Figure 1. Representation of the memory matrix state revision for mLSTM, Gated DeltaNet, and KronLSTM. Interference with stored values: mLSTM affects all values, Gated DeltaNet affects those with overlapping keys, and KronLSTM affects only those with overlapping key and value pairs.
Introduction
Large language models (LLMs) predominantly use the Transformer architecture ([41 , 54 , 56 ] (Qwen Team 2023)), whose softmax attention mechanism has computational costs that grow quadratically with context length, while its key–value cache grows linearly. Consequently, inference becomes increasingly costly as contexts grow. To reduce these costs, recent work has explored replacing softmax attention with linear attention mechanisms ([31 , 47 ] ), which can be traced back to the idea of fast weight programmers Schmidhuber, [48 ] . Linear attention compresses the context into a fixed-size memory matrix that is updated as new tokens arrive, yielding linear computational complexity and a constant memory footprint. Recently, recurrent architectures based on associative memories Anderson, [3 ] ; Kohonen [33 ] , including xLSTM Beck et al., [5 ] , as well as related state-space and memory-based approaches such as Mamba Dao and Gu, [18 ] ; Gu and Dao [22 ] ; Lahoti et al. [34 ] , DeltaNet Hatamizadeh et al., [24 ] ; Kimi Team et al. [32 ] ; Yang et al. [61 , 62 ] , RWKV-5 and 6 Peng et al., [43 ] , and other recent architectures Behrouz et al., [9 ] ; Liu et al. [35 ] ; Sun et al. [53 ] , have emerged as competitive alternatives to Transformers.
A crucial challenge for linear attention is controlling memory growth. If the memory of recurrent architectures grows without being bounded, its readout can become arbitrarily large, leading to numerical instabilities, particularly during learning. The original LSTM Hochreiter, [25 ] ; Hochreiter and Schmidhuber [26 ] uses a bounded squashing function to control the exposed cell state, while its forget gate regulates the accumulation of information in memory. RWKV-5 and RWKV-6 Peng et al., [43 ] similarly control memory growth through a forget gate. Mamba architectures Dao and Gu, [18 ] ; Gu and Dao [22 ] ; Lahoti et al. [34 ] also restrain memory growth through a forget mechanism derived from a stable state-space ordinary differential equation. However, forget-based memory control becomes increasingly restrictive for long contexts. To retain information over long time spans, forget gates must remain close to one; otherwise, information from the beginning of the context is progressively attenuated. xLSTM Beck et al., [5 ] addresses this limitation by allowing forget gates close to one and introducing normalizer and stabilizer states to control the scale of its matrix memory and readout. However, normalization based on ratios of potentially large quantities can itself lead to numerical difficulties and learning instabilities, particularly in larger xLSTM networks Beck et al., [7 ] . mLSTM’s numerical instabilities are illustrated by the stress test in Figure 2 .
Figure 2. Stress test with high learning rates for the same mLSTM and KronLSTM architectures on language modeling. Learning becomes unstable for mLSTM as the gradient norm spikes show, while KronLSTM yields stable learning.
The DeltaNet family Hatamizadeh et al., [24 ] ; Kimi Team et al. [32 ] ; Yang et al. [61 , 62 ] controls memory growth through Delta Rule updates, whose dynamics converge to a steady-state distribution with bounded memory ensuring stable learning dynamics. However, revising a stored value can perturb previously stored values, leading to unwanted memory interference for long sequences. We introduce KronLSTM, which, like (Gated) DeltaNet, provides bounded memory through a delta-style update, but uses the Kronecker Delta Rule. The Kronecker Delta Rule is less invasive: it modifies the memory only along a single direction in matrix space, reducing interference with previously stored associations. This targeted memory revision results in a less noisy gradient flow over time.
Our contributions are:
(i) Kronecker Delta Rule is introduced as a new update rule for associative matrix memories, which corrects the scalar associations while having minimal interference with previously stored values.
(ii) KronLSTM is introduced as a novel linear attention architecture based on the Kronecker Delta Rule equipped with a vector forget gate, a β \beta β gate, and a target gate.
(iii) Stability properties for the Kronecker Delta Rule are mathematically proved via bounds on the expected memory norm. These bounds ensure stable learning and proper gradient flow.
(iv) Experiments empirically validate the strong performance of KronLSTM at long-context language modeling and retrieval tasks. KronLSTM performs strongly across language modeling, recall, and state tracking benchmarks.
KronLSTM
In Subsection 2.1, we introduce the Kronecker Delta Rule and analyze its memory dynamics, and in Subsection 2.2, the Kronecker Delta Rule is embedded in the KronLSTM sequence-mixer block.
Kronecker Delta Rule
Linear attention update rules for matrix memory. Linear attention [31 ] compresses the context into a matrix memory S S S that is updated per token and read out per query,
S t + 1 = S t + v t k t T , h t = S t q t = ∑ i < t v i ( k i T q t ) , (1) S_{t+1} = S_t + v_t k_t^{\mathsf{T}}, \qquad h_t = S_t q_t = \sum_{i<t} v_i \left(k_i^{\mathsf{T}} q_t\right), \tag*{(1)} S t + 1 = S t + v t k t T , h t = S t q t = i < t ∑ v i ( k i T q t ) , (1)
with S t ∈ R d v × d k S_t \in\mathbb{R}^{d_v \times d_k} S t ∈ R d v × d k , keys k t k_t k t and queries q t q_t q t in R d k \mathbb{R}^{d_k} R d k , and values v t v_t v t in R d v \mathbb{R}^{d_v} R d v . Therefore, S S S is an associative memory [3 , 33 ] with a Hebbian rule, i.e., an outer product update rule. Modern linear attention architectures like xLSTM [5 ] , Mamba [18 , 22 , 34 ] , RWKV-5 and 6 [43 ] rely on these associative memories equipped with gates and transformations. The memory growth is controlled by a forget gate, which stabilizes the memory dynamics.
In contrast, the Delta Rule [47 , 59 ] corrects the memory with the difference of the retrieval to the current value when writing into the memory
S t + 1 = S t − β t ( S t k t − v t ) k t T = S t ( I − β t k t k t T ) + β t v t k t T , (2) S_{t+1} = S_t - \beta_t \left(S_t k_t - v_t\right) k_t^{\mathsf{T}} = S_t \left(I - \beta_t k_t k_t^{\mathsf{T}}\right) + \beta_t v_t k_t^{\mathsf{T}}, \tag*{(2)} S t + 1 = S t − β t ( S t k t − v t ) k t T = S t ( I − β t k t k t T ) + β t v t k t T , (2)
with step size β t \beta_t β t . Each step is gradient descent on 1 2 ∥ S t k t − v t ∥ 2 \frac{1}{2}\lVert S_t k_t - v_t\rVert^2 2 1 ∥ S t k t − v t ∥ 2 , and the transition I − β t k t k t T I - \beta_t k_t k_t^{\mathsf{T}} I − β t k t k t T is a generalized Householder matrix, which admits chunkwise-parallel training [61 ] .
The two rules differ in how an association, once stored, is changed. The Hebbian write of (1) is additive, so the memory grows with the sequence and an earlier value can be buried by later writes but never revised. The delta rule (2) revises: it removes the value currently stored at k t k_t k t before writing v t v_t v t . The correction, however, proceeds along ( S t k t − v t ) k t T \left(S_t k_t - v_t\right) k_t^{\mathsf{T}} ( S t k t − v t ) k t T , a direction determined by the current memory content, so values stored at keys that overlap with k t k_t k t are affected as well. We aim at an error-correcting write that acts solely on the pair ( k t , v t ) (k_t, v_t) ( k t , v t ) .
Kronecker Delta Rule. Let S t ∈ R d v × d k S_t \in\mathbb{R}^{d_v \times d_k} S t ∈ R d v × d k be the memory matrix, k t ∈ R d k k_t \in\mathbb{R}^{d_k} k t ∈ R d k and v t ∈ R d v v_t \in\mathbb{R}^{d_v} v t ∈ R d v the key and value at step t t t , c t ∈ R c_t \in\mathbb{R} c t ∈ R a target and β t \beta_t β t a step size. The scalar v t T S t k t v_t^{\mathsf{T}} S_t k_t v t T S t k t measures how strongly the memory associates k t k_t k t with v t v_t v t . The Kronecker Delta Rule corrects this scalar towards the target along the outer product of the pair as follows
S t + 1 = S t − β t ( v t T S t k t − c t ) v t k t T = S t − β t ( v t T S t k t ) v t k t T ⏟ erase + β t c t v t k t T ⏟ write . (3) S_{t+1} = S_t - \beta_t \left(v_t^{\mathsf{T}} S_t k_t - c_t\right) v_t k_t^{\mathsf{T}} = S_t - \underbrace{\beta_t \left(v_t^{\mathsf{T}} S_t k_t\right) v_t k_t^{\mathsf{T}}}_{\text{erase}} + \underbrace{\beta_t c_t v_t k_t^{\mathsf{T}}}_{\text{write}}. \tag*{(3)} S t + 1 = S t − β t ( v t T S t k t − c t ) v t k t T = S t − erase β t ( v t T S t k t ) v t k t T + write β t c t v t k t T . (3)
In place of the d v d_v d v -dimensional residual S t k t − v t S_t k_t - v_t S t k t − v t of the delta rule, a scalar is corrected, and the direction v t k t T v_t k_t^{\mathsf{T}} v t k t T of the correction is determined by the input pair alone without utilizing the memory content. A target c t ≡ 1 c_t \equiv1 c t ≡ 1 stores each association with unit strength, while c t = 0 c_t = 0 c t = 0 erases it. The target c t c_t c t may depend on the input, i.e., serves as a gate, so that what is written becomes input-dependent. The right-hand side of (3) separates the update into an erase and a write term, both scaled by β t \beta_t β t . Choosing the target c ~ t = c t / β t \tilde{c}_t = c_t / \beta_t c ~ t = c t / β t removes the step size from the write, so that β t \beta_t β t alone determines how much of the stored association is erased and c t c_t c t alone how much is written. KronLSTM uses this form, with both quantities computed from the input as gates (§ 2.2 ).
Properties of the Kronecker Delta Rule. (i) Error correction (see Appendix Property A2 and Property A6). An update is a step of gradient descent with step size β t \beta_t β t on 1 2 ( v t T S k t − c t ) 2 \frac{1}{2}\left(v_t^{\mathsf{T}} S k_t - c_t\right)^2 2 1 ( v t T S k t − c t ) 2 . Re-presenting the same pair gives v t T S t + 1 k t − c t = ( v t T S t k t − c t ) ( 1 − β t ∥ a t ∥ 2 ) v_t^{\mathsf{T}} S_{t+1} k_t - c_t = \left(v_t^{\mathsf{T}} S_t k_t - c_t\right)\left(1 - \beta_t \lVert a_t\rVert^2\right) v t T S t + 1 k t − c t = ( v t T S t k t − c t ) ( 1 − β t ∥ a t ∥ 2 ) , so it contracts the error iff 0 < β t < 2 / ∥ a t ∥ 2 0 < \beta_t < 2/\lVert a_t\rVert^2 0 < β t < 2/ ∥ a t ∥ 2 , and the Newton step β t = 1 / ∥ a t ∥ 2 \beta_t = 1/\lVert a_t\rVert^2 β t = 1/ ∥ a t ∥ 2 meets the target in a single update; for c t = 0 c_t = 0 c t = 0 this deletes the association. *(ii) Delta Rule in vectorized form (see Appendix Property Property A7). With the Kronecker product a t = k t ⊗ v t = vec ( v t k t ⊤ ) a_t = k_t \otimes v_t = \operatorname{vec}(v_t k_t^\top) a t = k t ⊗ v t = vec ( v t k t ⊤ ) and the vectorized memory s = vec ( S ) s = \operatorname{vec}(S) s = vec ( S ) , we have v t ⊤ S k t = a t ⊤ s v_t^\top S k_t = a_t^\top s v t ⊤ S k t = a t ⊤ s and ∥ a t ∥ 2 = ∥ k t ∥ 2 ∥ v t ∥ 2 \lVert a_t\rVert^2 = \lVert k_t\rVert^2 \lVert v_t\rVert^2 ∥ a t ∥ 2 = ∥ k t ∥ 2 ∥ v t ∥ 2 , and (3) becomes
s t + 1 = ( I − β t a t a t ⊤ ) s t + β t c t a t , (4) s_{t+1} = \left(I - \beta_t a_t a_t^\top\right)s_t + \beta_t c_t a_t, \tag*{(4)} s t + 1 = ( I − β t a t a t ⊤ ) s t + β t c t a t , (4)
the Widrow–Hoff Delta Rule (Widrow and Hoff, [59 ] ) with input a t a_t a t and target c t c_t c t , which gives the rule its name. The transition factor is a generalized Householder matrix that does not depend on c t c_t c t , so the chunkwise-parallel formulation of DeltaNet (Yang et al., [61 ] ) applies with a t a_t a t in place of k t k_t k t , the target entering only the driving term. For β t ∥ a t ∥ 2 ∈ ( 1 , 2 ) \beta_t \lVert a_t\rVert^2 \in(1, 2) β t ∥ a t ∥ 2 ∈ ( 1 , 2 ) the transition factor has a negative eigenvalue, the idea behind the state-tracking capabilities of Delta Rule models (Grazzi et al., [21 ] ).
(iii) Interference (see Appendix Property A9). For any other stored pair ( k i , v i ) (k_i, v_i) ( k i , v i ) , the update changes the association by
Δ ( v i ⊤ S k i ) = − β t ( v t ⊤ S k t − c t ) ( v i ⊤ v t ) ( k t ⊤ k i ) , (5) \Delta\left(v_i^\top S k_i\right) = -\beta_t\left(v_t^\top S k_t - c_t\right)\left(v_i^\top v_t\right)\left(k_t^\top k_i\right), \tag*{(5)} Δ ( v i ⊤ S k i ) = − β t ( v t ⊤ S k t − c t ) ( v i ⊤ v t ) ( k t ⊤ k i ) , (5)
the product of value overlap and key overlap. Orthogonality in either factor protects a stored association, whereas the Hebbian and the Delta rule interference is controlled by key overlap alone.
(iv) Capacity (see Appendix Property A11). Each triple ( k t , v t , c t ) (k_t, v_t, c_t) ( k t , v t , c t ) imposes one linear constraint a t ⊤ s = c t a_t^\top s = c_t a t ⊤ s = c t on the d k d v d_k d_v d k d v entries of S S S , so up to d k d v d_k d_v d k d v scalar associations can be stored exactly when the features { a i } \{a_i\} { a i } are linearly independent, against d k d_k d k value vectors for S k i = v i S k_i = v_i S k i = v i . In particular, a single key can hold several mutually orthogonal values, which the read-out S k S k S k returns as their sum. See Appendix A3 for more properties and more details on the above properties.
Convergence, steady state, and memory norm bounds. We will present expected bounds on the memory norm, which in turn imply stable memory dynamics that obviate numerical instabilities. Memories grow with the length of the context and the number of vectors or associations to store; therefore long contexts are most relevant when analyzing memory stability. We model a long context as a stream of i.i.d. triples ( k t , v t , c t ) ∼ p (k_t, v_t, c_t) \sim p ( k t , v t , c t ) ∼ p with fixed step size β \beta β and set
A ˉ : = E [ a a ⊤ ] ≻ 0 , b ˉ : = E [ c a ] , s L S : = A ˉ − 1 b ˉ , ρ : = c − a ⊤ s L S , κ : = β max i ∥ a i ∥ 2 , (6) \bar{A} := \mathbb{E}[a a^\top] \succ0,\qquad\bar{b} := \mathbb{E}[c a],\qquad s^{\mathrm{LS}} := \bar{A}^{-1}\bar{b},\qquad\rho:= c - a^\top s^{\mathrm{LS}},\qquad\kappa:= \beta\max_i \lVert a_i\rVert^2, \tag*{(6)} A ˉ := E [ a a ⊤ ] ≻ 0 , b ˉ := E [ c a ] , s LS := A ˉ − 1 b ˉ , ρ := c − a ⊤ s LS , κ := β i max ∥ a i ∥ 2 , (6)
so S L S = argmin S E [ ( v ⊤ S k − c ) 2 ] S^{\mathrm{LS}} = \operatorname*{argmin}_S \mathbb{E}[(v^\top S k-c)^2] S LS = argmin S E [( v ⊤ S k − c ) 2 ] is the least-squares memory and ρ \rho ρ the irreducible residual, with E [ ρ a ] = 0 \mathbb{E}[\rho a] = 0 E [ ρ a ] = 0 . Throughout, 0 < β ∥ a i ∥ 2 < 2 0 < \beta\lVert a_i\rVert^2 < 2 0 < β ∥ a i ∥ 2 < 2 , and q β : = 1 − β ( 2 − κ ) λ min ( A ˉ ) q_\beta:= 1 - \beta(2-\kappa)\lambda_{\min}(\bar{A}) q β := 1 − β ( 2 − κ ) λ m i n ( A ˉ ) .
Theorem 1 (Unbiased mean and steady state). The expected iterate obeys the exact recursion E [ s t + 1 ] = ( I − β A ˉ ) E [ s t ] + β b ˉ \mathbb{E}[s_{t+1}] = (I-\beta\bar{A})\mathbb{E}[s_t] + \beta\bar{b} E [ s t + 1 ] = ( I − β A ˉ ) E [ s t ] + β b ˉ , whose unique fixed point is s L S s^{\mathrm{LS}} s LS , so ∥ E [ S t ] − S L S ∥ F ≤ r t ∥ S 0 − S L S ∥ F \lVert\mathbb{E}[S_t]-S^{\mathrm{LS}}\rVert_F \le r^t\lVert S_0-S^{\mathrm{LS}}\rVert_F ∥ E [ S t ] − S LS ∥ F ≤ r t ∥ S 0 − S LS ∥ F with r = max k ∣ 1 − β λ k ( A ˉ ) ∣ < 1 r = \max_k \lvert1-\beta\lambda_k(\bar{A})\rvert< 1 r = max k ∣ 1 − β λ k ( A ˉ )∣ < 1 . Moreover the iterates form a time-homogeneous Markov chain on R d k d v \mathbb{R}^{d_kd_v} R d k d v whose kernel contracts on laws with finite second moment; by the standard theory of Markov chains on general state spaces (Meyn and Tweedie, 1993}) it has a unique stationary distribution π β \pi_\beta π β , the law of S t S_t S t converges to it geometrically from any initialization, and the stationary mean is exactly E π β [ S ] = S L S \mathbb{E}_{\pi_\beta}[S]=S^{\mathrm{LS}} E π β [ S ] = S LS .
Theorem 1 combines Appendix Theorem A6 and Appendix Theorem A7 which have been proven in Appendix A3.
Theorem 2 (Memory norm bounds). With λ min , λ max \lambda_{\min},\lambda_{\max} λ m i n , λ m a x the extreme eigenvalues of A ˉ \bar{A} A ˉ , the least-squares memory is sandwiched by the explained target energy,
E [ c 2 ] − E [ ρ 2 ] λ max ( A ˉ ) ≤ ∥ S L S ∥ F 2 ≤ E [ c 2 ] − E [ ρ 2 ] λ min ( A ˉ ) , (7) \frac{\mathbb{E}[c^2]-\mathbb{E}[\rho^2]}{\lambda_{\max}(\bar{A})}
\le\lVert S^{\mathrm{LS}}\rVert_F^2
\le
\frac{\mathbb{E}[c^2]-\mathbb{E}[\rho^2]}{\lambda_{\min}(\bar{A})}, \tag*{(7)} λ m a x ( A ˉ ) E [ c 2 ] − E [ ρ 2 ] ≤ ∥ S LS ∥ F 2 ≤ λ m i n ( A ˉ ) E [ c 2 ] − E [ ρ 2 ] , (7) and all its singular values obey the same upper bound; residual energy can only shrink it. In the non-expansive regime κ ≤ 1 \kappa\le1 κ ≤ 1 , the steady state and — started at S 0 = 0 S_0=0 S 0 = 0 — every iterate satisfies
E π β [ ∥ S ∥ F 2 ] ≤ E [ c 2 ] λ min ( A ˉ ) , E [ ∥ S t ∥ F 2 ] ≤ ( 1 + q β t / 2 ) 2 E [ c 2 ] λ min ( A ˉ ) ≤ 4 E [ c 2 ] λ min ( A ˉ ) , (8) \mathbb{E}_{\pi_\beta}\left[\lVert S\rVert_F^2\right]
\le\frac{\mathbb{E}[c^2]}{\lambda_{\min}(\bar{A})},\qquad
\mathbb{E}\left[\lVert S_t\rVert_F^2\right]
\le\left(1+q_\beta^{t/2}\right)^2\frac{\mathbb{E}[c^2]}{\lambda_{\min}(\bar{A})}
\le\frac{4\mathbb{E}[c^2]}{\lambda_{\min}(\bar{A})}, \tag*{(8)} E π β [ ∥ S ∥ F 2 ] ≤ λ m i n ( A ˉ ) E [ c 2 ] , E [ ∥ S t ∥ F 2 ] ≤ ( 1 + q β t /2 ) 2 λ m i n ( A ˉ ) E [ c 2 ] ≤ λ m i n ( A ˉ ) 4 E [ c 2 ] , (8) the transient factor decays geometrically from 4 to 1.
Theorem 2 combines Appendix Theorem A8, Appendix Theorem A9, and Appendix Theorem A10 which are proven in Appendix § 9 . From the first step onward, the expected memory of the rule is bounded by a constant determined by the data distribution alone. More details and theoretical results are given in Appendix § 9 .
The KronLSTM architecture
In modern deep learning architectures for sequence modeling, the matrix update rules are stabilized and enhanced with gating mechanisms. The mLSTM (Beck et al., [5 ] ) places the Hebbian write (1) under a sigmoid forget gate and an exponential input gate, and Gated DeltaNet (Yang et al., [62 ] ) decays the memory of (2) by a scalar before the correction,
S t + 1 = f t S t + i t v t v k T , S t + 1 = f t S t ( I − β t k t k t T ) + β t v t v k T . (9) S_{t+1}=f_tS_t+i_tv_tv_k^{\mathsf T},\qquad S_{t+1}=f_tS_t\left(I-\beta_tk_tk_t^{\mathsf T}\right)+\beta_tv_tv_k^{\mathsf T}. \tag*{(9)} S t + 1 = f t S t + i t v t v k T , S t + 1 = f t S t ( I − β t k t k t T ) + β t v t v k T . (9)
Kimi Delta Attention (Kimi Team et al., [32 ] ) makes the decay (key) channel-wise, and Gated DeltaNet-2 (Hatamizadeh et al., [24 ] ) replaces the single β t \beta_t β t by separate channel-wise erase and write gates. KronLSTM proceeds in the same way: the step size β t \beta_t β t and the target c t c_t c t of (3) become gates computed from the input; a forget gate decays the memory, and the rule is embedded in the projections, short convolutions, normalizations, and output gate of a standard sequence-mixer layer. All quantities below are per head, with key and value dimensions d k d_k d k and d v d_v d v ; we suppress the head index and write x t ∈ R d x_t \in\mathbb{R}^d x t ∈ R d for the layer input at step t t t .
KronLSTM update. The forget gate is a diagonal matrix F t F_t F t with diagonal entries in ( 0 , 1 ) (0,1) ( 0 , 1 ) . It acts on the key dimension and hence decays the memory from the right, S t F t S_tF_t S t F t , retaining each key direction separately. We use target c ~ t = c t / β t \tilde{c}_t=c_t/\beta_t c ~ t = c t / β t to decouple erase and write. The erase term of (3) is evaluated on the decayed memory S t F t S_tF_t S t F t . We obtain
S t + 1 = S t F t ⏟ forget − β t ( v t T S t F t k t ) v t k t T ⏟ erase + c t v t k t T ⏟ write , (10) S_{t+1}=\underbrace{S_tF_t}_{\text{forget}}-\underbrace{\beta_t\left(v_t^{\mathsf T}S_tF_tk_t\right)v_tk_t^{\mathsf T}}_{\text{erase}}+\underbrace{c_tv_tk_t^{\mathsf T}}_{\text{write}}, \tag*{(10)} S t + 1 = forget S t F t − erase β t ( v t T S t F t k t ) v t k t T + write c t v t k t T , (10)
so that forgetting, erasing, and writing are controlled by three separate gates, F t F_t F t , β t \beta_t β t and c t c_t c t . The standard coupled and the unit-target variant c t ≡ 1 c_t\equiv1 c t ≡ 1 are compared in Appendix § 12 , Table A3 .
Table A3 : Write-rule ablation at 1.5B/100B, relative to KRONLSTM. Avg-6 is the mean of the five commonsense reasoning tasks plus LAMBADA accuracy. Recall is the six-task real-world recall suite at 2k context. RULER is averaged at each evaluation length. Best per column in bold , second best underlined.
Table A3.
Gates. The erase gate β t \beta_t β t and the write gate c t c_t c t are per-head scalars obtained from linear projections of the input followed by a sigmoid,
β t = σ ( w β T x t + b β ) ∈ ( 0 , 1 ) , c t = σ ( w c T x t + b c ) ∈ ( 0 , 1 ) , (11) \beta_t=\sigma\left(w_\beta^{\mathsf T}x_t+b_\beta\right)\in(0,1),\qquad c_t=\sigma\left(w_c^{\mathsf T}x_t+b_c\right)\in(0,1), \tag*{(11)} β t = σ ( w β T x t + b β ) ∈ ( 0 , 1 ) , c t = σ ( w c T x t + b c ) ∈ ( 0 , 1 ) , (11)
both bounded by construction. The diagonal vector f t f_t f t of the forget gate F t = diag ( f t ) F_t=\operatorname{diag}(f_t) F t = diag ( f t ) is obtained by raising a sigmoid gate to a learned positive per-head power, following the parameterization of Mamba (Gu and Dao, [22 ] ) and Gated DeltaNet-2 (Hatamizadeh et al., [24 ] )
f t = σ ( − ( s f ) t ) α = ( 1 − σ ( ( s f ) t ) ) α , ( s f ) t = W f x t + b f ∈ R d k , α = exp ( a log ) ∈ R > 0 , (12) f_t=\sigma\left(-(s_f)_t\right)^\alpha=\left(1-\sigma\left((s_f)_t\right)\right)^\alpha,\qquad(s_f)_t=W_fx_t+b_f\in\mathbb{R}^{d_k},\qquad\alpha=\exp(a_{\log})\in\mathbb{R}_{>0}, \tag*{(12)} f t = σ ( − ( s f ) t ) α = ( 1 − σ ( ( s f ) t ) ) α , ( s f ) t = W f x t + b f ∈ R d k , α = exp ( a l o g ) ∈ R > 0 , (12)
where a log a_{\log} a l o g is a single learned scalar per head, broadcast over the d k d_k d k key dimensions, and the exponent is taken component-wise. The sigmoid sets retention separately for each key direction as a function of the input. At the same time, the exponent rescales the head’s retention horizon by 1 / α 1/\alpha 1/ α , so that heads can operate at widely different timescales without high pre-activations. Since a log a_{\log} a l o g is learned in log space, the baseline retention timescale spans orders of magnitude across heads, while within a head the per-direction variation comes entirely from the input. The read-out is gated by a sigmoid output gate o t = σ ( W o x t + b o ) ∈ ( 0 , 1 ) d v o_t=\sigma(W_ox_t+b_o)\in(0,1)^{d_v} o t = σ ( W o x t + b o ) ∈ ( 0 , 1 ) d v .
KronLSTM block. Queries, keys, and values are linear projections of x t x_t x t , each followed by a short causal depthwise convolution and a SiLU activation, and are then ℓ 2 \ell_2 ℓ 2 -normalized with q t , k t ∈ R d k q_t,k_t\in\mathbb{R}^{d_k} q t , k t ∈ R d k and v t ∈ R d v v_t\in\mathbb{R}^{d_v} v t ∈ R d v . The memory S t ∈ R d v × d k S_t\in\mathbb{R}^{d_v\times d_k} S t ∈ R d v × d k is updated by (10) and read with the query, h ~ t = S t q t ∈ R d v \tilde{h}_t=S_tq_t\in\mathbb{R}^{d_v} h ~ t = S t q t ∈ R d v . The read-out passes through the sigmoid output gate and a per-head RMSNorm, and the concatenated heads are mixed by the output projection,
h t = W o u t RMSNorm ( o t ⊙ h ~ t ) . h_t=W_{\mathrm{out}}\operatorname{RMSNorm}\left(o_t\odot\tilde{h}_t\right). h t = W out RMSNorm ( o t ⊙ h ~ t ) .
With ℓ 2 \ell_2 ℓ 2 -normalized keys and values and a sigmoid β t \beta_t β t , we have β t ∥ k t ∥ 2 ∥ v t ∥ 2 = β t < 1 \beta_t \lVert k_t\rVert^2 \lVert v_t\rVert^2 = \beta_t < 1 β t ∥ k t ∥ 2 ∥ v t ∥ 2 = β t < 1 , so every update is non-expansive and the memory operates in the regime of Theorem 2 ; the forget gate, with entries below one, only contracts it further. The architecture of the KronLSTM block is shown in Figure 3 . It carries no state besides the memories S t S_t S t of its heads: there is no normalizer state and no stabilizer state as in the mLSTM. KronLSTM can serve as a drop-in replacement for the attention layer in a Transformer block, analogous to mLSTM and sLSTM in the xLSTM architecture.
Figure 3. Architecture of the KronLSTM block.
Experimental Validation
We experimentally validate the KronLSTM architecture on different benchmarks with a focus on language modeling tasks. We compare KronLSTM with the following recurrent linear attention architectures: Gated DeltaNet (GDN) [62 ] , Kimi Delta Attention (KDA) [32 ] , Mamba-3 [34 ] , and Gated DeltaNet-2 (GDN-2) [24 ] . Recent deployments of linear attention in frontier models follow two hybrid-setting paradigms: either interleave the linear layers with a small number of softmax attention layers [32 , 37 , 40 ] or use sliding-window attention (SWA) instead [14 , 24 ] . The advantage of the latter is keeping the memory footprint constant with respect to the context length. Therefore, we evaluate every architecture in two settings: as a pure linear attention model, and as a hybrid with SWA layers. For the hybrid setting, we employ a window size of 2048 at a 5:1 ratio of linear attention to SWA layers. All models are pretrained on the FineWeb-Edu dataset using the Llama2 tokenizer [55 ] .
We train at a context of 8k, motivated by previous works [6 , 8 , 23 ] , where the throughput advantage of linear attention compared to the softmax attention emerges from 8k context onward. We evaluate the models within and extrapolating outside their context length (§ 3.2 ). The experiments in the following sections show 1.5B parameter models trained on 100B tokens. In Appendix § 12 , we provide a set of experiments at 400M model size.
Language Modeling
We assess the performance of KronLSTM compared to leading linear attention variants on standard language modeling benchmarks, concretely on five commonsense reasoning tasks (HellaSwag [63 ] , PIQA [12 ] , ARC-easy and ARC-challenge [17 ] , WinoGrande [46 ] ) and on LAMBADA [42 ] , reporting accuracy on all six tasks, their average (Avg-6), and LAMBADA perplexity. Table 1 shows the results of both linear and hybrid models. Among all competitors, KronLSTM has the highest average accuracy over the six tasks and the lowest perplexity on LAMBADA. As a linear model, KronLSTM performs best on five out of six tasks and improves on average on the strongest competitor, KDA, by 1.6 accuracy points. The hybrid setting confirms KronLSTM’s performance advantage, leading on four of six tasks and achieving the lowest perplexity on LAMBADA. Hybrid models with sliding-window attention change the averages by at most about one accuracy point. While mLSTM, Mamba-3 and GDN-2 benefit most from adding SWA, GDN and KDA change only minimally; KronLSTM remains robust, leading among all configurations.
Table 1. Downstream performance of 1.5B/100B models. Avg-6 is the mean accuracy over the five commonsense reasoning tasks and LAMBADA. Best per column in bold, second best underlined, separately for linear and hybrid models.
Retrieval Capabilities
On long-context tasks, the Kronecker Delta Rule should improve the most over previous update rules, as interference grows with context length. We assess the performance of KronLSTM on two benchmarks with different context lengths.
The first benchmark is the real-world recall suite of Arora et al. [4 ] (SWDE, SQuAD, FDA, TriviaQA, NQ, DROP) for 2k context. Results are given in Table 2 . KronLSTM has the best average accuracy, and it is the best method in three out of six tasks in both pure linear attention models and hybrid models. In general, hybrid models using SWA gain five to nine accuracy points.
Table 2: Results for 1.5B models on the real-world recall suite, given in accuracies for 2k context length. Best per column in bold , second best underlined.
Table 2.
The second benchmark is the RULER needle-in-a-haystack (NIAH) tasks (Hsieh et al., [29 ] ) where models are evaluated at 4k, 8k and 16k context length, with 16k assessing the extrapolation capabilities of the models. Results are presented in Table 3 . Among all linear attention models, KronLSTM is the best or second-best method across all sequence lengths, except for task S1. KronLSTM has the highest accuracy on S3 and MQ tasks across all sequence lengths. Among all hybrid models, KronLSTM achieves the highest accuracy (averaged over all lengths) on all tasks and has the highest accuracy at 16k on each task, confirming its strong long context capabilities. The improvement of KronLSTM over competitors like Gated DeltaNet is consistent with the interference analysis of Property A9. On S2 and S3, whose needles are unique sentences embedded in natural language text, KronLSTM outperforms GDN except for S2 at 16k. For hybrid models, KronLSTM outranks GDN-2 and Mamba-3 by 15.2 accuracy points on S2 at 4k. KronLSTM’s margin increases on MQ and MV with sequence length. On MQ, KronLSTM’s lead distance grows from 2.4 to 4.1 accuracy points when increasing the context from 4k to 16k. On MV, KronLSTM is 3.5 accuracy points behind the best method, GDN, for 4k context, while KronLSTM is the leading method for 8k and 16k, with a 1.1 and 3.8 accuracy point lead, respectively.
Table 3. Results of 1.5B models on the NIAH tasks for 8k pretraining context and 4k, 8k, and 16k evaluation context. S1–S3: S-NIAH-1/2/3; MK: MK-NIAH-1; MQ: MQ-NIAH; MV: MV-NIAH. Best per task and length in bold , second best underlined, separately for linear and hybrid models.
For longer contexts, the memory updates of GDN more frequently interfere with already stored content and perturb it, resulting in more degraded performance at 16k. In contrast, the Kronecker Delta Rule is less invasive and performs better at longer contexts. The exception is S1, where KronLSTM underperforms. S1 is the most artificial task with frequently repeated noise sentences, which is unrealistic for real-world text. We attribute the performance drop of the KronLSTM in this task to the residual accumulation for recurring keys. Hybrid models with SWA have a considerably smaller S1 accuracy, except KronLSTM. The hybrid models containing the baseline linear attention layers reach at most 51.8, 26.6, and 12.6 accuracy for 4k, 8k, and 16k context length, respectively. These accuracies closely match the fraction of the context covered by the attention window, indicating that the needle is primarily retrieved when it lies within the window, with Mamba-3 and mLSTM dropping to zero at 16k. The results suggest limited reliance on the recurrent architecture’s memory. KronLSTM exceeds these accuracies on average across all context lengths and achieves the best performance across all six tasks at 16k. § 3.3 examines how these methods perform after context extension.
Long Context Extension
In applications, language models do not extrapolate to longer context but are trained on them, typically by continued pretraining at increased context length [20 ] (Fu et al. 2024). This stage unlocks retrieval far beyond the original pretraining length (Fu et al., 2024). Linear attention models have no positional encoding to adapt; rather, they must learn to keep more associations over longer time spans. We therefore branch the 1.5B linear KronLSTM and GDN-2 runs at 75B tokens and train them for the remaining 25B tokens at a sequence length of 32k on a long-document mixture derived from ProLong-64K [20 ] , so that the context-length-extended models are token-matched to the 100B models of the previous sections. GDN-2 serves as a reference model in this experiment since it also decouples erase and write. Further details on training are described in Appendix § 13.1 . Table 4 shows the commonsense reasoning, LAMBADA, and recall results of the extended models. Both models benefit from length extension on the reasoning and recall tasks, and perplexity improves by about 4 perplexity points. KronLSTM keeps its lead: it has the higher average accuracy and the lower perplexity, and it is better on five of the six recall tasks.
Table 4. Long-context (32k) extension results for GDN-2 and KronLSTM models. Avg-6 is the mean accuracy of the five reasoning tasks plus LAMBADA acc; the remaining columns give accuracy on the real-world recall suite evaluated at 2048 tokens. The best per column is in bold .
State Tracking Capabilities
Results for context extension at the RULER tasks are given in Figure 4 and Table A4 . As expected, KronLSTM shows its advantage by extending the training context length. While both architectures perform strongly on single-needle retrieval at shorter contexts, KronLSTM increasingly outperforms GDN-2 on the more challenging NIAH tasks, ranging from 8k to 32k. At 16k, this advantage is particularly pronounced on S-3 and MV, with KronLSTM leading on most tasks except S1, where GDN-2 remains stronger. At 32k, KronLSTM maintains its lead across all five tasks, although both architectures experience a substantial drop in retrieval accuracy compared to 16k. The gap between the two models widens with haystack size, as predicted by the interference analysis in § 2.1 : longer contexts accumulate more interference, and KronLSTM’s update rule suffers less of it.
Figure 4. GDN-2 and KronLSTM results on NIAH for 32k context extension for linear and SWA-hybrid models.
Table A4: RULER NIAH for the 32k-context-extended linear models, evaluated on identical documents. Better of the two per task and length in bold .
Table A4.
State Tracking Capabilities
KronLSTM can perform state tracking by setting β t = 2 σ ( w β ⊤ x t + b β ) ∈ ( 0 , 2 ) \beta_t = 2\sigma(w_{\beta}^{\top} x_t + b_{\beta}) \in(0, 2) β t = 2 σ ( w β ⊤ x t + b β ) ∈ ( 0 , 2 ) , as described in Grazzi et al. [21 ] for DeltaNet. We evaluate KronLSTM’s state tracking capabilities on parity, S 3 S_3 S 3 permutation, and modulo arithmetic. We train up to sequence length 128 and evaluate up to 1024. Figure 5 shows that KronLSTM has similar state tracking capabilities as Gated DeltaNet on the parity and S 3 S_3 S 3 tasks, both ahead of Mamba-3, which wins on modulo arithmetic.
Figure 5. State tracking results for Gated DeltaNet (GDN), Mamba-3, and KronLSTM.
Training Efficiency
Figure 6 reports end-to-end training throughput of the purely linear 1.5B models on a single NVIDIA H100 (80 GB), following Hatamizadeh et al. [24 ] : each step processes 16,384 tokens, split from 2K×8 to 16K×1 (sequence length × batch size), and all models share the same training stack. KronLSTM reaches 36.8–44.5K tokens/s, 11–18% more than GDN-2 at every shape and within 4–8% of KDA up to 8K×2. GDN, whose scalar decay admits the cheapest kernels, remains the fastest. Throughput is nearly flat while a step holds at least two sequences; at 16K×1 all models slow down, since the inter-chunk recurrence parallelizes only over sequences and heads.
Figure 6. Training throughput of KronLSTM against baselines on a single H100-80GB.
Conclusion
We have introduced the Kronecker Delta Rule, an update rule for associative matrix memories that corrects a scalar association along a single direction, and KronLSTM, which embeds this rule in a gated linear attention mixer. We proved that the memory dynamics converge to a unique steady state and that the expected memory norm is bounded at every step. KronLSTM has the best average accuracy on commonsense reasoning and real-world recall, both as a linear model and as a hybrid. Averaged over the context lengths, it is the best or second-best linear model on every natural-text RULER task, and on S1 it is the only one that retrieves needles beyond the sliding attention window in hybrid models. After extending the context to 32k, KronLSTM outperforms GDN-2 on all natural-text retrieval tasks from 16k onward. Future work includes scaling up KronLSTM and training hybrid models containing full softmax attention layers.
AI use statement
The Kronecker Delta Rule and the KronLSTM architecture originated with the authors without AI assistance. LLMs were used to assist in developing the theoretical results and in implementing the method. Claude Fable 5 and Claude Fable 5.1 assisted in deriving the properties of the Kronecker Delta Rule and the convergence and memory norm bounds of § 2.1 and Appendix § 9 ; every statement and proof was checked independently by the authors, who take full responsibility for their correctness. Claude Fable 5 was further used to examine candidate formulations of the Kronecker Delta Rule, in particular the coupled, decoupled, and unit-target write variants, for their viability; the choice of the form in (10) based on was made by the authors on the basis of ablation studies such as in Appendix § 12 . Fable 5.1 was used to assist in deriving the chunkwise-parallel formulation of KronLSTM. The formulation was subsequently checked thoroughly by our team. Claude Fable 5 and Claude Fable 5.1 were used to optimize the chunkwise-parallel and recurrent kernels of KronLSTM; the resulting kernels were numerically verified against naive reference implementations written by the authors. We did not use generative AI tools to interpret results or to design experiments. We have not used generative AI tools to generate synthetic data or to propose hypotheses, and translation, dataset cleaning and reformatting, and qualitative or thematic data analysis do not apply to this work. Additionally, we used Claude Fable 5 and Claude Opus 5 to draft parts of the manuscript and to edit it for readability. All AI-drafted text was revised by the authors. We have reviewed all AI-assisted work and take responsibility for the final content of this work, including text, claims, and artifacts produced with the aid of generative AI.
Acknowledgments
This work was supported by the Austrian Science Fund (FWF) 10.55776/COE12 and the European Union’s Horizon Europe research and innovation program under grant agreement number 101214398 (ELLIOT). The ELLIS Unit Linz, the LIT AI Lab, and the Institute for Machine Learning are supported by the Federal State of Upper Austria. We acknowledge the EuroHPC Joint Undertaking for awarding us access to Leonardo at CINECA, Italy, to Marenostrum 5 at the Barcelona Supercomputing Centre (BSC), Spain, and to JUPITER at Jülich Supercomputing Centre (JSC), Germany. We would like to thank Stephanie Holly for checking the mathematical proofs of A3.5.4 and Bálint László Szarvas for proofreading and making suggestions to this manuscript.
References
Appendices
Notation Table
Table A1. Notation used throughout the paper and the appendix.
Related Work
To the best of our knowledge, no existing work uses exactly this update that is a scalar bilinear read-out v T S k v^{\mathsf{T}}Sk v T S k regressed onto the constant target 1. The established delta-rule linear attention family uses the vector read-out S k → v Sk \to v S k → v . The rule studied here is therefore the bilinear-readout, scalar-target member of the delta-rule. The constraints are ⟨ v k T , S ⟩ F = 1 \langle v_k^{\mathsf{T}}, S\rangle_{\mathrm{F}} = 1 ⟨ v k T , S ⟩ F = 1 rather than S k = v Sk = v S k = v .
Long Short-Term Memory
KronLSTM belongs to the xLSTM ([5 ] ) family that revisits LSTM ([25 , 26 ] ) for the era of large language models (LLMs) with two modifications: exponential gating and a redesigned memory. mLSTM replaces the scalar cell state by a matrix memory and drops memory mixing, making the recurrence fully parallelizable over the sequence. mLSTM enables recurrent LLMs with fast, constant-memory inference at scale ([6 ] ), and Beck et al. [8 ] establish scaling behavior competitive with Transformers at linear time complexity. xLSTM has also been applied to language modeling ([7 ] ), vision ([2 ] ), biological and chemical sequences ([49 ] ), and robotics ([50 ] ).
The matrix memory of mLSTM can be viewed as a classical correlation-based associative memory ([3 , 33 ] ) embedded within LSTM-style gating. In addition to providing an efficient mechanism for storing and retrieving associations, the exponential gates allow mLSTM to revise storage decisions. This distinguishes mLSTM from purely accumulative linear attention mechanisms. KronLSTM follows this line of research by retaining the matrix memory and the ability to revise storage decisions, while introducing a different memory-update rule.
A central challenge of recurrent architectures with growing memory is maintaining stable memory dynamics over long sequences. The original LSTM uses a bounded squashing function for the exposed cell state, while xLSTM introduces a normalizer state and a stabilizer state to control the scale of its matrix memory and retrievals. However, normalization based on ratios of large quantities can lead to numerical and optimization difficulties, particularly in large networks ([7 ] ). MC-LSTM ([28 ] ) instead redistributes its cell state with column-stochastic matrices, which conserve mass and never amplify the state. These considerations motivate alternative matrix-memory update rules that retain memory revision while providing stable memory dynamics. The Kronecker Delta Rule addresses this issue and provides convergence to a steady-state distribution with a norm-bounded memory in expectation. Furthermore, KronLSTM yields advantageous gradient flow over time compared with mLSTM ([27 ] ).
The delta-rule linear attention family
The direct ancestor is the fast weight programmer perspective of Schlag et al. [47 ] , who introduced the delta-rule write S t + 1 = S t − β ( S t k t − v t ) k t T S_{t+1} = S_t - \beta(S_t k_t - v_t)k_t^{\mathsf{T}} S t + 1 = S t − β ( S t k t − v t ) k t T into linear attention ([31 ] ), building on the fast-weight programmers of the early 1990s ([48 ] ). Yang et al. [61 ] (DeltaNet) made this rule hardware-efficient by exploiting the WY representation of the resulting products of generalized Householder matrices. This is exactly the product structure the bilinear rule inherits, with the Kronecker product k t ⊗ v t k_t \otimes v_t k t ⊗ v t taking the role of k t k_t k t . Subsequent extensions include Gated DeltaNet ([62 ] ), which adds a scalar forget gate α t S t \alpha_t S_t α t S t ; RWKV-7 ([44 ] ), which employs a generalized delta rule with a decoupled in-context learning rate; and DeltaProduct ([51 ] ), which applies multiple Householder factors per token to improve state tracking. This is directly relevant to the negative-eigenvalue regime β ∥ a t ∥ 2 ∈ ( 1 , 2 ) \beta\lVert a_t \rVert^2 \in(1,2) β ∥ a t ∥ 2 ∈ ( 1 , 2 ) of the transition factors I − β a t a t T I - \beta a_t a_t^{\mathsf{T}} I − β a t a t T , which allows state-tracking.
The online-learning perspective
Longhorn ([35 ] ) derives state-space recurrences explicitly as closed-form solutions of per-token online learning objectives; the bilinear rule fits this template with the objective 1 2 ( v T S k − 1 ) 2 \frac{1}{2}(v^{\mathsf{T}}Sk - 1)^2 2 1 ( v T S k − 1 ) 2 . Test-time training (TTT) ([53 ] ) and Titans ([9 ] ) treat the memory as a model trained by gradient descent at inference time. Most relevant as a unifying framework is the Miras taxonomy of Behrouz et al. [10 ] , which classifies memory architectures by their attentional-bias objective, retention mechanism, and optimizer. The new bilinear rule occupies a new place of that taxonomy: it is a bilinear attentional bias with a constant scalar target. A recent related observation is that test-time training with key–value binding is secretly linear attention Liu et al., [36 ] . The mesa-optimization line von Oswald et al., [58 ] provides the complementary least-squares-solver perspective on such layers.
Classical update and learning rules
When vectorized, the rule is the Widrow–Hoff LMS / delta rule Widrow and Hoff [59 ] applied to the Kronecker product input a t = k t ⊗ v t a_t = k_t \otimes v_t a t = k t ⊗ v t with target 1. The stability range β ∥ a t ∥ 2 ∈ ( 0 , 2 ) \beta\lVert a_t\rVert^2 \in(0, 2) β ∥ a t ∥ 2 ∈ ( 0 , 2 ) is the classical LMS step-size condition. The projection interpretation is Kaczmarz’s method Kaczmarz [30 ] : with the Newton step β ⋆ = 1 / ( ∥ k ∥ 2 ∥ v ∥ 2 ) \beta^\star= 1/(\lVert k\rVert^2 \lVert v\rVert^2) β ⋆ = 1/ (∥ k ∥ 2 ∥ v ∥ 2 ) , the update is the orthogonal projection onto the hyperplane { S : ⟨ v k ⊤ , S ⟩ F = 1 } \{S : \langle vk^\top, S\rangle_{\mathrm{F}} = 1\} { S : ⟨ v k ⊤ , S ⟩ F = 1 } , and general β \beta β gives relaxed projections. The modern convergence theory, including the linear rate under random sampling of constraints, is due to Strohmer and Vershynin [52 ] . The connections between Kaczmarz iterations, stochastic gradient descent, and learning are surveyed by Chen [16 ] . Fejér monotonicity and relaxed projection methods are standard in the projection-methods literature Cegielski [15 ] .
Relation to cognitive and neuroscience
Because S S S scores key–value pairs rather than retrieving values, the closest relative in the cognitive and neuroscience literature is familiarity (recognition) memory: the anti-Hebbian model of familiarity discrimination in perirhinal cortex of Bogacz and Brown [13 ] shows that scoring memories attain a far higher capacity than retrieval memories, which is the same d 2 d^2 d 2 -versus-d d d capacity separation derived for the bilinear rule. The classical retrieval-side background is given by correlation matrix memories [3 , 33 ] for the Hebbian write, and by delta-rule / pseudoinverse associative memories for capacity under Widrow–Hoff learning Prados and Kak [45 ] .
Kronecker Delta Rule: Setting, Name, Properties
Setup and notation.
We use the notation of the main paper (§ 2.1 ), specialized to d k = d v = : d d_k = d_v =: d d k = d v =: d as in our experiments (where d q k = d v d_{qk} = d_v d q k = d v ); the general d v × d k d_v \times d_k d v × d k case is identical with d 2 d^2 d 2 replaced by d k d v d_kd_v d k d v . Let S t ∈ R d × d S_t \in\mathbb{R}^{d \times d} S t ∈ R d × d be the memory matrix, k t ∈ R d k_t \in\mathbb{R}^d k t ∈ R d the key and v t ∈ R d v_t \in\mathbb{R}^d v t ∈ R d the value at step t t t , c t ∈ R c_t \in\mathbb{R} c t ∈ R a scalar target, and β t \beta_t β t a step size. The scalar v t ⊤ S t k t v_t^\top S_t k_t v t ⊤ S t k t measures how strongly the memory associates k t k_t k t with v t v_t v t ; the Kronecker Delta Rule corrects it towards the target c t c_t c t along the outer product v t k t ⊤ v_t k_t^\top v t k t ⊤ ,
S t + 1 = S t − β t ( v t ⊤ S t k t − c t ) v t k t ⊤ , (13) S_{t+1} = S_t - \beta_t \left(v_t^\top S_t k_t - c_t\right) v_t k_t^\top, \tag*{(13)} S t + 1 = S t − β t ( v t ⊤ S t k t − c t ) v t k t ⊤ , (13)
which is Eq. (3) of the main paper. For the analysis, we abbreviate the scalar read-out and its residual,
y t ≔ v t ⊤ S t k t y_t \coloneqq v_t^\top S_t k_t y t : = v t ⊤ S t k t
(scalar read-out),
ψ t ≔ y t − c t \psi_t \coloneqq y_t - c_t ψ t : = y t − c t
(residual),
so that the update reads compactly S t + 1 = S t − β t r t v t k t ⊤ S_{t+1} = S_t - \beta_t r_t v_t k_t^\top S t + 1 = S t − β t r t v t k t ⊤ . The target sequence c t c_t c t subsumes different variants: c t ≡ 1 c_t \equiv1 c t ≡ 1 is the basic Kronecker Delta Rule with target 1, c t ≡ c > 0 c_t \equiv c > 0 c t ≡ c > 0 the Constant-Target Kronecker Delta Rule, c t = ∥ v t ∥ 2 c_t = \lVert v_t\rVert^2 c t = ∥ v t ∥ 2 the Norm-Target Kronecker Delta Rule, and c t = ∥ k t ∥ ∥ v t ∥ c_t = \lVert k_t\rVert\lVert v_t\rVert c t = ∥ k t ∥ ∥ v t ∥ the Double-Norm-Target Kronecker Delta Rule. The target c t c_t c t may also be produced by a gate or a network, so that what is written becomes itself input-dependent. Throughout we use v ⊤ S k = Tr ( S k v ⊤ ) = ⟨ v k ⊤ , S ⟩ F v^\top Sk = \operatorname{Tr}(Skv^\top) = \langle vk^\top, S\rangle_{\mathrm{F}} v ⊤ S k = Tr ( S k v ⊤ ) = ⟨ v k ⊤ , S ⟩ F .
Name Kronecker Delta Rule
We use the Kronecker product
a t = k t ⊗ v t (14) a_t = k_t \otimes v_t \tag*{(14)} a t = k t ⊗ v t (14)
in a vectorized update rule. a t a_t a t is a vector of length d 2 d^2 d 2 , where the vector v t v_t v t is multiplied by each component of k t k_t k t and then stacked to one vector. However, we have v t k t ⊤ = v t ⊗ k t v_t k_t^\top= v_t \otimes k_t v t k t ⊤ = v t ⊗ k t . Let s = vec ( S ) ∈ R d 2 s = \operatorname{vec}(S) \in\mathbb{R}^{d^2} s = vec ( S ) ∈ R d 2 and a t = k t ⊗ v t ∈ R d 2 a_t = k_t \otimes v_t \in\mathbb{R}^{d^2} a t = k t ⊗ v t ∈ R d 2 , so that v t ⊤ S k t = a t ⊤ s v_t^\top S k_t = a_t^\top s v t ⊤ S k t = a t ⊤ s . The rule becomes
s t + 1 = ( I − β a t a t ⊤ ) s t + β c t a t , (15) s_{t+1} = \left(I - \beta a_t a_t^\top\right) s_t + \beta c_t a_t, \tag*{(15)} s t + 1 = ( I − β a t a t ⊤ ) s t + β c t a t , (15)
which is exactly Delta Rule with Kronecker product input a t \mathbf{a}_t a t and target c t c_t c t .
Properties
Property A1 (Memory Space is Hilbert Space and Kronecker Update Rule is a Projection onto a Hyperplane). The memory space R d × d \mathbb{R}^{d \times d} R d × d is a Hilbert space endowed with the Frobenius inner product
⟨ X , Y ⟩ F = Tr ( X T Y ) , \langle X,Y\rangle_{\mathrm{F}}=\operatorname{Tr}\left(X^{\mathsf{T}}Y\right), ⟨ X , Y ⟩ F = Tr ( X T Y ) , where X , Y ∈ R d × d X,Y \in\mathbb{R}^{d \times d} X , Y ∈ R d × d . Hence the target condition
v t T S k t = c t \mathbf{v}_t^{\mathsf{T}}S\mathbf{k}_t=c_t v t T S k t = c t is a hyperplane in this Hilbert space. The update
S t + 1 = S t − β ( v t T S k t − c t ) v t k t T S_{t+1}=S_t-\beta\left(\mathbf{v}_t^{\mathsf{T}}S\mathbf{k}_t-c_t\right)\mathbf{v}_t\mathbf{k}_t^{\mathsf{T}} S t + 1 = S t − β ( v t T S k t − c t ) v t k t T is a orthogonal projection onto it, if
β = 1 ∥ k t ∥ 2 2 ∥ v t ∥ 2 2 . \beta=\frac{1}{\lVert\mathbf{k}_t\rVert_2^2\lVert\mathbf{v}_t\rVert_2^2}. β = ∥ k t ∥ 2 2 ∥ v t ∥ 2 2 1 . The update
S t + 1 = S t − β ( v t T S k t − c t ) v t k t T S_{t+1}=S_t-\beta\left(\mathbf{v}_t^{\mathsf{T}}S\mathbf{k}_t-c_t\right)\mathbf{v}_t\mathbf{k}_t^{\mathsf{T}} S t + 1 = S t − β ( v t T S k t − c t ) v t k t T is a relaxed orthogonal projection onto it, with relaxation parameter
ω t = β ∥ k t ∥ 2 2 ∥ v t ∥ 2 2 . \omega_t=\beta\lVert\mathbf{k}_t\rVert_2^2\lVert\mathbf{v}_t\rVert_2^2. ω t = β ∥ k t ∥ 2 2 ∥ v t ∥ 2 2 .
Property A2 (Exact online gradient descent: a bilinear delta rule with target c t c_t c t ). The rule (A1) is stochastic gradient descent with step size β \beta β on the instantaneous loss
L t ( S ) = 1 2 ( v t T S k t − c t ) 2 , ∇ S L t ( S ) = ( v t T S k t − c t ) v t k t T . (16) L_t(S)=\frac{1}{2}\left(\mathbf{v}_t^{\mathsf{T}}S\mathbf{k}_t-c_t\right)^2,\qquad\nabla_S L_t(S)=\left(\mathbf{v}_t^{\mathsf{T}}S\mathbf{k}_t-c_t\right)\mathbf{v}_t\mathbf{k}_t^{\mathsf{T}}. \tag*{(16)} L t ( S ) = 2 1 ( v t T S k t − c t ) 2 , ∇ S L t ( S ) = ( v t T S k t − c t ) v t k t T . (16) It differs from DeltaNet, which performs SGD on 1 2 ∥ S k t − v t ∥ 2 \frac{1}{2}\lVert S\mathbf{k}_t-\mathbf{v}_t\rVert^2 2 1 ∥ S k t − v t ∥ 2 with update S t + 1 = S t − β ( S k t − v t ) k t T S_{t+1}=S_t-\beta(S\mathbf{k}_t-\mathbf{v}_t)\mathbf{k}_t^{\mathsf{T}} S t + 1 = S t − β ( S k t − v t ) k t T : there the target is the vector v t \mathbf{v}_t v t (imposing d d d constraints per pair), whereas here the target is the scalar c t c_t c t and the pair ( k t , v t ) (\mathbf{k}_t,\mathbf{v}_t) ( k t , v t ) jointly forms the input, imposing a single bilinear constraint v t T S k t = c t \mathbf{v}_t^{\mathsf{T}}S\mathbf{k}_t=c_t v t T S k t = c t per pair.
Property A3 (Superposition: the memory is linear in the target sequence). For a fixed pair sequence ( k t , v t ) t = 1 T (\mathbf{k}_t,\mathbf{v}_t)_{t=1}^{T} ( k t , v t ) t = 1 T and step size β \beta β , the map from the target sequence c = ( c 1 , … , c T ) \mathbf{c}=(c_1,\ldots,c_T) c = ( c 1 , … , c T ) to the final memory S T ( c ) S_T(\mathbf{c}) S T ( c ) with S 0 = 0 S_0=0 S 0 = 0 is linear:
S T ( c + c ′ ) = S T ( c ) + S T ( c ′ ) , S T ( α c ) = α S T ( c ) , (17) S_T(\mathbf{c}+\mathbf{c}')=S_T(\mathbf{c})+S_T(\mathbf{c}'),\qquad S_T(\alpha\mathbf{c})=\alpha S_T(\mathbf{c}), \tag*{(17)} S T ( c + c ′ ) = S T ( c ) + S T ( c ′ ) , S T ( α c ) = α S T ( c ) , (17) explicitly, with a t = k t ⊗ v t \mathbf{a}_t=\mathbf{k}_t\otimes\mathbf{v}_t a t = k t ⊗ v t ,
vec ( S T ( c ) ) = β ∑ t = 1 T c t [ ∏ τ = t + 1 T ( I − β a τ a τ T ) ] a t . (18) \operatorname{vec}\left(S_T(\mathbf{c})\right)=\beta\sum_{t=1}^{T}c_t\left[\prod_{\tau=t+1}^{T}\left(I-\beta\mathbf{a}_\tau\mathbf{a}_\tau^{\mathsf{T}}\right)\right]\mathbf{a}_t. \tag*{(18)} vec ( S T ( c ) ) = β t = 1 ∑ T c t [ τ = t + 1 ∏ T ( I − β a τ a τ T ) ] a t . (18) Memories written with different target sequences over the same pairs superpose exactly. The pure-gain homogeneity of the constant-target rule ( S t ( c ) = c S t ( 1 ) ) (S_t^{(c)}=cS_t^{(1)}) ( S t ( c ) = c S t ( 1 ) ) is the special case c = c 1 \mathbf{c}=c\mathbf{1} c = c 1 . The pair sequence determines where the memory can write (the reachable subspace span { a 1 , … , a T } \operatorname{span}\{\mathbf{a}_1,\ldots,\mathbf{a}_T\} span { a 1 , … , a T } and the interference pattern); the target sequence linearly selects what is written into it.
Property A4 (Per-step reparameterization to the unit target). For every step with c t ≠ 0 c_t\ne0 c t = 0 , rule (A1) with pair ( k t , v t ) (\mathbf{k}_t,\mathbf{v}_t) ( k t , v t ) and step β \beta β coincides exactly with the unit-target rule applied to the rescaled value v ~ t = v t / c t \widetilde{\mathbf{v}}_t=\mathbf{v}_t/c_t v t = v t / c t with the time-varying effective step β ~ t = β c t 2 \widetilde{\beta}_t=\beta c_t^2 β t = β c t 2 :
S − β ( v T S k − c t ) v k T = S − β ~ t ( v ~ T S k − 1 ) v ~ k T . (19) S-\beta\left(\mathbf{v}^{\mathsf{T}}S\mathbf{k}-c_t\right)\mathbf{v}\mathbf{k}^{\mathsf{T}}
=
S-\widetilde{\beta}_t\left(\widetilde{\mathbf{v}}^{\mathsf{T}}S\mathbf{k}-1\right)\widetilde{\mathbf{v}}\mathbf{k}^{\mathsf{T}}. \tag*{(19)} S − β ( v T S k − c t ) v k T = S − β t ( v T S k − 1 ) v k T . (19) In contrast to the constant-target case, the effective step size now varies over time, so the time-varying rule is not globally a reparameterized unit-target rule; steps with c t = 0 c_t=0 c t = 0 have no unit-target counterpart at all (see Property A5 (iii)).
Property A5 (Distinguished targets). Three choices of c t c_t c t are structurally special.
(i) Norm target, c t = ∥ v t ∥ 2 c_t = \lVert v_t\rVert^2 c t = ∥ v t ∥ 2 . The residual becomes the value-projected retrieval error r t = v t T ( S k t − v t ) r_t = v_t^{\mathsf T}(S k_t - v_t) r t = v t T ( S k t − v t ) ; every exact retrieval memory ( S k i = v i ) (S k_i = v_i) ( S k i = v i ) satisfies all constraints, and in the autoassociative case k t = v t k_t = v_t k t = v t the identity S = I S = I S = I is a universal fixed point.
(ii) Correlation target, c t = v t T k t c_t = v_t^{\mathsf T} k_t c t = v t T k t . The identity S = I S = I S = I is a universal fixed point even heteroassociatively: v T I k = v T k v^{\mathsf T} I k = v^{\mathsf T} k v T I k = v T k for all pairs. The rule then learns deviations of the bilinear map from the standard inner product, i.e. S S S is naturally parameterized around the identity, and the read-out v T S k − v T k v^{\mathsf T} S k - v^{\mathsf T} k v T S k − v T k measures the learned (in-context ) part of the similarity.
(iii) Erasure target, c t = 0 c_t = 0 c t = 0 . The update drives v t T S k t → 0 v_t^{\mathsf T} S k_t \to0 v t T S k t → 0 : with the Newton step β ⋆ = 1 / ( ∥ k t ∥ 2 ∥ v t ∥ 2 ) \beta^\star= 1/(\lVert k_t\rVert^2 \lVert v_t\rVert^2) β ⋆ = 1/ (∥ k t ∥ 2 ∥ v t ∥ 2 ) the association is deleted in one shot, while every read-out with k ′ ⊥ k t k' \perp k_t k ′ ⊥ k t or v ′ ⊥ v t v' \perp v_t v ′ ⊥ v t is exactly unchanged. Hence, c t = 0 c_t = 0 c t = 0 is a targeted unlearning of a single key–value binding. Alternating write targets ( c t ≠ 0 ) (c_t \ne0) ( c t = 0 ) and erasure targets ( c t = 0 ) (c_t = 0) ( c t = 0 ) turns the memory into an editable scalar store (see also Property A10 on overwriting).
Property A6 (One-step error contraction and stability range for β \beta β ). The target never enters the homogeneous part of the dynamics. Re-reading with the same pair and the same target after one update,
v t T S t + 1 k t = y t − β r t ∥ k t ∥ 2 ∥ v t ∥ 2 (20) v_t^{\mathsf T} S_{t+1} k_t = y_t - \beta r_t \lVert k_t\rVert^2 \lVert v_t\rVert^2 \tag*{(20)} v t T S t + 1 k t = y t − β r t ∥ k t ∥ 2 ∥ v t ∥ 2 (20) ⟺ v t T S t + 1 k t − c t = y t − c t − β r t ∥ k t ∥ 2 ∥ v t ∥ 2 (21) \Longleftrightarrow v_t^{\mathsf T} S_{t+1} k_t - c_t = y_t - c_t - \beta r_t \lVert k_t\rVert^2 \lVert v_t\rVert^2 \tag*{(21)} ⟺ v t T S t + 1 k t − c t = y t − c t − β r t ∥ k t ∥ 2 ∥ v t ∥ 2 (21) ⟺ r t + 1 = r t − β r t ∥ k t ∥ 2 ∥ v t ∥ 2 (22) \Longleftrightarrow r_{t+1} = r_t - \beta r_t \lVert k_t\rVert^2 \lVert v_t\rVert^2 \tag*{(22)} ⟺ r t + 1 = r t − β r t ∥ k t ∥ 2 ∥ v t ∥ 2 (22) ⟺ r t + 1 = ( 1 − β ∥ k t ∥ 2 ∥ v t ∥ 2 ) r t , (23) \Longleftrightarrow r_{t+1} = \left(1 - \beta\lVert k_t\rVert^2 \lVert v_t\rVert^2\right) r_t , \tag*{(23)} ⟺ r t + 1 = ( 1 − β ∥ k t ∥ 2 ∥ v t ∥ 2 ) r t , (23) where r t + 1 r_{t+1} r t + 1 is computed for the same ( k t , v t , c t ) (k_t, v_t, c_t) ( k t , v t , c t ) . The residual contracts if and only if
0 < β < 2 ∥ k t ∥ 2 ∥ v t ∥ 2 , (24) 0 < \beta< \frac{2}{\lVert k_t\rVert^2 \lVert v_t\rVert^2}, \tag*{(24)} 0 < β < ∥ k t ∥ 2 ∥ v t ∥ 2 2 , (24) and the loss shrinks by the factor ( 1 − β ∥ k t ∥ 2 ∥ v t ∥ 2 ) 2 \left(1 - \beta\lVert k_t\rVert^2 \lVert v_t\rVert^2\right)^2 ( 1 − β ∥ k t ∥ 2 ∥ v t ∥ 2 ) 2 . Since L t L_t L t is quadratic along the ray S − η r t v t k t T S - \eta r_t v_t k_t^{\mathsf T} S − η r t v t k t T , the choice
β ⋆ = 1 ∥ k t ∥ 2 ∥ v t ∥ 2 (25) \beta^\star= \frac{1}{\lVert k_t\rVert^2 \lVert v_t\rVert^2} \tag*{(25)} β ⋆ = ∥ k t ∥ 2 ∥ v t ∥ 2 1 (25) is a Newton step: the constraint v t T S k t = c t v_t^{\mathsf T} S k_t = c_t v t T S k t = c t is interpolated exactly in one update, for every value of c t c_t c t .
Property A7 (Delta Rule / Widrow–Hoff LMS in vectorized form). Let s = vec ( S ) ∈ R d 2 s = \operatorname{vec}(S) \in\mathbb{R}^{d^2} s = vec ( S ) ∈ R d 2 and a t = k t ⊗ v t ∈ R d 2 a_t = k_t \otimes v_t \in\mathbb{R}^{d^2} a t = k t ⊗ v t ∈ R d 2 the Kronecker product, so that v t T S k t = a t T s v_t^{\mathsf T} S k_t = a_t^{\mathsf T} s v t T S k t = a t T s and ∥ a t ∥ 2 = ∥ k t ∥ 2 ∥ v t ∥ 2 \lVert a_t\rVert^2 = \lVert k_t\rVert^2 \lVert v_t\rVert^2 ∥ a t ∥ 2 = ∥ k t ∥ 2 ∥ v t ∥ 2 . The rule becomes
s t + 1 = ( I − β a t a t T ) s t + β c t a t , (26) s_{t+1} = \left(I - \beta a_t a_t^{\mathsf T}\right)s_t + \beta c_t a_t , \tag*{(26)} s t + 1 = ( I − β a t a t T ) s t + β c t a t , (26) which is exactly the Widrow–Hoff LMS/delta rule with input a t a_t a t and target c t c_t c t . This is the classical setting of LMS, which regresses a time-varying desired signal onto the input. The transition matrix I − β a t a t T I - \beta a_t a_t^{\mathsf T} I − β a t a t T does not depend on c t c_t c t : it has eigenvalue 1 − β ∥ a t ∥ 2 1 - \beta\lVert a_t\rVert^2 1 − β ∥ a t ∥ 2 on span ( a t ) \operatorname{span}(a_t) span ( a t ) and eigenvalue 1 1 1 on the orthogonal complement (dimension d 2 − 1 d^2 - 1 d 2 − 1 ). Unrolling the recurrence,
s T = ∏ t = 1 T ( I − β a t a t T ) s 0 + β ∑ t = 1 T c t [ ∏ τ = t + 1 T ( I − β a τ a τ T ) ] a t , (27) s_T = \prod_{t=1}^{T}\left(I - \beta a_t a_t^{\mathsf T}\right)s_0 + \beta\sum_{t=1}^{T} c_t \left[\prod_{\tau=t+1}^{T}\left(I - \beta a_\tau a_\tau^{\mathsf T}\right)\right] a_t , \tag*{(27)} s T = t = 1 ∏ T ( I − β a t a t T ) s 0 + β t = 1 ∑ T c t [ τ = t + 1 ∏ T ( I − β a τ a τ T ) ] a t , (27) gives the same product-of-generalized-Householder structure as DeltaNet. Hence, the WY-representation / chunkwise-parallel training techniques apply verbatim with a t = k t ⊗ v t a_t = k_t \otimes v_t a t = k t ⊗ v t in place of k t k_t k t , and the targets c t c_t c t enter only the driving (inhomogeneous) term. Expressivity consequence: for β ∥ a t ∥ 2 ∈ ( 1 , 2 ) \beta\lVert a_t\rVert^2 \in(1,2) β ∥ a t ∥ 2 ∈ ( 1 , 2 ) the factor has a negative eigenvalue (reaching − 1 -1 − 1 at β ∥ a t ∥ 2 = 2 \beta\lVert a_t\rVert^2 = 2 β ∥ a t ∥ 2 = 2 ), the same mechanism that gives DeltaNet-style models state-tracking power beyond linear RNNs restricted to nonnegative eigenvalues. This is independent of the target sequence.
Property A8 (Homogeneous part of the update in non-expansive). For β ∥ a t ∥ 2 ∈ [ 0 , 2 ] \beta\lVert a_t\rVert^2 \in[0,2] β ∥ a t ∥ 2 ∈ [ 0 , 2 ] every factor I − β a t a t ⊤ I-\beta a_ta_t^\top I − β a t a t ⊤ has spectral norm at most 1, so the homogeneous part of the recurrence never amplifies and ∥ S t ∥ \lVert S_t\rVert ∥ S t ∥ cannot blow up. In contrast the Hebbian write S t + 1 = S t + v t k t ⊤ S_{t+1}=S_t+v_tk_t^\top S t + 1 = S t + v t k t ⊤ of vanilla linear attention leads to states that grow without bound. Moreover, the update is surgical: it modifies S S S only along the rank-one direction v t k t ⊤ v_tk_t^\top v t k t ⊤ . For any test pair with k ′ ⊥ k t k'\perp k_t k ′ ⊥ k t or v ′ ⊥ v t v'\perp v_t v ′ ⊥ v t the read-out v ′ ⊤ S k ′ v'^\top S k' v ′⊤ S k ′ is exactly unchanged.
Property A9 (Doubly gated interference). For a previously stored pair ( k i , v i ) (k_i,v_i) ( k i , v i ) , an update with ( k t , v t , c t ) (k_t,v_t,c_t) ( k t , v t , c t ) perturbs the stored read-out by
Δ ( v i ⊤ S k i ) = − β r t ( v i ⊤ v t ) ( k t ⊤ k i ) , (28) \Delta\left(v_i^\top S k_i\right)=-\beta r_t\left(v_i^\top v_t\right)\left(k_t^\top k_i\right), \tag*{(28)} Δ ( v i ⊤ S k i ) = − β r t ( v i ⊤ v t ) ( k t ⊤ k i ) , (28) i.e. interference is proportional to the product of the key overlap and the value overlap. Orthogonality in either factor suffices to protect a memory, which is a milder interference profile than the Hebbian rule (crosstalk from key overlap alone) and than DeltaNet (gated by key overlap only). The formula is target-independent, since the target enters only through the residual r t r_t r t . Consequently, writes whose read-out is already correct (r t ≈ 0 r_t\approx0 r t ≈ 0 ) cause no interference at all, regardless of c t c_t c t .
Property A10 (Relaxed Kaczmarz iteration, Fejér monotonicity, convergence, and overwriting). The constraint v t ⊤ S k t = c t v_t^\top S k_t=c_t v t ⊤ S k t = c t defines the hyperplane H t = { S : ⟨ v t k t ⊤ , S ⟩ = c t } H_t=\{S:\langle v_tk_t^\top,S\rangle=c_t\} H t = { S : ⟨ v t k t ⊤ , S ⟩ = c t } in R d × d \mathbb{R}^{d\times d} R d × d with Frobenius geometry. With β = β ∗ \beta=\beta^* β = β ∗ the update (A1) is the orthogonal projection onto H t H_t H t ; general β \beta β gives a relaxed projection. In contrast to a constant target, consistency of the system { v i ⊤ S k i = c i } i \{v_i^\top S k_i=c_i\}_i { v i ⊤ S k i = c i } i is now a genuine condition: it holds if and only if the target vector ( c i ) i (c_i)_i ( c i ) i lies in the range of the matrix stacking the rows a i ⊤ a_i^\top a i ⊤ . In particular, this always holds, whenever the { a i } \{a_i\} { a i } are linearly independent (at most d 2 d^2 d 2 of them). If the system is consistent, then for every solution S ∗ S^* S ∗ and any β t ∈ [ 0 , 2 / ∥ a t ∥ 2 ] \beta_t\in[0,2/\lVert a_t\rVert^2] β t ∈ [ 0 , 2/ ∥ a t ∥ 2 ] ,
∥ S t + 1 − S ∗ ∥ F ≤ ∥ S t − S ∗ ∥ F (29) \lVert S_{t+1}-S^*\rVert_F\leq\lVert S_t-S^*\rVert_F \tag*{(29)} ∥ S t + 1 − S ∗ ∥ F ≤ ∥ S t − S ∗ ∥ F (29) and cycling or randomly sampling over the pairs converges linearly to a solution, at a rate governed by the smallest singular value of the stacked a i ⊤ a_i^\top a i ⊤ . From S 0 = 0 S_0=0 S 0 = 0 the limit is the minimum-Frobenius-norm solution. The canonical inconsistent case is a repeated pair with a changed target, ( k , v , c ) (k,v,c) ( k , v , c ) followed later by ( k , v , c ′ ) (k,v,c') ( k , v , c ′ ) with c ′ ≠ c c'\ne c c ′ = c : the two hyperplanes are parallel, and the iteration resolves the conflict by overwriting. The contraction of Property A6 drives the read-out to the most recent target, in one shot for β = β ∗ \beta=\beta^* β = β ∗ . In expectation over a stationary distribution of triples ( k , v , c ) (k,v,c) ( k , v , c ) , the fixed-point condition is
E [ ( a ⊤ s − c ) a ] = 0 , (30) \mathbb{E}\left[(a^\top s-c)a\right]=0, \tag*{(30)} E [ ( a ⊤ s − c ) a ] = 0 , (30) the normal equation of the least-squares regression of the target c c c on the feature a = k ⊗ v a=k\otimes v a = k ⊗ v : the memory is an online linear regressor that learns, in context, to predict the scalar signal c t c_t c t from the key–value pair.
Property A11 (Capacity: memory can store d 2 d^2 d 2 associations). Each triple ( k i , v i , c i ) (k_i,v_i,c_i) ( k i , v i , c i ) contributes one linear constraint on the d 2 d^2 d 2 parameters of S S S , so up to d 2 d^2 d 2 arbitrary scalars c i c_i c i can be stored exactly for pairs with linearly independent { k i ⊗ v i } \{k_i\otimes v_i\} { k i ⊗ v i } (d 2 d^2 d 2 scalar components). Compare to DeltaNet with goal S k i = v i S k_i=v_i S k i = v i that represent d d d value vectors. In an orthonormal design ({ k i } \{k_i\} { k i } orthonormal and { v i } \{v_i\} { v i } orthonormal), the minimum-norm solution has the closed form
S ∗ = ∑ i c i v i k i ⊤ , v j ⊤ S ∗ k i = c i δ i j , (31) S^*=\sum_i c_i v_i k_i^\top,\qquad v_j^\top S^*k_i=c_i\delta_{ij}, \tag*{(31)} S ∗ = i ∑ c i v i k i ⊤ , v j ⊤ S ∗ k i = c i δ ij , (31) a target-weighted Hebbian superposition realizing the learned Kronecker delta with per-pair gains. The memory is thus best understood as a pair-addressed scalar table. It stores a number c i c_i c i at the address ( k i , v i ) (k_i,v_i) ( k i , v i ) , writes by rank-one updates, reads by the bilinear probe v ⊤ S k v^\top S k v ⊤ S k , rewrites by presenting the same pair with a new target, and deletes via c = 0 c=0 c = 0 (Property A5 (iii)).
Next, we exemplify the fact that KronLSTM can store d 2 d^2 d 2 numbers. Assume d d d orthonormal keys { k 1 , … , k d } \{k_1,\ldots,k_d\} { k 1 , … , k d } and d d d orthonormal values { v 1 , … , v d } \{v_1,\ldots,v_d\} { v 1 , … , v d } . Then we can build d 2 d^2 d 2 key-value pairs ( k i , v j ) (k_i,v_j) ( k i , v j ) with their associated d 2 d^2 d 2 numbers c i j c_{ij} c ij . We set β = 1 \beta=1 β = 1 and start with S 0 = 0 S_0=0 S 0 = 0 , then we iteratively update S t S_t S t with randomly chosen i i i and j j j :
S t + 1 = S t − β ( v j ⊤ S t k i − c i j ) v j k i ⊤ . (32) S_{t+1}=S_t-\beta\left(v_j^\top S_t k_i-c_{ij}\right)v_jk_i^\top. \tag*{(32)} S t + 1 = S t − β ( v j ⊤ S t k i − c ij ) v j k i ⊤ . (32)
If all pairs ( k i , v j ) (k_i, v_j) ( k i , v j ) or numbers c i j c_{ij} c ij are seen at least once, the update converged. If pair ( k i , v j ) (k_i, v_j) ( k i , v j ) has already been used in the update rule, then v j ⊤ S t k i = c i j v_j^\top S_t k_i = c_{ij} v j ⊤ S t k i = c ij and S t + 1 = S t S_{t+1} = S_t S t + 1 = S t . Therefore, if all pairs are seen, S t S_t S t does not change any more and the update converged. If pair ( k i , v j ) (k_i, v_j) ( k i , v j ) has NOT been stored, then v j ⊤ S t k i = 0 v_j^\top S_t k_i = 0 v j ⊤ S t k i = 0 and S t + 1 = S t + c i j v j k i ⊤ S_{t+1} = S_t + c_{ij}v_j k_i^\top S t + 1 = S t + c ij v j k i ⊤ . Therefore, if all pairs ( k i , v j ) (k_i, v_j) ( k i , v j ) or numbers c i j c_{ij} c ij are seen at least once, the memory matrix S t S_t S t is
S t = ∑ i = 1 , j = 1 d , d c i j v j k i ⊤ = V C ⊤ K ⊤ , (33) S_t = \sum_{i=1,j=1}^{d,d} c_{ij}v_j k_i^\top= V C^\top K^\top, \tag*{(33)} S t = i = 1 , j = 1 ∑ d , d c ij v j k i ⊤ = V C ⊤ K ⊤ , (33)
with
K = [ k 1 , … , k d ] ∈ R d × d , V = [ v 1 , … , v d ] ∈ R d × d , C = ( c i j ) i , j = 1 d ∈ R d × d . (34) K = [k_1,\ldots,k_d] \in\mathbb{R}^{d \times d}, \quad V = [v_1,\ldots,v_d] \in\mathbb{R}^{d \times d}, \quad C = (c_{ij})_{i,j=1}^{d} \in\mathbb{R}^{d \times d}. \tag*{(34)} K = [ k 1 , … , k d ] ∈ R d × d , V = [ v 1 , … , v d ] ∈ R d × d , C = ( c ij ) i , j = 1 d ∈ R d × d . (34)
Since v j ⊤ S t k i = c i j v_j^\top S_t k_i = c_{ij} v j ⊤ S t k i = c ij , we can now retrieve d 2 d^2 d 2 stored numbers. We have
C ⊤ = V ⊤ S t K , (35) C^\top= V^\top S_t K, \tag*{(35)} C ⊤ = V ⊤ S t K , (35)
since K K K and V V V are orthogonal. The transpose appears because the first index of C C C is the key index while the first index of S t S_t S t is the value index.
Property A12 (Norm and energy dynamics). Using ⟨ v t k t ⊤ , S t ⟩ F = y t = r t + c t \langle v_t k_t^\top, S_t\rangle_{\mathrm{F}} = y_t = r_t + c_t ⟨ v t k t ⊤ , S t ⟩ F = y t = r t + c t ,
∥ S t + 1 ∥ F 2 − ∥ S t ∥ F 2 = − 2 β r t ( r t + c t ) + β 2 r t 2 ∥ k t ∥ 2 ∥ v t ∥ 2 , (36) \lVert S_{t+1}\rVert_{\mathrm{F}}^2 - \lVert S_t\rVert_{\mathrm{F}}^2 = -2\beta r_t(r_t+c_t) + \beta^2 r_t^2 \lVert k_t\rVert^2 \lVert v_t\rVert^2 , \tag*{(36)} ∥ S t + 1 ∥ F 2 − ∥ S t ∥ F 2 = − 2 β r t ( r t + c t ) + β 2 r t 2 ∥ k t ∥ 2 ∥ v t ∥ 2 , (36)
so within the stable range, the state norm stays bounded whenever the constraint system is consistent. This is a consequence of Fejér monotonicity: the distance to any fixed solution is non-increasing.
Summary. The time-varying-target rule (A1) is the most general member of the Kronecker Delta Rule family: SGD on the scalar quadratic loss 1 2 ( v t ⊤ S k t − c t ) 2 \frac{1}{2}(v_t^\top S k_t-c_t)^2 2 1 ( v t ⊤ S k t − c t ) 2 , equivalently Widrow–Hoff LMS on the feature a t = k t ⊗ v t a_t = k_t \otimes v_t a t = k t ⊗ v t with desired signal c t c_t c t , equivalently relaxed Kaczmarz on the hyperplanes ⟨ v t k t ⊤ , S ⟩ F = c t \langle v_t k_t^\top,S\rangle_{\mathrm{F}} = c_t ⟨ v t k t ⊤ , S ⟩ F = c t . The entire analytic framework is target-independent. That comprises stability range and Newton step β ∗ \beta^* β ∗ , generalized-Householder transition factors with their parallelization and negative-eigenvalue state-tracking regime, doubly gated interference, Fejér-monotone linear convergence, d 2 d^2 d 2 capacity. The targets enter only the driving term, linearly : memories superpose in ( c t ) (c_t) ( c t ) , and in an orthonormal design the memory converges to the target-weighted Hebbian superposition ∑ i c i v i k i ⊤ \sum_i c_i v_i k_i^\top ∑ i c i v i k i ⊤ realizing v j ⊤ S k i = c i δ i j v_j^\top S k_i = c_i\delta_{ij} v j ⊤ S k i = c i δ ij . Distinguished targets recover c t ≡ 1 c_t \equiv1 c t ≡ 1 , c t ≡ c c_t \equiv c c t ≡ c , c t = ∥ v t ∥ 2 c_t = \lVert v_t\rVert^2 c t = ∥ v t ∥ 2 and add two new members: the correlation target c t = v t ⊤ k t c_t = v_t^\top k_t c t = v t ⊤ k t , for which the identity is a universal fixed point and S S S parameterizes learned deviations from the standard inner product, and the erasure target c t = 0 c_t = 0 c t = 0 , which deletes a single binding in one Newton step. Functionally, S S S is a pair-addressed scalar table with write, read, rewrite, and delete operations, therefore it is an editable d 2 d^2 d 2 -slot scalar memory. In the stochastic view, it is an online in-context regressor of the signal c t c_t c t on the bilinear feature.
Minimally Invasive Memory Revision
The Kronecker Delta Rule is not merely a correction of the addressed association: at the Newton step it is the smallest possible one, in every sense in which smallness can reasonably be asked for. This section elaborates on this topic. Throughout, S t ∈ R d × d S_t \in\mathbb{R}^{d \times d} S t ∈ R d × d , the read-out of the pair ( k t , v t ) (k_t,v_t) ( k t , v t ) is v t ⊤ S t k t v_t^\top S_t k_t v t ⊤ S t k t , the target is c t c_t c t , and
r t : = v t ⊤ S t k t − c t (37) r_t := v_t^\top S_t k_t - c_t \tag*{(37)} r t := v t ⊤ S t k t − c t (37)
is the residual that should be pushed to zero.
The admissible set
A correction is a matrix E ∈ R d × d E \in\mathbb{R}^{d \times d} E ∈ R d × d added to the memory. It corrects the addressed association if v t ⊤ ( S t + E ) k t = c t v_t^\top(S_t+E)k_t = c_t v t ⊤ ( S t + E ) k t = c t . Using v t ⊤ E k t = ⟨ v t k t ⊤ , E ⟩ F v_t^\top E k_t = \langle v_t k_t^\top,E\rangle_{\mathrm{F}} v t ⊤ E k t = ⟨ v t k t ⊤ , E ⟩ F , the set of such corrections is
E t : = { E ∈ R d × d : ⟨ v t k t ⊤ , E ⟩ F = − r t } , (38) \mathcal{E}_t := \left\{ E \in\mathbb{R}^{d \times d} : \langle v_t k_t^\top,E\rangle_{\mathrm{F}} = -r_t \right\}, \tag*{(38)} E t := { E ∈ R d × d : ⟨ v t k t ⊤ , E ⟩ F = − r t } , (38)
an affine hyperplane in the Hilbert space ( R d × d , ⟨ ⋅ , ⋅ ⟩ F ) (\mathbb{R}^{d\times d},\langle\cdot,\cdot\rangle_{\mathrm{F}}) ( R d × d , ⟨ ⋅ , ⋅ ⟩ F ) : it is nonempty, and it is a proper affine subspace because v t k t T ≠ 0 v_t k_t^{\mathsf{T}} \ne0 v t k t T = 0 whenever the pair is nondegenerate. Every statement below compares members of E t \mathcal{E}_t E t . We hold the requirement fixed (the addressed association must come out right) and ask which correction achieves it with the least collateral change. Write
E ⋆ ≔ − r t ∥ k t ∥ 2 ∥ v t ∥ 2 v t k t T = c t − v t T S t k t ∥ k t ∥ 2 ∥ v t ∥ 2 v t k t T , (39) E^\star\coloneqq-\frac{r_t}{\lVert k_t\rVert^2\lVert v_t\rVert^2}v_t k_t^{\mathsf{T}} = \frac{c_t-v_t^{\mathsf{T}}S_t k_t}{\lVert k_t\rVert^2\lVert v_t\rVert^2}v_t k_t^{\mathsf{T}}, \tag*{(39)} E ⋆ : = − ∥ k t ∥ 2 ∥ v t ∥ 2 r t v t k t T = ∥ k t ∥ 2 ∥ v t ∥ 2 c t − v t T S t k t v t k t T , (39)
which is exactly the Kronecker Delta Rule update at the Newton step β = β ⋆ = 1 / ( ∥ k t ∥ 2 ∥ v t ∥ 2 ) \beta=\beta^\star=1/(\lVert k_t\rVert^2\lVert v_t\rVert^2) β = β ⋆ = 1/ (∥ k t ∥ 2 ∥ v t ∥ 2 ) .
Minimality in norm
Theorem A1 (The Newton step is the smallest correction). E ⋆ ∈ E t E^\star\in\mathcal{E}_t E ⋆ ∈ E t , and for every E ∈ E t E\in\mathcal{E}_t E ∈ E t
∥ E ∥ F ≥ ∣ r t ∣ ∥ k t ∥ ∥ v t ∥ = ∥ E ⋆ ∥ F , (40) \lVert E\rVert_{\mathrm{F}} \ge\frac{\lvert r_t\rvert}{\lVert k_t\rVert\lVert v_t\rVert}=\lVert E^\star\rVert_{\mathrm{F}}, \tag*{(40)} ∥ E ∥ F ≥ ∥ k t ∥ ∥ v t ∥ ∣ r t ∣ = ∥ E ⋆ ∥ F , (40) with equality if and only if E = E ⋆ E=E^\star E = E ⋆ .
Proof. Admissibility is a computation: ⟨ v t k t T , E ⋆ ⟩ F = − r t ∥ v t k t T ∥ F 2 / ( ∥ k t ∥ 2 ∥ v t ∥ 2 ) = − r t \langle v_t k_t^{\mathsf{T}},E^\star\rangle_{\mathrm{F}}=-r_t\lVert v_t k_t^{\mathsf{T}}\rVert_{\mathrm{F}}^2/(\lVert k_t\rVert^2\lVert v_t\rVert^2)=-r_t ⟨ v t k t T , E ⋆ ⟩ F = − r t ∥ v t k t T ∥ F 2 / (∥ k t ∥ 2 ∥ v t ∥ 2 ) = − r t , using ∥ v t k t T ∥ F = ∥ k t ∥ ∥ v t ∥ \lVert v_t k_t^{\mathsf{T}}\rVert_{\mathrm{F}}=\lVert k_t\rVert\lVert v_t\rVert ∥ v t k t T ∥ F = ∥ k t ∥ ∥ v t ∥ . For the bound, apply Cauchy–Schwarz to the constraint in (A28):
∣ r t ∣ = ∣ ⟨ v t k t T , E ⟩ F ∣ ≤ ∥ v t k t T ∥ F ∥ E ∥ F = ∥ k t ∥ ∥ v t ∥ ∥ E ∥ F , (41) \lvert r_t\rvert=\left\lvert\left\langle v_t k_t^{\mathsf{T}},E\right\rangle_{\mathrm{F}}\right\rvert\le\left\lVert v_t k_t^{\mathsf{T}}\right\rVert_{\mathrm{F}}\lVert E\rVert_{\mathrm{F}}=\lVert k_t\rVert\lVert v_t\rVert\lVert E\rVert_{\mathrm{F}}, \tag*{(41)} ∣ r t ∣ = ⟨ v t k t T , E ⟩ F ≤ v t k t T F ∥ E ∥ F = ∥ k t ∥ ∥ v t ∥ ∥ E ∥ F , (41)
which is (A30). Cauchy–Schwarz holds with equality exactly when E E E is a scalar multiple of v t k t T v_t k_t^{\mathsf{T}} v t k t T , and the constraint then pins that scalar to − r t / ( ∥ k t ∥ 2 ∥ v t ∥ 2 ) -r_t/(\lVert k_t\rVert^2\lVert v_t\rVert^2) − r t / (∥ k t ∥ 2 ∥ v t ∥ 2 ) , i.e. E = E ⋆ E=E^\star E = E ⋆ .
Equivalently the geometric view: S t + E ⋆ S_t+E^\star S t + E ⋆ is the orthogonal projection of S t S_t S t onto the constraint hyperplane H t = { S : v t T S k t = c t } \mathcal{H}_t=\{S:v_t^{\mathsf{T}}Sk_t=c_t\} H t = { S : v t T S k t = c t } , so the Newton step moves the memory along the shortest path from where it is to where the constraint is satisfied.
Theorem A2 (Minimality in every unitarily invariant norm). Let ∥ ⋅ ∥ \lVert\cdot\rVert ∥ ⋅ ∥ be any unitarily invariant norm on R d × d \mathbb{R}^{d\times d} R d × d . Then
min E ∈ E t ∥ E ∥ = ∥ E ⋆ ∥ , (42) \min_{E\in\mathcal{E}_t}\lVert E\rVert=\lVert E^\star\rVert, \tag*{(42)} E ∈ E t min ∥ E ∥ = ∥ E ⋆ ∥ , (42) i.e. the same E ⋆ E^\star E ⋆ is optimal simultaneously for all of them. In particular, for every Schatten-p p p norm, 1 ≤ p ≤ ∞ 1\le p\le\infty 1 ≤ p ≤ ∞ ,
min E ∈ E t ∥ E ∥ S p = ∥ E ⋆ ∥ S p = ∣ r t ∣ ∥ k t ∥ ∥ v t ∥ , (43) \min_{E\in\mathcal{E}_t}\lVert E\rVert_{S_p}=\lVert E^\star\rVert_{S_p}=\frac{\lvert r_t\rvert}{\lVert k_t\rVert\lVert v_t\rVert}, \tag*{(43)} E ∈ E t min ∥ E ∥ S p = ∥ E ⋆ ∥ S p = ∥ k t ∥ ∥ v t ∥ ∣ r t ∣ , (43) the same number for all p p p . Therefore, the nuclear, Frobenius and spectral norms of the optimal correction coincide. The minimizer is unique for strictly convex norms (Schatten 1 < p < ∞ 1<p<\infty 1 < p < ∞ ). For the spectral and nuclear norms E ⋆ E^\star E ⋆ is a minimizer but need not be the only one.
Proof. Let ∥ ⋅ ∥ ′ \lVert\cdot\rVert' ∥ ⋅ ∥ ′ be the dual norm, so that ∣ ⟨ A , E ⟩ F ∣ ≤ ∥ A ∥ ′ ∥ E ∥ \lvert\langle A,E\rangle_{\mathrm{F}}\rvert\le\lVert A\rVert'\lVert E\rVert ∣⟨ A , E ⟩ F ∣ ≤ ∥ A ∥ ′ ∥ E ∥ for all A , E A,E A , E . Applying this to the constraint gives, for every E ∈ E t E\in\mathcal{E}_t E ∈ E t ,
∥ E ∥ ≥ ∣ r t ∣ ∥ v t k t T ∥ ′ . (44) \lVert E\rVert\ge\frac{\lvert r_t\rvert}{\lVert v_t k_t^{\mathsf{T}}\rVert'}. \tag*{(44)} ∥ E ∥ ≥ ∥ v t k t T ∥ ′ ∣ r t ∣ . (44)
By von Neumann’s theorem a unitarily invariant norm is a symmetric gauge function Φ \Phi Φ of the singular values, and the dual norm is the gauge Φ ′ \Phi' Φ ′ of the singular values. The matrix v t k t T v_t k_t^{\mathsf{T}} v t k t T has the single nonzero singular value ∥ k t ∥ ∥ v t ∥ \lVert k_t\rVert\lVert v_t\rVert ∥ k t ∥ ∥ v t ∥ , so
∥ v t k t T ∥ = ∥ k t ∥ ∥ v t ∥ Φ ( e 1 ) , ∥ v t k t T ∥ ′ = ∥ k t ∥ ∥ v t ∥ Φ ′ ( e 1 ) . (45) \lVert v_t k_t^{\mathsf{T}}\rVert=\lVert k_t\rVert\lVert v_t\rVert\Phi(e_1),\qquad\lVert v_t k_t^{\mathsf{T}}\rVert'=\lVert k_t\rVert\lVert v_t\rVert\Phi'(e_1). \tag*{(45)} ∥ v t k t T ∥ = ∥ k t ∥ ∥ v t ∥ Φ ( e 1 ) , ∥ v t k t T ∥ ′ = ∥ k t ∥ ∥ v t ∥ Φ ′ ( e 1 ) . (45)
Now Φ ( e 1 ) Φ ′ ( e 1 ) = 1 \Phi(e_1)\Phi'(e_1)=1 Φ ( e 1 ) Φ ′ ( e 1 ) = 1 : indeed Φ ′ ( e 1 ) = sup { x 1 : Φ ( x ) ≤ 1 } \Phi'(e_1)=\sup\{x_1:\Phi(x)\le1\} Φ ′ ( e 1 ) = sup { x 1 : Φ ( x ) ≤ 1 } , and since a symmetric gauge is nondecreasing in each ∣ x i ∣ \lvert x_i\rvert ∣ x i ∣ , replacing x x x by ( x 1 , 0 , … , 0 ) (x_1,0,\ldots,0) ( x 1 , 0 , … , 0 ) does not increase Φ \Phi Φ , so the supremum is attained on the first axis and equals 1 / Φ ( e 1 ) 1/\Phi(e_1) 1/Φ ( e 1 ) . Since E ⋆ E^\star E ⋆ is rank one with singular value ∣ r t ∣ / ( ∥ k t ∥ ∥ v t ∥ ) \lvert r_t\rvert/(\lVert k_t\rVert\lVert v_t\rVert) ∣ r t ∣ / (∥ k t ∥ ∥ v t ∥) ,
∥ E ⋆ ∥ = ∣ r t ∣ ∥ k t ∥ ∥ v t ∥ Φ ( e 1 ) = ∣ r t ∣ ∥ v t k t T ∥ ′ . (46) \lVert E^\star\rVert=\frac{\lvert r_t\rvert}{\lVert k_t\rVert\lVert v_t\rVert}\Phi(e_1)=\frac{\lvert r_t\rvert}{\lVert v_t k_t^{\mathsf{T}}\rVert'}. \tag*{(46)} ∥ E ⋆ ∥ = ∥ k t ∥ ∥ v t ∥ ∣ r t ∣ Φ ( e 1 ) = ∥ v t k t T ∥ ′ ∣ r t ∣ . (46)
which is exactly the lower bound (A34): E ⋆ E^\star E ⋆ attains it. The Schatten case follows from Φ p ( e 1 ) = 1 \Phi_p(e_1)=1 Φ p ( e 1 ) = 1 for every p p p . Uniqueness for a strictly convex norm holds because E t \mathcal{E}_t E t is convex and a strictly convex function has at most one minimizer on a convex set; for the spectral norm uniqueness fails already at d = 2 d=2 d = 2 with k = v = e 1 k=v=e_1 k = v = e 1 and r t = − 1 r_t=-1 r t = − 1 , where E = diag ( 1 , δ ) E=\operatorname{diag}(1,\delta) E = diag ( 1 , δ ) is admissible and has spectral norm 1 1 1 for every ∣ δ ∣ ≤ 1 |\delta|\leq1 ∣ δ ∣ ≤ 1 .
Minimality of support: the correction is surgical
Norm-optimality says the correction is small. The next theorem has a stronger statement even: it is the only correction that stays inside the addressed directions at all. No optimization is involved.
Theorem A3 (Unique correction with addressed support). E ⋆ E^\star E ⋆ is the unique E ∈ E t E\in\mathcal{E}_t E ∈ E t whose column space is contained in span { v t } \operatorname{span}\{v_t\} span { v t } and whose row space is contained in span { k t } \operatorname{span}\{k_t\} span { k t } .
Proof. A matrix has column space in span { v t } \operatorname{span}\{v_t\} span { v t } and row space in span { k t } \operatorname{span}\{k_t\} span { k t } if and only if it is of the form E = γ v t k t ⊤ E=\gamma v_t k_t^\top E = γ v t k t ⊤ for some γ ∈ R \gamma\in\mathbb{R} γ ∈ R . Admissibility forces γ ∥ k t ∥ 2 ∥ v t ∥ 2 = − r t \gamma\lVert k_t\rVert^2\lVert v_t\rVert^2=-r_t γ ∥ k t ∥ 2 ∥ v t ∥ 2 = − r t , hence γ = − r t / ( ∥ k t ∥ 2 ∥ v t ∥ 2 ) \gamma=-r_t/(\lVert k_t\rVert^2\lVert v_t\rVert^2) γ = − r t / (∥ k t ∥ 2 ∥ v t ∥ 2 ) and E = E ⋆ E=E^\star E = E ⋆ .
Every other admissible correction like every DeltaNet-style correction necessarily writes into at least one key direction other than k t k_t k t or one value direction other than v t v_t v t , i.e. into directions the task never asked about.
Minimality of interference
Smallness in norm translates into smallness of the damage done to other associations.
Theorem A4 (Interference caused by the correction). Let ( k ′ , v ′ ) (k',v') ( k ′ , v ′ ) be any probe pair and let Δ ( k ′ , v ′ ) : = v ′ ⊤ E k ′ \Delta(k',v'):=v'^\top Ek' Δ ( k ′ , v ′ ) := v ′⊤ E k ′ be the change its read-out suffers.
(i) Exact form and protection. For E = E ⋆ E=E^\star E = E ⋆ ,
Δ ( k ′ , v ′ ) = − r t ∥ k t ∥ 2 ∥ v t ∥ 2 ( v ′ ⊤ v t ) ( k t ⊤ k ′ ) (47) \Delta(k',v')=-\frac{r_t}{\lVert k_t\rVert^2\lVert v_t\rVert^2}(v'^\top v_t)(k_t^\top k') \tag*{(47)} Δ ( k ′ , v ′ ) = − ∥ k t ∥ 2 ∥ v t ∥ 2 r t ( v ′⊤ v t ) ( k t ⊤ k ′ ) (47) which vanishes whenever v ′ ⊥ v t v'\perp v_t v ′ ⊥ v t or k ′ ⊥ k t k'\perp k_t k ′ ⊥ k t , and obeys ∣ Δ ( k ′ , v ′ ) ∣ ≤ ∣ r t ∣ ∥ k ′ ∥ ∥ v ′ ∥ / ( ∥ k t ∥ ∥ v t ∥ ) |\Delta(k',v')|\leq|r_t|\lVert k'\rVert\lVert v'\rVert/(\lVert k_t\rVert\lVert v_t\rVert) ∣Δ ( k ′ , v ′ ) ∣ ≤ ∣ r t ∣ ∥ k ′ ∥ ∥ v ′ ∥ / (∥ k t ∥ ∥ v t ∥) . The condition is a property of the inputs alone: it can be checked in advance and does not depend on what S t S_t S t currently holds.
(ii) Minimum expected interference. Suppose probe pairs are drawn with k ′ k' k ′ and v ′ v' v ′ independent and isotropic, E [ k ′ k ′ ⊤ ] = σ k 2 I \mathbb{E}[k'k'^\top]=\sigma_k^2I E [ k ′ k ′⊤ ] = σ k 2 I and E [ v ′ v ′ ⊤ ] = σ v 2 I \mathbb{E}[v'v'^\top]=\sigma_v^2I E [ v ′ v ′⊤ ] = σ v 2 I . Then for every E ∈ R d × d E\in\mathbb{R}^{d\times d} E ∈ R d × d
E [ ( v ′ ⊤ E k ′ ) 2 ] = σ k 2 σ v 2 ∥ E ∥ F 2 . (48) \mathbb{E}\left[(v'^\top Ek')^2\right]=\sigma_k^2\sigma_v^2\lVert E\rVert_F^2. \tag*{(48)} E [ ( v ′⊤ E k ′ ) 2 ] = σ k 2 σ v 2 ∥ E ∥ F 2 . (48) Consequently
E ⋆ = argmin E ∈ E t E [ ( v ′ ⊤ E k ′ ) 2 ] , min E ∈ E t E [ Δ 2 ] = σ k 2 σ v 2 r t 2 ∥ k t ∥ 2 ∥ v t ∥ 2 , (49) E^\star=\underset{E\in\mathcal{E}_t}{\operatorname{argmin}}\ \mathbb{E}\left[(v'^\top Ek')^2\right],\qquad
\underset{E\in\mathcal{E}_t}{\min}\ \mathbb{E}[\Delta^2]=\frac{\sigma_k^2\sigma_v^2r_t^2}{\lVert k_t\rVert^2\lVert v_t\rVert^2}, \tag*{(49)} E ⋆ = E ∈ E t argmin E [ ( v ′⊤ E k ′ ) 2 ] , E ∈ E t min E [ Δ 2 ] = ∥ k t ∥ 2 ∥ v t ∥ 2 σ k 2 σ v 2 r t 2 , (49) and the minimizer is unique.
Proof. (i) Substitute (A29) and use v ′ ⊤ ( v t k t ⊤ ) k ′ = ( v ′ ⊤ v t ) ( k t ⊤ k ′ ) v'^\top(v_tk_t^\top)k'=(v'^\top v_t)(k_t^\top k') v ′⊤ ( v t k t ⊤ ) k ′ = ( v ′⊤ v t ) ( k t ⊤ k ′ ) ; the inequality is Cauchy–Schwarz on each factor. (ii) Vectorize: with e = vec ( E ) e=\operatorname{vec}(E) e = vec ( E ) and a ′ = k ′ ⊗ v ′ a'=k'\otimes v' a ′ = k ′ ⊗ v ′ we have v ′ ⊤ E k ′ = a ′ ⊤ e v'^\top Ek'=a'^\top e v ′⊤ E k ′ = a ′⊤ e , hence
E [ ( a ′ ⊤ e ) 2 ] = e ⊤ E [ a ′ a ′ ⊤ ] e = e ⊤ ( E [ k ′ k ′ ⊤ ] ⊗ E [ v ′ v ′ ⊤ ] ) e = σ k 2 σ v 2 ∥ e ∥ 2 , (50) \mathbb{E}[(a'^\top e)^2]=e^\top\mathbb{E}[a'a'^\top]e=e^\top\left(\mathbb{E}[k'k'^\top]\otimes\mathbb{E}[v'v'^\top]\right)e=\sigma_k^2\sigma_v^2\lVert e\rVert^2, \tag*{(50)} E [( a ′⊤ e ) 2 ] = e ⊤ E [ a ′ a ′⊤ ] e = e ⊤ ( E [ k ′ k ′⊤ ] ⊗ E [ v ′ v ′⊤ ] ) e = σ k 2 σ v 2 ∥ e ∥ 2 , (50)
using independence and ∥ e ∥ = ∥ E ∥ F \lVert e\rVert=\lVert E\rVert_F ∥ e ∥ = ∥ E ∥ F . The objective is therefore a positive multiple of ∥ E ∥ F 2 \lVert E\rVert_F^2 ∥ E ∥ F 2 , so minimizing it over E t \mathcal{E}_t E t is the problem already solved by Theorem A1, with the same unique solution.
Part (ii) is what justifies the name: for isotropic probe statistics, minimum norm and minimum expected interference are literally the same objective, so the Newton step is minimally invasive in the operational sense and not merely in the geometric one. It causes the least expected damage to the rest of the memory.
Relaxed steps: the correction never moves away from a correct memory
In practice, β \beta β is a learned gate rather than β ⋆ \beta^\star β ⋆ . Write β = λ β ⋆ \beta= \lambda\beta^\star β = λ β ⋆ , so that the actual update is S t + 1 = S t + λ E ⋆ S_{t+1} = S_t + \lambda E^\star S t + 1 = S t + λ E ⋆ with relaxation ω = β ∥ k t ∥ 2 ∥ v t ∥ 2 \omega= \beta\lVert k_t\rVert^2 \lVert v_t\rVert^2 ω = β ∥ k t ∥ 2 ∥ v t ∥ 2 .
Theorem A5 (Exact Fejér identity). Let S ∘ S^\circ S ∘ be any memory that already satisfies the addressed association, v t T S ∘ k t = c t v_t^\mathsf{T}S^\circ k_t = c_t v t T S ∘ k t = c t . Then
∥ S t + 1 − S ∘ ∥ F 2 = ∥ S t − S ∘ ∥ F 2 − λ ( 2 − λ ) r t 2 ∥ k t ∥ 2 ∥ v t ∥ 2 . (51) \lVert S_{t+1} - S^\circ\rVert_{\mathrm{F}}^2 = \lVert S_t - S^\circ\rVert_{\mathrm{F}}^2 - \lambda(2-\lambda)\frac{r_t^2}{\lVert k_t\rVert^2\lVert v_t\rVert^2}. \tag*{(51)} ∥ S t + 1 − S ∘ ∥ F 2 = ∥ S t − S ∘ ∥ F 2 − λ ( 2 − λ ) ∥ k t ∥ 2 ∥ v t ∥ 2 r t 2 . (51) Hence for every λ ∈ ( 0 , 2 ) \lambda\in(0,2) λ ∈ ( 0 , 2 ) the update strictly decreases the distance to every correct memory, the decrease is largest at the Newton step λ = 1 \lambda= 1 λ = 1 , and at λ ∈ { 0 , 2 } \lambda\in\{0,2\} λ ∈ { 0 , 2 } the distance is unchanged.
Proof. Expand, using ⟨ E ⋆ , S t − S ∘ ⟩ F = − r t ∥ k t ∥ 2 ∥ v t ∥ 2 ( v t T S t k t − v t T S ∘ k t ) = − r t 2 ∥ k t ∥ 2 ∥ v t ∥ 2 = − ∥ E ⋆ ∥ F 2 \langle E^\star,S_t-S^\circ\rangle_{\mathrm{F}}=-\frac{r_t}{\lVert k_t\rVert^2\lVert v_t\rVert^2}(v_t^\mathsf{T}S_tk_t-v_t^\mathsf{T}S^\circ k_t)=-\frac{r_t^2}{\lVert k_t\rVert^2\lVert v_t\rVert^2}=-\lVert E^\star\rVert_{\mathrm{F}}^2 ⟨ E ⋆ , S t − S ∘ ⟩ F = − ∥ k t ∥ 2 ∥ v t ∥ 2 r t ( v t T S t k t − v t T S ∘ k t ) = − ∥ k t ∥ 2 ∥ v t ∥ 2 r t 2 = − ∥ E ⋆ ∥ F 2 :
∥ S t − S ∘ + λ E ⋆ ∥ F 2 = ∥ S t − S ∘ ∥ F 2 + 2 λ ⟨ E ⋆ , S t − S ∘ ⟩ F + λ 2 ∥ E ⋆ ∥ F 2 , (52) \lVert S_t-S^\circ+\lambda E^\star\rVert_{\mathrm{F}}^2=\lVert S_t-S^\circ\rVert_{\mathrm{F}}^2+2\lambda\langle E^\star,S_t-S^\circ\rangle_{\mathrm{F}}+\lambda^2\lVert E^\star\rVert_{\mathrm{F}}^2, \tag*{(52)} ∥ S t − S ∘ + λ E ⋆ ∥ F 2 = ∥ S t − S ∘ ∥ F 2 + 2 λ ⟨ E ⋆ , S t − S ∘ ⟩ F + λ 2 ∥ E ⋆ ∥ F 2 , (52)
which is (A41) after inserting ∥ E ⋆ ∥ F 2 = r t 2 / ( ∥ k t ∥ 2 ∥ v t ∥ 2 ) \lVert E^\star\rVert_{\mathrm{F}}^2=r_t^2/(\lVert k_t\rVert^2\lVert v_t\rVert^2) ∥ E ⋆ ∥ F 2 = r t 2 / (∥ k t ∥ 2 ∥ v t ∥ 2 ) .
This is the strongest sense in which a single write is harmless: it does not merely perturb other associations little, it moves the memory closer to every memory that is consistent with the association being written and that simultaneously, for all of them.
Greedy minimality is global minimality
Per-step optimality does not by itself imply that a sequence of such steps is optimal. We show that our does.
Proposition 1 (The limit is the minimum-norm memory). Let { ( k i , v i , c i ) } i = 1 n \{(k_i,v_i,c_i)\}_{i=1}^n {( k i , v i , c i ) } i = 1 n be a consistent system of associations. Apply the rule from S 0 = 0 S_0=0 S 0 = 0 with relaxation ω ∈ ( 0 , 2 ) \omega\in(0,2) ω ∈ ( 0 , 2 ) , visiting each pair infinitely often either cyclically or by random sampling. Then S t S_t S t converges to
S ♯ = argmin S { ∥ S ∥ F : v i T S k i = c i for all i } , (53) S^\sharp=\underset{S}{\operatorname{argmin}}\left\{\lVert S\rVert_{\mathrm{F}}:v_i^\mathsf{T}Sk_i=c_i\ \text{for all }i\right\}, \tag*{(53)} S ♯ = S argmin { ∥ S ∥ F : v i T S k i = c i for all i } , (53) the minimum Frobenius-norm memory satisfying all the associations.
Proof. This is the classical convergence theorem for relaxed Kaczmarz iterations [15 , 30 ] , applied to the vectorized system a i T s = c i a_i^\mathsf{T}s=c_i a i T s = c i with a i = k i ⊗ v i a_i=k_i\otimes v_i a i = k i ⊗ v i . Each iterate stays in s 0 + span { a 1 , … , a n } s_0+\operatorname{span}\{a_1,\ldots,a_n\} s 0 + span { a 1 , … , a n } , which for s 0 = 0 s_0=0 s 0 = 0 is the row space of the system; the iterates converge to a solution, and the unique solution in the row space is the minimum-norm one.
So the memory built by a stream of minimally invasive revisions is itself the least-norm memory compatible with everything it was asked to store: locally and globally, the rule writes as little as the task permits.
Less invasive than DeltaNet
Theorems A1–A4 hold given the requirement encoded in E t \mathcal{E}_t E t : make the scalar read-out of the addressed pair equal to c t c_t c t , and do nothing else. DeltaNet’s step has to solve the d d d constraints S k t = v t Sk_t=v_t S k t = v t , therefore its correction is necessarily larger. The claim here is therefore not that DeltaNet is suboptimal for its own task, but that the two tasks differ: a recognition memory needs one scalar corrected, and the Kronecker Delta Rule corrects it with provably minimal disturbance, whereas a retrieval memory must rewrite an entire value vector and pays for it in footprint.
Convergence of the KronLSTM memory
We analyze the expected solution and steady state of the randomized Kronecker Delta Rule with more than d 2 d^2 d 2 associations. Running memories i.e. online updates are covered via the steady state considerations.
Overview
We analyze the Kronecker Delta Rule S t + 1 = S t − β ( v i t T S t k i t − c i t ) v i t k i t T S_{t+1}=S_t-\beta\left(v_{i_t}^{\mathsf{T}}S_t k_{i_t}-c_{i_t}\right)v_{i_t}k_{i_t}^{\mathsf{T}} S t + 1 = S t − β ( v i t T S t k i t − c i t ) v i t k i t T with randomly drawn indices when the number of associations exceeds the capacity: n > d 2 n>d^2 n > d 2 . The system is then generically inconsistent, and the iterates no longer converge to a point. Instead we have: (i) The expected iterate converges linearly to the sampling-weighted least-squares solution S L S = argmin S ∑ i p i ( v i T S k i − c i ) 2 S^{\mathrm{LS}}=\underset{S}{\operatorname{argmin}}\sum_i p_i\left(v_i^{\mathsf{T}}S k_i-c_i\right)^2 S LS = S argmin ∑ i p i ( v i T S k i − c i ) 2 . This is for uniform sampling exactly the minimizer of the loss L \mathrm{L} L . (ii) The iterates converge in distribution to a unique stationary distribution whose mean is exactly S L S S^{\mathrm{LS}} S LS (no bias at any order in β \beta β ) and whose covariance is O ( β ) O(\beta) O ( β ) , given by a Lyapunov-type equation driven by the residuals at S L S S^{\mathrm{LS}} S LS . (iii) The steady-state loss exceeds the minimum by the classical LMS misadjustment, E [ L ] − L min = β 2 E [ ρ i 2 ∥ k i ∥ 2 ∥ v i ∥ 2 ] + O ( β 2 ) \mathbb{E}[\mathrm{L}]-\mathrm{L}_{\min}=\frac{\beta}{2}\mathbb{E}[\rho_i^2\lVert k_i\rVert^2\lVert v_i\rVert^2]+O(\beta^2) E [ L ] − L m i n = 2 β E [ ρ i 2 ∥ k i ∥ 2 ∥ v i ∥ 2 ] + O ( β 2 ) , where ρ i \rho_i ρ i are the optimal residuals.
This appendix contains complete proofs of both theorems: the mean dynamics are solved exactly; existence, uniqueness, and geometric convergence of the stationary distribution are established by a synchronous-coupling contraction in the Wasserstein-2 metric; the stationary mean and the exact covariance equation are derived with all cross terms accounted for; the misadjustment follows from an exact trace identity with two-sided bounds; and the transient is controlled by an explicit geometric-plus-horizon estimate. Exact convergence can be recovered by diminishing step sizes or by iterate averaging, and it survives n > d 2 n>d^2 n > d 2 unharmed whenever the targets remain consistent. Notably, exact convergence is always obtain for the correlation target c i = v i T k i c_i=v_i^{\mathsf{T}}k_i c i = v i T k i and the norm target c i = ∥ v i ∥ 2 c_i=\lVert v_i\rVert^2 c i = ∥ v i ∥ 2 whenever an exact retrieval memory exists.
Setting and notation
Let n > d 2 n>d^2 n > d 2 associations ( k i , v i , c i ) (k_i,v_i,c_i) ( k i , v i , c i ) , 1 ≤ i ≤ n 1\leq i\leq n 1 ≤ i ≤ n , be given. Vectorized, with s = vec ( S ) ∈ R d 2 s=\operatorname{vec}(S)\in\mathbb{R}^{d^2} s = vec ( S ) ∈ R d 2 and a i = k i ⊗ v i a_i=k_i\otimes v_i a i = k i ⊗ v i , therefore ∥ a i ∥ = ∥ k i ∥ ∥ v i ∥ \lVert a_i\rVert=\lVert k_i\rVert\lVert v_i\rVert ∥ a i ∥ = ∥ k i ∥ ∥ v i ∥ . The associations form the linear system A s = c As=c A s = c with A ∈ R n × d 2 A\in\mathbb{R}^{n\times d^2} A ∈ R n × d 2 stacking the rows a i T a_i^{\mathsf{T}} a i T . We assume A A A has full column rank d 2 d^2 d 2 i.e. the features span R d 2 \mathbb{R}^{d^2} R d 2 . Since n > d 2 n>d^2 n > d 2 , the system is generically inconsistent: no S S S satisfies all associations, and
L ( S ) = 1 n ∑ i = 1 n ( v i T S k i − c i ) 2 (54) \mathrm{L}(S)=\frac{1}{n}\sum_{i=1}^{n}\left(v_i^{\mathsf{T}}S k_i-c_i\right)^2 \tag*{(54)} L ( S ) = n 1 i = 1 ∑ n ( v i T S k i − c i ) 2 (54)
has a positive minimum L min > 0 \mathrm{L}_{\min}>0 L m i n > 0 . At each step an index i t i_t i t is drawn with probabilities p i > 0 p_i>0 p i > 0 , independently of the past, and the rule is applied with fixed step size β \beta β satisfying
0 < β ∥ a i ∥ 2 < 2 for all i . (55) 0<\beta\lVert a_i\rVert^2<2\qquad\text{for all }i. \tag*{(55)} 0 < β ∥ a i ∥ 2 < 2 for all i . (55)
We define
A ˉ : = E [ a a T ] = ∑ i = 1 n p i a i a i T ≻ 0 , b ˉ : = E [ c a ] = ∑ i = 1 n p i c i a i , (56) \bar{A}:=\mathbb{E}[aa^{\mathsf{T}}]=\sum_{i=1}^{n}p_i a_i a_i^{\mathsf{T}}\succ0,\qquad
\bar{b}:=\mathbb{E}[ca]=\sum_{i=1}^{n}p_i c_i a_i, \tag*{(56)} A ˉ := E [ a a T ] = i = 1 ∑ n p i a i a i T ≻ 0 , b ˉ := E [ c a ] = i = 1 ∑ n p i c i a i , (56)
s L S : = A ˉ − 1 b ˉ = argmin s ∑ i = 1 n p i ( a i T s − c i ) 2 , ρ i : = c i − a i T s L S (optimal residuals) , (57) s^{\mathrm{LS}}:=\bar{A}^{-1}\bar{b}
=\underset{s}{\operatorname{argmin}}\sum_{i=1}^{n}p_i\left(a_i^{\mathsf{T}}s-c_i\right)^2,\qquad
\rho_i:=c_i-a_i^{\mathsf{T}}s^{\mathrm{LS}}\quad\text{(optimal residuals)}, \tag*{(57)} s LS := A ˉ − 1 b ˉ = s argmin i = 1 ∑ n p i ( a i T s − c i ) 2 , ρ i := c i − a i T s LS (optimal residuals) , (57)
so that the normal equations read
E [ ρ a ] = ∑ i = 1 n p i ρ i a i = 0 ⟺ b ˉ = A ˉ s L S . (58) \mathbb{E}[\rho a]=\sum_{i=1}^{n}p_i\rho_i a_i=0
\qquad\Longleftrightarrow\qquad
\bar{b}=\bar{A}s^{\mathrm{LS}}. \tag*{(58)} E [ ρ a ] = i = 1 ∑ n p i ρ i a i = 0 ⟺ b ˉ = A ˉ s LS . (58)
For uniform sampling, A ˉ = 1 n A T A \bar{A}=\frac{1}{n}A^{\mathsf{T}}A A ˉ = n 1 A T A and s L S = ( A T A ) − 1 A T c s^{\mathrm{LS}}=(A^{\mathsf{T}}A)^{-1}A^{\mathsf{T}}c s LS = ( A T A ) − 1 A T c is the ordinary least-squares solution, i.e. the unique minimizer of L \mathrm{L} L . Throughout, expectations without subscript are over i ∼ p i\sim p i ∼ p , and E π \mathbb{E}_{\pi} E π denotes expectation of a random state distributed according to a law π \pi π . Vectorized, the rule and its error form are
s t + 1 = M i t s t + β c i t a i t , e t + 1 = M i t e t + β ρ i t a i t , M i : = I − β a i a i T , (59) s_{t+1}=M_{i_t}s_t+\beta c_{i_t}a_{i_t},\qquad
e_{t+1}=M_{i_t}e_t+\beta\rho_{i_t}a_{i_t},\qquad
M_i:=I-\beta a_i a_i^{\mathsf{T}}, \tag*{(59)} s t + 1 = M i t s t + β c i t a i t , e t + 1 = M i t e t + β ρ i t a i t , M i := I − β a i a i T , (59)
with e t : = s t − s L S e_t:=s_t-s^{\mathrm{LS}} e t := s t − s LS ; the error form follows by inserting c i = a i T s L S + ρ i c_i=a_i^{\mathsf{T}}s^{\mathrm{LS}}+\rho_i c i = a i T s LS + ρ i . The constants
γ β : = min i β ( 2 − β ∥ a i ∥ 2 ) > 0 , ρ ˉ : = γ β λ min ( A ˉ ) ∈ ( 0 , 1 ] , σ 2 : = E [ ρ 2 ∥ a ∥ 2 ] (60) \gamma_\beta:=\min_i\beta\left(2-\beta\lVert a_i\rVert^2\right)>0,\qquad
\bar{\rho}:=\gamma_\beta\lambda_{\min}(\bar{A})\in(0,1],\qquad
\sigma^2:=\mathbb{E}\left[\rho^2\lVert a\rVert^2\right] \tag*{(60)} γ β := i min β ( 2 − β ∥ a i ∥ 2 ) > 0 , ρ ˉ := γ β λ m i n ( A ˉ ) ∈ ( 0 , 1 ] , σ 2 := E [ ρ 2 ∥ a ∥ 2 ] (60)
appear throughout (0 < ρ ˉ ≤ 1 0<\bar{\rho}\leq1 0 < ρ ˉ ≤ 1 is shown in Lemma A1).
Theorems on expected and steady state solution
Theorem A6 (Expected solution: exact, unbiased least squares). Under (A45), the expected iterate obeys the exact deterministic recursion
E [ s t + 1 ] = ( I − β A ˉ ) E [ s t ] + β b ˉ ⟺ E [ e t + 1 ] = ( I − β A ˉ ) E [ e t ] , (61) \mathbb{E}[s_{t+1}] = (I-\beta\bar{A})\mathbb{E}[s_t] + \beta\bar{b}
\quad\Longleftrightarrow\quad
\mathbb{E}[e_{t+1}] = (I-\beta\bar{A})\mathbb{E}[e_t], \tag*{(61)} E [ s t + 1 ] = ( I − β A ˉ ) E [ s t ] + β b ˉ ⟺ E [ e t + 1 ] = ( I − β A ˉ ) E [ e t ] , (61) whose unique fixed point is s L S s^{\mathrm{LS}} s LS , and
∥ E [ s t ] − s L S ∥ F ≤ r t ∥ s 0 − s L S ∥ F , r : = max k ∣ 1 − β λ k ( A ˉ ) ∣ < 1 , (62) \left\|\mathbb{E}[s_t]-s^{\mathrm{LS}}\right\|_{\mathrm{F}}
\le r^t\left\|s_0-s^{\mathrm{LS}}\right\|_{\mathrm{F}},
\qquad
r:=\max_k\left|1-\beta\lambda_k(\bar{A})\right|<1, \tag*{(62)} E [ s t ] − s LS F ≤ r t s 0 − s LS F , r := k max 1 − β λ k ( A ˉ ) < 1 , (62) so E [ s t ] → s L S \mathbb{E}[s_t]\to s^{\mathrm{LS}} E [ s t ] → s LS linearly. The convergence of the mean is exactly unbiased: the fixed point of (A51) is s L S s^{\mathrm{LS}} s LS itself, with no O ( β ) O(\beta) O ( β ) correction.
Theorem A7 (Steady state: stationary distribution around s L S s^{\mathrm{LS}} s LS ). Under (A45), the iterates form a Markov chain on R d 2 \mathbb{R}^{d^2} R d 2 that has a unique stationary distribution π β \pi_\beta π β with finite second moment , and the law of s t s_t s t converges to π β \pi_\beta π β geometrically in the Wasserstein-2 metric:
W 2 ( law ( s t ) , π β ) ≤ ( 1 − ρ ˉ ) t / 2 W 2 ( law ( s 0 ) , π β ) . (63) W_2\left(\operatorname{law}(s_t),\pi_\beta\right)
\le
\left(1-\bar{\rho}\right)^{t/2}
W_2\left(\operatorname{law}(s_0),\pi_\beta\right). \tag*{(63)} W 2 ( law ( s t ) , π β ) ≤ ( 1 − ρ ˉ ) t /2 W 2 ( law ( s 0 ) , π β ) . (63) Under π β \pi_\beta π β , with e : = s − s L S e:=s-s^{\mathrm{LS}} e := s − s LS :
(i) Mean: E π β [ S ] = S L S \mathbb{E}_{\pi_\beta}[S]=S^{\mathrm{LS}} E π β [ S ] = S LS exactly.
(ii) Covariance: V β : = E π β [ e e ⊤ ] V_\beta:=\mathbb{E}_{\pi_\beta}[ee^\top] V β := E π β [ e e ⊤ ] satisfies the exact stationarity equation
A ˉ V β + V β A ˉ − β E [ ( a ⊤ V β a ) a a ⊤ ] = β E [ ρ 2 a a ⊤ ] , (64) \bar{A}V_\beta+V_\beta\bar{A}-\beta\mathbb{E}\left[(a^\top V_\beta a)aa^\top\right]
=\beta\mathbb{E}\left[\rho^2aa^\top\right], \tag*{(64)} A ˉ V β + V β A ˉ − β E [ ( a ⊤ V β a ) a a ⊤ ] = β E [ ρ 2 a a ⊤ ] , (64) and obeys the two-sided trace bounds
β σ 2 2 ≤ tr ( A ˉ V β ) ≤ β σ 2 2 − β max i ∥ a i ∥ 2 , (65) \frac{\beta\sigma^2}{2}
\le\operatorname{tr}(\bar{A}V_\beta)
\le
\frac{\beta\sigma^2}{2-\beta\max_i\|a_i\|^2}, \tag*{(65)} 2 β σ 2 ≤ tr ( A ˉ V β ) ≤ 2 − β max i ∥ a i ∥ 2 β σ 2 , (65) E π β [ ∥ S − S L S ∥ F 2 ] ≤ β σ 2 ( 2 − β max i ∥ a i ∥ 2 ) λ min ( A ˉ ) ; \mathbb{E}_{\pi_\beta}\left[\|S-S^{\mathrm{LS}}\|_{\mathrm{F}}^2\right]
\le
\frac{\beta\sigma^2}
{\left(2-\beta\max_i\|a_i\|^2\right)\lambda_{\min}(\bar{A})}; E π β [ ∥ S − S LS ∥ F 2 ] ≤ ( 2 − β max i ∥ a i ∥ 2 ) λ m i n ( A ˉ ) β σ 2 ; in particular V β = O ( β ) V_\beta=O(\beta) V β = O ( β ) , and to leading order (A54) is the Lyapunov equation A ˉ V + V A ˉ = β E [ ρ 2 a a ⊤ ] \bar{A}V+V\bar{A}=\beta\mathbb{E}[\rho^2aa^\top] A ˉ V + V A ˉ = β E [ ρ 2 a a ⊤ ] .
(iii) Steady-state loss (misadjustment): for uniform sampling,
E π β [ L ( S ) ] − L min = tr ( A ˉ V β ) = β 2 ( σ 2 + E [ ( a ⊤ V β a ) ∥ a ∥ 2 ] ) , (66) \mathbb{E}_{\pi_\beta}[L(S)]-L_{\min}
=\operatorname{tr}(\bar{A}V_\beta)
=\frac{\beta}{2}\left(\sigma^2+\mathbb{E}\left[(a^\top V_\beta a)\|a\|^2\right]\right), \tag*{(66)} E π β [ L ( S )] − L m i n = tr ( A ˉ V β ) = 2 β ( σ 2 + E [ ( a ⊤ V β a ) ∥ a ∥ 2 ] ) , (66) an exact identity, whence
β 2 σ 2 ≤ E π β [ L ( S ) ] − L min ≤ β 2 σ 2 ⋅ 2 2 − β max i ∥ a i ∥ 2 , σ 2 = E [ ρ i 2 ∥ k i ∥ 2 ∥ v i ∥ 2 ] , (67) \frac{\beta}{2}\sigma^2
\le\mathbb{E}_{\pi_\beta}[L(S)]-L_{\min}
\le
\frac{\beta}{2}\sigma^2\cdot
\frac{2}{2-\beta\max_i\|a_i\|^2},
\qquad
\sigma^2=\mathbb{E}\left[\rho_i^2\|k_i\|^2\|v_i\|^2\right], \tag*{(67)} 2 β σ 2 ≤ E π β [ L ( S )] − L m i n ≤ 2 β σ 2 ⋅ 2 − β max i ∥ a i ∥ 2 2 , σ 2 = E [ ρ i 2 ∥ k i ∥ 2 ∥ v i ∥ 2 ] , (67) i.e. E π β [ L ] = L min + β 2 σ 2 + O ( β 2 ) \mathbb{E}_{\pi_\beta}[L]=L_{\min}+\frac{\beta}{2}\sigma^2+O(\beta^2) E π β [ L ] = L m i n + 2 β σ 2 + O ( β 2 ) . This is the classical Widrow–Hoff LMS misadjustment for the bilinear feature a = k ⊗ v a=k\otimes v a = k ⊗ v .
(iv) Transient: with w : = E [ ρ ∥ a ∥ 2 a ] w:=\mathbb{E}[\rho\|a\|^2a] w := E [ ρ ∥ a ∥ 2 a ] ,
E ∥ e t ∥ 2 ≤ ( 1 − ρ ˉ 2 ) t ∥ e 0 ∥ 2 + 2 β 2 σ 2 ρ ˉ + 4 β 4 ∥ w ∥ 2 ρ ˉ 2 , (68) \mathbb{E}\|e_t\|^2
\le
\left(1-\frac{\bar{\rho}}{2}\right)^t\|e_0\|^2
+\frac{2\beta^2\sigma^2}{\bar{\rho}}
+\frac{4\beta^4\|w\|^2}{\bar{\rho}^2}, \tag*{(68)} E ∥ e t ∥ 2 ≤ ( 1 − 2 ρ ˉ ) t ∥ e 0 ∥ 2 + ρ ˉ 2 β 2 σ 2 + ρ ˉ 2 4 β 4 ∥ w ∥ 2 , (68) a geometric approach to an O ( β ) O(\beta) O ( β ) horizon (note ρ ˉ = Θ ( β ) \bar{\rho}=\Theta(\beta) ρ ˉ = Θ ( β ) ): overload does not slow the transient, it only replaces the limit point by a limit distribution.
Proofs of the theorems
Throughout, F t : = σ ( s 0 , i 0 , … , i t − 1 ) \mathcal{F}_t:=\sigma(s_0,i_0,\ldots,i_{t-1}) F t := σ ( s 0 , i 0 , … , i t − 1 ) denotes the filtration of the past; i t i_t i t is independent of F t \mathcal{F}_t F t by assumption. We repeatedly use that (A49) is linear in the state with coefficients independent of the state.
Two lemmas crucial for the proofs of the theorems.
Lemma A1 (One-step mean-square contraction of the homogeneous part). For every fixed x ∈ R d 2 x \in\mathbb{R}^{d^2} x ∈ R d 2 ,
E [ ∥ M i x ∥ 2 ] = ∥ x ∥ 2 − ∑ i = 1 n p i β ( 2 − β ∥ a i ∥ 2 ) ( a i T x ) 2 ≤ ( 1 − ρ ˉ ) ∥ x ∥ 2 , (69) \mathbb{E}\left[\lVert M_i x\rVert^2\right] = \lVert x\rVert^2 - \sum_{i=1}^{n} p_i \beta(2-\beta\lVert a_i\rVert^2)(a_i^{\mathsf T}x)^2 \le(1-\bar{\rho})\lVert x\rVert^2, \tag*{(69)} E [ ∥ M i x ∥ 2 ] = ∥ x ∥ 2 − i = 1 ∑ n p i β ( 2 − β ∥ a i ∥ 2 ) ( a i T x ) 2 ≤ ( 1 − ρ ˉ ) ∥ x ∥ 2 , (69) and ρ ˉ ∈ ( 0 , 1 ] \bar{\rho}\in(0,1] ρ ˉ ∈ ( 0 , 1 ] .
Proof. Since M i M_i M i is symmetric,
∥ M i x ∥ 2 = x T ( I − β a i a i T ) 2 x = ∥ x ∥ 2 − 2 β ( a i T x ) 2 + β 2 ∥ a i ∥ 2 ( a i T x ) 2 , (70) \lVert M_i x\rVert^2=x^{\mathsf T}(I-\beta a_i a_i^{\mathsf T})^2x=\lVert x\rVert^2-2\beta(a_i^{\mathsf T}x)^2+\beta^2\lVert a_i\rVert^2(a_i^{\mathsf T}x)^2, \tag*{(70)} ∥ M i x ∥ 2 = x T ( I − β a i a i T ) 2 x = ∥ x ∥ 2 − 2 β ( a i T x ) 2 + β 2 ∥ a i ∥ 2 ( a i T x ) 2 , (70)
which is the equality in (A59) after averaging over i ∼ p i\sim p i ∼ p . By (A45), β ( 2 − β ∥ a i ∥ 2 ) ≥ γ β > 0 \beta(2-\beta\lVert a_i\rVert^2)\ge\gamma_\beta>0 β ( 2 − β ∥ a i ∥ 2 ) ≥ γ β > 0 for every i i i , and ∑ i p i ( a i T x ) 2 = x T A ˉ x ≥ λ min ( A ˉ ) ∥ x ∥ 2 \sum_i p_i(a_i^{\mathsf T}x)^2=x^{\mathsf T}\bar{A}x\ge\lambda_{\min}(\bar{A})\lVert x\rVert^2 ∑ i p i ( a i T x ) 2 = x T A ˉ x ≥ λ m i n ( A ˉ ) ∥ x ∥ 2 , hence the inequality with ρ ˉ = γ β λ min ( A ˉ ) \bar{\rho}=\gamma_\beta\lambda_{\min}(\bar{A}) ρ ˉ = γ β λ m i n ( A ˉ ) . Finally ρ ˉ ≤ 1 \bar{\rho}\le1 ρ ˉ ≤ 1 because the left-hand side of (A59) is nonnegative for every x x x . We assumed that A ˉ \bar{A} A ˉ has full rank, therefore 0 < λ min ( A ˉ ) 0<\lambda_{\min}(\bar{A}) 0 < λ m i n ( A ˉ ) and with 0 < γ β 0<\gamma_\beta 0 < γ β we have 0 < ρ ˉ 0<\bar{\rho} 0 < ρ ˉ .
Lemma A2 (The chain lives in P 2 \mathcal{P}_2 P 2 and its kernel is a W 2 \mathcal{W}_2 W 2 -contraction). Let P P P denote the transition kernel of (A49) and let P 2 \mathcal{P}_2 P 2 be the space of probability laws on R d 2 \mathbb{R}^{d^2} R d 2 with finite second moment, equipped with the Wasserstein-2 metric W 2 \mathcal{W}_2 W 2 . Then P P P maps P 2 \mathcal{P}_2 P 2 into P 2 \mathcal{P}_2 P 2 , and for all μ , ν ∈ P 2 \mu,\nu\in\mathcal{P}_2 μ , ν ∈ P 2 ,
W 2 ( μ P , ν P ) ≤ 1 − ρ ˉ W 2 ( μ , ν ) . (71) \mathcal{W}_2(\mu P,\nu P)\le\sqrt{1-\bar{\rho}}\,\mathcal{W}_2(\mu,\nu). \tag*{(71)} W 2 ( μ P , ν P ) ≤ 1 − ρ ˉ W 2 ( μ , ν ) . (71) Consequently P P P has a unique invariant law π β ∈ P 2 \pi_\beta\in\mathcal{P}_2 π β ∈ P 2 and (A53) holds.
Proof. P P P preserves P 2 \mathcal{P}_2 P 2 : for e ∼ μ ∈ P 2 e\sim\mu\in\mathcal{P}_2 e ∼ μ ∈ P 2 and an independent index i i i , E ∥ M i e + β ρ i a i ∥ 2 ≤ 2 E ∥ M i e ∥ 2 + 2 β 2 E [ ρ 2 ∥ a ∥ 2 ] ≤ 2 E ∥ e ∥ 2 + 2 β 2 σ 2 < ∞ \mathbb{E}\lVert M_i e+\beta\rho_i a_i\rVert^2\le2\mathbb{E}\lVert M_i e\rVert^2+2\beta^2\mathbb{E}[\rho^2\lVert a\rVert^2]\le2\mathbb{E}\lVert e\rVert^2+2\beta^2\sigma^2<\infty E ∥ M i e + β ρ i a i ∥ 2 ≤ 2 E ∥ M i e ∥ 2 + 2 β 2 E [ ρ 2 ∥ a ∥ 2 ] ≤ 2 E ∥ e ∥ 2 + 2 β 2 σ 2 < ∞ , using Lemma A1.
Contraction: let ( x , y ) (x,y) ( x , y ) be an optimal W 2 \mathcal{W}_2 W 2 -coupling of ( μ , ν ) (\mu,\nu) ( μ , ν ) and couple the two chains synchronously : draw one index i i i (independent of ( x , y ) (x,y) ( x , y ) ) and set x ′ = M i x + β ρ i a i x'=M_i x+\beta\rho_i a_i x ′ = M i x + β ρ i a i , y ′ = M i y + β ρ i a i y'=M_i y+\beta\rho_i a_i y ′ = M i y + β ρ i a i . The additive terms cancel in the difference,
x ′ − y ′ = M i ( x − y ) , (72) x'-y'=M_i(x-y), \tag*{(72)} x ′ − y ′ = M i ( x − y ) , (72)
so by Lemma A1, first conditionally on ( x , y ) (x,y) ( x , y ) and then in expectation,
E ∥ x ′ − y ′ ∥ 2 ≤ ( 1 − ρ ˉ ) E ∥ x − y ∥ 2 = ( 1 − ρ ˉ ) W 2 ( μ , ν ) 2 . (73) \mathbb{E}\lVert x'-y'\rVert^2\le(1-\bar{\rho})\mathbb{E}\lVert x-y\rVert^2=(1-\bar{\rho})\mathcal{W}_2(\mu,\nu)^2. \tag*{(73)} E ∥ x ′ − y ′ ∥ 2 ≤ ( 1 − ρ ˉ ) E ∥ x − y ∥ 2 = ( 1 − ρ ˉ ) W 2 ( μ , ν ) 2 . (73)
Since ( x ′ , y ′ ) (x',y') ( x ′ , y ′ ) is some coupling of ( μ P , ν P ) (\mu P,\nu P) ( μ P , ν P ) , W 2 ( μ P , ν P ) 2 ≤ E ∥ x ′ − y ′ ∥ 2 \mathcal{W}_2(\mu P,\nu P)^2\le\mathbb{E}\lVert x'-y'\rVert^2 W 2 ( μ P , ν P ) 2 ≤ E ∥ x ′ − y ′ ∥ 2 , which is (A61).
Conclusion: ( P 2 , W 2 ) (\mathcal{P}_2,\mathcal{W}_2) ( P 2 , W 2 ) is a complete metric space (see, e.g., [57 Theorem 6.18] ), so Banach’s fixed-point theorem yields a unique π β ∈ P 2 \pi_\beta\in\mathcal{P}_2 π β ∈ P 2 with π β P = π β \pi_\beta P=\pi_\beta π β P = π β and W 2 ( μ P t , π β ) ≤ ( 1 − ρ ˉ ) t / 2 W 2 ( μ , π β ) \mathcal{W}_2(\mu P^t,\pi_\beta)\le(1-\bar{\rho})^{t/2}\mathcal{W}_2(\mu,\pi_\beta) W 2 ( μ P t , π β ) ≤ ( 1 − ρ ˉ ) t /2 W 2 ( μ , π β ) for every μ ∈ P 2 \mu\in\mathcal{P}_2 μ ∈ P 2 , which is (A53). Convergence in W 2 \mathcal{W}_2 W 2 implies convergence in distribution together with convergence of first and second moments.
Comments to the proof of Lemma A2 in 9 bullet points.
1) Overview. How a mixture of non-contractive matrices produces a contraction: the kernel P P P and the coupling argument. The matrices M i M_i M i are not themselves contractions - by Lemma A1, ∥ M i ∥ 2 = 1 \lVert M_i\rVert_2=1 ∥ M i ∥ 2 = 1 for every single i i i , since M i M_i M i has eigenvalue 1 on a i ⊥ a_i^\perp a i ⊥ . Contractivity in Lemma A2 is a property of the mixture P P P , not of any individual M i M_i M i , and P P P is not a matrix at all: it is a Markov kernel acting on laws in P 2 \mathcal{P}_2 P 2 , and the relevant notion of size for it is the Wasserstein metric W 2 \mathcal{W}_2 W 2 on P 2 \mathcal{P}_2 P 2 itself - not any norm on R d 2 \mathbb{R}^{d^2} R d 2 from which P P P ’s contraction could be read off the M i M_i M i ’s directly. This note spells out how P P P is built from the M i M_i M i ’s, in what sense P P P does and does not have an “operator norm,” and why the coupling argument nonetheless produces the contraction (A61).
Φ i ( x ) : = M i x + β ρ i a i , i = 1 , … , n . (74) \Phi_i(x):=M_i x+\beta\rho_i a_i,\qquad i=1,\ldots,n. \tag*{(74)} Φ i ( x ) := M i x + β ρ i a i , i = 1 , … , n . (74)
One step of the chain from x x x lands at Φ i ( x ) \Phi_i(x) Φ i ( x ) with probability p i p_i p i , so the transition kernel is the mixture of point masses
P ( x , ⋅ ) : = ∑ i = 1 n p i δ Φ i ( x ) ( ⋅ ) , (75) P(x,\cdot):=\sum_{i=1}^{n}p_i\delta_{\Phi_i(x)}(\cdot), \tag*{(75)} P ( x , ⋅ ) := i = 1 ∑ n p i δ Φ i ( x ) ( ⋅ ) , (75)
and, for a law μ \mu μ , μ P \mu P μ P is defined by ( μ P ) ( B ) = ∫ P ( x , B ) μ ( d x ) (\mu P)(B) = \int P(x,B)\,\mu(\mathrm{d}x) ( μ P ) ( B ) = ∫ P ( x , B ) μ ( d x ) - equivalently, μ P \mu P μ P is the law of Φ I ( X ) \Phi_I(X) Φ I ( X ) for X ∼ μ X \sim\mu X ∼ μ and I ∼ p I \sim p I ∼ p drawn independently of X X X .
B B B is a Borel (measurable) subset of R d 2 \mathbb{R}^{d^2} R d 2 - an arbitrary test set , held fixed, to which the kernel assigns a probability. Concretely,
P ( x , B ) : = Pr ( Φ I ( x ) ∈ B ) = ∑ i = 1 n p i 1 [ Φ i ( x ) ∈ B ] = ∑ i = 1 n p i δ Φ i ( x ) ( B ) , (76) P(x,B) := \Pr\bigl(\Phi_I(x) \in B\bigr) = \sum_{i=1}^{n} p_i \, \mathbb{1}\bigl[\Phi_i(x) \in B\bigr] = \sum_{i=1}^{n} p_i \,\delta_{\Phi_i(x)}(B), \tag*{(76)} P ( x , B ) := Pr ( Φ I ( x ) ∈ B ) = i = 1 ∑ n p i 1 [ Φ i ( x ) ∈ B ] = i = 1 ∑ n p i δ Φ i ( x ) ( B ) , (76)
where Φ i ( x ) = M i x + β ρ i a i \Phi_i(x) = M_i x + \beta\rho_i a_i Φ i ( x ) = M i x + β ρ i a i is the (fixed, deterministic) image of x x x under transition i i i , and I ∼ p I \sim p I ∼ p is the random index selecting which transition fires. In words: P ( x , B ) P(x,B) P ( x , B ) is the probability that one step of the chain, started at x x x , lands in B B B - obtained by summing the weights p i p_i p i of every index i i i whose image Φ i ( x ) \Phi_i(x) Φ i ( x ) happens to fall inside B B B .
P P P as a function of each argument separately. The kernel P : R d 2 × B ( R d 2 ) → [ 0 , 1 ] P : \mathbb{R}^{d^2} \times\mathcal{B}(\mathbb{R}^{d^2}) \to[0,1] P : R d 2 × B ( R d 2 ) → [ 0 , 1 ] plays a different role in each of its two slots:
Fixed x x x , varying B B B : P ( x , ⋅ ) P(x,\cdot) P ( x , ⋅ ) is itself a probability measure on R d 2 \mathbb{R}^{d^2} R d 2 - concretely, the discrete measure
P ( x , ⋅ ) = ∑ i = 1 n p i δ Φ i ( x ) ( ⋅ ) (77) P(x,\cdot) = \sum_{i=1}^{n} p_i \,\delta_{\Phi_i(x)}(\cdot) \tag*{(77)} P ( x , ⋅ ) = i = 1 ∑ n p i δ Φ i ( x ) ( ⋅ ) (77)
putting mass p i p_i p i on the point Φ i ( x ) \Phi_i(x) Φ i ( x ) for each i i i (masses are summed if several Φ i ( x ) \Phi_i(x) Φ i ( x ) coincide). This is the law of the chain’s next state, given that it is currently at x x x .
Fixed B B B , varying x x x : P ( ⋅ , B ) P(\cdot,B) P ( ⋅ , B ) is a bounded measurable function of x x x , since each indicator 1 [ Φ i ( x ) ∈ B ] \mathbb{1}[\Phi_i(x) \in B] 1 [ Φ i ( x ) ∈ B ] is measurable in x x x and the sum is finite.
This is the standard convention for a Markov transition kernel: measurable in the first argument, a probability measure in the second.
the pushforward measure μ P \mu P μ P is well defined. For a law μ \mu μ of the current state, the one-step-ahead law μ P \mu P μ P is defined by integrating the second argument B B B (held fixed) against μ \mu μ acting on the first argument x x x :
( μ P ) ( B ) : = ∫ R d 2 P ( x , B ) μ ( d x ) . (78) (\mu P)(B) := \int_{\mathbb{R}^{d^2}} P(x,B)\,\mu(\mathrm{d}x). \tag*{(78)} ( μ P ) ( B ) := ∫ R d 2 P ( x , B ) μ ( d x ) . (78)
Equivalently, μ P \mu P μ P is the law of Φ I ( X ) \Phi_I(X) Φ I ( X ) for X ∼ μ X \sim\mu X ∼ μ and I ∼ p I \sim p I ∼ p drawn independently of X X X : draw a random current state X X X , draw an independent random index I I I , and output Φ I ( X ) \Phi_I(X) Φ I ( X ) . The two roles of B B B above are exactly what make this integral meaningful: for each fixed x x x in the integral, P ( x , B ) P(x,B) P ( x , B ) is a number in [ 0 , 1 ] [0,1] [ 0 , 1 ] (the probability mass the one-step law from x x x places on B B B ); integrating that number against μ ( d x ) \mu(\mathrm{d}x) μ ( d x ) averages it over the randomness in the current state, and the result, as a function of B B B , is again a probability measure. It is the law μ P \mu P μ P appearing throughout the steady-state analysis of the rule.
P P P has no operator norm on R d 2 \mathbb{R}^{d^2} R d 2 . Thus P P P is linear as an operator on (signed) measures ( ( α μ + ( 1 − α ) ν ) P = α μ P + ( 1 − α ) ν P ) ((\alpha\mu+ (1-\alpha)\nu)P = \alpha\mu P + (1-\alpha)\nu P) (( α μ + ( 1 − α ) ν ) P = α μ P + ( 1 − α ) ν P ) , but this linearity lives on the infinite-dimensional space of measures, not on R d 2 \mathbb{R}^{d^2} R d 2 : there is no d 2 × d 2 d^2 \times d^2 d 2 × d 2 matrix representing P P P , so P P P has no operator norm on ( R d 2 , ∥ ⋅ ∥ ) (\mathbb{R}^{d^2}, \lVert\cdot\rVert) ( R d 2 , ∥ ⋅ ∥) the way each M i M_i M i does. What P P P does have is an operator norm in the sense appropriate to its actual domain: a Lipschitz modulus on the metric space ( P 2 , W 2 ) (\mathcal{P}_2,\mathcal{W}_2) ( P 2 , W 2 ) , namely the smallest L L L such that W 2 ( μ P , ν P ) ≤ L W 2 ( μ , ν ) \mathcal{W}_2(\mu P,\nu P) \le L\mathcal{W}_2(\mu,\nu) W 2 ( μ P , ν P ) ≤ L W 2 ( μ , ν ) for all μ , ν ∈ P 2 \mu,\nu\in\mathcal{P}_2 μ , ν ∈ P 2 . That modulus is exactly what (A61) bounds by 1 − ρ ˉ < 1 \sqrt{1-\bar{\rho}} < 1 1 − ρ ˉ < 1 . It cannot be read off a per-map bound such as max i ∥ M i ∥ 2 \max_i \lVert M_i\rVert_2 max i ∥ M i ∥ 2 , which equals 1 1 1 by Lemma A1 and would certify no contraction whatsoever; obtaining a modulus strictly below 1 1 1 requires the coupling argument below, precisely because it is a statement about P P P acting on laws, not about any M i M_i M i acting on R d 2 \mathbb{R}^{d^2} R d 2 .
The Wasserstein metric is essential to prove the contraction. The definition
W 2 ( μ , ν ) 2 = inf γ ∈ Γ ( μ , ν ) E γ [ ∥ X − Y ∥ 2 ] (79) \mathcal{W}_2(\mu,\nu)^2 = \inf_{\gamma\in\Gamma(\mu,\nu)} \mathbb{E}_{\gamma}\left[\lVert X-Y\rVert^2\right] \tag*{(79)} W 2 ( μ , ν ) 2 = γ ∈ Γ ( μ , ν ) inf E γ [ ∥ X − Y ∥ 2 ] (79)
is an infimum over all couplings γ \gamma γ of ( μ , ν ) (\mu,\nu) ( μ , ν ) (joint laws with the prescribed marginals). This has an immediate but crucial consequence: to upper-bound W 2 ( μ P , ν P ) \mathcal{W}_2(\mu P,\nu P) W 2 ( μ P , ν P ) it suffices to exhibit any single coupling of ( μ P , ν P ) (\mu P,\nu P) ( μ P , ν P ) and compute its second moment. Note that we never need the optimal coupling of the outputs, let alone an explicit formula for P P P as an operator.
The synchronous coupling. We are free to choose how the randomness that drives one step of each chain is generated, and we exploit that freedom by reusing the same random index for both chains. Concretely: let ( x , y ) (x,y) ( x , y ) be an optimal coupling of ( μ , ν ) (\mu,\nu) ( μ , ν ) , i.e. a joint law with x ∼ μ x \sim\mu x ∼ μ , y ∼ ν y \sim\nu y ∼ ν , and E ∥ x − y ∥ 2 = W 2 ( μ , ν ) 2 \mathbb{E}\lVert x-y\rVert^2 = W_2(\mu,\nu)^2 E ∥ x − y ∥ 2 = W 2 ( μ , ν ) 2 (such a coupling exists because R d 2 \mathbb{R}^{d^2} R d 2 is Polish; see [57 Theorem 4.1] ). Draw a single index i ∼ p i \sim p i ∼ p , independent of ( x , y ) (x,y) ( x , y ) , and set
x ′ : = Φ i ( x ) = M i x + β ρ i a i , y ′ : = Φ i ( y ) = M i y + β ρ i a i . (80) x' := \Phi_i(x) = M_i x + \beta\rho_i a_i,\qquad y' := \Phi_i(y) = M_i y + \beta\rho_i a_i . \tag*{(80)} x ′ := Φ i ( x ) = M i x + β ρ i a i , y ′ := Φ i ( y ) = M i y + β ρ i a i . (80)
Marginally, x ′ ∼ μ P x' \sim\mu P x ′ ∼ μ P and y ′ ∼ ν P y' \sim\nu P y ′ ∼ ν P by the very definition (A65) of μ P \mu P μ P , applied separately to each starting law; hence ( x ′ , y ′ ) (x',y') ( x ′ , y ′ ) is a valid coupling of ( μ P , ν P ) (\mu P,\nu P) ( μ P , ν P ) . This coupling is not necessarily the optimal one, but that is all the argument requires.
The additive target terms cancel. Because x ′ x' x ′ and y ′ y' y ′ are built from the same draw i i i , the inhomogeneous term β ρ i a i \beta\rho_i a_i β ρ i a i is identical in both and cancels exactly in the difference:
x ′ − y ′ = ( M i x + β ρ i a i ) − ( M i y + β ρ i a i ) = M i ( x − y ) . (81) x' - y' = \left(M_i x + \beta\rho_i a_i\right) - \left(M_i y + \beta\rho_i a_i\right) = M_i(x-y). \tag*{(81)} x ′ − y ′ = ( M i x + β ρ i a i ) − ( M i y + β ρ i a i ) = M i ( x − y ) . (81)
The part of the update that cancels out is exactly the part carrying the targets c i c_i c i . This is the crux of the construction: it converts a statement about two inhomogeneous, randomly driven affine updates into a statement about a single homogeneous linear map, M i M_i M i , applied to the fixed vector x − y x-y x − y . This is exactly the object Lemma A1 controls. Had we instead coupled the two chains using independent indices for x ′ x' x ′ and y ′ y' y ′ , the noise terms would not cancel and the argument would not go through; the whole content of the synchronous-coupling trick is this cancellation.
Deriving the contraction. Since i i i is independent of ( x , y ) (x,y) ( x , y ) , condition on ( x , y ) (x,y) ( x , y ) first and average over i i i :
E [ ∥ x ′ − y ′ ∥ 2 ∣ x , y ] = ∑ i = 1 n p i ∥ M i ( x − y ) ∥ 2 ≤ ( 1 − ρ ˉ ) ∥ x − y ∥ 2 (82) \mathbb{E}\left[\left\lVert x'-y'\right\rVert^2 \mid x,y\right] = \sum_{i=1}^{n} p_i \left\lVert M_i(x-y)\right\rVert^2 \le(1-\bar{\rho})\lVert x-y\rVert^2 \tag*{(82)} E [ ∥ x ′ − y ′ ∥ 2 ∣ x , y ] = i = 1 ∑ n p i ∥ M i ( x − y ) ∥ 2 ≤ ( 1 − ρ ˉ ) ∥ x − y ∥ 2 (82)
by Lemma A1, applied pointwise to the vector x − y x-y x − y (fixed once we condition). Taking the outer expectation over ( x , y ) (x,y) ( x , y ) and using that this coupling was chosen optimal,
E ∥ x ′ − y ′ ∥ 2 ≤ ( 1 − ρ ˉ ) E ∥ x − y ∥ 2 = ( 1 − ρ ˉ ) W 2 ( μ , ν ) 2 . (83) \mathbb{E}\left\lVert x'-y'\right\rVert^2 \le(1-\bar{\rho})\mathbb{E}\lVert x-y\rVert^2 = (1-\bar{\rho})W_2(\mu,\nu)^2 . \tag*{(83)} E ∥ x ′ − y ′ ∥ 2 ≤ ( 1 − ρ ˉ ) E ∥ x − y ∥ 2 = ( 1 − ρ ˉ ) W 2 ( μ , ν ) 2 . (83)
Finally, since ( x ′ , y ′ ) (x',y') ( x ′ , y ′ ) is a coupling of ( μ P , ν P ) (\mu P,\nu P) ( μ P , ν P ) , the infimum defining W 2 ( μ P , ν P ) 2 W_2(\mu P,\nu P)^2 W 2 ( μ P , ν P ) 2 is at most the cost of this particular coupling:
W 2 ( μ P , ν P ) 2 ≤ E ∥ x ′ − y ′ ∥ 2 ≤ ( 1 − ρ ˉ ) W 2 ( μ , ν ) 2 , (84) W_2(\mu P,\nu P)^2 \le\mathbb{E}\left\lVert x'-y'\right\rVert^2 \le(1-\bar{\rho})W_2(\mu,\nu)^2 , \tag*{(84)} W 2 ( μ P , ν P ) 2 ≤ E ∥ x ′ − y ′ ∥ 2 ≤ ( 1 − ρ ˉ ) W 2 ( μ , ν ) 2 , (84)
and taking square roots recovers (A61) exactly.
Summary of the approach. Three ingredients combine, none of which is a bound on P P P as an operator on R d 2 \mathbb{R}^{d^2} R d 2 : (1) W 2 W_2 W 2 is an infimum over couplings, so any one coupling of the images gives a valid upper bound; (2) the synchronous coupling is reusing the same random index for both copies and is a legitimate choice precisely because P P P is defined by mixing over that same randomness, and it makes the target-dependent noise cancel identically; (3) what remains is a bound on E i ∥ M i z ∥ 2 \mathbb{E}_i\lVert M_i z\rVert^2 E i ∥ M i z ∥ 2 for the fixed vector z = x − y z=x-y z = x − y , which is exactly the pointwise mean-square contraction already established in Lemma A1. This lemma is a statement purely about the M i M_i M i 's and A ˉ = ∑ i p i a i a i T \bar{A}=\sum_i p_i a_i a_i^\mathsf{T} A ˉ = ∑ i p i a i a i T , requiring no coupling of its own. The contraction of P P P in W 2 W_2 W 2 is thus assembled entirely from the per-index contraction of Lemma A1, transported to the space of distributions by the coupling device. It is not, and could not be, a statement recoverable from max i ∥ M i ∥ 2 \max_i\lVert M_i\rVert_2 max i ∥ M i ∥ 2 alone.
Proof of Theorem A6.
Proof. Exact mean recursion. The update (A49) is linear in s t s_t s t , and v t v_t v t is independent of F t ⊃ σ ( s t ) \mathcal{F}_t \supset\sigma(s_t) F t ⊃ σ ( s t ) . Hence, conditioning on F t \mathcal{F}_t F t ,
E [ s t + 1 ∣ F t ] = E [ M i t ] s t + β E [ c i t a i t ] = ( I − β A ˉ ) s t + β b ˉ , (85) \mathbb{E}\left[s_{t+1}\mid\mathcal{F}_t\right] = \mathbb{E}\left[M_{i_t}\right]s_t + \beta\,\mathbb{E}\left[c_{i_t}a_{i_t}\right] = (I-\beta\bar{A})s_t+\beta\bar{b}, \tag*{(85)} E [ s t + 1 ∣ F t ] = E [ M i t ] s t + β E [ c i t a i t ] = ( I − β A ˉ ) s t + β b ˉ , (85)
and the tower property gives (A51). This equation holds with equality and is not an approximation: no linearization was needed because the update itself is linear. Subtracting the fixed-point relation s L S = ( I − β A ˉ ) s L S + β b ˉ s^{\mathrm{LS}}=(I-\beta\bar{A})s^{\mathrm{LS}}+\beta\bar{b} s LS = ( I − β A ˉ ) s LS + β b ˉ , which holds by the normal equations (A48) (b ˉ = A ˉ s L S \bar{b}=\bar{A}s^{\mathrm{LS}} b ˉ = A ˉ s LS ), yields the error form and, by induction,
E [ e t ] = ( I − β A ˉ ) t E [ e 0 ] . (86) \mathbb{E}[e_t] = (I-\beta\bar{A})^t\mathbb{E}[e_0]. \tag*{(86)} E [ e t ] = ( I − β A ˉ ) t E [ e 0 ] . (86)
Spectral radius. A ˉ \bar{A} A ˉ is symmetric positive definite; let 0 < λ min = λ 1 ≤ ⋯ ≤ λ d = λ max 0 < \lambda_{\min} = \lambda_1 \le\cdots\le\lambda_d = \lambda_{\max} 0 < λ m i n = λ 1 ≤ ⋯ ≤ λ d = λ m a x be its eigenvalues. For any unit vector x x x , x ⊤ A ˉ x = ∑ i p i ( a i ⊤ x ) 2 ≤ max i ∥ a i ∥ 2 x^{\top}\bar{A}x = \sum_i p_i(a_i^{\top}x)^2 \le\max_i \lVert a_i\rVert^2 x ⊤ A ˉ x = ∑ i p i ( a i ⊤ x ) 2 ≤ max i ∥ a i ∥ 2 , so
β λ max ≤ β max i ∥ a i ∥ 2 < 2 (87) \beta\lambda_{\max} \le\beta\max_i\lVert a_i\rVert^2 < 2 \tag*{(87)} β λ m a x ≤ β i max ∥ a i ∥ 2 < 2 (87)
by (A45). Hence, every eigenvalue 1 − β λ k 1-\beta\lambda_k 1 − β λ k of I − β A ˉ I-\beta\bar{A} I − β A ˉ lies in ( − 1 , 1 ) (-1,1) ( − 1 , 1 ) : 1 − β λ k < 1 1-\beta\lambda_k < 1 1 − β λ k < 1 since β λ k > 0 \beta\lambda_k > 0 β λ k > 0 , and 1 − β λ k > − 1 1-\beta\lambda_k > -1 1 − β λ k > − 1 since β λ k < 2 \beta\lambda_k < 2 β λ k < 2 . The matrix is symmetric, so its spectral norm equals r = max k ∣ 1 − β λ k ∣ < 1 r=\max_k\lvert1-\beta\lambda_k\rvert< 1 r = max k ∣ 1 − β λ k ∣ < 1 and ∥ E [ e t ] ∥ ≤ r t ∥ e 0 ∥ \lVert\mathbb{E}[e_t]\rVert\le r^t\lVert e_0\rVert ∥ E [ e t ]∥ ≤ r t ∥ e 0 ∥ , i.e. linear convergence of E [ S t ] \mathbb{E}[S_t] E [ S t ] to S L S S^{\mathrm{LS}} S LS in Frobenius norm (∥ s ∥ 2 = ∥ S ∥ F \lVert s\rVert_2 = \lVert S\rVert_F ∥ s ∥ 2 = ∥ S ∥ F ).
Exact unbiasedness. The fixed point of the affine map z ↦ ( I − β A ˉ ) z + β b ˉ z\mapsto(I-\beta\bar{A})z+\beta\bar{b} z ↦ ( I − β A ˉ ) z + β b ˉ is unique (β A ˉ \beta\bar{A} β A ˉ invertible) and equals A ˉ − 1 b ˉ = s L S \bar{A}^{-1}\bar{b}=s^{\mathrm{LS}} A ˉ − 1 b ˉ = s LS for every β \beta β in the admissible range: the limit of the mean does not depend on β \beta β and carries no O ( β ) O(\beta) O ( β ) correction.
Proof of Theorem A7.
Proof. Existence, uniqueness, membership in P 2 \mathcal{P}_2 P 2 , and the geometric W 2 W_2 W 2 -convergence (A53) are Lemma A2. We have to prove (i)–(iv). In stationarity we write e ∼ π β e\sim\pi_\beta e ∼ π β (centered at s L S s^{\mathrm{LS}} s LS ), draw a fresh index i i i independent of e e e , and use that
e ′ : = M i e + β ρ i a i = d e , (88) e' := M_i e+\beta\rho_i a_i \mathrel{\mathop{=}^{d}} e, \tag*{(88)} e ′ := M i e + β ρ i a i = d e , (88)
with all first and second moments finite (π β ∈ P 2 \pi_\beta\in\mathcal{P}_2 π β ∈ P 2 ). Here = d \mathrel{\mathop{=}^{d}} = d means equality in distribution.
(i) Stationary mean. Taking expectations in (A78) and using the independence of i i i and e e e together with the normal equations E [ ρ a ] = 0 \mathbb{E}[\rho a]=0 E [ ρ a ] = 0 :
E [ e ] = E [ e ′ ] = E [ M i ] E [ e ] + β E [ ρ i a i ] = ( I − β A ˉ ) E [ e ] , (89) \mathbb{E}[e]=\mathbb{E}[e']=\mathbb{E}[M_i]\mathbb{E}[e]+\beta\mathbb{E}[\rho_i a_i]=(I-\beta\bar{A})\mathbb{E}[e], \tag*{(89)} E [ e ] = E [ e ′ ] = E [ M i ] E [ e ] + β E [ ρ i a i ] = ( I − β A ˉ ) E [ e ] , (89)
hence β A ˉ E [ e ] = 0 \beta\bar{A}\mathbb{E}[e]=0 β A ˉ E [ e ] = 0 , and since A ˉ \bar{A} A ˉ is invertible, E [ e ] = 0 \mathbb{E}[e]=0 E [ e ] = 0 , i.e. E π β [ S ] = S L S \mathbb{E}_{\pi_\beta}[S]=S^{\mathrm{LS}} E π β [ S ] = S LS , which holds exactly.
(ii) Exact covariance equation. Expand e ′ e ′ ⊤ e'e'^{\top} e ′ e ′ ⊤ from (A78):
e ′ e ′ ⊤ = M i e e ⊤ M i ⊤ + β ρ i M i e a i ⊤ + β ρ i a i e ⊤ M i ⊤ + β 2 ρ i 2 a i a i ⊤ . (90) e'e'^{\top}=M_i ee^{\top}M_i^{\top}+\beta\rho_i M_i e a_i^{\top}+\beta\rho_i a_i e^{\top}M_i^{\top}+\beta^2\rho_i^2 a_i a_i^{\top}. \tag*{(90)} e ′ e ′ ⊤ = M i e e ⊤ M i ⊤ + β ρ i M i e a i ⊤ + β ρ i a i e ⊤ M i ⊤ + β 2 ρ i 2 a i a i ⊤ . (90)
Take expectations term by term, using the independence of i i i and e e e and writing V β = E [ e e ⊤ ] V_\beta=\mathbb{E}[ee^{\top}] V β = E [ e e ⊤ ] .
Homogeneous term. Since M i V β M i ⊤ = V β − β ( A ˉ V β + V β A ˉ ) + β 2 ( a i ⊤ V β a i ) a i a i ⊤ M_iV_\beta M_i^{\top}=V_\beta-\beta(\bar{A}V_\beta+V_\beta\bar{A})+\beta^2(a_i^{\top}V_\beta a_i)a_i a_i^{\top} M i V β M i ⊤ = V β − β ( A ˉ V β + V β A ˉ ) + β 2 ( a i ⊤ V β a i ) a i a i ⊤ (using a i a i ⊤ V β a i a i ⊤ = ( a i ⊤ V β a i ) a i a i ⊤ a_i a_i^{\top}V_\beta a_i a_i^{\top}=(a_i^{\top}V_\beta a_i)a_i a_i^{\top} a i a i ⊤ V β a i a i ⊤ = ( a i ⊤ V β a i ) a i a i ⊤ ),
E [ M i e e ⊤ M i ⊤ ] = V β − β ( A ˉ V β + V β A ˉ ) + β 2 E [ ( a ⊤ V β a ) a a ⊤ ] . (91) \mathbb{E}[M_i e e^{\top}M_i^{\top}]=V_\beta-\beta(\bar{A}V_\beta+V_\beta\bar{A})+\beta^2\mathbb{E}[(a^{\top}V_\beta a)aa^{\top}]. \tag*{(91)} E [ M i e e ⊤ M i ⊤ ] = V β − β ( A ˉ V β + V β A ˉ ) + β 2 E [( a ⊤ V β a ) a a ⊤ ] . (91)
Cross terms. They are linear in e e e , so independence factorizes them through E [ e ] \mathbb{E}[e] E [ e ] , which vanishes by part (i):
E [ ρ i M i e a i ⊤ ] = E i [ ρ i M i E [ e ] a i ⊤ ] = 0 , (92) \mathbb{E}[\rho_i M_i e a_i^{\top}]=\mathbb{E}_i[\rho_i M_i\mathbb{E}[e]a_i^{\top}]=0, \tag*{(92)} E [ ρ i M i e a i ⊤ ] = E i [ ρ i M i E [ e ] a i ⊤ ] = 0 , (92)
and likewise for its transpose. This is the step where exact unbiasedness pays off: no O ( β 2 ) O(\beta^2) O ( β 2 ) cross contribution survives.
Stationarity. Setting E [ e ′ e ′ ⊤ ] = V β \mathbb{E}[e'e'^{\top}]=V_\beta E [ e ′ e ′ ⊤ ] = V β and collecting,
V β = V β − β ( A ˉ V β + V β A ˉ ) + β 2 E [ ( a ⊤ V β a ) a a ⊤ ] + β 2 E [ ρ 2 a a ⊤ ] ; (93) V_\beta=V_\beta-\beta(\bar{A}V_\beta+V_\beta\bar{A})+\beta^2\mathbb{E}[(a^{\top}V_\beta a)aa^{\top}]+\beta^2\mathbb{E}[\rho^2aa^{\top}]; \tag*{(93)} V β = V β − β ( A ˉ V β + V β A ˉ ) + β 2 E [( a ⊤ V β a ) a a ⊤ ] + β 2 E [ ρ 2 a a ⊤ ] ; (93)
dividing by β \beta β gives (A54), which is an exact equation, not an expansion.
Trace bounds. Take the trace of (A54):
2 tr ( A ˉ V β ) = β E [ ( a ⊤ V β a ) ∥ a ∥ 2 ] + β σ 2 . (94) 2\operatorname{tr}(\bar{A}V_\beta)=\beta\mathbb{E}[(a^{\top}V_\beta a)\lVert a\rVert^2]+\beta\sigma^2. \tag*{(94)} 2 tr ( A ˉ V β ) = β E [( a ⊤ V β a ) ∥ a ∥ 2 ] + β σ 2 . (94)
Since V β ⪰ 0 V_\beta\succeq0 V β ⪰ 0 , we have a ⊤ V β a ≥ 0 a^{\top}V_\beta a\ge0 a ⊤ V β a ≥ 0 , which gives the lower bound in (A55); and E [ ( a ⊤ V β a ) ∥ a ∥ 2 ] ≤ max i ∥ a i ∥ 2 E [ a ⊤ V β a ] = max i ∥ a i ∥ 2 tr ( A ˉ V β ) \mathbb{E}[(a^{\top}V_\beta a)\lVert a\rVert^2]\le\max_i\lVert a_i\rVert^2\mathbb{E}[a^{\top}V_\beta a]=\max_i\lVert a_i\rVert^2\operatorname{tr}(\bar{A}V_\beta) E [( a ⊤ V β a ) ∥ a ∥ 2 ] ≤ max i ∥ a i ∥ 2 E [ a ⊤ V β a ] = max i ∥ a i ∥ 2 tr ( A ˉ V β ) , so
( 2 − β max i ∥ a i ∥ 2 ) tr ( A ˉ V β ) ≤ β σ 2 , (95) \left(2-\beta\max_i\lVert a_i\rVert^2\right)\operatorname{tr}(\bar{A}V_\beta)\le\beta\sigma^2, \tag*{(95)} ( 2 − β i max ∥ a i ∥ 2 ) tr ( A ˉ V β ) ≤ β σ 2 , (95)
the upper bound (the prefactor is positive by (A45)). Finally tr V β ≤ tr ( A ˉ V ˉ β ) / λ min ( A ˉ ) \operatorname{tr} V_\beta\le\operatorname{tr}(\bar{A}\bar{V}_\beta)/\lambda_{\min}(\bar{A}) tr V β ≤ tr ( A ˉ V ˉ β ) / λ m i n ( A ˉ ) gives the Frobenius bound in (A55), and tr V β = O ( β ) \operatorname{tr} V_\beta= O(\beta) tr V β = O ( β ) makes the quartic term in (A54) O ( β 2 ) O(\beta^2) O ( β 2 ) , leaving the Lyapunov equation at leading order.
(iii) Misadjustment. For uniform sampling, A ˉ = 1 n A ⊤ A \bar{A} = \frac{1}{n}A^\top A A ˉ = n 1 A ⊤ A and L ( s ) = 1 n ∥ A s − c ∥ 2 L(s) = \frac{1}{n}\lVert As-c\rVert^2 L ( s ) = n 1 ∥ A s − c ∥ 2 . Decompose with A s − c = A e − ρ As-c=Ae-\rho A s − c = A e − ρ , where ρ = ( ρ 1 , … , ρ n ) ⊤ \rho=(\rho_1,\ldots,\rho_n)^\top ρ = ( ρ 1 , … , ρ n ) ⊤ : since A ⊤ ρ = n E [ ρ a ] = 0 A^\top\rho=n\mathbb{E}[\rho a]=0 A ⊤ ρ = n E [ ρ a ] = 0 ,
L ( s ) = 1 n ∥ A e ∥ 2 − 2 n e ⊤ A ⊤ ρ + 1 n ∥ ρ ∥ 2 = 1 n ∥ A e ∥ 2 + L min , (96) L(s)=\frac{1}{n}\lVert Ae\rVert^2-\frac{2}{n}e^\top A^\top\rho+\frac{1}{n}\lVert\rho\rVert^2=\frac{1}{n}\lVert Ae\rVert^2+L_{\min}, \tag*{(96)} L ( s ) = n 1 ∥ A e ∥ 2 − n 2 e ⊤ A ⊤ ρ + n 1 ∥ ρ ∥ 2 = n 1 ∥ A e ∥ 2 + L m i n , (96)
where L min = 1 n ∥ ρ ∥ 2 L_{\min}=\frac{1}{n}\lVert\rho\rVert^2 L m i n = n 1 ∥ ρ ∥ 2 . This also re-proves that s L S s^{\mathrm{LS}} s LS minimizes L L L . Taking stationary expectations,
E π β [ L ] − L min = 1 n E [ e ⊤ A ⊤ A e ] = tr ( A ˉ V β ) , (97) \mathbb{E}_{\pi_\beta}[L]-L_{\min}=\frac{1}{n}\mathbb{E}[e^\top A^\top A e]=\operatorname{tr}(\bar{A}V_\beta), \tag*{(97)} E π β [ L ] − L m i n = n 1 E [ e ⊤ A ⊤ A e ] = tr ( A ˉ V β ) , (97)
and inserting the exact trace identity (A84) gives (A56). The two-sided bound (A57) follows from the trace bounds (A55). Since the upper and lower bounds differ by a factor 2 / ( 2 − β max i ∥ a i ∥ 2 ) = 1 + O ( β ) 2/(2-\beta\max_i\lVert a_i\rVert^2)=1+O(\beta) 2/ ( 2 − β max i ∥ a i ∥ 2 ) = 1 + O ( β ) , the excess loss equals β 2 σ 2 ( 1 + O ( β ) ) \frac{\beta}{2}\sigma^2(1+O(\beta)) 2 β σ 2 ( 1 + O ( β )) .
(iv) Transient. Condition on F t \mathcal{F}_t F t and expand (A49):
E [ ∥ e t + 1 ∥ 2 ∣ F t ] = E [ ∥ M i e t ∥ 2 ∣ F t ] + 2 β E [ ρ i a i ⊤ M i ] e t + β 2 σ 2 . (98) \mathbb{E}\left[\lVert e_{t+1}\rVert^2\mid\mathcal{F}_t\right]=\mathbb{E}\left[\lVert M_i e_t\rVert^2\mid\mathcal{F}_t\right]+2\beta\mathbb{E}\left[\rho_i a_i^\top M_i\right]e_t+\beta^2\sigma^2. \tag*{(98)} E [ ∥ e t + 1 ∥ 2 ∣ F t ] = E [ ∥ M i e t ∥ 2 ∣ F t ] + 2 β E [ ρ i a i ⊤ M i ] e t + β 2 σ 2 . (98)
For the middle term, a i ⊤ M i = ( 1 − β ∥ a i ∥ 2 ) a i ⊤ a_i^\top M_i=(1-\beta\lVert a_i\rVert^2)a_i^\top a i ⊤ M i = ( 1 − β ∥ a i ∥ 2 ) a i ⊤ , so by the normal equations
E [ ρ i a i ⊤ M i ] = E [ ρ a ] ⊤ − β E [ ρ ∥ a ∥ 2 a ] ⊤ = − β w ⊤ , (99) \mathbb{E}\left[\rho_i a_i^\top M_i\right]=\mathbb{E}[\rho a]^\top-\beta\mathbb{E}[\rho\lVert a\rVert^2a]^\top=-\beta w^\top, \tag*{(99)} E [ ρ i a i ⊤ M i ] = E [ ρ a ] ⊤ − β E [ ρ ∥ a ∥ 2 a ] ⊤ = − β w ⊤ , (99)
with w = E [ ρ ∥ a ∥ 2 a ] w=\mathbb{E}[\rho\lVert a\rVert^2a] w = E [ ρ ∥ a ∥ 2 a ] as in the statement. Outside stationarity this cross term does not vanish, and it is O ( β 2 ) O(\beta^2) O ( β 2 ) in total. Lemma A1 and Young’s inequality − 2 β 2 w ⊤ e ≤ ρ ˉ 2 ∥ e ∥ 2 + 2 β 4 ρ ˉ ∥ w ∥ 2 -2\beta^2w^\top e\le\frac{\bar{\rho}}{2}\lVert e\rVert^2+\frac{2\beta^4}{\bar{\rho}}\lVert w\rVert^2 − 2 β 2 w ⊤ e ≤ 2 ρ ˉ ∥ e ∥ 2 + ρ ˉ 2 β 4 ∥ w ∥ 2 give
E [ ∥ e t + 1 ∥ 2 ∣ F t ] ≤ ( 1 − ρ ˉ 2 ) ∥ e t ∥ 2 + β 2 σ 2 + 2 β 4 ρ ˉ ∥ w ∥ 2 . (100) \mathbb{E}\left[\lVert e_{t+1}\rVert^2\mid\mathcal{F}_t\right]\le\left(1-\frac{\bar{\rho}}{2}\right)\lVert e_t\rVert^2+\beta^2\sigma^2+\frac{2\beta^4}{\bar{\rho}}\lVert w\rVert^2. \tag*{(100)} E [ ∥ e t + 1 ∥ 2 ∣ F t ] ≤ ( 1 − 2 ρ ˉ ) ∥ e t ∥ 2 + β 2 σ 2 + ρ ˉ 2 β 4 ∥ w ∥ 2 . (100)
Taking full expectations and iterating the affine recursion (its geometric series is bounded by 2 / ρ ˉ 2/\bar{\rho} 2/ ρ ˉ ) yields (A58). Since ρ ˉ = γ β λ min = Θ ( β ) \bar{\rho}=\gamma\beta\lambda_{\min}=\Theta(\beta) ρ ˉ = γ β λ m i n = Θ ( β ) , the horizon 2 β 2 σ 2 / ρ ˉ + 4 β 4 ∥ w ∥ 2 / ρ ˉ 2 2\beta^2\sigma^2/\bar{\rho}+4\beta^4\lVert w\rVert^2/\bar{\rho}^2 2 β 2 σ 2 / ρ ˉ + 4 β 4 ∥ w ∥ 2 / ρ ˉ 2 is O ( β ) O(\beta) O ( β ) , consistent with (A55).
Corollary A1 (Recovering exact convergence to S L S S^{\mathrm{LS}} S LS ). Three standard mechanisms remove the O ( β ) O(\sqrt{\beta}) O ( β ) fluctuation:
(i) Diminishing step sizes (Robbins–Monro): β t ↓ 0 \beta_t\downarrow0 β t ↓ 0 , ∑ t β t = ∞ \sum_t\beta_t=\infty ∑ t β t = ∞ (e.g. β t ∝ 1 / t \beta_t\propto1/t β t ∝ 1/ t ) gives S t → S L S S_t\to S^{\mathrm{LS}} S t → S LS almost surely.
(ii) Iterate averaging (Polyak–Ruppert / ergodic average): with constant β \beta β , the time average S ˉ T = 1 T ∑ t ≤ T S t → E π β [ S ] = S L S \bar{S}_T=\frac{1}{T}\sum_{t\le T}S_t\to\mathbb{E}_{\pi_\beta}[S]=S^{\mathrm{LS}} S ˉ T = T 1 ∑ t ≤ T S t → E π β [ S ] = S LS almost surely - and by Theorem A7 (i) the average is exactly unbiased.
(iii) Small-β \beta β limit: by (A55), π β \pi_\beta π β concentrates on S L S S^{\mathrm{LS}} S LS as β → 0 \beta\to0 β → 0 , with the explicit bias–variance trade-off: larger β \beta β tracks nonstationary targets faster, smaller β \beta β reduces the steady-state loss excess (A57), which is the standard LMS trade-off, inherited verbatim.
Corollary A2 (Consistent targets). The inconsistency, not the count n > d 2 n>d^2 n > d 2 , is what forbids exact convergence. Let us assume that the targets happen to be realizable, that is c i = v i ⊤ S ∘ k i c_i=v_i^\top S^\circ k_i c i = v i ⊤ S ∘ k i for some S ∘ S^\circ S ∘ . In this case L min = 0 L_{\min}=0 L m i n = 0 , all residuals ρ i \rho_i ρ i vanish, the driving noise in (A49) disappears, and the Fejér-monotone argument of the consistent case applies unchanged: S t → S ∘ S_t\to S^\circ S t → S ∘ (unique by full column rank) almost surely, linearly, for arbitrarily many associations. Two distinguished targets realize this structurally:
(i) the correlation target c i = v i ⊤ k i c_i=v_i^\top k_i c i = v i ⊤ k i , solved by S = I S=I S = I for every n n n ;
(ii) the norm target c i = ∥ v i ∥ 2 c_i=\lVert v_i\rVert^2 c i = ∥ v i ∥ 2 , solved by any exact retrieval memory S k i = v i Sk_i=v_i S k i = v i (in particular autoassociatively by S = I S=I S = I ).
For these targets the recognition memory has effectively unbounded association count as long as realizability holds.
Memory for the overloaded setting. Beyond capacity, S S S is no longer a table but an online linear regressor : the read-out v ⊤ S L S k \mathbf{v}^{\top}S^{\mathrm{LS}}\mathbf{k} v ⊤ S LS k is the best linear prediction (in the p p p -weighted least-squares sense) of the scalar signal c c c from the bilinear feature k ⊗ v \mathbf{k}\otimes\mathbf{v} k ⊗ v . Degradation is graceful since overload produces averaging plus an O ( β ) O(\sqrt{\beta}) O ( β ) jitter, never blow-up, since the homogeneous factors keep spectral norm ≤ 1 \le1 ≤ 1 .
Assumptions for the proofs. Independence of t \mathbf{t} t from the past makes the mean recursion exact and factorizes the covariance cross terms; the normal equations kill the driving term in the mean and the cross terms in stationarity (via E [ e ] = 0 \mathbb{E}[\mathbf{e}]=0 E [ e ] = 0 ); the step condition (A45) enters three times: (1) γ β > 0 \gamma_{\beta}>0 γ β > 0 (contraction), (2) β λ max ( A ˉ ) < 2 \beta\lambda_{\max}(\bar{A})<2 β λ m a x ( A ˉ ) < 2 (mean stability), and (3) 2 − β max i ∥ a i ∥ 2 > 0 2-\beta\max_i\|\mathbf{a}_i\|^2>0 2 − β max i ∥ a i ∥ 2 > 0 (trace upper bound). Full column rank gives λ min ( A ˉ ) > 0 \lambda_{\min}(\bar{A})>0 λ m i n ( A ˉ ) > 0 , used for uniqueness of S L S S^{\mathrm{LS}} S LS , for ρ ˉ > 0 \bar{\rho}>0 ρ ˉ > 0 , and to convert tr ( A ˉ V ˉ β ) \operatorname{tr}(\bar{A}\bar{V}_{\beta}) tr ( A ˉ V ˉ β ) into a bound on tr V β \operatorname{tr}V_{\beta} tr V β . Linearity of the update in the state is what makes every step of the argument exact rather than a Taylor expansion which we obtain from the quadratic loss.
Conflicting targets on the same pair: averaging vs. overwriting. The canonical inconsistency is one pair ( k , v ) (\mathbf{k},\mathbf{v}) ( k , v ) presented with different targets. Under random sampling with probabilities p j p_j p j over targets c j c_j c j , the stationary mean read-out is the weighted average, v ⊤ E π β [ S ] k = ∑ j p j c j / ∑ j p j \mathbf{v}^{\top}\mathbb{E}_{\pi_{\beta}}[S]\mathbf{k}=\sum_j p_jc_j/\sum_j p_j v ⊤ E π β [ S ] k = ∑ j p j c j / ∑ j p j (the LS solution for identical rows). Under sequential presentation the same rule overwrites : the contraction drives the read-out to the most recent target. Random replay averages; ordered replay recencies, therefore the sampling protocol, not the rule, decides about overwrite behavior.
Forget gate: recency-weighted least squares. With the scalar gated variant S t + 1 = f t S t − β ( v ⊤ f t S t k t − c t ) v k t ⊤ S_{t+1}=f_tS_t-\beta(\mathbf{v}^{\top}f_tS_t\mathbf{k}_t-c_t)\mathbf{v}\mathbf{k}_t^{\top} S t + 1 = f t S t − β ( v ⊤ f t S t k t − c t ) v k t ⊤ , f t ∈ ( 0 , 1 ) f_t\in(0,1) f t ∈ ( 0 , 1 ) , the steady state is the corresponding exponentially weighted least-squares solution: old associations are geometrically down-weighted, so the effective association count is finite (∼ 1 / ( 1 − f ) \sim1/(1-f) ∼ 1/ ( 1 − f ) ) and the memory tracks a drifting target distribution. Therefore, the overloaded regime is exactly where the forget gate is most relevant.
Summary. For n > d 2 n>d^2 n > d 2 associations the randomized Kronecker Delta Rule with constant β \beta β does not converge to a point but to a stationary distribution: its mean is exactly the sampling-weighted least-squares memory S L S = argmin ∑ i p i ( v i ⊤ S k i − c i ) 2 S^{\mathrm{LS}}=\operatorname*{argmin}\sum_i p_i(\mathbf{v}_i^{\top}S\mathbf{k}_i-c_i)^2 S LS = argmin ∑ i p i ( v i ⊤ S k i − c i ) 2 (the minimizer of L L L for uniform sampling), its covariance is O ( β ) O(\beta) O ( β ) and solves the exact stationarity equation (A54) driven by the optimal residuals, and its loss sits at L min + β 2 E [ ρ i 2 ∥ k i ∥ 2 ∥ v i ∥ 2 ] L_{\min}+\frac{\beta}{2}\mathbb{E}[\rho_i^2\|\mathbf{k}_i\|^2\|\mathbf{v}_i\|^2] L m i n + 2 β E [ ρ i 2 ∥ k i ∥ 2 ∥ v i ∥ 2 ] up to an explicit 1 + O ( β ) 1+O(\beta) 1 + O ( β ) factor. All of this is proven from first principles: an exact linear mean recursion with spectral radius < 1 <1 < 1 ; a synchronous-coupling W 2 \mathcal{W}_2 W 2 -contraction plus Banach fixed point for existence, uniqueness, and geometric convergence of π β \pi_\beta π β ; exact moment identities in stationarity for mean and covariance; an exact trace identity for the misadjustment; and a Young-inequality recursion for the transient. Averaging or annealing β \beta β recovers S L S S^{\mathrm{LS}} S LS exactly; consistent target structures (correlation target, norm target with a retrieval solution) escape the capacity limit altogether; and a forget gate turns the steady state into recency-weighted least squares.
Norm Bounds for the KronLSTM with Steady-State Solution
We derive a two-sided norm bound for the steady state of the randomized Kronecker Delta Rule with more than d 2 d^2 d 2 associations. Running memories or data stream i.e. online updates are covered via the steady state, therefore we obtain an expected bound on the online updated memory.
Overview
For the overloaded KronLSTM (n n n associations, features a i = k i ⊗ v i \mathbf{a}_i=\mathbf{k}_i\otimes\mathbf{v}_i a i = k i ⊗ v i , sampling weights p i p_i p i ) the randomized Kronecker Delta Rule converges around the weighted least-squares memory S L S S^{\mathrm{LS}} S LS . We derive a two-sided bound on its Frobenius norm from an exact energy (Pythagoras) identity:
E [ c 2 ] − E [ ρ 2 ] λ max ( A ˉ ) ≤ ∥ S L S ∥ F 2 ≤ E [ c 2 ] − E [ ρ 2 ] λ min ( A ˉ ) , \frac{\mathbb{E}[c^2]-\mathbb{E}[\rho^2]}{\lambda_{\max}(\bar{A})}\leq\|S^{\mathrm{LS}}\|_F^2\leq\frac{\mathbb{E}[c^2]-\mathbb{E}[\rho^2]}{\lambda_{\min}(\bar{A})}, λ m a x ( A ˉ ) E [ c 2 ] − E [ ρ 2 ] ≤ ∥ S LS ∥ F 2 ≤ λ m i n ( A ˉ ) E [ c 2 ] − E [ ρ 2 ] ,
where ρ i \rho_i ρ i are the optimal residuals and A ˉ = E [ a a ⊤ ] \bar{A}=\mathbb{E}[\mathbf{a}\mathbf{a}^{\top}] A ˉ = E [ a a ⊤ ] is the feature second moment. The numerator is the explained target energy ; the bound is attained (both sides coincide) for isotropic designs, and in the orthonormal design it reproduces ∥ S ⋆ ∥ F 2 = ∑ i c i 2 \lVert S^{\star}\rVert_{\mathrm{F}}^{2} = \sum_i c_i^{2} ∥ S ⋆ ∥ F 2 = ∑ i c i 2 of the target-weighted Hebbian superposition exactly. Combined with the steady-state analysis, it yields an a-priori bound on the expected memory norm of the running iterates.
Setting and nomenclature
As before, we assume n n n associations ( k i , v i , c i ) (k_i, v_i, c_i) ( k i , v i , c i ) with features a i = k i ⊗ v i ∈ R d 2 a_i = k_i \otimes v_i \in\mathbb{R}^{d^2} a i = k i ⊗ v i ∈ R d 2 , ∥ a i ∥ = ∥ k i ∥ ∥ v i ∥ \lVert a_i\rVert= \lVert k_i\rVert\lVert v_i\rVert ∥ a i ∥ = ∥ k i ∥ ∥ v i ∥ , sampling probabilities p i > 0 p_i > 0 p i > 0 , and
A ˉ = E [ a a T ] = ∑ i = 1 n p i a i a i T , s L S = argmin s E [ ( a T s − c ) 2 ] = A ˉ − 1 E [ c a ] , (101) \bar{A} = \mathbb{E}[aa^{\mathsf{T}}] = \sum_{i=1}^{n} p_i a_i a_i^{\mathsf{T}}, \qquad s^{\mathrm{LS}} = \underset{s}{\operatorname{argmin}}\ \mathbb{E}\left[(a^{\mathsf{T}}s-c)^2\right] = \bar{A}^{-1}\mathbb{E}[ca], \tag*{(101)} A ˉ = E [ a a T ] = i = 1 ∑ n p i a i a i T , s LS = s argmin E [ ( a T s − c ) 2 ] = A ˉ − 1 E [ c a ] , (101)
ρ i = c i − a i T s L S , E [ ρ a ] = 0 (normal equations) , (102) \rho_i = c_i-a_i^{\mathsf{T}}s^{\mathrm{LS}}, \qquad\mathbb{E}[\rho a] = 0 \quad\text{(normal equations)}, \tag*{(102)} ρ i = c i − a i T s LS , E [ ρ a ] = 0 (normal equations) , (102)
assuming A ˉ ≻ 0 \bar{A} \succ0 A ˉ ≻ 0 (features span R d 2 \mathbb{R}^{d^2} R d 2 ; the rank-deficient case is treated in the remarks). All expectations are over i ∼ p i \sim p i ∼ p . Recall ∥ s ∥ 2 = ∥ S ∥ F \lVert s\rVert_2 = \lVert S\rVert_{\mathrm{F}} ∥ s ∥ 2 = ∥ S ∥ F for s = vec ( S ) s = \operatorname{vec}(S) s = vec ( S ) , and that for uniform sampling E [ ρ 2 ] = L min \mathbb{E}[\rho^2] = \mathrm{L}_{\min} E [ ρ 2 ] = L m i n , the minimum of the loss L \mathrm{L} L .
An exact energy identity
Lemma A3 (Energy identity). Let y i : = a i T s L S = v i T S L S k i y_i := a_i^{\mathsf{T}}s^{\mathrm{LS}} = v_i^{\mathsf{T}}S^{\mathrm{LS}}k_i y i := a i T s LS = v i T S LS k i be the optimal read-outs. Then E [ ρ y ] = 0 \mathbb{E}[\rho y] = 0 E [ ρ y ] = 0 and
E [ c 2 ] = E [ y 2 ] + E [ ρ 2 ] , ( s L S ) T A ˉ s L S = E [ y 2 ] = E [ c 2 ] − E [ ρ 2 ] . (103) \mathbb{E}[c^2] = \mathbb{E}[y^2] + \mathbb{E}[\rho^2], \qquad(s^{\mathrm{LS}})^{\mathsf{T}}\bar{A}s^{\mathrm{LS}} = \mathbb{E}[y^2] = \mathbb{E}[c^2]-\mathbb{E}[\rho^2]. \tag*{(103)} E [ c 2 ] = E [ y 2 ] + E [ ρ 2 ] , ( s LS ) T A ˉ s LS = E [ y 2 ] = E [ c 2 ] − E [ ρ 2 ] . (103)
Proof. E [ ρ y ] = E [ ρ a ] T s L S = 0 \mathbb{E}[\rho y] = \mathbb{E}[\rho a]^{\mathsf{T}}s^{\mathrm{LS}} = 0 E [ ρ y ] = E [ ρ a ] T s LS = 0 by the normal equations; with c = y + ρ c = y+\rho c = y + ρ this gives the first identity. The second follows from E [ y 2 ] = ( s L S ) T E [ a a T ] s L S \mathbb{E}[y^2] = (s^{\mathrm{LS}})^{\mathsf{T}}\mathbb{E}[aa^{\mathsf{T}}]s^{\mathrm{LS}} E [ y 2 ] = ( s LS ) T E [ a a T ] s LS .
The stored read-out energy of the least-squares memory equals the target energy minus the unexplained (residual) energy. Therefore, the memory can only be as large as the part of the target signal that the bilinear features can explain.
The Norm Bound
Theorem A8 (Two-sided Frobenius-norm bound). With λ min , λ max \lambda_{\min}, \lambda_{\max} λ m i n , λ m a x the extreme eigenvalues of A ˉ \bar{A} A ˉ ,
E [ c 2 ] − E [ ρ 2 ] λ max ( A ˉ ) ≤ ∥ S L S ∥ F 2 ≤ E [ c 2 ] − E [ ρ 2 ] λ min ( A ˉ ) . (104) \frac{\mathbb{E}[c^2]-\mathbb{E}[\rho^2]}{\lambda_{\max}(\bar{A})} \le\lVert S^{\mathrm{LS}}\rVert_{\mathrm{F}}^2 \le\frac{\mathbb{E}[c^2]-\mathbb{E}[\rho^2]}{\lambda_{\min}(\bar{A})}. \tag*{(104)} λ m a x ( A ˉ ) E [ c 2 ] − E [ ρ 2 ] ≤ ∥ S LS ∥ F 2 ≤ λ m i n ( A ˉ ) E [ c 2 ] − E [ ρ 2 ] . (104) In particular, since σ max ( S L S ) ≤ ∥ S L S ∥ F \sigma_{\max}(S^{\mathrm{LS}}) \le\lVert S^{\mathrm{LS}}\rVert_{\mathrm{F}} σ m a x ( S LS ) ≤ ∥ S LS ∥ F ,
σ max ( S L S ) ≤ E [ c 2 ] − E [ ρ 2 ] λ min ( A ˉ ) ≤ E [ c 2 ] λ min ( A ˉ ) . (105) \sigma_{\max}(S^{\mathrm{LS}}) \le\sqrt{\frac{\mathbb{E}[c^2]-\mathbb{E}[\rho^2]}{\lambda_{\min}(\bar{A})}} \le\sqrt{\frac{\mathbb{E}[c^2]}{\lambda_{\min}(\bar{A})}}. \tag*{(105)} σ m a x ( S LS ) ≤ λ m i n ( A ˉ ) E [ c 2 ] − E [ ρ 2 ] ≤ λ m i n ( A ˉ ) E [ c 2 ] . (105)
Proof. Sandwich the quadratic form: λ min ∥ s L S ∥ 2 ≤ ( s L S ) T A ˉ s L S ≤ λ max ∥ s L S ∥ 2 \lambda_{\min}\lVert s^{\mathrm{LS}}\rVert^2 \le(s^{\mathrm{LS}})^{\mathsf{T}}\bar{A}s^{\mathrm{LS}} \le\lambda_{\max}\lVert s^{\mathrm{LS}}\rVert^2 λ m i n ∥ s LS ∥ 2 ≤ ( s LS ) T A ˉ s LS ≤ λ m a x ∥ s LS ∥ 2 , and insert (A93).
Corollary A3 (First-moment variant). From λ min ∥ s ∥ 2 ≤ s T A ˉ s = s T E [ c a ] ≤ ∥ s ∥ E [ ∣ c ∣ ∥ a ∥ ] \lambda_{\min}\lVert s\rVert^2 \le s^{\mathsf{T}}\bar{A}s = s^{\mathsf{T}}\mathbb{E}[ca] \le\lVert s\rVert\mathbb{E}[|c|\lVert a\rVert] λ m i n ∥ s ∥ 2 ≤ s T A ˉ s = s T E [ c a ] ≤ ∥ s ∥ E [ ∣ c ∣ ∥ a ∥] ,
∥ S L S ∥ F ≤ E [ ∣ c i ∣ ∥ k i ∥ ∥ v i ∥ ] λ min ( A ˉ ) . (106) \lVert S^{\mathrm{LS}}\rVert_{\mathrm{F}} \le\frac{\mathbb{E}[|c_i|\lVert k_i\rVert\lVert v_i\rVert]}{\lambda_{\min}(\bar{A})}. \tag*{(106)} ∥ S LS ∥ F ≤ λ m i n ( A ˉ ) E [ ∣ c i ∣ ∥ k i ∥ ∥ v i ∥] . (106)
The accumulated write mass β c t ∥ k t ∥ ∥ v t ∥ \beta c_t\lVert k_t\rVert\lVert v_t\rVert β c t ∥ k t ∥ ∥ v t ∥ is exactly the quantity whose first moment appears here. The second moment bound (A94) is usually much sharper (it divides by λ min \lambda_{\min} λ m i n only once, under the square root).
Tightness and special cases
Isotropic design gives equality. If A ˉ = E [ ∥ a ∥ 2 ] d 2 I \bar{A} = \frac{\mathbb{E}[\lVert a\rVert^2]}{d^2}I A ˉ = d 2 E [∥ a ∥ 2 ] I (features spread evenly over R d 2 \mathbb{R}^{d^2} R d 2 ), then λ min = λ max \lambda_{\min} = \lambda_{\max} λ m i n = λ m a x and (A94) collapses to the identity
∥ S L S ∥ F 2 = d 2 ( E [ c 2 ] − E [ ρ 2 ] ) E [ ∥ a ∥ 2 ] . (107) \lVert S^{\mathrm{LS}}\rVert_{\mathrm{F}}^2 = \frac{d^2\left(\mathbb{E}[c^2]-\mathbb{E}[\rho^2]\right)}{\mathbb{E}[\lVert a\rVert^2]}. \tag*{(107)} ∥ S LS ∥ F 2 = E [∥ a ∥ 2 ] d 2 ( E [ c 2 ] − E [ ρ 2 ] ) . (107)
Orthonormal design recovers the Hebbian superposition norm. For n = d 2 n = d^2 n = d 2 with orthonormal { k i } \{k_i\} { k i } , orthonormal { v i } \{v_i\} { v i } , and uniform sampling, the features are orthonormal, A ˉ = I / d 2 \bar{A} = I/d^2 A ˉ = I / d 2 , the system is consistent (ρ ≡ 0 \rho\equiv0 ρ ≡ 0 ), and (A94) holds with equality:
∥ S ⋆ ∥ F 2 = d 2 E [ c 2 ] = ∑ i = 1 d 2 c i 2 (108) \lVert S^\star\rVert_F^2 = d^2 \mathbb{E}[c^2] = \sum_{i=1}^{d^2} c_i^2 \tag*{(108)} ∥ S ⋆ ∥ F 2 = d 2 E [ c 2 ] = i = 1 ∑ d 2 c i 2 (108)
exactly the Frobenius norm of the target-weighted Hebbian superposition S ⋆ = ∑ i c i v i k i ⊤ S^\star= \sum_i c_i v_i k_i^\top S ⋆ = ∑ i c i v i k i ⊤ . The bound is therefore attained, not merely an estimate.
Unit-norm keys and values give a d 2 d^2 d 2 conditioning factor. If ∥ k i ∥ = ∥ v i ∥ = 1 \lVert k_i\rVert= \lVert v_i\rVert= 1 ∥ k i ∥ = ∥ v i ∥ = 1 then tr A ˉ = E [ ∥ a ∥ 2 ] = 1 \operatorname{tr}\bar{A} = \mathbb{E}[\lVert a\rVert^2] = 1 tr A ˉ = E [∥ a ∥ 2 ] = 1 , so λ min ≤ 1 / d 2 ≤ λ max \lambda_{\min} \le1/d^2 \le\lambda_{\max} λ m i n ≤ 1/ d 2 ≤ λ m a x : the lower bound in (A94) is at most E [ c 2 ] − E [ ρ 2 ] \mathbb{E}[c^2] - \mathbb{E}[\rho^2] E [ c 2 ] − E [ ρ 2 ] , the upper bound at least d 2 ( E [ c 2 ] − E [ ρ 2 ] ) d^2(\mathbb{E}[c^2] - \mathbb{E}[\rho^2]) d 2 ( E [ c 2 ] − E [ ρ 2 ]) , with both collapsing to the latter in the isotropic case. Ill-spread designs pay in λ min \lambda_{\min} λ m i n : the worst case is a target profile aligned with the least-covered feature direction, and the 1 / λ min 1/\lambda_{\min} 1/ λ m i n dependence is unavoidable in general.
Consistent case where association errors are zero. If the system is consistent (ρ ≡ 0 \rho\equiv0 ρ ≡ 0 ; e.g. n ≤ d 2 n \le d^2 n ≤ d 2 with independent features, or the correlation/norm targets), then S L S = S ⋆ S^{\mathrm{LS}} = S^\star S LS = S ⋆ is the exact solution and
∥ S ⋆ ∥ F 2 ≤ E [ c 2 ] λ min ( A ˉ ) (109) \lVert S^\star\rVert_F^2 \le\frac{\mathbb{E}[c^2]}{\lambda_{\min}(\bar{A})} \tag*{(109)} ∥ S ⋆ ∥ F 2 ≤ λ m i n ( A ˉ ) E [ c 2 ] (109)
Residual energy can only shrink the memory: overload (inconsistency) makes the least-squares memory smaller than a consistent memory with the same target energy, never larger.
Rank-deficient features. If the features do not span R d 2 \mathbb{R}^{d^2} R d 2 (n < d 2 n < d^2 n < d 2 or dependent a i a_i a i ), replace S L S S^{\mathrm{LS}} S LS by the minimum-Frobenius-norm least-squares solution and λ min \lambda_{\min} λ m i n by the smallest nonzero eigenvalue of A ˉ \bar{A} A ˉ : since s L S ∈ span { a i } s^{\mathrm{LS}} \in\operatorname{span}\{a_i\} s LS ∈ span { a i } (as do the iterates from S 0 = 0 S_0 = 0 S 0 = 0 ), the sandwich argument applies on that subspace and (A94) holds verbatim with λ min + \lambda_{\min}^{+} λ m i n + .
Bound on the memory norm for steady-state and all updates
Bound on the memory norm for steady-state distribution. First, we consider a bound on the memory norm under the steady-state distribution.
Corollary A4 (Bound on the steady-state memory norm). Under the randomized rule with step size 0 < β < 2 / max i ∥ a i ∥ 2 0 < \beta< 2/\max_i \lVert a_i\rVert^2 0 < β < 2/ max i ∥ a i ∥ 2 , the stationary distribution π β \pi_\beta π β has mean exactly S L S S^{\mathrm{LS}} S LS , so the second moment decomposes orthogonally, E π β ∥ S ∥ F 2 = ∥ S L S ∥ F 2 + tr V β \mathbb{E}_{\pi_\beta}\lVert S\rVert_F^2 = \lVert S^{\mathrm{LS}}\rVert_F^2 + \operatorname{tr} V_\beta E π β ∥ S ∥ F 2 = ∥ S LS ∥ F 2 + tr V β , and with the covariance trace bound of the steady-state note,
E π β [ ∥ S ∥ F 2 ] ≤ E [ c 2 ] − E [ ρ 2 ] λ min ( A ˉ ) + β E [ ρ 2 ∥ k ∥ 2 ∥ v ∥ 2 ] ( 2 − β max i ∥ a i ∥ 2 ) λ min ( A ˉ ) (110) \mathbb{E}_{\pi_\beta}\left[\lVert S\rVert_F^2\right] \le\frac{\mathbb{E}[c^2] - \mathbb{E}[\rho^2]}{\lambda_{\min}(\bar{A})} + \frac{\beta\mathbb{E}[\rho^2 \lVert k\rVert^2 \lVert v\rVert^2]}{(2 - \beta\max_i \lVert a_i\rVert^2)\lambda_{\min}(\bar{A})} \tag*{(110)} E π β [ ∥ S ∥ F 2 ] ≤ λ m i n ( A ˉ ) E [ c 2 ] − E [ ρ 2 ] + ( 2 − β max i ∥ a i ∥ 2 ) λ m i n ( A ˉ ) β E [ ρ 2 ∥ k ∥ 2 ∥ v ∥ 2 ] (110) The first term is the size of the memory content, the second the O ( β ) O(\beta) O ( β ) fluctuation; for β → 0 \beta\to0 β → 0 or in the consistent case the bound reduces to Theorem A8.
Theorem A9 (Two-sided memory norm bound for the steady state). Let π β \pi_\beta π β be the steady-state law of (A16), V β = E π β [ e e ⊤ ] V_\beta= \mathbb{E}_{\pi_\beta}[ee^\top] V β = E π β [ e e ⊤ ] *its covariance as in Theorem A7 (ii) (the two coincide because the stationary mean is exactly s L S s^{\mathrm{LS}} s LS ), σ 2 = E [ ρ 2 ∥ u ∥ 2 ] \sigma^2 = \mathbb{E}[\rho^2\lVert u\rVert^2] σ 2 = E [ ρ 2 ∥ u ∥ 2 ] , and write
A max 2 : = max i ∥ a i ∥ 2 = max i ∥ k i ∥ 2 ∥ v i ∥ 2 , κ : = β A max 2 ∈ ( 0 , 2 ) , (111) A_{\max}^2 := \max_i \lVert a_i\rVert^2 = \max_i \lVert k_i\rVert^2 \lVert v_i\rVert^2,\qquad\kappa:= \beta A_{\max}^2 \in(0,2), \tag*{(111)} A m a x 2 := i max ∥ a i ∥ 2 = i max ∥ k i ∥ 2 ∥ v i ∥ 2 , κ := β A m a x 2 ∈ ( 0 , 2 ) , (111) the last inclusion being the step-size condition (A45). Then the energy identity
s L S ⊤ A ˉ s L S = E [ c 2 ] − E [ ρ 2 ] (112) s^{\mathrm{LS}\top}\bar{A}s^{\mathrm{LS}} = \mathbb{E}[c^2] - \mathbb{E}[\rho^2] \tag*{(112)} s LS ⊤ A ˉ s LS = E [ c 2 ] − E [ ρ 2 ] (112)
holds, hence the least-squares memory is pinned on both sides,
E [ c 2 ] − E [ ρ 2 ] λ max ( A ˉ ) ≤ ∥ S L S ∥ F 2 ≤ E [ c 2 ] − E [ ρ 2 ] λ min ( A ˉ ) , (113) \frac{\mathbb{E}[c^2]-\mathbb{E}[\rho^2]}{\lambda_{\max}(\bar{A})} \le\lVert S^{\mathrm{LS}}\rVert_{\mathrm{F}}^2 \le\frac{\mathbb{E}[c^2]-\mathbb{E}[\rho^2]}{\lambda_{\min}(\bar{A})}, \tag*{(113)} λ m a x ( A ˉ ) E [ c 2 ] − E [ ρ 2 ] ≤ ∥ S LS ∥ F 2 ≤ λ m i n ( A ˉ ) E [ c 2 ] − E [ ρ 2 ] , (113)
and the steady-state memory itself satisfies
E π β [ ∥ S ∥ F 2 ] = ∥ S L S ∥ F 2 + tr V β , tr V β ≤ β E [ ρ 2 ∥ a ∥ 2 ] ( 2 − κ ) λ min ( A ˉ ) . (114) \mathbb{E}_{\pi_\beta}\left[\lVert S\rVert_{\mathrm{F}}^2\right] = \lVert S^{\mathrm{LS}}\rVert_{\mathrm{F}}^2+\operatorname{tr}V_\beta,\qquad\operatorname{tr}V_\beta\le\frac{\beta\mathbb{E}[\rho^2\lVert a\rVert^2]}{(2-\kappa)\lambda_{\min}(\bar{A})}. \tag*{(114)} E π β [ ∥ S ∥ F 2 ] = ∥ S LS ∥ F 2 + tr V β , tr V β ≤ ( 2 − κ ) λ m i n ( A ˉ ) β E [ ρ 2 ∥ a ∥ 2 ] . (114)
The fluctuation term admits the exact decomposition
tr ( A ˉ V β ) = β 2 σ 2 + R β , (115) \operatorname{tr}(\bar{A}V_\beta)=\frac{\beta}{2}\sigma^2+R_\beta, \tag*{(115)} tr ( A ˉ V β ) = 2 β σ 2 + R β , (115)
R β ≔ β 2 E [ ( a T V β a ) ∥ a ∥ 2 ] , 0 ≤ R β ≤ κ 2 − κ ⋅ β σ 2 2 . R_\beta\coloneqq\frac{\beta}{2}\mathbb{E}\left[(a^{\mathsf{T}}V_\beta a)\lVert a\rVert^2\right],\qquad0\le R_\beta\le\frac{\kappa}{2-\kappa}\cdot\frac{\beta\sigma^2}{2}. R β : = 2 β E [ ( a T V β a ) ∥ a ∥ 2 ] , 0 ≤ R β ≤ 2 − κ κ ⋅ 2 β σ 2 .
R β R_\beta R β is bounded by β 2 A max 2 σ 2 / ( 2 ( 2 − κ ) ) \beta^2 A_{\max}^2\sigma^2/(2(2-\kappa)) β 2 A m a x 2 σ 2 / ( 2 ( 2 − κ )) . Consequently, since E [ ρ 2 ] ≤ E [ c 2 ] \mathbb{E}[\rho^2]\le\mathbb{E}[c^2] E [ ρ 2 ] ≤ E [ c 2 ] ,
E π β [ ∥ S ∥ F 2 ] ≤ 1 λ min ( A ˉ ) ( E [ c 2 ] − 2 ( 1 − κ ) 2 − κ E [ ρ 2 ] ) ≤ E [ c 2 ] λ min ( A ˉ ) max { 1 , κ 2 − κ } , (116) \mathbb{E}_{\pi_\beta}\left[\lVert S\rVert_{\mathrm{F}}^2\right]\le\frac{1}{\lambda_{\min}(\bar{A})}\left(\mathbb{E}[c^2]-\frac{2(1-\kappa)}{2-\kappa}\mathbb{E}[\rho^2]\right)\le\frac{\mathbb{E}[c^2]}{\lambda_{\min}(\bar{A})}\max\left\{1,\frac{\kappa}{2-\kappa}\right\}, \tag*{(116)} E π β [ ∥ S ∥ F 2 ] ≤ λ m i n ( A ˉ ) 1 ( E [ c 2 ] − 2 − κ 2 ( 1 − κ ) E [ ρ 2 ] ) ≤ λ m i n ( A ˉ ) E [ c 2 ] max { 1 , 2 − κ κ } , (116)
and in particular, whenever the step size satisfies β ∥ a t ∥ 2 ≤ 1 \beta\lVert a_t\rVert^2\le1 β ∥ a t ∥ 2 ≤ 1 (i.e. κ ≤ 1 \kappa\le1 κ ≤ 1 , the non-expansive regime),
E π β [ ∥ S ∥ F 2 ] ≤ E [ c 2 ] λ min ( A ˉ ) . (117) \mathbb{E}_{\pi_\beta}\left[\lVert S\rVert_{\mathrm{F}}^2\right]\le\frac{\mathbb{E}[c^2]}{\lambda_{\min}(\bar{A})}. \tag*{(117)} E π β [ ∥ S ∥ F 2 ] ≤ λ m i n ( A ˉ ) E [ c 2 ] . (117)
All statements are exact and non-asymptotic: they hold for every β \beta β admitted by (A45).
Proof. The normal equations A ˉ s L S = b \bar{A}s^{\mathrm{LS}}=b A ˉ s LS = b give E [ ρ 2 ] = E [ c 2 ] − 2 b T s L S + s L S T A ˉ s L S = E [ c 2 ] − s L S T A ˉ s L S \mathbb{E}[\rho^2]=\mathbb{E}[c^2]-2b^{\mathsf{T}}s^{\mathrm{LS}}+s^{\mathrm{LS}\mathsf{T}}\bar{A}s^{\mathrm{LS}}=\mathbb{E}[c^2]-s^{\mathrm{LS}\mathsf{T}}\bar{A}s^{\mathrm{LS}} E [ ρ 2 ] = E [ c 2 ] − 2 b T s LS + s LS T A ˉ s LS = E [ c 2 ] − s LS T A ˉ s LS , which is (A102); bounding the Rayleigh quotient s L S T A ˉ s L S s^{\mathrm{LS}\mathsf{T}}\bar{A}s^{\mathrm{LS}} s LS T A ˉ s LS between λ min ( A ˉ ) ∥ s L S ∥ 2 \lambda_{\min}(\bar{A})\lVert s^{\mathrm{LS}}\rVert^2 λ m i n ( A ˉ ) ∥ s LS ∥ 2 and λ max ( A ˉ ) ∥ s L S ∥ 2 \lambda_{\max}(\bar{A})\lVert s^{\mathrm{LS}}\rVert^2 λ m a x ( A ˉ ) ∥ s LS ∥ 2 , and using ∥ s ∥ 2 = ∥ S ∥ F \lVert s\rVert_2=\lVert S\rVert_{\mathrm{F}} ∥ s ∥ 2 = ∥ S ∥ F , gives (A103). The first part of (A104) is the bias–variance split $\mathbb{E}\lVert s\rVert^2=\lVert\mathbb{E}s\rVert^2+\operatorname{Tr}\operatorname{Cov}(s)$ together with the fact that the stationary mean is exactly $s^{\mathrm{LS}}$ (Theorem A7 (i)`).
Exact stationarity equation. Rather than the leading-order Lyapunov equation, we use the exact stationarity equation (A54) of Theorem A7 (ii),
A ˉ V β + V β A ˉ − β E [ ( a T V β a ) a a T ] = β E [ ρ 2 a a T ] , (118) \bar{A}V_\beta+V_\beta\bar{A}-\beta\mathbb{E}\left[(a^{\mathsf{T}}V_\beta a)aa^{\mathsf{T}}\right]=\beta\mathbb{E}[\rho^2aa^{\mathsf{T}}], \tag*{(118)} A ˉ V β + V β A ˉ − β E [ ( a T V β a ) a a T ] = β E [ ρ 2 a a T ] , (118)
whose trace is 2 tr ( A ˉ V β ) = β σ 2 + β E [ ( a T V β a ) ∥ a ∥ 2 ] 2\operatorname{tr}(\bar{A}V_\beta)=\beta\sigma^2+\beta\mathbb{E}[(a^{\mathsf{T}}V_\beta a)\lVert a\rVert^2] 2 tr ( A ˉ V β ) = β σ 2 + β E [( a T V β a ) ∥ a ∥ 2 ] . This is (A105), and it identifies the remainder: R β ≥ 0 R_\beta\ge0 R β ≥ 0 because V β ⪰ 0 V_\beta\succeq0 V β ⪰ 0 , while E [ ( a T V β a ) ∥ a ∥ 2 ] ≤ A max 2 E [ a T V β a ] = A max 2 tr ( A ˉ V β ) \mathbb{E}[(a^{\mathsf{T}}V_\beta a)\lVert a\rVert^2]\le A_{\max}^2\mathbb{E}[a^{\mathsf{T}}V_\beta a]=A_{\max}^2\operatorname{tr}(\bar{A}V_\beta) E [( a T V β a ) ∥ a ∥ 2 ] ≤ A m a x 2 E [ a T V β a ] = A m a x 2 tr ( A ˉ V β ) gives ( 2 − κ ) tr ( A ˉ V β ) ≤ β σ 2 (2-\kappa)\operatorname{tr}(\bar{A}V_\beta)\le\beta\sigma^2 ( 2 − κ ) tr ( A ˉ V β ) ≤ β σ 2 , the prefactor being positive by (A45). Substituting this bound back into R β = β A max 2 2 tr ( A ˉ V β ) R_\beta=\frac{\beta A_{\max}^2}{2}\operatorname{tr}(\bar{A}V_\beta) R β = 2 β A m a x 2 tr ( A ˉ V β ) ’s upper estimate yields the stated bound on R β R_\beta R β , and λ min ( A ˉ ) tr V β ≤ tr ( A ˉ V β ) \lambda_{\min}(\bar{A})\operatorname{tr}V_\beta\le\operatorname{tr}(\bar{A}V_\beta) λ m i n ( A ˉ ) tr V β ≤ tr ( A ˉ V β ) yields the second part of (A104).
Deriving the clean bound. S = 0 S=0 S = 0 is feasible, so E [ ρ 2 ] ≤ E [ c 2 ] \mathbb{E}[\rho^2]\le\mathbb{E}[c^2] E [ ρ 2 ] ≤ E [ c 2 ] ; moreover σ 2 = E [ ρ 2 ∥ a ∥ 2 ] ≤ A max 2 E [ ρ 2 ] \sigma^2=\mathbb{E}[\rho^2\lVert a\rVert^2]\le A_{\max}^2\mathbb{E}[\rho^2] σ 2 = E [ ρ 2 ∥ a ∥ 2 ] ≤ A m a x 2 E [ ρ 2 ] , hence β σ 2 ≤ κ E [ ρ 2 ] \beta\sigma^2\le\kappa\mathbb{E}[\rho^2] β σ 2 ≤ κ E [ ρ 2 ] and
tr V β ≤ κ 2 − κ ⋅ E [ ρ 2 ] λ min ( A ˉ ) . (119) \operatorname{tr}V_\beta\le\frac{\kappa}{2-\kappa}\cdot\frac{\mathbb{E}[\rho^2]}{\lambda_{\min}(\bar{A})}. \tag*{(119)} tr V β ≤ 2 − κ κ ⋅ λ m i n ( A ˉ ) E [ ρ 2 ] . (119)
Adding this to (A103) gives
E π β [ ∥ S ∥ F 2 ] ≤ 1 λ min ( A ˉ ) ( E [ c 2 ] − E [ ρ 2 ] + κ 2 − κ E [ ρ 2 ] ) , (120) \mathbb{E}_{\pi_\beta}\left[\lVert S\rVert_{\mathrm{F}}^2\right]\le\frac{1}{\lambda_{\min}(\bar{A})}\left(\mathbb{E}[c^2]-\mathbb{E}[\rho^2]+\frac{\kappa}{2-\kappa}\mathbb{E}[\rho^2]\right), \tag*{(120)} E π β [ ∥ S ∥ F 2 ] ≤ λ m i n ( A ˉ ) 1 ( E [ c 2 ] − E [ ρ 2 ] + 2 − κ κ E [ ρ 2 ] ) , (120)
and − 1 + κ / ( 2 − κ ) = − 2 ( 1 − κ ) / ( 2 − κ ) -1+\kappa/(2-\kappa)=-2(1-\kappa)/(2-\kappa) − 1 + κ / ( 2 − κ ) = − 2 ( 1 − κ ) / ( 2 − κ ) is the coefficient in (A106). For κ ≤ 1 \kappa\le1 κ ≤ 1 that coefficient is ≤ 0 \le0 ≤ 0 , so dropping the E [ ρ 2 ] \mathbb{E}[\rho^2] E [ ρ 2 ] term gives (A107); for κ > 1 \kappa>1 κ > 1 , using E [ ρ 2 ] ≤ E [ c 2 ] \mathbb{E}[\rho^2]\le\mathbb{E}[c^2] E [ ρ 2 ] ≤ E [ c 2 ] gives the factor 1 + 2 ( κ − 1 ) / ( 2 − κ ) = κ / ( 2 − κ ) 1+2(\kappa-1)/(2-\kappa)=\kappa/(2-\kappa) 1 + 2 ( κ − 1 ) / ( 2 − κ ) = κ / ( 2 − κ ) .
Residuals lower the bound. The E [ ρ 2 ] \mathbb{E}[\rho^2] E [ ρ 2 ] term in (A106) enters with a negative sign throughout the non-expansive regime κ ≤ 1 \kappa\le1 κ ≤ 1 : overload shrinks the bound but does not inflate it, because residual energy is energy the bilinear features cannot explain and therefore cannot store. The two effects of ρ \rho ρ are less memory content (A103) and more steady-state jitter (A104). They cancel exactly at κ = 1 \kappa=1 κ = 1 and the content term dominates below it. This is why the constant in (A107) is 1 rather than the 3/2 or 2 obtained by bounding the two contributions separately.
Bound on the memory norm for all updates. Next, we consider the bound even before the memory S t S_t S t reaches the steady state but is in the update phase.
Theorem A10 (The bound applies to the running or online memory at every step: not only in the steady state). Let μ t : = law ( s t ) = μ 0 P t ∈ P 2 \mu_t := \operatorname{law}(s_t) = \mu_0 P^t \in\mathcal{P}_2 μ t := law ( s t ) = μ 0 P t ∈ P 2 be the law of the iterate at step t t t under (A16), so that π β \pi_\beta π β is the unique fixed point of the same kernel, and set
q β : = 1 − β ( 2 − κ ) λ min ( A ˉ ) ∈ [ 0 , 1 ) . (121) q_\beta:= 1 - \beta(2-\kappa)\lambda_{\min}(\bar{A}) \in[0,1). \tag*{(121)} q β := 1 − β ( 2 − κ ) λ m i n ( A ˉ ) ∈ [ 0 , 1 ) . (121) Started at S 0 = 0 S_0 = 0 S 0 = 0 , the running memory obeys, for every t t t and in the non-expansive regime κ ≤ 1 \kappa\le1 κ ≤ 1 ,
E [ ∥ S t ∥ F 2 ] ≤ ( 1 + q β t / 2 ) 2 E [ c 2 ] λ min ( A ˉ ) ≤ 4 E [ c 2 ] λ min ( A ˉ ) , (122) \mathbb{E}\left[\lVert S_t\rVert_{\mathrm{F}}^2\right] \le\left(1+q_\beta^{t/2}\right)^2\frac{\mathbb{E}[c^2]}{\lambda_{\min}(\bar{A})} \le\frac{4\mathbb{E}[c^2]}{\lambda_{\min}(\bar{A})}, \tag*{(122)} E [ ∥ S t ∥ F 2 ] ≤ ( 1 + q β t /2 ) 2 λ m i n ( A ˉ ) E [ c 2 ] ≤ λ m i n ( A ˉ ) 4 E [ c 2 ] , (122) where the transient factor expands as ( 1 + q β t / 2 ) 2 = 1 + 2 q β t / 2 + q β t ↓ 1 \left(1+q_\beta^{t/2}\right)^2 = 1+2q_\beta^{t/2}+q_\beta^t \downarrow1 ( 1 + q β t /2 ) 2 = 1 + 2 q β t /2 + q β t ↓ 1 , so the bound decays geometrically from 4 to the steady-state constant 1 of (A107). For a general step size κ ∈ ( 0 , 2 ) \kappa\in(0,2) κ ∈ ( 0 , 2 ) the same statement holds with E [ c 2 ] \mathbb{E}[c^2] E [ c 2 ] replaced by E [ c 2 ] max { 1 , κ / ( 2 − κ ) } \mathbb{E}[c^2]\max\{1,\kappa/(2-\kappa)\} E [ c 2 ] max { 1 , κ / ( 2 − κ )} ; for a general deterministic start S 0 S_0 S 0 it reads
E [ ∥ S t ∥ F 2 ] ≤ ( B β + q β t / 2 ∥ S 0 − S L S ∥ F 2 + tr V β ) 2 , B β : = E π β [ ∥ S ∥ F 2 ] 1 / 2 . (123) \mathbb{E}\left[\lVert S_t\rVert_{\mathrm{F}}^2\right] \le\left(B_\beta+ q_\beta^{t/2}\sqrt{\lVert S_0-S^{\mathrm{LS}}\rVert_{\mathrm{F}}^2+\operatorname{tr}V_\beta}\right)^2,\qquad B_\beta:= \mathbb{E}_{\pi_\beta}\left[\lVert S\rVert_{\mathrm{F}}^2\right]^{1/2}. \tag*{(123)} E [ ∥ S t ∥ F 2 ] ≤ ( B β + q β t /2 ∥ S 0 − S LS ∥ F 2 + tr V β ) 2 , B β := E π β [ ∥ S ∥ F 2 ] 1/2 . (123) Nothing on the right-hand side depends on the sequence length: the expected squared memory norm never exceeds four times its steady-state bound from the very first step, and relaxes geometrically to that bound.
Proof. Rate. Couple two copies of (A16) synchronously, i.e. drive them with the same ( a t , c t ) (a_t,c_t) ( a t , c t ) . The additive term cancels in the difference d t d_t d t , leaving d t + 1 = ( I − β a t a t T ) d t d_{t+1}=(I-\beta a_ta_t^\mathsf{T})d_t d t + 1 = ( I − β a t a t T ) d t , and
E [ ( I − β a a T ) 2 ] = I − 2 β A ˉ + β 2 E [ ∥ a ∥ 2 a a T ] ⪯ I − β ( 2 − κ ) A ˉ , (124) \mathbb{E}\left[(I-\beta aa^\mathsf{T})^2\right] = I-2\beta\bar{A}+\beta^2\mathbb{E}[\lVert a\rVert^2aa^\mathsf{T}] \preceq I-\beta(2-\kappa)\bar{A}, \tag*{(124)} E [ ( I − β a a T ) 2 ] = I − 2 β A ˉ + β 2 E [∥ a ∥ 2 a a T ] ⪯ I − β ( 2 − κ ) A ˉ , (124)
using β 2 E [ ∥ a ∥ 2 a a T ] ⪯ β κ A ˉ \beta^2\mathbb{E}[\lVert a\rVert^2aa^\mathsf{T}]\preceq\beta\kappa\bar{A} β 2 E [∥ a ∥ 2 a a T ] ⪯ β κ A ˉ . Hence E ∥ d t + 1 ∥ 2 ≤ q β E ∥ d t ∥ 2 \mathbb{E}\lVert d_{t+1}\rVert^2\le q_\beta\mathbb{E}\lVert d_t\rVert^2 E ∥ d t + 1 ∥ 2 ≤ q β E ∥ d t ∥ 2 , and taking the infimum over couplings gives W 2 ( μ t , π β ) ≤ q β t / 2 W 2 ( μ 0 , π β ) W_2(\mu_t,\pi_\beta)\le q_\beta^{t/2}W_2(\mu_0,\pi_\beta) W 2 ( μ t , π β ) ≤ q β t /2 W 2 ( μ 0 , π β ) . (Equivalently q β = 1 − ρ ˉ q_\beta=1-\bar{\rho} q β = 1 − ρ ˉ with the ρ ˉ = γ β λ min ( A ˉ ) \bar{\rho}=\gamma_\beta\lambda_{\min}(\bar{A}) ρ ˉ = γ β λ m i n ( A ˉ ) of Lemma A1, since γ β ≥ β ( 2 − κ ) \gamma_\beta\ge\beta(2-\kappa) γ β ≥ β ( 2 − κ ) ; q β q_\beta q β is written out here only to keep the statement self-contained.) Since s ↦ ∥ s ∥ s\mapsto\lVert s\rVert s ↦ ∥ s ∥ is 1-Lipschitz, the reverse triangle inequality in L 2 L^2 L 2 gives E [ ∥ S t ∥ F 2 ] 1 / 2 ≤ B β + W 2 ( μ t , π β ) \mathbb{E}[\lVert S_t\rVert_{\mathrm{F}}^2]^{1/2}\le B_\beta+W_2(\mu_t,\pi_\beta) E [∥ S t ∥ F 2 ] 1/2 ≤ B β + W 2 ( μ t , π β ) .
The initial term is not an estimate. For a deterministic start the only coupling of μ 0 = δ S 0 \mu_0=\delta_{S_0} μ 0 = δ S 0 with π β \pi_\beta π β is the product, so the infimum defining W 2 W_2 W 2 is attained and
W 2 ( μ 0 , π β ) 2 = E π β [ ∥ S − S 0 ∥ F 2 ] = ∥ S 0 − S L S ∥ F 2 + tr V β , (125) W_2(\mu_0,\pi_\beta)^2=\mathbb{E}_{\pi_\beta}\left[\lVert S-S_0\rVert_{\mathrm{F}}^2\right]=\lVert S_0-S^{\mathrm{LS}}\rVert_{\mathrm{F}}^2+\operatorname{tr}V_\beta, \tag*{(125)} W 2 ( μ 0 , π β ) 2 = E π β [ ∥ S − S 0 ∥ F 2 ] = ∥ S 0 − S LS ∥ F 2 + tr V β , (125)
the second equality being the bias–variance split around the stationary mean S L S S^{\mathrm{LS}} S LS (Theorem A7 (i)). At S 0 = 0 S_0=0 S 0 = 0 the right-hand side is exactly ∥ S L S ∥ F 2 + tr V β = B β 2 \lVert S^{\mathrm{LS}}\rVert_{\mathrm{F}}^2+\operatorname{tr}V_\beta=B_\beta^2 ∥ S LS ∥ F 2 + tr V β = B β 2 by (A104), i.e. W 2 ( μ 0 , π β ) = B β W_2(\mu_0,\pi_\beta)=B_\beta W 2 ( μ 0 , π β ) = B β : the distance from the empty memory to the steady state is the steady-state size. The two terms therefore collapse into one,
E [ ∥ S t ∥ F 2 ] 1 / 2 ≤ ( 1 + q β t / 2 ) B β , (126) \mathbb{E}\left[\lVert S_t\rVert_{\mathrm{F}}^2\right]^{1/2}\le(1+q_\beta^{t/2})B_\beta, \tag*{(126)} E [ ∥ S t ∥ F 2 ] 1/2 ≤ ( 1 + q β t /2 ) B β , (126)
and squaring and inserting B β 2 ≤ E [ c 2 ] / λ min ( A ˉ ) B_\beta^2\le\mathbb{E}[c^2]/\lambda_{\min}(\bar{A}) B β 2 ≤ E [ c 2 ] / λ m i n ( A ˉ ) from (A107) gives (A112). Squaring preserves the inequality since both sides are nonnegative. Finally q β t / 2 ≤ 1 q_\beta^{t/2}\le1 q β t /2 ≤ 1 bounds the transient factor by 4. The general-κ \kappa κ variant replaces B β 2 B_\beta^2 B β 2 by the bound of (A106), and the general-S 0 S_0 S 0 variant (A113) keeps W 2 ( μ 0 , π β ) W_2(\mu_0,\pi_\beta) W 2 ( μ 0 , π β ) in the form (A115) instead of collapsing it.
Summary. The least-squares memory satisfies the exact energy identity ( S L S ) T A ˉ S L S = E [ c 2 ] − E [ ρ 2 ] (S^{\mathrm{LS}})^\mathsf{T}\bar{A}S^{\mathrm{LS}}=\mathbb{E}[c^2]-\mathbb{E}[\rho^2] ( S LS ) T A ˉ S LS = E [ c 2 ] − E [ ρ 2 ] , which sandwiches its Frobenius norm between ( E [ c 2 ] − E [ ρ 2 ] ) / λ max (\mathbb{E}[c^2]-\mathbb{E}[\rho^2])/\lambda_{\max} ( E [ c 2 ] − E [ ρ 2 ]) / λ m a x and ( E [ c 2 ] − E [ ρ 2 ] ) / λ min (\mathbb{E}[c^2]-\mathbb{E}[\rho^2])/\lambda_{\min} ( E [ c 2 ] − E [ ρ 2 ]) / λ m i n . This is the explained target energy divided by the extreme feature-covariance eigenvalues. The bound is an equality for isotropic designs, reproduces ∑ i c i 2 \sum_i c_i^2 ∑ i c i 2 in the orthonormal design, extends to the running iterates with an additive O ( β ) O(\beta) O ( β ) fluctuation term, shrinks under overload since the residual energy is subtracted.
DeltaNet Steady-State Norm Bound
Overview
The DeltaNet steady-state norm bound can be derived essentially verbatim to kronLSTM. DeltaNet’s update is affine in S t S_t S t and acts on each row of the memory independently; each row obeys exactly the scalar delta-rule recursion analyzed for KronLSTM, with the association direction a t = k t ⊗ v t a_t = k_t \otimes v_t a t = k t ⊗ v t replaced by the key k t k_t k t and the scalar target c t c_t c t replaced by the value coordinate v t , j v_{t,j} v t , j . Every step of the argument — the energy identity, the two-sided bound, the exact stationary covariance bound, and the Wasserstein contraction that carries all of it to the running memory — therefore transfers, and yields E π β [ ∥ S ∥ F 2 ] ≤ 2 E [ ∥ v ∥ 2 ] / λ min ( K ˉ ) \mathbb{E}_{\pi_\beta}[\lVert S\rVert_F^2] \le2\mathbb{E}[\lVert v\rVert^2]/\lambda_{\min}(\bar{K}) E π β [∥ S ∥ F 2 ] ≤ 2 E [∥ v ∥ 2 ] / λ m i n ( K ˉ ) with K ˉ = E [ k k ⊤ ] \bar{K} = \mathbb{E}[kk^\top] K ˉ = E [ k k ⊤ ] .
The practical consequence is a scoping correction: a length-independent steady-state bound is a property of the delta-rule mechanism , shared by KronLSTM and DeltaNet, which separates both from mLSTM and from any stability argument resting on a forget gate. We also record where the two genuinely differ: in the step-size condition, in what the bound is a bound on , and in the conditioning constant that sets its scale.
DeltaNet Affine Recursion
DeltaNet’s update is
S t + 1 = S t − β t ( S t k t − v t ) k t ⊤ = S t ( I − β t k t k t ⊤ ) + β t v t k t ⊤ , (127) S_{t+1}=S_t-\beta_t(S_tk_t-v_t)k_t^\top=S_t(I-\beta_tk_tk_t^\top)+\beta_tv_tk_t^\top, \tag*{(127)} S t + 1 = S t − β t ( S t k t − v t ) k t ⊤ = S t ( I − β t k t k t ⊤ ) + β t v t k t ⊤ , (127)
and Gated DeltaNet’s is the same with S t S_t S t replaced by f t S t f_tS_t f t S t in the first term. Two features are immediate. First, (A117) is affine in S t S_t S t , so the pair (memory, random input) generates a Markov chain of exactly the type analyzed for the Kronecker Delta Rule. Second, and decisively, the memory is acted on from the right only, so the rows of S S S do not mix. Transposing (A117),
S t + 1 ⊤ = ( I − β t k t k t ⊤ ) S t ⊤ + β t k t v t ⊤ , (128) S_{t+1}^{\top}=(I-\beta_tk_tk_t^\top)S_t^\top+\beta_tk_tv_t^\top, \tag*{(128)} S t + 1 ⊤ = ( I − β t k t k t ⊤ ) S t ⊤ + β t k t v t ⊤ , (128)
and reading off the j j j -th column, i.e. the j j j -th row s ( j ) : = S ⊤ e j s^{(j)}:=S^\top e_j s ( j ) := S ⊤ e j of S S S ,
s t + 1 ( j ) = ( I − β t k t k t ⊤ ) s t ( j ) + β t v t , j k t , j = 1 , … , d . (129) s_{t+1}^{(j)}=(I-\beta_tk_tk_t^\top)s_t^{(j)}+\beta_tv_{t,j}k_t,\qquad j=1,\ldots,d. \tag*{(129)} s t + 1 ( j ) = ( I − β t k t k t ⊤ ) s t ( j ) + β t v t , j k t , j = 1 , … , d . (129)
The structural correspondence. Equation (A119) is identical to the vectorized KronLSTM recursion s t + 1 = ( I − β t a t a t ⊤ ) s t + β t c t a t s_{t+1}=(I-\beta_ta_ta_t^\top)s_t+\beta_tc_ta_t s t + 1 = ( I − β t a t a t ⊤ ) s t + β t c t a t under the substitution
a t = k t ⊗ v t ⟼ k t , c t ⟼ v t , j , R d 2 ⟼ R d . (130) a_t=k_t\otimes v_t\longmapsto k_t,\qquad c_t\longmapsto v_{t,j},\qquad\mathbb{R}^{d^2}\longmapsto\mathbb{R}^d. \tag*{(130)} a t = k t ⊗ v t ⟼ k t , c t ⟼ v t , j , R d 2 ⟼ R d . (130)
DeltaNet is thus d d d independent copies of the same scalar delta rule all driven by a common key k t k_t k t and differing only in their scalar targets v t , j v_{t,j} v t , j . KronLSTM is a single copy of that rule on a d 2 d^2 d 2 -dimensional state with the richer direction k t ⊗ v t k_t\otimes v_t k t ⊗ v t . Neither is a generalization of the other; both are Kaczmarz/LMS iterations, which is why the same theory covers both.
Steady-State Norm Bound
Model the stream as i.i.d. triples ( k t , v t , β t ) ∼ p (k_t,v_t,\beta_t)\sim p ( k t , v t , β t ) ∼ p and put
K ˉ : = E [ k k ⊤ ] ≻ 0 , S L S : = E [ v k ⊤ ] K ˉ − 1 = argmin S E [ ∥ S k − v ∥ 2 ] , (131) \bar{K}:=\mathbb{E}[kk^\top]\succ0,\qquad S^{\mathrm{LS}}:=\mathbb{E}[vk^\top]\bar{K}^{-1}=\underset{S}{\operatorname{argmin}}\ \mathbb{E}[\lVert Sk-v\rVert^2], \tag*{(131)} K ˉ := E [ k k ⊤ ] ≻ 0 , S LS := E [ v k ⊤ ] K ˉ − 1 = S argmin E [∥ S k − v ∥ 2 ] , (131)
ρ : = v − S L S k , \rho:=v-S^{\mathrm{LS}}k, ρ := v − S LS k ,
so that E [ ρ k ⊤ ] = 0 \mathbb{E}[\rho k^\top]=0 E [ ρ k ⊤ ] = 0 . With a random, input-dependent β t \beta_t β t the same statements hold with K ˉ \bar{K} K ˉ replaced by the β \beta β -weighted second moment E [ β k k ⊤ ] \mathbb{E}[\beta kk^\top] E [ β k k ⊤ ] and S L S S^{\mathrm{LS}} S LS by the correspondingly weighted least-squares solution; nothing else changes. Assume the step size satisfies β t ∥ k t ∥ 2 ≤ 1 \beta_t\lVert k_t\rVert^2\le1 β t ∥ k t ∥ 2 ≤ 1 , which for Gated DeltaNet’s sigmoid-bounded β t ∈ ( 0 , 1 ) \beta_t\in(0,1) β t ∈ ( 0 , 1 ) and unit-norm keys holds by construction.
Proposition A1 (Two-sided steady-state norm bound for DeltaNet). Let π β \pi_\beta π β be the steady-state law of (A117) and Σ β \Sigma_\beta Σ β the covariance of vec ( S ) \operatorname{vec}(S) vec ( S ) under it. Then the energy identity
Tr ( S L S K ˉ S L S ⊤ ) = E [ ∥ v ∥ 2 ] − E [ ∥ ρ ∥ 2 ] (132) \operatorname{Tr}(S^{\mathrm{LS}}\bar{K}S^{\mathrm{LS}\top})=\mathbb{E}[\lVert v\rVert^2]-\mathbb{E}[\lVert\rho\rVert^2] \tag*{(132)} Tr ( S LS K ˉ S LS ⊤ ) = E [∥ v ∥ 2 ] − E [∥ ρ ∥ 2 ] (132)
holds, hence
E [ ∥ v ∥ 2 ] − E [ ∥ ρ ∥ 2 ] λ max ( K ˉ ) ≤ ∥ S L S ∥ F 2 ≤ E [ ∥ v ∥ 2 ] − E [ ∥ ρ ∥ 2 ] λ min ( K ˉ ) , (133) \frac{\mathbb{E}\left[\lVert v\rVert^2\right]-\mathbb{E}\left[\lVert\rho\rVert^2\right]}{\lambda_{\max}(\bar{K})}
\le\lVert S^{\mathrm{LS}}\rVert_{\mathrm{F}}^2
\le\frac{\mathbb{E}\left[\lVert v\rVert^2\right]-\mathbb{E}\left[\lVert\rho\rVert^2\right]}{\lambda_{\min}(\bar{K})}, \tag*{(133)} λ m a x ( K ˉ ) E [ ∥ v ∥ 2 ] − E [ ∥ ρ ∥ 2 ] ≤ ∥ S LS ∥ F 2 ≤ λ m i n ( K ˉ ) E [ ∥ v ∥ 2 ] − E [ ∥ ρ ∥ 2 ] , (133)
and the steady-state memory satisfies, exactly and without any expansion in β \beta β ,
E π β [ ∥ S ∥ F 2 ] = ∥ S L S ∥ F 2 + Tr Σ β , Tr Σ β ≤ β E [ ∥ ρ ∥ 2 ∥ k ∥ 2 ] λ min ( K ˉ ) , (134) \mathbb{E}_{\pi_\beta}\left[\lVert S\rVert_{\mathrm{F}}^2\right]
=
\lVert S^{\mathrm{LS}}\rVert_{\mathrm{F}}^2+\operatorname{Tr}\Sigma_\beta,
\qquad
\operatorname{Tr}\Sigma_\beta
\le
\frac{\beta\mathbb{E}\left[\lVert\rho\rVert^2\lVert k\rVert^2\right]}{\lambda_{\min}(\bar{K})}, \tag*{(134)} E π β [ ∥ S ∥ F 2 ] = ∥ S LS ∥ F 2 + Tr Σ β , Tr Σ β ≤ λ m i n ( K ˉ ) β E [ ∥ ρ ∥ 2 ∥ k ∥ 2 ] , (134)
so that in particular
E π β [ ∥ S ∥ F 2 ] ≤ 2 E [ ∥ v ∥ 2 ] λ min ( K ˉ ) . (135) \mathbb{E}_{\pi_\beta}\left[\lVert S\rVert_{\mathrm{F}}^2\right]
\le
\frac{2\mathbb{E}\left[\lVert v\rVert^2\right]}{\lambda_{\min}(\bar{K})}. \tag*{(135)} E π β [ ∥ S ∥ F 2 ] ≤ λ m i n ( K ˉ ) 2 E [ ∥ v ∥ 2 ] . (135)
Proof. Apply the KronLSTM argument to (A119) for each j j j and sum. The normal equations K ˉ s ( j ) , L S = E [ v j k ] \bar{K}s^{(j),\mathrm{LS}}=\mathbb{E}[v_jk] K ˉ s ( j ) , LS = E [ v j k ] give E [ ρ j 2 ] = E [ v j 2 ] − s ( j ) , L S T K ˉ s ( j ) , L S \mathbb{E}[\rho_j^2]=\mathbb{E}[v_j^2]-s^{(j),\mathrm{LS}\mathsf{T}}\bar{K}s^{(j),\mathrm{LS}} E [ ρ j 2 ] = E [ v j 2 ] − s ( j ) , LS T K ˉ s ( j ) , LS ; summing over j j j and using ∑ j s ( j ) T K ˉ s ( j ) = Tr ( S K ˉ S T ) \sum_j s^{(j)\mathsf{T}}\bar{K}s^{(j)}=\operatorname{Tr}(S\bar{K}S^{\mathsf{T}}) ∑ j s ( j ) T K ˉ s ( j ) = Tr ( S K ˉ S T ) yields (A122). Bounding each summand’s Rayleigh quotient between λ min ( K ˉ ) ∥ s ( j ) ∥ 2 \lambda_{\min}(\bar{K})\lVert s^{(j)}\rVert^2 λ m i n ( K ˉ ) ∥ s ( j ) ∥ 2 and λ max ( K ˉ ) ∥ s ( j ) ∥ 2 \lambda_{\max}(\bar{K})\lVert s^{(j)}\rVert^2 λ m a x ( K ˉ ) ∥ s ( j ) ∥ 2 and summing, together with ∑ j ∥ s ( j ) ∥ 2 = ∥ S ∥ F 2 \sum_j\lVert s^{(j)}\rVert^2=\lVert S\rVert_{\mathrm{F}}^2 ∑ j ∥ s ( j ) ∥ 2 = ∥ S ∥ F 2 , gives (A123). For (A124), write E : = S − S L S E:=S-S^{\mathrm{LS}} E := S − S LS , so that (A117) becomes E t + 1 = E t ( I − β t k t k t T ) + β t ρ t k t T E_{t+1}=E_t(I-\beta_tk_tk_t^{\mathsf{T}})+\beta_t\rho_tk_t^{\mathsf{T}} E t + 1 = E t ( I − β t k t k t T ) + β t ρ t k t T with ( k t , ρ t ) (k_t,\rho_t) ( k t , ρ t ) independent of E t E_t E t . Taking squared Frobenius norms in stationarity, the cross term vanishes because E [ E t ] = 0 \mathbb{E}[E_t]=0 E [ E t ] = 0 , leaving
E [ ∥ E ∥ F 2 ] = E [ Tr ( E E [ ( I − β k k T ) 2 ] E T ) ] + β 2 E [ ∥ ρ ∥ 2 ∥ k ∥ 2 ] . (136) \mathbb{E}\left[\lVert E\rVert_{\mathrm{F}}^2\right]
=
\mathbb{E}\left[\operatorname{Tr}\left(E\mathbb{E}\left[(I-\beta kk^{\mathsf{T}})^2\right]E^{\mathsf{T}}\right)\right]
+\beta^2\mathbb{E}\left[\lVert\rho\rVert^2\lVert k\rVert^2\right]. \tag*{(136)} E [ ∥ E ∥ F 2 ] = E [ Tr ( E E [ ( I − β k k T ) 2 ] E T ) ] + β 2 E [ ∥ ρ ∥ 2 ∥ k ∥ 2 ] . (136)
Since β ∥ k ∥ 2 ≤ 1 \beta\lVert k\rVert^2\le1 β ∥ k ∥ 2 ≤ 1 ,
E [ ( I − β k k T ) 2 ] = I − 2 β K ˉ + β 2 E [ ∥ k ∥ 2 k k T ] ⪯ I − β K ˉ , (137) \mathbb{E}\left[(I-\beta kk^{\mathsf{T}})^2\right]
=
I-2\beta\bar{K}+\beta^2\mathbb{E}\left[\lVert k\rVert^2kk^{\mathsf{T}}\right]
\preceq I-\beta\bar{K}, \tag*{(137)} E [ ( I − β k k T ) 2 ] = I − 2 β K ˉ + β 2 E [ ∥ k ∥ 2 k k T ] ⪯ I − β K ˉ , (137)
and cancelling E [ ∥ E ∥ F 2 ] \mathbb{E}\left[\lVert E\rVert_{\mathrm{F}}^2\right] E [ ∥ E ∥ F 2 ] from both sides gives E [ Tr ( E K ˉ E T ) ] ≤ β E [ ∥ ρ ∥ 2 ∥ k ∥ 2 ] \mathbb{E}[\operatorname{Tr}(E\bar{K}E^{\mathsf{T}})]\le\beta\mathbb{E}[\lVert\rho\rVert^2\lVert k\rVert^2] E [ Tr ( E K ˉ E T )] ≤ β E [∥ ρ ∥ 2 ∥ k ∥ 2 ] ; the claim follows from λ min ( K ˉ ) E [ ∥ E ∥ F 2 ] ≤ E [ Tr ( E K ˉ E T ) ] \lambda_{\min}(\bar{K})\mathbb{E}[\lVert E\rVert_{\mathrm{F}}^2]\le\mathbb{E}[\operatorname{Tr}(E\bar{K}E^{\mathsf{T}})] λ m i n ( K ˉ ) E [∥ E ∥ F 2 ] ≤ E [ Tr ( E K ˉ E T )] and Tr Σ β = E [ ∥ E ∥ F 2 ] \operatorname{Tr}\Sigma_\beta=\mathbb{E}[\lVert E\rVert_{\mathrm{F}}^2] Tr Σ β = E [∥ E ∥ F 2 ] . Finally S = 0 S=0 S = 0 is feasible, so E [ ∥ ρ ∥ 2 ] ≤ E [ ∥ v ∥ 2 ] \mathbb{E}[\lVert\rho\rVert^2]\le\mathbb{E}[\lVert v\rVert^2] E [∥ ρ ∥ 2 ] ≤ E [∥ v ∥ 2 ] , which bounds both terms of (A124) by E [ ∥ v ∥ 2 ] / λ min ( K ˉ ) \mathbb{E}[\lVert v\rVert^2]/\lambda_{\min}(\bar{K}) E [∥ v ∥ 2 ] / λ m i n ( K ˉ ) and gives (A125).
Corollary A5 (Transfer to the running or online memory). With μ t \mu_t μ t the law of S t S_t S t and B β : = E π β [ ∥ S ∥ F 2 ] 1 / 2 B_\beta:=\mathbb{E}_{\pi_\beta}[\lVert S\rVert_{\mathrm{F}}^2]^{1/2} B β := E π β [∥ S ∥ F 2 ] 1/2 ,
E [ ∥ S t ∥ F 2 ] 1 / 2 ≤ B β + ( 1 − β λ min ( K ˉ ) ) t / 2 W 2 ( μ 0 , π β ) , B β 2 ≤ 2 E [ ∥ v ∥ 2 ] λ min ( K ˉ ) . (138) \mathbb{E}\left[\lVert S_t\rVert_{\mathrm{F}}^2\right]^{1/2}
\le
B_\beta+(1-\beta\lambda_{\min}(\bar{K}))^{t/2}\mathcal{W}_2(\mu_0,\pi_\beta),
\qquad
B_\beta^2\le\frac{2\mathbb{E}[\lVert v\rVert^2]}{\lambda_{\min}(\bar{K})}. \tag*{(138)} E [ ∥ S t ∥ F 2 ] 1/2 ≤ B β + ( 1 − β λ m i n ( K ˉ ) ) t /2 W 2 ( μ 0 , π β ) , B β 2 ≤ λ m i n ( K ˉ ) 2 E [∥ v ∥ 2 ] . (138)
Proof. Synchronous coupling: driving two copies with the same ( k t , v t , β t ) (k_t,v_t,\beta_t) ( k t , v t , β t ) cancels the additive term, leaving D t + 1 = D t ( I − β t k t k t T ) D_{t+1}=D_t(I-\beta_tk_tk_t^{\mathsf{T}}) D t + 1 = D t ( I − β t k t k t T ) , whence E [ ∥ D t + 1 ∥ F 2 ] ≤ ( 1 − β λ min ( K ˉ ) ) E [ ∥ D t ∥ F 2 ] \mathbb{E}[\lVert D_{t+1}\rVert_{\mathrm{F}}^2]\le(1-\beta\lambda_{\min}(\bar{K}))\mathbb{E}[\lVert D_t\rVert_{\mathrm{F}}^2] E [∥ D t + 1 ∥ F 2 ] ≤ ( 1 − β λ m i n ( K ˉ )) E [∥ D t ∥ F 2 ] by the same positive-semidefinite inequality as above. Take the infimum over couplings and use that ∥ ⋅ ∥ F \lVert\cdot\rVert_{\mathrm{F}} ∥ ⋅ ∥ F is 1-Lipschitz.
Relation to the KronLSTM Bound
The bound is a consequence of the delta rule being a relaxed projection that is a correction proportional to a residual that shrinks as the memory improves. Both models have that structure, therefore separates the delta-rule family from mLSTM, whose Hebbian write is uncorrected and has no fixed point to contract toward. It separates the steady-state argument from the non-expansiveness argument, whose forget-gate bound β C A max / ( 1 − f ) \beta C A_{\max}/(1-f) β C A m a x / ( 1 − f ) is Ω ( T ) \Omega(T) Ω ( T ) once f → 1 f\to1 f → 1 as long contexts require.
The constants are not comparable. The two bounds read
KronLSTM: E [ c 2 ] λ min ( A ˉ ) , DeltaNet: 2 E [ ∥ v ∥ 2 ] λ min ( K ˉ ) , A ˉ = E [ a a T ] ∈ R d 2 × d 2 , K ˉ ∈ R d × d . (139) \text{KronLSTM:}\quad\frac{\mathbb{E}[c^2]}{\lambda_{\min}(\bar{A})},
\qquad
\text{DeltaNet:}\quad\frac{2\mathbb{E}[\lVert v\rVert^2]}{\lambda_{\min}(\bar{K})},
\qquad
\bar{A}=\mathbb{E}[aa^{\mathsf{T}}]\in\mathbb{R}^{d^2\times d^2},
\qquad
\bar{K}\in\mathbb{R}^{d\times d}. \tag*{(139)} KronLSTM: λ m i n ( A ˉ ) E [ c 2 ] , DeltaNet: λ m i n ( K ˉ ) 2 E [∥ v ∥ 2 ] , A ˉ = E [ a a T ] ∈ R d 2 × d 2 , K ˉ ∈ R d × d . (139)
These are bounds on the same object, ∥ S ∥ F \lVert S\rVert_{\mathrm{F}} ∥ S ∥ F , but they are not bounds for the same task: KronLSTM’s stream carries one scalar c t c_t c t per step, DeltaNet’s a d d d -vector v t v_t v t , so E [ c 2 ] \mathbb{E}[c^2] E [ c 2 ] and E [ ∥ v ∥ 2 ] \mathbb{E}[\lVert v\rVert^2] E [∥ v ∥ 2 ] measure different amounts of information being written. The denominators also differ structurally: K ˉ ≻ 0 \bar{K}\succ0 K ˉ ≻ 0 requires only that the keys span R d \mathbb{R}^d R d , whereas A ˉ ≻ 0 \bar{A}\succ0 A ˉ ≻ 0 requires the a i = k i ⊗ v i a_i=k_i\otimes v_i a i = k i ⊗ v i to span R d 2 \mathbb{R}^{d^2} R d 2 , and λ min ( A ˉ ) \lambda_{\min}(\bar{A}) λ m i n ( A ˉ ) is typically the smaller of the two. The reason is that for independent isotropic keys and values, A ˉ = E [ k k T ] ⊗ E [ v v T ] \bar{A}=\mathbb{E}[kk^{\mathsf{T}}]\otimes\mathbb{E}[vv^{\mathsf{T}}] A ˉ = E [ k k T ] ⊗ E [ v v T ] gives λ min ( A ˉ ) = λ min ( K ˉ ) λ min ( E [ v v T ] ) \lambda_{\min}(\bar{A})=\lambda_{\min}(\bar{K})\lambda_{\min}(\mathbb{E}[vv^{\mathsf{T}}]) λ m i n ( A ˉ ) = λ m i n ( K ˉ ) λ m i n ( E [ v v T ]) , a product of two small numbers. On the other hand the constant 2 appears in the bound of the DeltaNet. Which constant is smaller in a given setting is an empirical question, not a structural one.
DeltaNet has a smaller bound. The step-size condition is milder. KronLSTM requires β ∥ k t ∥ 2 ∥ v t ∥ 2 ≤ 1 \beta\lVert k_t\rVert^2 \lVert v_t\rVert^2 \le1 β ∥ k t ∥ 2 ∥ v t ∥ 2 ≤ 1 , so a large value norm eats directly into the admissible step size and hence into the contraction rate β λ min ( A ˉ ) \beta\lambda_{\min}(\bar{A}) β λ m i n ( A ˉ ) ; DeltaNet requires only β t ∥ k t ∥ 2 ≤ 1 \beta_t \lVert k_t\rVert^2 \le1 β t ∥ k t ∥ 2 ≤ 1 , in which v t v_t v t does not appear at all. It enters the drive but never the stability condition.
KronLSTM has advantages. d 2 d^2 d 2 rather than d d d exactly satisfiable associations per scalar slot, interference gated by the product of key and value overlap rather than key overlap alone, an update direction v t k t T v_t k_t^{\mathsf T} v t k t T fixed by the input pair rather than the state-dependent u t k t T u_t k_t^{\mathsf T} u t k t T so that a write’s footprint on unrelated memories does not compound with context length, single-binding write/rewrite/erase, and linearity of the memory in the target sequence. None of these follow from, or are affected by, the steady-state bound.
Retention Rates for KronLSTM and DeltaNet
KronLSTM Setting
Let n > d 2 n > d^2 n > d 2 associations ( k i , v i , c i ) (k_i, v_i, c_i) ( k i , v i , c i ) , 1 ≤ i ≤ n 1 \le i \le n 1 ≤ i ≤ n , be given. Vectorized, with s = vec ( S ) ∈ R d 2 s = \operatorname{vec}(S) \in\mathbb{R}^{d^2} s = vec ( S ) ∈ R d 2 and a i = k i ⊗ v i a_i = k_i \otimes v_i a i = k i ⊗ v i (∥ a i ∥ = ∥ k i ∥ ∥ v i ∥ \lVert a_i\rVert= \lVert k_i\rVert\lVert v_i\rVert ∥ a i ∥ = ∥ k i ∥ ∥ v i ∥ ), the associations form the linear system A s = c As = c A s = c with A ∈ R n × d 2 A \in\mathbb{R}^{n \times d^2} A ∈ R n × d 2 stacking the rows a i T a_i^{\mathsf T} a i T ; we assume A A A has full column rank d 2 d^2 d 2 (the features span R d 2 \mathbb{R}^{d^2} R d 2 ). At each step an index i t i_t i t is drawn with probabilities p i > 0 p_i > 0 p i > 0 , independently of the past, and the rule is applied with fixed step size β \beta β satisfying
0 < β ∥ a i ∥ 2 < 2 for all i . (140) 0 < \beta\lVert a_i\rVert^2 < 2 \qquad\text{for all } i . \tag*{(140)} 0 < β ∥ a i ∥ 2 < 2 for all i . (140)
Define
A ˉ : = E [ a a T ] = ∑ i = 1 n p i a i a i T ≻ 0 , b ˉ : = E [ c a ] = ∑ i = 1 n p i c i a i , (141) \bar{A} := \mathbb{E}[aa^{\mathsf T}] = \sum_{i=1}^{n} p_i a_i a_i^{\mathsf T} \succ0,\qquad
\bar{b} := \mathbb{E}[ca] = \sum_{i=1}^{n} p_i c_i a_i , \tag*{(141)} A ˉ := E [ a a T ] = i = 1 ∑ n p i a i a i T ≻ 0 , b ˉ := E [ c a ] = i = 1 ∑ n p i c i a i , (141)
s L S : = A ˉ − 1 b ˉ = argmin s ∑ i = 1 n p i ( a i T s − c i ) 2 , ρ i : = c i − a i T s L S (optimal residuals) , (142) s^{\mathrm{LS}} := \bar{A}^{-1}\bar{b} = \underset{s}{\operatorname{argmin}}\sum_{i=1}^{n} p_i(a_i^{\mathsf T}s-c_i)^2,\qquad
\rho_i := c_i-a_i^{\mathsf T}s^{\mathrm{LS}}\quad\text{(optimal residuals)} , \tag*{(142)} s LS := A ˉ − 1 b ˉ = s argmin i = 1 ∑ n p i ( a i T s − c i ) 2 , ρ i := c i − a i T s LS (optimal residuals) , (142)
so that the normal equations read
E [ ρ a ] = ∑ i = 1 n p i ρ i a i = 0 ⟺ b ˉ = A ˉ s L S . (143) \mathbb{E}[\rho a] = \sum_{i=1}^{n}p_i\rho_i a_i = 0
\qquad\Longleftrightarrow\qquad
\bar{b}=\bar{A}s^{\mathrm{LS}} . \tag*{(143)} E [ ρ a ] = i = 1 ∑ n p i ρ i a i = 0 ⟺ b ˉ = A ˉ s LS . (143)
For uniform sampling, A ˉ = 1 n A T A \bar{A}=\frac{1}{n}A^{\mathsf T}A A ˉ = n 1 A T A and s L S = ( A T A ) − 1 A T c s^{\mathrm{LS}}=(A^{\mathsf T}A)^{-1}A^{\mathsf T}c s LS = ( A T A ) − 1 A T c is the ordinary least-squares solution, i.e. the unique minimizer of L L L . Throughout, expectations without subscript are over i ∼ p i\sim p i ∼ p , and E π \mathbb{E}_{\pi} E π denotes expectation of a random state distributed according to a law π \pi π . Vectorized, the rule and its error form are
s t + 1 = M i t s t + β c i t a i t , e t + 1 = M i t e t + β ρ i t a i t , M i : = I − β a i a i T , (144) s_{t+1}=M_{i_t}s_t+\beta c_{i_t}a_{i_t},\qquad
e_{t+1}=M_{i_t}e_t+\beta\rho_{i_t}a_{i_t},\qquad
M_i:=I-\beta a_i a_i^{\mathsf T}, \tag*{(144)} s t + 1 = M i t s t + β c i t a i t , e t + 1 = M i t e t + β ρ i t a i t , M i := I − β a i a i T , (144)
with e t : = s t − s L S e_t:=s_t-s^{\mathrm{LS}} e t := s t − s LS ; the error form follows by inserting c i = a i T s L S + ρ i c_i=a_i^{\mathsf T}s^{\mathrm{LS}}+\rho_i c i = a i T s LS + ρ i . The constants
γ β : = min i β ( 2 − β ∥ a i ∥ 2 ) > 0 , ρ ˉ : = γ β λ min ( A ˉ ) ∈ ( 0 , 1 ] , σ 2 : = E [ β 2 ∥ a ∥ 2 ] (145) \gamma_\beta:=\min_i\beta(2-\beta\lVert a_i\rVert^2)>0,\qquad
\bar{\rho}:=\gamma_\beta\lambda_{\min}(\bar{A})\in(0,1],\qquad
\sigma^2:=\mathbb{E}[\beta^2\lVert a\rVert^2] \tag*{(145)} γ β := i min β ( 2 − β ∥ a i ∥ 2 ) > 0 , ρ ˉ := γ β λ m i n ( A ˉ ) ∈ ( 0 , 1 ] , σ 2 := E [ β 2 ∥ a ∥ 2 ] (145)
satisfy 0 < ρ ˉ ≤ 1 0<\bar{\rho}\le1 0 < ρ ˉ ≤ 1 . Under (A130), the expected iterate obeys the exact deterministic recursion
E [ s t + 1 ] = ( I − β A ˉ ) E [ s t ] + β b ˉ ⟺ E [ e t + 1 ] = ( I − β A ˉ ) E [ e t ] , (146) \mathbb{E}[s_{t+1}]=(I-\beta\bar{A})\mathbb{E}[s_t]+\beta\bar{b}
\qquad\Longleftrightarrow\qquad
\mathbb{E}[e_{t+1}]=(I-\beta\bar{A})\mathbb{E}[e_t], \tag*{(146)} E [ s t + 1 ] = ( I − β A ˉ ) E [ s t ] + β b ˉ ⟺ E [ e t + 1 ] = ( I − β A ˉ ) E [ e t ] , (146)
whose unique fixed point is s L S s^{\mathrm{LS}} s LS , and
∥ E [ S t ] − S L S ∥ F ≤ r t ∥ S 0 − S L S ∥ F , r : = max k ∣ 1 − β λ k ( A ˉ ) ∣ < 1 , (147) \lVert\mathbb{E}[S_t]-S^{\mathrm{LS}}\rVert_F
\le r^t\lVert S_0-S^{\mathrm{LS}}\rVert_F,\qquad
r:=\max_k\lvert1-\beta\lambda_k(\bar{A})\rvert<1, \tag*{(147)} ∥ E [ S t ] − S LS ∥ F ≤ r t ∥ S 0 − S LS ∥ F , r := k max ∣ 1 − β λ k ( A ˉ )∣ < 1 , (147)
so E [ S t ] → S L S \mathbb{E}[S_t]\to S^{\mathrm{LS}} E [ S t ] → S LS linearly. The convergence of the mean is exactly unbiased : the fixed point of (A136) is s L S s^{\mathrm{LS}} s LS itself, with no O ( β ) O(\beta) O ( β ) correction.
DeltaNet Setting
DeltaNet’s update is
S t + 1 = S t − β t ( S t k t − v t ) k t T = S t ( I − β t k t k t T ) + β t v t k t T , (148) S_{t+1}=S_t-\beta_t(S_tk_t-v_t)k_t^{\mathsf T}=S_t(I-\beta_tk_tk_t^{\mathsf T})+\beta_tv_tk_t^{\mathsf T}, \tag*{(148)} S t + 1 = S t − β t ( S t k t − v t ) k t T = S t ( I − β t k t k t T ) + β t v t k t T , (148)
and Gated DeltaNet’s is the same with S t S_t S t replaced by f t S t f_tS_t f t S t in the first term. Two features are immediate. First, (A138) is affine in S t S_t S t , so the pair (memory, random input) generates a Markov chain of exactly the type analyzed for the Kronecker Delta Rule. Second, and decisively, the memory is acted on from the right only, so the rows of S S S do not mix. Transposing (A138),
S t + 1 T = ( I − β t k t k t T ) S t T + β t k t v t T , (149) S_{t+1}^{\mathsf T}=(I-\beta_tk_tk_t^{\mathsf T})S_t^{\mathsf T}+\beta_tk_tv_t^{\mathsf T}, \tag*{(149)} S t + 1 T = ( I − β t k t k t T ) S t T + β t k t v t T , (149)
and reading off the j j j -th column, i.e. the j j j -th row s t ( j ) : = S t T e j s_t^{(j)}:=S_t^{\mathsf T}e_j s t ( j ) := S t T e j of S S S ,
s t + 1 ( j ) = ( I − β t k t k t T ) s t ( j ) + β t v t , j k t , j = 1 , … , d . (150) s_{t+1}^{(j)}=(I-\beta_tk_tk_t^{\mathsf T})s_t^{(j)}+\beta_tv_{t,j}k_t,\qquad j=1,\ldots,d. \tag*{(150)} s t + 1 ( j ) = ( I − β t k t k t T ) s t ( j ) + β t v t , j k t , j = 1 , … , d . (150)
We will use later the key covariance matrix K ˉ = E [ k k T ] \bar{K}=\mathbb{E}[kk^{\mathsf T}] K ˉ = E [ k k T ] and the value covariance matrix V ˉ = E [ v v T ] \bar{V}=\mathbb{E}[vv^{\mathsf T}] V ˉ = E [ v v T ] .
The structural correspondence. Equation (A140) is identical to the vectorized KronLSTM recursion s t + 1 = ( I − β a t a t T ) s t + β c t a t s_{t+1}=(I-\beta a_ta_t^{\mathsf T})s_t+\beta c_ta_t s t + 1 = ( I − β a t a t T ) s t + β c t a t under the substitution
a t = k t ⊗ v t ⟼ k t , c t ⟼ v t , j , R d 2 ⟼ R d . (151) a_t=k_t\otimes v_t\longmapsto k_t,\qquad c_t\longmapsto v_{t,j},\qquad\mathbb{R}^{d^2}\longmapsto\mathbb{R}^{d}. \tag*{(151)} a t = k t ⊗ v t ⟼ k t , c t ⟼ v t , j , R d 2 ⟼ R d . (151)
DeltaNet is thus d d d independent copies of the same scalar delta rule all driven by a common key k t k_t k t and differing only in their scalar targets v t , j v_{t,j} v t , j . KronLSTM is a single copy of that rule on a d 2 d^2 d 2 -dimensional state with the richer direction k t ⊗ v t k_t\otimes v_t k t ⊗ v t . Neither is a generalization of the other; both are Kaczmarz/LMS iterations, which is why the same theory covers both.
KronLSTM Retention of the Initial Memory
We consider KronLSTM and the Kronecker Delta Rule. We ask how much of an initial memory s 0 s_0 s 0 survives after t t t steps. The mean recursion (A136) is affine, so it separates the two contributions exactly. Writing M : = I − β A ˉ M:=I-\beta\bar{A} M := I − β A ˉ and unrolling,
E [ s t ] = M t s 0 + β ∑ τ = 0 t − 1 M τ b ˉ = M t s 0 ⏟ initial memory + ( I − M t ) s L S ⏟ acquired memory , (152) \mathbb{E}[s_t]=M^ts_0+\beta\sum_{\tau=0}^{t-1}M^\tau\bar{b}
=\underbrace{M^ts_0}_{\text{initial memory}}+\underbrace{(I-M^t)s^{\mathrm{LS}}}_{\text{acquired memory}}, \tag*{(152)} E [ s t ] = M t s 0 + β τ = 0 ∑ t − 1 M τ b ˉ = initial memory M t s 0 + acquired memory ( I − M t ) s LS , (152)
where the geometric sum was evaluated with β ∑ τ = 0 t − 1 M τ = ( I − M t ) ( I − M ) − 1 β = ( I − M t ) A ˉ − 1 \beta\sum_{\tau=0}^{t-1}M^\tau=(I-M^t)(I-M)^{-1}\beta=(I-M^t)\bar{A}^{-1} β ∑ τ = 0 t − 1 M τ = ( I − M t ) ( I − M ) − 1 β = ( I − M t ) A ˉ − 1 and A ˉ − 1 b ˉ = s L S \bar{A}^{-1}\bar{b}=s^{\mathrm{LS}} A ˉ − 1 b ˉ = s LS ; equivalently E [ s t ] − s L S = M t ( s 0 − s L S ) \mathbb{E}[s_t]-s^{\mathrm{LS}}=M^t(s_0-s^{\mathrm{LS}}) E [ s t ] − s LS = M t ( s 0 − s LS ) , which is the error recursion (A134) in the mean. Everything in E [ s t ] \mathbb{E}[s_t] E [ s t ] that depends on s 0 s_0 s 0 sits in the first term, and it depends on it linearly. The natural measure of retention is therefore the read-out of that term along the stored direction,
R t ( s 0 ) : = s 0 T ∂ E [ s t ] ∂ s 0 s 0 = s 0 T M t s 0 , ∥ s 0 ∥ = 1 , (153) R_t(s_0):=s_0^{\mathsf T}\frac{\partial\mathbb{E}[s_t]}{\partial s_0}s_0=s_0^{\mathsf T}M^ts_0,\qquad\lVert s_0\rVert=1, \tag*{(153)} R t ( s 0 ) := s 0 T ∂ s 0 ∂ E [ s t ] s 0 = s 0 T M t s 0 , ∥ s 0 ∥ = 1 , (153)
the fraction of a unit initial memory still legible after t t t steps. It equals 1 at t = 0 t=0 t = 0 and requires no assumption relating s 0 s_0 s 0 to s L S s^{\mathrm{LS}} s LS : the acquired term is filtered out by differentiation rather than by orthogonality.
Proposition A2 (Retention rate). Assume κ : = β A max 2 ≤ 1 \kappa:=\beta A_{\max}^2\leq1 κ := β A m a x 2 ≤ 1 with A max 2 = max i ∥ a i ∥ 2 A_{\max}^2=\max_i\lVert a_i\rVert^2 A m a x 2 = max i ∥ a i ∥ 2 , so that β λ max ( A ˉ ) ≤ 1 \beta\lambda_{\max}(\bar{A})\leq1 β λ m a x ( A ˉ ) ≤ 1 and M ⪰ 0 M\succeq0 M ⪰ 0 . Then for every unit s 0 s_0 s 0
( 1 − β λ max ( A ˉ ) ) t ≤ R t ( s 0 ) ≤ ( 1 − β λ min ( A ˉ ) ) t , (154) (1-\beta\lambda_{\max}(\bar{A}))^t\leq R_t(s_0)\leq(1-\beta\lambda_{\min}(\bar{A}))^t, \tag*{(154)} ( 1 − β λ m a x ( A ˉ ) ) t ≤ R t ( s 0 ) ≤ ( 1 − β λ m i n ( A ˉ ) ) t , (154) both bounds being attained, at the corresponding eigenvectors of A ˉ \bar{A} A ˉ . Averaged over a uniformly random unit s 0 s_0 s 0 the retention is exactly the normalized trace,
E s 0 [ R t ( s 0 ) ] = Tr M t d 2 = 1 d 2 ∑ k = 1 d 2 ( 1 − β λ k ( A ˉ ) ) t , (155) \mathbb{E}_{s_0}[R_t(s_0)]=\frac{\operatorname{Tr}M^t}{d^2}=\frac{1}{d^2}\sum_{k=1}^{d^2}(1-\beta\lambda_k(\bar{A}))^t, \tag*{(155)} E s 0 [ R t ( s 0 )] = d 2 Tr M t = d 2 1 k = 1 ∑ d 2 ( 1 − β λ k ( A ˉ ) ) t , (155) and the retention horizon — the number of steps after which the slowest mode has decayed to 1 / e 1/e 1/ e — is t 1 / e = 1 / log ( 1 − β λ min ( A ˉ ) ) − 1 ≃ 1 / ( β λ min ( A ˉ ) ) t_{1/e}=1/\log(1-\beta\lambda_{\min}(\bar{A}))^{-1}\simeq1/(\beta\lambda_{\min}(\bar{A})) t 1/ e = 1/ log ( 1 − β λ m i n ( A ˉ ) ) − 1 ≃ 1/ ( β λ m i n ( A ˉ )) .
Proof. A ˉ \bar{A} A ˉ is symmetric, so M = I − β A ˉ M=I-\beta\bar{A} M = I − β A ˉ is symmetric with eigenvalues 1 − β λ k ( A ˉ ) ∈ [ 0 , 1 ) 1-\beta\lambda_k(\bar{A})\in[0,1) 1 − β λ k ( A ˉ ) ∈ [ 0 , 1 ) under β λ max ≤ 1 \beta\lambda_{\max}\le1 β λ m a x ≤ 1 , hence M t M^t M t is symmetric positive semidefinite with eigenvalues ( 1 − β λ k ) t (1-\beta\lambda_k)^t ( 1 − β λ k ) t . Equation (A144) is the Rayleigh-quotient bound for M t M^t M t , with equality on its extreme eigenvectors, which are those of A ˉ \bar{A} A ˉ for λ max \lambda_{\max} λ m a x and λ min \lambda_{\min} λ m i n respectively. For (A145), E [ s 0 s 0 ⊤ ] = I / d 2 \mathbb{E}[s_0s_0^\top]=I/d^2 E [ s 0 s 0 ⊤ ] = I / d 2 for a uniform unit vector in R d 2 \mathbb{R}^{d^2} R d 2 , so E [ s 0 ⊤ M t s 0 ] = Tr ( M t E [ s 0 s 0 ⊤ ] ) = Tr ( M t ) / d 2 \mathbb{E}[s_0^\top M^t s_0]=\operatorname{Tr}(M^t\mathbb{E}[s_0s_0^\top])=\operatorname{Tr}(M^t)/d^2 E [ s 0 ⊤ M t s 0 ] = Tr ( M t E [ s 0 s 0 ⊤ ]) = Tr ( M t ) / d 2 . The step-size condition holds because λ max ( A ˉ ) ≤ max i ∥ a i ∥ 2 = A max 2 \lambda_{\max}(\bar{A})\le\max_i\lVert a_i\rVert^2=A_{\max}^2 λ m a x ( A ˉ ) ≤ max i ∥ a i ∥ 2 = A m a x 2 , A ˉ \bar{A} A ˉ being a convex combination of the a i a i ⊤ a_i a_i^\top a i a i ⊤ .
DeltaNet Retention of the Initial Memory
Corollary A6 (DeltaNet retention). By the row decomposition (A140), DeltaNet’s mean memory obeys E [ S t ] = S 0 N t + S L S ( I − N t ) \mathbb{E}[S_t]=S_0N^t+S^{\mathrm{LS}}(I-N^t) E [ S t ] = S 0 N t + S LS ( I − N t ) with N : = I − β K ˉ N:=I-\beta\bar{K} N := I − β K ˉ , so with R t ( S 0 ) : = ⟨ S 0 , S 0 N t ⟩ F R_t(S_0):=\langle S_0,S_0N^t\rangle_{\mathrm{F}} R t ( S 0 ) := ⟨ S 0 , S 0 N t ⟩ F and ∥ S 0 ∥ F = 1 \lVert S_0\rVert_{\mathrm{F}}=1 ∥ S 0 ∥ F = 1 ,
( 1 − β λ max ( K ˉ ) ) t ≤ R t ( S 0 ) ≤ ( 1 − β λ min ( K ˉ ) ) t , E S 0 [ R t ] = Tr N t d , (156) (1-\beta\lambda_{\max}(\bar{K}))^t\le R_t(S_0)\le(1-\beta\lambda_{\min}(\bar{K}))^t,\qquad\mathbb{E}_{S_0}[R_t]=\frac{\operatorname{Tr}N^t}{d}, \tag*{(156)} ( 1 − β λ m a x ( K ˉ ) ) t ≤ R t ( S 0 ) ≤ ( 1 − β λ m i n ( K ˉ ) ) t , E S 0 [ R t ] = d Tr N t , (156)
under β max i ∥ k i ∥ 2 ≤ 1 \beta\max_i\lVert k_i\rVert^2\le1 β max i ∥ k i ∥ 2 ≤ 1 . The statement is Proposition A2 under the substitution A ˉ ↦ K ˉ \bar{A}\mapsto\bar{K} A ˉ ↦ K ˉ , d 2 ↦ d d^2\mapsto d d 2 ↦ d of § 10.2 .
KronLSTM Retains Longer than DeltaNet
The two retention rates are governed by the spectra of A ˉ \bar{A} A ˉ and K ˉ \bar{K} K ˉ , and for independent keys and values these are related exactly.
Proposition A3 (Spectral comparison). Let k k k and v v v be independent and let V ˉ : = E [ v v ⊤ ] \bar{V}:=\mathbb{E}[vv^\top] V ˉ := E [ v v ⊤ ] . Then A ˉ = K ˉ ⊗ V ˉ \bar{A}=\bar{K}\otimes\bar{V} A ˉ = K ˉ ⊗ V ˉ , so the eigenvalues of A ˉ \bar{A} A ˉ are the d 2 d^2 d 2 pairwise products of those of K ˉ \bar{K} K ˉ and V ˉ \bar{V} V ˉ , and in particular λ min ( A ˉ ) = λ min ( K ˉ ) λ min ( V ˉ ) \lambda_{\min}(\bar{A})=\lambda_{\min}(\bar{K})\lambda_{\min}(\bar{V}) λ m i n ( A ˉ ) = λ m i n ( K ˉ ) λ m i n ( V ˉ ) and λ max ( A ˉ ) = λ max ( K ˉ ) λ max ( V ˉ ) \lambda_{\max}(\bar{A})=\lambda_{\max}(\bar{K})\lambda_{\max}(\bar{V}) λ m a x ( A ˉ ) = λ m a x ( K ˉ ) λ m a x ( V ˉ ) . If the values are ℓ 2 \ell_2 ℓ 2 -normalized, ∥ v ∥ = 1 \lVert v\rVert=1 ∥ v ∥ = 1 , then Tr V ˉ = 1 \operatorname{Tr}\bar{V}=1 Tr V ˉ = 1 and hence
λ min ( A ˉ ) ≤ λ min ( K ˉ ) d , λ max ( A ˉ ) ≤ λ max ( K ˉ ) , (157) \lambda_{\min}(\bar{A})\le\frac{\lambda_{\min}(\bar{K})}{d},\qquad\lambda_{\max}(\bar{A})\le\lambda_{\max}(\bar{K}), \tag*{(157)} λ m i n ( A ˉ ) ≤ d λ m i n ( K ˉ ) , λ m a x ( A ˉ ) ≤ λ m a x ( K ˉ ) , (157)
with equality in the first when the values are isotropic, V ˉ = I / d \bar{V}=I/d V ˉ = I / d .
Proof. a a ⊤ = ( k ⊗ v ) ( k ⊗ v ) ⊤ = k k ⊤ ⊗ v v ⊤ aa^\top=(k\otimes v)(k\otimes v)^\top=kk^\top\otimes vv^\top a a ⊤ = ( k ⊗ v ) ( k ⊗ v ) ⊤ = k k ⊤ ⊗ v v ⊤ by the mixed-product rule, and independence factorizes the expectation. The spectrum of a Kronecker product of symmetric matrices is the set of pairwise products. For normalized values λ min ( V ˉ ) ≤ Tr V ˉ / d = 1 / d \lambda_{\min}(\bar{V})\le\operatorname{Tr}\bar{V}/d=1/d λ m i n ( V ˉ ) ≤ Tr V ˉ / d = 1/ d and λ max ( V ˉ ) ≤ Tr V ˉ = 1 \lambda_{\max}(\bar{V})\le\operatorname{Tr}\bar{V}=1 λ m a x ( V ˉ ) ≤ Tr V ˉ = 1 , giving (A147); isotropy forces V ˉ = I / d \bar{V}=I/d V ˉ = I / d and the first inequality becomes an equality.
Because keys and values are ℓ 2 \ell_2 ℓ 2 -normalized in the KronLSTM block, ∥ a i ∥ 2 = ∥ k i ∥ 2 ∥ v i ∥ 2 = 1 = ∥ k i ∥ 2 \lVert a_i\rVert^2=\lVert k_i\rVert^2\lVert v_i\rVert^2=1=\lVert k_i\rVert^2 ∥ a i ∥ 2 = ∥ k i ∥ 2 ∥ v i ∥ 2 = 1 = ∥ k i ∥ 2 , so both models admit exactly the same step sizes β ≤ 1 \beta\le1 β ≤ 1 and the comparison at equal β \beta β is well posed. Under Propositions A2 and A3:
t 1 / e K r o n L S T M ≃ 1 β λ min ( A ˉ ) ≥ d β λ min ( K ˉ ) = d t 1 / e D e l t a N e t , (158) t_{1/e}^{\mathrm{KronLSTM}}\simeq\frac{1}{\beta\lambda_{\min}(\bar{A})}\ge\frac{d}{\beta\lambda_{\min}(\bar{K})}=d\,t_{1/e}^{\mathrm{DeltaNet}}, \tag*{(158)} t 1/ e KronLSTM ≃ β λ m i n ( A ˉ ) 1 ≥ β λ m i n ( K ˉ ) d = d t 1/ e DeltaNet , (158)
a factor d d d in context length at which a stored association is still legible, with equality for isotropic values.
No mode decays faster. By the second half of (A147), λ max ( A ˉ ) ≤ λ max ( K ˉ ) \lambda_{\max}(\bar{A})\le\lambda_{\max}(\bar{K}) λ m a x ( A ˉ ) ≤ λ m a x ( K ˉ ) , so even KronLSTM’s fastest-decaying mode is no faster than DeltaNet’s. The comparison is therefore not a trade of some directions against others: the whole spectrum of retention rates moves in KronLSTM’s favour. For isotropic values it moves by exactly the factor 1 / d 1/d 1/ d , λ k ( A ˉ ) = λ k ( K ˉ ) / d \lambda_k(\bar{A})=\lambda_k(\bar{K})/d λ k ( A ˉ ) = λ k ( K ˉ ) / d with multiplicity d d d each.
On average, likewise. Comparing (A145) with (A146) for isotropic values, Tr M t / d 2 = 1 d ∑ k ( 1 − β λ k ( K ˉ ) / d ) t ≥ 1 d ∑ k ( 1 − β λ k ( K ˉ ) ) t = Tr N t / d \operatorname{Tr}M^t/d^2=\frac{1}{d}\sum_k(1-\beta\lambda_k(\bar{K})/d)^t\ge\frac{1}{d}\sum_k(1-\beta\lambda_k(\bar{K}))^t=\operatorname{Tr}N^t/d Tr M t / d 2 = d 1 ∑ k ( 1 − β λ k ( K ˉ ) / d ) t ≥ d 1 ∑ k ( 1 − β λ k ( K ˉ ) ) t = Tr N t / d termwise.
Retention is bought from adaptation. The same matrix M t M^t M t governs both terms of (A142): the acquired memory approaches s L S s^{\mathrm{LS}} s LS at rate I − M t I-M^t I − M t . Retaining the initial memory d d d times longer therefore means approaching the least-squares memory d d d times more slowly, at fixed β \beta β . The statement is not that KronLSTM forgets less for free, but that its per-step write is d d d times gentler in every direction of an d 2 d^2 d 2 -dimensional memory, where DeltaNet’s is concentrated on a d d d -dimensional one — the same trade-off that appears in the capacity count (d 2 d^2 d 2 against d d d exactly satisfiable associations) and in the norm bound, seen through its time constant. Long contexts are precisely the regime in which the number of associations, not the speed of fitting any one of them, is binding.
Further Results
400M language models
Models with 400M/40B (Table A2 ) are closer to each other in their performance. KronLSTM has the best average and performs best on HellaSwag, ARC-easy, ARC-challenge, and WinoGrande. Mamba-3 is best on PIQA, KDA on LAMBADA, and GDN-2 on WinoGrande where it ties with KronLSTM. Compared with mLSTM, the matrix-memory xLSTM variant that KronLSTM builds on, KronLSTM is better on five of the six tasks, raises the average by 2.1 points, and lowers LAMBADA perplexity from 43.40 to 28.28. mLSTM is only marginally ahead on PIQA.
Table A2. Downstream performance of 400M/40B linear models. Avg-6 is the mean accuracy over the five commonsense reasoning tasks and LAMBADA. Best per column in bold , second best underlined.
Ablation on different formulations of KronLSTM
Table A3 ablates the components of the KronLSTM write rule at 1.5B/100B. We report Avg-6, the mean accuracy over the five commonsense reasoning tasks and LAMBADA, together with LAMBADA perplexity, the average over the six-task real-world recall suite at 2k context, and the average over the six RULER NIAH tasks of Table 3 at 4k, 8k and 16k. Three variants are compared with the base model of (10) :
(1) the ℓ 2 \ell_2 ℓ 2 -normalization of v t v_t v t is removed;
(2) erase and write are coupled by using the target c t c_t c t instead of c ~ t = c t / β t \tilde{c}_t=c_t/\beta_t c ~ t = c t / β t , so that β t \beta_t β t scales both terms as in Gated DeltaNet;
(3) the write gate is fixed to the unit target c t ≡ 1 c_t \equiv1 c t ≡ 1 .
On language modeling, variants (1) and (2) stay within 0.3 accuracy points and 0.25 perplexity points of the base model, with the coupled variant marginally ahead on both. Fixing the target costs 1.6 accuracy points and 0.8 perplexity points, the largest degradation among the ablations. On the recall suite, variants (1) and (2) gain 1.0 and 0.5 points, respectively. On NIAH the ordering reverses: at 4k the base model leads the coupled variant by 12.6 points and the unnormalized variant by 19.9 points, at 8k the unnormalized variant is ahead by 2.2 points, and at 16k the base model leads all three variants. The learned write gate and the decoupled write therefore matter most for retrieval within the pretraining context, while their effect on language modeling is small.
Long context extension results
Pretraining Details
All models are pretrained from scratch using lm-engine Mishra [39 ] with Triton kernels from flash-linear-attention Yang and Zhang [60 ] whenever available. Every mixer in a comparison shares the corpus, the tokenizer, the optimizer, the schedule, the batch size, and the seed; we only change the sequence mixer blocks. The models were trained on 16 nodes of 4 × 4\times 4 × NVIDIA GH200. Training is pure data parallelism under the FSDP-2 with bf16 mixed-precision.
Data and tokenization We train on the FineWeb-Edu 100BT sample, tokenized with the Llama 2 tokenizer with its standard vocab size of 32,000. Documents are concatenated, separated by the EOS token, and packed into fixed-length sequences of 8192 tokens; This is applied identically to every experiment.
Architecture All models are decoder-only and identical outside the sequence mixer. We implemented the block structures of every baseline as in the original works. Mamba-3 was trained using the standard expansion factor of 2, a dstate of 128, and a head dimension of 64. For Mamba-3, we utilized the SISO formulation. For the other baselines and our method, we utilized d q k = d v = 128 d_{qk}=d_v=128 d q k = d v = 128 . This makes the state size equal across all models. 1.5B models have 24 layers, a hidden dimension of 2048 with 16 heads. The 400M models have 24 layers with a hidden dimension of 1024 and 4 heads (here d v = 256 d_v = 256 d v = 256 ).
Optimization
Table 5.
Long-context extension
The context-extended models of § 3.3 branch from the 1.5B pretraining runs at 75B tokens and are trained for a further 25B tokens at a sequence length of 32,768, so that both extended models finish at exactly 100B tokens and are token-matched to the 8k-pretrained models. The architecture, tokenizer, and all model hyperparameters are unchanged. The global batch is held at the same 1M tokens per step, now as 32 sequences of 32,768 tokens instead of 128 sequences of 8,192.
Optimization. The optimizer state is re-initialized at the branch point for both models and the schedule restarts: linear warmup over 5% to a peak learning rate of 8.4 × 10 − 5 8.4 \times10^{-5} 8.4 × 1 0 − 5 , which is the value the parent cosine schedule had reached at the checkpoint, followed by cosine decay over 23,800 steps to 0.1× the peak. This mirrors the warmup-and-decay shape used by Gao et al. [20 ] . AdamW hyperparameters, weight decay, and gradient clipping are as in Appendix § 13 .
Data. The corpus takes the four long-document domains of ProLong-64K (Gao et al., [20 ] ), books, code, textbooks, and arXiv, re-tokenized with the Llama 2 tokenizer, and blends them at the ratios of Gao et al. [20 ] with FineWeb-Edu in place of their short-context mixture. Documents are concatenated and packed into sequences of 32,768 tokens as in pretraining.
Systems. Training runs on 8 nodes of 4×NVIDIA GH200 with ZeRO stage 2 and block-wise gradient checkpointing under bf16 mixed precision.
State Tracking Experiments
State-tracking tasks. Each sample is a token sequence of length ∼ U [ 2 , 128 ] \sim\mathcal{U}[2, 128] ∼ U [ 2 , 128 ] padded to the context length, with a single supervised position at its end. Parity: i.i.d. uniform bits, label the parity of the number of ones (chance 50%). S 3 S_3 S 3 : i.i.d. uniform transpositions ( 0 1 ) (0\ 1) ( 0 1 ) , ( 0 2 ) (0\ 2) ( 0 2 ) , ( 1 2 ) (1\ 2) ( 1 2 ) , label the prefix product x 1 ∘ ⋯ ∘ x ℓ x_1 \circ\cdots\circ x_\ell x 1 ∘ ⋯ ∘ x ℓ as one of the six group elements (chance 16.7%; tracking parity alone gives 33.3%). Modular arithmetic: an expression d 1 o 1 d 2 . . . d k = d_1 \ o_1 \ d_2 \ . . . \ d_k = d 1 o 1 d 2 ... d k = with uniform digits in { 0 , … , 4 } \{0, \ldots, 4\} { 0 , … , 4 } and operators in { + , − , × } \{+, -, \times\} { + , − , × } , evaluated with operator precedence modulo 5, the result supervised at the = token (chance 20%). Samples are generated on the fly and deterministically from the run seed (seeds 1-3); evaluation uses 50,000 fresh sequences per length L ∈ { 128 , 256 , 512 , 1024 } L \in\{128, 256, 512, 1024\} L ∈ { 128 , 256 , 512 , 1024 } .
Models and training. All models share a 128-dimensional embedding, pre-norm residual blocks (RMSNorm, no feed-forward sublayer) and a linear read-out; they differ only in the mixer: Gated DeltaNet with the eigenvalue range extended to [ − 1 , 1 ] [-1, 1] [ − 1 , 1 ] (Grazzi et al. [21 ] ), Mamba-3 (SISO, expansion 2, state size 128, data-dependent RoPE), and KronLSTM. For optimizer we used AdamW with weight decay 0.1, batch size 128, bf16. Parity and S 3 S_3 S 3 : two blocks, constant learning rate ∈ { 10 − 3 , 10 − 4 } \in\{10^{-3}, 10^{-4}\} ∈ { 1 0 − 3 , 1 0 − 4 } , 50k steps, gradient clipping 100. Modular arithmetic requires depth and a longer budget for all three architectures: three blocks, 150k steps. We report end-of-training accuracy, mean ± \pm ± standard deviation over three seeds, at the learning rate with the best mean accuracy over L ∈ { 256 } L \in\{256\} L ∈ { 256 } .
Chunkwise-Parallel Training of KronLSTM
The KronLSTM update (10) revises the matrix memory by a rank-one term whose coefficient depends on the memory only through a scalar. It therefore admits the chunkwise-parallel training scheme of Yang et al. [61 ] , Yang et al. [62 ] , Kimi Team et al. [32 ] , and Hatamizadeh et al. [24 ] : the sequence is split into chunks of length C C C , all token interactions inside a chunk are expressed as dense matrix products and one small triangular solve, and the memory is propagated only between chunks. Three points differ from the delta rule. (i) The triangular solve is built on the Gram matrix of the Kronecker features a t = k t ⊗ v t \boldsymbol{a}_t = \boldsymbol{k}_t \otimes\boldsymbol{v}_t a t = k t ⊗ v t , which by the mixed-product property is the Hadamard product of a (decay-weighted) key Gram and the value Gram; the d k d v d_kd_v d k d v -dimensional features are never formed. (ii) The pseudo-values are scalar multiples u r v r u_r v_r u r v r of the values. (iii) The incoming memory of a chunk enters the solve through C C C scalar scores v r T S 0 ( γ r ⊙ k r ) \boldsymbol{v}_r^{\mathsf{T}} S_0(\boldsymbol{\gamma}_r \odot\boldsymbol{k}_r) v r T S 0 ( γ r ⊙ k r ) instead of the d v d_v d v -dimensional read-outs S 0 k r S_0\boldsymbol{k}_r S 0 k r . The vector forget gate is handled as in KDA and Gated DeltaNet-2 by absorbing the cumulative decay into an asymmetric pair of keys.
Setting and notation
We consider one chunk of C C C tokens and suppress the chunk index. Let S 0 ∈ R d v × d k S_0 \in\mathbb{R}^{d_v \times d_k} S 0 ∈ R d v × d k be the memory entering the chunk and let q r , k r ∈ R d k \boldsymbol{q}_r,\boldsymbol{k}_r \in\mathbb{R}^{d_k} q r , k r ∈ R d k , v r ∈ R d v \boldsymbol{v}_r \in\mathbb{R}^{d_v} v r ∈ R d v , β r , c r ∈ R \beta_r,c_r \in\mathbb{R} β r , c r ∈ R and f r ∈ ( 0 , 1 ) d k \boldsymbol{f}_r \in(0,1)^{d_k} f r ∈ ( 0 , 1 ) d k , F r = diag ( f r ) F_r = \operatorname{diag}(\boldsymbol{f}_r) F r = diag ( f r ) , be the inputs and gates of the r r r -th token, r = 1 , … , C r = 1,\ldots,C r = 1 , … , C . Writing S r S_r S r for the memory after token r r r , the update (10) and the read-out are1
S r = S r − 1 F r + u r v r k r T , u r : = c r − β r y r , y r : = v r T S r − 1 F r k r , h r = S r q r . (159) S_r = S_{r-1}F_r + u_r\boldsymbol{v}_r\boldsymbol{k}_r^{\mathsf{T}}, \qquad u_r := c_r - \beta_r y_r, \qquad y_r := \boldsymbol{v}_r^{\mathsf{T}}S_{r-1}F_r\boldsymbol{k}_r, \qquad\boldsymbol{h}_r = S_r\boldsymbol{q}_r. \tag*{(159)} S r = S r − 1 F r + u r v r k r T , u r := c r − β r y r , y r := v r T S r − 1 F r k r , h r = S r q r . (159)
We call y r y_r y r the score of the pair ( k r , v r ) (\boldsymbol{k}_r,\boldsymbol{v}_r) ( k r , v r ) on the decayed memory and u r u_r u r the write coefficient, the coefficient of the rank-one write along v r k r T \boldsymbol{v}_r\boldsymbol{k}_r^{\mathsf{T}} v r k r T . It is the only quantity in (A149) that depends on the memory.
Cumulative decay. With ℓ r : = ∑ i ≤ r log f i \ell_r := \sum_{i \le r} \log f_i ℓ r := ∑ i ≤ r log f i (a cumulative sum inside the chunk) we set
γ r : = exp ( ℓ r ) = f 1 ⊙ ⋯ ⊙ f r ∈ ( 0 , 1 ] d k , γ 0 : = 1 , Γ r : = diag ( γ r ) = F 1 ⋯ F r . (160) \boldsymbol{\gamma}_r := \exp(\ell_r) = \boldsymbol{f}_1 \odot\cdots\odot\boldsymbol{f}_r \in(0,1]^{d_k}, \qquad\boldsymbol{\gamma}_0 := 1, \qquad\Gamma_r := \operatorname{diag}(\boldsymbol{\gamma}_r) = F_1 \cdots F_r. \tag*{(160)} γ r := exp ( ℓ r ) = f 1 ⊙ ⋯ ⊙ f r ∈ ( 0 , 1 ] d k , γ 0 := 1 , Γ r := diag ( γ r ) = F 1 ⋯ F r . (160)
All products, quotients and exponentials of γ \boldsymbol{\gamma} γ ’s are element-wise over the key dimension. For s ≤ r s \le r s ≤ r the ratio γ r / γ s = f s + 1 ⊙ ⋯ ⊙ f r \boldsymbol{\gamma}_r/\boldsymbol{\gamma}_s = \boldsymbol{f}_{s+1} \odot\cdots\odot\boldsymbol{f}_r γ r / γ s = f s + 1 ⊙ ⋯ ⊙ f r lies in ( 0 , 1 ] d k (0,1]^{d_k} ( 0 , 1 ] d k ; it is the decay that a key written at token s s s has undergone by token r r r . Four decay-scaled copies of the keys and queries appear:
k ^ r : = γ r ⊙ k r , k ‾ r : = k r / γ r , k ˇ r : = ( γ C / γ r ) ⊙ k r , q ^ r : = γ r ⊙ q r . (161) \widehat{\boldsymbol{k}}_r := \boldsymbol{\gamma}_r \odot\boldsymbol{k}_r, \qquad\overline{\boldsymbol{k}}_r := \boldsymbol{k}_r/\boldsymbol{\gamma}_r, \qquad\check{\boldsymbol{k}}_r := (\boldsymbol{\gamma}_C/\boldsymbol{\gamma}_r) \odot\boldsymbol{k}_r, \qquad\widehat{\boldsymbol{q}}_r := \boldsymbol{\gamma}_r \odot\boldsymbol{q}_r. \tag*{(161)} k r := γ r ⊙ k r , k r := k r / γ r , k ˇ r := ( γ C / γ r ) ⊙ k r , q r := γ r ⊙ q r . (161)
k ^ r \widehat{\boldsymbol{k}}_r k r is the key with which the memory is scored, k ‾ r \overline{\boldsymbol{k}}_r k r the key along which it is written in the decay-normalized frame of Section A8.2, and k ˇ r \check{\boldsymbol{k}}_r k ˇ r the key as it appears in the memory at the end of the chunk.
Stacked matrices. K , Q ∈ R C × d k K,Q \in\mathbb{R}^{C \times d_k} K , Q ∈ R C × d k and V ∈ R C × d v V \in\mathbb{R}^{C \times d_v} V ∈ R C × d v stack k r T \boldsymbol{k}_r^{\mathsf{T}} k r T , q r T \boldsymbol{q}_r^{\mathsf{T}} q r T , v r T \boldsymbol{v}_r^{\mathsf{T}} v r T as rows; K ^ , K ‾ , K ˇ , Q ^ \widehat{K},\overline{K},\check{K},\widehat{Q} K , K , K ˇ , Q stack the vectors of (A151); β , c , u , p ∈ R C \boldsymbol{\beta},\boldsymbol{c},\boldsymbol{u},\boldsymbol{p} \in\mathbb{R}^{C} β , c , u , p ∈ R C stack the scalars; H ∈ R C × d v H \in\mathbb{R}^{C \times d_v} H ∈ R C × d v stacks the read-outs h r T \boldsymbol{h}_r^{\mathsf{T}} h r T . The three C × C C \times C C × C matrices
G r s k k : = 1 [ r > s ] k ^ r T k ‾ s = 1 [ r > s ] ∑ d = 1 d k k r d k s d γ r d γ s d , G v v : = V V T , G r s q k : = 1 [ r ≥ s ] q ^ r T k ‾ s (162) G^{kk}_{rs} := \mathbb{1}[r > s]\,\widehat{\boldsymbol{k}}_r^{\mathsf{T}}\overline{\boldsymbol{k}}_s = \mathbb{1}[r > s]\sum_{d=1}^{d_k} k_{rd}k_{sd}\frac{\gamma_{rd}}{\gamma_{sd}}, \qquad G^{vv} := VV^{\mathsf{T}}, \qquad G^{qk}_{rs} := \mathbb{1}[r \ge s]\,\widehat{\boldsymbol{q}}_r^{\mathsf{T}}\overline{\boldsymbol{k}}_s \tag*{(162)} G r s k k := 1 [ r > s ] k r T k s = 1 [ r > s ] d = 1 ∑ d k k r d k s d γ s d γ r d , G v v := V V T , G r s q k := 1 [ r ≥ s ] q r T k s (162)
are the decay-weighted key Gram (strictly lower triangular), the value Gram, and the decay-weighted read-out Gram (lower triangular with diagonal). All entries of G k k G^{kk} G k k and G q k G^{qk} G q k involve only ratios γ r / γ s \gamma_r/\gamma_s γ r / γ s with s ≤ r s \le r s ≤ r , hence only factors in ( 0 , 1 ] (0,1] ( 0 , 1 ] .
Decay-normalized recurrence
The forget gate acts on the key dimension by a diagonal matrix, and diagonal matrices commute. This allows the cumulative decay to be taken out of the recurrence.
Proposition A4 (Decay-normalized recurrence). Let S ^ r : = S r Γ r − 1 \widehat{S}_r := S_r\Gamma_r^{-1} S r := S r Γ r − 1 , so that S ^ 0 = S 0 \widehat{S}_0 = S_0 S 0 = S 0 . Then, for r = 1 , … , C r = 1,\ldots,C r = 1 , … , C ,
S ^ r = S ^ r − 1 + u r v r k ˉ r T , y r = v r T S ^ r − 1 k ^ r (163) \widehat{S}_r = \widehat{S}_{r-1} + u_r v_r \bar{k}_r^{\mathsf{T}}, \qquad y_r = v_r^{\mathsf{T}}\widehat{S}_{r-1}\widehat{k}_r \tag*{(163)} S r = S r − 1 + u r v r k ˉ r T , y r = v r T S r − 1 k r (163) and consequently
S ^ r = S 0 + ∑ s ≤ r u s v s k ˉ s T . (164) \widehat{S}_r = S_0 + \sum_{s \le r} u_s v_s \bar{k}_s^{\mathsf{T}}. \tag*{(164)} S r = S 0 + s ≤ r ∑ u s v s k ˉ s T . (164) Proof. Since Γ r = Γ r − 1 F r \Gamma_r = \Gamma_{r-1}F_r Γ r = Γ r − 1 F r , we have S r − 1 F r = S ^ r − 1 Γ r − 1 F r = S ^ r − 1 Γ r S_{r-1}F_r = \widehat{S}_{r-1}\Gamma_{r-1}F_r = \widehat{S}_{r-1}\Gamma_r S r − 1 F r = S r − 1 Γ r − 1 F r = S r − 1 Γ r . Inserting this into (A149) gives S r = S ^ r − 1 Γ r + u r v r k ^ r T S_r = \widehat{S}_{r-1}\Gamma_r + u_rv_r\widehat{k}_r^{\mathsf{T}} S r = S r − 1 Γ r + u r v r k r T ; right-multiplication by Γ r − 1 \Gamma_r^{-1} Γ r − 1 yields the first identity, because k ^ r T Γ r − 1 = k ˉ r T \widehat{k}_r^{\mathsf{T}}\Gamma_r^{-1} = \bar{k}_r^{\mathsf{T}} k r T Γ r − 1 = k ˉ r T . The second identity is y r = v r T ( S ^ r − 1 Γ r ) k r = v r T S ^ r − 1 ( Γ r k r ) y_r = v_r^{\mathsf{T}}(\widehat{S}_{r-1}\Gamma_r)k_r = v_r^{\mathsf{T}}\widehat{S}_{r-1}(\Gamma_rk_r) y r = v r T ( S r − 1 Γ r ) k r = v r T S r − 1 ( Γ r k r ) . Unrolling the first identity from S ^ 0 = S 0 \widehat{S}_0 = S_0 S 0 = S 0 gives (A154).
In the normalized frame the recurrence is the Kronecker Delta Rule without forget gate, applied to an asymmetric key pair: the association is scored with the decayed key k ^ r \widehat{k}_r k r and written along the un-decayed key k ˉ r \bar{k}_r k ˉ r . This is the same device by which Gated DeltaNet-2 absorbs its channel-wise decay into asymmetric erase factors ([24 ] ). Equation (A154) is the additive representation of the memory, S r = ( S 0 + ∑ s ≤ r u s v s k ˉ s T ) Γ r S_r = \left(S_0 + \sum_{s \le r} u_sv_s\bar{k}_s^{\mathsf{T}}\right)\Gamma_r S r = ( S 0 + ∑ s ≤ r u s v s k ˉ s T ) Γ r ; the memory revisions do not have to be materialized, only the C C C scalars u s u_s u s .
The write coefficients solve a unit lower triangular system
Proposition A5 (Write coefficients). Let p r : = v r T S 0 k ^ r p_r := v_r^{\mathsf{T}}S_0\widehat{k}_r p r := v r T S 0 k r be the score of the r r r -th pair on the incoming memory, and let
N : = diag ( β ) ( G k k ⊙ G v v ) ∈ R C × C , Λ : = ( I + N ) − 1 . (165) N := \operatorname{diag}(\beta)\left(G^{kk} \odot G^{vv}\right) \in\mathbb{R}^{C \times C}, \qquad\Lambda:= (I + N)^{-1}. \tag*{(165)} N := diag ( β ) ( G k k ⊙ G v v ) ∈ R C × C , Λ := ( I + N ) − 1 . (165) N N N is strictly lower triangular, so I + N I + N I + N is unit lower triangular and invertible for every β \beta β and every input. The write coefficients are
u = Λ ( c − β ⊙ p ) = Λ c ⏟ = : u ∘ − Λ diag ( β ) p . (166) u = \Lambda(c - \beta\odot p) = \underbrace{\Lambda c}_{=:u^\circ} - \Lambda\operatorname{diag}(\beta)p. \tag*{(166)} u = Λ ( c − β ⊙ p ) = =: u ∘ Λ c − Λ diag ( β ) p . (166) Proof. Insert (A154) at r − 1 r-1 r − 1 into the score in (A153):
y r = v r T S 0 k ^ r + ∑ s < r u s ( v r T v s ) ( k ˉ s T k ^ r ) = p r + ∑ s < r ( G k k ⊙ G v v ) r s u s . y_r = v_r^{\mathsf{T}}S_0\widehat{k}_r + \sum_{s<r}u_s(v_r^{\mathsf{T}}v_s)(\bar{k}_s^{\mathsf{T}}\widehat{k}_r) = p_r + \sum_{s<r}(G^{kk}\odot G^{vv})_{rs}u_s. y r = v r T S 0 k r + s < r ∑ u s ( v r T v s ) ( k ˉ s T k r ) = p r + s < r ∑ ( G k k ⊙ G v v ) r s u s . Hence u r = c r − β r y r u_r = c_r - \beta_ry_r u r = c r − β r y r reads u r + ∑ s < r N r s u s = c r − β r p r u_r + \sum_{s<r}N_{rs}u_s = c_r - \beta_rp_r u r + ∑ s < r N r s u s = c r − β r p r , which is row r r r of ( I + N ) u = c − β ⊙ p (I+N)u = c-\beta\odot p ( I + N ) u = c − β ⊙ p .
The Gram of the Kronecker features. With a ^ r : = k ^ r ⊗ v r \widehat{a}_r := \widehat{k}_r \otimes v_r a r := k r ⊗ v r and a ˉ s : = k ˉ s ⊗ v s \bar{a}_s := \bar{k}_s \otimes v_s a ˉ s := k ˉ s ⊗ v s , the mixed-product property of the Kronecker product gives a ^ r T a ˉ s = ( k ^ r T k ˉ s ) ( v r T v s ) \widehat{a}_r^{\mathsf{T}}\bar{a}_s = (\widehat{k}_r^{\mathsf{T}}\bar{k}_s)(v_r^{\mathsf{T}}v_s) a r T a ˉ s = ( k r T k ˉ s ) ( v r T v s ) , so G k k ⊙ G v v G^{kk}\odot G^{vv} G k k ⊙ G v v is the strictly lower triangular part of the Gram matrix of the decay-weighted Kronecker features. Without forget gate (f r ≡ 1 f_r \equiv1 f r ≡ 1 ) it is the strictly lower part of the Gram matrix ( a r T a s ) r s (a_r^{\mathsf{T}}a_s)_{rs} ( a r T a s ) r s of the features a r = k r ⊗ v r a_r = k_r \otimes v_r a r = k r ⊗ v r , i.e., exactly the matrix that the UT transform of DeltaNet ([61 ] ) builds from K K T KK^{\mathsf{T}} K K T , with the feature a r a_r a r in place of k r k_r k r as anticipated by the vectorized form (4) . The d k d v d_kd_v d k d v -dimensional features are never formed: two C × C C \times C C × C Gram matrices of d k d_k d k - and d v d_v d v -dimensional vectors and one Hadamard product suffice.
Decoupled erase and write. The write gate c r c_r c r enters only the right-hand side of the solve, the erase gate β r \beta_r β r enters the system matrix and the coupling to S 0 S_0 S 0 . The state-independent part u ∘ = Λ c u^\circ= \Lambda c u ∘ = Λ c and the matrix Λ diag ( β ) \Lambda\operatorname{diag}(\beta) Λ diag ( β ) can be computed for all chunks in parallel; they are the analogues of the pseudo-values U = T V U = TV U = T V and of W = T K W = TK W = T K in DeltaNet. The only sequential quantity is the vector of scores p p p .
Chunkwise form
Theorem A11 (Chunkwise KronLSTM). Let V ~ : = diag ( u ) V ∈ R C × d v \widetilde{V} := \operatorname{diag}(u)V \in\mathbb{R}^{C \times d_v} V := diag ( u ) V ∈ R C × d v be the matrix of pseudo-values, with rows u r v r ⊤ u_r v_r^\top u r v r ⊤ , where u u u is given by Proposition A5 and
p = ( ( V S 0 ) ⊙ K ^ ) 1 (167) p = \left((VS_0) \odot\widehat{K}\right)\mathbf{1} \tag*{(167)} p = ( ( V S 0 ) ⊙ K ) 1 (167) (row sums of a C × d k C \times d_k C × d k matrix).
Then the memory at the end of the chunk and the read-outs of the chunk are
S C = S 0 Γ C + V ~ ⊤ K ~ ∈ R d v × d k , (168) S_C = S_0 \Gamma_C + \widetilde{V}^{\top}\widetilde{K} \in\mathbb{R}^{d_v \times d_k}, \tag*{(168)} S C = S 0 Γ C + V ⊤ K ∈ R d v × d k , (168) H = Q ^ S 0 ⊤ + G q k V ~ ∈ R C × d v . (169) H = \widehat{Q}S_0^{\top} + G^{qk}\widetilde{V} \in\mathbb{R}^{C \times d_v}. \tag*{(169)} H = Q S 0 ⊤ + G q k V ∈ R C × d v . (169)
Proof. Equation (A157) is p r = v r ⊤ S 0 k ^ r = ∑ d ( V S 0 ) r d K ^ r d p_r = v_r^\top S_0\widehat{k}_r = \sum_d (VS_0)_{rd}\widehat{K}_{rd} p r = v r ⊤ S 0 k r = ∑ d ( V S 0 ) r d K r d . For the state, S C = S ^ C Γ C = S 0 Γ C + ∑ s ≤ C u s v s k ‾ s ⊤ Γ C S_C = \widehat{S}_C\Gamma_C = S_0\Gamma_C + \sum_{s \le C} u_s v_s\overline{k}_s^\top\Gamma_C S C = S C Γ C = S 0 Γ C + ∑ s ≤ C u s v s k s ⊤ Γ C by (A154), and k ‾ s ⊤ Γ C = ( ( γ C / γ s ) ⊙ k s ) ⊤ = k ~ s ⊤ \overline{k}_s^\top\Gamma_C = ((\gamma_C/\gamma_s) \odot k_s)^\top= \widetilde{k}_s^\top k s ⊤ Γ C = (( γ C / γ s ) ⊙ k s ) ⊤ = k s ⊤ , which is (A158). For the read-out, h r = S r q r = S ^ r Γ r q r = S ^ r q ^ r = S 0 q ^ r + ∑ s ≤ r u s ( k ‾ s ⊤ q ^ r ) v s h_r = S_rq_r = \widehat{S}_r\Gamma_rq_r = \widehat{S}_r\widehat{q}_r = S_0\widehat{q}_r + \sum_{s \le r} u_s(\overline{k}_s^\top\widehat{q}_r)v_s h r = S r q r = S r Γ r q r = S r q r = S 0 q r + ∑ s ≤ r u s ( k s ⊤ q r ) v s by (A154); transposing and stacking over r r r gives (A159), the coefficient of u s v s ⊤ u_sv_s^\top u s v s ⊤ in row r r r being G r s q k G^{qk}_{rs} G r s q k .
Indexing the chunks by [ n ] [n] [ n ] and writing S [ n ] S_{[n]} S [ n ] for the memory entering chunk n n n , (A158) is the chunk-level recurrence
S [ n + 1 ] = S [ n ] Γ [ n ] , C + V ~ [ n ] ⊤ K ~ [ n ] , (170) S_{[n+1]} = S_{[n]}\Gamma_{[n],C} + \widetilde{V}_{[n]}^\top\widetilde{K}_{[n]}, \tag*{(170)} S [ n + 1 ] = S [ n ] Γ [ n ] , C + V [ n ] ⊤ K [ n ] , (170)
with S [ 0 ] = 0 S_{[0]} = 0 S [ 0 ] = 0 (or a given initial memory). Everything in (A152)–(A159) except p p p , u u u and V ~ \widetilde{V} V is local to a chunk and is computed for all chunks at once; the dependence S [ n ] → p → u → S [ n + 1 ] S_{[n]} \to p \to u \to S_{[n+1]} S [ n ] → p → u → S [ n + 1 ] is the only sequential part. Compared with the delta-rule family, the pseudo-value of token r r r is the value v r v_r v r scaled by the scalar u r u_r u r , and the memory couples into the chunk through the C C C scalars p p p rather than through the C C C vectors S 0 k r ∈ R d v S_0k_r \in\mathbb{R}^{d_v} S 0 k r ∈ R d v : computing p p p needs one matrix product V S 0 VS_0 V S 0 and a row-wise contraction over the value dimension, which is where a kernel that tiles the memory over d v d_v d v must exchange partial sums.
WY representation of the chunk transition
In vectorized form, s = vec ( S ) s = \operatorname{vec}(S) s = vec ( S ) with vec ( v k ⊤ ) = k ⊗ v \operatorname{vec}(vk^\top) = k \otimes v vec ( v k ⊤ ) = k ⊗ v and vec ( S F ) = ( F ⊗ I d v ) vec ( S ) \operatorname{vec}(SF) = (F \otimes I_{d_v})\operatorname{vec}(S) vec ( S F ) = ( F ⊗ I d v ) vec ( S ) , one step of (A149) is s r = ( I − β r a r a r ⊤ ) ( F r ⊗ I d v ) s r − 1 + c r a r s_r = (I - \beta_r a_ra_r^\top)(F_r \otimes I_{d_v})s_{r-1} + c_ra_r s r = ( I − β r a r a r ⊤ ) ( F r ⊗ I d v ) s r − 1 + c r a r with a r = k r ⊗ v r a_r = k_r \otimes v_r a r = k r ⊗ v r : a generalized Householder transition, as in (4) , preceded by a diagonal decay. Theorem A11 is the WY representation [11 ] of the product of these factors.
Corollary A7 (WY form). Let A ^ : = [ k ^ 1 ⊗ v 1 , … , k ^ C ⊗ v C ] \widehat{A} := [\widehat{k}_1 \otimes v_1, \ldots, \widehat{k}_C \otimes v_C] A := [ k 1 ⊗ v 1 , … , k C ⊗ v C ] and A ~ : = [ k ~ 1 ⊗ v 1 , … , k ~ C ⊗ v C ] \widetilde{A} := [\widetilde{k}_1 \otimes v_1, \ldots, \widetilde{k}_C \otimes v_C] A := [ k 1 ⊗ v 1 , … , k C ⊗ v C ] in R d k d v × C \mathbb{R}^{d_kd_v \times C} R d k d v × C . Then s C = P s 0 + d s_C = Ps_0 + d s C = P s 0 + d with
P = ∏ r = C 1 ( I − β r a r a r ⊤ ) ( F r ⊗ I d v ) = ( Γ C ⊗ I d v ) − A ~ Λ diag ( β ) A ^ ⊤ , d = A ~ Λ c . (171) P = \prod_{r=C}^{1}(I-\beta_ra_ra_r^\top)(F_r \otimes I_{d_v}) = (\Gamma_C \otimes I_{d_v}) - \widetilde{A}\Lambda\operatorname{diag}(\beta)\widehat{A}^\top,\qquad d = \widetilde{A}\Lambda c. \tag*{(171)} P = r = C ∏ 1 ( I − β r a r a r ⊤ ) ( F r ⊗ I d v ) = ( Γ C ⊗ I d v ) − A Λ diag ( β ) A ⊤ , d = A Λ c . (171)
Proof. By (A158), s C = ( Γ C ⊗ I ) s 0 + ∑ r u r ( k ~ r ⊗ v r ) = ( Γ C ⊗ I ) s 0 + A ~ u s_C = (\Gamma_C \otimes I)s_0 + \sum_r u_r(\widetilde{k}_r \otimes v_r) = (\Gamma_C \otimes I)s_0 + \widetilde{A}u s C = ( Γ C ⊗ I ) s 0 + ∑ r u r ( k r ⊗ v r ) = ( Γ C ⊗ I ) s 0 + A u . Since p r = ⟨ v r k ^ r ⊤ , S 0 ⟩ F = a ^ r ⊤ s 0 p_r = \langle v_r\widehat{k}_r^\top,S_0\rangle_F = \widehat{a}_r^\top s_0 p r = ⟨ v r k r ⊤ , S 0 ⟩ F = a r ⊤ s 0 , we have p = A ^ ⊤ s 0 p = \widehat{A}^\top s_0 p = A ⊤ s 0 , and (A156) gives u = Λ c − Λ diag ( β ) A ^ ⊤ s 0 u = \Lambda c - \Lambda\operatorname{diag}(\beta)\widehat{A}^\top s_0 u = Λ c − Λ diag ( β ) A ⊤ s 0 .
The correction to the diagonal decay has rank at most C C C , as for DeltaNet, where P = I − ∑ s w s k s ⊤ P = I - \sum_s w_sk_s^\top P = I − ∑ s w s k s ⊤ . The difference is that the columns of A ~ Λ diag ( β ) \widetilde{A}\Lambda\operatorname{diag}(\beta) A Λ diag ( β ) are linear combinations of Kronecker products k ~ s ⊗ v s \widetilde{k}_s \otimes v_s k s ⊗ v s and are in general not themselves Kronecker products, so the correction is not of the delta-rule form ∑ s ( S 0 w s ) k s ⊤ \sum_s(S_0w_s)k_s^\top ∑ s ( S 0 w s ) k s ⊤ , in which the memory enters through the d v d_v d v -dimensional read-outs S 0 w s S_0w_s S 0 w s . Instead the memory enters through the C C C scalars p p p , and each pseudo-value is v s v_s v s scaled by a linear function of S 0 S_0 S 0 . Table A5 summarizes the roles of the chunk quantities in the delta-rule family and in KronLSTM.
Table A5. Chunkwise forms of delta-rule memories and of KronLSTM, in the orientation S ∈ R d v × d k S\in\mathbb{R}^{d_v\times d_k} S ∈ R d v × d k of this paper. Λ \Lambda Λ is the inverse of the unit lower triangular matrix I + N I+N I + N built from the listed Gram; “coupling” is the term through which the incoming memory S 0 S_0 S 0 enters the write of token r r r ; stril \operatorname{stril} stril is the strictly lower triangular part. For KDA and Gated DeltaNet-2 the decay-weighted key Gram is G k k G^{kk} G k k of (A152); for Gated DeltaNet-2, b r ∈ [ 0 , 1 ] d k b_r\in[0,1]^{d_k} b r ∈ [ 0 , 1 ] d k and w r ∈ [ 0 , 1 ] d v w_r\in[0,1]^{d_v} w r ∈ [ 0 , 1 ] d v are its channel-wise erase and write gates.
Variants
The derivation uses only the structure “diagonal decay on the key side, rank-one write along v r k r ⊤ v_rk_r^\top v r k r ⊤ with a coefficient affine in the scalar score”:
Coupled erase and write (variant “β \beta β on both sides” of Table A3 ; target c r c_r c r instead of c ~ r = c r / β r \widetilde{c}_r=c_r/\beta_r c r = c r / β r ): replace c c c by β ⊙ c \beta\odot c β ⊙ c in (A156). Unit target (variant “non-learnable target”): c = 1 c=1 c = 1 , so u ∘ = Λ 1 u^\circ=\Lambda1 u ∘ = Λ1 .
State tracking , β r ∈ ( 0 , 2 ) \beta_r\in(0,2) β r ∈ ( 0 , 2 ) (§ 3.5 ): nothing changes. I + N I+N I + N is unit lower triangular for every β \beta β , so the solve needs no condition on the gates or on the data.
Scalar forget gate f r = f r 1 f_r=f_r1 f r = f r 1 (as in Gated DeltaNet): γ r \gamma_r γ r is a scalar and G r s k k = 1 [ r > s ] ( k r ⊤ k s ) γ r / γ s G^{kk}_{rs}=\mathbb{1}[r>s](k_r^\top k_s)\gamma_r/\gamma_s G r s k k = 1 [ r > s ] ( k r ⊤ k s ) γ r / γ s . No forget gate: G k k = stril ( K K ⊤ ) G^{kk}=\operatorname{stril}(KK^\top) G k k = stril ( K K ⊤ ) and K ^ = K ‾ = K ~ = K \widehat{K}=\overline{K}=\widetilde{K}=K K = K = K = K .
Values of arbitrary norm. The kernels accept unnormalized values by the substitution β r ← β r / ∥ v r ∥ 2 \beta_r\leftarrow\beta_r/\lVert v_r\rVert^2 β r ← β r / ∥ v r ∥ 2 (the erase step is capped so that β r ∥ a r ∥ 2 = β r \beta_r\lVert a_r\rVert^2=\beta_r β r ∥ a r ∥ 2 = β r for unit-norm keys and the contraction condition of Property A6 holds independently of ∥ v r ∥ \lVert v_r\rVert ∥ v r ∥ ); the write gate is unaffected. With the ℓ 2 \ell_2 ℓ 2 -normalized values of § 2.2 this is (A149).
Algorithm, cost and implementation
Algorithm (forward pass, one head). Given Q , K ∈ R L × d k Q,K\in\mathbb{R}^{L\times d_k} Q , K ∈ R L × d k , V ∈ R L × d v V\in\mathbb{R}^{L\times d_v} V ∈ R L × d v , β , c ∈ R L \beta,c\in\mathbb{R}^{L} β , c ∈ R L , log f ∈ R L × d k \log f\in\mathbb{R}^{L\times d_k} log f ∈ R L × d k , a chunk length C C C and an initial memory S [ 0 ] S_{[0]} S [ 0 ] :
Chunk-local, in parallel over chunks. For every chunk: ℓ ← cumsum ( log f ) \ell\leftarrow\operatorname{cumsum}(\log f) ℓ ← cumsum ( log f ) within the chunk, γ ← exp ( ℓ ) \gamma\leftarrow\exp(\ell) γ ← exp ( ℓ ) , and K ^ , K ‾ , K ~ , Q ^ \widehat{K},\overline{K},\widetilde{K},\widehat{Q} K , K , K , Q as in (A151); G k k , G v v , G q k G^{kk},G^{vv},G^{qk} G k k , G v v , G q k as in (A152), the entries of G k k G^{kk} G k k and G q k G^{qk} G q k via exp ( ℓ r − ℓ s ) \exp(\ell_r-\ell_s) exp ( ℓ r − ℓ s ) for s ≤ r s\le r s ≤ r ; N ← diag ( β ) ( G k k ⊙ G v v ) N\leftarrow\operatorname{diag}(\beta)(G^{kk}\odot G^{vv}) N ← diag ( β ) ( G k k ⊙ G v v ) and Λ ← ( I + N ) − 1 \Lambda\leftarrow(I+N)^{-1} Λ ← ( I + N ) − 1 by forward substitution; u ∘ ← Λ c u^\circ\leftarrow\Lambda c u ∘ ← Λ c and Λ β ← Λ diag ( β ) \Lambda_\beta\leftarrow\Lambda\operatorname{diag}(\beta) Λ β ← Λ diag ( β ) .
Scan, sequential over chunks. S ← S [ 0 ] S\leftarrow S_{[0]} S ← S [ 0 ] ; for n = 0 , 1 , … n=0,1,\ldots n = 0 , 1 , … : p ← ( ( V S ) ⊙ K ^ ) 1 p\leftarrow((VS)\odot\widehat{K})1 p ← (( V S ) ⊙ K ) 1 ; u ← u ∘ − Λ β p u\leftarrow u^\circ-\Lambda_\beta p u ← u ∘ − Λ β p ; V ~ ← diag ( u ) V \widetilde{V}\leftarrow\operatorname{diag}(u)V V ← diag ( u ) V ; H [ n ] ← Q ^ S ⊤ + G q k V ~ H_{[n]}\leftarrow\widehat{Q}S^\top+G^{qk}\widetilde{V} H [ n ] ← Q S ⊤ + G q k V ; S ← S diag ( γ C ) + V ~ ⊤ K ~ S\leftarrow S\operatorname{diag}(\gamma_C)+\widetilde{V}^{\top}\widetilde{K} S ← S diag ( γ C ) + V ⊤ K .
Return H ∈ R L × d v H\in\mathbb{R}^{L\times d_v} H ∈ R L × d v and the final memory S S S .
Cost. Per chunk and head the algorithm performs the two C × C C\times C C × C key Grams (2 C 2 d k 2C^2d_k 2 C 2 d k multiply–adds), the value Gram (C 2 d v C^2d_v C 2 d v ), one unit lower triangular solve applied to two right-hand sides (O ( C 3 ) O(C^3) O ( C 3 ) , in practice on 16 × 16 16\times16 16 × 16 blocks as in the delta-rule kernels), three products with the memory, V S VS V S , Q ^ S ⊤ \widehat{Q}S^\top Q S ⊤ and V ~ ⊤ K ~ \widetilde{V}^{\top}\widetilde{K} V ⊤ K (3 C d k d v 3Cd_kd_v 3 C d k d v ), and the intra-chunk product G q k V ~ G^{qk}\widetilde{V} G q k V (C 2 d v C^2d_v C 2 d v ). Over a sequence of length L L L this is O ( L C ( d k + d v ) + L d k d v ) O(LC(d_k+d_v)+Ld_kd_v) O ( L C ( d k + d v ) + L d k d v ) multiply–adds and O ( L C 2 ) O(LC^2) O ( L C 2 ) for the solves, the same order as Gated DeltaNet and KDA; relative to KDA the additional cost is the value Gram G v v G^{vv} G v v , and the coupling p p p replaces the product W S 0 WS_0 W S 0 of the delta rule at equal cost. The memory is materialized once per chunk, not per token.
Implementation notes. (i) The cumulative decays are kept in log space; the Gram entries use exp ( ℓ r − ℓ s ) \exp(\ell_r-\ell_s) exp ( ℓ r − ℓ s ) for s ≤ r s\le r s ≤ r only, so all decay factors lie in ( 0 , 1 ] (0,1] ( 0 , 1 ] and no quotient 1 / γ 1/\gamma 1/ γ is ever formed (the kernels use base-2 logarithms and exp 2 \exp_2 exp 2 ). (ii) The triangular solve is the precision-sensitive step and is carried out in IEEE fp32; the remaining products run on tensor cores. (iii) In the fused kernel the solve is not materialized as u ∘ u^\circ u ∘ and Λ β \Lambda_\beta Λ β but applied inside the scan as u = Λ ( c − β ⊙ p ) u=\Lambda(c-\beta\odot p) u = Λ ( c − β ⊙ p ) . (iv) Because p p p contracts over the full value dimension, programs that tile the memory over d v d_v d v exchange their partial row sums of ( V S ) ⊙ K ^ (VS)\odot\widehat{K} ( V S ) ⊙ K once per chunk; this is a synchronization, not a bandwidth cost (C C C floats per program against a d k × d v d_k\times d_v d k × d v memory). (v) Padding a sequence to a multiple of C C C with zero q , k , v , β = c = 0 q,k,v,\beta=c=0 q , k , v , β = c = 0 and log f = 0 \log f=0 log f = 0 neither writes nor decays. (vi) Queries are scaled by d k − 1 / 2 d_k^{-1/2} d k − 1/2 inside the kernel; the scale is immaterial after the RMSNorm of the block. In the supplied code, K ^ , K ˉ , K ˘ , Q ^ \widehat{K}, \bar{K}, \breve{K}, \widehat{Q} K , K ˉ , K ˘ , Q are k_up, k_dn, k_tail, q_up; G k k , G v v , G q k G^{kk}, G^{vv}, G^{qk} G k k , G v v , G q k are A_kk, A_vv, A_qk; Λ , u ∘ , Λ β \Lambda, u^\circ, \Lambda_\beta Λ , u ∘ , Λ β are Tinv, u_const, Tw; and p , u , V ~ p, u, \widetilde{V} p , u , V are p, u, uv.
Backward pass
The backward pass follows the forward decomposition chunk by chunk in reverse order. Let d H ∈ R C × d v \mathrm{d}H \in\mathbb{R}^{C \times d_v} d H ∈ R C × d v and d S C ∈ R d v × d k \mathrm{d}S_C \in\mathbb{R}^{d_v \times d_k} d S C ∈ R d v × d k be the upstream gradients of a chunk, the latter including the gradient d S 0 \mathrm{d}S_0 d S 0 returned by the following chunk. Write g : = c − β ⊙ p g := c - \beta\odot p g := c − β ⊙ p and Y 0 : = V S 0 ∈ R C × d k Y_0 := VS_0 \in\mathbb{R}^{C \times d_k} Y 0 := V S 0 ∈ R C × d k . With stril and tril the strictly lower and the lower triangular part, the vector–Jacobian products of (A158) and (A159) are
d S 0 = d S C Γ C + d H T Q ^ , d γ C = ( d S C ⊙ S 0 ) T 1 , d K ˘ = V ~ d S C , d V ~ = K ˘ d S C T + G q k T d H , d Q ^ = d H S 0 , d G q k = tril ( d H V ~ T ) . (172) \begin{aligned}
\mathrm{d}S_0 &= \mathrm{d}S_C\Gamma_C + \mathrm{d}H^{\mathsf{T}}\widehat{Q}, &
\mathrm{d}\gamma_C &= (\mathrm{d}S_C \odot S_0)^{\mathsf{T}}\mathbf{1}, &
\mathrm{d}\breve{K} &= \widetilde{V}\,\mathrm{d}S_C,\\
\mathrm{d}\widetilde{V} &= \breve{K}\,\mathrm{d}S_C^{\mathsf{T}} + G^{qk\mathsf{T}}\mathrm{d}H, &
\mathrm{d}\widehat{Q} &= \mathrm{d}H S_0, &
\mathrm{d}G^{qk} &= \operatorname{tril}(\mathrm{d}H\widetilde{V}^{\mathsf{T}}). \tag*{(172)}
\end{aligned} d S 0 d V = d S C Γ C + d H T Q , = K ˘ d S C T + G q k T d H , d γ C d Q = ( d S C ⊙ S 0 ) T 1 , = d H S 0 , d K ˘ d G q k = V d S C , = tril ( d H V T ) . (172)
those of the pseudo-values and of the solve (A156) are
d u = ( d V ~ ⊙ V ) 1 , d V + = diag ( u ) d V ~ , d Λ = d u g T , d g = Λ T d u , d c = d g , d β = − d g ⊙ p , d p = − d g ⊙ β , d N = − stril ( Λ T d Λ Λ T ) . (173) \begin{aligned}
\mathrm{d}u &= (\mathrm{d}\widetilde{V} \odot V)\mathbf{1}, &
\mathrm{d}V &\mathrel{+}= \operatorname{diag}(u)\,\mathrm{d}\widetilde{V}, &
\mathrm{d}\Lambda&= \mathrm{d}u\,g^{\mathsf{T}}, &
\mathrm{d}g &= \Lambda^{\mathsf{T}}\mathrm{d}u,\\
\mathrm{d}c &= \mathrm{d}g, &
\mathrm{d}\beta&= -\mathrm{d}g \odot p, &
\mathrm{d}p &= -\mathrm{d}g \odot\beta, &
\mathrm{d}N &= -\operatorname{stril}\left(\Lambda^{\mathsf{T}}\mathrm{d}\Lambda\Lambda^{\mathsf{T}}\right). \tag*{(173)}
\end{aligned} d u d c = ( d V ⊙ V ) 1 , = d g , d V d β + = diag ( u ) d V , = − d g ⊙ p , d Λ d p = d u g T , = − d g ⊙ β , d g d N = Λ T d u , = − stril ( Λ T d Λ Λ T ) . (173)
those of the scores (A157) and of the Gram matrices (A152), (A155) are
d V + = diag ( d p ) K ^ S 0 T + ( d G v v + d G v v T ) V , d S 0 + = V T diag ( d p ) K ^ , d β + = ( d N ⊙ G k k ⊙ G v v ) 1 , d G k k = diag ( β ) d N ⊙ G v v , d G v v = diag ( β ) d N ⊙ G k k , d K ^ = diag ( d p ) Y 0 + d G k k K ˉ , d K ˉ = d G k k T K ^ + d G q k T Q ^ , d Q ^ + = d G q k K ˉ . (174) \begin{aligned}
\mathrm{d}V &\mathrel{+}= \operatorname{diag}(\mathrm{d}p)\widehat{K}S_0^{\mathsf{T}} + (\mathrm{d}G^{vv}+\mathrm{d}G^{vv\mathsf{T}})V, &
\mathrm{d}S_0 &\mathrel{+}= V^{\mathsf{T}}\operatorname{diag}(\mathrm{d}p)\widehat{K},\\
\mathrm{d}\beta&\mathrel{+}= (\mathrm{d}N \odot G^{kk} \odot G^{vv})\mathbf{1}, &
\mathrm{d}G^{kk} &= \operatorname{diag}(\beta)\,\mathrm{d}N \odot G^{vv}, &
\mathrm{d}G^{vv} &= \operatorname{diag}(\beta)\,\mathrm{d}N \odot G^{kk},\\
\mathrm{d}\widehat{K} &= \operatorname{diag}(\mathrm{d}p)Y_0+\mathrm{d}G^{kk}\bar{K}, &
\mathrm{d}\bar{K} &= \mathrm{d}G^{kk\mathsf{T}}\widehat{K}+\mathrm{d}G^{qk\mathsf{T}}\widehat{Q}, &
\mathrm{d}\widehat{Q} &\mathrel{+}= \mathrm{d}G^{qk}\bar{K}. \tag*{(174)}
\end{aligned} d V d β d K + = diag ( d p ) K S 0 T + ( d G v v + d G v v T ) V , + = ( d N ⊙ G k k ⊙ G v v ) 1 , = diag ( d p ) Y 0 + d G k k K ˉ , d S 0 d G k k d K ˉ + = V T diag ( d p ) K , = diag ( β ) d N ⊙ G v v , = d G k k T K + d G q k T Q , d G v v d Q = diag ( β ) d N ⊙ G k k , + = d G q k K ˉ . (174)
and the decay-scaled copies (A151) and the cumulative sum (A150) give
d K = d K ^ ⊙ γ + d K ˉ / γ + d K ˘ ⊙ ( γ C / γ ) , d Q = d Q ^ ⊙ γ , d γ = d K ^ ⊙ K − d K ˉ ⊙ K / γ 2 + d Q ^ ⊙ Q − d K ˘ ⊙ K ⊙ γ C / γ 2 , d γ C + = ( d K ˘ ⊙ K / γ ) T 1 , d ℓ r = d γ r ⊙ γ r + 1 [ r = C ] d γ C ⊙ γ C , d log f i = ∑ r ≥ i d ℓ r (reverse cumulative sum). (175) \begin{aligned}
\mathrm{d}K &= \mathrm{d}\widehat{K}\odot\gamma+\mathrm{d}\bar{K}/\gamma+\mathrm{d}\breve{K}\odot(\gamma_C/\gamma), &
\mathrm{d}Q &= \mathrm{d}\widehat{Q}\odot\gamma,\\
\mathrm{d}\gamma&= \mathrm{d}\widehat{K}\odot K-\mathrm{d}\bar{K}\odot K/\gamma^2+\mathrm{d}\widehat{Q}\odot Q-\mathrm{d}\breve{K}\odot K\odot\gamma_C/\gamma^2, &
\mathrm{d}\gamma_C &\mathrel{+}= (\mathrm{d}\breve{K}\odot K/\gamma)^{\mathsf{T}}\mathbf{1},\\
\mathrm{d}\ell_r &= \mathrm{d}\gamma_r\odot\gamma_r+\mathbf{1}[r=C]\,\mathrm{d}\gamma_C\odot\gamma_C, &
\mathrm{d}\log f_i &= \sum_{r\geq i}\mathrm{d}\ell_r\quad\text{(reverse cumulative sum).} \tag*{(175)}
\end{aligned} d K d γ d ℓ r = d K ⊙ γ + d K ˉ / γ + d K ˘ ⊙ ( γ C / γ ) , = d K ⊙ K − d K ˉ ⊙ K / γ 2 + d Q ⊙ Q − d K ˘ ⊙ K ⊙ γ C / γ 2 , = d γ r ⊙ γ r + 1 [ r = C ] d γ C ⊙ γ C , d Q d γ C d log f i = d Q ⊙ γ , + = ( d K ˘ ⊙ K / γ ) T 1 , = r ≥ i ∑ d ℓ r (reverse cumulative sum). (175)
Here γ ∈ R C × d k \gamma\in\mathbb{R}^{C \times d_k} γ ∈ R C × d k stacks the γ r \gamma_r γ r and all products with it are element-wise. Two points differ from the delta-rule backward. The values enter the system matrix N N N through G v v G^{vv} G v v , so d V \mathrm{d}V d V receives a contribution through the inverse in (A163)–(A164), whereas in DeltaNet, Gated DeltaNet and KDA the values appear only in the right-hand side. And since the pseudo-values are u r v r u_rv_r u r v r , the gradient with respect to the write coefficients is the row-wise contraction d u = ( d V ~ ⊙ V ) 1 \mathrm{d}u=(\mathrm{d}\widetilde{V}\odot V)\mathbf{1} d u = ( d V ⊙ V ) 1 over the value dimension, mirroring the forward contraction that produces p p p . The gate factor diag ( β ) \operatorname{diag}(\beta) diag ( β ) sits inside the Hadamard product and must be kept inside the accumulation of d N \mathrm{d}N d N , as observed for the channel-wise gates of Gated DeltaNet-2 [24 ] .
Verification
We verified Theorem A11 and (A162)–(A165) numerically in double precision against the token-by-token recurrence (A149): outputs and final memory agree to machine precision (≤ 10 − 15 \leq10^{-15} ≤ 1 0 − 15 relative) for chunk lengths C ∈ { 1 , 8 , 16 , L } C \in\{1,8,16,L\} C ∈ { 1 , 8 , 16 , L } , sequence lengths that are and are not multiples of C C C , d k ≠ d v d_k \ne d_v d k = d v , nonzero initial memory, strong and weak decay, and β r ∈ ( 0 , 1 ) \beta_r \in(0,1) β r ∈ ( 0 , 1 ) as well as β r ∈ ( 0 , 2 ) \beta_r \in(0,2) β r ∈ ( 0 , 2 ) ; the operator identity (A161) holds against the explicitly formed product of d k d v × d k d v d_kd_v \times d_kd_v d k d v × d k d v factors; the no-decay reduction of N N N to the Gram of a r = k r ⊗ v r a_r=k_r \otimes v_r a r = k r ⊗ v r holds; and the gradients with respect to Q Q Q , K K K , V V V , β \beta β , c c c , log f \log f log f and S 0 S_0 S 0 agree with central finite differences for the recurrence to 10 − 9 10^{-9} 1 0 − 9 . The reference implementations in the supplementary material are checked against the same recurrence.