Introduction
Muon ([32 ] ) optimizes a matrix-shaped weight W t ∈ R m × n W_t \in\mathbb{R}^{m \times n} W t ∈ R m × n by orthogonalizing the momentum buffer instead of using it directly. From a stochastic gradient G t G_t G t ,
V t = β V t − 1 + ( 1 − β ) G t , D t = NS 5 ( V t ) ≈ ( V t V t T ) − 1 / 2 V t , W t + 1 = W t − η t D t , (1) \begin{aligned}
V_t &= \beta V_{t-1} + (1-\beta)G_t, \qquad D_t = \operatorname{NS}_5(V_t) \approx(V_tV_t^{\mathsf T})^{-1/2}V_t, \qquad W_{t+1} = W_t - \eta_tD_t, \tag*{(1)}
\end{aligned} V t = β V t − 1 + ( 1 − β ) G t , D t = NS 5 ( V t ) ≈ ( V t V t T ) − 1/2 V t , W t + 1 = W t − η t D t , (1)
with V 0 = G 0 V_0 = G_0 V 0 = G 0 and NS 5 \operatorname{NS}_5 NS 5 a fixed-point iteration for the orthogonal polar factor: if V t = P Σ Q T V_t = P\Sigma Q^{\mathsf T} V t = P Σ Q T is a thin SVD then D t = P Q T = ( V t V t T ) − 1 / 2 V t D_t = PQ^{\mathsf T} = (V_tV_t^{\mathsf T})^{-1/2}V_t D t = P Q T = ( V t V t T ) − 1/2 V t . A growing body of work studies why Muon works within general optimization frameworks: as momentum steepest descent under a chosen operator norm ((Bernstein & Newhouse, 2024b,a; Petrich et al., 2025; Lau et al., 2025)), as a non-Euclidean trust-region step ([34 ] ), under an isotropic curvature model [69 ] , with an implicit spectral-norm constraint on the iterates [10 ] , and with convergence guarantees under smoothness or Hölder smoothness, heavy-tailed noise, inexact orthogonalization, variance reduction, and nonsmooth or convex Lipschitz objectives [39 , 65 , 1 , 35 , 49 , 63 , 23 , 27 , 12 , 66 , 33 , 59 , 16 , 8 , 60 , 40 , 83 , 36 , 54 , 42 , 43 }). However, those works derive their analyses from abstract assumptions on the loss, such as smoothness or convexity, imposed uniformly over a problem class rather than from any specific model. The limitation of those works is that they cannot explain why Muon, a simple matrix preconditioner, works so well for neural networks specifically. The few theoretical results for specific models, such as matrix factorization and in-context learning with linear transformers {citep}] (Iiduka 2026)[45 ] , linear or bilinear classifiers on separable or imbalanced data {citep}[20 , 72 ] , and random-feature and transformer blocks at initialization {citep}[14 ] , sharpen the picture in particular regimes. Nevertheless, since the Hessian is the optimal preconditioner under a quadratic approximation of the loss, those previous works leave a basic structural question for preconditioning methods open: What makes the orthogonalization preconditioner suited to the Hessian of a neural network specifically? This paper gives a theoretical answer that Muon’s implicit preconditioner shares a similar row-block diagonally dominant heterogeneous structure to the neural-network Hessian described by {citet}Zhang et al. (2024a,b)`.
Preconditioner structure versus Hessian structure. Let us first take a view of the equivalent preconditioner form of orthogonalization. The orthogonalized update is D t = ( V t V t T ) − 1 / 2 V t D_t = (V_tV_t^{\mathsf{T}})^{-1/2}V_t D t = ( V t V t T ) − 1/2 V t , a left multiplication that mixes the rows of V t V_t V t . Stacking the rows of D t D_t D t and of V t V_t V t into vectors turns it into vec ( D t ) = ( ( V t V t T ) − 1 / 2 ⊗ I n ) vec ( V t ) \operatorname{vec}(D_t) = ((V_tV_t^{\mathsf{T}})^{-1/2} \otimes I_n)\operatorname{vec}(V_t) vec ( D t ) = (( V t V t T ) − 1/2 ⊗ I n ) vec ( V t ) , so Muon descends in the metric
H M u o n = P t 1 / 2 ⊗ I n , P t : = V t V t T ∈ R m × m , (2) H_{\mathrm{Muon}} = P_t^{1/2} \otimes I_n,\qquad P_t := V_tV_t^{\mathsf{T}} \in\mathbb{R}^{m \times m}, \tag*{(2)} H Muon = P t 1/2 ⊗ I n , P t := V t V t T ∈ R m × m , (2)
a Kronecker structure determined entirely by P t P_t P t . At initialization, V 0 = G 0 V_0 = G_0 V 0 = G 0 and hence P 0 = G 0 G 0 T P_0 = G_0G_0^{\mathsf{T}} P 0 = G 0 G 0 T . Previous work has shown that the neural-network Hessian has a special heterogeneous structure: for a weight W ∈ R m × n W \in\mathbb{R}^{m \times n} W ∈ R m × n , the m × m m \times m m × m blocks indexed by its rows (output neurons) are dominated by the diagonal ones, that is, the layer-wise Hessian is row-block diagonally dominant (Zhang et al., 2024a,b; Dong et al., 2025). Meanwhile, Deng et al. [15 ] point out empirically that G t G t T G_tG_t^{\mathsf{T}} G t G t T is diagonally dominant throughout training, and the larger the model, the stronger the dominance. When P t P_t P t is close to diagonal, H M u o n H_{\mathrm{Muon}} H Muon in (2) shares the row-block structure of the Hessian (Figure 1 ), so the question reduces to a clean property of the gradient.
Figure 1. Left: Hessian of a Transformer weight matrix at initialization, whose mass concentrates on the diagonal blocks. Right: the momentum self outer-product V t V t T V_tV_t^{\mathsf{T}} V t V t T of a Transformer weight matrix at initialization (log scale), whose mass concentrates on the diagonal.
Question. How does Neural Networks produce such a diagonally dominant V t V t T V_tV_t^{\mathsf{T}} V t V t T (equivalently, G t G t T G_tG_t^{\mathsf{T}} G t G t T at initialization)?
This paper answers this question for neural networks in an asymptotic regime, going a step further than the empirical observation of Deng et al. [15 ] by turning it into a quantitative asymptotic theoretical result at initialization. We prove that in four canonical neural-network models the diagonal of E [ G G T ] \mathbb{E}[GG^{\mathsf{T}}] E [ G G T ] outgrows the off-diagonal as the hidden dimensions of the weight matrices grow, at rates summarized in Table 1 . The overparameterization of the weight matrices is thus a key indicator of this diagonalization. Our contributions can be summarized as follows.
Table 1. Summary of the orders of E [ G G T ] \mathbb{E}[GG^{\mathsf{T}}] E [ G G T ] at a Gaussian initialization in the hidden dimensions of each model and, for attention, the token dimensions d 0 , d 2 d_0,d_2 d 0 , d 2 .
Hidden-dimension asymptotics theory for Muon’s preconditioner. For symmetric matrix factorization, deep linear networks, two-layer ReLU networks and single-layer linear attention, we prove that Muon’s implicit preconditioner becomes diagonally dominant as the hidden dimensions grow, with explicit rates (Theorems 1 to 4 and Table 1 ).
Empirical evaluation of the predicted scaling, from simulations to language models. Simulations of the four models reproduce the predicted rates (§ 5 ), and Qwen3.5, Llama-3.2 and Granite-4.0 at initialization show the dominance strengthening with model width (§ 6 ).
Related Work
Mechanistic analyses of Muon. Muon ([32 ] ) replaces the momentum of each weight matrix by its polar factor. This is the momentum form of steepest descent in the spectral norm, a view developed by Bernstein & Newhouse [6 ] . Work on why this update suits neural networks follows two routes. The first route models the curvature of the loss. Su [69 ] introduces an isotropic curvature model and shows that, under a growth condition on the curvature, the optimal update flattens the spectrum of the gradient, as orthogonalization does. Davis & Drusvyatskiy [14 ] give a layerwise condition, comparing the nuclear-to-Frobenius ratio of the gradient with the stable rank of the incoming activations, under which one spectral step decreases the loss more than one Euclidean step. Ma et al. [45 ] prove that a simplified Muon converges linearly on matrix factorization and on in-context learning with linear transformers, at a rate independent of the condition number and ahead of gradient descent and Adam. Nguyen et al. [50 ] recast Muon as a preconditioned gradient method and show, under a Kronecker-factored curvature model, that spectral flattening improves the convergence factor. Wang et al. [77 ] compare the one-step loss decrease of Muon and Adam under a second-order Taylor approximation and attribute Muon’s advantage to a smaller curvature penalty, and Wu et al. [81 ] probe the loss landscape along real training runs, estimating the loss-optimal step size along each singular direction of the momentum. Since the isotropic geometry behind Muon only approximates the true curvature, several curvature-aware variants have followed (Zhang et al., 2026c; Du & Su, [18 ] ; Che, [9 ] ; Zhang et al., 2026b). The second route studies the update itself. Beneventano et al. [4 ] characterize the polar factor as the entropy-maximizing bounded update that keeps the singular directions of the gradient and derive its spectral dynamics. Shumaylov et al. [67 ] find that random or inverted spectra train equally well, and Wang et al. [76 ] attribute Muon’s advantage on heavy-tailed data to its more isotropic update spectrum. Zhang (2026) separate spectral control from gauge equivariance, Eschenhagen et al. [19 ] relate Muon to Shampoo, and Zhang et al. (2026a) ask at which granularity attention weights should be orthogonalized. Both routes take a curvature model or a landscape as given. None of them compares the structure of Muon’s implicit preconditioner with that of the network’s Hessian, which is the question we study.
Block structure of the neural-network Hessian. The Hessian of a neural network is near block diagonal, with most of its mass in dense principal blocks. This observation goes back to Collobert [13 ] , and the same structure is assumed by the block-diagonal curvature approximations of Le Roux et al. [38 ] and Martens & Grosse [46 ] . Dong et al. [17 ] quantify the effect at random initialization: for linear and one-hidden-layer classifiers, the off-diagonal blocks vanish relative to the diagonal ones as the number of classes grows. The dense blocks within a layer are indexed by output neurons. This is the row-block structure we build on, and it is the partition by which Adam-mini assigns one learning rate per block to the value, projection and MLP weights (Zhang et al., 2024b). Beyond being block diagonal, the Hessian is heterogeneous across parameter blocks. Zhang et al. (2024a) show that in Transformers, unlike CNNs, the Hessian spectrum varies strongly from one weight matrix to another, so that no single learning rate serves all of them. Ormaniec et al. [52 ] derive the Hessian of a self-attention layer and trace this heterogeneity to the softmax and the query-key parameterization, Tomihari & Sato [70 ] show that gradient heterogeneity across blocks complements it, and Wang et al. [75 ] exploit the sharpness disparity between blocks to speed up pre-training. Within a block, Zhang et al. (2025) prove that under anisotropic inputs a layer-wise Kronecker-factored preconditioner recovers the feature-learning rate that SGD and Adam miss. Our work builds on the block-diagonal part of this picture: a preconditioner should be block diagonal in the row-block basis, and Muon’s ( V V T ) 1 / 2 ⊗ I (VV^{\mathsf{T}})^{1/2} \otimes I ( V V T ) 1/2 ⊗ I has this form precisely when V V T VV^{\mathsf{T}} V V T is diagonal. Further preconditioning methods are discussed in Appendix § 9 .
Preliminaries
Throughout, W ∈ R m × n W \in\mathbb{R}^{m \times n} W ∈ R m × n denotes a weight matrix, L \mathcal{L} L the population loss and G = ∂ L / ∂ W G = \partial\mathcal{L}/\partial W G = ∂ L / ∂ W the gradient of the loss with respect to that matrix. Our object of study is the m × m m \times m m × m matrix E [ G G T ] \mathbb{E}[GG^{\mathsf{T}}] E [ G G T ] at a Gaussian initialization. This matrix is the expectation of P 0 = V 0 V 0 T P_0 = V_0V_0^{\mathsf{T}} P 0 = V 0 V 0 T in (2) . We analyze it in four settings. § 3.1 introduces the four models, and § 3.2 states the initialization assumed for each. We write I d I_d I d for the d × d d \times d d × d identity, tr \operatorname{tr} tr for the trace and ∥ ⋅ ∥ F \lVert\cdot\rVert_F ∥ ⋅ ∥ F for the Frobenius norm. The ReLU is σ ( z ) = max ( z , 0 ) \sigma(z) = \max(z, 0) σ ( z ) = max ( z , 0 ) , with derivative σ ′ ( z ) = 1 [ z > 0 ] \sigma'(z) = 1[z > 0] σ ′ ( z ) = 1 [ z > 0 ] , and ⊗ \otimes ⊗ is the Kronecker product. Finally, a = Θ ( b ) a = \Theta(b) a = Θ ( b ) means that c b ≤ a ≤ C b c b \leq a \leq C b c b ≤ a ≤ C b for some constants 0 < c ≤ C 0 < c \leq C 0 < c ≤ C .
Setup
Symmetric matrix factorization. For positive integers d , k , r d, k, r d , k , r with k ≥ r k \geq r k ≥ r , a weight W ∈ R d × k W \in\mathbb{R}^{d \times k} W ∈ R d × k fits a fixed target M ⋆ ∈ R d × d M^\star\in\mathbb{R}^{d \times d} M ⋆ ∈ R d × d , symmetric positive semidefinite of rank r r r , under the over-parameterized loss
L ( W ) : = 1 4 ∥ W W T − M ⋆ ∥ F 2 , G : = ∂ L ∂ W = ( W W T − M ⋆ ) W ∈ R d × k . (3) \mathcal{L}(W) := \frac{1}{4}\lVert WW^{\mathsf{T}} - M^\star\rVert_F^2, \qquad G := \frac{\partial\mathcal{L}}{\partial W} = (WW^{\mathsf{T}} - M^\star)W \in\mathbb{R}^{d \times k}. \tag*{(3)} L ( W ) := 4 1 ∥ W W T − M ⋆ ∥ F 2 , G := ∂ W ∂ L = ( W W T − M ⋆ ) W ∈ R d × k . (3)
The hidden dimension is k k k .
Deep linear network. For L ≥ 1 L \geq1 L ≥ 1 and layer dimensions d 0 , … , d L d_0, \ldots, d_L d 0 , … , d L , weights W i ∈ R d i × d i − 1 W_i \in\mathbb{R}^{d_i \times d_{i-1}} W i ∈ R d i × d i − 1 fit a fixed linear target Φ ∈ R d L × d 0 \Phi\in\mathbb{R}^{d_L \times d_0} Φ ∈ R d L × d 0 on inputs x ∼ N ( 0 , I d 0 ) x \sim\mathcal{N}(0, I_{d_0}) x ∼ N ( 0 , I d 0 ) ,
f ( x ) = W L ⋯ W 1 x , L ( W 1 , … , W L ) : = 1 2 E x ∥ f ( x ) − Φ x ∥ 2 . (4) f(x) = W_L \cdots W_1x, \qquad\mathcal{L}(W_1, \ldots, W_L) := \frac{1}{2}\mathbb{E}_x \lVert f(x) - \Phi x \rVert^2. \tag*{(4)} f ( x ) = W L ⋯ W 1 x , L ( W 1 , … , W L ) := 2 1 E x ∥ f ( x ) − Φ x ∥ 2 . (4)
With B i : = W i − 1 ⋯ W 1 B_i := W_{i-1} \cdots W_1 B i := W i − 1 ⋯ W 1 (B 1 : = I B_1 := I B 1 := I ), A i : = W L ⋯ W i + 1 A_i := W_L \cdots W_{i+1} A i := W L ⋯ W i + 1 (A L : = I A_L := I A L := I ) and R : = W L ⋯ W 1 − Φ R := W_L \cdots W_1 - \Phi R := W L ⋯ W 1 − Φ , the layer-i i i gradient is
G i : = ∂ L ∂ W i = A i T R B i T ∈ R d i × d i − 1 . (5) G_i := \frac{\partial\mathcal{L}}{\partial W_i} = A_i^{\mathsf{T}} R B_i^{\mathsf{T}} \in\mathbb{R}^{d_i \times d_{i-1}}. \tag*{(5)} G i := ∂ W i ∂ L = A i T R B i T ∈ R d i × d i − 1 . (5)
The hidden dimensions are d 1 , … , d L − 1 d_1, \ldots, d_{L-1} d 1 , … , d L − 1 .
Two-layer ReLU network. For positive d 0 , d 1 , d 2 d_0,d_1,d_2 d 0 , d 1 , d 2 , weights W 1 ∈ R d 1 × d 0 W_1 \in\mathbb{R}^{d_1 \times d_0} W 1 ∈ R d 1 × d 0 and W 2 ∈ R d 2 × d 1 W_2 \in\mathbb{R}^{d_2 \times d_1} W 2 ∈ R d 2 × d 1 fit a fixed linear target Φ ∈ R d 2 × d 0 \Phi\in\mathbb{R}^{d_2 \times d_0} Φ ∈ R d 2 × d 0 on inputs x ∼ N ( 0 , I d 0 ) x \sim\mathcal{N}(0,I_{d_0}) x ∼ N ( 0 , I d 0 ) ,
f ( x ) = W 2 σ ( W 1 x ) , L ( W 1 , W 2 ) : = 1 2 E x ∥ f ( x ) − Φ x ∥ 2 . (6) f(x) = W_2\sigma(W_1x), \qquad\mathcal{L}(W_1,W_2) := \frac{1}{2}\mathbb{E}_x\left\|f(x)-\Phi x\right\|^2. \tag*{(6)} f ( x ) = W 2 σ ( W 1 x ) , L ( W 1 , W 2 ) := 2 1 E x ∥ f ( x ) − Φ x ∥ 2 . (6)
With h ( x ) : = σ ( W 1 x ) h(x) := \sigma(W_1x) h ( x ) := σ ( W 1 x ) the hidden features, r ( x ) : = f ( x ) − Φ x r(x) := f(x)-\Phi x r ( x ) := f ( x ) − Φ x the residual and D ( x ) : = diag ( σ ′ ( W 1 x ) ) D(x) := \operatorname{diag}(\sigma'(W_1x)) D ( x ) := diag ( σ ′ ( W 1 x )) , the gradients G i : = ∂ L / ∂ W i G_i := \partial\mathcal{L}/\partial W_i G i := ∂ L / ∂ W i are
G 1 = E x [ D ( x ) W 2 T r ( x ) x T ] ∈ R d 1 × d 0 , G 2 = E x [ r ( x ) h ( x ) T ] ∈ R d 2 × d 1 . (7) G_1 = \mathbb{E}_x\left[D(x)W_2^{\mathsf{T}}r(x)x^{\mathsf{T}}\right] \in\mathbb{R}^{d_1 \times d_0}, \qquad G_2 = \mathbb{E}_x\left[r(x)h(x)^{\mathsf{T}}\right] \in\mathbb{R}^{d_2 \times d_1}. \tag*{(7)} G 1 = E x [ D ( x ) W 2 T r ( x ) x T ] ∈ R d 1 × d 0 , G 2 = E x [ r ( x ) h ( x ) T ] ∈ R d 2 × d 1 . (7)
Let w a w_a w a denote the a a a -th row of W 1 W_1 W 1 . The hidden dimension is d 1 d_1 d 1 .
Single-layer linear attention. For positive T , d 0 , d k , d v , d 2 T,d_0,d_k,d_v,d_2 T , d 0 , d k , d v , d 2 , tokens X ∈ R T × d 0 X \in\mathbb{R}^{T \times d_0} X ∈ R T × d 0 (one per row) pass through weights W Q , W K ∈ R d k × d 0 W_Q,W_K \in\mathbb{R}^{d_k \times d_0} W Q , W K ∈ R d k × d 0 , W V ∈ R d v × d 0 W_V \in\mathbb{R}^{d_v \times d_0} W V ∈ R d v × d 0 , W O ∈ R d 2 × d v W_O \in\mathbb{R}^{d_2 \times d_v} W O ∈ R d 2 × d v , which fit a fixed linear target Φ ∈ R d 2 × d 0 \Phi\in\mathbb{R}^{d_2 \times d_0} Φ ∈ R d 2 × d 0 ,
A : = 1 d k X W Q T W K X T , f ( X ) = A X W V T W O T , A := \frac{1}{\sqrt{d_k}}XW_Q^{\mathsf{T}}W_KX^{\mathsf{T}}, \qquad f(X) = AXW_V^{\mathsf{T}}W_O^{\mathsf{T}}, A := d k 1 X W Q T W K X T , f ( X ) = A X W V T W O T ,
L ( W Q , W K , W V , W O ) : = 1 2 ∥ f ( X ) − X Φ T ∥ F 2 . (8) \mathcal{L}(W_Q,W_K,W_V,W_O) := \frac{1}{2}\left\|f(X)-X\Phi^{\mathsf{T}}\right\|_F^2. \tag*{(8)} L ( W Q , W K , W V , W O ) := 2 1 f ( X ) − X Φ T F 2 . (8)
Let C : = X T X C := X^{\mathsf{T}}X C := X T X , c p : = tr ( C p ) c_p := \operatorname{tr}(C^p) c p := tr ( C p ) , Z : = X W V T Z := XW_V^{\mathsf{T}} Z := X W V T and R : = f ( X ) − X Φ T R := f(X)-X\Phi^{\mathsf{T}} R := f ( X ) − X Φ T . The gradients G ∙ : = ∂ L / ∂ W ∙ G_\bullet:= \partial\mathcal{L}/\partial W_\bullet G ∙ := ∂ L / ∂ W ∙ , ∙ ∈ { Q , K , V , O } \bullet\in\{Q,K,V,O\} ∙ ∈ { Q , K , V , O } , are
G Q = 1 d k W K X T Z W O T R T X , G K = 1 d k W Q X T R W O Z T X , G_Q = \frac{1}{\sqrt{d_k}}W_KX^{\mathsf{T}}ZW_O^{\mathsf{T}}R^{\mathsf{T}}X, \qquad G_K = \frac{1}{\sqrt{d_k}}W_QX^{\mathsf{T}}RW_OZ^{\mathsf{T}}X, G Q = d k 1 W K X T Z W O T R T X , G K = d k 1 W Q X T R W O Z T X ,
G V = W O T R T A X , G O = R T A Z . (9) G_V = W_O^{\mathsf{T}}R^{\mathsf{T}}AX, \qquad G_O = R^{\mathsf{T}}AZ. \tag*{(9)} G V = W O T R T A X , G O = R T A Z . (9)
The hidden dimensions are d k d_k d k and d v d_v d v . In most practical language models the input and output dimensions of an attention block both equal the model width, d 0 = d 2 d_0=d_2 d 0 = d 2 ([73 , 62 , 7 , 71 , 22 , 47 , 61 , 28 ] ), and scaling a model up increases this width while d k , d v d_k,d_v d k , d v stay fixed, so for attention we also track d 0 d_0 d 0 and d 2 d_2 d 2 .
Assumptions
Each setting is analyzed at a standard Gaussian initialization, stated separately below.
Assumption 1 (Matrix factorization). The entries of W W W are i.i.d. N ( 0 , 1 ) \mathcal{N}(0,1) N ( 0 , 1 ) , and M ⋆ M^\star M ⋆ is deterministic and independent of W W W . Hence S : = W W T ∼ W d ( k , I d ) S := WW^{\mathsf{T}} \sim\mathcal{W}_{d}(k,I_d) S := W W T ∼ W d ( k , I d ) is Wishart with k k k degrees of freedom.
Assumption 2 (Deep linear network). The entries of W 1 , … , W L W_1,\ldots,W_L W 1 , … , W L are i.i.d. N ( 0 , 1 ) \mathcal{N}(0,1) N ( 0 , 1 ) ; the input x ∼ N ( 0 , I d 0 ) x \sim\mathcal{N}(0,I_{d_0}) x ∼ N ( 0 , I d 0 ) is integrated out in L \mathcal{L} L ; and Φ \Phi Φ is deterministic and independent of the weights. Expectations are over W 1 , … , W L W_1,\ldots,W_L W 1 , … , W L .
Assumption 3 (Two-layer ReLU network). The entries of W 1 W_1 W 1 and W 2 W_2 W 2 are i.i.d. N ( 0 , 1 ) \mathcal{N}(0,1) N ( 0 , 1 ) ; the input x ∼ N ( 0 , I d 0 ) x \sim\mathcal{N}(0,I_{d_0}) x ∼ N ( 0 , I d 0 ) is integrated out in L \mathcal{L} L ; and Φ \Phi Φ is deterministic and independent of W 1 , W 2 W_1,W_2 W 1 , W 2 . Expectations are over W 1 , W 2 W_1,W_2 W 1 , W 2 .
Assumption 4 (Linear attention). W Q , W K , W V , W O W_Q,W_K,W_V,W_O W Q , W K , W V , W O are mutually independent with i.i.d. N ( 0 , 1 ) \mathcal{N}(0,1) N ( 0 , 1 ) entries; the rows of X X X are i.i.d. N ( 0 , I d 0 ) \mathcal{N}(0,I_{d_0}) N ( 0 , I d 0 ) and independent of the weights, so that C ∼ W d 0 ( T , I d 0 ) C \sim\mathcal{W}_{d_0}(T,I_{d_0}) C ∼ W d 0 ( T , I d 0 ) is Wishart; and Φ \Phi Φ is deterministic. Expectations are over the weights and the tokens jointly.
Main Results
We state the closed form of E [ G G T ] \mathbb{E}[GG^{\mathsf{T}}] E [ G G T ] in each setup, discuss its diagonal dominance, and defer the proofs to Appendices B to E.
Symmetric matrix factorization
Theorem 1. Under Assumption 1 , with S : = W W ⊤ ∼ W d ( k , I d ) S := WW^{\top} \sim\mathcal{W}_{d}(k,I_{d}) S := W W ⊤ ∼ W d ( k , I d ) Wishart,
E [ G G ⊤ ] = α 3 I d − 2 α 2 M ⋆ + k ( M ⋆ ) 2 , (10) \mathbb{E}[GG^{\top}] = \alpha_{3}I_{d} - 2\alpha_{2}M^{\star} + k(M^{\star})^{2}, \tag*{(10)} E [ G G ⊤ ] = α 3 I d − 2 α 2 M ⋆ + k ( M ⋆ ) 2 , (10) where α 2 = k ( k + d + 1 ) \alpha_{2}=k(k+d+1) α 2 = k ( k + d + 1 ) and α 3 = k 3 + 3 ( d + 1 ) k 2 + ( d 2 + 3 d + 4 ) k \alpha_{3}=k^{3}+3(d+1)k^{2}+(d^{2}+3d+4)k α 3 = k 3 + 3 ( d + 1 ) k 2 + ( d 2 + 3 d + 4 ) k are the (isotropic) diagonal entries of E [ S 2 ] \mathbb{E}[S^{2}] E [ S 2 ] and E [ S 3 ] \mathbb{E}[S^{3}] E [ S 3 ] . Consequently, every diagonal entry α 3 − 2 α 2 M i i ⋆ + k ( M ⋆ ) i i 2 \alpha_{3}-2\alpha_{2}M^{\star}_{ii}+k(M^{\star})^{2}_{ii} α 3 − 2 α 2 M ii ⋆ + k ( M ⋆ ) ii 2 is Θ ( k 3 ) \Theta(k^{3}) Θ ( k 3 ) , and every off-diagonal entry − 2 α 2 M i j ⋆ + k ( M ⋆ ) i j 2 -2\alpha_{2}M^{\star}_{ij}+k(M^{\star})^{2}_{ij} − 2 α 2 M ij ⋆ + k ( M ⋆ ) ij 2 with M i j ⋆ ≠ 0 M^{\star}_{ij}\ne0 M ij ⋆ = 0 is Θ ( k 2 ) \Theta(k^{2}) Θ ( k 2 ) .
As the hidden dimension k k k grows, the ratio of diagonal to off-diagonal entries is therefore Θ ( k ) \Theta(k) Θ ( k ) , and E [ G G ⊤ ] \mathbb{E}[GG^{\top}] E [ G G ⊤ ] approaches the isotropic matrix α 3 I d \alpha_{3}I_{d} α 3 I d . At initialization the preconditioner thus reduces, to leading order, to a uniform rescaling of the momentum, and the target enters only through the lower-order terms − 2 α 2 M ⋆ + k ( M ⋆ ) 2 -2\alpha_{2}M^{\star}+k(M^{\star})^{2} − 2 α 2 M ⋆ + k ( M ⋆ ) 2 .
Deep linear network
Theorem 2. Under Assumption 2 , for every layer i ∈ { 1 , … , L } i\in\{1,\ldots,L\} i ∈ { 1 , … , L } ,
E [ G i G i ⊤ ] = V i s i I d i + T i , T i = { ( ∏ j = 1 L − 1 d j ) Φ Φ ⊤ , i = L , ( ∏ j ≠ i 1 ≤ j ≤ L − 1 d j ) ∥ Φ ∥ F 2 I d i , 1 ≤ i < L . (11) \mathbb{E}[G_{i}G_{i}^{\top}] = V_{i}s_{i}I_{d_{i}}+T_{i}, \qquad
T_{i} =
\begin{cases}
\left(\prod_{j=1}^{L-1}d_{j}\right)\Phi\Phi^{\top}, & i=L,\\
\left(\prod_{\substack{j\ne i\\1\le j\le L-1}}d_{j}\right)\lVert\Phi\rVert_{F}^{2}I_{d_{i}}, & 1\le i<L.
\end{cases} \tag*{(11)} E [ G i G i ⊤ ] = V i s i I d i + T i , T i = ⎩ ⎨ ⎧ ( ∏ j = 1 L − 1 d j ) Φ Φ ⊤ , ( ∏ j = i 1 ≤ j ≤ L − 1 d j ) ∥ Φ ∥ F 2 I d i , i = L , 1 ≤ i < L . (11) where V i , s i V_{i},s_{i} V i , s i are positive dimension-only scalars given by two-step recursions in ( d 0 , … , d L ) (d_{0},\ldots,d_{L}) ( d 0 , … , d L ) (Appendix Appendix § 11 ), with
V i s i = { Θ ( d i ∏ j = 1 j ≠ i L − 1 d j 2 ) , 1 ≤ i < L , Θ ( ∏ j = 1 L − 1 d j 2 ) , i = L . (12) V_{i}s_{i} =
\begin{cases}
\Theta\left(d_{i}\prod_{\substack{j=1\\j\ne i}}^{L-1}d_{j}^{2}\right), & 1\le i<L,\\
\Theta\left(\prod_{j=1}^{L-1}d_{j}^{2}\right), & i=L.
\end{cases} \tag*{(12)} V i s i = ⎩ ⎨ ⎧ Θ ( d i ∏ j = 1 j = i L − 1 d j 2 ) , Θ ( ∏ j = 1 L − 1 d j 2 ) , 1 ≤ i < L , i = L . (12) For a hidden layer (i < L i<L i < L ) the off-diagonal block T i T_{i} T i is a multiple of I d i I_{d_{i}} I d i , so E [ G i G i ⊤ ] \mathbb{E}[G_{i}G_{i}^{\top}] E [ G i G i ⊤ ] is exactly diagonal. At the output layer the diagonal noise V L s L = Θ ( ∏ j = 1 L − 1 d j 2 ) V_{L}s_{L}=\Theta\left(\prod_{j=1}^{L-1}d_{j}^{2}\right) V L s L = Θ ( ∏ j = 1 L − 1 d j 2 ) grows with the hidden dimensions while the off-diagonal stays proportional to ( ∏ j = 1 L − 1 d j ) ( Φ Φ ⊤ ) a b \left(\prod_{j=1}^{L-1}d_{j}\right)(\Phi\Phi^{\top})_{ab} ( ∏ j = 1 L − 1 d j ) ( Φ Φ ⊤ ) ab , so the ratio, of order ∏ j = 1 L − 1 d j \prod_{j=1}^{L-1}d_{j} ∏ j = 1 L − 1 d j , still diverges. For equal hidden dimensions d 1 = ⋯ = d L − 1 = d d_{1}=\cdots=d_{L-1}=d d 1 = ⋯ = d L − 1 = d the output-layer diagonal is Θ ( d 2 ( L − 1 ) ) \Theta(d^{2(L-1)}) Θ ( d 2 ( L − 1 ) ) against an off-diagonal of Θ ( d L − 1 ) \Theta(d^{L-1}) Θ ( d L − 1 ) .
Two-layer ReLU network
Let θ : = ∠ ( u , v ) \theta:=\angle(u,v) θ := ∠ ( u , v ) for independent u , v ∼ N ( 0 , I d 0 ) u,v\sim\mathcal{N}(0,I_{d_{0}}) u , v ∼ N ( 0 , I d 0 ) , and set the constants κ 1 : = E [ sin 2 θ + ( π − θ ) 2 4 π 2 + sin θ cos θ ( π − θ ) 2 π 2 ] \kappa_{1}:=\mathbb{E}\left[\frac{\sin^{2}\theta+(\pi-\theta)^{2}}{4\pi^{2}}+\frac{\sin\theta\cos\theta(\pi-\theta)}{2\pi^{2}}\right] κ 1 := E [ 4 π 2 s i n 2 θ + ( π − θ ) 2 + 2 π 2 s i n θ c o s θ ( π − θ ) ] and κ 2 : = E [ sin θ ( π − θ ) ] 2 π 2 \kappa_{2}:=\frac{\mathbb{E}[\sin\theta(\pi-\theta)]}{2\pi^{2}} κ 2 := 2 π 2 E [ s i n θ ( π − θ )] .
Theorem 3. Under Assumption 3 , with H : = E x [ h ( x ) h ( x ) ⊤ ] H:=\mathbb{E}_{x}[h(x)h(x)^{\top}] H := E x [ h ( x ) h ( x ) ⊤ ] and α : = E W 1 [ tr ( H 2 ) ] \alpha:=\mathbb{E}_{W_{1}}[\operatorname{tr}(H^{2})] α := E W 1 [ tr ( H 2 )] ,
E [ G 2 G 2 ⊤ ] = α I d 2 + d 1 4 Φ Φ ⊤ , α = Θ ( d 1 2 ) , \mathbb{E}[G_{2}G_{2}^{\top}] = \alpha I_{d_{2}}+\frac{d_{1}}{4}\Phi\Phi^{\top}, \qquad\alpha=\Theta(d_{1}^{2}), E [ G 2 G 2 ⊤ ] = α I d 2 + 4 d 1 Φ Φ ⊤ , α = Θ ( d 1 2 ) , and E [ G 1 G 1 ⊤ ] \mathbb{E}[G_{1}G_{1}^{\top}] E [ G 1 G 1 ⊤ ] is permutation-invariant in the hidden index with diagonal entries
E [ ( G 1 G 1 ⊤ ) a a ] = d 0 d 2 2 4 + d 0 d 2 2 [ ( d 1 − 1 ) κ 1 + 1 2 ] + ∥ Φ ∥ F 2 4 \mathbb{E}[(G_{1}G_{1}^{\top})_{aa}] = \frac{d_{0}d_{2}^{2}}{4}+d_{0}d_{2}^{2}\left[(d_{1}-1)\kappa_{1}+\frac{1}{2}\right]+\frac{\lVert\Phi\rVert_{F}^{2}}{4} E [( G 1 G 1 ⊤ ) aa ] = 4 d 0 d 2 2 + d 0 d 2 2 [ ( d 1 − 1 ) κ 1 + 2 1 ] + 4 ∥ Φ ∥ F 2 and off-diagonal entries E [ ( G 1 G 1 ⊤ ) a c ] = d 0 d 2 2 κ 2 + O ( d 2 d 0 ) \mathbb{E}[(G_{1}G_{1}^{\top})_{ac}]=d_{0}d_{2}^{2}\kappa_{2}+O(d_{2}\sqrt{d_{0}}) E [( G 1 G 1 ⊤ ) a c ] = d 0 d 2 2 κ 2 + O ( d 2 d 0 ) for a ≠ c a\ne c a = c . Consequently, for G 1 G_{1} G 1 the diagonal entries are Θ ( d 1 ) \Theta(d_{1}) Θ ( d 1 ) and the off-diagonal entries Θ ( 1 ) \Theta(1) Θ ( 1 ) , and for G 2 G_{2} G 2 they are Θ ( d 1 2 ) \Theta(d_{1}^{2}) Θ ( d 1 2 ) and Θ ( d 1 ) \Theta(d_{1}) Θ ( d 1 ) .
As the hidden dimension d 1 d_{1} d 1 grows, the ratio of diagonal to off-diagonal entries is therefore Θ ( d 1 ) \Theta(d_{1}) Θ ( d 1 ) for both layers. For G 2 G_{2} G 2 the off-diagonal d 1 4 ( Φ Φ ⊤ ) a c \frac{d_{1}}{4}(\Phi\Phi^{\top})_{ac} 4 d 1 ( Φ Φ ⊤ ) a c comes only from the target, as at the output layer of Theorem 2. For G 1 G_{1} G 1 the off-diagonal d 0 d 2 2 κ 2 d_{0}d_{2}^{2}\kappa_{2} d 0 d 2 2 κ 2 is target-free: it comes from a single term in the proof and traces back to ReLU’s lack of odd symmetry, which is why the dominance here is asymptotic rather than exact. Simulations match every formula (Appendix Appendix § 15 .§ 3 ).
Single-layer linear attention
Theorem 4. Under Assumption 4,
E [ G Q G Q T ] = c Q I d k , E [ G K G K T ] = c K I d k , E [ G V G V T ] = c V I d v , \mathbb{E}[G_QG_Q^{\mathsf{T}}] = c_Q I_{d_k}, \qquad\mathbb{E}[G_KG_K^{\mathsf{T}}] = c_K I_{d_k}, \qquad\mathbb{E}[G_VG_V^{\mathsf{T}}] = c_V I_{d_v}, E [ G Q G Q T ] = c Q I d k , E [ G K G K T ] = c K I d k , E [ G V G V T ] = c V I d v , E [ G O G O T ] = α ˉ O I d 2 + d v γ Φ Φ T , (13) \mathbb{E}[G_OG_O^{\mathsf{T}}] = \bar{\alpha}_O I_{d_2} + d_v \gamma\Phi\Phi^{\mathsf{T}}, \tag*{(13)} E [ G O G O T ] = α ˉ O I d 2 + d v γ Φ Φ T , (13) γ = 1 d 0 E [ c 2 2 ] = T [ ( T d 0 + 8 ) ( T + d 0 ) 2 + 2 ( T d 0 + 10 ) ( T + d 0 ) + 5 ( T d 0 + 4 ) ] , \gamma= \frac{1}{d_0}\mathbb{E}[c_2^2] = T\left[(Td_0+8)(T+d_0)^2 + 2(Td_0+10)(T+d_0) + 5(Td_0+4)\right], γ = d 0 1 E [ c 2 2 ] = T [ ( T d 0 + 8 ) ( T + d 0 ) 2 + 2 ( T d 0 + 10 ) ( T + d 0 ) + 5 ( T d 0 + 4 ) ] , where c Q , c K , c V , α ˉ O c_Q,c_K,c_V,\bar{\alpha}_O c Q , c K , c V , α ˉ O are explicit positive polynomials in the dimensions and ∥ Φ ∥ F 2 \lVert\Phi\rVert_F^2 ∥ Φ ∥ F 2 (Appendix § 13 ); for fixed Φ \Phi Φ ,
c Q ≍ c K = Θ ( d 2 2 d v 2 d 0 6 / d k ) , c V = Θ ( d 2 ( d 2 + d v ) d 0 6 ) , α ˉ O = Θ ( d 2 2 d 0 6 ) . c_Q \asymp c_K = \Theta\left(d_2^2d_v^2d_0^6/d_k\right), \qquad c_V = \Theta\left(d_2(d_2+d_v)d_0^6\right), \qquad\bar{\alpha}_O = \Theta\left(d_2^2d_0^6\right). c Q ≍ c K = Θ ( d 2 2 d v 2 d 0 6 / d k ) , c V = Θ ( d 2 ( d 2 + d v ) d 0 6 ) , α ˉ O = Θ ( d 2 2 d 0 6 ) . For W Q , W K , W V W_Q,W_K,W_V W Q , W K , W V the off-diagonal is exactly zero (Lemma 34 ). For W O W_O W O , as at the output layer of Theorem 2, the off-diagonal d v γ ( Φ Φ T ) a b = Θ ( d v d 0 3 ) d_v\gamma(\Phi\Phi^{\mathsf{T}})_{ab} = \Theta(d_vd_0^3) d v γ ( Φ Φ T ) ab = Θ ( d v d 0 3 ) comes only from the target while the diagonal α ˉ O = Θ ( d 2 2 d 0 6 ) \bar{\alpha}_O = \Theta(d_2^2d_0^6) α ˉ O = Θ ( d 2 2 d 0 6 ) is target-free noise, so the ratio Θ ( d v d 0 3 ) \Theta(d_vd_0^3) Θ ( d v d 0 3 ) diverges with d v d_v d v and, faster, with d 0 d_0 d 0 ; d k d_k d k does not affect it. With d 0 = d 2 d_0=d_2 d 0 = d 2 equal to the model width (§ 3.1 ), the ratio grows like the cube of the model width (§ 6 ).
Summary and proof sketch
In all four settings the diagonal of E [ G G T ] \mathbb{E}[GG^{\mathsf{T}}] E [ G G T ] outgrows the off-diagonal as the hidden dimension grows (Table 1), so P t = V t V t T P_t=V_tV_t^{\mathsf{T}} P t = V t V t T is asymptotically diagonal, H M u o n = P t 1 / 2 ⊗ I n H_{\mathrm{Muon}}=P_t^{1/2}\otimes I_n H Muon = P t 1/2 ⊗ I n is asymptotically block-diagonal in the Hessian’s row-block structure, and (Appendix § 14 ) orthogonalization is asymptotically a row-wise normalization.
The four proofs share a similar strategy. Each row of G G G is a random vector, and the ( a , b ) (a,b) ( a , b ) entry of G G T GG^{\mathsf{T}} G G T is the inner product of rows a a a and b b b , a measure of their correlation. A diagonal entry is the correlation of a row with itself, its squared norm, which adds up over all hidden coordinates. An off-diagonal entry is the correlation between two different rows, which are built from independent Gaussian vectors at initialization, so most of its terms cancel in expectation and only the lower-order terms carried by the fixed target remain. The gap between the orders of the diagonal and off-diagonal entries in Table 1 comes from this difference. In the proofs we compute these statistics of G G T GG^{\mathsf{T}} G G T exactly, using moments of the Wishart distribution for matrix factorization, deep linear networks and linear attention, and Stein’s lemma together with the arc-cosine kernel of Cho & Saul [11 ] for the ReLU network, whose nonlinearity is not a polynomial. Each closed form is reproduced by the simulations in § 5 .
Simulation of Theoretical Results
Simulation setup. We verify the dimension scalings of Theorems 1 to 4 under Assumptions 1 to 4 by varying the model dimensions. For each model, we vary the dimensions in the corresponding theoretical results and examine the resulting structure of the expected gradient self outer-product P : = E [ G G T ] ∈ R m × m P := \mathbb{E}[GG^{\mathsf{T}}] \in\mathbb{R}^{m \times m} P := E [ G G T ] ∈ R m × m , that is, the expectation of P 0 P_0 P 0 in (2) . For this matrix, we summarize the mean diagonal magnitude, the mean off-diagonal magnitude, and their ratio,
D ( P ) = 1 m ∑ i = 1 m ∣ P i i ∣ , OD ( P ) = 1 m ( m − 1 ) ∑ i ≠ j ∣ P i j ∣ , R ( P ) = D ( P ) OD ( P ) . (14) D(P) = \frac{1}{m}\sum_{i=1}^{m} |P_{ii}|,\qquad\operatorname{OD}(P) = \frac{1}{m(m-1)}\sum_{i\ne j}|P_{ij}|,\qquad R(P) = \frac{D(P)}{\operatorname{OD}(P)}. \tag*{(14)} D ( P ) = m 1 i = 1 ∑ m ∣ P ii ∣ , OD ( P ) = m ( m − 1 ) 1 i = j ∑ ∣ P ij ∣ , R ( P ) = OD ( P ) D ( P ) . (14)
We estimate P P P by E [ G G T ] ^ = 1 S ∑ s = 1 S G ( s ) G ( s ) T \widehat{\mathbb{E}[GG^{\mathsf{T}}]} = \frac{1}{S}\sum_{s=1}^{S}G^{(s)}G^{(s)\mathsf{T}} E [ G G T ] = S 1 ∑ s = 1 S G ( s ) G ( s ) T over S S S independent initializations; D ^ \widehat{D} D , OD ^ \widehat{\operatorname{OD}} OD , R ^ \widehat{R} R denote these estimates. Where the expected off-diagonal is exactly zero, the nonzero OD ^ \widehat{\operatorname{OD}} OD is Monte Carlo variance (details in Appendix § 15 ). We next apply this procedure to matrix factorization, deep linear networks, two-layer ReLU networks, and single-layer linear attention, varying in each case the dimensions governing the corresponding theoretical scaling.
Symmetric matrix factorization. We begin with symmetric matrix factorization, varying k ∈ { 5 , 10 , 20 , 40 , 80 , 160 , 320 , 640 } k \in\{5, 10, 20, 40, 80, 160, 320, 640\} k ∈ { 5 , 10 , 20 , 40 , 80 , 160 , 320 , 640 } with d = 25 d = 25 d = 25 and a fixed rank-5 target, using S = 10,000 S = 10{,}000 S = 10 , 000 initializations for each k k k . In Figure 3 (left), R ^ \widehat{R} R increases with k k k , in line with R = Θ ( k ) R = \Theta(k) R = Θ ( k ) from Theorem 1; D ^ \widehat{D} D and OD ^ \widehat{\operatorname{OD}} OD follow Θ ( k 3 ) \Theta(k^3) Θ ( k 3 ) and Θ ( k 2 ) \Theta(k^2) Θ ( k 2 ) (Appendix § 15 .§ 1 ).
Figure 3. Diagonal-dominance ratio R ^ \widehat{R} R for symmetric matrix factorization against k k k (left; d = 25 d = 25 d = 25 , rank-5 target) and for the two-layer ReLU network against d 1 d_1 d 1 (right; d 0 = 512 d_0 = 512 d 0 = 512 , d 2 = 12 d_2 = 12 d 2 = 12 ).
Deep linear network. We next consider a depth-4 linear network with baseline dimensions ( d 0 , … , d 4 ) = ( 512 , 80 , 80 , 80 , 512 ) (d_0,\ldots,d_4) = (512, 80, 80, 80, 512) ( d 0 , … , d 4 ) = ( 512 , 80 , 80 , 80 , 512 ) . We vary each hidden dimension d 1 , d 2 , d 3 d_1,d_2,d_3 d 1 , d 2 , d 3 in turn over the same grid while holding the remaining dimensions fixed, using S = 10,000 S = 10{,}000 S = 10 , 000 initializations per configuration. In Figure 4 , R ^ \widehat{R} R increases with every hidden dimension for all layers. For the output layer, Theorem 2 gives D = Θ ( ( d 1 d 2 d 3 ) 2 ) D = \Theta((d_1d_2d_3)^2) D = Θ (( d 1 d 2 d 3 ) 2 ) and OD = Θ ( d 1 d 2 d 3 ) \operatorname{OD} = \Theta(d_1d_2d_3) OD = Θ ( d 1 d 2 d 3 ) , and hence R = Θ ( d 1 d 2 d 3 ) R = \Theta(d_1d_2d_3) R = Θ ( d 1 d 2 d 3 ) ; accordingly, R ^ \widehat{R} R grows approximately linearly when any one hidden dimension is varied. For the hidden layers, the expected off-diagonal is zero (Appendix § 15 .§ 2 ).
Figure 4. Diagonal-dominance ratio R ^ \widehat{R} R of G 1 , … , G 4 G_1,\ldots,G_4 G 1 , … , G 4 for the depth-4 linear network when d 1 d_1 d 1 , d 2 d_2 d 2 or d 3 d_3 d 3 is varied (left to right). Each curve is normalized by its value at the baseline (d 1 = d 2 = d 3 = 80 d_1 = d_2 = d_3 = 80 d 1 = d 2 = d 3 = 80 ) to facilitate comparison across layers with different absolute scales.
Two-layer ReLU network. We then study the two-layer ReLU network, fixing d 0 = 512 d_0=512 d 0 = 512 and d 2 = 12 d_2=12 d 2 = 12 and varying the hidden width d 1 d_1 d 1 over the same grid. For each configuration, we use S = 10,000 S=10{,}000 S = 10 , 000 weight initializations and estimate the population gradient using 128,000 Gaussian inputs. In Figure 3 (right), R ^ \widehat{R} R increases with d 1 d_1 d 1 for both layers, matching R = Θ ( d 1 ) R=\Theta(d_1) R = Θ ( d 1 ) in Theorem 3 (Appendix § 15 .§ 3 ).
Single-layer linear attention. Finally, we consider single-layer linear attention with T = 32 T=32 T = 32 , d 0 = 512 d_0=512 d 0 = 512 , and d 2 = 12 d_2=12 d 2 = 12 . We examine four dimension sweeps: d k = d v d_k=d_v d k = d v jointly, d k d_k d k with d v = 80 d_v=80 d v = 80 , d v d_v d v with d k = 80 d_k=80 d k = 80 , and d 0 = d 2 d_0=d_2 d 0 = d 2 jointly. In Figure 5 , R ^ \widehat{R} R of G O G_O G O grows approximately linearly with d v d_v d v , is nearly unchanged with d k d_k d k , and grows superlinearly with d 0 d_0 d 0 , consistent with R = Θ ( d v d 0 3 ) R=\Theta(d_vd_0^3) R = Θ ( d v d 0 3 ) from Theorem 4. For G Q G_Q G Q , G K G_K G K , G V G_V G V , the expected off-diagonal entries are zero. Their finite R ^ \widehat{R} R values nevertheless follow the corresponding diagonal trends: they increase with d k = d v d_k=d_v d k = d v , increase with d v d_v d v , and grow strongly with d 0 = d 2 d_0=d_2 d 0 = d 2 , while G Q G_Q G Q and G K G_K G K decrease with d k d_k d k , consistent with D = Θ ( d 0 2 d v 2 d 0 6 / d k ) D=\Theta(d_0^2d_v^2d_0^6/d_k) D = Θ ( d 0 2 d v 2 d 0 6 / d k ) (Appendix § 15 .§ 4 ).
Figure 5. Diagonal-dominance ratio R ^ \widehat{R} R of the four attention projections when d k = d v d_k=d_v d k = d v , d k d_k d k , d v d_v d v , and d 0 = d 2 d_0=d_2 d 0 = d 2 are varied (left to right). Each curve is normalized by its value at the baseline (first three: T = 32 T=32 T = 32 , d 0 = 512 d_0=512 d 0 = 512 , d k = d v = 80 d_k=d_v=80 d k = d v = 80 , d 2 = 12 d_2=12 d 2 = 12 ; fourth: T = 16 T=16 T = 16 , d 0 = d 2 = 32 d_0=d_2=32 d 0 = d 2 = 32 , d k = d v = 20 d_k=d_v=20 d k = d v = 20 ) to facilitate comparison across projections with different absolute scales.
Empirical Evaluation on Large Language Models
We next test whether the diagonal dominance of Theorem 4 persists in Transformer-based language models at random initialization: Qwen3.5-0.8B-Base ([61 ] ), Granite-4.0-1B-Base ([28 ] ) and Llama-3.2-1B ([47 ] ). Figure 2 (right) shows the empirical E [ G G T ] \mathbb{E}[\mathbf{G}\mathbf{G}^{\mathsf{T}}] E [ G G T ] of the four projections in Qwen3.5, whose mass concentrates on the diagonal. For the first head of the first eligible attention block of each model we vary the model width d 0 = d 2 d_0=d_2 d 0 = d 2 at fixed head dimension d k = d v d_k=d_v d k = d v , using 25 random initializations and 64 fixed FineWeb-Edu ([55 ] ) sequences per configuration (details in Appendix § 16 .§ 2 ). Figure 6 shows R ^ \widehat{R} R increasing with model width for all four projections in all three models, as the Θ ( d v d 0 3 ) \Theta(d_vd_0^3) Θ ( d v d 0 3 ) ratio of Theorem 4 predicts.
Figure 2. Empirical E ^ [ G G ⊤ ] \widehat{\mathbb{E}}[GG^{\top}] E [ G G ⊤ ] (log scale) of the query, key, value and output projections at initialization, for single-layer linear attention (left, details in Appendix § 15 .§ 4 ) and the first attention block of Qwen3.5 (right, details in Appendix § 16 .§ 1 ). In both cases the mass concentrates on the diagonal. Each heatmap is normalized by its empirical mean diagonal magnitude.
Figure 6. Model width scaling of R ^ \widehat{R} R for the query, key, value, and output projections in the first head of the first eligible attention block of the three language models at random initialization.
Conclusion and Limitations
Conclusion. This work provides an asymptotic theory that shows Muon’s implicit preconditioner matches the row-block diagonally dominant structure of the neural-network Hessian, because E [ G G T ] \mathbb{E}[\mathbf{G}\mathbf{G}^{\mathsf{T}}] E [ G G T ] itself becomes diagonally dominant as the weight matrices are overparameterized, at the explicit rates of Table 1 . Specifically, in matrix factorization the isotropic Θ ( k 3 ) \Theta(k^3) Θ ( k 3 ) diagonal outgrows the target-driven Θ ( k 2 ) \Theta(k^2) Θ ( k 2 ) off-diagonal. In deep linear networks the hidden layers are exactly diagonal, and the ratio at the output layer grows with the product of all hidden dimensions. In the two-layer ReLU network the nonlinearity leaves an off-diagonal of constant order, yet the ratio still grows linearly with the hidden dimension. In linear attention the query, key and value weights are exactly diagonal. For the output weight the ratio Θ ( d v d 0 3 ) \Theta(d_vd_0^3) Θ ( d v d 0 3 ) grows with the head dimension and grows as the cube of the model width, so the dominance strengthens as models get wider. Our experiments on language models of increasing width match this intuition.
Limitations. Our theory is stated at initialization and in expectation: it does not characterize the training dynamics of V t V t T V_tV_t^{\mathsf{T}} V t V t T , although the empirical evidence of Deng et al. [15 ] indicates that the dominance persists during training. It is also worth studying whether this dominance reflects a degenerate behavior of the model that emerges at large scale. We leave these questions to future work.
Ethics statement
This work is primarily theoretical: it characterizes the expected gradient self outer-product of standard neural-network models at initialization. The experiments consist of synthetic simulations and training runs of publicly available language models on standard public corpora; they involve no human subjects, no personal or sensitive data, and no newly collected datasets. We do not foresee ethical concerns or direct negative societal impacts arising from this work.
Reproducibility statement
All results are stated under explicit assumptions (§ 3.2 ) and proved in full in the appendix: Appendix § 10 (symmetric matrix factorization), Appendix § 11 (deep linear networks), Appendix § 12 (two-layer ReLU networks) and Appendix § 13 (single-layer linear attention). Each appendix restates the corresponding theorem together with its standing assumption and all supporting lemmas, and the closed-form Wishart trace moments used for the attention layer are listed explicitly in Lemma 33 . The numerical checks (Appendix § 15 ) specify every dimension, sample size and initialization used, and the language-model experiments report the model sizes, datasets, optimizer settings and number of training steps; the code for both the simulations and the language-model runs is provided at the anonymous repository https://anonymous.4open.science/r/orthogonalization-hessian-CB2C/ .
References
Kang An, Yuxing Liu, Rui Pan, Yi Ren, Shiqian Ma, Donald Goldfarb, and Tong Zhang. ASGO: Adaptive structured gradient optimization. arXiv preprint arXiv:2503.20762 , 2025.
Rohan Anil, Vineet Gupta, Tomer Koren, and Yoram Singer. Memory-efficient adaptive optimization. In Advances in Neural Information Processing Systems , 2019.
Rohan Anil, Vineet Gupta, Tomer Koren, Kevin Regan, and Yoram Singer. Scalable second order optimization for deep learning. arXiv preprint arXiv:2002.09018 , 2020.
Pierfrancesco Beneventano, Mahmoud Abdelmoneum, and Tomaso Poggio. The spectral dynamics and noise geometry of Muon. arXiv preprint arXiv:2606.08388 , 2026.
Extended Related Work
Mechanistic analyses of Muon and the heterogeneity of the neural-network Hessian are discussed in § 2 ; this appendix covers the remaining line of work.
Preconditioning methods for neural-network optimization. Muon belongs to the family of optimizers defined by a linear minimization oracle (LMO) over a norm ball ([57 , 10 , 34 ] ), which also contains Scion and Gluon, unifies sign, spectral and Schatten-p updates ([37 , 1 , 53 , 56 ] ), and comes with convergence theory for the idealized and the practical algorithm ([39 , 65 , 35 ] ). Two-sided Kronecker preconditioners such as Shampoo and SOAP whiten both the row and the column side of a matrix gradient ([25 , 48 , 74 ] ); Anil et al. [3 ] make this second-order preconditioning practical at the scale of modern deep learning, and Anil et al. [2 ] go in the opposite direction, compressing the state of adaptive optimizers to a memory-efficient factored form. A parallel line keeps only the row side and drops optimizer state by preconditioning the gradient row by row: SWAN, gradient multi-normalization, a minimalist optimizer and SRON replace Adam’s state with a row-wise ℓ 2 \ell_2 ℓ 2 normalization ([44 , 64 , 21 , 78 Gu & Xie] (2026); Xu et al. [82 ] study its manifold-optimization and width-scaling sides; Pethick et al. [57 ] reach it from norm-constrained steepest descent; Lion et al. [41 ] make the row magnitudes an explicit optimizer variable, which Hübler et al. [26 ] show implicitly decays the angular step size; and Deng et al. [15 ] show that row-momentum normalization matches Muon at scale. Our analysis explains the common thread: once V V T VV^{\mathsf{T}} V V T is diagonally dominant, row normalization and orthogonalization are the same operation (Appendix § 14 ).
Symmetric Matrix Factorization
Setup
Let d d d , k k k , r r r be positive integers with k ≥ r k \ge r k ≥ r . The over-parameterized symmetric matrix-factorization loss is
L ( W ) : = 1 4 ∥ W W T − M ⋆ ∥ F 2 , W ∈ R d × k , (15) \mathcal{L}(\mathbf{W}) := \frac{1}{4}\left\lVert\mathbf{W}\mathbf{W}^{\mathsf{T}} - M^{\star}\right\rVert_{\mathrm{F}}^{2}, \qquad\mathbf{W} \in\mathbb{R}^{d \times k}, \tag*{(15)} L ( W ) := 4 1 W W T − M ⋆ F 2 , W ∈ R d × k , (15)
with target M ⋆ ∈ R d × d M^{\star} \in\mathbb{R}^{d \times d} M ⋆ ∈ R d × d symmetric positive semidefinite of rank r r r . A direct computation gives the gradient
G : = ∇ L ( W ) = ( W W T − M ⋆ ) W ∈ R d × k , (16) G := \nabla\mathcal{L}(\mathbf{W}) = \left(\mathbf{W}\mathbf{W}^{\mathsf{T}} - M^{\star}\right)\mathbf{W} \in\mathbb{R}^{d \times k}, \tag*{(16)} G := ∇ L ( W ) = ( W W T − M ⋆ ) W ∈ R d × k , (16)
and the object of interest is the d × d d \times d d × d matrix E [ G G T ] \mathbb{E}[GG^{\mathsf{T}}] E [ G G T ] .
Standing assumption
Assumption 5. The entries W i j W_{ij} W ij are i.i.d. N ( 0 , 1 ) \mathcal{N}(0,1) N ( 0 , 1 ) , and M ⋆ M^{\star} M ⋆ is deterministic and independent of W W W . Hence S : = W W T S := WW^{\mathsf{T}} S := W W T is Wishart with k k k degrees of freedom and identity scale, S ∼ W d ( k , I d ) S \sim\mathcal{W}_{d}(k,I_{d}) S ∼ W d ( k , I d ) .
Main results restated
Theorem (Theorem 1, restated). Under Assumption 5,
E [ G G T ] = α 3 ( d , k ) I d − 2 α 2 ( d , k ) M ⋆ + k ( M ⋆ ) 2 , (17) \mathbb{E}[GG^{\mathsf{T}}] = \alpha_{3}(d,k)I_{d} - 2\alpha_{2}(d,k)M^{\star} + k(M^{\star})^{2}, \tag*{(17)} E [ G G T ] = α 3 ( d , k ) I d − 2 α 2 ( d , k ) M ⋆ + k ( M ⋆ ) 2 , (17) where the scalars are the diagonal entries of the first three Wishart moments,
α 1 ( d , k ) = k , (18) \alpha_{1}(d,k) = k, \tag*{(18)} α 1 ( d , k ) = k , (18) α 2 ( d , k ) = k ( k + d + 1 ) , (19) \alpha_{2}(d,k) = k(k+d+1), \tag*{(19)} α 2 ( d , k ) = k ( k + d + 1 ) , (19) α 3 ( d , k ) = k 3 + 3 ( d + 1 ) k 2 + ( d 2 + 3 d + 4 ) k . (20) \alpha_{3}(d,k) = k^{3} + 3(d+1)k^{2} + (d^{2}+3d+4)k. \tag*{(20)} α 3 ( d , k ) = k 3 + 3 ( d + 1 ) k 2 + ( d 2 + 3 d + 4 ) k . (20)
Corollary 1. Under Assumption 5, E [ G G T ] \mathbb{E}[GG^{\mathsf{T}}] E [ G G T ] is invariant under any orthogonal change of basis fixing M ⋆ M^{\star} M ⋆ , with
E [ ( G G T ) i i ] = α 3 − 2 α 2 M i i ⋆ + k ( M ⋆ ) i i 2 , (21) \mathbb{E}[(GG^{\mathsf{T}})_{ii}] = \alpha_{3} - 2\alpha_{2}M^{\star}_{ii} + k(M^{\star})^{2}_{ii}, \tag*{(21)} E [( G G T ) ii ] = α 3 − 2 α 2 M ii ⋆ + k ( M ⋆ ) ii 2 , (21) E [ ( G G T ) i j ] = − 2 α 2 M i j ⋆ + k ( M ⋆ ) i j 2 ( i ≠ j ) . (22) \mathbb{E}[(GG^{\mathsf{T}})_{ij}] = -2\alpha_{2}M^{\star}_{ij} + k(M^{\star})^{2}_{ij} \qquad(i \ne j). \tag*{(22)} E [( G G T ) ij ] = − 2 α 2 M ij ⋆ + k ( M ⋆ ) ij 2 ( i = j ) . (22) As k → ∞ k \to\infty k → ∞ , E [ ( G G T ) i i ] = Θ ( k 3 ) \mathbb{E}[(GG^{\mathsf{T}})_{ii}] = \Theta(k^{3}) E [( G G T ) ii ] = Θ ( k 3 ) and E [ ( G G T ) i j ] = Θ ( k 2 ) \mathbb{E}[(GG^{\mathsf{T}})_{ij}] = \Theta(k^{2}) E [( G G T ) ij ] = Θ ( k 2 ) whenever M i j ⋆ ≠ 0 M^{\star}_{ij} \ne0 M ij ⋆ = 0 , so the dominance ratio is Θ ( k / M i j ⋆ ) \Theta(k/M^{\star}_{ij}) Θ ( k / M ij ⋆ ) .
Preliminary lemmas
Lemma 1. Let X ∈ R m × n X \in\mathbb{R}^{m \times n} X ∈ R m × n have i.i.d. N ( 0 , 1 ) \mathcal{N}(0,1) N ( 0 , 1 ) entries. Then for deterministic M ∈ R n × n M \in\mathbb{R}^{n \times n} M ∈ R n × n and N ∈ R m × m N \in\mathbb{R}^{m \times m} N ∈ R m × m ,
E [ X M X T ] = tr ( M ) I m , E [ X T N X ] = tr ( N ) I n . (23) \mathbb{E}[XMX^{\mathsf{T}}] = \operatorname{tr}(M)I_{m}, \qquad\mathbb{E}[X^{\mathsf{T}}NX] = \operatorname{tr}(N)I_{n}. \tag*{(23)} E [ X M X T ] = tr ( M ) I m , E [ X T N X ] = tr ( N ) I n . (23)
Proof. Using E [ X a p X b q ] = δ a b δ p q \mathbb{E}[X_{ap}X_{bq}] = \delta_{ab}\delta_{pq} E [ X a p X b q ] = δ ab δ pq ,
E [ ( X M X T ) a b ] = ∑ p , q M p q δ a b δ p q = δ a b tr ( M ) , \mathbb{E}[(XMX^{\mathsf{T}})_{ab}] = \sum_{p,q} M_{pq}\delta_{ab}\delta_{pq} = \delta_{ab}\operatorname{tr}(M), E [( X M X T ) ab ] = p , q ∑ M pq δ ab δ pq = δ ab tr ( M ) ,
which is the first identity; the second follows by applying it to X T X^{\mathsf{T}} X T (entries i.i.d. N ( 0 , 1 ) \mathcal{N}(0,1) N ( 0 , 1 ) ).
Lemma 2. Let S = W W ⊤ S = WW^\top S = W W ⊤ with W W W as in Assumption 5. For every m ∈ N m \in\mathbb{N} m ∈ N there is a scalar α m ( d , k ) \alpha_m(d,k) α m ( d , k ) with
E [ S m ] = α m ( d , k ) I d , α m ( d , k ) = 1 d E [ tr ( S m ) ] . (24) \mathbb{E}[S^m] = \alpha_m(d,k) I_d,\qquad\alpha_m(d,k) = \frac{1}{d}\mathbb{E}[\operatorname{tr}(S^m)]. \tag*{(24)} E [ S m ] = α m ( d , k ) I d , α m ( d , k ) = d 1 E [ tr ( S m )] . (24)
Proof. For Q ∈ O ( d ) Q \in O(d) Q ∈ O ( d ) , Q W = d W QW \overset{d}{=} W Q W = d W , hence Q S m Q ⊤ = d S m QS^mQ^\top\overset{d}{=} S^m Q S m Q ⊤ = d S m and
E [ S m ] = Q E [ S m ] Q ⊤ for all Q ∈ O ( d ) . \mathbb{E}[S^m] = Q\mathbb{E}[S^m]Q^\top\qquad\text{for all } Q \in O(d). E [ S m ] = Q E [ S m ] Q ⊤ for all Q ∈ O ( d ) .
By Schur’s lemma E [ S m ] = α m I d \mathbb{E}[S^m] = \alpha_m I_d E [ S m ] = α m I d ; taking the trace gives d α m = E [ tr ( S m ) ] d\alpha_m = \mathbb{E}[\operatorname{tr}(S^m)] d α m = E [ tr ( S m )] .
Lemma 3 (Wick/Isserlis). Let X 1 , … , X 2 n X_1,\ldots,X_{2n} X 1 , … , X 2 n be jointly centered Gaussian. Then
E [ X 1 ⋯ X 2 n ] = ∑ P ∏ { i , j } ∈ P E [ X i X j ] , (25) \mathbb{E}[X_1\cdots X_{2n}] = \sum_{P}\prod_{\{i,j\}\in P}\mathbb{E}[X_iX_j], \tag*{(25)} E [ X 1 ⋯ X 2 n ] = P ∑ { i , j } ∈ P ∏ E [ X i X j ] , (25) the sum over all pairings P P P of { 1 , … , 2 n } \{1,\ldots,2n\} { 1 , … , 2 n } ; odd moments vanish ([30 , 79 ] ).
Proof. Differentiate E [ exp ( t ⊤ X ) ] = exp ( 1 2 t ⊤ Σ t ) \mathbb{E}[\exp(t^\top X)] = \exp(\frac{1}{2}t^\top\Sigma t) E [ exp ( t ⊤ X )] = exp ( 2 1 t ⊤ Σ t ) in t 1 , … , t 2 n t_1,\ldots,t_{2n} t 1 , … , t 2 n at t = 0 t=0 t = 0 ; cf. Janson [31 ] .
Proofs
Lemma 4. With S : = W W ⊤ S := WW^\top S := W W ⊤ and B : = S − M ⋆ B := S - M^\star B := S − M ⋆ ,
G G ⊤ = B S B = S 3 − S 2 M ⋆ − M ⋆ S 2 + M ⋆ S M ⋆ . (26) GG^\top= BSB = S^3 - S^2M^\star- M^\star S^2 + M^\star S M^\star. \tag*{(26)} G G ⊤ = B S B = S 3 − S 2 M ⋆ − M ⋆ S 2 + M ⋆ S M ⋆ . (26)
Proof. Since B = B ⊤ B = B^\top B = B ⊤ ,
G G ⊤ = B W ( B W ) ⊤ = B W W ⊤ B ⊤ = B S B , GG^\top= BW(BW)^\top= BWW^\top B^\top= BSB, G G ⊤ = B W ( B W ) ⊤ = B W W ⊤ B ⊤ = B S B ,
and substituting B = S − M ⋆ B = S - M^\star B = S − M ⋆ and expanding gives (26) .
Lemma 5. Under Assumption 5, E [ S ] = k I d \mathbb{E}[S] = k I_d E [ S ] = k I d , hence α 1 ( d , k ) = k \alpha_1(d,k) = k α 1 ( d , k ) = k .
Proof. Let w a w_a w a denote the a a a -th column of W W W . Then
E [ S ] = E [ ∑ a = 1 k w a w a ⊤ ] = ∑ a = 1 k E [ w a w a ⊤ ] = k I d . \mathbb{E}[S] = \mathbb{E}\left[\sum_{a=1}^{k} w_aw_a^\top\right] = \sum_{a=1}^{k}\mathbb{E}[w_aw_a^\top] = kI_d. E [ S ] = E [ a = 1 ∑ k w a w a ⊤ ] = a = 1 ∑ k E [ w a w a ⊤ ] = k I d .
Lemma 6. Under Assumption 5, E [ S 2 ] = k ( k + d + 1 ) I d \mathbb{E}[S^2] = k(k+d+1)I_d E [ S 2 ] = k ( k + d + 1 ) I d , hence α 2 ( d , k ) = k ( k + d + 1 ) \alpha_2(d,k) = k(k+d+1) α 2 ( d , k ) = k ( k + d + 1 ) .
Proof. By Lemma 2, E [ S 2 ] = α 2 I d \mathbb{E}[S^2] = \alpha_2I_d E [ S 2 ] = α 2 I d . With columns w a ∼ i i d N ( 0 , I d ) w_a \overset{\mathrm{iid}}{\sim} \mathcal{N}(0,I_d) w a ∼ iid N ( 0 , I d ) ,
S 2 = ∑ a , b = 1 k ⟨ w a , w b ⟩ w a w b ⊤ . S^2 = \sum_{a,b=1}^{k} \langle w_a,w_b\rangle w_aw_b^\top. S 2 = a , b = 1 ∑ k ⟨ w a , w b ⟩ w a w b ⊤ .
Split into the k k k terms with a = b a=b a = b and the k ( k − 1 ) k(k-1) k ( k − 1 ) with a ≠ b a\ne b a = b . By Lemma 3 and independence,
E [ ∥ w a ∥ 2 w a w a ⊤ ] = ( d + 2 ) I d , \mathbb{E}[\lVert w_a\rVert^2w_aw_a^\top] = (d+2)I_d, E [∥ w a ∥ 2 w a w a ⊤ ] = ( d + 2 ) I d ,
E [ ⟨ w a , w b ⟩ w a w b ⊤ ] = ∑ p E [ w a , p w a ] E [ w b , p w b ⊤ ] = ∑ p e p e p ⊤ = I d . \mathbb{E}[\langle w_a,w_b\rangle w_aw_b^\top] = \sum_p\mathbb{E}[w_{a,p}w_a]\mathbb{E}[w_{b,p}w_b^\top] = \sum_p e_pe_p^\top= I_d. E [⟨ w a , w b ⟩ w a w b ⊤ ] = p ∑ E [ w a , p w a ] E [ w b , p w b ⊤ ] = p ∑ e p e p ⊤ = I d .
Summing, E [ S 2 ] = k ( d + 2 ) I d + k ( k − 1 ) I d = k ( k + d + 1 ) I d \mathbb{E}[S^2] = k(d+2)I_d + k(k-1)I_d = k(k+d+1)I_d E [ S 2 ] = k ( d + 2 ) I d + k ( k − 1 ) I d = k ( k + d + 1 ) I d .
Lemma 7. Under Assumption 5, E [ S 3 ] = [ k 3 + 3 ( d + 1 ) k 2 + ( d 2 + 3 d + 4 ) k ] I d \mathbb{E}[S^3] = [k^3 + 3(d+1)k^2 + (d^2+3d+4)k]I_d E [ S 3 ] = [ k 3 + 3 ( d + 1 ) k 2 + ( d 2 + 3 d + 4 ) k ] I d , hence α 3 ( d , k ) \alpha_3(d,k) α 3 ( d , k ) as in (20) .
Proof. By Lemma 2, α 3 = 1 d E [ tr ( S 3 ) ] \alpha_3 = \frac{1}{d}\mathbb{E}[\operatorname{tr}(S^3)] α 3 = d 1 E [ tr ( S 3 )] , and with S = ∑ a w a w a ⊤ S = \sum_a w_a w_a^\top S = ∑ a w a w a ⊤ and cyclicity of the trace,
tr ( S 3 ) = ∑ a , b , c ⟨ w a , w b ⟩ ⟨ w b , w c ⟩ ⟨ w c , w a ⟩ . \operatorname{tr}(S^3) = \sum_{a,b,c} \langle w_a,w_b\rangle\langle w_b,w_c\rangle\langle w_c,w_a\rangle. tr ( S 3 ) = a , b , c ∑ ⟨ w a , w b ⟩ ⟨ w b , w c ⟩ ⟨ w c , w a ⟩ .
Partition ( a , b , c ) ∈ { 1 , … , k } 3 (a,b,c) \in\{1,\ldots,k\}^3 ( a , b , c ) ∈ { 1 , … , k } 3 by multiplicity pattern:
# { a = b = c } = k : E [ ∥ w a ∥ 6 ] = d ( d + 2 ) ( d + 4 ) , # { ∣ { a , b , c } ∣ = 2 } = 3 k ( k − 1 ) : E [ ∥ w a ∥ 2 ⟨ w a , w c ⟩ 2 ] = E [ ∥ w a ∥ 4 ] = d ( d + 2 ) , # { a , b , c distinct } = k ( k − 1 ) ( k − 2 ) : ∑ p , q , r δ p r δ p q δ q r = d , \begin{aligned}
\#\{a=b=c\} &= k: &\quad\mathbb{E}[\lVert w_a\rVert^6] &= d(d+2)(d+4),\\
\#\{\lvert\{a,b,c\}\rvert=2\} &= 3k(k-1): &\quad\mathbb{E}[\lVert w_a\rVert^2\langle w_a,w_c\rangle^2] &= \mathbb{E}[\lVert w_a\rVert^4] = d(d+2),\\
\#\{a,b,c\ \text{distinct}\} &= k(k-1)(k-2): &\quad\sum_{p,q,r}\delta_{pr}\delta_{pq}\delta_{qr} &= d,
\end{aligned} # { a = b = c } # {∣{ a , b , c }∣ = 2 } # { a , b , c distinct } = k : = 3 k ( k − 1 ) : = k ( k − 1 ) ( k − 2 ) : E [∥ w a ∥ 6 ] E [∥ w a ∥ 2 ⟨ w a , w c ⟩ 2 ] p , q , r ∑ δ p r δ pq δ q r = d ( d + 2 ) ( d + 4 ) , = E [∥ w a ∥ 4 ] = d ( d + 2 ) , = d ,
where for the middle pattern, e.g. a = b ≠ c a=b\ne c a = b = c gives ∥ w a ∥ 2 ⟨ w a , w c ⟩ 2 \lVert w_a\rVert^2\langle w_a,w_c\rangle^2 ∥ w a ∥ 2 ⟨ w a , w c ⟩ 2 and, conditioning on w a w_a w a , ⟨ w a , w c ⟩ ∣ w a ∼ N ( 0 , ∥ w a ∥ 2 ) \langle w_a,w_c\rangle\mid w_a\sim\mathcal{N}(0,\lVert w_a\rVert^2) ⟨ w a , w c ⟩ ∣ w a ∼ N ( 0 , ∥ w a ∥ 2 ) ; the other two cases are symmetric. Hence
d α 3 = k d ( d + 2 ) ( d + 4 ) + 3 k ( k − 1 ) d ( d + 2 ) + k ( k − 1 ) ( k − 2 ) d , d\alpha_3 = kd(d+2)(d+4)+3k(k-1)d(d+2)+k(k-1)(k-2)d, d α 3 = k d ( d + 2 ) ( d + 4 ) + 3 k ( k − 1 ) d ( d + 2 ) + k ( k − 1 ) ( k − 2 ) d ,
α 3 = k ( d + 2 ) ( d + 4 ) + 3 k ( k − 1 ) ( d + 2 ) + k ( k − 1 ) ( k − 2 ) = k 3 + 3 ( d + 1 ) k 2 + ( d 2 + 3 d + 4 ) k . \alpha_3 = k(d+2)(d+4)+3k(k-1)(d+2)+k(k-1)(k-2)=k^3+3(d+1)k^2+(d^2+3d+4)k. α 3 = k ( d + 2 ) ( d + 4 ) + 3 k ( k − 1 ) ( d + 2 ) + k ( k − 1 ) ( k − 2 ) = k 3 + 3 ( d + 1 ) k 2 + ( d 2 + 3 d + 4 ) k .
Proof of Theorem 1. Take expectations in (26). Since M ⋆ M^\star M ⋆ is deterministic, Lemma 2 gives
E [ S 3 ] = α 3 I d , \mathbb{E}[S^3]=\alpha_3 I_d, E [ S 3 ] = α 3 I d ,
E [ S 2 M ⋆ ] = E [ M ⋆ S 2 ] = α 2 M ⋆ , \mathbb{E}[S^2M^\star]=\mathbb{E}[M^\star S^2]=\alpha_2M^\star, E [ S 2 M ⋆ ] = E [ M ⋆ S 2 ] = α 2 M ⋆ ,
E [ M ⋆ S M ⋆ ] = M ⋆ E [ S ] M ⋆ = k ( M ⋆ ) 2 , \mathbb{E}[M^\star S M^\star]=M^\star\mathbb{E}[S]M^\star=k(M^\star)^2, E [ M ⋆ S M ⋆ ] = M ⋆ E [ S ] M ⋆ = k ( M ⋆ ) 2 ,
so summing gives (17); the values of α 1 , α 2 , α 3 \alpha_1,\alpha_2,\alpha_3 α 1 , α 2 , α 3 are Lemmas 5 to 7. Corollary 1 follows from the entries of (17): with d , M ⋆ d,M^\star d , M ⋆ fixed, α 3 = Θ ( k 3 ) \alpha_3=\Theta(k^3) α 3 = Θ ( k 3 ) , α 2 = Θ ( k 2 ) \alpha_2=\Theta(k^2) α 2 = Θ ( k 2 ) , k ( M ⋆ ) ∙ 2 = Θ ( k ) k(M^\star)^2_{\bullet}=\Theta(k) k ( M ⋆ ) ∙ 2 = Θ ( k ) .
Remark 1. This computation answers the question raised by Ma et al. [45 ] of how gradient orthogonalization meets the curvature of training: the Θ ( k ) \Theta(k) Θ ( k ) gap between the diagonal and off-diagonal scales is the regime in which orthogonalization, which suppresses the isotropic Θ ( k 3 ) I d \Theta(k^3)I_d Θ ( k 3 ) I d part, acts as a real preconditioner relative to the target-aligned signal M ⋆ M^\star M ⋆ .
Deep Linear Networks
Setup
Let L ≥ 1 L\ge1 L ≥ 1 and d 0 , … , d L d_0,\ldots,d_L d 0 , … , d L be positive integers. The depth-L L L linear network
f ( x ) = W L W L − 1 ⋯ W 1 x , W i ∈ R d i × d i − 1 , (27) f(x)=W_LW_{L-1}\cdots W_1x,\qquad W_i\in\mathbb{R}^{d_i\times d_{i-1}}, \tag*{(27)} f ( x ) = W L W L − 1 ⋯ W 1 x , W i ∈ R d i × d i − 1 , (27)
learns a deterministic linear target Φ ∈ R d L × d 0 \Phi\in\mathbb{R}^{d_L\times d_0} Φ ∈ R d L × d 0 under the population squared loss
L ( W 1 , … , W L ) : = 1 2 E x ∥ W L ⋯ W 1 x − Φ x ∥ 2 . (28) \mathcal{L}(W_1,\ldots,W_L):=\frac{1}{2}\mathbb{E}_x\lVert W_L\cdots W_1x-\Phi x\rVert^2. \tag*{(28)} L ( W 1 , … , W L ) := 2 1 E x ∥ W L ⋯ W 1 x − Φ x ∥ 2 . (28)
Define the layer-i i i forward, backward and residual factors
B i : = W i − 1 ⋯ W 1 ∈ R d i − 1 × d 0 , A i : = W L ⋯ W i + 1 ∈ R d L × d i , R : = W L ⋯ W 1 − Φ , (29) B_i:=W_{i-1}\cdots W_1\in\mathbb{R}^{d_{i-1}\times d_0},\qquad A_i:=W_L\cdots W_{i+1}\in\mathbb{R}^{d_L\times d_i},\qquad R:=W_L\cdots W_1-\Phi, \tag*{(29)} B i := W i − 1 ⋯ W 1 ∈ R d i − 1 × d 0 , A i := W L ⋯ W i + 1 ∈ R d L × d i , R := W L ⋯ W 1 − Φ , (29)
with B 1 : = I d 0 B_1:=I_{d_0} B 1 := I d 0 and A L : = I d L A_L:=I_{d_L} A L := I d L . The layer-i i i gradient is
G i : = ∂ L ∂ W i = A i ⊤ R B i ⊤ ∈ R d i × d i − 1 . (30) G_i:=\frac{\partial\mathcal{L}}{\partial W_i}=A_i^\top R B_i^\top\in\mathbb{R}^{d_i\times d_{i-1}}. \tag*{(30)} G i := ∂ W i ∂ L = A i ⊤ R B i ⊤ ∈ R d i × d i − 1 . (30)
The closed form below uses two pairs of dimension-only scalar sequences. For the backward factors { B i } \{B_i\} { B i } ,
r 1 = 1 , v 1 = d 0 2 , (31) r_1=1,\qquad v_1=d_0^2, \tag*{(31)} r 1 = 1 , v 1 = d 0 2 , (31)
r i = v i − 1 + ( d i − 1 + 1 ) d i − 2 r i − 1 , v i = d i − 1 2 v i − 1 + 2 d i − 1 d i − 2 r i − 1 ( i ≥ 2 ) ; (32) r_i=v_{i-1}+(d_{i-1}+1)d_{i-2}r_{i-1},\qquad v_i=d_{i-1}^2v_{i-1}+2d_{i-1}d_{i-2}r_{i-1}\qquad(i\ge2); \tag*{(32)} r i = v i − 1 + ( d i − 1 + 1 ) d i − 2 r i − 1 , v i = d i − 1 2 v i − 1 + 2 d i − 1 d i − 2 r i − 1 ( i ≥ 2 ) ; (32)
for the forward factors { A i } \{A_i\} { A i } ,
s L = 1 , u L = d L 2 , (33) s_L = 1,\qquad u_L = d_L^2, \tag*{(33)} s L = 1 , u L = d L 2 , (33)
s i − 1 = u i + ( d i − 1 + 1 ) d i s i , u i − 1 = d i − 1 2 u i + 2 d i − 1 d i s i ( i ≤ L ) ; (34) s_{i-1} = u_i + (d_{i-1}+1)d_i s_i,\qquad u_{i-1} = d_{i-1}^2u_i + 2d_{i-1}d_i s_i\quad(i\le L); \tag*{(34)} s i − 1 = u i + ( d i − 1 + 1 ) d i s i , u i − 1 = d i − 1 2 u i + 2 d i − 1 d i s i ( i ≤ L ) ; (34)
and the composite Frobenius coefficient is
V i : = d i − 1 r i = E [ ∥ B i B i T ∥ F 2 ] . (35) V_i := d_{i-1}r_i = \mathbb{E}\left[\left\|B_iB_i^{\mathsf T}\right\|_{\mathrm F}^2\right]. \tag*{(35)} V i := d i − 1 r i = E [ B i B i T F 2 ] . (35)
Standing assumption
Assumption 6. The entries ( W i ) a b (W_i)_{ab} ( W i ) ab are i.i.d. N ( 0 , 1 ) \mathcal{N}(0,1) N ( 0 , 1 ) across all i , a , b i,a,b i , a , b ; x ∼ N ( 0 , I d 0 ) x\sim\mathcal{N}(0,I_{d_0}) x ∼ N ( 0 , I d 0 ) ; and Φ \Phi Φ is deterministic and independent of W 1 , … , W L W_1,\ldots,W_L W 1 , … , W L . Expectations E [ ⋅ ] \mathbb{E}[\cdot] E [ ⋅ ] are over W 1 , … , W L W_1,\ldots,W_L W 1 , … , W L (x x x has been integrated out in L \mathcal{L} L ).
Main results restated
Theorem (Theorem 2, restated). Under Assumption 6, for every layer i ∈ { 1 , … , L } i\in\{1,\ldots,L\} i ∈ { 1 , … , L } ,
E [ G i G i T ] = V i s i I d i + T i , T i = { ( ∏ j = 1 L − 1 d j ) Φ Φ T , i = L , ( ∏ j = 1 j ≠ i L − 1 d j ) ∥ Φ ∥ F 2 I d i , 1 ≤ i < L . (36) \mathbb{E}[G_iG_i^{\mathsf T}] = V_i s_i I_{d_i} + T_i,\qquad
T_i =
\begin{cases}
\left(\prod_{j=1}^{L-1}d_j\right)\Phi\Phi^{\mathsf T}, & i=L,\\
\left(\prod_{\substack{j=1\\j\ne i}}^{L-1}d_j\right)\|\Phi\|_{\mathrm F}^2 I_{d_i}, & 1\le i<L.
\end{cases} \tag*{(36)} E [ G i G i T ] = V i s i I d i + T i , T i = ⎩ ⎨ ⎧ ( ∏ j = 1 L − 1 d j ) Φ Φ T , ( ∏ j = 1 j = i L − 1 d j ) ∥Φ ∥ F 2 I d i , i = L , 1 ≤ i < L . (36) with V i , s i V_i,s_i V i , s i from (31) –(35) .
Corollary 2. Under Assumption 6, the diagonal entries of E [ G i G i T ] \mathbb{E}[G_iG_i^{\mathsf T}] E [ G i G i T ] are
E [ ( G i G i T ) a a ] = { V i s i + ( ∏ j ≠ i , 1 ≤ j ≤ L − 1 d j ) ∥ Φ ∥ F 2 , 1 ≤ i < L , V L s L + ( ∏ j = 1 L − 1 d j ) ( Φ Φ T ) a a , i = L , (37) \mathbb{E}\left[(G_iG_i^{\mathsf T})_{aa}\right] =
\begin{cases}
V_i s_i+\left(\prod_{\substack{j\ne i,\,1\le j\le L-1}}d_j\right)\|\Phi\|_{\mathrm F}^2, & 1\le i<L,\\
V_Ls_L+\left(\prod_{j=1}^{L-1}d_j\right)(\Phi\Phi^{\mathsf T})_{aa}, & i=L,
\end{cases} \tag*{(37)} E [ ( G i G i T ) aa ] = ⎩ ⎨ ⎧ V i s i + ( ∏ j = i , 1 ≤ j ≤ L − 1 d j ) ∥Φ ∥ F 2 , V L s L + ( ∏ j = 1 L − 1 d j ) ( Φ Φ T ) aa , 1 ≤ i < L , i = L , (37) and, for a ≠ b a\ne b a = b , the off-diagonal entries are E [ ( G i G i T ) a b ] = 0 \mathbb{E}[(G_iG_i^{\mathsf T})_{ab}]=0 E [( G i G i T ) ab ] = 0 if 1 ≤ i < L 1\le i<L 1 ≤ i < L and ( ∏ j = 1 L − 1 d j ) ( Φ Φ T ) a b \left(\prod_{j=1}^{L-1}d_j\right)(\Phi\Phi^{\mathsf T})_{ab} ( ∏ j = 1 L − 1 d j ) ( Φ Φ T ) ab if i = L i=L i = L . In particular E [ G i G i T ] \mathbb{E}[G_iG_i^{\mathsf T}] E [ G i G i T ] is a scalar multiple of I d i I_{d_i} I d i for every i < L i<L i < L . Moreover,
V i s i = Θ ( d i ∏ j = 1 j ≠ i L − 1 d j 2 ) ( 1 ≤ i < L ) , V L s L = Θ ( ∏ j = 1 L − 1 d j 2 ) . (38) V_i s_i=\Theta\left(d_i\prod_{\substack{j=1\\j\ne i}}^{L-1}d_j^2\right)\quad(1\le i<L),\qquad
V_Ls_L=\Theta\left(\prod_{j=1}^{L-1}d_j^2\right). \tag*{(38)} V i s i = Θ d i j = 1 j = i ∏ L − 1 d j 2 ( 1 ≤ i < L ) , V L s L = Θ ( j = 1 ∏ L − 1 d j 2 ) . (38)
Preliminary lemmas
Lemma 8. Same as Lemma 1.
Lemma 9 (Wick/Isserlis). Same as Lemma 3.
Lemma 10. Let X ∈ R m × n X\in\mathbb{R}^{m\times n} X ∈ R m × n have i.i.d. N ( 0 , 1 ) \mathcal{N}(0,1) N ( 0 , 1 ) entries and let M ∈ R n × n M\in\mathbb{R}^{n\times n} M ∈ R n × n , N ∈ R m × m N\in\mathbb{R}^{m\times m} N ∈ R m × m be deterministic symmetric. Then
E [ X M X T X M X T ] = ( ( tr ( M ) ) 2 + ( m + 1 ) ∥ M ∥ F 2 ) I m , (39) \mathbb{E}\left[XMX^{\mathsf T}XMX^{\mathsf T}\right]
=\left((\operatorname{tr}(M))^2+(m+1)\|M\|_{\mathrm F}^2\right)I_m, \tag*{(39)} E [ X M X T X M X T ] = ( ( tr ( M ) ) 2 + ( m + 1 ) ∥ M ∥ F 2 ) I m , (39) E [ X T N X X T N X ] = ( ( tr ( N ) ) 2 + ( n + 1 ) ∥ N ∥ F 2 ) I n . (40) \mathbb{E}\left[X^{\mathsf T}NXX^{\mathsf T}NX\right]
=\left((\operatorname{tr}(N))^2+(n+1)\|N\|_{\mathrm F}^2\right)I_n. \tag*{(40)} E [ X T N X X T N X ] = ( ( tr ( N ) ) 2 + ( n + 1 ) ∥ N ∥ F 2 ) I n . (40)
Proof. We prove (39) ; (40) follows by applying it to X T X^{\mathsf T} X T . For a , b , c , d ∈ { 1 , … , m } a,b,c,d\in\{1,\ldots,m\} a , b , c , d ∈ { 1 , … , m } , Lemma 9 gives
E [ X a p X b q X c r X d s ] = δ a b δ p q δ c d δ r s + δ a c δ p r δ b d δ q s + δ a d δ p s δ b c δ q r , \mathbb{E}[X_{ap}X_{bq}X_{cr}X_{ds}]
=\delta_{ab}\delta_{pq}\delta_{cd}\delta_{rs}
+\delta_{ac}\delta_{pr}\delta_{bd}\delta_{qs}
+\delta_{ad}\delta_{ps}\delta_{bc}\delta_{qr}, E [ X a p X b q X cr X d s ] = δ ab δ pq δ c d δ r s + δ a c δ p r δ b d δ q s + δ a d δ p s δ b c δ q r ,
and contracting against M p q M r s M_{pq}M_{rs} M pq M r s with M = M T M=M^{\mathsf T} M = M T ,
E [ ( X M X T ) a b ( X M X T ) c d ] = δ a b δ c d ( tr ( M ) ) 2 + ( δ a c δ b d + δ a d δ b c ) ∥ M ∥ F 2 . \mathbb{E}\left[(XMX^{\mathsf T})_{ab}(XMX^{\mathsf T})_{cd}\right]
=\delta_{ab}\delta_{cd}(\operatorname{tr}(M))^2
+(\delta_{ac}\delta_{bd}+\delta_{ad}\delta_{bc})\|M\|_{\mathrm F}^2. E [ ( X M X T ) ab ( X M X T ) c d ] = δ ab δ c d ( tr ( M ) ) 2 + ( δ a c δ b d + δ a d δ b c ) ∥ M ∥ F 2 .
Summing over b = c b=c b = c ,
E [ ( X M X T X M X T ) a c ] = ∑ b E [ ( X M X T ) a b ( X M X T ) b c ] = δ a c ( ( tr ( M ) ) 2 + ( m + 1 ) ∥ M ∥ F 2 ) . \mathbb{E}\left[(XMX^{\mathsf T}XMX^{\mathsf T})_{ac}\right]
=\sum_b\mathbb{E}\left[(XMX^{\mathsf T})_{ab}(XMX^{\mathsf T})_{bc}\right]
=\delta_{ac}\left((\operatorname{tr}(M))^2+(m+1)\|M\|_{\mathrm F}^2\right). E [ ( X M X T X M X T ) a c ] = b ∑ E [ ( X M X T ) ab ( X M X T ) b c ] = δ a c ( ( tr ( M ) ) 2 + ( m + 1 ) ∥ M ∥ F 2 ) .
Lemma 11. Let X ∈ R m × n X \in\mathbb{R}^{m \times n} X ∈ R m × n have i.i.d. N ( 0 , 1 ) \mathcal{N}(0,1) N ( 0 , 1 ) entries and M ∈ R n × n M \in\mathbb{R}^{n \times n} M ∈ R n × n be deterministic symmetric. Then
E [ tr ( X M X T ) 2 ] = m 2 tr ( M ) 2 + 2 m ∥ M ∥ F 2 . (41) \mathbb{E}\left[\operatorname{tr}\left(XMX^{\mathsf{T}}\right)^2\right] = m^2\operatorname{tr}(M)^2 + 2m\lVert M\rVert_{\mathrm{F}}^2. \tag*{(41)} E [ tr ( X M X T ) 2 ] = m 2 tr ( M ) 2 + 2 m ∥ M ∥ F 2 . (41)
Proof. Expanding,
tr ( X M X T ) 2 = ∑ a , a ′ , p , q , p ′ , q ′ X a p M p q X a q X a ′ p ′ M p ′ q ′ X a ′ q ′ ; \operatorname{tr}\left(XMX^{\mathsf{T}}\right)^2 = \sum_{a,a',p,q,p',q'} X_{ap}M_{pq}X_{aq}X_{a'p'}M_{p'q'}X_{a'q'}; tr ( X M X T ) 2 = a , a ′ , p , q , p ′ , q ′ ∑ X a p M pq X a q X a ′ p ′ M p ′ q ′ X a ′ q ′ ;
by Lemma 9 the three pair contractions of E [ X a p X a q X a ′ p ′ X a ′ q ′ ] \mathbb{E}[X_{ap}X_{aq}X_{a'p'}X_{a'q'}] E [ X a p X a q X a ′ p ′ X a ′ q ′ ] contribute (after summing over M M M M M M ) m 2 tr ( M ) 2 m^2\operatorname{tr}(M)^2 m 2 tr ( M ) 2 , m ∥ M ∥ F 2 m\lVert M\rVert_{\mathrm{F}}^2 m ∥ M ∥ F 2 and m tr ( M 2 ) = m ∥ M ∥ F 2 m\operatorname{tr}(M^2)=m\lVert M\rVert_{\mathrm{F}}^2 m tr ( M 2 ) = m ∥ M ∥ F 2 , which sum to (41) .
Proofs
The proof of Theorem 2 has three steps: a W i W_i W i -expansion; recursive moments of B i B i T B_iB_i^{\mathsf{T}} B i B i T and A i T A i A_i^{\mathsf{T}}A_i A i T A i ; and the target-dependent term.
Lemma 12. Under Assumption 6, for every i i i ,
E [ G i G i T ∣ A i , B i ] = tr ( B i B i T ) 2 ( A i T A i ) 2 + A i T Φ B i T B i Φ T A i . (42) \mathbb{E}[G_iG_i^{\mathsf{T}}\mid A_i,B_i] = \operatorname{tr}(B_iB_i^{\mathsf{T}})^2(A_i^{\mathsf{T}}A_i)^2 + A_i^{\mathsf{T}}\Phi B_i^{\mathsf{T}}B_i\Phi^{\mathsf{T}}A_i. \tag*{(42)} E [ G i G i T ∣ A i , B i ] = tr ( B i B i T ) 2 ( A i T A i ) 2 + A i T Φ B i T B i Φ T A i . (42)
Proof. Substituting R = A i W i B i − Φ R=A_iW_iB_i-\Phi R = A i W i B i − Φ into (30) ,
G i G i T = ( A i T A i ) W i ( B i B i T B i B i T ) W i T ( A i T A i ) + ( c r o s s ) + A i T Φ B i T B i Φ T A i , G_iG_i^{\mathsf{T}}=(A_i^{\mathsf{T}}A_i)W_i(B_iB_i^{\mathsf{T}}B_iB_i^{\mathsf{T}})W_i^{\mathsf{T}}(A_i^{\mathsf{T}}A_i)+(\mathrm{cross})+A_i^{\mathsf{T}}\Phi B_i^{\mathsf{T}}B_i\Phi^{\mathsf{T}}A_i, G i G i T = ( A i T A i ) W i ( B i B i T B i B i T ) W i T ( A i T A i ) + ( cross ) + A i T Φ B i T B i Φ T A i ,
where (cross) collects the two terms linear in W i W_i W i ; since W i = d − W i W_i \mathrel{\overset{d}{=}}-W_i W i = d − W i is independent of ( A i , B i ) (A_i,B_i) ( A i , B i ) , E [ ( c r o s s ) ∣ A i , B i ] = 0 \mathbb{E}[(\mathrm{cross})\mid A_i,B_i]=0 E [( cross ) ∣ A i , B i ] = 0 . By Lemma 8 with K = B i B i T B i B i T K=B_iB_i^{\mathsf{T}}B_iB_i^{\mathsf{T}} K = B i B i T B i B i T ,
E W i [ W i K W i T ∣ A i , B i ] = tr ( K ) I d i = tr ( ( B i B i T ) 2 ) I d i , \mathbb{E}_{W_i}[W_iKW_i^{\mathsf{T}}\mid A_i,B_i]=\operatorname{tr}(K)I_{d_i}=\operatorname{tr}\left((B_iB_i^{\mathsf{T}})^2\right)I_{d_i}, E W i [ W i K W i T ∣ A i , B i ] = tr ( K ) I d i = tr ( ( B i B i T ) 2 ) I d i ,
which yields (42) .
Lemma 13. Under Assumption 6, for every i ∈ { 1 , … , L } i\in\{1,\ldots,L\} i ∈ { 1 , … , L } ,
E [ ( B i B i T ) 2 ] = r i I d i − 1 , E [ tr ( B i B i T ) 2 ] = v i , (43) \mathbb{E}\left[(B_iB_i^{\mathsf{T}})^2\right]=r_iI_{d_{i-1}},\qquad\mathbb{E}\left[\operatorname{tr}(B_iB_i^{\mathsf{T}})^2\right]=v_i, \tag*{(43)} E [ ( B i B i T ) 2 ] = r i I d i − 1 , E [ tr ( B i B i T ) 2 ] = v i , (43) with ( r i , v i ) (r_i,v_i) ( r i , v i ) from (31) –(32) .
Proof. At i = 1 i=1 i = 1 , B 1 = I d 0 B_1=I_{d_0} B 1 = I d 0 , so ( B 1 B 1 T ) 2 = I d 0 (B_1B_1^{\mathsf{T}})^2=I_{d_0} ( B 1 B 1 T ) 2 = I d 0 and tr ( B 1 B 1 T ) 2 = d 0 2 \operatorname{tr}(B_1B_1^{\mathsf{T}})^2=d_0^2 tr ( B 1 B 1 T ) 2 = d 0 2 , matching r 1 = 1 r_1=1 r 1 = 1 , v 1 = d 0 2 v_1=d_0^2 v 1 = d 0 2 . For i ≥ 2 i\geq2 i ≥ 2 , B i = W i − 1 B i − 1 B_i=W_{i-1}B_{i-1} B i = W i − 1 B i − 1 with W i − 1 ∈ R d i − 1 × d i − 2 W_{i-1}\in\mathbb{R}^{d_{i-1}\times d_{i-2}} W i − 1 ∈ R d i − 1 × d i − 2 independent of B i − 1 B_{i-1} B i − 1 , so B i B i T = W i − 1 M W i − 1 T B_iB_i^{\mathsf{T}}=W_{i-1}MW_{i-1}^{\mathsf{T}} B i B i T = W i − 1 M W i − 1 T with M : = B i − 1 B i − 1 T M:=B_{i-1}B_{i-1}^{\mathsf{T}} M := B i − 1 B i − 1 T . By Lemma 10,
E [ ( B i B i T ) 2 ∣ B i − 1 ] = ( tr ( M ) 2 + ( d i − 1 + 1 ) ∥ M ∥ F 2 ) I d i − 1 , \mathbb{E}\left[(B_iB_i^{\mathsf{T}})^2\mid B_{i-1}\right]=\left(\operatorname{tr}(M)^2+(d_{i-1}+1)\lVert M\rVert_{\mathrm{F}}^2\right)I_{d_{i-1}}, E [ ( B i B i T ) 2 ∣ B i − 1 ] = ( tr ( M ) 2 + ( d i − 1 + 1 ) ∥ M ∥ F 2 ) I d i − 1 ,
and taking E \mathbb{E} E over B i − 1 B_{i-1} B i − 1 with E [ tr ( M 2 ) ] = d i − 2 r i − 1 \mathbb{E}[\operatorname{tr}(M^2)]=d_{i-2}r_{i-1} E [ tr ( M 2 )] = d i − 2 r i − 1 gives r i = v i − 1 + ( d i − 1 + 1 ) d i − 2 r i − 1 r_i=v_{i-1}+(d_{i-1}+1)d_{i-2}r_{i-1} r i = v i − 1 + ( d i − 1 + 1 ) d i − 2 r i − 1 . Likewise, by Lemma 11,
E [ tr ( B i B i T ) 2 ∣ B i − 1 ] = d i − 1 2 tr ( M ) 2 + 2 d i − 1 ∥ M ∥ F 2 , \mathbb{E}\left[\operatorname{tr}(B_iB_i^{\mathsf{T}})^2\mid B_{i-1}\right]=d_{i-1}^2\operatorname{tr}(M)^2+2d_{i-1}\lVert M\rVert_{\mathrm{F}}^2, E [ tr ( B i B i T ) 2 ∣ B i − 1 ] = d i − 1 2 tr ( M ) 2 + 2 d i − 1 ∥ M ∥ F 2 ,
so v i = d i − 1 2 v i − 1 + 2 d i − 1 d i − 2 r i − 1 v_i=d_{i-1}^2v_{i-1}+2d_{i-1}d_{i-2}r_{i-1} v i = d i − 1 2 v i − 1 + 2 d i − 1 d i − 2 r i − 1 .
Lemma 14. Under Assumption 6, for every i ∈ { 0 , … , L } i\in\{0,\ldots,L\} i ∈ { 0 , … , L } ,
E [ ( A i T A i ) 2 ] = s i I d i , E [ tr ( A i T A i ) 2 ] = u i , (44) \mathbb{E}\left[(A_i^{\mathsf{T}}A_i)^2\right]=s_iI_{d_i},\qquad\mathbb{E}\left[\operatorname{tr}(A_i^{\mathsf{T}}A_i)^2\right]=u_i, \tag*{(44)} E [ ( A i T A i ) 2 ] = s i I d i , E [ tr ( A i T A i ) 2 ] = u i , (44) with ( s i , u i ) (s_i,u_i) ( s i , u i ) from (33) –(34) .
Proof. At i = L i=L i = L , A L = I d L A_L=I_{d_L} A L = I d L , matching s L = 1 s_L=1 s L = 1 , u L = d L 2 u_L=d_L^2 u L = d L 2 . For i < L i<L i < L , A i = A i + 1 W i + 1 A_i=A_{i+1}W_{i+1} A i = A i + 1 W i + 1 with W i + 1 ∈ R d i + 1 × d i W_{i+1}\in\mathbb{R}^{d_{i+1}\times d_i} W i + 1 ∈ R d i + 1 × d i independent of A i + 1 A_{i+1} A i + 1 , so A i T A i = W i + 1 T N W i + 1 A_i^{\mathsf{T}}A_i=W_{i+1}^{\mathsf{T}}NW_{i+1} A i T A i = W i + 1 T N W i + 1 with N : = A i + 1 T A i + 1 N:=A_{i+1}^{\mathsf{T}}A_{i+1} N := A i + 1 T A i + 1 . By Lemma 10 (right identity),
E [ ( A i T A i ) 2 ∣ A i + 1 ] = ( tr ( N ) 2 + ( d i + 1 ) ∥ N ∥ F 2 ) I d i , \mathbb{E}\left[(A_i^{\mathsf{T}}A_i)^2\mid A_{i+1}\right]=\left(\operatorname{tr}(N)^2+(d_i+1)\lVert N\rVert_{\mathrm{F}}^2\right)I_{d_i}, E [ ( A i T A i ) 2 ∣ A i + 1 ] = ( tr ( N ) 2 + ( d i + 1 ) ∥ N ∥ F 2 ) I d i ,
so s i = u i + 1 + ( d i + 1 ) d i + 1 s i + 1 s_i=u_{i+1}+(d_i+1)d_{i+1}s_{i+1} s i = u i + 1 + ( d i + 1 ) d i + 1 s i + 1 ; the recursion for u i u_i u i is that of v i v_i v i with d i − 1 ↔ d i d_{i-1}\leftrightarrow d_i d i − 1 ↔ d i and d i − 2 ↔ d i + 1 d_{i-2}\leftrightarrow d_{i+1} d i − 2 ↔ d i + 1 , giving u i − 1 = d i − 1 2 u i + 2 d i − 1 d i s i u_{i-1}=d_{i-1}^2u_i+2d_{i-1}d_is_i u i − 1 = d i − 1 2 u i + 2 d i − 1 d i s i .
Lemma 15. Under Assumption 6, E [ A i T Φ B i T B i Φ T A i ] = T i \mathbb{E}[A_i^{\mathsf{T}}\Phi B_i^{\mathsf{T}}B_i\Phi^{\mathsf{T}}A_i]=T_i E [ A i T Φ B i T B i Φ T A i ] = T i , with T i T_i T i as in (36) .
Proof. By independence of A i A_i A i , B i B_i B i and Lemma 8 iterated through B i = W i − 1 ⋯ W 1 B_i=W_{i-1}\cdots W_1 B i = W i − 1 ⋯ W 1 ,
E [ B i T B i ] = ( ∏ j = 1 i − 1 d j ) I d 0 ⟹ E [ A i T Φ B i T B i Φ T A i ] = ( ∏ j = 1 i − 1 d j ) E [ A i T Φ Φ T A i ] . \mathbb{E}[B_i^{\mathsf{T}}B_i]=\left(\prod_{j=1}^{i-1}d_j\right)I_{d_0}
\quad\Longrightarrow\quad
\mathbb{E}[A_i^{\mathsf{T}}\Phi B_i^{\mathsf{T}}B_i\Phi^{\mathsf{T}}A_i]=\left(\prod_{j=1}^{i-1}d_j\right)\mathbb{E}[A_i^{\mathsf{T}}\Phi\Phi^{\mathsf{T}}A_i]. E [ B i T B i ] = ( j = 1 ∏ i − 1 d j ) I d 0 ⟹ E [ A i T Φ B i T B i Φ T A i ] = ( j = 1 ∏ i − 1 d j ) E [ A i T Φ Φ T A i ] .
(the empty product is 1 when i = 1 i=1 i = 1 ). Let h i ( M ) : = E [ A i T M A i ] h_i(M):=\mathbb{E}[A_i^{\mathsf{T}}MA_i] h i ( M ) := E [ A i T M A i ] for symmetric M ∈ R d L × d L M\in\mathbb{R}^{d_L\times d_L} M ∈ R d L × d L . Then h L ( M ) = M h_L(M)=M h L ( M ) = M , and for i < L i<L i < L , by Lemma 8,
h i ( M ) = E [ tr ( A i + 1 T M A i + 1 ) I d i ] = tr ( h i + 1 ( M ) ) I d i . h_i(M)=\mathbb{E}\left[\operatorname{tr}(A_{i+1}^{\mathsf{T}}MA_{i+1})I_{d_i}\right]=\operatorname{tr}(h_{i+1}(M))I_{d_i}. h i ( M ) = E [ tr ( A i + 1 T M A i + 1 ) I d i ] = tr ( h i + 1 ( M )) I d i .
Iterating,
h i ( M ) = { M , i = L , ( ∏ j = i + 1 L − 1 d j ) tr ( M ) I d i , i < L . h_i(M)=
\begin{cases}
M, & i=L,\\
\left(\prod_{j=i+1}^{L-1}d_j\right)\operatorname{tr}(M)I_{d_i}, & i<L.
\end{cases} h i ( M ) = { M , ( ∏ j = i + 1 L − 1 d j ) tr ( M ) I d i , i = L , i < L .
with ∏ j = 1 L − 1 d j = 1 \prod_{j=1}^{L-1}d_j=1 ∏ j = 1 L − 1 d j = 1 . Taking M = Φ Φ T M=\Phi\Phi^{\mathsf{T}} M = Φ Φ T (so tr ( M ) = ∥ Φ ∥ F 2 \operatorname{tr}(M)=\|\Phi\|_F^2 tr ( M ) = ∥Φ ∥ F 2 ) and multiplying by ∏ j = 1 i − 1 d j \prod_{j=1}^{i-1}d_j ∏ j = 1 i − 1 d j gives T i T_i T i . ▪
Proof of Theorem 2. By Lemma 12,
E [ G i G i T ] = E [ tr ( ( B i B i T ) 2 ) ( A i T A i ) 2 ] + E [ A i T Φ B i T B i Φ T A i ] . \mathbb{E}[G_iG_i^{\mathsf{T}}]=\mathbb{E}\left[\operatorname{tr}\left((B_iB_i^{\mathsf{T}})^2\right)(A_i^{\mathsf{T}}A_i)^2\right]+\mathbb{E}[A_i^{\mathsf{T}}\Phi B_i^{\mathsf{T}}B_i\Phi^{\mathsf{T}}A_i]. E [ G i G i T ] = E [ tr ( ( B i B i T ) 2 ) ( A i T A i ) 2 ] + E [ A i T Φ B i T B i Φ T A i ] .
Since A i , B i A_i,B_i A i , B i depend on disjoint sets of W j W_j W j , they are independent, so by Lemmas 13 and 14,
E [ tr ( ( B i B i T ) 2 ) ( A i T A i ) 2 ] = E [ tr ( ( B i B i T ) 2 ) ] E [ ( A i T A i ) 2 ] = d i − 1 r i ⋅ s i I d i = V i s i I d i , \mathbb{E}\left[\operatorname{tr}\left((B_iB_i^{\mathsf{T}})^2\right)(A_i^{\mathsf{T}}A_i)^2\right]=\mathbb{E}\left[\operatorname{tr}\left((B_iB_i^{\mathsf{T}})^2\right)\right]\mathbb{E}\left[(A_i^{\mathsf{T}}A_i)^2\right]=d_{i-1}r_i\cdot s_iI_{d_i}=V_is_iI_{d_i}, E [ tr ( ( B i B i T ) 2 ) ( A i T A i ) 2 ] = E [ tr ( ( B i B i T ) 2 ) ] E [ ( A i T A i ) 2 ] = d i − 1 r i ⋅ s i I d i = V i s i I d i ,
and the second term equals T i T_i T i by Lemma 15. Corollary 2 follows from the entries of (36) : for i < L i<L i < L , T i T_i T i is a multiple of I d i I_{d_i} I d i so the off-diagonal vanishes; for i = L i=L i = L , T L T_L T L contributes ( Φ Φ T ) a b (\Phi\Phi^{\mathsf{T}})_{ab} ( Φ Φ T ) ab entry-wise.
It remains to prove (38) . Let π i : = ∏ j = 0 i − 1 d j \pi_i:=\prod_{j=0}^{i-1}d_j π i := ∏ j = 0 i − 1 d j and σ i : = ∑ m = 0 i − 1 d m − 1 \sigma_i:=\sum_{m=0}^{i-1}d_m^{-1} σ i := ∑ m = 0 i − 1 d m − 1 , so that π i = d i − 1 π i − 1 \pi_i=d_{i-1}\pi_{i-1} π i = d i − 1 π i − 1 and σ i = σ i − 1 + d i − 1 − 1 \sigma_i=\sigma_{i-1}+d_{i-1}^{-1} σ i = σ i − 1 + d i − 1 − 1 . Since V i = d i − 1 r i V_i=d_{i-1}r_i V i = d i − 1 r i , the recursion (31) –(32) reads
V 1 = d 0 , v 1 = d 0 2 , V_1=d_0,\qquad v_1=d_0^2, V 1 = d 0 , v 1 = d 0 2 ,
V i = d i − 1 v i − 1 + d i − 1 ( d i − 1 + 1 ) V i − 1 , v i = d i − 1 2 v i − 1 + 2 d i − 1 V i − 1 ( i ≥ 2 ) . V_i=d_{i-1}v_{i-1}+d_{i-1}(d_{i-1}+1)V_{i-1},\qquad v_i=d_{i-1}^2v_{i-1}+2d_{i-1}V_{i-1}\qquad(i\ge2). V i = d i − 1 v i − 1 + d i − 1 ( d i − 1 + 1 ) V i − 1 , v i = d i − 1 2 v i − 1 + 2 d i − 1 V i − 1 ( i ≥ 2 ) .
Let a i : = v i / π i 2 a_i:=v_i/\pi_i^2 a i := v i / π i 2 and b i : = V i / ( π i 2 σ i ) b_i:=V_i/(\pi_i^2\sigma_i) b i := V i / ( π i 2 σ i ) , so a 1 = b 1 = 1 a_1=b_1=1 a 1 = b 1 = 1 . Dividing the two recursions by π i 2 \pi_i^2 π i 2 and by π i 2 σ i \pi_i^2\sigma_i π i 2 σ i gives
a i = a i − 1 + 2 σ i − 1 d i − 1 b i − 1 , b i = d i − 1 − 1 σ i a i − 1 + σ i − 1 σ i ( 1 + d i − 1 − 1 ) b i − 1 . a_i=a_{i-1}+\frac{2\sigma_{i-1}}{d_{i-1}}b_{i-1},\qquad b_i=\frac{d_{i-1}^{-1}}{\sigma_i}a_{i-1}+\frac{\sigma_{i-1}}{\sigma_i}(1+d_{i-1}^{-1})b_{i-1}. a i = a i − 1 + d i − 1 2 σ i − 1 b i − 1 , b i = σ i d i − 1 − 1 a i − 1 + σ i σ i − 1 ( 1 + d i − 1 − 1 ) b i − 1 .
The second identity expresses b i b_i b i as a convex combination of a i − 1 a_{i-1} a i − 1 and ( 1 + d i − 1 − 1 ) b i − 1 (1+d_{i-1}^{-1})b_{i-1} ( 1 + d i − 1 − 1 ) b i − 1 , so min ( a i − 1 , b i − 1 ) ≤ b i ≤ 2 max ( a i − 1 , b i − 1 ) \min(a_{i-1},b_{i-1})\le b_i\le2\max(a_{i-1},b_{i-1}) min ( a i − 1 , b i − 1 ) ≤ b i ≤ 2 max ( a i − 1 , b i − 1 ) ; the first gives a i − 1 ≤ a i ≤ a i − 1 + 2 ( i − 1 ) b i − 1 a_{i-1}\le a_i\le a_{i-1}+2(i-1)b_{i-1} a i − 1 ≤ a i ≤ a i − 1 + 2 ( i − 1 ) b i − 1 , using σ i − 1 ≤ i − 1 \sigma_{i-1}\le i-1 σ i − 1 ≤ i − 1 and d i − 1 ≥ 1 d_{i-1}\ge1 d i − 1 ≥ 1 . By induction, 1 ≤ a i , b i ≤ ( 2 L ) L 1\le a_i,b_i\le(2L)^L 1 ≤ a i , b i ≤ ( 2 L ) L for every i ≤ L i\le L i ≤ L , that is,
v i = Θ ( π i 2 ) , V i = Θ ( π i 2 σ i ) . v_i=\Theta(\pi_i^2),\qquad V_i=\Theta(\pi_i^2\sigma_i). v i = Θ ( π i 2 ) , V i = Θ ( π i 2 σ i ) .
The recursion (33) –(34) has the same form read from the output side: with S i : = d i s i S_i:=d_is_i S i := d i s i , ρ i : = ∏ j = i L d j \rho_i:=\prod_{j=i}^{L}d_j ρ i := ∏ j = i L d j and τ i : = ∑ m = i L d m − 1 \tau_i:=\sum_{m=i}^{L}d_m^{-1} τ i := ∑ m = i L d m − 1 ,
S L = d L , u L = d L 2 , S_L=d_L,\qquad u_L=d_L^2, S L = d L , u L = d L 2 ,
S i − 1 = d i − 1 u i + d i − 1 ( d i − 1 + 1 ) S i , u i − 1 = d i − 1 2 u i + 2 d i − 1 S i ( i ≤ L ) , S_{i-1}=d_{i-1}u_i+d_{i-1}(d_{i-1}+1)S_i,\qquad u_{i-1}=d_{i-1}^2u_i+2d_{i-1}S_i\qquad(i\le L), S i − 1 = d i − 1 u i + d i − 1 ( d i − 1 + 1 ) S i , u i − 1 = d i − 1 2 u i + 2 d i − 1 S i ( i ≤ L ) ,
and the same argument gives u i = Θ ( ρ i 2 ) u_i=\Theta(\rho_i^2) u i = Θ ( ρ i 2 ) and S i = Θ ( ρ i 2 τ i ) S_i=\Theta(\rho_i^2\tau_i) S i = Θ ( ρ i 2 τ i ) . Therefore
V i s i = V i S i d i = Θ ( π i 2 ρ i 2 d i σ i τ i ) , π i 2 ρ i 2 d i = 1 d i ( ∏ j = 0 L d j ) 2 . V_is_i=\frac{V_iS_i}{d_i}=\Theta\left(\frac{\pi_i^2\rho_i^2}{d_i}\sigma_i\tau_i\right),\qquad\frac{\pi_i^2\rho_i^2}{d_i}=\frac{1}{d_i}\left(\prod_{j=0}^{L}d_j\right)^2. V i s i = d i V i S i = Θ ( d i π i 2 ρ i 2 σ i τ i ) , d i π i 2 ρ i 2 = d i 1 ( j = 0 ∏ L d j ) 2 .
Since d 0 − 1 ≤ σ i ≤ L d_0^{-1}\le\sigma_i\le L d 0 − 1 ≤ σ i ≤ L and d L − 1 ≤ τ i ≤ L d_L^{-1}\le\tau_i\le L d L − 1 ≤ τ i ≤ L , the factor σ i τ i \sigma_i\tau_i σ i τ i is bounded above and below, and ( ∏ j = 0 L d j ) 2 / d i (\prod_{j=0}^{L}d_j)^2/d_i ( ∏ j = 0 L d j ) 2 / d i equals d 0 2 d L 2 d i ∏ j = 1 , j ≠ i L − 1 d j 2 d_0^2d_L^2d_i\prod_{j=1,j\ne i}^{L-1}d_j^2 d 0 2 d L 2 d i ∏ j = 1 , j = i L − 1 d j 2 for i < L i<L i < L and d 0 2 d L ∏ j = 1 L − 1 d j 2 d_0^2d_L\prod_{j=1}^{L-1}d_j^2 d 0 2 d L ∏ j = 1 L − 1 d j 2 for i = L i=L i = L , which gives (38) . ▪
Two-Layer ReLU Networks
Setup
Let d 0 , d 1 , d 2 d_0,d_1,d_2 d 0 , d 1 , d 2 be positive integers. The two-layer ReLU network
f ( x ) = W 2 σ ( W 1 x ) , W 1 ∈ R d 1 × d 0 , W 2 ∈ R d 2 × d 1 , (45) f(x)=W_2\sigma(W_1x),\qquad W_1\in\mathbb{R}^{d_1\times d_0},\qquad W_2\in\mathbb{R}^{d_2\times d_1}, \tag*{(45)} f ( x ) = W 2 σ ( W 1 x ) , W 1 ∈ R d 1 × d 0 , W 2 ∈ R d 2 × d 1 , (45)
acts on x ∈ R d 0 x\in\mathbb{R}^{d_0} x ∈ R d 0 via σ ( z ) = max ( z , 0 ) \sigma(z)=\max(z,0) σ ( z ) = max ( z , 0 ) , σ ′ ( z ) = 1 [ z > 0 ] \sigma'(z)=1[z>0] σ ′ ( z ) = 1 [ z > 0 ] . Let h ( x ) : = σ ( W 1 x ) ∈ R d 1 h(x):=\sigma(W_1x)\in\mathbb{R}^{d_1} h ( x ) := σ ( W 1 x ) ∈ R d 1 , and let w a ∈ R d 0 w_a\in\mathbb{R}^{d_0} w a ∈ R d 0 denote the a a a -th row of W 1 W_1 W 1 . The population loss against a deterministic linear target Φ ∈ R d 2 × d 0 \Phi\in\mathbb{R}^{d_2\times d_0} Φ ∈ R d 2 × d 0
L ( W 1 , W 2 ) : = 1 2 E x ∥ W 2 σ ( W 1 x ) − Φ x ∥ 2 , (46) \mathcal{L}(W_1,W_2):=\frac{1}{2}\mathbb{E}_x\left\|W_2\sigma(W_1x)-\Phi x\right\|^2, \tag*{(46)} L ( W 1 , W 2 ) := 2 1 E x ∥ W 2 σ ( W 1 x ) − Φ x ∥ 2 , (46)
and G i : = ∂ L / ∂ W i G_i:=\partial\mathcal{L}/\partial W_i G i := ∂ L / ∂ W i , i ∈ { 1 , 2 } i\in\{1,2\} i ∈ { 1 , 2 } . For independent u , v ∼ N ( 0 , I d 0 ) u,v\sim\mathcal{N}(0,I_{d_0}) u , v ∼ N ( 0 , I d 0 ) let θ : = ∠ ( u , v ) ∈ [ 0 , π ] \theta:=\angle(u,v)\in[0,\pi] θ := ∠ ( u , v ) ∈ [ 0 , π ] be their angle, so that cos θ = ⟨ u , v ⟩ / ( ∥ u ∥ ∥ v ∥ ) \cos\theta=\langle u,v\rangle/(\|u\|\|v\|) cos θ = ⟨ u , v ⟩ / ( ∥ u ∥∥ v ∥ ) , and define
κ 1 : = E [ sin 2 θ + ( π − θ ) 2 4 π 2 + sin θ cos θ ( π − θ ) 2 π 2 ] , (47) \kappa_1:=\mathbb{E}\left[\frac{\sin^2\theta+(\pi-\theta)^2}{4\pi^2}+\frac{\sin\theta\cos\theta(\pi-\theta)}{2\pi^2}\right], \tag*{(47)} κ 1 := E [ 4 π 2 sin 2 θ + ( π − θ ) 2 + 2 π 2 sin θ cos θ ( π − θ ) ] , (47)
κ 2 : = E [ sin θ ( π − θ ) ] 2 π 2 . (48) \kappa_2:=\frac{\mathbb{E}[\sin\theta(\pi-\theta)]}{2\pi^2}. \tag*{(48)} κ 2 := 2 π 2 E [ sin θ ( π − θ )] . (48)
Standing assumption
Assumption 7. ( W 1 ) i j (W_1)_{ij} ( W 1 ) ij and ( W 2 ) i j (W_2)_{ij} ( W 2 ) ij are i.i.d. N ( 0 , 1 ) \mathcal{N}(0,1) N ( 0 , 1 ) ; x ∼ N ( 0 , I d 0 ) x\sim\mathcal{N}(0,I_{d_0}) x ∼ N ( 0 , I d 0 ) ; Φ \Phi Φ is deterministic and independent of W 1 , W 2 W_1,W_2 W 1 , W 2 . Expectations E [ ⋅ ] \mathbb{E}[\cdot] E [ ⋅ ] are over W 1 , W 2 W_1,W_2 W 1 , W 2 (x x x has been integrated out in L \mathcal{L} L ).
Main results restated
Theorem (Theorem 3, restated). Under Assumption 7, with H : = E x [ h ( x ) h ( x ) ⊤ ] H:=\mathbb{E}_x[h(x)h(x)^\top] H := E x [ h ( x ) h ( x ) ⊤ ] and α : = E W 1 [ tr ( H 2 ) ] \alpha:=\mathbb{E}_{W_1}[\operatorname{tr}(H^2)] α := E W 1 [ tr ( H 2 )] ,
E [ G 2 G 2 ⊤ ] = α I d 2 + d 1 4 Φ Φ ⊤ , α = Θ ( d 1 2 ) , (49) \mathbb{E}[G_2G_2^\top]=\alpha I_{d_2}+\frac{d_1}{4}\Phi\Phi^\top,\qquad\alpha=\Theta(d_1^2), \tag*{(49)} E [ G 2 G 2 ⊤ ] = α I d 2 + 4 d 1 Φ Φ ⊤ , α = Θ ( d 1 2 ) , (49) and E [ G 1 G 1 ⊤ ] \mathbb{E}[G_1G_1^\top] E [ G 1 G 1 ⊤ ] is permutation-invariant in the hidden index with
E [ ( G 1 G 1 ⊤ ) a a ] = d 0 d 2 2 4 + d 0 d 2 [ ( d 1 − 1 ) κ 1 + 1 2 ] + ∥ Φ ∥ 2 4 , (50) \mathbb{E}[(G_1G_1^\top)_{aa}]=\frac{d_0d_2^2}{4}+d_0d_2\left[(d_1-1)\kappa_1+\frac{1}{2}\right]+\frac{\|\Phi\|^2}{4}, \tag*{(50)} E [( G 1 G 1 ⊤ ) aa ] = 4 d 0 d 2 2 + d 0 d 2 [ ( d 1 − 1 ) κ 1 + 2 1 ] + 4 ∥Φ ∥ 2 , (50) E [ ( G 1 G 1 ⊤ ) a c ] = d 0 d 2 κ 2 + O ( d 2 d 0 ) ( a ≠ c ) . (51) \mathbb{E}[(G_1G_1^\top)_{ac}]=d_0d_2\kappa_2+O(d_2\sqrt{d_0})\qquad(a\ne c). \tag*{(51)} E [( G 1 G 1 ⊤ ) a c ] = d 0 d 2 κ 2 + O ( d 2 d 0 ) ( a = c ) . (51) As d 0 → ∞ d_0\to\infty d 0 → ∞ , κ 1 → 1 4 π 2 + 1 16 \kappa_1\to\frac{1}{4\pi^2}+\frac{1}{16} κ 1 → 4 π 2 1 + 16 1 and κ 2 → 1 4 π \kappa_2\to\frac{1}{4\pi} κ 2 → 4 π 1 .
Preliminary lemmas
Lemma 16. Same as Lemma 1.
Lemma 17 (Gaussian integration by parts). Let x ∼ N ( 0 , I n ) x\sim\mathcal{N}(0,I_n) x ∼ N ( 0 , I n ) and f : R n → R f:\mathbb{R}^n\to\mathbb{R} f : R n → R be locally absolutely continuous along almost every line parallel to the j j j -th axis, with weak derivative ∂ j f \partial_jf ∂ j f , and E ∣ x j f ( x ) ∣ < ∞ \mathbb{E}|x_jf(x)|<\infty E ∣ x j f ( x ) ∣ < ∞ , E ∣ ∂ j f ( x ) ∣ < ∞ \mathbb{E}|\partial_jf(x)|<\infty E ∣ ∂ j f ( x ) ∣ < ∞ . Then
E [ x j f ( x ) ] = E [ ∂ j f ( x ) ] . (52) \mathbb{E}[x_jf(x)]=\mathbb{E}[\partial_jf(x)]. \tag*{(52)} E [ x j f ( x )] = E [ ∂ j f ( x )] . (52) In particular this holds whenever f f f is Lipschitz.
Proof. A standard identity ([68 , 51 ] ). The standard Gaussian density φ \varphi φ satisfies ∂ j φ ( x ) = − x j φ ( x ) \partial_j\varphi(x)=-x_j\varphi(x) ∂ j φ ( x ) = − x j φ ( x ) , so by Fubini and one-dimensional integration by parts in x j x_j x j (the boundary term vanishes by integrability),
E [ x j f ( x ) ] = − ∫ f ∂ j φ = ∫ ∂ j f φ = E [ ∂ j f ( x ) ] . \mathbb{E}[x_jf(x)]=-\int f\,\partial_j\varphi=\int\partial_jf\,\varphi=\mathbb{E}[\partial_jf(x)]. E [ x j f ( x )] = − ∫ f ∂ j φ = ∫ ∂ j f φ = E [ ∂ j f ( x )] .
The Lipschitz case follows from Rademacher’s theorem.
Lemma 18. For Z ∼ N ( 0 , s 2 ) Z \sim\mathcal{N}(0,s^2) Z ∼ N ( 0 , s 2 ) with s > 0 s > 0 s > 0 ,
E [ σ ( Z ) ] = s 2 π , E [ σ ( Z ) 2 ] = s 2 2 , E [ σ ′ ( Z ) ] = 1 2 . (53) \mathbb{E}[\sigma(Z)] = \frac{s}{\sqrt{2\pi}}, \qquad\mathbb{E}[\sigma(Z)^2] = \frac{s^2}{2}, \qquad\mathbb{E}[\sigma'(Z)] = \frac{1}{2}. \tag*{(53)} E [ σ ( Z )] = 2 π s , E [ σ ( Z ) 2 ] = 2 s 2 , E [ σ ′ ( Z )] = 2 1 . (53) Proof. Direct computation: E [ σ ( Z ) ] = ∫ 0 ∞ z ϕ s ( z ) d z = s / 2 π \mathbb{E}[\sigma(Z)] = \int_0^\infty z\phi_s(z)\,\mathrm{d}z = s/\sqrt{2\pi} E [ σ ( Z )] = ∫ 0 ∞ z ϕ s ( z ) d z = s / 2 π ; E [ σ ( Z ) 2 ] = ∫ 0 ∞ z 2 ϕ s ( z ) d z = s 2 / 2 \mathbb{E}[\sigma(Z)^2] = \int_0^\infty z^2\phi_s(z)\,\mathrm{d}z = s^2/2 E [ σ ( Z ) 2 ] = ∫ 0 ∞ z 2 ϕ s ( z ) d z = s 2 /2 ; E [ σ ′ ( Z ) ] = Pr ( Z > 0 ) = 1 / 2 \mathbb{E}[\sigma'(Z)] = \Pr(Z > 0) = 1/2 E [ σ ′ ( Z )] = Pr ( Z > 0 ) = 1/2 .
Lemma 19 (Cho–Saul kernel). Let u , v ∈ R n ∖ { 0 } u,v \in\mathbb{R}^n \setminus\{0\} u , v ∈ R n ∖ { 0 } , x ∼ N ( 0 , I n ) x \sim\mathcal{N}(0,I_n) x ∼ N ( 0 , I n ) , θ : = ∠ ( u , v ) \theta:= \angle(u,v) θ := ∠ ( u , v ) . Then Cho & Saul [11 ]
E [ σ ( u T x ) σ ( v T x ) ] = ∥ u ∥ ∥ v ∥ 2 π ( sin θ + ( π − θ ) cos θ ) , (54) \mathbb{E}[\sigma(u^{\mathsf{T}}x)\sigma(v^{\mathsf{T}}x)] = \frac{\lVert u\rVert\lVert v\rVert}{2\pi}\left(\sin\theta+ (\pi-\theta)\cos\theta\right), \tag*{(54)} E [ σ ( u T x ) σ ( v T x )] = 2 π ∥ u ∥ ∥ v ∥ ( sin θ + ( π − θ ) cos θ ) , (54) E [ σ ′ ( u T x ) σ ′ ( v T x ) ] = π − θ 2 π . (55) \mathbb{E}[\sigma'(u^{\mathsf{T}}x)\sigma'(v^{\mathsf{T}}x)] = \frac{\pi-\theta}{2\pi}. \tag*{(55)} E [ σ ′ ( u T x ) σ ′ ( v T x )] = 2 π π − θ . (55) Proof. Both quantities depend on ( u T x , v T x ) (u^{\mathsf{T}}x,v^{\mathsf{T}}x) ( u T x , v T x ) , bivariate Gaussian with variances ∥ u ∥ 2 , ∥ v ∥ 2 \lVert u\rVert^2,\lVert v\rVert^2 ∥ u ∥ 2 , ∥ v ∥ 2 and correlation cos θ \cos\theta cos θ . Equation (55) is Pr ( u T x > 0 , v T x > 0 ) \Pr(u^{\mathsf{T}}x > 0,v^{\mathsf{T}}x > 0) Pr ( u T x > 0 , v T x > 0 ) , the angular measure of the cone { Z 1 > 0 , Z 2 > 0 } \{Z_1 > 0,Z_2 > 0\} { Z 1 > 0 , Z 2 > 0 } for a standard bivariate Gaussian with correlation cos θ \cos\theta cos θ , namely ( π − θ ) / ( 2 π ) (\pi-\theta)/(2\pi) ( π − θ ) / ( 2 π ) . For (54) , by Price’s theorem (Price [58 ] ),
∂ ∂ ρ E [ σ ( u T x ) σ ( v T x ) ] = ∥ u ∥ ∥ v ∥ E [ σ ′ ( u T x ) σ ′ ( v T x ) ] = ∥ u ∥ ∥ v ∥ ( π − θ ) 2 π ; \frac{\partial}{\partial\rho}\mathbb{E}[\sigma(u^{\mathsf{T}}x)\sigma(v^{\mathsf{T}}x)] = \lVert u\rVert\lVert v\rVert\,\mathbb{E}[\sigma'(u^{\mathsf{T}}x)\sigma'(v^{\mathsf{T}}x)] = \frac{\lVert u\rVert\lVert v\rVert(\pi-\theta)}{2\pi}; ∂ ρ ∂ E [ σ ( u T x ) σ ( v T x )] = ∥ u ∥ ∥ v ∥ E [ σ ′ ( u T x ) σ ′ ( v T x )] = 2 π ∥ u ∥ ∥ v ∥ ( π − θ ) ; at ρ = − 1 \rho=-1 ρ = − 1 (i.e. θ = π \theta=\pi θ = π ) the product vanishes a.s., and integrating ∂ ρ \partial_\rho ∂ ρ from − 1 -1 − 1 to cos θ \cos\theta cos θ gives (54) .
Lemma 20 (Wick/Isserlis). Same as Lemma 3.
Lemma 21. Let x ∼ N ( 0 , I n ) x \sim\mathcal{N}(0,I_n) x ∼ N ( 0 , I n ) and u ∈ R n ∖ { 0 } u \in\mathbb{R}^n \setminus\{0\} u ∈ R n ∖ { 0 } . Then
E [ x x T 1 [ u T x > 0 ] ] = 1 2 I n . (56) \mathbb{E}[xx^{\mathsf{T}}1[u^{\mathsf{T}}x > 0]] = \frac{1}{2}I_n. \tag*{(56)} E [ x x T 1 [ u T x > 0 ]] = 2 1 I n . (56) Proof. Let A : = E [ x x T 1 [ u T x > 0 ] ] A := \mathbb{E}[xx^{\mathsf{T}}1[u^{\mathsf{T}}x > 0]] A := E [ x x T 1 [ u T x > 0 ]] , and decompose x = ( u T x / ∥ u ∥ 2 ) u + x ⊥ x = (u^{\mathsf{T}}x/\lVert u\rVert^2)u + x_\perp x = ( u T x / ∥ u ∥ 2 ) u + x ⊥ with x ⊥ x_\perp x ⊥ the projection onto u ⊥ u^\perp u ⊥ . In an orthonormal basis containing u / ∥ u ∥ u/\lVert u\rVert u / ∥ u ∥ , the off-diagonal entries of A A A vanish by the reflection symmetry of x ⊥ x_\perp x ⊥ ; the entry along u / ∥ u ∥ u/\lVert u\rVert u / ∥ u ∥ is E [ ( u T x / ∥ u ∥ ) 2 1 [ u T x > 0 ] ] = 1 2 \mathbb{E}[(u^{\mathsf{T}}x/\lVert u\rVert)^2 1[u^{\mathsf{T}}x > 0]] = \frac{1}{2} E [( u T x / ∥ u ∥ ) 2 1 [ u T x > 0 ]] = 2 1 (u T x u^{\mathsf{T}}x u T x symmetric), and each remaining diagonal entry is 1 2 E [ x j 2 ] = 1 2 \frac{1}{2}\mathbb{E}[x_j^2] = \frac{1}{2} 2 1 E [ x j 2 ] = 2 1 by independence of x ⊥ x_\perp x ⊥ from u T x u^{\mathsf{T}}x u T x and Pr ( u T x > 0 ) = 1 2 \Pr(u^{\mathsf{T}}x > 0) = \frac{1}{2} Pr ( u T x > 0 ) = 2 1 . Hence A = 1 2 I n A = \frac{1}{2}I_n A = 2 1 I n .
Proof of the gradient self outer-product of W 2 W_2 W 2
Lemma 22. G 2 = W 2 H − Φ C T G_2 = W_2H - \Phi C^{\mathsf{T}} G 2 = W 2 H − Φ C T , where H : = E x [ h ( x ) h ( x ) T ] ∈ R d 1 × d 1 H := \mathbb{E}_x[h(x)h(x)^{\mathsf{T}}] \in\mathbb{R}^{d_1 \times d_1} H := E x [ h ( x ) h ( x ) T ] ∈ R d 1 × d 1 and C : = E x [ h ( x ) x T ] ∈ R d 1 × d 0 C := \mathbb{E}_x[h(x)x^{\mathsf{T}}] \in\mathbb{R}^{d_1 \times d_0} C := E x [ h ( x ) x T ] ∈ R d 1 × d 0 .
Proof. Differentiating (46) in W 2 W_2 W 2 ,
G 2 = E x [ ( W 2 h ( x ) − Φ x ) h ( x ) T ] = W 2 E x [ h ( x ) h ( x ) T ] − Φ E x [ x h ( x ) T ] = W 2 H − Φ C T . G_2 = \mathbb{E}_x[(W_2h(x)-\Phi x)h(x)^{\mathsf{T}}] = W_2\mathbb{E}_x[h(x)h(x)^{\mathsf{T}}] - \Phi\mathbb{E}_x[xh(x)^{\mathsf{T}}] = W_2H - \Phi C^{\mathsf{T}}. G 2 = E x [( W 2 h ( x ) − Φ x ) h ( x ) T ] = W 2 E x [ h ( x ) h ( x ) T ] − Φ E x [ x h ( x ) T ] = W 2 H − Φ C T .
Lemma 23. C = 1 2 W 1 C = \frac{1}{2}W_1 C = 2 1 W 1 .
Proof. C a j = E x [ σ ( w a T x ) x j ] C_{aj} = \mathbb{E}_x[\sigma(w_a^{\mathsf{T}}x)x_j] C aj = E x [ σ ( w a T x ) x j ] , and f ( x ) : = σ ( w a T x ) f(x) := \sigma(w_a^{\mathsf{T}}x) f ( x ) := σ ( w a T x ) is Lipschitz with ∂ j f ( x ) = 1 [ w a T x > 0 ] ( w a ) j \partial_j f(x) = 1[w_a^{\mathsf{T}}x > 0](w_a)_j ∂ j f ( x ) = 1 [ w a T x > 0 ] ( w a ) j a.e. By Lemma 17 and Lemma 18 ,
C a j = E x [ ∂ j f ( x ) ] = ( w a ) j E x [ σ ′ ( w a T x ) ] = 1 2 ( w a ) j . C_{aj} = \mathbb{E}_x[\partial_j f(x)] = (w_a)_j\mathbb{E}_x[\sigma'(w_a^{\mathsf{T}}x)] = \frac{1}{2}(w_a)_j. C aj = E x [ ∂ j f ( x )] = ( w a ) j E x [ σ ′ ( w a T x )] = 2 1 ( w a ) j .
Proof of (49) . By Lemma 22,
G 2 G 2 ⊤ = W 2 H 2 W 2 ⊤ − W 2 H C Φ ⊤ − Φ C ⊤ H W 2 ⊤ + Φ C ⊤ C Φ ⊤ . G_2G_2^\top= W_2H^2W_2^\top- W_2HC\Phi^\top- \Phi C^\top HW_2^\top+ \Phi C^\top C\Phi^\top. G 2 G 2 ⊤ = W 2 H 2 W 2 ⊤ − W 2 H C Φ ⊤ − Φ C ⊤ H W 2 ⊤ + Φ C ⊤ C Φ ⊤ .
Since E [ W 2 ] = 0 \mathbb{E}[W_2]=0 E [ W 2 ] = 0 , the cross terms vanish under E W 2 [ ⋅ ∣ W 1 ] \mathbb{E}_{W_2}[\cdot\mid W_1] E W 2 [ ⋅ ∣ W 1 ] , and by Lemma 16 (left identity, M = H 2 M=H^2 M = H 2 ),
E [ G 2 G 2 ⊤ ∣ W 1 ] = tr ( H 2 ) I d 2 + Φ C ⊤ C Φ ⊤ . (57) \mathbb{E}[G_2G_2^\top\mid W_1]=\operatorname{tr}(H^2)I_{d_2}+\Phi C^\top C\Phi^\top. \tag*{(57)} E [ G 2 G 2 ⊤ ∣ W 1 ] = tr ( H 2 ) I d 2 + Φ C ⊤ C Φ ⊤ . (57)
By Lemma 23, C ⊤ C = 1 4 W 1 ⊤ W 1 C^\top C=\frac{1}{4}W_1^\top W_1 C ⊤ C = 4 1 W 1 ⊤ W 1 , and Lemma 16 (right identity, N = I d 1 N=I_{d_1} N = I d 1 ) gives E W 1 [ W 1 ⊤ W 1 ] = d 1 I d 0 \mathbb{E}_{W_1}[W_1^\top W_1]=d_1I_{d_0} E W 1 [ W 1 ⊤ W 1 ] = d 1 I d 0 , so E W 1 [ C ⊤ C ] = d 1 4 I d 0 \mathbb{E}_{W_1}[C^\top C]=\frac{d_1}{4}I_{d_0} E W 1 [ C ⊤ C ] = 4 d 1 I d 0 . Taking E W 1 \mathbb{E}_{W_1} E W 1 of (57) ,
E [ G 2 G 2 ⊤ ] = E W 1 [ tr ( H 2 ) ] I d 2 + d 1 4 Φ Φ ⊤ = α I d 2 + d 1 4 Φ Φ ⊤ . \mathbb{E}[G_2G_2^\top]=\mathbb{E}_{W_1}[\operatorname{tr}(H^2)]I_{d_2}+\frac{d_1}{4}\Phi\Phi^\top=\alpha I_{d_2}+\frac{d_1}{4}\Phi\Phi^\top. E [ G 2 G 2 ⊤ ] = E W 1 [ tr ( H 2 )] I d 2 + 4 d 1 Φ Φ ⊤ = α I d 2 + 4 d 1 Φ Φ ⊤ .
For the order of α \alpha α , split tr ( H 2 ) = ∑ a H a a 2 + ∑ a ≠ b H a b 2 \operatorname{tr}(H^2)=\sum_a H_{aa}^2+\sum_{a\ne b}H_{ab}^2 tr ( H 2 ) = ∑ a H aa 2 + ∑ a = b H ab 2 . By Lemma 18, H a a = 1 2 ∥ w a ∥ 2 H_{aa}=\frac{1}{2}\lVert w_a\rVert^2 H aa = 2 1 ∥ w a ∥ 2 with ∥ w a ∥ 2 ∼ χ d 0 2 \lVert w_a\rVert^2\sim\chi^2_{d_0} ∥ w a ∥ 2 ∼ χ d 0 2 , so
∑ a E [ H a a 2 ] = d 1 d 0 ( d 0 + 2 ) 4 = Θ ( d 1 ) . \sum_a\mathbb{E}[H_{aa}^2]=\frac{d_1d_0(d_0+2)}{4}=\Theta(d_1). a ∑ E [ H aa 2 ] = 4 d 1 d 0 ( d 0 + 2 ) = Θ ( d 1 ) .
By (54) , H a b = ∥ w a ∥ ∥ w b ∥ g ( θ a b ) H_{ab}=\lVert w_a\rVert\lVert w_b\rVert g(\theta_{ab}) H ab = ∥ w a ∥ ∥ w b ∥ g ( θ ab ) with g ( θ ) = ( sin θ + ( π − θ ) cos θ ) 2 π g(\theta)=\frac{(\sin\theta+(\pi-\theta)\cos\theta)}{2\pi} g ( θ ) = 2 π ( s i n θ + ( π − θ ) c o s θ ) and θ a b = ∠ ( w a , w b ) \theta_{ab}=\angle(w_a,w_b) θ ab = ∠ ( w a , w b ) independent of ∥ w a ∥ , ∥ w b ∥ \lVert w_a\rVert,\lVert w_b\rVert ∥ w a ∥ , ∥ w b ∥ , so E [ H a b 2 ] = d 0 2 E [ g ( θ ) 2 ] \mathbb{E}[H_{ab}^2]=d_0^2\mathbb{E}[g(\theta)^2] E [ H ab 2 ] = d 0 2 E [ g ( θ ) 2 ] , where E [ g ( θ ) 2 ] > 0 \mathbb{E}[g(\theta)^2]>0 E [ g ( θ ) 2 ] > 0 , and
∑ a ≠ b E [ H a b 2 ] = d 1 ( d 1 − 1 ) d 0 2 E [ g ( θ ) 2 ] = Θ ( d 1 2 ) . \sum_{a\ne b}\mathbb{E}[H_{ab}^2]=d_1(d_1-1)d_0^2\mathbb{E}[g(\theta)^2]=\Theta(d_1^2). a = b ∑ E [ H ab 2 ] = d 1 ( d 1 − 1 ) d 0 2 E [ g ( θ ) 2 ] = Θ ( d 1 2 ) .
The off-diagonal term dominates, so α = Θ ( d 1 2 ) \alpha=\Theta(d_1^2) α = Θ ( d 1 2 ) .
Proof of the gradient self outer-product of W 1 W_1 W 1
By the chain rule, with residual r ( x ) : = W 2 σ ( W 1 x ) − Φ x r(x):=W_2\sigma(W_1x)-\Phi x r ( x ) := W 2 σ ( W 1 x ) − Φ x and D ( x ) : = diag ( σ ′ ( W 1 x ) ) D(x):=\operatorname{diag}(\sigma'(W_1x)) D ( x ) := diag ( σ ′ ( W 1 x )) ,
G 1 = G 1 ( 1 ) − G 1 ( 2 ) , G_1=G_1^{(1)}-G_1^{(2)}, G 1 = G 1 ( 1 ) − G 1 ( 2 ) ,
G 1 ( 1 ) : = E x [ D ( x ) W 2 ⊤ W 2 σ ( W 1 x ) x ⊤ ] , G 1 ( 2 ) : = E x [ D ( x ) W 2 ⊤ Φ x x ⊤ ] . (58) G_1^{(1)}:=\mathbb{E}_x[D(x)W_2^\top W_2\sigma(W_1x)x^\top],\qquad G_1^{(2)}:=\mathbb{E}_x[D(x)W_2^\top\Phi xx^\top]. \tag*{(58)} G 1 ( 1 ) := E x [ D ( x ) W 2 ⊤ W 2 σ ( W 1 x ) x ⊤ ] , G 1 ( 2 ) := E x [ D ( x ) W 2 ⊤ Φ x x ⊤ ] . (58)
G 1 ( 1 ) G_1^{(1)} G 1 ( 1 ) is quadratic in W 2 W_2 W 2 and G 1 ( 2 ) G_1^{(2)} G 1 ( 2 ) is linear, so the cross terms in G 1 G 1 ⊤ G_1G_1^\top G 1 G 1 ⊤ are odd in W 2 W_2 W 2 and vanish under E W 2 [ ⋅ ∣ W 1 ] \mathbb{E}_{W_2}[\cdot\mid W_1] E W 2 [ ⋅ ∣ W 1 ] :
E W 2 [ G 1 G 1 ⊤ ∣ W 1 ] = E W 2 [ G 1 ( 1 ) ( G 1 ( 1 ) ) ⊤ ∣ W 1 ] + E W 2 [ G 1 ( 2 ) ( G 1 ( 2 ) ) ⊤ ∣ W 1 ] . (59) \mathbb{E}_{W_2}[G_1G_1^\top\mid W_1]=\mathbb{E}_{W_2}[G_1^{(1)}(G_1^{(1)})^\top\mid W_1]+\mathbb{E}_{W_2}[G_1^{(2)}(G_1^{(2)})^\top\mid W_1]. \tag*{(59)} E W 2 [ G 1 G 1 ⊤ ∣ W 1 ] = E W 2 [ G 1 ( 1 ) ( G 1 ( 1 ) ) ⊤ ∣ W 1 ] + E W 2 [ G 1 ( 2 ) ( G 1 ( 2 ) ) ⊤ ∣ W 1 ] . (59)
Lemma 24. G 1 ( 2 ) = 1 2 W 2 ⊤ Φ G_1^{(2)}=\frac{1}{2}W_2^\top\Phi G 1 ( 2 ) = 2 1 W 2 ⊤ Φ , and E W 2 [ G 1 ( 2 ) ( G 1 ( 2 ) ) ⊤ ∣ W 1 ] = ∥ Φ ∥ F 2 4 I d 1 \mathbb{E}_{W_2}[G_1^{(2)}(G_1^{(2)})^\top\mid W_1]=\frac{\lVert\Phi\rVert_F^2}{4}I_{d_1} E W 2 [ G 1 ( 2 ) ( G 1 ( 2 ) ) ⊤ ∣ W 1 ] = 4 ∥ Φ ∥ F 2 I d 1 .
Proof. Let v a : = Φ ⊤ ( W 2 ) : , a ∈ R d 0 v_a:=\Phi^\top(W_2)_{:,a}\in\mathbb{R}^{d_0} v a := Φ ⊤ ( W 2 ) : , a ∈ R d 0 , so ( W 2 ⊤ Φ ) a b = ( v a ) b (W_2^\top\Phi)_{ab}=(v_a)_b ( W 2 ⊤ Φ ) ab = ( v a ) b . Then, by Lemma 21,
( G 1 ( 2 ) ) a b = E x [ 1 [ w a ⊤ x > 0 ] x b ( x ⊤ v a ) ] = ( E x [ 1 [ w a ⊤ x > 0 ] x x ⊤ ] v a ) b = 1 2 ( v a ) b , (G_1^{(2)})_{ab}=\mathbb{E}_x[1[w_a^\top x>0]x_b(x^\top v_a)]=\left(\mathbb{E}_x[1[w_a^\top x>0]xx^\top]v_a\right)_b=\frac{1}{2}(v_a)_b, ( G 1 ( 2 ) ) ab = E x [ 1 [ w a ⊤ x > 0 ] x b ( x ⊤ v a )] = ( E x [ 1 [ w a ⊤ x > 0 ] x x ⊤ ] v a ) b = 2 1 ( v a ) b , so G 1 ( 2 ) = 1 2 W 2 ⊤ Φ G_1^{(2)}=\frac{1}{2}W_2^\top\Phi G 1 ( 2 ) = 2 1 W 2 ⊤ Φ and G 1 ( 2 ) ( G 1 ( 2 ) ) ⊤ = 1 4 W 2 ⊤ Φ Φ ⊤ W 2 G_1^{(2)}(G_1^{(2)})^\top=\frac{1}{4}W_2^\top\Phi\Phi^\top W_2 G 1 ( 2 ) ( G 1 ( 2 ) ) ⊤ = 4 1 W 2 ⊤ Φ Φ ⊤ W 2 ; Lemma 16 (right identity, N = Φ Φ ⊤ N=\Phi\Phi^\top N = Φ Φ ⊤ ) gives E W 2 [ W 2 ⊤ Φ Φ ⊤ W 2 ] = ∥ Φ ∥ F 2 I d 1 \mathbb{E}_{W_2}[W_2^\top\Phi\Phi^\top W_2]=\lVert\Phi\rVert_F^2I_{d_1} E W 2 [ W 2 ⊤ Φ Φ ⊤ W 2 ] = ∥ Φ ∥ F 2 I d 1 .
Lemma 25. Define F b ( u , v ) : = E x [ σ ′ ( u ⊤ x ) σ ( v ⊤ x ) x b ] F_b(u,v):=\mathbb{E}_x[\sigma'(u^\top x)\sigma(v^\top x)x_b] F b ( u , v ) := E x [ σ ′ ( u ⊤ x ) σ ( v ⊤ x ) x b ] . Then F b ( u , u ) = 1 2 u b F_b(u,u)=\frac{1}{2}u_b F b ( u , u ) = 2 1 u b , and for u ∉ span ( v ) u\notin\operatorname{span}(v) u ∈ / span ( v ) ,
F b ( u , v ) = u b ⋅ ∥ v ∥ sin θ 2 π ∥ u ∥ + v b ⋅ π − θ 2 π , θ : = ∠ ( u , v ) . (60) F_b(u,v)=u_b\cdot\frac{\lVert v\rVert\sin\theta}{2\pi\lVert u\rVert}+v_b\cdot\frac{\pi-\theta}{2\pi},\qquad\theta:=\angle(u,v). \tag*{(60)} F b ( u , v ) = u b ⋅ 2 π ∥ u ∥ ∥ v ∥ sin θ + v b ⋅ 2 π π − θ , θ := ∠ ( u , v ) . (60)
Proof. For u = v u=v u = v , σ ′ ( u ⊤ x ) σ ( u ⊤ x ) = σ ( u ⊤ x ) \sigma'(u^\top x)\sigma(u^\top x)=\sigma(u^\top x) σ ′ ( u ⊤ x ) σ ( u ⊤ x ) = σ ( u ⊤ x ) a.e., so by Lemma 17 (applied to the Lipschitz x ↦ σ ( u ⊤ x ) x\mapsto\sigma(u^\top x) x ↦ σ ( u ⊤ x ) ) and Lemma 18 , F b ( u , u ) = E x [ σ ( u ⊤ x ) x b ] = E x [ σ ′ ( u ⊤ x ) ] u b = 1 2 u b F_b(u,u)=\mathbb{E}_x[\sigma(u^\top x)x_b]=\mathbb{E}_x[\sigma'(u^\top x)]u_b=\frac{1}{2}u_b F b ( u , u ) = E x [ σ ( u ⊤ x ) x b ] = E x [ σ ′ ( u ⊤ x )] u b = 2 1 u b . For u ≠ v u\ne v u = v , f ( x ) = σ ′ ( u ⊤ x ) σ ( v ⊤ x ) f(x)=\sigma'(u^\top x)\sigma(v^\top x) f ( x ) = σ ′ ( u ⊤ x ) σ ( v ⊤ x ) is discontinuous across { u ⊤ x = 0 } \{u^\top x=0\} { u ⊤ x = 0 } , so we differentiate the Cho–Saul kernel instead. With K ( u , v ) : = E x [ σ ( u ⊤ x ) σ ( v ⊤ x ) ] K(u,v):=\mathbb{E}_x[\sigma(u^\top x)\sigma(v^\top x)] K ( u , v ) := E x [ σ ( u ⊤ x ) σ ( v ⊤ x )] , dominated convergence (since σ \sigma σ is Lipschitz with linear growth) gives
∂ K ∂ u b ( u , v ) = E x [ σ ′ ( u ⊤ x ) x b σ ( v ⊤ x ) ] = F b ( u , v ) . \frac{\partial K}{\partial u_b}(u,v)=\mathbb{E}_x[\sigma'(u^\top x)x_b\sigma(v^\top x)]=F_b(u,v). ∂ u b ∂ K ( u , v ) = E x [ σ ′ ( u ⊤ x ) x b σ ( v ⊤ x )] = F b ( u , v ) .
By Lemma 19 , K ( u , v ) = ∥ u ∥ ∥ v ∥ 2 π ( sin θ + ( π − θ ) cos θ ) K(u,v)=\frac{\lVert u\rVert\lVert v\rVert}{2\pi}(\sin\theta+(\pi-\theta)\cos\theta) K ( u , v ) = 2 π ∥ u ∥ ∥ v ∥ ( sin θ + ( π − θ ) cos θ ) , smooth for u ∉ span ( v ) u\notin\operatorname{span}(v) u ∈ / span ( v ) ; differentiating with ∂ ∥ u ∥ / ∂ u b = u b / ∥ u ∥ \partial\lVert u\rVert/\partial u_b=u_b/\lVert u\rVert ∂ ∥ u ∥ / ∂ u b = u b / ∥ u ∥ , ∂ cos θ / ∂ u b = 1 ∥ u ∥ ( v b / ∥ v ∥ − cos θ u b / ∥ u ∥ ) \partial\cos\theta/\partial u_b=\frac{1}{\lVert u\rVert}(v_b/\lVert v\rVert-\cos\theta\,u_b/\lVert u\rVert) ∂ cos θ / ∂ u b = ∥ u ∥ 1 ( v b / ∥ v ∥ − cos θ u b / ∥ u ∥) , ∂ θ / ∂ u b = − 1 sin θ ∂ cos θ / ∂ u b \partial\theta/\partial u_b=-\frac{1}{\sin\theta}\partial\cos\theta/\partial u_b ∂ θ / ∂ u b = − s i n θ 1 ∂ cos θ / ∂ u b , and simplifying yields (60) .
Lemma 26. E W 2 [ ( W 2 ⊤ W 2 ) a k ( W 2 ⊤ W 2 ) c l ] = d 2 2 δ a k δ c l + d 2 δ a c δ k l + d 2 δ a l δ c k \mathbb{E}_{W_2}[(W_2^\top W_2)_{ak}(W_2^\top W_2)_{cl}]=d_2^2\delta_{ak}\delta_{cl}+d_2\delta_{ac}\delta_{kl}+d_2\delta_{al}\delta_{ck} E W 2 [( W 2 ⊤ W 2 ) ak ( W 2 ⊤ W 2 ) c l ] = d 2 2 δ ak δ c l + d 2 δ a c δ k l + d 2 δ a l δ c k .
Proof. ( W 2 ⊤ W 2 ) a k = ∑ i = 1 d 2 ( W 2 ) i a ( W 2 ) i k (W_2^\top W_2)_{ak}=\sum_{i=1}^{d_2}(W_2)_{ia}(W_2)_{ik} ( W 2 ⊤ W 2 ) ak = ∑ i = 1 d 2 ( W 2 ) ia ( W 2 ) ik , and applying Lemma 20 to the four Gaussian factors ( W 2 ) i a (W_2)_{ia} ( W 2 ) ia , ( W 2 ) i k (W_2)_{ik} ( W 2 ) ik , ( W 2 ) j c (W_2)_{jc} ( W 2 ) j c , ( W 2 ) j l (W_2)_{jl} ( W 2 ) j l (summed over i , j i,j i , j ) the three pairings give
δ a k δ c l d 2 2 + δ a c δ k l d 2 + δ a l δ c k d 2 . \delta_{ak}\delta_{cl}d_2^2+\delta_{ac}\delta_{kl}d_2+\delta_{al}\delta_{ck}d_2. δ ak δ c l d 2 2 + δ a c δ k l d 2 + δ a l δ c k d 2 .
Proof of (50) –(51) . Permutation invariance follows from the exchangeability of the rows of W 1 W_1 W 1 and columns of W 2 W_2 W 2 . The G 1 ( 2 ) G_1^{(2)} G 1 ( 2 ) contribution to (59) is ∥ Φ ∥ 2 4 I d 1 \frac{\lVert\Phi\rVert^2}{4}I_{d_1} 4 ∥ Φ ∥ 2 I d 1 by Lemma 24. For G 1 ( 1 ) G_1^{(1)} G 1 ( 1 ) , we have ( G 1 ( 1 ) ) a b = ∑ k = 1 d 1 ( W 2 ⊤ W 2 ) a k F b ( w a , w k ) (G_1^{(1)})_{ab}=\sum_{k=1}^{d_1}(W_2^\top W_2)_{ak}F_b(w_a,w_k) ( G 1 ( 1 ) ) ab = ∑ k = 1 d 1 ( W 2 ⊤ W 2 ) ak F b ( w a , w k ) , so
E W 2 [ ( G 1 ( 1 ) ( G 1 ( 1 ) ) ⊤ ) a c ∣ W 1 ] = ∑ b , k , l E W 2 [ ( W 2 ⊤ W 2 ) a k ( W 2 ⊤ W 2 ) c l ] F b ( w a , w k ) F b ( w c , w l ) . \mathbb{E}_{W_2}[(G_1^{(1)}(G_1^{(1)})^\top)_{ac}\mid W_1]=\sum_{b,k,l}\mathbb{E}_{W_2}[(W_2^\top W_2)_{ak}(W_2^\top W_2)_{cl}]F_b(w_a,w_k)F_b(w_c,w_l). E W 2 [( G 1 ( 1 ) ( G 1 ( 1 ) ) ⊤ ) a c ∣ W 1 ] = b , k , l ∑ E W 2 [( W 2 ⊤ W 2 ) ak ( W 2 ⊤ W 2 ) c l ] F b ( w a , w k ) F b ( w c , w l ) .
and substituting Lemma 26 gives T 1 ( a c ) + T 2 ( a c ) + T 3 ( a c ) T_1^{(ac)}+T_2^{(ac)}+T_3^{(ac)} T 1 ( a c ) + T 2 ( a c ) + T 3 ( a c ) with
T 1 ( a c ) = d 2 2 ∑ b F b ( w a , w a ) F b ( w c , w c ) , (61) T_1^{(ac)}=d_2^2\sum_b F_b(w_a,w_a)F_b(w_c,w_c), \tag*{(61)} T 1 ( a c ) = d 2 2 b ∑ F b ( w a , w a ) F b ( w c , w c ) , (61)
T 2 ( a c ) = d 2 δ a c ∑ k = 1 d 1 ∑ b F b ( w a , w k ) 2 , (62) T_2^{(ac)}=d_2\delta_{ac}\sum_{k=1}^{d_1}\sum_b F_b(w_a,w_k)^2, \tag*{(62)} T 2 ( a c ) = d 2 δ a c k = 1 ∑ d 1 b ∑ F b ( w a , w k ) 2 , (62)
T 3 ( a c ) = d 2 ∑ b F b ( w a , w c ) F b ( w c , w a ) . (63) T_3^{(ac)}=d_2\sum_b F_b(w_a,w_c)F_b(w_c,w_a). \tag*{(63)} T 3 ( a c ) = d 2 b ∑ F b ( w a , w c ) F b ( w c , w a ) . (63)
Term T 1 T_1 T 1 . By Lemma 25, F b ( w a , w a ) = 1 2 ( w a ) b F_b(w_a,w_a)=\frac{1}{2}(w_a)_b F b ( w a , w a ) = 2 1 ( w a ) b , so T 1 ( a c ) = d 2 2 4 w a ⊤ w c T_1^{(ac)}=\frac{d_2^2}{4}w_a^\top w_c T 1 ( a c ) = 4 d 2 2 w a ⊤ w c , giving E [ T 1 ( a a ) ] = d 0 d 2 2 4 \mathbb{E}[T_1^{(aa)}]=\frac{d_0d_2^2}{4} E [ T 1 ( aa ) ] = 4 d 0 d 2 2 and E [ T 1 ( a c ) ] = 0 \mathbb{E}[T_1^{(ac)}]=0 E [ T 1 ( a c ) ] = 0 for a ≠ c a\ne c a = c .
Term T 2 T_2 T 2 . For u ≠ v u\ne v u = v , let A : = ∥ v ∥ sin θ 2 π ∥ u ∥ A:=\frac{\lVert v\rVert\sin\theta}{2\pi\lVert u\rVert} A := 2 π ∥ u ∥ ∥ v ∥ s i n θ and B : = π − θ 2 π B:=\frac{\pi-\theta}{2\pi} B := 2 π π − θ in (60) ; using u ⊤ v = ∥ u ∥ ∥ v ∥ cos θ u^\top v=\lVert u\rVert\lVert v\rVert\cos\theta u ⊤ v = ∥ u ∥ ∥ v ∥ cos θ ,
∑ b F b ( u , v ) 2 = A 2 ∥ u ∥ 2 + 2 A B ( u ⊤ v ) + B 2 ∥ v ∥ 2 = ∥ v ∥ 2 ψ ( θ ) , \sum_b F_b(u,v)^2=A^2\lVert u\rVert^2+2AB(u^\top v)+B^2\lVert v\rVert^2=\lVert v\rVert^2\psi(\theta), b ∑ F b ( u , v ) 2 = A 2 ∥ u ∥ 2 + 2 A B ( u ⊤ v ) + B 2 ∥ v ∥ 2 = ∥ v ∥ 2 ψ ( θ ) ,
ψ ( θ ) : = sin 2 θ + ( π − θ ) 2 4 π 2 + sin θ cos θ ( π − θ ) 2 π 2 . \psi(\theta):=\frac{\sin^2\theta+(\pi-\theta)^2}{4\pi^2}+\frac{\sin\theta\cos\theta(\pi-\theta)}{2\pi^2}. ψ ( θ ) := 4 π 2 sin 2 θ + ( π − θ ) 2 + 2 π 2 sin θ cos θ ( π − θ ) .
In T 2 ( a a ) T_2^{(aa)} T 2 ( aa ) the term k = a k=a k = a contributes ∑ b F b ( w a , w a ) 2 = 1 4 ∥ w a ∥ 2 \sum_bF_b(w_a,w_a)^2=\frac{1}{4}\lVert w_a\rVert^2 ∑ b F b ( w a , w a ) 2 = 4 1 ∥ w a ∥ 2 (expectation d 0 4 \frac{d_0}{4} 4 d 0 ), and each k ≠ a k\ne a k = a contributes E [ ∥ w k ∥ 2 ] E [ ψ ( θ a k ) ] = d 0 κ 1 \mathbb{E}[\lVert w_k\rVert^2]\mathbb{E}[\psi(\theta_{ak})]=d_0\kappa_1 E [∥ w k ∥ 2 ] E [ ψ ( θ ak )] = d 0 κ 1 , so
E [ T 2 ( a a ) ] = d 2 [ d 0 4 + ( d 1 − 1 ) d 0 κ 1 ] = d 0 d 2 [ ( d 1 − 1 ) κ 1 + 1 4 ] , \mathbb{E}[T_2^{(aa)}]=d_2\left[\frac{d_0}{4}+(d_1-1)d_0\kappa_1\right]=d_0d_2\left[(d_1-1)\kappa_1+\frac{1}{4}\right], E [ T 2 ( aa ) ] = d 2 [ 4 d 0 + ( d 1 − 1 ) d 0 κ 1 ] = d 0 d 2 [ ( d 1 − 1 ) κ 1 + 4 1 ] ,
and T 2 ( a c ) = 0 T_2^{(ac)}=0 T 2 ( a c ) = 0 for a ≠ c a\ne c a = c .
Term T 3 T_3 T 3 . For a = c a=c a = c , the sum ∑ b F b ( w a , w a ) 2 \sum_bF_b(w_a,w_a)^2 ∑ b F b ( w a , w a ) 2 equals 1 4 ∥ w a ∥ 2 \frac{1}{4}\lVert w_a\rVert^2 4 1 ∥ w a ∥ 2 , so E [ T 3 ( a a ) ] = d 0 d 2 4 \mathbb{E}[T_3^{(aa)}]=\frac{d_0d_2}{4} E [ T 3 ( aa ) ] = 4 d 0 d 2 . For a ≠ c a\ne c a = c , expanding F b ( w a , w c ) F b ( w c , w a ) F_b(w_a,w_c)F_b(w_c,w_a) F b ( w a , w c ) F b ( w c , w a ) via (60) (with θ : = θ a c \theta:=\theta_{ac} θ := θ a c ) and summing over b b b ,
∑ b F b ( w a , w c ) F b ( w c , w a ) = ∥ w a ∥ ∥ w c ∥ 4 π 2 [ cos θ ( sin 2 θ + ( π − θ ) 2 ) + 2 sin θ ( π − θ ) ] . \sum_bF_b(w_a,w_c)F_b(w_c,w_a)=\frac{\lVert w_a\rVert\lVert w_c\rVert}{4\pi^2}\left[\cos\theta\left(\sin^2\theta+(\pi-\theta)^2\right)+2\sin\theta(\pi-\theta)\right]. b ∑ F b ( w a , w c ) F b ( w c , w a ) = 4 π 2 ∥ w a ∥ ∥ w c ∥ [ cos θ ( sin 2 θ + ( π − θ ) 2 ) + 2 sin θ ( π − θ ) ] .
Here ∥ w a ∥ ∥ w c ∥ \lVert w_a\rVert\lVert w_c\rVert ∥ w a ∥ ∥ w c ∥ is independent of θ a c \theta_{ac} θ a c with E [ ∥ w a ∥ ∥ w c ∥ ] = d 0 + O ( 1 ) \mathbb{E}[\lVert w_a\rVert\lVert w_c\rVert] = d_0 + O(1) E [∥ w a ∥ ∥ w c ∥] = d 0 + O ( 1 ) ; as d 0 → ∞ d_0 \to\infty d 0 → ∞ , θ a c \theta_{ac} θ a c concentrates at π / 2 \pi/2 π /2 with E [ cos θ a c ] = 0 \mathbb{E}[\cos\theta_{ac}] = 0 E [ cos θ a c ] = 0 and Var ( cos θ a c ) = 1 / d 0 \operatorname{Var}(\cos\theta_{ac}) = 1/d_0 Var ( cos θ a c ) = 1/ d 0 , so ∣ E [ cos θ a c q ( θ a c ) ] ∣ = O ( 1 / d 0 ) \lvert\mathbb{E}[\cos\theta_{ac}q(\theta_{ac})]\rvert= O(1/\sqrt{d_0}) ∣ E [ cos θ a c q ( θ a c )]∣ = O ( 1/ d 0 ) for bounded q q q , while E [ sin θ a c ( π − θ a c ) ] = Θ ( 1 ) \mathbb{E}[\sin\theta_{ac}(\pi-\theta_{ac})] = \Theta(1) E [ sin θ a c ( π − θ a c )] = Θ ( 1 ) . The cos θ ( ⋅ ) \cos\theta(\cdot) cos θ ( ⋅ ) part contributes O ( d 2 d 0 ) O(d_2\sqrt{d_0}) O ( d 2 d 0 ) and the 2 sin θ ( π − θ ) 2\sin\theta(\pi-\theta) 2 sin θ ( π − θ ) part gives d 0 d 2 κ 2 + O ( d 2 ) d_0d_2\kappa_2 + O(d_2) d 0 d 2 κ 2 + O ( d 2 ) , so E [ T 3 ( a c ) ] = d 0 d 2 κ 2 + O ( d 2 d 0 ) \mathbb{E}[T_3^{(ac)}] = d_0d_2\kappa_2 + O(d_2\sqrt{d_0}) E [ T 3 ( a c ) ] = d 0 d 2 κ 2 + O ( d 2 d 0 ) .
Combining. For a = c a=c a = c , adding the three terms and the G 1 ( 2 ) G_1^{(2)} G 1 ( 2 ) contribution and merging the two d 0 d 2 4 \frac{d_0d_2}{4} 4 d 0 d 2 pieces,
E [ ( G 1 G 1 ⊤ ) a a ] = d 0 d 2 2 4 + d 0 d 2 [ ( d 1 − 1 ) κ 1 + 1 2 ] + ∥ Φ ∥ F 2 4 ; \mathbb{E}[(G_1G_1^\top)_{aa}] = \frac{d_0d_2^2}{4} + d_0d_2\left[(d_1-1)\kappa_1+\frac{1}{2}\right] + \frac{\lVert\Phi\rVert_{\mathrm F}^2}{4}; E [( G 1 G 1 ⊤ ) aa ] = 4 d 0 d 2 2 + d 0 d 2 [ ( d 1 − 1 ) κ 1 + 2 1 ] + 4 ∥ Φ ∥ F 2 ;
for a ≠ c a\ne c a = c only T 3 ( a c ) T_3^{(ac)} T 3 ( a c ) survives, E [ ( G 1 G 1 ⊤ ) a c ] = d 0 d 2 κ 2 + O ( d 2 d 0 ) \mathbb{E}[(G_1G_1^\top)_{ac}] = d_0d_2\kappa_2+O(d_2\sqrt{d_0}) E [( G 1 G 1 ⊤ ) a c ] = d 0 d 2 κ 2 + O ( d 2 d 0 ) . Finally ψ ( π / 2 ) = 1 + ( π / 2 ) 2 4 π 2 = 1 4 π 2 + 1 16 \psi(\pi/2)=\frac{1+(\pi/2)^2}{4\pi^2}=\frac{1}{4\pi^2}+\frac{1}{16} ψ ( π /2 ) = 4 π 2 1 + ( π /2 ) 2 = 4 π 2 1 + 16 1 and sin ( π / 2 ) ( π − π / 2 ) / ( 2 π 2 ) = 1 4 π \sin(\pi/2)(\pi-\pi/2)/(2\pi^2)=\frac{1}{4\pi} sin ( π /2 ) ( π − π /2 ) / ( 2 π 2 ) = 4 π 1 give the high-dimensional limits of κ 1 , κ 2 \kappa_1,\kappa_2 κ 1 , κ 2 .
Discussion
At initialization the gradient of W 2 W_2 W 2 is mostly target-independent: the Θ ( d 1 2 ) \Theta(d_1^2) Θ ( d 1 2 ) leading order of E [ ( G 2 G 2 ⊤ ) a a ] \mathbb{E}[(G_2G_2^\top)_{aa}] E [( G 2 G 2 ⊤ ) aa ] comes entirely from tr ( H 2 ) \operatorname{tr}(H^2) tr ( H 2 ) , which depends only on W 1 W_1 W 1 , while the target term d 1 4 ( Φ Φ ⊤ ) a a \frac{d_1}{4}(\Phi\Phi^\top)_{aa} 4 d 1 ( Φ Φ ⊤ ) aa is dominated unless ∥ Φ ∥ F 2 \lVert\Phi\rVert_{\mathrm F}^2 ∥ Φ ∥ F 2 is exceptionally large. The nonzero Θ ( 1 ) \Theta(1) Θ ( 1 ) off-diagonal of E [ G 1 G 1 ⊤ ] \mathbb{E}[G_1G_1^\top] E [ G 1 G 1 ⊤ ] comes from T 3 T_3 T 3 and reflects ReLU’s lack of odd symmetry: in a deep linear network the symmetry W 1 ↦ D W 1 W_1\mapsto DW_1 W 1 ↦ D W 1 , W 2 ↦ W 2 D W_2\mapsto W_2D W 2 ↦ W 2 D for D = diag ( ± 1 ) D=\operatorname{diag}(\pm1) D = diag ( ± 1 ) preserves the loss and forces off-diagonal expectations to zero (cf. Appendix § 11 ), whereas σ ( − z ) ≠ − σ ( z ) \sigma(-z)\ne-\sigma(z) σ ( − z ) = − σ ( z ) breaks it. The dominance is therefore asymptotic in d 1 d_1 d 1 rather than exact.
Single-Layer Linear Attention
Setup
Let T T T , d 0 d_0 d 0 , d k d_k d k , d v d_v d v , d 2 d_2 d 2 be positive integers. Single-layer linear attention with tokens X ∈ R T × d 0 X\in\mathbb{R}^{T\times d_0} X ∈ R T × d 0 (one per row) and weights W Q , W K ∈ R d k × d 0 W_Q,W_K\in\mathbb{R}^{d_k\times d_0} W Q , W K ∈ R d k × d 0 , W V ∈ R d v × d 0 W_V\in\mathbb{R}^{d_v\times d_0} W V ∈ R d v × d 0 , W O ∈ R d 2 × d v W_O\in\mathbb{R}^{d_2\times d_v} W O ∈ R d 2 × d v , and its loss against a deterministic linear target Φ ∈ R d 2 × d 0 \Phi\in\mathbb{R}^{d_2\times d_0} Φ ∈ R d 2 × d 0 , are
A : = 1 d k X W Q ⊤ W K X ⊤ , f ( X ) = A X W V ⊤ W O ⊤ , L : = 1 2 ∥ f ( X ) − X Φ ⊤ ∥ F 2 . (64) A := \frac{1}{\sqrt{d_k}}XW_Q^\top W_KX^\top,\qquad f(X)=AXW_V^\top W_O^\top,\qquad\mathcal{L}:=\frac{1}{2}\lVert f(X)-X\Phi^\top\rVert_{\mathrm F}^2. \tag*{(64)} A := d k 1 X W Q ⊤ W K X ⊤ , f ( X ) = A X W V ⊤ W O ⊤ , L := 2 1 ∥ f ( X ) − X Φ ⊤ ∥ F 2 . (64)
Let
Z : = X W V ⊤ , R : = f ( X ) − X Φ ⊤ , Y : = X Φ ⊤ , C : = X ⊤ X , c p : = tr ( C p ) , (65) Z:=XW_V^\top,\qquad R:=f(X)-X\Phi^\top,\qquad Y:=X\Phi^\top,\qquad C:=X^\top X,\qquad c_p:=\operatorname{tr}(C^p), \tag*{(65)} Z := X W V ⊤ , R := f ( X ) − X Φ ⊤ , Y := X Φ ⊤ , C := X ⊤ X , c p := tr ( C p ) , (65)
with c 0 = tr ( C 0 ) = d 0 c_0=\operatorname{tr}(C^0)=d_0 c 0 = tr ( C 0 ) = d 0 , and define the three data moments
A 1 : = E [ c 1 2 c 4 + c 2 c 4 + c 3 2 ] , A 2 : = E [ c 1 2 c 2 + 2 c 2 c 4 ] , A_1:=\mathbb{E}[c_1^2c_4+c_2c_4+c_3^2],\qquad A_2:=\mathbb{E}[c_1^2c_2+2c_2c_4], A 1 := E [ c 1 2 c 4 + c 2 c 4 + c 3 2 ] , A 2 := E [ c 1 2 c 2 + 2 c 2 c 4 ] ,
A 3 : = E [ c 1 2 c 4 + c 1 c 2 2 + 3 c 2 c 4 + c 2 3 ] = A 1 + A 2 , (66) A_3:=\mathbb{E}[c_1^2c_4+c_1c_2^2+3c_2c_4+c_2^3]=A_1+A_2, \tag*{(66)} A 3 := E [ c 1 2 c 4 + c 1 c 2 2 + 3 c 2 c 4 + c 2 3 ] = A 1 + A 2 , (66)
which are explicit polynomials in ( T , d 0 ) (T,d_0) ( T , d 0 ) (Lemma 33 ). The gradients G ∙ : = ∂ L / ∂ W ∙ G_\bullet:=\partial\mathcal{L}/\partial W_\bullet G ∙ := ∂ L / ∂ W ∙ , ∙ ∈ { Q , K , V , O } \bullet\in\{Q,K,V,O\} ∙ ∈ { Q , K , V , O } , are
G Q = 1 d k W K X ⊤ Z W O ⊤ R ⊤ X , G K = 1 d k W Q X ⊤ R W O Z ⊤ X , G_Q=\frac{1}{\sqrt{d_k}}W_KX^\top ZW_O^\top R^\top X,\qquad G_K=\frac{1}{\sqrt{d_k}}W_QX^\top RW_OZ^\top X, G Q = d k 1 W K X ⊤ Z W O ⊤ R ⊤ X , G K = d k 1 W Q X ⊤ R W O Z ⊤ X ,
G V = W O ⊤ R ⊤ A X , G O = R ⊤ A Z . (67) G_V=W_O^\top R^\top AX,\qquad G_O=R^\top AZ. \tag*{(67)} G V = W O ⊤ R ⊤ A X , G O = R ⊤ A Z . (67)
Muon acts per weight matrix, so the objects of study are the four gradient self outer-products G Q G Q ⊤ G_QG_Q^\top G Q G Q ⊤ and G K G K ⊤ G_KG_K^\top G K G K ⊤ (both d k × d k d_k\times d_k d k × d k ), G V G V ⊤ G_VG_V^\top G V G V ⊤ (d v × d v d_v\times d_v d v × d v ) and G O G O ⊤ G_OG_O^\top G O G O ⊤ (d 2 × d 2 d_2\times d_2 d 2 × d 2 ), as in Appendices B to D.
Standing assumption
Assumption 8. W Q , W K , W V , W O W_Q,W_K,W_V,W_O W Q , W K , W V , W O are mutually independent with i.i.d. N ( 0 , 1 ) \mathcal{N}(0,1) N ( 0 , 1 ) entries; the rows of X X X are i.i.d. N ( 0 , I d 0 ) \mathcal{N}(0,I_{d_0}) N ( 0 , I d 0 ) , independent of the weights, so C ∼ W d 0 ( T , I d 0 ) C\sim\mathcal{W}_{d_0}(T,I_{d_0}) C ∼ W d 0 ( T , I d 0 ) ; Φ \Phi Φ is deterministic. Expectations E [ ⋅ ] \mathbb{E}[\cdot] E [ ⋅ ] are over data and weights jointly, and E [ ⋅ ∣ X ] \mathbb{E}[\cdot\mid X] E [ ⋅ ∣ X ] over the weights only.
This is Assumption 4 of the main text; fan-in scaling is treated in Remark 6.
Main results restated
Every d k d_k d k dependence below enters through the single ratio A 2 / A 1 A_2/A_1 A 2 / A 1 of the moments (66) .
Theorem (Theorem 4, restated). Under Assumption 8, E [ G Q G Q ⊤ ] = c Q I d k \mathbb{E}[G_QG_Q^\top]=c_QI_{d_k} E [ G Q G Q ⊤ ] = c Q I d k , E [ G K G K ⊤ ] = c K I d k \mathbb{E}[G_KG_K^\top]=c_KI_{d_k} E [ G K G K ⊤ ] = c K I d k and E [ G V G V ⊤ ] = c V I d v \mathbb{E}[G_VG_V^\top]=c_VI_{d_v} E [ G V G V ⊤ ] = c V I d v with
c Q = 1 d k ( E [ c 2 β 1 ] + d v γ ∥ Φ ∥ F 2 ) + 1 d k 2 ( E [ c 2 β 1 ] + E [ c 2 β 2 ] ) , c_Q=\frac{1}{d_k}\left(\mathbb{E}[c_2\beta_1]+d_v\gamma\|\Phi\|_F^2\right)+\frac{1}{d_k^2}\left(\mathbb{E}[c_2\beta_1]+\mathbb{E}[c_2\beta_2]\right), c Q = d k 1 ( E [ c 2 β 1 ] + d v γ ∥Φ ∥ F 2 ) + d k 2 1 ( E [ c 2 β 1 ] + E [ c 2 β 2 ] ) , c K = 1 d k ( E [ c 2 β 1 ] + d v γ ∥ Φ ∥ F 2 ) + 1 d k 2 E [ ( c 1 2 + c 2 ) β 1 ] , c_K=\frac{1}{d_k}\left(\mathbb{E}[c_2\beta_1]+d_v\gamma\|\Phi\|_F^2\right)+\frac{1}{d_k^2}\mathbb{E}[(c_1^2+c_2)\beta_1], c K = d k 1 ( E [ c 2 β 1 ] + d v γ ∥Φ ∥ F 2 ) + d k 2 1 E [( c 1 2 + c 2 ) β 1 ] , c V = d 2 ( d 2 + d v + 1 ) ( A 1 + 1 d k A 3 ) + γ ∥ Φ ∥ F 2 , (68) c_V=d_2(d_2+d_v+1)\left(A_1+\frac{1}{d_k}A_3\right)+\gamma\|\Phi\|_F^2, \tag*{(68)} c V = d 2 ( d 2 + d v + 1 ) ( A 1 + d k 1 A 3 ) + γ ∥Φ ∥ F 2 , (68) where β 1 , β 2 \beta_1,\beta_2 β 1 , β 2 are the polynomials (75) ; and
E [ G O G O ⊤ ] = α ˉ O I d 2 + d v γ Φ Φ ⊤ , (69) \mathbb{E}[G_OG_O^\top]=\bar{\alpha}_O I_{d_2}+d_v\gamma\Phi\Phi^\top, \tag*{(69)} E [ G O G O ⊤ ] = α ˉ O I d 2 + d v γ Φ Φ ⊤ , (69) with
γ = 1 d 0 E [ c 2 2 ] = T [ ( T d 0 + 8 ) ( T + d 0 ) 2 + 2 ( T d 0 + 10 ) ( T + d 0 ) + 5 ( T d 0 + 4 ) ] , (70) \gamma=\frac{1}{d_0}\mathbb{E}[c_2^2]=T\left[(Td_0+8)(T+d_0)^2+2(Td_0+10)(T+d_0)+5(Td_0+4)\right], \tag*{(70)} γ = d 0 1 E [ c 2 2 ] = T [ ( T d 0 + 8 ) ( T + d 0 ) 2 + 2 ( T d 0 + 10 ) ( T + d 0 ) + 5 ( T d 0 + 4 ) ] , (70) α ˉ O = d v ( d v + 1 ) A 1 + d v A 2 + d v ( d v + 1 ) d k A 3 + 2 d v d k A 1 = Θ ( d v 2 d 0 6 ) . (71) \bar{\alpha}_O=d_v(d_v+1)A_1+d_vA_2+\frac{d_v(d_v+1)}{d_k}A_3+\frac{2d_v}{d_k}A_1=\Theta(d_v^2d_0^6). \tag*{(71)} α ˉ O = d v ( d v + 1 ) A 1 + d v A 2 + d k d v ( d v + 1 ) A 3 + d k 2 d v A 1 = Θ ( d v 2 d 0 6 ) . (71)
Corollary 3. Under Assumption 8 and for fixed Φ \Phi Φ ,
c Q ≍ c K = Θ ( d 2 2 d v 2 d 0 6 / d k ) , c V = Θ ( d 2 ( d 2 + d v ) d 0 6 ) , α ˉ O = Θ ( d v 2 d 0 6 ) , c_Q\asymp c_K=\Theta(d_2^2d_v^2d_0^6/d_k),\qquad c_V=\Theta(d_2(d_2+d_v)d_0^6),\qquad\bar{\alpha}_O=\Theta(d_v^2d_0^6), c Q ≍ c K = Θ ( d 2 2 d v 2 d 0 6 / d k ) , c V = Θ ( d 2 ( d 2 + d v ) d 0 6 ) , α ˉ O = Θ ( d v 2 d 0 6 ) , with c K − c Q = ( E [ c 1 2 β 1 ] − E [ c 2 β 2 ] ) / d k 2 c_K-c_Q=(\mathbb{E}[c_1^2\beta_1]-\mathbb{E}[c_2\beta_2])/d_k^2 c K − c Q = ( E [ c 1 2 β 1 ] − E [ c 2 β 2 ]) / d k 2 of relative size O ( 1 / d k ) O(1/d_k) O ( 1/ d k ) ; and for a ≠ b a\ne b a = b with ( Φ Φ ⊤ ) a b ≠ 0 (\Phi\Phi^\top)_{ab}\ne0 ( Φ Φ ⊤ ) ab = 0 ,
E [ ( G O G O ⊤ ) a a ] E [ ( G O G O ⊤ ) a b ] = Θ ( d v d 0 3 ) . \frac{\mathbb{E}[(G_OG_O^\top)_{aa}]}{\mathbb{E}[(G_OG_O^\top)_{ab}]}=\Theta(d_vd_0^3). E [( G O G O ⊤ ) ab ] E [( G O G O ⊤ ) aa ] = Θ ( d v d 0 3 ) .
Thus the hidden dimension d v d_v d v and the token dimension d 0 d_0 d 0 drive the diagonal dominance of E [ G O G O ⊤ ] \mathbb{E}[G_OG_O^\top] E [ G O G O ⊤ ] , while d k d_k d k enters c V c_V c V , α ˉ O \bar{\alpha}_O α ˉ O and the W O W_O W O ratio only through bounded factors.
Preliminary lemmas
Lemmas 29 to 32 are stated conditionally on X X X , i.e. with C C C fixed and c p = tr ( C p ) c_p=\operatorname{tr}(C^p) c p = tr ( C p ) ; Lemma 33 supplies the moments of C ∼ W d 0 ( T , I d 0 ) C\sim W_{d_0}(T,I_{d_0}) C ∼ W d 0 ( T , I d 0 ) needed to average over X X X .
Lemma 27. Same as Lemma 1.
Lemma 28 (Wick/Isserlis). Same as Lemma 3.
Lemma 29. Let W ∈ R n × d W\in\mathbb{R}^{n\times d} W ∈ R n × d have i.i.d. N ( 0 , 1 ) \mathcal{N}(0,1) N ( 0 , 1 ) entries, let S : = W ⊤ W ∼ W d ( n , I d ) S:=W^\top W\sim W_d(n,I_d) S := W ⊤ W ∼ W d ( n , I d ) , and let M ∈ R d × d M\in\mathbb{R}^{d\times d} M ∈ R d × d be deterministic symmetric. Then
E [ S a b S c d ] = n 2 δ a b δ c d + n ( δ a c δ b d + δ a d δ b c ) , E [ S M S ] = n ( n + 1 ) M + n tr ( M ) I d , (72) \mathbb{E}[S_{ab}S_{cd}]=n^2\delta_{ab}\delta_{cd}+n(\delta_{ac}\delta_{bd}+\delta_{ad}\delta_{bc}),\qquad\mathbb{E}[SMS]=n(n+1)M+n\operatorname{tr}(M)I_d, \tag*{(72)} E [ S ab S c d ] = n 2 δ ab δ c d + n ( δ a c δ b d + δ a d δ b c ) , E [ S M S ] = n ( n + 1 ) M + n tr ( M ) I d , (72) hence E [ tr ( M S M S ) ] = n ( n + 1 ) tr ( M 2 ) + n ( tr M ) 2 \mathbb{E}[\operatorname{tr}(MSMS)]=n(n+1)\operatorname{tr}(M^2)+n(\operatorname{tr}M)^2 E [ tr ( M S M S )] = n ( n + 1 ) tr ( M 2 ) + n ( tr M ) 2 and E [ S 2 ] = n ( n + d + 1 ) I d \mathbb{E}[S^2]=n(n+d+1)I_d E [ S 2 ] = n ( n + d + 1 ) I d .
Proof. Since S a b = ∑ α W α a W α b S_{ab}=\sum_\alpha W_{\alpha a}W_{\alpha b} S ab = ∑ α W α a W α b , Lemma 28 gives
E [ S a b S c d ] = ∑ α , β E [ W α a W α b W β c W β d ] = ∑ α , β ( δ a b δ c d + δ α β δ a c δ b d + δ α β δ a d δ b c ) , \mathbb{E}[S_{ab}S_{cd}]=\sum_{\alpha,\beta}\mathbb{E}[W_{\alpha a}W_{\alpha b}W_{\beta c}W_{\beta d}]=\sum_{\alpha,\beta}(\delta_{ab}\delta_{cd}+\delta_{\alpha\beta}\delta_{ac}\delta_{bd}+\delta_{\alpha\beta}\delta_{ad}\delta_{bc}), E [ S ab S c d ] = α , β ∑ E [ W α a W α b W β c W β d ] = α , β ∑ ( δ ab δ c d + δ α β δ a c δ b d + δ α β δ a d δ b c ) ,
which is the first identity. Contracting it against M b c M_{bc} M b c and using M = M ⊤ M=M^\top M = M ⊤ ,
E [ ( S M S ) a d ] = ∑ b , c M b c E [ S a b S c d ] = n 2 M a d + n M a d + n δ a d tr ( M ) , \mathbb{E}[(SMS)_{ad}]=\sum_{b,c}M_{bc}\mathbb{E}[S_{ab}S_{cd}]=n^2M_{ad}+nM_{ad}+n\delta_{ad}\operatorname{tr}(M), E [( S M S ) a d ] = b , c ∑ M b c E [ S ab S c d ] = n 2 M a d + n M a d + n δ a d tr ( M ) ,
which is the second identity; contracting once more against M d a M_{da} M d a gives the trace identity, and M = I d M=I_d M = I d gives E [ S 2 ] \mathbb{E}[S^2] E [ S 2 ] . ▪
Lemma 30. Under Assumption 8 , conditionally on X X X , let N : = W Q C W Q T ∈ R d k × d k N := W_QCW_Q^{\mathsf{T}} \in\mathbb{R}^{d_k \times d_k} N := W Q C W Q T ∈ R d k × d k . Then
E [ N i j N k l ∣ X ] = c 1 2 δ i j δ k l + c 2 ( δ i k δ j l + δ i l δ j k ) , \mathbb{E}[N_{ij}N_{kl} \mid X] = c_1^2 \delta_{ij}\delta_{kl} + c_2(\delta_{ik}\delta_{jl} + \delta_{il}\delta_{jk}), E [ N ij N k l ∣ X ] = c 1 2 δ ij δ k l + c 2 ( δ ik δ j l + δ i l δ j k ) , E [ ( tr N ) 2 ∣ X ] = d k 2 c 1 2 + 2 d k c 2 , E [ tr N 2 ∣ X ] = d k 2 c 2 + d k ( c 1 2 + c 2 ) . (73) \mathbb{E}[(\operatorname{tr} N)^2 \mid X] = d_k^2 c_1^2 + 2d_k c_2,\qquad\mathbb{E}[\operatorname{tr} N^2 \mid X] = d_k^2 c_2 + d_k(c_1^2+c_2). \tag*{(73)} E [( tr N ) 2 ∣ X ] = d k 2 c 1 2 + 2 d k c 2 , E [ tr N 2 ∣ X ] = d k 2 c 2 + d k ( c 1 2 + c 2 ) . (73) The same identities hold with C C C replaced by any deterministic symmetric E ∈ R d 0 × d 0 E \in\mathbb{R}^{d_0 \times d_0} E ∈ R d 0 × d 0 and ( c 1 , c 2 ) (c_1,c_2) ( c 1 , c 2 ) by ( tr E , tr E 2 ) (\operatorname{tr} E,\operatorname{tr} E^2) ( tr E , tr E 2 ) , and with W Q W_Q W Q replaced by W K W_K W K .
Proof. Since N i j = ∑ a , b ( W Q ) i a C a b ( W Q ) j b N_{ij} = \sum_{a,b}(W_Q)_{ia}C_{ab}(W_Q)_{jb} N ij = ∑ a , b ( W Q ) ia C ab ( W Q ) j b , Lemma 28 gives
E [ ( W Q ) i a ( W Q ) j b ( W Q ) k c ( W Q ) l d ] = δ i j δ a b δ k l δ c d + δ i k δ a c δ j l δ b d + δ i l δ a d δ j k δ b c , \mathbb{E}[(W_Q)_{ia}(W_Q)_{jb}(W_Q)_{kc}(W_Q)_{ld}] = \delta_{ij}\delta_{ab}\delta_{kl}\delta_{cd} + \delta_{ik}\delta_{ac}\delta_{jl}\delta_{bd} + \delta_{il}\delta_{ad}\delta_{jk}\delta_{bc}, E [( W Q ) ia ( W Q ) j b ( W Q ) k c ( W Q ) l d ] = δ ij δ ab δ k l δ c d + δ ik δ a c δ j l δ b d + δ i l δ a d δ j k δ b c , and contracting against C a b C c d C_{ab}C_{cd} C ab C c d , with ∑ a , b C a b C a b = tr ( C 2 ) = c 2 \sum_{a,b} C_{ab}C_{ab} = \operatorname{tr}(C^2) = c_2 ∑ a , b C ab C ab = tr ( C 2 ) = c 2 , gives the first line of (73) . Summing it over i = j i=j i = j and k = l k=l k = l gives E [ ( tr N ) 2 ∣ X ] \mathbb{E}[(\operatorname{tr} N)^2 \mid X] E [( tr N ) 2 ∣ X ] ; summing it over j = k j=k j = k and l = i l=i l = i gives E [ tr N 2 ∣ X ] \mathbb{E}[\operatorname{tr} N^2 \mid X] E [ tr N 2 ∣ X ] . Only the symmetry of C C C and the law of W Q W_Q W Q were used, which gives the last assertion. (These are Lemmas 10 and 11 with m = d k m=d_k m = d k and M = C M=C M = C .)
Lemma 31. Under Assumption 8 , conditionally on X X X , let N ∈ R d k × d k N \in\mathbb{R}^{d_k \times d_k} N ∈ R d k × d k be symmetric and independent of W K W_K W K , and let S : = W K T N W K S := W_K^{\mathsf{T}}NW_K S := W K T N W K . Then E [ S a b S c d ∣ X , N ] = ( tr N ) 2 δ a b δ c d + tr ( N 2 ) ( δ a c δ b d + δ a d δ b c ) \mathbb{E}[S_{ab}S_{cd} \mid X,N] = (\operatorname{tr} N)^2\delta_{ab}\delta_{cd} + \operatorname{tr}(N^2)(\delta_{ac}\delta_{bd} + \delta_{ad}\delta_{bc}) E [ S ab S c d ∣ X , N ] = ( tr N ) 2 δ ab δ c d + tr ( N 2 ) ( δ a c δ b d + δ a d δ b c ) , and
E [ tr ( S C 2 S C 2 ) ∣ X , N ] = ( tr N ) 2 c 4 + tr ( N 2 ) ( c 4 + c 2 2 ) , \mathbb{E}[\operatorname{tr}(SC^2SC^2) \mid X,N] = (\operatorname{tr} N)^2c_4 + \operatorname{tr}(N^2)(c_4+c_2^2), E [ tr ( S C 2 S C 2 ) ∣ X , N ] = ( tr N ) 2 c 4 + tr ( N 2 ) ( c 4 + c 2 2 ) , E [ ( tr ( S C 2 ) ) 2 ∣ X , N ] = ( tr N ) 2 c 2 2 + 2 tr ( N 2 ) c 4 . (74) \mathbb{E}[(\operatorname{tr}(SC^2))^2 \mid X,N] = (\operatorname{tr} N)^2c_2^2 + 2\operatorname{tr}(N^2)c_4. \tag*{(74)} E [( tr ( S C 2 ) ) 2 ∣ X , N ] = ( tr N ) 2 c 2 2 + 2 tr ( N 2 ) c 4 . (74) Proof. Since S a b = ∑ i , j ( W K ) i a N i j ( W K ) j b S_{ab} = \sum_{i,j}(W_K)_{ia}N_{ij}(W_K)_{jb} S ab = ∑ i , j ( W K ) ia N ij ( W K ) j b , Lemma 28 applied to E [ ( W K ) i a ( W K ) j b ( W K ) k c ( W K ) l d ] \mathbb{E}[(W_K)_{ia}(W_K)_{jb}(W_K)_{kc}(W_K)_{ld}] E [( W K ) ia ( W K ) j b ( W K ) k c ( W K ) l d ] as in the proof of Lemma 30 , followed by contraction against N i j N k l N_{ij}N_{kl} N ij N k l with N = N T N=N^{\mathsf{T}} N = N T , gives the first identity. Contracting the first identity against ( C 2 ) b c ( C 2 ) d a (C^2)_{bc}(C^2)_{da} ( C 2 ) b c ( C 2 ) d a , the three terms give ( tr N ) 2 tr ( C 4 ) (\operatorname{tr} N)^2\operatorname{tr}(C^4) ( tr N ) 2 tr ( C 4 ) , tr ( N 2 ) tr ( C 4 ) \operatorname{tr}(N^2)\operatorname{tr}(C^4) tr ( N 2 ) tr ( C 4 ) and tr ( N 2 ) tr ( C 2 ) 2 \operatorname{tr}(N^2)\operatorname{tr}(C^2)^2 tr ( N 2 ) tr ( C 2 ) 2 , which is the first line of (74) ; contracting against ( C 2 ) b a ( C 2 ) d c (C^2)_{ba}(C^2)_{dc} ( C 2 ) ba ( C 2 ) d c instead, they give ( tr N ) 2 tr ( C 2 ) 2 (\operatorname{tr} N)^2\operatorname{tr}(C^2)^2 ( tr N ) 2 tr ( C 2 ) 2 , tr ( N 2 ) tr ( C 4 ) \operatorname{tr}(N^2)\operatorname{tr}(C^4) tr ( N 2 ) tr ( C 4 ) and tr ( N 2 ) tr ( C 4 ) \operatorname{tr}(N^2)\operatorname{tr}(C^4) tr ( N 2 ) tr ( C 4 ) , which is the second line.
Lemma 32. Under Assumption 8 , conditionally on X X X , let H O : = W O T W O ∼ W d v ( d 2 , I d v ) H_O := W_O^{\mathsf{T}}W_O \sim W_{d_v}(d_2,I_{d_v}) H O := W O T W O ∼ W d v ( d 2 , I d v ) and E : = C W V T H O W V C E := C W_V^{\mathsf{T}}H_OW_VC E := C W V T H O W V C , and define
β 1 : = d 2 d v [ ( d 2 d v + d 2 + d v + 3 ) c 4 + ( d 2 + d v + 1 ) c 2 2 ] , \beta_1 := d_2d_v\left[(d_2d_v+d_2+d_v+3)c_4 + (d_2+d_v+1)c_2^2\right], β 1 := d 2 d v [ ( d 2 d v + d 2 + d v + 3 ) c 4 + ( d 2 + d v + 1 ) c 2 2 ] , β 2 : = d 2 d v [ ( d 2 d v + 2 ) c 2 2 + 2 ( d 2 + d v + 1 ) c 4 ] . (75) \beta_2 := d_2d_v\left[(d_2d_v+2)c_2^2 + 2(d_2+d_v+1)c_4\right]. \tag*{(75)} β 2 := d 2 d v [ ( d 2 d v + 2 ) c 2 2 + 2 ( d 2 + d v + 1 ) c 4 ] . (75) Then E [ tr ( E 2 ) ∣ X ] = β 1 \mathbb{E}[\operatorname{tr}(E^2) \mid X] = \beta_1 E [ tr ( E 2 ) ∣ X ] = β 1 and E [ ( tr E ) 2 ∣ X ] = β 2 \mathbb{E}[(\operatorname{tr} E)^2 \mid X] = \beta_2 E [( tr E ) 2 ∣ X ] = β 2 .
Proof. Let S V : = W V T H O W V S_V := W_V^{\mathsf{T}}H_OW_V S V := W V T H O W V , so that tr E = tr ( S V C 2 ) \operatorname{tr} E = \operatorname{tr}(S_VC^2) tr E = tr ( S V C 2 ) and tr E 2 = tr ( S V C 2 S V C 2 ) \operatorname{tr} E^2 = \operatorname{tr}(S_VC^2S_VC^2) tr E 2 = tr ( S V C 2 S V C 2 ) . Since H O H_O H O is symmetric and independent of W V W_V W V , Lemma 31 applies with ( W V , H O ) (W_V,H_O) ( W V , H O ) in place of ( W K , N ) (W_K,N) ( W K , N ) :
E [ tr E 2 ∣ X , H O ] = ( tr H O ) 2 c 4 + tr ( H O 2 ) ( c 4 + c 2 2 ) , \mathbb{E}[\operatorname{tr} E^2 \mid X,H_O] = (\operatorname{tr} H_O)^2c_4 + \operatorname{tr}(H_O^2)(c_4+c_2^2), E [ tr E 2 ∣ X , H O ] = ( tr H O ) 2 c 4 + tr ( H O 2 ) ( c 4 + c 2 2 ) , E [ ( tr E ) 2 ∣ X , H O ] = ( tr H O ) 2 c 2 2 + 2 tr ( H O 2 ) c 4 . \mathbb{E}[(\operatorname{tr} E)^2 \mid X,H_O] = (\operatorname{tr} H_O)^2c_2^2 + 2\operatorname{tr}(H_O^2)c_4. E [( tr E ) 2 ∣ X , H O ] = ( tr H O ) 2 c 2 2 + 2 tr ( H O 2 ) c 4 . By Lemma 29 with ( n , d ) = ( d 2 , d v ) (n,d)=(d_2,d_v) ( n , d ) = ( d 2 , d v ) , E [ ( tr H O ) 2 ] = d 2 d v ( d 2 d v + 2 ) \mathbb{E}[(\operatorname{tr} H_O)^2] = d_2d_v(d_2d_v+2) E [( tr H O ) 2 ] = d 2 d v ( d 2 d v + 2 ) and E [ tr H O 2 ] = d 2 d v ( d 2 + d v + 1 ) \mathbb{E}[\operatorname{tr} H_O^2] = d_2d_v(d_2+d_v+1) E [ tr H O 2 ] = d 2 d v ( d 2 + d v + 1 ) ; substituting gives β 1 \beta_1 β 1 and β 2 \beta_2 β 2 .
Lemma 33 (Wishart trace moments). For C ∼ W d 0 ( T , I d 0 ) C \sim W_{d_0}(T,I_{d_0}) C ∼ W d 0 ( T , I d 0 ) : E [ c 1 ] = T d 0 \mathbb{E}[c_1]=Td_0 E [ c 1 ] = T d 0 , E [ c 2 ] = T d 0 ( T + d 0 + 1 ) \mathbb{E}[c_2]=Td_0(T+d_0+1) E [ c 2 ] = T d 0 ( T + d 0 + 1 ) , and
E [ c 4 ] = T d 0 [ ( T + d 0 ) 3 + 6 ( T + d 0 ) 2 + 3 ( T d 0 + 7 ) ( T + d 0 ) + 5 ( T d 0 + 4 ) ] , (76) \mathbb{E}[c_4]=Td_0\left[(T+d_0)^3+6(T+d_0)^2+3(Td_0+7)(T+d_0)+5(Td_0+4)\right], \tag*{(76)} E [ c 4 ] = T d 0 [ ( T + d 0 ) 3 + 6 ( T + d 0 ) 2 + 3 ( T d 0 + 7 ) ( T + d 0 ) + 5 ( T d 0 + 4 ) ] , (76) E [ c 2 2 ] = T d 0 [ ( T d 0 + 8 ) ( T + d 0 ) 2 + 2 ( T d 0 + 10 ) ( T + d 0 ) + 5 ( T d 0 + 4 ) ] , (77) \mathbb{E}[c_2^2]=Td_0\left[(Td_0+8)(T+d_0)^2+2(Td_0+10)(T+d_0)+5(Td_0+4)\right], \tag*{(77)} E [ c 2 2 ] = T d 0 [ ( T d 0 + 8 ) ( T + d 0 ) 2 + 2 ( T d 0 + 10 ) ( T + d 0 ) + 5 ( T d 0 + 4 ) ] , (77) E [ c 2 c 4 ] = T d 0 [ ( T d 0 + 16 ) ( T + d 0 ) 4 + ( 7 T d 0 + 136 ) ( T + d 0 ) 3 + ( 3 T 2 d 0 2 + 99 T d 0 + 736 ) ( T + d 0 ) 2 + ( 8 T 2 d 0 2 + 305 T d 0 + 1608 ) ( T + d 0 ) + 21 T 2 d 0 2 + 420 T d 0 + 1344 ] , (78) \begin{aligned}
\mathbb{E}[c_2c_4]={}&Td_0\bigl[(Td_0+16)(T+d_0)^4+(7Td_0+136)(T+d_0)^3\\
&+(3T^2d_0^2+99Td_0+736)(T+d_0)^2+(8T^2d_0^2+305Td_0+1608)(T+d_0)\\
&+21T^2d_0^2+420Td_0+1344\bigr], \tag*{(78)}
\end{aligned} E [ c 2 c 4 ] = T d 0 [ ( T d 0 + 16 ) ( T + d 0 ) 4 + ( 7 T d 0 + 136 ) ( T + d 0 ) 3 + ( 3 T 2 d 0 2 + 99 T d 0 + 736 ) ( T + d 0 ) 2 + ( 8 T 2 d 0 2 + 305 T d 0 + 1608 ) ( T + d 0 ) + 21 T 2 d 0 2 + 420 T d 0 + 1344 ] , (78) E [ c 2 3 ] = T d 0 [ ( T 2 d 0 2 + 24 T d 0 + 160 ) ( T + d 0 ) 3 + ( 3 T 2 d 0 2 + 84 T d 0 + 704 ) ( T + d 0 ) 2 + ( 15 T 2 d 0 2 + 344 T d 0 + 1664 ) ( T + d 0 ) + 13 T 2 d 0 2 + 380 T d 0 + 1312 ] , (79) \begin{aligned}
\mathbb{E}[c_2^3]={}&Td_0\bigl[(T^2d_0^2+24Td_0+160)(T+d_0)^3+(3T^2d_0^2+84Td_0+704)(T+d_0)^2\\
&+(15T^2d_0^2+344Td_0+1664)(T+d_0)+13T^2d_0^2+380Td_0+1312\bigr], \tag*{(79)}
\end{aligned} E [ c 2 3 ] = T d 0 [ ( T 2 d 0 2 + 24 T d 0 + 160 ) ( T + d 0 ) 3 + ( 3 T 2 d 0 2 + 84 T d 0 + 704 ) ( T + d 0 ) 2 + ( 15 T 2 d 0 2 + 344 T d 0 + 1664 ) ( T + d 0 ) + 13 T 2 d 0 2 + 380 T d 0 + 1312 ] , (79) E [ c 1 2 c 4 ] = ( T d 0 + 8 ) ( T d 0 + 10 ) E [ c 4 ] , E [ c 1 2 c 2 2 ] = ( T d 0 + 8 ) ( T d 0 + 10 ) E [ c 2 ] . (80) \mathbb{E}[c_1^2c_4]=(Td_0+8)(Td_0+10)\mathbb{E}[c_4],\qquad
\mathbb{E}[c_1^2c_2^2]=(Td_0+8)(Td_0+10)\mathbb{E}[c_2]. \tag*{(80)} E [ c 1 2 c 4 ] = ( T d 0 + 8 ) ( T d 0 + 10 ) E [ c 4 ] , E [ c 1 2 c 2 2 ] = ( T d 0 + 8 ) ( T d 0 + 10 ) E [ c 2 ] . (80)
Consequently A 3 = A 1 + A 2 A_3=A_1+A_2 A 3 = A 1 + A 2 . As polynomials in d 0 d_0 d 0 , all six moments have positive coefficients, of degree 4 for E [ c 4 ] , E [ c 2 2 ] \mathbb{E}[c_4],\mathbb{E}[c_2^2] E [ c 4 ] , E [ c 2 2 ] and degree 6 for the other four; hence E [ c 4 ] , E [ c 2 2 ] = Θ ( d 0 4 ) \mathbb{E}[c_4],\mathbb{E}[c_2^2]=\Theta(d_0^4) E [ c 4 ] , E [ c 2 2 ] = Θ ( d 0 4 ) , E [ c 2 c 4 ] , E [ c 2 3 ] , E [ c 1 2 c 4 ] , E [ c 1 2 c 2 2 ] = Θ ( d 0 6 ) \mathbb{E}[c_2c_4],\mathbb{E}[c_2^3],\mathbb{E}[c_1^2c_4],\mathbb{E}[c_1^2c_2^2]=\Theta(d_0^6) E [ c 2 c 4 ] , E [ c 2 3 ] , E [ c 1 2 c 4 ] , E [ c 1 2 c 2 2 ] = Θ ( d 0 6 ) , and therefore γ = 1 d 0 E [ c 2 2 ] = Θ ( d 0 3 ) \gamma=\frac{1}{d_0}\mathbb{E}[c_2^2]=\Theta(d_0^3) γ = d 0 1 E [ c 2 2 ] = Θ ( d 0 3 ) and A 1 , A 2 = Θ ( d 0 6 ) A_1,A_2=\Theta(d_0^6) A 1 , A 2 = Θ ( d 0 6 ) .
Proof. Throughout the proof let s : = T + d 0 s:=T+d_0 s := T + d 0 and q : = T d 0 q:=Td_0 q := T d 0 . Since C = X T X C=X^\mathsf{T}X C = X T X with X ∈ R T × d 0 X\in\mathbb{R}^{T\times d_0} X ∈ R T × d 0 standard Gaussian, every moment below is the expectation of a polynomial in the entries of X X X , and Gaussian integration by parts, E [ X t a Ψ ( X ) ] = E [ ∂ Ψ / ∂ X t a ] \mathbb{E}[X_{ta}\Psi(X)]=\mathbb{E}[\partial\Psi/\partial X_{ta}] E [ X t a Ψ ( X )] = E [ ∂ Ψ/ ∂ X t a ] for polynomial Ψ \Psi Ψ , applies.
Step 1: a recursion in the total degree. For k ≥ 1 k\ge1 k ≥ 1 and g : = ∏ i = 1 n c m i g:=\prod_{i=1}^{n}c_{m_i} g := ∏ i = 1 n c m i with m i ≥ 1 m_i\ge1 m i ≥ 1 (g = 1 g=1 g = 1 if n = 0 n=0 n = 0 ),
E [ c k g ] = E [ ( ( T + k − 1 ) c k − 1 + ∑ j = 1 k − 1 c j c k − 1 − j ) g ] + 2 ∑ i = 1 n m i E [ c k + m i − 1 ∏ l ≠ i c m l ] . (81) \mathbb{E}[c_kg]=\mathbb{E}\left[\left((T+k-1)c_{k-1}+\sum_{j=1}^{k-1}c_jc_{k-1-j}\right)g\right]+2\sum_{i=1}^{n}m_i\mathbb{E}\left[c_{k+m_i-1}\prod_{l\ne i}c_{m_l}\right]. \tag*{(81)} E [ c k g ] = E [ ( ( T + k − 1 ) c k − 1 + j = 1 ∑ k − 1 c j c k − 1 − j ) g ] + 2 i = 1 ∑ n m i E c k + m i − 1 l = i ∏ c m l . (81)
Indeed, c k = tr ( X T X C k − 1 ) = ∑ t , a X t a ( X C k − 1 ) t a c_k=\operatorname{tr}(X^\mathsf{T}XC^{k-1})=\sum_{t,a}X_{ta}(XC^{k-1})_{ta} c k = tr ( X T X C k − 1 ) = ∑ t , a X t a ( X C k − 1 ) t a , so integration by parts in each X t a X_{ta} X t a gives
E [ c k g ] = ∑ t , a E [ ∂ ( X C k − 1 ) t a ∂ X t a g ] + ∑ t , a E [ ( X C k − 1 ) t a ∂ g ∂ X t a ] . (82) \mathbb{E}[c_kg]=\sum_{t,a}\mathbb{E}\left[\frac{\partial(XC^{k-1})_{ta}}{\partial X_{ta}}g\right]+\sum_{t,a}\mathbb{E}\left[(XC^{k-1})_{ta}\frac{\partial g}{\partial X_{ta}}\right]. \tag*{(82)} E [ c k g ] = t , a ∑ E [ ∂ X t a ∂ ( X C k − 1 ) t a g ] + t , a ∑ E [ ( X C k − 1 ) t a ∂ X t a ∂ g ] . (82)
From ∂ C b c / ∂ X t a = δ a b X t c + δ a c X t b \partial C_{bc}/\partial X_{ta}=\delta_{ab}X_{tc}+\delta_{ac}X_{tb} ∂ C b c / ∂ X t a = δ ab X t c + δ a c X t b and the product rule, for m ≥ 1 m\ge1 m ≥ 1 ,
∂ ( C m ) b c ∂ X t a = ∑ r = 0 m − 1 [ ( C r ) b a ( X C m − 1 − r ) t c + ( C r X T ) b t ( C m − 1 − r ) a c ] , \frac{\partial(C^m)_{bc}}{\partial X_{ta}}=\sum_{r=0}^{m-1}\left[(C^r)_{ba}(XC^{m-1-r})_{tc}+(C^rX^\mathsf{T})_{bt}(C^{m-1-r})_{ac}\right], ∂ X t a ∂ ( C m ) b c = r = 0 ∑ m − 1 [ ( C r ) ba ( X C m − 1 − r ) t c + ( C r X T ) b t ( C m − 1 − r ) a c ] ,
∂ c m ∂ X t a = ∑ b ∂ ( C m ) b b ∂ X t a = 2 m ( X C m − 1 ) t a . (83) \frac{\partial c_m}{\partial X_{ta}}=\sum_b\frac{\partial(C^m)_{bb}}{\partial X_{ta}}=2m(XC^{m-1})_{ta}. \tag*{(83)} ∂ X t a ∂ c m = b ∑ ∂ X t a ∂ ( C m ) bb = 2 m ( X C m − 1 ) t a . (83)
Since ( X C k − 1 ) t a = ∑ b X t b ( C k − 1 ) b a (XC^{k-1})_{ta}=\sum_bX_{tb}(C^{k-1})_{ba} ( X C k − 1 ) t a = ∑ b X t b ( C k − 1 ) ba , Eq. (83) with m = k − 1 m=k-1 m = k − 1 gives
∑ t , a ∂ ( X C k − 1 ) t a ∂ X t a = ∑ t , a ( C k − 1 ) a a + ∑ r = 0 k − 2 [ ∑ t , a ( X C r ) t a ( X C k − 2 − r ) t a ] + ∑ r = 0 k − 2 tr ( X C r X T ) tr ( C k − 2 − r ) = T c k − 1 + ( k − 1 ) c k − 1 + ∑ r = 0 k − 2 c r + 1 c k − 2 − r = ( T + k − 1 ) c k − 1 + ∑ j = 1 k − 1 c j c k − 1 − j . \begin{aligned}
\sum_{t,a}\frac{\partial(XC^{k-1})_{ta}}{\partial X_{ta}}
={}&\sum_{t,a}(C^{k-1})_{aa}+\sum_{r=0}^{k-2}\left[\sum_{t,a}(XC^r)_{ta}(XC^{k-2-r})_{ta}\right]\\
&+\sum_{r=0}^{k-2}\operatorname{tr}(XC^rX^\mathsf{T})\operatorname{tr}(C^{k-2-r})\\
={}&Tc_{k-1}+(k-1)c_{k-1}+\sum_{r=0}^{k-2}c_{r+1}c_{k-2-r}
=(T+k-1)c_{k-1}+\sum_{j=1}^{k-1}c_jc_{k-1-j}.
\end{aligned} t , a ∑ ∂ X t a ∂ ( X C k − 1 ) t a = = t , a ∑ ( C k − 1 ) aa + r = 0 ∑ k − 2 [ t , a ∑ ( X C r ) t a ( X C k − 2 − r ) t a ] + r = 0 ∑ k − 2 tr ( X C r X T ) tr ( C k − 2 − r ) T c k − 1 + ( k − 1 ) c k − 1 + r = 0 ∑ k − 2 c r + 1 c k − 2 − r = ( T + k − 1 ) c k − 1 + j = 1 ∑ k − 1 c j c k − 1 − j .
since ∑ t , a ( X C r ) t a ( X C k − 2 − r ) t a = tr ( C r X ⊤ X C k − 2 − r ) = c k − 1 \sum_{t,a}(XC^{r})_{ta}(XC^{k-2-r})_{ta}=\operatorname{tr}(C^{r}X^{\top}XC^{k-2-r})=c_{k-1} ∑ t , a ( X C r ) t a ( X C k − 2 − r ) t a = tr ( C r X ⊤ X C k − 2 − r ) = c k − 1 and tr ( X C r X ⊤ ) = c r + 1 \operatorname{tr}(XC^{r}X^{\top})=c_{r+1} tr ( X C r X ⊤ ) = c r + 1 ; this is the first term of (81) . For the second, ∂ g / ∂ X t a = ∑ i ( ∂ c m i / ∂ X t a ) ∏ l ≠ i c m l \partial g/\partial X_{ta}=\sum_{i}(\partial c_{m_i}/\partial X_{ta})\prod_{l\ne i}c_{m_l} ∂ g / ∂ X t a = ∑ i ( ∂ c m i / ∂ X t a ) ∏ l = i c m l and, by (83) ,
∑ t , a ( X C k − 1 ) t a ∂ c m ∂ X t a = 2 m ∑ t , a ( X C k − 1 ) t a ( X C m − 1 ) t a = 2 m tr ( C k − 1 X ⊤ X C m − 1 ) = 2 m c k + m − 1 . \sum_{t,a}(XC^{k-1})_{ta}\frac{\partial c_m}{\partial X_{ta}}=2m\sum_{t,a}(XC^{k-1})_{ta}(XC^{m-1})_{ta}
=2m\operatorname{tr}(C^{k-1}X^{\top}XC^{m-1})=2m c_{k+m-1}. t , a ∑ ( X C k − 1 ) t a ∂ X t a ∂ c m = 2 m t , a ∑ ( X C k − 1 ) t a ( X C m − 1 ) t a = 2 m tr ( C k − 1 X ⊤ X C m − 1 ) = 2 m c k + m − 1 .
Step 2: moments containing c 1 c_1 c 1 , and (80) . For k = 1 k=1 k = 1 the first bracket of (81) is T c 0 = q Tc_0=q T c 0 = q and each c m i c_{m_i} c m i is reproduced, so
E [ c 1 g ] = ( q + 2 deg g ) E [ g ] , deg g : = ∑ i m i , (84) \mathbb{E}[c_1g]=(q+2\deg g)\mathbb{E}[g],\qquad\deg g:=\sum_i m_i, \tag*{(84)} E [ c 1 g ] = ( q + 2 deg g ) E [ g ] , deg g := i ∑ m i , (84)
and iterating, E [ c 1 2 F ] = ( q + 2 d + 2 ) ( q + 2 d ) E [ F ] \mathbb{E}[c_1^2F]=(q+2d+2)(q+2d)\mathbb{E}[F] E [ c 1 2 F ] = ( q + 2 d + 2 ) ( q + 2 d ) E [ F ] for F F F homogeneous of degree d d d in C C C ; d = 4 d=4 d = 4 gives (80) .
Step 3: degree at most four. By (81) with g = 1 g=1 g = 1 and k = 1 , 2 , 3 k=1,2,3 k = 1 , 2 , 3 , and by (84) ,
E [ c 1 ] = q , E [ c 1 2 ] = ( q + 2 ) q , E [ c 2 ] = ( T + 1 + d 0 ) E [ c 1 ] = q ( s + 1 ) , \mathbb{E}[c_1]=q,\qquad\mathbb{E}[c_1^2]=(q+2)q,\qquad\mathbb{E}[c_2]=(T+1+d_0)\mathbb{E}[c_1]=q(s+1), E [ c 1 ] = q , E [ c 1 2 ] = ( q + 2 ) q , E [ c 2 ] = ( T + 1 + d 0 ) E [ c 1 ] = q ( s + 1 ) ,
E [ c 1 c 2 ] = ( q + 4 ) E [ c 2 ] = q ( q + 4 ) ( s + 1 ) , \mathbb{E}[c_1c_2]=(q+4)\mathbb{E}[c_2]=q(q+4)(s+1), E [ c 1 c 2 ] = ( q + 4 ) E [ c 2 ] = q ( q + 4 ) ( s + 1 ) ,
E [ c 3 ] = ( T + 2 + d 0 ) E [ c 2 ] + E [ c 1 2 ] = q [ ( s + 1 ) ( s + 2 ) + q + 2 ] = q [ s 2 + 3 s + q + 4 ] . \mathbb{E}[c_3]=(T+2+d_0)\mathbb{E}[c_2]+\mathbb{E}[c_1^2]
=q[(s+1)(s+2)+q+2]=q[s^2+3s+q+4]. E [ c 3 ] = ( T + 2 + d 0 ) E [ c 2 ] + E [ c 1 2 ] = q [( s + 1 ) ( s + 2 ) + q + 2 ] = q [ s 2 + 3 s + q + 4 ] .
Then (81) with ( k , g ) = ( 4 , 1 ) (k,g)=(4,1) ( k , g ) = ( 4 , 1 ) and ( 2 , c 2 ) (2,c_2) ( 2 , c 2 ) gives
E [ c 4 ] = ( T + 3 + d 0 ) E [ c 3 ] + 2 E [ c 1 c 2 ] = q [ ( s + 3 ) ( s 2 + 3 s + q + 4 ) + 2 ( s + 1 ) ( q + 4 ) ] = q [ s 3 + 6 s 2 + 3 ( q + 7 ) s + 5 ( q + 4 ) ] , \mathbb{E}[c_4]=(T+3+d_0)\mathbb{E}[c_3]+2\mathbb{E}[c_1c_2]
=q[(s+3)(s^2+3s+q+4)+2(s+1)(q+4)]
=q[s^3+6s^2+3(q+7)s+5(q+4)], E [ c 4 ] = ( T + 3 + d 0 ) E [ c 3 ] + 2 E [ c 1 c 2 ] = q [( s + 3 ) ( s 2 + 3 s + q + 4 ) + 2 ( s + 1 ) ( q + 4 )] = q [ s 3 + 6 s 2 + 3 ( q + 7 ) s + 5 ( q + 4 )] ,
E [ c 2 2 ] = ( T + 1 + d 0 ) E [ c 1 c 2 ] + 4 E [ c 3 ] = q [ ( s + 1 ) 2 ( q + 4 ) + 4 ( s 2 + 3 s + q + 4 ) ] = q [ ( q + 8 ) s 2 + 2 ( q + 10 ) s + 5 ( q + 4 ) ] , \mathbb{E}[c_2^2]=(T+1+d_0)\mathbb{E}[c_1c_2]+4\mathbb{E}[c_3]
=q[(s+1)^2(q+4)+4(s^2+3s+q+4)]
=q[(q+8)s^2+2(q+10)s+5(q+4)], E [ c 2 2 ] = ( T + 1 + d 0 ) E [ c 1 c 2 ] + 4 E [ c 3 ] = q [( s + 1 ) 2 ( q + 4 ) + 4 ( s 2 + 3 s + q + 4 )] = q [( q + 8 ) s 2 + 2 ( q + 10 ) s + 5 ( q + 4 )] ,
which are (76) and (77) .
Step 4: degree five. By (81) with ( k , g ) = ( 5 , 1 ) (k,g)=(5,1) ( k , g ) = ( 5 , 1 ) and ( 3 , c 2 ) (3,c_2) ( 3 , c 2 ) , and by (84) for E [ c 1 c 3 ] = ( q + 6 ) E [ c 3 ] \mathbb{E}[c_1c_3]=(q+6)\mathbb{E}[c_3] E [ c 1 c 3 ] = ( q + 6 ) E [ c 3 ] and E [ c 1 2 c 2 ] = ( q + 4 ) ( q + 6 ) E [ c 2 ] \mathbb{E}[c_1^2c_2]=(q+4)(q+6)\mathbb{E}[c_2] E [ c 1 2 c 2 ] = ( q + 4 ) ( q + 6 ) E [ c 2 ] ,
E [ c 5 ] = ( T + 4 + d 0 ) E [ c 4 ] + 2 E [ c 1 c 3 ] + E [ c 1 2 c 2 ] = q [ s 4 + 10 s 3 + ( 6 q + 65 ) s 2 + 5 ( 5 q + 32 ) s + ( q + 4 ) ( 2 q + 37 ) ] , \mathbb{E}[c_5]=(T+4+d_0)\mathbb{E}[c_4]+2\mathbb{E}[c_1c_3]+\mathbb{E}[c_1^2c_2]
=q[s^4+10s^3+(6q+65)s^2+5(5q+32)s+(q+4)(2q+37)], E [ c 5 ] = ( T + 4 + d 0 ) E [ c 4 ] + 2 E [ c 1 c 3 ] + E [ c 1 2 c 2 ] = q [ s 4 + 10 s 3 + ( 6 q + 65 ) s 2 + 5 ( 5 q + 32 ) s + ( q + 4 ) ( 2 q + 37 )] ,
E [ c 2 c 3 ] = ( T + 2 + d 0 ) E [ c 2 2 ] + E [ c 1 2 c 2 ] + 4 E [ c 4 ] = q [ ( q + 12 ) s 3 + 4 ( q + 15 ) s 2 + ( q + 7 ) ( q + 24 ) s + ( q + 4 ) ( q + 36 ) ] . \mathbb{E}[c_2c_3]=(T+2+d_0)\mathbb{E}[c_2^2]+\mathbb{E}[c_1^2c_2]+4\mathbb{E}[c_4]
=q[(q+12)s^3+4(q+15)s^2+(q+7)(q+24)s+(q+4)(q+36)]. E [ c 2 c 3 ] = ( T + 2 + d 0 ) E [ c 2 2 ] + E [ c 1 2 c 2 ] + 4 E [ c 4 ] = q [( q + 12 ) s 3 + 4 ( q + 15 ) s 2 + ( q + 7 ) ( q + 24 ) s + ( q + 4 ) ( q + 36 )] .
Step 5: degree six. By (81) with ( k , g ) = ( 4 , c 2 ) (k,g)=(4,c_2) ( k , g ) = ( 4 , c 2 ) and ( 2 , c 2 2 ) (2,c_2^2) ( 2 , c 2 2 ) , and by (84) for E [ c 1 c 2 2 ] = ( q + 8 ) E [ c 2 2 ] \mathbb{E}[c_1c_2^2]=(q+8)\mathbb{E}[c_2^2] E [ c 1 c 2 2 ] = ( q + 8 ) E [ c 2 2 ] ,
E [ c 2 c 4 ] = ( T + 3 + d 0 ) E [ c 2 c 3 ] + 2 E [ c 1 c 2 2 ] + 4 E [ c 5 ] = ( s + 3 ) E [ c 2 c 3 ] + 2 ( q + 8 ) E [ c 2 2 ] + 4 E [ c 5 ] , \mathbb{E}[c_2c_4]=(T+3+d_0)\mathbb{E}[c_2c_3]+2\mathbb{E}[c_1c_2^2]+4\mathbb{E}[c_5]
=(s+3)\mathbb{E}[c_2c_3]+2(q+8)\mathbb{E}[c_2^2]+4\mathbb{E}[c_5], E [ c 2 c 4 ] = ( T + 3 + d 0 ) E [ c 2 c 3 ] + 2 E [ c 1 c 2 2 ] + 4 E [ c 5 ] = ( s + 3 ) E [ c 2 c 3 ] + 2 ( q + 8 ) E [ c 2 2 ] + 4 E [ c 5 ] ,
E [ c 2 3 ] = ( T + 1 + d 0 ) E [ c 1 c 2 2 ] + 8 E [ c 2 c 3 ] = ( s + 1 ) ( q + 8 ) E [ c 2 2 ] + 8 E [ c 2 c 3 ] ; \mathbb{E}[c_2^3]=(T+1+d_0)\mathbb{E}[c_1c_2^2]+8\mathbb{E}[c_2c_3]
=(s+1)(q+8)\mathbb{E}[c_2^2]+8\mathbb{E}[c_2c_3]; E [ c 2 3 ] = ( T + 1 + d 0 ) E [ c 1 c 2 2 ] + 8 E [ c 2 c 3 ] = ( s + 1 ) ( q + 8 ) E [ c 2 2 ] + 8 E [ c 2 c 3 ] ;
substituting Steps 3 and 4 and collecting powers of s s s gives (78) and (79) .
Step 6: orders in d 0 d_0 d 0 . All coefficients above are positive, so each moment is Θ \Theta Θ of its highest power of d 0 d_0 d 0 , read off from the displayed formulas: d 0 ⋅ d 0 3 d_0\cdot d_0^3 d 0 ⋅ d 0 3 for E [ c 4 ] \mathbb{E}[c_4] E [ c 4 ] , d 0 ⋅ d 0 ⋅ d 0 2 d_0\cdot d_0\cdot d_0^2 d 0 ⋅ d 0 ⋅ d 0 2 for E [ c 2 2 ] \mathbb{E}[c_2^2] E [ c 2 2 ] , d 0 ⋅ d 0 ⋅ d 0 4 d_0\cdot d_0\cdot d_0^4 d 0 ⋅ d 0 ⋅ d 0 4 for E [ c 2 c 4 ] \mathbb{E}[c_2c_4] E [ c 2 c 4 ] , d 0 ⋅ d 0 2 ⋅ d 0 3 d_0\cdot d_0^2\cdot d_0^3 d 0 ⋅ d 0 2 ⋅ d 0 3 for E [ c 2 3 ] \mathbb{E}[c_2^3] E [ c 2 3 ] , and d 0 2 d_0^2 d 0 2 times E [ c 4 ] \mathbb{E}[c_4] E [ c 4 ] and E [ c 2 2 ] \mathbb{E}[c_2^2] E [ c 2 2 ] for (80) . Since A 1 A_1 A 1 and A 2 A_2 A 2 are positive combinations of degree-six moments by (66) , they are Θ ( d 0 6 ) \Theta(d_0^6) Θ ( d 0 6 ) ; A 3 = A 1 + A 2 A_3=A_1+A_2 A 3 = A 1 + A 2 holds by definition. ■
Proofs
The proof of Theorem 4 has three steps: the isotropy of E [ G ∙ G ∙ ⊤ ∣ X ] \mathbb{E}[G_{\bullet}G_{\bullet}^{\top}\mid X] E [ G ∙ G ∙ ⊤ ∣ X ] for ∙ ∈ { Q , K , V } \bullet\in\{Q,K,V\} ∙ ∈ { Q , K , V } (Lemma 34); the conditional closed forms of E [ G O G O ⊤ ∣ X ] \mathbb{E}[G_O G_O^{\top}\mid X] E [ G O G O ⊤ ∣ X ] (Lemma 35) and of the isotropic constants (Lemma 36); and the average over X X X .
Lemma 34 (Isotropy). Under Assumption 8, conditionally on X X X ,
E [ G Q G Q ⊤ ∣ X ] = c Q ( C ) I d k , E [ G K G K ⊤ ∣ X ] = c K ( C ) I d k , \mathbb{E}[G_QG_Q^{\top}\mid X]=c_Q(C)I_{d_k},\qquad
\mathbb{E}[G_KG_K^{\top}\mid X]=c_K(C)I_{d_k}, E [ G Q G Q ⊤ ∣ X ] = c Q ( C ) I d k , E [ G K G K ⊤ ∣ X ] = c K ( C ) I d k , E [ G V G V ⊤ ∣ X ] = c V ( C ) I d v , \mathbb{E}[G_VG_V^{\top}\mid X]=c_V(C)I_{d_v}, E [ G V G V ⊤ ∣ X ] = c V ( C ) I d v , with c Q ( C ) : = 1 d k E [ ∥ G Q ∥ F 2 ∣ X ] c_Q(C):=\frac{1}{d_k}\mathbb{E}[\|G_Q\|_F^2\mid X] c Q ( C ) := d k 1 E [ ∥ G Q ∥ F 2 ∣ X ] , c K ( C ) : = 1 d k E [ ∥ G K ∥ F 2 ∣ X ] c_K(C):=\frac{1}{d_k}\mathbb{E}[\|G_K\|_F^2\mid X] c K ( C ) := d k 1 E [ ∥ G K ∥ F 2 ∣ X ] and c V ( C ) : = 1 d v E [ ∥ G V ∥ F 2 ∣ X ] c_V(C):=\frac{1}{d_v}\mathbb{E}[\|G_V\|_F^2\mid X] c V ( C ) := d v 1 E [ ∥ G V ∥ F 2 ∣ X ] .
Proof. Fix X X X and let W : = ( W Q , W K , W V , W O ) W := (W_Q,W_K,W_V,W_O) W := ( W Q , W K , W V , W O ) . Let D ∈ R d k × d k D \in\mathbb{R}^{d_k \times d_k} D ∈ R d k × d k be diagonal with entries in { ± 1 } \{\pm1\} { ± 1 } , and let τ D \tau_D τ D denote the map ( W Q , W K , W V , W O ) ↦ ( D W Q , D W K , W V , W O ) (W_Q,W_K,W_V,W_O) \mapsto(DW_Q,DW_K,W_V,W_O) ( W Q , W K , W V , W O ) ↦ ( D W Q , D W K , W V , W O ) . Since ( D W Q ) ⊤ ( D W K ) = W Q ⊤ W K (DW_Q)^\top(DW_K) = W_Q^\top W_K ( D W Q ) ⊤ ( D W K ) = W Q ⊤ W K , the matrix A A A in (64) , and hence L \mathcal{L} L , is invariant: L ∘ τ D = L \mathcal{L} \circ\tau_D = \mathcal{L} L ∘ τ D = L . Differentiating this identity with respect to W Q W_Q W Q by the chain rule gives D ⊤ G Q ( τ D W ) = G Q ( W ) D^\top G_Q(\tau_D W) = G_Q(W) D ⊤ G Q ( τ D W ) = G Q ( W ) , i.e.
G Q ( τ D W ) = D G Q ( W ) . (85) G_Q(\tau_D W) = D G_Q(W). \tag*{(85)} G Q ( τ D W ) = D G Q ( W ) . (85)
Under Assumption 8, τ D W = d W \tau_D W \mathrel{\overset{d}{=}} W τ D W = d W , so
E [ G Q G Q ⊤ ∣ X ] = E [ G Q ( τ D W ) G Q ( τ D W ) ⊤ ∣ X ] = D E [ G Q G Q ⊤ ∣ X ] D . \mathbb{E}[G_QG_Q^\top\mid X] = \mathbb{E}[G_Q(\tau_D W)G_Q(\tau_D W)^\top\mid X] = D\mathbb{E}[G_QG_Q^\top\mid X]D. E [ G Q G Q ⊤ ∣ X ] = E [ G Q ( τ D W ) G Q ( τ D W ) ⊤ ∣ X ] = D E [ G Q G Q ⊤ ∣ X ] D .
For a ≠ b a \ne b a = b , choosing D a a = 1 D_{aa}=1 D aa = 1 and D b b = − 1 D_{bb}=-1 D bb = − 1 gives E [ ( G Q G Q ⊤ ) a b ∣ X ] = − E [ ( G Q G Q ⊤ ) a b ∣ X ] \mathbb{E}[(G_QG_Q^\top)_{ab} \mid X] = -\mathbb{E}[(G_QG_Q^\top)_{ab} \mid X] E [( G Q G Q ⊤ ) ab ∣ X ] = − E [( G Q G Q ⊤ ) ab ∣ X ] , hence 0 0 0 . Likewise, for a permutation matrix P ∈ R d k × d k P \in\mathbb{R}^{d_k \times d_k} P ∈ R d k × d k the map ( W Q , W K ) ↦ ( P W Q , P W K ) (W_Q,W_K) \mapsto(PW_Q,PW_K) ( W Q , W K ) ↦ ( P W Q , P W K ) preserves W Q ⊤ W K W_Q^\top W_K W Q ⊤ W K and the law of W W W , and the same argument gives E [ G Q G Q ⊤ ∣ X ] = P E [ G Q G Q ⊤ ∣ X ] P ⊤ \mathbb{E}[G_QG_Q^\top\mid X] = P\mathbb{E}[G_QG_Q^\top\mid X]P^\top E [ G Q G Q ⊤ ∣ X ] = P E [ G Q G Q ⊤ ∣ X ] P ⊤ ; hence all diagonal entries are equal, and their common value is 1 d k E [ tr ( G Q G Q ⊤ ) ∣ X ] = c Q ( C ) \frac{1}{d_k}\mathbb{E}[\operatorname{tr}(G_QG_Q^\top)\mid X] = c_Q(C) d k 1 E [ tr ( G Q G Q ⊤ ) ∣ X ] = c Q ( C ) . The same argument with W Q W_Q W Q and W K W_K W K exchanged gives the claim for G K G_K G K .
For G V G_V G V , let D ∈ R d v × d v D \in\mathbb{R}^{d_v \times d_v} D ∈ R d v × d v be diagonal with entries in { ± 1 } \{\pm1\} { ± 1 } and let τ D \tau_D τ D map ( W V , W O ) ↦ ( D W V , W O D ) (W_V,W_O) \mapsto(DW_V,W_OD) ( W V , W O ) ↦ ( D W V , W O D ) , fixing W Q , W K W_Q,W_K W Q , W K . Since f ( X ) = A X ( W O W V ) ⊤ f(X) = AX(W_OW_V)^\top f ( X ) = A X ( W O W V ) ⊤ depends on ( W V , W O ) (W_V,W_O) ( W V , W O ) only through the product W O W V W_OW_V W O W V , which equals ( W O D ) ( D W V ) (W_OD)(DW_V) ( W O D ) ( D W V ) , L ∘ τ D = L \mathcal{L} \circ\tau_D = \mathcal{L} L ∘ τ D = L ; differentiating with respect to W V W_V W V gives G V ( τ D W ) = D G V ( W ) G_V(\tau_D W) = DG_V(W) G V ( τ D W ) = D G V ( W ) , and ( D W V , W O D ) = d ( W V , W O ) (DW_V,W_OD) \mathrel{\overset{d}{=}} (W_V,W_O) ( D W V , W O D ) = d ( W V , W O ) under Assumption 8. The sign and permutation arguments above apply verbatim. ▪
Lemma 35 (Conditional closed form for W O W_O W O ). Under Assumption 8, conditionally on X X X ,
E [ G O G O ⊤ ∣ X ] = α O ( C ) I d 2 + d v c 2 Φ C 2 Φ ⊤ , (86) \mathbb{E}[G_OG_O^\top\mid X] = \alpha_O(C)I_{d_2} + d_v c_2 \Phi C^2 \Phi^\top, \tag*{(86)} E [ G O G O ⊤ ∣ X ] = α O ( C ) I d 2 + d v c 2 Φ C 2 Φ ⊤ , (86) α O ( C ) = d v ( d v + 1 ) ( c 1 2 c 4 + c 2 c 4 + c 2 3 ) + d v ( c 1 2 c 2 + 2 c 2 c 4 ) + d v ( d v + 1 ) d k ( c 1 2 c 4 + c 1 2 c 2 2 + 3 c 2 c 4 + c 2 3 ) + 2 d v d k ( c 1 2 c 4 + 2 c 2 c 4 + c 2 3 ) . (87) \alpha_O(C) = d_v(d_v+1)(c_1^2c_4+c_2c_4+c_2^3) + d_v(c_1^2c_2+2c_2c_4)
+ \frac{d_v(d_v+1)}{d_k}(c_1^2c_4+c_1^2c_2^2+3c_2c_4+c_2^3) + \frac{2d_v}{d_k}(c_1^2c_4+2c_2c_4+c_2^3). \tag*{(87)} α O ( C ) = d v ( d v + 1 ) ( c 1 2 c 4 + c 2 c 4 + c 2 3 ) + d v ( c 1 2 c 2 + 2 c 2 c 4 ) + d k d v ( d v + 1 ) ( c 1 2 c 4 + c 1 2 c 2 2 + 3 c 2 c 4 + c 2 3 ) + d k 2 d v ( c 1 2 c 4 + 2 c 2 c 4 + c 2 3 ) . (87)
Proof. Let Ω : = A Z Z ⊤ A ⊤ \Omega:= AZZ^\top A^\top Ω := A Z Z ⊤ A ⊤ . Substituting R = A Z W O ⊤ − Y R = AZW_O^\top- Y R = A Z W O ⊤ − Y into G O G O ⊤ = R ⊤ Ω R G_OG_O^\top= R^\top\Omega R G O G O ⊤ = R ⊤ Ω R ,
G O G O ⊤ = W O Z ⊤ A ⊤ Ω A Z W O ⊤ − W O Z ⊤ A ⊤ Ω Y − Y ⊤ Ω A Z W O ⊤ + Y ⊤ Ω Y . (88) G_OG_O^\top= W_OZ^\top A^\top\Omega AZW_O^\top
- W_OZ^\top A^\top\Omega Y
- Y^\top\Omega AZW_O^\top
+ Y^\top\Omega Y. \tag*{(88)} G O G O ⊤ = W O Z ⊤ A ⊤ Ω A Z W O ⊤ − W O Z ⊤ A ⊤ Ω Y − Y ⊤ Ω A Z W O ⊤ + Y ⊤ Ω Y . (88)
The terms T 2 T_2 T 2 , T 3 T_3 T 3 are linear in W O W_O W O , which is centered and independent of ( X , W Q , W K , W V ) (X,W_Q,W_K,W_V) ( X , W Q , W K , W V ) , so E [ T 2 ∣ X ] = E [ T 3 ∣ X ] = 0 \mathbb{E}[T_2 \mid X] = \mathbb{E}[T_3 \mid X] = 0 E [ T 2 ∣ X ] = E [ T 3 ∣ X ] = 0 .
The term T 4 T_4 T 4 . By Lemma 27, E [ Z Z ⊤ ∣ X , W Q , W K ] = X E [ W V ⊤ W V ] X ⊤ = d v X X ⊤ \mathbb{E}[ZZ^\top\mid X,W_Q,W_K] = X\mathbb{E}[W_V^\top W_V]X^\top= d_vXX^\top E [ Z Z ⊤ ∣ X , W Q , W K ] = X E [ W V ⊤ W V ] X ⊤ = d v X X ⊤ , hence
E [ T 4 ∣ X , W Q , W K ] = d v Φ ( X ⊤ A X X ⊤ A ⊤ X ) Φ ⊤ , \mathbb{E}[T_4 \mid X,W_Q,W_K] = d_v\Phi(X^\top AXX^\top A^\top X)\Phi^\top, E [ T 4 ∣ X , W Q , W K ] = d v Φ ( X ⊤ A X X ⊤ A ⊤ X ) Φ ⊤ ,
X ⊤ A X X ⊤ A ⊤ X = 1 d k C W Q ⊤ W K C 2 W K ⊤ W Q C . X^\top AXX^\top A^\top X = \frac{1}{d_k}CW_Q^\top W_KC^2W_K^\top W_QC. X ⊤ A X X ⊤ A ⊤ X = d k 1 C W Q ⊤ W K C 2 W K ⊤ W Q C .
By Lemma 27, E [ W K C 2 W K ⊤ ∣ X ] = c 2 I d k \mathbb{E}[W_KC^2W_K^\top\mid X] = c_2I_{d_k} E [ W K C 2 W K ⊤ ∣ X ] = c 2 I d k and then E [ W Q ⊤ W Q ] = d k I d 0 \mathbb{E}[W_Q^\top W_Q] = d_kI_{d_0} E [ W Q ⊤ W Q ] = d k I d 0 , so
E [ T 4 ∣ X ] = d v c 2 Φ C 2 Φ ⊤ . (89) \mathbb{E}[T_4 \mid X] = d_vc_2\Phi C^2\Phi^\top. \tag*{(89)} E [ T 4 ∣ X ] = d v c 2 Φ C 2 Φ ⊤ . (89)
The term T 1 T_1 T 1 . By Lemma 27 applied to W O W_O W O ,
E [ T 1 ∣ X , W Q , W K , W V ] = tr ( Z ⊤ A ⊤ Ω A Z ) I d 2 = tr ( ( A ⊤ A Z Z ⊤ ) 2 ) I d 2 . \mathbb{E}[T_1 \mid X,W_Q,W_K,W_V] = \operatorname{tr}(Z^\top A^\top\Omega AZ)I_{d_2} = \operatorname{tr}((A^\top AZZ^\top)^2)I_{d_2}. E [ T 1 ∣ X , W Q , W K , W V ] = tr ( Z ⊤ A ⊤ Ω A Z ) I d 2 = tr (( A ⊤ A Z Z ⊤ ) 2 ) I d 2 .
Let N : = W Q C W Q ⊤ N := W_QCW_Q^\top N := W Q C W Q ⊤ , S : = W K ⊤ N W K S := W_K^\top NW_K S := W K ⊤ N W K , H V : = W V ⊤ W V ∼ W d 0 ( d v , I d 0 ) H_V := W_V^\top W_V \sim\mathcal{W}_{d_0}(d_v,I_{d_0}) H V := W V ⊤ W V ∼ W d 0 ( d v , I d 0 ) and M : = C S C M := CSC M := C S C , which is symmetric. Then A ⊤ A = 1 d k X S X ⊤ A^\top A = \frac{1}{d_k}XSX^\top A ⊤ A = d k 1 X S X ⊤ and Z Z ⊤ = X H V X ⊤ ZZ^\top= XH_VX^\top Z Z ⊤ = X H V X ⊤ , so
tr ( ( A ⊤ A Z Z ⊤ ) 2 ) = 1 d k 2 tr ( M H V M H V ) . \operatorname{tr}((A^\top AZZ^\top)^2) = \frac{1}{d_k^2}\operatorname{tr}(MH_VMH_V). tr (( A ⊤ A Z Z ⊤ ) 2 ) = d k 2 1 tr ( M H V M H V ) .
Since H V H_V H V is independent of M M M , Lemma 29 with ( n , d ) = ( d v , d 0 ) (n,d)=(d_v,d_0) ( n , d ) = ( d v , d 0 ) gives
E [ tr ( M H V M H V ) ∣ X , W Q , W K ] = d v ( d v + 1 ) tr ( M 2 ) + d v ( tr M ) 2 . \mathbb{E}[\operatorname{tr}(MH_VMH_V)\mid X,W_Q,W_K] = d_v(d_v+1)\operatorname{tr}(M^2)+d_v(\operatorname{tr}M)^2. E [ tr ( M H V M H V ) ∣ X , W Q , W K ] = d v ( d v + 1 ) tr ( M 2 ) + d v ( tr M ) 2 .
Now M = tr ( S C 2 ) M=\operatorname{tr}(SC^{2}) M = tr ( S C 2 ) and tr M 2 = tr ( S C 2 S C 2 ) \operatorname{tr}M^{2}=\operatorname{tr}(SC^{2}SC^{2}) tr M 2 = tr ( S C 2 S C 2 ) , so Lemma 31 (conditionally on N N N ) followed by Lemma 30 give
E [ tr M 2 ∣ X ] = d k 2 ( c 1 2 c 4 + c 2 c 4 + c 3 2 ) + d k ( c 1 2 c 4 + c 1 2 c 2 2 + 3 c 2 c 4 + c 3 2 ) , E [ ( tr M ) 2 ∣ X ] = d k 2 ( c 1 2 c 2 2 + 2 c 2 c 4 ) + 2 d k ( c 1 2 c 4 + c 2 c 4 + c 3 2 ) . (90) \begin{aligned}
\mathbb{E}[\operatorname{tr}M^{2}\mid X]&=d_{k}^{2}(c_{1}^{2}c_{4}+c_{2}c_{4}+c_{3}^{2})+d_{k}(c_{1}^{2}c_{4}+c_{1}^{2}c_{2}^{2}+3c_{2}c_{4}+c_{3}^{2}),\\
\mathbb{E}[(\operatorname{tr}M)^{2}\mid X]&=d_{k}^{2}(c_{1}^{2}c_{2}^{2}+2c_{2}c_{4})+2d_{k}(c_{1}^{2}c_{4}+c_{2}c_{4}+c_{3}^{2}). \tag*{(90)}
\end{aligned} E [ tr M 2 ∣ X ] E [( tr M ) 2 ∣ X ] = d k 2 ( c 1 2 c 4 + c 2 c 4 + c 3 2 ) + d k ( c 1 2 c 4 + c 1 2 c 2 2 + 3 c 2 c 4 + c 3 2 ) , = d k 2 ( c 1 2 c 2 2 + 2 c 2 c 4 ) + 2 d k ( c 1 2 c 4 + c 2 c 4 + c 3 2 ) . (90)
Therefore E [ T 1 ∣ X ] = α O ( C ) I d 2 \mathbb{E}[T_{1}\mid X]=\alpha_{O}(C)I_{d_{2}} E [ T 1 ∣ X ] = α O ( C ) I d 2 with α O ( C ) = 1 d k 2 ( d v ( d v + 1 ) E [ tr M 2 ∣ X ] + d v E [ ( tr M ) 2 ∣ X ] ) \alpha_{O}(C)=\frac{1}{d_{k}^{2}}\left(d_{v}(d_{v}+1)\mathbb{E}[\operatorname{tr}M^{2}\mid X]+d_{v}\mathbb{E}[(\operatorname{tr}M)^{2}\mid X]\right) α O ( C ) = d k 2 1 ( d v ( d v + 1 ) E [ tr M 2 ∣ X ] + d v E [( tr M ) 2 ∣ X ] ) , which is (87) ; adding (89) gives (86) .
Lemma 36 (Conditional constants). Under Assumption 8 , conditionally on X X X , with t Φ : = tr ( Φ C 2 Φ ⊤ ) t_{\Phi}:=\operatorname{tr}(\Phi C^{2}\Phi^{\top}) t Φ := tr ( Φ C 2 Φ ⊤ ) , the constants of Lemma 34 are
c Q ( C ) = c 2 d k ( β 1 + d v t Φ ) + c 2 d k 2 ( β 1 + β 2 ) , (91) c_{Q}(C)=\frac{c_{2}}{d_{k}}(\beta_{1}+d_{v}t_{\Phi})+\frac{c_{2}}{d_{k}^{2}}(\beta_{1}+\beta_{2}), \tag*{(91)} c Q ( C ) = d k c 2 ( β 1 + d v t Φ ) + d k 2 c 2 ( β 1 + β 2 ) , (91) c K ( C ) = c 2 d k ( β 1 + d v t Φ ) + c 1 2 + c 2 d k 2 β 1 , (92) c_{K}(C)=\frac{c_{2}}{d_{k}}(\beta_{1}+d_{v}t_{\Phi})+\frac{c_{1}^{2}+c_{2}}{d_{k}^{2}}\beta_{1}, \tag*{(92)} c K ( C ) = d k c 2 ( β 1 + d v t Φ ) + d k 2 c 1 2 + c 2 β 1 , (92) c V ( C ) = d 2 ( d 2 + d v + 1 ) [ ( c 1 2 c 4 + c 2 c 4 + c 3 2 ) + 1 d k ( c 1 2 c 4 + c 1 2 c 2 2 + 3 c 2 c 4 + c 3 2 ) ] + c 2 t Φ . (93) c_{V}(C)=d_{2}(d_{2}+d_{v}+1)\left[\left(c_{1}^{2}c_{4}+c_{2}c_{4}+c_{3}^{2}\right)+\frac{1}{d_{k}}\left(c_{1}^{2}c_{4}+c_{1}^{2}c_{2}^{2}+3c_{2}c_{4}+c_{3}^{2}\right)\right]+c_{2}t_{\Phi}. \tag*{(93)} c V ( C ) = d 2 ( d 2 + d v + 1 ) [ ( c 1 2 c 4 + c 2 c 4 + c 3 2 ) + d k 1 ( c 1 2 c 4 + c 1 2 c 2 2 + 3 c 2 c 4 + c 3 2 ) ] + c 2 t Φ . (93)
Proof. Substituting R = A Z W O ⊤ − Y R=AZW_{O}^{\top}-Y R = A Z W O ⊤ − Y into (67) splits each gradient as G ∙ = a ∙ − b ∙ G_{\mathord{\bullet}}=a_{\mathord{\bullet}}-b_{\mathord{\bullet}} G ∙ = a ∙ − b ∙ , where a ∙ a_{\mathord{\bullet}} a ∙ collects the term containing A Z W O ⊤ AZW_{O}^{\top} A Z W O ⊤ and b ∙ b_{\mathord{\bullet}} b ∙ the term containing Y Y Y . For ∙ ∈ { Q , K } \mathord{\bullet}\in\{Q,K\} ∙ ∈ { Q , K } , a ∙ a_{\mathord{\bullet}} a ∙ is quadratic and b ∙ b_{\mathord{\bullet}} b ∙ linear in W O W_{O} W O ; for ∙ = V \mathord{\bullet}=V ∙ = V , a V a_{V} a V is quadratic and b V b_{V} b V linear in W O W_{O} W O as well. Since W O W_{O} W O is centered and independent of everything else, E [ tr ( a ∙ b ∙ ⊤ ) ∣ X ] = 0 \mathbb{E}[\operatorname{tr}(a_{\mathord{\bullet}}b_{\mathord{\bullet}}^{\top})\mid X]=0 E [ tr ( a ∙ b ∙ ⊤ ) ∣ X ] = 0 and
E [ ∥ G ∙ ∥ F 2 ∣ X ] = E [ ∥ a ∙ ∥ F 2 ∣ X ] + E [ ∥ b ∙ ∥ F 2 ∣ X ] . \mathbb{E}[\lVert G_{\mathord{\bullet}}\rVert_{F}^{2}\mid X]=\mathbb{E}[\lVert a_{\mathord{\bullet}}\rVert_{F}^{2}\mid X]+\mathbb{E}[\lVert b_{\mathord{\bullet}}\rVert_{F}^{2}\mid X]. E [∥ G ∙ ∥ F 2 ∣ X ] = E [∥ a ∙ ∥ F 2 ∣ X ] + E [∥ b ∙ ∥ F 2 ∣ X ] .
Throughout, H O : = W O ⊤ W O H_{O}:=W_{O}^{\top}W_{O} H O := W O ⊤ W O , E : = C W V ⊤ H O W V C E:=CW_{V}^{\top}H_{O}W_{V}C E := C W V ⊤ H O W V C (as in Lemma 32 ), N : = W Q C W Q ⊤ N:=W_{Q}CW_{Q}^{\top} N := W Q C W Q ⊤ (as in Lemma 30 ), and we use the identities X ⊤ Z = C W V ⊤ X^{\top}Z=CW_{V}^{\top} X ⊤ Z = C W V ⊤ and X ⊤ A ⊤ X = 1 d k C W K ⊤ W Q C X^{\top}A^{\top}X=\frac{1}{\sqrt{d_{k}}}CW_{K}^{\top}W_{Q}C X ⊤ A ⊤ X = d k 1 C W K ⊤ W Q C .
Target parts. From (67) , b Q = 1 d k W K C W V ⊤ W O ⊤ Φ C b_{Q}=\frac{1}{\sqrt{d_{k}}}W_{K}CW_{V}^{\top}W_{O}^{\top}\Phi C b Q = d k 1 W K C W V ⊤ W O ⊤ Φ C . By Lemma 27 applied to W K W_{K} W K and cyclicity of the trace,
E [ ∥ b Q ∥ F 2 ∣ X , W V , W O ] = tr ( W V C 2 W V ⊤ ⋅ W O ⊤ Φ C 2 Φ ⊤ W O ) , \mathbb{E}[\lVert b_{Q}\rVert_{F}^{2}\mid X,W_{V},W_{O}]=\operatorname{tr}(W_{V}C^{2}W_{V}^{\top}\cdot W_{O}^{\top}\Phi C^{2}\Phi^{\top}W_{O}), E [∥ b Q ∥ F 2 ∣ X , W V , W O ] = tr ( W V C 2 W V ⊤ ⋅ W O ⊤ Φ C 2 Φ ⊤ W O ) ,
and Lemma 27 applied to W V W_{V} W V (E [ W V C 2 W V ⊤ ] = c 2 I d v \mathbb{E}[W_{V}C^{2}W_{V}^{\top}]=c_{2}I_{d_{v}} E [ W V C 2 W V ⊤ ] = c 2 I d v ) and to W O W_{O} W O (E [ W O ⊤ Φ C 2 Φ ⊤ W O ] = t Φ I d v \mathbb{E}[W_{O}^{\top}\Phi C^{2}\Phi^{\top}W_{O}]=t_{\Phi}I_{d_{v}} E [ W O ⊤ Φ C 2 Φ ⊤ W O ] = t Φ I d v ) gives E [ ∥ b Q ∥ F 2 ∣ X ] = d v c 2 t Φ \mathbb{E}[\lVert b_{Q}\rVert_{F}^{2}\mid X]=d_{v}c_{2}t_{\Phi} E [∥ b Q ∥ F 2 ∣ X ] = d v c 2 t Φ . The same computation applies to b K = 1 d k W Q C Φ ⊤ W O W V C b_{K}=\frac{1}{\sqrt{d_{k}}}W_{Q}C\Phi^{\top}W_{O}W_{V}C b K = d k 1 W Q C Φ ⊤ W O W V C (Lemma 27 applied to W Q W_{Q} W Q ) and to b V = W O ⊤ A X b_{V}=W_{O}^{\top}A X b V = W O ⊤ A X (Lemma 27 applied to W O W_{O} W O , then (89) for the average of X ⊤ A X X ⊤ A ⊤ X X^{\top}AXX^{\top}A^{\top}X X ⊤ A X X ⊤ A ⊤ X over W K , W Q W_{K},W_{Q} W K , W Q ), so
E [ ∥ b Q ∥ F 2 ∣ X ] = E [ ∥ b K ∥ F 2 ∣ X ] = E [ ∥ b V ∥ F 2 ∣ X ] = d v c 2 t Φ . \mathbb{E}[\lVert b_{Q}\rVert_{F}^{2}\mid X]=\mathbb{E}[\lVert b_{K}\rVert_{F}^{2}\mid X]=\mathbb{E}[\lVert b_{V}\rVert_{F}^{2}\mid X]=d_{v}c_{2}t_{\Phi}. E [∥ b Q ∥ F 2 ∣ X ] = E [∥ b K ∥ F 2 ∣ X ] = E [∥ b V ∥ F 2 ∣ X ] = d v c 2 t Φ .
Model part of G Q G_{Q} G Q . Using X ⊤ Z = C W V ⊤ X^{\top}Z=CW_{V}^{\top} X ⊤ Z = C W V ⊤ and Z ⊤ A ⊤ X = W V X ⊤ A ⊤ X = 1 d k W V C W K ⊤ W Q C Z^{\top}A^{\top}X=W_{V}X^{\top}A^{\top}X=\frac{1}{\sqrt{d_{k}}}W_{V}CW_{K}^{\top}W_{Q}C Z ⊤ A ⊤ X = W V X ⊤ A ⊤ X = d k 1 W V C W K ⊤ W Q C ,
a Q = 1 d k W K E W K ⊤ W Q C = 1 d k J W Q C , ∥ a Q ∥ F 2 = 1 d k 2 tr ( J 2 N ~ ) , a_{Q}=\frac{1}{d_{k}}W_{K}EW_{K}^{\top}W_{Q}C=\frac{1}{d_{k}}JW_{Q}C,\qquad\lVert a_{Q}\rVert_{F}^{2}=\frac{1}{d_{k}^{2}}\operatorname{tr}(J^{2}\widetilde{N}), a Q = d k 1 W K E W K ⊤ W Q C = d k 1 J W Q C , ∥ a Q ∥ F 2 = d k 2 1 tr ( J 2 N ) ,
where J : = W K E W K ⊤ J:=W_{K}EW_{K}^{\top} J := W K E W K ⊤ and N ~ : = W Q C 2 W Q ⊤ \widetilde{N}:=W_{Q}C^{2}W_{Q}^{\top} N := W Q C 2 W Q ⊤ . Given X X X , J J J and N ~ \widetilde{N} N are independent, and E [ N ~ ∣ X ] = c 2 I d k \mathbb{E}[\widetilde{N}\mid X]=c_{2}I_{d_{k}} E [ N ∣ X ] = c 2 I d k by Lemma 27, so E [ ∥ a Q ∥ F 2 ∣ X ] = c 2 d k 2 E [ tr J 2 ∣ X ] \mathbb{E}[\lVert a_{Q}\rVert_{F}^{2}\mid X]=\frac{c_{2}}{d_{k}^{2}}\mathbb{E}[\operatorname{tr}J^{2}\mid X] E [∥ a Q ∥ F 2 ∣ X ] = d k 2 c 2 E [ tr J 2 ∣ X ] . By Lemma 30 with E E E in place of C C C and W K W_{K} W K in place of W Q W_{Q} W Q , E [ tr J 2 ∣ X , E ] = d k 2 tr ( E 2 ) + d k ( ( tr E ) 2 + tr ( E 2 ) ) \mathbb{E}[\operatorname{tr}J^{2}\mid X,E]=d_{k}^{2}\operatorname{tr}(E^{2})+d_{k}((\operatorname{tr}E)^{2}+\operatorname{tr}(E^{2})) E [ tr J 2 ∣ X , E ] = d k 2 tr ( E 2 ) + d k (( tr E ) 2 + tr ( E 2 )) , and by Lemma 32 , E [ tr J 2 ∣ X ] = d k 2 β 1 + d k ( β 1 + β 2 ) \mathbb{E}[\operatorname{tr}J^{2}\mid X]=d_{k}^{2}\beta_{1}+d_{k}(\beta_{1}+\beta_{2}) E [ tr J 2 ∣ X ] = d k 2 β 1 + d k ( β 1 + β 2 ) . Hence E [ ∥ a Q ∥ F 2 ∣ X ] = c 2 β 1 + c 2 d k ( β 1 + β 2 ) \mathbb{E}[\lVert a_{Q}\rVert_{F}^{2}\mid X]=c_{2}\beta_{1}+\frac{c_{2}}{d_{k}}(\beta_{1}+\beta_{2}) E [∥ a Q ∥ F 2 ∣ X ] = c 2 β 1 + d k c 2 ( β 1 + β 2 ) , and c Q ( C ) = 1 d k ( E [ ∥ a Q ∥ F 2 ∣ X ] + E [ ∥ b Q ∥ F 2 ∣ X ] ) c_{Q}(C)=\frac{1}{d_{k}}\left(\mathbb{E}[\lVert a_{Q}\rVert_{F}^{2}\mid X]+\mathbb{E}[\lVert b_{Q}\rVert_{F}^{2}\mid X]\right) c Q ( C ) = d k 1 ( E [∥ a Q ∥ F 2 ∣ X ] + E [∥ b Q ∥ F 2 ∣ X ] ) is (91) .
Model part of G K G_{K} G K . Using X ⊤ A Z = 1 d k C W Q ⊤ W K C W V ⊤ X^{\top}AZ=\frac{1}{\sqrt{d_{k}}}CW_{Q}^{\top}W_{K}CW_{V}^{\top} X ⊤ A Z = d k 1 C W Q ⊤ W K C W V ⊤ and Z ⊤ X = W V C Z^{\top}X=W_{V}C Z ⊤ X = W V C ,
a K = 1 d k N W K E , ∥ a K ∥ F 2 = 1 d k 2 tr ( N 2 W K E 2 W K ⊤ ) . a_{K}=\frac{1}{d_{k}}NW_{K}E,\qquad\lVert a_{K}\rVert_{F}^{2}=\frac{1}{d_{k}^{2}}\operatorname{tr}(N^{2}W_{K}E^{2}W_{K}^{\top}). a K = d k 1 N W K E , ∥ a K ∥ F 2 = d k 2 1 tr ( N 2 W K E 2 W K ⊤ ) .
Given X X X , the matrices N N N , W K W_{K} W K and E E E are independent. By Lemma 27, E [ W K E 2 W K ⊤ ∣ X , E ] = tr ( E 2 ) I d k \mathbb{E}[W_{K}E^{2}W_{K}^{\top}\mid X,E]=\operatorname{tr}(E^{2})I_{d_{k}} E [ W K E 2 W K ⊤ ∣ X , E ] = tr ( E 2 ) I d k ; by Lemma 30 , E [ tr N 2 ∣ X ] = d k 2 c 2 + d k ( c 1 2 + c 2 ) \mathbb{E}[\operatorname{tr}N^{2}\mid X]=d_{k}^{2}c_{2}+d_{k}(c_{1}^{2}+c_{2}) E [ tr N 2 ∣ X ] = d k 2 c 2 + d k ( c 1 2 + c 2 ) ; and by Lemma 32 , E [ tr E 2 ∣ X ] = β 1 \mathbb{E}[\operatorname{tr}E^{2}\mid X]=\beta_{1} E [ tr E 2 ∣ X ] = β 1 . Hence E [ ∥ a K ∥ F 2 ∣ X ] = ( c 2 + c 1 2 + c 2 d k ) β 1 \mathbb{E}[\lVert a_{K}\rVert_{F}^{2}\mid X]=(c_{2}+\frac{c_{1}^{2}+c_{2}}{d_{k}})\beta_{1} E [∥ a K ∥ F 2 ∣ X ] = ( c 2 + d k c 1 2 + c 2 ) β 1 , which together with the target part gives (92) . Model part of G V G_{\mathrm{V}} G V . Here a V = W O ⊤ W O Z ⊤ A ⊤ A X = H O Z ⊤ A ⊤ A X a_{\mathrm{V}}=W_{\mathrm{O}}^{\top}W_{\mathrm{O}}Z^{\top}A^{\top}AX=H_{\mathrm{O}}Z^{\top}A^{\top}AX a V = W O ⊤ W O Z ⊤ A ⊤ A X = H O Z ⊤ A ⊤ A X , so
∥ a V ∥ F 2 = tr ( H O 2 M V ) , M V : = Z ⊤ A ⊤ A X X ⊤ A ⊤ A Z = 1 d k 2 W V C S C 2 C W V ⊤ . \lVert a_{\mathrm{V}}\rVert_{\mathrm{F}}^{2}=\operatorname{tr}(H_{\mathrm{O}}^{2}M_{\mathrm{V}}),\qquad M_{\mathrm{V}}:=Z^{\top}A^{\top}AXX^{\top}A^{\top}AZ=\frac{1}{d_{k}^{2}}W_{\mathrm{V}}CSC^{2}C W_{\mathrm{V}}^{\top}. ∥ a V ∥ F 2 = tr ( H O 2 M V ) , M V := Z ⊤ A ⊤ A X X ⊤ A ⊤ A Z = d k 2 1 W V C S C 2 C W V ⊤ .
with S S S as in the proof of Lemma 35. Given X X X , H O H_{\mathrm{O}} H O is independent of M V M_{\mathrm{V}} M V and E [ H O 2 ] = d 2 ( d 2 + d V + 1 ) I d V \mathbb{E}[H_{\mathrm{O}}^{2}]=d_{2}(d_{2}+d_{\mathrm{V}}+1)I_{d_{\mathrm{V}}} E [ H O 2 ] = d 2 ( d 2 + d V + 1 ) I d V by Lemma 29 , so E [ ∥ a V ∥ F 2 ∣ X ] = d 2 ( d 2 + d V + 1 ) E [ tr M V ∣ X ] \mathbb{E}[\lVert a_{\mathrm{V}}\rVert_{\mathrm{F}}^{2}\mid X]=d_{2}(d_{2}+d_{\mathrm{V}}+1)\mathbb{E}[\operatorname{tr}M_{\mathrm{V}}\mid X] E [∥ a V ∥ F 2 ∣ X ] = d 2 ( d 2 + d V + 1 ) E [ tr M V ∣ X ] . Taking the expectation over W V W_{\mathrm{V}} W V (E [ W V ⊤ W V ] = d V I d 0 \mathbb{E}[W_{\mathrm{V}}^{\top}W_{\mathrm{V}}]=d_{\mathrm{V}}I_{d_{0}} E [ W V ⊤ W V ] = d V I d 0 ) and then over W K , W Q W_{\mathrm{K}},W_{\mathrm{Q}} W K , W Q by Lemmas 30 and 31 ,
E [ tr M V ∣ X ] = d V d k 2 E [ tr ( S C 2 S C 2 ) ∣ X ] = d V [ ( c 1 2 c 4 + c 2 c 4 + c 3 2 ) + 1 d k ( c 1 2 c 4 + c 1 2 c 2 2 + 3 c 2 c 4 + c 3 2 ) ] . \mathbb{E}[\operatorname{tr}M_{\mathrm{V}}\mid X]=\frac{d_{\mathrm{V}}}{d_{k}^{2}}\mathbb{E}[\operatorname{tr}(SC^{2}SC^{2})\mid X]=d_{\mathrm{V}}\left[\left(c_{1}^{2}c_{4}+c_{2}c_{4}+c_{3}^{2}\right)+\frac{1}{d_{k}}\left(c_{1}^{2}c_{4}+c_{1}^{2}c_{2}^{2}+3c_{2}c_{4}+c_{3}^{2}\right)\right]. E [ tr M V ∣ X ] = d k 2 d V E [ tr ( S C 2 S C 2 ) ∣ X ] = d V [ ( c 1 2 c 4 + c 2 c 4 + c 3 2 ) + d k 1 ( c 1 2 c 4 + c 1 2 c 2 2 + 3 c 2 c 4 + c 3 2 ) ] .
which is the first line of (90) divided by d k 2 d_{k}^{2} d k 2 . Adding the target part and dividing by d V d_{\mathrm{V}} d V gives (93) .
Proof of Theorem 4. Take expectations over X X X in Lemmas 34 to 36 ; by the tower property and Assumption 8,
E [ G O G O ⊤ ] = E [ α O ( C ) ] I d 2 + d V Φ E [ c 2 C 2 ] Φ ⊤ , E [ G ∙ G ∙ ⊤ ] = E [ c ∙ ( C ) ] I ( ∙ ∈ { Q , K , V } ) . \mathbb{E}[G_{\mathrm{O}}G_{\mathrm{O}}^{\top}]=\mathbb{E}[\alpha_{\mathrm{O}}(C)]I_{d_{2}}+d_{\mathrm{V}}\Phi\mathbb{E}[c_{2}C^{2}]\Phi^{\top},\qquad\mathbb{E}[G_{\bullet}G_{\bullet}^{\top}]=\mathbb{E}[c_{\bullet}(C)]I\quad(\bullet\in\{Q,K,V\}). E [ G O G O ⊤ ] = E [ α O ( C )] I d 2 + d V Φ E [ c 2 C 2 ] Φ ⊤ , E [ G ∙ G ∙ ⊤ ] = E [ c ∙ ( C )] I ( ∙ ∈ { Q , K , V }) .
For Q ∈ O ( d 0 ) Q\in O(d_{0}) Q ∈ O ( d 0 ) , Q C Q ⊤ = d C QCQ^{\top}\mathrel{\overset{d}{=}}C QC Q ⊤ = d C and c 2 c_{2} c 2 is invariant, so E [ c 2 C 2 ] = Q E [ c 2 C 2 ] Q ⊤ \mathbb{E}[c_{2}C^{2}]=Q\mathbb{E}[c_{2}C^{2}]Q^{\top} E [ c 2 C 2 ] = Q E [ c 2 C 2 ] Q ⊤ for all Q Q Q , and by Schur’s lemma E [ c 2 C 2 ] = γ I d 0 \mathbb{E}[c_{2}C^{2}]=\gamma I_{d_{0}} E [ c 2 C 2 ] = γ I d 0 with γ = 1 d 0 E [ c 2 tr C 2 ] = 1 d 0 E [ c 2 2 ] \gamma=\frac{1}{d_{0}}\mathbb{E}[c_{2}\operatorname{tr}C^{2}]=\frac{1}{d_{0}}\mathbb{E}[c_{2}^{2}] γ = d 0 1 E [ c 2 tr C 2 ] = d 0 1 E [ c 2 2 ] ; (77) gives the polynomial and the order in (70) . This proves (69) , and averaging (87) termwise with (66) gives the closed form of α ‾ O = E [ α O ( C ) ] \overline{\alpha}_{\mathrm{O}}=\mathbb{E}[\alpha_{\mathrm{O}}(C)] α O = E [ α O ( C )] in (71) . The same isotropy argument gives E [ c 2 Φ ] = tr ( Φ E [ c 2 C 2 ] Φ ⊤ ) = γ ∥ Φ ∥ F 2 \mathbb{E}[c_{2}\Phi]=\operatorname{tr}(\Phi\mathbb{E}[c_{2}C^{2}]\Phi^{\top})=\gamma\lVert\Phi\rVert_{\mathrm{F}}^{2} E [ c 2 Φ ] = tr ( Φ E [ c 2 C 2 ] Φ ⊤ ) = γ ∥ Φ ∥ F 2 , so averaging (91) –(93) gives (68) , where by (75)
E [ c 2 β 1 ] = d 2 d V [ ( d 2 d V + d 2 + d V + 3 ) E [ c 2 c 4 ] + ( d 2 + d V + 1 ) E [ c 2 3 ] ] , \mathbb{E}[c_{2}\beta_{1}]=d_{2}d_{\mathrm{V}}\left[(d_{2}d_{\mathrm{V}}+d_{2}+d_{\mathrm{V}}+3)\mathbb{E}[c_{2}c_{4}]+(d_{2}+d_{\mathrm{V}}+1)\mathbb{E}[c_{2}^{3}]\right], E [ c 2 β 1 ] = d 2 d V [ ( d 2 d V + d 2 + d V + 3 ) E [ c 2 c 4 ] + ( d 2 + d V + 1 ) E [ c 2 3 ] ] ,
E [ c 2 β 2 ] = d 2 d V [ ( d 2 d V + 2 ) E [ c 2 3 ] + 2 ( d 2 + d V + 1 ) E [ c 2 c 4 ] ] , \mathbb{E}[c_{2}\beta_{2}]=d_{2}d_{\mathrm{V}}\left[(d_{2}d_{\mathrm{V}}+2)\mathbb{E}[c_{2}^{3}]+2(d_{2}+d_{\mathrm{V}}+1)\mathbb{E}[c_{2}c_{4}]\right], E [ c 2 β 2 ] = d 2 d V [ ( d 2 d V + 2 ) E [ c 2 3 ] + 2 ( d 2 + d V + 1 ) E [ c 2 c 4 ] ] ,
E [ ( c 1 2 + c 2 ) β 1 ] = d 2 d V [ ( d 2 d V + d 2 + d V + 3 ) ( E [ c 1 2 c 4 ] + E [ c 2 c 4 ] ) + ( d 2 + d V + 1 ) ( E [ c 1 2 c 2 2 ] + E [ c 2 3 ] ) ] . (94) \mathbb{E}[(c_{1}^{2}+c_{2})\beta_{1}]=d_{2}d_{\mathrm{V}}\left[(d_{2}d_{\mathrm{V}}+d_{2}+d_{\mathrm{V}}+3)(\mathbb{E}[c_{1}^{2}c_{4}]+\mathbb{E}[c_{2}c_{4}])+(d_{2}+d_{\mathrm{V}}+1)(\mathbb{E}[c_{1}^{2}c_{2}^{2}]+\mathbb{E}[c_{2}^{3}])\right]. \tag*{(94)} E [( c 1 2 + c 2 ) β 1 ] = d 2 d V [ ( d 2 d V + d 2 + d V + 3 ) ( E [ c 1 2 c 4 ] + E [ c 2 c 4 ]) + ( d 2 + d V + 1 ) ( E [ c 1 2 c 2 2 ] + E [ c 2 3 ]) ] . (94)
all explicit by Lemma 33 . Finally, substituting A 3 = A 1 + A 2 A_{3}=A_{1}+A_{2} A 3 = A 1 + A 2 into the closed form of α ‾ O \overline{\alpha}_{\mathrm{O}} α O ,
α ‾ O = d V [ ( d V + 1 ) ( 1 + 1 d k ) + 2 d k ] A 1 + d V [ 1 + d V + 1 d k ] A 2 = Θ ( d 2 2 d 0 6 ) , (95) \overline{\alpha}_{\mathrm{O}}=d_{\mathrm{V}}\left[(d_{\mathrm{V}}+1)\left(1+\frac{1}{d_{k}}\right)+\frac{2}{d_{k}}\right]A_{1}+d_{\mathrm{V}}\left[1+\frac{d_{\mathrm{V}}+1}{d_{k}}\right]A_{2}=\Theta(d_{2}^{2}d_{0}^{6}), \tag*{(95)} α O = d V [ ( d V + 1 ) ( 1 + d k 1 ) + d k 2 ] A 1 + d V [ 1 + d k d V + 1 ] A 2 = Θ ( d 2 2 d 0 6 ) , (95)
all terms being nonnegative and A 1 , A 2 = Θ ( d 0 6 ) A_{1},A_{2}=\Theta(d_{0}^{6}) A 1 , A 2 = Θ ( d 0 6 ) by Lemma 33 , which is the order stated in (71) .
Proof of Corollary 3. All Wishart moments of Lemma 33 are positive, and d V , d k ≥ 1 d_{\mathrm{V}},d_{k}\ge1 d V , d k ≥ 1 throughout.
Step 1: the β \beta β -moments. By (94) , each of
E [ c 2 β 1 ] = d 2 d V [ ( d 2 d V + d 2 + d V + 3 ) E [ c 2 c 4 ] + ( d 2 + d V + 1 ) E [ c 2 3 ] ] , \mathbb{E}[c_{2}\beta_{1}]=d_{2}d_{\mathrm{V}}\left[(d_{2}d_{\mathrm{V}}+d_{2}+d_{\mathrm{V}}+3)\mathbb{E}[c_{2}c_{4}]+(d_{2}+d_{\mathrm{V}}+1)\mathbb{E}[c_{2}^{3}]\right], E [ c 2 β 1 ] = d 2 d V [ ( d 2 d V + d 2 + d V + 3 ) E [ c 2 c 4 ] + ( d 2 + d V + 1 ) E [ c 2 3 ] ] ,
E [ c 2 β 2 ] = d 2 d V [ ( d 2 d V + 2 ) E [ c 2 3 ] + 2 ( d 2 + d V + 1 ) E [ c 2 c 4 ] ] , \mathbb{E}[c_{2}\beta_{2}]=d_{2}d_{\mathrm{V}}\left[(d_{2}d_{\mathrm{V}}+2)\mathbb{E}[c_{2}^{3}]+2(d_{2}+d_{\mathrm{V}}+1)\mathbb{E}[c_{2}c_{4}]\right], E [ c 2 β 2 ] = d 2 d V [ ( d 2 d V + 2 ) E [ c 2 3 ] + 2 ( d 2 + d V + 1 ) E [ c 2 c 4 ] ] ,
E [ ( c 1 2 + c 2 ) β 1 ] = d 2 d V [ ( d 2 d V + d 2 + d V + 3 ) ( E [ c 1 2 c 4 ] + E [ c 2 c 4 ] ) + ( d 2 + d V + 1 ) ( E [ c 1 2 c 2 2 ] + E [ c 2 3 ] ) ] \mathbb{E}[(c_{1}^{2}+c_{2})\beta_{1}]=d_{2}d_{\mathrm{V}}\left[(d_{2}d_{\mathrm{V}}+d_{2}+d_{\mathrm{V}}+3)(\mathbb{E}[c_{1}^{2}c_{4}]+\mathbb{E}[c_{2}c_{4}])+(d_{2}+d_{\mathrm{V}}+1)(\mathbb{E}[c_{1}^{2}c_{2}^{2}]+\mathbb{E}[c_{2}^{3}])\right] E [( c 1 2 + c 2 ) β 1 ] = d 2 d V [ ( d 2 d V + d 2 + d V + 3 ) ( E [ c 1 2 c 4 ] + E [ c 2 c 4 ]) + ( d 2 + d V + 1 ) ( E [ c 1 2 c 2 2 ] + E [ c 2 3 ]) ]
is d 2 d V d_{2}d_{\mathrm{V}} d 2 d V times a polynomial of degree one in d V d_{\mathrm{V}} d V whose coefficients are positive combinations of the degree-six moments E [ c 2 c 4 ] \mathbb{E}[c_{2}c_{4}] E [ c 2 c 4 ] , E [ c 2 3 ] \mathbb{E}[c_{2}^{3}] E [ c 2 3 ] , E [ c 1 2 c 4 ] \mathbb{E}[c_{1}^{2}c_{4}] E [ c 1 2 c 4 ] , E [ c 1 2 c 2 2 ] = Θ ( d 0 6 ) \mathbb{E}[c_{1}^{2}c_{2}^{2}]=\Theta(d_{0}^{6}) E [ c 1 2 c 2 2 ] = Θ ( d 0 6 ) of Lemma 33 ; since d 2 d V ≥ 1 2 ( d 2 + d V ) d_{2}d_{\mathrm{V}}\ge\frac{1}{2}(d_{2}+d_{\mathrm{V}}) d 2 d V ≥ 2 1 ( d 2 + d V ) , each is
E [ c 2 β 1 ] = Θ ( d 2 2 d V 2 d 0 6 ) , E [ c 2 β 2 ] = Θ ( d 2 2 d V 2 d 0 6 ) , E [ ( c 1 2 + c 2 ) β 1 ] = Θ ( d 2 2 d V 2 d 0 6 ) . (96) \mathbb{E}[c_{2}\beta_{1}]=\Theta(d_{2}^{2}d_{\mathrm{V}}^{2}d_{0}^{6}),\qquad\mathbb{E}[c_{2}\beta_{2}]=\Theta(d_{2}^{2}d_{\mathrm{V}}^{2}d_{0}^{6}),\qquad\mathbb{E}[(c_{1}^{2}+c_{2})\beta_{1}]=\Theta(d_{2}^{2}d_{\mathrm{V}}^{2}d_{0}^{6}). \tag*{(96)} E [ c 2 β 1 ] = Θ ( d 2 2 d V 2 d 0 6 ) , E [ c 2 β 2 ] = Θ ( d 2 2 d V 2 d 0 6 ) , E [( c 1 2 + c 2 ) β 1 ] = Θ ( d 2 2 d V 2 d 0 6 ) . (96)
Step 2: c Q c_{Q} c Q and c K c_{K} c K . Substituting (96) into (68) ,
c Q = 1 d k ( E [ c 2 β 1 ] + d V γ ∥ Φ ∥ F 2 ) ⏟ Θ ( d 2 2 d V 2 d 0 6 ) Θ ( d V d 0 3 ) + 1 d k 2 ( E [ c 2 β 1 ] + E [ c 2 β 2 ] ) ⏟ Θ ( d 2 2 d V 2 d 0 6 ) , c K = 1 d k ( E [ c 2 β 1 ] + d V γ ∥ Φ ∥ F 2 ) ⏟ Θ ( d 2 2 d V 2 d 0 6 ) Θ ( d V d 0 3 ) + 1 d k 2 E [ ( c 1 2 + c 2 ) β 1 ] ⏟ Θ ( d 2 2 d V 2 d 0 6 ) . \begin{aligned}
c_{Q}&=\frac{1}{d_{k}}\underbrace{\left(\mathbb{E}[c_{2}\beta_{1}]+d_{\mathrm{V}}\gamma\lVert\Phi\rVert_{\mathrm{F}}^{2}\right)}_{\Theta(d_{2}^{2}d_{\mathrm{V}}^{2}d_{0}^{6})\qquad\Theta(d_{\mathrm{V}}d_{0}^{3})}+\frac{1}{d_{k}^{2}}\underbrace{\left(\mathbb{E}[c_{2}\beta_{1}]+\mathbb{E}[c_{2}\beta_{2}]\right)}_{\Theta(d_{2}^{2}d_{\mathrm{V}}^{2}d_{0}^{6})},\\
c_{K}&=\frac{1}{d_{k}}\underbrace{\left(\mathbb{E}[c_{2}\beta_{1}]+d_{\mathrm{V}}\gamma\lVert\Phi\rVert_{\mathrm{F}}^{2}\right)}_{\Theta(d_{2}^{2}d_{\mathrm{V}}^{2}d_{0}^{6})\qquad\Theta(d_{\mathrm{V}}d_{0}^{3})}+\frac{1}{d_{k}^{2}}\underbrace{\mathbb{E}[(c_{1}^{2}+c_{2})\beta_{1}]}_{\Theta(d_{2}^{2}d_{\mathrm{V}}^{2}d_{0}^{6})}.
\end{aligned} c Q c K = d k 1 Θ ( d 2 2 d V 2 d 0 6 ) Θ ( d V d 0 3 ) ( E [ c 2 β 1 ] + d V γ ∥ Φ ∥ F 2 ) + d k 2 1 Θ ( d 2 2 d V 2 d 0 6 ) ( E [ c 2 β 1 ] + E [ c 2 β 2 ] ) , = d k 1 Θ ( d 2 2 d V 2 d 0 6 ) Θ ( d V d 0 3 ) ( E [ c 2 β 1 ] + d V γ ∥ Φ ∥ F 2 ) + d k 2 1 Θ ( d 2 2 d V 2 d 0 6 ) E [( c 1 2 + c 2 ) β 1 ] .
where γ = Θ ( d 0 3 ) \gamma= \Theta(d_{0}^{3}) γ = Θ ( d 0 3 ) by Lemma 33 . All terms are positive, and since d 2 , d v , d k , d 0 ≥ 1 d_{2}, d_{v}, d_{k}, d_{0} \ge1 d 2 , d v , d k , d 0 ≥ 1 the second and third terms are at most a constant multiple of the first; hence
c Q ≍ c K = Θ ( d 2 2 d v 2 d 0 6 d k ) , c K − c Q = E [ c 1 2 β 1 ] − E [ c 2 β 2 ] d k 2 = O ( d 2 2 d v 2 d 0 6 d k 2 ) = O ( 1 d k ) c Q . c_{Q} \asymp c_{K} = \Theta\left(\frac{d_{2}^{2}d_{v}^{2}d_{0}^{6}}{d_{k}}\right), \qquad c_{K} - c_{Q} = \frac{\mathbb{E}[c_{1}^{2}\beta_{1}] - \mathbb{E}[c_{2}\beta_{2}]}{d_{k}^{2}} = O\left(\frac{d_{2}^{2}d_{v}^{2}d_{0}^{6}}{d_{k}^{2}}\right) = O\left(\frac{1}{d_{k}}\right)c_{Q}. c Q ≍ c K = Θ ( d k d 2 2 d v 2 d 0 6 ) , c K − c Q = d k 2 E [ c 1 2 β 1 ] − E [ c 2 β 2 ] = O ( d k 2 d 2 2 d v 2 d 0 6 ) = O ( d k 1 ) c Q .
Step 3: c V c_{V} c V . By (68) with A 3 = A 1 + A 2 A_{3} = A_{1} + A_{2} A 3 = A 1 + A 2 ,
c V = d 2 ( d 2 + d v + 1 ) ( A 1 + A 3 d k ) + γ ∥ Φ ∥ F 2 , d 2 A 1 ≤ d 2 ( A 1 + A 3 d k ) ≤ d 2 ( A 1 + A 3 ) , c_{V} = d_{2}(d_{2}+d_{v}+1)\left(A_{1}+\frac{A_{3}}{d_{k}}\right) + \gamma\|\Phi\|_{F}^{2}, \qquad d_{2}A_{1} \le d_{2}\left(A_{1}+\frac{A_{3}}{d_{k}}\right) \le d_{2}(A_{1}+A_{3}), c V = d 2 ( d 2 + d v + 1 ) ( A 1 + d k A 3 ) + γ ∥Φ ∥ F 2 , d 2 A 1 ≤ d 2 ( A 1 + d k A 3 ) ≤ d 2 ( A 1 + A 3 ) ,
so c V c_{V} c V is affine in d v d_{v} d v with slope between d 2 A 1 d_{2}A_{1} d 2 A 1 and d 2 ( A 1 + A 3 ) d_{2}(A_{1}+A_{3}) d 2 ( A 1 + A 3 ) and intercept between d 2 ( d 2 + 1 ) A 1 d_{2}(d_{2}+1)A_{1} d 2 ( d 2 + 1 ) A 1 and d 2 ( d 2 + 1 ) ( A 1 + A 3 ) + γ ∥ Φ ∥ F 2 d_{2}(d_{2}+1)(A_{1}+A_{3})+\gamma\|\Phi\|_{F}^{2} d 2 ( d 2 + 1 ) ( A 1 + A 3 ) + γ ∥Φ ∥ F 2 ; since A 1 , A 2 = Θ ( d 0 6 ) A_{1}, A_{2} = \Theta(d_{0}^{6}) A 1 , A 2 = Θ ( d 0 6 ) and γ = Θ ( d 0 3 ) \gamma= \Theta(d_{0}^{3}) γ = Θ ( d 0 3 ) (Lemma 33 ), c V = Θ ( d 2 ( d 2 + d v ) d 0 6 ) c_{V} = \Theta\left(d_{2}(d_{2}+d_{v})d_{0}^{6}\right) c V = Θ ( d 2 ( d 2 + d v ) d 0 6 ) .
Step 4: α ˉ O \bar{\alpha}_{O} α ˉ O . Let α ˉ O ∞ : = d v ( d v + 1 ) A 1 + d v A 2 \bar{\alpha}_{O}^{\infty} := d_{v}(d_{v}+1)A_{1}+d_{v}A_{2} α ˉ O ∞ := d v ( d v + 1 ) A 1 + d v A 2 be the d k d_{k} d k -free part of (95) . Then
α ˉ O − α ˉ O ∞ = 1 d k [ d v ( d v + 1 ) A 3 + 2 d v A 1 ] ≤ 1 d k [ d v ( d v + 1 ) ( A 1 + A 2 ) + 2 d v ( d v + 1 ) A 1 ] = 3 A 1 + A 2 A 1 ⋅ d v ( d v + 1 ) A 1 d k ≤ 3 A 1 + A 2 A 1 ⋅ α ˉ O ∞ d k , \begin{aligned}
\bar{\alpha}_{O}-\bar{\alpha}_{O}^{\infty}=\frac{1}{d_{k}}\left[d_{v}(d_{v}+1)A_{3}+2d_{v}A_{1}\right]\le\frac{1}{d_{k}}\left[d_{v}(d_{v}+1)(A_{1}+A_{2})+2d_{v}(d_{v}+1)A_{1}\right] \\
&=\frac{3A_{1}+A_{2}}{A_{1}}\cdot\frac{d_{v}(d_{v}+1)A_{1}}{d_{k}}\le\frac{3A_{1}+A_{2}}{A_{1}}\cdot\frac{\bar{\alpha}_{O}^{\infty}}{d_{k}},
\end{aligned} α ˉ O − α ˉ O ∞ = d k 1 [ d v ( d v + 1 ) A 3 + 2 d v A 1 ] ≤ d k 1 [ d v ( d v + 1 ) ( A 1 + A 2 ) + 2 d v ( d v + 1 ) A 1 ] = A 1 3 A 1 + A 2 ⋅ d k d v ( d v + 1 ) A 1 ≤ A 1 3 A 1 + A 2 ⋅ d k α ˉ O ∞ ,
so that
α ˉ O ∞ ≤ α ˉ O ≤ ( 1 + 3 A 1 + A 2 A 1 d k ) α ˉ O ∞ , d v 2 A 1 ≤ α ˉ O ∞ ≤ 2 d v 2 A 1 + d v 2 A 2 . (97) \bar{\alpha}_{O}^{\infty}\le\bar{\alpha}_{O}\le\left(1+\frac{3A_{1}+A_{2}}{A_{1}d_{k}}\right)\bar{\alpha}_{O}^{\infty}, \qquad d_{v}^{2}A_{1}\le\bar{\alpha}_{O}^{\infty}\le2d_{v}^{2}A_{1}+d_{v}^{2}A_{2}. \tag*{(97)} α ˉ O ∞ ≤ α ˉ O ≤ ( 1 + A 1 d k 3 A 1 + A 2 ) α ˉ O ∞ , d v 2 A 1 ≤ α ˉ O ∞ ≤ 2 d v 2 A 1 + d v 2 A 2 . (97)
and therefore, with A 1 , A 2 = Θ ( d 0 6 ) A_{1}, A_{2}=\Theta(d_{0}^{6}) A 1 , A 2 = Θ ( d 0 6 ) , α ˉ O = Θ ( d v 2 d 0 6 ) \bar{\alpha}_{O}=\Theta(d_{v}^{2}d_{0}^{6}) α ˉ O = Θ ( d v 2 d 0 6 ) .
Step 5: the W O W_{O} W O ratio. By (69) , for a ≠ b a\ne b a = b with ( Φ Φ T ) a b ≠ 0 (\Phi\Phi^{\mathsf{T}})_{ab}\ne0 ( Φ Φ T ) ab = 0 ,
E [ ( G O G O T ) a a ] = α ˉ O + d v γ ( Φ Φ T ) a a = Θ ( d v 2 d 0 6 ) , \mathbb{E}[(G_{O}G_{O}^{\mathsf{T}})_{aa}]=\bar{\alpha}_{O}+d_{v}\gamma(\Phi\Phi^{\mathsf{T}})_{aa}=\Theta(d_{v}^{2}d_{0}^{6}), E [( G O G O T ) aa ] = α ˉ O + d v γ ( Φ Φ T ) aa = Θ ( d v 2 d 0 6 ) ,
E [ ( G O G O T ) a b ] = d v γ ( Φ Φ T ) a b = Θ ( d v d 0 3 ) , \mathbb{E}[(G_{O}G_{O}^{\mathsf{T}})_{ab}]=d_{v}\gamma(\Phi\Phi^{\mathsf{T}})_{ab}=\Theta(d_{v}d_{0}^{3}), E [( G O G O T ) ab ] = d v γ ( Φ Φ T ) ab = Θ ( d v d 0 3 ) ,
and dividing gives the ratio Θ ( d v d 0 3 ) \Theta(d_{v}d_{0}^{3}) Θ ( d v d 0 3 ) . Finally, Steps 3 and 4 show that d k d_{k} d k enters c V c_{V} c V and α ˉ O \bar{\alpha}_{O} α ˉ O only through the factors 1 + A 3 / ( A 1 d k ) ∈ [ 1 , 1 + A 3 / A 1 ] 1+A_{3}/(A_{1}d_{k})\in[1,1+A_{3}/A_{1}] 1 + A 3 / ( A 1 d k ) ∈ [ 1 , 1 + A 3 / A 1 ] and the bracket in (97) , both bounded above and below; the same is then true of the ratio.
Discussion
Remark 3 (Scope of the isotropy). This remark records how far the exact zeros of Theorem 4 for W Q , W K , W V W_{Q},W_{K},W_{V} W Q , W K , W V extend beyond the setting of Assumption 8 . The proof of Lemma 34 uses only that f f f depends on ( W Q , W K ) (W_{Q},W_{K}) ( W Q , W K ) through W Q T W K W_{Q}^{\mathsf{T}}W_{K} W Q T W K and on ( W V , W O ) (W_{V},W_{O}) ( W V , W O ) through W O W V W_{O}W_{V} W O W V , and that the law of the weights is invariant under the sign and permutation maps. Hence the isotropy of G Q , G K , G V G_{Q},G_{K},G_{V} G Q , G K , G V holds conditionally on every dataset, for any differentiable loss, for any initialization whose law has these symmetries, and, applying the sign matrix to stacked heads, for multi-head attention. RoPE inserts a position-dependent matrix between queries and keys and is not covered. For these three matrices P ∝ I P\propto I P ∝ I in expectation, so orthogonalization degenerates to the scalar normalization G / ∥ G ∥ 2 G/\|G\|_{2} G /∥ G ∥ 2 .
Remark 4 (Role of data randomness). This remark identifies which part of the argument actually uses the Gaussian tokens, since the answer differs from the ReLU setting. Unlike the ReLU setting, where Gaussian data creates the cancellation (κ 1 , κ 2 \kappa_{1},\kappa_{2} κ 1 , κ 2 are input integrals), here the data enters no cancellation: the zeros are weight-space symmetries holding per dataset, and (86) is an identity for every dataset. For instance, since α O ( C ) ≥ d v 2 c 2 \alpha_{O}(C)\ge d_{v}^{2}c_{2} α O ( C ) ≥ d v 2 c 2 and c 2 ≥ λ max ( C ) 2 c_{2}\ge\lambda_{\max}(C)^{2} c 2 ≥ λ m a x ( C ) 2 , (86) gives E [ ( G O G O T ) a a ∣ X ] / E [ ( G O G O T ) a b ∣ X ] ≥ d v λ max ( C ) 2 / ( ∥ φ a ∥ ∥ φ b ∥ ) \mathbb{E}[(G_{O}G_{O}^{\mathsf{T}})_{aa}\mid X]/\mathbb{E}[(G_{O}G_{O}^{\mathsf{T}})_{ab}\mid X]\ge d_{v}\lambda_{\max}(C)^{2}/(\|\varphi_{a}\|\|\varphi_{b}\|) E [( G O G O T ) aa ∣ X ] / E [( G O G O T ) ab ∣ X ] ≥ d v λ m a x ( C ) 2 / ( ∥ φ a ∥∥ φ b ∥ ) deterministically, with φ a \varphi_{a} φ a the rows of Φ \Phi Φ . Gaussian tokens only translate c 1 , c 2 , c 4 c_{1},c_{2},c_{4} c 1 , c 2 , c 4 into ( T , d 0 ) (T,d_{0}) ( T , d 0 ) via Lemma 33 ; what is genuinely needed is independence of the data from the initialization, since a data-dependent initialization breaks the symmetry argument.
Remark 5 (Sign of c K − c Q c_{K}-c_{Q} c K − c Q ). This remark settles which of the query and key constants is larger, a question that Corollary 3 leaves open because it only bounds their difference. By (68) and (94) , c K − c Q = d 2 d v [ ( d 2 d v + 2 ) Δ 1 + ( d 2 + d v + 1 ) Δ 2 ] / d k 2 c_{K}-c_{Q}=d_{2}d_{v}[(d_{2}d_{v}+2)\Delta_{1}+(d_{2}+d_{v}+1)\Delta_{2}]/d_{k}^{2} c K − c Q = d 2 d v [( d 2 d v + 2 ) Δ 1 + ( d 2 + d v + 1 ) Δ 2 ] / d k 2 with Δ 1 : = E [ c 1 2 c 4 ] − E [ c 2 3 ] \Delta_{1}:=\mathbb{E}[c_{1}^{2}c_{4}]-\mathbb{E}[c_{2}^{3}] Δ 1 := E [ c 1 2 c 4 ] − E [ c 2 3 ] and Δ 2 : = E [ c 1 2 c 2 ] − 2 E [ c 2 c 4 ] \Delta_{2}:=\mathbb{E}[c_{1}^{2}c_{2}]-2\mathbb{E}[c_{2}c_{4}] Δ 2 := E [ c 1 2 c 2 ] − 2 E [ c 2 c 4 ] . When min ( T , d 0 ) = 1 \min(T,d_{0})=1 min ( T , d 0 ) = 1 , C C C has rank one, all four moments equal E [ ( χ max ( T , d 0 ) 2 ) 6 ] \mathbb{E}[(\chi_{\max(T,d_{0})}^{2})^{6}] E [( χ m a x ( T , d 0 ) 2 ) 6 ] , and c K = c Q c_{K}=c_{Q} c K = c Q . For every integer 2 ≤ T , d 0 ≤ 512 2\le T,d_{0}\le512 2 ≤ T , d 0 ≤ 512 the polynomials of Lemma 33 give Δ 1 , Δ 2 > 0 \Delta_{1},\Delta_{2}>0 Δ 1 , Δ 2 > 0 , hence c K > c Q c_{K}>c_{Q} c K > c Q for all d 2 , d v d_{2},d_{v} d 2 , d v in that range.
Asymptotic Approximation between Orthogonalization and Row Normalization
What does diagonal dominance say about the orthogonalization step itself? Suppose V t V t T V_tV_t^{\mathsf T} V t V t T is diagonal. Then V t V t T = diag ( V t V t T ) V_tV_t^{\mathsf T}=\operatorname{diag}(V_tV_t^{\mathsf T}) V t V t T = diag ( V t V t T ) , so ( V t V t T ) − 1 / 2 = diag ( V t V t T ) − 1 / 2 (V_tV_t^{\mathsf T})^{-1/2}=\operatorname{diag}(V_tV_t^{\mathsf T})^{-1/2} ( V t V t T ) − 1/2 = diag ( V t V t T ) − 1/2 , and since diag ( V t V t T ) i i = ∥ ( V t ) i , : ∥ 2 2 \operatorname{diag}(V_tV_t^{\mathsf T})_{ii}=\lVert(V_t)_{i,:}\rVert_2^2 diag ( V t V t T ) ii = ∥( V t ) i , : ∥ 2 2 ,
[ ( V t V t T ) − 1 / 2 V t ] i , : = ( V t ) i , : ∥ ( V t ) i , : ∥ 2 . (98) \left[(V_tV_t^{\mathsf T})^{-1/2}V_t\right]_{i,:}=\frac{(V_t)_{i,:}}{\lVert(V_t)_{i,:}\rVert_2}. \tag*{(98)} [ ( V t V t T ) − 1/2 V t ] i , : = ∥( V t ) i , : ∥ 2 ( V t ) i , : . (98)
That is, when V t V t T V_tV_t^{\mathsf T} V t V t T is diagonal the orthogonalization in (1) is just each row of the momentum divided by its own ℓ 2 \ell_2 ℓ 2 norm, which is exactly the row normalization used by several recent stateless optimizers (Ma et al., [44 ] ; Scetbon et al., [64 ] ; Glentis et al., [21 ] ; Wen et al., [78 ] ; Pethick et al., [57 ] ; Deng et al., [15 ] ); equivalently, its implicit preconditioner is diag ( P t ) 1 / 2 ⊗ I n \operatorname{diag}(P_t)^{1/2}\otimes I_n diag ( P t ) 1/2 ⊗ I n , which equals H M u o n H_{\mathrm{Muon}} H Muon when P t P_t P t is diagonal. Putting (98) together with Theorems 1 to 4: at a Gaussian initialization, as the hidden dimension grows, Muon’s Θ ( m n min ( m , n ) ) \Theta(mn\min(m,n)) Θ ( mn min ( m , n )) orthogonalization collapses to a Θ ( m n ) \Theta(mn) Θ ( mn ) row-wise ℓ 2 \ell_2 ℓ 2 normalization with the same preconditioner structure, and for the query, key and value weights of attention the collapse is exact in expectation. Deng et al. [15 ] check that this equivalence holds throughout the training of GPT-2 and LLaMA, and that the dominance ratio grows with model size, in line with the hidden-dimension dependence in Theorem 3 and Theorem 4. Carrying the analysis past initialization, and to deep nonlinear networks, is left for future work.
Simulation of Theoretical Results
We numerically verify the dimension-scaling predictions of Theorems 1 to 4 under the Gaussian initialization of § 3.2 . For each model, we perform two experiments. First, at a representative configuration, we estimate the expected gradient self outer-product and examine its diagonal and off-diagonal structure. Second, we vary the relevant model dimensions and measure how the diagonal magnitude, off-diagonal magnitude, and their ratio change.
To summarize the structure of a matrix P ∈ R m × m P\in\mathbb{R}^{m\times m} P ∈ R m × m , we define the mean diagonal magnitude, mean off-diagonal magnitude, and diagonal-dominance ratio as
D ( P ) = 1 m ∑ i = 1 m ∣ P i i ∣ , OD ( P ) = 1 m ( m − 1 ) ∑ i ≠ j ∣ P i j ∣ , R ( P ) = D ( P ) OD ( P ) . (99) D(P)=\frac{1}{m}\sum_{i=1}^{m}|P_{ii}|,\qquad\operatorname{OD}(P)=\frac{1}{m(m-1)}\sum_{i\ne j}|P_{ij}|,\qquad R(P)=\frac{D(P)}{\operatorname{OD}(P)}. \tag*{(99)} D ( P ) = m 1 i = 1 ∑ m ∣ P ii ∣ , OD ( P ) = m ( m − 1 ) 1 i = j ∑ ∣ P ij ∣ , R ( P ) = OD ( P ) D ( P ) . (99)
Given S S S independent initializations with gradients G ( 1 ) , … , G ( S ) G^{(1)},\ldots,G^{(S)} G ( 1 ) , … , G ( S ) , we estimate the expected gradient self outer-product by
E ^ [ G G T ] = 1 S ∑ s = 1 S G ( s ) ( G ( s ) ) T . (100) \widehat{\mathbb{E}}[GG^{\mathsf T}]=\frac{1}{S}\sum_{s=1}^{S}G^{(s)}(G^{(s)})^{\mathsf T}. \tag*{(100)} E [ G G T ] = S 1 s = 1 ∑ S G ( s ) ( G ( s ) ) T . (100)
We use D ^ \widehat{D} D , OD ^ \widehat{\operatorname{OD}} OD , and R ^ \widehat{R} R to denote the corresponding empirical estimates of D ( E [ G G T ] ) D(\mathbb{E}[GG^{\mathsf T}]) D ( E [ G G T ]) , OD ( E [ G G T ] ) \operatorname{OD}(\mathbb{E}[GG^{\mathsf T}]) OD ( E [ G G T ]) , and R ( E [ G G T ] ) R(\mathbb{E}[GG^{\mathsf T}]) R ( E [ G G T ]) , respectively.
Symmetric Matrix Factorization
Setup. We next verify the predictions of Theorem 1 and Corollary 1 for symmetric matrix factorization. As defined in (16) , the gradient is
G = ( W W T − M ⋆ ) W , G = (WW^{\mathsf{T}} - M^\star)W, G = ( W W T − M ⋆ ) W ,
where W ∈ R d × k W \in\mathbb{R}^{d \times k} W ∈ R d × k and M ⋆ ∈ R d × d M^\star\in\mathbb{R}^{d \times d} M ⋆ ∈ R d × d is symmetric positive semidefinite. Under Assumption 5, Theorem 1 gives
E [ G G T ] = α 3 I d − 2 α 2 M ⋆ + k ( M ⋆ ) 2 , (101) \mathbb{E}[GG^{\mathsf{T}}] = \alpha_3 I_d - 2\alpha_2 M^\star+ k(M^\star)^2, \tag*{(101)} E [ G G T ] = α 3 I d − 2 α 2 M ⋆ + k ( M ⋆ ) 2 , (101)
where α 2 = k ( k + d + 1 ) \alpha_2 = k(k + d + 1) α 2 = k ( k + d + 1 ) and α 3 = k 3 + 3 ( d + 1 ) k 2 + ( d 2 + 3 d + 4 ) k \alpha_3 = k^3 + 3(d + 1)k^2 + (d^2 + 3d + 4)k α 3 = k 3 + 3 ( d + 1 ) k 2 + ( d 2 + 3 d + 4 ) k . Corollary 1 gives the corresponding diagonal and off-diagonal entries,
E [ ( G G T ) i i ] = α 3 − 2 α 2 M i i ⋆ + k ( ( M ⋆ ) 2 ) i i , E [ ( G G T ) i j ] = − 2 α 2 M i j ⋆ + k ( ( M ⋆ ) 2 ) i j , i ≠ j . (102) \begin{aligned}
\mathbb{E}[(GG^{\mathsf{T}})_{ii}] &= \alpha_3 - 2\alpha_2 M^\star_{ii} + k((M^\star)^2)_{ii},\\
\mathbb{E}[(GG^{\mathsf{T}})_{ij}] &= -2\alpha_2 M^\star_{ij} + k((M^\star)^2)_{ij}, \qquad i \ne j. \tag*{(102)}
\end{aligned} E [( G G T ) ii ] E [( G G T ) ij ] = α 3 − 2 α 2 M ii ⋆ + k (( M ⋆ ) 2 ) ii , = − 2 α 2 M ij ⋆ + k (( M ⋆ ) 2 ) ij , i = j . (102)
For fixed d d d and a fixed target with nonzero off-diagonal structure, this gives D ( E [ G G T ] ) = Θ ( k 3 ) D(\mathbb{E}[GG^{\mathsf{T}}]) = \Theta(k^3) D ( E [ G G T ]) = Θ ( k 3 ) and O D ( E [ G G T ] ) = Θ ( k 2 ) OD(\mathbb{E}[GG^{\mathsf{T}}]) = \Theta(k^2) O D ( E [ G G T ]) = Θ ( k 2 ) , and hence R ( E [ G G T ] ) = Θ ( k ) R(\mathbb{E}[GG^{\mathsf{T}}]) = \Theta(k) R ( E [ G G T ]) = Θ ( k ) .
Empirical structure of the expected gradient Gram. We first examine the structure of the expected gradient Gram at a representative hidden dimension. We set d = 25 d = 25 d = 25 , k = 80 k = 80 k = 80 , and construct a fixed rank-5 target M ⋆ = Q Λ Q T M^\star= Q\Lambda Q^{\mathsf{T}} M ⋆ = Q Λ Q T , where Q ∈ R 25 × 5 Q \in\mathbb{R}^{25 \times5} Q ∈ R 25 × 5 has orthonormal columns obtained from the QR factorization of a Gaussian matrix and Λ = diag ( 3 , 2.5 , 2 , 1.5 , 1 ) \Lambda= \operatorname{diag}(3, 2.5, 2, 1.5, 1) Λ = diag ( 3 , 2.5 , 2 , 1.5 , 1 ) . We draw S = 10,000 S = 10{,}000 S = 10 , 000 independent initializations W ( 1 ) , … , W ( S ) W^{(1)}, \ldots, W^{(S)} W ( 1 ) , … , W ( S ) , with i.i.d. entries from N ( 0 , 1 ) \mathcal{N}(0,1) N ( 0 , 1 ) . For each s ∈ { 1 , … , S } s \in\{1, \ldots, S\} s ∈ { 1 , … , S } , let G ( s ) G^{(s)} G ( s ) denote the gradient corresponding to W ( s ) W^{(s)} W ( s ) . Using (100) , we estimate E [ G G T ] \mathbb{E}[GG^{\mathsf{T}}] E [ G G T ] from these 10,000 initializations.
Figure 7 shows a strongly diagonal expected gradient Gram, with substantially smaller off-diagonal entries. This is consistent with the diagonal-dominance behavior predicted by Theorem 1.
Figure 7. Empirical expected gradient self outer-product for symmetric matrix factorization with d = 25 d = 25 d = 25 , k = 80 k = 80 k = 80 , and a fixed rank-5 target, estimated over 10,000 independent initializations. Each heatmap is normalized by its empirical mean diagonal magnitude. The panel uses the logarithmic color scale as shown.
Hidden-dimension scaling. We next vary k k k over 15 values from 5 to 640, while keeping d = 25 d = 25 d = 25 and M ⋆ M^\star M ⋆ fixed, using S = 10,000 S = 10{,}000 S = 10 , 000 independent initializations at each hidden dimension.
To reduce Monte Carlo noise in the smaller off-diagonal entries, we subtract a term whose expected off-diagonal contribution is zero. By Lemma 4, setting M ⋆ = 0 M^\star= 0 M ⋆ = 0 gives G ( 0 ) G ( 0 ) T = S 3 G(0)G(0)^{\mathsf{T}} = S^3 G ( 0 ) G ( 0 ) T = S 3 , where S = W W T S = WW^{\mathsf{T}} S = W W T . Moreover, Lemma 7 gives E [ S 3 ] = α 3 I d \mathbb{E}[S^3] = \alpha_3 I_d E [ S 3 ] = α 3 I d , so this term has zero expected off-diagonal entries. We therefore define
Δ M F ( s ) = G ( s ) ( M ⋆ ) G ( s ) ( M ⋆ ) T − G ( s ) ( 0 ) G ( s ) ( 0 ) T . (103) \Delta_{\mathrm{MF}}^{(s)} = G^{(s)}(M^\star)G^{(s)}(M^\star)^{\mathsf{T}} - G^{(s)}(0)G^{(s)}(0)^{\mathsf{T}}. \tag*{(103)} Δ MF ( s ) = G ( s ) ( M ⋆ ) G ( s ) ( M ⋆ ) T − G ( s ) ( 0 ) G ( s ) ( 0 ) T . (103)
For i ≠ j i \ne j i = j ,
E [ ( Δ M F ( s ) ) i j ] = E [ ( G G T ) i j ] . \mathbb{E}\left[\left(\Delta_{\mathrm{MF}}^{(s)}\right)_{ij}\right] = \mathbb{E}\left[(GG^{\mathsf{T}})_{ij}\right]. E [ ( Δ MF ( s ) ) ij ] = E [ ( G G T ) ij ] .
Thus, the paired difference removes the S 3 S^3 S 3 contribution while preserving the expected off-diagonal entries.
For each hidden dimension, we compute D ^ \widehat{D} D from the full empirical average (100) . For O D ^ \widehat{\mathrm{OD}} OD , we first average Δ M F ( s ) \Delta_{\mathrm{MF}}^{(s)} Δ MF ( s ) over the same S S S initializations and then apply O D ( ⋅ ) \mathrm{OD}(\cdot) OD ( ⋅ ) . We then compute R ^ = D ^ / O D ^ \widehat{R} = \widehat{D}/\widehat{\mathrm{OD}} R = D / OD .
Figure 8 shows that the diagonal magnitude grows faster with k k k than the off-diagonal magnitude, while R ^ \widehat{R} R grows approximately linearly with k k k . This is consistent with the theoretical predictions D = Θ ( k 3 ) D = \Theta(k^3) D = Θ ( k 3 ) , O D = Θ ( k 2 ) \mathrm{OD} = \Theta(k^2) OD = Θ ( k 2 ) , and R = Θ ( k ) R = \Theta(k) R = Θ ( k ) .
Figure 8. Hidden-dimension scaling for symmetric matrix factorization with d = 25 d = 25 d = 25 and a fixed rank-5 target. From left to right, the panels show D ^ \widehat{D} D , O D ^ \widehat{\mathrm{OD}} OD , and R ^ = D ^ / O D ^ \widehat{R} = \widehat{D}/\widehat{\mathrm{OD}} R = D / OD as the factorization hidden dimension k k k is varied. Empirical estimates use 10,000 independent initializations at each hidden dimension; O D ^ \widehat{\mathrm{OD}} OD uses the paired estimator in (103) .
Deep Linear Network
Setup. We next verify the predictions of Theorem 2 and Corollary 2 for the deep linear network. Under the Gaussian initialization of Assumption 2, Theorem 2 gives
E [ G i G i T ] = V i s i I d i + T i , T i = { ( ∏ j = 1 L − 1 d j ) Φ Φ T , i = L , ( ∏ 1 ≤ j ≤ L − 1 j ≠ i d j ) ∥ Φ ∥ F 2 I d i , i < L . (104) \mathbb{E}[G_iG_i^{\mathsf{T}}] = V_i s_i I_{d_i} + T_i,\qquad T_i =
\begin{cases}
\left(\prod_{j=1}^{L-1} d_j\right)\Phi\Phi^{\mathsf{T}}, & i=L,\\
\left(\prod_{\substack{1\leq j\leq L-1\\j\ne i}} d_j\right)\lVert\Phi\rVert_{\mathrm{F}}^2 I_{d_i}, & i<L.
\end{cases} \tag*{(104)} E [ G i G i T ] = V i s i I d i + T i , T i = ⎩ ⎨ ⎧ ( ∏ j = 1 L − 1 d j ) Φ Φ T , ( ∏ 1 ≤ j ≤ L − 1 j = i d j ) ∥ Φ ∥ F 2 I d i , i = L , i < L . (104)
Corollary 2 therefore gives zero expected off-diagonal entries for every hidden layer i < L i<L i < L , while for the output layer
E [ ( G L G L T ) a b ] = ( ∏ j = 1 L − 1 d j ) ( Φ Φ T ) a b , a ≠ b . (105) \mathbb{E}[(G_LG_L^{\mathsf{T}})_{ab}] = \left(\prod_{j=1}^{L-1} d_j\right)(\Phi\Phi^{\mathsf{T}})_{ab},\qquad a\ne b. \tag*{(105)} E [( G L G L T ) ab ] = ( j = 1 ∏ L − 1 d j ) ( Φ Φ T ) ab , a = b . (105)
Empirical structure of the expected gradient Gram. We first examine the empirical expected gradient Grams at a representative configuration. We use depth L = 4 L=4 L = 4 with dimensions [ d 0 , d 1 , d 2 , d 3 , d 4 ] = [ 512 , 80 , 80 , 80 , 512 ] [d_0,d_1,d_2,d_3,d_4] = [512,80,80,80,512] [ d 0 , d 1 , d 2 , d 3 , d 4 ] = [ 512 , 80 , 80 , 80 , 512 ] and a fixed target Φ ∈ R 512 × 512 \Phi\in\mathbb{R}^{512\times512} Φ ∈ R 512 × 512 with entries sampled from N ( 0 , 1 ) \mathcal{N}(0,1) N ( 0 , 1 ) . We use S = 10,000 S=10{,}000 S = 10 , 000 independent Gaussian weight initializations and estimate the expected Gram for each G 1 , … , G 4 G_1,\ldots,G_4 G 1 , … , G 4 using (100) .
Figure 9 shows strongly diagonal expected Grams for G 1 G_1 G 1 , G 2 G_2 G 2 , G 3 G_3 G 3 , consistent with Corollary 2, which gives zero expected off-diagonal entries for the hidden layers. The output-layer Gram retains visible off-diagonal structure, consistent with the Φ Φ T \Phi\Phi^{\mathsf{T}} Φ Φ T term in Theorem 2.
Figure 9. Empirical expected gradient self outer-products for the depth-4 linear network with baseline dimensions [ 512 , 80 , 80 , 80 , 512 ] [512,80,80,80,512] [ 512 , 80 , 80 , 80 , 512 ] , estimated from 10,000 independent initializations. The four panels show E [ G i G i T ] ^ \widehat{\mathbb{E}[G_iG_i^{\mathsf{T}}]} E [ G i G i T ] for i = 1 , … , 4 i=1,\ldots,4 i = 1 , … , 4 . Each heatmap is normalized by its empirical mean diagonal magnitude. A common logarithmic color scale is used across all panels.
Dimension scaling. We next vary each hidden dimension d 1 d_1 d 1 , d 2 d_2 d 2 , d 3 d_3 d 3 individually while holding the remaining dimensions fixed at the baseline values above. Each hidden dimension is varied over a 12-point grid { 5 , 10 , 20 , 40 , 60 , 80 , 120 , 160 , 200 , 240 , 280 , 320 } \{5,10,20,40,60,80,120,160,200,240,280,320\} { 5 , 10 , 20 , 40 , 60 , 80 , 120 , 160 , 200 , 240 , 280 , 320 } , and we evaluate G 1 , … , G 4 G_1,\ldots,G_4 G 1 , … , G 4 . A fixed 512 × 512 512 \times512 512 × 512 target is sampled with entries from N ( 0 , 1 ) \mathcal{N}(0,1) N ( 0 , 1 ) . We use S = 10,000 S=10{,}000 S = 10 , 000 independent initializations for every configuration. Because the expected off-diagonal entries are much smaller than the diagonal entries, direct Monte Carlo estimation of OD \operatorname{OD} OD has high variance. We therefore use a paired construction to reduce this Monte Carlo variance while preserving the expected off-diagonal entries. From (29) and (30) , for fixed weights, the gradient is affine in Φ \Phi Φ . For each s ∈ { 1 , … , S } s \in\{1,\ldots,S\} s ∈ { 1 , … , S } , let G i ( s ) ( Φ ) G_i^{(s)}(\Phi) G i ( s ) ( Φ ) denote the layer-i i i gradient for the s s s -th weight initialization and target Φ \Phi Φ . Using the same weights, we evaluate the gradient at Φ \Phi Φ , − Φ -\Phi − Φ , and 0 0 0 , and define
Δ i ( s ) = 1 2 [ G i ( s ) ( Φ ) G i ( s ) ( Φ ) T + G i ( s ) ( − Φ ) G i ( s ) ( − Φ ) T ] − G i ( s ) ( 0 ) G i ( s ) ( 0 ) T . (106) \Delta_i^{(s)}=\frac{1}{2}\left[G_i^{(s)}(\Phi)G_i^{(s)}(\Phi)^{\mathsf{T}}+G_i^{(s)}(-\Phi)G_i^{(s)}(-\Phi)^{\mathsf{T}}\right]-G_i^{(s)}(0)G_i^{(s)}(0)^{\mathsf{T}}. \tag*{(106)} Δ i ( s ) = 2 1 [ G i ( s ) ( Φ ) G i ( s ) ( Φ ) T + G i ( s ) ( − Φ ) G i ( s ) ( − Φ ) T ] − G i ( s ) ( 0 ) G i ( s ) ( 0 ) T . (106)
The symmetric combination cancels the target-independent and target-linear terms. The remaining term is
Δ i ( s ) = ( A i ( s ) ) T Φ ( B i ( s ) ) T B i ( s ) Φ T A i ( s ) . (107) \Delta_i^{(s)}=(A_i^{(s)})^{\mathsf{T}}\Phi(B_i^{(s)})^{\mathsf{T}}B_i^{(s)}\Phi^{\mathsf{T}}A_i^{(s)}. \tag*{(107)} Δ i ( s ) = ( A i ( s ) ) T Φ ( B i ( s ) ) T B i ( s ) Φ T A i ( s ) . (107)
By Lemma 15,
E [ Δ i ( s ) ] = T i . (108) \mathbb{E}[\Delta_i^{(s)}]=T_i. \tag*{(108)} E [ Δ i ( s ) ] = T i . (108)
Since the remaining term V s i I d i V_{s_i}I_{d_i} V s i I d i in Theorem 2 is diagonal, (106) has exactly the same expected off-diagonal entries as the full gradient Gram.
For each configuration, we compute D ^ \widehat{D} D from the full empirical expected Gram. For OD ^ \widehat{\operatorname{OD}} OD , we first average Δ i ( s ) \Delta_i^{(s)} Δ i ( s ) over the same S S S initializations and then apply OD ( ⋅ ) \operatorname{OD}(\cdot) OD ( ⋅ ) . We then compute R ^ = D ^ / OD ^ \widehat{R}=\widehat{D}/\widehat{\operatorname{OD}} R = D / OD .
Figure 10 shows the layer-dependent scaling predicted by Theorem 2 and Corollary 2. When d j d_j d j is varied, D ^ \widehat{D} D grows more slowly for the corresponding gradient G j G_j G j than for the other hidden-layer gradients. This is consistent with D = Θ ( d j ) D=\Theta(d_j) D = Θ ( d j ) for G j G_j G j , compared with D = Θ ( d j 2 ) D=\Theta(d_j^2) D = Θ ( d j 2 ) for G i G_i G i , i ≠ j i\ne j i = j , when the remaining dimensions are fixed.
Figure 10. Empirical dimension scaling for the depth-4 linear network. Columns vary d 1 d_1 d 1 , d 2 d_2 d 2 , and d 3 d_3 d 3 individually while holding the remaining dimensions fixed at their baseline values. Rows show D ^ \widehat{D} D , O D ^ \widehat{OD} O D , and R ^ = D ^ / O D ^ \widehat{R}=\widehat{D}/\widehat{OD} R = D / O D for G 1 , … , G 4 G_1,\ldots,G_4 G 1 , … , G 4 , each normalized by its value at the baseline configuration [ d 0 , d 1 , d 2 , d 3 , d 4 ] = [ 512 , 80 , 80 , 80 , 512 ] [d_0,d_1,d_2,d_3,d_4]=[512,80,80,80,512] [ d 0 , d 1 , d 2 , d 3 , d 4 ] = [ 512 , 80 , 80 , 80 , 512 ] to facilitate comparison across layers with different absolute scales. Empirical estimates use 10,000 independent initializations per configuration. The off-diagonal statistic is estimated using (106) .
Corollary 2 gives zero expected off-diagonal entries for i < L i<L i < L , and the remaining nonzero values are finite-sample Monte Carlo variances. For the output layer, the normalized D ^ \widehat{D} D curves show approximately quadratic growth, while the normalized OD ^ \widehat{\operatorname{OD}} OD curves grow approximately linearly when any one of d 1 , d 2 , d 3 d_1,d_2,d_3 d 1 , d 2 , d 3 is varied. This is consistent with D = Θ ( d j 2 ) D=\Theta(d_j^2) D = Θ ( d j 2 ) , OD = Θ ( d j ) \operatorname{OD}=\Theta(d_j) OD = Θ ( d j ) , and hence R = Θ ( d j ) R=\Theta(d_j) R = Θ ( d j ) .
Two-Layer ReLU Network
Setup. We next verify the predictions of Theorem 3 for the two-layer ReLU network f ( x ) = W 2 σ ( W 1 x ) f(x)=W_2\sigma(W_1x) f ( x ) = W 2 σ ( W 1 x ) , with hidden dimension d 1 d_1 d 1 . Unlike the hidden layers of the deep linear network, the expected gradient self outer-products are not exactly diagonal.
Under the Gaussian initialization of Assumption 3, Theorem 3 gives, for G 1 G_1 G 1 ,
E [ ( G 1 G 1 T ) a a ] = d 0 d 2 2 4 + d 0 d 2 ( ( d 1 − 1 ) κ 1 + 1 2 ) + ∥ Φ ∥ F 2 4 . (109) \mathbb{E}\left[(G_1G_1^{\mathsf{T}})_{aa}\right]=\frac{d_0d_2^2}{4}+d_0d_2\left((d_1-1)\kappa_1+\frac{1}{2}\right)+\frac{\lVert\Phi\rVert_{\mathrm{F}}^2}{4}. \tag*{(109)} E [ ( G 1 G 1 T ) aa ] = 4 d 0 d 2 2 + d 0 d 2 ( ( d 1 − 1 ) κ 1 + 2 1 ) + 4 ∥ Φ ∥ F 2 . (109)
while for a ≠ c a \ne c a = c ,
E [ ( G 1 G 1 T ) a c ] = d 0 d 2 κ 2 + O ( d 2 d 0 ) , (110) \mathbb{E}\left[(G_1G_1^{\mathsf{T}})_{ac}\right]=d_0d_2\kappa_2+O(d_2\sqrt{d_0}), \tag*{(110)} E [ ( G 1 G 1 T ) a c ] = d 0 d 2 κ 2 + O ( d 2 d 0 ) , (110)
where κ 1 \kappa_1 κ 1 and κ 2 \kappa_2 κ 2 are defined in Theorem 3 and depend only on d 0 d_0 d 0 . Holding d 0 d_0 d 0 and d 2 d_2 d 2 fixed while varying d 1 d_1 d 1 therefore gives
D ( E [ G 1 G 1 T ] ) = Θ ( d 1 ) , O D ( E [ G 1 G 1 T ] ) = Θ ( 1 ) , R ( E [ G 1 G 1 T ] ) = Θ ( d 1 ) . (111) D\left(\mathbb{E}\left[G_1G_1^{\mathsf{T}}\right]\right)=\Theta(d_1),\qquad
OD\left(\mathbb{E}\left[G_1G_1^{\mathsf{T}}\right]\right)=\Theta(1),\qquad
R\left(\mathbb{E}\left[G_1G_1^{\mathsf{T}}\right]\right)=\Theta(d_1). \tag*{(111)} D ( E [ G 1 G 1 T ] ) = Θ ( d 1 ) , O D ( E [ G 1 G 1 T ] ) = Θ ( 1 ) , R ( E [ G 1 G 1 T ] ) = Θ ( d 1 ) . (111)
For G 2 G_2 G 2 , Theorem 3 gives
E [ G 2 G 2 T ] = α I d 2 + d 1 4 Φ Φ T , α = Θ ( d 1 2 ) . (112) \mathbb{E}\left[G_2G_2^{\mathsf{T}}\right]=\alpha I_{d_2}+\frac{d_1}{4}\Phi\Phi^{\mathsf{T}},\qquad\alpha=\Theta(d_1^2). \tag*{(112)} E [ G 2 G 2 T ] = α I d 2 + 4 d 1 Φ Φ T , α = Θ ( d 1 2 ) . (112)
Thus, for fixed d 0 d_0 d 0 , d 2 d_2 d 2 , and a fixed target with nonzero off-diagonal structure,
D ( E [ G 2 G 2 T ] ) = Θ ( d 1 2 ) , O D ( E [ G 2 G 2 T ] ) = Θ ( d 1 ) , R ( E [ G 2 G 2 T ] ) = Θ ( d 1 ) . (113) D\left(\mathbb{E}\left[G_2G_2^{\mathsf{T}}\right]\right)=\Theta(d_1^2),\qquad
OD\left(\mathbb{E}\left[G_2G_2^{\mathsf{T}}\right]\right)=\Theta(d_1),\qquad
R\left(\mathbb{E}\left[G_2G_2^{\mathsf{T}}\right]\right)=\Theta(d_1). \tag*{(113)} D ( E [ G 2 G 2 T ] ) = Θ ( d 1 2 ) , O D ( E [ G 2 G 2 T ] ) = Θ ( d 1 ) , R ( E [ G 2 G 2 T ] ) = Θ ( d 1 ) . (113)
Empirical structure of the expected gradient Gram. We first examine the empirical expected gradient Grams at a representative configuration. We use [ d 0 , d 1 , d 2 ] = [ 512 , 80 , 12 ] [d_0,d_1,d_2]=[512,80,12] [ d 0 , d 1 , d 2 ] = [ 512 , 80 , 12 ] and a fixed target Φ ∈ R 12 × 512 \Phi\in\mathbb{R}^{12 \times512} Φ ∈ R 12 × 512 whose entries are sampled from N ( 0 , 9 ) \mathcal{N}(0,9) N ( 0 , 9 ) . For each of S = 10,000 S=10{,}000 S = 10 , 000 independent Gaussian weight initializations, we estimate the population gradients using 128,000 independent inputs sampled from N ( 0 , I d 0 ) \mathcal{N}(0,I_{d_0}) N ( 0 , I d 0 ) , and estimate the expected gradient Grams using (100) .
Figure 11 shows diagonally dominant empirical expected Grams for both G 1 G_1 G 1 and G 2 G_2 G 2 , with smaller but nonzero off-diagonal entries. This is consistent with Theorem 3, which predicts asymptotic diagonal dominance.
Figure 11. Empirical expected gradient self outer-products for the two-layer ReLU network with dimensions [ d 0 , d 1 , d 2 ] = [ 512 , 80 , 12 ] [d_0,d_1,d_2]=[512,80,12] [ d 0 , d 1 , d 2 ] = [ 512 , 80 , 12 ] , estimated from 10,000 independent weight initializations. The panels show E ^ [ G 1 G 1 T ] \widehat{\mathbb{E}}[G_1G_1^{\mathsf{T}}] E [ G 1 G 1 T ] and E ^ [ G 2 G 2 T ] \widehat{\mathbb{E}}[G_2G_2^{\mathsf{T}}] E [ G 2 G 2 T ] . Each heatmap is normalized by its empirical mean diagonal magnitude. A common logarithmic color scale is used across all panels.
Dimension scaling. We set d 0 = 512 d_0=512 d 0 = 512 and d 2 = 12 d_2=12 d 2 = 12 , and vary d 1 ∈ { 5 , 10 , 20 , 40 , 60 , 80 , 120 , 160 , 200 , 240 , 280 , 320 , 400 , 480 , 640 } d_1 \in\{5,10,20,40,60,80,120,160,200,240,280,320,400,480,640\} d 1 ∈ { 5 , 10 , 20 , 40 , 60 , 80 , 120 , 160 , 200 , 240 , 280 , 320 , 400 , 480 , 640 } . The same fixed target is used throughout the sweep, and each population gradient is estimated from 128,000 Gaussian inputs for each of S = 10,000 S=10{,}000 S = 10 , 000 independent weight initializations.
Because the expected off-diagonal entries are much smaller than the diagonal entries, direct Monte Carlo estimation of OD is noisy. We therefore use the following expressions to reduce this Monte Carlo variance while preserving the expected off-diagonal entries.
For G 1 G_1 G 1 , (58) gives
G 1 = G 1 ( 1 ) − G 1 ( 2 ) . (114) G_1=G_1^{(1)}-G_1^{(2)}. \tag*{(114)} G 1 = G 1 ( 1 ) − G 1 ( 2 ) . (114)
By (59) , the cross terms vanish in expectation, and Lemma 24 gives
G 1 ( 2 ) = 1 2 W 2 T Φ , E [ G 1 ( 2 ) ( G 1 ( 2 ) ) T ] = ∥ Φ ∥ F 2 4 I d 1 . (115) G_1^{(2)}=\frac{1}{2}W_2^{\mathsf{T}}\Phi,\qquad\mathbb{E}\left[G_1^{(2)}(G_1^{(2)})^{\mathsf{T}}\right]=\frac{\lVert\Phi\rVert_{\mathrm{F}}^2}{4}I_{d_1}. \tag*{(115)} G 1 ( 2 ) = 2 1 W 2 T Φ , E [ G 1 ( 2 ) ( G 1 ( 2 ) ) T ] = 4 ∥ Φ ∥ F 2 I d 1 . (115)
Hence,
E [ G 1 G 1 T ] = E [ G 1 ( 1 ) ( G 1 ( 1 ) ) T ] + ∥ Φ ∥ F 2 4 I d 1 . (116) \mathbb{E}[G_1G_1^{\mathsf{T}}]=\mathbb{E}\left[G_1^{(1)}(G_1^{(1)})^{\mathsf{T}}\right]+\frac{\lVert\Phi\rVert_{\mathrm{F}}^2}{4}I_{d_1}. \tag*{(116)} E [ G 1 G 1 T ] = E [ G 1 ( 1 ) ( G 1 ( 1 ) ) T ] + 4 ∥ Φ ∥ F 2 I d 1 . (116)
The second term is diagonal, so all expected off-diagonal entries come from G 1 ( 1 ) G_1^{(1)} G 1 ( 1 ) . Moreover, setting Φ = 0 \Phi=0 Φ = 0 in (58) gives G 1 ( 0 ) = G 1 ( 1 ) G_1(0)=G_1^{(1)} G 1 ( 0 ) = G 1 ( 1 ) . For each s ∈ { 1 , … , S } s\in\{1,\ldots,S\} s ∈ { 1 , … , S } , we therefore define
Δ 1 ( s ) = G 1 ( s ) ( 0 ) G 1 ( s ) ( 0 ) T . (117) \Delta_1^{(s)}=G_1^{(s)}(0)G_1^{(s)}(0)^{\mathsf{T}}. \tag*{(117)} Δ 1 ( s ) = G 1 ( s ) ( 0 ) G 1 ( s ) ( 0 ) T . (117)
For a ≠ c a\ne c a = c ,
E [ ( Δ 1 ( s ) ) a c ] = E [ ( G 1 G 1 T ) a c ] . (118) \mathbb{E}\left[(\Delta_1^{(s)})_{ac}\right]=\mathbb{E}\left[(G_1G_1^{\mathsf{T}})_{ac}\right]. \tag*{(118)} E [ ( Δ 1 ( s ) ) a c ] = E [ ( G 1 G 1 T ) a c ] . (118)
Thus, (117) preserves the expected off-diagonal entries while removing the diagonal target contribution.
For G 2 G_2 G 2 , Lemma 22 gives G 2 = W 2 H − Φ C T G_2=W_2H-\Phi C^{\mathsf{T}} G 2 = W 2 H − Φ C T . For each s ∈ { 1 , … , S } s\in\{1,\ldots,S\} s ∈ { 1 , … , S } , let G 2 ( s ) ( Φ ) G_2^{(s)}(\Phi) G 2 ( s ) ( Φ ) denote the output-layer gradient for the s s s -th weight initialization and target Φ \Phi Φ . Using the same weights and input samples, we evaluate the gradient at Φ \Phi Φ , − Φ -\Phi − Φ , and 0 0 0 , and define
Δ 2 ( s ) = 1 2 [ G 2 ( s ) ( Φ ) G 2 ( s ) ( Φ ) T + G 2 ( s ) ( − Φ ) G 2 ( s ) ( − Φ ) T ] − G 2 ( s ) ( 0 ) G 2 ( s ) ( 0 ) T . (119) \Delta_2^{(s)}=\frac{1}{2}\left[G_2^{(s)}(\Phi)G_2^{(s)}(\Phi)^{\mathsf{T}}+G_2^{(s)}(-\Phi)G_2^{(s)}(-\Phi)^{\mathsf{T}}\right]-G_2^{(s)}(0)G_2^{(s)}(0)^{\mathsf{T}}. \tag*{(119)} Δ 2 ( s ) = 2 1 [ G 2 ( s ) ( Φ ) G 2 ( s ) ( Φ ) T + G 2 ( s ) ( − Φ ) G 2 ( s ) ( − Φ ) T ] − G 2 ( s ) ( 0 ) G 2 ( s ) ( 0 ) T . (119)
Substituting Lemma 22 into (119) cancels the terms independent of Φ \Phi Φ and linear in Φ \Phi Φ , leaving
Δ 2 ( s ) = Φ ( C ( s ) ) T C ( s ) Φ T . (120) \Delta_2^{(s)} = \Phi(C^{(s)})^{\mathsf{T}} C^{(s)}\Phi^{\mathsf{T}}. \tag*{(120)} Δ 2 ( s ) = Φ ( C ( s ) ) T C ( s ) Φ T . (120)
By Lemma 23 , C = 1 2 W 1 C = \frac{1}{2}W_1 C = 2 1 W 1 , and Lemma 16 gives E [ W 1 T W 1 ] = d 1 I d 0 \mathbb{E}[W_1^{\mathsf{T}}W_1] = d_1 I_{d_0} E [ W 1 T W 1 ] = d 1 I d 0 . Hence
E [ Δ 2 ( s ) ] = d 1 4 Φ Φ T . (121) \mathbb{E}[\Delta_2^{(s)}] = \frac{d_1}{4}\Phi\Phi^{\mathsf{T}}. \tag*{(121)} E [ Δ 2 ( s ) ] = 4 d 1 Φ Φ T . (121)
Since the remaining term α I d 2 \alpha I_{d_2} α I d 2 in (112) is diagonal, (119) has exactly the same expected off-diagonal entries as the full G 2 G 2 T G_2G_2^{\mathsf{T}} G 2 G 2 T .
For both layers, we compute D ^ \widehat{D} D from the full empirical expected Gram. For G 1 G_1 G 1 , we compute O D ‾ \overline{\mathrm{OD}} OD from the zero-target empirical Gram. For G 2 G_2 G 2 , we first average Δ 2 ( s ) \Delta_2^{(s)} Δ 2 ( s ) over the same S S S initializations and then apply O D ( ⋅ ) \mathrm{OD}(\cdot) OD ( ⋅ ) . We then compute R ^ = D ^ / O D ‾ \widehat{R} = \widehat{D}/\overline{\mathrm{OD}} R = D / OD .
Figure 12 shows the different hidden-dimension scalings predicted by Theorem 3. For G 1 G_1 G 1 , D ^ \widehat{D} D grows approximately linearly with d 1 d_1 d 1 , while O D ‾ \overline{\mathrm{OD}} OD remains approximately constant, consistent with D = Θ ( d 1 ) D = \Theta(d_1) D = Θ ( d 1 ) and O D = Θ ( 1 ) \mathrm{OD} = \Theta(1) OD = Θ ( 1 ) in (111) . For G 2 G_2 G 2 , D ^ \widehat{D} D grows approximately quadratically with d 1 d_1 d 1 , while O D ‾ \overline{\mathrm{OD}} OD grows approximately linearly, consistent with (113) . Accordingly, R ^ \widehat{R} R increases with d 1 d_1 d 1 for both layers, consistent with the predicted R = Θ ( d 1 ) R = \Theta(d_1) R = Θ ( d 1 ) .
Figure 12. Hidden-dimension scaling for the two-layer ReLU network with d 0 = 512 d_0 = 512 d 0 = 512 and d 2 = 12 d_2 = 12 d 2 = 12 . The top and bottom rows correspond to G 1 G_1 G 1 and G 2 G_2 G 2 , respectively, and show D ^ \widehat{D} D , O D ‾ \overline{\mathrm{OD}} OD , and R ^ = D ^ / O D ‾ \widehat{R} = \widehat{D}/\overline{\mathrm{OD}} R = D / OD as d 1 d_1 d 1 is varied. For G 1 G_1 G 1 , O D ‾ \overline{\mathrm{OD}} OD is estimated from the zero-target Gram; for G 2 G_2 G 2 , it is estimated using (119) .
Single-Layer Attention
Setup. Finally, we verify the predictions of Theorem 4 and Corollary 3 for single-layer attention defined in (8) . We consider the gradients G Q G_Q G Q , G K G_K G K , G V G_V G V , G O G_O G O of the query, key, value, and output projections, respectively.
Figure 13. Empirical expected gradient self outer-products for single-layer attention with T = 16 T = 16 T = 16 , d 0 = 32 d_0 = 32 d 0 = 32 , d k = d v = 20 d_k = d_v = 20 d k = d v = 20 , and d 2 = 12 d_2 = 12 d 2 = 12 , estimated from 10,000 independent initializations. The four panels show E ^ [ G p G p T ] \hat{\mathbb{E}}[G_pG_p^{\mathsf{T}}] E ^ [ G p G p T ] for p ∈ { Q , K , V , O } p \in\{Q,K,V,O\} p ∈ { Q , K , V , O } . Each heatmap is normalized by its empirical mean diagonal magnitude. A common logarithmic color scale is used across all panels.
Under the Gaussian initialization of Assumption 4, Theorem 4 gives
E [ G Q G Q T ] = c Q I d k , E [ G K G K T ] = c K I d k , E [ G V G V T ] = c V I d v , E [ G O G O T ] = α ˉ O I d 2 + d v γ Φ Φ T , (122) \begin{aligned}
\mathbb{E}[G_QG_Q^{\mathsf{T}}] &= c_Q I_{d_k},\\
\mathbb{E}[G_KG_K^{\mathsf{T}}] &= c_K I_{d_k},\\
\mathbb{E}[G_VG_V^{\mathsf{T}}] &= c_V I_{d_v},\\
\mathbb{E}[G_OG_O^{\mathsf{T}}] &= \bar{\alpha}_O I_{d_2} + d_v\gamma\Phi\Phi^{\mathsf{T}}, \tag*{(122)}
\end{aligned} E [ G Q G Q T ] E [ G K G K T ] E [ G V G V T ] E [ G O G O T ] = c Q I d k , = c K I d k , = c V I d v , = α ˉ O I d 2 + d v γ Φ Φ T , (122)
where c Q c_Q c Q , c K c_K c K , c V c_V c V , α ˉ O \bar{\alpha}_O α ˉ O , and γ \gamma γ are given explicitly in Theorem 4 and Appendix § 13 .§ 3 . Thus, the expected gradient self outer-products for the query, key, and value projections are exactly diagonal,
OD ( E [ G p G p T ] ) = 0 , p ∈ { Q , K , V } . (123) \operatorname{OD}\left(\mathbb{E}[G_pG_p^{\mathsf{T}}]\right) = 0,\qquad p \in\{Q,K,V\}. \tag*{(123)} OD ( E [ G p G p T ] ) = 0 , p ∈ { Q , K , V } . (123)
Combining Theorem 4 and Corollary 3, for a fixed target with OD ( Φ Φ T ) > 0 \operatorname{OD}(\Phi\Phi^{\mathsf{T}}) > 0 OD ( Φ Φ T ) > 0 , we obtain
R ( E [ G O G O T ] ) = Θ ( d v d 0 3 ) . (124) R\left(\mathbb{E}[G_OG_O^{\mathsf{T}}]\right) = \Theta(d_vd_0^3). \tag*{(124)} R ( E [ G O G O T ] ) = Θ ( d v d 0 3 ) . (124)
Hence increasing the hidden dimension d v d_v d v or the token dimension d 0 d_0 d 0 makes the expected output-projection Gram increasingly diagonally dominant, while the hidden dimension d k d_k d k enters the ratio only through a bounded factor.
Empirical structure of the expected gradient Gram. We first examine the empirical expected gradient Grams at a representative configuration. We use linear attention with T = 16 T = 16 T = 16 , d 0 = 32 d_0 = 32 d 0 = 32 , d k = d v = 20 d_k = d_v = 20 d k = d v = 20 , and d 2 = 12 d_2 = 12 d 2 = 12 . The entries of the fixed target Φ ∈ R d 2 × d 0 \Phi\in\mathbb{R}^{d_2 \times d_0} Φ ∈ R d 2 × d 0 are sampled from N ( 0 , 1 ) \mathcal{N}(0,1) N ( 0 , 1 ) . We use S = 10,000 S = 10{,}000 S = 10 , 000 independent initializations. For each initialization, the rows of X ∈ R T × d 0 X \in\mathbb{R}^{T \times d_0} X ∈ R T × d 0 are sampled independently from N ( 0 , I d 0 ) \mathcal{N}(0,I_{d_0}) N ( 0 , I d 0 ) , independently of the Gaussian weights, as specified in Assumption 4. We estimate the expected gradient Gram for each projection using (100) .
Figure 13 shows strongly diagonal empirical expected Grams for the query, key, and value projections, consistent with the exact isotropy in Theorem 4. The output-projection Gram is also diagonally dominant but retains nonzero off-diagonal structure, consistent with the d v γ Φ Φ T d_v\gamma\Phi\Phi^{\mathsf{T}} d v γ Φ Φ T term in (122) .
Dimension scaling. We examine four dimension sweeps: d k = d v d_k = d_v d k = d v jointly, d k d_k d k with d v = 80 d_v = 80 d v = 80 , d v d_v d v with d k = 80 d_k = 80 d k = 80 , and d 0 = d 2 d_0 = d_2 d 0 = d 2 jointly. For the first three sweeps, we set T = 32 T = 32 T = 32 , d 0 = 512 d_0 = 512 d 0 = 512 , and d 2 = 12 d_2 = 12 d 2 = 12 , and use the 12-point grid { 5 , 10 , 20 , 40 , 60 , 80 , 120 , 160 , 200 , 240 , 280 , 320 } \{5, 10, 20, 40, 60, 80, 120, 160, 200, 240, 280, 320\} { 5 , 10 , 20 , 40 , 60 , 80 , 120 , 160 , 200 , 240 , 280 , 320 } . For the joint d 0 = d 2 d_0 = d_2 d 0 = d 2 sweep, we set T = 16 T = 16 T = 16 and d k = d v = 20 d_k = d_v = 20 d k = d v = 20 , and use the 13-point grid { 32 , 64 , 128 , 192 , 256 , 320 , 384 , 448 , 512 , 640 , 768 , 896 , 1024 } \{32, 64, 128, 192, 256, 320, 384, 448, 512, 640, 768, 896, 1024\} { 32 , 64 , 128 , 192 , 256 , 320 , 384 , 448 , 512 , 640 , 768 , 896 , 1024 } . Further, for the joint d 0 = d 2 d_0 = d_2 d 0 = d 2 sweep, we rescale the target Φ \Phi Φ at each dimension to have the same Frobenius norm as the $32 \times 32$ reference target. This controls for the explicit target dependence in Theorem 4: the diagonal constants c Q c_Q c Q , c K c_K c K , c V c_V c V contain terms proportional to ∥ Φ ∥ F 2 \lVert\Phi\rVert_{\mathrm{F}}^2 ∥ Φ ∥ F 2 in (68) , while the output Gram contains the term d v γ Φ Φ T d_v\gamma\Phi\Phi^{\mathsf{T}} d v γ Φ Φ T in (69) . Holding ∥ Φ ∥ F \lVert\Phi\rVert_{\mathrm{F}} ∥ Φ ∥ F fixed therefore isolates the effect of increasing d 0 = d 2 d_0 = d_2 d 0 = d 2 from the increase in target magnitude from the larger target matrix. For every configuration, we use $S = 10{,}000 independent initializations. For each initialization, the rows of X X X are sampled independently from N ( 0 , I d 0 ) \mathcal{N}(0, I_{d_0}) N ( 0 , I d 0 ) , independently of the weights.
Since the expected off-diagonal entries are much smaller than the diagonal entries, direct Monte Carlo estimation of OD is noisy. We therefore use the following expressions to reduce this Monte Carlo variance while preserving the expected off-diagonal entries. From (65) and (67) , each gradient is affine in Φ \Phi Φ when the weights and X X X are fixed. For each s ∈ { 1 , … , S } s \in\{1,\ldots,S\} s ∈ { 1 , … , S } , let G p ( s ) ( Φ ) G_p^{(s)}(\Phi) G p ( s ) ( Φ ) denote the gradient for projection p ∈ { Q , K , V , O } p \in\{Q,K,V,O\} p ∈ { Q , K , V , O } under the s s s -th initialization and target Φ \Phi Φ . Using the same weights and X X X , we evaluate the gradient at Φ \Phi Φ , − Φ -\Phi − Φ , and 0 0 0 , and define
Δ p ( s ) = 1 2 [ G p ( s ) ( Φ ) G p ( s ) ( Φ ) ⊤ + G p ( s ) ( − Φ ) G p ( s ) ( − Φ ) ⊤ ] − G p ( s ) ( 0 ) G p ( s ) ( 0 ) ⊤ . (125) \Delta_p^{(s)}=\frac{1}{2}\left[G_p^{(s)}(\Phi)G_p^{(s)}(\Phi)^{\top}+G_p^{(s)}(-\Phi)G_p^{(s)}(-\Phi)^{\top}\right]-G_p^{(s)}(0)G_p^{(s)}(0)^{\top}. \tag*{(125)} Δ p ( s ) = 2 1 [ G p ( s ) ( Φ ) G p ( s ) ( Φ ) ⊤ + G p ( s ) ( − Φ ) G p ( s ) ( − Φ ) ⊤ ] − G p ( s ) ( 0 ) G p ( s ) ( 0 ) ⊤ . (125)
The symmetric combination cancels the terms independent of Φ \Phi Φ and linear in Φ \Phi Φ , leaving the contribution quadratic in the target. Combining this cancellation with the target-dependent terms in (68) and (69) gives
E [ Δ Q ( s ) ] = d v γ d k ∥ Φ ∥ F 2 I d k , E [ Δ K ( s ) ] = d v γ d k ∥ Φ ∥ F 2 I d k , E [ Δ V ( s ) ] = γ ∥ Φ ∥ F 2 I d v , E [ Δ O ( s ) ] = d v γ Φ Φ ⊤ . (126) \begin{aligned}
\mathbb{E}[\Delta_Q^{(s)}] &= \frac{d_v\gamma}{d_k}\lVert\Phi\rVert_F^2 I_{d_k},\\
\mathbb{E}[\Delta_K^{(s)}] &= \frac{d_v\gamma}{d_k}\lVert\Phi\rVert_F^2 I_{d_k},\\
\mathbb{E}[\Delta_V^{(s)}] &= \gamma\lVert\Phi\rVert_F^2 I_{d_v},\\
\mathbb{E}[\Delta_O^{(s)}] &= d_v\gamma\Phi\Phi^{\top}. \tag*{(126)}
\end{aligned} E [ Δ Q ( s ) ] E [ Δ K ( s ) ] E [ Δ V ( s ) ] E [ Δ O ( s ) ] = d k d v γ ∥ Φ ∥ F 2 I d k , = d k d v γ ∥ Φ ∥ F 2 I d k , = γ ∥ Φ ∥ F 2 I d v , = d v γ Φ Φ ⊤ . (126)
Thus, Δ p ( s ) \Delta_p^{(s)} Δ p ( s ) in (125) has zero expected off-diagonal entries for p ∈ { Q , K , V } p \in\{Q,K,V\} p ∈ { Q , K , V } , exactly as in (123) , while for p = O p=O p = O it preserves the off-diagonal term in (122) .
For each configuration, we compute D ^ \widehat{D} D from the full empirical expected Gram. For O D ^ \widehat{\mathrm{OD}} OD , we first average Δ p ( s ) \Delta_p^{(s)} Δ p ( s ) over the same S S S initializations and then apply O D ( ⋅ ) \mathrm{OD}(\cdot) OD ( ⋅ ) . We then compute R ^ = D ^ / O D ^ \widehat{R}=\widehat{D}/\widehat{\mathrm{OD}} R = D / OD .
Figure 14 shows the dimension-dependent diagonal scaling predicted by Theorem 4 and Corollary 3. When d k d_k d k is varied with d v d_v d v fixed, D ^ \widehat{D} D for G Q G_Q G Q and G K G_K G K decreases, consistent with D = Θ ( d 0 2 d v 2 / d k ) D=\Theta(d_0^2d_v^2/d_k) D = Θ ( d 0 2 d v 2 / d k ) , while G V G_V G V and G O G_O G O depend only weakly on d k d_k d k . Increasing d v d_v d v increases D ^ \widehat{D} D for all four projections; the stronger growth of G Q G_Q G Q , G K G_K G K , G O G_O G O relative to G V G_V G V is consistent with their quadratic versus linear dependence on d v d_v d v . When d k = d v d_k=d_v d k = d v are increased jointly, D ^ \widehat{D} D grows approximately linearly for G Q G_Q G Q , G K G_K G K , G V G_V G V and more rapidly for G O G_O G O , again matching the predicted orders.
Figure 14. Empirical dimension scaling for single-layer attention. Rows vary d k = d v d_k=d_v d k = d v jointly, d k d_k d k , d v d_v d v , and d 0 = d 2 d_0=d_2 d 0 = d 2 (top to bottom). Columns show D ^ \widehat{D} D , O D ^ \widehat{OD} O D , and R ^ = D ^ / O D ^ \widehat{R}=\widehat{D}/\widehat{OD} R = D / O D for G Q G_Q G Q , G K G_K G K , G V G_V G V , and G O G_O G O , each normalized by its value at the baseline configuration (first three rows: T = 32 T=32 T = 32 , d 0 = 512 d_0=512 d 0 = 512 , d k = d v = 80 d_k=d_v=80 d k = d v = 80 , d 2 = 12 d_2=12 d 2 = 12 ; fourth row: T = 16 T=16 T = 16 , d 0 = d 2 = 32 d_0=d_2=32 d 0 = d 2 = 32 , d k = d v = 20 d_k=d_v=20 d k = d v = 20 ), to facilitate comparison across projections with different absolute scales. Empirical estimates use 10,000 independent initializations per configuration; the off-diagonal statistics are estimated using (125).
For Q Q Q , K K K , and V V V , the population off-diagonal entries are exactly zero by Theorem 4; the nonzero O D ^ \widehat{\mathrm{OD}} OD values are finite-sample Monte Carlo variances. For the output projection, O D ^ \widehat{\mathrm{OD}} OD grows approximately linearly with d v d_v d v and is nearly unchanged when only d k d_k d k is varied, while R ^ \widehat{R} R grows approximately linearly with d v d_v d v , consistent with Corollary 3, which gives D = Θ ( d v 2 d 0 6 ) D=\Theta(d_v^2d_0^6) D = Θ ( d v 2 d 0 6 ) , O D = Θ ( d v d 0 3 ) \mathrm{OD}=\Theta(d_vd_0^3) OD = Θ ( d v d 0 3 ) , and R = Θ ( d v d 0 3 ) R=\Theta(d_vd_0^3) R = Θ ( d v d 0 3 ) . For the joint d 0 = d 2 = m d_0=d_2=m d 0 = d 2 = m sweep, Corollary 3 gives D = Θ ( m 8 ) D=\Theta(m^8) D = Θ ( m 8 ) for G Q G_Q G Q , G K G_K G K , G V G_V G V , while for G O G_O G O , D = Θ ( m 6 ) D=\Theta(m^6) D = Θ ( m 6 ) , O D = Θ ( m 3 ) \mathrm{OD}=\Theta(m^3) OD = Θ ( m 3 ) , and hence R = Θ ( m 3 ) R=\Theta(m^3) R = Θ ( m 3 ) .
Empirical Evaluation on Large Language Models
We next test whether the diagonal dominance behavior of Theorem 4 persists in transformer language models. We consider Qwen3.5-0.8B-Base ([61 ] ), Granite-4.0-1B-Base ([28 ] ), and Llama-3.2-1B ([47 ] ). We perform two experiments. First, we examine the expected gradient Grams of individual attention projections at random initialization. Second, we measure how their diagonal dominance changes with model width across the three models.
Gradient Gram Structure at Initialization
We first examine the gradient Gram of attention projections in Qwen3.5-0.8B-Base, Granite-4.0-1B-Base, and Llama-3.2-1B at random initialization. The models are instantiated from their released Hugging Face configurations ([80 ] ) without pretrained weights. We set the sequence length to T = 256 T=256 T = 256 , retain each model’s released model width and head dimension, and set the number of key/value heads equal to the number of query heads. Native RoPE behavior is used throughout.
We analyze head 0 in block 3 for Qwen and in block 0 for Granite and Llama. For Qwen, DeltaNet blocks are excluded, so block 3 is the first full-attention block considered. For each projection p ∈ { Q , K , V , O } p \in\{Q, K, V, O\} p ∈ { Q , K , V , O } , we estimate
P ^ p = 1 N W N X ∑ w = 1 N W ∑ x = 1 N X G p , w , x G p , w , x ⊤ (127) \widehat{P}_p = \frac{1}{N_W N_X} \sum_{w=1}^{N_W} \sum_{x=1}^{N_X} G_{p,w,x}G_{p,w,x}^{\top} \tag*{(127)} P p = N W N X 1 w = 1 ∑ N W x = 1 ∑ N X G p , w , x G p , w , x ⊤ (127)
using N W = 25 N_W = 25 N W = 25 independent random initializations and N X = 64 N_X = 64 N X = 64 fixed FineWeb-Edu ([55 ] ) sequences. We use N W = 25 N_W = 25 N W = 25 independent random initializations obtained from each model’s native Hugging Face initialization scheme, which uses zero-mean Gaussian initialization for the attention projection weights with model-specific scales. The same input sequences are reused across all initializations. For Qwen, the gating rows of the query projection are excluded when forming the query Gram.
Figure 15 shows the resulting empirical expected Grams. All four attention projections exhibit a clear diagonal component across all three models. This indicates that the diagonal structure predicted for the single-layer attention model in Theorem 4 also appears in deeper transformer architectures at random initialization.
Model Width Scaling
We next examine how diagonal dominance changes with model width. Using the same random initialization protocol as above, we vary the transformer hidden width (hidden_size) while holding the head dimension (head_dim) fixed at its released value. In the notation of our single-layer attention model, this corresponds to varying the residual input and output dimensions d 0 = d 2 d_0 = d_2 d 0 = d 2 while keeping d k = d v d_k = d_v d k = d v fixed. The released head dimensions are 256 for Qwen3.5-0.8B-Base, 128 for Granite-4.0-1B-Base, and 64 for Llama-3.2-1B. The query-head count (num_attention_heads) is held fixed, and the number of key/value heads (num_key_value_heads) is set equal to the number of query heads to enable analogous dimension sweep in the models. The sequence length is fixed at T = 256 T = 256 T = 256 .
We consider the model-width grid
{ 128 , 256 , 512 , 768 , 1024 , 1536 , 2048 , 3072 , 4096 , 6144 , 8192 , 12288 , 16384 } . \{128, 256, 512, 768, 1024, 1536, 2048, 3072, 4096, 6144, 8192, 12288, 16384\}. { 128 , 256 , 512 , 768 , 1024 , 1536 , 2048 , 3072 , 4096 , 6144 , 8192 , 12288 , 16384 } .
At each width, we use 25 independent random initializations (obtained from each model’s native Hugging Face initialization scheme, which uses zero-mean Gaussian initialization for the attention projection weights with model-specific scales) and the same 64 fixed FineWeb-Edu sequences. We analyze head 0 in block 3 for Qwen and in block 0 for Granite and Llama. For each projection p ∈ { Q , K , V , O } p \in\{Q, K, V, O\} p ∈ { Q , K , V , O } , we report the empirical diagonal-dominance ratio R ^ p = R ( P ^ p ) \widehat{R}_p = R(\widehat{P}_p) R p = R ( P p ) , with R ( ⋅ ) R(\cdot) R ( ⋅ ) defined in (99) .
Figure 16 shows that R ^ \widehat{R} R increases with model width for all four attention projections across all three models. For G O G_O G O , this is consistent with the R = Θ ( d v d 0 3 ) R = \Theta(d_v d_0^3) R = Θ ( d v d 0 3 ) scaling in Corollary 3. The increase is strongest for G Q G_Q G Q and G K G_K G K in Qwen, and for G V G_V G V and G O G_O G O in Granite and Llama.
Figure 15. Empirical expected gradient self outer-products for Qwen3.5-0.8B-Base, Granite-4.0-1B-Base, and Llama-3.2-1B (columns), estimated from 25 independent random initializations and 64 fixed FineWeb-Edu sequences of length T = 256 T = 256 T = 256 . The rows show E ^ [ G Q G Q ⊤ ] \widehat{\mathbb{E}}[G_QG_Q^\top] E [ G Q G Q ⊤ ] , E ^ [ G K G K ⊤ ] \widehat{\mathbb{E}}[G_KG_K^\top] E [ G K G K ⊤ ] , E ^ [ G V G V ⊤ ] \widehat{\mathbb{E}}[G_VG_V^\top] E [ G V G V ⊤ ] , and E ^ [ G O G O ⊤ ] \widehat{\mathbb{E}}[G_OG_O^\top] E [ G O G O ⊤ ] . We use head 0 of block 3 for Qwen and head 0 of block 0 for Granite and Llama. Each heatmap is normalized by its empirical mean diagonal magnitude. A common logarithmic color scale is used across all panels.
Figure 16. Model width scaling of the empirical diagonal-dominance ratio R ^ \hat{R} R ^ for G Q G_Q G Q , G K G_K G K , G V G_V G V , and G O G_O G O in Qwen3.5-0.8B-Base, Granite-4.0-1B-Base, and Llama-3.2-1B. Model width (hidden_size) is varied while the head dimension (head_dim) is held fixed at its released value. We analyze head 0 in block 3 for Qwen and in block 0 for Granite and Llama. Each point is estimated from 25 independent random initializations and 64 fixed FineWeb-Edu sequences of length T = 256 T = 256 T = 256 .