Introduction
The central problem in statistical learning is to infer structure from an i.i.d. sample x 1 , … , x n x_1,\ldots,x_n x 1 , … , x n , with or without paired observations y 1 , … , y n y_1,\ldots,y_n y 1 , … , y n . The accompanying central challenge of high-dimensionality in modern machine learning arises from increasingly complex datasets (e.g., ImageNet [34 ] , LAION-5B [70 ] , and FineWeb [66 ] ) and model architectures (e.g., diffusion [51 , 67 ] and LLMs [23 , 46 ] ), going beyond the classical understanding of large-sample asymptotics [77 ] and sparsity [79 ] . Random matrix theory (RMT) has become popular in theoretical machine learning in those regimes where learning is possible but challenging. Not only useful for modeling frontier learning models such as kernel methods, random features, and neural networks [57 , 63 , 33 , 64 ] , it is also successful in understanding many empirical phenomena in machine learning, such as double-descent, benign-overfitting, memorization, and scaling laws [12 , 61 , 29 , 27 , 65 , 4 ] .
Existing tools in RMT primarily focus on the proportional setup when n / d n/d n / d converges to a constant, in which x i ∈ R d x_i \in\mathbb{R}^d x i ∈ R d has population covariance Σ \boldsymbol{\Sigma} Σ and the sample covariance $\widehat{\boldsymbol{\Sigma}}=\boldsymbol{X}^{*}\boldsymbol{X}/n does not concentrate [60 ] . This setup, including its extension to polynomial scalings when n / d ℓ n/d^{\ell} n / d ℓ converges [18 , 59 ] , still allows the input dimension to grow. In functional data analysis and in regression with a fixed kernel, it is natural to increase n n n while keeping the underlying Hilbert space H \mathcal{H} H and its covariance Σ \boldsymbol{\Sigma} Σ fixed, even when H \mathcal{H} H is infinite-dimensional. Empirically, an infinite-dimensional Σ \boldsymbol{\Sigma} Σ whose population spectrum σ 1 ≥ σ 2 ≥ ⋯ \sigma_1 \ge\sigma_2 \ge\cdots σ 1 ≥ σ 2 ≥ ⋯ is trace-class (Tr Σ < ∞ \operatorname{Tr} \boldsymbol{\Sigma} < \infty Tr Σ < ∞ ), e.g., Zipf’s law with polynomial decay, σ i = i − 1 − α \sigma_i = i^{-1-\alpha} σ i = i − 1 − α for some α > 0 \alpha> 0 α > 0 , is often a better model for real-world data, as shown in Figure 1 . We observe that the sample spectrum from a fixed Hilbert space with growing size n n n is a better proxy for the sample spectrum of CIFAR-10 features.
Figure 1. Spectra of X X ∗ / n XX^*/n X X ∗ / n pooled over 2,000 trials, of (i) proportional isotropic Gaussian features with n = 4 d n = 4d n = 4 d , (ii) Zipf’s law with i.i.d. Gaussian coordinates, σ i = i − 1.2 \sigma_i = i^{-1.2} σ i = i − 1.2 and (iii) CIFAR-10 ResNet-50 second-to-last layer features (d = 2 , 048 d = 2,048 d = 2 , 048 ). The zoomed in view show rescaled histograms. For CIFAR-10, we estimate σ ^ n \widehat{\sigma}_n σ n from the separate training set.
The pioneering work of Cheng and Montanari [28 ] addressed this setting for the prediction risk of ridge regression problems, with later works studying closely-related problems [63 , 33 ] . However, a more applicable and complete dimension-free spectral theory at the bulk scale, beyond ridge regression, is still missing. In this work, we develop nonasymptotic and deterministic approximations for (i) spectral parameters and laws and (ii) the prediction risk of a general class of spectral filters. Given i.i.d. x 1 , … , x n ∈ H x_1,\ldots,x_n \in\mathcal{H} x 1 , … , x n ∈ H , we study the two following problems:
Unsupervised learning: the empirical spectrum. How can we characterize the (top) empirical spectrum of the sample covariance Σ ^ \widehat{\boldsymbol{\Sigma}} Σ ?
Equivalently, we consider the Gram matrix X X ∗ / n XX^*/n X X ∗ / n . Let σ ^ 1 ≥ ⋯ ≥ σ ^ n ≥ 0 \widehat{\sigma}_1 \ge\cdots\ge\widehat{\sigma}_n \ge0 σ 1 ≥ ⋯ ≥ σ n ≥ 0 be the sample eigenvalues of X X ∗ / n XX^*/n X X ∗ / n . We seek a sharp characterization of the unnormalized empirical spectral distribution
μ ^ n : = 1 n ∑ j = 1 n δ n σ ^ j (1) \widehat{\mu}_n := \frac{1}{n}\sum_{j=1}^{n}\delta_{n\widehat{\sigma}_j} \tag*{(1)} μ n := n 1 j = 1 ∑ n δ n σ j (1)
at the bulk scale, i.e., on any interval E ⊂ R E \subset\mathbb{R} E ⊂ R such μ ^ n ( E ) ∈ ( c , 1 − c ) \widehat{\mu}_n(E) \in(c,1-c) μ n ( E ) ∈ ( c , 1 − c ) for some c > 0 c > 0 c > 0 .
Supervised learning: risk of spectral filters. How can we characterize the prediction error of a spectral filter ϕ \phi ϕ ?
We consider a linear model y = X β + ε \boldsymbol{y} = \boldsymbol{X}\boldsymbol{\beta} + \boldsymbol{\varepsilon} y = X β + ε with noise E [ ε ∣ X ] = 0 \mathbb{E}[\boldsymbol{\varepsilon} \mid\boldsymbol{X}] = 0 E [ ε ∣ X ] = 0 . The spectral filter Φ \Phi Φ multiplies each sample eigendirection at eigenvalue x x x by Φ ( x ) \Phi(x) Φ ( x ) . Let the residual function be ψ ( x ) = 1 − x Φ ( x ) \psi(x) = 1 - x\Phi(x) ψ ( x ) = 1 − x Φ ( x ) ; this yields the estimator β ^ Φ = Φ ( Σ ^ ) X ∗ y / n \widehat{\boldsymbol{\beta}}_{\Phi} = \Phi(\widehat{\boldsymbol{\Sigma}})\boldsymbol{X}^{*}\boldsymbol{y}/n β Φ = Φ ( Σ ) X ∗ y / n , which covers a broad class of spectral algorithms, e.g., ψ ( x ) = λ / ( x + λ ) \psi(x) = \lambda/(x + \lambda) ψ ( x ) = λ / ( x + λ ) for ridge, ψ ( x ) = ( λ / ( x + λ ) ) k \psi(x) = (\lambda/(x + \lambda))^{k} ψ ( x ) = ( λ / ( x + λ ) ) k for iterated ridge, ψ ( x ) = e − t x \psi(x) = e^{-tx} ψ ( x ) = e − t x for gradient flow, and ψ ( x ) = 1 { x < λ } \psi(x) = \mathbb{1}\{x < \lambda\} ψ ( x ) = 1 { x < λ } for principal component regression (PCR). We seek sharp characterizations for the conditional generalization error
R X [ Φ ] : = E [ ∣ ⟨ x n e w , β ^ Φ − β ⟩ ∣ 2 ∣ X ] = V X [ Φ ] + B X [ Φ ] . (2) \mathcal{R}_{X}[\Phi] := \mathbb{E}[|\langle\boldsymbol{x}_{\mathrm{new}},\widehat{\boldsymbol{\beta}}_{\Phi} - \boldsymbol{\beta}\rangle|^{2} \mid\boldsymbol{X}] = \mathcal{V}_{X}[\Phi] + \mathcal{B}_{X}[\Phi]. \tag*{(2)} R X [ Φ ] := E [ ∣ ⟨ x new , β Φ − β ⟩ ∣ 2 ∣ X ] = V X [ Φ ] + B X [ Φ ] . (2)
Here x n e w \boldsymbol{x}_{\mathrm{new}} x new denotes a test feature with the same law as x 1 \boldsymbol{x}_{1} x 1 , drawn independently of the training data. The expectation averages over the test feature and training noise conditional on X \boldsymbol{X} X . The term V X [ Φ ] = τ 2 Tr ( Σ Σ ^ Φ ( Σ ^ ) 2 ) / n \mathcal{V}_{X}[\Phi] = \tau^{2}\operatorname{Tr}(\boldsymbol{\Sigma}\widehat{\boldsymbol{\Sigma}}\Phi(\widehat{\boldsymbol{\Sigma}})^{2})/n V X [ Φ ] = τ 2 Tr ( Σ Σ Φ ( Σ ) 2 ) / n is the conditional variance when E [ ε ε ∗ ∣ X ] = τ 2 I n \mathbb{E}[\boldsymbol{\varepsilon}\boldsymbol{\varepsilon}^{*} \mid\boldsymbol{X}] = \tau^{2}I_{n} E [ ε ε ∗ ∣ X ] = τ 2 I n , and B X [ Φ ] = ∥ Σ 1 / 2 ψ ( Σ ^ ) β ∥ 2 \mathcal{B}_{X}[\Phi] = \|\boldsymbol{\Sigma}^{1/2}\psi(\widehat{\boldsymbol{\Sigma}})\boldsymbol{\beta}\|^{2} B X [ Φ ] = ∥ Σ 1/2 ψ ( Σ ) β ∥ 2 is the squared bias.
Our contributions
We give our solutions to the two problems, and summarize our contributions in the paper.
Deterministic equivalents to the empirical spectrum. In § 2.2 , we track the empirical spectral law μ ^ n \widehat{\mu}_{n} μ n and its Stieltjes transform s ^ n \widehat{s}_{n} s n by their deterministic counterparts:
( s n ( Σ ) , μ n ( Σ ) ) ⟷ deterministic equivalent closely approximates ( s ^ n ( X ) , μ ^ n ( X ) ) . (s_{n}(\boldsymbol{\Sigma}), \mu_{n}(\boldsymbol{\Sigma}))
\underset{\text{closely approximates}}{\overset{\text{deterministic equivalent}}{\longleftrightarrow}}
(\widehat{s}_{n}(\boldsymbol{X}), \widehat{\mu}_{n}(\boldsymbol{X})). ( s n ( Σ ) , μ n ( Σ )) closely approximates ⟷ deterministic equivalent ( s n ( X ) , μ n ( X )) .
At fixed height on the bulk scale, the Stieltjes transform bound captures the optimal n − 1 n^{-1} n − 1 power for averaged fluctuations (cf. [54 ] ).
Deterministic equivalents for prediction risk. In § 2.3 , for a given analytic spectral filter, at fixed signal and noise level, we obtain
( V n [ Φ ; Σ ] , B n [ Φ ; Σ ] ) ⟷ deterministic equivalent closely approximates ( V X [ Φ ] , B X [ Φ ] ) . (\mathcal{V}_{n}[\Phi;\boldsymbol{\Sigma}],\mathcal{B}_{n}[\Phi;\boldsymbol{\Sigma}])
\underset{\text{closely approximates}}{\overset{\text{deterministic equivalent}}{\longleftrightarrow}}
(\mathcal{V}_{X}[\Phi],\mathcal{B}_{X}[\Phi]). ( V n [ Φ ; Σ ] , B n [ Φ ; Σ ]) closely approximates ⟷ deterministic equivalent ( V X [ Φ ] , B X [ Φ ]) .
Our bounds capture the optimal corresponding averaged and local fluctuation rate, n − 1 n^{-1} n − 1 and n − 1 / 2 n^{-1/2} n − 1/2 , respectively.
Technical innovation: the backward interpolation path. In § 5 , we present our technical core, where we achieve the preceding goals by approximating weighted traces of resolvents and mixed resolvents. For a deterministic weight Q \boldsymbol{Q} Q , we have
Tr ( Q Σ ( Σ − ω ( z / n ) I ) − 1 ) ⟷ deterministic equivalent closely approximates Tr ( Q X ∗ ( X X ∗ − z I n ) − 1 X ) . \operatorname{Tr}\left(\boldsymbol{Q}\boldsymbol{\Sigma}(\boldsymbol{\Sigma} - \omega(z/n)I)^{-1}\right)
\underset{\text{closely approximates}}{\overset{\text{deterministic equivalent}}{\longleftrightarrow}}
\operatorname{Tr}\left(\boldsymbol{Q}\boldsymbol{X}^{*}(\boldsymbol{X}\boldsymbol{X}^{*} - zI_{n})^{-1}\boldsymbol{X}\right). Tr ( Q Σ ( Σ − ω ( z / n ) I ) − 1 ) closely approximates ⟷ deterministic equivalent Tr ( Q X ∗ ( X X ∗ − z I n ) − 1 X ) .
Similar deterministic equivalents exist for the mixed-resolvent traces. We obtain these comparisons along a deterministic backward path, inspired by and innovated upon the analysis of Cheng and Montanari [28 ] . This adjustment cancels the leading drift at each step and connects the sample trace to its population counterpart while keeping each step deterministic.
We emphasize that all our approximations are nonasymptotic and that all deterministic equivalents can vary in n n n , even if they could have a limit in n n n . For power law and critical decay, the spectral approximation remains valid at mesoscopic heights h = n − δ h = n^{-\delta} h = n − δ in the interior of the bulk, with h h h measured relative to the bulk width and 0 < δ < 1 / 17 0 < \delta< 1/17 0 < δ < 1/17 .
Related work
Random matrix theory. The study of random sample covariance matrices is classical and dates back to Wishart’s characterization of their distribution under Gaussian sampling [80 ] . The Marchenko-Pastur law subsequently described their limiting empirical spectrum when dimension and sample size grow proportionally [60 ] . More recently, these global descriptions have been sharpened to local and anisotropic laws, controlling resolvents and spectral statistics on increasingly fine scales [54 ] . Very recent work also establishes quantitative approximations for mixed-resolvent traces in the proportional regime [73 ] .
Relevant works include extensions to polynomial dimensional scalings [35 ] . However, these settings do not directly address an infinite decaying population spectrum with a shrinking empirical bulk. Removing restrictions on the population condition number is essential here, since even finite-dimensional truncations become increasingly ill-conditioned. Cheng and Montanari [28 ] provided a methodological foundation through their leave-one-out interpolation strategy. Although they do not directly derive spectral laws and local laws over the bulk, their ridge formulas suggest the deterministic equivalent through analytic continuation.
Kernel random matrices. Our work closely relates to kernel methods, which provide a natural Hilbert-space setting through a possibly infinite-dimensional feature map, even when the input space is fixed [74 Section 4.5] . Despite having the Hilbert space embedding, much of the kernel random matrix literature instead considers growing input dimension. The foundational work of El Karoui [40 ] studies smooth inner-product and distance kernels when input dimension grows proportionally to sample size, showing that their spectra admit a linear approximation involving the sample Gram matrix. Subsequent work treats nonsmooth kernels, universality and spectral edges [30 , 36 , 43 ] . Further extensions include polynomial dimensional scalings that keep higher-degree feature components [62 , 38 ] . Our framework instead permits a fixed kernel and input distribution, and studies the shrinking bulk of a decaying feature spectrum under quadratic concentration.
Spectral regularization and risk characterization. Studying the risk of spectral regularization and spectral filters permits a common framework for ridge regression, iterative regularization and early stopping. Classical analyses in this direction give risk bounds and establish learning rates and statistical optimality [58 , 24 ] .
Precise risk characterizations are available under proportional asymptotics, including for ridge regression and gradient flow [37 , 2 ] . Beyond this regime, Cheng and Montanari [28 ] establish nonasymptotic, dimension-free deterministic equivalents for ridge prediction risk at the bulk scale, with nonnegative regularization. Li et al. [56 ] establish asymptotic risk formulas for analytic spectral algorithms under power-law decay, with regularization polynomially above the bulk scale. Their formulas apply the filter directly to the population covariance, whereas ours preserve corrections from a self-consistent spectral parameter at the bulk scale. Notably, Velikanov et al. [78 ] derive closely related risk formulas for an infinite Gaussian feature model, relying on concentration and stability approximations whose justification in that setting is left open. Our results give deterministic equivalents for analytic filters at the bulk scale under dimension-free concentration assumptions, including negative ridge and gradient flow, with explicit nonasymptotic errors for both variance and squared bias.
Dimension-free learning theory. Beyond the ridge regression work [28 ] , dimension-free deterministic equivalents have been used to analyze kernel ridge regression and generalized cross-validation [63 ] and random feature regression [33 ] . Related random matrix approaches have also been used to study scaling laws [4 ] . These works relate prediction risk to spectral decay and signal alignment, making it possible to investigate how to choose regularization, how many random features are required to achieve statistical accuracy, and how learning rates vary with sample size.
Our extension to analytic filters provides quantitative inputs for studying iterative regularization and early stopping through the same spectral quantities. In particular, the deterministic risk expressions maintain the source-dependent bias-variance trade-off over the course of training. As a result, they offer a foundation for comparing regularization methods and studying stopping rules, with direct control over the approximation error. More generally, our spectral laws connect these learning problems to the geometry of the empirical spectrum.
Notation. For a positive integer n n n , let [ n ] : = { 1 , … , n } [n] := \{1,\ldots,n\} [ n ] := { 1 , … , n } and let C + : = { z ∈ C : Im z > 0 } \mathbb{C}_{+} := \{z \in\mathbb{C} : \operatorname{Im} z > 0\} C + := { z ∈ C : Im z > 0 } . For sets A , B ⊆ C A,B \subseteq\mathbb{C} A , B ⊆ C , define dist ( A , B ) : = inf { ∣ a − b ∣ : a ∈ A , b ∈ B } \operatorname{dist}(A,B) := \inf\{|a-b| : a \in A,\ b \in B\} dist ( A , B ) := inf { ∣ a − b ∣ : a ∈ A , b ∈ B } , with points identified with singleton sets. We write a ∧ b : = min { a , b } a \wedge b := \min\{a,b\} a ∧ b := min { a , b } and a ∨ b : = max { a , b } a \vee b := \max\{a,b\} a ∨ b := max { a , b } . Let H \mathcal{H} H be a real or complex separable Hilbert space. We denote by L ( H ) \mathcal{L}(\mathcal{H}) L ( H ) the Banach space of bounded linear operators acting on H \mathcal{H} H . The inner product on H \mathcal{H} H , denoted ⟨ x , y ⟩ H \langle x,y\rangle_{\mathcal{H}} ⟨ x , y ⟩ H , is assumed to be conjugate-linear in its first argument; with respect to an orthonormal basis, it can be expressed in coordinates as x ∗ y x^{*}y x ∗ y . For A ∈ L ( H ) A \in\mathcal{L}(\mathcal{H}) A ∈ L ( H ) , we denote its adjoint by A ∗ A^{*} A ∗ , its operator norm by ∥ A ∥ \|A\| ∥ A ∥ , its Hilbert-Schmidt (Frobenius) norm by ∥ A ∥ F \|A\|_{F} ∥ A ∥ F , its spectrum by Spec ( A ) \operatorname{Spec}(A) Spec ( A ) , and its trace by Tr ( A ) \operatorname{Tr}(A) Tr ( A ) whenever the latter is well defined. We write ∣ A ∣ = ( A ∗ A ) 1 / 2 |A| = (A^{*}A)^{1/2} ∣ A ∣ = ( A ∗ A ) 1/2 for the operator modulus and Tr ∣ A ∣ \operatorname{Tr}|A| Tr ∣ A ∣ for the trace norm when A A A is trace-class. The induced Hilbert-space norm is denoted by ∥ x ∥ H 2 : = ⟨ x , x ⟩ H \|x\|_{\mathcal{H}}^{2} := \langle x,x\rangle_{\mathcal{H}} ∥ x ∥ H 2 := ⟨ x , x ⟩ H . For any bounded positive semidefinite operator A A A on H \mathcal{H} H , define
⟨ x , y ⟩ A : = ⟨ x , A y ⟩ H , ∥ x ∥ A 2 : = ⟨ x , x ⟩ A = ∥ A 1 / 2 x ∥ H 2 . \langle x,y\rangle_{A} := \langle x,Ay\rangle_{\mathcal{H}}, \qquad\|x\|_{A}^{2} := \langle x,x\rangle_{A} = \|A^{1/2}x\|_{\mathcal{H}}^{2}. ⟨ x , y ⟩ A := ⟨ x , A y ⟩ H , ∥ x ∥ A 2 := ⟨ x , x ⟩ A = ∥ A 1/2 x ∥ H 2 .
When A A A is not injective, ∥ ⋅ ∥ A \|\cdot\|_{A} ∥ ⋅ ∥ A is understood as a seminorm. We use I I I for the identity operator and 0 \mathbf{0} 0 and 1 \mathbf{1} 1 for the all-zero and all-one vectors, respectively, with the underlying dimensions clear from context. For self-adjoint operators A , B A,B A , B , the relation A ⪰ B A \succeq B A ⪰ B means that A − B A-B A − B is positive semidefinite. We write λ min + ( A ) \lambda_{\min}^{+}(A) λ m i n + ( A ) for the smallest positive eigenvalue of a finite-rank positive semidefinite operator A A A . For x , y ∈ H x,y \in\mathcal{H} x , y ∈ H , the rank-one operator x y ∗ xy^{*} x y ∗ acts by ( x y ∗ ) v = ⟨ y , v ⟩ H x (xy^{*})v = \langle y,v\rangle_{\mathcal{H}}x ( x y ∗ ) v = ⟨ y , v ⟩ H x .
For a real or complex random variable Z Z Z and 1 ≤ p < ∞ 1 \leq p < \infty 1 ≤ p < ∞ , write ∥ Z ∥ L p : = ( E ∣ Z ∣ p ) 1 / p \|Z\|_{L^{p}} := (\mathbb{E}|Z|^{p})^{1/p} ∥ Z ∥ L p := ( E ∣ Z ∣ p ) 1/ p , with ∥ Z ∥ L ∞ \|Z\|_{L^{\infty}} ∥ Z ∥ L ∞ its essential supremum in modulus. For a function of a random element Y Y Y , we use L p ( Y ) L^{p}(Y) L p ( Y ) when specifying that the norm is taken under the law of Y Y Y . For square-integrable complex random variables, we use the bilinear covariance Cov ( Z , W ) : = E [ ( Z − E Z ) ( W − E W ) ] \operatorname{Cov}(Z,W) := \mathbb{E}[(Z-\mathbb{E}Z)(W-\mathbb{E}W)] Cov ( Z , W ) := E [( Z − E Z ) ( W − E W )] .
For nonnegative quantities f , g f,g f , g , we write f = O ( g ) f = O(g) f = O ( g ) if f ≤ C g f \leq Cg f ≤ C g and f = Ω ( g ) f = \Omega(g) f = Ω ( g ) if f ≥ c g f \geq cg f ≥ c g on the stated range, for positive constants C , c C,c C , c . We write f ≍ g f \asymp g f ≍ g or f = Θ ( g ) f = \Theta(g) f = Θ ( g ) , when both bounds hold. For signed or complex quantities the comparisons apply to their absolute values. We write f = o ( g ) f = o(g) f = o ( g ) if ∣ f / g ∣ → 0 |f/g| \to0 ∣ f / g ∣ → 0 along the specified limit, with n → ∞ n \to\infty n → ∞ understood for sequences unless stated otherwise. We use subscripts on O O O , Ω \Omega Ω , Θ \Theta Θ and o o o to indicate the parameters on which the implicit constants may depend.
Model and main results
Model setup
Dimension-free sample covariances. Let H \mathcal{H} H be a real separable Hilbert space of dimension d ∈ N ∪ { ∞ } d \in\mathbb{N} \cup\{\infty\} d ∈ N ∪ { ∞ } . We observe independent and identically distributed feature vectors x 1 , … , x n ∈ H x_{1},\ldots,x_{n} \in\mathcal{H} x 1 , … , x n ∈ H with mean zero and population covariance operator Σ : = E [ x i x i ∗ ] \boldsymbol{\Sigma} := \mathbb{E}[x_i x_i^*] Σ := E [ x i x i ∗ ] . For k ∈ [ n ] k \in[n] k ∈ [ n ] , define the partial data operator X k : H → R k \mathbf{X}_k : \mathcal{H} \to\mathbb{R}^k X k : H → R k and its unnormalized partial sample covariance operator by
( X k v ) j : = ⟨ x j , v ⟩ H , j ∈ [ k ] , and T k : = X k ∗ X k = ∑ j = 1 k x j x j ∗ . (\mathbf{X}_k v)_j := \langle x_j, v\rangle_{\mathcal{H}},\quad j \in[k], \qquad\text{and} \qquad T_k := \mathbf{X}_k^* \mathbf{X}_k = \sum_{j=1}^{k} x_j x_j^*. ( X k v ) j := ⟨ x j , v ⟩ H , j ∈ [ k ] , and T k := X k ∗ X k = j = 1 ∑ k x j x j ∗ .
Setting T 0 : = 0 T_0 := 0 T 0 := 0 and writing X : = X n \mathbf{X} := \mathbf{X}_n X := X n , we denote by r n : = rank ( X ) r_n := \operatorname{rank}(\mathbf{X}) r n := rank ( X ) . Let the positive eigenvalues of the normalized sample covariance operator X ∗ X / n \mathbf{X}^* \mathbf{X}/n X ∗ X / n be σ ^ 1 , n ≥ ⋯ ≥ σ ^ r n , n > 0 \widehat{\sigma}_{1,n} \ge\cdots\ge\widehat{\sigma}_{r_n,n} > 0 σ 1 , n ≥ ⋯ ≥ σ r n , n > 0 . In the following discussion, we let F k : = σ ( x 1 , … , x k ) \mathcal{F}_k := \sigma(x_1,\ldots,x_k) F k := σ ( x 1 , … , x k ) , with F 0 \mathcal{F}_0 F 0 trivial, and we abbreviate conditional expectation by E k [ ⋅ ] : = E [ ⋅ ∣ F k ] \mathbb{E}_k[\cdot] := \mathbb{E}[\cdot\mid\mathcal{F}_k] E k [ ⋅ ] := E [ ⋅ ∣ F k ] .
H \mathcal{H} H -valued feature distribution. We impose the two distributional conditions used in earlier dimension-free ridge-regression analysis [28 ] , and [63 , 33 ] impose closely related concentration conditions on kernel eigenfunctions or random feature maps. Either condition below yields the same dimension-free Hanson-Wright inequality, while condition (ii) permits dependence among the coordinates.
Assumption 1 (Feature distribution). The covariance operator Σ \boldsymbol{\Sigma} Σ is positive semi-definite and trace class. Without loss of generality, we assume ∥ Σ ∥ = 1 \|\boldsymbol{\Sigma}\| = 1 ∥ Σ ∥ = 1 . Let
Σ = ∑ j ≥ 1 σ j u j u j ∗ , 1 = σ 1 ≥ σ 2 ≥ ⋯ > 0 , ∑ j ≥ 1 σ j < ∞ , \boldsymbol{\Sigma} = \sum_{j \ge1} \sigma_j u_j u_j^*, \qquad1 = \sigma_1 \ge\sigma_2 \ge\cdots> 0, \qquad\sum_{j \ge1} \sigma_j < \infty, Σ = j ≥ 1 ∑ σ j u j u j ∗ , 1 = σ 1 ≥ σ 2 ≥ ⋯ > 0 , j ≥ 1 ∑ σ j < ∞ , where the sum is finite when d < ∞ d < \infty d < ∞ and zero-covariance directions are omitted. The features admit
x i = Σ 1 / 2 z i : = ∑ j ≥ 1 σ j z i j u j , (3) x_i = \boldsymbol{\Sigma}^{1/2} z_i := \sum_{j \ge1} \sqrt{\sigma_j}\,z_{ij}u_j, \tag*{(3)} x i = Σ 1/2 z i := j ≥ 1 ∑ σ j z ij u j , (3) *where the coordinate sequences are centered and isotropic1 , i.e., E [ z i ] = 0 \mathbb{E}[z_i] = 0 E [ z i ] = 0 and E [ z i z i ∗ ] = I \mathbb{E}[z_i z_i^*] = I E [ z i z i ∗ ] = I . There exists a constant C x > 0 C_x > 0 C x > 0 such that one of the following conditions holds:*
(i) (Independent sub-Gaussian coordinates). For every i i i , the coordinates { z i j } j ≥ 1 \{z_{ij}\}_{j \ge1} { z ij } j ≥ 1 are independent, not necessarily identically distributed, and
∥ z i j ∥ ψ 2 : = sup p ≥ 1 p − 1 / 2 ( E ∣ z i j ∣ p ) 1 / p ≤ C x . \|z_{ij}\|_{\psi_2} := \sup_{p \ge1} p^{-1/2}(\mathbb{E}|z_{ij}|^p)^{1/p} \le C_x. ∥ z ij ∥ ψ 2 := p ≥ 1 sup p − 1/2 ( E ∣ z ij ∣ p ) 1/ p ≤ C x . (ii) (Convex concentration). For every m ≤ d m \le d m ≤ d when d < ∞ d < \infty d < ∞ , and every m ≥ 1 m \ge1 m ≥ 1 when d = ∞ d = \infty d = ∞ , every convex 1 1 1 -Lipschitz function φ : R m → R \varphi: \mathbb{R}^m \to\mathbb{R} φ : R m → R , and every t > 0 t > 0 t > 0 ,
P { ∣ φ ( z i 1 , … , z i m ) − E φ ( z i 1 , … , z i m ) ∣ ≥ t } ≤ 2 exp ( − t 2 C x 2 ) . \mathbb{P}\left\{|\varphi(z_{i1},\ldots,z_{im})-\mathbb{E}\varphi(z_{i1},\ldots,z_{im})| \ge t\right\} \le2\exp\left(-\frac{t^2}{C_x^2}\right). P { ∣ φ ( z i 1 , … , z im ) − E φ ( z i 1 , … , z im ) ∣ ≥ t } ≤ 2 exp ( − C x 2 t 2 ) . When d = ∞ d = \infty d = ∞ , the series in Eq. (3) converges in H \mathcal{H} H almost surely and in L 2 L^2 L 2 .
The convex-concentration condition in (ii) includes several important cases, such as independent uniformly bounded coordinates and distributions satisfying a dimension-free log-Sobolev inequality; in particular, it includes uniformly strongly log-concave distributions [21 , 11 ] . Under either condition, the following Hanson-Wright inequality holds in H \mathcal{H} H . For finite-dimensional real vectors, the bound under condition (i) follows from the original Hanson-Wright inequality [68 Theorem 1.1] , whereas the bound under condition (ii) follows from its convex-concentration extension [1 Theorem 2.3] . The Hilbert-space extension by finite-rank approximation is also used in dimension-free ridge-regression analysis [28 ] .
Proposition 1 (Hanson-Wright inequality). Let x x x be an independent copy of x i x_i x i satisfying Assumption 1. For any bounded operator A \mathbf{A} A on the complexification of H \mathcal{H} H and any t ≥ 0 t \ge0 t ≥ 0 , there exists a universal constant c > 0 c > 0 c > 0 such that
P ( ∣ x ∗ A x − Tr ( Σ A ) ∣ ≥ t ) ≤ 4 exp { − c ( t 2 C x 4 ∥ Σ 1 / 2 A Σ 1 / 2 ∥ F 2 ∧ t C x 2 ∥ Σ 1 / 2 A Σ 1 / 2 ∥ ) } . (4) \mathbb{P}\left(\left|x^* \mathbf{A}x-\operatorname{Tr}(\boldsymbol{\Sigma}\mathbf{A})\right|\ge t\right)\le4\exp\left\{-c\left(\frac{t^2}{C_x^4\left\|\boldsymbol{\Sigma}^{1/2}\mathbf{A}\boldsymbol{\Sigma}^{1/2}\right\|_{\mathrm{F}}^2}\wedge\frac{t}{C_x^2\left\|\boldsymbol{\Sigma}^{1/2}\mathbf{A}\boldsymbol{\Sigma}^{1/2}\right\|}\right)\right\}. \tag*{(4)} P ( ∣ x ∗ A x − Tr ( Σ A ) ∣ ≥ t ) ≤ 4 exp ⎩ ⎨ ⎧ − c C x 4 Σ 1/2 A Σ 1/2 F 2 t 2 ∧ C x 2 Σ 1/2 A Σ 1/2 t ⎭ ⎬ ⎫ . (4) For real self-adjoint A \mathbf{A} A , the prefactor 4 can be replaced by 2. If A \mathbf{A} A is measurable with respect to a σ \sigma σ -field G \mathcal{G} G and x x x is independent of G \mathcal{G} G , then the same bound holds conditionally on G \mathcal{G} G .
Main examples: power law and critical law decay. To showcase how our results apply, we examine two infinite-dimensional populations as primary illustrative examples. For α , β > 0 \alpha,\beta>0 α , β > 0 , their eigenvalues are
(Power law) σ i = 1 i α + 1 and (Critical law) σ i = 1 i ( 1 + log i ) β + 1 . (5) \text{(Power law)}\quad\sigma_i=\frac{1}{i^{\alpha+1}}
\qquad\text{and}\qquad
\text{(Critical law)}\quad\sigma_i=\frac{1}{i(1+\log i)^{\beta+1}}. \tag*{(5)} (Power law) σ i = i α + 1 1 and (Critical law) σ i = i ( 1 + log i ) β + 1 1 . (5)
In the critical case, the logarithmic factor makes the spectrum summable at the boundary exponent i − 1 i^{-1} i − 1 . Of the two spectra in Eq. (5) , only the critical one satisfies the tail effective-rank criterion for benign overfitting in minimum-norm linear regression [12 ] . Tsigler and Bartlett [76 ] extended this analysis to ridge regression. For these same spectra, Cheng and Montanari [28 ] obtained dimension-free deterministic equivalents for ridge bias and variance. Subsequent work extended this approach to kernel ridge regression [63 ] and to random feature ridge regression, where it also yielded excess risk rates under power-law decay [33 ] . These analyses are building blocks for understanding scaling laws for prediction error in kernel and random feature regression [19 , 6 , 4 ] . Here we study the empirical Gram spectrum itself and the risk of general spectral filters in these two regimes.
Main result I: Spectral parameter and bulk laws
Our first results characterize the empirical spectral distributions (e.s.d.) of the sample covariance operators T n = X ∗ X T_n=X^*X T n = X ∗ X at scales adapted to the population covariance Σ \boldsymbol{\Sigma} Σ in a dimension-free manner, capturing the bulk behavior. The r n r_n r n positive eigenvalues of T n T_n T n are also the positive eigenvalues of X X ∗ XX^* X X ∗ . We therefore study the e.s.d. μ ^ n \widehat{\mu}_n μ n of the unscaled Gram matrix X X ∗ XX^* X X ∗ in Eq. (1) , and its associated empirical Stieltjes transform s ^ n \widehat{s}_n s n . As previewed in the introduction, the main result in this section is essentially
( s n ( Σ ) , μ n ( Σ ) ) ≈ ( s ^ n ( X ) , μ ^ n ( X ) ) , \left(s_n(\boldsymbol{\Sigma}),\mu_n(\boldsymbol{\Sigma})\right)\approx\left(\widehat{s}_n(\mathbf{X}),\widehat{\mu}_n(\mathbf{X})\right), ( s n ( Σ ) , μ n ( Σ ) ) ≈ ( s n ( X ) , μ n ( X ) ) ,
where the deterministic pair ( s n , μ n ) (s_n,\mu_n) ( s n , μ n ) depends only on n n n and Σ \boldsymbol{\Sigma} Σ . Theorem 1 quantifies the approximation for Stieltjes transforms and interval masses.
Key to our analysis, we consider the effective spectral parameter ω ( z ) \omega(z) ω ( z ) , defined as the unique solution in C + \mathbb{C}_+ C + to
z ω ( z ) − 1 = 1 + 1 n Tr ( Σ ( ω ( z ) I − Σ ) − 1 ) . (6) z\omega(z)^{-1}=1+\frac{1}{n}\operatorname{Tr}\left(\boldsymbol{\Sigma}\left(\omega(z)\mathbf{I}-\boldsymbol{\Sigma}\right)^{-1}\right). \tag*{(6)} z ω ( z ) − 1 = 1 + n 1 Tr ( Σ ( ω ( z ) I − Σ ) − 1 ) . (6)
The map z ↦ ω ( z ) z \mapsto\omega(z) z ↦ ω ( z ) is analytic on C + \mathbb{C}_{+} C + . Written equivalently, it is clear that the effective spectral parameter ω ( z ) \omega(z) ω ( z ) is closely connected to the notion of effective regularization λ ⋆ \lambda_{\star} λ ⋆ defined in [28 ] for dimension free ridge regression. This relationship can be interpreted in terms of the deterministic variance and bias quantities that arise in proportional-asymptotic linear regression [13 , 50 ] . In particular, for λ ∈ ( 0 , + ∞ ) \lambda\in(0,+\infty) λ ∈ ( 0 , + ∞ ) , one has
n ( z ω ( z ) − 1 ) = Tr ( Σ ( ω ( z ) I − Σ ) − 1 ) and n ( 1 − λ λ ⋆ ( λ ) ) = Tr ( Σ ( Σ + λ ⋆ ( λ ) I ) − 1 ) , (7) n\left(\frac{z}{\omega(z)}-1\right)=\operatorname{Tr}\left(\boldsymbol{\Sigma}(\omega(z)I-\boldsymbol{\Sigma})^{-1}\right)
\quad\text{and}\quad
n\left(1-\frac{\lambda}{\lambda_{\star}(\lambda)}\right)=\operatorname{Tr}\left(\boldsymbol{\Sigma}(\boldsymbol{\Sigma}+\lambda_{\star}(\lambda)I)^{-1}\right), \tag*{(7)} n ( ω ( z ) z − 1 ) = Tr ( Σ ( ω ( z ) I − Σ ) − 1 ) and n ( 1 − λ ⋆ ( λ ) λ ) = Tr ( Σ ( Σ + λ ⋆ ( λ ) I ) − 1 ) , (7)
and we have the one-to-one correspondence ω ( − λ ) = − λ ⋆ ( λ ) \omega(-\lambda)=-\lambda_{\star}(\lambda) ω ( − λ ) = − λ ⋆ ( λ ) . The deeper connection originates from the spectral analysis of random matrix theory under the finite-dimensional proportional regime. On the normalized covariance scale, − 1 / ω ( z / n ) -1/\omega(z/n) − 1/ ω ( z / n ) is the deterministic companion Stieltjes transform evaluated at z / n z/n z / n , as used in classical support analysis [71 , 8 ] . We next use ω ( z ) \omega(z) ω ( z ) to study the finite-n n n spectral law. For real t ∈ R t \in\mathbb{R} t ∈ R , we additionally let ω ( t ) \omega(t) ω ( t ) be the boundary value whenever lim η ↓ 0 ω ( t + i η ) \lim_{\eta\downarrow0}\omega(t+i\eta) lim η ↓ 0 ω ( t + i η ) is finite.
For the unscaled Gram matrix X X ∗ \mathbf{X}\mathbf{X}^{*} X X ∗ , its n n n eigenvalues, counted with multiplicity, consist of n σ ^ j , n n\widehat{\sigma}_{j,n} n σ j , n for j ∈ [ r n ] j \in[r_n] j ∈ [ r n ] and n − r n n-r_n n − r n zeros. Thus its empirical law and Stieltjes transform are
μ ^ n = 1 n ∑ j = 1 r n δ n σ ^ j , n + n − r n n δ 0 , s ^ n ( z ) : = ∫ μ ^ n ( d x ) x − z = 1 n Tr ( ( X X ∗ − z I n ) − 1 ) . \widehat{\mu}_n=\frac{1}{n}\sum_{j=1}^{r_n}\delta_{n\widehat{\sigma}_{j,n}}+\frac{n-r_n}{n}\delta_0,
\qquad
\widehat{s}_n(z):=\int\frac{\widehat{\mu}_n(dx)}{x-z}=\frac{1}{n}\operatorname{Tr}\left((\mathbf{X}\mathbf{X}^{*}-zI_n)^{-1}\right). μ n = n 1 j = 1 ∑ r n δ n σ j , n + n n − r n δ 0 , s n ( z ) := ∫ x − z μ n ( d x ) = n 1 Tr ( ( X X ∗ − z I n ) − 1 ) .
On the same scale, set s n ( z ) : = − 1 / ( n ω ( z / n ) ) s_n(z):=-1/(n\omega(z/n)) s n ( z ) := − 1/ ( nω ( z / n )) . Substituting ω ( z / n ) = − 1 / ( n s n ( z ) ) \omega(z/n)=-1/(ns_n(z)) ω ( z / n ) = − 1/ ( n s n ( z )) into Eq. (6) at z / n z/n z / n gives the self-consistent equation
s n ( z ) − 1 = − z + ∑ j ≥ 1 σ j 1 + n σ j s n ( z ) . (8) s_n(z)^{-1}=-z+\sum_{j\geq1}\frac{\sigma_j}{1+n\sigma_js_n(z)}. \tag*{(8)} s n ( z ) − 1 = − z + j ≥ 1 ∑ 1 + n σ j s n ( z ) σ j . (8)
By Lemma A.4 , this equation has a unique solution s n ( z ) ∈ C + s_n(z)\in\mathbb{C}_{+} s n ( z ) ∈ C + for z ∈ C + z\in\mathbb{C}_{+} z ∈ C + and a unique positive solution for real z < 0 z<0 z < 0 . Denote by μ n \mu_n μ n the unique probability measure on [ 0 , ∞ ) [0,\infty) [ 0 , ∞ ) with Stieltjes transform s n s_n s n , whose existence follows from the same lemma. We will establish that the empirical pair ( s ^ n , μ ^ n ) (\widehat{s}_n,\widehat{\mu}_n) ( s n , μ n ) “concentrates” around this deterministic pair ( s n , μ n ) (s_n,\mu_n) ( s n , μ n ) . The general strategy of recovering the Gram matrix’s spectral law from the effective regularization parameter λ ⋆ \lambda_{\star} λ ⋆ was sketched in the Ph.D. dissertation of Cheng [26 Chapter 4] , and in this paper, we establish rigorously in Theorem 1 with a finite-sample L p L^p L p bound for s ^ n ( z ) − s n ( z ) \widehat{s}_n(z)-s_n(z) s n ( z ) − s n ( z ) and transfers it to a bound on interval masses.
To approximate s ^ n ( z ) \widehat{s}_n(z) s n ( z ) , we introduce necessary notations to characterize stability of the population spectrum. When d < ∞ d<\infty d < ∞ , extend the population eigenvalues by zero beyond d d d . For p ≥ 1 p\geq1 p ≥ 1 , define the tail trace and tail effective rank parameters by
τ p ( k ) : = ∑ j > k σ j p and r p ( k ) : = ∑ j > k ( σ j σ k + 1 ) p . (9) \tau_p(k):=\sum_{j>k}\sigma_j^p
\quad\text{and}\quad
r_p(k):=\sum_{j>k}\left(\frac{\sigma_j}{\sigma_{k+1}}\right)^p. \tag*{(9)} τ p ( k ) := j > k ∑ σ j p and r p ( k ) := j > k ∑ ( σ k + 1 σ j ) p . (9)
We set r p ( k ) = 0 r_p(k)=0 r p ( k ) = 0 when σ k + 1 = 0 \sigma_{k+1}=0 σ k + 1 = 0 . Fix z = t + i η ∈ C + ∪ R < 0 z=t+i\eta\in\mathbb{C}_{+}\cup\mathbb{R}_{<0} z = t + i η ∈ C + ∪ R < 0 and write z − : = ( t ∧ 0 ) + i η z_-:=(t\wedge0)+i\eta z − := ( t ∧ 0 ) + i η , so that ∣ z − ∣ = dist ( z , R ≥ 0 ) |z_-|=\operatorname{dist}(z,\mathbb{R}_{\geq0}) ∣ z − ∣ = dist ( z , R ≥ 0 ) . There is a constant ρ = ρ ( C x ) ∈ ( 0 , 1 / 2 ) \rho=\rho(C_x)\in(0,1/2) ρ = ρ ( C x ) ∈ ( 0 , 1/2 ) , depending only on the feature constant in Assumption 1, for which the results below hold. For each integer 1 ≤ m ≤ d 1\leq m\leq d 1 ≤ m ≤ d , define
Λ n ( z , m ) : = ∣ z ∣ + n σ m + 1 + τ 1 ( m ) log ( n ) log ( n r 1 ( m ) ) ∣ z − ∣ . \Lambda_n(z,m):=\frac{|z|+n\sigma_{m+1}+\tau_1(m)\log(n)\log(nr_1(m))}{|z_-|}. Λ n ( z , m ) := ∣ z − ∣ ∣ z ∣ + n σ m + 1 + τ 1 ( m ) log ( n ) log ( n r 1 ( m )) .
When σ m + 1 = 0 \sigma_{m+1}=0 σ m + 1 = 0 , we set Λ n ( z , m ) = ∣ z ∣ / ∣ z − ∣ \Lambda_n(z,m)=|z|/|z_-| Λ n ( z , m ) = ∣ z ∣/∣ z − ∣ . The following assumption quantifies the required sample size n n n and the Stieltjes parameter scale z z z in C + ∪ R < 0 \mathbb{C}_{+}\cup\mathbb{R}_{<0} C + ∪ R < 0 .
Assumption 2 (Spectral stability). For ν , κ > 0 \nu,\kappa>0 ν , κ > 0 , define the spectral stability parameter
Λ n ( z ) : = min 1 ≤ m ≤ d ∧ ρ ( ⌊ n / 2 ⌋ − 2 ) Λ n ( z , m ) . (10) \Lambda_n(z) := \min_{1\le m\le d\wedge\rho(\lfloor n/2\rfloor-2)} \Lambda_n(z,m). \tag*{(10)} Λ n ( z ) := 1 ≤ m ≤ d ∧ ρ (⌊ n /2 ⌋ − 2 ) min Λ n ( z , m ) . (10) Assume n = Ω C x , ν , κ ( Λ n ( z ) 16 ) n=\Omega_{C_{x,\nu,\kappa}}(\Lambda_n(z)^{16}) n = Ω C x , ν , κ ( Λ n ( z ) 16 ) and ∣ z − ∣ ≥ n − ν |z_-|\ge n^{-\nu} ∣ z − ∣ ≥ n − ν .
We point out that the same m m m attains the minimum for every z z z , since the m m m -dependent part of the numerator of Λ n ( z , m ) \Lambda_n(z,m) Λ n ( z , m ) is independent of z z z . The selection of κ \kappa κ is primarily a technical parameter, as it specifies the admissible range 2 ≤ p ≤ κ log n 2\le p\le\kappa\log n 2 ≤ p ≤ κ log n on which the L p L^p L p bounds (presented below) for the approximation error are valid uniformly. For an interval E = [ a , b ] E=[a,b] E = [ a , b ] and 0 < h < ( b − a ) / 2 0<h<(b-a)/2 0 < h < ( b − a ) /2 , write E h + = [ a − h , b + h ] E_h^+=[a-h,b+h] E h + = [ a − h , b + h ] and E h − = [ a + h , b − h ] E_h^-=[a+h,b-h] E h − = [ a + h , b − h ] , we have the following deterministic equivalents for the spectrum. See its proof in Appendix B.1.
Theorem 1 (Deterministic equivalents of the spectra). Under Assumption 1, fix ν , κ > 0 \nu,\kappa>0 ν , κ > 0 .
(i) Let z ∈ C + ∪ R < 0 z\in\mathbb{C}_+\cup\mathbb{R}_{<0} z ∈ C + ∪ R < 0 satisfy Assumption 2. For 2 ≤ p ≤ κ log n 2\le p\le\kappa\log n 2 ≤ p ≤ κ log n , the Stieltjes transform s ^ n \widehat{s}_n s n of the e.s.d. of X X ∗ \mathbf{X}\mathbf{X}^* X X ∗ and its deterministic equivalent s n s_n s n satisfy
∥ s ^ n ( z ) − s n ( z ) ∥ L p = O C x , ν , κ ( p 2 Λ n ( z ) 17 n ∣ z ∣ ) . (11) \lVert\widehat{s}_n(z)-s_n(z)\rVert_{L^p}=O_{C_{x,\nu,\kappa}}\left(\frac{p^2\Lambda_n(z)^{17}}{n|z|}\right). \tag*{(11)} ∥ s n ( z ) − s n ( z ) ∥ L p = O C x , ν , κ ( n ∣ z ∣ p 2 Λ n ( z ) 17 ) . (11) (ii) Fix E = [ a , b ] E=[a,b] E = [ a , b ] , 0 < h < ( b − a ) / 2 0<h<(b-a)/2 0 < h < ( b − a ) /2 , and D , η > 0 D,\eta>0 D , η > 0 . Suppose κ log n ≥ 2 \kappa\log n\ge2 κ log n ≥ 2 and Assumption 2 holds at t + i η t+\mathrm{i}\eta t + i η for every t ∈ [ a − h , b + h ] t\in[a-h,b+h] t ∈ [ a − h , b + h ] . Then with probability 1 − O ( n − D ) 1-O(n^{-D}) 1 − O ( n − D ) ,
∣ μ ^ n ( E ) − μ n ( E ) ∣ ≤ O C x , ν , κ , D ( ( log n ) 2 n ∫ a − h b + h Λ n ( t + i η ) 17 ∣ t + i η ∣ d t + η h ) + μ n ( E h + ∖ E h − ) . (12) |\widehat{\mu}_n(E)-\mu_n(E)|\le O_{C_{x,\nu,\kappa,D}}\left(\frac{(\log n)^2}{n}\int_{a-h}^{b+h}\frac{\Lambda_n(t+\mathrm{i}\eta)^{17}}{|t+\mathrm{i}\eta|}\,\mathrm{d}t+\frac{\eta}{h}\right)+\mu_n(E_h^+\setminus E_h^-). \tag*{(12)} ∣ μ n ( E ) − μ n ( E ) ∣ ≤ O C x , ν , κ , D ( n ( log n ) 2 ∫ a − h b + h ∣ t + i η ∣ Λ n ( t + i η ) 17 d t + h η ) + μ n ( E h + ∖ E h − ) . (12)
This deterministic equivalent theorem provides a dimension-free, nonasymptotic, and quantitative description for the e.s.d. of the unnormalized Gram matrix X X ∗ \mathbf{X}\mathbf{X}^* X X ∗ . The user can adapt to specific scenarios, depending on the population Σ \boldsymbol{\Sigma} Σ and the sample size n n n , choosing an appropriate scale z = b n + a n ζ z=b_n+a_n\zeta z = b n + a n ζ and verifying Assumption 2 under that scale. Here ζ \zeta ζ ranges over a fixed compact subset of C + \mathbb{C}_+ C + , a n a_n a n is the appropriate “bulk width” and b n b_n b n is the appropriate “bulk shift”.
We summarize the choices to apply our results to the main examples in the paper in Table 1 . Under these rescalings, we plot the deterministic equivalent measure μ n \mu_n μ n in ζ \zeta ζ in Figure 2 against its asymptotic limit. We remark that the probability measures depend on n n n , and we further observe that, under the critical law, convergence to its limit is substantially slower in the critical case.
Table 1. Spectral scale and stability bounds for applying Theorem 1.
Figure 2. Densities of the deterministic laws μ n \mu_n μ n (solid) and their limits μ ⋆ \mu_\star μ ⋆ (dashed), in the bulk coordinates of Table 1. We show magnified views of each lower edge. μ n \mu_n μ n and μ ⋆ \mu_\star μ ⋆ are determined by Eqs. (8) and (12b), respectively.
In the next Proposition 2, we give quantitative nonasymptotic approximation bounds for the power and critical laws in Eq. (5) . In both cases the bulk width is a n = n σ n a_n=n\sigma_n a n = n σ n . In the critical case the shift b n = τ 1 ( n ) b_n=\tau_1(n) b n = τ 1 ( n ) is larger by a factor of order log n \log n log n .
Proposition 2 (Power and critical bulk laws). Under Assumption 1, take either spectrum in Eq. (5) and let z = z n ( ζ ) z=z_n(\zeta) z = z n ( ζ ) be the corresponding entry of Table 1 . Fix κ > 0 \kappa>0 κ > 0 and a compact K ⊂ C + K\subset\mathbb{C}_+ K ⊂ C + . For all sufficiently large n n n and 2 ≤ p ≤ κ log n 2\le p\le\kappa\log n 2 ≤ p ≤ κ log n ,
sup ζ ∈ K ∥ a n ( s ^ n ( z ) − s n ( z ) ) ∥ L p = p 2 n { O C x , ν , κ , K , α ( ( log n ) 34 ) , power law ; O C x , ν , κ , K , β ( ( log n ) 50 ) , critical law . (13) \sup_{\zeta\in K}\left\|a_n\left(\widehat{s}_n(z)-s_n(z)\right)\right\|_{L^p}
=\frac{p^2}{n}
\begin{cases}
O_{C_{x,\nu,\kappa,K,\alpha}}\left((\log n)^{34}\right), & \text{power law};\\
O_{C_{x,\nu,\kappa,K,\beta}}\left((\log n)^{50}\right), & \text{critical law}.
\end{cases} \tag*{(13)} ζ ∈ K sup ∥ a n ( s n ( z ) − s n ( z ) ) ∥ L p = n p 2 { O C x , ν , κ , K , α ( ( log n ) 34 ) , O C x , ν , κ , K , β ( ( log n ) 50 ) , power law ; critical law . (13) After the corresponding centering and scaling, μ n \mu_n μ n and μ ^ n \widehat{\mu}_n μ n converge weakly to a fixed probability measure μ ∗ \mu_* μ ∗ , almost surely for μ ^ n \widehat{\mu}_n μ n . Its Stieltjes transform s ( ζ ) s(\zeta) s ( ζ ) is the locally uniform limit of a n s n ( z ) a_ns_n(z) a n s n ( z )
on C + \mathbb{C}_+ C + and the unique solution in C + \mathbb{C}_+ C + of
s ( ζ ) − 1 = − ζ + { π s ( ζ ) − α / ( 1 + α ) ( 1 + α ) sin ( π / ( 1 + α ) ) , power law ; − log s ( ζ ) , critical law . (14) s(\zeta)^{-1}=-\zeta+\begin{cases}
\dfrac{\pi s(\zeta)^{-\alpha/(1+\alpha)}}{(1+\alpha)\sin(\pi/(1+\alpha))}, & \text{power law};\\
-\log s(\zeta), & \text{critical law}.
\end{cases} \tag*{(14)} s ( ζ ) − 1 = − ζ + ⎩ ⎨ ⎧ ( 1 + α ) sin ( π / ( 1 + α )) π s ( ζ ) − α / ( 1 + α ) , − log s ( ζ ) , power law ; critical law . (14)
Let κ ⋆ : = inf supp ( μ ⋆ ) \kappa_\star:=\inf\operatorname{supp}(\mu_\star) κ ⋆ := inf supp ( μ ⋆ ) . In each case, μ ⋆ \mu_\star μ ⋆ has support [ κ ⋆ , ∞ ) [\kappa_\star,\infty) [ κ ⋆ , ∞ ) and a bounded density ρ ⋆ \rho_\star ρ ⋆ satisfying
κ ⋆ = { 1 α ( π α ( 1 + α ) 2 sin ( π / ( 1 + α ) ) ) 1 + α , power law ; − 1 , critical law , ∥ ρ ⋆ ∥ ∞ ≤ { 1 π α κ ⋆ , power law ; 1 π , critical law . (15) \kappa_\star=\begin{cases}
\dfrac{1}{\alpha}\left(\dfrac{\pi\alpha}{(1+\alpha)^2\sin(\pi/(1+\alpha))}\right)^{1+\alpha}, & \text{power law};\\
-1, & \text{critical law},
\end{cases}
\qquad
\lVert\rho_\star\rVert_\infty\le\begin{cases}
\dfrac{1}{\pi\alpha\kappa_\star}, & \text{power law};\\
\dfrac{1}{\pi}, & \text{critical law}.
\end{cases} \tag*{(15)} κ ⋆ = ⎩ ⎨ ⎧ α 1 ( ( 1 + α ) 2 sin ( π / ( 1 + α )) π α ) 1 + α , − 1 , power law ; critical law , ∥ ρ ⋆ ∥ ∞ ≤ ⎩ ⎨ ⎧ π α κ ⋆ 1 , π 1 , power law ; critical law . (15)
The proof is in Appendix § 10 .§ 2 , which also contains the exact parameterizations of both limiting densities. We also point out that for the critical law, its limiting rescaled law μ ⋆ \mu_\star μ ⋆ is independent of the exponent β \beta β , unlike the power law decay, whose μ ⋆ \mu_\star μ ⋆ and κ ⋆ \kappa_\star κ ⋆ rely on α \alpha α .
Intriguingly, the limiting laws identified by Eq. (12b) are free stable laws characterized in [15 ] in free probability. Power law decay gives a scaled positive free stable law of index 1 / ( 1 + α ) 1/(1+\alpha) 1/ ( 1 + α ) , while critical decay gives the reflection of the exceptional free 1-stable law studied in [49 ] .
Remark 1 (Gram matrix geometry and benign overfitting). In the critical case, a n / b n ∼ β / log n → 0 a_n/b_n\sim\beta/\log n\to0 a n / b n ∼ β / log n → 0 , so Proposition 2 implies that the empirical spectral distribution of b n − 1 X X ∗ b_n^{-1}XX^* b n − 1 X X ∗ converges almost surely to a point mass δ 1 \delta_1 δ 1 , as in the classical fixed-dimensional asymptotics. For power-law decay, the bulk of a n − 1 X X ∗ a_n^{-1}XX^* a n − 1 X X ∗ retains a nondegenerate spread, as in proportional asymptotics. This gives a spectral interpretation of the effective-rank criterion for benign overfitting in Bartlett et al. [12 ] , Tsigler and Bartlett [76 ] , whose bounds show that, under their hypotheses, the prediction variance of ridgeless regression vanishes in the critical case and persists under power-law decay. We give a quantitative interpretation here by Gram matrix spectral geometry.
To wrap up the discussion on our first set of main results, we point the readers to two additional extensions of our bulk law results in Appendix § 10 .§ 3 . The first is Proposition 6 , which extends the spectral height from O ( 1 ) O(1) O ( 1 ) in bulk scale to Im ζ = n − δ \operatorname{Im}\zeta=n^{-\delta} Im ζ = n − δ for fixed 0 < δ < 1 / 17 0<\delta<1/17 0 < δ < 1/17 , providing its mesoscopic version (albeit, suboptimal as δ < 1 \delta<1 δ < 1 ). The second is Proposition 7 , where we recover the classical generalized Marchenko–Pastur law, showing the applicability and adaptability of our master Theorem 1.
We next turn to the second set of our results, which provides the deterministic equivalents of prediction risk of analytic spectral filters, with explicit formulas of effective variance and bias.
Main result II: Effective risk of spectral filters
Our second main result characterizes the conditional excess prediction error of spectral filters in linear regression. For the design X X X of § 2.1 and a fixed signal β ∈ H \beta\in\mathcal{H} β ∈ H , we observe
y = X β + ε where E [ ε ∣ X ] = 0 , E [ ε ε ∗ ∣ X ] = τ 2 I n . y=X\beta+\varepsilon\qquad\text{where}\qquad\mathbb{E}[\varepsilon\mid X]=0,\qquad\mathbb{E}[\varepsilon\varepsilon^{*}\mid X]=\tau^{2}I_n. y = X β + ε where E [ ε ∣ X ] = 0 , E [ ε ε ∗ ∣ X ] = τ 2 I n .
This includes the regime of centered independent noise ε i \varepsilon_i ε i for 1 ≤ i ≤ n 1\leq i\leq n 1 ≤ i ≤ n , with common variance τ 2 \tau^2 τ 2 and noise vector independent of X X X . We make the signal assumption β = Σ 1 / 2 θ \beta=\Sigma^{1/2}\theta β = Σ 1/2 θ with ∥ θ ∥ < ∞ \lVert\theta\rVert<\infty ∥ θ ∥ < ∞ , as is in [28 ] . For a given filter ϕ \phi ϕ and noise level τ \tau τ , we seek the deterministic approximation
( V n [ ϕ ; Σ ] , B n [ ϕ ; Σ ] ) ≈ ( V X [ ϕ ] , B X [ ϕ ] ) . (V_n[\phi;\Sigma],B_n[\phi;\Sigma])\approx(\mathcal{V}_X[\phi],\mathcal{B}_X[\phi]). ( V n [ ϕ ; Σ ] , B n [ ϕ ; Σ ]) ≈ ( V X [ ϕ ] , B X [ ϕ ]) .
for the two terms in Eq. (2) . Recall that with Σ ^ = X ∗ X / n \widehat{\Sigma}=X^{*}X/n Σ = X ∗ X / n , the variance and squared bias are
V X [ ϕ ] = τ 2 n Tr ( Σ Σ ^ ϕ ( Σ ^ ) 2 ) and B X [ ϕ ] = ∥ Σ 1 / 2 ψ ( Σ ^ ) β ∥ 2 . (16) \mathcal{V}_X[\phi]=\frac{\tau^2}{n}\operatorname{Tr}\left(\Sigma\widehat{\Sigma}\phi(\widehat{\Sigma})^2\right)
\qquad\text{and}\qquad
\mathcal{B}_X[\phi]=\left\lVert\Sigma^{1/2}\psi(\widehat{\Sigma})\beta\right\rVert^2. \tag*{(16)} V X [ ϕ ] = n τ 2 Tr ( Σ Σ ϕ ( Σ ) 2 ) and B X [ ϕ ] = Σ 1/2 ψ ( Σ ) β 2 . (16)
In literature, instance-wise risk bounds have been used to compare gradient descent with ridge regression [81 ] and to establish the optimality, up to constant factors, of optimally tuned PCR among monotone spectral filters [53 ] . Our deterministic equivalents could provide sharper estimates of risk differences and be useful in extending the comparisons. In this paper, we focus on analytic filters, for which a characterization via Cauchy’s formula is available, directly connecting prediction risk to their resolvents on C \mathbb{C} C .
To incorporate spectral filters ϕ \phi ϕ that may be undefined or unbounded at or near zero, such as ridgeless and negative ridge estimators, we choose η n ≥ 0 \eta_n\geq0 η n ≥ 0 such that [ n η n , ∞ ) = : I n [n\eta_n,\infty)=:I_n [ n η n , ∞ ) =: I n contains supp μ n \operatorname{supp}\mu_n supp μ n . On a hard edge event, that is σ ^ n ≥ η n \widehat{\sigma}_n\geq\eta_n σ n ≥ η n , the spectrum of X ∗ X X^{*}X X ∗ X lies in { 0 } ∪ I n \{0\}\cup I_n { 0 } ∪ I n . Let ψ ( x ) = 1 − x ϕ ( x ) \psi(x)=1-x\phi(x) ψ ( x ) = 1 − x ϕ ( x ) for the filter residual.
Assumption 3 (Analytic spectral filters). For supp μ n ⊆ I n = [ n η n , ∞ ) \operatorname{supp}\mu_n\subseteq I_n=[n\eta_n,\infty) supp μ n ⊆ I n = [ n η n , ∞ ) , let Ω \Omega Ω be a conjugation-symmetric open neighborhood of { 0 } ∪ I n \{0\}\cup I_n { 0 } ∪ I n , with piecewise smooth boundary Γ = ∂ Ω \Gamma=\partial\Omega Γ = ∂ Ω . The set Ω \Omega Ω may be disconnected. The restriction of the filter z ↦ Φ ( z / n ) z\mapsto\Phi(z/n) z ↦ Φ ( z / n ) to I n I_n I n extends holomorphically to a neighborhood of Ω ‾ \overline{\Omega} Ω , with Φ ( z / n ) ‾ = Φ ( z ‾ / n ) \overline{\Phi(z/n)}=\Phi(\overline{z}/n) Φ ( z / n ) = Φ ( z / n ) . For this extension, let ψ ( z / n ) = 1 − ( z / n ) Φ ( z / n ) \psi(z/n)=1-(z/n)\Phi(z/n) ψ ( z / n ) = 1 − ( z / n ) Φ ( z / n ) and assume that
lim ∣ z ∣ → ∞ ψ ( z / n ) = 0 for z ∈ Ω ‾ , and ∫ Γ ∣ ψ ( z / n ) ∣ ⋅ ∣ d z ∣ < ∞ when Γ is unbounded . \lim_{\lvert z\rvert\to\infty}\psi(z/n)=0\qquad\text{for }z\in\overline{\Omega},
\qquad\text{and}\qquad
\int_{\Gamma}\lvert\psi(z/n)\rvert\cdot\lvert dz\rvert<\infty\quad\text{when }\Gamma\text{ is unbounded}. ∣ z ∣ → ∞ lim ψ ( z / n ) = 0 for z ∈ Ω , and ∫ Γ ∣ ψ ( z / n )∣ ⋅ ∣ d z ∣ < ∞ when Γ is unbounded . We remark that under Assumption 3, we need only specify x ↦ ϕ ( x / n ) x\mapsto\phi(x/n) x ↦ ϕ ( x / n ) on I n I_n I n , which means ϕ \phi ϕ need only be defined (analytically) on [ η n , ∞ ) [\eta_n,\infty) [ η n , ∞ ) . For any extension as in the assumption, the residual ψ ( z / n ) = 1 − ( z / n ) ϕ ( z / n ) \psi(z/n)=1-(z/n)\phi(z/n) ψ ( z / n ) = 1 − ( z / n ) ϕ ( z / n ) satisfies ψ ( 0 ) = 1 \psi(0)=1 ψ ( 0 ) = 1 , since 0 ∈ Ω 0\in\Omega 0 ∈ Ω . Setting α ϕ ( d z ) = ψ ( z / n ) d z / ( 2 π i z ) \alpha_\phi(dz)=\psi(z/n)dz/(2\pi iz) α ϕ ( d z ) = ψ ( z / n ) d z / ( 2 π i z ) on the
positively oriented boundary, we can apply Cauchy’s formula at z = n x ∈ I n z = nx \in I_n z = n x ∈ I n to z ↦ ψ ( z / n ) z \mapsto\psi(z/n) z ↦ ψ ( z / n ) and z ↦ ϕ ( z / n ) z \mapsto\phi(z/n) z ↦ ϕ ( z / n ) , which yields the following representations, respectively:
ψ ( x ) = ∫ Γ z z − n x α ϕ ( d z ) and ϕ ( x ) = ∫ Γ 1 x − z / n α ϕ ( d z ) . (17) \psi(x) = \int_{\Gamma} \frac{z}{z-nx}\alpha_{\phi}(dz) \qquad\text{and}\qquad\phi(x) = \int_{\Gamma} \frac{1}{x-z/n}\alpha_{\phi}(dz). \tag*{(17)} ψ ( x ) = ∫ Γ z − n x z α ϕ ( d z ) and ϕ ( x ) = ∫ Γ x − z / n 1 α ϕ ( d z ) . (17)
Since ψ ( 0 ) = 1 \psi(0) = 1 ψ ( 0 ) = 1 , it also holds by Cauchy’s formula at z = 0 z = 0 z = 0 that ∫ Γ α ϕ ( d z ) = 1 \int_{\Gamma}\alpha_{\phi}(dz) = 1 ∫ Γ α ϕ ( d z ) = 1 . Writing ∣ α ϕ ∣ ( d z ) = ∣ ψ ( z / n ) ∣ / ( 2 π ∣ z ∣ ) ⋅ ∣ d z ∣ \lvert\alpha_{\phi}\rvert(dz) = \lvert\psi(z/n)\rvert/(2\pi\lvert z\rvert)\cdot\lvert dz\rvert ∣ α ϕ ∣ ( d z ) = ∣ ψ ( z / n )∣ / ( 2 π ∣ z ∣) ⋅ ∣ d z ∣ , we let ∥ α ϕ ∥ T V : = ∫ Γ ∣ α ϕ ∣ ( d z ) \|\alpha_{\phi}\|_{\mathrm{TV}} := \int_{\Gamma}\lvert\alpha_{\phi}\rvert(dz) ∥ α ϕ ∥ TV := ∫ Γ ∣ α ϕ ∣ ( d z ) . Applying the Cauchy representations in Eq. (17) to the two conditional risk formulas in Eq. (16) , we have the following contour-integral representations. Its proof is in Appendix C.1.
Lemma 2.1. Under Assumption 3, on { σ ^ n ≥ η n } \{\hat{\sigma}_n \ge\eta_n\} { σ ^ n ≥ η n } , the conditional variance and squared bias satisfy
V X [ ϕ ] = τ 2 ∬ Γ 2 z Tr ( Σ ( T n − z I ) − 1 ) − z ′ Tr ( Σ ( T n − z ′ I ) − 1 ) z − z ′ α ϕ ( d z ) α ϕ ( d z ′ ) , \mathcal{V}_X[\phi] = \tau^2 \iint_{\Gamma^2} \frac{z\operatorname{Tr}\left(\Sigma(T_n-zI)^{-1}\right)-z'\operatorname{Tr}\left(\Sigma(T_n-z'I)^{-1}\right)}{z-z'}\alpha_{\phi}(dz)\alpha_{\phi}(dz'), V X [ ϕ ] = τ 2 ∬ Γ 2 z − z ′ z Tr ( Σ ( T n − z I ) − 1 ) − z ′ Tr ( Σ ( T n − z ′ I ) − 1 ) α ϕ ( d z ) α ϕ ( d z ′ ) , B X [ ϕ ] = ∬ Γ 2 z z ′ β ∗ ( T n − z I ) − 1 Σ ( T n − z ′ I ) − 1 β α ϕ ( d z ) α ϕ ( d z ′ ) . \mathcal{B}_X[\phi] = \iint_{\Gamma^2} zz'\beta^*(T_n-zI)^{-1}\Sigma(T_n-z'I)^{-1}\beta\alpha_{\phi}(dz)\alpha_{\phi}(dz'). B X [ ϕ ] = ∬ Γ 2 z z ′ β ∗ ( T n − z I ) − 1 Σ ( T n − z ′ I ) − 1 β α ϕ ( d z ) α ϕ ( d z ′ ) .
The variance integrand at z ′ = z z' = z z ′ = z is understood to be the derivative of z Tr ( Σ ( T n − z I ) − 1 ) z\operatorname{Tr}\left(\Sigma(T_n-zI)^{-1}\right) z Tr ( Σ ( T n − z I ) − 1 ) in $z.
To see how this contour-integral can be useful, we first rescale the effective spectral parameter in Eq. (6) on the scale of the contour variables, i.e., ω ~ ( z ) : = n ω ( z / n ) = − 1 / s n ( z ) \widetilde{\omega}(z) := n\omega(z/n) = -1/s_n(z) ω ( z ) := nω ( z / n ) = − 1/ s n ( z ) for z ∉ I n z \notin I_n z ∈ / I n . Following the notation in [75 Section 2.1] , we write its first divided difference at z , z ′ ∉ I n z,z'\notin I_n z , z ′ ∈ / I n as
ω ~ [ z , z ′ ] : = ω ~ ( z ) − ω ~ ( z ′ ) z − z ′ ( z ≠ z ′ ) , and ω ~ [ z , z ] : = ω ~ ′ ( z ) = ω ′ ( z / n ) . \widetilde{\omega}[z,z'] := \frac{\widetilde{\omega}(z)-\widetilde{\omega}(z')}{z-z'} \quad(z\ne z'), \qquad\text{and}\qquad\widetilde{\omega}[z,z] := \widetilde{\omega}'(z)=\omega'(z/n). ω [ z , z ′ ] := z − z ′ ω ( z ) − ω ( z ′ ) ( z = z ′ ) , and ω [ z , z ] := ω ′ ( z ) = ω ′ ( z / n ) .
We now invoke the resolvent and mixed-resolvent deterministic equivalents in Theorem 4 and Theorem 5, whose details are in the later § 5 . Informally, we have
Tr ( Σ ( T n − z I ) − 1 ) ≃ Theorem 4 ω ~ ( z ) z − 1 , \operatorname{Tr}\left(\Sigma(T_n-zI)^{-1}\right) \mathrel{\mathop{\simeq}^{\text{Theorem 4}}} \frac{\widetilde{\omega}(z)}{z}-1, Tr ( Σ ( T n − z I ) − 1 ) ≃ Theorem 4 z ω ( z ) − 1 ,
β ∗ ( T n − z I ) − 1 Σ ( T n − z ′ I ) − 1 β ≃ Theorem 5 ω ~ [ z , z ′ ] z z ′ β ∗ ( I − n Σ ω ~ ( z ) ) − 1 Σ ( I − n Σ ω ~ ( z ′ ) ) − 1 β . \beta^*(T_n-zI)^{-1}\Sigma(T_n-z'I)^{-1}\beta\mathrel{\mathop{\simeq}^{\text{Theorem 5}}} \frac{\widetilde{\omega}[z,z']}{zz'}\beta^*\left(I-\frac{n\Sigma}{\widetilde{\omega}(z)}\right)^{-1}\Sigma\left(I-\frac{n\Sigma}{\widetilde{\omega}(z')}\right)^{-1}\beta. β ∗ ( T n − z I ) − 1 Σ ( T n − z ′ I ) − 1 β ≃ Theorem 5 z z ′ ω [ z , z ′ ] β ∗ ( I − ω ( z ) n Σ ) − 1 Σ ( I − ω ( z ′ ) n Σ ) − 1 β .
We can now immediately see that the effective risk formula R n [ ϕ ; Σ ] = V n [ ϕ ; Σ ] + B n [ ϕ ; Σ ] = V n [ ϕ ] + B n [ ϕ ] R_n[\phi;\Sigma] = V_n[\phi;\Sigma] + B_n[\phi;\Sigma] = V_n[\phi] + B_n[\phi] R n [ ϕ ; Σ ] = V n [ ϕ ; Σ ] + B n [ ϕ ; Σ ] = V n [ ϕ ] + B n [ ϕ ] follows by substitution into Lemma 2.1.
V n [ ϕ ] = τ 2 ∬ Γ 2 ( ω ~ [ z , z ′ ] − 1 ) α ϕ ( d z ) α ϕ ( d z ′ ) , (18) V_n[\phi] = \tau^2 \iint_{\Gamma^2}\left(\widetilde{\omega}[z,z']-1\right)\alpha_{\phi}(dz)\alpha_{\phi}(dz'), \tag*{(18)} V n [ ϕ ] = τ 2 ∬ Γ 2 ( ω [ z , z ′ ] − 1 ) α ϕ ( d z ) α ϕ ( d z ′ ) , (18)
B n [ Φ ] = ∬ Γ 2 ω ~ [ z , z ′ ] β ∗ ( I − n Σ ω ~ ( z ) ) − 1 Σ ( I − n Σ ω ~ ( z ′ ) ) − 1 β α ϕ ( d z ) α ϕ ( d z ′ ) . (19) B_n[\Phi] = \iint_{\Gamma^2} \widetilde{\omega}[z,z']\beta^* \left(I-\frac{n\Sigma}{\widetilde{\omega}(z)}\right)^{-1}\Sigma\left(I-\frac{n\Sigma}{\widetilde{\omega}(z')}\right)^{-1}\beta\,\alpha_\phi(dz)\alpha_\phi(dz'). \tag*{(19)} B n [ Φ ] = ∬ Γ 2 ω [ z , z ′ ] β ∗ ( I − ω ( z ) n Σ ) − 1 Σ ( I − ω ( z ′ ) n Σ ) − 1 β α ϕ ( d z ) α ϕ ( d z ′ ) . (19)
The rigorous replacement requires more caution. As Γ \Gamma Γ passes through the gap ( 0 , n η n ) (0,n\eta_n) ( 0 , n η n ) on the real axis, Theorems 4 and 5 do not apply directly there. Using a contour argument allows us to bound the log-norm of the error at z z z near the gap by its average over a small disk, by the submean inequality. The disk is large enough to apply Theorems 4 and 5 to control the function away from a narrow horizontal strip about the real axis that occupies only a small fraction of the disk. Using this strategy, the following theorem quantifies how closely these deterministic quantities approximate the conditional risks. See its proof in Appendix § 11 .§ 2 .
Theorem 2 (Effective risk of analytic spectral filters). Under Assumptions 1 and 3 , let κ > 0 \kappa> 0 κ > 0 and α ϕ \alpha_\phi α ϕ be the measure in Eq. (17) . Let Δ Γ : = Λ n ( − dist ( Γ , I n ) ) ∨ sup z ∈ Γ ( ∣ z ∣ / dist ( z , I n ) ) \Delta_\Gamma:= \Lambda_n(-\operatorname{dist}(\Gamma,I_n)) \vee\sup_{z\in\Gamma}(|z|/\operatorname{dist}(z,I_n)) Δ Γ := Λ n ( − dist ( Γ , I n )) ∨ sup z ∈ Γ ( ∣ z ∣/ dist ( z , I n )) denote the spectral stability of the contour Γ \Gamma Γ . Assume β = Σ 1 / 2 θ \beta= \Sigma^{1/2}\theta β = Σ 1/2 θ with ∥ θ ∥ < ∞ \lVert\theta\rVert< \infty ∥ θ ∥ < ∞ , and for some constant C Γ > 0 C_\Gamma> 0 C Γ > 0 ,
− log ( 1 ∧ dist ( Γ , I n ) ) ≤ C Γ log n and Δ Γ ≤ C Γ n 1 / 16 / log n . -\log(1 \wedge\operatorname{dist}(\Gamma,I_n)) \le C_\Gamma\log n
\qquad\text{and}\qquad
\Delta_\Gamma\le C_\Gamma n^{1/16}/\log n. − log ( 1 ∧ dist ( Γ , I n )) ≤ C Γ log n and Δ Γ ≤ C Γ n 1/16 / log n . Then for all large n n n and 2 ≤ p ≤ κ log n 2 \le p \le\kappa\log n 2 ≤ p ≤ κ log n , the effective variance and squared bias in Eq. (15) satisfy
∥ 1 { σ ^ n ≥ η n } ( V X [ ϕ ] − V n [ ϕ ] ) ∥ L p = O C X , κ , C Γ ( τ 2 p 2 Δ Γ 23 ∥ α ϕ ∥ T V n ∫ Γ ∣ z ∣ ∨ n η n dist ( z , I n ) ⋅ ∣ α ϕ ∣ ( d z ) ) , \left\lVert\mathbb{1}\{\widehat{\sigma}_n \ge\eta_n\}\left(\mathcal{V}_X[\phi]-V_n[\phi]\right)\right\rVert_{L^p}
=
O_{C_X,\kappa,C_\Gamma}\left(
\frac{\tau^2p^2\Delta_\Gamma^{23}\lVert\alpha_\phi\rVert_{\mathrm{TV}}}{n}
\int_\Gamma\frac{|z|\vee n\eta_n}{\operatorname{dist}(z,I_n)}\cdot|\alpha_\phi|(dz)
\right), ∥ 1 { σ n ≥ η n } ( V X [ ϕ ] − V n [ ϕ ] ) ∥ L p = O C X , κ , C Γ ( n τ 2 p 2 Δ Γ 23 ∥ α ϕ ∥ TV ∫ Γ dist ( z , I n ) ∣ z ∣ ∨ n η n ⋅ ∣ α ϕ ∣ ( d z ) ) , ∥ 1 { σ ^ n ≥ η n } ( B X [ ϕ ] − B n [ ϕ ] ) ∥ L p = O C X , κ , C Γ ( p 3 / 2 Δ Γ 26 ∥ θ ∥ 2 n 5 / 2 ( ∫ Γ ( ∣ z ∣ ∨ n η n ) ⋅ ∣ α ϕ ∣ ( d z ) ) 2 ) . \left\lVert\mathbb{1}\{\widehat{\sigma}_n \ge\eta_n\}\left(\mathcal{B}_X[\phi]-B_n[\phi]\right)\right\rVert_{L^p}
=
O_{C_X,\kappa,C_\Gamma}\left(
\frac{p^{3/2}\Delta_\Gamma^{26}\lVert\theta\rVert^2}{n^{5/2}}
\left(
\int_\Gamma(|z|\vee n\eta_n)\cdot|\alpha_\phi|(dz)
\right)^2
\right). ∥ 1 { σ n ≥ η n } ( B X [ ϕ ] − B n [ ϕ ] ) ∥ L p = O C X , κ , C Γ ( n 5/2 p 3/2 Δ Γ 26 ∥ θ ∥ 2 ( ∫ Γ ( ∣ z ∣ ∨ n η n ) ⋅ ∣ α ϕ ∣ ( d z ) ) 2 ) .
We now establish that the variance and squared bias of spectral filters concentrate around their deterministic equivalents in Eq. (15). However, it may occur to the reader that these quantities appear to depend on the contour Γ \Gamma Γ . On { σ ^ n ≥ η n } \{\widehat{\sigma}_n \ge\eta_n\} { σ n ≥ η n } , all possible singularities of the integrands for V X [ ϕ ] \mathcal{V}_X[\phi] V X [ ϕ ] , B X [ ϕ ] \mathcal{B}_X[\phi] B X [ ϕ ] and their equivalents V n [ ϕ ] V_n[\phi] V n [ ϕ ] , B n [ ϕ ] B_n[\phi] B n [ ϕ ] lie in { 0 } ∪ I n \{0\} \cup I_n { 0 } ∪ I n , which is enclosed by Γ \Gamma Γ . By Cauchy’s formula, we can deform the contour within the domain of analyticity of the filter, and the integrals depend only on the behavior of the integrands near the enclosed singularities. On the population side, for each σ j > 0 \sigma_j > 0 σ j > 0 (pole at spectral parameter ω \omega ω ), define
s ω , j ( z ) : = ω ( z ) z ( σ j − ω ( z ) ) = − 1 z ( 1 + n σ j s n ( n z ) ) , z ∈ C + . (20) s_{\omega,j}(z) := \frac{\omega(z)}{z(\sigma_j-\omega(z))} = -\frac{1}{z(1+n\sigma_j s_n(nz))}, \qquad z\in\mathbb{C}_+. \tag*{(20)} s ω , j ( z ) := z ( σ j − ω ( z )) ω ( z ) = − z ( 1 + n σ j s n ( n z )) 1 , z ∈ C + . (20)
As verified in Appendix § 11 .§ 3 (along with the proof of the following Proposition) using Lemma A.3, s ω , j s_{\omega,j} s ω , j is the Stieltjes transform of a probability measure μ ω , j \mu_{\omega,j} μ ω , j on { 0 } ∪ [ η n , ∞ ) \{0\}\cup[\eta_n,\infty) { 0 } ∪ [ η n , ∞ ) , so s ω , j ( z ) = ∫ ( x − z ) − 1 μ ω , j ( d x ) s_{\omega,j}(z)=\int(x-z)^{-1}\mu_{\omega,j}(dx) s ω , j ( z ) = ∫ ( x − z ) − 1 μ ω , j ( d x ) . Define the finite positive measure μ ω \mu_\omega μ ω by the weighted sum μ ω ( d x ) : = ( x / n ) ∑ j σ j μ ω , j ( d x ) \mu_\omega(dx):=(x/n)\sum_j\sigma_j\mu_{\omega,j}(dx) μ ω ( d x ) := ( x / n ) ∑ j σ j μ ω , j ( d x ) . In Figure 4 , we make plots of μ ω , j \mu_{\omega,j} μ ω , j and μ ω \mu_\omega μ ω for the power law and critical law, showing how the spectral mass shifts with the population eigenvalue index j j j . Finally, we establish the following representation of R n R_n R n .
Figure 4. Densities of μ ω , j \mu_{\omega,j} μ ω , j and their weighted measure μ ω \mu_{\omega} μ ω for σ i = i − 1.2 \sigma_i=i^{-1.2} σ i = i − 1.2 (power) and σ i = i − 1 ( 1 + log i ) − 1.2 \sigma_i=i^{-1}(1+\log i)^{-1.2} σ i = i − 1 ( 1 + log i ) − 1.2 (critical) for n ∈ { 8 , 64 } n\in\{8,64\} n ∈ { 8 , 64 } and j ∈ { 1 , n / 4 , n / 2 , n } j\in\{1,n/4,n/2,n\} j ∈ { 1 , n /4 , n /2 , n } . The bar plots show μ ω , j ( { 0 } ) \mu_{\omega,j}(\{0\}) μ ω , j ({ 0 }) .
Proposition 3. Under the setup of Theorem 2 , the effective risks in Eq. (15) satisfy
V n [ ϕ ] = τ 2 ∫ ϕ ( x ) 2 μ ω ( d x ) = τ 2 n ∑ j σ j ∫ x ϕ ( x ) 2 μ ω , j ( d x ) , (21) V_n[\phi] = \tau^2 \int\phi(x)^2\,\mu_\omega(dx)
= \frac{\tau^2}{n}\sum_j \sigma_j\int x\phi(x)^2\,\mu_{\omega,j}(dx), \tag*{(21)} V n [ ϕ ] = τ 2 ∫ ϕ ( x ) 2 μ ω ( d x ) = n τ 2 j ∑ σ j ∫ x ϕ ( x ) 2 μ ω , j ( d x ) , (21) B n [ ϕ ] = ∑ j σ j ∣ ⟨ ψ j , β ⟩ ∣ 2 { ( ∫ ψ ( x ) μ ω , j ( d x ) ) 2 + ∫ ( ∫ ψ ( x ) − ψ ( x ′ ) x ′ − x μ ω , j ( d x ) ) 2 μ ω ( d x ′ ) } . (22) B_n[\phi] = \sum_j \sigma_j|\langle\psi_j,\beta\rangle|^2
\left\{
\left(\int\psi(x)\,\mu_{\omega,j}(dx)\right)^2
+
\int\left(\int\frac{\psi(x)-\psi(x')}{x'-x}\,\mu_{\omega,j}(dx)\right)^2\mu_\omega(dx')
\right\}. \tag*{(22)} B n [ ϕ ] = j ∑ σ j ∣ ⟨ ψ j , β ⟩ ∣ 2 { ( ∫ ψ ( x ) μ ω , j ( d x ) ) 2 + ∫ ( ∫ x ′ − x ψ ( x ) − ψ ( x ′ ) μ ω , j ( d x ) ) 2 μ ω ( d x ′ ) } . (22)
At x = x ′ > 0 x=x'>0 x = x ′ > 0 , the divided difference in Eq. (17b) equals − ψ ′ ( x ′ ) -\psi'(x') − ψ ′ ( x ′ ) .
We note that Eqs. (15) and (17) provide equivalent deterministic risk expressions, each useful from a different viewpoint. The first contour representation captures the filter’s analytic regularity, and its holomorphic extension determines the explicit approximation error through its behavior on Γ \Gamma Γ in Theorem 2. On the other hand, Proposition 3 provides real-line evaluation formulas, without separate approximation guarantees while avoiding working on the complex plane.
In practice, both formulas can be useful for evaluating the deterministic risk; while they should theoretically yield the same result, one may be simpler to compute than the other. We will see this in the next section.
Applications
We will apply our main result II in § 2.3 to ridge-type filters and gradient flow under the power and critical laws in Eq. (5) . Recall that in those results, we work under the hard edge event { σ ^ n ≥ η n } \{\widehat{\sigma}_n\geq\eta_n\} { σ n ≥ η n } . This is true unconditionally setting η n = 0 \eta_n=0 η n = 0 , by p.s.d. of X X ∗ \mathbf{X}\mathbf{X}^{*} X X ∗ . Under the working Assumption 1 of quadratic concentration, we can indeed take without loss of generality that
n η n = Θ ( τ 1 ( n ) ) such that P ( σ ^ n ≥ η n ) ≥ 1 − 2 e − n , n\eta_n=\Theta(\tau_1(n)) \qquad\text{such that} \qquad\mathbb{P}(\widehat{\sigma}_n\geq\eta_n)\geq1-2e^{-n}, n η n = Θ ( τ 1 ( n )) such that P ( σ n ≥ η n ) ≥ 1 − 2 e − n ,
for all n = Ω ( 1 ) n=\Omega(1) n = Ω ( 1 ) (with dependence on the decay exponent and C x C_x C x suppressed). We defer discussion on the hard edge event to Lemma A.6. At the same time, for our power law and critical law example, this is at the correct scale, since inf supp μ n = Θ ( τ 1 ( n ) ) \inf\operatorname{supp}\mu_n=\Theta(\tau_1(n)) inf supp μ n = Θ ( τ 1 ( n )) (cf. Appendix § 12 .§ 3 ).
It is possible to strengthen and take n η n = ( 1 − c ) inf supp μ n n\eta_n=(1-c)\inf\operatorname{supp}\mu_n n η n = ( 1 − c ) inf supp μ n for arbitrarily small c > 0 c>0 c > 0 , or even vanishing c = c n c=c_n c = c n and still have σ ^ n ≥ η n \widehat{\sigma}_n\geq\eta_n σ n ≥ η n with overwhelmingly high probability, provided with stronger concentration of the features, e.g., finer Gaussian edge bounds (cf. [22 Section 3.3] ). For both of our applications, having n η n = Θ ( inf supp μ n ) n\eta_n=\Theta(\inf\operatorname{supp}\mu_n) n η n = Θ ( inf supp μ n ) is enough, so we leave this for future work.
Ridge-type filters
We begin with a unifying framework for ridge-type filters outlined in the introduction. Indeed, with
ψ k , λ ( x ) = ( λ x + λ ) k and ϕ k , λ ( x ) = 1 − ψ k , λ ( x ) x , (23) \psi_{k,\lambda}(x)=\left(\frac{\lambda}{x+\lambda}\right)^{k}
\qquad\text{and}\qquad
\phi_{k,\lambda}(x)=\frac{1-\psi_{k,\lambda}(x)}{x}, \tag*{(23)} ψ k , λ ( x ) = ( x + λ λ ) k and ϕ k , λ ( x ) = x 1 − ψ k , λ ( x ) , (23)
we can recover regular ridge (k = 1 , λ > 0 k = 1, \lambda> 0 k = 1 , λ > 0 ), ridgeless (λ = 0 \lambda= 0 λ = 0 ), negative ridge (k = 1 , λ < 0 k = 1, \lambda< 0 k = 1 , λ < 0 ) and their iterated version (k > 1 k > 1 k > 1 ). We can also view these filters as estimators from an iterative algorithm, fits a ridge correction to the current residual at each step, initialized at β ^ 0 , λ = 0 \widehat{\beta}_{0,\lambda} = 0 β 0 , λ = 0 and defined recursively for k = 1 , 2 , … k = 1, 2, \ldots k = 1 , 2 , … by
β ^ k , λ = β ^ k − 1 , λ + arg min b ∈ ran ( X ∗ ) { 1 2 n ∥ y − X β ^ k − 1 , λ − X b ∥ 2 + λ 2 ∥ b ∥ 2 } . \widehat{\beta}_{k,\lambda} = \widehat{\beta}_{k-1,\lambda} + \underset{b \in\operatorname{ran}(X^*)}{\arg\min}\left\{\frac{1}{2n}\left\|y - X\widehat{\beta}_{k-1,\lambda} - Xb\right\|^2 + \frac{\lambda}{2}\|b\|^2\right\}. β k , λ = β k − 1 , λ + b ∈ ran ( X ∗ ) arg min { 2 n 1 y − X β k − 1 , λ − X b 2 + 2 λ ∥ b ∥ 2 } .
Here ran ( X ∗ ) \operatorname{ran}(X^*) ran ( X ∗ ) denotes the range of X ∗ X^* X ∗ —it is important to have this constraint to allow for negative curvature in the regularization term, as the ambient space is in H \mathcal{H} H , which can have arbitrarily many descent directions when λ < 0 \lambda< 0 λ < 0 . For λ > 0 \lambda> 0 λ > 0 this is iterated Tikhonov regularization [58 ] .
To apply Theorem 2, it is instructive to construct the contour Γ \Gamma Γ in Assumption 3. The two-circle contour in Figure 5 suffices for all bounded λ ≥ − η n / 2 \lambda\ge-\eta_n/2 λ ≥ − η n /2 and fixed k k k . We highlight that on the component around zero, we separately define ϕ k , λ ( z / n ) = 0 \phi_{k,\lambda}(z/n) = 0 ϕ k , λ ( z / n ) = 0 and hence ψ k , λ ( z / n ) = 1 \psi_{k,\lambda}(z/n) = 1 ψ k , λ ( z / n ) = 1 , while the rational formula is used on the component containing I n I_n I n . Since these components are disjoint, the two definitions give a holomorphic function on their union (cf. [32 Chapter III] ). In the ridge-type filters case, it is advantageous to directly use the contour formula in Eq. (17) and use the Cauchy’s formula to obtain the exact forms of V n V_n V n and B n B_n B n . We write ω [ x , x ′ ] ≔ ω ~ [ n x , n x ′ ] \omega[x,x'] \coloneqq\widetilde{\omega}[nx,nx'] ω [ x , x ′ ] : = ω [ n x , n x ′ ] with the same divided-difference convention, summarize our results next, and defer its proof to Appendix § 12 .§ 1 .
Figure 5. Two-circle contours Γ \Gamma Γ for negative ridge, ridgeless, and positive ridge and its enclosed Ω \Omega Ω , which is the union of the inner disk around 0 0 0 and the exterior of the outer circle containing I n I_n I n . The dot marks − n λ -n\lambda − nλ .
Proposition 4 (Risk of ridge-type filters). Under Assumption 1 , take either spectrum in Eq. (5) and let β = Σ 1 / 2 θ \beta= \Sigma^{1/2}\theta β = Σ 1/2 θ , where ∥ θ ∥ < ∞ \|\theta\| < \infty ∥ θ ∥ < ∞ . Fix κ > 0 \kappa> 0 κ > 0 and an integer k ≥ 1 k \ge1 k ≥ 1 . There exists a deterministic η n = Θ ( τ 1 ( n ) / n ) \eta_n = \Theta(\tau_1(n)/n) η n = Θ ( τ 1 ( n ) / n ) satisfying P ( σ ^ n ≥ η n ) ≥ 1 − 2 e − n \mathbb{P}(\widehat{\sigma}_n \ge\eta_n) \ge1 - 2e^{-n} P ( σ n ≥ η n ) ≥ 1 − 2 e − n , such that the following holds for all sufficiently large n n n and λ ≥ − η n / 2 \lambda\ge-\eta_n/2 λ ≥ − η n /2 with λ = O ( 1 ) \lambda= O(1) λ = O ( 1 ) .
(i) For the iterated ridge filter ϕ k , λ \phi_{k,\lambda} ϕ k , λ , the effective variance and squared bias are
V n [ ϕ k , λ ] = τ 2 { ∑ m = 1 2 k − 1 ( k − ∣ k − m ∣ ) λ m − 1 ω ( m ) ( − λ ) m ! − 1 } , V_n[\phi_{k,\lambda}] = \tau^2\left\{\sum_{m=1}^{2k-1}\frac{(k-|k-m|)\lambda^{m-1}\omega^{(m)}(-\lambda)}{m!}-1\right\}, V n [ ϕ k , λ ] = τ 2 { m = 1 ∑ 2 k − 1 m ! ( k − ∣ k − m ∣ ) λ m − 1 ω ( m ) ( − λ ) − 1 } , B n [ ϕ k , λ ] = ∑ a , b = 0 k − 1 λ a + b a ! b ! d a + b d x a d x ′ b { ω [ x , x ′ ] β ∗ Σ ( I − Σ ω ( x ) ) − 1 ( I − Σ ω ( x ′ ) ) − 1 β } ∣ x = x ′ = − λ . B_n[\phi_{k,\lambda}] = \left.\sum_{a,b=0}^{k-1}\frac{\lambda^{a+b}}{a!b!}\frac{\mathrm{d}^{a+b}}{\mathrm{d}x^a\mathrm{d}x'^{b}}\left\{\omega[x,x']\beta^*\Sigma\left(I-\frac{\Sigma}{\omega(x)}\right)^{-1}\left(I-\frac{\Sigma}{\omega(x')}\right)^{-1}\beta\right\}\right|_{x=x'=-\lambda}. B n [ ϕ k , λ ] = a , b = 0 ∑ k − 1 a ! b ! λ a + b d x a d x ′ b d a + b { ω [ x , x ′ ] β ∗ Σ ( I − ω ( x ) Σ ) − 1 ( I − ω ( x ′ ) Σ ) − 1 β } x = x ′ = − λ .
We use O ~ \widetilde{O} O to extend the big-Oh notation by suppressing factors polynomial in log n \log n log n , and leave dependence on the decay exponent α \alpha α or β \beta β implicit. For 2 ≤ p ≤ κ log n 2 \le p \le\kappa\log n 2 ≤ p ≤ κ log n , we have
∥ 1 { σ ^ n ≥ η n } ( V X [ ϕ k , λ ] − V n [ ϕ k , λ ] ) ∥ L p = O ~ C X , κ , k ( τ 2 n ) , \left\|\mathbf{1}\{\widehat{\sigma}_n \ge\eta_n\}\left(\mathcal{V}_X[\phi_{k,\lambda}] - V_n[\phi_{k,\lambda}]\right)\right\|_{L^p} = \widetilde{O}_{C_X,\kappa,k}\left(\frac{\tau^2}{n}\right), ∥ 1 { σ n ≥ η n } ( V X [ ϕ k , λ ] − V n [ ϕ k , λ ] ) ∥ L p = O C X , κ , k ( n τ 2 ) ,
∥ 1 { σ ^ n ≥ η n } ( B X [ ϕ k , λ ] − B n [ ϕ k , λ ] ) ∥ L p = O ~ C x , κ , k ( ( λ + η n ) 2 ∥ θ ∥ 2 n ) . \left\|\mathbb{1}\{\widehat{\sigma}_n \ge\eta_n\}\left(\mathcal{B}_X[\phi_{k,\lambda}]-B_n[\phi_{k,\lambda}]\right)\right\|_{L^p}
=\widetilde{O}_{C_x,\kappa,k}\left(\frac{(\lambda+\eta_n)^2\|\theta\|^2}{\sqrt{n}}\right). ∥ 1 { σ n ≥ η n } ( B X [ ϕ k , λ ] − B n [ ϕ k , λ ] ) ∥ L p = O C x , κ , k ( n ( λ + η n ) 2 ∥ θ ∥ 2 ) .
(ii) For k = 1 k=1 k = 1 , let λ ⋆ = − ω ( − λ ) \lambda_\star=-\omega(-\lambda) λ ⋆ = − ω ( − λ ) be the effective regularization in Eq. (7) , and set J n ( λ ) = n − 1 Tr ( Σ 2 ( Σ + λ ⋆ I ) − 2 ) < 1 J_n(\lambda)=n^{-1}\operatorname{Tr}\left(\Sigma^2(\Sigma+\lambda_\star I)^{-2}\right)<1 J n ( λ ) = n − 1 Tr ( Σ 2 ( Σ + λ ⋆ I ) − 2 ) < 1 . Then the formulas in (i) reduce to
V n [ ϕ 1 , λ ] = τ 2 J n ( λ ) 1 − J n ( λ ) and B n [ ϕ 1 , λ ] = λ ⋆ 2 1 − J n ( λ ) ∥ Σ 1 / 2 ( Σ + λ ⋆ I ) − 1 β ∥ 2 . V_n[\phi_{1,\lambda}]=\tau^2\frac{J_n(\lambda)}{1-J_n(\lambda)}
\qquad\text{and}\qquad
B_n[\phi_{1,\lambda}]=\frac{\lambda_\star^2}{1-J_n(\lambda)}\left\|\Sigma^{1/2}(\Sigma+\lambda_\star I)^{-1}\beta\right\|^2. V n [ ϕ 1 , λ ] = τ 2 1 − J n ( λ ) J n ( λ ) and B n [ ϕ 1 , λ ] = 1 − J n ( λ ) λ ⋆ 2 Σ 1/2 ( Σ + λ ⋆ I ) − 1 β 2 .
For τ > 0 \tau>0 τ > 0 and θ ≠ 0 \theta\ne0 θ = 0 , dividing the bounds in (i) by these quantities yields
∥ 1 { σ ^ n ≥ η n } ( V X [ ϕ 1 , λ ] V n [ ϕ 1 , λ ] − 1 ) ∥ L p = O ~ C x , κ ( 1 n ⋅ 1 − J n ( λ ) J n ( λ ) ) , \left\|\mathbb{1}\{\widehat{\sigma}_n\ge\eta_n\}\left(\frac{\mathcal{V}_X[\phi_{1,\lambda}]}{V_n[\phi_{1,\lambda}]}-1\right)\right\|_{L^p}
=\widetilde{O}_{C_x,\kappa}\left(\frac{1}{n}\cdot\frac{1-J_n(\lambda)}{J_n(\lambda)}\right), 1 { σ n ≥ η n } ( V n [ ϕ 1 , λ ] V X [ ϕ 1 , λ ] − 1 ) L p = O C x , κ ( n 1 ⋅ J n ( λ ) 1 − J n ( λ ) ) ,
∥ 1 { σ ^ n ≥ η n } ( B X [ ϕ 1 , λ ] B n [ ϕ 1 , λ ] − 1 ) ∥ L p = O ~ C x , κ ( 1 n ⋅ ( λ + η n ) 2 ( 1 − J n ( λ ) ) ∥ θ ∥ 2 λ ⋆ 2 ∥ Σ ( Σ + λ ⋆ I ) − 1 θ ∥ 2 ) . \left\|\mathbb{1}\{\widehat{\sigma}_n\ge\eta_n\}\left(\frac{\mathcal{B}_X[\phi_{1,\lambda}]}{B_n[\phi_{1,\lambda}]}-1\right)\right\|_{L^p}
=\widetilde{O}_{C_x,\kappa}\left(\frac{1}{\sqrt{n}}\cdot\frac{(\lambda+\eta_n)^2(1-J_n(\lambda))\|\theta\|^2}{\lambda_\star^2\|\Sigma(\Sigma+\lambda_\star I)^{-1}\theta\|^2}\right). 1 { σ n ≥ η n } ( B n [ ϕ 1 , λ ] B X [ ϕ 1 , λ ] − 1 ) L p = O C x , κ ( n 1 ⋅ λ ⋆ 2 ∥Σ ( Σ + λ ⋆ I ) − 1 θ ∥ 2 ( λ + η n ) 2 ( 1 − J n ( λ )) ∥ θ ∥ 2 ) .
Remark 2. The relative (multiplicative) bounds in (ii) preserve the population spectrum and the source dependence explicitly, and can still vanish when these factors grow sufficiently slowly. For polynomially varying spectra, the ridgeless bias bounds of [28 ] do not, in general, yield a vanishing relative error [28 Remark 4.2] . We remove this barrier and recover optimal fluctuation rates O ~ ( n − 1 ) \widetilde{O}(n^{-1}) O ( n − 1 ) and O ~ ( n − 1 / 2 ) \widetilde{O}(n^{-1/2}) O ( n − 1/2 ) for the variance and squared bias. The proposition also covers negative ridge with − η n / 2 ≤ λ < 0 -\eta_n/2\le\lambda<0 − η n /2 ≤ λ < 0 and its iterated versions.
Gradient flow
We next consider gradient flow for the empirical squared loss in the Hilbert space H \mathcal{H} H initialized at zero:
d d t β ^ t = 1 n X ∗ ( y − X β ^ t ) and β ^ 0 = 0. \frac{\mathrm{d}}{\mathrm{d}t}\widehat{\beta}_t=\frac{1}{n}X^\ast(y-X\widehat{\beta}_t)
\qquad\text{and}\qquad
\widehat{\beta}_0=0. d t d β t = n 1 X ∗ ( y − X β t ) and β 0 = 0.
Solving this equation yields β ^ t = ϕ t ( Σ ^ ) X ∗ y / n \widehat{\beta}_t=\phi_t(\widehat{\Sigma})X^\ast y/n β t = ϕ t ( Σ ) X ∗ y / n , with filter and residual
ϕ t ( x ) = 1 − e − t x x and ψ t ( x ) = e − t x , \phi_t(x)=\frac{1-e^{-tx}}{x}
\qquad\text{and}\qquad
\psi_t(x)=e^{-tx}, ϕ t ( x ) = x 1 − e − t x and ψ t ( x ) = e − t x ,
where ϕ t ( 0 ) = t \phi_t(0)=t ϕ t ( 0 ) = t by continuity. The stopping time t t t controls which directions are fitted, since the residual is small when t x tx t x is large. This filter also arises from iterated ridge by taking λ = k / t \lambda=k/t λ = k / t and k → ∞ k\to\infty k → ∞ , since ( 1 + t x / k ) − k → e − t x (1+tx/k)^{-k}\to e^{-tx} ( 1 + t x / k ) − k → e − t x .
To invoke Theorem 2, we consider the two-ray contour indicated in Figure 3 , with the origin of the rays at − n / t -n/t − n / t . To evaluate the effective variance and squared bias, it is in fact convenient to use the real-line formulas in Proposition 3, which reduce to the Laplace transforms of the probability measures in Eq. (20) through
Figure 3. Example of Ω \Omega Ω and the contour Γ \Gamma Γ in Assumption 3. The illustrated two-ray contour Γ \Gamma Γ is chosen for evaluating the gradient flow estimator’s effective terms V n V_n V n and B n B_n B n in § 3 .
L ω , j ( t ) : = ∫ [ 0 , ∞ ) e − t x μ ω , j ( d x ) , t ≥ 0. \mathcal{L}_{\omega,j}(t):=\int_{[0,\infty)}e^{-tx}\mu_{\omega,j}(\mathrm{d}x),
\qquad t\ge0. L ω , j ( t ) := ∫ [ 0 , ∞ ) e − t x μ ω , j ( d x ) , t ≥ 0.
We summarize the resulting formulas and approximation bounds next, and defer the proof to Appendix § 12 .§ 2 .
Proposition 5 (Risk of gradient flow). Under the same setup of Proposition 4, for all sufficiently large n n n and t = t n > 0 t=t_n>0 t = t n > 0 , the effective variance and squared bias of ϕ t \phi_t ϕ t are
V n [ ϕ t ] = τ 2 n ∑ j σ j ∫ 0 t ( L ω , j ( r ) − L ω , j ( r + t ) ) d r , V_n[\phi_t]=\frac{\tau^2}{n}\sum_j\sigma_j\int_0^t\left(\mathcal{L}_{\omega,j}(r)-\mathcal{L}_{\omega,j}(r+t)\right)\,\mathrm{d}r, V n [ ϕ t ] = n τ 2 j ∑ σ j ∫ 0 t ( L ω , j ( r ) − L ω , j ( r + t ) ) d r , B n [ ϕ t ] = ∑ j σ j ∣ ⟨ u j , β ⟩ ∣ 2 { L ω , j ( t ) 2 + ∫ [ 0 , ∞ ) ( ∫ 0 t e − ( t − r ) x L ω , j ( r ) d r ) 2 μ ω ( d x ) } . B_n[\phi_t]=\sum_j\sigma_j\lvert\langle u_j,\beta\rangle\rvert^2\left\{\mathcal{L}_{\omega,j}(t)^2+\int_{[0,\infty)}\left(\int_0^t e^{-(t-r)x}\mathcal{L}_{\omega,j}(r)\,\mathrm{d}r\right)^2\mu_\omega(\mathrm{d}x)\right\}. B n [ ϕ t ] = j ∑ σ j ∣⟨ u j , β ⟩ ∣ 2 { L ω , j ( t ) 2 + ∫ [ 0 , ∞ ) ( ∫ 0 t e − ( t − r ) x L ω , j ( r ) d r ) 2 μ ω ( d x ) } . For 2 ≤ p ≤ κ log n 2\le p\le\kappa\log n 2 ≤ p ≤ κ log n , using the O ~ \widetilde{O} O convention of Proposition 4, we have the following bounds with implied constants independent of t t t :
∥ 1 { σ ^ n ≥ η n } ( V X [ ϕ t ] − V n [ ϕ t ] ) ∥ L p = O ~ C X , κ ( τ 2 n ) , \left\|\mathbf{1}\{\widehat{\sigma}_n\ge\eta_n\}\left(\mathcal{V}_X[\phi_t]-V_n[\phi_t]\right)\right\|_{L^p}
=\widetilde{O}_{C_X,\kappa}\left(\frac{\tau^2}{n}\right), ∥ 1 { σ n ≥ η n } ( V X [ ϕ t ] − V n [ ϕ t ] ) ∥ L p = O C X , κ ( n τ 2 ) , ∥ 1 { σ ^ n ≥ η n } ( B X [ ϕ t ] − B n [ ϕ t ] ) ∥ L p = O ~ C X , κ ( ( η n + t − 1 ) 2 ∥ θ ∥ 2 n ) . \left\|\mathbf{1}\{\widehat{\sigma}_n\ge\eta_n\}\left(\mathcal{B}_X[\phi_t]-B_n[\phi_t]\right)\right\|_{L^p}
=\widetilde{O}_{C_X,\kappa}\left(\frac{(\eta_n+t^{-1})^2\|\theta\|^2}{\sqrt{n}}\right). ∥ 1 { σ n ≥ η n } ( B X [ ϕ t ] − B n [ ϕ t ] ) ∥ L p = O C X , κ ( n ( η n + t − 1 ) 2 ∥ θ ∥ 2 ) .
The reader may notice that our bias approximation bounds can deteriorate near initialization t → 0 + t\to0^+ t → 0 + or under strong ridge regularization λ → ∞ \lambda\to\infty λ → ∞ . This is because our bounds are adapted to bulk scales. Indeed, strong regularization in those cases are easier regimes, and can be controlled separately by standard concentration arguments. We focus on growing stopping times t → ∞ t\to\infty t → ∞ up to the bulk scale t = O ( η n − 1 ) t=O(\eta_n^{-1}) t = O ( η n − 1 ) .
Remark 3. Recent work on early stopping and gradient descent with Hilbert-space features uses concentration and nonasymptotic risk bounds to compare with ridge regression [81 ] , while existing asymptotic risk formulas exclude the power-law bulk scale [56 , 55 ] . For both regimes considered here, the bulk stopping scale is t = Θ ( η n − 1 ) = Θ ( n / τ 1 ( n ) ) t=\Theta(\eta_n^{-1})=\Theta(n/\tau_1(n)) t = Θ ( η n − 1 ) = Θ ( n / τ 1 ( n )) . Proposition 5 supplies explicit deterministic equivalents at and beyond this scale, with quantitative errors at the n − 1 n^{-1} n − 1 and n − 1 / 2 n^{-1/2} n − 1/2 fluctuation rates after the displayed scale factors.
An interesting immediate observation is that these formulas make the source-dependent variance-bias trade-off explicit through ∣ ⟨ u j , β ⟩ ∣ 2 \lvert\langle u_j,\beta\rangle\rvert^2 ∣⟨ u j , β ⟩ ∣ 2 . Indeed, letting ( e − x ⋅ ∗ L ω , j ) ( t ) = ∫ 0 t e − ( t − r ) x L ω , j ( r ) d r (e^{-x\cdot}*\mathcal{L}_{\omega,j})(t)=\int_0^t e^{-(t-r)x}\mathcal{L}_{\omega,j}(r)\,\mathrm{d}r ( e − x ⋅ ∗ L ω , j ) ( t ) = ∫ 0 t e − ( t − r ) x L ω , j ( r ) d r and differentiating the formulas in Proposition 5 yields
d d t V n [ ϕ t ] = 2 τ 2 n ∑ j σ j ( L ω , j ( t ) − L ω , j ( 2 t ) ) , \frac{\mathrm{d}}{\mathrm{d}t}V_n[\phi_t]=\frac{2\tau^2}{n}\sum_j\sigma_j\left(\mathcal{L}_{\omega,j}(t)-\mathcal{L}_{\omega,j}(2t)\right), d t d V n [ ϕ t ] = n 2 τ 2 j ∑ σ j ( L ω , j ( t ) − L ω , j ( 2 t ) ) ,
d d t B n [ ϕ t ] = 2 ∑ j σ j ∣ ⟨ u j , β ⟩ ∣ 2 { L ω , j L ω , j ′ + ∫ [ 0 , ∞ ) ( e − x ⋅ ∗ L ω , j ) ( L ω , j − x ( e − x ⋅ ∗ L ω , j ) ) μ ω ( d x ) } . \frac{\mathrm{d}}{\mathrm{d}t}B_n[\phi_t]=2\sum_j\sigma_j\lvert\langle u_j,\beta\rangle\rvert^2\left\{\mathcal{L}_{\omega,j}\mathcal{L}'_{\omega,j}+\int_{[0,\infty)}(e^{-x\cdot}*\mathcal{L}_{\omega,j})(\mathcal{L}_{\omega,j}-x(e^{-x\cdot}*\mathcal{L}_{\omega,j}))\mu_\omega(\mathrm{d}x)\right\}. d t d B n [ ϕ t ] = 2 j ∑ σ j ∣⟨ u j , β ⟩ ∣ 2 { L ω , j L ω , j ′ + ∫ [ 0 , ∞ ) ( e − x ⋅ ∗ L ω , j ) ( L ω , j − x ( e − x ⋅ ∗ L ω , j )) μ ω ( d x ) } .
In the second line, all transforms and convolutions are evaluated at t t t , with
L ω , j ′ ( t ) = − ∫ [ 0 , ∞ ) x e − t x μ ω , j ( d x ) \mathcal{L}'_{\omega,j}(t)=-\int_{[0,\infty)}xe^{-tx}\mu_{\omega,j}(\mathrm{d}x) L ω , j ′ ( t ) = − ∫ [ 0 , ∞ ) x e − t x μ ω , j ( d x )
This suggests an explicit stationary condition for the optimal (early) stopping time t o p t > 0 t_{\mathrm{opt}}>0 t opt > 0 , upon its existence, satisfying
d d t V n [ ϕ t ] ∣ t = t o p t = − d d t B n [ ϕ t ] ∣ t = t o p t . \left.\frac{\mathrm{d}}{\mathrm{d}t}V_n[\phi_t]\right|_{t=t_{\mathrm{opt}}}
=-\left.\frac{\mathrm{d}}{\mathrm{d}t}B_n[\phi_t]\right|_{t=t_{\mathrm{opt}}}. d t d V n [ ϕ t ] t = t opt = − d t d B n [ ϕ t ] t = t opt .
This could yield further insights into optimal stopping and how it relates to the source distribution over high and low frequency features. We defer the analysis of this trade-off and the resulting derivation of stopping rules to future work.
Numerical experiments
We complement our theoretical results with three numerical experiments: (i) empirical spectral laws, (ii) ridge-type filters, and (iii) gradient flow. We compare the deterministic predictions with simulations under the power and critical laws of § 2.1 . Codes are in the public repo.
Throughout these experiments, we use Gaussian covariates with diagonal covariance, taking decay exponents 0.2 and 0.05. We truncate the spectra at d = 2 24 d = 2^{24} d = 2 24 for computational efficiency and compute the predictions using the same truncated covariance. For the regression experiments, we take a Poisson source—to vary the source weights across population eigendirections—β = Σ 1 / 2 θ \beta= \Sigma^{1/2}\theta β = Σ 1/2 θ , where
θ ∈ { θ q : q > 0 } , ( θ q ) j ∝ q j − 1 ( j − 1 ) ! , j = 1 , … , d . \theta\in\{\theta_q : q > 0\}, \qquad(\theta_q)_j \propto\frac{q^{j-1}}{(j-1)!}, \qquad j = 1,\ldots,d. θ ∈ { θ q : q > 0 } , ( θ q ) j ∝ ( j − 1 )! q j − 1 , j = 1 , … , d .
θ q \theta_q θ q has coordinates proportional to the Poisson probabilities and we normalize so that ∥ θ q ∥ 2 = 1 \lVert\theta_q\rVert_2 = 1 ∥ θ q ∥ 2 = 1 . The details of the sample sizes, filter parameters, and source parameter q q q are in each experiment below.
Empirical and predicted spectral laws. We first test our main result I in Theorem 1, comparing the empirical spectral laws μ ^ n \widehat{\mu}_n μ n with their deterministic predictions μ n \mu_n μ n under the four covariance settings above. For the purpose of nonasymptotic comparisons, we choose relatively small n ∈ { 2 , 8 , 32 , 128 } n \in\{2, 8, 32, 128\} n ∈ { 2 , 8 , 32 , 128 } . Following Proposition 2, we rescale the eigenvalues by a n = n σ n a_n = n\sigma_n a n = n σ n in the power law case and center by b n = τ 1 ( n ) b_n = \tau_1(n) b n = τ 1 ( n ) from Eq. (9) before rescaling by a n a_n a n in the critical law case. Figure 6 presents the empirical/predicted PDFs and CDFs.
Figure 6. Empirical vs. predicted spectral laws under power and critical decay regimes with n ∈ { 2 , 8 , 32 , 128 } n \in\{2, 8, 32, 128\} n ∈ { 2 , 8 , 32 , 128 } . We average over 2,000 independent Gaussian designs. Solid curves show the theoretical predictions, and error bars give pointwise 95% Monte Carlo intervals computed across matrices. For eigenvalues x x x of X X ∗ XX^{*} X X ∗ , the horizontal coordinate is x / a n x/a_n x / a n for the power law and ( x − b n ) / a n (x-b_n)/a_n ( x − b n ) / a n for the critical law. We use shifted logarithmic x-axes for better visualization.
We observe that in four settings, the empirical histograms and CDFs agree increasingly well with the finite-sample predictions as n n n grows. The predictions capture both the bulk shape and its dependence on n n n . The visible discrepancies occur only near the lower edge when n n n is small.
Ridge-type filters. We next examine our main result II, comparing empirical results with the theoretical predictions in Eq. (15) for ridge-type filters first. We use the Poisson source q = 16 q = 16 q = 16 , noise standard deviation τ = 0.01 \tau= 0.01 τ = 0.01 , and sample sizes n ∈ { 16 , 32 , 64 , 128 , 192 , 256 , 384 , 512 } n \in\{16, 32, 64, 128, 192, 256, 384, 512\} n ∈ { 16 , 32 , 64 , 128 , 192 , 256 , 384 , 512 } . We take λ ∈ { − 0.6 , 0 , 0.5 , 1 , 1.5 , 2 } κ n \lambda\in\{-0.6, 0, 0.5, 1, 1.5, 2\}\kappa_n λ ∈ { − 0.6 , 0 , 0.5 , 1 , 1.5 , 2 } κ n , where κ n : = n − 1 inf supp μ n \kappa_n := n^{-1} \inf\operatorname{supp}\mu_n κ n := n − 1 inf supp μ n is the predicted lower spectral edge of X X ∗ / n \mathbf{X}\mathbf{X}^{*}/n X X ∗ / n for the truncated covariance.
Figure 7a shows the unnormalized results for k = 1 k = 1 k = 1 . Figures 7b and 7c compare these quantities relative to ridgeless regression for k = 1 k = 1 k = 1 and k = 2 k = 2 k = 2 , respectively. The normalized predictions are V n / V 0 V_n/V_0 V n / V 0 , B n / B 0 B_n/B_0 B n / B 0 , and R n / R 0 R_n/R_0 R n / R 0 , using (theoretical or empirical) ridgeless baselines at the same n n n and Σ \boldsymbol{\Sigma} Σ . The exact effective risk formulas for k = 2 k = 2 k = 2 , and additional corresponding results for a different shape source q = 24 q = 24 q = 24 are in Appendix § 15 .§ 1 .
Figure 7a. Ridge-type filters with q = 16 q = 16 q = 16 and k = 1 k = 1 k = 1 . Rows show V n V_n V n , B n B_n B n , and R n R_n R n . Solid curves show the theoretical predictions. We show markers for the simulated results, with error bars by pointwise 95% Monte Carlo intervals over 400 designs.
From the plots, we see the empirical means closely match the theoretical predictions, with error bars that are nearly imperceptible. It is important to note that, unlike standard ridge regression—where V n V_n V n decreases monotonically with λ \lambda λ —iterated ridge does not generally exhibit monotonic behavior, as shown in Figure 7c. We also observe there is no universal dominance of negative, positive or ridgeless choices for ridge-type filters. The optimal choice depends on the spectrum, sample size and also the source shape, parameterized by Σ \boldsymbol{\Sigma} Σ , n n n , q q q respectively in our experiments.
Figure 7b. Ridge-type filters with q = 16 q = 16 q = 16 and k = 1 k = 1 k = 1 , normalized by the respective ridgeless baselines.
Figure 7c. Iterated ridge with q = 16 q = 16 q = 16 and k = 2 k = 2 k = 2 , normalized by the respective ridgeless baselines.
Gradient flow. In the last experiment, we investigate gradient flow risks with the theoretical predictions in Proposition 5. We use q ∈ { 16 , 24 } q \in\{16,24\} q ∈ { 16 , 24 } , τ = 0.01 \tau= 0.01 τ = 0.01 , and n ∈ { 64 , 128 , 192 , 256 , 384 , 512 } n \in\{64,128,192,256,384,512\} n ∈ { 64 , 128 , 192 , 256 , 384 , 512 } . Figure 8 compares empirical and predicted risks, normalized by their respective ridgeless baselines (t = ∞ t = \infty t = ∞ ), against t ~ = t τ 1 ( n ) / n ∈ [ 0 , 4 ] \tilde{t} = t\tau_1(n)/n \in[0,4] t ~ = t τ 1 ( n ) / n ∈ [ 0 , 4 ] . We also make the plot with the predicted optimal stopping times from the stationary condition derived in § 3.2 .
Figure 8. Gradient-flow risk relative to ridgeless regression. Surfaces show empirical means over 400 designs, meshes show theoretical predictions, and curves show predicted finite stopping optima. Envelopes show pointwise 10% to 90% quantiles across designs. See Appendix § 15 .§ 2 for details.
We once again see our theoretical predictions closely match the empirical mean risks across the four spectral settings and both sources. However, the benefit of early stopping is more nuanced, depending on the spectrum, sample size, and source distribution. For the power law with α = 0.05 \alpha= 0.05 α = 0.05 , our numerical optimization favors the ridgeless limit throughout the displayed n n n range and therefore no optimal curve is shown—however, the numerical evaluation of R n R_n R n suggests an apparent phase transition at larger n n n . This transition lies beyond the empirical range n ≤ 512 n \le512 n ≤ 512 in Figure 8 .
It is possible that there exists a finite n 0 n_0 n 0 such that t o p t = ∞ t_{\mathrm{opt}} = \infty t opt = ∞ when n < n 0 n < n_0 n < n 0 , and t o p t < ∞ t_{\mathrm{opt}} < \infty t opt < ∞ when n > n 0 n > n_0 n > n 0 . However, the discontinuities in the t o p t t_{\mathrm{opt}} t opt curve itself do not always imply such transition point. We give further numerical illustrations of this complicated behavior in Appendix § 15 .§ 2 .
The deterministic self-consistent interpolation
We now introduce the interpolation framework used throughout the paper. To this end, we first specify the appropriate domains on which we can analyze the matrix trace functionals of interest. Recall z = t + i η z = t + i\eta z = t + i η and z − = ( t ∧ 0 ) + i η z_- = (t \wedge0) + i\eta z − = ( t ∧ 0 ) + i η from § 2.2 , with ∣ z − ∣ = dist ( z , R ≥ 0 ) |z_-| = \operatorname{dist}(z,\mathbb{R}_{\ge0}) ∣ z − ∣ = dist ( z , R ≥ 0 ) . We write additionally w = u + i v w = u + iv w = u + i v with u , v ∈ R u,v \in\mathbb{R} u , v ∈ R . Denote by t + : = t ∨ 0 t_+ := t \vee0 t + := t ∨ 0 and t − : = ( − t ) ∨ 0 t_- := (-t) \vee0 t − := ( − t ) ∨ 0 , so z − = z − t + = − t − + i η z_- = z - t_+ = -t_- + i\eta z − = z − t + = − t − + i η .
Throughout this section, z z z denotes the spectral parameter on the unnormalized scale of T k T_k T k , and thus z / n z/n z / n is the corresponding spectral parameter for the normalized sample covariance operator. Let their natural domain be D : = { ( z , w ) : η > 0 , v ≥ 0 } \mathcal{D} := \{(z,w): \eta> 0, v \ge0\} D := {( z , w ) : η > 0 , v ≥ 0 } . We extend their domain to include subsets of the real line by
D k : = { ( z , w ) : η , v ≥ 0 , 0 ∉ Spec ( T k − w Σ − z I ) } . \mathcal{D}_k := \{(z,w): \eta, v \ge0, 0 \notin\operatorname{Spec}(T_k-w\Sigma-zI)\}. D k := {( z , w ) : η , v ≥ 0 , 0 ∈ / Spec ( T k − w Σ − z I )} .
The domain D k \mathcal{D}_k D k depends on the sample. Since Im ( T k − w Σ − z I ) = − v Σ − η I ⪯ − η I \operatorname{Im}(\mathbf{T}_k-w\mathbf{\Sigma}-zI)=-v\mathbf{\Sigma}-\eta I\preceq-\eta I Im ( T k − w Σ − z I ) = − v Σ − η I ⪯ − η I , we have D ⊂ D k \mathcal{D}\subset\mathcal{D}_k D ⊂ D k for all realizations T k \mathbf{T}_k T k . For ( z , w ) ∈ D k (z,w)\in\mathcal{D}_k ( z , w ) ∈ D k and Q ∈ L ( H ) Q\in\mathcal{L}(\mathcal{H}) Q ∈ L ( H ) , define the relative-resolvent trace H k Q ( z , w ) H_k^Q(z,w) H k Q ( z , w ) ; and for ( z , w ) ∈ D k − 1 (z,w)\in\mathcal{D}_{k-1} ( z , w ) ∈ D k − 1 , the leave-one-out trace G k Q ( z , w ) G_k^Q(z,w) G k Q ( z , w ) by
R k ( z , w ) ≔ ( T k − w Σ − z I ) − 1 , (24) R_k(z,w)\coloneqq(\mathbf{T}_k-w\mathbf{\Sigma}-zI)^{-1}, \tag*{(24)} R k ( z , w ) : = ( T k − w Σ − z I ) − 1 , (24)
H k Q ( z , w ) ≔ Tr ( Q ( T k − w Σ ) R k ( z , w ) ) , G k Q ( z , w ) ≔ Tr ( Σ 1 2 Q Σ 1 2 ( T k − 1 − w Σ − z I ) − 1 ) (25) H_k^Q(z,w)\coloneqq\operatorname{Tr}\left(Q(\mathbf{T}_k-w\mathbf{\Sigma})R_k(z,w)\right),\qquad G_k^Q(z,w)\coloneqq\operatorname{Tr}\left(\mathbf{\Sigma}^{\frac{1}{2}}Q\mathbf{\Sigma}^{\frac{1}{2}}(\mathbf{T}_{k-1}-w\mathbf{\Sigma}-zI)^{-1}\right) \tag*{(25)} H k Q ( z , w ) : = Tr ( Q ( T k − w Σ ) R k ( z , w ) ) , G k Q ( z , w ) : = Tr ( Σ 2 1 Q Σ 2 1 ( T k − 1 − w Σ − z I ) − 1 ) (25)
The traces in Eq. (23) are finite because Σ \mathbf{\Sigma} Σ and T k \mathbf{T}_k T k are trace-class, and we suppress their dependence in Q Q Q when Q = I Q=I Q = I by writing H k ≔ H k I H_k\coloneqq H_k^I H k : = H k I and G k ≔ G k I G_k\coloneqq G_k^I G k : = G k I . We explain the terms relative-resolvent and leave-one-out traces by the following identities when z ≠ 0 z\ne0 z = 0 :
(Relative-resolvent trace) H k ( z , w ) = z Tr ( R k ( z , w ) − ( − z I ) − 1 ) H_k(z,w)=z\operatorname{Tr}\left(R_k(z,w)-(-zI)^{-1}\right) H k ( z , w ) = z Tr ( R k ( z , w ) − ( − z I ) − 1 ) ; (18c)
(Leave-one-out trace) G k ( z , w ) = E x k [ ⟨ x k , ( T k − 1 − w Σ − z I ) − 1 x k ⟩ H ] G_k(z,w)=\mathbb{E}_{x_k}\left[\left\langle x_k,(\mathbf{T}_{k-1}-w\mathbf{\Sigma}-zI)^{-1}x_k\right\rangle_{\mathcal{H}}\right] G k ( z , w ) = E x k [ ⟨ x k , ( T k − 1 − w Σ − z I ) − 1 x k ⟩ H ] . (18d)
We point out that: (i) in Eq. (23) for H k H_k H k , neither resolvent within the right hand side need be trace class, but their difference is, and H k H_k H k measures (the rescaled trace of) R k R_k R k relative to the resolvent of the zero operator in L ( H ) \mathcal{L}(\mathcal{H}) L ( H ) ; (ii) in Eq. (23) , E x k \mathbb{E}_{x_k} E x k denotes expectation with respect to x k x_k x k alone with T k − 1 \mathbf{T}_{k-1} T k − 1 held fixed. We also identify the quantity G k Q G_k^Q G k Q corresponds to R k − 1 ( Q ) \mathcal{R}_{k-1}(Q) R k − 1 ( Q ) used by Cheng and Montanari with real arguments. We highlight that these weights Q Q Q in the resolvents allow us to probe selected directions, and their derivatives help the analysis of general spectral filters, especially for the bias term.
Trace interpolation for spectral and regression analysis
The preceding resolvent traces in Eqs. (23) and (23) are central to study both the spectrum of the sample covariance operator and the bias and variance of general regression estimators defined by spectral filters. We will construct a deterministic parameter sequence { w k } k = 0 n \{w_k\}_{k=0}^{n} { w k } k = 0 n with w n = 0 w_n=0 w n = 0 such that H k ( z , w k ) H_k(z,w_k) H k ( z , w k ) changes only by a second-order error in expectation. Its endpoints are
H 0 ( z , w 0 ) = Tr ( ( − w 0 Σ ) ( − w 0 Σ − z I ) − 1 ) , H n ( z , w n ) = H n ( z , 0 ) = Tr ( T n ( T n − z I ) − 1 ) . H_0(z,w_0)=\operatorname{Tr}\left((-w_0\mathbf{\Sigma})(-w_0\mathbf{\Sigma}-zI)^{-1}\right),\qquad H_n(z,w_n)=H_n(z,0)=\operatorname{Tr}\left(\mathbf{T}_n(\mathbf{T}_n-zI)^{-1}\right). H 0 ( z , w 0 ) = Tr ( ( − w 0 Σ ) ( − w 0 Σ − z I ) − 1 ) , H n ( z , w n ) = H n ( z , 0 ) = Tr ( T n ( T n − z I ) − 1 ) .
We call { H k ( z , w k ) } k = 0 n \{H_k(z,w_k)\}_{k=0}^{n} { H k ( z , w k ) } k = 0 n the relative-resolvent interpolation process , or simply the interpolation process . For retrieval of the spectrum, the terminal quantity H n ( z , 0 ) H_n(z,0) H n ( z , 0 ) contains exactly the spectral information carried by the empirical Stieltjes transform. Indeed, since T n = X ∗ X \mathbf{T}_n=X^*X T n = X ∗ X and X X ∗ XX^* X X ∗ has the same nonzero eigenvalues,
s ^ n ( z ) = 1 n Tr ( ( X X ∗ − z I n ) − 1 ) = 1 z ( 1 n H n ( z , 0 ) − 1 ) . (26) \widehat{s}_n(z)=\frac{1}{n}\operatorname{Tr}\left((XX^*-zI_n)^{-1}\right)=\frac{1}{z}\left(\frac{1}{n}H_n(z,0)-1\right). \tag*{(26)} s n ( z ) = n 1 Tr ( ( X X ∗ − z I n ) − 1 ) = z 1 ( n 1 H n ( z , 0 ) − 1 ) . (26)
Thus H n ( z , 0 ) H_n(z,0) H n ( z , 0 ) determines the Stieltjes transform of the finite empirical Gram matrix while remaining well-defined on an infinite-dimensional feature space. By contrast, Tr ( R n ( z , 0 ) ) \operatorname{Tr}(R_n(z,0)) Tr ( R n ( z , 0 )) is not defined when dim ( H ) = ∞ \dim(\mathcal{H})=\infty dim ( H ) = ∞ as R n ( z , 0 ) R_n(z,0) R n ( z , 0 ) equals − z − 1 I -z^{-1}I − z − 1 I on the infinite-dimensional kernel of T n \mathbf{T}_n T n .
For z = − λ < 0 z=-\lambda<0 z = − λ < 0 , H n ( − λ , 0 ) = Tr ( T n ( T n + λ I ) − 1 ) H_n(-\lambda,0)=\operatorname{Tr}\left(\mathbf{T}_n(\mathbf{T}_n+\lambda I)^{-1}\right) H n ( − λ , 0 ) = Tr ( T n ( T n + λ I ) − 1 ) is the empirical effective dimension familiar from kernel learning [83 , 24 , 17 ] . The subtraction of a reference resolvent, arranged so that the resulting difference belongs to the trace class, is a standard technique in the context of trace ideals and spectral shift theory [69 , 45 , 72 ] . Misiakiewicz and Saeed [63 ] consider exactly n − 1 H n ( − λ , 0 ) n^{-1}H_n(-\lambda,0) n − 1 H n ( − λ , 0 ) as one of their matrix functionals and derive its deterministic equivalent by a leave-one-out argument as an ingredient in their analysis of kernel ridge regression training error and generalized cross-validation. Our aim, more universal, is to develop a common interpolation framework for spectral analysis and regression. Along the same deterministic path, we control weighted resolvent traces and their products, which provide the quantities needed to analyze the variance and bias of general spectral filters.
The ingredients underlying the parameter update are classical. Expected leave-one-out resolvents and their self-consistent equations are standard tools for deterministic equivalents of sample covariance matrices [47 , 31 ] , but they typically reduce to a simple linear interpolation. In [28 ] , the authors construct a closely related stochastic interpolation process to cancel the first order drift terms. Our novel construction to be introduced next combines the stochastic interpolation with an expectation-level simplification—its parameter is deterministic and is chosen to cancel the leading unconditional mean drift. We call the resulting parameter sequence the deterministic self-consistent path , distinguishing it from both the adapted random path and the standard linear path.
The backward path and its stability
We formally construct the alluded deterministic path. Fix z ≠ 0 z \ne0 z = 0 , and for ( z , w ) ∈ D ∪ { ( z , w ) ∈ R 2 : z < 0 , w ≤ 0 } (z,w) \in\mathcal{D} \cup\{(z,w) \in\mathbb{R}^{2}: z < 0,\ w \le0\} ( z , w ) ∈ D ∪ {( z , w ) ∈ R 2 : z < 0 , w ≤ 0 } , define the expected leave-one-out trace
G ‾ k ( z , w ) : = E [ G k ( z , w ) ] = E Tr ( Σ ( T k − 1 − w Σ − z I ) − 1 ) . \overline{G}_{k}(z,w) := \mathbb{E}[G_{k}(z,w)] = \mathbb{E}\operatorname{Tr}\left(\boldsymbol{\Sigma}(T_{k-1}-w\boldsymbol{\Sigma}-zI)^{-1}\right). G k ( z , w ) := E [ G k ( z , w )] = E Tr ( Σ ( T k − 1 − w Σ − z I ) − 1 ) .
Starting from w n = 0 w_n=0 w n = 0 , we use this deterministic trace to define the backward updates for k = n , … , 1 k=n,\ldots,1 k = n , … , 1 as below (cf. Theorem 3.(i) guarantees that each update is well-defined)
(Self-consistent backward path)
w k − 1 : = w k − 1 1 + G ‾ k ( z , w k ) . (27) w_{k-1} := w_k-\frac{1}{1+\overline{G}_k(z,w_k)}. \tag*{(27)} w k − 1 := w k − 1 + G k ( z , w k ) 1 . (27)
We highlight that the path itself depend only on the unweighted leave-one-out traces, and the same path will also be used for weighted traces. It is instructive to compare this path with the standard linear interpolation and the adapted forward construction of Cheng and Montanari [28 ] , written after a change of variables, start from w 0 l i n = μ 0 = − z / ω ( z / n ) w^{\mathrm{lin}}_0=\mu_0=-z/\omega(z/n) w 0 lin = μ 0 = − z / ω ( z / n ) and satisfy
(Linear interpolation path)
w k − 1 l i n : = w k l i n − 1 1 + G ‾ 1 ( z , w 0 l i n ) . (28) w^{\mathrm{lin}}_{k-1}:=w^{\mathrm{lin}}_k-\frac{1}{1+\overline{G}_1(z,w^{\mathrm{lin}}_0)}. \tag*{(28)} w k − 1 lin := w k lin − 1 + G 1 ( z , w 0 lin ) 1 . (28)
(Self-consistent forward path)
μ k − 1 : = μ k − 1 1 + G k ( z , μ k ) . (29) \mu_{k-1}:=\mu_k-\frac{1}{1+G_k(z,\mu_k)}. \tag*{(29)} μ k − 1 := μ k − 1 + G k ( z , μ k ) 1 . (29)
The defining equation for ω ( z / n ) \omega(z/n) ω ( z / n ) implies 1 + G ‾ 1 ( z , w 0 l i n ) = − n / w 0 l i n 1+\overline{G}_1(z,w^{\mathrm{lin}}_0)=-n/w^{\mathrm{lin}}_0 1 + G 1 ( z , w 0 lin ) = − n / w 0 lin , so the linear path has constant increments and w k l i n = ( 1 − k / n ) w 0 l i n w^{\mathrm{lin}}_k=(1-k/n)w^{\mathrm{lin}}_0 w k lin = ( 1 − k / n ) w 0 lin , while the backward path instead updates its increment using the current expected trace.
Remark 4. We highlight the beneficial deterministic definition of the backward path, as we can impose the target endpoint w n = 0 w_n=0 w n = 0 and evaluate Eq. (20a) explicitly. While the martingale updates in Eq. (20c) used by Cheng and Montanari [28 ] are generated forward from implicit self-consistent equations, which require inverting the map w ↦ w − ( 1 + G k ( z , w ) ) − 1 w \mapsto w-(1+G_k(z,w))^{-1} w ↦ w − ( 1 + G k ( z , w ) ) − 1 at every step. This introduces questions of existence and possible instability. The backward definition circumvents solving the implicit iteration and fixes the terminal relative-resolvent trace H n ( z , 0 ) H_n(z,0) H n ( z , 0 ) exactly. Figure 9 illustrates the resulting accuracy: backward interpolation has smaller endpoint trace errors than forward interpolation over much of the bulk at small spectral heights, particularly in L 2 L^2 L 2 . Related linear-interpolation estimates appear in Hastie et al. [50 Theorem 5] and Misiakiewicz and Saeed [63 Theorem 4 and Remark 5.1] .
Figure 9. Endpoint trace errors for backward, linear, and forward interpolation under power law (left) and critical law (right) spectra. Rows show empirical L 1 L^{1} L 1 and L 2 L^{2} L 2 errors. We use brighter colors to indicate larger errors. See Appendix § 15 for error definitions and numerical details.
We first present the following stability theorem regarding the deterministic backward path, whose proof is in Appendix § 13 .§ 1 .
Theorem 3 (Stability of the self-consistent backward path). Let Assumption 1 hold and fix z = t + i η ∈ C + ∪ R < 0 z=t+i\eta\in\mathbb{C}_{+}\cup\mathbb{R}_{<0} z = t + i η ∈ C + ∪ R < 0 .
(i) (Well-posedness and monotonicity). For every positive integer n n n , the recursion (27) uniquely defines a deterministic path { w k } k = 0 n \{w_k\}_{k=0}^{n} { w k } k = 0 n with w n = 0 w_n=0 w n = 0 . If η > 0 \eta>0 η > 0 , then
Im w 0 > Im w 1 > ⋯ > Im w n = 0. \operatorname{Im} w_0>\operatorname{Im} w_1>\cdots>\operatorname{Im} w_n=0. Im w 0 > Im w 1 > ⋯ > Im w n = 0. If t < 0 t<0 t < 0 , then also
Re w 0 < Re w 1 < ⋯ < Re w n = 0. \operatorname{Re} w_0<\operatorname{Re} w_1<\cdots<\operatorname{Re} w_n=0. Re w 0 < Re w 1 < ⋯ < Re w n = 0. For z < 0 z<0 z < 0 , the path remains real for all $0\leq k\leq n.
(ii) (Expected-trace stability). Under Assumption 2 at z z z with κ = 1 \kappa=1 κ = 1 , the expected leave-one-out traces satisfy
max 1 ≤ k ≤ n { ∣ G ‾ k ( z , w k ) − G ‾ 1 ( z , w 0 ) ∣ ∨ ∣ G ‾ k ( z , w k ) − G ‾ n ( z , 0 ) ∣ } = O C x , ν ( Λ n ( z ) 15 n ) . \max_{1\leq k\leq n}\left\{\left|\overline{G}_k(z,w_k)-\overline{G}_1(z,w_0)\right|\vee\left|\overline{G}_k(z,w_k)-\overline{G}_n(z,0)\right|\right\}=O_{C_{x,\nu}}\left(\frac{\Lambda_n(z)^{15}}{n}\right). 1 ≤ k ≤ n max { G k ( z , w k ) − G 1 ( z , w 0 ) ∨ G k ( z , w k ) − G n ( z , 0 ) } = O C x , ν ( n Λ n ( z ) 15 ) . (iii) (Path stability). Under the conditions of (ii), the backward path satisfies for 0 ≤ k ≤ n 0\leq k\leq n 0 ≤ k ≤ n ,
∣ w k − w k l i n ∣ = O C x , ν ( Λ n ( z ) 17 ⋅ n − k n ) , \left|w_k-w_k^{\mathrm{lin}}\right|=O_{C_{x,\nu}}\left(\Lambda_n(z)^{17}\cdot\frac{n-k}{n}\right), w k − w k lin = O C x , ν ( Λ n ( z ) 17 ⋅ n n − k ) , where w k l i n = − ( 1 − k / n ) z / ω ( z / n ) w_k^{\mathrm{lin}}=-(1-k/n)z/\omega(z/n) w k lin = − ( 1 − k / n ) z / ω ( z / n ) .
The above Theorem 3 controls the expected leave-one-out trace along the backward path and compares that path with the linear interpolation determined by ω ( z / n ) \omega(z/n) ω ( z / n ) . We highlight that Theorem 3 is non-random: the bounds in (ii) and (iii) are deterministic and hold simultaneously for any deterministic set of spectral parameters on which the assumptions are satisfied uniformly, in contrast to the stochastic approximation in [28 ] .
To provide insights from a theoretical point of view, Figure 10 illustrates the two-phase argument underlying Theorem 3. For large sample indices, sample covariance concentration controls the resolvents, while for small sample indices, we use explicit one-step path error control.
Figure 10. The illustration of the proof strategies along the backward path { w k } \{w_k\} { w k } on C \mathbb{C} C with Phase I (large sample) and Phase II (small sample). The real-axis markers show the reflected terminal ESD as a reference overlay after rescaling.
Remark 5 (The polynomial, critical and proportional regimes). Let ζ \zeta ζ range over a fixed compact subset of C + \mathbb{C}_{+} C + , and take z z z at the scales in Table 1 . Proposition 2 and the discussions in Appendix § 10 .§ 3 provide Λ n ( z ) = O ( ( log n ) 2 ) \Lambda_n(z)=O((\log n)^2) Λ n ( z ) = O (( log n ) 2 ) for the power law and proportional regime, and O ( ( log n ) 3 ) O((\log n)^3) O (( log n ) 3 ) for the critical law. Thus, in all three regimes, Theorem 3.(ii) and (iii) control the expected traces and the backward path with only logarithmic losses.
Remark 6 (Logarithmic and local-scale refinement). Under the hypotheses of Zhivotovskiy [84, Theorem 1 and Example 1], the dimension-free covariance bound can potentially sharpen the real resolvent spectral parameter when the tail effective rank is comparable to the sample size. Reaching optimal local scales would require sharpening the dependence on Λ n ( z ) \Lambda_n(z) Λ n ( z ) in both the approximation bounds and the sample-size condition to linear, up to logarithmic factors, as discussed in Appendix § 10 .§ 3 . We leave these refinements for future work.
Stability of weighted single-resolvents and mixed-resolvents
The backward path { w k } k = 0 n \{w_k\}_{k=0}^{n} { w k } k = 0 n and its stability allow us to track the random weighted resolvent traces H n Q H^Q_n H n Q and G n + 1 Q G^Q_{n+1} G n + 1 Q in Eqs. (23) and (18d) to their deterministic equivalents H 0 Q H^Q_0 H 0 Q and G 1 Q G^Q_1 G 1 Q , controlling both their bias and fluctuations with nonasymptotic error bounds. On a high level, our path construction allows us to write
H k Q ( z , w k ) − H k − 1 Q ( z , w k − 1 ) = ξ k H , Q + ρ k H , Q and G k + 1 Q ( z , w k + 1 ) − G k Q ( z , w k ) = ξ k G , Q + ρ k G , Q , H^Q_k(z,w_k)-H^Q_{k-1}(z,w_{k-1})=\xi^{H,Q}_k+\rho^{H,Q}_k\quad\text{and}\quad G^Q_{k+1}(z,w_{k+1})-G^Q_k(z,w_k)=\xi^{G,Q}_k+\rho^{G,Q}_k, H k Q ( z , w k ) − H k − 1 Q ( z , w k − 1 ) = ξ k H , Q + ρ k H , Q and G k + 1 Q ( z , w k + 1 ) − G k Q ( z , w k ) = ξ k G , Q + ρ k G , Q ,
where the ξ \xi ξ -terms are martingale differences and the β \beta β -terms form a small accumulated drift, both controlled in L p L^p L p . Finally, the path stability in Theorem 3.(iii) allows us to transfer the backward path at w 0 w_0 w 0 to w 0 l i n w_0^{\mathrm{lin}} w 0 lin .
Thus, the linear interpolation path makes the deterministic equivalents explicit by comparing the terminal values and the backward path gives sharp bounds. Recall Eq. (23) , at the sample terminal point, we have
H n Q ( z , 0 ) = Tr ( Q T n ( T n − z I ) − 1 ) and G n + 1 Q ( z , 0 ) = Tr ( Q Σ 1 2 R n ( z , 0 ) Σ 1 2 ) . H_n^Q(z,0)=\operatorname{Tr}\left(QT_n(T_n-zI)^{-1}\right)
\qquad\text{and}\qquad
G_{n+1}^Q(z,0)=\operatorname{Tr}\left(Q\Sigma^{\frac{1}{2}}R_n(z,0)\Sigma^{\frac{1}{2}}\right). H n Q ( z , 0 ) = Tr ( Q T n ( T n − z I ) − 1 ) and G n + 1 Q ( z , 0 ) = Tr ( Q Σ 2 1 R n ( z , 0 ) Σ 2 1 ) .
At the population terminal point, i.e. the linear initial point, substituting T 0 = 0 T_0=0 T 0 = 0 and w 0 l i n = − z / ω ( z / n ) w_0^{\mathrm{lin}}=-z/\omega(z/n) w 0 lin = − z / ω ( z / n ) in Eq. (23) , we have
H 0 Q ( z , w 0 l i n ) = Tr ( Q Σ ( Σ − ω ( z / n ) I ) − 1 ) and G 1 Q ( z , w 0 l i n ) = 1 z Tr ( Q Σ ( Σ / ω ( z / n ) − I ) − 1 ) . (30) H_0^Q(z,w_0^{\mathrm{lin}})=\operatorname{Tr}\left(Q\Sigma(\Sigma-\omega(z/n)I)^{-1}\right)
\qquad\text{and}\qquad
G_1^Q(z,w_0^{\mathrm{lin}})=\frac{1}{z}\operatorname{Tr}\left(Q\Sigma(\Sigma/\omega(z/n)-I)^{-1}\right). \tag*{(30)} H 0 Q ( z , w 0 lin ) = Tr ( Q Σ ( Σ − ω ( z / n ) I ) − 1 ) and G 1 Q ( z , w 0 lin ) = z 1 Tr ( Q Σ ( Σ/ ω ( z / n ) − I ) − 1 ) . (30)
Using ∣ z − ∣ = dist ( z , R ≥ 0 ) |z_-|=\operatorname{dist}(z,\mathbb{R}_{\ge0}) ∣ z − ∣ = dist ( z , R ≥ 0 ) as in Assumption 2 and the shorthand H n Q = H n Q ( z , 0 ) H_n^Q=H_n^Q(z,0) H n Q = H n Q ( z , 0 ) , H 0 Q = H 0 Q ( z , w 0 l i n ) H_0^Q=H_0^Q(z,w_0^{\mathrm{lin}}) H 0 Q = H 0 Q ( z , w 0 lin ) , as well as G n + 1 Q = G n + 1 Q ( z , 0 ) G_{n+1}^Q=G_{n+1}^Q(z,0) G n + 1 Q = G n + 1 Q ( z , 0 ) , G 1 Q = G 1 Q ( z , w 0 l i n ) G_1^Q=G_1^Q(z,w_0^{\mathrm{lin}}) G 1 Q = G 1 Q ( z , w 0 lin ) , the next theorem provides the approximation bounds at their terminal values. The proof is in Appendix E.2.
Theorem 4 (Deterministic equivalents of single-resolvents). Let z ∈ C + ∪ R < 0 z\in\mathbb{C}_+\cup\mathbb{R}_{<0} z ∈ C + ∪ R < 0 , κ > 0 \kappa>0 κ > 0 and r r r *be a nonnegative integer. Under Assumptions 1 and 2 at z z z with parameters ν , κ \nu,\kappa ν , κ , it holds for all 2 ≤ p ≤ κ log n 2\le p\le\kappa\log n 2 ≤ p ≤ κ log n and ∥ Q ∥ < ∞ \lVert Q\rVert<\infty ∥ Q ∥ < ∞ that
∥ 1 n ⋅ d r d z r ( H n Q − H 0 Q ) ∥ L p ∨ ∥ d r d z r ( G n + 1 Q − G 1 Q ) ∥ L p = O C x , ν , κ , r ( p 2 Λ n ( z ) 23 n ∣ z − ∣ r ⋅ ( ∥ Q ∥ ∧ ∥ Q ∥ F n ) ) . (31) \left\lVert\frac{1}{n}\cdot\frac{\mathrm{d}^r}{\mathrm{d}z^r}\left(H_n^Q-H_0^Q\right)\right\rVert_{L^p}
\vee
\left\lVert\frac{\mathrm{d}^r}{\mathrm{d}z^r}\left(G_{n+1}^Q-G_1^Q\right)\right\rVert_{L^p}
=O_{C_{x,\nu,\kappa,r}}\left(\frac{p^2\Lambda_n(z)^{23}}{n|z_-|^r}\cdot\left(\lVert Q\rVert\wedge\frac{\lVert Q\rVert_F}{\sqrt{n}}\right)\right). \tag*{(31)} n 1 ⋅ d z r d r ( H n Q − H 0 Q ) L p ∨ d z r d r ( G n + 1 Q − G 1 Q ) L p = O C x , ν , κ , r ( n ∣ z − ∣ r p 2 Λ n ( z ) 23 ⋅ ( ∥ Q ∥ ∧ n ∥ Q ∥ F ) ) . (31)
At r = 0 r=0 r = 0 , Theorem 4 approximates H n Q / n H_n^Q/n H n Q / n and G n + 1 Q G_{n+1}^Q G n + 1 Q by H 0 Q / n H_0^Q/n H 0 Q / n and G 1 Q G_1^Q G 1 Q in Eq. (30) . At the polynomial and centered critical bulk scales of Remark 5, both errors are of order n − 1 n^{-1} n − 1 up to powers of log n \log n log n for ∥ Q ∥ ≤ 1 \lVert Q\rVert\le1 ∥ Q ∥ ≤ 1 . For r ≥ 1 r\ge1 r ≥ 1 , differentiation includes the dependence of the population traces on w 0 l i n ( z ) = − z / ω ( z / n ) w_0^{\mathrm{lin}}(z)=-z/\omega(z/n) w 0 lin ( z ) = − z / ω ( z / n ) . For each admissible z z z and fixed r r r , applying Markov’s inequality to Eq. (31) at p = κ log n p=\kappa\log n p = κ log n gives a pointwise high probability bound for both errors with probability 1 − O ( n − D ) 1-O(n^{-D}) 1 − O ( n − D ) for any fixed D > 0 D>0 D > 0 .
For our main application, regression with a spectral filter, the prediction variance conditional on the design involves a single trace weighted by Σ \Sigma Σ . Its resolvent representation uses divided differences of z G n + 1 I ( z , 0 ) zG_{n+1}^I(z,0) z G n + 1 I ( z , 0 ) , with a derivative when the two spectral parameters coincide. Theorem 4 controls both quantities. To control the concentration of its bias term, we need traces of two weighted resolvents. We study these mixed resolvents next.
Expanding T n T_n T n over the observations allows us to approximate the mixed trace by a divided difference of terminal single traces, with prefactors determined by the two terminal path increments. Applying Theorems 3 and 4 replaces these quantities by their linear initial values; the initial resolvent identity then yields the divided-difference factor of ω \omega ω . In the next theorem, we bound the trace of weighted mixed resolvents around their identified deterministic equivalents. Define the weighted resolvent R ~ k ( z , w ) : = Σ 1 2 R k ( z , w ) Σ 1 2 \widetilde{R}_k(z,w):=\Sigma^{\frac{1}{2}}R_k(z,w)\Sigma^{\frac{1}{2}} R k ( z , w ) := Σ 2 1 R k ( z , w ) Σ 2 1 . For simplicity of the statement, we write R ~ n = R ~ n ( z , 0 ) \widetilde{R}_n=\widetilde{R}_n(z,0) R n = R n ( z , 0 ) and R ~ 0 = R ~ 0 ( z , w 0 l i n ( z ) ) \widetilde{R}_0=\widetilde{R}_0(z,w_0^{\mathrm{lin}}(z)) R 0 = R 0 ( z , w 0 lin ( z )) , with R ~ n ′ = R ~ n ( z ′ , 0 ) \widetilde{R}'_n=\widetilde{R}_n(z',0) R n ′ = R n ( z ′ , 0 ) and R ~ 0 ′ = R ~ 0 ( z ′ , w 0 l i n ( z ′ ) ) \widetilde{R}'_0=\widetilde{R}_0(z',w_0^{\mathrm{lin}}(z')) R 0 ′ = R 0 ( z ′ , w 0 lin ( z ′ )) at the second parameter. We also use the shorthand Λ n = Λ n ( z ) \Lambda_n=\Lambda_n(z) Λ n = Λ n ( z ) , Λ n ( z ′ ) = Λ n ′ \Lambda_n(z')=\Lambda'_n Λ n ( z ′ ) = Λ n ′ and suppress the dependence on C x , ν , κ C_{x,\nu,\kappa} C x , ν , κ in its big-Oh notation. See its proof in Appendix E.3.
Theorem 5 (Deterministic equivalents of mixed-resolvents). Fix z , z ′ ∈ C + ∪ R < 0 z,z'\in\mathbb{C}_+\cup\mathbb{R}_{<0} z , z ′ ∈ C + ∪ R < 0 , κ > 0 \kappa>0 κ > 0 , and nonnegative integers r , r ′ r,r' r , r ′ . Suppose Assumption 1 holds and Assumption 2 holds at z , z ′ z,z' z , z ′ with the
same ν , κ \nu,\kappa ν , κ . For every deterministic Q \mathbf{Q} Q with ∥ Q ∥ F < ∞ \|\mathbf{Q}\|_{F}<\infty ∥ Q ∥ F < ∞ and every 2 ≤ p ≤ κ log n 2\le p\le\kappa\log n 2 ≤ p ≤ κ log n , we have
∥ d r + r ′ d z r d ( z ′ ) r ′ Tr [ Q ( R ~ n R ~ n ′ − ω ( z / n ) − ω ( z ′ / n ) ( z − z ′ ) / n R ~ 0 R ~ 0 ′ ) ] ∥ L p = O r , r ′ ( p 3 / 2 ∥ Q ∥ F ⋅ ( Λ n ∨ Λ n ′ ) 26 n 5 / 2 ∣ z − z ′ ∣ r ∣ z ′ ∣ r ′ ) . (32) \left\|\frac{\mathrm{d}^{r+r'}}{\mathrm{d}z^{r}\mathrm{d}(z')^{r'}}\operatorname{Tr}\left[\mathbf{Q}\left(\widetilde{R}_{n}\widetilde{R}_{n}'-\frac{\omega(z/n)-\omega(z'/n)}{(z-z')/n}\widetilde{R}_{0}\widetilde{R}_{0}'\right)\right]\right\|_{L^{p}}=O_{r,r'}\left(\frac{p^{3/2}\|\mathbf{Q}\|_{F}\cdot(\Lambda_{n}\vee\Lambda_{n}')^{26}}{n^{5/2}|z-z'|^{r}|z'|^{r'}}\right). \tag*{(32)} d z r d ( z ′ ) r ′ d r + r ′ Tr [ Q ( R n R n ′ − ( z − z ′ ) / n ω ( z / n ) − ω ( z ′ / n ) R 0 R 0 ′ ) ] L p = O r , r ′ ( n 5/2 ∣ z − z ′ ∣ r ∣ z ′ ∣ r ′ p 3/2 ∥ Q ∥ F ⋅ ( Λ n ∨ Λ n ′ ) 26 ) . (32)
When z ′ = z z'=z z ′ = z the divided difference becomes ω ′ ( z / n ) \omega'(z/n) ω ′ ( z / n ) , and the derivatives act separately.
We point out the initial explicit factors R ~ 0 ( z , w 0 l i n ( z ) ) = ( ω ( z / n ) / z ) Σ ( Σ − ω ( z / n ) I ) − 1 \widetilde{R}_{0}(z,w_{0}^{\mathrm{lin}}(z))=(\omega(z/n)/z)\boldsymbol{\Sigma}(\boldsymbol{\Sigma}-\omega(z/n)I)^{-1} R 0 ( z , w 0 lin ( z )) = ( ω ( z / n ) / z ) Σ ( Σ − ω ( z / n ) I ) − 1 . In addition, the same bound above extends to either parameter in the lower half-plane, by reflecting the corresponding path and taking adjoints (cf. Appendix § 13 .§ 3 ).
Discussion
Our results establish nonasymptotic deterministic equivalents for spectral laws and prediction risk in a Hilbert-space setting. As a concluding remark, we outline unresolved questions, future research directions and potential applications that merit further investigation.
Optimal local laws at local scale. The mesoscopic estimates in Appendix § 10 .§ 3 leave a gap to the optimal local scale known in the proportional regime [54 ] . As discussed there, a natural target for future refinement is linear dependence on Λ n ( z ) \Lambda_n(z) Λ n ( z ) , up to logarithmic factors, in both the error bound and the sample-size requirement. This would allow spectral heights approaching the typical bulk spacing in our canonical examples. Achieving this requires sharper control along the backward interpolation path, and most likely a completely new interpolation scheme. It is also interesting to determine how far such estimates hold under the quadratic concentration assumption alone.
Natural spikes and the transition to the bulk. As the sample size naturally grows, the decaying population spectrum in a Hilbert space can produce separated leading sample eigenvalues without an added finite-rank perturbation. This produces the natural “spiked plus noise” random matrix model. The question of characterizing their locations, eigendirection overlaps, and eventual transition into the bulk would give a natural analogue of the spiked Wigner and Wishart theories [10 , 14 ] . The self-consistent equation and weighted resolvent approximations provide a starting point for identifying the corresponding shifts and separation criteria.
Central limit theorems and uncertainty quantification. Central limit theorems could provide confidence intervals for spectral quantities and prediction risk. The martingale decompositions in § 5.3 suggest a route beyond classical results for linear spectral statistics [9 ] . This requires identifying and consistently estimating the fluctuation variance, and making the drift and centering errors negligible at the fluctuation scale or correcting for them. However, our current bounds do not show that these errors are negligible at the fluctuation scale.
Lower-edge behavior and universality. § 3 uses a smallest-eigenvalue bound at the correct order. A sharper characterization would locate the smallest sample eigenvalue near the lower edge of the finite-n n n law μ n \mu_n μ n and determine its fluctuations. At a regular positive edge, a natural question is whether Tracy-Widom limits hold after appropriate centering and rescaling, as in classical covariance models [42 ] . This requires finer spectral control and potentially stronger distributional assumptions, and it would enable quantification of the extent to which a negative ridge pole can approach the empirical spectrum. However, the latter consideration is likely of greater theoretical than practical interest.
PCR and nonsmooth spectral filters. The theory in our paper is restricted to analytic filters. Principal component regression (PCR) uses a discontinuous spectral cutoff and therefore falls outside our analytic framework. It would be interesting to extend our deterministic risk approximations to PCR and other nonanalytic spectral filters, including filters with limited smoothness. Indeed, we can nevertheless formally substitute more general filters into Proposition 3. It remains open to quantify the resulting approximation errors and how these errors depend on the filter’s smoothness.
Applications in learning theory. Finally, our effective risk formulas could guide source-dependent regularization and stopping rules, as suggested in § 3.2 . In practice, developing data-driven procedures would require estimating the relevant spectral and signal quantities, with generalized cross-validation providing a related example [63 ] . More broadly, these tools could potentially refine analyses of random features and neural scaling laws [33 , 6 ] . An important and interesting future direction is to extend the theory to learned representations.
Statement of AI use
We disclose the use of AI, and if used, acknowledge it in the following two categories.
(i) Primary contributions: Yes, by C.C., who uses GPT-5.6 to find the covariance tensorization (bilinear form of the Efron-Stein) to control the expected traces of weighted resolvents in the proof of Lemma E.11, which supports the proof of Lemma E.4 .
(ii) Supporting contributions: Yes, by C.C., who uses GPT-5.6 and GPT-6 models for producing numerical experiments, assisting in writing, suggesting literature, and auditing and optimizing existing proofs. GPT-6 suggested the logarithmic submean inequality on L p L^p L p norms for bounding the contour integral across the spectral gap in Lemma C.1 .
The authors took full responsibility for the contents of the paper.
Acknowledgment
The author thanks Theodor Misiakiewicz for helpful discussions. This work made use of the Illinois Campus Cluster, a computing resource that is operated by the Illinois Campus Cluster Program (ICCP) in conjunction with the National Center for Supercomputing Applications (NCSA) and which is supported by funds from the University of Illinois Urbana-Champaign.
Technical lemmas
We record the following moment form of the Efron-Stein inequality, adapted from Boucheron et al. [20 Theorem 2] .
Lemma A.1 (Moment Efron-Stein inequality). Let X 1 , … , X N X_1,\ldots,X_N X 1 , … , X N be independent random elements, let X 1 ′ , … , X N ′ X'_1,\ldots,X'_N X 1 ′ , … , X N ′ be independent copies that are also independent of X 1 , … , X N X_1,\ldots,X_N X 1 , … , X N , and let
Z = f ( X 1 , … , X N ) , Z=f(X_1,\ldots,X_N), Z = f ( X 1 , … , X N ) , Z ( i ) = f ( X 1 , … , X i − 1 , X i ′ , X i + 1 , … , X N ) . Z^{(i)}=f(X_1,\ldots,X_{i-1},X'_i,X_{i+1},\ldots,X_N). Z ( i ) = f ( X 1 , … , X i − 1 , X i ′ , X i + 1 , … , X N ) . For every q ≥ 2 q \ge2 q ≥ 2 such that the right-hand side is finite and some universal constant C > 0 C>0 C > 0 ,
∥ Z − E Z ∥ L q ≤ C q ⋅ ( ∑ i = 1 N ∥ Z − Z ( i ) ∥ L q 2 ) 1 / 2 . (33) \lVert Z-\mathbb{E}Z\rVert_{L^q}\le C\sqrt{q}\cdot\left(\sum_{i=1}^{N}\lVert Z-Z^{(i)}\rVert_{L^q}^{2}\right)^{1/2}. \tag*{(33)} ∥ Z − E Z ∥ L q ≤ C q ⋅ ( i = 1 ∑ N ∥ Z − Z ( i ) ∥ L q 2 ) 1/2 . (33) The statement holds for both Z ∈ R Z\in\mathbb{R} Z ∈ R and Z ∈ C Z\in\mathbb{C} Z ∈ C .
Proof. For real-valued Z Z Z , applying Boucheron et al. [20 Theorem 2] to Z Z Z and − Z -Z − Z gives
∥ ( Z − E Z ) + ∥ L q ≤ 2 κ q q ⋅ ∥ V + 1 / 2 ∥ L q , ∥ ( Z − E Z ) − ∥ L q ≤ 2 κ q q ⋅ ∥ V − 1 / 2 ∥ L q , \lVert(Z-\mathbb{E}Z)_+\rVert_{L^q}\le\sqrt{2\kappa_q q}\cdot\lVert V_+^{1/2}\rVert_{L^q},\qquad
\lVert(Z-\mathbb{E}Z)_-\rVert_{L^q}\le\sqrt{2\kappa_q q}\cdot\lVert V_-^{1/2}\rVert_{L^q}, ∥( Z − E Z ) + ∥ L q ≤ 2 κ q q ⋅ ∥ V + 1/2 ∥ L q , ∥( Z − E Z ) − ∥ L q ≤ 2 κ q q ⋅ ∥ V − 1/2 ∥ L q ,
where κ q = O ( 1 ) \kappa_q=O(1) κ q = O ( 1 ) and
V ± : = ∑ i = 1 N E X i ′ [ ( Z − Z ( i ) ) ± 2 ] . V_\pm:=\sum_{i=1}^{N}\mathbb{E}_{X'_i}\left[(Z-Z^{(i)})_\pm^2\right]. V ± := i = 1 ∑ N E X i ′ [ ( Z − Z ( i ) ) ± 2 ] .
Set V : = ∑ i = 1 N E X i ′ [ ∣ Z − Z ( i ) ∣ 2 ] V:=\sum_{i=1}^{N}\mathbb{E}_{X'_i}\left[|Z-Z^{(i)}|^2\right] V := ∑ i = 1 N E X i ′ [ ∣ Z − Z ( i ) ∣ 2 ] . Since V ± ≤ V V_\pm\le V V ± ≤ V , we have
∥ Z − E Z ∥ L q = ( ∥ ( Z − E Z ) + ∥ L q q + ∥ ( Z − E Z ) − ∥ L q q ) 1 / q ≤ 2 1 / q 2 κ q q ⋅ ∥ V 1 / 2 ∥ L q = O ( q ∥ V 1 / 2 ∥ L q ) , \lVert Z-\mathbb{E}Z\rVert_{L^q}=\left(\lVert(Z-\mathbb{E}Z)_+\rVert_{L^q}^q+\lVert(Z-\mathbb{E}Z)_-\rVert_{L^q}^q\right)^{1/q}\le2^{1/q}\sqrt{2\kappa_q q}\cdot\lVert V^{1/2}\rVert_{L^q}=O\left(\sqrt{q}\,\lVert V^{1/2}\rVert_{L^q}\right), ∥ Z − E Z ∥ L q = ( ∥( Z − E Z ) + ∥ L q q + ∥( Z − E Z ) − ∥ L q q ) 1/ q ≤ 2 1/ q 2 κ q q ⋅ ∥ V 1/2 ∥ L q = O ( q ∥ V 1/2 ∥ L q ) ,
where the big-Oh notation subsumes the universal bounds 2 1 / q ≤ 2 2^{1/q}\le\sqrt{2} 2 1/ q ≤ 2 and sup q ≥ 2 κ q < ∞ \sup_{q\ge2}\kappa_q<\infty sup q ≥ 2 κ q < ∞ . For complex-valued Z Z Z , applying the real bound to Re Z \operatorname{Re}Z Re Z and Im Z \operatorname{Im}Z Im Z and using Minkowski’s inequality gives the same estimate. It remains to bound ∥ V 1 / 2 ∥ L q \lVert V^{1/2}\rVert_{L^q} ∥ V 1/2 ∥ L q in terms of the replacement moments. Using q / 2 ≥ 1 q/2\ge1 q /2 ≥ 1 , we write
∥ V 1 / 2 ∥ L q 2 = ∥ ∑ i = 1 N E X i ′ [ ∣ Z − Z ( i ) ∣ 2 ] ∥ L q / 2 ≤ ( i ) ∑ i = 1 N ∥ E X i ′ [ ∣ Z − Z ( i ) ∣ 2 ] ∥ L q / 2 ≤ ( i i ) ∑ i = 1 N ∥ Z − Z ( i ) ∥ L q 2 . \lVert V^{1/2}\rVert_{L^q}^2=\left\lVert\sum_{i=1}^{N}\mathbb{E}_{X'_i}\left[|Z-Z^{(i)}|^2\right]\right\rVert_{L^{q/2}}\overset{\mathrm{(i)}}{\le}\sum_{i=1}^{N}\left\lVert\mathbb{E}_{X'_i}\left[|Z-Z^{(i)}|^2\right]\right\rVert_{L^{q/2}}\overset{\mathrm{(ii)}}{\le}\sum_{i=1}^{N}\lVert Z-Z^{(i)}\rVert_{L^q}^2. ∥ V 1/2 ∥ L q 2 = i = 1 ∑ N E X i ′ [ ∣ Z − Z ( i ) ∣ 2 ] L q /2 ≤ ( i ) i = 1 ∑ N E X i ′ [ ∣ Z − Z ( i ) ∣ 2 ] L q /2 ≤ ( ii ) i = 1 ∑ N ∥ Z − Z ( i ) ∥ L q 2 .
We make use of Minkowski’s inequality in L q / 2 L^{q/2} L q /2 in (i), and Jensen’s inequality in (ii). Taking square roots and substituting into the preceding estimate proves Eq. (33) . □
The following covariance tensorization inequality is the bilinear form of the Efron-Stein inequality and follows from its standard martingale proof. See, e.g., Boucheron et al. [21 Theorem 3.1] .
Lemma A.2 (Covariance tensorization). Let X 1 , … , X N X_1,\ldots,X_N X 1 , … , X N be independent random elements, let Z = Z ( X 1 , … , X N ) Z=Z(X_1,\ldots,X_N) Z = Z ( X 1 , … , X N ) and W = W ( X 1 , … , X N ) W=W(X_1,\ldots,X_N) W = W ( X 1 , … , X N ) be complex-valued random variables in L 2 L^2 L 2 , and let Z ( i ) Z^{(i)} Z ( i ) and W ( i ) W^{(i)} W ( i ) be obtained by replacing X i X_i X i by an independent copy. Then
∣ E [ ( Z − E Z ) ( W − E W ) ] ∣ ≤ ∑ i = 1 N ∥ Z − Z ( i ) ∥ L 2 ∥ W − W ( i ) ∥ L 2 . \left|\mathbb{E}\left[(Z-\mathbb{E}Z)(W-\mathbb{E}W)\right]\right|\le\sum_{i=1}^{N}\lVert Z-Z^{(i)}\rVert_{L^2}\lVert W-W^{(i)}\rVert_{L^2}. ∣ E [ ( Z − E Z ) ( W − E W ) ] ∣ ≤ i = 1 ∑ N ∥ Z − Z ( i ) ∥ L 2 ∥ W − W ( i ) ∥ L 2 .
Proof. Let F i = σ ( X 1 , … , X i ) \mathcal{F}_i = \sigma(X_1,\ldots,X_i) F i = σ ( X 1 , … , X i ) , with F 0 \mathcal{F}_0 F 0 trivial, and let D i Z = E [ Z ∣ F i ] − E [ Z ∣ F i − 1 ] D_iZ = \mathbb{E}[Z \mid\mathcal{F}_i] - \mathbb{E}[Z \mid\mathcal{F}_{i-1}] D i Z = E [ Z ∣ F i ] − E [ Z ∣ F i − 1 ] , with D i W D_iW D i W defined analogously. Orthogonality of martingale differences gives E [ ( Z − E Z ) ( W − E W ) ] = ∑ i = 1 N E [ D i Z D i W ] \mathbb{E}[(Z-\mathbb{E}Z)(W-\mathbb{E}W)] = \sum_{i=1}^{N}\mathbb{E}[D_iZ\,D_iW] E [( Z − E Z ) ( W − E W )] = ∑ i = 1 N E [ D i Z D i W ] . If X i ′ X_i' X i ′ is the independent copy used in Z ( i ) Z^{(i)} Z ( i ) , conditioning first on F i \mathcal{F}_i F i and then integrating over X i ′ X_i' X i ′ and the future variables shows D i Z = E [ Z − Z ( i ) ∣ F i ] D_iZ = \mathbb{E}[Z-Z^{(i)} \mid\mathcal{F}_i] D i Z = E [ Z − Z ( i ) ∣ F i ] . Hence conditional Jensen’s inequality gives ∥ D i Z ∥ L 2 ≤ ∥ Z − Z ( i ) ∥ L 2 \lVert D_iZ\rVert_{L^2} \leq\lVert Z-Z^{(i)}\rVert_{L^2} ∥ D i Z ∥ L 2 ≤ ∥ Z − Z ( i ) ∥ L 2 , and similarly for W W W . The conclusion follows from Cauchy-Schwarz, applied to each martingale-difference product. For complex-valued variables, the same argument applies to the bilinear product; alternatively, apply the real-valued result to the real and imaginary parts.
Let s ( z ) s(z) s ( z ) be defined on z ∈ C + z \in\mathbb{C}_+ z ∈ C + . The following lemma helps us to determine when s ( z ) s(z) s ( z ) is indeed a Stieltjes transform of a probability measure, often from only s ( z ) s(z) s ( z ) satisfying a certain fixed-point equation (or system of equations). s ( z ) s(z) s ( z ) is known to be a Herglotz-Nevanlinna function if it is analytic on C + \mathbb{C}_+ C + and Im s ( z ) ≥ 0 \operatorname{Im}s(z) \geq0 Im s ( z ) ≥ 0 for all z ∈ C + z \in\mathbb{C}_+ z ∈ C + . We have the following representation lemma [52 ] .
Lemma A.3 (Herglotz representation). The function s ( z ) s(z) s ( z ) is a Herglotz-Nevanlinna function if and only if there are unique numbers a ∈ R a \in\mathbb{R} a ∈ R , b ≥ 0 b \geq0 b ≥ 0 and a (positive) Borel measure μ \mu μ with ∫ R 1 1 + t 2 d μ ( t ) < ∞ \int_{\mathbb{R}} \frac{1}{1+t^2}\,\mathrm{d}\mu(t) < \infty ∫ R 1 + t 2 1 d μ ( t ) < ∞ such that
s ( z ) = a + b z + ∫ − ∞ + ∞ ( 1 t − z − t 1 + t 2 ) μ ( d t ) . s(z) = a + bz + \int_{-\infty}^{+\infty} \left(\frac{1}{t-z} - \frac{t}{1+t^2}\right)\mu(\mathrm{d}t). s ( z ) = a + b z + ∫ − ∞ + ∞ ( t − z 1 − 1 + t 2 t ) μ ( d t ) .
Lemma A.4 (Stability of the effective spectral parameter equation for ω ( z ) \omega(z) ω ( z ) ). Let Σ ⪰ 0 \Sigma\succeq0 Σ ⪰ 0 be trace-class. For every z ∈ C + z \in\mathbb{C}_+ z ∈ C + ,
s ( z ) − 1 = − z + ∑ j ≥ 1 σ j 1 + n σ j s ( z ) s(z)^{-1} = -z + \sum_{j \geq1} \frac{\sigma_j}{1+n\sigma_j s(z)} s ( z ) − 1 = − z + j ≥ 1 ∑ 1 + n σ j s ( z ) σ j has a unique solution s ( z ) = s n ( z ) ∈ C + s(z) = s_n(z) \in\mathbb{C}_+ s ( z ) = s n ( z ) ∈ C + . It is the Stieltjes transform of a probability measure μ n \mu_n μ n supported on [ 0 , ∞ ) [0,\infty) [ 0 , ∞ ) , and is equal to − ( n ω ( z / n ) ) − 1 -(n\omega(z/n))^{-1} − ( nω ( z / n ) ) − 1 . It extends analytically across ( − ∞ , 0 ) (-\infty,0) ( − ∞ , 0 ) , where it is the unique positive solution of Eq. (8) . For z , z ′ ∈ C + ∪ R < 0 z,z' \in\mathbb{C}_+ \cup\mathbb{R}_{<0} z , z ′ ∈ C + ∪ R < 0 ,
∣ s n ( z ) ∣ ≤ ∣ z ∣ − 1 and ∣ s n ( z ′ ) − s n ( z ) ∣ ≤ ∣ z ′ − z ∣ ∣ z ′ ∣ ⋅ ∣ z ∣ . \lvert s_n(z)\rvert\leq\lvert z\rvert^{-1}
\qquad\text{and}\qquad
\lvert s_n(z')-s_n(z)\rvert\leq\frac{\lvert z'-z\rvert}{\lvert z'\rvert\cdot\lvert z\rvert}. ∣ s n ( z )∣ ≤ ∣ z ∣ − 1 and ∣ s n ( z ′ ) − s n ( z )∣ ≤ ∣ z ′ ∣ ⋅ ∣ z ∣ ∣ z ′ − z ∣ .
We defer the proof of Lemma A.4 to Appendix A.1. The next lemma quantifies the spectral transfer from approximation errors of Stieltjes transforms to their probability measures over an interval E E E . For a probability measure μ \mu μ with Stieltjes transform s s s , the Stieltjes inversion formula [7 Theorem B.8] recovers the mass of E = [ a , b ] E=[a,b] E = [ a , b ] when μ ( { a , b } ) = 0 \mu(\{a,b\})=0 μ ({ a , b }) = 0 , giving
μ ( E ) = lim η ↓ 0 1 π ∫ E Im s ( t + i η ) d t . \mu(E) = \lim_{\eta\downarrow0} \frac{1}{\pi}\int_E \operatorname{Im}s(t+\mathrm{i}\eta)\,\mathrm{d}t. μ ( E ) = η ↓ 0 lim π 1 ∫ E Im s ( t + i η ) d t .
The following elementary smoothing inequality bounds μ ( E ) \mu(E) μ ( E ) at finite η \eta η with explicit error terms. A related estimate for eigenvalue counts near the spectral edge follows from the local law in the proof of Erdős et al. [41 Lemma 6.1] . Additionally, let the inflated and deflated intervals be E h + = [ a − h , b + h ] E_h^+=[a-h,b+h] E h + = [ a − h , b + h ] and E h − = [ a + h , b − h ] E_h^-=[a+h,b-h] E h − = [ a + h , b − h ] for 0 < h < ( b − a ) / 2 0<h<(b-a)/2 0 < h < ( b − a ) /2 .
Lemma A.5. Let s ( z ) s(z) s ( z ) be the Stieltjes transform of a probability measure μ \mu μ on R \mathbb{R} R , and E = [ a , b ] E=[a,b] E = [ a , b ] for a < b a<b a < b . For E h ± E_h^{\pm} E h ± defined above, it holds for any 0 < η < π h / 2 0<\eta<\pi h/2 0 < η < π h /2 that
1 π ∫ E h − Im s ( t + i η ) d t − η π h 1 − η π h ≤ μ ( E ) ≤ 1 π ∫ E h + Im s ( t + i η ) d t 1 − 2 η π h . \frac{\frac{1}{\pi}\int_{E_h^-}\operatorname{Im}s(t+\mathrm{i}\eta)\,\mathrm{d}t-\frac{\eta}{\pi h}}{1-\frac{\eta}{\pi h}}
\leq\mu(E) \leq
\frac{\frac{1}{\pi}\int_{E_h^+}\operatorname{Im}s(t+\mathrm{i}\eta)\,\mathrm{d}t}{1-\frac{2\eta}{\pi h}}. 1 − π h η π 1 ∫ E h − Im s ( t + i η ) d t − π h η ≤ μ ( E ) ≤ 1 − π h 2 η π 1 ∫ E h + Im s ( t + i η ) d t .
Proof. By Fubini,
1 π ∫ E Im s ( t + i η ) d t = ∫ − ∞ + ∞ ( ∫ E η π ( ( x − t ) 2 + η 2 ) d x ) μ ( d t ) . \frac{1}{\pi}\int_{E}\operatorname{Im}s(t+i\eta)\,dt=\int_{-\infty}^{+\infty}\left(\int_{E}\frac{\eta}{\pi((x-t)^2+\eta^2)}\,dx\right)\mu(dt). π 1 ∫ E Im s ( t + i η ) d t = ∫ − ∞ + ∞ ( ∫ E π (( x − t ) 2 + η 2 ) η d x ) μ ( d t ) .
We use the fact that for any t ∈ E t\in E t ∈ E ,
∫ E h + η π ( ( x − t ) 2 + η 2 ) d x ≥ 1 − 1 π ∫ ∣ x ∣ ≥ h η x 2 + η 2 d x ≥ 1 − 2 π arctan η h ≥ 1 − 2 η π h , \int_{E_h^+}\frac{\eta}{\pi((x-t)^2+\eta^2)}\,dx\geq1-\frac{1}{\pi}\int_{\lvert x\rvert\geq h}\frac{\eta}{x^2+\eta^2}\,dx\geq1-\frac{2}{\pi}\arctan\frac{\eta}{h}\geq1-\frac{2\eta}{\pi h}, ∫ E h + π (( x − t ) 2 + η 2 ) η d x ≥ 1 − π 1 ∫ ∣ x ∣ ≥ h x 2 + η 2 η d x ≥ 1 − π 2 arctan h η ≥ 1 − π h 2 η ,
and
1 π ∫ E h + Im s ( t + i η ) d t ≥ ∫ − ∞ + ∞ ( 1 − 2 η π h ) 1 { t ∈ E } μ ( d t ) = ( 1 − 2 η π h ) μ ( E ) . \frac{1}{\pi}\int_{E_h^+}\operatorname{Im}s(t+i\eta)\,dt\geq\int_{-\infty}^{+\infty}\left(1-\frac{2\eta}{\pi h}\right)\mathbf{1}\{t\in E\}\,\mu(dt)=\left(1-\frac{2\eta}{\pi h}\right)\mu(E). π 1 ∫ E h + Im s ( t + i η ) d t ≥ ∫ − ∞ + ∞ ( 1 − π h 2 η ) 1 { t ∈ E } μ ( d t ) = ( 1 − π h 2 η ) μ ( E ) .
Similarly for t ∉ E t\notin E t ∈ / E ,
∫ E h − η π ( ( x − t ) 2 + η 2 ) d x ≤ 1 π ∫ ∣ x ∣ ≥ h η x 2 + η 2 d x ≤ η π h , \int_{E_h^-}\frac{\eta}{\pi((x-t)^2+\eta^2)}\,dx\leq\frac{1}{\pi}\int_{\lvert x\rvert\geq h}\frac{\eta}{x^2+\eta^2}\,dx\leq\frac{\eta}{\pi h}, ∫ E h − π (( x − t ) 2 + η 2 ) η d x ≤ π 1 ∫ ∣ x ∣ ≥ h x 2 + η 2 η d x ≤ π h η ,
and
1 π ∫ E h − Im s ( t + i η ) d t ≤ ∫ − ∞ + ∞ ( η π h 1 { t ∉ E } + 1 { t ∈ E } ) μ ( d t ) = η π h ( 1 − μ ( E ) ) + μ ( E ) . \frac{1}{\pi}\int_{E_h^-}\operatorname{Im}s(t+i\eta)\,dt\leq\int_{-\infty}^{+\infty}\left(\frac{\eta}{\pi h}\mathbf{1}\{t\notin E\}+\mathbf{1}\{t\in E\}\right)\mu(dt)=\frac{\eta}{\pi h}(1-\mu(E))+\mu(E). π 1 ∫ E h − Im s ( t + i η ) d t ≤ ∫ − ∞ + ∞ ( π h η 1 { t ∈ / E } + 1 { t ∈ E } ) μ ( d t ) = π h η ( 1 − μ ( E )) + μ ( E ) .
The proof is complete.
In § 2.3 , we bound the approximation error between a spectral filter’s prediction risk and its deterministic equivalent, under the hypothesis of a spectral event { σ ^ n ≥ η n } \{\widehat{\sigma}_n\geq\eta_n\} { σ n ≥ η n } , where n σ ^ n n\widehat{\sigma}_n n σ n is the smallest eigenvalue of the unnormalized sample Gram matrix X X ∗ \mathbf{X}\mathbf{X}^{*} X X ∗ . Indeed, under the quadratic form concentration Assumption 1 of Hanson-Wright, we can already derive a sufficiently informative estimate for the power law and critical law decay regimes, as stated in Lemma A.6. We understand d = ∞ d=\infty d = ∞ and σ i = 0 \sigma_i=0 σ i = 0 for i i i beyond Σ \boldsymbol{\Sigma} Σ ’s rank in what follows. See its proof in Appendix § 9 .§ 2 .
Lemma A.6 (Smallest eigenvalue bound). Under Assumption 1, there exists C τ = C τ ( C x ) > 0 C_{\tau}=C_{\tau}(C_x)>0 C τ = C τ ( C x ) > 0 such that, for every deterministic integer m ≥ 0 m\geq0 m ≥ 0 and u ≥ 0 u\geq0 u ≥ 0 , with probability at least 1 − 2 e − u 1-2e^{-u} 1 − 2 e − u ,
n σ ^ n ≥ τ 1 ( m ) − C τ { ( n + u ) τ 2 ( m ) + ( n + u ) σ m + 1 } . (34) n\widehat{\sigma}_n\geq\tau_1(m)-C_{\tau}\left\{\sqrt{(n+u)\tau_2(m)}+(n+u)\sigma_{m+1}\right\}. \tag*{(34)} n σ n ≥ τ 1 ( m ) − C τ { ( n + u ) τ 2 ( m ) + ( n + u ) σ m + 1 } . (34) In particular, if τ 1 ( m ) ≥ 2 C τ { 2 n τ 2 ( m ) + 2 n σ m + 1 } \tau_1(m)\geq2C_{\tau}\left\{\sqrt{2n\tau_2(m)}+2n\sigma_{m+1}\right\} τ 1 ( m ) ≥ 2 C τ { 2 n τ 2 ( m ) + 2 n σ m + 1 } , then P ( σ ^ n ≥ τ 1 ( m ) / ( 2 n ) ) ≥ 1 − 2 e − n \mathbb{P}\left(\widehat{\sigma}_n\geq\tau_1(m)/(2n)\right)\geq1-2e^{-n} P ( σ n ≥ τ 1 ( m ) / ( 2 n ) ) ≥ 1 − 2 e − n .
Proof of Lemma A.4
Part I: Unique analytic solution as the Stieltjes transform. We first establish existence, uniqueness, and analyticity by applying a global inversion theorem. If Σ = 0 \boldsymbol{\Sigma}=0 Σ = 0 , then s = − 1 / z s=-1/z s = − 1/ z is the unique analytic solution. Otherwise, substituting u = − 1 / s u=-1/s u = − 1/ s transforms Eq. (8) into H ( u ) = z H(u)=z H ( u ) = z , where
H ( u ) : = u + ∑ j σ j 2 u u − n σ j = u + Tr Σ + n ∑ j σ j 2 u − n σ j , u ∈ C + . H(u):=u+\sum_j\frac{\sigma_j^2u}{u-n\sigma_j}=u+\operatorname{Tr}\boldsymbol{\Sigma}+n\sum_j\frac{\sigma_j^2}{u-n\sigma_j},\qquad u\in\mathbb{C}_+. H ( u ) := u + j ∑ u − n σ j σ j 2 u = u + Tr Σ + n j ∑ u − n σ j σ j 2 , u ∈ C + .
To apply Biane [16, Lemma 4, p. 710], define the probability measure ρ : = Tr ( Σ 2 ) − 1 ∑ j σ j 2 δ n σ j \rho:=\operatorname{Tr}(\boldsymbol{\Sigma}^2)^{-1}\sum_j\sigma_j^2\delta_{n\sigma_j} ρ := Tr ( Σ 2 ) − 1 ∑ j σ j 2 δ n σ j , which is well-defined because 0 < Tr ( Σ 2 ) ≤ ∥ Σ ∥ Tr Σ < ∞ 0<\operatorname{Tr}(\boldsymbol{\Sigma}^2)\leq\lVert\boldsymbol{\Sigma}\rVert\operatorname{Tr}\boldsymbol{\Sigma}<\infty 0 < Tr ( Σ 2 ) ≤ ∥ Σ ∥ Tr Σ < ∞ . We have
H ( u ) − Tr Σ = u + n Tr ( Σ 2 ) ∫ R ρ ( d x ) u − x . (35) H(u)-\operatorname{Tr}\boldsymbol{\Sigma}=u+n\operatorname{Tr}(\boldsymbol{\Sigma}^2)\int_{\mathbb{R}}\frac{\rho(dx)}{u-x}. \tag*{(35)} H ( u ) − Tr Σ = u + n Tr ( Σ 2 ) ∫ R u − x ρ ( d x ) . (35)
This is Biane’s map with probability measure ρ \rho ρ and variance parameter n Tr ( Σ 2 ) n\operatorname{Tr}(\Sigma^{2}) n Tr ( Σ 2 ) , whose domain is easily checked to be { u ∈ C + : Im H ( u ) > 0 } \{u \in\mathbb{C}_{+} : \operatorname{Im} H(u) > 0\} { u ∈ C + : Im H ( u ) > 0 } . Applying that lemma shows that H − Tr Σ H-\operatorname{Tr}\Sigma H − Tr Σ maps this domain conformally onto C + \mathbb{C}_{+} C + . Since translation by the real number Tr Σ \operatorname{Tr}\Sigma Tr Σ preserves C + \mathbb{C}_{+} C + , the same holds for H H H , which therefore has an analytic inverse. Thus s n ( z ) = − 1 / H − 1 ( z ) ∈ C + s_n(z)=-1/H^{-1}(z) \in\mathbb{C}_{+} s n ( z ) = − 1/ H − 1 ( z ) ∈ C + solves Eq. (8) . Every solution s ∈ C + s \in\mathbb{C}_{+} s ∈ C + corresponds to u = − 1 / s u=-1/s u = − 1/ s in this domain. Hence the solution is unique and s n ( z ) = − 1 / H − 1 ( z ) s_n(z)=-1/H^{-1}(z) s n ( z ) = − 1/ H − 1 ( z ) is analytic on C + \mathbb{C}_{+} C + .
We next identify the solution as a Stieltjes transform and prove the bounds. Taking imaginary parts in Eq. (8) implies ( Im z ) ∣ s n ( z ) ∣ 2 ≤ Im s n ( z ) ≤ ∣ s n ( z ) ∣ (\operatorname{Im} z)|s_n(z)|^2 \leq\operatorname{Im} s_n(z) \leq|s_n(z)| ( Im z ) ∣ s n ( z ) ∣ 2 ≤ Im s n ( z ) ≤ ∣ s n ( z ) ∣ , and hence ∣ s n ( z ) ∣ ≤ ( Im z ) − 1 |s_n(z)| \leq(\operatorname{Im} z)^{-1} ∣ s n ( z ) ∣ ≤ ( Im z ) − 1 , since s n ( z ) ≠ 0 s_n(z) \ne0 s n ( z ) = 0 . For y > 2 n ∥ Σ ∥ y > 2n\|\Sigma\| y > 2 n ∥Σ∥ , we have ∣ n σ j s n ( i y ) ∣ ≤ 1 / 2 |n\sigma_j s_n(iy)| \leq1/2 ∣ n σ j s n ( i y ) ∣ ≤ 1/2 , and hence ∣ ∑ j σ j / ( 1 + n σ j s n ( i y ) ) ∣ ≤ 2 Tr Σ \left|\sum_j \sigma_j/(1+n\sigma_j s_n(iy))\right| \leq2\operatorname{Tr}\Sigma ∑ j σ j / ( 1 + n σ j s n ( i y )) ≤ 2 Tr Σ . Multiplying Eq. (8) by s n ( i y ) s_n(iy) s n ( i y ) at z = i y z=iy z = i y therefore implies
∣ i y s n ( i y ) + 1 ∣ = ∣ s n ( i y ) ∑ j σ j 1 + n σ j s n ( i y ) ∣ ≤ 2 Tr Σ y ⟶ 0 as y → + ∞ . |iy s_n(iy)+1|=\left|s_n(iy)\sum_j\frac{\sigma_j}{1+n\sigma_j s_n(iy)}\right|\leq\frac{2\operatorname{Tr}\Sigma}{y}\longrightarrow0 \quad\text{as } y\to+\infty. ∣ i y s n ( i y ) + 1∣ = s n ( i y ) j ∑ 1 + n σ j s n ( i y ) σ j ≤ y 2 Tr Σ ⟶ 0 as y → + ∞.
Thus y s n ( i y ) → i y s_n(iy)\to i y s n ( i y ) → i as y → ∞ y\to\infty y → ∞ , with n n n and Σ \Sigma Σ fixed. Applying Lemma A.3 to the analytic map s n : C + → C + s_n:\mathbb{C}_{+}\to\mathbb{C}_{+} s n : C + → C + yields a Herglotz representation. In this paragraph, let a ∈ R a\in\mathbb{R} a ∈ R , b ≥ 0 b\geq0 b ≥ 0 , and μ \mu μ denote its coefficients and measure. Taking imaginary parts at z = i y z=iy z = i y and using the preceding limit implies
y Im s n ( i y ) = b y 2 + ∫ R y 2 t 2 + y 2 μ ( d t ) ⟶ 1 as y → + ∞ . y\operatorname{Im}s_n(iy)=by^2+\int_{\mathbb{R}}\frac{y^2}{t^2+y^2}\,\mu(\mathrm{d}t)\longrightarrow1 \quad\text{as } y\to+\infty. y Im s n ( i y ) = b y 2 + ∫ R t 2 + y 2 y 2 μ ( d t ) ⟶ 1 as y → + ∞.
Both terms on the right are nonnegative, so b = 0 b=0 b = 0 . Since y 2 / ( t 2 + y 2 ) y^2/(t^2+y^2) y 2 / ( t 2 + y 2 ) increases to 1 1 1 as y → ∞ y\to\infty y → ∞ , applying monotone convergence then yields μ ( R ) = 1 \mu(\mathbb{R})=1 μ ( R ) = 1 . In particular,
∣ ∫ R 1 t − i y μ ( d t ) ∣ ≤ 1 y ⟶ 0 as y → + ∞ . \left|\int_{\mathbb{R}}\frac{1}{t-iy}\,\mu(\mathrm{d}t)\right|\leq\frac{1}{y}\longrightarrow0 \quad\text{as } y\to+\infty. ∫ R t − i y 1 μ ( d t ) ≤ y 1 ⟶ 0 as y → + ∞.
We obtain by comparing the above display with s n ( i y ) → 0 s_n(iy)\to0 s n ( i y ) → 0 as y → ∞ y\to\infty y → ∞ in the Herglotz representation that a − ∫ R t ( 1 + t 2 ) − 1 μ ( d t ) = 0 a-\int_{\mathbb{R}}t(1+t^2)^{-1}\,\mu(\mathrm{d}t)=0 a − ∫ R t ( 1 + t 2 ) − 1 μ ( d t ) = 0 . This last integral is finite as ∣ t ∣ / ( 1 + t 2 ) ≤ 1 / 2 |t|/(1+t^2)\leq1/2 ∣ t ∣/ ( 1 + t 2 ) ≤ 1/2 and μ ( R ) = 1 \mu(\mathbb{R})=1 μ ( R ) = 1 . Consequently, we can establish s n ( z ) s_n(z) s n ( z ) is a Stieltjes transform:
s n ( z ) = ∫ R 1 t − z μ ( d t ) . s_n(z)=\int_{\mathbb{R}}\frac{1}{t-z}\,\mu(\mathrm{d}t). s n ( z ) = ∫ R t − z 1 μ ( d t ) .
Part II: Analytic extension and quantitative bounds of s n ( z ) s_n(z) s n ( z ) . We next verify that μ \mu μ has no negative support so that we can extend analytically to ( − ∞ , 0 ) (-\infty,0) ( − ∞ , 0 ) for s n ( z ) s_n(z) s n ( z ) . For a fixed x < 0 x<0 x < 0 ,
1 = ( − x ) s + ∑ j σ j s 1 + n σ j s , s > 0 , 1=(-x)s+\sum_j\frac{\sigma_j s}{1+n\sigma_j s},\qquad s>0, 1 = ( − x ) s + j ∑ 1 + n σ j s σ j s , s > 0 ,
has a unique solution because the right-hand side is continuous and strictly increasing from 0 0 0 to + ∞ +\infty + ∞ as s s s ranges over ( 0 , ∞ ) (0,\infty) ( 0 , ∞ ) . To apply the analytic inverse function theorem, we verify that the derivative of the right-hand side of Eq. (8) is positive at this solution. Indeed,
n ∑ j σ j 2 s 2 ( 1 + n σ j s ) 2 ≤ ∑ j σ j s 1 + n σ j s = 1 + x s < 1. n\sum_j\frac{\sigma_j^2s^2}{(1+n\sigma_j s)^2}\leq\sum_j\frac{\sigma_j s}{1+n\sigma_j s}=1+xs<1. n j ∑ ( 1 + n σ j s ) 2 σ j 2 s 2 ≤ j ∑ 1 + n σ j s σ j s = 1 + x s < 1.
Consequently, at this solution,
d d s ( − s − 1 + ∑ j σ j 1 + n σ j s ) = 1 s 2 ( 1 − n ∑ j σ j 2 s 2 ( 1 + n σ j s ) 2 ) ≥ − x s > 0. \frac{\mathrm{d}}{\mathrm{d}s}\left(-s^{-1}+\sum_j\frac{\sigma_j}{1+n\sigma_j s}\right)=\frac{1}{s^2}\left(1-n\sum_j\frac{\sigma_j^2s^2}{(1+n\sigma_j s)^2}\right)\geq-\frac{x}{s}>0. d s d ( − s − 1 + j ∑ 1 + n σ j s σ j ) = s 2 1 ( 1 − n j ∑ ( 1 + n σ j s ) 2 σ j 2 s 2 ) ≥ − s x > 0.
Applying the analytic inverse function theorem therefore supplies a local inverse with a positive imaginary part just above x x x . By uniqueness in C + \mathbb{C}_{+} C + , this inverse agrees with s n s_n s n . Hence, s n s_n s n extends analytically and is real on ( − ∞ , 0 ) (-\infty,0) ( − ∞ , 0 ) . Applying Stieltjes inversion on compact negative intervals gives μ ( ( − ∞ , 0 ) ) = 0 \mu((-\infty,0))=0 μ (( − ∞ , 0 )) = 0 . The representation now yields the following bounds for every z , z ′ ∈ C + ∪ R < 0 z,z'\in\mathbb{C}_{+}\cup\mathbb{R}_{<0} z , z ′ ∈ C + ∪ R < 0 , since μ \mu μ is a probability measure supported on [ 0 , ∞ ) [0,\infty) [ 0 , ∞ ) :
∣ s n ( z ) ∣ ≤ 1 ∣ z ∣ and ∣ s n ( z ′ ) − s n ( z ) ∣ ≤ ∣ z ′ − z ∣ ∫ [ 0 , ∞ ) μ ( d t ) ∣ t − z ′ ∣ ∣ t − z ∣ ≤ ∣ z ′ − z ∣ ∣ z ′ ∣ ∣ z ∣ . \lvert s_n(z)\rvert\leq\frac{1}{\lvert z\rvert}
\qquad\text{and}\qquad
\lvert s_n(z')-s_n(z)\rvert\leq\lvert z'-z\rvert\int_{[0,\infty)}\frac{\mu(\mathrm{d}t)}{\lvert t-z'\rvert\lvert t-z\rvert}\leq\frac{\lvert z'-z\rvert}{\lvert z'\rvert\lvert z\rvert}. ∣ s n ( z )∣ ≤ ∣ z ∣ 1 and ∣ s n ( z ′ ) − s n ( z )∣ ≤ ∣ z ′ − z ∣ ∫ [ 0 , ∞ ) ∣ t − z ′ ∣ ∣ t − z ∣ μ ( d t ) ≤ ∣ z ′ ∣ ∣ z ∣ ∣ z ′ − z ∣ .
Finally, substituting s = − ( n ω ( z / n ) ) − 1 s=-(n\omega(z/n))^{-1} s = − ( nω ( z / n ) ) − 1 into Eq. (8) gives Eq. (6) . Its uniqueness therefore identifies s n ( z ) = − ( n ω ( z / n ) ) − 1 s_n(z)=-(n\omega(z/n))^{-1} s n ( z ) = − ( nω ( z / n ) ) − 1 and completes the proof.
Proof of Lemma A.6
We bound the least eigenvalue by projecting each x i x_i x i onto the tail eigenspaces indexed by j > m j>m j > m , using the orthogonal projection Q m : = I − ∑ j = 1 m u j u j ∗ Q_m:=I-\sum_{j=1}^{m}u_ju_j^{*} Q m := I − ∑ j = 1 m u j u j ∗ . Let S ∈ R n × n S\in\mathbb{R}^{n\times n} S ∈ R n × n have entries S i j : = 1 { i ≠ j } ⟨ Q m x i , Q m x j ⟩ S_{ij}:=\mathbf{1}\{i\neq j\}\langle Q_mx_i,Q_mx_j\rangle S ij := 1 { i = j } ⟨ Q m x i , Q m x j ⟩ . Since X X ∗ ⪰ X Q m X ∗ = ( ⟨ Q m x i , Q m x j ⟩ ) i , j ≤ n XX^{*}\succeq XQ_mX^{*}=(\langle Q_mx_i,Q_mx_j\rangle)_{i,j\leq n} X X ∗ ⪰ X Q m X ∗ = (⟨ Q m x i , Q m x j ⟩ ) i , j ≤ n , separating the diagonal and off-diagonal terms yields
n σ ^ n = λ min ( X X ∗ ) ≥ min i ≤ n ∥ Q m x i ∥ 2 − ∥ S ∥ ≥ τ 1 ( m ) − max i ≤ n ∣ ∥ Q m x i ∥ 2 − τ 1 ( m ) ∣ − ∥ S ∥ . n\widehat{\sigma}_n=\lambda_{\min}(XX^{*})\geq\min_{i\leq n}\lVert Q_mx_i\rVert^{2}-\lVert S\rVert\geq\tau_1(m)-\max_{i\leq n}\left\lvert\lVert Q_mx_i\rVert^{2}-\tau_1(m)\right\rvert-\lVert S\rVert. n σ n = λ m i n ( X X ∗ ) ≥ i ≤ n min ∥ Q m x i ∥ 2 − ∥ S ∥ ≥ τ 1 ( m ) − i ≤ n max ∥ Q m x i ∥ 2 − τ 1 ( m ) − ∥ S ∥ .
It suffices to control the diagonal fluctuations and ∥ S ∥ \lVert S\rVert ∥ S ∥ separately. Applying Proposition 1 with A = Q m A=Q_m A = Q m and taking a union bound yields control for the second term immediately, for t > 0 t>0 t > 0 ,
P ( max i ≤ n ∣ ∥ Q m x i ∥ 2 − τ 1 ( m ) ∣ > t ) ≤ 2 n exp { − Ω C x ( t 2 τ 2 ( m ) ∧ t σ m + 1 ) } . \mathbb{P}\left(\max_{i\leq n}\left\lvert\lVert Q_mx_i\rVert^{2}-\tau_1(m)\right\rvert>t\right)\leq2n\exp\left\{-\Omega_{C_x}\left(\frac{t^2}{\tau_2(m)}\wedge\frac{t}{\sigma_{m+1}}\right)\right\}. P ( i ≤ n max ∥ Q m x i ∥ 2 − τ 1 ( m ) > t ) ≤ 2 n exp { − Ω C x ( τ 2 ( m ) t 2 ∧ σ m + 1 t ) } .
It remains to control ∥ S ∥ \lVert S\rVert ∥ S ∥ . We will use the standard ϵ \epsilon ϵ -net argument, and start with the quadratic forms of S S S . We use the Hilbert-space Hanson-Wright inequality [25 Proposition 2.5] , based on the decoupling argument in [68 ] . The vectors Q m x i Q_mx_i Q m x i are independent and centered, with positive-definite, trace-class covariance Σ > m : = Q m Σ Q m \Sigma_{>m}:=Q_m\Sigma Q_m Σ > m := Q m Σ Q m on the closed span of the nonzero tail eigenvectors. Applying the rank-one case of Proposition 1 shows that, for every h h h in this space, E e ⟨ h , Q m x i ⟩ ≤ exp { O C x ( h ∗ Σ > m h ) } \mathbb{E}e^{\langle h,Q_mx_i\rangle}\leq\exp\{O_{C_x}(h^{*}\Sigma_{>m}h)\} E e ⟨ h , Q m x i ⟩ ≤ exp { O C x ( h ∗ Σ > m h )} , verifying the required sub-Gaussian assumption. For each unit v ∈ R n v\in\mathbb{R}^{n} v ∈ R n , apply the cited inequality with covariance proxy Σ > m \Sigma_{>m} Σ > m and coefficient matrices ± v v ∗ \pm vv^{*} ± v v ∗ , whose operator and Frobenius norms equal one. Substituting ∥ Σ > m ∥ = σ m + 1 \lVert\Sigma_{>m}\rVert=\sigma_{m+1} ∥ Σ > m ∥ = σ m + 1 and ∥ Σ > m ∥ F 2 = τ 2 ( m ) \lVert\Sigma_{>m}\rVert_{F}^{2}=\tau_2(m) ∥ Σ > m ∥ F 2 = τ 2 ( m ) yields
P ( ∣ v ∗ S v ∣ > t ) = P ( ∣ ∑ i ≠ j v i v j ⟨ Q m x i , Q m x j ⟩ ∣ > t ) ≤ 2 exp { − Ω C x ( t 2 τ 2 ( m ) ∧ t σ m + 1 ) } . \mathbb{P}(\lvert v^{*}Sv\rvert>t)=\mathbb{P}\left(\left\lvert\sum_{i\neq j}v_iv_j\langle Q_mx_i,Q_mx_j\rangle\right\rvert>t\right)\leq2\exp\left\{-\Omega_{C_x}\left(\frac{t^2}{\tau_2(m)}\wedge\frac{t}{\sigma_{m+1}}\right)\right\}. P (∣ v ∗ S v ∣ > t ) = P i = j ∑ v i v j ⟨ Q m x i , Q m x j ⟩ > t ≤ 2 exp { − Ω C x ( τ 2 ( m ) t 2 ∧ σ m + 1 t ) } .
We close the proof by the standard net argument. Using a 1 / 4 1/4 1/4 -net with at most 9 n 9^n 9 n points and the union bound yields
P ( n σ ^ n < τ 1 ( m ) − 3 t ) ≤ 2 ( n + 9 n ) exp { − Ω C x ( t 2 τ 2 ( m ) ∧ t σ m + 1 ) } . \mathbb{P}\left(n\widehat{\sigma}_n<\tau_1(m)-3t\right)\leq2(n+9^n)\exp\left\{-\Omega_{C_x}\left(\frac{t^2}{\tau_2(m)}\wedge\frac{t}{\sigma_{m+1}}\right)\right\}. P ( n σ n < τ 1 ( m ) − 3 t ) ≤ 2 ( n + 9 n ) exp { − Ω C x ( τ 2 ( m ) t 2 ∧ σ m + 1 t ) } .
Taking 3 t = C τ { ( n + u ) τ 2 ( m ) + ( n + u ) σ m + 1 } 3t=C_{\tau}\left\{\sqrt{(n+u)\tau_2(m)}+(n+u)\sigma_{m+1}\right\} 3 t = C τ { ( n + u ) τ 2 ( m ) + ( n + u ) σ m + 1 } with C τ = C τ ( C x ) C_{\tau}=C_{\tau}(C_x) C τ = C τ ( C x ) sufficiently large absorbs the union-bound factor, since log ( n + 9 n ) = O ( n ) \log(n+9^n)=O(n) log ( n + 9 n ) = O ( n ) , and bounds the right-hand side by 2 e − u 2e^{-u} 2 e − u , proving Eq. (34) . The final assertion follows by taking u = n u=n u = n .
Proofs for Section 2.2: Spectral laws
Proof of Theorem 1
Part I: Stieltjes transform. The proof follows from the backward path { w k } \{w_k\} { w k } and relative-resolvent traces { H k } \{H_k\} { H k } in § 5 , which is made clear by Eqs. (26) and (8) . Indeed, the previous equations identify
H n ( z , 0 ) / n = 1 + z s ^ n ( z ) and H 0 ( z , w 0 l i n ) / n = 1 + z s n ( z ) . H_n(z,0)/n = 1 + z\widehat{s}_n(z) \qquad\text{and} \qquad H_0(z,w_0^{\mathrm{lin}})/n = 1 + zs_n(z). H n ( z , 0 ) / n = 1 + z s n ( z ) and H 0 ( z , w 0 lin ) / n = 1 + z s n ( z ) .
We then obtain directly from Theorem 4 with r = 0 r=0 r = 0 and Q = I \mathbf{Q}=\mathbf{I} Q = I that
∥ s ^ n ( z ) − s n ( z ) ∥ L p = ∥ H n ( z , 0 ) − H 0 ( z , w 0 l i n ) ∥ L p n ∣ z ∣ = O C x , ν , κ ( p 2 Λ n ( z ) 23 n ∣ z ∣ ) . \left\|\widehat{s}_n(z)-s_n(z)\right\|_{L^p} = \frac{\left\|H_n(z,0)-H_0(z,w_0^{\mathrm{lin}})\right\|_{L^p}}{n|z|} = O_{C_x,\nu,\kappa}\left(\frac{p^2\Lambda_n(z)^{23}}{n|z|}\right). ∥ s n ( z ) − s n ( z ) ∥ L p = n ∣ z ∣ H n ( z , 0 ) − H 0 ( z , w 0 lin ) L p = O C x , ν , κ ( n ∣ z ∣ p 2 Λ n ( z ) 23 ) .
A slightly more refined analysis using the proof details can sharpen the exponent on Λ n ( z ) \Lambda_n(z) Λ n ( z ) , using Lemma E.5 and the subsequent Eq. (64a) to express H n ( z , 0 ) − H 0 ( z , w 0 l i n ) H_n(z,0)-H_0(z,w_0^{\mathrm{lin}}) H n ( z , 0 ) − H 0 ( z , w 0 lin ) as the sum of the martingale and drift terms with indices 1 ≤ k ≤ n 1\leq k\leq n 1 ≤ k ≤ n and 0 ≤ k ≤ n 0\leq k\leq n 0 ≤ k ≤ n , respectively. In the proof of Lemma E.6 , Eqs. (121) and (128) , bound the martingale and drift sums in L p L^p L p by O ( p 2 Λ n ( z ) 9 ) O(p^2\Lambda_n(z)^9) O ( p 2 Λ n ( z ) 9 ) and O ( p 2 Λ n ( z ) 17 ) O(p^2\Lambda_n(z)^{17}) O ( p 2 Λ n ( z ) 17 ) , respectively. For k = 0 k=0 k = 0 , Theorem 3.(iii) and Eqs. (75) and (76) in its proof imply for the initial drift β 0 H , I = w 0 − w 0 l i n + ( w 0 / z ) ε n ( z ) = O ( Λ n ( z ) 17 ) \beta_0^{H,I}=w_0-w_0^{\mathrm{lin}}+(w_0/z)\varepsilon_n(z)=O(\Lambda_n(z)^{17}) β 0 H , I = w 0 − w 0 lin + ( w 0 / z ) ε n ( z ) = O ( Λ n ( z ) 17 ) , using ∣ w 0 ∣ ≤ n Λ n ( z ) |w_0|\leq n\Lambda_n(z) ∣ w 0 ∣ ≤ n Λ n ( z ) as in Part IV-i of that proof. Dividing the sum of these bounds by n ∣ z ∣ n|z| n ∣ z ∣ yields the strengthened bound in Eq. (11a).
Part II: Interval mass. We transfer the L p L^p L p bound on Stieltjes transforms to their probability measure masses. Fix E E E and h h h as in Theorem 1.(ii). To compare interval masses by Lemma A.5 , it suffices to control the integrated transform error Δ ( E h + ) : = π − 1 ∫ a − h b + h ∣ s ^ n ( t + i η ) − s n ( t + i η ) ∣ d t \Delta(E_h^+):=\pi^{-1}\int_{a-h}^{b+h}|\widehat{s}_n(t+i\eta)-s_n(t+i\eta)|\,dt Δ ( E h + ) := π − 1 ∫ a − h b + h ∣ s n ( t + i η ) − s n ( t + i η ) ∣ d t . For p = κ log n p=\kappa\log n p = κ log n , applying Minkowski’s integral inequality and then Part I at each t + i η t+i\eta t + i η , we have
∥ Δ ( E h + ) ∥ L p ≤ 1 π ∫ a − h b + h ∥ s ^ n ( t + i η ) − s n ( t + i η ) ∥ L p d t ≤ O C x , ν , κ ( ( log n ) 2 n ∫ a − h b + h Λ n ( t + i η ) 17 ∣ t + i η ∣ d t ) . \left\|\Delta(E_h^+)\right\|_{L^p}\leq\frac{1}{\pi}\int_{a-h}^{b+h}\left\|\widehat{s}_n(t+i\eta)-s_n(t+i\eta)\right\|_{L^p}\,dt\leq O_{C_x,\nu,\kappa}\left(\frac{(\log n)^2}{n}\int_{a-h}^{b+h}\frac{\Lambda_n(t+i\eta)^{17}}{|t+i\eta|}\,dt\right). Δ ( E h + ) L p ≤ π 1 ∫ a − h b + h ∥ s n ( t + i η ) − s n ( t + i η ) ∥ L p d t ≤ O C x , ν , κ ( n ( log n ) 2 ∫ a − h b + h ∣ t + i η ∣ Λ n ( t + i η ) 17 d t ) .
Applying Markov’s inequality to Δ ( E h + ) \Delta(E_h^+) Δ ( E h + ) , it holds Δ ( E h + ) ≤ e D / κ ∥ Δ ( E h + ) ∥ L p \Delta(E_h^+)\leq e^{D/\kappa}\left\|\Delta(E_h^+)\right\|_{L^p} Δ ( E h + ) ≤ e D / κ Δ ( E h + ) L p with probability at least 1 − n − D 1-n^{-D} 1 − n − D .
If η / h > π / 16 \eta/h>\pi/16 η / h > π /16 , Eq. (11b) follows from ∣ μ ^ n ( E ) − μ n ( E ) ∣ ≤ 1 |\widehat{\mu}_n(E)-\mu_n(E)|\leq1 ∣ μ n ( E ) − μ n ( E ) ∣ ≤ 1 . Otherwise, we apply Lemma A.5 to μ ^ n \widehat{\mu}_n μ n on E E E with deflation/inflation width h / 2 h/2 h /2 , which implies
1 π ∫ E h / 2 − Im s ^ n ( t + i η ) d t − O ( η / h ) ≤ μ ^ n ( E ) ≤ 1 π ∫ E h / 2 + Im s ^ n ( t + i η ) d t + O ( η / h ) . \frac{1}{\pi}\int_{E_{h/2}^-}\operatorname{Im}\widehat{s}_n(t+i\eta)\,dt-O(\eta/h)\leq\widehat{\mu}_n(E)\leq\frac{1}{\pi}\int_{E_{h/2}^+}\operatorname{Im}\widehat{s}_n(t+i\eta)\,dt+O(\eta/h). π 1 ∫ E h /2 − Im s n ( t + i η ) d t − O ( η / h ) ≤ μ n ( E ) ≤ π 1 ∫ E h /2 + Im s n ( t + i η ) d t + O ( η / h ) .
The condition η ≤ π h / 16 \eta\leq\pi h/16 η ≤ π h /16 keeps both denominators in the lemma bounded away from zero, and each smoothed mass is at most one. To compare with μ n \mu_n μ n , we replace s ^ n \widehat{s}_n s n with s n s_n s n at a cost of at most Δ ( E h + ) \Delta(E_h^+) Δ ( E h + ) in either integral, since E h / 2 − ⊂ E h + E_{h/2}^-\subset E_h^+ E h /2 − ⊂ E h + , and use the same Poisson tail estimate as in the proof of the lemma. For x ∈ E h − x\in E_h^- x ∈ E h − , the two tails outside E h / 2 − E_{h/2}^- E h /2 − each begin at distance at least h / 2 h/2 h /2 from x x x ; and for x ∉ E h + x\notin E_h^+ x ∈ / E h + , the interval E h / 2 + E_{h/2}^+ E h /2 + lies in one such tail. Each tail has mass at most π − 1 ∫ h / 2 ∞ η t − 2 d t = 2 η / ( π h ) \pi^{-1}\int_{h/2}^{\infty}\eta t^{-2}\,dt=2\eta/(\pi h) π − 1 ∫ h /2 ∞ η t − 2 d t = 2 η / ( π h ) . Integrating against μ n \mu_n μ n therefore implies
( 1 − 4 η π h ) μ n ( E h − ) ≤ 1 π ∫ E h / 2 − Im s n ( t + i η ) d t and 1 π ∫ E h / 2 + Im s n ( t + i η ) d t ≤ μ n ( E h + ) + 2 η π h . \left(1-\frac{4\eta}{\pi h}\right)\mu_n(E_h^-)\leq\frac{1}{\pi}\int_{E_{h/2}^-}\operatorname{Im}s_n(t+i\eta)\,dt
\qquad\text{and}\qquad
\frac{1}{\pi}\int_{E_{h/2}^+}\operatorname{Im}s_n(t+i\eta)\,dt\leq\mu_n(E_h^+)+\frac{2\eta}{\pi h}. ( 1 − π h 4 η ) μ n ( E h − ) ≤ π 1 ∫ E h /2 − Im s n ( t + i η ) d t and π 1 ∫ E h /2 + Im s n ( t + i η ) d t ≤ μ n ( E h + ) + π h 2 η .
We therefore finish the proof by combining the above displays, concluding with ∣ μ ^ n ( E ) − μ n ( E ) ∣ ≤ Δ ( E h + ) + O ( η / h ) + μ n ( E h + ∖ E h − ) |\widehat{\mu}_n(E)-\mu_n(E)|\leq\Delta(E_h^+)+O(\eta/h)+\mu_n(E_h^+\setminus E_h^-) ∣ μ n ( E ) − μ n ( E ) ∣ ≤ Δ ( E h + ) + O ( η / h ) + μ n ( E h + ∖ E h − ) .
Proof of Proposition 2
We will prove the three main conclusions in the following three parts. In Part I we prove concentration of the empirical Stieltjes transform to its deterministic equivalent. In Part II we prove weak convergence of the spectral laws to μ ⋆ \mu_{\star} μ ⋆ in power law and critical law, respectively. In Part III, we identify the lower edges of the limiting laws, and their density bounds. Throughout, z = z n ( ζ ) z=z_n(\zeta) z = z n ( ζ ) denotes the entry of Table 1 for the spectrum under consideration.
Part I: Concentration of s ^ n \widehat{s}_n s n on compact subsets of C + \mathbb{C}_{+} C + . We first apply Theorem 1.(i) at the two spectral scales in Table 1 to prove Eq. (12a). Once Assumption 2 is verified, Eq. (11a) then reads
∥ a n ( s ^ n ( z ) − s n ( z ) ) ∥ L p = O C x , ν , κ ( p 2 a n n ∣ z ∣ Λ n ( z ) 17 ) . \left\|a_n\left(\widehat{s}_n(z)-s_n(z)\right)\right\|_{L^p}
=O_{C_x,\nu,\kappa}\left(\frac{p^2a_n}{n\lvert z\rvert}\Lambda_n(z)^{17}\right). ∥ a n ( s n ( z ) − s n ( z ) ) ∥ L p = O C x , ν , κ ( n ∣ z ∣ p 2 a n Λ n ( z ) 17 ) .
It therefore suffices to verify the assumption and bound ( a n / ∣ z ∣ ) Λ n ( z ) 17 (a_n/\lvert z\rvert)\Lambda_n(z)^{17} ( a n / ∣ z ∣) Λ n ( z ) 17 uniformly for ζ ∈ K \zeta\in K ζ ∈ K . To verify the assumption, we choose the largest admissible cutoff m = ⌊ ρ ( ⌊ n / 2 ⌋ − 2 ) ⌋ = Θ ( n ) m=\lfloor\rho(\lfloor n/2\rfloor-2)\rfloor=\Theta(n) m = ⌊ ρ (⌊ n /2 ⌋ − 2 )⌋ = Θ ( n ) , so Λ n ( z ) ≤ Λ n ( z , m ) \Lambda_n(z)\leq\Lambda_n(z,m) Λ n ( z ) ≤ Λ n ( z , m ) . For the power law, comparing the tail sum with its integral gives a n = n − α a_n=n^{-\alpha} a n = n − α and τ 1 ( m ) = Θ ( a n ) \tau_1(m)=\Theta(a_n) τ 1 ( m ) = Θ ( a n ) and n σ m + 1 = Θ ( a n ) n\sigma_{m+1}=\Theta(a_n) n σ m + 1 = Θ ( a n ) . For the critical law, a n = ( 1 + log n ) − 1 − β a_n=(1+\log n)^{-1-\beta} a n = ( 1 + log n ) − 1 − β and
b n ∼ ∫ n ∞ d x x ( 1 + log x ) 1 + β = 1 β ( 1 + log n ) β , τ 1 ( m ) = Θ ( b n ) , n σ m + 1 = Θ ( a n ) . b_n\sim\int_n^{\infty}\frac{\mathrm{d}x}{x(1+\log x)^{1+\beta}}
=\frac{1}{\beta(1+\log n)^\beta},
\qquad
\tau_1(m)=\Theta(b_n),
\qquad
n\sigma_{m+1}=\Theta(a_n). b n ∼ ∫ n ∞ x ( 1 + log x ) 1 + β d x = β ( 1 + log n ) β 1 , τ 1 ( m ) = Θ ( b n ) , n σ m + 1 = Θ ( a n ) .
Consequently r 1 ( m ) = τ 1 ( m ) / σ m + 1 r_1(m)=\tau_1(m)/\sigma_{m+1} r 1 ( m ) = τ 1 ( m ) / σ m + 1 is Θ ( n ) \Theta(n) Θ ( n ) and Θ ( n log n ) \Theta(n\log n) Θ ( n log n ) , respectively, so log ( n r 1 ( m ) ) = Θ ( log n ) \log(nr_1(m))=\Theta(\log n) log ( n r 1 ( m )) = Θ ( log n ) in both cases. Thus,
Λ n ( z , m ) = 1 ∣ z − ∣ { ∣ z ∣ + Θ ( a n ( log n ) 2 ) , power law ; ∣ z ∣ + Θ ( b n ( log n ) 2 ) , critical law . \Lambda_n(z,m)=\frac{1}{\lvert z_{-}\rvert}
\begin{cases}
\lvert z\rvert+\Theta\left(a_n(\log n)^2\right), & \text{power law};\\
\lvert z\rvert+\Theta\left(b_n(\log n)^2\right), & \text{critical law}.
\end{cases} Λ n ( z , m ) = ∣ z − ∣ 1 { ∣ z ∣ + Θ ( a n ( log n ) 2 ) , ∣ z ∣ + Θ ( b n ( log n ) 2 ) , power law ; critical law .
It remains to lower bound ∣ z − ∣ \lvert z_{-}\rvert ∣ z − ∣ and upper bound ∣ z ∣ \lvert z\rvert ∣ z ∣ at the two spectral scales. Since K K K is compact in C + \mathbb{C}_{+} C + , ∣ z − ∣ ≥ a n min ζ ∈ K Im ζ \lvert z_{-}\rvert\geq a_n\min_{\zeta\in K}\operatorname{Im}\zeta ∣ z − ∣ ≥ a n min ζ ∈ K Im ζ in both cases. The power law z = a n ζ z=a_n\zeta z = a n ζ gives ∣ z ∣ = Θ ( a n ) \lvert z\rvert=\Theta(a_n) ∣ z ∣ = Θ ( a n ) , whereas the critical law z = b n + a n ζ z=b_n+a_n\zeta z = b n + a n ζ implies ∣ z ∣ = Θ ( b n ) \lvert z\rvert=\Theta(b_n) ∣ z ∣ = Θ ( b n ) with b n = Θ ( a n log n ) b_n=\Theta(a_n\log n) b n = Θ ( a n log n ) . Substituting these bounds into the preceding display gives, uniformly for ζ ∈ K \zeta\in K ζ ∈ K ,
Λ n ( z ) ≤ Λ n ( z , m ) = { O ( a n + a n ( log n ) 2 a n ) = O ( ( log n ) 2 ) , power law ; O ( b n + b n ( log n ) 2 a n ) = O ( ( log n ) 3 ) , critical law . \Lambda_n(z)\leq\Lambda_n(z,m)=
\begin{cases}
O\left(\dfrac{a_n+a_n(\log n)^2}{a_n}\right)=O\left((\log n)^2\right), & \text{power law};\\
O\left(\dfrac{b_n+b_n(\log n)^2}{a_n}\right)=O\left((\log n)^3\right), & \text{critical law}.
\end{cases} Λ n ( z ) ≤ Λ n ( z , m ) = ⎩ ⎨ ⎧ O ( a n a n + a n ( log n ) 2 ) = O ( ( log n ) 2 ) , O ( a n b n + b n ( log n ) 2 ) = O ( ( log n ) 3 ) , power law ; critical law .
Thus Λ n ( z ) 16 = O ( ( log n ) 48 ) = o ( n ) \Lambda_n(z)^{16}=O((\log n)^{48})=o(n) Λ n ( z ) 16 = O (( log n ) 48 ) = o ( n ) in both cases, verifying the sample-size condition of Assumption 2 holds for large enough n n n . For its distance condition, ∣ z − ∣ ≥ a n min ζ ∈ K Im ζ \lvert z_{-}\rvert\geq a_n\min_{\zeta\in K}\operatorname{Im}\zeta ∣ z − ∣ ≥ a n min ζ ∈ K Im ζ guarantees ∣ z − ∣ ≥ n − ( 1 + α ) \lvert z_{-}\rvert\geq n^{-(1+\alpha)} ∣ z − ∣ ≥ n − ( 1 + α ) for the power law since a n = n − α a_n=n^{-\alpha} a n = n − α , and ∣ z − ∣ ≥ n − 1 \lvert z_{-}\rvert\geq n^{-1} ∣ z − ∣ ≥ n − 1 for the critical law since a n = ( 1 + log n ) − 1 − β a_n=(1+\log n)^{-1-\beta} a n = ( 1 + log n ) − 1 − β . The assumption holds uniformly on K K K for all sufficiently large n n n with ν = 1 + α \nu=1+\alpha ν = 1 + α and ν = 1 \nu=1 ν = 1 , respectively. Returning to the first display, the power law contributes ( log n ) 2 × 17 = ( log n ) 34 (\log n)^{2\times17}=(\log n)^{34} ( log n ) 2 × 17 = ( log n ) 34 . For the critical law the additional factor a n / ∣ z ∣ = O ( ( log n ) − 1 ) a_n/\lvert z\rvert=O((\log n)^{-1}) a n / ∣ z ∣ = O (( log n ) − 1 ) gives ( log n ) 3 × 17 − 1 = ( log n ) 50 (\log n)^{3\times17-1}=(\log n)^{50} ( log n ) 3 × 17 − 1 = ( log n ) 50 . Since these bounds are uniform on K K K , we complete the proof by taking the supremum of the L p L^p L p norms.
Part II: Convergence of the rescaled spectral laws μ ^ n \hat{\mu}_n μ ^ n , μ n \mu_n μ n to μ ⋆ \mu_\star μ ⋆ . We first show the rescaled fixed-point equations have a limit. Define u = a n s n ( z n ( ζ ) ) u=a_n s_n(z_n(\zeta)) u = a n s n ( z n ( ζ )) to be the Stieltjes transform at ζ \zeta ζ of the law of x / a n x/a_n x / a n or ( x − b n ) / a n (x-b_n)/a_n ( x − b n ) / a n , respectively, for x ∼ μ n x \sim\mu_n x ∼ μ n . To identify its limit, substitute s n ( z n ( ζ ) ) = u / a n s_n(z_n(\zeta))=u/a_n s n ( z n ( ζ )) = u / a n and a n = n σ n a_n=n\sigma_n a n = n σ n into Eq. (8) . Since z n ( ζ ) z_n(\zeta) z n ( ζ ) equals a n ζ a_n\zeta a n ζ or b n + a n ζ b_n+a_n\zeta b n + a n ζ , dividing Eq. (8) by a n a_n a n yields the inverse equation ζ = ζ n ( u ) \zeta=\zeta_n(u) ζ = ζ n ( u ) , where
ζ n ( u ) + 1 u = 1 n ∑ j ≥ 1 1 σ n / σ j + u − { 0 , power law ; b n / a n , critical law . \zeta_n(u)+\frac{1}{u}=\frac{1}{n}\sum_{j\geq1}\frac{1}{\sigma_n/\sigma_j+u}-
\begin{cases}
0, & \text{power law};\\
b_n/a_n, & \text{critical law}.
\end{cases} ζ n ( u ) + u 1 = n 1 j ≥ 1 ∑ σ n / σ j + u 1 − { 0 , b n / a n , power law ; critical law .
We show that ζ n ( u ) → ζ ⋆ ( u ) \zeta_n(u)\to\zeta_\star(u) ζ n ( u ) → ζ ⋆ ( u ) locally uniformly on C + \mathbb{C}_+ C + , where ζ ⋆ ( u ) \zeta_\star(u) ζ ⋆ ( u ) is the spectral parameter in Eq. (12b) at s ( ζ ) = u s(\zeta)=u s ( ζ ) = u .
For the power law, σ n / σ j = ( j / n ) 1 + α \sigma_n/\sigma_j=(j/n)^{1+\alpha} σ n / σ j = ( j / n ) 1 + α . For u > 0 u>0 u > 0 the sum is 1 / u 1/u 1/ u times the “regularly varying spectrum” trace sum in [28 ] at ridge parameter σ n / u \sigma_n/u σ n / u . The same Riemann sum gives, for u ∈ C + u\in\mathbb{C}_+ u ∈ C + ,
ζ n ( u ) + 1 u → ∫ 0 ∞ d t t 1 + α + u = π u − α / ( 1 + α ) ( 1 + α ) sin ( π / ( 1 + α ) ) . \zeta_n(u)+\frac{1}{u}\to\int_0^\infty\frac{dt}{t^{1+\alpha}+u}=\frac{\pi u^{-\alpha/(1+\alpha)}}{(1+\alpha)\sin(\pi/(1+\alpha))}. ζ n ( u ) + u 1 → ∫ 0 ∞ t 1 + α + u d t = ( 1 + α ) sin ( π / ( 1 + α )) π u − α / ( 1 + α ) .
For the critical law, b n / a n = n − 1 ∑ j > n σ j / σ n b_n/a_n=n^{-1}\sum_{j>n}\sigma_j/\sigma_n b n / a n = n − 1 ∑ j > n σ j / σ n , and we have
ζ n ( u ) + 1 u = 1 n ∑ j ≤ n 1 σ n / σ j + u + 1 n ∑ j > n ( 1 σ n / σ j + u − σ j σ n ) → ∫ 0 1 d t t + u + ∫ 1 ∞ ( 1 t + u − 1 t ) d t = − log u . \zeta_n(u)+\frac{1}{u}=\frac{1}{n}\sum_{j\leq n}\frac{1}{\sigma_n/\sigma_j+u}+\frac{1}{n}\sum_{j>n}\left(\frac{1}{\sigma_n/\sigma_j+u}-\frac{\sigma_j}{\sigma_n}\right)\to\int_0^1\frac{dt}{t+u}+\int_1^\infty\left(\frac{1}{t+u}-\frac{1}{t}\right)dt=-\log u. ζ n ( u ) + u 1 = n 1 j ≤ n ∑ σ n / σ j + u 1 + n 1 j > n ∑ ( σ n / σ j + u 1 − σ n σ j ) → ∫ 0 1 t + u d t + ∫ 1 ∞ ( t + u 1 − t 1 ) d t = − log u .
Indeed, σ n / σ ⌈ n t ⌉ → t \sigma_n/\sigma_{\lceil nt\rceil}\to t σ n / σ ⌈ n t ⌉ → t for t > 0 t>0 t > 0 and this ratio is at least t t t for t > 1 t>1 t > 1 . As the power law summand at j = ⌈ n t ⌉ j=\lceil nt\rceil j = ⌈ n t ⌉ is O ( ( 1 + t 1 + α ) − 1 ) O((1+t^{1+\alpha})^{-1}) O (( 1 + t 1 + α ) − 1 ) , while the critical law summand is bounded for 0 < t ≤ 1 0<t\leq1 0 < t ≤ 1 and its centered tail is O ( t − 2 ) O(t^{-2}) O ( t − 2 ) for t > 1 t>1 t > 1 , locally uniform convergence follows on compact subsets of C + \mathbb{C}_+ C + by dominated convergence.
We next show the convergence of the laws and identify μ ⋆ \mu_\star μ ⋆ . To this end, it suffices to show that every vague subsequential limit of the rescaled laws has mass one and that all such limits agree. Fix one such limit μ \mu μ , and let s s s be its Stieltjes transform. For u ∈ C + u\in\mathbb{C}_+ u ∈ C + with ζ = ζ ⋆ ( u ) ∈ C + \zeta=\zeta_\star(u)\in\mathbb{C}_+ ζ = ζ ⋆ ( u ) ∈ C + , we have ζ n ( u ) ∈ C + \zeta_n(u)\in\mathbb{C}_+ ζ n ( u ) ∈ C + for all large n n n , and thus applying Lemma A.4 at z n ( ζ n ( u ) ) z_n(\zeta_n(u)) z n ( ζ n ( u )) implies a n s n ( z n ( ζ n ( u ) ) ) = u a_n s_n(z_n(\zeta_n(u)))=u a n s n ( z n ( ζ n ( u ))) = u . Invoking the resolvent identity at ζ \zeta ζ and ζ n ( u ) \zeta_n(u) ζ n ( u ) , we have
∣ a n s n ( z n ( ζ ) ) − u ∣ ≤ ∣ ζ − ζ n ( u ) ∣ Im ζ ⋅ Im ζ n ( u ) → 0. \left|a_n s_n(z_n(\zeta))-u\right|\leq\frac{|\zeta-\zeta_n(u)|}{\operatorname{Im}\zeta\cdot\operatorname{Im}\zeta_n(u)}\to0. ∣ a n s n ( z n ( ζ )) − u ∣ ≤ Im ζ ⋅ Im ζ n ( u ) ∣ ζ − ζ n ( u ) ∣ → 0.
Along the chosen subsequence, vague convergence implies a n s n ( z n ( ζ ) ) → s ( ζ ) a_n s_n(z_n(\zeta))\to s(\zeta) a n s n ( z n ( ζ )) → s ( ζ ) since x ↦ ( x − ζ ) − 1 x\mapsto(x-\zeta)^{-1} x ↦ ( x − ζ ) − 1 is continuous and vanishes at infinity. Hence s ( ζ ⋆ ( u ) ) = u s(\zeta_\star(u))=u s ( ζ ⋆ ( u )) = u , proving uniqueness whenever a solution exists. To verify that μ \mu μ is a probability measure, take u = i / y u=i/y u = i / y and ζ = ζ ⋆ ( u ) = i y + o ( y ) \zeta=\zeta_\star(u)=iy+o(y) ζ = ζ ⋆ ( u ) = i y + o ( y ) as y → ∞ y\to\infty y → ∞ . Taking imaginary parts as in the proof of Lemma A.4 yields
Im ζ y = ( Im ζ ) Im s ( ζ ) = ∫ ( Im ζ ) 2 ∣ x − ζ ∣ 2 μ ( d x ) → μ ( R ) . \frac{\operatorname{Im}\zeta}{y}=(\operatorname{Im}\zeta)\operatorname{Im}s(\zeta)=\int\frac{(\operatorname{Im}\zeta)^2}{|x-\zeta|^2}\,\mu(dx)\to\mu(\mathbb{R}). y Im ζ = ( Im ζ ) Im s ( ζ ) = ∫ ∣ x − ζ ∣ 2 ( Im ζ ) 2 μ ( d x ) → μ ( R ) .
The convergence follows by dominated convergence as y → ∞ y\to\infty y → ∞ . Since Im ζ / y → 1 \operatorname{Im}\zeta/y\to1 Im ζ / y → 1 , we obtain μ ( R ) = 1 \mu(\mathbb{R})=1 μ ( R ) = 1 . Thus s ( ζ ) ∈ C + s(\zeta)\in\mathbb{C}_+ s ( ζ ) ∈ C + for every ζ ∈ C + \zeta\in\mathbb{C}_+ ζ ∈ C + . Passing to the limit along the subsequence in ζ n ( a n s n ( z n ( ζ ) ) ) = ζ \zeta_n(a_n s_n(z_n(\zeta)))=\zeta ζ n ( a n s n ( z n ( ζ ))) = ζ , it holds ζ ⋆ ( s ( ζ ) ) = ζ \zeta_\star(s(\zeta))=\zeta ζ ⋆ ( s ( ζ )) = ζ by locally uniform convergence of ζ n \zeta_n ζ n . By the uniqueness of s s s established above, all vague subsequential limits therefore have the same transform s s s , proving weak convergence to μ = μ ⋆ \mu=\mu_\star μ = μ ⋆ . The resolvent identity makes the rescaled transforms equicontinuous on each compact K ⊂ C + K\subset\mathbb{C}_+ K ⊂ C + , so their convergence is locally uniform. Finally, setting p = 2 p = 2 p = 2 in Eq. (12a) yields a summable sequence of mean-square errors for each fixed ζ \zeta ζ . By invoking Chebyshev’s inequality together with the Borel-Cantelli lemma, we deduce almost sure convergence of the empirical Stieltjes transforms to s s s on a countable dense subset of C + \mathbb{C}_{+} C + . This is sufficient to conclude almost sure weak convergence via the Stieltjes continuity theorem [7 Theorem B.9] .
Part III: Lower edges and density bounds. To obtain the edge formula and the density bound, we reparameterize by u = s ( ζ ) = r e i θ u = s(\zeta) = r e^{i\theta} u = s ( ζ ) = r e i θ , 0 < θ < π 0 < \theta< \pi 0 < θ < π . In the power law case, we additionally set γ = 1 / ( 1 + α ) \gamma= 1/(1+\alpha) γ = 1/ ( 1 + α ) and c = π γ / sin ( π γ ) c = \pi\gamma/\sin(\pi\gamma) c = π γ / sin ( π γ ) . Then, take imaginary parts for both cases in Eq. (12b), and since Im ζ > 0 \operatorname{Im}\zeta> 0 Im ζ > 0 , we have
0 < Im ζ = sin θ r = { c r γ − 1 sin ( ( 1 − γ ) θ ) , power law ; θ , critical law . r < { ( sin θ c sin ( ( 1 − γ ) θ ) ) 1 / γ , power law ; sin θ θ , critical law . 0 < \operatorname{Im}\zeta= \frac{\sin\theta}{r} =
\begin{cases}
c r^{\gamma-1}\sin((1-\gamma)\theta), & \text{power law};\\
\theta, & \text{critical law}.
\end{cases}
\qquad
r <
\begin{cases}
\left(\dfrac{\sin\theta}{c\sin((1-\gamma)\theta)}\right)^{1/\gamma}, & \text{power law};\\
\dfrac{\sin\theta}{\theta}, & \text{critical law}.
\end{cases} 0 < Im ζ = r sin θ = { c r γ − 1 sin (( 1 − γ ) θ ) , θ , power law ; critical law . r < ⎩ ⎨ ⎧ ( c sin (( 1 − γ ) θ ) sin θ ) 1/ γ , θ sin θ , power law ; critical law .
Using sin ( ( 1 − γ ) θ ) ≥ ( 1 − γ ) sin θ \sin((1-\gamma)\theta) \ge(1-\gamma)\sin\theta sin (( 1 − γ ) θ ) ≥ ( 1 − γ ) sin θ and sin θ ≤ θ \sin\theta\le\theta sin θ ≤ θ , we have Im s ( ζ ) ≤ r ≤ ( c ( 1 − γ ) ) − 1 / γ \operatorname{Im}s(\zeta) \le r \le(c(1-\gamma))^{-1/\gamma} Im s ( ζ ) ≤ r ≤ ( c ( 1 − γ ) ) − 1/ γ for the power law decay and Im s ( ζ ) ≤ 1 \operatorname{Im}s(\zeta) \le1 Im s ( ζ ) ≤ 1 for the critical law decay. By Stieltjes inversion on any interval ( a , b ) (a,b) ( a , b ) ,
μ ⋆ ( ( a , b ) ) ≤ lim η ↓ 0 1 π ∫ a b Im s ( t + i η ) d t ≤ b − a π { ( c ( 1 − γ ) ) − 1 / γ , power law ; 1 , critical law . \mu_{\star}((a,b)) \le\lim_{\eta\downarrow0}\frac{1}{\pi}\int_a^b \operatorname{Im}s(t+i\eta)\,\mathrm{d}t
\le\frac{b-a}{\pi}
\begin{cases}
(c(1-\gamma))^{-1/\gamma}, & \text{power law};\\
1, & \text{critical law}.
\end{cases} μ ⋆ (( a , b )) ≤ η ↓ 0 lim π 1 ∫ a b Im s ( t + i η ) d t ≤ π b − a { ( c ( 1 − γ ) ) − 1/ γ , 1 , power law ; critical law .
which proves the density bound on ∥ ρ ⋆ ∥ ∞ \lVert\rho_{\star}\rVert_{\infty} ∥ ρ ⋆ ∥ ∞ for μ ⋆ \mu_{\star} μ ⋆ .
Next we find κ ⋆ \kappa_{\star} κ ⋆ by differentiating Eq. (12b) on u > 0 u > 0 u > 0 . The edge equation (cf. [71 Section 4] ) is
0 = u 2 ζ ⋆ ′ ( u ) = { 1 − c ( 1 − γ ) u γ , power law ; 1 − u , critical law . (36) 0 = u^{2}\zeta_{\star}'(u) =
\begin{cases}
1-c(1-\gamma)u^{\gamma}, & \text{power law};\\
1-u, & \text{critical law}.
\end{cases} \tag*{(36)} 0 = u 2 ζ ⋆ ′ ( u ) = { 1 − c ( 1 − γ ) u γ , 1 − u , power law ; critical law . (36)
On the increasing branch from u = 0 u = 0 u = 0 , the inverse agrees with s s s by Part II, so there is no mass below max u > 0 ζ ⋆ ( u ) \max_{u>0}\zeta_{\star}(u) max u > 0 ζ ⋆ ( u ) . Evaluating this maximum at u = ( c ( 1 − γ ) ) − 1 / γ u = (c(1-\gamma))^{-1/\gamma} u = ( c ( 1 − γ ) ) − 1/ γ and u = 1 u = 1 u = 1 , respectively, we obtain the two edges κ ⋆ = ( c ( 1 − γ ) ) 1 / γ / α \kappa_{\star} = (c(1-\gamma))^{1/\gamma}/\alpha κ ⋆ = ( c ( 1 − γ ) ) 1/ γ / α and κ ⋆ = − 1 \kappa_{\star} = -1 κ ⋆ = − 1 in Eq. (12c).
It only remains to show that the support contains every x > κ ⋆ x > \kappa_{\star} x > κ ⋆ . Indeed, we will derive the exact densities below using ( r , θ ) (r,\theta) ( r , θ ) parameterization. Taking equality in the bound for r r r sets Im ζ ⋆ ( r e i θ ) = 0 \operatorname{Im}\zeta_{\star}(r e^{i\theta}) = 0 Im ζ ⋆ ( r e i θ ) = 0 in the first display of Part III. Thus x = ζ ⋆ ( r e i θ ) x = \zeta_{\star}(r e^{i\theta}) x = ζ ⋆ ( r e i θ ) is real and continuous in θ \theta θ . As θ ↓ 0 \theta\downarrow0 θ ↓ 0 , r r r tends to the positive critical point computed above (use sin θ ∼ θ \sin\theta\sim\theta sin θ ∼ θ ), so x → κ ⋆ x \to\kappa_{\star} x → κ ⋆ . As θ ↑ π \theta\uparrow\pi θ ↑ π , r → 0 r \to0 r → 0 and x ∼ 1 / r → ∞ x \sim1/r \to\infty x ∼ 1/ r → ∞ . The intermediate value theorem therefore gives such a θ \theta θ for every x > κ ⋆ x > \kappa_{\star} x > κ ⋆ . Since ζ ⋆ ′ ( r e i θ ) ≠ 0 \zeta_{\star}'(r e^{i\theta}) \ne0 ζ ⋆ ′ ( r e i θ ) = 0 by the derivative formula, its local inverse at x x x takes values in C + \mathbb{C}_{+} C + near x x x and agrees with s s s there for Im ζ > 0 \operatorname{Im}\zeta> 0 Im ζ > 0 by Part II. Hence s ( x + i 0 ) = r e i θ s(x+i0) = r e^{i\theta} s ( x + i 0 ) = r e i θ . Stieltjes inversion yields ρ ⋆ ( x ) = r sin θ / π > 0 \rho_{\star}(x) = r\sin\theta/\pi> 0 ρ ⋆ ( x ) = r sin θ / π > 0 , proving that the support is [ κ ⋆ , ∞ ) [\kappa_{\star},\infty) [ κ ⋆ , ∞ ) .
Mesoscopic scales and the generalized Marchenko-Pastur law
We provide two additional secondary implications of Theorem 1 below.
Extension I: Mesoscopic scale. To set the stage, we write h = Im ζ = Im z / a n h = \operatorname{Im}\zeta= \operatorname{Im}z/a_n h = Im ζ = Im z / a n for the spectral height in units of the bulk width a n a_n a n . The results presented in Proposition 2 treat the macroscopic scale h = Θ ( 1 ) h = \Theta(1) h = Θ ( 1 ) . At a fixed interior bulk energy, typical finer mesoscopic results concern heights satisfying h → 0 h \to0 h → 0 and n h → ∞ nh \to\infty nh → ∞ . In fact, our Theorem 1 can already give such a refinement at h = n − δ h = n^{-\delta} h = n − δ with 0 < δ < 1 / 17 0 < \delta< 1/17 0 < δ < 1/17 . To our knowledge, such a dimension-free Stieltjes transform estimate was previously unavailable for the infinite-rank power and critical laws under Assumption 1. Related dimension-free estimates for ridge regression concern prediction risk at nonnegative real regularization parameters [28 ] .
We briefly discuss what is known to be optimal in the classical proportional regime, where d / n d/n d / n is bounded above and away from zero, the population eigenvalues lie in a fixed compact interval, and the normalized Gram matrix X X ∗ / n XX^*/n X X ∗ / n has bulk width of order one. At a regular bulk energy, the averaged local law gives a Stieltjes transform error of order 1 / ( n h ) 1/(nh) 1/ ( nh ) in the sense of stochastic domination down to h = n − 1 + ε h = n^{-1+\varepsilon} h = n − 1 + ε for any fixed ε > 0 \varepsilon> 0 ε > 0 [54 Theorem 3.6] . This scale is optimal up to n ε n^{\varepsilon} n ε , since the mean bulk spacing is of order 1 / n 1/n 1/ n . For identity covariance, the aspect ratio may vary polynomially [18 ] . In the proportional regime, the averaged local law also holds for dependent coordinates under quadratic-form concentration [44 Theorem 2.5] . It is reasonable to expect such the similar height refinement can hold in our setup; however, neither result covers an infinite-rank covariance with decaying eigenvalues and a shrinking bulk width as in Proposition 2.
In Theorem 1.(i), Λ n ( z ) = Ω ( 1 / h ) \Lambda_n(z) = \Omega(1/h) Λ n ( z ) = Ω ( 1/ h ) and ( a n / ∣ z ∣ ) Λ n ( z ) = O ( ( log n ) 2 / h ) (a_n/|z|)\Lambda_n(z) = O((\log n)^2/h) ( a n /∣ z ∣ ) Λ n ( z ) = O (( log n ) 2 / h ) , so one power in the error bound would give O ( p 2 ( log n ) 2 / ( n h ) ) O(p^2(\log n)^2/(nh)) O ( p 2 ( log n ) 2 / ( nh )) . This provides an explicit target to refine, i.e. replacing the sample-size condition n = Ω ( Λ n ( z ) 16 ) n = \Omega(\Lambda_n(z)^{16}) n = Ω ( Λ n ( z ) 16 ) by n = Ω ( Λ n ( z ) ) n = \Omega(\Lambda_n(z)) n = Ω ( Λ n ( z )) , which would admit h = n − 1 + ε h = n^{-1+\varepsilon} h = n − 1 + ε for every fixed ε > 0 \varepsilon> 0 ε > 0 . Along the interpolation path, this calls for (a potentially completely new strategy for) stability in Theorem 3 at those heights and an unweighted trace error in Theorem 4 linear in Λ n ( z ) \Lambda_n(z) Λ n ( z ) up to logarithmic factors. We leave the optimal local law open and give the mesoscopic bound below.
Proposition 6 (Mesoscopic bulk scales). Use the setup and scaling z = z n ( ζ ) z = z_n(\zeta) z = z n ( ζ ) of Proposition 2, with Re ζ \operatorname{Re}\zeta Re ζ fixed in the interior of the limiting bulk and h = Im ζ ∈ ( 0 , 1 ] h = \operatorname{Im}\zeta\in(0,1] h = Im ζ ∈ ( 0 , 1 ] . If z z z satisfies Assumption 2, then for all large n n n and 2 ≤ p ≤ κ log n 2 \le p \le\kappa\log n 2 ≤ p ≤ κ log n ,
∥ a n ( s ^ n ( z ) − s n ( z ) ) ∥ L p = p 2 n h 17 { O C x , ν , κ , Re ζ , α ( ( log n ) 34 ) , power law ; O C x , ν , κ , Re ζ , β ( ( log n ) 50 ) , critical law . (37) \left\|a_n\left(\widehat{s}_n(z)-s_n(z)\right)\right\|_{L^p}
=
\frac{p^2}{nh^{17}}
\begin{cases}
O_{C_{x,\nu,\kappa,\operatorname{Re}\zeta,\alpha}}\left((\log n)^{34}\right), & \text{power law};\\
O_{C_{x,\nu,\kappa,\operatorname{Re}\zeta,\beta}}\left((\log n)^{50}\right), & \text{critical law}.
\end{cases} \tag*{(37)} ∥ a n ( s n ( z ) − s n ( z ) ) ∥ L p = n h 17 p 2 { O C x , ν , κ , Re ζ , α ( ( log n ) 34 ) , O C x , ν , κ , Re ζ , β ( ( log n ) 50 ) , power law ; critical law . (37)
This implies mesoscopic refinement, as for h = n − δ h = n^{-\delta} h = n − δ with fixed 0 < δ < 1 / 17 0 < \delta< 1/17 0 < δ < 1/17 , the assumption holds for all large n n n with ν > α + δ \nu> \alpha+ \delta ν > α + δ or ν > δ \nu> \delta ν > δ , respectively, and the bound is o ( 1 ) o(1) o ( 1 ) uniformly in p p p .
Extension II: Proportional regime. In the main paper, our examples focus on the less well-understood settings (power law and critical law decay) involving infinitely many population eigenvalues. Still, it is useful to illustrate how our results apply to the standard proportional framework, which follows as a more straightforward application of the same theorem. Suppose d / n d/n d / n stays within a fixed compact subset of ( 0 , + ∞ ) (0,+\infty) ( 0 , + ∞ ) , and let π n = d − 1 ∑ j = 1 d δ σ j \pi_n = d^{-1}\sum_{j=1}^{d}\delta_{\sigma_j} π n = d − 1 ∑ j = 1 d δ σ j be the population spectral law. At z = n ζ z = n\zeta z = n ζ with ζ \zeta ζ in a fixed compact set K ⊂ C + K \subset\mathbb{C}_+ K ⊂ C + , simply choosing the cutoff m = 1 m = 1 m = 1 verifies Assumption 2. The finite-n n n comparison law for X X ∗ / n XX^*/n X X ∗ / n is the generalized Marchenko–Pastur law on the Gram scale determined by d / n d/n d / n and π n \pi_n π n . By Eq. (8) , its Stieltjes transform u = n s n ( n ζ ) u = ns_n(n\zeta) u = n s n ( n ζ ) satisfies
ζ = − 1 u + d n ∫ x 1 + x u π n ( d x ) . (38) \zeta= -\frac{1}{u}+\frac{d}{n}\int\frac{x}{1+xu}\,\pi_n(\mathrm{d}x). \tag*{(38)} ζ = − u 1 + n d ∫ 1 + xu x π n ( d x ) . (38)
These results are standardly known. At fixed ζ ∈ C + \zeta\in\mathbb{C}_+ ζ ∈ C + , sub-Gaussian concentration for all 1-Lipschitz functions of the full data matrix in Frobenius norm gives an almost sure O ( log n / n ) O(\sqrt{\log n}/n) O ( log n / n ) transform error for the same finite-n n n law [31 Corollary 2.5] . To see how our result applies, under Assumption 1, which allows independent sub-Gaussian coordinates or convex concentration of each standardized observation, Proposition 7 applies Theorem 1 directly to recover this finite-n n n law at the 1 / n 1/n 1/ n scale up to logarithmic factors, with an explicit L p L^p L p bound on K K K . If d / n → ϕ d/n \to\phi d / n → ϕ and π n ⇒ π \pi_n \Rightarrow\pi π n ⇒ π , the limiting law is the classical Marchenko-Pastur law for identity covariance [60 ] and the generalized Marchenko-Pastur law for population spectrum π \pi π [71 ] . However, a quantitative rate to this limit would also depend on how fast d / n d/n d / n and π n \pi_n π n converge, which is not covered by the statement of the result itself in what follows.
Proposition 7 (Generalized Marchenko-Pastur law). In the proportional setup above, suppose Assumption 1 holds and fix κ > 0 \kappa> 0 κ > 0 . For all large n n n and 2 ≤ p ≤ κ log n 2 \le p \le\kappa\log n 2 ≤ p ≤ κ log n ,
sup ζ ∈ K ∥ n s ^ n ( n ζ ) − n s n ( n ζ ) ∥ L p = O C x , κ , K ( p 2 ( log n ) 34 n ) . \sup_{\zeta\in K} \left\|\widehat{n s}_n(n\zeta)-n s_n(n\zeta)\right\|_{L^p}=O_{C_x,\kappa,K}\left(\frac{p^2(\log n)^{34}}{n}\right). ζ ∈ K sup ∥ n s n ( n ζ ) − n s n ( n ζ ) ∥ L p = O C x , κ , K ( n p 2 ( log n ) 34 ) . If d / n → ϕ ∈ ( 0 , ∞ ) d/n \to\phi\in(0,\infty) d / n → ϕ ∈ ( 0 , ∞ ) and π n \pi_n π n converges weakly to π \pi π , the empirical law of X X ∗ / n \mathbf{X}\mathbf{X}^{*}/n X X ∗ / n converges almost surely to the generalized Marchenko-Pastur law. Its Stieltjes transform uniquely satisfies in C + \mathbb{C}_+ C + that
ζ = − 1 s + ϕ ∫ x 1 + x s π ( d x ) . (39) \zeta=-\frac{1}{s}+\phi\int\frac{x}{1+xs}\,\pi(\mathrm{d}x). \tag*{(39)} ζ = − s 1 + ϕ ∫ 1 + x s x π ( d x ) . (39)
Proof of Proposition 6
The proof is a simple application of Theorem 1.(i), using the cutoff and the bound on Λ n ( z , m ) \Lambda_n(z,m) Λ n ( z , m ) from Part I of the proof of Proposition 2. In the power law case, Re ζ > κ ⋆ > 0 \operatorname{Re}\zeta>\kappa_\star>0 Re ζ > κ ⋆ > 0 by Eq. (12c), so ∣ z ∣ = a n ∣ ζ ∣ = Θ ( a n ) |z|=a_n|\zeta|=\Theta(a_n) ∣ z ∣ = a n ∣ ζ ∣ = Θ ( a n ) . In the critical law case, z = b n + a n ζ z=b_n+a_n\zeta z = b n + a n ζ and a n / b n → 0 a_n/b_n\to0 a n / b n → 0 give ∣ z ∣ = Θ ( b n ) |z|=\Theta(b_n) ∣ z ∣ = Θ ( b n ) , uniformly for 0 < h ≤ 1 0<h\le1 0 < h ≤ 1 . In both cases Re z > 0 \operatorname{Re}z>0 Re z > 0 for large n n n , so ∣ z − ∣ = a n h |z_-|=a_nh ∣ z − ∣ = a n h . The same bound on Λ n ( z , m ) \Lambda_n(z,m) Λ n ( z , m ) therefore acquires a factor h − 1 h^{-1} h − 1 . Applying Theorem 1.(i) gives Eq. (37) , with the factor h − 17 h^{-17} h − 17 and the same logarithmic powers as Eq. (12a).
Proof of Proposition 7
We apply Theorem 1.(i) at z = n ζ z=n\zeta z = n ζ by taking m = 1 m=1 m = 1 , which is admissible for all large n n n since d / n d/n d / n is bounded away from zero. The normalization ∥ Σ ∥ = 1 \|\Sigma\|=1 ∥Σ∥ = 1 gives σ 2 ≤ 1 \sigma_2\le1 σ 2 ≤ 1 , τ 1 ( 1 ) ≤ d \tau_1(1)\le d τ 1 ( 1 ) ≤ d and r 1 ( 1 ) ≤ d r_1(1)\le d r 1 ( 1 ) ≤ d . Uniformly for ζ ∈ K \zeta\in K ζ ∈ K , ∣ z ∣ = O K ( n ) |z|=O_K(n) ∣ z ∣ = O K ( n ) and ∣ z − ∣ ≥ n min ζ ∈ K Im ζ |z_-|\ge n\min_{\zeta\in K}\operatorname{Im}\zeta ∣ z − ∣ ≥ n min ζ ∈ K Im ζ . Hence, when σ 2 > 0 \sigma_2>0 σ 2 > 0 ,
Λ n ( n ζ ) ≤ Λ n ( n ζ , 1 ) ≤ n sup ζ ∈ K ∣ ζ ∣ + n + d log n log ( n d ) n min ζ ∈ K Im ζ = O K ( ( log n ) 2 ) . \Lambda_n(n\zeta)\le\Lambda_n(n\zeta,1)\le\frac{n\sup_{\zeta\in K}|\zeta|+n+d\log n\log(nd)}{n\min_{\zeta\in K}\operatorname{Im}\zeta}=O_K((\log n)^2). Λ n ( n ζ ) ≤ Λ n ( n ζ , 1 ) ≤ n min ζ ∈ K Im ζ n sup ζ ∈ K ∣ ζ ∣ + n + d log n log ( n d ) = O K (( log n ) 2 ) .
Here and in the statement, the implicit constant may depend on the fixed range of d / n d/n d / n . If σ 2 = 0 \sigma_2=0 σ 2 = 0 , the defining convention gives Λ n ( n ζ , 1 ) = ∣ z ∣ / ∣ z − ∣ = O K ( 1 ) \Lambda_n(n\zeta,1)=|z|/|z_-|=O_K(1) Λ n ( n ζ , 1 ) = ∣ z ∣/∣ z − ∣ = O K ( 1 ) . Thus Λ n ( n ζ ) 16 = o ( n ) \Lambda_n(n\zeta)^{16}=o(n) Λ n ( n ζ ) 16 = o ( n ) and ∣ z − ∣ ≥ n − 1 |z_-|\ge n^{-1} ∣ z − ∣ ≥ n − 1 for all large $n, verifying Assumption 2 with ν = 1 \nu=1 ν = 1 . The empirical transform of X X ∗ / n \mathbf{X}\mathbf{X}^{*}/n X X ∗ / n is n s ^ n ( n ζ ) n\widehat{s}_n(n\zeta) n s n ( n ζ ) . Multiplying Eq. (11a) in Theorem 1.(i) by n n n gives the stated bound since ∣ ζ ∣ |\zeta| ∣ ζ ∣ is bounded away from zero on K K K .
To identify the limit, we apply the inverse comparison in Part II of the proof of Proposition 2. For u = n s n ( n ζ ) u=ns_n(n\zeta) u = n s n ( n ζ ) , the finite-n n n inverse equation is Eq. (38) . Its right-hand side converges locally uniformly in u ∈ C + u\in\mathbb{C}_+ u ∈ C + to the right-hand side of Eq. (39) , since d / n → ϕ d/n\to\phi d / n → ϕ , π n \pi_n π n converges weakly to π \pi π . At u = i / y u=i/y u = i / y the limiting right-hand side is i y + O ( 1 ) iy+O(1) i y + O ( 1 ) as y → ∞ y\to\infty y → ∞ , since ∣ x / ( 1 + x i / y ) ∣ ≤ 1 |x/(1+xi/y)|\le1 ∣ x / ( 1 + x i / y ) ∣ ≤ 1 on [ 0 , 1 ] [0,1] [ 0 , 1 ] . These are precisely the two inputs to the inverse comparison and mass-one calculation used in Part II. Applying them with z n ( ζ ) = n ζ z_n(\zeta)=n\zeta z n ( ζ ) = n ζ shows that the deterministic laws converge weakly to the probability law whose transform is the unique upper-half-plane solution of Eq. (39) . At p = 2 p=2 p = 2 the displayed rate in Proposition 7 gives summable mean-square errors for every fixed ζ \zeta ζ . The same countable-set argument as in Part II therefore gives almost sure weak convergence of the empirical laws.
For π = δ 1 \pi=\delta_1 π = δ 1 , Eq. (39) reduces to ζ s 2 + ( ζ + 1 − ϕ ) s + 1 = 0 \zeta s^2+(\zeta+1-\phi)s+1=0 ζ s 2 + ( ζ + 1 − ϕ ) s + 1 = 0 , the Marchenko-Pastur equation for the n × n n\times n n × n Gram matrix [71 ] .
Proofs for Section 2.3: Risk of spectral filters
Proof of Lemma 2.1
Starting from the conditional variance in Eq. (16) , we have
V X [ ϕ ] = τ 2 n Tr ( Σ Σ ^ ϕ ( Σ ^ ) 2 ) = τ 2 ∬ Γ 2 Tr ( Σ T n ( T n − z I ) − 1 ( T n − z ′ I ) − 1 ) α ϕ ( d z ) α ϕ ( d z ′ ) = τ 2 ∬ Γ 2 z Tr ( Σ ( T n − z I ) − 1 ) − z ′ Tr ( Σ ( T n − z ′ I ) − 1 ) z − z ′ α ϕ ( d z ) α ϕ ( d z ′ ) , \begin{aligned}
\mathcal{V}_{X}[\phi] = \frac{\tau^{2}}{n}\operatorname{Tr}\left(\Sigma\widehat{\Sigma}\phi(\widehat{\Sigma})^{2}\right)
= \tau^{2}\iint_{\Gamma^{2}}\operatorname{Tr}\left(\Sigma T_{n}(T_{n}-zI)^{-1}(T_{n}-z'I)^{-1}\right)\alpha_{\phi}(\mathrm{d}z)\alpha_{\phi}(\mathrm{d}z') \\
&= \tau^{2}\iint_{\Gamma^{2}}\frac{z\operatorname{Tr}\left(\Sigma(T_{n}-zI)^{-1}\right)-z'\operatorname{Tr}\left(\Sigma(T_{n}-z'I)^{-1}\right)}{z-z'}\alpha_{\phi}(\mathrm{d}z)\alpha_{\phi}(\mathrm{d}z'),
\end{aligned} V X [ ϕ ] = n τ 2 Tr ( Σ Σ ϕ ( Σ ) 2 ) = τ 2 ∬ Γ 2 Tr ( Σ T n ( T n − z I ) − 1 ( T n − z ′ I ) − 1 ) α ϕ ( d z ) α ϕ ( d z ′ ) = τ 2 ∬ Γ 2 z − z ′ z Tr ( Σ ( T n − z I ) − 1 ) − z ′ Tr ( Σ ( T n − z ′ I ) − 1 ) α ϕ ( d z ) α ϕ ( d z ′ ) ,
where in (i), we substitute Eq. (17) for the squared filter factors with a double integral and use Σ ^ = T n / n \widehat{\Sigma}=T_{n}/n Σ = T n / n , and in (ii) we apply the resolvent identity at z ≠ z ′ z\ne z' z = z ′ and use T n ( T n − z I ) − 1 = I + z ( T n − z I ) − 1 T_{n}(T_{n}-zI)^{-1}=I+z(T_{n}-zI)^{-1} T n ( T n − z I ) − 1 = I + z ( T n − z I ) − 1 . At z ′ = z z'=z z ′ = z the quotient equals the derivative of z Tr ( Σ ( T n − z I ) − 1 ) z\operatorname{Tr}(\Sigma(T_{n}-zI)^{-1}) z Tr ( Σ ( T n − z I ) − 1 ) , which can be verified directly at step (ii). Similarly, for the conditional squared bias in Eq. (16) , we have
B X [ ϕ ] = ∬ Γ 2 z z ′ β ∗ ( T n − z I ) − 1 Σ ( T n − z ′ I ) − 1 β α ϕ ( d z ) α ϕ ( d z ′ ) , \mathcal{B}_{X}[\phi]=\iint_{\Gamma^{2}}zz'\beta^{*}(T_{n}-zI)^{-1}\Sigma(T_{n}-z'I)^{-1}\beta\alpha_{\phi}(\mathrm{d}z)\alpha_{\phi}(\mathrm{d}z'), B X [ ϕ ] = ∬ Γ 2 z z ′ β ∗ ( T n − z I ) − 1 Σ ( T n − z ′ I ) − 1 β α ϕ ( d z ) α ϕ ( d z ′ ) ,
where we substitute ψ ( Σ ^ ) = − ∫ Γ z ( T n − z I ) − 1 α ϕ ( d z ) \psi(\widehat{\Sigma})=-\int_{\Gamma}z(T_{n}-zI)^{-1}\alpha_{\phi}(\mathrm{d}z) ψ ( Σ ) = − ∫ Γ z ( T n − z I ) − 1 α ϕ ( d z ) from Eq. (17) on both sides of Σ \Sigma Σ . Since 0 ∈ Ω 0\in\Omega 0 ∈ Ω , Cauchy’s formula at zero covers ker Σ ^ \operatorname{ker}\widehat{\Sigma} ker Σ as well.
We implicitly used the dominated convergence theorem to interchange the double integral and the trace, since the resolvent kernels are bounded on Γ 2 \Gamma^{2} Γ 2 and ∥ α ϕ ∥ T V 2 Tr Σ < ∞ \|\alpha_{\phi}\|_{\mathrm{TV}}^{2}\operatorname{Tr}\Sigma<\infty ∥ α ϕ ∥ TV 2 Tr Σ < ∞ .
Proof of Theorem 2
Our goal is to replace the two integrands in Lemma 2.1 with their deterministic equivalents in Theorems 4 and 5, respectively. We then integrate the errors over Γ 2 \Gamma^{2} Γ 2 . Accordingly, we advise readers to refer to Section 5 first, as the present proof relies on the path-interpolation estimates established there for the weighted resolvent trace and the mixed-resolvent trace. Throughout, we adopt the notation introduced in that section. We begin by computing the differences V X [ ϕ ] − V n [ ϕ ] \mathcal{V}_{X}[\phi]-V_{n}[\phi] V X [ ϕ ] − V n [ ϕ ] and B X [ ϕ ] − B n [ ϕ ] \mathcal{B}_{X}[\phi]-B_{n}[\phi] B X [ ϕ ] − B n [ ϕ ] in a form to which we can apply Theorems 4 and 5 of Section 5. We first work on the event { σ ^ n ≥ η n } \{\widehat{\sigma}_{n}\ge\eta_{n}\} { σ n ≥ η n } .
For the variance terms, we will take Q = I Q=I Q = I in Theorem 4. Recall that G n + 1 ( z ) = G n + 1 ( z , 0 ) = Tr ( Σ ( T n − z I ) − 1 ) G_{n+1}(z)=G_{n+1}(z,0)=\operatorname{Tr}(\Sigma(T_{n}-zI)^{-1}) G n + 1 ( z ) = G n + 1 ( z , 0 ) = Tr ( Σ ( T n − z I ) − 1 ) and G 1 ( z ) = G 1 ( z , w 0 l i n ( z ) ) G_{1}(z)=G_{1}(z,w_{0}^{\mathrm{lin}}(z)) G 1 ( z ) = G 1 ( z , w 0 lin ( z )) at the two endpoints. Combining Eqs. (6) and (30) , we have
z G 1 ( z ) = ∑ j σ j ω ( z / n ) σ j − ω ( z / n ) = ω ~ ( z ) − z . zG_{1}(z)=\sum_{j}\frac{\sigma_{j}\omega(z/n)}{\sigma_{j}-\omega(z/n)}=\widetilde{\omega}(z)-z. z G 1 ( z ) = j ∑ σ j − ω ( z / n ) σ j ω ( z / n ) = ω ( z ) − z .
Therefore, denote by χ G ( z ) : = z ( G n + 1 ( z , 0 ) − G 1 ( z , w 0 l i n ( z ) ) ) \chi^{G}(z):=z\left(G_{n+1}(z,0)-G_{1}(z,w_{0}^{\mathrm{lin}}(z))\right) χ G ( z ) := z ( G n + 1 ( z , 0 ) − G 1 ( z , w 0 lin ( z )) ) and its divided difference by χ G [ z , z ′ ] \chi^{G}[z,z'] χ G [ z , z ′ ] , we can write
V X [ ϕ ] − V n [ ϕ ] = τ 2 ∬ Γ 2 χ G [ z , z ′ ] α ϕ ( d z ) α ϕ ( d z ′ ) . \mathcal{V}_{X}[\phi]-V_{n}[\phi]=\tau^{2}\iint_{\Gamma^{2}}\chi^{G}[z,z']\alpha_{\phi}(\mathrm{d}z)\alpha_{\phi}(\mathrm{d}z'). V X [ ϕ ] − V n [ ϕ ] = τ 2 ∬ Γ 2 χ G [ z , z ′ ] α ϕ ( d z ) α ϕ ( d z ′ ) .
For bias terms, we take Q = θ θ ∗ Q=\theta\theta^{*} Q = θ θ ∗ in Theorem 5. Since β = Σ 1 / 2 θ \beta=\Sigma^{1/2}\theta β = Σ 1/2 θ , multiplying the sample trace by z z ′ zz' z z ′ recovers z z ′ β ∗ ( T n − z I ) − 1 Σ ( T n − z ′ I ) − 1 β zz'\beta^{*}(T_{n}-zI)^{-1}\Sigma(T_{n}-z'I)^{-1}\beta z z ′ β ∗ ( T n − z I ) − 1 Σ ( T n − z ′ I ) − 1 β . As ω ( z / n ) ( ω ( z / n ) I − Σ ) − 1 = ( I − n Σ / ω ~ ( z ) ) − 1 \omega(z/n)(\omega(z/n)I-\Sigma)^{-1}=(I-n\Sigma/\widetilde{\omega}(z))^{-1} ω ( z / n ) ( ω ( z / n ) I − Σ ) − 1 = ( I − n Σ/ ω ( z ) ) − 1 at each of the two arguments R ~ 0 R ~ 0 ′ \widetilde{R}_{0}\widetilde{R}_{0}' R 0 R 0 ′ , we recover the integrand of B n [ ϕ ] B_{n}[\phi] B n [ ϕ ] in Eq. (15). Denoting the trace difference in Eq. (32) at r = r ′ = 0 r=r'=0 r = r ′ = 0 by (cf. the opening of the proof of Theorem 5 in Appendix § 13 .§ 3 for the same notation used)
χ Q ( z , z ′ ) = Tr ( Q ( R ~ n R ~ n ′ − ω ~ [ z , z ′ ] R ~ 0 R ~ 0 ′ ) ) , \chi^{Q}(z,z')=\operatorname{Tr}\left(Q\left(\widetilde{R}_{n}\widetilde{R}_{n}'-\widetilde{\omega}[z,z']\widetilde{R}_{0}\widetilde{R}_{0}'\right)\right), χ Q ( z , z ′ ) = Tr ( Q ( R n R n ′ − ω [ z , z ′ ] R 0 R 0 ′ ) ) ,
With Q = θ θ ∗ Q=\theta\theta^{*} Q = θ θ ∗ , subtracting Eq. (15) from Lemma 2.1 yields
B X [ ϕ ] − B n [ ϕ ] = ∬ Γ 2 z z ′ χ Q ( z , z ′ ) α ϕ ( d z ) α ϕ ( d z ′ ) . \mathcal{B}_{X}[\phi]-B_{n}[\phi]=\iint_{\Gamma^{2}}zz'\chi^{Q}(z,z')\alpha_{\phi}(dz)\alpha_{\phi}(dz'). B X [ ϕ ] − B n [ ϕ ] = ∬ Γ 2 z z ′ χ Q ( z , z ′ ) α ϕ ( d z ) α ϕ ( d z ′ ) .
It remains to bound these errors for each pair on Γ 2 \Gamma^{2} Γ 2 . As sketched in the paragraph before Theorem 2, when the contour crosses the gap ( 0 , n η n ) (0,n\eta_{n}) ( 0 , n η n ) , Theorems 4 and 5 do not apply directly. The following lemma extends the approximation errors to the entire contour. We defer its proof to Appendix § 11 .§ 2.1 .
Lemma C.1. Under the assumptions of Theorem 2, for z , z ′ ∈ Γ z,z'\in\Gamma z , z ′ ∈ Γ and Q = θ θ ∗ Q=\theta\theta^{*} Q = θ θ ∗ ,
∥ 1 { σ ^ n ≥ η n } χ G [ z , z ′ ] ∥ L p = O ( p 2 Δ Γ 23 n ⋅ ( ∣ z ∣ ∨ n η n dist ( z , I n ) + ∣ z ′ ∣ ∨ n η n dist ( z ′ , I n ) ) ) , (40) \left\|\mathbf{1}\{\widehat{\sigma}_{n}\geq\eta_{n}\}\chi^{G}[z,z']\right\|_{L^{p}}=O\left(\frac{p^{2}\Delta_{\Gamma}^{23}}{n}\cdot\left(\frac{|z|\vee n\eta_{n}}{\operatorname{dist}(z,I_{n})}+\frac{|z'|\vee n\eta_{n}}{\operatorname{dist}(z',I_{n})}\right)\right), \tag*{(40)} 1 { σ n ≥ η n } χ G [ z , z ′ ] L p = O ( n p 2 Δ Γ 23 ⋅ ( dist ( z , I n ) ∣ z ∣ ∨ n η n + dist ( z ′ , I n ) ∣ z ′ ∣ ∨ n η n ) ) , (40) ∥ 1 { σ ^ n ≥ η n } z z ′ χ Q ( z , z ′ ) ∥ L p = O ( p 3 / 2 Δ Γ 26 ∥ θ ∥ 2 n 5 / 2 ⋅ ( ∣ z ∣ ∨ n η n ) ⋅ ( ∣ z ′ ∣ ∨ n η n ) ) , (41) \left\|\mathbf{1}\{\widehat{\sigma}_{n}\geq\eta_{n}\}zz'\chi^{Q}(z,z')\right\|_{L^{p}}=O\left(\frac{p^{3/2}\Delta_{\Gamma}^{26}\|\theta\|^{2}}{n^{5/2}}\cdot\left(|z|\vee n\eta_{n}\right)\cdot\left(|z'|\vee n\eta_{n}\right)\right), \tag*{(41)} 1 { σ n ≥ η n } z z ′ χ Q ( z , z ′ ) L p = O ( n 5/2 p 3/2 Δ Γ 26 ∥ θ ∥ 2 ⋅ ( ∣ z ∣ ∨ n η n ) ⋅ ( ∣ z ′ ∣ ∨ n η n ) ) , (41) where χ G [ z , z ] = ( χ G ) ′ ( z ) \chi^{G}[z,z]=(\chi^{G})'(z) χ G [ z , z ] = ( χ G ) ′ ( z ) .
Applying Minkowski’s integral inequality to the two contour identities and invoking Lemma C.1 , Eqs. (30a) and (30b), yields both bounds in Theorem 2. For the variance, the two summands have equal integrals by symmetry, and integration in the other variable gives ∥ α ϕ ∥ T V \|\alpha_{\phi}\|_{\mathrm{TV}} ∥ α ϕ ∥ TV . For the squared bias, the double integral factors as ( ∫ Γ ( ∣ z ∣ ∨ n η n ) ∣ α ϕ ∣ ( d z ) ) 2 \left(\int_{\Gamma}(|z|\vee n\eta_{n})|\alpha_{\phi}|(dz)\right)^{2} ( ∫ Γ ( ∣ z ∣ ∨ n η n ) ∣ α ϕ ∣ ( d z ) ) 2 .
Proof of Lemma C.1
Our proof consists of three parts. To simplify the notation, we use A n = { σ ^ n ≥ η n } A_{n}=\{\widehat{\sigma}_{n}\geq\eta_{n}\} A n = { σ n ≥ η n } in the proof.
In Part I, for variance, we reduce to a bound on χ G \chi^{G} χ G in a local disk centered at each z ∈ Γ z\in\Gamma z ∈ Γ .
In Part II, we apply Theorems 4 and 5 where spectral stability holds, and use a crude bound at smaller spectral heights.
In Part III, we upgrade the crude bounds by applying submean inequality, averaging the logarithm of each normalized error over the disks to recover the desired bounds.
Part I: Reduction for χ G \chi^{G} χ G . The key observation is that we can reduce the divided difference into contour integrals on the boundary of two disks. For z , z ′ ∈ Γ z,z'\in\Gamma z , z ′ ∈ Γ , applying Cauchy’s formula on the union of the local disks centered at z , z ′ z,z' z , z ′ with radii dist ( z , I n ) / 4 , dist ( z ′ , I n ) / 4 \operatorname{dist}(z,I_{n})/4,\operatorname{dist}(z',I_{n})/4 dist ( z , I n ) /4 , dist ( z ′ , I n ) /4 , we immediately have
χ G [ z , z ′ ] = 1 2 π i ∮ Γ z ∣ z ′ ∪ Γ z ′ ∣ z χ G ( ζ ) ( ζ − z ) ( ζ − z ′ ) d ζ , \chi^{G}[z,z']=\frac{1}{2\pi\mathrm{i}}\oint_{\Gamma_{z|z'}\cup\Gamma_{z'|z}}\frac{\chi^{G}(\zeta)}{(\zeta-z)(\zeta-z')}\,d\zeta, χ G [ z , z ′ ] = 2 π i 1 ∮ Γ z ∣ z ′ ∪ Γ z ′ ∣ z ( ζ − z ) ( ζ − z ′ ) χ G ( ζ ) d ζ ,
where Γ z ∣ z ′ \Gamma_{z|z'} Γ z ∣ z ′ and Γ z ′ ∣ z \Gamma_{z'|z} Γ z ′ ∣ z are the two arcs on the boundary, centered at z z z and z ′ z' z ′ respectively. The integral is positively oriented on their union and χ G \chi^{G} χ G is zero outside A n A_{n} A n . This naturally recovers the derivative convention, as at z = z ′ z=z' z = z ′ , this is Cauchy’s derivative formula. For ζ ∈ Γ z ∣ z ′ \zeta\in\Gamma_{z|z'} ζ ∈ Γ z ∣ z ′ , ∣ ζ − z ∣ = dist ( z , I n ) / 4 |\zeta-z|=\operatorname{dist}(z,I_n)/4 ∣ ζ − z ∣ = dist ( z , I n ) /4 and ∣ ζ − z ′ ∣ ≥ dist ( z ′ , I n ) / 4 |\zeta-z'|\ge\operatorname{dist}(z',I_n)/4 ∣ ζ − z ′ ∣ ≥ dist ( z ′ , I n ) /4 . Hence
3 4 dist ( z , I n ) ≤ dist ( ζ , I n ) ≤ ∣ ζ − z ′ ∣ + dist ( z ′ , I n ) ≤ 5 ∣ ζ − z ′ ∣ . \frac{3}{4}\operatorname{dist}(z,I_n)\le\operatorname{dist}(\zeta,I_n)\le|\zeta-z'|+\operatorname{dist}(z',I_n)\le5|\zeta-z'|. 4 3 dist ( z , I n ) ≤ dist ( ζ , I n ) ≤ ∣ ζ − z ′ ∣ + dist ( z ′ , I n ) ≤ 5∣ ζ − z ′ ∣.
Since Γ z ∣ z ′ \Gamma_{z|z'} Γ z ∣ z ′ lies on the circle centered at z z z with radius dist ( z , I n ) / 4 \operatorname{dist}(z,I_n)/4 dist ( z , I n ) /4 , its length is at most the circumference 2 π ⋅ dist ( z , I n ) / 4 = O ( dist ( z , I n ) ) 2\pi\cdot\operatorname{dist}(z,I_n)/4=O(\operatorname{dist}(z,I_n)) 2 π ⋅ dist ( z , I n ) /4 = O ( dist ( z , I n )) . We can apply Minkowski’s inequality and the preceding denominator bound to conclude that
∥ 1 2 π i ∫ Γ z ∣ z ′ χ G ( ζ ) ( ζ − z ) ( ζ − z ′ ) d ζ ∥ L p = O ( 1 dist ( z , I n ) sup ∣ ζ − z ∣ ≤ dist ( z , I n ) / 4 ∥ χ G ( ζ ) ∥ L p ) . \left\|\frac{1}{2\pi i}\int_{\Gamma_{z|z'}}\frac{\chi^G(\zeta)}{(\zeta-z)(\zeta-z')}\,\mathrm{d}\zeta\right\|_{L^p}
=O\left(\frac{1}{\operatorname{dist}(z,I_n)}\sup_{|\zeta-z|\le\operatorname{dist}(z,I_n)/4}\|\chi^G(\zeta)\|_{L^p}\right). 2 π i 1 ∫ Γ z ∣ z ′ ( ζ − z ) ( ζ − z ′ ) χ G ( ζ ) d ζ L p = O ( dist ( z , I n ) 1 ∣ ζ − z ∣ ≤ dist ( z , I n ) /4 sup ∥ χ G ( ζ ) ∥ L p ) .
Similarly, we have the integral bound over Γ z ′ ∣ z \Gamma_{z'|z} Γ z ′ ∣ z . Combine those two bounds and compare with Eq. (30a), it is then clear that it suffices to prove the following local disk bound for the variance:
sup ∣ z ′ − z ∣ ≤ dist ( z , I n ) / 4 ∥ 1 A n χ G ( z ′ ) ∥ L p = O ( p 2 Δ Γ 23 n ( ∣ z ∣ ∨ n η n ) ) , z ∈ Γ . (42) \sup_{|z'-z|\le\operatorname{dist}(z,I_n)/4}\|\mathbb{1}_{A_n}\chi^G(z')\|_{L^p}
=O\left(\frac{p^2\Delta_\Gamma^{23}}{n}(|z|\vee n\eta_n)\right),\qquad z\in\Gamma. \tag*{(42)} ∣ z ′ − z ∣ ≤ dist ( z , I n ) /4 sup ∥ 1 A n χ G ( z ′ ) ∥ L p = O ( n p 2 Δ Γ 23 ( ∣ z ∣ ∨ n η n ) ) , z ∈ Γ. (42)
Part II: Separate bounds inside and outside the horizontal strip. We bound χ G ( z ′ ) \chi^G(z') χ G ( z ′ ) on the local disk in Eq. (42) and z z ′ χ Q ( z , z ′ ) zz'\chi^Q(z,z') z z ′ χ Q ( z , z ′ ) for z , z ′ ∈ Γ z,z'\in\Gamma z , z ′ ∈ Γ . Theorems 4 and 5 apply directly outside the horizontal strip
{ ζ ∈ C : ∣ Im ζ ∣ < h n dist ( z , I n ) } . \{\zeta\in\mathbb{C}:|\operatorname{Im}\zeta|<h_n\operatorname{dist}(z,I_n)\}. { ζ ∈ C : ∣ Im ζ ∣ < h n dist ( z , I n )} .
associated with each z ∈ Γ z\in\Gamma z ∈ Γ . We will select the relative height h n > 0 h_n>0 h n > 0 later.
Our first step is to apply Theorems 4 and 5 outside the strip, by verifying Assumption 2 uniformly there. Let D z : = { ζ ∈ C : ∣ ζ − z ∣ ≤ dist ( z , I n ) / 4 } D_z:=\{\zeta\in\mathbb{C}:|\zeta-z|\le\operatorname{dist}(z,I_n)/4\} D z := { ζ ∈ C : ∣ ζ − z ∣ ≤ dist ( z , I n ) /4 } . For variance, we take z ∈ D z z\in D_z z ∈ D z and ζ ∈ D z ′ \zeta\in D_{z'} ζ ∈ D z ′ . For squared bias we take ( ζ , ζ ′ ) ∈ D z × D z ′ (\zeta,\zeta')\in D_z\times D_{z'} ( ζ , ζ ′ ) ∈ D z × D z ′ with z , z ′ ∈ Γ z,z'\in\Gamma z , z ′ ∈ Γ . Since distance to I n I_n I n is 1-Lipschitz, dist ( z ′ , I n ) ≍ dist ( z , I n ) \operatorname{dist}(z',I_n)\asymp\operatorname{dist}(z,I_n) dist ( z ′ , I n ) ≍ dist ( z , I n ) when z ′ ∈ D z z'\in D_z z ′ ∈ D z . Thus in either case ∣ ζ − z ∣ = O ( dist ( z , I n ) ) |\zeta-z|=O(\operatorname{dist}(z,I_n)) ∣ ζ − z ∣ = O ( dist ( z , I n )) and ∣ ζ ∣ = O ( ∣ z ∣ ∨ n η n ) |\zeta|=O(|z|\vee n\eta_n) ∣ ζ ∣ = O ( ∣ z ∣ ∨ n η n ) . Outside the strip, ∣ ζ − ⋅ ∣ ≥ ∣ Im ζ ∣ ≥ h n dist ( z , I n ) |\zeta-\cdot|\ge|\operatorname{Im}\zeta|\ge h_n\operatorname{dist}(z,I_n) ∣ ζ − ⋅ ∣ ≥ ∣ Im ζ ∣ ≥ h n dist ( z , I n ) .
To verify Assumption 2, we make use of the fact that ∣ ζ − ⋅ ∣ Λ n ( ζ ) − ∣ ζ ∣ |\zeta-\cdot|\Lambda_n(\zeta)-|\zeta| ∣ ζ − ⋅ ∣ Λ n ( ζ ) − ∣ ζ ∣ is the minimized spectral tail and hence takes the same value at ζ \zeta ζ and at − dist ( z , I n ) -\operatorname{dist}(z,I_n) − dist ( z , I n ) . Since r ↦ Λ n ( − r ) r\mapsto\Lambda_n(-r) r ↦ Λ n ( − r ) is nonincreasing for r > 0 r>0 r > 0 , we have Λ n ( − dist ( z , I n ) ) ≤ Δ Γ \Lambda_n(-\operatorname{dist}(z,I_n))\le\Delta_\Gamma Λ n ( − dist ( z , I n )) ≤ Δ Γ and ∣ z ∣ ≤ Δ Γ dist ( z , I n ) |z|\le\Delta_\Gamma\operatorname{dist}(z,I_n) ∣ z ∣ ≤ Δ Γ dist ( z , I n ) from Theorem 2. For any ∣ ζ − z ∣ ≤ 3 dist ( z , I n ) / 4 |\zeta-z|\le3\operatorname{dist}(z,I_n)/4 ∣ ζ − z ∣ ≤ 3 dist ( z , I n ) /4 and Im ζ ≠ 0 \operatorname{Im}\zeta\ne0 Im ζ = 0 , it holds
∣ ζ − ⋅ ∣ Λ n ( ζ ) = ∣ ζ ∣ + dist ( z , I n ) ⋅ ( Λ n ( − dist ( z , I n ) ) − 1 ) ≤ 2 Δ Γ dist ( z , I n ) . (43) |\zeta-\cdot|\Lambda_n(\zeta)=|\zeta|+\operatorname{dist}(z,I_n)\cdot\left(\Lambda_n(-\operatorname{dist}(z,I_n))-1\right)\le2\Delta_\Gamma\operatorname{dist}(z,I_n). \tag*{(43)} ∣ ζ − ⋅ ∣ Λ n ( ζ ) = ∣ ζ ∣ + dist ( z , I n ) ⋅ ( Λ n ( − dist ( z , I n )) − 1 ) ≤ 2 Δ Γ dist ( z , I n ) . (43)
To see that we can use Eq. (43) for ζ ∈ D z ′ \zeta\in D_{z'} ζ ∈ D z ′ with z ′ ∈ D z z'\in D_z z ′ ∈ D z . By dist ( z ′ , I n ) ≤ dist ( z , I n ) + ∣ z ′ − z ∣ ≤ 5 dist ( z , I n ) / 4 \operatorname{dist}(z',I_n)\le\operatorname{dist}(z,I_n)+|z'-z|\le5\operatorname{dist}(z,I_n)/4 dist ( z ′ , I n ) ≤ dist ( z , I n ) + ∣ z ′ − z ∣ ≤ 5 dist ( z , I n ) /4 , it holds for ζ ∈ D z ′ \zeta\in D_{z'} ζ ∈ D z ′ ,
∣ ζ − z ∣ ≤ ∣ ζ − z ′ ∣ + ∣ z ′ − z ∣ ≤ dist ( z ′ , I n ) 4 + dist ( z , I n ) 4 ≤ ( 5 16 + 1 4 ) dist ( z , I n ) < 3 4 dist ( z , I n ) . |\zeta-z|\le|\zeta-z'|+|z'-z|\le\frac{\operatorname{dist}(z',I_n)}{4}+\frac{\operatorname{dist}(z,I_n)}{4}\le\left(\frac{5}{16}+\frac{1}{4}\right)\operatorname{dist}(z,I_n)<\frac{3}{4}\operatorname{dist}(z,I_n). ∣ ζ − z ∣ ≤ ∣ ζ − z ′ ∣ + ∣ z ′ − z ∣ ≤ 4 dist ( z ′ , I n ) + 4 dist ( z , I n ) ≤ ( 16 5 + 4 1 ) dist ( z , I n ) < 4 3 dist ( z , I n ) .
Therefore, for the variance term and ζ ∈ D z ∪ D z ′ \zeta\in D_z\cup D_{z'} ζ ∈ D z ∪ D z ′ outside the strip, Eq. (43) implies Λ n ( ζ ) ≤ 2 Δ Γ / h n \Lambda_n(\zeta)\le2\Delta_\Gamma/h_n Λ n ( ζ ) ≤ 2 Δ Γ / h n . We can choose h n = C Δ Γ n − 1 / 16 h_n=C\Delta_\Gamma n^{-1/16} h n = C Δ Γ n − 1/16 and ν = C Γ + 1 \nu=C_\Gamma+1 ν = C Γ + 1 , which yields
Λ n ( ζ ) 16 ≤ ( 2 / C ) 16 n and ∣ ζ − ⋅ ∣ ≥ h n dist ( Γ , I n ) ≥ C Δ Γ n − 1 / 16 − C Γ ≥ n − ν . \Lambda_n(\zeta)^{16}\le(2/C)^{16}n\qquad\text{and}\qquad|\zeta-\cdot|\ge h_n\operatorname{dist}(\Gamma,I_n)\ge C\Delta_\Gamma n^{-1/16-C_\Gamma}\ge n^{-\nu}. Λ n ( ζ ) 16 ≤ ( 2/ C ) 16 n and ∣ ζ − ⋅ ∣ ≥ h n dist ( Γ , I n ) ≥ C Δ Γ n − 1/16 − C Γ ≥ n − ν .
Choosing C C C sufficiently large as a function of C x , κ , C Γ C_x,\kappa,C_\Gamma C x , κ , C Γ ensures that Assumption 2 holds uniformly for all sufficiently large n n n . Moreover, the requirement on C Γ C_\Gamma C Γ in Theorem 2 implies that h n = O ( 1 / log n ) h_n=O(1/\log n) h n = O ( 1/ log n ) .
With Assumption 2 verified, we apply Theorem 4 with Q = I Q=I Q = I and, if ∣ Im ζ ′ ∣ ≥ h n dist ( z ′ , I n ) |\operatorname{Im}\zeta'|\ge h_n\operatorname{dist}(z',I_n) ∣ Im ζ ′ ∣ ≥ h n dist ( z ′ , I n ) as well, Theorem 5 with Q = θ θ ∗ Q=\theta\theta^* Q = θ θ ∗ to bound χ G \chi^G χ G and χ Q \chi^Q χ Q . At derivative order zero, using reflection for lower-half-plane arguments (for opposite half-planes, see the extension after Theorem 5), we have results outside the strip,
∥ 1 A n χ G ( ζ ) ∥ L p = O ( p 2 Δ Γ 23 n ( ∣ z ∣ ∨ n η n ) { 1 ∨ ( dist ( z , I n ) ∣ Im ζ ∣ ) 23 } ) . \left\|\mathbf{1}_{\mathcal{A}_n}\chi^G(\zeta)\right\|_{L^p}=O\left(\frac{p^2\Delta_{\Gamma}^{23}}{n}(|z|\vee n\eta_n)\left\{1\vee\left(\frac{\operatorname{dist}(z,I_n)}{|\operatorname{Im}\zeta|}\right)^{23}\right\}\right). 1 A n χ G ( ζ ) L p = O ( n p 2 Δ Γ 23 ( ∣ z ∣ ∨ n η n ) { 1 ∨ ( ∣ Im ζ ∣ dist ( z , I n ) ) 23 } ) .
∥ 1 A n ζ ζ ′ χ Q ( ζ , ζ ′ ) ∥ L p = O ( p 3 / 2 Δ Γ 26 ∥ θ ∥ 2 n 5 / 2 ( ∣ z ∣ ∨ n η n ) ( ∣ z ′ ∣ ∨ n η n ) { 1 ∨ dist ( z , I n ) ∣ Im ζ ∣ ∨ dist ( z ′ , I n ) ∣ Im ζ ′ ∣ } 26 ) . (44) \left\|\mathbf{1}_{\mathcal{A}_n}\zeta\zeta'\chi^Q(\zeta,\zeta')\right\|_{L^p}=O\left(\frac{p^{3/2}\Delta_{\Gamma}^{26}\|\theta\|^2}{n^{5/2}}(|z|\vee n\eta_n)(|z'|\vee n\eta_n)\left\{1\vee\frac{\operatorname{dist}(z,I_n)}{|\operatorname{Im}\zeta|}\vee\frac{\operatorname{dist}(z',I_n)}{|\operatorname{Im}\zeta'|}\right\}^{26}\right). \tag*{(44)} 1 A n ζ ζ ′ χ Q ( ζ , ζ ′ ) L p = O ( n 5/2 p 3/2 Δ Γ 26 ∥ θ ∥ 2 ( ∣ z ∣ ∨ n η n ) ( ∣ z ′ ∣ ∨ n η n ) { 1 ∨ ∣ Im ζ ∣ dist ( z , I n ) ∨ ∣ Im ζ ′ ∣ dist ( z ′ , I n ) } 26 ) . (44)
When either ζ \zeta ζ or ζ ′ \zeta' ζ ′ lies inside its strip, we will instead use the polynomial bounds in the following lemma, proved in Appendix C.2.2.
Lemma C.2. Under the setup of Lemma C.1, when A n \mathcal{A}_n A n holds, the following bounds hold for some fixed K K K and all sufficiently large n n n , uniformly in the contour points.
sup z ′ ∈ D z sup ζ ∈ D z ′ ∣ χ G ( ζ ) ∣ ≤ n K ∣ z ∣ ∨ n η n n , (45) \sup_{z'\in D_z}\sup_{\zeta\in D_{z'}}|\chi^G(\zeta)|\le n^K\frac{|z|\vee n\eta_n}{n}, \tag*{(45)} z ′ ∈ D z sup ζ ∈ D z ′ sup ∣ χ G ( ζ ) ∣ ≤ n K n ∣ z ∣ ∨ n η n , (45) sup ( ζ , ζ ′ ) ∈ D z × D z ′ ∣ ζ ζ ′ χ Q ( ζ , ζ ′ ) ∣ ≤ n K ∥ θ ∥ 2 n 5 / 2 ( ∣ z ∣ ∨ n η n ) ( ∣ z ′ ∣ ∨ n η n ) . (46) \sup_{(\zeta,\zeta')\in D_z\times D_{z'}}|\zeta\zeta'\chi^Q(\zeta,\zeta')|\le n^K\frac{\|\theta\|^2}{n^{5/2}}(|z|\vee n\eta_n)(|z'|\vee n\eta_n). \tag*{(46)} ( ζ , ζ ′ ) ∈ D z × D z ′ sup ∣ ζ ζ ′ χ Q ( ζ , ζ ′ ) ∣ ≤ n K n 5/2 ∥ θ ∥ 2 ( ∣ z ∣ ∨ n η n ) ( ∣ z ′ ∣ ∨ n η n ) . (46)
Combining the crude and fine-grained estimates by the logarithmic submean inequality, we can close the proof in the next part.
Part III: Combining bounds by the submean inequality. For z ∈ Γ z\in\Gamma z ∈ Γ and z ′ ∈ D z z'\in D_z z ′ ∈ D z , on A n \mathcal{A}_n A n the scaled trace error is analytic samplewise on D z ′ D_{z'} D z ′ by spectral separation and ω ~ = − 1 / s n \widetilde{\omega}=-1/s_n ω = − 1/ s n , and the uniform bound in Lemma C.2 makes 1 A n χ G \mathbf{1}_{\mathcal{A}_n}\chi^G 1 A n χ G analytic there in L p L^p L p by Cauchy’s formula. Applying the submean inequality to its log-norm [5 Lemma 3.4.6] , with log + x : = log ( 1 ∨ x ) \log^+x:=\log(1\vee x) log + x := log ( 1 ∨ x ) , yields
log + n ∥ 1 A n χ G ( z ′ ) ∥ L p p 2 Δ Γ 23 ( ∣ z ∣ ∨ n η n ) ≤ 1 ∣ D z ′ ∣ ∫ D z ′ log + n ∥ 1 A n χ G ( ζ ) ∥ L p p 2 Δ Γ 23 ( ∣ z ∣ ∨ n η n ) d A ( ζ ) . \log^+\frac{n\|\mathbf{1}_{\mathcal{A}_n}\chi^G(z')\|_{L^p}}{p^2\Delta_{\Gamma}^{23}(|z|\vee n\eta_n)}
\le\frac{1}{|D_{z'}|}\int_{D_{z'}}\log^+\frac{n\|\mathbf{1}_{\mathcal{A}_n}\chi^G(\zeta)\|_{L^p}}{p^2\Delta_{\Gamma}^{23}(|z|\vee n\eta_n)}\,dA(\zeta). log + p 2 Δ Γ 23 ( ∣ z ∣ ∨ n η n ) n ∥ 1 A n χ G ( z ′ ) ∥ L p ≤ ∣ D z ′ ∣ 1 ∫ D z ′ log + p 2 Δ Γ 23 ( ∣ z ∣ ∨ n η n ) n ∥ 1 A n χ G ( ζ ) ∥ L p d A ( ζ ) .
Here d A dA d A denotes area measure, and the denominator depends only on the fixed contour point z z z . It suffices to bound the average on the right by O ( 1 ) O(1) O ( 1 ) . By splitting at the strip boundary, i.e. ∣ Im ζ ∣ = h n dist ( z , I n ) |\operatorname{Im}\zeta|=h_n\operatorname{dist}(z,I_n) ∣ Im ζ ∣ = h n dist ( z , I n ) , we apply Lemma C.2 inside the strip and Eq. (44) outside to obtain
1 ∣ D z ′ ∣ ∫ D z ′ log + n ∥ 1 A n χ G ( ζ ) ∥ L p p 2 Δ Γ 23 ( ∣ z ∣ ∨ n η n ) d A ( ζ ) ≤ O ( h n K log n ) + O ( 1 + 23 ∫ h n 1 log ( 1 / y ) d y ) = O ( 1 ) . \frac{1}{|D_{z'}|}\int_{D_{z'}}\log^+\frac{n\|\mathbf{1}_{\mathcal{A}_n}\chi^G(\zeta)\|_{L^p}}{p^2\Delta_{\Gamma}^{23}(|z|\vee n\eta_n)}\,dA(\zeta)
\le O(h_nK\log n)+O\left(1+23\int_{h_n}^{1}\log(1/y)\,dy\right)=O(1). ∣ D z ′ ∣ 1 ∫ D z ′ log + p 2 Δ Γ 23 ( ∣ z ∣ ∨ n η n ) n ∥ 1 A n χ G ( ζ ) ∥ L p d A ( ζ ) ≤ O ( h n K log n ) + O ( 1 + 23 ∫ h n 1 log ( 1/ y ) d y ) = O ( 1 ) .
Inside the strip, Lemma C.2 bounds the ratio inside log + \log^+ log + by n K / ( p 2 Δ Γ 23 ) ≤ n K n^K/(p^2\Delta_{\Gamma}^{23})\le n^K n K / ( p 2 Δ Γ 23 ) ≤ n K , since the factor ( ∣ z ∣ ∨ n η n ) / n (|z|\vee n\eta_n)/n ( ∣ z ∣ ∨ n η n ) / n cancels and p , Δ Γ ≥ 1 p,\Delta_{\Gamma}\ge1 p , Δ Γ ≥ 1 . Outside the strip, the full prefactor in Eq. (44) cancels, leaving at most log + C + 23 log + ( 1 / y ) \log^+C+23\log^+(1/y) log + C + 23 log + ( 1/ y ) with y = ∣ Im ζ ∣ / dist ( z , I n ) y=|\operatorname{Im}\zeta|/\operatorname{dist}(z,I_n) y = ∣ Im ζ ∣/ dist ( z , I n ) . The within strip term thus contributes O ( h n K log n ) O(h_nK\log n) O ( h n K log n ) , and the remaining integral is bounded by ∫ 0 1 log ( 1 / y ) d y = 1 \int_0^1\log(1/y)\,dy=1 ∫ 0 1 log ( 1/ y ) d y = 1 . Since h n log n = O ( 1 ) h_n\log n=O(1) h n log n = O ( 1 ) , both contributions are bounded. Exponentiating the submean bound proves Eq. (42) uniformly for z ′ ∈ D z z'\in D_z z ′ ∈ D z , and Part I closes the variance bound of Lemma C.1.
For the squared bias, we repeat the same argument by applying the submean inequality to 1 A n ζ ζ ′ χ Q ( ζ , ζ ′ ) \mathbf{1}_{\mathcal{A}_n}\zeta\zeta'\chi^Q(\zeta,\zeta') 1 A n ζ ζ ′ χ Q ( ζ , ζ ′ ) successively on D z D_z D z and D z ′ D_{z'} D z ′ . We can therefore have Eq. (30b).
Proof of Lemma C.2
Part I: Proof of Eq. (34a). On A n \mathcal{A}_n A n , we bound ∣ χ G ( ζ ) ∣ |\chi^G(\zeta)| ∣ χ G ( ζ ) ∣ for z ′ ∈ D z z' \in D_z z ′ ∈ D z and ζ ∈ D z ′ \zeta\in D_{z'} ζ ∈ D z ′ . We work on the larger disk ∣ ζ − z ∣ ≤ 3 dist ( z , I n ) / 4 |\zeta-z| \le3\operatorname{dist}(z,I_n)/4 ∣ ζ − z ∣ ≤ 3 dist ( z , I n ) /4 , which contains every such D z ′ D_{z'} D z ′ by the disk inclusion following Eq. (43) . Part II will use the bound for ∣ ω ~ ( ζ ) − ζ ∣ |\widetilde{\omega}(\zeta)-\zeta| ∣ ω ( ζ ) − ζ ∣ on these same disks to estimate a divided difference, so we provide bounds on them together. Taking Q = I Q=I Q = I in Eq. (30) and using ω ~ = − 1 / s n \widetilde{\omega}=-1/s_n ω = − 1/ s n , we can write both endpoints as weighted traces.
ζ G n + 1 ( ζ ) = Tr ( Σ ζ ( T n − ζ I ) − 1 ) , ζ G 1 ( ζ ) = Tr ( Σ ( Σ ω ( ζ / n ) − I ) − 1 ) = − Tr ( Σ ( I + n s n ( ζ ) Σ ) − 1 ) = ω ~ ( ζ ) − ζ . \zeta G_{n+1}(\zeta)=\operatorname{Tr}\left(\Sigma\zeta(T_n-\zeta I)^{-1}\right),
\qquad
\zeta G_1(\zeta)=\operatorname{Tr}\left(\Sigma\left(\frac{\Sigma}{\omega(\zeta/n)}-I\right)^{-1}\right)=-\operatorname{Tr}\left(\Sigma(I+ns_n(\zeta)\Sigma)^{-1}\right)=\widetilde{\omega}(\zeta)-\zeta. ζ G n + 1 ( ζ ) = Tr ( Σ ζ ( T n − ζ I ) − 1 ) , ζ G 1 ( ζ ) = Tr ( Σ ( ω ( ζ / n ) Σ − I ) − 1 ) = − Tr ( Σ ( I + n s n ( ζ ) Σ ) − 1 ) = ω ( ζ ) − ζ .
It thus suffices to bound the two operator norms. For the sample endpoint, the spectrum of T n T_n T n lies in { 0 } ∪ I n \{0\}\cup I_n { 0 } ∪ I n on A n \mathcal{A}_n A n , with ζ / ( 0 − ζ ) = − 1 \zeta/(0-\zeta)=-1 ζ / ( 0 − ζ ) = − 1 by continuation. For the population endpoint, diagonalize Σ \Sigma Σ . If Re ζ ≤ n η n \operatorname{Re}\zeta\le n\eta_n Re ζ ≤ n η n , then Re s n ( ζ ) ≥ 0 \operatorname{Re}s_n(\zeta)\ge0 Re s n ( ζ ) ≥ 0 since supp μ n ⊂ I n \operatorname{supp}\mu_n\subset I_n supp μ n ⊂ I n , so ∣ 1 + n σ j s n ( ζ ) ∣ ≥ 1 |1+n\sigma_j s_n(\zeta)|\ge1 ∣1 + n σ j s n ( ζ ) ∣ ≥ 1 . Otherwise dist ( ζ , I n ) = ∣ Im ζ ∣ \operatorname{dist}(\zeta,I_n)=|\operatorname{Im}\zeta| dist ( ζ , I n ) = ∣ Im ζ ∣ , and Im ( ζ s n ( ζ ) ) = Im ζ ∫ I n x ∣ x − ζ ∣ − 2 μ n ( d x ) \operatorname{Im}(\zeta s_n(\zeta))=\operatorname{Im}\zeta\int_{I_n}x|x-\zeta|^{-2}\mu_n(dx) Im ( ζ s n ( ζ )) = Im ζ ∫ I n x ∣ x − ζ ∣ − 2 μ n ( d x ) implies ∣ ζ ∣ ⋅ ∣ 1 + n σ j s n ( ζ ) ∣ ≥ ∣ Im ( ζ ( 1 + n σ j s n ( ζ ) ) ) ∣ ≥ ∣ Im ζ ∣ |\zeta|\cdot|1+n\sigma_j s_n(\zeta)|\ge|\operatorname{Im}(\zeta(1+n\sigma_j s_n(\zeta)))|\ge|\operatorname{Im}\zeta| ∣ ζ ∣ ⋅ ∣1 + n σ j s n ( ζ ) ∣ ≥ ∣ Im ( ζ ( 1 + n σ j s n ( ζ ))) ∣ ≥ ∣ Im ζ ∣ . Hence
∥ ζ ( T n − ζ I ) − 1 ∥ ∨ ∥ ( I + n s n ( ζ ) Σ ) − 1 ∥ ≤ 1 ∨ ∣ ζ ∣ dist ( ζ , I n ) = O ( Δ Γ ) . \left\|\zeta(T_n-\zeta I)^{-1}\right\|\vee\left\|(I+ns_n(\zeta)\Sigma)^{-1}\right\|\le1\vee\frac{|\zeta|}{\operatorname{dist}(\zeta,I_n)}=O(\Delta_\Gamma). ζ ( T n − ζ I ) − 1 ∨ ( I + n s n ( ζ ) Σ ) − 1 ≤ 1 ∨ dist ( ζ , I n ) ∣ ζ ∣ = O ( Δ Γ ) .
where dist ( ζ , I n ) ≥ dist ( z , I n ) / 4 \operatorname{dist}(\zeta,I_n)\ge\operatorname{dist}(z,I_n)/4 dist ( ζ , I n ) ≥ dist ( z , I n ) /4 and ∣ z ∣ ≤ Δ Γ dist ( z , I n ) |z|\le\Delta_\Gamma\operatorname{dist}(z,I_n) ∣ z ∣ ≤ Δ Γ dist ( z , I n ) . Applying ∣ Tr ( Σ A ) ∣ ≤ Tr ( Σ ) ∥ A ∥ |\operatorname{Tr}(\Sigma A)|\le\operatorname{Tr}(\Sigma)\|A\| ∣ Tr ( Σ A ) ∣ ≤ Tr ( Σ ) ∥ A ∥ to each endpoint, we have
∣ χ G ( ζ ) ∣ ≤ ∣ ζ G n + 1 ( ζ ) ∣ + ∣ ζ G 1 ( ζ ) ∣ ≤ Tr ( Σ ) ( ∥ ζ ( T n − ζ I ) − 1 ∥ + ∥ ( I + n s n ( ζ ) Σ ) − 1 ∥ ) = O ( Δ Γ Tr Σ ) . |\chi^G(\zeta)|\le|\zeta G_{n+1}(\zeta)|+|\zeta G_1(\zeta)|\le\operatorname{Tr}(\Sigma)\left(\left\|\zeta(T_n-\zeta I)^{-1}\right\|+\left\|(I+ns_n(\zeta)\Sigma)^{-1}\right\|\right)=O(\Delta_\Gamma\operatorname{Tr}\Sigma). ∣ χ G ( ζ ) ∣ ≤ ∣ ζ G n + 1 ( ζ ) ∣ + ∣ ζ G 1 ( ζ ) ∣ ≤ Tr ( Σ ) ( ζ ( T n − ζ I ) − 1 + ( I + n s n ( ζ ) Σ ) − 1 ) = O ( Δ Γ Tr Σ ) .
Note that the same bound holds for ∣ ω ~ ( ζ ) − ζ ∣ |\widetilde{\omega}(\zeta)-\zeta| ∣ ω ( ζ ) − ζ ∣ .
We are now ready to close the proof of Eq. (34a). Taking a minimizing cutoff m m m in Λ n ( − dist ( Γ , I n ) ) \Lambda_n(-\operatorname{dist}(\Gamma,I_n)) Λ n ( − dist ( Γ , I n )) and using ∥ Σ ∥ = 1 \|\Sigma\|=1 ∥Σ∥ = 1 and Eq. (10) , we have Tr Σ ≤ m + τ 1 ( m ) ≤ n + dist ( Γ , I n ) Δ Γ \operatorname{Tr}\Sigma\le m+\tau_1(m)\le n+\operatorname{dist}(\Gamma,I_n)\Delta_\Gamma Tr Σ ≤ m + τ 1 ( m ) ≤ n + dist ( Γ , I n ) Δ Γ . Since dist ( Γ , I n ) ≤ ∣ z − n η n ∣ ≤ 2 ( ∣ z ∣ ∨ n η n ) \operatorname{dist}(\Gamma,I_n)\le|z-n\eta_n|\le2(|z|\vee n\eta_n) dist ( Γ , I n ) ≤ ∣ z − n η n ∣ ≤ 2 ( ∣ z ∣ ∨ n η n ) ,
n ∣ χ G ( ζ ) ∣ ∣ z ∣ ∨ n η n = O ( n Δ Γ ⋅ { n dist ( Γ , I n ) + Δ Γ } ) . \frac{n|\chi^G(\zeta)|}{|z|\vee n\eta_n}=O\left(n\Delta_\Gamma\cdot\left\{\frac{n}{\operatorname{dist}(\Gamma,I_n)}+\Delta_\Gamma\right\}\right). ∣ z ∣ ∨ n η n n ∣ χ G ( ζ ) ∣ = O ( n Δ Γ ⋅ { dist ( Γ , I n ) n + Δ Γ } ) .
As Theorem 2 assumes dist ( Γ , I n ) ≥ n − C Γ \operatorname{dist}(\Gamma,I_n)\ge n^{-C_\Gamma} dist ( Γ , I n ) ≥ n − C Γ and Δ Γ = O ( n 1 / 16 / log n ) \Delta_\Gamma=O(n^{1/16}/\log n) Δ Γ = O ( n 1/16 / log n ) , we conclude that the right-hand side is at most n K n^K n K for some fixed K K K . We complete the proof for the first bound.
Part II: Proof of Eq. (34b). For ( ζ , ζ ′ ) ∈ D z × D z ′ (\zeta,\zeta')\in D_z\times D_{z'} ( ζ , ζ ′ ) ∈ D z × D z ′ , applying the two resolvent bounds from Part I to the mixed-trace identity in Appendix § 11 .§ 2 , with β = Σ 1 / 2 θ \beta=\Sigma^{1/2}\theta β = Σ 1/2 θ and ∥ Σ ∥ = 1 \|\Sigma\|=1 ∥Σ∥ = 1 , yields
∣ ζ ζ ′ χ Q ( ζ , ζ ′ ) ∣ ≤ O ( ∥ θ ∥ 2 Δ Γ 2 ) ( 1 + ∣ ω ~ [ ζ , ζ ′ ] ∣ ) . |\zeta\zeta'\chi^Q(\zeta,\zeta')|\le O(\|\theta\|^2\Delta_\Gamma^2)(1+|\widetilde{\omega}[\zeta,\zeta']|). ∣ ζ ζ ′ χ Q ( ζ , ζ ′ ) ∣ ≤ O ( ∥ θ ∥ 2 Δ Γ 2 ) ( 1 + ∣ ω [ ζ , ζ ′ ] ∣ ) .
It remains to bound the divided difference. Since ω ~ [ ζ , ζ ′ ] − 1 \widetilde{\omega}[\zeta,\zeta']-1 ω [ ζ , ζ ′ ] − 1 is the divided difference of ω ~ ( ζ ) − ζ \widetilde{\omega}(\zeta)-\zeta ω ( ζ ) − ζ , we employ the same strategy on Γ z ∣ z ′ \Gamma_{z|z'} Γ z ∣ z ′ and Γ z ′ ∣ z \Gamma_{z'|z} Γ z ′ ∣ z from Part I of the proof of Lemma C.1 , with ( z , z ′ ) (z,z') ( z , z ′ ) there replaced by ( ζ , ζ ′ ) (\zeta,\zeta') ( ζ , ζ ′ ) . Each arc lies in the corresponding larger disk from Part I, and dist ( ζ , I n ) ≥ 3 dist ( z , I n ) / 4 \operatorname{dist}(\zeta,I_n)\ge3\operatorname{dist}(z,I_n)/4 dist ( ζ , I n ) ≥ 3 dist ( z , I n ) /4 , with the same comparison for ζ ′ \zeta' ζ ′ . Inserting the bound from Part I on each arc gives
∣ ω ~ [ ζ , ζ ′ ] − 1 ∣ = O ( Δ Γ Tr Σ ⋅ { 1 dist ( z , I n ) + 1 dist ( z ′ , I n ) } ) = O ( Δ Γ ⋅ { dist ( Γ , I n ) + Δ Γ } ) . |\widetilde{\omega}[\zeta,\zeta']-1|=O\left(\Delta_\Gamma\operatorname{Tr}\Sigma\cdot\left\{\frac{1}{\operatorname{dist}(z,I_n)}+\frac{1}{\operatorname{dist}(z',I_n)}\right\}\right)=O\left(\Delta_\Gamma\cdot\left\{\frac{\operatorname{dist}(\Gamma,I_n)}{}+\Delta_\Gamma\right\}\right). ∣ ω [ ζ , ζ ′ ] − 1∣ = O ( Δ Γ Tr Σ ⋅ { dist ( z , I n ) 1 + dist ( z ′ , I n ) 1 } ) = O ( Δ Γ ⋅ { dist ( Γ , I n ) + Δ Γ } ) .
Finally, ( ∣ z ∣ ∨ n η n ) ( ∣ z ′ ∣ ∨ n η n ) ≥ dist ( Γ , I n ) 2 / 4 (|z|\vee n\eta_n)(|z'|\vee n\eta_n)\ge\operatorname{dist}(\Gamma,I_n)^2/4 ( ∣ z ∣ ∨ n η n ) ( ∣ z ′ ∣ ∨ n η n ) ≥ dist ( Γ , I n ) 2 /4 . Thus multiplying the mixed-trace bound by n 5 / 2 / ( ( ∣ z ∣ ∨ n η n ) ( ∣ z ′ ∣ ∨ n η n ) ) n^{5/2}/((|z|\vee n\eta_n)(|z'|\vee n\eta_n)) n 5/2 / (( ∣ z ∣ ∨ n η n ) ( ∣ z ′ ∣ ∨ n η n )) yields at most n K ∥ θ ∥ 2 n^K\|\theta\|^2 n K ∥ θ ∥ 2 by the same polynomial bounds as in Part I. We prove Eq. (34b) and thus the lemma.
Proof of Proposition 3
We shall begin with the Stieltjes representation in Eq. (20) , with associated probability measure μ ω , j \mu_{\omega,j} μ ω , j and weighted measure μ ω \mu_\omega μ ω . After establishing the formulas in Eq. (17) from Eq. (15) in Parts I and II, we come back to verify that s ω , j s_{\omega,j} s ω , j are indeed Stieltjes transforms of probability measures and that μ ω \mu_\omega μ ω is finite.
Part I: Proof of (21) . Starting from Eq. (15a), we have
V n [ ϕ ] = τ 2 ∬ Γ 2 ( ω ~ [ z , z ′ ] − 1 ) α ϕ ( d z ) α ϕ ( d z ′ ) . V_n[\phi]=\tau^2\iint_{\Gamma^2}\left(\widetilde{\omega}[z,z']-1\right)\alpha_\phi(\mathrm{d}z)\alpha_\phi(\mathrm{d}z'). V n [ ϕ ] = τ 2 ∬ Γ 2 ( ω [ z , z ′ ] − 1 ) α ϕ ( d z ) α ϕ ( d z ′ ) .
Next, we use the fixed-point equation for ω \omega ω and Eq. (20) at z / n z/n z / n to write
ω ~ ( z ) − z + Tr Σ = ∑ j σ j ( 1 + z n s ω , j ( z / n ) ) = ∑ j σ j ∫ n x n x − z μ ω , j ( d x ) = n 2 ∫ μ ω ( d x ) n x − z . \widetilde{\omega}(z)-z+\operatorname{Tr}\Sigma=\sum_j\sigma_j\left(1+\frac{z}{n}s_{\omega,j}(z/n)\right)=\sum_j\sigma_j\int\frac{nx}{nx-z}\,\mu_{\omega,j}(\mathrm{d}x)=n^2\int\frac{\mu_\omega(\mathrm{d}x)}{nx-z}. ω ( z ) − z + Tr Σ = j ∑ σ j ( 1 + n z s ω , j ( z / n ) ) = j ∑ σ j ∫ n x − z n x μ ω , j ( d x ) = n 2 ∫ n x − z μ ω ( d x ) .
Taking the difference of the above identity between z z z and z ′ z' z ′ and dividing by z − z ′ z-z' z − z ′ , we have
ω ~ [ z , z ′ ] − 1 = ( ω ~ ( z ) − z ) − ( ω ~ ( z ′ ) − z ′ ) z − z ′ = n 2 ∫ μ ω ( d x ) ( n x − z ) ( n x − z ′ ) . (47) \widetilde{\omega}[z,z']-1=\frac{\left(\widetilde{\omega}(z)-z\right)-\left(\widetilde{\omega}(z')-z'\right)}{z-z'}=n^2\int\frac{\mu_\omega(\mathrm{d}x)}{(nx-z)(nx-z')}. \tag*{(47)} ω [ z , z ′ ] − 1 = z − z ′ ( ω ( z ) − z ) − ( ω ( z ′ ) − z ′ ) = n 2 ∫ ( n x − z ) ( n x − z ′ ) μ ω ( d x ) . (47)
As we will show later in Part III, supp μ ω ⊆ [ η n , ∞ ) \operatorname{supp}\mu_\omega\subseteq[\eta_n,\infty) supp μ ω ⊆ [ η n , ∞ ) , so we have n x ∈ I n nx\in I_n n x ∈ I n on its support. Thus, by substituting into the variance and applying Eq. (17) in z z z and z ′ z' z ′ , we recover Eq. (17a):
V n [ ϕ ] = τ 2 ∫ ( ∫ Γ n α ϕ ( d z ) n x − z ) ( ∫ Γ n α ϕ ( d z ′ ) n x − z ′ ) μ ω ( d x ) = τ 2 ∫ ϕ ( x ) 2 μ ω ( d x ) . V_n[\phi]=\tau^2\int\left(\int_\Gamma\frac{n\alpha_\phi(\mathrm{d}z)}{nx-z}\right)\left(\int_\Gamma\frac{n\alpha_\phi(\mathrm{d}z')}{nx-z'}\right)\mu_\omega(\mathrm{d}x)=\tau^2\int\phi(x)^2\,\mu_\omega(\mathrm{d}x). V n [ ϕ ] = τ 2 ∫ ( ∫ Γ n x − z n α ϕ ( d z ) ) ( ∫ Γ n x − z ′ n α ϕ ( d z ′ ) ) μ ω ( d x ) = τ 2 ∫ ϕ ( x ) 2 μ ω ( d x ) .
Part II: Proof of (22) . By Eq. (15b) and writing the two population resolvents in the eigenbasis of Σ \Sigma Σ , we have
B n [ ϕ ] = ∑ j σ j ∣ ⟨ u j , β ⟩ ∣ 2 ∬ Γ 2 ω ~ [ z , z ′ ] α ϕ ( d z ) α ϕ ( d z ′ ) ( 1 − n σ j / ω ~ ( z ) ) ( 1 − n σ j / ω ~ ( z ′ ) ) . B_n[\phi]=\sum_j\sigma_j\lvert\langle u_j,\beta\rangle\rvert^2\iint_{\Gamma^2}\frac{\widetilde{\omega}[z,z']\,\alpha_\phi(\mathrm{d}z)\alpha_\phi(\mathrm{d}z')}{\left(1-n\sigma_j/\widetilde{\omega}(z)\right)\left(1-n\sigma_j/\widetilde{\omega}(z')\right)}. B n [ ϕ ] = j ∑ σ j ∣⟨ u j , β ⟩ ∣ 2 ∬ Γ 2 ( 1 − n σ j / ω ( z ) ) ( 1 − n σ j / ω ( z ′ ) ) ω [ z , z ′ ] α ϕ ( d z ) α ϕ ( d z ′ ) .
To identify the two terms in Eq. (17b), we apply Eq. (47) to ω ~ [ z , z ′ ] = 1 + ( ω ~ [ z , z ′ ] − 1 ) \widetilde{\omega}[z,z']=1+\left(\widetilde{\omega}[z,z']-1\right) ω [ z , z ′ ] = 1 + ( ω [ z , z ′ ] − 1 ) . Therefore it follows that
B n [ ϕ ] = ∑ j σ j ∣ ⟨ u j , β ⟩ ∣ 2 { ( ∫ Γ α ϕ ( d z ) 1 − n σ j / ω ~ ( z ) ) 2 + ∫ ( ∫ Γ n n x ′ − z α ϕ ( d z ) 1 − n σ j / ω ~ ( z ) ) 2 μ ω ( d x ′ ) } . B_n[\phi]=\sum_j\sigma_j\lvert\langle u_j,\beta\rangle\rvert^2\left\{\left(\int_\Gamma\frac{\alpha_\phi(\mathrm{d}z)}{1-n\sigma_j/\widetilde{\omega}(z)}\right)^2+\int\left(\int_\Gamma\frac{n}{nx'-z}\frac{\alpha_\phi(\mathrm{d}z)}{1-n\sigma_j/\widetilde{\omega}(z)}\right)^2\mu_\omega(\mathrm{d}x')\right\}. B n [ ϕ ] = j ∑ σ j ∣⟨ u j , β ⟩ ∣ 2 { ( ∫ Γ 1 − n σ j / ω ( z ) α ϕ ( d z ) ) 2 + ∫ ( ∫ Γ n x ′ − z n 1 − n σ j / ω ( z ) α ϕ ( d z ) ) 2 μ ω ( d x ′ ) } .
Indeed, the z z z and z ′ z' z ′ integrals coincide in each summand, with x ′ x' x ′ fixed in the second. By Eq. (20) at z / n z/n z / n , the common resolvent factor is
( 1 − n σ j ω ~ ( z ) ) − 1 = − z n s ω , j ( z / n ) = ∫ z z − n x μ ω , j ( d x ) . \left(1-\frac{n\sigma_j}{\widetilde{\omega}(z)}\right)^{-1}=-\frac{z}{n}s_{\omega,j}(z/n)=\int\frac{z}{z-nx}\,\mu_{\omega,j}(\mathrm{d}x). ( 1 − ω ( z ) n σ j ) − 1 = − n z s ω , j ( z / n ) = ∫ z − n x z μ ω , j ( d x ) .
Substituting this representation and applying Eq. (17) , we have
( I ) = ∫ [ ∫ Γ z z − n x α ϕ ( d z ) ] μ ω , j ( d x ) = ∫ ψ ( x ) μ ω , j ( d x ) , (\mathrm{I})=\int\left[\int_\Gamma\frac{z}{z-nx}\,\alpha_\phi(\mathrm{d}z)\right]\mu_{\omega,j}(\mathrm{d}x)=\int\psi(x)\,\mu_{\omega,j}(\mathrm{d}x), ( I ) = ∫ [ ∫ Γ z − n x z α ϕ ( d z ) ] μ ω , j ( d x ) = ∫ ψ ( x ) μ ω , j ( d x ) ,
( I I ) = ∫ [ 1 x ′ − x ∫ Γ ( z z − n x − z z − n x ′ ) α ϕ ( d z ) ] μ ω , j ( d x ) = ∫ ψ ( x ) − ψ ( x ′ ) x ′ − x μ ω , j ( d x ) . \mathrm{(II)}=\int\left[\frac{1}{x'-x}\int_{\Gamma}\left(\frac{z}{z-nx}-\frac{z}{z-nx'}\right)\alpha\phi(dz)\right]\mu_{\omega,j}(dx)=\int\frac{\psi(x)-\psi(x')}{x'-x}\mu_{\omega,j}(dx). ( II ) = ∫ [ x ′ − x 1 ∫ Γ ( z − n x z − z − n x ′ z ) α ϕ ( d z ) ] μ ω , j ( d x ) = ∫ x ′ − x ψ ( x ) − ψ ( x ′ ) μ ω , j ( d x ) .
In the second row we use partial fractions for x ≠ x ′ x\ne x' x = x ′ and the continuous extension − ψ ′ ( x ′ ) -\psi'(x') − ψ ′ ( x ′ ) at x = x ′ > 0 x=x'>0 x = x ′ > 0 . Substituting (I) and (II) into the expansion of B n [ ϕ ] B_n[\phi] B n [ ϕ ] proves Eq. (17b).
We implicitly applied Fubini’s theorem in Parts I and II. We can apply because the contour bounds in Theorem 2 make both kernels uniformly bounded for fixed n n n , the measures have finite total variation, and ∑ j σ j ∣ ⟨ u j , β ⟩ ∣ 2 < ∞ \sum_j\sigma_j|\langle u_j,\beta\rangle|^2<\infty ∑ j σ j ∣ ⟨ u j , β ⟩ ∣ 2 < ∞ .
Part III: Probability measures in Eq. (20) . Lastly, we check μ ω , j \mu_{\omega,j} μ ω , j are indeed probability measures and the representation is valid. From Eq. (20) , s ω , j ( z ) − 1 = − z ( 1 + n σ j s n ( n z ) ) s_{\omega,j}(z)^{-1}=-z(1+n\sigma_js_n(nz)) s ω , j ( z ) − 1 = − z ( 1 + n σ j s n ( n z )) for σ j > 0 \sigma_j>0 σ j > 0 . Using the Stieltjes representation of s n s_n s n , we obtain that
Im ( z ( 1 + n σ j s n ( n z ) ) ) = Im z ( 1 + n σ j ∫ x ∣ x − n z ∣ 2 μ n ( d x ) ) > 0 , z ∈ C + . \operatorname{Im}\left(z(1+n\sigma_js_n(nz))\right)=\operatorname{Im}z\left(1+n\sigma_j\int\frac{x}{|x-nz|^2}\mu_n(dx)\right)>0,\qquad z\in\mathbb{C}_+. Im ( z ( 1 + n σ j s n ( n z )) ) = Im z ( 1 + n σ j ∫ ∣ x − n z ∣ 2 x μ n ( d x ) ) > 0 , z ∈ C + .
( − i y ) s ω , j ( i y ) = 1 1 + n σ j s n ( n i y ) ⟶ 1 , y → ∞ , (-iy)s_{\omega,j}(iy)=\frac{1}{1+n\sigma_js_n(niy)}\longrightarrow1,\qquad y\to\infty, ( − i y ) s ω , j ( i y ) = 1 + n σ j s n ( ni y ) 1 ⟶ 1 , y → ∞ ,
where the second equation makes use of ∣ s n ( n i y ) ∣ ≤ 1 / ( n y ) |s_n(niy)|\le1/(ny) ∣ s n ( ni y ) ∣ ≤ 1/ ( n y ) . By the first equation, z ( 1 + n σ j s n ( n z ) ) z(1+n\sigma_js_n(nz)) z ( 1 + n σ j s n ( n z )) never vanishes on C + \mathbb{C}_+ C + . Its negative reciprocal s ω , j s_{\omega,j} s ω , j is therefore analytic there, with
Im s ω , j ( z ) = Im ( z ( 1 + n σ j s n ( n z ) ) ) ∣ z ( 1 + n σ j s n ( n z ) ) ∣ 2 > 0. \operatorname{Im}s_{\omega,j}(z)=\frac{\operatorname{Im}\left(z(1+n\sigma_js_n(nz))\right)}{|z(1+n\sigma_js_n(nz))|^2}>0. Im s ω , j ( z ) = ∣ z ( 1 + n σ j s n ( n z )) ∣ 2 Im ( z ( 1 + n σ j s n ( n z )) ) > 0.
Thus s ω , j s_{\omega,j} s ω , j is Herglotz-Nevanlinna, and the second equation gives y Im s ω , j ( i y ) → 1 y\operatorname{Im}s_{\omega,j}(iy)\to1 y Im s ω , j ( i y ) → 1 and s ω , j ( i y ) → 0 s_{\omega,j}(iy)\to0 s ω , j ( i y ) → 0 as y → ∞ y\to\infty y → ∞ . Applying Lemma A.3 with these limits, as in the proof of Lemma A.4 , yields a unique positive measure μ ω , j \mu_{\omega,j} μ ω , j satisfying
s ω , j ( z ) = ∫ R μ ω , j ( d x ) x − z and μ ω , j ( R ) = lim y → ∞ y Im s ω , j ( i y ) = 1. s_{\omega,j}(z)=\int_{\mathbb{R}}\frac{\mu_{\omega,j}(dx)}{x-z}
\qquad\text{and}\qquad
\mu_{\omega,j}(\mathbb{R})=\lim_{y\to\infty}y\operatorname{Im}s_{\omega,j}(iy)=1. s ω , j ( z ) = ∫ R x − z μ ω , j ( d x ) and μ ω , j ( R ) = y → ∞ lim y Im s ω , j ( i y ) = 1.
By Assumption 3, s n ( n z ) > 0 s_n(nz)>0 s n ( n z ) > 0 for real z < η n z<\eta_n z < η n , so s ω , j s_{\omega,j} s ω , j is real analytic there except possibly at zero. Stieltjes inversion gives supp μ ω , j ⊆ { 0 } ∪ [ η n , ∞ ) \operatorname{supp}\mu_{\omega,j}\subseteq\{0\}\cup[\eta_n,\infty) supp μ ω , j ⊆ { 0 } ∪ [ η n , ∞ ) . Applying monotone convergence to λ x / ( x + λ ) → x \lambda x/(x+\lambda)\to x λ x / ( x + λ ) → x and using Eq. (20) at − λ -\lambda − λ implies
∫ x μ ω , j ( d x ) = lim λ → ∞ λ ( 1 − λ s ω , j ( − λ ) ) = σ j , \int x\,\mu_{\omega,j}(dx)=\lim_{\lambda\to\infty}\lambda(1-\lambda s_{\omega,j}(-\lambda))=\sigma_j, ∫ x μ ω , j ( d x ) = λ → ∞ lim λ ( 1 − λ s ω , j ( − λ )) = σ j ,
since n λ s n ( − n λ ) → 1 n\lambda s_n(-n\lambda)\to1 nλ s n ( − nλ ) → 1 . Thus μ ω ( R ) = n − 1 ∑ j σ j 2 = Tr ( Σ 2 ) / n < ∞ \mu_\omega(\mathbb{R})=n^{-1}\sum_j\sigma_j^2=\operatorname{Tr}(\Sigma^2)/n<\infty μ ω ( R ) = n − 1 ∑ j σ j 2 = Tr ( Σ 2 ) / n < ∞ , and the factor x x x in its definition removes any mass at zero, concluding the proof with supp μ ω ⊆ [ η n , ∞ ) \operatorname{supp}\mu_\omega\subseteq[\eta_n,\infty) supp μ ω ⊆ [ η n , ∞ ) .
Proofs for Applications
Throughout the application proofs, we leave dependence on the decay exponent α \alpha α or β \beta β implicit for simplicity.
We first introduce the lemma that provides the hard edge guarantee on the event { σ ^ n ≥ η n } \{\hat{\sigma}_n\ge\eta_n\} { σ ^ n ≥ η n } , which serves as the common input for ridge-type filters and gradient flow. The lemma also gives quantitative bounds that helps us check assumptions and conditions to apply Theorem 2. We shall take η n = Θ C x ( τ 1 ( n ) / n ) \eta_n=\Theta_{C_x}(\tau_1(n)/n) η n = Θ C x ( τ 1 ( n ) / n ) next for both propositions. See its proof in Appendix § 12 .§ 3 .
Lemma D.1. Under Assumption 1, take either spectrum in Eq. (5) . We can take η n \eta_n η n above such that for all n = Ω C x ( 1 ) n=\Omega_{C_x}(1) n = Ω C x ( 1 ) , it holds P ( σ ^ n ≥ η n ) ≥ 1 − 2 e − n \mathbb{P}(\hat{\sigma}_n\ge\eta_n)\ge1-2e^{-n} P ( σ ^ n ≥ η n ) ≥ 1 − 2 e − n and supp μ n ⊆ [ n η n , ∞ ) \operatorname{supp}\mu_n\subseteq[n\eta_n,\infty) supp μ n ⊆ [ n η n , ∞ ) , and the following consequences hold under both laws
Λ n ( − n η n / 4 ) = O C x ( ( log n ) 2 ) and − log ( 1 ∧ n η n ) = O C x ( log n ) . \Lambda_n(-n\eta_n/4)=O_{C_x}\left((\log n)^2\right)
\qquad\text{and}\qquad
-\log(1\wedge n\eta_n)=O_{C_x}(\log n). Λ n ( − n η n /4 ) = O C x ( ( log n ) 2 ) and − log ( 1 ∧ n η n ) = O C x ( log n ) .
Proof of Proposition 4
Part I: Bounding approximation errors in (i). In the proof, we write λ + ≔ λ ∨ 0 \lambda_{+} \coloneqq\lambda\vee0 λ + : = λ ∨ 0 . We will first apply Theorem 2 to obtain the error control before computing their explicit formulas and specializing to k = 1 k=1 k = 1 in the next part. To this end, Lemma D.1 supplies the event probability and the inclusion supp μ n ⊆ I n \operatorname{supp}\mu_n \subseteq I_n supp μ n ⊆ I n . It remains to check the contour conditions and bound the two contour integrals. We use the two-circle construction (cf. Figure 5 for illustration) with the explicit choice
Ω = { ∣ z ∣ < n η n / 4 } ∪ { ∣ z + n λ + ∣ > n ( λ + + 3 η n / 4 ) } ≔ D i n ∪ D o u t . \Omega= \{ |z| < n\eta_n/4 \} \cup\{ |z+n\lambda_{+}| > n(\lambda_{+}+3\eta_n/4) \} \coloneqq D_{\mathrm{in}} \cup D_{\mathrm{out}}. Ω = { ∣ z ∣ < n η n /4 } ∪ { ∣ z + n λ + ∣ > n ( λ + + 3 η n /4 )} : = D in ∪ D out .
We point out that the orientation convention is that boundaries ∂ D i n \partial D_{\mathrm{in}} ∂ D in and ∂ D o u t \partial D_{\mathrm{out}} ∂ D out are counterclockwise and clockwise, respectively. Define the residual ψ k , λ ( z / n ) \psi_{k,\lambda}(z/n) ψ k , λ ( z / n ) to equal 1 1 1 in D i n D_{\mathrm{in}} D in and ( λ / ( z / n + λ ) ) k (\lambda/(z/n+\lambda))^k ( λ / ( z / n + λ ) ) k on D o u t D_{\mathrm{out}} D out . The pole − n λ -n\lambda − nλ lies strictly inside the excluded outer disk. Thus the corresponding filter is holomorphic on both components, agrees with z ↦ ϕ k , λ ( z / n ) z \mapsto\phi_{k,\lambda}(z/n) z ↦ ϕ k , λ ( z / n ) on I n I_n I n , and has residual tending to zero at infinity, verifying Assumption 3. Both circles lie in Re z ≤ 3 n η n / 4 \operatorname{Re} z \le3n\eta_n/4 Re z ≤ 3 n η n /4 . Consequently,
dist ( Γ , I n ) = n η n / 4 and sup z ∈ Γ ∣ z ∣ ∨ n η n dist ( z , I n ) ≤ 5. \operatorname{dist}(\Gamma,I_n)=n\eta_n/4
\qquad\text{and}\qquad
\sup_{z\in\Gamma}\frac{|z|\vee n\eta_n}{\operatorname{dist}(z,I_n)}\le5. dist ( Γ , I n ) = n η n /4 and z ∈ Γ sup dist ( z , I n ) ∣ z ∣ ∨ n η n ≤ 5.
Indeed, dist ( z , I n ) = ∣ z − n η n ∣ \operatorname{dist}(z,I_n)=|z-n\eta_n| dist ( z , I n ) = ∣ z − n η n ∣ on Γ \Gamma Γ and ∣ z ∣ ≤ ∣ z − n η n ∣ + n η n |z|\le|z-n\eta_n|+n\eta_n ∣ z ∣ ≤ ∣ z − n η n ∣ + n η n . Applying Lemma D.1 yields
Δ Γ ≤ Λ n ( − n η n / 4 ) ∨ 5 = O C x ( ( log n ) 2 ) and − log ( 1 ∧ dist ( Γ , I n ) ) = O C x ( log n ) . \Delta_\Gamma\le\Lambda_n(-n\eta_n/4)\vee5 = O_{C_x}((\log n)^2)
\qquad\text{and}\qquad
-\log(1\wedge\operatorname{dist}(\Gamma,I_n))=O_{C_x}(\log n). Δ Γ ≤ Λ n ( − n η n /4 ) ∨ 5 = O C x (( log n ) 2 ) and − log ( 1 ∧ dist ( Γ , I n )) = O C x ( log n ) .
Thus the two numerical conditions of Theorem 2 hold for all sufficiently large n n n . It remains to evaluate the three quantities entering the error bounds of Theorem 2:
∥ α ϕ k , λ ∥ T V , ∫ Γ ∣ z ∣ ∨ n η n dist ( z , I n ) ∣ α ϕ k , λ ∣ ( d z ) , ∫ Γ ( ∣ z ∣ ∨ n η n ) ∣ α ϕ k , λ ∣ ( d z ) . \|\alpha_{\phi_{k,\lambda}}\|_{\mathrm{TV}},\qquad
\int_\Gamma\frac{|z|\vee n\eta_n}{\operatorname{dist}(z,I_n)}|\alpha_{\phi_{k,\lambda}}|(dz),\qquad
\int_\Gamma(|z|\vee n\eta_n)|\alpha_{\phi_{k,\lambda}}|(dz). ∥ α ϕ k , λ ∥ TV , ∫ Γ dist ( z , I n ) ∣ z ∣ ∨ n η n ∣ α ϕ k , λ ∣ ( d z ) , ∫ Γ ( ∣ z ∣ ∨ n η n ) ∣ α ϕ k , λ ∣ ( d z ) .
First, we control the term ∥ α ϕ k , λ ∥ T V \|\alpha_{\phi_{k,\lambda}}\|_{\mathrm{TV}} ∥ α ϕ k , λ ∥ TV . On ∂ D i n \partial D_{\mathrm{in}} ∂ D in , we have ∣ α ϕ k , λ ∣ ( d z ) = ∣ d z ∣ / ( 2 π ∣ z ∣ ) |\alpha_{\phi_{k,\lambda}}|(dz)=|dz|/(2\pi|z|) ∣ α ϕ k , λ ∣ ( d z ) = ∣ d z ∣/ ( 2 π ∣ z ∣ ) , so this circle contributes exactly 1. On ∂ D o u t \partial D_{\mathrm{out}} ∂ D out , by a change of variable z / n = ( λ + + 3 η n / 4 ) e i θ − λ + z/n=(\lambda_{+}+3\eta_n/4)e^{i\theta}-\lambda_{+} z / n = ( λ + + 3 η n /4 ) e i θ − λ + , if − η n / 2 ≤ λ ≤ 0 -\eta_n/2\le\lambda\le0 − η n /2 ≤ λ ≤ 0 , then ∣ ( 3 η n / 4 ) e i θ + λ ∣ ≥ 3 η n / 4 − ∣ λ ∣ ≥ η n / 4 |(3\eta_n/4)e^{i\theta}+\lambda|\ge3\eta_n/4-|\lambda|\ge\eta_n/4 ∣ ( 3 η n /4 ) e i θ + λ ∣ ≥ 3 η n /4 − ∣ λ ∣ ≥ η n /4 , so
∣ α ϕ k , λ ∣ ( d z ) = ∣ λ ( 3 η n / 4 ) e i θ + λ ∣ k d θ 2 π ≤ 2 k d θ 2 π . |\alpha_{\phi_{k,\lambda}}|(dz)
=
\left|\frac{\lambda}{(3\eta_n/4)e^{i\theta}+\lambda}\right|^k\frac{d\theta}{2\pi}
\le2^k\frac{d\theta}{2\pi}. ∣ α ϕ k , λ ∣ ( d z ) = ( 3 η n /4 ) e i θ + λ λ k 2 π d θ ≤ 2 k 2 π d θ .
If λ > 0 \lambda>0 λ > 0 , the density on ∂ D o u t \partial D_{\mathrm{out}} ∂ D out is instead
∣ α ϕ k , λ ∣ ( d z ) = ( λ λ + 3 η n / 4 ) k λ + 3 η n / 4 ∣ ( λ + 3 η n / 4 ) e i θ − λ ∣ d θ 2 π . |\alpha_{\phi_{k,\lambda}}|(dz)
=
\left(\frac{\lambda}{\lambda+3\eta_n/4}\right)^k
\frac{\lambda+3\eta_n/4}{|(\lambda+3\eta_n/4)e^{i\theta}-\lambda|}
\frac{d\theta}{2\pi}. ∣ α ϕ k , λ ∣ ( d z ) = ( λ + 3 η n /4 λ ) k ∣ ( λ + 3 η n /4 ) e i θ − λ ∣ λ + 3 η n /4 2 π d θ .
Using ∣ ( λ + 3 η n / 4 ) e i θ − λ ∣ 2 = ( 3 η n / 4 ) 2 + 4 λ ( λ + 3 η n / 4 ) sin 2 ( θ / 2 ) |(\lambda+3\eta_n/4)e^{i\theta}-\lambda|^2=(3\eta_n/4)^2+4\lambda(\lambda+3\eta_n/4)\sin^2(\theta/2) ∣ ( λ + 3 η n /4 ) e i θ − λ ∣ 2 = ( 3 η n /4 ) 2 + 4 λ ( λ + 3 η n /4 ) sin 2 ( θ /2 ) and sin ( θ / 2 ) ≥ θ / π \sin(\theta/2)\ge\theta/\pi sin ( θ /2 ) ≥ θ / π on [ 0 , π ] [0,\pi] [ 0 , π ] , we obtain
∫ ∂ D o u t ∣ α ϕ k , λ ∣ ( d z ) ≤ λ π ∫ 0 π d θ ( 3 η n / 4 ) 2 + 4 λ 2 θ 2 / π 2 ≤ 1 2 log ( 1 + 16 λ 3 η n ) , \int_{\partial D_{\mathrm{out}}}|\alpha_{\phi_{k,\lambda}}|(dz)
\le\frac{\lambda}{\pi}\int_0^\pi
\frac{d\theta}{\sqrt{(3\eta_n/4)^2+4\lambda^2\theta^2/\pi^2}}
\le\frac{1}{2}\log\left(1+\frac{16\lambda}{3\eta_n}\right), ∫ ∂ D out ∣ α ϕ k , λ ∣ ( d z ) ≤ π λ ∫ 0 π ( 3 η n /4 ) 2 + 4 λ 2 θ 2 / π 2 d θ ≤ 2 1 log ( 1 + 3 η n 16 λ ) ,
where we use in (i) that ( λ / ( λ + 3 η n / 4 ) ) k ⋅ ( λ + 3 η n / 4 ) ≤ λ (\lambda/(\lambda+3\eta_n/4))^k\cdot(\lambda+3\eta_n/4)\le\lambda ( λ / ( λ + 3 η n /4 ) ) k ⋅ ( λ + 3 η n /4 ) ≤ λ . Combining the two cases, and adding the contribution from ∂ D i n \partial D_{\mathrm{in}} ∂ D in , we have
∥ α ϕ k , λ ∥ T V = O k ( 1 + log ( 1 + λ + / η n ) ) = O C x , k ( log n ) . (48) \|\alpha_{\phi_{k,\lambda}}\|_{\mathrm{TV}}
=O_k(1+\log(1+\lambda_{+}/\eta_n))
=O_{C_x,k}(\log n). \tag*{(48)} ∥ α ϕ k , λ ∥ TV = O k ( 1 + log ( 1 + λ + / η n )) = O C x , k ( log n ) . (48)
where the last step uses λ = O ( 1 ) \lambda= O(1) λ = O ( 1 ) and − log η n = O C x ( log n ) -\log\eta_n = O_{C_x}(\log n) − log η n = O C x ( log n ) . The second quantity then follows immediately, applying the geometric bound already proved on Γ \Gamma Γ ,
∫ Γ ∣ z ∣ ∨ n η n dist ( z , I n ) ∣ α ϕ k , λ ∣ ( d z ) ≤ 5 ∥ α ϕ k , λ ∥ T V = O C x , k ( log n ) . (49) \int_{\Gamma} \frac{|z| \vee n\eta_n}{\operatorname{dist}(z,I_n)} |\alpha_{\phi_k,\lambda}|(dz) \le5\|\alpha_{\phi_k,\lambda}\|_{\mathrm{TV}} = O_{C_x,k}(\log n). \tag*{(49)} ∫ Γ dist ( z , I n ) ∣ z ∣ ∨ n η n ∣ α ϕ k , λ ∣ ( d z ) ≤ 5∥ α ϕ k , λ ∥ TV = O C x , k ( log n ) . (49)
Thirdly, we control the integral weighted by ∣ z ∣ ∨ n η n |z| \vee n\eta_n ∣ z ∣ ∨ n η n . On ∂ D i n \partial D_{\mathrm{in}} ∂ D in , this weight equals n η n n\eta_n n η n . On ∂ D o u t \partial D_{\mathrm{out}} ∂ D out , the preceding density bounds imply
∫ ∂ D o u t ∣ z ∣ ∣ α ϕ k , λ ∣ ( d z ) ≤ { 2 k ⋅ 3 n η n / 4 , − η n / 2 ≤ λ ≤ 0 , n ( λ + 3 η n / 4 ) ( λ λ + 3 η n / 4 ) k , λ > 0 , \int_{\partial D_{\mathrm{out}}} |z| |\alpha_{\phi_k,\lambda}|(dz) \le
\begin{cases}
2^k \cdot3n\eta_n/4, & -\eta_n/2 \le\lambda\le0, \\
n(\lambda+ 3\eta_n/4)\left(\frac{\lambda}{\lambda+ 3\eta_n/4}\right)^k, & \lambda> 0,
\end{cases} ∫ ∂ D out ∣ z ∣∣ α ϕ k , λ ∣ ( d z ) ≤ ⎩ ⎨ ⎧ 2 k ⋅ 3 n η n /4 , n ( λ + 3 η n /4 ) ( λ + 3 η n /4 λ ) k , − η n /2 ≤ λ ≤ 0 , λ > 0 ,
which are both bounded by O k ( n ( λ + η n ) ) O_k(n(\lambda+ \eta_n)) O k ( n ( λ + η n )) . Combining with the inner contribution and using ∣ z ∣ ∨ n η n ≤ ∣ z ∣ + n η n |z| \vee n\eta_n \le|z| + n\eta_n ∣ z ∣ ∨ n η n ≤ ∣ z ∣ + n η n on the outer circle, we conclude that
∫ Γ ( ∣ z ∣ ∨ n η n ) ∣ α ϕ k , λ ∣ ( d z ) ≤ n η n ∥ α ϕ k , λ ∥ T V + ∫ ∂ D o u t ∣ z ∣ ∣ α ϕ k , λ ∣ ( d z ) = O k ( n ( λ + η n ) + n η n log ( 1 + λ + / η n ) ) = O k ( n ( λ + η n ) ) , (50) \begin{aligned}
\int_{\Gamma} (|z| \vee n\eta_n)|\alpha_{\phi_k,\lambda}|(dz)
&\le n\eta_n\|\alpha_{\phi_k,\lambda}\|_{\mathrm{TV}} + \int_{\partial D_{\mathrm{out}}} |z||\alpha_{\phi_k,\lambda}|(dz) \\
&= O_k\left(n(\lambda+ \eta_n) + n\eta_n\log(1+\lambda_+/\eta_n)\right)
= O_k(n(\lambda+ \eta_n)), \tag*{(50)}
\end{aligned} ∫ Γ ( ∣ z ∣ ∨ n η n ) ∣ α ϕ k , λ ∣ ( d z ) ≤ n η n ∥ α ϕ k , λ ∥ TV + ∫ ∂ D out ∣ z ∣∣ α ϕ k , λ ∣ ( d z ) = O k ( n ( λ + η n ) + n η n log ( 1 + λ + / η n ) ) = O k ( n ( λ + η n )) , (50)
where the last step uses log ( 1 + x ) ≤ x \log(1+x) \le x log ( 1 + x ) ≤ x and λ + ≤ λ + η n \lambda_+ \le\lambda+ \eta_n λ + ≤ λ + η n . Substituting Eqs. (48) , (49) and (50) into Theorem 2 there yields
∥ 1 { σ ^ n ≥ η n } ( V X [ ϕ k , λ ] − V n [ ϕ k , λ ] ) ∥ L p = O C x , κ , k ( τ 2 p 2 Δ Γ 23 ( log n ) 2 n ) . \left\|\mathbb{1}\{\widehat{\sigma}_n \ge\eta_n\}\left(\mathcal{V}_X[\phi_k,\lambda]-\mathcal{V}_n[\phi_k,\lambda]\right)\right\|_{L^p}
= O_{C_x,\kappa,k}\left(\frac{\tau^2p^2\Delta_\Gamma^{23}(\log n)^2}{n}\right). ∥ 1 { σ n ≥ η n } ( V X [ ϕ k , λ ] − V n [ ϕ k , λ ] ) ∥ L p = O C x , κ , k ( n τ 2 p 2 Δ Γ 23 ( log n ) 2 ) .
∥ 1 { σ ^ n ≥ η n } ( B X [ ϕ k , λ ] − B n [ ϕ k , λ ] ) ∥ L p = O C x , κ , k ( p 3 / 2 Δ Γ 26 ( λ + η n ) 2 ∥ θ ∥ 2 n ) . \left\|\mathbb{1}\{\widehat{\sigma}_n \ge\eta_n\}\left(\mathcal{B}_X[\phi_k,\lambda]-\mathcal{B}_n[\phi_k,\lambda]\right)\right\|_{L^p}
= O_{C_x,\kappa,k}\left(\frac{p^{3/2}\Delta_\Gamma^{26}(\lambda+\eta_n)^2\|\theta\|^2}{\sqrt{n}}\right). ∥ 1 { σ n ≥ η n } ( B X [ ϕ k , λ ] − B n [ ϕ k , λ ] ) ∥ L p = O C x , κ , k ( n p 3/2 Δ Γ 26 ( λ + η n ) 2 ∥ θ ∥ 2 ) .
Here Δ Γ = O C x ( ( log n ) 2 ) \Delta_\Gamma= O_{C_x}((\log n)^2) Δ Γ = O C x (( log n ) 2 ) and p ≤ κ log n p \le\kappa\log n p ≤ κ log n contribute only powers of log n \log n log n . This proves the two approximation bounds in (i).
Part II: Evaluating the effective formulas in (i) and (ii). It remains to concretely compute the effective risk R n R_n R n , and the Cauchy’s formula is helpful in this case. We first reduce both contour integrals to the pole at − λ -\lambda − λ . For F F F analytic on a neighborhood of the closed outer disk in units ζ = z / n \zeta= z/n ζ = z / n , deform the inner integral of F ( ζ ) / ζ F(\zeta)/\zeta F ( ζ ) / ζ to its outer boundary. Reversing the exterior contribution gives
∫ Γ F ( z / n ) α ϕ k , λ ( d z ) = − 1 2 π i ∫ ∂ D o u t F ( z / n ) 1 − ( λ / ( z / n + λ ) ) k z d z = ∑ a = 0 k − 1 λ a a ! F ( a ) ( − λ ) . (51) \int_{\Gamma} F(z/n)\alpha_{\phi_k,\lambda}(dz)
= -\frac{1}{2\pi i}\int_{\partial D_{\mathrm{out}}}F(z/n)\frac{1-(\lambda/(z/n+\lambda))^k}{z}\,dz
= \sum_{a=0}^{k-1}\frac{\lambda^a}{a!}F^{(a)}(-\lambda). \tag*{(51)} ∫ Γ F ( z / n ) α ϕ k , λ ( d z ) = − 2 π i 1 ∫ ∂ D out F ( z / n ) z 1 − ( λ / ( z / n + λ ) ) k d z = a = 0 ∑ k − 1 a ! λ a F ( a ) ( − λ ) . (51)
Here ∂ D o u t \partial D_{\mathrm{out}} ∂ D out is clockwise, and after substituting ζ = z / n \zeta= z/n ζ = z / n , the last equality follows by expanding the kernel as ∑ a = 0 k − 1 λ a / ( ζ + λ ) a + 1 \sum_{a=0}^{k-1}\lambda^a/(\zeta+\lambda)^{a+1} ∑ a = 0 k − 1 λ a / ( ζ + λ ) a + 1 and applying Cauchy’s derivative formula. At λ = 0 \lambda= 0 λ = 0 it is simply 1 / ζ 1/\zeta 1/ ζ , giving F ( 0 ) F(0) F ( 0 ) for every k k k . By Eq. (15), for variance and squared bias we first apply Eq. (51) in ζ = z / n \zeta= z/n ζ = z / n , with ζ ′ = z ′ / n \zeta' = z'/n ζ ′ = z ′ / n fixed, taking
F ( ζ ) = { τ 2 ( ω [ ζ , ζ ′ ] − 1 ) , for computing V n , ω [ ζ , ζ ′ ] β ∗ Σ ( I − Σ ω ( ζ ) ) − 1 ( I − Σ ω ( ζ ′ ) ) − 1 β , for computing B n . F(\zeta)=
\begin{cases}
\tau^2\left(\omega[\zeta,\zeta']-1\right), & \text{for computing } \mathcal{V}_n, \\
\omega[\zeta,\zeta']\beta^*\Sigma\left(I-\dfrac{\Sigma}{\omega(\zeta)}\right)^{-1}\left(I-\dfrac{\Sigma}{\omega(\zeta')}\right)^{-1}\beta, & \text{for computing } \mathcal{B}_n.
\end{cases} F ( ζ ) = ⎩ ⎨ ⎧ τ 2 ( ω [ ζ , ζ ′ ] − 1 ) , ω [ ζ , ζ ′ ] β ∗ Σ ( I − ω ( ζ ) Σ ) − 1 ( I − ω ( ζ ′ ) Σ ) − 1 β , for computing V n , for computing B n .
Substituting the second kernel into Eq. (51) in both variables directly recovers the squared-bias formula in (i). For variance, applying the same identity yields
V n [ ϕ k , λ ] = ( i ) τ 2 { ∑ a , b = 0 k − 1 λ a + b ω ( a + b + 1 ) ( − λ ) ( a + b + 1 ) ! − 1 } = τ 2 { ∑ m = 1 2 k − 1 ( k − ∣ k − m ∣ ) λ m − 1 ω ( m ) ( − λ ) m ! − 1 } . V_n[\phi_{k,\lambda}] \mathrel{\overset{\mathrm{(i)}}{=}} \tau^2 \left\{ \sum_{a,b=0}^{k-1} \frac{\lambda^{a+b}\omega^{(a+b+1)}(-\lambda)}{(a+b+1)!} - 1 \right\} = \tau^2 \left\{ \sum_{m=1}^{2k-1} \frac{(k-\lvert k-m\rvert)\lambda^{m-1}\omega^{(m)}(-\lambda)}{m!} - 1 \right\}. V n [ ϕ k , λ ] = ( i ) τ 2 ⎩ ⎨ ⎧ a , b = 0 ∑ k − 1 ( a + b + 1 )! λ a + b ω ( a + b + 1 ) ( − λ ) − 1 ⎭ ⎬ ⎫ = τ 2 { m = 1 ∑ 2 k − 1 m ! ( k − ∣ k − m ∣) λ m − 1 ω ( m ) ( − λ ) − 1 } .
In (i), the Taylor expansion of ω \omega ω at − λ -\lambda − λ gives the coefficient ω ( a + b + 1 ) ( − λ ) / ( a + b + 1 ) ! \omega^{(a+b+1)}(-\lambda)/(a+b+1)! ω ( a + b + 1 ) ( − λ ) / ( a + b + 1 )! of ( ζ + λ ) a ( ζ ′ + λ ) b (\zeta+\lambda)^a(\zeta^{\prime}+\lambda)^b ( ζ + λ ) a ( ζ ′ + λ ) b in ω [ ζ , ζ ′ ] \omega[\zeta,\zeta^{\prime}] ω [ ζ , ζ ′ ] ; the constant − 1 -1 − 1 contributes only when a = b = 0 a=b=0 a = b = 0 . In the last equality, we group the terms with a + b = m − 1 a+b=m-1 a + b = m − 1 , of which there are k − ∣ k − m ∣ k-\lvert k-m\rvert k − ∣ k − m ∣ for 0 ≤ a , b ≤ k − 1 0\le a,b\le k-1 0 ≤ a , b ≤ k − 1 . This proves the variance formula in (i).
We move on to specialization for (ii), using λ ⋆ = − ω ( − λ ) = ( n s n ( − n λ ) ) − 1 > 0 \lambda_{\star}=-\omega(-\lambda)=(n s_n(-n\lambda))^{-1}>0 λ ⋆ = − ω ( − λ ) = ( n s n ( − nλ ) ) − 1 > 0 . Rearranging the fixed-point equation in Eq. (7) and differentiating in λ ⋆ = λ ⋆ ( λ ) \lambda_{\star}=\lambda_{\star}(\lambda) λ ⋆ = λ ⋆ ( λ ) implies
λ = λ ⋆ ⋅ ( 1 − 1 n Tr ( Σ ( Σ + λ ⋆ I ) − 1 ) ) and ω ′ ( − λ ) = 1 1 − J n ( λ ) > 0. \lambda=\lambda_{\star}\cdot\left(1-\frac{1}{n}\operatorname{Tr}\left(\boldsymbol{\Sigma}(\boldsymbol{\Sigma}+\lambda_{\star}I)^{-1}\right)\right)
\qquad\text{and}\qquad
\omega^{\prime}(-\lambda)=\frac{1}{1-J_n(\lambda)}>0. λ = λ ⋆ ⋅ ( 1 − n 1 Tr ( Σ ( Σ + λ ⋆ I ) − 1 ) ) and ω ′ ( − λ ) = 1 − J n ( λ ) 1 > 0.
Substituting k = 1 k=1 k = 1 into the formulas in (i) yields both effective risks in (ii). Finally, we have from β = Σ 1 / 2 θ \beta=\boldsymbol{\Sigma}^{1/2}\theta β = Σ 1/2 θ that
τ 2 V n [ ϕ 1 , λ ] = 1 − J n ( λ ) J n ( λ ) and ( λ + η n ) 2 ∥ θ ∥ 2 B n [ ϕ 1 , λ ] = ( λ + η n ) 2 ( 1 − J n ( λ ) ) ∥ θ ∥ 2 λ ⋆ 2 ∥ Σ ( Σ + λ ⋆ I ) − 1 θ ∥ 2 . \frac{\tau^2}{V_n[\phi_{1,\lambda}]}=\frac{1-J_n(\lambda)}{J_n(\lambda)}
\qquad\text{and}\qquad
\frac{(\lambda+\eta_n)^2\lVert\theta\rVert^2}{B_n[\phi_{1,\lambda}]}=\frac{(\lambda+\eta_n)^2(1-J_n(\lambda))\lVert\theta\rVert^2}{\lambda_{\star}^2\lVert\boldsymbol{\Sigma}(\boldsymbol{\Sigma}+\lambda_{\star}I)^{-1}\theta\rVert^2}. V n [ ϕ 1 , λ ] τ 2 = J n ( λ ) 1 − J n ( λ ) and B n [ ϕ 1 , λ ] ( λ + η n ) 2 ∥ θ ∥ 2 = λ ⋆ 2 ∥ Σ ( Σ + λ ⋆ I ) − 1 θ ∥ 2 ( λ + η n ) 2 ( 1 − J n ( λ )) ∥ θ ∥ 2 .
We close the proof with the multiplicative bounds.
Proof of Proposition 5
Part I: Bounding approximation errors. We repeat largely the same proof in Appendix D.1. Applying Theorem 2 and Lemma D.1 , we compute the contour integrals on the positively oriented boundary Γ = ∂ Ω \Gamma=\partial\Omega Γ = ∂ Ω of Ω = { z : ∣ arg ( z + n / t ) ∣ < π / 4 } \Omega=\{z:\lvert\arg(z+n/t)\rvert<\pi/4\} Ω = { z : ∣ arg ( z + n / t )∣ < π /4 } , as in Figure 3 . The filter ϕ t ( z / n ) \phi_t(z/n) ϕ t ( z / n ) is entire, and its residual e − t z / n e^{-tz/n} e − t z / n decays exponentially along the two rays, verifying Assumption 3. Here
dist ( Γ , I n ) = n ( η n + t − 1 ) 2 and sup z ∈ Γ ∣ z ∣ ∨ n η n dist ( z , I n ) ≤ 3. \operatorname{dist}(\Gamma,I_n)=\frac{n(\eta_n+t^{-1})}{\sqrt{2}}
\qquad\text{and}\qquad
\sup_{z\in\Gamma}\frac{\lvert z\rvert\vee n\eta_n}{\operatorname{dist}(z,I_n)}\le3. dist ( Γ , I n ) = 2 n ( η n + t − 1 ) and z ∈ Γ sup dist ( z , I n ) ∣ z ∣ ∨ n η n ≤ 3.
For the second bound, use ∣ z ∣ ≤ ∣ z − n η n ∣ + n η n ≤ ( 1 + 2 ) dist ( z , I n ) ≤ 3 dist ( z , I n ) \lvert z\rvert\le\lvert z-n\eta_n\rvert+n\eta_n\le(1+\sqrt{2})\operatorname{dist}(z,I_n)\le3\operatorname{dist}(z,I_n) ∣ z ∣ ≤ ∣ z − n η n ∣ + n η n ≤ ( 1 + 2 ) dist ( z , I n ) ≤ 3 dist ( z , I n ) when Re z ≤ n η n \operatorname{Re}z\le n\eta_n Re z ≤ n η n , where 2 = 1 / sin ( π / 4 ) \sqrt{2}=1/\sin(\pi/4) 2 = 1/ sin ( π /4 ) comes from the angle of the rays, and dist ( z , I n ) = ∣ Im z ∣ = Re z + n / t \operatorname{dist}(z,I_n)=\lvert\operatorname{Im}z\rvert=\operatorname{Re}z+n/t dist ( z , I n ) = ∣ Im z ∣ = Re z + n / t otherwise. Since dist ( Γ , I n ) ≥ n η n / 4 \operatorname{dist}(\Gamma,I_n)\ge n\eta_n/4 dist ( Γ , I n ) ≥ n η n /4 , applying Lemma D.1 and the monotonicity of Λ n ( − a ) \Lambda_n(-a) Λ n ( − a ) in Eq. (10) yields Δ Γ = O C x ( ( log n ) 2 ) \Delta_\Gamma=O_{C_x}((\log n)^2) Δ Γ = O C x (( log n ) 2 ) and − log ( 1 ∧ dist ( Γ , I n ) ) = O C x ( log n ) -\log(1\wedge\operatorname{dist}(\Gamma,I_n))=O_{C_x}(\log n) − log ( 1 ∧ dist ( Γ , I n )) = O C x ( log n ) . Thus the conditions hold for all sufficiently large n n n , independently of t t t . As in Appendix D.1, the remaining quantities are
∥ α ϕ t ∥ T V , ∫ Γ ∣ z ∣ ∨ n η n dist ( z , I n ) ∣ α ϕ t ∣ ( d z ) , ∫ Γ ( ∣ z ∣ ∨ n η n ) ∣ α ϕ t ∣ ( d z ) . \lVert\alpha\phi_t\rVert_{\mathrm{TV}},\qquad
\int_{\Gamma}\frac{\lvert z\rvert\vee n\eta_n}{\operatorname{dist}(z,I_n)}\lvert\alpha\phi_t\rvert(dz),\qquad
\int_{\Gamma}(\lvert z\rvert\vee n\eta_n)\lvert\alpha\phi_t\rvert(dz). ∥ α ϕ t ∥ TV , ∫ Γ dist ( z , I n ) ∣ z ∣ ∨ n η n ∣ α ϕ t ∣ ( d z ) , ∫ Γ (∣ z ∣ ∨ n η n ) ∣ α ϕ t ∣ ( d z ) .
We bound them one-by-one next. For the first, parametrize each ray by z = ( n / t ) ( − 1 + r e ± i π / 4 ) z=(n/t)(-1+re^{\pm i\pi/4}) z = ( n / t ) ( − 1 + r e ± iπ /4 ) , r ≥ 0 r\ge0 r ≥ 0 , so that
∣ α ϕ t ∣ ( d z ) = e 1 − r / 2 2 π ∣ − 1 + r e ± i π / 4 ∣ d r and ∣ − 1 + r e ± i π / 4 ∣ 2 = ( r − 1 / 2 ) 2 + 1 / 2. \lvert\alpha\phi_t\rvert(dz)=\frac{e^{1-r/\sqrt{2}}}{2\pi\lvert-1+re^{\pm i\pi/4}\rvert}\,dr
\qquad\text{and}\qquad
\lvert-1+re^{\pm i\pi/4}\rvert^2=(r-1/\sqrt{2})^2+1/2. ∣ α ϕ t ∣ ( d z ) = 2 π ∣ − 1 + r e ± iπ /4 ∣ e 1 − r / 2 d r and ∣ − 1 + r e ± iπ /4 ∣ 2 = ( r − 1/ 2 ) 2 + 1/2.
Integrating over both rays yields
∥ α ϕ t ∥ T V ≤ 2 π ∫ 0 ∞ e 1 − r / 2 d r = O ( 1 ) . (52) \lVert\alpha\phi_t\rVert_{\mathrm{TV}}\le\frac{\sqrt{2}}{\pi}\int_0^\infty e^{1-r/\sqrt{2}}\,dr=O(1). \tag*{(52)} ∥ α ϕ t ∥ TV ≤ π 2 ∫ 0 ∞ e 1 − r / 2 d r = O ( 1 ) . (52)
For the second quantity, applying the geometric bound above yields
∫ Γ ∣ z ∣ ∨ n η n dist ( z , I n ) ∣ α ϕ t ∣ ( d z ) ≤ 3 ∥ α ϕ t ∥ T V = O ( 1 ) . (53) \int_{\Gamma} \frac{|z| \vee n\eta_n}{\operatorname{dist}(z,I_n)}|\alpha\phi_t|(dz) \le3\|\alpha\phi_t\|_{\mathrm{TV}}=O(1). \tag*{(53)} ∫ Γ dist ( z , I n ) ∣ z ∣ ∨ n η n ∣ α ϕ t ∣ ( d z ) ≤ 3∥ α ϕ t ∥ TV = O ( 1 ) . (53)
For the third quantity, it follows from ∣ z ∣ ∨ n η n ≤ ∣ z ∣ + n η n |z| \vee n\eta_n \le|z|+n\eta_n ∣ z ∣ ∨ n η n ≤ ∣ z ∣ + n η n and the cancellation of ∣ z ∣ |z| ∣ z ∣ that
∫ Γ ( ∣ z ∣ ∨ n η n ) ∣ α ϕ t ∣ ( d z ) ≤ n π t ∫ 0 ∞ e 1 − r / 2 d r + n η n ∥ α ϕ t ∥ T V = O ( n ( η n + t − 1 ) ) . (54) \int_{\Gamma} (|z| \vee n\eta_n)|\alpha\phi_t|(dz) \le\frac{n}{\pi t}\int_0^\infty e^{1-r/\sqrt{2}}\,dr+n\eta_n\|\alpha\phi_t\|_{\mathrm{TV}}=O\left(n(\eta_n+t^{-1})\right). \tag*{(54)} ∫ Γ ( ∣ z ∣ ∨ n η n ) ∣ α ϕ t ∣ ( d z ) ≤ π t n ∫ 0 ∞ e 1 − r / 2 d r + n η n ∥ α ϕ t ∥ TV = O ( n ( η n + t − 1 ) ) . (54)
Substituting Eqs. (52) , (53) and (54) into Theorem 2 proves the two approximation bounds. As in Part I of Appendix D.1, the factors p 2 Δ Γ 3 p^2\Delta_\Gamma^3 p 2 Δ Γ 3 and p 3 / 2 Δ Γ 6 p^{3/2}\Delta_\Gamma^6 p 3/2 Δ Γ 6 are absorbed into O ~ C x , κ \widetilde{O}_{C_x,\kappa} O C x , κ , with implied constants independent of t t t .
Part II: Evaluating the effective formulas. To obtain the effective-risk formulas from Proposition 3, write the two filter expressions there as
x ϕ t ( x ) 2 = ∫ 0 t ( e − r x − e − ( r + t ) x ) d r and ψ t ( x ) − ψ t ( x ′ ) x ′ − x = ∫ 0 t e − r x − ( t − r ) x ′ d r . x\phi_t(x)^2=\int_0^t\left(e^{-rx}-e^{-(r+t)x}\right)dr
\qquad\text{and}\qquad
\frac{\psi_t(x)-\psi_t(x')}{x'-x}=\int_0^t e^{-rx-(t-r)x'}\,dr. x ϕ t ( x ) 2 = ∫ 0 t ( e − r x − e − ( r + t ) x ) d r and x ′ − x ψ t ( x ) − ψ t ( x ′ ) = ∫ 0 t e − r x − ( t − r ) x ′ d r .
The identities extend continuously to zero and to x = x ′ x=x' x = x ′ . Integrating against μ ω , j \mu_{\omega,j} μ ω , j and applying Fubini-Tonelli by nonnegativity yields
∫ [ 0 , ∞ ) x ϕ t ( x ) 2 μ ω , j ( d x ) = ∫ 0 t ( L ω , j ( r ) − L ω , j ( r + t ) ) d r , \int_{[0,\infty)}x\phi_t(x)^2\mu_{\omega,j}(dx)=\int_0^t\left(\mathcal{L}_{\omega,j}(r)-\mathcal{L}_{\omega,j}(r+t)\right)dr, ∫ [ 0 , ∞ ) x ϕ t ( x ) 2 μ ω , j ( d x ) = ∫ 0 t ( L ω , j ( r ) − L ω , j ( r + t ) ) d r ,
∫ [ 0 , ∞ ) ψ t ( x ) − ψ t ( x ′ ) x ′ − x μ ω , j ( d x ) = ∫ 0 t e − ( t − r ) x ′ L ω , j ( r ) d r . \int_{[0,\infty)}\frac{\psi_t(x)-\psi_t(x')}{x'-x}\mu_{\omega,j}(dx)=\int_0^t e^{-(t-r)x'}\mathcal{L}_{\omega,j}(r)dr. ∫ [ 0 , ∞ ) x ′ − x ψ t ( x ) − ψ t ( x ′ ) μ ω , j ( d x ) = ∫ 0 t e − ( t − r ) x ′ L ω , j ( r ) d r .
Substituting the first identity into Eq. (17a) proves the variance formula. Substituting the second into Eq. (17b), with ∫ ψ t ( x ) μ ω , j ( d x ) = L ω , j ( t ) \int\psi_t(x)\mu_{\omega,j}(dx)=\mathcal{L}_{\omega,j}(t) ∫ ψ t ( x ) μ ω , j ( d x ) = L ω , j ( t ) , proves the squared-bias formula.
Proof of Lemma D.1
Part I: η n \eta_n η n for the hard edge event. By Lemma A.6, to obtain P ( σ ^ n ≥ η n ) ≥ 1 − 2 e − n \mathbb{P}(\widehat{\sigma}_n\ge\eta_n)\ge1-2e^{-n} P ( σ n ≥ η n ) ≥ 1 − 2 e − n , it suffices to take η n ≔ τ 1 ( m ) / ( 2 n ) \eta_n\coloneqq\tau_1(m)/(2n) η n : = τ 1 ( m ) / ( 2 n ) for some m m m , provided that
τ 1 ( m ) = Ω C x ( 2 n τ 2 ( m ) + 2 n σ m + 1 ) . \tau_1(m)=\Omega_{C_x}\left(\sqrt{2n\tau_2(m)}+2n\sigma_{m+1}\right). τ 1 ( m ) = Ω C x ( 2 n τ 2 ( m ) + 2 n σ m + 1 ) .
We verify this condition for both laws when n = Ω C x ( 1 ) n=\Omega_{C_x}(1) n = Ω C x ( 1 ) . We take m = ⌈ C n ⌉ m=\lceil Cn\rceil m = ⌈ C n ⌉ and m = n m=n m = n in the power law and critical law, respectively. Here C ≥ 1 C\ge1 C ≥ 1 depends only on C x C_x C x and α \alpha α . By standard elementary estimates to the tails of the relevant series,
2 n τ 2 ( m ) + 2 n σ m + 1 τ 1 ( m ) = { O α ( C − 1 / 2 + C − 1 ) , power law , O β ( ( 1 + log n ) − 1 ) , critical law . \frac{\sqrt{2n\tau_2(m)}+2n\sigma_{m+1}}{\tau_1(m)}
=
\begin{cases}
O_\alpha(C^{-1/2}+C^{-1}), & \text{power law},\\
O_\beta((1+\log n)^{-1}), & \text{critical law}.
\end{cases} τ 1 ( m ) 2 n τ 2 ( m ) + 2 n σ m + 1 = { O α ( C − 1/2 + C − 1 ) , O β (( 1 + log n ) − 1 ) , power law , critical law .
We meet the condition by choosing C C C large enough and then taking n n n large enough. Thus the above choices of η n = τ 1 ( m ) / ( 2 n ) \eta_n=\tau_1(m)/(2n) η n = τ 1 ( m ) / ( 2 n ) suffice and satisfy n η n = Θ C x ( τ 1 ( n ) ) n\eta_n=\Theta_{C_x}(\tau_1(n)) n η n = Θ C x ( τ 1 ( n )) .
For the deterministic support, we apply the edge characterization in Eq. (26) and the paragraph following it in Part III of the proof of Proposition 2 to Eq. (8) to obtain
n η n ≤ τ 1 ( m ) − 2 n τ 2 ( m ) ≤ inf supp μ n = max s > 0 { − 1 s + ∑ j ≥ 1 σ j 1 + n σ j s } ≤ τ 1 ( n ) = Θ C x ( n η n ) . n\eta_n\le\tau_1(m)-2\sqrt{n\tau_2(m)}\le\inf\operatorname{supp}\mu_n=\max_{s>0}\left\{-\frac{1}{s}+\sum_{j\ge1}\frac{\sigma_j}{1+n\sigma_js}\right\}\le\tau_1(n)=\Theta_{C_x}(n\eta_n). n η n ≤ τ 1 ( m ) − 2 n τ 2 ( m ) ≤ inf supp μ n = s > 0 max { − s 1 + j ≥ 1 ∑ 1 + n σ j s σ j } ≤ τ 1 ( n ) = Θ C x ( n η n ) .
We explain the bounds above. For the lower bound, we preserve the tail terms, use ( 1 + x ) − 1 ≥ 1 − x (1+x)^{-1}\ge1-x ( 1 + x ) − 1 ≥ 1 − x , and evaluate at s = 1 / n τ 2 ( m ) s=1/\sqrt{n\tau_2(m)} s = 1/ n τ 2 ( m ) . For the upper bound, it is directly obtained by bounding the first n n n summands by 1 / ( n s ) 1/(ns) 1/ ( n s ) and the remaining sum by τ 1 ( n ) \tau_1(n) τ 1 ( n ) .
Part II: Preparing bounds for invoking Theorem 2. In the previous Part I, we choose η n \eta_n η n explicitly. For the spectral stability parameter, we use the admissible cutoff m = ⌊ ρ n / 4 ⌋ m=\lfloor\rho n/4\rfloor m = ⌊ ρ n /4 ⌋ in Eq. (10) (note that this m m m is different from the m m m we used for the eigenvalue lower bound lemma). From the tail bounds, n σ m + 1 + τ 1 ( m ) = O C x ( τ 1 ( n ) ) n\sigma_{m+1}+\tau_1(m)=O_{C_x}(\tau_1(n)) n σ m + 1 + τ 1 ( m ) = O C x ( τ 1 ( n )) and log ( n r 1 ( m ) ) = O C x ( log n ) \log(nr_1(m))=O_{C_x}(\log n) log ( n r 1 ( m )) = O C x ( log n ) , so
Λ n ( − n η n / 4 ) ≤ 1 + 4 { n σ m + 1 + τ 1 ( m ) log n log ( n r 1 ( m ) ) } n η n = O C x ( ( log n ) 2 ) . \Lambda_n(-n\eta_n/4)\le1+\frac{4\{n\sigma_{m+1}+\tau_1(m)\log n\log(nr_1(m))\}}{n\eta_n}=O_{C_x}((\log n)^2). Λ n ( − n η n /4 ) ≤ 1 + n η n 4 { n σ m + 1 + τ 1 ( m ) log n log ( n r 1 ( m ))} = O C x (( log n ) 2 ) .
For the remaining logarithmic bound, the scale of η n \eta_n η n established in Part I implies − log ( 1 ∧ n η n ) = O C x ( log n ) -\log(1\wedge n\eta_n)=O_{C_x}(\log n) − log ( 1 ∧ n η n ) = O C x ( log n ) , completing the proof.
Proofs for the deterministic self-consistent interpolation
In the proofs, we will repeatedly use the elementary identity Im ( M − 1 ) = − M − ∗ ( Im M ) M − 1 \operatorname{Im}(M^{-1})=-M^{-*}(\operatorname{Im}M)M^{-1} Im ( M − 1 ) = − M −∗ ( Im M ) M − 1 for any bounded invertible operator M M M on (the complexification of) H \mathcal{H} H . If, in addition, Im M \operatorname{Im}M Im M is bounded and invertible, then Im ( M − 1 ) − 1 = − M ( Im M ) − 1 M ∗ \operatorname{Im}(M^{-1})^{-1}=-M(\operatorname{Im}M)^{-1}M^* Im ( M − 1 ) − 1 = − M ( Im M ) − 1 M ∗ .
Proof of Theorem 3
We prove the theorem for z = t + i η ∈ C + ∪ R < 0 z=t+\mathrm{i}\eta\in\mathbb{C}_+\cup\mathbb{R}_{<0} z = t + i η ∈ C + ∪ R < 0 in four parts.
In Part I, we prove (i) and collect the weighted resolvent bounds needed for stability.
In Part II, we bound the one-step change ∣ G ‾ k + 1 ( z , w k + 1 ) − G ‾ k ( z , w k ) ∣ |\overline{G}_{k+1}(z,w_{k+1})-\overline{G}_k(z,w_k)| ∣ G k + 1 ( z , w k + 1 ) − G k ( z , w k ) ∣ for every 1 ≤ k ≤ n − 1 1\le k\le n-1 1 ≤ k ≤ n − 1 by canceling its leading drift through leave-one-out identities.
In Part III, we prove (ii), the pathwise expected-trace stability for { G ‾ k ( z , w k ) } \{\overline{G}_k(z,w_k)\} { G k ( z , w k )} by controlling the resolvent norms separately for large and small sample sizes, then sum the one-step errors.
In Part IV, we prove (iii) for the backward path stability for { w k } \{w_k\} { w k } , by converting expected-trace stability into nearly constant increments and comparing the initial endpoint with the effective spectral parameter equation.
Part I: Proof of (i) and resolvent bounds. We first invoke the following lemma to establish well-posedness and monotonicity of the imaginary part and, when t < 0 t<0 t < 0 , the real part of the backward path. We then prepare the weighted resolvent estimates needed for stability in the following parts. The lemma collects the resolvent and expected-trace bounds from which we derive these path properties. See its proof in Appendix § 13 .4.1.
Lemma E.1. For any fixed z = t + i η ∈ C + ∪ R < 0 z=t+\mathrm{i}\eta\in\mathbb{C}_+\cup\mathbb{R}_{<0} z = t + i η ∈ C + ∪ R < 0 , the backward path { w k } k = 0 n \{w_k\}_{k=0}^n { w k } k = 0 n is uniquely defined, and the following bounds hold.
(i) (Resolvent bounds). For j ∈ [ n ] j\in[n] j ∈ [ n ] and ( z , w ) ∈ D (z,w)\in\mathcal{D} ( z , w ) ∈ D (and in particular, z ∈ C + z\in\mathbb{C}_+ z ∈ C + ), the resolvent exists and ∥ R j − 1 ( z , w ) ∥ ≤ η − 1 \|R_{j-1}(z,w)\|\le\eta^{-1} ∥ R j − 1 ( z , w ) ∥ ≤ η − 1 . In addition, allowing η = 0 \eta=0 η = 0 , along the path, we have for every j ∈ [ n ] j\in[n] j ∈ [ n ] , 0 ≤ k ≤ n 0\le k\le n 0 ≤ k ≤ n and x ∈ H x\in\mathcal{H} x ∈ H ,
∥ R j − 1 ( z , w k ) ∥ ≤ 1 ∣ z − ∣ and ∣ 1 + x ∗ R j − 1 ( z , w k ) x ∣ − 1 ≤ ∣ z ∣ ∣ z − ∣ . (55) \|R_{j-1}(z,w_k)\|\le\frac{1}{|z_-|}\qquad\text{and}\qquad|1+x^*R_{j-1}(z,w_k)x|^{-1}\le\frac{|z|}{|z_-|}. \tag*{(55)} ∥ R j − 1 ( z , w k ) ∥ ≤ ∣ z − ∣ 1 and ∣1 + x ∗ R j − 1 ( z , w k ) x ∣ − 1 ≤ ∣ z − ∣ ∣ z ∣ . (55)
(ii) (Expected-trace bounds ). For j ∈ [ n ] j \in[n] j ∈ [ n ] and ( z , w ) ∈ D (z,w) \in\mathcal{D} ( z , w ) ∈ D , the imaginary parts of the leave-one-out trace and expected-trace are positive, i.e. Im G j ( z , w ) > 0 \operatorname{Im} G_j(z,w)>0 Im G j ( z , w ) > 0 and Im G ‾ j ( z , w ) > 0 \operatorname{Im} \overline{G}_j(z,w)>0 Im G j ( z , w ) > 0 . In addition, allowing η = 0 \eta=0 η = 0 , along the path, we have for every j ∈ [ n ] j \in[n] j ∈ [ n ] and 0 ≤ k ≤ n 0 \le k \le n 0 ≤ k ≤ n ,
∣ 1 + G ‾ j ( z , w k ) ∣ ≥ ∣ z − ∣ ∣ z ∣ . (56) \lvert1+\overline{G}_j(z,w_k)\rvert\ge\frac{\lvert z_{-}\rvert}{\lvert z\rvert}. \tag*{(56)} ∣ 1 + G j ( z , w k )∣ ≥ ∣ z ∣ ∣ z − ∣ . (56)
If t < 0 t<0 t < 0 , then also Re G ‾ j ( z , w k ) ≥ 0 \operatorname{Re}\overline{G}_j(z,w_k)\ge0 Re G j ( z , w k ) ≥ 0 .
(iii) Along the backward path, we also have for 0 ≤ k ≤ n 0 \le k \le n 0 ≤ k ≤ n
Im ( w k z ‾ ) ≥ 0 and Im ( w 0 z ‾ ) ≥ η ∣ w 0 ∣ 2 n . (57) \operatorname{Im}(w_k\overline{z})\ge0 \qquad\text{and}\qquad\operatorname{Im}(w_0\overline{z})\ge\frac{\eta\lvert w_0\rvert^2}{n}. \tag*{(57)} Im ( w k z ) ≥ 0 and Im ( w 0 z ) ≥ n η ∣ w 0 ∣ 2 . (57)
We are now equipped to prove the monotonicity results of the backward path. By Lemma E.1, the backward path is well-defined. For η > 0 \eta>0 η > 0 , applying the recursion and Lemma E.1.(ii) yields
Im w k − 1 − Im w k = Im G ‾ k ( z , w k ) ∣ 1 + G ‾ k ( z , w k ) ∣ 2 > 0. \operatorname{Im} w_{k-1}-\operatorname{Im} w_k=\frac{\operatorname{Im}\overline{G}_k(z,w_k)}{\lvert1+\overline{G}_k(z,w_k)\rvert^2}>0. Im w k − 1 − Im w k = ∣ 1 + G k ( z , w k ) ∣ 2 Im G k ( z , w k ) > 0.
For t < 0 t<0 t < 0 , applying the same recursion and using Re G ‾ k ( z , w k ) ≥ 0 \operatorname{Re}\overline{G}_k(z,w_k)\ge0 Re G k ( z , w k ) ≥ 0 from Lemma E.1.(ii), and we have similarly
Re w k − Re w k − 1 = 1 + Re G ‾ k ( z , w k ) ∣ 1 + G ‾ k ( z , w k ) ∣ 2 > 0. \operatorname{Re} w_k-\operatorname{Re} w_{k-1}=\frac{1+\operatorname{Re}\overline{G}_k(z,w_k)}{\lvert1+\overline{G}_k(z,w_k)\rvert^2}>0. Re w k − Re w k − 1 = ∣ 1 + G k ( z , w k ) ∣ 2 1 + Re G k ( z , w k ) > 0.
Combined with w n = 0 w_n=0 w n = 0 , this shows that the real parts are decreasing, namely Re w 0 < Re w 1 < ⋯ < Re w n = 0 \operatorname{Re} w_0<\operatorname{Re} w_1<\cdots<\operatorname{Re} w_n=0 Re w 0 < Re w 1 < ⋯ < Re w n = 0 . For real z < 0 z<0 z < 0 , the backward induction directly implies w k ∈ R w_k\in\mathbb{R} w k ∈ R for every k k k , and the recursion determines w 0 < w 1 < ⋯ < w n = 0 w_0<w_1<\cdots<w_n=0 w 0 < w 1 < ⋯ < w n = 0 . Together with w n = 0 w_n=0 w n = 0 , these inequalities prove Theorem 3.(i).
The remainder of Part I supplies additional bounds for the weighted resolvents R ~ k ( z , w ) : = Σ 1 / 2 R k ( z , w ) Σ 1 / 2 \widetilde{R}_k(z,w):=\boldsymbol{\Sigma}^{1/2}R_k(z,w)\boldsymbol{\Sigma}^{1/2} R k ( z , w ) := Σ 1/2 R k ( z , w ) Σ 1/2 . For large sample sizes, we compare them with the real resolvent A k ( ζ ) : = Σ 1 / 2 ( T k + ζ I ) − 1 Σ 1 / 2 A_k(\zeta):=\boldsymbol{\Sigma}^{1/2}(T_k+\zeta I)^{-1}\boldsymbol{\Sigma}^{1/2} A k ( ζ ) := Σ 1/2 ( T k + ζ I ) − 1 Σ 1/2 at ζ = ∣ z − ∣ \zeta=\lvert z_{-}\rvert ζ = ∣ z − ∣ . For small sample sizes, we use the coefficient − Re ( e i θ w j ) -\operatorname{Re}(e^{i\theta}w_j) − Re ( e i θ w j ) of Σ \boldsymbol{\Sigma} Σ in the rotated inverse resolvent. The following lemma provides both estimates, whose proof is in Appendix E.4.2.
Lemma E.2 (Weighted resolvent bounds). For any fixed z = t + i η ∈ C + ∪ R < 0 z=t+i\eta\in\mathbb{C}_{+}\cup\mathbb{R}_{<0} z = t + i η ∈ C + ∪ R < 0 and every 0 ≤ k , j ≤ n 0\le k,j\le n 0 ≤ k , j ≤ n , let R ~ k = R ~ k ( z , w j ) \widetilde{R}_k=\widetilde{R}_k(z,w_j) R k = R k ( z , w j ) . Along the backward path, the real resolvents provide the first type of bounds:
∥ R ~ k ∥ ≤ 3 ∣ z ∣ ∣ z − ∣ ⋅ ∥ A k ( ∣ z − ∣ ) ∥ , ∥ R ~ k ∥ F ≤ 3 ∣ z ∣ ∣ z − ∣ ⋅ ∥ A k ( ∣ z − ∣ ) ∥ F , ∣ Tr R ~ k ∣ ≤ 3 ∣ z ∣ ∣ z − ∣ ⋅ Tr A k ( ∣ z − ∣ ) . (58) \lVert\widetilde{R}_k\rVert\le\frac{3\lvert z\rvert}{\lvert z_{-}\rvert}\cdot\lVert A_k(\lvert z_{-}\rvert)\rVert,\qquad
\lVert\widetilde{R}_k\rVert_F \le\frac{3\lvert z\rvert}{\lvert z_{-}\rvert}\cdot\lVert A_k(\lvert z_{-}\rvert)\rVert_F,\qquad
\lvert\operatorname{Tr}\widetilde{R}_k\rvert\le\frac{3\lvert z\rvert}{\lvert z_{-}\rvert}\cdot\operatorname{Tr}A_k(\lvert z_{-}\rvert). \tag*{(58)} ∥ R k ∥ ≤ ∣ z − ∣ 3 ∣ z ∣ ⋅ ∥ A k (∣ z − ∣)∥ , ∥ R k ∥ F ≤ ∣ z − ∣ 3 ∣ z ∣ ⋅ ∥ A k (∣ z − ∣) ∥ F , ∣ Tr R k ∣ ≤ ∣ z − ∣ 3 ∣ z ∣ ⋅ Tr A k (∣ z − ∣) . (58) Set θ = arctan ( 1 + 2 ( t ∨ 0 ) / η ) \theta=\arctan(1+2(t\vee0)/\eta) θ = arctan ( 1 + 2 ( t ∨ 0 ) / η ) for η > 0 \eta>0 η > 0 and θ = π / 4 \theta=\pi/4 θ = π /4 for real z < 0 z<0 z < 0 . For j < n j<n j < n , − Re ( e i θ w j ) > 0 -\operatorname{Re}(e^{i\theta}w_j)>0 − Re ( e i θ w j ) > 0 and, for every integer 1 ≤ m ≤ d 1\le m\le d 1 ≤ m ≤ d , the path parameter itself provides the second type of bounds:
∥ R ~ k ∥ ≤ 1 − Re ( e i θ w j ) and ∥ R ~ k ∥ F 2 ≤ m Re ( e i θ w j ) 2 + 4 τ 2 ( m ) ∣ z − ∣ 2 . (59) \lVert\widetilde{R}_k\rVert\le\frac{1}{-\operatorname{Re}(e^{i\theta}w_j)}
\qquad\text{and}\qquad
\lVert\widetilde{R}_k\rVert_F^2\le\frac{m}{\operatorname{Re}(e^{i\theta}w_j)^2}+\frac{4\tau_2(m)}{\lvert z_{-}\rvert^2}. \tag*{(59)} ∥ R k ∥ ≤ − Re ( e i θ w j ) 1 and ∥ R k ∥ F 2 ≤ Re ( e i θ w j ) 2 m + ∣ z − ∣ 2 4 τ 2 ( m ) . (59)
To use Lemma E.2, Eq. (44a), we need moment bounds for A k ( ζ ) A_k(\zeta) A k ( ζ ) . We obtain them by separating the first m m m population eigenvectors from the tail, following the truncation and intrinsic-dimension matrix Bernstein argument of Cheng and Montanari [28 Lemma 7.2] . The tail block is controlled by the following real resolvent spectral parameter, expressed using the tail quantities in Eq. (9) : for k ≥ 2 k \ge2 k ≥ 2 and 1 ≤ m < k 1 \le m < k 1 ≤ m < k ,
C A ( k , m ) ≔ k σ m + 1 + k σ m + 1 τ 1 ( m ) log ( k ) log ( k r 1 ( m ) ) + τ 1 ( m ) log ( k ) log ( k r 1 ( m ) ) , C_A(k,m) \coloneqq k\sigma_{m+1}+\sqrt{k\sigma_{m+1}\tau_1(m)\log(k)\log(kr_1(m))}+\tau_1(m)\log(k)\log(kr_1(m)), C A ( k , m ) : = k σ m + 1 + k σ m + 1 τ 1 ( m ) log ( k ) log ( k r 1 ( m )) + τ 1 ( m ) log ( k ) log ( k r 1 ( m )) ,
with C A ( k , m ) = 0 C_A(k,m)=0 C A ( k , m ) = 0 when σ m + 1 = 0 \sigma_{m+1}=0 σ m + 1 = 0 . The next lemma converts this tail estimate into the required resolvent moments. Its proof is in Appendix E.4.3.
Lemma E.3 (L p L^p L p -type norm bounds for A k ( ζ ) A_k(\zeta) A k ( ζ ) ). Let Assumption 1 hold. With ρ = ρ ( C x ) \rho=\rho(C_x) ρ = ρ ( C x ) from Assumption 2 , for every fixed κ , D > 0 \kappa,D>0 κ , D > 0 , k = Ω C x , κ , D ( 1 ) k=\Omega_{C_x,\kappa,D}(1) k = Ω C x , κ , D ( 1 ) , 1 ≤ m ≤ ρ k ∧ d 1\le m\le\rho k\wedge d 1 ≤ m ≤ ρ k ∧ d , ζ > 0 \zeta>0 ζ > 0 , and 1 ≤ p ≤ κ log k 1\le p\le\kappa\log k 1 ≤ p ≤ κ log k , we have
∥ A k ( ζ ) ∥ L p ( T k ) ≤ O C x , κ , D ( 1 k ( 1 + C A ( k , m ) ζ ) + k − D ζ ) , (60) \left\lVert A_k(\zeta)\right\rVert_{L^p(\mathcal{T}_k)}
\le O_{C_x,\kappa,D}\left(\frac{1}{k}\left(1+\frac{C_A(k,m)}{\zeta}\right)+\frac{k^{-D}}{\zeta}\right), \tag*{(60)} ∥ A k ( ζ ) ∥ L p ( T k ) ≤ O C x , κ , D ( k 1 ( 1 + ζ C A ( k , m ) ) + ζ k − D ) , (60) ∥ A k ( ζ ) ∥ F , L p ( T k ) 2 ≤ O C x , κ , D ( m k 2 ( 1 + C A ( k , m ) ζ ) 2 + τ 2 ( m ) ζ 2 + ∥ Σ ∥ F 2 k − D ζ 2 ) , (61) \left\lVert A_k(\zeta)\right\rVert_{\mathrm{F},L^p(\mathcal{T}_k)}^2
\le O_{C_x,\kappa,D}\left(\frac{m}{k^2}\left(1+\frac{C_A(k,m)}{\zeta}\right)^2+\frac{\tau_2(m)}{\zeta^2}+\frac{\lVert\Sigma\rVert_{\mathrm{F}}^2 k^{-D}}{\zeta^2}\right), \tag*{(61)} ∥ A k ( ζ ) ∥ F , L p ( T k ) 2 ≤ O C x , κ , D ( k 2 m ( 1 + ζ C A ( k , m ) ) 2 + ζ 2 τ 2 ( m ) + ζ 2 ∥ Σ ∥ F 2 k − D ) , (61) ∥ Tr A k ( ζ ) ∥ L p ( T k ) ≤ O C x , κ , D ( m k ( 1 + C A ( k , m ) ζ ) + τ 1 ( m ) ζ + Tr ( Σ ) k − D ζ ) . (62) \left\lVert\operatorname{Tr} A_k(\zeta)\right\rVert_{L^p(\mathcal{T}_k)}
\le O_{C_x,\kappa,D}\left(\frac{m}{k}\left(1+\frac{C_A(k,m)}{\zeta}\right)+\frac{\tau_1(m)}{\zeta}+\frac{\operatorname{Tr}(\Sigma)k^{-D}}{\zeta}\right). \tag*{(62)} ∥ Tr A k ( ζ ) ∥ L p ( T k ) ≤ O C x , κ , D ( k m ( 1 + ζ C A ( k , m ) ) + ζ τ 1 ( m ) + ζ Tr ( Σ ) k − D ) . (62)
This wraps up Part I, in which we have proved Theorem 3.(i), and Lemmas E.2 and E.3 provide the weighted resolvent bounds needed in the next steps. Part II first expresses the one-step error in terms of these norm bounds before we evaluate them in Part III.
Part II: One-step stability by leave-one-out
We next bound ∣ G ‾ k + 1 ( z , w k + 1 ) − G ‾ k ( z , w k ) ∣ \left|\overline{G}_{k+1}(z,w_{k+1})-\overline{G}_k(z,w_k)\right| G k + 1 ( z , w k + 1 ) − G k ( z , w k ) in terms of the operator and Frobenius norms of the weighted resolvents. The leave-one-out calculation uses sample sizes k − 2 k-2 k − 2 , k − 1 k-1 k − 1 , k k k and the two adjacent path parameters. Accordingly, for k ∈ [ n − 1 ] k\in[n-1] k ∈ [ n − 1 ] , let I k ≔ { ( k − 2 ) ∨ 0 , k − 1 , k } × { w k , w k + 1 } \mathcal{I}_k\coloneqq\{(k-2)\vee0,k-1,k\}\times\{w_k,w_{k+1}\} I k : = {( k − 2 ) ∨ 0 , k − 1 , k } × { w k , w k + 1 } and define
γ k ≔ max ( j , ζ ) ∈ I k ∥ R ~ j ( z , ζ ) ∥ L 4 ( T j ) and φ k ≔ max ( j , ζ ) ∈ I k ∥ R ~ j ( z , ζ ) ∥ F , L 4 ( T j ) . \gamma_k\coloneqq\max_{(j,\zeta)\in\mathcal{I}_k}\left\lVert\widetilde{R}_j(z,\zeta)\right\rVert_{L^4(\mathcal{T}_j)}
\qquad\text{and}\qquad
\varphi_k\coloneqq\max_{(j,\zeta)\in\mathcal{I}_k}\left\lVert\widetilde{R}_j(z,\zeta)\right\rVert_{\mathrm{F},L^4(\mathcal{T}_j)}. γ k : = ( j , ζ ) ∈ I k max R j ( z , ζ ) L 4 ( T j ) and φ k : = ( j , ζ ) ∈ I k max R j ( z , ζ ) F , L 4 ( T j ) .
The next lemma bounds the one-step error
c G ‾ ( k ) = ∣ G ‾ k + 1 ( z , w k + 1 ) − G ‾ k ( z , w k ) ∣ c_{\overline{G}}(k)=\left|\overline{G}_{k+1}(z,w_{k+1})-\overline{G}_k(z,w_k)\right| c G ( k ) = G k + 1 ( z , w k + 1 ) − G k ( z , w k )
in terms of γ k \gamma_k γ k and φ k \varphi_k φ k . Its proof, including the leading-order cancellation, is in Appendix E.4.4.
Lemma E.4. Let Assumption 1 hold. For a fixed z = t + i η ∈ C + ∪ R < 0 z=t+i\eta\in\mathbb{C}_+\cup\mathbb{R}_{<0} z = t + i η ∈ C + ∪ R < 0 and all 1 ≤ k ≤ n − 1 1\le k\le n-1 1 ≤ k ≤ n − 1 ,
c G ‾ ( k ) = O C x ( ∣ z ∣ γ k 2 ∣ z − ∣ + ∣ z ∣ 2 ( k γ k 3 + γ k φ k 2 ) ∣ z − ∣ 2 + ∣ z ∣ 3 ( φ k 2 ( k γ k + φ k ) 2 + γ k ( k γ k + φ k ) 3 ) ∣ z − ∣ 3 ) . (63) c_{\overline{G}}(k)=O_{C_x}\left(\frac{|z|\gamma_k^2}{|z_-|}+\frac{|z|^2(k\gamma_k^3+\gamma_k\varphi_k^2)}{|z_-|^2}+\frac{|z|^3\left(\varphi_k^2(\sqrt{k}\gamma_k+\varphi_k)^2+\gamma_k(\sqrt{k}\gamma_k+\varphi_k)^3\right)}{|z_-|^3}\right). \tag*{(63)} c G ( k ) = O C x ∣ z − ∣ ∣ z ∣ γ k 2 + ∣ z − ∣ 2 ∣ z ∣ 2 ( k γ k 3 + γ k φ k 2 ) + ∣ z − ∣ 3 ∣ z ∣ 3 ( φ k 2 ( k γ k + φ k ) 2 + γ k ( k γ k + φ k ) 3 ) . (63) If in addition, for some Δ > 0 \Delta>0 Δ > 0 satisfying Δ = Ω ( k γ k + φ k ) \Delta=\Omega(\sqrt{k}\gamma_k+\varphi_k) Δ = Ω ( k γ k + φ k ) and ∣ 1 + G ‾ k + 1 ( z , w k + 1 ) ∣ ≥ Δ \left|1+\overline{G}_{k+1}(z,w_{k+1})\right|\ge\Delta 1 + G k + 1 ( z , w k + 1 ) ≥ Δ , we have
c G ‾ ( k ) = O C x , Δ ( ( γ k + φ k 2 ) 2 + k ( γ k + φ k 2 ) 3 ) . (64) c_{\overline{G}}(k)=O_{C_x,\Delta}\left((\gamma_k+\varphi_k^2)^2+k(\gamma_k+\varphi_k^2)^3\right). \tag*{(64)} c G ( k ) = O C x , Δ ( ( γ k + φ k 2 ) 2 + k ( γ k + φ k 2 ) 3 ) . (64)
Heuristically, we can view the “typical scaling” above as γ k = Θ ( φ k 2 ) \gamma_k=\Theta(\varphi_k^2) γ k = Θ ( φ k 2 ) and φ k 2 = O ~ ( n − 1 ) \varphi_k^2=\widetilde{O}(n^{-1}) φ k 2 = O ( n − 1 ) , where O ~ \widetilde{O} O suppresses polynomial factors in the path stability parameter Λ n ( z ) \Lambda_n(z) Λ n ( z ) . Since k ≤ n k\le n k ≤ n and ∣ z ∣ / ∣ z − ∣ ≤ Λ n ( z ) |z|/|z_-|\le\Lambda_n(z) ∣ z ∣/∣ z − ∣ ≤ Λ n ( z ) , substituting this scaling into Lemma E.4, Eq. (46a), gives c G ‾ ( k ) = O ~ C x , ν ( n − 2 ) c_{\overline{G}}(k)=\widetilde{O}_{C_x,\nu}(n^{-2}) c G ( k ) = O C x , ν ( n − 2 ) , and hence an accumulated error of order O ~ C x , ν ( n − 1 ) \widetilde{O}_{C_x,\nu}(n^{-1}) O C x , ν ( n − 1 ) over the path, which is the desired conclusion of Theorem 3.(ii). We will justify this scaling in Part III uniformly in k k k , using two different approaches in the large and small sample size regimes.
Part III: Proof of (ii), expected-trace stability. To prove Theorem 3.(ii), we need to compare every G ‾ k ( z , w k ) \overline{G}_k(z,w_k) G k ( z , w k ) with G ‾ 1 ( z , w 0 ) \overline{G}_1(z,w_0) G 1 ( z , w 0 ) . The norm estimates from Lemma E.3 above are sufficient to control the moments in Lemma E.4 when the sample size is on the order of n n n , but they do not apply when the sample size is close to zero. Accordingly, we further divide this part into two steps.
In Part III-i, we first aggregate the large-sample error terms, and then leverage the resulting trace stability to obtain a uniform lower bound on the rotated coefficient − Re ( e i θ w k ) -\operatorname{Re}(e^{i\theta}w_k) − Re ( e i θ w k ) for all k ≤ ℓ k \le\ell k ≤ ℓ .
In Part III-ii, we use this lower bound for the small sample indices, where the upper bounds in Eq. (44b) of Lemma E.2 provide the remaining norm bounds required for telescoping.
The proof of (ii) then follows by combining the steps above, and accounting separately for the final change from w 1 w_1 w 1 to w 0 w_0 w 0 in G ‾ 1 \overline{G}_1 G 1 . To illustrate this strategy, Figure 10 summarizes the two sample-size regimes. Throughout, fix ν > 0 \nu> 0 ν > 0 as in Assumption 2 and choose m m m attaining the minimum in Eq. (10) . The same m m m works for every z z z , since ∣ z ∣ |z| ∣ z ∣ and ∣ z − ∣ |z_-| ∣ z − ∣ are independent of the cutoff. We write
δ k = δ k ( z ) ≔ 1 1 + G ‾ k ( z , w k ) = w k − w k − 1 (65) \delta_k=\delta_k(z)\coloneqq\frac{1}{1+\overline{G}_k(z,w_k)}=w_k-w_{k-1} \tag*{(65)} δ k = δ k ( z ) : = 1 + G k ( z , w k ) 1 = w k − w k − 1 (65)
for the backward increments when 1 ≤ k ≤ n 1 \le k \le n 1 ≤ k ≤ n . We split the path at ℓ ≔ ⌊ n / 2 ⌋ \ell\coloneqq\lfloor n/2\rfloor ℓ : = ⌊ n /2 ⌋ .
Part III-i: Large sample sizes and a lower bound for the path parameter. We begin by controlling the segment from n n n to ℓ \ell ℓ , since the cumulative increments there will in turn govern the rest of the segment. To do this, we first need uniform bounds on γ k \gamma_k γ k and φ k 2 \varphi_k^2 φ k 2 for ℓ ≤ k ≤ n − 1 \ell\le k \le n-1 ℓ ≤ k ≤ n − 1 , which then allows us to sum the one-step errors from Lemma E.4 .
For ℓ ≤ k ≤ n − 1 \ell\le k \le n-1 ℓ ≤ k ≤ n − 1 , every index j ∈ { k − 2 , k − 1 , k } j \in\{k-2,k-1,k\} j ∈ { k − 2 , k − 1 , k } defining γ k , φ k \gamma_k,\varphi_k γ k , φ k satisfies j ≥ ℓ − 2 = ⌊ n / 2 ⌋ − 2 ≍ n j \ge\ell-2=\lfloor n/2\rfloor-2\asymp n j ≥ ℓ − 2 = ⌊ n /2 ⌋ − 2 ≍ n . Thus the cutoff chosen above satisfies m ≤ ρ ( C x ) ( ℓ − 2 ) ≤ ρ ( C x ) j m \le\rho(C_x)(\ell-2)\le\rho(C_x)j m ≤ ρ ( C x ) ( ℓ − 2 ) ≤ ρ ( C x ) j for every such j j j . For fixed D > 0 D>0 D > 0 , apply Lemma E.3 , Eq. (45a), with p = 4 p=4 p = 4 for γ k \gamma_k γ k , and Eq. (45b) with p = 2 p=2 p = 2 for φ k 2 \varphi_k^2 φ k 2 . Using C A ( j , m ) ≤ C A ( n , m ) C_A(j,m)\le C_A(n,m) C A ( j , m ) ≤ C A ( n , m ) and then Eq. (44a) in Lemma E.2 implies
γ k = O C x , D ( ∣ z ∣ ∣ z − ∣ { 1 n ( 1 + C A ( n , m ) ∣ z − ∣ ) + n − D ∣ z − ∣ } ) , \gamma_k=O_{C_x,D}\left(\frac{|z|}{|z_-|}\left\{\frac{1}{n}\left(1+\frac{C_A(n,m)}{|z_-|}\right)+\frac{n^{-D}}{|z_-|}\right\}\right), γ k = O C x , D ( ∣ z − ∣ ∣ z ∣ { n 1 ( 1 + ∣ z − ∣ C A ( n , m ) ) + ∣ z − ∣ n − D } ) ,
φ k 2 = O C x , D ( ∣ z ∣ 2 ∣ z − ∣ 2 { m n 2 ( 1 + C A ( n , m ) ∣ z − ∣ ) 2 + τ 2 ( m ) ∣ z − ∣ 2 + ∥ Σ ∥ F 2 n − D ∣ z − ∣ 2 } ) . \varphi_k^2=O_{C_x,D}\left(\frac{|z|^2}{|z_-|^2}\left\{\frac{m}{n^2}\left(1+\frac{C_A(n,m)}{|z_-|}\right)^2+\frac{\tau_2(m)}{|z_-|^2}+\frac{\|\Sigma\|_F^2n^{-D}}{|z_-|^2}\right\}\right). φ k 2 = O C x , D ( ∣ z − ∣ 2 ∣ z ∣ 2 { n 2 m ( 1 + ∣ z − ∣ C A ( n , m ) ) 2 + ∣ z − ∣ 2 τ 2 ( m ) + ∣ z − ∣ 2 ∥Σ ∥ F 2 n − D } ) .
To express these bounds in terms of Λ n ( z ) \Lambda_n(z) Λ n ( z ) , we need to control C A ( n , m ) C_A(n,m) C A ( n , m ) and τ 2 ( m ) \tau_2(m) τ 2 ( m ) by Λ n ( z ) \Lambda_n(z) Λ n ( z ) . If σ m + 1 = 0 \sigma_{m+1}=0 σ m + 1 = 0 , then C A ( n , m ) = τ 2 ( m ) = 0 C_A(n,m)=\tau_2(m)=0 C A ( n , m ) = τ 2 ( m ) = 0 and both comparisons below follow from Λ n ( z ) ≥ 1 \Lambda_n(z)\ge1 Λ n ( z ) ≥ 1 . Otherwise r 1 ( m ) ≥ 1 r_1(m)\ge1 r 1 ( m ) ≥ 1 and log ( n ) log ( n r 1 ( m ) ) ≥ 1 \log(n)\log(nr_1(m))\ge1 log ( n ) log ( n r 1 ( m )) ≥ 1 for n ≥ 3 n\ge3 n ≥ 3 . The middle term n σ m + 1 τ 1 ( m ) log ( n ) log ( n r 1 ( m ) ) \sqrt{n}\sigma_{m+1}\tau_1(m)\log(n)\log(nr_1(m)) n σ m + 1 τ 1 ( m ) log ( n ) log ( n r 1 ( m )) in C A ( n , m ) C_A(n,m) C A ( n , m ) is the geometric mean of its first and last terms. We then use τ 2 ( m ) ≤ σ m + 1 τ 1 ( m ) \tau_2(m)\le\sigma_{m+1}\tau_1(m) τ 2 ( m ) ≤ σ m + 1 τ 1 ( m ) and 2 a b ≤ a + b 2\sqrt{ab}\le a+b 2 ab ≤ a + b in the definitions of C A ( n , m ) C_A(n,m) C A ( n , m ) and Λ n ( z ) \Lambda_n(z) Λ n ( z ) to conclude that
n τ 2 ( m ) ∣ z − ∣ 2 ≤ Λ n ( z ) 2 and 1 + C A ( n , m ) ∣ z − ∣ ≤ 3 2 Λ n ( z ) . (66) \frac{n\tau_2(m)}{|z_-|^2}\le\Lambda_n(z)^2
\qquad\text{and}\qquad
1+\frac{C_A(n,m)}{|z_-|}\le\frac{3}{2}\Lambda_n(z). \tag*{(66)} ∣ z − ∣ 2 n τ 2 ( m ) ≤ Λ n ( z ) 2 and 1 + ∣ z − ∣ C A ( n , m ) ≤ 2 3 Λ n ( z ) . (66)
We then choose D D D so that the last remainder terms in γ k \gamma_k γ k and φ k 2 \varphi_k^2 φ k 2 preserve the n − 1 n^{-1} n − 1 scale of the norm estimates inside the brackets. Since ∥ Σ ∥ = 1 \|\Sigma\|=1 ∥Σ∥ = 1 , ∥ Σ ∥ F 2 ≤ Tr Σ ≤ m + τ 1 ( m ) ≤ n + Λ n ( z ) ∣ z − ∣ \|\Sigma\|_F^2\le\operatorname{Tr}\Sigma\le m+\tau_1(m)\le n+\Lambda_n(z)|z_-| ∥Σ ∥ F 2 ≤ Tr Σ ≤ m + τ 1 ( m ) ≤ n + Λ n ( z ) ∣ z − ∣ . Using ∣ z − ∣ ≥ n − ν |z_-|\ge n^{-\nu} ∣ z − ∣ ≥ n − ν , it follows that
n − D ∣ z − ∣ ≤ n ν − D and ∥ Σ ∥ F 2 n − D ∣ z − ∣ 2 ≤ n 1 + 2 ν − D + Λ n ( z ) n ν − D . \frac{n^{-D}}{|z_-|}\le n^{\nu-D}
\qquad\text{and}\qquad
\frac{\|\Sigma\|_F^2n^{-D}}{|z_-|^2}\le n^{1+2\nu-D}+\Lambda_n(z)n^{\nu-D}. ∣ z − ∣ n − D ≤ n ν − D and ∣ z − ∣ 2 ∥Σ ∥ F 2 n − D ≤ n 1 + 2 ν − D + Λ n ( z ) n ν − D .
Taking D = 2 ν + 2 D=2\nu+2 D = 2 ν + 2 makes n 1 + 2 ν − D = n − 1 n^{1+2\nu-D}=n^{-1} n 1 + 2 ν − D = n − 1 , so both remainders are O ( Λ n ( z ) / n ) O(\Lambda_n(z)/n) O ( Λ n ( z ) / n ) . Since all implicit constants from this point on depend only on C x C_x C x and ν \nu ν , we will suppress them from the big-Oh notations for simplicity. Consequently, by collecting all the preceding bounds, for all ℓ ≤ k ≤ n − 1 \ell\leq k\leq n-1 ℓ ≤ k ≤ n − 1 , we have
γ k = O ( Λ n ( z ) 2 n ) and φ k 2 = O ( Λ n ( z ) 4 n ) . \gamma_k=O\left(\frac{\Lambda_n(z)^2}{n}\right)\qquad\text{and}\qquad\varphi_k^2=O\left(\frac{\Lambda_n(z)^4}{n}\right). γ k = O ( n Λ n ( z ) 2 ) and φ k 2 = O ( n Λ n ( z ) 4 ) .
Since k ≤ n k\leq n k ≤ n , these two bounds give k γ k + φ k = O ( Λ n ( z ) 2 / n ) \sqrt{k}\gamma_k+\varphi_k=O(\Lambda_n(z)^2/\sqrt{n}) k γ k + φ k = O ( Λ n ( z ) 2 / n ) . Substituting the preceding bounds into Eq. (46a) in Lemma E.4 implies the following without the additional denominator stability assumption on Δ \Delta Δ :
c G ‾ ( k ) = O ( ∣ z ∣ γ k 2 ∣ z − ∣ + ∣ z ∣ 2 ( k γ k 3 + γ k φ k 2 ) ∣ z − ∣ 2 + ∣ z ∣ 3 ( φ k 2 ( k γ k + φ k ) 2 + γ k ( k γ k + φ k ) 3 ) ∣ z − ∣ 3 ) = O ( Λ n ( z ) 5 n 2 + Λ n ( z ) 8 n 2 + Λ n ( z ) 11 n 2 ) = O ( Λ n ( z ) 11 n 2 ) . (67) \begin{aligned}
c_{\overline{G}}(k)=O\left(\frac{|z|\gamma_k^2}{|z_-|}+\frac{|z|^2(k\gamma_k^3+\gamma_k\varphi_k^2)}{|z_-|^2}+\frac{|z|^3\left(\varphi_k^2(\sqrt{k}\gamma_k+\varphi_k)^2+\gamma_k(\sqrt{k}\gamma_k+\varphi_k)^3\right)}{|z_-|^3}\right) \\
&=O\left(\frac{\Lambda_n(z)^5}{n^2}+\frac{\Lambda_n(z)^8}{n^2}+\frac{\Lambda_n(z)^{11}}{n^2}\right)=O\left(\frac{\Lambda_n(z)^{11}}{n^2}\right). \tag*{(67)}
\end{aligned} c G ( k ) = O ∣ z − ∣ ∣ z ∣ γ k 2 + ∣ z − ∣ 2 ∣ z ∣ 2 ( k γ k 3 + γ k φ k 2 ) + ∣ z − ∣ 3 ∣ z ∣ 3 ( φ k 2 ( k γ k + φ k ) 2 + γ k ( k γ k + φ k ) 3 ) = O ( n 2 Λ n ( z ) 5 + n 2 Λ n ( z ) 8 + n 2 Λ n ( z ) 11 ) = O ( n 2 Λ n ( z ) 11 ) . (67)
Summing G ‾ k + 1 ( z , w k + 1 ) − G ‾ k ( z , w k ) \overline{G}_{k+1}(z,w_{k+1})-\overline{G}_k(z,w_k) G k + 1 ( z , w k + 1 ) − G k ( z , w k ) from k = j k=j k = j to n − 1 n-1 n − 1 and applying Eq. (67) to the n − j n-j n − j summands, with w n = 0 w_n=0 w n = 0 , yields, for ℓ ≤ j ≤ n \ell\leq j\leq n ℓ ≤ j ≤ n ,
∣ G ‾ j ( z , w j ) − G ‾ n ( z , 0 ) ∣ ≤ ∑ k = j n − 1 c G ‾ ( k ) = O ( Λ n ( z ) 11 n ) . (68) \left|\overline{G}_j(z,w_j)-\overline{G}_n(z,0)\right|\leq\sum_{k=j}^{n-1}c_{\overline{G}}(k)=O\left(\frac{\Lambda_n(z)^{11}}{n}\right). \tag*{(68)} G j ( z , w j ) − G n ( z , 0 ) ≤ k = j ∑ n − 1 c G ( k ) = O ( n Λ n ( z ) 11 ) . (68)
For the small-sample segment, Lemma E.2, Eq. (44b), reduces the norm estimates to a lower bound on − Re ( e i θ w k ) -\operatorname{Re}(e^{i\theta}w_k) − Re ( e i θ w k ) for k ≤ ℓ k\leq\ell k ≤ ℓ . Since w k = − ∑ j = k + 1 n δ j w_k=-\sum_{j=k+1}^{n}\delta_j w k = − ∑ j = k + 1 n δ j , it suffices to show that every rotated increment has positive real part and that the last n − ℓ n-\ell n − ℓ increments contribute at least order n / Λ n ( z ) 3 n/\Lambda_n(z)^3 n / Λ n ( z ) 3 in total.
Fix j ∈ [ n ] j\in[n] j ∈ [ n ] and use the rotation in the proof of Lemma E.2, Eq. (96) , at U = e i θ R j − 1 ( z , w j ) − 1 U=e^{i\theta}R_{j-1}(z,w_j)^{-1} U = e i θ R j − 1 ( z , w j ) − 1 . Since Re U ≻ 0 \operatorname{Re}U\succ0 Re U ≻ 0 , we have Re ( e − i θ R j − 1 ( z , w j ) ) = U − ∗ ( Re U ) U − 1 ⪰ 0 \operatorname{Re}(e^{-i\theta}R_{j-1}(z,w_j))=U^{-*}(\operatorname{Re}U)U^{-1}\succeq0 Re ( e − i θ R j − 1 ( z , w j )) = U −∗ ( Re U ) U − 1 ⪰ 0 . Taking the Σ \Sigma Σ -weighted trace and expectation gives Re ( e − i θ G ‾ j ( z , w j ) ) ≥ 0 \operatorname{Re}(e^{-i\theta}\overline{G}_j(z,w_j))\geq0 Re ( e − i θ G j ( z , w j )) ≥ 0 . Using δ j = ( 1 + G ‾ j ( z , w j ) ) − 1 \delta_j=(1+\overline{G}_j(z,w_j))^{-1} δ j = ( 1 + G j ( z , w j ) ) − 1 therefore yields
Re ( e i θ δ j ) = cos θ + Re ( e − i θ G ‾ j ( z , w j ) ) ∣ 1 + G ‾ j ( z , w j ) ∣ 2 ≥ cos θ ∣ δ j ∣ 2 > 0. \operatorname{Re}(e^{i\theta}\delta_j)=\frac{\cos\theta+\operatorname{Re}(e^{-i\theta}\overline{G}_j(z,w_j))}{|1+\overline{G}_j(z,w_j)|^2}\geq\cos\theta\,|\delta_j|^2>0. Re ( e i θ δ j ) = ∣1 + G j ( z , w j ) ∣ 2 cos θ + Re ( e − i θ G j ( z , w j )) ≥ cos θ ∣ δ j ∣ 2 > 0.
Since we need a lower bound on − Re ( e i θ w k ) -\operatorname{Re}(e^{i\theta}w_k) − Re ( e i θ w k ) for k ≤ ℓ k\leq\ell k ≤ ℓ , we retain the last n − ℓ n-\ell n − ℓ terms in its sum. The preceding inequality reduces this to bounding ∣ δ j ∣ |\delta_j| ∣ δ j ∣ from below, or equivalently ∣ δ j − 1 ∣ |\delta_j^{-1}| ∣ δ j − 1 ∣ from above, for j > ℓ j>\ell j > ℓ . By Eq. (65) , δ j − 1 − δ n − 1 = G ‾ j ( z , w j ) − G ‾ n ( z , 0 ) \delta_j^{-1}-\delta_n^{-1}=\overline{G}_j(z,w_j)-\overline{G}_n(z,0) δ j − 1 − δ n − 1 = G j ( z , w j ) − G n ( z , 0 ) . Applying the triangle inequality and Eq. (68) yields ∣ δ j − 1 ∣ ≤ ∣ δ n − 1 ∣ + O ( Λ n ( z ) 11 / n ) |\delta_j^{-1}|\leq|\delta_n^{-1}|+O(\Lambda_n(z)^{11}/n) ∣ δ j − 1 ∣ ≤ ∣ δ n − 1 ∣ + O ( Λ n ( z ) 11 / n ) , so it suffices to bound the terminal reciprocal.
Since w n = 0 w_n=0 w n = 0 , we can compare the terminal trace directly with the real resolvent at ∣ z ∣ |z| ∣ z ∣ . This choice cancels the factor ∣ z ∣ |z| ∣ z ∣ in the comparison against the denominators ζ = ∣ z ∣ \zeta=|z| ζ = ∣ z ∣ in the real trace bound, leaving a single factor ∣ z − ∣ − 1 |z_-|^{-1} ∣ z − ∣ − 1 . For x ≥ 0 x\geq0 x ≥ 0 , we have ( x + ∣ z ∣ ) / ∣ x − z ∣ ≤ 1 + 2 ∣ z ∣ / ∣ z − ∣ ≤ 3 ∣ z ∣ / ∣ z − ∣ (x+|z|)/|x-z|\leq1+2|z|/|z_-|\leq3|z|/|z_-| ( x + ∣ z ∣ ) /∣ x − z ∣ ≤ 1 + 2∣ z ∣/∣ z − ∣ ≤ 3∣ z ∣/∣ z − ∣ . Applying this inequality in the eigenbasis of T n − 1 T_{n-1} T n − 1 and then applying Lemma E.3, Eq. (45c), at ζ = ∣ z ∣ \zeta=|z| ζ = ∣ z ∣ , sample size n − 1 n-1 n − 1 , p = 1 p=1 p = 1 and D = 2 ν + 2 D=2\nu+2 D = 2 ν + 2 yields
∣ δ n − 1 ∣ = ∣ 1 + G ‾ n ( z , 0 ) ∣ ≤ (i) 1 + 3 ∣ z ∣ ∣ z − ∣ E Tr A n − 1 ( ∣ z ∣ ) |\delta_n^{-1}|=|1+\overline{G}_n(z,0)|\mathrel{\mathop{\leq}^{\text{(i)}}}1+\frac{3|z|}{|z_-|}\mathbb{E}\operatorname{Tr}A_{n-1}(|z|) ∣ δ n − 1 ∣ = ∣1 + G n ( z , 0 ) ∣ ≤ (i) 1 + ∣ z − ∣ 3∣ z ∣ E Tr A n − 1 ( ∣ z ∣ )
≤ (ii) O ( ∣ z ∣ + C A ( n , m ) + τ 1 ( m ) + n − D Tr Σ ∣ z − ∣ ) = O ( Λ n ( z ) ) . (69) \mathrel{\mathop{\leq}^{\text{(ii)}}}O\left(\frac{|z|+C_A(n,m)+\tau_1(m)+n^{-D}\operatorname{Tr}\Sigma}{|z_-|}\right)=O(\Lambda_n(z)). \tag*{(69)} ≤ (ii) O ( ∣ z − ∣ ∣ z ∣ + C A ( n , m ) + τ 1 ( m ) + n − D Tr Σ ) = O ( Λ n ( z )) . (69)
In (i), the trace assigns nonnegative weights from Σ \boldsymbol{\Sigma} Σ to the eigenvalues of T n − 1 T_{n-1} T n − 1 . In (ii), multiplying Lemma E.3 , Eq. (45c), by ∣ z ∣ |z| ∣ z ∣ yields
∣ z ∣ E Tr A n − 1 ( ∣ z ∣ ) = O ( m n − 1 ( ∣ z ∣ + C A ( n − 1 , m ) ) + τ 1 ( m ) + ( n − 1 ) − D Tr Σ ) . |z|\mathbb{E}\operatorname{Tr} A_{n-1}(|z|)=O\left(\frac{m}{n-1}\left(|z|+C_A(n-1,m)\right)+\tau_1(m)+(n-1)^{-D}\operatorname{Tr}\boldsymbol{\Sigma}\right). ∣ z ∣ E Tr A n − 1 ( ∣ z ∣ ) = O ( n − 1 m ( ∣ z ∣ + C A ( n − 1 , m ) ) + τ 1 ( m ) + ( n − 1 ) − D Tr Σ ) .
We use m ≤ n − 1 m\le n-1 m ≤ n − 1 , C A ( n − 1 , m ) ≤ C A ( n , m ) C_A(n-1,m)\le C_A(n,m) C A ( n − 1 , m ) ≤ C A ( n , m ) . The last bound follows from Eq. (66) , Tr Σ ≤ m + τ 1 ( m ) ≤ n + Λ n ( z ) ∣ z − ∣ \operatorname{Tr}\boldsymbol{\Sigma}\le m+\tau_1(m)\le n+\Lambda_n(z)|z_-| Tr Σ ≤ m + τ 1 ( m ) ≤ n + Λ n ( z ) ∣ z − ∣ and ∣ z − ∣ ≥ n − ν |z_-|\ge n^{-\nu} ∣ z − ∣ ≥ n − ν , as in the choice of D D D after Eq. (66) . Using n = Ω ( Λ n ( z ) 16 ) n=\Omega(\Lambda_n(z)^{16}) n = Ω ( Λ n ( z ) 16 ) and Λ n ( z ) ≥ 1 \Lambda_n(z)\ge1 Λ n ( z ) ≥ 1 in Eqs. (68) and (69) , we have
max ℓ < j ≤ n ∣ δ j − 1 ∣ ≤ ∣ δ n − 1 ∣ + max ℓ < j ≤ n ∣ G ‾ j ( z , w j ) − G ‾ n ( z , 0 ) ∣ = O ( Λ n ( z ) + Λ n ( z ) 11 n ) = O ( Λ n ( z ) ) . \max_{\ell<j\le n}|\delta_j^{-1}|\le|\delta_n^{-1}|+\max_{\ell<j\le n}|\overline{G}_j(z,w_j)-\overline{G}_n(z,0)|=O\left(\Lambda_n(z)+\frac{\Lambda_n(z)^{11}}{n}\right)=O(\Lambda_n(z)). ℓ < j ≤ n max ∣ δ j − 1 ∣ ≤ ∣ δ n − 1 ∣ + ℓ < j ≤ n max ∣ G j ( z , w j ) − G n ( z , 0 ) ∣ = O ( Λ n ( z ) + n Λ n ( z ) 11 ) = O ( Λ n ( z )) .
For every k ≤ ℓ k\le\ell k ≤ ℓ , positivity allows us to retain just the last n − ℓ n-\ell n − ℓ terms. The preceding bound gives min ℓ < j ≤ n ∣ δ j ∣ = Ω ( 1 / Λ n ( z ) ) \min_{\ell<j\le n}|\delta_j|=\Omega(1/\Lambda_n(z)) min ℓ < j ≤ n ∣ δ j ∣ = Ω ( 1/ Λ n ( z )) . Using cos θ ≥ ∣ z − ∣ / ( 3 ∣ z ∣ ) ≥ 1 / ( 3 Λ n ( z ) ) \cos\theta\ge|z_-|/(3|z|)\ge1/(3\Lambda_n(z)) cos θ ≥ ∣ z − ∣/ ( 3∣ z ∣ ) ≥ 1/ ( 3 Λ n ( z )) , as in Part II of the proof of Lemma E.2 , and n − ℓ ≥ n / 2 n-\ell\ge n/2 n − ℓ ≥ n /2 yields, for 0 ≤ k ≤ ℓ 0\le k\le\ell 0 ≤ k ≤ ℓ ,
− Re ( e i θ w k ) = ∑ j = k + 1 n Re ( e i θ δ j ) ≥ n − ℓ 3 Λ n ( z ) min ℓ < j ≤ n ∣ δ j ∣ 2 = Ω ( n Λ n ( z ) 3 ) . (70) -\operatorname{Re}(e^{\mathrm{i}\theta}w_k)=\sum_{j=k+1}^{n}\operatorname{Re}(e^{\mathrm{i}\theta}\delta_j)\ge\frac{n-\ell}{3\Lambda_n(z)}\min_{\ell<j\le n}|\delta_j|^2=\Omega\left(\frac{n}{\Lambda_n(z)^3}\right). \tag*{(70)} − Re ( e i θ w k ) = j = k + 1 ∑ n Re ( e i θ δ j ) ≥ 3 Λ n ( z ) n − ℓ ℓ < j ≤ n min ∣ δ j ∣ 2 = Ω ( Λ n ( z ) 3 n ) . (70)
Part III-ii: Small sample sizes and stability of the expected traces. Our goal is now to bound every remaining one-step error and then sum along the entire path. For 1 ≤ k < ℓ 1\le k<\ell 1 ≤ k < ℓ , the weighted resolvents defining γ k \gamma_k γ k and φ k \varphi_k φ k use path indices j ∈ { k , k + 1 } ⊆ { 0 , … , ℓ } j\in\{k,k+1\}\subseteq\{0,\ldots,\ell\} j ∈ { k , k + 1 } ⊆ { 0 , … , ℓ } . Thus Eq. (70) applies to both. Applying Eq. (44b) in Lemma E.2 , we have the following estimates as
γ k = O ( Λ n ( z ) 3 n ) and φ k 2 = O ( m Λ n ( z ) 6 n 2 + τ 2 ( m ) ∣ z − ∣ 2 ) = O ( Λ n ( z ) 6 n ) . \gamma_k=O\left(\frac{\Lambda_n(z)^3}{n}\right)\qquad\text{and}\qquad\varphi_k^2=O\left(\frac{m\Lambda_n(z)^6}{n^2}+\frac{\tau_2(m)}{|z_-|^2}\right)=O\left(\frac{\Lambda_n(z)^6}{n}\right). γ k = O ( n Λ n ( z ) 3 ) and φ k 2 = O ( n 2 m Λ n ( z ) 6 + ∣ z − ∣ 2 τ 2 ( m ) ) = O ( n Λ n ( z ) 6 ) .
where we use m ≤ n m\le n m ≤ n and Eq. (66) . In particular, this implies k γ k + φ k = O ( Λ n ( z ) 3 / n ) \sqrt{k}\gamma_k+\varphi_k=O(\Lambda_n(z)^3/\sqrt{n}) k γ k + φ k = O ( Λ n ( z ) 3 / n ) , and these bounds for γ k \gamma_k γ k , φ k \varphi_k φ k and k γ k + φ k \sqrt{k}\gamma_k+\varphi_k k γ k + φ k combined with Eq. (46a) in Lemma E.4 imply that
c G ‾ ( k ) = O ( Λ n ( z ) 7 n 2 + Λ n ( z ) 11 n 2 + Λ n ( z ) 15 n 2 ) = O ( Λ n ( z ) 15 n 2 ) . c_{\overline{G}}(k)=O\left(\frac{\Lambda_n(z)^7}{n^2}+\frac{\Lambda_n(z)^{11}}{n^2}+\frac{\Lambda_n(z)^{15}}{n^2}\right)=O\left(\frac{\Lambda_n(z)^{15}}{n^2}\right). c G ( k ) = O ( n 2 Λ n ( z ) 7 + n 2 Λ n ( z ) 11 + n 2 Λ n ( z ) 15 ) = O ( n 2 Λ n ( z ) 15 ) .
Combining this estimate with the large-sample bound c G ‾ ( k ) = O ( Λ n ( z ) 11 / n 2 ) c_{\overline{G}}(k)=O(\Lambda_n(z)^{11}/n^2) c G ( k ) = O ( Λ n ( z ) 11 / n 2 ) from Part III-i, Eq. (67) , and using Λ n ( z ) ≥ 1 \Lambda_n(z)\ge1 Λ n ( z ) ≥ 1 , we obtain c G ‾ ( k ) = O ( Λ n ( z ) 15 / n 2 ) c_{\overline{G}}(k)=O(\Lambda_n(z)^{15}/n^2) c G ( k ) = O ( Λ n ( z ) 15 / n 2 ) for every 1 ≤ k ≤ n − 1 1\le k\le n-1 1 ≤ k ≤ n − 1 . Summing the increments between any two indices proves
max 1 ≤ i , j ≤ n ∣ G ‾ i ( z , w i ) − G ‾ j ( z , w j ) ∣ ≤ ∑ k = 1 n − 1 c G ‾ ( k ) = O ( Λ n ( z ) 15 n ) . (71) \max_{1\le i,j\le n}|\overline{G}_i(z,w_i)-\overline{G}_j(z,w_j)|\le\sum_{k=1}^{n-1}c_{\overline{G}}(k)=O\left(\frac{\Lambda_n(z)^{15}}{n}\right). \tag*{(71)} 1 ≤ i , j ≤ n max ∣ G i ( z , w i ) − G j ( z , w j ) ∣ ≤ k = 1 ∑ n − 1 c G ( k ) = O ( n Λ n ( z ) 15 ) . (71)
To establish Theorem 3.(ii), which also compares G ‾ k ( z , w k ) \overline{G}_k(z,w_k) G k ( z , w k ) with G ‾ 1 ( z , w 0 ) \overline{G}_1(z,w_0) G 1 ( z , w 0 ) , we also need to bound the offset G ‾ 1 ( z , w 1 ) − G ‾ 1 ( z , w 0 ) \overline{G}_1(z,w_1)-\overline{G}_1(z,w_0) G 1 ( z , w 1 ) − G 1 ( z , w 0 ) . For this last change of parameter, we use the same path lower bound at the two indices 0 and 1. Applying Lemma E.2 , Eq. (44b), with sample index 0 and using Eqs. (66) and (70) yields, for i ∈ { 0 , 1 } i\in\{0,1\} i ∈ { 0 , 1 } ,
∥ R ~ 0 ( z , w i ) ∥ F 2 ≤ m Re ( e i θ w i ) 2 + 4 τ 2 ( m ) ∣ z − ∣ 2 = O ( Λ n ( z ) 6 n ) . \|\widetilde{R}_0(z,w_i)\|_F^2\le\frac{m}{\operatorname{Re}(e^{\mathrm{i}\theta}w_i)^2}+\frac{4\tau_2(m)}{|z_-|^2}=O\left(\frac{\Lambda_n(z)^6}{n}\right). ∥ R 0 ( z , w i ) ∥ F 2 ≤ Re ( e i θ w i ) 2 m + ∣ z − ∣ 2 4 τ 2 ( m ) = O ( n Λ n ( z ) 6 ) .
Here ∣ δ 1 ∣ ≤ Λ n ( z ) |\delta_1|\le\Lambda_n(z) ∣ δ 1 ∣ ≤ Λ n ( z ) , and therefore by trace Cauchy–Schwarz,
∣ G ‾ 1 ( z , w 1 ) − G ‾ 1 ( z , w 0 ) ∣ ≤ ∣ δ 1 ∣ ∥ R ~ 0 ( z , w 1 ) ∥ F ∥ R ~ 0 ( z , w 0 ) ∥ F = O ( Λ n ( z ) 7 n ) . |\overline{G}_1(z,w_1)-\overline{G}_1(z,w_0)|\le|\delta_1|\|\widetilde{R}_0(z,w_1)\|_F\|\widetilde{R}_0(z,w_0)\|_F=O\left(\frac{\Lambda_n(z)^7}{n}\right). ∣ G 1 ( z , w 1 ) − G 1 ( z , w 0 ) ∣ ≤ ∣ δ 1 ∣∥ R 0 ( z , w 1 ) ∥ F ∥ R 0 ( z , w 0 ) ∥ F = O ( n Λ n ( z ) 7 ) .
Combining the bound for ∣ G ‾ 1 ( z , w 1 ) − G ‾ 1 ( z , w 0 ) ∣ |\overline{G}_1(z,w_1)-\overline{G}_1(z,w_0)| ∣ G 1 ( z , w 1 ) − G 1 ( z , w 0 ) ∣ with Eq. (71) yields
max 1 ≤ k ≤ n ∣ G ‾ k ( z , w k ) − G ‾ 1 ( z , w 0 ) ∣ = O ( Λ n ( z ) 15 n ) . (72) \max_{1\leq k\leq n}|\overline{G}_k(z,w_k)-\overline{G}_1(z,w_0)|=O\left(\frac{\Lambda_n(z)^{15}}{n}\right). \tag*{(72)} 1 ≤ k ≤ n max ∣ G k ( z , w k ) − G 1 ( z , w 0 ) ∣ = O ( n Λ n ( z ) 15 ) . (72)
Combining with taking j = n j=n j = n in Eq. (71) completes the proof of (ii).
Part IV: Proof of (iii), path stability. To prove (iii), we use Eq. (72) to compare backward increments with ( 1 + G ‾ 1 ( z , w 0 ) ) − 1 (1+\overline{G}_1(z,w_0))^{-1} ( 1 + G 1 ( z , w 0 ) ) − 1 . Summing yields an approximate equation for w 0 w_0 w 0 , and Lemma A.4 then compares w 0 w_0 w 0 and w 0 l i n w_0^{\mathrm{lin}} w 0 lin . The proof concludes by summing the increment differences.
Part IV-i: Stable increments and the endpoint equation. Let δ 0 : = ( 1 + G ‾ 1 ( z , w 0 ) ) − 1 \delta_0:=(1+\overline{G}_1(z,w_0))^{-1} δ 0 := ( 1 + G 1 ( z , w 0 ) ) − 1 . Applying Lemma E.1 .(ii), Eq. (43b), we have max 0 ≤ j ≤ n ∣ δ j ∣ ≤ Λ n ( z ) \max_{0\leq j\leq n}|\delta_j|\leq\Lambda_n(z) max 0 ≤ j ≤ n ∣ δ j ∣ ≤ Λ n ( z ) . Using the reciprocal identity and Eq. (72) , we obtain, for 1 ≤ j ≤ n 1\leq j\leq n 1 ≤ j ≤ n ,
δ 0 − δ j = δ 0 δ j ( G ‾ j ( z , w j ) − G ‾ 1 ( z , w 0 ) ) and ∣ δ j − δ 0 ∣ = O ( Λ n ( z ) 17 n ) . (73) \delta_0-\delta_j=\delta_0\delta_j\left(\overline{G}_j(z,w_j)-\overline{G}_1(z,w_0)\right)
\qquad\text{and}\qquad
|\delta_j-\delta_0|=O\left(\frac{\Lambda_n(z)^{17}}{n}\right). \tag*{(73)} δ 0 − δ j = δ 0 δ j ( G j ( z , w j ) − G 1 ( z , w 0 ) ) and ∣ δ j − δ 0 ∣ = O ( n Λ n ( z ) 17 ) . (73)
Here ∣ δ 0 δ j ∣ ≤ Λ n ( z ) 2 |\delta_0\delta_j|\leq\Lambda_n(z)^2 ∣ δ 0 δ j ∣ ≤ Λ n ( z ) 2 , while Eq. (72) bounds the trace difference by O ( Λ n ( z ) 15 / n ) O(\Lambda_n(z)^{15}/n) O ( Λ n ( z ) 15 / n ) uniformly in j j j . It then follows from summing these differences, with ∣ ∑ j = 1 n ( δ 0 − δ j ) ∣ ≤ n max 1 ≤ j ≤ n ∣ δ 0 − δ j ∣ |\sum_{j=1}^n(\delta_0-\delta_j)|\leq n\max_{1\leq j\leq n}|\delta_0-\delta_j| ∣ ∑ j = 1 n ( δ 0 − δ j ) ∣ ≤ n max 1 ≤ j ≤ n ∣ δ 0 − δ j ∣ , and using w 0 = − ∑ j = 1 n δ j w_0=-\sum_{j=1}^n\delta_j w 0 = − ∑ j = 1 n δ j from Eq. (65) and w n = 0 w_n=0 w n = 0 that
w 0 + n δ 0 = ∑ j = 1 n ( δ 0 − δ j ) = O ( Λ n ( z ) 17 ) . (74) w_0+n\delta_0=\sum_{j=1}^n(\delta_0-\delta_j)=O(\Lambda_n(z)^{17}). \tag*{(74)} w 0 + n δ 0 = j = 1 ∑ n ( δ 0 − δ j ) = O ( Λ n ( z ) 17 ) . (74)
We next convert this residual into a perturbation of the effective spectral parameter equation, which will allow us to compare w 0 w_0 w 0 with w 0 l i n w_0^{\mathrm{lin}} w 0 lin .
Part IV-ii: Comparison with the population endpoint. To compare w 0 w_0 w 0 with w 0 l i n = n z s n ( z ) w_0^{\mathrm{lin}}=nzs_n(z) w 0 lin = n z s n ( z ) , we will identify w 0 / ( n z ) w_0/(nz) w 0 / ( n z ) with s n s_n s n using an approximate fixed-point argument and apply Lemma A.4 . Indeed, w 0 l i n / ( n z ) w_0^{\mathrm{lin}}/(nz) w 0 lin / ( n z ) satisfies the exact fixed-point equation in Eq. (8) ,
z = − ( w 0 l i n n z ) − 1 + ∑ j ≥ 1 σ j 1 + σ j w 0 l i n / z . z=-\left(\frac{w_0^{\mathrm{lin}}}{nz}\right)^{-1}+\sum_{j\geq1}\frac{\sigma_j}{1+\sigma_jw_0^{\mathrm{lin}}/z}. z = − ( n z w 0 lin ) − 1 + j ≥ 1 ∑ 1 + σ j w 0 lin / z σ j .
This boils down to controlling the perturbed equation, which we will identify next. By Eq. (70) , we have w 0 ≠ 0 w_0\ne0 w 0 = 0 . For z ∈ C + z\in\mathbb{C}_+ z ∈ C + , applying Lemma E.1 .(iii) gives Im ( w 0 / ( n z ) ) ≥ η ∣ w 0 / ( n z ) ∣ 2 > 0 \operatorname{Im}(w_0/(nz))\geq\eta|w_0/(nz)|^2>0 Im ( w 0 / ( n z )) ≥ η ∣ w 0 / ( n z ) ∣ 2 > 0 . Since T 0 = 0 T_0=0 T 0 = 0 , the trace G ‾ 1 ( z , w 0 ) \overline{G}_1(z,w_0) G 1 ( z , w 0 ) is deterministic and satisfies G ‾ 1 ( z , w 0 ) = − z − 1 ∑ j ≥ 1 σ j / ( 1 + σ j w 0 / z ) \overline{G}_1(z,w_0)=-z^{-1}\sum_{j\geq1}\sigma_j/(1+\sigma_jw_0/z) G 1 ( z , w 0 ) = − z − 1 ∑ j ≥ 1 σ j / ( 1 + σ j w 0 / z ) . Thus δ 0 − 1 = 1 + G ‾ 1 ( z , w 0 ) \delta_0^{-1}=1+\overline{G}_1(z,w_0) δ 0 − 1 = 1 + G 1 ( z , w 0 ) implies ∑ j ≥ 1 σ j / ( 1 + σ j w 0 / z ) = z ( 1 − δ 0 − 1 ) \sum_{j\geq1}\sigma_j/(1+\sigma_jw_0/z)=z(1-\delta_0^{-1}) ∑ j ≥ 1 σ j / ( 1 + σ j w 0 / z ) = z ( 1 − δ 0 − 1 ) . Subtracting n z / w 0 nz/w_0 n z / w 0 from both sides of the preceding equation and using z ( 1 − δ 0 − 1 ) − n z / w 0 = z − z ( w 0 + n δ 0 ) / ( w 0 δ 0 ) z(1-\delta_0^{-1})-nz/w_0=z-z(w_0+n\delta_0)/(w_0\delta_0) z ( 1 − δ 0 − 1 ) − n z / w 0 = z − z ( w 0 + n δ 0 ) / ( w 0 δ 0 ) yields the perturbed equation
z + ε n ( z ) = − ( w 0 n z ) − 1 + ∑ j ≥ 1 σ j 1 + σ j w 0 / z where ε n ( z ) : = − z w 0 δ 0 ( w 0 + n δ 0 ) . (75) z+\varepsilon_n(z)=-\left(\frac{w_0}{nz}\right)^{-1}+\sum_{j\geq1}\frac{\sigma_j}{1+\sigma_jw_0/z}
\qquad\text{where}\qquad
\varepsilon_n(z):=-\frac{z}{w_0\delta_0}(w_0+n\delta_0). \tag*{(75)} z + ε n ( z ) = − ( n z w 0 ) − 1 + j ≥ 1 ∑ 1 + σ j w 0 / z σ j where ε n ( z ) := − w 0 δ 0 z ( w 0 + n δ 0 ) . (75)
The residual ε n ( z ) = − z / ( w 0 δ 0 ) ⋅ ∑ j = 1 n ( δ 0 − δ j ) \varepsilon_n(z)=-z/(w_0\delta_0)\cdot\sum_{j=1}^n(\delta_0-\delta_j) ε n ( z ) = − z / ( w 0 δ 0 ) ⋅ ∑ j = 1 n ( δ 0 − δ j ) by Eq. (74) , which can be controlled by trace differences already bounded in Part III. From Eq. (73) ,
ε n ( z ) = − z w 0 ∑ j = 1 n δ j ( G ‾ j ( z , w j ) − G ‾ 1 ( z , w 0 ) ) . \varepsilon_n(z)=-\frac{z}{w_0}\sum_{j=1}^n\delta_j\left(\overline{G}_j(z,w_j)-\overline{G}_1(z,w_0)\right). ε n ( z ) = − w 0 z j = 1 ∑ n δ j ( G j ( z , w j ) − G 1 ( z , w 0 ) ) .
To bound the ratio of ∑ j ∣ δ j ∣ \sum_j|\delta_j| ∑ j ∣ δ j ∣ to ∣ w 0 ∣ |w_0| ∣ w 0 ∣ , compare each increment with δ n \delta_n δ n before summing. By Lemma E.1 .(ii), Eq. (43b), we have ∣ δ j ∣ ≤ Λ n ( z ) |\delta_j|\leq\Lambda_n(z) ∣ δ j ∣ ≤ Λ n ( z ) , and Eq. (71) bounds the trace difference with the terminal index by O ( Λ n ( z ) 15 / n ) O(\Lambda_n(z)^{15}/n) O ( Λ n ( z ) 15 / n ) . Thus
max 1 ≤ j ≤ n ∣ δ j δ n − 1 ∣ = max 1 ≤ j ≤ n ∣ δ j ( G ‾ n ( z , 0 ) − G ‾ j ( z , w j ) ) ∣ = O ( Λ n ( z ) 16 n ) ≤ 1 2 . \max_{1\leq j\leq n}\left|\frac{\delta_j}{\delta_n}-1\right|=\max_{1\leq j\leq n}|\delta_j(\overline{G}_n(z,0)-\overline{G}_j(z,w_j))|=O\left(\frac{\Lambda_n(z)^{16}}{n}\right)\leq\frac{1}{2}. 1 ≤ j ≤ n max δ n δ j − 1 = 1 ≤ j ≤ n max ∣ δ j ( G n ( z , 0 ) − G j ( z , w j )) ∣ = O ( n Λ n ( z ) 16 ) ≤ 2 1 .
The last inequality follows by choosing the implicit constant in Assumption 2. Since w 0 = − ∑ j = 1 n δ j w_0=-\sum_{j=1}^n\delta_j w 0 = − ∑ j = 1 n δ j , we have ∣ w 0 + n δ n ∣ ≤ n ∣ δ n ∣ / 2 |w_0+n\delta_n|\leq n|\delta_n|/2 ∣ w 0 + n δ n ∣ ≤ n ∣ δ n ∣/2 , so ∣ w 0 ∣ ≥ n ∣ δ n ∣ / 2 |w_0|\geq n|\delta_n|/2 ∣ w 0 ∣ ≥ n ∣ δ n ∣/2 and ∑ j = 1 n ∣ δ j ∣ ≤ 3 n ∣ δ n ∣ / 2 \sum_{j=1}^n|\delta_j|\leq3n|\delta_n|/2 ∑ j = 1 n ∣ δ j ∣ ≤ 3 n ∣ δ n ∣/2 . Hence ∑ j = 1 n ∣ δ j ∣ / ∣ w 0 ∣ ≤ 3 \sum_{j=1}^n|\delta_j|/|w_0|\leq3 ∑ j = 1 n ∣ δ j ∣/∣ w 0 ∣ ≤ 3 . Applying this bound and Eq. (72) to the preceding identity yields
∣ ε n ( z ) ∣ ≤ ∣ z ∣ ∣ w 0 ∣ ∑ j = 1 n ∣ δ j ∣ max 1 ≤ j ≤ n ∣ G ‾ j ( z , w j ) − G ‾ 1 ( z , w 0 ) ∣ = O ( ∣ z ∣ Λ n ( z ) 15 n ) = O ( ∣ z − ∣ Λ n ( z ) 16 n ) . (76) |\varepsilon_n(z)|\leq\frac{|z|}{|w_0|}\sum_{j=1}^n|\delta_j|\max_{1\leq j\leq n}|\overline{G}_j(z,w_j)-\overline{G}_1(z,w_0)|=O\left(\frac{|z|\Lambda_n(z)^{15}}{n}\right)=O\left(\frac{|z_-|\Lambda_n(z)^{16}}{n}\right). \tag*{(76)} ∣ ε n ( z ) ∣ ≤ ∣ w 0 ∣ ∣ z ∣ j = 1 ∑ n ∣ δ j ∣ 1 ≤ j ≤ n max ∣ G j ( z , w j ) − G 1 ( z , w 0 ) ∣ = O ( n ∣ z ∣ Λ n ( z ) 15 ) = O ( n ∣ z − ∣ Λ n ( z ) 16 ) . (76)
By Eq. (76) , choosing the implicit constant in n = Ω ( Λ n ( z ) 16 ) n=\Omega(\Lambda_n(z)^{16}) n = Ω ( Λ n ( z ) 16 ) sufficiently large ensures ∣ ε n ( z ) ∣ ≤ ∣ z − ∣ / 4 |\varepsilon_n(z)|\leq|z_-|/4 ∣ ε n ( z ) ∣ ≤ ∣ z − ∣/4 . For z ∈ C + z\in\mathbb{C}_+ z ∈ C + , since w 0 / ( n z ) ∈ C + w_0/(nz)\in\mathbb{C}_+ w 0 / ( n z ) ∈ C + and Σ ⪰ 0 \Sigma\succeq0 Σ ⪰ 0 is trace-class, assuming z + ε n ( z ) ∈ C + z+\varepsilon_n(z)\in\mathbb{C}_+ z + ε n ( z ) ∈ C + and applying uniqueness in Lemma A.4 to Eq. (75) , we have w 0 / ( n z ) = s n ( z + ε n ( z ) ) w_0/(nz)=s_n(z+\varepsilon_n(z)) w 0 / ( n z ) = s n ( z + ε n ( z )) and further
∣ ( z + ε n ( z ) ) − ∣ = dist ( z + ε n ( z ) , R ≥ 0 ) ≥ ∣ z − ∣ − ∣ ε n ( z ) ∣ ≥ 3 4 ∣ z − ∣ . |(z+\varepsilon_n(z))_-|=\operatorname{dist}(z+\varepsilon_n(z),\mathbb{R}_{\geq0})\geq|z_-|-|\varepsilon_n(z)|\geq\frac{3}{4}|z_-|. ∣ ( z + ε n ( z ) ) − ∣ = dist ( z + ε n ( z ) , R ≥ 0 ) ≥ ∣ z − ∣ − ∣ ε n ( z ) ∣ ≥ 4 3 ∣ z − ∣.
The fixed-point stability bound in the same lemma at z z z and z + ε n ( z ) z+\varepsilon_n(z) z + ε n ( z ) implies then
∣ w 0 n z − s n ( z ) ∣ = ∣ s n ( z + ε n ( z ) ) − s n ( z ) ∣ ≤ ∣ ε n ( z ) ∣ ∣ ( z + ε n ( z ) ) − ∣ ∣ z − ∣ ≤ 4 ∣ ε n ( z ) ∣ 3 ∣ z − ∣ 2 . \left|\frac{w_0}{nz}-s_n(z)\right|=|s_n(z+\varepsilon_n(z))-s_n(z)|\leq\frac{|\varepsilon_n(z)|}{|(z+\varepsilon_n(z))_-||z_-|}\leq\frac{4|\varepsilon_n(z)|}{3|z_-|^2}. n z w 0 − s n ( z ) = ∣ s n ( z + ε n ( z )) − s n ( z ) ∣ ≤ ∣ ( z + ε n ( z ) ) − ∣∣ z − ∣ ∣ ε n ( z ) ∣ ≤ 3∣ z − ∣ 2 4∣ ε n ( z ) ∣ .
By w 0 l i n = n z s n ( z ) w_0^{\mathrm{lin}}=nz s_n(z) w 0 lin = n z s n ( z ) , Eq. (76) , and ∣ z ∣ / ∣ z − ∣ ≤ Λ n ( z ) |z|/|z_-|\leq\Lambda_n(z) ∣ z ∣/∣ z − ∣ ≤ Λ n ( z ) , we can control the terminal path difference by
∣ w 0 − w 0 l i n ∣ = n ∣ z ∣ ∣ w 0 n z − s n ( z ) ∣ = O ( ∣ z ∣ ∣ z − ∣ Λ n ( z ) 16 ) = O ( Λ n ( z ) 17 ) . (77) |w_0-w_0^{\mathrm{lin}}|=n|z|\left|\frac{w_0}{nz}-s_n(z)\right|=O\left(\frac{|z|}{|z_-|}\Lambda_n(z)^{16}\right)=O(\Lambda_n(z)^{17}). \tag*{(77)} ∣ w 0 − w 0 lin ∣ = n ∣ z ∣ n z w 0 − s n ( z ) = O ( ∣ z − ∣ ∣ z ∣ Λ n ( z ) 16 ) = O ( Λ n ( z ) 17 ) . (77)
It thus remains only to verify z + ε n ( z ) ∈ C + z+\varepsilon_n(z)\in\mathbb{C}_+ z + ε n ( z ) ∈ C + , which can be shown by the following short proof by contradiction. Suppose that Im ( z + ε n ( z ) ) ≤ 0 \operatorname{Im}(z+\varepsilon_n(z))\leq0 Im ( z + ε n ( z )) ≤ 0 . Since η ≤ ∣ ε n ( z ) ∣ < ∣ z − ∣ \eta\leq|\varepsilon_n(z)|<|z_-| η ≤ ∣ ε n ( z ) ∣ < ∣ z − ∣ , we must have t < 0 t<0 t < 0 . Moreover, Re ( z + ε n ( z ) ) < 0 \operatorname{Re}(z+\varepsilon_n(z))<0 Re ( z + ε n ( z )) < 0 , since otherwise ∣ ε n ( z ) ∣ 2 ≥ t 2 + η 2 = ∣ z − ∣ 2 |\varepsilon_n(z)|^2\geq t^2+\eta^2=|z_-|^2 ∣ ε n ( z ) ∣ 2 ≥ t 2 + η 2 = ∣ z − ∣ 2 , yielding a contradiction. By the path signs from Part I, Re ( w 0 / ( n z ) ) = ( t Re w 0 + η Im w 0 ) / ( n ∣ z ∣ 2 ) > 0 \operatorname{Re}(w_0/(nz))=(t\operatorname{Re}w_0+\eta\operatorname{Im}w_0)/(n|z|^2)>0 Re ( w 0 / ( n z )) = ( t Re w 0 + η Im w 0 ) / ( n ∣ z ∣ 2 ) > 0 . Together with Im ( w 0 / ( n z ) ) > 0 \operatorname{Im}(w_0/(nz))>0 Im ( w 0 / ( n z )) > 0 , these signs imply
Im ( ( z + ε n ( z ) ) w 0 / ( n z ) ) = Im ( z + ε n ( z ) ) Re ( w 0 / ( n z ) ) + Re ( z + ε n ( z ) ) Im ( w 0 / ( n z ) ) < 0. \operatorname{Im}((z+\varepsilon_n(z))w_0/(nz))=\operatorname{Im}(z+\varepsilon_n(z))\operatorname{Re}(w_0/(nz))+\operatorname{Re}(z+\varepsilon_n(z))\operatorname{Im}(w_0/(nz))<0. Im (( z + ε n ( z )) w 0 / ( n z )) = Im ( z + ε n ( z )) Re ( w 0 / ( n z )) + Re ( z + ε n ( z )) Im ( w 0 / ( n z )) < 0.
However, multiplying Eq. (75) by w 0 / ( n z ) w_0/(nz) w 0 / ( n z ) and taking imaginary parts implies
Im ( ( z + ε n ( z ) ) w 0 / ( n z ) ) = Im ( w 0 / ( n z ) ) ⋅ ∑ j ≥ 1 σ j ∣ 1 + σ j w 0 / z ∣ 2 ≥ 0 , \operatorname{Im}((z+\varepsilon_n(z))w_0/(nz))=\operatorname{Im}(w_0/(nz))\cdot\sum_{j\geq1}\frac{\sigma_j}{|1+\sigma_jw_0/z|^2}\geq0, Im (( z + ε n ( z )) w 0 / ( n z )) = Im ( w 0 / ( n z )) ⋅ j ≥ 1 ∑ ∣1 + σ j w 0 / z ∣ 2 σ j ≥ 0 ,
yielding a contradiction. Thus z + ε n ( z ) ∈ C + z+\varepsilon_n(z)\in\mathbb{C}_+ z + ε n ( z ) ∈ C + , which proves Eq. (77) for z ∈ C + z\in\mathbb{C}_+ z ∈ C + .
For z = t < 0 z=t<0 z = t < 0 , we have from Part I w 0 < 0 w_0<0 w 0 < 0 and δ 0 > 0 \delta_0>0 δ 0 > 0 . Hence w 0 / ( n t ) > 0 w_0/(nt)>0 w 0 / ( n t ) > 0 and ε n ( t ) ∈ R \varepsilon_n(t)\in\mathbb{R} ε n ( t ) ∈ R . Since ∣ ε n ( t ) ∣ ≤ ∣ t ∣ / 4 |\varepsilon_n(t)|\leq|t|/4 ∣ ε n ( t ) ∣ ≤ ∣ t ∣/4 , we also have t + ε n ( t ) < 0 t+\varepsilon_n(t)<0 t + ε n ( t ) < 0 . Applying the uniqueness of the positive solution in Lemma A.4 to Eq. (75) identifies w 0 / ( n t ) = s n ( t + ε n ( t ) ) w_0/(nt)=s_n(t+\varepsilon_n(t)) w 0 / ( n t ) = s n ( t + ε n ( t )) , and subsequently, we obtain from its difference bound that
∣ w 0 − w 0 l i n ∣ = n ∣ t ∣ ∣ s n ( t + ε n ( t ) ) − s n ( t ) ∣ ≤ n ∣ ε n ( t ) ∣ ∣ t + ε n ( t ) ∣ ≤ 4 n ∣ ε n ( t ) ∣ 3 ∣ t ∣ = O ( Λ n ( t ) 16 ) , |w_0-w_0^{\mathrm{lin}}|=n|t||s_n(t+\varepsilon_n(t))-s_n(t)|\leq\frac{n|\varepsilon_n(t)|}{|t+\varepsilon_n(t)|}\leq\frac{4n|\varepsilon_n(t)|}{3|t|}=O(\Lambda_n(t)^{16}), ∣ w 0 − w 0 lin ∣ = n ∣ t ∣∣ s n ( t + ε n ( t )) − s n ( t ) ∣ ≤ ∣ t + ε n ( t ) ∣ n ∣ ε n ( t ) ∣ ≤ 3∣ t ∣ 4 n ∣ ε n ( t ) ∣ = O ( Λ n ( t ) 16 ) ,
. s c a l a b l y t y p e d .scalablytyped . sc a l ab l y t y p e d
where we use Eq. (76) and ∣ z − ∣ = ∣ t ∣ |z_-|=|t| ∣ z − ∣ = ∣ t ∣ . Since Λ n ( t ) ≥ 1 \Lambda_n(t)\ge1 Λ n ( t ) ≥ 1 , Eq. (77) also follows for z = t < 0 z=t<0 z = t < 0 .
We also transfer the terminal reciprocal bound in Eq. (69) to the constant increment δ lin ( z ) : = − w 0 lin / n = − z s n ( z ) \delta^{\operatorname{lin}}(z):=-w_0^{\operatorname{lin}}/n=-z s_n(z) δ lin ( z ) := − w 0 lin / n = − z s n ( z ) of the linear path. For both z ∈ C + z\in\mathbb{C}_+ z ∈ C + and z < 0 z<0 z < 0 , we have just identified w 0 / ( n z ) = s n ( z + ε n ( z ) ) w_0/(nz)=s_n(z+\varepsilon_n(z)) w 0 / ( n z ) = s n ( z + ε n ( z )) . Rearranging the definition of ε n ( z ) \varepsilon_n(z) ε n ( z ) in Eq. (75) gives − n / w 0 = δ 0 − 1 + ε n ( z ) / z -n/w_0=\delta_0^{-1}+\varepsilon_n(z)/z − n / w 0 = δ 0 − 1 + ε n ( z ) / z . Substitution yields the exact identity
1 δ lin ( z ) = s n ( z + ε n ( z ) ) s n ( z ) ( 1 δ 0 + ε n ( z ) z ) . (78) \frac{1}{\delta^{\operatorname{lin}}(z)}
=
\frac{s_n(z+\varepsilon_n(z))}{s_n(z)}
\left(\frac{1}{\delta_0}+\frac{\varepsilon_n(z)}{z}\right). \tag*{(78)} δ lin ( z ) 1 = s n ( z ) s n ( z + ε n ( z )) ( δ 0 1 + z ε n ( z ) ) . (78)
To bound the multiplier, write s n ( z ) = ∫ ( x − z ) − 1 μ ( d x ) s_n(z)=\int(x-z)^{-1}\mu(\mathrm{d}x) s n ( z ) = ∫ ( x − z ) − 1 μ ( d x ) as in Lemma A.4. Since ∣ ε n ( z ) ∣ ≤ ∣ z − ∣ / 4 |\varepsilon_n(z)|\le|z_-|/4 ∣ ε n ( z ) ∣ ≤ ∣ z − ∣/4 by Eq. (76) , subtracting the two Stieltjes transforms yields
∣ s n ( z + ε n ( z ) ) s n ( z ) − 1 ∣ ≤ 4 ∣ ε n ( z ) ∣ 3 ∣ s n ( z ) ∣ ∫ μ ( d x ) ∣ x − z ∣ 2 ≤ 4 ∣ ε n ( z ) ∣ 3 ∣ z − ∣ ≤ 1 3 . \left|\frac{s_n(z+\varepsilon_n(z))}{s_n(z)}-1\right|
\le
\frac{4|\varepsilon_n(z)|}{3|s_n(z)|}\int\frac{\mu(\mathrm{d}x)}{|x-z|^2}
\le
\frac{4|\varepsilon_n(z)|}{3|z_-|}
\le\frac{1}{3}. s n ( z ) s n ( z + ε n ( z )) − 1 ≤ 3∣ s n ( z ) ∣ 4∣ ε n ( z ) ∣ ∫ ∣ x − z ∣ 2 μ ( d x ) ≤ 3∣ z − ∣ 4∣ ε n ( z ) ∣ ≤ 3 1 .
For the second inequality, Im s n ( z ) = η ∫ ∣ x − z ∣ − 2 μ ( d x ) \operatorname{Im}s_n(z)=\eta\int|x-z|^{-2}\mu(\mathrm{d}x) Im s n ( z ) = η ∫ ∣ x − z ∣ − 2 μ ( d x ) suffices when t ≥ 0 t\ge0 t ≥ 0 . When t < 0 t<0 t < 0 , also use Re s n ( z ) ≥ ( − t ) ∫ ∣ x − z ∣ − 2 μ ( d x ) \operatorname{Re}s_n(z)\ge(-t)\int|x-z|^{-2}\mu(\mathrm{d}x) Re s n ( z ) ≥ ( − t ) ∫ ∣ x − z ∣ − 2 μ ( d x ) and ∣ z − ∣ = t 2 + η 2 |z_-|=\sqrt{t^2+\eta^2} ∣ z − ∣ = t 2 + η 2 . Finally, Eq. (72) gives ∣ δ 0 − 1 − ( δ n lin ) − 1 ∣ = O ( Λ n ( z ) 15 / n ) = O ( 1 ) |\delta_0^{-1}-(\delta_n^{\operatorname{lin}})^{-1}|=O(\Lambda_n(z)^{15}/n)=O(1) ∣ δ 0 − 1 − ( δ n lin ) − 1 ∣ = O ( Λ n ( z ) 15 / n ) = O ( 1 ) . Substituting into Eq. (78) and using ∣ ε n ( z ) / z ∣ ≤ 1 / 4 |\varepsilon_n(z)/z|\le1/4 ∣ ε n ( z ) / z ∣ ≤ 1/4 and Eq. (69) yields ∣ δ lin ( z ) ∣ − 1 = O ( Λ n ( z ) ) |\delta^{\operatorname{lin}}(z)|^{-1}=O(\Lambda_n(z)) ∣ δ lin ( z ) ∣ − 1 = O ( Λ n ( z )) .
Part IV-iii: Comparison of the paths. We are now ready to finish the proof of Theorem 3.(iii) for path comparisons when 1 ≤ k ≤ n 1\le k\le n 1 ≤ k ≤ n . To compare the path parameters, we first compare each increment δ j \delta_j δ j with − w 0 lin / n -w_0^{\operatorname{lin}}/n − w 0 lin / n , i.e. the constant backward increment of w k lin w_k^{\operatorname{lin}} w k lin . By Eqs. (73) , (74) , and (77) , and Λ n ( z ) ≥ 1 \Lambda_n(z)\ge1 Λ n ( z ) ≥ 1 , we have for 1 ≤ j ≤ n 1\le j\le n 1 ≤ j ≤ n ,
∣ δ j + w 0 lin n ∣ ≤ ∣ δ j − δ 0 ∣ + ∣ w 0 + n δ 0 ∣ + ∣ w 0 − w 0 lin ∣ n = O ( Λ n ( z ) 17 n ) . \left|\delta_j+\frac{w_0^{\operatorname{lin}}}{n}\right|
\le
|\delta_j-\delta_0|+
\frac{|w_0+n\delta_0|+|w_0-w_0^{\operatorname{lin}}|}{n}
=
O\left(\frac{\Lambda_n(z)^{17}}{n}\right). δ j + n w 0 lin ≤ ∣ δ j − δ 0 ∣ + n ∣ w 0 + n δ 0 ∣ + ∣ w 0 − w 0 lin ∣ = O ( n Λ n ( z ) 17 ) .
Since w k = − ∑ j = k + 1 n δ j w_k=-\sum_{j=k+1}^{n}\delta_j w k = − ∑ j = k + 1 n δ j and w k lin = ( n − k ) w 0 lin / n w_k^{\operatorname{lin}}=(n-k)w_0^{\operatorname{lin}}/n w k lin = ( n − k ) w 0 lin / n , summing from j = k + 1 j=k+1 j = k + 1 to n n n yields
∣ w k − w k lin ∣ = ∣ ∑ j = k + 1 n ( δ j + w 0 lin n ) ∣ ≤ ∑ j = k + 1 n ∣ δ j + w 0 lin n ∣ = O ( Λ n ( z ) 17 ⋅ n − k n ) . |w_k-w_k^{\operatorname{lin}}|
=
\left|\sum_{j=k+1}^{n}\left(\delta_j+\frac{w_0^{\operatorname{lin}}}{n}\right)\right|
\le
\sum_{j=k+1}^{n}\left|\delta_j+\frac{w_0^{\operatorname{lin}}}{n}\right|
=
O\left(\Lambda_n(z)^{17}\cdot\frac{n-k}{n}\right). ∣ w k − w k lin ∣ = j = k + 1 ∑ n ( δ j + n w 0 lin ) ≤ j = k + 1 ∑ n δ j + n w 0 lin = O ( Λ n ( z ) 17 ⋅ n n − k ) .
for every 0 ≤ k ≤ n 0\le k\le n 0 ≤ k ≤ n . The last display proves (iii) for z ∈ C + ∪ R < 0 z\in\mathbb{C}_+\cup\mathbb{R}_{<0} z ∈ C + ∪ R < 0 . With Parts I and III, the proof of Theorem 3 is complete.
Proof of Theorem 4
We track the random variables H k Q H_k^Q H k Q and G k Q G_k^Q G k Q along the backward path to their deterministic equivalents. We will bound the two fluctuations below, including the additional correction from w 0 w_0 w 0 to w 0 lin w_0^{\operatorname{lin}} w 0 lin .
Part I: The exact path decomposition. We first identify this decomposition in a one-step update. Each trace increment combines a rank-one sample update with a change in the parameter. We first isolate the sample update by fixing w w w , since the conditional centering of this update will produce the martingale difference. For R = R k − 1 ( z , w ) R=R_{k-1}(z,w) R = R k − 1 ( z , w ) , the Sherman–Morrison formula identifies the corresponding trace updates as
V k H , Q ( w ) : = z x k ∗ R Q R x k 1 + x k ∗ R x k and V k G , Q ( w ) : = x k ∗ R Σ 1 / 2 Q Σ 1 / 2 R x k 1 + x k ∗ R x k . (79) V_k^{H,Q}(w):=\frac{z x_k^* R Q R x_k}{1+x_k^*R x_k}
\qquad\text{and}\qquad
V_k^{G,Q}(w):=\frac{x_k^*R\Sigma^{1/2}Q\Sigma^{1/2}R x_k}{1+x_k^*R x_k}. \tag*{(79)} V k H , Q ( w ) := 1 + x k ∗ R x k z x k ∗ R QR x k and V k G , Q ( w ) := 1 + x k ∗ R x k x k ∗ R Σ 1/2 Q Σ 1/2 R x k . (79)
Because the backward path is deterministic, the sample updates can be analyzed in the forward filtration { F k } \{\mathcal{F}_k\} { F k } where x k x_k x k is independent of F k − 1 \mathcal{F}_{k-1} F k − 1 , and all path parameters are fixed. Writing δ k = w k − w k − 1 = ( 1 + G k ( z , w k ) ) − 1 \delta_k = w_k-w_{k-1}=(1+G_k(z,w_k))^{-1} δ k = w k − w k − 1 = ( 1 + G k ( z , w k ) ) − 1 , we obtain the following decomposition. Its proof is in Appendix § 13 .§ 5.1 .
Lemma E.5 (One-step trace decomposition). Under Assumption 1, along the backward path { w k } \{w_k\} { w k } ,
H k Q ( z , w k ) − H k − 1 Q ( z , w k − 1 ) = ξ k H , Q + β k H , Q , 1 ≤ k ≤ n , (80) H_k^Q(z,w_k)-H_{k-1}^Q(z,w_{k-1})=\xi_k^{H,Q}+\beta_k^{H,Q},\qquad1\leq k\leq n, \tag*{(80)} H k Q ( z , w k ) − H k − 1 Q ( z , w k − 1 ) = ξ k H , Q + β k H , Q , 1 ≤ k ≤ n , (80) G k + 1 Q ( z , w k + 1 ) − G k Q ( z , w k ) = ξ k G , Q + β k G , Q , 1 ≤ k < n , (81) G_{k+1}^Q(z,w_{k+1})-G_k^Q(z,w_k)=\xi_k^{G,Q}+\beta_k^{G,Q},\qquad1\leq k<n, \tag*{(81)} G k + 1 Q ( z , w k + 1 ) − G k Q ( z , w k ) = ξ k G , Q + β k G , Q , 1 ≤ k < n , (81) where
ξ k H , Q = E k − 1 V k H , Q ( w k ) − V k H , Q ( w k ) and ξ k G , Q = E k − 1 V k G , Q ( w k + 1 ) − V k G , Q ( w k + 1 ) , \xi_k^{H,Q}=\mathbb{E}_{k-1}V_k^{H,Q}(w_k)-V_k^{H,Q}(w_k)\qquad\text{and}\qquad\xi_k^{G,Q}=\mathbb{E}_{k-1}V_k^{G,Q}(w_{k+1})-V_k^{G,Q}(w_{k+1}), ξ k H , Q = E k − 1 V k H , Q ( w k ) − V k H , Q ( w k ) and ξ k G , Q = E k − 1 V k G , Q ( w k + 1 ) − V k G , Q ( w k + 1 ) , and
β k H , Q = z δ k Tr [ Q R k − 1 ( z , w k ) Σ R k − 1 ( z , w k − 1 ) ] − E k − 1 V k H , Q ( w k ) , \beta_k^{H,Q}=z\delta_k\operatorname{Tr}\left[QR_{k-1}(z,w_k)\Sigma R_{k-1}(z,w_{k-1})\right]-\mathbb{E}_{k-1}V_k^{H,Q}(w_k), β k H , Q = z δ k Tr [ Q R k − 1 ( z , w k ) Σ R k − 1 ( z , w k − 1 ) ] − E k − 1 V k H , Q ( w k ) , β k G , Q = δ k + 1 Tr [ Q Σ 1 / 2 R k − 1 ( z , w k + 1 ) Σ R k − 1 ( z , w k ) Σ 1 / 2 ] − E k − 1 V k G , Q ( w k + 1 ) . (82) \beta_k^{G,Q}=\delta_{k+1}\operatorname{Tr}\left[Q\Sigma^{1/2}R_{k-1}(z,w_{k+1})\Sigma R_{k-1}(z,w_k)\Sigma^{1/2}\right]-\mathbb{E}_{k-1}V_k^{G,Q}(w_{k+1}). \tag*{(82)} β k G , Q = δ k + 1 Tr [ Q Σ 1/2 R k − 1 ( z , w k + 1 ) Σ R k − 1 ( z , w k ) Σ 1/2 ] − E k − 1 V k G , Q ( w k + 1 ) . (82) Both the { ξ k H , Q } \{\xi_k^{H,Q}\} { ξ k H , Q } and { ξ k G , Q } \{\xi_k^{G,Q}\} { ξ k G , Q } sequences are martingale differences for { F k } \{\mathcal{F}_k\} { F k } , and both { β k H , Q } \{\beta_k^{H,Q}\} { β k H , Q } and { β k G , Q } \{\beta_k^{G,Q}\} { β k G , Q } sequences (drift terms) are F k − 1 \mathcal{F}_{k-1} F k − 1 -measurable.
We can now telescope from the backward initial point w 0 w_0 w 0 , and further connect to the effective spectral parameter initial point (i.e., linear interpolation) w 0 l i n w_0^{\mathrm{lin}} w 0 lin by including the additional corrections β 0 H , Q : = H 0 Q ( z , w 0 ) − H 0 Q ( z , w 0 l i n ) \beta_0^{H,Q}:=H_0^Q(z,w_0)-H_0^Q(z,w_0^{\mathrm{lin}}) β 0 H , Q := H 0 Q ( z , w 0 ) − H 0 Q ( z , w 0 lin ) and β 0 G , Q : = G 1 Q ( z , w 1 ) − G 1 Q ( z , w 0 l i n ) \beta_0^{G,Q}:=G_1^Q(z,w_1)-G_1^Q(z,w_0^{\mathrm{lin}}) β 0 G , Q := G 1 Q ( z , w 1 ) − G 1 Q ( z , w 0 lin ) , which implies
H k Q ( z , w k ) − H 0 Q ( z , w 0 l i n ) = ∑ j = 1 k ξ j H , Q + ∑ j = 0 k β j H , Q , 0 ≤ k ≤ n , (83) H_k^Q(z,w_k)-H_0^Q(z,w_0^{\mathrm{lin}})=\sum_{j=1}^{k}\xi_j^{H,Q}+\sum_{j=0}^{k}\beta_j^{H,Q},\qquad0\leq k\leq n, \tag*{(83)} H k Q ( z , w k ) − H 0 Q ( z , w 0 lin ) = j = 1 ∑ k ξ j H , Q + j = 0 ∑ k β j H , Q , 0 ≤ k ≤ n , (83)
G k Q ( z , w k ) − G 1 Q ( z , w 0 l i n ) = ∑ j = 1 k − 1 ξ j G , Q + ∑ j = 0 k − 1 β j G , Q , 1 ≤ k ≤ n . (84) G_k^Q(z,w_k)-G_1^Q(z,w_0^{\mathrm{lin}})=\sum_{j=1}^{k-1}\xi_j^{G,Q}+\sum_{j=0}^{k-1}\beta_j^{G,Q},\qquad1\leq k\leq n. \tag*{(84)} G k Q ( z , w k ) − G 1 Q ( z , w 0 lin ) = j = 1 ∑ k − 1 ξ j G , Q + j = 0 ∑ k − 1 β j G , Q , 1 ≤ k ≤ n . (84)
Part II: Controlling the two error terms. The backward path { w k } k = 0 n \{w_k\}_{k=0}^n { w k } k = 0 n allows us to apply Theorem 3 to bound the drift and martingale contributions in the exact decompositions above. The two-phase argument in its proof additionally supplies resolvent moments for all sample indices, and the endpoint comparison in Theorem 3.(iii) controls the correction from w 0 w_0 w 0 to w 0 l i n w_0^{\mathrm{lin}} w 0 lin .
Lemma E.6. Under the same setup of Theorem 4, we control the errors in Eq. (64).
(i) The martingale partial sums obey
∥ max 0 ≤ k ≤ n 1 n ∣ ∑ j = 1 k ξ j H , Q ∣ ∥ L p ∨ ∥ max 1 ≤ k ≤ n 1 n ∣ ∑ j = 1 k − 1 ξ j G , Q ∣ ∥ L p = O C x , ν , κ ( p 2 Λ n ( z ) 11 n ⋅ ( ∥ Q ∥ ∧ ∥ Q ∥ F n ) ) . (85) \left\|\max_{0\leq k\leq n}\frac{1}{n}\left|\sum_{j=1}^{k}\xi_j^{H,Q}\right|\right\|_{L_p}\vee\left\|\max_{1\leq k\leq n}\frac{1}{n}\left|\sum_{j=1}^{k-1}\xi_j^{G,Q}\right|\right\|_{L_p}
=O_{C_x,\nu,\kappa}\left(\frac{p^2\Lambda_n(z)^{11}}{n}\cdot\left(\|Q\|\wedge\frac{\|Q\|_F}{\sqrt{n}}\right)\right). \tag*{(85)} 0 ≤ k ≤ n max n 1 j = 1 ∑ k ξ j H , Q L p ∨ 1 ≤ k ≤ n max n 1 j = 1 ∑ k − 1 ξ j G , Q L p = O C x , ν , κ ( n p 2 Λ n ( z ) 11 ⋅ ( ∥ Q ∥ ∧ n ∥ Q ∥ F ) ) . (85) (ii) The accumulated drifts, including the initial corrections, obey
∥ max 0 ≤ k ≤ n 1 n ∣ ∑ j = 0 k β j H , Q ∣ ∥ L p ∨ ∥ max 1 ≤ k ≤ n 1 n ∣ ∑ j = 0 k − 1 β j G , Q ∣ ∥ L p = O C x , ν , κ ( p 2 Λ n ( z ) 23 n ⋅ ( ∥ Q ∥ ∧ ∥ Q ∥ F n ) ) . (86) \left\|\max_{0\leq k\leq n}\frac{1}{n}\left|\sum_{j=0}^{k}\beta_j^{H,Q}\right|\right\|_{L_p}\vee\left\|\max_{1\leq k\leq n}\frac{1}{n}\left|\sum_{j=0}^{k-1}\beta_j^{G,Q}\right|\right\|_{L_p}
=O_{C_x,\nu,\kappa}\left(\frac{p^2\Lambda_n(z)^{23}}{n}\cdot\left(\|Q\|\wedge\frac{\|Q\|_F}{\sqrt{n}}\right)\right). \tag*{(86)} 0 ≤ k ≤ n max n 1 j = 0 ∑ k β j H , Q L p ∨ 1 ≤ k ≤ n max n 1 j = 0 ∑ k − 1 β j G , Q L p = O C x , ν , κ ( n p 2 Λ n ( z ) 23 ⋅ ( ∥ Q ∥ ∧ n ∥ Q ∥ F ) ) . (86) The proof is in Appendix § 13 .§ 5.2 . We are now ready to closes the trace bounds in the next part, and we will obtain the additional derivative estimates by Cauchy’s formula.
Part III: Terminal traces and derivatives. Taking k = n k=n k = n in Eqs. (64a) and (64b), applying the triangle inequality and invoking Lemma E.6 implies
∥ H n Q − H 0 Q ∥ L p n ∨ ∥ G n Q ( z , 0 ) − G 1 Q ∥ L p = O ( p 2 Λ n ( z ) 23 n min { ∥ Q ∥ , ∥ Q ∥ F n } ) . \frac{\left\|H_n^Q-H_0^Q\right\|_{L^p}}{n}\vee\left\|G_n^Q(z,0)-G_1^Q\right\|_{L^p}=O\left(\frac{p^2\Lambda_n(z)^{23}}{n}\min\left\{\lVert Q\rVert,\frac{\lVert Q\rVert_F}{\sqrt{n}}\right\}\right). n H n Q − H 0 Q L p ∨ G n Q ( z , 0 ) − G 1 Q L p = O ( n p 2 Λ n ( z ) 23 min { ∥ Q ∥ , n ∥ Q ∥ F } ) .
For the last G G G update, by Sherman-Morrison at w n = 0 w_n=0 w n = 0 , it holds G n + 1 Q ( z , 0 ) − G n Q ( z , 0 ) = − V n G , Q ( 0 ) G_{n+1}^Q(z,0)-G_n^Q(z,0)=-V_n^{G,Q}(0) G n + 1 Q ( z , 0 ) − G n Q ( z , 0 ) = − V n G , Q ( 0 ) . We additionally invoke Eq. (137) from Part III of the proof of Lemma E.12 at ( s , j ) = ( n − 1 , n ) (s,j)=(n-1,n) ( s , j ) = ( n − 1 , n ) to control
∥ G n + 1 Q ( z , 0 ) − G n Q ( z , 0 ) ∥ L p = ∥ V n G , Q ( 0 ) ∥ L p = O ( Λ n ( z ) 7 n min { ∥ Q ∥ , ∥ Q ∥ F n } ) . \left\|G_{n+1}^Q(z,0)-G_n^Q(z,0)\right\|_{L^p}=\left\|V_n^{G,Q}(0)\right\|_{L^p}=O\left(\frac{\Lambda_n(z)^7}{n}\min\left\{\lVert Q\rVert,\frac{\lVert Q\rVert_F}{\sqrt{n}}\right\}\right). G n + 1 Q ( z , 0 ) − G n Q ( z , 0 ) L p = V n G , Q ( 0 ) L p = O ( n Λ n ( z ) 7 min { ∥ Q ∥ , n ∥ Q ∥ F } ) .
Adding this bound to the preceding estimate for G n Q ( z , 0 ) − G 1 Q G_n^Q(z,0)-G_1^Q G n Q ( z , 0 ) − G 1 Q proves Theorem 4 for r = 0 r=0 r = 0 .
Both terminal errors are analytic on C ∖ R ≥ 0 \mathbb{C}\setminus\mathbb{R}_{\geq0} C ∖ R ≥ 0 by the sample resolvent formulas and Eq. (30) . We can then transfer the analytic function value bounds to their derivatives when r ≥ 1 r\geq1 r ≥ 1 by Cauchy’s formula on the circle ∣ ζ − z ∣ = ∣ z − ∣ / 4 \lvert\zeta-z\rvert=\lvert z_-\rvert/4 ∣ ζ − z ∣ = ∣ z − ∣ /4 , as we have 3 ∣ z − ∣ / 4 ≤ dist ( ζ , R ≥ 0 ) ≤ 5 ∣ z − ∣ / 4 3\lvert z_-\rvert/4\leq\operatorname{dist}(\zeta,\mathbb{R}_{\geq0})\leq5\lvert z_-\rvert/4 3 ∣ z − ∣ /4 ≤ dist ( ζ , R ≥ 0 ) ≤ 5 ∣ z − ∣ /4 and ∣ ζ ∣ ≍ ∣ z ∣ \lvert\zeta\rvert\asymp\lvert z\rvert ∣ ζ ∣ ≍ ∣ z ∣ . With the same minimizing cutoff m m m , Eq. (10) of Assumption 2 guarantees Λ n ( ζ ) = O ( Λ n ( z ) ) \Lambda_n(\zeta)=O(\Lambda_n(z)) Λ n ( ζ ) = O ( Λ n ( z )) for Im ζ ≥ 0 \operatorname{Im}\zeta\geq0 Im ζ ≥ 0 , and 3 ∣ z − ∣ / 4 ≥ n − ( ν + 1 ) 3\lvert z_-\rvert/4\geq n^{-(\nu+1)} 3 ∣ z − ∣ /4 ≥ n − ( ν + 1 ) for sufficiently large n n n . Thus the r = 0 r=0 r = 0 bound applies uniformly on the circle, by conjugation with Q ∗ Q^* Q ∗ when Im ζ < 0 \operatorname{Im}\zeta<0 Im ζ < 0 . Applying Cauchy’s formula and Minkowski’s inequality for r ≥ 1 r\geq1 r ≥ 1 gives
1 n ∥ d r d z r ( H n Q − H 0 Q ) ∥ L p ≤ r ! 2 π n ∫ ∣ ζ − z ∣ = ∣ z − ∣ / 4 ∥ H n Q ( ζ , 0 ) − H 0 Q ( ζ , w 0 l i n ( ζ ) ) ∥ L p ∣ ζ − z ∣ r + 1 ∣ d ζ ∣ ≤ r ! ( 4 ∣ z − ∣ ) r sup ∣ ζ − z ∣ = ∣ z − ∣ / 4 ∥ H n Q ( ζ , 0 ) − H 0 Q ( ζ , w 0 l i n ( ζ ) ) ∥ L p n . \begin{aligned}
\frac{1}{n}\left\|\frac{\mathrm{d}^r}{\mathrm{d}z^r}\left(H_n^Q-H_0^Q\right)\right\|_{L^p}\leq\frac{r!}{2\pi n}\int_{\lvert\zeta-z\rvert=\lvert z_-\rvert/4}\frac{\left\|H_n^Q(\zeta,0)-H_0^Q(\zeta,w_0^{\mathrm{lin}}(\zeta))\right\|_{L^p}}{\lvert\zeta-z\rvert^{r+1}}\,\lvert\mathrm{d}\zeta\rvert\\
&\leq r!\left(\frac{4}{\lvert z_-\rvert}\right)^r\sup_{\lvert\zeta-z\rvert=\lvert z_-\rvert/4}\frac{\left\|H_n^Q(\zeta,0)-H_0^Q(\zeta,w_0^{\mathrm{lin}}(\zeta))\right\|_{L^p}}{n}.
\end{aligned} n 1 d z r d r ( H n Q − H 0 Q ) L p ≤ 2 π n r ! ∫ ∣ ζ − z ∣ = ∣ z − ∣ /4 ∣ ζ − z ∣ r + 1 H n Q ( ζ , 0 ) − H 0 Q ( ζ , w 0 lin ( ζ )) L p ∣ d ζ ∣ ≤ r ! ( ∣ z − ∣ 4 ) r ∣ ζ − z ∣ = ∣ z − ∣ /4 sup n H n Q ( ζ , 0 ) − H 0 Q ( ζ , w 0 lin ( ζ )) L p .
The same argument applies to G n + 1 Q ( ζ , 0 ) − G 1 Q ( ζ , w 0 l i n ( ζ ) ) G_{n+1}^Q(\zeta,0)-G_1^Q(\zeta,w_0^{\mathrm{lin}}(\zeta)) G n + 1 Q ( ζ , 0 ) − G 1 Q ( ζ , w 0 lin ( ζ )) with the same multiplicative factor r ! ( 4 / ∣ z − ∣ ) r r!(4/\lvert z_-\rvert)^r r ! ( 4/ ∣ z − ∣ ) r . Substituting the r = 0 r=0 r = 0 bounds on the circle completes the proof.
Remark 7 (Derivative estimates). It is easy to control the derivatives by Cauchy’s formula, as we allow the spectral parameter to take complex values from the r = 0 r=0 r = 0 bound in Theorem 4. For comparison, to estimate a first derivative from values on the real line, Cheng and Montanari [28 ] expand the function at equally spaced points and invert the resulting Vandermonde matrix. To bound the higher derivatives, they apply Faà di Bruno’s formula to the composition of the deterministic trace with the implicit spectral parameter. Here Cauchy’s formula gives all derivative orders directly from the r = 0 r=0 r = 0 bound.
Proof of Theorem 5
Let χ Q ( z , z ′ ) \chi^Q(z,z') χ Q ( z , z ′ ) denote the mixed-resolvent trace difference on the left-hand side of Eq. (32) at r = r ′ = 0 r=r'=0 r = r ′ = 0 . Throughout the proof, we work by taking ∥ Q ∥ F = n \lVert Q\rVert_F=\sqrt{n} ∥ Q ∥ F = n without loss of generality, by linearity in Q Q Q . We begin by establishing concentration of Tr ( Q R ~ n R ~ n ′ ) \operatorname{Tr}(Q\widetilde{R}_n\widetilde{R}'_n) Tr ( Q R n R n ′ ) around its deterministic equivalent at zeroth order, i.e., for r = r ′ = 0 r=r'=0 r = r ′ = 0 , and subsequently extend the result to arbitrary derivative orders using Cauchy’s integral formula. We extend our shorthand notations for R ~ n − 1 = R ~ n − 1 ( z , 0 ) \widetilde{R}_{n-1}=\widetilde{R}_{n-1}(z,0) R n − 1 = R n − 1 ( z , 0 ) , R ~ n − 1 ′ = R ~ n − 1 ( z ′ , 0 ) \widetilde{R}'_{n-1}=\widetilde{R}_{n-1}(z',0) R n − 1 ′ = R n − 1 ( z ′ , 0 ) and δ k = δ k ( z ) \delta_k=\delta_k(z) δ k = δ k ( z ) , δ k ′ = δ k ( z ′ ) \delta'_k=\delta_k(z') δ k ′ = δ k ( z ′ ) (cf. Eq. (47)), and additionally define δ k l i n ( z ) = − w 0 l i n ( z ) / n = z / ( n ω ( z / n ) ) \delta_k^{\mathrm{lin}}(z)=-w_0^{\mathrm{lin}}(z)/n=z/(n\omega(z/n)) δ k lin ( z ) = − w 0 lin ( z ) / n = z / ( nω ( z / n )) and similarly for δ k l i n ( z ′ ) \delta_k^{\mathrm{lin}}(z') δ k lin ( z ′ ) . We are now ready to first explain the heuristic and outline of the proof below.
Tr ( Q R ~ n R ~ n ′ ) ≈ E Tr ( Q R ~ n − 1 R ~ n − 1 ′ ) ≈ z E G n + 1 Q ( z , 0 ) − z ′ E G n + 1 Q ( z ′ , 0 ) n δ n δ n ′ ( z − z ′ ) (Part I: LOO concentration) ≈ 1 δ l i n ( z ) δ l i n ( z ′ ) z G 1 Q ( z , w 0 l i n ( z ) ) − z ′ G 1 Q ( z ′ , w 0 l i n ( z ′ ) ) n ( z − z ′ ) \begin{aligned}
\operatorname{Tr}(Q\widetilde{R}_n\widetilde{R}'_n)\approx\mathbb{E}\operatorname{Tr}(Q\widetilde{R}_{n-1}\widetilde{R}'_{n-1})\approx\frac{z\mathbb{E}G_{n+1}^Q(z,0)-z'\mathbb{E}G_{n+1}^Q(z',0)}{n\delta_n\delta'_n(z-z')}
\qquad\text{(Part I: LOO concentration)} \\
&\approx\frac{1}{\delta^{\mathrm{lin}}(z)\delta^{\mathrm{lin}}(z')} \frac{zG^{Q}_{1}(z,w^{\mathrm{lin}}_{0}(z))-z'G^{Q}_{1}(z',w^{\mathrm{lin}}_{0}(z'))}{n(z-z')}
\end{aligned} Tr ( Q R n R n ′ ) ≈ E Tr ( Q R n − 1 R n − 1 ′ ) ≈ n δ n δ n ′ ( z − z ′ ) z E G n + 1 Q ( z , 0 ) − z ′ E G n + 1 Q ( z ′ , 0 ) (Part I: LOO concentration) ≈ δ lin ( z ) δ lin ( z ′ ) 1 n ( z − z ′ ) z G 1 Q ( z , w 0 lin ( z )) − z ′ G 1 Q ( z ′ , w 0 lin ( z ′ ))
(Part II: Stability by path)
= ω ( z / n ) − ω ( z ′ / n ) ( z − z ′ ) / n Tr ( Q R ~ 0 R ~ 0 ′ ) . = \frac{\omega(z/n)-\omega(z'/n)}{(z-z')/n}\operatorname{Tr}(Q\widetilde{R}_{0}\widetilde{R}'_{0}). = ( z − z ′ ) / n ω ( z / n ) − ω ( z ′ / n ) Tr ( Q R 0 R 0 ′ ) .
(Exact resolvent identity)
In the first line of approximation, we will use leave-one-out on the terminal sample covariance matrix T n T_n T n and concentration by Hanson-Wright, whose details will be in Part I of the proof; in Part II, we establish the second line of approximation, where path and single resolvent stability allow us to rewrite the previous step in its deterministic equivalent form at the zero sample point; finally, in Part III (as well as the reduction within the steps of Part I), we make use of the exact mixed-resolvent identity expressed by single resolvent divided differences to close the proof. Concretely, we will prove by adding the errors in the first two lines of the roadmap that
∥ χ Q ( z , z ′ ) ∥ L p ≤ O ( p 3 / 2 ( Λ n ∨ Λ n ′ ) 13 + ( Λ n ∨ Λ n ′ ) 16 n 2 ) + O ( ( Λ n ∨ Λ n ′ ) 26 n 2 ) . \lVert\chi^{Q}(z,z')\rVert_{L_p} \le O\left(\frac{p^{3/2}(\Lambda_n\vee\Lambda'_n)^{13}+(\Lambda_n\vee\Lambda'_n)^{16}}{n^2}\right)+O\left(\frac{(\Lambda_n\vee\Lambda'_n)^{26}}{n^2}\right). ∥ χ Q ( z , z ′ ) ∥ L p ≤ O ( n 2 p 3/2 ( Λ n ∨ Λ n ′ ) 13 + ( Λ n ∨ Λ n ′ ) 16 ) + O ( n 2 ( Λ n ∨ Λ n ′ ) 26 ) .
For the third line, we have the exact identity formulas below, reducing our remaining work to establishing the above inequality.
Lemma E.7. Under the setup of Theorem 5, the sample and zero-sample endpoints satisfy
z G n + 1 Q ( z , 0 ) − z ′ G n + 1 Q ( z ′ , 0 ) z − z ′ = Tr ( Q Σ 1 / 2 R n ( z , 0 ) T n R n ( z ′ , 0 ) Σ 1 / 2 ) , (87) \frac{zG^{Q}_{n+1}(z,0)-z'G^{Q}_{n+1}(z',0)}{z-z'}=\operatorname{Tr}\left(Q\Sigma^{1/2}R_n(z,0)T_nR_n(z',0)\Sigma^{1/2}\right), \tag*{(87)} z − z ′ z G n + 1 Q ( z , 0 ) − z ′ G n + 1 Q ( z ′ , 0 ) = Tr ( Q Σ 1/2 R n ( z , 0 ) T n R n ( z ′ , 0 ) Σ 1/2 ) , (87) z G 1 Q ( z , w 0 l i n ( z ) ) − z ′ G 1 Q ( z ′ , w 0 l i n ( z ′ ) ) z − z ′ = z z ′ ( ω ( z / n ) − ω ( z ′ / n ) ) ( z − z ′ ) ω ( z / n ) ω ( z ′ / n ) Tr ( Q R ~ 0 R ~ 0 ′ ) . (88) \frac{zG^{Q}_{1}(z,w^{\mathrm{lin}}_{0}(z))-z'G^{Q}_{1}(z',w^{\mathrm{lin}}_{0}(z'))}{z-z'}=\frac{zz'(\omega(z/n)-\omega(z'/n))}{(z-z')\omega(z/n)\omega(z'/n)}\operatorname{Tr}(Q\widetilde{R}_{0}\widetilde{R}'_{0}). \tag*{(88)} z − z ′ z G 1 Q ( z , w 0 lin ( z )) − z ′ G 1 Q ( z ′ , w 0 lin ( z ′ )) = ( z − z ′ ) ω ( z / n ) ω ( z ′ / n ) z z ′ ( ω ( z / n ) − ω ( z ′ / n )) Tr ( Q R 0 R 0 ′ ) . (88) At z = z ′ z=z' z = z ′ the divided differences are evaluated by differentiation.
The proof is in Appendix § 13 .6.1. Note that δ l i n ( z ) = z / ( n ω ( z / n ) ) \delta^{\mathrm{lin}}(z)=z/(n\omega(z/n)) δ lin ( z ) = z / ( nω ( z / n )) , and multiplying Eq. (66b) of Lemma E.7 by 1 / ( n δ l i n ( z ) δ l i n ( z ′ ) ) = n ω ( z / n ) ω ( z ′ / n ) / ( z z ′ ) 1/(n\delta^{\mathrm{lin}}(z)\delta^{\mathrm{lin}}(z'))=n\omega(z/n)\omega(z'/n)/(zz') 1/ ( n δ lin ( z ) δ lin ( z ′ )) = nω ( z / n ) ω ( z ′ / n ) / ( z z ′ ) leaves the prefactor ( ω ( z / n ) − ω ( z ′ / n ) ) / ( ( z − z ′ ) / n ) (\omega(z/n)-\omega(z'/n))/((z-z')/n) ( ω ( z / n ) − ω ( z ′ / n )) / (( z − z ′ ) / n ) . This yields the final equality in the roadmap. We now focus on its first line of approximation.
Part I: Leave-one-out concentration at the terminal sample. As promised, we study the errors in the two leave-one-out approximations outlined at the opening of the proof through
χ 1 : = Tr ( Q R ~ n R ~ n ′ ) − E Tr ( Q R ~ n − 1 R ~ n − 1 ′ ) , \chi_1:=\operatorname{Tr}(Q\widetilde{R}_n\widetilde{R}'_n)-\mathbb{E}\operatorname{Tr}(Q\widetilde{R}_{n-1}\widetilde{R}'_{n-1}), χ 1 := Tr ( Q R n R n ′ ) − E Tr ( Q R n − 1 R n − 1 ′ ) ,
χ 2 : = E Tr ( Q R ~ n − 1 R ~ n − 1 ′ ) − z E G n + 1 Q ( z , 0 ) − z ′ E G n + 1 Q ( z ′ , 0 ) n δ n δ n ′ ( z − z ′ ) . \chi_2:=\mathbb{E}\operatorname{Tr}(Q\widetilde{R}_{n-1}\widetilde{R}'_{n-1})-\frac{z\mathbb{E}G^{Q}_{n+1}(z,0)-z'\mathbb{E}G^{Q}_{n+1}(z',0)}{n\delta_n\delta'_n(z-z')}. χ 2 := E Tr ( Q R n − 1 R n − 1 ′ ) − n δ n δ n ′ ( z − z ′ ) z E G n + 1 Q ( z , 0 ) − z ′ E G n + 1 Q ( z ′ , 0 ) .
The first residual error χ 1 \chi_1 χ 1 compares the full mixed trace with the expected trace after removing one observation. For the second residual χ 2 \chi_2 χ 2 , taking expectation in Eq. (66a) of Lemma E.7 , and expanding T n = ∑ i = 1 n x i x i ∗ T_n=\sum_{i=1}^{n}x_ix_i^* T n = ∑ i = 1 n x i x i ∗ gives n n n equal expected contributions. After removing x n x_n x n , the row numerator has conditional mean Tr ( Q R ~ n − 1 R ~ n − 1 ′ ) \operatorname{Tr}(Q\widetilde{R}_{n-1}\widetilde{R}'_{n-1}) Tr ( Q R n − 1 R n − 1 ′ ) given F n − 1 \mathcal{F}_{n-1} F n − 1 . Thus χ 2 \chi_2 χ 2 measures the error of replacing the two row reciprocals by δ n \delta_n δ n and δ n ′ \delta'_n δ n ′ in expectation. We explicitly bound the terms below, and defer these detailed proofs to Appendix § 13 .6.2.
Lemma E.8. Under the setup within the proof, we have
∥ χ 1 ∥ L p = O ( p 3 / 2 ( Λ n ∨ Λ n ′ ) 13 n 2 ) and ∣ χ 2 ∣ = O ( ( Λ n ∨ Λ n ′ ) 16 n 2 ) . \lVert\chi_1\rVert_{L_p}=O\left(\frac{p^{3/2}(\Lambda_n\vee\Lambda'_n)^{13}}{n^2}\right)
\qquad\text{and}\qquad
|\chi_2|=O\left(\frac{(\Lambda_n\vee\Lambda'_n)^{16}}{n^2}\right). ∥ χ 1 ∥ L p = O ( n 2 p 3/2 ( Λ n ∨ Λ n ′ ) 13 ) and ∣ χ 2 ∣ = O ( n 2 ( Λ n ∨ Λ n ′ ) 16 ) .
Part II: Stability of the increments and single traces. We turn to the second line of the roadmap by first replacing the terminal increments by their linear values while keeping the terminal traces fixed. We then replace the expected terminal traces by their zero-sample values with the linear increments fixed. The corresponding residuals are
χ 3 ≔ ( 1 δ n δ n ′ − 1 δ l i n ( z ) δ l i n ( z ′ ) ) ⋅ z E G n + 1 Q ( z , 0 ) − z ′ E G n + 1 Q ( z ′ , 0 ) n ( z − z ′ ) . \chi_{3} \coloneqq\left(\frac{1}{\delta_{n}\delta_{n}^{\prime}}-\frac{1}{\delta^{\mathrm{lin}}(z)\delta^{\mathrm{lin}}(z^{\prime})}\right)\cdot\frac{z\mathbb{E}G_{n+1}^{Q}(z,0)-z^{\prime}\mathbb{E}G_{n+1}^{Q}(z^{\prime},0)}{n(z-z^{\prime})}. χ 3 : = ( δ n δ n ′ 1 − δ lin ( z ) δ lin ( z ′ ) 1 ) ⋅ n ( z − z ′ ) z E G n + 1 Q ( z , 0 ) − z ′ E G n + 1 Q ( z ′ , 0 ) .
χ 4 ≔ z ( E G n + 1 Q ( z , 0 ) − G 1 Q ( z , w 0 l i n ( z ) ) ) − z ′ ( E G n + 1 Q ( z ′ , 0 ) − G 1 Q ( z ′ , w 0 l i n ( z ′ ) ) ) n δ l i n ( z ) δ l i n ( z ′ ) ( z − z ′ ) . \chi_{4} \coloneqq\frac{z\left(\mathbb{E}G_{n+1}^{Q}(z,0)-G_{1}^{Q}(z,w_{0}^{\mathrm{lin}}(z))\right)-z^{\prime}\left(\mathbb{E}G_{n+1}^{Q}(z^{\prime},0)-G_{1}^{Q}(z^{\prime},w_{0}^{\mathrm{lin}}(z^{\prime}))\right)}{n\delta^{\mathrm{lin}}(z)\delta^{\mathrm{lin}}(z^{\prime})(z-z^{\prime})}. χ 4 : = n δ lin ( z ) δ lin ( z ′ ) ( z − z ′ ) z ( E G n + 1 Q ( z , 0 ) − G 1 Q ( z , w 0 lin ( z )) ) − z ′ ( E G n + 1 Q ( z ′ , 0 ) − G 1 Q ( z ′ , w 0 lin ( z ′ )) ) .
The next lemma controls the terms above, whose proof details are in Appendix E.6.3.
Lemma E.9. Under the setup within the proof, we have
∣ χ 3 ∣ = O ( ( Λ n ∨ Λ n ′ ) 24 n 2 ) and ∣ χ 4 ∣ = O ( ( Λ n ∨ Λ n ′ ) 26 n 2 ) . \lvert\chi_{3}\rvert=O\left(\frac{(\Lambda_{n}\vee\Lambda_{n}^{\prime})^{24}}{n^{2}}\right)\quad\text{and}\quad\lvert\chi_{4}\rvert=O\left(\frac{(\Lambda_{n}\vee\Lambda_{n}^{\prime})^{26}}{n^{2}}\right). ∣ χ 3 ∣ = O ( n 2 ( Λ n ∨ Λ n ′ ) 24 ) and ∣ χ 4 ∣ = O ( n 2 ( Λ n ∨ Λ n ′ ) 26 ) .
Identifying χ Q ( z , z ′ ) = χ 1 + χ 2 + χ 3 + χ 4 \chi^{Q}(z,z^{\prime})=\chi_{1}+\chi_{2}+\chi_{3}+\chi_{4} χ Q ( z , z ′ ) = χ 1 + χ 2 + χ 3 + χ 4 , we combine Lemmas E.8 and E.9 with Minkowski’s inequality and restore the factor ∥ Q ∥ F / n \lVert Q\rVert_{F}/\sqrt{n} ∥ Q ∥ F / n by linearity to obtain Eq. (32) for r = r ′ = 0 r=r^{\prime}=0 r = r ′ = 0 ,
∥ χ Q ( z , z ′ ) ∥ L p ≤ ∥ χ 1 ∥ L p + ∣ χ 2 ∣ + ∣ χ 3 ∣ + ∣ χ 4 ∣ = O ( p 3 / 2 ∥ Q ∥ F ( Λ n ∨ Λ n ′ ) 26 n 5 / 2 ) . (89) \left\lVert\chi^{Q}(z,z^{\prime})\right\rVert_{L^{p}}\leq\left\lVert\chi_{1}\right\rVert_{L^{p}}+\lvert\chi_{2}\rvert+\lvert\chi_{3}\rvert+\lvert\chi_{4}\rvert=O\left(\frac{p^{3/2}\lVert Q\rVert_{F}(\Lambda_{n}\vee\Lambda_{n}^{\prime})^{26}}{n^{5/2}}\right). \tag*{(89)} χ Q ( z , z ′ ) L p ≤ ∥ χ 1 ∥ L p + ∣ χ 2 ∣ + ∣ χ 3 ∣ + ∣ χ 4 ∣ = O ( n 5/2 p 3/2 ∥ Q ∥ F ( Λ n ∨ Λ n ′ ) 26 ) . (89)
Part III: Derivatives in the two spectral parameters. Let Γ z \Gamma_{z} Γ z and Γ z ′ \Gamma_{z^{\prime}} Γ z ′ be the positively oriented circles centered at z z z and z ′ z^{\prime} z ′ with radii ∣ z − ∣ / 4 \lvert z_{-}\rvert/4 ∣ z − ∣ /4 and ∣ z − ′ ∣ / 4 \lvert z_{-}^{\prime}\rvert/4 ∣ z − ′ ∣ /4 , respectively. We can then apply the same circle arguments from Part III of the proof of Theorem 4 in each variable, which enables us to use Eq. (89) on Γ z × Γ z ′ \Gamma_{z}\times\Gamma_{z^{\prime}} Γ z × Γ z ′ (taking adjoints for reflected resolvents when needed). The trace difference is analytic in each variable, with the divided difference extended by continuity on the diagonal. Applying Cauchy’s formula on the two circles and then Minkowski’s inequality, we have
∥ d r + r ′ d z r d ( z ′ ) r ′ χ Q ( z , z ′ ) ∥ L p ≤ r ! r ′ ! ( 2 π ) 2 ∥ ∬ Γ z × Γ z ′ χ Q ( ζ , ζ ′ ) ( ζ − z ) r + 1 ( ζ ′ − z ′ ) r ′ + 1 d ζ ′ d ζ ∥ L p ≤ r ! r ′ ! ( 4 ∣ z − ∣ ) r ( 4 ∣ z − ′ ∣ ) r ′ sup Γ z × Γ z ′ ∥ χ Q ( ζ , ζ ′ ) ∥ L p . \begin{aligned}
\left\lVert\frac{\mathrm{d}^{r+r^{\prime}}}{\mathrm{d}z^{r}\,\mathrm{d}(z^{\prime})^{r^{\prime}}}\chi^{Q}(z,z^{\prime})\right\rVert_{L^{p}}\leq\frac{r!\,r^{\prime}!}{(2\pi)^{2}}\left\lVert\iint_{\Gamma_{z}\times\Gamma_{z^{\prime}}}\frac{\chi^{Q}(\zeta,\zeta^{\prime})}{(\zeta-z)^{r+1}(\zeta^{\prime}-z^{\prime})^{r^{\prime}+1}}\,\mathrm{d}\zeta^{\prime}\,\mathrm{d}\zeta\right\rVert_{L^{p}} \\
&\leq r!\,r^{\prime}!\left(\frac{4}{\lvert z_{-}\rvert}\right)^{r}\left(\frac{4}{\lvert z_{-}^{\prime}\rvert}\right)^{r^{\prime}}\sup_{\Gamma_{z}\times\Gamma_{z^{\prime}}}\left\lVert\chi^{Q}(\zeta,\zeta^{\prime})\right\rVert_{L^{p}}.
\end{aligned} d z r d ( z ′ ) r ′ d r + r ′ χ Q ( z , z ′ ) L p ≤ ( 2 π ) 2 r ! r ′ ! ∬ Γ z × Γ z ′ ( ζ − z ) r + 1 ( ζ ′ − z ′ ) r ′ + 1 χ Q ( ζ , ζ ′ ) d ζ ′ d ζ L p ≤ r ! r ′ ! ( ∣ z − ∣ 4 ) r ( ∣ z − ′ ∣ 4 ) r ′ Γ z × Γ z ′ sup χ Q ( ζ , ζ ′ ) L p .
Substituting the result for r = r ′ = 0 r=r^{\prime}=0 r = r ′ = 0 into the preceding bound completes the proof for any fixed order of mixed derivatives.
Auxiliary proofs for Theorem 3
Proof of Lemma E.1
We will first prove the path-dependent bounds in (i) and (ii) at fixed parameters satisfying Im ( w z ‾ ) ≥ 0 \operatorname{Im}(w\overline{z})\geq0 Im ( w z ) ≥ 0 (or w ≤ 0 w\leq0 w ≤ 0 when z < 0 z<0 z < 0 ) in the first two parts, and then verify these conditions along the backward path (as well as its well-posedness) in the proof of (iii). Recall that z = t + i η z=t+\mathrm{i}\eta z = t + i η , w = u + i v w=u+\mathrm{i}v w = u + i v and therefore Im ( w z ‾ ) = t v − η u \operatorname{Im}(w\overline{z})=tv-\eta u Im ( w z ) = t v − η u .
Part I: Resolvent bounds in (i). We first prove the existence and the bound ∥ R j − 1 ( z , w ) ∥ ≤ η − 1 \lVert R_{j-1}(z,w)\rVert\le\eta^{-1} ∥ R j − 1 ( z , w )∥ ≤ η − 1 at a general fixed ( z , w ) ∈ D (z,w) \in D ( z , w ) ∈ D , where η > 0 \eta> 0 η > 0 . Fixing j ∈ [ n ] j \in[n] j ∈ [ n ] and using the shorthand M j − 1 = M j − 1 ( z , w ) ≔ T j − 1 − w Σ − z I M_{j-1}=M_{j-1}(z,w) \coloneqq T_{j-1}-w\Sigma-zI M j − 1 = M j − 1 ( z , w ) : = T j − 1 − w Σ − z I and R j − 1 = R j − 1 ( z , w ) R_{j-1}=R_{j-1}(z,w) R j − 1 = R j − 1 ( z , w ) whenever the inverse exists, we have for h ∈ H h \in\mathcal{H} h ∈ H ,
− Im ⟨ h , M j − 1 h ⟩ H = η ∥ h ∥ H 2 + v ∥ h ∥ Σ 2 ≥ η ∥ h ∥ H 2 . -\operatorname{Im}\langle h,M_{j-1}h\rangle_{\mathcal{H}}=\eta\lVert h\rVert_{\mathcal{H}}^{2}+v\lVert h\rVert_{\Sigma}^{2}\ge\eta\lVert h\rVert_{\mathcal{H}}^{2}. − Im ⟨ h , M j − 1 h ⟩ H = η ∥ h ∥ H 2 + v ∥ h ∥ Σ 2 ≥ η ∥ h ∥ H 2 .
It follows that ∥ M j − 1 h ∥ H ≥ η ∥ h ∥ H \lVert M_{j-1}h\rVert_{\mathcal{H}}\ge\eta\lVert h\rVert_{\mathcal{H}} ∥ M j − 1 h ∥ H ≥ η ∥ h ∥ H . Hence, M j − 1 M_{j-1} M j − 1 is bounded below by η \eta η and, in particular, is injective and has closed range. Applying the same argument to its adjoint implies M j − 1 ∗ M_{j-1}^{*} M j − 1 ∗ has a trivial kernel, and by Hilbert space duality, M j − 1 M_{j-1} M j − 1 has a dense range that equals H \mathcal{H} H . Thus the inverse exists and ∥ R j − 1 ∥ ≤ η − 1 \lVert R_{j-1}\rVert\le\eta^{-1} ∥ R j − 1 ∥ ≤ η − 1 . For the path-dependent results, we shall assume Im ( w z ‾ ) ≥ 0 \operatorname{Im}(w\overline{z})\ge0 Im ( w z ) ≥ 0 and prove the two bounds in (i) at this fixed parameter w w w , as is mentioned in the proof outline.
We begin with proving the resolvent norm bound ∥ R j − 1 ∥ ≤ ∣ z − ∣ − 1 \lVert R_{j-1}\rVert\le\lvert z_{-}\rvert^{-1} ∥ R j − 1 ∥ ≤ ∣ z − ∣ − 1 . When t ≥ 0 t\ge0 t ≥ 0 , this is implied by the previous general bound ∥ R j − 1 ∥ ≤ η − 1 \lVert R_{j-1}\rVert\le\eta^{-1} ∥ R j − 1 ∥ ≤ η − 1 as z − = i η z_{-}=i\eta z − = i η . When t < 0 t<0 t < 0 , this condition implies u ≤ t v / η ≤ 0 u\le tv/\eta\le0 u ≤ t v / η ≤ 0 . Consequently, applying Cauchy-Schwarz for every h ∈ H h\in\mathcal{H} h ∈ H ,
∥ M j − 1 h ∥ H 2 ∥ h ∥ H 2 ≥ ∣ ⟨ h , M j − 1 h ⟩ H ∣ 2 = ( Re ⟨ h , M j − 1 h ⟩ H ) 2 + ( Im ⟨ h , M j − 1 h ⟩ H ) 2 ≥ ( t 2 + η 2 ) ∥ h ∥ H 4 . \lVert M_{j-1}h\rVert_{\mathcal{H}}^{2}\lVert h\rVert_{\mathcal{H}}^{2}\ge\lvert\langle h,M_{j-1}h\rangle_{\mathcal{H}}\rvert^{2}=\left(\operatorname{Re}\langle h,M_{j-1}h\rangle_{\mathcal{H}}\right)^{2}+\left(\operatorname{Im}\langle h,M_{j-1}h\rangle_{\mathcal{H}}\right)^{2}\ge(t^{2}+\eta^{2})\lVert h\rVert_{\mathcal{H}}^{4}. ∥ M j − 1 h ∥ H 2 ∥ h ∥ H 2 ≥ ∣⟨ h , M j − 1 h ⟩ H ∣ 2 = ( Re ⟨ h , M j − 1 h ⟩ H ) 2 + ( Im ⟨ h , M j − 1 h ⟩ H ) 2 ≥ ( t 2 + η 2 ) ∥ h ∥ H 4 .
In the last step, we additionally make use of the parameter condition u ≤ 0 u\le0 u ≤ 0 for the real part, i.e., Re ⟨ h , M j − 1 h ⟩ H = ⟨ h , T j − 1 h ⟩ H − u ∥ Σ 1 / 2 h ∥ H 2 − t ∥ h ∥ H 2 ≥ ( − t ) ∥ h ∥ H 2 \operatorname{Re}\langle h,M_{j-1}h\rangle_{\mathcal{H}}=\langle h,T_{j-1}h\rangle_{\mathcal{H}}-u\lVert\Sigma^{1/2}h\rVert_{\mathcal{H}}^{2}-t\lVert h\rVert_{\mathcal{H}}^{2}\ge(-t)\lVert h\rVert_{\mathcal{H}}^{2} Re ⟨ h , M j − 1 h ⟩ H = ⟨ h , T j − 1 h ⟩ H − u ∥ Σ 1/2 h ∥ H 2 − t ∥ h ∥ H 2 ≥ ( − t ) ∥ h ∥ H 2 . Hence, ∥ M j − 1 h ∥ H ≥ ∣ z − ∣ ∥ h ∥ H \lVert M_{j-1}h\rVert_{\mathcal{H}}\ge\lvert z_{-}\rvert\lVert h\rVert_{\mathcal{H}} ∥ M j − 1 h ∥ H ≥ ∣ z − ∣ ∥ h ∥ H , and using the inverse already established concludes with ∥ R j − 1 ∥ ≤ ∣ z ∣ − 1 = ∣ z − ∣ − 1 \lVert R_{j-1}\rVert\le\lvert z\rvert^{-1}=\lvert z_{-}\rvert^{-1} ∥ R j − 1 ∥ ≤ ∣ z ∣ − 1 = ∣ z − ∣ − 1 .
Next, we establish the quadratic-form bound by exploiting the positive semidefinite property of Im ( z R j − 1 ) \operatorname{Im}(zR_{j-1}) Im ( z R j − 1 ) . Since M j − 1 R j − 1 = I M_{j-1}R_{j-1}=I M j − 1 R j − 1 = I and R j − 1 ∗ M j − 1 ∗ = I R_{j-1}^{*}M_{j-1}^{*}=I R j − 1 ∗ M j − 1 ∗ = I , we can write
Im ( z R j − 1 ) = R j − 1 ∗ ⋅ z M j − 1 ∗ − z ‾ M j − 1 2 i ⋅ R j − 1 = R j − 1 ∗ ( η T j − 1 + Im ( w z ‾ ) Σ ) R j − 1 . (90) \operatorname{Im}(zR_{j-1})=R_{j-1}^{*}\cdot\frac{zM_{j-1}^{*}-\overline{z}M_{j-1}}{2i}\cdot R_{j-1}=R_{j-1}^{*}\left(\eta T_{j-1}+\operatorname{Im}(w\overline{z})\Sigma\right)R_{j-1}. \tag*{(90)} Im ( z R j − 1 ) = R j − 1 ∗ ⋅ 2 i z M j − 1 ∗ − z M j − 1 ⋅ R j − 1 = R j − 1 ∗ ( η T j − 1 + Im ( w z ) Σ ) R j − 1 . (90)
Since T j − 1 ⪰ 0 T_{j-1}\succeq0 T j − 1 ⪰ 0 and Im ( w z ‾ ) ≥ 0 \operatorname{Im}(w\overline{z})\ge0 Im ( w z ) ≥ 0 , the right-hand side is positive semidefinite. Hence, for every x ∈ H x\in\mathcal{H} x ∈ H ,
Im ( z ( 1 + x ∗ R j − 1 x ) ) = η + x ∗ Im ( z R j − 1 ) x ≥ η , \operatorname{Im}\left(z(1+x^{*}R_{j-1}x)\right)=\eta+x^{*}\operatorname{Im}(zR_{j-1})x\ge\eta, Im ( z ( 1 + x ∗ R j − 1 x ) ) = η + x ∗ Im ( z R j − 1 ) x ≥ η ,
which implies ∣ 1 + x ∗ R j − 1 x ∣ ≥ η / ∣ z ∣ \lvert1+x^{*}R_{j-1}x\rvert\ge\eta/\lvert z\rvert ∣ 1 + x ∗ R j − 1 x ∣ ≥ η / ∣ z ∣ . Similarly, we only need to prove the case when t < 0 t<0 t < 0 as η = ∣ z − ∣ \eta=\lvert z_{-}\rvert η = ∣ z − ∣ when t ≥ 0 t\ge0 t ≥ 0 . We again use u ≤ t v / η ≤ 0 u\le tv/\eta\le0 u ≤ t v / η ≤ 0 and additionally the inverse real-part identity:
Re R j − 1 = R j − 1 ∗ ( T j − 1 − u Σ − t I ) R j − 1 ⪰ 0. (91) \operatorname{Re}R_{j-1}=R_{j-1}^{*}(T_{j-1}-u\Sigma-tI)R_{j-1}\succeq0. \tag*{(91)} Re R j − 1 = R j − 1 ∗ ( T j − 1 − u Σ − t I ) R j − 1 ⪰ 0. (91)
It then follows that ∣ 1 + x ∗ R j − 1 x ∣ ≥ Re ( 1 + x ∗ R j − 1 x ) ≥ 1 = ∣ z − ∣ / ∣ z ∣ \lvert1+x^{*}R_{j-1}x\rvert\ge\operatorname{Re}(1+x^{*}R_{j-1}x)\ge1=\lvert z_{-}\rvert/\lvert z\rvert ∣ 1 + x ∗ R j − 1 x ∣ ≥ Re ( 1 + x ∗ R j − 1 x ) ≥ 1 = ∣ z − ∣ / ∣ z ∣ . Thus ∣ 1 + x ∗ R j − 1 x ∣ − 1 ≤ ∣ z ∣ / ∣ z − ∣ \lvert1+x^{*}R_{j-1}x\rvert^{-1}\le\lvert z\rvert/\lvert z_{-}\rvert ∣ 1 + x ∗ R j − 1 x ∣ − 1 ≤ ∣ z ∣ / ∣ z − ∣ in both cases.
We finally cover the case of z < 0 z<0 z < 0 and real w ≤ 0 w\le0 w ≤ 0 . Now the operator is self-adjoint and
M j − 1 ( z , w ) = T j − 1 − w Σ + ∣ z ∣ I ⪰ ∣ z ∣ I , 0 ⪯ R j − 1 ( z , w ) ⪯ ∣ z ∣ − 1 I . (92) M_{j-1}(z,w)=T_{j-1}-w\Sigma+\lvert z\rvert I\succeq\lvert z\rvert I,\qquad0\preceq R_{j-1}(z,w)\preceq\lvert z\rvert^{-1}I. \tag*{(92)} M j − 1 ( z , w ) = T j − 1 − w Σ + ∣ z ∣ I ⪰ ∣ z ∣ I , 0 ⪯ R j − 1 ( z , w ) ⪯ ∣ z ∣ − 1 I . (92)
Thus the inverse exists and 1 + x ∗ R j − 1 ( z , w ) x ≥ 1 1+x^{*}R_{j-1}(z,w)x\ge1 1 + x ∗ R j − 1 ( z , w ) x ≥ 1 . Since ∣ z − ∣ = ∣ z ∣ \lvert z_{-}\rvert=\lvert z\rvert ∣ z − ∣ = ∣ z ∣ , this proves both bounds in (i) at these fixed real parameters as well. It remains to verify the required conditions on w w w along the path in Part III, i.e. Im ( w k z ‾ ) ≥ 0 \operatorname{Im}(w_k\overline{z})\ge0 Im ( w k z ) ≥ 0 when η > 0 \eta>0 η > 0 and w k ≤ 0 w_k\le0 w k ≤ 0 when t < 0 , η = 0 t<0,\eta=0 t < 0 , η = 0 .
Part II: Expected-trace bounds in (ii). We first prove the strict positivity results at a general ( z , w ) ∈ D (z,w)\in D ( z , w ) ∈ D . By the inverse imaginary-part identity, we have
Im R j − 1 = R j − 1 ∗ ⋅ M j − 1 ∗ − M j − 1 2 i ⋅ R j − 1 = R j − 1 ∗ ( η I + v Σ ) R j − 1 ⪰ 0. \operatorname{Im}R_{j-1}=R_{j-1}^{*}\cdot\frac{M_{j-1}^{*}-M_{j-1}}{2i}\cdot R_{j-1}=R_{j-1}^{*}(\eta I+v\Sigma)R_{j-1}\succeq0. Im R j − 1 = R j − 1 ∗ ⋅ 2 i M j − 1 ∗ − M j − 1 ⋅ R j − 1 = R j − 1 ∗ ( η I + v Σ ) R j − 1 ⪰ 0.
Recalling the definition of G j G_j G j in Eq. (23) , we obtain from v ≥ 0 v \ge0 v ≥ 0 that
Im G j ( z , w ) = Tr ( Σ ‾ 1 2 R j − 1 ∗ ( η I + v Σ ) R j − 1 Σ ‾ 1 2 ) ≥ η ∥ R j − 1 Σ ‾ 1 2 ∥ F 2 ≥ η Tr ( Σ ) ∥ M j − 1 ∥ 2 > 0 , \operatorname{Im} G_j(z,w) = \operatorname{Tr}\left(\overline{\Sigma}^{\frac{1}{2}} R_{j-1}^{*}(\eta I+v\Sigma)R_{j-1}\overline{\Sigma}^{\frac{1}{2}}\right) \ge\eta\left\|R_{j-1}\overline{\Sigma}^{\frac{1}{2}}\right\|_{F}^{2} \ge\frac{\eta\operatorname{Tr}(\Sigma)}{\left\|M_{j-1}\right\|^{2}} > 0, Im G j ( z , w ) = Tr ( Σ 2 1 R j − 1 ∗ ( η I + v Σ ) R j − 1 Σ 2 1 ) ≥ η R j − 1 Σ 2 1 F 2 ≥ ∥ M j − 1 ∥ 2 η Tr ( Σ ) > 0 ,
where the last lower bound follows from M j − 1 R j − 1 = I M_{j-1}R_{j-1}=I M j − 1 R j − 1 = I and Tr ( Σ ) 1 2 = ∥ Σ 1 2 ∥ F ≤ ∥ M j − 1 ∥ ∥ R j − 1 Σ 1 2 ∥ F \operatorname{Tr}(\Sigma)^{\frac{1}{2}}=\left\|\Sigma^{\frac{1}{2}}\right\|_{F}\le\left\|M_{j-1}\right\|\left\|R_{j-1}\Sigma^{\frac{1}{2}}\right\|_{F} Tr ( Σ ) 2 1 = Σ 2 1 F ≤ ∥ M j − 1 ∥ R j − 1 Σ 2 1 F . Applying the norm bound from Part I gives ∣ G j ( z , w ) ∣ = ∣ Tr ( Σ R j − 1 ( z , w ) ) ∣ ≤ Tr ( Σ ) ∥ R j − 1 ( z , w ) ∥ ≤ Tr ( Σ ) / η \left|G_j(z,w)\right|=\left|\operatorname{Tr}\left(\Sigma R_{j-1}(z,w)\right)\right|\le\operatorname{Tr}(\Sigma)\left\|R_{j-1}(z,w)\right\|\le\operatorname{Tr}(\Sigma)/\eta ∣ G j ( z , w ) ∣ = ∣ Tr ( Σ R j − 1 ( z , w ) ) ∣ ≤ Tr ( Σ ) ∥ R j − 1 ( z , w ) ∥ ≤ Tr ( Σ ) / η , so we can take expectation by boundedness, and Im G j ( z , w ) = E [ Im G j ( z , w ) ] > 0 \operatorname{Im} G_j(z,w)=\mathbb{E}[\operatorname{Im}G_j(z,w)]>0 Im G j ( z , w ) = E [ Im G j ( z , w )] > 0 .
We then use the operator identities from Part I to obtain the denominator and real-part bounds under Im ( w ‾ z ) ≥ 0 \operatorname{Im}(\overline{w}z)\ge0 Im ( w z ) ≥ 0 . Taking the weighted trace and expectation in Eq. (90) implies
Im ( z ( 1 + G j ‾ ( z , w ) ) ) = η + E Tr ( Σ 1 2 Im ( z R j − 1 ) Σ 1 2 ) ≥ η . (93) \operatorname{Im}\left(z(1+\overline{G_j}(z,w))\right)=\eta+\mathbb{E}\operatorname{Tr}\left(\Sigma^{\frac{1}{2}}\operatorname{Im}(zR_{j-1})\Sigma^{\frac{1}{2}}\right)\ge\eta. \tag*{(93)} Im ( z ( 1 + G j ( z , w )) ) = η + E Tr ( Σ 2 1 Im ( z R j − 1 ) Σ 2 1 ) ≥ η . (93)
We obtain the denominator bound when t ≥ 0 t\ge0 t ≥ 0 since ∣ z ∣ ⋅ ∣ 1 + G j ‾ ( z , w ) ∣ ≥ η = ∣ z − ∣ \lvert z\rvert\cdot\lvert1+\overline{G_j}(z,w)\rvert\ge\eta=\lvert z_-\rvert ∣ z ∣ ⋅ ∣ 1 + G j ( z , w )∣ ≥ η = ∣ z − ∣ . For t < 0 t<0 t < 0 , taking the weighted trace and expectation in Eq. (91) instead yields
Re G j ‾ ( z , w ) = E Tr ( Σ 1 2 Re R j − 1 Σ 1 2 ) ≥ 0 and ∣ 1 + G j ‾ ( z , w ) ∣ ≥ 1 + Re G j ‾ ( z , w ) ≥ 1 = ∣ z − ∣ ∣ z ∣ . \operatorname{Re}\overline{G_j}(z,w)=\mathbb{E}\operatorname{Tr}\left(\Sigma^{\frac{1}{2}}\operatorname{Re}R_{j-1}\Sigma^{\frac{1}{2}}\right)\ge0
\qquad\text{and}\qquad
\lvert1+\overline{G_j}(z,w)\rvert\ge1+\operatorname{Re}\overline{G_j}(z,w)\ge1=\frac{\lvert z_-\rvert}{\lvert z\rvert}. Re G j ( z , w ) = E Tr ( Σ 2 1 Re R j − 1 Σ 2 1 ) ≥ 0 and ∣ 1 + G j ( z , w )∣ ≥ 1 + Re G j ( z , w ) ≥ 1 = ∣ z ∣ ∣ z − ∣ .
This also proves the real-part positivity in (ii).
Lastly, for the case of z < 0 z<0 z < 0 and real w ≤ 0 w\le0 w ≤ 0 , we obtain from taking the weighted trace and expectation in Eq. (92) that
0 ≤ G j ‾ ( z , w ) ≤ Tr ( Σ ) ∣ z ∣ , ∣ 1 + G j ‾ ( z , w ) ∣ = 1 + G j ‾ ( z , w ) ≥ 1 = ∣ z ∣ ∣ z ∣ . (94) 0\le\overline{G_j}(z,w)\le\frac{\operatorname{Tr}(\Sigma)}{\lvert z\rvert},
\qquad
\lvert1+\overline{G_j}(z,w)\rvert=1+\overline{G_j}(z,w)\ge1=\frac{\lvert z\rvert}{\lvert z\rvert}. \tag*{(94)} 0 ≤ G j ( z , w ) ≤ ∣ z ∣ Tr ( Σ ) , ∣ 1 + G j ( z , w )∣ = 1 + G j ( z , w ) ≥ 1 = ∣ z ∣ ∣ z ∣ . (94)
Thus (ii) holds at each fixed parameter satisfying the same path parameter conditions of Part I. We now verify these conditions along the backward path.
Part III: Path and endpoint bounds in (iii). For z ∈ C + z\in\mathbb{C}_{+} z ∈ C + , we prove by backward induction that ( z , w k ) ∈ D (z,w_k)\in\mathcal{D} ( z , w k ) ∈ D and Im ( w k z ‾ ) ≥ 0 \operatorname{Im}(w_k\overline{z})\ge0 Im ( w k z ) ≥ 0 , which also justifies applying Parts I and II at every path parameter. The exact analysis also yields a quantitative lower bound for the latter increment, whose sum will give the second endpoint estimate conclusion in (iii).
We start the induction backward at w n = 0 w_n=0 w n = 0 . Suppose the two conditions hold at some 1 ≤ k ≤ n 1\le k\le n 1 ≤ k ≤ n . From the strict positivity proved in Part II, namely Im G k ‾ ( z , w k ) > 0 \operatorname{Im}\overline{G_k}(z,w_k)>0 Im G k ( z , w k ) > 0 , we have that the denominator in the backward recursion is nonzero and the imaginary part increases strictly
Im w k − 1 = Im w k + Im G k ‾ ( z , w k ) ∣ 1 + G k ‾ ( z , w k ) ∣ 2 > 0. \operatorname{Im}w_{k-1}=\operatorname{Im}w_k+\frac{\operatorname{Im}\overline{G_k}(z,w_k)}{\lvert1+\overline{G_k}(z,w_k)\rvert^{2}}>0. Im w k − 1 = Im w k + ∣ 1 + G k ( z , w k ) ∣ 2 Im G k ( z , w k ) > 0.
Thus ( z , w k − 1 ) ∈ D (z,w_{k-1})\in\mathcal{D} ( z , w k − 1 ) ∈ D . For the second condition, it holds from the same recursion that
Im ( w k − 1 z ‾ ) − Im ( w k z ‾ ) = Im ( z ( 1 + G k ‾ ( z , w k ) ) ) ∣ 1 + G k ‾ ( z , w k ) ∣ 2 ≥ η ∣ 1 + G k ‾ ( z , w k ) ∣ 2 > 0 (95) \operatorname{Im}(w_{k-1}\overline{z})-\operatorname{Im}(w_k\overline{z})=\frac{\operatorname{Im}\left(z(1+\overline{G_k}(z,w_k))\right)}{\lvert1+\overline{G_k}(z,w_k)\rvert^{2}}\ge\frac{\eta}{\lvert1+\overline{G_k}(z,w_k)\rvert^{2}}>0 \tag*{(95)} Im ( w k − 1 z ) − Im ( w k z ) = ∣ 1 + G k ( z , w k ) ∣ 2 Im ( z ( 1 + G k ( z , w k )) ) ≥ ∣ 1 + G k ( z , w k ) ∣ 2 η > 0 (95)
The last inequality applies Eq. (93) with j = k j=k j = k and w = w k w=w_k w = w k , and we close the positivity results in Lemma E.1.(iii) by induction. Substituting w = w k w=w_k w = w k into Parts I and II we also close the loop and prove Lemma E.1.(i) and (ii).
It only remains to establish the endpoint bound. To this end, we first sum Eq. (95) from w n = 0 w_n=0 w n = 0 ,
Im ( w 0 z ‾ ) ≥ η ∑ k = 1 n 1 ∣ 1 + G k ‾ ( z , w k ) ∣ 2 . \operatorname{Im}(w_0\overline{z})\ge\eta\sum_{k=1}^{n}\frac{1}{\lvert1+\overline{G_k}(z,w_k)\rvert^{2}}. Im ( w 0 z ) ≥ η k = 1 ∑ n ∣ 1 + G k ( z , w k ) ∣ 2 1 .
Telescoping the backward recursion and then applying Cauchy–Schwarz implies
∣ w 0 ∣ 2 = ∣ ∑ k = 1 n 1 1 + G ‾ k ( z , w k ) ∣ 2 ≤ n ∑ k = 1 n 1 ∣ 1 + G ‾ k ( z , w k ) ∣ 2 ≤ n η Im ( w 0 z ‾ ) . |w_0|^2 = \left|\sum_{k=1}^{n} \frac{1}{1+\overline{G}_k(z,w_k)}\right|^2 \le n\sum_{k=1}^{n}\frac{1}{|1+\overline{G}_k(z,w_k)|^2} \le\frac{n}{\eta}\operatorname{Im}(w_0\overline{z}). ∣ w 0 ∣ 2 = k = 1 ∑ n 1 + G k ( z , w k ) 1 2 ≤ n k = 1 ∑ n ∣1 + G k ( z , w k ) ∣ 2 1 ≤ η n Im ( w 0 z ) .
where the last inequality uses the preceding display. Therefore, rearranging proves the second bound in Lemma E.1,(iii), Eq. (57) .
Finally, for z < 0 z<0 z < 0 , it suffices to verify that the recursion stays on the real half-line w ≤ 0 w\le0 w ≤ 0 . Starting from w n = 0 w_n=0 w n = 0 , suppose w k ≤ 0 w_k\le0 w k ≤ 0 is real. Applying Eq. (94) with j = k j=k j = k gives G ‾ k ( z , w k ) ≥ 0 \overline{G}_k(z,w_k)\ge0 G k ( z , w k ) ≥ 0 , so the next value is real and satisfies
w k − 1 = w k − 1 1 + G ‾ k ( z , w k ) < w k ≤ 0. w_{k-1}=w_k-\frac{1}{1+\overline{G}_k(z,w_k)}<w_k\le0. w k − 1 = w k − 1 + G k ( z , w k ) 1 < w k ≤ 0.
This closes the real-parameter induction. Finally, the backward recursion uniquely defines { w k } k = 0 n \{w_k\}_{k=0}^{n} { w k } k = 0 n and all resolvents and expected traces appearing in it are well-defined, and the proof is completed.
Proof of Lemma E.2
We prove both types of bounds using the same rotation argument of the inverse resolvent. In Part I, we bound the norms and trace of the weighted resolvent in terms of the real part of the rotated inverse. In Part II, keeping the T k T_k T k term explicit provides the comparison with A k ( ∣ z − ∣ ) A_k(|z_-|) A k ( ∣ z − ∣ ) . In Part III, using Σ \Sigma Σ term implies the bounds in terms of − Re ( e i θ w j ) -\operatorname{Re}(e^{i\theta}w_j) − Re ( e i θ w j ) .
Recall that z = t + i η z=t+i\eta z = t + i η , w = u + i v w=u+iv w = u + i v , and R ~ k = R ~ k ( z , w ) \widetilde{R}_k=\widetilde{R}_k(z,w) R k = R k ( z , w ) . We work first at ( z , w ) ∈ D (z,w)\in D ( z , w ) ∈ D with Im ( w z ‾ ) = t v − η u ≥ 0 \operatorname{Im}(w\overline{z})=tv-\eta u\ge0 Im ( w z ) = t v − η u ≥ 0 , or at real z < 0 z<0 z < 0 and w ≤ 0 w\le0 w ≤ 0 . The path properties established in Part I of Appendix E.1 verify these conditions at every w = w j w=w_j w = w j .
Part I: The common rotation and comparison. For t + = t ∨ 0 t_+=t\vee0 t + = t ∨ 0 , we set the rotation angle θ = arctan ( 1 + 2 t + / η ) \theta=\arctan(1+2t_+/\eta) θ = arctan ( 1 + 2 t + / η ) when η > 0 \eta>0 η > 0 , and θ = π / 4 \theta=\pi/4 θ = π /4 when z < 0 z<0 z < 0 is real. For U = e i θ ( T k − w Σ − z I ) U=e^{i\theta}(T_k-w\Sigma-zI) U = e i θ ( T k − w Σ − z I ) , we will prove the two lower bounds
Re U ⪰ cos θ ⋅ ( T k + ∣ z − ∣ I ) and Re U ⪰ 1 2 ∣ z − ∣ I − Re ( e i θ w ) Σ . (96) \operatorname{Re}U\succeq\cos\theta\cdot(T_k+|z_-|I)
\qquad\text{and}\qquad
\operatorname{Re}U\succeq\frac{1}{2}|z_-|I-\operatorname{Re}(e^{i\theta}w)\Sigma. \tag*{(96)} Re U ⪰ cos θ ⋅ ( T k + ∣ z − ∣ I ) and Re U ⪰ 2 1 ∣ z − ∣ I − Re ( e i θ w ) Σ. (96)
First, for t ≥ 0 t\ge0 t ≥ 0 , we have η > 0 \eta>0 η > 0 and u ≤ t v / η u\le tv/\eta u ≤ t v / η , so
Re U = cos θ ⋅ ( T k + ( ( 1 + 2 t / η ) v − u ) Σ + ( η + t ) I ) ⪰ cos θ ⋅ T k + η + t η 2 + ( η + 2 t ) 2 ( v Σ + η I ) . \operatorname{Re}U=\cos\theta\cdot\left(T_k+\left((1+2t/\eta)v-u\right)\Sigma+(\eta+t)I\right)\succeq\cos\theta\cdot T_k+\frac{\eta+t}{\sqrt{\eta^2+(\eta+2t)^2}}(v\Sigma+\eta I). Re U = cos θ ⋅ ( T k + ( ( 1 + 2 t / η ) v − u ) Σ + ( η + t ) I ) ⪰ cos θ ⋅ T k + η 2 + ( η + 2 t ) 2 η + t ( v Σ + η I ) .
The coefficient in the lower bound is at least 1 / 2 1/2 1/2 , because η 2 + ( η + 2 t ) 2 ≤ 4 ( η + t ) 2 \eta^2+(\eta+2t)^2\le4(\eta+t)^2 η 2 + ( η + 2 t ) 2 ≤ 4 ( η + t ) 2 . In the first equality above, the coefficient of Σ \Sigma Σ is cos θ ⋅ ( ( 1 + 2 t / η ) v − u ) = − Re ( e i θ w ) \cos\theta\cdot((1+2t/\eta)v-u)=-\operatorname{Re}(e^{i\theta}w) cos θ ⋅ (( 1 + 2 t / η ) v − u ) = − Re ( e i θ w ) , and the coefficient of I I I is ( η + t ) cos θ = η ( η + t ) / η 2 + ( η + 2 t ) 2 ≥ η / 2 (\eta+t)\cos\theta=\eta(\eta+t)/\sqrt{\eta^2+(\eta+2t)^2}\ge\eta/2 ( η + t ) cos θ = η ( η + t ) / η 2 + ( η + 2 t ) 2 ≥ η /2 . This proves the second bound in Eq. (96) , since ∣ z − ∣ = η |z_-|=\eta ∣ z − ∣ = η . The first equality also proves the first bound in Eq. (96) , as its Σ \Sigma Σ coefficient is nonnegative and η + t ≥ ∣ z − ∣ \eta+t\ge|z_-| η + t ≥ ∣ z − ∣ . Second, for t < 0 t<0 t < 0 and η > 0 \eta>0 η > 0 , the same condition and v ≥ 0 v\ge0 v ≥ 0 imply u ≤ t v / η ≤ 0 u\le tv/\eta\le0 u ≤ t v / η ≤ 0 . Third, for real z < 0 z<0 z < 0 , we assumed u = w ≤ 0 u=w\le0 u = w ≤ 0 and v = 0 v=0 v = 0 . In both cases θ = π / 4 \theta=\pi/4 θ = π /4 , so
Re U = T k + ( v − u ) Σ + ( η − t ) I 2 ⪰ T k + ( v − u ) Σ + ∣ z − ∣ I 2 , \operatorname{Re}U=\frac{T_k+(v-u)\Sigma+(\eta-t)I}{\sqrt{2}}\succeq\frac{T_k+(v-u)\Sigma+|z_-|I}{\sqrt{2}}, Re U = 2 T k + ( v − u ) Σ + ( η − t ) I ⪰ 2 T k + ( v − u ) Σ + ∣ z − ∣ I ,
where we use η − t ≥ ∣ z − ∣ \eta-t\ge|z_-| η − t ≥ ∣ z − ∣ and v − u ≥ 0 v-u\ge0 v − u ≥ 0 . Here ( v − u ) / 2 = − Re ( e i θ w ) (v-u)/\sqrt{2}=-\operatorname{Re}(e^{i\theta}w) ( v − u ) / 2 = − Re ( e i θ w ) and ∣ z − ∣ / 2 ≥ ∣ z − ∣ / 2 |z_-|/\sqrt{2}\ge|z_-|/2 ∣ z − ∣/ 2 ≥ ∣ z − ∣/2 . Both lower bounds in Eq. (96) still follow.
Next we derive the common norm comparisons used in Parts II and III, by setting V : = Re U ≻ 0 V := \operatorname{Re} U \succ0 V := Re U ≻ 0 . The inverse of U U U exists by Part I of the proof of Lemma E.1 . We use U + U ∗ = 2 V U + U^* = 2V U + U ∗ = 2 V , V ∗ = V V^* = V V ∗ = V and then take the congruence by U − ∗ Σ 1 / 2 U^{-*}\Sigma^{1/2} U −∗ Σ 1/2 to obtain that
U ∗ V − 1 U − V = ( U − V ) ∗ V − 1 ( U − V ) ⪰ 0 and Σ 1 / 2 U − ∗ V U − 1 Σ 1 / 2 ⪯ Σ 1 / 2 V − 1 Σ 1 / 2 . (97) U^*V^{-1}U - V = (U-V)^*V^{-1}(U-V) \succeq0 \quad\text{and} \quad\Sigma^{1/2}U^{-*}VU^{-1}\Sigma^{1/2} \preceq\Sigma^{1/2}V^{-1}\Sigma^{1/2}. \tag*{(97)} U ∗ V − 1 U − V = ( U − V ) ∗ V − 1 ( U − V ) ⪰ 0 and Σ 1/2 U −∗ V U − 1 Σ 1/2 ⪯ Σ 1/2 V − 1 Σ 1/2 . (97)
Identify that R ~ k = e i θ ( Σ 1 / 2 V − 1 / 2 ) ( V 1 / 2 U − 1 Σ 1 / 2 ) \widetilde{R}_k = e^{i\theta}(\Sigma^{1/2}V^{-1/2})(V^{1/2}U^{-1}\Sigma^{1/2}) R k = e i θ ( Σ 1/2 V − 1/2 ) ( V 1/2 U − 1 Σ 1/2 ) , and we apply sub-multiplicativity for the operator norm and trace Cauchy–Schwarz for the Frobenius norm and trace to obtain
∥ R ~ k ∥ ≤ ∥ Σ 1 / 2 V − 1 / 2 ∥ ∥ V 1 / 2 U − 1 Σ 1 / 2 ∥ ≤ ∥ Σ 1 / 2 V − 1 Σ 1 / 2 ∥ , (98) \lVert\widetilde{R}_k\rVert\leq\lVert\Sigma^{1/2}V^{-1/2}\rVert\lVert V^{1/2}U^{-1}\Sigma^{1/2}\rVert\leq\lVert\Sigma^{1/2}V^{-1}\Sigma^{1/2}\rVert, \tag*{(98)} ∥ R k ∥ ≤ ∥ Σ 1/2 V − 1/2 ∥ ∥ V 1/2 U − 1 Σ 1/2 ∥ ≤ ∥ Σ 1/2 V − 1 Σ 1/2 ∥ , (98)
∥ R ~ k ∥ F 2 = Tr ( ( V − 1 / 2 Σ V − 1 / 2 ) ( V 1 / 2 U − 1 Σ U − ∗ V 1 / 2 ) ) ≤ ∥ Σ 1 / 2 V − 1 Σ 1 / 2 ∥ F ∥ Σ 1 / 2 U − ∗ V U − 1 Σ 1 / 2 ∥ F ≤ ∥ Σ 1 / 2 V − 1 Σ 1 / 2 ∥ F 2 , (99) \begin{aligned}
\lVert\widetilde{R}_k\rVert_F^2 = \operatorname{Tr}\left((V^{-1/2}\Sigma V^{-1/2})(V^{1/2}U^{-1}\Sigma U^{-*}V^{1/2})\right) \\
&\leq\lVert\Sigma^{1/2}V^{-1}\Sigma^{1/2}\rVert_F\lVert\Sigma^{1/2}U^{-*}VU^{-1}\Sigma^{1/2}\rVert_F \leq\lVert\Sigma^{1/2}V^{-1}\Sigma^{1/2}\rVert_F^2, \tag*{(99)}
\end{aligned} ∥ R k ∥ F 2 = Tr ( ( V − 1/2 Σ V − 1/2 ) ( V 1/2 U − 1 Σ U −∗ V 1/2 ) ) ≤ ∥ Σ 1/2 V − 1 Σ 1/2 ∥ F ∥ Σ 1/2 U −∗ V U − 1 Σ 1/2 ∥ F ≤ ∥ Σ 1/2 V − 1 Σ 1/2 ∥ F 2 , (99)
∣ Tr R ~ k ∣ ≤ ∥ Σ 1 / 2 V − 1 / 2 ∥ F ∥ V 1 / 2 U − 1 Σ 1 / 2 ∥ F ≤ Tr ( Σ 1 / 2 V − 1 Σ 1 / 2 ) . (100) \left|\operatorname{Tr}\widetilde{R}_k\right| \leq\lVert\Sigma^{1/2}V^{-1/2}\rVert_F\lVert V^{1/2}U^{-1}\Sigma^{1/2}\rVert_F \leq\operatorname{Tr}(\Sigma^{1/2}V^{-1}\Sigma^{1/2}). \tag*{(100)} Tr R k ≤ ∥ Σ 1/2 V − 1/2 ∥ F ∥ V 1/2 U − 1 Σ 1/2 ∥ F ≤ Tr ( Σ 1/2 V − 1 Σ 1/2 ) . (100)
Each final inequality above follows from Eq. (97) . In particular, the Frobenius norm is monotone for positive operators since 0 ⪯ C ⪯ D 0 \preceq C \preceq D 0 ⪯ C ⪯ D implies Tr ( D 2 ) − Tr ( C 2 ) = Tr ( ( D − C ) ( D + C ) ) ≥ 0 \operatorname{Tr}(D^2)-\operatorname{Tr}(C^2)=\operatorname{Tr}((D-C)(D+C))\geq0 Tr ( D 2 ) − Tr ( C 2 ) = Tr (( D − C ) ( D + C )) ≥ 0 .
Part II: Comparison with the real weighted resolvent. To prove Eq. (44a), it suffices to bound Σ 1 / 2 V − 1 Σ 1 / 2 \Sigma^{1/2}V^{-1}\Sigma^{1/2} Σ 1/2 V − 1 Σ 1/2 by 3 ∣ z ∣ A k ( ∣ z − ∣ ) / ∣ z − ∣ 3|z|A_k(|z_-|)/|z_-| 3∣ z ∣ A k ( ∣ z − ∣ ) /∣ z − ∣ . We use the first lower bound in Eq. (96) . Taking inverses reverses the positive-operator order, and taking the congruence by Σ 1 / 2 \Sigma^{1/2} Σ 1/2 yields
Σ 1 / 2 V − 1 Σ 1 / 2 ⪯ 1 cos θ Σ 1 / 2 ( T k + ∣ z − ∣ I ) − 1 Σ 1 / 2 = 1 cos θ A k ( ∣ z − ∣ ) . \Sigma^{1/2}V^{-1}\Sigma^{1/2} \preceq\frac{1}{\cos\theta}\Sigma^{1/2}(T_k+|z_-|I)^{-1}\Sigma^{1/2} = \frac{1}{\cos\theta}A_k(|z_-|). Σ 1/2 V − 1 Σ 1/2 ⪯ cos θ 1 Σ 1/2 ( T k + ∣ z − ∣ I ) − 1 Σ 1/2 = cos θ 1 A k ( ∣ z − ∣ ) .
By monotonicity in p.s.d. cone for Eqs. (76a) to (76c), we establish all three bounds with factor ( cos θ ) − 1 (\cos\theta)^{-1} ( cos θ ) − 1 . Therefore it only remains to bound this factor. If t < 0 t<0 t < 0 , then ( cos θ ) − 1 = 2 (\cos\theta)^{-1}=\sqrt{2} ( cos θ ) − 1 = 2 and ∣ z ∣ = ∣ z − ∣ |z|=|z_-| ∣ z ∣ = ∣ z − ∣ . If t ≥ 0 t\geq0 t ≥ 0 , then ∣ z − ∣ = η |z_-|=\eta ∣ z − ∣ = η and
( cos θ ) − 1 = η 2 + ( η + 2 t ) 2 η ≤ 4 η 2 + 6 t 2 η ≤ 6 ∣ z ∣ η ≤ 3 ∣ z ∣ ∣ z − ∣ . (\cos\theta)^{-1}=\frac{\sqrt{\eta^2+(\eta+2t)^2}}{\eta}\leq\frac{\sqrt{4\eta^2+6t^2}}{\eta}\leq\frac{\sqrt{6}|z|}{\eta}\leq\frac{3|z|}{|z_-|}. ( cos θ ) − 1 = η η 2 + ( η + 2 t ) 2 ≤ η 4 η 2 + 6 t 2 ≤ η 6 ∣ z ∣ ≤ ∣ z − ∣ 3∣ z ∣ .
The first inequality uses 4 η t ≤ 2 η 2 + 2 t 2 4\eta t\leq2\eta^2+2t^2 4 η t ≤ 2 η 2 + 2 t 2 . Taking w = w j w=w_j w = w j now proves Eq. (44a).
Part III: Bounds using the path parameter. Finally, we prove Eq. (44b). Write w j = u j + i v j w_j=u_j+iv_j w j = u j + i v j . For j < n j<n j < n , the path monotonicity established in Part I of Appendix E.1 gives v j > 0 v_j>0 v j > 0 when z ∈ C + z\in\mathbb{C}_+ z ∈ C + , and u j = w j < 0 u_j=w_j<0 u j = w j < 0 , v j = 0 v_j=0 v j = 0 when z < 0 z<0 z < 0 is real. For t ≥ 0 t\geq0 t ≥ 0 , using u j ≤ t v j / η u_j\leq tv_j/\eta u j ≤ t v j / η in the first calculation in Part I yields − Re ( e i θ w j ) ≥ cos θ ( 1 + t / η ) v j > 0 -\operatorname{Re}(e^{i\theta}w_j)\geq\cos\theta(1+t/\eta)v_j>0 − Re ( e i θ w j ) ≥ cos θ ( 1 + t / η ) v j > 0 . For t < 0 t<0 t < 0 , the same coefficient is ( v j − u j ) / 2 > 0 (v_j-u_j)/\sqrt{2}>0 ( v j − u j ) / 2 > 0 , since u j ≤ 0 u_j\leq0 u j ≤ 0 and either v j > 0 v_j>0 v j > 0 or w j = u j < 0 w_j=u_j<0 w j = u j < 0 is real. We may therefore first derive the bounds for any w w w with − Re ( e i θ w ) > 0 -\operatorname{Re}(e^{i\theta}w)>0 − Re ( e i θ w ) > 0 and then substitute w = w j w=w_j w = w j .
The second lower bound in Eq. (96) compares V V V with an operator diagonal in the population eigenbasis. Taking inverses and the congruence by Σ 1 / 2 \Sigma^{1/2} Σ 1/2 gives
Σ 1 / 2 V − 1 Σ 1 / 2 ⪯ 2 Σ 1 / 2 ⋅ ( ∣ z − ∣ I − 2 Re ( e i θ w ) Σ ) − 1 Σ 1 / 2 . \Sigma^{1/2}V^{-1}\Sigma^{1/2}\preceq2\Sigma^{1/2}\cdot\left(|z_-|I-2\operatorname{Re}(e^{i\theta}w)\Sigma\right)^{-1}\Sigma^{1/2}. Σ 1/2 V − 1 Σ 1/2 ⪯ 2 Σ 1/2 ⋅ ( ∣ z − ∣ I − 2 Re ( e i θ w ) Σ ) − 1 Σ 1/2 .
From Eqs. (76a) and (76b), monotonicity in the p.s.d. cone, and splitting the eigenvalue sum at m m m , we have
∥ R ~ k ∥ ≤ 2 sup i σ i ∣ z − ∣ − 2 Re ( e i θ w ) σ i ≤ 1 − Re ( e i θ w ) , \lVert\widetilde{R}_k\rVert\leq2\sup_i\frac{\sigma_i}{|z_-|-2\operatorname{Re}(e^{i\theta}w)\sigma_i}\leq\frac{1}{-\operatorname{Re}(e^{i\theta}w)}, ∥ R k ∥ ≤ 2 i sup ∣ z − ∣ − 2 Re ( e i θ w ) σ i σ i ≤ − Re ( e i θ w ) 1 ,
∥ R ~ k ∥ F 2 ≤ 4 ∑ i σ i 2 ( ∣ z − ∣ − 2 Re ( e i θ w ) σ i ) 2 ≤ ∑ i ≤ m 1 Re ( e i θ w ) 2 + 4 ∑ i > m σ i 2 ∣ z − ∣ 2 = m Re ( e i θ w ) 2 + 4 τ 2 ( m ) ∣ z − ∣ 2 . \lVert\widetilde{R}_k\rVert_F^2\leq4\sum_i\frac{\sigma_i^2}{\left(|z_-|-2\operatorname{Re}(e^{i\theta}w)\sigma_i\right)^2}\leq\sum_{i\leq m}\frac{1}{\operatorname{Re}(e^{i\theta}w)^2}+4\sum_{i>m}\frac{\sigma_i^2}{|z_-|^2}=\frac{m}{\operatorname{Re}(e^{i\theta}w)^2}+\frac{4\tau_2(m)}{|z_-|^2}. ∥ R k ∥ F 2 ≤ 4 i ∑ ( ∣ z − ∣ − 2 Re ( e i θ w ) σ i ) 2 σ i 2 ≤ i ≤ m ∑ Re ( e i θ w ) 2 1 + 4 i > m ∑ ∣ z − ∣ 2 σ i 2 = Re ( e i θ w ) 2 m + ∣ z − ∣ 2 4 τ 2 ( m ) .
Taking w = w j = u j + i v j w=w_j=u_j+iv_j w = w j = u j + i v j proves Eq. (44b) and the proof is complete.
Proof of Lemma E.3
We need bounds for the operator norm, squared Frobenius norm, and trace of A k ( ζ ) A_k(\zeta) A k ( ζ ) , uniformly over 1 ≤ p ≤ κ log k 1 \le p \le\kappa\log k 1 ≤ p ≤ κ log k . In Part I, we reduce these quantities to the norm of the lower population block on an event where the top Gram matrix is well-conditioned. In Part II, we integrate a tail bound for the lower block. In Part III, we combine these estimates and bound the exceptional-event contribution by the deterministic inequality 0 ⪯ A k ( ζ ) ⪯ Σ / ζ 0 \preceq A_k(\zeta) \preceq\Sigma/\zeta 0 ⪯ A k ( ζ ) ⪯ Σ/ ζ .
Part I: Reduction to the lower block. The following deterministic inequality separates the top and lower population eigenspaces without requiring independence between them.
Lemma E.10 ([28 ] , Lemma C.1). Let U , W : H → R k U,W:\mathcal{H}\to\mathbb{R}^k U , W : H → R k be bounded operators. For every ζ > 0 \zeta>0 ζ > 0 ,
ζ I + ( U + W ) ∗ ( U + W ) ⪰ ζ 2 I + ( 1 + 2 ∥ W ∗ W ∥ ζ ) − 1 U ∗ U \zeta I+(U+W)^{*}(U+W) \succeq\frac{\zeta}{2}I+\left(1+\frac{2\lVert W^{*}W\rVert}{\zeta}\right)^{-1}U^{*}U ζ I + ( U + W ) ∗ ( U + W ) ⪰ 2 ζ I + ( 1 + ζ 2 ∥ W ∗ W ∥ ) − 1 U ∗ U
Let P m P_m P m be the orthogonal projection onto the first m m m population eigenvectors, let Q m = I − P m Q_m=I-P_m Q m = I − P m , and write Σ ≤ m = P m Σ P m \Sigma_{\leq m}=P_m\Sigma P_m Σ ≤ m = P m Σ P m and Σ > m = Q m Σ Q m \Sigma_{>m}=Q_m\Sigma Q_m Σ > m = Q m Σ Q m . Recall that X k : H → R k X_k:\mathcal{H}\to\mathbb{R}^k X k : H → R k and T k = X k ∗ X k T_k=X_k^{*}X_k T k = X k ∗ X k . Decompose into lower and upper blocks,
X k = U k , m + W k , m , U k , m : = X k P m , W k , m : = X k Q m . X_k=U_{k,m}+W_{k,m},\qquad U_{k,m}:=X_kP_m,\qquad W_{k,m}:=X_kQ_m. X k = U k , m + W k , m , U k , m := X k P m , W k , m := X k Q m .
Letting Z k , m Z_{k,m} Z k , m be the k × m k\times m k × m matrix whose i i i -th row consists of the first m m m coordinates of z i z_i z i (we identify R m \mathbb{R}^m R m with P m H P_m\mathcal{H} P m H through its ordered population eigenbasis), then
U k , m ∗ U k , m = Σ ≤ m 1 / 2 Z k , m ∗ Z k , m Σ ≤ m 1 / 2 . U_{k,m}^{*}U_{k,m}=\Sigma_{\leq m}^{1/2}Z_{k,m}^{*}Z_{k,m}\Sigma_{\leq m}^{1/2}. U k , m ∗ U k , m = Σ ≤ m 1/2 Z k , m ∗ Z k , m Σ ≤ m 1/2 .
Applying Lemma E.10 with U = U k , m U=U_{k,m} U = U k , m and W = W k , m W=W_{k,m} W = W k , m gives the deterministic reduction
A k ( ζ ) ⪯ Σ 1 / 2 ( ζ 2 I + ( 1 + 2 ∥ W k , m ∗ W k , m ∥ ζ ) − 1 Σ ≤ m 1 / 2 Z k , m ∗ Z k , m Σ ≤ m 1 / 2 ) − 1 Σ 1 / 2 . (101) A_k(\zeta)\preceq\Sigma^{1/2}\left(\frac{\zeta}{2}I+\left(1+\frac{2\lVert W_{k,m}^{*}W_{k,m}\rVert}{\zeta}\right)^{-1}\Sigma_{\leq m}^{1/2}Z_{k,m}^{*}Z_{k,m}\Sigma_{\leq m}^{1/2}\right)^{-1}\Sigma^{1/2}. \tag*{(101)} A k ( ζ ) ⪯ Σ 1/2 ( 2 ζ I + ( 1 + ζ 2 ∥ W k , m ∗ W k , m ∥ ) − 1 Σ ≤ m 1/2 Z k , m ∗ Z k , m Σ ≤ m 1/2 ) − 1 Σ 1/2 . (101)
We therefore need a lower bound for Z k , m ∗ Z k , m Z_{k,m}^{*}Z_{k,m} Z k , m ∗ Z k , m and an upper bound for ∥ W k , m ∗ W k , m ∥ \lVert W_{k,m}^{*}W_{k,m}\rVert ∥ W k , m ∗ W k , m ∥ , as in Cheng and Montanari [28 ] , Proof of Lemma 7.2. We first establish the former. Under Assumption 1, we can repeat the fourth-moment verification in Cheng and Montanari [28 ] , Appendix C.2, Part II with sample size k k k and cutoff m m m , which yields isotropic rows of Z k , m Z_{k,m} Z k , m whose unit linear forms have fourth moments bounded by C 2 C^2 C 2 , for C = O C x ( 1 ) C=O_{C_x}(1) C = O C x ( 1 ) . Then we invoke Yaskov [82 ] , Theorem 2.2 with dimension m m m , sample size k k k , and lower-tail parameter k / ( 4 C ) \sqrt{k}/(4C) k / ( 4 C ) . Taking C = C ( C x ) ≥ 1 C=C(C_x)\geq1 C = C ( C x ) ≥ 1 uniformly over the upper block coordinates and setting ρ ( C x ) = ( 16 C ) − 2 \rho(C_x)=(16C)^{-2} ρ ( C x ) = ( 16 C ) − 2 gives 4 C ρ ( C x ) = 1 / 4 4C\sqrt{\rho(C_x)}=1/4 4 C ρ ( C x ) = 1/4 . With this choice, we obtain the following bounds for E ( k , m ) : = { Z k , m ∗ Z k , m ⪰ ( k / 2 ) I m } \mathcal{E}(k,m):=\{Z_{k,m}^{*}Z_{k,m}\succeq(k/2)I_m\} E ( k , m ) := { Z k , m ∗ Z k , m ⪰ ( k /2 ) I m } when m ≤ ρ k m\leq\rho k m ≤ ρ k :
P ( E ( k , m ) c ) ≤ exp { − Ω C x ( k ) } , Z k , m ∗ Z k , m ⪰ k 2 I m , ∥ ( Z k , m ∗ Z k , m ) − 1 ∥ ≤ 2 k . (102) \mathbb{P}\left(\mathcal{E}(k,m)^{c}\right)\leq\exp\{-\Omega_{C_x}(k)\},\qquad Z_{k,m}^{*}Z_{k,m}\succeq\frac{k}{2}I_m,\qquad\left\lVert\left(Z_{k,m}^{*}Z_{k,m}\right)^{-1}\right\rVert\leq\frac{2}{k}. \tag*{(102)} P ( E ( k , m ) c ) ≤ exp { − Ω C x ( k )} , Z k , m ∗ Z k , m ⪰ 2 k I m , ( Z k , m ∗ Z k , m ) − 1 ≤ k 2 . (102)
Here the last two inequalities hold on E ( k , m ) \mathcal{E}(k,m) E ( k , m ) . We next make explicit how this top Gram matrix bound controls the two resolvent norms. On E ( k , m ) \mathcal{E}(k,m) E ( k , m ) , substituting Z k , m ∗ Z k , m ⪰ ( k / 2 ) I m Z_{k,m}^{*}Z_{k,m}\succeq(k/2)I_m Z k , m ∗ Z k , m ⪰ ( k /2 ) I m into Eq. (101) gives
A k ( ζ ) ⪯ Σ 1 / 2 ( ζ 2 I + k 2 ( 1 + 2 ∥ W k , m ∗ W k , m ∥ ζ ) − 1 Σ ≤ m ) − 1 Σ 1 / 2 ⪯ 2 k ( 1 + 2 ∥ W k , m ∗ W k , m ∥ ζ ) P m + 2 ζ Σ > m . A_k(\zeta)\preceq\Sigma^{1/2}\left(\frac{\zeta}{2}I+\frac{k}{2}\left(1+\frac{2\lVert W_{k,m}^{*}W_{k,m}\rVert}{\zeta}\right)^{-1}\Sigma_{\leq m}\right)^{-1}\Sigma^{1/2}\preceq\frac{2}{k}\left(1+\frac{2\lVert W_{k,m}^{*}W_{k,m}\rVert}{\zeta}\right)P_m+\frac{2}{\zeta}\Sigma_{>m}. A k ( ζ ) ⪯ Σ 1/2 ( 2 ζ I + 2 k ( 1 + ζ 2 ∥ W k , m ∗ W k , m ∥ ) − 1 Σ ≤ m ) − 1 Σ 1/2 ⪯ k 2 ( 1 + ζ 2 ∥ W k , m ∗ W k , m ∥ ) P m + ζ 2 Σ > m .
For the second inequality, diagonalizing in the population eigenbasis gives, on the first m m m coordinates,
σ i ζ 2 + k 2 ( 1 + 2 ∥ W k , m ∗ W k , m ∥ / ζ ) − 1 σ i ≤ 2 k ( 1 + 2 ∥ W k , m ∗ W k , m ∥ ζ ) . \frac{\sigma_i}{\frac{\zeta}{2}+\frac{k}{2}\left(1+2\lVert W_{k,m}^{*}W_{k,m}\rVert/\zeta\right)^{-1}\sigma_i}\leq\frac{2}{k}\left(1+\frac{2\lVert W_{k,m}^{*}W_{k,m}\rVert}{\zeta}\right). 2 ζ + 2 k ( 1 + 2 ∥ W k , m ∗ W k , m ∥ / ζ ) − 1 σ i σ i ≤ k 2 ( 1 + ζ 2 ∥ W k , m ∗ W k , m ∥ ) .
On the complement, the eigenvalues of the upper bound are 2 σ i / ζ 2\sigma_i/\zeta 2 σ i / ζ . The two terms in the upper bound of A k ( ζ ) A_k(\zeta) A k ( ζ ) above act on orthogonal subspaces, and the first has rank at most m m m . Taking spectral and squared Frobenius norms respectively therefore gives on the event E ( k , m ) \mathcal{E}(k,m) E ( k , m )
∥ A k ( ζ ) ∥ ≤ 2 k ( 1 + 2 ∥ W k , m ∗ W k , m ∥ ζ ) + 2 σ m + 1 ζ , \lVert A_k(\zeta)\rVert\leq\frac{2}{k}\left(1+\frac{2\lVert W_{k,m}^{*}W_{k,m}\rVert}{\zeta}\right)+\frac{2\sigma_{m+1}}{\zeta}, ∥ A k ( ζ )∥ ≤ k 2 ( 1 + ζ 2 ∥ W k , m ∗ W k , m ∥ ) + ζ 2 σ m + 1 ,
∥ A k ( ζ ) ∥ F 2 ≤ 4 m k 2 ( 1 + 2 ∥ W k , m ∗ W k , m ∥ ζ ) 2 + 4 τ 2 ( m ) ζ 2 . (103) \lVert A_k(\zeta)\rVert_{\mathrm{F}}^2\leq\frac{4m}{k^2}\left(1+\frac{2\lVert W_{k,m}^{*}W_{k,m}\rVert}{\zeta}\right)^2+\frac{4\tau_2(m)}{\zeta^2}. \tag*{(103)} ∥ A k ( ζ ) ∥ F 2 ≤ k 2 4 m ( 1 + ζ 2 ∥ W k , m ∗ W k , m ∥ ) 2 + ζ 2 4 τ 2 ( m ) . (103)
These are the estimates whose L p L^p L p norms we need to control. The random tail norm enters the spectral norm bound linearly and the squared Frobenius norm bound quadratically, so an L 2 p L^{2p} L 2 p bound on ∥ W k , m ∗ W k , m ∥ \lVert W_{k,m}^{*}W_{k,m}\rVert ∥ W k , m ∗ W k , m ∥ will control both contributions on E ( k , m ) \mathcal{E}(k,m) E ( k , m ) . Part II derives a uniform bound on this moment for all 1 ≤ p ≤ κ log k 1\leq p\leq\kappa\log k 1 ≤ p ≤ κ log k . Part III subsequently applies this estimate to obtain the stated upper bounds on the corresponding norms and the associated trace bound, and then controls the remaining contribution using the high probability event E ( k , m ) \mathcal{E}(k,m) E ( k , m ) in Eq. (102) .
Part II: Moments of the lower block. By Eq. (103) , it suffices to prove ∥ ∥ W k , m ∗ W k , m ∥ ∥ L p = O C x , κ ( C A ( k , m ) ) \lVert\lVert W_{k,m}^{*}W_{k,m}\rVert\rVert_{L^p}=O_{C_x,\kappa}(C_A(k,m)) ∥∥ W k , m ∗ W k , m ∥ ∥ L p = O C x , κ ( C A ( k , m )) for 1 ≤ p ≤ 2 κ log k 1\leq p\leq2\kappa\log k 1 ≤ p ≤ 2 κ log k . Replacing p p p by 2 p 2p 2 p will then supply the moment required for the squared Frobenius norm. We first derive a tail bound with an explicit tail parameter a ≥ 1 a\geq1 a ≥ 1 , and then integrate it. The lower block is W k , m ∗ W k , m = ∑ i = 1 k ( Q m x i ) ( Q m x i ) ∗ W_{k,m}^{*}W_{k,m}=\sum_{i=1}^{k}(Q_mx_i)(Q_mx_i)^{*} W k , m ∗ W k , m = ∑ i = 1 k ( Q m x i ) ( Q m x i ) ∗ , with covariance parameters
E [ ( Q m x i ) ( Q m x i ) ∗ ] = Σ > m , ∥ Σ > m ∥ = σ m + 1 , Tr Σ > m = τ 1 ( m ) , ∥ Σ > m ∥ F 2 = τ 2 ( m ) . \mathbb{E}\left[(Q_mx_i)(Q_mx_i)^{*}\right]=\Sigma_{>m},\qquad\lVert\Sigma_{>m}\rVert=\sigma_{m+1},\qquad\operatorname{Tr}\Sigma_{>m}=\tau_1(m),\qquad\lVert\Sigma_{>m}\rVert_{\mathrm{F}}^2=\tau_2(m). E [ ( Q m x i ) ( Q m x i ) ∗ ] = Σ > m , ∥ Σ > m ∥ = σ m + 1 , Tr Σ > m = τ 1 ( m ) , ∥ Σ > m ∥ F 2 = τ 2 ( m ) .
If σ m + 1 = 0 \sigma_{m+1}=0 σ m + 1 = 0 , the lower block vanishes almost surely and its moment bound is immediate. Otherwise, its covariance effective rank is r 1 ( m ) = τ 1 ( m ) / σ m + 1 ≥ 1 r_1(m)=\tau_1(m)/\sigma_{m+1}\geq1 r 1 ( m ) = τ 1 ( m ) / σ m + 1 ≥ 1 . Recall the deterministic threshold
C A ( k , m ) : = k σ m + 1 + k σ m + 1 τ 1 ( m ) log ( k ) log ( k r 1 ( m ) ) + τ 1 ( m ) log ( k ) log ( k r 1 ( m ) ) . C_A(k,m):=k\sigma_{m+1}+\sqrt{k\sigma_{m+1}\tau_1(m)\log(k)\log(kr_1(m))}+\tau_1(m)\log(k)\log(kr_1(m)). C A ( k , m ) := k σ m + 1 + k σ m + 1 τ 1 ( m ) log ( k ) log ( k r 1 ( m )) + τ 1 ( m ) log ( k ) log ( k r 1 ( m )) .
We use the truncation argument of Cheng and Montanari [28 Lemma C.2] , keeping the truncation level dependent on a a a so that the resulting probability bound can be integrated at the required moment orders. Applying Proposition 1, Eq. (4) , with A = Q m A=Q_m A = Q m , we have for u > 0 u>0 u > 0 ,
P ( ∥ Q m x i ∥ 2 > τ 1 ( m ) + u ) ≤ 2 exp { − Ω C x ( u 2 τ 2 ( m ) ∧ u σ m + 1 ) } . \mathbb{P}\left(\lVert Q_mx_i\rVert^2>\tau_1(m)+u\right)\leq2\exp\left\{-\Omega_{C_x}\left(\frac{u^2}{\tau_2(m)}\wedge\frac{u}{\sigma_{m+1}}\right)\right\}. P ( ∥ Q m x i ∥ 2 > τ 1 ( m ) + u ) ≤ 2 exp { − Ω C x ( τ 2 ( m ) u 2 ∧ σ m + 1 u ) } .
Since τ 2 ( m ) ≤ τ 1 ( m ) \sqrt{\tau_2(m)}\leq\tau_1(m) τ 2 ( m ) ≤ τ 1 ( m ) and σ m + 1 ≤ τ 1 ( m ) \sigma_{m+1}\leq\tau_1(m) σ m + 1 ≤ τ 1 ( m ) , we can choose a deterministic truncation level satisfying
L a = C τ 1 ( m ) ( log k + a ) , P ( ∥ Q m x i ∥ 2 > L a ) ≤ 2 e − a k , L_a=C\tau_1(m)(\log k+a),\qquad\mathbb{P}\left(\lVert Q_mx_i\rVert^2>L_a\right)\leq\frac{2e^{-a}}{k}, L a = C τ 1 ( m ) ( log k + a ) , P ( ∥ Q m x i ∥ 2 > L a ) ≤ k 2 e − a ,
where C C C is sufficiently large and depends only on C x C_x C x . Define the independent truncated summands, for i ∈ [ k ] i\in[k] i ∈ [ k ] , by
Y i , a : = ( Q m x i ) ( Q m x i ) ∗ 1 { ∥ Q m x i ∥ 2 ≤ L a } . Y_{i,a}:=(Q_mx_i)(Q_mx_i)^{*}\mathbf{1}\{\lVert Q_mx_i\rVert^2\leq L_a\}. Y i , a := ( Q m x i ) ( Q m x i ) ∗ 1 {∥ Q m x i ∥ 2 ≤ L a } .
Their positivity and rank-one form imply
0 ⪯ Y i , a ⪯ L a I , E Y i , a ⪯ Σ > m , ( Y i , a ) 2 = ∥ Q m x i ∥ 2 Y i , a ⪯ L a Y i , a . 0 \preceq Y_{i,a} \preceq L_a I,\qquad\mathbb{E}Y_{i,a} \preceq\Sigma_{>m},\qquad(Y_{i,a})^2 = \lVert Q_m x_i\rVert^2Y_{i,a} \preceq L_aY_{i,a}. 0 ⪯ Y i , a ⪯ L a I , E Y i , a ⪯ Σ > m , ( Y i , a ) 2 = ∥ Q m x i ∥ 2 Y i , a ⪯ L a Y i , a .
Thus, the centered summands have a norm of at most L a L_a L a , and their variance satisfies
∑ i = 1 k E ( Y i , a − E Y i , a ) 2 = ∑ i = 1 k [ E ( Y i , a ) 2 − ( E Y i , a ) 2 ] ⪯ k L a Σ > m . \sum_{i=1}^{k}\mathbb{E}(Y_{i,a}-\mathbb{E}Y_{i,a})^2=\sum_{i=1}^{k}\left[\mathbb{E}(Y_{i,a})^2-(\mathbb{E}Y_{i,a})^2\right]\preceq kL_a\Sigma_{>m}. i = 1 ∑ k E ( Y i , a − E Y i , a ) 2 = i = 1 ∑ k [ E ( Y i , a ) 2 − ( E Y i , a ) 2 ] ⪯ k L a Σ > m .
The deterministic variance upper bound k L a Σ > m kL_a\Sigma_{>m} k L a Σ > m has norm k L a σ m + 1 kL_a\sigma_{m+1} k L a σ m + 1 and intrinsic dimension r 1 ( m ) r_1(m) r 1 ( m ) . We can then apply the intrinsic-dimension Bernstein inequality in Cheng and Montanari [28 Proof of Lemma C.2] with summand bound L a L_a L a and this variance upper bound for s ≥ k L a σ m + 1 + L a / 3 s\ge\sqrt{kL_a\sigma_{m+1}}+L_a/3 s ≥ k L a σ m + 1 + L a /3 ,
P ( ∥ ∑ i = 1 k ( Y i , a − E Y i , a ) ∥ > s ) ≤ 8 r 1 ( m ) ⋅ exp { − s 2 2 ( k L a σ m + 1 + L a s / 3 ) } . \mathbb{P}\left(\left\lVert\sum_{i=1}^{k}(Y_{i,a}-\mathbb{E}Y_{i,a})\right\rVert>s\right)\le8r_1(m)\cdot\exp\left\{-\frac{s^2}{2(kL_a\sigma_{m+1}+L_as/3)}\right\}. P ( i = 1 ∑ k ( Y i , a − E Y i , a ) > s ) ≤ 8 r 1 ( m ) ⋅ exp { − 2 ( k L a σ m + 1 + L a s /3 ) s 2 } .
Taking s s s to be a sufficiently large constant times the expression below, and using log r 1 ( m ) + a ≥ 1 \log r_1(m)+a\ge1 log r 1 ( m ) + a ≥ 1 , bounds the failure probability by O ( e − a ) O(e^{-a}) O ( e − a ) , which implies
∥ ∑ i = 1 k ( Y i , a − E Y i , a ) ∥ = O ( k L a σ m + 1 ( log r 1 ( m ) + a ) + L a ( log r 1 ( m ) + a ) ) . \left\lVert\sum_{i=1}^{k}(Y_{i,a}-\mathbb{E}Y_{i,a})\right\rVert=O\left(\sqrt{kL_a\sigma_{m+1}(\log r_1(m)+a)}+L_a(\log r_1(m)+a)\right). i = 1 ∑ k ( Y i , a − E Y i , a ) = O ( k L a σ m + 1 ( log r 1 ( m ) + a ) + L a ( log r 1 ( m ) + a ) ) .
By the union bound, with probability at least 1 − 2 e − a 1-2e^{-a} 1 − 2 e − a we have W k , m ∗ W k , m = ∑ i Y i , a W_{k,m}^*W_{k,m}=\sum_iY_{i,a} W k , m ∗ W k , m = ∑ i Y i , a , while ∥ ∑ i = 1 k E Y i , a ∥ ≤ k σ m + 1 \left\lVert\sum_{i=1}^{k}\mathbb{E}Y_{i,a}\right\rVert\le k\sigma_{m+1} ∑ i = 1 k E Y i , a ≤ k σ m + 1 . Adding the mean and the centered fluctuation consequently proves, with probability 1 − O ( e − a ) 1-O(e^{-a}) 1 − O ( e − a ) ,
∥ W k , m ∗ W k , m ∥ = O C x ( k σ m + 1 + k σ m + 1 τ 1 ( m ) ( log k + a ) ( log r 1 ( m ) + a ) + τ 1 ( m ) ( log k + a ) ( log r 1 ( m ) + a ) ) . (104) \left\lVert W_{k,m}^*W_{k,m}\right\rVert
=O_{C_x}\left(k\sigma_{m+1}+\sqrt{k\sigma_{m+1}\tau_1(m)(\log k+a)(\log r_1(m)+a)}+\tau_1(m)(\log k+a)(\log r_1(m)+a)\right). \tag*{(104)} W k , m ∗ W k , m = O C x ( k σ m + 1 + k σ m + 1 τ 1 ( m ) ( log k + a ) ( log r 1 ( m ) + a ) + τ 1 ( m ) ( log k + a ) ( log r 1 ( m ) + a ) ) . (104)
To express this threshold in terms of C A ( k , m ) C_A(k,m) C A ( k , m ) , observe that
( log k + a ) ( log r 1 ( m ) + a ) ≤ ( 1 + a log k ) 2 log ( k ) log ( k r 1 ( m ) ) . (\log k+a)(\log r_1(m)+a)\le\left(1+\frac{a}{\log k}\right)^2\log(k)\log(kr_1(m)). ( log k + a ) ( log r 1 ( m ) + a ) ≤ ( 1 + log k a ) 2 log ( k ) log ( k r 1 ( m )) .
Thus Eq. (104) bounds ∥ W k , m ∗ W k , m ∥ \left\lVert W_{k,m}^*W_{k,m}\right\rVert W k , m ∗ W k , m by O C x ( ( 1 + a / log k ) 2 C A ( k , m ) ) O_{C_x}((1+a/\log k)^2C_A(k,m)) O C x (( 1 + a / log k ) 2 C A ( k , m )) with failure probability O ( e − a ) O(e^{-a}) O ( e − a ) for every a ≥ 1 a\ge1 a ≥ 1 . We are now ready to combine this tail bound with its failure probability and bound the L p L^p L p norm by integrating the tail. For every p ≥ 1 p\ge1 p ≥ 1 ,
∥ ∥ W k , m ∗ W k , m ∥ ∥ L p = { p ∫ 0 ∞ t p − 1 P ( ∥ W k , m ∗ W k , m ∥ > t ) d t } 1 / p ≤ O C x ( C A ( k , m ) ) { ( 1 + 1 log k ) 2 p + 2 p log k ∫ 1 ∞ ( 1 + a log k ) 2 p − 1 e − a d a } 1 / p ≤ O C x ( C A ( k , m ) ) { ∫ 0 ∞ ( 1 + a log k ) 2 p e − a d a } 1 / p ≤ O C x ( C A ( k , m ) ) ( 1 + Γ ( 2 p + 1 ) 1 / ( 2 p ) log k ) 2 . (105) \begin{aligned}
\left\lVert\left\lVert W_{k,m}^*W_{k,m}\right\rVert\right\rVert_{L^p}
&=\left\{p\int_0^\infty t^{p-1}\mathbb{P}\left(\left\lVert W_{k,m}^*W_{k,m}\right\rVert>t\right)\,dt\right\}^{1/p}\\
&\le O_{C_x}(C_A(k,m))\left\{\left(1+\frac{1}{\log k}\right)^{2p}+\frac{2p}{\log k}\int_1^\infty\left(1+\frac{a}{\log k}\right)^{2p-1}e^{-a}\,da\right\}^{1/p}\\
&\le O_{C_x}(C_A(k,m))\left\{\int_0^\infty\left(1+\frac{a}{\log k}\right)^{2p}e^{-a}\,da\right\}^{1/p}\\
&\le O_{C_x}(C_A(k,m))\left(1+\frac{\Gamma(2p+1)^{1/(2p)}}{\log k}\right)^2. \tag*{(105)}
\end{aligned} W k , m ∗ W k , m L p = { p ∫ 0 ∞ t p − 1 P ( W k , m ∗ W k , m > t ) d t } 1/ p ≤ O C x ( C A ( k , m )) { ( 1 + log k 1 ) 2 p + log k 2 p ∫ 1 ∞ ( 1 + log k a ) 2 p − 1 e − a d a } 1/ p ≤ O C x ( C A ( k , m )) { ∫ 0 ∞ ( 1 + log k a ) 2 p e − a d a } 1/ p ≤ O C x ( C A ( k , m )) ( 1 + log k Γ ( 2 p + 1 ) 1/ ( 2 p ) ) 2 . (105)
In (i), we split at the threshold corresponding to a = 1 a=1 a = 1 , use the bound 1 below it, and substitute t = C A ( k , m ) ( 1 + a / log k ) 2 t=C_A(k,m)(1+a/\log k)^2 t = C A ( k , m ) ( 1 + a / log k ) 2 above it. The fixed constants in the tail probability are absorbed using p ≥ 1 p\ge1 p ≥ 1 . For (ii), integration by parts gives
2 p log k ∫ 1 ∞ ( 1 + a log k ) 2 p − 1 e − a d a = − e − 1 ( 1 + 1 log k ) 2 p + ∫ 1 ∞ ( 1 + a log k ) 2 p e − a d a . \frac{2p}{\log k}\int_1^\infty\left(1+\frac{a}{\log k}\right)^{2p-1}e^{-a}\,\mathrm{d}a=-e^{-1}\left(1+\frac{1}{\log k}\right)^{2p}+\int_1^\infty\left(1+\frac{a}{\log k}\right)^{2p}e^{-a}\,\mathrm{d}a. log k 2 p ∫ 1 ∞ ( 1 + log k a ) 2 p − 1 e − a d a = − e − 1 ( 1 + log k 1 ) 2 p + ∫ 1 ∞ ( 1 + log k a ) 2 p e − a d a .
The remaining boundary term is absorbed because ∫ 1 ∞ ( 1 + a / log k ) 2 p e − a d a ≥ e − 1 ( 1 + 1 / log k ) 2 p \int_1^\infty(1+a/\log k)^{2p}e^{-a}\,\mathrm{d}a\ge e^{-1}(1+1/\log k)^{2p} ∫ 1 ∞ ( 1 + a / log k ) 2 p e − a d a ≥ e − 1 ( 1 + 1/ log k ) 2 p and e 1 / p ≤ e e^{1/p}\le e e 1/ p ≤ e . In (iii), we apply Minkowski in L 2 p ( [ 0 , ∞ ) , e − a d a ) L^{2p}([0,\infty),e^{-a}\mathrm{d}a) L 2 p ([ 0 , ∞ ) , e − a d a ) and then square the resulting bound. Finally, using Γ ( 2 p + 1 ) 1 / ( 2 p ) = O ( p ) \Gamma(2p+1)^{1/(2p)}=O(p) Γ ( 2 p + 1 ) 1/ ( 2 p ) = O ( p ) in Eq. (105) proves
∥ ∥ W k , m ∗ W k , m ∥ ∥ L p = O C x , κ ( C A ( k , m ) ) , 1 ≤ p ≤ 2 κ log k . \left\|\left\|W_{k,m}^{*}W_{k,m}\right\|\right\|_{L^p}=O_{C_x,\kappa}(C_A(k,m)),\qquad1\le p\le2\kappa\log k. W k , m ∗ W k , m L p = O C x , κ ( C A ( k , m )) , 1 ≤ p ≤ 2 κ log k .
This is the moment bound required in Part I.
Remark 8. Under the stronger effective-rank condition k = Ω C x ( r 1 ( m ) ) k=\Omega_{C_x}(r_1(m)) k = Ω C x ( r 1 ( m )) , applying [84 Theorem 1 and Example 1] instead gives ∥ W k , m ∗ W k , m ∥ = O C x ( k σ m + 1 ) \left\|W_{k,m}^{*}W_{k,m}\right\|=O_{C_x}(k\sigma_{m+1}) W k , m ∗ W k , m = O C x ( k σ m + 1 ) with failure probability exp { − Ω C x ( k ) } \exp\{-\Omega_{C_x}(k)\} exp { − Ω C x ( k )} . Thus one can remove the logarithmic factors in the tail estimate in this regime. However, we do not pursue this refinement, since it improves the downstream bounds only by logarithmic factors.
Part III: Completing the resolvent estimates. We now apply the moment bound from Part II to the resolvent estimates on E ( k , m ) \mathcal{E}(k,m) E ( k , m ) from Part I, and then control the contribution of E ( k , m ) c \mathcal{E}(k,m)^c E ( k , m ) c . Throughout this part, 1 ≤ p ≤ κ log k 1\le p\le\kappa\log k 1 ≤ p ≤ κ log k . For the first contribution, we apply the Minkowski inequality and the L 2 p L^{2p} L 2 p bound from Part II to get
∥ 1 + 2 ∥ W k , m ∗ W k , m ∥ ζ ∥ L 2 p ≤ 1 + 2 ζ ∥ ∥ W k , m ∗ W k , m ∥ ∥ L 2 p = O C x , κ ( 1 + C A ( k , m ) ζ ) . \left\|1+\frac{2\left\|W_{k,m}^{*}W_{k,m}\right\|}{\zeta}\right\|_{L^{2p}}\le1+\frac{2}{\zeta}\left\|\left\|W_{k,m}^{*}W_{k,m}\right\|\right\|_{L^{2p}}=O_{C_x,\kappa}\left(1+\frac{C_A(k,m)}{\zeta}\right). 1 + ζ 2 W k , m ∗ W k , m L 2 p ≤ 1 + ζ 2 W k , m ∗ W k , m L 2 p = O C x , κ ( 1 + ζ C A ( k , m ) ) .
Thus it controls both terms involving the tail block in Eq. (103) . To obtain the trace estimate as well, taking the trace in the operator comparison preceding that equation gives, on E ( k , m ) \mathcal{E}(k,m) E ( k , m ) ,
Tr A k ( ζ ) ≤ 2 m k ( 1 + 2 ∥ W k , m ∗ W k , m ∥ ζ ) + 2 τ 1 ( m ) ζ . \operatorname{Tr}A_k(\zeta)\le\frac{2m}{k}\left(1+\frac{2\left\|W_{k,m}^{*}W_{k,m}\right\|}{\zeta}\right)+\frac{2\tau_1(m)}{\zeta}. Tr A k ( ζ ) ≤ k 2 m 1 + ζ 2 W k , m ∗ W k , m + ζ 2 τ 1 ( m ) .
We again apply Minkowski’s inequality, using σ m + 1 ≤ C A ( k , m ) / k \sigma_{m+1}\le C_A(k,m)/k σ m + 1 ≤ C A ( k , m ) / k to obtain
( ∫ E ( k , m ) ∥ A k ( ζ ) ∥ p d P ) 1 / p = O C x , κ ( 1 k ( 1 + C A ( k , m ) ζ ) ) , \left(\int_{\mathcal{E}(k,m)}\left\|A_k(\zeta)\right\|^p\mathrm{d}P\right)^{1/p}
=O_{C_x,\kappa}\left(\frac{1}{k}\left(1+\frac{C_A(k,m)}{\zeta}\right)\right), ( ∫ E ( k , m ) ∥ A k ( ζ ) ∥ p d P ) 1/ p = O C x , κ ( k 1 ( 1 + ζ C A ( k , m ) ) ) ,
( ∫ E ( k , m ) ∥ A k ( ζ ) ∥ F 2 p d P ) 1 / p = O C x , κ ( m k 2 ( 1 + C A ( k , m ) ζ ) 2 + τ 2 ( m ) ζ 2 ) , \left(\int_{\mathcal{E}(k,m)}\left\|A_k(\zeta)\right\|_F^{2p}\mathrm{d}P\right)^{1/p}
=O_{C_x,\kappa}\left(\frac{m}{k^2}\left(1+\frac{C_A(k,m)}{\zeta}\right)^2+\frac{\tau_2(m)}{\zeta^2}\right), ( ∫ E ( k , m ) ∥ A k ( ζ ) ∥ F 2 p d P ) 1/ p = O C x , κ ( k 2 m ( 1 + ζ C A ( k , m ) ) 2 + ζ 2 τ 2 ( m ) ) ,
( ∫ E ( k , m ) ( Tr A k ( ζ ) ) p d P ) 1 / p = O C x , κ ( m k ( 1 + C A ( k , m ) ζ ) + τ 1 ( m ) ζ ) . \left(\int_{\mathcal{E}(k,m)}\left(\operatorname{Tr}A_k(\zeta)\right)^p\mathrm{d}P\right)^{1/p}
=O_{C_x,\kappa}\left(\frac{m}{k}\left(1+\frac{C_A(k,m)}{\zeta}\right)+\frac{\tau_1(m)}{\zeta}\right). ( ∫ E ( k , m ) ( Tr A k ( ζ ) ) p d P ) 1/ p = O C x , κ ( k m ( 1 + ζ C A ( k , m ) ) + ζ τ 1 ( m ) ) .
It remains to control the same three quantities on E ( k , m ) c \mathcal{E}(k,m)^c E ( k , m ) c . Here we control deterministically by 0 ⪯ A k ( ζ ) ⪯ Σ / ζ 0\preceq A_k(\zeta)\preceq\Sigma/\zeta 0 ⪯ A k ( ζ ) ⪯ Σ/ ζ . Applying Eq. (102) and using p ≤ κ log k p\le\kappa\log k p ≤ κ log k implies
P ( E ( k , m ) c ) 1 / p ≤ exp { − Ω C x ( k / p ) } ≤ exp { − Ω C x ( k / ( κ log k ) ) } ≤ k − D . \mathbb{P}(\mathcal{E}(k,m)^c)^{1/p}\le\exp\{-\Omega_{C_x}(k/p)\}\le\exp\{-\Omega_{C_x}(k/(\kappa\log k))\}\le k^{-D}. P ( E ( k , m ) c ) 1/ p ≤ exp { − Ω C x ( k / p )} ≤ exp { − Ω C x ( k / ( κ log k ))} ≤ k − D .
Since k / ( log k ) 2 → ∞ k/(\log k)^2 \to\infty k / ( log k ) 2 → ∞ as k → ∞ k \to\infty k → ∞ , the last inequality holds for k = Ω C x , κ , D ( 1 ) k = \Omega_{C_x,\kappa,D}(1) k = Ω C x , κ , D ( 1 ) , uniformly over the stated range of p p p . Using ∥ Σ ∥ = 1 \lVert\Sigma\rVert= 1 ∥ Σ ∥ = 1 , we therefore obtain
( ∫ E ( k , m ) c ∣ A k ( ζ ) ∣ p d P ) 1 / p ≤ P ( E ( k , m ) c ) 1 / p ζ ≤ k − D ζ , ( ∫ E ( k , m ) c ∥ A k ( ζ ) ∥ F 2 p d P ) 1 / p ≤ ∥ Σ ∥ F 2 P ( E ( k , m ) c ) 1 / p ζ 2 ≤ ∥ Σ ∥ F 2 k − D ζ 2 , ( ∫ E ( k , m ) c ( Tr A k ( ζ ) ) p d P ) 1 / p ≤ Tr ( Σ ) P ( E ( k , m ) c ) 1 / p ζ ≤ Tr ( Σ ) k − D ζ . \begin{aligned}
\left(\int_{\mathcal{E}(k,m)^c} \lvert A_k(\zeta)\rvert^p \, \mathrm{d}P\right)^{1/p} &\le\frac{P(\mathcal{E}(k,m)^c)^{1/p}}{\zeta} \le\frac{k^{-D}}{\zeta},\\
\left(\int_{\mathcal{E}(k,m)^c} \lVert A_k(\zeta)\rVert_F^{2p} \, \mathrm{d}P\right)^{1/p} &\le\frac{\lVert\Sigma\rVert_F^2 P(\mathcal{E}(k,m)^c)^{1/p}}{\zeta^2} \le\frac{\lVert\Sigma\rVert_F^2 k^{-D}}{\zeta^2},\\
\left(\int_{\mathcal{E}(k,m)^c} (\operatorname{Tr} A_k(\zeta))^p \, \mathrm{d}P\right)^{1/p} &\le\frac{\operatorname{Tr}(\Sigma)P(\mathcal{E}(k,m)^c)^{1/p}}{\zeta} \le\frac{\operatorname{Tr}(\Sigma)k^{-D}}{\zeta}.
\end{aligned} ( ∫ E ( k , m ) c ∣ A k ( ζ ) ∣ p d P ) 1/ p ( ∫ E ( k , m ) c ∥ A k ( ζ ) ∥ F 2 p d P ) 1/ p ( ∫ E ( k , m ) c ( Tr A k ( ζ ) ) p d P ) 1/ p ≤ ζ P ( E ( k , m ) c ) 1/ p ≤ ζ k − D , ≤ ζ 2 ∥ Σ ∥ F 2 P ( E ( k , m ) c ) 1/ p ≤ ζ 2 ∥ Σ ∥ F 2 k − D , ≤ ζ Tr ( Σ ) P ( E ( k , m ) c ) 1/ p ≤ ζ Tr ( Σ ) k − D .
Finally, splitting each quantity over E ( k , m ) \mathcal{E}(k,m) E ( k , m ) and E ( k , m ) c \mathcal{E}(k,m)^c E ( k , m ) c and applying the Minkowski inequality in L p L^p L p one last time proves all stated results in Lemma E.3, Eqs. (45) uniformly in p p p .
Proof of Lemma E.4
To bound the one-step difference, we compare both endpoints with their common Euler approximation. Their leading terms then cancel exactly, leaving two remainders to estimate. Recall that δ k + 1 = ( 1 + G ‾ k + 1 ( z , w k + 1 ) ) − 1 \delta_{k+1} = (1+\overline{G}_{k+1}(z,w_{k+1}))^{-1} δ k + 1 = ( 1 + G k + 1 ( z , w k + 1 ) ) − 1 . For real z < 0 z<0 z < 0 , the path parameters are nonpositive, so T j − w k Σ − z I ⪰ ∣ z ∣ I \mathbf{T}_j-w_k\Sigma-zI \succeq\lvert z\rvert I T j − w k Σ − z I ⪰ ∣ z ∣ I and both the quadratic-form and expected-trace denominators are at least one. Thus the resolvent identities and moment estimates below apply directly in this case as well. We insert the Euler approximation as follows:
G ‾ k + 1 ( z , w k + 1 ) ⟷ G ‾ k ( z , w k + 1 ) − δ k + 1 ∂ ∂ w G ‾ k ( z , w k + 1 ) ⟷ G ‾ k ( z , w k ) . (106) \overline{G}_{k+1}(z,w_{k+1}) \ \longleftrightarrow\ \overline{G}_k(z,w_{k+1})-\delta_{k+1}\frac{\partial}{\partial w}\overline{G}_k(z,w_{k+1}) \ \longleftrightarrow\ \overline{G}_k(z,w_k). \tag*{(106)} G k + 1 ( z , w k + 1 ) ⟷ G k ( z , w k + 1 ) − δ k + 1 ∂ w ∂ G k ( z , w k + 1 ) ⟷ G k ( z , w k ) . (106)
The first comparison adds one sample while keeping w = w k + 1 w=w_{k+1} w = w k + 1 fixed, whose leading contribution is
G ‾ k + 1 ( z , w k + 1 ) − G ‾ k ( z , w k + 1 ) = − δ k + 1 ∂ ∂ w G ‾ k ( z , w k + 1 ) + higher-order fluctuation . \overline{G}_{k+1}(z,w_{k+1})-\overline{G}_k(z,w_{k+1})=-\delta_{k+1}\frac{\partial}{\partial w}\overline{G}_k(z,w_{k+1})+\text{higher-order fluctuation}. G k + 1 ( z , w k + 1 ) − G k ( z , w k + 1 ) = − δ k + 1 ∂ w ∂ G k ( z , w k + 1 ) + higher-order fluctuation .
The second comparison moves w k + 1 w_{k+1} w k + 1 to w k = w k + 1 − δ k + 1 w_k=w_{k+1}-\delta_{k+1} w k = w k + 1 − δ k + 1 while keeping the sample size fixed, so
G ‾ k ( z , w k ) − G ‾ k ( z , w k + 1 ) = − δ k + 1 ∂ ∂ w G ‾ k ( z , w k + 1 ) + higher-order resolvent remainder . \overline{G}_k(z,w_k)-\overline{G}_k(z,w_{k+1})=-\delta_{k+1}\frac{\partial}{\partial w}\overline{G}_k(z,w_{k+1})+\text{higher-order resolvent remainder}. G k ( z , w k ) − G k ( z , w k + 1 ) = − δ k + 1 ∂ w ∂ G k ( z , w k + 1 ) + higher-order resolvent remainder .
The two increments have the same leading term, which cancels when we subtract them. Only the two higher-order remainders remain. The identity δ k + 1 = ( 1 + G ‾ k + 1 ( z , w k + 1 ) ) − 1 \delta_{k+1}=(1+\overline{G}_{k+1}(z,w_{k+1}))^{-1} δ k + 1 = ( 1 + G k + 1 ( z , w k + 1 ) ) − 1 is exactly what matches the two leading terms. Viewed in this way, the comparisons in (106) are the expected-trace analogue of the cancellation strategy in Cheng and Montanari [28, Proof of Theorem 6, Part II]. Their strategy requires inserting an auxiliary resolvent between consecutive iterates and splitting the increment into a rank-one update and a coefficient update, and their update rule explicitly cancels these conditional leading terms.
In the comparisons in Eq. (106) , subtracting the common Euler approximation leaves a nonlinear rank-one fluctuation in the first comparison and a resolvent remainder in the second. We now derive their exact formulas and bound them separately.
Part I: Explicit higher-order terms χ 1 \chi_1 χ 1 and χ 2 \chi_2 χ 2 . We first write the two endpoints and their common Euler approximation as expected traces:
G ‾ k + 1 ( z , w k + 1 ) = E Tr ( Σ 1 / 2 R k ( z , w k + 1 ) Σ 1 / 2 ) , \overline{G}_{k+1}(z,w_{k+1})=\mathbb{E}\operatorname{Tr}\left(\Sigma^{1/2}R_k(z,w_{k+1})\Sigma^{1/2}\right), G k + 1 ( z , w k + 1 ) = E Tr ( Σ 1/2 R k ( z , w k + 1 ) Σ 1/2 ) ,
G ‾ k ( z , w k + 1 ) − δ k + 1 ∂ ∂ w G ‾ k ( z , w k + 1 ) = E Tr ( Σ 1 2 R k − 1 ( z , w k + 1 ) Σ 1 2 ) − δ k + 1 E Tr ( Σ 1 2 R k − 1 ( z , w k + 1 ) Σ 1 2 ) 2 , \overline{G}_k(z,w_{k+1})-\delta_{k+1}\frac{\partial}{\partial w}\overline{G}_k(z,w_{k+1})=\mathbb{E}\operatorname{Tr}\left(\boldsymbol{\Sigma}^{\frac{1}{2}}R_{k-1}(z,w_{k+1})\boldsymbol{\Sigma}^{\frac{1}{2}}\right)-\delta_{k+1}\mathbb{E}\operatorname{Tr}\left(\boldsymbol{\Sigma}^{\frac{1}{2}}R_{k-1}(z,w_{k+1})\boldsymbol{\Sigma}^{\frac{1}{2}}\right)^2, G k ( z , w k + 1 ) − δ k + 1 ∂ w ∂ G k ( z , w k + 1 ) = E Tr ( Σ 2 1 R k − 1 ( z , w k + 1 ) Σ 2 1 ) − δ k + 1 E Tr ( Σ 2 1 R k − 1 ( z , w k + 1 ) Σ 2 1 ) 2 ,
G ‾ k ( z , w k ) = E Tr ( Σ 1 2 R k − 1 ( z , w k ) Σ 1 2 ) . \overline{G}_k(z,w_k)=\mathbb{E}\operatorname{Tr}\left(\boldsymbol{\Sigma}^{\frac{1}{2}}R_{k-1}(z,w_k)\boldsymbol{\Sigma}^{\frac{1}{2}}\right). G k ( z , w k ) = E Tr ( Σ 2 1 R k − 1 ( z , w k ) Σ 2 1 ) .
Using the weighted resolvent notation R ~ j ( z , w ) \widetilde{R}_j(z,w) R j ( z , w ) from Lemma E.2 , applying the Sherman–Morrison identity, and then using the isotropy of z k z_k z k conditional on F k − 1 \mathcal{F}_{k-1} F k − 1 gives
G ‾ k + 1 ( z , w k + 1 ) = E Tr ( R ~ k − 1 ( z , w k + 1 ) ) − E [ z k ∗ ( R ~ k − 1 ( z , w k + 1 ) ) 2 z k 1 + z k ∗ R ~ k − 1 ( z , w k + 1 ) z k ] = E Tr ( R ~ k − 1 ( z , w k + 1 ) ) − δ k + 1 E Tr ( R ~ k − 1 ( z , w k + 1 ) ) 2 + ( E [ z k ∗ ( R ~ k − 1 ( z , w k + 1 ) ) 2 z k ] 1 + G ‾ k + 1 ( z , w k + 1 ) − E [ z k ∗ ( R ~ k − 1 ( z , w k + 1 ) ) 2 z k 1 + z k ∗ R ~ k − 1 ( z , w k + 1 ) z k ] ) . \begin{aligned}
\overline{G}_{k+1}(z,w_{k+1})
&=\mathbb{E}\operatorname{Tr}\left(\widetilde{R}_{k-1}(z,w_{k+1})\right)
-\mathbb{E}\left[\frac{z_k^*\left(\widetilde{R}_{k-1}(z,w_{k+1})\right)^2z_k}{1+z_k^*\widetilde{R}_{k-1}(z,w_{k+1})z_k}\right]\\
&=\mathbb{E}\operatorname{Tr}\left(\widetilde{R}_{k-1}(z,w_{k+1})\right)
-\delta_{k+1}\mathbb{E}\operatorname{Tr}\left(\widetilde{R}_{k-1}(z,w_{k+1})\right)^2\\
&\quad+\left(\frac{\mathbb{E}\left[z_k^*\left(\widetilde{R}_{k-1}(z,w_{k+1})\right)^2z_k\right]}{1+\overline{G}_{k+1}(z,w_{k+1})}
-\mathbb{E}\left[\frac{z_k^*\left(\widetilde{R}_{k-1}(z,w_{k+1})\right)^2z_k}{1+z_k^*\widetilde{R}_{k-1}(z,w_{k+1})z_k}\right]\right).
\end{aligned} G k + 1 ( z , w k + 1 ) = E Tr ( R k − 1 ( z , w k + 1 ) ) − E 1 + z k ∗ R k − 1 ( z , w k + 1 ) z k z k ∗ ( R k − 1 ( z , w k + 1 ) ) 2 z k = E Tr ( R k − 1 ( z , w k + 1 ) ) − δ k + 1 E Tr ( R k − 1 ( z , w k + 1 ) ) 2 + 1 + G k + 1 ( z , w k + 1 ) E [ z k ∗ ( R k − 1 ( z , w k + 1 ) ) 2 z k ] − E 1 + z k ∗ R k − 1 ( z , w k + 1 ) z k z k ∗ ( R k − 1 ( z , w k + 1 ) ) 2 z k .
It is equally instructive to expand the third equation around the same trace at w k + 1 w_{k+1} w k + 1 . Applying the resolvent identity twice gives
G ‾ k ( z , w k ) = E Tr ( R ~ k − 1 ( z , w k + 1 ) ) − δ k + 1 E Tr ( R ~ k − 1 ( z , w k + 1 ) ) 2 + δ k + 1 2 E Tr [ R ~ k − 1 ( z , w k ) ( R ~ k − 1 ( z , w k + 1 ) ) 2 ] . \overline{G}_k(z,w_k)=\mathbb{E}\operatorname{Tr}\left(\widetilde{R}_{k-1}(z,w_{k+1})\right)-\delta_{k+1}\mathbb{E}\operatorname{Tr}\left(\widetilde{R}_{k-1}(z,w_{k+1})\right)^2+\delta_{k+1}^2\mathbb{E}\operatorname{Tr}\left[\widetilde{R}_{k-1}(z,w_k)\left(\widetilde{R}_{k-1}(z,w_{k+1})\right)^2\right]. G k ( z , w k ) = E Tr ( R k − 1 ( z , w k + 1 ) ) − δ k + 1 E Tr ( R k − 1 ( z , w k + 1 ) ) 2 + δ k + 1 2 E Tr [ R k − 1 ( z , w k ) ( R k − 1 ( z , w k + 1 ) ) 2 ] .
Thus the second equation by Euler approximation exactly recover the common first-order approximation to both endpoints in the two displays above, and χ 1 , χ 2 \chi_1,\chi_2 χ 1 , χ 2 are the remaining two higher-order terms. From the resolvent identity, we can write
R ~ k − 1 ( z , w k + 1 ) − R ~ k − 1 ( z , w k ) = δ k + 1 R ~ k − 1 ( z , w k + 1 ) R ~ k − 1 ( z , w k ) = δ k + 1 R ~ k − 1 ( z , w k ) R ~ k − 1 ( z , w k + 1 ) . \begin{aligned}
\widetilde{R}_{k-1}(z,w_{k+1})-\widetilde{R}_{k-1}(z,w_k)
=\delta_{k+1}\widetilde{R}_{k-1}(z,w_{k+1})\widetilde{R}_{k-1}(z,w_k) \\
&=\delta_{k+1}\widetilde{R}_{k-1}(z,w_k)\widetilde{R}_{k-1}(z,w_{k+1}).
\end{aligned} R k − 1 ( z , w k + 1 ) − R k − 1 ( z , w k ) = δ k + 1 R k − 1 ( z , w k + 1 ) R k − 1 ( z , w k ) = δ k + 1 R k − 1 ( z , w k ) R k − 1 ( z , w k + 1 ) .
In this notation, the comparisons in (106) combine into the exact decomposition
G ‾ k + 1 ( z , w k + 1 ) − G ‾ k ( z , w k ) = { G ‾ k + 1 ( z , w k + 1 ) − ( G ‾ k ( z , w k + 1 ) − δ k + 1 ∂ ∂ w G ‾ k ( z , w k + 1 ) ) } + { ( G ‾ k ( z , w k + 1 ) − δ k + 1 ∂ ∂ w G ‾ k ( z , w k + 1 ) ) − G ‾ k ( z , w k ) } = χ 1 − χ 2 . (107) \begin{aligned}
\overline{G}_{k+1}(z,w_{k+1})-\overline{G}_k(z,w_k)
&=\left\{\overline{G}_{k+1}(z,w_{k+1})-\left(\overline{G}_k(z,w_{k+1})-\delta_{k+1}\frac{\partial}{\partial w}\overline{G}_k(z,w_{k+1})\right)\right\}\\
&\quad+\left\{\left(\overline{G}_k(z,w_{k+1})-\delta_{k+1}\frac{\partial}{\partial w}\overline{G}_k(z,w_{k+1})\right)-\overline{G}_k(z,w_k)\right\}\\
&=\chi_1-\chi_2. \tag*{(107)}
\end{aligned} G k + 1 ( z , w k + 1 ) − G k ( z , w k ) = { G k + 1 ( z , w k + 1 ) − ( G k ( z , w k + 1 ) − δ k + 1 ∂ w ∂ G k ( z , w k + 1 ) ) } + { ( G k ( z , w k + 1 ) − δ k + 1 ∂ w ∂ G k ( z , w k + 1 ) ) − G k ( z , w k ) } = χ 1 − χ 2 . (107)
The two remainders also take the following equivalent forms,
χ 1 = E [ z k ∗ R ~ k − 1 ( z , w k + 1 ) 2 z k ] 1 + G ‾ k + 1 ( z , w k + 1 ) − E [ z k ∗ R ~ k − 1 ( z , w k + 1 ) 2 z k 1 + z k ∗ R ~ k − 1 ( z , w k + 1 ) z k ] , \chi_1=\frac{\mathbb{E}\left[z_k^*\widetilde{R}_{k-1}(z,w_{k+1})^2z_k\right]}{1+\overline{G}_{k+1}(z,w_{k+1})}
-\mathbb{E}\left[\frac{z_k^*\widetilde{R}_{k-1}(z,w_{k+1})^2z_k}{1+z_k^*\widetilde{R}_{k-1}(z,w_{k+1})z_k}\right], χ 1 = 1 + G k + 1 ( z , w k + 1 ) E [ z k ∗ R k − 1 ( z , w k + 1 ) 2 z k ] − E [ 1 + z k ∗ R k − 1 ( z , w k + 1 ) z k z k ∗ R k − 1 ( z , w k + 1 ) 2 z k ] ,
χ 2 = δ k + 1 2 E Tr ( R ~ k − 1 ( z , w k ) R ~ k − 1 ( z , w k + 1 ) 2 ) . \chi_2=\delta_{k+1}^2\mathbb{E}\operatorname{Tr}\left(\widetilde{R}_{k-1}(z,w_k)\widetilde{R}_{k-1}(z,w_{k+1})^2\right). χ 2 = δ k + 1 2 E Tr ( R k − 1 ( z , w k ) R k − 1 ( z , w k + 1 ) 2 ) .
The parameter error term χ 2 \chi_2 χ 2 is immediate to control. From its explicit formula, applying the trace Cauchy–Schwarz and then the Hölder’s inequality gives
∣ χ 2 ∣ ≤ ∣ δ k + 1 ∣ 2 E [ ∥ R ~ k − 1 ( z , w k ) ∥ F ∥ R ~ k − 1 ( z , w k + 1 ) ∥ ∥ R ~ k − 1 ( z , w k + 1 ) ∥ F ] = O ( ∣ δ k + 1 ∣ 2 γ k φ k 2 ) . (108) \left|\chi_2\right|\leq\left|\delta_{k+1}\right|^2\mathbb{E}\left[\left\|\widetilde{R}_{k-1}(z,w_k)\right\|_F\left\|\widetilde{R}_{k-1}(z,w_{k+1})\right\|\left\|\widetilde{R}_{k-1}(z,w_{k+1})\right\|_F\right]=O\left(\left|\delta_{k+1}\right|^2\gamma_k\varphi_k^2\right). \tag*{(108)} ∣ χ 2 ∣ ≤ ∣ δ k + 1 ∣ 2 E [ R k − 1 ( z , w k ) F R k − 1 ( z , w k + 1 ) R k − 1 ( z , w k + 1 ) F ] = O ( ∣ δ k + 1 ∣ 2 γ k φ k 2 ) . (108)
In contrast, the analysis of χ 1 \chi_1 χ 1 requires a more fine-grained approach. It is a nonlinear rank-one fluctuation whose numerator and denominator depend on the same fresh sample. The subsequent parts analyze χ 1 \chi_1 χ 1 in detail, and we first present a useful decomposition of χ 1 \chi_1 χ 1 .
Part II: A bias-covariance decomposition for χ 1 \chi_1 χ 1 . We start from the expression for χ 1 \chi_1 χ 1 obtained in Part I. In the remainder of this proof and the proof of Lemma E.11, we suppress the fixed arguments ( z , w k + 1 ) (z,w_{k+1}) ( z , w k + 1 ) from G ‾ k + 1 \overline{G}_{k+1} G k + 1 and the weighted resolvents R ~ j ( z , w k + 1 ) \widetilde{R}_j(z,w_{k+1}) R j ( z , w k + 1 ) . Set
U k : = z k ∗ R ~ k − 1 z k , V k : = z k ∗ R ~ k − 1 2 z k 1 + z k ∗ R ~ k − 1 z k . U_k := z_k^*\widetilde{R}_{k-1}z_k,\qquad V_k := \frac{z_k^*\widetilde{R}_{k-1}^2z_k}{1+z_k^*\widetilde{R}_{k-1}z_k}. U k := z k ∗ R k − 1 z k , V k := 1 + z k ∗ R k − 1 z k z k ∗ R k − 1 2 z k .
Using δ k + 1 ( 1 + G ‾ k + 1 ) = 1 \delta_{k+1}(1+\overline{G}_{k+1})=1 δ k + 1 ( 1 + G k + 1 ) = 1 gives
χ 1 = δ k + 1 E [ z k ∗ R ~ k − 1 2 z k ( 1 − 1 + G ‾ k + 1 1 + z k ∗ R ~ k − 1 z k ) ] = δ k + 1 E [ ( U k − G ‾ k + 1 ) V k ] . \chi_1=\delta_{k+1}\mathbb{E}\left[z_k^*\widetilde{R}_{k-1}^2z_k\left(1-\frac{1+\overline{G}_{k+1}}{1+z_k^*\widetilde{R}_{k-1}z_k}\right)\right]=\delta_{k+1}\mathbb{E}\left[(U_k-\overline{G}_{k+1})V_k\right]. χ 1 = δ k + 1 E [ z k ∗ R k − 1 2 z k ( 1 − 1 + z k ∗ R k − 1 z k 1 + G k + 1 ) ] = δ k + 1 E [ ( U k − G k + 1 ) V k ] .
We can simplify the expectation of the first factor U k − G ‾ k + 1 U_k-\overline{G}_{k+1} U k − G k + 1 using E [ z k z k ∗ ] = I \mathbb{E}[z_kz_k^*]=I E [ z k z k ∗ ] = I and then applying Sherman-Morrison on R ~ k − 1 \widetilde{R}_{k-1} R k − 1 :
E U k − G ‾ k + 1 = E Tr ( R ~ k − 1 ) − G ‾ k + 1 = E V k . (109) \mathbb{E}U_k-\overline{G}_{k+1}=\mathbb{E}\operatorname{Tr}(\widetilde{R}_{k-1})-\overline{G}_{k+1}=\mathbb{E}V_k. \tag*{(109)} E U k − G k + 1 = E Tr ( R k − 1 ) − G k + 1 = E V k . (109)
Subtracting the product of the means therefore yields the exact bias-covariance decomposition
χ 1 = δ k + 1 ( ( E V k ) 2 + Cov ( U k , V k ) ) . (110) \chi_1=\delta_{k+1}\left((\mathbb{E}V_k)^2+\operatorname{Cov}(U_k,V_k)\right). \tag*{(110)} χ 1 = δ k + 1 ( ( E V k ) 2 + Cov ( U k , V k ) ) . (110)
The bias term is relatively easy to estimate. Applying the rank-one identity gives a direct bound for the bias. The weighted resolvent R ~ k = R ~ k ( z , w k + 1 ) \widetilde{R}_k=\widetilde{R}_k(z,w_{k+1}) R k = R k ( z , w k + 1 ) is obtained from R ~ k − 1 \widetilde{R}_{k-1} R k − 1 by adding the k k k -th sample. Applying Sherman-Morrison shows V k = Tr ( R ~ k − 1 − R ~ k ) V_k=\operatorname{Tr}(\widetilde{R}_{k-1}-\widetilde{R}_k) V k = Tr ( R k − 1 − R k ) , and
∣ V k ∣ = ∣ Tr ( R ~ k − 1 − R ~ k ) ∣ ≤ ∥ R ~ k − 1 ∥ + ∥ R ~ k ∥ . (111) \lvert V_k\rvert=\lvert\operatorname{Tr}(\widetilde{R}_{k-1}-\widetilde{R}_k)\rvert\leq\lVert\widetilde{R}_{k-1}\rVert+\lVert\widetilde{R}_k\rVert. \tag*{(111)} ∣ V k ∣ = ∣ Tr ( R k − 1 − R k )∣ ≤ ∥ R k − 1 ∥ + ∥ R k ∥ . (111)
The last inequality is a consequence of R ~ k − 1 − R ~ k \widetilde{R}_{k-1}-\widetilde{R}_k R k − 1 − R k having rank at most one. Both resolvents are indexed by I k I_k I k , so ∥ V k ∥ L 4 = O C x ( γ k ) \lVert V_k\rVert_{L^4}=O_{C_x}(\gamma_k) ∥ V k ∥ L 4 = O C x ( γ k ) . Consequently,
∣ δ k + 1 ( E V k ) 2 ∣ = O C x ( ∣ δ k + 1 ∣ γ k 2 ) . (112) \left\lvert\delta_{k+1}(\mathbb{E}V_k)^2\right\rvert=O_{C_x}\left(\lvert\delta_{k+1}\rvert\gamma_k^2\right). \tag*{(112)} δ k + 1 ( E V k ) 2 = O C x ( ∣ δ k + 1 ∣ γ k 2 ) . (112)
Thus it remains only to control the covariance term, which we analyze in the next part.
Part III: Bounding the covariance term δ k + 1 Cov ( U k , V k ) \delta_{k+1}\operatorname{Cov}(U_k,V_k) δ k + 1 Cov ( U k , V k ) . We first derive a decomposition around the stable denominator 1 + G ‾ k + 1 1+\overline{G}_{k+1} 1 + G k + 1 to replace the random denominator 1 + U k 1+U_k 1 + U k . The decomposition allows us to control the covariance between U k U_k U k and V k V_k V k by U k U_k U k with smaller and low-rank terms. We first derive the exact denominator expansion that organizes the proof. Since 1 + U k = 1 + G ‾ k + 1 + ( U k − G ‾ k + 1 ) 1+U_k=1+\overline{G}_{k+1}+(U_k-\overline{G}_{k+1}) 1 + U k = 1 + G k + 1 + ( U k − G k + 1 ) , we can explicitly expand
1 1 + U k = 1 1 + G ‾ k + 1 − U k − G ‾ k + 1 ( 1 + G ‾ k + 1 ) ( 1 + U k ) = 1 1 + G ‾ k + 1 − U k − G ‾ k + 1 ( 1 + G ‾ k + 1 ) 2 + ( U k − G ‾ k + 1 ) 2 ( 1 + G ‾ k + 1 ) 2 ( 1 + U k ) . \frac{1}{1+U_k}=\frac{1}{1+\overline{G}_{k+1}}-\frac{U_k-\overline{G}_{k+1}}{(1+\overline{G}_{k+1})(1+U_k)}
=\frac{1}{1+\overline{G}_{k+1}}-\frac{U_k-\overline{G}_{k+1}}{(1+\overline{G}_{k+1})^2}+\frac{(U_k-\overline{G}_{k+1})^2}{(1+\overline{G}_{k+1})^2(1+U_k)}. 1 + U k 1 = 1 + G k + 1 1 − ( 1 + G k + 1 ) ( 1 + U k ) U k − G k + 1 = 1 + G k + 1 1 − ( 1 + G k + 1 ) 2 U k − G k + 1 + ( 1 + G k + 1 ) 2 ( 1 + U k ) ( U k − G k + 1 ) 2 .
Multiplying this identity by z k ∗ R ~ k − 1 2 z k z_k^*\widetilde{R}_{k-1}^2z_k z k ∗ R k − 1 2 z k on both sides, by the definition of V k V_k V k we can write
V k = z k ∗ R ~ k − 1 2 z k 1 + U k = z k ∗ R ~ k − 1 2 z k 1 + G ‾ k + 1 − z k ∗ R ~ k − 1 2 z k ( U k − G ‾ k + 1 ) ( 1 + G ‾ k + 1 ) 2 + V k ( U k − G ‾ k + 1 ) 2 ( 1 + G ‾ k + 1 ) 2 . (113) V_k=\frac{z_k^*\widetilde{R}_{k-1}^2z_k}{1+U_k}
=\frac{z_k^*\widetilde{R}_{k-1}^2z_k}{1+\overline{G}_{k+1}}
-\frac{z_k^*\widetilde{R}_{k-1}^2z_k(U_k-\overline{G}_{k+1})}{(1+\overline{G}_{k+1})^2}
+\frac{V_k(U_k-\overline{G}_{k+1})^2}{(1+\overline{G}_{k+1})^2}. \tag*{(113)} V k = 1 + U k z k ∗ R k − 1 2 z k = 1 + G k + 1 z k ∗ R k − 1 2 z k − ( 1 + G k + 1 ) 2 z k ∗ R k − 1 2 z k ( U k − G k + 1 ) + ( 1 + G k + 1 ) 2 V k ( U k − G k + 1 ) 2 . (113)
Taking covariance with U k U_k U k in Eq. (113) and by its linearity in the second argument, we derive
Cov ( U k , V k ) = Cov ( U k , z k ∗ R ~ k − 1 2 z k ) 1 + G ‾ k + 1 − Cov ( U k , z k ∗ R ~ k − 1 2 z k ( U k − G ‾ k + 1 ) ) ( 1 + G ‾ k + 1 ) 2 + Cov ( U k , V k ( U k − G ‾ k + 1 ) 2 ) ( 1 + G ‾ k + 1 ) 2 . = δ k + 1 Cov ( U k , z k ∗ R ~ k − 1 2 z k ) − δ k + 1 2 Cov ( U k , z k ∗ R ~ k − 1 2 z k ( U k − G ‾ k + 1 ) ) + δ k + 1 2 Cov ( U k , V k ( U k − G ‾ k + 1 ) 2 ) . (114) \begin{aligned}
\operatorname{Cov}(U_k,V_k)=\frac{\operatorname{Cov}(U_k,z_k^*\widetilde{R}_{k-1}^2z_k)}{1+\overline{G}_{k+1}}-\frac{\operatorname{Cov}(U_k,z_k^*\widetilde{R}_{k-1}^2z_k(U_k-\overline{G}_{k+1}))}{(1+\overline{G}_{k+1})^2}+\frac{\operatorname{Cov}(U_k,V_k(U_k-\overline{G}_{k+1})^2)}{(1+\overline{G}_{k+1})^2}. \\
&= \delta_{k+1}\operatorname{Cov}(U_k,z_k^{*}\widetilde{R}_{k-1}^{2}z_k)-\delta_{k+1}^{2}\operatorname{Cov}(U_k,z_k^{*}\widetilde{R}_{k-1}^{2}z_k(U_k-\overline{G}_{k+1}))+\delta_{k+1}^{2}\operatorname{Cov}(U_k,V_k(U_k-\overline{G}_{k+1})^{2}). \tag*{(114)}
\end{aligned} Cov ( U k , V k ) = 1 + G k + 1 Cov ( U k , z k ∗ R k − 1 2 z k ) − ( 1 + G k + 1 ) 2 Cov ( U k , z k ∗ R k − 1 2 z k ( U k − G k + 1 )) + ( 1 + G k + 1 ) 2 Cov ( U k , V k ( U k − G k + 1 ) 2 ) . = δ k + 1 Cov ( U k , z k ∗ R k − 1 2 z k ) − δ k + 1 2 Cov ( U k , z k ∗ R k − 1 2 z k ( U k − G k + 1 )) + δ k + 1 2 Cov ( U k , V k ( U k − G k + 1 ) 2 ) . (114)
This decomposition replaces the random denominator 1 + U k 1+U_k 1 + U k with powers of the deterministic denominator 1 + G ‾ k + 1 1+\overline{G}_{k+1} 1 + G k + 1 . Thus Cov ( U k , V k ) = δ k + 1 ψ 1 − δ k + 1 2 ψ 2 + δ k + 1 2 ψ 3 \operatorname{Cov}(U_k,V_k)=\delta_{k+1}\psi_1-\delta_{k+1}^{2}\psi_2+\delta_{k+1}^{2}\psi_3 Cov ( U k , V k ) = δ k + 1 ψ 1 − δ k + 1 2 ψ 2 + δ k + 1 2 ψ 3 . We bound the three covariances separately from the deterministic factors involving δ k + 1 \delta_{k+1} δ k + 1 , which we collect in the following lemma. We defer its proof to Appendix E.4.5.
Lemma E.11. Under the assumptions of Lemma E.4 , let ψ 1 , ψ 2 , ψ 3 \psi_1,\psi_2,\psi_3 ψ 1 , ψ 2 , ψ 3 be the covariances defined in Eq. (114) . Then
∣ ψ 1 ∣ = O C x ( k γ k 3 + γ k φ k 2 ) , ∣ ψ 2 ∣ = O C x ( φ k 2 ( k γ k + φ k ) 2 ) , ∣ ψ 3 ∣ = O C x ( γ k ( k γ k + φ k ) 3 ) . |\psi_1|=O_{C_x}\left(k\gamma_k^3+\gamma_k\varphi_k^2\right),\qquad|\psi_2|=O_{C_x}\left(\varphi_k^2(\sqrt{k}\gamma_k+\varphi_k)^2\right),\qquad|\psi_3|=O_{C_x}\left(\gamma_k(\sqrt{k}\gamma_k+\varphi_k)^3\right). ∣ ψ 1 ∣ = O C x ( k γ k 3 + γ k φ k 2 ) , ∣ ψ 2 ∣ = O C x ( φ k 2 ( k γ k + φ k ) 2 ) , ∣ ψ 3 ∣ = O C x ( γ k ( k γ k + φ k ) 3 ) .
We are now ready to combine all bounds in the final step.
Part IV: Combining the bounds. Applying Lemma E.11 to the decomposition in Eq. (114) and using the triangle inequality gives
∣ δ k + 1 Cov ( U k , V k ) ∣ = O C x ( ∣ δ k + 1 ∣ 2 ⋅ ( k γ k 3 + γ k φ k 2 ) + ∣ δ k + 1 ∣ 3 ⋅ ( φ k 2 ( k γ k + φ k ) 2 + γ k ( k γ k + φ k ) 3 ) ) . (115) |\delta_{k+1}\operatorname{Cov}(U_k,V_k)|=O_{C_x}\left(|\delta_{k+1}|^2\cdot(k\gamma_k^3+\gamma_k\varphi_k^2)+|\delta_{k+1}|^3\cdot\left(\varphi_k^2(\sqrt{k}\gamma_k+\varphi_k)^2+\gamma_k(\sqrt{k}\gamma_k+\varphi_k)^3\right)\right). \tag*{(115)} ∣ δ k + 1 Cov ( U k , V k ) ∣ = O C x ( ∣ δ k + 1 ∣ 2 ⋅ ( k γ k 3 + γ k φ k 2 ) + ∣ δ k + 1 ∣ 3 ⋅ ( φ k 2 ( k γ k + φ k ) 2 + γ k ( k γ k + φ k ) 3 ) ) . (115)
Combining Eq. (115) with the decomposition in Eqs. (107) and (110) , the Euler-remainder bound in Eq. (108) , and the bias bound in Eq. (112) , then applying ∣ δ k + 1 ∣ ≤ ∣ z ∣ / ∣ z − ∣ |\delta_{k+1}|\le|z|/|z_-| ∣ δ k + 1 ∣ ≤ ∣ z ∣/∣ z − ∣ from Lemma E.1 , Eq. (43b), proves Lemma E.4 , Eq. (46a). For its second bound, Eq. (46b), we instead use ∣ δ k + 1 ∣ ≤ Δ − 1 |\delta_{k+1}|\le\Delta^{-1} ∣ δ k + 1 ∣ ≤ Δ − 1 and
γ k 2 + γ k φ k 2 + k γ k 3 + φ k 2 ( k γ k + φ k ) 2 = O ( ( γ k + φ k 2 ) 2 + k ( γ k + φ k 2 ) 3 ) , \gamma_k^2+\gamma_k\varphi_k^2+k\gamma_k^3+\varphi_k^2(\sqrt{k}\gamma_k+\varphi_k)^2=O\left((\gamma_k+\varphi_k^2)^2+k(\gamma_k+\varphi_k^2)^3\right), γ k 2 + γ k φ k 2 + k γ k 3 + φ k 2 ( k γ k + φ k ) 2 = O ( ( γ k + φ k 2 ) 2 + k ( γ k + φ k 2 ) 3 ) ,
γ k ( k γ k + φ k ) 3 ≤ 2 ( k γ k + φ k ) ( k γ k 3 + γ k φ k 2 ) . \gamma_k(\sqrt{k}\gamma_k+\varphi_k)^3\le2(\sqrt{k}\gamma_k+\varphi_k)(k\gamma_k^3+\gamma_k\varphi_k^2). γ k ( k γ k + φ k ) 3 ≤ 2 ( k γ k + φ k ) ( k γ k 3 + γ k φ k 2 ) .
The condition Δ = Ω ( k γ k + φ k ) \Delta=\Omega(\sqrt{k}\gamma_k+\varphi_k) Δ = Ω ( k γ k + φ k ) subsumes the last factor into the big-Oh notation O C x , Δ O_{C_x,\Delta} O C x , Δ . Combining these estimates once again with Eqs. (107) , (108) , (110) , and (112) proves Eq. (46b) under its additional assumptions. The proof is complete.
Proof of Lemma E.11
We use the same notations in Appendix E.4.4 (i.e. proof of Lemma E.4 ). The covariances ψ 2 \psi_2 ψ 2 and ψ 3 \psi_3 ψ 3 are more straightforward to control, while ψ 1 \psi_1 ψ 1 contains two sources of randomness—the fresh sample z k z_k z k and the leave-one-out resolvent R ~ k − 1 \widetilde{R}_{k-1} R k − 1 . We will condition on the first k − 1 k-1 k − 1 samples to separate the two contributions. We control the first source by quadratic-form concentration, and the second source by covariance tensorization (cf. Lemma A.2 ).
Part I: Bounding ψ 1 \psi_1 ψ 1 . Since R ~ k − 1 \widetilde{R}_{k-1} R k − 1 depends only on the first k − 1 k-1 k − 1 samples, conditional isotropy provides E [ U k ∣ T k − 1 ] = Tr ( R ~ k − 1 ) \mathbb{E}[U_k\mid\mathcal{T}_{k-1}]=\operatorname{Tr}(\widetilde{R}_{k-1}) E [ U k ∣ T k − 1 ] = Tr ( R k − 1 ) and E [ z k ∗ R ~ k − 1 2 z k ∣ T k − 1 ] = Tr ( R ~ k − 1 2 ) \mathbb{E}[z_k^{*}\widetilde{R}_{k-1}^{2}z_k\mid\mathcal{T}_{k-1}]=\operatorname{Tr}(\widetilde{R}_{k-1}^{2}) E [ z k ∗ R k − 1 2 z k ∣ T k − 1 ] = Tr ( R k − 1 2 ) . Subtracting these conditional means in the two quadratic forms gives
ψ 1 = E [ ( U k − Tr R ~ k − 1 ) ( z k ∗ R ~ k − 1 2 z k − Tr ( R ~ k − 1 2 ) ) ] + Cov ( Tr ( R ~ k − 1 ) , Tr ( R ~ k − 1 2 ) ) . (116) \psi_1=\mathbb{E}\left[(U_k-\operatorname{Tr}\widetilde{R}_{k-1})\left(z_k^{*}\widetilde{R}_{k-1}^{2}z_k-\operatorname{Tr}(\widetilde{R}_{k-1}^{2})\right)\right]+\operatorname{Cov}\left(\operatorname{Tr}(\widetilde{R}_{k-1}),\operatorname{Tr}(\widetilde{R}_{k-1}^{2})\right). \tag*{(116)} ψ 1 = E [ ( U k − Tr R k − 1 ) ( z k ∗ R k − 1 2 z k − Tr ( R k − 1 2 ) ) ] + Cov ( Tr ( R k − 1 ) , Tr ( R k − 1 2 ) ) . (116)
The first term measures the fresh-sample fluctuation, whereas the second measures the fluctuation of the two leave-one-out traces.
For the first term, applying the conditional Hanson-Wright in Proposition 1 and integrating over tail yields
∥ U k − Tr R ~ k − 1 ∥ L 2 ( z k ∣ T k − 1 ) = O C x ( ∥ R ~ k − 1 ∥ F ) , \left\|U_k-\operatorname{Tr}\widetilde{R}_{k-1}\right\|_{L^2(z_k\mid T_{k-1})}=O_{C_x}\left(\left\|\widetilde{R}_{k-1}\right\|_F\right), U k − Tr R k − 1 L 2 ( z k ∣ T k − 1 ) = O C x ( R k − 1 F ) ,
∥ z k ∗ R ~ k − 1 2 z k − Tr ( R ~ k − 1 2 ) ∥ L 2 ( z k ∣ T k − 1 ) = O C x ( ∥ R ~ k − 1 ∥ ∥ R ~ k − 1 ∥ F ) . (117) \left\|z_k^*\widetilde{R}_{k-1}^2z_k-\operatorname{Tr}\left(\widetilde{R}_{k-1}^2\right)\right\|_{L^2(z_k\mid T_{k-1})}=O_{C_x}\left(\left\|\widetilde{R}_{k-1}\right\|\left\|\widetilde{R}_{k-1}\right\|_F\right). \tag*{(117)} z k ∗ R k − 1 2 z k − Tr ( R k − 1 2 ) L 2 ( z k ∣ T k − 1 ) = O C x ( R k − 1 R k − 1 F ) . (117)
where we use ∥ R ~ k − 1 2 ∥ F ≤ ∥ R ~ k − 1 ∥ ∥ R ~ k − 1 ∥ F \left\|\widetilde{R}_{k-1}^2\right\|_F\leq\left\|\widetilde{R}_{k-1}\right\|\left\|\widetilde{R}_{k-1}\right\|_F R k − 1 2 F ≤ R k − 1 R k − 1 F and ∥ R ~ k − 1 ∥ ≤ ∥ R ~ k − 1 ∥ F \left\|\widetilde{R}_{k-1}\right\|\leq\left\|\widetilde{R}_{k-1}\right\|_F R k − 1 ≤ R k − 1 F . For complex operators, the same estimates follow by repeating the same analyses to their Hermitian real and imaginary parts. The Hölder’s inequality with exponents ( 2 , 4 , 4 ) (2,4,4) ( 2 , 4 , 4 ) therefore implies
∣ E [ ( U k − Tr R ~ k − 1 ) ( z k ∗ R ~ k − 1 2 z k − Tr ( R ~ k − 1 2 ) ) ] ∣ = O C x ( E [ ∥ R ~ k − 1 ∥ ∥ R ~ k − 1 ∥ F 2 ] ) = O C x ( γ k φ k 2 ) . \left|\mathbb{E}\left[(U_k-\operatorname{Tr}\widetilde{R}_{k-1})\left(z_k^*\widetilde{R}_{k-1}^2z_k-\operatorname{Tr}\left(\widetilde{R}_{k-1}^2\right)\right)\right]\right|=O_{C_x}\left(\mathbb{E}\left[\left\|\widetilde{R}_{k-1}\right\|\left\|\widetilde{R}_{k-1}\right\|_F^2\right]\right)=O_{C_x}\left(\gamma_k\varphi_k^2\right). E [ ( U k − Tr R k − 1 ) ( z k ∗ R k − 1 2 z k − Tr ( R k − 1 2 ) ) ] = O C x ( E [ R k − 1 R k − 1 F 2 ] ) = O C x ( γ k φ k 2 ) .
This completes the control of the first term in Eq. (116) . To control the second covariance of traces term, we estimate how each trace changes when one of the first k − 1 k-1 k − 1 observations is replaced. For 1 ≤ i ≤ k − 1 1\leq i\leq k-1 1 ≤ i ≤ k − 1 , let R ~ k − 1 ( i ) \widetilde{R}_{k-1}^{(i)} R k − 1 ( i ) denote R ~ k − 1 \widetilde{R}_{k-1} R k − 1 with z i z_i z i replaced by an independent copy. The two sample covariance operators differ by an operator of rank at most two; applying the resolvent identity therefore shows that R ~ k − 1 − R ~ k − 1 ( i ) \widetilde{R}_{k-1}-\widetilde{R}_{k-1}^{(i)} R k − 1 − R k − 1 ( i ) also has rank at most two. Consequently,
∣ Tr ( R ~ k − 1 ) − Tr ( R ~ k − 1 ( i ) ) ∣ ≤ 2 ∥ R ~ k − 1 − R ~ k − 1 ( i ) ∥ ≤ 2 ( ∥ R ~ k − 1 ∥ + ∥ R ~ k − 1 ( i ) ∥ ) . \left|\operatorname{Tr}(\widetilde{R}_{k-1})-\operatorname{Tr}(\widetilde{R}_{k-1}^{(i)})\right|\leq2\left\|\widetilde{R}_{k-1}-\widetilde{R}_{k-1}^{(i)}\right\|\leq2\left(\left\|\widetilde{R}_{k-1}\right\|+\left\|\widetilde{R}_{k-1}^{(i)}\right\|\right). Tr ( R k − 1 ) − Tr ( R k − 1 ( i ) ) ≤ 2 R k − 1 − R k − 1 ( i ) ≤ 2 ( R k − 1 + R k − 1 ( i ) ) .
For the trace of the squared resolvent, using R ~ k − 1 2 − ( R ~ k − 1 ( i ) ) 2 = R ~ k − 1 ( R ~ k − 1 − R ~ k − 1 ( i ) ) + ( R ~ k − 1 − R ~ k − 1 ( i ) ) R ~ k − 1 ( i ) \widetilde{R}_{k-1}^2-(\widetilde{R}_{k-1}^{(i)})^2=\widetilde{R}_{k-1}(\widetilde{R}_{k-1}-\widetilde{R}_{k-1}^{(i)})+(\widetilde{R}_{k-1}-\widetilde{R}_{k-1}^{(i)})\widetilde{R}_{k-1}^{(i)} R k − 1 2 − ( R k − 1 ( i ) ) 2 = R k − 1 ( R k − 1 − R k − 1 ( i ) ) + ( R k − 1 − R k − 1 ( i ) ) R k − 1 ( i ) shows that this difference has rank at most four and operator norm at most ( ∥ R ~ k − 1 ∥ + ∥ R ~ k − 1 ( i ) ∥ ) ⋅ ∥ R ~ k − 1 − R ~ k − 1 ( i ) ∥ \left(\left\|\widetilde{R}_{k-1}\right\|+\left\|\widetilde{R}_{k-1}^{(i)}\right\|\right)\cdot\left\|\widetilde{R}_{k-1}-\widetilde{R}_{k-1}^{(i)}\right\| ( R k − 1 + R k − 1 ( i ) ) ⋅ R k − 1 − R k − 1 ( i ) . Since R ~ k − 1 ( i ) \widetilde{R}_{k-1}^{(i)} R k − 1 ( i ) has the same distribution as R ~ k − 1 \widetilde{R}_{k-1} R k − 1 , taking moments and applying Hölder’s inequality with exponents ( 4 , 4 ) (4,4) ( 4 , 4 ) to the product yields
∥ Tr ( R ~ k − 1 ) − Tr ( R ~ k − 1 ( i ) ) ∥ L 4 = O C x ( γ k ) and ∥ Tr ( R ~ k − 1 2 ) − Tr ( ( R ~ k − 1 ( i ) ) 2 ) ∥ L 2 = O C x ( γ k 2 ) . (118) \left\|\operatorname{Tr}(\widetilde{R}_{k-1})-\operatorname{Tr}(\widetilde{R}_{k-1}^{(i)})\right\|_{L^4}=O_{C_x}(\gamma_k)\quad\text{and}\quad\left\|\operatorname{Tr}(\widetilde{R}_{k-1}^2)-\operatorname{Tr}\left((\widetilde{R}_{k-1}^{(i)})^2\right)\right\|_{L^2}=O_{C_x}(\gamma_k^2). \tag*{(118)} Tr ( R k − 1 ) − Tr ( R k − 1 ( i ) ) L 4 = O C x ( γ k ) and Tr ( R k − 1 2 ) − Tr ( ( R k − 1 ( i ) ) 2 ) L 2 = O C x ( γ k 2 ) . (118)
Both traces are functions of the independent samples z 1 , … , z k − 1 z_1,\ldots,z_{k-1} z 1 , … , z k − 1 and belong to L 2 L^2 L 2 . We can thus apply Efron-Stein via covariance tensorization in Lemma A.2 with these two traces. Using Eq. (118) , we have
∣ Cov ( Tr ( R ~ k − 1 ) , Tr ( R ~ k − 1 2 ) ) ∣ ≤ ∑ i = 1 k − 1 ∥ Tr ( R ~ k − 1 ) − Tr ( R ~ k − 1 ( i ) ) ∥ L 2 ∥ Tr ( R ~ k − 1 2 ) − Tr ( ( R ~ k − 1 ( i ) ) 2 ) ∥ L 2 = O C x ( k γ k 3 ) . \begin{aligned}
\left|\operatorname{Cov}\left(\operatorname{Tr}(\widetilde{R}_{k-1}),\operatorname{Tr}(\widetilde{R}_{k-1}^2)\right)\right|\leq\sum_{i=1}^{k-1}\left\|\operatorname{Tr}(\widetilde{R}_{k-1})-\operatorname{Tr}(\widetilde{R}_{k-1}^{(i)})\right\|_{L^2}\left\|\operatorname{Tr}(\widetilde{R}_{k-1}^2)-\operatorname{Tr}\left((\widetilde{R}_{k-1}^{(i)})^2\right)\right\|_{L^2} \\
&=O_{C_x}\left(k\gamma_k^3\right).
\end{aligned} Cov ( Tr ( R k − 1 ) , Tr ( R k − 1 2 ) ) ≤ i = 1 ∑ k − 1 Tr ( R k − 1 ) − Tr ( R k − 1 ( i ) ) L 2 Tr ( R k − 1 2 ) − Tr ( ( R k − 1 ( i ) ) 2 ) L 2 = O C x ( k γ k 3 ) .
Combining the two contributions in Eq. (116) completes the proof for
∣ ψ 1 ∣ = O C x ( k γ k 3 + γ k φ k 2 ) . |\psi_1|=O_{C_x}\left(k\gamma_k^3+\gamma_k\varphi_k^2\right). ∣ ψ 1 ∣ = O C x ( k γ k 3 + γ k φ k 2 ) .
Part II: Moment bounds for the remaining covariances. The covariances ψ 2 \psi_2 ψ 2 and ψ 3 \psi_3 ψ 3 contain moments of U k − G ‾ k + 1 U_k-\overline{G}_{k+1} U k − G k + 1 , which we aim to control first. Applying Eq. (109) gives the decomposition
U k − G ‾ k + 1 = ( U k − Tr R ~ k − 1 ) + ( Tr R ~ k − 1 − E Tr R ~ k − 1 ) + E V k . U_k-\overline{G}_{k+1}=(U_k-\operatorname{Tr}\widetilde{R}_{k-1})+(\operatorname{Tr}\widetilde{R}_{k-1}-\mathbb{E}\operatorname{Tr}\widetilde{R}_{k-1})+\mathbb{E}V_k. U k − G k + 1 = ( U k − Tr R k − 1 ) + ( Tr R k − 1 − E Tr R k − 1 ) + E V k .
The first summand is the fresh quadratic-form fluctuation. Applying Hanson-Wright in Proposition 1 conditionally with moment order 4 4 4 and then averaging over T k − 1 T_{k-1} T k − 1 gives ∥ U k − Tr R ~ k − 1 ∥ L 4 = O C x ( φ k ) \left\|U_k-\operatorname{Tr}\widetilde{R}_{k-1}\right\|_{L^4}=O_{C_x}(\varphi_k) U k − Tr R k − 1 L 4 = O C x ( φ k ) .
For the second summand, we use the replacement estimate already established in the previous part (cf. Eq. (118) ). By Efron-Stein in Lemma A.1 with q = 4 q=4 q = 4 , the conditional Jensen’s inequality and Minkowski’s inequality for the square-function bound, we can bound
∥ Tr R ~ k − 1 − E Tr R ~ k − 1 ∥ L 4 = O ( ( ∑ i = 1 k − 1 ∣ Tr ( R ~ k − 1 ) − Tr ( R ~ k − 1 ( i ) ) ∣ L 4 2 ) 1 / 2 ) = O C x ( k γ k ) . \left\|\operatorname{Tr}\widetilde{R}_{k-1}-\mathbb{E}\operatorname{Tr}\widetilde{R}_{k-1}\right\|_{L^4}=O\left(\left(\sum_{i=1}^{k-1}\left|\operatorname{Tr}(\widetilde{R}_{k-1})-\operatorname{Tr}(\widetilde{R}_{k-1}^{(i)})\right|_{L^4}^{2}\right)^{1/2}\right)=O_{C_x}\left(\sqrt{k}\gamma_k\right). Tr R k − 1 − E Tr R k − 1 L 4 = O ( i = 1 ∑ k − 1 Tr ( R k − 1 ) − Tr ( R k − 1 ( i ) ) L 4 2 ) 1/2 = O C x ( k γ k ) .
The bound ∣ E V k ∣ = O C x ( γ k ) \left|\mathbb{E}V_k\right|=O_{C_x}(\gamma_k) ∣ E V k ∣ = O C x ( γ k ) was already proved in Part II of Lemma E.4 , using Eq. (111) . Combining these estimates by the triangle inequality yields
∥ U k − G ‾ k + 1 ∥ L 4 = O C x ( k γ k + φ k ) . (119) \left\|U_k-\overline{G}_{k+1}\right\|_{L^4}=O_{C_x}\left(\sqrt{k}\gamma_k+\varphi_k\right). \tag*{(119)} U k − G k + 1 L 4 = O C x ( k γ k + φ k ) . (119)
The remaining components in ψ 2 \psi_2 ψ 2 and ψ 3 \psi_3 ψ 3 include z k ∗ R ~ k − 1 2 z k z_k^*\widetilde{R}_{k-1}^2z_k z k ∗ R k − 1 2 z k and V k V_k V k . Using the conditional estimate in Eq. (117) with ∣ Tr ( R ~ k − 1 2 ) ∣ ≤ ∥ R ~ k − 1 ∥ F 2 \left|\operatorname{Tr}(\widetilde{R}_{k-1}^2)\right|\leq\left\|\widetilde{R}_{k-1}\right\|_F^2 Tr ( R k − 1 2 ) ≤ R k − 1 F 2 and ∥ R ~ k − 1 ∥ ≤ ∥ R ~ k − 1 ∥ F \left\|\widetilde{R}_{k-1}\right\|\leq\left\|\widetilde{R}_{k-1}\right\|_F R k − 1 ≤ R k − 1 F , then averaging over T k − 1 T_{k-1} T k − 1 , gives the first bound below. The second follows from the rank-one bound (111) , as already used above:
∥ z k ∗ R ~ k − 1 2 z k ∥ L 2 = O C x ( φ k 2 ) and ∥ V k ∥ L 4 = O C x ( γ k ) . (120) \left\|z_k^*\widetilde{R}_{k-1}^2z_k\right\|_{L^2}=O_{C_x}\left(\varphi_k^2\right)
\qquad\text{and}\qquad
\left\|V_k\right\|_{L^4}=O_{C_x}\left(\gamma_k\right). \tag*{(120)} z k ∗ R k − 1 2 z k L 2 = O C x ( φ k 2 ) and ∥ V k ∥ L 4 = O C x ( γ k ) . (120)
Part III: Bounding ψ 2 \psi_2 ψ 2 and $\psi_3. Recall we can write
ψ 2 = E [ ( U k − E U k ) ( z k ∗ R ~ k − 1 2 z k ) ( U k − G ‾ k + 1 ) ] and ψ 3 = E [ ( U k − E U k ) V k ( U k − G ‾ k + 1 ) 2 ] . \psi_2=\mathbb{E}\left[(U_k-\mathbb{E}U_k)(z_k^*\widetilde{R}_{k-1}^2z_k)(U_k-\overline{G}_{k+1})\right]
\qquad\text{and}\qquad
\psi_3=\mathbb{E}\left[(U_k-\mathbb{E}U_k)V_k(U_k-\overline{G}_{k+1})^2\right]. ψ 2 = E [ ( U k − E U k ) ( z k ∗ R k − 1 2 z k ) ( U k − G k + 1 ) ] and ψ 3 = E [ ( U k − E U k ) V k ( U k − G k + 1 ) 2 ] .
Although U k − G ‾ k + 1 U_k-\overline{G}_{k+1} U k − G k + 1 need not be centered, applying Jensen’s inequality gives ∣ E U k − G ‾ k + 1 ∣ ≤ ∥ U k − G ‾ k + 1 ∥ L 4 \left|\mathbb{E}U_k-\overline{G}_{k+1}\right|\leq\left\|U_k-\overline{G}_{k+1}\right\|_{L^4} E U k − G k + 1 ≤ U k − G k + 1 L 4 and thus ∥ U k − E U k ∥ L 4 ≤ 2 ∥ U k − G ‾ k + 1 ∥ L 4 \left\|U_k-\mathbb{E}U_k\right\|_{L^4}\leq2\left\|U_k-\overline{G}_{k+1}\right\|_{L^4} ∥ U k − E U k ∥ L 4 ≤ 2 U k − G k + 1 L 4 . Invoking Hölder’s inequality with exponents ( 4 , 2 , 4 ) (4,2,4) ( 4 , 2 , 4 ) to ψ 2 \psi_2 ψ 2 and using Eqs. (119) and (120) implies
∣ ψ 2 ∣ ≤ 2 ∥ z k ∗ R ~ k − 1 2 z k ∥ L 2 ⋅ ∥ U k − G ‾ k + 1 ∥ L 4 2 = O C x ( φ k 2 ( k γ k + φ k ) 2 ) . |\psi_2|\leq2\left\|z_k^*\widetilde{R}_{k-1}^2z_k\right\|_{L^2}\cdot\left\|U_k-\overline{G}_{k+1}\right\|_{L^4}^2=O_{C_x}\left(\varphi_k^2\left(\sqrt{k}\gamma_k+\varphi_k\right)^2\right). ∣ ψ 2 ∣ ≤ 2 z k ∗ R k − 1 2 z k L 2 ⋅ U k − G k + 1 L 4 2 = O C x ( φ k 2 ( k γ k + φ k ) 2 ) .
For ψ 3 \psi_3 ψ 3 , applying Hölder with exponents ( 4 , 4 , 4 , 4 ) (4,4,4,4) ( 4 , 4 , 4 , 4 ) yields
∣ ψ 3 ∣ ≤ 2 ∥ V k ∥ L 4 ⋅ ∥ U k − G ‾ k + 1 ∥ L 4 3 = O C x ( γ k ( k γ k + φ k ) 3 ) . |\psi_3|\leq2\left\|V_k\right\|_{L^4}\cdot\left\|U_k-\overline{G}_{k+1}\right\|_{L^4}^3=O_{C_x}\left(\gamma_k\left(\sqrt{k}\gamma_k+\varphi_k\right)^3\right). ∣ ψ 3 ∣ ≤ 2 ∥ V k ∥ L 4 ⋅ U k − G k + 1 L 4 3 = O C x ( γ k ( k γ k + φ k ) 3 ) .
This proves all three bounds and we complete the proof.
Auxiliary proofs for Theorem 4
Proof of Lemma E.5
The proof is a pure algebraic verification. We compute directly along the backward path, first for the resolvents
R k ( z , w k ) − R k − 1 ( z , w k − 1 ) = ( R k ( z , w k ) − R k − 1 ( z , w k ) ) + ( R k − 1 ( z , w k ) − R k − 1 ( z , w k − 1 ) ) = − R k − 1 ( z , w k ) x k x k ∗ R k − 1 ( z , w k ) 1 + x k ∗ R k − 1 ( z , w k ) x k + δ k R k − 1 ( z , w k ) Σ R k − 1 ( z , w k − 1 ) , \begin{aligned}
R_k(z,w_k)-R_{k-1}(z,w_{k-1})=\left(R_k(z,w_k)-R_{k-1}(z,w_k)\right)+\left(R_{k-1}(z,w_k)-R_{k-1}(z,w_{k-1})\right) \\
&=-\frac{R_{k-1}(z,w_k)x_kx_k^*R_{k-1}(z,w_k)}{1+x_k^*R_{k-1}(z,w_k)x_k}+\delta_kR_{k-1}(z,w_k)\Sigma R_{k-1}(z,w_{k-1}),
\end{aligned} R k ( z , w k ) − R k − 1 ( z , w k − 1 ) = ( R k ( z , w k ) − R k − 1 ( z , w k ) ) + ( R k − 1 ( z , w k ) − R k − 1 ( z , w k − 1 ) ) = − 1 + x k ∗ R k − 1 ( z , w k ) x k R k − 1 ( z , w k ) x k x k ∗ R k − 1 ( z , w k ) + δ k R k − 1 ( z , w k ) Σ R k − 1 ( z , w k − 1 ) ,
where we use the Sherman-Morrison formula and the path update δ k = w k − w k − 1 \delta_k=w_k-w_{k-1} δ k = w k − w k − 1 . Next, we can recall the definitions of H k Q H_k^Q H k Q and G k Q G_k^Q G k Q in Eq. (23) . Using ( T j − w Σ ) R j ( z , w ) = I + z R j ( z , w ) (T_j-w\Sigma)R_j(z,w)=I+zR_j(z,w) ( T j − w Σ ) R j ( z , w ) = I + z R j ( z , w ) , we can make the one-step update explicit using the calculation above. For 1 ≤ k ≤ n 1 \le k \le n 1 ≤ k ≤ n , the resolvent update implies
H k Q ( z , w k ) − H k − 1 Q ( z , w k − 1 ) = z Tr ( Q ( R k ( z , w k ) − R k − 1 ( z , w k − 1 ) ) ) = − V k H , Q ( w k ) + z δ k Tr ( Q R k − 1 ( z , w k ) Σ R k − 1 ( z , w k − 1 ) ) = ξ k H , Q + β k H , Q . \begin{aligned}
H_k^{Q}(z,w_k)-H_{k-1}^{Q}(z,w_{k-1})=z\operatorname{Tr}\left(Q\left(R_k(z,w_k)-R_{k-1}(z,w_{k-1})\right)\right) \\
&=-V_k^{H,Q}(w_k)+z\delta_k\operatorname{Tr}\left(QR_{k-1}(z,w_k)\Sigma R_{k-1}(z,w_{k-1})\right)=\xi_k^{H,Q}+\beta_k^{H,Q}.
\end{aligned} H k Q ( z , w k ) − H k − 1 Q ( z , w k − 1 ) = z Tr ( Q ( R k ( z , w k ) − R k − 1 ( z , w k − 1 ) ) ) = − V k H , Q ( w k ) + z δ k Tr ( Q R k − 1 ( z , w k ) Σ R k − 1 ( z , w k − 1 ) ) = ξ k H , Q + β k H , Q .
For G k Q G_k^Q G k Q , the parameter pair is ( w k + 1 , w k ) (w_{k+1},w_k) ( w k + 1 , w k ) . Applying the same resolvent calculation with this pair and taking the trace weighted by Σ 1 / 2 Q Σ 1 / 2 \Sigma^{1/2}Q\Sigma^{1/2} Σ 1/2 Q Σ 1/2 , we have for 1 ≤ k < n 1 \le k < n 1 ≤ k < n ,
G k + 1 Q ( z , w k + 1 ) − G k Q ( z , w k ) = Tr ( Σ 1 / 2 Q Σ 1 / 2 ( R k ( z , w k + 1 ) − R k − 1 ( z , w k ) ) ) = − V k G , Q ( w k + 1 ) + δ k + 1 Tr ( Q Σ 1 / 2 R k − 1 ( z , w k + 1 ) Σ R k − 1 ( z , w k ) Σ 1 / 2 ) = ξ k G , Q + β k G , Q . \begin{aligned}
G_{k+1}^{Q}(z,w_{k+1})-G_k^{Q}(z,w_k)=\operatorname{Tr}\left(\Sigma^{1/2}Q\Sigma^{1/2}\left(R_k(z,w_{k+1})-R_{k-1}(z,w_k)\right)\right) \\
&=-V_k^{G,Q}(w_{k+1})+\delta_{k+1}\operatorname{Tr}\left(Q\Sigma^{1/2}R_{k-1}(z,w_{k+1})\Sigma R_{k-1}(z,w_k)\Sigma^{1/2}\right)=\xi_k^{G,Q}+\beta_k^{G,Q}.
\end{aligned} G k + 1 Q ( z , w k + 1 ) − G k Q ( z , w k ) = Tr ( Σ 1/2 Q Σ 1/2 ( R k ( z , w k + 1 ) − R k − 1 ( z , w k ) ) ) = − V k G , Q ( w k + 1 ) + δ k + 1 Tr ( Q Σ 1/2 R k − 1 ( z , w k + 1 ) Σ R k − 1 ( z , w k ) Σ 1/2 ) = ξ k G , Q + β k G , Q .
The deterministic path makes the parameter-change terms F k − 1 \mathcal{F}_{k-1} F k − 1 -measurable. Thus both drift term sequences are predictable, and conditional centering gives E k − 1 ξ k H , Q = E k − 1 ξ k G , Q = 0 \mathbb{E}_{k-1}\xi_k^{H,Q}=\mathbb{E}_{k-1}\xi_k^{G,Q}=0 E k − 1 ξ k H , Q = E k − 1 ξ k G , Q = 0 . We therefore complete the proof.
Proof of Lemma E.6
We first bound the martingale partial sums in (i). For (ii), we separate each predictable increment into its full expectation and its centered fluctuation. Indeed, taking full expectation recovers the Euler decomposition in Lemma E.4 , which makes the role of our backward path explicit. It suffices to work under either ∥ Q ∥ ≤ 1 \lVert Q\rVert\le1 ∥ Q ∥ ≤ 1 or ∥ Q ∥ F ≤ n \lVert Q\rVert_F \le\sqrt{n} ∥ Q ∥ F ≤ n , as rescaling restores the factor ∥ Q ∥ ∧ ∥ Q ∥ F / n \lVert Q\rVert\wedge\lVert Q\rVert_F/\sqrt{n} ∥ Q ∥ ∧ ∥ Q ∥ F / n .
Part I: Bounding the martingale partial sums. To control the sums of martingale differences ξ k H , Q \xi_k^{H,Q} ξ k H , Q and ξ k G , Q \xi_k^{G,Q} ξ k G , Q , it suffices to bound their L p L^p L p norms, as the martingale inequality below will show. By Lemma E.5 , each ξ \xi ξ -term is the negative of a conditionally centered V V V -term. The next lemma supplies a general bound for V V V -term fluctuations, which will also be used later in the proof for the drift β \beta β -terms analysis. We defer its proof to Appendix § 13 .§ 5.3 .
Lemma E.12. Under the same setup as Theorem 4, let 0 ≤ j ≤ n 0 \le j \le n 0 ≤ j ≤ n and ( j − 3 ) ∨ 0 ≤ s ≤ j (j-3)\vee0 \le s \le j ( j − 3 ) ∨ 0 ≤ s ≤ j . Form the ratios V s + 1 H , Q ( w j ) V_{s+1}^{H,Q}(w_j) V s + 1 H , Q ( w j ) and V s + 1 G , Q ( w j ) V_{s+1}^{G,Q}(w_j) V s + 1 G , Q ( w j ) of Lemma E.5 , Eq. (79) , using R s ( z , w j ) R_s(z,w_j) R s ( z , w j ) and a fresh observation x s + 1 x_{s+1} x s + 1 independent of F s \mathcal{F}_s F s . Then, for 2 ≤ p ≤ κ log n 2 \le p \le\kappa\log n 2 ≤ p ≤ κ log n ,
max { ∥ V s + 1 H , Q ( w j ) − E s V s + 1 H , Q ( w j ) ∥ L p n , ∥ V s + 1 G , Q ( w j ) − E s V s + 1 G , Q ( w j ) ∥ L p } = O C x , ν , κ ( p Λ n ( z ) 11 n 3 / 2 ⋅ ( ∥ Q ∥ ∧ ∥ Q ∥ F n ) ) . \max\left\{\frac{\left\|V_{s+1}^{H,Q}(w_j)-\mathbb{E}_sV_{s+1}^{H,Q}(w_j)\right\|_{L^p}}{n},\left\|V_{s+1}^{G,Q}(w_j)-\mathbb{E}_sV_{s+1}^{G,Q}(w_j)\right\|_{L^p}\right\}
=O_{C_x,\nu,\kappa}\left(\frac{p\Lambda_n(z)^{11}}{n^{3/2}}\cdot\left(\lVert Q\rVert\wedge\frac{\lVert Q\rVert_F}{\sqrt{n}}\right)\right). max ⎩ ⎨ ⎧ n V s + 1 H , Q ( w j ) − E s V s + 1 H , Q ( w j ) L p , V s + 1 G , Q ( w j ) − E s V s + 1 G , Q ( w j ) L p ⎭ ⎬ ⎫ = O C x , ν , κ ( n 3/2 p Λ n ( z ) 11 ⋅ ( ∥ Q ∥ ∧ n ∥ Q ∥ F ) ) .
We immediately recover the ξ \xi ξ -terms in the martingale partial sums by taking ( s , j ) = ( k − 1 , k ) (s,j)=(k-1,k) ( s , j ) = ( k − 1 , k ) for H H H and ( s , j ) = ( k − 1 , k + 1 ) (s,j)=(k-1,k+1) ( s , j ) = ( k − 1 , k + 1 ) for G G G in the above Lemma E.12 . For H H H we use the separate power 9 9 9 in Part II of the proof of that lemma. Therefore, we obtain
1 n ∥ max 0 ≤ j ≤ n ∑ k = 1 j ξ k H , Q ∥ L p ≤ O ( p ) n ( ∑ k = 1 n ∥ ξ k H , Q ∥ L p 2 ) 1 / 2 = O ( p 2 Λ n ( z ) 9 n ) , \frac{1}{n}\left\|\max_{0\le j\le n}\sum_{k=1}^{j}\xi_k^{H,Q}\right\|_{L^p}
\le\frac{O(p)}{n}\left(\sum_{k=1}^{n}\lVert\xi_k^{H,Q}\rVert_{L^p}^{2}\right)^{1/2}
=O\left(\frac{p^2\Lambda_n(z)^9}{n}\right), n 1 0 ≤ j ≤ n max k = 1 ∑ j ξ k H , Q L p ≤ n O ( p ) ( k = 1 ∑ n ∥ ξ k H , Q ∥ L p 2 ) 1/2 = O ( n p 2 Λ n ( z ) 9 ) ,
∥ max 0 ≤ j < n ∑ k = 1 j ξ k G , Q ∥ L p ≤ O ( p ) ( ∑ k = 1 n − 1 ∥ ξ k G , Q ∥ L p 2 ) 1 / 2 = O ( p 2 Λ n ( z ) 11 n ) . (121) \left\|\max_{0\le j<n}\sum_{k=1}^{j}\xi_k^{G,Q}\right\|_{L^p}
\le O(p)\left(\sum_{k=1}^{n-1}\lVert\xi_k^{G,Q}\rVert_{L^p}^{2}\right)^{1/2}
=O\left(\frac{p^2\Lambda_n(z)^{11}}{n}\right). \tag*{(121)} 0 ≤ j < n max k = 1 ∑ j ξ k G , Q L p ≤ O ( p ) ( k = 1 ∑ n − 1 ∥ ξ k G , Q ∥ L p 2 ) 1/2 = O ( n p 2 Λ n ( z ) 11 ) . (121)
In both inequalities above, we apply Doob’s maximal inequality [39 Theorem 4.4.4] and Burkholder’s inequality [48 Theorem 2.10] to the real and imaginary parts, and then apply the Minkowski inequality in L p / 2 L^{p/2} L p /2 to reduce to sum of L p L^p L p norms of ξ \xi ξ -terms. We suppress dependence on C x , ν , κ C_x,\nu,\kappa C x , ν , κ in the big-Oh notation to simplify presentation, and we shall keep this convention throughout the remainder of this proof. This completes the proof of Lemma E.6(i).
Part II: Bounding the accumulated drift. We next derive an upper bound for the accumulated β \beta β -terms. This entails establishing bounds on both the full expectation and the centered fluctuation of each predictable increment, in addition to controlling the initial correction term. We start with the exact decomposition
β k H , Q = E β k H , Q + ( β k H , Q − E β k H , Q ) and β k G , Q = E β k G , Q + ( β k G , Q − E β k G , Q ) . \beta_k^{H,Q}=\mathbb{E}\beta_k^{H,Q}+(\beta_k^{H,Q}-\mathbb{E}\beta_k^{H,Q}) \qquad\text{and}\qquad\beta_k^{G,Q}=\mathbb{E}\beta_k^{G,Q}+(\beta_k^{G,Q}-\mathbb{E}\beta_k^{G,Q}). β k H , Q = E β k H , Q + ( β k H , Q − E β k H , Q ) and β k G , Q = E β k G , Q + ( β k G , Q − E β k G , Q ) .
We point out how the drift terms connect to our proof for the path stability of { w k } k = 0 n \{w_k\}_{k=0}^n { w k } k = 0 n . Taking full expectation in Lemma E.5, Eq. (62b), with Q = I Q=I Q = I recovers Eq. (107) in the proof of Lemma E.4:
E β k G , I = G ‾ k + 1 ( z , w k + 1 ) − G ‾ k ( z , w k ) = χ 1 − χ 2 , \mathbb{E}\beta_k^{G,I}=\overline{G}_{k+1}(z,w_{k+1})-\overline{G}_k(z,w_k)=\chi_1-\chi_2, E β k G , I = G k + 1 ( z , w k + 1 ) − G k ( z , w k ) = χ 1 − χ 2 ,
since E ξ k G , I = 0 \mathbb{E}\xi_k^{G,I}=0 E ξ k G , I = 0 . This gives rise to our proof strategy by repeating the Euler calculation with the test Q Q Q under conditional expectation E k − 1 \mathbb{E}_{k-1} E k − 1 , so that the conditional trace of the weighted resolvent, Tr R ~ k − 1 = E k − 1 [ z k ∗ R ~ k − 1 z k ] \operatorname{Tr}\widetilde{R}_{k-1}=\mathbb{E}_{k-1}[z_k^*\widetilde{R}_{k-1}z_k] Tr R k − 1 = E k − 1 [ z k ∗ R k − 1 z k ] , remains explicit. Below, we shall suppress ( z , w k ) (z,w_k) ( z , w k ) in the β k H , Q \beta_k^{H,Q} β k H , Q terms and ( z , w k + 1 ) (z,w_{k+1}) ( z , w k + 1 ) in the β k G , Q \beta_k^{G,Q} β k G , Q terms, and write U k = z k ∗ R ~ k − 1 z k U_k=z_k^*\widetilde{R}_{k-1}z_k U k = z k ∗ R k − 1 z k at the corresponding evaluation point. Applying the Euler identity from Part I of the proof of Lemma E.4, Eq. (107) , to Eq. (82) , we can write
β k H , Q = χ 1 H , Q − χ 2 H , Q and β k G , Q = χ 1 G , Q − χ 2 G , Q , (122) \beta_k^{H,Q}=\chi_1^{H,Q}-\chi_2^{H,Q}\qquad\text{and}\qquad\beta_k^{G,Q}=\chi_1^{G,Q}-\chi_2^{G,Q}, \tag*{(122)} β k H , Q = χ 1 H , Q − χ 2 H , Q and β k G , Q = χ 1 G , Q − χ 2 G , Q , (122)
where conditional isotropy and δ j ( 1 + G ‾ j ) = 1 \delta_j(1+\overline{G}_j)=1 δ j ( 1 + G j ) = 1 identify
χ 1 H , Q = δ k E k − 1 [ ( U k − G ‾ k ) V k H , Q ] , χ 1 G , Q = δ k + 1 E k − 1 [ ( U k − G ‾ k + 1 ) V k G , Q ] , \chi_1^{H,Q}=\delta_k\mathbb{E}_{k-1}[(U_k-\overline{G}_k)V_k^{H,Q}],\qquad\chi_1^{G,Q}=\delta_{k+1}\mathbb{E}_{k-1}[(U_k-\overline{G}_{k+1})V_k^{G,Q}], χ 1 H , Q = δ k E k − 1 [( U k − G k ) V k H , Q ] , χ 1 G , Q = δ k + 1 E k − 1 [( U k − G k + 1 ) V k G , Q ] ,
χ 2 H , Q = z δ k 2 Tr [ Q R k − 1 Σ R k − 1 ( z , w k − 1 ) Σ R k − 1 ] , χ 2 G , Q = δ k + 1 2 Tr [ Q R ~ k − 1 R ~ k − 1 ( z , w k ) R ~ k − 1 ] . \chi_2^{H,Q}=z\delta_k^2\operatorname{Tr}[Q R_{k-1}\Sigma R_{k-1}(z,w_{k-1})\Sigma R_{k-1}],\qquad\chi_2^{G,Q}=\delta_{k+1}^2\operatorname{Tr}[Q\widetilde{R}_{k-1}\widetilde{R}_{k-1}(z,w_k)\widetilde{R}_{k-1}]. χ 2 H , Q = z δ k 2 Tr [ Q R k − 1 Σ R k − 1 ( z , w k − 1 ) Σ R k − 1 ] , χ 2 G , Q = δ k + 1 2 Tr [ Q R k − 1 R k − 1 ( z , w k ) R k − 1 ] .
We use R k − 1 ( z , w k − 1 ) = R k − 1 − δ k R k − 1 ( z , w k − 1 ) Σ R k − 1 R_{k-1}(z,w_{k-1})=R_{k-1}-\delta_kR_{k-1}(z,w_{k-1})\Sigma R_{k-1} R k − 1 ( z , w k − 1 ) = R k − 1 − δ k R k − 1 ( z , w k − 1 ) Σ R k − 1 inside the trace for the H H H -terms. The G G G -terms calculation is the weighted version of the same identity. We first bound the remainders χ 2 H , Q \chi_2^{H,Q} χ 2 H , Q and χ 2 G , Q \chi_2^{G,Q} χ 2 G , Q directly. For χ 2 G , Q \chi_2^{G,Q} χ 2 G , Q , trace Cauchy-Schwarz and the operator-Frobenius product inequality give
∣ Tr ( Q R ~ k − 1 R ~ k − 1 ( z , w k ) R ~ k − 1 ) ∣ ≤ ∥ R ~ k − 1 ( z , w k ) ∥ ( ∥ R ~ k − 1 ∥ F 2 ∨ n ∥ R ~ k − 1 ∥ ∥ R ~ k − 1 ∥ F ) . \left|\operatorname{Tr}\left(Q\widetilde{R}_{k-1}\widetilde{R}_{k-1}(z,w_k)\widetilde{R}_{k-1}\right)\right|\leq\left\|\widetilde{R}_{k-1}(z,w_k)\right\|\left(\left\|\widetilde{R}_{k-1}\right\|_F^2\vee\sqrt{n}\left\|\widetilde{R}_{k-1}\right\|\left\|\widetilde{R}_{k-1}\right\|_F\right). Tr ( Q R k − 1 R k − 1 ( z , w k ) R k − 1 ) ≤ R k − 1 ( z , w k ) ( R k − 1 F 2 ∨ n R k − 1 R k − 1 F ) .
The two alternatives correspond to ∥ Q ∥ ≤ 1 \|Q\|\leq1 ∥ Q ∥ ≤ 1 and ∥ Q ∥ F ≤ n \|Q\|_F\leq\sqrt{n} ∥ Q ∥ F ≤ n , respectively. For χ 2 H , Q \chi_2^{H,Q} χ 2 H , Q , we can factor the trace through R k − 1 Σ 1 / 2 R_{k-1}\Sigma^{1/2} R k − 1 Σ 1/2 , R ~ k − 1 ( z , w k − 1 ) \widetilde{R}_{k-1}(z,w_{k-1}) R k − 1 ( z , w k − 1 ) , and Σ 1 / 2 R k − 1 \Sigma^{1/2}R_{k-1} Σ 1/2 R k − 1 . Applying the same product inequalities and Eq. (133) from Part I-ii of the proof of Lemma E.12 gives
∣ χ 2 H , Q ∣ n ≤ 2 ∣ z ∣ ∣ δ k ∣ 2 n ∣ z − ∣ ∥ R ~ k − 1 ( z , w k − 1 ) ∥ ( Tr ∣ R ~ k − 1 ∣ ∨ n ∥ R ~ k − 1 ∥ Tr ∣ R ~ k − 1 ∣ ) . \frac{|\chi_2^{H,Q}|}{n}\leq\frac{2|z||\delta_k|^2}{n|z_-|}\left\|\widetilde{R}_{k-1}(z,w_{k-1})\right\|\left(\operatorname{Tr}|\widetilde{R}_{k-1}|\vee\sqrt{n}\|\widetilde{R}_{k-1}\|\operatorname{Tr}|\widetilde{R}_{k-1}|\right). n ∣ χ 2 H , Q ∣ ≤ n ∣ z − ∣ 2∣ z ∣∣ δ k ∣ 2 R k − 1 ( z , w k − 1 ) ( Tr ∣ R k − 1 ∣ ∨ n ∥ R k − 1 ∥ Tr ∣ R k − 1 ∣ ) .
Using ∣ δ j ∣ ≤ Λ n ( z ) |\delta_j|\leq\Lambda_n(z) ∣ δ j ∣ ≤ Λ n ( z ) and ∣ z ∣ / ∣ z − ∣ ≤ Λ n ( z ) |z|/|z_-|\leq\Lambda_n(z) ∣ z ∣/∣ z − ∣ ≤ Λ n ( z ) , then applying Hölder and the resolvent moments in Eq. (135) from Part II of the proof of Lemma E.12, we obtain
∥ χ 2 G , Q ∥ L p = O ( Λ n ( z ) 2 Λ n ( z ) 3 Λ n ( z ) 6 n ) = O ( Λ n ( z ) 11 n 2 ) , \left\|\chi_2^{G,Q}\right\|_{L^p}=O\left(\Lambda_n(z)^2\frac{\Lambda_n(z)^3\Lambda_n(z)^6}{n}\right)=O\left(\frac{\Lambda_n(z)^{11}}{n^2}\right), χ 2 G , Q L p = O ( Λ n ( z ) 2 n Λ n ( z ) 3 Λ n ( z ) 6 ) = O ( n 2 Λ n ( z ) 11 ) ,
∥ χ 2 H , Q ∥ L p n = O ( Λ n ( z ) 3 n Λ n ( z ) 3 n Λ n ( z ) 3 ) = O ( Λ n ( z ) 9 n 2 ) (123) \frac{\lVert\chi_{2}^{H,Q}\rVert_{L^{p}}}{n}=O\left(\frac{\Lambda_{n}(z)^{3}}{n}\frac{\Lambda_{n}(z)^{3}}{n}\Lambda_{n}(z)^{3}\right)=O\left(\frac{\Lambda_{n}(z)^{9}}{n^{2}}\right) \tag*{(123)} n ∥ χ 2 H , Q ∥ L p = O ( n Λ n ( z ) 3 n Λ n ( z ) 3 Λ n ( z ) 3 ) = O ( n 2 Λ n ( z ) 9 ) (123)
using the Hölder inequality with exponents ( 3 , 3 , 3 ) (3,3,3) ( 3 , 3 , 3 ) in the first line, and in the second line ( 2 , 2 ) (2,2) ( 2 , 2 ) first, followed by ( 2 , 2 ) (2,2) ( 2 , 2 ) for the square-root product. The weighted resolvent moments used above have order at most 3 p ≤ 4 κ log n 3p\le4\kappa\log n 3 p ≤ 4 κ log n , and thus inequalities in Eq. (135) apply.
It remains to bound χ 1 H , Q \chi_{1}^{H,Q} χ 1 H , Q and χ 1 G , Q \chi_{1}^{G,Q} χ 1 G , Q . Both expansions contain the conditional trace fluctuation Tr R ~ k − 1 − E Tr R ~ k − 1 \operatorname{Tr}\widetilde{R}_{k-1}-\mathbb{E}\operatorname{Tr}\widetilde{R}_{k-1} Tr R k − 1 − E Tr R k − 1 . Using E k − 1 U k = Tr R ~ k − 1 \mathbb{E}_{k-1}U_{k}=\operatorname{Tr}\widetilde{R}_{k-1} E k − 1 U k = Tr R k − 1 and the mean identity in Part II of the proof of Lemma E.4 , Eq. (109) , we have for χ 1 G , Q \chi_{1}^{G,Q} χ 1 G , Q ,
χ 1 G , Q = δ k + 1 E k − 1 [ ( U k − E U k ) V k G , Q ] + δ k + 1 ⋅ E V k G , I ⋅ E k − 1 V k G , Q = δ k + 1 ( Tr R ~ k − 1 − E Tr R ~ k − 1 ) E k − 1 V k G , Q + δ k + 1 Cov ( U k , V k G , Q ∣ F k − 1 ) + δ k + 1 ⋅ E V k G , I ⋅ E k − 1 V k G , Q (124) \begin{aligned}
\chi_{1}^{G,Q}=\delta_{k+1}\mathbb{E}_{k-1}\left[(U_{k}-\mathbb{E}U_{k})V_{k}^{G,Q}\right]+\delta_{k+1}\cdot\mathbb{E}V_{k}^{G,I}\cdot\mathbb{E}_{k-1}V_{k}^{G,Q} \\
&=\delta_{k+1}\left(\operatorname{Tr}\widetilde{R}_{k-1}-\mathbb{E}\operatorname{Tr}\widetilde{R}_{k-1}\right)\mathbb{E}_{k-1}V_{k}^{G,Q}+\delta_{k+1}\operatorname{Cov}(U_{k},V_{k}^{G,Q}\mid\mathcal{F}_{k-1})+\delta_{k+1}\cdot\mathbb{E}V_{k}^{G,I}\cdot\mathbb{E}_{k-1}V_{k}^{G,Q} \tag*{(124)}
\end{aligned} χ 1 G , Q = δ k + 1 E k − 1 [ ( U k − E U k ) V k G , Q ] + δ k + 1 ⋅ E V k G , I ⋅ E k − 1 V k G , Q = δ k + 1 ( Tr R k − 1 − E Tr R k − 1 ) E k − 1 V k G , Q + δ k + 1 Cov ( U k , V k G , Q ∣ F k − 1 ) + δ k + 1 ⋅ E V k G , I ⋅ E k − 1 V k G , Q (124)
For the second line, we split U k − E U k = ( U k − E k − 1 U k ) + ( Tr R ~ k − 1 − E Tr R ~ k − 1 ) U_{k}-\mathbb{E}U_{k}=(U_{k}-\mathbb{E}_{k-1}U_{k})+(\operatorname{Tr}\widetilde{R}_{k-1}-\mathbb{E}\operatorname{Tr}\widetilde{R}_{k-1}) U k − E U k = ( U k − E k − 1 U k ) + ( Tr R k − 1 − E Tr R k − 1 ) and use E k − 1 ( U k − E k − 1 U k ) = 0 \mathbb{E}_{k-1}(U_{k}-\mathbb{E}_{k-1}U_{k})=0 E k − 1 ( U k − E k − 1 U k ) = 0 . For χ 1 H , Q \chi_{1}^{H,Q} χ 1 H , Q , we have similarly
χ 1 H , Q = δ k E k − 1 [ ( U k − E U k ) V k H , Q ] = δ k ( Tr R ~ k − 1 − E Tr R ~ k − 1 ) E k − 1 V k H , Q + δ k Cov ( U k , V k H , Q ∣ F k − 1 ) (125) \begin{aligned}
\chi_{1}^{H,Q}=\delta_{k}\mathbb{E}_{k-1}\left[(U_{k}-\mathbb{E}U_{k})V_{k}^{H,Q}\right] \\
&=\delta_{k}\left(\operatorname{Tr}\widetilde{R}_{k-1}-\mathbb{E}\operatorname{Tr}\widetilde{R}_{k-1}\right)\mathbb{E}_{k-1}V_{k}^{H,Q}+\delta_{k}\operatorname{Cov}(U_{k},V_{k}^{H,Q}\mid\mathcal{F}_{k-1}) \tag*{(125)}
\end{aligned} χ 1 H , Q = δ k E k − 1 [ ( U k − E U k ) V k H , Q ] = δ k ( Tr R k − 1 − E Tr R k − 1 ) E k − 1 V k H , Q + δ k Cov ( U k , V k H , Q ∣ F k − 1 ) (125)
We note that only the χ 1 G , Q \chi_{1}^{G,Q} χ 1 G , Q expansion has the additional mean correction. The following lemma bounds χ 1 \chi_{1} χ 1 -terms and the initial correction, making use of the decomposition in the above displays. We postpone its proof to Appendix § 13 .5.4.
Lemma E.13. Under the same setup as Theorem 4, for 2 ≤ p ≤ κ log n 2\le p\le\kappa\log n 2 ≤ p ≤ κ log n ,
∥ χ 1 H , Q ∥ L p n ∨ ∥ χ 1 G , Q ∥ L p = O C x , ν , κ ( p 2 Λ n ( z ) 19 n 2 ⋅ ( ∥ Q ∥ ∧ ∥ Q ∥ F n ) ) (126) \frac{\lVert\chi_{1}^{H,Q}\rVert_{L^{p}}}{n}\vee\lVert\chi_{1}^{G,Q}\rVert_{L^{p}}=O_{C_{x},\nu,\kappa}\left(\frac{p^{2}\Lambda_{n}(z)^{19}}{n^{2}}\cdot\left(\lVert Q\rVert\wedge\frac{\lVert Q\rVert_{F}}{\sqrt{n}}\right)\right) \tag*{(126)} n ∥ χ 1 H , Q ∥ L p ∨ ∥ χ 1 G , Q ∥ L p = O C x , ν , κ ( n 2 p 2 Λ n ( z ) 19 ⋅ ( ∥ Q ∥ ∧ n ∥ Q ∥ F ) ) (126) ∣ β 0 H , Q ∣ n ∨ ∣ β 0 G , Q ∣ = O C x , ν , κ ( Λ n ( z ) 23 n ⋅ ( ∥ Q ∥ ∧ ∥ Q ∥ F n ) ) (127) \frac{\lvert\beta_{0}^{H,Q}\rvert}{n}\vee\lvert\beta_{0}^{G,Q}\rvert=O_{C_{x},\nu,\kappa}\left(\frac{\Lambda_{n}(z)^{23}}{n}\cdot\left(\lVert Q\rVert\wedge\frac{\lVert Q\rVert_{F}}{\sqrt{n}}\right)\right) \tag*{(127)} n ∣ β 0 H , Q ∣ ∨ ∣ β 0 G , Q ∣ = O C x , ν , κ ( n Λ n ( z ) 23 ⋅ ( ∥ Q ∥ ∧ n ∥ Q ∥ F ) ) (127)
We can now combine the two parts of each drift. Applying Lemma E.13 above to the decomposition in Eq. (122) , we obtain
∥ β k H , Q ∥ L p n ∨ ∥ β k G , Q ∥ L p ≤ ( ∥ χ 1 H , Q ∥ L p n ∨ ∥ χ 1 G , Q ∥ L p ) + ( ∥ χ 2 H , Q ∥ L p n ∨ ∥ χ 2 G , Q ∥ L p ) = O ( p 2 Λ n ( z ) 19 n 2 ) . \frac{\lVert\beta_{k}^{H,Q}\rVert_{L^{p}}}{n}\vee\lVert\beta_{k}^{G,Q}\rVert_{L^{p}}\le\left(\frac{\lVert\chi_{1}^{H,Q}\rVert_{L^{p}}}{n}\vee\lVert\chi_{1}^{G,Q}\rVert_{L^{p}}\right)+\left(\frac{\lVert\chi_{2}^{H,Q}\rVert_{L^{p}}}{n}\vee\lVert\chi_{2}^{G,Q}\rVert_{L^{p}}\right)=O\left(\frac{p^{2}\Lambda_{n}(z)^{19}}{n^{2}}\right). n ∥ β k H , Q ∥ L p ∨ ∥ β k G , Q ∥ L p ≤ ( n ∥ χ 1 H , Q ∥ L p ∨ ∥ χ 1 G , Q ∥ L p ) + ( n ∥ χ 2 H , Q ∥ L p ∨ ∥ χ 2 G , Q ∥ L p ) = O ( n 2 p 2 Λ n ( z ) 19 ) .
For the scalar estimate in Theorem 1 we preserve a separate H H H bound here. For χ 2 H , Q \chi_{2}^{H,Q} χ 2 H , Q , the second line of Eq. (123) gives the exponent 9. For χ 1 H , Q \chi_{1}^{H,Q} χ 1 H , Q , the last display in Part II of the proof of Lemma E.13 bounds the two terms of Eq. (125) separately. The conditional trace term has exponent 17 and the conditional covariance has power 13. Summing from k = 1 k=1 k = 1 and applying Minkowski’s inequality to the n n n one-step drifts yields a sharpened bound
1 n ∥ ∑ k = 1 n β k H , Q ∥ L p = O ( p 2 Λ n ( z ) 17 n ) . (128) \frac{1}{n}\left\lVert\sum_{k=1}^{n}\beta_{k}^{H,Q}\right\rVert_{L^{p}}=O\left(\frac{p^{2}\Lambda_{n}(z)^{17}}{n}\right). \tag*{(128)} n 1 k = 1 ∑ n β k H , Q L p = O ( n p 2 Λ n ( z ) 17 ) . (128)
Applying Minkowski’s inequality to the n n n increments for H H H and the n − 1 n-1 n − 1 increments for G G G , we have
max { 1 n ∥ ∑ k = 1 n β k H , Q ∥ L p , ∥ ∑ k = 1 n − 1 β k G , Q ∥ L p } = O ( p 2 Λ n ( z ) 19 n ) . \max\left\{\frac{1}{n}\left\lVert\sum_{k=1}^{n}\beta_{k}^{H,Q}\right\rVert_{L^{p}},\left\lVert\sum_{k=1}^{n-1}\beta_{k}^{G,Q}\right\rVert_{L^{p}}\right\}=O\left(\frac{p^{2}\Lambda_{n}(z)^{19}}{n}\right). max { n 1 k = 1 ∑ n β k H , Q L p , k = 1 ∑ n − 1 β k G , Q L p } = O ( n p 2 Λ n ( z ) 19 ) .
The proof is complete by adding the initial corrections bounded in Lemma E.13, Eq. (103b).
Proof of Lemma E.12
Within the proof, we assume either ∥ Q ∥ ≤ 1 \lVert Q\rVert\le1 ∥ Q ∥ ≤ 1 or ∥ Q ∥ F ≤ n \lVert Q\rVert_F \le\sqrt{n} ∥ Q ∥ F ≤ n as linearity in Q Q Q allows rescaling.
Part I: Reduction to numerator and denominator moments. For any fixed s , j s,j s , j under the setup of Lemma E.12, let R ~ s ( z , w j ) = Σ 1 / 2 R s ( z , w j ) Σ 1 / 2 \widetilde{R}_s(z,w_j)=\Sigma^{1/2}R_s(z,w_j)\Sigma^{1/2} R s ( z , w j ) = Σ 1/2 R s ( z , w j ) Σ 1/2 , and we shall suppress the dependence on ( z , w j ) (z,w_j) ( z , w j ) for notational simplicity. Starting from the left-hand side of Eq. (97), we apply conditional Jensen’s inequality by freezing the denominator first, and then the triangle inequality to obtain
max { ∥ V s + 1 H , Q − E s V s + 1 H , Q ∥ L p n , ∥ V s + 1 G , Q − E s V s + 1 G , Q ∥ L p } \max\left\{\frac{\left\lVert V_{s+1}^{H,Q}-\mathbb{E}_sV_{s+1}^{H,Q}\right\rVert_{L^p}}{n},\left\lVert V_{s+1}^{G,Q}-\mathbb{E}_sV_{s+1}^{G,Q}\right\rVert_{L^p}\right\} max ⎩ ⎨ ⎧ n V s + 1 H , Q − E s V s + 1 H , Q L p , V s + 1 G , Q − E s V s + 1 G , Q L p ⎭ ⎬ ⎫
≤ 2 max { 1 n ∥ V s + 1 H , Q − z Tr ( Q R s Σ R s ) 1 + Tr R ~ s ∥ L p , ∥ V s + 1 G , Q − Tr ( Q R ~ s 2 ) 1 + Tr R ~ s ∥ L p } . \le2\max\left\{\frac{1}{n}\left\lVert V_{s+1}^{H,Q}-\frac{z\operatorname{Tr}(Q R_s\Sigma R_s)}{1+\operatorname{Tr}\widetilde{R}_s}\right\rVert_{L^p},\left\lVert V_{s+1}^{G,Q}-\frac{\operatorname{Tr}(Q\widetilde{R}_s^2)}{1+\operatorname{Tr}\widetilde{R}_s}\right\rVert_{L^p}\right\}. ≤ 2 max { n 1 V s + 1 H , Q − 1 + Tr R s z Tr ( Q R s Σ R s ) L p , V s + 1 G , Q − 1 + Tr R s Tr ( Q R s 2 ) L p } .
It then boils down to bounding the right hand side in the above display. To see what estimates we need, for the G G G term, we expand
V s + 1 G , Q − Tr ( Q R ~ s 2 ) 1 + Tr R ~ s = z s + 1 ∗ R ~ s Q R ~ s z s + 1 − Tr ( Q R ~ s 2 ) 1 + z s + 1 ∗ R ~ s z s + 1 − Tr ( Q R ~ s 2 ) ( z s + 1 ∗ R ~ s z s + 1 − Tr R ~ s ) ( 1 + Tr R ~ s ) ( 1 + z s + 1 ∗ R ~ s z s + 1 ) . (129) V_{s+1}^{G,Q}-\frac{\operatorname{Tr}(Q\widetilde{R}_s^2)}{1+\operatorname{Tr}\widetilde{R}_s}
=\frac{z_{s+1}^*\widetilde{R}_sQ\widetilde{R}_sz_{s+1}-\operatorname{Tr}(Q\widetilde{R}_s^2)}{1+z_{s+1}^*\widetilde{R}_sz_{s+1}}-\frac{\operatorname{Tr}(Q\widetilde{R}_s^2)(z_{s+1}^*\widetilde{R}_sz_{s+1}-\operatorname{Tr}\widetilde{R}_s)}{(1+\operatorname{Tr}\widetilde{R}_s)(1+z_{s+1}^*\widetilde{R}_sz_{s+1})}. \tag*{(129)} V s + 1 G , Q − 1 + Tr R s Tr ( Q R s 2 ) = 1 + z s + 1 ∗ R s z s + 1 z s + 1 ∗ R s Q R s z s + 1 − Tr ( Q R s 2 ) − ( 1 + Tr R s ) ( 1 + z s + 1 ∗ R s z s + 1 ) Tr ( Q R s 2 ) ( z s + 1 ∗ R s z s + 1 − Tr R s ) . (129)
We bound the denominators above first. The quadratic-form can be controlled by Lemma E.1.(i). For the trace denominator, note that Tr R ~ s = G s + 1 ( z , w j ) \operatorname{Tr}\widetilde{R}_s=G_{s+1}(z,w_j) Tr R s = G s + 1 ( z , w j ) . Repeating the argument for G s + 1 G_{s+1} G s + 1 in Part II of that lemma’s proof before taking expectation gives Im ( z ( 1 + Tr R ~ s ) ) ≥ η \operatorname{Im}(z(1+\operatorname{Tr}\widetilde{R}_s))\ge\eta Im ( z ( 1 + Tr R s )) ≥ η , as in Eq. (93) , and Re Tr R ~ s ≥ 0 \operatorname{Re}\operatorname{Tr}\widetilde{R}_s\ge0 Re Tr R s ≥ 0 when t < 0 t<0 t < 0 . Thus both denominators satisfy
∣ 1 + z s + 1 ∗ R ~ s z s + 1 ∣ − 1 ∨ ∣ 1 + Tr R ~ s ∣ − 1 ≤ ∣ z ∣ ∣ z − ∣ ≤ Λ n ( z ) . \lvert1+z_{s+1}^*\widetilde{R}_sz_{s+1}\rvert^{-1}\vee\lvert1+\operatorname{Tr}\widetilde{R}_s\rvert^{-1}\le\frac{\lvert z\rvert}{\lvert z_-\rvert}\le\Lambda_n(z). ∣ 1 + z s + 1 ∗ R s z s + 1 ∣ − 1 ∨ ∣ 1 + Tr R s ∣ − 1 ≤ ∣ z − ∣ ∣ z ∣ ≤ Λ n ( z ) .
Applying the denominator bounds and Cauchy–Schwarz to Eq. (129) yields
∥ V s + 1 G , Q − E s V s + 1 G , Q ∥ L p ≤ 2 Λ n ( z ) ∥ z s + 1 ∗ R ~ s Q R ~ s z s + 1 − Tr ( Q R ~ s 2 ) ∥ L p + 2 Λ n ( z ) 2 ∥ Tr ( Q R ~ s 2 ) ∥ L 2 p ∥ z s + 1 ∗ R ~ s z s + 1 − Tr R ~ s ∥ L 2 p . (130) \left\lVert V_{s+1}^{G,Q}-\mathbb{E}_sV_{s+1}^{G,Q}\right\rVert_{L^p}\le2\Lambda_n(z)\left\lVert z_{s+1}^*\widetilde{R}_sQ\widetilde{R}_sz_{s+1}-\operatorname{Tr}(Q\widetilde{R}_s^2)\right\rVert_{L^p}
+2\Lambda_n(z)^2\left\lVert\operatorname{Tr}(Q\widetilde{R}_s^2)\right\rVert_{L^{2p}}\left\lVert z_{s+1}^*\widetilde{R}_sz_{s+1}-\operatorname{Tr}\widetilde{R}_s\right\rVert_{L^{2p}}. \tag*{(130)} V s + 1 G , Q − E s V s + 1 G , Q L p ≤ 2 Λ n ( z ) z s + 1 ∗ R s Q R s z s + 1 − Tr ( Q R s 2 ) L p + 2 Λ n ( z ) 2 Tr ( Q R s 2 ) L 2 p z s + 1 ∗ R s z s + 1 − Tr R s L 2 p . (130)
Similarly, for V s + 1 H , Q V_{s+1}^{H,Q} V s + 1 H , Q , we have
∥ V s + 1 H , Q − E s V s + 1 H , Q ∥ L p n ≤ 2 Λ n ( z ) n ∥ z s + 1 ∗ Σ 1 / 2 R s Q R s Σ 1 / 2 z s + 1 − z Tr ( Q R s Σ R s ) ∥ L p + 2 Λ n ( z ) 2 n ∥ z Tr ( Q R s Σ R s ) ∥ L 2 p ∥ z s + 1 ∗ R ~ s z s + 1 − Tr R ~ s ∥ L 2 p . (131) \frac{\left\lVert V_{s+1}^{H,Q}-\mathbb{E}_sV_{s+1}^{H,Q}\right\rVert_{L^p}}{n}\le\frac{2\Lambda_n(z)}{n}\left\lVert z_{s+1}^*\Sigma^{1/2}R_sQR_s\Sigma^{1/2}z_{s+1}-z\operatorname{Tr}(Q R_s\Sigma R_s)\right\rVert_{L^p}
+\frac{2\Lambda_n(z)^2}{n}\left\lVert z\operatorname{Tr}(Q R_s\Sigma R_s)\right\rVert_{L^{2p}}\left\lVert z_{s+1}^*\widetilde{R}_sz_{s+1}-\operatorname{Tr}\widetilde{R}_s\right\rVert_{L^{2p}}. \tag*{(131)} n V s + 1 H , Q − E s V s + 1 H , Q L p ≤ n 2 Λ n ( z ) z s + 1 ∗ Σ 1/2 R s Q R s Σ 1/2 z s + 1 − z Tr ( Q R s Σ R s ) L p + n 2 Λ n ( z ) 2 ∥ z Tr ( Q R s Σ R s ) ∥ L 2 p z s + 1 ∗ R s z s + 1 − Tr R s L 2 p . (131)
Thus it suffices to control the numerator moments on the right-hand sides above.
Part I-i: Reduction of G G G -terms numerator moments. For 2 ≤ q ≤ 2 κ log n 2\le q\le2\kappa\log n 2 ≤ q ≤ 2 κ log n , applying conditional Hanson-Wright and integrating the tail, as in Eq. (117) , we have
∥ z s + 1 ∗ R ~ s Q R ~ s z s + 1 − Tr ( Q R ~ s 2 ) ∥ L q ≤ O C x ( q ) ∥ ∥ R ~ s Q R ~ s ∥ F ∥ L q ≤ O C x ( q ) ⋅ ( ∥ ∥ R ~ s ∥ ∥ R ~ s ∥ F ∥ L q ∨ ∥ n ∥ R ~ s ∥ 2 ∥ L q ) , \left\lVert z_{s+1}^*\widetilde{R}_sQ\widetilde{R}_sz_{s+1}-\operatorname{Tr}(Q\widetilde{R}_s^2)\right\rVert_{L^q}\le O C_x(q)\left\lVert\left\lVert\widetilde{R}_sQ\widetilde{R}_s\right\rVert_F\right\rVert_{L^q}
\le O C_x(q)\cdot\left(\left\lVert\left\lVert\widetilde{R}_s\right\rVert\left\lVert\widetilde{R}_s\right\rVert_F\right\rVert_{L^q}\vee\left\lVert\sqrt{n}\left\lVert\widetilde{R}_s\right\rVert^2\right\rVert_{L^q}\right), z s + 1 ∗ R s Q R s z s + 1 − Tr ( Q R s 2 ) L q ≤ O C x ( q ) R s Q R s F L q ≤ O C x ( q ) ⋅ ( R s R s F L q ∨ n R s 2 L q ) ,
∥ z s + 1 ∗ R ~ s z s + 1 − Tr R ~ s ∥ L q ≤ O C x ( q ) ⋅ ∥ ∥ R ~ s ∥ F ∥ L q , (132) \left\lVert z_{s+1}^*\widetilde{R}_sz_{s+1}-\operatorname{Tr}\widetilde{R}_s\right\rVert_{L^q}\le O C_x(q)\cdot\left\lVert\left\lVert\widetilde{R}_s\right\rVert_F\right\rVert_{L^q}, \tag*{(132)} z s + 1 ∗ R s z s + 1 − Tr R s L q ≤ O C x ( q ) ⋅ R s F L q , (132)
where we invoke the operator-Frobenius product inequality in (i), using either ∥ Q ∥ ≤ 1 \lVert Q\rVert\le1 ∥ Q ∥ ≤ 1 or ∥ Q ∥ F ≤ n \lVert Q\rVert_F\le\sqrt{n} ∥ Q ∥ F ≤ n . For the conditional trace in Eq. (106a), we apply the trace Cauchy-Schwarz and the same product inequality to obtain
∥ Tr ( Q R ~ s 2 ) ∥ L q ≤ ∥ ∥ R ~ s ∥ F 2 ∥ L q ∨ ∥ n ∥ R ~ s ∥ ∥ R ~ s ∥ F ∥ L q . \left\lVert\operatorname{Tr}(Q\widetilde{R}_s^2)\right\rVert_{L^q}\le\left\lVert\left\lVert\widetilde{R}_s\right\rVert_F^2\right\rVert_{L^q}\vee\left\lVert\sqrt{n}\left\lVert\widetilde{R}_s\right\rVert\left\lVert\widetilde{R}_s\right\rVert_F\right\rVert_{L^q}. Tr ( Q R s 2 ) L q ≤ R s F 2 L q ∨ n R s R s F L q .
Part I-ii: Reduction of H H H -terms numerator moments. For the H H H -terms numerator, we will express the bound through the same weighted resolvent R ~ s \widetilde{R}_s R s . We define the operator modulus by its semidefinite square-root ∣ A ∣ ≔ ( A ∗ A ) 1 / 2 |A| \coloneqq(A^{*}A)^{1/2} ∣ A ∣ : = ( A ∗ A ) 1/2 when A A A is Hilbert–Schmidt. Taking congruences by R s R_s R s and R s ∗ R_s^{*} R s ∗ in the lower bound Re ( e i θ R s − 1 ) ⪰ ∣ z − ∣ I / 2 \operatorname{Re}(e^{i\theta}R_s^{-1}) \succeq|z_-|I/2 Re ( e i θ R s − 1 ) ⪰ ∣ z − ∣ I /2 from Part I of the proof of Lemma E.2, Eq. (96) and multiplying by Σ 1 / 2 \Sigma^{1/2} Σ 1/2 and taking norms or traces imply
∥ R s Σ 1 / 2 ∥ 2 2 ∨ ∥ Σ 1 / 2 R s ∥ 2 2 ≤ 2 ∥ R ~ s ∥ ∣ z − ∣ and ∥ R s Σ 1 / 2 ∥ F 2 ∨ ∥ Σ 1 / 2 R s ∥ F 2 ≤ 2 Tr ∣ R ~ s ∣ ∣ z − ∣ . (133) \lVert R_s\Sigma^{1/2}\rVert_2^2 \vee\lVert\Sigma^{1/2}R_s\rVert_2^2 \leq\frac{2\lVert\widetilde{R}_s\rVert}{|z_-|} \quad\text{and} \quad\lVert R_s\Sigma^{1/2}\rVert_F^2 \vee\lVert\Sigma^{1/2}R_s\rVert_F^2 \leq\frac{2\operatorname{Tr}|\widetilde{R}_s|}{|z_-|}. \tag*{(133)} ∥ R s Σ 1/2 ∥ 2 2 ∨ ∥ Σ 1/2 R s ∥ 2 2 ≤ ∣ z − ∣ 2 ∥ R s ∥ and ∥ R s Σ 1/2 ∥ F 2 ∨ ∥ Σ 1/2 R s ∥ F 2 ≤ ∣ z − ∣ 2 Tr ∣ R s ∣ . (133)
Therefore, using ∣ z ∣ / ∣ z − ∣ ≤ Λ n ( z ) |z|/|z_-| \leq\Lambda_n(z) ∣ z ∣/∣ z − ∣ ≤ Λ n ( z ) , we have
∥ z Σ 1 / 2 R s Q R s Σ 1 / 2 ∥ F ≤ 2 Λ n ( z ) ⋅ ( ∥ R ~ s ∥ Tr ∣ R ~ s ∣ ∨ ( n ∥ R ~ s ∥ ) ) , \left\lVert z\Sigma^{1/2}R_sQR_s\Sigma^{1/2}\right\rVert_F \leq2\Lambda_n(z)\cdot\left(\sqrt{\lVert\widetilde{R}_s\rVert\operatorname{Tr}|\widetilde{R}_s|}\vee\left(\sqrt{n}\lVert\widetilde{R}_s\rVert\right)\right), z Σ 1/2 R s Q R s Σ 1/2 F ≤ 2 Λ n ( z ) ⋅ ( ∥ R s ∥ Tr ∣ R s ∣ ∨ ( n ∥ R s ∥ ) ) ,
∣ z Tr ( Q R s Σ R s ) ∣ ≤ 2 Λ n ( z ) ⋅ ( Tr ∣ R ~ s ∣ ∨ n ∥ R ~ s ∥ Tr ∣ R ~ s ∣ ) . \left|z\operatorname{Tr}(QR_s\Sigma R_s)\right| \leq2\Lambda_n(z)\cdot\left(\operatorname{Tr}|\widetilde{R}_s|\vee\sqrt{n}\lVert\widetilde{R}_s\rVert\operatorname{Tr}|\widetilde{R}_s|\right). ∣ z Tr ( Q R s Σ R s ) ∣ ≤ 2 Λ n ( z ) ⋅ ( Tr ∣ R s ∣ ∨ n ∥ R s ∥ Tr ∣ R s ∣ ) .
The first alternative in each maximum applies when ∥ Q ∥ ≤ 1 \lVert Q\rVert\leq1 ∥ Q ∥ ≤ 1 , and the second when ∥ Q ∥ F ≤ n \lVert Q\rVert_F\leq\sqrt{n} ∥ Q ∥ F ≤ n . Applying the same conditional Hanson-Wright calculation for G G G -terms, it holds
∥ z z s + 1 ∗ Σ 1 / 2 R s Q R s Σ 1 / 2 z s + 1 − z Tr ( Q R s Σ R s ) ∥ L q ≤ O C x ( q Λ n ( z ) ) ⋅ ( ( ∥ R ~ s ∥ Tr ∣ R ~ s ∣ ) L q ∨ ( n ∥ R ~ s ∥ ) L q ) . \left\lVert zz_{s+1}^{*}\Sigma^{1/2}R_sQR_s\Sigma^{1/2}z_{s+1}-z\operatorname{Tr}(QR_s\Sigma R_s)\right\rVert_{L^q}
\leq O_{C_x}\bigl(q\Lambda_n(z)\bigr)\cdot\left(\left(\sqrt{\lVert\widetilde{R}_s\rVert\operatorname{Tr}|\widetilde{R}_s|}\right)_{L^q}\vee\left(\sqrt{n}\lVert\widetilde{R}_s\rVert\right)_{L^q}\right). z z s + 1 ∗ Σ 1/2 R s Q R s Σ 1/2 z s + 1 − z Tr ( Q R s Σ R s ) L q ≤ O C x ( q Λ n ( z ) ) ⋅ ( ( ∥ R s ∥ Tr ∣ R s ∣ ) L q ∨ ( n ∥ R s ∥ ) L q ) .
Combining the reduction to weighted resolvent moments above and substituting into Eqs. (106a) and (106b), we have
∥ V s + 1 G , Q − E s V s + 1 G , Q ∥ L p ≤ O C x ( p ) ⋅ [ Λ n ( z ) 2 ∥ R ~ s ∥ ∥ R ~ s ∥ F + n ∥ R ~ s ∥ 2 ] L p + Λ n ( z ) 2 ∥ ∥ R ~ s ∥ F 2 + n ∥ R ~ s ∥ ∥ R ~ s ∥ F ∥ L 2 p ∥ ∥ R ~ s ∥ F ∥ L 2 p , \left\lVert V_{s+1}^{G,Q}-\mathbb{E}_sV_{s+1}^{G,Q}\right\rVert_{L^p}
\leq O_{C_x}(p)\cdot\left[\Lambda_n(z)^2\lVert\widetilde{R}_s\rVert\lVert\widetilde{R}_s\rVert_F+\sqrt{n}\lVert\widetilde{R}_s\rVert^2\right]_{L^p}
+\Lambda_n(z)^2\left\lVert\lVert\widetilde{R}_s\rVert_F^2+\sqrt{n}\lVert\widetilde{R}_s\rVert\lVert\widetilde{R}_s\rVert_F\right\rVert_{L^{2p}}\lVert\lVert\widetilde{R}_s\rVert_F\rVert_{L^{2p}}, V s + 1 G , Q − E s V s + 1 G , Q L p ≤ O C x ( p ) ⋅ [ Λ n ( z ) 2 ∥ R s ∥ ∥ R s ∥ F + n ∥ R s ∥ 2 ] L p + Λ n ( z ) 2 ∥ R s ∥ F 2 + n ∥ R s ∥ ∥ R s ∥ F L 2 p ∥∥ R s ∥ F ∥ L 2 p ,
∥ V s + 1 H , Q − E s V s + 1 H , Q ∥ L p n ≤ O C x ( p ) n ⋅ [ Λ n ( z ) 2 ∥ ∥ R ~ s ∥ Tr ∣ R ~ s ∣ + n ∥ R ~ s ∥ ∥ L p + Λ n ( z ) 3 ∥ Tr ∣ R ~ s ∣ + n ∥ R ~ s ∥ Tr ∣ R ~ s ∣ ∥ L 2 p ∥ ∥ R ~ s ∥ F ∥ L 2 p ] . (134) \frac{\left\lVert V_{s+1}^{H,Q}-\mathbb{E}_sV_{s+1}^{H,Q}\right\rVert_{L^p}}{n}
\leq\frac{O_{C_x}(p)}{n}\cdot\left[\Lambda_n(z)^2\left\lVert\sqrt{\lVert\widetilde{R}_s\rVert\operatorname{Tr}|\widetilde{R}_s|}+\sqrt{n}\lVert\widetilde{R}_s\rVert\right\rVert_{L^p}
+\Lambda_n(z)^3\left\lVert\operatorname{Tr}|\widetilde{R}_s|+\sqrt{n}\lVert\widetilde{R}_s\rVert\operatorname{Tr}|\widetilde{R}_s|\right\rVert_{L^{2p}}\lVert\lVert\widetilde{R}_s\rVert_F\rVert_{L^{2p}}\right]. \tag*{(134)} n V s + 1 H , Q − E s V s + 1 H , Q L p ≤ n O C x ( p ) ⋅ [ Λ n ( z ) 2 ∥ R s ∥ Tr ∣ R s ∣ + n ∥ R s ∥ L p + Λ n ( z ) 3 Tr ∣ R s ∣ + n ∥ R s ∥ Tr ∣ R s ∣ L 2 p ∥∥ R s ∥ F ∥ L 2 p ] . (134)
It remains to control the operator, Frobenius, and trace-norm moments of R ~ s \widetilde{R}_s R s .
Part II: Bounding the resolvent moments. Throughout this part, we suppress dependence on C x , ν , κ C_x,\nu,\kappa C x , ν , κ in big-Oh notations. To close Eq. (134) , we prove, for 2 ≤ q ≤ 4 κ log n 2\leq q\leq4\kappa\log n 2 ≤ q ≤ 4 κ log n ,
∥ ∥ R ~ s ∥ ∥ L q = O ( Λ n ( z ) 3 n ) , ∥ ∥ R ~ s ∥ F ∥ L q = O ( Λ n ( z ) 3 n ) , ∥ Tr ∣ R ~ s ∣ ∥ L q = O ( Λ n ( z ) 3 ) . (135) \lVert\lVert\widetilde{R}_s\rVert\rVert_{L^q}=O\left(\frac{\Lambda_n(z)^3}{n}\right),\qquad
\lVert\lVert\widetilde{R}_s\rVert_F\rVert_{L^q}=O\left(\frac{\Lambda_n(z)^3}{\sqrt{n}}\right),\qquad
\lVert\operatorname{Tr}|\widetilde{R}_s|\rVert_{L^q}=O\left(\Lambda_n(z)^3\right). \tag*{(135)} ∥∥ R s ∥ ∥ L q = O ( n Λ n ( z ) 3 ) , ∥∥ R s ∥ F ∥ L q = O ( n Λ n ( z ) 3 ) , ∥ Tr ∣ R s ∣ ∥ L q = O ( Λ n ( z ) 3 ) . (135)
The first two bounds follow from our existing arguments. For j > ℓ ≔ ⌊ n / 2 ⌋ j>\ell\coloneqq\lfloor n/2\rfloor j > ℓ : = ⌊ n /2 ⌋ , combining Lemma E.2, Eq. (44a), with Lemma E.3, Eqs. (45a) and (45b), at ζ = ∣ z − ∣ \zeta=|z_-| ζ = ∣ z − ∣ and moment orders q , q / 2 q,q/2 q , q /2 gives the (strengthened) first two bounds, with Λ n ( z ) 3 \Lambda_n(z)^3 Λ n ( z ) 3 replaced by Λ n ( z ) 2 \Lambda_n(z)^2 Λ n ( z ) 2 . To see how these lemmas apply, note that s ≥ ℓ − 2 ≍ n s\geq\ell-2\asymp n s ≥ ℓ − 2 ≍ n , so the minimizing cutoff in Assumption 2 satisfies m ≤ ρ ( C x ) ( ℓ − 2 ) ≤ ρ ( C x ) s m\leq\rho(C_x)(\ell-2)\leq\rho(C_x)s m ≤ ρ ( C x ) ( ℓ − 2 ) ≤ ρ ( C x ) s and, for sufficiently large n n n , q ≤ 4 κ log n ≤ 8 κ log s q\leq4\kappa\log n\leq8\kappa\log s q ≤ 4 κ log n ≤ 8 κ log s . We therefore apply Lemma E.3 with κ \kappa κ replaced by 8 κ 8\kappa 8 κ for both orders q q q and q / 2 q/2 q /2 . To evaluate the resulting bounds, use Eq. (66) from Part III-i of the proof of Theorem 3 to control the spectral terms by Λ n ( z ) \Lambda_n(z) Λ n ( z ) , and take D = 2 ν + 2 D=2\nu+2 D = 2 ν + 2 to absorb the exceptional-event terms, as verified immediately after that equation. Lastly, for j ≤ ℓ j\leq\ell j ≤ ℓ , combining Lemma E.2 , Eq. (44b), with the path lower bound in Eq. (70) from that same part gives the first two bounds pointwise.
For the third estimate, we use the rotation from Part I of the proof of Lemma E.2 , with U = e i θ R s − 1 U=e^{i\theta}R_s^{-1} U = e i θ R s − 1 and V = Re U V=\operatorname{Re}U V = Re U and obtain that
Tr ∣ R ~ s ∣ = Tr ∣ ( Σ 1 / 2 V − 1 / 2 ) ( V 1 / 2 U − 1 Σ 1 / 2 ) ∣ ≤ ∥ Σ 1 / 2 V − 1 / 2 ∥ F ∥ V 1 / 2 U − 1 Σ 1 / 2 ∥ F = Tr ( Σ 1 / 2 V − 1 Σ 1 / 2 ) Tr ( Σ 1 / 2 U − ∗ V U − 1 Σ 1 / 2 ) ≤ Tr ( Σ 1 / 2 V − 1 Σ 1 / 2 ) . \begin{aligned}
\operatorname{Tr}|\widetilde{R}_s|=\operatorname{Tr}\left|\left(\Sigma^{1/2}V^{-1/2}\right)\left(V^{1/2}U^{-1}\Sigma^{1/2}\right)\right| \\
&\leq\left\|\Sigma^{1/2}V^{-1/2}\right\|_F\left\|V^{1/2}U^{-1}\Sigma^{1/2}\right\|_F=\sqrt{\operatorname{Tr}\left(\Sigma^{1/2}V^{-1}\Sigma^{1/2}\right)}\sqrt{\operatorname{Tr}\left(\Sigma^{1/2}U^{-*}VU^{-1}\Sigma^{1/2}\right)} \\
&\leq\operatorname{Tr}\left(\Sigma^{1/2}V^{-1}\Sigma^{1/2}\right).
\end{aligned} Tr ∣ R s ∣ = Tr ( Σ 1/2 V − 1/2 ) ( V 1/2 U − 1 Σ 1/2 ) ≤ Σ 1/2 V − 1/2 F V 1/2 U − 1 Σ 1/2 F = Tr ( Σ 1/2 V − 1 Σ 1/2 ) Tr ( Σ 1/2 U −∗ V U − 1 Σ 1/2 ) ≤ Tr ( Σ 1/2 V − 1 Σ 1/2 ) .
Here in (i), we apply the trace-norm Cauchy-Schwarz, i.e. Tr ∣ A B ∣ ≤ ∥ A ∥ F ∥ B ∥ F \operatorname{Tr}|AB|\leq\|A\|_F\|B\|_F Tr ∣ A B ∣ ≤ ∥ A ∥ F ∥ B ∥ F , to the two displayed factors. In (ii), we take the trace in the positive-operator inequality from Eq. (97) . We similarly discuss the large sample and small sample regimes separately. For j > ℓ j>\ell j > ℓ , the first lower bound in Eq. (96) in Part I of the proof of Lemma E.2 implies
∥ Tr ∣ R ~ s ∣ ∥ L q ≤ 3 ∣ z ∣ ∣ z − ∣ ∥ Tr A s ( ∣ z − ∣ ) ∥ L q = O ( Λ n ( z ) 2 ) . \left\|\operatorname{Tr}|\widetilde{R}_s|\right\|_{L^q}\leq\frac{3|z|}{|z_-|}\left\|\operatorname{Tr}A_s(|z_-|)\right\|_{L^q}=O\left(\Lambda_n(z)^2\right). Tr ∣ R s ∣ L q ≤ ∣ z − ∣ 3∣ z ∣ ∥ Tr A s ( ∣ z − ∣ ) ∥ L q = O ( Λ n ( z ) 2 ) .
Using the same parameter choices as for the first two estimates, we apply Lemma E.3 , Eq. (45c), at order q q q . The bounds τ 1 ( m ) / ∣ z − ∣ ≤ Λ n ( z ) \tau_1(m)/|z_-|\leq\Lambda_n(z) τ 1 ( m ) /∣ z − ∣ ≤ Λ n ( z ) and Tr Σ ≤ n + τ 1 ( m ) \operatorname{Tr}\Sigma\leq n+\tau_1(m) Tr Σ ≤ n + τ 1 ( m ) control the last two terms, giving the displayed O ( Λ n ( z ) 2 ) O(\Lambda_n(z)^2) O ( Λ n ( z ) 2 ) bound. For j ≤ ℓ j\leq\ell j ≤ ℓ , we use the second lower bound in Eq. (96) from Part I of the proof of Lemma E.2 instead to estimate a pointwise bound:
Tr ∣ R ~ s ∣ ≤ 2 ∑ i σ i ∣ z − ∣ − 2 Re ( e i θ w j ) σ i ≤ m − Re ( e i θ w j ) + 2 τ 1 ( m ) ∣ z − ∣ = O ( Λ n ( z ) 3 ) . \operatorname{Tr}|\widetilde{R}_s|\leq2\sum_i\frac{\sigma_i}{|z_-|-2\operatorname{Re}(e^{i\theta}w_j)\sigma_i}\leq\frac{m}{-\operatorname{Re}(e^{i\theta}w_j)}+\frac{2\tau_1(m)}{|z_-|}=O\left(\Lambda_n(z)^3\right). Tr ∣ R s ∣ ≤ 2 i ∑ ∣ z − ∣ − 2 Re ( e i θ w j ) σ i σ i ≤ − Re ( e i θ w j ) m + ∣ z − ∣ 2 τ 1 ( m ) = O ( Λ n ( z ) 3 ) .
The last step uses m ≤ n m\leq n m ≤ n and the path lower bound in Eq. (70) from Part III-i of the proof of Theorem 3. We complete the proof of the third estimate in Eq. (135) by taking its L q L^q L q norm. Therefore, substituting all weighted resolvent bounds in Eq. (134) and using Cauchy-Schwarz, we conclude that
∥ V s + 1 G , Q − E s V s + 1 G , Q ∥ L p = O ( p Λ n ( z ) 7 n 3 / 2 + p Λ n ( z ) 11 n 3 / 2 ) , \left\|V_{s+1}^{G,Q}-\mathbb{E}_sV_{s+1}^{G,Q}\right\|_{L^p}=O\left(\frac{p\Lambda_n(z)^7}{n^{3/2}}+\frac{p\Lambda_n(z)^{11}}{n^{3/2}}\right), V s + 1 G , Q − E s V s + 1 G , Q L p = O ( n 3/2 p Λ n ( z ) 7 + n 3/2 p Λ n ( z ) 11 ) ,
∥ V s + 1 H , Q − E s V s + 1 H , Q ∥ L p n = O ( p Λ n ( z ) 5 n 3 / 2 + p Λ n ( z ) 9 n 3 / 2 ) . \frac{\left\|V_{s+1}^{H,Q}-\mathbb{E}_sV_{s+1}^{H,Q}\right\|_{L^p}}{n}=O\left(\frac{p\Lambda_n(z)^5}{n^{3/2}}+\frac{p\Lambda_n(z)^9}{n^{3/2}}\right). n V s + 1 H , Q − E s V s + 1 H , Q L p = O ( n 3/2 p Λ n ( z ) 5 + n 3/2 p Λ n ( z ) 9 ) .
All resolvent moments above have order at most 4 p ≤ 4 κ log n 4p\leq4\kappa\log n 4 p ≤ 4 κ log n . Since Λ n ( z ) ≥ 1 \Lambda_n(z)\geq1 Λ n ( z ) ≥ 1 , we complete the proof by rescaling Q Q Q .
Part III: Additional uncentered bounds. For the later estimation of β \beta β -terms, we record a useful (and cruder) consequence for 2 ≤ q ≤ 2 κ log n 2\leq q\leq2\kappa\log n 2 ≤ q ≤ 2 κ log n . Substituting the weighted resolvent bounds in Eq. (135) into the numerator inequalities from Part I and applying Hölder, we have
∥ Tr ( Q R ~ s 2 ) ∥ L q = O ( Λ n ( z ) 6 n ) , ∥ ∥ R ~ s Q R ~ s ∥ F ∥ L q = O ( Λ n ( z ) 6 n 3 / 2 ) , \left\|\operatorname{Tr}(Q\widetilde{R}_s^2)\right\|_{L^q}=O\left(\frac{\Lambda_n(z)^6}{n}\right),\qquad\left\|\left\|\widetilde{R}_sQ\widetilde{R}_s\right\|_F\right\|_{L^q}=O\left(\frac{\Lambda_n(z)^6}{n^{3/2}}\right), Tr ( Q R s 2 ) L q = O ( n Λ n ( z ) 6 ) , R s Q R s F L q = O ( n 3/2 Λ n ( z ) 6 ) ,
∥ z Tr ( Q R s Σ R s ) ∥ L q = O ( Λ n ( z ) 4 ) , ∥ ∥ z Σ 1 / 2 R s Q R s Σ 1 / 2 ∥ F ∥ L q = O ( Λ n ( z ) 4 n ) . (136) \left\|z\operatorname{Tr}(QR_s\Sigma R_s)\right\|_{L^q}=O\left(\Lambda_n(z)^4\right),\qquad\left\|\left\|z\Sigma^{1/2}R_sQR_s\Sigma^{1/2}\right\|_F\right\|_{L^q}=O\left(\frac{\Lambda_n(z)^4}{\sqrt{n}}\right). \tag*{(136)} ∥ z Tr ( Q R s Σ R s ) ∥ L q = O ( Λ n ( z ) 4 ) , z Σ 1/2 R s Q R s Σ 1/2 F L q = O ( n Λ n ( z ) 4 ) . (136)
To bound the uncentered ratios, apply the reciprocal bound from Part I, then conditional Hanson-Wright and Eq. (136) :
∥ V s + 1 G , Q ∥ L q n ≤ Λ n ( z ) ( ∥ Tr ( Q R ~ s 2 ) ∥ L q + O C x ( q ) ∥ R ~ s Q R ~ s ∥ F ∥ L q ) = O ( Λ n ( z ) 7 n ) , \frac{\lVert V_{s+1}^{G,Q}\rVert_{L^q}}{n} \le\Lambda_n(z)\left(\lVert\operatorname{Tr}(Q\widetilde{R}_s^2)\rVert_{L^q}+O_{C_x}(q)\lVert\widetilde{R}_s Q\widetilde{R}_s\rVert_F\big\rVert_{L^q}\right)=O\left(\frac{\Lambda_n(z)^7}{n}\right), n ∥ V s + 1 G , Q ∥ L q ≤ Λ n ( z ) ( ∥ Tr ( Q R s 2 ) ∥ L q + O C x ( q ) ∥ R s Q R s ∥ F L q ) = O ( n Λ n ( z ) 7 ) ,
∥ V s + 1 H , Q ∥ L q n ≤ Λ n ( z ) n ( ∥ z Tr ( Q R s Σ R s ) ∥ L q + O C x ( q ) ∥ z Σ 1 / 2 R s Q R s Σ 1 / 2 ∥ F ∥ L q ) = O ( Λ n ( z ) 5 n ) . \begin{aligned}
\frac{\lVert V_{s+1}^{H,Q}\rVert_{L^q}}{n}\le\frac{\Lambda_n(z)}{n}\left(\lVert z\operatorname{Tr}(QR_s\Sigma R_s)\rVert_{L^q}+O_{C_x}(q)\lVert z\Sigma^{1/2}R_sQR_s\Sigma^{1/2}\rVert_F\big\rVert_{L^q}\right) \\
&=O\left(\frac{\Lambda_n(z)^5}{n}\right).
\end{aligned} n ∥ V s + 1 H , Q ∥ L q ≤ n Λ n ( z ) ( ∥ z Tr ( Q R s Σ R s ) ∥ L q + O C x ( q ) ∥ z Σ 1/2 R s Q R s Σ 1/2 ∥ F L q ) = O ( n Λ n ( z ) 5 ) .
where we use q ≤ 2 κ log n q\le2\kappa\log n q ≤ 2 κ log n . We thus obtain from Λ n ( z ) ≥ 1 \Lambda_n(z)\ge1 Λ n ( z ) ≥ 1 that
∥ V s + 1 H , Q ∥ L q n ∨ ∥ V s + 1 G , Q ∥ L q = O ( Λ n ( z ) 7 n ) , (137) \frac{\lVert V_{s+1}^{H,Q}\rVert_{L^q}}{n}\vee\lVert V_{s+1}^{G,Q}\rVert_{L^q}=O\left(\frac{\Lambda_n(z)^7}{n}\right), \tag*{(137)} n ∥ V s + 1 H , Q ∥ L q ∨ ∥ V s + 1 G , Q ∥ L q = O ( n Λ n ( z ) 7 ) , (137)
Here, as in the preceding parts, either ∥ Q ∥ ≤ 1 \lVert Q\rVert\le1 ∥ Q ∥ ≤ 1 or ∥ Q ∥ F ≤ n \lVert Q\rVert_F\le\sqrt{n} ∥ Q ∥ F ≤ n .
Proof of Lemma E.13
Our goal is to control each term in the expansion of χ 1 \chi_1 χ 1 -terms in Eqs. Eqs. (124) and (125) , and additionally provide the zero-sample endpoint errors. Throughout the proof, we shall assume either ∥ Q ∥ ≤ 1 \lVert Q\rVert\le1 ∥ Q ∥ ≤ 1 or ∥ Q ∥ F ≤ n \lVert Q\rVert_F\le\sqrt{n} ∥ Q ∥ F ≤ n , and suppress dependence on C x , ν , κ C_x,\nu,\kappa C x , ν , κ in big-Oh notations. We first bound the conditional trace fluctuation appearing in the first terms of Eqs. (124) and (125) .
Part I: The conditional trace fluctuation. To upper-bound the leading term in each expansion in Part II of the proof of Lemma E.6, namely in Eqs. (124) and (125) , we first require a concentration result for Tr R ~ k − 1 ( z , w j ) \operatorname{Tr}\widetilde{R}_{k-1}(z,w_j) Tr R k − 1 ( z , w j ) for j = k , k + 1 j=k,k+1 j = k , k + 1 . We would like to use Efron-Stein in Lemma A.1, and therefore for k ≥ 2 k\ge2 k ≥ 2 , we replace observation i < k i<k i < k and condition on the common remaining sample. This is equivalent to invoking Lemma E.12 with V s + 1 G , I V_{s+1}^{G,I} V s + 1 G , I . Applying Sherman-Morrison with common resolvent ( T k − 1 − x i x i ∗ − w j Σ − z I ) − 1 (T_{k-1}-x_i x_i^*-w_j\Sigma-zI)^{-1} ( T k − 1 − x i x i ∗ − w j Σ − z I ) − 1 allows us to use Lemma E.12 with κ \kappa κ replaced by 6 κ 6\kappa 6 κ , Eq. (97), with s = k − 2 s=k-2 s = k − 2 and j = k j=k j = k or k + 1 k+1 k + 1 since j − s ≤ 3 j-s\le3 j − s ≤ 3 , and implies
∥ Tr ( R ~ k − 1 − R ~ k − 1 ( i ) ) ∥ L q = O ( q Λ n ( z ) 11 n 3 / 2 ) . \left\lVert\operatorname{Tr}(\widetilde{R}_{k-1}-\widetilde{R}_{k-1}^{(i)})\right\rVert_{L^q}=O\left(\frac{q\Lambda_n(z)^{11}}{n^{3/2}}\right). Tr ( R k − 1 − R k − 1 ( i ) ) L q = O ( n 3/2 q Λ n ( z ) 11 ) .
We can then use Efron-Stein (cf. Lemma A.1) to the first k − 1 k-1 k − 1 observations and conclude that for 2 ≤ q ≤ 6 κ log n 2\le q\le6\kappa\log n 2 ≤ q ≤ 6 κ log n ,
∥ Tr R ~ k − 1 − E Tr R ~ k − 1 ∥ L q = O ( q ) × ( ∑ i = 1 k − 1 ∥ Tr ( R ~ k − 1 − R ~ k − 1 ( i ) ) ∥ L q 2 ) 1 / 2 = O ( q 3 / 2 Λ n ( z ) 11 n ) . (138) \left\lVert\operatorname{Tr}\widetilde{R}_{k-1}-\mathbb{E}\operatorname{Tr}\widetilde{R}_{k-1}\right\rVert_{L^q}
=O(\sqrt{q})\times\left(\sum_{i=1}^{k-1}\left\lVert\operatorname{Tr}(\widetilde{R}_{k-1}-\widetilde{R}_{k-1}^{(i)})\right\rVert_{L^q}^{2}\right)^{1/2}
=O\left(\frac{q^{3/2}\Lambda_n(z)^{11}}{n}\right). \tag*{(138)} Tr R k − 1 − E Tr R k − 1 L q = O ( q ) × ( i = 1 ∑ k − 1 Tr ( R k − 1 − R k − 1 ( i ) ) L q 2 ) 1/2 = O ( n q 3/2 Λ n ( z ) 11 ) . (138)
We point out this generalizes the L 4 L^4 L 4 bound already obtained in Part II of the proof of Lemma E.11, and for k = 1 k=1 k = 1 the trace is deterministic.
Part II: Bounding χ 1 H , Q \chi_1^{H,Q} χ 1 H , Q and χ 1 G , Q \chi_1^{G,Q} χ 1 G , Q . We begin by bounding the first terms in Eqs. (124) and (125) . By Cauchy-Schwarz and the conditional Jensen’s inequality, it follows that
∥ δ k ( Tr R ~ k − 1 − E Tr R ~ k − 1 ) E k − 1 V k H , Q ∥ L p n ≤ Λ n ( z ) ∥ Tr R ~ k − 1 − E Tr R ~ k − 1 ∥ L 2 p ∥ V k H , Q ∥ L 2 p n , \frac{\left\lVert\delta_k(\operatorname{Tr}\widetilde{R}_{k-1}-\mathbb{E}\operatorname{Tr}\widetilde{R}_{k-1})\mathbb{E}_{k-1}V_k^{H,Q}\right\rVert_{L^p}}{n}
\le\Lambda_n(z)\left\lVert\operatorname{Tr}\widetilde{R}_{k-1}-\mathbb{E}\operatorname{Tr}\widetilde{R}_{k-1}\right\rVert_{L^{2p}}\frac{\left\lVert V_k^{H,Q}\right\rVert_{L^{2p}}}{n}, n δ k ( Tr R k − 1 − E Tr R k − 1 ) E k − 1 V k H , Q L p ≤ Λ n ( z ) Tr R k − 1 − E Tr R k − 1 L 2 p n V k H , Q L 2 p ,
∥ δ k + 1 ( Tr R ~ k − 1 − E Tr R ~ k − 1 ) E k − 1 V k G , Q ∥ L p ≤ Λ n ( z ) ∥ Tr R ~ k − 1 − E Tr R ~ k − 1 ∥ L 2 p ∥ V k G , Q ∥ L 2 p . \left\lVert\delta_{k+1}(\operatorname{Tr}\widetilde{R}_{k-1}-\mathbb{E}\operatorname{Tr}\widetilde{R}_{k-1})\mathbb{E}_{k-1}V_k^{G,Q}\right\rVert_{L^p}
\le\Lambda_n(z)\left\lVert\operatorname{Tr}\widetilde{R}_{k-1}-\mathbb{E}\operatorname{Tr}\widetilde{R}_{k-1}\right\rVert_{L^{2p}}\left\lVert V_k^{G,Q}\right\rVert_{L^{2p}}. δ k + 1 ( Tr R k − 1 − E Tr R k − 1 ) E k − 1 V k G , Q L p ≤ Λ n ( z ) Tr R k − 1 − E Tr R k − 1 L 2 p V k G , Q L 2 p .
Here the resolvents are evaluated at ( z , w k ) (z,w_k) ( z , w k ) and ( z , w k + 1 ) (z,w_{k+1}) ( z , w k + 1 ) , respectively, and we use ∣ δ k ∣ ∨ ∣ δ k + 1 ∣ ≤ Λ n ( z ) |\delta_k|\vee|\delta_{k+1}|\le\Lambda_n(z) ∣ δ k ∣ ∨ ∣ δ k + 1 ∣ ≤ Λ n ( z ) . Applying the conditional trace bound in Eq. (138) from Part I and Eq. (137) from Part III of the proof of Lemma E.12 , both at order 2 p 2p 2 p , bounds each right-hand side by
O ( Λ n ( z ) p 3 / 2 Λ n ( z ) 11 n Λ n ( z ) 7 n ) = O ( p 3 / 2 Λ n ( z ) 19 n 2 ) . O\left(\Lambda_n(z)\frac{p^{3/2}\Lambda_n(z)^{11}}{n}\frac{\Lambda_n(z)^7}{n}\right)=O\left(\frac{p^{3/2}\Lambda_n(z)^{19}}{n^2}\right). O ( Λ n ( z ) n p 3/2 Λ n ( z ) 11 n Λ n ( z ) 7 ) = O ( n 2 p 3/2 Λ n ( z ) 19 ) .
For H H H , the uncentered ratio estimate in Part III of the proof of Lemma E.12 gives ∥ V k H , Q ∥ L 2 p / n = O ( Λ n ( z ) 5 / n ) \lVert V_k^{H,Q}\rVert_{L^{2p}}/n=O(\Lambda_n(z)^5/n) ∥ V k H , Q ∥ L 2 p / n = O ( Λ n ( z ) 5 / n ) . Thus the first H H H term has order O ( p 3 / 2 Λ n ( z ) 1 + 11 + 5 / n 2 ) O(p^{3/2}\Lambda_n(z)^{1+11+5}/n^2) O ( p 3/2 Λ n ( z ) 1 + 11 + 5 / n 2 ) . For the second terms in the same two expansions, recall U k = z k ∗ R ~ k − 1 z k U_k=z_k^*\widetilde{R}_{k-1}z_k U k = z k ∗ R k − 1 z k and we have similarly
∥ δ k Cov ( U k , V k H , Q ∣ F k − 1 ) ∥ L p n ≤ Λ n ( z ) ∥ U k − E k − 1 U k ∥ L 2 p ∥ V k H , Q − E k − 1 V k H , Q ∥ L 2 p n , \frac{\lVert\delta_k\operatorname{Cov}(U_k,V_k^{H,Q}\mid\mathcal{F}_{k-1})\rVert_{L^p}}{n}\le\Lambda_n(z)\lVert U_k-\mathbb{E}_{k-1}U_k\rVert_{L^{2p}}\frac{\lVert V_k^{H,Q}-\mathbb{E}_{k-1}V_k^{H,Q}\rVert_{L^{2p}}}{n}, n ∥ δ k Cov ( U k , V k H , Q ∣ F k − 1 ) ∥ L p ≤ Λ n ( z ) ∥ U k − E k − 1 U k ∥ L 2 p n ∥ V k H , Q − E k − 1 V k H , Q ∥ L 2 p ,
∥ δ k + 1 Cov ( U k , V k G , Q ∣ F k − 1 ) ∥ L p ≤ Λ n ( z ) ∥ U k − E k − 1 U k ∥ L 2 p ∥ V k G , Q − E k − 1 V k G , Q ∥ L 2 p . \lVert\delta_{k+1}\operatorname{Cov}(U_k,V_k^{G,Q}\mid\mathcal{F}_{k-1})\rVert_{L^p}\le\Lambda_n(z)\lVert U_k-\mathbb{E}_{k-1}U_k\rVert_{L^{2p}}\lVert V_k^{G,Q}-\mathbb{E}_{k-1}V_k^{G,Q}\rVert_{L^{2p}}. ∥ δ k + 1 Cov ( U k , V k G , Q ∣ F k − 1 ) ∥ L p ≤ Λ n ( z ) ∥ U k − E k − 1 U k ∥ L 2 p ∥ V k G , Q − E k − 1 V k G , Q ∥ L 2 p .
Here E k − 1 U k = Tr R ~ k − 1 \mathbb{E}_{k-1}U_k=\operatorname{Tr}\widetilde{R}_{k-1} E k − 1 U k = Tr R k − 1 exactly, and we can therefore apply Eq. (132) from Part I of the proof of Lemma E.12 to bound the first factors following Λ n ( z ) \Lambda_n(z) Λ n ( z ) above directly. By Eq. (135) , we have
∥ U k − E k − 1 U k ∥ L 2 p ≤ O ( p ) ⋅ ∥ R ~ k − 1 ∥ F , L 2 p = O ( p Λ n ( z ) 3 n ) . \lVert U_k-\mathbb{E}_{k-1}U_k\rVert_{L^{2p}}\le O(p)\cdot\lVert\widetilde{R}_{k-1}\rVert_{F,L^{2p}}=O\left(\frac{p\Lambda_n(z)^3}{\sqrt{n}}\right). ∥ U k − E k − 1 U k ∥ L 2 p ≤ O ( p ) ⋅ ∥ R k − 1 ∥ F , L 2 p = O ( n p Λ n ( z ) 3 ) .
For the last factor, apply Lemma E.12 with κ \kappa κ replaced by 2 κ 2\kappa 2 κ and moment order 2 p 2p 2 p , taking ( s , j ) = ( k − 1 , k + 1 ) (s,j)=(k-1,k+1) ( s , j ) = ( k − 1 , k + 1 ) for G G G and ( s , j ) = ( k − 1 , k ) (s,j)=(k-1,k) ( s , j ) = ( k − 1 , k ) for H H H . This yields the desired control, so we can bound each right-hand side by
O ( Λ n ( z ) p Λ n ( z ) 3 n p Λ n ( z ) 11 n 3 / 2 ) = O ( p 2 Λ n ( z ) 15 n 2 ) . O\left(\Lambda_n(z)\frac{p\Lambda_n(z)^3}{\sqrt{n}}\frac{p\Lambda_n(z)^{11}}{n^{3/2}}\right)=O\left(\frac{p^2\Lambda_n(z)^{15}}{n^2}\right). O ( Λ n ( z ) n p Λ n ( z ) 3 n 3/2 p Λ n ( z ) 11 ) = O ( n 2 p 2 Λ n ( z ) 15 ) .
For H H H , the centered ratio estimate in Part II of the proof of Lemma E.12 has power 9, giving O ( p 2 Λ n ( z ) 1 + 3 + 9 / n 2 ) O(p^2\Lambda_n(z)^{1+3+9}/n^2) O ( p 2 Λ n ( z ) 1 + 3 + 9 / n 2 ) for the second H H H term. It only remains to bound the third term in Eq. (124) . Applying Eq. (111) from Part II of the proof of Lemma E.4 , followed by Eq. (135) from Part II of the proof of Lemma E.12 , we obtain
∣ E V k G , I ∣ ≤ E ∣ V k G , I ∣ ≤ E ∥ R ~ k − 1 ( z , w k + 1 ) ∥ + E ∥ R ~ k ( z , w k + 1 ) ∥ = O ( Λ n ( z ) 3 n ) . |\mathbb{E}V_k^{G,I}|\le\mathbb{E}|V_k^{G,I}|\le\mathbb{E}\lVert\widetilde{R}_{k-1}(z,w_{k+1})\rVert+\mathbb{E}\lVert\widetilde{R}_k(z,w_{k+1})\rVert=O\left(\frac{\Lambda_n(z)^3}{n}\right). ∣ E V k G , I ∣ ≤ E ∣ V k G , I ∣ ≤ E ∥ R k − 1 ( z , w k + 1 )∥ + E ∥ R k ( z , w k + 1 )∥ = O ( n Λ n ( z ) 3 ) .
Further by the conditional Jensen’s inequality and Eq. (137) from Part III of the proof of Lemma E.12 , we bound the last term by
∥ δ k + 1 ⋅ E V k G , I ⋅ E k − 1 V k G , Q ∥ L p ≤ Λ n ( z ) ∣ E V k G , I ∣ ∥ V k G , Q ∥ L p = O ( Λ n ( z ) 11 n 2 ) . \lVert\delta_{k+1}\cdot\mathbb{E}V_k^{G,I}\cdot\mathbb{E}_{k-1}V_k^{G,Q}\rVert_{L^p}\le\Lambda_n(z)|\mathbb{E}V_k^{G,I}|\lVert V_k^{G,Q}\rVert_{L^p}=O\left(\frac{\Lambda_n(z)^{11}}{n^2}\right). ∥ δ k + 1 ⋅ E V k G , I ⋅ E k − 1 V k G , Q ∥ L p ≤ Λ n ( z ) ∣ E V k G , I ∣ ∥ V k G , Q ∥ L p = O ( n 2 Λ n ( z ) 11 ) .
Collecting all the bounds from the preceding displays and substituting them into Eqs. (124) and (125) enables us to complete the proof of Eq. (103a) by
∥ χ 1 G , Q ∥ L p = O ( p 3 / 2 Λ n ( z ) 19 + p 2 Λ n ( z ) 15 + Λ n ( z ) 11 n 2 ) , \lVert\chi_1^{G,Q}\rVert_{L^p}=O\left(\frac{p^{3/2}\Lambda_n(z)^{19}+p^2\Lambda_n(z)^{15}+\Lambda_n(z)^{11}}{n^2}\right), ∥ χ 1 G , Q ∥ L p = O ( n 2 p 3/2 Λ n ( z ) 19 + p 2 Λ n ( z ) 15 + Λ n ( z ) 11 ) ,
∥ χ 1 H , Q ∥ L p n = O ( p 3 / 2 Λ n ( z ) 17 + p 2 Λ n ( z ) 13 n 2 ) . \frac{\lVert\chi_1^{H,Q}\rVert_{L^p}}{n}=O\left(\frac{p^{3/2}\Lambda_n(z)^{17}+p^2\Lambda_n(z)^{13}}{n^2}\right). n ∥ χ 1 H , Q ∥ L p = O ( n 2 p 3/2 Λ n ( z ) 17 + p 2 Λ n ( z ) 13 ) .
Since p ≥ 2 p\ge2 p ≥ 2 and Λ n ( z ) ≥ 1 \Lambda_n(z)\ge1 Λ n ( z ) ≥ 1 , both bounds are O ( p 2 Λ n ( z ) 19 / n 2 ) O(p^2\Lambda_n(z)^{19}/n^2) O ( p 2 Λ n ( z ) 19 / n 2 ) .
Part III: The initial corrections. Finally, we control the endpoint corrections β 0 H , Q = H 0 Q ( z , w 0 ) − H 0 Q ( z , w 0 l i n ) \beta_0^{H,Q}=H_0^Q(z,w_0)-H_0^Q(z,w_0^{\mathrm{lin}}) β 0 H , Q = H 0 Q ( z , w 0 ) − H 0 Q ( z , w 0 lin ) and β 0 G , Q = G 0 Q ( z , w 1 ) − G 0 Q ( z , w 0 l i n ) \beta_0^{G,Q}=G_0^Q(z,w_1)-G_0^Q(z,w_0^{\mathrm{lin}}) β 0 G , Q = G 0 Q ( z , w 1 ) − G 0 Q ( z , w 0 lin ) . We apply the resolvent identity and identify
β 0 H , Q = ( w 0 − w 0 l i n ) ⋅ z Tr ( Q R 0 ( z , w 0 ) Σ R 0 ( z , w 0 l i n ) ) , \beta_0^{H,Q}=(w_0-w_0^{\mathrm{lin}})\cdot z\operatorname{Tr}\left(QR_0(z,w_0)\Sigma R_0(z,w_0^{\mathrm{lin}})\right), β 0 H , Q = ( w 0 − w 0 lin ) ⋅ z Tr ( Q R 0 ( z , w 0 ) Σ R 0 ( z , w 0 lin ) ) ,
β 0 G , Q = ( w 1 − w 0 l i n ) ⋅ Tr ( Q R ~ 0 ( z , w 1 ) R ~ 0 ( z , w 0 l i n ) ) . \beta_0^{G,Q}=(w_1-w_0^{\mathrm{lin}})\cdot\operatorname{Tr}\left(Q\widetilde{R}_0(z,w_1)\widetilde{R}_0(z,w_0^{\mathrm{lin}})\right). β 0 G , Q = ( w 1 − w 0 lin ) ⋅ Tr ( Q R 0 ( z , w 1 ) R 0 ( z , w 0 lin ) ) .
We bound the first path differences factors by applying Eq. (77) from the proof of Theorem 3, i.e., ∣ w 0 − w 0 l i n ∣ = O ( Λ n ( z ) 17 ) |w_0-w_0^{\mathrm{lin}}|=O(\Lambda_n(z)^{17}) ∣ w 0 − w 0 lin ∣ = O ( Λ n ( z ) 17 ) , and ∣ δ 1 ∣ ≤ Λ n ( z ) |\delta_1|\leq\Lambda_n(z) ∣ δ 1 ∣ ≤ Λ n ( z ) implying ∣ w 1 − w 0 l i n ∣ ≤ ∣ w 0 − w 0 l i n ∣ + ∣ δ 1 ∣ = O ( Λ n ( z ) 17 ) |w_1-w_0^{\mathrm{lin}}|\leq|w_0-w_0^{\mathrm{lin}}|+|\delta_1|=O(\Lambda_n(z)^{17}) ∣ w 1 − w 0 lin ∣ ≤ ∣ w 0 − w 0 lin ∣ + ∣ δ 1 ∣ = O ( Λ n ( z ) 17 ) .
To bound the next two tracial terms, we recall Eq. (136) from Part III of the proof of Lemma E.12 with argument ( z , w 0 ) (z,w_0) ( z , w 0 ) as follows
∥ z Tr ( Q R 0 Σ R 0 ) ∥ L p = O ( Λ n ( z ) 4 ) and ∥ Tr ( Q R ~ 0 2 ) ∥ L p = O ( Λ n ( z ) 6 / n ) . \left\|z\operatorname{Tr}(QR_0\Sigma R_0)\right\|_{L^p}=O(\Lambda_n(z)^4)
\qquad\text{and}\qquad
\left\|\operatorname{Tr}(Q\widetilde{R}_0^2)\right\|_{L^p}=O(\Lambda_n(z)^6/n). ∥ z Tr ( Q R 0 Σ R 0 ) ∥ L p = O ( Λ n ( z ) 4 ) and Tr ( Q R 0 2 ) L p = O ( Λ n ( z ) 6 / n ) .
It thus remains to replace R 0 ( z , w 0 ) R_0(z,w_0) R 0 ( z , w 0 ) with R 0 ( z , w 0 l i n ) R_0(z,w_0^{\mathrm{lin}}) R 0 ( z , w 0 lin ) and R ~ 0 ( z , w 0 ) 2 \widetilde{R}_0(z,w_0)^2 R 0 ( z , w 0 ) 2 with R ~ 0 ( z , w 1 ) R ~ 0 ( z , w 0 l i n ) \widetilde{R}_0(z,w_1)\widetilde{R}_0(z,w_0^{\mathrm{lin}}) R 0 ( z , w 1 ) R 0 ( z , w 0 lin ) in the above display. As in the zero-sample endpoint, T 0 = 0 T_0=0 T 0 = 0 , the resolvents in both traces are diagonal in the eigenbasis of Σ \Sigma Σ . On the eigenspace of σ i \sigma_i σ i , replacing one resolvent at w 0 w_0 w 0 by the one at w 0 l i n w_0^{\mathrm{lin}} w 0 lin multiplies its entry by ( z + w 0 σ i ) / ( z + w 0 l i n σ i ) (z+w_0\sigma_i)/(z+w_0^{\mathrm{lin}}\sigma_i) ( z + w 0 σ i ) / ( z + w 0 lin σ i ) . For the G G G trace we also replace the other factor at w 0 w_0 w 0 by the one at w 1 w_1 w 1 , which contributes ( z + w 0 σ i ) / ( z + w 1 σ i ) (z+w_0\sigma_i)/(z+w_1\sigma_i) ( z + w 0 σ i ) / ( z + w 1 σ i ) . It suffices to bound both ratios uniformly in i i i .
We start from the second ratio, using w 1 − w 0 = δ 1 w_1-w_0=\delta_1 w 1 − w 0 = δ 1 and applying Eq. (135) from the proof of Lemma E.12 at ( s , j ) = ( 0 , 1 ) (s,j)=(0,1) ( s , j ) = ( 0 , 1 ) . Since ∣ δ 1 ∣ ≤ Λ n ( z ) |\delta_1|\leq\Lambda_n(z) ∣ δ 1 ∣ ≤ Λ n ( z ) , we obtain
∣ z + w 0 σ i z + w 1 σ i ∣ = ∣ 1 − δ 1 σ i z + w 1 σ i ∣ ≤ 1 + ∣ δ 1 ∣ ∥ R ~ 0 ( z , w 1 ) ∥ = 1 + O ( Λ n ( z ) 4 n ) = O ( 1 ) . \left|\frac{z+w_0\sigma_i}{z+w_1\sigma_i}\right|
=\left|1-\frac{\delta_1\sigma_i}{z+w_1\sigma_i}\right|
\leq1+|\delta_1|\|\widetilde{R}_0(z,w_1)\|
=1+O\left(\frac{\Lambda_n(z)^4}{n}\right)
=O(1). z + w 1 σ i z + w 0 σ i = 1 − z + w 1 σ i δ 1 σ i ≤ 1 + ∣ δ 1 ∣∥ R 0 ( z , w 1 ) ∥ = 1 + O ( n Λ n ( z ) 4 ) = O ( 1 ) .
For the first ratio, recall the fixed-point equations comparison with Eq. (75) from Part IV-ii of the proof of Theorem 3, i.e., s n ( z ) = w 0 l i n / ( n z ) s_n(z)=w_0^{\mathrm{lin}}/(nz) s n ( z ) = w 0 lin / ( n z ) and s n ( z + ε n ( z ) ) = w 0 / ( n z ) s_n(z+\varepsilon_n(z))=w_0/(nz) s n ( z + ε n ( z )) = w 0 / ( n z ) with ∣ ε n ( z ) ∣ ≤ ∣ z − ∣ / 4 |\varepsilon_n(z)|\leq|z_-|/4 ∣ ε n ( z ) ∣ ≤ ∣ z − ∣/4 by Eq. (76) . Writing s n ( z ) = ∫ ( x − z ) − 1 μ ( d x ) s_n(z)=\int(x-z)^{-1}\mu(dx) s n ( z ) = ∫ ( x − z ) − 1 μ ( d x ) as in Lemma A.4, we obtain
∣ z + w 0 σ i z + w 0 l i n σ i − 1 ∣ = n σ i ∣ s n ( z + ε n ( z ) ) − s n ( z ) ∣ ∣ 1 + n σ i s n ( z ) ∣ ≤ ( i ) 4 ∣ ε n ( z ) ∣ n σ i ∫ ∣ x − z ∣ − 2 μ ( d x ) 3 ∣ 1 + n σ i s n ( z ) ∣ ≤ ( i i ) 4 ∣ ε n ( z ) ∣ 3 ∣ z − ∣ ≤ 1 3 . \left|\frac{z+w_0\sigma_i}{z+w_0^{\mathrm{lin}}\sigma_i}-1\right|
=\frac{n\sigma_i|s_n(z+\varepsilon_n(z))-s_n(z)|}{|1+n\sigma_i s_n(z)|}
\mathrel{\mathop{\leq}^{\mathrm{(i)}}}
\frac{4|\varepsilon_n(z)|\,n\sigma_i\int|x-z|^{-2}\mu(dx)}{3|1+n\sigma_i s_n(z)|}
\mathrel{\mathop{\leq}^{\mathrm{(ii)}}}
\frac{4|\varepsilon_n(z)|}{3|z_-|}
\leq\frac{1}{3}. z + w 0 lin σ i z + w 0 σ i − 1 = ∣1 + n σ i s n ( z ) ∣ n σ i ∣ s n ( z + ε n ( z )) − s n ( z ) ∣ ≤ ( i ) 3∣1 + n σ i s n ( z ) ∣ 4∣ ε n ( z ) ∣ n σ i ∫ ∣ x − z ∣ − 2 μ ( d x ) ≤ ( ii ) 3∣ z − ∣ 4∣ ε n ( z ) ∣ ≤ 3 1 .
In (i) we subtract the two Stieltjes transforms and use ∣ x − z − ε n ( z ) ∣ ≥ 3 ∣ x − z ∣ / 4 |x-z-\varepsilon_n(z)|\geq3|x-z|/4 ∣ x − z − ε n ( z ) ∣ ≥ 3∣ x − z ∣/4 . For (ii), taking the imaginary part of the Stieltjes representation yields Im s n ( z ) = η ∫ ∣ x − z ∣ − 2 μ ( d x ) \operatorname{Im}s_n(z)=\eta\int|x-z|^{-2}\mu(dx) Im s n ( z ) = η ∫ ∣ x − z ∣ − 2 μ ( d x ) . If t ≥ 0 t\geq0 t ≥ 0 , then ∣ z − ∣ = η |z_-|=\eta ∣ z − ∣ = η , so ∣ 1 + n σ i s n ( z ) ∣ ≥ n σ i Im s n ( z ) |1+n\sigma_i s_n(z)|\geq n\sigma_i\operatorname{Im}s_n(z) ∣1 + n σ i s n ( z ) ∣ ≥ n σ i Im s n ( z ) and we have the desired bound. If t < 0 t<0 t < 0 , then x − t ≥ − t x-t\geq-t x − t ≥ − t on the support of μ \mu μ , so Re s n ( z ) = ∫ ( x − t ) ∣ x − z ∣ − 2 μ ( d x ) ≥ ( − t ) ∫ ∣ x − z ∣ − 2 μ ( d x ) \operatorname{Re}s_n(z)=\int(x-t)|x-z|^{-2}\mu(dx)\geq(-t)\int|x-z|^{-2}\mu(dx) Re s n ( z ) = ∫ ( x − t ) ∣ x − z ∣ − 2 μ ( d x ) ≥ ( − t ) ∫ ∣ x − z ∣ − 2 μ ( d x ) . Combining the two parts and using ∣ z − ∣ = t 2 + η 2 |z_-|=\sqrt{t^2+\eta^2} ∣ z − ∣ = t 2 + η 2 implies (ii) as
∣ 1 + n σ i s n ( z ) ∣ ≥ n σ i ∣ s n ( z ) ∣ ≥ n σ i ∣ z − ∣ ∫ ∣ x − z ∣ − 2 μ ( d x ) . |1+n\sigma_i s_n(z)|\geq n\sigma_i|s_n(z)|\geq n\sigma_i|z_-|\int|x-z|^{-2}\mu(dx). ∣1 + n σ i s n ( z ) ∣ ≥ n σ i ∣ s n ( z ) ∣ ≥ n σ i ∣ z − ∣ ∫ ∣ x − z ∣ − 2 μ ( d x ) .
Multiplying Q Q Q by the bounded diagonal multipliers preserves both test norm bounds up to a constant. Taken together, the above displays complete the proof as
∣ β 0 H , Q ∣ n = O ( ∣ w 0 − w 0 l i n ∣ ⋅ sup i ≥ 1 ∣ z + w 0 σ i z + w 0 l i n σ i ∣ ⋅ Λ n ( z ) 4 n ) = O ( Λ n ( z ) 21 n ) , \frac{|\beta_0^{H,Q}|}{n}
=O\left(|w_0-w_0^{\mathrm{lin}}|\cdot\sup_{i\geq1}\left|\frac{z+w_0\sigma_i}{z+w_0^{\mathrm{lin}}\sigma_i}\right|\cdot\frac{\Lambda_n(z)^4}{n}\right)
=O\left(\frac{\Lambda_n(z)^{21}}{n}\right), n ∣ β 0 H , Q ∣ = O ( ∣ w 0 − w 0 lin ∣ ⋅ i ≥ 1 sup z + w 0 lin σ i z + w 0 σ i ⋅ n Λ n ( z ) 4 ) = O ( n Λ n ( z ) 21 ) ,
∣ β 0 G , Q ∣ = O ( ∣ w 1 − w 0 l i n ∣ ⋅ sup i ≥ 1 ∣ z + w 0 σ i z + w 1 σ i ∣ ⋅ sup i ≥ 1 ∣ z + w 0 σ i z + w 0 l i n σ i ∣ ⋅ Λ n ( z ) 6 n ) = O ( Λ n ( z ) 23 n ) . |\beta_0^{G,Q}|
=O\left(|w_1-w_0^{\mathrm{lin}}|\cdot\sup_{i\geq1}\left|\frac{z+w_0\sigma_i}{z+w_1\sigma_i}\right|\cdot\sup_{i\geq1}\left|\frac{z+w_0\sigma_i}{z+w_0^{\mathrm{lin}}\sigma_i}\right|\cdot\frac{\Lambda_n(z)^6}{n}\right)
=O\left(\frac{\Lambda_n(z)^{23}}{n}\right). ∣ β 0 G , Q ∣ = O ( ∣ w 1 − w 0 lin ∣ ⋅ i ≥ 1 sup z + w 1 σ i z + w 0 σ i ⋅ i ≥ 1 sup z + w 0 lin σ i z + w 0 σ i ⋅ n Λ n ( z ) 6 ) = O ( n Λ n ( z ) 23 ) .
Auxiliary proofs for Theorem 5
Proof of Lemma E.7
At the full sample endpoint, z R n ( z , 0 ) = T n R n ( z , 0 ) − I zR_n(z,0)=T_nR_n(z,0)-I z R n ( z , 0 ) = T n R n ( z , 0 ) − I . Therefore, subtraction of the same identity at z ′ z' z ′ and applying the resolvent identity implies the following:
z R n ( z , 0 ) − z ′ R n ( z ′ , 0 ) = ( z − z ′ ) R n ( z , 0 ) T n R n ( z ′ , 0 ) . zR_n(z,0)-z'R_n(z',0)=(z-z')R_n(z,0)T_nR_n(z',0). z R n ( z , 0 ) − z ′ R n ( z ′ , 0 ) = ( z − z ′ ) R n ( z , 0 ) T n R n ( z ′ , 0 ) .
We recover Eq. (66a) by taking the weighted trace.
At the zero sample endpoint, we make use of Eq. (30) to express z G 1 Q ( z , w 0 lin ( z ) ) zG_1^Q(z,w_0^{\operatorname{lin}}(z)) z G 1 Q ( z , w 0 lin ( z )) as the trace against Q Q Q of a diagonal operator in the eigenbasis of Σ \Sigma Σ . On the eigenspace for σ i \sigma_i σ i , it holds
ω ( z / n ) σ i σ i − ω ( z / n ) − ω ( z ′ / n ) σ i σ i − ω ( z ′ / n ) = ( ω ( z / n ) − ω ( z ′ / n ) ) σ i 2 ( σ i − ω ( z / n ) ) ( σ i − ω ( z ′ / n ) ) . \frac{\omega(z/n)\sigma_i}{\sigma_i-\omega(z/n)}-\frac{\omega(z'/n)\sigma_i}{\sigma_i-\omega(z'/n)}=\frac{(\omega(z/n)-\omega(z'/n))\sigma_i^2}{(\sigma_i-\omega(z/n))(\sigma_i-\omega(z'/n))}. σ i − ω ( z / n ) ω ( z / n ) σ i − σ i − ω ( z ′ / n ) ω ( z ′ / n ) σ i = ( σ i − ω ( z / n )) ( σ i − ω ( z ′ / n )) ( ω ( z / n ) − ω ( z ′ / n )) σ i 2 .
Dividing the display by z − z ′ z-z' z − z ′ , using R ~ 0 = ( ω ( z / n ) / z ) Σ ( Σ − ω ( z / n ) I ) − 1 \widetilde{R}_0=(\omega(z/n)/z)\Sigma(\Sigma-\omega(z/n)I)^{-1} R 0 = ( ω ( z / n ) / z ) Σ ( Σ − ω ( z / n ) I ) − 1 and that of R ~ 0 ′ \widetilde{R}_0' R 0 ′ at both spectral parameters and taking the trace against Q Q Q proves Eq. (66b). It follows by taking z → z ′ z\to z' z → z ′ to see that both endpoint identities extend to z = z ′ z=z' z = z ′ by differentiation.
Proof of Lemma E.8
Part I: Bounding the term χ 1 \chi_1 χ 1 . We write χ 1 \chi_1 χ 1 into the following two parts:
χ 1 = E Tr ( Q ( R ~ n R ~ n ′ − R ~ n − 1 R ~ n − 1 ′ ) ) + Tr ( Q R ~ n R ~ n ′ ) − E Tr ( Q R ~ n R ~ n ′ ) , \chi_1=\mathbb{E}\operatorname{Tr}\left(Q(\widetilde{R}_n\widetilde{R}_n'-\widetilde{R}_{n-1}\widetilde{R}_{n-1}')\right)+\operatorname{Tr}(Q\widetilde{R}_n\widetilde{R}_n')-\mathbb{E}\operatorname{Tr}(Q\widetilde{R}_n\widetilde{R}_n'), χ 1 = E Tr ( Q ( R n R n ′ − R n − 1 R n − 1 ′ ) ) + Tr ( Q R n R n ′ ) − E Tr ( Q R n R n ′ ) ,
where the first term is : = ξ 1 \mathrel{:=}\xi_1 := ξ 1 and the second term is : = ξ 2 \mathrel{:=}\xi_2 := ξ 2 .
where ξ 1 = E χ 1 \xi_1=\mathbb{E}\chi_1 ξ 1 = E χ 1 . Therefore, ξ 2 \xi_2 ξ 2 can be viewed as its fluctuation, and ξ 1 \xi_1 ξ 1 as its residual bias. We will bound those two terms separately. The residual bias ξ 1 \xi_1 ξ 1 can be bounded by a leave-one-out argument of x n x_n x n , and the fluctuation term ξ 2 \xi_2 ξ 2 can be bounded by reusing the analysis of ξ 1 \xi_1 ξ 1 applying the Efron-Stein inequality. We start with ξ 1 \xi_1 ξ 1 next.
Part I-i: Bounding χ 1 \chi_1 χ 1 ’s residual bias ξ 1 \xi_1 ξ 1 . Following the definition of U k U_k U k at the start of Part II of the proof of Lemma E.4, we write
U n : = z n ∗ R ~ n − 1 z n and U n ′ : = z n ∗ R ~ n − 1 ′ z n U_n:=z_n^*\widetilde{R}_{n-1}z_n
\qquad\text{and}\qquad
U_n':=z_n^*\widetilde{R}_{n-1}'z_n U n := z n ∗ R n − 1 z n and U n ′ := z n ∗ R n − 1 ′ z n
for the two terminal parameters ( z , 0 ) (z,0) ( z , 0 ) and ( z ′ , 0 ) (z',0) ( z ′ , 0 ) , respectively. By applying the Sherman-Morrison formula to both resolvents, we can write
Tr ( Q ( R ~ n R ~ n ′ − R ~ n − 1 R ~ n − 1 ′ ) ) = − z n ∗ R ~ n − 1 R ~ n − 1 ′ Q R ~ n − 1 z n 1 + U n − z n ∗ R ~ n − 1 ′ Q R ~ n − 1 R ~ n − 1 ′ z n 1 + U n ′ + ( z n ∗ R ~ n − 1 R ~ n − 1 ′ z n ) ( z n ∗ R ~ n − 1 ′ Q R ~ n − 1 z n ) ( 1 + U n ) ( 1 + U n ′ ) . (139) \begin{aligned}
\operatorname{Tr}\left(Q(\widetilde{R}_n\widetilde{R}_n'-\widetilde{R}_{n-1}\widetilde{R}_{n-1}')\right)
={}&-\frac{z_n^*\widetilde{R}_{n-1}\widetilde{R}_{n-1}'Q\widetilde{R}_{n-1}z_n}{1+U_n}
-\frac{z_n^*\widetilde{R}_{n-1}'Q\widetilde{R}_{n-1}\widetilde{R}_{n-1}'z_n}{1+U_n'}\\
&+\frac{(z_n^*\widetilde{R}_{n-1}\widetilde{R}_{n-1}'z_n)(z_n^*\widetilde{R}_{n-1}'Q\widetilde{R}_{n-1}z_n)}{(1+U_n)(1+U_n')}. \tag*{(139)}
\end{aligned} Tr ( Q ( R n R n ′ − R n − 1 R n − 1 ′ ) ) = − 1 + U n z n ∗ R n − 1 R n − 1 ′ Q R n − 1 z n − 1 + U n ′ z n ∗ R n − 1 ′ Q R n − 1 R n − 1 ′ z n + ( 1 + U n ) ( 1 + U n ′ ) ( z n ∗ R n − 1 R n − 1 ′ z n ) ( z n ∗ R n − 1 ′ Q R n − 1 z n ) . (139)
To bound ξ 1 = E χ 1 \xi_1=\mathbb{E}\chi_1 ξ 1 = E χ 1 , it suffices to bound the L p L^p L p norm of the right-hand side of Eq. (139) , since the absolute value of its expectation is at most its L p L^p L p norm by Hölder’s inequality. To bound each term, we start by bounding the reciprocals of the denominators. By Lemma E.1 .(i) with j = k = n j=k=n j = k = n , w n = 0 w_n=0 w n = 0 and x = x n x=x_n x = x n , since U n = x n ∗ R n − 1 ( z , 0 ) x n U_n=x_n^*R_{n-1}(z,0)x_n U n = x n ∗ R n − 1 ( z , 0 ) x n and the same identity holds at z ′ z' z ′ , we obtain
∣ 1 + U n ∣ − 1 ≤ ∣ z ∣ / ∣ z − ∣ ≤ Λ n and ∣ 1 + U n ′ ∣ − 1 ≤ ∣ z ′ ∣ / ∣ z − ′ ∣ ≤ Λ n ′ . |1+U_n|^{-1}\le|z|/|z_-|\le\Lambda_n
\qquad\text{and}\qquad
|1+U_n'|^{-1}\le|z'|/|z_-'|\le\Lambda_n'. ∣1 + U n ∣ − 1 ≤ ∣ z ∣/∣ z − ∣ ≤ Λ n and ∣1 + U n ′ ∣ − 1 ≤ ∣ z ′ ∣/∣ z − ′ ∣ ≤ Λ n ′ .
For the numerators, they are the same type of quadratic forms conditioned on F n − 1 \mathcal{F}_{n-1} F n − 1 , by writing A A A for the sandwiched matrices between z n ∗ z_n^* z n ∗ and z n z_n z n in any of the four quadratic forms in Eq. (139) . Indeed, this allows us to invoke Hanson-Wright (cf. Proposition 1). Integrating the conditional Hanson-Wright tail as in Part I of the proof of Lemma E.11, Eq. (117) , implies ∥ z n ∗ A z n − Tr A ∥ L p = O ( p ) ⋅ ∥ ∥ A ∥ F ∥ L p \lVert z_n^*Az_n-\operatorname{Tr} A\rVert_{L_p}=O(p)\cdot\lVert\lVert A\rVert_F\rVert_{L_p} ∥ z n ∗ A z n − Tr A ∥ L p = O ( p ) ⋅ ∥∥ A ∥ F ∥ L p for all of the four terms above. Finally, by the moment bounds at ( s , j ) = ( n − 1 , n ) (s,j)=(n-1,n) ( s , j ) = ( n − 1 , n ) (cf. Eq. (132) and the paragraph after Eq. (135) ), we have the following bounds, where the first line applies to the first two numerators in Eq. (139) , and the second to each quadratic-form factor of its third numerator.
∥ Tr A ∥ L p = O ( ( Λ n ∨ Λ n ′ ) 6 n 2 ) , ∥ z n ∗ A z n − Tr A ∥ L p = O ( p ( Λ n ∨ Λ n ′ ) 6 n 5 / 2 ) , \lVert\operatorname{Tr} A\rVert_{L_p}=O\left(\frac{(\Lambda_n\vee\Lambda_n')^6}{n^2}\right),\qquad\lVert z_n^*Az_n-\operatorname{Tr} A\rVert_{L_p}=O\left(\frac{p(\Lambda_n\vee\Lambda_n')^6}{n^{5/2}}\right), ∥ Tr A ∥ L p = O ( n 2 ( Λ n ∨ Λ n ′ ) 6 ) , ∥ z n ∗ A z n − Tr A ∥ L p = O ( n 5/2 p ( Λ n ∨ Λ n ′ ) 6 ) ,
∥ Tr A ∥ L 2 p = O ( ( Λ n ∨ Λ n ′ ) 4 n ) , ∥ z n ∗ A z n − Tr A ∥ L 2 p = O ( p ( Λ n ∨ Λ n ′ ) 4 n 3 / 2 ) . (140) \lVert\operatorname{Tr} A\rVert_{L_{2p}}=O\left(\frac{(\Lambda_n\vee\Lambda_n')^4}{n}\right),\qquad\lVert z_n^*Az_n-\operatorname{Tr} A\rVert_{L_{2p}}=O\left(\frac{p(\Lambda_n\vee\Lambda_n')^4}{n^{3/2}}\right). \tag*{(140)} ∥ Tr A ∥ L 2 p = O ( n ( Λ n ∨ Λ n ′ ) 4 ) , ∥ z n ∗ A z n − Tr A ∥ L 2 p = O ( n 3/2 p ( Λ n ∨ Λ n ′ ) 4 ) . (140)
We point out that these bounds hold for p = O ( log n ) p=O(\log n) p = O ( log n ) . Combining the displays above and substituting them into Eq. (139) with an additional application of Hölder, we conclude that
∥ Tr ( Q ( R ~ n R ~ n ′ − R ~ n − 1 R ~ n − 1 ′ ) ) ∥ L p = O ( ( Λ n + Λ n ′ ) ( Λ n ∨ Λ n ′ ) 6 + Λ n Λ n ′ ( Λ n ∨ Λ n ′ ) 8 n 2 ) = O ( ( Λ n ∨ Λ n ′ ) 10 n 2 ) . (141) \left\lVert\operatorname{Tr}\left(Q\left(\widetilde{R}_n\widetilde{R}_n'-\widetilde{R}_{n-1}\widetilde{R}_{n-1}'\right)\right)\right\rVert_{L_p}=O\left(\frac{(\Lambda_n+\Lambda_n')(\Lambda_n\vee\Lambda_n')^6+\Lambda_n\Lambda_n'(\Lambda_n\vee\Lambda_n')^8}{n^2}\right)=O\left(\frac{(\Lambda_n\vee\Lambda_n')^{10}}{n^2}\right). \tag*{(141)} Tr ( Q ( R n R n ′ − R n − 1 R n − 1 ′ ) ) L p = O ( n 2 ( Λ n + Λ n ′ ) ( Λ n ∨ Λ n ′ ) 6 + Λ n Λ n ′ ( Λ n ∨ Λ n ′ ) 8 ) = O ( n 2 ( Λ n ∨ Λ n ′ ) 10 ) . (141)
Taking the expectation proves ∣ ξ 1 ∣ = O ( ( Λ n ∨ Λ n ′ ) 10 / n 2 ) |\xi_1|=O((\Lambda_n\vee\Lambda_n')^{10}/n^2) ∣ ξ 1 ∣ = O (( Λ n ∨ Λ n ′ ) 10 / n 2 ) .
Part I-ii: Bounding χ 1 \chi_1 χ 1 ’s fluctuation ξ 2 \xi_2 ξ 2 . We will invoke Efron-Stein in Lemma A.1 , which boils down to bounding each ∥ ξ 2 − ξ 2 ( i ) ∥ L p \lVert\xi_2-\xi_2^{(i)}\rVert_{L_p} ∥ ξ 2 − ξ 2 ( i ) ∥ L p for 1 ≤ i ≤ n 1\le i\le n 1 ≤ i ≤ n , where ξ 2 ( i ) \xi_2^{(i)} ξ 2 ( i ) denotes ξ 2 \xi_2 ξ 2 ’s replacement of z i z_i z i by its independent copy, using the convention in Lemma A.1 . By exchangeability, we take i = n i=n i = n and let z ^ n \widehat{z}_n z n be a copy of z n z_n z n independent of the entire sample. Canceling the common expectation in ξ 2 − ξ 2 ( n ) \xi_2-\xi_2^{(n)} ξ 2 − ξ 2 ( n ) and subtracting the same mixed trace based on the first n − 1 n-1 n − 1 observations yields
ξ 2 − ξ 2 ( n ) = Tr ( Q ( R ~ n R ~ n ′ − R ~ n − 1 R ~ n − 1 ′ ) ) − Tr ( Q ( R ~ n ( n ) ( R ~ n ′ ) ( n ) − R ~ n − 1 R ~ n − 1 ′ ) ) . \xi_2-\xi_2^{(n)}=\operatorname{Tr}\left(Q\left(\widetilde{R}_n\widetilde{R}_n'-\widetilde{R}_{n-1}\widetilde{R}_{n-1}'\right)\right)-\operatorname{Tr}\left(Q\left(\widetilde{R}_n^{(n)}(\widetilde{R}_n')^{(n)}-\widetilde{R}_{n-1}\widetilde{R}_{n-1}'\right)\right). ξ 2 − ξ 2 ( n ) = Tr ( Q ( R n R n ′ − R n − 1 R n − 1 ′ ) ) − Tr ( Q ( R n ( n ) ( R n ′ ) ( n ) − R n − 1 R n − 1 ′ ) ) .
and this allows us to reuse the three terms analysis from Eq. (139) . Let U n ( n ) = z ^ n ∗ R ~ n − 1 z ^ n U_n^{(n)}=\widehat{z}_n^*\widetilde{R}_{n-1}\widehat{z}_n U n ( n ) = z n ∗ R n − 1 z n and ( U n ′ ) ( n ) = z ^ n ∗ R ~ n − 1 ′ z ^ n (U_n')^{(n)}=\widehat{z}_n^*\widetilde{R}_{n-1}'\widehat{z}_n ( U n ′ ) ( n ) = z n ∗ R n − 1 ′ z n . It boils down to controlling z n ∗ A z n / ( 1 + U n ) − z ^ n ∗ A z ^ n / ( 1 + U n ( n ) ) z_n^*Az_n/(1+U_n)-\widehat{z}_n^*A\widehat{z}_n/(1+U_n^{(n)}) z n ∗ A z n / ( 1 + U n ) − z n ∗ A z n / ( 1 + U n ( n ) ) (and the ones measured at z n ′ z_n' z n ′ ), for A A A the numerator matrices following the same bounds in Eq. (140) . Since A A A and G n = G n ( z , 0 ) G_n=G_n(z,0) G n = G n ( z , 0 ) are F n − 1 \mathcal{F}_{n-1} F n − 1 -measurable, both ratios have the same conditional law. Subtracting Tr A / ( 1 + G n ) \operatorname{Tr} A/(1+G_n) Tr A / ( 1 + G n ) from each and applying the triangle inequality yields
∥ z n ∗ A z n 1 + U n − z ^ n ∗ A z ^ n 1 + U n ( n ) ∥ L p ≤ 2 ∥ z n ∗ A z n 1 + U n − Tr A 1 + G n ∥ L p = 2 ∥ z n ∗ A z n − Tr A 1 + U n − Tr A ( U n − G n ) ( 1 + U n ) ( 1 + G n ) ∥ L p . \left\lVert\frac{z_n^*Az_n}{1+U_n}-\frac{\widehat{z}_n^*A\widehat{z}_n}{1+U_n^{(n)}}\right\rVert_{L_p}\le2\left\lVert\frac{z_n^*Az_n}{1+U_n}-\frac{\operatorname{Tr} A}{1+G_n}\right\rVert_{L_p}=2\left\lVert\frac{z_n^*Az_n-\operatorname{Tr} A}{1+U_n}-\frac{\operatorname{Tr} A(U_n-G_n)}{(1+U_n)(1+G_n)}\right\rVert_{L_p}. 1 + U n z n ∗ A z n − 1 + U n ( n ) z n ∗ A z n L p ≤ 2 1 + U n z n ∗ A z n − 1 + G n Tr A L p = 2 1 + U n z n ∗ A z n − Tr A − ( 1 + U n ) ( 1 + G n ) Tr A ( U n − G n ) L p .
See Eq. (129) in the proof of Lemma E.12 for a similar analysis already done, and use its denominator bound at ( s , j ) = ( n − 1 , n ) (s,j)=(n-1,n) ( s , j ) = ( n − 1 , n ) , which implies ∣ 1 + G n ∣ − 1 ≤ Λ n |1+G_n|^{-1}\le\Lambda_n ∣1 + G n ∣ − 1 ≤ Λ n . Combining with the same upper bound for ∣ 1 + U n ∣ − 1 ≤ Λ n |1+U_n|^{-1}\le\Lambda_n ∣1 + U n ∣ − 1 ≤ Λ n shown in Part I-i, and ∥ U n − G n ∥ L 2 p = O ( p Λ n 2 / n ) \lVert U_n-G_n\rVert_{L_{2p}}=O(p\Lambda_n^2/\sqrt{n}) ∥ U n − G n ∥ L 2 p = O ( p Λ n 2 / n ) (cf. Eq. (132) and the paragraph after Eq. (135) ), it follows that
∥ z n ∗ A z n 1 + U n − z ^ n ∗ A z ^ n 1 + U n ( n ) ∥ L p ≤ O ( ( Λ n ∨ Λ n ′ ) ∥ z n ∗ A z n − Tr A ∥ L p + p ( Λ n ∨ Λ n ′ ) 4 n ∥ Tr A ∥ L 2 p ) . \left\lVert\frac{z_n^*Az_n}{1+U_n}-\frac{\widehat{z}_n^*A\widehat{z}_n}{1+U_n^{(n)}}\right\rVert_{L_p}\le O\left((\Lambda_n\vee\Lambda_n')\lVert z_n^*Az_n-\operatorname{Tr} A\rVert_{L_p}+\frac{p(\Lambda_n\vee\Lambda_n')^4}{\sqrt{n}}\lVert\operatorname{Tr} A\rVert_{L_{2p}}\right). 1 + U n z n ∗ A z n − 1 + U n ( n ) z n ∗ A z n L p ≤ O ( ( Λ n ∨ Λ n ′ ) ∥ z n ∗ A z n − Tr A ∥ L p + n p ( Λ n ∨ Λ n ′ ) 4 ∥ Tr A ∥ L 2 p ) .
For the first two differences in Eq. (139) , use the first line of Eq. (140) at p p p for the centered numerator and at 2 p 2p 2 p for the trace. For the third difference, change one ratio at a time and apply Hölder to the two factors in L 2 p L^{2p} L 2 p . The second line of Eq. (140) bounds the centered numerator in L 2 p L^{2p} L 2 p as written and the trace in L 4 p L^{4p} L 4 p upon replacing p p p by 2 p 2p 2 p . Combining all these bounds, we conclude that
∥ ξ 2 − ξ ~ 2 ( n ) ∥ L p ≤ O ( 2 p ( Λ n ∨ Λ n ′ ) 10 n 5 / 2 + 2 p ( Λ n ∨ Λ n ′ ) 8 ( Λ n ∨ Λ n ′ ) 5 n 3 / 2 n ) = O ( p ( Λ n ∨ Λ n ′ ) 13 n 5 / 2 ) . \left\|\xi_2-\tilde{\xi}_2^{(n)}\right\|_{L^p} \le O\left(2^p\frac{(\Lambda_n\vee\Lambda'_n)^{10}}{n^{5/2}}+2^p\frac{(\Lambda_n\vee\Lambda'_n)^8(\Lambda_n\vee\Lambda'_n)^5}{n^{3/2}n}\right)=O\left(\frac{p(\Lambda_n\vee\Lambda'_n)^{13}}{n^{5/2}}\right). ξ 2 − ξ ~ 2 ( n ) L p ≤ O ( 2 p n 5/2 ( Λ n ∨ Λ n ′ ) 10 + 2 p n 3/2 n ( Λ n ∨ Λ n ′ ) 8 ( Λ n ∨ Λ n ′ ) 5 ) = O ( n 5/2 p ( Λ n ∨ Λ n ′ ) 13 ) .
We are now ready to substitute this bound for every coordinate into Lemma A.1 , Eq. (33) , and we have
∥ ξ 2 ∥ L p ≤ O ( p ⋅ ( ∑ i = 1 n ∥ ξ 2 − ξ ~ 2 ( i ) ∥ L p 2 ) 1 / 2 ) = O ( n p ⋅ p ( Λ n ∨ Λ n ′ ) 13 n 5 / 2 ) = O ( p 3 / 2 ( Λ n ∨ Λ n ′ ) 13 n 2 ) . \left\|\xi_2\right\|_{L^p}\le O\left(\sqrt{p}\cdot\left(\sum_{i=1}^n\left\|\xi_2-\tilde{\xi}_2^{(i)}\right\|_{L^p}^2\right)^{1/2}\right)=O\left(\sqrt{np}\cdot\frac{p(\Lambda_n\vee\Lambda'_n)^{13}}{n^{5/2}}\right)=O\left(\frac{p^{3/2}(\Lambda_n\vee\Lambda'_n)^{13}}{n^2}\right). ∥ ξ 2 ∥ L p ≤ O p ⋅ ( i = 1 ∑ n ξ 2 − ξ ~ 2 ( i ) L p 2 ) 1/2 = O ( n p ⋅ n 5/2 p ( Λ n ∨ Λ n ′ ) 13 ) = O ( n 2 p 3/2 ( Λ n ∨ Λ n ′ ) 13 ) .
Combining the bound above with Part I-i and using χ 1 = ξ 1 + ξ 2 \chi_1=\xi_1+\xi_2 χ 1 = ξ 1 + ξ 2 , we finish the proof of Part I.
Part II: Bounding the term χ 2 \chi_2 χ 2 . By Part III-i of the proof of Theorem 3, Eq. (69) , ∣ δ n δ n ′ ∣ − 1 = O ( ( Λ n ∨ Λ n ′ ) 2 ) \left|\delta_n\delta'_n\right|^{-1}=O((\Lambda_n\vee\Lambda'_n)^2) ∣ δ n δ n ′ ∣ − 1 = O (( Λ n ∨ Λ n ′ ) 2 ) . Thus to prove the asserted bound for ∣ χ 2 ∣ \left|\chi_2\right| ∣ χ 2 ∣ , it suffices to show ∣ δ n δ n ′ χ 2 ∣ = O ( ( Λ n ∨ Λ n ′ ) 14 / n 2 ) \left|\delta_n\delta'_n\chi_2\right|=O((\Lambda_n\vee\Lambda'_n)^{14}/n^2) ∣ δ n δ n ′ χ 2 ∣ = O (( Λ n ∨ Λ n ′ ) 14 / n 2 ) . Take A = R ~ n − 1 Q R ~ n − 1 ′ A=\widetilde{R}_{n-1}Q\widetilde{R}_{n-1}' A = R n − 1 Q R n − 1 ′ , so Tr A = Tr ( Q R ~ n − 1 R ~ n − 1 ′ ) \operatorname{Tr}A=\operatorname{Tr}(Q\widetilde{R}_{n-1}\widetilde{R}_{n-1}') Tr A = Tr ( Q R n − 1 R n − 1 ′ ) . Applying the terminal identity and then centering the numerator, we have
δ n δ n ′ χ 2 = ( i ) δ n δ n ′ E Tr A − E [ z n ∗ R ~ n ′ Q R ~ n z n ] = ( i i ) δ n δ n ′ E Tr A − E [ z n ∗ A z n ( 1 + U n ) ( 1 + U n ′ ) ] = − E Tr A ⋅ ( E [ 1 ( 1 + U n ) ( 1 + U n ′ ) ] − δ n δ n ′ ) − E [ ( z n ∗ A z n − E Tr A ) ( 1 ( 1 + U n ) ( 1 + U n ′ ) − δ n δ n ′ ) ] . (142) \begin{aligned}
\delta_n\delta'_n\chi_2 \mathrel{\overset{(i)}{=}} \delta_n\delta'_n\mathbb{E}\operatorname{Tr}A-\mathbb{E}\left[z_n^*\widetilde{R}_n'Q\widetilde{R}_nz_n\right]\mathrel{\overset{(ii)}{=}}\delta_n\delta'_n\mathbb{E}\operatorname{Tr}A-\mathbb{E}\left[\frac{z_n^*Az_n}{(1+U_n)(1+U'_n)}\right] \\
&=-\mathbb{E}\operatorname{Tr}A\cdot\left(\mathbb{E}\left[\frac{1}{(1+U_n)(1+U'_n)}\right]-\delta_n\delta'_n\right)-\mathbb{E}\left[(z_n^*Az_n-\mathbb{E}\operatorname{Tr}A)\left(\frac{1}{(1+U_n)(1+U'_n)}-\delta_n\delta'_n\right)\right]. \tag*{(142)}
\end{aligned} δ n δ n ′ χ 2 = ( i ) δ n δ n ′ E Tr A − E [ z n ∗ R n ′ Q R n z n ] = ( ii ) δ n δ n ′ E Tr A − E [ ( 1 + U n ) ( 1 + U n ′ ) z n ∗ A z n ] = − E Tr A ⋅ ( E [ ( 1 + U n ) ( 1 + U n ′ ) 1 ] − δ n δ n ′ ) − E [ ( z n ∗ A z n − E Tr A ) ( ( 1 + U n ) ( 1 + U n ′ ) 1 − δ n δ n ′ ) ] . (142)
In (i), apply Lemma E.7 , Eq. (66a), and use exchangeability in T n = ∑ i = 1 n x i x i ∗ T_n=\sum_{i=1}^n x_i x_i^* T n = ∑ i = 1 n x i x i ∗ . At z ′ = z z'=z z ′ = z , use the diagonal identity of the same lemma and the local resolvent bound to differentiate under the expectation. In (ii), we apply Sherman-Morrison to R ~ n z n \widetilde{R}_nz_n R n z n and z n ∗ R ~ n ′ z_n^*\widetilde{R}'_n z n ∗ R n ′ .
Part II-i: Bounding χ 2 \chi_2 χ 2 ’s bias ξ 3 \xi_3 ξ 3 . In ξ 3 \xi_3 ξ 3 from Eq. (142) , the factor ∣ E Tr A ∣ \left|\mathbb{E}\operatorname{Tr}A\right| ∣ E Tr A ∣ is O ( ( Λ n ∨ Λ n ′ ) 4 / n ) O((\Lambda_n\vee\Lambda'_n)^4/n) O (( Λ n ∨ Λ n ′ ) 4 / n ) by the second line of Eq. (140) . It suffices to bound the mean error of the reciprocal product. At p = 2 p=2 p = 2 , combining Part I-ii with Eq. (138) in Part I of the proof of Lemma E.13 at ( k , j ) = ( n , n ) (k,j)=(n,n) ( k , j ) = ( n , n ) yields ∥ U n − E U n ∥ L 2 = O ( Λ n 2 / n + Λ n 11 / n ) = O ( Λ n 3 / n ) \|U_n-\mathbb{E}U_n\|_{L^2}=O(\Lambda_n^2/\sqrt{n}+\Lambda_n^{11}/n)=O(\Lambda_n^3/\sqrt{n}) ∥ U n − E U n ∥ L 2 = O ( Λ n 2 / n + Λ n 11 / n ) = O ( Λ n 3 / n ) , since n = Ω ( Λ n 16 ) n=\Omega(\Lambda_n^{16}) n = Ω ( Λ n 16 ) . Applying the reciprocal expansion immediately before Eq. (113) in Part III of the proof of Lemma E.4 , centered at E U n = G n ( z , 0 ) \mathbb{E}U_n=G_n(z,0) E U n = G n ( z , 0 ) , with the denominator bounds of Lemma E.1 .(i)–(ii), Eqs. (43a) and (56) , yields ∥ ( 1 + U n ) − 1 − δ n ∥ L 2 = O ( Λ n 5 / n ) \|(1+U_n)^{-1}-\delta_n\|_{L^2}=O(\Lambda_n^5/\sqrt{n}) ∥ ( 1 + U n ) − 1 − δ n ∥ L 2 = O ( Λ n 5 / n ) and ∣ E ( 1 + U n ) − 1 − δ n ∣ = O ( Λ n 9 / n ) \left|\mathbb{E}(1+U_n)^{-1}-\delta_n\right|=O(\Lambda_n^9/n) E ( 1 + U n ) − 1 − δ n = O ( Λ n 9 / n ) , since the linear term has zero mean.
As the same estimates hold at z ′ z' z ′ , we can expand the product in Eq. (142) and apply the mean bounds to the two linear terms. By Cauchy-Schwarz to the third term below, we finally have
ξ 3 = E Tr A ⋅ E [ δ n ′ ( 1 1 + U n − δ n ) + δ n ( 1 1 + U n ′ − δ n ′ ) + ( 1 1 + U n − δ n ) ( 1 1 + U n ′ − δ n ′ ) ] = O ( ( Λ n ∨ Λ n ′ ) 4 n ⋅ Λ n ′ Λ n 9 + Λ n ( Λ n ′ ) 9 + Λ n 5 ( Λ n ′ ) 5 n ) = O ( ( Λ n ∨ Λ n ′ ) 14 n 2 ) . \begin{aligned}
\xi_3=\mathbb{E}\operatorname{Tr}A\cdot\mathbb{E}\left[\delta'_n\left(\frac{1}{1+U_n}-\delta_n\right)+\delta_n\left(\frac{1}{1+U'_n}-\delta'_n\right)+\left(\frac{1}{1+U_n}-\delta_n\right)\left(\frac{1}{1+U'_n}-\delta'_n\right)\right] \\
&=O\left(\frac{(\Lambda_n\vee\Lambda'_n)^4}{n}\cdot\frac{\Lambda'_n\Lambda_n^9+\Lambda_n(\Lambda'_n)^9+\Lambda_n^5(\Lambda'_n)^5}{n}\right)=O\left(\frac{(\Lambda_n\vee\Lambda'_n)^{14}}{n^2}\right).
\end{aligned} ξ 3 = E Tr A ⋅ E [ δ n ′ ( 1 + U n 1 − δ n ) + δ n ( 1 + U n ′ 1 − δ n ′ ) + ( 1 + U n 1 − δ n ) ( 1 + U n ′ 1 − δ n ′ ) ] = O ( n ( Λ n ∨ Λ n ′ ) 4 ⋅ n Λ n ′ Λ n 9 + Λ n ( Λ n ′ ) 9 + Λ n 5 ( Λ n ′ ) 5 ) = O ( n 2 ( Λ n ∨ Λ n ′ ) 14 ) .
Part II-ii: Bounding χ 2 \chi_2 χ 2 ’s covariance ξ 4 \xi_4 ξ 4 . We bound the two terms (the numerator and the denominator fluctuations) in ξ 4 \xi_4 ξ 4 separately. For the numerator, the second line of Eq. (140) at p = 2 p=2 p = 2 bounds ∥ z n ∗ A z n − Tr A ∥ L 2 \lVert z_n^* A z_n-\operatorname{Tr} A\rVert_{L^2} ∥ z n ∗ A z n − Tr A ∥ L 2 by O ( ( Λ n ∨ Λ n ′ ) 4 / n 3 / 2 ) O((\Lambda_n\vee\Lambda_n')^4/n^{3/2}) O (( Λ n ∨ Λ n ′ ) 4 / n 3/2 ) . To change the center to E Tr A \mathbb{E}\operatorname{Tr} A E Tr A , recall that Tr A = Tr ( Q R ~ n − 1 R ~ n − 1 ′ ) \operatorname{Tr} A=\operatorname{Tr}(Q\widetilde{R}_{n-1}\widetilde{R}_{n-1}^{\prime}) Tr A = Tr ( Q R n − 1 R n − 1 ′ ) . Inserting the full trace yields
Tr A − E Tr A = ( Tr A − Tr ( Q R ~ n R ~ n ′ ) ) + ( Tr ( Q R ~ n R ~ n ′ ) − E Tr A ) = ( Tr A − Tr ( Q R ~ n R ~ n ′ ) ) + χ 1 . \operatorname{Tr} A-\mathbb{E}\operatorname{Tr} A=(\operatorname{Tr} A-\operatorname{Tr}(Q\widetilde{R}_n\widetilde{R}_n^{\prime}))+(\operatorname{Tr}(Q\widetilde{R}_n\widetilde{R}_n^{\prime})-\mathbb{E}\operatorname{Tr} A)=(\operatorname{Tr} A-\operatorname{Tr}(Q\widetilde{R}_n\widetilde{R}_n^{\prime}))+\chi_1. Tr A − E Tr A = ( Tr A − Tr ( Q R n R n ′ )) + ( Tr ( Q R n R n ′ ) − E Tr A ) = ( Tr A − Tr ( Q R n R n ′ )) + χ 1 .
At p = 2 p=2 p = 2 , applying Eq. (141) to the first difference yields an L 2 L^2 L 2 bound of O ( ( Λ n ∨ Λ n ′ ) 10 / n 2 ) O((\Lambda_n\vee\Lambda_n')^{10}/n^2) O (( Λ n ∨ Λ n ′ ) 10 / n 2 ) . Combining this with ∥ χ 1 ∥ L 2 = O ( ( Λ n ∨ Λ n ′ ) 13 / n 2 ) \lVert\chi_1\rVert_{L^2}=O((\Lambda_n\vee\Lambda_n')^{13}/n^2) ∥ χ 1 ∥ L 2 = O (( Λ n ∨ Λ n ′ ) 13 / n 2 ) from Part I yields ∥ Tr A − E Tr A ∥ L 2 = O ( ( Λ n ∨ Λ n ′ ) 13 / n 2 ) \lVert\operatorname{Tr} A-\mathbb{E}\operatorname{Tr} A\rVert_{L^2}=O((\Lambda_n\vee\Lambda_n')^{13}/n^2) ∥ Tr A − E Tr A ∥ L 2 = O (( Λ n ∨ Λ n ′ ) 13 / n 2 ) . Adding this to the centered bound yields ∥ z n ∗ A z n − E Tr A ∥ L 2 = O ( ( Λ n ∨ Λ n ′ ) 5 / n 3 / 2 ) \lVert z_n^*Az_n-\mathbb{E}\operatorname{Tr} A\rVert_{L^2}=O((\Lambda_n\vee\Lambda_n')^5/n^{3/2}) ∥ z n ∗ A z n − E Tr A ∥ L 2 = O (( Λ n ∨ Λ n ′ ) 5 / n 3/2 ) , since n = Ω ( ( Λ n ∨ Λ n ′ ) 16 ) n=\Omega((\Lambda_n\vee\Lambda_n')^{16}) n = Ω (( Λ n ∨ Λ n ′ ) 16 ) . Indeed, Assumption 2 at both parameters implies ( Λ n ∨ Λ n ′ ) 8 = O ( n ) (\Lambda_n\vee\Lambda_n')^8=O(\sqrt{n}) ( Λ n ∨ Λ n ′ ) 8 = O ( n ) , so the term with exponent 13 contributes at most O ( ( Λ n ∨ Λ n ′ ) 5 / n 3 / 2 ) O((\Lambda_n\vee\Lambda_n')^5/n^{3/2}) O (( Λ n ∨ Λ n ′ ) 5 / n 3/2 ) . For the denominator term, using Part II-i and changing the two factors one at a time, we have
∥ 1 ( 1 + U n ) ( 1 + U n ′ ) − δ n δ n ′ ∥ L 2 = ∥ ( 1 1 + U n − δ n ) 1 1 + U n ′ + δ n ( 1 1 + U n ′ − δ n ′ ) ∥ L 2 ≤ Λ n ′ ∥ 1 1 + U n − δ n ∥ L 2 + ∣ δ n ∣ ∥ 1 1 + U n ′ − δ n ′ ∥ L 2 = O ( ( Λ n ∨ Λ n ′ ) 6 n ) . \begin{aligned}
\left\lVert\frac{1}{(1+U_n)(1+U_n')}-\delta_n\delta_n'\right\rVert_{L^2}
=
\left\lVert\left(\frac{1}{1+U_n}-\delta_n\right)\frac{1}{1+U_n'}+\delta_n\left(\frac{1}{1+U_n'}-\delta_n'\right)\right\rVert_{L^2} \\
&\leq\Lambda_n'\left\lVert\frac{1}{1+U_n}-\delta_n\right\rVert_{L^2}+|\delta_n|\left\lVert\frac{1}{1+U_n'}-\delta_n'\right\rVert_{L^2}
=O\left(\frac{(\Lambda_n\vee\Lambda_n')^6}{\sqrt{n}}\right).
\end{aligned} ( 1 + U n ) ( 1 + U n ′ ) 1 − δ n δ n ′ L 2 = ( 1 + U n 1 − δ n ) 1 + U n ′ 1 + δ n ( 1 + U n ′ 1 − δ n ′ ) L 2 ≤ Λ n ′ 1 + U n 1 − δ n L 2 + ∣ δ n ∣ 1 + U n ′ 1 − δ n ′ L 2 = O ( n ( Λ n ∨ Λ n ′ ) 6 ) .
Applying Cauchy–Schwarz in Eq. (142) therefore implies ∣ ξ 4 ∣ = O ( ( ( Λ n ∨ Λ n ′ ) 5 / n 3 / 2 ) ( ( Λ n ∨ Λ n ′ ) 6 / n ) ) = O ( ( Λ n ∨ Λ n ′ ) 11 / n 2 ) |\xi_4|=O(((\Lambda_n\vee\Lambda_n')^5/n^{3/2})((\Lambda_n\vee\Lambda_n')^6/\sqrt{n}))=O((\Lambda_n\vee\Lambda_n')^{11}/n^2) ∣ ξ 4 ∣ = O ((( Λ n ∨ Λ n ′ ) 5 / n 3/2 ) (( Λ n ∨ Λ n ′ ) 6 / n )) = O (( Λ n ∨ Λ n ′ ) 11 / n 2 ) . Combining the bounds for ξ 3 \xi_3 ξ 3 and ξ 4 \xi_4 ξ 4 in Eq. (142) gives ∣ δ n δ n ′ χ 2 ∣ = O ( ( Λ n ∨ Λ n ′ ) 14 / n 2 ) |\delta_n\delta_n'\chi_2|=O((\Lambda_n\vee\Lambda_n')^{14}/n^2) ∣ δ n δ n ′ χ 2 ∣ = O (( Λ n ∨ Λ n ′ ) 14 / n 2 ) . Applying Eq. (69) from Part III-i of the proof of Theorem 3 yields ∣ χ 2 ∣ = O ( ( Λ n ∨ Λ n ′ ) 16 / n 2 ) |\chi_2|=O((\Lambda_n\vee\Lambda_n')^{16}/n^2) ∣ χ 2 ∣ = O (( Λ n ∨ Λ n ′ ) 16 / n 2 ) .
Proof of Lemma E.9
Part I: Bounding the term χ 3 \chi_3 χ 3 . Using the definition of χ 2 \chi_2 χ 2 to substitute for the terminal divided difference in χ 3 \chi_3 χ 3 yields
χ 3 = ( 1 − δ n δ n ′ δ lin ( z ) δ lin ( z ′ ) ) ( E Tr ( Q R ~ n − 1 R ~ n − 1 ′ ) − χ 2 ) . \chi_3=\left(1-\frac{\delta_n\delta_n'}{\delta^{\operatorname{lin}}(z)\delta^{\operatorname{lin}}(z')}\right)\left(\mathbb{E}\operatorname{Tr}(Q\widetilde{R}_{n-1}\widetilde{R}_{n-1}')-\chi_2\right). χ 3 = ( 1 − δ lin ( z ) δ lin ( z ′ ) δ n δ n ′ ) ( E Tr ( Q R n − 1 R n − 1 ′ ) − χ 2 ) .
The second factor is O ( ( Λ n ∨ Λ n ′ ) 4 / n ) O((\Lambda_n\vee\Lambda_n')^4/n) O (( Λ n ∨ Λ n ′ ) 4 / n ) . Indeed, taking A = R ~ n − 1 ′ Q R ~ n − 1 A=\widetilde{R}_{n-1}'Q\widetilde{R}_{n-1} A = R n − 1 ′ Q R n − 1 in the second line of Eq. (140) in Part I-i of the proof of Lemma E.8 gives this bound for the expected trace. The bound ∣ χ 2 ∣ = O ( ( Λ n ∨ Λ n ′ ) 16 / n 2 ) |\chi_2|=O((\Lambda_n\vee\Lambda_n')^{16}/n^2) ∣ χ 2 ∣ = O (( Λ n ∨ Λ n ′ ) 16 / n 2 ) in that lemma is absorbed since n = Ω ( ( Λ n ∨ Λ n ′ ) 16 ) n=\Omega((\Lambda_n\vee\Lambda_n')^{16}) n = Ω (( Λ n ∨ Λ n ′ ) 16 ) .
For the first factor, Part IV-iii of the proof of Theorem 3 bounds ∣ δ n − δ lin ( z ) ∣ |\delta_n-\delta^{\operatorname{lin}}(z)| ∣ δ n − δ lin ( z ) ∣ by O ( Λ n 17 / n ) O(\Lambda_n^{17}/n) O ( Λ n 17 / n ) . By Lemma A.4, ∣ δ lin ( z ) ∣ = ∣ z s n ( z ) ∣ ≤ ∣ z ∣ / ∣ z − ∣ ≤ Λ n |\delta^{\operatorname{lin}}(z)|=|zs_n(z)|\leq|z|/|z_-|\leq\Lambda_n ∣ δ lin ( z ) ∣ = ∣ z s n ( z ) ∣ ≤ ∣ z ∣/∣ z − ∣ ≤ Λ n , and ∣ δ n ∣ ≤ Λ n |\delta_n|\leq\Lambda_n ∣ δ n ∣ ≤ Λ n by Lemma E.1.(ii), Eq. (43b). Applying Eqs. (69) and (78) in the proof of Theorem 3 at z z z and z ′ z' z ′ yields ∣ δ lin ( z ) δ lin ( z ′ ) ∣ − 1 = O ( ( Λ n ∨ Λ n ′ ) 2 ) |\delta^{\operatorname{lin}}(z)\delta^{\operatorname{lin}}(z')|^{-1}=O((\Lambda_n\vee\Lambda_n')^2) ∣ δ lin ( z ) δ lin ( z ′ ) ∣ − 1 = O (( Λ n ∨ Λ n ′ ) 2 ) . Changing one increment at a time then yields
∣ 1 − δ n δ n ′ δ lin ( z ) δ lin ( z ′ ) ∣ ≤ ∣ δ lin ( z ) − δ n ∣ ∣ δ lin ( z ′ ) ∣ + ∣ δ n ∣ ∣ δ lin ( z ′ ) − δ n ′ ∣ ∣ δ lin ( z ) δ lin ( z ′ ) ∣ = O ( ( Λ n ∨ Λ n ′ ) 20 n ) . \left|1-\frac{\delta_n\delta_n'}{\delta^{\operatorname{lin}}(z)\delta^{\operatorname{lin}}(z')}\right|
\leq
\frac{|\delta^{\operatorname{lin}}(z)-\delta_n||\delta^{\operatorname{lin}}(z')|+|\delta_n||\delta^{\operatorname{lin}}(z')-\delta_n'|}{|\delta^{\operatorname{lin}}(z)\delta^{\operatorname{lin}}(z')|}
=O\left(\frac{(\Lambda_n\vee\Lambda_n')^{20}}{n}\right). 1 − δ lin ( z ) δ lin ( z ′ ) δ n δ n ′ ≤ ∣ δ lin ( z ) δ lin ( z ′ ) ∣ ∣ δ lin ( z ) − δ n ∣∣ δ lin ( z ′ ) ∣ + ∣ δ n ∣∣ δ lin ( z ′ ) − δ n ′ ∣ = O ( n ( Λ n ∨ Λ n ′ ) 20 ) .
Multiplying the two factors yields ∣ χ 3 ∣ = O ( ( Λ n ∨ Λ n ′ ) 24 / n 2 ) |\chi_3|=O((\Lambda_n\vee\Lambda_n')^{24}/n^2) ∣ χ 3 ∣ = O (( Λ n ∨ Λ n ′ ) 24 / n 2 ) .
Part II: Bounding the term χ 4 \chi_4 χ 4 . For the final term, it suffices to bound the divided difference n δ lin ( z ) δ lin ( z ′ ) χ 4 n\delta^{\operatorname{lin}}(z)\delta^{\operatorname{lin}}(z')\chi_4 n δ lin ( z ) δ lin ( z ′ ) χ 4 by O ( ( Λ n ∨ Λ n ′ ) 24 / n ) O((\Lambda_n\vee\Lambda_n')^{24}/n) O (( Λ n ∨ Λ n ′ ) 24 / n ) , because applying Eqs. (69) and (78) in the proof of Theorem 3 at z z z and z ′ z' z ′ bounds ∣ n δ lin ( z ) δ lin ( z ′ ) ∣ − 1 |n\delta^{\operatorname{lin}}(z)\delta^{\operatorname{lin}}(z')|^{-1} ∣ n δ lin ( z ) δ lin ( z ′ ) ∣ − 1 by O ( ( Λ n ∨ Λ n ′ ) 2 / n ) O((\Lambda_n\vee\Lambda_n')^2/n) O (( Λ n ∨ Λ n ′ ) 2 / n ) , and the desired bound on χ 4 \chi_4 χ 4 then follows.
Let Γ \Gamma Γ be the positively oriented boundary of the union of the disks centered at z z z and z ′ z' z ′ with radii ∣ z − ∣ / 4 |z_-|/4 ∣ z − ∣/4 and ∣ z − ′ ∣ / 4 |z'_-|/4 ∣ z − ′ ∣/4 , respectively. We can apply Cauchy’s formula on this union to obtain
n δ l i n ( z ) δ l i n ( z ′ ) χ 4 = 1 2 π i ∮ Γ ζ [ G n + 1 Q ( ζ , 0 ) − G 1 Q ( ζ , w 0 l i n ( ζ ) ) ] ( ζ − z ) ( ζ − z ′ ) d ζ . n\delta^{\mathrm{lin}}(z)\delta^{\mathrm{lin}}(z')\chi_4 = \frac{1}{2\pi i}\oint_{\Gamma}\frac{\zeta\left[G^{Q}_{n+1}(\zeta,0)-G^{Q}_{1}(\zeta,w^{\mathrm{lin}}_0(\zeta))\right]}{(\zeta-z)(\zeta-z')}\,\mathrm{d}\zeta. n δ lin ( z ) δ lin ( z ′ ) χ 4 = 2 π i 1 ∮ Γ ( ζ − z ) ( ζ − z ′ ) ζ [ G n + 1 Q ( ζ , 0 ) − G 1 Q ( ζ , w 0 lin ( ζ )) ] d ζ .
This allows us to invoke Theorem 4 for the single resolvents. We bound the other terms first. Since ζ ∈ Γ \zeta\in\Gamma ζ ∈ Γ lies outside the open disk centered at z z z , we have ∣ ζ − z ∣ ≥ ∣ z − ∣ / 4 |\zeta-z|\geq|z_-|/4 ∣ ζ − z ∣ ≥ ∣ z − ∣/4 . By the triangle inequality for the distance to R ≥ 0 \mathbb{R}_{\geq0} R ≥ 0 ,
∣ ζ − ∣ ≤ ∣ ζ − z ∣ + ∣ z − ∣ ≤ 5 ∣ ζ − z ∣ . |\zeta_-|\leq|\zeta-z|+|z_-|\leq5|\zeta-z|. ∣ ζ − ∣ ≤ ∣ ζ − z ∣ + ∣ z − ∣ ≤ 5∣ ζ − z ∣.
The same argument at z ′ z' z ′ implies ∣ ζ − z ′ ∣ ≥ ∣ z − ′ ∣ / 4 |\zeta-z'|\geq|z'_-|/4 ∣ ζ − z ′ ∣ ≥ ∣ z − ′ ∣/4 and ∣ ζ − ∣ ≤ ∣ ζ − z ′ ∣ + ∣ z − ′ ∣ ≤ 5 ∣ ζ − z ′ ∣ |\zeta_-|\leq|\zeta-z'|+|z'_-|\leq5|\zeta-z'| ∣ ζ − ∣ ≤ ∣ ζ − z ′ ∣ + ∣ z − ′ ∣ ≤ 5∣ ζ − z ′ ∣ . On the boundary arc centered at z z z , the latter bound yields ∣ ζ − ∣ / ( ∣ ζ − z ∣ ∣ ζ − z ′ ∣ ) ≤ 5 / ∣ ζ − z ∣ |\zeta_-|/(|\zeta-z||\zeta-z'|)\leq5/|\zeta-z| ∣ ζ − ∣/ ( ∣ ζ − z ∣∣ ζ − z ′ ∣ ) ≤ 5/∣ ζ − z ∣ . Integrating along this arc gives at most 10 π 10\pi 10 π , since ∣ ζ − z ∣ |\zeta-z| ∣ ζ − z ∣ is its radius. Interchanging z z z and z ′ z' z ′ gives the same integral bound on the other arc, and adding the two contributions and dividing by 2 π 2\pi 2 π yields
∣ n δ l i n ( z ) δ l i n ( z ′ ) χ 4 ∣ ≤ 10 sup ζ ∈ Γ ∣ ζ ∣ ∣ ζ − ∣ ∣ G n + 1 Q ( ζ , 0 ) − G 1 Q ( ζ , w 0 l i n ( ζ ) ) ∣ = O ( ( Λ n ∨ Λ n ′ ) 24 n ) . |n\delta^{\mathrm{lin}}(z)\delta^{\mathrm{lin}}(z')\chi_4|\leq10\sup_{\zeta\in\Gamma}\frac{|\zeta|}{|\zeta_-|}\left|G^{Q}_{n+1}(\zeta,0)-G^{Q}_{1}(\zeta,w^{\mathrm{lin}}_0(\zeta))\right|=O\left(\frac{(\Lambda_n\vee\Lambda'_n)^{24}}{n}\right). ∣ n δ lin ( z ) δ lin ( z ′ ) χ 4 ∣ ≤ 10 ζ ∈ Γ sup ∣ ζ − ∣ ∣ ζ ∣ G n + 1 Q ( ζ , 0 ) − G 1 Q ( ζ , w 0 lin ( ζ )) = O ( n ( Λ n ∨ Λ n ′ ) 24 ) .
We apply Theorem 4 with p = 2 p=2 p = 2 and r = 0 r=0 r = 0 , followed by Jensen’s inequality in the last step. As in Part III of the proof of Theorem 4, on each circle Λ n ( ζ ) = O ( Λ n ∨ Λ n ′ ) \Lambda_n(\zeta)=O(\Lambda_n\vee\Lambda'_n) Λ n ( ζ ) = O ( Λ n ∨ Λ n ′ ) with the same minimizing cutoff m m m from Assumption 2 and distance exponent ν + 1 \nu+1 ν + 1 , using reflection with Q ∗ \mathbf{Q}^* Q ∗ for potential values of ζ \zeta ζ in the lower half-plane. The remaining factor satisfies ∣ ζ ∣ / ∣ ζ − ∣ ≤ Λ n ( ζ ) |\zeta|/|\zeta_-|\leq\Lambda_n(\zeta) ∣ ζ ∣/∣ ζ − ∣ ≤ Λ n ( ζ ) by Assumption 2. The proof is complete.
Supplementary details for the numerical experiments
We provide further details on the setups, formulas, and numerical methods in experiments. The corresponding code is available in the public repository.
Figure 1 . For CIFAR-10, we use a fixed ImageNet-pretrained feature extractor, center the features using the training-set mean, and scale them so that the training covariance has largest eigenvalue one. Each trial samples n n n distinct test images, and histogram heights use all n n n eigenvalues. The colors indicate pooled eigenvalue quantiles, and cropped insets preserve the full-distribution normalization. For the polynomial model, we generate d = max { 2 14 , 256 n } d=\max\{2^{14},256n\} d = max { 2 14 , 256 n } Gaussian coordinates and approximate the remaining contribution to X X ∗ / n XX^*/n X X ∗ / n by its expectation, τ 1 ( d ) I n / n \tau_1(d)I_n/n τ 1 ( d ) I n / n . This differs from the fixed covariance truncation in § 4 .
Figure 2 . We numerically solve for the boundary values of the self-consistent equation and recover the densities by Stieltjes inversion. We sum the population terms explicitly up to a cutoff, then separate the first-order tail and approximate the quadratic remainder by quadrature. The limiting densities use the parametrizations in Appendix § 10 .§ 2 .
Figure 4 . Each μ ω , j \mu_{\omega,j} μ ω , j is a probability measure, including its atom at zero. We plot μ ω \mu_\omega μ ω without renormalization, so that it has mass Tr ( Σ 2 ) / n \operatorname{Tr}(\Sigma^2)/n Tr ( Σ 2 ) / n and no atom at zero.
Figure 9 . We use n = 400 n = 400 n = 400 Gaussian observations under the power law and critical law with α = β = 0.2 \alpha= \beta= 0.2 α = β = 0.2 , truncated at d = 2 16 d = 2^{16} d = 2 16 . With a n = n σ n a_n = n\sigma_n a n = n σ n and b n = ∑ j = n + 1 ∞ σ j b_n = \sum_{j=n+1}^{\infty}\sigma_j b n = ∑ j = n + 1 ∞ σ j , we plot z = a n ζ z = a_n\zeta z = a n ζ and z = b n + a n ζ z = b_n + a_n\zeta z = b n + a n ζ , respectively. Each grid has 25 equally spaced real coordinates on the displayed interval and 17 geometrically spaced heights.
For backward, linear, and forward interpolation in Eqs. (20a)–(20c), respectively, the rows estimate these errors with p = 1 p = 1 p = 1 and p = 2 p = 2 p = 2 :
( ∥ G n ( z , 0 ) − G 1 ( z , w 0 ) ∥ L p , ∥ G n ( z , 0 ) − G 1 ( z , w 0 l i n ) ∥ L p , ∥ G n ( z , μ n ) − G 1 ( z , w 1 l i n ) ∥ L p ) 1 + ∣ G 1 ( z , w 0 l i n ) ∣ \frac{\left(\|G_n(z,0)-G_1(z,w_0)\|_{L_p},\ \|G_n(z,0)-G_1(z,w_0^{\mathrm{lin}})\|_{L_p},\ \|G_n(z,\mu_n)-G_1(z,w_1^{\mathrm{lin}})\|_{L_p}\right)}{1+\left|G_1(z,w_0^{\mathrm{lin}})\right|} 1 + G 1 ( z , w 0 lin ) ( ∥ G n ( z , 0 ) − G 1 ( z , w 0 ) ∥ L p , ∥ G n ( z , 0 ) − G 1 ( z , w 0 lin ) ∥ L p , ∥ G n ( z , μ n ) − G 1 ( z , w 1 lin ) ∥ L p )
We compute the mean absolute and root mean squared errors over 400 trials shared across methods and grid points, using 1,024 independent draws to approximate the backward expectations. Here G n G_n G n uses n − 1 n - 1 n − 1 observations, and μ n \mu_n μ n is the actual forward endpoint. Forward updates use fixed-point iteration with a root-solver fallback.
Additional experiments for ridge-type filters
We begin with computing the effective risk formulas when k = 2 k = 2 k = 2 . By Faà di Bruno’s formula [3 ] , also used by Cheng and Montanari [28 ] for their calculations, from Eq. (7) , we can calculate the derivatives below, where λ ⋆ = − ω ( − λ ) \lambda_{\star} = -\omega(-\lambda) λ ⋆ = − ω ( − λ ) and
λ = λ ⋆ ( 1 − 1 n Tr ( Σ ( Σ + λ ⋆ I ) − 1 ) ) and J n ( λ ) = 1 n Tr ( Σ 2 ( Σ + λ ⋆ I ) − 2 ) . \lambda= \lambda_{\star}\left(1-\frac{1}{n}\operatorname{Tr}\left(\boldsymbol{\Sigma}(\boldsymbol{\Sigma}+\lambda_{\star}I)^{-1}\right)\right)
\qquad\text{and}\qquad
J_n(\lambda)=\frac{1}{n}\operatorname{Tr}\left(\boldsymbol{\Sigma}^{2}(\boldsymbol{\Sigma}+\lambda_{\star}I)^{-2}\right). λ = λ ⋆ ( 1 − n 1 Tr ( Σ ( Σ + λ ⋆ I ) − 1 ) ) and J n ( λ ) = n 1 Tr ( Σ 2 ( Σ + λ ⋆ I ) − 2 ) .
It follows from that formula:
ω ′ ( − λ ) = 1 1 − J n ( λ ) , ω ′ ′ ( − λ ) = 2 Tr ( Σ 2 ( Σ + λ ⋆ I ) − 3 ) n ( 1 − J n ( λ ) ) 3 , \omega'(-\lambda)=\frac{1}{1-J_n(\lambda)},
\qquad
\omega''(-\lambda)=\frac{2\operatorname{Tr}\left(\boldsymbol{\Sigma}^{2}(\boldsymbol{\Sigma}+\lambda_{\star}I)^{-3}\right)}{n(1-J_n(\lambda))^{3}}, ω ′ ( − λ ) = 1 − J n ( λ ) 1 , ω ′′ ( − λ ) = n ( 1 − J n ( λ ) ) 3 2 Tr ( Σ 2 ( Σ + λ ⋆ I ) − 3 ) ,
ω ′ ′ ′ ( − λ ) = 6 Tr ( Σ 2 ( Σ + λ ⋆ I ) − 4 ) n ( 1 − J n ( λ ) ) 4 + 12 ( Tr ( Σ 2 ( Σ + λ ⋆ I ) − 3 ) ) 2 n 2 ( 1 − J n ( λ ) ) 5 . \omega'''(-\lambda)=\frac{6\operatorname{Tr}\left(\boldsymbol{\Sigma}^{2}(\boldsymbol{\Sigma}+\lambda_{\star}I)^{-4}\right)}{n(1-J_n(\lambda))^{4}}+\frac{12\left(\operatorname{Tr}\left(\boldsymbol{\Sigma}^{2}(\boldsymbol{\Sigma}+\lambda_{\star}I)^{-3}\right)\right)^{2}}{n^{2}(1-J_n(\lambda))^{5}}. ω ′′′ ( − λ ) = n ( 1 − J n ( λ ) ) 4 6 Tr ( Σ 2 ( Σ + λ ⋆ I ) − 4 ) + n 2 ( 1 − J n ( λ ) ) 5 12 ( Tr ( Σ 2 ( Σ + λ ⋆ I ) − 3 ) ) 2 .
Substituting into Proposition 4.(i), we have the explicit variance and squared bias for k = 2 k = 2 k = 2 :
V n [ ϕ 2 , λ ] = τ 2 1 − J n ( λ ) { J n ( λ ) + 2 λ Tr ( Σ 2 ( Σ + λ ⋆ I ) − 3 ) n ( 1 − J n ( λ ) ) 2 + λ 2 Tr ( Σ 2 ( Σ + λ ⋆ I ) − 4 ) n ( 1 − J n ( λ ) ) 3 + 2 λ 2 ( Tr ( Σ 2 ( Σ + λ ⋆ I ) − 3 ) ) 2 n 2 ( 1 − J n ( λ ) ) 4 } . V_n[\phi_{2,\lambda}]=\frac{\tau^{2}}{1-J_n(\lambda)}\left\{J_n(\lambda)+\frac{2\lambda\operatorname{Tr}\left(\boldsymbol{\Sigma}^{2}(\boldsymbol{\Sigma}+\lambda_{\star}I)^{-3}\right)}{n(1-J_n(\lambda))^{2}}+\frac{\lambda^{2}\operatorname{Tr}\left(\boldsymbol{\Sigma}^{2}(\boldsymbol{\Sigma}+\lambda_{\star}I)^{-4}\right)}{n(1-J_n(\lambda))^{3}}+\frac{2\lambda^{2}\left(\operatorname{Tr}\left(\boldsymbol{\Sigma}^{2}(\boldsymbol{\Sigma}+\lambda_{\star}I)^{-3}\right)\right)^{2}}{n^{2}(1-J_n(\lambda))^{4}}\right\}. V n [ ϕ 2 , λ ] = 1 − J n ( λ ) τ 2 { J n ( λ ) + n ( 1 − J n ( λ ) ) 2 2 λ Tr ( Σ 2 ( Σ + λ ⋆ I ) − 3 ) + n ( 1 − J n ( λ ) ) 3 λ 2 Tr ( Σ 2 ( Σ + λ ⋆ I ) − 4 ) + n 2 ( 1 − J n ( λ ) ) 4 2 λ 2 ( Tr ( Σ 2 ( Σ + λ ⋆ I ) − 3 ) ) 2 } .
B n [ ϕ 2 , λ ] = λ ⋆ 2 β ⋆ Σ ( Σ + λ ⋆ I ) − 2 β 1 − J n ( λ ) { 1 + 2 λ Tr ( Σ 2 ( Σ + λ ⋆ I ) − 3 ) n ( 1 − J n ( λ ) ) 2 + λ 2 Tr ( Σ 2 ( Σ + λ ⋆ I ) − 4 ) n ( 1 − J n ( λ ) ) 3 + 2 λ 2 ( Tr ( Σ 2 ( Σ + λ ⋆ I ) − 3 ) ) 2 n 2 ( 1 − J n ( λ ) ) 4 } − 2 λ λ ⋆ ( 1 − J n ( λ ) ) 2 { 1 + λ Tr ( Σ 2 ( Σ + λ ⋆ I ) − 3 ) n ( 1 − J n ( λ ) ) 2 } × β ⋆ Σ 2 ( Σ + λ ⋆ I ) − 3 β + λ 2 ( 1 − J n ( λ ) ) 3 β ⋆ Σ 3 ( Σ + λ ⋆ I ) − 4 β . \begin{aligned}
B_n[\phi_{2,\lambda}]={}&\frac{\lambda_{\star}^{2}\beta^{\star}\boldsymbol{\Sigma}(\boldsymbol{\Sigma}+\lambda_{\star}I)^{-2\beta}}{1-J_n(\lambda)}
\left\{1+\frac{2\lambda\operatorname{Tr}\left(\boldsymbol{\Sigma}^{2}(\boldsymbol{\Sigma}+\lambda_{\star}I)^{-3}\right)}{n(1-J_n(\lambda))^{2}}+\frac{\lambda^{2}\operatorname{Tr}\left(\boldsymbol{\Sigma}^{2}(\boldsymbol{\Sigma}+\lambda_{\star}I)^{-4}\right)}{n(1-J_n(\lambda))^{3}}\right.\\
&\left.+\frac{2\lambda^{2}\left(\operatorname{Tr}\left(\boldsymbol{\Sigma}^{2}(\boldsymbol{\Sigma}+\lambda_{\star}I)^{-3}\right)\right)^{2}}{n^{2}(1-J_n(\lambda))^{4}}\right\}
-\frac{2\lambda\lambda_{\star}}{(1-J_n(\lambda))^{2}}\left\{1+\frac{\lambda\operatorname{Tr}\left(\boldsymbol{\Sigma}^{2}(\boldsymbol{\Sigma}+\lambda_{\star}I)^{-3}\right)}{n(1-J_n(\lambda))^{2}}\right\}\\
&\times\beta^{\star}\boldsymbol{\Sigma}^{2}(\boldsymbol{\Sigma}+\lambda_{\star}I)^{-3}\beta+\frac{\lambda^{2}}{(1-J_n(\lambda))^{3}}\beta^{\star}\boldsymbol{\Sigma}^{3}(\boldsymbol{\Sigma}+\lambda_{\star}I)^{-4}\beta.
\end{aligned} B n [ ϕ 2 , λ ] = 1 − J n ( λ ) λ ⋆ 2 β ⋆ Σ ( Σ + λ ⋆ I ) − 2 β { 1 + n ( 1 − J n ( λ ) ) 2 2 λ Tr ( Σ 2 ( Σ + λ ⋆ I ) − 3 ) + n ( 1 − J n ( λ ) ) 3 λ 2 Tr ( Σ 2 ( Σ + λ ⋆ I ) − 4 ) + n 2 ( 1 − J n ( λ ) ) 4 2 λ 2 ( Tr ( Σ 2 ( Σ + λ ⋆ I ) − 3 ) ) 2 } − ( 1 − J n ( λ ) ) 2 2 λ λ ⋆ { 1 + n ( 1 − J n ( λ ) ) 2 λ Tr ( Σ 2 ( Σ + λ ⋆ I ) − 3 ) } × β ⋆ Σ 2 ( Σ + λ ⋆ I ) − 3 β + ( 1 − J n ( λ ) ) 3 λ 2 β ⋆ Σ 3 ( Σ + λ ⋆ I ) − 4 β .
Figures 11a to 11c repeat Figures 7a to 7c with q = 24 q = 24 q = 24 . We keep all other settings and plotting conventions.
Figure 11a. Ridge-type filters with q = 24 q = 24 q = 24 and k = 1 k = 1 k = 1 . Rows show V n V_n V n , B n B_n B n , and R n R_n R n .
Figure 11b. Ridge-type filters with q = 24 q = 24 q = 24 and k = 1 k = 1 k = 1 , normalized by the respective ridgeless baselines.
Figure 11c. Iterated ridge with q = 24 q=24 q = 24 and k = 2 k=2 k = 2 , normalized by the respective ridgeless baselines.
Additional experiments for gradient flow
We explain our numerical approach to find the t o p t t_{\mathrm{opt}} t opt curve first. We evaluate the risk in Proposition 5 and its derivative using the formulas in § 3.2 . We locate sign changes of the risk derivative on a logarithmic grid in t ~ = t τ 1 ( n ) / n \tilde{t}=t\tau_1(n)/n t ~ = t τ 1 ( n ) / n , refine each detected root by Brent’s method in logarithmic time, and compare the risks at these stationary points with those at t = 0 t=0 t = 0 and the ridgeless limit t = ∞ t=\infty t = ∞ . The smallest value among these candidates determines the plotted optimum. For Figure 8 , we do not assume a unique stationary point, and the search uses 151 logarithmically spaced times in [ 10 − 12 , 10 3 ] [10^{-12},10^3] [ 1 0 − 12 , 1 0 3 ] , separately from the 91 displayed times in [ 0 , 4 ] [0,4] [ 0 , 4 ] .
Next, Figure 12 supplements these results with theoretical risk surfaces for q ∈ { 28 , 32 , 36 , 40 } q\in\{28,32,36,40\} q ∈ { 28 , 32 , 36 , 40 } , using 161 search times in [ 10 − 10 , 10 4 ] [10^{-10},10^4] [ 1 0 − 10 , 1 0 4 ] and otherwise the same settings. Relative improvements over ridgeless of at most 10 − 8 10^{-8} 1 0 − 8 are treated as numerical ties. Among these risk surfaces, the critical law with β = 0.2 \beta=0.2 β = 0.2 and q = 32 q=32 q = 32 illustrates a jump between two finite stopping times, whereas the power law with α = 0.2 \alpha=0.2 α = 0.2 exhibits an apparent ridgeless-to-finite transition. It remains intriguing to investigate these complicated behaviors further.
Figure 12. Gradient-flow theoretical risk surfaces relative to ridgeless for q = 28 , 32 , 36 , 40 q = 28, 32, 36, 40 q = 28 , 32 , 36 , 40 (top to bottom), with columns as in Figure 8 . Curves and points show predicted finite stopping optima. Triangles and dashed connectors indicate jumps between sampled minimizing branches.