betaFeedback

WorksMathematics

Dimension-free spectral laws and the risk of spectral filters

Abstract

Spectral properties of the sample covariance are key to modern machine learning with high-dimensional data. Random matrix theory (RMT) supplies powerful tools; however, most classical results assume proportional asymptotics, where d/nd/n tends to a finite limit. In functional data analysis and kernel methods, a fixed infinite-dimensional Hilbert space is often the more natural setting for the observations, beyond the scope of proportional asymptotics. This calls for a dimension-free spectral theory for i.i.d. observations x1,…,xn\boldsymbol{x}_1, \dots, \boldsymbol{x}_n in a Hilbert space H\mathcal{H} with trace-class population covariance ΣΣ. Building on the dimension-free ridge regression analysis of Cheng and Montanari (2024), we establish two deterministic equivalents with explicit nonasymptotic error bounds. The first approximates the empirical spectral law μ^n\widehatμ_n of the sample Gram matrix by a deterministic law μnμ_n. The second characterizes the conditional prediction risk of analytic spectral filters in linear regression. Previous sharp approximations to prediction risk in this setting have focused on ridge regression and closely-related problems. Our analysis covers a broader class of filters at the bulk scale, with an explicit user-friendly spectral stability condition to check. We apply our theory to power and critical spectral laws, and we recover the proportional regime. For prediction, we study (negative) ridge estimators, iterated ridge, and gradient flow. Mathematically, building on the idea of stochastic interpolation, we construct a deterministic backward path to interpolate the spectral parameter. The resulting bounds on resolvent and mixed-resolvent traces give mesoscopic approximations to the spectral law within the bulk and, by Cauchy's formula, deterministic equivalents for the prediction variance and squared bias of analytic spectral filters.

Introduction

The central problem in statistical learning is to infer structure from an i.i.d. sample x1,…,xnx_1,\ldots,x_n, with or without paired observations y1,…,yny_1,\ldots,y_n. The accompanying central challenge of high-dimensionality in modern machine learning arises from increasingly complex datasets (e.g., ImageNet [34], LAION-5B [70], and FineWeb [66]) and model architectures (e.g., diffusion [51, 67] and LLMs [23, 46]), going beyond the classical understanding of large-sample asymptotics [77] and sparsity [79]. Random matrix theory (RMT) has become popular in theoretical machine learning in those regimes where learning is possible but challenging. Not only useful for modeling frontier learning models such as kernel methods, random features, and neural networks [57, 63, 33, 64], it is also successful in understanding many empirical phenomena in machine learning, such as double-descent, benign-overfitting, memorization, and scaling laws [12, 61, 29, 27, 65, 4].

Existing tools in RMT primarily focus on the proportional setup when n/dn/d converges to a constant, in which xi∈Rdx_i \in\mathbb{R}^d has population covariance Σ\boldsymbol{\Sigma} and the sample covariance $\widehat{\boldsymbol{\Sigma}}=\boldsymbol{X}^{*}\boldsymbol{X}/n does not concentrate [60]. This setup, including its extension to polynomial scalings when n/dℓn/d^{\ell} converges [18, 59], still allows the input dimension to grow. In functional data analysis and in regression with a fixed kernel, it is natural to increase nn while keeping the underlying Hilbert space H\mathcal{H} and its covariance Σ\boldsymbol{\Sigma} fixed, even when H\mathcal{H} is infinite-dimensional. Empirically, an infinite-dimensional Σ\boldsymbol{\Sigma} whose population spectrum σ1≥σ2≥⋯\sigma_1 \ge\sigma_2 \ge\cdots is trace-class (Tr⁡Σ<∞\operatorname{Tr} \boldsymbol{\Sigma} < \infty), e.g., Zipf’s law with polynomial decay, σi=i−1−α\sigma_i = i^{-1-\alpha} for some α>0\alpha> 0, is often a better model for real-world data, as shown in Figure 1. We observe that the sample spectrum from a fixed Hilbert space with growing size nn is a better proxy for the sample spectrum of CIFAR-10 features.

Figure 1. Spectra of XX∗/nXX^*/n pooled over 2,000 trials, of (i) proportional isotropic Gaussian features with n=4dn = 4d, (ii) Zipf’s law with i.i.d. Gaussian coordinates, σi=i−1.2\sigma_i = i^{-1.2} and (iii) CIFAR-10 ResNet-50 second-to-last layer features (d=2,048d = 2,048). The zoomed in view show rescaled histograms. For CIFAR-10, we estimate σ^n\widehat{\sigma}_n from the separate training set.

The pioneering work of Cheng and Montanari [28] addressed this setting for the prediction risk of ridge regression problems, with later works studying closely-related problems [63, 33]. However, a more applicable and complete dimension-free spectral theory at the bulk scale, beyond ridge regression, is still missing. In this work, we develop nonasymptotic and deterministic approximations for (i) spectral parameters and laws and (ii) the prediction risk of a general class of spectral filters. Given i.i.d. x1,…,xn∈Hx_1,\ldots,x_n \in\mathcal{H}, we study the two following problems:

  1. Unsupervised learning: the empirical spectrum. How can we characterize the (top) empirical spectrum of the sample covariance Σ^\widehat{\boldsymbol{\Sigma}}?

Equivalently, we consider the Gram matrix XX∗/nXX^*/n. Let σ^1≥⋯≥σ^n≥0\widehat{\sigma}_1 \ge\cdots\ge\widehat{\sigma}_n \ge0 be the sample eigenvalues of XX∗/nXX^*/n. We seek a sharp characterization of the unnormalized empirical spectral distribution

μ^n:=1n∑j=1nδnσ^j(1)\widehat{\mu}_n := \frac{1}{n}\sum_{j=1}^{n}\delta_{n\widehat{\sigma}_j} \tag*{(1)}

at the bulk scale, i.e., on any interval E⊂RE \subset\mathbb{R} such μ^n(E)∈(c,1−c)\widehat{\mu}_n(E) \in(c,1-c) for some c>0c > 0.

  1. Supervised learning: risk of spectral filters. How can we characterize the prediction error of a spectral filter ϕ\phi?

We consider a linear model y=Xβ+ε\boldsymbol{y} = \boldsymbol{X}\boldsymbol{\beta} + \boldsymbol{\varepsilon} with noise E[ε∣X]=0\mathbb{E}[\boldsymbol{\varepsilon} \mid\boldsymbol{X}] = 0. The spectral filter Φ\Phi multiplies each sample eigendirection at eigenvalue xx by Φ(x)\Phi(x). Let the residual function be ψ(x)=1−xΦ(x)\psi(x) = 1 - x\Phi(x); this yields the estimator β^Φ=Φ(Σ^)X∗y/n\widehat{\boldsymbol{\beta}}_{\Phi} = \Phi(\widehat{\boldsymbol{\Sigma}})\boldsymbol{X}^{*}\boldsymbol{y}/n, which covers a broad class of spectral algorithms, e.g., ψ(x)=λ/(x+λ)\psi(x) = \lambda/(x + \lambda) for ridge, ψ(x)=(λ/(x+λ))k\psi(x) = (\lambda/(x + \lambda))^{k} for iterated ridge, ψ(x)=e−tx\psi(x) = e^{-tx} for gradient flow, and ψ(x)=1{x<λ}\psi(x) = \mathbb{1}\{x < \lambda\} for principal component regression (PCR). We seek sharp characterizations for the conditional generalization error

RX[Φ]:=E[∣⟨xnew,β^Φ−β⟩∣2∣X]=VX[Φ]+BX[Φ].(2)\mathcal{R}_{X}[\Phi] := \mathbb{E}[|\langle\boldsymbol{x}_{\mathrm{new}},\widehat{\boldsymbol{\beta}}_{\Phi} - \boldsymbol{\beta}\rangle|^{2} \mid\boldsymbol{X}] = \mathcal{V}_{X}[\Phi] + \mathcal{B}_{X}[\Phi]. \tag*{(2)}

Here xnew\boldsymbol{x}_{\mathrm{new}} denotes a test feature with the same law as x1\boldsymbol{x}_{1}, drawn independently of the training data. The expectation averages over the test feature and training noise conditional on X\boldsymbol{X}. The term VX[Φ]=τ2Tr⁡(ΣΣ^Φ(Σ^)2)/n\mathcal{V}_{X}[\Phi] = \tau^{2}\operatorname{Tr}(\boldsymbol{\Sigma}\widehat{\boldsymbol{\Sigma}}\Phi(\widehat{\boldsymbol{\Sigma}})^{2})/n is the conditional variance when E[εε∗∣X]=τ2In\mathbb{E}[\boldsymbol{\varepsilon}\boldsymbol{\varepsilon}^{*} \mid\boldsymbol{X}] = \tau^{2}I_{n}, and BX[Φ]=∥Σ1/2ψ(Σ^)β∥2\mathcal{B}_{X}[\Phi] = \|\boldsymbol{\Sigma}^{1/2}\psi(\widehat{\boldsymbol{\Sigma}})\boldsymbol{\beta}\|^{2} is the squared bias.

Our contributions

We give our solutions to the two problems, and summarize our contributions in the paper.

  1. Deterministic equivalents to the empirical spectrum. In § 2.2, we track the empirical spectral law μ^n\widehat{\mu}_{n} and its Stieltjes transform s^n\widehat{s}_{n} by their deterministic counterparts:

(sn(Σ),μn(Σ))⟷deterministic equivalentclosely approximates(s^n(X),μ^n(X)).(s_{n}(\boldsymbol{\Sigma}), \mu_{n}(\boldsymbol{\Sigma})) \underset{\text{closely approximates}}{\overset{\text{deterministic equivalent}}{\longleftrightarrow}} (\widehat{s}_{n}(\boldsymbol{X}), \widehat{\mu}_{n}(\boldsymbol{X})).

At fixed height on the bulk scale, the Stieltjes transform bound captures the optimal n−1n^{-1} power for averaged fluctuations (cf. [54]).

  1. Deterministic equivalents for prediction risk. In § 2.3, for a given analytic spectral filter, at fixed signal and noise level, we obtain

(Vn[Φ;Σ],Bn[Φ;Σ])⟷deterministic equivalentclosely approximates(VX[Φ],BX[Φ]).(\mathcal{V}_{n}[\Phi;\boldsymbol{\Sigma}],\mathcal{B}_{n}[\Phi;\boldsymbol{\Sigma}]) \underset{\text{closely approximates}}{\overset{\text{deterministic equivalent}}{\longleftrightarrow}} (\mathcal{V}_{X}[\Phi],\mathcal{B}_{X}[\Phi]).

Our bounds capture the optimal corresponding averaged and local fluctuation rate, n−1n^{-1} and n−1/2n^{-1/2}, respectively.

  1. Technical innovation: the backward interpolation path. In § 5, we present our technical core, where we achieve the preceding goals by approximating weighted traces of resolvents and mixed resolvents. For a deterministic weight Q\boldsymbol{Q}, we have

Tr⁡(QΣ(Σ−ω(z/n)I)−1)⟷deterministic equivalentclosely approximatesTr⁡(QX∗(XX∗−zIn)−1X).\operatorname{Tr}\left(\boldsymbol{Q}\boldsymbol{\Sigma}(\boldsymbol{\Sigma} - \omega(z/n)I)^{-1}\right) \underset{\text{closely approximates}}{\overset{\text{deterministic equivalent}}{\longleftrightarrow}} \operatorname{Tr}\left(\boldsymbol{Q}\boldsymbol{X}^{*}(\boldsymbol{X}\boldsymbol{X}^{*} - zI_{n})^{-1}\boldsymbol{X}\right).

Similar deterministic equivalents exist for the mixed-resolvent traces. We obtain these comparisons along a deterministic backward path, inspired by and innovated upon the analysis of Cheng and Montanari [28]. This adjustment cancels the leading drift at each step and connects the sample trace to its population counterpart while keeping each step deterministic.

We emphasize that all our approximations are nonasymptotic and that all deterministic equivalents can vary in nn, even if they could have a limit in nn. For power law and critical decay, the spectral approximation remains valid at mesoscopic heights h=n−δh = n^{-\delta} in the interior of the bulk, with hh measured relative to the bulk width and 0<δ<1/170 < \delta< 1/17.

Related work

Random matrix theory. The study of random sample covariance matrices is classical and dates back to Wishart’s characterization of their distribution under Gaussian sampling [80]. The Marchenko-Pastur law subsequently described their limiting empirical spectrum when dimension and sample size grow proportionally [60]. More recently, these global descriptions have been sharpened to local and anisotropic laws, controlling resolvents and spectral statistics on increasingly fine scales [54]. Very recent work also establishes quantitative approximations for mixed-resolvent traces in the proportional regime [73].

Relevant works include extensions to polynomial dimensional scalings [35]. However, these settings do not directly address an infinite decaying population spectrum with a shrinking empirical bulk. Removing restrictions on the population condition number is essential here, since even finite-dimensional truncations become increasingly ill-conditioned. Cheng and Montanari [28] provided a methodological foundation through their leave-one-out interpolation strategy. Although they do not directly derive spectral laws and local laws over the bulk, their ridge formulas suggest the deterministic equivalent through analytic continuation.

Kernel random matrices. Our work closely relates to kernel methods, which provide a natural Hilbert-space setting through a possibly infinite-dimensional feature map, even when the input space is fixed [74 Section 4.5]. Despite having the Hilbert space embedding, much of the kernel random matrix literature instead considers growing input dimension. The foundational work of El Karoui [40] studies smooth inner-product and distance kernels when input dimension grows proportionally to sample size, showing that their spectra admit a linear approximation involving the sample Gram matrix. Subsequent work treats nonsmooth kernels, universality and spectral edges [30, 36, 43]. Further extensions include polynomial dimensional scalings that keep higher-degree feature components [62, 38]. Our framework instead permits a fixed kernel and input distribution, and studies the shrinking bulk of a decaying feature spectrum under quadratic concentration.

Spectral regularization and risk characterization. Studying the risk of spectral regularization and spectral filters permits a common framework for ridge regression, iterative regularization and early stopping. Classical analyses in this direction give risk bounds and establish learning rates and statistical optimality [58, 24].

Precise risk characterizations are available under proportional asymptotics, including for ridge regression and gradient flow [37, 2]. Beyond this regime, Cheng and Montanari [28] establish nonasymptotic, dimension-free deterministic equivalents for ridge prediction risk at the bulk scale, with nonnegative regularization. Li et al. [56] establish asymptotic risk formulas for analytic spectral algorithms under power-law decay, with regularization polynomially above the bulk scale. Their formulas apply the filter directly to the population covariance, whereas ours preserve corrections from a self-consistent spectral parameter at the bulk scale. Notably, Velikanov et al. [78] derive closely related risk formulas for an infinite Gaussian feature model, relying on concentration and stability approximations whose justification in that setting is left open. Our results give deterministic equivalents for analytic filters at the bulk scale under dimension-free concentration assumptions, including negative ridge and gradient flow, with explicit nonasymptotic errors for both variance and squared bias.

Dimension-free learning theory. Beyond the ridge regression work [28], dimension-free deterministic equivalents have been used to analyze kernel ridge regression and generalized cross-validation [63] and random feature regression [33]. Related random matrix approaches have also been used to study scaling laws [4]. These works relate prediction risk to spectral decay and signal alignment, making it possible to investigate how to choose regularization, how many random features are required to achieve statistical accuracy, and how learning rates vary with sample size.

Our extension to analytic filters provides quantitative inputs for studying iterative regularization and early stopping through the same spectral quantities. In particular, the deterministic risk expressions maintain the source-dependent bias-variance trade-off over the course of training. As a result, they offer a foundation for comparing regularization methods and studying stopping rules, with direct control over the approximation error. More generally, our spectral laws connect these learning problems to the geometry of the empirical spectrum.

Notation. For a positive integer nn, let [n]:={1,…,n}[n] := \{1,\ldots,n\} and let C+:={z∈C:Im⁡z>0}\mathbb{C}_{+} := \{z \in\mathbb{C} : \operatorname{Im} z > 0\}. For sets A,B⊆CA,B \subseteq\mathbb{C}, define dist⁡(A,B):=inf⁡{∣a−b∣:a∈A, b∈B}\operatorname{dist}(A,B) := \inf\{|a-b| : a \in A,\ b \in B\}, with points identified with singleton sets. We write a∧b:=min⁡{a,b}a \wedge b := \min\{a,b\} and a∨b:=max⁡{a,b}a \vee b := \max\{a,b\}. Let H\mathcal{H} be a real or complex separable Hilbert space. We denote by L(H)\mathcal{L}(\mathcal{H}) the Banach space of bounded linear operators acting on H\mathcal{H}. The inner product on H\mathcal{H}, denoted ⟨x,y⟩H\langle x,y\rangle_{\mathcal{H}}, is assumed to be conjugate-linear in its first argument; with respect to an orthonormal basis, it can be expressed in coordinates as x∗yx^{*}y. For A∈L(H)A \in\mathcal{L}(\mathcal{H}), we denote its adjoint by A∗A^{*}, its operator norm by ∥A∥\|A\|, its Hilbert-Schmidt (Frobenius) norm by ∥A∥F\|A\|_{F}, its spectrum by Spec⁡(A)\operatorname{Spec}(A), and its trace by Tr⁡(A)\operatorname{Tr}(A) whenever the latter is well defined. We write ∣A∣=(A∗A)1/2|A| = (A^{*}A)^{1/2} for the operator modulus and Tr⁡∣A∣\operatorname{Tr}|A| for the trace norm when AA is trace-class. The induced Hilbert-space norm is denoted by ∥x∥H2:=⟨x,x⟩H\|x\|_{\mathcal{H}}^{2} := \langle x,x\rangle_{\mathcal{H}}. For any bounded positive semidefinite operator AA on H\mathcal{H}, define

⟨x,y⟩A:=⟨x,Ay⟩H,∥x∥A2:=⟨x,x⟩A=∥A1/2x∥H2.\langle x,y\rangle_{A} := \langle x,Ay\rangle_{\mathcal{H}}, \qquad\|x\|_{A}^{2} := \langle x,x\rangle_{A} = \|A^{1/2}x\|_{\mathcal{H}}^{2}.

References

  1. [1]R. Adamczak. A note on the Hanson-Wright inequality for random vectors with dependencies. Electronic Communications in Probability, 20(72):1–13, 2015.arxiv.org/abs/1409.8457
  2. [2]A. Ali, J. Z. Kolter, and R. J. Tibshirani. A continuous-time view of early stopping for least squares regression. In Proceedings of the Twenty-Second International Conference on Artificial Intelligence and Statistics, volume 89 of Proceedings of Machine Learning Research, pages 1370–1378, 2019.arxiv.org/abs/1810.10082
  3. [3]T. M. Apostol. Calculating higher derivatives of inverses. The American Mathematical Monthly, 107(8):738–741, 2000.DOI
  4. [4]A. Atanasov, J. A. Zavatone-Veth, and C. Pehlevan. Scaling and renormalization in high-dimensional regression. Journal of Statistical Mechanics: Theory and Experiment, 2026(4):043404, 2026.DOI
  5. [5]B. Aupetit. A Primer on Spectral Theory. Universitext. Springer-Verlag, New York, 1991.DOI
  6. [6]Y. Bahri, E. Dyer, J. Kaplan, J. Lee, and U. Sharma. Explaining neural scaling laws. Proceedings of the National Academy of Sciences, 121(27):e2311878121, 2024.DOI
  7. [7]Z. Bai and J. W. Silverstein. Spectral Analysis of Large Dimensional Random Matrices. Springer, 2 edition, 2010.DOI
  8. [8]Z. D. Bai and J. W. Silverstein. Exact separation of eigenvalues of large dimensional sample covariance matrices. The Annals of Probability, 27(3):1536–1555, 1999.DOI
  9. [9]Z. D. Bai and J. W. Silverstein. CLT for linear spectral statistics of large-dimensional sample covariance matrices. The Annals of Probability, 32(1A):553–605, 2004.
  10. [10]J. Baik, G. Ben Arous, and S. Péché. Phase transition of the largest eigenvalue for nonnull complex sample covariance matrices. The Annals of Probability, 33(5):1643–1697, 2005.DOI
  11. [11]D. Bakry, I. Gentil, and M. Ledoux. Analysis and Geometry of Markov Diffusion Operators, volume 348 of Grundlehren der mathematischen Wissenschaften. Springer, Cham, 2014.DOI
  12. [12]P. L. Bartlett, P. M. Long, G. Lugosi, and A. Tsigler. Benign overfitting in linear regression. Proceedings of the National Academy of Sciences, 117(48):30063–30070, 2020.DOI
  13. [13]P. L. Bartlett, A. Montanari, and A. Rakhlin. Deep learning: a statistical viewpoint. Acta Numerica, 30:87–201, 2021.arxiv.org/abs/2103.09177
  14. [14]F. Benaych-Georges and R. R. Nadakuditi. The eigenvalues and eigenvectors of finite, low rank perturbations of large random matrices. Advances in Mathematics, 227(1):494–521, 2011.
  15. [15]H. Bercovici and V. Pata. Stable laws and domains of attraction in free probability theory. Annals of Mathematics, 149(3):1023–1060, 1999. With an appendix by Philippe Biane.
  16. [17]G. Blanchard and N. Mücke. Kernel regression, minimax rates and effective dimensionality: Beyond the regular case. Analysis and Applications, 18(4):683–696, 2020.arxiv.org/abs/1611.03979
  17. [18]A. Bloemendal, L. Erdős, A. Knowles, H.-T. Yau, and J. Yin. Isotropic local laws for sample covariance and generalized Wigner matrices. Electronic Journal of Probability, 19:1–53, 2014.
  18. [19]B. Bordelon, A. Canatar, and C. Pehlevan. Spectrum dependent learning curves in kernel regression and wide neural networks. In Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 1024–1034, 2020.arxiv.org/abs/2002.02561
  19. [20]S. Boucheron, O. Bousquet, G. Lugosi, and P. Massart. Moment inequalities for functions of independent random variables. The Annals of Probability, 33(2):514–560, 2005.
  20. [21]S. Boucheron, G. Lugosi, and P. Massart. Concentration Inequalities: A Nonasymptotic Theory of Independence. Oxford University Press, Oxford, 2013.
  21. [22]T. Brailovskaya and R. van Handel. Universality and sharp matrix concentration inequalities. Geometric and Functional Analysis, 34:1734–1838, 2024.arxiv.org/abs/2201.05142
  22. [23]T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, et al. Language models are few-shot learners. In Advances in Neural Information Processing Systems, volume 33, pages 1877–1901, 2020.arxiv.org/abs/2005.14165
  23. [24]A. Caponnetto and E. De Vito. Optimal rates for the regularized least-squares algorithm. Foundations of Computational Mathematics, 7(3):331–368, 2007.DOI
  24. [25]X. Chen and Y. Yang. Hanson-Wright inequality in Hilbert spaces with application to K-means clustering for non-Euclidean data. Bernoulli, 27(1):586–614, 2021.arxiv.org/abs/1810.11180
  25. [26]C. Cheng. High Dimensionality in Modern Machine Learning: A Random Matrix Theory Perspective. PhD thesis, Stanford University, 2025.
  26. [27]C. Cheng and R. F. Barber. Is memorization helpful or harmful? prior information sets the threshold. In Proceedings of the Thirty-Ninth Conference on Learning Theory, volume 336 of Proceedings of Machine Learning Research, pages 1399–1433, 2026.arxiv.org/abs/2602.09405
  27. [28]C. Cheng and A. Montanari. Dimension free ridge regression. The Annals of Statistics, 52(6):2879–2912, 2024.arxiv.org/abs/2210.08571
  28. [29]C. Cheng, J. Duchi, and R. Kuditipudi. Memorize to generalize: On the necessity of interpolation in high dimensional linear regression. In Proceedings of the Thirty-Fifth Conference on Learning Theory, volume 178 of Proceedings of Machine Learning Research, pages 5528–5560, 2022.arxiv.org/abs/2202.09889
  29. [30]X. Cheng and A. Singer. The spectrum of random inner-product kernel matrices. Random Matrices: Theory and Applications, 2(4):1350010, 2013.arxiv.org/abs/1202.3155
  30. [31]C. Chouard. Quantitative deterministic equivalent of sample covariance matrices with a general dependence structure. arXiv preprint arXiv:2211.13044, 2022.arxiv.org/abs/2211.13044
  31. [32]J. B. Conway. Functions of One Complex Variable I, volume 11 of Graduate Texts in Mathematics. Springer, 2 edition, 1978.DOI
  32. [33]L. Defilippis, B. Loureiro, and T. Misiakiewicz. Dimension-free deterministic equivalents and scaling laws for random feature regression. In Advances in Neural Information Processing Systems, volume 37, pages 104630–104693, 2024.arxiv.org/abs/2405.15699
  33. [34]J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. ImageNet: A large-scale hierarchical image database. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 248–255, 2009.
  34. [35]X. Ding and Z. Wang. Global and local CLTs for linear spectral statistics of general sample covariance matrices when the dimension is much larger than the sample size with applications. IEEE Transactions on Information Theory, 71(8):6248–6296, 2025.arxiv.org/abs/2308.08646
  35. [36]Y. Do and V. Vu. The spectrum of random kernel matrices: Universality results for rough and varying kernels. Random Matrices: Theory and Applications, 2(3):1350005, 2013.
  36. [37]E. Dobriban and S. Wager. High-dimensional asymptotics of prediction: Ridge regression and classification. The Annals of Statistics, 46(1):247–279, 2018.arxiv.org/abs/1507.03003
  37. [38]S. Dubova, Y. M. Lu, B. McKenna, and H.-T. Yau. Universality for the global spectrum of random inner-product kernel matrices in the polynomial regime. The Annals of Applied Probability, 36(5):4465–4515, 2026.arxiv.org/abs/2310.18280
  38. [39]R. Durrett. Probability: Theory and Examples. Cambridge University Press, 5 edition, 2019.
  39. [40]N. El Karoui. The spectrum of kernel random matrices. The Annals of Statistics, 38(1):1–50, 2010.
  40. [41]L. Erdős, H.-T. Yau, and J. Yin. Rigidity of eigenvalues of generalized Wigner matrices. Advances in Mathematics, 229(3):1435–1515, 2012.arxiv.org/abs/1007.4652
  41. [42]Z. Fan and I. M. Johnstone. Tracy-Widom at each edge of real covariance and MANOVA estimators. The Annals of Applied Probability, 32(4):2967–3003, 2022.pubmed.ncbi.nlm.nih.gov/36034074
  42. [43]Z. Fan and A. Montanari. The spectral norm of random inner-product kernel matrices. Probability Theory and Related Fields, 173:27–85, 2019.arxiv.org/abs/1507.05343
  43. [44]Z. Fan, R. Ma, E. Paquette, and Z. Wang. Anisotropic local law for non-separable sample covariance matrices. arXiv preprint arXiv:2602.17960, 2026.arxiv.org/abs/2602.17960
  44. [45]I. C. Gohberg and M. G. Krein. Introduction to the Theory of Linear Nonselfadjoint Operators in Hilbert Space, volume 18 of Translations of Mathematical Monographs. American Mathematical Society, Providence, RI, 1969.DOI
  45. [46]A. Grattafiori, A. Dubey, A. Jauhari, A. Pandey, A. Kadian, A. Al-Dahle, et al. The Llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024.arxiv.org/abs/2407.21783
  46. [47]W. Hachem, P. Loubaton, and J. Najim. Deterministic equivalents for certain functionals of large random matrices. The Annals of Applied Probability, 17(3):875–930, 2007.
  47. [48]P. Hall and C. C. Heyde. Martingale Limit Theory and Its Application. Academic Press, 1980.
  48. [49]T. Hasebe, T. Simon, and M. Wang. Some properties of the free stable distributions. Annales de l’Institut Henri Poincaré, Probabilités et Statistiques, 56(1):296–325, 2020.arxiv.org/abs/1805.01133
  49. [50]T. Hastie, A. Montanari, S. Rosset, and R. J. Tibshirani. Surprises in high-dimensional ridgeless least squares interpolation. The Annals of Statistics, 50(2):949–986, 2022.DOI
  50. [51]J. Ho, A. Jain, and P. Abbeel. Denoising diffusion probabilistic models. In Advances in Neural Information Processing Systems, volume 33, pages 6840–6851, 2020.arxiv.org/abs/2006.11239
  51. [52]I. S. Kac and M. G. Krein. R-functions—analytic functions mapping the upper halfplane into itself. American Mathematical Society Translations, Series 2, 103:1–18, 1974.
  52. [53]J. Kim, H. Fu, P. Bartlett, J. D. Lee, and J. Wu. Principal component regression dominates all monotone spectral filters for linear regression. arXiv preprint arXiv:2609.39440, 2026.arxiv.org/abs/2609.39440
  53. [54]A. Knowles and J. Yin. Anisotropic local laws for random matrices. Probability Theory and Related Fields, 169:257–352, 2017.arxiv.org/abs/1410.3516
  54. [55]Y. Li, Y. Cheng, Z. Chen, and Q. Lin. Risk equivalence between RKHS regression and sequence models for lipschitz spectral algorithms. arXiv preprint arXiv:2609.08817, 2026.arxiv.org/abs/2609.08817
  55. [56]Y. Li, W. Gan, Z. Shi, and Q. Lin. Generalization error curves for analytic spectral algorithms under power-law decay. Applied and Computational Harmonic Analysis, 87:101920, 2027.DOI
  56. [57]T. Liang and A. Rakhlin. Just interpolate: Kernel “ridgeless” regression can generalize. The Annals of Statistics, 48(3):1329–1347, 2020.
  57. [58]L. Lo Gerfo, L. Rosasco, F. Odone, E. De Vito, and A. Verri. Spectral algorithms for supervised learning. Neural Computation, 20(7):1873–1897, 2008.
  58. [59]Y. M. Lu and H.-T. Yau. An equivalence principle for the spectrum of random inner-product kernel matrices with polynomial scalings. The Annals of Applied Probability, 35(4):2411–2470, 2025.arxiv.org/abs/2205.06308
  59. [60]V. A. Marchenko and L. A. Pastur. Distribution of eigenvalues for some sets of random matrices. Mathematics of the USSR-Sbornik, 1(4):457–483, 1967.
  60. [61]S. Mei and A. Montanari. The generalization error of random features regression: Precise asymptotics and the double descent curve. Communications on Pure and Applied Mathematics, 75(4):667–766, 2022.
  61. [62]T. Misiakiewicz. Spectrum of inner-product kernel matrices in the polynomial regime and multiple descent phenomenon in kernel ridge regression. arXiv preprint arXiv:2204.10425, 2022.arxiv.org/abs/2204.10425
  62. [63]T. Misiakiewicz and B. Saeed. A non-asymptotic theory of kernel ridge regression: deterministic equivalents, test error, and GCV estimator. arXiv preprint arXiv:2403.08938, 2024.arxiv.org/abs/2403.08938
  63. [64]A. Montanari and Y. Zhong. The interpolation phase transition in neural networks: Memorization and generalization under lazy training. The Annals of Statistics, 50(5):2816–2847, 2022.arxiv.org/abs/2007.12826
  64. [65]E. Paquette, C. Paquette, L. Xiao, and J. Pennington. 4+3 phases of compute-optimal neural scaling laws. In Advances in Neural Information Processing Systems, volume 37, pages 16459–16537, 2024.arxiv.org/abs/2405.15074
  65. [66]G. Penedo, H. Kydlíček, L. Ben Allal, A. Lozhkov, M. Mitchell, C. Raffel, L. von Werra, and T. Wolf. The FineWeb datasets: Decanting the web for the finest text data at scale. In Advances in Neural Information Processing Systems, volume 37, pages 30811–30849, 2024.arxiv.org/abs/2406.17557
  66. [67]R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10684–10695, 2022.arxiv.org/abs/2112.10752
  67. [68]M. Rudelson and R. Vershynin. Hanson-Wright inequality and sub-Gaussian concentration. Electronic Communications in Probability, 18(82):1–9, 2013.arxiv.org/abs/1306.2872
  68. [69]R. Schatten. Norm Ideals of Completely Continuous Operators, volume 27 of Ergebnisse der Mathematik und ihrer Grenzgebiete. Springer-Verlag, Berlin, 1960.
  69. [70]C. Schuhmann, R. Beaumont, R. Vencu, C. Gordon, R. Wightman, M. Cherti, T. Coombes, A. Katta, C. Mullis, M. Wortsman, P. Schramowski, S. Kundurthy, K. Crowson, L. Schmidt, R. Kaczmarczyk, and J. Jitsev. LAION-5B: An open large-scale dataset for training next generation image-text models. In Advances in Neural Information Processing Systems, volume 35, pages 25278–25294, 2022.
  70. [71]J. W. Silverstein and S.-I. Choi. Analysis of the limiting spectral distribution of large dimensional random matrices. Journal of Multivariate Analysis, 54(2):295–309, 1995.
  71. [72]B. Simon. Trace Ideals and Their Applications, volume 120 of Mathematical Surveys and Monographs. American Mathematical Society, Providence, RI, 2 edition, 2005.
  72. [73]T. Stark and N. Dörnemann. Quantitative resolvent calculus for sample covariance matrices. arXiv preprint arXiv:2610.33087, 2026.arxiv.org/abs/2610.33087
  73. [74]I. Steinwart and A. Christmann. Support Vector Machines. Information Science and Statistics. Springer, 2008.
  74. [75]R. J. Tibshirani. Divided differences, falling factorials, and discrete splines. Foundations and Trends in Machine Learning, 15(6):694–846, 2022.
  75. [76]A. Tsigler and P. L. Bartlett. Benign overfitting in ridge regression. Journal of Machine Learning Research, 24(123):1–76, 2023.
  76. [77]A. W. van der Vaart and J. A. Wellner. Weak Convergence and Empirical Processes: With Applications to Statistics. Springer, 2 edition, 2023.
  77. [78]M. Velikanov, M. Panov, and D. Yarotsky. Generalization error of spectral algorithms. In The Twelfth International Conference on Learning Representations, 2024.arxiv.org/abs/2403.11696
  78. [79]M. J. Wainwright. High-Dimensional Statistics: A Non-Asymptotic Viewpoint. Cambridge University Press, 2019.
  79. [80]J. Wishart. The generalised product moment distribution in samples from a normal multivariate population. Biometrika, 20A(1-2):32–52, 1928.DOI
  80. [81]J. Wu, P. L. Bartlett, S. M. Kakade, J. D. Lee, and B. Yu. Risk comparisons in linear regression: implicit regularization dominates explicit regularization (extended abstract). In Proceedings of Thirty Ninth Conference on Learning Theory, volume 336 of Proceedings of Machine Learning Research, pages 6849–6851, 2026.arxiv.org/abs/2509.17251
  81. [82]P. Yaskov. Lower bounds on the smallest eigenvalue of a sample covariance matrix. Electronic Communications in Probability, 19(83):1–10, 2014.DOI
  82. [83]T. Zhang. Effective dimension and generalization of kernel learning. In Advances in Neural Information Processing Systems, volume 15, pages 454–461, 2002.
  83. [84]N. Zhivotovskiy. Dimension-free bounds for sums of independent matrices and simple tensors via the variational principle. Electronic Journal of Probability, 29(13):1–28, 2024.arxiv.org/abs/2108.08198

Paper details

vlnc:W_6s4y4e9g
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Mathematics

Not recorded for this paper

View statement
Looking for related papers…
Contents