WorksMathematics
Training-point consistency of kernel ridge regression with varying correlation scales
Abstract
The scale parameter in kernel ridge regression plays a pivotal role in modeling the correlation structure of data. In modern applications, sampling density and the underlying correlation range often change simultaneously, making a fixed-scale assumption highly restrictive. However, establishing asymptotic theories under varying scale parameters is fundamentally challenging. To address this gap, this paper develops a unified theoretical framework for kernel ridge regression under varying scale parameters. We establish that, on uniform tensor grids with product exponential kernels, the consistency of the training-point risk holds if and only if the effective local sample size diverges to infinity. This sharp criterion universally covers all regimes along arbitrary deterministic scale sequences, seamlessly unifying the long-range, fixed-scale, and short-range. Furthermore, we provide an explicit, signal-independent construction of a deterministic regularization penalty that guarantees this consistency. Numerical experiments illustrate the criterion and the corresponding penalty across all the three regimes.
Introduction
Consider the nonparametric regression model
where the design points , is a bounded domain, is the unknown regression function, and are random noises. Kernel ridge regression and Gaussian process regression are fundamental methods for estimating the signal in nonparametric statistics [18], machine learning [11], spatial statistics [13], and computer experiments [1]. The two methods combine observations at different sites through a correlation kernel that relies on a range/scale parameter. The scale parameter controls correlation decay: a smaller one makes distant sites more similar, whereas a larger one retains substantial correlation only between near neighbors. Therefore, the estimation performance is highly sensitive to selection of the scale parameter in the kernel [9, 22].
The asymptotic properties of kernel ridge regression and Gaussian process regression have been extensively studied in the literature. Caponnetto and De Vito [4] established optimal rates for regularized least squares under source conditions [4]. Steinwart, Hush, and Scovel [14] derived optimal rates for regularized least squares regression [14]. Dicker, Foster, and Hsu [6] clarified the relation between kernel ridge regression and principal component regression through minimax bounds and regularization qualification [6]. Wang and Jing [19] derived upper and lower error bounds for Gaussian process regression with possibly misspecified correlation functions and established corresponding convergence results for kernel ridge regression with a deterministic target [19]. The posterior asymptotics from the Bayesian perspective were investigated in [15, 16, 17]. However, these studies are all restricted to a fixed scale parameter. As is known, in many scenarios, particularly when the scale parameter is selected using data-driven methods such as cross-validation, the scale parameter (or its reciprocal) is not necessarily bounded [10].
Some studies have investigated the case of varying scale parameters for fixed sample sizes [2, 3, 7]. With a diverging sample size, sampling refinement and kernel scale adjustment can occur simultaneously. Therefore, fixed-domain asymptotics should address changing design density and changing correlation range together. To the best of our knowledge, the asymptotic properties of kernel ridge regression and Gaussian process regression with a varying scale parameter have not yet been systematically studied in the literature.
It is fundamentally challenging to extend the asymptotic properties with fixed scale parameters to varying scales. First, as the scale changes, the kernel matrix undergoes a continuous phase transition from a nearly flat (rank-deficient) structure to an almost diagonal one, rendering classical fixed-kernel spectral tools ineffective. Second, the reproducing kernel Hilbert space (RKHS) norm of a fixed signal depends heavily on the scale, shifting the bias-variance balance dynamically. Third, the simultaneous growth of the sample size and the scale creates a double asymptotic regime governed by the local sample size, where standard empirical process theory struggles to quantify whether local information suffices to offset the noise. Finally, in non-periodic bounded domains, boundary effects are amplified by scale variations and resist Fourier diagonalization. These intertwined challenges necessitate a unified framework that can bridge the long-range, fixed-scale, and short-range regimes.
To overcome these fundamental obstacles, we develop a unified asymptotic framework that fully characterizes the recoverability of signals under varying scale parameters. Our main theorem reveals a sharp phase transition governed entirely by the local sample size: the training-point risk of kernel ridge regression converges to zero if and only if the local sample size goes to infinity. This necessary and sufficient condition is valid for any nonzero signal and any deterministic scale sequence, seamlessly unifying the long-range, fixed-scale, and short-range. Furthermore, we provide a single, signal-independent penalty sequence that realizes this consistency, demonstrating that the bias-variance trade-off can be explicitly balanced across all regimes. The proof circumvents the aforementioned difficulties by combining shape-dependent RKHS norm bounds, an effective dimension estimate via product spectral grouping, and a finite-grid treatment of boundary terms.
The rest of this paper is organized as follows. Section 2 presents the exact risk decomposition and fixed-sample risk with extreme correlation scales. Section 3 establishes the unified framework. Section 4 reports reproducible numerical experiments. Section 5 concludes with discussion. The proofs are given in the Appendix.
Fixed-sample risk with extreme correlation scales
Under the model (1), write , abbreviated as when the sample size is clear, , and . Let be a positive-semidefinite correlation kernel with a scale parameter . Here controls correlation decay: a smaller makes distant sites more similar, whereas a larger retains substantial correlation only between near neighbors. The correlation matrix is defined by for .
Kernel ridge regression estimates the regression function in (1) by solving the optimization problem
where is a tuning penalty parameter, is the reproducing kernel Hilbert space (RKHS) induced by the kernel function , and is the corresponding norm. By (2), the fitted training-point vector is
This formula can also be derived from a Bayesian perspective. Under a zero-mean Gaussian prior with covariance and independent Gaussian noise of variance , computing the posterior mean at the training points yields (3) with .
We study the fixed-design training-point risk
Assumption 1. The noise satisfies
where denotes the identity matrix of order .
The following theorem presents a bias-variance decomposition of the risk. Let and denote the bias and variance of the kernel ridge estimator under the fixed-design training-point risk (4), respectively.
Theorem 1. Under Assumption 1,
We now study the following two extreme cases
for fixed and , where denotes the -vector with all elements being one. Write , , and .
Theorem 2. (i) For the long-range correlation case ,
(ii) For the short-range correlation case , if , then
Training-point risk consistency with varying correlation scales
Theorem 2 shows that, regardless of the choice of , the risk does not converge to zero for extreme values of and fixed . This section considers joint asymptotics as and vary simultaneously, and answers whether there exists such that the risk tends to zero. For convenience, we denote the sample size by instead of in the following. We introduce the following definition.
Definition 1. Given a scale sequence and a fixed , suppose that there exists a sequence such that
We then say that the corresponding kernel ridge estimator has training-point risk consistency with respect to for .
Because the risk is the second moment of the squared training-point error, its convergence to zero is equivalent to convergence of the training-point error in the corresponding space. Markov’s inequality further yields convergence in probability.
In our analysis, the dimension is fixed. We need the following assumptions on the design and the correlation function.
Assumption 2. Let and with some integer . The design points are
Assumption 3. For , the correlation function is
Under Assumptions 2 and 3, we denote the correlation matrix by with
for , where . Thus
Write
which can be viewed as local sample sizes in one dimension and dimension, respectively.
For a multi-index , write . Let
denote the mixed first-order Sobolev space equipped with the norm
This space is the reproducing-kernel Hilbert space of the product exponential kernel at a fixed scale, and its elements have continuous representatives. We always take the continuous representative, and grid evaluations and pointwise Riemann sums are well defined.
The following theorem presents the necessary and sufficient condition for training-point risk consistency.
Theorem 3. Under Assumptions 1, 2, and 3, let be fixed and nonzero, and let be an arbitrary deterministic sequence of positive numbers. The corresponding kernel ridge estimator has training-point risk consistency with respect to for if
and only if
More specifically, the following two statements hold.
(i) If , selecting
gives .
(ii) If , then for every deterministic sequence ,
This theorem covers all the cases of , fixed or bounded , and . The two branches of (10) have different orders at . This discontinuity arises from using the long-range spectral estimate and the coarse short-range variance bound on different sides; it serves only the consistency proof and does not represent a discontinuity in an optimal penalty or an optimal risk rate.
In the classical theory on consistency of kernel ridge regression for fixed , it is required that and ; see, e.g. [4]. Set to be fixed in Theorem 3, and our in (10) satisfies this condition.
Sanz-Alonso [12] gave a scale-consistent risk analysis of Matérn kernel ridge regression on periodic domains with quasi-uniform designs and Sobolev target classes [12]. Its kernel-scale information quantity has the same order as our local effective sample size at fixed dimension and fixed signal-to-noise ratio. Wenzel and Santin [21] studied asymptotically optimal shape parameters for finitely smooth Sobolev kernels and their relation to superconvergence [21]. Their results concern asymptotic approximation rates for noiseless kernel interpolation, whereas we study noisy kernel ridge regression.
Numerical experiments
We now illustrate the theoretical results with numerical examples. Consider the following one-dimensional and two-dimensional test functions,
in the regression model (1). For the one-dimensional case, the sample size is set as . For the two-dimensional case, set and , Let in Assumption 1. Under the grid design in Assumption 2, generate data with normal noises, and compute the kernel ridge estimator with the correlation function in Assumption 3.
For the one-dimensional case, consider the following four cases of the correlation parameter in kernel ridge regression: (i) fixed , corresponding to the local sample size ; (ii) short range , corresponding to ; (iii) long range , corresponding to ; (iv) too short range , corresponding to bounded local sample size . Cases (i)-(iii) satisfy the diverging-local-sample-size condition (9) in Theorem 3, and we use in (10). For case (iv) where condition (9) does not hold, we select that yields the minimum value of training-point risk.
For the two-dimensional case, consider the following four cases similarly: (i) fixed , corresponding to the local sample size ; (ii) short range , corresponding to ; (iii) long range , corresponding to ; (iv) too short range , corresponding to bounded local sample size . Cases (i)-(iii) satisfy condition (9), while case (iv) does not. We select in the same manner as in the one-dimensional case.
We compute the fixed-design training-point risk (4) by the analytic formula (5), not a Monte Carlo average over random realizations. The results are displayed in Figure 1. It can be seen that the risk converges to zero, albeit slowly, if the diverging-local-sample-size condition size condition holds, and remains bounded away from zero otherwise. The performance is consistent with the sufficiency and the necessity of the diverging-local-sample-size condition. The slow convergence may be due to the fact that we provide only a condition for convergence, rather than a condition for achieving the optimal convergence rate.
Figure 1. Training-point risk under four correlation regimes in one and two dimensions.
Discussion
This paper establishes a unified signal-recovery framework on fixed-dimensional bounded uniform grids with product exponential kernels, placing long-range correlation, fixed scale, and short-range correlation under one necessary and sufficient criterion and one penalty construction. For every fixed nonzero signal in the mixed first-order Sobolev space, training-point risk consistency is equivalent to the diverging-local-sample-size condition . The statement holds along the entire sequence and therefore also covers mixed scale paths containing subsequences from different regimes.
The present study is restricted to specific conditions, leaving several open problems for future research. First, it is unclear whether the diverging-local-sample-size condition is valid for general Matérn kernels, nonuniform designs, out-of-sample prediction, and continuous recovery. Second, we establish the validity of prespecified deterministic penalty sequences, while the conditions under which cross-validation and other data-driven rules succeed are left open. Third, the numerical results show that the penalty sequence we proposed is far from optimal, and regime-specific optimal penalties and optimal risk rates remain to be studied.
Proofs
Proofs in Section 2
Proof of Theorem 1. Right-multiplying by gives
Consequently,
Expanding the squared norm, we find that the expected cross term is
because . The deterministic part divided by is . For the noise part,
where the last step uses the symmetry of . Division by gives (5).
Proof of Theorem 2. The identity directly gives
This yields the fitted vector . Orthogonally decomposing into and , and using , gives
which reaches its minimum at , and this completes the proof of (i).
By , we can immediately get
Some algebra gives its minimizer . Substitution gives (ii).
Proofs in Section 3
For an correlation matrix and -vector , define , and let denote the operator 2-norm/maximum singular value of . Let denote the reproducing-kernel Hilbert space of the continuous product exponential kernel in Assumption 3 equipped with the norm . The following lemma presents the general risk bounds of kernel ridge regression. As in Theorem 1, and denote the bias and variance of the kernel ridge estimator under the fixed-design training-point risk (4), respectively.
Lemma 1. Under Assumption 1, we have
the upper bounds , ; (12)
and the lower bounds , . (13)
Moreover, under Assumptions 1, 2, and 3, if and , then
Proof. Write the eigenvalue decomposition . By the reproducingkernel representation, there exists such that and . Therefore,
Since and ,
For the lower bounds, gives
Similarly, using with the Frobenius norm , we obtain
Finally, minimum-norm interpolation [20] gives
and taking yields (12).
The following lemma shows the effect of the correlation scale on the RKHS norm.
Lemma 2. Under Assumptions 2 and 3, for , there exists a constant , depending only on and , such that
Proof. We first construct a bounded extension of to . Taking the even reflection of successively about 0 and 1 in each coordinate gives an extension on . Each even reflection preserves the first weak derivative in that coordinate and its integrability. Coordinatewise application therefore preserves all mixed weak derivatives and gives a Sobolev-norm bound depending only on .
Next, take an infinitely differentiable function , defined on , which equals 1 on and is supported in . Let and . Then agrees with on and has compact support. Applying the multivariate Leibniz rule to compute the partial derivatives term by term for gives
where is a constant depending only on and the norm is defined by (8). The right-hand side of (14) is therefore a constant that depends only on and . Regard the kernel function on as the restriction of a kernel on . Let denote the RKHS induced by this kernel on . The original RKHS induced by the restricted kernel consists of restrictions of functions in . We have
Consider the Fourier transform . The spectral density of the one-dimensional exponential kernel is
and thus the product kernel has spectral density . The Fourier representation of the RKHS norm and Parseval’s identity [13, 20] give
If , then ; if , then . Combining (14) and (15) proves (13).
Lemma 2 unifies the two bias controls. For , (12) and (13) give
for , they give
The constant may differ between the two cases.
The coarse variance bound in (12) remains valid under extremely long correlation, but a sharper spectral bound will be presented. We first estimate the nonprincipal eigenvalues of the one-dimensional factor in the following lemma.
Lemma 3. Under Assumptions 2 and 3, let be the eigenvalues of in (7). There exists an absolute constant such that, for all , , and ,
Proof. Note that in (6). We have
Direct expansion for every gives
Discarding the last two nonnegative terms, and letting be the path-graph Laplacian [5] defined by , yields
Since and for , taking gives
The th eigenvalue of the path-graph Laplacian satisfies
where we use for . Applying the Courant–Fischer theorem [8] to the increasingly ordered eigenvalues of gives
Taking reciprocals proves (18).
The following lemma provides a sharper upper bound on the variance in Lemma 1 under long correlation.
Lemma 4. Under Assumptions 1, 2, and 3, let , , and . For , there exists a constant , depending only on , such that
Proof. The eigenvalues of the product matrix are indexed by multi-indices :
By Theorem 1,
For , the inequalities give
We group the terms by .
When , the group contains at most multi-indices. Bounding each summand by and adding the groups , we find that their contribution to is at most
When , every . Lemma 3 gives
Consequently, the sum over finite terms in the right-hand side of (20) is controlled by the unrestricted sum, i.e.,
where (we may take in Lemma 3, and thus implies ).
We now estimate both the body region and all remaining tail terms. Define the lattice count
Clearly, . If , then
Hence the body-region contribution is at most
For the tail, let and use the dyadic shells
These shells cover every index with . Any overlap with the body region is negligible since our estimate is used for an upper bound. Every term in satisfies
Combining this with (22), we bound the tail contribution by
The second inequality uses the ratio of consecutive terms, : beyond a sufficiently large , comparison with a geometric series applies, and the finitely many earlier terms are absorbed into a constant depending only on .
Substituting and absorbing constants depending only on , we find that the full-coordinate group contributes at most
to . Combining (21) and (25) implies (19). □
Proof of Theorem 3. (i) We first prove sufficiency. Suppose .
When , take
For sufficiently large , . Lemma 4 therefore gives
Thus the risk tends to zero in this regime.
When , take . By (16) and (12),
Both terms tend to zero as . Combining the above two cases proves the sufficiency of (10).
(ii) We now prove necessity. Suppose . Choose a subsequence and a constant such that
To simplify notation, we continue to write for this subsequence. The spectrum of a Kronecker product consists of products of the factor spectra, and hence by (7). Since is symmetric, its operator 2-norm is at most its maximum absolute row sum. We have
Along the subsequence satisfying (26), , and therefore . It follows that
For every , if , then (13) gives
if , then combining (13) and (27) gives
Therefore, along this subsequence satisfying (26) and for every deterministic ,
Note that continuity of the fixed and Riemann summation on the bounded domain give
The right-hand side of (28) has a positive limit inferior, which proves (11).
Acknowledgements
This work is partially supported by the National Natural Science Foundation of China (grant number 12571276).
References
- [1]Baker, E., Barbillon, P., Fadikar, A., Gramacy, R. B., Herbei, R., Higdon, D., Huang, J., Johnson, L. R., Ma, P., Mondal, A., Pires, B., Sacks, J., & Sokolov, V. (2022). Analyzing stochastic computer models: A review with opportunities. Statistical Science, 37, 64–89.arxiv.org/abs/2002.01321
- [2]Barthélé, S., Amblard, P. O., Tremblay, N., & Usevich, K. (2023). Gaussian process regression in the flat limit. The Annals of Statistics, 51(6), 2471–2505.arxiv.org/abs/2201.01074
- [3]Barthelmè, S., & Usevich, K. (2021). Spectral properties of kernel matrices in the flat limit. SIAM Journal on Matrix Analysis and Applications, 42(1), 17–57.DOI
- [4]Caponnetto, A., & De Vito, E. (2007). Optimal rates for the regularized least-squares algorithm. Foundations of Computational Mathematics, 7(3), 331–368.DOI
- [5]Chung, F. R. K. (1997). Spectral Graph Theory. American Mathematical Society.
- [6]Dicker, L. H., Foster, D. P., & Hsu, D. (2017). Kernel ridge vs. principal component regression: Minimax bounds and the qualification of regularization operators. Electronic Journal of Statistics, 11(1), 1022–1047.
- [7]Driscoll, T. A., & Fornberg, B. (2002). Interpolation in the limit of increasingly flat radial basis functions. Computers & Mathematics with Applications, 43(3–5), 413–422.DOI
- [8]Horn, R. A., & Johnson, C. R. (2012). Matrix Analysis (2nd ed.). Cambridge University Press.
- [9]Kaufman, C. G., & Shaby, B. A. (2013). The role of the range parameter for estimation and prediction in geostatistics. Biometrika, 100(2), 473–484.arxiv.org/abs/1108.1851
- [10]Pramanick, S., & Karvonen, T. (2026). Lengthscale estimation for the Gaussian kernel and deterministic response functions. arXiv:2611.13930. https://doi.org/10.48550/arXiv.2611.13930
- [11]Rasmussen, C. E., & Williams, C. K. I. (2006). Gaussian processes for machine learning. MIT Press.DOI
- [12]Sanz-Alonso, D. (2026). Non-asymptotic analysis of Matérn regression: The roles of target and kernel lengthscales. arXiv:2608.22553. https://arxiv.org/abs/2608.22553
- [13]Stein, M. L. (1999). Interpolation of spatial data: Some theory for kriging. Springer.
- [14]Steinwart, I., Hush, D. R., & Scovel, C. (2009). Optimal rates for regularized least squares regression. In Proceedings of the 22nd Annual Conference on Learning Theory (pp. 79–93).
- [15]van der Vaart, A. W., & van Zanten, J. H. (2008). Rates of contraction of posterior distributions based on Gaussian process priors. The Annals of Statistics, 36(3), 1435–1463.DOI
- [16]van der Vaart, A. W., & van Zanten, J. H. (2009). Adaptive Bayesian estimation using a Gaussian random field with inverse Gamma bandwidth. The Annals of Statistics, 37(5B), 2655–2675.
- [17]van der Vaart, A. W., & van Zanten, J. H. (2011). Information rates of nonparametric Gaussian process methods. Journal of Machine Learning Research, 12, 2095–2119.DOI
- [18]Wahba, G. (1990). Spline models for observational data. SIAM.
- [19]Wang, W., & Jing, B.-Y. (2022). Gaussian process regression: Optimality, robustness, and relationship with kernel ridge regression. Journal of Machine Learning Research, 23(193), 1–67.
- [20]Wendland, H. (2005). Scattered Data Approximation. Cambridge University Press, Cambridge.DOI
- [21]Wenzel, T., & Santin, G. (2026). On the optimal shape parameter for kernel methods: Sharp direct and inverse statements. arXiv:2601.14070. https://arxiv.org/abs/2601.14070
- [22]Zhang, H. (2004). Inconsistent estimation and asymptotically equal interpolations in model-based geostatistics. Journal of the American Statistical Association, 99(465), 250–261.DOI
