Introduction
Bilevel optimization is a class of hierarchical optimization problems in which the solution to an outer problem depends on the solution of an inner problem [28 ] . These methods naturally arise in numerous machine learning applications, including hyperparameter optimization [68 , 37 , 40 ] , meta-learning [41 , 38 ] , neural architecture search [65 ] , and reinforcement learning [51 , 74 ] . The main challenge in bilevel problems is that the outer objective is defined implicitly through the solution of the inner problem, making both gradient computation and generalization analysis nontrivial [77 ] .
Bilevel optimization poses two intertwined challenges: numerically, one must approximate not only the inner-level solution but also its sensitivity to outer variables in order to compute hypergradients [31 , 13 , 42 ] ; and statistically, one must understand how the outer solution generalizes as the sample size grows [33 , 54 , 6 ] . A large body of work in the well-specified setting with a unique inner level solution addresses the numerical difficulty. Crucially, they typically require the inner objective to be strongly convex, which ensures a unique, smoothly varying inner solution and is therefore amenable to stability and convergence analyses [43 , 1 , 20 ] . Modern machine learning applications, however, usually employ neural networks as inner models [11 , 92 , 65 ] ; the resulting inner problems are nonconvex and often overparameterized, so multiple inner solutions can exist and their dependence on the outer variables becomes ambiguous [90 , 4 ] . This dependence can lead to instability, both because inner solutions may be non-unique and because small perturbations in the lower-level problem can induce large changes in the hypergradient [66 , 7 , 21 ] . As a consequence, strong generalization guarantees in the nonconvex setting are still largely limited, and the behavior of the resulting solutions on unseen data is still poorly understood. On the statistical side, most generalization results adopt a fixed finite-dimensional parametric viewpoint [29 , 52 ] . This perspective becomes restrictive for overparameterized networks and nonparametric methods, where the effective parameter space (or feature map) can grow with the sample size and the parameters obtained at different sample sizes are not directly comparable. Bridging this gap therefore requires analytical tools that can handle (i) nonconvex inner landscapes with possibly multiple solutions arising from overparameterization, and (ii) nonparametric function spaces, such as a Reproducing Kernel Hilbert Space (RKHS), in order to obtain meaningful stability and generalization guarantees for bilevel learning in modern settings.
To address these challenges, the functional bilevel optimization framework, introduced in [78 ] , provides a principled framework that shifts focus from parameter-centric approaches to a function-space perspective. By formulating the inner problem as an optimization over an infinite-dimensional Hilbert space of functions, such as an RKHS [33 ] , this approach naturally accommodates multiple inner solutions, which frequently arise in overparameterized neural networks. A crucial advantage of this functional perspective is that it only requires strong convexity in the function space, rather than with respect to the parameter vector, the latter being rarely satisfied by modern neural networks [64 ] . This ensures the uniqueness of the inner solution for any outer variable, providing a well-defined dependency of the inner problem on the outer parameters and making stability and generalization analyses tractable in practice. In a related line of work, El Khoury et al. [33 ] provided generalization guarantees for functional bilevel optimization where the considered space is an RKHS. However, these results remain restrictive in the deep learning setting, as they do not accommodate highly expressive models such as overparameterized networks, leaving the question of generalization in bilevel optimization in such settings largely open.
In the present paper, we address this gap by studying Overparameterized Bilevel Optimization (OBO) problems in which the lower-level model is overparameterized , i.e. , it contains more parameters than training samples and thus may have multiple global minimizers for the inner problem [90 ] . Formally, we consider problems of the form:
min ω ∈ Ω F ( ω ) : = L o u t ( ω , h ω ⋆ ) such that h ω ⋆ = argmin h ∈ H L i n ( ω , h ) , \min_{\omega\in\Omega} \mathcal{F}(\omega) := L_{\mathrm{out}}(\omega,h_\omega^\star)
\quad\text{such that} \quad
h_\omega^\star= \underset{h \in\mathcal{H}}{\operatorname{argmin}}\ L_{\mathrm{in}}(\omega,h), ω ∈ Ω min F ( ω ) := L out ( ω , h ω ⋆ ) such that h ω ⋆ = h ∈ H argmin L in ( ω , h ) ,
where Ω ⊂ R d \Omega\subset\mathbb{R}^{d} Ω ⊂ R d is nonempty, compact, and convex, H \mathcal{H} H is a Hilbert space of functions on X ⊂ R p \mathcal{X} \subset\mathbb{R}^{p} X ⊂ R p taking values in Y ⊂ R \mathcal{Y} \subset\mathbb{R} Y ⊂ R , and L i n , L o u t : Ω × H → R L_{\mathrm{in}},L_{\mathrm{out}}:\Omega\times\mathcal{H} \to\mathbb{R} L in , L out : Ω × H → R denote the inner and outer objective functions, respectively. Throughout this work, both L i n L_{\mathrm{in}} L in and L o u t L_{\mathrm{out}} L out take the form of expected squared losses, a choice motivated by its practical relevance in machine learning [41 , 77 ] and its ubiquity as a theoretical framework [23 , 9 ] .
Contributions. We study generalization in bilevel least squares , i.e. , bilevel optimization problems where both levels are least-squares regressions. This setting is advantageous, as it expresses the hypergradient ∇ F \nabla\mathcal{F} ∇ F in terms of two regression functions, the inner solution and the outer regression function. The latter does not depend on the outer variable and can thus be estimated once and for all, which simplifies both the analysis and the algorithmic complexity. Our contributions are as follows.
(i) Convergence under inexact hypergradients. We bound the near-stationarity measure of a projected mini-batch stochastic gradient descent (SGD) method driven by a plug-in hypergradient estimator, in terms of the two estimators’ generalization errors (Theorem 2.4 ).
(ii) Generalization rates in the NTK regime. When these two estimators are two-layer ReLU networks with bias in both layers trained by gradient flow with early stopping, we derive finite-sample generalization bounds in the neural tangent kernel (NTK) regime (Proposition 3.1). These bounds hold in expectation, and uniformly over the outer variable for the inner estimator. To obtain them, we truncate the trained networks, i.e. , set them to zero, on a rare event outside of which they provably remain in the NTK regime.
(iii) End-to-end sample complexity. Plugging the rates of (ii) into the bound of (i) yields the sample complexity of the bilevel method in the NTK regime (Corollary 3.2): the statistical error is governed by the slower of the two regression rates, so the inner and outer sample sizes should be balanced to prevent either level from becoming a bottleneck.
(iv) Implicit bias beyond the NTK regime. Synthetic experiments suggest an initialization-dependent implicit bias of the bilevel structure, which promotes inner-level feature learning and improves performance over the NTK regime.
Overparameterized bilevel optimization with squared losses
In this section, we introduce the problem under consideration, derive the expression of its hypergradient, and present the gradient-based algorithm we analyze along with its convergence guarantee.
Problem formulation
In many machine learning problems with a bilevel structure, the inner and outer objectives L i n L_{\mathrm{in}} L in and L o u t L_{\mathrm{out}} L out are typically expectations of pointwise losses over some data distributions. In this work, we instantiate both as squared losses , a choice that arises widely in regression [46 ] and causal inference [93 , 56 ] , and that allows us to develop our theoretical analysis in a simplified setting. Specifically, we consider:
L i n ( ω , h ) = 1 2 E ( x , z ) ∼ P X Z [ ∣ h ( x ) − g ω ( z ) ∣ 2 ] , L o u t ( h ) = 1 2 E ( x , y ) ∼ P X Y [ ∣ h ( x ) − y ∣ 2 ] , (1) L_{\mathrm{in}}(\omega,h)=\frac{1}{2}\mathbb{E}_{(x,z)\sim\mathbb{P}_{XZ}}\left[\lvert h(x)-g_{\omega}(z)\rvert^{2}\right],\qquad L_{\mathrm{out}}(h)=\frac{1}{2}\mathbb{E}_{(x,y)\sim\mathbb{P}_{XY}}\left[\lvert h(x)-y\rvert^{2}\right], \tag*{(1)} L in ( ω , h ) = 2 1 E ( x , z ) ∼ P X Z [ ∣ h ( x ) − g ω ( z ) ∣ 2 ] , L out ( h ) = 2 1 E ( x , y ) ∼ P X Y [ ∣ h ( x ) − y ∣ 2 ] , (1)
where g ω : Z ⊂ R r → R g_{\omega}:\mathcal{Z}\subset\mathbb{R}^{r}\to\mathbb{R} g ω : Z ⊂ R r → R is a model parameterized by ω \omega ω . Note that we omit the explicit dependence of L o u t L_{\mathrm{out}} L out on ω \omega ω , as it occurs only implicitly through h ω ⋆ h_{\omega}^{\star} h ω ⋆ . The distributions P X Z \mathbb{P}_{XZ} P X Z and P X Y \mathbb{P}_{XY} P X Y are the marginals of a joint distribution P \mathbb{P} P over triplets ( x , y , z ) (x,y,z) ( x , y , z ) in X × Y × Z \mathcal{X}\times\mathcal{Y}\times\mathcal{Z} X × Y × Z , sharing the same marginal P X \mathbb{P}_{X} P X over elements x ∈ X x\in\mathcal{X} x ∈ X . A natural space of functions H \mathcal{H} H in this context is L 2 ( X ) L^{2}(\mathcal{X}) L 2 ( X ) , the space of square-integrable functions with respect to P X \mathbb{P}_{X} P X , defined on X \mathcal{X} X and taking values in Y \mathcal{Y} Y . Provided that z ↦ g ω ( z ) z\mapsto g_{\omega}(z) z ↦ g ω ( z ) is square integrable, this problem is guaranteed to admit a unique solution to the inner problem, which takes the simple form of a conditional expectation, i.e., h ω ⋆ ( x ) = E P [ g ω ( z ) ∣ x ] h_{\omega}^{\star}(x)=\mathbb{E}_{\mathbb{P}}[g_{\omega}(z)\mid x] h ω ⋆ ( x ) = E P [ g ω ( z ) ∣ x ] , for any ω ∈ Ω \omega\in\Omega ω ∈ Ω . In practice, such conditional expectation is only estimated, typically using a neural network, and thus requires careful analysis of the generalization error, which is a central challenge in this study. Typical applications of this problem in machine learning include instrumental variable regression and value function fitting in reinforcement learning, as further discussed in Appendix § 9 . In what follows, ∥ ⋅ ∥ \lVert\cdot\rVert ∥ ⋅ ∥ denotes the Euclidean norm, ∥ ⋅ ∥ L 2 ( X ) \lVert\cdot\rVert_{L^{2}(\mathcal{X})} ∥ ⋅ ∥ L 2 ( X ) denotes the L 2 L^{2} L 2 norm with respect to P X \mathbb{P}_{X} P X , and ⟨ ⋅ , ⋅ ⟩ L 2 ( X ) \langle\cdot,\cdot\rangle_{L^{2}(\mathcal{X})} ⟨ ⋅ , ⋅ ⟩ L 2 ( X ) denotes the corresponding inner product. For notational simplicity, we will write P \mathbb{P} P without subscript in the remainder of the paper whenever the meaning can be inferred from context.
Throughout the paper, we assume that the model g ω g_{\omega} g ω is uniformly smooth with bounded values and gradients, that the target space is bounded, that the input space is compact, and that P X \mathbb{P}_{X} P X admits a density p P p_{\mathbb{P}} p P with respect to the Lebesgue measure on X \mathcal{X} X that is bounded above and bounded away from zero on some open ball. These assumptions are formalized below.
Assumption A. (Parameterization properties of g ω g_{\omega} g ω ) There exist an open convex set U ⊃ Ω U\supset\Omega U ⊃ Ω such that, for all z ∈ Z z\in\mathcal{Z} z ∈ Z , the map ( ω , z ) ↦ g ω ( z ) (\omega,z)\mapsto g_{\omega}(z) ( ω , z ) ↦ g ω ( z ) is differentiable on U U U with respect to ω \omega ω and its partial derivative ∂ ω g ω ( z ) ∈ R d \partial_{\omega}g_{\omega}(z)\in\mathbb{R}^{d} ∂ ω g ω ( z ) ∈ R d is L L L -Lipschitz on U U U . Moreover, there exist constants G , D > 0 G,D>0 G , D > 0 such that:
sup ω ∈ U , z ∈ Z ∥ ∂ ω g ω ( z ) ∥ ≤ G and sup ω ∈ U , z ∈ Z ∣ g ω ( z ) ∣ ≤ D . \sup_{\omega\in U,z\in\mathcal{Z}}\lVert\partial_{\omega}g_{\omega}(z)\rVert\le G
\qquad\text{and}\qquad
\sup_{\omega\in U,z\in\mathcal{Z}}\lvert g_{\omega}(z)\rvert\le D. ω ∈ U , z ∈ Z sup ∥ ∂ ω g ω ( z )∥ ≤ G and ω ∈ U , z ∈ Z sup ∣ g ω ( z )∣ ≤ D .
Assumption B. (Boundedness of Y \mathcal{Y} Y ) The target space Y ⊂ R \mathcal{Y}\subset\mathbb{R} Y ⊂ R is bounded, i.e., there exists a constant B > 0 B>0 B > 0 such that ∣ y ∣ ≤ B \lvert y\rvert\le B ∣ y ∣ ≤ B for any y ∈ Y y\in\mathcal{Y} y ∈ Y .
Assumption C. (Compactness of X \mathcal{X} X ) The input space X ⊂ R p \mathcal{X}\subset\mathbb{R}^{p} X ⊂ R p is compact with a constant A > 0 A>0 A > 0 such that ∥ x ∥ ≤ A \lVert x\rVert\le A ∥ x ∥ ≤ A for any x ∈ X x\in\mathcal{X} x ∈ X .
Assumption D. (Bounded density) There exists a constant M P > 0 M_{\mathbb{P}}>0 M P > 0 such that p P ≤ M P p_{\mathbb{P}}\le M_{\mathbb{P}} p P ≤ M P . Moreover, p P p_{\mathbb{P}} p P is bounded below by a positive constant on some open ball contained in X \mathcal{X} X .
These are relatively mild conditions introduced mainly to simplify the analysis as further discussed in Appendix § 10 . Additional assumptions will be introduced in later sections whenever needed.
Optimization algorithm for solving (OBO)
We aim to solve (OBO) with a gradient-based algorithm, driven by the hypergradient ∇ F \nabla\mathcal{F} ∇ F . The following result establishes the differentiability of F \mathcal{F} F and provides an explicit expression for ∇ F \nabla\mathcal{F} ∇ F , by applying the functional implicit differentiation theorem [78 Proposition 2.2] .
Proposition 2.1 (Hypergradient of F \mathcal{F} F ). Under Assumptions A \mathbf{A} A and B \mathbf{B} B , F \mathcal{F} F is differentiable on U U U , and its hypergradient is given, for any ω ∈ U \omega\in U ω ∈ U , by:
∇ F ( ω ) = − E P [ ∂ ω g ω ( z ) a ω ⋆ ( x ) ] ∈ R d , (2) \nabla\mathcal{F}(\omega)=-\mathbb{E}_{\mathbb{P}}\left[\partial_{\omega}g_{\omega}(z)a_{\omega}^{\star}(x)\right]\in\mathbb{R}^{d}, \tag*{(2)} ∇ F ( ω ) = − E P [ ∂ ω g ω ( z ) a ω ⋆ ( x ) ] ∈ R d , (2) where a ω ⋆ = − h ω ⋆ + b ⋆ ∈ L 2 ( X ) a_{\omega}^{\star}=-h_{\omega}^{\star}+b^{\star}\in L^{2}(\mathcal{X}) a ω ⋆ = − h ω ⋆ + b ⋆ ∈ L 2 ( X ) is the adjoint function, h ω ⋆ : x ↦ E P [ g ω ( z ) ∣ x ] h_{\omega}^{\star}:x\mapsto\mathbb{E}_{\mathbb{P}}[g_{\omega}(z)\mid x] h ω ⋆ : x ↦ E P [ g ω ( z ) ∣ x ] is the inner solution, and b ⋆ : x ↦ E P [ y ∣ x ] b^{\star}:x\mapsto\mathbb{E}_{\mathbb{P}}[y\mid x] b ⋆ : x ↦ E P [ y ∣ x ] is the regression function, defined as the minimizer of L o u t L_{\mathrm{out}} L out over L 2 ( X ) L^{2}(\mathcal{X}) L 2 ( X ) . Moreover, F \mathcal{F} F is L \mathcal{L} L -smooth on U U U , in particular on Ω \Omega Ω , with L : = G 2 + L ( D + B ) \mathcal{L}:=G^{2}+L(D+B) L := G 2 + L ( D + B ) where the constants G G G , L L L , D D D and B B B are defined in Assumptions A \mathbf{A} A and B \mathbf{B} B .
While the expression in Equation (2) is exact, it requires full knowledge of P \mathbb{P} P , h ω ⋆ h_{\omega}^{\star} h ω ⋆ , and b ⋆ b^{\star} b ⋆ , which are unavailable in practice, making it necessary to estimate ∇ F ( ω ) \nabla\mathcal{F}(\omega) ∇ F ( ω ) from data. Its structure is particularly appealing: since ∂ ω g ω ( z ) \partial_{\omega}g_{\omega}(z) ∂ ω g ω ( z ) is known, the statistical difficulty reduces entirely to approximating the adjoint function a ω ⋆ a_{\omega}^{\star} a ω ⋆ , which itself decomposes into two regression functions b ⋆ b^{\star} b ⋆ and h ω ⋆ h_{\omega}^{\star} h ω ⋆ . This structure is key to our analysis, as it allows us to approximate h ω ⋆ h_{\omega}^{\star} h ω ⋆ and b ⋆ b^{\star} b ⋆ independently and combine their estimates to obtain an approximation of the hypergradient using finite-sample estimates.
Empirical losses. Estimating h ω ⋆ h_{\omega}^{\star} h ω ⋆ and b ⋆ b^{\star} b ⋆ requires a finite-sample approximation of (OBO). To this end, we assume access to two datasets D i n = { ( x i , z i ) } i = 1 n i n \mathcal{D}_{\mathrm{in}}=\{(x_i,z_i)\}_{i=1}^{n_{\mathrm{in}}} D in = {( x i , z i ) } i = 1 n in and D o u t = { ( x ~ j , y ~ j ) } j = 1 n o u t \mathcal{D}_{\mathrm{out}}=\{(\widetilde{x}_j,\widetilde{y}_j)\}_{j=1}^{n_{\mathrm{out}}} D out = {( x j , y j ) } j = 1 n out consisting of i.i.d. samples drawn from the distributions P X Z \mathbb{P}_{XZ} P X Z and P X Y \mathbb{P}_{XY} P X Y , respectively. This is a standard setting in machine learning, since it leads to tractable approximations of (OBO). We therefore consider the corresponding empirical problem, obtained by replacing the population losses L i n L_{\mathrm{in}} L in and L o u t L_{\mathrm{out}} L out with their empirical counterparts:
L ^ i n ( ω , h ) = 1 2 n i n ∑ i = 1 n i n ( h ( x i ) − g ω ( z i ) ) 2 , L ^ o u t ( h ) = 1 2 n o u t ∑ j = 1 n o u t ( h ( x ~ j ) − y ~ j ) 2 . (3) \widehat{L}_{\mathrm{in}}(\omega,h)=\frac{1}{2n_{\mathrm{in}}}\sum_{i=1}^{n_{\mathrm{in}}}(h(x_i)-g_{\omega}(z_i))^{2},\qquad\widehat{L}_{\mathrm{out}}(h)=\frac{1}{2n_{\mathrm{out}}}\sum_{j=1}^{n_{\mathrm{out}}}(h(\widetilde{x}_j)-\widetilde{y}_j)^{2}. \tag*{(3)} L in ( ω , h ) = 2 n in 1 i = 1 ∑ n in ( h ( x i ) − g ω ( z i ) ) 2 , L out ( h ) = 2 n out 1 j = 1 ∑ n out ( h ( x j ) − y j ) 2 . (3)
These serve as training objectives for the estimators h ^ ω \widehat{h}_{\omega} h ω and b ^ \widehat{b} b of h ω ⋆ h_{\omega}^{\star} h ω ⋆ and b ⋆ b^{\star} b ⋆ : the former is trained by minimizing L ^ i n ( ω , ⋅ ) \widehat{L}_{\mathrm{in}}(\omega,\cdot) L in ( ω , ⋅ ) for a given ω \omega ω , while the latter is estimated by minimizing L ^ o u t \widehat{L}_{\mathrm{out}} L out .
Hypergradient estimator. With h ^ ω \widehat{h}_{\omega} h ω and b ^ \widehat{b} b at hand, it remains to approximate the expectation in Equation (2) . Accordingly, we assume access, at each use of the estimator, to a fresh mini-batch B g = { ( x l , z l ) } l = 1 n g \mathcal{B}_{g}=\{(x_l,z_l)\}_{l=1}^{n_g} B g = {( x l , z l ) } l = 1 n g of i.i.d. samples drawn from P X Z \mathbb{P}_{XZ} P X Z , independent of D i n \mathcal{D}_{\mathrm{in}} D in , D o u t \mathcal{D}_{\mathrm{out}} D out and of the previously drawn batches. We then employ a plug-in estimator for ∇ F ( ω ) \nabla\mathcal{F}(\omega) ∇ F ( ω ) obtained by replacing the population expectation in Equation (2) with the empirical average over B g \mathcal{B}_{g} B g :
∇ F ^ ( ω ) = − 1 n g ∑ l = 1 n g ∂ ω g ω ( z l ) ( − h ^ ω ( x l ) + b ^ ( x l ) ) , (4) \widehat{\nabla\mathcal{F}}(\omega)=-\frac{1}{n_g}\sum_{l=1}^{n_g}\partial_{\omega}g_{\omega}(z_l)\left(-\widehat{h}_{\omega}(x_l)+\widehat{b}(x_l)\right), \tag*{(4)} ∇ F ( ω ) = − n g 1 l = 1 ∑ n g ∂ ω g ω ( z l ) ( − h ω ( x l ) + b ( x l ) ) , (4)
where we approximate a ω ⋆ a_{\omega}^{\star} a ω ⋆ by its finite-sample estimate of the form − h ^ ω + b ^ -\widehat{h}_{\omega}+\widehat{b} − h ω + b since a ω ⋆ = − h ω ⋆ + b ⋆ a_{\omega}^{\star}=-h_{\omega}^{\star}+b^{\star} a ω ⋆ = − h ω ⋆ + b ⋆ .
Algorithm. We use the plug-in estimator ∇ F ^ \widehat{\nabla\mathcal{F}} ∇ F defined in Equation (4) to drive a gradient-based bilevel algorithm that approximately solves (OBO). Starting from ω 0 ∈ Ω \omega_{0}\in\Omega ω 0 ∈ Ω , we first compute b ^ \widehat{b} b from D o u t \mathcal{D}_{\mathrm{out}} D out once and for all, since it does not depend on the outer variable. At each outer iteration k = 0 , … , K − 1 k=0,\ldots,K-1 k = 0 , … , K − 1 , we then re-estimate h ^ ω k \widehat{h}_{\omega_k} h ω k from D i n \mathcal{D}_{\mathrm{in}} D in at the current outer variable, draw a fresh mini-batch B g k \mathcal{B}_{g}^{k} B g k of fixed size n g n_g n g , independent of the previous ones, and update ω k + 1 = Π Ω ( ω k − η ∇ F ^ ( ω k ) ) \omega_{k+1}=\Pi_{\Omega}(\omega_k-\eta\widehat{\nabla\mathcal{F}}(\omega_k)) ω k + 1 = Π Ω ( ω k − η ∇ F ( ω k )) , where η > 0 \eta>0 η > 0 is the stepsize and Π Ω \Pi_{\Omega} Π Ω is the Euclidean projection onto the feasible set Ω \Omega Ω . Only the mini-batches are redrawn along the iterations: D i n \mathcal{D}_{\mathrm{in}} D in and D o u t \mathcal{D}_{\mathrm{out}} D out are sampled once and reused at every ω k \omega_k ω k , so that h ^ ω \widehat{h}_{\omega} h ω is required to be accurate uniformly over Ω \Omega Ω . Since h ^ ω \widehat{h}_{\omega} h ω and b ^ \widehat{b} b are finite-sample approximations
of h ω ⋆ h_\omega^\star h ω ⋆ and b ⋆ b^\star b ⋆ , the resulting ∇ F ^ \widehat{\nabla F} ∇ F is a biased estimator of ∇ F \nabla F ∇ F . As the iterates are constrained to Ω \Omega Ω , we refer to this scheme as projected mini-batch biased stochastic gradient descent (SGD) .
Remark 2.2 (Full-batch variant). Instead of drawing a fresh mini-batch at each outer iteration, one can draw a single dataset once and use all of it at every iteration. Since the iterates then depend on this dataset, the analysis requires the hypergradient estimate to be accurate simultaneously for all ω ∈ Ω \omega\in\Omega ω ∈ Ω , and not only at the current iterate. Such a uniform control can be obtained through a covering argument over Ω \Omega Ω , but it requires a dataset substantially larger than a single mini-batch, and thus full-batch gradient evaluations, which are costly. As both approaches require a comparable number of outer iterations, and each iteration processes its own batch, the full-batch variant is computationally more expensive than Algorithm 1.
Remark 2.3 (Computational benefit of the squared loss setting). In general functional bilevel problems, the adjoint a ω ⋆ a_\omega^\star a ω ⋆ is defined as the minimizer of a quadratic problem depending on ω \omega ω . Hence, both a ω ⋆ a_\omega^\star a ω ⋆ and h ω ⋆ h_\omega^\star h ω ⋆ must be re-estimated at each outer iteration [78 , 33 ] . Our setting simplifies this step considerably. Indeed, in the estimator − h ^ ω + b ^ -\widehat{h}_\omega+\widehat{b} − h ω + b of a ω ⋆ a_\omega^\star a ω ⋆ , only h ^ ω \widehat{h}_\omega h ω depends on ω \omega ω , while b ^ \widehat{b} b can be estimated once and kept fixed throughout the optimization. This reduces the computational cost of the algorithm: each outer iteration involves a single re-estimation of h ^ ω k \widehat{h}_{\omega_k} h ω k , n g n_g n g evaluations of the summands in Equation (4) and one projection onto Ω \Omega Ω , so that K K K outer iterations require K + 1 K+1 K + 1 functional estimations instead of 2 K 2K 2 K in the general case. Among these operations, the dominant one is the re-estimation of h ^ ω k \widehat{h}_{\omega_k} h ω k , the projection being inexpensive for simple sets such as Euclidean balls or boxes.
General convergence result for the projected mini-batch biased SGD
Since F F F is nonconvex and the iterates are constrained to Ω \Omega Ω , we quantify the convergence of the algorithm through the gradient mapping [14 Definition 10.5] , defined for any λ > 0 \lambda>0 λ > 0 by:
G λ ( ω ) : = λ − 1 ( ω − Π Ω ( ω − λ ∇ F ( ω ) ) ) , G_\lambda(\omega) := \lambda^{-1}\left(\omega-\Pi_\Omega\left(\omega-\lambda\nabla F(\omega)\right)\right), G λ ( ω ) := λ − 1 ( ω − Π Ω ( ω − λ ∇ F ( ω ) ) ) ,
and adopt ∥ G λ ( ω ) ∥ \lVert G_\lambda(\omega)\rVert ∥ G λ ( ω )∥ as our near-stationarity measure for (OBO). This quantity measures the displacement produced by a single exact projected gradient step at stepsize λ \lambda λ , and vanishes precisely at the stationary points of (OBO). We now state our main convergence result for Algorithm 1 when using general estimators h ^ ω \widehat{h}_\omega h ω and b ^ \widehat{b} b .
Theorem 2.4 (Convergence guarantee of projected mini-batch biased SGD). Assume A A A and B B B , and fix ω 0 ∈ Ω \omega_0 \in\Omega ω 0 ∈ Ω . Let Δ F : = F ( ω 0 ) − F ⋆ \Delta_F := F(\omega_0)-F^\star Δ F := F ( ω 0 ) − F ⋆ with F ⋆ : = min ω ∈ Ω F ( ω ) F^\star:= \min_{\omega\in\Omega}F(\omega) F ⋆ := min ω ∈ Ω F ( ω ) and { ω k } k ≥ 0 \{\omega_k\}_{k\geq0} { ω k } k ≥ 0 be generated by ω k + 1 = Π Ω ( ω k − η ∇ F ( ω k ) ) \omega_{k+1}=\Pi_\Omega(\omega_k-\eta\nabla F(\omega_k)) ω k + 1 = Π Ω ( ω k − η ∇ F ( ω k )) with stepsize 0 < η ≤ 1 2 L 0<\eta\leq\frac{1}{2L} 0 < η ≤ 2 L 1 , where L L L is the smoothness constant of F F F provided in Proposition 2.1. For any K ≥ 1 K\geq1 K ≥ 1 , the iterates { ω k } k ≥ 0 \{\omega_k\}_{k\geq0} { ω k } k ≥ 0 satisfy:
1 K ∑ k = 0 K − 1 E [ ∥ G 1 / L ( ω k ) ∥ 2 ] ≤ c [ Δ F η K + 4 G 2 ( e h + e b ) + 4 L G 2 η n g ( ( D + B ) 2 + 2 ( e h + e b ) ) ] , \frac{1}{K}\sum_{k=0}^{K-1}\mathbb{E}\left[\left\lVert G_{1/L}(\omega_k)\right\rVert^2\right]\leq c\left[\frac{\Delta_F}{\eta K}+4G^2(e_h+e_b)+\frac{4LG^2\eta}{n_g}\left((D+B)^2+2(e_h+e_b)\right)\right], K 1 k = 0 ∑ K − 1 E [ G 1/ L ( ω k ) 2 ] ≤ c [ η K Δ F + 4 G 2 ( e h + e b ) + n g 4 L G 2 η ( ( D + B ) 2 + 2 ( e h + e b ) ) ] , where c = 18 + 12 2 c=18+12\sqrt{2} c = 18 + 12 2 , e h = E [ sup ω ∈ Ω ∥ h ^ ω − h ω ⋆ ∥ L 2 ( χ ) 2 ] e_h=\mathbb{E}\left[\sup_{\omega\in\Omega}\left\lVert\widehat{h}_\omega-h_\omega^\star\right\rVert_{L^2(\chi)}^2\right] e h = E [ sup ω ∈ Ω h ω − h ω ⋆ L 2 ( χ ) 2 ] , e b = E [ ∥ b ^ − b ⋆ ∥ L 2 ( χ ) 2 ] e_b=\mathbb{E}\left[\left\lVert\widehat{b}-b^\star\right\rVert_{L^2(\chi)}^2\right] e b = E [ b − b ⋆ L 2 ( χ ) 2 ] , and B B B , D D D , G G G are the constants of Assumptions A A A and B B B . The expectation is over all sources of randomness, namely the samples D i n D_{\mathrm{in}} D in and D o u t D_{\mathrm{out}} D out used to construct h ^ ω \widehat{h}_\omega h ω and b ^ \widehat{b} b , and the mini-batches { B g k } k ≥ 0 \{B_g^k\}_{k\geq0} { B g k } k ≥ 0 .
The proof of Theorem 2.4 is given in Appendix § 11 . It proceeds by controlling the bias and the variance of the hypergradient estimator in terms of e h e_h e h and e b e_b e b (Lemmas C.2 and C.3), combining these bounds with a one-step descent inequality along the projected update (Lemma C.4 ), and telescoping over the iterations. The descent inequality is established for the Moreau envelope of F F F , whose gradient is equivalent to G 1 / L G_{1/L} G 1/ L up to absolute constants in our case [30 Section 2.2] . Note that the same bound holds for E [ ∥ G 1 / L ( ω k ⋆ ) ∥ 2 ] \mathbb{E}[\lVert G_{1/L}(\omega_{k^\star})\rVert^2] E [∥ G 1/ L ( ω k ⋆ ) ∥ 2 ] , where k ⋆ k^\star k ⋆ is drawn uniformly at random in { 0 , … , K − 1 } \{0,\ldots,K-1\} { 0 , … , K − 1 } , so that the guarantee applies to an iterate Algorithm 1 can return.
Error decomposition. The bound decomposes into three terms of distinct origins. The first is the usual optimization error, and vanishes as the number of iterations K K K grows. The second is a bias term: it quantifies the two-stage error incurred by replacing the adjoint a ω ⋆ a_\omega^\star a ω ⋆ with − h ^ ω + b ^ -\widehat{h}_\omega+\widehat{b} − h ω + b , and is unaffected by K K K , η \eta η and n g n_g n g . The third is a variance term, reflecting the sampling noise of the empirical average over the mini-batch. For a fixed batch size n g n_g n g , it is proportional to η \eta η and can therefore be reduced by decreasing the stepsize. The stepsize thus trades off the first and third terms, while leaving the bias unchanged. Choosing η \eta η achieving the best trade-off makes the algorithmic part of the error decay at the rate C Δ F / ( K n g ) \sqrt{C\Delta_F/(Kn_g)} C Δ F / ( K n g ) (Corollary C.6).
Towards explicit rates for e h e_h e h and e b e_b e b . Theorem 2.4 is stated in terms of the generalization errors e h e_h e h and e b e_b e b alone, and is therefore agnostic to how h ^ ω \hat{h}_\omega h ^ ω and b ^ \hat{b} b ^ are obtained. These errors are statistical in nature and decrease only as the two regression estimates improve, through larger sample sizes n i n n_{in} n in and n o u t n_{out} n o u t or hypothesis spaces better suited to h ω ∗ h_\omega^* h ω ∗ and b ∗ b^* b ∗ . We make them explicit for overparameterized two-layer ReLU neural networks trained by gradient flow, whose dynamics we describe in Equation (6) and whose generalization bounds we derive in the NTK regime in Proposition 3.1.
Explicit statistical rate for Algorithm 1 in the NTK regime
In this section, we build estimators for both targets as truncated neural networks trained by gradient flow, derive their finite-sample generalization rates in the NTK regime, and deduce the sample complexity of Algorithm 1.
Neural network estimators: training dynamics and truncation
Architecture. We construct estimators h ^ ω \hat{h}_\omega h ^ ω and b ^ \hat{b} b ^ of the target functions h ω ∗ h_\omega^* h ω ∗ and b ∗ b^* b ∗ using overparameterized neural networks with a common architecture: two-layer ReLU networks of the form:
f θ ( x ) = 1 m ∑ i = 1 m v i σ ( u i T x + c i ) + a , (5) f_\theta(x) = \frac{1}{\sqrt{m}}\sum_{i=1}^{m} v_i \sigma\left(u_i^\mathsf{T} x + c_i\right) + a, \tag*{(5)} f θ ( x ) = m 1 i = 1 ∑ m v i σ ( u i T x + c i ) + a , (5)
where σ ( z ) = max ( 0 , z ) \sigma(z) = \max(0,z) σ ( z ) = max ( 0 , z ) is the ReLU activation function, m m m is the network’s (even) width, and θ = ( v , u , c , a ) \theta= (v,u,c,a) θ = ( v , u , c , a ) collects the parameters. The first-layer weights are u ∈ R m × p u \in\mathbb{R}^{m \times p} u ∈ R m × p with corresponding biases c ∈ R m c \in\mathbb{R}^m c ∈ R m , the second-layer weights are v ∈ R m v \in\mathbb{R}^m v ∈ R m , and a ∈ R a \in\mathbb{R} a ∈ R is the output bias. The width m m m and the parameters θ \theta θ above are generic: we write m h m_h m h and θ h \theta^h θ h for the width and parameters of the network approximating h ω ∗ h_\omega^* h ω ∗ , and m b m_b m b and θ b \theta^b θ b for those of the network approximating b ∗ b^* b ∗ . The scaling 1 / m 1/\sqrt{m} 1/ m allows reaching the NTK regime [53 ] as m m m grows and under the initialization discussed next.
Initialization. We set the initial output bias at a = 0 a = 0 a = 0 , while the rest of the parameters of both networks are drawn, independently of each other and of the data, from a standard Gaussian distribution using a mirrored initialization [58 , 62 ] , to ensure the output is exactly zero initially. Specifically, for 1 ≤ i ≤ m / 2 1 \leq i \leq m/2 1 ≤ i ≤ m /2 , we draw u i u_i u i , c i c_i c i and v i v_i v i independently from a standard Gaussian and set u i + m / 2 = u i u_{i+m/2} = u_i u i + m /2 = u i , c i + m / 2 = c i c_{i+m/2} = c_i c i + m /2 = c i and v i + m / 2 = − v i v_{i+m/2} = -v_i v i + m /2 = − v i , so that the contributions of neurons i i i and i + m / 2 i+m/2 i + m /2 cancel. For the network approximating h ω ∗ h_\omega^* h ω ∗ , the parameters are drawn once and shared by all ω ∈ Ω \omega\in\Omega ω ∈ Ω , so that their dependence on ω \omega ω arises only through training.
Gradient flow training. While neural networks are commonly trained with discrete-time gradient-based algorithms, we consider here their continuous-time limit, gradient flow , for theoretical tractability [35 ] . Starting from the mirrored initialization, and for a given ω ∈ Ω \omega\in\Omega ω ∈ Ω , the parameters of both networks evolve in time according to the following ordinary differential equations:
d θ t , ω h d t = − ∂ θ L ^ i n ( ω , θ t h ) , d θ t b d t = − ∂ θ L ^ o u t ( θ t b ) . (6) \frac{\mathrm{d}\theta^h_{t,\omega}}{\mathrm{d}t} = -\partial_\theta\widehat{\mathcal{L}}_{in}(\omega,\theta^h_t), \qquad\frac{\mathrm{d}\theta^b_t}{\mathrm{d}t} = -\partial_\theta\widehat{\mathcal{L}}_{out}(\theta^b_t). \tag*{(6)} d t d θ t , ω h = − ∂ θ L in ( ω , θ t h ) , d t d θ t b = − ∂ θ L o u t ( θ t b ) . (6)
where we defined the training objectives in terms of the network parameters: L ^ i n ( ω , θ h ) : = L ^ i n ( ω , f θ h ) \widehat{\mathcal{L}}_{in}(\omega,\theta^h) := \widehat{\mathcal{L}}_{in}(\omega,f_{\theta^h}) L in ( ω , θ h ) := L in ( ω , f θ h ) and L ^ o u t ( θ b ) : = L ^ o u t ( f θ b ) \widehat{\mathcal{L}}_{out}(\theta^b) := \widehat{\mathcal{L}}_{out}(f_{\theta^b}) L o u t ( θ b ) := L o u t ( f θ b ) , while ∂ θ \partial_\theta ∂ θ denotes the gradient with respect to the network parameters. We next introduce the NTK, which is needed later to define our final estimators. As shown in [53 ] , the evolution of the predictor functions f θ t h f_{\theta^h_t} f θ t h and f θ t b f_{\theta^b_t} f θ t b is governed by the finite-width NTKs, defined for any x , x ′ ∈ X x,x' \in\mathcal{X} x , x ′ ∈ X by:
K t m h ( x , x ′ ) = ( ∂ θ f θ t h ( x ) ) T ∂ θ f θ t h ( x ′ ) , K t m b ( x , x ′ ) = ( ∂ θ f θ t b ( x ) ) T ∂ θ f θ t b ( x ′ ) . K^{m_h}_t(x,x') = \left(\partial_\theta f_{\theta^h_t}(x)\right)^\mathsf{T}\partial_\theta f_{\theta^h_t}(x'), \qquad K^{m_b}_t(x,x') = \left(\partial_\theta f_{\theta^b_t}(x)\right)^\mathsf{T}\partial_\theta f_{\theta^b_t}(x'). K t m h ( x , x ′ ) = ( ∂ θ f θ t h ( x ) ) T ∂ θ f θ t h ( x ′ ) , K t m b ( x , x ′ ) = ( ∂ θ f θ t b ( x ) ) T ∂ θ f θ t b ( x ′ ) .
As the widths m h m_h m h and m b m_b m b increase, the network’s scaling and initialization choices ensure that both kernels K t m h K^{m_h}_t K t m h and K t m b K^{m_b}_t K t m b converge to a common deterministic, time-invariant infinite-width limit [53 , 4 , 59 ] , denoted here by K N T K K^{\mathrm{NTK}} K NTK , whose explicit expression is given in Equation (38) of Appendix § 13 .§ 2 . Consequently, the predictor’s dynamics converge to those of a kernel regression trained by gradient flow, which we refer to as NTK regression .
Early stopping. Since we consider an overparameterized setting in which the network’s width can greatly exceed the number of training samples, the predictors f θ h t f_{\theta_h}^{t} f θ h t and f θ b t f_{\theta_b}^{t} f θ b t may interpolate the training data as t → ∞ t \to\infty t → ∞ . Although interpolation can still yield good generalization performance [47 , 16 ] , precisely characterizing generalization in this regime is beyond the scope of this work. Instead, we adopt an early stopping strategy, halting the gradient flows at carefully chosen stopping times t o p h t_{\mathrm{op}_h} t op h and t o p b t_{\mathrm{op}_b} t op b , which allows us to leverage techniques similar to those developed in [62 ] . Early stopping acts as an implicit regularizer [82 , 95 ] , with the stopping time playing the role of a regularization parameter that controls the bias-variance trade-off and thereby improves generalization.
Truncated estimators. We could, in principle, use the early-stopped predictors as our final estimators. However, obtaining an explicit finite-sample convergence guarantee from Theorem 2.4 requires bounds on their expected generalization errors. Existing results, such as [62 Proposition 13] , are not directly applicable: they provide high-probability bounds, holding with probability at least 1 − δ 1-\delta 1 − δ , under width requirements that depend on the training samples and on the confidence level δ \delta δ , preventing the usual integration over δ \delta δ to obtain an expectation bound for a fixed-width network. Instead, one would need to (i) address the dependence of the required width on the training data and (ii) control rare events on which the early-stopped predictors fall outside the high-probability guarantee and may potentially take arbitrarily large values. The former is addressed through a finer high-probability bound in Proposition E.22, which makes the required width a function of the sample size and δ \delta δ only, while the latter is addressed by instead considering truncated estimators:
h ^ ω = f θ t o p h 1 E h , b ^ = f θ t o p b 1 E b , (7) \hat{h}_{\omega}=f_{\theta_{t_{\mathrm{op}_h}}}\mathbb{1}_{\mathcal{E}_h},\qquad\hat{b}=f_{\theta_{t_{\mathrm{op}_b}}}\mathbb{1}_{\mathcal{E}_b}, \tag*{(7)} h ^ ω = f θ t op h 1 E h , b ^ = f θ t op b 1 E b , (7)
where E h \mathcal{E}^{h} E h and E b \mathcal{E}^{b} E b depend only on the training data and initialization. These events are constructed to occur with high probability and ensure that, whenever they occur, the early-stopped predictors satisfy the required probabilistic bounds. The truncated estimators therefore coincide with the early-stopped networks on truncation events, inheriting their high-probability guarantees, and are set to zero on the low-probability complements, where no precise control over the early-stopped networks is available.
Truncation events. We construct each event E h \mathcal{E}^{h} E h and E b \mathcal{E}^{b} E b as the intersection of four explicit high-probability events depending on the data and initialization used to estimate h ω ∗ h_{\omega}^{*} h ω ∗ and b ∗ b^{*} b ∗ . Since these events have the same structure for both estimators, we drop the superscript h / b h/b h / b for simplicity and state them using the notation of Equation (5) , for a width-m m m network trained on inputs { x i } i = 1 n \{x_i\}_{i=1}^{n} { x i } i = 1 n . The first three events depend only on the random initialization θ 0 \theta_0 θ 0 , whereas the fourth depends only on the training inputs. For conciseness, we provide only a high-level description here and refer the reader to Appendix § 13 .8.2 for the detailed definitions and discussion.
Small initial weights event (E θ 0 \mathcal{E}_{\theta_0} E θ 0 ): the event when initial weights satisfy: ∥ θ 0 ∥ ∞ ≤ 3 ln m \lVert\theta_0\rVert_{\infty} \le\sqrt{3\ln m} ∥ θ 0 ∥ ∞ ≤ 3 ln m .
Initial pre-activation margin event (R \mathcal{R} R ): the event when, uniformly over the input domain, only a small fraction O ( m κ ) \mathcal{O}(m^{\kappa}) O ( m κ ) of initial pre-activations are within a prescribed margin of 0 0 0 , for κ < 1 \kappa<1 κ < 1 .
Initial NTK concentration event (C \mathcal{C} C ): the event when, uniformly over a grid on the input domain, the finite-width NTK at initialization is within order ln m / m \sqrt{\ln m/m} ln m / m of its infinite-width limit.
NTK non-degeneracy event (E λ 0 \mathcal{E}_{\lambda_0} E λ 0 ): the event when the Gram matrix of the infinite-width NTK on training data has its smallest eigenvalue λ 0 \lambda_0 λ 0 bounded below by a prescribed quantity λ ⋆ ( n , δ ) > 0 \lambda_{\star}(n,\delta)>0 λ ⋆ ( n , δ ) > 0 .
Under ( E θ 0 ) (\mathcal{E}_{\theta_0}) ( E θ 0 ) , it is possible to show that the gradients remain bounded along training, which helps keep the parameters in a controlled neighborhood of θ 0 \theta_0 θ 0 . Under ( R ) (\mathcal{R}) ( R ) , it can be shown that the signs of their pre-activations cannot change unless the parameters move far. This property is essential in proving that wide neural networks trained in the NTK regime remain close to their initialization [53 , 4 ] . ( C ) (\mathcal{C}) ( C ) is expected for large m m m , since the finite-width NTK is an average over the m m m neurons of terms that are random at initialization and whose mean is K N T K K^{\mathrm{NTK}} K NTK . Finally, ( E λ 0 ) (\mathcal{E}_{\lambda_0}) ( E λ 0 ) guarantees that the gradient flow of the NTK regression converges on the training data.
Finite-sample generalization bounds
To state our generalization rates, we need a source condition , i.e., a notion of regularity for the target functions h ω ∗ h_{\omega}^{*} h ω ∗ and b ∗ b^{*} b ∗ . A natural way to quantify this regularity is through how well these targets can be approximated by elements of the RKHS H N T K \mathcal{H}^{\mathrm{NTK}} H NTK associated with the limiting kernel K N T K K^{\mathrm{NTK}} K NTK . Indeed, in the infinite-width limit, the predictors used to approximate the targets are elements of H N T K \mathcal{H}^{\mathrm{NTK}} H NTK . Rather than requiring these targets to also belong to H N T K \mathcal{H}^{\mathrm{NTK}} H NTK , we will instead assume they lie in an interpolation space between L 2 ( X ) L^{2}(\mathcal{X}) L 2 ( X ) and H N T K \mathcal{H}^{\mathrm{NTK}} H NTK , as this allows capturing intermediate regularity properties. To formally define these spaces, we introduce the integral operator T : L 2 ( X ) → L 2 ( X ) T:L^2(\mathcal{X})\to L^2(\mathcal{X}) T : L 2 ( X ) → L 2 ( X ) associated with K N T K K^{\mathrm{NTK}} K NTK , given for every f ∈ L 2 ( X ) f\in L^2(\mathcal{X}) f ∈ L 2 ( X ) and x ∈ X x\in\mathcal{X} x ∈ X by:
( T f ) ( x ) = E X ∼ P X [ K N T K ( x , X ) f ( X ) ] . (Tf)(x)=\mathbb{E}_{X\sim P_X}\left[K^{\mathrm{NTK}}(x,X)f(X)\right]. ( T f ) ( x ) = E X ∼ P X [ K NTK ( x , X ) f ( X ) ] .
Since K N T K K^{\mathrm{NTK}} K NTK is a Mercer kernel, T T T is compact, self-adjoint, and positive semi-definite. Hence, by the spectral theorem [86 Theorem A.5.13] , T T T admits a spectral decomposition with non-increasing eigenvalues ( λ i ) i ≥ 1 (\lambda_i)_{i\geq1} ( λ i ) i ≥ 1 and corresponding orthonormal eigenfunctions ( e i ) i ≥ 1 (e_i)_{i\geq1} ( e i ) i ≥ 1 forming a basis of L 2 ( X ) L^2(\mathcal{X}) L 2 ( X ) . We then define the interpolation space [ H N T K ] s [\mathcal{H}^{\mathrm{NTK}}]_s [ H NTK ] s with parameter s ≥ 0 s\geq0 s ≥ 0 as the subspace of functions f ∈ L 2 ( X ) f\in L^2(\mathcal{X}) f ∈ L 2 ( X ) such that ∥ f ∥ [ H N T K ] s < ∞ \|f\|_{[\mathcal{H}^{\mathrm{NTK}}]_s}<\infty ∥ f ∥ [ H NTK ] s < ∞ , where ∥ ⋅ ∥ [ H N T K ] s \|\cdot\|_{[\mathcal{H}^{\mathrm{NTK}}]_s} ∥ ⋅ ∥ [ H NTK ] s is the interpolation norm defined as the sum of the squared projections onto each e i e_i e i , weighted by λ i − s \lambda_i^{-s} λ i − s , i.e., ∥ f ∥ [ H N T K ] s 2 = ∑ i ≥ 1 λ i − s ⟨ f , e i ⟩ L 2 ( X ) 2 \|f\|^2_{[\mathcal{H}^{\mathrm{NTK}}]_s}=\sum_{i\geq1}\lambda_i^{-s}\langle f,e_i\rangle_{L^2(\mathcal{X})}^2 ∥ f ∥ [ H NTK ] s 2 = ∑ i ≥ 1 λ i − s ⟨ f , e i ⟩ L 2 ( X ) 2 .2
We now give the source conditions needed to establish generalization bounds.
Assumption E. (Source condition for h ω ∗ h^*_\omega h ω ∗ ) There exist s h , R h > 0 s_h,R_h>0 s h , R h > 0 such that ∥ h ω ∗ ∥ [ H N T K ] s h ≤ R h \|h^*_\omega\|_{[\mathcal{H}^{\mathrm{NTK}}]_{s_h}}\leq R_h ∥ h ω ∗ ∥ [ H NTK ] s h ≤ R h for every ω ∈ Ω \omega\in\Omega ω ∈ Ω , hence in particular at every iterate of Algorithm 1.
Assumption F. (Source condition for b ∗ b^* b ∗ ) There exist s b , R b > 0 s_b,R_b>0 s b , R b > 0 such that ∥ b ∗ ∥ [ H N T K ] s b ≤ R b \|b^*\|_{[\mathcal{H}^{\mathrm{NTK}}]_{s_b}}\leq R_b ∥ b ∗ ∥ [ H NTK ] s b ≤ R b .
Assumptions E and F are standard source conditions in kernel regression [63 , 23 , 39 ] , with s h s_h s h and s b s_b s b controlling the regularity of h ω ∗ h^*_\omega h ω ∗ and b ∗ b^* b ∗ relative to H N T K \mathcal{H}^{\mathrm{NTK}} H NTK : larger values correspond to smoother, easier to learn targets. In addition, Assumption E requires this regularity bound to hold uniformly over ω \omega ω , ensuring that the difficulty of estimating h ω ∗ h^*_\omega h ω ∗ remains controlled throughout the optimization. When X \mathcal{X} X is a bounded Lipschitz domain, such as a ball or a cube, we show in Appendix § 15 that [ H N T K ] s [\mathcal{H}^{\mathrm{NTK}}]_s [ H NTK ] s coincides with the Sobolev space H s ( p + 1 ) 2 ( X ) H^{\frac{s(p+1)}{2}}(\mathcal{X}) H 2 s ( p + 1 ) ( X ) for 0 ≤ s ≤ 1 0\leq s\leq1 0 ≤ s ≤ 1 . Thus, when s h , s b ≤ 1 s_h,s_b\leq1 s h , s b ≤ 1 , Assumptions E and F can be expressed as Sobolev regularity conditions, uniformly over ω ∈ Ω \omega\in\Omega ω ∈ Ω for h ω ∗ h^*_\omega h ω ∗ .
We can now derive generalization bounds for the estimators h ^ ω \hat{h}_\omega h ^ ω and b ^ \hat{b} b ^ from § 3.1 .
Proposition 3.1. (Expected generalization bounds). Assume p ≥ 2 p\geq2 p ≥ 2 and let Ω ⊂ R d \Omega\subset\mathbb{R}^d Ω ⊂ R d be compact. Let Assumptions A to F hold with s h , s b ∈ ( p + 1 p , 2 ] s_h,s_b\in\left(\frac{p+1}{p},2\right] s h , s b ∈ ( p p + 1 , 2 ] , and choose the stopping times t o p h = n i n γ h t_{\mathrm{op}_h}=n_{\mathrm{in}}^{\gamma_h} t op h = n in γ h and t o p b = n o u t γ b t_{\mathrm{op}_b}=n_{\mathrm{out}}^{\gamma_b} t op b = n out γ b , with γ h = p + 1 s h ( p + 1 ) + p \gamma_h=\frac{p+1}{s_h(p+1)+p} γ h = s h ( p + 1 ) + p p + 1 and γ b = p + 1 s b ( p + 1 ) + p \gamma_b=\frac{p+1}{s_b(p+1)+p} γ b = s b ( p + 1 ) + p p + 1 . There exist two polynomials poly h \operatorname{poly}_h poly h and poly b \operatorname{poly}_b poly b such that, when n i n n_{\mathrm{in}} n in and n o u t n_{\mathrm{out}} n out are sufficiently large and the widths satisfy m h ≥ poly h ( n i n , ln n i n ) m_h\geq\operatorname{poly}_h(n_{\mathrm{in}},\ln n_{\mathrm{in}}) m h ≥ poly h ( n in , ln n in ) and m b ≥ poly b ( n o u t , ln n o u t ) m_b\geq\operatorname{poly}_b(n_{\mathrm{out}},\ln n_{\mathrm{out}}) m b ≥ poly b ( n out , ln n out ) , the truncated predictors of Equation (7) satisfy:
E [ sup ω ∈ Ω ∥ h ^ ω − h ω ∗ ∥ L 2 ( X ) 2 ] ≲ d 4 ( ln n i n ) 4 n i n − s h γ h , E [ ∥ b ^ − b ∗ ∥ L 2 ( X ) 2 ] ≲ ( ln n o u t ) 4 n o u t − s b γ b , \mathbb{E}\left[\sup_{\omega\in\Omega}\left\|\hat{h}_\omega-h^*_\omega\right\|_{L^2(\mathcal{X})}^2\right]\lesssim d^4(\ln n_{\mathrm{in}})^4n_{\mathrm{in}}^{-s_h\gamma_h},\qquad
\mathbb{E}\left[\left\|\hat{b}-b^*\right\|_{L^2(\mathcal{X})}^2\right]\lesssim(\ln n_{\mathrm{out}})^4n_{\mathrm{out}}^{-s_b\gamma_b}, E [ ω ∈ Ω sup h ^ ω − h ω ∗ L 2 ( X ) 2 ] ≲ d 4 ( ln n in ) 4 n in − s h γ h , E [ b ^ − b ∗ L 2 ( X ) 2 ] ≲ ( ln n out ) 4 n out − s b γ b , where ≲ \lesssim ≲ hides constants independent of n i n n_{\mathrm{in}} n in , n o u t n_{\mathrm{out}} n out and d d d , and the expectations are taken over the training samples and the random initialization.
Proposition 3.1 shows that the truncated predictors achieve the minimax-optimal rates of the corresponding regression problems, up to logarithmic factors [19 , 96 ] . We defer the detailed proof, including the explicit dependence of the constants and polynomial coefficients on the problem parameters, to Appendices § 12 .§ 2 and § 13 .8.2. The proof first shows that, with high probability, sufficiently wide networks remain uniformly close to their NTK regression counterparts, and then relies on truncation to turn these high-probability guarantees into bounds in expectation. Compared with existing results in the NTK literature [62 , 58 , 24 ] , the main challenge is to obtain a bound that holds uniformly over ω ∈ Ω \omega\in\Omega ω ∈ Ω . This requires deriving bounds with constants independent of ω \omega ω and combining uniformity argument over Ω \Omega Ω with the Lipschitz continuity of the NTK regression error in ω \omega ω (Lemma D.3 ).
The above bounds are precisely the quantities ϵ h \epsilon_h ϵ h and ϵ b \epsilon_b ϵ b arising in Theorem 2.4 . Thus combining both the convergence result in Theorem 2.4 with the generalization bounds in Proposition 3.1 yields the following explicit convergence result for the iterates produced by Algorithm 1 when using the truncated estimators in Equation (7) .
Corollary 3.2. (Sample complexity of Algorithm 1). Let Assumptions A to F hold and consider the iterates ω k \omega_k ω k obtained from Algorithm 1 using stepsize 0 < η ≤ 1 2 L 0<\eta\leq\frac{1}{2\mathcal{L}} 0 < η ≤ 2 L 1 , gradient estimator in Equation (4) and truncated predictors in Equation (7) with width parameters, sample sizes and stopping times as in Proposition 3.1. Then, for any K ≥ 1 K\geq1 K ≥ 1 , the iterates { ω k } k ≥ 0 \{\omega_k\}_{k\geq0} { ω k } k ≥ 0 satisfy:
1 K ∑ k = 0 K − 1 E [ ∥ g 1 / L ( ω k ) ∥ 2 ] ≲ Δ F η K + L G 2 η ( D + B ) 2 n g + G 2 ( d 4 ( ln n i n ) 4 n i n − s h γ h + ( ln n o u t ) 4 n o u t − s b γ b ) . \frac{1}{K}\sum_{k=0}^{K-1}\mathbb{E}\left[\left\|g_{1/\mathcal{L}}(\omega_k)\right\|^2\right]\lesssim\frac{\Delta_F}{\eta K}+\frac{\mathcal{L}G^2\eta(D+B)^2}{n_g}+G^2\left(d^4(\ln n_{\mathrm{in}})^4n_{\mathrm{in}}^{-s_h\gamma_h}+(\ln n_{\mathrm{out}})^4n_{\mathrm{out}}^{-s_b\gamma_b}\right). K 1 k = 0 ∑ K − 1 E [ g 1/ L ( ω k ) 2 ] ≲ η K Δ F + n g L G 2 η ( D + B ) 2 + G 2 ( d 4 ( ln n in ) 4 n in − s h γ h + ( ln n out ) 4 n out − s b γ b ) .
where ≲ \lesssim ≲ hides constants which are independent of K K K , η \eta η , n g n_g n g , n i n n_{in} n in , n o u t n_{out} n o u t and d d d .
In Corollary 3.2, the statistical error is dominated by the slower of the two rates n i n − s h γ h n_{in}^{-s_h\gamma_h} n in − s h γ h and n o u t − s b γ b n_{out}^{-s_b\gamma_b} n o u t − s b γ b . This suggests balancing the two sample sizes so that n i n − s h γ h ≈ n o u t − s b γ b n_{in}^{-s_h\gamma_h} \approx n_{out}^{-s_b\gamma_b} n in − s h γ h ≈ n o u t − s b γ b up to constant and logarithmic factors, ensuring that neither level becomes a bottleneck. Moreover, since the exponents s h γ h s_h\gamma_h s h γ h and s b γ b s_b\gamma_b s b γ b increase with s h s_h s h and s b s_b s b , respectively, the level whose target has the larger smoothness parameter needs fewer samples to reach the same accuracy.
Related work
El Khoury et al. [33 ] and Chen et al. [25 ] study generalization in bilevel nonparametric settings: the former uses full-batch gradient descent to optimize the inner level over an RKHS, while the latter relies on mean-field Langevin dynamics for two-stage least squares. Both works assume a fixed regularization parameter to control the bias-variance trade-off, leaving open the question of how the optimal choice of the regularization, as a function of the sample sizes, would affect sample complexity and whether it could lead to faster rates. In contrast, in our setting, this trade-off is achieved by the early stopping time, which we select as a function of the sample sizes, yielding explicit finite-sample bounds that quantify the sample complexity of the problem. Relatedly, Patel and Vlatakis-Gkaragkounis [76 ] provide convergence guarantees for training overparameterized neural networks in min-max (zero-sum) games in the NTK regime, which can be viewed as a particular bilevel structure. Our work differs in that we study bilevel learning and generalization, and quantify how stochastic, approximate NTK training affects hypergradient accuracy and excess risk rather than game-theoretic optimization dynamics. Also, Kim et al. [56 ] analyze the deep feature instrumental variable regression, which shares structural similarities with our bilevel setup. They establish minimax optimal convergence rates when the target function belongs to a Besov space. However, their analysis focuses on statistical aspects, whereas we also account for optimization. Finally, El Khoury et al. [34 ] study the same problem as ours and debias the hypergradient to attain parametric rates, whereas we derive generalization bounds for the plug-in estimator with neural networks in the NTK regime.
Numerical experiments
Empirical validation of the theory in the NTK regime on synthetic data
We empirically validate the convergence bound of Corollary 3.2 on targets whose smoothness is known in closed form. We consider inputs x ∈ [ 0 , 1 ] 2 x \in[0,1]^2 x ∈ [ 0 , 1 ] 2 , outer parameters ω ∈ R 4 \omega\in\mathbb{R}^4 ω ∈ R 4 , and cusp targets h ω ∗ ( x ) = ψ ( x ) T ω h_\omega^*(x)=\psi(x)^\mathsf{T}\omega h ω ∗ ( x ) = ψ ( x ) T ω and b ∗ ( x ) = ∥ x − x 0 ∥ q b b^*(x)=\lVert x-x_0\rVert^{q_b} b ∗ ( x ) = ∥ x − x 0 ∥ q b , where ψ ( x ) : = [ ∥ x − c 1 ∥ q h , … , ∥ x − c 4 ∥ q h ] T \psi(x):=[\lVert x-c_1\rVert^{q_h},\ldots,\lVert x-c_4\rVert^{q_h}]^\mathsf{T} ψ ( x ) := [∥ x − c 1 ∥ q h , … , ∥ x − c 4 ∥ q h ] T , the centers c 1 , … , c 4 c_1,\ldots,c_4 c 1 , … , c 4 form the grid { 1 4 , 3 4 } 2 \{\frac{1}{4},\frac{3}{4}\}^2 { 4 1 , 4 3 } 2 , and x 0 = ( 1 2 , 1 2 ) x_0=(\frac{1}{2},\frac{1}{2}) x 0 = ( 2 1 , 2 1 ) . By [84 Section 2.3.1, Lemma 1] , h ω ∗ h_\omega^* h ω ∗ and b ∗ b^* b ∗ satisfy the source conditions uniformly over ω \omega ω for every s h : = 2 q h + 2 3 s_h:=\frac{2q_h+2}{3} s h := 3 2 q h + 2 and s b : = 2 q b + 2 3 s_b:=\frac{2q_b+2}{3} s b := 3 2 q b + 2 , with q h , q b ∈ { 0.05 , 0.25 , 0.45 } q_h,q_b\in\{0.05,0.25,0.45\} q h , q b ∈ { 0.05 , 0.25 , 0.45 } , giving s ˉ h , s ˉ b ∈ { 0.70 , 0.83 , 0.97 } \bar{s}_h,\bar{s}_b\in\{0.70,0.83,0.97\} s ˉ h , s ˉ b ∈ { 0.70 , 0.83 , 0.97 } and decay rates of 0.51, 0.56, and 0.59, respectively. Both targets are approximated by networks of the form in Equation (5) trained by full-batch gradient descent. We follow Algorithm 1 with hypergradients evaluated on fresh batches of size 200. The sample sizes n i n n_{in} n in and n o u t n_{out} n o u t vary independently from 100 to 25,600. Further details about the setup are given in Appendix § 18 .§ 1 . Figure 1 plots the near-stationarity measure averaged over the outer iterations against n i n n_{in} n in or n o u t n_{out} n o u t , minus a constant C C C collecting the terms of the bound in Corollary 3.2 that are independent of the varied sample size. In panel (a), n i n n_{in} n in varies for s h ∈ { 0.70 , 0.83 , 0.97 } s_h\in\{0.70,0.83,0.97\} s h ∈ { 0.70 , 0.83 , 0.97 } , with n o u t = 25 , 600 n_{out}=25,600 n o u t = 25 , 600 and s b = 0.70 s_b=0.70 s b = 0.70 ; panel (b) swaps the inner and outer roles. The curves follow the slopes predicted by Corollary 3.2, thus validating our theory.
Figure 1. Illustration of Algorithm 1 on synthetic data with targets of known smoothness. Dashed lines show the theoretical slopes. The curves are averaged over 5 runs and displayed on a log scale. Solid lines represent the mean across runs, and shaded regions indicate the 95% confidence intervals.
Insights into feature learning beyond the NTK regime
Our analysis relies on the NTK regime, which assumes fixed features and thus precludes feature learning . For a single network, the initialization scale governs the transition between the NTK and feature learning regimes [91 ] . In the bilevel setting, the inner and outer initializations may jointly determine the regime. We investigate this insight, which lies beyond the scope of our theory, on a synthetic problem in which h ω ∗ h_\omega^* h ω ∗ and b ∗ b^* b ∗ are estimated by two-layer networks of the form in Equation (5) normalized by m m m instead of m \sqrt{m} m to allow feature learning [26 ] , with first-layer initialization variances σ 0 2 \sigma_0^2 σ 0 2 and τ 0 2 \tau_0^2 τ 0 2 , respectively. Figure 2 shows that both the test inner and outer losses are lowest when both initializations are small (feature learning) and degrade as either grows (NTK), indicating a *joint effect of the two initializations on the effective regime. The full setup and additional results, showing that this ranking holds along the outer iterations and across sample sizes, are given in Appendix § 18 .§ 2 .
Figure 2. Test inner (left) and outer (right) losses at the last outer iteration as a function of ( σ 0 2 , τ 0 2 ) (\sigma_0^2, \tau_0^2) ( σ 0 2 , τ 0 2 ) : both losses are lowest when both initializations are small (feature learning) and degrade as either grows large (NTK). The heatmaps are averaged over 5 runs and displayed on a log scale.
Conclusion and perspectives
Summary. In this paper, we provided a general convergence guarantee for inexact bilevel gradient methods when both objectives are least-squares regressions. The resulting bound depends on the estimators of h ω ⋆ h_{\omega}^{\star} h ω ⋆ and b ⋆ b^{\star} b ⋆ only through their uniform generalization errors. In the NTK regime, we derived finite-sample bounds on these errors when both are estimated by overparameterized two-layer ReLU networks trained by gradient flow with early stopping. Plugging these rates into the general bound yields an explicit convergence guarantee for Algorithm 1. Our experimental results suggest that, beyond the NTK regime, the bilevel formulation introduces an implicit bias that promotes feature learning and leads to improved performance.
Limitations and future work. Our theoretical analysis does not address minimax optimality, and establishing minimax rates for (OBO) remains an open problem. While the transition between the NTK and feature learning regimes is well studied for single networks [91 , 26 , 94 ] , its theoretical characterization in the bilevel setting is still unexplored, which could perhaps be approached through the lens of grokking [79 ] . Indeed, this phenomenon has been interpreted as a transition between the NTK and feature learning regimes [67 , 57 , 69 ] , potentially offering an intermediate trade-off. Further directions include extending our analysis to deeper architectures, relaxing the polynomial width requirement typical of the NTK regime, and considering general losses, for which the adjoint function no longer reduces to two independently estimable regression functions.
Acknowledgments and Disclosure of Funding
FEK would like to thank Elie Louis for helpful discussions related to this paper. This work was supported by the ANR project BONSAI (grant ANR-23-CE23-0012-01). EP acknowledges funding from the AI Interdisciplinary Institute ANITI through the French Investments for the Future – PIA3 program under grant agreement ANR-19-PI3A-0004; the Air Force Office of Scientific Research, Air Force Materiel Command, USAF under grant number FA8655-22-1-7012; TSE-P; the ANR project Chess (grant ANR-17-EURE-0010); the ANR project Regulia; and the Institut Universitaire de France (IUF). EP and SV acknowledge support from the ANR project MAD. SV also thanks the PEPR PDE-AI program (ANR-23-PEIA-0004) and the 3IA Côte d’Azur Chair BOGL.
Appendices
Roadmap. Appendix A presents concrete instances of (OBO), namely instrumental variable regression and value function fitting in reinforcement learning. Appendix B discusses Assumptions A to D. Appendix C establishes convergence guarantees for projected mini-batch biased SGD applied to the empirical approximation of (OBO). Within it, Appendix C.1 derives the hypergradient and the smoothness of the value function, Appendix C.2 analyzes the statistical properties of the hypergradient estimator, and Appendix C.3 proves the main convergence theorem. These guarantees in the NTK regime rely on the generalization bounds for (OBO) derived in Appendix D. Those bounds build on Appendix E, which establishes non-asymptotic generalization bounds for neural network gradient flow in the NTK regime and provides the theoretical foundation used to approximate both the inner solution h ⋆ h_\star h ⋆ and the regression function b ⋆ b^\star b ⋆ . Differentiability of the objectives and properties of the optimal solutions are provided in Appendix F. Appendix G shows that the RKHS of the infinite-width NTK and its interpolation spaces are Sobolev spaces. Background on integral and covariance operators as well as interpolation spaces is given in Appendix H. Auxiliary technical lemmas used throughout the proofs are collected in Appendix I. Finally, further experimental details and additional numerical results are provided in Appendix J.
Notations. ⟨ ⋅ , ⋅ ⟩ \langle\cdot,\cdot\rangle ⟨ ⋅ , ⋅ ⟩ and ∥ ⋅ ∥ \|\cdot\| ∥ ⋅ ∥ denote the Euclidean inner product and its induced norm, ⟨ ⋅ , ⋅ ⟩ H \langle\cdot,\cdot\rangle_{\mathcal{H}} ⟨ ⋅ , ⋅ ⟩ H and ∥ ⋅ ∥ H \|\cdot\|_{\mathcal{H}} ∥ ⋅ ∥ H denote the inner product and norm of a Hilbert space H \mathcal{H} H , ∥ ⋅ ∥ o p \|\cdot\|_{\mathrm{op}} ∥ ⋅ ∥ op denotes the operator norm, ∥ ⋅ ∥ F \|\cdot\|_{\mathrm{F}} ∥ ⋅ ∥ F denotes the Frobenius norm, and ∥ ⋅ ∥ H S \|\cdot\|_{\mathrm{HS}} ∥ ⋅ ∥ HS denotes the Hilbert-Schmidt norm. ⊙ \odot ⊙ denotes the Hadamard product. For any v 1 , … , v p ∈ R v_1,\ldots,v_p\in\mathbb{R} v 1 , … , v p ∈ R , diag ( v 1 , … , v p ) ∈ R p × p \operatorname{diag}(v_1,\ldots,v_p)\in\mathbb{R}^{p\times p} diag ( v 1 , … , v p ) ∈ R p × p denotes the diagonal matrix of size p × p p\times p p × p , where the diagonal entries are v 1 , … , v p v_1,\ldots,v_p v 1 , … , v p and all the off-diagonal entries are 0 0 0 . 1 p \mathbf{1}_p 1 p denotes a vector of size p p p where all entries are 1 1 1 . 1 p × p \mathbf{1}_{p\times p} 1 p × p denotes the identity matrix of size p p p . For x ∈ R x\in\mathbb{R} x ∈ R , ⌊ x ⌋ \lfloor x\rfloor ⌊ x ⌋ denotes the largest integer less than or equal to x x x , and ⌈ x ⌉ \lceil x\rceil ⌈ x ⌉ denotes the smallest integer greater than or equal to x x x . For any event E \mathcal{E} E , 1 E \mathbf{1}_{\mathcal{E}} 1 E denotes the indicator function of E \mathcal{E} E . Given a compact set K \mathcal{K} K , diam ( K ) \operatorname{diam}(\mathcal{K}) diam ( K ) denotes its diameter. Id H \operatorname{Id}_{\mathcal{H}} Id H denotes the identity operator on a Hilbert space H \mathcal{H} H . For a set A \mathcal{A} A , A ∁ \mathcal{A}^{\complement} A ∁ denotes its complement. H 1 ≅ H 2 \mathcal{H}_1\cong\mathcal{H}_2 H 1 ≅ H 2 means H 1 = H 2 \mathcal{H}_1=\mathcal{H}_2 H 1 = H 2 as sets with equivalent norms. a ≍ b a\asymp b a ≍ b means that there exist constants c , C > 0 c,C>0 c , C > 0 such that c b ≤ a ≤ C b cb\leq a\leq Cb c b ≤ a ≤ C b .
Concrete Examples of (OBO)
Instrumental variable regression. It is an approach that handles endogeneity in statistical modeling, relying on instruments to identify causal relationships [73 ] . The aim is to learn a function z ↦ g ω ( z ) z \mapsto g_\omega(z) z ↦ g ω ( z ) , parameterized by a vector ω \omega ω , satisfying y = g ω ( z ) + ϵ y = g_\omega(z) + \epsilon y = g ω ( z ) + ϵ , where y ∈ Y y \in\mathcal{Y} y ∈ Y denotes the observed outcome, z ∈ Z z \in\mathcal{Z} z ∈ Z the treatment, and ϵ \epsilon ϵ the error term. The central difficulty lies in the endogeneity of z z z , namely its correlation with ϵ \epsilon ϵ , which renders direct regression inconsistent. Exogeneity no longer holds, since z z z and ϵ \epsilon ϵ are no longer independent, and the estimates of g ω g_\omega g ω are distorted as a result. A remedy is to bring in an instrumental variable x ∈ X x \in\mathcal{X} x ∈ X , correlated with z z z but not with ϵ \epsilon ϵ , which retrieves the link between y y y and z z z while remaining immune to the distortion carried by ϵ \epsilon ϵ , classically through two-stage least squares regression [5 ] . Following [93 ] , this formulation translates directly into a bilevel problem whose inner and outer objectives are those given in Equation (1) , where h h h is taken in L 2 ( X ) L^2(\mathcal{X}) L 2 ( X ) .
Value function fitting in reinforcement learning. A second instance arises in reinforcement learning. Let x = ( s , a ) x = (s, a) x = ( s , a ) be a state–action pair and z = ( s , a , r , s ′ ) z = (s, a, r, s') z = ( s , a , r , s ′ ) the transition it generates, where r r r is the reward received and s ′ s' s ′ is the successor state. Let g ω ( z ) = r + γ V ω ( s ′ ) g_\omega(z) = r + \gamma V_\omega(s') g ω ( z ) = r + γ V ω ( s ′ ) be the one-step Bellman target associated with a parametric value function V ω V_\omega V ω , where γ ∈ ( 0 , 1 ) \gamma\in(0, 1) γ ∈ ( 0 , 1 ) is the discount factor. This target is random given x x x , since it depends on which reward and successor state are drawn. The inner problem averages it over that randomness, returning the Bellman backup h ω ⋆ ( s , a ) = E r , s ′ [ r + γ V ω ( s ′ ) ∣ s , a ] h^\star_\omega(s, a) = \mathbb{E}_{r,s'}[r + \gamma V_\omega(s') \mid s, a] h ω ⋆ ( s , a ) = E r , s ′ [ r + γ V ω ( s ′ ) ∣ s , a ] , which fitted value iteration estimates by regressing the target on ( s , a ) (s, a) ( s , a ) [71 ] . The outer problem then fits this backup to the discounted return y y y observed along a trajectory started at ( s , a ) (s, a) ( s , a ) , whose conditional expectation given ( s , a ) (s, a) ( s , a ) is the true action-value function. Minimizing F \mathcal{F} F therefore selects the value function whose averaged one-step target h ω ⋆ ( s , a ) h^\star_\omega(s, a) h ω ⋆ ( s , a ) is closest, in L 2 ( X ) L^2(\mathcal{X}) L 2 ( X ) , to the observed return y y y .
Discussion of Assumptions
Assumption C is mild and standard in practice, as inputs are naturally bounded in most applications, e.g. , images and physical measurements, or can be made so through preprocessing procedures such as normalization or clipping. Assumption B is common in supervised learning, as the output naturally takes values in a bounded range in most regression tasks, e.g. , probabilities in [ 0 , 1 ] [0, 1] [ 0 , 1 ] , normalized scores, or physical measurements on a known scale, and can always be enforced by output normalization otherwise. Assumption A holds in practice for a broad class of models, including linear models with bounded features, e.g., random Fourier features [80 ] , when the parameter space is compact. It also applies to feedforward neural networks or transformers with smooth activations, e.g., SiLU [81 ] or GeLU [48 ] , provided that the parameter space and the input domain are compact. Assumption D is a standard regularity condition on the input data distribution. Together with Assumption C, it is satisfied, for instance, by the uniform distribution on any compact set with nonempty interior, by Gaussian distributions truncated to a ball, and by finite mixtures of such distributions. It excludes, however, distributions without a Lebesgue density, such as discrete inputs (e.g., binary or one-hot features) and distributions concentrated on lower-dimensional sets (e.g., the uniform distribution on the unit sphere S p − 1 S^{p-1} S p − 1 , or compositional data constrained to the simplex), as well as densities with singularities (e.g., the arcsine distribution Beta ( 1 2 , 1 2 ) \operatorname{Beta}\left(\frac{1}{2},\frac{1}{2}\right) Beta ( 2 1 , 2 1 ) on [ 0 , 1 ] [0,1] [ 0 , 1 ] , whose density diverges at both endpoints).
Convergence of Projected Mini-Batch Biased SGD
Hypergradient and smoothness property of the value function
Proof of Proposition 2.1. The differentiability of F \mathcal{F} F as well as the expression of its hypergradient ∇ F \nabla\mathcal{F} ∇ F are a direct consequence of [78 Proposition 2.2] together with Lemmas F.1 and F.2 . Next, we prove the smoothness property of F \mathcal{F} F . Let ω 1 , ω 2 ∈ U \omega_1,\omega_2 \in U ω 1 , ω 2 ∈ U . We have:
∥ ∇ F ( ω 1 ) − ∇ F ( ω 2 ) ∥ = ∥ E P [ ∂ ω g ω 1 ( z ) a ω 1 ⋆ ( x ) − ∂ ω g ω 2 ( z ) a ω 2 ⋆ ( x ) ] ∥ ≤ E P [ ∥ ∂ ω g ω 1 ( z ) a ω 1 ⋆ ( x ) − ∂ ω g ω 2 ( z ) a ω 2 ⋆ ( x ) ∥ ] = E P [ ∥ ∂ ω g ω 1 ( z ) ( a ω 1 ⋆ ( x ) − a ω 2 ⋆ ( x ) ) + ( ∂ ω g ω 1 ( z ) − ∂ ω g ω 2 ( z ) ) a ω 2 ⋆ ( x ) ∥ ] ≤ E P [ ∥ ∂ ω g ω 1 ( z ) ∥ ∣ a ω 1 ⋆ ( x ) − a ω 2 ⋆ ( x ) ∣ + ∥ ∂ ω g ω 1 ( z ) − ∂ ω g ω 2 ( z ) ∥ ∣ a ω 2 ⋆ ( x ) ∣ ] = E P [ ∥ ∂ ω g ω 1 ( z ) ∥ ∣ h ω 1 ⋆ ( x ) − h ω 2 ⋆ ( x ) ∣ + ∥ ∂ ω g ω 1 ( z ) − ∂ ω g ω 2 ( z ) ∥ ∣ a ω 2 ⋆ ( x ) ∣ ] , (8) \begin{aligned}
\left\|\nabla\mathcal{F}(\omega_1)-\nabla\mathcal{F}(\omega_2)\right\|
&= \left\|\mathbb{E}_{P}\left[\partial_\omega g_{\omega_1}(z)a_{\omega_1}^{\star}(x)-\partial_\omega g_{\omega_2}(z)a_{\omega_2}^{\star}(x)\right]\right\| \\
&\le\mathbb{E}_{P}\left[\left\|\partial_\omega g_{\omega_1}(z)a_{\omega_1}^{\star}(x)-\partial_\omega g_{\omega_2}(z)a_{\omega_2}^{\star}(x)\right\|\right] \\
&= \mathbb{E}_{P}\left[\left\|\partial_\omega g_{\omega_1}(z)(a_{\omega_1}^{\star}(x)-a_{\omega_2}^{\star}(x))+(\partial_\omega g_{\omega_1}(z)-\partial_\omega g_{\omega_2}(z))a_{\omega_2}^{\star}(x)\right\|\right] \\
&\le\mathbb{E}_{P}\left[\|\partial_\omega g_{\omega_1}(z)\|\left|a_{\omega_1}^{\star}(x)-a_{\omega_2}^{\star}(x)\right|+\|\partial_\omega g_{\omega_1}(z)-\partial_\omega g_{\omega_2}(z)\|\left|a_{\omega_2}^{\star}(x)\right|\right] \\
&= \mathbb{E}_{P}\left[\|\partial_\omega g_{\omega_1}(z)\|\left|h_{\omega_1}^{\star}(x)-h_{\omega_2}^{\star}(x)\right|+\|\partial_\omega g_{\omega_1}(z)-\partial_\omega g_{\omega_2}(z)\|\left|a_{\omega_2}^{\star}(x)\right|\right], \tag*{(8)}
\end{aligned} ∥ ∇ F ( ω 1 ) − ∇ F ( ω 2 ) ∥ = E P [ ∂ ω g ω 1 ( z ) a ω 1 ⋆ ( x ) − ∂ ω g ω 2 ( z ) a ω 2 ⋆ ( x ) ] ≤ E P [ ∂ ω g ω 1 ( z ) a ω 1 ⋆ ( x ) − ∂ ω g ω 2 ( z ) a ω 2 ⋆ ( x ) ] = E P [ ∂ ω g ω 1 ( z ) ( a ω 1 ⋆ ( x ) − a ω 2 ⋆ ( x )) + ( ∂ ω g ω 1 ( z ) − ∂ ω g ω 2 ( z )) a ω 2 ⋆ ( x ) ] ≤ E P [ ∥ ∂ ω g ω 1 ( z ) ∥ a ω 1 ⋆ ( x ) − a ω 2 ⋆ ( x ) + ∥ ∂ ω g ω 1 ( z ) − ∂ ω g ω 2 ( z ) ∥ a ω 2 ⋆ ( x ) ] = E P [ ∥ ∂ ω g ω 1 ( z ) ∥ h ω 1 ⋆ ( x ) − h ω 2 ⋆ ( x ) + ∥ ∂ ω g ω 1 ( z ) − ∂ ω g ω 2 ( z ) ∥ a ω 2 ⋆ ( x ) ] , (8)
where, in the third line, we use Jensen’s inequality, and in the last line, we use a ω 1 ⋆ − a ω 2 ⋆ = − ( h ω 1 ⋆ − h ω 2 ⋆ ) a_{\omega_1}^{\star}-a_{\omega_2}^{\star}=-(h_{\omega_1}^{\star}-h_{\omega_2}^{\star}) a ω 1 ⋆ − a ω 2 ⋆ = − ( h ω 1 ⋆ − h ω 2 ⋆ ) , the function b ⋆ b^{\star} b ⋆ being independent of ω \omega ω . By Assumption A, sup ω ∈ U , z ∈ Z ∥ ∂ ω g ω ( z ) ∥ ≤ G \sup_{\omega\in U,z\in Z}\|\partial_\omega g_\omega(z)\|\le G sup ω ∈ U , z ∈ Z ∥ ∂ ω g ω ( z ) ∥ ≤ G and ω ↦ g ω ( z ) \omega\mapsto g_\omega(z) ω ↦ g ω ( z ) is L L L -smooth for any z ∈ Z z\in Z z ∈ Z , that is, ∥ ∂ ω g ω 1 ( z ) − ∂ ω g ω 2 ( z ) ∥ ≤ L ∥ ω 1 − ω 2 ∥ \|\partial_\omega g_{\omega_1}(z)-\partial_\omega g_{\omega_2}(z)\|\le L\|\omega_1-\omega_2\| ∥ ∂ ω g ω 1 ( z ) − ∂ ω g ω 2 ( z ) ∥ ≤ L ∥ ω 1 − ω 2 ∥ . Furthermore, by Lemma F.3 , ∥ a ω 2 ⋆ ∥ ∞ ≤ sup ω ∈ U ∥ a ω ⋆ ∥ ∞ ≤ D + B \|a_{\omega_2}^{\star}\|_\infty\le\sup_{\omega\in U}\|a_\omega^{\star}\|_\infty\le D+B ∥ a ω 2 ⋆ ∥ ∞ ≤ sup ω ∈ U ∥ a ω ⋆ ∥ ∞ ≤ D + B . Combining these bounds in Equation (8) , we obtain:
∥ ∇ F ( ω 1 ) − ∇ F ( ω 2 ) ∥ ≤ G E P [ ∣ h ω 1 ⋆ ( x ) − h ω 2 ⋆ ( x ) ∣ ] + L ( D + B ) ∥ ω 1 − ω 2 ∥ = G ∥ h ω 1 ⋆ − h ω 2 ⋆ ∥ L 1 ( X ) + L ( D + B ) ∥ ω 1 − ω 2 ∥ ≤ G ∥ h ω 1 ⋆ − h ω 2 ⋆ ∥ L 2 ( X ) + L ( D + B ) ∥ ω 1 − ω 2 ∥ ≤ ( G 2 + L ( D + B ) ) ⏟ L ∥ ω 1 − ω 2 ∥ , \begin{aligned}
\|\nabla\mathcal{F}(\omega_1)-\nabla\mathcal{F}(\omega_2)\|
&\le G\mathbb{E}_{P}\left[|h_{\omega_1}^{\star}(x)-h_{\omega_2}^{\star}(x)|\right]+L(D+B)\|\omega_1-\omega_2\| \\
&=G\|h_{\omega_1}^{\star}-h_{\omega_2}^{\star}\|_{L^1(\mathcal{X})}+L(D+B)\|\omega_1-\omega_2\| \\
&\le G\|h_{\omega_1}^{\star}-h_{\omega_2}^{\star}\|_{L^2(\mathcal{X})}+L(D+B)\|\omega_1-\omega_2\| \\
&\le\underbrace{\left(G^2+L(D+B)\right)}_{\mathcal{L}}\|\omega_1-\omega_2\|,
\end{aligned} ∥∇ F ( ω 1 ) − ∇ F ( ω 2 ) ∥ ≤ G E P [ ∣ h ω 1 ⋆ ( x ) − h ω 2 ⋆ ( x ) ∣ ] + L ( D + B ) ∥ ω 1 − ω 2 ∥ = G ∥ h ω 1 ⋆ − h ω 2 ⋆ ∥ L 1 ( X ) + L ( D + B ) ∥ ω 1 − ω 2 ∥ ≤ G ∥ h ω 1 ⋆ − h ω 2 ⋆ ∥ L 2 ( X ) + L ( D + B ) ∥ ω 1 − ω 2 ∥ ≤ L ( G 2 + L ( D + B ) ) ∥ ω 1 − ω 2 ∥ ,
where, in the third line, we use ∥ ⋅ ∥ L 1 ( X ) ≤ ∥ ⋅ ∥ L 2 ( X ) \|\cdot\|_{L^1(\mathcal{X})}\le\|\cdot\|_{L^2(\mathcal{X})} ∥ ⋅ ∥ L 1 ( X ) ≤ ∥ ⋅ ∥ L 2 ( X ) , valid since P X P_X P X is a probability measure, and where L 1 ( X ) L^1(\mathcal{X}) L 1 ( X ) denotes the space of integrable functions on X \mathcal{X} X with respect to P X P_X P X , endowed with the norm ∥ f ∥ L 1 ( X ) = E x ∼ P X [ ∣ f ( x ) ∣ ] \|f\|_{L^1(\mathcal{X})}=\mathbb{E}_{x\sim P_X}[|f(x)|] ∥ f ∥ L 1 ( X ) = E x ∼ P X [ ∣ f ( x ) ∣ ] . In the last line, we use the fact that ω ↦ h ω ⋆ \omega\mapsto h_\omega^{\star} ω ↦ h ω ⋆ is G G G -Lipschitz continuous, as established in Proposition F.4 . This concludes the proof. □
Statistical properties of the hypergradient estimator
Lemma C.1 (Conditional mean of the hypergradient estimator). Fix ω ∈ Ω \omega\in\Omega ω ∈ Ω . Let h ^ ω \hat{h}_\omega h ^ ω and b ^ \hat{b} b ^ be any estimators of the inner solution h ω ⋆ h_\omega^{\star} h ω ⋆ and of the regression function b ⋆ b^{\star} b ⋆ , built from D i n \mathcal{D}_{in} D in and D o u t \mathcal{D}_{out} D o u t , and set a ^ ω = − h ^ ω + b ^ \hat{a}_\omega=-\hat{h}_\omega+\hat{b} a ^ ω = − h ^ ω + b ^ . Then:
∇ F ‾ ( ω ) : = E [ ∇ F ^ ( ω ) ∣ a ^ ω ] = − E P [ ∂ ω g ω ( z ) a ^ ω ( x ) ] . \overline{\nabla\mathcal{F}}(\omega):=\mathbb{E}\left[\left.\widehat{\nabla\mathcal{F}}(\omega)\right|\hat{a}_\omega\right]=-\mathbb{E}_{P}\left[\partial_\omega g_\omega(z)\hat{a}_\omega(x)\right]. ∇ F ( ω ) := E [ ∇ F ( ω ) a ^ ω ] = − E P [ ∂ ω g ω ( z ) a ^ ω ( x ) ] .
where E P \mathbb{E}_{P} E P integrates over ( x , z ) ∼ P X Z (x,z)\sim P_{XZ} ( x , z ) ∼ P X Z only, a ^ ω \hat{a}_\omega a ^ ω being held fixed.
Proof. Let ω ∈ Ω \omega\in\Omega ω ∈ Ω . By Equation (4) , we have:
∇ F ~ ( ω ) = − 1 n g ∑ l = 1 n g ∂ ω g ω ( z l ) a ^ ω ( x l ) . \widetilde{\nabla F}(\omega)=-\frac{1}{n_g}\sum_{l=1}^{n_g}\partial_{\omega}g_{\omega}(z_l)\widehat{a}_{\omega}(x_l). ∇ F ( ω ) = − n g 1 l = 1 ∑ n g ∂ ω g ω ( z l ) a ω ( x l ) .
where B g = { ( x l , z l ) } l = 1 n g B_g=\{(x_l,z_l)\}_{l=1}^{n_g} B g = {( x l , z l ) } l = 1 n g is the mini-batch used at ω \omega ω . Since a ^ ω \widehat{a}_{\omega} a ω is built from D i n D_{in} D in and D o u t D_{out} D o u t , which are independent of B g B_g B g , the pairs ( x l , z l ) (x_l,z_l) ( x l , z l ) remain i.i.d. with law P X Z \mathbb{P}_{XZ} P X Z conditionally on a ^ ω \widehat{a}_{\omega} a ω . Hence, each summand has the same conditional expectation:
E [ ∂ ω g ω ( z l ) a ^ ω ( x l ) ∣ a ^ ω ] = E P [ ∂ ω g ω ( z ) a ^ ω ( x ) ] , l = 1 , … , n g , \mathbb{E}\left[\partial_{\omega}g_{\omega}(z_l)\widehat{a}_{\omega}(x_l)\mid\widehat{a}_{\omega}\right]=\mathbb{E}_{\mathbb{P}}\left[\partial_{\omega}g_{\omega}(z)\widehat{a}_{\omega}(x)\right],\qquad l=1,\ldots,n_g, E [ ∂ ω g ω ( z l ) a ω ( x l ) ∣ a ω ] = E P [ ∂ ω g ω ( z ) a ω ( x ) ] , l = 1 , … , n g ,
and averaging over l l l yields the announced identity by linearity of the expectation. □ \square □
Lemma C.2 (Bias control). Let Assumption A hold. Let { h ^ ω } ω ∈ Ω \{\widehat{h}_{\omega}\}_{\omega\in\Omega} { h ω } ω ∈ Ω and b ^ \widehat{b} b be any estimators of { h ω ⋆ } ω ∈ Ω \{h_{\omega}^{\star}\}_{\omega\in\Omega} { h ω ⋆ } ω ∈ Ω and b ⋆ b^{\star} b ⋆ built from D i n D_{in} D in and D o u t D_{out} D o u t , possibly using additional randomness independent of the mini-batches, and set a ^ ω = − h ^ ω + b ^ \widehat{a}_{\omega}=-\widehat{h}_{\omega}+\widehat{b} a ω = − h ω + b and a ω ⋆ = − h ω ⋆ + b ⋆ a_{\omega}^{\star}=-h_{\omega}^{\star}+b^{\star} a ω ⋆ = − h ω ⋆ + b ⋆ for every ω ∈ Ω \omega\in\Omega ω ∈ Ω . Then:
E [ sup ω ∈ Ω ∥ ∇ F ~ ( ω ) − ∇ F ( ω ) ∥ 2 ] ≤ G 2 E [ sup ω ∈ Ω ∥ a ^ ω − a ω ⋆ ∥ L 2 ( X ) 2 ] ≤ 2 G 2 ( e h + e b ) , \mathbb{E}\left[\sup_{\omega\in\Omega}\left\|\widetilde{\nabla F}(\omega)-\nabla F(\omega)\right\|^{2}\right]\leq G^{2}\mathbb{E}\left[\sup_{\omega\in\Omega}\left\|\widehat{a}_{\omega}-a_{\omega}^{\star}\right\|_{L^{2}(\mathcal{X})}^{2}\right]\leq2G^{2}(e_h+e_b), E [ ω ∈ Ω sup ∇ F ( ω ) − ∇ F ( ω ) 2 ] ≤ G 2 E [ ω ∈ Ω sup ∥ a ω − a ω ⋆ ∥ L 2 ( X ) 2 ] ≤ 2 G 2 ( e h + e b ) , where ∇ F ~ ( ω ) = E [ ∇ F ~ ( ω ) ∣ a ^ ω ] \widetilde{\nabla F}(\omega)=\mathbb{E}[\widetilde{\nabla F}(\omega)\mid\widehat{a}_{\omega}] ∇ F ( ω ) = E [ ∇ F ( ω ) ∣ a ω ] , e h = E [ sup ω ∈ Ω ∥ h ^ ω − h ω ⋆ ∥ L 2 ( X ) 2 ] e_h=\mathbb{E}[\sup_{\omega\in\Omega}\|\widehat{h}_{\omega}-h_{\omega}^{\star}\|_{L^{2}(\mathcal{X})}^{2}] e h = E [ sup ω ∈ Ω ∥ h ω − h ω ⋆ ∥ L 2 ( X ) 2 ] , e b = E [ ∥ b ^ − b ⋆ ∥ L 2 ( X ) 2 ] e_b=\mathbb{E}[\|\widehat{b}-b^{\star}\|_{L^{2}(\mathcal{X})}^{2}] e b = E [ ∥ b − b ⋆ ∥ L 2 ( X ) 2 ] , and the expectation is taken over the randomness used to construct h ^ ω \widehat{h}_{\omega} h ω and b ^ \widehat{b} b .
Proof. Fix ω ∈ Ω \omega\in\Omega ω ∈ Ω . By Lemma C.1 and Proposition 2.1, we have:
∇ F ~ ( ω ) − ∇ F ( ω ) = − E P [ ∂ ω g ω ( z ) ( a ^ ω ( x ) − a ω ⋆ ( x ) ) ] . \widetilde{\nabla F}(\omega)-\nabla F(\omega)=-\mathbb{E}_{\mathbb{P}}\left[\partial_{\omega}g_{\omega}(z)\left(\widehat{a}_{\omega}(x)-a_{\omega}^{\star}(x)\right)\right]. ∇ F ( ω ) − ∇ F ( ω ) = − E P [ ∂ ω g ω ( z ) ( a ω ( x ) − a ω ⋆ ( x ) ) ] .
Since ∥ ⋅ ∥ 2 \|\cdot\|^{2} ∥ ⋅ ∥ 2 is convex, Jensen’s inequality gives:
∥ ∇ F ~ ( ω ) − ∇ F ( ω ) ∥ 2 ≤ E P [ ∥ ∂ ω g ω ( z ) ∥ 2 ( a ^ ω ( x ) − a ω ⋆ ( x ) ) 2 ] ≤ G 2 ∥ a ^ ω − a ω ⋆ ∥ L 2 ( X ) 2 , \left\|\widetilde{\nabla F}(\omega)-\nabla F(\omega)\right\|^{2}\leq\mathbb{E}_{\mathbb{P}}\left[\|\partial_{\omega}g_{\omega}(z)\|^{2}\left(\widehat{a}_{\omega}(x)-a_{\omega}^{\star}(x)\right)^{2}\right]\leq G^{2}\left\|\widehat{a}_{\omega}-a_{\omega}^{\star}\right\|_{L^{2}(\mathcal{X})}^{2}, ∇ F ( ω ) − ∇ F ( ω ) 2 ≤ E P [ ∥ ∂ ω g ω ( z ) ∥ 2 ( a ω ( x ) − a ω ⋆ ( x ) ) 2 ] ≤ G 2 ∥ a ω − a ω ⋆ ∥ L 2 ( X ) 2 ,
where the last inequality uses sup ω ∈ Ω , z ∈ Z ∥ ∂ ω g ω ( z ) ∥ ≤ G \sup_{\omega\in\Omega,z\in\mathcal{Z}}\|\partial_{\omega}g_{\omega}(z)\|\leq G sup ω ∈ Ω , z ∈ Z ∥ ∂ ω g ω ( z ) ∥ ≤ G from Assumption A , together with the fact that L 2 ( X ) L^{2}(\mathcal{X}) L 2 ( X ) is taken with respect to P X \mathbb{P}_{X} P X , so that E P [ ( a ^ ω ( x ) − a ω ⋆ ( x ) ) 2 ] = ∥ a ^ ω − a ω ⋆ ∥ L 2 ( X ) 2 \mathbb{E}_{\mathbb{P}}[(\widehat{a}_{\omega}(x)-a_{\omega}^{\star}(x))^{2}]=\|\widehat{a}_{\omega}-a_{\omega}^{\star}\|_{L^{2}(\mathcal{X})}^{2} E P [( a ω ( x ) − a ω ⋆ ( x ) ) 2 ] = ∥ a ω − a ω ⋆ ∥ L 2 ( X ) 2 . This inequality holds for every ω ∈ Ω \omega\in\Omega ω ∈ Ω and every realization of the randomness used to construct h ^ ω \widehat{h}_{\omega} h ω and b ^ \widehat{b} b , so we may take the supremum over ω ∈ Ω \omega\in\Omega ω ∈ Ω on both sides and then the expectation, which gives the first bound. For the second, the triangle inequality and ( u + v ) 2 ≤ 2 u 2 + 2 v 2 (u+v)^{2}\leq2u^{2}+2v^{2} ( u + v ) 2 ≤ 2 u 2 + 2 v 2 yield, for every $\omega\in\Omega:
∥ a ^ ω − a ω ⋆ ∥ L 2 ( X ) 2 ≤ ( ∥ h ^ ω − h ω ⋆ ∥ L 2 ( X ) + ∥ b ^ − b ⋆ ∥ L 2 ( X ) ) 2 \left\|\widehat{a}_{\omega}-a_{\omega}^{\star}\right\|_{L^{2}(\mathcal{X})}^{2}\leq\left(\left\|\widehat{h}_{\omega}-h_{\omega}^{\star}\right\|_{L^{2}(\mathcal{X})}+\left\|\widehat{b}-b^{\star}\right\|_{L^{2}(\mathcal{X})}\right)^{2} ∥ a ω − a ω ⋆ ∥ L 2 ( X ) 2 ≤ ( h ω − h ω ⋆ L 2 ( X ) + b − b ⋆ L 2 ( X ) ) 2
≤ 2 ( ∥ h ^ ω − h ω ⋆ ∥ L 2 ( X ) 2 + ∥ b ^ − b ⋆ ∥ L 2 ( X ) 2 ) . \leq2\left(\left\|\widehat{h}_{\omega}-h_{\omega}^{\star}\right\|_{L^{2}(\mathcal{X})}^{2}+\left\|\widehat{b}-b^{\star}\right\|_{L^{2}(\mathcal{X})}^{2}\right). ≤ 2 ( h ω − h ω ⋆ L 2 ( X ) 2 + b − b ⋆ L 2 ( X ) 2 ) .
Taking the supremum over ω ∈ Ω \omega\in\Omega ω ∈ Ω and then the expectation concludes the proof. □ \square □
Lemma C.3 (Variance control). Let Assumptions A and B hold. Let { h ^ ω } ω ∈ Ω \{\widehat{h}_{\omega}\}_{\omega\in\Omega} { h ω } ω ∈ Ω and b ^ \widehat{b} b be any estimators of { h ω ⋆ } ω ∈ Ω \{h_{\omega}^{\star}\}_{\omega\in\Omega} { h ω ⋆ } ω ∈ Ω and b ⋆ b^{\star} b ⋆ built from D i n D_{in} D in and D o u t D_{out} D o u t , possibly using additional randomness independent of the mini-batches, and set a ^ ω = − h ^ ω + b ^ \widehat{a}_{\omega}=-\widehat{h}_{\omega}+\widehat{b} a ω = − h ω + b and a ω ⋆ = − h ω ⋆ + b ⋆ a_{\omega}^{\star}=-h_{\omega}^{\star}+b^{\star} a ω ⋆ = − h ω ⋆ + b ⋆ for every ω ∈ Ω \omega\in\Omega ω ∈ Ω . Then, for every ω ∈ Ω \omega\in\Omega ω ∈ Ω , we have:
E [ ∥ ∇ F ~ ( ω ) − ∇ F ( ω ) ∥ 2 ] ≤ 2 G 2 n g ( E [ sup ω ′ ∈ Ω ∥ a ^ ω ′ − a ω ′ ⋆ ∥ L 2 ( X ) 2 ] + ( D + B ) 2 ) \mathbb{E}\left[\left\|\widetilde{\nabla F}(\omega)-\nabla F(\omega)\right\|^{2}\right]\leq\frac{2G^{2}}{n_g}\left(\mathbb{E}\left[\sup_{\omega'\in\Omega}\left\|\widehat{a}_{\omega'}-a_{\omega'}^{\star}\right\|_{L^{2}(\mathcal{X})}^{2}\right]+(D+B)^{2}\right) E [ ∇ F ( ω ) − ∇ F ( ω ) 2 ] ≤ n g 2 G 2 ( E [ ω ′ ∈ Ω sup ∥ a ω ′ − a ω ′ ⋆ ∥ L 2 ( X ) 2 ] + ( D + B ) 2 ) ≤ 2 G 2 n g [ 2 ( e h + e b ) + ( D + B ) 2 ] , \leq\frac{2G^{2}}{n_g}\left[2(e_h+e_b)+(D+B)^{2}\right], ≤ n g 2 G 2 [ 2 ( e h + e b ) + ( D + B ) 2 ] , where ∇ F ~ ( ω ) = E [ ∇ F ~ ( ω ) ∣ a ^ ω ] \widetilde{\nabla F}(\omega)=\mathbb{E}[\widetilde{\nabla F}(\omega)\mid\widehat{a}_{\omega}] ∇ F ( ω ) = E [ ∇ F ( ω ) ∣ a ω ] , e h = E [ sup ω ∈ Ω ∥ h ^ ω − h ω ⋆ ∥ L 2 ( X ) 2 ] e_h=\mathbb{E}[\sup_{\omega\in\Omega}\|\widehat{h}_{\omega}-h_{\omega}^{\star}\|_{L^{2}(\mathcal{X})}^{2}] e h = E [ sup ω ∈ Ω ∥ h ω − h ω ⋆ ∥ L 2 ( X ) 2 ] , e b = E [ ∥ b ^ − b ⋆ ∥ L 2 ( X ) 2 ] e_b=\mathbb{E}[\|\widehat{b}-b^{\star}\|_{L^{2}(\mathcal{X})}^{2}] e b = E [ ∥ b − b ⋆ ∥ L 2 ( X ) 2 ] , and the expectation is taken over all sources of randomness.
Proof. Fix ω ∈ Ω \omega\in\Omega ω ∈ Ω . Let B g = { ( x l , z l ) } l = 1 n g \mathcal{B}_g = \{(x_l,z_l)\}_{l=1}^{n_g} B g = {( x l , z l ) } l = 1 n g be the mini-batch used at ω \omega ω , and set:
U l ( ω ) = − ∂ ω g ω ( z l ) a ^ ω ( x l ) , l = 1 , … , n g , U_l(\omega)=-\partial_\omega g_\omega(z_l)\hat{a}_\omega(x_l), \qquad l=1,\ldots,n_g, U l ( ω ) = − ∂ ω g ω ( z l ) a ^ ω ( x l ) , l = 1 , … , n g ,
so that ∇ F ( ω ) = 1 n g ∑ l = 1 n g U l ( ω ) \nabla\mathcal{F}(\omega)=\frac{1}{n_g}\sum_{l=1}^{n_g}U_l(\omega) ∇ F ( ω ) = n g 1 ∑ l = 1 n g U l ( ω ) and, by Lemma C.1, ∇ F ( ω ) = E [ U l ( ω ) ∣ a ^ ω ] \nabla F(\omega)=\mathbb{E}[U_l(\omega)\mid\hat{a}_\omega] ∇ F ( ω ) = E [ U l ( ω ) ∣ a ^ ω ] for every l l l . As B g \mathcal{B}_g B g is independent of a ^ ω \hat{a}_\omega a ^ ω , the variables U 1 ( ω ) , … , U n g ( ω ) U_1(\omega),\ldots,U_{n_g}(\omega) U 1 ( ω ) , … , U n g ( ω ) are i.i.d. conditionally on a ^ ω \hat{a}_\omega a ^ ω , so that the conditional variance of their average scales as 1 / n g 1/n_g 1/ n g :
E [ ∥ ∇ F ( ω ) − ∇ F ( ω ) ∥ 2 | a ^ ω ] ≤ 1 n g E [ ∥ U 1 ( ω ) ∥ 2 | a ^ ω ] . \mathbb{E}\left[\left\|\nabla\mathcal{F}(\omega)-\nabla F(\omega)\right\|^2\middle|\hat{a}_\omega\right]\leq\frac{1}{n_g}\mathbb{E}\left[\|U_1(\omega)\|^2\middle|\hat{a}_\omega\right]. E [ ∥ ∇ F ( ω ) − ∇ F ( ω ) ∥ 2 a ^ ω ] ≤ n g 1 E [ ∥ U 1 ( ω ) ∥ 2 a ^ ω ] .
By Assumption A , ∥ U 1 ( ω ) ∥ 2 ≤ G 2 ∣ a ^ ω ( x 1 ) ∣ 2 \|U_1(\omega)\|^2\leq G^2|\hat{a}_\omega(x_1)|^2 ∥ U 1 ( ω ) ∥ 2 ≤ G 2 ∣ a ^ ω ( x 1 ) ∣ 2 , and since x 1 ∼ P X x_1\sim P_X x 1 ∼ P X conditionally on a ^ ω \hat{a}_\omega a ^ ω :
E [ ∥ U 1 ( ω ) ∥ 2 | a ^ ω ] ≤ G 2 ∥ a ^ ω ∥ L 2 ( X ) 2 ≤ 2 G 2 ( ∥ a ^ ω − a ⋆ ∥ L 2 ( X ) 2 + ∥ a ⋆ ∥ L 2 ( X ) 2 ) \mathbb{E}\left[\|U_1(\omega)\|^2\middle|\hat{a}_\omega\right]\leq G^2\|\hat{a}_\omega\|_{L^2(\mathcal{X})}^2\leq2G^2\left(\|\hat{a}_\omega-a^\star\|_{L^2(\mathcal{X})}^2+\|a^\star\|_{L^2(\mathcal{X})}^2\right) E [ ∥ U 1 ( ω ) ∥ 2 a ^ ω ] ≤ G 2 ∥ a ^ ω ∥ L 2 ( X ) 2 ≤ 2 G 2 ( ∥ a ^ ω − a ⋆ ∥ L 2 ( X ) 2 + ∥ a ⋆ ∥ L 2 ( X ) 2 )
≤ 2 G 2 ( sup ω ′ ∈ Ω ∥ a ^ ω ′ − a ⋆ ∥ L 2 ( X ) 2 + ( D + B ) 2 ) , \leq2G^2\left(\sup_{\omega'\in\Omega}\|\hat{a}_{\omega'}-a^\star\|_{L^2(\mathcal{X})}^2+(D+B)^2\right), ≤ 2 G 2 ( ω ′ ∈ Ω sup ∥ a ^ ω ′ − a ⋆ ∥ L 2 ( X ) 2 + ( D + B ) 2 ) ,
where the second inequality follows from the elementary inequality ∥ u + v ∥ 2 ≤ 2 ( ∥ u ∥ 2 + ∥ v ∥ 2 ) \|u+v\|^2\leq2(\|u\|^2+\|v\|^2) ∥ u + v ∥ 2 ≤ 2 ( ∥ u ∥ 2 + ∥ v ∥ 2 ) , and the third bounds ∥ a ^ ω − a ⋆ ∥ L 2 ( X ) 2 \|\hat{a}_\omega-a^\star\|_{L^2(\mathcal{X})}^2 ∥ a ^ ω − a ⋆ ∥ L 2 ( X ) 2 by its supremum over Ω \Omega Ω and uses ∥ a ⋆ ∥ L 2 ( X ) ≤ ∥ a ⋆ ∥ ∞ \|a^\star\|_{L^2(\mathcal{X})}\leq\|a^\star\|_\infty ∥ a ⋆ ∥ L 2 ( X ) ≤ ∥ a ⋆ ∥ ∞ , valid since P X P_X P X is a probability measure, together with the bound ∥ a ⋆ ∥ ∞ ≤ D + B \|a^\star\|_\infty\leq D+B ∥ a ⋆ ∥ ∞ ≤ D + B established in Lemma F.3. Taking the total expectation gives the first bound. For the second, the triangle inequality and ( u + v ) 2 ≤ 2 u 2 + 2 v 2 (u+v)^2\leq2u^2+2v^2 ( u + v ) 2 ≤ 2 u 2 + 2 v 2 yield, for every ω ′ ∈ Ω \omega'\in\Omega ω ′ ∈ Ω :
∥ a ^ ω ′ − a ⋆ ∥ L 2 ( X ) 2 ≤ 2 ∥ h ^ ω ′ − h ⋆ ∥ L 2 ( X ) 2 + 2 ∥ b ^ − b ⋆ ∥ L 2 ( X ) 2 , \|\hat{a}_{\omega'}-a^\star\|_{L^2(\mathcal{X})}^2\leq2\|\hat{h}_{\omega'}-h^\star\|_{L^2(\mathcal{X})}^2+2\|\hat{b}-b^\star\|_{L^2(\mathcal{X})}^2, ∥ a ^ ω ′ − a ⋆ ∥ L 2 ( X ) 2 ≤ 2∥ h ^ ω ′ − h ⋆ ∥ L 2 ( X ) 2 + 2∥ b ^ − b ⋆ ∥ L 2 ( X ) 2 ,
so that taking the supremum over ω ′ ∈ Ω \omega'\in\Omega ω ′ ∈ Ω and then the expectation gives E [ sup ω ′ ∈ Ω ∥ a ^ ω ′ − a ⋆ ∥ L 2 ( X ) 2 ] ≤ 2 ( e h + e b ) \mathbb{E}[\sup_{\omega'\in\Omega}\|\hat{a}_{\omega'}-a^\star\|_{L^2(\mathcal{X})}^2]\leq2(e_h+e_b) E [ sup ω ′ ∈ Ω ∥ a ^ ω ′ − a ⋆ ∥ L 2 ( X ) 2 ] ≤ 2 ( e h + e b ) , since b ^ \hat{b} b ^ does not depend on ω ′ \omega' ω ′ .
Main theorem: convergence of projected mini-batch biased SGD
We write δ Ω \delta_\Omega δ Ω for the indicator function of Ω \Omega Ω , that is δ Ω ( ω ) = 0 \delta_\Omega(\omega)=0 δ Ω ( ω ) = 0 if ω ∈ Ω \omega\in\Omega ω ∈ Ω and + ∞ +\infty + ∞ otherwise—a standard device that folds the constraint ω ∈ Ω \omega\in\Omega ω ∈ Ω into the objective itself, so that minimizing F + δ Ω F+\delta_\Omega F + δ Ω over all of R d \mathbb{R}^d R d is the same problem as minimizing F F F over Ω \Omega Ω alone. We set f : = F + δ Ω f:=F+\delta_\Omega f := F + δ Ω . For ω \omega ω with f ( ω ) < ∞ f(\omega)<\infty f ( ω ) < ∞ , ∂ f ( ω ) \partial f(\omega) ∂ f ( ω ) denotes the (limiting) subdifferential of f f f at ω \omega ω : a set of generalized slopes, reducing to { ∇ f ( ω ) } \{\nabla f(\omega)\} { ∇ f ( ω )} where f f f is differentiable, with 0 ∈ ∂ f ( ω ) 0\in\partial f(\omega) 0 ∈ ∂ f ( ω ) the first-order stationarity condition for min ω ∈ Ω F ( ω ) \min_{\omega\in\Omega}F(\omega) min ω ∈ Ω F ( ω ) . For a point p p p and a set S S S , dist ( p , S ) : = inf s ∈ S ∥ p − s ∥ \operatorname{dist}(p,S):=\inf_{s\in S}\|p-s\| dist ( p , S ) := inf s ∈ S ∥ p − s ∥ is the ordinary distance from p p p to S S S ; we use it only as dist ( 0 , ∂ f ( ω ) ) \operatorname{dist}(0,\partial f(\omega)) dist ( 0 , ∂ f ( ω )) , a continuous relaxation of exact stationarity (zero exactly when 0 ∈ ∂ f ( ω ) 0\in\partial f(\omega) 0 ∈ ∂ f ( ω ) , small when some element of ∂ f ( ω ) \partial f(\omega) ∂ f ( ω ) is nearly zero). A function ϕ \phi ϕ is ρ \rho ρ -weakly convex if ϕ + ρ 2 ∥ ⋅ ∥ 2 \phi+\frac{\rho}{2}\|\cdot\|^2 ϕ + 2 ρ ∥ ⋅ ∥ 2 is convex. We note that every L L L -smooth function is L L L -weakly convex.
The Moreau envelope. Since F F F is L L L -smooth on Ω \Omega Ω (Proposition 2.1), F + L 2 ∥ ⋅ ∥ 2 F+\frac{L}{2}\|\cdot\|^2 F + 2 L ∥ ⋅ ∥ 2 is convex, so F F F is L L L -weakly convex. Moreover, since Ω \Omega Ω is convex, δ Ω \delta_\Omega δ Ω is convex, so f = F + δ Ω f=F+\delta_\Omega f = F + δ Ω is L L L -weakly convex as well, being a sum of the two. Fix λ ∈ ( 0 , L − 1 ) \lambda\in(0,L^{-1}) λ ∈ ( 0 , L − 1 ) and define, for ω ∈ Ω \omega\in\Omega ω ∈ Ω , the Moreau envelope and the proximal map of F F F as:
F λ ( ω ) : = min y ∈ Ω { F ( y ) + 1 2 λ ∥ y − ω ∥ 2 } , prox λ F ( ω ) : = argmin y ∈ Ω { F ( y ) + 1 2 λ ∥ y − ω ∥ 2 } . (9) F_\lambda(\omega):=\min_{y\in\Omega}\left\{F(y)+\frac{1}{2\lambda}\|y-\omega\|^2\right\},\qquad\operatorname{prox}_{\lambda F}(\omega):=\underset{y\in\Omega}{\operatorname{argmin}}\left\{F(y)+\frac{1}{2\lambda}\|y-\omega\|^2\right\}. \tag*{(9)} F λ ( ω ) := y ∈ Ω min { F ( y ) + 2 λ 1 ∥ y − ω ∥ 2 } , prox λ F ( ω ) := y ∈ Ω argmin { F ( y ) + 2 λ 1 ∥ y − ω ∥ 2 } . (9)
Since λ < L − 1 \lambda<L^{-1} λ < L − 1 , the minimized objective is ( λ − 1 − L ) (\lambda^{-1}-L) ( λ − 1 − L ) -strongly convex on Ω \Omega Ω , so the minimum is attained at a unique point and prox λ F ( ω ) \operatorname{prox}_{\lambda F}(\omega) prox λ F ( ω ) is well defined. By the standard Moreau-envelope calculus for weakly convex functions [70 , 83 ] , F λ F_\lambda F λ is C 1 C^1 C 1 with:
∇ F λ ( ω ) = 1 λ ( ω − prox λ F ( ω ) ) , ∥ prox λ F ( ω ) − ω ∥ = λ ∥ ∇ F λ ( ω ) ∥ . (10) \nabla F_\lambda(\omega)=\frac{1}{\lambda}\left(\omega-\operatorname{prox}_{\lambda F}(\omega)\right),\qquad\|\operatorname{prox}_{\lambda F}(\omega)-\omega\|=\lambda\|\nabla F_\lambda(\omega)\|. \tag*{(10)} ∇ F λ ( ω ) = λ 1 ( ω − prox λ F ( ω ) ) , ∥ prox λ F ( ω ) − ω ∥ = λ ∥∇ F λ ( ω ) ∥. (10)
Writing u k : = prox λ F ( ω k ) u_k:=\operatorname{prox}_{\lambda F}(\omega_k) u k := prox λ F ( ω k ) , we have f ( u k ) < f ( ω k ) f(u_k)<f(\omega_k) f ( u k ) < f ( ω k ) , since ω k \omega_k ω k itself is a feasible, zero-penalty competitor in the minimization defining F λ ( ω k ) F_\lambda(\omega_k) F λ ( ω k ) , and dist ( 0 , ∂ f ( u k ) ) ≤ ∥ ∇ F λ ( ω k ) ∥ \operatorname{dist}(0,\partial f(u_k))\leq\|\nabla F_\lambda(\omega_k)\| dist ( 0 , ∂ f ( u k )) ≤ ∥∇ F λ ( ω k ) ∥ , since the first-order optimality condition for u k u_k u k reads exactly ∇ F λ ( ω k ) ∈ ∂ f ( u k ) \nabla F_\lambda(\omega_k)\in\partial f(u_k) ∇ F λ ( ω k ) ∈ ∂ f ( u k ) by the subdifferential sum rule, valid here because the added quadratic term is smooth. A small ∥ ∇ F λ ( ω k ) ∥ \|\nabla F_\lambda(\omega_k)\| ∥∇ F λ ( ω k ) ∥ therefore certifies that ω k \omega_k ω k is close to a point u k u_k u k that is itself near-stationary for (OBO), with no loss of objective value. This is the C 1 C^1 C 1 surrogate for dist ( 0 , ∂ f ( ω k ) ) \operatorname{dist}(0,\partial f(\omega_k)) dist ( 0 , ∂ f ( ω k )) itself, which is generally discontinuous in ω k \omega_k ω k and hence unsuitable for a descent-type stochastic analysis.
Lemma C.4 (One-step envelope descent). Let Assumptions A A A and B B B hold and let λ ∈ ( 0 , L − 1 ) \lambda\in(0,L^{-1}) λ ∈ ( 0 , L − 1 ) , where L L L is the smoothness parameter of F F F provided in Proposition 2.1. Let { ω k } k ≥ 0 \{\omega_k\}_{k\ge0} { ω k } k ≥ 0 be generated by ω k + 1 = Π Ω ( ω k − η ∇ F ~ ( ω k ) ) \omega_{k+1}=\Pi_{\Omega}\left(\omega_k-\eta\nabla\widetilde{F}(\omega_k)\right) ω k + 1 = Π Ω ( ω k − η ∇ F ( ω k ) ) with 0 < η ≤ 1 2 L 0<\eta\le\frac{1}{2L} 0 < η ≤ 2 L 1 . Then, for every k ≥ 0 k\ge0 k ≥ 0 :
E [ F λ ( ω k + 1 ) ∣ a ^ ω k ] ≤ F λ ( ω k ) − 3 η 4 ( 1 − λ L ) ∥ ∇ F λ ( ω k ) ∥ 2 + η ( 1 1 − λ L + 2 η λ ) ∥ ∇ F ~ ( ω k ) − ∇ F ( ω k ) ∥ 2 \mathbb{E}\left[\mathcal{F}_{\lambda}(\omega_{k+1})\mid\widehat{a}_{\omega_k}\right]
\le\mathcal{F}_{\lambda}(\omega_k)-\frac{3\eta}{4}(1-\lambda L)\left\|\nabla F_{\lambda}(\omega_k)\right\|^2+\eta\left(\frac{1}{1-\lambda L}+\frac{2\eta}{\lambda}\right)\left\|\nabla\widetilde{F}(\omega_k)-\nabla F(\omega_k)\right\|^2 E [ F λ ( ω k + 1 ) ∣ a ω k ] ≤ F λ ( ω k ) − 4 3 η ( 1 − λ L ) ∥ ∇ F λ ( ω k ) ∥ 2 + η ( 1 − λ L 1 + λ 2 η ) ∇ F ( ω k ) − ∇ F ( ω k ) 2 + 2 η 2 λ E [ ∥ ∇ F ~ ( ω k ) − ∇ F ~ ( ω k ) ∥ 2 ∣ a ^ ω k ] + η λ ( F ( ω k ) − E [ F ( ω k + 1 ) ∣ a ^ ω k ] ) . +\frac{2\eta^2}{\lambda}\mathbb{E}\left[\left\|\nabla\widetilde{F}(\omega_k)-\nabla\widetilde{F}(\omega_k)\right\|^2\mid\widehat{a}_{\omega_k}\right]+\frac{\eta}{\lambda}\left(\mathcal{F}(\omega_k)-\mathbb{E}\left[F(\omega_{k+1})\mid\widehat{a}_{\omega_k}\right]\right). + λ 2 η 2 E [ ∇ F ( ω k ) − ∇ F ( ω k ) 2 ∣ a ω k ] + λ η ( F ( ω k ) − E [ F ( ω k + 1 ) ∣ a ω k ] ) .
Proof. Let k ≥ 0 k\ge0 k ≥ 0 . Write u k : = prox λ F ( ω k ) ∈ Ω u_k:=\operatorname{prox}_{\lambda F}(\omega_k)\in\Omega u k := prox λ F ( ω k ) ∈ Ω for the proximal point of F F F at ω k \omega_k ω k , that is, the unique minimizer in Equation (9) , so that ∥ u k − ω k ∥ = λ ∥ ∇ F λ ( ω k ) ∥ \|u_k-\omega_k\|=\lambda\|\nabla F_{\lambda}(\omega_k)\| ∥ u k − ω k ∥ = λ ∥∇ F λ ( ω k ) ∥ (Equation (10) ). Write also E k [ ⋅ ] : = E [ ⋅ ∣ a ^ ω k ] \mathbb{E}_k[\cdot]:=\mathbb{E}[\cdot\mid\widehat{a}_{\omega_k}] E k [ ⋅ ] := E [ ⋅ ∣ a ω k ] , with g k = ∇ F ~ ( ω k ) g_k=\nabla\widetilde{F}(\omega_k) g k = ∇ F ( ω k ) , v k : = E k [ ∇ F ~ ( ω k ) ] = ∇ F ~ ( ω k ) v_k:=\mathbb{E}_k[\nabla\widetilde{F}(\omega_k)]=\nabla\widetilde{F}(\omega_k) v k := E k [ ∇ F ( ω k )] = ∇ F ( ω k ) , b k : = v k − ∇ F ( ω k ) b_k:=v_k-\nabla F(\omega_k) b k := v k − ∇ F ( ω k ) , ξ k : = g k − v k \xi_k:=g_k-v_k ξ k := g k − v k , and Δ k : = ω k + 1 − ω k \Delta_k:=\omega_{k+1}-\omega_k Δ k := ω k + 1 − ω k .3
Step 1: A one-step bound on E k [ F λ ] \mathbb{E}_k[F_{\lambda}] E k [ F λ ] . By construction, it holds that ω k + 1 = Π Ω ( ω k − η g k ) = arg min y ∈ Ω h ( y ) \omega_{k+1}=\Pi_{\Omega}(\omega_k-\eta g_k)=\mathop{\arg\min}_{y\in\Omega}h(y) ω k + 1 = Π Ω ( ω k − η g k ) = arg min y ∈ Ω h ( y ) with h ( y ) : = ⟨ g k , y − ω k ⟩ + 1 2 η ∥ y − ω k ∥ 2 h(y):=\langle g_k,y-\omega_k\rangle+\frac{1}{2\eta}\|y-\omega_k\|^2 h ( y ) := ⟨ g k , y − ω k ⟩ + 2 η 1 ∥ y − ω k ∥ 2 , which is 1 η \frac{1}{\eta} η 1 -strongly convex. Since h h h is quadratic, for any y ∈ Ω y\in\Omega y ∈ Ω , one has the exact expansion h ( y ) = h ( ω k + 1 ) + ⟨ ∇ h ( ω k + 1 ) , y − ω k + 1 ⟩ + 1 2 η ∥ y − ω k + 1 ∥ 2 h(y)=h(\omega_{k+1})+\langle\nabla h(\omega_{k+1}),y-\omega_{k+1}\rangle+\frac{1}{2\eta}\|y-\omega_{k+1}\|^2 h ( y ) = h ( ω k + 1 ) + ⟨ ∇ h ( ω k + 1 ) , y − ω k + 1 ⟩ + 2 η 1 ∥ y − ω k + 1 ∥ 2 . In addition, the first-order optimality condition for the constrained minimizer ω k + 1 \omega_{k+1} ω k + 1 gives ⟨ ∇ h ( ω k + 1 ) , y − ω k + 1 ⟩ ≥ 0 \langle\nabla h(\omega_{k+1}),y-\omega_{k+1}\rangle\ge0 ⟨ ∇ h ( ω k + 1 ) , y − ω k + 1 ⟩ ≥ 0 for all y ∈ Ω y\in\Omega y ∈ Ω [22 Proposition 1.3] . Hence, taking y = u k ∈ Ω y=u_k\in\Omega y = u k ∈ Ω , we obtain:
h ( u k ) ≥ h ( ω k + 1 ) + 1 2 η ∥ u k − ω k + 1 ∥ 2 . h(u_k)\ge h(\omega_{k+1})+\frac{1}{2\eta}\|u_k-\omega_{k+1}\|^2. h ( u k ) ≥ h ( ω k + 1 ) + 2 η 1 ∥ u k − ω k + 1 ∥ 2 .
Expanding both sides via the definition of h h h and multiplying through by 2 η > 0 2\eta>0 2 η > 0 yields:
∥ u k − ω k + 1 ∥ 2 ≤ ∥ u k − ω k ∥ 2 − ∥ Δ k ∥ 2 + 2 η ⟨ g k , u k − ω k + 1 ⟩ . (11) \|u_k-\omega_{k+1}\|^2\le\|u_k-\omega_k\|^2-\|\Delta_k\|^2+2\eta\langle g_k,u_k-\omega_{k+1}\rangle. \tag*{(11)} ∥ u k − ω k + 1 ∥ 2 ≤ ∥ u k − ω k ∥ 2 − ∥ Δ k ∥ 2 + 2 η ⟨ g k , u k − ω k + 1 ⟩ . (11)
This transfers to F λ F_{\lambda} F λ as follows. Since u k ∈ Ω u_k\in\Omega u k ∈ Ω is feasible for the minimization defining F λ ( ω k + 1 ) F_{\lambda}(\omega_{k+1}) F λ ( ω k + 1 ) , but not necessarily its minimizer, we get:
F λ ( ω k + 1 ) ≤ F ( u k ) + 1 2 λ ∥ u k − ω k + 1 ∥ 2 , F_{\lambda}(\omega_{k+1})\le F(u_k)+\frac{1}{2\lambda}\|u_k-\omega_{k+1}\|^2, F λ ( ω k + 1 ) ≤ F ( u k ) + 2 λ 1 ∥ u k − ω k + 1 ∥ 2 ,
and by definition F ( u k ) + 1 2 λ ∥ u k − ω k ∥ 2 = F λ ( ω k ) F(u_k)+\frac{1}{2\lambda}\|u_k-\omega_k\|^2=F_{\lambda}(\omega_k) F ( u k ) + 2 λ 1 ∥ u k − ω k ∥ 2 = F λ ( ω k ) exactly. Multiplying Equation (11) by 1 2 λ \frac{1}{2\lambda} 2 λ 1 and substituting:
F λ ( ω k + 1 ) ≤ F λ ( ω k ) − 1 2 λ ∥ Δ k ∥ 2 + η λ ⟨ g k , u k − ω k + 1 ⟩ . (12) F_{\lambda}(\omega_{k+1})\le F_{\lambda}(\omega_k)-\frac{1}{2\lambda}\|\Delta_k\|^2+\frac{\eta}{\lambda}\langle g_k,u_k-\omega_{k+1}\rangle. \tag*{(12)} F λ ( ω k + 1 ) ≤ F λ ( ω k ) − 2 λ 1 ∥ Δ k ∥ 2 + λ η ⟨ g k , u k − ω k + 1 ⟩ . (12)
Since g k = ∇ F ( ω k ) + b k + ξ k g_k=\nabla F(\omega_k)+b_k+\xi_k g k = ∇ F ( ω k ) + b k + ξ k and u k − ω k + 1 = ( u k − ω k ) − Δ k u_k-\omega_{k+1}=(u_k-\omega_k)-\Delta_k u k − ω k + 1 = ( u k − ω k ) − Δ k , we have:
⟨ g k , u k − ω k + 1 ⟩ = ⟨ g k , u k − ω k ⟩ − ⟨ ∇ F ( ω k ) , Δ k ⟩ − ⟨ b k , Δ k ⟩ − ⟨ ξ k , Δ k ⟩ . \langle g_k,u_k-\omega_{k+1}\rangle=\langle g_k,u_k-\omega_k\rangle-\langle\nabla F(\omega_k),\Delta_k\rangle-\langle b_k,\Delta_k\rangle-\langle\xi_k,\Delta_k\rangle. ⟨ g k , u k − ω k + 1 ⟩ = ⟨ g k , u k − ω k ⟩ − ⟨ ∇ F ( ω k ) , Δ k ⟩ − ⟨ b k , Δ k ⟩ − ⟨ ξ k , Δ k ⟩ .
By L L L -smoothness of F F F (Proposition 2.1), it holds that − ⟨ ∇ F ( ω k ) , Δ k ⟩ ≤ F ( ω k ) − F ( ω k + 1 ) + L 2 ∥ Δ k ∥ 2 -\langle\nabla F(\omega_k),\Delta_k\rangle\le F(\omega_k)-F(\omega_{k+1})+\frac{L}{2}\|\Delta_k\|^2 − ⟨ ∇ F ( ω k ) , Δ k ⟩ ≤ F ( ω k ) − F ( ω k + 1 ) + 2 L ∥ Δ k ∥ 2 [72 Lemma 1.2.3] . By Young’s inequality with parameter 1 4 η \frac{1}{4\eta} 4 η 1 , − ⟨ b k , Δ k ⟩ ≤ 2 η ∥ b k ∥ 2 + 1 8 η ∥ Δ k ∥ 2 -\langle b_k,\Delta_k\rangle\le2\eta\|b_k\|^2+\frac{1}{8\eta}\|\Delta_k\|^2 − ⟨ b k , Δ k ⟩ ≤ 2 η ∥ b k ∥ 2 + 8 η 1 ∥ Δ k ∥ 2 and likewise − ⟨ ξ k , Δ k ⟩ ≤ 2 η ∥ ξ k ∥ 2 + 1 8 η ∥ Δ k ∥ 2 -\langle\xi_k,\Delta_k\rangle\le2\eta\|\xi_k\|^2+\frac{1}{8\eta}\|\Delta_k\|^2 − ⟨ ξ k , Δ k ⟩ ≤ 2 η ∥ ξ k ∥ 2 + 8 η 1 ∥ Δ k ∥ 2 . Multiplying these three bounds by η λ \frac{\eta}{\lambda} λ η and inserting into Equation (12) , the coefficient of ∥ Δ k ∥ 2 \|\Delta_k\|^2 ∥ Δ k ∥ 2 becomes 1 λ ( − 1 2 + η L 2 + 1 8 + 1 8 ) = 1 λ ( η L 2 − 1 4 ) \frac{1}{\lambda}\left(-\frac{1}{2}+\frac{\eta L}{2}+\frac{1}{8}+\frac{1}{8}\right)=\frac{1}{\lambda}\left(\frac{\eta L}{2}-\frac{1}{4}\right) λ 1 ( − 2 1 + 2 η L + 8 1 + 8 1 ) = λ 1 ( 2 η L − 4 1 ) , which is ≤ 0 \le0 ≤ 0 for η ≤ 1 2 L \eta\le\frac{1}{2L} η ≤ 2 L 1 , independently of λ \lambda λ . Dropping this nonpositive term, we get:
F λ ( ω k + 1 ) ≤ F λ ( ω k ) + η λ ⟨ g k , u k − ω k ⟩ + η λ ( F ( ω k ) − F ( ω k + 1 ) ) + 2 η 2 λ ( ∥ b k ∥ 2 + ∥ ξ k ∥ 2 ) . (13) F_{\lambda}(\omega_{k+1})\le F_{\lambda}(\omega_k)+\frac{\eta}{\lambda}\langle g_k,u_k-\omega_k\rangle+\frac{\eta}{\lambda}\left(F(\omega_k)-F(\omega_{k+1})\right)+\frac{2\eta^2}{\lambda}\left(\|b_k\|^2+\|\xi_k\|^2\right). \tag*{(13)} F λ ( ω k + 1 ) ≤ F λ ( ω k ) + λ η ⟨ g k , u k − ω k ⟩ + λ η ( F ( ω k ) − F ( ω k + 1 ) ) + λ 2 η 2 ( ∥ b k ∥ 2 + ∥ ξ k ∥ 2 ) . (13)
The quantities F λ ( ω k ) F_{\lambda}(\omega_k) F λ ( ω k ) , F ( ω k ) F(\omega_k) F ( ω k ) and b k b_k b k are a ^ ω k \widehat{a}_{\omega_k} a ω k -measurable, and E k [ ξ k ] = 0 \mathbb{E}_k[\xi_k]=0 E k [ ξ k ] = 0 since B g k B_g^k B g k is fresh, so that E k [ ⟨ g k , u k − ω k ⟩ ] = ⟨ v k , u k − ω k ⟩ \mathbb{E}_k[\langle g_k,u_k-\omega_k\rangle]=\langle v_k,u_k-\omega_k\rangle E k [⟨ g k , u k − ω k ⟩] = ⟨ v k , u k − ω k ⟩ . Taking E k \mathbb{E}_k E k in Equation (13) gives:
E k [ F λ ( ω k + 1 ) ] ≤ F λ ( ω k ) + η λ ⟨ v k , u k − ω k ⟩ + η λ ( F ( ω k ) − E k [ F ( ω k + 1 ) ] ) + 2 η 2 λ ( ∥ b k ∥ 2 + E k [ ∥ ξ k ∥ 2 ] ) . (14) \mathbb{E}_k[F_{\lambda}(\omega_{k+1})]\le F_{\lambda}(\omega_k)+\frac{\eta}{\lambda}\langle v_k,u_k-\omega_k\rangle+\frac{\eta}{\lambda}\left(F(\omega_k)-\mathbb{E}_k[F(\omega_{k+1})]\right)+\frac{2\eta^2}{\lambda}\left(\|b_k\|^2+\mathbb{E}_k[\|\xi_k\|^2]\right). \tag*{(14)} E k [ F λ ( ω k + 1 )] ≤ F λ ( ω k ) + λ η ⟨ v k , u k − ω k ⟩ + λ η ( F ( ω k ) − E k [ F ( ω k + 1 )] ) + λ 2 η 2 ( ∥ b k ∥ 2 + E k [ ∥ ξ k ∥ 2 ] ) . (14)
Step 2: Control on the cross term ⟨ v k , u k − ω k ⟩ \langle v_k,u_k-\omega_k\rangle ⟨ v k , u k − ω k ⟩ . Write v k = ∇ F ( ω k ) + b k v_k=\nabla F(\omega_k)+b_k v k = ∇ F ( ω k ) + b k , so:
⟨ v k , u k − ω k ⟩ = ⟨ ∇ F ( ω k ) , u k − ω k ⟩ + ⟨ b k , u k − ω k ⟩ . (15) \langle v_k,u_k-\omega_k\rangle=\langle\nabla F(\omega_k),u_k-\omega_k\rangle+\langle b_k,u_k-\omega_k\rangle. \tag*{(15)} ⟨ v k , u k − ω k ⟩ = ⟨ ∇ F ( ω k ) , u k − ω k ⟩ + ⟨ b k , u k − ω k ⟩ . (15)
Since F F F is L \mathcal{L} L -smooth (Proposition 2.1), [72 Lemma 1.2.3] gives:
F ( u k ) ≥ F ( ω k ) + ⟨ ∇ F ( ω k ) , u k − ω k ⟩ − L 2 ∥ u k − ω k ∥ 2 , F(u_k)\ge F(\omega_k)+\langle\nabla F(\omega_k),u_k-\omega_k\rangle-\frac{\mathcal{L}}{2}\lVert u_k-\omega_k\rVert^2, F ( u k ) ≥ F ( ω k ) + ⟨ ∇ F ( ω k ) , u k − ω k ⟩ − 2 L ∥ u k − ω k ∥ 2 ,
so
⟨ ∇ F ( ω k ) , u k − ω k ⟩ ≤ F ( u k ) − F ( ω k ) + L 2 ∥ u k − ω k ∥ 2 . (16) \langle\nabla F(\omega_k),u_k-\omega_k\rangle\le F(u_k)-F(\omega_k)+\frac{\mathcal{L}}{2}\lVert u_k-\omega_k\rVert^2. \tag*{(16)} ⟨ ∇ F ( ω k ) , u k − ω k ⟩ ≤ F ( u k ) − F ( ω k ) + 2 L ∥ u k − ω k ∥ 2 . (16)
We have by Young’s inequality with parameter 1 2 ( λ − 1 − L ) > 0 \frac{1}{2}(\lambda^{-1}-\mathcal{L})>0 2 1 ( λ − 1 − L ) > 0 :
⟨ b k , u k − ω k ⟩ ≤ 1 λ − 1 − L ∥ b k ∥ 2 + λ − 1 − L 4 ∥ u k − ω k ∥ 2 . (17) \langle b_k,u_k-\omega_k\rangle\le\frac{1}{\lambda^{-1}-\mathcal{L}}\lVert b_k\rVert^2+\frac{\lambda^{-1}-\mathcal{L}}{4}\lVert u_k-\omega_k\rVert^2. \tag*{(17)} ⟨ b k , u k − ω k ⟩ ≤ λ − 1 − L 1 ∥ b k ∥ 2 + 4 λ − 1 − L ∥ u k − ω k ∥ 2 . (17)
The map ψ k ( y ) : = F ( y ) + 1 2 λ ∥ y − ω k ∥ 2 \psi_k(y):=F(y)+\frac{1}{2\lambda}\lVert y-\omega_k\rVert^2 ψ k ( y ) := F ( y ) + 2 λ 1 ∥ y − ω k ∥ 2 is ( λ − 1 − L ) (\lambda^{-1}-\mathcal{L}) ( λ − 1 − L ) -strongly convex on Ω \Omega Ω (sum of the convex function F + L 2 ∥ ⋅ ∥ 2 F+\frac{\mathcal{L}}{2}\lVert\cdot\rVert^2 F + 2 L ∥ ⋅ ∥ 2 , up to an affine term, and a quadratic of curvature λ − 1 − L \lambda^{-1}-\mathcal{L} λ − 1 − L , which is positive since λ < L − 1 \lambda<\mathcal{L}^{-1} λ < L − 1 ), with unique minimizer u k u_k u k . Comparing to the feasible point y = ω k y=\omega_k y = ω k , strong convexity gives:
ψ k ( ω k ) ≥ ψ k ( u k ) + λ − 1 − L 2 ∥ ω k − u k ∥ 2 . \psi_k(\omega_k)\ge\psi_k(u_k)+\frac{\lambda^{-1}-\mathcal{L}}{2}\lVert\omega_k-u_k\rVert^2. ψ k ( ω k ) ≥ ψ k ( u k ) + 2 λ − 1 − L ∥ ω k − u k ∥ 2 .
Since ψ k ( ω k ) = F ( ω k ) \psi_k(\omega_k)=F(\omega_k) ψ k ( ω k ) = F ( ω k ) and ψ k ( u k ) = F ( u k ) + 1 2 λ ∥ u k − ω k ∥ 2 \psi_k(u_k)=F(u_k)+\frac{1}{2\lambda}\lVert u_k-\omega_k\rVert^2 ψ k ( u k ) = F ( u k ) + 2 λ 1 ∥ u k − ω k ∥ 2 , this rearranges to:
F ( u k ) − F ( ω k ) ≤ − ( λ − 1 − L 2 ) ∥ u k − ω k ∥ 2 . (18) F(u_k)-F(\omega_k)\le-\left(\lambda^{-1}-\frac{\mathcal{L}}{2}\right)\lVert u_k-\omega_k\rVert^2. \tag*{(18)} F ( u k ) − F ( ω k ) ≤ − ( λ − 1 − 2 L ) ∥ u k − ω k ∥ 2 . (18)
Adding (16) and (17) reconstructs the right-hand side of (15) . Substituting (18) for F ( u k ) − F ( ω k ) F(u_k)-F(\omega_k) F ( u k ) − F ( ω k ) and combining the three ∥ u k − ω k ∥ 2 \lVert u_k-\omega_k\rVert^2 ∥ u k − ω k ∥ 2 coefficients, − ( λ − 1 − L 2 ) + L 2 + λ − 1 − L 4 = − 3 4 ( λ − 1 − L ) -\left(\lambda^{-1}-\frac{\mathcal{L}}{2}\right)+\frac{\mathcal{L}}{2}+\frac{\lambda^{-1}-\mathcal{L}}{4}=-\frac{3}{4}(\lambda^{-1}-\mathcal{L}) − ( λ − 1 − 2 L ) + 2 L + 4 λ − 1 − L = − 4 3 ( λ − 1 − L ) :
⟨ v k , u k − ω k ⟩ ≤ − 3 4 ( λ − 1 − L ) ∥ u k − ω k ∥ 2 + 1 λ − 1 − L ∥ b k ∥ 2 . (19) \langle v_k,u_k-\omega_k\rangle\le-\frac{3}{4}(\lambda^{-1}-\mathcal{L})\lVert u_k-\omega_k\rVert^2+\frac{1}{\lambda^{-1}-\mathcal{L}}\lVert b_k\rVert^2. \tag*{(19)} ⟨ v k , u k − ω k ⟩ ≤ − 4 3 ( λ − 1 − L ) ∥ u k − ω k ∥ 2 + λ − 1 − L 1 ∥ b k ∥ 2 . (19)
Step 3: Assembling the descent inequality. Plugging (19) into (14) and using ∥ u k − ω k ∥ 2 = λ 2 ∥ ∇ F λ ( ω k ) ∥ 2 \lVert u_k-\omega_k\rVert^2=\lambda^2\lVert\nabla F_\lambda(\omega_k)\rVert^2 ∥ u k − ω k ∥ 2 = λ 2 ∥ ∇ F λ ( ω k ) ∥ 2 , the descent term becomes:
− η λ ⋅ 3 4 ( λ − 1 − L ) ⋅ λ 2 ∥ ∇ F λ ( ω k ) ∥ 2 = − 3 η 4 ( 1 − λ L ) ∥ ∇ F λ ( ω k ) ∥ 2 , -\frac{\eta}{\lambda}\cdot\frac{3}{4}(\lambda^{-1}-\mathcal{L})\cdot\lambda^2\lVert\nabla F_\lambda(\omega_k)\rVert^2=-\frac{3\eta}{4}(1-\lambda\mathcal{L})\lVert\nabla F_\lambda(\omega_k)\rVert^2, − λ η ⋅ 4 3 ( λ − 1 − L ) ⋅ λ 2 ∥ ∇ F λ ( ω k ) ∥ 2 = − 4 3 η ( 1 − λ L ) ∥ ∇ F λ ( ω k ) ∥ 2 ,
while the coefficient of ∥ b k ∥ 2 \lVert b_k\rVert^2 ∥ b k ∥ 2 becomes η λ ⋅ 1 λ − 1 − L + 2 η 2 λ = η ( 1 1 − λ L + 2 η λ ) \frac{\eta}{\lambda}\cdot\frac{1}{\lambda^{-1}-\mathcal{L}}+\frac{2\eta^2}{\lambda}=\eta\left(\frac{1}{1-\lambda\mathcal{L}}+\frac{2\eta}{\lambda}\right) λ η ⋅ λ − 1 − L 1 + λ 2 η 2 = η ( 1 − λ L 1 + λ 2 η ) . Hence:
E k [ F λ ( ω k + 1 ) ] ≤ F λ ( ω k ) − 3 η 4 ( 1 − λ L ) ∥ ∇ F λ ( ω k ) ∥ 2 + η ( 1 1 − λ L + 2 η λ ) ∥ b k ∥ 2 + 2 η 2 λ E k [ ∥ ξ k ∥ 2 ] + η λ ( F ( ω k ) − E k [ F ( ω k + 1 ) ] ) . \begin{aligned}
\mathbb{E}_k[F_\lambda(\omega_{k+1})]\le{}&F_\lambda(\omega_k)-\frac{3\eta}{4}(1-\lambda\mathcal{L})\lVert\nabla F_\lambda(\omega_k)\rVert^2\\
&+\eta\left(\frac{1}{1-\lambda\mathcal{L}}+\frac{2\eta}{\lambda}\right)\lVert b_k\rVert^2\\
&+\frac{2\eta^2}{\lambda}\mathbb{E}_k[\lVert\xi_k\rVert^2]+\frac{\eta}{\lambda}\left(F(\omega_k)-\mathbb{E}_k[F(\omega_{k+1})]\right).
\end{aligned} E k [ F λ ( ω k + 1 )] ≤ F λ ( ω k ) − 4 3 η ( 1 − λ L ) ∥ ∇ F λ ( ω k ) ∥ 2 + η ( 1 − λ L 1 + λ 2 η ) ∥ b k ∥ 2 + λ 2 η 2 E k [∥ ξ k ∥ 2 ] + λ η ( F ( ω k ) − E k [ F ( ω k + 1 )] ) .
Recalling b k = ∇ ~ F ( ω k ) − ∇ F ( ω k ) b_k=\widetilde{\nabla}F(\omega_k)-\nabla F(\omega_k) b k = ∇ F ( ω k ) − ∇ F ( ω k ) and ξ k = ∇ ~ F ( ω k ) − ∇ ~ F ( ω k ) \xi_k=\widetilde{\nabla}F(\omega_k)-\widetilde{\nabla}F(\omega_k) ξ k = ∇ F ( ω k ) − ∇ F ( ω k ) gives exactly the claimed inequality.
Theorem C.5 (Convergence guarantee of projected mini-batch biased SGD). Fix ω 0 ∈ Ω \omega_0\in\Omega ω 0 ∈ Ω . Let F ⋆ : = min ω ∈ Ω F ( ω ) F^\star:=\min_{\omega\in\Omega}F(\omega) F ⋆ := min ω ∈ Ω F ( ω ) and Δ F : = F ( ω 0 ) − F ⋆ \Delta_F:=F(\omega_0)-F^\star Δ F := F ( ω 0 ) − F ⋆ . Let { ω k } k ≥ 0 \{\omega_k\}_{k\ge0} { ω k } k ≥ 0 be generated by ω k + 1 = Π Ω ( ω k − η ∇ ~ F ( ω k ) ) \omega_{k+1}=\Pi_\Omega(\omega_k-\eta\widetilde{\nabla}F(\omega_k)) ω k + 1 = Π Ω ( ω k − η ∇ F ( ω k )) with stepsize 0 < η ≤ 1 2 L 0<\eta\le\frac{1}{2\mathcal{L}} 0 < η ≤ 2 L 1 , where L \mathcal{L} L is the smoothness parameter of F F F provided in Proposition 2.1. Under Assumptions A and B, for any K ≥ 1 K\ge1 K ≥ 1 and any λ ∈ ( 0 , L − 1 ) \lambda\in(0,\mathcal{L}^{-1}) λ ∈ ( 0 , L − 1 ) , the iterates { ω k } k ≥ 0 \{\omega_k\}_{k\ge0} { ω k } k ≥ 0 satisfy:
1 K ∑ k = 0 K − 1 E [ ∥ ∇ F λ ( ω k ) ∥ 2 ] ≤ 4 3 ( 1 − λ L ) [ ( 1 η + 1 λ ) Δ F K + 2 G 2 ( 1 1 − λ L + 2 η λ ) ( e h + e b ) + 4 G 2 η λ n g ( D + B ) 2 + 2 ( e h + e b ) ] . \frac{1}{K}\sum_{k=0}^{K-1}\mathbb{E}\left[\lVert\nabla F_\lambda(\omega_k)\rVert^2\right]\le\frac{4}{3(1-\lambda\mathcal{L})}\left[\left(\frac{1}{\eta}+\frac{1}{\lambda}\right)\frac{\Delta_F}{K}+2G^2\left(\frac{1}{1-\lambda\mathcal{L}}+\frac{2\eta}{\lambda}\right)(e_h+e_b)+\frac{4G^2\eta}{\lambda n_g}(D+B)^2+2(e_h+e_b)\right]. K 1 k = 0 ∑ K − 1 E [ ∥ ∇ F λ ( ω k ) ∥ 2 ] ≤ 3 ( 1 − λ L ) 4 [ ( η 1 + λ 1 ) K Δ F + 2 G 2 ( 1 − λ L 1 + λ 2 η ) ( e h + e b ) + λ n g 4 G 2 η ( D + B ) 2 + 2 ( e h + e b ) ] . where e h = E [ sup ω ∈ Ω ∥ h ^ ω − h ω ⋆ ∥ L 2 ( X ) 2 ] e_h=\mathbb{E}[\sup_{\omega\in\Omega}\lVert\hat h_\omega-h^\star_\omega\rVert^2_{L^2(\mathcal{X})}] e h = E [ sup ω ∈ Ω ∥ h ^ ω − h ω ⋆ ∥ L 2 ( X ) 2 ] , e b = E [ ∥ b ^ − b ⋆ ∥ L 2 ( X ) 2 ] e_b=\mathbb{E}[\lVert\hat b-b^\star\rVert^2_{L^2(\mathcal{X})}] e b = E [∥ b ^ − b ⋆ ∥ L 2 ( X ) 2 ] , and B B B , D D D , G G G are the constants of Assumptions A and B. The expectation is over all sources of randomness, namely the samples D i n \mathcal{D}_{\mathrm{in}} D in and D o u t \mathcal{D}_{\mathrm{out}} D out used to construct h ^ ω \hat h_\omega h ^ ω and b ^ \hat b b ^ , and the mini-batches { B g k } k ≥ 0 \{B_g^k\}_{k\ge0} { B g k } k ≥ 0 ; the expectations defining e h e_h e h and e b e_b e b are over D i n \mathcal{D}_{\mathrm{in}} D in and D o u t \mathcal{D}_{\mathrm{out}} D out only.
Proof. Let k ≥ 0 k \ge0 k ≥ 0 , and recall that b k : = ∇ F ~ ( ω k ) − ∇ F ( ω k ) b_k := \widetilde{\nabla F}(\omega_k) - \nabla F(\omega_k) b k := ∇ F ( ω k ) − ∇ F ( ω k ) is the bias of the plug-in hypergradient estimator ∇ F ~ ( ω k ) \widetilde{\nabla F}(\omega_k) ∇ F ( ω k ) conditional on ω ^ k \widehat{\omega}_k ω k , and ξ k : = ∇ F ~ ( ω k ) − ∇ F ~ ( ω k ) \xi_k := \widetilde{\nabla F}(\omega_k) - \widetilde{\nabla F}(\omega_k) ξ k := ∇ F ( ω k ) − ∇ F ( ω k ) is its conditionally zero-mean noise.
Taking full expectation in Lemma C.4 and summing over k = 0 , … , K − 1 k = 0,\ldots,K-1 k = 0 , … , K − 1 , both the F λ F_\lambda F λ - and the F F F -sums telescope, and ω 0 \omega_0 ω 0 is deterministic, so:
3 η 4 ( 1 − λ L ) ∑ k = 0 K − 1 E [ ∥ ∇ F λ ( ω k ) ∥ 2 ] ≤ F λ ( ω 0 ) − E [ F λ ( ω K ) ] + η λ ( F ( ω 0 ) − E [ F ( ω K ) ] ) + η ( 1 1 − λ L + 2 η λ ) ∑ k = 0 K − 1 E [ ∥ b k ∥ 2 ] \frac{3\eta}{4}(1-\lambda\mathcal{L})\sum_{k=0}^{K-1}\mathbb{E}\left[\left\|\nabla F_\lambda(\omega_k)\right\|^2\right]
\leq F_\lambda(\omega_0)-\mathbb{E}[F_\lambda(\omega_K)]+\frac{\eta}{\lambda}\left(F(\omega_0)-\mathbb{E}[F(\omega_K)]\right)+\eta\left(\frac{1}{1-\lambda\mathcal{L}}+\frac{2\eta}{\lambda}\right)\sum_{k=0}^{K-1}\mathbb{E}\left[\left\|b_k\right\|^2\right] 4 3 η ( 1 − λ L ) k = 0 ∑ K − 1 E [ ∥ ∇ F λ ( ω k ) ∥ 2 ] ≤ F λ ( ω 0 ) − E [ F λ ( ω K )] + λ η ( F ( ω 0 ) − E [ F ( ω K )] ) + η ( 1 − λ L 1 + λ 2 η ) k = 0 ∑ K − 1 E [ ∥ b k ∥ 2 ]
+ 2 η 2 λ ∑ k = 0 K − 1 E [ ∥ ξ k ∥ 2 ] . \qquad+\frac{2\eta^2}{\lambda}\sum_{k=0}^{K-1}\mathbb{E}\left[\left\|\xi_k\right\|^2\right]. + λ 2 η 2 k = 0 ∑ K − 1 E [ ∥ ξ k ∥ 2 ] .
Since F λ ( ω ) ≤ F ( ω ) F_\lambda(\omega)\leq F(\omega) F λ ( ω ) ≤ F ( ω ) and F λ ( ω ) , F ( ω ) ≥ F ⋆ F_\lambda(\omega),F(\omega)\geq F^\star F λ ( ω ) , F ( ω ) ≥ F ⋆ for every ω ∈ Ω \omega\in\Omega ω ∈ Ω , the two differences on the right-hand side are bounded by ( 1 + η λ ) Δ F (1+\frac{\eta}{\lambda})\Delta_F ( 1 + λ η ) Δ F . Dividing by 3 η 4 ( 1 − λ L ) K \frac{3\eta}{4}(1-\lambda\mathcal{L})K 4 3 η ( 1 − λ L ) K , which is positive because λ < L − 1 \lambda<\mathcal{L}^{-1} λ < L − 1 , we get:
1 K ∑ k = 0 K − 1 E [ ∥ ∇ F λ ( ω k ) ∥ 2 ] ≤ 4 3 ( 1 − λ L ) ( 1 η + 1 λ ) Δ F K + 4 3 ( 1 − λ L ) ( 1 1 − λ L + 2 η λ ) 1 K ∑ k = 0 K − 1 E [ ∥ b k ∥ 2 ] \frac{1}{K}\sum_{k=0}^{K-1}\mathbb{E}\left[\left\|\nabla F_\lambda(\omega_k)\right\|^2\right]
\leq\frac{4}{3(1-\lambda\mathcal{L})}\left(\frac{1}{\eta}+\frac{1}{\lambda}\right)\frac{\Delta_F}{K}+\frac{4}{3(1-\lambda\mathcal{L})}\left(\frac{1}{1-\lambda\mathcal{L}}+\frac{2\eta}{\lambda}\right)\frac{1}{K}\sum_{k=0}^{K-1}\mathbb{E}\left[\left\|b_k\right\|^2\right] K 1 k = 0 ∑ K − 1 E [ ∥ ∇ F λ ( ω k ) ∥ 2 ] ≤ 3 ( 1 − λ L ) 4 ( η 1 + λ 1 ) K Δ F + 3 ( 1 − λ L ) 4 ( 1 − λ L 1 + λ 2 η ) K 1 k = 0 ∑ K − 1 E [ ∥ b k ∥ 2 ]
+ 8 η 3 λ ( 1 − λ L ) K ∑ k = 0 K − 1 E [ ∥ ξ k ∥ 2 ] . (20) \qquad+\frac{8\eta}{3\lambda(1-\lambda\mathcal{L})K}\sum_{k=0}^{K-1}\mathbb{E}\left[\left\|\xi_k\right\|^2\right]. \tag*{(20)} + 3 λ ( 1 − λ L ) K 8 η k = 0 ∑ K − 1 E [ ∥ ξ k ∥ 2 ] . (20)
The iterate ω k \omega_k ω k depends on D i n D_{\mathrm{in}} D in and D o u t D_{\mathrm{out}} D out , so we bound both error terms uniformly over Ω \Omega Ω . By Lemmas C.2 and C.3, for every k k k , we have:
E [ ∥ b k ∥ 2 ] ≤ 2 G 2 ( e h + e b ) and E [ ∥ ξ k ∥ 2 ] ≤ 2 G 2 n g [ ( D + B ) 2 + 2 ( e h + e b ) ] . \mathbb{E}\left[\left\|b_k\right\|^2\right]\leq2G^2(e_h+e_b)
\qquad\text{and}\qquad
\mathbb{E}\left[\left\|\xi_k\right\|^2\right]\leq\frac{2G^2}{n_g}\left[(D+B)^2+2(e_h+e_b)\right]. E [ ∥ b k ∥ 2 ] ≤ 2 G 2 ( e h + e b ) and E [ ∥ ξ k ∥ 2 ] ≤ n g 2 G 2 [ ( D + B ) 2 + 2 ( e h + e b ) ] .
Substituting these bounds into Equation (20) gives exactly the claimed bound.
Proof of Theorem 2.4. We specialize Theorem C.5 to λ = 1 2 L \lambda=\frac{1}{2\mathcal{L}} λ = 2 L 1 , which is admissible since 1 2 L < L − 1 \frac{1}{2\mathcal{L}}<\mathcal{L}^{-1} 2 L 1 < L − 1 . With this choice, λ L = 1 2 \lambda\mathcal{L}=\frac{1}{2} λ L = 2 1 , so the prefactor equals 4 3 ( 1 − λ L ) = 8 3 \frac{4}{3(1-\lambda\mathcal{L})}=\frac{8}{3} 3 ( 1 − λ L ) 4 = 3 8 and 1 λ = 2 L \frac{1}{\lambda}=2\mathcal{L} λ 1 = 2 L . Using η ≤ 1 2 L \eta\leq\frac{1}{2\mathcal{L}} η ≤ 2 L 1 , that is 2 L ≤ 1 η 2\mathcal{L}\leq\frac{1}{\eta} 2 L ≤ η 1 and η L ≤ 1 2 \eta\mathcal{L}\leq\frac{1}{2} η L ≤ 2 1 , the three brackets are bounded by:
1 η + 1 λ = 1 η + 2 L ≤ 2 η , 2 G 2 ( 1 1 − λ L + 2 η λ ) = 2 G 2 ( 2 + 4 η L ) ≤ 8 G 2 , 4 G 2 η λ n g = 8 L G 2 η n g . \frac{1}{\eta}+\frac{1}{\lambda}=\frac{1}{\eta}+2\mathcal{L}\leq\frac{2}{\eta},
\qquad
2G^2\left(\frac{1}{1-\lambda\mathcal{L}}+\frac{2\eta}{\lambda}\right)=2G^2(2+4\eta\mathcal{L})\leq8G^2,
\qquad
\frac{4G^2\eta}{\lambda n_g}=\frac{8\mathcal{L}G^2\eta}{n_g}. η 1 + λ 1 = η 1 + 2 L ≤ η 2 , 2 G 2 ( 1 − λ L 1 + λ 2 η ) = 2 G 2 ( 2 + 4 η L ) ≤ 8 G 2 , λ n g 4 G 2 η = n g 8 L G 2 η .
so that Theorem C.5 gives:
1 K ∑ k = 0 K − 1 E [ ∥ ∇ F 1 / 2 L ( ω k ) ∥ 2 ] ≤ 16 Δ F 3 η K + 64 G 2 ( e h + e b ) 3 + 64 L G 2 η 3 n g [ ( D + B ) 2 + 2 ( e h + e b ) ] . (21) \frac{1}{K}\sum_{k=0}^{K-1}\mathbb{E}\left[\left\|\nabla F_{1/2\mathcal{L}}(\omega_k)\right\|^2\right]\leq\frac{16\Delta_F}{3\eta K}+\frac{64G^2(e_h+e_b)}{3}+\frac{64\mathcal{L}G^2\eta}{3n_g}\left[(D+B)^2+2(e_h+e_b)\right]. \tag*{(21)} K 1 k = 0 ∑ K − 1 E [ ∇ F 1/2 L ( ω k ) 2 ] ≤ 3 η K 16 Δ F + 3 64 G 2 ( e h + e b ) + 3 n g 64 L G 2 η [ ( D + B ) 2 + 2 ( e h + e b ) ] . (21)
It remains to pass from the Moreau envelope to the gradient mapping. Applying [30 Section 2.2] , we obtain, for every ω ∈ Ω \omega\in\Omega ω ∈ Ω :
∥ G 1 / L ( ω ) ∥ ≤ 3 2 ( 1 + 1 2 ) ∥ ∇ F 1 / 2 L ( ω ) ∥ , hence ∥ G 1 / L ( ω ) ∥ 2 ≤ c ∥ ∇ F 1 / 2 L ( ω ) ∥ 2 , \left\|\mathcal{G}_{1/\mathcal{L}}(\omega)\right\|\leq\frac{3}{2}\left(1+\frac{1}{\sqrt{2}}\right)\left\|\nabla F_{1/2\mathcal{L}}(\omega)\right\|,
\qquad\text{hence}\qquad
\left\|\mathcal{G}_{1/\mathcal{L}}(\omega)\right\|^2\leq c\left\|\nabla F_{1/2\mathcal{L}}(\omega)\right\|^2, G 1/ L ( ω ) ≤ 2 3 ( 1 + 2 1 ) ∇ F 1/2 L ( ω ) , hence G 1/ L ( ω ) 2 ≤ c ∇ F 1/2 L ( ω ) 2 ,
with c = 9 4 ( 3 2 + 2 ) c=\frac{9}{4}\left(\frac{3}{2}+\sqrt{2}\right) c = 4 9 ( 2 3 + 2 ) . Taking expectations, averaging over k = 0 , … , K − 1 k=0,\ldots,K-1 k = 0 , … , K − 1 and combining with Equation (21) gives the claimed bound.
Corollary C.6 (Balanced stepsize). Let the assumptions of Theorem 2.4 hold, assume Δ F > 0 \Delta_F > 0 Δ F > 0 , and set σ 2 ≔ ( D + B ) 2 + 2 ( e h + e b ) \sigma^2 \coloneqq(D+B)^2+2(e_h+e_b) σ 2 : = ( D + B ) 2 + 2 ( e h + e b ) . For any K ≥ 1 K \ge1 K ≥ 1 , the choice
η = min { 1 2 L , η ⋆ } , η ⋆ ≔ 1 2 G σ Δ F n g L K , (22) \eta=\min\left\{\frac{1}{2\mathcal{L}},\eta_\star\right\},\qquad\eta_\star\coloneqq\frac{1}{2G\sigma}\sqrt{\frac{\Delta_F n_g}{\mathcal{L}K}}, \tag*{(22)} η = min { 2 L 1 , η ⋆ } , η ⋆ : = 2 G σ 1 L K Δ F n g , (22) is admissible and yields:
1 K ∑ k = 0 K − 1 E [ ∥ G 1 / L ( ω k ) ∥ 2 ] ≤ 2 c [ L Δ F K + 2 G σ L Δ F K n g + 2 G 2 ( e h + e b ) ] , (23) \frac{1}{K}\sum_{k=0}^{K-1}\mathbb{E}\left[\left\|\mathcal{G}_{1/\mathcal{L}}(\omega_k)\right\|^2\right]\le2c\left[\frac{\mathcal{L}\Delta_F}{K}+2G\sigma\sqrt{\frac{\mathcal{L}\Delta_F}{Kn_g}}+2G^2(e_h+e_b)\right], \tag*{(23)} K 1 k = 0 ∑ K − 1 E [ G 1/ L ( ω k ) 2 ] ≤ 2 c [ K L Δ F + 2 G σ K n g L Δ F + 2 G 2 ( e h + e b ) ] , (23) where c = 18 + 12 2 c=18+12\sqrt{2} c = 18 + 12 2 is the constant of Theorem 2.4 .
Proof. Write the bound of Theorem 2.4 as c [ c o p t / η + c v a r η + c b i a s ] c\left[c_{\mathrm{opt}}/\eta+c_{\mathrm{var}}\eta+c_{\mathrm{bias}}\right] c [ c opt / η + c var η + c bias ] , valid for every 0 < η ≤ 1 2 L 0<\eta\le\frac{1}{2\mathcal{L}} 0 < η ≤ 2 L 1 , where:
c o p t = Δ F K , c v a r = 4 L G 2 σ 2 n g , c b i a s = 4 G 2 ( e h + e b ) , c = 18 + 12 2 . c_{\mathrm{opt}}=\frac{\Delta_F}{K},\qquad c_{\mathrm{var}}=\frac{4\mathcal{L}G^2\sigma^2}{n_g},\qquad c_{\mathrm{bias}}=4G^2(e_h+e_b),\qquad c=18+12\sqrt{2}. c opt = K Δ F , c var = n g 4 L G 2 σ 2 , c bias = 4 G 2 ( e h + e b ) , c = 18 + 12 2 .
Since the constant c c c does not depend on η \eta η , minimizing this bound amounts to minimizing η ↦ c o p t / η + c v a r η \eta\mapsto c_{\mathrm{opt}}/\eta+c_{\mathrm{var}}\eta η ↦ c opt / η + c var η , which is convex on ( 0 , + ∞ ) (0,+\infty) ( 0 , + ∞ ) and attains its minimum at c o p t / c v a r \sqrt{c_{\mathrm{opt}}/c_{\mathrm{var}}} c opt / c var , and
c o p t c v a r = Δ F / K 4 L G 2 σ 2 / n g = 1 2 G σ Δ F n g L K = η ⋆ , \sqrt{\frac{c_{\mathrm{opt}}}{c_{\mathrm{var}}}}=\sqrt{\frac{\Delta_F/K}{4\mathcal{L}G^2\sigma^2/n_g}}=\frac{1}{2G\sigma}\sqrt{\frac{\Delta_F n_g}{\mathcal{L}K}}=\eta_\star, c var c opt = 4 L G 2 σ 2 / n g Δ F / K = 2 G σ 1 L K Δ F n g = η ⋆ ,
so that Equation (22) reads η = min { 1 2 L , η ⋆ } \eta=\min\{\frac{1}{2\mathcal{L}},\eta_\star\} η = min { 2 L 1 , η ⋆ } and is admissible in Theorem 2.4 . Since 1 η = max { 2 L , η ⋆ − 1 } ≤ 2 L + η ⋆ − 1 \frac{1}{\eta}=\max\{2\mathcal{L},\eta_\star^{-1}\}\le2\mathcal{L}+\eta_\star^{-1} η 1 = max { 2 L , η ⋆ − 1 } ≤ 2 L + η ⋆ − 1 and η ≤ η ⋆ \eta\le\eta_\star η ≤ η ⋆ , we obtain:
c o p t η + c v a r η ≤ 2 L c o p t + c o p t η ⋆ + c v a r η ⋆ = 2 L c o p t + 2 c o p t c v a r , \frac{c_{\mathrm{opt}}}{\eta}+c_{\mathrm{var}}\eta\le2\mathcal{L}c_{\mathrm{opt}}+\frac{c_{\mathrm{opt}}}{\eta_\star}+c_{\mathrm{var}}\eta_\star=2\mathcal{L}c_{\mathrm{opt}}+2\sqrt{c_{\mathrm{opt}}c_{\mathrm{var}}}, η c opt + c var η ≤ 2 L c opt + η ⋆ c opt + c var η ⋆ = 2 L c opt + 2 c opt c var ,
where we used c o p t / η ⋆ = c o p t c v a r / c o p t = c o p t c v a r c_{\mathrm{opt}}/\eta_\star=c_{\mathrm{opt}}\sqrt{c_{\mathrm{var}}/c_{\mathrm{opt}}}=\sqrt{c_{\mathrm{opt}}c_{\mathrm{var}}} c opt / η ⋆ = c opt c var / c opt = c opt c var and c v a r η ⋆ = c v a r c o p t / c v a r = c o p t c v a r c_{\mathrm{var}}\eta_\star=c_{\mathrm{var}}\sqrt{c_{\mathrm{opt}}/c_{\mathrm{var}}}=\sqrt{c_{\mathrm{opt}}c_{\mathrm{var}}} c var η ⋆ = c var c opt / c var = c opt c var . Finally:
2 L c o p t = 2 L Δ F K , 2 c o p t c v a r = 2 Δ F K ⋅ 4 L G 2 σ 2 n g = 4 G σ L Δ F K n g , 2\mathcal{L}c_{\mathrm{opt}}=\frac{2\mathcal{L}\Delta_F}{K},\qquad2\sqrt{c_{\mathrm{opt}}c_{\mathrm{var}}}=2\sqrt{\frac{\Delta_F}{K}\cdot\frac{4\mathcal{L}G^2\sigma^2}{n_g}}=4G\sigma\sqrt{\frac{\mathcal{L}\Delta_F}{Kn_g}}, 2 L c opt = K 2 L Δ F , 2 c opt c var = 2 K Δ F ⋅ n g 4 L G 2 σ 2 = 4 G σ K n g L Δ F ,
which, together with c b i a s c_{\mathrm{bias}} c bias and after multiplying by c c c , gives Equation (23) .
Remark C.7 (Dominant regime). The second branch of Equation (22) is active as soon as K ≥ L Δ F n g G 2 σ 2 K\ge\frac{\mathcal{L}\Delta_F n_g}{G^2\sigma^2} K ≥ G 2 σ 2 L Δ F n g . In that regime, the ratio of the first term of Equation (23) to the second equals 1 2 G σ L Δ F n g K \frac{1}{2G\sigma}\sqrt{\frac{\mathcal{L}\Delta_F n_g}{K}} 2 G σ 1 K L Δ F n g , which is at most 1 2 \frac{1}{2} 2 1 , so that the algorithmic error is of order G σ L Δ F K n g G\sigma\sqrt{\frac{\mathcal{L}\Delta_F}{Kn_g}} G σ K n g L Δ F . Only the total number K n g Kn_g K n g of hypergradient samples then matters: a given accuracy is reached either with many small batches or with fewer larger ones, as long as K K K remains above this threshold.
Generalization Bounds for (OBO) in the NTK Regime
Appendix D proves the bounds on e h e_h e h and e b e_b e b stated in Proposition 3.1. The bound on e b e_b e b follows directly from the single-regression result of Proposition E.25. The bound on e h e_h e h is more delicate because of the supremum over ω \omega ω , and proceeds in four steps: (i) we split the error of h ^ ω \hat{h}_\omega h ^ ω into the gap between the finite-width network and its infinite-width NTK counterpart, which is controlled uniformly in ω \omega ω since the truncation event does not depend on ω \omega ω , and the error of the NTK predictor itself; (ii) we show that the NTK predictor is linear in the target vector, so that its error is Lipschitz continuous in ω \omega ω ; (iii) we control the NTK error on a finite net of Ω \Omega Ω by a union bound and extend it to all of Ω \Omega Ω by Lipschitz continuity; (iv) we choose the confidence level δ \delta δ to balance the resulting terms.
Throughout Appendix D, we fix a nonempty, compact and convex parameter set Ω ⊂ R d \Omega\subset\mathbb{R}^d Ω ⊂ R d with diameter diam ( Ω ) \operatorname{diam}(\Omega) diam ( Ω ) . We write K = [ K N T K ( x i , x j ) ] i , j = 1 n i n ∈ R n i n × n i n K=[K^{\mathrm{NTK}}(x_i,x_j)]_{i,j=1}^{n_{\mathrm{in}}}\in\mathbb{R}^{n_{\mathrm{in}}\times n_{\mathrm{in}}} K = [ K NTK ( x i , x j ) ] i , j = 1 n in ∈ R n in × n in for the inner infinite-width NTK Gram matrix on the samples of D i n \mathcal{D}_{\mathrm{in}} D in , and λ 0 \lambda_0 λ 0 for its smallest eigenvalue. For ω ∈ Ω \omega\in\Omega ω ∈ Ω , let Y ω = [ g ω ( z 1 ) , … , g ω ( z n i n ) ] ⊤ ∈ R n i n Y_\omega=[g_\omega(z_1),\ldots,g_\omega(z_{n_{\mathrm{in}}})]^\top\in\mathbb{R}^{n_{\mathrm{in}}} Y ω = [ g ω ( z 1 ) , … , g ω ( z n in ) ] ⊤ ∈ R n in denote the inner target vector. The event E λ 0 = { λ 0 ≥ λ ⋆ } \mathcal{E}^{\lambda_0}=\{\lambda_0\ge\lambda_\star\} E λ 0 = { λ 0 ≥ λ ⋆ } is the NTK non-degeneracy event of § 3.1 and Appendix § 13 .8.2, whose threshold λ ⋆ = λ ⋆ ( n i n , δ ) > 0 \lambda_{\star}=\lambda_{\star}(n_{in},\delta)>0 λ ⋆ = λ ⋆ ( n in , δ ) > 0 depends only on n i n n_{in} n in , δ \delta δ , p p p , A A A and M P M_{\mathbb{P}} M P . We denote by h t , ω N T K h_{t,\omega}^{\mathrm{NTK}} h t , ω NTK the inner NTK gradient flow predictor of ω \omega ω whose dynamics are described as follows for any x ∈ X x\in\mathcal{X} x ∈ X :
d d t h t , ω N T K ( x ) = − 1 n i n ∑ i = 1 n i n K N T K ( x , x i ) ( h t , ω N T K ( x i ) − g ω ( z i ) ) , h 0 , ω N T K = 0. (24) \frac{\mathrm{d}}{\mathrm{d}t}h_{t,\omega}^{\mathrm{NTK}}(x)=-\frac{1}{n_{in}}\sum_{i=1}^{n_{in}}K^{\mathrm{NTK}}(x,x_i)\left(h_{t,\omega}^{\mathrm{NTK}}(x_i)-g_{\omega}(z_i)\right),\qquad h_{0,\omega}^{\mathrm{NTK}}=0. \tag*{(24)} d t d h t , ω NTK ( x ) = − n in 1 i = 1 ∑ n in K NTK ( x , x i ) ( h t , ω NTK ( x i ) − g ω ( z i ) ) , h 0 , ω NTK = 0. (24)
For brevity, we state the generalization bounds without writing the exact lower bounds on the inner and outer sample sizes n i n n_{in} n in and n o u t n_{out} n o u t required for the results to hold. We refer the reader to Proposition E.25 for a detailed statement including these conditions.
Lipschitz continuity of the inner NTK approximation error in the outer variable
Lemma D.1 (Linear representation of the inner NTK predictor). On the event E λ 0 h \mathcal{E}_{\lambda_0}^{h} E λ 0 h , the inner NTK gradient flow predictor started from the mirrored initialization admits, for every ω ∈ Ω \omega\in\Omega ω ∈ Ω and every t ≥ 0 t\ge0 t ≥ 0 , the following representation:
h t , ω N T K = Φ t Y ω , Φ t : = 1 n i n K N T K ( ⋅ , X ) T ( 1 n i n × n i n − exp ( − t n i n K ) ) ( 1 n i n K ) − 1 , h_{t,\omega}^{\mathrm{NTK}}=\Phi_tY_\omega,\qquad\Phi_t:=\frac{1}{n_{in}}K^{\mathrm{NTK}}(\cdot,\mathbf{X})^{\mathsf{T}}\left(\mathbb{1}_{n_{in}\times n_{in}}-\exp\left(-\frac{t}{n_{in}}\mathbf{K}\right)\right)\left(\frac{1}{n_{in}}\mathbf{K}\right)^{-1}, h t , ω NTK = Φ t Y ω , Φ t := n in 1 K NTK ( ⋅ , X ) T ( 1 n in × n in − exp ( − n in t K ) ) ( n in 1 K ) − 1 , where K N T K ( ⋅ , X ) = [ K N T K ( ⋅ , x 1 ) , … , K N T K ( ⋅ , x n i n ) ] T K^{\mathrm{NTK}}(\cdot,\mathbf{X})=[K^{\mathrm{NTK}}(\cdot,x_1),\ldots,K^{\mathrm{NTK}}(\cdot,x_{n_{in}})]^{\mathsf{T}} K NTK ( ⋅ , X ) = [ K NTK ( ⋅ , x 1 ) , … , K NTK ( ⋅ , x n in ) ] T . The operator Φ t \Phi_t Φ t depends only on K \mathbf{K} K , the inputs x 1 , … , x n i n x_1,\ldots,x_{n_{in}} x 1 , … , x n in , and t t t ; all dependence on ω \omega ω enters through Y ω Y_\omega Y ω .
Proof. We write A : = 1 n i n K \mathbf{A}:=\frac{1}{n_{in}}\mathbf{K} A := n in 1 K and let H ( t ) : = [ h t , ω N T K ( x 1 ) , … , h t , ω N T K ( x n i n ) ] T H(t):=[h_{t,\omega}^{\mathrm{NTK}}(x_1),\ldots,h_{t,\omega}^{\mathrm{NTK}}(x_{n_{in}})]^{\mathsf{T}} H ( t ) := [ h t , ω NTK ( x 1 ) , … , h t , ω NTK ( x n in ) ] T . Evaluating the kernel gradient flow in Equation (24) at the training points gives the linear inhomogeneous ODE H ˙ ( t ) = − A ( H ( t ) − Y ω ) \dot{H}(t)=-\mathbf{A}(H(t)-Y_\omega) H ˙ ( t ) = − A ( H ( t ) − Y ω ) with H ( 0 ) = 0 H(0)=0 H ( 0 ) = 0 due to the mirrored initialization. Its unique solution is H ( t ) = ( 1 n i n × n i n − exp ( − t A ) ) Y ω H(t)=(\mathbb{1}_{n_{in}\times n_{in}}-\exp(-t\mathbf{A}))Y_\omega H ( t ) = ( 1 n in × n in − exp ( − t A )) Y ω , so the training residual is H ( t ) − Y ω = − exp ( − t A ) Y ω H(t)-Y_\omega=-\exp(-t\mathbf{A})Y_\omega H ( t ) − Y ω = − exp ( − t A ) Y ω . Substituting into the flow at an arbitrary x ∈ X x\in\mathcal{X} x ∈ X , we get:
d d t h t , ω N T K ( x ) = − 1 n i n K N T K ( x , X ) T ( H ( t ) − Y ω ) = 1 n i n K N T K ( x , X ) T exp ( − t A ) Y ω . \frac{\mathrm{d}}{\mathrm{d}t}h_{t,\omega}^{\mathrm{NTK}}(x)=-\frac{1}{n_{in}}K^{\mathrm{NTK}}(x,\mathbf{X})^{\mathsf{T}}(H(t)-Y_\omega)=\frac{1}{n_{in}}K^{\mathrm{NTK}}(x,\mathbf{X})^{\mathsf{T}}\exp(-t\mathbf{A})Y_\omega. d t d h t , ω NTK ( x ) = − n in 1 K NTK ( x , X ) T ( H ( t ) − Y ω ) = n in 1 K NTK ( x , X ) T exp ( − t A ) Y ω .
The right-hand side does not involve h t , ω N T K ( x ) h_{t,\omega}^{\mathrm{NTK}}(x) h t , ω NTK ( x ) . Integrating from 0 0 0 to t t t with h 0 , ω N T K = 0 h_{0,\omega}^{\mathrm{NTK}}=0 h 0 , ω NTK = 0 and using ∫ 0 t exp ( − s A ) d s = A − 1 ( 1 n i n × n i n − exp ( − t A ) ) \int_0^t\exp(-s\mathbf{A})\mathrm{d}s=\mathbf{A}^{-1}(\mathbb{1}_{n_{in}\times n_{in}}-\exp(-t\mathbf{A})) ∫ 0 t exp ( − s A ) d s = A − 1 ( 1 n in × n in − exp ( − t A )) (valid since A \mathbf{A} A is invertible on E λ 0 h \mathcal{E}_{\lambda_0}^{h} E λ 0 h ) yields h t , ω N T K ( x ) = 1 n i n K N T K ( x , X ) T A − 1 ( 1 n i n × n i n − exp ( − t A ) ) Y ω h_{t,\omega}^{\mathrm{NTK}}(x)=\frac{1}{n_{in}}K^{\mathrm{NTK}}(x,\mathbf{X})^{\mathsf{T}}\mathbf{A}^{-1}(\mathbb{1}_{n_{in}\times n_{in}}-\exp(-t\mathbf{A}))Y_\omega h t , ω NTK ( x ) = n in 1 K NTK ( x , X ) T A − 1 ( 1 n in × n in − exp ( − t A )) Y ω , which concludes the proof. □ \square □
Lemma D.2 (Operator-norm bound on Φ t \Phi_t Φ t ). Let Assumption C hold. On the event E λ 0 h \mathcal{E}_{\lambda_0}^{h} E λ 0 h , for every t ≥ 0 t\ge0 t ≥ 0 , we have:
∥ Φ t ∥ ℓ 2 → L 2 ( X ) ≤ κ N T K n i n λ 0 , with κ N T K : = ( p + 2 ) ( A 2 + 1 ) + 1 , \|\Phi_t\|_{\ell_2\to L^2(\mathcal{X})}\le\frac{\kappa^{\mathrm{NTK}}\sqrt{n_{in}}}{\lambda_0},\qquad\text{with }\kappa^{\mathrm{NTK}}:=(p+2)(A^2+1)+1, ∥ Φ t ∥ ℓ 2 → L 2 ( X ) ≤ λ 0 κ NTK n in , with κ NTK := ( p + 2 ) ( A 2 + 1 ) + 1 , where ∥ Φ t ∥ ℓ 2 → L 2 ( X ) : = sup Y ≠ 0 ∥ Φ t Y ∥ L 2 ( X ) / ∥ Y ∥ \|\Phi_t\|_{\ell_2\to L^2(\mathcal{X})}:=\sup_{Y\ne0}\|\Phi_tY\|_{L^2(\mathcal{X})}/\|Y\| ∥ Φ t ∥ ℓ 2 → L 2 ( X ) := sup Y = 0 ∥ Φ t Y ∥ L 2 ( X ) /∥ Y ∥ denotes the operator norm of Φ t \Phi_t Φ t as a map from ( R n i n , ∥ ⋅ ∥ ) (\mathbb{R}^{n_{in}},\|\cdot\|) ( R n in , ∥ ⋅ ∥ ) to L 2 ( X ) L^2(\mathcal{X}) L 2 ( X ) .
Proof. We factor Φ t = E M t \Phi_t=\mathcal{E}M_t Φ t = E M t , where M t : = ( 1 n i n × n i n − exp ( − t n i n K ) ) ( 1 n i n K ) − 1 ∈ R n i n × n i n M_t:=\left(\mathbb{1}_{n_{in}\times n_{in}}-\exp\left(-\frac{t}{n_{in}}\mathbf{K}\right)\right)\left(\frac{1}{n_{in}}\mathbf{K}\right)^{-1}\in\mathbb{R}^{n_{in}\times n_{in}} M t := ( 1 n in × n in − exp ( − n in t K ) ) ( n in 1 K ) − 1 ∈ R n in × n in and E : R n i n → L 2 ( X ) \mathcal{E}:\mathbb{R}^{n_{in}}\to L^2(\mathcal{X}) E : R n in → L 2 ( X ) is the kernel-evaluation operator defined by:
( E c ) ( x ) = 1 n i n ∑ i = 1 n i n K N T K ( x , x i ) c i , for c ∈ R n i n and x ∈ X . (\mathcal{E}c)(x)=\frac{1}{n_{in}}\sum_{i=1}^{n_{in}}K^{\mathrm{NTK}}(x,x_i)c_i,\qquad\text{for }c\in\mathbb{R}^{n_{in}}\text{ and }x\in\mathcal{X}. ( E c ) ( x ) = n in 1 i = 1 ∑ n in K NTK ( x , x i ) c i , for c ∈ R n in and x ∈ X .
Control of ∥ M t ∥ o p \|M_t\|_{\mathrm{op}} ∥ M t ∥ op . M t = g t ( K ) M_t=g_t(\mathbf{K}) M t = g t ( K ) is a function of K \mathbf{K} K with scalar filter g t ( λ ) = n i n λ ( 1 − exp ( − t n i n λ ) ) g_t(\lambda)=\frac{n_{in}}{\lambda}\left(1-\exp\left(-\frac{t}{n_{in}}\lambda\right)\right) g t ( λ ) = λ n in ( 1 − exp ( − n in t λ ) ) . Since K \mathbf{K} K is symmetric positive definite on E λ 0 h \mathcal{E}_{\lambda_0}^{h} E λ 0 h , ∥ M t ∥ o p = max 1 ≤ i ≤ n i n g t ( λ i ) \|M_t\|_{\mathrm{op}}=\max_{1\le i\le n_{in}}g_t(\lambda_i) ∥ M t ∥ op = max 1 ≤ i ≤ n in g t ( λ i ) where { λ i } i = 1 n i n \{\lambda_i\}_{i=1}^{n_{in}} { λ i } i = 1 n in are the eigenvalues of K \mathbf{K} K . Using 0 ≤ 1 − exp ( − a ) ≤ 1 0\le1-\exp(-a)\le1 0 ≤ 1 − exp ( − a ) ≤ 1 provided that a ≥ 0 a\ge0 a ≥ 0 , we obtain:
g t ( λ ) = n i n λ ( 1 − exp ( − t n i n λ ) ) ≤ n i n λ , hence ∥ M t ∥ o p ≤ n i n min 1 ≤ i ≤ n i n λ i = n i n λ 0 . g_t(\lambda)=\frac{n_{in}}{\lambda}\left(1-\exp\left(-\frac{t}{n_{in}}\lambda\right)\right)\le\frac{n_{in}}{\lambda},\qquad\text{hence }\|M_t\|_{\mathrm{op}}\le\frac{n_{in}}{\min_{1\le i\le n_{in}}\lambda_i}=\frac{n_{in}}{\lambda_0}. g t ( λ ) = λ n in ( 1 − exp ( − n in t λ ) ) ≤ λ n in , hence ∥ M t ∥ op ≤ min 1 ≤ i ≤ n in λ i n in = λ 0 n in .
a bound independent of t t t .
Control of ∥ E ∥ ℓ 2 → L 2 ( X ) \lVert E\rVert_{\ell_2\to L^2(\mathcal{X})} ∥ E ∥ ℓ 2 → L 2 ( X ) . Fix c ∈ R n i n c \in\mathbb{R}^{n_{\mathrm{in}}} c ∈ R n in and x ∈ X x \in\mathcal{X} x ∈ X . By the Cauchy-Schwarz inequality and ∣ K N T K ( x , x i ) ∣ ≤ κ N T K \lvert K^{\mathrm{NTK}}(x,x_i)\rvert\le\kappa^{\mathrm{NTK}} ∣ K NTK ( x , x i )∣ ≤ κ NTK (Lemma E.2), we get:
∣ 1 n i n ∑ i = 1 n i n K N T K ( x , x i ) c i ∣ ≤ 1 n i n ∥ K N T K ( x , X ) ∥ ∥ c ∥ ≤ κ N T K n i n ∥ c ∥ . \left\lvert\frac{1}{n_{\mathrm{in}}}\sum_{i=1}^{n_{\mathrm{in}}}K^{\mathrm{NTK}}(x,x_i)c_i\right\rvert\le\frac{1}{n_{\mathrm{in}}}\lVert K^{\mathrm{NTK}}(x,\mathcal{X})\rVert\lVert c\rVert\le\frac{\kappa^{\mathrm{NTK}}}{\sqrt{n_{\mathrm{in}}}}\lVert c\rVert. n in 1 i = 1 ∑ n in K NTK ( x , x i ) c i ≤ n in 1 ∥ K NTK ( x , X )∥ ∥ c ∥ ≤ n in κ NTK ∥ c ∥ .
where K N T K ( x , X ) = [ K N T K ( x , x 1 ) , … , K N T K ( x , x n i n ) ] T K^{\mathrm{NTK}}(x,\mathcal{X})=[K^{\mathrm{NTK}}(x,x_1),\ldots,K^{\mathrm{NTK}}(x,x_{n_{\mathrm{in}}})]^{\mathsf{T}} K NTK ( x , X ) = [ K NTK ( x , x 1 ) , … , K NTK ( x , x n in ) ] T . Since the marginal on X \mathcal{X} X is a probability measure, we have:
∥ E c ∥ L 2 ( X ) 2 ≤ ( κ N T K n i n ) 2 ∥ c ∥ 2 , \lVert Ec\rVert_{L^2(\mathcal{X})}^2\le\left(\frac{\kappa^{\mathrm{NTK}}}{\sqrt{n_{\mathrm{in}}}}\right)^2\lVert c\rVert^2, ∥ E c ∥ L 2 ( X ) 2 ≤ ( n in κ NTK ) 2 ∥ c ∥ 2 ,
which implies that ∥ E ∥ ℓ 2 → L 2 ( X ) ≤ κ N T K n i n \lVert E\rVert_{\ell_2\to L^2(\mathcal{X})}\le\frac{\kappa^{\mathrm{NTK}}}{\sqrt{n_{\mathrm{in}}}} ∥ E ∥ ℓ 2 → L 2 ( X ) ≤ n in κ NTK .
By submultiplicativity of the operator norm, we get:
∥ Φ t ∥ ℓ 2 → L 2 ( X ) ≤ ∥ E ∥ ℓ 2 → L 2 ( X ) ∥ M t ∥ o p ≤ κ N T K n i n ⋅ n i n λ 0 = κ N T K n i n λ 0 . \lVert\Phi_t\rVert_{\ell_2\to L^2(\mathcal{X})}\le\lVert E\rVert_{\ell_2\to L^2(\mathcal{X})}\lVert M_t\rVert_{\mathrm{op}}\le\frac{\kappa^{\mathrm{NTK}}}{\sqrt{n_{\mathrm{in}}}}\cdot\frac{n_{\mathrm{in}}}{\lambda_0}=\frac{\kappa^{\mathrm{NTK}}\sqrt{n_{\mathrm{in}}}}{\lambda_0}. ∥ Φ t ∥ ℓ 2 → L 2 ( X ) ≤ ∥ E ∥ ℓ 2 → L 2 ( X ) ∥ M t ∥ op ≤ n in κ NTK ⋅ λ 0 n in = λ 0 κ NTK n in .
Lemma D.3 (Lipschitz continuity in ω \omega ω of the inner NTK error). Let Assumptions A A A to C C C hold. On the event E λ 0 h \mathcal{E}_{\lambda_0}^{h} E λ 0 h , for any t ≥ 0 t\ge0 t ≥ 0 , the map Ψ : ω ↦ h t , ω N T K − h ω ∗ \Psi:\omega\mapsto h_{t,\omega}^{\mathrm{NTK}}-h_\omega^* Ψ : ω ↦ h t , ω NTK − h ω ∗ is L Ψ L_\Psi L Ψ -Lipschitz continuous from Ω \Omega Ω to $L^2(\mathcal{X}) with :
L Ψ : = G ( 1 + κ N T K n i n λ 0 ) , with κ N T K : = ( p + 2 ) ( A 2 + 1 ) + 1. L_\Psi:=G\left(1+\frac{\kappa^{\mathrm{NTK}}n_{\mathrm{in}}}{\lambda_0}\right),\qquad\text{with }\kappa^{\mathrm{NTK}}:=(p+2)(A^2+1)+1. L Ψ := G ( 1 + λ 0 κ NTK n in ) , with κ NTK := ( p + 2 ) ( A 2 + 1 ) + 1. Proof. Let t ≥ 0 t\ge0 t ≥ 0 and ω , ω ′ ∈ Ω \omega,\omega'\in\Omega ω , ω ′ ∈ Ω . By the triangle inequality, we have:
∥ Ψ ( ω ) − Ψ ( ω ′ ) ∥ L 2 ( X ) ≤ ∥ h t , ω N T K − h t , ω ′ N T K ∥ L 2 ( X ) + ∥ h ω ∗ − h ω ′ ∗ ∥ L 2 ( X ) . \lVert\Psi(\omega)-\Psi(\omega')\rVert_{L^2(\mathcal{X})}\le\lVert h_{t,\omega}^{\mathrm{NTK}}-h_{t,\omega'}^{\mathrm{NTK}}\rVert_{L^2(\mathcal{X})}+\lVert h_\omega^*-h_{\omega'}^*\rVert_{L^2(\mathcal{X})}. ∥ Ψ ( ω ) − Ψ ( ω ′ ) ∥ L 2 ( X ) ≤ ∥ h t , ω NTK − h t , ω ′ NTK ∥ L 2 ( X ) + ∥ h ω ∗ − h ω ′ ∗ ∥ L 2 ( X ) . For the target term, Proposition F.4 gives ∥ h ω ∗ − h ω ′ ∗ ∥ L 2 ( X ) ≤ G ∥ ω − ω ′ ∥ \lVert h_\omega^*-h_{\omega'}^*\rVert_{L^2(\mathcal{X})}\le G\lVert\omega-\omega'\rVert ∥ h ω ∗ − h ω ′ ∗ ∥ L 2 ( X ) ≤ G ∥ ω − ω ′ ∥ . For the estimator term, Lemma D.1 and the linearity of Φ t \Phi_t Φ t give h t , ω N T K − h t , ω ′ N T K = Φ t ( Y ω − Y ω ′ ) h_{t,\omega}^{\mathrm{NTK}}-h_{t,\omega'}^{\mathrm{NTK}}=\Phi_t(Y_\omega-Y_{\omega'}) h t , ω NTK − h t , ω ′ NTK = Φ t ( Y ω − Y ω ′ ) , hence by Lemma D.2:
∥ h t , ω N T K − h t , ω ′ N T K ∥ L 2 ( X ) ≤ ∥ Φ t ∥ ℓ 2 → L 2 ( X ) ∥ Y ω − Y ω ′ ∥ ≤ κ N T K n i n λ 0 ∥ Y ω − Y ω ′ ∥ . \lVert h_{t,\omega}^{\mathrm{NTK}}-h_{t,\omega'}^{\mathrm{NTK}}\rVert_{L^2(\mathcal{X})}\le\lVert\Phi_t\rVert_{\ell_2\to L^2(\mathcal{X})}\lVert Y_\omega-Y_{\omega'}\rVert\le\frac{\kappa^{\mathrm{NTK}}\sqrt{n_{\mathrm{in}}}}{\lambda_0}\lVert Y_\omega-Y_{\omega'}\rVert. ∥ h t , ω NTK − h t , ω ′ NTK ∥ L 2 ( X ) ≤ ∥ Φ t ∥ ℓ 2 → L 2 ( X ) ∥ Y ω − Y ω ′ ∥ ≤ λ 0 κ NTK n in ∥ Y ω − Y ω ′ ∥ . Since the open set U ⊃ Ω U\supset\Omega U ⊃ Ω of Assumption A A A is convex, the segment ω s : = ω ′ + s ( ω − ω ′ ) \omega_s:=\omega'+s(\omega-\omega') ω s := ω ′ + s ( ω − ω ′ ) , s ∈ [ 0 , 1 ] s\in[0,1] s ∈ [ 0 , 1 ] , lies in U U U . Hence, applying the Cauchy-Schwarz inequality and Assumption A A A , we get for each 1 ≤ i ≤ n i n 1\le i\le n_{\mathrm{in}} 1 ≤ i ≤ n in :
∣ g ω ( z i ) − g ω ′ ( z i ) ∣ = ∣ ∫ 0 1 ⟨ ∂ ω g ω s ( z i ) , ω − ω ′ ⟩ d s ∣ ≤ G ∥ ω − ω ′ ∥ , \lvert g_\omega(z_i)-g_{\omega'}(z_i)\rvert=\left\lvert\int_0^1\langle\partial_\omega g_{\omega_s}(z_i),\omega-\omega'\rangle\,\mathrm{d}s\right\rvert\le G\lVert\omega-\omega'\rVert, ∣ g ω ( z i ) − g ω ′ ( z i )∣ = ∫ 0 1 ⟨ ∂ ω g ω s ( z i ) , ω − ω ′ ⟩ d s ≤ G ∥ ω − ω ′ ∥ , and therefore ∥ Y ω − Y ω ′ ∥ ≤ n i n G ∥ ω − ω ′ ∥ \lVert Y_\omega-Y_{\omega'}\rVert\le\sqrt{n_{\mathrm{in}}}G\lVert\omega-\omega'\rVert ∥ Y ω − Y ω ′ ∥ ≤ n in G ∥ ω − ω ′ ∥ . Combining both, we get:
∥ h t , ω N T K − h t , ω ′ N T K ∥ L 2 ( X ) ≤ κ N T K n i n λ 0 G ∥ ω − ω ′ ∥ . \lVert h_{t,\omega}^{\mathrm{NTK}}-h_{t,\omega'}^{\mathrm{NTK}}\rVert_{L^2(\mathcal{X})}\le\frac{\kappa^{\mathrm{NTK}}n_{\mathrm{in}}}{\lambda_0}G\lVert\omega-\omega'\rVert. ∥ h t , ω NTK − h t , ω ′ NTK ∥ L 2 ( X ) ≤ λ 0 κ NTK n in G ∥ ω − ω ′ ∥ . Adding the two contributions concludes the proof.
Generalization bounds for the inner solution, regression and adjoint functions
Proposition D.4 (Uniform-in-ω \omega ω expected error in the inner solution). Consider n i n n_{\mathrm{in}} n in i.i.d. samples { ( x i , z i ) } i = 1 n i n ∼ P X Z \{(x_i,z_i)\}_{i=1}^{n_{\mathrm{in}}}\sim P_{XZ} {( x i , z i ) } i = 1 n in ∼ P X Z and assume p ≥ 2 p\ge2 p ≥ 2 . Fix a compact Ω ⊂ R d \Omega\subset\mathbb{R}^d Ω ⊂ R d and let Assumptions A A A and C C C to E E E hold. If we choose a stopping time t o p h = n i n p + 1 s h ( p + 1 ) + p t_{\mathrm{op}_h}=n_{\mathrm{in}}^{\frac{p+1}{s_h(p+1)+p}} t op h = n in s h ( p + 1 ) + p p + 1 with 1 p + 1 < s h ≤ 2 \frac{1}{p+1}<s_h\le2 p + 1 1 < s h ≤ 2 , then, considering the early-stopped truncated neural network predictor h ^ ω ∗ \widehat{h}_\omega* h ω ∗ for the inner solution h ω ∗ h_\omega^* h ω ∗ , there exist a positive constant C 1 > 0 C_1>0 C 1 > 0 and a polynomial poly h ∗ \operatorname{poly}_h * poly h ∗ such that when n i n n_{\mathrm{in}} n in is sufficiently large and the width satisfies m h ≥ poly h ( n i n , ln n i n ) m_h\ge\operatorname{poly}_h(n_{\mathrm{in}},\ln n_{\mathrm{in}}) m h ≥ poly h ( n in , ln n in ) , the following holds:
E [ sup ω ∈ Ω ∥ h ^ ω − h ω ∗ ∥ L 2 ( X ) 2 ] ≤ C 1 d 4 ( ln n i n ) 4 n i n − s h ( p + 1 ) s h ( p + 1 ) + p , \mathbb{E}\left[\sup_{\omega\in\Omega}\lVert\widehat{h}_\omega-h_\omega^*\rVert_{L^2(\mathcal{X})}^2\right]\le C_1d^4(\ln n_{\mathrm{in}})^4n_{\mathrm{in}}^{-\frac{s_h(p+1)}{s_h(p+1)+p}}, E [ ω ∈ Ω sup ∥ h ω − h ω ∗ ∥ L 2 ( X ) 2 ] ≤ C 1 d 4 ( ln n in ) 4 n in − s h ( p + 1 ) + p s h ( p + 1 ) , where the constant C 1 > 0 C_1>0 C 1 > 0 is independent of n i n n_{\mathrm{in}} n in and t o p h t_{\mathrm{op}_h} t op h , but depends on s h s_h s h , p p p , A A A , D D D , G G G , M P M_{\mathbb{P}} M P , diam ( Ω ) \operatorname{diam}(\Omega) diam ( Ω ) , and R h R_h R h , and the coefficients of poly h \operatorname{poly}_h poly h depend on s h s_h s h , A A A , D D D , G G G , M P M_{\mathbb{P}} M P , diam ( Ω ) \operatorname{diam}(\Omega) diam ( Ω ) , p p p , and R h R_h R h . The expectation here is with respect to random samples and random initialization.
Proof. Throughout, we write r h : = 1 2 s h γ h r_h := \frac{1}{2}s_h\gamma_h r h := 2 1 s h γ h with γ h = p + 1 s h ( p + 1 ) + p \gamma_h = \frac{p+1}{s_h(p+1)+p} γ h = s h ( p + 1 ) + p p + 1 . By the triangle inequality and the identity ( a + b ) 2 ≤ 2 ( a 2 + b 2 ) (a+b)^2 \le2(a^2+b^2) ( a + b ) 2 ≤ 2 ( a 2 + b 2 ) , we have:
E [ sup ω ∈ Ω ∥ h ^ ω − h ω ∗ ∥ L 2 ( X ) 2 ] ≤ 2 ( E [ sup ω ∈ Ω ∥ h ^ ω − h t o p h , ω , ω N T K ∥ L 2 ( X ) 2 ] + E [ sup ω ∈ Ω ∥ h t o p h , ω , ω N T K − h ω ∗ ∥ L 2 ( X ) 2 ] ) . (25) \mathbb{E}\left[\sup_{\omega\in\Omega}\left\|\hat{h}_{\omega}-h^*_{\omega}\right\|_{L^2(\mathcal{X})}^2\right]\leq2\left(\mathbb{E}\left[\sup_{\omega\in\Omega}\left\|\hat{h}_{\omega}-h^{\mathrm{NTK}}_{t_{\mathrm{op}_h,\omega}\!,\omega}\right\|_{L^2(\mathcal{X})}^2\right]+\mathbb{E}\left[\sup_{\omega\in\Omega}\left\|h^{\mathrm{NTK}}_{t_{\mathrm{op}_h,\omega}\!,\omega}-h^*_{\omega}\right\|_{L^2(\mathcal{X})}^2\right]\right). \tag*{(25)} E [ ω ∈ Ω sup h ^ ω − h ω ∗ L 2 ( X ) 2 ] ≤ 2 ( E [ ω ∈ Ω sup h ^ ω − h t op h , ω , ω NTK L 2 ( X ) 2 ] + E [ ω ∈ Ω sup h t op h , ω , ω NTK − h ω ∗ L 2 ( X ) 2 ] ) . (25)
Upper-bound on E [ sup ω ∈ Ω ∥ h ^ ω − h t o p h , ω , ω N T K ∥ L 2 ( X ) 2 ] \mathbb{E}\left[\sup_{\omega\in\Omega}\left\|\hat{h}_{\omega}-h^{\mathrm{NTK}}_{t_{\mathrm{op}_h,\omega}\!,\omega}\right\|_{L^2(\mathcal{X})}^2\right] E [ sup ω ∈ Ω h ^ ω − h t op h , ω , ω NTK L 2 ( X ) 2 ] . The bound on this term is obtained by following the first part of the proof of Proposition E.25, the only difference being the presence of the supremum over Ω \Omega Ω . The supremum is harmless here for two reasons. First, the favorable event E h \mathcal{E}^h E h is independent of ω \omega ω : the truncated predictor h ^ ω \hat{h}_{\omega} h ^ ω therefore vanishes on ( E h ) c (\mathcal{E}^h)^c ( E h ) c for all ω ∈ Ω \omega\in\Omega ω ∈ Ω at once, so the decomposition over E h \mathcal{E}^h E h and ( E h ) c (\mathcal{E}^h)^c ( E h ) c and the ensuing control of Pr ( ( E h ) c ) \Pr((\mathcal{E}^h)^c) Pr (( E h ) c ) are unchanged. Second, the outer variable enters the argument only through the uniform target bound ∣ g ω ( z i ) ∣ ≤ D \lvert g_{\omega}(z_i)\rvert\leq D ∣ g ω ( z i )∣ ≤ D (Assumption A), so all constants remain ω \omega ω -independent and each estimate holds uniformly over Ω \Omega Ω . We thus obtain, exactly as in Proposition E.25:
E [ sup ω ∈ Ω ∥ h ^ ω − h t o p h , ω , ω N T K ∥ L 2 ( X ) 2 ] ≤ c 1 2 m h − 1 2 ln m h + 4 ( κ N T K ) 2 D 2 n i n − 2 r h δ , (26) \mathbb{E}\left[\sup_{\omega\in\Omega}\left\|\hat{h}_{\omega}-h^{\mathrm{NTK}}_{t_{\mathrm{op}_h,\omega}\!,\omega}\right\|_{L^2(\mathcal{X})}^2\right]\leq c_1^2m_h^{-\frac{1}{2}}\ln m_h+4(\kappa^{\mathrm{NTK}})^2D^2n_{\mathrm{in}}^{-2r_h}\delta, \tag*{(26)} E [ ω ∈ Ω sup h ^ ω − h t op h , ω , ω NTK L 2 ( X ) 2 ] ≤ c 1 2 m h − 2 1 ln m h + 4 ( κ NTK ) 2 D 2 n in − 2 r h δ , (26)
where δ ∈ ( 0 , 1 ) \delta\in(0,1) δ ∈ ( 0 , 1 ) , κ N T K = ( p + 2 ) ( A 2 + 1 ) + 1 \kappa^{\mathrm{NTK}}=(p+2)(A^2+1)+1 κ NTK = ( p + 2 ) ( A 2 + 1 ) + 1 , and c 1 > 0 c_1>0 c 1 > 0 is a constant that is independent of δ \delta δ , n i n n_{\mathrm{in}} n in , and t o p h t_{\mathrm{op}_h} t op h , but depends on A A A . We note that Equation (26) is valid whenever m h ≥ poly ( n i n , 1 / δ , ln n i n , ln ( 1 / δ ) ) m_h\geq\operatorname{poly}(n_{\mathrm{in}},1/\delta,\ln n_{\mathrm{in}},\ln(1/\delta)) m h ≥ poly ( n in , 1/ δ , ln n in , ln ( 1/ δ )) , where the coefficients of poly \operatorname{poly} poly depend on A A A , D D D , M P M_{\mathrm{P}} M P , and p p p .
Upper-bound on E [ sup ω ∈ Ω ∥ h t o p h , ω , ω N T K − h ω ∗ ∥ L 2 ( X ) 2 ] \mathbb{E}\left[\sup_{\omega\in\Omega}\left\|h^{\mathrm{NTK}}_{t_{\mathrm{op}_h,\omega}\!,\omega}-h^*_{\omega}\right\|_{L^2(\mathcal{X})}^2\right] E [ sup ω ∈ Ω h t op h , ω , ω NTK − h ω ∗ L 2 ( X ) 2 ] . Let λ ∗ : = λ ∗ ( n i n , δ ) > 0 \lambda_*:=\lambda_*(n_{\mathrm{in}},\delta)>0 λ ∗ := λ ∗ ( n in , δ ) > 0 denote the threshold in the definition of E λ 0 h \mathcal{E}^h_{\lambda_0} E λ 0 h , so that E λ 0 h = { λ 0 ≥ λ ∗ } \mathcal{E}^h_{\lambda_0}=\{\lambda_0\geq\lambda_*\} E λ 0 h = { λ 0 ≥ λ ∗ } . It depends only on n i n n_{\mathrm{in}} n in , δ \delta δ , p p p , A A A and M P M_{\mathrm{P}} M P , and not on D i n D_{\mathrm{in}} D in . Define the deterministic constants:
L Ψ ∗ : = G ( 1 + κ N T K n i n λ ∗ ) and ε : = ( L Ψ ∗ ) − 1 n i n − r h . L^*_{\Psi}:=G\left(1+\frac{\kappa^{\mathrm{NTK}}n_{\mathrm{in}}}{\lambda_*}\right)
\qquad\text{and}\qquad
\varepsilon:=(L^*_{\Psi})^{-1}n_{\mathrm{in}}^{-r_h}. L Ψ ∗ := G ( 1 + λ ∗ κ NTK n in ) and ε := ( L Ψ ∗ ) − 1 n in − r h .
On E λ 0 h \mathcal{E}^h_{\lambda_0} E λ 0 h , the Lipschitz constant L Ψ L_{\Psi} L Ψ of Lemma D.3 satisfies L Ψ ≤ L Ψ ∗ L_{\Psi}\leq L^*_{\Psi} L Ψ ≤ L Ψ ∗ , since λ 0 ≥ λ ∗ \lambda_0\geq\lambda_* λ 0 ≥ λ ∗ there. Let N = { ω 1 , … , ω N } ⊂ Ω \mathcal{N}=\{\omega_1,\ldots,\omega_N\}\subset\Omega N = { ω 1 , … , ω N } ⊂ Ω be a maximal ε \varepsilon ε -separated subset of Ω \Omega Ω , i.e., ∥ ω j − ω k ∥ ≥ ε \|\omega_j-\omega_k\|\geq\varepsilon ∥ ω j − ω k ∥ ≥ ε for all j ≠ k j\ne k j = k and no point of Ω \Omega Ω can be added to N \mathcal{N} N without violating this property. By maximality, every ω ∈ Ω \omega\in\Omega ω ∈ Ω satisfies ∥ ω − ω k ∥ < ε \|\omega-\omega_k\|<\varepsilon ∥ ω − ω k ∥ < ε for some 1 ≤ k ≤ N 1\leq k\leq N 1 ≤ k ≤ N , so N \mathcal{N} N is an ε \varepsilon ε -net of Ω \Omega Ω whose points lie in Ω \Omega Ω . In particular, Lemma D.3 applies to every pair ( ω , ω k ) (\omega,\omega_k) ( ω , ω k ) . Note that since ε \varepsilon ε is deterministic, N \mathcal{N} N and N N N are fixed before the data are drawn, and are therefore independent of D i n D_{\mathrm{in}} D in . Let also E n e t : = ⋂ k = 1 N E ω k N T K \mathcal{E}_{\mathrm{net}}:=\bigcap_{k=1}^{N}\mathcal{E}^{\mathrm{NTK}}_{\omega_k} E net := ⋂ k = 1 N E ω k NTK , where, for each 1 ≤ k ≤ N 1\leq k\leq N 1 ≤ k ≤ N , E ω k N T K \mathcal{E}^{\mathrm{NTK}}_{\omega_k} E ω k NTK is the event of Lemma E.21 at ω k \omega_k ω k with deviation parameter δ / N \delta/N δ / N , namely:
E ω k N T K : = { ∥ h t o p h , ω k , ω k N T K − h ω k ∗ ∥ L 2 ( X ) ≤ c 2 ( ln 6 N δ ) 2 n i n − r h } . \mathcal{E}^{\mathrm{NTK}}_{\omega_k}:=\left\{\left\|h^{\mathrm{NTK}}_{t_{\mathrm{op}_h,\omega_k}\!,\omega_k}-h^*_{\omega_k}\right\|_{L^2(\mathcal{X})}\leq c_2\left(\ln\frac{6N}{\delta}\right)^2n_{\mathrm{in}}^{-r_h}\right\}. E ω k NTK := { h t op h , ω k , ω k NTK − h ω k ∗ L 2 ( X ) ≤ c 2 ( ln δ 6 N ) 2 n in − r h } .
where c 2 > 0 c_2>0 c 2 > 0 is a constant that is independent of δ \delta δ , n i n n_{\mathrm{in}} n in , t o p h t_{\mathrm{op}_h} t op h , and ω k \omega_k ω k , but depends on s h s_h s h , p p p , A A A , D D D , M P M_{\mathrm{P}} M P , and R h R_h R h . Since ω k \omega_k ω k is deterministic, Lemma E.21 applies to the regression problem with target h ω k ∗ h^*_{\omega_k} h ω k ∗ and gives Pr ( ( E ω k N T K ) c ) ≤ δ / N \Pr((\mathcal{E}^{\mathrm{NTK}}_{\omega_k})^c)\leq\delta/N Pr (( E ω k NTK ) c ) ≤ δ / N . Therefore, by a union bound, Pr ( E n e t c ) ≤ δ \Pr(\mathcal{E}_{\mathrm{net}}^c)\leq\delta Pr ( E net c ) ≤ δ . On E n e t ∩ E λ 0 h \mathcal{E}_{\mathrm{net}}\cap\mathcal{E}^h_{\lambda_0} E net ∩ E λ 0 h , for every ω ∈ Ω \omega\in\Omega ω ∈ Ω and its nearest net point ω k \omega_k ω k , i.e., ∥ ω − ω k ∥ ≤ ε \|\omega-\omega_k\|\leq\varepsilon ∥ ω − ω k ∥ ≤ ε , applying Lemma D.3 together with L Ψ ≤ L Ψ ∗ L_{\Psi}\leq L^*_{\Psi} L Ψ ≤ L Ψ ∗ gives:
∥ h t o p h , ω , ω N T K − h ω ∗ ∥ L 2 ( X ) ≤ ∥ h t o p h , ω k , ω k N T K − h ω k ∗ ∥ L 2 ( X ) + L Ψ ∗ ε ≤ c 2 ( ln 6 N δ ) 2 n i n − r h + n i n − r h , \left\|h^{\mathrm{NTK}}_{t_{\mathrm{op}_h,\omega}\!,\omega}-h^*_{\omega}\right\|_{L^2(\mathcal{X})}\leq\left\|h^{\mathrm{NTK}}_{t_{\mathrm{op}_h,\omega_k}\!,\omega_k}-h^*_{\omega_k}\right\|_{L^2(\mathcal{X})}+L^*_{\Psi}\varepsilon\leq c_2\left(\ln\frac{6N}{\delta}\right)^2n_{\mathrm{in}}^{-r_h}+n_{\mathrm{in}}^{-r_h}, h t op h , ω , ω NTK − h ω ∗ L 2 ( X ) ≤ h t op h , ω k , ω k NTK − h ω k ∗ L 2 ( X ) + L Ψ ∗ ε ≤ c 2 ( ln δ 6 N ) 2 n in − r h + n in − r h ,
by the choice L Ψ ⋆ ϵ = n i n − r h L_{\Psi}^{\star}\epsilon=n_{\mathrm{in}}^{-r_h} L Ψ ⋆ ϵ = n in − r h . Setting F : = E n e t ∩ E λ 0 ⋆ h \mathcal{F}:=\mathcal{E}_{\mathrm{net}}\cap\mathcal{E}_{\lambda_{0}^{\star}}^{h} F := E net ∩ E λ 0 ⋆ h , so that Pr ( F ∁ ) ≤ 2 δ \Pr(\mathcal{F}^{\complement})\le2\delta Pr ( F ∁ ) ≤ 2 δ , decomposing over F \mathcal{F} F and F ∁ \mathcal{F}^{\complement} F ∁ , and using the identity ( a + b ) 2 ≤ 2 ( a 2 + b 2 ) (a+b)^2\le2(a^2+b^2) ( a + b ) 2 ≤ 2 ( a 2 + b 2 ) , we obtain:
E [ sup ω ∈ Ω ∥ h t o p , ω N T K − h ω ⋆ ∥ L 2 ( X ) 2 ] = E [ sup ω ∈ Ω ∥ h t o p , ω N T K − h ω ⋆ ∥ L 2 ( X ) 2 1 F ] + E [ sup ω ∈ Ω ∥ h t o p , ω N T K − h ω ⋆ ∥ L 2 ( X ) 2 1 F ∁ ] ≤ 2 c 2 2 ( ln 6 N δ ) 4 n i n − 2 r h + 2 n i n − 2 r h + 4 ( ( κ N T K ) 2 D 2 n i n 2 γ h + D 2 ) δ , (27) \begin{aligned}
\mathbb{E}\left[\sup_{\omega\in\Omega}\left\|h_{t_{\mathrm{op},\omega}}^{\mathrm{NTK}}-h_{\omega}^{\star}\right\|_{L^2(\mathcal{X})}^{2}\right]
={}&\mathbb{E}\left[\sup_{\omega\in\Omega}\left\|h_{t_{\mathrm{op},\omega}}^{\mathrm{NTK}}-h_{\omega}^{\star}\right\|_{L^2(\mathcal{X})}^{2}\mathbb{1}_{\mathcal{F}}\right]\\
&+\mathbb{E}\left[\sup_{\omega\in\Omega}\left\|h_{t_{\mathrm{op},\omega}}^{\mathrm{NTK}}-h_{\omega}^{\star}\right\|_{L^2(\mathcal{X})}^{2}\mathbb{1}_{\mathcal{F}^{\complement}}\right]\\
\le{}&2c_2^2\left(\ln\frac{6N}{\delta}\right)^4n_{\mathrm{in}}^{-2r_h}+2n_{\mathrm{in}}^{-2r_h}+4\left((\kappa^{\mathrm{NTK}})^2D^2n_{\mathrm{in}}^{2\gamma_h}+D^2\right)\delta, \tag*{(27)}
\end{aligned} E [ ω ∈ Ω sup h t op , ω NTK − h ω ⋆ L 2 ( X ) 2 ] = ≤ E [ ω ∈ Ω sup h t op , ω NTK − h ω ⋆ L 2 ( X ) 2 1 F ] + E [ ω ∈ Ω sup h t op , ω NTK − h ω ⋆ L 2 ( X ) 2 1 F ∁ ] 2 c 2 2 ( ln δ 6 N ) 4 n in − 2 r h + 2 n in − 2 r h + 4 ( ( κ NTK ) 2 D 2 n in 2 γ h + D 2 ) δ , (27)
where we apply Lemmas E.15 and F.3 on the event F ∁ \mathcal{F}^{\complement} F ∁ to bound ∥ h t o p , ω N T K ∥ L 2 ( X ) 2 \left\|h_{t_{\mathrm{op},\omega}}^{\mathrm{NTK}}\right\|_{L^2(\mathcal{X})}^{2} h t op , ω NTK L 2 ( X ) 2 by ( κ N T K ) 2 D 2 n i n 2 γ h (\kappa^{\mathrm{NTK}})^2D^2n_{\mathrm{in}}^{2\gamma_h} ( κ NTK ) 2 D 2 n in 2 γ h and ∥ h ω ⋆ ∥ L 2 ( X ) 2 \left\|h_{\omega}^{\star}\right\|_{L^2(\mathcal{X})}^{2} ∥ h ω ⋆ ∥ L 2 ( X ) 2 by D 2 D^2 D 2 for any ω ∈ Ω \omega\in\Omega ω ∈ Ω .
Expected generalization bound. Plugging Equations (26) and (27) into Equation (25) and recalling that n i n − 2 r h = n i n − s h γ h n_{\mathrm{in}}^{-2r_h}=n_{\mathrm{in}}^{-s_h\gamma_h} n in − 2 r h = n in − s h γ h , we obtain:
E [ sup ω ∈ Ω ∥ h ^ ω − h ω ⋆ ∥ L 2 ( X ) 2 ] ≤ 2 c 1 2 n i n − 1 2 ln m h + 4 c 2 2 ( ln 6 N δ ) 4 n i n − s h γ h + 4 n i n − s h γ h + a 1 n i n 2 γ h δ , (28) \mathbb{E}\left[\sup_{\omega\in\Omega}\left\|\hat{h}_{\omega}-h_{\omega}^{\star}\right\|_{L^2(\mathcal{X})}^{2}\right]\le2c_1^2n_{\mathrm{in}}^{-\frac{1}{2}}\ln m_h+4c_2^2\left(\ln\frac{6N}{\delta}\right)^4n_{\mathrm{in}}^{-s_h\gamma_h}+4n_{\mathrm{in}}^{-s_h\gamma_h}+a_1n_{\mathrm{in}}^{2\gamma_h}\delta, \tag*{(28)} E [ ω ∈ Ω sup h ^ ω − h ω ⋆ L 2 ( X ) 2 ] ≤ 2 c 1 2 n in − 2 1 ln m h + 4 c 2 2 ( ln δ 6 N ) 4 n in − s h γ h + 4 n in − s h γ h + a 1 n in 2 γ h δ , (28)
where a 1 : = 16 ( κ N T K ) 2 D 2 + 8 D 2 a_1:=16(\kappa^{\mathrm{NTK}})^2D^2+8D^2 a 1 := 16 ( κ NTK ) 2 D 2 + 8 D 2 collects the three δ \delta δ -terms, using n i n 2 γ h ≥ 1 n_{\mathrm{in}}^{2\gamma_h}\ge1 n in 2 γ h ≥ 1 . For each fixed δ ∈ ( 0 , 1 ) \delta\in(0,1) δ ∈ ( 0 , 1 ) , the net above is deterministic, so Equation (28) holds for every δ ∈ ( 0 , 1 ) \delta\in(0,1) δ ∈ ( 0 , 1 ) . We now control the δ \delta δ -dependent terms. The balls B ( ω k , ϵ / 2 ) B(\omega_k,\epsilon/2) B ( ω k , ϵ /2 ) , 1 ≤ k ≤ N 1\le k\le N 1 ≤ k ≤ N , are pairwise disjoint and are all contained in B ( ω 1 , diam ( Ω ) + ϵ / 2 ) B(\omega_1,\operatorname{diam}(\Omega)+\epsilon/2) B ( ω 1 , diam ( Ω ) + ϵ /2 ) , since ∥ ω k − ω 1 ∥ ≤ diam ( Ω ) \|\omega_k-\omega_1\|\le\operatorname{diam}(\Omega) ∥ ω k − ω 1 ∥ ≤ diam ( Ω ) . Comparing volumes, and using ϵ ≤ diam ( Ω ) \epsilon\le\operatorname{diam}(\Omega) ϵ ≤ diam ( Ω ) for n i n n_{\mathrm{in}} n in large, we obtain by [89 Proposition 4.2.10] :
N ≤ ( 1 + 2 diam ( Ω ) ϵ ) d ≤ ( 3 diam ( Ω ) ϵ ) d , hence ln N ≤ d [ ln ( 3 diam ( Ω ) ) + ln L Ψ ⋆ + r h ln n i n ] . N\le\left(1+\frac{2\operatorname{diam}(\Omega)}{\epsilon}\right)^d\le\left(\frac{3\operatorname{diam}(\Omega)}{\epsilon}\right)^d,
\qquad\text{hence}\qquad
\ln N\le d\left[\ln(3\operatorname{diam}(\Omega))+\ln L_{\Psi}^{\star}+r_h\ln n_{\mathrm{in}}\right]. N ≤ ( 1 + ϵ 2 diam ( Ω ) ) d ≤ ( ϵ 3 diam ( Ω ) ) d , hence ln N ≤ d [ ln ( 3 diam ( Ω )) + ln L Ψ ⋆ + r h ln n in ] .
By Corollary E.10, the deterministic threshold λ ⋆ \lambda_{\star} λ ⋆ satisfies:
λ ⋆ − 1 ≤ c 3 n i n 8 p δ − 4 p ( ln n i n + ln 1 δ ) 3 , \lambda_{\star}^{-1}\le c_3n_{\mathrm{in}}^{\frac{8}{p}}\delta^{-\frac{4}{p}}\left(\ln n_{\mathrm{in}}+\ln\frac{1}{\delta}\right)^3, λ ⋆ − 1 ≤ c 3 n in p 8 δ − p 4 ( ln n in + ln δ 1 ) 3 ,
where c 3 > 0 c_3>0 c 3 > 0 depends only on p p p , A A A and M P M_{\mathbb{P}} M P . Since L Ψ ⋆ = G ( 1 + κ N T K n i n / λ ⋆ ) ≤ 2 G κ N T K n i n λ ⋆ − 1 L_{\Psi}^{\star}=G(1+\kappa^{\mathrm{NTK}}n_{\mathrm{in}}/\lambda_{\star})\le2G\kappa^{\mathrm{NTK}}n_{\mathrm{in}}\lambda_{\star}^{-1} L Ψ ⋆ = G ( 1 + κ NTK n in / λ ⋆ ) ≤ 2 G κ NTK n in λ ⋆ − 1 for n i n n_{\mathrm{in}} n in large, taking logarithms, and using the bound on λ ⋆ \lambda_{\star} λ ⋆ above and the inequality ln y ≤ y \ln y\le y ln y ≤ y for y > 0 y>0 y > 0 to bound the last term, we get:
ln L Ψ ⋆ ≤ ln ( 2 G κ N T K ) + ln n i n + ln λ ⋆ − 1 ≤ ln ( 2 G κ N T K c 3 ) + ( 1 + 8 p ) ln n i n + 4 p ln 1 δ + 3 ln ( ln n i n + ln 1 δ ) ≤ C Ψ ( ln n i n + ln 1 δ ) , \begin{aligned}
\ln L_{\Psi}^{\star}\le\ln(2G\kappa^{\mathrm{NTK}})+\ln n_{\mathrm{in}}+\ln\lambda_{\star}^{-1} \\
&\le\ln(2G\kappa^{\mathrm{NTK}}c_3)+\left(1+\frac{8}{p}\right)\ln n_{\mathrm{in}}+\frac{4}{p}\ln\frac{1}{\delta}+3\ln\left(\ln n_{\mathrm{in}}+\ln\frac{1}{\delta}\right) \\
&\le C_{\Psi}\left(\ln n_{\mathrm{in}}+\ln\frac{1}{\delta}\right),
\end{aligned} ln L Ψ ⋆ ≤ ln ( 2 G κ NTK ) + ln n in + ln λ ⋆ − 1 ≤ ln ( 2 G κ NTK c 3 ) + ( 1 + p 8 ) ln n in + p 4 ln δ 1 + 3 ln ( ln n in + ln δ 1 ) ≤ C Ψ ( ln n in + ln δ 1 ) ,
with C Ψ : = ∣ ln ( 2 G κ N T K c 3 ) ∣ + 4 + 8 p > 0 C_{\Psi}:=\left|\ln(2G\kappa^{\mathrm{NTK}}c_3)\right|+4+\frac{8}{p}>0 C Ψ := ln ( 2 G κ NTK c 3 ) + 4 + p 8 > 0 , where the last inequality uses ln n i n ≥ 1 \ln n_{\mathrm{in}}\ge1 ln n in ≥ 1 for n i n ≥ 3 n_{\mathrm{in}}\ge3 n in ≥ 3 . Using also r h ≤ 1 r_h\le1 r h ≤ 1 and ln n i n ≥ 1 \ln n_{\mathrm{in}}\ge1 ln n in ≥ 1 , we obtain:
ln N ≤ c d ( ln n i n + ln 1 δ ) , with c : = ∣ ln ( 3 diam ( Ω ) ) ∣ + C Ψ + 1. \ln N\le cd\left(\ln n_{\mathrm{in}}+\ln\frac{1}{\delta}\right),
\qquad\text{with}\qquad
c:=|\ln(3\operatorname{diam}(\Omega))|+C_{\Psi}+1. ln N ≤ c d ( ln n in + ln δ 1 ) , with c := ∣ ln ( 3 diam ( Ω )) ∣ + C Ψ + 1.
This implies that ln 6 N δ = ln 6 + ln N + ln 1 δ ≤ c ′ d ( ln n i n + ln 1 δ ) \ln\frac{6N}{\delta}=\ln6+\ln N+\ln\frac{1}{\delta}\le c'd\left(\ln n_{\mathrm{in}}+\ln\frac{1}{\delta}\right) ln δ 6 N = ln 6 + ln N + ln δ 1 ≤ c ′ d ( ln n in + ln δ 1 ) with c ′ : = c + 2 c':=c+2 c ′ := c + 2 , using ln 6 ≤ 2 ln n i n \ln6\le2\ln n_{\mathrm{in}} ln 6 ≤ 2 ln n in for n i n ≥ 3 n_{\mathrm{in}}\ge3 n in ≥ 3 and d ≥ 1 d\ge1 d ≥ 1 , where c ′ > 0 c'>0 c ′ > 0 is a constant depending on diam ( Ω ) \operatorname{diam}(\Omega) diam ( Ω ) , p p p , A A A , M P M_{\mathbb{P}} M P , and G G G . Hence, using the identity ( u + v ) 4 ≤ 8 ( u 4 + v 4 ) (u+v)^4\le8(u^4+v^4) ( u + v ) 4 ≤ 8 ( u 4 + v 4 ) and ln 1 δ ≤ ln 6 δ \ln\frac{1}{\delta}\le\ln\frac{6}{\delta} ln δ 1 ≤ ln δ 6 , we obtain:
( ln 6 N δ ) 4 ≤ 8 c ′ 4 d 4 [ ( ln n i n ) 4 + ( ln 1 δ ) 4 ] ≤ 8 c ′ 4 d 4 [ ( ln n i n ) 4 + ( ln 6 δ ) 4 ] . \left(\ln\frac{6N}{\delta}\right)^4\le8c'^4d^4\left[(\ln n_{\mathrm{in}})^4+\left(\ln\frac{1}{\delta}\right)^4\right]\le8c'^4d^4\left[(\ln n_{\mathrm{in}})^4+\left(\ln\frac{6}{\delta}\right)^4\right]. ( ln δ 6 N ) 4 ≤ 8 c ′4 d 4 [ ( ln n in ) 4 + ( ln δ 1 ) 4 ] ≤ 8 c ′4 d 4 [ ( ln n in ) 4 + ( ln δ 6 ) 4 ] .
Substituting into Equation (28) , the δ \delta δ -dependent part of the bound is at most φ ( δ ) \varphi(\delta) φ ( δ ) , where:
φ ( δ ) : = a 2 d 4 ( ln 6 δ ) 4 n i n − s h γ h + a 1 n i n 2 γ h δ , a 2 : = 32 c 2 2 c ′ 4 , \varphi(\delta):=a_2d^4\left(\ln\frac{6}{\delta}\right)^4n_{\mathrm{in}}^{-s_h\gamma_h}+a_1n_{\mathrm{in}}^{2\gamma_h}\delta,\qquad a_2:=32c_2^2c'^4, φ ( δ ) := a 2 d 4 ( ln δ 6 ) 4 n in − s h γ h + a 1 n in 2 γ h δ , a 2 := 32 c 2 2 c ′4 ,
and the remaining, δ \delta δ -free part is 2 c 1 2 m h 1 2 ln m h + ( a 2 d 4 ( ln n i n ) 4 + 4 ) n i n − s h γ h 2c_1^2m_h^{\frac{1}{2}}\ln m_h+(a_2d^4(\ln n_{in})^4+4)n_{in}^{-s_h\gamma_h} 2 c 1 2 m h 2 1 ln m h + ( a 2 d 4 ( ln n in ) 4 + 4 ) n in − s h γ h . Minimizing φ \varphi φ over δ ∈ ( 0 , 1 ) \delta\in(0,1) δ ∈ ( 0 , 1 ) exactly as in the proof of Proposition E.25 gives:
δ ⋆ = 108 a 2 d 4 a 1 n i n − ( s h + 2 ) γ h W 0 3 ( z h ) , z h : = ( a 1 18 a 2 d 4 ) 1 3 n i n 1 3 ( s h + 2 ) γ h (29) \delta^\star=\frac{108a_2d^4}{a_1}n_{in}^{-(s_h+2)\gamma_h}W_0^3(z_h),\qquad z_h:=\left(\frac{a_1}{18a_2d^4}\right)^{\frac{1}{3}}n_{in}^{\frac{1}{3}(s_h+2)\gamma_h} \tag*{(29)} δ ⋆ = a 1 108 a 2 d 4 n in − ( s h + 2 ) γ h W 0 3 ( z h ) , z h := ( 18 a 2 d 4 a 1 ) 3 1 n in 3 1 ( s h + 2 ) γ h (29)
and φ ( δ ⋆ ) = 27 a 2 d 4 W 0 3 ( z h ) ( 3 W 0 ( z h ) + 4 ) n i n − s h γ h \varphi(\delta^\star)=27a_2d^4W_0^3(z_h)(3W_0(z_h)+4)n_{in}^{-s_h\gamma_h} φ ( δ ⋆ ) = 27 a 2 d 4 W 0 3 ( z h ) ( 3 W 0 ( z h ) + 4 ) n in − s h γ h , where W 0 W_0 W 0 is the principal branch of the Lambert W W W function. Since d ≥ 1 d\ge1 d ≥ 1 , we have z h ≤ ( a 1 / ( 18 a 2 ) ) 1 / 3 n i n ( s h + 2 ) γ h / 3 z_h\le(a_1/(18a_2))^{1/3}n_{in}^{(s_h+2)\gamma_h/3} z h ≤ ( a 1 / ( 18 a 2 ) ) 1/3 n in ( s h + 2 ) γ h /3 , so Lemma I.3 bounds W 0 3 ( z h ) ( 3 W 0 ( z h ) + 4 ) W_0^3(z_h)(3W_0(z_h)+4) W 0 3 ( z h ) ( 3 W 0 ( z h ) + 4 ) by a constant multiple of ( ln n i n ) 4 (\ln n_{in})^4 ( ln n in ) 4 that does not involve d d d . This yields the rate d 4 ( ln n i n ) 4 n i n − s h γ h d^4(\ln n_{in})^4n_{in}^{-s_h\gamma_h} d 4 ( ln n in ) 4 n in − s h γ h . For the width, note that taking n i n n_{in} n in large enough so that n i n ( s h + 2 ) γ h ≥ 18 e 3 a 2 d 4 / a 1 n_{in}^{(s_h+2)\gamma_h}\ge18e^3a_2d^4/a_1 n in ( s h + 2 ) γ h ≥ 18 e 3 a 2 d 4 / a 1 gives z h ≥ e z_h\ge e z h ≥ e , hence W 0 ( z h ) ≥ 1 W_0(z_h)\ge1 W 0 ( z h ) ≥ 1 and
δ ⋆ ≥ 108 a 2 d 4 a 1 n i n − ( s h + 2 ) γ h ≥ 108 a 2 a 1 n i n − ( s h + 2 ) γ h , \delta^\star\ge\frac{108a_2d^4}{a_1}n_{in}^{-(s_h+2)\gamma_h}\ge\frac{108a_2}{a_1}n_{in}^{-(s_h+2)\gamma_h}, δ ⋆ ≥ a 1 108 a 2 d 4 n in − ( s h + 2 ) γ h ≥ a 1 108 a 2 n in − ( s h + 2 ) γ h ,
so that, using ( s h + 2 ) γ h ≤ 4 (s_h+2)\gamma_h\le4 ( s h + 2 ) γ h ≤ 4 and ln n i n ≥ 1 \ln n_{in}\ge1 ln n in ≥ 1 , we obtain:
1 δ ⋆ ≤ a 1 108 a 2 n i n 4 , ln 1 δ ⋆ ≤ ( ∣ ln a 1 108 a 2 ∣ + 4 ) ln n i n . \frac{1}{\delta^\star}\le\frac{a_1}{108a_2}n_{in}^4,\qquad\ln\frac{1}{\delta^\star}\le\left(\left|\ln\frac{a_1}{108a_2}\right|+4\right)\ln n_{in}. δ ⋆ 1 ≤ 108 a 2 a 1 n in 4 , ln δ ⋆ 1 ≤ ( ln 108 a 2 a 1 + 4 ) ln n in .
Both bounds are free of d d d , so the condition m h ≥ poly h ( n i n , 1 / δ ⋆ , ln n i n , ln ( 1 / δ ⋆ ) ) m_h\ge\operatorname{poly}_h(n_{in},1/\delta^\star,\ln n_{in},\ln(1/\delta^\star)) m h ≥ poly h ( n in , 1/ δ ⋆ , ln n in , ln ( 1/ δ ⋆ )) is implied by m h ≥ poly h ( n i n , ln n i n ) m_h\ge\operatorname{poly}_h(n_{in},\ln n_{in}) m h ≥ poly h ( n in , ln n in ) . As a consequence, we obtain:
E [ sup ω ∈ Ω ∥ h ^ ω − h ω ⋆ ∥ L 2 ( X ) 2 ] ≤ C 1 d 4 ( ln n i n ) 4 n i n − s h γ h , \mathbb{E}\left[\sup_{\omega\in\Omega}\left\|\hat{h}_\omega-h_\omega^\star\right\|_{L^2(\mathcal{X})}^2\right]\le C_1d^4(\ln n_{in})^4n_{in}^{-s_h\gamma_h}, E [ ω ∈ Ω sup h ^ ω − h ω ⋆ L 2 ( X ) 2 ] ≤ C 1 d 4 ( ln n in ) 4 n in − s h γ h ,
provided that m h ≥ poly h ( n i n , ln n i n ) m_h\ge\operatorname{poly}_h(n_{in},\ln n_{in}) m h ≥ poly h ( n in , ln n in ) , where C 1 > 0 C_1>0 C 1 > 0 is a constant independent of n i n n_{in} n in and t o p h t_{\mathrm{op}_h} t op h , but depends on s h s_h s h , p p p , A A A , D D D , G G G , M P M_{\mathbb{P}} M P , diam ( Ω ) \operatorname{diam}(\Omega) diam ( Ω ) , and R h R_h R h , and the coefficients of poly h \operatorname{poly}_h poly h depend on s h s_h s h , A A A , D D D , G G G , M P M_{\mathbb{P}} M P , diam ( Ω ) \operatorname{diam}(\Omega) diam ( Ω ) , p p p , and R h R_h R h . This concludes the proof.
Proposition D.5 (Expected error in the regression function). Consider n o u t n_{out} n o u t i.i.d. samples { ( x ~ j , y ~ j ) } j = 1 n o u t ∼ P X Y \{(\tilde{x}_j,\tilde{y}_j)\}_{j=1}^{n_{out}}\sim P_{XY} {( x ~ j , y ~ j ) } j = 1 n o u t ∼ P X Y and assume p ≥ 2 p\ge2 p ≥ 2 . Let Assumptions B to D and F hold. If we choose a stopping time t o p b = n o u t s b ( p + 1 ) + p s b ( p + 1 ) + p t_{\mathrm{op}_b}=n_{out}^{\frac{s_b(p+1)+p}{s_b(p+1)+p}} t op b = n o u t s b ( p + 1 ) + p s b ( p + 1 ) + p with 1 p + 1 < s b ≤ 2 \frac{1}{p+1}<s_b\le2 p + 1 1 < s b ≤ 2 , then, considering the early-stopped truncated neural network predictor b ^ \hat{b} b ^ for the regression function b ⋆ b^\star b ⋆ , there exist a positive constant C 2 > 0 C_2>0 C 2 > 0 and a polynomial poly b \operatorname{poly}_b poly b such that when n o u t n_{out} n o u t is sufficiently large and the width satisfies m b ≥ poly b ( n o u t , ln n o u t ) m_b\ge\operatorname{poly}_b(n_{out},\ln n_{out}) m b ≥ poly b ( n o u t , ln n o u t ) , the following holds:
E [ ∥ b ^ − b ⋆ ∥ L 2 ( X ) 2 ] ≤ C 2 ( ln n o u t ) 4 n o u t − s b ( p + 1 ) s b ( p + 1 ) + p , \mathbb{E}\left[\left\|\hat{b}-b^\star\right\|_{L^2(\mathcal{X})}^2\right]\le C_2(\ln n_{out})^4n_{out}^{-\frac{s_b(p+1)}{s_b(p+1)+p}}, E [ b ^ − b ⋆ L 2 ( X ) 2 ] ≤ C 2 ( ln n o u t ) 4 n o u t − s b ( p + 1 ) + p s b ( p + 1 ) , where C 2 > 0 C_2>0 C 2 > 0 is a constant that is independent of n o u t n_{out} n o u t and t o p b t_{\mathrm{op}_b} t op b , but depends on s b s_b s b , p p p , A A A , B B B , M P M_{\mathbb{P}} M P , and R b R_b R b , and the coefficients of poly b \operatorname{poly}_b poly b depend on s b s_b s b , A A A , B B B , M P M_{\mathbb{P}} M P , R b R_b R b , and p p p . The expectation here is with respect to random samples and random initialization.
Proof. This is a direct consequence of Proposition E.25, where the neural network approximating b ⋆ b^\star b ⋆ of the form in Equation (5) is trained using the empirical loss L ^ o u t \hat{L}_{out} L ^ o u t , defined for any h ∈ L 2 ( X ) h\in L^2(\mathcal{X}) h ∈ L 2 ( X ) by:
L ^ o u t ( h ) = 1 2 n o u t ∑ j = 1 n o u t ( h ( x ~ j ) − y ~ j ) 2 . \hat{L}_{out}(h)=\frac{1}{2n_{out}}\sum_{j=1}^{n_{out}}(h(\tilde{x}_j)-\tilde{y}_j)^2. L ^ o u t ( h ) = 2 n o u t 1 j = 1 ∑ n o u t ( h ( x ~ j ) − y ~ j ) 2 .
Proposition D.6 (Uniform-in-ω \omega ω expected error in the adjoint function). Consider n i n n_{in} n in i.i.d. samples { ( x i , z i ) } i = 1 n i n ∼ P X Z \{(x_i,z_i)\}_{i=1}^{n_{in}}\sim P_{XZ} {( x i , z i ) } i = 1 n in ∼ P X Z , n o u t n_{out} n o u t i.i.d. samples { ( x ~ j , y ~ j ) } j = 1 n o u t ∼ P X Y \{(\tilde{x}_j,\tilde{y}_j)\}_{j=1}^{n_{out}}\sim P_{XY} {( x ~ j , y ~ j ) } j = 1 n o u t ∼ P X Y , and assume p ≥ 2 p\ge2 p ≥ 2 . Let Assumptions A to F hold. If we choose stopping times t o p h = n i n s h ( p + 1 ) + p s h ( p + 1 ) + p t_{\mathrm{op}_h}=n_{in}^{\frac{s_h(p+1)+p}{s_h(p+1)+p}} t op h = n in s h ( p + 1 ) + p s h ( p + 1 ) + p and t o p b = n o u t s b ( p + 1 ) + p s b ( p + 1 ) + p t_{\mathrm{op}_b}=n_{out}^{\frac{s_b(p+1)+p}{s_b(p+1)+p}} t op b = n o u t s b ( p + 1 ) + p s b ( p + 1 ) + p with 1 p + 1 < s h ≤ 2 \frac{1}{p+1}<s_h\le2 p + 1 1 < s h ≤ 2 and 1 p + 1 < s b ≤ 2 \frac{1}{p+1}<s_b\le2 p + 1 1 < s b ≤ 2 , then, for every ω ∈ Ω \omega\in\Omega ω ∈ Ω , considering the early-stopped truncated neural network predictors h ^ ω \hat{h}_\omega h ^ ω and b ^ \hat{b} b ^ for the inner solution h ω ⋆ h_\omega^\star h ω ⋆ and the regression function b ⋆ b^\star b ⋆ , respectively, there exist two positive constants C 1 , C 2 > 0 C_1,C_2>0 C 1 , C 2 > 0 and two polynomials poly h \operatorname{poly}_h poly h and poly b \operatorname{poly}_b poly b
such that when n i n n_{\mathrm{in}} n in and n o u t n_{\mathrm{out}} n out are sufficiently large and the widths satisfy m h ≥ poly h ( n i n , ln n i n ) m_h \ge\operatorname{poly}_h(n_{\mathrm{in}},\ln n_{\mathrm{in}}) m h ≥ poly h ( n in , ln n in ) and m b ≥ poly b ( n o u t , ln n o u t ) m_b \ge\operatorname{poly}_b(n_{\mathrm{out}},\ln n_{\mathrm{out}}) m b ≥ poly b ( n out , ln n out ) , the following holds:
E [ sup ω ∈ Ω ∥ a ^ ω − a ω ⋆ ∥ L 2 ( X ) 2 ] ≤ 2 C 1 d 4 ( ln n i n ) 4 n i n − s h ( p + 1 ) s h ( p + 1 ) + p + 2 C 2 ( ln n o u t ) 4 n o u t − s b ( p + 1 ) s b ( p + 1 ) + p . \mathbb{E}\left[\sup_{\omega\in\Omega}\left\|\hat{a}_{\omega}-a_{\omega}^{\star}\right\|_{L^2(\mathcal{X})}^2\right]\leq2C_1d^4(\ln n_{\mathrm{in}})^4n_{\mathrm{in}}^{-\frac{s_h(p+1)}{s_h(p+1)+p}}+2C_2(\ln n_{\mathrm{out}})^4n_{\mathrm{out}}^{-\frac{s_b(p+1)}{s_b(p+1)+p}}. E [ ω ∈ Ω sup ∥ a ^ ω − a ω ⋆ ∥ L 2 ( X ) 2 ] ≤ 2 C 1 d 4 ( ln n in ) 4 n in − s h ( p + 1 ) + p s h ( p + 1 ) + 2 C 2 ( ln n out ) 4 n out − s b ( p + 1 ) + p s b ( p + 1 ) .
where a ^ ω = − h ^ ω + b ^ \hat{a}_{\omega}=-\hat{h}_{\omega}+\hat{b} a ^ ω = − h ^ ω + b ^ and a ω ⋆ = − h ω ⋆ + b ⋆ a_{\omega}^{\star}=-h_{\omega}^{\star}+b^{\star} a ω ⋆ = − h ω ⋆ + b ⋆ . C 1 , C 2 > 0 C_1,C_2>0 C 1 , C 2 > 0 and poly h , poly b \operatorname{poly}_h,\operatorname{poly}_b poly h , poly b are the same as in Propositions D.4 and D.5, respectively. The expectation here is with respect to random samples and random initialization.
Proof. Let ω ∈ Ω \omega\in\Omega ω ∈ Ω . Using the triangle inequality and the identity ( a + b ) 2 ≤ 2 ( a 2 + b 2 ) (a+b)^2\leq2(a^2+b^2) ( a + b ) 2 ≤ 2 ( a 2 + b 2 ) , we have:
∥ a ^ ω − a ω ⋆ ∥ L 2 ( X ) 2 = ∥ ( − h ^ ω + b ^ ) − ( − h ω ⋆ + b ⋆ ) ∥ L 2 ( X ) 2 \left\|\hat{a}_{\omega}-a_{\omega}^{\star}\right\|_{L^2(\mathcal{X})}^2=\left\|\left(-\hat{h}_{\omega}+\hat{b}\right)-\left(-h_{\omega}^{\star}+b^{\star}\right)\right\|_{L^2(\mathcal{X})}^2 ∥ a ^ ω − a ω ⋆ ∥ L 2 ( X ) 2 = ( − h ^ ω + b ^ ) − ( − h ω ⋆ + b ⋆ ) L 2 ( X ) 2
≤ 2 ( ∥ h ^ ω − h ω ⋆ ∥ L 2 ( X ) 2 + ∥ b ^ − b ⋆ ∥ L 2 ( X ) 2 ) . \leq2\left(\left\|\hat{h}_{\omega}-h_{\omega}^{\star}\right\|_{L^2(\mathcal{X})}^2+\left\|\hat{b}-b^{\star}\right\|_{L^2(\mathcal{X})}^2\right). ≤ 2 ( h ^ ω − h ω ⋆ L 2 ( X ) 2 + b ^ − b ⋆ L 2 ( X ) 2 ) .
Taking the supremum over ω ∈ Ω \omega\in\Omega ω ∈ Ω followed by the expectation over the random samples and random initialization on both sides, we obtain:
E [ sup ω ∈ Ω ∥ a ^ ω − a ω ⋆ ∥ L 2 ( X ) 2 ] ≤ 2 ( E [ sup ω ∈ Ω ∥ h ^ ω − h ω ⋆ ∥ L 2 ( X ) 2 ] + E [ ∥ b ^ − b ⋆ ∥ L 2 ( X ) 2 ] ) . (30) \mathbb{E}\left[\sup_{\omega\in\Omega}\left\|\hat{a}_{\omega}-a_{\omega}^{\star}\right\|_{L^2(\mathcal{X})}^2\right]\leq2\left(\mathbb{E}\left[\sup_{\omega\in\Omega}\left\|\hat{h}_{\omega}-h_{\omega}^{\star}\right\|_{L^2(\mathcal{X})}^2\right]+\mathbb{E}\left[\left\|\hat{b}-b^{\star}\right\|_{L^2(\mathcal{X})}^2\right]\right). \tag*{(30)} E [ ω ∈ Ω sup ∥ a ^ ω − a ω ⋆ ∥ L 2 ( X ) 2 ] ≤ 2 ( E [ ω ∈ Ω sup h ^ ω − h ω ⋆ L 2 ( X ) 2 ] + E [ b ^ − b ⋆ L 2 ( X ) 2 ] ) . (30)
By Proposition D.4 , we know that there exist a positive constant C 1 > 0 C_1>0 C 1 > 0 and a polynomial poly h \operatorname{poly}_h poly h such that when n i n n_{\mathrm{in}} n in is sufficiently large and m h ≥ poly h ( n i n , ln n i n ) m_h\geq\operatorname{poly}_h(n_{\mathrm{in}},\ln n_{\mathrm{in}}) m h ≥ poly h ( n in , ln n in ) , the following holds:
E [ sup ω ∈ Ω ∥ h ^ ω − h ω ⋆ ∥ L 2 ( X ) 2 ] ≤ C 1 d 4 ( ln n i n ) 4 n i n − s h ( p + 1 ) s h ( p + 1 ) + p , \mathbb{E}\left[\sup_{\omega\in\Omega}\left\|\hat{h}_{\omega}-h_{\omega}^{\star}\right\|_{L^2(\mathcal{X})}^2\right]\leq C_1d^4(\ln n_{\mathrm{in}})^4n_{\mathrm{in}}^{-\frac{s_h(p+1)}{s_h(p+1)+p}}, E [ ω ∈ Ω sup h ^ ω − h ω ⋆ L 2 ( X ) 2 ] ≤ C 1 d 4 ( ln n in ) 4 n in − s h ( p + 1 ) + p s h ( p + 1 ) ,
where the exact dependence of C 1 C_1 C 1 and the coefficients of poly h \operatorname{poly}_h poly h are specified in Proposition D.4 . Additionally, by Proposition D.5, we know that there exist a positive constant C 2 > 0 C_2>0 C 2 > 0 and a polynomial poly b \operatorname{poly}_b poly b such that when n o u t n_{\mathrm{out}} n out is sufficiently large and m b ≥ poly b ( n o u t , ln n o u t ) m_b\geq\operatorname{poly}_b(n_{\mathrm{out}},\ln n_{\mathrm{out}}) m b ≥ poly b ( n out , ln n out ) , the following holds:
E [ ∥ b ^ − b ⋆ ∥ L 2 ( X ) 2 ] ≤ C 2 ( ln n o u t ) 4 n o u t − s b ( p + 1 ) s b ( p + 1 ) + p , \mathbb{E}\left[\left\|\hat{b}-b^{\star}\right\|_{L^2(\mathcal{X})}^2\right]\leq C_2(\ln n_{\mathrm{out}})^4n_{\mathrm{out}}^{-\frac{s_b(p+1)}{s_b(p+1)+p}}, E [ b ^ − b ⋆ L 2 ( X ) 2 ] ≤ C 2 ( ln n out ) 4 n out − s b ( p + 1 ) + p s b ( p + 1 ) ,
where the exact dependence of C 2 > 0 C_2>0 C 2 > 0 and the coefficients of poly b \operatorname{poly}_b poly b are provided in Proposition D.5. After combining both upper-bounds in Equation (30) , we get:
E [ sup ω ∈ Ω ∥ a ^ ω − a ω ⋆ ∥ L 2 ( X ) 2 ] ≤ 2 C 1 d 4 ( ln n i n ) 4 n i n − s h ( p + 1 ) s h ( p + 1 ) + p + 2 C 2 ( ln n o u t ) 4 n o u t − s b ( p + 1 ) s b ( p + 1 ) + p . \mathbb{E}\left[\sup_{\omega\in\Omega}\left\|\hat{a}_{\omega}-a_{\omega}^{\star}\right\|_{L^2(\mathcal{X})}^2\right]\leq2C_1d^4(\ln n_{\mathrm{in}})^4n_{\mathrm{in}}^{-\frac{s_h(p+1)}{s_h(p+1)+p}}+2C_2(\ln n_{\mathrm{out}})^4n_{\mathrm{out}}^{-\frac{s_b(p+1)}{s_b(p+1)+p}}. E [ ω ∈ Ω sup ∥ a ^ ω − a ω ⋆ ∥ L 2 ( X ) 2 ] ≤ 2 C 1 d 4 ( ln n in ) 4 n in − s h ( p + 1 ) + p s h ( p + 1 ) + 2 C 2 ( ln n out ) 4 n out − s b ( p + 1 ) + p s b ( p + 1 ) .
Generalization Analysis of Neural Network Gradient Flow in the NTK Regime
Problem setting
Let p p p be a fixed positive integer, X ⊂ R p \mathcal{X}\subset\mathbb{R}^p X ⊂ R p , Y ⊂ R \mathcal{Y}\subset\mathbb{R} Y ⊂ R , and ρ \rho ρ be a fixed but unknown probability distribution supported on X × Y \mathcal{X}\times\mathcal{Y} X × Y . We denote by μ \mu μ the marginal distribution of ρ \rho ρ on X \mathcal{X} X , which we assume is absolutely continuous with respect to the Lebesgue measure on R p \mathbb{R}^p R p , i.e., it admits a density function f μ f_\mu f μ . Let L 2 ( X , μ ) L^2(\mathcal{X},\mu) L 2 ( X , μ ) and L ∞ ( X ) L^\infty(\mathcal{X}) L ∞ ( X ) denote respectively the spaces of square-integrable with respect to μ \mu μ and essentially bounded functions defined on the input space X \mathcal{X} X and taking values in Y \mathcal{Y} Y , endowed with the following norms:
∥ f ∥ L 2 ( X , μ ) = ( E x ∼ μ [ ∣ f ( x ) ∣ 2 ] ) 1 2 , ∥ f ∥ ∞ = ess sup x ∈ X ∣ f ( x ) ∣ . \|f\|_{L^2(\mathcal{X},\mu)}=\left(\mathbb{E}_{x\sim\mu}\left[|f(x)|^2\right]\right)^{\frac{1}{2}},\qquad\|f\|_{\infty}=\operatorname*{ess\,sup}_{x\in\mathcal{X}}|f(x)|. ∥ f ∥ L 2 ( X , μ ) = ( E x ∼ μ [ ∣ f ( x ) ∣ 2 ] ) 2 1 , ∥ f ∥ ∞ = x ∈ X ess sup ∣ f ( x ) ∣.
For conciseness, we write L 2 ( X ) L^2(\mathcal{X}) L 2 ( X ) for L 2 ( X , μ ) L^2(\mathcal{X},\mu) L 2 ( X , μ ) . Suppose we are given a training set of n n n i.i.d. samples { ( x i , y i ) } i = 1 n ⊂ X × Y \{(x_i,y_i)\}_{i=1}^n\subset\mathcal{X}\times\mathcal{Y} {( x i , y i ) } i = 1 n ⊂ X × Y drawn from the distribution ρ \rho ρ . Using those n n n samples, our goal is to learn a neural network that approximates the regression function f ⋆ : X → Y f^{\star}:\mathcal{X}\to\mathcal{Y} f ⋆ : X → Y in L 2 ( X ) L^2(\mathcal{X}) L 2 ( X ) , defined for any x ∈ X x\in\mathcal{X} x ∈ X by f ⋆ ( x ) = E y ∼ ρ ( ⋅ ∣ x ) [ y ∣ x ] f^{\star}(x)=\mathbb{E}_{y\sim\rho(\cdot\mid x)}[y\mid x] f ⋆ ( x ) = E y ∼ ρ ( ⋅ ∣ x ) [ y ∣ x ] , which minimizes the following population loss over L 2 ( X ) L^2(\mathcal{X}) L 2 ( X ) :
L ( f ) = 1 2 E ( x , y ) ∼ ρ [ ∣ f ( x ) − y ∣ 2 ] . L(f)=\frac{1}{2}\mathbb{E}_{(x,y)\sim\rho}\left[|f(x)-y|^2\right]. L ( f ) = 2 1 E ( x , y ) ∼ ρ [ ∣ f ( x ) − y ∣ 2 ] .
Throughout Appendix E, we denote by X = [ x 1 , … , x n ] T ∈ R n × p \mathbf{X}=[x_1,\ldots,x_n]^{\mathsf{T}}\in\mathbb{R}^{n\times p} X = [ x 1 , … , x n ] T ∈ R n × p the design matrix, by Y = [ y 1 , … , y n ] T ∈ R n \mathbf{Y}=[y_1,\ldots,y_n]^{\mathsf{T}}\in\mathbb{R}^n Y = [ y 1 , … , y n ] T ∈ R n the target vector, and by S p = { x ∈ R p + 1 ∣ ∥ x ∥ = 1 } \mathbb{S}^{p}=\{x\in\mathbb{R}^{p+1}\mid\|x\|=1\} S p = { x ∈ R p + 1 ∣ ∥ x ∥ = 1 } the unit sphere in R p + 1 \mathbb{R}^{p+1} R p + 1 .
Neural network architecture and infinite-width training dynamics
Neural network architecture. We consider a two-layer ReLU neural network predictor f θ N N f^{\mathrm{NN}}_{\theta} f θ NN for f ⋆ f^{\star} f ⋆ , defined for any x ∈ X x \in\mathcal{X} x ∈ X by:
f θ N N ( x ) = 1 m ∑ i = 1 m v i σ ( u ~ i ⊤ x ~ ) + a , with parameters θ = ( v , u , b , a ) ∈ Θ , (31) f^{\mathrm{NN}}_{\theta}(x) = \frac{1}{\sqrt{m}}\sum_{i=1}^{m}v_i\sigma(\tilde{u}_i^{\top}\tilde{x}) + a,\quad\text{with parameters }\theta=(v,u,b,a)\in\Theta, \tag*{(31)} f θ NN ( x ) = m 1 i = 1 ∑ m v i σ ( u ~ i ⊤ x ~ ) + a , with parameters θ = ( v , u , b , a ) ∈ Θ , (31)
where the augmented input is x ~ = [ x ⊤ , 1 ] ⊤ ∈ R p + 1 \tilde{x}=[x^{\top},1]^{\top}\in\mathbb{R}^{p+1} x ~ = [ x ⊤ , 1 ] ⊤ ∈ R p + 1 and the augmented inner weights u ~ i ∈ R p + 1 \tilde{u}_i\in\mathbb{R}^{p+1} u ~ i ∈ R p + 1 incorporate a neuron-wise bias b i b_i b i via:
u ~ i = [ u i b i ] ∈ R p + 1 , \tilde{u}_i=\begin{bmatrix}u_i\\b_i\end{bmatrix}\in\mathbb{R}^{p+1}, u ~ i = [ u i b i ] ∈ R p + 1 ,
where u = [ u 1 , … , u m ] ⊤ ∈ R m × p u=[u_1,\ldots,u_m]^{\top}\in\mathbb{R}^{m\times p} u = [ u 1 , … , u m ] ⊤ ∈ R m × p are the inner layer weights, b = [ b 1 , … , b m ] ⊤ ∈ R m b=[b_1,\ldots,b_m]^{\top}\in\mathbb{R}^{m} b = [ b 1 , … , b m ] ⊤ ∈ R m are the inner layer biases, v = [ v 1 , … , v m ] ⊤ ∈ R m v=[v_1,\ldots,v_m]^{\top}\in\mathbb{R}^{m} v = [ v 1 , … , v m ] ⊤ ∈ R m are the outer layer weights, a ∈ R a\in\mathbb{R} a ∈ R is the outer bias, Θ = R m × R m × p × R m × R \Theta=\mathbb{R}^{m}\times\mathbb{R}^{m\times p}\times\mathbb{R}^{m}\times\mathbb{R} Θ = R m × R m × p × R m × R is the parameter space, and σ \sigma σ is the ReLU activation function, defined as σ ( x ) = max ( 0 , x ) \sigma(x)=\max(0,x) σ ( x ) = max ( 0 , x ) . The derivative of σ \sigma σ is σ ′ ( x ) = 1 \sigma'(x)=1 σ ′ ( x ) = 1 for x > 0 x>0 x > 0 and σ ′ ( x ) = 0 \sigma'(x)=0 σ ′ ( x ) = 0 for x < 0 x<0 x < 0 . Since σ \sigma σ is not differentiable at 0 0 0 , we adopt the convention σ ′ ( 0 ) = 0 \sigma'(0)=0 σ ′ ( 0 ) = 0 ; hence we write σ ′ ( x ) = 1 x > 0 \sigma'(x)=\mathbb{1}_{x>0} σ ′ ( x ) = 1 x > 0 .
Initialization. We use a mirrored initialization to ensure that the network output is exactly zero at initialization. Specifically, the m m m neurons are arranged in m / 2 m/2 m /2 pairs (assuming m m m is even). For each pair ( i , i + m / 2 ) (i,i+m/2) ( i , i + m /2 ) with 1 ≤ i ≤ m / 2 1\leq i\leq m/2 1 ≤ i ≤ m /2 , the inner weights and biases are initialized symmetrically:
u i = u i + m / 2 ∼ N ( 0 , 1 p × p ) , b i = b i + m / 2 ∼ N ( 0 , 1 ) , u_i=u_{i+m/2}\sim\mathcal{N}(0,\mathbb{1}_{p\times p}),\qquad b_i=b_{i+m/2}\sim\mathcal{N}(0,1), u i = u i + m /2 ∼ N ( 0 , 1 p × p ) , b i = b i + m /2 ∼ N ( 0 , 1 ) ,
and the outer weights are initialized with opposite signs:
v i = − v i + m / 2 ∼ N ( 0 , 1 ) . v_i=-v_{i+m/2}\sim\mathcal{N}(0,1). v i = − v i + m /2 ∼ N ( 0 , 1 ) .
The outer bias is initialized as a = 0 a=0 a = 0 . Moreover, v i v_i v i , u i u_i u i , and b i b_i b i are drawn independently from one another. Under this initialization, for each pair ( i , i + m / 2 ) (i,i+m/2) ( i , i + m /2 ) and any input x ∈ X x\in\mathcal{X} x ∈ X :
v i σ ( u ~ i ⊤ x ~ ) + v i + m / 2 σ ( u ~ i + m / 2 ⊤ x ~ ) = v i σ ( u ~ i ⊤ x ~ ) − v i σ ( u ~ i ⊤ x ~ ) = 0. v_i\sigma(\tilde{u}_i^{\top}\tilde{x})+v_{i+m/2}\sigma(\tilde{u}_{i+m/2}^{\top}\tilde{x})=v_i\sigma(\tilde{u}_i^{\top}\tilde{x})-v_i\sigma(\tilde{u}_i^{\top}\tilde{x})=0. v i σ ( u ~ i ⊤ x ~ ) + v i + m /2 σ ( u ~ i + m /2 ⊤ x ~ ) = v i σ ( u ~ i ⊤ x ~ ) − v i σ ( u ~ i ⊤ x ~ ) = 0.
Summing over all pairs, the network output at initialization satisfies f θ N N ( x ) = 0 f^{\mathrm{NN}}_{\theta}(x)=0 f θ NN ( x ) = 0 for all x ∈ X x\in\mathcal{X} x ∈ X .
Training dynamics. Given a training set of n n n i.i.d. samples { ( x i , y i ) } i = 1 n ⊂ X × Y \{(x_i,y_i)\}_{i=1}^{n}\subset\mathcal{X}\times\mathcal{Y} {( x i , y i ) } i = 1 n ⊂ X × Y drawn from the distribution ρ \rho ρ , we consider the following empirical loss:
L ^ ( f θ N N ) : = L ^ ( θ ) = 1 2 n ∑ i = 1 n ( f θ N N ( x i ) − y i ) 2 , \widehat{\mathcal{L}}(f^{\mathrm{NN}}_{\theta}):=\widehat{\mathcal{L}}(\theta)=\frac{1}{2n}\sum_{i=1}^{n}\left(f^{\mathrm{NN}}_{\theta}(x_i)-y_i\right)^2, L ( f θ NN ) := L ( θ ) = 2 n 1 i = 1 ∑ n ( f θ NN ( x i ) − y i ) 2 ,
and we train the network by gradient flow , where the parameters are updated according to the following ordinary differential equation:
d d t θ t = − ∂ θ L ^ ( θ t ) = − 1 n ∑ i = 1 n ∂ θ f θ t N N ( x i ) ( f θ t N N ( x i ) − y i ) , \frac{\mathrm{d}}{\mathrm{d}t}\theta_t=-\partial_{\theta}\widehat{\mathcal{L}}(\theta_t)=-\frac{1}{n}\sum_{i=1}^{n}\partial_{\theta}f^{\mathrm{NN}}_{\theta_t}(x_i)\left(f^{\mathrm{NN}}_{\theta_t}(x_i)-y_i\right), d t d θ t = − ∂ θ L ( θ t ) = − n 1 i = 1 ∑ n ∂ θ f θ t NN ( x i ) ( f θ t NN ( x i ) − y i ) ,
where θ t = ( v t , u t , b t , a t ) \theta_t=(v_t,u_t,b_t,a_t) θ t = ( v t , u t , b t , a t ) and ∂ θ f θ t N N ( x i ) ∈ R dim ( Θ ) \partial_{\theta}f^{\mathrm{NN}}_{\theta_t}(x_i)\in\mathbb{R}^{\dim(\Theta)} ∂ θ f θ t NN ( x i ) ∈ R d i m ( Θ ) denotes the Jacobian. Using the chain rule, for any x ∈ X x\in\mathcal{X} x ∈ X , the network’s output evolves as follows:
d d t f θ t N N ( x ) = ( ∂ θ f θ t N N ( x ) ) ⊤ d d t θ t = − 1 n ∑ i = 1 n ( ∂ θ f θ t N N ( x ) ) ⊤ ∂ θ f θ t N N ( x i ) ( f θ t N N ( x i ) − y i ) . \frac{\mathrm{d}}{\mathrm{d}t}f^{\mathrm{NN}}_{\theta_t}(x)=\left(\partial_{\theta}f^{\mathrm{NN}}_{\theta_t}(x)\right)^{\top}\frac{\mathrm{d}}{\mathrm{d}t}\theta_t=-\frac{1}{n}\sum_{i=1}^{n}\left(\partial_{\theta}f^{\mathrm{NN}}_{\theta_t}(x)\right)^{\top}\partial_{\theta}f^{\mathrm{NN}}_{\theta_t}(x_i)\left(f^{\mathrm{NN}}_{\theta_t}(x_i)-y_i\right). d t d f θ t NN ( x ) = ( ∂ θ f θ t NN ( x ) ) ⊤ d t d θ t = − n 1 i = 1 ∑ n ( ∂ θ f θ t NN ( x ) ) ⊤ ∂ θ f θ t NN ( x i ) ( f θ t NN ( x i ) − y i ) .
We define the finite-width Neural Tangent Kernel (NTK) K θ m K^{m}_{\theta} K θ m for any x , x ′ ∈ X x,x'\in\mathcal{X} x , x ′ ∈ X by:
K θ m ( x , x ′ ) = ( ∂ θ f θ N N ( x ) ) ⊤ ∂ θ f θ N N ( x ′ ) ∈ R . (32) K^{m}_{\theta}(x,x')=\left(\partial_{\theta}f^{\mathrm{NN}}_{\theta}(x)\right)^{\top}\partial_{\theta}f^{\mathrm{NN}}_{\theta}(x')\in\mathbb{R}. \tag*{(32)} K θ m ( x , x ′ ) = ( ∂ θ f θ NN ( x ) ) ⊤ ∂ θ f θ NN ( x ′ ) ∈ R . (32)
Using this, the output dynamics becomes:
d d t f θ t N N ( x ) = − 1 n ∑ i = 1 n K θ t m ( x , x i ) ( f θ t N N ( x i ) − y i ) . (33) \frac{\mathrm{d}}{\mathrm{d}t}f^{\mathrm{NN}}_{\theta_t}(x)=-\frac{1}{n}\sum_{i=1}^{n}K^{m}_{\theta_t}(x,x_i)\left(f^{\mathrm{NN}}_{\theta_t}(x_i)-y_i\right). \tag*{(33)} d t d f θ t NN ( x ) = − n 1 i = 1 ∑ n K θ t m ( x , x i ) ( f θ t NN ( x i ) − y i ) . (33)
For the neural network defined above in Equation (31) , K θ m K_\theta^m K θ m admits an explicit expression. Indeed, computing the gradients with respect to the parameters for any 1 ≤ i ≤ m 1 \leq i \leq m 1 ≤ i ≤ m , we get:
∂ v i f θ N N ( x ) = 1 m σ ( u ~ i ⊤ x ~ ) , ∂ u ~ i f θ N N ( x ) = 1 m v i σ ′ ( u ~ i ⊤ x ~ ) x ~ , ∂ a f θ N N ( x ) = 1. \partial_{v_i}f_\theta^{\mathrm{NN}}(x)=\frac{1}{\sqrt{m}}\sigma(\tilde{u}_i^{\top}\tilde{x}),\qquad\partial_{\tilde{u}_i}f_\theta^{\mathrm{NN}}(x)=\frac{1}{\sqrt{m}}v_i\sigma'(\tilde{u}_i^{\top}\tilde{x})\tilde{x},\qquad\partial_{a}f_\theta^{\mathrm{NN}}(x)=1. ∂ v i f θ NN ( x ) = m 1 σ ( u ~ i ⊤ x ~ ) , ∂ u ~ i f θ NN ( x ) = m 1 v i σ ′ ( u ~ i ⊤ x ~ ) x ~ , ∂ a f θ NN ( x ) = 1.
Using the definition of K θ m K_\theta^m K θ m in Equation (32) , we obtain:
K θ m ( x , x ′ ) = ∑ i = 1 m ⟨ ∂ v i f θ N N ( x ) , ∂ v i f θ N N ( x ′ ) ⟩ + ∑ i = 1 m ⟨ ∂ u ~ i f θ N N ( x ) , ∂ u ~ i f θ N N ( x ′ ) ⟩ + 1 = 1 m ∑ i = 1 m σ ( u ~ i ⊤ x ~ ) σ ( u ~ i ⊤ x ~ ′ ) + 1 m ∑ i = 1 m v i 2 σ ′ ( u ~ i ⊤ x ~ ) σ ′ ( u ~ i ⊤ x ~ ′ ) x ~ ⊤ x ~ ′ + 1. (34) \begin{aligned}
K_\theta^m(x,x')&=\sum_{i=1}^{m}\langle\partial_{v_i}f_\theta^{\mathrm{NN}}(x),\partial_{v_i}f_\theta^{\mathrm{NN}}(x')\rangle+\sum_{i=1}^{m}\langle\partial_{\tilde{u}_i}f_\theta^{\mathrm{NN}}(x),\partial_{\tilde{u}_i}f_\theta^{\mathrm{NN}}(x')\rangle+1\\
&=\frac{1}{m}\sum_{i=1}^{m}\sigma(\tilde{u}_i^{\top}\tilde{x})\sigma(\tilde{u}_i^{\top}\tilde{x}')+\frac{1}{m}\sum_{i=1}^{m}v_i^2\sigma'(\tilde{u}_i^{\top}\tilde{x})\sigma'(\tilde{u}_i^{\top}\tilde{x}')\tilde{x}^{\top}\tilde{x}'+1. \tag*{(34)}
\end{aligned} K θ m ( x , x ′ ) = i = 1 ∑ m ⟨ ∂ v i f θ NN ( x ) , ∂ v i f θ NN ( x ′ )⟩ + i = 1 ∑ m ⟨ ∂ u ~ i f θ NN ( x ) , ∂ u ~ i f θ NN ( x ′ )⟩ + 1 = m 1 i = 1 ∑ m σ ( u ~ i ⊤ x ~ ) σ ( u ~ i ⊤ x ~ ′ ) + m 1 i = 1 ∑ m v i 2 σ ′ ( u ~ i ⊤ x ~ ) σ ′ ( u ~ i ⊤ x ~ ′ ) x ~ ⊤ x ~ ′ + 1. (34)
Infinite-width limit. When the network’s width m m m goes to + ∞ +\infty + ∞ , K θ m K_\theta^m K θ m converges to a time-invariant deterministic kernel K N T K K^{\mathrm{NTK}} K NTK , referred to as the infinite-width NTK [4 , 53 , 59 ] . Let f t N T K f_t^{\mathrm{NTK}} f t NTK denote the function that describes the limiting training dynamics of f θ t N N f_{\theta_t}^{\mathrm{NN}} f θ t NN when m → + ∞ m\to+\infty m → + ∞ . Then, in this regime, the dynamics of f t N T K f_t^{\mathrm{NTK}} f t NTK are described by:
d d t f t N T K ( x ) = − 1 n ∑ i = 1 n K N T K ( x , x i ) ( f t N T K ( x i ) − y i ) . (35) \frac{\mathrm{d}}{\mathrm{d}t}f_t^{\mathrm{NTK}}(x)=-\frac{1}{n}\sum_{i=1}^{n}K^{\mathrm{NTK}}(x,x_i)\left(f_t^{\mathrm{NTK}}(x_i)-y_i\right). \tag*{(35)} d t d f t NTK ( x ) = − n 1 i = 1 ∑ n K NTK ( x , x i ) ( f t NTK ( x i ) − y i ) . (35)
Similarly to the neural network initialization, we also have that f θ 0 N T K = 0 f_{\theta_0}^{\mathrm{NTK}}=0 f θ 0 NTK = 0 due to the mirrored initialization. For the neural network defined above in Equation (31) , K N T K K^{\mathrm{NTK}} K NTK also admits an explicit expression. Indeed, as m → + ∞ m\to+\infty m → + ∞ , by the law of large numbers and the Gaussian initialization of the parameters, the kernel K θ m K_\theta^m K θ m , whose closed-form expression is provided in Equation (34) , converges to a deterministic kernel K N T K K^{\mathrm{NTK}} K NTK given for any x , x ′ ∈ X x,x'\in\mathcal{X} x , x ′ ∈ X by:
K N T K ( x , x ′ ) = E u ~ ∼ N ( 0 , 1 ( p + 1 ) × ( p + 1 ) ) [ σ ( u ~ ⊤ x ~ ) σ ( u ~ ⊤ x ~ ′ ) ] + E v ∼ N ( 0 , 1 ) , u ~ ∼ N ( 0 , 1 ( p + 1 ) × ( p + 1 ) ) [ v 2 σ ′ ( u ~ ⊤ x ~ ) σ ′ ( u ~ ⊤ x ~ ′ ) ] x ~ ⊤ x ~ ′ + 1 = E u ~ ∼ N ( 0 , 1 ( p + 1 ) × ( p + 1 ) ) [ σ ( u ~ ⊤ x ~ ) σ ( u ~ ⊤ x ~ ′ ) ] + E u ~ ∼ N ( 0 , 1 ( p + 1 ) × ( p + 1 ) ) [ σ ′ ( u ~ ⊤ x ~ ) σ ′ ( u ~ ⊤ x ~ ′ ) ] x ~ ⊤ x ~ ′ + 1 , (36) \begin{aligned}
K^{\mathrm{NTK}}(x,x')&=\mathbb{E}_{\tilde{u}\sim\mathcal{N}(0,1_{(p+1)\times(p+1)})}\left[\sigma(\tilde{u}^{\top}\tilde{x})\sigma(\tilde{u}^{\top}\tilde{x}')\right]\\
&\quad+\mathbb{E}_{v\sim\mathcal{N}(0,1),\tilde{u}\sim\mathcal{N}(0,1_{(p+1)\times(p+1)})}\left[v^2\sigma'(\tilde{u}^{\top}\tilde{x})\sigma'(\tilde{u}^{\top}\tilde{x}')\right]\tilde{x}^{\top}\tilde{x}'+1\\
&=\mathbb{E}_{\tilde{u}\sim\mathcal{N}(0,1_{(p+1)\times(p+1)})}\left[\sigma(\tilde{u}^{\top}\tilde{x})\sigma(\tilde{u}^{\top}\tilde{x}')\right]\\
&\quad+\mathbb{E}_{\tilde{u}\sim\mathcal{N}(0,1_{(p+1)\times(p+1)})}\left[\sigma'(\tilde{u}^{\top}\tilde{x})\sigma'(\tilde{u}^{\top}\tilde{x}')\right]\tilde{x}^{\top}\tilde{x}'+1, \tag*{(36)}
\end{aligned} K NTK ( x , x ′ ) = E u ~ ∼ N ( 0 , 1 ( p + 1 ) × ( p + 1 ) ) [ σ ( u ~ ⊤ x ~ ) σ ( u ~ ⊤ x ~ ′ ) ] + E v ∼ N ( 0 , 1 ) , u ~ ∼ N ( 0 , 1 ( p + 1 ) × ( p + 1 ) ) [ v 2 σ ′ ( u ~ ⊤ x ~ ) σ ′ ( u ~ ⊤ x ~ ′ ) ] x ~ ⊤ x ~ ′ + 1 = E u ~ ∼ N ( 0 , 1 ( p + 1 ) × ( p + 1 ) ) [ σ ( u ~ ⊤ x ~ ) σ ( u ~ ⊤ x ~ ′ ) ] + E u ~ ∼ N ( 0 , 1 ( p + 1 ) × ( p + 1 ) ) [ σ ′ ( u ~ ⊤ x ~ ) σ ′ ( u ~ ⊤ x ~ ′ ) ] x ~ ⊤ x ~ ′ + 1 , (36)
where in the second equality, we use the independence between v v v and u ~ \tilde{u} u ~ , as well as E v ∼ N ( 0 , 1 ) [ v 2 ] = 1 \mathbb{E}_{v\sim\mathcal{N}(0,1)}[v^2]=1 E v ∼ N ( 0 , 1 ) [ v 2 ] = 1 . The NTK matrix K N T K = [ K N T K ( x i , x j ) ] i , j = 1 n K^{\mathrm{NTK}}=[K^{\mathrm{NTK}}(x_i,x_j)]_{i,j=1}^{n} K NTK = [ K NTK ( x i , x j ) ] i , j = 1 n can thus be expressed as:
K N T K = E u ~ ∼ N ( 0 , 1 ( p + 1 ) × ( p + 1 ) ) [ σ ( X ~ u ~ ) σ ( u ~ ⊤ X ~ ⊤ ) ] + E u ~ ∼ N ( 0 , 1 ( p + 1 ) × ( p + 1 ) ) [ σ ′ ( X ~ u ~ ) σ ′ ( u ~ ⊤ X ~ ⊤ ) ] ⊙ X ~ X ~ ⊤ + 1 n 1 n ⊤ . (37) \begin{aligned}
K^{\mathrm{NTK}}&=\mathbb{E}_{\tilde{u}\sim\mathcal{N}(0,1_{(p+1)\times(p+1)})}\left[\sigma(\widetilde{X}\tilde{u})\sigma(\tilde{u}^{\top}\widetilde{X}^{\top})\right]\\
&\quad+\mathbb{E}_{\tilde{u}\sim\mathcal{N}(0,1_{(p+1)\times(p+1)})}\left[\sigma'(\widetilde{X}\tilde{u})\sigma'(\tilde{u}^{\top}\widetilde{X}^{\top})\right]\odot\widetilde{X}\widetilde{X}^{\top}+1_n1_n^{\top}. \tag*{(37)}
\end{aligned} K NTK = E u ~ ∼ N ( 0 , 1 ( p + 1 ) × ( p + 1 ) ) [ σ ( X u ~ ) σ ( u ~ ⊤ X ⊤ ) ] + E u ~ ∼ N ( 0 , 1 ( p + 1 ) × ( p + 1 ) ) [ σ ′ ( X u ~ ) σ ′ ( u ~ ⊤ X ⊤ ) ] ⊙ X X ⊤ + 1 n 1 n ⊤ . (37)
where X ~ = [ x ~ 1 , … , x ~ n ] ⊤ ∈ R n × ( p + 1 ) \widetilde{X}=[\tilde{x}_1,\ldots,\tilde{x}_n]^{\top}\in\mathbb{R}^{n\times(p+1)} X = [ x ~ 1 , … , x ~ n ] ⊤ ∈ R n × ( p + 1 ) is the design matrix of the dataset { x ~ i } i = 1 n \{\tilde{x}_i\}_{i=1}^{n} { x ~ i } i = 1 n and ⊙ \odot ⊙ is the Hadamard product. In Equation (37) , σ \sigma σ and σ ′ \sigma' σ ′ are applied entrywise to vectors or matrices. Moreover, we have the following explicit formula for the infinite-width NTK [27 ] :
K N T K ( x , x ′ ) = ∥ x ~ ∥ ∥ x ~ ′ ∥ 2 π ( sin φ + ( π − φ ) cos φ ) + π − φ 2 π x ~ ⊤ x ~ ′ + 1 , (38) K^{\mathrm{NTK}}(x,x')=\frac{\|\tilde{x}\|\|\tilde{x}'\|}{2\pi}\left(\sin\varphi+(\pi-\varphi)\cos\varphi\right)+\frac{\pi-\varphi}{2\pi}\tilde{x}^{\top}\tilde{x}'+1, \tag*{(38)} K NTK ( x , x ′ ) = 2 π ∥ x ~ ∥∥ x ~ ′ ∥ ( sin φ + ( π − φ ) cos φ ) + 2 π π − φ x ~ ⊤ x ~ ′ + 1 , (38)
where φ = arccos ( x ~ ⊤ x ~ ′ ∥ x ~ ∥ ∥ x ~ ′ ∥ ) \varphi=\arccos\left(\frac{\tilde{x}^{\top}\tilde{x}'}{\|\tilde{x}\|\|\tilde{x}'\|}\right) φ = arccos ( ∥ x ~ ∥∥ x ~ ′ ∥ x ~ ⊤ x ~ ′ ) is the angle between the augmented inputs x ~ = [ x ⊤ , 1 ] ⊤ \tilde{x}=[x^{\top},1]^{\top} x ~ = [ x ⊤ , 1 ] ⊤ and x ~ ′ = [ x ′ ⊤ , 1 ] ⊤ \tilde{x}'=[x'^{\top},1]^{\top} x ~ ′ = [ x ′ ⊤ , 1 ] ⊤ . In the following, for notational simplicity, we denote f θ t N N f_{\theta_t}^{\mathrm{NN}} f θ t NN , f θ t N T K f_{\theta_t}^{\mathrm{NTK}} f θ t NTK , and K θ t m K_{\theta_t}^m K θ t m by f t N N f_t^{\mathrm{NN}} f t NN , f t N T K f_t^{\mathrm{NTK}} f t NTK , and K t m K_t^m K t m , respectively. In the remainder of Appendix § 13 , we denote by H N T K \mathcal{H}^{\mathrm{NTK}} H NTK the RKHS associated with the infinite-width kernel K N T K K^{\mathrm{NTK}} K NTK , by [ H N T K ] s [\mathcal{H}^{\mathrm{NTK}}]_s [ H NTK ] s its interpolation space, and by ( λ i ) i ≥ 1 ⊂ ( 0 , + ∞ ) (\lambda_i)_{i\geq1}\subset(0,+\infty) ( λ i ) i ≥ 1 ⊂ ( 0 , + ∞ ) the non-increasing sequence of non-zero eigenvalues of the integral operator T T T (equivalently, of the covariance operator Σ \Sigma Σ ), with associated orthonormal family ( e i ) i ≥ 1 (e_i)_{i\geq1} ( e i ) i ≥ 1 in L 2 ( X ) L^2(\mathcal{X}) L 2 ( X ) . We refer the reader to Appendix § 16 for details on interpolation spaces and the spectral decomposition. We further denote by λ 0 \lambda_0 λ 0 the smallest eigenvalue of the infinite-width NTK matrix K N T K = [ K N T K ( x i , x j ) ] i , j = 1 n ∈ R n × n K^{\mathrm{NTK}}=[K^{\mathrm{NTK}}(x_i,x_j)]_{i,j=1}^{n}\in\mathbb{R}^{n\times n} K NTK = [ K NTK ( x i , x j ) ] i , j = 1 n ∈ R n × n , which controls the convergence of the NTK gradient flow.
Assumptions
We make the following assumptions.
Assumption a. (Compactness of X \mathcal{X} X ) The input space X ⊂ R p \mathcal{X} \subset\mathbb{R}^{p} X ⊂ R p is compact with a constant A > 0 A > 0 A > 0 such that ∥ x ∥ ≤ A \lVert x\rVert\le A ∥ x ∥ ≤ A for any x ∈ X x \in\mathcal{X} x ∈ X .
Assumption b. (Boundedness of Y \mathcal{Y} Y ) The target space Y ⊂ R \mathcal{Y} \subset\mathbb{R} Y ⊂ R is bounded, i.e. , there exists a constant B > 0 B > 0 B > 0 such that ∣ y ∣ ≤ B \lvert y\rvert\le B ∣ y ∣ ≤ B for any y ∈ Y y \in\mathcal{Y} y ∈ Y .
Assumption c. (Source condition for f ⋆ f^\star f ⋆ ) There exist s , R > 0 s, R > 0 s , R > 0 such that ∥ f ⋆ ∥ [ H N T K ] s ≤ R \lVert f^\star\rVert_{[\mathcal{H}^{\mathrm{NTK}}]^{s}} \le R ∥ f ⋆ ∥ [ H NTK ] s ≤ R .
Assumption d. (Bounded density) There exists a positive constant M μ > 0 M_{\mu} > 0 M μ > 0 such that f μ ≤ M μ f_{\mu} \le M_{\mu} f μ ≤ M μ . Moreover, there exist an open ball Ω μ ⊂ X \Omega_{\mu} \subset\mathcal{X} Ω μ ⊂ X and a constant c μ > 0 c_{\mu} > 0 c μ > 0 such that f μ ≥ c μ f_{\mu} \ge c_{\mu} f μ ≥ c μ on Ω μ \Omega_{\mu} Ω μ .
We now derive some consequences of these assumptions that will be useful later.
Lemma E.1 (Boundedness of f ⋆ f^\star f ⋆ ). Under Assumption b , f ⋆ ∈ L ∞ ( X ) f^\star\in L^{\infty}(\mathcal{X}) f ⋆ ∈ L ∞ ( X ) and ∥ f ⋆ ∥ ∞ ≤ B \lVert f^\star\rVert_{\infty} \le B ∥ f ⋆ ∥ ∞ ≤ B .
Proof. For any x ∈ X x \in\mathcal{X} x ∈ X , we have ∣ f ⋆ ( x ) ∣ = ∣ E y ∼ ρ [ y ∣ x ] ∣ ≤ B \lvert f^\star(x)\rvert= \lvert\mathbb{E}_{y\sim\rho}[y\mid x]\rvert\le B ∣ f ⋆ ( x )∣ = ∣ E y ∼ ρ [ y ∣ x ]∣ ≤ B by Jensen’s inequality and Assumption b. Taking the supremum over x ∈ X x \in\mathcal{X} x ∈ X allows us to conclude.
Lemma E.2 (Boundedness of K N T K K^{\mathrm{NTK}} K NTK ). Under Assumption a , K N T K K^{\mathrm{NTK}} K NTK is bounded by κ N T K : = ( p + 2 ) ( A 2 + 1 ) + 1 \kappa^{\mathrm{NTK}} := (p+2)(A^{2}+1)+1 κ NTK := ( p + 2 ) ( A 2 + 1 ) + 1 on X × X \mathcal{X}\times\mathcal{X} X × X .
Proof. We start by recalling the expression of K N T K K^{\mathrm{NTK}} K NTK given in Equation (36) for any x , x ′ ∈ X x,x' \in\mathcal{X} x , x ′ ∈ X by:
K N T K ( x , x ′ ) = E u ~ ∼ N ( 0 , 1 ( p + 1 ) × ( p + 1 ) ) [ σ ( u ~ T x ~ ) σ ( u ~ T x ~ ′ ) ] + E u ~ ∼ N ( 0 , 1 ( p + 1 ) × ( p + 1 ) ) [ σ ′ ( u ~ T x ~ ) σ ′ ( u ~ T x ~ ′ ) ] x ~ T x ~ ′ + 1 , K^{\mathrm{NTK}}(x,x') =
\mathbb{E}_{\widetilde{u}\sim\mathcal{N}(0,1_{(p+1)\times(p+1)})}\left[\sigma(\widetilde{u}^{\mathsf{T}}\widetilde{x})\sigma(\widetilde{u}^{\mathsf{T}}\widetilde{x}')\right]
+\mathbb{E}_{\widetilde{u}\sim\mathcal{N}(0,1_{(p+1)\times(p+1)})}\left[\sigma'(\widetilde{u}^{\mathsf{T}}\widetilde{x})\sigma'(\widetilde{u}^{\mathsf{T}}\widetilde{x}')\right]\widetilde{x}^{\mathsf{T}}\widetilde{x}'+1, K NTK ( x , x ′ ) = E u ∼ N ( 0 , 1 ( p + 1 ) × ( p + 1 ) ) [ σ ( u T x ) σ ( u T x ′ ) ] + E u ∼ N ( 0 , 1 ( p + 1 ) × ( p + 1 ) ) [ σ ′ ( u T x ) σ ′ ( u T x ′ ) ] x T x ′ + 1 , By the triangle inequality, we have:
∣ K N T K ( x , x ′ ) ∣ ≤ ∣ E u ~ ∼ N ( 0 , 1 ( p + 1 ) × ( p + 1 ) ) [ σ ( u ~ T x ~ ) σ ( u ~ T x ~ ′ ) ] ∣ + ∣ E u ~ ∼ N ( 0 , 1 ( p + 1 ) × ( p + 1 ) ) [ σ ′ ( u ~ T x ~ ) σ ′ ( u ~ T x ~ ′ ) ] x ~ T x ~ ′ ∣ + 1. (39) \begin{aligned}
\left|K^{\mathrm{NTK}}(x,x')\right|
&\le\left|\mathbb{E}_{\widetilde{u}\sim\mathcal{N}(0,1_{(p+1)\times(p+1)})}\left[\sigma(\widetilde{u}^{\mathsf{T}}\widetilde{x})\sigma(\widetilde{u}^{\mathsf{T}}\widetilde{x}')\right]\right| \\
&\quad+\left|\mathbb{E}_{\widetilde{u}\sim\mathcal{N}(0,1_{(p+1)\times(p+1)})}\left[\sigma'(\widetilde{u}^{\mathsf{T}}\widetilde{x})\sigma'(\widetilde{u}^{\mathsf{T}}\widetilde{x}')\right]\widetilde{x}^{\mathsf{T}}\widetilde{x}'\right|+1. \tag*{(39)}
\end{aligned} K NTK ( x , x ′ ) ≤ E u ∼ N ( 0 , 1 ( p + 1 ) × ( p + 1 ) ) [ σ ( u T x ) σ ( u T x ′ ) ] + E u ∼ N ( 0 , 1 ( p + 1 ) × ( p + 1 ) ) [ σ ′ ( u T x ) σ ′ ( u T x ′ ) ] x T x ′ + 1. (39) Using the triangle inequality, the identities ∣ σ ( z ) ∣ ≤ ∣ z ∣ \lvert\sigma(z)\rvert\le\lvert z\rvert ∣ σ ( z )∣ ≤ ∣ z ∣ and σ ′ ( z ) ≤ 1 \sigma'(z)\le1 σ ′ ( z ) ≤ 1 , Cauchy-Schwarz inequality, and Assumption a, we get:
∣ E u ~ ∼ N ( 0 , 1 ( p + 1 ) × ( p + 1 ) ) [ σ ( u ~ T x ~ ) σ ( u ~ T x ~ ′ ) ] ∣ ≤ E u ~ ∼ N ( 0 , 1 ( p + 1 ) × ( p + 1 ) ) [ ∣ u ~ T x ~ ∣ ∣ u ~ T x ~ ′ ∣ ] ≤ E u ~ ∼ N ( 0 , 1 ( p + 1 ) × ( p + 1 ) ) [ ∥ u ~ ∥ 2 ] ∥ x ~ ∥ ∥ x ~ ′ ∥ ≤ ( p + 1 ) ( A 2 + 1 ) , \begin{aligned}
\left|\mathbb{E}_{\widetilde{u}\sim\mathcal{N}(0,1_{(p+1)\times(p+1)})}\left[\sigma(\widetilde{u}^{\mathsf{T}}\widetilde{x})\sigma(\widetilde{u}^{\mathsf{T}}\widetilde{x}')\right]\right|
&\le\mathbb{E}_{\widetilde{u}\sim\mathcal{N}(0,1_{(p+1)\times(p+1)})}\left[\left|\widetilde{u}^{\mathsf{T}}\widetilde{x}\right|\left|\widetilde{u}^{\mathsf{T}}\widetilde{x}'\right|\right] \\
&\le\mathbb{E}_{\widetilde{u}\sim\mathcal{N}(0,1_{(p+1)\times(p+1)})}\left[\lVert\widetilde{u}\rVert^{2}\right]\lVert\widetilde{x}\rVert\lVert\widetilde{x}'\rVert\\
&\le(p+1)(A^{2}+1),
\end{aligned} E u ∼ N ( 0 , 1 ( p + 1 ) × ( p + 1 ) ) [ σ ( u T x ) σ ( u T x ′ ) ] ≤ E u ∼ N ( 0 , 1 ( p + 1 ) × ( p + 1 ) ) [ u T x u T x ′ ] ≤ E u ∼ N ( 0 , 1 ( p + 1 ) × ( p + 1 ) ) [ ∥ u ∥ 2 ] ∥ x ∥ ∥ x ′ ∥ ≤ ( p + 1 ) ( A 2 + 1 ) , and
∣ E u ~ ∼ N ( 0 , 1 ( p + 1 ) × ( p + 1 ) ) [ σ ′ ( u ~ T x ~ ) σ ′ ( u ~ T x ~ ′ ) ] x ~ T x ~ ′ ∣ ≤ ∥ x ~ ∥ ∥ x ~ ′ ∥ ≤ A 2 + 1. \left|\mathbb{E}_{\widetilde{u}\sim\mathcal{N}(0,1_{(p+1)\times(p+1)})}\left[\sigma'(\widetilde{u}^{\mathsf{T}}\widetilde{x})\sigma'(\widetilde{u}^{\mathsf{T}}\widetilde{x}')\right]\widetilde{x}^{\mathsf{T}}\widetilde{x}'\right|
\le\lVert\widetilde{x}\rVert\lVert\widetilde{x}'\rVert\le A^{2}+1. E u ∼ N ( 0 , 1 ( p + 1 ) × ( p + 1 ) ) [ σ ′ ( u T x ) σ ′ ( u T x ′ ) ] x T x ′ ≤ ∥ x ∥ ∥ x ′ ∥ ≤ A 2 + 1. Substituting both inequalities into Equation (39) concludes the proof.
Remark E.3. As pointed out in Appendix § 16 , Lemma E.2 implies that the RKHS H N T K \mathcal{H}^{\mathrm{NTK}} H NTK is included in L 2 ( X ) L^{2}(\mathcal{X}) L 2 ( X ) . This allows us to consider the closure of H N T K \mathcal{H}^{\mathrm{NTK}} H NTK in L 2 ( X ) L^{2}(\mathcal{X}) L 2 ( X ) , as we do in Lemma E.4.
Lemma E.4. Under Assumptions a and c , f ⋆ f^\star f ⋆ lies in the closure of H N T K \mathcal{H}^{\mathrm{NTK}} H NTK in L 2 ( X ) L^{2}(\mathcal{X}) L 2 ( X ) .
Proof. It suffices to show that there exists a sequence in H N T K \mathcal{H}^{\mathrm{NTK}} H NTK that converges to f ⋆ f^\star f ⋆ in L 2 ( X ) L^{2}(\mathcal{X}) L 2 ( X ) . Under Assumption c, we know that f ⋆ ∈ [ H N T K ] s f^\star\in[\mathcal{H}^{\mathrm{NTK}}]^{s} f ⋆ ∈ [ H NTK ] s for some s > 0 s > 0 s > 0 . Therefore, f ⋆ f^\star f ⋆ can be written as (see Appendix § 16 ):
f ⋆ = ∑ i ≥ 1 a i λ i s 2 e i , with ∑ i ≥ 1 a i 2 < + ∞ . f^\star= \sum_{i\ge1} a_i\lambda_i^{\frac{s}{2}}e_i,\qquad\text{with}\quad\sum_{i\ge1}a_i^{2}<+\infty. f ⋆ = i ≥ 1 ∑ a i λ i 2 s e i , with i ≥ 1 ∑ a i 2 < + ∞. Let ( h n ) n ≥ 1 (h_n)_{n\ge1} ( h n ) n ≥ 1 be a sequence in H N T K \mathcal{H}^{\mathrm{NTK}} H NTK defined for any n ≥ 1 n\ge1 n ≥ 1 by:
h n = ∑ i = 1 n a i λ i s 2 e i = ∑ i = 1 n ( a i λ i s − 1 2 ) λ i 1 2 e i . h_n=\sum_{i=1}^{n}a_i\lambda_i^{\frac{s}{2}}e_i
=\sum_{i=1}^{n}\left(a_i\lambda_i^{\frac{s-1}{2}}\right)\lambda_i^{\frac{1}{2}}e_i. h n = i = 1 ∑ n a i λ i 2 s e i = i = 1 ∑ n ( a i λ i 2 s − 1 ) λ i 2 1 e i .
Indeed, h n ∈ H N T K h_n \in\mathcal{H}^{\mathrm{NTK}} h n ∈ H NTK for any n ≥ 1 n \ge1 n ≥ 1 since h n h_n h n is a finite linear combination of the basis functions λ i 1 2 e i \lambda_i^{\frac{1}{2}}e_i λ i 2 1 e i . Moreover, we have:
∥ h n − f ⋆ ∥ L 2 ( X ) 2 = ∥ ∑ i > n a i λ i s 2 e i ∥ L 2 ( X ) 2 = ∑ i > n a i 2 λ i s ≤ λ 1 s ∑ i > n a i 2 . \left\|h_n-f^\star\right\|_{L^2(\mathcal{X})}^2=\left\|\sum_{i>n}a_i\lambda_i^{\frac{s}{2}}e_i\right\|_{L^2(\mathcal{X})}^2=\sum_{i>n}a_i^2\lambda_i^s\leq\lambda_1^s\sum_{i>n}a_i^2. ∥ h n − f ⋆ ∥ L 2 ( X ) 2 = i > n ∑ a i λ i 2 s e i L 2 ( X ) 2 = i > n ∑ a i 2 λ i s ≤ λ 1 s i > n ∑ a i 2 .
where the second equality follows from Parseval’s identity, and the inequality uses the fact that the eigenvalues ( λ i ) i ≥ 1 (\lambda_i)_{i\geq1} ( λ i ) i ≥ 1 are sorted in non-increasing order. Given that the series ( a i 2 ) i ≥ 1 (a_i^2)_{i\geq1} ( a i 2 ) i ≥ 1 is convergent, we know that its tail ∑ i > n a i 2 ⟶ 0 \sum_{i>n}a_i^2\longrightarrow0 ∑ i > n a i 2 ⟶ 0 as n → ∞ n\to\infty n → ∞ . Hence, we obtain:
∥ h n − f ⋆ ∥ L 2 ( X ) 2 → n → ∞ 0 , \left\|h_n-f^\star\right\|_{L^2(\mathcal{X})}^2\xrightarrow[n\to\infty]{}0, ∥ h n − f ⋆ ∥ L 2 ( X ) 2 n → ∞ 0 ,
which shows that the sequence ( h n ) n ≥ 1 (h_n)_{n\geq1} ( h n ) n ≥ 1 converges to f ⋆ f^\star f ⋆ in L 2 ( X ) L^2(\mathcal{X}) L 2 ( X ) . This concludes the proof. □
Lower-bound on the smallest eigenvalue of K N T K K^{\mathrm{NTK}} K NTK
Recall that we are given a dataset x 1 , … , x n ∈ X ⊂ R p x_1,\ldots,x_n\in\mathcal{X}\subset\mathbb{R}^p x 1 , … , x n ∈ X ⊂ R p sampled from a distribution μ \mu μ and an augmented dataset x ~ 1 , … , x ~ n ∈ R p + 1 \widetilde{x}_1,\ldots,\widetilde{x}_n\in\mathbb{R}^{p+1} x 1 , … , x n ∈ R p + 1 where x ~ i = [ x i ⊤ , 1 ] ⊤ \widetilde{x}_i=[x_i^\top,1]^\top x i = [ x i ⊤ , 1 ] ⊤ for 1 ≤ i ≤ n 1\leq i\leq n 1 ≤ i ≤ n , with design matrices X = [ x 1 , … , x n ] ⊤ ∈ R n × p X=[x_1,\ldots,x_n]^\top\in\mathbb{R}^{n\times p} X = [ x 1 , … , x n ] ⊤ ∈ R n × p and X ~ = [ x ~ 1 , … , x ~ n ] ⊤ ∈ R n × ( p + 1 ) \widetilde{X}=[\widetilde{x}_1,\ldots,\widetilde{x}_n]^\top\in\mathbb{R}^{n\times(p+1)} X = [ x 1 , … , x n ] ⊤ ∈ R n × ( p + 1 ) , respectively. Recall also that σ \sigma σ is the ReLU activation function, and σ ′ \sigma' σ ′ is its derivative. For ψ ∈ { p + 1 σ , σ ′ } \psi\in\{\sqrt{p+1}\sigma,\sigma'\} ψ ∈ { p + 1 σ , σ ′ } , let K ψ ∞ ∈ R n × n K_\psi^\infty\in\mathbb{R}^{n\times n} K ψ ∞ ∈ R n × n be the matrix defined as:
K ψ ∞ = E w ∼ U ( S p ) [ ψ ( X ^ w ) ψ ( w ⊤ X ^ ⊤ ) ] , K_\psi^\infty=\mathbb{E}_{w\sim\mathcal{U}(\mathbb{S}^p)}\left[\psi(\widehat{X}w)\psi(w^\top\widehat{X}^\top)\right], K ψ ∞ = E w ∼ U ( S p ) [ ψ ( X w ) ψ ( w ⊤ X ⊤ ) ] ,
where U ( S p ) \mathcal{U}(\mathbb{S}^p) U ( S p ) denotes the uniform distribution on the unit sphere S p \mathbb{S}^p S p , ψ \psi ψ is applied entrywise to the vector X ^ w ∈ R n \widehat{X}w\in\mathbb{R}^n X w ∈ R n and its transpose, with X ^ = [ x ^ 1 , … , x ^ n ] ⊤ ∈ R n × ( p + 1 ) \widehat{X}=[\widehat{x}_1,\ldots,\widehat{x}_n]^\top\in\mathbb{R}^{n\times(p+1)} X = [ x 1 , … , x n ] ⊤ ∈ R n × ( p + 1 ) being the design matrix of the normalized counterparts of x ~ 1 , … , x ~ n \widetilde{x}_1,\ldots,\widetilde{x}_n x 1 , … , x n , defined by:
x ^ i = x ~ i ∥ x ~ i ∥ = 1 ∥ x i ∥ 2 + 1 [ x i 1 ] ∈ S p for 1 ≤ i ≤ n . (40) \widehat{x}_i=\frac{\widetilde{x}_i}{\|\widetilde{x}_i\|}=\frac{1}{\sqrt{\|x_i\|^2+1}}\begin{bmatrix}x_i\\1\end{bmatrix}\in\mathbb{S}^p\quad\text{for }1\leq i\leq n. \tag*{(40)} x i = ∥ x i ∥ x i = ∥ x i ∥ 2 + 1 1 [ x i 1 ] ∈ S p for 1 ≤ i ≤ n . (40)
Definition E.5 (ϵ \epsilon ϵ -separated dataset [55 ] ). A dataset { x i } i = 1 n \{x_i\}_{i=1}^n { x i } i = 1 n is ϵ \epsilon ϵ -separated for ϵ > 0 \epsilon>0 ϵ > 0 if:
min i ≠ k min ( ∥ x i − x k ∥ , ∥ x i + x k ∥ ) ≥ ϵ . \min_{i\neq k}\min\left(\|x_i-x_k\|,\|x_i+x_k\|\right)\geq\epsilon. i = k min min ( ∥ x i − x k ∥ , ∥ x i + x k ∥ ) ≥ ϵ .
Lemma E.6 (Lemma 7 in [55 ] ). Let X ^ ∈ R n × ( p + 1 ) \widehat{X}\in\mathbb{R}^{n\times(p+1)} X ∈ R n × ( p + 1 ) be the design matrix of the dataset x ^ 1 , … , x ^ n ∈ S p \widehat{x}_1,\ldots,\widehat{x}_n\in\mathbb{S}^p x 1 , … , x n ∈ S p with p ≥ 2 p\geq2 p ≥ 2 , and ψ ∈ { p + 1 σ , σ ′ } \psi\in\{\sqrt{p+1}\sigma,\sigma'\} ψ ∈ { p + 1 σ , σ ′ } . Assume 0 < ϵ ≤ 1 0<\epsilon\leq1 0 < ϵ ≤ 1 . Suppose that this dataset is ϵ \epsilon ϵ -separated. Then, for all w ∈ R n w\in\mathbb{R}^n w ∈ R n with ∥ w ∥ = 1 \|w\|=1 ∥ w ∥ = 1 , we have:
⟨ K ψ ∞ w , w ⟩ ≥ { C ( 1 + g ( ϵ ) ) − 3 ϵ 4 if ψ = σ ′ , C ( 1 + g ( ϵ ) ) − 3 ϵ 4 if ψ = p + 1 σ , \left\langle K_\psi^\infty w,w\right\rangle\geq
\begin{cases}
C(1+g(\epsilon))^{-3}\epsilon^4 & \text{if }\psi=\sigma',\\
C(1+g(\epsilon))^{-3}\epsilon^4 & \text{if }\psi=\sqrt{p+1}\sigma,
\end{cases} ⟨ K ψ ∞ w , w ⟩ ≥ { C ( 1 + g ( ϵ ) ) − 3 ϵ 4 C ( 1 + g ( ϵ ) ) − 3 ϵ 4 if ψ = σ ′ , if ψ = p + 1 σ , where C > 0 C>0 C > 0 is an absolute positive constant and g ( ϵ ) = ( p + 1 ) ln ( 1 / ϵ ) ln ( p + 1 ) g(\epsilon)=\frac{(p+1)\ln(1/\epsilon)}{\ln(p+1)} g ( ϵ ) = l n ( p + 1 ) ( p + 1 ) l n ( 1/ ϵ ) .
Lemma E.7 (Deterministic lower-bound on the smallest eigenvalue of K N T K K^{\mathrm{NTK}} K NTK ). Let X ~ ∈ R n × ( p + 1 ) \widetilde{X}\in\mathbb{R}^{n\times(p+1)} X ∈ R n × ( p + 1 ) be the design matrix of the augmented dataset x ~ i = [ x i ⊤ , 1 ] ⊤ \widetilde{x}_i=[x_i^\top,1]^\top x i = [ x i ⊤ , 1 ] ⊤ and X ^ ∈ R n × ( p + 1 ) \widehat{X}\in\mathbb{R}^{n\times(p+1)} X ∈ R n × ( p + 1 ) be the design matrix of the normalized dataset x ^ i = x ~ i / ∥ x ~ i ∥ \widehat{x}_i=\widetilde{x}_i/\|\widetilde{x}_i\| x i = x i /∥ x i ∥ , with p ≥ 2 p\geq2 p ≥ 2 . Assume 0 < ϵ ≤ 1 0<\epsilon\leq1 0 < ϵ ≤ 1 . Suppose that the normalized dataset x ^ 1 , … , x ^ n ∈ S p \widehat{x}_1,\ldots,\widehat{x}_n\in\mathbb{S}^p x 1 , … , x n ∈ S p is ϵ \epsilon ϵ -separated. Then, we have the following lower-bound on the smallest eigenvalue λ 0 \lambda_0 λ 0 of K N T K K^{\mathrm{NTK}} K NTK :
λ 0 ≥ C ( 1 + g ( ϵ ) ) − 3 ϵ 4 , \lambda_0\geq C(1+g(\epsilon))^{-3}\epsilon^4, λ 0 ≥ C ( 1 + g ( ϵ ) ) − 3 ϵ 4 , where C > 0 C>0 C > 0 is an absolute positive constant and g ( ϵ ) = ( p + 1 ) ln ( 1 / ϵ ) ln ( p + 1 ) g(\epsilon)=\frac{(p+1)\ln(1/\epsilon)}{\ln(p+1)} g ( ϵ ) = l n ( p + 1 ) ( p + 1 ) l n ( 1/ ϵ ) .
Proof. By Equation (37) , the infinite-width NTK can be written as:
K N T K = E u ~ ∼ N ( 0 , I ( p + 1 ) × ( p + 1 ) ) [ σ ( X ~ u ~ ) σ ( u ~ ⊤ X ~ ⊤ ) ] + E u ~ ∼ N ( 0 , I ( p + 1 ) × ( p + 1 ) ) [ σ ′ ( X ~ u ~ ) σ ′ ( u ~ ⊤ X ~ ⊤ ) ] ⊙ X ~ X ~ ⊤ + 1 n 1 n ⊤ . K^{\mathrm{NTK}}=\mathbb{E}_{\widetilde{u}\sim\mathcal{N}(0,I_{(p+1)\times(p+1)})}\left[\sigma(\widetilde{X}\widetilde{u})\sigma(\widetilde{u}^{\top}\widetilde{X}^{\top})\right]
+\mathbb{E}_{\widetilde{u}\sim\mathcal{N}(0,I_{(p+1)\times(p+1)})}\left[\sigma'(\widetilde{X}\widetilde{u})\sigma'(\widetilde{u}^{\top}\widetilde{X}^{\top})\right]\odot\widetilde{X}\widetilde{X}^{\top}+\mathbf{1}_n\mathbf{1}_n^\top. K NTK = E u ∼ N ( 0 , I ( p + 1 ) × ( p + 1 ) ) [ σ ( X u ) σ ( u ⊤ X ⊤ ) ] + E u ∼ N ( 0 , I ( p + 1 ) × ( p + 1 ) ) [ σ ′ ( X u ) σ ′ ( u ⊤ X ⊤ ) ] ⊙ X X ⊤ + 1 n 1 n ⊤ .
Since u ~ ∼ N ( 0 , I ( p + 1 ) × ( p + 1 ) ) \tilde{u} \sim\mathcal{N}(0, \mathbb{I}_{(p+1)\times(p+1)}) u ~ ∼ N ( 0 , I ( p + 1 ) × ( p + 1 ) ) , using the Gaussian polar decomposition, it holds that u ~ = r w \tilde{u}=rw u ~ = r w , where r = ∥ u ~ ∥ r=\|\tilde{u}\| r = ∥ u ~ ∥ , w ∼ U ( S p ) w\sim\mathcal{U}(S^p) w ∼ U ( S p ) , and r r r and w w w are independent. We note also that r 2 = ∥ u ~ ∥ 2 = u ~ 1 2 + ⋯ + u ~ p + 1 2 r^2=\|\tilde{u}\|^2=\tilde{u}_1^2+\cdots+\tilde{u}_{p+1}^2 r 2 = ∥ u ~ ∥ 2 = u ~ 1 2 + ⋯ + u ~ p + 1 2 , where u ~ 1 , … , u ~ p + 1 \tilde{u}_1,\ldots,\tilde{u}_{p+1} u ~ 1 , … , u ~ p + 1 are independent and sampled from the standard Gaussian distribution N ( 0 , 1 ) \mathcal{N}(0,1) N ( 0 , 1 ) . This implies that r 2 ∼ χ p + 1 2 r^2\sim\chi^2_{p+1} r 2 ∼ χ p + 1 2 (the chi-squared distribution with p + 1 p+1 p + 1 degrees of freedom) with E [ r 2 ] = p + 1 \mathbb{E}[r^2]=p+1 E [ r 2 ] = p + 1 . Writing x ~ i = ∥ x ~ i ∥ x ^ i \tilde{x}_i=\|\tilde{x}_i\|\hat{x}_i x ~ i = ∥ x ~ i ∥ x ^ i and letting D = diag ( ∥ x ~ 1 ∥ , … , ∥ x ~ n ∥ ) D=\operatorname{diag}(\|\tilde{x}_1\|,\ldots,\|\tilde{x}_n\|) D = diag ( ∥ x ~ 1 ∥ , … , ∥ x ~ n ∥ ) , so that X ~ w = D X ^ w \tilde{X}w=D\hat{X}w X ~ w = D X ^ w , and using the facts that σ \sigma σ is homogeneous of degree 1 and σ ′ \sigma' σ ′ is scale-invariant together with the independence of r r r and w w w , we obtain:
K N T K = E r , w ∼ U ( S p ) [ σ ( X ~ r w ) σ ( r w T X ~ T ) ] + E r , w ∼ U ( S p ) [ σ ′ ( X ~ r w ) σ ′ ( r w T X ~ T ) ] ⊙ X ~ X ~ T + 1 n 1 n T = E r 2 ∼ χ p + 1 2 [ r 2 ] E w ∼ U ( S p ) [ σ ( X ~ w ) σ ( w T X ~ T ) ] + E w ∼ U ( S p ) [ σ ′ ( X ~ w ) σ ′ ( w T X ~ T ) ] ⊙ X ~ X ~ T + 1 n 1 n T = D K σ p + 1 ∞ D + K σ ′ ∞ ⊙ X ~ X ~ T + 1 n 1 n T , \begin{aligned}
K^{\mathrm{NTK}}
=\mathbb{E}_{r,w\sim\mathcal{U}(S^p)}\left[\sigma(\tilde{X}rw)\sigma(rw^{\mathsf{T}}\tilde{X}^{\mathsf{T}})\right]+\mathbb{E}_{r,w\sim\mathcal{U}(S^p)}\left[\sigma'(\tilde{X}rw)\sigma'(rw^{\mathsf{T}}\tilde{X}^{\mathsf{T}})\right]\odot\tilde{X}\tilde{X}^{\mathsf{T}}+\mathbb{1}_n\mathbb{1}_n^{\mathsf{T}} \\
&=\mathbb{E}_{r^2\sim\chi^2_{p+1}}[r^2]\mathbb{E}_{w\sim\mathcal{U}(S^p)}\left[\sigma(\tilde{X}w)\sigma(w^{\mathsf{T}}\tilde{X}^{\mathsf{T}})\right]+\mathbb{E}_{w\sim\mathcal{U}(S^p)}\left[\sigma'(\tilde{X}w)\sigma'(w^{\mathsf{T}}\tilde{X}^{\mathsf{T}})\right]\odot\tilde{X}\tilde{X}^{\mathsf{T}}+\mathbb{1}_n\mathbb{1}_n^{\mathsf{T}} \\
&=DK^\infty_{\frac{\sigma}{\sqrt{p+1}}}D+K^\infty_{\sigma'}\odot\tilde{X}\tilde{X}^{\mathsf{T}}+\mathbb{1}_n\mathbb{1}_n^{\mathsf{T}},
\end{aligned} K NTK = E r , w ∼ U ( S p ) [ σ ( X ~ r w ) σ ( r w T X ~ T ) ] + E r , w ∼ U ( S p ) [ σ ′ ( X ~ r w ) σ ′ ( r w T X ~ T ) ] ⊙ X ~ X ~ T + 1 n 1 n T = E r 2 ∼ χ p + 1 2 [ r 2 ] E w ∼ U ( S p ) [ σ ( X ~ w ) σ ( w T X ~ T ) ] + E w ∼ U ( S p ) [ σ ′ ( X ~ w ) σ ′ ( w T X ~ T ) ] ⊙ X ~ X ~ T + 1 n 1 n T = D K p + 1 σ ∞ D + K σ ′ ∞ ⊙ X ~ X ~ T + 1 n 1 n T ,
where the last line uses σ ( X ~ w ) = D σ ( X ^ w ) \sigma(\tilde{X}w)=D\sigma(\hat{X}w) σ ( X ~ w ) = D σ ( X ^ w ) and σ ′ ( X ~ w ) = σ ′ ( X ^ w ) \sigma'(\tilde{X}w)=\sigma'(\hat{X}w) σ ′ ( X ~ w ) = σ ′ ( X ^ w ) . The matrices D K σ p + 1 ∞ D DK^\infty_{\frac{\sigma}{\sqrt{p+1}}}D D K p + 1 σ ∞ D , K σ ′ ∞ ⊙ X ~ X ~ T K^\infty_{\sigma'}\odot\tilde{X}\tilde{X}^{\mathsf{T}} K σ ′ ∞ ⊙ X ~ X ~ T , and 1 n 1 n T \mathbb{1}_n\mathbb{1}_n^{\mathsf{T}} 1 n 1 n T are all positive semi-definite (the second by the Schur product theorem). Hence, by Weyl’s inequality, we obtain:
λ 0 ≥ λ min ( D K σ p + 1 ∞ D ) + λ min ( K σ ′ ∞ ⊙ X ~ X ~ T ) + λ min ( 1 n 1 n T ) ≥ λ min ( D K σ p + 1 ∞ D ) , (41) \lambda_0\geq\lambda_{\min}\left(DK^\infty_{\frac{\sigma}{\sqrt{p+1}}}D\right)+\lambda_{\min}\left(K^\infty_{\sigma'}\odot\tilde{X}\tilde{X}^{\mathsf{T}}\right)+\lambda_{\min}\left(\mathbb{1}_n\mathbb{1}_n^{\mathsf{T}}\right)\geq\lambda_{\min}\left(DK^\infty_{\frac{\sigma}{\sqrt{p+1}}}D\right), \tag*{(41)} λ 0 ≥ λ m i n ( D K p + 1 σ ∞ D ) + λ m i n ( K σ ′ ∞ ⊙ X ~ X ~ T ) + λ m i n ( 1 n 1 n T ) ≥ λ m i n ( D K p + 1 σ ∞ D ) , (41)
where we use the fact that λ min ( K σ ′ ∞ ⊙ X ~ X ~ T ) ≥ 0 \lambda_{\min}\left(K^\infty_{\sigma'}\odot\tilde{X}\tilde{X}^{\mathsf{T}}\right)\geq0 λ m i n ( K σ ′ ∞ ⊙ X ~ X ~ T ) ≥ 0 and λ min ( 1 n 1 n T ) ≥ 0 \lambda_{\min}\left(\mathbb{1}_n\mathbb{1}_n^{\mathsf{T}}\right)\geq0 λ m i n ( 1 n 1 n T ) ≥ 0 given that K σ ′ ∞ ⊙ X ~ X ~ T K^\infty_{\sigma'}\odot\tilde{X}\tilde{X}^{\mathsf{T}} K σ ′ ∞ ⊙ X ~ X ~ T and 1 n 1 n T \mathbb{1}_n\mathbb{1}_n^{\mathsf{T}} 1 n 1 n T are both positive semi-definite. By definition, for any unit vector w ∈ R n w\in\mathbb{R}^n w ∈ R n , we have:
λ min ( D K σ p + 1 ∞ D ) = inf ∥ w ∥ = 1 ⟨ D K σ p + 1 ∞ D w , w ⟩ = inf ∥ w ∥ = 1 ∥ D w ∥ 2 ⟨ K σ p + 1 ∞ D w ∥ D w ∥ , D w ∥ D w ∥ ⟩ ≥ λ min ( K σ p + 1 ∞ ) ⋅ λ min ( D 2 ) , \begin{aligned}
\lambda_{\min}\left(DK^\infty_{\frac{\sigma}{\sqrt{p+1}}}D\right)=\inf_{\|w\|=1}\left\langle DK^\infty_{\frac{\sigma}{\sqrt{p+1}}}Dw,w\right\rangle\\
&=\inf_{\|w\|=1}\|Dw\|^2\left\langle K^\infty_{\frac{\sigma}{\sqrt{p+1}}}\frac{Dw}{\|Dw\|},\frac{Dw}{\|Dw\|}\right\rangle\\
&\geq\lambda_{\min}\left(K^\infty_{\frac{\sigma}{\sqrt{p+1}}}\right)\cdot\lambda_{\min}(D^2),
\end{aligned} λ m i n ( D K p + 1 σ ∞ D ) = ∥ w ∥ = 1 inf ⟨ D K p + 1 σ ∞ D w , w ⟩ = ∥ w ∥ = 1 inf ∥ D w ∥ 2 ⟨ K p + 1 σ ∞ ∥ D w ∥ D w , ∥ D w ∥ D w ⟩ ≥ λ m i n ( K p + 1 σ ∞ ) ⋅ λ m i n ( D 2 ) ,
since D w / ∥ D w ∥ Dw/\|Dw\| D w /∥ D w ∥ is a unit vector and ∥ D w ∥ 2 ≥ λ min ( D 2 ) ∥ w ∥ 2 = λ min ( D 2 ) \|Dw\|^2\geq\lambda_{\min}(D^2)\|w\|^2=\lambda_{\min}(D^2) ∥ D w ∥ 2 ≥ λ m i n ( D 2 ) ∥ w ∥ 2 = λ m i n ( D 2 ) . Since each x ~ i \tilde{x}_i x ~ i has an appended 1, we have λ min ( D 2 ) = min i ∥ x ~ i ∥ 2 = min i ( ∥ x i ∥ 2 + 1 ) ≥ 1 \lambda_{\min}(D^2)=\min_i\|\tilde{x}_i\|^2=\min_i(\|x_i\|^2+1)\geq1 λ m i n ( D 2 ) = min i ∥ x ~ i ∥ 2 = min i ( ∥ x i ∥ 2 + 1 ) ≥ 1 . Moreover, by definition, we have:
λ min ( K σ p + 1 ∞ ) = inf ∥ w ∥ = 1 ⟨ K σ p + 1 ∞ w , w ⟩ . \lambda_{\min}\left(K^\infty_{\frac{\sigma}{\sqrt{p+1}}}\right)=\inf_{\|w\|=1}\left\langle K^\infty_{\frac{\sigma}{\sqrt{p+1}}}w,w\right\rangle. λ m i n ( K p + 1 σ ∞ ) = ∥ w ∥ = 1 inf ⟨ K p + 1 σ ∞ w , w ⟩ .
It then follows from Lemma E.6 that:
λ min ( D K σ p + 1 ∞ D ) ≥ C ( 1 + g ( ϵ ) ) − 3 ϵ 4 , \lambda_{\min}\left(DK^\infty_{\frac{\sigma}{\sqrt{p+1}}}D\right)\geq C(1+g(\epsilon))^{-3}\epsilon^4, λ m i n ( D K p + 1 σ ∞ D ) ≥ C ( 1 + g ( ϵ ) ) − 3 ϵ 4 ,
where C > 0 C>0 C > 0 is an absolute positive constant and g ( ϵ ) = ( p + 1 ) ln ( 1 / ϵ ) ln ( p + 1 ) g(\epsilon)=\frac{(p+1)\ln(1/\epsilon)}{\ln(p+1)} g ( ϵ ) = l n ( p + 1 ) ( p + 1 ) l n ( 1/ ϵ ) . Substituting this lower-bound into Equation (41) concludes the proof. □
Lemma E.8 (High-probability separation of homogeneous normalized points on S p S^p S p ). Let Assumptions a a a and d d d hold. Define the homogeneous normalized points { x ^ i } i = 1 n \{\hat{x}_i\}_{i=1}^n { x ^ i } i = 1 n as in Equation (40). Then, for any δ ∈ ( 0 , 1 ) \delta\in(0,1) δ ∈ ( 0 , 1 ) , with probability at least 1 − δ 1-\delta 1 − δ with respect to random samples, the normalized dataset { x ^ i } i = 1 n \{\hat{x}_i\}_{i=1}^n { x ^ i } i = 1 n is ϵ \epsilon ϵ -separated in the sense of Definition E.5 with
ϵ ≤ min ( 2 A 2 + 1 , 2 π ( 2 δ M μ ( A 2 + 1 ) ( p + 1 ) / 2 v p n 2 ) 1 p ) , \epsilon\leq\min\left(\frac{2}{\sqrt{A^2+1}},\frac{2}{\pi}\left(\frac{2\delta}{M_\mu(A^2+1)^{(p+1)/2}v_pn^2}\right)^{\frac{1}{p}}\right), ϵ ≤ min ( A 2 + 1 2 , π 2 ( M μ ( A 2 + 1 ) ( p + 1 ) /2 v p n 2 2 δ ) p 1 ) , where v p = Vol ( B ( 0 , 1 ) ) v_p=\operatorname{Vol}(B(0,1)) v p = Vol ( B ( 0 , 1 )) is the volume of the unit ball in R p \mathbb{R}^p R p .
Proof. Let ϕ : X ⊂ R p → S p ⊂ R p + 1 \phi:\mathcal{X}\subset\mathbb{R}^p\to S^p\subset\mathbb{R}^{p+1} ϕ : X ⊂ R p → S p ⊂ R p + 1 be the normalization map defined for any x ∈ X x\in\mathcal{X} x ∈ X by:
ϕ ( x ) = 1 ∥ x ∥ 2 + 1 [ x 1 ] (42) \phi(x)=\frac{1}{\sqrt{\|x\|^2+1}}\begin{bmatrix}x\\1\end{bmatrix} \tag*{(42)} ϕ ( x ) = ∥ x ∥ 2 + 1 1 [ x 1 ] (42)
so that x ^ i = ϕ ( x i ) \hat{x}_i=\phi(x_i) x ^ i = ϕ ( x i ) . Define the event that the dataset is not ϵ \epsilon ϵ -separated:
E ϵ = { min i ≠ k min ( ∥ x ^ i − x ^ k ∥ , ∥ x ^ i + x ^ k ∥ ) < ϵ } . \mathcal{E}_{\epsilon}=\left\{\min_{i\ne k}\min\left(\lVert\hat{x}_i-\hat{x}_k\rVert,\lVert\hat{x}_i+\hat{x}_k\rVert\right)<\epsilon\right\}. E ϵ = { i = k min min ( ∥ x ^ i − x ^ k ∥ , ∥ x ^ i + x ^ k ∥ ) < ϵ } .
i.e., there exists at least one pair i ≠ k i\ne k i = k for which the points are either too close together, or too close to antipodal. Hence, we obtain the following inclusion:
E ϵ ⊂ ⋃ i < k ( { ∥ x ^ i − x ^ k ∥ < ϵ } ∪ { ∥ x ^ i + x ^ k ∥ < ϵ } ) . \mathcal{E}_{\epsilon}\subset\bigcup_{i<k}\left(\left\{\lVert\hat{x}_i-\hat{x}_k\rVert<\epsilon\right\}\cup\left\{\lVert\hat{x}_i+\hat{x}_k\rVert<\epsilon\right\}\right). E ϵ ⊂ i < k ⋃ ( { ∥ x ^ i − x ^ k ∥ < ϵ } ∪ { ∥ x ^ i + x ^ k ∥ < ϵ } ) .
Note that the antipodal term ∥ x ^ i + x ^ k ∥ < ϵ \lVert\hat{x}_i+\hat{x}_k\rVert<\epsilon ∥ x ^ i + x ^ k ∥ < ϵ need not be considered provided that ϵ \epsilon ϵ is chosen small enough such that ϵ ≤ 2 / A 2 + 1 \epsilon\le2/\sqrt{A^2+1} ϵ ≤ 2/ A 2 + 1 . Indeed, for any x ∈ X x\in\mathcal{X} x ∈ X with ∥ x ∥ ≤ A \lVert x\rVert\le A ∥ x ∥ ≤ A (Assumption a), the last coordinate of ϕ ( x ) \phi(x) ϕ ( x ) is 1 / ∥ x ∥ 2 + 1 ≥ 1 / A 2 + 1 > 0 1/\sqrt{\lVert x\rVert^2+1}\ge1/\sqrt{A^2+1}>0 1/ ∥ x ∥ 2 + 1 ≥ 1/ A 2 + 1 > 0 , so all points x ^ i \hat{x}_i x ^ i lie in the open upper hemisphere S + p : = { z ∈ S p ∣ z p + 1 > 0 } \mathbb{S}^{p}_{+}:=\{z\in\mathbb{S}^{p}\mid z_{p+1}>0\} S + p := { z ∈ S p ∣ z p + 1 > 0 } , bounded away from the equator by the constant 1 / A 2 + 1 1/\sqrt{A^2+1} 1/ A 2 + 1 . Therefore ( x ^ i + x ^ k ) p + 1 ≥ 2 / A 2 + 1 (\hat{x}_i+\hat{x}_k)_{p+1}\ge2/\sqrt{A^2+1} ( x ^ i + x ^ k ) p + 1 ≥ 2/ A 2 + 1 for all i ≠ k i\ne k i = k . Since the last coordinate of a vector is dominated by its Euclidean norm, we have ∥ x ^ i + x ^ k ∥ ≥ ( x ^ i + x ^ k ) p + 1 ≥ 2 / A 2 + 1 \lVert\hat{x}_i+\hat{x}_k\rVert\ge(\hat{x}_i+\hat{x}_k)_{p+1}\ge2/\sqrt{A^2+1} ∥ x ^ i + x ^ k ∥ ≥ ( x ^ i + x ^ k ) p + 1 ≥ 2/ A 2 + 1 . Hence, provided that ϵ ≤ 2 / A 2 + 1 \epsilon\le2/\sqrt{A^2+1} ϵ ≤ 2/ A 2 + 1 , the event { ∥ x ^ i + x ^ k ∥ < ϵ } \{\lVert\hat{x}_i+\hat{x}_k\rVert<\epsilon\} {∥ x ^ i + x ^ k ∥ < ϵ } is empty for every pair i ≠ k i\ne k i = k . This implies that the antipodal event cannot occur. Thus:
E ϵ ⊂ ⋃ i < k { ∥ x ^ i − x ^ k ∥ < ϵ } . \mathcal{E}_{\epsilon}\subset\bigcup_{i<k}\left\{\lVert\hat{x}_i-\hat{x}_k\rVert<\epsilon\right\}. E ϵ ⊂ i < k ⋃ { ∥ x ^ i − x ^ k ∥ < ϵ } .
Therefore, by the union bound, we get:
Pr ( E ϵ ) ≤ ∑ i < k Pr ( ∥ x ^ i − x ^ k ∥ < ϵ ) . (43) \Pr(\mathcal{E}_{\epsilon})\le\sum_{i<k}\Pr\left(\lVert\hat{x}_i-\hat{x}_k\rVert<\epsilon\right). \tag*{(43)} Pr ( E ϵ ) ≤ i < k ∑ Pr ( ∥ x ^ i − x ^ k ∥ < ϵ ) . (43)
Let ν = ϕ # μ \nu=\phi_{\#}\mu ν = ϕ # μ be the pushforward distribution of μ \mu μ under the map ϕ \phi ϕ , so that x ^ i ∼ ν \hat{x}_i\sim\nu x ^ i ∼ ν on S p \mathbb{S}^{p} S p . Since x ^ i \hat{x}_i x ^ i and x ^ k \hat{x}_k x ^ k are independent and identically distributed samples from ν \nu ν on S p \mathbb{S}^{p} S p , conditioning on x ^ k = z \hat{x}_k=z x ^ k = z and applying the tower property yield:
Pr ( ∥ x ^ i − x ^ k ∥ < ϵ ) = E [ Pr ( ∥ x ^ i − z ∥ < ϵ ∣ x ^ k = z ) ] = E [ ν ( B S p ( z , ϵ ) ) ] , (44) \Pr\left(\lVert\hat{x}_i-\hat{x}_k\rVert<\epsilon\right)=\mathbb{E}\left[\Pr\left(\lVert\hat{x}_i-z\rVert<\epsilon\mid\hat{x}_k=z\right)\right]=\mathbb{E}\left[\nu\left(B_{\mathbb{S}^{p}}(z,\epsilon)\right)\right], \tag*{(44)} Pr ( ∥ x ^ i − x ^ k ∥ < ϵ ) = E [ Pr ( ∥ x ^ i − z ∥ < ϵ ∣ x ^ k = z ) ] = E [ ν ( B S p ( z , ϵ ) ) ] , (44)
where B S p ( z , ϵ ) = { x ^ ∈ S p ∣ ∥ x ^ − z ∥ < ϵ } B_{\mathbb{S}^{p}}(z,\epsilon)=\{\hat{x}\in\mathbb{S}^{p}\mid\lVert\hat{x}-z\rVert<\epsilon\} B S p ( z , ϵ ) = { x ^ ∈ S p ∣ ∥ x ^ − z ∥ < ϵ } is the Euclidean ball of radius ϵ \epsilon ϵ centered at z z z intersected with S p \mathbb{S}^{p} S p . We now derive the density of ν \nu ν with respect to the surface measure σ \sigma σ on S p \mathbb{S}^{p} S p . For any measurable set M ⊂ S p \mathcal{M}\subset\mathbb{S}^{p} M ⊂ S p , we have:
ν ( M ) = μ ( ϕ − 1 ( M ) ) = ∫ ϕ − 1 ( M ) f μ ( x ) d x = ∫ R p g ( x ) J ϕ ( x ) d x , (45) \nu(\mathcal{M})=\mu\left(\phi^{-1}(\mathcal{M})\right)=\int_{\phi^{-1}(\mathcal{M})}f_{\mu}(x)\,dx=\int_{\mathbb{R}^{p}}g(x)J_{\phi}(x)\,dx, \tag*{(45)} ν ( M ) = μ ( ϕ − 1 ( M ) ) = ∫ ϕ − 1 ( M ) f μ ( x ) d x = ∫ R p g ( x ) J ϕ ( x ) d x , (45)
where g ( x ) = f μ ( x ) / J ϕ ( x ) ⋅ 1 ϕ ( x ) ∈ M g(x)=f_{\mu}(x)/J_{\phi}(x)\cdot\mathbf{1}_{\phi(x)\in\mathcal{M}} g ( x ) = f μ ( x ) / J ϕ ( x ) ⋅ 1 ϕ ( x ) ∈ M , J ϕ ( x ) = det ( D ϕ ( x ) ⊤ D ϕ ( x ) ) J_{\phi}(x)=\sqrt{\det\left(D\phi(x)^{\top}D\phi(x)\right)} J ϕ ( x ) = det ( D ϕ ( x ) ⊤ D ϕ ( x ) ) with D ϕ ( x ) ∈ R ( p + 1 ) × p D\phi(x)\in\mathbb{R}^{(p+1)\times p} D ϕ ( x ) ∈ R ( p + 1 ) × p being the Jacobian matrix. Since ϕ \phi ϕ is C ∞ C^{\infty} C ∞ on R p \mathbb{R}^{p} R p , it is Lipschitz on the compact domain X \mathcal{X} X . The area formula [36 Theorem 3.9] therefore gives:
∫ R p g ( x ) J ϕ ( x ) d x = ∫ S p [ ∑ x ∈ ϕ − 1 ( z ) g ( x ) ] d σ ( z ) . \int_{\mathbb{R}^{p}}g(x)J_{\phi}(x)\,dx=\int_{\mathbb{S}^{p}}\left[\sum_{x\in\phi^{-1}(z)}g(x)\right]d\sigma(z). ∫ R p g ( x ) J ϕ ( x ) d x = ∫ S p x ∈ ϕ − 1 ( z ) ∑ g ( x ) d σ ( z ) .
Given that ϕ \phi ϕ is injective on R p \mathbb{R}^{p} R p , the preimage ϕ − 1 ( z ) \phi^{-1}(z) ϕ − 1 ( z ) contains exactly one point for each z ∈ S + p z\in\mathbb{S}^{p}_{+} z ∈ S + p and is empty for z ∉ S + p z\notin\mathbb{S}^{p}_{+} z ∈ / S + p . Using injectivity so that the sum has exactly one term for z ∈ S + p z\in\mathbb{S}^{p}_{+} z ∈ S + p , we obtain:
∫ R p g ( x ) J ϕ ( x ) d x = ∫ S p g ( ϕ − 1 ( z ) ) d σ ( z ) = ∫ M f μ ( ϕ − 1 ( z ) ) J ϕ ( ϕ − 1 ( z ) ) d σ ( z ) . \int_{\mathbb{R}^{p}}g(x)J_{\phi}(x)\,dx=\int_{\mathbb{S}^{p}}g\left(\phi^{-1}(z)\right)d\sigma(z)=\int_{\mathcal{M}}\frac{f_{\mu}\left(\phi^{-1}(z)\right)}{J_{\phi}\left(\phi^{-1}(z)\right)}d\sigma(z). ∫ R p g ( x ) J ϕ ( x ) d x = ∫ S p g ( ϕ − 1 ( z ) ) d σ ( z ) = ∫ M J ϕ ( ϕ − 1 ( z ) ) f μ ( ϕ − 1 ( z ) ) d σ ( z ) .
Next, we find the expression of J ϕ ( x ) J_{\phi}(x) J ϕ ( x ) . Write s = ∥ x ∥ 2 + 1 s=\lVert x\rVert^{2}+1 s = ∥ x ∥ 2 + 1 for brevity. D ϕ ( x ) D\phi(x) D ϕ ( x ) has entries, for 1 ≤ i ≤ p 1\le i\le p 1 ≤ i ≤ p and 1 ≤ j ≤ p 1\le j\le p 1 ≤ j ≤ p , obtained by the product rule applied to ϕ i ( x ) = x i s − 1 / 2 \phi_i(x)=x_i s^{-1/2} ϕ i ( x ) = x i s − 1/2 :
∂ ϕ i ∂ x j ( x ) = δ i j s − 1 / 2 + x i ⋅ ( − x j s − 3 / 2 ) = δ i j s − x i x j s 3 / 2 , \frac{\partial\phi_i}{\partial x_j}(x)=\delta_{ij}s^{-1/2}+x_i\cdot\left(-x_j s^{-3/2}\right)=\frac{\delta_{ij}s-x_ix_j}{s^{3/2}}, ∂ x j ∂ ϕ i ( x ) = δ ij s − 1/2 + x i ⋅ ( − x j s − 3/2 ) = s 3/2 δ ij s − x i x j ,
where δ i j \delta_{ij} δ ij is the Kronecker delta, and for the last row, using ϕ p + 1 ( x ) = s − 1 / 2 \phi_{p+1}(x)=s^{-1/2} ϕ p + 1 ( x ) = s − 1/2 and the chain rule:
∂ ϕ p + 1 ∂ x j ( x ) = − x j s 3 / 2 . \frac{\partial\phi_{p+1}}{\partial x_j}(x)=-\frac{x_j}{s^{3/2}}. ∂ x j ∂ ϕ p + 1 ( x ) = − s 3/2 x j .
Therefore, the full Jacobian matrix is:
D ϕ ( x ) = 1 s 3 / 2 [ s 1 p × p − x x T − x T ] ∈ R ( p + 1 ) × p . D\phi(x)=\frac{1}{s^{3/2}}\begin{bmatrix}s\mathbf{1}_{p\times p}-xx^{\mathsf{T}}\\-x^{\mathsf{T}}\end{bmatrix}\in\mathbb{R}^{(p+1)\times p}. D ϕ ( x ) = s 3/2 1 [ s 1 p × p − x x T − x T ] ∈ R ( p + 1 ) × p .
We compute the Gram matrix D ϕ ( x ) T D ϕ ( x ) ∈ R p × p D\phi(x)^{\mathsf{T}}D\phi(x)\in\mathbb{R}^{p\times p} D ϕ ( x ) T D ϕ ( x ) ∈ R p × p . Expanding the block matrix product and using ( x x T ) ( x x T ) = ∥ x ∥ 2 x x T = ( s − 1 ) x x T (xx^{\mathsf{T}})(xx^{\mathsf{T}})=\lVert x\rVert^{2}xx^{\mathsf{T}}=(s-1)xx^{\mathsf{T}} ( x x T ) ( x x T ) = ∥ x ∥ 2 x x T = ( s − 1 ) x x T , we obtain:
D ϕ ( x ) T D ϕ ( x ) = 1 s 3 [ ( s 1 p × p − x x T ) 2 + x x T ] = 1 s 3 [ s 2 1 p × p − ( s + 1 ) x x T + x x T ] = 1 s ( 1 p × p − x x T s ) . D\phi(x)^{\mathsf{T}}D\phi(x)=\frac{1}{s^{3}}\left[(s\mathbf{1}_{p\times p}-xx^{\mathsf{T}})^{2}+xx^{\mathsf{T}}\right]=\frac{1}{s^{3}}\left[s^{2}\mathbf{1}_{p\times p}-(s+1)xx^{\mathsf{T}}+xx^{\mathsf{T}}\right]
=\frac{1}{s}\left(\mathbf{1}_{p\times p}-\frac{xx^{\mathsf{T}}}{s}\right). D ϕ ( x ) T D ϕ ( x ) = s 3 1 [ ( s 1 p × p − x x T ) 2 + x x T ] = s 3 1 [ s 2 1 p × p − ( s + 1 ) x x T + x x T ] = s 1 ( 1 p × p − s x x T ) .
Applying the matrix determinant lemma det ( 1 p × p + u v T ) = 1 + v T u \det(\mathbf{1}_{p\times p}+uv^{\mathsf{T}})=1+v^{\mathsf{T}}u det ( 1 p × p + u v T ) = 1 + v T u with u = − x / s u=-x/\sqrt{s} u = − x / s and v = x / s v=x/\sqrt{s} v = x / s :
det ( 1 p × p − x x T s ) = 1 − ∥ x ∥ 2 s = 1 − s − 1 s = 1 s . \det\left(\mathbf{1}_{p\times p}-\frac{xx^{\mathsf{T}}}{s}\right)=1-\frac{\lVert x\rVert^{2}}{s}=1-\frac{s-1}{s}=\frac{1}{s}. det ( 1 p × p − s x x T ) = 1 − s ∥ x ∥ 2 = 1 − s s − 1 = s 1 .
Therefore:
det ( D ϕ ( x ) T D ϕ ( x ) ) = 1 s p ⋅ 1 s = 1 ( ∥ x ∥ 2 + 1 ) p + 1 . \det\left(D\phi(x)^{\mathsf{T}}D\phi(x)\right)=\frac{1}{s^{p}}\cdot\frac{1}{s}=\frac{1}{(\lVert x\rVert^{2}+1)^{p+1}}. det ( D ϕ ( x ) T D ϕ ( x ) ) = s p 1 ⋅ s 1 = (∥ x ∥ 2 + 1 ) p + 1 1 .
and the Jacobian factor is:
J ϕ ( x ) = det ( D ϕ ( x ) T D ϕ ( x ) ) = 1 ( ∥ x ∥ 2 + 1 ) ( p + 1 ) / 2 . J_{\phi}(x)=\sqrt{\det\left(D\phi(x)^{\mathsf{T}}D\phi(x)\right)}=\frac{1}{(\lVert x\rVert^{2}+1)^{(p+1)/2}}. J ϕ ( x ) = det ( D ϕ ( x ) T D ϕ ( x ) ) = (∥ x ∥ 2 + 1 ) ( p + 1 ) /2 1 .
Consequently, we obtain:
∫ R p g ( x ) J ϕ ( x ) d x = ∫ M f μ ( ϕ − 1 ( z ) ) ⋅ ( ∥ ϕ − 1 ( z ) ∥ 2 + 1 ) ( p + 1 ) / 2 d σ ( z ) . \int_{\mathbb{R}^{p}}g(x)J_{\phi}(x)\,dx=\int_{\mathcal{M}}f_{\mu}\left(\phi^{-1}(z)\right)\cdot\left(\lVert\phi^{-1}(z)\rVert^{2}+1\right)^{(p+1)/2}\,d\sigma(z). ∫ R p g ( x ) J ϕ ( x ) d x = ∫ M f μ ( ϕ − 1 ( z ) ) ⋅ ( ∥ ϕ − 1 ( z ) ∥ 2 + 1 ) ( p + 1 ) /2 d σ ( z ) .
Substituting this into Equation (45) , we get:
ν ( M ) = ∫ M f μ ( ϕ − 1 ( z ) ) ⋅ ( ∥ ϕ − 1 ( z ) ∥ 2 + 1 ) ( p + 1 ) / 2 d σ ( z ) . \nu(\mathcal{M})=\int_{\mathcal{M}}f_{\mu}\left(\phi^{-1}(z)\right)\cdot\left(\lVert\phi^{-1}(z)\rVert^{2}+1\right)^{(p+1)/2}\,d\sigma(z). ν ( M ) = ∫ M f μ ( ϕ − 1 ( z ) ) ⋅ ( ∥ ϕ − 1 ( z ) ∥ 2 + 1 ) ( p + 1 ) /2 d σ ( z ) .
Since this holds for any measurable set M ⊂ S p \mathcal{M}\subset\mathbb{S}^{p} M ⊂ S p , by uniqueness of the Radon-Nikodym derivative, ν \nu ν has the following density with respect to σ \sigma σ :
f ν ( z ) = f μ ( ϕ − 1 ( z ) ) ⋅ ( ∥ ϕ − 1 ( z ) ∥ 2 + 1 ) ( p + 1 ) / 2 . f_{\nu}(z)=f_{\mu}\left(\phi^{-1}(z)\right)\cdot\left(\lVert\phi^{-1}(z)\rVert^{2}+1\right)^{(p+1)/2}. f ν ( z ) = f μ ( ϕ − 1 ( z ) ) ⋅ ( ∥ ϕ − 1 ( z ) ∥ 2 + 1 ) ( p + 1 ) /2 .
The inverse map is ϕ − 1 ( z ) = z 1 : p / z p + 1 \phi^{-1}(z)=z_{1:p}/z_{p+1} ϕ − 1 ( z ) = z 1 : p / z p + 1 , where z 1 : p = [ z 1 , … , z p ] T z_{1:p}=[z_{1},\ldots,z_{p}]^{\mathsf{T}} z 1 : p = [ z 1 , … , z p ] T denotes the first p p p coordinates of z z z . Since z ∈ S p z\in\mathbb{S}^{p} z ∈ S p we have ∥ z 1 : p ∥ 2 + z p + 1 2 = 1 \lVert z_{1:p}\rVert^{2}+z_{p+1}^{2}=1 ∥ z 1 : p ∥ 2 + z p + 1 2 = 1 , therefore:
∥ ϕ − 1 ( z ) ∥ 2 + 1 = ∥ z 1 : p ∥ 2 z p + 1 2 + 1 = ∥ z 1 : p ∥ 2 + z p + 1 2 z p + 1 2 = 1 z p + 1 2 = ∥ x ∥ 2 + 1. \lVert\phi^{-1}(z)\rVert^{2}+1=\frac{\lVert z_{1:p}\rVert^{2}}{z_{p+1}^{2}}+1=\frac{\lVert z_{1:p}\rVert^{2}+z_{p+1}^{2}}{z_{p+1}^{2}}=\frac{1}{z_{p+1}^{2}}=\lVert x\rVert^{2}+1. ∥ ϕ − 1 ( z ) ∥ 2 + 1 = z p + 1 2 ∥ z 1 : p ∥ 2 + 1 = z p + 1 2 ∥ z 1 : p ∥ 2 + z p + 1 2 = z p + 1 2 1 = ∥ x ∥ 2 + 1.
Given that ∥ x ∥ ≤ A \lVert x\rVert\le A ∥ x ∥ ≤ A for all x ∈ X x\in\mathcal{X} x ∈ X by Assumption a, we have ∥ ϕ − 1 ( z ) ∥ 2 + 1 ≤ A 2 + 1 \lVert\phi^{-1}(z)\rVert^{2}+1\le A^{2}+1 ∥ ϕ − 1 ( z ) ∥ 2 + 1 ≤ A 2 + 1 . Moreover, we have by Assumption d that ∥ f μ ∥ ∞ ≤ M μ \lVert f_{\mu}\rVert_{\infty}\le M_{\mu} ∥ f μ ∥ ∞ ≤ M μ . This yields:
∥ f ν ∥ ∞ ≤ ∥ f μ ∥ ∞ ⋅ sup x ∈ X ( ∥ x ∥ 2 + 1 ) ( p + 1 ) / 2 ≤ M μ ( A 2 + 1 ) ( p + 1 ) / 2 . \lVert f_{\nu}\rVert_{\infty}\le\lVert f_{\mu}\rVert_{\infty}\cdot\sup_{x\in\mathcal{X}}(\lVert x\rVert^{2}+1)^{(p+1)/2}\le M_{\mu}(A^{2}+1)^{(p+1)/2}. ∥ f ν ∥ ∞ ≤ ∥ f μ ∥ ∞ ⋅ x ∈ X sup (∥ x ∥ 2 + 1 ) ( p + 1 ) /2 ≤ M μ ( A 2 + 1 ) ( p + 1 ) /2 .
From this bound and Lemma I.4, we bound the ν \nu ν -measure of B S p ( z , ϵ ) B_{\mathbb{S}^{p}}(z,\epsilon) B S p ( z , ϵ ) as follows:
ν ( B S p ( z , ϵ ) ) = ∫ B S p ( z , ϵ ) f ν ( x ^ ) d σ ( x ^ ) ≤ ∥ f ν ∥ ∞ ⋅ σ ( B S p ( z , ϵ ) ) ≤ M μ ( A 2 + 1 ) ( p + 1 ) / 2 v p ( π 2 ϵ ) p , \nu\left(B_{\mathbb{S}^{p}}(z,\epsilon)\right)=\int_{B_{\mathbb{S}^{p}}(z,\epsilon)}f_{\nu}(\hat{x})\,d\sigma(\hat{x})\le\lVert f_{\nu}\rVert_{\infty}\cdot\sigma\left(B_{\mathbb{S}^{p}}(z,\epsilon)\right)\le M_{\mu}(A^{2}+1)^{(p+1)/2}v_{p}\left(\frac{\pi}{2}\epsilon\right)^{p}, ν ( B S p ( z , ϵ ) ) = ∫ B S p ( z , ϵ ) f ν ( x ^ ) d σ ( x ^ ) ≤ ∥ f ν ∥ ∞ ⋅ σ ( B S p ( z , ϵ ) ) ≤ M μ ( A 2 + 1 ) ( p + 1 ) /2 v p ( 2 π ϵ ) p ,
where v p v_{p} v p is the volume of the unit ball in R p \mathbb{R}^{p} R p . Since this bound is uniform in z z z , after substituting it into Equation (44) , we obtain:
Pr ( ∥ x ^ i − x ^ k ∥ < ϵ ) ≤ M μ ( A 2 + 1 ) ( p + 1 ) / 2 v p ( π 2 ϵ ) p . \Pr\left(\lVert\hat{x}_{i}-\hat{x}_{k}\rVert<\epsilon\right)\le M_{\mu}(A^{2}+1)^{(p+1)/2}v_{p}\left(\frac{\pi}{2}\epsilon\right)^{p}. Pr ( ∥ x ^ i − x ^ k ∥ < ϵ ) ≤ M μ ( A 2 + 1 ) ( p + 1 ) /2 v p ( 2 π ϵ ) p .
Putting this bound into the union bound in Equation (43) and using ( n 2 ) ≤ n 2 / 2 \binom{n}{2} \le n^2/2 ( 2 n ) ≤ n 2 /2 , we get:
Pr ( E ϵ ) ≤ ∑ i < k M μ ( A 2 + 1 ) ( p + 1 ) / 2 v p ( π 2 ϵ ) p = M μ ( A 2 + 1 ) ( p + 1 ) / 2 v p ( π 2 ϵ ) p ∑ i < k 1 = M μ ( A 2 + 1 ) ( p + 1 ) / 2 v p ( π 2 ϵ ) p ( n 2 ) ≤ M μ ( A 2 + 1 ) ( p + 1 ) / 2 v p ( π 2 ϵ ) p n 2 2 . \begin{aligned}
\Pr(\mathcal{E}_{\epsilon}) \le\sum_{i<k} M_{\mu}(A^2+1)^{(p+1)/2}v_p\left(\frac{\pi}{2}\epsilon\right)^p \\
&= M_{\mu}(A^2+1)^{(p+1)/2}v_p\left(\frac{\pi}{2}\epsilon\right)^p \sum_{i<k}1 \\
&= M_{\mu}(A^2+1)^{(p+1)/2}v_p\left(\frac{\pi}{2}\epsilon\right)^p \binom{n}{2} \\
&\le M_{\mu}(A^2+1)^{(p+1)/2}v_p\left(\frac{\pi}{2}\epsilon\right)^p\frac{n^2}{2}.
\end{aligned} Pr ( E ϵ ) ≤ i < k ∑ M μ ( A 2 + 1 ) ( p + 1 ) /2 v p ( 2 π ϵ ) p = M μ ( A 2 + 1 ) ( p + 1 ) /2 v p ( 2 π ϵ ) p i < k ∑ 1 = M μ ( A 2 + 1 ) ( p + 1 ) /2 v p ( 2 π ϵ ) p ( 2 n ) ≤ M μ ( A 2 + 1 ) ( p + 1 ) /2 v p ( 2 π ϵ ) p 2 n 2 .
Setting this quantity less than or equal to δ \delta δ and solving for ϵ \epsilon ϵ :
M μ ( A 2 + 1 ) ( p + 1 ) / 2 v p ( π 2 ϵ ) p n 2 2 ≤ δ ⟹ ϵ ≤ 2 π ( 2 δ M μ ( A 2 + 1 ) ( p + 1 ) / 2 v p n 2 ) 1 / p . M_{\mu}(A^2+1)^{(p+1)/2}v_p\left(\frac{\pi}{2}\epsilon\right)^p\frac{n^2}{2}\le\delta\quad\Longrightarrow\quad\epsilon\le\frac{2}{\pi}\left(\frac{2\delta}{M_{\mu}(A^2+1)^{(p+1)/2}v_pn^2}\right)^{1/p}. M μ ( A 2 + 1 ) ( p + 1 ) /2 v p ( 2 π ϵ ) p 2 n 2 ≤ δ ⟹ ϵ ≤ π 2 ( M μ ( A 2 + 1 ) ( p + 1 ) /2 v p n 2 2 δ ) 1/ p .
Therefore, for any ϵ ≤ 2 π ( 2 δ M μ ( A 2 + 1 ) ( p + 1 ) / 2 v p n 2 ) 1 / p \epsilon\le\frac{2}{\pi}\left(\frac{2\delta}{M_{\mu}(A^2+1)^{(p+1)/2}v_pn^2}\right)^{1/p} ϵ ≤ π 2 ( M μ ( A 2 + 1 ) ( p + 1 ) /2 v p n 2 2 δ ) 1/ p , we have Pr ( E ϵ ) ≤ δ \Pr(\mathcal{E}_{\epsilon})\le\delta Pr ( E ϵ ) ≤ δ , i.e., the normalized dataset { x ^ i } i = 1 n \{\hat{x}_i\}_{i=1}^n { x ^ i } i = 1 n is ϵ \epsilon ϵ -separated with probability at least 1 − δ 1-\delta 1 − δ . Combining both upper-bounds on ϵ \epsilon ϵ , we obtain:
ϵ ≤ min ( 2 A 2 + 1 , 2 π ( 2 δ M μ ( A 2 + 1 ) ( p + 1 ) / 2 v p n 2 ) 1 / p ) , \epsilon\le\min\left(\frac{2}{\sqrt{A^2+1}},\frac{2}{\pi}\left(\frac{2\delta}{M_{\mu}(A^2+1)^{(p+1)/2}v_pn^2}\right)^{1/p}\right), ϵ ≤ min ( A 2 + 1 2 , π 2 ( M μ ( A 2 + 1 ) ( p + 1 ) /2 v p n 2 2 δ ) 1/ p ) ,
which concludes the proof.
Lemma E.9. (High-probability lower-bound on the smallest eigenvalue of K N T K K^{\mathrm{NTK}} K NTK ). Fix p ≥ 2 p\ge2 p ≥ 2 and let Assumptions a and d hold . Define the homogeneous normalized points { x ^ i } i = 1 n \{\hat{x}_i\}_{i=1}^n { x ^ i } i = 1 n as in Equation (40). Then, for any δ ∈ ( 0 , 1 ) \delta\in(0,1) δ ∈ ( 0 , 1 ) , with probability at least 1 − δ 1-\delta 1 − δ with respect to random samples, the smallest eigenvalue λ 0 \lambda_0 λ 0 of K N T K K^{\mathrm{NTK}} K NTK satisfies:
λ 0 ≥ C ( 1 + g ( ϵ ) ) − 3 ϵ 4 for every 0 < ϵ ≤ min ( 1 , 2 A 2 + 1 , 2 π ( 2 δ M μ ( A 2 + 1 ) ( p + 1 ) / 2 v p n 2 ) 1 / p ) , \lambda_0\ge C(1+g(\epsilon))^{-3}\epsilon^4\quad\text{for every}\quad0<\epsilon\le\min\left(1,\frac{2}{\sqrt{A^2+1}},\frac{2}{\pi}\left(\frac{2\delta}{M_{\mu}(A^2+1)^{(p+1)/2}v_pn^2}\right)^{1/p}\right), λ 0 ≥ C ( 1 + g ( ϵ ) ) − 3 ϵ 4 for every 0 < ϵ ≤ min ( 1 , A 2 + 1 2 , π 2 ( M μ ( A 2 + 1 ) ( p + 1 ) /2 v p n 2 2 δ ) 1/ p ) , where C > 0 C>0 C > 0 is an absolute positive constant, g ( ϵ ) = ( p + 1 ) ln ( 1 / ϵ ) ln ( p + 1 ) g(\epsilon)=\frac{(p+1)\ln(1/\epsilon)}{\ln(p+1)} g ( ϵ ) = l n ( p + 1 ) ( p + 1 ) l n ( 1/ ϵ ) , and v p = Vol ( B ( 0 , 1 ) ) v_p=\operatorname{Vol}(B(0,1)) v p = Vol ( B ( 0 , 1 )) is the volume of the unit ball in R p \mathbb{R}^p R p .
Proof. By Lemma E.8 , with probability at least 1 − δ 1-\delta 1 − δ with respect to random samples, the dataset { x ^ i } i = 1 n \{\hat{x}_i\}_{i=1}^n { x ^ i } i = 1 n is ϵ ˉ \bar{\epsilon} ϵ ˉ -separated with:
ϵ ˉ ≤ min ( 2 A 2 + 1 , 2 π ( 2 δ M μ ( A 2 + 1 ) ( p + 1 ) / 2 v p n 2 ) 1 / p ) . \bar{\epsilon}\le\min\left(\frac{2}{\sqrt{A^2+1}},\frac{2}{\pi}\left(\frac{2\delta}{M_{\mu}(A^2+1)^{(p+1)/2}v_pn^2}\right)^{1/p}\right). ϵ ˉ ≤ min ( A 2 + 1 2 , π 2 ( M μ ( A 2 + 1 ) ( p + 1 ) /2 v p n 2 2 δ ) 1/ p ) .
On this event, it is also ϵ \epsilon ϵ -separated for every 0 < ϵ ≤ ϵ ˉ 0<\epsilon\le\bar{\epsilon} 0 < ϵ ≤ ϵ ˉ , since the condition in Definition E.5 becomes weaker as ϵ \epsilon ϵ decreases. Consequently, by Lemma E.7 , we obtain on this event, for every 0 < ϵ ≤ min ( 1 , ϵ ˉ ) 0<\epsilon\le\min(1,\bar{\epsilon}) 0 < ϵ ≤ min ( 1 , ϵ ˉ ) , the following lower-bound on λ 0 \lambda_0 λ 0 :
λ 0 ≥ C ( 1 + g ( ϵ ) ) − 3 ϵ 4 , \lambda_0\ge C(1+g(\epsilon))^{-3}\epsilon^4, λ 0 ≥ C ( 1 + g ( ϵ ) ) − 3 ϵ 4 ,
where C > 0 C>0 C > 0 is an absolute positive constant and g ( ϵ ) = ( p + 1 ) ln ( 1 / ϵ ) ln ( p + 1 ) g(\epsilon)=\frac{(p+1)\ln(1/\epsilon)}{\ln(p+1)} g ( ϵ ) = l n ( p + 1 ) ( p + 1 ) l n ( 1/ ϵ ) . Note that we require ϵ ≤ 1 \epsilon\le1 ϵ ≤ 1 to ensure that g ( ϵ ) ≥ 0 g(\epsilon)\ge0 g ( ϵ ) ≥ 0 , and hence 1 + g ( ϵ ) ≥ 1 1+g(\epsilon)\ge1 1 + g ( ϵ ) ≥ 1 , so that the lower-bound is well defined. This concludes the proof.
Corollary E.10. (High-probability upper-bound on the inverse of the smallest eigenvalue of K N T K K^{\mathrm{NTK}} K NTK ). Fix p ≥ 2 p\ge2 p ≥ 2 and let Assumptions a and d hold . Define the homogeneous normalized points { x ^ i } i = 1 n \{\hat{x}_i\}_{i=1}^n { x ^ i } i = 1 n as in Equation (40), and, for n ≥ 1 n\ge1 n ≥ 1 and δ ∈ ( 0 , 1 ) \delta\in(0,1) δ ∈ ( 0 , 1 ) , the deterministic threshold λ ⋆ ( n , δ ) : = h ( min ( ϵ A , ϵ 0 ( n , δ ) ) ) > 0 \lambda_{\star}(n,\delta):=h(\min(\epsilon_A,\epsilon_0(n,\delta)))>0 λ ⋆ ( n , δ ) := h ( min ( ϵ A , ϵ 0 ( n , δ ))) > 0 , where h ( ϵ ) : = C ( 1 + g ( ϵ ) ) − 3 ϵ 4 h(\epsilon):=C(1+g(\epsilon))^{-3}\epsilon^4 h ( ϵ ) := C ( 1 + g ( ϵ ) ) − 3 ϵ 4 , ϵ A : = min ( 1 , 2 / A 2 + 1 ) \epsilon_A:=\min(1,2/\sqrt{A^2+1}) ϵ A := min ( 1 , 2/ A 2 + 1 )
and ϵ 0 ( n , δ ) : = 2 π ( 2 δ M μ ( A 2 + 1 ) ( p + 1 ) / 2 v p n 2 ) 1 / p \epsilon_0(n,\delta) := \frac{2}{\pi}\left(\frac{2\delta}{M_\mu(A^2+1)^{(p+1)/2}v_p n^2}\right)^{1/p} ϵ 0 ( n , δ ) := π 2 ( M μ ( A 2 + 1 ) ( p + 1 ) /2 v p n 2 2 δ ) 1/ p , with C C C , g g g and v p v_p v p as in Lemma E.9. Then, there exists a constant C ′ > 0 C' > 0 C ′ > 0 , depending only on A A A , M μ M_\mu M μ , and p p p , such that, for any n ≥ 3 n \ge3 n ≥ 3 and δ ∈ ( 0 , 1 ) \delta\in(0,1) δ ∈ ( 0 , 1 ) , with probability 1 − δ 1-\delta 1 − δ with respect to random samples, the smallest eigenvalue λ 0 \lambda_0 λ 0 of K N T K \mathbf{K}^{\mathrm{NTK}} K NTK satisfies λ 0 ≥ λ ⋆ ( n , δ ) \lambda_0 \ge\lambda_\star(n,\delta) λ 0 ≥ λ ⋆ ( n , δ ) , and hence:
λ 0 − 1 ≤ λ ⋆ ( n , δ ) − 1 ≤ C ′ n 8 / p δ − 4 / p ( ln n + ln ( 1 / δ ) ) 3 . \lambda_0^{-1} \le\lambda_\star(n,\delta)^{-1} \le C'n^{8/p}\delta^{-4/p}\left(\ln n+\ln(1/\delta)\right)^3. λ 0 − 1 ≤ λ ⋆ ( n , δ ) − 1 ≤ C ′ n 8/ p δ − 4/ p ( ln n + ln ( 1/ δ ) ) 3 .
Proof. Since ϵ A ≤ 1 \epsilon_A \le1 ϵ A ≤ 1 , we have g ( ϵ A ) ≥ 0 g(\epsilon_A) \ge0 g ( ϵ A ) ≥ 0 , hence 1 + g ( ϵ A ) ≥ 1 1+g(\epsilon_A) \ge1 1 + g ( ϵ A ) ≥ 1 and κ : = h ( ϵ A ) > 0 \kappa:= h(\epsilon_A) > 0 κ := h ( ϵ A ) > 0 , a constant depending on A A A and p p p . Similarly, λ ⋆ ( n , δ ) > 0 \lambda_\star(n,\delta) > 0 λ ⋆ ( n , δ ) > 0 , since min ( ϵ A , ϵ 0 ( n , δ ) ) ∈ ( 0 , 1 ] \min(\epsilon_A,\epsilon_0(n,\delta)) \in(0,1] min ( ϵ A , ϵ 0 ( n , δ )) ∈ ( 0 , 1 ] , which is admissible in Lemma E.9. Therefore, with probability at least 1 − δ 1-\delta 1 − δ with respect to random samples, λ 0 ≥ λ ⋆ ( n , δ ) \lambda_0 \ge\lambda_\star(n,\delta) λ 0 ≥ λ ⋆ ( n , δ ) . It remains to bound λ ⋆ ( n , δ ) − 1 \lambda_\star(n,\delta)^{-1} λ ⋆ ( n , δ ) − 1 . To this end, we define:
L : = ln n + ln ( 1 / δ ) , F : = n 8 / p δ − 4 / p L 3 , M : = M μ ( A 2 + 1 ) ( p + 1 ) / 2 v p . L := \ln n+\ln(1/\delta), \qquad F := n^{8/p}\delta^{-4/p}L^3, \qquad M := M_\mu(A^2+1)^{(p+1)/2}v_p. L := ln n + ln ( 1/ δ ) , F := n 8/ p δ − 4/ p L 3 , M := M μ ( A 2 + 1 ) ( p + 1 ) /2 v p .
Since n ≥ 3 n \ge3 n ≥ 3 , we have ln n ≥ ln 3 > 1 \ln n \ge\ln3 > 1 ln n ≥ ln 3 > 1 , and since δ ∈ ( 0 , 1 ) \delta\in(0,1) δ ∈ ( 0 , 1 ) , we have ln ( 1 / δ ) > 0 \ln(1/\delta) > 0 ln ( 1/ δ ) > 0 ; hence L ≥ 1 L \ge1 L ≥ 1 . Moreover, n 8 / p ≥ 1 n^{8/p} \ge1 n 8/ p ≥ 1 and δ − 4 / p ≥ 1 \delta^{-4/p} \ge1 δ − 4/ p ≥ 1 , so that F ≥ 1 F \ge1 F ≥ 1 . We distinguish two cases.
Case 1: ϵ 0 ( n , δ ) > ϵ A \epsilon_0(n,\delta) > \epsilon_A ϵ 0 ( n , δ ) > ϵ A . Then λ ⋆ ( n , δ ) = h ( ϵ A ) = κ \lambda_\star(n,\delta) = h(\epsilon_A) = \kappa λ ⋆ ( n , δ ) = h ( ϵ A ) = κ , so that, using F ≥ 1 F \ge1 F ≥ 1 , we get λ ⋆ ( n , δ ) − 1 = κ − 1 ≤ κ − 1 F \lambda_\star(n,\delta)^{-1} = \kappa^{-1} \le\kappa^{-1}F λ ⋆ ( n , δ ) − 1 = κ − 1 ≤ κ − 1 F .
Case 2: ϵ 0 ( n , δ ) ≤ ϵ A \epsilon_0(n,\delta) \le\epsilon_A ϵ 0 ( n , δ ) ≤ ϵ A . Then λ ⋆ ( n , δ ) = h ( ϵ 0 ( n , δ ) ) \lambda_\star(n,\delta) = h(\epsilon_0(n,\delta)) λ ⋆ ( n , δ ) = h ( ϵ 0 ( n , δ )) . Since ϵ 0 ( n , δ ) ≤ 1 \epsilon_0(n,\delta) \le1 ϵ 0 ( n , δ ) ≤ 1 , the quantity
Λ : = ln 1 ϵ 0 ( n , δ ) = ln ( π 2 ) + 1 p ln ( M n 2 2 δ ) \Lambda:= \ln\frac{1}{\epsilon_0(n,\delta)}=\ln\left(\frac{\pi}{2}\right)+\frac{1}{p}\ln\left(\frac{Mn^2}{2\delta}\right) Λ := ln ϵ 0 ( n , δ ) 1 = ln ( 2 π ) + p 1 ln ( 2 δ M n 2 )
is non-negative, and 1 + g ( ϵ 0 ( n , δ ) ) = 1 + ( p + 1 ) Λ ln ( p + 1 ) ≥ 1 1+g(\epsilon_0(n,\delta))=1+\frac{(p+1)\Lambda}{\ln(p+1)}\ge1 1 + g ( ϵ 0 ( n , δ )) = 1 + l n ( p + 1 ) ( p + 1 ) Λ ≥ 1 . Substituting ϵ 0 ( n , δ ) \epsilon_0(n,\delta) ϵ 0 ( n , δ ) gives:
λ ⋆ ( n , δ ) − 1 = h ( ϵ 0 ( n , δ ) ) − 1 = C − 1 ( π 2 ) 4 ( M 2 ) 4 / p ( 1 + ( p + 1 ) Λ ln ( p + 1 ) ) 3 n 8 / p δ − 4 / p . \lambda_\star(n,\delta)^{-1}=h(\epsilon_0(n,\delta))^{-1}=C^{-1}\left(\frac{\pi}{2}\right)^4\left(\frac{M}{2}\right)^{4/p}\left(1+\frac{(p+1)\Lambda}{\ln(p+1)}\right)^3 n^{8/p}\delta^{-4/p}. λ ⋆ ( n , δ ) − 1 = h ( ϵ 0 ( n , δ ) ) − 1 = C − 1 ( 2 π ) 4 ( 2 M ) 4/ p ( 1 + ln ( p + 1 ) ( p + 1 ) Λ ) 3 n 8/ p δ − 4/ p .
Expanding the logarithm, we have Λ = ln ( π 2 ) + 1 p ln M 2 + 2 p ln n + 1 p ln 1 δ \Lambda=\ln\left(\frac{\pi}{2}\right)+\frac{1}{p}\ln\frac{M}{2}+\frac{2}{p}\ln n+\frac{1}{p}\ln\frac{1}{\delta} Λ = ln ( 2 π ) + p 1 ln 2 M + p 2 ln n + p 1 ln δ 1 . Since p ≥ 2 p\ge2 p ≥ 2 and ln n , ln ( 1 / δ ) ≥ 0 \ln n,\ln(1/\delta)\ge0 ln n , ln ( 1/ δ ) ≥ 0 , this gives Λ ≤ a 0 + L \Lambda\le a_0+L Λ ≤ a 0 + L , where a 0 : = ∣ ln ( π 2 ) + 1 p ln M 2 ∣ a_0:=\left|\ln\left(\frac{\pi}{2}\right)+\frac{1}{p}\ln\frac{M}{2}\right| a 0 := ln ( 2 π ) + p 1 ln 2 M . Using L ≥ 1 L\ge1 L ≥ 1 , we have a 0 + L ≤ ( a 0 + 1 ) L a_0+L\le(a_0+1)L a 0 + L ≤ ( a 0 + 1 ) L , hence:
1 + ( p + 1 ) Λ ln ( p + 1 ) ≤ L + ( p + 1 ) ( a 0 + 1 ) ln ( p + 1 ) L = b L , 1+\frac{(p+1)\Lambda}{\ln(p+1)}
\le L+\frac{(p+1)(a_0+1)}{\ln(p+1)}L=bL, 1 + ln ( p + 1 ) ( p + 1 ) Λ ≤ L + ln ( p + 1 ) ( p + 1 ) ( a 0 + 1 ) L = b L ,
where b : = 1 + ( p + 1 ) ( a 0 + 1 ) ln ( p + 1 ) b:=1+\frac{(p+1)(a_0+1)}{\ln(p+1)} b := 1 + l n ( p + 1 ) ( p + 1 ) ( a 0 + 1 ) . Both sides being positive, cubing preserves the inequality, and we obtain:
λ ⋆ ( n , δ ) − 1 ≤ c F , where c : = C − 1 ( π 2 ) 4 ( M 2 ) 4 / p b 3 . \lambda_\star(n,\delta)^{-1}\le cF,\qquad\text{where}\qquad c:=C^{-1}\left(\frac{\pi}{2}\right)^4\left(\frac{M}{2}\right)^{4/p}b^3. λ ⋆ ( n , δ ) − 1 ≤ c F , where c := C − 1 ( 2 π ) 4 ( 2 M ) 4/ p b 3 .
Combining both cases yields λ ⋆ ( n , δ ) − 1 ≤ max ( κ − 1 , c ) F ≤ ( κ − 1 + c ) F \lambda_\star(n,\delta)^{-1}\le\max(\kappa^{-1},c)F\le(\kappa^{-1}+c)F λ ⋆ ( n , δ ) − 1 ≤ max ( κ − 1 , c ) F ≤ ( κ − 1 + c ) F . Setting C ′ : = κ − 1 + c C':=\kappa^{-1}+c C ′ := κ − 1 + c concludes the proof, since C C C is an absolute constant, κ \kappa κ depends on A A A and p p p , and M M M , a 0 a_0 a 0 and b b b depend on A A A , M μ M_\mu M μ and p p p .
Eigenvalue decay rate of H N T K \mathcal{H}^{\mathrm{NTK}} H NTK
We now derive the eigenvalue decay rate of H N T K \mathcal{H}^{\mathrm{NTK}} H NTK in our setting in the sense of Definition E.11 . The eigenvalues of H N T K \mathcal{H}^{\mathrm{NTK}} H NTK are understood as those of its associated integral/covariance operator (see Appendix § 16 for more details). Throughout, λ i ( k ; E , ν ) \lambda_i(k;E,\nu) λ i ( k ; E , ν ) denotes the i i i -th largest eigenvalue of the integral operator of a kernel k k k on the space L 2 ( E , ν ) L^2(E,\nu) L 2 ( E , ν ) .
Definition E.11 (Eigenvalue decay rate). Let K : X × X → R K:\mathcal{X}\times\mathcal{X}\to\mathbb{R} K : X × X → R be a continuous kernel associated to an RKHS H \mathcal{H} H , and ( λ i ) i ≥ 1 (\lambda_i)_{i\ge1} ( λ i ) i ≥ 1 be the non-zero eigenvalues of its associated integral operator in non-increasing order. We say that H \mathcal{H} H has eigenvalue decay rate β > 1 \beta>1 β > 1 if there exist two constants k 1 , k 2 > 0 k_1,k_2>0 k 1 , k 2 > 0 such that k 1 i − β ≤ λ i ≤ k 2 i − β k_1i^{-\beta}\le\lambda_i\le k_2i^{-\beta} k 1 i − β ≤ λ i ≤ k 2 i − β , for any i ≥ 1 i\ge1 i ≥ 1 .
Lemma E.12 (Eigenvalue decay rate of H N T K \mathcal{H}^{\mathrm{NTK}} H NTK ). Under Assumptions a and d , the eigenvalue decay rate of the integral/covariance operator associated with K N T K K^{\mathrm{NTK}} K NTK in Equation (36) is p + 1 p \frac{p+1}{p} p p + 1 .
Proof. Let ϕ ( x ) : = x ~ / ∥ x ∥ ∈ S p \phi(x) := \tilde{x}/\lVert x\rVert\in\mathbb{S}^{p} ϕ ( x ) := x ~ / ∥ x ∥ ∈ S p , ρ ( x ) : = ∥ x ∥ / 2 \rho(x) := \lVert x\rVert/\sqrt{2} ρ ( x ) := ∥ x ∥ / 2 and K 0 ( y , y ′ ) : = κ ( ⟨ y , y ′ ⟩ ) K_{0}(y,y') := \kappa(\langle y,y'\rangle) K 0 ( y , y ′ ) := κ (⟨ y , y ′ ⟩) with y , y ′ ∈ S p y,y' \in\mathbb{S}^{p} y , y ′ ∈ S p , where, for any u ∈ [ − 1 , 1 ] u \in[-1,1] u ∈ [ − 1 , 1 ] , κ ( u ) = u κ 0 ( u ) + κ 1 ( u ) \kappa(u) = u\kappa_{0}(u) + \kappa_{1}(u) κ ( u ) = u κ 0 ( u ) + κ 1 ( u ) with κ 0 ( u ) : = 1 π ( π − arccos ( u ) ) \kappa_{0}(u) := \frac{1}{\pi}(\pi-\arccos(u)) κ 0 ( u ) := π 1 ( π − arccos ( u )) and κ 1 ( u ) : = 1 π ( u ( π − arccos ( u ) ) + 1 − u 2 ) \kappa_{1}(u) := \frac{1}{\pi}(u(\pi-\arccos(u))+\sqrt{1-u^{2}}) κ 1 ( u ) := π 1 ( u ( π − arccos ( u )) + 1 − u 2 ) . Thus, we can write K N T K = K 1 + 1 K^{\mathrm{NTK}} = K_{1} + 1 K NTK = K 1 + 1 , where K 1 ( x , x ′ ) : = ρ ( x ) K 0 ( ϕ ( x ) , ϕ ( x ′ ) ) ρ ( x ′ ) K_{1}(x,x') := \rho(x)K_{0}(\phi(x),\phi(x'))\rho(x') K 1 ( x , x ′ ) := ρ ( x ) K 0 ( ϕ ( x ) , ϕ ( x ′ )) ρ ( x ′ ) . Let ν ~ : = ϕ # ( ρ 2 μ ) \tilde{\nu} := \phi_{\#}(\rho^{2}\mu) ν ~ := ϕ # ( ρ 2 μ ) be the push-forward measure.
Upper bound. Since ϕ \phi ϕ is injective, [62 Propositions 1 and 3] give:
λ i ( K 1 ; X , μ ) = λ i ( K 0 ∘ ( ϕ × ϕ ) ; X , ρ 2 μ ) = λ i ( K 0 ; S p , ν ~ ) . \lambda_i(K_1;\mathcal{X},\mu)=\lambda_i(K_0\circ(\phi\times\phi);\mathcal{X},\rho^2\mu)=\lambda_i(K_0;\mathbb{S}^p,\tilde{\nu}). λ i ( K 1 ; X , μ ) = λ i ( K 0 ∘ ( ϕ × ϕ ) ; X , ρ 2 μ ) = λ i ( K 0 ; S p , ν ~ ) .
By the change of variables computed in the proof of Lemma E.8 , ν ~ \tilde{\nu} ν ~ has the density q ( y ) = 1 2 ( ∥ x ∥ 2 + 1 ) ( p + 3 ) / 2 f μ ( x ) q(y)=\frac{1}{2}(\lVert x\rVert^2+1)^{(p+3)/2}f_\mu(x) q ( y ) = 2 1 (∥ x ∥ 2 + 1 ) ( p + 3 ) /2 f μ ( x ) , with x = ϕ − 1 ( y ) x=\phi^{-1}(y) x = ϕ − 1 ( y ) , with respect to the surface measure σ \sigma σ on S p \mathbb{S}^p S p , and q = 0 q=0 q = 0 outside ϕ ( X ) \phi(\mathcal{X}) ϕ ( X ) . By Assumptions a and d, q ≤ C q : = 1 2 ( A 2 + 1 ) ( p + 3 ) / 2 M μ q\leq C_q:=\frac{1}{2}(A^2+1)^{(p+3)/2}M_\mu q ≤ C q := 2 1 ( A 2 + 1 ) ( p + 3 ) /2 M μ . Hence [62 Lemma 2] gives λ i ( K 0 ; S p , ν ~ ) ≤ C q λ i ( K 0 ; S p , σ ) \lambda_i(K_0;\mathbb{S}^p,\tilde{\nu})\leq C_q\lambda_i(K_0;\mathbb{S}^p,\sigma) λ i ( K 0 ; S p , ν ~ ) ≤ C q λ i ( K 0 ; S p , σ ) . By [45 Lemma B.1] , λ i ( K 0 ; S p , σ ) ≤ C i − ( p + 1 ) / p \lambda_i(K_0;\mathbb{S}^p,\sigma)\leq Ci^{-(p+1)/p} λ i ( K 0 ; S p , σ ) ≤ C i − ( p + 1 ) / p for all i ≥ 1 i\geq1 i ≥ 1 , where C > 0 C>0 C > 0 . Finally, the constant kernel 1 1 1 has rank one, so Weyl’s inequality gives λ i + 1 ( K N T K ; X , μ ) ≤ λ i ( K 1 ; X , μ ) \lambda_{i+1}(K^{\mathrm{NTK}};\mathcal{X},\mu)\leq\lambda_i(K_1;\mathcal{X},\mu) λ i + 1 ( K NTK ; X , μ ) ≤ λ i ( K 1 ; X , μ ) for all i ≥ 1 i\geq1 i ≥ 1 . Together with λ 1 ( K N T K ; X , μ ) ≤ ∫ X K N T K ( x , x ) d μ ( x ) ≤ κ N T K \lambda_1(K^{\mathrm{NTK}};\mathcal{X},\mu)\leq\int_{\mathcal{X}}K^{\mathrm{NTK}}(x,x)d\mu(x)\leq\kappa^{\mathrm{NTK}} λ 1 ( K NTK ; X , μ ) ≤ ∫ X K NTK ( x , x ) d μ ( x ) ≤ κ NTK (Lemma E.2), this yields λ i ( K N T K ; X , μ ) ≤ k 2 i − ( p + 1 ) / p \lambda_i(K^{\mathrm{NTK}};\mathcal{X},\mu)\leq k_2i^{-(p+1)/p} λ i ( K NTK ; X , μ ) ≤ k 2 i − ( p + 1 ) / p for all i ≥ 1 i\geq1 i ≥ 1 , with k 2 : = max ( κ N T K , 2 ( p + 1 ) / p C q C ) k_2:=\max(\kappa^{\mathrm{NTK}},2^{(p+1)/p}C_qC) k 2 := max ( κ NTK , 2 ( p + 1 ) / p C q C ) .
Lower bound. By Assumption d, f μ ≥ c μ f_\mu\geq c_\mu f μ ≥ c μ on the open ball Ω μ ⊂ X \Omega_\mu\subset\mathcal{X} Ω μ ⊂ X . The integral operator of K N T K K^{\mathrm{NTK}} K NTK on L 2 ( Ω μ , μ ) L^2(\Omega_\mu,\mu) L 2 ( Ω μ , μ ) is the compression of the one on L 2 ( X , μ ) L^2(\mathcal{X},\mu) L 2 ( X , μ ) to functions vanishing outside Ω μ \Omega_\mu Ω μ , so the minimax principle [62 Lemma 49] gives λ i ( K N T K ; X , μ ) ≥ λ i ( K N T K ; Ω μ , μ ) \lambda_i(K^{\mathrm{NTK}};\mathcal{X},\mu)\geq\lambda_i(K^{\mathrm{NTK}};\Omega_\mu,\mu) λ i ( K NTK ; X , μ ) ≥ λ i ( K NTK ; Ω μ , μ ) . Since d μ / d x ≥ c μ d\mu/dx\geq c_\mu d μ / d x ≥ c μ on Ω μ \Omega_\mu Ω μ , [62 Lemma 2] gives λ i ( K N T K ; Ω μ , μ ) ≥ c μ λ i ( K N T K ; Ω μ , d x ) \lambda_i(K^{\mathrm{NTK}};\Omega_\mu,\mu)\geq c_\mu\lambda_i(K^{\mathrm{NTK}};\Omega_\mu,dx) λ i ( K NTK ; Ω μ , μ ) ≥ c μ λ i ( K NTK ; Ω μ , d x ) . The ball Ω μ \Omega_\mu Ω μ is a bounded Lipschitz domain, so Theorem G.12 shows that the RKHS of K N T K K^{\mathrm{NTK}} K NTK restricted to Ω μ \Omega_\mu Ω μ equals the Sobolev space H p + 1 2 ( Ω μ ) H^{\frac{p+1}{2}}(\Omega_\mu) H 2 p + 1 ( Ω μ ) , with equivalent norms. The eigenvalues λ i ( K N T K ; Ω μ , d x ) \lambda_i(K^{\mathrm{NTK}};\Omega_\mu,dx) λ i ( K NTK ; Ω μ , d x ) are the squares of the singular values of the embedding of this RKHS into L 2 ( Ω μ ) L^2(\Omega_\mu) L 2 ( Ω μ ) [87 ] , and equivalent norms change these singular values by at most constant factors. The singular values of the embedding H p + 1 2 ( Ω μ ) ↪ L 2 ( Ω μ ) H^{\frac{p+1}{2}}(\Omega_\mu)\hookrightarrow L^2(\Omega_\mu) H 2 p + 1 ( Ω μ ) ↪ L 2 ( Ω μ ) are of exact order i − p + 1 2 p i^{-\frac{p+1}{2p}} i − 2 p p + 1 [32 § 3.3.4] . Hence λ i ( K N T K ; Ω μ , d x ) ≥ c ′ i − ( p + 1 ) / p \lambda_i(K^{\mathrm{NTK}};\Omega_\mu,dx)\geq c'i^{-(p+1)/p} λ i ( K NTK ; Ω μ , d x ) ≥ c ′ i − ( p + 1 ) / p , and therefore λ i ( K N T K ; X , μ ) ≥ k 1 i − ( p + 1 ) / p \lambda_i(K^{\mathrm{NTK}};\mathcal{X},\mu)\geq k_1i^{-(p+1)/p} λ i ( K NTK ; X , μ ) ≥ k 1 i − ( p + 1 ) / p with k 1 : = c μ c ′ k_1:=c_\mu c' k 1 := c μ c ′ , where c ′ c' c ′ depends on p p p , A A A and the radius of Ω μ \Omega_\mu Ω μ .
Bounds on the training dynamics
For t ≥ 0 t\geq0 t ≥ 0 , we recall that f t N N f_t^{\mathrm{NN}} f t NN and f t N T K f_t^{\mathrm{NTK}} f t NTK are the solutions of the gradient flows provided in Equations (33) and (35) , respectively.
Lemma E.13 (Training residual bounds). Consider n n n i.i.d. samples { ( x i , y i ) } i = 1 n ∼ ρ \{(x_i,y_i)\}_{i=1}^{n}\sim\rho {( x i , y i ) } i = 1 n ∼ ρ . For any t ≥ 0 t\geq0 t ≥ 0 , let:
r N T K ( t ) = f t N T K ( X ) − Y = [ f t N T K ( x 1 ) − y 1 , … , f t N T K ( x n ) − y n ] ⊤ ∈ R n , r^{\mathrm{NTK}}(t)=f_t^{\mathrm{NTK}}(X)-Y=[f_t^{\mathrm{NTK}}(x_1)-y_1,\ldots,f_t^{\mathrm{NTK}}(x_n)-y_n]^\top\in\mathbb{R}^{n}, r NTK ( t ) = f t NTK ( X ) − Y = [ f t NTK ( x 1 ) − y 1 , … , f t NTK ( x n ) − y n ] ⊤ ∈ R n , r N N ( t ) = f t N N ( X ) − Y = [ f t N N ( x 1 ) − y 1 , … , f t N N ( x n ) − y n ] ⊤ ∈ R n . r^{\mathrm{NN}}(t)=f_t^{\mathrm{NN}}(X)-Y=[f_t^{\mathrm{NN}}(x_1)-y_1,\ldots,f_t^{\mathrm{NN}}(x_n)-y_n]^\top\in\mathbb{R}^{n}. r NN ( t ) = f t NN ( X ) − Y = [ f t NN ( x 1 ) − y 1 , … , f t NN ( x n ) − y n ] ⊤ ∈ R n . Under Assumption b, it holds that:
∥ r N T K ( t ) ∥ ≤ exp ( − λ 0 n t ) n B , \lVert r^{\mathrm{NTK}}(t)\rVert\leq\exp\left(-\frac{\lambda_0}{n}t\right)\sqrt{n}B, ∥ r NTK ( t )∥ ≤ exp ( − n λ 0 t ) n B , ∥ r N N ( t ) − r N T K ( t ) ∥ ≤ Δ t exp ( ( Δ − λ 0 n ) t ) n B , \lVert r^{\mathrm{NN}}(t)-r^{\mathrm{NTK}}(t)\rVert\leq\Delta t\exp\left(\left(\Delta-\frac{\lambda_0}{n}\right)t\right)\sqrt{n}B, ∥ r NN ( t ) − r NTK ( t )∥ ≤ Δ t exp ( ( Δ − n λ 0 ) t ) n B , where λ 0 \lambda_0 λ 0 is the smallest eigenvalue of the infinite-width NTK matrix K N T K K^{\mathrm{NTK}} K NTK and
Δ = sup t ≥ 0 sup ∥ x ∥ ≤ A , ∥ x ′ ∥ ≤ A ∣ K t m ( x , x ′ ) − K N T K ( x , x ′ ) ∣ . \Delta=\sup_{t\geq0}\sup_{\lVert x\rVert\leq A,\lVert x'\rVert\leq A}\left|K_t^m(x,x')-K^{\mathrm{NTK}}(x,x')\right|. Δ = t ≥ 0 sup ∥ x ∥ ≤ A , ∥ x ′ ∥ ≤ A sup K t m ( x , x ′ ) − K NTK ( x , x ′ ) .
Remark E.14. The quantity Δ \Delta Δ is random, as it depends on both the random initialization of the network and the random training samples. Lemma E.18 shows that, for sufficiently large width m m m , it is finite with high probability, and all subsequent results involving Δ \Delta Δ are understood to hold on this high-probability event.
Proof of Lemma E.13. Let t ≥ 0 t \ge0 t ≥ 0 . We recall that the gradient flows of f t N N f_t^{\mathrm{NN}} f t NN and f t N T K f_t^{\mathrm{NTK}} f t NTK evaluated at the samples { ( x i , y i ) } i = 1 n \{(x_i,y_i)\}_{i=1}^n {( x i , y i ) } i = 1 n are:
d d t r N N ( t ) = − 1 n K t m r N N ( t ) , (46) \frac{\mathrm{d}}{\mathrm{d}t}r^{\mathrm{NN}}(t)=-\frac{1}{n}K_t^m r^{\mathrm{NN}}(t), \tag*{(46)} d t d r NN ( t ) = − n 1 K t m r NN ( t ) , (46)
d d t r N T K ( t ) = − 1 n K N T K r N T K ( t ) , (47) \frac{\mathrm{d}}{\mathrm{d}t}r^{\mathrm{NTK}}(t)=-\frac{1}{n}K^{\mathrm{NTK}}r^{\mathrm{NTK}}(t), \tag*{(47)} d t d r NTK ( t ) = − n 1 K NTK r NTK ( t ) , (47)
with r N N ( 0 ) = r N T K ( 0 ) = − Y r^{\mathrm{NN}}(0)=r^{\mathrm{NTK}}(0)=-Y r NN ( 0 ) = r NTK ( 0 ) = − Y .
Upper-bound on ∥ r N T K ( t ) ∥ \lVert r^{\mathrm{NTK}}(t)\rVert ∥ r NTK ( t )∥ . A closed-form expression of the solution to the ordinary differential equation in Equation (47) is:
r N T K ( t ) = exp ( − 1 n t K N T K ) r N T K ( 0 ) = − exp ( − 1 n t K N T K ) Y . r^{\mathrm{NTK}}(t)=\exp\left(-\frac{1}{n}tK^{\mathrm{NTK}}\right)r^{\mathrm{NTK}}(0)=-\exp\left(-\frac{1}{n}tK^{\mathrm{NTK}}\right)Y. r NTK ( t ) = exp ( − n 1 t K NTK ) r NTK ( 0 ) = − exp ( − n 1 t K NTK ) Y .
where the second equality holds because f 0 N T K = 0 f_0^{\mathrm{NTK}}=0 f 0 NTK = 0 . Thus, under Assumption b, we get:
∥ r N T K ( t ) ∥ ≤ ∥ exp ( − 1 n t K N T K ) ∥ o p ∥ Y ∥ = exp ( − λ 0 n t ) ∥ Y ∥ ≤ exp ( − λ 0 n t ) n B , (48) \lVert r^{\mathrm{NTK}}(t)\rVert\leq\left\lVert\exp\left(-\frac{1}{n}tK^{\mathrm{NTK}}\right)\right\rVert_{\mathrm{op}}\lVert Y\rVert=\exp\left(-\frac{\lambda_0}{n}t\right)\lVert Y\rVert\leq\exp\left(-\frac{\lambda_0}{n}t\right)\sqrt{nB}, \tag*{(48)} ∥ r NTK ( t )∥ ≤ exp ( − n 1 t K NTK ) op ∥ Y ∥ = exp ( − n λ 0 t ) ∥ Y ∥ ≤ exp ( − n λ 0 t ) n B , (48)
where we use Lemma I.5 to find the operator norm of the matrix.
Upper-bound on ∥ r N N ( t ) − r N T K ( t ) ∥ \lVert r^{\mathrm{NN}}(t)-r^{\mathrm{NTK}}(t)\rVert ∥ r NN ( t ) − r NTK ( t )∥ . Define the error e ( t ) = r N N ( t ) − r N T K ( t ) e(t)=r^{\mathrm{NN}}(t)-r^{\mathrm{NTK}}(t) e ( t ) = r NN ( t ) − r NTK ( t ) . Using the gradient flow expressions, we obtain:
d d t e ( t ) = d d t r N N ( t ) − d d t r N T K ( t ) = − 1 n K t m r N N ( t ) + 1 n K N T K r N T K ( t ) . \frac{\mathrm{d}}{\mathrm{d}t}e(t)=\frac{\mathrm{d}}{\mathrm{d}t}r^{\mathrm{NN}}(t)-\frac{\mathrm{d}}{\mathrm{d}t}r^{\mathrm{NTK}}(t)=-\frac{1}{n}K_t^m r^{\mathrm{NN}}(t)+\frac{1}{n}K^{\mathrm{NTK}}r^{\mathrm{NTK}}(t). d t d e ( t ) = d t d r NN ( t ) − d t d r NTK ( t ) = − n 1 K t m r NN ( t ) + n 1 K NTK r NTK ( t ) .
Adding and subtracting 1 n K N T K r N N ( t ) \frac{1}{n}K^{\mathrm{NTK}}r^{\mathrm{NN}}(t) n 1 K NTK r NN ( t ) leads to:
d d t e ( t ) = − 1 n ( K t m − K N T K ) r N N ( t ) − 1 n K N T K ( r N N ( t ) − r N T K ( t ) ) . \frac{\mathrm{d}}{\mathrm{d}t}e(t)=-\frac{1}{n}\left(K_t^m-K^{\mathrm{NTK}}\right)r^{\mathrm{NN}}(t)-\frac{1}{n}K^{\mathrm{NTK}}\left(r^{\mathrm{NN}}(t)-r^{\mathrm{NTK}}(t)\right). d t d e ( t ) = − n 1 ( K t m − K NTK ) r NN ( t ) − n 1 K NTK ( r NN ( t ) − r NTK ( t ) ) .
This is a linear ordinary differential equation with initialization e ( 0 ) = 0 e(0)=0 e ( 0 ) = 0 . Applying Duhamel’s principle, i.e. , the variation of constants method, we get:
e ( t ) = ∫ 0 t exp ( − 1 n ( t − s ) K N T K ) ( − 1 n ( K s m − K N T K ) r N N ( s ) ) d s . e(t)=\int_0^t\exp\left(-\frac{1}{n}(t-s)K^{\mathrm{NTK}}\right)\left(-\frac{1}{n}\left(K_s^m-K^{\mathrm{NTK}}\right)r^{\mathrm{NN}}(s)\right)\mathrm{d}s. e ( t ) = ∫ 0 t exp ( − n 1 ( t − s ) K NTK ) ( − n 1 ( K s m − K NTK ) r NN ( s ) ) d s .
Taking norms and applying the triangle inequality together with Lemma I.5 gives:
∥ e ( t ) ∥ ≤ 1 n ∫ 0 t ∥ exp ( − 1 n ( t − s ) K N T K ) ∥ o p ∥ K s m − K N T K ∥ o p ∥ r N N ( s ) ∥ d s = 1 n ∫ 0 t exp ( − 1 n λ 0 ( t − s ) ) ∥ K s m − K N T K ∥ o p ∥ r N N ( s ) ∥ d s . \begin{aligned}
\lVert e(t)\rVert\leq\frac{1}{n}\int_0^t\left\lVert\exp\left(-\frac{1}{n}(t-s)K^{\mathrm{NTK}}\right)\right\rVert_{\mathrm{op}}\left\lVert K_s^m-K^{\mathrm{NTK}}\right\rVert_{\mathrm{op}}\lVert r^{\mathrm{NN}}(s)\rVert\mathrm{d}s \\
&=\frac{1}{n}\int_0^t\exp\left(-\frac{1}{n}\lambda_0(t-s)\right)\left\lVert K_s^m-K^{\mathrm{NTK}}\right\rVert_{\mathrm{op}}\lVert r^{\mathrm{NN}}(s)\rVert\mathrm{d}s.
\end{aligned} ∥ e ( t )∥ ≤ n 1 ∫ 0 t exp ( − n 1 ( t − s ) K NTK ) op K s m − K NTK op ∥ r NN ( s )∥ d s = n 1 ∫ 0 t exp ( − n 1 λ 0 ( t − s ) ) K s m − K NTK op ∥ r NN ( s )∥ d s .
Noticing by the triangle inequality that ∥ r N N ( s ) ∥ ≤ ∥ e ( s ) ∥ + ∥ r N T K ( s ) ∥ \lVert r^{\mathrm{NN}}(s)\rVert\leq\lVert e(s)\rVert+\lVert r^{\mathrm{NTK}}(s)\rVert ∥ r NN ( s )∥ ≤ ∥ e ( s )∥ + ∥ r NTK ( s )∥ yields:
∥ e ( t ) ∥ ≤ 1 n ∫ 0 t exp ( − 1 n λ 0 ( t − s ) ) ∥ K s m − K N T K ∥ o p ∥ r N T K ( s ) ∥ d s + 1 n ∫ 0 t exp ( − 1 n λ 0 ( t − s ) ) ∥ K s m − K N T K ∥ o p ∥ e ( s ) ∥ d s . \lVert e(t)\rVert\leq\frac{1}{n}\int_0^t\exp\left(-\frac{1}{n}\lambda_0(t-s)\right)\left\lVert K_s^m-K^{\mathrm{NTK}}\right\rVert_{\mathrm{op}}\lVert r^{\mathrm{NTK}}(s)\rVert\mathrm{d}s
+\frac{1}{n}\int_0^t\exp\left(-\frac{1}{n}\lambda_0(t-s)\right)\left\lVert K_s^m-K^{\mathrm{NTK}}\right\rVert_{\mathrm{op}}\lVert e(s)\rVert\mathrm{d}s. ∥ e ( t )∥ ≤ n 1 ∫ 0 t exp ( − n 1 λ 0 ( t − s ) ) K s m − K NTK op ∥ r NTK ( s )∥ d s + n 1 ∫ 0 t exp ( − n 1 λ 0 ( t − s ) ) K s m − K NTK op ∥ e ( s )∥ d s .
Multiplying both sides by exp ( 1 n λ 0 t ) \exp\left(\frac{1}{n}\lambda_0t\right) exp ( n 1 λ 0 t ) gives f ( t ) ≤ g ( t ) + ∫ 0 t h ( s ) f ( s ) d s f(t)\leq g(t)+\int_0^t h(s)f(s)\mathrm{d}s f ( t ) ≤ g ( t ) + ∫ 0 t h ( s ) f ( s ) d s , where:
f ( t ) = exp ( 1 n λ 0 t ) ∥ e ( t ) ∥ , f(t)=\exp\left(\frac{1}{n}\lambda_0t\right)\lVert e(t)\rVert, f ( t ) = exp ( n 1 λ 0 t ) ∥ e ( t )∥ ,
g ( t ) = 1 n ∫ 0 t exp ( 1 n λ 0 s ) ∥ K s m − K N T K ∥ o p ∥ r N T K ( s ) ∥ d s , g(t)=\frac{1}{n}\int_0^t\exp\left(\frac{1}{n}\lambda_0s\right)\left\lVert K_s^m-K^{\mathrm{NTK}}\right\rVert_{\mathrm{op}}\lVert r^{\mathrm{NTK}}(s)\rVert\mathrm{d}s, g ( t ) = n 1 ∫ 0 t exp ( n 1 λ 0 s ) K s m − K NTK op ∥ r NTK ( s )∥ d s ,
h ( t ) = 1 n ∥ K t m − K N T K ∥ o p . h(t)=\frac{1}{n}\left\lVert K_t^m-K^{\mathrm{NTK}}\right\rVert_{\mathrm{op}}. h ( t ) = n 1 K t m − K NTK op .
We notice that t ↦ h ( t ) t \mapsto h(t) t ↦ h ( t ) is non-negative and t ↦ g ( t ) t \mapsto g(t) t ↦ g ( t ) is non-decreasing. Hence, it follows from Grönwall’s inequality that:
f ( t ) ≤ g ( t ) exp ( ∫ 0 t h ( s ) d s ) . f(t) \leq g(t) \exp\left(\int_{0}^{t} h(s)\mathrm{d}s\right). f ( t ) ≤ g ( t ) exp ( ∫ 0 t h ( s ) d s ) .
Plugging in the definitions of f ( t ) , g ( t ) f(t),g(t) f ( t ) , g ( t ) , and h ( t ) h(t) h ( t ) , then dividing by exp ( 1 n λ 0 t ) \exp\left(\frac{1}{n}\lambda_{0}t\right) exp ( n 1 λ 0 t ) , we get:
∥ e ( t ) ∥ ≤ 1 n ∫ 0 t exp ( − 1 n λ 0 ( t − s ) ) ∥ K s m − K N T K ∥ o p ∥ r N T K ( s ) ∥ d s \lVert e(t)\rVert\leq\frac{1}{n}\int_{0}^{t}\exp\left(-\frac{1}{n}\lambda_{0}(t-s)\right)\left\lVert\boldsymbol{K}_{s}^{m}-\boldsymbol{K}^{\mathrm{NTK}}\right\rVert_{\mathrm{op}}\left\lVert r^{\mathrm{NTK}}(s)\right\rVert\mathrm{d}s ∥ e ( t )∥ ≤ n 1 ∫ 0 t exp ( − n 1 λ 0 ( t − s ) ) K s m − K NTK op r NTK ( s ) d s
× exp ( ∫ 0 t 1 n ∥ K s m − K N T K ∥ o p d s ) . \qquad\times\exp\left(\int_{0}^{t}\frac{1}{n}\left\lVert\boldsymbol{K}_{s}^{m}-\boldsymbol{K}^{\mathrm{NTK}}\right\rVert_{\mathrm{op}}\mathrm{d}s\right). × exp ( ∫ 0 t n 1 K s m − K NTK op d s ) .
Let
Δ = sup t ≥ 0 sup ∥ x ∥ ≤ A , ∥ x ′ ∥ ≤ A ∣ K t m ( x , x ′ ) − K N T K ( x , x ′ ) ∣ . \Delta= \sup_{t \geq0}\sup_{\lVert x\rVert\leq A,\lVert x'\rVert\leq A}\left\lvert K_{t}^{m}(x,x')-K^{\mathrm{NTK}}(x,x')\right\rvert. Δ = t ≥ 0 sup ∥ x ∥ ≤ A , ∥ x ′ ∥ ≤ A sup K t m ( x , x ′ ) − K NTK ( x , x ′ ) .
Using Equation (48) to bound ∥ r N T K ( s ) ∥ \lVert r^{\mathrm{NTK}}(s)\rVert ∥ r NTK ( s )∥ and the fact that ∥ K m − K N T K ∥ o p ≤ n Δ \left\lVert\boldsymbol{K}^{m}-\boldsymbol{K}^{\mathrm{NTK}}\right\rVert_{\mathrm{op}}\leq n\Delta K m − K NTK op ≤ n Δ , we obtain:
∥ e ( t ) ∥ ≤ Δ t exp ( ( Δ − λ 0 n ) t ) n B . \lVert e(t)\rVert\leq\Delta t\exp\left(\left(\Delta-\frac{\lambda_{0}}{n}\right)t\right)\sqrt{n}B. ∥ e ( t )∥ ≤ Δ t exp ( ( Δ − n λ 0 ) t ) n B .
Lemma E.15 (Bound on ∥ f t N T K ∥ L 2 ( X ) \lVert f_{t}^{\mathrm{NTK}}\rVert_{L^{2}(\mathcal{X})} ∥ f t NTK ∥ L 2 ( X ) ). Let t ≥ 0 t \geq0 t ≥ 0 . Under Assumptions a and b , it holds that:
∥ f t N T K ∥ L 2 ( X ) ≤ κ N T K B t , \lVert f_{t}^{\mathrm{NTK}}\rVert_{L^{2}(\mathcal{X})}\leq\kappa^{\mathrm{NTK}}Bt, ∥ f t NTK ∥ L 2 ( X ) ≤ κ NTK B t , where κ N T K : = ( p + 2 ) ( A 2 + 1 ) + 1 \kappa^{\mathrm{NTK}} := (p+2)(A^{2}+1)+1 κ NTK := ( p + 2 ) ( A 2 + 1 ) + 1 .
Proof. Let x ∈ X x\in\mathcal{X} x ∈ X and t ≥ 0 t\geq0 t ≥ 0 . We recall that the gradient flow of f t N T K f_{t}^{\mathrm{NTK}} f t NTK is the following:
d d t f t N T K ( x ) = − 1 n ∑ i = 1 n K N T K ( x , x i ) r i N T K ( t ) , (49) \frac{\mathrm{d}}{\mathrm{d}t}f_{t}^{\mathrm{NTK}}(x)=-\frac{1}{n}\sum_{i=1}^{n}K^{\mathrm{NTK}}(x,x_{i})r_{i}^{\mathrm{NTK}}(t), \tag*{(49)} d t d f t NTK ( x ) = − n 1 i = 1 ∑ n K NTK ( x , x i ) r i NTK ( t ) , (49) where r i N T K ( t ) = f t N T K ( x i ) − y i r_{i}^{\mathrm{NTK}}(t)=f_{t}^{\mathrm{NTK}}(x_{i})-y_{i} r i NTK ( t ) = f t NTK ( x i ) − y i . Using the fact that f 0 N T K = 0 f_{0}^{\mathrm{NTK}}=0 f 0 NTK = 0 and the triangle inequality, we obtain:
∣ f t N T K ( x ) ∣ = ∣ f t N T K ( x ) − f 0 N T K ( x ) ∣ = ∣ ∫ 0 t d d s f s N T K ( x ) d s ∣ ≤ ∫ 0 t ∣ d d s f s N T K ( x ) ∣ d s . (50) \left\lvert f_{t}^{\mathrm{NTK}}(x)\right\rvert=\left\lvert f_{t}^{\mathrm{NTK}}(x)-f_{0}^{\mathrm{NTK}}(x)\right\rvert=\left\lvert\int_{0}^{t}\frac{\mathrm{d}}{\mathrm{d}s}f_{s}^{\mathrm{NTK}}(x)\mathrm{d}s\right\rvert\leq\int_{0}^{t}\left\lvert\frac{\mathrm{d}}{\mathrm{d}s}f_{s}^{\mathrm{NTK}}(x)\right\rvert\mathrm{d}s. \tag*{(50)} f t NTK ( x ) = f t NTK ( x ) − f 0 NTK ( x ) = ∫ 0 t d s d f s NTK ( x ) d s ≤ ∫ 0 t d s d f s NTK ( x ) d s . (50) Next, we find an upper-bound on ∣ d d s f s N T K ( x ) ∣ \left\lvert\frac{\mathrm{d}}{\mathrm{d}s}f_{s}^{\mathrm{NTK}}(x)\right\rvert d s d f s NTK ( x ) . We have:
∣ d d s f s N T K ( x ) ∣ = 1 n ∣ ∑ i = 1 n K N T K ( x , x i ) r i N T K ( s ) ∣ ≤ κ N T K n ∑ i = 1 n ∣ r i N T K ( s ) ∣ , \left\lvert\frac{\mathrm{d}}{\mathrm{d}s}f_{s}^{\mathrm{NTK}}(x)\right\rvert=\frac{1}{n}\left\lvert\sum_{i=1}^{n}K^{\mathrm{NTK}}(x,x_{i})r_{i}^{\mathrm{NTK}}(s)\right\rvert\leq\frac{\kappa^{\mathrm{NTK}}}{n}\sum_{i=1}^{n}\left\lvert r_{i}^{\mathrm{NTK}}(s)\right\rvert, d s d f s NTK ( x ) = n 1 i = 1 ∑ n K NTK ( x , x i ) r i NTK ( s ) ≤ n κ NTK i = 1 ∑ n r i NTK ( s ) , where we use Lemma E.2 to bound K N T K ( x , x i ) K^{\mathrm{NTK}}(x,x_{i}) K NTK ( x , x i ) by κ N T K : = ( p + 2 ) ( A 2 + 1 ) + 1 \kappa^{\mathrm{NTK}} := (p+2)(A^{2}+1)+1 κ NTK := ( p + 2 ) ( A 2 + 1 ) + 1 in the inequality. Using the notation r N T K ( s ) = [ r 1 N T K ( s ) , … , r n N T K ( s ) ] T ∈ R n r^{\mathrm{NTK}}(s)=[r_{1}^{\mathrm{NTK}}(s),\ldots,r_{n}^{\mathrm{NTK}}(s)]^{\mathsf{T}}\in\mathbb{R}^{n} r NTK ( s ) = [ r 1 NTK ( s ) , … , r n NTK ( s ) ] T ∈ R n , we obtain by the Cauchy-Schwarz inequality:
∣ d d s f s N T K ( x ) ∣ ≤ κ N T K n ∥ r N T K ( s ) ∥ . (51) \left\lvert\frac{\mathrm{d}}{\mathrm{d}s}f_{s}^{\mathrm{NTK}}(x)\right\rvert\leq\frac{\kappa^{\mathrm{NTK}}}{\sqrt{n}}\left\lVert r^{\mathrm{NTK}}(s)\right\rVert. \tag*{(51)} d s d f s NTK ( x ) ≤ n κ NTK r NTK ( s ) . (51) By Lemma E.13, we get:
∥ r N T K ( s ) ∥ ≤ exp ( − λ 0 n s ) n B , \left\lVert r^{\mathrm{NTK}}(s)\right\rVert\leq\exp\left(-\frac{\lambda_{0}}{n}s\right)\sqrt{n}B, r NTK ( s ) ≤ exp ( − n λ 0 s ) n B , where λ 0 \lambda_{0} λ 0 is the smallest eigenvalue of the matrix K N T K \boldsymbol{K}^{\mathrm{NTK}} K NTK . Substituting this bound into Equation (51) , we get:
∣ d d s f s N T K ( x ) ∣ ≤ κ N T K B exp ( − λ 0 n s ) . \left\lvert\frac{\mathrm{d}}{\mathrm{d}s}f_{s}^{\mathrm{NTK}}(x)\right\rvert\leq\kappa^{\mathrm{NTK}}B\exp\left(-\frac{\lambda_{0}}{n}s\right). d s d f s NTK ( x ) ≤ κ NTK B exp ( − n λ 0 s ) . Inserting this bound into Equation (50) gives:
∣ f t N T K ( x ) ∣ ≤ κ N T K B ∫ 0 t exp ( − λ 0 n s ) d s = κ N T K B n λ 0 ( 1 − exp ( − λ 0 n t ) ) ≤ κ N T K B t , \left\lvert f_{t}^{\mathrm{NTK}}(x)\right\rvert\leq\kappa^{\mathrm{NTK}}B\int_{0}^{t}\exp\left(-\frac{\lambda_{0}}{n}s\right)\mathrm{d}s=\frac{\kappa^{\mathrm{NTK}}Bn}{\lambda_{0}}\left(1-\exp\left(-\frac{\lambda_{0}}{n}t\right)\right)\leq\kappa^{\mathrm{NTK}}Bt, f t NTK ( x ) ≤ κ NTK B ∫ 0 t exp ( − n λ 0 s ) d s = λ 0 κ NTK B n ( 1 − exp ( − n λ 0 t ) ) ≤ κ NTK B t , where we use the identity 1 − exp ( − a ) ≤ a 1-\exp(-a)\leq a 1 − exp ( − a ) ≤ a for all a ≥ 0 a\geq0 a ≥ 0 . Since μ \mu μ is a probability measure, the claimed L 2 ( X ) L^{2}(\mathcal{X}) L 2 ( X ) bound follows.
Lemma E.16 (High-probability entrywise upper-bound on the parameters at initialization). Let θ 0 = ( v 0 , u 0 , b 0 , a 0 ) \theta_0=(v_0,u_0,b_0,a_0) θ 0 = ( v 0 , u 0 , b 0 , a 0 ) be the parameters at initialization, where u 0 , i ∼ N ( 0 , I p × p ) u_{0,i}\sim\mathcal{N}(0,I_{p\times p}) u 0 , i ∼ N ( 0 , I p × p ) , b 0 , i ∼ N ( 0 , 1 ) b_{0,i}\sim\mathcal{N}(0,1) b 0 , i ∼ N ( 0 , 1 ) , and v 0 , i ∼ N ( 0 , 1 ) v_{0,i}\sim\mathcal{N}(0,1) v 0 , i ∼ N ( 0 , 1 ) are drawn independently for i = 1 , … , m / 2 i=1,\ldots,m/2 i = 1 , … , m /2 , with mirrored copies u 0 , i + m / 2 = u 0 , i u_{0,i+m/2}=u_{0,i} u 0 , i + m /2 = u 0 , i , b 0 , i + m / 2 = b 0 , i b_{0,i+m/2}=b_{0,i} b 0 , i + m /2 = b 0 , i , v 0 , i + m / 2 = − v 0 , i v_{0,i+m/2}=-v_{0,i} v 0 , i + m /2 = − v 0 , i , and a 0 = 0 a_0=0 a 0 = 0 . Let m ≥ 2 m\ge2 m ≥ 2 . Then, the following holds with probability at least 1 − p + 2 2 π m − 1 / 2 1-\frac{p+2}{\sqrt{2\pi}}m^{-1/2} 1 − 2 π p + 2 m − 1/2 with respect to random initialization:
max 1 ≤ i ≤ m max ( ∣ v 0 , i ∣ , ∣ b 0 , i ∣ , ∥ u 0 , i ∥ ∞ ) ≤ 3 ln m . \max_{1\le i\le m}\max\left(\lvert v_{0,i}\rvert,\lvert b_{0,i}\rvert,\lVert u_{0,i}\rVert_\infty\right)\le\sqrt{3\ln m}. 1 ≤ i ≤ m max max ( ∣ v 0 , i ∣ , ∣ b 0 , i ∣ , ∥ u 0 , i ∥ ∞ ) ≤ 3 ln m . Proof. By the mirrored initialization, the parameters of neuron i + m / 2 i+m/2 i + m /2 coincide with those of neuron i i i , up to the sign of the outer weight. It therefore suffices to control the m ( p + 2 ) / 2 m(p+2)/2 m ( p + 2 ) /2 independent standard Gaussian variables ( u 0 , i ) j (u_{0,i})_j ( u 0 , i ) j , b 0 , i b_{0,i} b 0 , i , and v 0 , i v_{0,i} v 0 , i , for 1 ≤ i ≤ m / 2 1\le i\le m/2 1 ≤ i ≤ m /2 and 1 ≤ j ≤ p 1\le j\le p 1 ≤ j ≤ p . For Z ∼ N ( 0 , 1 ) Z\sim\mathcal{N}(0,1) Z ∼ N ( 0 , 1 ) and t > 0 t>0 t > 0 , the classical Gaussian tail bound gives Pr ( ∣ Z ∣ ≥ t ) ≤ 2 2 π t exp ( − t 2 / 2 ) \Pr(\lvert Z\rvert\ge t)\le\frac{2}{\sqrt{2\pi}t}\exp(-t^2/2) Pr (∣ Z ∣ ≥ t ) ≤ 2 π t 2 exp ( − t 2 /2 ) . Since m ≥ 2 m\ge2 m ≥ 2 , we have 3 ln m ≥ 3 ln 2 > 1 \sqrt{3\ln m}\ge\sqrt{3\ln2}>1 3 ln m ≥ 3 ln 2 > 1 , hence Pr ( ∣ Z ∣ ≥ 3 ln m ) ≤ 2 2 π exp ( − 3 ln m / 2 ) = 2 2 π m − 3 / 2 \Pr(\lvert Z\rvert\ge\sqrt{3\ln m})\le\frac{2}{\sqrt{2\pi}}\exp(-3\ln m/2)=\frac{2}{\sqrt{2\pi}}m^{-3/2} Pr (∣ Z ∣ ≥ 3 ln m ) ≤ 2 π 2 exp ( − 3 ln m /2 ) = 2 π 2 m − 3/2 . A union bound over the m ( p + 2 ) / 2 m(p+2)/2 m ( p + 2 ) /2 variables shows that the bound fails with probability at most m ( p + 2 ) 2 × 2 2 π m − 3 / 2 = p + 2 2 π m − 1 / 2 \frac{m(p+2)}{2}\times\frac{2}{\sqrt{2\pi}}m^{-3/2}=\frac{p+2}{\sqrt{2\pi}}m^{-1/2} 2 m ( p + 2 ) × 2 π 2 m − 3/2 = 2 π p + 2 m − 1/2 , which concludes the proof. □ \square □
Proposition E.17, borrowed from [62 ] , establishes that the empirical NTK of a sufficiently wide fully connected ReLU neural network with mirrored initialization converges uniformly to the infinite-width NTK, both over the entire training trajectory and over all inputs in a bounded domain. This is the key result that allows us to approximate the dynamics of the neural network by those of its NTK surrogate throughout training, which we use in Lemma E.18 .
Proposition E.17 (Kernel uniform convergence – Proposition 45 in [62 ] ). Consider n n n i.i.d. samples { ( x i , y i ) } i = 1 n \{(x_i,y_i)\}_{i=1}^n {( x i , y i ) } i = 1 n and their corresponding augmented inputs { x ~ i } i = 1 n \{\widetilde{x}_i\}_{i=1}^n { x i } i = 1 n , where x ~ i = [ x i , 1 ] ⊤ \widetilde{x}_i=[x_i,1]^\top x i = [ x i , 1 ] ⊤ for 1 ≤ i ≤ n 1\le i\le n 1 ≤ i ≤ n . Let M ~ X = ∑ i = 1 n ∥ x ~ i ∥ \widetilde{M}_X=\sum_{i=1}^n\lVert\widetilde{x}_i\rVert M X = ∑ i = 1 n ∥ x i ∥ , B r = { x ∈ X ∣ 1 ≤ ∥ x ~ ∥ ≤ r } B_r=\{x\in\mathcal{X}\mid1\le\lVert\widetilde{x}\rVert\le r\} B r = { x ∈ X ∣ 1 ≤ ∥ x ∥ ≤ r } for r ≥ 1 r\ge1 r ≥ 1 , and λ 0 \lambda_0 λ 0 be the smallest eigenvalue of the infinite-width NTK kernel matrix. There exist a constant C > 0 C>0 C > 0 and a polynomial poly \operatorname{poly} poly such that, for any δ ∈ ( 0 , 1 ) \delta\in(0,1) δ ∈ ( 0 , 1 ) and k > 0 k>0 k > 0 , when the width satisfies m ≥ poly ( n , M ~ X , λ 0 − 1 , ∣ y 1 ∣ , … , ∣ y n ∣ , ln ( 1 / δ ) , k ) m\ge\operatorname{poly}(n,\widetilde{M}_X,\lambda_0^{-1},\lvert y_1\rvert,\ldots,\lvert y_n\rvert,\ln(1/\delta),k) m ≥ poly ( n , M X , λ 0 − 1 , ∣ y 1 ∣ , … , ∣ y n ∣ , ln ( 1/ δ ) , k ) and m ≥ r k m\ge r^k m ≥ r k , with probability at least 1 − δ 1-\delta 1 − δ with respect to random samples and random initialization, we have:
sup t ≥ 0 sup x , x ′ ∈ B r ∣ K t m ( x , x ′ ) − K N T K ( x , x ′ ) ∣ ≤ C r 2 m − 1 / 12 ln m , \sup_{t\ge0}\sup_{x,x'\in B_r}\left\lvert K_t^m(x,x')-K^{\mathrm{NTK}}(x,x')\right\rvert\le Cr^2m^{-1/12}\sqrt{\ln m}, t ≥ 0 sup x , x ′ ∈ B r sup K t m ( x , x ′ ) − K NTK ( x , x ′ ) ≤ C r 2 m − 1/12 ln m , where the constant C > 0 C>0 C > 0 is independent of δ \delta δ and n n n .
Lemma E.18 (Kernel uniform convergence – Application of Proposition E.17 to our setting). Consider n ≥ 3 n\ge3 n ≥ 3 i.i.d. samples { ( x i , y i ) } i = 1 n ∼ ρ \{(x_i,y_i)\}_{i=1}^n\sim\rho {( x i , y i ) } i = 1 n ∼ ρ and fix p ≥ 2 p\ge2 p ≥ 2 . Under Assumptions a, b and d, there exist a constant C > 0 C>0 C > 0 and a polynomial poly \operatorname{poly} poly such that, for any δ ∈ ( 0 , 1 ) \delta\in(0,1) δ ∈ ( 0 , 1 ) , when the width satisfies m ≥ poly ( n , 1 / δ , ln n , ln ( 1 / δ ) ) m\ge\operatorname{poly}(n,1/\delta,\ln n,\ln(1/\delta)) m ≥ poly ( n , 1/ δ , ln n , ln ( 1/ δ )) , with probability at least 1 − δ 1-\delta 1 − δ with respect to random samples and random initialization, we have:
sup t ≥ 0 sup ∥ x ∥ ≤ A , ∥ x ′ ∥ ≤ A ∣ K t m ( x , x ′ ) − K N T K ( x , x ′ ) ∣ ≤ C m − 1 / 12 ln m , \sup_{t\ge0}\sup_{\lVert x\rVert\le A,\lVert x'\rVert\le A}\left\lvert K_t^m(x,x')-K^{\mathrm{NTK}}(x,x')\right\rvert\le Cm^{-1/12}\sqrt{\ln m}, t ≥ 0 sup ∥ x ∥ ≤ A , ∥ x ′ ∥ ≤ A sup K t m ( x , x ′ ) − K NTK ( x , x ′ ) ≤ C m − 1/12 ln m , where the constant C > 0 C>0 C > 0 is independent of δ \delta δ and n n n , but depends on A A A , and the coefficients of poly \operatorname{poly} poly depend on A A A , B B B , M μ M_\mu M μ , and p p p .
Proof. In this proof, we apply the statement of Proposition E.17 to our setting. Under Assumption a, we know that ∥ x ∥ ≤ A \lVert x\rVert\le A ∥ x ∥ ≤ A , for any x ∈ X x\in\mathcal{X} x ∈ X , which implies that M ~ X = ∑ i = 1 n ∥ x ~ i ∥ ≤ n A 2 + 1 \widetilde{M}_X=\sum_{i=1}^n\lVert\widetilde{x}_i\rVert\le n\sqrt{A^2+1} M X = ∑ i = 1 n ∥ x i ∥ ≤ n A 2 + 1 . Moreover, under Assumption b, we know that ∣ y i ∣ ≤ B \lvert y_i\rvert\le B ∣ y i ∣ ≤ B , for 1 ≤ i ≤ n 1\le i\le n 1 ≤ i ≤ n . By taking k = 1 k=1 k = 1 and r = A 2 + 1 r=\sqrt{A^2+1} r = A 2 + 1 in Proposition E.17, we require that m m m should satisfy m ≥ poly ( n , λ 0 − 1 , ln ( 1 / δ ) ) m\ge\operatorname{poly}(n,\lambda_0^{-1},\ln(1/\delta)) m ≥ poly ( n , λ 0 − 1 , ln ( 1/ δ )) and m ≥ A 2 + 1 m\ge\sqrt{A^2+1} m ≥ A 2 + 1 , where the coefficients of poly \operatorname{poly} poly depend on A A A and B B B . Hence, we get that m ≥ max ( poly ( n , λ 0 − 1 , ln ( 1 / δ ) ) , A 2 + 1 ) m\ge\max(\operatorname{poly}(n,\lambda_0^{-1},\ln(1/\delta)),\sqrt{A^2+1}) m ≥ max ( poly ( n , λ 0 − 1 , ln ( 1/ δ )) , A 2 + 1 ) . After redefining the polynomial poly \operatorname{poly} poly that dominates both lower bounds on m m m , we simplify the requirement to m ≥ poly ( n , λ 0 − 1 , ln ( 1 / δ ) ) m\ge\operatorname{poly}(n,\lambda_0^{-1},\ln(1/\delta)) m ≥ poly ( n , λ 0 − 1 , ln ( 1/ δ )) . By Corollary E.10, since n ≥ 3 n\ge3 n ≥ 3 , the following holds with probability at least 1 − δ 1-\delta 1 − δ with respect to random samples: λ 0 − 1 ≤ P ( n , 1 / δ , ln n , ln ( 1 / δ ) ) \lambda_0^{-1}\le P(n,1/\delta,\ln n,\ln(1/\delta)) λ 0 − 1 ≤ P ( n , 1/ δ , ln n , ln ( 1/ δ )) , where P ( n , 1 / δ , ln n , ln ( 1 / δ ) ) : = P(n,1/\delta,\ln n,\ln(1/\delta)):= P ( n , 1/ δ , ln n , ln ( 1/ δ )) :=
C ′ n 8 / p δ − 4 / p ( ln n + ln ( 1 / δ ) ) 3 C'n^{8/p}\delta^{-4/p}(\ln n+\ln(1/\delta))^3 C ′ n 8/ p δ − 4/ p ( ln n + ln ( 1/ δ ) ) 3 is a polynomial whose coefficients depend on A A A , M μ M_\mu M μ , and p p p . Consequently, it suffices to choose m ≥ poly ( n , P ( n , 1 / δ , ln n , ln ( 1 / δ ) ) , ln ( 1 / δ ) ) m \ge\operatorname{poly}(n, P(n,1/\delta,\ln n,\ln(1/\delta)),\ln(1/\delta)) m ≥ poly ( n , P ( n , 1/ δ , ln n , ln ( 1/ δ )) , ln ( 1/ δ )) . After redefining poly ( n , 1 / δ , ln n , ln ( 1 / δ ) ) \operatorname{poly}(n,1/\delta,\ln n,\ln(1/\delta)) poly ( n , 1/ δ , ln n , ln ( 1/ δ )) to be the polynomial poly ( n , P ( n , 1 / δ , ln n , ln ( 1 / δ ) ) , ln ( 1 / δ ) ) \operatorname{poly}(n,P(n,1/\delta,\ln n,\ln(1/\delta)),\ln(1/\delta)) poly ( n , P ( n , 1/ δ , ln n , ln ( 1/ δ )) , ln ( 1/ δ )) , we obtain that m ≥ poly ( n , 1 / δ , ln n , ln ( 1 / δ ) ) m \ge\operatorname{poly}(n,1/\delta,\ln n,\ln(1/\delta)) m ≥ poly ( n , 1/ δ , ln n , ln ( 1/ δ )) where the coefficients of poly \operatorname{poly} poly depend on A A A , B B B , M μ M_\mu M μ , and p p p . Since we want both events in Proposition E.17 and Corollary E.10 to hold simultaneously, then, using a union bound, they both hold with probability at least 1 − 2 δ 1-2\delta 1 − 2 δ . Redefining δ \delta δ to be 2 δ 2\delta 2 δ concludes the proof.
Lemma E.19 (Uniform NTK approximation of the neural network predictor). Consider n ≥ 3 n \ge3 n ≥ 3 i.i.d. samples { ( x i , y i ) } i = 1 n ∼ ρ \{(x_i,y_i)\}_{i=1}^n \sim\rho {( x i , y i ) } i = 1 n ∼ ρ and fix p ≥ 2 p \ge2 p ≥ 2 . Under Assumptions a a a , b b b and d d d , there exist a constant C > 0 C > 0 C > 0 and a polynomial poly \operatorname{poly} poly such that, for any δ ∈ ( 0 , 1 ) \delta\in(0,1) δ ∈ ( 0 , 1 ) , when the width satisfies m ≥ poly ( n , 1 / δ , ln n , ln ( 1 / δ ) ) m \ge\operatorname{poly}(n,1/\delta,\ln n,\ln(1/\delta)) m ≥ poly ( n , 1/ δ , ln n , ln ( 1/ δ )) , with probability at least 1 − δ 1-\delta 1 − δ with respect to random samples and random initialization, we have:
sup t ≥ 0 sup ∣ x ∣ ≤ A ∣ f t N N ( x ) − f t N T K ( x ) ∣ ≤ C m − 1 24 ln m , \sup_{t \ge0}\sup_{\lvert x\rvert\le A}\left|f_t^{\mathrm{NN}}(x)-f_t^{\mathrm{NTK}}(x)\right| \le Cm^{-\frac{1}{24}}\sqrt{\ln m}, t ≥ 0 sup ∣ x ∣ ≤ A sup f t NN ( x ) − f t NTK ( x ) ≤ C m − 24 1 ln m , where the constant C > 0 C > 0 C > 0 is independent of δ \delta δ and n n n , but depends on A A A , and the coefficients of poly \operatorname{poly} poly depend on A A A , B B B , M μ M_\mu M μ , and p p p .
Proof. We start by recalling that f 0 N N f_0^{\mathrm{NN}} f 0 NN and f 0 N T K f_0^{\mathrm{NTK}} f 0 NTK are assumed to be equal to 0. Using this, for any x ∈ X x \in\mathcal{X} x ∈ X , we have:
∣ f t N N ( x ) − f t N T K ( x ) ∣ = ∣ ∫ 0 t ( d d s f s N N ( x ) − d d s f s N T K ( x ) ) d s ∣ ≤ ∫ 0 t ∣ d d s f s N N ( x ) − d d s f s N T K ( x ) ∣ d s , \begin{aligned}
\left|f_t^{\mathrm{NN}}(x)-f_t^{\mathrm{NTK}}(x)\right|=\left|\int_0^t\left(\frac{\mathrm{d}}{\mathrm{d}s}f_s^{\mathrm{NN}}(x)-\frac{\mathrm{d}}{\mathrm{d}s}f_s^{\mathrm{NTK}}(x)\right)\mathrm{d}s\right| \\
&\le\int_0^t\left|\frac{\mathrm{d}}{\mathrm{d}s}f_s^{\mathrm{NN}}(x)-\frac{\mathrm{d}}{\mathrm{d}s}f_s^{\mathrm{NTK}}(x)\right|\mathrm{d}s,
\end{aligned} f t NN ( x ) − f t NTK ( x ) = ∫ 0 t ( d s d f s NN ( x ) − d s d f s NTK ( x ) ) d s ≤ ∫ 0 t d s d f s NN ( x ) − d s d f s NTK ( x ) d s ,
where we use the triangle inequality in the second line. We recall that the gradient flows of f s N N f_s^{\mathrm{NN}} f s NN and f s N T K f_s^{\mathrm{NTK}} f s NTK are the following:
d d s f s N N ( x ) = − 1 n K s m ( x , X ) T r N N ( s ) and d d s f s N T K ( x ) = − 1 n K N T K ( x , X ) T r N T K ( s ) , \frac{\mathrm{d}}{\mathrm{d}s}f_s^{\mathrm{NN}}(x)=-\frac{1}{n}K_s^m(x,\mathbf{X})^\mathsf{T}r^{\mathrm{NN}}(s)\quad\text{and}\quad\frac{\mathrm{d}}{\mathrm{d}s}f_s^{\mathrm{NTK}}(x)=-\frac{1}{n}K^{\mathrm{NTK}}(x,\mathbf{X})^\mathsf{T}r^{\mathrm{NTK}}(s), d s d f s NN ( x ) = − n 1 K s m ( x , X ) T r NN ( s ) and d s d f s NTK ( x ) = − n 1 K NTK ( x , X ) T r NTK ( s ) ,
where:
K s m ( x , X ) = [ K s m ( x , x 1 ) , … , K s m ( x , x n ) ] T ∈ R n , K_s^m(x,\mathbf{X})=[K_s^m(x,x_1),\ldots,K_s^m(x,x_n)]^\mathsf{T}\in\mathbb{R}^n, K s m ( x , X ) = [ K s m ( x , x 1 ) , … , K s m ( x , x n ) ] T ∈ R n ,
K N T K ( x , X ) = [ K N T K ( x , x 1 ) , … , K N T K ( x , x n ) ] T ∈ R n , K^{\mathrm{NTK}}(x,\mathbf{X})=[K^{\mathrm{NTK}}(x,x_1),\ldots,K^{\mathrm{NTK}}(x,x_n)]^\mathsf{T}\in\mathbb{R}^n, K NTK ( x , X ) = [ K NTK ( x , x 1 ) , … , K NTK ( x , x n ) ] T ∈ R n ,
r N N ( s ) = f s N N ( X ) − Y = [ f s N N ( x 1 ) − y 1 , … , f s N N ( x n ) − y n ] T ∈ R n , r^{\mathrm{NN}}(s)=f_s^{\mathrm{NN}}(\mathbf{X})-\mathbf{Y}=[f_s^{\mathrm{NN}}(x_1)-y_1,\ldots,f_s^{\mathrm{NN}}(x_n)-y_n]^\mathsf{T}\in\mathbb{R}^n, r NN ( s ) = f s NN ( X ) − Y = [ f s NN ( x 1 ) − y 1 , … , f s NN ( x n ) − y n ] T ∈ R n ,
r N T K ( s ) = f s N T K ( X ) − Y = [ f s N T K ( x 1 ) − y 1 , … , f s N T K ( x n ) − y n ] T ∈ R n . r^{\mathrm{NTK}}(s)=f_s^{\mathrm{NTK}}(\mathbf{X})-\mathbf{Y}=[f_s^{\mathrm{NTK}}(x_1)-y_1,\ldots,f_s^{\mathrm{NTK}}(x_n)-y_n]^\mathsf{T}\in\mathbb{R}^n. r NTK ( s ) = f s NTK ( X ) − Y = [ f s NTK ( x 1 ) − y 1 , … , f s NTK ( x n ) − y n ] T ∈ R n .
Consequently, we get:
∣ f t N N ( x ) − f t N T K ( x ) ∣ ≤ ∫ 0 t ∣ d d s f s N N ( x ) − d d s f s N T K ( x ) ∣ d s = ∫ 0 t ∣ − 1 n K s m ( x , X ) T r N N ( s ) + 1 n K N T K ( x , X ) T r N T K ( s ) ∣ d s = 1 n ∫ 0 t ∣ K s m ( x , X ) T r N N ( s ) − ( K N T K ( x , X ) − K s m ( x , X ) + K s m ( x , X ) ) T r N T K ( s ) ∣ d s = 1 n ∫ 0 t ∣ K s m ( x , X ) T ( r N N ( s ) − r N T K ( s ) ) + ( K s m ( x , X ) − K N T K ( x , X ) ) T r N T K ( s ) ∣ d s ≤ 1 n ∫ 0 t ( ( ∥ K s m ( x , X ) − K N T K ( x , X ) ∥ + ∥ K N T K ( x , X ) ∥ ) ∥ r N N ( s ) − r N T K ( s ) ∥ ) d s + 1 n ∫ 0 t ∥ K s m ( x , X ) − K N T K ( x , X ) ∥ ∥ r N T K ( s ) ∥ d s , (52) \begin{aligned}
\left|f_t^{\mathrm{NN}}(x)-f_t^{\mathrm{NTK}}(x)\right|
&\le\int_0^t\left|\frac{\mathrm{d}}{\mathrm{d}s}f_s^{\mathrm{NN}}(x)-\frac{\mathrm{d}}{\mathrm{d}s}f_s^{\mathrm{NTK}}(x)\right|\mathrm{d}s\\
&=\int_0^t\left|-\frac{1}{n}K_s^m(x,\mathbf{X})^\mathsf{T}r^{\mathrm{NN}}(s)+\frac{1}{n}K^{\mathrm{NTK}}(x,\mathbf{X})^\mathsf{T}r^{\mathrm{NTK}}(s)\right|\mathrm{d}s\\
&=\frac{1}{n}\int_0^t\left|K_s^m(x,\mathbf{X})^\mathsf{T}r^{\mathrm{NN}}(s)-\left(K^{\mathrm{NTK}}(x,\mathbf{X})-K_s^m(x,\mathbf{X})+K_s^m(x,\mathbf{X})\right)^\mathsf{T}r^{\mathrm{NTK}}(s)\right|\mathrm{d}s\\
&=\frac{1}{n}\int_0^t\left|K_s^m(x,\mathbf{X})^\mathsf{T}\left(r^{\mathrm{NN}}(s)-r^{\mathrm{NTK}}(s)\right)+\left(K_s^m(x,\mathbf{X})-K^{\mathrm{NTK}}(x,\mathbf{X})\right)^\mathsf{T}r^{\mathrm{NTK}}(s)\right|\mathrm{d}s\\
&\le\frac{1}{n}\int_0^t\left(\left(\left\|K_s^m(x,\mathbf{X})-K^{\mathrm{NTK}}(x,\mathbf{X})\right\|+\left\|K^{\mathrm{NTK}}(x,\mathbf{X})\right\|\right)\left\|r^{\mathrm{NN}}(s)-r^{\mathrm{NTK}}(s)\right\|\right)\mathrm{d}s\\
&\quad+\frac{1}{n}\int_0^t\left\|K_s^m(x,\mathbf{X})-K^{\mathrm{NTK}}(x,\mathbf{X})\right\|\left\|r^{\mathrm{NTK}}(s)\right\|\mathrm{d}s, \tag*{(52)}
\end{aligned} f t NN ( x ) − f t NTK ( x ) ≤ ∫ 0 t d s d f s NN ( x ) − d s d f s NTK ( x ) d s = ∫ 0 t − n 1 K s m ( x , X ) T r NN ( s ) + n 1 K NTK ( x , X ) T r NTK ( s ) d s = n 1 ∫ 0 t K s m ( x , X ) T r NN ( s ) − ( K NTK ( x , X ) − K s m ( x , X ) + K s m ( x , X ) ) T r NTK ( s ) d s = n 1 ∫ 0 t K s m ( x , X ) T ( r NN ( s ) − r NTK ( s ) ) + ( K s m ( x , X ) − K NTK ( x , X ) ) T r NTK ( s ) d s ≤ n 1 ∫ 0 t ( ( K s m ( x , X ) − K NTK ( x , X ) + K NTK ( x , X ) ) r NN ( s ) − r NTK ( s ) ) d s + n 1 ∫ 0 t K s m ( x , X ) − K NTK ( x , X ) r NTK ( s ) d s , (52)
where we use the triangle inequality in the last inequality. Let:
Δ = sup t ≥ 0 sup ∥ x ∥ ≤ A , ∥ x ′ ∥ ≤ A ∣ K t m ( x , x ′ ) − K N T K ( x , x ′ ) ∣ . \Delta= \sup_{t \ge0}\sup_{\lVert x\rVert\le A,\lVert x'\rVert\le A}\left|K^{m}_{t}(x,x')-K^{\mathrm{NTK}}(x,x')\right|. Δ = t ≥ 0 sup ∥ x ∥ ≤ A , ∥ x ′ ∥ ≤ A sup K t m ( x , x ′ ) − K NTK ( x , x ′ ) .
We notice that:
∥ K s m ( x , X ) − K N T K ( x , X ) ∥ = ∑ i = 1 n ∣ K s m ( x , x i ) − K N T K ( x , x i ) ∣ 2 ≤ n Δ , (53) \left\lVert K^{m}_{s}(x,X)-K^{\mathrm{NTK}}(x,X)\right\rVert=\sqrt{\sum_{i=1}^{n}\left|K^{m}_{s}(x,x_i)-K^{\mathrm{NTK}}(x,x_i)\right|^{2}}\le\sqrt{n}\Delta, \tag*{(53)} K s m ( x , X ) − K NTK ( x , X ) = i = 1 ∑ n ∣ K s m ( x , x i ) − K NTK ( x , x i ) ∣ 2 ≤ n Δ , (53)
∥ K N T K ( x , X ) ∥ = ∑ i = 1 n ∣ K N T K ( x , x i ) ∣ 2 ≤ n κ N T K , (54) \left\lVert K^{\mathrm{NTK}}(x,X)\right\rVert=\sqrt{\sum_{i=1}^{n}\left|K^{\mathrm{NTK}}(x,x_i)\right|^{2}}\le\sqrt{n}\kappa^{\mathrm{NTK}}, \tag*{(54)} K NTK ( x , X ) = i = 1 ∑ n ∣ K NTK ( x , x i ) ∣ 2 ≤ n κ NTK , (54)
where in the second line we use Lemma E.2 to bound ∣ K N T K ( x , x i ) ∣ \left|K^{\mathrm{NTK}}(x,x_i)\right| K NTK ( x , x i ) by κ N T K ≔ ( p + 2 ) ( A 2 + 1 ) + 1 \kappa^{\mathrm{NTK}}\coloneqq(p+2)(A^{2}+1)+1 κ NTK : = ( p + 2 ) ( A 2 + 1 ) + 1 . Moreover, by Lemma E.13 , we get:
∥ r N T K ( s ) ∥ ≤ exp ( − λ 0 n s ) n B , (55) \left\lVert r^{\mathrm{NTK}}(s)\right\rVert\le\exp\left(-\frac{\lambda_0}{n}s\right)\sqrt{n}B, \tag*{(55)} r NTK ( s ) ≤ exp ( − n λ 0 s ) n B , (55)
∥ r N N ( s ) − r N T K ( s ) ∥ ≤ Δ s exp ( γ s ) n B , (56) \left\lVert r^{\mathrm{NN}}(s)-r^{\mathrm{NTK}}(s)\right\rVert\le\Delta s\exp(\gamma s)\sqrt{n}B, \tag*{(56)} r NN ( s ) − r NTK ( s ) ≤ Δ s exp ( γ s ) n B , (56)
where λ 0 \lambda_0 λ 0 is the smallest eigenvalue of the matrix K N T K K^{\mathrm{NTK}} K NTK and γ = Δ − λ 0 n \gamma=\Delta-\frac{\lambda_0}{n} γ = Δ − n λ 0 . Substituting Equations (53) to (56) into Equation (52) , we obtain:
∣ f t N N ( x ) − f t N T K ( x ) ∣ ≤ ( Δ + κ N T K ) Δ B ∫ 0 t s exp ( γ s ) d s + Δ B ∫ 0 t exp ( − λ 0 n s ) d s . \left|f^{\mathrm{NN}}_{t}(x)-f^{\mathrm{NTK}}_{t}(x)\right|\le(\Delta+\kappa^{\mathrm{NTK}})\Delta B\int_{0}^{t}s\exp(\gamma s)\,\mathrm{d}s+\Delta B\int_{0}^{t}\exp\left(-\frac{\lambda_0}{n}s\right)\,\mathrm{d}s. f t NN ( x ) − f t NTK ( x ) ≤ ( Δ + κ NTK ) Δ B ∫ 0 t s exp ( γ s ) d s + Δ B ∫ 0 t exp ( − n λ 0 s ) d s .
Applying integration by parts gives:
∫ 0 t s exp ( γ s ) d s = [ s γ exp ( γ s ) ] s = 0 s = t − 1 γ ∫ 0 t exp ( γ s ) d s = t γ exp ( γ t ) − 1 γ 2 ( exp ( γ t ) − 1 ) = exp ( γ t ) γ 2 ( γ t − 1 ) + 1 γ 2 . \begin{aligned}
\int_{0}^{t}s\exp(\gamma s)\,\mathrm{d}s=\left[\frac{s}{\gamma}\exp(\gamma s)\right]_{s=0}^{s=t}-\frac{1}{\gamma}\int_{0}^{t}\exp(\gamma s)\,\mathrm{d}s=\frac{t}{\gamma}\exp(\gamma t)-\frac{1}{\gamma^{2}}\left(\exp(\gamma t)-1\right) \\
&=\frac{\exp(\gamma t)}{\gamma^{2}}(\gamma t-1)+\frac{1}{\gamma^{2}}.
\end{aligned} ∫ 0 t s exp ( γ s ) d s = [ γ s exp ( γ s ) ] s = 0 s = t − γ 1 ∫ 0 t exp ( γ s ) d s = γ t exp ( γ t ) − γ 2 1 ( exp ( γ t ) − 1 ) = γ 2 exp ( γ t ) ( γ t − 1 ) + γ 2 1 .
Also, we have:
∫ 0 t exp ( − λ 0 n s ) d s = [ − n λ 0 exp ( − λ 0 n s ) ] s = 0 s = t = n λ 0 [ 1 − exp ( − λ 0 n t ) ] ≤ n λ 0 . \int_{0}^{t}\exp\left(-\frac{\lambda_0}{n}s\right)\,\mathrm{d}s=\left[-\frac{n}{\lambda_0}\exp\left(-\frac{\lambda_0}{n}s\right)\right]_{s=0}^{s=t}=\frac{n}{\lambda_0}\left[1-\exp\left(-\frac{\lambda_0}{n}t\right)\right]\le\frac{n}{\lambda_0}. ∫ 0 t exp ( − n λ 0 s ) d s = [ − λ 0 n exp ( − n λ 0 s ) ] s = 0 s = t = λ 0 n [ 1 − exp ( − n λ 0 t ) ] ≤ λ 0 n .
Thus:
∣ f t N N ( x ) − f t N T K ( x ) ∣ ≤ Δ B [ ( Δ + κ N T K ) ( exp ( γ t ) γ 2 ( γ t − 1 ) + 1 γ 2 ) + n λ 0 ] . (57) \left|f^{\mathrm{NN}}_{t}(x)-f^{\mathrm{NTK}}_{t}(x)\right|\le\Delta B\left[(\Delta+\kappa^{\mathrm{NTK}})\left(\frac{\exp(\gamma t)}{\gamma^{2}}(\gamma t-1)+\frac{1}{\gamma^{2}}\right)+\frac{n}{\lambda_0}\right]. \tag*{(57)} f t NN ( x ) − f t NTK ( x ) ≤ Δ B [ ( Δ + κ NTK ) ( γ 2 exp ( γ t ) ( γ t − 1 ) + γ 2 1 ) + λ 0 n ] . (57)
By Lemma E.18 , we know that with probability at least 1 − δ 1-\delta 1 − δ with respect to random samples and random initialization, it holds that Δ ≤ C m − 1 2 ln m \Delta\le Cm^{-\frac{1}{2}}\sqrt{\ln m} Δ ≤ C m − 2 1 ln m when m m m is large enough such that m ≥ poly ( n , 1 / δ , ln n , ln ( 1 / δ ) ) m\ge\operatorname{poly}(n,1/\delta,\ln n,\ln(1/\delta)) m ≥ poly ( n , 1/ δ , ln n , ln ( 1/ δ )) , where the constant C > 0 C>0 C > 0 is independent of n n n and m m m , but depends on n n n , and the coefficients of poly depend on n n n , B B B , A A A , M μ M_{\mu} M μ , and p p p . When m ≥ max ( ( 2 C 12 n λ 0 ) 24 , 1 ) m\ge\max\left(\left(\frac{2C\sqrt{12n}}{\lambda_0}\right)^{24},1\right) m ≥ max ( ( λ 0 2 C 12 n ) 24 , 1 ) , we obtain by Lemma I.9 that Δ ≤ λ 0 2 n \Delta\le\frac{\lambda_0}{2n} Δ ≤ 2 n λ 0 , which implies that γ ≤ − λ 0 2 n < 0 \gamma\le-\frac{\lambda_0}{2n}<0 γ ≤ − 2 n λ 0 < 0 . Similarly, when m ≥ max ( ( C 12 ) 24 , 1 ) m\ge\max\left((C\sqrt{12})^{24},1\right) m ≥ max ( ( C 12 ) 24 , 1 ) , it holds by Lemma I.9 that Δ ≤ 1 \Delta\le1 Δ ≤ 1 .
Hence, when m ≥ max ( poly ( n , 1 / δ , ln n , ln ( 1 / δ ) ) , ( 2 C 12 n λ 0 ) 24 , ( C 12 ) 24 , 1 ) m\ge\max\left(\operatorname{poly}(n,1/\delta,\ln n,\ln(1/\delta)),\left(\frac{2C\sqrt{12n}}{\lambda_0}\right)^{24},(C\sqrt{12})^{24},1\right) m ≥ max ( poly ( n , 1/ δ , ln n , ln ( 1/ δ )) , ( λ 0 2 C 12 n ) 24 , ( C 12 ) 24 , 1 ) , it holds that Δ ≤ min ( C m − 1 2 ln m , λ 0 2 n , 1 ) \Delta\le\min\left(Cm^{-\frac{1}{2}}\sqrt{\ln m},\frac{\lambda_0}{2n},1\right) Δ ≤ min ( C m − 2 1 ln m , 2 n λ 0 , 1 ) . Consequently, with probability at least 1 − δ 1-\delta 1 − δ with respect to random samples and random initialization, Equation (57) simplifies to:
∣ f t N N ( x ) − f t N T K ( x ) ∣ ≤ C m − 1 2 ln m B [ ( 1 + κ N T K ) 4 n 2 λ 0 2 + n λ 0 ] ≤ C m − 1 2 ln m B n 2 [ ( 1 + κ N T K ) 4 λ 0 2 + 1 λ 0 ] . (58) \begin{aligned}
\left|f^{\mathrm{NN}}_{t}(x)-f^{\mathrm{NTK}}_{t}(x)\right|\le Cm^{-\frac{1}{2}}\sqrt{\ln m}B\left[\left(1+\kappa^{\mathrm{NTK}}\right)\frac{4n^{2}}{\lambda_0^{2}}+\frac{n}{\lambda_0}\right] \\
&\le Cm^{-\frac{1}{2}}\sqrt{\ln m}Bn^{2}\left[\left(1+\kappa^{\mathrm{NTK}}\right)\frac{4}{\lambda_0^{2}}+\frac{1}{\lambda_0}\right]. \tag*{(58)}
\end{aligned} f t NN ( x ) − f t NTK ( x ) ≤ C m − 2 1 ln m B [ ( 1 + κ NTK ) λ 0 2 4 n 2 + λ 0 n ] ≤ C m − 2 1 ln m B n 2 [ ( 1 + κ NTK ) λ 0 2 4 + λ 0 1 ] . (58)
By Corollary E.10, since n ≥ 3 n \ge3 n ≥ 3 , the following holds with probability at least 1 − δ 1-\delta 1 − δ with respect to random samples: λ 0 − 1 ≤ poly 1 ( n , 1 / δ , ln n , ln ( 1 / δ ) ) \lambda_0^{-1} \le\operatorname{poly}_1(n,1/\delta,\ln n,\ln(1/\delta)) λ 0 − 1 ≤ poly 1 ( n , 1/ δ , ln n , ln ( 1/ δ )) , where poly 1 ( n , 1 / δ , ln n , ln ( 1 / δ ) ) : = C ′ n 8 / p δ − 4 / p ( ln n + ln ( 1 / δ ) ) 3 \operatorname{poly}_1(n,1/\delta,\ln n,\ln(1/\delta)) := C^{\prime}n^{8/p}\delta^{-4/p}(\ln n+\ln(1/\delta))^3 poly 1 ( n , 1/ δ , ln n , ln ( 1/ δ )) := C ′ n 8/ p δ − 4/ p ( ln n + ln ( 1/ δ ) ) 3 is a polynomial whose coefficients depend on A A A , M μ M_\mu M μ , and p p p . Consequently, we obtain:
λ 0 − 1 ≤ poly 1 ( n , 1 / δ , ln n , ln n , ln ( 1 / δ ) ) and λ 0 − 2 ≤ ( poly 1 ( n , 1 / δ , ln n , ln ( 1 / δ ) ) ) 2 . (59) \lambda_0^{-1} \le\operatorname{poly}_1(n,1/\delta,\ln n,\ln n,\ln(1/\delta)) \quad\text{and} \quad\lambda_0^{-2} \le\left(\operatorname{poly}_1(n,1/\delta,\ln n,\ln(1/\delta))\right)^2. \tag*{(59)} λ 0 − 1 ≤ poly 1 ( n , 1/ δ , ln n , ln n , ln ( 1/ δ )) and λ 0 − 2 ≤ ( poly 1 ( n , 1/ δ , ln n , ln ( 1/ δ )) ) 2 . (59)
Using this, we require that m ≥ P 1 ( n , 1 / δ , ln n , ln ( 1 / δ ) ) m \ge P_1(n,1/\delta,\ln n,\ln(1/\delta)) m ≥ P 1 ( n , 1/ δ , ln n , ln ( 1/ δ )) where P 1 P_1 P 1 is a polynomial that dominates:
max ( poly ( n , 1 / δ , ln n , ln ( 1 / δ ) ) , ( 2 C 12 n poly 1 ( n , 1 δ , ln n , ln 1 δ ) ) 24 , ( C 12 ) 24 , 1 ) . \max\left(\operatorname{poly}(n,1/\delta,\ln n,\ln(1/\delta)),\left(2C\sqrt{12n}\operatorname{poly}_1\left(n,\frac{1}{\delta},\ln n,\ln\frac{1}{\delta}\right)\right)^{24},(C\sqrt{12})^{24},1\right). max ( poly ( n , 1/ δ , ln n , ln ( 1/ δ )) , ( 2 C 12 n poly 1 ( n , δ 1 , ln n , ln δ 1 ) ) 24 , ( C 12 ) 24 , 1 ) .
Plugging Equation (59) in Equation (58) , we get:
∣ f t N N ( x ) − f t N T K ( x ) ∣ ≤ C m − 1 12 ln m P 2 ( n , 1 δ , ln n , ln 1 δ ) . \left|f_t^{\mathrm{NN}}(x)-f_t^{\mathrm{NTK}}(x)\right| \le Cm^{-\frac{1}{12}}\sqrt{\ln m}P_2\left(n,\frac{1}{\delta},\ln n,\ln\frac{1}{\delta}\right). f t NN ( x ) − f t NTK ( x ) ≤ C m − 12 1 ln m P 2 ( n , δ 1 , ln n , ln δ 1 ) .
where P 2 P_2 P 2 is a polynomial expressed as:
P 2 ( n , 1 δ , ln n , ln 1 δ ) = B n 2 [ 4 ( 1 + κ N T K ) ( poly 1 ( n , 1 δ , ln n , ln 1 δ ) ) 2 + poly 1 ( n , 1 δ , ln n , ln 1 δ ) ] . P_2\left(n,\frac{1}{\delta},\ln n,\ln\frac{1}{\delta}\right)=Bn^2\left[4(1+\kappa^{\mathrm{NTK}})\left(\operatorname{poly}_1\left(n,\frac{1}{\delta},\ln n,\ln\frac{1}{\delta}\right)\right)^2+\operatorname{poly}_1\left(n,\frac{1}{\delta},\ln n,\ln\frac{1}{\delta}\right)\right]. P 2 ( n , δ 1 , ln n , ln δ 1 ) = B n 2 [ 4 ( 1 + κ NTK ) ( poly 1 ( n , δ 1 , ln n , ln δ 1 ) ) 2 + poly 1 ( n , δ 1 , ln n , ln δ 1 ) ] .
When m ≥ max ( P 1 ( n , 1 / δ , ln n , ln ( 1 / δ ) ) , ( P 2 ( n , 1 / δ , ln n , ln ( 1 / δ ) ) ) 24 ) m \ge\max\left(P_1(n,1/\delta,\ln n,\ln(1/\delta)),(P_2(n,1/\delta,\ln n,\ln(1/\delta)))^{24}\right) m ≥ max ( P 1 ( n , 1/ δ , ln n , ln ( 1/ δ )) , ( P 2 ( n , 1/ δ , ln n , ln ( 1/ δ )) ) 24 ) , we obtain:
∣ f t N N ( x ) − f t N T K ( x ) ∣ ≤ C m − 1 24 ln m . \left|f_t^{\mathrm{NN}}(x)-f_t^{\mathrm{NTK}}(x)\right| \le Cm^{-\frac{1}{24}}\sqrt{\ln m}. f t NN ( x ) − f t NTK ( x ) ≤ C m − 24 1 ln m .
We conclude the proof by redefining poly \operatorname{poly} poly to be the polynomial that dominates:
max ( P 1 ( n , 1 / δ , ln n , ln ( 1 / δ ) ) , ( P 2 ( n , 1 / δ , ln n , ln ( 1 / δ ) ) ) 24 ) . \max\left(P_1(n,1/\delta,\ln n,\ln(1/\delta)),(P_2(n,1/\delta,\ln n,\ln(1/\delta)))^{24}\right). max ( P 1 ( n , 1/ δ , ln n , ln ( 1/ δ )) , ( P 2 ( n , 1/ δ , ln n , ln ( 1/ δ )) ) 24 ) .
Generalization guarantees
High-probability generalization bounds
Corollary E.20, taken from [63 ] , provides a high-probability bound on the generalization error of a broad class of spectral algorithms [63 Algorithm 1] applied to kernel regression in an RKHS. This class includes gradient methods [63 Example 3.2] and ridge regression [63 Example 3.3] as special cases, unified through the notion of a filter function [63 Definition 3.1] . The result shows that the estimator produced by any of these algorithms converges to the best approximation of the regression function in the RKHS at a rate that depends on the smoothness of the target function through the source condition , and on the complexity of the hypothesis space through the capacity condition . This is the key result that we use in Lemma E.21 to control the estimation error of the NTK gradient flow, by identifying the RKHS with the one associated to the infinite-width NTK and verifying that the assumptions required by this corollary are satisfied in our setting.
Corollary E.20 (Probabilistic generalization bound of the NTK predictor – Corollary 4.4 in [63 ] ). Let H \mathcal{H} H be an RKHS associated with a Mercer kernel K : X × X → R K:\mathcal{X}\times\mathcal{X}\to\mathbb{R} K : X × X → R , T : L 2 ( X ) → L 2 ( X ) T:L^2(\mathcal{X})\to L^2(\mathcal{X}) T : L 2 ( X ) → L 2 ( X ) be the integral operator, and Σ : H → H \Sigma:\mathcal{H}\to\mathcal{H} Σ : H → H be the covariance operator, all defined in Appendix § 16 . Denote by f H f_{\mathcal{H}} f H the projection of the regression function f ⋆ f^\star f ⋆ onto the closure of H \mathcal{H} H in L 2 ( X ) L^2(\mathcal{X}) L 2 ( X ) . Given i.i.d. samples { ( x i , y i ) } i = 1 n \{(x_i,y_i)\}_{i=1}^n {( x i , y i ) } i = 1 n drawn from a distribution ρ \rho ρ with marginal μ \mu μ on X \mathcal{X} X , we consider the following empirical risk minimization problem:
f λ = argmin f ∈ H 1 2 n ∑ i = 1 n ( f ( x i ) − y i ) 2 , f_\lambda=\underset{f\in\mathcal{H}}{\operatorname{argmin}}\ \frac{1}{2n}\sum_{i=1}^{n}(f(x_i)-y_i)^2, f λ = f ∈ H argmin 2 n 1 i = 1 ∑ n ( f ( x i ) − y i ) 2 , which we want to solve using a spectral algorithm with filter function G λ G_\lambda G λ indexed with λ > 0 \lambda>0 λ > 0 . We make the following assumptions:
Assumption 1. (Bounded kernel) There exists a constant κ ≥ 1 \kappa\ge1 κ ≥ 1 such that K ( x , x ′ ) ≤ κ 2 K(x,x') \le\kappa^2 K ( x , x ′ ) ≤ κ 2 for any x , x ′ ∈ X x,x' \in\mathcal{X} x , x ′ ∈ X .
Assumption 2. (Moment condition) There exist positive constants Q , J > 0 Q,J>0 Q , J > 0 such that for all integers l ≥ 2 l \ge2 l ≥ 2 , we have E y ∼ ρ [ ∣ y ∣ l ∣ x ] ≤ 1 2 l ! J l − 2 Q 2 \mathbb{E}_{y\sim\rho}[|y|^l \mid x] \le\frac{1}{2}l!J^{l-2}Q^2 E y ∼ ρ [ ∣ y ∣ l ∣ x ] ≤ 2 1 l ! J l − 2 Q 2 .
Assumption 3. (Regularity/smoothness of f H f_{\mathcal{H}} f H ) f H f_{\mathcal{H}} f H satisfies E x ∼ μ [ ∣ f H ( x ) − f ⋆ ( x ) ∣ 2 K ( x , ⋅ ) ⊗ K ( x , ⋅ ) ] ⪯ U 2 Σ \mathbb{E}_{x\sim\mu}[|f_{\mathcal{H}}(x)-f^\star(x)|^2K(x,\cdot)\otimes K(x,\cdot)] \preceq U^2\Sigma E x ∼ μ [ ∣ f H ( x ) − f ⋆ ( x ) ∣ 2 K ( x , ⋅ ) ⊗ K ( x , ⋅ )] ⪯ U 2 Σ and the following source condition f H = ϕ ( T ) g 0 f_{\mathcal{H}}=\phi(T)g_0 f H = ϕ ( T ) g 0 with ∥ g 0 ∥ L 2 ( X ) ≤ S \|g_0\|_{L^2(\mathcal{X})}\le S ∥ g 0 ∥ L 2 ( X ) ≤ S , where U , S ≥ 0 U,S\ge0 U , S ≥ 0 , ϕ ( u ) = κ − 2 max ( 0 , ζ − 1 ) u ζ \phi(u)=\kappa^{-2\max(0,\zeta-1)}u^\zeta ϕ ( u ) = κ − 2 m a x ( 0 , ζ − 1 ) u ζ , and ζ ≥ 0 \zeta\ge0 ζ ≥ 0 .
Assumption 4. (Capacity condition) For some γ ∈ ] 0 , 1 ] \gamma\in]0,1] γ ∈ ] 0 , 1 ] and c γ > 0 c_\gamma>0 c γ > 0 , the effective dimension of H \mathcal{H} H satisfies d e f f ( λ ) ≤ c γ λ − γ d_{\mathrm{eff}}(\lambda)\le c_\gamma\lambda^{-\gamma} d eff ( λ ) ≤ c γ λ − γ for all λ > 0 \lambda>0 λ > 0 , where d e f f ( λ ) = Trace ( Σ ( Σ + λ 1 H ) − 1 ) d_{\mathrm{eff}}(\lambda)=\operatorname{Trace}(\Sigma(\Sigma+\lambda1_{\mathcal{H}})^{-1}) d eff ( λ ) = Trace ( Σ ( Σ + λ 1 H ) − 1 ) .
Let a ∈ [ 0 , min ( 1 2 , ζ ) ] a\in[0,\min(\frac{1}{2},\zeta)] a ∈ [ 0 , min ( 2 1 , ζ )] and λ o p = n − 1 2 ζ + γ \lambda_{\mathrm{op}}=n^{-\frac{1}{2\zeta+\gamma}} λ op = n − 2 ζ + γ 1 . There exists a constant C > 0 C>0 C > 0 such that, for any δ ∈ ( 0 , 1 ) \delta\in(0,1) δ ∈ ( 0 , 1 ) , with probability at least 1 − δ 1-\delta 1 − δ with respect to random samples, we have:
∥ T − a ( f λ o p − f H ) ∥ L 2 ( X ) ≤ C ( ln 6 δ ) 2 − a n − ζ − a 2 ζ + γ if 2 ζ + γ > 1 , \left\|\mathcal{T}^{-a}(f_{\lambda_{\mathrm{op}}}-f_{\mathcal{H}})\right\|_{L^2(\mathcal{X})}
\le C\left(\ln\frac{6}{\delta}\right)^{2-a}n^{-\frac{\zeta-a}{2\zeta+\gamma}}
\quad\text{if }2\zeta+\gamma>1, T − a ( f λ op − f H ) L 2 ( X ) ≤ C ( ln δ 6 ) 2 − a n − 2 ζ + γ ζ − a if 2 ζ + γ > 1 ,
where the constant C > 0 C>0 C > 0 is independent of δ \delta δ , n n n , and λ o p \lambda_{\mathrm{op}} λ op , but depends on κ \kappa κ , c γ c_\gamma c γ , γ \gamma γ , ζ \zeta ζ , U U U , J J J , Q Q Q , S S S , and a a a .
Lemma E.21 (Probabilistic generalization bound of the NTK predictor – Application of Corollary E.20 to our setting). Consider n n n i.i.d. samples { ( x i , y i ) } i = 1 n ∼ ρ \{(x_i,y_i)\}_{i=1}^n\sim\rho {( x i , y i ) } i = 1 n ∼ ρ . Let Assumptions a a a to d d d hold. If we choose a stopping time t o p = n p + 1 s ( p + 1 ) + p t_{\mathrm{op}}=n^{\frac{p+1}{s(p+1)+p}} t op = n s ( p + 1 ) + p p + 1 with 1 p + 1 < s ≤ 2 \frac{1}{p+1}<s\le2 p + 1 1 < s ≤ 2 , then there exists a constant C > 0 C>0 C > 0 such that, for any δ ∈ ( 0 , 1 ) \delta\in(0,1) δ ∈ ( 0 , 1 ) , with probability at least 1 − δ 1-\delta 1 − δ with respect to random samples, we have:
∥ f t o p N T K − f ⋆ ∥ L 2 ( X ) ≤ C ( ln 6 δ ) 2 n − 1 2 s ( p + 1 ) s ( p + 1 ) + p , \left\|f_{t_{\mathrm{op}}}^{\mathrm{NTK}}-f^\star\right\|_{L^2(\mathcal{X})}
\le C\left(\ln\frac{6}{\delta}\right)^2n^{-\frac{1}{2}\frac{s(p+1)}{s(p+1)+p}}, f t op NTK − f ⋆ L 2 ( X ) ≤ C ( ln δ 6 ) 2 n − 2 1 s ( p + 1 ) + p s ( p + 1 ) , where the constant C > 0 C>0 C > 0 is independent of δ \delta δ , n n n , and t o p t_{\mathrm{op}} t op , but depends on s s s , p p p , A A A , B B B , M μ M_\mu M μ , and R R R .
Proof. In this proof, we apply the result in Corollary E.20 to our setting using our assumptions. For that, we set a = 0 a=0 a = 0 , s = 2 ζ s=2\zeta s = 2 ζ to be the smoothness parameter of the interpolation space [ H N T K ] s [\mathcal{H}^{\mathrm{NTK}}]^s [ H NTK ] s , and β : = p + 1 p = 1 γ \beta:=\frac{p+1}{p}=\frac{1}{\gamma} β := p p + 1 = γ 1 to be the eigenvalue decay rate of K N T K K^{\mathrm{NTK}} K NTK as shown in Lemma E.12 under Assumptions a a a and d d d , where a a a , ζ \zeta ζ , and γ \gamma γ are the notations used in Corollary E.20. We verify that the assumptions in Corollary E.20 hold in our setting for K N T K K^{\mathrm{NTK}} K NTK .
Assumption [2 ] holds in our setting under Assumption b b b since the fact that ∣ y ∣ ≤ B |y|\le B ∣ y ∣ ≤ B implies E y ∼ ρ [ ∣ y ∣ l ∣ x ] ≤ B l = B l − 2 B 2 ≤ 1 2 l ! B l − 2 B 2 \mathbb{E}_{y\sim\rho}[|y|^l\mid x]\le B^l=B^{l-2}B^2\le\frac{1}{2}l!B^{l-2}B^2 E y ∼ ρ [ ∣ y ∣ l ∣ x ] ≤ B l = B l − 2 B 2 ≤ 2 1 l ! B l − 2 B 2 for all l ≥ 2 l\ge2 l ≥ 2 , so it suffices to choose J = B J=B J = B and Q = B Q=B Q = B .
The first part of Assumption [3 ] is satisfied in our setting under Assumptions a a a and c c c for U = 0 U=0 U = 0 , because f ⋆ f^\star f ⋆ coincides with its projection onto the closure of H N T K \mathcal{H}^{\mathrm{NTK}} H NTK in L 2 ( X ) L^2(\mathcal{X}) L 2 ( X ) . In fact, we show in Lemma E.4 that f ⋆ f^\star f ⋆ belongs to the closure of H N T K \mathcal{H}^{\mathrm{NTK}} H NTK in L 2 ( X ) L^2(\mathcal{X}) L 2 ( X ) , which means that its projection onto that closure is itself. The second part of Assumption [3 ] is satisfied for S = R S=R S = R with 0 ≤ ζ ≤ 1 0\le\zeta\le1 0 ≤ ζ ≤ 1 under Assumption c c c . The condition on ζ \zeta ζ implies that 0 ≤ s ≤ 2 0\le s\le2 0 ≤ s ≤ 2 .
Assumption [4 ] is a consequence of Lemma E.12 . In fact, letting ( λ i ) i ≥ 1 (\lambda_i)_{i\ge1} ( λ i ) i ≥ 1 denote the eigenvalues of Σ \Sigma Σ , we have the following upper-bound on the effective dimension of H \mathcal{H} H :
d e f f ( λ ) = Trace ( Σ ( Σ + λ 1 H ) − 1 ) = ∑ i ≥ 1 λ i λ i + λ ≤ ∑ i ≥ 1 min ( 1 , k 2 i − β λ ) ≤ ∫ 0 ∞ min ( 1 , k 2 x − β λ ) d x = β β − 1 k 2 1 β λ − 1 β , \begin{aligned}
d_{\mathrm{eff}}(\lambda)=\operatorname{Trace}(\Sigma(\Sigma+\lambda1_{\mathcal{H}})^{-1})=\sum_{i\ge1}\frac{\lambda_i}{\lambda_i+\lambda}\le\sum_{i\ge1}\min\left(1,\frac{k_2i^{-\beta}}{\lambda}\right) \\
&\le\int_0^\infty\min\left(1,\frac{k_2x^{-\beta}}{\lambda}\right)\,dx=\frac{\beta}{\beta-1}k_2^{\frac{1}{\beta}}\lambda^{-\frac{1}{\beta}},
\end{aligned} d eff ( λ ) = Trace ( Σ ( Σ + λ 1 H ) − 1 ) = i ≥ 1 ∑ λ i + λ λ i ≤ i ≥ 1 ∑ min ( 1 , λ k 2 i − β ) ≤ ∫ 0 ∞ min ( 1 , λ k 2 x − β ) d x = β − 1 β k 2 β 1 λ − β 1 ,
where we use λ i λ i + λ ≤ min ( 1 , λ i λ ) \frac{\lambda_i}{\lambda_i+\lambda}\le\min(1,\frac{\lambda_i}{\lambda}) λ i + λ λ i ≤ min ( 1 , λ λ i ) , the upper bound λ i ≤ k 2 i − β \lambda_i\le k_2i^{-\beta} λ i ≤ k 2 i − β of Lemma E.12 , and the fact that x ↦ min ( 1 , k 2 x − β / λ ) x\mapsto\min(1,k_2x^{-\beta}/\lambda) x ↦ min ( 1 , k 2 x − β / λ ) is non-increasing. The integral equals x 0 + x 0 β − 1 x_0+\frac{x_0}{\beta-1} x 0 + β − 1 x 0 with x 0 = ( k 2 / λ ) 1 / β x_0=(k_2/\lambda)^{1/\beta} x 0 = ( k 2 / λ ) 1/ β . This is precisely Assumption [4 ] with c γ = β β − 1 k 1 p p + 1 = ( p + 1 ) k 1 p p + 1 c_\gamma=\frac{\beta}{\beta-1}k_1^{\frac{p}{p+1}}=(p+1)k_1^{\frac{p}{p+1}} c γ = β − 1 β k 1 p + 1 p = ( p + 1 ) k 1 p + 1 p , since β = p + 1 p \beta=\frac{p+1}{p} β = p p + 1 .
It remains to identify the filter function and the optimal stopping time. According to [58 Lemma D.1] , for a given λ > 0 \lambda>0 λ > 0 , the filter function of the gradient flow is given by:
G λ ( u ) = 1 − exp ( − u λ ) u , G_\lambda(u)=\frac{1-\exp\left(-\frac{u}{\lambda}\right)}{u}, G λ ( u ) = u 1 − exp ( − λ u ) ,
where 0 ≤ u ≤ κ N T K 0\leq u\leq\kappa^{\mathrm{NTK}} 0 ≤ u ≤ κ NTK with κ N T K : = ( p + 2 ) ( A 2 + 1 ) + 1 \kappa^{\mathrm{NTK}}:=(p+2)(A^2+1)+1 κ NTK := ( p + 2 ) ( A 2 + 1 ) + 1 by Lemma E.2. From [58 Proof of Lemma D.1] , the continuous time parameter is t = 1 λ t=\frac{1}{\lambda} t = λ 1 . Using the fact that λ o p − 1 = n 2 ζ + γ s β + 1 \lambda_{\mathrm{op}}^{-1}=n^{\frac{2\zeta+\gamma}{s\beta+1}} λ op − 1 = n s β + 1 2 ζ + γ allows us to obtain that t o p = n β s β + 1 = n p + 1 s ( p + 1 ) + p t_{\mathrm{op}}=n^{\frac{\beta}{s\beta+1}}=n^{\frac{p+1}{s(p+1)+p}} t op = n s β + 1 β = n s ( p + 1 ) + p p + 1 , where t o p t_{\mathrm{op}} t op is the stopping time of the gradient flow algorithm. It is important to note that the result in Corollary E.20 only holds for s > 1 − 1 β = 1 p + 1 s>1-\frac{1}{\beta}=\frac{1}{p+1} s > 1 − β 1 = p + 1 1 . Combining this with 0 ≤ s ≤ 2 0\leq s\leq2 0 ≤ s ≤ 2 provided earlier gives 1 p + 1 < s ≤ 2 \frac{1}{p+1}<s\leq2 p + 1 1 < s ≤ 2 .
Proposition E.22 (Probabilistic generalization bound). Consider n ≥ 3 n\geq3 n ≥ 3 i.i.d. samples { ( x i , y i ) } i = 1 n ∼ ρ \{(x_i,y_i)\}_{i=1}^n\sim\rho {( x i , y i ) } i = 1 n ∼ ρ and fix p ≥ 2 p\geq2 p ≥ 2 . Let Assumptions a to d hold. If we choose a stopping time t o p = n p + 1 s ( p + 1 ) + p t_{\mathrm{op}}=n^{\frac{p+1}{s(p+1)+p}} t op = n s ( p + 1 ) + p p + 1 with 1 p + 1 < s ≤ 2 \frac{1}{p+1}<s\leq2 p + 1 1 < s ≤ 2 , then there exist a constant C > 0 C>0 C > 0 and a polynomial poly \operatorname{poly} poly such that, for any δ ∈ ( 0 , 1 ) \delta\in(0,1) δ ∈ ( 0 , 1 ) , when the width satisfies m ≥ poly ( n , 1 / δ , ln n , ln ( 1 / δ ) ) m\geq\operatorname{poly}(n,1/\delta,\ln n,\ln(1/\delta)) m ≥ poly ( n , 1/ δ , ln n , ln ( 1/ δ )) , with probability at least 1 − δ 1-\delta 1 − δ with respect to random samples and random initialization, we have:
∥ f t o p N N − f ⋆ ∥ L 2 ( X ) ≤ C ( ln 6 δ ) 2 n − 1 2 s ( p + 1 ) s ( p + 1 ) + p , \left\|f^{\mathrm{NN}}_{t_{\mathrm{op}}}-f^\star\right\|_{L^2(\mathcal{X})}\leq C\left(\ln\frac{6}{\delta}\right)^2n^{-\frac{1}{2}\frac{s(p+1)}{s(p+1)+p}}, f t op NN − f ⋆ L 2 ( X ) ≤ C ( ln δ 6 ) 2 n − 2 1 s ( p + 1 ) + p s ( p + 1 ) , where the constant C > 0 C>0 C > 0 is independent of δ \delta δ , n n n and t o p t_{\mathrm{op}} t op , but depends on s s s , p p p , A A A , B B B , M μ M_\mu M μ , and R R R , and the coefficients of poly \operatorname{poly} poly depend on A A A , B B B , M μ M_\mu M μ , and $p.
Proof. By the triangle inequality and the identity ∥ ⋅ ∥ L 2 ( X ) ≤ ∥ ⋅ ∥ ∞ \|\cdot\|_{L^2(\mathcal{X})}\leq\|\cdot\|_\infty ∥ ⋅ ∥ L 2 ( X ) ≤ ∥ ⋅ ∥ ∞ , we have:
∥ f t o p N N − f ⋆ ∥ L 2 ( X ) ≤ ∥ f t o p N N − f t o p N T K ∥ L 2 ( X ) + ∥ f t o p N T K − f ⋆ ∥ L 2 ( X ) ≤ ∥ f t o p N N − f t o p N T K ∥ ∞ + ∥ f t o p N T K − f ⋆ ∥ L 2 ( X ) . (60) \left\|f^{\mathrm{NN}}_{t_{\mathrm{op}}}-f^\star\right\|_{L^2(\mathcal{X})}
\leq
\left\|f^{\mathrm{NN}}_{t_{\mathrm{op}}}-f^{\mathrm{NTK}}_{t_{\mathrm{op}}}\right\|_{L^2(\mathcal{X})}
+
\left\|f^{\mathrm{NTK}}_{t_{\mathrm{op}}}-f^\star\right\|_{L^2(\mathcal{X})}
\leq
\left\|f^{\mathrm{NN}}_{t_{\mathrm{op}}}-f^{\mathrm{NTK}}_{t_{\mathrm{op}}}\right\|_\infty
+
\left\|f^{\mathrm{NTK}}_{t_{\mathrm{op}}}-f^\star\right\|_{L^2(\mathcal{X})}. \tag*{(60)} f t op NN − f ⋆ L 2 ( X ) ≤ f t op NN − f t op NTK L 2 ( X ) + f t op NTK − f ⋆ L 2 ( X ) ≤ f t op NN − f t op NTK ∞ + f t op NTK − f ⋆ L 2 ( X ) . (60)
Applying Lemma E.21, we obtain under Assumptions a to d that there exists a constant C 1 > 0 C_1>0 C 1 > 0 such that, for any δ ∈ ( 0 , 1 ) \delta\in(0,1) δ ∈ ( 0 , 1 ) , the following holds with probability at least 1 − δ 2 1-\frac{\delta}{2} 1 − 2 δ with respect to random samples:
∥ f t o p N T K − f ⋆ ∥ L 2 ( X ) ≤ C 1 ( ln 6 δ ) 2 n − 1 2 s ( p + 1 ) s ( p + 1 ) + p , (61) \left\|f^{\mathrm{NTK}}_{t_{\mathrm{op}}}-f^\star\right\|_{L^2(\mathcal{X})}\leq C_1\left(\ln\frac{6}{\delta}\right)^2n^{-\frac{1}{2}\frac{s(p+1)}{s(p+1)+p}}, \tag*{(61)} f t op NTK − f ⋆ L 2 ( X ) ≤ C 1 ( ln δ 6 ) 2 n − 2 1 s ( p + 1 ) + p s ( p + 1 ) , (61)
where C 1 > 0 C_1>0 C 1 > 0 is a constant independent of δ \delta δ , n n n , and t o p t_{\mathrm{op}} t op , and that depends on s s s , p p p , A A A , B B B , M μ M_\mu M μ , and R R R .
By Lemma E.19, under Assumptions a, b and d, we know that there exist a constant C 2 > 0 C_2>0 C 2 > 0 and a polynomial P P P such that, for any δ ∈ ( 0 , 1 ) \delta\in(0,1) δ ∈ ( 0 , 1 ) , when m ≥ P ( n , 1 / δ , ln n , ln ( 1 / δ ) ) m\geq P(n,1/\delta,\ln n,\ln(1/\delta)) m ≥ P ( n , 1/ δ , ln n , ln ( 1/ δ )) , with probability at least 1 − δ 2 1-\frac{\delta}{2} 1 − 2 δ with respect to random samples and random initialization, we have:
∥ f t o p N N − f t o p N T K ∥ ∞ ≤ sup ∥ x ∥ ≤ A ∣ f t o p N N ( x ) − f t o p N T K ( x ) ∣ ≤ C 2 m − 1 24 ln m , \left\|f^{\mathrm{NN}}_{t_{\mathrm{op}}}-f^{\mathrm{NTK}}_{t_{\mathrm{op}}}\right\|_\infty\leq\sup_{\lVert x\rVert\leq A}\left|f^{\mathrm{NN}}_{t_{\mathrm{op}}}(x)-f^{\mathrm{NTK}}_{t_{\mathrm{op}}}(x)\right|\leq C_2m^{-\frac{1}{24}}\sqrt{\ln m}, f t op NN − f t op NTK ∞ ≤ ∥ x ∥ ≤ A sup f t op NN ( x ) − f t op NTK ( x ) ≤ C 2 m − 24 1 ln m ,
where the constant C 2 > 0 C_2>0 C 2 > 0 is independent of δ \delta δ and n n n , but depends on A A A , and the coefficients of P P P depend on A A A , B B B , M μ M_\mu M μ , and p p p . Choosing m ≥ max ( P ( n , 1 / δ , ln n , ln ( 1 / δ ) ) , n 24 ( ln n ) 12 ) m\geq\max\left(P(n,1/\delta,\ln n,\ln(1/\delta)),n^{24}(\ln n)^{12}\right) m ≥ max ( P ( n , 1/ δ , ln n , ln ( 1/ δ )) , n 24 ( ln n ) 12 ) yields m − 1 24 ln m ≤ 6 n − 1 m^{-\frac{1}{24}}\sqrt{\ln m}\leq6n^{-1} m − 24 1 ln m ≤ 6 n − 1 by Lemma I.8 since n ≥ 3 n\geq3 n ≥ 3 . This implies that:
∥ f t o p N N − f t o p N T K ∥ ∞ ≤ 6 C 2 n − 1 . (62) \left\|f^{\mathrm{NN}}_{t_{\mathrm{op}}}-f^{\mathrm{NTK}}_{t_{\mathrm{op}}}\right\|_\infty\leq6C_2n^{-1}. \tag*{(62)} f t op NN − f t op NTK ∞ ≤ 6 C 2 n − 1 . (62)
Substituting the bounds in Equations (61) and (62) into Equation (60) , we obtain that:
∥ f t o p N N − f ⋆ ∥ L 2 ( X ) ≤ 6 C 2 n − 1 + C 1 ( ln 6 δ ) 2 n − 1 2 s ( p + 1 ) s ( p + 1 ) + p . \left\|f^{\mathrm{NN}}_{t_{\mathrm{op}}}-f^\star\right\|_{L^2(\mathcal{X})}\leq6C_2n^{-1}+C_1\left(\ln\frac{6}{\delta}\right)^2n^{-\frac{1}{2}\frac{s(p+1)}{s(p+1)+p}}. f t op NN − f ⋆ L 2 ( X ) ≤ 6 C 2 n − 1 + C 1 ( ln δ 6 ) 2 n − 2 1 s ( p + 1 ) + p s ( p + 1 ) .
Noticing that n − 1 ≤ n − 1 2 s ( p + 1 ) s ( p + 1 ) + p n^{-1} \le n^{-\frac{1}{2}\frac{s(p+1)}{s(p+1)+p}} n − 1 ≤ n − 2 1 s ( p + 1 ) + p s ( p + 1 ) , we get:
∥ f t o p N N − f ∗ ∥ L 2 ( X ) ≤ ( 6 C 2 + C 1 ( ln 6 δ ) 2 ) n − 1 2 s ( p + 1 ) s ( p + 1 ) + p . \left\|f^{\mathrm{NN}}_{t_{\mathrm{op}}}-f^*\right\|_{L^2(\mathcal{X})}\leq\left(6C_2+C_1\left(\ln\frac{6}{\delta}\right)^2\right)n^{-\frac{1}{2}\frac{s(p+1)}{s(p+1)+p}}. f t op NN − f ∗ L 2 ( X ) ≤ ( 6 C 2 + C 1 ( ln δ 6 ) 2 ) n − 2 1 s ( p + 1 ) + p s ( p + 1 ) .
Given that δ ∈ ( 0 , 1 ) \delta\in(0,1) δ ∈ ( 0 , 1 ) , it holds that ln 6 δ > ln 6 > 1 \ln\frac{6}{\delta}>\ln6>1 ln δ 6 > ln 6 > 1 . Hence, we get ( ln 6 δ ) 2 > 1 \left(\ln\frac{6}{\delta}\right)^2>1 ( ln δ 6 ) 2 > 1 . Therefore, 6 C 2 ≤ 6 C 2 ( ln 6 δ ) 2 6C_2\leq6C_2\left(\ln\frac{6}{\delta}\right)^2 6 C 2 ≤ 6 C 2 ( ln δ 6 ) 2 . As a result, combining the terms gives:
∥ f t o p N N − f ∗ ∥ L 2 ( X ) ≤ ( 6 C 2 + C 1 ) ( ln 6 δ ) 2 n − 1 2 s ( p + 1 ) s ( p + 1 ) + p . \left\|f^{\mathrm{NN}}_{t_{\mathrm{op}}}-f^*\right\|_{L^2(\mathcal{X})}\leq\left(6C_2+C_1\right)\left(\ln\frac{6}{\delta}\right)^2n^{-\frac{1}{2}\frac{s(p+1)}{s(p+1)+p}}. f t op NN − f ∗ L 2 ( X ) ≤ ( 6 C 2 + C 1 ) ( ln δ 6 ) 2 n − 2 1 s ( p + 1 ) + p s ( p + 1 ) .
Defining the constant C : = 6 C 2 + C 1 C:=6C_2+C_1 C := 6 C 2 + C 1 establishes the desired convergence rate. Since we want both bounds in Equations (61) and (62) to hold simultaneously, then, using a union bound, they both hold with probability at least 1 − δ 1-\delta 1 − δ . We conclude the proof by choosing m ≥ poly ( n , 1 / δ , ln n , ln ( 1 / δ ) ) m\geq\operatorname{poly}(n,1/\delta,\ln n,\ln(1/\delta)) m ≥ poly ( n , 1/ δ , ln n , ln ( 1/ δ )) , where poly is a polynomial that dominates max ( P ( n , 1 / δ , ln n , ln ( 1 / δ ) ) , n 24 ( ln n ) 12 ) \max\left(P(n,1/\delta,\ln n,\ln(1/\delta)),n^{24}(\ln n)^{12}\right) max ( P ( n , 1/ δ , ln n , ln ( 1/ δ )) , n 24 ( ln n ) 12 ) .
Expected generalization bounds
Deriving generalization bounds in expectation is essential to characterize the convergence rate of optimization algorithms, whose guarantees are naturally stated in expectation. However, the standard approach of converting high-probability bounds to expected bounds via E [ X ] = ∫ 0 ∞ Pr ( X > t ) d t \mathbb{E}[X]=\int_0^\infty\Pr(X>t)\mathrm{d}t E [ X ] = ∫ 0 ∞ Pr ( X > t ) d t for X ≥ 0 X\geq0 X ≥ 0 does not apply here, as the required network width m m m depends on δ \delta δ (see Proposition E.22 ) in a way that prevents a direct integration over δ \delta δ . To overcome this, we introduce truncated neural network predictors, obtained by setting the networks to zero on a carefully defined low-probability event, which allows us to pass from high-probability bounds to expected bounds while maintaining control over the contribution of the low-probability event. To this end, we introduce six events that will be used to control the generalization error of the early-stopped neural network in expectation.
High-probability events governing the generalization error. Recall that f t o p N N f^{\mathrm{NN}}_{t_{\mathrm{op}}} f t op NN denotes the neural network trained by gradient flow and stopped at time t o p t_{\mathrm{op}} t op . To analyze the generalization error of f t o p N N f^{\mathrm{NN}}_{t_{\mathrm{op}}} f t op NN , we decompose it as:
∥ f t o p N N − f ∗ ∥ L 2 ( X ) ≤ ∥ f t o p N N − f t o p N T K ∥ L 2 ( X ) + ∥ f t o p N T K − f ∗ ∥ L 2 ( X ) , \left\|f^{\mathrm{NN}}_{t_{\mathrm{op}}}-f^*\right\|_{L^2(\mathcal{X})}\leq\left\|f^{\mathrm{NN}}_{t_{\mathrm{op}}}-f^{\mathrm{NTK}}_{t_{\mathrm{op}}}\right\|_{L^2(\mathcal{X})}+\left\|f^{\mathrm{NTK}}_{t_{\mathrm{op}}}-f^*\right\|_{L^2(\mathcal{X})}, f t op NN − f ∗ L 2 ( X ) ≤ f t op NN − f t op NTK L 2 ( X ) + f t op NTK − f ∗ L 2 ( X ) ,
where f t o p N T K f^{\mathrm{NTK}}_{t_{\mathrm{op}}} f t op NTK denotes the corresponding NTK gradient flow stopped at the same time t o p t_{\mathrm{op}} t op . The first term measures how well the finite-width neural network remains uniformly close to its infinite-width NTK surrogate, while the second term is the estimation error of the NTK gradient flow itself. The following are the two events under which each of these two terms can be made small.
E N T K = { ∥ f t o p N T K − f ∗ ∥ L 2 ( X ) ≤ C 1 ( ln 6 δ ) 2 n − 1 2 s ( p + 1 ) s ( p + 1 ) + p } \mathcal{E}_{\mathrm{NTK}}=\left\{\left\|f^{\mathrm{NTK}}_{t_{\mathrm{op}}}-f^*\right\|_{L^2(\mathcal{X})}\leq C_1\left(\ln\frac{6}{\delta}\right)^2n^{-\frac{1}{2}\frac{s(p+1)}{s(p+1)+p}}\right\} E NTK = { f t op NTK − f ∗ L 2 ( X ) ≤ C 1 ( ln δ 6 ) 2 n − 2 1 s ( p + 1 ) + p s ( p + 1 ) } , where δ ∈ ( 0 , 1 ) \delta\in(0,1) δ ∈ ( 0 , 1 ) and C 1 > 0 C_1>0 C 1 > 0 is a constant that is independent of δ \delta δ , n n n , and t o p t_{\mathrm{op}} t op , but depends on s s s , p p p , A A A , B B B , M μ M_\mu M μ , and R R R . It is the event that the NTK gradient flow achieves the target estimation rate. By Lemma E.21 , E N T K \mathcal{E}_{\mathrm{NTK}} E NTK holds with probability at least 1 − δ 1-\delta 1 − δ with respect to random samples.
E N N = { ∥ f t o p N N − f t o p N T K ∥ L 2 ( X ) ≤ C 2 m − 1 4 ln m } \mathcal{E}_{\mathrm{NN}}=\left\{\left\|f^{\mathrm{NN}}_{t_{\mathrm{op}}}-f^{\mathrm{NTK}}_{t_{\mathrm{op}}}\right\|_{L^2(\mathcal{X})}\leq C_2m^{-\frac{1}{4}}\sqrt{\ln m}\right\} E NN = { f t op NN − f t op NTK L 2 ( X ) ≤ C 2 m − 4 1 ln m } , where C 2 > 0 C_2>0 C 2 > 0 is a constant that is independent of δ \delta δ , n n n and t o p t_{\mathrm{op}} t op with δ ∈ ( 0 , 1 ) \delta\in(0,1) δ ∈ ( 0 , 1 ) , but depends on A A A . By Lemma E.19, for n ≥ 3 n\geq3 n ≥ 3 , E N N \mathcal{E}_{\mathrm{NN}} E NN holds with probability at least 1 − δ 1-\delta 1 − δ with respect to random samples and random initialization, provided that m ≥ poly ( n , 1 / δ , ln n , ln ( 1 / δ ) ) m\geq\operatorname{poly}(n,1/\delta,\ln n,\ln(1/\delta)) m ≥ poly ( n , 1/ δ , ln n , ln ( 1/ δ )) , where the coefficients of poly depend on A A A , B B B , M μ M_\mu M μ , and p p p .
Rather than bounding the probability of E N N \mathcal{E}_{\mathrm{NN}} E NN directly, we instead exhibit the following events, each depending only on the random initialization or the empirical NTK at initialization, and whose intersection is included in E N N \mathcal{E}_{\mathrm{NN}} E NN , as we discuss in Remark E.23.
E θ 0 = { max 1 ≤ i ≤ m max ( ∣ v 0 , i ∣ , ∣ b 0 , i ∣ , ∥ u 0 , i ∥ ∞ ) ≤ 3 ln m } \mathcal{E}_{\theta_0}=\left\{\max_{1\leq i\leq m}\max\left(\left|v_{0,i}\right|,\left|b_{0,i}\right|,\left\|\boldsymbol{u}_{0,i}\right\|_{\infty}\right)\leq\sqrt{3\ln m}\right\} E θ 0 = { max 1 ≤ i ≤ m max ( ∣ v 0 , i ∣ , ∣ b 0 , i ∣ , ∥ u 0 , i ∥ ∞ ) ≤ 3 ln m } is the event that no parameter of the initialization θ 0 \theta_0 θ 0 is not atypically large, ensuring that the parameters remain in a controlled neighborhood of θ 0 \theta_0 θ 0 during training. By Lemma E.16 , this event holds with probability at least 1 − p + 2 2 π m − 1 / 2 1-\frac{p+2}{\sqrt{2\pi}}m^{-1/2} 1 − 2 π p + 2 m − 1/2 with respect to random initialization.
R = { Card ( { i ∈ { 1 , … , m } : ∣ u ~ 0 , i T x ~ ∣ ≤ 2 π 4 m − η } ) ≤ ⌊ m κ ⌋ , ∀ x ∈ X } \mathcal{R} = \left\{ \operatorname{Card}\left(\left\{i \in\{1,\ldots,m\}: \left|\tilde{\mathbf{u}}_{0,i}^{\mathsf{T}}\tilde{x}\right| \le\frac{\sqrt{2\pi}}{4}m^{-\eta}\right\}\right) \le\left\lfloor m^{\kappa}\right\rfloor,\ \forall x \in\mathcal{X}\right\} R = { Card ( { i ∈ { 1 , … , m } : u ~ 0 , i T x ~ ≤ 4 2 π m − η } ) ≤ ⌊ m κ ⌋ , ∀ x ∈ X } , where u ~ 0 , i = [ u 0 , i T , b 0 , i ] T \tilde{\mathbf{u}}_{0,i} = [\mathbf{u}_{0,i}^{\mathsf{T}},b_{0,i}]^{\mathsf{T}} u ~ 0 , i = [ u 0 , i T , b 0 , i ] T are the augmented first-layer weights at initialization, x ~ = [ x T , 1 ] T \tilde{x} = [x^{\mathsf{T}},1]^{\mathsf{T}} x ~ = [ x T , 1 ] T is the augmented input, 0 < η < ϱ ≤ 1 0 < \eta< \varrho\le1 0 < η < ϱ ≤ 1 , τ > 1 2 \tau> \frac{1}{2} τ > 2 1 , and κ > max ( 1 − η , τ ) \kappa> \max(1-\eta,\tau) κ > max ( 1 − η , τ ) . It is the event that, uniformly over all inputs, at most ⌊ m κ ⌋ \left\lfloor m^{\kappa}\right\rfloor ⌊ m κ ⌋ neurons have a small pre-activation value ∣ u ~ 0 , i T x ~ ∣ \left|\tilde{\mathbf{u}}_{0,i}^{\mathsf{T}}\tilde{x}\right| u ~ 0 , i T x ~ at initialization. For the remaining neurons, the pre-activation value is large enough that changing their activation pattern would require the parameters to travel far from their initialization. Consequently, R \mathcal{R} R guarantees that the activation pattern of the majority of neurons is effectively frozen throughout training, which is precisely the condition needed for the network to operate in the lazy training NTK regime . By [58 ] , Lemma C.2, when the width m m m is sufficiently large such that:
m 1 − η + m τ ≤ ⌊ m κ ⌋ and 2 A p ln m ⌊ m ϱ ⌋ ≤ 2 π 4 m − η , (63) m^{1-\eta}+m^{\tau}\le\left\lfloor m^{\kappa}\right\rfloor\qquad\text{and} \qquad\frac{2Ap\sqrt{\ln m}}{\left\lfloor m^{\varrho}\right\rfloor}\le\frac{\sqrt{2\pi}}{4}m^{-\eta}, \tag*{(63)} m 1 − η + m τ ≤ ⌊ m κ ⌋ and ⌊ m ϱ ⌋ 2 A p ln m ≤ 4 2 π m − η , (63)
the event R \mathcal{R} R holds with probability at least 1 − m p ϱ exp ( − 2 m 2 τ − 1 ) 1-m^{p\varrho}\exp\left(-2m^{2\tau-1}\right) 1 − m pϱ exp ( − 2 m 2 τ − 1 ) with respect to random initialization. By Lemma I.10 , the conditions on m m m in Equation (63) are satisfied when m m m is sufficiently large such that m ≥ max ( 2 1 2 ℓ , ( 128 A 2 p 2 π ( ϱ − η ) 2 ) 1 κ − max ( 1 − η , τ ) , 3 1 κ − max ( 1 − η , τ ) ) m \ge\max\left(2^{\frac{1}{2\ell}},\left(\frac{128A^{2}p^{2}}{\pi(\varrho-\eta)^{2}}\right)^{\frac{1}{\kappa-\max(1-\eta,\tau)}},3^{\frac{1}{\kappa-\max(1-\eta,\tau)}}\right) m ≥ max ( 2 2 ℓ 1 , ( π ( ϱ − η ) 2 128 A 2 p 2 ) κ − m a x ( 1 − η , τ ) 1 , 3 κ − m a x ( 1 − η , τ ) 1 ) .
C = { sup z , z ′ ∈ N ϵ ∣ K 0 m ( z , z ′ ) − K N T K ( z , z ′ ) ∣ ≤ C 3 ln m m } \mathcal{C} = \left\{\sup_{z,z'\in\mathcal{N}_{\epsilon}}\left|K_{0}^{m}(z,z')-K^{\mathrm{NTK}}(z,z')\right|\le C_{3}\sqrt{\frac{\ln m}{m}}\right\} C = { sup z , z ′ ∈ N ϵ K 0 m ( z , z ′ ) − K NTK ( z , z ′ ) ≤ C 3 m l n m } , where N ϵ \mathcal{N}_{\epsilon} N ϵ denotes a uniform grid on the compact X \mathcal{X} X with mesh size ϵ = 2 A ⌊ m ϱ ⌋ \epsilon=\frac{2A}{\lfloor m^{\varrho}\rfloor} ϵ = ⌊ m ϱ ⌋ 2 A where 0 < ϱ ≤ 1 0<\varrho\le1 0 < ϱ ≤ 1 , and C 3 > 0 C_{3}>0 C 3 > 0 is a constant that depends on p p p , A A A , and ϱ \varrho ϱ . It is the event that the empirical NTK at initialization K 0 m K_{0}^{m} K 0 m concentrates uniformly around the infinite-width NTK K N T K K^{\mathrm{NTK}} K NTK over the grid N ϵ \mathcal{N}_{\epsilon} N ϵ with a discrepancy of order ln m m \sqrt{\frac{\ln m}{m}} m l n m . By [58 ] , Lemma C.3, when m m m is large enough such that 3 p ϱ c 0 ln m ≤ m \frac{3p\varrho}{c_{0}}\ln m\le m c 0 3 pϱ ln m ≤ m with c 0 > 0 c_{0}>0 c 0 > 0 being an absolute positive constant, the event C \mathcal{C} C holds with probability at least 1 − 4 m − p ϱ 1-4m^{-p\varrho} 1 − 4 m − pϱ with respect to random initialization. By Lemma I.10 , it is enough to require m ≥ max ( 36 p 2 ϱ 2 c 0 2 , 1 ) m\ge\max\left(\frac{36p^{2}\varrho^{2}}{c_{0}^{2}},1\right) m ≥ max ( c 0 2 36 p 2 ϱ 2 , 1 ) to ensure that 3 p ϱ c 0 ln m ≤ m \frac{3p\varrho}{c_{0}}\ln m\le m c 0 3 pϱ ln m ≤ m .
Controlling the smallest eigenvalue λ 0 \lambda_{0} λ 0 of the infinite-width NTK matrix K N T K K^{\mathrm{NTK}} K NTK is essential for the convergence of the gradient flow. To see this, recall from Lemma E.13 that the explicit solution of the NTK gradient flow at the training points ( X , Y ) = { ( x i , y i ) } i = 1 n (X,Y)=\{(x_i,y_i)\}_{i=1}^{n} ( X , Y ) = {( x i , y i ) } i = 1 n satisfies at t ≥ 0 t\ge0 t ≥ 0 :
∥ f t N T K ( X ) − Y ∥ ≤ exp ( − λ 0 n t ) n B . \left\|f_{t}^{\mathrm{NTK}}(X)-Y\right\|\le\exp\left(-\frac{\lambda_{0}}{n}t\right)\sqrt{n}B. f t NTK ( X ) − Y ≤ exp ( − n λ 0 t ) n B .
A positive lower bound λ 0 ≥ λ ⋆ > 0 \lambda_{0}\ge\lambda_{\star}>0 λ 0 ≥ λ ⋆ > 0 therefore guarantees exponential convergence of the training error at rate λ ⋆ n \frac{\lambda_{\star}}{n} n λ ⋆ . If λ 0 = 0 \lambda_{0}=0 λ 0 = 0 , the training error never decays, and the gradient flow fails to converge entirely. Since the samples x 1 , … , x n x_{1},\ldots,x_{n} x 1 , … , x n are drawn randomly, the smallest eigenvalue λ 0 \lambda_{0} λ 0 of the infinite-width NTK at initialization is itself a random quantity, and must therefore be lower-bounded with high probability, which motivates the following event.
E λ 0 = { λ 0 ≥ λ ⋆ ( n , δ ) } \mathcal{E}_{\lambda_{0}}=\{\lambda_{0}\ge\lambda_{\star}(n,\delta)\} E λ 0 = { λ 0 ≥ λ ⋆ ( n , δ )} , where λ ⋆ ( n , δ ) : = C 4 ( 1 + g ( ϵ ⋆ ( n , δ ) ) ) − 3 ϵ ⋆ ( n , δ ) 4 > 0 \lambda_{\star}(n,\delta):=C_{4}(1+g(\epsilon_{\star}(n,\delta)))^{-3}\epsilon_{\star}(n,\delta)^{4}>0 λ ⋆ ( n , δ ) := C 4 ( 1 + g ( ϵ ⋆ ( n , δ )) ) − 3 ϵ ⋆ ( n , δ ) 4 > 0 , ϵ ⋆ ( n , δ ) : = min ( ϵ A , ϵ 0 ( n , δ ) ) ∈ ( 0 , 1 ] \epsilon_{\star}(n,\delta):=\min(\epsilon_{A},\epsilon_{0}(n,\delta))\in(0,1] ϵ ⋆ ( n , δ ) := min ( ϵ A , ϵ 0 ( n , δ )) ∈ ( 0 , 1 ] , ϵ A : = min ( 1 , 2 A 2 + 1 ) \epsilon_{A}:=\min\left(1,\frac{2}{\sqrt{A^{2}+1}}\right) ϵ A := min ( 1 , A 2 + 1 2 ) , ϵ 0 ( n , δ ) : = 2 π ( 2 δ M μ ( A 2 + 1 ) ( p + 1 ) / 2 v p n 2 ) 1 / p \epsilon_{0}(n,\delta):=\frac{2}{\pi}\left(\frac{2\delta}{M_{\mu}(A^{2}+1)(p+1)/2v_{p}n^{2}}\right)^{1/p} ϵ 0 ( n , δ ) := π 2 ( M μ ( A 2 + 1 ) ( p + 1 ) /2 v p n 2 2 δ ) 1/ p , g ( ϵ ) : = ( p + 1 ) ln ( 1 / ϵ ) ln ( p + 1 ) g(\epsilon):=\frac{(p+1)\ln(1/\epsilon)}{\ln(p+1)} g ( ϵ ) := l n ( p + 1 ) ( p + 1 ) l n ( 1/ ϵ ) , δ ∈ ( 0 , 1 ) \delta\in(0,1) δ ∈ ( 0 , 1 ) , v p = Vol ( B ( 0 , 1 ) ) v_{p}=\operatorname{Vol}(B(0,1)) v p = Vol ( B ( 0 , 1 )) is the volume of the unit ball in R p \mathbb{R}^{p} R p , M μ M_{\mu} M μ is the bound on the probability density function f μ f_{\mu} f μ , and C 4 > 0 C_{4}>0 C 4 > 0 is an absolute positive constant. It is the event that the smallest eigenvalue of the infinite-width NTK at initialization is bounded away from zero, a necessary condition for the gradient flow to converge. By Lemma E.9, E λ 0 \mathcal{E}_{\lambda_{0}} E λ 0 holds with probability at least 1 − δ 1-\delta 1 − δ with respect to random samples.
Remark E.23 (Inclusion of events). According to [58 ] , Proof of Proposition 3.3, we have the following inclusion:
E λ 0 ∩ R ∩ C ⊆ { sup t ≥ 0 sup ∥ x ∥ ≤ A , ∥ x ′ ∥ ≤ A ∣ K t m ( x , x ′ ) − K N T K ( x , x ′ ) ∣ ≤ C m − 1 12 ln m } . \mathcal{E}_{\lambda_{0}}\cap\mathcal{R}\cap\mathcal{C}\subseteq\left\{\sup_{t\ge0}\sup_{\lVert x\rVert\le A,\lVert x'\rVert\le A}\left|K_{t}^{m}(x,x')-K^{\mathrm{NTK}}(x,x')\right|\le Cm^{-\frac{1}{12}}\sqrt{\ln m}\right\}. E λ 0 ∩ R ∩ C ⊆ { t ≥ 0 sup ∥ x ∥ ≤ A , ∥ x ′ ∥ ≤ A sup K t m ( x , x ′ ) − K NTK ( x , x ′ ) ≤ C m − 12 1 ln m } .
Furthermore, in the proof of Lemma E.19, we show that:
{ sup t ≥ 0 sup ∥ x ∥ ≤ A , ∥ x ′ ∥ ≤ A ∣ K t m ( x , x ′ ) − K N T K ( x , x ′ ) ∣ ≤ C m − 1 12 ln m } ∩ E λ 0 ⊂ E N N . \left\{\sup_{t\ge0}\sup_{\lVert x\rVert\le A,\lVert x'\rVert\le A}\left|K_t^m(x,x')-K^{\mathrm{NTK}}(x,x')\right|\le C m^{-\frac{1}{12}}\sqrt{\ln m}\right\}\cap\mathcal{E}_{\lambda_0}\subset\mathcal{E}_{\mathrm{NN}}. { t ≥ 0 sup ∥ x ∥ ≤ A , ∥ x ′ ∥ ≤ A sup K t m ( x , x ′ ) − K NTK ( x , x ′ ) ≤ C m − 12 1 ln m } ∩ E λ 0 ⊂ E NN .
Consequently, we obtain E : = E θ 0 ∩ R ∩ C ∩ E λ 0 ⊂ E N N \mathcal{E}:=\mathcal{E}_{\theta_0}\cap\mathcal{R}\cap\mathcal{C}\cap\mathcal{E}_{\lambda_0}\subset\mathcal{E}_{\mathrm{NN}} E := E θ 0 ∩ R ∩ C ∩ E λ 0 ⊂ E NN . This inclusion motivates the following definition, in which we truncate the neural network on E \mathcal{E} E rather than on E N N \mathcal{E}_{\mathrm{NN}} E NN directly, since E \mathcal{E} E is contained in E N N \mathcal{E}_{\mathrm{NN}} E NN and its probability can be bounded explicitly from its four constituent parts.
Definition E.24 (Truncated neural network). Let E = E θ 0 ∩ R ∩ C ∩ E λ 0 \mathcal{E}=\mathcal{E}_{\theta_0}\cap\mathcal{R}\cap\mathcal{C}\cap\mathcal{E}_{\lambda_0} E = E θ 0 ∩ R ∩ C ∩ E λ 0 . For a chosen stopping time t o p t_{\mathrm{op}} t op , the truncated neural network associated with f t o p N N f_{t_{\mathrm{op}}}^{\mathrm{NN}} f t op NN is defined by f ~ t o p N N : = f t o p N N 1 E \widetilde{f}_{t_{\mathrm{op}}}^{\mathrm{NN}}:=f_{t_{\mathrm{op}}}^{\mathrm{NN}}\mathbf{1}_{\mathcal{E}} f t op NN := f t op NN 1 E .
The event E \mathcal{E} E and the truncated predictor depend on a deterministic confidence parameter δ ∈ ( 0 , 1 ) \delta\in(0,1) δ ∈ ( 0 , 1 ) , through λ ⋆ ( n , δ ) \lambda_{\star}(n,\delta) λ ⋆ ( n , δ ) . The inner and outer estimators in Equation (7) choose δ \delta δ to be the corresponding minimizer δ ⋆ \delta^\star δ ⋆ , specified in Equations (29) and (69) in the proofs of Propositions D.4 and E.25, respectively.
By truncating the neural network on E \mathcal{E} E , the truncated predictor f ~ t o p N N \widetilde{f}_{t_{\mathrm{op}}}^{\mathrm{NN}} f t op NN vanishes on E C \mathcal{E}^{\mathrm{C}} E C , which allows us to pass from the high-probability bound of Proposition E.22 to the expected bound of Proposition E.25 by decomposing the expected error into a contribution on E \mathcal{E} E , controlled by the high-probability bound, and a contribution on E C \mathcal{E}^{\mathrm{C}} E C , which vanishes by definition of the truncation. The explicit probability bounds on each constituent event of E \mathcal{E} E are then used to show that Pr ( E C ) \Pr(\mathcal{E}^{\mathrm{C}}) Pr ( E C ) is small enough to be absorbed into the overall rate.
Proposition E.25 (Expected generalization bound). Consider n n n i.i.d. samples { ( x i , y i ) } i = 1 n ∼ ρ \{(x_i,y_i)\}_{i=1}^n\sim\rho {( x i , y i ) } i = 1 n ∼ ρ and fix p ≥ 2 p\ge2 p ≥ 2 . Let Assumptions a a a to d d d hold. If we choose a stopping time t o p = n s ( p + 1 ) + p s ( p + 1 ) + p t_{\mathrm{op}}=n^{\frac{s(p+1)+p}{s(p+1)+p}} t op = n s ( p + 1 ) + p s ( p + 1 ) + p with 1 p + 1 < s ≤ 2 \frac{1}{p+1}<s\le2 p + 1 1 < s ≤ 2 , then, considering the early-stopped truncated neural network predictor f ~ t o p N N \widetilde{f}_{t_{\mathrm{op}}}^{\mathrm{NN}} f t op NN for the target function f ⋆ f^\star f ⋆ , there exist a constant C > 0 C>0 C > 0 and a polynomial poly \operatorname{poly} poly such that when
n ≥ max ( 3 , ⌈ ( 3 C 1 2 ε 3 2 ( κ N T K ) 2 B 2 ) s ( p + 1 ) + p ( s + 2 ) ( p + 1 ) + p ⌉ , ⌈ ( 1 3 ( κ N T K ) 2 ) s ( p + 1 ) + p 2 ( p + 1 ) ⌉ , ⌈ ( C 1 2 ( ln 6 ) 3 3 ( κ N T K ) 2 B 2 ) s ( p + 1 ) + p ( s + 2 ) ( p + 1 ) + p ⌉ + 1 ) , n\ge\max\left(3,\left\lceil\left(\frac{3C_1^2\varepsilon^3}{2(\kappa^{\mathrm{NTK}})^2B^2}\right)^{\frac{s(p+1)+p}{(s+2)(p+1)+p}}\right\rceil,\left\lceil\left(\frac{1}{3(\kappa^{\mathrm{NTK}})^2}\right)^{\frac{s(p+1)+p}{2(p+1)}}\right\rceil,\left\lceil\left(\frac{C_1^2(\ln6)^3}{3(\kappa^{\mathrm{NTK}})^2B^2}\right)^{\frac{s(p+1)+p}{(s+2)(p+1)+p}}\right\rceil+1\right), n ≥ max 3 , ( 2 ( κ NTK ) 2 B 2 3 C 1 2 ε 3 ) ( s + 2 ) ( p + 1 ) + p s ( p + 1 ) + p , ( 3 ( κ NTK ) 2 1 ) 2 ( p + 1 ) s ( p + 1 ) + p , ( 3 ( κ NTK ) 2 B 2 C 1 2 ( ln 6 ) 3 ) ( s + 2 ) ( p + 1 ) + p s ( p + 1 ) + p + 1 ,
and the width satisfies m ≥ poly ( n , ln n ) m\ge\operatorname{poly}(n,\ln n) m ≥ poly ( n , ln n ) , the following holds:
E [ ∥ f ~ t o p N N − f ⋆ ∥ L 2 ( X ) 2 ] ≤ C ( ln n ) 4 n − s ( p + 1 ) s ( p + 1 ) + p , \mathbb{E}\left[\left\|\widetilde{f}_{t_{\mathrm{op}}}^{\mathrm{NN}}-f^\star\right\|_{L^2(\mathcal{X})}^2\right]\le C(\ln n)^4n^{-\frac{s(p+1)}{s(p+1)+p}}, E [ f t op NN − f ⋆ L 2 ( X ) 2 ] ≤ C ( ln n ) 4 n − s ( p + 1 ) + p s ( p + 1 ) ,
where κ N T K : = ( p + 2 ) ( A 2 + 1 ) + 1 \kappa^{\mathrm{NTK}}:=(p+2)(A^2+1)+1 κ NTK := ( p + 2 ) ( A 2 + 1 ) + 1 , the constant C > 0 C>0 C > 0 is independent of n n n and t o p t_{\mathrm{op}} t op , but depends on s s s , p p p , A A A , B B B , M μ M_\mu M μ , and R R R , and the coefficients of poly \operatorname{poly} poly depend on s s s , p p p , A A A , B B B , M μ M_\mu M μ , and R R R . The expectation here is with respect to random samples and random initialization.
Proof. By the triangle inequality and the identity ( a + b ) 2 ≤ 2 ( a 2 + b 2 ) (a+b)^2\le2(a^2+b^2) ( a + b ) 2 ≤ 2 ( a 2 + b 2 ) , we have:
E [ ∥ f ~ t o p N N − f ⋆ ∥ L 2 ( X ) 2 ] ≤ 2 ( E [ ∥ f ~ t o p N N − f t o p N T K ∥ L 2 ( X ) 2 ] + E [ ∥ f t o p N T K − f ⋆ ∥ L 2 ( X ) 2 ] ) . (64) \mathbb{E}\left[\left\|\widetilde{f}_{t_{\mathrm{op}}}^{\mathrm{NN}}-f^\star\right\|_{L^2(\mathcal{X})}^2\right]\le2\left(\mathbb{E}\left[\left\|\widetilde{f}_{t_{\mathrm{op}}}^{\mathrm{NN}}-f_{t_{\mathrm{op}}}^{\mathrm{NTK}}\right\|_{L^2(\mathcal{X})}^2\right]+\mathbb{E}\left[\left\|f_{t_{\mathrm{op}}}^{\mathrm{NTK}}-f^\star\right\|_{L^2(\mathcal{X})}^2\right]\right). \tag*{(64)} E [ f t op NN − f ⋆ L 2 ( X ) 2 ] ≤ 2 ( E [ f t op NN − f t op NTK L 2 ( X ) 2 ] + E [ f t op NTK − f ⋆ L 2 ( X ) 2 ] ) . (64)
Next, we upper-bound each expectation on the right-hand side of the inequality, and then combine the resulting bounds.
Upper-bound on E [ ∥ f ~ t o p N N − f t o p N T K ∥ L 2 ( X ) 2 ] \mathbb{E}\left[\left\|\widetilde{f}_{t_{\mathrm{op}}}^{\mathrm{NN}}-f_{t_{\mathrm{op}}}^{\mathrm{NTK}}\right\|_{L^2(\mathcal{X})}^2\right] E [ f t op NN − f t op NTK L 2 ( X ) 2 ] . Let E = E θ 0 ∩ R ∩ C ∩ E λ 0 \mathcal{E}=\mathcal{E}_{\theta_0}\cap\mathcal{R}\cap\mathcal{C}\cap\mathcal{E}_{\lambda_0} E = E θ 0 ∩ R ∩ C ∩ E λ 0 . By the law of total expectation, we decompose the error over the disjoint events E \mathcal{E} E and E C \mathcal{E}^{\mathrm{C}} E C as follows:
E [ ∥ f ~ t o p N N − f t o p N T K ∥ L 2 ( X ) 2 ] = E [ ∥ f ~ t o p N N − f t o p N T K ∥ L 2 ( X ) 2 1 E ] + E [ ∥ f ~ t o p N N − f t o p N T K ∥ L 2 ( X ) 2 1 E C ] = E [ ∥ f ~ t o p N N − f t o p N T K ∥ L 2 ( X ) 2 1 E ] + E [ ∥ f ~ t o p N N ∥ L 2 ( X ) 2 1 E C ] ≤ E [ ∥ f ~ t o p N N − f t o p N T K ∥ L 2 ( X ) 2 1 E N N ] + ( κ N T K ) 2 B 2 t o p 2 Pr ( E C ) ≤ C 2 2 m − 1 12 ln m + ( κ N T K ) 2 B 2 n 2 ( p + 1 ) s ( p + 1 ) + p Pr ( E C ) , (65) \begin{aligned}
\mathbb{E}\left[\left\|\widetilde{f}_{t_{\mathrm{op}}}^{\mathrm{NN}}-f_{t_{\mathrm{op}}}^{\mathrm{NTK}}\right\|_{L^2(\mathcal{X})}^2\right]
={}&\mathbb{E}\left[\left\|\widetilde{f}_{t_{\mathrm{op}}}^{\mathrm{NN}}-f_{t_{\mathrm{op}}}^{\mathrm{NTK}}\right\|_{L^2(\mathcal{X})}^2\mathbf{1}_{\mathcal{E}}\right]\\
&+\mathbb{E}\left[\left\|\widetilde{f}_{t_{\mathrm{op}}}^{\mathrm{NN}}-f_{t_{\mathrm{op}}}^{\mathrm{NTK}}\right\|_{L^2(\mathcal{X})}^2\mathbf{1}_{\mathcal{E}^{\mathrm{C}}}\right]\\
={}&\mathbb{E}\left[\left\|\widetilde{f}_{t_{\mathrm{op}}}^{\mathrm{NN}}-f_{t_{\mathrm{op}}}^{\mathrm{NTK}}\right\|_{L^2(\mathcal{X})}^2\mathbf{1}_{\mathcal{E}}\right]\\
&+\mathbb{E}\left[\left\|\widetilde{f}_{t_{\mathrm{op}}}^{\mathrm{NN}}\right\|_{L^2(\mathcal{X})}^2\mathbf{1}_{\mathcal{E}^{\mathrm{C}}}\right]\\
\le{}&\mathbb{E}\left[\left\|\widetilde{f}_{t_{\mathrm{op}}}^{\mathrm{NN}}-f_{t_{\mathrm{op}}}^{\mathrm{NTK}}\right\|_{L^2(\mathcal{X})}^2\mathbf{1}_{\mathcal{E}_{\mathrm{NN}}}\right]+(\kappa^{\mathrm{NTK}})^2B^2t_{\mathrm{op}}^2\Pr\left(\mathcal{E}^{\mathrm{C}}\right)\\
\le{}&C_2^2m^{-\frac{1}{12}}\ln m+(\kappa^{\mathrm{NTK}})^2B^2n^{\frac{2(p+1)}{s(p+1)+p}}\Pr\left(\mathcal{E}^{\mathrm{C}}\right), \tag*{(65)}
\end{aligned} E [ f t op NN − f t op NTK L 2 ( X ) 2 ] = = ≤ ≤ E [ f t op NN − f t op NTK L 2 ( X ) 2 1 E ] + E [ f t op NN − f t op NTK L 2 ( X ) 2 1 E C ] E [ f t op NN − f t op NTK L 2 ( X ) 2 1 E ] + E [ f t op NN L 2 ( X ) 2 1 E C ] E [ f t op NN − f t op NTK L 2 ( X ) 2 1 E NN ] + ( κ NTK ) 2 B 2 t op 2 Pr ( E C ) C 2 2 m − 12 1 ln m + ( κ NTK ) 2 B 2 n s ( p + 1 ) + p 2 ( p + 1 ) Pr ( E C ) , (65)
where, in the second line, we use the definition of the truncated neural network in Definition E.24 which gives f ~ t o p N N = 0 \tilde{f}^{\mathrm{NN}}_{t_{\mathrm{op}}}=0 f ~ t op NN = 0 on E c \mathcal{E}^{c} E c ; in the third line, we use E ⊂ E N N \mathcal{E}\subset\mathcal{E}_{\mathrm{NN}} E ⊂ E NN as explained in Remark E.23, and Lemma E.15 to upper-bound ∥ f t o p N T K ∥ L 2 ( X ) \left\|f^{\mathrm{NTK}}_{t_{\mathrm{op}}}\right\|_{L^{2}(\mathcal{X})} f t op NTK L 2 ( X ) by κ N T K B t o p \kappa^{\mathrm{NTK}}B t_{\mathrm{op}} κ NTK B t op with κ N T K : = ( p + 2 ) ( A 2 + 1 ) + 1 \kappa^{\mathrm{NTK}}:=(p+2)(A^{2}+1)+1 κ NTK := ( p + 2 ) ( A 2 + 1 ) + 1 ; and in the fourth line, we upper-bound ∥ f t o p N N − f t o p N T K ∥ L 2 ( X ) 2 \left\|f^{\mathrm{NN}}_{t_{\mathrm{op}}}-f^{\mathrm{NTK}}_{t_{\mathrm{op}}}\right\|_{L^{2}(\mathcal{X})}^{2} f t op NN − f t op NTK L 2 ( X ) 2 by C 2 2 m − 1 12 ln m C_{2}^{2}m^{-\frac{1}{12}}\ln m C 2 2 m − 12 1 ln m on the event E N N \mathcal{E}_{\mathrm{NN}} E NN and use t o p = n p + 1 s ( p + 1 ) + p t_{\mathrm{op}}=n^{\frac{p+1}{s(p+1)+p}} t op = n s ( p + 1 ) + p p + 1 . Furthermore, by De Morgan’s law and a union bound, we have that:
Pr ( E c ) = Pr ( ( E θ 0 ∩ R ∩ C ∩ E λ 0 ) c ) = Pr ( E θ 0 c ∪ R c ∪ C c ∪ E λ 0 c ) ≤ Pr ( E θ 0 c ) + Pr ( R c ) + Pr ( C c ) + Pr ( E λ 0 c ) ≤ p + 2 2 π m − 1 / 2 + m p ϱ exp ( − 2 m 2 τ − 1 ) + 4 m − p ϱ + δ . \begin{aligned}
\Pr\left(\mathcal{E}^{c}\right)&=\Pr\left(\left(\mathcal{E}_{\theta_{0}}\cap\mathcal{R}\cap\mathcal{C}\cap\mathcal{E}_{\lambda_{0}}\right)^{c}\right)\\
&=\Pr\left(\mathcal{E}_{\theta_{0}}^{c}\cup\mathcal{R}^{c}\cup\mathcal{C}^{c}\cup\mathcal{E}_{\lambda_{0}}^{c}\right)\\
&\leq\Pr\left(\mathcal{E}_{\theta_{0}}^{c}\right)+\Pr\left(\mathcal{R}^{c}\right)+\Pr\left(\mathcal{C}^{c}\right)+\Pr\left(\mathcal{E}_{\lambda_{0}}^{c}\right)\\
&\leq\frac{p+2}{\sqrt{2\pi}}m^{-1/2}+m^{p\varrho}\exp\left(-2m^{2\tau-1}\right)+4m^{-p\varrho}+\delta.
\end{aligned} Pr ( E c ) = Pr ( ( E θ 0 ∩ R ∩ C ∩ E λ 0 ) c ) = Pr ( E θ 0 c ∪ R c ∪ C c ∪ E λ 0 c ) ≤ Pr ( E θ 0 c ) + Pr ( R c ) + Pr ( C c ) + Pr ( E λ 0 c ) ≤ 2 π p + 2 m − 1/2 + m pϱ exp ( − 2 m 2 τ − 1 ) + 4 m − pϱ + δ .
We note that p + 2 2 π m − 1 / 2 ≤ δ \frac{p+2}{\sqrt{2\pi}}m^{-1/2}\leq\delta 2 π p + 2 m − 1/2 ≤ δ and 4 m − p ϱ ≤ δ 4m^{-p\varrho}\leq\delta 4 m − pϱ ≤ δ when we choose m m m large enough such that m ≥ ( p + 2 ) 2 2 π δ 2 m\geq\frac{(p+2)^{2}}{2\pi\delta^{2}} m ≥ 2 π δ 2 ( p + 2 ) 2 and m ≥ ( 4 δ ) 1 p ϱ m\geq\left(\frac{4}{\delta}\right)^{\frac{1}{p\varrho}} m ≥ ( δ 4 ) pϱ 1 , respectively. Moreover, by Lemma I.7 , m p ϱ exp ( − 2 m 2 τ − 1 ) ≤ δ m^{p\varrho}\exp\left(-2m^{2\tau-1}\right)\leq\delta m pϱ exp ( − 2 m 2 τ − 1 ) ≤ δ holds whenever m m m is large enough such that m ≥ max ( 1 , M 1 ( 1 / δ ) ) m\geq\max(1,M_{1}(1/\delta)) m ≥ max ( 1 , M 1 ( 1/ δ )) where:
M 1 ( 1 δ ) = ( [ p ϱ 2 τ − 1 ( ln ( p ϱ 2 τ − 1 ) − 1 ) + ln ( 1 δ ) ] + ) 1 2 τ − 1 , M_{1}\left(\frac{1}{\delta}\right)=\left(\left[\frac{p\varrho}{2\tau-1}\left(\ln\left(\frac{p\varrho}{2\tau-1}\right)-1\right)+\ln\left(\frac{1}{\delta}\right)\right]_{+}\right)^{\frac{1}{2\tau-1}}, M 1 ( δ 1 ) = ( [ 2 τ − 1 pϱ ( ln ( 2 τ − 1 pϱ ) − 1 ) + ln ( δ 1 ) ] + ) 2 τ − 1 1 ,
with [ t ] + : = max ( t , 0 ) [t]_{+}:=\max(t,0) [ t ] + := max ( t , 0 ) .
R \mathcal{R} R and C \mathcal{C} C hold with high-probability when m ≥ max ( 2 1 ϵ , ( 128 A 2 p 2 π ( ϱ − η ) ) 1 ϵ − η , 3 1 κ − max ( 1 − η , τ ) ) m\geq\max\left(2^{\frac{1}{\epsilon}},\left(\frac{128A^{2}p^{2}}{\pi(\varrho-\eta)}\right)^{\frac{1}{\epsilon-\eta}},3^{\frac{1}{\kappa-\max(1-\eta,\tau)}}\right) m ≥ max ( 2 ϵ 1 , ( π ( ϱ − η ) 128 A 2 p 2 ) ϵ − η 1 , 3 κ − m a x ( 1 − η , τ ) 1 ) and m ≥ max ( 36 p 2 ϱ 2 c 0 2 , 1 ) m\geq\max\left(\frac{36p^{2}\varrho^{2}}{c_{0}^{2}},1\right) m ≥ max ( c 0 2 36 p 2 ϱ 2 , 1 ) , respectively, where c 0 c_{0} c 0 is an absolute positive constant, as discussed in Items 4 and 5. Combining all these conditions, we conclude that when m m m is large enough such that m ≥ M 2 ( 1 / δ ) m\geq M_{2}(1/\delta) m ≥ M 2 ( 1/ δ ) where:
M 2 ( 1 δ ) = max ( ( p + 2 ) 2 2 π δ 2 , ( 4 δ ) 1 p ϱ , M 1 ( 1 δ ) , 2 1 ϵ , ( 128 A 2 p 2 π ( ϱ − η ) ) 1 ϵ − η , 3 1 κ − max ( 1 − η , τ ) , 36 p 2 ϱ 2 c 0 2 ) , M_{2}\left(\frac{1}{\delta}\right)=\max\left(\frac{(p+2)^{2}}{2\pi\delta^{2}},\left(\frac{4}{\delta}\right)^{\frac{1}{p\varrho}},M_{1}\left(\frac{1}{\delta}\right),2^{\frac{1}{\epsilon}},\left(\frac{128A^{2}p^{2}}{\pi(\varrho-\eta)}\right)^{\frac{1}{\epsilon-\eta}},3^{\frac{1}{\kappa-\max(1-\eta,\tau)}},\frac{36p^{2}\varrho^{2}}{c_{0}^{2}}\right), M 2 ( δ 1 ) = max ( 2 π δ 2 ( p + 2 ) 2 , ( δ 4 ) pϱ 1 , M 1 ( δ 1 ) , 2 ϵ 1 , ( π ( ϱ − η ) 128 A 2 p 2 ) ϵ − η 1 , 3 κ − m a x ( 1 − η , τ ) 1 , c 0 2 36 p 2 ϱ 2 ) ,
we get Pr ( E c ) ≤ 4 δ \Pr(\mathcal{E}^{c})\leq4\delta Pr ( E c ) ≤ 4 δ . Substituting this into Equation (65) , we obtain:
E [ ∥ f ~ t o p N N − f t o p N T K ∥ L 2 ( X ) 2 ] ≤ C 2 2 m − 1 12 ln m + 4 ( κ N T K ) 2 B 2 n 2 ( p + 1 ) s ( p + 1 ) + p δ . (66) \mathbb{E}\left[\left\|\tilde{f}^{\mathrm{NN}}_{t_{\mathrm{op}}}-f^{\mathrm{NTK}}_{t_{\mathrm{op}}}\right\|_{L^{2}(\mathcal{X})}^{2}\right]\leq C_{2}^{2}m^{-\frac{1}{12}}\ln m+4(\kappa^{\mathrm{NTK}})^{2}B^{2}n^{\frac{2(p+1)}{s(p+1)+p}}\delta. \tag*{(66)} E [ f ~ t op NN − f t op NTK L 2 ( X ) 2 ] ≤ C 2 2 m − 12 1 ln m + 4 ( κ NTK ) 2 B 2 n s ( p + 1 ) + p 2 ( p + 1 ) δ . (66)
Since m ≥ poly ( n , 1 / δ , ln n , ln ( 1 / δ ) ) m\geq\operatorname{poly}(n,1/\delta,\ln n,\ln(1/\delta)) m ≥ poly ( n , 1/ δ , ln n , ln ( 1/ δ )) should be satisfied for E N N \mathcal{E}_{\mathrm{NN}} E NN to hold with high-probability, we require m ≥ M 3 ( 1 / δ ) m\geq M_{3}(1/\delta) m ≥ M 3 ( 1/ δ ) where M 3 ( 1 / δ ) = max ( poly ( n , 1 / δ , ln n , ln ( 1 / δ ) ) , M 2 ( 1 / δ ) ) M_{3}(1/\delta)=\max(\operatorname{poly}(n,1/\delta,\ln n,\ln(1/\delta)),M_{2}(1/\delta)) M 3 ( 1/ δ ) = max ( poly ( n , 1/ δ , ln n , ln ( 1/ δ )) , M 2 ( 1/ δ )) . Redefining poly as the polynomial dominating M 3 ( 1 / δ ) M_{3}(1/\delta) M 3 ( 1/ δ ) , we obtain m ≥ poly ( n , 1 / δ , ln n , ln ( 1 / δ ) ) m\geq\operatorname{poly}(n,1/\delta,\ln n,\ln(1/\delta)) m ≥ poly ( n , 1/ δ , ln n , ln ( 1/ δ )) , which should be satisfied so that the bound on E [ ∥ f ~ t o p N N − f t o p N T K ∥ L 2 ( X ) 2 ] \mathbb{E}\left[\left\|\tilde{f}^{\mathrm{NN}}_{t_{\mathrm{op}}}-f^{\mathrm{NTK}}_{t_{\mathrm{op}}}\right\|_{L^{2}(\mathcal{X})}^{2}\right] E [ f ~ t op NN − f t op NTK L 2 ( X ) 2 ] holds.
Upper-bound on E [ ∥ f t o p N T K − f ⋆ ∥ L 2 ( X ) 2 ] \mathbb{E}\left[\left\|f^{\mathrm{NTK}}_{t_{\mathrm{op}}}-f^\star\right\|_{L^2(\mathcal{X})}^2\right] E [ f t op NTK − f ⋆ L 2 ( X ) 2 ] . By the law of total expectation, we decompose the error over the disjoint events E N T K \mathcal{E}_{\mathrm{NTK}} E NTK and E N T K c \mathcal{E}_{\mathrm{NTK}}^c E NTK c as follows:
E [ ∥ f t o p N T K − f ⋆ ∥ L 2 ( X ) 2 ] = E [ ∥ f t o p N T K − f ⋆ ∥ L 2 ( X ) 2 1 E N T K ] + E [ ∥ f t o p N T K − f ⋆ ∥ L 2 ( X ) 2 1 E N T K c ] ≤ C 1 2 ( ln 6 δ ) 4 n − s ( p + 1 ) s ( p + 1 ) + p + E [ ( ∥ f t o p N T K ∥ L 2 ( X ) + ∥ f ⋆ ∥ ∞ ) 2 1 E N T K c ] ≤ C 1 2 ( ln 6 δ ) 4 n − s ( p + 1 ) s ( p + 1 ) + p + 2 E [ ( ∥ f t o p N T K ∥ L 2 ( X ) 2 + ∥ f ⋆ ∥ ∞ 2 ) 1 E N T K c ] ≤ C 1 2 ( ln 6 δ ) 4 n − s ( p + 1 ) s ( p + 1 ) + p + 2 ( ( κ N T K ) 2 B 2 t o p 2 + B 2 ) Pr ( E N T K c ) ≤ C 1 2 ( ln 6 δ ) 4 n − s ( p + 1 ) s ( p + 1 ) + p + 2 ( ( κ N T K ) 2 B 2 n 2 ( p + 1 ) s ( p + 1 ) + p + B 2 ) δ (67) \begin{aligned}
\mathbb{E}\left[\left\|f^{\mathrm{NTK}}_{t_{\mathrm{op}}}-f^\star\right\|_{L^2(\mathcal{X})}^2\right]
=
\mathbb{E}\left[\left\|f^{\mathrm{NTK}}_{t_{\mathrm{op}}}-f^\star\right\|_{L^2(\mathcal{X})}^2\mathbb{1}_{\mathcal{E}_{\mathrm{NTK}}}\right]
+
\mathbb{E}\left[\left\|f^{\mathrm{NTK}}_{t_{\mathrm{op}}}-f^\star\right\|_{L^2(\mathcal{X})}^2\mathbb{1}_{\mathcal{E}_{\mathrm{NTK}}^c}\right] \\
&\le C_1^2\left(\ln\frac{6}{\delta}\right)^4n^{-\frac{s(p+1)}{s(p+1)+p}}
+
\mathbb{E}\left[\left(\left\|f^{\mathrm{NTK}}_{t_{\mathrm{op}}}\right\|_{L^2(\mathcal{X})}+\left\|f^\star\right\|_\infty\right)^2\mathbb{1}_{\mathcal{E}_{\mathrm{NTK}}^c}\right] \\
&\le C_1^2\left(\ln\frac{6}{\delta}\right)^4n^{-\frac{s(p+1)}{s(p+1)+p}}
+
2\mathbb{E}\left[\left(\left\|f^{\mathrm{NTK}}_{t_{\mathrm{op}}}\right\|_{L^2(\mathcal{X})}^2+\left\|f^\star\right\|_\infty^2\right)\mathbb{1}_{\mathcal{E}_{\mathrm{NTK}}^c}\right] \\
&\le C_1^2\left(\ln\frac{6}{\delta}\right)^4n^{-\frac{s(p+1)}{s(p+1)+p}}
+
2\left((\kappa^{\mathrm{NTK}})^2B^2t_{\mathrm{op}}^2+B^2\right)\Pr\left(\mathcal{E}_{\mathrm{NTK}}^c\right) \\
&\le C_1^2\left(\ln\frac{6}{\delta}\right)^4n^{-\frac{s(p+1)}{s(p+1)+p}}
+
2\left((\kappa^{\mathrm{NTK}})^2B^2n^{\frac{2(p+1)}{s(p+1)+p}}+B^2\right)\delta \tag*{(67)}
\end{aligned} E [ f t op NTK − f ⋆ L 2 ( X ) 2 ] = E [ f t op NTK − f ⋆ L 2 ( X ) 2 1 E NTK ] + E [ f t op NTK − f ⋆ L 2 ( X ) 2 1 E NTK c ] ≤ C 1 2 ( ln δ 6 ) 4 n − s ( p + 1 ) + p s ( p + 1 ) + E [ ( f t op NTK L 2 ( X ) + ∥ f ⋆ ∥ ∞ ) 2 1 E NTK c ] ≤ C 1 2 ( ln δ 6 ) 4 n − s ( p + 1 ) + p s ( p + 1 ) + 2 E [ ( f t op NTK L 2 ( X ) 2 + ∥ f ⋆ ∥ ∞ 2 ) 1 E NTK c ] ≤ C 1 2 ( ln δ 6 ) 4 n − s ( p + 1 ) + p s ( p + 1 ) + 2 ( ( κ NTK ) 2 B 2 t op 2 + B 2 ) Pr ( E NTK c ) ≤ C 1 2 ( ln δ 6 ) 4 n − s ( p + 1 ) + p s ( p + 1 ) + 2 ( ( κ NTK ) 2 B 2 n s ( p + 1 ) + p 2 ( p + 1 ) + B 2 ) δ (67)
where, in the second line, we upper-bound ∥ f t o p N T K − f ⋆ ∥ L 2 ( X ) 2 \left\|f^{\mathrm{NTK}}_{t_{\mathrm{op}}}-f^\star\right\|_{L^2(\mathcal{X})}^2 f t op NTK − f ⋆ L 2 ( X ) 2 by C 1 2 ( ln 6 δ ) 4 n − s ( p + 1 ) s ( p + 1 ) + p C_1^2\left(\ln\frac{6}{\delta}\right)^4n^{-\frac{s(p+1)}{s(p+1)+p}} C 1 2 ( ln δ 6 ) 4 n − s ( p + 1 ) + p s ( p + 1 ) on the event E N T K \mathcal{E}_{\mathrm{NTK}} E NTK and use the triangle inequality together with the identity ∥ f ⋆ ∥ L 2 ( X ) ≤ ∥ f ⋆ ∥ ∞ \left\|f^\star\right\|_{L^2(\mathcal{X})}\le\left\|f^\star\right\|_\infty ∥ f ⋆ ∥ L 2 ( X ) ≤ ∥ f ⋆ ∥ ∞ ; in the third line, we apply the identity ( a + b ) 2 ≤ 2 ( a 2 + b 2 ) (a+b)^2\le2(a^2+b^2) ( a + b ) 2 ≤ 2 ( a 2 + b 2 ) ; in the fourth line, we use Lemmas E.1 and E.15 to upper-bound ∥ f t o p N T K ∥ L 2 ( X ) \left\|f^{\mathrm{NTK}}_{t_{\mathrm{op}}}\right\|_{L^2(\mathcal{X})} f t op NTK L 2 ( X ) by κ N T K B t o p \kappa^{\mathrm{NTK}}B t_{\mathrm{op}} κ NTK B t op with κ N T K : = ( p + 2 ) ( A 2 + 1 ) + 1 \kappa^{\mathrm{NTK}}:=(p+2)(A^2+1)+1 κ NTK := ( p + 2 ) ( A 2 + 1 ) + 1 and ∥ f ⋆ ∥ ∞ \left\|f^\star\right\|_\infty ∥ f ⋆ ∥ ∞ by B B B ; and in the last line, we use t o p = n p + 1 s ( p + 1 ) + p t_{\mathrm{op}}=n^{\frac{p+1}{s(p+1)+p}} t op = n s ( p + 1 ) + p p + 1 .
Expected generalization bound. Plugging Equations (66) and (67) into Equation (64) yields:
E [ ∥ f ~ t o p N N − f ⋆ ∥ L 2 ( X ) 2 ] ≤ 2 ( C 2 2 m − 1 2 ln m + C 1 2 ( ln 6 δ ) 4 n − s ( p + 1 ) s ( p + 1 ) + p + ( 6 ( κ N T K ) 2 B 2 n 2 ( p + 1 ) s ( p + 1 ) + p + 2 B 2 ) δ ) . \mathbb{E}\left[\left\|\widetilde f^{\mathrm{NN}}_{t_{\mathrm{op}}}-f^\star\right\|_{L^2(\mathcal{X})}^2\right]
\le
2\left(C_2^2m^{-\frac{1}{2}}\ln m+C_1^2\left(\ln\frac{6}{\delta}\right)^4n^{-\frac{s(p+1)}{s(p+1)+p}}+\left(6(\kappa^{\mathrm{NTK}})^2B^2n^{\frac{2(p+1)}{s(p+1)+p}}+2B^2\right)\delta\right). E [ f t op NN − f ⋆ L 2 ( X ) 2 ] ≤ 2 ( C 2 2 m − 2 1 ln m + C 1 2 ( ln δ 6 ) 4 n − s ( p + 1 ) + p s ( p + 1 ) + ( 6 ( κ NTK ) 2 B 2 n s ( p + 1 ) + p 2 ( p + 1 ) + 2 B 2 ) δ ) .
Since n ≥ ⌈ ( 1 3 ( κ N T K ) 2 ) s ( p + 1 ) + p 2 ( p + 1 ) ⌉ n\ge\left\lceil\left(\frac{1}{3(\kappa^{\mathrm{NTK}})^2}\right)^{\frac{s(p+1)+p}{2(p+1)}}\right\rceil n ≥ ⌈ ( 3 ( κ NTK ) 2 1 ) 2 ( p + 1 ) s ( p + 1 ) + p ⌉ , we have 3 ( κ N T K ) 2 B 2 n 2 ( p + 1 ) s ( p + 1 ) + p ≥ B 2 3(\kappa^{\mathrm{NTK}})^2B^2n^{\frac{2(p+1)}{s(p+1)+p}}\ge B^2 3 ( κ NTK ) 2 B 2 n s ( p + 1 ) + p 2 ( p + 1 ) ≥ B 2 , therefore:
E [ ∥ f ~ t o p N N − f ⋆ ∥ L 2 ( X ) 2 ] ≤ 2 ( C 2 2 m − 1 2 ln m + φ ( δ ) ) (68) \mathbb{E}\left[\left\|\widetilde f^{\mathrm{NN}}_{t_{\mathrm{op}}}-f^\star\right\|_{L^2(\mathcal{X})}^2\right]
\le2\left(C_2^2m^{-\frac{1}{2}}\ln m+\varphi(\delta)\right) \tag*{(68)} E [ f t op NN − f ⋆ L 2 ( X ) 2 ] ≤ 2 ( C 2 2 m − 2 1 ln m + φ ( δ ) ) (68)
where φ : ( 0 , 1 ) → ( 0 , + ∞ ) \varphi:(0,1)\to(0,+\infty) φ : ( 0 , 1 ) → ( 0 , + ∞ ) is the function defined for any δ ∈ ( 0 , 1 ) \delta\in(0,1) δ ∈ ( 0 , 1 ) by:
φ ( δ ) = C 1 2 ( ln 6 δ ) 4 n − s ( p + 1 ) s ( p + 1 ) + p + 12 ( κ N T K ) 2 B 2 n 2 ( p + 1 ) s ( p + 1 ) + p δ . \varphi(\delta)=C_1^2\left(\ln\frac{6}{\delta}\right)^4n^{-\frac{s(p+1)}{s(p+1)+p}}+12(\kappa^{\mathrm{NTK}})^2B^2n^{\frac{2(p+1)}{s(p+1)+p}}\delta. φ ( δ ) = C 1 2 ( ln δ 6 ) 4 n − s ( p + 1 ) + p s ( p + 1 ) + 12 ( κ NTK ) 2 B 2 n s ( p + 1 ) + p 2 ( p + 1 ) δ .
We minimize φ \varphi φ over δ ∈ ( 0 , 1 ) \delta\in(0,1) δ ∈ ( 0 , 1 ) . By Lemma I.1 , since n ≥ ⌈ ( C 1 2 ( ln 6 ) 3 3 ( κ N T K ) 2 B 2 ) s ( p + 1 ) + p ( s + 2 ) ( p + 1 ) + 1 ⌉ n\ge\left\lceil\left(\frac{C_1^2(\ln6)^3}{3(\kappa^{\mathrm{NTK}})^2B^2}\right)^{\frac{s(p+1)+p}{(s+2)(p+1)}}+1\right\rceil n ≥ ⌈ ( 3 ( κ NTK ) 2 B 2 C 1 2 ( l n 6 ) 3 ) ( s + 2 ) ( p + 1 ) s ( p + 1 ) + p + 1 ⌉ , the minimizer δ ⋆ ∈ ( 0 , 1 ) \delta^\star\in(0,1) δ ⋆ ∈ ( 0 , 1 ) and the corresponding minimum value φ ( δ ⋆ ) \varphi(\delta^\star) φ ( δ ⋆ ) are given by:
δ ⋆ = 9 C 1 2 ( κ N T K ) 2 B 2 n − s + 2 ) ( p + 1 ) s ( p + 1 ) + p W 0 3 ( n 1 3 ( s + 2 ) ( p + 1 ) s ( p + 1 ) + p ( 2 ( κ N T K ) 2 B 2 3 C 1 2 ) 1 3 ) (69) \delta^\star=
\frac{9C_1^2}{(\kappa^{\mathrm{NTK}})^2B^2}
n^{-\frac{s+2)(p+1)}{s(p+1)+p}}
W_0^3\left(n^{\frac{1}{3}\frac{(s+2)(p+1)}{s(p+1)+p}}\left(\frac{2(\kappa^{\mathrm{NTK}})^2B^2}{3C_1^2}\right)^{\frac{1}{3}}\right) \tag*{(69)} δ ⋆ = ( κ NTK ) 2 B 2 9 C 1 2 n − s ( p + 1 ) + p s + 2 ) ( p + 1 ) W 0 3 ( n 3 1 s ( p + 1 ) + p ( s + 2 ) ( p + 1 ) ( 3 C 1 2 2 ( κ NTK ) 2 B 2 ) 3 1 ) (69)
and
φ ( δ ⋆ ) = 27 C 1 2 W 0 3 ( n 1 3 ( s + 2 ) ( p + 1 ) s ( p + 1 ) + p ( 2 ( κ N T K ) 2 B 2 3 C 1 2 ) 1 3 ) × ( 3 W 0 ( n 1 3 ( s + 2 ) ( p + 1 ) s ( p + 1 ) + p ( 2 ( κ N T K ) 2 B 2 3 C 1 2 ) 1 3 ) + 4 ) n − s ( p + 1 ) s ( p + 1 ) + p , \begin{aligned}
\varphi(\delta^\star)=27C_1^2W_0^3\left(n^{\frac{1}{3}\frac{(s+2)(p+1)}{s(p+1)+p}}\left(\frac{2(\kappa^{\mathrm{NTK}})^2B^2}{3C_1^2}\right)^{\frac{1}{3}}\right) \\
&\times\left(3W_0\left(n^{\frac{1}{3}\frac{(s+2)(p+1)}{s(p+1)+p}}\left(\frac{2(\kappa^{\mathrm{NTK}})^2B^2}{3C_1^2}\right)^{\frac{1}{3}}\right)+4\right)
n^{-\frac{s(p+1)}{s(p+1)+p}},
\end{aligned} φ ( δ ⋆ ) = 27 C 1 2 W 0 3 ( n 3 1 s ( p + 1 ) + p ( s + 2 ) ( p + 1 ) ( 3 C 1 2 2 ( κ NTK ) 2 B 2 ) 3 1 ) × ( 3 W 0 ( n 3 1 s ( p + 1 ) + p ( s + 2 ) ( p + 1 ) ( 3 C 1 2 2 ( κ NTK ) 2 B 2 ) 3 1 ) + 4 ) n − s ( p + 1 ) + p s ( p + 1 ) ,
where W 0 W_0 W 0 is the principal branch of the Lambert W W W function. Note that since 1 p + 1 < s ≤ 2 \frac{1}{p+1}<s\leq2 p + 1 1 < s ≤ 2 , one can verify that ( s + 2 ) ( p + 1 ) s ( p + 1 ) + p ≤ 4 \frac{(s+2)(p+1)}{s(p+1)+p}\leq4 s ( p + 1 ) + p ( s + 2 ) ( p + 1 ) ≤ 4 , so that 1 3 ( s + 2 ) ( p + 1 ) s ( p + 1 ) + p ≤ 4 3 \frac{1}{3}\frac{(s+2)(p+1)}{s(p+1)+p}\leq\frac{4}{3} 3 1 s ( p + 1 ) + p ( s + 2 ) ( p + 1 ) ≤ 3 4 . Additionally, since:
n ≥ max ( 3 , ⌈ ( 3 C 1 2 ε 3 2 ( κ N T K ) 2 B 2 ) s ( p + 1 ) + p s ( p + 1 ) + 1 + p ⌉ ) ≥ max ( e , ( 3 C 1 2 ε 3 2 ( κ N T K ) 2 B 2 ) s ( p + 1 ) + p s ( p + 1 ) + 1 ) , n\geq\max\left(3,\left\lceil\left(\frac{3C_1^2\varepsilon^3}{2(\kappa^{\mathrm{NTK}})^2B^2}\right)^{\frac{s(p+1)+p}{s(p+1)+1+p}}\right\rceil\right)\geq\max\left(e,\left(\frac{3C_1^2\varepsilon^3}{2(\kappa^{\mathrm{NTK}})^2B^2}\right)^{\frac{s(p+1)+p}{s(p+1)+1}}\right), n ≥ max 3 , ( 2 ( κ NTK ) 2 B 2 3 C 1 2 ε 3 ) s ( p + 1 ) + 1 + p s ( p + 1 ) + p ≥ max e , ( 2 ( κ NTK ) 2 B 2 3 C 1 2 ε 3 ) s ( p + 1 ) + 1 s ( p + 1 ) + p ,
applying Lemma 1.3 yields:
W 0 3 ( n 1 3 ( s + 2 ) ( p + 1 ) s ( p + 1 ) + p ( 2 ( κ N T K ) 2 B 2 3 C 1 2 ) 1 3 ) ( 3 W 0 ( n 1 3 ( s + 2 ) ( p + 1 ) s ( p + 1 ) + p ( 2 ( κ N T K ) 2 B 2 3 C 1 2 ) 1 3 ) + 4 ) ≤ A 1 ( ( 2 ( κ N T K ) 2 B 2 3 C 1 2 ) 1 3 ) ( ln n ) 4 , \begin{aligned}
W_0^3\left(n^{\frac{1}{3}\frac{(s+2)(p+1)}{s(p+1)+p}}\left(\frac{2(\kappa^{\mathrm{NTK}})^2B^2}{3C_1^2}\right)^{\frac{1}{3}}\right)\left(3W_0\left(n^{\frac{1}{3}\frac{(s+2)(p+1)}{s(p+1)+p}}\left(\frac{2(\kappa^{\mathrm{NTK}})^2B^2}{3C_1^2}\right)^{\frac{1}{3}}\right)+4\right) \\
&\leq A_1\left(\left(\frac{2(\kappa^{\mathrm{NTK}})^2B^2}{3C_1^2}\right)^{\frac{1}{3}}\right)(\ln n)^4,
\end{aligned} W 0 3 ( n 3 1 s ( p + 1 ) + p ( s + 2 ) ( p + 1 ) ( 3 C 1 2 2 ( κ NTK ) 2 B 2 ) 3 1 ) ( 3 W 0 ( n 3 1 s ( p + 1 ) + p ( s + 2 ) ( p + 1 ) ( 3 C 1 2 2 ( κ NTK ) 2 B 2 ) 3 1 ) + 4 ) ≤ A 1 ( ( 3 C 1 2 2 ( κ NTK ) 2 B 2 ) 3 1 ) ( ln n ) 4 ,
where A 1 A_1 A 1 is defined for any u > 0 u>0 u > 0 by:
A 1 ( u ) = 512 27 + 448 9 ∣ ln u ∣ + 48 ( ln u ) 2 + 20 ∣ ln u ∣ 3 + 3 ( ln u ) 4 . A_1(u)=\frac{512}{27}+\frac{448}{9}|\ln u|+48(\ln u)^2+20|\ln u|^3+3(\ln u)^4. A 1 ( u ) = 27 512 + 9 448 ∣ ln u ∣ + 48 ( ln u ) 2 + 20∣ ln u ∣ 3 + 3 ( ln u ) 4 .
Substituting this bound into the expression of φ ( δ ⋆ ) \varphi(\delta^\star) φ ( δ ⋆ ) , we obtain:
φ ( δ ⋆ ) ≤ C ′ ( ln n ) 4 n − s ( p + 1 ) s ( p + 1 ) + p , \varphi(\delta^\star)\leq C'(\ln n)^4n^{-\frac{s(p+1)}{s(p+1)+p}}, φ ( δ ⋆ ) ≤ C ′ ( ln n ) 4 n − s ( p + 1 ) + p s ( p + 1 ) ,
where C ′ = 27 C 1 2 A 1 ( ( 2 ( κ N T K ) 2 B 2 3 C 1 2 ) 1 3 ) C'=27C_1^2A_1\left(\left(\frac{2(\kappa^{\mathrm{NTK}})^2B^2}{3C_1^2}\right)^{\frac{1}{3}}\right) C ′ = 27 C 1 2 A 1 ( ( 3 C 1 2 2 ( κ NTK ) 2 B 2 ) 3 1 ) . Plugging this bound in Equation (68) yields:
E [ ∥ f ~ t o p N N − f ⋆ ∥ L 2 ( X ) 2 ] ≤ 2 ( C 2 2 m − 1 2 ln m + C ′ ( ln n ) 4 n − s ( p + 1 ) s ( p + 1 ) + p ) . \mathbb{E}\left[\left\|\tilde{f}^{\mathrm{NN}}_{\mathrm{top}}-f^\star\right\|_{L^2(\mathcal{X})}^2\right]\leq2\left(C_2^2m^{-\frac{1}{2}}\ln m+C'(\ln n)^4n^{-\frac{s(p+1)}{s(p+1)+p}}\right). E [ f ~ top NN − f ⋆ L 2 ( X ) 2 ] ≤ 2 ( C 2 2 m − 2 1 ln m + C ′ ( ln n ) 4 n − s ( p + 1 ) + p s ( p + 1 ) ) .
Choosing m ≥ max ( poly ( n , 1 / δ , ln n , ln ( 1 / δ ) ) , n 12 ( ln n ) 12 ) m\geq\max\left(\operatorname{poly}(n,1/\delta,\ln n,\ln(1/\delta)),n^{12}(\ln n)^{12}\right) m ≥ max ( poly ( n , 1/ δ , ln n , ln ( 1/ δ )) , n 12 ( ln n ) 12 ) gives m − 1 2 ln m ≤ 24 n − 1 m^{-\frac{1}{2}}\ln m\leq24n^{-1} m − 2 1 ln m ≤ 24 n − 1 by Lemma 1.8. Since s > 0 s>0 s > 0 and p > 0 p>0 p > 0 , we have 0 < s ( p + 1 ) s ( p + 1 ) + p < 1 0<\frac{s(p+1)}{s(p+1)+p}<1 0 < s ( p + 1 ) + p s ( p + 1 ) < 1 , so n − 1 ≤ n − s ( p + 1 ) s ( p + 1 ) + p n^{-1}\leq n^{-\frac{s(p+1)}{s(p+1)+p}} n − 1 ≤ n − s ( p + 1 ) + p s ( p + 1 ) . Moreover, n ≥ 3 > e n\geq3>e n ≥ 3 > e implies 1 ≤ ( ln n ) 4 1\leq(\ln n)^4 1 ≤ ( ln n ) 4 . Combining these observations leads to:
E [ ∥ f ~ t o p N N − f ⋆ ∥ L 2 ( X ) 2 ] ≤ 2 ( 24 C 2 2 + C ′ ) ( ln n ) 4 n − s ( p + 1 ) s ( p + 1 ) + p . \mathbb{E}\left[\left\|\tilde{f}^{\mathrm{NN}}_{\mathrm{top}}-f^\star\right\|_{L^2(\mathcal{X})}^2\right]\leq2\left(24C_2^2+C'\right)(\ln n)^4n^{-\frac{s(p+1)}{s(p+1)+p}}. E [ f ~ top NN − f ⋆ L 2 ( X ) 2 ] ≤ 2 ( 24 C 2 2 + C ′ ) ( ln n ) 4 n − s ( p + 1 ) + p s ( p + 1 ) .
The desired bound follows by setting C = 2 ( 24 C 2 2 + C ′ ) C=2\left(24C_2^2+C'\right) C = 2 ( 24 C 2 2 + C ′ ) .
We now derive an explicit lower-bound on δ ⋆ \delta^\star δ ⋆ to characterize the precise quantities on which m m m depends. Recall from Equation (69) that:
δ ⋆ = 9 C 1 2 ( κ N T K ) 2 B 2 n − ( s + 2 ) ( p + 1 ) s ( p + 1 ) + p W 0 3 ( z ) , z : = n 1 3 ( s + 2 ) ( p + 1 ) s ( p + 1 ) + p ( 2 ( κ N T K ) 2 B 2 3 C 1 2 ) 1 3 . \delta^\star=\frac{9C_1^2}{(\kappa^{\mathrm{NTK}})^2B^2}n^{-\frac{(s+2)(p+1)}{s(p+1)+p}}W_0^3(z),\qquad z:=n^{\frac{1}{3}\frac{(s+2)(p+1)}{s(p+1)+p}}\left(\frac{2(\kappa^{\mathrm{NTK}})^2B^2}{3C_1^2}\right)^{\frac{1}{3}}. δ ⋆ = ( κ NTK ) 2 B 2 9 C 1 2 n − s ( p + 1 ) + p ( s + 2 ) ( p + 1 ) W 0 3 ( z ) , z := n 3 1 s ( p + 1 ) + p ( s + 2 ) ( p + 1 ) ( 3 C 1 2 2 ( κ NTK ) 2 B 2 ) 3 1 .
Since n ≥ ⌈ ( 3 C 1 2 ε 3 2 ( κ N T K ) 2 B 2 ) s ( p + 1 ) + p s ( p + 1 ) + 1 + p ⌉ n\geq\left\lceil\left(\frac{3C_1^2\varepsilon^3}{2(\kappa^{\mathrm{NTK}})^2B^2}\right)^{\frac{s(p+1)+p}{s(p+1)+1+p}}\right\rceil n ≥ ⌈ ( 2 ( κ NTK ) 2 B 2 3 C 1 2 ε 3 ) s ( p + 1 ) + 1 + p s ( p + 1 ) + p ⌉ , we have z ≥ e z\geq e z ≥ e . As W 0 W_0 W 0 is increasing on [ 0 , + ∞ ) [0,+\infty) [ 0 , + ∞ ) with W 0 ( e ) = 1 W_0(e)=1 W 0 ( e ) = 1 , this gives W 0 ( z ) ≥ 1 W_0(z)\geq1 W 0 ( z ) ≥ 1 , hence:
δ ⋆ ≥ 9 C 1 2 ( κ N T K ) 2 B 2 n − ( s + 2 ) ( p + 1 ) s ( p + 1 ) + p . \delta^\star\geq\frac{9C_1^2}{(\kappa^{\mathrm{NTK}})^2B^2}n^{-\frac{(s+2)(p+1)}{s(p+1)+p}}. δ ⋆ ≥ ( κ NTK ) 2 B 2 9 C 1 2 n − s ( p + 1 ) + p ( s + 2 ) ( p + 1 ) .
Using ( s + 2 ) ( p + 1 ) s ( p + 1 ) + p ≤ 4 \frac{(s+2)(p+1)}{s(p+1)+p}\leq4 s ( p + 1 ) + p ( s + 2 ) ( p + 1 ) ≤ 4 , we obtain:
1 δ ⋆ ≤ ( κ N T K ) 2 B 2 9 C 1 2 n ( s + 2 ) ( p + 1 ) s ( p + 1 ) + p ≤ ( κ N T K ) 2 B 2 9 C 1 2 n 4 , \frac{1}{\delta^\star}\leq\frac{(\kappa^{\mathrm{NTK}})^2B^2}{9C_1^2}n^{\frac{(s+2)(p+1)}{s(p+1)+p}}\leq\frac{(\kappa^{\mathrm{NTK}})^2B^2}{9C_1^2}n^4, δ ⋆ 1 ≤ 9 C 1 2 ( κ NTK ) 2 B 2 n s ( p + 1 ) + p ( s + 2 ) ( p + 1 ) ≤ 9 C 1 2 ( κ NTK ) 2 B 2 n 4 ,
and, taking logarithms and using ln n ≥ 1 \ln n\geq1 ln n ≥ 1 for n ≥ 3 n\geq3 n ≥ 3 , we get:
ln 1 δ ⋆ ≤ ∣ ln ( ( κ N T K ) 2 B 2 9 C 1 2 ) ∣ + 4 ln n ≤ ( ∣ ln ( ( κ N T K ) 2 B 2 9 C 1 2 ) ∣ + 4 ) ln n . \ln\frac{1}{\delta^\star}\leq\left|\ln\left(\frac{(\kappa^{\mathrm{NTK}})^2B^2}{9C_1^2}\right)\right|+4\ln n\leq\left(\left|\ln\left(\frac{(\kappa^{\mathrm{NTK}})^2B^2}{9C_1^2}\right)\right|+4\right)\ln n. ln δ ⋆ 1 ≤ ln ( 9 C 1 2 ( κ NTK ) 2 B 2 ) + 4 ln n ≤ ( ln ( 9 C 1 2 ( κ NTK ) 2 B 2 ) + 4 ) ln n .
Both 1 / δ ⋆ 1/\delta^\star 1/ δ ⋆ and ln ( 1 / δ ⋆ ) \ln(1/\delta^\star) ln ( 1/ δ ⋆ ) are therefore dominated by a polynomial in ( n , ln n ) (n,\ln n) ( n , ln n ) whose coefficients depend only on s , p , A , B , M μ s,p,A,B,M_\mu s , p , A , B , M μ and R R R . Redefining poly accordingly, the condition m ≥ poly ( n , 1 / δ ⋆ , ln n , ln ( 1 / δ ⋆ ) ) m\geq\operatorname{poly}(n,1/\delta^\star,\ln n,\ln(1/\delta^\star)) m ≥ poly ( n , 1/ δ ⋆ , ln n , ln ( 1/ δ ⋆ )) is implied by m ≥ poly ( n , ln n ) m\geq\operatorname{poly}(n,\ln n) m ≥ poly ( n , ln n ) , which concludes the proof.
Differentiability of the Objectives and Properties of the Optimal Solutions
The next two Lemmas follow directly from Lemma D.3 and Lemma D.4 in [78 ] , respectively. Here, differentiability is understood in the Fréchet sense. We write ∂ h L o u t ( h ) ∈ L 2 ( X ) \partial_h L_{\mathrm{out}}(h) \in L^2(\mathcal{X}) ∂ h L out ( h ) ∈ L 2 ( X ) and ∂ h L i n ( ω , h ) ∈ L 2 ( X ) \partial_h L_{\mathrm{in}}(\omega,h) \in L^2(\mathcal{X}) ∂ h L in ( ω , h ) ∈ L 2 ( X ) for the partial derivatives with respect to h h h , ∂ h 2 L i n ( ω , h ) : L 2 ( X ) → L 2 ( X ) \partial_h^2 L_{\mathrm{in}}(\omega,h):L^2(\mathcal{X}) \to L^2(\mathcal{X}) ∂ h 2 L in ( ω , h ) : L 2 ( X ) → L 2 ( X ) and ∂ ω , h 2 L i n ( ω , h ) : L 2 ( X ) → R \partial_{\omega,h}^2 L_{\mathrm{in}}(\omega,h):L^2(\mathcal{X}) \to\mathbb{R} ∂ ω , h 2 L in ( ω , h ) : L 2 ( X ) → R for the second-order partial derivatives of L i n L_{\mathrm{in}} L in with respect to h h h and to ( ω , h ) (\omega,h) ( ω , h ) .
Lemma F.1 (Differentiability of L o u t L_{\mathrm{out}} L out ). Under Assumption B B B , L o u t L_{\mathrm{out}} L out is differentiable on L 2 ( X ) L^2(\mathcal{X}) L 2 ( X ) , with partial derivative given, for any h ∈ L 2 ( X ) h \in L^2(\mathcal{X}) h ∈ L 2 ( X ) , by:
∂ h L o u t ( h ) ∈ L 2 ( X ) : x ↦ h ( x ) − E P [ y ∣ x ] . \partial_h L_{\mathrm{out}}(h) \in L^2(\mathcal{X}):x \mapsto h(x)-\mathbb{E}_{\mathbb{P}}[y\mid x]. ∂ h L out ( h ) ∈ L 2 ( X ) : x ↦ h ( x ) − E P [ y ∣ x ] .
Lemma F.2 (Differentiability of ∂ h L i n \partial_h L_{\mathrm{in}} ∂ h L in ). Under Assumption A A A , the function ( ω , h ) ↦ ∂ h L i n ( ω , h ) (\omega,h)\mapsto\partial_h L_{\mathrm{in}}(\omega,h) ( ω , h ) ↦ ∂ h L in ( ω , h ) is differentiable on U × L 2 ( X ) U\times L^2(\mathcal{X}) U × L 2 ( X ) , with partial derivatives given, for any ω ∈ U \omega\in U ω ∈ U and h ∈ L 2 ( X ) h\in L^2(\mathcal{X}) h ∈ L 2 ( X ) , by:
∂ h 2 L i n ( ω , h ) = Id L 2 ( X ) , ∂ ω , h 2 L i n ( ω , h ) f = − E P [ ∂ ω g ω ( z ) f ( x ) ] , for any f ∈ L 2 ( X ) . \partial_h^2 L_{\mathrm{in}}(\omega,h)=\operatorname{Id}_{L^2(\mathcal{X})},\qquad\partial_{\omega,h}^2 L_{\mathrm{in}}(\omega,h)f=-\mathbb{E}_{\mathbb{P}}[\partial_\omega g_\omega(z)f(x)],\quad\text{for any }f\in L^2(\mathcal{X}). ∂ h 2 L in ( ω , h ) = Id L 2 ( X ) , ∂ ω , h 2 L in ( ω , h ) f = − E P [ ∂ ω g ω ( z ) f ( x )] , for any f ∈ L 2 ( X ) .
Lemma F.3 (Uniform boundedness of target functions). Under Assumptions A A A and B B B , we have:
h ω ⋆ ∈ L ∞ ( X ) h_\omega^\star\in L^\infty(\mathcal{X}) h ω ⋆ ∈ L ∞ ( X ) for any ω ∈ U \omega\in U ω ∈ U , and sup ω ∈ U ∥ h ω ⋆ ∥ ∞ ≤ D \sup_{\omega\in U}\lVert h_\omega^\star\rVert_\infty\le D sup ω ∈ U ∥ h ω ⋆ ∥ ∞ ≤ D ;
b ⋆ ∈ L ∞ ( X ) b^\star\in L^\infty(\mathcal{X}) b ⋆ ∈ L ∞ ( X ) and ∥ b ⋆ ∥ ∞ ≤ B \lVert b^\star\rVert_\infty\le B ∥ b ⋆ ∥ ∞ ≤ B ;
a ω ⋆ ∈ L ∞ ( X ) a_\omega^\star\in L^\infty(\mathcal{X}) a ω ⋆ ∈ L ∞ ( X ) for any ω ∈ U \omega\in U ω ∈ U , and sup ω ∈ U ∥ a ω ⋆ ∥ ∞ ≤ D + B \sup_{\omega\in U}\lVert a_\omega^\star\rVert_\infty\le D+B sup ω ∈ U ∥ a ω ⋆ ∥ ∞ ≤ D + B .
Proof. Uniform boundedness of h ω ⋆ h_\omega^\star h ω ⋆ . Let ω ∈ U \omega\in U ω ∈ U . By Assumption A A A , ∣ g ω ( z ) ∣ ≤ D \lvert g_\omega(z)\rvert\le D ∣ g ω ( z )∣ ≤ D for any z ∈ Z z\in\mathcal{Z} z ∈ Z , so that, by Jensen’s inequality, for P X \mathbb{P}_{\mathcal{X}} P X -almost every x ∈ X x\in\mathcal{X} x ∈ X , ∣ h ω ⋆ ( x ) ∣ = ∣ E P [ g ω ( z ) ∣ x ] ∣ ≤ E P [ ∣ g ω ( z ) ∣ ∣ x ] ≤ D \lvert h_\omega^\star(x)\rvert=\lvert\mathbb{E}_{\mathbb{P}}[g_\omega(z)\mid x]\rvert\le\mathbb{E}_{\mathbb{P}}[\lvert g_\omega(z)\rvert\mid x]\le D ∣ h ω ⋆ ( x )∣ = ∣ E P [ g ω ( z ) ∣ x ]∣ ≤ E P [∣ g ω ( z )∣ ∣ x ] ≤ D . Taking the essential supremum over x x x and then the supremum over ω ∈ U \omega\in U ω ∈ U gives sup ω ∈ U ∥ h ω ⋆ ∥ ∞ ≤ D \sup_{\omega\in U}\lVert h_\omega^\star\rVert_\infty\le D sup ω ∈ U ∥ h ω ⋆ ∥ ∞ ≤ D .
Boundedness of b ⋆ b^\star b ⋆ . Similarly, by Assumption B B B , ∣ y ∣ ≤ B \lvert y\rvert\le B ∣ y ∣ ≤ B , so that ∣ b ⋆ ( x ) ∣ = ∣ E P [ y ∣ x ] ∣ ≤ E P [ ∣ y ∣ ∣ x ] ≤ B \lvert b^\star(x)\rvert=\lvert\mathbb{E}_{\mathbb{P}}[y\mid x]\rvert\le\mathbb{E}_{\mathbb{P}}[\lvert y\rvert\mid x]\le B ∣ b ⋆ ( x )∣ = ∣ E P [ y ∣ x ]∣ ≤ E P [∣ y ∣ ∣ x ] ≤ B for P X \mathbb{P}_{\mathcal{X}} P X -almost every x ∈ X x\in\mathcal{X} x ∈ X , hence ∥ b ⋆ ∥ ∞ ≤ B \lVert b^\star\rVert_\infty\le B ∥ b ⋆ ∥ ∞ ≤ B .
Uniform boundedness of a ω ⋆ a_\omega^\star a ω ⋆ . Let ω ∈ U \omega\in U ω ∈ U . By the triangle inequality, ∥ a ω ⋆ ∥ ∞ = ∥ − h ω ⋆ + b ⋆ ∥ ∞ ≤ ∥ h ω ⋆ ∥ ∞ + ∥ b ⋆ ∥ ∞ \lVert a_\omega^\star\rVert_\infty=\lVert-h_\omega^\star+b^\star\rVert_\infty\le\lVert h_\omega^\star\rVert_\infty+\lVert b^\star\rVert_\infty ∥ a ω ⋆ ∥ ∞ = ∥ − h ω ⋆ + b ⋆ ∥ ∞ ≤ ∥ h ω ⋆ ∥ ∞ + ∥ b ⋆ ∥ ∞ . Taking the supremum over ω ∈ U \omega\in U ω ∈ U together with the two previous bounds gives sup ω ∈ U ∥ a ω ⋆ ∥ ∞ ≤ D + B \sup_{\omega\in U}\lVert a_\omega^\star\rVert_\infty\le D+B sup ω ∈ U ∥ a ω ⋆ ∥ ∞ ≤ D + B .
Proposition F.4 (Lipschitz continuity of ω ↦ h ω ⋆ \omega\mapsto h_\omega^\star ω ↦ h ω ⋆ ). Under Assumption A A A , the function ω ↦ h ω ⋆ \omega\mapsto h_\omega^\star ω ↦ h ω ⋆ is G G G -Lipschitz continuous on U U U .
Proof. Let ω 1 , ω 2 ∈ U \omega_1,\omega_2\in U ω 1 , ω 2 ∈ U . Since U U U is convex, the segment ω s : = ω 2 + s ( ω 1 − ω 2 ) \omega_s:=\omega_2+s(\omega_1-\omega_2) ω s := ω 2 + s ( ω 1 − ω 2 ) , s ∈ [ 0 , 1 ] s\in[0,1] s ∈ [ 0 , 1 ] , lies in U U U , so that applying the Cauchy-Schwarz inequality and Assumption A A A gives for any z ∈ Z z\in\mathcal{Z} z ∈ Z :
∣ g ω 1 ( z ) − g ω 2 ( z ) ∣ ≤ ∣ ∫ 0 1 ⟨ ∂ ω g ω s ( z ) , ω 1 − ω 2 ⟩ d s ∣ ≤ G ∥ ω 1 − ω 2 ∥ . \lvert g_{\omega_1}(z)-g_{\omega_2}(z)\rvert\le\left\lvert\int_0^1\langle\partial_\omega g_{\omega_s}(z),\omega_1-\omega_2\rangle\,\mathrm{d}s\right\rvert\le G\lVert\omega_1-\omega_2\rVert. ∣ g ω 1 ( z ) − g ω 2 ( z )∣ ≤ ∫ 0 1 ⟨ ∂ ω g ω s ( z ) , ω 1 − ω 2 ⟩ d s ≤ G ∥ ω 1 − ω 2 ∥ . Hence, by Jensen’s inequality, for P X \mathbb{P}_{\mathcal{X}} P X -almost every x ∈ X x\in\mathcal{X} x ∈ X , we have:
∣ h ω 1 ⋆ ( x ) − h ω 2 ⋆ ( x ) ∣ = ∣ E P [ g ω 1 ( z ) − g ω 2 ( z ) ∣ x ] ∣ ≤ E P [ ∣ g ω 1 ( z ) − g ω 2 ( z ) ∣ ∣ x ] ≤ G ∥ ω 1 − ω 2 ∥ . \lvert h_{\omega_1}^\star(x)-h_{\omega_2}^\star(x)\rvert=\lvert\mathbb{E}_{\mathbb{P}}[g_{\omega_1}(z)-g_{\omega_2}(z)\mid x]\rvert\le\mathbb{E}_{\mathbb{P}}[\lvert g_{\omega_1}(z)-g_{\omega_2}(z)\rvert\mid x]\le G\lVert\omega_1-\omega_2\rVert. ∣ h ω 1 ⋆ ( x ) − h ω 2 ⋆ ( x )∣ = ∣ E P [ g ω 1 ( z ) − g ω 2 ( z ) ∣ x ]∣ ≤ E P [∣ g ω 1 ( z ) − g ω 2 ( z )∣ ∣ x ] ≤ G ∥ ω 1 − ω 2 ∥ . Since P X \mathbb{P}_{\mathcal{X}} P X is a probability measure, ∥ ⋅ ∥ L 2 ( X ) ≤ ∥ ⋅ ∥ ∞ \lVert\cdot\rVert_{L^2(\mathcal{X})}\le\lVert\cdot\rVert_\infty ∥ ⋅ ∥ L 2 ( X ) ≤ ∥ ⋅ ∥ ∞ , and therefore:
∥ h ω 1 ⋆ − h ω 2 ⋆ ∥ L 2 ( X ) ≤ G ∥ ω 1 − ω 2 ∥ . \lVert h_{\omega_1}^\star-h_{\omega_2}^\star\rVert_{L^2(\mathcal{X})}\le G\lVert\omega_1-\omega_2\rVert. ∥ h ω 1 ⋆ − h ω 2 ⋆ ∥ L 2 ( X ) ≤ G ∥ ω 1 − ω 2 ∥ . which concludes the proof.
A Sobolev Space Characterization of the Infinite-Width NTK’s RKHS
In this section, our goal is to show that the RKHS H N T K \mathcal{H}^{\mathrm{NTK}} H NTK of the infinite-width NTK K N T K K^{\mathrm{NTK}} K NTK of the two-layer ReLU network of the form in Equation (31) , and its corresponding interpolation spaces, are Sobolev spaces when the input space X \mathcal{X} X is a bounded Lipschitz domain (Definition G.1).
Preliminaries
For a positive semi-definite kernel k k k on a set E E E , we write H ( k ) \mathcal{H}(k) H ( k ) for its RKHS. Standard facts about H ( k ) \mathcal{H}(k) H ( k ) are used repeatedly: (restriction ) for E 0 ⊂ E E_0 \subset E E 0 ⊂ E , H ( k ∣ E 0 ) = { f ∣ E 0 : f ∈ H ( k ) } \mathcal{H}(k|_{E_0}) = \{f|_{E_0}: f \in\mathcal{H}(k)\} H ( k ∣ E 0 ) = { f ∣ E 0 : f ∈ H ( k )} with ∥ g ∥ H ( k ∣ E 0 ) = min { ∥ f ∥ H ( k ) : f ∣ E 0 = g } \|g\|_{\mathcal{H}(k|_{E_0})} = \min\{\|f\|_{\mathcal{H}(k)}: f|_{E_0} = g\} ∥ g ∥ H ( k ∣ E 0 ) = min { ∥ f ∥ H ( k ) : f ∣ E 0 = g } ; (sums ) H ( k 1 + k 2 ) = H ( k 1 ) + H ( k 2 ) \mathcal{H}(k_1 + k_2) = \mathcal{H}(k_1) + \mathcal{H}(k_2) H ( k 1 + k 2 ) = H ( k 1 ) + H ( k 2 ) with ∥ f ∥ H ( k 1 + k 2 ) 2 = min { ∥ f 1 ∥ H ( k 1 ) 2 + ∥ f 2 ∥ H ( k 2 ) 2 : f = f 1 + f 2 } \|f\|^2_{\mathcal{H}(k_1+k_2)} = \min\{\|f_1\|^2_{\mathcal{H}(k_1)} + \|f_2\|^2_{\mathcal{H}(k_2)}: f = f_1 + f_2\} ∥ f ∥ H ( k 1 + k 2 ) 2 = min { ∥ f 1 ∥ H ( k 1 ) 2 + ∥ f 2 ∥ H ( k 2 ) 2 : f = f 1 + f 2 } ; and (scaling ) for c > 0 c > 0 c > 0 , H ( c k ) = H ( k ) \mathcal{H}(ck) = \mathcal{H}(k) H ( c k ) = H ( k ) as sets, with ∥ f ∥ H ( c k ) = c − 1 / 2 ∥ f ∥ H ( k ) \|f\|_{\mathcal{H}(ck)} = c^{-1/2}\|f\|_{\mathcal{H}(k)} ∥ f ∥ H ( c k ) = c − 1/2 ∥ f ∥ H ( k ) [8 ] . For a subset F F F of a normed space H H H , we say F F F embeds continuously into a normed space H ′ H' H ′ if F ⊂ H ′ F \subset H' F ⊂ H ′ and there is a constant C > 0 C > 0 C > 0 such that ∥ f ∥ H ′ ≤ C ∥ f ∥ H \|f\|_{H'} \le C\|f\|_H ∥ f ∥ H ′ ≤ C ∥ f ∥ H for every f ∈ F f \in F f ∈ F ; we write this F ↪ H ′ F \hookrightarrow H' F ↪ H ′ . For a function f f f on a topological space E E E , the support of f f f , written supp f \operatorname{supp} f supp f , is the closure of { x ∈ E : f ( x ) ≠ 0 } \{x \in E : f(x) \ne0\} { x ∈ E : f ( x ) = 0 } . f f f is said to be compactly supported in an open set U ⊂ E U \subset E U ⊂ E if supp f \operatorname{supp} f supp f is a compact subset of U U U . Let ϕ : R p → S p ⊂ R p + 1 \phi: \mathbb{R}^{p} \to\mathbb{S}^{p} \subset\mathbb{R}^{p+1} ϕ : R p → S p ⊂ R p + 1 be the map defined for any x ∈ R p x \in\mathbb{R}^{p} x ∈ R p by ϕ ( x ) = x ~ / ∥ x ~ ∥ \phi(x) = \tilde{x}/\|\tilde{x}\| ϕ ( x ) = x ~ /∥ x ~ ∥ with x ~ = [ x T , 1 ] T \tilde{x} = [x^{\mathsf{T}},1]^{\mathsf{T}} x ~ = [ x T , 1 ] T . We write S + p : = { y ∈ S p : y p + 1 > 0 } \mathbb{S}^{p}_{+} := \{y \in\mathbb{S}^{p}: y_{p+1} > 0\} S + p := { y ∈ S p : y p + 1 > 0 } and S − p : = { y ∈ S p : y p + 1 < 0 } \mathbb{S}^{p}_{-} := \{y \in\mathbb{S}^{p}: y_{p+1} < 0\} S − p := { y ∈ S p : y p + 1 < 0 } for the open upper and lower hemispheres of S p \mathbb{S}^{p} S p . For any u ∈ [ − 1 , 1 ] u \in[-1,1] u ∈ [ − 1 , 1 ] , we define κ ( u ) : = u κ 0 ( u ) + κ 1 ( u ) \kappa(u) := u\kappa_0(u) + \kappa_1(u) κ ( u ) := u κ 0 ( u ) + κ 1 ( u ) where κ 0 ( u ) : = 1 π ( π − arccos ( u ) ) \kappa_0(u) := \frac{1}{\pi}(\pi- \arccos(u)) κ 0 ( u ) := π 1 ( π − arccos ( u )) and κ 1 ( u ) : = 1 π ( u ( π − arccos ( u ) ) + 1 − u 2 ) \kappa_1(u) := \frac{1}{\pi}(u(\pi- \arccos(u)) + \sqrt{1-u^2}) κ 1 ( u ) := π 1 ( u ( π − arccos ( u )) + 1 − u 2 ) . Using Equation (38) :
K N T K ( x , x ′ ) = 1 2 ∥ x ~ ∥ ∥ x ~ ′ ∥ κ ( ⟨ ϕ ( x ) , ϕ ( x ′ ) ⟩ ) + 1 , K^{\mathrm{NTK}}(x,x') = \frac{1}{2}\|\tilde{x}\|\|\tilde{x}'\|\kappa(\langle\phi(x),\phi(x')\rangle)+1, K NTK ( x , x ′ ) = 2 1 ∥ x ~ ∥∥ x ~ ′ ∥ κ (⟨ ϕ ( x ) , ϕ ( x ′ )⟩) + 1 ,
Setting K 0 ( y , y ′ ) : = κ ( ⟨ y , y ′ ⟩ ) K_0(y,y') := \kappa(\langle y,y'\rangle) K 0 ( y , y ′ ) := κ (⟨ y , y ′ ⟩) for y , y ′ ∈ S p y,y' \in\mathbb{S}^{p} y , y ′ ∈ S p , and using the notation M g [ k ] ( x , x ′ ) : = g ( x ) k ( x , x ′ ) g ( x ′ ) M_g[k](x,x') := g(x)k(x,x')g(x') M g [ k ] ( x , x ′ ) := g ( x ) k ( x , x ′ ) g ( x ′ ) for a function g g g and kernel k k k and ϕ ∗ K 0 ( x , x ′ ) : = K 0 ( ϕ ( x ) , ϕ ( x ′ ) ) \phi^*K_0(x,x') := K_0(\phi(x),\phi(x')) ϕ ∗ K 0 ( x , x ′ ) := K 0 ( ϕ ( x ) , ϕ ( x ′ )) for x , x ′ ∈ X x,x' \in\mathcal{X} x , x ′ ∈ X :
K N T K = 1 2 M q [ ϕ ∗ K 0 ] + I X , where q ( x ) : = ∥ x ~ ∥ and I X ( x , x ′ ) = 1 for x , x ′ ∈ X . (70) K^{\mathrm{NTK}} = \frac{1}{2}M_q[\phi^*K_0] + I_{\mathcal{X}}, \qquad\text{where }q(x) := \|\tilde{x}\|\text{ and }I_{\mathcal{X}}(x,x')=1\text{ for }x,x'\in\mathcal{X}. \tag*{(70)} K NTK = 2 1 M q [ ϕ ∗ K 0 ] + I X , where q ( x ) := ∥ x ~ ∥ and I X ( x , x ′ ) = 1 for x , x ′ ∈ X . (70)
In this section, X \mathcal{X} X is a bounded Lipschitz domain , in the sense of Definition G.1 below. We write L 2 ( X , d x ) L^2(\mathcal{X},\mathrm{d}x) L 2 ( X , d x ) for the space of square-integrable functions on X \mathcal{X} X with respect to the Lebesgue measure d x \mathrm{d}x d x , and recall L 2 ( X ) : = L 2 ( X , μ ) L^2(\mathcal{X}) := L^2(\mathcal{X},\mu) L 2 ( X ) := L 2 ( X , μ ) . Let L l o c 1 ( X , d x ) L^1_{\mathrm{loc}}(\mathcal{X},\mathrm{d}x) L loc 1 ( X , d x ) denote the space of locally integrable functions on X \mathcal{X} X , i.e. , functions f : X → R f:\mathcal{X}\to\mathbb{R} f : X → R such that ∫ K ∣ f ( x ) ∣ d x < ∞ \int_K |f(x)|\mathrm{d}x < \infty ∫ K ∣ f ( x ) ∣ d x < ∞ for every compact K ⊂ X K \subset\mathcal{X} K ⊂ X , and let C c ∞ ( X ) C^\infty_c(\mathcal{X}) C c ∞ ( X ) denote the space of smooth, compactly supported functions on X \mathcal{X} X . Write N 0 : = { 0 , 1 , 2 , … } \mathbb{N}_0 := \{0,1,2,\ldots\} N 0 := { 0 , 1 , 2 , … } for the nonnegative integers.
Definition G.1 (Bounded Lipschitz domain [44 Definition 1.2.1.1] ). X \mathcal{X} X is bounded Lipschitz if it is open, bounded, and nonempty, and its boundary ∂ X \partial\mathcal{X} ∂ X is locally the graph of a Lipschitz function: for every x 0 ∈ ∂ X x_0 \in\partial\mathcal{X} x 0 ∈ ∂ X , there exist r > 0 r > 0 r > 0 , an orthonormal coordinate system ( z 1 , … , z p − 1 , z p ) (z_1,\ldots,z_{p-1},z_p) ( z 1 , … , z p − 1 , z p ) centered at x 0 x_0 x 0 , and a Lipschitz function γ : R p − 1 → R \gamma:\mathbb{R}^{p-1}\to\mathbb{R} γ : R p − 1 → R such that:
X ∩ B ( x 0 , r ) = { ( z 1 , … , z p ) ∈ B ( x 0 , r ) : z p > γ ( z 1 , … , z p − 1 ) } , \mathcal{X}\cap B(x_0,r)=\{(z_1,\ldots,z_p)\in B(x_0,r):z_p>\gamma(z_1,\ldots,z_{p-1})\}, X ∩ B ( x 0 , r ) = {( z 1 , … , z p ) ∈ B ( x 0 , r ) : z p > γ ( z 1 , … , z p − 1 )} , and
∂ X ∩ B ( x 0 , r ) = { ( z 1 , … , z p ) ∈ B ( x 0 , r ) : z p = γ ( z 1 , … , z p − 1 ) } , \partial\mathcal{X}\cap B(x_0,r)=\{(z_1,\ldots,z_p)\in B(x_0,r):z_p=\gamma(z_1,\ldots,z_{p-1})\}, ∂ X ∩ B ( x 0 , r ) = {( z 1 , … , z p ) ∈ B ( x 0 , r ) : z p = γ ( z 1 , … , z p − 1 )} , where B ( x 0 , r ) B(x_0,r) B ( x 0 , r ) is the open ball centered at x 0 x_0 x 0 of radius r r r .
Definition G.2 (Weak derivative). For f ∈ L l o c 1 ( X , d x ) f \in L^1_{\mathrm{loc}}(\mathcal{X},\mathrm{d}x) f ∈ L loc 1 ( X , d x ) and a multi-index α = ( α 1 , … , α p ) ∈ N 0 p \alpha=(\alpha_1,\ldots,\alpha_p)\in\mathbb{N}_0^p α = ( α 1 , … , α p ) ∈ N 0 p with ∣ α ∣ = α 1 + ⋯ + α p |\alpha|=\alpha_1+\cdots+\alpha_p ∣ α ∣ = α 1 + ⋯ + α p , v ∈ L l o c 1 ( X , d x ) v\in L^1_{\mathrm{loc}}(\mathcal{X},\mathrm{d}x) v ∈ L loc 1 ( X , d x ) is the weak α \alpha α -th derivative of f f f , written v = D α f v=D^\alpha f v = D α f , if
∫ X f ( x ) D α ϕ ( x ) d x = ( − 1 ) ∣ α ∣ ∫ X v ( x ) ϕ ( x ) d x \int_{\mathcal{X}} f(x)D^\alpha\phi(x)\mathrm{d}x=(-1)^{|\alpha|}\int_{\mathcal{X}}v(x)\phi(x)\mathrm{d}x ∫ X f ( x ) D α ϕ ( x ) d x = ( − 1 ) ∣ α ∣ ∫ X v ( x ) ϕ ( x ) d x for all ϕ ∈ C c ∞ ( X ) \phi\in C^\infty_c(\mathcal{X}) ϕ ∈ C c ∞ ( X ) .
The following are the definitions of Sobolev spaces used throughout this section, following the standard treatment in, e.g. , [3 Section 5.1] .
Definition G.3 (Sobolev space, integer order). For m ∈ N 0 m\in\mathbb{N}_0 m ∈ N 0 , the Sobolev space of integer order m m m is:
H m ( X ) : = { f ∈ L 2 ( X , d x ) : D α f ∈ L 2 ( X , d x ) for every α with ∣ α ∣ ≤ m } , H^m(\mathcal{X}) := \{f\in L^2(\mathcal{X},\mathrm{d}x):D^\alpha f\in L^2(\mathcal{X},\mathrm{d}x)\text{ for every }\alpha\text{ with }|\alpha|\le m\}, H m ( X ) := { f ∈ L 2 ( X , d x ) : D α f ∈ L 2 ( X , d x ) for every α with ∣ α ∣ ≤ m } , with ∥ f ∥ H m ( X ) 2 : = ∑ ∣ α ∣ ≤ m ∥ D α f ∥ L 2 ( X , d x ) 2 \|f\|^2_{H^m(\mathcal{X})} := \sum_{|\alpha|\le m}\|D^\alpha f\|^2_{L^2(\mathcal{X},\mathrm{d}x)} ∥ f ∥ H m ( X ) 2 := ∑ ∣ α ∣ ≤ m ∥ D α f ∥ L 2 ( X , d x ) 2 .
Definition G.4 (Sobolev space, fractional order). For τ = m + ϑ \tau=m+\vartheta τ = m + ϑ with m ∈ N 0 m\in\mathbb{N}_0 m ∈ N 0 and ϑ ∈ ( 0 , 1 ) \vartheta\in(0,1) ϑ ∈ ( 0 , 1 ) , define the Gagliardo (Sobolev–Slobodeckij) seminorm as
[ f ] τ , X 2 : = ∑ ∣ α ∣ = m ∬ X × X ∣ D α f ( x ) − D α f ( y ) ∣ 2 ∥ x − y ∥ p + 2 ϑ d x d y . [f]^2_{\tau,\mathcal{X}} := \sum_{|\alpha|=m}\iint_{\mathcal{X}\times\mathcal{X}}\frac{|D^\alpha f(x)-D^\alpha f(y)|^2}{\|x-y\|^{p+2\vartheta}}\mathrm{d}x\mathrm{d}y. [ f ] τ , X 2 := ∣ α ∣ = m ∑ ∬ X × X ∥ x − y ∥ p + 2 ϑ ∣ D α f ( x ) − D α f ( y ) ∣ 2 d x d y . The Sobolev space of fractional order τ \tau τ is H τ ( X ) : = { f ∈ H m ( X ) : [ f ] τ , X < ∞ } H^\tau(\mathcal{X}) := \{f\in H^m(\mathcal{X}):[f]_{\tau,\mathcal{X}}<\infty\} H τ ( X ) := { f ∈ H m ( X ) : [ f ] τ , X < ∞ } , with ∥ f ∥ H τ ( X ) 2 : = ∥ f ∥ H m ( X ) 2 + [ f ] τ , X 2 \|f\|^2_{H^\tau(\mathcal{X})}:=\|f\|^2_{H^m(\mathcal{X})}+[f]^2_{\tau,\mathcal{X}} ∥ f ∥ H τ ( X ) 2 := ∥ f ∥ H m ( X ) 2 + [ f ] τ , X 2 . For τ ∈ N 0 \tau\in\mathbb{N}_0 τ ∈ N 0 , i.e. , ϑ = 0 \vartheta=0 ϑ = 0 , H τ ( X ) H^\tau(\mathcal{X}) H τ ( X ) is understood as in Definition G.3.
Corollary G.5 (Domination absorbs a summand). Let k 1 , k 2 k_1, k_2 k 1 , k 2 be positive semi-definite kernels on the same set. If H ( k 1 ) ↪ H ( k 2 ) \mathcal{H}(k_1) \hookrightarrow\mathcal{H}(k_2) H ( k 1 ) ↪ H ( k 2 ) continuously, then H ( k 1 + k 2 ) ≅ H ( k 2 ) \mathcal{H}(k_1 + k_2) \cong\mathcal{H}(k_2) H ( k 1 + k 2 ) ≅ H ( k 2 ) .
Proof. By the sum property of RKHSs, H ( k 1 + k 2 ) = H ( k 1 ) + H ( k 2 ) \mathcal{H}(k_1 + k_2) = \mathcal{H}(k_1) + \mathcal{H}(k_2) H ( k 1 + k 2 ) = H ( k 1 ) + H ( k 2 ) . Since H ( k 1 ) ⊂ H ( k 2 ) \mathcal{H}(k_1) \subset\mathcal{H}(k_2) H ( k 1 ) ⊂ H ( k 2 ) , this equals H ( k 2 ) \mathcal{H}(k_2) H ( k 2 ) as a set. Let C C C be the embedding constant, then ∥ f 1 ∥ H ( k 2 ) ≤ C ∥ f 1 ∥ H ( k 1 ) \lVert f_1\rVert_{\mathcal{H}(k_2)} \le C\lVert f_1\rVert_{\mathcal{H}(k_1)} ∥ f 1 ∥ H ( k 2 ) ≤ C ∥ f 1 ∥ H ( k 1 ) . For f = f 1 + f 2 ∈ H ( k 1 ) + H ( k 2 ) f = f_1 + f_2 \in\mathcal{H}(k_1) + \mathcal{H}(k_2) f = f 1 + f 2 ∈ H ( k 1 ) + H ( k 2 ) , we have:
∥ f ∥ H ( k 2 ) ≤ ∥ f 1 ∥ H ( k 2 ) + ∥ f 2 ∥ H ( k 2 ) ≤ C ∥ f 1 ∥ H ( k 1 ) + ∥ f 2 ∥ H ( k 2 ) ≤ ( C + 1 ) ∥ f 1 ∥ H ( k 1 ) 2 + ∥ f 2 ∥ H ( k 2 ) 2 . \lVert f\rVert_{\mathcal{H}(k_2)} \le\lVert f_1\rVert_{\mathcal{H}(k_2)}+\lVert f_2\rVert_{\mathcal{H}(k_2)} \le C\lVert f_1\rVert_{\mathcal{H}(k_1)}+\lVert f_2\rVert_{\mathcal{H}(k_2)} \le(C+1)\sqrt{\lVert f_1\rVert_{\mathcal{H}(k_1)}^2+\lVert f_2\rVert_{\mathcal{H}(k_2)}^2}. ∥ f ∥ H ( k 2 ) ≤ ∥ f 1 ∥ H ( k 2 ) + ∥ f 2 ∥ H ( k 2 ) ≤ C ∥ f 1 ∥ H ( k 1 ) + ∥ f 2 ∥ H ( k 2 ) ≤ ( C + 1 ) ∥ f 1 ∥ H ( k 1 ) 2 + ∥ f 2 ∥ H ( k 2 ) 2 .
Taking the infimum over decompositions gives ∥ f ∥ H ( k 2 ) ≤ ( C + 1 ) ∥ f ∥ H ( k 1 + k 2 ) \lVert f\rVert_{\mathcal{H}(k_2)} \le(C + 1)\lVert f\rVert_{\mathcal{H}(k_1+k_2)} ∥ f ∥ H ( k 2 ) ≤ ( C + 1 ) ∥ f ∥ H ( k 1 + k 2 ) , and the trivial decomposition f 1 = 0 f_1 = 0 f 1 = 0 gives the reverse.
Lemma G.6. Let k k k be a positive semi-definite kernel on a set E 2 E_2 E 2 , let φ 0 : E 1 → E 2 \varphi_0 : E_1 \to E_2 φ 0 : E 1 → E 2 be a surjection, and let g : E 1 → R g : E_1 \to\mathbb{R} g : E 1 → R be nowhere zero. Then the map f ↦ g ⋅ ( f ∘ φ 0 ) f \mapsto g \cdot(f \circ\varphi_0) f ↦ g ⋅ ( f ∘ φ 0 ) is an isometric isomorphism from H ( k ) \mathcal{H}(k) H ( k ) onto H ( M g [ φ 0 ∗ k ] ) \mathcal{H}(M_g[\varphi_0^*k]) H ( M g [ φ 0 ∗ k ]) .
Proof. If Ψ : E 2 → W \Psi: E_2 \to W Ψ : E 2 → W is a feature map for k k k , i.e., k ( y , y ′ ) = ⟨ Ψ ( y ) , Ψ ( y ′ ) ⟩ W k(y,y') = \langle\Psi(y),\Psi(y')\rangle_W k ( y , y ′ ) = ⟨ Ψ ( y ) , Ψ ( y ′ ) ⟩ W , then Ψ ′ ( x ) : = g ( x ) Ψ ( φ 0 ( x ) ) \Psi'(x) := g(x)\Psi(\varphi_0(x)) Ψ ′ ( x ) := g ( x ) Ψ ( φ 0 ( x )) is a feature map for M g [ φ 0 ∗ k ] M_g[\varphi_0^*k] M g [ φ 0 ∗ k ] , since ⟨ Ψ ′ ( x ) , Ψ ′ ( x ′ ) ⟩ W = g ( x ) g ( x ′ ) k ( φ 0 ( x ) , φ 0 ( x ′ ) ) = M g [ φ 0 ∗ k ] ( x , x ′ ) \langle\Psi'(x),\Psi'(x')\rangle_W = g(x)g(x')k(\varphi_0(x),\varphi_0(x')) = M_g[\varphi_0^*k](x,x') ⟨ Ψ ′ ( x ) , Ψ ′ ( x ′ ) ⟩ W = g ( x ) g ( x ′ ) k ( φ 0 ( x ) , φ 0 ( x ′ )) = M g [ φ 0 ∗ k ] ( x , x ′ ) . Write Φ ( w ) : = ⟨ w , Ψ ( ⋅ ) ⟩ W \Phi(w) := \langle w,\Psi(\cdot)\rangle_W Φ ( w ) := ⟨ w , Ψ ( ⋅ ) ⟩ W and Φ ′ ( w ) : = ⟨ w , Ψ ′ ( ⋅ ) ⟩ W \Phi'(w) := \langle w,\Psi'(\cdot)\rangle_W Φ ′ ( w ) := ⟨ w , Ψ ′ ( ⋅ ) ⟩ W for the corresponding representation maps, so that H ( k ) = { Φ ( w ) : w ∈ W } \mathcal{H}(k) = \{\Phi(w) : w \in W\} H ( k ) = { Φ ( w ) : w ∈ W } , H ( M g [ φ 0 ∗ k ] ) = { Φ ′ ( w ) : w ∈ W } \mathcal{H}(M_g[\varphi_0^*k]) = \{\Phi'(w) : w \in W\} H ( M g [ φ 0 ∗ k ]) = { Φ ′ ( w ) : w ∈ W } , and, for f ∈ H ( k ) f \in\mathcal{H}(k) f ∈ H ( k ) , ∥ f ∥ H ( k ) = min { ∥ w ∥ W : Φ ( w ) = f } \lVert f\rVert_{\mathcal{H}(k)} = \min\{\lVert w\rVert_W : \Phi(w) = f\} ∥ f ∥ H ( k ) = min {∥ w ∥ W : Φ ( w ) = f } , with the analogous expression for H ( M g [ φ 0 ∗ k ] ) \mathcal{H}(M_g[\varphi_0^*k]) H ( M g [ φ 0 ∗ k ]) via Φ ′ \Phi' Φ ′ . Since Φ ′ ( w ) ( x ) = ⟨ w , Ψ ′ ( x ) ⟩ W = g ( x ) ⟨ w , Ψ ( φ 0 ( x ) ) ⟩ W = g ( x ) Φ ( w ) ( φ 0 ( x ) ) \Phi'(w)(x) = \langle w,\Psi'(x)\rangle_W = g(x)\langle w,\Psi(\varphi_0(x))\rangle_W = g(x)\Phi(w)(\varphi_0(x)) Φ ′ ( w ) ( x ) = ⟨ w , Ψ ′ ( x ) ⟩ W = g ( x ) ⟨ w , Ψ ( φ 0 ( x )) ⟩ W = g ( x ) Φ ( w ) ( φ 0 ( x )) for every w ∈ W w \in W w ∈ W , we have Φ ′ = T ∘ Φ \Phi' = T \circ\Phi Φ ′ = T ∘ Φ , where T ( f ) : = g ⋅ ( f ∘ φ 0 ) T(f) := g \cdot(f \circ\varphi_0) T ( f ) := g ⋅ ( f ∘ φ 0 ) . In particular, any w 0 w_0 w 0 representing f f f on the left (i.e., Φ ( w 0 ) = f \Phi(w_0) = f Φ ( w 0 ) = f ) automatically represents T ( f ) T(f) T ( f ) on the right (i.e., Φ ′ ( w 0 ) = T ( f ) \Phi'(w_0) = T(f) Φ ′ ( w 0 ) = T ( f ) ). We claim that, for such w 0 w_0 w 0 , the set of all w ∈ W w \in W w ∈ W representing f f f on the left coincides with the set of all w ∈ W w \in W w ∈ W representing T ( f ) T(f) T ( f ) on the right. Since Φ \Phi Φ and Φ ′ \Phi' Φ ′ are linear, the former set is the coset w 0 + N w_0 + N w 0 + N and the latter is w 0 + N ′ w_0 + N' w 0 + N ′ , where N : = ker Φ N := \ker\Phi N := ker Φ and N ′ : = ker Φ ′ N' := \ker\Phi' N ′ := ker Φ ′ , so it suffices to show N = N ′ N = N' N = N ′ . Indeed, w ∈ N w \in N w ∈ N means ⟨ w , Ψ ( y ) ⟩ W = 0 \langle w,\Psi(y)\rangle_W = 0 ⟨ w , Ψ ( y ) ⟩ W = 0 for every y ∈ E 2 y \in E_2 y ∈ E 2 , which, since φ 0 \varphi_0 φ 0 is surjective, holds if and only if ⟨ w , Ψ ( φ 0 ( x ) ) ⟩ W = 0 \langle w,\Psi(\varphi_0(x))\rangle_W = 0 ⟨ w , Ψ ( φ 0 ( x )) ⟩ W = 0 for every x ∈ E 1 x \in E_1 x ∈ E 1 ; and since g g g is nowhere zero, this holds if and only if g ( x ) ⟨ w , Ψ ( φ 0 ( x ) ) ⟩ W = ⟨ w , Ψ ′ ( x ) ⟩ W = 0 g(x)\langle w,\Psi(\varphi_0(x))\rangle_W = \langle w,\Psi'(x)\rangle_W = 0 g ( x ) ⟨ w , Ψ ( φ 0 ( x )) ⟩ W = ⟨ w , Ψ ′ ( x ) ⟩ W = 0 for every x ∈ E 1 x \in E_1 x ∈ E 1 , i.e., w ∈ N ′ w \in N' w ∈ N ′ . Hence N = N ′ N = N' N = N ′ , and the two cosets w 0 + N w_0 + N w 0 + N , w 0 + N ′ w_0 + N' w 0 + N ′ are identical as subsets of W W W . Consequently, for every f ∈ H ( k ) f \in\mathcal{H}(k) f ∈ H ( k ) , the set of w ∈ W w \in W w ∈ W representing f f f on the left, namely w 0 + N w_0 + N w 0 + N , is exactly the set of w ∈ W w \in W w ∈ W representing T ( f ) T(f) T ( f ) on the right, namely w 0 + N ′ = w 0 + N w_0 + N' = w_0 + N w 0 + N ′ = w 0 + N . Minimizing ∥ w ∥ W \lVert w\rVert_W ∥ w ∥ W over this common set gives:
∥ f ∥ H ( k ) = min w ∈ w 0 + N ∥ w ∥ W = min w ∈ w 0 + N ′ ∥ w ∥ W = ∥ T ( f ) ∥ H ( M g [ φ 0 ∗ k ] ) = ∥ g ⋅ ( f ∘ φ 0 ) ∥ H ( M g [ φ 0 ∗ k ] ) . \lVert f\rVert_{\mathcal{H}(k)} = \min_{w\in w_0+N}\lVert w\rVert_W = \min_{w\in w_0+N'}\lVert w\rVert_W = \lVert T(f)\rVert_{\mathcal{H}(M_g[\varphi_0^*k])} = \lVert g\cdot(f\circ\varphi_0)\rVert_{\mathcal{H}(M_g[\varphi_0^*k])}. ∥ f ∥ H ( k ) = w ∈ w 0 + N min ∥ w ∥ W = w ∈ w 0 + N ′ min ∥ w ∥ W = ∥ T ( f ) ∥ H ( M g [ φ 0 ∗ k ]) = ∥ g ⋅ ( f ∘ φ 0 ) ∥ H ( M g [ φ 0 ∗ k ]) .
In addition, T T T is a bijection from H ( k ) \mathcal{H}(k) H ( k ) onto H ( M g [ φ 0 ∗ k ] ) \mathcal{H}(M_g[\varphi_0^*k]) H ( M g [ φ 0 ∗ k ]) . Indeed, if T ( f 1 ) = T ( f 2 ) T(f_1) = T(f_2) T ( f 1 ) = T ( f 2 ) , then g ( x ) f 1 ( φ 0 ( x ) ) = g ( x ) f 2 ( φ 0 ( x ) ) g(x)f_1(\varphi_0(x)) = g(x)f_2(\varphi_0(x)) g ( x ) f 1 ( φ 0 ( x )) = g ( x ) f 2 ( φ 0 ( x )) for all x ∈ E 1 x \in E_1 x ∈ E 1 . Since g g g is nowhere zero, f 1 ( φ 0 ( x ) ) = f 2 ( φ 0 ( x ) ) f_1(\varphi_0(x)) = f_2(\varphi_0(x)) f 1 ( φ 0 ( x )) = f 2 ( φ 0 ( x )) for all x ∈ E 1 x \in E_1 x ∈ E 1 and since φ 0 \varphi_0 φ 0 is surjective, f 1 = f 2 f_1 = f_2 f 1 = f 2 . Thus T T T is injective. Surjectivity holds because every h ∈ H ( M g [ φ 0 ∗ k ] ) h \in\mathcal{H}(M_g[\varphi_0^*k]) h ∈ H ( M g [ φ 0 ∗ k ]) equals Φ ′ ( w ) \Phi'(w) Φ ′ ( w ) for some w w w , hence h = T ( Φ ( w ) ) h = T(\Phi(w)) h = T ( Φ ( w )) . Together with the norm identity above, T T T is an isometric isomorphism.
Lemma G.7. The map ϕ \phi ϕ is a real-analytic diffeomorphism of R p \mathbb{R}^p R p onto the open upper hemisphere S + p : = { y ∈ S p : y p + 1 > 0 } \mathbb{S}_+^p := \{y \in\mathbb{S}^p : y_{p+1} > 0\} S + p := { y ∈ S p : y p + 1 > 0 } , with inverse ϕ − 1 ( y 1 , … , y p ) = y / y p + 1 \phi^{-1}(y_1,\ldots,y_p) = y/y_{p+1} ϕ − 1 ( y 1 , … , y p ) = y / y p + 1 . Moreover, under the assumption that there exists A > 0 A > 0 A > 0 such that ∥ x ∥ ≤ A \lVert x\rVert\le A ∥ x ∥ ≤ A for x ∈ X x \in\mathcal{X} x ∈ X , it holds that:
ϕ ( x ) p + 1 = ( 1 + ∥ x ∥ 2 ) − 1 / 2 ≥ ( 1 + A 2 ) − 1 / 2 > 0 for all x ∈ X , (71) \phi(x)_{p+1} = (1+\lVert x\rVert^2)^{-1/2} \ge(1+A^2)^{-1/2} > 0 \qquad\text{for all } x \in\mathcal{X}, \tag*{(71)} ϕ ( x ) p + 1 = ( 1 + ∥ x ∥ 2 ) − 1/2 ≥ ( 1 + A 2 ) − 1/2 > 0 for all x ∈ X , (71) so X ^ : = ϕ ( X ) \widehat{\mathcal{X}} := \phi(\mathcal{X}) X := ϕ ( X ) has closure contained in the compact cap { y ∈ S p : y p + 1 ≥ ( 1 + A 2 ) − 1 / 2 } ⊂ S + p \{y \in\mathbb{S}^p : y_{p+1} \ge(1+A^2)^{-1/2}\} \subset\mathbb{S}_+^p { y ∈ S p : y p + 1 ≥ ( 1 + A 2 ) − 1/2 } ⊂ S + p . Consequently, there exists χ ∈ C ∞ ( S p ) \chi\in C^\infty(\mathbb{S}^p) χ ∈ C ∞ ( S p ) with 0 ≤ χ ≤ 1 0 \le\chi\le1 0 ≤ χ ≤ 1 , χ = 1 \chi= 1 χ = 1 on X ^ \widehat{\mathcal{X}} X , and supp χ ⊂ S + p \operatorname{supp}\chi\subset\mathbb{S}_+^p supp χ ⊂ S + p .
Proof. The map ϕ \phi ϕ is real-analytic on R p \mathbb{R}^p R p , since its coordinate functions are compositions of polynomials with the real-analytic function t ↦ t − 1 / 2 t \mapsto t^{-1/2} t ↦ t − 1/2 on ( 0 , ∞ ) (0,\infty) ( 0 , ∞ ) . Likewise, ϕ − 1 \phi^{-1} ϕ − 1 is real-analytic on S + p \mathbb{S}_+^p S + p , since y p + 1 > 0 y_{p+1} > 0 y p + 1 > 0 there, and hence t ↦ t − 1 t \mapsto t^{-1} t ↦ t − 1 is real-analytic on ( 0 , ∞ ) (0,\infty) ( 0 , ∞ ) . A direct calculation shows that ϕ − 1 ∘ ϕ = Id R p \phi^{-1} \circ\phi= \operatorname{Id}_{\mathbb{R}^p} ϕ − 1 ∘ ϕ = Id R p and ϕ ∘ ϕ − 1 = Id S + p \phi\circ\phi^{-1} = \operatorname{Id}_{\mathbb{S}_+^p} ϕ ∘ ϕ − 1 = Id S + p , so ϕ \phi ϕ is a real-analytic diffeomorphism. Equation (71) follows from ∥ x ∥ ≤ A \lVert x\rVert\le A ∥ x ∥ ≤ A since t ↦ ( 1 + t ) − 1 / 2 t \mapsto(1+t)^{-1/2} t ↦ ( 1 + t ) − 1/2 is decreasing. Let C A : = { y ∈ S p : y p + 1 ≥ ( 1 + A 2 ) − 1 / 2 } C_A := \{y \in\mathbb{S}^p : y_{p+1} \ge(1+A^2)^{-1/2}\} C A := { y ∈ S p : y p + 1 ≥ ( 1 + A 2 ) − 1/2 } . By Equation (71) , X ^ ⊂ C A \widehat{\mathcal{X}} \subset C_A X ⊂ C A , and since C A C_A C A is closed (the preimage of a closed half-line under the
continuous map y ↦ y p + 1 y \mapsto y_{p+1} y ↦ y p + 1 , X ^ ⊂ C A \widehat{\mathcal{X}} \subset C_A X ⊂ C A as well. In particular, X ^ \widehat{\mathcal{X}} X is compact being a closed subset of the compact C A C_A C A . Moreover, C A ⊂ S + p C_A \subset\mathbb{S}^{p}_{+} C A ⊂ S + p since ( 1 + A 2 ) − 1 / 2 > 0 (1+A^2)^{-1/2}>0 ( 1 + A 2 ) − 1/2 > 0 . A smooth bump function for the pair ( X ^ , S + p ) (\widehat{\mathcal{X}},\mathbb{S}^{p}_{+}) ( X , S + p ) , which exists since X ^ \widehat{\mathcal{X}} X is compact and contained in the open set S + p \mathbb{S}^{p}_{+} S + p , gives χ \chi χ [60 Proposition 2.26] .
Corollary G.8. Define X ^ : = ϕ ( X ) \widehat{\mathcal{X}} := \phi(\mathcal{X}) X := ϕ ( X ) . It holds that:
H N T K = q ⋅ [ H ( K 0 ∣ X ^ ) ∘ ϕ ] + R ⋅ 1 X , (72) \mathcal{H}^{\mathrm{NTK}} = q \cdot\left[\mathcal{H}(K_0|_{\widehat{\mathcal{X}}}) \circ\phi\right] + \mathbb{R} \cdot\mathbf{1}_{\mathcal{X}}, \tag*{(72)} H NTK = q ⋅ [ H ( K 0 ∣ X ) ∘ ϕ ] + R ⋅ 1 X , (72)
where 1 X : X → R \mathbf{1}_{\mathcal{X}} : \mathcal{X} \to\mathbb{R} 1 X : X → R denotes the constant-one function, i.e., 1 X ( x ) = 1 \mathbf{1}_{\mathcal{X}}(x)=1 1 X ( x ) = 1 for all x ∈ X x \in\mathcal{X} x ∈ X .
Proof. Since X ^ : = ϕ ( X ) \widehat{\mathcal{X}} := \phi(\mathcal{X}) X := ϕ ( X ) , the restriction ϕ ∣ X \phi|_{\mathcal{X}} ϕ ∣ X is a surjection onto X ^ \widehat{\mathcal{X}} X , and q ≥ 1 > 0 q \geq1 > 0 q ≥ 1 > 0 on X \mathcal{X} X . Lemma G.6 therefore applies to the kernel M q [ ϕ ∗ K 0 ] M_q[\phi^*K_0] M q [ ϕ ∗ K 0 ] appearing in Equation (70) , giving H ( M q [ ϕ ∗ K 0 ] ) = q ⋅ H ( K 0 ∣ X ^ ) ∘ ϕ \mathcal{H}(M_q[\phi^*K_0]) = q \cdot\mathcal{H}(K_0|_{\widehat{\mathcal{X}}}) \circ\phi H ( M q [ ϕ ∗ K 0 ]) = q ⋅ H ( K 0 ∣ X ) ∘ ϕ isometrically. The factor 1 2 \frac{1}{2} 2 1 rescales the norm only, and the sum property of H ( ⋅ ) \mathcal{H}(\cdot) H ( ⋅ ) recalled above assembles the two terms of Equation (70) into Equation (72) .
The sphere kernel: parity gap and the hemisphere lemma
Denote by L 2 ( S p ) L^2(\mathbb{S}^p) L 2 ( S p ) the space of square-integrable functions on S p \mathbb{S}^p S p with respect to its surface measure. Let { Y k , j } k ≥ 0 , 1 ≤ j ≤ a k \{Y_{k,j}\}_{k\geq0,\,1\leq j\leq a_k} { Y k , j } k ≥ 0 , 1 ≤ j ≤ a k denote an orthonormal basis of L 2 ( S p ) L^2(\mathbb{S}^p) L 2 ( S p ) , with Y k , j Y_{k,j} Y k , j a spherical harmonic of degree k k k and a k a_k a k the dimension of the degree-k k k harmonics Y k \mathcal{Y}_k Y k . Every f ∈ L 2 ( S p ) f \in L^2(\mathbb{S}^p) f ∈ L 2 ( S p ) decomposes uniquely as f = ∑ k ≥ 0 f k f=\sum_{k\geq0}f_k f = ∑ k ≥ 0 f k , f k : = ∑ j = 1 a k ⟨ f , Y k , j ⟩ Y k , j ∈ Y k f_k:=\sum_{j=1}^{a_k}\langle f,Y_{k,j}\rangle Y_{k,j}\in\mathcal{Y}_k f k := ∑ j = 1 a k ⟨ f , Y k , j ⟩ Y k , j ∈ Y k . For τ ≥ 0 \tau\geq0 τ ≥ 0 , the Sobolev space H τ ( S p ) H^\tau(\mathbb{S}^p) H τ ( S p ) is defined spectrally by ∥ f ∥ H τ ( S p ) 2 : = ∑ k ≥ 0 ( 1 + k ( k + p − 1 ) ) τ ∥ f k ∥ L 2 ( S p ) 2 ≍ ∑ k ≥ 0 ( 1 + k ) 2 τ ∥ f k ∥ L 2 ( S p ) 2 \|f\|^2_{H^\tau(\mathbb{S}^p)}:=\sum_{k\geq0}(1+k(k+p-1))^\tau\|f_k\|^2_{L^2(\mathbb{S}^p)}\asymp\sum_{k\geq0}(1+k)^{2\tau}\|f_k\|^2_{L^2(\mathbb{S}^p)} ∥ f ∥ H τ ( S p ) 2 := ∑ k ≥ 0 ( 1 + k ( k + p − 1 ) ) τ ∥ f k ∥ L 2 ( S p ) 2 ≍ ∑ k ≥ 0 ( 1 + k ) 2 τ ∥ f k ∥ L 2 ( S p ) 2 . Given a dot-product kernel κ ( ⟨ y , y ′ ⟩ ) \kappa(\langle y,y'\rangle) κ (⟨ y , y ′ ⟩) on S p \mathbb{S}^p S p with Mercer decomposition κ ( ⟨ y , y ′ ⟩ ) = ∑ k ≥ 0 μ k ∑ j = 1 a k Y k , j ( y ) Y k , j ( y ′ ) \kappa(\langle y,y'\rangle)=\sum_{k\geq0}\mu_k\sum_{j=1}^{a_k}Y_{k,j}(y)Y_{k,j}(y') κ (⟨ y , y ′ ⟩) = ∑ k ≥ 0 μ k ∑ j = 1 a k Y k , j ( y ) Y k , j ( y ′ ) , its RKHS is:
H ( κ ) : = { f = ∑ k ≥ 0 ∑ j = 1 a k c k , j Y k , j : ∥ f ∥ H ( κ ) 2 : = ∑ k ≥ 0 ∑ j = 1 a k c k , j 2 μ k < ∞ } . \mathcal{H}(\kappa):=\left\{f=\sum_{k\geq0}\sum_{j=1}^{a_k}c_{k,j}Y_{k,j}:\|f\|^2_{\mathcal{H}(\kappa)}:=\sum_{k\geq0}\sum_{j=1}^{a_k}\frac{c_{k,j}^2}{\mu_k}<\infty\right\}. H ( κ ) := { f = k ≥ 0 ∑ j = 1 ∑ a k c k , j Y k , j : ∥ f ∥ H ( κ ) 2 := k ≥ 0 ∑ j = 1 ∑ a k μ k c k , j 2 < ∞ } .
with the convention, forced by finiteness, that c k , j = 0 c_{k,j}=0 c k , j = 0 whenever μ k = 0 \mu_k=0 μ k = 0 . Moreover, for an open subset V ⊂ S p V\subset\mathbb{S}^p V ⊂ S p , the Sobolev space H τ ( V ) H^\tau(V) H τ ( V ) is defined by restriction H τ ( V ) : = { U ∣ V : U ∈ H τ ( S p ) } H^\tau(V):=\{U|_V:U\in H^\tau(\mathbb{S}^p)\} H τ ( V ) := { U ∣ V : U ∈ H τ ( S p )} with norm ∥ g ∥ H τ ( V ) : = inf { ∥ U ∥ H τ ( S p ) : U ∈ H τ ( S p ) , U ∣ V = g } \|g\|_{H^\tau(V)}:=\inf\{\|U\|_{H^\tau(\mathbb{S}^p)}:U\in H^\tau(\mathbb{S}^p),U|_V=g\} ∥ g ∥ H τ ( V ) := inf { ∥ U ∥ H τ ( S p ) : U ∈ H τ ( S p ) , U ∣ V = g } . For more details, we refer the reader to [10 , 12 , 49 ] .
Corollary G.9 (Parity gap on S p \mathbb{S}^p S p ). Recall K 0 ( y , y ′ ) : = κ ( ⟨ y , y ′ ⟩ ) K_0(y,y'):=\kappa(\langle y,y'\rangle) K 0 ( y , y ′ ) := κ (⟨ y , y ′ ⟩) for y , y ′ ∈ S p y,y'\in\mathbb{S}^p y , y ′ ∈ S p , with Mercer eigenvalues ( μ k ) k ≥ 0 (\mu_k)_{k\geq0} ( μ k ) k ≥ 0 . Then, we have:
μ 0 , μ 1 > 0 , μ k = 0 for every odd k ≥ 3 , μ k ∼ C ( p + 1 ) k − ( p + 1 ) otherwise as k → ∞ , \mu_0,\mu_1>0,\qquad\mu_k=0\text{ for every odd }k\geq3,\qquad\mu_k\sim C(p+1)k^{-(p+1)}\text{ otherwise as }k\to\infty, μ 0 , μ 1 > 0 , μ k = 0 for every odd k ≥ 3 , μ k ∼ C ( p + 1 ) k − ( p + 1 ) otherwise as k → ∞ ,
for a constant C ( p + 1 ) > 0 C(p+1)>0 C ( p + 1 ) > 0 . Consequently, writing E : = ⨁ k e v e n Y k ⊕ Y 1 ⊂ L 2 ( S p ) \mathcal{E}:=\bigoplus_{k\ \mathrm{even}}\mathcal{Y}_k\oplus\mathcal{Y}_1\subset L^2(\mathbb{S}^p) E := ⨁ k even Y k ⊕ Y 1 ⊂ L 2 ( S p ) , it holds that:
H ( K 0 ) ≅ E ∩ H p + 1 2 ( S p ) ⊊ H p + 1 2 ( S p ) . \mathcal{H}(K_0)\cong\mathcal{E}\cap H^{\frac{p+1}{2}}(\mathbb{S}^p)\subsetneq H^{\frac{p+1}{2}}(\mathbb{S}^p). H ( K 0 ) ≅ E ∩ H 2 p + 1 ( S p ) ⊊ H 2 p + 1 ( S p ) .
Proof. The first part of this corollary comes from [18 Proposition 5] .
Identifying E \mathcal{E} E . Since L 2 ( S p ) = ⨁ k ≥ 0 Y k L^2(\mathbb{S}^p)=\bigoplus_{k\geq0}\mathcal{Y}_k L 2 ( S p ) = ⨁ k ≥ 0 Y k is an orthogonal decomposition, the closed subspace E \mathcal{E} E consists exactly of those f = ∑ k ≥ 0 f k ∈ L 2 ( S p ) f=\sum_{k\geq0}f_k\in L^2(\mathbb{S}^p) f = ∑ k ≥ 0 f k ∈ L 2 ( S p ) with f k = 0 f_k=0 f k = 0 for every odd k ≥ 3 k\geq3 k ≥ 3 . We note that the spherical-harmonic expansion already converges in L 2 ( S p ) L^2(\mathbb{S}^p) L 2 ( S p ) , since we have by Parseval’s identity that ∥ f ∥ L 2 ( S p ) 2 = ∑ k ∥ f k ∥ L 2 ( S p ) 2 < ∞ \|f\|^2_{L^2(\mathbb{S}^p)}=\sum_k\|f_k\|^2_{L^2(\mathbb{S}^p)}<\infty ∥ f ∥ L 2 ( S p ) 2 = ∑ k ∥ f k ∥ L 2 ( S p ) 2 < ∞ for any f ∈ L 2 ( S p ) f\in L^2(\mathbb{S}^p) f ∈ L 2 ( S p ) .
A two-sided bound on μ k − 1 \mu_k^{-1} μ k − 1 . Inverting μ k ∼ C ( p + 1 ) k − ( p + 1 ) \mu_k\sim C(p+1)k^{-(p+1)} μ k ∼ C ( p + 1 ) k − ( p + 1 ) gives μ k − 1 ∼ C ( p + 1 ) − 1 k p + 1 \mu_k^{-1}\sim C(p+1)^{-1}k^{p+1} μ k − 1 ∼ C ( p + 1 ) − 1 k p + 1 as k → ∞ k\to\infty k → ∞ . By the definition of asymptotic equivalence, there exist constants C 0 , C 0 ′ > 0 C_0,C_0'>0 C 0 , C 0 ′ > 0 depending only on p p p and K ≥ 1 K\geq1 K ≥ 1 such that c 0 k p + 1 ≤ μ k − 1 ≤ C 0 k p + 1 c_0k^{p+1}\leq\mu_k^{-1}\leq C_0k^{p+1} c 0 k p + 1 ≤ μ k − 1 ≤ C 0 k p + 1 , for k ≥ K k\geq K k ≥ K . It remains to control the finitely many indices k < K k<K k < K . Since μ k > 0 \mu_k>0 μ k > 0 for every k ∈ { 1 } ∪ 2 N 0 k\in\{1\}\cup2\mathbb{N}_0 k ∈ { 1 } ∪ 2 N 0 with 2 N 0 = { 0 , 2 , 4 , 6 , … } 2\mathbb{N}_0=\{0,2,4,6,\ldots\} 2 N 0 = { 0 , 2 , 4 , 6 , … } , the quantities μ k − 1 ( 1 + k ) p + 1 \frac{\mu_k^{-1}}{(1+k)^{p+1}} ( 1 + k ) p + 1 μ k − 1 , with k ∈ { 1 } ∪ 2 N 0 k\in\{1\}\cup2\mathbb{N}_0 k ∈ { 1 } ∪ 2 N 0 and k < K k<K k < K , form a finite collection of strictly positive finite numbers. Thus, their minimum is positive and their maximum is finite. Combining these bounds with the preceding estimate for k ≥ K k \ge K k ≥ K , and using k p + 1 ≍ ( 1 + k ) p + 1 k^{p+1} \asymp(1+k)^{p+1} k p + 1 ≍ ( 1 + k ) p + 1 for k ≥ 1 k \ge1 k ≥ 1 , yields constants c , C > 0 c,C>0 c , C > 0 , depending only on p p p , such that c ( 1 + k ) p + 1 ≤ μ k − 1 ≤ C ( 1 + k ) p + 1 c(1+k)^{p+1} \le\mu_k^{-1} \le C(1+k)^{p+1} c ( 1 + k ) p + 1 ≤ μ k − 1 ≤ C ( 1 + k ) p + 1 for k ∈ { 1 } ∪ 2 N 0 k \in\{1\} \cup2\mathbb{N}_0 k ∈ { 1 } ∪ 2 N 0 .
( ⊆ ) (\subseteq) ( ⊆ ) . Let f ∈ H ( K 0 ) f \in\mathcal{H}(K_0) f ∈ H ( K 0 ) , so f = ∑ k , j c k , j Y k , j f=\sum_{k,j}c_{k,j}Y_{k,j} f = ∑ k , j c k , j Y k , j , with ∑ k , j c k , j 2 / μ k < ∞ \sum_{k,j}c_{k,j}^2/\mu_k<\infty ∑ k , j c k , j 2 / μ k < ∞ and c k , j = 0 c_{k,j}=0 c k , j = 0 forced wherever μ k = 0 \mu_k=0 μ k = 0 , i.e., at every odd k ≥ 3 k\ge3 k ≥ 3 . By the identification above this already gives f ∈ E f\in\mathcal{E} f ∈ E . Summing the norm condition over j j j (Parseval within each degree, ∑ j c k , j 2 = ∥ f k ∥ L 2 ( S p ) 2 \sum_j c_{k,j}^2=\lVert f_k\rVert_{L^2(\mathbb{S}^p)}^2 ∑ j c k , j 2 = ∥ f k ∥ L 2 ( S p ) 2 ) and using the lower-bound on μ k − 1 \mu_k^{-1} μ k − 1 , we get:
∥ f ∥ H p + 1 2 ( S p ) 2 = ∑ k ≥ 0 ( 1 + k ) p + 1 ∥ f k ∥ L 2 ( S p ) 2 = ∑ k ∈ { 1 } ∪ 2 N 0 ( 1 + k ) p + 1 ∥ f k ∥ L 2 ( S p ) 2 ≤ 1 c ∑ k ∈ { 1 } ∪ 2 N 0 μ k − 1 ∥ f k ∥ L 2 ( S p ) 2 = 1 c ∥ f ∥ H ( K 0 ) 2 < ∞ . \lVert f\rVert_{H^{\frac{p+1}{2}}(\mathbb{S}^p)}^2
=\sum_{k\ge0}(1+k)^{p+1}\lVert f_k\rVert_{L^2(\mathbb{S}^p)}^2
=\sum_{k\in\{1\}\cup2\mathbb{N}_0}(1+k)^{p+1}\lVert f_k\rVert_{L^2(\mathbb{S}^p)}^2
\le\frac{1}{c}\sum_{k\in\{1\}\cup2\mathbb{N}_0}\mu_k^{-1}\lVert f_k\rVert_{L^2(\mathbb{S}^p)}^2
=\frac{1}{c}\lVert f\rVert_{\mathcal{H}(K_0)}^2<\infty. ∥ f ∥ H 2 p + 1 ( S p ) 2 = k ≥ 0 ∑ ( 1 + k ) p + 1 ∥ f k ∥ L 2 ( S p ) 2 = k ∈ { 1 } ∪ 2 N 0 ∑ ( 1 + k ) p + 1 ∥ f k ∥ L 2 ( S p ) 2 ≤ c 1 k ∈ { 1 } ∪ 2 N 0 ∑ μ k − 1 ∥ f k ∥ L 2 ( S p ) 2 = c 1 ∥ f ∥ H ( K 0 ) 2 < ∞.
where the first equality holds because f k = 0 f_k=0 f k = 0 for k ∉ { 1 } ∪ 2 N 0 k\notin\{1\}\cup2\mathbb{N}_0 k ∈ / { 1 } ∪ 2 N 0 ; so f ∈ H p + 1 2 ( S p ) f\in H^{\frac{p+1}{2}}(\mathbb{S}^p) f ∈ H 2 p + 1 ( S p ) . Hence f ∈ E ∩ H p + 1 2 ( S p ) f\in\mathcal{E}\cap H^{\frac{p+1}{2}}(\mathbb{S}^p) f ∈ E ∩ H 2 p + 1 ( S p ) .
( ⊇ ) (\supseteq) ( ⊇ ) . Let f ∈ E ∩ H p + 1 2 ( S p ) f\in\mathcal{E}\cap H^{\frac{p+1}{2}}(\mathbb{S}^p) f ∈ E ∩ H 2 p + 1 ( S p ) . Then, f k = 0 f_k=0 f k = 0 for every odd k ≥ 3 k\ge3 k ≥ 3 , which is exactly the forcing condition defining H ( K 0 ) \mathcal{H}(K_0) H ( K 0 ) above, so f f f is an admissible candidate for H ( K 0 ) \mathcal{H}(K_0) H ( K 0 ) . Since f ∈ H p + 1 2 ( S p ) f\in H^{\frac{p+1}{2}}(\mathbb{S}^p) f ∈ H 2 p + 1 ( S p ) , ∑ k ≥ 0 ( 1 + k ) p + 1 ∥ f k ∥ L 2 ( S p ) 2 < ∞ \sum_{k\ge0}(1+k)^{p+1}\lVert f_k\rVert_{L^2(\mathbb{S}^p)}^2<\infty ∑ k ≥ 0 ( 1 + k ) p + 1 ∥ f k ∥ L 2 ( S p ) 2 < ∞ . As f ∈ E f\in\mathcal{E} f ∈ E , this sum is already supported on { 1 } ∪ 2 N 0 \{1\}\cup2\mathbb{N}_0 { 1 } ∪ 2 N 0 , so the upper-bound on μ k − 1 \mu_k^{-1} μ k − 1 gives:
∥ f ∥ H ( K 0 ) 2 = ∑ k ∈ { 1 } ∪ 2 N 0 μ k − 1 ∥ f k ∥ L 2 ( S p ) 2 ≤ C ∑ k ∈ { 1 } ∪ 2 N 0 ( 1 + k ) p + 1 ∥ f k ∥ L 2 ( S p ) 2 = C ∥ f ∥ H p + 1 2 ( S p ) 2 < ∞ , \lVert f\rVert_{\mathcal{H}(K_0)}^2
=\sum_{k\in\{1\}\cup2\mathbb{N}_0}\mu_k^{-1}\lVert f_k\rVert_{L^2(\mathbb{S}^p)}^2
\le C\sum_{k\in\{1\}\cup2\mathbb{N}_0}(1+k)^{p+1}\lVert f_k\rVert_{L^2(\mathbb{S}^p)}^2
=C\lVert f\rVert_{H^{\frac{p+1}{2}}(\mathbb{S}^p)}^2<\infty, ∥ f ∥ H ( K 0 ) 2 = k ∈ { 1 } ∪ 2 N 0 ∑ μ k − 1 ∥ f k ∥ L 2 ( S p ) 2 ≤ C k ∈ { 1 } ∪ 2 N 0 ∑ ( 1 + k ) p + 1 ∥ f k ∥ L 2 ( S p ) 2 = C ∥ f ∥ H 2 p + 1 ( S p ) 2 < ∞ ,
where we use the fact that f ∈ H p + 1 2 ( S p ) f\in H^{\frac{p+1}{2}}(\mathbb{S}^p) f ∈ H 2 p + 1 ( S p ) . Hence, f ∈ H ( K 0 ) f\in\mathcal{H}(K_0) f ∈ H ( K 0 ) .
The two inclusions give H ( K 0 ) = E ∩ H p + 1 2 ( S p ) \mathcal{H}(K_0)=\mathcal{E}\cap H^{\frac{p+1}{2}}(\mathbb{S}^p) H ( K 0 ) = E ∩ H 2 p + 1 ( S p ) as sets, and the same constants c , C c,C c , C , one used in each direction, give the equivalence of norms. Together, this gives H ( K 0 ) ≅ E ∩ H p + 1 2 ( S p ) \mathcal{H}(K_0)\cong\mathcal{E}\cap H^{\frac{p+1}{2}}(\mathbb{S}^p) H ( K 0 ) ≅ E ∩ H 2 p + 1 ( S p ) .
Properness. A nonzero Y ∈ Y 3 Y\in\mathcal{Y}_3 Y ∈ Y 3 is a polynomial, hence lies in every H τ ( S p ) H^\tau(\mathbb{S}^p) H τ ( S p ) for τ ≥ 0 \tau\ge0 τ ≥ 0 . Its only nonzero component is at degree 3 3 3 , which is odd and ≥ 3 \ge3 ≥ 3 , so Y ∉ E Y\notin\mathcal{E} Y ∈ / E by the identification above, hence Y ∉ H ( K 0 ) Y\notin\mathcal{H}(K_0) Y ∈ / H ( K 0 ) . So the containment H ( K 0 ) ⊊ H p + 1 2 ( S p ) \mathcal{H}(K_0)\subsetneq H^{\frac{p+1}{2}}(\mathbb{S}^p) H ( K 0 ) ⊊ H 2 p + 1 ( S p ) is strict.
For y ∈ S p y\in\mathbb{S}^p y ∈ S p , write a ( y ) : = − y a(y):=-y a ( y ) := − y for the antipodal map. We first record how composition with a a a acts on the spherical-harmonic expansion of a function.
Lemma G.10. For any h ∈ L 2 ( S p ) h\in L^2(\mathbb{S}^p) h ∈ L 2 ( S p ) with expansion h = ∑ k ≥ 0 h k h=\sum_{k\ge0}h_k h = ∑ k ≥ 0 h k , h k ∈ Y k h_k\in\mathcal{Y}_k h k ∈ Y k , the function h ∘ a h\circ a h ∘ a has expansion ( h ∘ a ) k = ( − 1 ) k h k (h\circ a)_k=(-1)^kh_k ( h ∘ a ) k = ( − 1 ) k h k . Consequently, ∥ h ∘ a ∥ H τ ( S p ) = ∥ h ∥ H τ ( S p ) \lVert h\circ a\rVert_{H^\tau(\mathbb{S}^p)}=\lVert h\rVert_{H^\tau(\mathbb{S}^p)} ∥ h ∘ a ∥ H τ ( S p ) = ∥ h ∥ H τ ( S p ) for every τ ≥ 0 \tau\ge0 τ ≥ 0 .
Proof. Every Y ∈ Y k Y\in\mathcal{Y}_k Y ∈ Y k is the restriction to S p \mathbb{S}^p S p of a polynomial on R p + 1 \mathbb{R}^{p+1} R p + 1 that is homogeneous of degree k k k , i.e., Y ( λ x ) = λ k Y ( x ) Y(\lambda x)=\lambda^kY(x) Y ( λ x ) = λ k Y ( x ) for every scalar λ \lambda λ . Taking λ = − 1 \lambda=-1 λ = − 1 gives Y ( − y ) = ( − 1 ) k Y ( y ) Y(-y)=(-1)^kY(y) Y ( − y ) = ( − 1 ) k Y ( y ) for every y ∈ S p y\in\mathbb{S}^p y ∈ S p . Applying this to each term of h = ∑ k ≥ 0 h k h=\sum_{k\ge0}h_k h = ∑ k ≥ 0 h k yields:
h ( a ( y ) ) = h ( − y ) = ∑ k ≥ 0 h k ( − y ) = ∑ k ≥ 0 ( − 1 ) k h k ( y ) , h(a(y))=h(-y)=\sum_{k\ge0}h_k(-y)=\sum_{k\ge0}(-1)^kh_k(y), h ( a ( y )) = h ( − y ) = k ≥ 0 ∑ h k ( − y ) = k ≥ 0 ∑ ( − 1 ) k h k ( y ) ,
so h ∘ a = ∑ k ≥ 0 ( − 1 ) k h k h\circ a=\sum_{k\ge0}(-1)^kh_k h ∘ a = ∑ k ≥ 0 ( − 1 ) k h k . Since ( − 1 ) k h k ∈ Y k (-1)^kh_k\in\mathcal{Y}_k ( − 1 ) k h k ∈ Y k , this is precisely the spherical-harmonic expansion of h ∘ a h\circ a h ∘ a , giving ( h ∘ a ) k = ( − 1 ) k h k (h\circ a)_k=(-1)^kh_k ( h ∘ a ) k = ( − 1 ) k h k . Since ∥ ( − 1 ) k h k ∥ L 2 ( S p ) = ∥ h k ∥ L 2 ( S p ) \lVert(-1)^kh_k\rVert_{L^2(\mathbb{S}^p)}=\lVert h_k\rVert_{L^2(\mathbb{S}^p)} ∥( − 1 ) k h k ∥ L 2 ( S p ) = ∥ h k ∥ L 2 ( S p ) for every k k k , we obtain:
∥ h ∘ a ∥ H τ ( S p ) 2 = ∑ k ≥ 0 ( 1 + k ( k + p − 1 ) ) τ ∥ ( h ∘ a ) k ∥ L 2 ( S p ) 2 = ∑ k ≥ 0 ( 1 + k ( k + p − 1 ) ) τ ∥ h k ∥ L 2 ( S p ) 2 = ∥ h ∥ H τ ( S p ) 2 . \lVert h\circ a\rVert_{H^\tau(\mathbb{S}^p)}^2
=\sum_{k\ge0}(1+k(k+p-1))^\tau\lVert(h\circ a)_k\rVert_{L^2(\mathbb{S}^p)}^2
=\sum_{k\ge0}(1+k(k+p-1))^\tau\lVert h_k\rVert_{L^2(\mathbb{S}^p)}^2
=\lVert h\rVert_{H^\tau(\mathbb{S}^p)}^2. ∥ h ∘ a ∥ H τ ( S p ) 2 = k ≥ 0 ∑ ( 1 + k ( k + p − 1 ) ) τ ∥( h ∘ a ) k ∥ L 2 ( S p ) 2 = k ≥ 0 ∑ ( 1 + k ( k + p − 1 ) ) τ ∥ h k ∥ L 2 ( S p ) 2 = ∥ h ∥ H τ ( S p ) 2 .
Lemma G.11 (Hemisphere lemma). Let V ⊂ S p \mathcal{V}\subset\mathbb{S}^p V ⊂ S p be open, bounded, with Lipschitz boundary, with V ‾ \overline{\mathcal{V}} V compactly contained in S + p \mathbb{S}^p_+ S + p . Then it holds that H ( K 0 ∣ V ) ≅ H p + 1 2 ( V ) \mathcal{H}(K_0|_{\mathcal{V}})\cong H^{\frac{p+1}{2}}(\mathcal{V}) H ( K 0 ∣ V ) ≅ H 2 p + 1 ( V ) .
Proof. (⊆ \subseteq ⊆ ). Let g ∈ H ( K 0 ∣ V ) g \in\mathcal{H}(K_{0\mid\mathcal{V}}) g ∈ H ( K 0 ∣ V ) . By the restriction property of RKHSs, there exists F ∈ H ( K 0 ) F \in\mathcal{H}(K_0) F ∈ H ( K 0 ) with F ∣ V = g F|_{\mathcal{V}} = g F ∣ V = g and ∥ F ∥ H ( K 0 ) = ∥ g ∥ H ( K 0 ∣ V ) \|F\|_{\mathcal{H}(K_0)} = \|g\|_{\mathcal{H}(K_{0\mid\mathcal{V}})} ∥ F ∥ H ( K 0 ) = ∥ g ∥ H ( K 0 ∣ V ) . By Corollary G.9, it holds that H ( K 0 ) ⊊ H p + 1 2 ( S p ) \mathcal{H}(K_0) \subsetneq H^{\frac{p+1}{2}}(\mathbb{S}^p) H ( K 0 ) ⊊ H 2 p + 1 ( S p ) and there is a constant C 1 > 0 C_1 > 0 C 1 > 0 , depending only on p p p , with ∥ F ∥ H p + 1 2 ( S p ) ≤ C 1 ∥ F ∥ H ( K 0 ) \|F\|_{H^{\frac{p+1}{2}}(\mathbb{S}^p)} \le C_1 \|F\|_{\mathcal{H}(K_0)} ∥ F ∥ H 2 p + 1 ( S p ) ≤ C 1 ∥ F ∥ H ( K 0 ) . Since F F F is in particular one extension of g g g to S p \mathbb{S}^p S p , the definition of the restriction norm on V \mathcal{V} V as an infimum over all such extensions gives ∥ g ∥ H p + 1 2 ( V ) ≤ ∥ F ∥ H p + 1 2 ( S p ) \|g\|_{H^{\frac{p+1}{2}}(\mathcal{V})} \le\|F\|_{H^{\frac{p+1}{2}}(\mathbb{S}^p)} ∥ g ∥ H 2 p + 1 ( V ) ≤ ∥ F ∥ H 2 p + 1 ( S p ) . Chaining the three displays, we get:
∥ g ∥ H p + 1 2 ( V ) ≤ ∥ F ∥ H p + 1 2 ( S p ) ≤ C 1 ∥ F ∥ H ( K 0 ) = C 1 ∥ g ∥ H ( K 0 ∣ V ) < ∞ , \|g\|_{H^{\frac{p+1}{2}}(\mathcal{V})} \le\|F\|_{H^{\frac{p+1}{2}}(\mathbb{S}^p)} \le C_1 \|F\|_{\mathcal{H}(K_0)} = C_1\|g\|_{\mathcal{H}(K_{0\mid\mathcal{V}})} < \infty, ∥ g ∥ H 2 p + 1 ( V ) ≤ ∥ F ∥ H 2 p + 1 ( S p ) ≤ C 1 ∥ F ∥ H ( K 0 ) = C 1 ∥ g ∥ H ( K 0 ∣ V ) < ∞ ,
so g ∈ H p + 1 2 ( V ) g \in H^{\frac{p+1}{2}}(\mathcal{V}) g ∈ H 2 p + 1 ( V ) with ∥ g ∥ H p + 1 2 ( V ) ≤ C 1 ∥ g ∥ H ( K 0 ∣ V ) \|g\|_{H^{\frac{p+1}{2}}(\mathcal{V})} \le C_1\|g\|_{\mathcal{H}(K_{0\mid\mathcal{V}})} ∥ g ∥ H 2 p + 1 ( V ) ≤ C 1 ∥ g ∥ H ( K 0 ∣ V ) .
(⊇ \supseteq ⊇ ). Let g ∈ H p + 1 2 ( V ) g \in H^{\frac{p+1}{2}}(\mathcal{V}) g ∈ H 2 p + 1 ( V ) . Our goal is to produce a function on the whole sphere, in H ( K 0 ) \mathcal{H}(K_0) H ( K 0 ) , that restricts to g g g on V \mathcal{V} V .
Step 1: extend g g g to the sphere, then confine the extension to the upper hemisphere. Since V \mathcal{V} V is bounded Lipschitz, there is a bounded linear extension operator [85 Chapter VI, Theorem 5] , giving G ~ ∈ H p + 1 2 ( S p ) \widetilde{G} \in H^{\frac{p+1}{2}}(\mathbb{S}^p) G ∈ H 2 p + 1 ( S p ) with G ~ ∣ V = g \widetilde{G}|_{\mathcal{V}} = g G ∣ V = g and ∥ G ~ ∥ H p + 1 2 ( S p ) ≤ C 2 ∥ g ∥ H p + 1 2 ( V ) \|\widetilde{G}\|_{H^{\frac{p+1}{2}}(\mathbb{S}^p)} \le C_2\|g\|_{H^{\frac{p+1}{2}}(\mathcal{V})} ∥ G ∥ H 2 p + 1 ( S p ) ≤ C 2 ∥ g ∥ H 2 p + 1 ( V ) for a constant C 2 > 0 C_2 > 0 C 2 > 0 depending only on p p p and the Lipschitz character of V \mathcal{V} V . Let χ ∈ C ∞ ( S p ) \chi\in C^\infty(\mathbb{S}^p) χ ∈ C ∞ ( S p ) be the smooth bump function from Lemma G.7, equal to 1 on V \mathcal{V} V and supported in S + p \mathbb{S}^p_+ S + p , and set G : = χ G ~ G := \chi\widetilde{G} G := χ G . By the multiplier property of Sobolev spaces [88 §4.2] , there is a constant C 3 > 0 C_3 > 0 C 3 > 0 , depending only on χ \chi χ , such that:
∥ G ∥ H p + 1 2 ( S p ) ≤ C 3 ∥ G ~ ∥ H p + 1 2 ( S p ) ≤ C 2 C 3 ∥ g ∥ H p + 1 2 ( V ) . \|G\|_{H^{\frac{p+1}{2}}(\mathbb{S}^p)} \le C_3\|\widetilde{G}\|_{H^{\frac{p+1}{2}}(\mathbb{S}^p)} \le C_2C_3\|g\|_{H^{\frac{p+1}{2}}(\mathcal{V})}. ∥ G ∥ H 2 p + 1 ( S p ) ≤ C 3 ∥ G ∥ H 2 p + 1 ( S p ) ≤ C 2 C 3 ∥ g ∥ H 2 p + 1 ( V ) .
Moreover, supp G ⊂ supp χ ⊂ S + p \operatorname{supp} G \subset\operatorname{supp}\chi\subset\mathbb{S}^p_+ supp G ⊂ supp χ ⊂ S + p , and since χ ≡ 1 \chi\equiv1 χ ≡ 1 on V \mathcal{V} V , we have G ∣ V = χ ∣ V ⋅ G ~ ∣ V = G ~ ∣ V = g G|_{\mathcal{V}} = \chi|_{\mathcal{V}}\cdot\widetilde{G}|_{\mathcal{V}} = \widetilde{G}|_{\mathcal{V}} = g G ∣ V = χ ∣ V ⋅ G ∣ V = G ∣ V = g .
Step 2: symmetrize across the equator. Define F : = G + G ∘ a F := G + G \circ a F := G + G ∘ a . We verify three properties of F F F .
(i) F F F restricts to g g g on V \mathcal{V} V . For y ∈ V ⊂ S + p y \in\mathcal{V} \subset\mathbb{S}^p_+ y ∈ V ⊂ S + p , the antipodal point − y -y − y lies in the open lower hemisphere S − p : = { y : y p + 1 < 0 } \mathbb{S}^p_- := \{y : y_{p+1} < 0\} S − p := { y : y p + 1 < 0 } , which is disjoint from supp G ⊂ S + p \operatorname{supp}G \subset\mathbb{S}^p_+ supp G ⊂ S + p . Hence, G ( − y ) = 0 G(-y) = 0 G ( − y ) = 0 and F ( y ) = G ( y ) + G ( − y ) = G ( y ) = g ( y ) F(y) = G(y) + G(-y) = G(y) = g(y) F ( y ) = G ( y ) + G ( − y ) = G ( y ) = g ( y ) .
(ii) F ∈ H p + 1 2 ( S p ) F \in H^{\frac{p+1}{2}}(\mathbb{S}^p) F ∈ H 2 p + 1 ( S p ) , with ∥ F ∥ H p + 1 2 ( S p ) ≤ 2 ∥ G ∥ H p + 1 2 ( S p ) \|F\|_{H^{\frac{p+1}{2}}(\mathbb{S}^p)} \le2\|G\|_{H^{\frac{p+1}{2}}(\mathbb{S}^p)} ∥ F ∥ H 2 p + 1 ( S p ) ≤ 2∥ G ∥ H 2 p + 1 ( S p ) . By Lemma G.10, ∥ G ∘ a ∥ H p + 1 2 ( S p ) = ∥ G ∥ H p + 1 2 ( S p ) \|G \circ a\|_{H^{\frac{p+1}{2}}(\mathbb{S}^p)} = \|G\|_{H^{\frac{p+1}{2}}(\mathbb{S}^p)} ∥ G ∘ a ∥ H 2 p + 1 ( S p ) = ∥ G ∥ H 2 p + 1 ( S p ) . The triangle inequality gives the bound on F = G + G ∘ a F = G + G \circ a F = G + G ∘ a .
(iii) F F F has no spherical-harmonic component of odd degree. By Lemma G.10, F k = G k + ( G ∘ a ) k = G k + ( − 1 ) k G k = ( 1 + ( − 1 ) k ) G k F_k = G_k + (G \circ a)_k = G_k + (-1)^kG_k = (1 + (-1)^k)G_k F k = G k + ( G ∘ a ) k = G k + ( − 1 ) k G k = ( 1 + ( − 1 ) k ) G k . For odd k k k , 1 + ( − 1 ) k = 0 1 + (-1)^k = 0 1 + ( − 1 ) k = 0 , so F k = 0 F_k = 0 F k = 0 .
By (ii) and (iii), F ∈ E ∩ H p + 1 2 ( S p ) F \in\mathcal{E} \cap H^{\frac{p+1}{2}}(\mathbb{S}^p) F ∈ E ∩ H 2 p + 1 ( S p ) . By Corollary G.9, F ∈ H ( K 0 ) F \in\mathcal{H}(K_0) F ∈ H ( K 0 ) with ∥ F ∥ H ( K 0 ) ≤ C ∥ F ∥ H p + 1 2 ( S p ) ≤ 2 C C 2 C 3 ∥ g ∥ H p + 1 2 ( V ) \|F\|_{\mathcal{H}(K_0)} \le C\|F\|_{H^{\frac{p+1}{2}}(\mathbb{S}^p)} \le2CC_2C_3\|g\|_{H^{\frac{p+1}{2}}(\mathcal{V})} ∥ F ∥ H ( K 0 ) ≤ C ∥ F ∥ H 2 p + 1 ( S p ) ≤ 2 C C 2 C 3 ∥ g ∥ H 2 p + 1 ( V ) , where C > 0 C > 0 C > 0 depends only on p p p . By (i), F F F is an admissible extension of g g g , so by the restriction property of RKHSs, g ∈ H ( K 0 ∣ V ) g \in\mathcal{H}(K_{0\mid\mathcal{V}}) g ∈ H ( K 0 ∣ V ) with ∥ g ∥ H ( K 0 ∣ V ) ≤ ∥ F ∥ H ( K 0 ) ≤ 2 C C 2 C 3 ∥ g ∥ H p + 1 2 ( V ) \|g\|_{\mathcal{H}(K_{0\mid\mathcal{V}})} \le\|F\|_{\mathcal{H}(K_0)} \le2CC_2C_3\|g\|_{H^{\frac{p+1}{2}}(\mathcal{V})} ∥ g ∥ H ( K 0 ∣ V ) ≤ ∥ F ∥ H ( K 0 ) ≤ 2 C C 2 C 3 ∥ g ∥ H 2 p + 1 ( V ) . □ \square □
The RKHS H N T K \mathcal{H}^{\mathrm{NTK}} H NTK is a Sobolev space
Theorem G.12 (The RKHS of K N T K K^{\mathrm{NTK}} K NTK is a Sobolev space). Let p ≥ 1 p \ge1 p ≥ 1 and let X ⊂ R p \mathcal{X} \subset\mathbb{R}^p X ⊂ R p be a nonempty bounded Lipschitz domain, with ∥ x ∥ ≤ A \|x\| \le A ∥ x ∥ ≤ A for any x ∈ X x \in\mathcal{X} x ∈ X . It holds that H N T K ≅ H p + 1 2 ( X ) \mathcal{H}^{\mathrm{NTK}} \cong H^{\frac{p+1}{2}}(\mathcal{X}) H NTK ≅ H 2 p + 1 ( X ) with equivalence constants depending only on p p p , A A A , and the Lipschitz character of X \mathcal{X} X .
Proof. By Corollary G.8, we have H N T K = q ⋅ [ H ( K 0 ∣ X ^ ) ∘ ϕ ] + R ⋅ 1 X \mathcal{H}^{\mathrm{NTK}} = q \cdot[\mathcal{H}(K_{0\mid\widehat{\mathcal{X}}}) \circ\phi] + \mathbb{R}\cdot\mathbf{1}_{\mathcal{X}} H NTK = q ⋅ [ H ( K 0 ∣ X ) ∘ ϕ ] + R ⋅ 1 X .
Step 1: identify the spherical factor. We check the three hypotheses of Lemma G.11 for X ^ = ϕ ( X ) \widehat{\mathcal{X}} = \phi(\mathcal{X}) X = ϕ ( X ) .
X ^ \widehat{\mathcal{X}} X is open. By Lemma G.7, ϕ \phi ϕ is a diffeomorphism, hence an open map. Since X \mathcal{X} X is open, so is X ^ \widehat{\mathcal{X}} X .
X ^ \widehat{\mathcal{X}} X is bounded with Lipschitz boundary. Boundedness is immediate, since S p \mathbb{S}^p S p is compact and X ^ ⊂ S p \widehat{\mathcal{X}} \subset\mathbb{S}^p X ⊂ S p . For the boundary, let A \mathcal{A} A be the maximal real-analytic atlas of S p \mathbb{S}^p S p (for its standard structure as a submanifold of R p + 1 \mathbb{R}^{p+1} R p + 1 ), and declare an open Ω ^ ⊂ S p \widehat{\Omega} \subset\mathbb{S}^p Ω ⊂ S p to have Lipschitz boundary if there is a chart ( ψ , U ) ∈ A (\psi,U) \in\mathcal{A} ( ψ , U ) ∈ A with Ω ^ ⊂ U \widehat{\Omega} \subset U Ω ⊂ U such that ψ ( Ω ^ ) \psi(\widehat{\Omega}) ψ ( Ω ) is a bounded Lipschitz domain in R p \mathbb{R}^p R p . This does not on the chart. Indeed, let ( ψ 1 , U 1 ) , ( ψ 2 , U 2 ) ∈ A (\psi_1,U_1),(\psi_2,U_2)\in\mathcal{A} ( ψ 1 , U 1 ) , ( ψ 2 , U 2 ) ∈ A both satisfy Ω ^ ⊂ U 1 \widehat{\Omega}\subset U_1 Ω ⊂ U 1 and Ω ^ ⊂ U 2 \widehat{\Omega}\subset U_2 Ω ⊂ U 2 . Since S p S^p S p is compact, Ω ^ \widehat{\Omega} Ω is compact. Given that ψ 1 : U 1 → R p \psi_1:U_1\to\mathbb{R}^p ψ 1 : U 1 → R p and ψ 2 : U 2 → R p \psi_2:U_2\to\mathbb{R}^p ψ 2 : U 2 → R p are homeomorphisms, we have ψ 1 ( Ω ^ ) = ψ 1 ( Ω ) ‾ \psi_1(\widehat{\Omega})=\overline{\psi_1(\Omega)} ψ 1 ( Ω ) = ψ 1 ( Ω ) and ψ 2 ( Ω ^ ) = ψ 2 ( Ω ) ‾ \psi_2(\widehat{\Omega})=\overline{\psi_2(\Omega)} ψ 2 ( Ω ) = ψ 2 ( Ω ) . Since Ω ^ ⊂ U 1 ∩ U 2 \widehat{\Omega}\subset U_1\cap U_2 Ω ⊂ U 1 ∩ U 2 , we get ψ 1 ( Ω ^ ) ⊂ ψ 1 ( U 1 ∩ U 2 ) \psi_1(\widehat{\Omega})\subset\psi_1(U_1\cap U_2) ψ 1 ( Ω ) ⊂ ψ 1 ( U 1 ∩ U 2 ) and ψ 2 ( Ω ^ ) ⊂ ψ 2 ( U 1 ∩ U 2 ) \psi_2(\widehat{\Omega})\subset\psi_2(U_1\cap U_2) ψ 2 ( Ω ) ⊂ ψ 2 ( U 1 ∩ U 2 ) . The transition map τ 0 = ψ 2 ∘ ψ 1 − 1 : ψ 1 ( U 1 ∩ U 2 ) → ψ 2 ( U 1 ∩ U 2 ) \tau_0=\psi_2\circ\psi_1^{-1}:\psi_1(U_1\cap U_2)\to\psi_2(U_1\cap U_2) τ 0 = ψ 2 ∘ ψ 1 − 1 : ψ 1 ( U 1 ∩ U 2 ) → ψ 2 ( U 1 ∩ U 2 ) is a real-analytic C 1 C^1 C 1 diffeomorphism, with inverse τ 0 − 1 = ψ 1 ∘ ψ 2 − 1 \tau_0^{-1}=\psi_1\circ\psi_2^{-1} τ 0 − 1 = ψ 1 ∘ ψ 2 − 1 . By the two displayed inclusions, τ 0 \tau_0 τ 0 is defined on a neighborhood of ψ 1 ( Ω ^ ) \psi_1(\widehat{\Omega}) ψ 1 ( Ω ) and τ 0 − 1 \tau_0^{-1} τ 0 − 1 is defined on a neighborhood of ψ 2 ( Ω ^ ) \psi_2(\widehat{\Omega}) ψ 2 ( Ω ) , so [50 Theorem 4.1] applies to each: it shows ψ 1 ( Ω ^ ) \psi_1(\widehat{\Omega}) ψ 1 ( Ω ) is a bounded Lipschitz domain if and only if τ 0 ( ψ 1 ( Ω ^ ) ) = ψ 2 ( Ω ^ ) \tau_0(\psi_1(\widehat{\Omega}))=\psi_2(\widehat{\Omega}) τ 0 ( ψ 1 ( Ω )) = ψ 2 ( Ω ) is, which proves the chart-independence property.
We now specialize to ϕ − 1 \phi^{-1} ϕ − 1 . The map ϕ − 1 : S + p → R p \phi^{-1}:S^p_+\to\mathbb{R}^p ϕ − 1 : S + p → R p , ϕ − 1 ( y ) = ( y 1 , … , y p ) / y p + 1 \phi^{-1}(y)=(y_1,\ldots,y_p)/y_{p+1} ϕ − 1 ( y ) = ( y 1 , … , y p ) / y p + 1 , is a real-analytic diffeomorphism with real-analytic inverse ϕ ( x ) = x ~ / 1 + ∥ x ∥ 2 \phi(x)=\widetilde{x}/\sqrt{1+\lVert x\rVert^2} ϕ ( x ) = x / 1 + ∥ x ∥ 2 , so ( ϕ − 1 , S + p ) ∈ A (\phi^{-1},S^p_+)\in\mathcal{A} ( ϕ − 1 , S + p ) ∈ A . Since X X X is bounded, X ‾ \overline{X} X is compact. The fact that ϕ : R p → S + p \phi:\mathbb{R}^p\to S^p_+ ϕ : R p → S + p is a homeomorphism gives X ^ = ϕ ( X ) = ϕ ( X ‾ ) \widehat{X}=\phi(X)=\phi(\overline{X}) X = ϕ ( X ) = ϕ ( X ) . Boundedness of X ‾ \overline{X} X also keeps ϕ ( x ) p + 1 = 1 / 1 + ∥ x ∥ 2 \phi(x)_{p+1}=1/\sqrt{1+\lVert x\rVert^2} ϕ ( x ) p + 1 = 1/ 1 + ∥ x ∥ 2 bounded away from 0 on X ‾ \overline{X} X , so X ^ = ϕ ( X ‾ ) ⊂ S + p \widehat{X}=\phi(\overline{X})\subset S^p_+ X = ϕ ( X ) ⊂ S + p . Thus ( ϕ − 1 , S + p ) (\phi^{-1},S^p_+) ( ϕ − 1 , S + p ) is a chart with X ^ ⊂ S + p \widehat{X}\subset S^p_+ X ⊂ S + p , and ϕ − 1 ( X ^ ) = X \phi^{-1}(\widehat{X})=X ϕ − 1 ( X ) = X is a bounded Lipschitz domain by hypothesis. Hence, X ^ \widehat{X} X has Lipschitz boundary.
X ^ \widehat{X} X is compactly contained in S + p S^p_+ S + p . By Lemma G.7, X ^ \widehat{X} X lies in the closed cap { y ∈ S p : y p + 1 ≥ ( 1 + A 2 ) − 1 / 2 } \{y\in S^p:y_{p+1}\ge(1+A^2)^{-1/2}\} { y ∈ S p : y p + 1 ≥ ( 1 + A 2 ) − 1/2 } , which is compact and contained in S + p S^p_+ S + p . Hence, X ^ \widehat{X} X is compactly contained in S + p S^p_+ S + p .
All three hypotheses hold, so Lemma G.11 applies and gives H ( K 0 ∣ X ^ ) ≅ H p + 1 2 ( X ^ ) \mathcal{H}(K_0|_{\widehat{X}})\cong H^{\frac{p+1}{2}}(\widehat{X}) H ( K 0 ∣ X ) ≅ H 2 p + 1 ( X ) .
Step 2: transport back to X X X . We prove in this step that q ⋅ H ( K 0 ∣ X ^ ) ∘ ϕ − 1 ≅ H p + 1 2 ( X ) q\cdot\mathcal{H}(K_0|_{\widehat{X}})\circ\phi^{-1}\cong H^{\frac{p+1}{2}}(X) q ⋅ H ( K 0 ∣ X ) ∘ ϕ − 1 ≅ H 2 p + 1 ( X ) .
Undoing ϕ \phi ϕ . Let A \mathcal{A} A be the maximal real-analytic atlas of S p S^p S p , fixed in Step 1. For any two charts ( ψ 1 , U 1 ) , ( ψ 2 , U 2 ) ∈ A (\psi_1,U_1),(\psi_2,U_2)\in\mathcal{A} ( ψ 1 , U 1 ) , ( ψ 2 , U 2 ) ∈ A , the transition map τ 0 : = ψ 2 ∘ ψ 1 − 1 \tau_0:=\psi_2\circ\psi_1^{-1} τ 0 := ψ 2 ∘ ψ 1 − 1 is a bijection from ψ 1 ( U 1 ∩ U 2 ) \psi_1(U_1\cap U_2) ψ 1 ( U 1 ∩ U 2 ) onto ψ 2 ( U 1 ∩ U 2 ) \psi_2(U_1\cap U_2) ψ 2 ( U 1 ∩ U 2 ) . Therefore, τ 0 \tau_0 τ 0 and its inverse τ 0 − 1 = ψ 1 ∘ ψ 2 − 1 \tau_0^{-1}=\psi_1\circ\psi_2^{-1} τ 0 − 1 = ψ 1 ∘ ψ 2 − 1 are real-analytic, since ψ 1 , ψ 2 \psi_1,\psi_2 ψ 1 , ψ 2 are. Thus, τ 0 \tau_0 τ 0 is a real-analytic diffeomorphism between these two open subsets of R p \mathbb{R}^p R p , and in particular, being real-analytic, τ 0 \tau_0 τ 0 is C ∞ C^\infty C ∞ , matching the hypothesis of [15 Theorem 80] . For a bounded open Ω ^ ⊂ S p \widehat{\Omega}\subset S^p Ω ⊂ S p with Ω ^ ⊂ U \widehat{\Omega}\subset U Ω ⊂ U for some chart ( ψ , U ) ∈ A (\psi,U)\in\mathcal{A} ( ψ , U ) ∈ A , define ∥ g ∥ H p + 1 2 ( Ω ^ ) , ψ : = ∥ g ∘ ψ − 1 ∥ H p + 1 2 ( ψ ( Ω ^ ) ) \lVert g\rVert_{H^{\frac{p+1}{2}}(\widehat{\Omega}),\psi}:=\lVert g\circ\psi^{-1}\rVert_{H^{\frac{p+1}{2}}(\psi(\widehat{\Omega}))} ∥ g ∥ H 2 p + 1 ( Ω ) , ψ := ∥ g ∘ ψ − 1 ∥ H 2 p + 1 ( ψ ( Ω )) . We claim there is a constant C 1 > 0 C_1>0 C 1 > 0 , depending only on p p p and the pair of charts ( ψ 1 , U 1 ) , ( ψ 2 , U 2 ) (\psi_1,U_1),(\psi_2,U_2) ( ψ 1 , U 1 ) , ( ψ 2 , U 2 ) , such that:
1 C 1 ∥ g ∥ H p + 1 2 ( Ω ^ ) , ψ 1 ≤ ∥ g ∥ H p + 1 2 ( Ω ^ ) , ψ 2 ≤ C 1 ∥ g ∥ H p + 1 2 ( Ω ^ ) , ψ 1 , (73) \frac{1}{C_1}\lVert g\rVert_{H^{\frac{p+1}{2}}(\widehat{\Omega}),\psi_1}\le\lVert g\rVert_{H^{\frac{p+1}{2}}(\widehat{\Omega}),\psi_2}\le C_1\lVert g\rVert_{H^{\frac{p+1}{2}}(\widehat{\Omega}),\psi_1}, \tag*{(73)} C 1 1 ∥ g ∥ H 2 p + 1 ( Ω ) , ψ 1 ≤ ∥ g ∥ H 2 p + 1 ( Ω ) , ψ 2 ≤ C 1 ∥ g ∥ H 2 p + 1 ( Ω ) , ψ 1 , (73)
whenever ( ψ 1 , U 1 ) , ( ψ 2 , U 2 ) ∈ A (\psi_1,U_1),(\psi_2,U_2)\in\mathcal{A} ( ψ 1 , U 1 ) , ( ψ 2 , U 2 ) ∈ A both contain Ω ^ \widehat{\Omega} Ω . Indeed, since Ω ^ ⊂ U 1 ∩ U 2 \widehat{\Omega}\subset U_1\cap U_2 Ω ⊂ U 1 ∩ U 2 , and a diffeomorphism restricted to an open subset of its domain remains a diffeomorphism onto the corresponding open subset of its codomain, τ 0 \tau_0 τ 0 restricts to a real-analytic diffeomorphism τ 0 ∣ ψ 1 ( Ω ^ ) : ψ 1 ( Ω ^ ) → ψ 2 ( Ω ^ ) \tau_0|_{\psi_1(\widehat{\Omega})}:\psi_1(\widehat{\Omega})\to\psi_2(\widehat{\Omega}) τ 0 ∣ ψ 1 ( Ω ) : ψ 1 ( Ω ) → ψ 2 ( Ω ) , with real-analytic inverse τ 0 − 1 ∣ ψ 2 ( Ω ^ ) : ψ 2 ( Ω ^ ) → ψ 1 ( Ω ^ ) \tau_0^{-1}|_{\psi_2(\widehat{\Omega})}:\psi_2(\widehat{\Omega})\to\psi_1(\widehat{\Omega}) τ 0 − 1 ∣ ψ 2 ( Ω ) : ψ 2 ( Ω ) → ψ 1 ( Ω ) . We retain the notation τ 0 , τ 0 − 1 \tau_0,\tau_0^{-1} τ 0 , τ 0 − 1 for these restrictions. We have g ∘ ψ 2 − 1 = ( g ∘ ψ 1 − 1 ) ∘ τ 0 − 1 g\circ\psi_2^{-1}=(g\circ\psi_1^{-1})\circ\tau_0^{-1} g ∘ ψ 2 − 1 = ( g ∘ ψ 1 − 1 ) ∘ τ 0 − 1 , and [15 Theorem 80] applied to τ 0 − 1 \tau_0^{-1} τ 0 − 1 between ψ 2 ( Ω ^ ) \psi_2(\widehat{\Omega}) ψ 2 ( Ω ) and ψ 1 ( Ω ^ ) \psi_1(\widehat{\Omega}) ψ 1 ( Ω ) gives a constant C 1 ′ > 0 C_1'>0 C 1 ′ > 0 with:
∥ ( g ∘ ψ 1 − 1 ) ∘ τ 0 − 1 ∥ H p + 1 2 ( ψ 2 ( Ω ^ ) ) ≤ C 1 ′ ∥ g ∘ ψ 1 − 1 ∥ H p + 1 2 ( ψ 1 ( Ω ^ ) ) , \lVert(g\circ\psi_1^{-1})\circ\tau_0^{-1}\rVert_{H^{\frac{p+1}{2}}(\psi_2(\widehat{\Omega}))}\le C_1'\lVert g\circ\psi_1^{-1}\rVert_{H^{\frac{p+1}{2}}(\psi_1(\widehat{\Omega}))}, ∥( g ∘ ψ 1 − 1 ) ∘ τ 0 − 1 ∥ H 2 p + 1 ( ψ 2 ( Ω )) ≤ C 1 ′ ∥ g ∘ ψ 1 − 1 ∥ H 2 p + 1 ( ψ 1 ( Ω )) ,
i.e., ∥ g ∥ H p + 1 2 ( Ω ^ ) , ψ 2 ≤ C 1 ′ ∥ g ∥ H p + 1 2 ( Ω ^ ) , ψ 1 \lVert g\rVert_{H^{\frac{p+1}{2}}(\widehat{\Omega}),\psi_2}\le C_1'\lVert g\rVert_{H^{\frac{p+1}{2}}(\widehat{\Omega}),\psi_1} ∥ g ∥ H 2 p + 1 ( Ω ) , ψ 2 ≤ C 1 ′ ∥ g ∥ H 2 p + 1 ( Ω ) , ψ 1 . Conversely, g ∘ ψ 1 − 1 = ( g ∘ ψ 2 − 1 ) ∘ τ 0 g\circ\psi_1^{-1}=(g\circ\psi_2^{-1})\circ\tau_0 g ∘ ψ 1 − 1 = ( g ∘ ψ 2 − 1 ) ∘ τ 0 , and applying [15 Theorem 80] directly to τ 0 \tau_0 τ 0 between ψ 1 ( Ω ^ ) \psi_1(\widehat{\Omega}) ψ 1 ( Ω ) and ψ 2 ( Ω ^ ) \psi_2(\widehat{\Omega}) ψ 2 ( Ω ) gives a constant C 1 ′ ′ > 0 C_1''>0 C 1 ′′ > 0 with:
∥ ( g ∘ ψ 2 − 1 ) ∘ τ 0 ∥ H p + 1 2 ( ψ 1 ( Ω ^ ) ) ≤ C 1 ′ ′ ∥ g ∘ ψ 2 − 1 ∥ H p + 1 2 ( ψ 2 ( Ω ^ ) ) , \lVert(g\circ\psi_2^{-1})\circ\tau_0\rVert_{H^{\frac{p+1}{2}}(\psi_1(\widehat{\Omega}))}\le C_1''\lVert g\circ\psi_2^{-1}\rVert_{H^{\frac{p+1}{2}}(\psi_2(\widehat{\Omega}))}, ∥( g ∘ ψ 2 − 1 ) ∘ τ 0 ∥ H 2 p + 1 ( ψ 1 ( Ω )) ≤ C 1 ′′ ∥ g ∘ ψ 2 − 1 ∥ H 2 p + 1 ( ψ 2 ( Ω )) ,
i.e., ∥ g ∥ H p + 1 2 ( Ω ^ ) , ψ 1 ≤ C 1 ′ ′ ∥ g ∥ H p + 1 2 ( Ω ^ ) , ψ 2 \lVert g\rVert_{H^{\frac{p+1}{2}}(\widehat{\Omega}),\psi_1}\le C_1''\lVert g\rVert_{H^{\frac{p+1}{2}}(\widehat{\Omega}),\psi_2} ∥ g ∥ H 2 p + 1 ( Ω ) , ψ 1 ≤ C 1 ′′ ∥ g ∥ H 2 p + 1 ( Ω ) , ψ 2 . Taking C 1 : = max ( C 1 ′ , C 1 ′ ′ ) C_1:=\max(C_1',C_1'') C 1 := max ( C 1 ′ , C 1 ′′ ) gives Equation (72).
Specialization to ϕ − 1 \phi^{-1} ϕ − 1 . The map ϕ − 1 : S + p → R p \phi^{-1}:S^p_+\to\mathbb{R}^p ϕ − 1 : S + p → R p , ϕ − 1 ( y ) = ( y 1 , … , y p ) / y p + 1 \phi^{-1}(y)=(y_1,\ldots,y_p)/y_{p+1} ϕ − 1 ( y ) = ( y 1 , … , y p ) / y p + 1 , is a real-analytic diffeomorphism with real-analytic inverse ϕ ( x ) = x ~ / 1 + ∥ x ∥ 2 \phi(x)=\widetilde{x}/\sqrt{1+\lVert x\rVert^2} ϕ ( x ) = x / 1 + ∥ x ∥ 2 , so ( ϕ − 1 , S + p ) ∈ A (\phi^{-1},S^p_+)\in\mathcal{A} ( ϕ − 1 , S + p ) ∈ A by maximality of A \mathcal{A} A . As shown above, X ^ = ϕ ( X ‾ ) ⊂ S + p \widehat{X}=\phi(\overline{X})\subset S^p_+ X = ϕ ( X ) ⊂ S + p , so ( ϕ − 1 , S + p ) (\phi^{-1},S^p_+) ( ϕ − 1 , S + p ) is a chart of A \mathcal{A} A containing X ^ \widehat{X} X in its domain, exactly as required by (73) . Fix any anchor chart ( ψ 0 , U 0 ) ∈ A (\psi_0,U_0)\in\mathcal{A} ( ψ 0 , U 0 ) ∈ A with X ^ ⊂ U 0 \widehat{X}\subset U_0 X ⊂ U 0 , and set ∥ g ∥ H p + 1 2 ( X ^ ) : = ∥ g ∥ H p + 1 2 ( X ^ ) , ψ 0 \lVert g\rVert_{H^{\frac{p+1}{2}}(\widehat{X})}:=\lVert g\rVert_{H^{\frac{p+1}{2}}(\widehat{X}),\psi_0} ∥ g ∥ H 2 p + 1 ( X ) := ∥ g ∥ H 2 p + 1 ( X ) , ψ 0 , which by Step 1 is independent of this choice of chart. Applying (73) directly with Ω ^ : = X ^ \widehat{\Omega}:=\widehat{X} Ω := X , ψ 1 : = ψ 0 \psi_1:=\psi_0 ψ 1 := ψ 0 , ψ 2 : = ϕ − 1 \psi_2:=\phi^{-1} ψ 2 := ϕ − 1 gives a constant C 2 > 0 C_2>0 C 2 > 0 with:
1 C 2 ∥ g ∥ H p + 1 2 ( X ^ ) ≤ ∥ g ∥ H p + 1 2 ( X ^ ) , ϕ − 1 ≤ C 2 ∥ g ∥ H p + 1 2 ( X ^ ) . (74) \frac{1}{C_2}\lVert g\rVert_{H^{\frac{p+1}{2}}(\widehat{X})}\leq\lVert g\rVert_{H^{\frac{p+1}{2}}(\widehat{X}),\phi^{-1}}\leq C_2\lVert g\rVert_{H^{\frac{p+1}{2}}(\widehat{X})}. \tag*{(74)} C 2 1 ∥ g ∥ H 2 p + 1 ( X ) ≤ ∥ g ∥ H 2 p + 1 ( X ) , ϕ − 1 ≤ C 2 ∥ g ∥ H 2 p + 1 ( X ) . (74)
Since ∥ g ∥ H p + 1 2 ( X ^ ) , ϕ − 1 = ∥ g ∘ ϕ ∥ H p + 1 2 ( X ) \lVert g\rVert_{H^{\frac{p+1}{2}}(\widehat{X}),\phi^{-1}}=\lVert g\circ\phi\rVert_{H^{\frac{p+1}{2}}(\mathcal{X})} ∥ g ∥ H 2 p + 1 ( X ) , ϕ − 1 = ∥ g ∘ ϕ ∥ H 2 p + 1 ( X ) , Equation (74) gives ∥ g ∥ H p + 1 2 ( X ^ ) ≍ ∥ g ∘ ϕ ∥ H p + 1 2 ( X ) \lVert g\rVert_{H^{\frac{p+1}{2}}(\widehat{X})}\asymp\lVert g\circ\phi\rVert_{H^{\frac{p+1}{2}}(\mathcal{X})} ∥ g ∥ H 2 p + 1 ( X ) ≍ ∥ g ∘ ϕ ∥ H 2 p + 1 ( X ) , i.e., H p + 1 2 ( X ^ ) ≅ H p + 1 2 ( X ) H^{\frac{p+1}{2}}(\widehat{X})\cong H^{\frac{p+1}{2}}(\mathcal{X}) H 2 p + 1 ( X ) ≅ H 2 p + 1 ( X ) via the pullback g ↦ g ∘ ϕ g\mapsto g\circ\phi g ↦ g ∘ ϕ .
Undoing q q q . The function q ( x ) = ∥ x ~ ∥ q(x)=\lVert\widetilde{x}\rVert q ( x ) = ∥ x ∥ is real-analytic with 1 ≤ q ≤ 1 + A 2 1\leq q\leq\sqrt{1+A^2} 1 ≤ q ≤ 1 + A 2 on X \mathcal{X} X . Since X \mathcal{X} X is a bounded Lipschitz domain, the multiplication theorem for Sobolev spaces on Lipschitz domains [15 Corollary 3] shows H p + 1 2 ( X ) H^{\frac{p+1}{2}}(\mathcal{X}) H 2 p + 1 ( X ) is a Banach algebra under pointwise multiplication. Applying this to q q q and to 1 / q 1/q 1/ q (likewise real-analytic and bounded on X \mathcal{X} X , since q ≥ 1 q\geq1 q ≥ 1 ), there is a constant C 3 > 0 C_3>0 C 3 > 0 , depending only on A A A and p p p , such that multiplication by q q q is an isomorphism of H p + 1 2 ( X ) H^{\frac{p+1}{2}}(\mathcal{X}) H 2 p + 1 ( X ) , with:
1 C 3 ∥ f ∥ H p + 1 2 ( X ) ≤ ∥ q f ∥ H p + 1 2 ( X ) ≤ C 3 ∥ f ∥ H p + 1 2 ( X ) . \frac{1}{C_3}\lVert f\rVert_{H^{\frac{p+1}{2}}(\mathcal{X})}\leq\lVert qf\rVert_{H^{\frac{p+1}{2}}(\mathcal{X})}\leq C_3\lVert f\rVert_{H^{\frac{p+1}{2}}(\mathcal{X})}. C 3 1 ∥ f ∥ H 2 p + 1 ( X ) ≤ ∥ q f ∥ H 2 p + 1 ( X ) ≤ C 3 ∥ f ∥ H 2 p + 1 ( X ) .
Composing. Combining the two isomorphisms above with the one from Step 1 (H ( K 0 ∣ X ^ ) ≅ H p + 1 2 ( X ^ ) \mathcal{H}(K_0|_{\widehat{X}})\cong H^{\frac{p+1}{2}}(\widehat{X}) H ( K 0 ∣ X ) ≅ H 2 p + 1 ( X ) ), we get q ⋅ [ H ( K 0 ∣ X ^ ) ∘ ϕ ] ≅ H p + 1 2 ( X ) q\cdot[\mathcal{H}(K_0|_{\widehat{X}})\circ\phi]\cong H^{\frac{p+1}{2}}(\mathcal{X}) q ⋅ [ H ( K 0 ∣ X ) ∘ ϕ ] ≅ H 2 p + 1 ( X ) .
Step 3: absorb the outer bias. Recall from Corollary G.8 that H N T K = q ⋅ [ H ( K 0 ∣ X ^ ) ∘ ϕ ] + R ⋅ 1 X \mathcal{H}^{\mathrm{NTK}}=q\cdot[\mathcal{H}(K_0|_{\widehat{X}})\circ\phi]+\mathbb{R}\cdot1_{\mathcal{X}} H NTK = q ⋅ [ H ( K 0 ∣ X ) ∘ ϕ ] + R ⋅ 1 X . Write k 1 : = 1 X k_1:=1_{\mathcal{X}} k 1 := 1 X for the constant kernel and k 2 k_2 k 2 for the kernel underlying the first summand, so that H ( k 2 ) = q ⋅ [ H ( K 0 ∣ X ^ ) ∘ ϕ ] \mathcal{H}(k_2)=q\cdot[\mathcal{H}(K_0|_{\widehat{X}})\circ\phi] H ( k 2 ) = q ⋅ [ H ( K 0 ∣ X ) ∘ ϕ ] . By Step 2, H ( k 2 ) ≅ H p + 1 2 ( X ) \mathcal{H}(k_2)\cong H^{\frac{p+1}{2}}(\mathcal{X}) H ( k 2 ) ≅ H 2 p + 1 ( X ) . Since 1 X 1_{\mathcal{X}} 1 X has vanishing weak derivatives of every order, only the L 2 L^2 L 2 term of the Sobolev norm survives, and ∥ 1 X ∥ H p + 1 2 ( X ) = ∥ 1 X ∥ L 2 ( X ) = ∫ X 1 d x = Vol ( X ) < ∞ \lVert1_{\mathcal{X}}\rVert_{H^{\frac{p+1}{2}}(\mathcal{X})}=\lVert1_{\mathcal{X}}\rVert_{L^2(\mathcal{X})}=\int_{\mathcal{X}}1\,dx=\operatorname{Vol}(\mathcal{X})<\infty ∥ 1 X ∥ H 2 p + 1 ( X ) = ∥ 1 X ∥ L 2 ( X ) = ∫ X 1 d x = Vol ( X ) < ∞ , where Vol ( ⋅ ) \operatorname{Vol}(\cdot) Vol ( ⋅ ) denotes the volume. Finiteness follows from the boundedness of X \mathcal{X} X . Thus 1 X ∈ H p + 1 2 ( X ) 1_{\mathcal{X}}\in H^{\frac{p+1}{2}}(\mathcal{X}) 1 X ∈ H 2 p + 1 ( X ) . Since H ( k 1 ) = R ⋅ 1 X \mathcal{H}(k_1)=\mathbb{R}\cdot1_{\mathcal{X}} H ( k 1 ) = R ⋅ 1 X with ∥ c 1 X ∥ H ( k 1 ) = ∣ c ∣ \lVert c1_{\mathcal{X}}\rVert_{\mathcal{H}(k_1)}=|c| ∥ c 1 X ∥ H ( k 1 ) = ∣ c ∣ for c ∈ R c\in\mathbb{R} c ∈ R , the homogeneity of the H p + 1 2 ( X ) H^{\frac{p+1}{2}}(\mathcal{X}) H 2 p + 1 ( X ) norm and the previous display give ∥ c 1 X ∥ H p + 1 2 ( X ) = ∣ c ∣ Vol ( X ) = Vol ( X ) ∥ c 1 X ∥ H ( k 1 ) \lVert c1_{\mathcal{X}}\rVert_{H^{\frac{p+1}{2}}(\mathcal{X})}=|c|\sqrt{\operatorname{Vol}(\mathcal{X})}=\sqrt{\operatorname{Vol}(\mathcal{X})}\lVert c1_{\mathcal{X}}\rVert_{\mathcal{H}(k_1)} ∥ c 1 X ∥ H 2 p + 1 ( X ) = ∣ c ∣ Vol ( X ) = Vol ( X ) ∥ c 1 X ∥ H ( k 1 ) . Hence, H ( k 1 ) \mathcal{H}(k_1) H ( k 1 ) embeds continuously into H p + 1 2 ( X ) ≅ H ( k 2 ) H^{\frac{p+1}{2}}(\mathcal{X})\cong\mathcal{H}(k_2) H 2 p + 1 ( X ) ≅ H ( k 2 ) , with embedding constant Vol ( X ) \sqrt{\operatorname{Vol}(\mathcal{X})} Vol ( X ) . Applying Corollary G.5 with this k 1 k_1 k 1 and k 2 k_2 k 2 yields H N T K = H ( k 1 ) + H ( k 2 ) ≅ H ( k 2 ) ≅ H p + 1 2 ( X ) \mathcal{H}^{\mathrm{NTK}}=\mathcal{H}(k_1)+\mathcal{H}(k_2)\cong\mathcal{H}(k_2)\cong H^{\frac{p+1}{2}}(\mathcal{X}) H NTK = H ( k 1 ) + H ( k 2 ) ≅ H ( k 2 ) ≅ H 2 p + 1 ( X ) , which is the claimed identification.
The interpolation spaces of H N T K \mathcal{H}^{\mathrm{NTK}} H NTK are Sobolev spaces
For two Hilbert spaces E 0 , E 1 E_0,E_1 E 0 , E 1 continuously embedded in a common space, we denote the real interpolation space by ( E 0 , E 1 ) s , 2 (E_0,E_1)_{s,2} ( E 0 , E 1 ) s , 2 , s ∈ ( 0 , 1 ) s\in(0,1) s ∈ ( 0 , 1 ) , defined via the standard K K K -method [17 Chapter 3] .
Corollary G.13 (A Sobolev characterization of [ H N T K ] s [\mathcal{H}^{\mathrm{NTK}}]_s [ H NTK ] s ). Let p ≥ 1 p\geq1 p ≥ 1 and X ⊂ R p \mathcal{X}\subset\mathbb{R}^p X ⊂ R p be a nonempty bounded Lipschitz domain, with ∥ x ∥ ≤ A \lVert x\rVert\leq A ∥ x ∥ ≤ A for any x ∈ X x\in\mathcal{X} x ∈ X . Assume that μ \mu μ , the data distribution on X \mathcal{X} X , has a density f μ f_\mu f μ bounded away from 0 0 0 and ∞ \infty ∞ , i.e., there exist c μ , M μ > 0 c_\mu,M_\mu>0 c μ , M μ > 0 such that c μ ≤ f μ ≤ M μ c_\mu\leq f_\mu\leq M_\mu c μ ≤ f μ ≤ M μ . Then, for every s ∈ [ 0 , 1 ] s\in[0,1] s ∈ [ 0 , 1 ] , [ H N T K ] s ≅ H s ( p + 1 ) 2 ( X ) [\mathcal{H}^{\mathrm{NTK}}]_s\cong H^{\frac{s(p+1)}{2}}(\mathcal{X}) [ H NTK ] s ≅ H 2 s ( p + 1 ) ( X ) , with equivalence constants depending on p p p , s s s , A A A , the Lipschitz character of X \mathcal{X} X , and the density bounds c μ , M μ c_\mu,M_\mu c μ , M μ .
Proof. By Theorem G.12, we have H N T K ≅ H p + 1 2 ( X ) \mathcal{H}^{\mathrm{NTK}}\cong H^{\frac{p+1}{2}}(\mathcal{X}) H NTK ≅ H 2 p + 1 ( X ) . Since 0 < c μ ≤ f μ ≤ M μ 0<c_\mu\leq f_\mu\leq M_\mu 0 < c μ ≤ f μ ≤ M μ , the norms ∥ ⋅ ∥ L 2 ( X , d μ ) \lVert\cdot\rVert_{L^2(\mathcal{X},d\mu)} ∥ ⋅ ∥ L 2 ( X , d μ ) and ∥ ⋅ ∥ L 2 ( X ) \lVert\cdot\rVert_{L^2(\mathcal{X})} ∥ ⋅ ∥ L 2 ( X ) are equivalent, so L 2 ( X , d μ ) ≅ L 2 ( X ) L^2(\mathcal{X},d\mu)\cong L^2(\mathcal{X}) L 2 ( X , d μ ) ≅ L 2 ( X ) . We use this identification below.
Interior case s ∈ ( 0 , 1 ) s\in(0,1) s ∈ ( 0 , 1 ) . Let C c ∞ ( X ) C^\infty_c(\mathcal{X}) C c ∞ ( X ) be the space of smooth, compactly supported functions on X \mathcal{X} X . It is standard that C c ∞ ( X ) C^\infty_c(\mathcal{X}) C c ∞ ( X ) is dense in L 2 ( X , d x ) L^2(\mathcal{X},dx) L 2 ( X , d x ) . By the identification above, C c ∞ ( X ) C^\infty_c(\mathcal{X}) C c ∞ ( X ) is therefore dense in L 2 ( X ) L^2(\mathcal{X}) L 2 ( X ) . Since H p + 1 2 ( X ) ⊃ C c ∞ ( X ) H^{\frac{p+1}{2}}(\mathcal{X})\supset C^\infty_c(\mathcal{X}) H 2 p + 1 ( X ) ⊃ C c ∞ ( X ) , it follows that H p + 1 2 ( X ) H^{\frac{p+1}{2}}(\mathcal{X}) H 2 p + 1 ( X ) is dense in L 2 ( X ) L^2(\mathcal{X}) L 2 ( X ) .
The embedding H p + 1 2 ( X ) ↪ L 2 ( X ) H^{\frac{p+1}{2}}(\mathcal{X})\hookrightarrow L^2(\mathcal{X}) H 2 p + 1 ( X ) ↪ L 2 ( X ) is compact. Write κ : = ⌊ p + 1 2 ⌋ \kappa:=\left\lfloor\frac{p+1}{2}\right\rfloor κ := ⌊ 2 p + 1 ⌋ . If p p p is odd, p + 1 2 = κ \frac{p+1}{2}=\kappa 2 p + 1 = κ is already an integer, and the Rellich–Kondrachov theorem [2 Theorem. 6.3, Part II, Eq. (6)] gives directly that H p + 1 2 ( X ) = H κ ( X ) ↪ L 2 ( X , d x ) H^{\frac{p+1}{2}}(\mathcal{X})=H^\kappa(\mathcal{X})\hookrightarrow L^2(\mathcal{X},dx) H 2 p + 1 ( X ) = H κ ( X ) ↪ L 2 ( X , d x ) is compact. If p p p is even, κ = p 2 < p + 1 2 \kappa=\frac{p}{2}<\frac{p+1}{2} κ = 2 p < 2 p + 1 , and the Rellich–Kondrachov theorem [2 Theorem. 6.3, Part I, Eq. (4)] shows that H κ ( X ) ↪ L 2 ( X , d x ) H^\kappa(\mathcal{X})\hookrightarrow L^2(\mathcal{X},dx) H κ ( X ) ↪ L 2 ( X , d x ) is compact. By definition, H p + 1 2 ( X ) ↪ H κ ( X ) H^{\frac{p+1}{2}}(\mathcal{X}) \hookrightarrow H^{\kappa}(\mathcal{X}) H 2 p + 1 ( X ) ↪ H κ ( X ) is continuous, and composing the two gives H p + 1 2 ( X ) ↪ L 2 ( X , d x ) H^{\frac{p+1}{2}}(\mathcal{X}) \hookrightarrow L^2(\mathcal{X},\mathrm{d}x) H 2 p + 1 ( X ) ↪ L 2 ( X , d x ) compact. In either case, by the identification L 2 ( X , d x ) ≅ L 2 ( X ) L^2(\mathcal{X},\mathrm{d}x) \cong L^2(\mathcal{X}) L 2 ( X , d x ) ≅ L 2 ( X ) , H p + 1 2 ( X ) ↪ L 2 ( X ) H^{\frac{p+1}{2}}(\mathcal{X}) \hookrightarrow L^2(\mathcal{X}) H 2 p + 1 ( X ) ↪ L 2 ( X ) is compact.
By [87 Theorem 4.6] , [ H N T K ] s ≅ ( L 2 ( X ) , H N T K ) s , 2 [\mathcal{H}^{\mathrm{NTK}}]^s \cong(L^2(\mathcal{X}),\mathcal{H}^{\mathrm{NTK}})_{s,2} [ H NTK ] s ≅ ( L 2 ( X ) , H NTK ) s , 2 . Since H N T K ≅ H p + 1 2 ( X ) \mathcal{H}^{\mathrm{NTK}} \cong H^{\frac{p+1}{2}}(\mathcal{X}) H NTK ≅ H 2 p + 1 ( X ) , [24 Lemma H.5] gives ( L 2 ( X ) , H N T K ) s , 2 ≅ ( L 2 ( X ) , H p + 1 2 ( X ) ) s , 2 (L^2(\mathcal{X}),\mathcal{H}^{\mathrm{NTK}})_{s,2} \cong(L^2(\mathcal{X}),H^{\frac{p+1}{2}}(\mathcal{X}))_{s,2} ( L 2 ( X ) , H NTK ) s , 2 ≅ ( L 2 ( X ) , H 2 p + 1 ( X ) ) s , 2 , so [ H N T K ] s ≅ ( L 2 ( X ) , H p + 1 2 ( X ) ) s , 2 [\mathcal{H}^{\mathrm{NTK}}]^s \cong(L^2(\mathcal{X}),H^{\frac{p+1}{2}}(\mathcal{X}))_{s,2} [ H NTK ] s ≅ ( L 2 ( X ) , H 2 p + 1 ( X ) ) s , 2 . Using the identification L 2 ( X , d x ) ≅ L 2 ( X ) L^2(\mathcal{X},\mathrm{d}x) \cong L^2(\mathcal{X}) L 2 ( X , d x ) ≅ L 2 ( X ) and the fact that L 2 ( X , d x ) ≅ H 0 ( X ) L^2(\mathcal{X},\mathrm{d}x) \cong H^0(\mathcal{X}) L 2 ( X , d x ) ≅ H 0 ( X ) , we get L 2 ( X ) ≅ H 0 ( X ) L^2(\mathcal{X}) \cong H^0(\mathcal{X}) L 2 ( X ) ≅ H 0 ( X ) . [24 Lemma H.6] then gives ( L 2 ( X ) , H p + 1 2 ( X ) ) s , 2 ≅ ( H 0 ( X ) , H p + 1 2 ( X ) ) s , 2 (L^2(\mathcal{X}),H^{\frac{p+1}{2}}(\mathcal{X}))_{s,2} \cong(H^0(\mathcal{X}),H^{\frac{p+1}{2}}(\mathcal{X}))_{s,2} ( L 2 ( X ) , H 2 p + 1 ( X ) ) s , 2 ≅ ( H 0 ( X ) , H 2 p + 1 ( X ) ) s , 2 . Therefore, by [17 Theorem 6.4.5] , we obtain [ H N T K ] s ≅ ( H 0 ( X ) , H p + 1 2 ( X ) ) s , 2 = H s ( p + 1 ) 2 ( X ) [\mathcal{H}^{\mathrm{NTK}}]^s \cong(H^0(\mathcal{X}),H^{\frac{p+1}{2}}(\mathcal{X}))_{s,2}=H^{\frac{s(p+1)}{2}}(\mathcal{X}) [ H NTK ] s ≅ ( H 0 ( X ) , H 2 p + 1 ( X ) ) s , 2 = H 2 s ( p + 1 ) ( X ) .
Endpoints s ∈ { 0 , 1 } s \in\{0,1\} s ∈ { 0 , 1 } . Directly from the definition: [ H N T K ] 0 = L 2 ( X ) ≅ H 0 ( X ) [\mathcal{H}^{\mathrm{NTK}}]^0=L^2(\mathcal{X}) \cong H^0(\mathcal{X}) [ H NTK ] 0 = L 2 ( X ) ≅ H 0 ( X ) , and [ H N T K ] 1 = H N T K ≅ H p + 1 2 ( X ) [\mathcal{H}^{\mathrm{NTK}}]^1=\mathcal{H}^{\mathrm{NTK}} \cong H^{\frac{p+1}{2}}(\mathcal{X}) [ H NTK ] 1 = H NTK ≅ H 2 p + 1 ( X ) by Theorem G.12.
Background on Integral/Covariance Operators and Interpolation Spaces
Let X ⊂ R p \mathcal{X} \subset\mathbb{R}^p X ⊂ R p be the input space, Y ⊂ R \mathcal{Y} \subset\mathbb{R} Y ⊂ R be the target space, and ρ \rho ρ be an unknown probability distribution supported on X × Y \mathcal{X} \times\mathcal{Y} X × Y with marginal distribution μ \mu μ on X \mathcal{X} X . Let L 2 ( X ) L^2(\mathcal{X}) L 2 ( X ) denote the space of square-integrable functions with respect to μ \mu μ defined on the input space X \mathcal{X} X and taking values in Y \mathcal{Y} Y . Let H \mathcal{H} H be an RKHS defined on X \mathcal{X} X and associated with a kernel K : X × X → R K:\mathcal{X} \times\mathcal{X} \to\mathbb{R} K : X × X → R . We assume that K K K is a Mercer kernel, i.e., a continuous symmetric positive semi-definite kernel defined on a compact space X \mathcal{X} X . This implies that K K K is bounded, i.e., there exists κ ≥ 0 \kappa\ge0 κ ≥ 0 such that K ( x , x ) ≤ κ 2 K(x,x) \le\kappa^2 K ( x , x ) ≤ κ 2 for any x ∈ X x \in\mathcal{X} x ∈ X , and that H \mathcal{H} H is separable, i.e., it contains a countable dense subset [86 Lemma 4.33] . Under this boundedness condition, it holds that H ⊂ L 2 ( X ) \mathcal{H} \subset L^2(\mathcal{X}) H ⊂ L 2 ( X ) . Indeed, for any f ∈ H f \in\mathcal{H} f ∈ H and x ∈ X x \in\mathcal{X} x ∈ X , we can write by the reproducing property that f ( x ) = ⟨ f , K ( x , ⋅ ) ⟩ H f(x)=\langle f,K(x,\cdot)\rangle_{\mathcal{H}} f ( x ) = ⟨ f , K ( x , ⋅ ) ⟩ H , which implies by Cauchy-Schwarz inequality that ∣ f ( x ) ∣ ≤ ∥ f ∥ H K ( x , x ) ≤ κ ∥ f ∥ H |f(x)| \le\|f\|_{\mathcal{H}}\sqrt{K(x,x)} \le\kappa\|f\|_{\mathcal{H}} ∣ f ( x ) ∣ ≤ ∥ f ∥ H K ( x , x ) ≤ κ ∥ f ∥ H . This gives that ∥ f ∥ L 2 ( X ) 2 ≤ κ 2 ∥ f ∥ H 2 < + ∞ \|f\|^2_{L^2(\mathcal{X})} \le\kappa^2\|f\|^2_{\mathcal{H}} < +\infty ∥ f ∥ L 2 ( X ) 2 ≤ κ 2 ∥ f ∥ H 2 < + ∞ , and hence that f ∈ L 2 ( X ) f \in L^2(\mathcal{X}) f ∈ L 2 ( X ) .
Integral and covariance operators. Let I : H → L 2 ( X ) \mathcal{I}:\mathcal{H} \to L^2(\mathcal{X}) I : H → L 2 ( X ) be the natural embedding inclusion operator defined for any f ∈ H f \in\mathcal{H} f ∈ H by I ( f ) = f \mathcal{I}(f)=f I ( f ) = f . Its adjoint I ⋆ : L 2 ( X ) → H \mathcal{I}^{\star}:L^2(\mathcal{X}) \to\mathcal{H} I ⋆ : L 2 ( X ) → H is defined for any f ∈ L 2 ( X ) f \in L^2(\mathcal{X}) f ∈ L 2 ( X ) by I ⋆ ( f ) = ∫ X K ( x , ⋅ ) f ( x ) d μ ( x ) ∈ H \mathcal{I}^{\star}(f)=\int_{\mathcal{X}}K(x,\cdot)f(x)\mathrm{d}\mu(x) \in\mathcal{H} I ⋆ ( f ) = ∫ X K ( x , ⋅ ) f ( x ) d μ ( x ) ∈ H . Both I \mathcal{I} I and I ⋆ \mathcal{I}^{\star} I ⋆ are Hilbert-Schmidt operators, with Hilbert-Schmidt norms:
∥ I ∥ H S = ∥ I ⋆ ∥ H S = ∫ X K ( x , x ) d μ ( x ) ≤ κ , \|\mathcal{I}\|_{\mathrm{HS}}=\|\mathcal{I}^{\star}\|_{\mathrm{HS}}=\sqrt{\int_{\mathcal{X}}K(x,x)\mathrm{d}\mu(x)}\le\kappa, ∥ I ∥ HS = ∥ I ⋆ ∥ HS = ∫ X K ( x , x ) d μ ( x ) ≤ κ ,
and are in particular compact. Let Σ = I ⋆ I : H → H \Sigma=\mathcal{I}^{\star}\mathcal{I}:\mathcal{H}\to\mathcal{H} Σ = I ⋆ I : H → H be the covariance operator defined by Σ = E x ∼ μ [ K ( x , ⋅ ) ⊗ K ( x , ⋅ ) ] \Sigma=\mathbb{E}_{x\sim\mu}[K(x,\cdot)\otimes K(x,\cdot)] Σ = E x ∼ μ [ K ( x , ⋅ ) ⊗ K ( x , ⋅ )] , where ( a ⊗ b ) c = ⟨ b , c ⟩ H a (a\otimes b)c=\langle b,c\rangle_{\mathcal{H}}a ( a ⊗ b ) c = ⟨ b , c ⟩ H a for a , b , c ∈ H a,b,c\in\mathcal{H} a , b , c ∈ H . Let T = I I ⋆ : L 2 ( X ) → L 2 ( X ) T=\mathcal{I}\mathcal{I}^{\star}:L^2(\mathcal{X})\to L^2(\mathcal{X}) T = I I ⋆ : L 2 ( X ) → L 2 ( X ) be the integral operator defined for any f ∈ L 2 ( X ) f\in L^2(\mathcal{X}) f ∈ L 2 ( X ) by T f = ∫ X K ( ⋅ , t ) f ( t ) d μ ( t ) Tf=\int_{\mathcal{X}}K(\cdot,t)f(t)\mathrm{d}\mu(t) T f = ∫ X K ( ⋅ , t ) f ( t ) d μ ( t ) . Since K K K is a Mercer kernel, it follows that T T T is compact, self-adjoint, and positive definite on L 2 ( X ) L^2(\mathcal{X}) L 2 ( X ) [86 Theorem 4.27] . By the spectral theorem , there exists an at most countable index set N \mathcal{N} N , a non-increasing summable sequence ( λ i ) i ∈ N ⊂ ( 0 , + ∞ ) (\lambda_i)_{i\in\mathcal{N}}\subset(0,+\infty) ( λ i ) i ∈ N ⊂ ( 0 , + ∞ ) , and an orthonormal family ( e i ) i ∈ N ⊂ L 2 ( X ) (e_i)_{i\in\mathcal{N}}\subset L^2(\mathcal{X}) ( e i ) i ∈ N ⊂ L 2 ( X ) forming an orthonormal basis of Range ( I ) ⊂ L 2 ( X ) \operatorname{Range}(\mathcal{I})\subset L^2(\mathcal{X}) Range ( I ) ⊂ L 2 ( X ) , such that ( λ i e i ) i ∈ N (\sqrt{\lambda_i}e_i)_{i\in\mathcal{N}} ( λ i e i ) i ∈ N is an orthonormal basis of H \mathcal{H} H [86 Section 4.5] . The operators T T T and Σ \Sigma Σ admit the spectral decompositions:
T = ∑ i ∈ N λ i ⟨ ⋅ , e i ⟩ L 2 ( X ) e i and Σ = ∑ i ∈ N λ i ⟨ ⋅ , λ i e i ⟩ H λ i e i , T=\sum_{i\in\mathcal{N}}\lambda_i\langle\cdot,e_i\rangle_{L^2(\mathcal{X})}e_i
\quad\text{and}\quad
\Sigma=\sum_{i\in\mathcal{N}}\lambda_i\langle\cdot,\sqrt{\lambda_i}e_i\rangle_{\mathcal{H}}\sqrt{\lambda_i}e_i, T = i ∈ N ∑ λ i ⟨ ⋅ , e i ⟩ L 2 ( X ) e i and Σ = i ∈ N ∑ λ i ⟨ ⋅ , λ i e i ⟩ H λ i e i ,
and share the same non-zero eigenvalues ( λ i ) i ∈ N (\lambda_i)_{i\in\mathcal{N}} ( λ i ) i ∈ N , since T = I I ⋆ T=\mathcal{I}\mathcal{I}^{\star} T = I I ⋆ and Σ = I ⋆ I \Sigma=\mathcal{I}^{\star}\mathcal{I} Σ = I ⋆ I have the same non-zero spectrum. Moreover, by Mercer’s theorem [86 Theorem 4.49] , we have for any x , x ′ ∈ X x,x' \in\mathcal{X} x , x ′ ∈ X :
K ( x , x ′ ) = ∑ i ∈ N λ i e i ( x ) e i ( x ′ ) , K(x,x')=\sum_{i\in\mathcal{N}}\lambda_i e_i(x)e_i(x'), K ( x , x ′ ) = i ∈ N ∑ λ i e i ( x ) e i ( x ′ ) ,
where the convergence is absolute and uniform. Since N \mathcal{N} N is at most countable, there exists a bijection ϕ : N → N \phi:\mathbb{N}\to\mathcal{N} ϕ : N → N ; hence, by reindexing via ϕ \phi ϕ , we may write the eigenpairs as ( λ ϕ ( i ) , e ϕ ( i ) ) i ≥ 1 (\lambda_{\phi(i)},e_{\phi(i)})_{i\ge1} ( λ ϕ ( i ) , e ϕ ( i ) ) i ≥ 1 , and by a slight abuse of notation, simply denote them by ( λ i , e i ) i ≥ 1 (\lambda_i,e_i)_{i\ge1} ( λ i , e i ) i ≥ 1 .
Interpolation space. For s ≥ 0 s \ge0 s ≥ 0 , we define the interpolation space [ H ] s [\mathcal{H}]^{s} [ H ] s by [39 Section 2] :
[ H ] s = Range ( T s 2 ) = { ∑ i ≥ 1 a i λ i s 2 e i | ∑ i ≥ 1 a i 2 < + ∞ } ⊂ L 2 ( X ) [\mathcal{H}]^{s} = \operatorname{Range}(T^{\frac{s}{2}}) = \left\{ \sum_{i \ge1} a_i \lambda_i^{\frac{s}{2}} e_i \,\middle|\, \sum_{i \ge1} a_i^2 < +\infty\right\} \subset L^2(\mathcal{X}) [ H ] s = Range ( T 2 s ) = { i ≥ 1 ∑ a i λ i 2 s e i i ≥ 1 ∑ a i 2 < + ∞ } ⊂ L 2 ( X )
equipped with the norm ∥ ∑ i ≥ 1 a i λ i s 2 e i ∥ [ H ] s = ( ∑ i ≥ 1 a i 2 ) 1 2 \left\|\sum_{i \ge1} a_i \lambda_i^{\frac{s}{2}} e_i\right\|_{[\mathcal{H}]^s} = \left(\sum_{i \ge1} a_i^2\right)^{\frac{1}{2}} ∑ i ≥ 1 a i λ i 2 s e i [ H ] s = ( ∑ i ≥ 1 a i 2 ) 2 1 . Note that [ H ] 0 ⊂ L 2 ( X ) [\mathcal{H}]^0 \subset L^2(\mathcal{X}) [ H ] 0 ⊂ L 2 ( X ) and [ H ] 1 = H [\mathcal{H}]^1 = \mathcal{H} [ H ] 1 = H . The functions in [ H ] s [\mathcal{H}]^s [ H ] s with smaller values of s s s exhibit lower smoothness and are therefore more difficult for kernel regression algorithms to estimate.
Auxiliary Technical Lemmas
In Lemmas I.1 to I.3, let W 0 W_0 W 0 be the principal branch of the Lambert W W W function, i.e. , the real-valued solution v = W 0 ( z ) ≥ 0 v = W_0(z) \ge0 v = W 0 ( z ) ≥ 0 of an equation of the form v exp ( v ) = z v\exp(v) = z v exp ( v ) = z for z ≥ 0 z \ge0 z ≥ 0 .
Lemma I.1. Let C 1 , C 2 > 0 C_1, C_2 > 0 C 1 , C 2 > 0 , 0 < α < 1 0 < \alpha< 1 0 < α < 1 , 0 < γ < 2 0 < \gamma< 2 0 < γ < 2 , and x ≥ [ ( 4 C 1 ( ln 6 ) 3 C 2 ) 1 α + γ ] + 1 x \ge\left[\left(\frac{4C_1(\ln6)^3}{C_2}\right)^{\frac{1}{\alpha+\gamma}}\right] + 1 x ≥ [ ( C 2 4 C 1 ( l n 6 ) 3 ) α + γ 1 ] + 1 . Let φ : ( 0 , 1 ) → ( 0 , + ∞ ) \varphi: (0,1) \to(0,+\infty) φ : ( 0 , 1 ) → ( 0 , + ∞ ) be the function defined for any δ ∈ ( 0 , 1 ) \delta\in(0,1) δ ∈ ( 0 , 1 ) by:
φ ( δ ) = C 1 ( ln 6 δ ) 3 x − α + C 2 x γ δ . \varphi(\delta) = C_1\left(\ln\frac{6}{\delta}\right)^3x^{-\alpha} + C_2x^\gamma\delta. φ ( δ ) = C 1 ( ln δ 6 ) 3 x − α + C 2 x γ δ . Then, φ \varphi φ has a unique minimizer δ ⋆ ∈ ( 0 , 1 ) \delta^\star\in(0,1) δ ⋆ ∈ ( 0 , 1 ) , given by:
δ ⋆ = 108 C 1 C 2 x − ( α + γ ) W 0 3 ( x α + γ 3 ( C 2 18 C 1 ) 1 3 ) , \delta^\star= \frac{108C_1}{C_2}x^{-(\alpha+\gamma)}W_0^3\left(x^{\frac{\alpha+\gamma}{3}}\left(\frac{C_2}{18C_1}\right)^{\frac{1}{3}}\right), δ ⋆ = C 2 108 C 1 x − ( α + γ ) W 0 3 ( x 3 α + γ ( 18 C 1 C 2 ) 3 1 ) , and the corresponding minimum value is:
φ ( δ ⋆ ) = 27 C 1 W 0 3 ( x α + γ 3 ( C 2 18 C 1 ) 1 3 ) ( 3 W 0 ( x α + γ 3 ( C 2 18 C 1 ) 1 3 ) + 4 ) x − α . \varphi(\delta^\star) = 27C_1W_0^3\left(x^{\frac{\alpha+\gamma}{3}}\left(\frac{C_2}{18C_1}\right)^{\frac{1}{3}}\right)\left(3W_0\left(x^{\frac{\alpha+\gamma}{3}}\left(\frac{C_2}{18C_1}\right)^{\frac{1}{3}}\right)+4\right)x^{-\alpha}. φ ( δ ⋆ ) = 27 C 1 W 0 3 ( x 3 α + γ ( 18 C 1 C 2 ) 3 1 ) ( 3 W 0 ( x 3 α + γ ( 18 C 1 C 2 ) 3 1 ) + 4 ) x − α .
Proof. Existence and uniqueness of δ ⋆ ∈ ( 0 , 1 ) \delta^\star\in(0,1) δ ⋆ ∈ ( 0 , 1 ) . The derivative of φ \varphi φ for any δ ∈ ( 0 , 1 ) \delta\in(0,1) δ ∈ ( 0 , 1 ) is:
φ ′ ( δ ) = − 4 C 1 ( ln 6 δ ) 3 x − α δ + C 2 x γ . \varphi'(\delta) = -\frac{4C_1\left(\ln\frac{6}{\delta}\right)^3x^{-\alpha}}{\delta} + C_2x^\gamma. φ ′ ( δ ) = − δ 4 C 1 ( ln δ 6 ) 3 x − α + C 2 x γ .
The first term is strictly increasing in δ \delta δ on ( 0 , 1 ) (0,1) ( 0 , 1 ) , since ln ( 6 / δ ) \ln(6/\delta) ln ( 6/ δ ) and 1 / δ 1/\delta 1/ δ are positive and decrease as δ \delta δ increases, while the second term is constant. Hence, φ ′ \varphi' φ ′ is strictly increasing on ( 0 , 1 ) (0,1) ( 0 , 1 ) . Moreover, φ ′ ( δ ) → − ∞ \varphi'(\delta) \to-\infty φ ′ ( δ ) → − ∞ as δ → 0 + \delta\to0^+ δ → 0 + , and φ ′ ( δ ) → − 4 C 1 ( ln 6 ) 3 x − α + C 2 x γ \varphi'(\delta) \to-4C_1(\ln6)^3x^{-\alpha} + C_2x^\gamma φ ′ ( δ ) → − 4 C 1 ( ln 6 ) 3 x − α + C 2 x γ as δ → 1 − \delta\to1^- δ → 1 − . Since x ≥ [ ( 4 C 1 ( ln 6 ) 3 C 2 ) 1 α + γ ] + 1 > ( 4 C 1 ( ln 6 ) 3 C 2 ) 1 α + γ x \ge\left[\left(\frac{4C_1(\ln6)^3}{C_2}\right)^{\frac{1}{\alpha+\gamma}}\right] + 1 > \left(\frac{4C_1(\ln6)^3}{C_2}\right)^{\frac{1}{\alpha+\gamma}} x ≥ [ ( C 2 4 C 1 ( l n 6 ) 3 ) α + γ 1 ] + 1 > ( C 2 4 C 1 ( l n 6 ) 3 ) α + γ 1 , we have x α + γ > 4 C 1 ( ln 6 ) 3 C 2 x^{\alpha+\gamma} > \frac{4C_1(\ln6)^3}{C_2} x α + γ > C 2 4 C 1 ( l n 6 ) 3 , so that the limit − 4 C 1 ( ln 6 ) 3 x − α + C 2 x γ -4C_1(\ln6)^3x^{-\alpha} + C_2x^\gamma − 4 C 1 ( ln 6 ) 3 x − α + C 2 x γ is positive, and φ ′ ( δ ) > 0 \varphi'(\delta) > 0 φ ′ ( δ ) > 0 for every δ \delta δ close enough to 1 1 1 . Therefore, by the intermediate value theorem applied to the continuous function φ ′ \varphi' φ ′ , there exists δ ⋆ ∈ ( 0 , 1 ) \delta^\star\in(0,1) δ ⋆ ∈ ( 0 , 1 ) such that φ ′ ( δ ⋆ ) = 0 \varphi'(\delta^\star) = 0 φ ′ ( δ ⋆ ) = 0 . Since φ ′ \varphi' φ ′ is strictly increasing, this zero is unique, and since φ ′ \varphi' φ ′ changes sign from negative to positive at δ ⋆ \delta^\star δ ⋆ , it is the unique minimizer of φ \varphi φ on ( 0 , 1 ) (0,1) ( 0 , 1 ) .
Formula for δ ⋆ \delta^\star δ ⋆ . The minimizer δ ⋆ \delta^\star δ ⋆ satisfies φ ′ ( δ ⋆ ) = 0 \varphi'(\delta^\star) = 0 φ ′ ( δ ⋆ ) = 0 , that is:
4 C 1 x − α δ ⋆ ( ln 6 δ ⋆ ) 3 = C 2 x γ . \frac{4C_1x^{-\alpha}}{\delta^\star}\left(\ln\frac{6}{\delta^\star}\right)^3 = C_2x^\gamma. δ ⋆ 4 C 1 x − α ( ln δ ⋆ 6 ) 3 = C 2 x γ .
Setting y = ln 6 δ ⋆ y = \ln\frac{6}{\delta^\star} y = ln δ ⋆ 6 , so that δ ⋆ = 6 exp ( − y ) \delta^\star= 6\exp(-y) δ ⋆ = 6 exp ( − y ) , we obtain:
y 3 exp ( y ) = 3 C 2 2 C 1 x α + γ ⟺ y exp ( y 3 ) = ( 3 C 2 2 C 1 x α + γ ) 1 3 . y^3\exp(y) = \frac{3C_2}{2C_1}x^{\alpha+\gamma} \Longleftrightarrow y\exp\left(\frac{y}{3}\right) = \left(\frac{3C_2}{2C_1}x^{\alpha+\gamma}\right)^{\frac{1}{3}}. y 3 exp ( y ) = 2 C 1 3 C 2 x α + γ ⟺ y exp ( 3 y ) = ( 2 C 1 3 C 2 x α + γ ) 3 1 .
Equivalently, setting z = x α + γ 3 ( C 2 18 C 1 ) 1 3 z = x^{\frac{\alpha+\gamma}{3}}\left(\frac{C_2}{18C_1}\right)^{\frac{1}{3}} z = x 3 α + γ ( 18 C 1 C 2 ) 3 1 , y 3 exp ( y 3 ) = z \frac{y}{3}\exp\left(\frac{y}{3}\right) = z 3 y exp ( 3 y ) = z , since 1 3 ( 3 C 2 2 C 1 ) 1 3 = ( C 2 18 C 1 ) 1 3 \frac{1}{3}\left(\frac{3C_2}{2C_1}\right)^{\frac{1}{3}} = \left(\frac{C_2}{18C_1}\right)^{\frac{1}{3}} 3 1 ( 2 C 1 3 C 2 ) 3 1 = ( 18 C 1 C 2 ) 3 1 . Using the identity W 0 ( w ) exp ( W 0 ( w ) ) = w W_0(w)\exp(W_0(w)) = w W 0 ( w ) exp ( W 0 ( w )) = w , we get y = 3 W 0 ( z ) y = 3W_0(z) y = 3 W 0 ( z ) . Since δ ⋆ = 6 exp ( − y ) \delta^\star= 6\exp(-y) δ ⋆ = 6 exp ( − y ) , the identity exp ( − W 0 ( w ) ) = W 0 ( w ) w \exp(-W_0(w))=\frac{W_0(w)}{w} exp ( − W 0 ( w )) = w W 0 ( w ) gives δ ⋆ = 6 [ exp ( − W 0 ( z ) ) ] 3 = 6 W 0 3 ( z ) z 3 = 108 C 1 C 2 x − ( α + γ ) W 0 3 ( z ) \delta^\star=6\left[\exp(-W_0(z))\right]^3=\frac{6W_0^3(z)}{z^3}=\frac{108C_1}{C_2}x^{-(\alpha+\gamma)}W_0^3(z) δ ⋆ = 6 [ exp ( − W 0 ( z )) ] 3 = z 3 6 W 0 3 ( z ) = C 2 108 C 1 x − ( α + γ ) W 0 3 ( z ) , where we used z 3 = C 2 18 C 1 x α + γ z^3=\frac{C_2}{18C_1}x^{\alpha+\gamma} z 3 = 18 C 1 C 2 x α + γ in the last equality.
Formula for φ ( δ ⋆ ) \varphi(\delta^\star) φ ( δ ⋆ ) . From the optimality condition, C 2 x γ δ ⋆ = 4 C 1 y 3 x − α C_2x^\gamma\delta^\star=4C_1y^3x^{-\alpha} C 2 x γ δ ⋆ = 4 C 1 y 3 x − α . As a consequence:
φ ( δ ⋆ ) = C 1 ( ln 6 δ ⋆ ) 4 x − α + C 2 x γ δ ⋆ = C 1 y 4 x − α + 4 C 1 y 3 x − α = C 1 y 3 ( y + 4 ) x − α = 27 C 1 W 0 3 ( z ) ( 3 W 0 ( z ) + 4 ) x − α . \begin{aligned}
\varphi(\delta^\star)=C_1\left(\ln\frac{6}{\delta^\star}\right)^4x^{-\alpha}+C_2x^\gamma\delta^\star=C_1y^4x^{-\alpha}+4C_1y^3x^{-\alpha} \\
&=C_1y^3(y+4)x^{-\alpha}=27C_1W_0^3(z)(3W_0(z)+4)x^{-\alpha}.
\end{aligned} φ ( δ ⋆ ) = C 1 ( ln δ ⋆ 6 ) 4 x − α + C 2 x γ δ ⋆ = C 1 y 4 x − α + 4 C 1 y 3 x − α = C 1 y 3 ( y + 4 ) x − α = 27 C 1 W 0 3 ( z ) ( 3 W 0 ( z ) + 4 ) x − α .
Auxiliary Technical Lemmas
Lemma I.2 (Upper-bound on W 0 W_0 W 0 ). For any x ≥ e x\ge e x ≥ e , it holds that W 0 ( x ) ≤ ln x W_0(x)\le\ln x W 0 ( x ) ≤ ln x .
Proof. Define F ( x ) = ln x − W 0 ( x ) F(x)=\ln x-W_0(x) F ( x ) = ln x − W 0 ( x ) for any x ≥ e x\ge e x ≥ e . We have that:
d d x F ( x ) = 1 x − d d x W 0 ( x ) . (75) \frac{\mathrm{d}}{\mathrm{d}x}F(x)=\frac{1}{x}-\frac{\mathrm{d}}{\mathrm{d}x}W_0(x). \tag*{(75)} d x d F ( x ) = x 1 − d x d W 0 ( x ) . (75)
To compute the derivative of W 0 ( x ) W_0(x) W 0 ( x ) with respect to x x x , we differentiate the identity W 0 ( x ) exp ( W 0 ( x ) ) = x W_0(x)\exp(W_0(x))=x W 0 ( x ) exp ( W 0 ( x )) = x . Using the product and chain rules yields:
( d d x W 0 ( x ) ) exp ( W 0 ( x ) ) + W 0 ( x ) exp ( W 0 ( x ) ) ( d d x W 0 ( x ) ) = 1. \left(\frac{\mathrm{d}}{\mathrm{d}x}W_0(x)\right)\exp(W_0(x))+W_0(x)\exp(W_0(x))\left(\frac{\mathrm{d}}{\mathrm{d}x}W_0(x)\right)=1. ( d x d W 0 ( x ) ) exp ( W 0 ( x )) + W 0 ( x ) exp ( W 0 ( x )) ( d x d W 0 ( x ) ) = 1.
After rearranging the terms and using the identity W 0 ( x ) exp ( W 0 ( x ) ) = x W_0(x)\exp(W_0(x))=x W 0 ( x ) exp ( W 0 ( x )) = x , we obtain:
d d x W 0 ( x ) = W 0 ( x ) x ( 1 + W 0 ( x ) ) . \frac{\mathrm{d}}{\mathrm{d}x}W_0(x)=\frac{W_0(x)}{x(1+W_0(x))}. d x d W 0 ( x ) = x ( 1 + W 0 ( x )) W 0 ( x ) .
Note that this derivative is defined for any x ≥ e x\ge e x ≥ e because x ( 1 + W 0 ( x ) ) ≠ 0 x(1+W_0(x))\ne0 x ( 1 + W 0 ( x )) = 0 , which is due to the fact that W 0 ( x ) ≥ 0 W_0(x)\ge0 W 0 ( x ) ≥ 0 for any x ≥ e x\ge e x ≥ e . Substituting the obtained derivative into Equation (75) gives:
d d x F ( x ) = 1 x − W 0 ( x ) x ( 1 + W 0 ( x ) ) = 1 x ( 1 + W 0 ( x ) ) ≥ 0. \frac{\mathrm{d}}{\mathrm{d}x}F(x)=\frac{1}{x}-\frac{W_0(x)}{x(1+W_0(x))}=\frac{1}{x(1+W_0(x))}\ge0. d x d F ( x ) = x 1 − x ( 1 + W 0 ( x )) W 0 ( x ) = x ( 1 + W 0 ( x )) 1 ≥ 0.
Thus, F F F is increasing on [ e , + ∞ ) [e,+\infty) [ e , + ∞ ) . Since F ( e ) = ln e − W 0 ( e ) = 0 F(e)=\ln e-W_0(e)=0 F ( e ) = ln e − W 0 ( e ) = 0 , we get F ( x ) ≥ 0 F(x)\ge0 F ( x ) ≥ 0 for any x ≥ e x\ge e x ≥ e , which implies that W 0 ( x ) ≤ ln x W_0(x)\le\ln x W 0 ( x ) ≤ ln x . This concludes the proof.
Lemma I.3. Let 0 < α < γ 0<\alpha<\gamma 0 < α < γ and κ > 0 \kappa>0 κ > 0 . When x ≥ max ( e , ( e κ ) 1 α ) x\ge\max\left(e,\left(\frac{e}{\kappa}\right)^{\frac{1}{\alpha}}\right) x ≥ max ( e , ( κ e ) α 1 ) , the following holds:
W 0 3 ( x α κ ) ( 3 W 0 ( x α κ ) + 4 ) ≤ ( 4 γ 3 + 3 γ 4 + 12 ( γ 2 + γ 3 ) ∣ ln κ ∣ + ( 12 γ + 18 γ 2 ) ( ln κ ) 2 + ( 4 + 12 γ ) ∣ ln κ ∣ 3 + 3 ( ln κ ) 4 ) ( ln x ) 4 . W_0^3(x^\alpha\kappa)(3W_0(x^\alpha\kappa)+4)\le\left(4\gamma^3+3\gamma^4+12(\gamma^2+\gamma^3)|\ln\kappa|+(12\gamma+18\gamma^2)(\ln\kappa)^2+(4+12\gamma)|\ln\kappa|^3+3(\ln\kappa)^4\right)(\ln x)^4. W 0 3 ( x α κ ) ( 3 W 0 ( x α κ ) + 4 ) ≤ ( 4 γ 3 + 3 γ 4 + 12 ( γ 2 + γ 3 ) ∣ ln κ ∣ + ( 12 γ + 18 γ 2 ) ( ln κ ) 2 + ( 4 + 12 γ ) ∣ ln κ ∣ 3 + 3 ( ln κ ) 4 ) ( ln x ) 4 . Proof. Fix 0 < α < γ 0<\alpha<\gamma 0 < α < γ and κ > 0 \kappa>0 κ > 0 . We have x α κ ≥ e x^\alpha\kappa\ge e x α κ ≥ e since it is assumed that x ≥ ( e κ ) 1 α x\ge\left(\frac{e}{\kappa}\right)^{\frac{1}{\alpha}} x ≥ ( κ e ) α 1 . Therefore, by Lemma I.2 , we get W 0 ( x α κ ) ≤ ln ( x α κ ) = α ln x + ln κ W_0(x^\alpha\kappa)\le\ln(x^\alpha\kappa)=\alpha\ln x+\ln\kappa W 0 ( x α κ ) ≤ ln ( x α κ ) = α ln x + ln κ . Using this, we have:
W 0 3 ( x α κ ) ≤ ( α ln x + ln κ ) 3 = α 3 ( ln x ) 3 + 3 α 2 ( ln x ) 2 ln κ + 3 α ln x ( ln κ ) 2 + ( ln κ ) 3 , W_0^3(x^\alpha\kappa)\le(\alpha\ln x+\ln\kappa)^3=\alpha^3(\ln x)^3+3\alpha^2(\ln x)^2\ln\kappa+3\alpha\ln x(\ln\kappa)^2+(\ln\kappa)^3, W 0 3 ( x α κ ) ≤ ( α ln x + ln κ ) 3 = α 3 ( ln x ) 3 + 3 α 2 ( ln x ) 2 ln κ + 3 α ln x ( ln κ ) 2 + ( ln κ ) 3 , and
W 0 3 ( x α κ ) ( 3 W 0 ( x α κ ) + 4 ) ≤ ( α ln x + ln κ ) 3 ( 3 α ln x + 3 ln κ + 4 ) = ( α 3 ( ln x ) 3 + 3 α 2 ( ln x ) 2 ln κ + 3 α ln x ( ln κ ) 2 + ( ln κ ) 3 ) ( 3 α ln x + 3 ln κ + 4 ) = 3 α 4 ( ln x ) 4 + 3 α 3 ln κ ( ln x ) 3 + 4 α 3 ( ln x ) 3 + 9 α 3 ( ln x ) 3 ln κ + 9 α 2 ( ln x ) 2 ( ln κ ) 2 + 12 α 2 ( ln x ) 2 ln κ + 9 α 2 ( ln x ) 2 ( ln κ ) 2 + 9 α ln x ( ln κ ) 3 + 12 α ln x ( ln κ ) 2 + 3 α ln x ( ln κ ) 3 + 3 ( ln κ ) 4 + 4 ( ln κ ) 3 = 3 α 4 ( ln x ) 4 + 12 α 3 ln κ ( ln x ) 3 + 4 α 3 ( ln x ) 3 + 18 α 2 ( ln x ) 2 ( ln κ ) 2 + 12 α 2 ( ln x ) 2 ln κ + 12 α ln x ( ln κ ) 3 + 12 α ln x ( ln κ ) 2 + 3 ( ln κ ) 4 + 4 ( ln κ ) 3 . \begin{aligned}
W_0^3(x^\alpha\kappa)(3W_0(x^\alpha\kappa)+4)
&\le(\alpha\ln x+\ln\kappa)^3(3\alpha\ln x+3\ln\kappa+4)\\
&=(\alpha^3(\ln x)^3+3\alpha^2(\ln x)^2\ln\kappa+3\alpha\ln x(\ln\kappa)^2+(\ln\kappa)^3)(3\alpha\ln x+3\ln\kappa+4)\\
&=3\alpha^4(\ln x)^4+3\alpha^3\ln\kappa(\ln x)^3+4\alpha^3(\ln x)^3+9\alpha^3(\ln x)^3\ln\kappa+9\alpha^2(\ln x)^2(\ln\kappa)^2\\
&\quad+12\alpha^2(\ln x)^2\ln\kappa+9\alpha^2(\ln x)^2(\ln\kappa)^2+9\alpha\ln x(\ln\kappa)^3+12\alpha\ln x(\ln\kappa)^2\\
&\quad+3\alpha\ln x(\ln\kappa)^3+3(\ln\kappa)^4+4(\ln\kappa)^3\\
&=3\alpha^4(\ln x)^4+12\alpha^3\ln\kappa(\ln x)^3+4\alpha^3(\ln x)^3+18\alpha^2(\ln x)^2(\ln\kappa)^2+12\alpha^2(\ln x)^2\ln\kappa\\
&\quad+12\alpha\ln x(\ln\kappa)^3+12\alpha\ln x(\ln\kappa)^2+3(\ln\kappa)^4+4(\ln\kappa)^3.
\end{aligned} W 0 3 ( x α κ ) ( 3 W 0 ( x α κ ) + 4 ) ≤ ( α ln x + ln κ ) 3 ( 3 α ln x + 3 ln κ + 4 ) = ( α 3 ( ln x ) 3 + 3 α 2 ( ln x ) 2 ln κ + 3 α ln x ( ln κ ) 2 + ( ln κ ) 3 ) ( 3 α ln x + 3 ln κ + 4 ) = 3 α 4 ( ln x ) 4 + 3 α 3 ln κ ( ln x ) 3 + 4 α 3 ( ln x ) 3 + 9 α 3 ( ln x ) 3 ln κ + 9 α 2 ( ln x ) 2 ( ln κ ) 2 + 12 α 2 ( ln x ) 2 ln κ + 9 α 2 ( ln x ) 2 ( ln κ ) 2 + 9 α ln x ( ln κ ) 3 + 12 α ln x ( ln κ ) 2 + 3 α ln x ( ln κ ) 3 + 3 ( ln κ ) 4 + 4 ( ln κ ) 3 = 3 α 4 ( ln x ) 4 + 12 α 3 ln κ ( ln x ) 3 + 4 α 3 ( ln x ) 3 + 18 α 2 ( ln x ) 2 ( ln κ ) 2 + 12 α 2 ( ln x ) 2 ln κ + 12 α ln x ( ln κ ) 3 + 12 α ln x ( ln κ ) 2 + 3 ( ln κ ) 4 + 4 ( ln κ ) 3 .
It is also assumed that x ≥ e x \ge e x ≥ e , hence ( ln x ) i ≤ ( ln x ) j (\ln x)^i \le(\ln x)^j ( ln x ) i ≤ ( ln x ) j for 0 ≤ i < j ≤ 4 0 \le i < j \le4 0 ≤ i < j ≤ 4 . Also, we know that α < γ \alpha< \gamma α < γ . Consequently, we get
W 0 3 ( x α κ ) ≤ ( γ 3 + 3 γ 2 ∣ ln κ ∣ + 3 γ ( ln κ ) 2 + ∣ ln κ ∣ 3 ) ( ln x ) 3 W_0^3(x^\alpha\kappa) \le\left(\gamma^3+3\gamma^2|\ln\kappa|+3\gamma(\ln\kappa)^2+|\ln\kappa|^3\right)(\ln x)^3 W 0 3 ( x α κ ) ≤ ( γ 3 + 3 γ 2 ∣ ln κ ∣ + 3 γ ( ln κ ) 2 + ∣ ln κ ∣ 3 ) ( ln x ) 3
and
W 0 3 ( x α κ ) ( 3 W 0 ( x α κ ) + 4 ) ≤ ( 3 γ 4 + 12 γ 3 ∣ ln κ ∣ + 4 γ 3 + 18 γ 2 ( ln κ ) 2 + 12 γ 2 ∣ ln κ ∣ + 12 γ ∣ ln κ ∣ 3 + 12 γ ( ln κ ) 2 + 3 ( ln κ ) 4 + 4 ∣ ln κ ∣ 3 ) ( ln x ) 4 = ( 4 γ 3 + 3 γ 4 + 12 ( γ 2 + γ 3 ) ∣ ln κ ∣ + ( 12 γ + 18 γ 2 ) ( ln κ ) 2 + ( 4 + 12 γ ) ∣ ln κ ∣ 3 + 3 ( ln κ ) 4 ) ( ln x ) 4 . \begin{aligned}
W_0^3(x^\alpha\kappa)(3W_0(x^\alpha\kappa)+4)
&\le\left(3\gamma^4+12\gamma^3|\ln\kappa|+4\gamma^3+18\gamma^2(\ln\kappa)^2+12\gamma^2|\ln\kappa| \right.\\
&\quad\left.{}+12\gamma|\ln\kappa|^3+12\gamma(\ln\kappa)^2+3(\ln\kappa)^4+4|\ln\kappa|^3\right)(\ln x)^4\\
&= \left(4\gamma^3+3\gamma^4+12(\gamma^2+\gamma^3)|\ln\kappa|+(12\gamma+18\gamma^2)(\ln\kappa)^2\right.\\
&\quad\left.{}+(4+12\gamma)|\ln\kappa|^3+3(\ln\kappa)^4\right)(\ln x)^4.
\end{aligned} W 0 3 ( x α κ ) ( 3 W 0 ( x α κ ) + 4 ) ≤ ( 3 γ 4 + 12 γ 3 ∣ ln κ ∣ + 4 γ 3 + 18 γ 2 ( ln κ ) 2 + 12 γ 2 ∣ ln κ ∣ + 12 γ ∣ ln κ ∣ 3 + 12 γ ( ln κ ) 2 + 3 ( ln κ ) 4 + 4∣ ln κ ∣ 3 ) ( ln x ) 4 = ( 4 γ 3 + 3 γ 4 + 12 ( γ 2 + γ 3 ) ∣ ln κ ∣ + ( 12 γ + 18 γ 2 ) ( ln κ ) 2 + ( 4 + 12 γ ) ∣ ln κ ∣ 3 + 3 ( ln κ ) 4 ) ( ln x ) 4 .
This concludes the proof.
Lemma I.4. (Surface measure of Euclidean spherical caps). Let y ^ ∈ S p \hat y \in\mathbb{S}^p y ^ ∈ S p and ϵ > 0 \epsilon> 0 ϵ > 0 . Define the Euclidean spherical cap of radius ϵ \epsilon ϵ centered at y ^ \hat y y ^ as the Euclidean ball of radius ϵ \epsilon ϵ centered at y ^ \hat y y ^ intersected with the sphere: B S p ( y ^ , ϵ ) : = { x ^ ∈ S p ∣ ∥ x ^ − y ^ ∥ ≤ ϵ } B_{\mathbb{S}^p}(\hat y,\epsilon):=\{\hat x\in\mathbb{S}^p\mid\|\hat x-\hat y\|\le\epsilon\} B S p ( y ^ , ϵ ) := { x ^ ∈ S p ∣ ∥ x ^ − y ^ ∥ ≤ ϵ } . Then, it holds that σ ( B S p ( y ^ , ϵ ) ) ≤ v p ( π 2 ϵ ) p \sigma(B_{\mathbb{S}^p}(\hat y,\epsilon))\le v_p\left(\frac{\pi}{2}\epsilon\right)^p σ ( B S p ( y ^ , ϵ )) ≤ v p ( 2 π ϵ ) p , where σ \sigma σ is the surface measure on S p \mathbb{S}^p S p and v p = Vol ( B ( 0 , 1 ) ) v_p=\operatorname{Vol}(B(0,1)) v p = Vol ( B ( 0 , 1 )) is the volume of the unit ball in R p \mathbb{R}^p R p .
Proof. We denote by d g d_g d g the geodesic distance and fix x ^ , y ^ ∈ S p \hat x,\hat y\in\mathbb{S}^p x ^ , y ^ ∈ S p . On the unit sphere S p \mathbb{S}^p S p , we have d g ( x ^ , y ^ ) = θ d_g(\hat x,\hat y)=\theta d g ( x ^ , y ^ ) = θ , where θ ∈ [ 0 , π ] \theta\in[0,\pi] θ ∈ [ 0 , π ] is the angle between x ^ \hat x x ^ and y ^ \hat y y ^ . Since ∥ x ^ ∥ = ∥ y ^ ∥ = 1 \|\hat x\|=\|\hat y\|=1 ∥ x ^ ∥ = ∥ y ^ ∥ = 1 , it follows that:
∥ x ^ − y ^ ∥ 2 = ∥ x ^ ∥ 2 + ∥ y ^ ∥ 2 − 2 ⟨ x ^ , y ^ ⟩ = 2 − 2 cos θ = 4 sin 2 ( θ 2 ) . \|\hat x-\hat y\|^2=\|\hat x\|^2+\|\hat y\|^2-2\langle\hat x,\hat y\rangle=2-2\cos\theta=4\sin^2\left(\frac{\theta}{2}\right). ∥ x ^ − y ^ ∥ 2 = ∥ x ^ ∥ 2 + ∥ y ^ ∥ 2 − 2 ⟨ x ^ , y ^ ⟩ = 2 − 2 cos θ = 4 sin 2 ( 2 θ ) .
where we use the identity 1 − cos θ = 2 sin 2 ( θ 2 ) 1-\cos\theta=2\sin^2\left(\frac{\theta}{2}\right) 1 − cos θ = 2 sin 2 ( 2 θ ) in the last equality. Hence, we obtain ∥ x ^ − y ^ ∥ = 2 sin ( θ 2 ) \|\hat x-\hat y\|=2\sin\left(\frac{\theta}{2}\right) ∥ x ^ − y ^ ∥ = 2 sin ( 2 θ ) . Solving for θ \theta θ gives d g ( x ^ , y ^ ) : = θ = 2 arcsin ( ∥ x ^ − y ^ ∥ 2 ) d_g(\hat x,\hat y):=\theta=2\arcsin\left(\frac{\|\hat x-\hat y\|}{2}\right) d g ( x ^ , y ^ ) := θ = 2 arcsin ( 2 ∥ x ^ − y ^ ∥ ) . Jordan’s inequality states that sin θ ≥ 2 θ π \sin\theta\ge\frac{2\theta}{\pi} sin θ ≥ π 2 θ for θ ∈ [ 0 , π 2 ] \theta\in[0,\frac{\pi}{2}] θ ∈ [ 0 , 2 π ] , or equivalently arcsin ( t ) ≤ π t 2 \arcsin(t)\le\frac{\pi t}{2} arcsin ( t ) ≤ 2 π t for t ∈ [ 0 , 1 ] t\in[0,1] t ∈ [ 0 , 1 ] . Since ∥ x ^ − y ^ ∥ 2 ∈ [ 0 , 1 ] \frac{\|\hat x-\hat y\|}{2}\in[0,1] 2 ∥ x ^ − y ^ ∥ ∈ [ 0 , 1 ] for any x ^ , y ^ ∈ S p \hat x,\hat y\in\mathbb{S}^p x ^ , y ^ ∈ S p , we obtain d g ( x ^ , y ^ ) ≤ π 2 ∥ x ^ − y ^ ∥ d_g(\hat x,\hat y)\le\frac{\pi}{2}\|\hat x-\hat y\| d g ( x ^ , y ^ ) ≤ 2 π ∥ x ^ − y ^ ∥ . Therefore, we get B S p ( y ^ , ϵ ) ⊂ B g ( y ^ , π 2 ϵ ) B_{\mathbb{S}^p}(\hat y,\epsilon)\subset B_g(\hat y,\frac{\pi}{2}\epsilon) B S p ( y ^ , ϵ ) ⊂ B g ( y ^ , 2 π ϵ ) , where B g ( y ^ , π 2 ϵ ) : = { x ^ ∈ S p ∣ d g ( x ^ , y ^ ) ≤ π 2 ϵ } B_g(\hat y,\frac{\pi}{2}\epsilon):=\{\hat x\in\mathbb{S}^p\mid d_g(\hat x,\hat y)\le\frac{\pi}{2}\epsilon\} B g ( y ^ , 2 π ϵ ) := { x ^ ∈ S p ∣ d g ( x ^ , y ^ ) ≤ 2 π ϵ } is the geodesic ball on the sphere of radius π 2 ϵ \frac{\pi}{2}\epsilon 2 π ϵ and centered at y ^ \hat y y ^ . The surface measure of a geodesic ball of radius r r r on S p \mathbb{S}^p S p satisfies [61 ] :
σ ( B g ( y ^ , r ) ) = w p − 1 ∫ 0 r sin p − 1 t d t ≤ w p − 1 ∫ 0 r t p − 1 d t = w p − 1 p r p , \sigma(B_g(\hat y,r))=w_{p-1}\int_0^r\sin^{p-1}t\,dt\le w_{p-1}\int_0^r t^{p-1}\,dt=\frac{w_{p-1}}{p}r^p, σ ( B g ( y ^ , r )) = w p − 1 ∫ 0 r sin p − 1 t d t ≤ w p − 1 ∫ 0 r t p − 1 d t = p w p − 1 r p ,
where we use the fact that sin t ≤ t \sin t\le t sin t ≤ t for t ≥ 0 t\ge0 t ≥ 0 , with w p − 1 w_{p-1} w p − 1 being the surface area of S p − 1 \mathbb{S}^{p-1} S p − 1 . Additionally, we have that w p − 1 = 2 π p / 2 Γ ( p / 2 ) w_{p-1}=\frac{2\pi^{p/2}}{\Gamma(p/2)} w p − 1 = Γ ( p /2 ) 2 π p /2 and v p = π p / 2 Γ ( p / 2 + 1 ) v_p=\frac{\pi^{p/2}}{\Gamma(p/2+1)} v p = Γ ( p /2 + 1 ) π p /2 , where v p v_p v p is the volume of the unit ball in R p \mathbb{R}^p R p and Γ \Gamma Γ is the gamma function. Using the identity Γ ( x + 1 ) = x Γ ( x ) \Gamma(x+1)=x\Gamma(x) Γ ( x + 1 ) = x Γ ( x ) , we obtain:
v p = π p / 2 ( p / 2 ) Γ ( p / 2 ) = w p − 1 p , v_p=\frac{\pi^{p/2}}{(p/2)\Gamma(p/2)}=\frac{w_{p-1}}{p}, v p = ( p /2 ) Γ ( p /2 ) π p /2 = p w p − 1 ,
which yields σ ( B g ( y ^ , r ) ) ≤ v p r p \sigma(B_g(\hat y,r))\le v_p r^p σ ( B g ( y ^ , r )) ≤ v p r p . Using this inequality and the already established inclusion B S p ( y ^ , ϵ ) ⊂ B g ( y ^ , π 2 ϵ ) B_{\mathbb{S}^p}(\hat y,\epsilon)\subset B_g(\hat y,\frac{\pi}{2}\epsilon) B S p ( y ^ , ϵ ) ⊂ B g ( y ^ , 2 π ϵ ) , we get:
σ ( B S p ( y ^ , ϵ ) ) ≤ σ ( B g ( y ^ , π 2 ϵ ) ) ≤ v p ( π 2 ϵ ) p . \sigma(B_{\mathbb{S}^p}(\hat y,\epsilon))\le\sigma\left(B_g\left(\hat y,\frac{\pi}{2}\epsilon\right)\right)\le v_p\left(\frac{\pi}{2}\epsilon\right)^p. σ ( B S p ( y ^ , ϵ )) ≤ σ ( B g ( y ^ , 2 π ϵ ) ) ≤ v p ( 2 π ϵ ) p .
This concludes the proof.
Lemma I.5. Let K ∈ R n × n K\in\mathbb{R}^{n\times n} K ∈ R n × n be a symmetric positive definite matrix. For any t ≥ 0 t\ge0 t ≥ 0 , it holds that:
∥ exp ( − 1 n t K ) ∥ o p = exp ( − 1 n λ 0 t ) , \left\|\exp\left(-\frac{1}{n}tK\right)\right\|_{\mathrm{op}}=\exp\left(-\frac{1}{n}\lambda_0t\right), exp ( − n 1 t K ) op = exp ( − n 1 λ 0 t ) , where λ 0 \lambda_0 λ 0 is the smallest eigenvalue of K K K .
Proof. Since K K K is a symmetric positive definite matrix, it can be diagonalized as follows K = Q Λ Q ⊤ K = Q\Lambda Q^\top K = Q Λ Q ⊤ , where Q Q Q is orthogonal (i.e., Q − 1 = Q ⊤ Q^{-1} = Q^\top Q − 1 = Q ⊤ ), Λ = diag ( λ 1 , … , λ n ) \Lambda= \operatorname{diag}(\lambda_1,\ldots,\lambda_n) Λ = diag ( λ 1 , … , λ n ) , and λ 1 , … , λ n > 0 \lambda_1,\ldots,\lambda_n > 0 λ 1 , … , λ n > 0 are the eigenvalues of K K K . Then, we have:
exp ( − 1 n t K ) = exp ( − 1 n t Q Λ Q ⊤ ) = ∑ k = 0 + ∞ ( − 1 n t Q Λ Q ⊤ ) k k ! = Q ( ∑ k = 0 + ∞ ( − 1 n t Λ ) k k ! ) Q ⊤ = Q exp ( − 1 n t Λ ) Q ⊤ = Q diag ( exp ( − 1 n λ 1 t ) , … , exp ( − 1 n λ n t ) ) Q ⊤ . \begin{aligned}
\exp\left(-\frac{1}{n}tK\right)=\exp\left(-\frac{1}{n}tQ\Lambda Q^\top\right)=\sum_{k=0}^{+\infty}\frac{\left(-\frac{1}{n}tQ\Lambda Q^\top\right)^k}{k!}=Q\left(\sum_{k=0}^{+\infty}\frac{\left(-\frac{1}{n}t\Lambda\right)^k}{k!}\right)Q^\top\\
&=Q\exp\left(-\frac{1}{n}t\Lambda\right)Q^\top=Q\operatorname{diag}\left(\exp\left(-\frac{1}{n}\lambda_1t\right),\ldots,\exp\left(-\frac{1}{n}\lambda_nt\right)\right)Q^\top.
\end{aligned} exp ( − n 1 t K ) = exp ( − n 1 tQ Λ Q ⊤ ) = k = 0 ∑ + ∞ k ! ( − n 1 tQ Λ Q ⊤ ) k = Q ( k = 0 ∑ + ∞ k ! ( − n 1 t Λ ) k ) Q ⊤ = Q exp ( − n 1 t Λ ) Q ⊤ = Q diag ( exp ( − n 1 λ 1 t ) , … , exp ( − n 1 λ n t ) ) Q ⊤ .
Consequently, we get:
∥ exp ( − 1 n t K ) ∥ o p = max 1 ≤ i ≤ n exp ( − 1 n λ i t ) = exp ( − 1 n λ 0 t ) , \left\|\exp\left(-\frac{1}{n}tK\right)\right\|_{\mathrm{op}}=\max_{1\le i\le n}\exp\left(-\frac{1}{n}\lambda_it\right)=\exp\left(-\frac{1}{n}\lambda_0t\right), exp ( − n 1 t K ) op = 1 ≤ i ≤ n max exp ( − n 1 λ i t ) = exp ( − n 1 λ 0 t ) ,
where λ 0 : = min ( λ 1 , … , λ n ) \lambda_0 := \min(\lambda_1,\ldots,\lambda_n) λ 0 := min ( λ 1 , … , λ n ) is the smallest eigenvalue of K K K . This concludes the proof.
Lemma I.6. For any m ≥ 1 m \ge1 m ≥ 1 and α , β > 0 \alpha,\beta> 0 α , β > 0 , it holds that:
ln m ≤ m α α , β ln m − m α ≤ β α ( ln ( β α ) − 1 ) . \ln m\le\frac{m^\alpha}{\alpha},\qquad\beta\ln m-m^\alpha\le\frac{\beta}{\alpha}\left(\ln\left(\frac{\beta}{\alpha}\right)-1\right). ln m ≤ α m α , β ln m − m α ≤ α β ( ln ( α β ) − 1 ) . Proof. First inequality. Let g ( m ) = m α α − ln m g(m)=\frac{m^\alpha}{\alpha}-\ln m g ( m ) = α m α − ln m for m > 0 m>0 m > 0 . We have:
g ′ ( m ) = m α − 1 − 1 m = m α − 1 m . g'(m)=m^{\alpha-1}-\frac{1}{m}=\frac{m^\alpha-1}{m}. g ′ ( m ) = m α − 1 − m 1 = m m α − 1 . Since 1 m > 0 \frac{1}{m}>0 m 1 > 0 , the sign of g ′ g' g ′ is determined by m α − 1 m^\alpha-1 m α − 1 : g ′ ( m ) < 0 g'(m)<0 g ′ ( m ) < 0 for m < 1 m<1 m < 1 and g ′ ( m ) > 0 g'(m)>0 g ′ ( m ) > 0 for m > 1 m>1 m > 1 , so m ⋆ = 1 m^\star=1 m ⋆ = 1 is a global minimum of g g g on ( 0 , + ∞ ) (0,+\infty) ( 0 , + ∞ ) . Since g ( 1 ) = 1 α ≥ 0 g(1)=\frac{1}{\alpha}\ge0 g ( 1 ) = α 1 ≥ 0 , we have g ( m ) ≥ 1 α g(m)\ge\frac{1}{\alpha} g ( m ) ≥ α 1 for all m > 0 m>0 m > 0 , which gives:
ln m ≤ m α − 1 α ≤ m α α . \ln m\le\frac{m^\alpha-1}{\alpha}\le\frac{m^\alpha}{\alpha}. ln m ≤ α m α − 1 ≤ α m α . In particular, this holds for all m ≥ 1 m\ge1 m ≥ 1 .
Second inequality. Let h ( m ) = β ln m − m α h(m)=\beta\ln m-m^\alpha h ( m ) = β ln m − m α for m > 0 m>0 m > 0 . We have:
h ′ ( m ) = β m − α m α − 1 = β − α m α m . h'(m)=\frac{\beta}{m}-\alpha m^{\alpha-1}=\frac{\beta-\alpha m^\alpha}{m}. h ′ ( m ) = m β − α m α − 1 = m β − α m α . Since 1 m > 0 \frac{1}{m}>0 m 1 > 0 , the sign of h ′ h' h ′ is determined by β − α m α \beta-\alpha m^\alpha β − α m α : h ′ ( m ) > 0 h'(m)>0 h ′ ( m ) > 0 for m < m ⋆ m<m^\star m < m ⋆ and h ′ ( m ) < 0 h'(m)<0 h ′ ( m ) < 0 for m > m ⋆ m>m^\star m > m ⋆ , where m ⋆ = ( β α ) 1 α m^\star=\left(\frac{\beta}{\alpha}\right)^{\frac{1}{\alpha}} m ⋆ = ( α β ) α 1 . Hence m ⋆ m^\star m ⋆ is a global maximum of h h h on ( 0 , + ∞ ) (0,+\infty) ( 0 , + ∞ ) , so for all m ≥ 1 m\ge1 m ≥ 1 :
β ln m − m α ≤ h ( m ⋆ ) = β ⋅ 1 α ln β α − β α = β α ( ln ( β α ) − 1 ) . \beta\ln m-m^\alpha\le h(m^\star)=\beta\cdot\frac{1}{\alpha}\ln\frac{\beta}{\alpha}-\frac{\beta}{\alpha}=\frac{\beta}{\alpha}\left(\ln\left(\frac{\beta}{\alpha}\right)-1\right). β ln m − m α ≤ h ( m ⋆ ) = β ⋅ α 1 ln α β − α β = α β ( ln ( α β ) − 1 ) .
Lemma I.7. Let p ≥ 1 p\ge1 p ≥ 1 , 0 < ϱ ≤ 1 0<\varrho\le1 0 < ϱ ≤ 1 , τ > 1 2 \tau>\frac{1}{2} τ > 2 1 , and δ ∈ ( 0 , 1 ) \delta\in(0,1) δ ∈ ( 0 , 1 ) . Then, it holds that m p ϱ exp ( − 2 m 2 τ − 1 ) ≤ δ m^{p\varrho}\exp(-2m^{2\tau-1})\le\delta m pϱ exp ( − 2 m 2 τ − 1 ) ≤ δ , for every m ≥ max ( 1 , M ) m\ge\max(1,M) m ≥ max ( 1 , M ) , where:
M : = ( [ p ϱ 2 τ − 1 ( ln ( p ϱ 2 τ − 1 ) − 1 ) + ln ( 1 δ ) ] + ) 1 2 τ − 1 M:=\left(\left[\frac{p\varrho}{2\tau-1}\left(\ln\left(\frac{p\varrho}{2\tau-1}\right)-1\right)+\ln\left(\frac{1}{\delta}\right)\right]_+\right)^{\frac{1}{2\tau-1}} M := ( [ 2 τ − 1 pϱ ( ln ( 2 τ − 1 pϱ ) − 1 ) + ln ( δ 1 ) ] + ) 2 τ − 1 1 with [ t ] + : = max ( t , 0 ) [t]_+:=\max(t,0) [ t ] + := max ( t , 0 ) .
Proof. Set α : = 2 τ − 1 > 0 \alpha:=2\tau-1>0 α := 2 τ − 1 > 0 , β : = p ϱ > 0 \beta:=p\varrho>0 β := pϱ > 0 and u : = β α u:=\frac{\beta}{\alpha} u := α β . Taking logarithms, the desired inequality is equivalent to:
p ϱ ln m − 2 m 2 τ − 1 ≤ ln δ . (76) p\varrho\ln m-2m^{2\tau-1}\le\ln\delta. \tag*{(76)} pϱ ln m − 2 m 2 τ − 1 ≤ ln δ . (76) By Lemma I.6, we have β ln m − m α ≤ u ( ln u − 1 ) \beta\ln m-m^\alpha\le u(\ln u-1) β ln m − m α ≤ u ( ln u − 1 ) for any m ≥ 1 m\ge1 m ≥ 1 , hence:
p ϱ ln m − 2 m 2 τ − 1 = ( β ln m − m α ) − m 2 τ − 1 ≤ u ( ln u − 1 ) − m 2 τ − 1 . p\varrho\ln m-2m^{2\tau-1}=(\beta\ln m-m^\alpha)-m^{2\tau-1}\le u(\ln u-1)-m^{2\tau-1}. pϱ ln m − 2 m 2 τ − 1 = ( β ln m − m α ) − m 2 τ − 1 ≤ u ( ln u − 1 ) − m 2 τ − 1 . It is therefore sufficient that:
m 2 τ − 1 ≥ u ( ln u − 1 ) + ln ( 1 δ ) . (77) m^{2\tau-1}\ge u(\ln u-1)+\ln\left(\frac{1}{\delta}\right). \tag*{(77)} m 2 τ − 1 ≥ u ( ln u − 1 ) + ln ( δ 1 ) . (77) If the right-hand side of Equation (77) is nonpositive, then Equation (77) holds for every m ≥ 1 m\ge1 m ≥ 1 , since m 2 τ − 1 ≥ 1 m^{2\tau-1}\ge1 m 2 τ − 1 ≥ 1 . Otherwise, Equation (77) is equivalent to m ≥ M m\ge M m ≥ M . In both situations, m ≥ max ( 1 , M ) m\ge\max(1,M) m ≥ max ( 1 , M ) implies Equation (77) , hence Equation (76) , which concludes the proof.
Lemma I.8. Let n ≥ 3 n \ge3 n ≥ 3 , a ≥ 1 a \ge1 a ≥ 1 , 0 < b ≤ 1 0 < b \le1 0 < b ≤ 1 , and define the function f : ( 1 , + ∞ ) → R f : (1,+\infty) \to\mathbb{R} f : ( 1 , + ∞ ) → R by
f ( x ) = x − 1 a ( ln x ) b . f(x)=x^{-\frac{1}{a}}(\ln x)^b. f ( x ) = x − a 1 ( ln x ) b . Then, for all x ≥ n a ( ln n ) a b x \ge n^a(\ln n)^{ab} x ≥ n a ( ln n ) ab , it holds that f ( x ) ≤ a b ( 1 + b ) b n − 1 f(x) \le a^b(1+b)^b n^{-1} f ( x ) ≤ a b ( 1 + b ) b n − 1 .
Proof. We first study the monotonicity of f f f . For x > 1 x>1 x > 1 , we have by the product rule:
f ′ ( x ) = − 1 a x − 1 a − 1 ( ln x ) b + x − 1 a b ( ln x ) b − 1 1 x = x − 1 a − 1 ( ln x ) b − 1 ( − 1 a ln x + b ) . f'(x)=-\frac{1}{a}x^{-\frac{1}{a}-1}(\ln x)^b+x^{-\frac{1}{a}}b(\ln x)^{b-1}\frac{1}{x}=x^{-\frac{1}{a}-1}(\ln x)^{b-1}\left(-\frac{1}{a}\ln x+b\right). f ′ ( x ) = − a 1 x − a 1 − 1 ( ln x ) b + x − a 1 b ( ln x ) b − 1 x 1 = x − a 1 − 1 ( ln x ) b − 1 ( − a 1 ln x + b ) . Since x − 1 a − 1 ( ln x ) b − 1 > 0 x^{-\frac{1}{a}-1}(\ln x)^{b-1}>0 x − a 1 − 1 ( ln x ) b − 1 > 0 for x > 1 x>1 x > 1 , the sign of f ′ f' f ′ is entirely determined by − 1 a ln x + b -\frac{1}{a}\ln x+b − a 1 ln x + b . Hence, f ′ ( x ) < 0 f'(x)<0 f ′ ( x ) < 0 whenever x > e a b x>e^{ab} x > e ab . Therefore, f f f is decreasing on ( e a b , + ∞ ) (e^{ab},+\infty) ( e ab , + ∞ ) . Let c = n a ( ln n ) a b c=n^a(\ln n)^{ab} c = n a ( ln n ) ab where n ≥ 3 n\ge3 n ≥ 3 . We have ln c = a ln n + a b ln ln n > a b \ln c=a\ln n+ab\ln\ln n>ab ln c = a ln n + ab ln ln n > ab , so c > e a b c>e^{ab} c > e ab . Thus, for all x ≥ c x\ge c x ≥ c , we have f ( x ) ≤ f ( c ) f(x)\le f(c) f ( x ) ≤ f ( c ) . We compute:
f ( c ) = n − 1 ( ln n ) − b a b b ( a ln n + a b ln ln n ) b = n − 1 ( ln n ) − b a b ( ln n + b ln ln n ) b f(c)=n^{-1}(\ln n)^{-b^a b^b}(a\ln n+ab\ln\ln n)^b=n^{-1}(\ln n)^{-b}a^b(\ln n+b\ln\ln n)^b f ( c ) = n − 1 ( ln n ) − b a b b ( a ln n + ab ln ln n ) b = n − 1 ( ln n ) − b a b ( ln n + b ln ln n ) b ≤ n − 1 ( ln n ) − b a b ( ln n + b ( ln n − 1 ) ) b ≤ n − 1 ( ln n ) − b a b ( ( 1 + b ) ln n ) b \le n^{-1}(\ln n)^{-b}a^b(\ln n+b(\ln n-1))^b\le n^{-1}(\ln n)^{-b}a^b((1+b)\ln n)^b ≤ n − 1 ( ln n ) − b a b ( ln n + b ( ln n − 1 ) ) b ≤ n − 1 ( ln n ) − b a b (( 1 + b ) ln n ) b = n − 1 a b ( 1 + b ) b , =n^{-1}a^b(1+b)^b, = n − 1 a b ( 1 + b ) b , where in the second line we use ln ln n ≤ ln n − 1 \ln\ln n\le\ln n-1 ln ln n ≤ ln n − 1 for n > 1 n>1 n > 1 . Therefore, f ( x ) ≤ a b ( 1 + b ) b n − 1 f(x)\le a^b(1+b)^b n^{-1} f ( x ) ≤ a b ( 1 + b ) b n − 1 for all x ≥ n a ( ln n ) a b x\ge n^a(\ln n)^{ab} x ≥ n a ( ln n ) ab , which concludes the proof.
Lemma I.9. Let C 1 , C 2 > 0 C_1,C_2>0 C 1 , C 2 > 0 . When m ≥ max ( ( C 1 12 C 2 ) 24 , 1 ) m\ge\max\left(\left(\frac{C_1\sqrt{12}}{C_2}\right)^{24},1\right) m ≥ max ( ( C 2 C 1 12 ) 24 , 1 ) , we have C 1 m − 1 12 ln m ≤ C 2 C_1m^{-\frac{1}{12}}\sqrt{\ln m}\le C_2 C 1 m − 12 1 ln m ≤ C 2 .
Proof. When m ≥ ( C 1 12 C 2 ) 24 m\ge\left(\frac{C_1\sqrt{12}}{C_2}\right)^{24} m ≥ ( C 2 C 1 12 ) 24 , we get C 1 12 m − 1 24 ≤ C 2 C_1\sqrt{12}m^{-\frac{1}{24}}\le C_2 C 1 12 m − 24 1 ≤ C 2 . By Lemma I.6 , we have ln m ≤ 12 m 1 12 \ln m\le12m^{\frac{1}{12}} ln m ≤ 12 m 12 1 when m ≥ 1 m\ge1 m ≥ 1 . Thus, combining both inequalities, when m ≥ max ( ( C 1 12 C 2 ) 24 , 1 ) m\ge\max\left(\left(\frac{C_1\sqrt{12}}{C_2}\right)^{24},1\right) m ≥ max ( ( C 2 C 1 12 ) 24 , 1 ) , we obtain
C 1 m − 1 12 ln m ≤ C 1 m − 1 12 12 m 1 24 = C 1 12 m − 1 24 ≤ C 2 . C_1m^{-\frac{1}{12}}\sqrt{\ln m}\le C_1m^{-\frac{1}{12}}\sqrt{12}m^{\frac{1}{24}}=C_1\sqrt{12}m^{-\frac{1}{24}}\le C_2. C 1 m − 12 1 ln m ≤ C 1 m − 12 1 12 m 24 1 = C 1 12 m − 24 1 ≤ C 2 . This concludes the proof.
Lemma I.10. Let 0 < α < β ≤ 1 0<\alpha<\beta\le1 0 < α < β ≤ 1 , δ > 1 2 \delta>\frac{1}{2} δ > 2 1 , γ > max ( 1 − α , δ ) \gamma>\max(1-\alpha,\delta) γ > max ( 1 − α , δ ) , B > 0 B>0 B > 0 , d ≥ 1 d\ge1 d ≥ 1 , and c 0 > 0 c_0>0 c 0 > 0 be an absolute constant. The following statements hold:
(i) When m ≥ 3 1 γ − max ( 1 − α , δ ) m\ge3^{\frac{1}{\gamma-\max(1-\alpha,\delta)}} m ≥ 3 γ − m a x ( 1 − α , δ ) 1 , it holds that m 1 − α + m δ ≤ ⌊ m γ ⌋ m^{1-\alpha}+m^\delta\le\lfloor m^\gamma\rfloor m 1 − α + m δ ≤ ⌊ m γ ⌋ .
(ii) When m ≥ max ( 2 1 β , ( 128 B 2 d 2 π ( β − α ) ) 1 β − α ) m\ge\max\left(2^{\frac{1}{\beta}},\left(\frac{128B^2d^2}{\pi(\beta-\alpha)}\right)^{\frac{1}{\beta-\alpha}}\right) m ≥ max ( 2 β 1 , ( π ( β − α ) 128 B 2 d 2 ) β − α 1 ) , it holds that 2 B d ln m ⌊ m β ⌋ ≤ 2 π 4 m − α \frac{2Bd\sqrt{\ln m}}{\lfloor m^\beta\rfloor}\le\frac{\sqrt{2\pi}}{4}m^{-\alpha} ⌊ m β ⌋ 2 B d l n m ≤ 4 2 π m − α .
(iii) When m ≥ max ( 36 d 2 β 2 c 0 2 , 1 ) m\ge\max\left(\frac{36d^2\beta^2}{c_0^2},1\right) m ≥ max ( c 0 2 36 d 2 β 2 , 1 ) , it holds that 3 d β c 0 ln m ≤ m \frac{3d\beta}{c_0}\ln m\le m c 0 3 d β ln m ≤ m .
Proof. Item (i). Since m ≥ 1 m\ge1 m ≥ 1 , we have 1 ≤ m max ( 1 − α , δ ) 1\le m^{\max(1-\alpha,\delta)} 1 ≤ m m a x ( 1 − α , δ ) , and therefore:
m 1 − α + m δ ≤ 2 m max ( 1 − α , δ ) = ( 2 m max ( 1 − α , δ ) + 1 ) − 1 ≤ 3 m max ( 1 − α , δ ) − 1. m^{1-\alpha}+m^\delta\le2m^{\max(1-\alpha,\delta)}=(2m^{\max(1-\alpha,\delta)}+1)-1\le3m^{\max(1-\alpha,\delta)}-1. m 1 − α + m δ ≤ 2 m m a x ( 1 − α , δ ) = ( 2 m m a x ( 1 − α , δ ) + 1 ) − 1 ≤ 3 m m a x ( 1 − α , δ ) − 1. Moreover, since m ≥ 3 1 γ − max ( 1 − α , δ ) m\ge3^{\frac{1}{\gamma-\max(1-\alpha,\delta)}} m ≥ 3 γ − m a x ( 1 − α , δ ) 1 , we have m γ − max ( 1 − α , δ ) ≥ 3 m^{\gamma-\max(1-\alpha,\delta)}\ge3 m γ − m a x ( 1 − α , δ ) ≥ 3 , which gives 3 m max ( 1 − α , δ ) ≤ m γ 3m^{\max(1-\alpha,\delta)}\le m^\gamma 3 m m a x ( 1 − α , δ ) ≤ m γ . Combining with the standard inequality ⌊ m γ ⌋ ≥ m γ − 1 \lfloor m^\gamma\rfloor\ge m^\gamma-1 ⌊ m γ ⌋ ≥ m γ − 1 , we conclude:
m 1 − α + m δ ≤ 3 m max ( 1 − α , δ ) − 1 ≤ m γ − 1 ≤ ⌊ m γ ⌋ . m^{1-\alpha}+m^\delta\le3m^{\max(1-\alpha,\delta)}-1\le m^\gamma-1\le\lfloor m^\gamma\rfloor. m 1 − α + m δ ≤ 3 m m a x ( 1 − α , δ ) − 1 ≤ m γ − 1 ≤ ⌊ m γ ⌋ . Item (ii). Since m ≥ 2 1 β m\ge2^{\frac{1}{\beta}} m ≥ 2 β 1 , we have m β 2 ≥ 1 \frac{m^\beta}{2}\ge1 2 m β ≥ 1 . This implies that m β 2 ≤ m β − 1 ≤ ⌊ m β ⌋ \frac{m^\beta}{2}\le m^\beta-1\le\lfloor m^\beta\rfloor 2 m β ≤ m β − 1 ≤ ⌊ m β ⌋ . Consequently, we obtain:
2 B d ln m ⌊ m β ⌋ ≤ 4 B d ln m m β ≤ 4 B d m − β + α 2 β − α , \frac{2Bd\sqrt{\ln m}}{\lfloor m^\beta\rfloor}\le\frac{4Bd\sqrt{\ln m}}{m^\beta}\le\frac{4Bdm^{-\frac{\beta+\alpha}{2}}}{\sqrt{\beta-\alpha}}, ⌊ m β ⌋ 2 B d ln m ≤ m β 4 B d ln m ≤ β − α 4 B d m − 2 β + α , where, in the second inequality, we use the identity ln m ≤ m β − α β − α \ln m\le\frac{m^{\beta-\alpha}}{\beta-\alpha} ln m ≤ β − α m β − α provided in Lemma I.6 . So, it suffices to have:
4 B d m − β + α 2 β − α ≤ 2 π 4 m − α ⟺ 16 B d 2 π ( β − α ) ≤ m β − α 2 , (78) \frac{4Bdm^{-\frac{\beta+\alpha}{2}}}{\sqrt{\beta-\alpha}}\le\frac{\sqrt{2\pi}}{4}m^{-\alpha}\Longleftrightarrow\frac{16Bd}{\sqrt{2\pi(\beta-\alpha)}}\le m^{\frac{\beta-\alpha}{2}}, \tag*{(78)} β − α 4 B d m − 2 β + α ≤ 4 2 π m − α ⟺ 2 π ( β − α ) 16 B d ≤ m 2 β − α , (78)
which is satisfied since m ≥ ( 128 B 2 d 2 π ( β − α ) ) 1 β − α m \ge\left(\frac{128B^2d^2}{\pi(\beta-\alpha)}\right)^{\frac{1}{\beta-\alpha}} m ≥ ( π ( β − α ) 128 B 2 d 2 ) β − α 1 .
Item (iii). By Lemma I.6, for any m ≥ 1 m \ge1 m ≥ 1 , we have ln m ≤ 2 m \ln m \le2\sqrt{m} ln m ≤ 2 m . Using this, we have:
3 d β c 0 ln m ≤ 6 d β c 0 m . \frac{3d\beta}{c_0}\ln m \le\frac{6d\beta}{c_0}\sqrt{m}. c 0 3 d β ln m ≤ c 0 6 d β m .
The result then follows since m ≥ 36 d 2 β 2 c 0 2 m \ge\frac{36d^2\beta^2}{c_0^2} m ≥ c 0 2 36 d 2 β 2 .
Further Details on Experiments and Additional Numerical Results
Our code is available at https://github.com/fareselkhoury/OBO .
Empirical validation of the theory in the NTK regime on synthetic data
Statistical model. We generate synthetic data as follows:
x ∼ Unif ( [ 0 , 1 ] 2 ) , z j = ∥ x − c j ∥ q h + ξ j ( j = 1 , … , 4 ) , y = ∥ x − x 0 ∥ q b + ξ y , x \sim\operatorname{Unif}([0,1]^2), \qquad z_j = \lVert x-c_j\rVert^{q_h}+\xi_j \ (j=1,\ldots,4), \qquad y=\lVert x-x_0\rVert^{q_b}+\xi_y, x ∼ Unif ([ 0 , 1 ] 2 ) , z j = ∥ x − c j ∥ q h + ξ j ( j = 1 , … , 4 ) , y = ∥ x − x 0 ∥ q b + ξ y ,
where ξ j , ξ y ∼ N ( 0 , τ 2 ) \xi_j,\xi_y \sim\mathcal{N}(0,\tau^2) ξ j , ξ y ∼ N ( 0 , τ 2 ) i.i.d. with τ = 0.01 \tau=0.01 τ = 0.01 , and the centers c 1 , … , c 4 c_1,\ldots,c_4 c 1 , … , c 4 and the point x 0 x_0 x 0 are as defined in the main text. x x x , ξ j \xi_j ξ j , and ξ y \xi_y ξ y are all redrawn independently at every sample. The feature map g ω g_\omega g ω is given by g ω ( z ) = ω ⊤ z g_\omega(z)=\omega^\top z g ω ( z ) = ω ⊤ z , where z : = [ z 1 , z 2 , z 3 , z 4 ] ⊤ z := [z_1,z_2,z_3,z_4]^\top z := [ z 1 , z 2 , z 3 , z 4 ] ⊤ . Since ξ j , ξ y \xi_j,\xi_y ξ j , ξ y are zero-mean and independent of x x x , we obtain h ω ⋆ ( x ) = E [ g ω ( z ) ∣ x ] = ψ ( x ) ⊤ ω h^\star_\omega(x)=\mathbb{E}[g_\omega(z)\mid x]=\psi(x)^\top\omega h ω ⋆ ( x ) = E [ g ω ( z ) ∣ x ] = ψ ( x ) ⊤ ω and b ⋆ ( x ) = E [ y ∣ x ] = ∥ x − x 0 ∥ q b b^\star(x)=\mathbb{E}[y\mid x]=\lVert x-x_0\rVert^{q_b} b ⋆ ( x ) = E [ y ∣ x ] = ∥ x − x 0 ∥ q b , where ψ ( x ) : = [ ∥ x − c 1 ∥ q h , … , ∥ x − c 4 ∥ q h ] ⊤ \psi(x):=[\lVert x-c_1\rVert^{q_h},\ldots,\lVert x-c_4\rVert^{q_h}]^\top ψ ( x ) := [∥ x − c 1 ∥ q h , … , ∥ x − c 4 ∥ q h ] ⊤ .
Setup. We use a width of m = 8,000 m=8{,}000 m = 8 , 000 for both networks approximating h ω ⋆ h^\star_\omega h ω ⋆ and b ⋆ b^\star b ⋆ , with input dimension p = 2 p=2 p = 2 and outer dimension d = 4 d=4 d = 4 . Both networks are trained by full-batch gradient descent at stepsize 1, for n s t e p s = max ( 1 , round ( t o p / 1.0 ) ) n_{\mathrm{steps}}=\max(1,\operatorname{round}(t_{\mathrm{op}}/1.0)) n steps = max ( 1 , round ( t op /1.0 )) steps, where t o p t_{\mathrm{op}} t op is the continuous-time stopping rule of Proposition 3.1. Because h ⋆ ω h^\omega_\star h ⋆ ω and b ⋆ b^\star b ⋆ are available in closed form here, we also compute the true hypergradient ∇ F ( ω k ) \nabla F(\omega_k) ∇ F ( ω k ) by a Monte Carlo estimate over 20,000 samples at every outer iteration, giving an accurate near-stationarity measure ∥ G 1 / L ( ω k ) ∥ \lVert G_1/\mathcal{L}(\omega_k)\rVert ∥ G 1 / L ( ω k )∥ . We estimate G = sup z ∥ z ∥ G=\sup_z\lVert z\rVert G = sup z ∥ z ∥ by the empirical maximum over 5,000 Monte Carlo samples, and use L = G 2 \mathcal{L}=G^2 L = G 2 exactly, since g ω ( z ) = ω ⊤ z g_\omega(z)=\omega^\top z g ω ( z ) = ω ⊤ z is linear in ω \omega ω and ∂ ω g ω ( z ) = z \partial_\omega g_\omega(z)=z ∂ ω g ω ( z ) = z does not depend on ω \omega ω . The projection set is Ω = { ω ∈ R d : ∥ ω ∥ ≤ 2 } \Omega=\{\omega\in\mathbb{R}^d:\lVert\omega\rVert\le2\} Ω = { ω ∈ R d : ∥ ω ∥ ≤ 2 } , the outer stepsize is η = 0.05 \eta=0.05 η = 0.05 , the outer loop runs for K = 200 K=200 K = 200 iterations, and the hypergradient batch size is n g = 200 n_g=200 n g = 200 . The outer parameter is initialized as ω 0 ∼ N ( 0 , 1 d 1 d × d ) \omega_0\sim\mathcal{N}(0,\frac{1}{d}\mathbf{1}_{d\times d}) ω 0 ∼ N ( 0 , d 1 1 d × d ) , so that E [ ∥ ω 0 ∥ 2 ] = 1 \mathbb{E}[\lVert\omega_0\rVert^2]=1 E [∥ ω 0 ∥ 2 ] = 1 regardless of d d d , well inside the projection set Ω \Omega Ω . For each smoothness level and each sample size, we repeat the estimation of h ω ⋆ h^\star_\omega h ω ⋆ and b ⋆ b^\star b ⋆ over 5 independent seeds and a held-out test set of 4,000 points, reporting the mean and using the standard error of the mean for the confidence intervals. The experiment is implemented using the PyTorch framework [75 ] and runs on a single NVIDIA RTX A5000 GPU. It takes approximately 3 hours to complete.
Insights into feature learning beyond the NTK regime
The NTK analysis is theoretically sound but often conservative , as it assumes fixed features and therefore precludes feature learning . Following [91 ] , a single network can transition from the NTK to a feature learning regime depending on initialization scale. The bilevel structure, however, introduces an additional degree of freedom : the interplay between inner and outer initializations may jointly govern whether the network operates in the NTK or feature learning regime, potentially inducing feature learning even when a single network would remain in the NTK regime. This raises a natural question: Could the bilevel structure exhibit a feature learning regime where the network adapts its features and exceeds NTK predictions? Our experiments explore this possibility.
Statistical model. We generate synthetic data as follows:
x ∼ N ( 0 , 1 p 1 p × p ) , z = 2 ( 1 p ⊤ x + ε ) 1 r + η , y = g ⋆ ( z ) + ϵ , x\sim\mathcal{N}\left(0,\frac{1}{p}\mathbf{1}_{p\times p}\right), \qquad z=2(\mathbf{1}_p^\top x+\varepsilon)\mathbf{1}_r+\eta, \qquad y=g^\star(z)+\epsilon, x ∼ N ( 0 , p 1 1 p × p ) , z = 2 ( 1 p ⊤ x + ε ) 1 r + η , y = g ⋆ ( z ) + ϵ ,
where ϵ ∼ N ( 0 , 0.01 2 ) \epsilon\sim\mathcal{N}(0,0.01^2) ϵ ∼ N ( 0 , 0.0 1 2 ) , η ∼ N ( 0 , 0.01 2 4 1 r × r ) \eta\sim\mathcal{N}\left(0,\frac{0.01^2}{4}\mathbf{1}_{r\times r}\right) η ∼ N ( 0 , 4 0.0 1 2 1 r × r ) , g ⋆ ( z ) = ϕ ( z ) ⊤ ω ⋆ g^\star(z)=\phi(z)^\top\omega^\star g ⋆ ( z ) = ϕ ( z ) ⊤ ω ⋆ , ϕ ( z ) = SiLU ( A z + b ) ∈ R d \phi(z)=\operatorname{SiLU}(Az+b)\in\mathbb{R}^d ϕ ( z ) = SiLU ( A z + b ) ∈ R d with A ∈ R d × r A\in\mathbb{R}^{d\times r} A ∈ R d × r having i.i.d. entries drawn from N ( 0 , 1 r ) \mathcal{N}(0,\frac{1}{r}) N ( 0 , r 1 ) , b ∼ N ( 0 , 10 − 4 1 d × d ) b\sim\mathcal{N}(0,10^{-4}\mathbf{1}_{d\times d}) b ∼ N ( 0 , 1 0 − 4 1 d × d ) , and ω ⋆ ∼ N ( 0 , 1 d 1 d × d ) \omega^\star\sim\mathcal{N}(0,\frac{1}{d}\mathbf{1}_{d\times d}) ω ⋆ ∼ N ( 0 , d 1 1 d × d ) , all drawn once at initialization and fixed throughout.
Setup. We consider the empirical inner and outer objectives defined in Equation (3) with g ω : z ↦ ϕ ( z ) ⊤ ω g_\omega: z \mapsto\phi(z)^\top\omega g ω : z ↦ ϕ ( z ) ⊤ ω , where ϕ ( z ) = SiLU ( A z + b ) ∈ R d \phi(z) = \operatorname{SiLU}(Az + b) \in\mathbb{R}^d ϕ ( z ) = SiLU ( A z + b ) ∈ R d is a fixed random feature map, SiLU ( x ) = x / ( 1 + exp ( − x ) ) \operatorname{SiLU}(x) = x/(1 + \exp(-x)) SiLU ( x ) = x / ( 1 + exp ( − x )) is applied entrywise, A ∈ R d × r A \in\mathbb{R}^{d \times r} A ∈ R d × r and b ∈ R d b \in\mathbb{R}^d b ∈ R d are drawn at initialization. We generate synthetic data with 100 training samples and 20 validation and test samples at each level, that remain fixed across all runs. Both h ω ∗ h_\omega^* h ω ∗ and b ∗ b^* b ∗ are approximated by neural networks h ω N N h_\omega^{\mathrm{NN}} h ω NN and b N N b^{\mathrm{NN}} b NN of width 500, of the form in Equation (5) without second-layer bias, normalized by m m m rather than m \sqrt{m} m to be consistent with the feature learning limit as m m m increases [26 ] . The first-layer parameters are initialized from a centered Gaussian of variance σ 0 2 \sigma_0^2 σ 0 2 for h ω N N h_\omega^{\mathrm{NN}} h ω NN and τ 0 2 \tau_0^2 τ 0 2 for b N N b^{\mathrm{NN}} b NN , while the second-layer weights are initialized to 0. Since b N N b^{\mathrm{NN}} b NN is independent of the outer variable, it is trained once at the beginning; h ω N N h_\omega^{\mathrm{NN}} h ω NN is retrained at every outer iteration with a warm-start strategy [90 ] . Both networks are trained with full-batch gradient descent until the training error falls below 10 − 15 10^{-15} 1 0 − 15 . Early stopping is applied based on validation loss, selecting the iterate with the lowest validation error. At the outer level, we use full-batch gradient descent for 5000 iterations, which is appropriate here because the sample size is small. We use the PyTorch framework [75 ] on an NVIDIA RTX 6000 ADA GPU. The experiments take around 10 hours to complete for each pair ( σ 0 2 , τ 0 2 ) (\sigma_0^2,\tau_0^2) ( σ 0 2 , τ 0 2 ) . We set d = 30 d = 30 d = 30 , p = 20 p = 20 p = 20 , and r = 10 r = 10 r = 10 . The stepsize to train the two neural networks is fixed at 10, whereas the stepsize for the outer optimization is 0.01. We vary the initialization variances σ 0 2 \sigma_0^2 σ 0 2 and τ 0 2 \tau_0^2 τ 0 2 in { 10 − 3 , 10 − 2 , 10 − 1 , 1 , 10 , 100 } \{10^{-3}, 10^{-2}, 10^{-1}, 1, 10, 100\} { 1 0 − 3 , 1 0 − 2 , 1 0 − 1 , 1 , 10 , 100 } independently, across training sizes and random seeds, to study the transition between the NTK and feature learning regimes.
Additional numerical results. Figure 3 shows the test outer loss over outer iterations for all combinations of ( σ 0 2 , τ 0 2 ) (\sigma_0^2,\tau_0^2) ( σ 0 2 , τ 0 2 ) . When both initializations are small, the network operates in the feature learning regime and converges to a low test loss. As either σ 0 2 \sigma_0^2 σ 0 2 or τ 0 2 \tau_0^2 τ 0 2 increases, the network transitions toward the NTK regime, with curves plateauing at higher loss values. This is most visible in the bottom-right panels, where both initializations are large and all curves converge to similarly high loss regardless of σ 0 2 \sigma_0^2 σ 0 2 , suggesting that large τ 0 2 \tau_0^2 τ 0 2 alone is sufficient to push the network into the NTK regime.
Figure 3. Test outer loss as a function of outer iterations for varying σ 0 2 \sigma_0^2 σ 0 2 and τ 0 2 \tau_0^2 τ 0 2 : small initializations (feature learning) converge to lower loss, while large initializations (NTK) plateau at higher values. The plots are averaged over 5 runs and displayed on a log scale. The line represents the mean across all runs, and the shaded region indicates the 95% confidence interval.
Figure 4 shows the final test outer loss as a function of the sample size for all combinations of ( σ 0 2 , τ 0 2 ) (\sigma_0^2,\tau_0^2) ( σ 0 2 , τ 0 2 ) . Three observations emerge. First, all curves decrease as the sample size grows, reflecting improved generalization with more data regardless of the regime. Second, the ranking between regimes is preserved at every sample size: when both initializations are small (feature learning), the network consistently achieves lower loss than when either initialization is large (NTK), confirming that the regime effect observed in Figure 2 is not a finite-sample artifact. Third, when τ 0 2 \tau_0^2 τ 0 2 is large, the curves collapse and become insensitive to σ 0 2 \sigma_0^2 σ 0 2 , further confirming that large τ 0 2 \tau_0^2 τ 0 2 alone is sufficient to push the network into the NTK regime, regardless of σ 0 2 \sigma_0^2 σ 0 2 .
Figure 4. Test outer loss at the last iteration as a function of the sample size for varying σ 0 2 \sigma_0^2 σ 0 2 and τ 0 2 \tau_0^2 τ 0 2 : all curves decrease with the sample size, with the ranking between feature learning and NTK regimes preserved throughout. The plots are averaged over 5 runs and displayed on a log scale. The line represents the mean across all runs, and the shaded region indicates the 95% confidence interval.
Figure 5 presents the same results as in Figure 2 in line plot form, showing the test inner and outer losses at the last outer iteration as a function of σ 0 2 \sigma_0^2 σ 0 2 for varying τ 0 2 \tau_0^2 τ 0 2 . The confidence intervals confirm that the ranking between regimes is stable across runs.
Figure 5. Test inner (left) and outer (right) losses at the last outer iteration as a function of σ 0 2 \sigma_{0}^{2} σ 0 2 for varying τ 0 2 \tau_{0}^{2} τ 0 2 : both losses are lowest when both initializations are small (feature learning regime) and degrade as either grows large (NTK regime). The plots are averaged over 5 runs and displayed on a log scale. The line represents the mean across all runs, and the shaded region indicates the 95% confidence interval.