We study direct and inverse approximation for linearized shallow ReLUk networks on Sd with quasi-uniform hidden parameters. For 0<α<2dd+2k+1, under the coefficient growth condition (n∑j=1naj,n2)21≤nαM, we prove that approximation at the order n−2dd+2k+1+α is equivalent to membership in the parity-compatible Besov space B2,∞,∗2d+2k+1−αd(Sd). Mere convergence under the same coefficient budget does not imply this regularity. For unconstrained approximation along a nested parameter sequence, a single ridge function gives a counterexample when α<21, whereas for α>21 the endpoint Besov inverse conclusion follows from a Bernstein inequality under antipodal quasi-uniformity. The inverse conclusion is also recovered when the prescribed rate holds for two quasi-uniform parameter sequences whose union is antipodally separated.
Introduction
The approximation theory of shallow neural networks seeks to quantify how the approximation error depends on the network width, the activation function, the regularity of the target, and the size of the representing coefficients. For Barron and variation-type spaces, dimension-independent rates are commonly obtained through probabilistic or greedy discretization arguments [20, 2, 3, 12, 1, 21, 40, 42, 41, 27]. For Sobolev classes, the rate instead reflects both smoothness and dimension [29, 37, 38]. These direct estimates give sufficient conditions for attaining a prescribed rate. The converse question—what regularity is forced by that rate—is subtler and is closely related to Bernstein inequalities, inverse approximation theorems, and the structure of neural-network approximation classes [42, 43, 17, 14, 4, 18].
We focus on the ReLUk activation σk(t)=(max{0,t})k, k∈N0, with the usual Heaviside convention when k=0, and on linearized shallow networks, in which the hidden parameters are fixed and only the output coefficients vary. By the positive homogeneity of σk, the parameter variables may be normalized to directions on the unit sphere. We consider the approximation problem on
Sd={η∈Rd+1:∣η∣=1},
where the basic ridge functions take the form σk(θ⋅η) with θ,η∈Sd.
The spherical setting is useful both as a normalized model for affine ReLUk networks and as an approximation problem in its own right. Spherical neural networks arise naturally for data with directional or rotational structure [7, 19, 9], and their approximation and generalization properties have been studied in [15, 16, 44]. Harmonic analysis on the sphere also provides localized polynomial kernels, cubature formulas, and stable discretizations on quasi-uniform point sets [28, 30, 31, 32, 33, 34]. These tools connect the geometry of the hidden parameters with approximation accuracy and coefficient stability.
We write Lnk(Θn) when no constraint is imposed on the coefficients. The constant M is always independent of n; in a direct approximation statement for a given f, it may depend on f. Schwartz inequality shows the coefficient bound of the Barron space [43] is consistent with the constraint in (1) when α=0:
j=1∑n∣aj,n∣≤(nj=1∑naj,n2)21≤M
The identity
σk(θ⋅η)+(−1)kσk(−θ⋅η)=(θ⋅η)k
shows that every element of Lnk(Θn) has a prescribed parity up to a spherical polynomial of degree at most k. Therefore, for a function space S(Sd) containing Pk(Sd), we write
S∗(Sd):={f∈S(Sd):Πmf=0 for every m>k with m−k even}.
Here Πm denotes the spherical harmonic projection introduced below. Equivalently, up to Pk(Sd), S∗(Sd) consists of even functions when k is odd and odd functions when k is even.
A sharp direct approximation result was established in [23]. For a quasi-uniform parameter sequence {Θn}n=1∞ and 0≤α≤2dd+2k+1, one has, with M controlled by the Sobolev norm of f,
The endpoint 2d+2k+1 is determined by the decay of the spherical harmonic coefficients of a ReLUk ridge function.
Two converse results provide the immediate background for this work. First, the integral characterization in [23] shows that if, for some quasi-uniform sequence {Θn}n=1∞,
n→∞limfn∈Lnk,α(Θn)inf∥f−fn∥L2(Sd)=0,(3)
then f∈H∗2d+2k+1(Sd). Second, the Bernstein inequality established in [24] gives an inverse theorem for the unconstrained spaces. If {Θn}n≥1 is nested and antipodally quasi-uniform and 21<α<2dd+2k+1, then
implies f∈H∗r(Sd) for every 0≤r<2d+2k+1−αd. We strengthen these subcritical Sobolev conclusions below to the endpoint Besov space.
Direct and inverse estimates of this kind are widespread in classical approximation theory: together they determine whether a rate is sharp and whether improved convergence forces additional regularity of the target [5, 36, 13, 45, 25]. One of the key concepts in this theory is the notion of an approximation space (see [10]) corresponding to a sequence of linear spaces. Specifically, let Vn⊂L2(Sd) be a linear space of dimension n. For a given decay rate s>0, the approximation spaces with respect to the sequence of linear spaces Vn is defined by the semi-norm
∣f∣As({Vn}n=1∞):=n≥1supnsE(f,Vn),(5)
where
E(f,Vn)=fn∈Vninf∥f−fn∥L2(Sd)(6)
is the error of best approximation by the space Vn.
A useful classical comparison to our results is provided by the setting of linear approximation by splines. The simplest case of this is linear approximation by piecewise constants, which is detailed in Section 3.1 in [10], for instance. In this theory, the approximation spaces for a sequence of (non-nested) piecewise constant linear spaces Vn are shown to be Besov spaces, but only for decay rates up to n−s with s<1 (see [11], Chapter 7, Section 7). This is due to the fact that higher-order smoothness cannot be captured by piecewise constants, and indeed for higher order decay rates the approximation spaces cannot be described in a simple form.
When we consider the linear spaces Vn:=Lnk(Θn) for a sequence of quasi-uniform points Θn, one might expect that the same characterization of the approximation spaces holds. Our results show that this is indeed the case, up to approximation order s<21+2d2k+1, provided that we additionally constrain the coefficients of the approximant as in the definition (1.1), i.e., we consider
instead of E(f,Vn) for the linear spaces Vn=Lnk(Θn). This additional constraint is required since the ReLUk ridge functions are globally supported and highly correlated, in contrast to the spline basis. Motivated by this comparison, we address the following two questions:
Which function classes can be approximated when a prescribed growth bound is imposed on the coefficients?
Which regularity properties can be inferred from a prescribed approximation rate, with or without an explicit coefficient bound?
For α≥0, the natural coefficient and approximation scales considered here are
For α=0, these are the coefficient scale and endpoint rate in (1.2). Allowing the coefficient norm to grow like nα weakens the corresponding regularity and approximation order by the same factor.
Our first result identifies the exact approximation space at this scale. For 0<α<2dd+2k+1 and every quasi-uniform sequence {Θn}n=1∞,
f∈B2,∞,∗2dd+2k+1−α(Sd)⟺ there exists M>0 such that fn∈Ln,Mk(Θn)inf∥f−fn∥L2(Sd)≲n−2dd+2k+1+α.(9)
Thus there is no loss of smoothness in the inverse conclusion. The rate assumption remains essential: when α>0, mere convergence by approximants satisfying the same coefficient growth bound does not imply membership in B2,∞,∗2dd+2k+1−α(Sd). This contrasts with the endpoint case α=0 in (3).
We next remove the coefficient constraint and retain only the approximation rate. The resulting picture is more delicate. If 0<α<21 and {Θn}n=1∞ is nested, the estimate
fn∈Lnk(Θn)inf∥f−fn∥L2(Sd)≲n−2dd+2k+1+α(10)
does not imply that f∈B2,∞,∗2dd+2k+1−α(Sd). Indeed, nestedness preserves at least one ridge function at all levels, and a single ridge has exact Besov smoothness k+21. If α>21 and the nested sequence is antipodally quasi-uniform, the Bernstein inequality yields the converse implication at the endpoint order 2dd+2k+1−α.
The inverse conclusion can nevertheless be recovered from two geometrically independent parameter sequences. Suppose the same target satisfies (10) for quasi-uniform sequences {Θn}n=1∞ and {Θn}n=1∞, and that Θn∪Θn is antipodally separated at scale n−1/d. Subtracting approximants on the two parameter sets and applying a coefficient lower bound on their union yields
j=1∑naj,n2≲n2α−1.
Thus the coefficient growth required above is recovered from the two approximation estimates themselves, and the Besov inverse conclusion follows from (9).
These results separate two pieces of information that are often conflated in neural-network approximation: the size of the approximation error and the stability of the representation producing it. A single parameter sequence may approximate a target rapidly for geometric reasons that reveal little about its smoothness. An explicit coefficient bound, or approximation on two suitably separated parameter sequences, rules out this degeneracy and restores a sharp inverse statement. The result therefore has a saturation-type interpretation once the admissible coefficient growth is fixed, but it is not a classical saturation theorem for a single fixed approximation process. This perspective complements recent work relating neural-network norms, approximation classes, Bernstein inequalities, and generalization [42, 41, 43, 27, 18].
The remainder of the paper is organized as follows. Section 2 states the main theorem. Section 3 introduces the spherical harmonic notation, the ReLUk expansion, and the quadrature formula used in the analysis. Section 4 contains the proofs.
Main Results
Theorem 2.1. Let d∈N, k∈N0, 0<α<2dd+2k+1, and let {Θn}n=1∞ be a sequence of quasi-uniform collections on Sd.
Let f∈L2(Sd), then f∈B2,∞,∗2d+2k+1−αd(Sd) if and only if there exists M independent of n, such that
then it follows that f∈B2,∞,∗2d+2k+1−αd(Sd). On the other hand, for every α≤1/2, this characterization fails, i.e., there exists an f∈/B2,∞,∗2d+2k+1−αd(Sd) which satisfies (13).
Let f∈L2(Sd), and suppose there exists another sequence of antipodally quasi-uniform collections {Θn}n=1∞, with Θn={θj,n∗}j=1n, such that #(Θn∪Θn)=2n,
Throughout, dη denotes the normalized rotation-invariant measure on Sd. We write Pm(Sd) for the restrictions to Sd of polynomials on Rd+1 of total degree at most m.
Spherical harmonics, Besov spaces, and Sobolev spaces
For m∈N0, let Ym be the space of spherical harmonics of degree m, and fix an orthonormal basis {Ym,ℓ}ℓ=1N(m). Its dimension is
The collection Θn is called quasi-uniform if hn≲hn, and antipodally quasi-uniform if hn≲hn. A sequence {Θn}n=1∞ has either property when the corresponding constant is uniform in n, and it is called nested if Θn⊂Θn+1 for every n. In particular,
hn≃hn≃n−1/d
for a quasi-uniform sequence, while
hn≃hn≃n−1/d
in the antipodally quasi-uniform case. When one value of n is fixed, we write h=hn.
Related ultraspherical expansions of ReLU-type zonal functions appear in [33].
Polynomial quadrature on quasi-uniform collections
We shall use the following positive quadrature rule.
Lemma 3.1 ([28 Theorem 4.1], [35 Theorem 3.2]). There exist nonnegative weights τ1,n,…,τn,n satisfying τj,n≲hd, and a constant c1 independent of n and h, such that
The theorem is the endpoint specialization of the spherical direct approximation theorem in [23]. Although its parity condition is commonly stated for the pure even or odd component, the same construction covers the additional Pk(Sd) component: all degrees 0≤m≤k belong to Eσk and are reproduced by the quadrature formula.
We also record the stability estimate used in items 3 and 4. Its Bernstein part is the estimate proved in [24]; the same localization argument also gives the coefficient lower bound. We include the short common argument so that the latter is not left implicit.
Theorem 3.2 (Theorem 1, [24]; Theorem 1, [22]). Let d≥2, k∈N0, and {Θn}n=1∞ be antipodally quasi-uniform. Then
Throughout this section, c2 and τj,n are as in Lemma 3.1. In particular, τj,n≃n−1.
Proof of item 1
Direct implication. Let f∈B2,∞,∗2d+2k+1−αd(Sd). Choose Q=Q(n) so that 2Q≤nd1<2Q+1 and put gn:=P2Qf. The dyadic definition of the Besov norm and the orthogonality of the blocks give
For every sufficiently large q, choose n=nq:=⌈C2qd⌉, where the fixed constant C is large enough that 2q≤⌊c2nq/d⌋=J. Thus nq≃2qd. By the assumed approximation estimate, there exists fn∈Ln,Mk,α such that
Equation (4.10) and the finitely many low-frequency blocks give
q≥0sup2q(2d+2k+1−αd)∥Δqf∥L2(Sd)<∞,
so f∈B2,∞2d+2k+1−αd(Sd). It remains to verify the parity condition. For every m>k with m−k even, one has σ^k(m)=0, so Πmfn=0. Since 2d+2k+1−αd>0, (4.7) implies fn→f in L2(Sd) along the above choice of n. Consequently, Πmf=0 for every such m, and hence f∈B2,∞,∗2d+2k+1−αd(Sd).
Proof of item 2
Fix arbitrary ϵ>0 and fix L such that
∥f−ΠLf∥L2(Sd)≤ϵ.(42)
By Theorem 3.1, for each n∈N, there exists M0>0 and an,1,…,an,n with
Case 1:α<21. Since {Θn}n=1∞ is nested, we may fix θ1,1∗∈Θ1, so that θ1,1∗∈Θn for every n. Let
f(η):=σk(θ1,1∗⋅η).
Then f∈Lnk(Θn) for every n, so the left-hand side of (2.3) is identically zero. It remains to verify the claimed lack of regularity. By (3.8), (3.9), and (3.10), for m≥k+1 with m−k odd,
Hence f∈B2,∞2d+2k+1−αd(Sd). Finally, vq→f in L2, and every vq has zero projection onto Ym whenever m>k and m−k is even. Therefore f∈B2,∞,∗2d+2k+1−αd(Sd).
Case 3:α=21. By nestedness, after relabeling the points if necessary, we may write
Θn={θ1∗,…,θn∗},n=1,2,….
Since dimPJ(Sd)≍Jd, we may choose s0 sufficiently large and integers Js, s≥s0, such that
Js+2∈Eσk,Js≍2ds+1,dimPJs(Sd)≤2s−1,(48)
and Js+2≤c22ds+1, which lies below the quadrature degree associated with Θ2s+1.
For each s≥s0, we follow the notations in [22 (4.28)–(4.34)] and denote the orthogonal complement of the orthonormal vectors in (4.3):
Therefore, for some M>0 independent of n, fn∈Ln,Mk,α(Θn), while (54) retains the rate in item 1. The inverse implication in item 1 therefore gives f∈B2,∞,∗2dd+2k+1−αd(Sd), proving item 4.
References
[1]F. Bach. Breaking the curse of dimensionality with convex neural networks. The Journal of Machine Learning Research, 18(1):629–681, 2017.
[2]A. R. Barron. Universal approximation bounds for superpositions of a sigmoidal function. IEEE Transactions on Information theory, 39(3):930–945, 1993.DOI
[3]A. R. Barron. Approximation and estimation bounds for artificial neural networks. Machine learning, 14:115–133, 1994.
[4]J. Bernstein, A. Vahdat, Y. Yue, and M.-Y. Liu. On the distance between two neural networks and the stability of learning. Advances in Neural Information Processing Systems, 33:21370–21381, 2020.arxiv.org/abs/2002.03432
[5]S. N. Bernstein. Sur l’ordre de la meilleure approximation des fonctions continues par les polynômes de degré donné. Mem. Cl. Sci. Acad. Roy. Belg., 4:1–103, 1912.
[6]J. Bourgain and J. Lindenstrauss. Projection bodies. In Geometric Aspects of Functional Analysis: Israel Seminar (GAFA) 1986–87, pages 250–270. Springer, 2006.
[7]T. S. Cohen, M. Geiger, J. Köhler, and M. Welling. Spherical CNNs. In International Conference on Learning Representations, 2018.arxiv.org/abs/1801.10130
[8]F. Dai and Y. Xu. Approximation theory and harmonic analysis on spheres and balls. Springer, 2013.
[9]M. Defferrard, N. Perraudin, T. Kacprzak, and R. Sgier. DeepSphere: Towards an equivariant graph-based spherical CNN. In ICLR Workshop on Representation Learning on Graphs and Manifolds, 2019.arxiv.org/abs/1904.05146
[10]R. A. DeVore. Nonlinear approximation. Acta Numerica, 7:51–150, 1998.DOI
[11]R. A. DeVore and G. G. Lorentz. Constructive approximation, volume 303. Springer Science & Business Media, 1993.
[12]R. A. DeVore and V. N. Temlyakov. Some remarks on greedy algorithms. Advances in computational Mathematics, 5(1):173–187, 1996.DOI
[13]R. H. DeVore, Ronald A and C. Michelli. Optimal nonlinear approximation. Manuscripta mathematica, 58(3):295–306, 1989.DOI
[14]W. E and B. Yu. A priori estimates and analysis of a numerical method for machine learning. Science China Mathematics, 62(11):2303–2330, 2019.
[15]Z. Fang, H. Feng, S. Huang, and D.-X. Zhou. Theory of deep convolutional neural networks ii: Spherical analysis. Neural Networks, 131:154–162, 2020.arxiv.org/abs/2007.14285
[16]H. Feng, S. Huang, and D.-X. Zhou. Generalization analysis of CNNs for classification on spheres. IEEE Transactions on Neural Networks and Learning Systems, 2023.
[17]Z.-C. Guo and D.-X. Zhou. Optimal approximation with relu neural networks. Neural Networks, 128:96–109, 2020.
[18]S. Huang, J. Zhou, H. Feng, and D.-X. Zhou. Generalization analysis of pairwise learning for ranking with deep neural networks. Neural Computation, 35(6):1135–1158, 2023.DOI
[19]C. Jiang, J. Huang, K. Kashinath, Prabhat, P. Marcus, and M. Niessner. Spherical CNNs on unstructured grids. In International Conference on Learning Representations, 2019.arxiv.org/abs/1901.02039
[20]L. K. Jones. A simple lemma on greedy approximation in hilbert space and convergence rates for projection pursuit regression and neural network training. The Annals of Statistics, 20(1):608–613, 1992.DOI
[21]J. M. Klusowski and A. R. Barron. Approximation by combinations of relu and squared relu ridge functions with l1 and l0 controls. IEEE Transactions on Information Theory, 64(12):7649–7656, 2018.arxiv.org/abs/1607.07819
[22]X. Liu, T. Mao, and J. Xu. Condition numbers and eigenvalue spectra of shallow networks on spheres. arXiv preprint arXiv:2511.02625, 2025.arxiv.org/abs/2511.02625
[23]X. Liu, T. Mao, and J. Xu. Integral representations of sobolev spaces via ReLU^k activation function and optimal error estimates for linearized networks. arXiv preprint arXiv:2505.00351, 2025.arxiv.org/abs/2505.00351
[24]X. Liu, T. Mao, and J. Xu. Optimal neural network approximation via empirical least squares with deterministic samples. arXiv preprint arXiv:2608.06687, 2026.arxiv.org/abs/2608.06687
[25]V. Maiorov. Pseudo-dimension and entropy of manifolds formed by affine-invariant dictionary. Advances in Computational Mathematics, 25:435–450, 2006.DOI
[26]T. Mao and J. Xu. Sharp lower bounds for linearized relu^k approximation on the sphere. arXiv preprint arXiv:2510.04060, 2025.
[27]H. Mhaskar and T. Mao. Tractability of approximation by general shallow networks. arXiv preprint arXiv:2308.03230, 2023.arxiv.org/abs/2308.03230
[28]H. Mhaskar, F. Narcowich, and J. Ward. Spherical marcinkiewicz-zygmund inequalities and positive quadrature. Mathematics of computation, 70(235):1113–1130, 2001.DOI
[29]H. N. Mhaskar. Approximation properties of a multilayered feedforward artificial neural network. Advances in Computational Mathematics, 1:61–80, 1993.DOI
[30]H. N. Mhaskar. Polynomial operators and local smoothness classes on the unit interval. Journal of Approximation Theory, 131(2):243–267, 2004.DOI
[31]H. N. Mhaskar. On the representation of smooth functions on the sphere using finitely many bits. Applied and Computational Harmonic Analysis, 18(3):215–233, 2005.DOI
[32]H. N. Mhaskar. Weighted quadrature formulas and approximation by zonal function networks on the sphere. Journal of Complexity, 22(3):348–370, 2006.DOI
[33]H. N. Mhaskar. Function approximation with zonal function networks with activation functions analogous to the rectified linear unit functions. Journal of Complexity, 51:1–19, 2019.arxiv.org/abs/1709.08174
[34]H. N. Mhaskar. Kernel-based analysis of massive data. Frontiers in Applied Mathematics and Statistics, 6:30, 2020.arxiv.org/abs/2003.13226
[35]F. J. Narcowich, J. D. Ward, and H. Wendland. Sobolev error estimates and a bernstein inequality for scattered data interpolation via radial basis functions. Constructive Approximation, 24:175–186, 2006.DOI
[36]S. M. Nikolskii. Approximation of Functions of Several Variables and Imbedding Theorems. Springer, Berlin, 1975.DOI
[37]P. P. Petrushev. Approximation by ridge functions and neural networks. SIAM Journal on Mathematical Analysis, 30(1):155–189, 1998.DOI
[38]A. Pinkus. Approximation theory of the mlp model in neural networks. Acta numerica, 8:143–195, 1999.DOI
[39]R. Schneider. Zu einem problem von shephard über die projektionen konvexer körper. Mathematische Zeitschrift, 101:71–82, 1967.DOI
[40]J. W. Siegel and J. Xu. High-order approximation rates for shallow neural networks with cosine and ReLUk activation functions. Applied and Computational Harmonic Analysis, 58:1–26, 2022.
[41]J. W. Siegel and J. Xu. Optimal convergence rates for the orthogonal greedy algorithm. IEEE Transactions on Information Theory, 68(5):3354–3361, 2022.arxiv.org/abs/2106.15000
[42]J. W. Siegel and J. Xu. Sharp bounds on the approximation rates, metric entropy, and n-widths of shallow neural networks. Foundations of Computational Mathematics, pages 1–57, 2022.DOI
[43]J. W. Siegel and J. Xu. Characterization of the variation spaces corresponding to shallow neural networks. Constructive Approximation, 57(3):1109–1132, 2023.arxiv.org/abs/2106.15002
[44]Y. Yang, S. Huang, H. Feng, and D.-X. Zhou. Spherical analysis of learning nonlinear functionals. Journal of Machine Learning Research, 2026.arxiv.org/abs/2410.01047
[45]D.-X. Zhou and K. Jetter. Approximation with polynomial kernels and svm classifiers. Advances in Computational Mathematics, 25(1):323–344, 2006.DOI