Introduction
A central question in the Bayesian theory of neural networks is whether posteriors under network priors attain optimal frequentist rates of estimation. For deep ReLU networks, near-minimax contraction has been established under spike-and-slab priors whose nonzero network parameters are uniform on a bounded interval [23 ] , under Gaussian and other network-parameter laws with positive densities [16 ] , and under heavy-tailed network-parameter laws such as Student t t t and Cauchy distributions [8 ] . In each case, however, the joint prior changes with the sample size n n n : some part of the architecture, its distribution, or the network-parameter scale is chosen as a function of n n n . This allows the prior specification to be calibrated to the statistical resolution available at sample size n n n , but means that a different prior is used as the amount of data changes. We show that this dependence on n n n is unnecessary. Here, sieve-free refers to a prior fixed across sample sizes; this use of the term should not be confused with mixtures over finite-dimensional models, which are also called sieve priors [3 ] .
Our main aim is to isolate the mechanism that allows a single fixed prior to attain the same rates. A fixed prior cannot be tuned to one statistical accuracy; it must instead allocate enough probability simultaneously across all relevant approximation scales. For one-hidden-layer networks, width provides a natural index for these scales. Given a target and an accuracy ε > 0 \varepsilon> 0 ε > 0 , we use the approximation bounds of Yang and Zhou [28 , 29 ] to select an approximating network of suitable width k k k , and ask how much probability the same prior assigns to a neighbourhood of its parameters. The hidden-layer coefficients and biases of these approximators remain bounded, while the required output coefficients grow at most polynomially with the approximation complexity. These bounds can be dominated by rules depending only on the sampled width and the input dimension. This suggests a hierarchical construction: specify once and for all a law on all finite widths and, conditional on width k k k , a law on the corresponding network parameters. Neither n n n nor the unknown smoothness or radius of the target need enter the prior.
The argument separates the approximation problem from a local prior-mass requirement: the prior must assign sufficient probability to a suitable width and to accurate parameters at that width. We verify this requirement for three qualitatively different conditional network-parameter families: bounded uniform laws with width-dependent output support, Gaussian laws with a width-dependent output variance, and fixed Student t t t laws, including the Cauchy distribution. The same calculation applies to fixed network-parameter densities satisfying a polynomial lower bound. These examples identify local mass near approximating networks as the property that permits adaptation.
We work with one-hidden-layer networks in Gaussian random-design regression, with fixed input dimension d ≥ 1 d \ge1 d ≥ 1 and design law P X P_X P X on [ 0 , 1 ] d [0,1]^d [ 0 , 1 ] d . Width indexes approximation complexity, and uniform approximation gives likelihood neighbourhoods for arbitrary design laws. Following Bhattacharya et al. [7 ] and Castillo and Egels [8 ] , we study a fractional posterior, which raises the likelihood to a fixed power 0 < η < 1 0 < \eta< 1 0 < η < 1 . Its contraction argument requires only sufficient local prior mass, avoiding the additional global control of large models needed for ordinary-posterior contraction. The order-η \eta η Rényi divergence arises directly from the fractional likelihood ratio. For Hölder smoothness 0 < β ≤ 1 0 < \beta\le1 0 < β ≤ 1 and radius M > 0 M > 0 M > 0 , the fractional posterior under the same fixed prior contracts in Rényi divergence at order ( log n / n ) 2 β / ( 2 β + d ) (\log n/n)^{2\beta/(2\beta+d)} ( log n / n ) 2 β / ( 2 β + d ) , uniformly over the corresponding Hölder ball.
Rényi contraction isolates this mechanism, but squared prediction error is a stronger and more familiar frequentist loss. The passage between the two creates a separate difficulty: posterior draws under our fixed priors need not be uniformly bounded, and large discrepancies on sets of small P X P_X P X -probability can have a much greater effect on squared error than on Rényi divergence. Rather than impose additional restrictions on the prior or a clipping level depending on the unknown target radius, we clip posterior draws at a level estimated from the data. Under a design density bounded away from zero, this level contains the range of the target with exponentially high probability. Once both functions are bounded, Rényi divergence controls squared prediction error. The resulting posterior mean has risk ( log n / n ) 2 β / ( 2 β + d ) (\log n/n)^{2\beta/(2\beta+d)} ( log n / n ) 2 β / ( 2 β + d ) under the lower density bound. If the density is also bounded above, the classical Hölder minimax lower bound shows that this rate is optimal up to a logarithmic factor. An alternative clipping construction retains the same upper risk rate under arbitrary design laws, at the cost of an additional polynomial term in the posterior-tail bound. Clipping also ensures that the posterior mean exists. For Cauchy and Student network-parameter priors with at most one degree of freedom, the unclipped mean can fail to exist when a unit is inactive at every observed covariate; see Appendix § 11 . Whether squared prediction-loss contraction holds without clipping, or the unclipped mean attains the same risk rate under conditions ensuring its existence, remains open.
To the best of our knowledge, this is the first adaptive near-minimax fractional-posterior contraction result for dense shallow ReLU networks under a prior independent of the sample size, with bounded parameter distributions conditional on width.
Approximation and frequentist estimation
Classical shallow-network rates such as those of Barron [5 ] concern Fourier-moment classes rather than Hölder balls. For deep ReLU networks, quantitative approximation bounds (Yarotsky, [30 ] ) underpin near-minimax estimation for compositional targets (Schmidt-Hieber, [24 ] ), fully connected architectures (Kohler and Langer, [15 ] ), and Besov and mixed-smoothness classes (Suzuki, [27 ] ).
We use the shallow-network approximation rate k − β / d k^{-\beta/d} k − β / d for width k k k and output-coefficient bounds of Yang and Zhou [29 ] , valid for 0 < β < ( d + 3 ) / 2 0 < \beta< (d+3)/2 0 < β < ( d + 3 ) /2 ; see also Yang and Zhou [28 ] . Rather than select an architecture for a frequentist procedure, these bounds identify accurate parameter neighbourhoods whose probability we evaluate under one fixed prior. We compare the resulting risk with the classical Hölder minimax lower bound of Stone [26 ] .
Bayesian contraction and architecture selection
Random widths already appear in the shallow-network consistency results of Lee [18 ] ; hierarchical dimension priors are also standard in Bayesian nonparametrics [3 ] . For deep ReLU networks, Polson and Ročková [23 ] obtain near-minimax contraction with uniform-slab spike-and-slab priors and random width, but retain other n n n -dependent architectural choices. Related approaches use sparse connections [19 , 20 ] , hidden-unit selection [17 ] , or variational inference [9 , 4 , 21 ] .
For dense ordinary posteriors, Kong and Kim [16 ] cover Gaussian and other network-parameter laws through a local density lower bound on every bounded parameter box, with network dimensions varying with n n n . Our condition instead controls neighbourhoods of approximation-compatible parameters, allowing width-dependent bounded supports.
Heavy tails and fractional posteriors
Heavy-tailed series priors adapt to unknown smoothness [1 ] . For deep ReLU networks, Castillo and Egels [8 ] obtain near-minimax fractional-posterior contraction with heavy-tailed network parameters and n n n -dependent network specifications and coefficient scales. The adaptive shallow-network construction of Agapiou et al. [2 ] , with fixed shifts and random output coefficients, likewise uses n n n -dependent quantities.
Polynomial density lower bounds make coefficient magnitude contribute only logarithmically to the negative log-probability of local parameter intervals. Our uniform and Gaussian constructions satisfy the same local-mass criterion. Fractional posteriors isolate this criterion from the global control needed for ordinary posteriors; clipping then recovers squared prediction-risk bounds.
Statistical model and fractional posterior
All functions are assumed measurable. Random objects are defined on a probability space ( Ω , A , P ) (\Omega,\mathcal{A},\mathbb{P}) ( Ω , A , P ) , with Borel σ \sigma σ -algebras on their state spaces. Fix d ≥ 1 d \ge1 d ≥ 1 and write X = [ 0 , 1 ] d \mathcal{X} = [0,1]^d X = [ 0 , 1 ] d . For an unknown regression function f 0 : X → R f_0 : \mathcal{X} \to\mathbb{R} f 0 : X → R , we observe D n = ( ( X i , Y i ) ) i = 1 n \mathcal{D}_n = ((X_i,Y_i))_{i=1}^n D n = (( X i , Y i ) ) i = 1 n from
Y i = f 0 ( X i ) + ξ i , i = 1 , … , n , (1) Y_i = f_0(X_i) + \xi_i,\qquad i = 1,\ldots,n, \tag*{(1)} Y i = f 0 ( X i ) + ξ i , i = 1 , … , n , (1)
where the covariates X , X i : Ω → X X,X_i : \Omega\to\mathcal{X} X , X i : Ω → X , i ≥ 1 i \ge1 i ≥ 1 , are independent with law P X P_X P X and the errors ξ i : Ω → R \xi_i : \Omega\to\mathbb{R} ξ i : Ω → R are independent standard normal variables, independent of the covariates. The design law is fixed but otherwise unrestricted unless a density condition is stated. Write E f 0 \mathbb{E}_{f_0} E f 0 and P f 0 \mathbb{P}_{f_0} P f 0 for expectation and probability under this data law.
Let C ( X ) C(\mathcal{X}) C ( X ) be the continuous real-valued functions on X \mathcal{X} X , equipped with the supremum norm and its Borel σ \sigma σ -algebra. For a real vector v = ( v 1 , … , v ℓ ) v=(v_1,\ldots,v_\ell) v = ( v 1 , … , v ℓ ) with ℓ ≥ 1 \ell\ge1 ℓ ≥ 1 coordinates, write ∥ v ∥ 1 = ∑ j = 1 ℓ ∣ v j ∣ \lVert v\rVert_1=\sum_{j=1}^{\ell}|v_j| ∥ v ∥ 1 = ∑ j = 1 ℓ ∣ v j ∣ , ∥ v ∥ 2 = ( ∑ j = 1 ℓ v j 2 ) 1 / 2 \lVert v\rVert_2=(\sum_{j=1}^{\ell}v_j^2)^{1/2} ∥ v ∥ 2 = ( ∑ j = 1 ℓ v j 2 ) 1/2 and ∥ v ∥ ∞ = max 1 ≤ j ≤ ℓ ∣ v j ∣ \lVert v\rVert_\infty=\max_{1\le j\le\ell}|v_j| ∥ v ∥ ∞ = max 1 ≤ j ≤ ℓ ∣ v j ∣ . For a real-valued function f f f with domain S S S , write ∥ f ∥ ∞ = sup x ∈ S ∣ f ( x ) ∣ \lVert f\rVert_\infty=\sup_{x\in S}|f(x)| ∥ f ∥ ∞ = sup x ∈ S ∣ f ( x ) ∣ . For 0 < β ≤ 1 0<\beta\le1 0 < β ≤ 1 and M > 0 M>0 M > 0 , define
H d β ( M ) = { f ∈ C ( X ) : ∥ f ∥ ∞ ≤ M , ∣ f ( x ) − f ( y ) ∣ ≤ M ∥ x − y ∥ 2 β for all x , y ∈ X } . (2) \mathcal{H}^{\beta}_d(M)=\{f\in C(\mathcal{X}):\lVert f\rVert_\infty\le M,\ |f(x)-f(y)|\le M\lVert x-y\rVert_2^\beta\text{ for all }x,y\in\mathcal{X}\}. \tag*{(2)} H d β ( M ) = { f ∈ C ( X ) : ∥ f ∥ ∞ ≤ M , ∣ f ( x ) − f ( y ) ∣ ≤ M ∥ x − y ∥ 2 β for all x , y ∈ X } . (2)
The prediction loss is ∥ f − f 0 ∥ L 2 ( P X ) 2 = ∫ X ( f − f 0 ) 2 d P X \lVert f-f_0\rVert_{L^2(P_X)}^2=\int_{\mathcal{X}}(f-f_0)^2\,dP_X ∥ f − f 0 ∥ L 2 ( P X ) 2 = ∫ X ( f − f 0 ) 2 d P X . Given a prior Π \Pi Π on C ( X ) C(\mathcal{X}) C ( X ) and a fixed 0 < η < 1 0<\eta<1 0 < η < 1 , the fractional posterior is
Π n η ( d f ∣ D n ) ∝ exp { − η 2 ∑ i = 1 n ( Y i − f ( X i ) ) 2 } Π ( d f ) . (3) \Pi_n^\eta(df\mid\mathcal{D}_n)\propto\exp\left\{-\frac{\eta}{2}\sum_{i=1}^n(Y_i-f(X_i))^2\right\}\Pi(df). \tag*{(3)} Π n η ( df ∣ D n ) ∝ exp { − 2 η i = 1 ∑ n ( Y i − f ( X i ) ) 2 } Π ( df ) . (3)
The positive bounded likelihood gives a finite, nonzero normalizing constant for every data set. Write E X \mathbb{E}_X E X for expectation under X ∼ P X X\sim P_X X ∼ P X . The one-observation order-η \eta η Rényi divergence in this model is
D η ( f , f 0 ) = − 1 1 − η log E X e − η ( 1 − η ) ( f ( X ) − f 0 ( X ) ) 2 / 2 . (4) D_{\eta}(f,f_0)=-\frac{1}{1-\eta}\log\mathbb{E}_{X}e^{-\eta(1-\eta)(f(X)-f_0(X))^2/2}. \tag*{(4)} D η ( f , f 0 ) = − 1 − η 1 log E X e − η ( 1 − η ) ( f ( X ) − f 0 ( X ) ) 2 /2 . (4)
We write a ≲ b a\lesssim b a ≲ b if a ≤ C b a\leq Cb a ≤ C b for a fixed positive constant C C C , a ≳ b a\gtrsim b a ≳ b if b ≲ a b\lesssim a b ≲ a , and a ≍ b a\asymp b a ≍ b if both inequalities hold. For positive sequences, a n ≫ b n a_n\gg b_n a n ≫ b n means a n / b n → ∞ a_n/b_n\to\infty a n / b n → ∞ . The symbols ∨ \vee ∨ and ∧ \wedge ∧ denote maximum and minimum. Constants in uniform bounds may depend on d d d , β \beta β , M M M , η \eta η and the chosen prior, but not on n n n , the approximation accuracy, or f 0 f_0 f 0 within the stated Hölder ball. They are independent of P X P_X P X unless dependence on design-density bounds is stated.
Where the prior must place mass
Lemma 1 specifies how much local prior mass contraction needs; Proposition 1 identifies where to place it. Both apply before choosing a particular network prior.
How much mass is needed. The following form of the fractional-posterior principle of Bhattacharya et al. [7 ] converts uniform-approximation mass into Rényi contraction.
Lemma 1 (Prior mass implies fractional contraction). Let Π \Pi Π be any prior on C ( X ) C(\mathcal{X}) C ( X ) , and let ϵ n ↓ 0 \epsilon_n\downarrow0 ϵ n ↓ 0 with n ϵ n 2 → ∞ n\epsilon_n^2\to\infty n ϵ n 2 → ∞ . Suppose that, for some fixed c 0 > 0 c_0>0 c 0 > 0 , uniformly over a class of targets,
Π { f : ∥ f − f 0 ∥ ∞ ≤ ϵ n } ≥ e − c 0 n ϵ n 2 . \Pi\{f:\|f-f_0\|_\infty\leq\epsilon_n\}\geq e^{-c_0n\epsilon_n^2}. Π { f : ∥ f − f 0 ∥ ∞ ≤ ϵ n } ≥ e − c 0 n ϵ n 2 . For 0 < η < 1 0<\eta<1 0 < η < 1 and J > ( c 0 + η ) / ( 1 − η ) J>(c_0+\eta)/(1-\eta) J > ( c 0 + η ) / ( 1 − η ) ,
sup f 0 E f 0 Π n η ( D η ( f , f 0 ) > J ϵ n 2 ∣ D n ) ≤ e − n ϵ n 2 / 8 + e − { ( 1 − η ) J − c 0 − η } n ϵ n 2 . (5) \sup_{f_0}\mathbb{E}_{f_0}\Pi_n^\eta\left(D_\eta(f,f_0)>J\epsilon_n^2\mid D_n\right)\leq e^{-n\epsilon_n^2/8}+e^{-\{(1-\eta)J-c_0-\eta\}n\epsilon_n^2}. \tag*{(5)} f 0 sup E f 0 Π n η ( D η ( f , f 0 ) > J ϵ n 2 ∣ D n ) ≤ e − n ϵ n 2 /8 + e − {( 1 − η ) J − c 0 − η } n ϵ n 2 . (5)
At ϵ n ≍ ( log n / n ) β / ( 2 β + d ) \epsilon_n\asymp(\log n/n)^{\beta/(2\beta+d)} ϵ n ≍ ( log n / n ) β / ( 2 β + d ) , n ϵ n 2 ≍ ϵ n − d / β log ( 1 / ϵ n ) n\epsilon_n^2\asymp\epsilon_n^{-d/\beta}\log(1/\epsilon_n) n ϵ n 2 ≍ ϵ n − d / β log ( 1/ ϵ n ) . It therefore suffices that
− log Π { f : ∥ f − f 0 ∥ ∞ ≤ ϵ } ≲ ϵ − d / β log ( 1 / ϵ ) -\log\Pi\{f:\|f-f_0\|_\infty\leq\epsilon\}\lesssim\epsilon^{-d/\beta}\log(1/\epsilon) − log Π { f : ∥ f − f 0 ∥ ∞ ≤ ϵ } ≲ ϵ − d / β log ( 1/ ϵ )
for all small ϵ \epsilon ϵ , uniformly over each H d β ( M ) \mathcal{H}_d^\beta(M) H d β ( M ) . The lemma is proved in Appendix H.
Where the mass is needed. For a width k ≥ 1 k\geq1 k ≥ 1 , let θ = ( ( a m , u m , b m ) ) m = 1 k ∈ R ( d + 2 ) k \theta=((a_m,u_m,b_m))_{m=1}^k\in\mathbb{R}^{(d+2)k} θ = (( a m , u m , b m ) ) m = 1 k ∈ R ( d + 2 ) k , where a m , b m ∈ R a_m,b_m\in\mathbb{R} a m , b m ∈ R are output coefficients and biases, and u m ∈ R d u_m\in\mathbb{R}^d u m ∈ R d are hidden-layer coefficient vectors. With the ReLU activation ρ ( t ) = max { t , 0 } \rho(t)=\max\{t,0\} ρ ( t ) = max { t , 0 } , define
f θ ( x ) = ∑ m = 1 k a m ρ ( u m T x + b m ) , x ∈ X . f_\theta(x)=\sum_{m=1}^{k}a_m\rho(u_m^{\mathsf{T}}x+b_m),\qquad x\in\mathcal{X}. f θ ( x ) = m = 1 ∑ k a m ρ ( u m T x + b m ) , x ∈ X .
The following is Yang and Zhou [29 ] , with the accuracy expressed through the width and with zero units added when the approximating network has fewer than k k k units. Put ω = ( d + 3 ) / ( 2 d ) \omega=(d+3)/(2d) ω = ( d + 3 ) / ( 2 d ) .
Proposition 1 (Approximation by shallow networks). Fix 0 < β ≤ 1 0<\beta\leq1 0 < β ≤ 1 and M > 0 M>0 M > 0 . There is C > 0 C>0 C > 0 such that, for every f 0 ∈ H d β ( M ) f_0\in\mathcal{H}_d^\beta(M) f 0 ∈ H d β ( M ) and every k ≥ 1 k\geq1 k ≥ 1 , some θ k ∗ = ( ( a m ∗ , u m ∗ , b m ∗ ) ) m = 1 k \theta_k^*=((a_m^*,u_m^*,b_m^*))_{m=1}^k θ k ∗ = (( a m ∗ , u m ∗ , b m ∗ ) ) m = 1 k satisfies
∥ f θ k ∗ − f 0 ∥ ∞ ≤ C k − β / d , (6) \|f_{\theta_k^*}-f_0\|_\infty\leq Ck^{-\beta/d}, \tag*{(6)} ∥ f θ k ∗ − f 0 ∥ ∞ ≤ C k − β / d , (6) max 1 ≤ m ≤ k { ∥ u m ∗ ∥ ∞ , ∣ b m ∗ ∣ } ≤ 1 , (7) \max_{1\leq m\leq k}\{\|u_m^*\|_\infty,|b_m^*|\}\leq1, \tag*{(7)} 1 ≤ m ≤ k max { ∥ u m ∗ ∥ ∞ , ∣ b m ∗ ∣ } ≤ 1 , (7) ∑ m = 1 k ∣ a m ∗ ∣ ≤ C k ω − β / d . (8) \sum_{m=1}^{k}|a_m^*|\leq Ck^{\omega-\beta/d}. \tag*{(8)} m = 1 ∑ k ∣ a m ∗ ∣ ≤ C k ω − β / d . (8)
Choose k ≍ ε − d / β k \asymp\varepsilon^{-d/\beta} k ≍ ε − d / β so the approximator is ε / 2 \varepsilon/2 ε /2 -accurate. The ReLU Lipschitz bound ensures that a parameter box about θ k ∗ \theta_k^* θ k ∗ of radius δ > 0 \delta> 0 δ > 0 polynomially small in ε \varepsilon ε contains only ε \varepsilon ε -accurate functions. Let K denote the random width and θ k \theta_k θ k the parameter vector conditional on K = k K = k K = k . The prior probability of these functions is therefore at least
P ( K = k ) ⋅ P ( ∥ θ k − θ k ∗ ∥ ∞ ≤ δ ∣ K = k ) . \mathbb{P}(K = k) \cdot\mathbb{P}\left(\lVert\theta_k - \theta_k^*\rVert_\infty\leq\delta\mid K = k\right). P ( K = k ) ⋅ P ( ∥ θ k − θ k ∗ ∥ ∞ ≤ δ ∣ K = k ) .
It suffices to bound each negative log-probability by C k log ( k / δ ) ≍ k log ( 1 / ε ) Ck\log(k/\delta) \asymp k\log(1/\varepsilon) C k log ( k / δ ) ≍ k log ( 1/ ε ) . Section 4 imposes target-independent conditions giving these bounds; the approximation and perturbation details are in Appendices A and B.
Sufficient conditions on the prior
We now give conditions on the width and network-parameter laws that ensure the mass required in Section 3, without reference to n n n , β \beta β or M M M .
We consider one fixed hierarchical prior on finite-width ReLU networks. Draw a positive integer width K : Ω → N K : \Omega\to\mathbb{N} K : Ω → N with mass function π K ( k ) = P ( K = k ) \pi_K(k) = \mathbb{P}(K = k) π K ( k ) = P ( K = k ) and then draw θ k : Ω → R ( d + 2 ) k \theta_k : \Omega\to\mathbb{R}^{(d+2)k} θ k : Ω → R ( d + 2 ) k with θ k = ( ( A m , U m , B m ) ) m = 1 k \theta_k = ((A_m, U_m, B_m))_{m=1}^k θ k = (( A m , U m , B m ) ) m = 1 k , from a law Π k θ \Pi_k^\theta Π k θ on R ( d + 2 ) k \mathbb{R}^{(d+2)k} R ( d + 2 ) k , conditional on K = k K = k K = k . Write U m = ( U m 1 , … , U m d ) U_m = (U_{m1}, \ldots, U_{md}) U m = ( U m 1 , … , U m d ) for the hidden-layer coefficient vector. The width and parameters are independent of the covariates and errors. The random function F : Ω → C ( X ) F : \Omega\to C(\mathcal{X}) F : Ω → C ( X ) is
F ( x ) = f θ K ( x ) = ∑ m = 1 K A m ρ ( U m T x + B m ) , x ∈ X . (9) F(x) = f_{\theta_K}(x) = \sum_{m=1}^K A_m \rho(U_m^\mathsf{T}x + B_m), \qquad x \in\mathcal{X}. \tag*{(9)} F ( x ) = f θ K ( x ) = m = 1 ∑ K A m ρ ( U m T x + B m ) , x ∈ X . (9)
Its law Π \Pi Π on C ( X ) C(\mathcal{X}) C ( X ) is well defined because the parameter-to-function map is continuous at each finite width. Neither the width law nor the conditional network-parameter laws depend on n n n , β \beta β , or M M M , and independence between network-parameter coordinates is not assumed.
General sufficient conditions
Width probabilities. The width law must not make large widths too improbable. For some fixed C π > 0 C_\pi> 0 C π > 0 , require
log π K ( k ) ≥ − C π k log ( e k ) , k ≥ 1. (10) \log\pi_K(k) \geq-C_\pi k\log(ek), \qquad k \geq1. \tag*{(10)} log π K ( k ) ≥ − C π k log ( e k ) , k ≥ 1. (10)
This permits a log-cost of order k log ( 1 / ε ) k\log(1/\varepsilon) k log ( 1/ ε ) at the required width. Examples include π K ( k ) = 2 − k \pi_K(k) = 2^{-k} π K ( k ) = 2 − k , K ∼ 1 + Poisson ( λ ) K \sim1 + \operatorname{Poisson}(\lambda) K ∼ 1 + Poisson ( λ ) for fixed λ > 0 \lambda> 0 λ > 0 , and π K ( k ) ∝ k − a \pi_K(k) \propto k^{-a} π K ( k ) ∝ k − a for fixed a > 1 a > 1 a > 1 . A width law with probabilities proportional to e − k 2 e^{-k^2} e − k 2 is an example that fails condition (10).
Conditional network-parameter probabilities. Since C k ω ε − β / d ≤ 1 2 k ω C k^\omega\varepsilon^{-\beta/d} \leq\frac{1}{2}k^\omega C k ω ε − β / d ≤ 2 1 k ω eventually for every β \beta β , M M M , the approximators satisfy the following bounds at all large widths. Call θ ∗ = ( ( a m ∗ , u m ∗ , b m ∗ ) ) m = 1 k \theta^* = ((a_m^*, u_m^*, b_m^*))_{m=1}^k θ ∗ = (( a m ∗ , u m ∗ , b m ∗ ) ) m = 1 k an admissible centre if
max 1 ≤ m ≤ k { ∥ u m ∗ ∥ ∞ , ∣ b m ∗ ∣ } ≤ 1 , ∑ m = 1 k ∣ a m ∗ ∣ ≤ 1 2 k ω . (11) \max_{1 \leq m \leq k}\{\lVert u_m^*\rVert_\infty, |b_m^*|\} \leq1, \qquad\sum_{m=1}^k |a_m^*| \leq\frac{1}{2}k^\omega. \tag*{(11)} 1 ≤ m ≤ k max {∥ u m ∗ ∥ ∞ , ∣ b m ∗ ∣ } ≤ 1 , m = 1 ∑ k ∣ a m ∗ ∣ ≤ 2 1 k ω . (11)
For some fixed C ω > 0 C_\omega> 0 C ω > 0 , require
− log Π k θ { θ : ∥ θ − θ ∗ ∥ ∞ ≤ δ } ≤ C ω k log ( e k / δ ) , (12) -\log\Pi_k^\theta\{\theta: \lVert\theta- \theta^*\rVert_\infty\leq\delta\} \leq C_\omega k\log(ek/\delta), \tag*{(12)} − log Π k θ { θ : ∥ θ − θ ∗ ∥ ∞ ≤ δ } ≤ C ω k log ( e k / δ ) , (12)
for every k ≥ 1 k \geq1 k ≥ 1 , every admissible centre θ ∗ \theta^* θ ∗ at width k k k , and every 0 < δ ≤ 1 / 2 0 < \delta\leq1/2 0 < δ ≤ 1/2 .
Condition (12) requires mass at least ( δ / e k ) C ω k (\delta/ek)^{C_\omega k} ( δ / e k ) C ω k near each admissible centre: a log-cost proportional to the number of parameters times the logarithm of the required precision. This is the order obtained from uniform laws on polynomially growing boxes. Only local mass matters; bounded supports and dependent coordinates are allowed. Together with (10), it gives the required function-space mass.
Proposition 2 (Local prior mass). Let Π \Pi Π satisfy (10) and (12). For every 0 < β ≤ 1 0 < \beta\le1 0 < β ≤ 1 and M > 0 M > 0 M > 0 there are C , ε 0 > 0 C,\varepsilon_0 > 0 C , ε 0 > 0 such that, for all f 0 ∈ H d β ( M ) f_0 \in\mathcal{H}^{\beta}_d(M) f 0 ∈ H d β ( M ) and 0 < ε < ε 0 0 < \varepsilon< \varepsilon_0 0 < ε < ε 0 ,
Π { f : ∥ f − f 0 ∥ ∞ ≤ ε } ≥ exp { − C ε − d / β log ( 1 / ε ) } . (13) \Pi\{f:\|f-f_0\|_\infty\le\varepsilon\} \ge\exp\{-C\varepsilon^{-d/\beta}\log(1/\varepsilon)\}. \tag*{(13)} Π { f : ∥ f − f 0 ∥ ∞ ≤ ε } ≥ exp { − C ε − d / β log ( 1/ ε )} . (13) The proof is in Appendix B. The conditions are sufficient, not necessary: only centres supplied by Proposition 1 are used in the proof.
Priors satisfying the conditions
We verify (12) for three familiar network-parameter families: uniform, analogous to the slab component of Polson and Ročková [23 ] ; Gaussian, covered by Kong and Kim [16 ] ; and Student t t t , including Cauchy, as considered by Castillo and Egels [8 ] . Table 1 specifies our conditional laws and width scalings, not those papers’ joint priors. For a < b a < b a < b , Unif [ a , b ] \operatorname{Unif}[a,b] Unif [ a , b ] denotes the uniform distribution on [ a , b ] [a,b] [ a , b ] . Here N ( 0 , s 2 ) \mathcal{N}(0,s^2) N ( 0 , s 2 ) denotes the normal distribution with mean zero and variance s 2 s^2 s 2 , and t ν t_\nu t ν denotes the standard Student t t t distribution with ν > 0 \nu> 0 ν > 0 degrees of freedom.
Table 1. Conditional network-parameter priors satisfying (12).
Lemma 2 (Uniform, Gaussian and Student network-parameter laws). Fix ν > 0 \nu> 0 ν > 0 . For each k ≥ 1 k \ge1 k ≥ 1 , let the coordinates A m A_m A m , U m j U_{mj} U mj and B m B_m B m (1 ≤ m ≤ k 1 \le m \le k 1 ≤ m ≤ k , 1 ≤ j ≤ d 1 \le j \le d 1 ≤ j ≤ d ) of θ k \theta_k θ k be independent under Π k θ \Pi_k^\theta Π k θ , with laws given by one row of Table 1 . Then Π k θ \Pi_k^\theta Π k θ satisfies (12) with a constant C w C_w C w depending only on d d d for the uniform and Gaussian rows, and only on d d d and ν \nu ν for the Student row.
The families differ in how they reach large output coefficients: uniform support expands to contain them, Gaussian variance grows to control their quadratic log-penalty, while Student tails incur only logarithmic penalties without rescaling (ν = 1 \nu= 1 ν = 1 gives Cauchy). Hidden-layer coefficients and biases need no rescaling. Unit-variance Gaussian output coefficients would fail (12), which allows a single coefficient of size 1 2 k ω \frac{1}{2}k^\omega 2 1 k ω . The Student argument extends to any fixed density with a polynomial lower bound.
Proposition 3 (Polynomial density lower bounds). Let q q q be a probability density on R \mathbb{R} R such that
q ( z ) ≥ c ( 1 + ∣ z ∣ ) − r , z ∈ R , q(z) \ge c(1+|z|)^{-r}, \qquad z \in\mathbb{R}, q ( z ) ≥ c ( 1 + ∣ z ∣ ) − r , z ∈ R , for some c > 0 c > 0 c > 0 and r > 1 r > 1 r > 1 . If, for each k ≥ 1 k \ge1 k ≥ 1 , the coordinates of θ k \theta_k θ k are independent under Π k θ \Pi_k^\theta Π k θ , each with density q q q , then Π k θ \Pi_k^\theta Π k θ satisfies (12) with a constant C w C_w C w depending only on d d d , c c c and r r r .
This includes Student t ν t_\nu t ν (r = ν + 1 r = \nu+ 1 r = ν + 1 ) and any fixed density mixture with a positive Cauchy component. Paired with any width law satisfying (10), these give priors covered by the results below. The probability estimates are in Appendix G.
Contraction and prediction risk
Lemma 1 and Proposition 2 give the near-minimax rate derived in Section 3. Theorem 1 states the resulting posterior-mean risk; Theorem 2 provides the underlying contraction bounds.
To pass from Rényi divergence to squared prediction loss, we need bounded functions. Write C b ( t ) = ( − b ) ∨ ( t ∧ b ) C_b(t)=(-b)\vee(t\wedge b) C b ( t ) = ( − b ) ∨ ( t ∧ b ) for clipping at b > 0 b > 0 b > 0 . The data-based choice below avoids a known target bound.
Theorem 1 (Near-minimax posterior-mean risk). Let Π \Pi Π be a fixed hierarchical prior satisfying (10) and (12) , and fix 0 < η < 1 0 < \eta< 1 0 < η < 1 . There is a sequence of finite positive data-dependent cutoffs B ~ n = B ~ n ( D n ) \widetilde{B}_n = \widetilde{B}_n(D_n) B n = B n ( D n ) , n ≥ 3 n \ge3 n ≥ 3 , depending only on n n n , d d d and the data, with the following property. For every 0 < β ≤ 1 0 < \beta\le1 0 < β ≤ 1 , M > 0 M > 0 M > 0 , and design law P X P_X P X with Lebesgue density p X p_X p X satisfying, for fixed 0 < c X ≤ C X < ∞ 0 < c_X \le C_X < \infty 0 < c X ≤ C X < ∞ and Lebesgue-almost every x ∈ X x \in\mathcal{X} x ∈ X ,
0 < c X ≤ p X ( x ) ≤ C X < ∞ , (14) 0 < c_X \le p_X(x) \le C_X < \infty, \tag*{(14)} 0 < c X ≤ p X ( x ) ≤ C X < ∞ , (14) the posterior means
f ~ n ( x ) = ∫ C ( X ) C B ~ n ( f ( x ) ) Π n η ( d f ∣ D n ) (15) \widetilde{f}_n(x) = \int_{\mathcal{C}(\mathcal{X})} C_{\widetilde{B}_n}(f(x)) \Pi_n^\eta(df \mid D_n) \tag*{(15)} f n ( x ) = ∫ C ( X ) C B n ( f ( x )) Π n η ( df ∣ D n ) (15) satisfy, for all sufficiently large n n n ,
sup f 0 ∈ H d β ( M ) E f 0 ∥ f ~ n − f 0 ∥ L 2 ( P X ) 2 ≲ ( log n / n ) 2 β / ( 2 β + d ) . (16) \sup_{f_0 \in\mathcal{H}_d^\beta(M)} \mathbb{E}_{f_0}\|\widetilde{f}_n-f_0\|_{L^2(P_X)}^2 \lesssim(\log n/n)^{2\beta/(2\beta+d)}. \tag*{(16)} f 0 ∈ H d β ( M ) sup E f 0 ∥ f n − f 0 ∥ L 2 ( P X ) 2 ≲ ( log n / n ) 2 β / ( 2 β + d ) . (16) The upper bound requires only the lower density bound; under both bounds in (14) ,
inf f n sup f 0 ∈ H d β ( M ) E f 0 ∥ f n − f 0 ∥ L 2 ( P X ) 2 ≳ n − 2 β / ( 2 β + d ) , (17) \inf_{f_n}\sup_{f_0 \in\mathcal{H}_d^\beta(M)} \mathbb{E}_{f_0}\|f_n-f_0\|_{L^2(P_X)}^2 \gtrsim n^{-2\beta/(2\beta+d)}, \tag*{(17)} f n inf f 0 ∈ H d β ( M ) sup E f 0 ∥ f n − f 0 ∥ L 2 ( P X ) 2 ≳ n − 2 β / ( 2 β + d ) , (17) where the infimum is over all measurable estimators. Constants may depend on c X c_X c X and C X C_X C X .
The same estimator adapts to every stated Hölder ball and design law, losing only ( log n ) 2 β / ( 2 β + d ) (\log n)^{2\beta/(2\beta+d)} ( log n ) 2 β / ( 2 β + d ) relative to the classical minimax bound (Stone, [26 ] ); see § 7.1 . Its upper risk bound follows from the next contraction result.
Theorem 2 (Fractional-posterior contraction). Let Π \Pi Π be a fixed hierarchical prior satisfying (10) and (12) . Fix 0 < β ≤ 1 0 < \beta\le1 0 < β ≤ 1 , M > 0 M > 0 M > 0 , and 0 < η < 1 0 < \eta< 1 0 < η < 1 . There are A , C , c > 0 A,C,c > 0 A , C , c > 0 such that, with
ϵ n = A ( log n / n ) β / ( 2 β + d ) , n ≥ 3 , (18) \epsilon_n = A(\log n/n)^{\beta/(2\beta+d)}, \qquad n \ge3, \tag*{(18)} ϵ n = A ( log n / n ) β / ( 2 β + d ) , n ≥ 3 , (18) the following hold for all sufficiently large n n n .
(i) Rényi contraction. For every design law P X P_X P X ,
sup f 0 ∈ H d β ( M ) E f 0 Π n η ( D η ( f , f 0 ) > C ϵ n 2 ∣ D n ) ≤ 2 e − c n ϵ n 2 . (19) \sup_{f_0 \in\mathcal{H}_d^\beta(M)} \mathbb{E}_{f_0}\Pi_n^\eta\left(D_\eta(f,f_0) > C\epsilon_n^2 \mid D_n\right) \le2e^{-cn\epsilon_n^2}. \tag*{(19)} f 0 ∈ H d β ( M ) sup E f 0 Π n η ( D η ( f , f 0 ) > C ϵ n 2 ∣ D n ) ≤ 2 e − c n ϵ n 2 . (19) (ii) L 2 L^2 L 2 contraction after clipping. The cutoffs B ~ n \widetilde{B}_n B n of Theorem 1 can be chosen so that, in addition, the image Π ~ n η \widetilde{\Pi}_n^\eta Π n η of Π n η ( ⋅ ∣ D n ) \Pi_n^\eta(\cdot\mid D_n) Π n η ( ⋅ ∣ D n ) under f ↦ C B ~ n ∘ f f \mapsto C_{\widetilde{B}_n} \circ f f ↦ C B n ∘ f satisfies, whenever P X P_X P X has a density p X ≥ c X > 0 p_X \ge c_X > 0 p X ≥ c X > 0 Lebesgue-almost everywhere,
sup f 0 ∈ H d β ( M ) E f 0 Π ~ n η ( ∥ g − f 0 ∥ L 2 ( P X ) 2 > C ′ ϵ n 2 ∣ D n ) ≤ C ′ e − c ′ n ϵ n 2 (20) \sup_{f_0 \in\mathcal{H}_d^\beta(M)} \mathbb{E}_{f_0}\widetilde{\Pi}_n^\eta\left(\|g-f_0\|_{L^2(P_X)}^2 > C'\epsilon_n^2 \mid D_n\right) \le C'e^{-c'n\epsilon_n^2} \tag*{(20)} f 0 ∈ H d β ( M ) sup E f 0 Π n η ( ∥ g − f 0 ∥ L 2 ( P X ) 2 > C ′ ϵ n 2 ∣ D n ) ≤ C ′ e − c ′ n ϵ n 2 (20) for constants $C',c' > 0 that may also depend on c X c_X c X .
The prior is the same for every n n n , β \beta β and M M M , and the cutoff rule uses only n n n , d d d and the data.
Remark 1 (Computing the cutoff). For n ≥ 3 n \ge3 n ≥ 3 , put m n = ⌈ log n ⌉ m_n = \lceil\log n\rceil m n = ⌈ log n ⌉ and partition X \mathcal{X} X into J n = m n d J_n = m_n^d J n = m n d cubes Q j , n Q_{j,n} Q j , n of side 1 / m n 1/m_n 1/ m n , using half-open intervals with the outer faces included. Let N j , n N_{j,n} N j , n be the number of observations in cube j j j and Y ‾ j , n \overline{Y}_{j,n} Y j , n their response mean, set to zero when N j , n = 0 N_{j,n}=0 N j , n = 0 , and define
B ~ n = 2 + max 1 ≤ j ≤ J n ∣ Y ‾ j , n ∣ . (21) \widetilde{B}_n = 2 + \max_{1\le j\le J_n}|\overline{Y}_{j,n}|. \tag*{(21)} B n = 2 + 1 ≤ j ≤ J n max ∣ Y j , n ∣. (21)
Counts and response sums give this cutoff in one pass and O ( d n + J n ) O(dn+J_n) O ( d n + J n ) operations, without sample splitting or knowledge of β \beta β , M M M or P X P_X P X . The margin 2 absorbs cell-mean noise and within-cell variation. Under the lower density bound, for all large n n n ,
sup f 0 ∈ H d β ( M ) P f 0 { ∥ f 0 ∥ ∞ > B ~ n or B ~ n > M + 3 } ≤ C J n e − c n / J n . \sup_{f_0\in\mathcal{H}^{\beta}_d(M)} \mathbb{P}_{f_0}\{\|f_0\|_\infty> \widetilde{B}_n \text{ or } \widetilde{B}_n > M+3\} \leq C J_n e^{-cn/J_n}. f 0 ∈ H d β ( M ) sup P f 0 { ∥ f 0 ∥ ∞ > B n or B n > M + 3 } ≤ C J n e − c n / J n .
On the complementary event, clipping cannot increase Rényi divergence, and boundedness lets it control squared L 2 ( P X ) L^2(P_X) L 2 ( P X ) loss.
Proofs are in Appendices C and D. Appendix E extends the Rényi and arbitrary-design clipping results to bounded Borel covariate supports, using ambient Hölder continuity in R d \mathbb{R}^d R d .
Numerical assessment
Simulation setup
For d = 1 d=1 d = 1 , P X = Unif [ 0 , 1 ] P_X=\operatorname{Unif}[0,1] P X = Unif [ 0 , 1 ] , Gaussian errors with variance one, and η = 0.9 \eta=0.9 η = 0.9 , we assess the squared prediction error in Theorem 1 at the fixed targets
f β ( x ) = ∣ sin ( 2 π x ) ∣ β , β ∈ { 1 / 2 , 1 } . f_\beta(x)=|\sin(2\pi x)|^\beta,\qquad\beta\in\{1/2,1\}. f β ( x ) = ∣ sin ( 2 π x ) ∣ β , β ∈ { 1/2 , 1 } .
Their global Hölder bounds are 2 π \sqrt{2}\pi 2 π and 2 π 2\pi 2 π . Both have cusps at 1 / 2 1/2 1/2 and peaks at 1 / 4 1/4 1/4 and 3 / 4 3/4 3/4 . Neither has an exact finite shallow ReLU representation; see Appendix § 14 for verification of these facts.
For each target, sample size and parameter family, we use 50 replications and retain the first computation stage that passes the diagnostics in Appendix § 14 .§ 3 . The sample sizes are n ∈ { 2 j : j = 6 , … , 12 } n\in\{2^j:j=6,\ldots,12\} n ∈ { 2 j : j = 6 , … , 12 } .
The baseline (N/G) uses the Gaussian row of Table 1 and geometric width π G ( k ) = 2 − k \pi_G(k)=2^{-k} π G ( k ) = 2 − k . We estimate B ~ n \widetilde{B}_n B n from the fitting data and average clipped draws as in (15). Both targets are bounded by one and B ~ n ≥ 2 \widetilde{B}_n\geq2 B n ≥ 2 . Computation makes the simplification of restricting the width to K ≤ 64 K\leq64 K ≤ 64 and uses a 1009-point grid of values for numerical integration.
Figure 1 shows a Hölder 1/2 fit and a Lipschitz fit at each displayed sample size under the uniform parameter prior. The true central-notch value zero lies in none of the displayed pointwise intervals, although we note that these fractional-posterior intervals carry no frequentist coverage guarantee.
Figure 1. Example fits of the Hölder 1 / 2 1/2 1/2 case (left) and Lipschitz case (right), at n = 1024 n = 1024 n = 1024 (top) and 4096 (bottom), using the uniform parameter prior with geometric width and η = 0.9 \eta= 0.9 η = 0.9 . Truth (black), clipped posterior mean (blue), and pointwise 95% credible intervals (grey).
Rate and prior sensitivity
Figure 2 reports mean integrated squared errors with 95% normal Monte Carlo confidence intervals across the 50 replications per case (Appendix § 14 .§ 2 ). The grey guides show q n , β = ( log n / n ) 2 β / ( 2 β + 1 ) q_{n,\beta} = (\log n/n)^{2\beta/(2\beta+1)} q n , β = ( log n / n ) 2 β / ( 2 β + 1 ) (upper) and n − 2 β / ( 2 β + 1 ) n^{-2\beta/(2\beta+1)} n − 2 β / ( 2 β + 1 ) (lower), both matched to N/G at n = 64 n = 64 n = 64 with illustrative constants.
At n = 4096 n = 4096 n = 4096 , N/G mean errors are 0.00597 for β = 1 / 2 \beta= 1/2 β = 1/2 and 0.00408 for β = 1 \beta= 1 β = 1 . U/G and t 10 t_{10} t 10 /G mean errors are 0.00719 and 0.00681 for β = 1 / 2 \beta= 1/2 β = 1/2 , and 0.00469 and 0.00423 for β = 1 \beta= 1 β = 1 . Across sample sizes and both reweighted Gaussian width laws, the mean errors differ from N/G by at most 9.5% for β = 1 / 2 \beta= 1/2 β = 1/2 and 6.9% for β = 1 \beta= 1 β = 1 . For N/G, from 2048 to 4096, the estimated decay exponents are 0.749 for β = 1 / 2 \beta= 1/2 β = 1/2 and 0.707 for β = 1 \beta= 1 β = 1 , with estimated 95% paired-bootstrap intervals [0.630, 0.879] and [0.575, 0.839], respectively. These intervals are obtained by resampling the 50 replications with replacement, using the same sampled indices at both sample sizes, and recalculating the decay exponent [10 ] . They quantify simulation uncertainty while preserving the dependence between errors calculated from the same underlying data stream; Appendix § 14 .§ 2 gives details.
Posterior samples obtained under Gaussian network parameters and geometric width are reweighted to estimate the posterior mean under fixed shifted Poisson and power-law width priors satisfying (10) , conditional on K ≤ 64 K \le64 K ≤ 64 . Each case uses 50 replications whose fits pass the diagnostics in Appendix § 14 .§ 3 .
Discussion
The main conclusions depend on two local probability conditions: one on the width law and one on the conditional network-parameter law. A single prior satisfying them gives the stated rate for every Hölder smoothness and radius under consideration. Uniform, Gaussian, and Student network-parameter priors are distinct examples of this principle, rather than separate mechanisms. Data-dependent clipping then gives the same posterior-mean risk rate without a known target bound.
The logarithmic gap
The local prior-mass bound (13) has a log ( 1 / ε ) \log(1/\varepsilon) log ( 1/ ε ) factor, while the minimax lower bound has no logarithm. Requiring the network parameters to lie in narrow intervals around one approximator contributes this factor, and the width law can contribute a cost of the same order. These estimates do not show that the logarithmic loss is necessary.
A uniform bound − log Π { f : ∥ f − f 0 ∥ ∞ ≤ ε } ≲ ε − d / β -\log\Pi\{f : \lVert f-f_0\rVert_\infty\le\varepsilon\} \lesssim\varepsilon^{-d/\beta} − log Π { f : ∥ f − f 0 ∥ ∞ ≤ ε } ≲ ε − d / β would give the exact minimax rate through Lemma 1. A heavier-tailed width law alone does not remove the interval-probability cost. Possible improvements include collecting the probability of many accurate representations, using dependence between network parameters, and working with likelihood or prediction neighbourhoods rather than a single uniform-approximation box. The sufficient condition (12) is designed for transparent verification, not as a characterization of all priors that attain the rate.
Deep networks and other approximations
Using shallow networks keeps our approach simple, with explicit architecture and perturbation estimates. The method can also be considered for deep networks. For deep networks, the corresponding problem is to specify one prior on finite architectures and, conditional on each architecture, assign enough mass near accurate parameter configurations. The relevant approximation and stability bounds must track depth and parameter count, as well as coefficient size. Combining such bounds with a fixed architecture law would extend the same principle without choosing a new prior for each sample size or smoothness class. The resulting rates must be established from those bounds; they do not follow merely by replacing width with total parameter count.
The prior-mass principle is also not specific to neural networks. Lemma 1 applies to any prior on C ( X ) \mathcal{C}(\mathcal{X}) C ( X ) in the Gaussian regression model. Finite series, spline expansions, and other finite-dimensional approximation families offer settings in which to formulate analogous conditions on model size and local parameter probabilities. The objective is to identify verifiable conditions on the approximation family and prior that guarantee enough mass across all relevant scales. Outside Gaussian regression, the corresponding likelihood-neighbourhood and loss-transfer arguments would also need to be established; the clipping result here is a separate regression-specific step.
Further questions
Our results concern a fractional posterior with 0 < η < 1 0 < \eta< 1 0 < η < 1 . Its expected fractional likelihood ratio decays exponentially in Rényi divergence. At η = 1 \eta= 1 η = 1 the expected likelihood ratio is one, so this particular argument gives no contraction bound. An ordinary-posterior result would require additional control of inaccurate networks, including very large architectures; compare Kong and Kim [16 ] . The local conditions in this paper do not supply that global control.
We treat Hölder smoothness up to one. For 1 < β < ( d + 3 ) / 2 1 < \beta< (d + 3)/2 1 < β < ( d + 3 ) /2 , the shallow approximation bounds of Yang and Zhou [29 ] offer a starting point, with smoothness defined through derivatives and an appropriate extension beyond the cube. The clipping construction could then use the resulting Lipschitz bound. Deep-network results for smoother classes are available in Kong and Kim [16 ] .
Adaptive clipping removes the need to know the target range, but at small sample sizes its interval may fail to cover the whole range. A lower design-density bound gives eventual exponential control of this event. Remark 3 retains the same risk rate without that assumption, at the cost of an additional polynomial term in the posterior-tail bound. The present analysis does not determine the risk of the unclipped posterior mean, when it exists, or frequentist coverage of posterior credible balls.
The design law can permit faster rates than the uniform Hölder-ball upper bound. When the covariates equal one point, only one function value needs to be estimated and squared risk has order n − 1 n^{-1} n − 1 . Rates for designs concentrated on lower-dimensional sets require geometric information about the support; Castillo and Egels [8 ] study deep-network rates in terms of its upper Minkowski dimension.
The numerical width cap excludes larger networks, while finite posterior sampling adds computational error. Bounds keeping these errors below the statistical risk would connect the theory more directly to implementation. Cauchy priors also motivate samplers that explore large coefficients reliably. Extending the variational analysis of Castillo and Egels [8 ] to a fixed random-architecture prior is another route to studying computational approximations.
AI Use Statement
OpenAI ChatGPT and Codex were used for exposition, mathematical formulation, proof checking, proofreading, and formatting. The authors take responsibility for all parts of this manuscript.
Approximation theory
Lemma 3 (Hölder extension). Let A A A be a nonempty subset of a metric space ( S , d S ) (S,d_S) ( S , d S ) and let 0 < β ≤ 1 0 < \beta\le1 0 < β ≤ 1 . If h : A → R h:A \to\mathbb{R} h : A → R satisfies ∣ h ( a ) − h ( a ′ ) ∣ ≤ H d S ( a , a ′ ) β |h(a)-h(a')| \le H d_S(a,a')^\beta ∣ h ( a ) − h ( a ′ ) ∣ ≤ H d S ( a , a ′ ) β for all a , a ′ ∈ A a,a' \in A a , a ′ ∈ A and some H > 0 H > 0 H > 0 , then
h ~ ( z ) = inf a ∈ A { h ( a ) + H d S ( z , a ) β } , z ∈ S , (23) \widetilde{h}(z)=\inf_{a\in A}\{h(a)+H d_S(z,a)^\beta\},\qquad z\in S, \tag*{(23)} h ( z ) = a ∈ A inf { h ( a ) + H d S ( z , a ) β } , z ∈ S , (23) is a finite extension of h h h with the same Hölder constant. If h h h is bounded by M M M , clipping h ~ \widetilde{h} h to [ − M , M ] [-M,M] [ − M , M ] preserves both the extension and its Hölder bound.
Proof. For z , z ′ ∈ S z,z' \in S z , z ′ ∈ S and a ∈ A a \in A a ∈ A , subadditivity of t ↦ t β t \mapsto t^\beta t ↦ t β gives
d S ( z , a ) β ≤ d S ( z , z ′ ) β + d S ( z ′ , a ) β . d_S(z,a)^\beta\le d_S(z,z')^\beta+d_S(z',a)^\beta. d S ( z , a ) β ≤ d S ( z , z ′ ) β + d S ( z ′ , a ) β .
We first check that the infimum in (23) is finite. Fix a 0 ∈ A a_0 \in A a 0 ∈ A . The Hölder condition and the same subadditivity give, for every a ∈ A a \in A a ∈ A ,
h ( a ) + H d S ( z , a ) β ≥ h ( a 0 ) − H d S ( z , a 0 ) β . h(a)+H d_S(z,a)^\beta\ge h(a_0)-H d_S(z,a_0)^\beta. h ( a ) + H d S ( z , a ) β ≥ h ( a 0 ) − H d S ( z , a 0 ) β .
This bounds the infimum below; taking a = a 0 a=a_0 a = a 0 bounds it above.
If z ∈ A z \in A z ∈ A , the choice a = z a=z a = z gives h ~ ( z ) ≤ h ( z ) \widetilde{h}(z)\le h(z) h ( z ) ≤ h ( z ) . The Hölder condition gives h ( a ) + H d S ( z , a ) β ≥ h ( z ) h(a)+H d_S(z,a)^\beta\ge h(z) h ( a ) + H d S ( z , a ) β ≥ h ( z ) for every a ∈ A a \in A a ∈ A , proving equality. For arbitrary z , z ′ ∈ S z,z' \in S z , z ′ ∈ S , subadditivity gives
h ( a ) + H d S ( z , a ) β ≤ h ( a ) + H d S ( z ′ , a ) β + H d S ( z , z ′ ) β . h(a)+H d_S(z,a)^\beta\le h(a)+H d_S(z',a)^\beta+H d_S(z,z')^\beta. h ( a ) + H d S ( z , a ) β ≤ h ( a ) + H d S ( z ′ , a ) β + H d S ( z , z ′ ) β .
Taking infima and then interchanging z , z ′ z,z' z , z ′ proves the asserted Hölder bound. Finally, projection onto [ − M , M ] [-M,M] [ − M , M ] is one-Lipschitz and fixes every value of h h h , so clipping preserves both conclusions.
Proof of Proposition 1. We first place the target on the unit ball used by the approximation result, then rescale the approximating network back to X \mathcal{X} X . Let B d \mathbb{B}^d B d be the closed Euclidean unit ball in R d \mathbb{R}^d R d and S d \mathbb{S}^d S d the Euclidean unit sphere in R d + 1 \mathbb{R}^{d+1} R d + 1 . Set R = max { 1 , d } R=\max\{1,\sqrt{d}\} R = max { 1 , d } and X R = R − 1 X ⊆ B d \mathcal{X}_R=R^{-1}\mathcal{X}\subseteq\mathbb{B}^d X R = R − 1 X ⊆ B d . The function h ( z ) = f 0 ( R z ) h(z)=f_0(Rz) h ( z ) = f 0 ( R z ) on X R \mathcal{X}_R X R satisfies ∥ h ∥ ∞ ≤ M \|h\|_\infty\le M ∥ h ∥ ∞ ≤ M and
∣ h ( z ) − h ( z ′ ) ∣ ≤ M R β ∥ z − z ′ ∥ 2 β . |h(z)-h(z')|\le MR^\beta\|z-z'\|_2^\beta. ∣ h ( z ) − h ( z ′ ) ∣ ≤ M R β ∥ z − z ′ ∥ 2 β .
Lemma 3, followed by clipping to [ − M , M ] [-M,M] [ − M , M ] , extends h h h to R d \mathbb{R}^d R d with these same bounds. There is consequently a fixed K 0 ≥ 1 K_0\ge1 K 0 ≥ 1 , depending only on d d d , β \beta β , M M M , such that the restriction of h / K 0 h/K_0 h / K 0 to B d \mathbb{B}^d B d belongs to the unit Hölder ball in Yang and Zhou [29 ] .
For every positive integer N N N and bound V > 0 V>0 V > 0 on the sum of absolute output coefficients, that corollary supplies a network
g 0 ( z ) = ∑ m = 1 N c m ρ ( ( z , 1 ) ⊤ v m ) , v m ∈ S d , ∑ m = 1 N ∣ c m ∣ ≤ V , g_0(z)=\sum_{m=1}^{N}c_m\rho((z,1)^\top v_m),\qquad v_m\in\mathbb{S}^d,\qquad\sum_{m=1}^{N}|c_m|\le V, g 0 ( z ) = m = 1 ∑ N c m ρ (( z , 1 ) ⊤ v m ) , v m ∈ S d , m = 1 ∑ N ∣ c m ∣ ≤ V ,
with supremum error relative to h / K 0 h/K_0 h / K 0 on B d \mathbb{B}^d B d at most
C { N − β / d ∨ V − 2 β / ( d + 3 − 2 β ) } . C\left\{N^{-\beta/d}\vee V^{-2\beta/(d+3-2\beta)}\right\}. C { N − β / d ∨ V − 2 β / ( d + 3 − 2 β ) } .
Its smoothness condition holds because 0 < β ≤ 1 < ( d + 3 ) / 2 0<\beta\le1<(d+3)/2 0 < β ≤ 1 < ( d + 3 ) /2 . Take N = k N=k N = k and V = k ω − β / d V=k^{\omega-\beta/d} V = k ω − β / d . Since
2 β ( ω − β / d ) d + 3 − 2 β = β d , \frac{2\beta(\omega-\beta/d)}{d+3-2\beta}=\frac{\beta}{d}, d + 3 − 2 β 2 β ( ω − β / d ) = d β ,
both terms in the error bound have order k − β / d k^{-\beta/d} k − β / d . Write v m = ( w m , t m ) ∈ R d × R v_m=(w_m,t_m)\in\mathbb{R}^{d}\times\mathbb{R} v m = ( w m , t m ) ∈ R d × R and set
a m ∗ = K 0 c m , u m ∗ = w m / R , b m ∗ = t m . a_m^*=K_0c_m,\qquad u_m^*=w_m/R,\qquad b_m^*=t_m. a m ∗ = K 0 c m , u m ∗ = w m / R , b m ∗ = t m .
Then f θ k ∗ ( x ) = K 0 g 0 ( x / R ) f_{\theta_k^*}(x)=K_0g_0(x/R) f θ k ∗ ( x ) = K 0 g 0 ( x / R ) has supremum error at most C k − β / d Ck^{-\beta/d} C k − β / d on X \mathcal{X} X and a sum of absolute output coefficients at most K 0 k ω − β / d K_0k^{\omega-\beta/d} K 0 k ω − β / d . Units with zero parameters may be added if the representation uses fewer than k k k units. The sphere constraint and R ≥ 1 R\ge1 R ≥ 1 imply ∥ u m ∗ ∥ ∞ ≤ 1 \lVert u_m^*\rVert_\infty\le1 ∥ u m ∗ ∥ ∞ ≤ 1 and ∣ b m ∗ ∣ ≤ 1 \lvert b_m^*\rvert\le1 ∣ b m ∗ ∣ ≤ 1 . Every constant used above is uniform over the stated Hölder ball.
Prior probability of approximating networks
Fix 0 < β ≤ 1 0<\beta\le1 0 < β ≤ 1 and M > 0 M>0 M > 0 , and set
p = d / β , κ = ( d + 3 − 2 β ) / ( 2 β ) . (24) p=d/\beta,\qquad\kappa=(d+3-2\beta)/(2\beta). \tag*{(24)} p = d / β , κ = ( d + 3 − 2 β ) / ( 2 β ) . (24)
Choose C N C_N C N sufficiently large and put k ε = ⌈ C N ε − p ⌉ k_\varepsilon=\lceil C_N\varepsilon^{-p}\rceil k ε = ⌈ C N ε − p ⌉ . Proposition 1 gives an approximating network f ε = f θ k ε ∗ f_\varepsilon=f_{\theta_{k_\varepsilon}^*} f ε = f θ k ε ∗ with error at most ε / 4 \varepsilon/4 ε /4 and sum of absolute output coefficients V ε = ∑ m = 1 k ε ∣ a m ∗ ∣ ≲ ε − κ V_\varepsilon=\sum_{m=1}^{k_\varepsilon}\lvert a_m^*\rvert\lesssim\varepsilon^{-\kappa} V ε = ∑ m = 1 k ε ∣ a m ∗ ∣ ≲ ε − κ . For a fixed c δ > 0 c_\delta>0 c δ > 0 to be chosen below, define
δ ε = c δ ε / ( k ε + 1 + V ε ) , (25) \delta_\varepsilon=c_\delta\varepsilon/(k_\varepsilon+1+V_\varepsilon), \tag*{(25)} δ ε = c δ ε / ( k ε + 1 + V ε ) , (25)
and the parameter box
E ε = { θ ∈ R ( d + 2 ) k ε : ∥ θ − θ k ε ∗ ∥ ∞ ≤ δ ε } . (26) E_\varepsilon=\{\theta\in\mathbb{R}^{(d+2)k_\varepsilon}:\lVert\theta-\theta_{k_\varepsilon}^*\rVert_\infty\le\delta_\varepsilon\}. \tag*{(26)} E ε = { θ ∈ R ( d + 2 ) k ε : ∥ θ − θ k ε ∗ ∥ ∞ ≤ δ ε } . (26)
All constants below are uniform over the target Hölder ball.
Lemma 4 (Box stability). For sufficiently small fixed c δ > 0 c_\delta>0 c δ > 0 and all sufficiently small ε \varepsilon ε , every parameter vector in E ε E_\varepsilon E ε represents a function f f f satisfying ∥ f − f ε ∥ ∞ ≤ ε / 4 \lVert f-f_\varepsilon\rVert_\infty\le\varepsilon/4 ∥ f − f ε ∥ ∞ ≤ ε /4 and ∥ f − f 0 ∥ ∞ ≤ ε / 2 \lVert f-f_0\rVert_\infty\le\varepsilon/2 ∥ f − f 0 ∥ ∞ ≤ ε /2 . Moreover, log ( 1 / δ ε ) ≲ log ( 1 / ε ) \log(1/\delta_\varepsilon)\lesssim\log(1/\varepsilon) log ( 1/ δ ε ) ≲ log ( 1/ ε ) .
Proof of Lemma 4. Put k = k ε k=k_\varepsilon k = k ε and δ = δ ε \delta=\delta_\varepsilon δ = δ ε . Fix θ = ( ( a m , u m , b m ) ) m = 1 k ∈ E ε \theta=((a_m,u_m,b_m))_{m=1}^k\in E_\varepsilon θ = (( a m , u m , b m ) ) m = 1 k ∈ E ε and let f f f be its network. Put D = d + 1 D=d+1 D = d + 1 and abbreviate V = V ε V=V_\varepsilon V = V ε . For each x ∈ X x\in\mathcal{X} x ∈ X ,
∣ ( u m − u m ∗ ) T x + b m − b m ∗ ∣ ≤ D δ . \left|(u_m-u_m^*)^\mathsf{T}x+b_m-b_m^*\right|\le D\delta. ( u m − u m ∗ ) T x + b m − b m ∗ ≤ D δ .
For sufficiently small ε \varepsilon ε , we have δ ≤ 1 \delta\le1 δ ≤ 1 . Since the approximating hidden-layer coefficients and biases are bounded by one, ∣ ρ ( u m T x + b m ) ∣ ≤ 2 D \lvert\rho(u_m^\mathsf{T}x+b_m)\rvert\le2D ∣ ρ ( u m T x + b m )∣ ≤ 2 D . Subtract and add a m ∗ ρ ( u m T x + b m ) a_m^*\rho(u_m^\mathsf{T}x+b_m) a m ∗ ρ ( u m T x + b m ) to separate the perturbation of the output coefficient from that of the hidden-layer coefficients and bias. ReLU is one-Lipschitz, so
∣ a m ρ ( u m T x + b m ) − a m ∗ ρ ( ( u m ∗ ) T x + b m ∗ ) ∣ ≤ ∣ a m − a m ∗ ∣ ∣ ρ ( u m T x + b m ) ∣ + ∣ a m ∗ ∣ D δ ≤ 2 D δ + D ∣ a m ∗ ∣ δ . \begin{aligned}
\left|a_m\rho(u_m^\mathsf{T}x+b_m)-a_m^*\rho((u_m^*)^\mathsf{T}x+b_m^*)\right|
&\le|a_m-a_m^*|\lvert\rho(u_m^\mathsf{T}x+b_m)\rvert+|a_m^*|D\delta\\
&\le2D\delta+D|a_m^*|\delta.
\end{aligned} a m ρ ( u m T x + b m ) − a m ∗ ρ (( u m ∗ ) T x + b m ∗ ) ≤ ∣ a m − a m ∗ ∣ ∣ ρ ( u m T x + b m )∣ + ∣ a m ∗ ∣ D δ ≤ 2 D δ + D ∣ a m ∗ ∣ δ .
Summing these inequalities gives
∥ f − f ε ∥ ∞ ≤ D δ ( 2 k + V ) ≤ 2 D c δ ε . \lVert f-f_\varepsilon\rVert_\infty\le D\delta(2k+V)\le2Dc_\delta\varepsilon. ∥ f − f ε ∥ ∞ ≤ D δ ( 2 k + V ) ≤ 2 D c δ ε .
Taking c δ ≤ ( 8 D ) − 1 c_\delta\le(8D)^{-1} c δ ≤ ( 8 D ) − 1 proves the first error bound. Proposition 1 and the triangle inequality give ∥ f − f 0 ∥ ∞ ≤ ε / 2 \lVert f-f_0\rVert_\infty\le\varepsilon/2 ∥ f − f 0 ∥ ∞ ≤ ε /2 .
Finally, k + 1 + V ≲ ε − max { p , κ } k+1+V\lesssim\varepsilon^{-\max\{p,\kappa\}} k + 1 + V ≲ ε − m a x { p , κ } . Thus δ ≳ ε 1 + max { p , κ } \delta\gtrsim\varepsilon^{1+\max\{p,\kappa\}} δ ≳ ε 1 + m a x { p , κ } , and taking logarithms proves the remaining assertion. These estimates and choices of constants are uniform over the target class.
For sufficiently small ε \varepsilon ε , the centre is admissible in (11) , because V ε ≤ C k ε ω − β / d ≤ k ε ω / 2 V_\varepsilon\le Ck_\varepsilon^{\omega-\beta/d}\le k_\varepsilon^\omega/2 V ε ≤ C k ε ω − β / d ≤ k ε ω /2 . Also δ ε ≤ 1 / 2 \delta_\varepsilon\le1/2 δ ε ≤ 1/2 . Condition (12) therefore gives
− log Π k ε θ ( E ε ) ≲ k ε log ( 1 / ε ) . (27) -\log\Pi_{k_\varepsilon}^{\theta}(E_\varepsilon)\lesssim k_\varepsilon\log(1/\varepsilon). \tag*{(27)} − log Π k ε θ ( E ε ) ≲ k ε log ( 1/ ε ) . (27)
Proof of Proposition 2. If the sampled width is k ε k_\varepsilon k ε and every network parameter lies in its interval in E ε \mathcal{E}_\varepsilon E ε , Lemma 4 gives ∥ F − f 0 ∥ ∞ ≤ ε / 2 \lVert F-f_0\rVert_\infty\leq\varepsilon/2 ∥ F − f 0 ∥ ∞ ≤ ε /2 for the resulting network F F F . Thus { K = k ε , θ k ε ∈ E ε } \{K=k_\varepsilon,\theta_{k_\varepsilon}\in\mathcal{E}_\varepsilon\} { K = k ε , θ k ε ∈ E ε } is contained in { ∥ F − f 0 ∥ ∞ ≤ ε } \{\lVert F-f_0\rVert_\infty\leq\varepsilon\} {∥ F − f 0 ∥ ∞ ≤ ε } . The probability of choosing this width and these parameter values is π K ( k ε ) Π k ε θ ( E ε ) \pi_K(k_\varepsilon)\Pi^\theta_{k_\varepsilon}(\mathcal{E}_\varepsilon) π K ( k ε ) Π k ε θ ( E ε ) . Therefore
− log Π { f : ∥ f − f 0 ∥ ∞ ≤ ε } ≤ − log π K ( k ε ) − log Π k ε θ ( E ε ) -\log\Pi\{f:\lVert f-f_0\rVert_\infty\leq\varepsilon\}\leq-\log\pi_K(k_\varepsilon)-\log\Pi^\theta_{k_\varepsilon}(\mathcal{E}_\varepsilon) − log Π { f : ∥ f − f 0 ∥ ∞ ≤ ε } ≤ − log π K ( k ε ) − log Π k ε θ ( E ε )
≲ k ε log ( 1 / ε ) ≲ ε − d / β log ( 1 / ε ) . \lesssim k_\varepsilon\log(1/\varepsilon)\lesssim\varepsilon^{-d/\beta}\log(1/\varepsilon). ≲ k ε log ( 1/ ε ) ≲ ε − d / β log ( 1/ ε ) .
The second inequality uses (10) and (27) ; the last uses the chosen width k ε ≍ ε − d / β k_\varepsilon\asymp\varepsilon^{-d/\beta} k ε ≍ ε − d / β . All constants are uniform over f 0 ∈ H d β ( M ) f_0\in\mathcal{H}^{\beta}_d(M) f 0 ∈ H d β ( M ) .
Fractional posterior and clipped loss
The clipping level B ~ n \widetilde{B}_n B n is a finite measurable function of the data. The map ( b , f ) ↦ C b ∘ f (b,f)\mapsto C_b\circ f ( b , f ) ↦ C b ∘ f is continuous for b > 0 b>0 b > 0 and f ∈ C ( X ) f\in C(\mathcal{X}) f ∈ C ( X ) , so the clipped posterior Π ~ n η \widetilde{\Pi}^{\eta}_n Π n η is a measurable probability kernel. For each data set, the integrand in (15) is bounded by B ~ n \widetilde{B}_n B n . Dominated convergence therefore makes f ~ n \widetilde{f}_n f n continuous in x x x . Its values at a countable dense subset of X \mathcal{X} X are measurable functions of the data, which makes f ~ n \widetilde{f}_n f n a measurable C ( X ) C(\mathcal{X}) C ( X ) -valued estimator.
Existence of the unclipped posterior mean. Take d = 1 d=1 d = 1 , P X = Unif [ 0 , 1 ] P_X=\operatorname{Unif}[0,1] P X = Unif [ 0 , 1 ] , f 0 = 0 f_0=0 f 0 = 0 , and independent Student t ν t_\nu t ν network parameters with 0 < ν ≤ 1 0<\nu\leq1 0 < ν ≤ 1 , including Cauchy network parameters at ν = 1 \nu=1 ν = 1 . Condition (10) gives π K ( 1 ) > 0 \pi_K(1)>0 π K ( 1 ) > 0 . Fix a realized data set with m = max i ≤ n X i < 1 m=\max_{i\leq n}X_i<1 m = max i ≤ n X i < 1 and finite responses; this holds almost surely at every finite sample size. Set t 0 = ( m + 1 ) / 2 t_0=(m+1)/2 t 0 = ( m + 1 ) /2 , I = ( t 0 , 1 ] I=(t_0,1] I = ( t 0 , 1 ] , and
H m = { 1 < U 1 < 2 , m < − B 1 / U 1 < t 0 } . \mathcal{H}_m=\{1<U_1<2,\ m<-B_1/U_1<t_0\}. H m = { 1 < U 1 < 2 , m < − B 1 / U 1 < t 0 } .
This event has positive prior probability conditional on K = 1 K=1 K = 1 . On { K = 1 } ∩ H m \{K=1\}\cap\mathcal{H}_m { K = 1 } ∩ H m , all training activations vanish, whereas ρ ( U 1 x + B 1 ) ≥ x − t 0 > 0 \rho(U_1x+B_1)\geq x-t_0>0 ρ ( U 1 x + B 1 ) ≥ x − t 0 > 0 for x ∈ I x\in I x ∈ I . The fractional likelihood is therefore L η = exp { − η ∑ i = 1 n Y i 2 / 2 } > 0 L_\eta=\exp\{-\eta\sum_{i=1}^{n}Y_i^2/2\}>0 L η = exp { − η ∑ i = 1 n Y i 2 /2 } > 0 , independently of A 1 A_1 A 1 . Let Z η ∈ ( 0 , 1 ] Z_\eta\in(0,1] Z η ∈ ( 0 , 1 ] be the normalizing constant in (3) , and write z + = max { z , 0 } z_+=\max\{z,0\} z + = max { z , 0 } and z − = max { − z , 0 } z_-=\max\{-z,0\} z − = max { − z , 0 } . The Student density has tails of order ∣ a ∣ − ν − 1 |a|^{-\nu-1} ∣ a ∣ − ν − 1 , so both E [ ( A 1 ) + ∣ K = 1 ] \mathbb{E}[(A_1)_+\mid K=1] E [( A 1 ) + ∣ K = 1 ] and E [ ( A 1 ) − ∣ K = 1 ] \mathbb{E}[(A_1)_-\mid K=1] E [( A 1 ) − ∣ K = 1 ] are infinite. Independence and Tonelli’s theorem give, for every x ∈ I x\in I x ∈ I ,
∫ C ( X ) f ( x ) ± Π n η ( d f ∣ D n ) ≥ π K ( 1 ) L η Z η ( x − t 0 ) P ( H m ∣ K = 1 ) \int_{C(\mathcal{X})}f(x)_\pm\Pi^\eta_n(df\mid\mathcal{D}_n)\geq\frac{\pi_K(1)L_\eta}{Z_\eta}(x-t_0)\mathbb{P}(\mathcal{H}_m\mid K=1) ∫ C ( X ) f ( x ) ± Π n η ( df ∣ D n ) ≥ Z η π K ( 1 ) L η ( x − t 0 ) P ( H m ∣ K = 1 )
× E [ ( A 1 ) ± ∣ K = 1 ] = ∞ . {}\times\mathbb{E}[(A_1)_\pm\mid K=1]=\infty. × E [( A 1 ) ± ∣ K = 1 ] = ∞.
Thus the unclipped posterior mean is undefined on an interval of positive P X P_X P X -measure. This concerns the mean’s existence, rather than posterior contraction in probability. The clipped averages remain well defined without any moment condition on the network parameters.
Lemma 5 (Clipping with target-range bias). For any b > 0 b>0 b > 0 , f , f 0 ∈ C ( X ) f,f_0\in C(\mathcal{X}) f , f 0 ∈ C ( X ) , and 0 < η < 1 0<\eta<1 0 < η < 1 ,
∥ C b ∘ f − f 0 ∥ L 2 ( P X ) 2 ≤ 2 K η ( b ) D η ( f , f 0 ) + 2 ∥ C b ∘ f 0 − f 0 ∥ L 2 ( P X ) 2 , (28) \lVert C_b\circ f-f_0\rVert_{L^2(P_X)}^2\leq2K_\eta(b)D_\eta(f,f_0)+2\lVert C_b\circ f_0-f_0\rVert_{L^2(P_X)}^2, \tag*{(28)} ∥ C b ∘ f − f 0 ∥ L 2 ( P X ) 2 ≤ 2 K η ( b ) D η ( f , f 0 ) + 2 ∥ C b ∘ f 0 − f 0 ∥ L 2 ( P X ) 2 , (28) where K η ( b ) = 4 ( 1 − η ) b 2 / { 1 − e − 2 η ( 1 − η ) b 2 } K_\eta(b)=4(1-\eta)b^2/\{1-e^{-2\eta(1-\eta)b^2}\} K η ( b ) = 4 ( 1 − η ) b 2 / { 1 − e − 2 η ( 1 − η ) b 2 } . If b ≥ ∥ f 0 ∥ ∞ b\geq\lVert f_0\rVert_\infty b ≥ ∥ f 0 ∥ ∞ , the second term vanishes and the factor 2 in front of the first term can be omitted.
Proof. Set a = η ( 1 − η ) / 2 a=\eta(1-\eta)/2 a = η ( 1 − η ) /2 and T = ( f ( X ) − f 0 ( X ) ) 2 T=(f(X)-f_0(X))^2 T = ( f ( X ) − f 0 ( X ) ) 2 . Clipping is one-Lipschitz, and both clipped values lie in [ − b , b ] [-b,b] [ − b , b ] . Hence
∣ C b ( f ( X ) ) − C b ( f 0 ( X ) ) ∣ 2 ≤ min { T , 4 b 2 } . |C_b(f(X))-C_b(f_0(X))|^2\leq\min\{T,4b^2\}. ∣ C b ( f ( X )) − C b ( f 0 ( X )) ∣ 2 ≤ min { T , 4 b 2 } .
On [ 0 , 4 b 2 ] [0,4b^{2}] [ 0 , 4 b 2 ] , the concave function t ↦ 1 − e − a t t \mapsto1-e^{-at} t ↦ 1 − e − a t lies above the chord joining its endpoints. For t ≥ 4 b 2 t \ge4b^{2} t ≥ 4 b 2 it is increasing. Combining these two facts gives
1 − e − a T ≥ 1 − e − 4 a b 2 4 b 2 min { T , 4 b 2 } . (29) 1-e^{-aT} \ge\frac{1-e^{-4ab^{2}}}{4b^{2}}\min\{T,4b^{2}\}. \tag*{(29)} 1 − e − a T ≥ 4 b 2 1 − e − 4 a b 2 min { T , 4 b 2 } . (29)
By (4) and − log z ≥ 1 − z -\log z \ge1-z − log z ≥ 1 − z ,
( 1 − η ) D η ( f , f 0 ) ≥ E X ( 1 − e − a T ) . (1-\eta)D_{\eta}(f,f_{0}) \ge\mathbb{E}_{X}(1-e^{-aT}). ( 1 − η ) D η ( f , f 0 ) ≥ E X ( 1 − e − a T ) .
Taking expectations in (29) therefore yields
∥ C b ∘ f − C b ∘ f 0 ∥ L 2 ( P X ) 2 ≤ K η ( b ) D η ( f , f 0 ) . \|C_{b}\circ f-C_{b}\circ f_{0}\|_{L^{2}(P_{X})}^{2} \le K_{\eta}(b)D_{\eta}(f,f_{0}). ∥ C b ∘ f − C b ∘ f 0 ∥ L 2 ( P X ) 2 ≤ K η ( b ) D η ( f , f 0 ) .
Now write C b ∘ f − f 0 C_{b}\circ f-f_{0} C b ∘ f − f 0 as the sum of C b ∘ f − C b ∘ f 0 C_{b}\circ f-C_{b}\circ f_{0} C b ∘ f − C b ∘ f 0 and C b ∘ f 0 − f 0 C_{b}\circ f_{0}-f_{0} C b ∘ f 0 − f 0 . The inequality ( u + v ) 2 ≤ 2 u 2 + 2 v 2 (u+v)^{2}\le2u^{2}+2v^{2} ( u + v ) 2 ≤ 2 u 2 + 2 v 2 , integrated over P X P_{X} P X , proves (28) . If b ≥ ∥ f 0 ∥ ∞ b\ge\|f_{0}\|_{\infty} b ≥ ∥ f 0 ∥ ∞ , then C b ∘ f 0 = f 0 C_{b}\circ f_{0}=f_{0} C b ∘ f 0 = f 0 , so this final step and its factor two are unnecessary.
Proof of Theorem 2. The prior-probability bound (13) is uniform over H d β ( M ) \mathcal{H}_{d}^{\beta}(M) H d β ( M ) . For ε n \varepsilon_{n} ε n in (18) ,
ε n − d / β log ( 1 / ε n ) ≲ A − d / β n d / ( 2 β + d ) ( log n ) 2 β / ( 2 β + d ) , \varepsilon_{n}^{-d/\beta}\log(1/\varepsilon_{n})\lesssim A^{-d/\beta}n^{d/(2\beta+d)}(\log n)^{2\beta/(2\beta+d)}, ε n − d / β log ( 1/ ε n ) ≲ A − d / β n d / ( 2 β + d ) ( log n ) 2 β / ( 2 β + d ) ,
whereas n ε n 2 = A 2 n d / ( 2 β + d ) ( log n ) 2 β / ( 2 β + d ) n\varepsilon_{n}^{2}=A^{2}n^{d/(2\beta+d)}(\log n)^{2\beta/(2\beta+d)} n ε n 2 = A 2 n d / ( 2 β + d ) ( log n ) 2 β / ( 2 β + d ) . Choose the fixed A A A large enough that the prior probability is at least e − n ε n 2 e^{-n\varepsilon_{n}^{2}} e − n ε n 2 for all sufficiently large n n n . Since n ε n 2 → ∞ n\varepsilon_{n}^{2}\to\infty n ε n 2 → ∞ , Lemma 1 applies with c 0 = 1 c_{0}=1 c 0 = 1 . Choose J 0 > ( 1 + η ) / ( 1 − η ) J_{0}>(1+\eta)/(1-\eta) J 0 > ( 1 + η ) / ( 1 − η ) and put
C = J 0 , c = min { 1 / 8 , ( 1 − η ) J 0 − 1 − η } > 0. C=J_{0},\qquad c=\min\{1/8,(1-\eta)J_{0}-1-\eta\}>0. C = J 0 , c = min { 1/8 , ( 1 − η ) J 0 − 1 − η } > 0.
The two terms in that lemma are at most e − c n ε n 2 e^{-cn\varepsilon_{n}^{2}} e − c n ε n 2 each, proving (19) .
Write p j = P X ( Q j , n ) p_{j}=P_{X}(Q_{j,n}) p j = P X ( Q j , n ) . Equal cell volumes and the lower density bound imply p j ≥ c X / J n p_{j}\ge c_{X}/J_{n} p j ≥ c X / J n . Let E n \mathcal{E}_{n} E n be the event that every cell contains at least c X n / ( 2 J n ) c_{X}n/(2J_{n}) c X n / ( 2 J n ) observations. The binomial Chernoff bound and a union bound over cells give
P ( E n c ) ≤ J n exp { − c X n / ( 8 J n ) } . \mathbb{P}(\mathcal{E}_{n}^{c})\le J_{n}\exp\{-c_{X}n/(8J_{n})\}. P ( E n c ) ≤ J n exp { − c X n / ( 8 J n )} .
Let ξ ˉ j , n \bar{\xi}_{j,n} ξ ˉ j , n denote the noise mean in cell j j j , with value zero if the cell is empty. Conditional on the covariates, each occupied-cell noise mean has law N ( 0 , 1 / N j , n ) N(0,1/N_{j,n}) N ( 0 , 1/ N j , n ) . Consequently, on E n \mathcal{E}_{n} E n ,
P { max j ∣ ξ ˉ j , n ∣ > 1 ∣ X 1 , … , X n } ≤ 2 J n exp { − c X n / ( 4 J n ) } . \mathbb{P}\{\max_{j}|\bar{\xi}_{j,n}|>1\mid X_{1},\ldots,X_{n}\}\le2J_{n}\exp\{-c_{X}n/(4J_{n})\}. P { j max ∣ ξ ˉ j , n ∣ > 1 ∣ X 1 , … , X n } ≤ 2 J n exp { − c X n / ( 4 J n )} .
Define G n = E n ∩ { max j ∣ ξ ˉ j , n ∣ ≤ 1 } \mathcal{G}_{n}=\mathcal{E}_{n}\cap\{\max_{j}|\bar{\xi}_{j,n}|\le1\} G n = E n ∩ { max j ∣ ξ ˉ j , n ∣ ≤ 1 } . Combining the two failure probabilities gives
P f 0 ( G n c ) ≤ 3 J n exp { − c X n / ( 8 J n ) } . (30) \mathbb{P}_{f_{0}}(\mathcal{G}_{n}^{c})\le3J_{n}\exp\{-c_{X}n/(8J_{n})\}. \tag*{(30)} P f 0 ( G n c ) ≤ 3 J n exp { − c X n / ( 8 J n )} . (30)
This estimate is uniform over the target class. Since J n ≍ ( log n ) d J_{n}\asymp(\log n)^{d} J n ≍ ( log n ) d ,
n / J n n ε n 2 ≲ A − 2 n 2 β / ( 2 β + d ) ( log n ) − d − 2 β / ( 2 β + d ) ⟶ ∞ . \frac{n/J_{n}}{n\varepsilon_{n}^{2}}\lesssim A^{-2}n^{2\beta/(2\beta+d)}(\log n)^{-d-2\beta/(2\beta+d)}\longrightarrow\infty. n ε n 2 n / J n ≲ A − 2 n 2 β / ( 2 β + d ) ( log n ) − d − 2 β / ( 2 β + d ) ⟶ ∞.
Also log J n = o ( n / J n ) \log J_{n}=o(n/J_{n}) log J n = o ( n / J n ) . Thus the right side of (30) is eventually at most e − c n ε n 2 e^{-cn\varepsilon_{n}^{2}} e − c n ε n 2 for the same fixed c c c chosen above.
For large enough n n n , M m n − β ≤ 1 Mm_{n}^{-\beta}\le1 M m n − β ≤ 1 . On G n \mathcal{G}_{n} G n , every cell is occupied. Its max-norm diameter is 1 / m n 1/m_{n} 1/ m n , so for x ∈ Q j , n x\in Q_{j,n} x ∈ Q j , n , averaging the Hölder inequality over the sampled covariates in that cell gives
∣ f 0 ( x ) ∣ ≤ ∣ Y ‾ j , n ∣ + ∣ ξ ˉ j , n ∣ + M m n − β ≤ B ‾ n . (31) |f_{0}(x)|\le|\overline{Y}_{j,n}|+|\bar{\xi}_{j,n}|+Mm_{n}^{-\beta}\le\overline{B}_{n}. \tag*{(31)} ∣ f 0 ( x ) ∣ ≤ ∣ Y j , n ∣ + ∣ ξ ˉ j , n ∣ + M m n − β ≤ B n . (31)
In addition, ∣ Y ‾ j , n ∣ ≤ M + 1 \lvert\overline{Y}_{j,n}\rvert\le M+1 ∣ Y j , n ∣ ≤ M + 1 on this event, so B ~ n ≤ M + 3 \widetilde{B}_n\le M+3 B n ≤ M + 3 . By (31) , clipping leaves the target unchanged on G n \mathcal{G}_n G n . The final assertion of Lemma 5 gives
∥ C B ~ n ∘ f − f 0 ∥ L 2 ( P X ) 2 ≤ K η ( M + 3 ) D η ( f , f 0 ) on G n . (32) \left\|C_{\widetilde{B}_n}\circ f-f_0\right\|_{L^2(P_X)}^2\le K_\eta(M+3)D_\eta(f,f_0)\quad\text{on }\mathcal{G}_n. \tag*{(32)} C B n ∘ f − f 0 L 2 ( P X ) 2 ≤ K η ( M + 3 ) D η ( f , f 0 ) on G n . (32)
Here K η ( b ) K_\eta(b) K η ( b ) is increasing in b > 0 b>0 b > 0 , as follows from the monotonicity of t / ( 1 − e − t ) t/(1-e^{-t}) t / ( 1 − e − t ) for t > 0 t>0 t > 0 . Take C ′ = max { C , C K η ( M + 3 ) , 3 } C'=\max\{C,CK_\eta(M+3),3\} C ′ = max { C , C K η ( M + 3 ) , 3 } and c ′ = c c'=c c ′ = c . On G n \mathcal{G}_n G n , clipped loss exceeding C ′ ε n 2 C'\varepsilon_n^2 C ′ ε n 2 implies D η ( f , f 0 ) > C ε n 2 D_\eta(f,f_0)>C\varepsilon_n^2 D η ( f , f 0 ) > C ε n 2 . The expected posterior probability is therefore bounded by the Rényi tail plus P f 0 ( G n c ) P_{f_0}(\mathcal{G}_n^c) P f 0 ( G n c ) , hence by 3 e − c n ε n 2 3e^{-cn\varepsilon_n^2} 3 e − c n ε n 2 . Our choice C ′ ≥ 3 C'\ge3 C ′ ≥ 3 proves (20).
If a fixed B ≥ M B\ge M B ≥ M is prescribed, Lemma 5 gives ∥ C B ∘ f − f 0 ∥ L 2 ( P X ) 2 ≤ K η ( B ) D η ( f , f 0 ) \left\|C_B\circ f-f_0\right\|_{L^2(P_X)}^2\le K_\eta(B)D_\eta(f,f_0) ∥ C B ∘ f − f 0 ∥ L 2 ( P X ) 2 ≤ K η ( B ) D η ( f , f 0 ) . The Rényi bound then gives a pure exponential posterior tail at a suitable constant multiple of ε n 2 \varepsilon_n^2 ε n 2 . This comparison does not require estimating B B B .
A clipping level for arbitrary designs
For the alternative in Remark 3, set s n = max { 1 , ⌊ n 1 / ( 4 d ) ⌋ } s_n=\max\{1,\lfloor n^{1/(4d)}\rfloor\} s n = max { 1 , ⌊ n 1/ ( 4 d ) ⌋} . Divide each coordinate interval into s n s_n s n equal intervals and take their Cartesian products. Use left-closed, right-open intervals, except the last interval, which includes one. This gives a disjoint partition of X \mathcal{X} X into H n = s n d ≤ n 1 / 4 H_n=s_n^d\le n^{1/4} H n = s n d ≤ n 1/4 cubes Q j , n ∘ Q^\circ_{j,n} Q j , n ∘ , each with max-norm diameter 1 / s n 1/s_n 1/ s n . Write N j , n ∘ N^\circ_{j,n} N j , n ∘ and Y ‾ j , n ∘ \overline{Y}^\circ_{j,n} Y j , n ∘ for their counts and response means. Set t n = ⌈ 8 log ( e n ) ⌉ t_n=\lceil8\log(en)\rceil t n = ⌈ 8 log ( e n )⌉ . A cell is eligible if it contains at least t n t_n t n observations. Define
B ~ n ∘ = 2 + max j : N j , n ∘ ≥ t n ∣ Y ‾ j , n ∘ ∣ , f ~ n ∘ ( x ) = ∫ C ( X ) C B ~ n ∘ ( f ( x ) ) Π η ( d f ∣ D n ) , (33) \widetilde{B}^\circ_n=2+\max_{j:N^\circ_{j,n}\ge t_n}\left\lvert\overline{Y}^\circ_{j,n}\right\rvert,\qquad
\widetilde{f}^\circ_n(x)=\int_{\mathcal{C}(\mathcal{X})}C_{\widetilde{B}^\circ_n}(f(x))\,\Pi_\eta(df\mid D_n), \tag*{(33)} B n ∘ = 2 + j : N j , n ∘ ≥ t n max Y j , n ∘ , f n ∘ ( x ) = ∫ C ( X ) C B n ∘ ( f ( x )) Π η ( df ∣ D n ) , (33)
with an empty maximum equal to zero. This procedure also uses neither β \beta β nor M M M .
Proof of Remark 3. Choose the fixed Rényi threshold J 0 > ( 1 + η ) / ( 1 − η ) J_0>(1+\eta)/(1-\eta) J 0 > ( 1 + η ) / ( 1 − η ) as in the proof of Theorem 2. Let G n ∘ \mathcal{G}^\circ_n G n ∘ be the event that every eligible cell has noise mean of absolute value at most one. Let
U n ∘ = ⋃ j : N j , n ∘ < t n Q j , n ∘ \mathcal{U}^\circ_n=\bigcup_{j:N^\circ_{j,n}<t_n}Q^\circ_{j,n} U n ∘ = j : N j , n ∘ < t n ⋃ Q j , n ∘
be the union of the ineligible cells.
Conditional Gaussian tails and N j , n ∘ ≥ t n N^\circ_{j,n}\ge t_n N j , n ∘ ≥ t n for eligible cells give
P f 0 { ( G n ∘ ) c } ≤ 2 H n e − t n / 2 ≲ n − 15 / 4 . (34) P_{f_0}\{(\mathcal{G}^\circ_n)^c\}\le2H_ne^{-t_n/2}\lesssim n^{-15/4}. \tag*{(34)} P f 0 {( G n ∘ ) c } ≤ 2 H n e − t n /2 ≲ n − 15/4 . (34)
For sufficiently large n n n , M s n − β ≤ 1 Ms_n^{-\beta}\le1 M s n − β ≤ 1 . On G n ∘ \mathcal{G}^\circ_n G n ∘ , the clipping level lies in [ 2 , M + 3 ] [2,M+3] [ 2 , M + 3 ] . Within every eligible cell, the averaging argument used in (31) gives ∣ f 0 ( x ) ∣ ≤ ∣ Y ‾ j , n ∘ ∣ + 2 ≤ B ~ n ∘ \lvert f_0(x)\rvert\le\lvert\overline{Y}^\circ_{j,n}\rvert+2\le\widetilde{B}^\circ_n ∣ f 0 ( x )∣ ≤ ∣ Y j , n ∘ ∣ + 2 ≤ B n ∘ . Thus clipping can change the target only on U n ∘ \mathcal{U}^\circ_n U n ∘ . Since ∣ f 0 ∣ ≤ M \lvert f_0\rvert\le M ∣ f 0 ∣ ≤ M , it follows that
∥ C B ~ n ∘ ∘ f 0 − f 0 ∥ L 2 ( P X ) 2 ≤ M 2 P X ( U n ∘ ) on G n ∘ . (35) \left\|C_{\widetilde{B}^\circ_n}\circ f_0-f_0\right\|_{L^2(P_X)}^2\le M^2P_X(\mathcal{U}^\circ_n)\quad\text{on }\mathcal{G}^\circ_n. \tag*{(35)} C B n ∘ ∘ f 0 − f 0 L 2 ( P X ) 2 ≤ M 2 P X ( U n ∘ ) on G n ∘ . (35)
Write p j = P X ( Q j , n ∘ ) p_j=P_X(Q^\circ_{j,n}) p j = P X ( Q j , n ∘ ) . For p ∈ [ 0 , 1 ] p\in[0,1] p ∈ [ 0 , 1 ] and an integer t ≥ 1 t\ge1 t ≥ 1 , if N ∼ Bin ( n , p ) N\sim\operatorname{Bin}(n,p) N ∼ Bin ( n , p ) and N ′ ∼ Bin ( n + 1 , p ) N'\sim\operatorname{Bin}(n+1,p) N ′ ∼ Bin ( n + 1 , p ) , comparison of the binomial masses at l l l and l + 1 l+1 l + 1 gives
p P ( N < t ) = 1 n + 1 E { N ′ 1 { N ′ ≤ t } } ≤ t n + 1 . pP(N<t)=\frac{1}{n+1}\mathbb{E}\{N'\mathbf{1}_{\{N'\le t\}}\}\le\frac{t}{n+1}. pP ( N < t ) = n + 1 1 E { N ′ 1 { N ′ ≤ t } } ≤ n + 1 t .
Applying this identity to each cell and summing yields, uniformly over all P X P_X P X ,
E f 0 P X ( U n ∘ ) ≤ H n t n n + 1 ≲ n − 3 / 4 log ( e n ) = o ( ε n 2 ) . (36) \mathbb{E}_{f_0}P_X(\mathcal{U}^\circ_n)\le\frac{H_nt_n}{n+1}\lesssim n^{-3/4}\log(en)=o(\varepsilon_n^2). \tag*{(36)} E f 0 P X ( U n ∘ ) ≤ n + 1 H n t n ≲ n − 3/4 log ( e n ) = o ( ε n 2 ) . (36)
For the last comparison, 2 β / ( 2 β + d ) ≤ 2 / 3 2\beta/(2\beta+d)\le2/3 2 β / ( 2 β + d ) ≤ 2/3 under our assumptions, whereas 3 / 4 > 2 / 3 3/4>2/3 3/4 > 2/3 . We also need a tail bound for this random probability mass. For u > 0 u>0 u > 0 , if P X ( U n ∘ ) > u P_X(U_n^\circ)>u P X ( U n ∘ ) > u , the subset of ineligible cells has total probability greater than u u u and total count below H n t n H_nt_n H n t n . For each fixed subset I I I , its total count is Bin ( n , p I ) \operatorname{Bin}(n,p_I) Bin ( n , p I ) , where p I p_I p I is its total probability. If p I > u p_I>u p I > u and n u ≥ 2 H n t n nu\ge2H_nt_n n u ≥ 2 H n t n , Chernoff’s inequality bounds the probability that its count is below H n t n H_nt_n H n t n by e − n u / 8 e^{-nu/8} e − n u /8 . There are at most 2 H n 2^{H_n} 2 H n subsets, so
P f 0 { P X ( U n ∘ ) > u } ≤ 2 H n e − n u / 8 . (37) P_{f_0}\{P_X(U_n^\circ)>u\}\le2^{H_n}e^{-nu/8}. \tag*{(37)} P f 0 { P X ( U n ∘ ) > u } ≤ 2 H n e − n u /8 . (37)
Since d / ( 2 β + d ) ≥ 1 / 3 > 1 / 4 d/(2\beta+d)\ge1/3>1/4 d / ( 2 β + d ) ≥ 1/3 > 1/4 , we have n ε n 2 ≫ H n t n + H n n\varepsilon_n^2\gg H_nt_n+H_n n ε n 2 ≫ H n t n + H n . Taking u = a 0 ε n 2 u=a_0\varepsilon_n^2 u = a 0 ε n 2 with any fixed a 0 > 0 a_0>0 a 0 > 0 therefore makes (37) at most e − c n ε n 2 e^{-cn\varepsilon_n^2} e − c n ε n 2 for sufficiently large n n n .
On G n ∘ ∩ { P X ( U n ∘ ) ≤ a 0 ε n 2 } G_n^\circ\cap\{P_X(U_n^\circ)\le a_0\varepsilon_n^2\} G n ∘ ∩ { P X ( U n ∘ ) ≤ a 0 ε n 2 } , Lemma 5 implies that D η ( f , f 0 ) ≤ J 0 ε n 2 D_\eta(f,f_0)\le J_0\varepsilon_n^2 D η ( f , f 0 ) ≤ J 0 ε n 2 entails
∥ C B ~ n ∘ ∘ f − f 0 ∥ L 2 ( P X ) 2 ≤ { 2 K η ( M + 3 ) J 0 + 2 M 2 a 0 } ε n 2 . \left\|\mathcal{C}\widetilde{B}_n^\circ\circ f-f_0\right\|_{L^2(P_X)}^2\le\{2K_\eta(M+3)J_0+2M^2a_0\}\varepsilon_n^2. C B n ∘ ∘ f − f 0 L 2 ( P X ) 2 ≤ { 2 K η ( M + 3 ) J 0 + 2 M 2 a 0 } ε n 2 .
Lemma 1, (34), and (37) prove (22).
For expected risk, conditional Jensen’s inequality bounds the loss of the posterior mean by the posterior mean of the clipped losses. On G n ∘ G_n^\circ G n ∘ , split that integral at D η = J 0 ε n 2 D_\eta=J_0\varepsilon_n^2 D η = J 0 ε n 2 . The resulting bound is
2 K η ( M + 3 ) J 0 ε n 2 + 2 M 2 P X ( U n ∘ ) + ( 2 M + 3 ) 2 Π n η { D η ( f , f 0 ) > J 0 ε n 2 ∣ D n } . 2K_\eta(M+3)J_0\varepsilon_n^2+2M^2P_X(U_n^\circ)+(2M+3)^2\Pi_n^\eta\{D_\eta(f,f_0)>J_0\varepsilon_n^2\mid D_n\}. 2 K η ( M + 3 ) J 0 ε n 2 + 2 M 2 P X ( U n ∘ ) + ( 2 M + 3 ) 2 Π n η { D η ( f , f 0 ) > J 0 ε n 2 ∣ D n } .
Its expectation is O ( ε n 2 ) O(\varepsilon_n^2) O ( ε n 2 ) by (36) and Lemma 1. On ( G n ∘ ) c (G_n^\circ)^c ( G n ∘ ) c , B ~ n ∘ ≤ 2 + M + max i ≤ n ∣ ξ i ∣ \widetilde{B}_n^\circ\le2+M+\max_{i\le n}|\xi_i| B n ∘ ≤ 2 + M + max i ≤ n ∣ ξ i ∣ , so every clipped loss is at most ( 2 M + 2 + max i ≤ n ∣ ξ i ∣ ) 2 (2M+2+\max_{i\le n}|\xi_i|)^2 ( 2 M + 2 + max i ≤ n ∣ ξ i ∣ ) 2 . The fourth moment of this Gaussian maximum is O ( log 2 ( e n ) ) O(\log^2(en)) O ( log 2 ( e n )) . Cauchy–Schwarz and (34) bound the exceptional risk contribution by O M ( n − 15 / 8 log ( e n ) ) = o ( ε n 2 ) O_M(n^{-15/8}\log(en))=o(\varepsilon_n^2) O M ( n − 15/8 log ( e n )) = o ( ε n 2 ) . This proves the upper risk rate in (16) for f ~ n ∘ \widetilde{f}_n^\circ f n ∘ without any density assumption or moment assumption on the network parameters.
Minimax rates
Proof of Theorem 1, upper bound. Define the clipped-draw loss ℓ n ( f ) = ∥ C B ~ n ∘ ∘ f − f 0 ∥ L 2 ( P X ) 2 \ell_n(f)=\|\mathcal{C}\widetilde{B}_n^\circ\circ f-f_0\|_{L^2(P_X)}^2 ℓ n ( f ) = ∥ C B n ∘ ∘ f − f 0 ∥ L 2 ( P X ) 2 . The posterior mean is an average of clipped draws, so conditional Jensen’s inequality gives
∥ f ~ n − f 0 ∥ L 2 ( P X ) 2 ≤ ∫ C ( X ) ℓ n ( f ) Π n η ( d f ∣ D n ) . \|\widetilde{f}_n-f_0\|_{L^2(P_X)}^2\le\int_{\mathcal{C}(\mathcal{X})}\ell_n(f)\Pi_n^\eta(df\mid D_n). ∥ f n − f 0 ∥ L 2 ( P X ) 2 ≤ ∫ C ( X ) ℓ n ( f ) Π n η ( df ∣ D n ) .
We bound this integral separately on the clipping event G n G_n G n from the proof of Theorem 2 and its complement.
On G n G_n G n , B ~ n ≤ M + 3 \widetilde{B}_n\le M+3 B n ≤ M + 3 , so ℓ n ( f ) ≤ ( 2 M + 3 ) 2 \ell_n(f)\le(2M+3)^2 ℓ n ( f ) ≤ ( 2 M + 3 ) 2 for every f f f . For draws satisfying D η ( f , f 0 ) ≤ J 0 ε n 2 D_\eta(f,f_0)\le J_0\varepsilon_n^2 D η ( f , f 0 ) ≤ J 0 ε n 2 , (32) gives the sharper bound K η ( M + 3 ) J 0 ε n 2 K_\eta(M+3)J_0\varepsilon_n^2 K η ( M + 3 ) J 0 ε n 2 . Splitting the posterior integral into these two sets gives
1 G n ∫ C ( X ) ℓ n ( f ) Π n η ( d f ∣ D n ) ≤ K η ( M + 3 ) J 0 ε n 2 + ( 2 M + 3 ) 2 Π n η { D η ( f , f 0 ) > J 0 ε n 2 ∣ D n } . (38) \mathbf{1}_{G_n}\int_{\mathcal{C}(\mathcal{X})}\ell_n(f)\Pi_n^\eta(df\mid D_n)\le K_\eta(M+3)J_0\varepsilon_n^2
+(2M+3)^2\Pi_n^\eta\{D_\eta(f,f_0)>J_0\varepsilon_n^2\mid D_n\}. \tag*{(38)} 1 G n ∫ C ( X ) ℓ n ( f ) Π n η ( df ∣ D n ) ≤ K η ( M + 3 ) J 0 ε n 2 + ( 2 M + 3 ) 2 Π n η { D η ( f , f 0 ) > J 0 ε n 2 ∣ D n } . (38)
Lemma 1 bounds the expectation of the last posterior probability by an exponentially small quantity. Hence the risk contribution on G n G_n G n is O ( ε n 2 ) O(\varepsilon_n^2) O ( ε n 2 ) .
On G n c G_n^c G n c , every cell response mean has absolute value at most M + max i ≤ n ∣ ξ i ∣ M+\max_{i\le n}|\xi_i| M + max i ≤ n ∣ ξ i ∣ . Thus B ~ n ≤ 2 + M + max i ≤ n ∣ ξ i ∣ \widetilde{B}_n\le2+M+\max_{i\le n}|\xi_i| B n ≤ 2 + M + max i ≤ n ∣ ξ i ∣ and ℓ n ( f ) ≤ ( 2 M + 2 + max i ≤ n ∣ ξ i ∣ ) 2 \ell_n(f)\le(2M+2+\max_{i\le n}|\xi_i|)^2 ℓ n ( f ) ≤ ( 2 M + 2 + max i ≤ n ∣ ξ i ∣ ) 2 . The fourth moment of max i ≤ n ∣ ξ i ∣ \max_{i\le n}|\xi_i| max i ≤ n ∣ ξ i ∣ is O ( log 2 ( e n ) ) O(\log^2(en)) O ( log 2 ( e n )) . Cauchy–Schwarz and (30) therefore bound the exceptional risk contribution by
C M log ( e n ) J n exp { − c χ n / ( 16 J n ) } = o ( ε n 2 ) . C_M\log(en)\sqrt{J_n}\exp\{-c\chi n/(16J_n)\}=o(\varepsilon_n^2). C M log ( e n ) J n exp { − cχ n / ( 16 J n )} = o ( ε n 2 ) .
Combining the two contributions proves (16). Only bounded clipped loss is integrated under the posterior; no posterior moment of the Rényi divergence is needed.
Proof of Theorem 1, lower bound. Choose a nonzero Lipschitz function ψ : [ 0 , 1 ] d → [ 0 , 1 ] \psi: [0,1]^d \to[0,1] ψ : [ 0 , 1 ] d → [ 0 , 1 ] supported in [ 1 / 4 , 3 / 4 ] d [1/4,3/4]^d [ 1/4 , 3/4 ] d , and extend it by zero outside the unit cube. Let C ψ C_\psi C ψ be a Lipschitz constant of ψ \psi ψ in maximum norm, let m ≥ 1 m \ge1 m ≥ 1 be an integer, and let a > 0 a > 0 a > 0 be an amplitude to be chosen below. Partition [ 0 , 1 ] d [0,1]^d [ 0 , 1 ] d into m d m^d m d cubes Q j Q_j Q j of side h = 1 / m h = 1/m h = 1/ m , with lower corners x j x_j x j . Define
ψ j ( x ) = a h β ψ ( ( x − x j ) / h ) 1 { x ∈ Q j } , f θ = ∑ j = 1 m d θ j ψ j , θ ∈ { 0 , 1 } m d . \psi_j(x) = ah^\beta\psi((x-x_j)/h)1\{x \in Q_j\}, \qquad f_\theta= \sum_{j=1}^{m^d} \theta_j\psi_j, \qquad\theta\in\{0,1\}^{m^d}. ψ j ( x ) = a h β ψ (( x − x j ) / h ) 1 { x ∈ Q j } , f θ = j = 1 ∑ m d θ j ψ j , θ ∈ { 0 , 1 } m d .
Each bump has height at most a h β ah^\beta a h β and Lipschitz constant at most C ψ a h β − 1 C_\psi ah^{\beta-1} C ψ a h β − 1 . Its difference between two values at maximum-norm distance r r r is bounded by min { C ψ a h β − 1 r , 2 a h β } \min\{C_\psi ah^{\beta-1}r, 2ah^\beta\} min { C ψ a h β − 1 r , 2 a h β } , which is at most C a r β Car^\beta C a r β . Distinct bump supports are separated by at least h / 2 h/2 h /2 in max norm, and the bumps vanish near cell boundaries. For points at distance less than h / 2 h/2 h /2 , at most one bump contributes nonzero values, so its bound applies. For larger distances, the height bound gives the same estimate with a fixed larger constant. Thus the sums have Hölder seminorm at most C a Ca C a , and ∥ f θ ∥ ∞ ≤ a \|f_\theta\|_\infty\le a ∥ f θ ∥ ∞ ≤ a . Choose a fixed a > 0 a > 0 a > 0 small enough that every f θ ∈ H d β ( M ) f_\theta\in\mathcal{H}_d^\beta(M) f θ ∈ H d β ( M ) , uniformly in m m m .
Indexing vectors at Hamming distance one differ in a single coordinate j j j , so their targets differ by ψ j \psi_j ψ j up to sign. Put I ψ = ∫ [ 0 , 1 ] d ψ ( u ) 2 d u > 0 I_\psi= \int_{[0,1]^d}\psi(u)^2\,du > 0 I ψ = ∫ [ 0 , 1 ] d ψ ( u ) 2 d u > 0 . The squared L 2 ( P X ) L^2(P_X) L 2 ( P X ) distance between these targets, denoted by Δ j 2 \Delta_j^2 Δ j 2 , satisfies
c X a 2 h 2 β + d I ψ ≤ Δ j 2 ≤ C X a 2 h 2 β + d I ψ . c_Xa^2h^{2\beta+d}I_\psi\le\Delta_j^2 \le C_Xa^2h^{2\beta+d}I_\psi. c X a 2 h 2 β + d I ψ ≤ Δ j 2 ≤ C X a 2 h 2 β + d I ψ .
The n n n -sample Kullback–Leibler divergence between these two Gaussian-regression experiments is n Δ j 2 / 2 n\Delta_j^2/2 n Δ j 2 /2 . Take m = ⌈ n 1 / ( 2 β + d ) ⌉ m = \lceil n^{1/(2\beta+d)}\rceil m = ⌈ n 1/ ( 2 β + d ) ⌉ and reduce the fixed a a a , if necessary, so that C X a 2 I ψ / 2 ≤ 1 / 8 C_Xa^2I_\psi/2 \le1/8 C X a 2 I ψ /2 ≤ 1/8 . Since n h 2 β + d ≤ 1 nh^{2\beta+d} \le1 n h 2 β + d ≤ 1 , the divergence between any such neighbouring experiments is at most 1 / 8 1/8 1/8 . Pinsker’s inequality then bounds its total variation by 1 / 4 1/4 1/4 .
For any estimator f n f_n f n , choose θ ^ j ∈ { 0 , 1 } \widehat{\theta}_j \in\{0,1\} θ j ∈ { 0 , 1 } to minimize ∫ Q j ( f n ( x ) − b ψ j ( x ) ) 2 P X ( d x ) \int_{Q_j}(f_n(x)-b\psi_j(x))^2P_X(dx) ∫ Q j ( f n ( x ) − b ψ j ( x ) ) 2 P X ( d x ) over b ∈ { 0 , 1 } b \in\{0,1\} b ∈ { 0 , 1 } , taking zero in a tie. An estimator with infinite risk already satisfies the desired lower bound, so it suffices to consider finite-risk estimators. If the decision is wrong, its chosen candidate is no farther from f n f_n f n than the true candidate in the local norm ( ∫ Q j ∣ g ( x ) ∣ 2 P X ( d x ) ) 1 / 2 \left(\int_{Q_j}|g(x)|^2P_X(dx)\right)^{1/2} ( ∫ Q j ∣ g ( x ) ∣ 2 P X ( d x ) ) 1/2 . The triangle inequality then bounds Δ j \Delta_j Δ j by twice the local estimation error, so the local squared loss is at least Δ j 2 / 4 \Delta_j^2/4 Δ j 2 /4 .
Average over the uniform distribution on { 0 , 1 } m d \{0,1\}^{m^d} { 0 , 1 } m d . For coordinate j j j , mixing over the other coordinates preserves the 1 / 4 1/4 1/4 total-variation bound by convexity. The average error probability for each binary decision is therefore at least ( 1 − 1 / 4 ) / 2 = 3 / 8 (1-1/4)/2=3/8 ( 1 − 1/4 ) /2 = 3/8 . Summing the local losses gives
inf f n sup θ E f θ ∥ f n − f θ ∥ L 2 ( P X ) 2 ≥ c m d a 2 h 2 β + d I ψ ≍ h 2 β ≍ n − 2 β / ( 2 β + d ) . \inf_{f_n}\sup_\theta\mathbb{E}_{f_\theta}\|f_n-f_\theta\|_{L^2(P_X)}^2 \ge cm^da^2h^{2\beta+d}I_\psi\asymp h^{2\beta} \asymp n^{-2\beta/(2\beta+d)}. f n inf θ sup E f θ ∥ f n − f θ ∥ L 2 ( P X ) 2 ≥ c m d a 2 h 2 β + d I ψ ≍ h 2 β ≍ n − 2 β / ( 2 β + d ) .
The finite family lies in the stated Hölder ball, so the bound also holds for minimax risk over that ball.
Bounded covariate supports
The contraction and risk upper bounds do not require the support of P X P_X P X to be a cube. Let S ⊂ R d S \subset\mathbb{R}^d S ⊂ R d be a fixed bounded Borel set, P X ( S ) = 1 P_X(S)=1 P X ( S ) = 1 , and write ∥ f ∥ ∞ , S = sup x ∈ S ∣ f ( x ) ∣ \|f\|_{\infty,S}=\sup_{x\in S}|f(x)| ∥ f ∥ ∞ , S = sup x ∈ S ∣ f ( x ) ∣ . Define
H β ( S ; M ) = { f : S → R : ∥ f ∥ ∞ , S ≤ M , ∣ f ( x ) − f ( y ) ∣ ≤ M ∥ x − y ∥ ∞ β for all x , y ∈ S } . \mathcal{H}^\beta(S;M)=\{f:S\to\mathbb{R}:\|f\|_{\infty,S}\le M,\ |f(x)-f(y)|\le M\|x-y\|_\infty^\beta\text{ for all }x,y\in S\}. H β ( S ; M ) = { f : S → R : ∥ f ∥ ∞ , S ≤ M , ∣ f ( x ) − f ( y ) ∣ ≤ M ∥ x − y ∥ ∞ β for all x , y ∈ S } .
The condition uses distances in the surrounding space R d \mathbb{R}^d R d , including between different components of S S S . Choose a fixed closed cube Q ⊃ S Q \supset S Q ⊃ S of side length ℓ > 0 \ell>0 ℓ > 0 , and regard the same network and parameter priors as a law on the continuous functions C ( Q ) C(Q) C ( Q ) . The eligible-cell clipping rule uses this enclosing cube, not the precise shape of S S S . Partition Q Q Q into s n d s_n^d s n d cells and use (33) with their counts and means.
Proposition 4. For every fixed 0 < β ≤ 1 0 < \beta\le1 0 < β ≤ 1 and M > 0 M > 0 M > 0 , the Rényi bound (19) and the clipped-draw and risk assertions of Remark 3 hold with H β ( S ; M ) \mathcal{H}^{\beta}(S;M) H β ( S ; M ) in place of H d β ( M ) \mathcal{H}^{\beta}_{d}(M) H d β ( M ) and L 2 ( P X ) \mathcal{L}^{2}(P_X) L 2 ( P X ) evaluated on S S S . Their constants and the sample size beyond which they hold may depend on Q Q Q . Neither the pure exponential adaptive bound (20) nor the regular-design lower bound (17) is asserted for arbitrary S S S .
Proof. Lemma 3, followed by clipping to [ − M , M ] [-M,M] [ − M , M ] , extends each target from S S S to R d \mathbb{R}^{d} R d without increasing its supremum bound or Hölder constant. Choose a fixed R ≥ max { 1 , sup x ∈ Q ∥ x ∥ 2 } R \ge\max\{1,\sup_{x\in Q}\lVert x\rVert_{2}\} R ≥ max { 1 , sup x ∈ Q ∥ x ∥ 2 } . The rescaled extension z ↦ f 0 ( R z ) z \mapsto f_{0}(Rz) z ↦ f 0 ( R z ) has a uniformly bounded Hölder norm on the unit ball. Apply the argument of Proposition 1 to this function and rescale the network back to Q Q Q . The bounds on width and on the sum of absolute output coefficients retain their orders in ε \varepsilon ε , and the approximating hidden-layer coefficients and biases remain bounded by one. In Lemma 4, the bound for perturbing a hidden-layer coefficient or bias changes only by the fixed factor 1 + sup x ∈ Q ∥ x ∥ 1 1+\sup_{x\in Q}\lVert x\rVert_{1} 1 + sup x ∈ Q ∥ x ∥ 1 . The approximating centres are eventually admissible in (11) . The parameter-box condition (12) and the width condition (10) therefore give the same uniform prior-probability bound on Q Q Q . Lemma 1 then proves (19) , with constants allowed to depend on Q Q Q .
Intersect the equal cells of the enclosing cube with S S S . These intersections partition S S S into at most s n d s_{n}^{d} s n d sets, each of max-norm diameter at most ℓ / s n \ell/s_{n} ℓ / s n . For sufficiently large n n n , M ( ℓ / s n ) β ≤ 1 M(\ell/s_{n})^{\beta} \le1 M ( ℓ / s n ) β ≤ 1 . The proofs of (34) –(37) use only the number of cells, their probabilities, and their binomial counts. They therefore apply to these intersections, including any having probability zero. The clipped-loss and risk bounds follow as in Remark 3. The separate pure exponential bound for coarse-partition clipping would require an additional lower bound on the probabilities of its relevant cells; no such bound is assumed here.
Simulation design and computation
All simulation code and reproducibility materials are available at https://github.com/hiendn/bayesian-relu-minimax. The study uses likelihood power η = 0.9 \eta= 0.9 η = 0.9 and sample sizes from 64 to 4096. For each target, sample size and parameter family, we fit 50 replications. The Gaussian width-prior comparisons use the same fits within each target and sample size. Supplementary Tables 6 and 7 give all mean errors, Monte Carlo standard errors and pointwise 95% Monte Carlo confidence intervals.
Targets, prior, and loss
The numerical study uses d = 1 d = 1 d = 1 , P X = Unif [ 0 , 1 ] P_X = \operatorname{Unif}[0,1] P X = Unif [ 0 , 1 ] , and independent N ( 0 , 1 ) N(0,1) N ( 0 , 1 ) errors. The two fixed targets are
\begin{aligned}
f_{1/2}(x) &= \sqrt{\lvert\sin(2\pi x)\rvert},\\
f_{1}(x) &= \lvert\sin(2\pi x)\rvert,
\end{aligned}
\qquad x \in[0,1]. \tag*{(39)}\end{aligned}
Both have supremum norm one. For every x , y ∈ [ 0 , 1 ] x,y \in[0,1] x , y ∈ [ 0 , 1 ] , ∣ ∣ sin ( 2 π x ) ∣ − ∣ sin ( 2 π y ) ∣ ∣ ≤ 2 π ∣ x − y ∣ \lvert\lvert\sin(2\pi x)\rvert-\lvert\sin(2\pi y)\rvert\rvert\le2\pi\lvert x-y\rvert ∣∣ sin ( 2 π x )∣ − ∣ sin ( 2 π y )∣∣ ≤ 2 π ∣ x − y ∣ and ∣ u − v ∣ ≤ ∣ u − v ∣ \lvert\sqrt{u}-\sqrt{v}\rvert\le\sqrt{\lvert u-v\rvert} ∣ u − v ∣ ≤ ∣ u − v ∣ for u , v ≥ 0 u,v \ge0 u , v ≥ 0 give
∣ f 1 / 2 ( x ) − f 1 / 2 ( y ) ∣ ≤ 2 π ∣ x − y ∣ 1 / 2 , ∣ f 1 ( x ) − f 1 ( y ) ∣ ≤ 2 π ∣ x − y ∣ . \begin{aligned}
\lvert f_{1/2}(x)-f_{1/2}(y)\rvert&\le\sqrt{2\pi}\lvert x-y\rvert^{1/2},\\
\lvert f_{1}(x)-f_{1}(y)\rvert&\le2\pi\lvert x-y\rvert.
\end{aligned} ∣ f 1/2 ( x ) − f 1/2 ( y )∣ ∣ f 1 ( x ) − f 1 ( y )∣ ≤ 2 π ∣ x − y ∣ 1/2 , ≤ 2 π ∣ x − y ∣ .
Thus f 1 / 2 ∈ H 1 1 / 2 ( 2 π ) f_{1/2} \in\mathcal{H}_{1}^{1/2}(\sqrt{2\pi}) f 1/2 ∈ H 1 1/2 ( 2 π ) and f 1 ∈ H 1 ( 2 π ) f_{1} \in\mathcal{H}_{1}(2\pi) f 1 ∈ H 1 ( 2 π ) ; in particular, both belong to their respective classes with the common radius 2 π 2\pi 2 π . Both functions have two peaks, at 1 / 4 1/4 1/4 and 3 / 4 3/4 3/4 . At the interior zero x 0 = 1 / 2 x_{0}=1/2 x 0 = 1/2 ,
f 1 / 2 ( x 0 + h ) − f 1 / 2 ( x 0 ) = 2 π ∣ h ∣ 1 / 2 + O ( ∣ h ∣ 5 / 2 ) , f_{1/2}(x_{0}+h)-f_{1/2}(x_{0})=\sqrt{2\pi}\lvert h\rvert^{1/2}+O(\lvert h\rvert^{5/2}), f 1/2 ( x 0 + h ) − f 1/2 ( x 0 ) = 2 π ∣ h ∣ 1/2 + O (∣ h ∣ 5/2 ) ,
so f 1 / 2 f_{1/2} f 1/2 has no larger Hölder exponent at this point and is not Lipschitz. The one-sided derivatives of f 1 f_{1} f 1 differ by 4 π 4\pi 4 π at x 0 x_{0} x 0 . Within each lobe, f 1 ′ ′ = − ( 2 π ) 2 f 1 < 0 f_{1}''=-(2\pi)^{2}f_{1}<0 f 1 ′′ = − ( 2 π ) 2 f 1 < 0 , so f 1 f_{1} f 1 is not piecewise linear.
Every finite shallow ReLU network is Lipschitz and piecewise linear. Consequently, neither target is represented exactly by such a network. Both functions are evaluated directly using elementary operations. The clipping procedure uses neither smoothness parameter nor radius.
The Gaussian baseline has π K ( k ) = 2 − k \pi_K(k) = 2^{-k} π K ( k ) = 2 − k , U m , B m ∼ N ( 0 , 1 ) U_m, B_m \sim\mathrm{N}(0,1) U m , B m ∼ N ( 0 , 1 ) , and A m ∣ K = k ∼ N ( 0 , k 3 ) A_m \mid K = k \sim\mathrm{N}(0,k^3) A m ∣ K = k ∼ N ( 0 , k 3 ) , independently across all scalar network parameters. The fractional power is η = 0.9 \eta= 0.9 η = 0.9 . The same fixed prior is used for both targets and every sample size. Numerical computation conditions on K ≤ 64 K \le64 K ≤ 64 ; the theorem uses unbounded width.
The additional uniform and Student t 10 t_{10} t 10 comparisons use geometric width. In the uniform case, U m , B m ∼ Unif [ − 2 , 2 ] U_m, B_m \sim\operatorname{Unif}[-2,2] U m , B m ∼ Unif [ − 2 , 2 ] and A m ∣ K = k ∼ Unif [ − k 2 , k 2 ] A_m \mid K = k \sim\operatorname{Unif}[-k^2,k^2] A m ∣ K = k ∼ Unif [ − k 2 , k 2 ] . In the Student case, all network parameters independently follow a unit-scale t 10 t_{10} t 10 law, with no width-dependent scaling. Their Figure 2 labels are U/G and t 10 t_{10} t 10 /G.
Figure 2. Mean squared errors with 95% Monte Carlo confidence intervals from 50 replications per case. N/U/t 10 t_{10} t 10 : Gaussian/uniform/Student t 10 t_{10} t 10 parameters; G/P/H: geometric/shifted Poisson/power-law widths. Grey illustrative guides: lower (dashed), upper (dotted).
Original replicate indices 1 , … , 50 1,\ldots,50 1 , … , 50 identify the data streams. For a fixed target, the same indexed stream is used across parameter families and sample sizes, with nested data as n n n increases. Every original replication is included, and neither risk nor credible coverage is used to select a computation stage. The released seeds and settings reproduce these fits.
Adaptive clipping and prediction risk
For each data set we divide [ 0 , 1 ] [0,1] [ 0 , 1 ] into m n = ⌈ log n ⌉ m_n = \lceil\log n \rceil m n = ⌈ log n ⌉ equal cells and calculate the cutoff in (21) , setting an empty cell mean to zero. The same responses determine the clipping and the fractional posterior. We average draws clipped at B ~ n \widetilde{B}_n B n as in (15) . Since B ~ n ≥ 2 \widetilde{B}_n \ge2 B n ≥ 2 and both targets are bounded by one, the clipping interval always covers each target in this study.
Across the reported fits, the smallest coarse-cell counts were 4 for β = 1 / 2 \beta= 1/2 β = 1/2 and 6 for β = 1 \beta= 1 β = 1 .
Write Π n , 64 0.9 \Pi_{n,64}^{0.9} Π n , 64 0.9 for the numerical fractional posterior conditional on K ≤ 64 K \le64 K ≤ 64 and q n , β = ( log n / n ) 2 β / ( 2 β + 1 ) q_{n,\beta} = (\log n/n)^{2\beta/(2\beta+1)} q n , β = ( log n / n ) 2 β / ( 2 β + 1 ) .
We approximate integrals under the uniform design by equally weighted averages over a fixed grid, with one point in each of 1009 equal subintervals:
x g = g − 1 / 2 + 2 − 1 1009 , g = 1 , … , 1009. (40) x_g = \frac{g - 1/2 + \sqrt{2} - 1}{1009}, \qquad g = 1,\ldots,1009. \tag*{(40)} x g = 1009 g − 1/2 + 2 − 1 , g = 1 , … , 1009. (40)
This shifted rectangle rule uses a fixed offset of 2 − 1 \sqrt{2} - 1 2 − 1 grid spacings so that its points do not coincide with the targets’ central zero at 1 / 2 1/2 1/2 or peaks at 1 / 4 1/4 1/4 and $3/4. Related uses of irrational evaluation points to avoid exact alignment are discussed by Bartuska et al. [6 ] and Shampine [25 ] . These precedents motivate avoiding exact alignment; they do not establish improved accuracy or a convergence rate for our chosen offset. The same grid is used for every target, prior, sample size and replication; it is fixed independently of the observations and fitted functions. For the Gaussian fits in Figure 3 , we independently recomputed the clipped posterior means from the sampled widths and hidden-layer parameters, integrating over the output coefficients, and then recomputed their squared errors on this grid. These were compared with the estimates produced during fitting. The largest absolute discrepancies were below 10 − 10 10^{-10} 1 0 − 10 . This agreement checks the calculation of the estimates; it does not bound numerical integration error or residual Markov-chain error.
Figure 3. True functions, clipped posterior means, and pointwise 95% credible intervals for both targets at n = 1024 , 4096 n = 1024, 4096 n = 1024 , 4096 . Original replicate one; Gaussian parameters, geometric width, η = 0.9 \eta= 0.9 η = 0.9 , and K ≤ 64 K \le64 K ≤ 64 .
If f ~ n , r \widetilde{f}_{n,r} f n , r is the numerical adaptive posterior mean for replicate r r r , its approximate integrated squared error is
R ~ n , r , β = 1 1009 ∑ g = 1 1009 { f ~ n , r ( x g ) − f β ( x g ) } 2 . (41) \widetilde{R}_{n,r,\beta} = \frac{1}{1009} \sum_{g=1}^{1009} \left\{\widetilde{f}_{n,r}(x_g) - f_\beta(x_g)\right\}^{2}. \tag*{(41)} R n , r , β = 1009 1 g = 1 ∑ 1009 { f n , r ( x g ) − f β ( x g ) } 2 . (41)
For Gaussian network parameters, we integrate the clipped prediction conditional on the width, hidden-layer coefficients and biases, then average over the posterior samples as described in Appendix § 14 .§ 3 . This integration reduces Monte Carlo noise. Clipping the posterior mean after averaging would define a different estimator.
Each reported risk is the mean of 50 original diagnostic-passing replicate errors. Let R ‾ n , β \overline{R}_{n,\beta} R n , β be their mean and s n , β s_{n,\beta} s n , β their sample standard deviation. The Monte Carlo standard error is s n , β / 50 s_{n,\beta}/\sqrt{50} s n , β / 50 . We report pointwise 95% asymptotic Monte Carlo confidence intervals using the usual normal approximation. Let Φ \Phi Φ be the standard normal distribution function and z 0.975 = Φ − 1 ( 0.975 ) ≈ 1.95996 z_{0.975}=\Phi^{-1}(0.975)\approx1.95996 z 0.975 = Φ − 1 ( 0.975 ) ≈ 1.95996 . The interval is
R ‾ n , β ± z 0.975 s n , β 50 . \overline{R}_{n,\beta}\ \pm\ z_{0.975}\frac{s_{n,\beta}}{\sqrt{50}}. R n , β ± z 0.975 50 s n , β .
These intervals quantify simulation uncertainty in the mean squared error across all 50 original data replications within each case. They are not posterior credible intervals and do not bound residual Markov-chain bias. Since each target is bounded by one, the clipping bound satisfies B ‾ n ≤ 3 + max i ≤ n ∣ ξ i ∣ \overline{B}_n \le3+\max_{i\le n}|\xi_i| B n ≤ 3 + max i ≤ n ∣ ξ i ∣ , where ξ i \xi_i ξ i are the regression errors in (1), and each replicate error is at most ( 4 + max i ≤ n ∣ ξ i ∣ ) 2 (4+\max_{i\le n}|\xi_i|)^2 ( 4 + max i ≤ n ∣ ξ i ∣ ) 2 . It therefore has finite variance, supporting the normal approximation as the number of reported completed replications increases. In each risk-figure panel, the upper-rate scale q n , β q_{n,\beta} q n , β and lower-rate scale n − 2 β / ( 2 β + 1 ) n^{-2\beta/(2\beta+1)} n − 2 β / ( 2 β + 1 ) are both normalized to the Gaussian/geometric mean error at n = 64 n=64 n = 64 . Their vertical constants are illustrative and do not estimate the unknown theoretical constants. The minimax lower rate is a class benchmark rather than a lower bound on the error at either fixed target.
For adjacent sample sizes and for the two endpoints, we also report the decay exponent
p a , b = log ( R ‾ a , β / R ‾ b , β ) log ( b / a ) , a < b . p_{a,b}=\frac{\log(\overline{R}_{a,\beta}/\overline{R}_{b,\beta})}{\log(b/a)},\qquad a<b. p a , b = log ( b / a ) log ( R a , β / R b , β ) , a < b .
Its 95% percentile bootstrap interval uses 2000 resamples of the 50 original replicate indices, paired across sample sizes for each target. This is a nonparametric percentile bootstrap applied to paired replication errors [10 Sections 2, 7–8] : the two errors from a replication form one resampling unit, and the interval endpoints are the empirical 2.5th and 97.5th percentiles of the recalculated exponents. This preserves dependence from the reused data streams. These fixed-target intervals do not bound residual Markov-chain error.
These exponents describe the displayed targets over a finite range of sample sizes and do not estimate worst-case Hölder-ball rates.
The function estimates in Figure 1 use original replicate one for both targets at n = 1024 n=1024 n = 1024 and 4096, under uniform parameters and geometric width. We evaluate and clip the regression function for each posterior draw of the output coefficients and hidden parameters, combine the draws from the independent ladders, and calculate the posterior mean and empirical type-1 equal-tailed 95% intervals. Repeating these simulations with the same data, seeds and settings reproduces the predictions and diagnostic summaries exactly. The intervals concern the latent regression function, with no observation noise added. The true central-notch value zero lies in none of the intervals. At n = 4096 n=4096 n = 4096 , the Hölder 1/2 mean is 0.400525 with interval [0.194452, 0.593273]; the Lipschitz mean is 0.168787 with interval [0.025630, 0.298567].
Figure 3 shows original replicate one for both targets at n = 1024 n=1024 n = 1024 and 4096, under Gaussian parameters and geometric width. Conditional on each posterior draw of width and hidden parameters, the regression function is normal with the mean and variance below. We invert the equal-weight mixture at probabilities 0.025 and 0.975, then clip the quantiles at ± B ‾ n \pm\overline{B}_n ± B n . The intervals include output-coefficient, width and hidden-parameter uncertainty. At the central zero, none of the intervals contain zero. At n = 4096 n=4096 n = 4096 , the Hölder 1/2 mean is 0.373204 with interval [0.186481, 0.570900]; the Lipschitz mean is 0.155984 with interval [0.003281, 0.280728].
Posterior computation
Numerical target. The accompanying R code implements the sampling transitions directly. For this one-dimensional study, a full network state is θ = ( k , a , u , b ) \theta=(k,a,u,b) θ = ( k , a , u , b ) , with a , u , b ∈ R k a,u,b\in\mathbb{R}^k a , u , b ∈ R k , on
T 64 = ⨆ k = 1 64 ( { k } × R 3 k ) \mathcal{T}_{64} = \bigsqcup_{k=1}^{64}(\{k\} \times\mathbb{R}^{3k}) T 64 = ⨆ k = 1 64 ({ k } × R 3 k ) . Write p k ( a , u , b ) p_k(a,u,b) p k ( a , u , b ) for the conditional parameter density of the chosen row of Table 1 , and π 64 ( k ) = 2 − k / ( 1 − 2 − 64 ) \pi_{64}(k) = 2^{-k}/(1 - 2^{-64}) π 64 ( k ) = 2 − k / ( 1 − 2 − 64 ) , 1 ≤ k ≤ 64 1 \le k \le64 1 ≤ k ≤ 64 , for the conditioned geometric width law. At likelihood power 0 ≤ λ ≤ η = 0.9 0 \le\lambda\le\eta= 0.9 0 ≤ λ ≤ η = 0.9 , the target density is
p λ , 64 ( θ ∣ D n ) ∝ π 64 ( k ) p k ( a , u , b ) exp { − λ R ( θ ) / 2 } , R ( θ ) = ∑ i = 1 n { Y i − f θ ( X i ) } 2 . (42) p_{\lambda,64}(\theta\mid\mathcal{D}_n) \propto\pi_{64}(k)p_k(a,u,b)\exp\{-\lambda R(\theta)/2\}, \qquad R(\theta) = \sum_{i=1}^{n}\{Y_i-f_\theta(X_i)\}^2. \tag*{(42)} p λ , 64 ( θ ∣ D n ) ∝ π 64 ( k ) p k ( a , u , b ) exp { − λ R ( θ ) /2 } , R ( θ ) = i = 1 ∑ n { Y i − f θ ( X i ) } 2 . (42)
The reference measure counts widths and uses Lebesgue measure on the scalar network parameters. The likelihood uses the unclipped network; clipping enters the prediction average in (15). The chain at power 0.9 targets the numerical fractional posterior. At power zero the target is the conditioned prior.
Gaussian output coefficients. For Gaussian network parameters, the output coefficients can be integrated analytically. Conditional on width k k k and the hidden-layer coefficients and biases, write Φ \Phi Φ for the n × k n \times k n × k matrix with entries ρ ( U m X i + B m ) \rho(U_mX_i+B_m) ρ ( U m X i + B m ) and Y = ( Y 1 , … , Y n ) T Y=(Y_1,\ldots,Y_n)^\mathsf{T} Y = ( Y 1 , … , Y n ) T . Let I j I_j I j denote the j × j j \times j j × j identity matrix. For vector arguments, N ( μ , Σ ) \mathcal{N}(\mu,\Sigma) N ( μ , Σ ) denotes the normal distribution with mean vector μ \mu μ and covariance matrix Σ \Sigma Σ . In this one-dimensional experiment, the output vector has prior N ( 0 , k 3 I k ) \mathcal{N}(0,k^3I_k) N ( 0 , k 3 I k ) . At likelihood power 0 ≤ λ ≤ η 0 \le\lambda\le\eta 0 ≤ λ ≤ η , define
Q λ = k − 3 I k + λ Φ T Φ , m λ = λ Q λ − 1 Φ T Y . (43) Q_\lambda= k^{-3}I_k+\lambda\Phi^\mathsf{T}\Phi,\qquad m_\lambda=\lambda Q_\lambda^{-1}\Phi^\mathsf{T}Y. \tag*{(43)} Q λ = k − 3 I k + λ Φ T Φ , m λ = λ Q λ − 1 Φ T Y . (43)
For an output coefficient vector a ∈ R k a \in\mathbb{R}^k a ∈ R k , multiplying the prior density by the powered Gaussian likelihood and completing the square gives
k − 3 a T a + λ ∥ Y − Φ a ∥ 2 2 = ( a − m λ ) T Q λ ( a − m λ ) + λ Y T Y − m λ T Q λ m λ . k^{-3}a^\mathsf{T}a+\lambda\lVert Y-\Phi a\rVert_2^2=(a-m_\lambda)^\mathsf{T}Q_\lambda(a-m_\lambda)+\lambda Y^\mathsf{T}Y-m_\lambda^\mathsf{T}Q_\lambda m_\lambda. k − 3 a T a + λ ∥ Y − Φ a ∥ 2 2 = ( a − m λ ) T Q λ ( a − m λ ) + λ Y T Y − m λ T Q λ m λ .
Hence the conditional output posterior is N ( m λ , Q λ − 1 ) \mathcal{N}(m_\lambda,Q_\lambda^{-1}) N ( m λ , Q λ − 1 ) . If φ k ( a ; 0 , k 3 I k ) \varphi_k(a;0,k^3I_k) φ k ( a ; 0 , k 3 I k ) is the Gaussian prior density, integration of this completed square yields
∫ R k ( 2 π ) − λ n / 2 e − λ ∥ Y − Φ a ∥ 2 2 / 2 φ k ( a ; 0 , k 3 I k ) d a = ( 2 π ) − λ n / 2 k − 3 k / 2 det ( Q λ ) − 1 / 2 exp { − λ 2 Y T Y + 1 2 m λ T Q λ m λ } . \int_{\mathbb{R}^k}(2\pi)^{-\lambda n/2}e^{-\lambda\lVert Y-\Phi a\rVert_2^2/2}\varphi_k(a;0,k^3I_k)\,da
= (2\pi)^{-\lambda n/2}k^{-3k/2}\det(Q_\lambda)^{-1/2}\exp\left\{-\frac{\lambda}{2}Y^\mathsf{T}Y+\frac{1}{2}m_\lambda^\mathsf{T}Q_\lambda m_\lambda\right\}. ∫ R k ( 2 π ) − λn /2 e − λ ∥ Y − Φ a ∥ 2 2 /2 φ k ( a ; 0 , k 3 I k ) d a = ( 2 π ) − λn /2 k − 3 k /2 det ( Q λ ) − 1/2 exp { − 2 λ Y T Y + 2 1 m λ T Q λ m λ } .
Using k 3 k det ( Q λ ) = det ( I k + λ k 3 Φ T Φ ) = det ( I n + λ k 3 Φ Φ T ) k^{3k}\det(Q_\lambda)=\det(I_k+\lambda k^3\Phi^\mathsf{T}\Phi)=\det(I_n+\lambda k^3\Phi\Phi^\mathsf{T}) k 3 k det ( Q λ ) = det ( I k + λ k 3 Φ T Φ ) = det ( I n + λ k 3 Φ Φ T ) and the Woodbury identity yields
det ( I n + λ k 3 Φ Φ T ) − 1 / 2 exp { − λ 2 Y T ( I n + λ k 3 Φ Φ T ) − 1 Y } , (44) \det(I_n+\lambda k^3\Phi\Phi^\mathsf{T})^{-1/2}\exp\left\{-\frac{\lambda}{2}Y^\mathsf{T}(I_n+\lambda k^3\Phi\Phi^\mathsf{T})^{-1}Y\right\}, \tag*{(44)} det ( I n + λ k 3 Φ Φ T ) − 1/2 exp { − 2 λ Y T ( I n + λ k 3 Φ Φ T ) − 1 Y } , (44)
after omitting ( 2 π ) − λ n / 2 (2\pi)^{-\lambda n/2} ( 2 π ) − λn /2 , which is independent of the network state. We evaluate the factor using a k × k k\times k k × k Cholesky factorization of Q λ Q_\lambda Q λ . The implementation also omits − λ Y T Y / 2 -\lambda Y^\mathsf{T}Y/2 − λ Y T Y /2 from its log value. These terms depend only on the data and the assigned power, so they cancel in both within-level moves and exchanges between levels; all terms depending on width or hidden-layer coefficients and biases are retained. The calculation takes O ( n k 2 + k 3 ) O(nk^2+k^3) O ( n k 2 + k 3 ) time and O ( n k + k 2 ) O(nk+k^2) O ( nk + k 2 ) working memory. Grid prediction adds O ( 1009 k 2 ) O(1009k^2) O ( 1009 k 2 ) time.
For a Gaussian state s = ( k , u , b ) s=(k,u,b) s = ( k , u , b ) , containing the width, hidden-layer coefficients and biases, let ℓ λ ( s ) \ell_\lambda(s) ℓ λ ( s ) be the log of the factor in (44), and let φ \varphi φ be the standard normal density. The sampled marginal density is
p λ , 64 G ( s ∣ D n ) ∝ π 64 ( k ) ∏ m = 1 k φ ( u m ) φ ( b m ) e ℓ λ ( s ) . p_{\lambda,64}^{\mathrm{G}}(s\mid D_n)\propto\pi_{64}(k)\prod_{m=1}^{k}\varphi(u_m)\varphi(b_m)e^{\ell_\lambda(s)}. p λ , 64 G ( s ∣ D n ) ∝ π 64 ( k ) m = 1 ∏ k φ ( u m ) φ ( b m ) e ℓ λ ( s ) .
The prior densities of the hidden-layer coefficients and biases remain in this density. Gaussian integration removes the k k k output coefficients from the sampled state and retains their exact conditional distribution. The factor is evaluated separately at each power: integrating a powered likelihood generally differs from powering its integral.
Gaussian Metropolis–Hastings updates. At a fixed width, for a proposal density q λ ( s ′ ∣ s ) q_\lambda(s'\mid s) q λ ( s ′ ∣ s ) , the acceptance rule is
1 ∧ p λ , 64 G ( s ′ ∣ D n ) q λ ( s ∣ s ′ ) p λ , 64 G ( s ∣ D n ) q λ ( s ′ ∣ s ) . 1\wedge\frac{p_{\lambda,64}^{\mathrm{G}}(s'\mid D_n)q_\lambda(s\mid s')}{p_{\lambda,64}^{\mathrm{G}}(s\mid D_n)q_\lambda(s'\mid s)}. 1 ∧ p λ , 64 G ( s ∣ D n ) q λ ( s ′ ∣ s ) p λ , 64 G ( s ′ ∣ D n ) q λ ( s ∣ s ′ ) .
Each iteration selects a width change with probability 0.25. Otherwise it chooses one unit uniformly. With probability 0.2, the hidden-layer coefficient and bias of that unit are independently redrawn from N ( 0 , 1 ) \mathcal{N}(0,1) N ( 0 , 1 ) . The prior and proposal densities cancel, giving acceptance probability 1 ∧ e Δ ℓ λ 1\wedge e^{\Delta\ell_\lambda} 1 ∧ e Δ ℓ λ , where Δ ℓ λ = ℓ λ ( s ′ ) − ℓ λ ( s ) \Delta\ell_\lambda=\ell_\lambda(s')-\ell_\lambda(s) Δ ℓ λ = ℓ λ ( s ′ ) − ℓ λ ( s ) . Otherwise either its hidden-layer coefficient or its bias is chosen with equal probability. Given its current value z z z , the proposed value is the random variable Z ′ = z + ζ Z'=z+\zeta Z ′ = z + ζ , where ζ ∼ N ( 0 , 0.65 2 ) \zeta\sim\mathcal{N}(0,0.65^2) ζ ∼ N ( 0 , 0.6 5 2 ) . For a realized proposal z ′ z' z ′ , the acceptance probability is
1 ∧ exp { Δ ℓ λ − ( z ′ 2 − z 2 ) / 2 } . 1\wedge\exp\{\Delta\ell_\lambda-(z'^2-z^2)/2\}. 1 ∧ exp { Δ ℓ λ − ( z ′2 − z 2 ) /2 } .
The proposal is symmetric, and the final term is the Gaussian prior-density ratio for the selected parameter.
For width changes, define the birth probability
b k = { 1 , k = 1 , 1 / 2 , 1 < k < 64 , 0 , k = 64 , d k = 1 − b k . b_k=
\begin{cases}
1, & k=1,\\
1/2, & 1<k<64,\\
0, & k=64,
\end{cases}
\qquad d_k=1-b_k. b k = ⎩ ⎨ ⎧ 1 , 1/2 , 0 , k = 1 , 1 < k < 64 , k = 64 , d k = 1 − b k .
A birth chooses one of the k + 1 k+1 k + 1 insertion positions uniformly and draws the new unit’s hidden-layer coefficient and bias from their Gaussian priors. A death chooses one of the k k k active positions uniformly and deletes that unit. Their acceptance probabilities are
α b = 1 ∧ ( e Δ ℓ λ d k + 1 2 b k ) , α d = 1 ∧ ( e Δ ℓ λ 2 b k − 1 d k ) , (45) \alpha_{\mathrm{b}} = 1 \wedge\left(e^{\Delta\ell_\lambda}\frac{d_{k+1}}{2b_k}\right), \qquad\alpha_{\mathrm{d}} = 1 \wedge\left(e^{\Delta\ell_\lambda}\frac{2b_{k-1}}{d_k}\right), \tag*{(45)} α b = 1 ∧ ( e Δ ℓ λ 2 b k d k + 1 ) , α d = 1 ∧ ( e Δ ℓ λ d k 2 b k − 1 ) , (45)
for permitted births and deaths, respectively. Forward insertion and reverse deletion choose the same position with equal probability, and the prior density of the new hidden-layer coefficient and bias cancels with its proposal density. The factors 1 / 2 1/2 1/2 and 2 2 2 are width-prior ratios. The insertion map has Jacobian magnitude one; units are represented in an ordered list, so no factorial correction is needed ([12 ] ). The marginal factor is recomputed at the proposed width, including the changed k 3 k^3 k 3 output-prior variance and all width-dependent normalizers.
Uniform and Student t 10 t_{10} t 10 updates. The uniform and Student t 10 t_{10} t 10 samplers retain output coefficients, hidden-layer coefficients and biases, and use the full density in (42). Let z m = ( a m , u m , b m ) z_m = (a_m,u_m,b_m) z m = ( a m , u m , b m ) and g k ( z m ) g_k(z_m) g k ( z m ) be the product of the prior densities of these three parameters, so that p k = ∏ m = 1 k g k ( z m ) p_k = \prod_{m=1}^{k} g_k(z_m) p k = ∏ m = 1 k g k ( z m ) . A birth draws the new triple from g k + 1 g_{k+1} g k + 1 and inserts it at a uniformly chosen position; a death deletes a uniformly chosen triple. For a birth from θ \theta θ to θ ′ \theta' θ ′ , the acceptance rule is
1 ∧ [ e − λ { R ( θ ′ ) − R ( θ ) } / 2 d k + 1 2 b k ∏ m = 1 k g k + 1 ( z m ) g k ( z m ) ] . (46) 1 \wedge\left[e^{-\lambda\{R(\theta')-R(\theta)\}/2}\frac{d_{k+1}}{2b_k}\prod_{m=1}^{k}\frac{g_{k+1}(z_m)}{g_k(z_m)}\right]. \tag*{(46)} 1 ∧ [ e − λ { R ( θ ′ ) − R ( θ )} /2 2 b k d k + 1 m = 1 ∏ k g k ( z m ) g k + 1 ( z m ) ] . (46)
For a death from width k k k to k − 1 k-1 k − 1 , write z 1 ′ , … , z k − 1 ′ z'_1,\ldots,z'_{k-1} z 1 ′ , … , z k − 1 ′ for the remaining triples. Its rule is
1 ∧ [ e − λ { R ( θ ′ ) − R ( θ ) } / 2 2 b k − 1 d k ∏ m = 1 k − 1 g k − 1 ( z m ′ ) g k ( z m ′ ) ] . 1 \wedge\left[e^{-\lambda\{R(\theta')-R(\theta)\}/2}\frac{2b_{k-1}}{d_k}\prod_{m=1}^{k-1}\frac{g_{k-1}(z'_m)}{g_k(z'_m)}\right]. 1 ∧ [ e − λ { R ( θ ′ ) − R ( θ )} /2 d k 2 b k − 1 m = 1 ∏ k − 1 g k ( z m ′ ) g k − 1 ( z m ′ ) ] .
The inserted or deleted triple’s density cancels with its proposal factor. Prior densities for the retained units must still be evaluated at the destination width. In the uniform case the birth product is ( k / ( k + 1 ) ) 2 k (k/(k+1))^{2k} ( k / ( k + 1 ) ) 2 k ; the death product is ( k / ( k − 1 ) ) 2 ( k − 1 ) (k/(k-1))^{2(k-1)} ( k / ( k − 1 ) ) 2 ( k − 1 ) when every remaining ∣ a m ′ ∣ ≤ ( k − 1 ) 2 |a'_m| \le(k-1)^2 ∣ a m ′ ∣ ≤ ( k − 1 ) 2 , and is zero otherwise. For Student network parameters, the products equal one because the parameter priors do not depend on width.
Each direct-sampler iteration makes one width move with probability 0.25, otherwise one update of a hidden-layer coefficient or bias, followed by updates of all output coefficients in a random order. For the update of a hidden-layer coefficient or bias, the sampler chooses one unit uniformly and then chooses its coefficient or bias with equal probability. With probability 0.25 it redraws the selected parameter from its prior, giving acceptance probability 1 ∧ e − λ { R ( θ ′ ) − R ( θ ) } / 2 1 \wedge e^{-\lambda\{R(\theta')-R(\theta)\}/2} 1 ∧ e − λ { R ( θ ′ ) − R ( θ )} /2 . Otherwise it proposes a Gaussian random walk with standard deviation 0.4, retaining the selected parameter’s prior-density ratio in the Metropolis–Hastings rule. When the parameter prior is uniform, proposals outside its support are rejected.
For the updates of the output coefficients, put ϕ m = ( ρ ( u m X i + b m ) ) i = 1 n \phi_m = (\rho(u_m X_i+b_m))_{i=1}^{n} ϕ m = ( ρ ( u m X i + b m ) ) i = 1 n , r − m = Y − ∑ j ≠ m k a j ϕ j r_{-m}=Y-\sum_{j\ne m}^{k} a_j\phi_j r − m = Y − ∑ j = m k a j ϕ j , v m = ∥ ϕ m ∥ 2 2 v_m=\lVert\phi_m\rVert_2^2 v m = ∥ ϕ m ∥ 2 2 , and c m = ϕ m T r − m c_m=\phi_m^{\mathsf{T}}r_{-m} c m = ϕ m T r − m . For uniform output coefficients, if λ v m > 0 \lambda v_m>0 λ v m > 0 the conditional law is
a m ∣ other parameters , D n ∼ N ( c m v m , ( λ v m ) − 1 ) truncated to [ − k 2 , k 2 ] . a_m \mid\text{other parameters},D_n \sim\mathcal{N}\left(\frac{c_m}{v_m},(\lambda v_m)^{-1}\right)\ \text{truncated to }[-k^2,k^2]. a m ∣ other parameters , D n ∼ N ( v m c m , ( λ v m ) − 1 ) truncated to [ − k 2 , k 2 ] .
If λ v m = 0 \lambda v_m=0 λ v m = 0 , it is the uniform prior on that interval. For α , γ > 0 \alpha,\gamma>0 α , γ > 0 , IG ( α , γ ) \operatorname{IG}(\alpha,\gamma) IG ( α , γ ) denotes the inverse gamma distribution with density proportional to v − α − 1 e − γ / v v^{-\alpha-1}e^{-\gamma/v} v − α − 1 e − γ / v on v > 0 v>0 v > 0 . For Student output coefficients, introduce an auxiliary variance V m ∼ IG ( 5 , 5 ) V_m\sim\operatorname{IG}(5,5) V m ∼ IG ( 5 , 5 ) with a m ∣ V m ∼ N ( 0 , V m ) a_m\mid V_m\sim\mathcal{N}(0,V_m) a m ∣ V m ∼ N ( 0 , V m ) . The two conditional updates are
V m ∣ a m ∼ IG ( 11 / 2 , ( 10 + a m 2 ) / 2 ) , V_m \mid a_m \sim\operatorname{IG}\left(11/2,(10+a_m^2)/2\right), V m ∣ a m ∼ IG ( 11/2 , ( 10 + a m 2 ) /2 ) ,
a m ∣ V m , other parameters , D n ∼ N ( λ c m λ v m + V m − 1 , ( λ v m + V m − 1 ) − 1 ) . (47) a_m \mid V_m,\text{ other parameters},D_n \sim\mathcal{N}\left(\frac{\lambda c_m}{\lambda v_m+V_m^{-1}},(\lambda v_m+V_m^{-1})^{-1}\right). \tag*{(47)} a m ∣ V m , other parameters , D n ∼ N ( λ v m + V m − 1 λ c m , ( λ v m + V m − 1 ) − 1 ) . (47)
They define a Gibbs transition for the output coefficient, with the auxiliary variance regenerated during each sweep. Residuals are updated after each output coefficient. These conditional updates and the preceding Metropolis–Hastings moves preserve the target at their assigned power. At width k k k , network evaluation and the vector arithmetic in a sweep over the output coefficients take O ( n k ) O(nk) O ( nk ) operations per tempering level and O ( n k ) O(nk) O ( nk ) working memory, excluding retained diagnostics. A grid prediction takes O ( 100 9 k ) O(100^{9}k) O ( 10 0 9 k ) operations.
Exchanges between likelihood powers. Parallel tempering [13 ] uses 16 interacting chains at fixed powers
0 , 0.0009 , 0.0018 , 0.0036 , 0.0072 , 0.0144 , 0.0288 , 0.0576 , 0, 0.0009, 0.0018, 0.0036, 0.0072, 0.0144, 0.0288, 0.0576, 0 , 0.0009 , 0.0018 , 0.0036 , 0.0072 , 0.0144 , 0.0288 , 0.0576 ,
0.1152 , 0.2304 , 0.3456 , 0.4608 , 0.576 , 0.6912 , 0.8064 , 0.9. 0.1152, 0.2304, 0.3456, 0.4608, 0.576, 0.6912, 0.8064, 0.9. 0.1152 , 0.2304 , 0.3456 , 0.4608 , 0.576 , 0.6912 , 0.8064 , 0.9.
This set of chains is called a ladder. Its invariant joint distribution is the product of the 16 distributions at these powers. Every iteration updates every level once as described above. Every fifth iteration proposes exchanges between disjoint adjacent pairs, alternating ( 1 , 2 ) , ( 3 , 4 ) , … (1,2),(3,4),\ldots ( 1 , 2 ) , ( 3 , 4 ) , … and ( 2 , 3 ) , ( 4 , 5 ) , … (2,3),(4,5),\ldots ( 2 , 3 ) , ( 4 , 5 ) , … . For Gaussian states s , t s,t s , t at powers λ , λ ′ \lambda,\lambda' λ , λ ′ , the exchange probability is
1 ∧ exp { ℓ λ ( t ) + ℓ λ ′ ( s ) − ℓ λ ( s ) − ℓ λ ′ ( t ) } . 1 \wedge\exp\{\ell_{\lambda}(t) + \ell_{\lambda'}(s) - \ell_{\lambda}(s) - \ell_{\lambda'}(t)\}. 1 ∧ exp { ℓ λ ( t ) + ℓ λ ′ ( s ) − ℓ λ ( s ) − ℓ λ ′ ( t )} .
Each state is evaluated at its destination power and the common prior terms cancel. For the direct samplers it simplifies to
1 ∧ exp { λ − λ ′ 2 [ R ( θ ) − R ( θ ′ ) ] } . 1 \wedge\exp\left\{\frac{\lambda-\lambda'}{2}\left[R(\theta)-R(\theta')\right]\right\}. 1 ∧ exp { 2 λ − λ ′ [ R ( θ ) − R ( θ ′ ) ] } .
where θ , θ ′ \theta,\theta' θ , θ ′ are the states currently at λ , λ ′ \lambda,\lambda' λ , λ ′ . The exchanges preserve the joint target. Only states at the final power 0.9 are retained for estimation.
Prediction from posterior samples. For a Gaussian state at power η \eta η , let ϕ x = ( ρ ( u m x + b m ) ) m = 1 k \phi_x = \left(\rho(u_m x + b_m)\right)_{m=1}^k ϕ x = ( ρ ( u m x + b m ) ) m = 1 k . The conditional latent prediction is normal with μ x = ϕ x T m η \mu_x = \phi_x^{\mathsf T}m_\eta μ x = ϕ x T m η and σ x 2 = ϕ x T Q η − 1 ϕ x \sigma_x^2 = \phi_x^{\mathsf T}Q_\eta^{-1}\phi_x σ x 2 = ϕ x T Q η − 1 ϕ x . Put B ~ = B η \widetilde{B} = B_\eta B = B η for this data set. If σ x > 0 \sigma_x > 0 σ x > 0 , set α x = ( − B − μ x ) / σ x \alpha_x = (-B-\mu_x)/\sigma_x α x = ( − B − μ x ) / σ x and γ x = ( B − μ x ) / σ x \gamma_x = (B-\mu_x)/\sigma_x γ x = ( B − μ x ) / σ x . The conditional clipped mean is
h x ( s ) = − B Φ ( α x ) + B { 1 − Φ ( γ x ) } + μ x { Φ ( γ x ) − Φ ( α x ) } + σ x { ϕ ( α x ) − ϕ ( γ x ) } . (48) h_x(s) = -B\Phi(\alpha_x) + B\{1-\Phi(\gamma_x)\}
+ \mu_x\{\Phi(\gamma_x)-\Phi(\alpha_x)\} + \sigma_x\{\phi(\alpha_x)-\phi(\gamma_x)\}. \tag*{(48)} h x ( s ) = − B Φ ( α x ) + B { 1 − Φ ( γ x )} + μ x { Φ ( γ x ) − Φ ( α x )} + σ x { ϕ ( α x ) − ϕ ( γ x )} . (48)
If σ x = 0 \sigma_x=0 σ x = 0 , set h x ( s ) = C B ( μ x ) h_x(s)=C_B(\mu_x) h x ( s ) = C B ( μ x ) . For the combined posterior samples s 1 , … , s S s_1,\ldots,s_S s 1 , … , s S , the Gaussian estimate is S − 1 ∑ j = 1 S h x ( s j ) S^{-1}\sum_{j=1}^{S}h_x(s_j) S − 1 ∑ j = 1 S h x ( s j ) . The direct samplers instead average C B ( f θ j ( x ) ) C_B(f_{\theta_j}(x)) C B ( f θ j ( x )) over draws of all network parameters θ 1 , … , θ S \theta_1,\ldots,\theta_S θ 1 , … , θ S . Both averages approximate the clipped posterior mean in (15) ; σ x 2 \sigma_x^2 σ x 2 is the conditional variance of the latent function, without observation-noise variance.
Initialization, retention, and diagnostics. Separate ladders use independent seeds. For Gaussian network parameters the chain at power η = 0.9 \eta=0.9 η = 0.9 in ladder j j j starts at width j j j , giving widths 1 , … , L 1,\ldots,L 1 , … , L for a run with L L L ladders. Other Gaussian levels, and all direct-sampler levels, start at k = min { 64 , 1 + Z } k=\min\{64,1+Z\} k = min { 64 , 1 + Z } , where P ( Z = z ) = 2 − ( z + 1 ) \mathbb{P}(Z=z)=2^{-(z+1)} P ( Z = z ) = 2 − ( z + 1 ) , z = 0 , 1 , … z=0,1,\ldots z = 0 , 1 , … . Network parameters are drawn from their conditional priors; Gaussian output coefficients are integrated rather than initialized. These are starting distributions; the acceptance ratios use the conditioned prior in (42) .
Table 2 specifies the eight computation stages, all fixed before this η = 0.9 \eta=0.9 η = 0.9 study. A later stage is considered only after all preceding stages fail the unchanged diagnostic screens, and the first accepted stage is retained. Every stage uses the same data, priors, 16 likelihood powers, parameter updates, clipping, width cap and diagnostic thresholds. Independent chain seeds depend on the stage index. We discard the first half of each ladder as burn-in and retain every twentieth subsequent state at power η = 0.9 \eta= 0.9 η = 0.9 .
Table 2: Computation stages. Later stages are used only after every preceding stage fails the unchanged diagnostics. All eight stages were fixed before the study.
Table 2.
We require potential scale reduction factor R ^ ≤ 1.10 \hat{R} \le1.10 R ^ ≤ 1.10 [11 ] and pooled effective sample size at least 50 for width and for clipped predictions at five points near 0.1, 0.3, 0.5, 0.7, 0.9. For uniform and Student network parameters, the corresponding unclipped prediction diagnostics must also pass. We calculate these diagnostics with coda [22 ] .
For each diagnostic, the pooled effective sample size is the sum of the per-ladder estimates. We also require the width to remain below 64 at every post-burn-in iteration at likelihood power 0.9, including iterations omitted by thinning. For each parameter family, target and sample size, the fits for all 50 replications pass these checks. Estimated risk is never used to select fits or computation stages.
The calculations used R 4.4.0 on an arm64 macOS computer. The repository provides the settings and seeds for each reported fit. For the displayed Gaussian fits, independently recomputing posterior means and squared errors from the sampled parameters gives agreement to within 10 − 10 10^{-10} 1 0 − 10 , as detailed in Appendix § 14 .§ 2 .
All fits pass the diagnostic criteria. The original eight computation stages were specified before the study. After the sole unresolved original fit, U/G for the Hölder 1 / 2 1/2 1/2 target at n = 4096 n = 4096 n = 4096 , replication 15, failed all eight stages, four additional stages were specified before their execution. These stages use 6.144, 12.288, 24.576 and 49.152 million iterations per ladder, respectively, with eight independent ladders, half burn-in and thinning 20; the first passing stage is retained. The unresolved fit first passed the diagnostics at stage 10. The original data, seed formula, priors, sampler, 16 likelihood-power levels and diagnostic thresholds are unchanged. No fit occupies the width cap after burn-in. These diagnostic screens do not prove exploration of every posterior mode or bound residual sampling error.
The finite chain lengths and the use of grid averages to approximate design integrals may introduce residual error. Conditioning on width at most 64 changes the prior, and convergence diagnostics from the sampled chains do not bound posterior mass beyond that limit.
Under geometric width and Gaussian network parameters, mean errors at n = 64 n = 64 n = 64 and 4096 are 0.08501 and 0.00597 for β = 1 / 2 \beta= 1/2 β = 1/2 , and 0.12262 and 0.00408 for β = 1 \beta= 1 β = 1 . The endpoint exponents ρ 64 , 4096 \rho_{64,4096} ρ 64 , 4096 are 0.639 and 0.818, with 95% paired-bootstrap intervals [0.609, 0.670] and [0.791, 0.846]. These finite-range comparisons describe fixed targets rather than worst-case Hölder-ball risk.
Table 3 gives the Gaussian/geometric errors and clipping means at all sample sizes.
Table 3. Squared prediction error with Gaussian parameters and geometric width. Each estimate uses 50 independent replications. The SE column gives Monte Carlo standard errors. Supplementary Tables 6 and 7 give pointwise 95% normal Monte Carlo confidence intervals.
Sensitivity to the fixed width law
We keep the Gaussian conditional network-parameter law and use the fixed width laws
π G ( k ) = 2 − k , π P ( k ) = e − 1 ( k − 1 ) ! , π H ( k ) = k − 5 / 2 ∑ j = 1 ∞ j − 5 / 2 , k ≥ 1. (49) \pi_G(k)=2^{-k},\qquad\pi_P(k)=\frac{e^{-1}}{(k-1)!},\qquad\pi_H(k)=\frac{k^{-5/2}}{\sum_{j=1}^{\infty}j^{-5/2}},\qquad k\ge1. \tag*{(49)} π G ( k ) = 2 − k , π P ( k ) = ( k − 1 )! e − 1 , π H ( k ) = ∑ j = 1 ∞ j − 5/2 k − 5/2 , k ≥ 1. (49)
They satisfy (10) and have prior mean widths 2, 2, and approximately 1.95, respectively. Each numerical law is conditioned on K ≤ 64 K \le64 K ≤ 64 .
Let Q n , 64 η , π Q_{n,64}^{\eta,\pi} Q n , 64 η , π be the posterior on T 64 \mathcal{T}_{64} T 64 of ( K , θ K ) (K,\theta_K) ( K , θ K ) under width law π \pi π , conditional on K ≤ 64 K \le64 K ≤ 64 . For any integrable function h h h of the pair,
∫ T 64 h d Q n , 64 η , π = ∫ T 64 h w π ( K ) d Q n , 64 η , π G ∫ T 64 w π ( K ) d Q n , 64 η , π G , w π ( k ) = π ( k ) π G ( k ) . (50) \int_{\mathcal{T}_{64}} h\,dQ_{n,64}^{\eta,\pi}
=
\frac{\int_{\mathcal{T}_{64}} h\,w_\pi(K)\,dQ_{n,64}^{\eta,\pi_G}}{\int_{\mathcal{T}_{64}} w_\pi(K)\,dQ_{n,64}^{\eta,\pi_G}},
\qquad
w_\pi(k)=\frac{\pi(k)}{\pi_G(k)}. \tag*{(50)} ∫ T 64 h d Q n , 64 η , π = ∫ T 64 w π ( K ) d Q n , 64 η , π G ∫ T 64 h w π ( K ) d Q n , 64 η , π G , w π ( k ) = π G ( k ) π ( k ) . (50)
The conditional network-parameter law is unchanged and the truncation normalizers cancel. We apply this identity to the conditional expectation of the clipped output, using the same clipping and data for all laws. Importance weights are normalized within each data set before calculating its posterior mean and squared error. For the S S S posterior draws with widths k 1 , … , k S k_1,\ldots,k_S k 1 , … , k S , set w s = w π ( k s ) w_s=w_\pi(k_s) w s = w π ( k s ) . The Kish effective sample size, expressed as a fraction of S S S ([14 pp. 190–191] ), ( ∑ s = 1 S w s ) 2 / ( S ∑ s = 1 S w s 2 ) (\sum_{s=1}^{S}w_s)^2/(S\sum_{s=1}^{S}w_s^2) ( ∑ s = 1 S w s ) 2 / ( S ∑ s = 1 S w s 2 ) , measures the concentration of the importance weights among the posterior draws. It does not account for their Markov dependence or unsampled modes.
Across the Gaussian fits for each target, minimum Kish effective sample size fractions for shifted Poisson reweighting are 0.403 and 0.482 for β = 1 / 2 \beta=1/2 β = 1/2 and β = 1 \beta=1 β = 1 . For power-law width they are 0.306 and 0.194, respectively.
Table 4 reports normalized errors and standard errors across the 50 original data replications in each case. The Gaussian width comparisons reweight the same posterior samples. This comparison does not establish a uniform rate over all width laws satisfying (10).
Table 4: Squared prediction error divided by q n , β = ( log n / n ) 2 β / ( 2 β + 1 ) q_{n,\beta} = (\log n/n)^{2\beta/(2\beta+1)} q n , β = ( log n / n ) 2 β / ( 2 β + 1 ) under three width laws with Gaussian parameters. Each estimate uses 50 independent replications. Parentheses give Monte Carlo standard errors. Supplementary Tables 6 and 7 give pointwise 95% normal Monte Carlo confidence intervals.
Table 4.
Sensitivity to the parameter law
Uniform and Student t 10 t_{10} t 10 simulations use geometric width and the computational methods above. Figure 2 and the table below report 50 original diagnostic-passing data replications per case.
Table 5: Squared prediction error for uniform and Student t 10 t_{10} t 10 parameters with geometric width. Each estimate uses 50 independent replications. Parentheses give Monte Carlo standard errors. Supplementary Tables 6 and 7 give pointwise 95% normal Monte Carlo confidence intervals.
Table 5.
Supplementary numerical results
Supplementary Table 6 and Table 7 report the unnormalized mean squared errors, Monte Carlo standard errors and confidence intervals for all prior comparisons at every sample size. They use the same 50 replications and interval formula as Figure 2 .
Table 6. Supplementary squared-error estimates for the Hölder 1 / 2 1/2 1/2 target. Each mean uses 50 replications; SE is the Monte Carlo standard error and CI is the pointwise 95% normal Monte Carlo confidence interval. N/U/t 10 t_{10} t 10 denote Gaussian/uniform/Student parameters; G/P/H denote geometric/shifted Poisson/power-law width.
Table 7 caption: Supplementary squared-error estimates for the Lipschitz target. Each mean uses 50 replications; SE is the Monte Carlo standard error and CI is the pointwise 95% normal Monte Carlo confidence interval. N/U/t 10 t_{10} t 10 denote Gaussian/uniform/Student parameters; G/P/H denote geometric/shifted Poisson/power-law width.
Table 7. Supplementary squared-error estimates for the Lipschitz target. Each mean uses 50 replications; SE is the Monte Carlo standard error and CI is the pointwise 95% normal Monte Carlo confidence interval. N/U/t 10 t_{10} t 10 denote Gaussian/uniform/Student parameters; G/P/H denote geometric/shifted Poisson/power-law width.
Parameter-law probability calculations
For an admissible centre θ ∗ \theta^* θ ∗ in (11) and 0 < δ ≤ 1 / 2 0 < \delta\le1/2 0 < δ ≤ 1/2 , write
B k ( θ ∗ , δ ) = { θ ∈ R ( d + 2 ) k : ∥ θ − θ ∗ ∥ ∞ ≤ δ } . B_k(\theta^*,\delta)=\{\theta\in\mathbb{R}^{(d+2)k}:\|\theta-\theta^*\|_\infty\le\delta\}. B k ( θ ∗ , δ ) = { θ ∈ R ( d + 2 ) k : ∥ θ − θ ∗ ∥ ∞ ≤ δ } .
We prove the bound (12) for every such centre, radius, and width, with constants independent of them.
The output intervals lie inside [ − k ω , k ω ] [-k^\omega,k^\omega] [ − k ω , k ω ] because ∣ a m ∗ ∣ + δ ≤ k ω / 2 + 1 / 2 ≤ k ω |a_m^*|+\delta\le k^\omega/2+1/2\le k^\omega ∣ a m ∗ ∣ + δ ≤ k ω /2 + 1/2 ≤ k ω . The hidden-layer coefficient and bias centres are bounded by one, so their intervals lie inside [ − 2 , 2 ] [-2,2] [ − 2 , 2 ] . Independence gives
Π k θ { B k ( θ ∗ , δ ) } = ( δ k ω ) k ( δ 2 ) ( d + 1 ) k . (51) \Pi_k^\theta\{B_k(\theta^*,\delta)\}=\left(\frac{\delta}{k^\omega}\right)^k\left(\frac{\delta}{2}\right)^{(d+1)k}. \tag*{(51)} Π k θ { B k ( θ ∗ , δ )} = ( k ω δ ) k ( 2 δ ) ( d + 1 ) k . (51)
Consequently,
− log Π k θ { B k ( θ ∗ , δ ) } = k ω log k + ( d + 2 ) k log ( 1 / δ ) + ( d + 1 ) k log 2 ≲ k log ( e k / δ ) . (52) -\log\Pi_k^\theta\{B_k(\theta^*,\delta)\}=k\omega\log k+(d+2)k\log(1/\delta)+(d+1)k\log2\lesssim k\log(ek/\delta). \tag*{(52)} − log Π k θ { B k ( θ ∗ , δ )} = k ω log k + ( d + 2 ) k log ( 1/ δ ) + ( d + 1 ) k log 2 ≲ k log ( e k / δ ) . (52)
This proves the uniform case of Lemma 2. At a fixed width the parameter support is compact, and its continuous image in C ( X ) C(\mathcal{X}) C ( X ) is compact. The full mixture allows arbitrarily large widths and does not have compact function support.
Gaussian parameters
For Z ∼ N ( 0 , τ 2 ) Z\sim N(0,\tau^2) Z ∼ N ( 0 , τ 2 ) , τ ≥ 1 \tau\ge1 τ ≥ 1 , and z ∈ R z\in\mathbb{R} z ∈ R , integrating the density on [ z − δ , z + δ ] [z-\delta,z+\delta] [ z − δ , z + δ ] gives
P { ∣ Z − z ∣ ≤ δ } ≥ 2 δ 2 π τ exp { − ( ∣ z ∣ + δ ) 2 2 τ 2 } . (53) \mathbb{P}\{|Z-z|\le\delta\}\ge\frac{2\delta}{\sqrt{2\pi\tau}}\exp\left\{-\frac{(|z|+\delta)^2}{2\tau^2}\right\}. \tag*{(53)} P { ∣ Z − z ∣ ≤ δ } ≥ 2 π τ 2 δ exp { − 2 τ 2 ( ∣ z ∣ + δ ) 2 } . (53)
Since ( ∣ z ∣ + δ ) 2 ≤ 2 z 2 + 2 δ 2 (|z|+\delta)^2\le2z^2+2\delta^2 ( ∣ z ∣ + δ ) 2 ≤ 2 z 2 + 2 δ 2 , the negative logarithm is at most C + log ( τ / δ ) + z 2 / τ 2 C+\log(\tau/\delta)+z^2/\tau^2 C + log ( τ / δ ) + z 2 / τ 2 . The bounded hidden-layer coefficient and bias centres add only O ( k ) O(k) O ( k ) beyond the precision terms. With output standard deviation τ k = k 3 / ( 2 d ) \tau_k=k^{3/(2d)} τ k = k 3/ ( 2 d ) , independence therefore yields
− log Π k θ { B k ( θ ∗ , δ ) } ≤ C k + ( d + 2 ) k log ( 1 / δ ) + k log τ k + τ k − 2 ∑ m = 1 k ∣ a m ∗ ∣ 2 . (54) \begin{aligned}
-\log\Pi_k^\theta\{B_k(\theta^*,\delta)\}
&\le Ck+(d+2)k\log(1/\delta)\\
&\quad+k\log\tau_k+\tau_k^{-2}\sum_{m=1}^{k}|a_m^*|^2. \tag*{(54)}
\end{aligned} − log Π k θ { B k ( θ ∗ , δ )} ≤ C k + ( d + 2 ) k log ( 1/ δ ) + k log τ k + τ k − 2 m = 1 ∑ k ∣ a m ∗ ∣ 2 . (54)
Admissibility controls the final term:
τ k − 2 ∑ m = 1 k ∣ a m ∗ ∣ 2 ≤ k − 3 / d ( ∑ m = 1 k ∣ a m ∗ ∣ ) 2 ≤ 1 4 k − 3 / d k 2 ω = 1 4 k . (55) \tau_k^{-2}\sum_{m=1}^{k}|a_m^*|^2\le k^{-3/d}\left(\sum_{m=1}^{k}|a_m^*|\right)^2\le\frac14 k^{-3/d}k^{2\omega}=\frac14 k. \tag*{(55)} τ k − 2 m = 1 ∑ k ∣ a m ∗ ∣ 2 ≤ k − 3/ d ( m = 1 ∑ k ∣ a m ∗ ∣ ) 2 ≤ 4 1 k − 3/ d k 2 ω = 4 1 k . (55)
Also k log τ k = ( 3 / ( 2 d ) ) k log k k\log\tau_k=(3/(2d))k\log k k log τ k = ( 3/ ( 2 d )) k log k . Thus (54) is O ( k log ( e k / δ ) ) O(k\log(ek/\delta)) O ( k log ( e k / δ )) , proving the Gaussian case of Lemma 2. For unit-variance Gaussian output parameters, an admissible centre with a 1 ∗ = k ω / 2 a_1^*=k^\omega/2 a 1 ∗ = k ω /2 and all other coordinates zero has a quadratic log-probability cost of order k 2 ω k^{2\omega} k 2 ω . Since 2 ω = 1 + 3 / d > 1 2\omega=1+3/d>1 2 ω = 1 + 3/ d > 1 , it fails (12) at any fixed positive radius. This failure of the sufficient condition does not determine that prior’s posterior rate.
Student t t t and Cauchy parameters
For every fixed ν > 0 \nu> 0 ν > 0 , the density of t ν t_\nu t ν is
q ν ( z ) = Γ ( ( ν + 1 ) / 2 ) ν π Γ ( ν / 2 ) ( 1 + z 2 ν ) − ( ν + 1 ) / 2 , z ∈ R . q_\nu(z) = \frac{\Gamma((\nu+ 1)/2)}{\sqrt{\nu\pi}\Gamma(\nu/2)}\left(1 + \frac{z^2}{\nu}\right)^{-(\nu+1)/2}, \qquad z \in\mathbb{R}. q ν ( z ) = ν π Γ ( ν /2 ) Γ (( ν + 1 ) /2 ) ( 1 + ν z 2 ) − ( ν + 1 ) /2 , z ∈ R .
It satisfies q ν ( z ) ≥ c ν ( 1 + ∣ z ∣ ) − ( ν + 1 ) q_\nu(z) \ge c_\nu(1 + |z|)^{-(\nu+1)} q ν ( z ) ≥ c ν ( 1 + ∣ z ∣ ) − ( ν + 1 ) . If Z Z Z has a probability density q q q on R \mathbb{R} R satisfying q ( z ) ≥ c ( 1 + ∣ z ∣ ) − r q(z) \ge c(1 + |z|)^{-r} q ( z ) ≥ c ( 1 + ∣ z ∣ ) − r for all z ∈ R z \in\mathbb{R} z ∈ R , then integration over [ z − δ , z + δ ] [z-\delta,z+\delta] [ z − δ , z + δ ] gives
P { ∣ Z − z ∣ ≤ δ } ≥ 2 c δ 2 − r ( 1 + ∣ z ∣ ) − r . (56) \mathbb{P}\{|Z-z| \le\delta\} \ge2c\delta2^{-r}(1+|z|)^{-r}. \tag*{(56)} P { ∣ Z − z ∣ ≤ δ } ≥ 2 cδ 2 − r ( 1 + ∣ z ∣ ) − r . (56)
For Student parameters, use r = ν + 1 r=\nu+1 r = ν + 1 . All hidden-layer and bias centres are bounded by one, and admissibility gives ∣ a m ∗ ∣ ≤ k ω / 2 |a_m^*| \le k^\omega/2 ∣ a m ∗ ∣ ≤ k ω /2 . Summing the negative logarithms of the independent interval probabilities gives
− log Π k 0 { B k ( θ ∗ , δ ) } ≤ C k + ( d + 2 ) k log ( 1 / δ ) + ( ν + 1 ) k log ( 1 + k ω / 2 ) ≲ k log ( e k / δ ) . (57) -\log\Pi_k^0\{B_k(\theta^*,\delta)\} \le Ck + (d+2)k\log(1/\delta)
+(\nu+1)k\log(1+k^\omega/2) \lesssim k\log(ek/\delta). \tag*{(57)} − log Π k 0 { B k ( θ ∗ , δ )} ≤ C k + ( d + 2 ) k log ( 1/ δ ) + ( ν + 1 ) k log ( 1 + k ω /2 ) ≲ k log ( e k / δ ) . (57)
This proves the Student case and completes the proof of Lemma 2. The Cauchy law is t 1 t_1 t 1 , and no moment of the parameter law is used.
Polynomial density lower bounds
Proof of Proposition 3. Apply (56) to each scalar parameter. As in the Student calculation, all hidden-layer and bias centres are bounded by one and ∣ a m ∗ ∣ ≤ k ω / 2 |a_m^*| \le k^\omega/2 ∣ a m ∗ ∣ ≤ k ω /2 . Independence bounds the negative log probability of B k ( θ ∗ , δ ) B_k(\theta^*,\delta) B k ( θ ∗ , δ ) by
C k + ( d + 2 ) k log ( 1 / δ ) + r k log ( 1 + k ω / 2 ) ≲ k log ( e k / δ ) . Ck + (d+2)k\log(1/\delta) + rk\log(1+k^\omega/2) \lesssim k\log(ek/\delta). C k + ( d + 2 ) k log ( 1/ δ ) + r k log ( 1 + k ω /2 ) ≲ k log ( e k / δ ) .
The constant depends only on d , c , r d,c,r d , c , r . This proves (12); paired with (10), the prior is therefore covered by Proposition 2 and the contraction and risk theorems. No moment assumption is needed.
Fractional-posterior prior-probability argument
Proof of Lemma 1. We first give a lower bound for the posterior denominator and then an upper bound for the expected numerator outside the stated Rényi neighbourhood. Let B n = { f ∈ C ( X ) : ∥ f − f 0 ∥ ∞ ≤ ε n } B_n = \{f \in C(\mathcal{X}) : \|f-f_0\|_\infty\le\varepsilon_n\} B n = { f ∈ C ( X ) : ∥ f − f 0 ∥ ∞ ≤ ε n } and Π B n = Π ( ⋅ ∣ B n ) \Pi_{B_n} = \Pi(\cdot\mid B_n) Π B n = Π ( ⋅ ∣ B n ) . The likelihood ratio under f f f relative to f 0 f_0 f 0 is
log R n ( f ) = − 1 2 ∑ i = 1 n { ( Y i − f ( X i ) ) 2 − ( Y i − f 0 ( X i ) ) 2 } . \log R_n(f) = -\frac{1}{2}\sum_{i=1}^{n}\{(Y_i-f(X_i))^2-(Y_i-f_0(X_i))^2\}. log R n ( f ) = − 2 1 i = 1 ∑ n {( Y i − f ( X i ) ) 2 − ( Y i − f 0 ( X i ) ) 2 } .
The powered denominator is Z n = ∫ C ( X ) R n ( f ) η Π ( d f ) Z_n = \int_{C(\mathcal{X})} R_n(f)^\eta\Pi(\mathrm{d}f) Z n = ∫ C ( X ) R n ( f ) η Π ( d f ) . Restricting this integral to B n B_n B n and applying Jensen’s inequality gives
Z n ≥ Π ( B n ) e η Λ n , Λ n = ∫ B n log R n ( f ) Π B n ( d f ) . (58) Z_n \ge\Pi(B_n)e^{\eta\Lambda_n}, \qquad\Lambda_n = \int_{B_n}\log R_n(f)\Pi_{B_n}(\mathrm{d}f). \tag*{(58)} Z n ≥ Π ( B n ) e η Λ n , Λ n = ∫ B n log R n ( f ) Π B n ( d f ) . (58)
Write h i ( f ) = f ( X i ) − f 0 ( X i ) h_i(f)=f(X_i)-f_0(X_i) h i ( f ) = f ( X i ) − f 0 ( X i ) . Since Y i = f 0 ( X i ) + ξ i Y_i=f_0(X_i)+\xi_i Y i = f 0 ( X i ) + ξ i , expansion of the squares gives
Λ n = ∑ i = 1 n ξ i ∫ B n h i ( f ) Π B n ( d f ) − 1 2 ∑ i = 1 n ∫ B n h i ( f ) 2 Π B n ( d f ) . \Lambda_n = \sum_{i=1}^{n}\xi_i\int_{B_n}h_i(f)\Pi_{B_n}(\mathrm{d}f)-\frac{1}{2}\sum_{i=1}^{n}\int_{B_n}h_i(f)^2\Pi_{B_n}(\mathrm{d}f). Λ n = i = 1 ∑ n ξ i ∫ B n h i ( f ) Π B n ( d f ) − 2 1 i = 1 ∑ n ∫ B n h i ( f ) 2 Π B n ( d f ) .
Conditional on the covariates, the integrals are deterministic and the noises are independent standard normals. Thus Λ n \Lambda_n Λ n is Gaussian, with conditional mean and variance
μ n = − 1 2 ∑ i = 1 n ∫ B n h i ( f ) 2 Π B n ( d f ) ≥ − 1 2 n ε n 2 , \mu_n=-\frac{1}{2}\sum_{i=1}^{n}\int_{\mathcal B_n}h_i(f)^2\Pi_{\mathcal B_n}(\mathrm{d}f)\geq-\frac{1}{2}n\varepsilon_n^2, μ n = − 2 1 i = 1 ∑ n ∫ B n h i ( f ) 2 Π B n ( d f ) ≥ − 2 1 n ε n 2 ,
σ n 2 = ∑ i = 1 n { ∫ B n h i ( f ) Π B n ( d f ) } 2 ≤ n ε n 2 . \sigma_n^2=\sum_{i=1}^{n}\left\{\int_{\mathcal B_n}h_i(f)\Pi_{\mathcal B_n}(\mathrm{d}f)\right\}^2\leq n\varepsilon_n^2. σ n 2 = i = 1 ∑ n { ∫ B n h i ( f ) Π B n ( d f ) } 2 ≤ n ε n 2 .
The bounds use ∣ h i ( f ) ∣ ≤ ε n |h_i(f)|\leq\varepsilon_n ∣ h i ( f ) ∣ ≤ ε n on B n \mathcal B_n B n . A Gaussian tail bound gives P f 0 { Λ n < − n ε n 2 } ≤ e − n ε n 2 / 8 \mathbb P_{f_0}\{\Lambda_n<-n\varepsilon_n^2\}\leq e^{-n\varepsilon_n^2/8} P f 0 { Λ n < − n ε n 2 } ≤ e − n ε n 2 /8 ; the zero-variance case satisfies the same bound. The assumed prior probability and (58) imply
P f 0 { Z n < e − ( c 0 + η ) n ε n 2 } ≤ e − n ε n 2 / 8 . (59) \mathbb P_{f_0}\{Z_n<e^{-(c_0+\eta)n\varepsilon_n^2}\}\leq e^{-n\varepsilon_n^2/8}. \tag*{(59)} P f 0 { Z n < e − ( c 0 + η ) n ε n 2 } ≤ e − n ε n 2 /8 . (59)
Set A n = { f ∈ C ( X ) : D η ( f , f 0 ) > J ε n 2 } \mathcal A_n=\{f\in C(\mathcal X):D_\eta(f,f_0)>J\varepsilon_n^2\} A n = { f ∈ C ( X ) : D η ( f , f 0 ) > J ε n 2 } . Integrating the Gaussian noises conditional on the covariates gives
E f 0 { R n ( f ) η ∣ X 1 , … , X n } = exp { − η ( 1 − η ) 2 ∑ i = 1 n ( f ( X i ) − f 0 ( X i ) ) 2 } . \mathbb E_{f_0}\{R_n(f)^\eta\mid X_1,\ldots,X_n\}=\exp\left\{-\frac{\eta(1-\eta)}{2}\sum_{i=1}^{n}(f(X_i)-f_0(X_i))^2\right\}. E f 0 { R n ( f ) η ∣ X 1 , … , X n } = exp { − 2 η ( 1 − η ) i = 1 ∑ n ( f ( X i ) − f 0 ( X i ) ) 2 } .
Integrating the independent covariates and using (4) therefore yields
E f 0 R n ( f ) η = e − ( 1 − η ) n D η ( f , f 0 ) . \mathbb E_{f_0}R_n(f)^\eta=e^{-(1-\eta)nD_\eta(f,f_0)}. E f 0 R n ( f ) η = e − ( 1 − η ) n D η ( f , f 0 ) .
By Tonelli’s theorem,
E f 0 ∫ A n R n ( f ) η Π ( d f ) ≤ e − ( 1 − η ) J n ε n 2 . \mathbb E_{f_0}\int_{\mathcal A_n}R_n(f)^\eta\Pi(\mathrm{d}f)\leq e^{-(1-\eta)Jn\varepsilon_n^2}. E f 0 ∫ A n R n ( f ) η Π ( d f ) ≤ e − ( 1 − η ) J n ε n 2 .
On the event Z n ≥ e − ( c 0 + η ) n ε n 2 Z_n\geq e^{-(c_0+\eta)n\varepsilon_n^2} Z n ≥ e − ( c 0 + η ) n ε n 2 , the posterior probability of A n \mathcal A_n A n is at most its numerator multiplied by e ( c 0 + η ) n ε n 2 e^{(c_0+\eta)n\varepsilon_n^2} e ( c 0 + η ) n ε n 2 . On the complementary event it is at most one. Taking expectations and applying (59) gives
E f 0 Π η ( A n ∣ D n ) ≤ e − n ε n 2 / 8 + e − { ( 1 − η ) J − c 0 − η } n ε n 2 . \mathbb E_{f_0}\Pi^\eta(\mathcal A_n\mid\mathcal D_n)\leq e^{-n\varepsilon_n^2/8}+e^{-\{(1-\eta)J-c_0-\eta\}n\varepsilon_n^2}. E f 0 Π η ( A n ∣ D n ) ≤ e − n ε n 2 /8 + e − {( 1 − η ) J − c 0 − η } n ε n 2 .
All estimates are uniform over the assumed class of targets, proving (5) . □ \square □