betaFeedback

WorksMathematics

Sieve-free ReLU Bayesian priors with near-minimax posterior contraction

Abstract

We establish fractional-posterior contraction for shallow ReLU networks in Gaussian random-design regression with a ββ-Hölder target, using priors independent of the sample size and smoothness parameters. For any fixed design law PXP_X on [0,1]d[0,1]^d, we obtain contraction in Rényi divergence and, after clipping posterior draws at a bound on the target range, in squared L2(PX)\mathcal{L}^2(P_X) loss at rate (log⁡n/n)2β/(2β+d)(\log n/n)^{2β/(2β+d)}. This matches the classical Hölder minimax rate up to a logarithmic factor. These priors have random finite width and use width-dependent uniform or Gaussian weight distributions, or width-independent Student tt distributions, including Cauchy distributions.

Introduction

A central question in the Bayesian theory of neural networks is whether posteriors under network priors attain optimal frequentist rates of estimation. For deep ReLU networks, near-minimax contraction has been established under spike-and-slab priors whose nonzero network parameters are uniform on a bounded interval [23], under Gaussian and other network-parameter laws with positive densities [16], and under heavy-tailed network-parameter laws such as Student tt and Cauchy distributions [8]. In each case, however, the joint prior changes with the sample size nn: some part of the architecture, its distribution, or the network-parameter scale is chosen as a function of nn. This allows the prior specification to be calibrated to the statistical resolution available at sample size nn, but means that a different prior is used as the amount of data changes. We show that this dependence on nn is unnecessary. Here, sieve-free refers to a prior fixed across sample sizes; this use of the term should not be confused with mixtures over finite-dimensional models, which are also called sieve priors [3].

Our main aim is to isolate the mechanism that allows a single fixed prior to attain the same rates. A fixed prior cannot be tuned to one statistical accuracy; it must instead allocate enough probability simultaneously across all relevant approximation scales. For one-hidden-layer networks, width provides a natural index for these scales. Given a target and an accuracy ε>0\varepsilon> 0, we use the approximation bounds of Yang and Zhou [28, 29] to select an approximating network of suitable width kk, and ask how much probability the same prior assigns to a neighbourhood of its parameters. The hidden-layer coefficients and biases of these approximators remain bounded, while the required output coefficients grow at most polynomially with the approximation complexity. These bounds can be dominated by rules depending only on the sampled width and the input dimension. This suggests a hierarchical construction: specify once and for all a law on all finite widths and, conditional on width kk, a law on the corresponding network parameters. Neither nn nor the unknown smoothness or radius of the target need enter the prior.

The argument separates the approximation problem from a local prior-mass requirement: the prior must assign sufficient probability to a suitable width and to accurate parameters at that width. We verify this requirement for three qualitatively different conditional network-parameter families: bounded uniform laws with width-dependent output support, Gaussian laws with a width-dependent output variance, and fixed Student tt laws, including the Cauchy distribution. The same calculation applies to fixed network-parameter densities satisfying a polynomial lower bound. These examples identify local mass near approximating networks as the property that permits adaptation.

We work with one-hidden-layer networks in Gaussian random-design regression, with fixed input dimension d≥1d \ge1 and design law PXP_X on [0,1]d[0,1]^d. Width indexes approximation complexity, and uniform approximation gives likelihood neighbourhoods for arbitrary design laws. Following Bhattacharya et al. [7] and Castillo and Egels [8], we study a fractional posterior, which raises the likelihood to a fixed power 0<η<10 < \eta< 1. Its contraction argument requires only sufficient local prior mass, avoiding the additional global control of large models needed for ordinary-posterior contraction. The order-η\eta Rényi divergence arises directly from the fractional likelihood ratio. For Hölder smoothness 0<β≤10 < \beta\le1 and radius M>0M > 0, the fractional posterior under the same fixed prior contracts in Rényi divergence at order (log⁡n/n)2β/(2β+d)(\log n/n)^{2\beta/(2\beta+d)}, uniformly over the corresponding Hölder ball.

Rényi contraction isolates this mechanism, but squared prediction error is a stronger and more familiar frequentist loss. The passage between the two creates a separate difficulty: posterior draws under our fixed priors need not be uniformly bounded, and large discrepancies on sets of small PXP_X-probability can have a much greater effect on squared error than on Rényi divergence. Rather than impose additional restrictions on the prior or a clipping level depending on the unknown target radius, we clip posterior draws at a level estimated from the data. Under a design density bounded away from zero, this level contains the range of the target with exponentially high probability. Once both functions are bounded, Rényi divergence controls squared prediction error. The resulting posterior mean has risk (log⁡n/n)2β/(2β+d)(\log n/n)^{2\beta/(2\beta+d)} under the lower density bound. If the density is also bounded above, the classical Hölder minimax lower bound shows that this rate is optimal up to a logarithmic factor. An alternative clipping construction retains the same upper risk rate under arbitrary design laws, at the cost of an additional polynomial term in the posterior-tail bound. Clipping also ensures that the posterior mean exists. For Cauchy and Student network-parameter priors with at most one degree of freedom, the unclipped mean can fail to exist when a unit is inactive at every observed covariate; see Appendix § 11. Whether squared prediction-loss contraction holds without clipping, or the unclipped mean attains the same risk rate under conditions ensuring its existence, remains open.

To the best of our knowledge, this is the first adaptive near-minimax fractional-posterior contraction result for dense shallow ReLU networks under a prior independent of the sample size, with bounded parameter distributions conditional on width.

Approximation and frequentist estimation

Classical shallow-network rates such as those of Barron [5] concern Fourier-moment classes rather than Hölder balls. For deep ReLU networks, quantitative approximation bounds (Yarotsky, [30]) underpin near-minimax estimation for compositional targets (Schmidt-Hieber, [24]), fully connected architectures (Kohler and Langer, [15]), and Besov and mixed-smoothness classes (Suzuki, [27]).

We use the shallow-network approximation rate k−β/dk^{-\beta/d} for width kk and output-coefficient bounds of Yang and Zhou [29], valid for 0<β<(d+3)/20 < \beta< (d+3)/2; see also Yang and Zhou [28]. Rather than select an architecture for a frequentist procedure, these bounds identify accurate parameter neighbourhoods whose probability we evaluate under one fixed prior. We compare the resulting risk with the classical Hölder minimax lower bound of Stone [26].

Bayesian contraction and architecture selection

Random widths already appear in the shallow-network consistency results of Lee [18]; hierarchical dimension priors are also standard in Bayesian nonparametrics [3]. For deep ReLU networks, Polson and Ročková [23] obtain near-minimax contraction with uniform-slab spike-and-slab priors and random width, but retain other nn-dependent architectural choices. Related approaches use sparse connections [19, 20], hidden-unit selection [17], or variational inference [9, 4, 21].

For dense ordinary posteriors, Kong and Kim [16] cover Gaussian and other network-parameter laws through a local density lower bound on every bounded parameter box, with network dimensions varying with nn. Our condition instead controls neighbourhoods of approximation-compatible parameters, allowing width-dependent bounded supports.

Heavy tails and fractional posteriors

Heavy-tailed series priors adapt to unknown smoothness [1]. For deep ReLU networks, Castillo and Egels [8] obtain near-minimax fractional-posterior contraction with heavy-tailed network parameters and nn-dependent network specifications and coefficient scales. The adaptive shallow-network construction of Agapiou et al. [2], with fixed shifts and random output coefficients, likewise uses nn-dependent quantities.

Polynomial density lower bounds make coefficient magnitude contribute only logarithmically to the negative log-probability of local parameter intervals. Our uniform and Gaussian constructions satisfy the same local-mass criterion. Fractional posteriors isolate this criterion from the global control needed for ordinary posteriors; clipping then recovers squared prediction-risk bounds.

Statistical model and fractional posterior

All functions are assumed measurable. Random objects are defined on a probability space (Ω,A,P)(\Omega,\mathcal{A},\mathbb{P}), with Borel σ\sigma-algebras on their state spaces. Fix d≥1d \ge1 and write X=[0,1]d\mathcal{X} = [0,1]^d. For an unknown regression function f0:X→Rf_0 : \mathcal{X} \to\mathbb{R}, we observe Dn=((Xi,Yi))i=1n\mathcal{D}_n = ((X_i,Y_i))_{i=1}^n from

Yi=f0(Xi)+ξi,i=1,…,n,(1)Y_i = f_0(X_i) + \xi_i,\qquad i = 1,\ldots,n, \tag*{(1)}

where the covariates X,Xi:Ω→XX,X_i : \Omega\to\mathcal{X}, i≥1i \ge1, are independent with law PXP_X and the errors ξi:Ω→R\xi_i : \Omega\to\mathbb{R} are independent standard normal variables, independent of the covariates. The design law is fixed but otherwise unrestricted unless a density condition is stated. Write Ef0\mathbb{E}_{f_0} and Pf0\mathbb{P}_{f_0} for expectation and probability under this data law.

Let C(X)C(\mathcal{X}) be the continuous real-valued functions on X\mathcal{X}, equipped with the supremum norm and its Borel σ\sigma-algebra. For a real vector v=(v1,…,vℓ)v=(v_1,\ldots,v_\ell) with ℓ≥1\ell\ge1 coordinates, write ∥v∥1=∑j=1ℓ∣vj∣\lVert v\rVert_1=\sum_{j=1}^{\ell}|v_j|, ∥v∥2=(∑j=1ℓvj2)1/2\lVert v\rVert_2=(\sum_{j=1}^{\ell}v_j^2)^{1/2} and ∥v∥∞=max⁡1≤j≤ℓ∣vj∣\lVert v\rVert_\infty=\max_{1\le j\le\ell}|v_j|. For a real-valued function ff with domain SS, write ∥f∥∞=sup⁡x∈S∣f(x)∣\lVert f\rVert_\infty=\sup_{x\in S}|f(x)|. For 0<β≤10<\beta\le1 and M>0M>0, define

Hdβ(M)={f∈C(X):∥f∥∞≤M, ∣f(x)−f(y)∣≤M∥x−y∥2β for all x,y∈X}.(2)\mathcal{H}^{\beta}_d(M)=\{f\in C(\mathcal{X}):\lVert f\rVert_\infty\le M,\ |f(x)-f(y)|\le M\lVert x-y\rVert_2^\beta\text{ for all }x,y\in\mathcal{X}\}. \tag*{(2)}

The prediction loss is ∥f−f0∥L2(PX)2=∫X(f−f0)2 dPX\lVert f-f_0\rVert_{L^2(P_X)}^2=\int_{\mathcal{X}}(f-f_0)^2\,dP_X. Given a prior Π\Pi on C(X)C(\mathcal{X}) and a fixed 0<η<10<\eta<1, the fractional posterior is

Πnη(df∣Dn)∝exp⁡{−η2∑i=1n(Yi−f(Xi))2}Π(df).(3)\Pi_n^\eta(df\mid\mathcal{D}_n)\propto\exp\left\{-\frac{\eta}{2}\sum_{i=1}^n(Y_i-f(X_i))^2\right\}\Pi(df). \tag*{(3)}

The positive bounded likelihood gives a finite, nonzero normalizing constant for every data set. Write EX\mathbb{E}_X for expectation under X∼PXX\sim P_X. The one-observation order-η\eta Rényi divergence in this model is

Dη(f,f0)=−11−ηlog⁡EXe−η(1−η)(f(X)−f0(X))2/2.(4)D_{\eta}(f,f_0)=-\frac{1}{1-\eta}\log\mathbb{E}_{X}e^{-\eta(1-\eta)(f(X)-f_0(X))^2/2}. \tag*{(4)}

We write a≲ba\lesssim b if a≤Cba\leq Cb for a fixed positive constant CC, a≳ba\gtrsim b if b≲ab\lesssim a, and a≍ba\asymp b if both inequalities hold. For positive sequences, an≫bna_n\gg b_n means an/bn→∞a_n/b_n\to\infty. The symbols ∨\vee and ∧\wedge denote maximum and minimum. Constants in uniform bounds may depend on dd, β\beta, MM, η\eta and the chosen prior, but not on nn, the approximation accuracy, or f0f_0 within the stated Hölder ball. They are independent of PXP_X unless dependence on design-density bounds is stated.

Where the prior must place mass

Lemma 1 specifies how much local prior mass contraction needs; Proposition 1 identifies where to place it. Both apply before choosing a particular network prior.

How much mass is needed. The following form of the fractional-posterior principle of Bhattacharya et al. [7] converts uniform-approximation mass into Rényi contraction.

Lemma 1 (Prior mass implies fractional contraction). Let Π\Pi be any prior on C(X)C(\mathcal{X}), and let ϵn↓0\epsilon_n\downarrow0 with nϵn2→∞n\epsilon_n^2\to\infty. Suppose that, for some fixed c0>0c_0>0, uniformly over a class of targets,

Π{f:∥f−f0∥∞≤ϵn}≥e−c0nϵn2.\Pi\{f:\|f-f_0\|_\infty\leq\epsilon_n\}\geq e^{-c_0n\epsilon_n^2}.

For 0<η<10<\eta<1 and J>(c0+η)/(1−η)J>(c_0+\eta)/(1-\eta),

sup⁡f0Ef0Πnη(Dη(f,f0)>Jϵn2∣Dn)≤e−nϵn2/8+e−{(1−η)J−c0−η}nϵn2.(5)\sup_{f_0}\mathbb{E}_{f_0}\Pi_n^\eta\left(D_\eta(f,f_0)>J\epsilon_n^2\mid D_n\right)\leq e^{-n\epsilon_n^2/8}+e^{-\{(1-\eta)J-c_0-\eta\}n\epsilon_n^2}. \tag*{(5)}

At ϵn≍(log⁡n/n)β/(2β+d)\epsilon_n\asymp(\log n/n)^{\beta/(2\beta+d)}, nϵn2≍ϵn−d/βlog⁡(1/ϵn)n\epsilon_n^2\asymp\epsilon_n^{-d/\beta}\log(1/\epsilon_n). It therefore suffices that

−log⁡Π{f:∥f−f0∥∞≤ϵ}≲ϵ−d/βlog⁡(1/ϵ)-\log\Pi\{f:\|f-f_0\|_\infty\leq\epsilon\}\lesssim\epsilon^{-d/\beta}\log(1/\epsilon)

for all small ϵ\epsilon, uniformly over each Hdβ(M)\mathcal{H}_d^\beta(M). The lemma is proved in Appendix H.

References

  1. [1]Sergios Agapiou and Ismaël Castillo. Heavy-tailed bayesian nonparametric adaptation. The Annals of Statistics, 52(4):1433–1459, 2024.arxiv.org/abs/2308.04916
  2. [2]Sergios Agapiou, Ismaël Castillo, and Paul Engels. Leveraging tails for adaptation, 2026. arXiv:2606.20480v2.arxiv.org/abs/2606.20480
  3. [3]Julyan Arbel, Ghislaine Gayraud, and Judith Rousseau. Bayesian optimal adaptive estimation using a sieve prior. Scandinavian Journal of Statistics, 40(3):549–570, 2013.
  4. [4]Jincheng Bai, Qifan Song, and Guang Cheng. Efficient variational inference for sparse deep learning with theoretical guarantee. In Advances in Neural Information Processing Systems 33, 2020.arxiv.org/abs/2011.07439
  5. [5]Andrew R. Barron. Universal approximation bounds for superpositions of a sigmoidal function. IEEE Transactions on Information Theory, 39(3):930–945, 1993.DOI
  6. [6]Arved Bartuska, André Gustavo Carlon, Luis Espath, Sebastian Krumscheid, and Raúl Tempone. Multilevel randomized quasi-Monte Carlo estimator for nested integration, 2025. arXiv:2412.07723v2.arxiv.org/abs/2412.07723
  7. [7]Anirban Bhattacharya, Debdeep Pati, and Yun Yang. Bayesian fractional posteriors. The Annals of Statistics, 47(1):39–66, 2019.arxiv.org/abs/1611.01125
  8. [8]Ismaël Castillo and Paul Egels. Posterior and variational inference for deep neural networks with heavy-tailed weights. Journal of Machine Learning Research, 26(122):1–58, 2025.arxiv.org/abs/2406.03369
  9. [9]Badr-Eddine Chérief-Abdellatif. Convergence rates of variational inference in sparse deep learning. In Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 1831–1842, 2020.
  10. [10]Bradley Efron and Robert Tibshirani. The bootstrap method for assessing statistical accuracy. Behaviormetrika, 12(17):1–35, 1985.
  11. [11]Andrew Gelman and Donald B Rubin. Inference from iterative simulation using multiple sequences. Statistical Science, 7(4):457–472, 1992.DOI
  12. [12]Peter J. Green. Reversible jump Markov chain Monte Carlo computation and Bayesian model determination. Biometrika, 82(4):711–732, 1995.DOI
  13. [13]Ajay Jasra, David A. Stephens, and Christopher C. Holmes. Population-based reversible jump Markov chain Monte Carlo. Biometrika, 94(4):787–807, 2007.arxiv.org/abs/0711.0186
  14. [14]Leslie Kish. Weighting for unequal P_i. Journal of Official Statistics, 8(2):183–200, 1992.
  15. [15]Michael Kohler and Sophie Langer. On the rate of convergence of fully connected deep neural network regression estimates. The Annals of Statistics, 49(4):2231–2249, 2021.
  16. [16]Insung Kong and Yongdai Kim. Posterior concentrations of fully-connected Bayesian neural networks with general priors on the weights. Journal of Machine Learning Research, 26(94):1–60, 2025.arxiv.org/abs/2403.14225
  17. [17]Insung Kong, Dongyoon Yang, Jongjin Lee, Ilsang Ohn, Gyuseung Baek, and Yongdai Kim. Masked Bayesian neural networks: Theoretical guarantee and its posterior inference. In International Conference on Machine Learning, volume 202, pages 17462–17491, 2023.arxiv.org/abs/2305.14765
  18. [18]Herbert K. H. Lee. Consistency of posterior distributions for neural networks. Neural Networks, 13(6):629–642, 2000.pubmed.ncbi.nlm.nih.gov/10987516
  19. [19]Kyeongwon Lee and Jaeyong Lee. Asymptotic properties for Bayesian neural network in Besov space. In Advances in Neural Information Processing Systems, 2022.arxiv.org/abs/2206.00241
  20. [20]Kyeongwon Lee, Lizhen Lin, Jaewoo Park, and Seonghyun Jeong. Posterior contraction for sparse neural networks in Besov spaces with intrinsic dimensionality. In Advances in Neural Information Processing Systems, 2025.arxiv.org/abs/2506.19144
  21. [21]Ilsang Ohn and Lizhen Lin. Adaptive variational Bayes: Optimality, computation and applications. The Annals of Statistics, 52(1):335–363, 2024.DOI
  22. [22]Martyn Plummer, Nicky Best, Kate Cowles, and Karen Vines. CODA: Convergence diagnosis and output analysis for MCMC. R News, 6(1):7–11, 2006.
  23. [23]Nicholas G. Polson and Veronika Ročková. Posterior concentration for sparse deep learning. In Advances in Neural Information Processing Systems 31, 2018.arxiv.org/abs/1803.09138
  24. [24]Johannes Schmidt-Hieber. Nonparametric regression using deep neural networks with ReLU activation function. The Annals of Statistics, 48(4):1875–1897, 2020.
  25. [25]L. F. Shampine. Vectorized adaptive quadrature in MATLAB. Journal of Computational and Applied Mathematics, 211(2):131–140, 2008. https://doi.org/10.1016/j.cam.2006.11.021.
  26. [26]Charles J. Stone. Optimal global rates of convergence for nonparametric regression. The Annals of Statistics, 10(4):1040–1053, 1982.DOI
  27. [27]Taiji Suzuki. Adaptivity of deep ReLU network for learning in Besov and mixed smooth Besov spaces: Optimal rate and curse of dimensionality. In International Conference on Learning Representations, 2019.arxiv.org/abs/1810.08033
  28. [28]Yunfei Yang and Ding-Xuan Zhou. Nonparametric regression using over-parameterized shallow ReLU neural networks. Journal of Machine Learning Research, 25(165):1–35, 2024.arxiv.org/abs/2306.08321
  29. [29]Yunfei Yang and Ding-Xuan Zhou. Optimal rates of approximation by shallow ReLU^k neural networks and applications to nonparametric regression. Constructive Approximation, 62(2):329–360, 2025.DOI
  30. [30]Dmitry Yarotsky. Error bounds for approximations with deep ReLU networks. Neural Networks, 94:103–114, 2017.DOI

Paper details

vlnc:W_dvaydgse
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 9

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Mathematics

OpenAI ChatGPT, Codex

View statement
Looking for related papers…
Contents