betaFeedback

WorksMathematics

Recursive moment updates for scalable Bayesian nonparametric regression

Abstract

Bayesian nonparametric regression methods have wide applications in statistics and machine learning, but inference can often be expensive or require approximations. This paper introduces a class of recursive moment function updates which are closed form, interpretable, expedient, and fully quantify Bayesian uncertainty through the martingale posterior. The proposed recursive updates can be used off-the-shelf and encompass challenging scenarios such as nonparametric heteroscedasticity, heavy-tailed or skewed noise, and importantly generalize to the multi-output case with missing outcomes without any significant loss in tractability or speed. We show frequentist consistency under weak assumptions in the aforementioned scenarios, and demonstrate empirically that our proposed method is orders of magnitude faster than competing Bayesian nonparametric approaches, with competitive or even superior prediction and estimation accuracy.

Introduction

Bayesian nonparametric regression has wide applications in areas like machine learning, spatial statistics, and biostatistics. Unfortunately, methods such as dependent mixture models [26, 30] and Gaussian processes (GP, Rasmussen and Williams [ [31]) can have prohibitively expensive computation due to Markov chain Monte Carlo (MCMC) or large matrix inversions. Furthermore, these models have many moving parts, and prior and model elicitation often lacks clear guidance. These issues are further compounded in multi-output regression [3], where the outcome is multivariate.

An alternative MCMC-free approach to Bayesian nonparametric inference for mixture models is based on recursive density updates, pioneered by the predictive recursion method [27] and more recently by Hahn et al. [ [16]. Advances within the predictive Bayesian literature have demonstrated recursive density updates can even be utilized to obtain full posterior uncertainty through the imputation of unseen future variables [13, 7, 12], resulting in the martingale posterior.

Within the single-output regression setting, Fong et al. [ [12], Cappello and Walker [ [9] extend the bivariate copula update of Hahn et al. [ [16] to conditional density estimation, and Lin and Tokdar [ [25] extend predictive recursion to estimate a covariate-dependent mixing density. However, in many cases in regression, we are often primarily interested in the mean or variance function, as opposed to the entire conditional density. While the previously mentioned recursive updates are flexible, they all require numerical integration on a grid if the moment functions are of interest, which is already computationally demanding in the single-output case, and quickly becomes infeasible as the dimension of the outcome grows.

In this paper, we introduce flexible, tractable, and interpretable recursive updates for the covariate-dependent moment functions, with uncertainty quantification through a semiparametric martingale posterior. Our moment updates easily handle heteroscedasticity, heavy-tailed or skewed residuals, and importantly scale very well to the multi-output setting with missing outcomes and covariate-dependent correlations. Our approach achieves competitive or superior statistical performance compared to other recursive approaches and traditional Bayesian nonparametric methods, while being orders of magnitude faster as we do not rely on numerical integration. Hyperparameter tuning is automatic, so our approach can be used off-the-shelf for Bayesian nonparametric regression. Finally, tractability of the moment updates allows thorough study of predictive and frequentist asymptotics.

Methodology for univariate outcomes

In this section, we consider single outcome regression with random covariates to illustrate the main idea, before extending to the multi-output case in § 3. Let Y∈RY \in\mathbb{R} denote the outcome and X∈X⊆RpX \in\mathcal{X} \subseteq\mathbb{R}^{p} denote the covariates where X\mathcal{X} is compact. Let (Yi,Xi)i=1,…,n(Y_i, X_i)_{i=1,\ldots,n} be nn independently and identically distributed (i.i.d.) copies of (Y,X)(Y, X) from an unknown distribution P∗P^{*} with Lebesgue density p∗(y,x)p^{*}(y,x). Furthermore, let pX∗(x)p_{X}^{*}(x) denote the marginal density of XX, and let p∗(y∣x)p^{*}(y \mid x) denote the conditional density of interest which we assume exists for all x∈Xx \in\mathcal{X}.

Recursive bivariate copula updates

We begin by revisiting a recursive approach for conditional density estimation introduced in Fong et al. [ [12] using bivariate copula densities. Whilst not the main focus, our recursive moment updates will be derived from this copula update. The starting point is that the Dirichlet process (DP) mixture is tractable when only a single datum is observed [27]. In the covariate-free DP mixture, Hahn et al. [ [16] derived the predictive density update in terms of a Gaussian copula density, and proposed extending the n=1n=1 rule recursively for n>1n>1.

The same idea can be carried out for the dependent DP mixture model with a Gaussian kernel and base measure. Let pi(y∣x)p_i(y \mid x) and ui=Pi(y∣x)u_i=P_i(y \mid x) denote the conditional probability density function (PDF) and cumulative distribution function (CDF) after ii observations. Fong et al. [ [12] suggests the recursive update

pi(y∣x)=[1−αi(x,Xi)+αi(x,Xi)cρ(ui−1,vi−1)]pi−1(y∣x),(1)p_i(y \mid x) = [1-\alpha_i(x,X_i)+\alpha_i(x,X_i)c_{\rho}(u_{i-1},v_{i-1})]p_{i-1}(y \mid x), \tag*{(1)}

where vi−1=Pi−1(Yi∣Xi)v_{i-1}=P_{i-1}(Y_i \mid X_i), and cρc_{\rho} is the bivariate Gaussian copula density:

cρ(u,v)=ϕ2(Φ−1(u),Φ−1(v);ρ)ϕ(Φ−1(u))ϕ(Φ−1(v))c_{\rho}(u,v)=\frac{\phi_2(\Phi^{-1}(u),\Phi^{-1}(v);\rho)}{\phi(\Phi^{-1}(u))\phi(\Phi^{-1}(v))}

and ϕ2(y1,y2;ρ)\phi_2(y_1,y_2;\rho) is the standard bivariate Gaussian density with correlation ρ\rho, ϕ(y)\phi(y) is the standard Gaussian density, Φ−1\Phi^{-1} is the Gaussian quantile function, ρ∈(0,1)\rho\in(0,1) controls the smoothness of the density, and the covariate-dependent learning rate αi(x,Xi)\alpha_i(x,X_i) determines the information borrowed between covariates.

Covariate-dependent learning rate

We consider a general form of αi(x,x′)\alpha_i(x,x') based on the ‘conditional’ approach of Fong et al. [ [12]. Let s(x,x′)s(x,x') be a similarity function,

s(x,x′)=EΨ[k(x;Ψ)k(x′;Ψ)]EΨ[k(x;Ψ)]EΨ[k(x′;Ψ)](2)s(x,x') = \frac{E_{\Psi}[k(x;\Psi)k(x';\Psi)]}{E_{\Psi}[k(x;\Psi)]E_{\Psi}[k(x';\Psi)]} \tag*{(2)}

where Ψ∼Π\Psi\sim\Pi, the kernel k(⋅;ψ)k(\cdot;\psi) is a PDF for all ψ∈supp⁡(Π)\psi\in\operatorname{supp}(\Pi), and we assume that s(x,x′)s(x,x') is well-defined for all (x,x′)∈X×X(x,x') \in\mathcal{X} \times\mathcal{X}. We then define

αi(x,x′)=αis(x,x′)1−αi+αis(x,x′).\alpha_i(x,x') = \frac{\alpha_i s(x,x')}{1-\alpha_i+\alpha_i s(x,x')}.

where 0<αi<10 < \alpha_i < 1 and αi=O(i−m)\alpha_i = O(i^{-m}) for some 0.5<m≤10.5 < m \le1. In our implementations, we choose the learning rate αi=(2−i−1)(i+1)−0.9\alpha_i = (2-i^{-1})(i+1)^{-0.9}, which works well in practice. Further discussions on the role of the learning rate can be found in Fortini and Petrone [2020], Agnoletto et al. [2026] and Fong et al. [2023, Appendix E].

Importantly, we have αi(x,x′)→0\alpha_i(x,x') \to0 if s(x,x′)→0s(x,x') \to0, so the datum (Yi,Xi)(Y_i,X_i) is less relevant for the conditional density at xx if s(x,Xi)s(x,X_i) is small, and αi(x,x′)→0\alpha_i(x,x') \to0 as i→∞i \to\infty. Like Antoniano-Villalobos et al. [2014], our model elicitation involves a kernel k(x;ψ)k(x;\psi) with a prior for ψ\psi. We now illustrate an example of s(x,x′)s(x,x').

Example 1. Consider the Gaussian kernel k(x;ψ)=N(x;ψ,1−τ−1)k(x;\psi) = \mathcal{N}(x;\psi,1-\tau^{-1}) and Π=N(0,τ−1)\Pi= \mathcal{N}(0,\tau^{-1}) for a fixed τ>1\tau> 1. In this case, we recover s(x,x′)=cρx{P0(x),P0(x′)}s(x,x') = c_{\rho_x}\{P_0(x),P_0(x')\} where P0=ΦP_0 = \Phi is the normal CDF and ρx=τ−1\rho_x = \tau^{-1}, as considered in Fong et al. [ [12].

More generally, a closed form s(x,x′)s(x,x') can be obtained if k(⋅;ψ)k(\cdot;\psi) is an exponential family with conjugate prior Π\Pi; we outline more examples in Appendix § 13.§ 1. Multivariate covariates can be handled by defining k(x;ψ)=∏j=1pk(x(j);ψ(j))k(x;\psi) = \prod_{j=1}^{p} k(x^{(j)};\psi^{(j)}), and hyperparameters of s(x,x′)s(x,x'), e.g. ρx\rho_x for the Gaussian copula, can be tuned in an empirical Bayesian manner using the prequential log-likelihood. In the multivariate case, the bandwidth tuning procedure can also carry out variable selection.

The main property we assert on s(x,x′)s(x,x') is that the family {k(x;ψ):ψ∈supp⁡(Π)}\{k(x;\psi):\psi\in\operatorname{supp}(\Pi)\} is complete [22] as defined in Definition 1. Completeness will be necessary for consistency of the recursive moment updates, and is satisfied by many examples including Example 1. Intuitively, this means that s(x,x′)s(x,x') does not have any ‘gaps’ which may impede consistency.

Definition 1. The family {k(x;ψ):ψ∈supp⁡(Π)}\{k(x;\psi):\psi\in\operatorname{supp}(\Pi)\} is complete if for every measurable function f:Rp→Rf:\mathbb{R}^p \to\mathbb{R}, the condition ∫f(x)k(x;ψ) dx=0\int f(x)k(x;\psi)\,dx = 0 for all ψ∈supp⁡(Π)\psi\in\operatorname{supp}(\Pi) implies f(x)=0f(x) = 0 for k(⋅;ψ)k(\cdot;\psi)-almost all xx for every ψ∈supp⁡(Π)\psi\in\operatorname{supp}(\Pi).

Recursive moment updates

In nonparametric regression, the mean function E[Y∣X=x]E[Y \mid X = x] is often of primary interest for prediction, or one may be interested in assessing heteroscedasticity through the conditional variance Var⁡(Y∣X=x)\operatorname{Var}(Y \mid X = x). A significant drawback of the nonparametric copula approach is that the moment functions must be computed numerically on a grid from the conditional density estimates p(y∣x)p(y \mid x), which can be very computationally expensive, and quickly becomes completely infeasible in the multi-output setting in the next Section. In addition, the selection of the yy-grid is non-trivial, especially under strong heteroscedasticity or heavy tails where a very high resolution is needed to cover the range of yy values. We highlight that a grid over the mixing parameter is also needed for the predictive recursion approach of Lin and Tokdar [2026], which will scale exponentially poorly with outcome dimensionality.

To that end, we introduce a semiparametric recursive update, which can be viewed as a projection of the nonparametric copula update onto a parametric family. Instead of a recursive update for the conditional density, we restrict pi(y∣x)p_i(y \mid x) to a specific parametric family, and derive tractable recursive updates for the moment functions directly. The moment updates are simple, flexible and interpretable, and are significantly more expedient than the copula update and other Bayesian competitors. Importantly, uncertainty quantification of the moment functions can still be obtained through the martingale posterior, and § 4.2 will illustrate that consistency can still be attained even if the family is misspecified.

Gaussian family

The core idea is to distill the full nonparametric copula update into its first two moments, then project the updated density onto the Gaussian family through moment matching. Let us restrict pi−1(y∣x)p_{i-1}(y \mid x) to be Gaussian, that is

pi−1(y∣x)=N(y;μi−1(x),σi−12(x)).p_{i-1}(y \mid x) = \mathcal{N}\left(y; \mu_{i-1}(x), \sigma_{i-1}^{2}(x)\right).

Whilst applying (1) to the above yields a non-Gaussian pi(y∣x)p_i(y \mid x), its updated mean and variance can be computed exactly from conjugacy of the predictive CDF with the Gaussian copula. We then use these updated moments to define the Gaussian predictive density for the next step. To derive the moment updates, note that Pi−1(y∣x)=Φ{(y−μi−1(x))/σi−1(x)}P_{i-1}(y \mid x) = \Phi\{(y - \mu_{i-1}(x))/\sigma_{i-1}(x)\}, and Gaussian conditioning then gives

cρ(ui−1,vi−1)pi−1(y∣x)=N(y;μi−1(x)+ρσi−1(x)Zi,(1−ρ2)σi−12(x)),(3)c_{\rho}(u_{i-1}, v_{i-1})p_{i-1}(y \mid x) = \mathcal{N}\left(y; \mu_{i-1}(x) + \rho\sigma_{i-1}(x)Z_i, (1-\rho^2)\sigma_{i-1}^{2}(x)\right), \tag*{(3)}

where ZiZ_i is the standardized residual of the form

References

  1. [2]M. Altamirano, F.-X. Briol, and J. Knoblauch. Robust and conjugate Gaussian process regression. In ICML, 2024.arxiv.org/abs/2311.00463
  2. [3]M. A. Alvarez and N. D. Lawrence. Computationally efficient convolved multiple output Gaussian processes. The Journal of Machine Learning Research, 12:1459–1500, 2011.
  3. [4]I. Antoniano-Villalobos, S. Wade, and S. G. Walker. A Bayesian nonparametric regression model with normalized weights: A study of hippocampal atrophy in Alzheimer’s disease. Journal of the American Statistical Association, 109(506):477–490, 2014.
  4. [5]P. Berti, L. Pratelli, and P. Rigo. Limit theorems for a class of identically distributed random variables. The Annals of Probability, 32(3):2029 – 2052, 2004.DOI
  5. [7]P. Berti, E. Dreassi, F. Leisen, L. Pratelli, and P. Rigo. Bayesian predictive inference without a prior. Statistica Sinica, 33(4):2405–2429, 2023.arxiv.org/abs/2104.11643
  6. [8]G. E. P. Box and D. R. Cox. An analysis of transformations. Journal of the Royal Statistical Society Series B: Statistical Methodology, 26(2):211–243, 1964.DOI
  7. [9]L. Cappello and S. G. Walker. Recursive nonparametric predictive for a discrete regression model. Computational Statistics & Data Analysis, page 108275, 2025.DOI
  8. [10]Z. Drmač, M. Omladič, and K. Veselić. On the perturbation of the Cholesky factorization. SIAM Journal on Matrix Analysis and Applications, 15(4):1319–1332, 1994.DOI
  9. [11]E. Fong and A. Yiu. Bayesian quantile estimation and regression with martingale posteriors. Journal of the Royal Statistical Society Series B: Statistical Methodology, 88(4):1130–1159, 2026.arxiv.org/abs/2406.03358
  10. [12]E. Fong, C. Holmes, and S. G. Walker. Martingale posterior distributions. Journal of the Royal Statistical Society Series B: Statistical Methodology, 85(5):1357–1391, 2023.arxiv.org/abs/2103.15671
  11. [13]S. Fortini and S. Petrone. Quasi-Bayes properties of a procedure for sequential learning in mixture models. Journal of the Royal Statistical Society Series B: Statistical Methodology, 82(4):1087–1114, 2020.DOI
  12. [14]S. Garelli, F. Leisen, L. Pratelli, and P. Rigo. Asymptotics of predictive distributions driven by sample means and variances. Journal of Applied Probability, 2026.arxiv.org/abs/2403.16828
  13. [15]J. Gentilli. Estimating dew point from minimum temperature. Bulletin of the American Meteorological Society, 36(3):128–131, 1955.DOI
  14. [16]P. R. Hahn, R. Martin, and S. G. Walker. On recursive Bayesian predictive distributions. Journal of the American Statistical Association, 113(523):1085–1093, 2018.DOI
  15. [18]R. Hiptmair and V. Gradinaru. Numerical methods for computational science and engineering. 2009. URL https://people.math.ethz.ch/~hiptmair/tmp/NumCSE09.pdf.
  16. [19]N. L. Johnson. Systems of frequency curves generated by methods of translation. Biometrika, 36(1/2):149–176, 1949.pubmed.ncbi.nlm.nih.gov/18132090
  17. [20]P. Jylänki, J. Vanhaatalo, and A. Vehtari. Robust Gaussian process regression with a Student-t likelihood. Journal of Machine Learning Research, 12:3227–3257, 2011.
  18. [21]N. V. Kelso and T. Patterson. Introducing Natural Earth Data - naturalearthdata.com. Geographia Technica, 5:82–89, 2010.
  19. [22]E. L. Lehmann and J. P. Romano. Testing Statistical Hypotheses. Springer Science & Business Media, 2006.
  20. [23]Y. Lei, L. Shi, and Z.-C. Guo. Convergence of unregularized online learning algorithms. Journal of Machine Learning Research, 18(171):1–33, 2018.arxiv.org/abs/1708.02939
  21. [24]G. Leoni. A first course in Sobolev spaces. American Mathematical Society, 2017.
  22. [25]J. Lin and S. Tokdar. Fast semiparametric density regression with weight-localized predictive recursion. arXiv preprint arXiv:2605.02146, 2026.arxiv.org/abs/2605.02146
  23. [26]S. N. MacEachern. Dependent Dirichlet processes. Technical Report, Department of Statistics, The Ohio State University, 2000.
  24. [27]M. A. Newton, F. A. Quintana, and Y. Zhang. Nonparametric Bayes methods using predictive updating. In Practical Nonparametric and Semiparametric Bayesian Statistics, pages 45–61. Springer, 1998.DOI
  25. [28]K. B. Petersen and M. S. Pedersen. The matrix cookbook, Nov. 2012. URL http://www2.compute.dtu.dk/pubdb/pubs/3274-full.html. Version 20121115.DOI
  26. [29]G. Pisier. Martingales in Banach Spaces, volume 155. Cambridge University Press, 2016.DOI
  27. [30]F. A. Quintana, P. Müller, A. Jara, and S. N. MacEachern. The dependent Dirichlet process and related models. Statistical Science, 37(1):24–41, 2022.arxiv.org/abs/2007.06129
  28. [31]C. E. Rasmussen and C. K. I. Williams. Gaussian Processes for Machine Learning. The MIT Press, 2006.DOI
  29. [32]J. Requima, W. Tebbutt, W. Bruinsma, and R. E. Turner. The Gaussian process autoregressive regression model (GPAR). In The 22nd International Conference on Artificial Intelligence and Statistics, pages 1860–1869. PMLR, 2019.
  30. [33]H. Robbins and D. Siegmund. A convergence theorem for non negative almost super-martingales and some applications. In Optimizing Methods in Statistics, pages 233–257. Elsevier, 1971.
  31. [34]J. Rooijakkers, L. Rønneberg, F.-X. Briol, J. Knoblauch, and M. Altamirano. Multi-output robust and conjugate Gaussian processes. arXiv preprint arXiv:2510.26401, 2025.arxiv.org/abs/2510.26401
  32. [35]A. Smith, N. Lott, and R. Vose. The integrated surface database: Recent developments and partnerships. Bulletin of the American Meteorological Society, 92(6):704–708, 2011.DOI
  33. [37]G. W. Stewart. On the perturbation of LU, Cholesky, and QR factorizations. SIAM Journal on Matrix Analysis and Applications, 14(4):1141–1145, 1993.DOI
  34. [39]S. T. Tokdar, R. Martin, and J. K. Ghosh. Consistency of a recursive estimate of mixing distributions. The Annals of Statistics, pages 2502–2522, 2009.
  35. [40]S. Wade and V. Inácio. Bayesian dependent mixture models: A predictive comparison and survey. Statistical Science, 40(1):81–108, 2025.arxiv.org/abs/2307.16298
  36. [41]A. G. Wilson, D. A. Knowles, and Z. Ghahramani. Gaussian process regression networks. In Proceedings of the 29th International Conference on Machine Learning, pages 1139–1146, 2012.arxiv.org/abs/1110.4411
  37. [42]Y. Ying and M. Pontil. Online gradient descent learning algorithms. Foundations of Computational Mathematics, 8(5):561–596, 2008.DOI

Paper details

vlnc:W_865mkns8
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 1

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Mathematics

ChatGPT Astra

View statement
Looking for related papers…
Contents