betaFeedback

WorksMathematics

Martingale posteriors for flexible regression via recursive least squares

Abstract

Predictive Bayes quantifies uncertainty by specifying a sequence of one-step-ahead predictive distributions, inducing a martingale posterior without an explicit prior. Existing parametric constructions rely on stochastic-gradient updates and have seen limited development for flexible regression. We propose a predictive Bayes framework based on recursive least squares, whose martingale posterior has an explicit representation: given future covariates, it is the law of the posterior mean plus a weighted series of independent draws from the standardized predictive distribution. The posterior mean is a penalized least squares estimator, available in closed form and invariant to data ordering, whereas the predictive distribution, for example heavy tailed, is chosen separately without changing the estimator or algorithm. We derive the exact posterior variance under a known design and an asymptotically Gaussian approximation, and obtain the martingale posterior in closed form for Gaussian predictive distributions. Posterior draws are independent by construction, with a known truncation error, and require no Markov chain Monte Carlo. The construction extends to additive, mixed and Gaussian process models. In simulations and an analysis of NHANES triglyceride data, the method attains accuracy comparable to that of a fully Bayesian model with the same response distribution at a fraction of its computational cost.

Introduction

Flexible regression is central to modern statistical practice, yet Bayesian analyses in this setting can be awkward when the scientifically appropriate sampling distribution is not paired with a convenient prior or computational scheme. Heavy tails and rich mean structures are readily motivated in applications, but can complicate posterior computation. Laplace approximations [34] and variational Bayes [10] avoid Markov chain Monte Carlo (MCMC) in such settings, but their accuracy depends on the approximating family, and variational methods tend to understate posterior uncertainty.

Generalized Bayesian approaches, which replace the likelihood with moment conditions or a loss [11, 8], typically rely on MCMC, and valid uncertainty quantification hinges on calibrating a weighting matrix or learning rate [8, 2]. Predictive Bayes offers a different route to uncertainty quantification: rather than starting from a prior, it specifies a sequence of one-step-ahead predictive distributions. When the corresponding parameter estimates form a martingale, their random limit has an induced martingale posterior distribution. This construction retains a Bayesian interpretation while making the sequential predictive rule, rather than an explicit prior, the primary modeling object [23, 28, 19].

The regression setting raises two obstacles for existing predictive Bayes methods. First, a useful regression procedure should allow the predictive distribution to reflect the response distribution while retaining a stable and interpretable estimator of the conditional mean. Second, its finite-sample estimator should not depend on an arbitrary ordering of the observed data. Much recent work develops predictive constructions for density estimation or discrete data [23, 3]. Parametric martingale posteriors have mainly been built from stochastic-gradient or online-gradient updates [39, 28, 20, 24]. These updates are valuable general tools, but they require delicate learning-rate choices and typically yield order-dependent finite-sample estimates. They also do not directly provide a flexible regression framework with transparent posterior uncertainty.

We address these issues by constructing martingale posteriors for regression from recursive least squares (RLS). The RLS recursion is a classical and computationally inexpensive estimator, but here it is used as the update of a predictive Bayesian model. The resulting posterior mean is invariant to permutations of the observed data and coincides with a ridge-type regression estimator. Crucially, the plug-in predictive distribution can be chosen separately from this update: it may be Gaussian or heavy-tailed, for example, while preserving the same conditional mean update, so that changing the response distribution requires neither a new estimator nor a new sampler. Posterior uncertainty is obtained by forward predictive resampling, which produces independent draws, with no convergence diagnostics and a truncation error known in closed form. Hyperparameters, such as smoothing parameters, can be selected by maximizing an order-invariant leave-one-out prequential log-likelihood. In our simulations and application, the resulting posterior mean has an accuracy comparable to that of a fully Bayesian model with the same response distribution, at a fraction of its computational cost.

Our main contributions are as follows. We show that the RLS coefficient sequence is a martingale and converges to a random limit, establishing a martingale posterior with an order-invariant finite-sample posterior mean and asymptotic exchangeability of the induced sequence. Conditional on a known covariate design, we derive the exact posterior variance and establish an asymptotically Gaussian approximation that uses only observed covariates. Under Gaussian predictive distributions, we obtain the exact martingale posterior in closed form, and we show that, with a suitable predictive variance, the construction coincides with the conjugate Bayesian linear model, for any covariate distribution. Finally, we show how the construction can be used for additive modeling and to accommodate correlated coefficients, as in mixed effects and Gaussian process regressions.

Predictive Bayes and martingale posteriors

Predictive Bayes starts from a sequential specification rather than an explicit prior. Let (Yn)n≥1(Y_n)_{n\ge1} be a sequence of random variables and write

Y1∼P0,Yn+1∣y1:n∼Pn,n≥1,Y_1 \sim P_0,\qquad Y_{n+1}\mid y_{1:n}\sim P_n,\qquad n\ge1,

where PnP_n is the one-step-ahead predictive distribution after observing y1:ny_{1:n}. By the Ionescu-Tulcea theorem, this sequence of predictive distributions defines a unique joint probability law P\mathbb{P} for (Yn)n≥1(Y_n)_{n\ge1} [7]. By de Finetti’s theorem, if (Yn)n≥1(Y_n)_{n\ge1} is exchangeable, then it admits a representation in terms of a statistical model P~\widetilde{P} and a prior distribution Π\Pi such that Yn∣P~∼iidP~Y_n\mid\widetilde{P}\overset{\mathrm{iid}}{\sim}\widetilde{P}, for any n≥1n\ge1, with P~∼Π\widetilde{P}\sim\Pi, and moreover the sequence of predictive distributions (Pn)n≥1(P_n)_{n\ge1} converges weakly to P~\widetilde{P}, P\mathbb{P}-a.s. Except for well-known cases, constructing sequences of predictive distributions that ensure exchangeability is challenging in practice. A more tractable goal is asymptotic exchangeability: if PnP_n converges weakly to a random distribution P~\widetilde{P}, P\mathbb{P}-a.s., then (Yn+1,Yn+2,…)→d(Z1,Z2,…)(Y_{n+1},Y_{n+2},\ldots)\xrightarrow{d}(Z_1,Z_2,\ldots) as n→∞n\to\infty, where (Z1,Z2,…)(Z_1,Z_2,\ldots) is an exchangeable sequence with directing random probability measure P~\widetilde{P}, and →d\xrightarrow{d} denotes convergence in distribution [4, 22, 24]. Thus, a convergent predictive rule induces a well-defined distribution for the random limit P~\widetilde{P}.

For a parametric family {P(⋅∣θ):θ∈Θ}\{P(\cdot\mid\theta):\theta\in\Theta\}, a convenient construction specifies a sequence of estimates θn=θ(y1:n)\theta_n=\theta(y_{1:n}) and uses the plug-in predictive distribution Pn(⋅)=P(⋅∣θn)P_n(\cdot)=P(\cdot\mid\theta_n). Specifically, a proper Bayesian model is built by sequentially alternating

θn=θ(y1:n)andYn+1∼Pn(⋅)=P(⋅∣θn),(1)\theta_n=\theta(y_{1:n})\qquad\text{and}\qquad Y_{n+1}\sim P_n(\cdot)=P(\cdot\mid\theta_n), \tag*{(1)}

for n≥1n\ge1. If (θn)n≥1(\theta_n)_{n\ge1} is an L1L_1-bounded martingale, namely

E(θn+1∣y1:n)=θn,n≥1,andsup⁡nE(∣θn∣)<∞,\mathbb{E}(\theta_{n+1}\mid y_{1:n})=\theta_n,\qquad n\ge1,\qquad\text{and}\qquad\sup_n\mathbb{E}(|\theta_n|)<\infty,

then by Doob’s martingale convergence theorem, there exists a random θ~∈Θ\widetilde{\theta}\in\Theta such that θn→θ~\theta_n\to\widetilde{\theta} P\mathbb{P}-a.s. as n→∞n\to\infty [16]. When P(⋅∣θ)P(\cdot\mid\theta) is continuous in θ\theta, the predictive distributions converge to P(⋅∣θ~)P(\cdot\mid\widetilde{\theta}) and the conditional law of θ~\widetilde{\theta} given y1:ny_{1:n} is called the martingale posterior [39, 28]. The posterior is generally not available analytically. It can, however, be sampled by predictive resampling: starting from PnP_n, repeatedly sample future observations and update the plug-in predictive distributions as in (1), obtaining θN\theta_N for N≫nN \gg n. Independent forward paths yield independent approximate draws from the martingale posterior [23, 19].

Existing parametric methods rely on predictive updates that resemble stochastic gradient descent [28, 20, 24, 37]. Because the resulting θn\theta_n typically depends on the order of the observed data and requires careful tuning of the learning rate, most of these methods use the predictive construction only for predictive resampling and compute the initial θn\theta_n separately through a different scheme. In the next section, we instead use recursive least squares as the predictive update. This yields a posterior mean that is invariant to the ordering of the data, giving an effective method for flexible regression.

Predictive linear regression via recursive least squares

A predictive regression model

We now construct a martingale posterior for the regression coefficient β∈Rp\beta\in\mathbb{R}^p. Let Xn∈RpX_n \in\mathbb{R}^p and Yn∈RY_n \in\mathbb{R} denote the covariates and response, with observed values xnx_n and yny_n. We take XnX_n to be iid from an unknown distribution PXP_X independent of past responses. At the end of this section, we also introduce the case of fixed covariates, which is used in § 3.2 and 3.3. All proofs are given in Appendix § 11.

Assumption 1. Conditional on the covariates, the predictive distribution belongs to the location-scale family Pβ={P(⋅∣x,β,σ):x∈Rp,β∈Rp,σ>0}\mathcal{P}_{\beta} = \{P(\cdot\mid x, \beta, \sigma) : x \in\mathbb{R}^p, \beta\in\mathbb{R}^p, \sigma> 0\}, where P(⋅∣x,β,σ)P(\cdot\mid x, \beta, \sigma) is the law of xTβ+σξx^{\mathsf{T}}\beta+ \sigma\xi and ξ∼Pξ\xi\sim P_{\xi}, a fixed distribution on R\mathbb{R} with E(ξ)=0\mathbb{E}(\xi) = 0 and E(ξ2)=1\mathbb{E}(\xi^2) = 1.

Hence, P(⋅∣x,β,σ)P(\cdot\mid x, \beta, \sigma) has mean xTβx^{\mathsf{T}}\beta and variance σ2\sigma^2. A Gaussian PξP_{\xi} gives Gaussian predictive distributions, whereas, for example, a standardized Student-tt PξP_{\xi} gives heavy-tailed ones. Starting from β0\beta_0 and a positive-definite matrix V0V_0, the responses are generated sequentially as

Y1∣x1∼P0(⋅),andYn+1∣y1:n,x1:n+1∼Pn(⋅),(2)Y_1 \mid x_1 \sim P_0(\cdot), \qquad\text{and} \qquad Y_{n+1} \mid y_{1:n}, x_{1:n+1} \sim P_n(\cdot), \tag*{(2)}

for n≥1n \ge1, where Pn(⋅)=P(⋅∣xn+1,βn,σn+1)∈PβP_n(\cdot) = P(\cdot\mid x_{n+1}, \beta_n, \sigma_{n+1}) \in\mathcal{P}_{\beta} for n≥0n \ge0, βn\beta_n denotes an estimator of β\beta based on the first nn observations (x1:n,y1:n)(x_{1:n}, y_{1:n}), and σn+1\sigma_{n+1} controls the scale of the next predictive distribution. Together with Xn+1∼PXX_{n+1} \sim P_X independently of (X1:n,Y1:n)(X_{1:n}, Y_{1:n}), Equation (2) defines the joint law P\mathbb{P} of (Xn,Yn)n≥1(X_n, Y_n)_{n \ge1}, under which (X1:n,Y1:n)(X_{1:n}, Y_{1:n}) has distribution ∏i=1nPX(dxi)Pi−1(dyi)\prod_{i=1}^{n} P_X(\mathrm{d}x_i) P_{i-1}(\mathrm{d}y_i) for every n≥1n \ge1. The key modeling decision is the update of βn\beta_n. We use recursive least squares:

βn=βn−1+anVn−1xn(yn−xnTβn−1),(3)\beta_n = \beta_{n-1} + a_n V_{n-1}x_n(y_n - x_n^{\mathsf{T}}\beta_{n-1}), \tag*{(3)}

where an=(1+xnTVn−1xn)−1a_n = (1 + \mathbf{x}_n^{\mathsf{T}}V_{n-1}\mathbf{x}_n)^{-1} and VnV_n is a p×pp \times p matrix updated as

Vn=Vn−1−anVn−1xnxnTVn−1.(4)V_n = V_{n-1} - a_n V_{n-1}\mathbf{x}_n\mathbf{x}_n^{\mathsf{T}}V_{n-1}. \tag*{(4)}

The update is the classical recursive least-squares estimator, written through the Sherman–Morrison formula to avoid repeated matrix inversions [32]. Its role here is different from its usual role in online estimation: it is the predictive update that defines the posterior mean and, as the next result shows, induces a martingale posterior.

References

  1. [1]Adiels, M., J. Borén, M. J. Caslake, P. Stewart, A. Soro, J. Westerbacka, B. Wennberg, S.-O. Olofsson, C. Packard, and M.-R. Taskinen (2005). Overproduction of VLDL1 driven by hyperglycemia is a dominant feature of diabetic dyslipidemia. Arteriosclerosis, Thrombosis, and Vascular Biology 25(8), 1697–1703.pubmed.ncbi.nlm.nih.gov/15947244
  2. [2]Agnoletto, D., T. Rigon, and D. B. Dunson (2025). Bayesian inference for generalized linear models via quasi-posteriors. Biometrika 112(2), asaf022.
  3. [3]Agnoletto, D., T. Rigon, and D. B. Dunson (2026). Nonparametric predictive inference for discrete data via Metropolis-adjusted Dirichlet sequences. Journal of the American Statistical Association, 1–14.
  4. [4]Aldous, D. J. (1985). Exchangeability and related topics. In École d’Été de Probabilités de Saint-Flour XIII – 1983, Volume 1117 of Lecture Notes in Mathematics, pp. 1–198. Springer.DOI
  5. [5]Azzalini, A. and A. Capitanio (2003). Distributions generated by perturbation of symmetry with emphasis on a multivariate skew t-distribution. Journal of the Royal Statistical Society Series B: Statistical Methodology 65(2), 367–389.
  6. [6]Barter, P. J., H. B. Brewer Jr, M. J. Chapman, C. H. Hennekens, D. J. Rader, and A. R. Tall (2003). Cholesteryl ester transfer protein: a novel target for raising hdl and inhibiting atherosclerosis. Arteriosclerosis, Thrombosis, and Vascular Biology 23(2), 160–167.pubmed.ncbi.nlm.nih.gov/12588754
  7. [7]Berti, P., E. Dreassi, F. Leisen, L. Pratelli, and P. Rigo (2025). A probabilistic view on predictive constructions for Bayesian learning. Statistical Science 40(1), 25–39.
  8. [8]Bissiri, P. G., C. C. Holmes, and S. G. Walker (2016). A general framework for updating belief distributions. Journal of the Royal Statistical Society Series B: Statistical Methodology 78(5), 1103–1130.
  9. [9]Blackwell, D. and L. Dubins (1962). Merging of opinions with increasing information. The Annals of Mathematical Statistics 33(3), 882–886.
  10. [10]Blei, D. M., A. Kucukelbir, and J. D. McAuliffe (2017). Variational inference: A review for statisticians. Journal of the American Statistical Association 112(518), 859–877.
  11. [11]Chernozhukov, V. and H. Hong (2003). An MCMC approach to classical estimation. Journal of Econometrics 115(2), 293–346.DOI
  12. [12]Chipman, H. A., E. I. George, and R. E. McCulloch (2010). BART: Bayesian additive regression trees. The Annals of Applied Statistics 4(1), 266–298.
  13. [13]Çınlar, E. (2011). Probability and Stochastics, Volume 261 of Graduate Texts in Mathematics. Springer.DOI
  14. [14]D’Aristotile, A., P. Diaconis, and D. Freedman (1988). On merging of probabilities. Sankhyā: The Indian Journal of Statistics, Series A 50(3), 363–380.
  15. [15]Dawid, A. P. (1984). Present position and potential developments: some personal views. Statistical theory: the prequential approach. Journal of the Royal Statistical Society: Series A (General) 147(2), 278–290.DOI
  16. [16]Doob, J. L. (1953). Stochastic Processes. Wiley.
  17. [17]Eilers, P. H. and B. D. Marx (1996). Flexible smoothing with B-splines and penalties. Statistical Science 11(2), 89–121.
  18. [18]Fong, E. and C. C. Holmes (2020). On the marginal likelihood and cross-validation. Biometrika 107(2), 489–496.arxiv.org/abs/1905.08737
  19. [19]Fong, E., C. C. Holmes, and S. G. Walker (2023). Martingale posterior distributions. Journal of the Royal Statistical Society Series B: Statistical Methodology 85(5), 1357–1391.arxiv.org/abs/2103.15671
  20. [20]Fong, E. and A. Yiu (2026). Asymptotics for a class of parametric martingale posteriors. Biometrika 113(2), asag007.arxiv.org/abs/2410.17692
  21. [21]Fortini, S., K. Ng, S. Petrone, J. Rousseau, and S. Wei (2026). A principled framework for uncertainty decomposition in TabPFN. arXiv preprint arXiv:2602.04596.
  22. [22]Fortini, S. and S. Petrone (2012). Predictive construction of priors in Bayesian nonparametrics. Brazilian Journal of Probability and Statistics 26(4), 423–449.DOI
  23. [23]Fortini, S. and S. Petrone (2020). Quasi-Bayes properties of a procedure for sequential learning in mixture models. Journal of the Royal Statistical Society Series B: Statistical Methodology 82(4), 1087–1114.DOI
  24. [24]Fortini, S. and S. Petrone (2025). Exchangeability, prediction and predictive modeling in Bayesian statistics. Statistical Science 40(1), 40–67.arxiv.org/abs/2402.10126
  25. [25]Hastie, T. and R. Tibshirani (1986). Generalized additive models. Statistical Science 1(3), 297–310.
  26. [26]Hegele, R. A., H. N. Ginsberg, M. J. Chapman, B. G. Nordestgaard, J. A. Kuivenhoven, M. Averna, J. Borén, E. Bruckert, A. L. Catapano, O. S. Descamps, et al. (2014). The polygenic nature of hypertriglyceridaemia: implications for definition, diagnosis, and management. The Lancet Diabetes & Endocrinology 2(8), 655–666.DOI
  27. [27]Hokanson, J. E. and M. A. Austin (1996). Plasma triglyceride level is a risk factor for cardiovascular disease independent of high-density lipoprotein cholesterol level: a metaanalysis of population-based prospective studies. European Journal of Cardiovascular Prevention & Rehabilitation 3(2), 213–219.DOI
  28. [28]Holmes, C. C. and S. G. Walker (2023). Statistical inference with exchangeability and martingales. Philosophical Transactions of the Royal Society A 381 (2247), 20220143.DOI
  29. [29]Magnus, J. R. and H. Neudecker (2019). Matrix Differential Calculus with Applications in Statistics and Econometrics (Third ed.). Chichester: John Wiley & Sons.
  30. [31]Patsch, J., S. Prasad, A. Gotto, G. Bengtsson-Olivecrona, et al. (1984). Postprandial lipemia. A key for the conversion of high density lipoprotein₂ into high density lipoprotein₃ by hepatic lipase. The Journal of Clinical Investigation 74(6), 2017–2023.pubmed.ncbi.nlm.nih.gov/6439739
  31. [32]Plackett, R. L. (1950). Some theorems in least squares. Biometrika 37(1/2), 149–157.pubmed.ncbi.nlm.nih.gov/15420260
  32. [33]Reaven, G. M. (1988). Role of insulin resistance in human disease. Diabetes 37(12), 1595–1607.
  33. [34]Rue, H., S. Martino, and N. Chopin (2009). Approximate Bayesian inference for latent Gaussian models by using integrated nested Laplace approximations. Journal of the Royal Statistical Society Series B: Statistical Methodology 71(2), 319–392.DOI
  34. [35]Sarwar, N., J. Danesh, G. Eiriksdottir, G. Sigurdsson, N. Wareham, S. Bingham, S. M. Boekholdt, K.-T. Khaw, and V. Gudnason (2007). Triglycerides and the risk of coronary heart disease: 10 158 incident cases among 262 525 participants in 29 Western prospective studies. Circulation 115(4), 450–458.pubmed.ncbi.nlm.nih.gov/17190864
  35. [36]Sparks, J. D., C. E. Sparks, and K. Adeli (2012). Selective hepatic insulin resistance, VLDL over-production, and hypertriglyceridemia. Arteriosclerosis, Thrombosis, and Vascular Biology 32(9), 2104–2112.DOI
  36. [37]Sun, W. and E. Fong (2026). On the design distribution for predictive Bayesian regression. arXiv preprint arXiv:2606.14544.arxiv.org/abs/2606.14544
  37. [38]Van der Vaart, A. W. (2000). Asymptotic Statistics, Volume 3 of Cambridge Series in Statistical and Probabilistic Mathematics. Cambridge University Press.
  38. [39]Walker, S. G. (2022). A new look at Bayesian uncertainty. In Handbook of Statistics, Volume 47, pp. 83–101. Elsevier.DOI
  39. [40]Williams, D. (1991). Probability with Martingales. Cambridge University Press.DOI
  40. [41]Wood, S. N. (2011). Fast stable restricted maximum likelihood and marginal likelihood estimation of semiparametric generalized linear models. Journal of the Royal Statistical Society Series B: Statistical Methodology 73(1), 3–36.DOI
  41. [42]Wood, S. N. (2025). Generalized additive models. Annual Review of Statistics and Its Application 12(1), 497–526.

Paper details

vlnc:W_29jt2uy9
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Mathematics

Claude (Anthropic)

View statement
Looking for related papers…
Contents