Introduction
Flexible regression is central to modern statistical practice, yet Bayesian analyses in this setting can be awkward when the scientifically appropriate sampling distribution is not paired with a convenient prior or computational scheme. Heavy tails and rich mean structures are readily motivated in applications, but can complicate posterior computation. Laplace approximations [34 ] and variational Bayes [10 ] avoid Markov chain Monte Carlo (MCMC) in such settings, but their accuracy depends on the approximating family, and variational methods tend to understate posterior uncertainty.
Generalized Bayesian approaches, which replace the likelihood with moment conditions or a loss [11 , 8 ] , typically rely on MCMC, and valid uncertainty quantification hinges on calibrating a weighting matrix or learning rate [8 , 2 ] . Predictive Bayes offers a different route to uncertainty quantification: rather than starting from a prior, it specifies a sequence of one-step-ahead predictive distributions. When the corresponding parameter estimates form a martingale, their random limit has an induced martingale posterior distribution. This construction retains a Bayesian interpretation while making the sequential predictive rule, rather than an explicit prior, the primary modeling object [23 , 28 , 19 ] .
The regression setting raises two obstacles for existing predictive Bayes methods. First, a useful regression procedure should allow the predictive distribution to reflect the response distribution while retaining a stable and interpretable estimator of the conditional mean. Second, its finite-sample estimator should not depend on an arbitrary ordering of the observed data. Much recent work develops predictive constructions for density estimation or discrete data [23 , 3 ] . Parametric martingale posteriors have mainly been built from stochastic-gradient or online-gradient updates [39 , 28 , 20 , 24 ] . These updates are valuable general tools, but they require delicate learning-rate choices and typically yield order-dependent finite-sample estimates. They also do not directly provide a flexible regression framework with transparent posterior uncertainty.
We address these issues by constructing martingale posteriors for regression from recursive least squares (RLS). The RLS recursion is a classical and computationally inexpensive estimator, but here it is used as the update of a predictive Bayesian model. The resulting posterior mean is invariant to permutations of the observed data and coincides with a ridge-type regression estimator. Crucially, the plug-in predictive distribution can be chosen separately from this update: it may be Gaussian or heavy-tailed, for example, while preserving the same conditional mean update, so that changing the response distribution requires neither a new estimator nor a new sampler. Posterior uncertainty is obtained by forward predictive resampling, which produces independent draws, with no convergence diagnostics and a truncation error known in closed form. Hyperparameters, such as smoothing parameters, can be selected by maximizing an order-invariant leave-one-out prequential log-likelihood. In our simulations and application, the resulting posterior mean has an accuracy comparable to that of a fully Bayesian model with the same response distribution, at a fraction of its computational cost.
Our main contributions are as follows. We show that the RLS coefficient sequence is a martingale and converges to a random limit, establishing a martingale posterior with an order-invariant finite-sample posterior mean and asymptotic exchangeability of the induced sequence. Conditional on a known covariate design, we derive the exact posterior variance and establish an asymptotically Gaussian approximation that uses only observed covariates. Under Gaussian predictive distributions, we obtain the exact martingale posterior in closed form, and we show that, with a suitable predictive variance, the construction coincides with the conjugate Bayesian linear model, for any covariate distribution. Finally, we show how the construction can be used for additive modeling and to accommodate correlated coefficients, as in mixed effects and Gaussian process regressions.
Predictive Bayes and martingale posteriors
Predictive Bayes starts from a sequential specification rather than an explicit prior. Let ( Y n ) n ≥ 1 (Y_n)_{n\ge1} ( Y n ) n ≥ 1 be a sequence of random variables and write
Y 1 ∼ P 0 , Y n + 1 ∣ y 1 : n ∼ P n , n ≥ 1 , Y_1 \sim P_0,\qquad Y_{n+1}\mid y_{1:n}\sim P_n,\qquad n\ge1, Y 1 ∼ P 0 , Y n + 1 ∣ y 1 : n ∼ P n , n ≥ 1 ,
where P n P_n P n is the one-step-ahead predictive distribution after observing y 1 : n y_{1:n} y 1 : n . By the Ionescu-Tulcea theorem, this sequence of predictive distributions defines a unique joint probability law P \mathbb{P} P for ( Y n ) n ≥ 1 (Y_n)_{n\ge1} ( Y n ) n ≥ 1 [7 ] . By de Finetti’s theorem, if ( Y n ) n ≥ 1 (Y_n)_{n\ge1} ( Y n ) n ≥ 1 is exchangeable, then it admits a representation in terms of a statistical model P ~ \widetilde{P} P and a prior distribution Π \Pi Π such that Y n ∣ P ~ ∼ i i d P ~ Y_n\mid\widetilde{P}\overset{\mathrm{iid}}{\sim}\widetilde{P} Y n ∣ P ∼ iid P , for any n ≥ 1 n\ge1 n ≥ 1 , with P ~ ∼ Π \widetilde{P}\sim\Pi P ∼ Π , and moreover the sequence of predictive distributions ( P n ) n ≥ 1 (P_n)_{n\ge1} ( P n ) n ≥ 1 converges weakly to P ~ \widetilde{P} P , P \mathbb{P} P -a.s. Except for well-known cases, constructing sequences of predictive distributions that ensure exchangeability is challenging in practice. A more tractable goal is asymptotic exchangeability: if P n P_n P n converges weakly to a random distribution P ~ \widetilde{P} P , P \mathbb{P} P -a.s., then ( Y n + 1 , Y n + 2 , … ) → d ( Z 1 , Z 2 , … ) (Y_{n+1},Y_{n+2},\ldots)\xrightarrow{d}(Z_1,Z_2,\ldots) ( Y n + 1 , Y n + 2 , … ) d ( Z 1 , Z 2 , … ) as n → ∞ n\to\infty n → ∞ , where ( Z 1 , Z 2 , … ) (Z_1,Z_2,\ldots) ( Z 1 , Z 2 , … ) is an exchangeable sequence with directing random probability measure P ~ \widetilde{P} P , and → d \xrightarrow{d} d denotes convergence in distribution [4 , 22 , 24 ] . Thus, a convergent predictive rule induces a well-defined distribution for the random limit P ~ \widetilde{P} P .
For a parametric family { P ( ⋅ ∣ θ ) : θ ∈ Θ } \{P(\cdot\mid\theta):\theta\in\Theta\} { P ( ⋅ ∣ θ ) : θ ∈ Θ } , a convenient construction specifies a sequence of estimates θ n = θ ( y 1 : n ) \theta_n=\theta(y_{1:n}) θ n = θ ( y 1 : n ) and uses the plug-in predictive distribution P n ( ⋅ ) = P ( ⋅ ∣ θ n ) P_n(\cdot)=P(\cdot\mid\theta_n) P n ( ⋅ ) = P ( ⋅ ∣ θ n ) . Specifically, a proper Bayesian model is built by sequentially alternating
θ n = θ ( y 1 : n ) and Y n + 1 ∼ P n ( ⋅ ) = P ( ⋅ ∣ θ n ) , (1) \theta_n=\theta(y_{1:n})\qquad\text{and}\qquad Y_{n+1}\sim P_n(\cdot)=P(\cdot\mid\theta_n), \tag*{(1)} θ n = θ ( y 1 : n ) and Y n + 1 ∼ P n ( ⋅ ) = P ( ⋅ ∣ θ n ) , (1)
for n ≥ 1 n\ge1 n ≥ 1 . If ( θ n ) n ≥ 1 (\theta_n)_{n\ge1} ( θ n ) n ≥ 1 is an L 1 L_1 L 1 -bounded martingale, namely
E ( θ n + 1 ∣ y 1 : n ) = θ n , n ≥ 1 , and sup n E ( ∣ θ n ∣ ) < ∞ , \mathbb{E}(\theta_{n+1}\mid y_{1:n})=\theta_n,\qquad n\ge1,\qquad\text{and}\qquad\sup_n\mathbb{E}(|\theta_n|)<\infty, E ( θ n + 1 ∣ y 1 : n ) = θ n , n ≥ 1 , and n sup E ( ∣ θ n ∣ ) < ∞ ,
then by Doob’s martingale convergence theorem, there exists a random θ ~ ∈ Θ \widetilde{\theta}\in\Theta θ ∈ Θ such that θ n → θ ~ \theta_n\to\widetilde{\theta} θ n → θ P \mathbb{P} P -a.s. as n → ∞ n\to\infty n → ∞ [16 ] . When P ( ⋅ ∣ θ ) P(\cdot\mid\theta) P ( ⋅ ∣ θ ) is continuous in θ \theta θ , the predictive distributions converge to P ( ⋅ ∣ θ ~ ) P(\cdot\mid\widetilde{\theta}) P ( ⋅ ∣ θ ) and the conditional law of θ ~ \widetilde{\theta} θ given y 1 : n y_{1:n} y 1 : n is called the martingale posterior [39 , 28 ] . The posterior is generally not available analytically. It can, however, be sampled by predictive resampling: starting from P n P_n P n , repeatedly sample future observations and update the plug-in predictive distributions as in (1) , obtaining θ N \theta_N θ N for N ≫ n N \gg n N ≫ n . Independent forward paths yield independent approximate draws from the martingale posterior [23 , 19 ] .
Existing parametric methods rely on predictive updates that resemble stochastic gradient descent [28 , 20 , 24 , 37 ] . Because the resulting θ n \theta_n θ n typically depends on the order of the observed data and requires careful tuning of the learning rate, most of these methods use the predictive construction only for predictive resampling and compute the initial θ n \theta_n θ n separately through a different scheme. In the next section, we instead use recursive least squares as the predictive update. This yields a posterior mean that is invariant to the ordering of the data, giving an effective method for flexible regression.
Predictive linear regression via recursive least squares
A predictive regression model
We now construct a martingale posterior for the regression coefficient β ∈ R p \beta\in\mathbb{R}^p β ∈ R p . Let X n ∈ R p X_n \in\mathbb{R}^p X n ∈ R p and Y n ∈ R Y_n \in\mathbb{R} Y n ∈ R denote the covariates and response, with observed values x n x_n x n and y n y_n y n . We take X n X_n X n to be iid from an unknown distribution P X P_X P X independent of past responses. At the end of this section, we also introduce the case of fixed covariates, which is used in § 3.2 and 3.3. All proofs are given in Appendix § 11 .
Assumption 1. Conditional on the covariates, the predictive distribution belongs to the location-scale family P β = { P ( ⋅ ∣ x , β , σ ) : x ∈ R p , β ∈ R p , σ > 0 } \mathcal{P}_{\beta} = \{P(\cdot\mid x, \beta, \sigma) : x \in\mathbb{R}^p, \beta\in\mathbb{R}^p, \sigma> 0\} P β = { P ( ⋅ ∣ x , β , σ ) : x ∈ R p , β ∈ R p , σ > 0 } , where P ( ⋅ ∣ x , β , σ ) P(\cdot\mid x, \beta, \sigma) P ( ⋅ ∣ x , β , σ ) is the law of x T β + σ ξ x^{\mathsf{T}}\beta+ \sigma\xi x T β + σ ξ and ξ ∼ P ξ \xi\sim P_{\xi} ξ ∼ P ξ , a fixed distribution on R \mathbb{R} R with E ( ξ ) = 0 \mathbb{E}(\xi) = 0 E ( ξ ) = 0 and E ( ξ 2 ) = 1 \mathbb{E}(\xi^2) = 1 E ( ξ 2 ) = 1 .
Hence, P ( ⋅ ∣ x , β , σ ) P(\cdot\mid x, \beta, \sigma) P ( ⋅ ∣ x , β , σ ) has mean x T β x^{\mathsf{T}}\beta x T β and variance σ 2 \sigma^2 σ 2 . A Gaussian P ξ P_{\xi} P ξ gives Gaussian predictive distributions, whereas, for example, a standardized Student-t t t P ξ P_{\xi} P ξ gives heavy-tailed ones. Starting from β 0 \beta_0 β 0 and a positive-definite matrix V 0 V_0 V 0 , the responses are generated sequentially as
Y 1 ∣ x 1 ∼ P 0 ( ⋅ ) , and Y n + 1 ∣ y 1 : n , x 1 : n + 1 ∼ P n ( ⋅ ) , (2) Y_1 \mid x_1 \sim P_0(\cdot), \qquad\text{and} \qquad Y_{n+1} \mid y_{1:n}, x_{1:n+1} \sim P_n(\cdot), \tag*{(2)} Y 1 ∣ x 1 ∼ P 0 ( ⋅ ) , and Y n + 1 ∣ y 1 : n , x 1 : n + 1 ∼ P n ( ⋅ ) , (2)
for n ≥ 1 n \ge1 n ≥ 1 , where P n ( ⋅ ) = P ( ⋅ ∣ x n + 1 , β n , σ n + 1 ) ∈ P β P_n(\cdot) = P(\cdot\mid x_{n+1}, \beta_n, \sigma_{n+1}) \in\mathcal{P}_{\beta} P n ( ⋅ ) = P ( ⋅ ∣ x n + 1 , β n , σ n + 1 ) ∈ P β for n ≥ 0 n \ge0 n ≥ 0 , β n \beta_n β n denotes an estimator of β \beta β based on the first n n n observations ( x 1 : n , y 1 : n ) (x_{1:n}, y_{1:n}) ( x 1 : n , y 1 : n ) , and σ n + 1 \sigma_{n+1} σ n + 1 controls the scale of the next predictive distribution. Together with X n + 1 ∼ P X X_{n+1} \sim P_X X n + 1 ∼ P X independently of ( X 1 : n , Y 1 : n ) (X_{1:n}, Y_{1:n}) ( X 1 : n , Y 1 : n ) , Equation (2) defines the joint law P \mathbb{P} P of ( X n , Y n ) n ≥ 1 (X_n, Y_n)_{n \ge1} ( X n , Y n ) n ≥ 1 , under which ( X 1 : n , Y 1 : n ) (X_{1:n}, Y_{1:n}) ( X 1 : n , Y 1 : n ) has distribution ∏ i = 1 n P X ( d x i ) P i − 1 ( d y i ) \prod_{i=1}^{n} P_X(\mathrm{d}x_i) P_{i-1}(\mathrm{d}y_i) ∏ i = 1 n P X ( d x i ) P i − 1 ( d y i ) for every n ≥ 1 n \ge1 n ≥ 1 . The key modeling decision is the update of β n \beta_n β n . We use recursive least squares:
β n = β n − 1 + a n V n − 1 x n ( y n − x n T β n − 1 ) , (3) \beta_n = \beta_{n-1} + a_n V_{n-1}x_n(y_n - x_n^{\mathsf{T}}\beta_{n-1}), \tag*{(3)} β n = β n − 1 + a n V n − 1 x n ( y n − x n T β n − 1 ) , (3)
where a n = ( 1 + x n T V n − 1 x n ) − 1 a_n = (1 + \mathbf{x}_n^{\mathsf{T}}V_{n-1}\mathbf{x}_n)^{-1} a n = ( 1 + x n T V n − 1 x n ) − 1 and V n V_n V n is a p × p p \times p p × p matrix updated as
V n = V n − 1 − a n V n − 1 x n x n T V n − 1 . (4) V_n = V_{n-1} - a_n V_{n-1}\mathbf{x}_n\mathbf{x}_n^{\mathsf{T}}V_{n-1}. \tag*{(4)} V n = V n − 1 − a n V n − 1 x n x n T V n − 1 . (4)
The update is the classical recursive least-squares estimator, written through the Sherman–Morrison formula to avoid repeated matrix inversions [32 ] . Its role here is different from its usual role in online estimation: it is the predictive update that defines the posterior mean and, as the next result shows, induces a martingale posterior.
Assumption 2 (Random design). The covariates satisfy X n + 1 ∼ P X \mathbf{X}_{n+1} \sim P_X X n + 1 ∼ P X independently of ( X 1 : n , Y 1 : n ) (X_{1:n},Y_{1:n}) ( X 1 : n , Y 1 : n ) , for n ≥ 0 n \ge0 n ≥ 0 , so that ( X n , Y n ) n ≥ 1 (X_n,Y_n)_{n\ge1} ( X n , Y n ) n ≥ 1 has probability law P P P . The sequence ( β n ) n ≥ 1 (\beta_n)_{n\ge1} ( β n ) n ≥ 1 is defined by (3) , β 0 \beta_0 β 0 is fixed, and V 0 V_0 V 0 is fixed and positive-definite. The predictive scales are σ n + 1 = s n ( X n + 1 ) \sigma_{n+1} = s_n(X_{n+1}) σ n + 1 = s n ( X n + 1 ) , where s n : R p → ( 0 , σ ∗ ] s_n : \mathbb{R}^p \to(0,\sigma_*] s n : R p → ( 0 , σ ∗ ] is a measurable function of x \mathbf{x} x and X 1 : n X_{1:n} X 1 : n , for some σ ∗ < ∞ \sigma_* < \infty σ ∗ < ∞ , and, P \mathbb{P} P -almost surely, s n ( x ) → σ ∈ ( 0 , ∞ ) s_n(\mathbf{x}) \to\sigma\in(0,\infty) s n ( x ) → σ ∈ ( 0 , ∞ ) for P X P_X P X -almost every x \mathbf{x} x .
Theorem 1. Under Assumptions 1 and 2,
(a) the sequence of random vectors ( β n ) n ≥ 1 (\beta_n)_{n\ge1} ( β n ) n ≥ 1 converges P \mathbb{P} P -almost surely and in L 2 L_2 L 2 to a random limit β ~ \widetilde{\beta} β , and for every n ≥ 1 n \ge1 n ≥ 1 , E ( β ~ ∣ Y 1 : n , X 1 : n ) = β n \mathbb{E}(\widetilde{\beta} \mid Y_{1:n},X_{1:n}) = \beta_n E ( β ∣ Y 1 : n , X 1 : n ) = β n ;
(b) the sequence ( X n , Y n ) n ≥ 1 (X_n,Y_n)_{n\ge1} ( X n , Y n ) n ≥ 1 is asymptotically exchangeable with directing random probability measure P ~ X , Y ( d x , d y ) = P X ( d x ) P ( d y ∣ x , β ~ , σ ) \widetilde{P}_{X,Y}(\mathrm{d}x,\mathrm{d}y) = P_X(\mathrm{d}x)\,P(\mathrm{d}y \mid x,\widetilde{\beta},\sigma) P X , Y ( d x , d y ) = P X ( d x ) P ( d y ∣ x , β , σ ) ; in particular, the conditional distribution of Y Y Y given X = x X = x X = x under P ~ X , Y \widetilde{P}_{X,Y} P X , Y is P ( ⋅ ∣ x , β ~ , σ ) P(\cdot\mid x,\widetilde{\beta},\sigma) P ( ⋅ ∣ x , β , σ ) .
The theorem provides the Bayesian interpretation of the construction. The law of β ~ \widetilde{\beta} β is the implicit prior induced by the predictive update, and the conditional law of β ~ \widetilde{\beta} β given the observed data is the martingale posterior. Its mean is exactly the RLS estimate β n \beta_n β n . Although the law of ( X n , Y n ) n ≥ 1 (X_n,Y_n)_{n\ge1} ( X n , Y n ) n ≥ 1 is in general not exchangeable, but only asymptotically so, the estimator β n \beta_n β n does not depend on the ordering of the observations.
Remark 1. For every n ≥ 1 n \ge1 n ≥ 1 , the recursive least squares estimator β n \beta_n β n is invariant to permutations of the observations ( x 1 , y 1 ) , … , ( x n , y n ) (x_1,y_1),\ldots,(x_n,y_n) ( x 1 , y 1 ) , … , ( x n , y n ) . Indeed, (3) and (4) can be rewritten as
β n = V n ( V 0 − 1 β 0 + ∑ i = 1 n y i x i ) , and V n = ( V 0 − 1 + ∑ i = 1 n x i x i T ) − 1 , (5) \beta_n = V_n\left(V_0^{-1}\beta_0 + \sum_{i=1}^{n} y_i\mathbf{x}_i\right), \qquad\text{and} \qquad V_n = \left(V_0^{-1} + \sum_{i=1}^{n}\mathbf{x}_i\mathbf{x}_i^{\mathsf{T}}\right)^{-1}, \tag*{(5)} β n = V n ( V 0 − 1 β 0 + i = 1 ∑ n y i x i ) , and V n = ( V 0 − 1 + i = 1 ∑ n x i x i T ) − 1 , (5)
respectively, which are symmetric functions of ( x 1 , y 1 ) , … , ( x n , y n ) (x_1,y_1),\ldots,(x_n,y_n) ( x 1 , y 1 ) , … , ( x n , y n ) .
Equation (5) identifies the finite-sample posterior mean as a generalized ridge estimator. In particular, setting β 0 = 0 \beta_0 = 0 β 0 = 0 and V 0 = κ I p V_0 = \kappa I_p V 0 = κ I p gives ridge regression, with ordinary least squares recovered as κ → ∞ \kappa\to\infty κ → ∞ when ∑ i = 1 n x i x i T \sum_{i=1}^{n}\mathbf{x}_i\mathbf{x}_i^{\mathsf{T}} ∑ i = 1 n x i x i T is invertible. Equation (5) also coincides with the posterior mean of a Gaussian linear model with variance σ 2 \sigma^2 σ 2 and a Gaussian prior on the regression coefficients having mean β 0 \beta_0 β 0 and covariance matrix σ 2 V 0 \sigma^{2}\mathbf{V}_{0} σ 2 V 0 . The analogy with the posterior mean under a normal linear model is useful for interpreting β 0 \beta_{0} β 0 and V 0 \mathbf{V}_{0} V 0 , but the present construction is not that Bayesian model: the predictive distribution may be non-Gaussian, and even a Gaussian plug-in predictive distribution does not in general reproduce the ordinary Gaussian–Gaussian posterior, which is recovered only as a special case (see Section 3.3).
In many applications, the covariates are not regarded as random, either because they are set by design or simply because the analysis is conditional on them. The whole covariate sequence x 1 : ∞ \mathbf{x}_{1:\infty} x 1 : ∞ is then treated as fixed, and Equation (2) defines a law P x 1 : ∞ \mathbb{P}_{\mathbf{x}_{1:\infty}} P x 1 : ∞ of ( Y n ) n ≥ 1 (Y_n)_{n\geq1} ( Y n ) n ≥ 1 . The two designs are naturally connected, since P x 1 : ∞ \mathbb{P}_{\mathbf{x}_{1:\infty}} P x 1 : ∞ is the conditional law of ( Y n ) n ≥ 1 (Y_n)_{n\geq1} ( Y n ) n ≥ 1 under P \mathbb{P} P given X 1 : ∞ = x 1 : ∞ \mathbf{X}_{1:\infty}=\mathbf{x}_{1:\infty} X 1 : ∞ = x 1 : ∞ .
Assumption 3 (Fixed design). The covariates x 1 : ∞ \mathbf{x}_{1:\infty} x 1 : ∞ are a known and bounded sequence, and ( Y n ) n ≥ 1 (Y_n)_{n\geq1} ( Y n ) n ≥ 1 has probability law P x 1 : ∞ \mathbb{P}_{\mathbf{x}_{1:\infty}} P x 1 : ∞ . The sequence ( β n ) n ≥ 1 (\beta_n)_{n\geq1} ( β n ) n ≥ 1 is defined by (3) , β 0 \beta_0 β 0 is fixed, and V 0 \mathbf{V}_0 V 0 is fixed and positive-definite. The predictive scales are σ n = s n − 1 ( x n ) \sigma_n=s_{n-1}(\mathbf{x}_n) σ n = s n − 1 ( x n ) , where s n − 1 : R p → ( 0 , σ ∗ ) s_{n-1}:\mathbb{R}^{p}\to(0,\sigma_*) s n − 1 : R p → ( 0 , σ ∗ ) depends only on x 1 : n − 1 \mathbf{x}_{1:n-1} x 1 : n − 1 , for some σ ∗ < ∞ \sigma_*<\infty σ ∗ < ∞ , and σ n → σ ∈ ( 0 , ∞ ) \sigma_n\to\sigma\in(0,\infty) σ n → σ ∈ ( 0 , ∞ ) . Moreover, we assume that Ω : = lim n → ∞ n − 1 ∑ i = 1 n x i x i T \boldsymbol{\Omega}:=\lim_{n\to\infty}n^{-1}\sum_{i=1}^{n}\mathbf{x}_i\mathbf{x}_i^{\mathsf{T}} Ω := lim n → ∞ n − 1 ∑ i = 1 n x i x i T exists and is positive definite.
Proposition 1. Under Assumptions 1 and 3,
(a) the sequence ( β n ) n ≥ 1 (\beta_n)_{n\geq1} ( β n ) n ≥ 1 converges P x 1 : ∞ \mathbb{P}_{\mathbf{x}_{1:\infty}} P x 1 : ∞ -almost surely and in L 2 L_2 L 2 to a random limit β ~ \widetilde{\beta} β , and for every n ≥ 1 n\geq1 n ≥ 1 , E ( β ~ ∣ y 1 : n ) = β n \mathbb{E}(\widetilde{\beta}\mid y_{1:n})=\beta_n E ( β ∣ y 1 : n ) = β n ;
(b) P x 1 : ∞ \mathbb{P}_{\mathbf{x}_{1:\infty}} P x 1 : ∞ -almost surely, the predictive distributions P n P_n P n and P ( ⋅ ∣ x n + 1 , β ~ , σ ) P(\mathord{\cdot}\mid\mathbf{x}_{n+1},\widetilde{\beta},\sigma) P ( ⋅ ∣ x n + 1 , β , σ ) merge weakly, that is, ∫ f d P n − ∫ f d P ( ⋅ ∣ x n + 1 , β ~ , σ ) → 0 \int f\,\mathrm{d}P_n-\int f\,\mathrm{d}P(\mathord{\cdot}\mid\mathbf{x}_{n+1},\widetilde{\beta},\sigma)\to0 ∫ f d P n − ∫ f d P ( ⋅ ∣ x n + 1 , β , σ ) → 0 as n → ∞ n\to\infty n → ∞ for every bounded continuous f f f .
Part (b) is the fixed-design counterpart of Theorem 1(b). Since x n + 1 \mathbf{x}_{n+1} x n + 1 changes with n n n , P n P_n P n need not converge, and convergence is replaced by weak merging, a notion from D’Aristotle et al. (1988), weaker than the merging in total variation of Blackwell and Dubins [9 ] . Informally, for large n n n , P n P_n P n is approximately P ( ⋅ ∣ x n + 1 , β ~ , σ ) P(\mathord{\cdot}\mid\mathbf{x}_{n+1},\widetilde{\beta},\sigma) P ( ⋅ ∣ x n + 1 , β , σ ) .
Under Assumption 3, the predictive residuals are Y i − x i T β i − 1 = σ i ξ i Y_i-\mathbf{x}_i^{\mathsf{T}}\beta_{i-1}=\sigma_i\xi_i Y i − x i T β i − 1 = σ i ξ i , where, by Assumption 1, the ξ i \xi_i ξ i are iid from P ξ P_{\xi} P ξ , each independent of y 1 : i − 1 y_{1:i-1} y 1 : i − 1 . Hence, by (3) and Proposition 1,
β ~ = β n + ∑ i > n w i ξ i , w i = σ i a i V i − 1 x i , (6) \widetilde{\beta}=\beta_n+\sum_{i>n}w_i\xi_i,\qquad w_i=\sigma_i a_i\mathbf{V}_{i-1}\mathbf{x}_i, \tag*{(6)} β = β n + i > n ∑ w i ξ i , w i = σ i a i V i − 1 x i , (6)
where the weights w i w_i w i are deterministic and the series converges almost surely and in L 2 L_2 L 2 . Since ξ n + 1 , ξ n + 2 , … \xi_{n+1},\xi_{n+2},\ldots ξ n + 1 , ξ n + 2 , … are independent of y 1 : n y_{1:n} y 1 : n , the martingale posterior is a location family centered at β n \beta_n β n , whose shape does not depend on y 1 : n y_{1:n} y 1 : n .
As for the choice of the sequence ( σ n ) n ≥ 1 (\sigma_n)_{n\geq1} ( σ n ) n ≥ 1 , one could simply set σ n = σ \sigma_n=\sigma σ n = σ for all n ≥ 1 n\geq1 n ≥ 1 . A better default, however, is σ n 2 = σ 2 / a n = σ 2 ( 1 + x n T V n − 1 x n ) \sigma_n^2=\sigma^2/a_n=\sigma^2(1+\mathbf{x}_n^{\mathsf{T}}\mathbf{V}_{n-1}\mathbf{x}_n) σ n 2 = σ 2 / a n = σ 2 ( 1 + x n T V n − 1 x n ) , which is the predictive variance of the conjugate Gaussian linear model with prior covariance matrix σ 2 V 0 \sigma^2\mathbf{V}_0 σ 2 V 0 and makes the posterior covariance matrix equal to σ 2 V n \sigma^{2}V_n σ 2 V n (see § 3.2 ); it satisfies Assumption 2 when P X P_X P X has bounded support and E P X ( X X T ) \mathbb{E}_{P_X}(\mathbf{X}\mathbf{X}^{\mathsf{T}}) E P X ( X X T ) is positive definite, and Assumption 3 for any bounded design with Ω \mathbf{\Omega} Ω positive definite. The scale σ \sigma σ can then be estimated from the data or assigned a prior; see § 4.3 for further details. A pipeline for selecting the most appropriate P n P_n P n in applications is illustrated in Appendix § 13 .
Posterior variance
While Theorem 1 and Proposition 1 identify the posterior mean for any choice of P ξ P_{\xi} P ξ , we now characterize the posterior variance under the fixed design of Assumption 3.
Proposition 2. For every n ≥ 0 n \ge0 n ≥ 0 , let Σ n : = ∑ i > n w i w i T = ∑ i > n σ i 2 a i ( V i − 1 − V i ) \boldsymbol{\Sigma}_n := \sum_{i>n}\mathbf{w}_i\mathbf{w}_i^{\mathsf{T}} = \sum_{i>n}\sigma_i^2a_i(\mathbf{V}_{i-1}-\mathbf{V}_i) Σ n := ∑ i > n w i w i T = ∑ i > n σ i 2 a i ( V i − 1 − V i ) , with w i = σ i a i V i − 1 x i \mathbf{w}_i=\sigma_i a_i\mathbf{V}_{i-1}\mathbf{x}_i w i = σ i a i V i − 1 x i . Under Assumptions 1 and 3, for any n ≥ 1 n \ge1 n ≥ 1 ,
var ( β ~ ∣ y 1 : n ) = Σ n , and var ( β ~ ) = Σ 0 . \operatorname{var}(\widetilde{\boldsymbol{\beta}}\mid y_{1:n})=\boldsymbol{\Sigma}_n,\qquad\text{and}\qquad\operatorname{var}(\widetilde{\boldsymbol{\beta}})=\boldsymbol{\Sigma}_0. var ( β ∣ y 1 : n ) = Σ n , and var ( β ) = Σ 0 . In particular, if σ n 2 = σ 2 / a n = σ 2 ( 1 + x n T V n − 1 x n ) \sigma_n^2=\sigma^2/a_n=\sigma^2(1+\mathbf{x}_n^{\mathsf{T}}\mathbf{V}_{n-1}\mathbf{x}_n) σ n 2 = σ 2 / a n = σ 2 ( 1 + x n T V n − 1 x n ) , then Σ n = σ 2 V n \boldsymbol{\Sigma}_n=\sigma^2\mathbf{V}_n Σ n = σ 2 V n and Σ 0 = σ 2 V 0 \boldsymbol{\Sigma}_0=\sigma^2\mathbf{V}_0 Σ 0 = σ 2 V 0 .
Thus, Σ n = ∑ i > n σ i 2 a i ( V i − 1 − V i ) \boldsymbol{\Sigma}_n=\sum_{i>n}\sigma_i^2a_i(\mathbf{V}_{i-1}-\mathbf{V}_i) Σ n = ∑ i > n σ i 2 a i ( V i − 1 − V i ) is the exact posterior variance for any choice of P ξ P_{\xi} P ξ , whereas Σ 0 \boldsymbol{\Sigma}_0 Σ 0 is the exact prior variance. Both quantities depend on past and future covariates. For σ n 2 = σ 2 / a n \sigma_n^2=\sigma^2/a_n σ n 2 = σ 2 / a n , the posterior variance of a traditional Bayesian linear model is recovered, and Σ n = σ 2 V n \boldsymbol{\Sigma}_n=\sigma^2\mathbf{V}_n Σ n = σ 2 V n depends only on the observed covariates; this is the only choice of σ n \sigma_n σ n for which Σ n = σ 2 V n \boldsymbol{\Sigma}_n=\sigma^2\mathbf{V}_n Σ n = σ 2 V n for every n n n .
For a general choice of σ n \sigma_n σ n , Proposition 2 may seem to contradict the interpretation of σ 2 V 0 \sigma^2\mathbf{V}_0 σ 2 V 0 as a prior guess for the covariance of β ~ \widetilde{\boldsymbol{\beta}} β , and raises the question of how V n \mathbf{V}_n V n and Σ n \boldsymbol{\Sigma}_n Σ n are related. This is clarified by the following proposition.
Proposition 3. Under Assumptions 1 and 3, n V n → Ω − 1 n\mathbf{V}_n\to\mathbf{\Omega}^{-1} n V n → Ω − 1 and n Σ n → σ 2 Ω − 1 n\boldsymbol{\Sigma}_n\to\sigma^2\mathbf{\Omega}^{-1} n Σ n → σ 2 Ω − 1 .
By Proposition 3, both n Σ n n\boldsymbol{\Sigma}_n n Σ n and n σ 2 V n n\sigma^2\mathbf{V}_n n σ 2 V n converge to σ 2 Ω − 1 \sigma^2\mathbf{\Omega}^{-1} σ 2 Ω − 1 . Hence, σ 2 V n \sigma^2\mathbf{V}_n σ 2 V n , which depends only on the observed covariates, is asymptotically equivalent to the posterior covariance for any choice of σ n \sigma_n σ n . The result follows naturally from (5) , which writes V n \mathbf{V}_n V n as the inverse of a regularized empirical second-moment matrix. Lemma A.3 in the Appendix, on which the proof of Proposition 3 is based, also shows that, if σ n = σ \sigma_n=\sigma σ n = σ for all n ≥ 1 n\ge1 n ≥ 1 , then Σ n ⪯ σ 2 V n \boldsymbol{\Sigma}_n\preceq\sigma^2\mathbf{V}_n Σ n ⪯ σ 2 V n , where A ⪯ B A\preceq B A ⪯ B means that B − A B-A B − A is positive semidefinite. Hence, with a constant predictive variance, the posterior covariance never exceeds σ 2 V n \sigma^2\mathbf{V}_n σ 2 V n , obtained with σ n 2 = σ 2 / a n \sigma_n^2=\sigma^2/a_n σ n 2 = σ 2 / a n .
Asymptotic and exact distributions of β ~ \widetilde{\boldsymbol{\beta}} β
Building on the exact variance Σ n \boldsymbol{\Sigma}_n Σ n and its large-sample approximation σ 2 V n \sigma^2\mathbf{V}_n σ 2 V n , the next theorem gives an asymptotic approximation to the martingale posterior.
Theorem 2. Under Assumptions 1 1 1 and 3 3 3 ,
Σ n − 1 / 2 ( β ~ − β n ) → d Z , σ − 1 V n − 1 / 2 ( β ~ − β n ) → d Z , Z ∼ N p ( 0 , I p ) , \boldsymbol{\Sigma}_{n}^{-1/2}(\widetilde{\boldsymbol{\beta}}-\boldsymbol{\beta}_{n}) \xrightarrow{d} \boldsymbol{Z}, \qquad\sigma^{-1}\mathbf{V}_{n}^{-1/2}(\widetilde{\boldsymbol{\beta}}-\boldsymbol{\beta}_{n}) \xrightarrow{d} \boldsymbol{Z}, \qquad\boldsymbol{Z} \sim\mathcal{N}_{p}(\mathbf{0},\mathbf{I}_{p}), Σ n − 1/2 ( β − β n ) d Z , σ − 1 V n − 1/2 ( β − β n ) d Z , Z ∼ N p ( 0 , I p ) , conditionally on y 1 : n y_{1:n} y 1 : n , as n → ∞ n \to\infty n → ∞ , where → d \xrightarrow{d} d denotes convergence in distribution.
Consequently, for large n n n ,
β ~ ∣ y 1 : n , x 1 : n ∼ ˙ N p ( β n , σ 2 V n ) . (7) \widetilde{\boldsymbol{\beta}} \mid y_{1:n},\mathbf{x}_{1:n} \mathrel{\dot\sim} \mathcal{N}_{p}(\boldsymbol{\beta}_{n},\sigma^{2}\mathbf{V}_{n}). \tag*{(7)} β ∣ y 1 : n , x 1 : n ∼ ˙ N p ( β n , σ 2 V n ) . (7)
Approximation (7) always has the exact posterior mean and, when σ n 2 = σ 2 / a n \sigma_{n}^{2}=\sigma^{2}/a_{n} σ n 2 = σ 2 / a n , also the exact posterior variance; for other choices of σ n \sigma_{n} σ n the variance is approximate. Notably, it depends only on observed covariates, even though Theorem 2 assumes that x 1 : ∞ \mathbf{x}_{1:\infty} x 1 : ∞ is known. Theorem 2 is a predictive, Bayesian result on the martingale posterior, not on the sampling distribution of β n \boldsymbol{\beta}_{n} β n , and gives no frequentist guarantees. When P ξ P_{\xi} P ξ is Gaussian, β ~ − β n = ∑ i > n w i ξ i \widetilde{\boldsymbol{\beta}}-\boldsymbol{\beta}_{n}=\sum_{i>n}\mathbf{w}_{i}\xi_{i} β − β n = ∑ i > n w i ξ i is a series of independent Gaussian vectors, and the exact prior and posterior distributions of β ~ \widetilde{\boldsymbol{\beta}} β can be derived.
Theorem 3. Under Assumptions 1 1 1 and 3 3 3 , assume also that P ξ = N ( 0 , 1 ) P_{\xi}=\mathcal{N}(0,1) P ξ = N ( 0 , 1 ) . Then,
β ~ ∼ N p ( β 0 , Σ 0 ) , and β ~ ∣ y 1 : n ∼ N p ( β n , Σ n ) , for any n ≥ 1. \widetilde{\boldsymbol{\beta}} \sim\mathcal{N}_{p}(\boldsymbol{\beta}_{0},\boldsymbol{\Sigma}_{0}), \qquad\text{and} \qquad\widetilde{\boldsymbol{\beta}} \mid y_{1:n} \sim\mathcal{N}_{p}(\boldsymbol{\beta}_{n},\boldsymbol{\Sigma}_{n}), \qquad\text{for any } n \ge1. β ∼ N p ( β 0 , Σ 0 ) , and β ∣ y 1 : n ∼ N p ( β n , Σ n ) , for any n ≥ 1. In particular, if σ n 2 = σ 2 / a n = σ 2 ( 1 + x n T V n − 1 − 1 x n ) \sigma_{n}^{2}=\sigma^{2}/a_{n}=\sigma^{2}(1+\mathbf{x}_{n}^{\mathsf{T}}\mathbf{V}_{n-1}^{-1}\mathbf{x}_{n}) σ n 2 = σ 2 / a n = σ 2 ( 1 + x n T V n − 1 − 1 x n ) , then
β ~ ∼ N p ( β 0 , σ 2 V 0 ) , and β ~ ∣ y 1 : n ∼ N p ( β n , σ 2 V n ) , for any n ≥ 1. \widetilde{\boldsymbol{\beta}} \sim\mathcal{N}_{p}(\boldsymbol{\beta}_{0},\sigma^{2}\mathbf{V}_{0}), \qquad\text{and} \qquad\widetilde{\boldsymbol{\beta}} \mid y_{1:n} \sim\mathcal{N}_{p}(\boldsymbol{\beta}_{n},\sigma^{2}\mathbf{V}_{n}), \qquad\text{for any } n \ge1. β ∼ N p ( β 0 , σ 2 V 0 ) , and β ∣ y 1 : n ∼ N p ( β n , σ 2 V n ) , for any n ≥ 1.
When σ n 2 = σ 2 / a n \sigma_{n}^{2}=\sigma^{2}/a_{n} σ n 2 = σ 2 / a n and P ξ P_{\xi} P ξ is Gaussian, the ordinary Gaussian–Gaussian prior and posterior are recovered, because the whole construction then coincides with the conjugate model β ∼ N p ( β 0 , σ 2 V 0 ) \boldsymbol{\beta}\sim\mathcal{N}_{p}(\boldsymbol{\beta}_{0},\sigma^{2}\mathbf{V}_{0}) β ∼ N p ( β 0 , σ 2 V 0 ) , Y i ∣ x i , β ∼ i n d N ( x i T β , σ 2 ) Y_{i}\mid\mathbf{x}_{i},\boldsymbol{\beta}\mathrel{\mathop{\sim}^{\mathrm{ind}}}\mathcal{N}(\mathbf{x}_{i}^{\mathsf{T}}\boldsymbol{\beta},\sigma^{2}) Y i ∣ x i , β ∼ ind N ( x i T β , σ 2 ) . Indeed, the predictive distribution in (2) is then N { x n + 1 T β n , σ 2 ( 1 + x n + 1 T V n x n + 1 ) } \mathcal{N}\{\mathbf{x}_{n+1}^{\mathsf{T}}\boldsymbol{\beta}_{n},\sigma^{2}(1+\mathbf{x}_{n+1}^{\mathsf{T}}\mathbf{V}_{n}\mathbf{x}_{n+1})\} N { x n + 1 T β n , σ 2 ( 1 + x n + 1 T V n x n + 1 )} , which is the posterior predictive distribution of the conjugate model, whose posterior mean and covariance are β n \boldsymbol{\beta}_{n} β n and σ 2 V n \sigma^{2}\mathbf{V}_{n} σ 2 V n . By the Ionescu-Tulcea theorem, the two models have the same law of ( Y n ) n ≥ 1 (Y_{n})_{n\ge1} ( Y n ) n ≥ 1 .
The Gaussian case serves mainly as a reference point: it makes the induced prior and posterior fully explicit, whereas non-Gaussian predictive choices retain the same posterior mean but require predictive resampling to characterize the rest of the posterior.
We finally return to the random design of Assumption 2 2 2 , with X n ∼ i i d P X \mathbf{X}_{n}\mathrel{\mathop{\sim}^{\mathrm{iid}}}P_{X} X n ∼ iid P X having bounded support and E P X ( X X T ) \mathbb{E}_{P_{X}}(\mathbf{X}\mathbf{X}^{\mathsf{T}}) E P X ( X X T ) positive definite, and with σ n → σ \sigma_{n}\to\sigma σ n → σ almost surely, as for σ n 2 = σ 2 / a n \sigma_{n}^{2}=\sigma^{2}/a_{n} σ n 2 = σ 2 / a n . Then, Assumption 3 3 3 holds almost surely given X 1 : ∞ \mathbf{X}_{1:\infty} X 1 : ∞ , and E ( β ~ ∣ y 1 : n , X 1 : ∞ ) = β n \mathbb{E}(\widetilde{\boldsymbol{\beta}}\mid y_{1:n},\mathbf{X}_{1:\infty})=\boldsymbol{\beta}_{n} E ( β ∣ y 1 : n , X 1 : ∞ ) = β n does not depend on the future covariates. By the law of total variance, var ( β ~ ∣ y 1 : n , x 1 : n ) = E ( Σ n ∣ x 1 : n ) \operatorname{var}(\widetilde{\boldsymbol{\beta}}\mid y_{1:n},\mathbf{x}_{1:n})=\mathbb{E}(\boldsymbol{\Sigma}_{n}\mid\mathbf{x}_{1:n}) var ( β ∣ y 1 : n , x 1 : n ) = E ( Σ n ∣ x 1 : n ) for every n ≥ 0 n\ge0 n ≥ 0 , where the expectation is over X n + 1 , X n + 2 , … ∼ i i d P X \mathbf{X}_{n+1},\mathbf{X}_{n+2},\ldots\mathrel{\mathop{\sim}^{\mathrm{iid}}}P_{X} X n + 1 , X n + 2 , … ∼ iid P X . With σ n 2 = σ 2 / a n \sigma_{n}^{2}=\sigma^{2}/a_{n} σ n 2 = σ 2 / a n , this equals σ 2 V n \sigma^{2}\mathbf{V}_{n} σ 2 V n , so neither the posterior mean nor the posterior covariance depends on P X P_{X} P X . This is the weak design invariance property discussed by Sun and Fong [37 ] : P X P_{X} P X can be replaced in predictive resampling by any distribution with bounded support and E P X ( X X T ) \mathbb{E}_{P_{X}}(\mathbf{X}\mathbf{X}^{\mathsf{T}}) E P X ( X X T ) positive definite, also when n < p n<p n < p . If, in addition, P ξ P_{\xi} P ξ is Gaussian, the equivalence with the conjugate model extends to the joint law of ( X n , Y n ) n ≥ 1 (X_n,Y_n)_{n\geq1} ( X n , Y n ) n ≥ 1 , which is therefore exchangeable. Moreover, since V n → 0 V_n \to0 V n → 0 almost surely, the limit β ~ = lim n → ∞ β n \widetilde{\beta} = \lim_{n\to\infty}\beta_n β = lim n → ∞ β n is the regression parameter β \beta β of the conjugate model, and β ~ ∣ X 1 : n , Y 1 : n ∼ N p ( β n , σ 2 V n ) \widetilde{\beta} \mid X_{1:n},Y_{1:n} \sim\mathcal{N}_p(\beta_n,\sigma^2V_n) β ∣ X 1 : n , Y 1 : n ∼ N p ( β n , σ 2 V n ) for any P X P_X P X with E P X ( X X T ) \mathbb{E}_{P_X}(XX^{\mathsf{T}}) E P X ( X X T ) positive definite, even without bounded support. In this case, the whole martingale posterior, and not only its first two moments, does not depend on P X P_X P X . With a Gaussian P ξ P_\xi P ξ , the update (3) is also a special case of the score-based recursion of Sun and Fong [37 ] ; here, it is kept for any P ξ P_\xi P ξ .
Posterior sampling
Except for the Gaussian case of Theorem 3, the martingale posterior distribution for β ~ \widetilde{\beta} β is not available in closed form. Posterior sampling is carried out by predictive resampling. Algorithm 1 presents the standard predictive resampling when using recursive least squares. Starting from β n \beta_n β n and V n V_n V n , for i = n + 1 , … , N i=n+1,\ldots,N i = n + 1 , … , N , the algorithm performs B B B independent forward samples, each sequentially generating a new pair ( x i , y i ) (x_i,y_i) ( x i , y i ) and updating β i \beta_i β i and V i V_i V i using (3) –(4) . The B B B paths are independent, so they can be run in parallel and the effective sample size of the draws is exactly B B B , whereas MCMC methods may suffer from poor efficiency. In practice, P X P_X P X is usually unknown and must be replaced by an estimator P ^ X \widehat{P}_X P X ; the empirical distribution of the observed x i x_i x i is typically a good candidate when n > p n>p n > p .
Standard predictive resampling can be further accelerated by using the same covariate sequence X n + 1 : N X_{n+1:N} X n + 1 : N across all B B B sample paths, as also done by Sun and Fong [37 ] and detailed in Algorithm 2. This is the fixed-design setting of § 3.1 , with all future covariates drawn before forward sampling begins. Given the covariates, (3) gives β N = β n + ∑ i = n + 1 N w i ξ i \beta_N=\beta_n+\sum_{i=n+1}^{N}w_i\xi_i β N = β n + ∑ i = n + 1 N w i ξ i , the series in (6) truncated at N N N , with ξ n + 1 , … , ξ N \xi_{n+1},\ldots,\xi_N ξ n + 1 , … , ξ N iid from P ξ P_\xi P ξ . The weights w i = σ i a i V i − 1 X i w_i=\sigma_i a_iV_{i-1}X_i w i = σ i a i V i − 1 X i depend only on the covariates, since σ i = s i − 1 ( X i ) \sigma_i=s_{i-1}(X_i) σ i = s i − 1 ( X i ) . They can therefore be computed once, and each draw only requires sampling ξ n + 1 , … , ξ N \xi_{n+1},\ldots,\xi_N ξ n + 1 , … , ξ N and computing a weighted sum. The effect of stopping at N N N can also be checked directly: given the covariates, var ( β N ∣ y 1 : n ) = ∑ i = n + 1 N w i w i T = Σ n − Σ N \operatorname{var}(\beta_N\mid y_{1:n})=\sum_{i=n+1}^{N}w_iw_i^{\mathsf{T}}=\Sigma_n-\Sigma_N var ( β N ∣ y 1 : n ) = ∑ i = n + 1 N w i w i T = Σ n − Σ N . With σ n 2 = σ 2 / a n \sigma_n^2=\sigma^2/a_n σ n 2 = σ 2 / a n , this is σ 2 ( V n − V N ) \sigma^2(V_n-V_N) σ 2 ( V n − V N ) , against the exact σ 2 V n \sigma^2V_n σ 2 V n , so the omitted variance σ 2 V N \sigma^2V_N σ 2 V N is a by-product of Algorithm 2. When V 0 − 1 V_0^{-1} V 0 − 1 is negligible, V N V_N V N is roughly n / N n/N n / N times V n V_n V n . Although sharing the covariates might appear to understate uncertainty in the covariate space, we show in § 5.1 that Algorithms 1 and 2 produce approximately the same posterior distribution, with Algorithm 2 being massively faster.
The omitted part is then added back, as in the hybrid algorithm of Fong and Yiu [20 ] . By Theorem 2 applied at N N N , β ~ − β N \widetilde{\beta}-\beta_N β − β N is approximately Gaussian with covariance Σ N \Sigma_N Σ N , equal to σ 2 V N \sigma^2V_N σ 2 V N if σ n 2 = σ 2 / a n \sigma_n^2=\sigma^2/a_n σ n 2 = σ 2 / a n , and close to it if σ n = σ \sigma_n=\sigma σ n = σ (Appendix § 15 ), so the last step of Algorithms 1 and 2 adds to each draw an independent draw from N p ( 0 , σ 2 V N ) \mathcal{N}_p(0,\sigma^2V_N) N p ( 0 , σ 2 V N ) . Thus, with σ n 2 = σ 2 / a n \sigma_n^2=\sigma^2/a_n σ n 2 = σ 2 / a n , the draws have mean β n \beta_n β n and covariance σ 2 V n \sigma^2V_n σ 2 V n exactly, for any N N N . If only these two moments are of interest, they are available in closed form; predictive resampling is needed for the other features of the martingale posterior, such as quantiles, credible intervals and nonlinear functionals. We use N = 10 n N=10n N = 10 n throughout: Algorithm 1 Standard predictive resampling for RLS
Compute ( β n , V n ) (\beta_n,V_n) ( β n , V n ) from the observed data using (3)–(4)
Choose N ≫ n N \gg n N ≫ n and the number of draws B B B
for b = 1 , … , B b=1,\ldots,B b = 1 , … , B do
Set β n ( b ) = β n \beta_n^{(b)}=\beta_n β n ( b ) = β n and V n ( b ) = V n V_n^{(b)}=V_n V n ( b ) = V n
for i = n + 1 , … , N i=n+1,\ldots,N i = n + 1 , … , N do
Sample X i ( b ) ∼ P X X_i^{(b)}\sim P_X X i ( b ) ∼ P X
Sample Y i ( b ) ∣ X i ( b ) ∼ P ( ⋅ ∣ X i ( b ) , β i ( b ) , σ i ) Y_i^{(b)}\mid X_i^{(b)}\sim P(\cdot\mid X_i^{(b)},\beta_i^{(b)},\sigma_i) Y i ( b ) ∣ X i ( b ) ∼ P ( ⋅ ∣ X i ( b ) , β i ( b ) , σ i )
Update a i ( b ) = { 1 + ( X i ( b ) ) T V i − 1 ( b ) X i ( b ) } − 1 a_i^{(b)}=\{1+(X_i^{(b)})^\mathsf{T}V_{i-1}^{(b)}X_i^{(b)}\}^{-1} a i ( b ) = { 1 + ( X i ( b ) ) T V i − 1 ( b ) X i ( b ) } − 1
Update β i ( b ) \beta_i^{(b)} β i ( b ) and V i ( b ) V_i^{(b)} V i ( b ) using (3)–(4)
end for
Sample Z ( b ) ∼ N p ( 0 , I p ) Z^{(b)}\sim\mathcal{N}_p(0,I_p) Z ( b ) ∼ N p ( 0 , I p ) and set
β N ( b ) ← β N ( b ) + σ ( V N ( b ) ) 1 / 2 Z ( b ) \beta_N^{(b)}\leftarrow\beta_N^{(b)}+\sigma(V_N^{(b)})^{1/2}Z^{(b)} β N ( b ) ← β N ( b ) + σ ( V N ( b ) ) 1/2 Z ( b )
end for
return β N ( 1 ) , … , β N ( B ) \beta_N^{(1)},\ldots,\beta_N^{(B)} β N ( 1 ) , … , β N ( B )
Algorithm 2 Accelerated predictive resampling for RLS
Compute ( β n , V n ) (\beta_n,V_n) ( β n , V n ) from the observed data using (3)–(4)
Choose N ≫ n N \gg n N ≫ n and the number of draws B B B
for i = n + 1 , … , N i=n+1,\ldots,N i = n + 1 , … , N do
Sample X i ∼ P X X_i\sim P_X X i ∼ P X
Update a i = ( 1 + X i T V i − 1 X i ) − 1 a_i=(1+X_i^\mathsf{T}V_{i-1}X_i)^{-1} a i = ( 1 + X i T V i − 1 X i ) − 1
Set w i = σ a i V i − 1 X i w_i=\sigma a_iV_{i-1}X_i w i = σ a i V i − 1 X i
Update V i V_i V i using (4)
end for
for b = 1 , … , B b=1,\ldots,B b = 1 , … , B do
Sample ξ n + 1 ( b ) , … , ξ N ( b ) ∼ i i d P ξ \xi_{n+1}^{(b)},\ldots,\xi_N^{(b)}\mathrel{\overset{\mathrm{iid}}{\sim}}P_\xi ξ n + 1 ( b ) , … , ξ N ( b ) ∼ iid P ξ
Set β N ( b ) = β n + ∑ i = n + 1 N w i ξ i ( b ) \beta_N^{(b)}=\beta_n+\sum_{i=n+1}^{N}w_i\xi_i^{(b)} β N ( b ) = β n + ∑ i = n + 1 N w i ξ i ( b )
Sample Z ( b ) ∼ N p ( 0 , I p ) Z^{(b)}\sim\mathcal{N}_p(0,I_p) Z ( b ) ∼ N p ( 0 , I p ) and set
β N ( b ) ← β N ( b ) + σ V N 1 / 2 Z ( b ) \beta_N^{(b)}\leftarrow\beta_N^{(b)}+\sigma V_N^{1/2}Z^{(b)} β N ( b ) ← β N ( b ) + σ V N 1/2 Z ( b )
end for
return β N ( 1 ) , … , β N ( B ) \beta_N^{(1)},\ldots,\beta_N^{(B)} β N ( 1 ) , … , β N ( B )
in all our examples, the resulting credible intervals are close to those obtained with a much larger N N N (Appendix § 15 ).
Flexible regression extensions and hyperparameter selection
The predictive framework defined in § 3 is not limited to linear regression. It can be applied whenever the mean of the response is a linear combination of a fixed set of features of the covariates and the coefficients are estimated by penalized least squares. Each such model is specified by two choices: the features z i z_i z i , which replace x i x_i x i in (3)–(4), and the matrix V 0 V_0 V 0 , whose inverse acts as a penalty matrix or, equivalently, for which σ 2 V 0 \sigma^2V_0 σ 2 V 0 plays the role of a prior covariance matrix, as in the conjugate analogy of § 3.1 . The rest of the construction of § 3 is unchanged, and the predictive distribution can still be selected separately to reflect the response distribution. Additive models (§ 4.1 ) and mixed and Gaussian process models (§ 4.2 ) arise in very different applied contexts, but are two instances of this principle. We then discuss a data-driven approach for selecting the hyperparameters (§ 4.3 ).
Additive models
In their simplest form, additive models ([25 , 42 ] ) assume that the mean of the response variable decomposes into a sum of covariate-specific smooth univariate functions, E ( Y ∣ x ) = μ + ∑ j = 1 p f j ( x j ) \operatorname{E}(Y\mid x)=\mu+\sum_{j=1}^{p}f_j(x_j) E ( Y ∣ x ) = μ + ∑ j = 1 p f j ( x j ) , where x j x_j x j denotes the j j j th covariate, f j ( ⋅ ) f_j(\cdot) f j ( ⋅ ) the corresponding additive component, and μ ∈ R \mu\in\mathbb{R} μ ∈ R the global mean. Each f j f_j f j is commonly estimated through a spline basis expansion, that is, f j ( x ) = h j ( x ) T γ j f_j(x) = h_j(x)^{\mathsf{T}}\gamma_j f j ( x ) = h j ( x ) T γ j , with h j ( x ) ∈ R r j h_j(x) \in\mathbb{R}^{r_j} h j ( x ) ∈ R r j containing the evaluations of r j r_j r j basis functions at x x x for component j j j , and γ j ∈ R r j \gamma_j \in\mathbb{R}^{r_j} γ j ∈ R r j the corresponding vector of coefficients. Since μ \mu μ and f j f_j f j , j = 1 , … , p j = 1,\ldots,p j = 1 , … , p , are not jointly identifiable, each component is centered so that ∑ i = 1 n f j ( x i , j ) = 0 \sum_{i=1}^{n} f_j(x_{i,j}) = 0 ∑ i = 1 n f j ( x i , j ) = 0 ; the resulting basis is kept fixed for future observations. When penalized splines are employed, the estimation of β = ( μ , γ 1 T , … , γ p T ) T \beta= (\mu, \gamma_1^{\mathsf{T}},\ldots,\gamma_p^{\mathsf{T}})^{\mathsf{T}} β = ( μ , γ 1 T , … , γ p T ) T from the observations ( y 1 : n , x 1 : n ) (y_{1:n}, x_{1:n}) ( y 1 : n , x 1 : n ) reduces to the minimization of a penalized loss, whose solution is the penalized least squares estimator
β n = V n ∑ i = 1 n y i z i , with V n − 1 = Λ + ∑ i = 1 n z i z i T , (8) \beta_n = V_n \sum_{i=1}^{n} y_i z_i, \qquad\text{with} \qquad V_n^{-1} = \Lambda+ \sum_{i=1}^{n} z_i z_i^{\mathsf{T}}, \tag*{(8)} β n = V n i = 1 ∑ n y i z i , with V n − 1 = Λ + i = 1 ∑ n z i z i T , (8)
where z i = [ 1 , h 1 ( x i , 1 ) T , … , h p ( x i , p ) T ] T ∈ R r z_i = [1, h_1(x_{i,1})^{\mathsf{T}},\ldots,h_p(x_{i,p})^{\mathsf{T}}]^{\mathsf{T}} \in\mathbb{R}^{r} z i = [ 1 , h 1 ( x i , 1 ) T , … , h p ( x i , p ) T ] T ∈ R r , x i = ( x i , 1 , … , x i , p ) x_i = (x_{i,1},\ldots,x_{i,p}) x i = ( x i , 1 , … , x i , p ) , r = 1 + ∑ j = 1 p r j r = 1 + \sum_{j=1}^{p} r_j r = 1 + ∑ j = 1 p r j , and Λ \Lambda Λ is a block-diagonal matrix defined as Λ = diag ( λ μ , λ 1 Λ 1 , … , λ p Λ p ) \Lambda= \operatorname{diag}(\lambda_\mu,\lambda_1\Lambda_1,\ldots,\lambda_p\Lambda_p) Λ = diag ( λ μ , λ 1 Λ 1 , … , λ p Λ p ) , where λ μ ≥ 0 \lambda_\mu\ge0 λ μ ≥ 0 penalizes the global mean (λ μ = 0 \lambda_\mu= 0 λ μ = 0 in our implementation), λ j > 0 \lambda_j > 0 λ j > 0 are smoothing parameters and Λ j ∈ R r j × r j \Lambda_j \in\mathbb{R}^{r_j \times r_j} Λ j ∈ R r j × r j are component-specific penalty matrices, j = 1 , … , p j = 1,\ldots,p j = 1 , … , p . The penalty matrix Λ \Lambda Λ is singular in general, so Λ − 1 \Lambda^{-1} Λ − 1 may not exist and cannot be interpreted as a prior covariance matrix. We therefore set V 0 = ( Λ + ε I r ) − 1 V_0 = (\Lambda+ \varepsilon I_r)^{-1} V 0 = ( Λ + ε I r ) − 1 with a small ε > 0 \varepsilon> 0 ε > 0 (ε = 10 − 10 \varepsilon= 10^{-10} ε = 1 0 − 10 in our implementation), which recovers (8) as ε → 0 \varepsilon\to0 ε → 0 . Thus, the features are the spline evaluations z i z_i z i and V 0 V_0 V 0 is a regularized inverse of the penalty matrix: the penalized least squares estimator β n = ( μ n , γ n , 1 T , … , γ n , p T ) T \beta_n = (\mu_n,\gamma_{n,1}^{\mathsf{T}},\ldots,\gamma_{n,p}^{\mathsf{T}})^{\mathsf{T}} β n = ( μ n , γ n , 1 T , … , γ n , p T ) T in (8) is then obtained from (3) –(4) with β 0 = 0 \beta_0 = 0 β 0 = 0 and x i x_i x i replaced by z i z_i z i , and admits the Bayesian interpretation of § 3 .
Example 1 (Penalized regression splines). Low-rank thin plate regression splines ([42 ] ) are obtained with h j h_j h j the thin plate regression spline basis with r j ≪ n r_j \ll n r j ≪ n functions and [ Λ j ] ℓ , ℓ ′ = ∫ h j , ℓ ′ ′ ( x ) h j , ℓ ′ ′ ′ ( x ) d x [\Lambda_j]_{\ell,\ell'} = \int h_{j,\ell}''(x)h_{j,\ell'}''(x)\,\mathrm{d}x [ Λ j ] ℓ , ℓ ′ = ∫ h j , ℓ ′′ ( x ) h j , ℓ ′ ′′ ( x ) d x , ℓ , ℓ ′ = 1 , … , r j \ell,\ell' = 1,\ldots,r_j ℓ , ℓ ′ = 1 , … , r j .
Example 2 (P-splines). P-splines ([17 ] ) are obtained with h j h_j h j the B-splines of degree q q q on equally spaced knots and Λ j = D d j T D d j \Lambda_j = D_{d_j}^{\mathsf{T}}D_{d_j} Λ j = D d j T D d j , where D d j ∈ R ( r j − d j ) × r j D_{d_j} \in\mathbb{R}^{(r_j-d_j)\times r_j} D d j ∈ R ( r j − d j ) × r j is the d j d_j d j th order difference matrix.
Mixed models and Gaussian process regression
Suppose that the observations are divided into q q q groups, and let g i ∈ { 1 , … , q } g_i \in\{1,\ldots,q\} g i ∈ { 1 , … , q } denote the group of observation i i i . A random intercept model assumes
E ( Y i ∣ x i ) = c i T η + α g i , \mathbb{E}(Y_i \mid x_i) = c_i^{\mathsf{T}}\eta+ \alpha_{g_i}, E ( Y i ∣ x i ) = c i T η + α g i ,
where c i ∈ R d c_i \in\mathbb{R}^{d} c i ∈ R d are individual-specific covariates with effects η \eta η and α 1 , … , α q \alpha_1,\ldots,\alpha_q α 1 , … , α q are group-specific intercepts; a global intercept, or a constant in c i c_i c i , would be collinear with the group indicators. This mean is linear in β = ( η T , α 1 , … , α q ) T \beta= (\eta^{\mathsf{T}},\alpha_1,\ldots,\alpha_q)^{\mathsf{T}} β = ( η T , α 1 , … , α q ) T , with features z i = ( c i T , m i T ) T z_i = (c_i^{\mathsf{T}},m_i^{\mathsf{T}})^{\mathsf{T}} z i = ( c i T , m i T ) T , where m i ∈ { 0 , 1 } q m_i \in\{0,1\}^{q} m i ∈ { 0 , 1 } q has j j j th entry equal to 1 if g i = j g_i = j g i = j and 0 otherwise. Hence, the RLS estimator β n \beta_n β n contains the estimates of the covariate effects and of the group-specific intercepts, and the type of model is determined by the choice of V 0 = blockdiag ( D , K ) V_0 = \operatorname{blockdiag}(D, K) V 0 = blockdiag ( D , K ) , where D ∈ R d × d D \in\mathbb{R}^{d \times d} D ∈ R d × d and K ∈ R q × q K \in\mathbb{R}^{q \times q} K ∈ R q × q are positive definite and play the role of prior covariance matrices, up to σ 2 \sigma^2 σ 2 , of η \boldsymbol{\eta} η and of the group-specific intercepts, respectively. A common choice is D = diag ( v 1 , … , v d ) D = \operatorname{diag}(v_1, \ldots, v_d) D = diag ( v 1 , … , v d ) , but any positive definite D D D can be used.
Setting K = diag ( v d + 1 , … , v d + q ) K = \operatorname{diag}(v_{d+1}, \ldots, v_{d+q}) K = diag ( v d + 1 , … , v d + q ) yields a model with group-specific fixed intercepts, whereas K = τ I q + v μ 1 q 1 q T K = \tau I_q + v_\mu\mathbf{1}_q \mathbf{1}_q^\mathsf{T} K = τ I q + v μ 1 q 1 q T gives the corresponding random intercept model, with α j = μ + u j \alpha_j = \mu+ u_j α j = μ + u j for a global intercept μ \mu μ and independent group effects u j u_j u j , where 1 q \mathbf{1}_q 1 q is the vector of ones and τ \tau τ is either selected by maximizing the LOO-PLL or assigned a prior, with the prequential log-likelihood used as a likelihood; see Section 4.3. A non-diagonal K K K allows the group-specific intercepts to be correlated. In particular, when the groups are q q q fixed locations with coordinates s 1 , … , s q \boldsymbol{s}_1, \ldots, \boldsymbol{s}_q s 1 , … , s q and K K K is a kernel covariance matrix, with entries k ( s j , s ℓ ) k(s_j, s_\ell) k ( s j , s ℓ ) for a kernel function k k k , we obtain Gaussian process regression at fixed locations, and α 1 , … , α q \alpha_1, \ldots, \alpha_q α 1 , … , α q are the spatial effects.
Under a Gaussian likelihood, all these models are easily estimated within the traditional Bayesian framework. Under a non-Gaussian one, estimation can become much more expensive, whereas here the posterior mean β n \boldsymbol{\beta}_n β n is unchanged and only the predictive distribution P n P_n P n needs to be adapted.
Hyperparameter selection
As discussed in the previous sections, the RLS estimator may depend on hyperparameters contained in V 0 V_0 V 0 , while predictive resampling depends on the scale and shape parameters of P n P_n P n . It is therefore desirable to select these hyperparameters in a data-driven way. A common approach in predictive Bayes methods [19 , 20 , 3 ] is to rely on the so-called prequential log-likelihood [15 ] , ∑ i = 1 n log p i − 1 ( y i ∣ x i ) \sum_{i=1}^{n} \log p_{i-1}(y_i \mid x_i) ∑ i = 1 n log p i − 1 ( y i ∣ x i ) , which can be interpreted as a log-likelihood arising from a sequential construction. The prequential log-likelihood can then either be maximized to select the hyperparameters or, after placing a prior on them, be used as a likelihood in a fully Bayesian setting.
However, this quantity has the important drawback of depending on the ordering of the observations, so it should be averaged over several permutations to heuristically ensure invariance to the ordering. Moreover, we have observed in practice that it can be numerically unstable when maximized. Therefore, rather than averaging over permutations, which increases the computational burden, we propose using the leave-one-out prequential log-likelihood (LOO-PLL), defined as
∑ i = 1 n log p ( y i ∣ x i , β − i , σ ( i ) ) , \sum_{i=1}^{n} \log p(y_i \mid x_i, \boldsymbol{\beta}_{-i}, \sigma_{(i)}), i = 1 ∑ n log p ( y i ∣ x i , β − i , σ ( i ) ) ,
where p ( ⋅ ) p(\cdot) p ( ⋅ ) denotes the density function of the distribution in P \mathcal{P} P , β − i \boldsymbol{\beta}_{-i} β − i is the estimate of the regression coefficients based on all observations except the i i i th, and σ ( i ) \sigma_{(i)} σ ( i ) is the predictive scale of the i i i th observation when it comes last, e.g. σ ( i ) 2 = σ 2 / ( 1 − x i T V n x i ) \sigma_{(i)}^2 = \sigma^2/(1 - x_i^\mathsf{T}V_nx_i) σ ( i ) 2 = σ 2 / ( 1 − x i T V n x i ) when σ n 2 = σ 2 / a n \sigma_n^2 = \sigma^2/a_n σ n 2 = σ 2 / a n . Up to a factor n n n , this is equivalent to evaluating the prequential log-likelihood contribution of the last observation based on the coefficients estimated from the other n − 1 n-1 n − 1 observations and averaging it over all possible permutations of the data, since each observation comes last in ( n − 1 ) ! (n-1)! ( n − 1 )! of the n ! n! n ! permutations and, by Remark 1, β − i \boldsymbol{\beta}_{-i} β − i does not depend on the ordering of the other observations. An analogous identity relates the marginal likelihood to exhaustive cross-validation [18 ] . The LOO-PLL is therefore invariant to the ordering of the observations. It can also be computed efficiently, since the leave-one-out predictions satisfy x i T β − i = y i − ( y i − x i T β n ) / ( 1 − x i T V n x i ) x_i^\mathsf{T}\boldsymbol{\beta}_{-i}=y_i-(y_i-x_i^\mathsf{T}\boldsymbol{\beta}_n)/(1-x_i^\mathsf{T}V_nx_i) x i T β − i = y i − ( y i − x i T β n ) / ( 1 − x i T V n x i ) for i = 1 , … , n i=1,\ldots,n i = 1 , … , n , so β n \boldsymbol{\beta}_n β n and V n V_n V n need to be computed only once. The LOO-PLL can thus replace the standard prequential log-likelihood for hyperparameter selection, and in our experience it performs better in practice.
Simulation study
Comparison between uncertainty quantification methods
We first check predictive resampling in a linear regression setting in which the exact posterior is available. The response is Gaussian with known standard deviation σ = 2 \sigma=2 σ = 2 , the p = 50 p=50 p = 50 covariates are standard multivariate normal, and we consider two scenarios, n = 3 p n=3p n = 3 p and n = 10 p n=10p n = 10 p , each with 30 simulated datasets; the true coefficients are drawn once from N p ( 0 , 4 I p ) \mathcal{N}_p(0,4I_p) N p ( 0 , 4 I p ) . We set β 0 = 0 \boldsymbol{\beta}_0=0 β 0 = 0 , V 0 = 1000 I p V_0=1000I_p V 0 = 1000 I p , P ^ X \widehat{P}_X P X equal to the empirical distribution of the observed covariates, B = 10,000 B=10{,}000 B = 10 , 000 and N = 10 n N=10n N = 10 n , and consider both σ n 2 = σ 2 / a n \sigma_n^2=\sigma^2/a_n σ n 2 = σ 2 / a n and σ n 2 = σ 2 \sigma_n^2=\sigma^2 σ n 2 = σ 2 . For each choice, we compare Algorithm 1 (PS-1) and Algorithm 2 (PS-2) with B B B independent draws from the exact posterior N p ( β n , Σ n ) \mathcal{N}_p(\boldsymbol{\beta}_n,\boldsymbol{\Sigma}_n) N p ( β n , Σ n ) of Theorem 3 (EXACT), where, if σ n 2 = σ 2 \sigma_n^2=\sigma^2 σ n 2 = σ 2 , Σ n \boldsymbol{\Sigma}_n Σ n is computed given a sequence of future covariates drawn from P ^ X \widehat{P}_X P X , whose first N − n N-n N − n elements are those of PS-2. We report the errors of the sample mean β ^ \widehat{\boldsymbol{\beta}} β and covariance Σ ^ \widehat{\boldsymbol{\Sigma}} Σ of the draws, the average width and the frequentist coverage of the 95% credible intervals for the coefficients, and the computing time on an Apple M2 chip with 8 cores, with an R/C++ implementation. An analogous study for heavy-tailed data is reported in Appendix § 12 .§ 1 .
The results are reported in Table 1 . With the correction, the draws of both algorithms have exact mean and, if σ n 2 = σ 2 / a n \sigma_n^2=\sigma^2/a_n σ n 2 = σ 2 / a n , exact covariance, so this experiment mainly checks the implementation: as expected, the errors of PS-1 and PS-2 are comparable with the Monte Carlo errors of EXACT, and the widths and coverages essentially coincide with the exact ones. Less obviously, PS-2 performs as PS-1 also when σ n 2 = σ 2 \sigma_n^2=\sigma^2 σ n 2 = σ 2 , where Σ n \boldsymbol{\Sigma}_n Σ n depends on the future covariates: sharing them across paths has a negligible effect. With σ n 2 = σ 2 / a n \sigma_n^2=\sigma^2/a_n σ n 2 = σ 2 / a n , the coverage is close to the nominal level, as expected since the martingale posterior then coincides with the posterior of the conjugate model; with σ n 2 = σ 2 \sigma_n^2=\sigma^2 σ n 2 = σ 2 , the intervals are narrower, in agreement with the bound Σ n ⪯ σ 2 V n \boldsymbol{\Sigma}_n\preceq\sigma^2V_n Σ n ⪯ σ 2 V n of § 3.2 , and the coverage is slightly below the nominal level. The correction is not negligible: the omitted part of the posterior variance is about n / N = 10 % n/N=10\% n / N = 10% , and without it the intervals would be about 5% narrower. Finally, PS-2 is about 80 times faster than PS-1, and is therefore preferable in practice.
Table 1. Comparison of uncertainty quantification from predictive resampling, with Algorithms 1 (PS-1) and 2 (PS-2), with B B B draws from the exact posterior N p ( β n , Σ n ) \mathcal{N}_p(\beta_n,\Sigma_n) N p ( β n , Σ n ) (EXACT): logarithm of the mean squared error of the posterior mean, log ( ∥ β ^ − β n ∥ 2 / p ) \log(\lVert\hat{\beta}-\beta_n\rVert^2/p) log (∥ β ^ − β n ∥ 2 / p ) , logarithm of the Frobenius norm of the lower-triangular part of σ − 2 ( Σ ^ − Σ n ) \sigma^{-2}(\hat{\Sigma}-\Sigma_n) σ − 2 ( Σ ^ − Σ n ) , average width and coverage of the 95% intervals for the p p p regression coefficients, and computation time in seconds. For EXACT, the errors are Monte Carlo errors and the intervals are the exact Gaussian ones. Values are averages over the simulated datasets, with standard deviations in parentheses.
Misspecification of the predictive distribution
We now illustrate how the choice of P n P_n P n affects the martingale posterior of the regression coefficients, through two simulated examples with n = 15 n=15 n = 15 observations and p = 5 p=5 p = 5 covariates. We generate y i = x i T β 0 ∗ + ϕ 0 ξ i y_i=x_i^{\mathsf{T}}\beta_0^*+\phi_0\xi_i y i = x i T β 0 ∗ + ϕ 0 ξ i , i = 1 , … , n i=1,\ldots,n i = 1 , … , n , with x i x_i x i iid from N p ( 0 , I p ) \mathcal{N}_p(0,I_p) N p ( 0 , I p ) , β 0 ∗ = ( − 1.54 , 1.81 , − 2.21 , − 0.22 , − 3.48 ) T \beta_0^*=(-1.54,1.81,-2.21,-0.22,-3.48)^{\mathsf{T}} β 0 ∗ = ( − 1.54 , 1.81 , − 2.21 , − 0.22 , − 3.48 ) T and ϕ 0 = 0.2 \phi_0=0.2 ϕ 0 = 0.2 . In the first example, the ξ i \xi_i ξ i are iid from a Student-t t t distribution with ν 0 = 3 \nu_0=3 ν 0 = 3 degrees of freedom; in the second, they are iid from a skew Student-t t t distribution (Azzalini and Capitanio, [5 ] ) with ν 0 = 3 \nu_0=3 ν 0 = 3 and skewness parameter α 0 = 3 \alpha_0=3 α 0 = 3 , centered to have zero mean. In each example, we compare a P n P_n P n that matches the error distribution, namely a location-scale Student-t t t and a skew Student-t t t centered to have mean x T β x^{\mathsf{T}}\beta x T β , respectively, with a Gaussian one, which, with σ n 2 = σ 2 / a n \sigma_n^2=\sigma^2/a_n σ n 2 = σ 2 / a n , coincides with the conjugate Bayesian linear model. We set β 0 = 0 \beta_0=0 β 0 = 0 , V 0 = 1000 I p V_0=1000I_p V 0 = 1000 I p and σ n 2 = σ 2 / a n \sigma_n^2=\sigma^2/a_n σ n 2 = σ 2 / a n , select hyperparameters by maximizing the LOO-PLL, and run Algorithm 2 with N = 10 n N = 10n N = 10 n , P ^ X \widehat{P}_X P X equal to the empirical distribution of the observed covariates, and B = 10,000 B = 10{,}000 B = 10 , 000 . The LOO-PLL estimates are ϕ ^ = 0.21 \widehat{\phi} = 0.21 ϕ = 0.21 and ν ^ = 2.60 \widehat{\nu} = 2.60 ν = 2.60 for the Student-t t t P n P_n P n in the first example, and ϕ ^ = 0.26 \widehat{\phi} = 0.26 ϕ = 0.26 , ν ^ = 4.07 \widehat{\nu} = 4.07 ν = 4.07 and α ^ = 3.80 \widehat{\alpha} = 3.80 α = 3.80 for the skew Student-t t t P n P_n P n in the second; in the first example, the Gaussian P n P_n P n inflates the scale, ϕ ^ = 0.33 \widehat{\phi} = 0.33 ϕ = 0.33 , to match the error variance.
Figure 1 shows the posterior distribution of the fourth regression coefficient. With a matching P n P_n P n , the heavy tails of the errors, and in the second example also their skewness, carry over to the martingale posterior, which is more dispersed than, and in the second example skewed relative to, the posterior of the Gaussian linear model. The choice of P n P_n P n thus affects not only predictive quantities, such as the quantiles of Y ∣ x Y \mid x Y ∣ x , but also inference on the regression coefficients.
Figure 1. Posterior distribution of a regression coefficient obtained with a P n P_n P n matching the error distribution and with a Gaussian P n P_n P n , for Student-t t t errors (left) and skew Student-t t t errors (right).
Additive models with Student-t predictive distribution
We now compare our approach with well-established Bayesian methods in a nonlinear regression setting with heavy-tailed errors. We generate y i = μ + f ( x i ) + ϕ 0 ξ i y_i = \mu+ f(x_i) + \phi_0 \xi_i y i = μ + f ( x i ) + ϕ 0 ξ i , i = 1 , … , n i = 1, \ldots, n i = 1 , … , n , with covariates iid uniform on ( − 2 , 2 ) (-2, 2) ( − 2 , 2 ) , ξ i \xi_i ξ i iid from a Student-t t t distribution with ν 0 = 5 \nu_0 = 5 ν 0 = 5 degrees of freedom, μ = 1 \mu= 1 μ = 1 , ϕ 0 = 1 \phi_0 = 1 ϕ 0 = 1 and f ( x i ) = ∑ j = 1 p f j ( x i , j ) f(x_i) = \sum_{j=1}^{p} f_j(x_{i,j}) f ( x i ) = ∑ j = 1 p f j ( x i , j ) , where f j = f k ∗ f_j = f_k^* f j = f k ∗ , with k = 1 + { ( j − 1 ) m o d 4 } k = 1 + \{(j - 1) \mathbin{\mathrm{mod}} 4\} k = 1 + {( j − 1 ) mod 4 } , is centered over the sample and
f 1 ∗ ( x ) = 1 2 x 3 − x , f 2 ∗ ( x ) = 3 x exp { − x 2 } , f 3 ∗ ( x ) = x 2 − 1 , f 4 ∗ ( x ) = 2 sin ( π x ) . f_1^*(x) = \frac{1}{2}x^3 - x,\quad f_2^*(x) = 3x\exp\{-x^2\},\quad f_3^*(x) = x^2 - 1,\quad f_4^*(x) = 2\sin(\pi x). f 1 ∗ ( x ) = 2 1 x 3 − x , f 2 ∗ ( x ) = 3 x exp { − x 2 } , f 3 ∗ ( x ) = x 2 − 1 , f 4 ∗ ( x ) = 2 sin ( π x ) .
We set p = 15 p = 15 p = 15 and consider two scenarios, n = 30 p n = 30p n = 30 p and n = 100 p n = 100p n = 100 p , each with 30 simulated datasets. On a test set of size 10,000, we compute the mean squared error (MSE) of the posterior mean of f ( x ) f(x) f ( x ) and the average coverage and width of the 95% credible intervals for f ( x ) f(x) f ( x ) , where the coverage at a test point is the proportion of datasets whose interval contains the true value; we do the same for Q 0.90 ( x ) Q_{0.90}(x) Q 0.90 ( x ) , the 0.90 quantile of the response at x x x . We also report the computing time on an Apple M2 chip with 8 cores, for the posterior mean alone, including the selection of the hyperparameters, and in total.
Our method (RLS-t t t ) uses the penalized regression splines of § 4.1 with a location-scale Student-t t t P n P_n P n and σ n 2 = σ 2 / a n \sigma_n^2 = \sigma^2/a_n σ n 2 = σ 2 / a n , where ν > 2 \nu> 2 ν > 2 and σ 2 = ϕ 2 ν ( ν − 2 ) − 1 \sigma^2 = \phi^2\nu(\nu- 2)^{-1} σ 2 = ϕ 2 ν ( ν − 2 ) − 1 ; ϕ \phi ϕ , ν \nu ν and the smoothing parameters are selected by maximizing the LOO-PLL, and Algorithm 2 is run with N = 10 n N = 10n N = 10 n and B = 5,000 B = 5{,}000 B = 5 , 000 . We also include its Gaussian version (RLS-n n n ). The first competitor is the traditional Bayesian counterpart of our model, with a Student-t t t likelihood (BLM). The coefficients have a Gaussian prior with block-diagonal precision matrix, whose blocks are the spline penalty matrices multiplied by a smoothing parameter and divided by ϕ 2 \phi^2 ϕ 2 , improper along the null space of each penalty, and each smoothing parameter has a Gamma(2, 0.5) prior, with shape 2 and rate 0.5. We sample from its posterior with the R package brms, based on STAN, running five chains of 2,000 draws and discarding the first 1,000 as burn-in; the sampler reports no divergent transitions. The second competitor is Bayesian additive regression trees (BART; [12 ] ), with 55,000 draws, of which the first 50,000 are discarded as burn-in. The MCMC chain lengths are chosen, when possible, to give an effective sample size (ESS) close to B B B (Appendix § 12 .§ 3 ), and the times exclude the compilation of the STAN model.
The results are reported in Table 2 . BLM yields the most accurate point estimates, except for Q 0.90 ( x ) Q_{0.90}(x) Q 0.90 ( x ) when n = 30 p n = 30p n = 30 p , where RLS-t t t is the most accurate; overall, RLS-t t t is close to BLM and much more accurate than BART. This is expected, since the true f f f is additive, whereas BART also accommodates interactions among the covariates. The point estimates of RLS-t t t and RLS-n n n differ only through their estimated smoothing parameters. The coverage of RLS-t t t is slightly below the nominal level, and that of BLM slightly above it, except for Q 0.90 ( x ) Q_{0.90}(x) Q 0.90 ( x ) when n = 30 p n = 30p n = 30 p ; RLS-n n n has lower coverage for Q 0.90 ( x ) Q_{0.90}(x) Q 0.90 ( x ) , which depends more heavily on the tails of P n P_n P n , and BART attains coverage at or above the nominal level with intervals more than twice as wide. Finally, our posterior mean requires no posterior sampling, whereas the competitors must run the full MCMC to obtain it, so for them the two times in Table 2 coincide. In both scenarios, RLS-t t t and RLS-n n n are more than an order of magnitude faster than the competitors, both for the posterior mean and in total time. Appendix B.2 compares our method also with frequentist additive models fitted with the R package mgcv ([41 ] ): the Student-t t t additive model gives more accurate point estimates, while our method is faster, even including predictive resampling.
Table 2. Comparison between our proposed methods, with Student-t t t (RLS-t t t ) and Gaussian (RLS-n n n ) predictive distributions, the traditional Bayesian linear model with Student-t t t likelihood (BLM) and BART in terms of out-of-sample MSE, average coverage and width of credible intervals for f ( x ) f(x) f ( x ) and Q 0.90 ( x ) Q_{0.90}(x) Q 0.90 ( x ) , and time for obtaining the posterior mean (PM) and total, respectively. Values are averages over the simulated datasets, with standard deviations in parentheses.
Analysis of NHANES serum triglycerides dataset
We analyze fasting serum triglyceride concentrations (mg/dL) from the 2021–2023 cycle of the National Health and Nutrition Examination Survey (NHANES), focusing on US-born males aged 12 to 40 years tested between May and October. NHANES is a stratified, multistage probability sample of the civilian population of the United States. Triglycerides are the circulating storage form of dietary and hepatically synthesized fat, transported in chylomicrons and very-low-density lipoproteins (VLDL). Monitoring the level of plasma triglycerides is particularly important because it is a risk-enhancing factor for cardiovascular diseases ([27 , 35 ] ). Since both triglycerides and glucose respond to recent food intake, the analysis is restricted to the 119 participants who fasted between 12 and 24 hours before the examination. For each participant, the dataset records eight characteristics: age (years), body mass index (BMI), waist circumference, high-density lipoprotein (HDL) cholesterol, fasting plasma glucose, average systolic blood pressure (SBP), the ratio of family income to the federal poverty threshold, and ethnicity (Mexican American, other Hispanic, non-Hispanic white, non-Hispanic black, non-Hispanic Asian, and other or multiracial). Modeling the relation between these covariates and triglyceride concentration is of practical interest: BMI, HDL cholesterol, and fasting glucose are all proxies for insulin resistance, the principal driver of hypertriglyceridemia ([36 ] ).
Several relations between the covariates and the logarithm of triglyceride concentration are expected to be nonlinear, thereby motivating modeling the mean through additive smooth components. Specifically, we include the covariates age, BMI, HDL, plasma glucose, average SBP, and income level via penalized regression splines, while ethnicity is encoded with five dummy indicators, with non-Hispanic white as the baseline. We exclude waist circumference from the analysis because it is highly correlated with BMI (correlation 0.96 in the training set). A location-scale Student-t t t predictive distribution is a natural choice for modeling log-triglyceride concentration, since the distribution is typically heavy-tailed [26 ] . Additional details on the choice of P n P_n P n are contained in Appendix C.
We compare the results obtained with our approach against the same model fitted under the traditional Bayesian approach with the R package brms. We randomly split the dataset into a training set of 83 observations and a test set of 36 observations, used to estimate the models and to evaluate out-of-sample predictive accuracy, respectively. Out-of-sample accuracy is evaluated through the MSE between the posterior mean of E ( Y ∣ x ) \mathrm{E}(Y \mid x) E ( Y ∣ x ) and the corresponding observed responses on the test set. Importantly, we also assess the computational efficiency of the posterior sampling algorithm. For the proposed method, we set σ n 2 = σ 2 / a n \sigma_n^2 = \sigma^2/a_n σ n 2 = σ 2 / a n , as in § 5.3 , leave the coefficients of the ethnicity indicators unpenalized, as the global mean, and select the degrees of freedom ν \nu ν , the scale parameter ϕ \phi ϕ , and the smoothing parameters by maximizing the LOO-PLL, obtaining ν = 13.85 \nu= 13.85 ν = 13.85 and ϕ = 0.36 \phi= 0.36 ϕ = 0.36 . Uncertainty quantification is carried out via Algorithm 2 with B = 10,000 B = 10{,}000 B = 10 , 000 and N = 10 n N = 10n N = 10 n . For the traditional Bayesian model, we use the prior of § 5.3 , with a Gamma( 2 , 0.5 ) (2, 0.5) ( 2 , 0.5 ) prior on each smoothing parameter, and we run five MCMC chains of 3,000 iterations each and discard the first 1,000 as burn-in; the target acceptance rate of STAN is raised from 0.8 to 0.95, since the default yields divergent transitions, also with a longer burn-in.
Our approach obtains an accuracy similar to that of the traditional Bayesian additive model with Student-t t t likelihood (MSE of 0.263 versus 0.249). Selecting the hyperparameters and computing the posterior mean take 0.02 seconds and predictive resampling takes 0.09 seconds, whereas MCMC requires 6.68 seconds. In addition, predictive resampling yields posterior draws that are independent by construction, given the shared future covariates, whereas the average ESS of the MCMC draws across regression parameters is 5,424 out of 10,000. The traditional Bayesian model is, however, sensitive to the prior on the smoothing parameters: with a vague Gamma( 0.001 , 0.001 ) (0.001, 0.001) ( 0.001 , 0.001 ) prior, the sampler reports 235 divergent transitions out of 10,000 draws, the average ESS drops to 1,266, and MCMC requires 39.21 seconds. Our approach requires no such prior, since its hyperparameters are selected by maximizing the LOO-PLL.
The results obtained with our predictive additive model are shown in Figure 2 . The relation between HDL and log-triglycerides is clearly nonlinear. The inferred shape is consistent with a well-documented saturating exchange process [31 , 6 ] . The effect of age on log-triglycerides is also nonlinear: it decreases up to about age 30 and increases afterwards. For fasting glucose, BMI and SBP, the estimated effect is linear and positive, which is consistent with the literature [33 , 1 ] . The remaining covariates, including ethnicity, show no clear association with log-triglyceride concentration, as their 95% credible bands and intervals include zero. In the analyzed sample, age, BMI, HDL cholesterol, fasting glucose and SBP are thus associated with log-triglycerides; the analysis does not use the NHANES survey weights.
Figure 2. Estimated covariate-specific effects with pointwise 95% highest posterior density bands, obtained using our predictive additive model, and partial residuals. The bands are truncated to the range of the partial residuals.
Discussion
Recursive least squares is a popular estimator in the frequentist literature for streaming data. We have proposed a novel interpretation within a predictive Bayes framework, expressing it as a predictive rule that characterizes an implicit prior on the regression coefficients of a linear model. The proposed method addresses some limitations of existing predictive Bayes approaches: it provides a new method for regression and, despite assuming that observations are generated sequentially, it is invariant to their ordering. While in traditional Bayesian models the choice of likelihood and prior is often restricted to computationally convenient combinations, our approach offers greater flexibility, since the posterior mean β n \beta_n β n always coincides with the one obtained under a Gaussian–Gaussian model whereas the plug-in predictive distribution P n P_n P n may be chosen separately. The approach extends to nonlinear additive models, mixed models, and Gaussian process regression through an appropriate choice of the initial V 0 V_0 V 0 .
Some limitations remain. First, the approach does not generalize to the full class of generalized linear models, as no closed-form least squares estimator is available in that setting; for logistic regression, one possibility could be to adopt the construction of Fortini and Petrone [24 ] and study how extensions beyond the simple regression case can be accommodated. Second, β n \beta_n β n is the generalized ridge estimator for every choice of P ξ P_{\xi} P ξ : a heavy-tailed P n P_n P n changes the spread and shape of the posterior, but not the point estimate, which is therefore not robust to outliers. Third, our asymptotic results assume a fixed number of covariates p p p and a positive definite Ω \Omega Ω ; when p p p is large relative to n n n , the construction still applies, since V 0 V_0 V 0 is positive definite and β n \beta_n β n is a ridge-type shrinkage estimator, and with σ n 2 = σ 2 / a n \sigma_n^2 = \sigma^2/a_n σ n 2 = σ 2 / a n the posterior covariance σ 2 V n \sigma^2 V_n σ 2 V n is exact also when n < p n < p n < p , but regimes in which p p p grows with n n n are not covered by our theory. Fourth, the hyperparameters selected by the LOO-PLL are fixed in predictive resampling, so their uncertainty is not propagated, as shown by the equal widths for f ( x ) f(x) f ( x ) and Q 0.90 ( x ) Q_{0.90}(x) Q 0.90 ( x ) in Table 2 ; a prior on them, as in § 4.3 , would address this. Finally, the predictive scales are required to converge to a constant σ \sigma σ , so that the limiting model is homoscedastic; allowing a covariate-dependent limit would accommodate heteroscedastic responses, and we leave this extension to future work.
While the proposed method represents a substantial first step towards predictive Bayes methods for regression, an interesting direction for future work is to develop updates for a general nonparametric f n ( x ) f_n(x) f n ( x ) that does not assume linearity or additivity. In this direction, Ng et al. (2025); Fortini et al. [21 ] have recently provided a predictive Bayes interpretation of TabPFN, a popular machine learning model for tabular data. Although several candidates appear to yield convergent predictive rules, how to establish the martingale condition, or asymptotic exchangeability, for a general nonparametric estimator remains unclear. Predictive Bayes methods for multivariate responses, allowing inference on their dependence structure, are a further ambitious direction. More broadly, establishing theoretical guarantees for more general and flexible updates that retain computational scalability is an important next step.
Data availability
The NHANES data analyzed in § 6 are publicly available from the website of the US National Center for Health Statistics (www.cdc.gov/nchs/nhanes). Code to download the data and to reproduce all the analyses is available in the repository github.com/davideagno/BayesianRLS.
Acknowledgments
This research was partially supported by the National Institutes of Health (grant ID R01ES035625), by the Office of Naval Research (N000142412626), and by NSF award #2426762. The authors used Claude (Anthropic), an AI assistant, to help with language editing, with checking the consistency of the text and proofs, and with revising and rerunning the code. The authors reviewed all changes and take full responsibility for the content of the paper.
Appendix
The Appendix contains the proofs of all theorems and propositions, additional simulation studies, additional results for the analysis of § 6 , and details on the computations. Code to replicate our analyses is available as described in the Data availability section.
Proofs
This section collects the proofs of the results in § 3 of the paper. We first prove three preliminary lemmas on the matrices V n \mathbf{V}_n V n , and then prove the results in the order in which they appear in the paper. Equation numbers without a letter prefix refer to the paper.
Preliminary results
Lemma A.1. For every u ∈ R p u \in\mathbb{R}^p u ∈ R p and any n ≥ 1 n \ge1 n ≥ 1 , u T V n u ≤ u T V n − 1 u u^{\mathsf{T}}\mathbf{V}_n u \le u^{\mathsf{T}}\mathbf{V}_{n-1}u u T V n u ≤ u T V n − 1 u .
Proof. The lemma states that the matrices V n \mathbf{V}_n V n can only decrease as new observations arrive. By (4) , we can write V n − 1 − V n = a n V n − 1 x n x n T V n − 1 \mathbf{V}_{n-1}-\mathbf{V}_n=a_n\mathbf{V}_{n-1}x_nx_n^{\mathsf{T}}\mathbf{V}_{n-1} V n − 1 − V n = a n V n − 1 x n x n T V n − 1 , and therefore
u T ( V n − 1 − V n ) u = a n u T V n − 1 x n x n T V n − 1 u = a n ( u T V n − 1 x n ) 2 ≥ 0 , u^{\mathsf{T}}(\mathbf{V}_{n-1}-\mathbf{V}_n)u=a_nu^{\mathsf{T}}\mathbf{V}_{n-1}x_nx_n^{\mathsf{T}}\mathbf{V}_{n-1}u=a_n(u^{\mathsf{T}}\mathbf{V}_{n-1}x_n)^2\ge0, u T ( V n − 1 − V n ) u = a n u T V n − 1 x n x n T V n − 1 u = a n ( u T V n − 1 x n ) 2 ≥ 0 ,
since a n > 0 a_n>0 a n > 0 . Rearranging the terms, u T V n u ≤ u T V n − 1 u u^{\mathsf{T}}\mathbf{V}_n u \le u^{\mathsf{T}}\mathbf{V}_{n-1}u u T V n u ≤ u T V n − 1 u .
Lemma A.2. Assume that the sequence of observed covariates x 1 : ∞ x_{1:\infty} x 1 : ∞ is known and assume that Ω : = lim n → ∞ n − 1 ∑ i = 1 n x i x i T \boldsymbol{\Omega}:=\lim_{n\to\infty}n^{-1}\sum_{i=1}^{n}x_ix_i^{\mathsf{T}} Ω := lim n → ∞ n − 1 ∑ i = 1 n x i x i T exists and is positive definite. Then, x n T V n − 1 x n → 0 x_n^{\mathsf{T}}\mathbf{V}_{n-1}x_n\to0 x n T V n − 1 x n → 0 as n → ∞ n\to\infty n → ∞ .
Proof. We show that the largest eigenvalue of V n \mathbf{V}_n V n decreases at rate n − 1 n^{-1} n − 1 , whereas ∥ x n ∥ 2 \lVert x_n\rVert^2 ∥ x n ∥ 2 grows more slowly than n n n . Since V n \mathbf{V}_n V n is positive definite, for any u ∈ R p u\in\mathbb{R}^p u ∈ R p , u T V n u ≤ λ max ( V n ) ∥ u ∥ 2 u^{\mathsf{T}}\mathbf{V}_n u\le\lambda_{\max}(\mathbf{V}_n)\lVert u\rVert^2 u T V n u ≤ λ m a x ( V n ) ∥ u ∥ 2 , where λ max ( V n ) \lambda_{\max}(\mathbf{V}_n) λ m a x ( V n ) denotes the maximum eigenvalue of the matrix V n \mathbf{V}_n V n . Let λ ∗ = ζ λ min ( Ω ) \lambda_*=\zeta\lambda_{\min}(\boldsymbol{\Omega}) λ ∗ = ζ λ m i n ( Ω ) , with ζ ∈ ( 0 , 1 ) \zeta\in(0,1) ζ ∈ ( 0 , 1 ) . Since by assumption n − 1 ∑ i = 1 n x i x i T → Ω n^{-1}\sum_{i=1}^{n}x_ix_i^{\mathsf{T}}\to\boldsymbol{\Omega} n − 1 ∑ i = 1 n x i x i T → Ω as n → ∞ n\to\infty n → ∞ , with Ω \boldsymbol{\Omega} Ω positive definite, there exists n 0 n_0 n 0 such that, for all n ≥ n 0 n\ge n_0 n ≥ n 0 and all u ∈ R p u\in\mathbb{R}^p u ∈ R p ,
u T ( n − 1 ∑ i = 1 n x i x i T ) u ≥ u T λ ∗ I p u , u^{\mathsf{T}}\left(n^{-1}\sum_{i=1}^{n}x_ix_i^{\mathsf{T}}\right)u\ge u^{\mathsf{T}}\lambda_*\mathbf{I}_p u, u T ( n − 1 i = 1 ∑ n x i x i T ) u ≥ u T λ ∗ I p u ,
and therefore, multiplying by n n n both sides,
u T ( ∑ i = 1 n x i x i T ) u ≥ u T n λ ∗ I p u . u^{\mathsf{T}}\left(\sum_{i=1}^{n}x_ix_i^{\mathsf{T}}\right)u\ge u^{\mathsf{T}}n\lambda_*\mathbf{I}_p u. u T ( i = 1 ∑ n x i x i T ) u ≥ u T n λ ∗ I p u .
Then, by (5) and since u T V 0 − 1 u > 0 u^{\mathsf{T}}\mathbf{V}_0^{-1}u>0 u T V 0 − 1 u > 0 ,
u T V n − 1 u = u T V 0 − 1 u + u T ( ∑ i = 1 n x i x i T ) u > u T ( ∑ i = 1 n x i x i T ) u ≥ u T n λ ∗ I p u , u^{\mathsf{T}}\mathbf{V}_n^{-1}u=u^{\mathsf{T}}\mathbf{V}_0^{-1}u+u^{\mathsf{T}}\left(\sum_{i=1}^{n}x_ix_i^{\mathsf{T}}\right)u>u^{\mathsf{T}}\left(\sum_{i=1}^{n}x_ix_i^{\mathsf{T}}\right)u\ge u^{\mathsf{T}}n\lambda_*\mathbf{I}_p u, u T V n − 1 u = u T V 0 − 1 u + u T ( i = 1 ∑ n x i x i T ) u > u T ( i = 1 ∑ n x i x i T ) u ≥ u T n λ ∗ I p u ,
that is, all the eigenvalues of V n − 1 \mathbf{V}_n^{-1} V n − 1 are larger than n λ ∗ n\lambda_* n λ ∗ . Therefore, for the inverse matrix V n \mathbf{V}_n V n , whose eigenvalues are the reciprocals of those of V n − 1 \mathbf{V}_n^{-1} V n − 1 , all the eigenvalues are smaller than n − 1 λ ∗ − 1 n^{-1}\lambda_*^{-1} n − 1 λ ∗ − 1 , that is, u T V n u ≤ u T n − 1 λ ∗ − 1 I p u u^{\mathsf{T}}\mathbf{V}_n u\le u^{\mathsf{T}}n^{-1}\lambda_*^{-1}\mathbf{I}_p u u T V n u ≤ u T n − 1 λ ∗ − 1 I p u for all u ∈ R p u\in\mathbb{R}^p u ∈ R p , and λ max ( V n ) ≤ n − 1 λ ∗ − 1 \lambda_{\max}(\mathbf{V}_n)\le n^{-1}\lambda_*^{-1} λ m a x ( V n ) ≤ n − 1 λ ∗ − 1 . Then, as n → ∞ n\to\infty n → ∞ ,
0 ≤ x n T V n − 1 x n ≤ λ max ( V n − 1 ) ∥ x n ∥ 2 ≤ λ ∗ − 1 ( n − 1 ) − 1 ∥ x n ∥ 2 ⟶ 0 , (9) 0\le x_n^{\mathsf{T}}\mathbf{V}_{n-1}x_n\le\lambda_{\max}(\mathbf{V}_{n-1})\lVert x_n\rVert^2\le\lambda_*^{-1}(n-1)^{-1}\lVert x_n\rVert^2\longrightarrow0, \tag*{(9)} 0 ≤ x n T V n − 1 x n ≤ λ m a x ( V n − 1 ) ∥ x n ∥ 2 ≤ λ ∗ − 1 ( n − 1 ) − 1 ∥ x n ∥ 2 ⟶ 0 , (9)
since n − 1 ∥ x n ∥ 2 → 0 n^{-1}\lVert x_n\rVert^2 \to0 n − 1 ∥ x n ∥ 2 → 0 . This last fact follows from
n − 1 x n x n ⊤ = n − 1 ∑ i = 1 n x i x i ⊤ − n − 1 n { ( n − 1 ) − 1 ∑ i = 1 n − 1 x i x i ⊤ } → Ω − Ω = 0 , n^{-1}x_nx_n^\top=n^{-1}\sum_{i=1}^{n}x_ix_i^\top-\frac{n-1}{n}\left\{(n-1)^{-1}\sum_{i=1}^{n-1}x_ix_i^\top\right\}\to\Omega-\Omega=0, n − 1 x n x n ⊤ = n − 1 i = 1 ∑ n x i x i ⊤ − n n − 1 { ( n − 1 ) − 1 i = 1 ∑ n − 1 x i x i ⊤ } → Ω − Ω = 0 ,
whose trace is n − 1 ∥ x n ∥ 2 n^{-1}\lVert x_n\rVert^2 n − 1 ∥ x n ∥ 2 .
Lemma A.3. Under Assumption 3, let Σ n \Sigma_n Σ n be as in Proposition 2 of the paper, and let ϵ n : = sup i > n x i ⊤ V i − 1 x i \epsilon_n := \sup_{i>n} x_i^\top V_{i-1}x_i ϵ n := sup i > n x i ⊤ V i − 1 x i , s n − : = inf i > n σ i 2 s_n^- := \inf_{i>n}\sigma_i^2 s n − := inf i > n σ i 2 and s n + : = sup i > n σ i 2 s_n^+ := \sup_{i>n}\sigma_i^2 s n + := sup i > n σ i 2 . Then, for every n ≥ 0 n \ge0 n ≥ 0 , Σ n \Sigma_n Σ n is positive definite and
s n − ( 1 − ϵ n ) V n ⪯ Σ n ⪯ s n + V n . s_n^-(1-\epsilon_n)V_n \preceq\Sigma_n \preceq s_n^+V_n. s n − ( 1 − ϵ n ) V n ⪯ Σ n ⪯ s n + V n . Here, for symmetric matrices A \mathbf{A} A and B \mathbf{B} B , A ⪯ B \mathbf{A} \preceq\mathbf{B} A ⪯ B means that u ⊤ A u ≤ u ⊤ B u \mathbf{u}^\top\mathbf{A}\mathbf{u} \le\mathbf{u}^\top\mathbf{B}\mathbf{u} u ⊤ Au ≤ u ⊤ Bu for all u ∈ R p \mathbf{u} \in\mathbb{R}^p u ∈ R p , that is, B − A \mathbf{B}-\mathbf{A} B − A is positive semidefinite.
Proof. Recall that Σ n = ∑ i > n σ i 2 a i ( V i − 1 − V i ) \Sigma_n=\sum_{i>n}\sigma_i^2a_i(V_{i-1}-V_i) Σ n = ∑ i > n σ i 2 a i ( V i − 1 − V i ) , where a i = ( 1 + x i ⊤ V i − 1 x i ) − 1 a_i=(1+x_i^\top V_{i-1}x_i)^{-1} a i = ( 1 + x i ⊤ V i − 1 x i ) − 1 . We show below that also V n = ∑ i > n ( V i − 1 − V i ) V_n=\sum_{i>n}(V_{i-1}-V_i) V n = ∑ i > n ( V i − 1 − V i ) . Hence, Σ n \Sigma_n Σ n and V n V_n V n are weighted sums of the same positive semidefinite matrices V i − 1 − V i V_{i-1}-V_i V i − 1 − V i , with weights σ i 2 a i \sigma_i^2a_i σ i 2 a i and 1 1 1 , respectively. The two bounds then follow by bounding the weights σ i 2 a i \sigma_i^2a_i σ i 2 a i , for i > n i>n i > n , from above by s n + s_n^+ s n + and from below by s n − ( 1 − ϵ n ) s_n^-(1-\epsilon_n) s n − ( 1 − ϵ n ) . We first show that ( 1 − ϵ n ) V n ⪯ S n ⪯ V n (1-\epsilon_n)V_n\preceq S_n\preceq V_n ( 1 − ϵ n ) V n ⪯ S n ⪯ V n , where S n : = ∑ i > n a i ( V i − 1 − V i ) S_n:=\sum_{i>n}a_i(V_{i-1}-V_i) S n := ∑ i > n a i ( V i − 1 − V i ) is the matrix Σ n \Sigma_n Σ n with all σ i 2 \sigma_i^2 σ i 2 replaced by 1. By Equation (5) and Assumption 3, n − 1 V n − 1 = n − 1 V 0 − 1 + n − 1 ∑ i = 1 n x i x i ⊤ → Ω n^{-1}V_n^{-1}=n^{-1}V_0^{-1}+n^{-1}\sum_{i=1}^{n}x_ix_i^\top\to\Omega n − 1 V n − 1 = n − 1 V 0 − 1 + n − 1 ∑ i = 1 n x i x i ⊤ → Ω , so that lim n → ∞ V n = 0 \lim_{n\to\infty}V_n=0 lim n → ∞ V n = 0 . Since the sum ∑ i = n + 1 N ( V i − 1 − V i ) = V n − V N \sum_{i=n+1}^{N}(V_{i-1}-V_i)=V_n-V_N ∑ i = n + 1 N ( V i − 1 − V i ) = V n − V N telescopes, we can write
V n = V n − lim N → ∞ V N = lim N → ∞ ∑ i = n + 1 N ( V i − 1 − V i ) = ∑ i > n ( V i − 1 − V i ) . V_n=V_n-\lim_{N\to\infty}V_N=\lim_{N\to\infty}\sum_{i=n+1}^{N}(V_{i-1}-V_i)=\sum_{i>n}(V_{i-1}-V_i). V n = V n − N → ∞ lim V N = N → ∞ lim i = n + 1 ∑ N ( V i − 1 − V i ) = i > n ∑ ( V i − 1 − V i ) .
The series defining S n S_n S n and Σ n \Sigma_n Σ n also converge, since their terms are positive semidefinite by Lemma A.1 and bounded, in the order ⪯ \preceq ⪯ , by V i − 1 − V i V_{i-1}-V_i V i − 1 − V i and σ i 2 ( V i − 1 − V i ) \sigma_i^2(V_{i-1}-V_i) σ i 2 ( V i − 1 − V i ) , respectively, because a i ≤ 1 a_i\le1 a i ≤ 1 and σ i 2 ≤ σ ∗ 2 \sigma_i^2\le\sigma_*^2 σ i 2 ≤ σ ∗ 2 . Then,
V n − S n = ∑ i > n ( V i − 1 − V i ) − ∑ i > n a i ( V i − 1 − V i ) = ∑ i > n ( 1 − a i ) ( V i − 1 − V i ) ⪰ 0 , V_n-S_n=\sum_{i>n}(V_{i-1}-V_i)-\sum_{i>n}a_i(V_{i-1}-V_i)=\sum_{i>n}(1-a_i)(V_{i-1}-V_i)\succeq0, V n − S n = i > n ∑ ( V i − 1 − V i ) − i > n ∑ a i ( V i − 1 − V i ) = i > n ∑ ( 1 − a i ) ( V i − 1 − V i ) ⪰ 0 ,
because each V i − 1 − V i V_{i-1}-V_i V i − 1 − V i is positive semidefinite by Lemma A.1 and a i ≤ 1 a_i\le1 a i ≤ 1 . This shows that S n ⪯ V n S_n\preceq V_n S n ⪯ V n . Moreover, for every i > n i>n i > n ,
0 ≤ 1 − a i = x i ⊤ V i − 1 x i 1 + x i ⊤ V i − 1 x i ≤ x i ⊤ V i − 1 x i ≤ ϵ n , 0\le1-a_i=\frac{x_i^\top V_{i-1}x_i}{1+x_i^\top V_{i-1}x_i}\le x_i^\top V_{i-1}x_i\le\epsilon_n, 0 ≤ 1 − a i = 1 + x i ⊤ V i − 1 x i x i ⊤ V i − 1 x i ≤ x i ⊤ V i − 1 x i ≤ ϵ n ,
where the second inequality holds because the denominator a i − 1 = 1 + x i ⊤ V i − 1 x i a_i^{-1}=1+x_i^\top V_{i-1}x_i a i − 1 = 1 + x i ⊤ V i − 1 x i is at least 1, and the third because ϵ n \epsilon_n ϵ n is the supremum of x i ⊤ V i − 1 x i x_i^\top V_{i-1}x_i x i ⊤ V i − 1 x i over i > n i>n i > n . Hence, recalling that V n − S n = ∑ i > n ( 1 − a i ) ( V i − 1 − V i ) V_n-S_n=\sum_{i>n}(1-a_i)(V_{i-1}-V_i) V n − S n = ∑ i > n ( 1 − a i ) ( V i − 1 − V i ) , and since each V i − 1 − V i V_{i-1}-V_i V i − 1 − V i is positive semidefinite, so that the scalar bound 1 − a i ≤ ϵ n 1-a_i\le\epsilon_n 1 − a i ≤ ϵ n carries over to each term of the sum, we obtain
V n − S n ⪯ ϵ n ∑ i > n ( V i − 1 − V i ) = ϵ n V n , V_n-S_n\preceq\epsilon_n\sum_{i>n}(V_{i-1}-V_i)=\epsilon_nV_n, V n − S n ⪯ ϵ n i > n ∑ ( V i − 1 − V i ) = ϵ n V n ,
that is, ( 1 − ϵ n ) V n ⪯ S n (1-\epsilon_n)V_n \preceq S_n ( 1 − ϵ n ) V n ⪯ S n . Together with S n ⪯ V n S_n \preceq V_n S n ⪯ V n , shown above, this gives
( 1 − ϵ n ) V n ⪯ S n ⪯ V n . (1-\epsilon_n)V_n \preceq S_n \preceq V_n. ( 1 − ϵ n ) V n ⪯ S n ⪯ V n .
We now bring the weights σ i 2 \sigma_i^2 σ i 2 into play. By Lemma A.1, a i ( V i − 1 − V i ) a_i(V_{i-1}-V_i) a i ( V i − 1 − V i ) is positive semidefinite for every i > n i>n i > n . Then
Σ n − s n − S n = ∑ i > n ( σ i 2 − s n − ) a i ( V i − 1 − V i ) ⪰ 0 \Sigma_n-s_n^-S_n=\sum_{i>n}(\sigma_i^2-s_n^-)a_i(V_{i-1}-V_i)\succeq0 Σ n − s n − S n = i > n ∑ ( σ i 2 − s n − ) a i ( V i − 1 − V i ) ⪰ 0
since σ i 2 − s n − ≥ 0 \sigma_i^2-s_n^- \ge0 σ i 2 − s n − ≥ 0 for i > n i>n i > n , and analogously
s n + S n − Σ n = ∑ i > n ( s n + − σ i 2 ) a i ( V i − 1 − V i ) ⪰ 0 s_n^+S_n-\Sigma_n=\sum_{i>n}(s_n^+-\sigma_i^2)a_i(V_{i-1}-V_i)\succeq0 s n + S n − Σ n = i > n ∑ ( s n + − σ i 2 ) a i ( V i − 1 − V i ) ⪰ 0
since s n + − σ i 2 ≥ 0 s_n^+-\sigma_i^2\ge0 s n + − σ i 2 ≥ 0 for i > n i>n i > n . Therefore, s n − S n ⪯ Σ n ⪯ s n + S n s_n^-S_n\preceq\Sigma_n\preceq s_n^+S_n s n − S n ⪯ Σ n ⪯ s n + S n and, by what we have previously proved,
s n − ( 1 − ϵ n ) V n ⪯ s n − S n ⪯ Σ n ⪯ s n + S n ⪯ s n + V n . s_n^-(1-\epsilon_n)V_n\preceq s_n^-S_n\preceq\Sigma_n\preceq s_n^+S_n\preceq s_n^+V_n. s n − ( 1 − ϵ n ) V n ⪯ s n − S n ⪯ Σ n ⪯ s n + S n ⪯ s n + V n .
Note also that ϵ n → 0 \epsilon_n\to0 ϵ n → 0 as n → ∞ n\to\infty n → ∞ : by Lemma A.2, x i T V i − 1 x i → 0 x_i^\mathsf{T}V_{i-1}x_i\to0 x i T V i − 1 x i → 0 as i → ∞ i\to\infty i → ∞ , and ϵ n \epsilon_n ϵ n is the supremum of the tail ( x i T V i − 1 x i ) i > n (x_i^\mathsf{T}V_{i-1}x_i)_{i>n} ( x i T V i − 1 x i ) i > n of this sequence. Hence, the lower bound is informative, that is, 1 − ϵ n > 0 1-\epsilon_n>0 1 − ϵ n > 0 , for all n n n large enough.
Finally, Σ n \Sigma_n Σ n is positive definite for every n ≥ 0 n\ge0 n ≥ 0 . For i > n i>n i > n , a i = ( 1 + x i T V i − 1 x i ) − 1 ≥ ( 1 + ϵ n ) − 1 a_i=(1+x_i^\mathsf{T}V_{i-1}x_i)^{-1}\ge(1+\epsilon_n)^{-1} a i = ( 1 + x i T V i − 1 x i ) − 1 ≥ ( 1 + ϵ n ) − 1 and σ i 2 ≥ s n − \sigma_i^2\ge s_n^- σ i 2 ≥ s n − , so that, since each V i − 1 − V i V_{i-1}-V_i V i − 1 − V i is positive semidefinite,
Σ n = ∑ i > n σ i 2 a i ( V i − 1 − V i ) ⪰ s n − 1 + ϵ n ∑ i > n ( V i − 1 − V i ) = s n − 1 + ϵ n V n . \Sigma_n=\sum_{i>n}\sigma_i^2a_i(V_{i-1}-V_i)\succeq\frac{s_n^-}{1+\epsilon_n}\sum_{i>n}(V_{i-1}-V_i)=\frac{s_n^-}{1+\epsilon_n}V_n. Σ n = i > n ∑ σ i 2 a i ( V i − 1 − V i ) ⪰ 1 + ϵ n s n − i > n ∑ ( V i − 1 − V i ) = 1 + ϵ n s n − V n .
The right-hand side is positive definite, because V n V_n V n is positive definite, ϵ n \epsilon_n ϵ n is finite, as the supremum of a sequence that converges to 0, and s n − > 0 s_n^->0 s n − > 0 , as the infimum of positive values that converge to σ 2 > 0 \sigma^2>0 σ 2 > 0 by Assumption 3.
Proof of Theorem 1
The proof follows the standard route for parametric martingale posteriors (see, e.g., Holmes and Walker, [28 ] ; Fong and Yiu, [20 ] ). In part (a), we show that, for every fixed u ∈ R p u\in\mathbb{R}^p u ∈ R p , the scalar sequence u T β n u^\mathsf{T}\beta_n u T β n is a martingale bounded in L 2 L_2 L 2 , and conclude with Doob’s martingale convergence theorem (Doob, [16 ] ). In part (b), we derive asymptotic exchangeability from the almost sure weak convergence of the one-step predictive distributions, a standard predictive approach to exchangeability (Fortini and Petrone, [22 , 24 ] ).
(a) Let u ∈ R p u\in\mathbb{R}^p u ∈ R p and θ n = θ n ( u ) = u T β n \theta_n=\theta_n(u)=u^\mathsf{T}\beta_n θ n = θ n ( u ) = u T β n . Multiplying Equation (3) on the left by u T u^\mathsf{T} u T gives
u T β n = u T β n − 1 + a n u T V n − 1 X n ( Y n − X n T β n − 1 ) , u^\mathsf{T}\beta_n=u^\mathsf{T}\beta_{n-1}+a_nu^\mathsf{T}V_{n-1}X_n(Y_n-X_n^\mathsf{T}\beta_{n-1}), u T β n = u T β n − 1 + a n u T V n − 1 X n ( Y n − X n T β n − 1 ) ,
that is, θ n = θ n − 1 + c n e n \theta_n=\theta_{n-1}+c_ne_n θ n = θ n − 1 + c n e n , where c n : = a n u T V n − 1 X n c_n := a_n\mathbf{u}^{\mathsf{T}}V_{n-1}X_n c n := a n u T V n − 1 X n and e n : = Y n − X n T β n − 1 e_n := Y_n-X_n^{\mathsf{T}}\beta_{n-1} e n := Y n − X n T β n − 1 . Let ( F n ) n ≥ 0 (\mathcal{F}_n)_{n\geq0} ( F n ) n ≥ 0 be the filtration with F 0 \mathcal{F}_0 F 0 the trivial σ \sigma σ -field and F n = σ ( X 1 , Y 1 , … , X n , Y n ) \mathcal{F}_n=\sigma(X_1,Y_1,\ldots,X_n,Y_n) F n = σ ( X 1 , Y 1 , … , X n , Y n ) for n ≥ 1 n\geq1 n ≥ 1 ; note that θ 0 = u T β 0 \theta_0=\mathbf{u}^{\mathsf{T}}\beta_0 θ 0 = u T β 0 is a constant.
Both c n c_n c n and σ n = s n − 1 ( X n ) \sigma_n=s_{n-1}(X_n) σ n = s n − 1 ( X n ) are measurable with respect to F n − 1 ∨ σ ( X n ) \mathcal{F}_{n-1}\vee\sigma(X_n) F n − 1 ∨ σ ( X n ) , that is, they are known once the past observations and the new covariate X n X_n X n are known. By Equation (2) and Assumption 1, given F n − 1 \mathcal{F}_{n-1} F n − 1 and X n X_n X n , the response Y n Y_n Y n has mean X n T β n − 1 X_n^{\mathsf{T}}\beta_{n-1} X n T β n − 1 and variance σ n 2 \sigma_n^2 σ n 2 , so that
E ( e n ∣ F n − 1 , X n ) = 0 , E ( e n 2 ∣ F n − 1 , X n ) = σ n 2 ≤ σ ∗ 2 . \mathbb{E}(e_n\mid\mathcal{F}_{n-1},X_n)=0,\qquad\mathbb{E}(e_n^2\mid\mathcal{F}_{n-1},X_n)=\sigma_n^2\leq\sigma_*^2. E ( e n ∣ F n − 1 , X n ) = 0 , E ( e n 2 ∣ F n − 1 , X n ) = σ n 2 ≤ σ ∗ 2 .
Consequently, by the tower property of conditional expectations,
E ( θ n ∣ F n − 1 ) = E ( θ n − 1 + c n e n ∣ F n − 1 ) = θ n − 1 + E { c n E ( e n ∣ F n − 1 , X n ) ∣ F n − 1 } = θ n − 1 . \begin{aligned}
\mathbb{E}(\theta_n\mid\mathcal{F}_{n-1})=\mathbb{E}(\theta_{n-1}+c_ne_n\mid\mathcal{F}_{n-1}) \\
&=\theta_{n-1}+\mathbb{E}\{c_n\mathbb{E}(e_n\mid\mathcal{F}_{n-1},X_n)\mid\mathcal{F}_{n-1}\}=\theta_{n-1}.
\end{aligned} E ( θ n ∣ F n − 1 ) = E ( θ n − 1 + c n e n ∣ F n − 1 ) = θ n − 1 + E { c n E ( e n ∣ F n − 1 , X n ) ∣ F n − 1 } = θ n − 1 .
Then, ( θ n ) n ≥ 0 (\theta_n)_{n\geq0} ( θ n ) n ≥ 0 is a martingale with respect to the filtration ( F n ) n ≥ 0 (\mathcal{F}_n)_{n\geq0} ( F n ) n ≥ 0 ; integrability, E ∣ θ n ∣ < ∞ \mathbb{E}|\theta_n|<\infty E ∣ θ n ∣ < ∞ , follows from the bound on E ( θ n 2 ) \mathbb{E}(\theta_n^2) E ( θ n 2 ) below.
Now we show that sup n E ( θ n 2 ) < ∞ \sup_n\mathbb{E}(\theta_n^2)<\infty sup n E ( θ n 2 ) < ∞ . Squaring θ n = θ n − 1 + c n e n \theta_n=\theta_{n-1}+c_ne_n θ n = θ n − 1 + c n e n and conditioning on F n − 1 \mathcal{F}_{n-1} F n − 1 and X n X_n X n , with respect to which θ n − 1 \theta_{n-1} θ n − 1 and c n c_n c n are measurable, we obtain, from the two conditional moments of e n e_n e n displayed above,
E ( θ n 2 ∣ F n − 1 , X n ) = θ n − 1 2 + 2 θ n − 1 c n E ( e n ∣ F n − 1 , X n ) + c n 2 E ( e n 2 ∣ F n − 1 , X n ) = θ n − 1 2 + σ n 2 c n 2 ≤ θ n − 1 2 + σ ∗ 2 c n 2 . \begin{aligned}
\mathbb{E}(\theta_n^2\mid\mathcal{F}_{n-1},X_n)=\theta_{n-1}^2+2\theta_{n-1}c_n\mathbb{E}(e_n\mid\mathcal{F}_{n-1},X_n)+c_n^2\mathbb{E}(e_n^2\mid\mathcal{F}_{n-1},X_n) \\
&=\theta_{n-1}^2+\sigma_n^2c_n^2\leq\theta_{n-1}^2+\sigma_*^2c_n^2.
\end{aligned} E ( θ n 2 ∣ F n − 1 , X n ) = θ n − 1 2 + 2 θ n − 1 c n E ( e n ∣ F n − 1 , X n ) + c n 2 E ( e n 2 ∣ F n − 1 , X n ) = θ n − 1 2 + σ n 2 c n 2 ≤ θ n − 1 2 + σ ∗ 2 c n 2 .
Taking expectations and iterating over n n n gives E ( θ n 2 ) ≤ θ 0 2 + σ ∗ 2 E ( ∑ i = 1 n c i 2 ) \mathbb{E}(\theta_n^2)\leq\theta_0^2+\sigma_*^2\mathbb{E}(\sum_{i=1}^n c_i^2) E ( θ n 2 ) ≤ θ 0 2 + σ ∗ 2 E ( ∑ i = 1 n c i 2 ) . The quantity c n 2 c_n^2 c n 2 can be rewritten using Equation (4) , which gives V n − 1 − V n = a n V n − 1 X n X n T V n − 1 V_{n-1}-V_n=a_nV_{n-1}X_nX_n^{\mathsf{T}}V_{n-1} V n − 1 − V n = a n V n − 1 X n X n T V n − 1 :
c n 2 = a n 2 ( u T V n − 1 X n ) 2 = a n u T ( V n − 1 − V n ) u ≤ u T V n − 1 u − u T V n u , c_n^2=a_n^2(\mathbf{u}^{\mathsf{T}}V_{n-1}X_n)^2=a_n\mathbf{u}^{\mathsf{T}}(V_{n-1}-V_n)\mathbf{u}\leq\mathbf{u}^{\mathsf{T}}V_{n-1}\mathbf{u}-\mathbf{u}^{\mathsf{T}}V_n\mathbf{u}, c n 2 = a n 2 ( u T V n − 1 X n ) 2 = a n u T ( V n − 1 − V n ) u ≤ u T V n − 1 u − u T V n u ,
since a n ∈ ( 0 , 1 ] a_n\in(0,1] a n ∈ ( 0 , 1 ] and u T ( V n − 1 − V n ) u ≥ 0 \mathbf{u}^{\mathsf{T}}(V_{n-1}-V_n)\mathbf{u}\geq0 u T ( V n − 1 − V n ) u ≥ 0 by Lemma A.1. Summing these inequalities over i = 1 , … , n i=1,\ldots,n i = 1 , … , n , the right-hand side telescopes, so that, for every realization of the covariates,
∑ i = 1 n c i 2 ≤ ∑ i = 1 n ( u T V i − 1 u − u T V i u ) = u T V 0 u − u T V n u ≤ u T V 0 u , \sum_{i=1}^n c_i^2\leq\sum_{i=1}^n(\mathbf{u}^{\mathsf{T}}V_{i-1}\mathbf{u}-\mathbf{u}^{\mathsf{T}}V_i\mathbf{u})=\mathbf{u}^{\mathsf{T}}V_0\mathbf{u}-\mathbf{u}^{\mathsf{T}}V_n\mathbf{u}\leq\mathbf{u}^{\mathsf{T}}V_0\mathbf{u}, i = 1 ∑ n c i 2 ≤ i = 1 ∑ n ( u T V i − 1 u − u T V i u ) = u T V 0 u − u T V n u ≤ u T V 0 u ,
where the last inequality holds because V n V_n V n is positive definite. Hence, for every n ≥ 1 n\geq1 n ≥ 1 ,
E ( θ n 2 ) ≤ θ 0 2 + σ ∗ 2 u T V 0 u , \mathbb{E}(\theta_n^2)\leq\theta_0^2+\sigma_*^2\mathbf{u}^{\mathsf{T}}V_0\mathbf{u}, E ( θ n 2 ) ≤ θ 0 2 + σ ∗ 2 u T V 0 u ,
which is finite, since θ 0 = u T β 0 \theta_0=\mathbf{u}^{\mathsf{T}}\beta_0 θ 0 = u T β 0 is fixed, and does not depend on n n n . Therefore, ( θ n ) n ≥ 0 (\theta_n)_{n\geq0} ( θ n ) n ≥ 0 is a martingale bounded in L 2 L_2 L 2 . By Doob’s convergence theorem for such martingales [40 Theorem 12.1] , θ n \theta_n θ n converges almost surely and in L 2 L_2 L 2 to a limit random variable θ ~ ( u ) \widetilde{\theta}(\mathbf{u}) θ ( u ) . Moreover, since convergence in L 2 L_2 L 2 implies convergence in L 1 L_1 L 1 , and conditional expectations are continuous in L 1 L_1 L 1 , letting m → ∞ m\to\infty m → ∞ in E ( θ m ∣ F n ) = θ n \mathbb{E}(\theta_m\mid\mathcal{F}_n)=\theta_n E ( θ m ∣ F n ) = θ n , for m > n m>n m > n , gives E { θ ~ ( u ) ∣ F n } = θ n \mathbb{E}\{\widetilde{\theta}(\mathbf{u})\mid\mathcal{F}_n\}=\theta_n E { θ ( u ) ∣ F n } = θ n . Taking u \mathbf{u} u equal to each vector of the canonical basis of R p \mathbb{R}^p R p , we obtain a random vector β ~ \widetilde{\beta} β such that β n \beta_n β n converges to β ~ \widetilde{\beta} β almost surely and in L 2 L_2 L 2 , and E ( β ~ ∣ F n ) = β n \mathbb{E}(\widetilde{\beta}\mid\mathcal{F}_n)=\beta_n E ( β ∣ F n ) = β n .
(b) By Lemma 8.2(b) in Aldous [4 ] , a sequence of random elements of a Polish space is asymptotically exchangeable if its one-step predictive distributions converge weakly almost surely to a random probability measure, which is then its directing random measure. We identify the predictive distribution Q n Q_n Q n of ( X n + 1 , Y n + 1 ) (\mathbf{X}_{n+1},Y_{n+1}) ( X n + 1 , Y n + 1 ) in Step 1 and prove its convergence in Step 2; since Q n Q_n Q n depends on n n n only through β n \beta_n β n and s n s_n s n , this follows from part (a) and Assumption 2. Step 3 applies the lemma.
Step 1: the predictive distribution. Let R n = ( X n , Y n ) ∈ S : = R p × R \mathbf{R}_n=(\mathbf{X}_n,Y_n)\in S:=\mathbb{R}^p\times\mathbb{R} R n = ( X n , Y n ) ∈ S := R p × R , so that F n = σ ( R 1 : n ) \mathcal{F}_n=\sigma(\mathbf{R}_{1:n}) F n = σ ( R 1 : n ) . By Equation (2) and Assumptions 1–2, R n + 1 = ( X n + 1 , X n + 1 T β n + s n ( X n + 1 ) ξ n + 1 ) \mathbf{R}_{n+1}=(\mathbf{X}_{n+1},\mathbf{X}_{n+1}^{\mathsf{T}}\beta_n+s_n(\mathbf{X}_{n+1})\xi_{n+1}) R n + 1 = ( X n + 1 , X n + 1 T β n + s n ( X n + 1 ) ξ n + 1 ) , where ξ n + 1 \xi_{n+1} ξ n + 1 is defined by this identity, since s n > 0 s_n>0 s n > 0 , and X n + 1 ∼ P X \mathbf{X}_{n+1}\sim P_X X n + 1 ∼ P X and ξ n + 1 ∼ P ξ \xi_{n+1}\sim P_\xi ξ n + 1 ∼ P ξ are independent of each other and of F n \mathcal{F}_n F n : indeed, X n + 1 \mathbf{X}_{n+1} X n + 1 is independent of F n \mathcal{F}_n F n by Assumption 2, and the conditional law of ξ n + 1 \xi_{n+1} ξ n + 1 given F n \mathcal{F}_n F n and X n + 1 \mathbf{X}_{n+1} X n + 1 is P ξ P_\xi P ξ , which does not depend on them. Hence, holding R 1 : n \mathbf{R}_{1:n} R 1 : n fixed and integrating over ( X n + 1 , ξ n + 1 ) (\mathbf{X}_{n+1},\xi_{n+1}) ( X n + 1 , ξ n + 1 ) ([13 Exercise IV.2.26] ), a regular version of P ( R n + 1 ∈ ⋅ ∣ F n ) \mathbb{P}(\mathbf{R}_{n+1}\in\cdot\mid\mathcal{F}_n) P ( R n + 1 ∈ ⋅ ∣ F n ) is
Q n ( d x , d y ) = P X ( d x ) P { d y ∣ x , β n , s n ( x ) } . Q_n(dx,dy)=P_X(dx)\,P\{dy\mid x,\beta_n,s_n(x)\}. Q n ( d x , d y ) = P X ( d x ) P { d y ∣ x , β n , s n ( x )} .
Step 2: weak convergence of the predictive distributions. Let Ω 0 \Omega_0 Ω 0 be the set on which β n → β ~ \beta_n\to\tilde{\beta} β n → β ~ and s n ( x ) → σ s_n(x)\to\sigma s n ( x ) → σ for P X P_X P X -almost every x x x . By Fubini’s theorem, Ω 0 \Omega_0 Ω 0 is an event, since s n s_n s n is jointly measurable in x x x and X 1 : n \mathbf{X}_{1:n} X 1 : n , and it has probability one by part (a) and Assumption 2. Fix ω ∈ Ω 0 \omega\in\Omega_0 ω ∈ Ω 0 . For P X P_X P X -almost every x x x , we have x T β n + s n ( x ) ξ → x T β ~ + σ ξ x^{\mathsf{T}}\beta_n+s_n(x)\xi\to x^{\mathsf{T}}\tilde{\beta}+\sigma\xi x T β n + s n ( x ) ξ → x T β ~ + σ ξ for every ξ ∈ R \xi\in\mathbb{R} ξ ∈ R ; taking ξ ∼ P ξ \xi\sim P_\xi ξ ∼ P ξ , by Assumption 1, P { ⋅ ∣ x , β n , s n ( x ) } → P ( ⋅ ∣ x , β ~ , σ ) P\{\cdot\mid x,\beta_n,s_n(x)\}\to P(\cdot\mid x,\tilde{\beta},\sigma) P { ⋅ ∣ x , β n , s n ( x )} → P ( ⋅ ∣ x , β ~ , σ ) weakly, as pointwise convergence implies convergence in distribution. Hence, for every bounded continuous f : S → R f:S\to\mathbb{R} f : S → R , since y ↦ f ( x , y ) y\mapsto f(x,y) y ↦ f ( x , y ) is bounded and continuous, the inner integral ∫ f ( x , y ) P { d y ∣ x , β n , s n ( x ) } \int f(x,y)P\{dy\mid x,\beta_n,s_n(x)\} ∫ f ( x , y ) P { d y ∣ x , β n , s n ( x )} converges to ∫ f ( x , y ) P ( d y ∣ x , β ~ , σ ) \int f(x,y)P(dy\mid x,\tilde{\beta},\sigma) ∫ f ( x , y ) P ( d y ∣ x , β ~ , σ ) for P X P_X P X -almost every x x x , and it is bounded by ∥ f ∥ ∞ : = sup ∣ f ∣ \lVert f\rVert_\infty:=\sup|f| ∥ f ∥ ∞ := sup ∣ f ∣ . Integrating with respect to P X P_X P X , dominated convergence gives ∫ f d Q n → ∫ f d P ~ X , Y \int f\,dQ_n\to\int f\,d\tilde{P}_{X,Y} ∫ f d Q n → ∫ f d P ~ X , Y , where
P ~ X , Y ( d x , d y ) = P X ( d x ) P ( d y ∣ x , β ~ , σ ) \tilde{P}_{X,Y}(dx,dy)=P_X(dx)P(dy\mid x,\tilde{\beta},\sigma) P ~ X , Y ( d x , d y ) = P X ( d x ) P ( d y ∣ x , β ~ , σ )
is a random probability measure, by the measurability of β ~ \tilde{\beta} β ~ and Fubini’s theorem. Thus, Q n → P ~ X , Y Q_n\to\tilde{P}_{X,Y} Q n → P ~ X , Y weakly on Ω 0 \Omega_0 Ω 0 , that is, P \mathbb{P} P -almost surely.
Step 3: asymptotic exchangeability. Since S S S is Polish, by Lemma 8.2(b) in Aldous [4 ] the almost sure weak convergence of the one-step predictive distributions implies that ( R n ) n ≥ 1 (\mathbf{R}_n)_{n\geq1} ( R n ) n ≥ 1 is asymptotically exchangeable with directing random measure P ~ X , Y \tilde{P}_{X,Y} P ~ X , Y . By construction, its X \mathbf{X} X -marginal is P X P_X P X and P ( ⋅ ∣ x , β ~ , σ ) P(\cdot\mid x,\tilde{\beta},\sigma) P ( ⋅ ∣ x , β ~ , σ ) is a version of its conditional distribution of Y Y Y given X = x \mathbf{X}=x X = x .
Proof of Proposition 1
The proof parallels that of Theorem 1. Part (a) repeats the martingale argument, now with known covariates. In part (b), the predictive distribution P n P_n P n need not converge, because x n + 1 x_{n+1} x n + 1 changes with n n n ; we show instead, by a coupling argument, that it becomes close to P ( ⋅ ∣ x n + 1 , β ~ , σ ) P(\cdot\mid x_{n+1}, \widetilde{\beta}, \sigma) P ( ⋅ ∣ x n + 1 , β , σ ) , in the sense of weak merging [14 ] , a notion weaker than the merging in total variation of Blackwell and Dubins [9 ] . We conclude by deriving the series representation of β ~ \widetilde{\beta} β in Equation (6) of the paper.
(a) The argument is that of part (a) of Theorem 1, with three simplifications. Under Assumption 3 the covariates are known, so that c n = a n u T V n − 1 x n c_n = a_n u^{\mathsf{T}} V_{n-1} x_n c n = a n u T V n − 1 x n and σ n \sigma_n σ n are deterministic, and e n = Y n − x n T β n − 1 e_n = Y_n - x_n^{\mathsf{T}}\beta_{n-1} e n = Y n − x n T β n − 1 ; the filtration is F n = σ ( Y 1 , … , Y n ) \mathcal{F}_n = \sigma(Y_1,\ldots,Y_n) F n = σ ( Y 1 , … , Y n ) for n ≥ 1 n \ge1 n ≥ 1 , with F 0 \mathcal{F}_0 F 0 the trivial σ \sigma σ -field; and it suffices to condition on F n − 1 \mathcal{F}_{n-1} F n − 1 , rather than on F n − 1 \mathcal{F}_{n-1} F n − 1 and X n X_n X n . Indeed, by Equation (2) and Assumption 1, E ( e n ∣ F n − 1 ) = 0 \mathbb{E}(e_n \mid\mathcal{F}_{n-1}) = 0 E ( e n ∣ F n − 1 ) = 0 and E ( e n 2 ∣ F n − 1 ) = σ n 2 ≤ σ ∗ 2 \mathbb{E}(e_n^2 \mid\mathcal{F}_{n-1}) = \sigma_n^2 \le\sigma_*^2 E ( e n 2 ∣ F n − 1 ) = σ n 2 ≤ σ ∗ 2 , so that the same computations show that ( θ n ) n ≥ 0 (\theta_n)_{n \ge0} ( θ n ) n ≥ 0 is a martingale with E ( θ n 2 ) ≤ θ 0 2 + σ ∗ 2 u T V 0 u \mathbb{E}(\theta_n^2) \le\theta_0^2 + \sigma_*^2 u^{\mathsf{T}} V_0 u E ( θ n 2 ) ≤ θ 0 2 + σ ∗ 2 u T V 0 u for every n n n . Doob’s martingale convergence theorem then gives, as in Theorem 1, that β n → β ~ \beta_n \to\widetilde{\beta} β n → β almost surely and in L 2 L_2 L 2 , and that E ( β ~ ∣ F n ) = β n \mathbb{E}(\widetilde{\beta} \mid\mathcal{F}_n) = \beta_n E ( β ∣ F n ) = β n .
(b) Fix an outcome on which β n → β ~ \beta_n \to\widetilde{\beta} β n → β , which has P x 1 : ∞ \mathbb{P}_{x_{1:\infty}} P x 1 : ∞ -probability one by part (a); on this outcome, β n \beta_n β n and β ~ \widetilde{\beta} β are fixed vectors. Let ξ ∼ P ξ \xi\sim P_\xi ξ ∼ P ξ be an auxiliary random variable. By Assumption 1, the random variables
Y n + 1 ′ : = x n + 1 T β n + σ n + 1 ξ and Y ~ n + 1 : = x n + 1 T β ~ + σ ξ Y'_{n+1} := x_{n+1}^{\mathsf{T}}\beta_n + \sigma_{n+1}\xi
\qquad\text{and}\qquad
\widetilde{Y}_{n+1} := x_{n+1}^{\mathsf{T}}\widetilde{\beta} + \sigma\xi Y n + 1 ′ := x n + 1 T β n + σ n + 1 ξ and Y n + 1 := x n + 1 T β + σ ξ
have distributions P n P_n P n and P ( ⋅ ∣ x n + 1 , β ~ , σ ) P(\cdot\mid x_{n+1},\widetilde{\beta},\sigma) P ( ⋅ ∣ x n + 1 , β , σ ) , respectively. Thus, Y n + 1 ′ Y'_{n+1} Y n + 1 ′ has the same law as Y n + 1 Y_{n+1} Y n + 1 given y 1 : n y_{1:n} y 1 : n , but, unlike Y n + 1 Y_{n+1} Y n + 1 , it is built from the same ξ \xi ξ as Y ~ n + 1 \widetilde{Y}_{n+1} Y n + 1 . Let f : R → R f:\mathbb{R}\to\mathbb{R} f : R → R be bounded and continuous, and let
d n : = ∫ f d P n − ∫ f d P ( ⋅ ∣ x n + 1 , β ~ , σ ) = E { f ( Y n + 1 ′ ) − f ( Y ~ n + 1 ) } . d_n := \int f\,\mathrm{d}P_n - \int f\,\mathrm{d}P(\cdot\mid x_{n+1},\widetilde{\beta},\sigma)
= \mathbb{E}\{f(Y'_{n+1}) - f(\widetilde{Y}_{n+1})\}. d n := ∫ f d P n − ∫ f d P ( ⋅ ∣ x n + 1 , β , σ ) = E { f ( Y n + 1 ′ ) − f ( Y n + 1 )} .
We need to show that d n → 0 d_n \to0 d n → 0 as n → ∞ n \to\infty n → ∞ , for every such f f f . Unlike in the proof of Theorem 1(b), we cannot let n → ∞ n \to\infty n → ∞ in each of the two terms, because x n + 1 x_{n+1} x n + 1 , and hence Y n + 1 ′ Y'_{n+1} Y n + 1 ′ and Y ~ n + 1 \widetilde{Y}_{n+1} Y n + 1 , need not converge. We will show that Y n + 1 ′ − Y ~ n + 1 → 0 Y'_{n+1} - \widetilde{Y}_{n+1} \to0 Y n + 1 ′ − Y n + 1 → 0 , but this alone does not imply that f ( Y n + 1 ′ ) − f ( Y ~ n + 1 ) → 0 f(Y'_{n+1}) - f(\widetilde{Y}_{n+1}) \to0 f ( Y n + 1 ′ ) − f ( Y n + 1 ) → 0 , because a bounded continuous f f f need not be uniformly continuous on R \mathbb{R} R , as for f ( y ) = sin ( y 2 ) f(y) = \sin(y^2) f ( y ) = sin ( y 2 ) . Therefore, we also show that both variables lie in a compact interval, independent of n n n , on which f f f is uniformly continuous.
Let M : = sup n ≥ 1 ∥ x n ∥ M := \sup_{n \ge1}\lVert x_n\rVert M := sup n ≥ 1 ∥ x n ∥ , which is finite because the sequence x 1 : ∞ x_{1:\infty} x 1 : ∞ is bounded by Assumption 3. Subtracting the two definitions, Y n + 1 ′ − Y ~ n + 1 = x n + 1 T ( β n − β ~ ) + ( σ n + 1 − σ ) ξ Y'_{n+1} - \widetilde{Y}_{n+1} = x_{n+1}^{\mathsf{T}}(\beta_n - \widetilde{\beta}) + (\sigma_{n+1} - \sigma)\xi Y n + 1 ′ − Y n + 1 = x n + 1 T ( β n − β ) + ( σ n + 1 − σ ) ξ , and therefore
∣ Y n + 1 ′ − Y ~ n + 1 ∣ ≤ ∣ x n + 1 T ( β n − β ~ ) ∣ + ∣ σ n + 1 − σ ∣ ∣ ξ ∣ ≤ ∥ x n + 1 ∥ ∥ β n − β ~ ∥ + ∣ σ n + 1 − σ ∣ ∣ ξ ∣ ≤ M ∥ β n − β ~ ∥ + ∣ σ n + 1 − σ ∣ ∣ ξ ∣ , \begin{aligned}
\left|Y'_{n+1} - \widetilde{Y}_{n+1}\right|
\le\left|x_{n+1}^{\mathsf{T}}(\beta_n - \widetilde{\beta})\right| + \left|\sigma_{n+1} - \sigma\right|\lvert\xi\rvert\\
&\le\lVert x_{n+1}\rVert\lVert\beta_n - \widetilde{\beta}\rVert+ \left|\sigma_{n+1} - \sigma\right|\lvert\xi\rvert\\
&\le M\lVert\beta_n - \widetilde{\beta}\rVert+ \left|\sigma_{n+1} - \sigma\right|\lvert\xi\rvert,
\end{aligned} Y n + 1 ′ − Y n + 1 ≤ x n + 1 T ( β n − β ) + ∣ σ n + 1 − σ ∣ ∣ ξ ∣ ≤ ∥ x n + 1 ∥ ∥ β n − β ∥ + ∣ σ n + 1 − σ ∣ ∣ ξ ∣ ≤ M ∥ β n − β ∥ + ∣ σ n + 1 − σ ∣ ∣ ξ ∣ ,
where the first inequality is the triangle inequality, the second is the Cauchy–Schwarz inequality applied to the vectors x n + 1 \mathbf{x}_{n+1} x n + 1 and β n − β ~ \boldsymbol{\beta}_n-\widetilde{\boldsymbol{\beta}} β n − β , and the third holds because ∥ x n + 1 ∥ ≤ M \lVert\mathbf{x}_{n+1}\rVert\le M ∥ x n + 1 ∥ ≤ M . The right-hand side tends to 0, because β n → β ~ \boldsymbol{\beta}_n \to\widetilde{\boldsymbol{\beta}} β n → β and σ n → σ \sigma_n \to\sigma σ n → σ by Assumption 3, so that Y n + 1 ′ − Y ~ n + 1 → 0 Y'_{n+1}-\widetilde{Y}_{n+1} \to0 Y n + 1 ′ − Y n + 1 → 0 for every value of ξ \xi ξ .
For the interval, note that on the fixed outcome ( β n ) n ≥ 0 (\boldsymbol{\beta}_n)_{n\ge0} ( β n ) n ≥ 0 is a convergent sequence in R p \mathbb{R}^p R p , hence bounded, and its limit satisfies ∥ β ~ ∥ ≤ sup n ≥ 0 ∥ β n ∥ \lVert\widetilde{\boldsymbol{\beta}}\rVert\le\sup_{n\ge0}\lVert\boldsymbol{\beta}_n\rVert ∥ β ∥ ≤ sup n ≥ 0 ∥ β n ∥ ; moreover, σ n + 1 ≤ σ ∗ \sigma_{n+1} \le\sigma_* σ n + 1 ≤ σ ∗ and σ ≤ σ ∗ \sigma\le\sigma_* σ ≤ σ ∗ , the latter because σ \sigma σ is the limit of σ n \sigma_n σ n . The same two inequalities as above then give ∣ Y n + 1 ′ ∣ ≤ M ∥ β n ∥ + σ n + 1 ∣ ξ ∣ \lvert Y'_{n+1}\rvert\le M\lVert\boldsymbol{\beta}_n\rVert+\sigma_{n+1}\lvert\xi\rvert ∣ Y n + 1 ′ ∣ ≤ M ∥ β n ∥ + σ n + 1 ∣ ξ ∣ and ∣ Y ~ n + 1 ∣ ≤ M ∥ β ~ ∥ + σ ∣ ξ ∣ \lvert\widetilde{Y}_{n+1}\rvert\le M\lVert\widetilde{\boldsymbol{\beta}}\rVert+\sigma\lvert\xi\rvert ∣ Y n + 1 ∣ ≤ M ∥ β ∥ + σ ∣ ξ ∣ , so that both variables lie in the interval [ − C − σ ∗ ∣ ξ ∣ , C + σ ∗ ∣ ξ ∣ ] [-C-\sigma_*\lvert\xi\rvert,C+\sigma_*\lvert\xi\rvert] [ − C − σ ∗ ∣ ξ ∣ , C + σ ∗ ∣ ξ ∣] , with C : = M sup n ≥ 0 ∥ β n ∥ < ∞ C:=M\sup_{n\ge0}\lVert\boldsymbol{\beta}_n\rVert<\infty C := M sup n ≥ 0 ∥ β n ∥ < ∞ , for every n n n .
By the Heine–Cantor theorem, f f f is uniformly continuous on this compact interval. Since Y n + 1 ′ − Y ~ n + 1 → 0 Y'_{n+1}-\widetilde{Y}_{n+1}\to0 Y n + 1 ′ − Y n + 1 → 0 and both variables lie in it, it follows that f ( Y n + 1 ′ ) − f ( Y ~ n + 1 ) → 0 f(Y'_{n+1})-f(\widetilde{Y}_{n+1})\to0 f ( Y n + 1 ′ ) − f ( Y n + 1 ) → 0 for every value of ξ \xi ξ . As ∣ f ( Y n + 1 ′ ) − f ( Y ~ n + 1 ) ∣ ≤ 2 ∥ f ∥ ∞ \lvert f(Y'_{n+1})-f(\widetilde{Y}_{n+1})\rvert\le2\lVert f\rVert_\infty ∣ f ( Y n + 1 ′ ) − f ( Y n + 1 )∣ ≤ 2 ∥ f ∥ ∞ , dominated convergence with respect to P ξ P_\xi P ξ gives d n → 0 d_n\to0 d n → 0 . Since the outcome was arbitrary in an event of probability one, P n P_n P n and P ( ⋅ ∣ x n + 1 , β ~ , σ ) P(\mathord\cdot\mid\mathbf{x}_{n+1},\widetilde{\boldsymbol{\beta}},\sigma) P ( ⋅ ∣ x n + 1 , β , σ ) merge weakly P x 1 : ∞ \mathbb{P}_{\mathbf{x}_{1:\infty}} P x 1 : ∞ -almost surely.
Finally, we derive Equation (6) of the paper. For i ≥ 1 i\ge1 i ≥ 1 , let ξ i : = ( Y i − x i ⊤ β i − 1 ) / σ i \xi_i:=(Y_i-\mathbf{x}_i^\top\boldsymbol{\beta}_{i-1})/\sigma_i ξ i := ( Y i − x i ⊤ β i − 1 ) / σ i . By Equation (2) and Assumption 1, the ξ i \xi_i ξ i are iid from P ξ P_\xi P ξ , and those with i > n i>n i > n are independent of y 1 : n \mathbf{y}_{1:n} y 1 : n . By Equation (3) , β i − β i − 1 = w i ξ i \boldsymbol{\beta}_i-\boldsymbol{\beta}_{i-1}=w_i\xi_i β i − β i − 1 = w i ξ i , with w i = σ i a i V i − 1 x i w_i=\sigma_i a_i V_{i-1}\mathbf{x}_i w i = σ i a i V i − 1 x i deterministic under Assumption 3. Summing from i = n + 1 i=n+1 i = n + 1 to N N N , the left-hand side telescopes, so that β N − β n = ∑ i = n + 1 N w i ξ i \boldsymbol{\beta}_N-\boldsymbol{\beta}_n=\sum_{i=n+1}^{N}w_i\xi_i β N − β n = ∑ i = n + 1 N w i ξ i . As N → ∞ N\to\infty N → ∞ , β N → β ~ \boldsymbol{\beta}_N\to\widetilde{\boldsymbol{\beta}} β N → β almost surely and in L 2 L_2 L 2 by part (a), and hence the partial sums converge, in the same senses, to β ~ − β n \widetilde{\boldsymbol{\beta}}-\boldsymbol{\beta}_n β − β n . This gives
β ~ − β n = ∑ i > n w i ξ i , \widetilde{\boldsymbol{\beta}}-\boldsymbol{\beta}_n=\sum_{i>n}w_i\xi_i, β − β n = i > n ∑ w i ξ i ,
where the series converges almost surely and in L 2 L_2 L 2 . In particular, β ~ − β n \widetilde{\boldsymbol{\beta}}-\boldsymbol{\beta}_n β − β n depends only on ξ n + 1 , ξ n + 2 , … \xi_{n+1},\xi_{n+2},\ldots ξ n + 1 , ξ n + 2 , … and is therefore independent of y 1 : n \mathbf{y}_{1:n} y 1 : n .
Proof of Proposition 2
By Equation (6) of the paper, derived at the end of the proof of Proposition 1, β ~ − β n = ∑ i > n w i ξ i \widetilde{\boldsymbol{\beta}}-\boldsymbol{\beta}_n=\sum_{i>n}w_i\xi_i β − β n = ∑ i > n w i ξ i is independent of y 1 : n \mathbf{y}_{1:n} y 1 : n , with deterministic weights and ξ i \xi_i ξ i iid with zero mean and unit variance. Hence, its partial sums ∑ i = n + 1 N w i ξ i \sum_{i=n+1}^{N}w_i\xi_i ∑ i = n + 1 N w i ξ i have variance ∑ i = n + 1 N w i w i ⊤ = Σ n − Σ N \sum_{i=n+1}^{N}w_iw_i^\top=\boldsymbol{\Sigma}_n-\boldsymbol{\Sigma}_N ∑ i = n + 1 N w i w i ⊤ = Σ n − Σ N . They converge in L 2 L_2 L 2 as N → ∞ N\to\infty N → ∞ , so that their variances converge, by the Cauchy–Schwarz inequality, and Σ N → 0 \boldsymbol{\Sigma}_N\to0 Σ N → 0 , as the tail of the series defining Σ 0 \boldsymbol{\Sigma}_0 Σ 0 , which converges by the proof of Lemma A.3. Therefore, since β n \boldsymbol{\beta}_n β n is a function of y 1 : n \mathbf{y}_{1:n} y 1 : n ,
var ( β ~ ∣ y 1 : n ) = var ( β ~ − β n ∣ y 1 : n ) = var ( β ~ − β n ) = lim N → ∞ ( Σ n − Σ N ) = Σ n . \operatorname{var}(\widetilde{\boldsymbol{\beta}}\mid\mathbf{y}_{1:n})=\operatorname{var}(\widetilde{\boldsymbol{\beta}}-\boldsymbol{\beta}_n\mid\mathbf{y}_{1:n})=\operatorname{var}(\widetilde{\boldsymbol{\beta}}-\boldsymbol{\beta}_n)=\lim_{N\to\infty}(\boldsymbol{\Sigma}_n-\boldsymbol{\Sigma}_N)=\boldsymbol{\Sigma}_n. var ( β ∣ y 1 : n ) = var ( β − β n ∣ y 1 : n ) = var ( β − β n ) = N → ∞ lim ( Σ n − Σ N ) = Σ n .
The same argument with n = 0 n=0 n = 0 , where β 0 \beta_0 β 0 is fixed, gives var ( β ~ ) = Σ 0 \operatorname{var}(\widetilde{\beta})=\Sigma_0 var ( β ) = Σ 0 . When σ n 2 = σ 2 / a n \sigma_n^2=\sigma^2/a_n σ n 2 = σ 2 / a n , the weights σ i 2 a i \sigma_i^2a_i σ i 2 a i are all equal to σ 2 \sigma^2 σ 2 , so that Σ n = σ 2 ∑ i > n ( V i − 1 − V i ) = σ 2 V n \Sigma_n=\sigma^2\sum_{i>n}(V_{i-1}-V_i)=\sigma^2V_n Σ n = σ 2 ∑ i > n ( V i − 1 − V i ) = σ 2 V n , where the last equality is shown in the proof of Lemma A.3.
Proof of Proposition 3
Both limits follow from Equation (5) and the bounds of Lemma A.3. For V n V_n V n , by Equation (5) , n − 1 V n − 1 = n − 1 V 0 − 1 + n − 1 ∑ i = 1 n x i x i ⊤ → Ω n^{-1}V_n^{-1}=n^{-1}V_0^{-1}+n^{-1}\sum_{i=1}^{n}x_ix_i^\top\to\Omega n − 1 V n − 1 = n − 1 V 0 − 1 + n − 1 ∑ i = 1 n x i x i ⊤ → Ω , and therefore, by continuity of matrix inversion, n V n → Ω − 1 nV_n\to\Omega^{-1} n V n → Ω − 1 as n → ∞ n\to\infty n → ∞ .
For Σ n \Sigma_n Σ n , multiplying by n n n the inequalities of Lemma A.3 gives
s ‾ n ( 1 − ϵ n ) n V n ⪯ n Σ n ⪯ s ‾ n + n V n . \overline{s}_n(1-\epsilon_n)nV_n\preceq n\Sigma_n\preceq\overline{s}_n^{+}nV_n. s n ( 1 − ϵ n ) n V n ⪯ n Σ n ⪯ s n + n V n .
As n → ∞ n\to\infty n → ∞ , ϵ n → 0 \epsilon_n\to0 ϵ n → 0 by Lemma A.2, s ‾ n − → σ 2 \overline{s}_n^{-}\to\sigma^2 s n − → σ 2 and s ‾ n + → σ 2 \overline{s}_n^{+}\to\sigma^2 s n + → σ 2 , since they are the infimum and the supremum of the tail ( σ i 2 ) i > n (\sigma_i^2)_{i>n} ( σ i 2 ) i > n of a sequence that converges to σ 2 \sigma^2 σ 2 by Assumption 3, and n V n → Ω − 1 nV_n\to\Omega^{-1} n V n → Ω − 1 . Hence both bounds converge to σ 2 Ω − 1 \sigma^2\Omega^{-1} σ 2 Ω − 1 , so that u ⊤ n Σ n u → σ 2 u ⊤ Ω − 1 u u^\top n\Sigma_nu\to\sigma^2u^\top\Omega^{-1}u u ⊤ n Σ n u → σ 2 u ⊤ Ω − 1 u for every u ∈ R p u\in\mathbb{R}^p u ∈ R p . Since the matrices are symmetric, this implies n Σ n → σ 2 Ω − 1 n\Sigma_n\to\sigma^2\Omega^{-1} n Σ n → σ 2 Ω − 1 .
Proof of Theorem 2
Asymptotic normality of martingale posteriors is typically derived from martingale central limit theorems ([23 , 20 ] ). Here, the terms of the series in Equation (6) of the paper are independent, and we apply the Lindeberg–Feller theorem to the standardized version of β ~ − β n \widetilde{\beta}-\beta_n β − β n .
Step 1: representation. By Equation (6) of the paper, derived at the end of the proof of Proposition 1,
β ~ − β n = ∑ i > n w i ξ i , and ∑ i > n w i w i ⊤ = ∑ i > n σ i 2 a i ( V i − 1 − V i ) = Σ n , \widetilde{\beta}-\beta_n=\sum_{i>n}w_i\xi_i,\qquad\text{and}\qquad\sum_{i>n}w_iw_i^\top=\sum_{i>n}\sigma_i^2a_i(V_{i-1}-V_i)=\Sigma_n, β − β n = i > n ∑ w i ξ i , and i > n ∑ w i w i ⊤ = i > n ∑ σ i 2 a i ( V i − 1 − V i ) = Σ n ,
with deterministic weights w i = σ i a i V i − 1 x i w_i=\sigma_ia_iV_{i-1}x_i w i = σ i a i V i − 1 x i , ξ i \xi_i ξ i iid from P ξ P_\xi P ξ , and the series converging almost surely and in L 2 L_2 L 2 . Since β ~ − β n \widetilde{\beta}-\beta_n β − β n is independent of y 1 : n y_{1:n} y 1 : n , its conditional distribution given y 1 : n y_{1:n} y 1 : n coincides with its distribution under P x 1 : ∞ P_{x_{1:\infty}} P x 1 : ∞ , and it suffices to prove convergence in distribution.
Step 2: reduction to the Lindeberg condition. Consider the standardized vector
W n : = Σ n − 1 / 2 ( β ~ − β n ) = ∑ i > n Σ n − 1 / 2 w i ξ i = ∑ i > n v n , i ξ i , v n , i : = Σ n − 1 / 2 w i , W_n:=\Sigma_n^{-1/2}(\widetilde{\beta}-\beta_n)=\sum_{i>n}\Sigma_n^{-1/2}w_i\xi_i=\sum_{i>n}v_{n,i}\xi_i,\qquad v_{n,i}:=\Sigma_n^{-1/2}w_i, W n := Σ n − 1/2 ( β − β n ) = i > n ∑ Σ n − 1/2 w i ξ i = i > n ∑ v n , i ξ i , v n , i := Σ n − 1/2 w i ,
where Σ n \Sigma_n Σ n is invertible by Lemma A.3 and the second equality follows from Equation (6) of the paper. By Step 1, ∑ i > n v n , i v n , i ⊤ = Σ n − 1 / 2 Σ n Σ n − 1 / 2 = I p \sum_{i>n}v_{n,i}v_{n,i}^{\top}=\Sigma_n^{-1/2}\Sigma_n\Sigma_n^{-1/2}=I_p ∑ i > n v n , i v n , i ⊤ = Σ n − 1/2 Σ n Σ n − 1/2 = I p . The Lindeberg–Feller theorem concerns sums of finitely many independent terms, whereas W n W_n W n is an infinite series. We therefore split W n W_n W n into a finite sum, to which the theorem applies, and a remainder, which we show to be negligible. Taking traces in the identity ∑ i > n v n , i v n , i T = I p \sum_{i>n}\mathbf{v}_{n,i}\mathbf{v}_{n,i}^{\mathsf T}=I_p ∑ i > n v n , i v n , i T = I p , and recalling that ∥ v ∥ 2 = tr ( v v T ) \lVert v\rVert^2=\operatorname{tr}(vv^{\mathsf T}) ∥ v ∥ 2 = tr ( v v T ) , gives ∑ i > n ∥ v n , i ∥ 2 = tr ( I p ) = p < ∞ \sum_{i>n}\lVert\mathbf{v}_{n,i}\rVert^2=\operatorname{tr}(I_p)=p<\infty ∑ i > n ∥ v n , i ∥ 2 = tr ( I p ) = p < ∞ . As this series converges, we can choose N n > n N_n>n N n > n such that ∑ i > N n ∥ v n , i ∥ 2 ≤ n − 1 \sum_{i>N_n}\lVert\mathbf{v}_{n,i}\rVert^2\le n^{-1} ∑ i > N n ∥ v n , i ∥ 2 ≤ n − 1 , and write W n = T n + U n W_n=T_n+U_n W n = T n + U n , with
T n : = ∑ i = n + 1 N n v n , i ξ i and U n : = ∑ i > N n v n , i ξ i . T_n:=\sum_{i=n+1}^{N_n}\mathbf{v}_{n,i}\xi_i
\qquad\text{and}\qquad
U_n:=\sum_{i>N_n}\mathbf{v}_{n,i}\xi_i. T n := i = n + 1 ∑ N n v n , i ξ i and U n := i > N n ∑ v n , i ξ i .
Any bound that tends to 0 would do in place of n − 1 n^{-1} n − 1 : it makes the remainder vanish.
For the remainder, the terms v n , i ξ i \mathbf{v}_{n,i}\xi_i v n , i ξ i have zero mean and are uncorrelated, because the ξ i \xi_i ξ i are iid with zero mean and unit variance. Hence, the partial sums of U n U_n U n have expected squared norm equal to the corresponding partial sums of ∥ v n , i ∥ 2 \lVert\mathbf{v}_{n,i}\rVert^2 ∥ v n , i ∥ 2 , and, passing to the limit by the L 2 L_2 L 2 convergence of the series in Step 1, E ( ∥ U n ∥ 2 ) = ∑ i > N n ∥ v n , i ∥ 2 ≤ n − 1 \mathbb{E}(\lVert U_n\rVert^2)=\sum_{i>N_n}\lVert\mathbf{v}_{n,i}\rVert^2\le n^{-1} E (∥ U n ∥ 2 ) = ∑ i > N n ∥ v n , i ∥ 2 ≤ n − 1 . This shows that U n → 0 U_n\to0 U n → 0 in L 2 L_2 L 2 , and therefore in probability, as n → ∞ n\to\infty n → ∞ .
For the finite part, T n T_n T n is a sum of the N n − n N_n-n N n − n independent vectors v n , i ξ i \mathbf{v}_{n,i}\xi_i v n , i ξ i , i = n + 1 , … , N n i=n+1,\ldots,N_n i = n + 1 , … , N n , which have zero mean and covariance matrices v n , i v n , i T \mathbf{v}_{n,i}\mathbf{v}_{n,i}^{\mathsf T} v n , i v n , i T . These vectors form a triangular array, as they change with n n n . Since ∑ i > n v n , i v n , i T = I p \sum_{i>n}\mathbf{v}_{n,i}\mathbf{v}_{n,i}^{\mathsf T}=I_p ∑ i > n v n , i v n , i T = I p , their total covariance is
∑ i = n + 1 N n v n , i v n , i T = I p − ∑ i > N n v n , i v n , i T ⟶ I p , \sum_{i=n+1}^{N_n}\mathbf{v}_{n,i}\mathbf{v}_{n,i}^{\mathsf T}
=I_p-\sum_{i>N_n}\mathbf{v}_{n,i}\mathbf{v}_{n,i}^{\mathsf T}
\longrightarrow I_p, i = n + 1 ∑ N n v n , i v n , i T = I p − i > N n ∑ v n , i v n , i T ⟶ I p ,
because the last sum is positive semidefinite with trace ∑ i > N n ∥ v n , i ∥ 2 ≤ n − 1 \sum_{i>N_n}\lVert\mathbf{v}_{n,i}\rVert^2\le n^{-1} ∑ i > N n ∥ v n , i ∥ 2 ≤ n − 1 , and each entry of a positive semidefinite matrix is bounded in absolute value by its trace. By the Lindeberg–Feller theorem ([38 Proposition 2.27] ), if
∑ i = n + 1 N n E { ∥ v n , i ξ i ∥ 2 1 ( ∥ v n , i ξ i ∥ > ϵ ) } → 0 (10) \sum_{i=n+1}^{N_n}\mathbb{E}\{\lVert\mathbf{v}_{n,i}\xi_i\rVert^2\mathbf{1}(\lVert\mathbf{v}_{n,i}\xi_i\rVert>\epsilon)\}\to0 \tag*{(10)} i = n + 1 ∑ N n E {∥ v n , i ξ i ∥ 2 1 (∥ v n , i ξ i ∥ > ϵ )} → 0 (10)
for every ϵ > 0 \epsilon>0 ϵ > 0 as n → ∞ n\to\infty n → ∞ , then T n → Z T_n\to Z T n → Z in distribution, with Z ∼ N p ( 0 , I p ) Z\sim N_p(0,I_p) Z ∼ N p ( 0 , I p ) . Together with U n → 0 U_n\to0 U n → 0 in probability, Slutsky’s theorem then gives W n = T n + U n → Z W_n=T_n+U_n\to Z W n = T n + U n → Z in distribution.
Step 3: the Lindeberg condition. Let ρ n : = sup i > n w i T Σ n − 1 w i \rho_n:=\sup_{i>n}\mathbf{w}_i^{\mathsf T}\Sigma_n^{-1}\mathbf{w}_i ρ n := sup i > n w i T Σ n − 1 w i , the largest weight of a single term. Since ∥ Σ n − 1 / 2 w i ξ i ∥ 2 = ∥ Σ n − 1 / 2 w i ∥ 2 ξ i 2 = w i T Σ n − 1 w i ξ i 2 \lVert\Sigma_n^{-1/2}\mathbf{w}_i\xi_i\rVert^2=\lVert\Sigma_n^{-1/2}\mathbf{w}_i\rVert^2\xi_i^2=\mathbf{w}_i^{\mathsf T}\Sigma_n^{-1}\mathbf{w}_i\xi_i^2 ∥ Σ n − 1/2 w i ξ i ∥ 2 = ∥ Σ n − 1/2 w i ∥ 2 ξ i 2 = w i T Σ n − 1 w i ξ i 2 , and ∥ Σ n − 1 / 2 w i ξ i ∥ ≥ 0 \lVert\Sigma_n^{-1/2}\mathbf{w}_i\xi_i\rVert\ge0 ∥ Σ n − 1/2 w i ξ i ∥ ≥ 0 ,
∑ i = n + 1 N n E { ∥ v n , i ξ i ∥ 2 1 ( ∥ v n , i ξ i ∥ > ϵ ) } ≤ ∑ i > n E { ∥ Σ n − 1 / 2 w i ξ i ∥ 2 1 ( ∥ Σ n − 1 / 2 w i ξ i ∥ > ϵ ) } = ∑ i > n w i T Σ n − 1 w i E { ξ i 2 1 ( w i T Σ n − 1 w i ξ i 2 > ϵ 2 ) } ≤ ∑ i > n w i T Σ n − 1 w i E { ξ i 2 1 ( ρ n ξ i 2 > ϵ 2 ) } = ∑ i > n w i T Σ n − 1 w i E { ξ i 2 1 ( ξ i 2 > ϵ 2 / ρ n ) } = p E { ξ 2 1 ( ξ 2 > ϵ 2 / ρ n ) } , \begin{aligned}
\sum_{i=n+1}^{N_n}\mathbb{E}\{\lVert\mathbf{v}_{n,i}\xi_i\rVert^2\mathbf{1}(\lVert\mathbf{v}_{n,i}\xi_i\rVert>\epsilon)\}
&\le\sum_{i>n}\mathbb{E}\{\lVert\Sigma_n^{-1/2}\mathbf{w}_i\xi_i\rVert^2\mathbf{1}(\lVert\Sigma_n^{-1/2}\mathbf{w}_i\xi_i\rVert>\epsilon)\}\\
&=\sum_{i>n}\mathbf{w}_i^{\mathsf T}\Sigma_n^{-1}\mathbf{w}_i\mathbb{E}\{\xi_i^2\mathbf{1}(\mathbf{w}_i^{\mathsf T}\Sigma_n^{-1}\mathbf{w}_i\xi_i^2>\epsilon^2)\}\\
&\le\sum_{i>n}\mathbf{w}_i^{\mathsf T}\Sigma_n^{-1}\mathbf{w}_i\mathbb{E}\{\xi_i^2\mathbf{1}(\rho_n\xi_i^2>\epsilon^2)\}\\
&=\sum_{i>n}\mathbf{w}_i^{\mathsf T}\Sigma_n^{-1}\mathbf{w}_i\mathbb{E}\{\xi_i^2\mathbf{1}(\xi_i^2>\epsilon^2/\rho_n)\}\\
&=p\mathbb{E}\{\xi^2\mathbf{1}(\xi^2>\epsilon^2/\rho_n)\},
\end{aligned} i = n + 1 ∑ N n E {∥ v n , i ξ i ∥ 2 1 (∥ v n , i ξ i ∥ > ϵ )} ≤ i > n ∑ E {∥ Σ n − 1/2 w i ξ i ∥ 2 1 (∥ Σ n − 1/2 w i ξ i ∥ > ϵ )} = i > n ∑ w i T Σ n − 1 w i E { ξ i 2 1 ( w i T Σ n − 1 w i ξ i 2 > ϵ 2 )} ≤ i > n ∑ w i T Σ n − 1 w i E { ξ i 2 1 ( ρ n ξ i 2 > ϵ 2 )} = i > n ∑ w i T Σ n − 1 w i E { ξ i 2 1 ( ξ i 2 > ϵ 2 / ρ n )} = p E { ξ 2 1 ( ξ 2 > ϵ 2 / ρ n )} ,
because E { ξ i 2 1 ( ξ i 2 > ϵ 2 / ρ n ) } \mathbb{E}\{\xi_i^2\mathbf{1}(\xi_i^2>\epsilon^2/\rho_n)\} E { ξ i 2 1 ( ξ i 2 > ϵ 2 / ρ n )} is the same for every i > n i>n i > n and ∑ i > n w i ⊤ Σ n − 1 w i = tr ( Σ n − 1 Σ n ) = p \sum_{i>n} w_i^\top\Sigma_n^{-1}w_i=\operatorname{tr}(\Sigma_n^{-1}\Sigma_n)=p ∑ i > n w i ⊤ Σ n − 1 w i = tr ( Σ n − 1 Σ n ) = p . Since E ( ξ 2 ) = 1 < ∞ \mathbb{E}(\xi^2)=1<\infty E ( ξ 2 ) = 1 < ∞ , the last expectation tends to 0 0 0 by dominated convergence whenever ϵ 2 / ρ n → ∞ \epsilon^2/\rho_n\to\infty ϵ 2 / ρ n → ∞ . Therefore, (A.1) is satisfied if ρ n → 0 \rho_n\to0 ρ n → 0 as n → ∞ n\to\infty n → ∞ .
To show that ρ n → 0 \rho_n\to0 ρ n → 0 , let ϵ n \epsilon_n ϵ n and s ˉ n \bar{s}_n s ˉ n be as in Lemma A.3, and take n n n large enough that ϵ n < 1 \epsilon_n<1 ϵ n < 1 . By Lemma A.3, Σ n ⪰ s ˉ n ( 1 − ϵ n ) V n \Sigma_n\succeq\bar{s}_n(1-\epsilon_n)V_n Σ n ⪰ s ˉ n ( 1 − ϵ n ) V n , where both sides are positive definite. Since inversion reverses the ordering between positive definite matrices, that is, A ⪯ B A\preceq B A ⪯ B implies B − 1 ⪯ A − 1 B^{-1}\preceq A^{-1} B − 1 ⪯ A − 1 , we obtain
Σ n − 1 ⪯ { s ˉ n ( 1 − ϵ n ) } − 1 V n − 1 . \Sigma_n^{-1}\preceq\{\bar{s}_n(1-\epsilon_n)\}^{-1}V_n^{-1}. Σ n − 1 ⪯ { s ˉ n ( 1 − ϵ n ) } − 1 V n − 1 .
Moreover, for i > n i>n i > n , V i − 1 ⪯ V n V_{i-1}\preceq V_n V i − 1 ⪯ V n by Lemma A.1, and thus, for the inverses, V n − 1 ⪯ V i − 1 − 1 V_n^{-1}\preceq V_{i-1}^{-1} V n − 1 ⪯ V i − 1 − 1 . Multiplying on both sides by the symmetric matrix V i − 1 − 1 V_{i-1}^{-1} V i − 1 − 1 , which preserves the ordering since A ⪯ B A\preceq B A ⪯ B implies C A C ⪯ C B C CAC\preceq CBC C A C ⪯ C B C for every symmetric C C C , we obtain V i − 1 − 1 V n − 1 V i − 1 − 1 ⪯ V i − 1 − 1 V i − 1 − 1 V i − 1 − 1 = V i − 1 − 1 V_{i-1}^{-1}V_n^{-1}V_{i-1}^{-1}\preceq V_{i-1}^{-1}V_{i-1}^{-1}V_{i-1}^{-1}=V_{i-1}^{-1} V i − 1 − 1 V n − 1 V i − 1 − 1 ⪯ V i − 1 − 1 V i − 1 − 1 V i − 1 − 1 = V i − 1 − 1 .
Therefore, since a i ≤ 1 a_i\le1 a i ≤ 1 and σ i ≤ σ ∗ \sigma_i\le\sigma_* σ i ≤ σ ∗ ,
w i ⊤ Σ n − 1 w i ≤ σ ∗ 2 s ˉ n ( 1 − ϵ n ) x i ⊤ V i − 1 − 1 V n − 1 V i − 1 − 1 x i ≤ σ ∗ 2 s ˉ n ( 1 − ϵ n ) x i ⊤ V i − 1 − 1 x i ≤ σ ∗ 2 s ˉ n ( 1 − ϵ n ) ϵ n . w_i^\top\Sigma_n^{-1}w_i\le\frac{\sigma_*^2}{\bar{s}_n(1-\epsilon_n)}x_i^\top V_{i-1}^{-1}V_n^{-1}V_{i-1}^{-1}x_i\le\frac{\sigma_*^2}{\bar{s}_n(1-\epsilon_n)}x_i^\top V_{i-1}^{-1}x_i\le\frac{\sigma_*^2}{\bar{s}_n(1-\epsilon_n)}\epsilon_n. w i ⊤ Σ n − 1 w i ≤ s ˉ n ( 1 − ϵ n ) σ ∗ 2 x i ⊤ V i − 1 − 1 V n − 1 V i − 1 − 1 x i ≤ s ˉ n ( 1 − ϵ n ) σ ∗ 2 x i ⊤ V i − 1 − 1 x i ≤ s ˉ n ( 1 − ϵ n ) σ ∗ 2 ϵ n .
so ρ n → 0 \rho_n\to0 ρ n → 0 , because ϵ n → 0 \epsilon_n\to0 ϵ n → 0 by Lemma A.2 and s ˉ n → σ 2 > 0 \bar{s}_n\to\sigma^2>0 s ˉ n → σ 2 > 0 by Assumption 3. This proves the first statement.
Step 4: the second statement. We can write σ n − 1 / 2 V n − 1 / 2 ( β ~ − β n ) = G n W n \sigma_n^{-1/2}V_n^{-1/2}(\widetilde{\beta}-\beta_n)=G_nW_n σ n − 1/2 V n − 1/2 ( β − β n ) = G n W n , with
G n : = σ − 1 V n − 1 / 2 Σ n 1 / 2 = ( σ n 2 V n ) − 1 / 2 ( n Σ n ) 1 / 2 ⟶ ( σ 2 Ω − 1 ) − 1 / 2 ( σ 2 Ω − 1 ) 1 / 2 = I p G_n:=\sigma^{-1}V_n^{-1/2}\Sigma_n^{1/2}=(\sigma_n^2V_n)^{-1/2}(n\Sigma_n)^{1/2}\longrightarrow(\sigma^2\Omega^{-1})^{-1/2}(\sigma^2\Omega^{-1})^{1/2}=I_p G n := σ − 1 V n − 1/2 Σ n 1/2 = ( σ n 2 V n ) − 1/2 ( n Σ n ) 1/2 ⟶ ( σ 2 Ω − 1 ) − 1/2 ( σ 2 Ω − 1 ) 1/2 = I p
by Proposition 3 and the continuity of the matrix square root. Since W n → Z W_n\to Z W n → Z in distribution, Slutsky’s theorem gives G n W n → Z G_nW_n\to Z G n W n → Z in distribution, which is the second statement.
Proof of Theorem 3
By Equation (6) of the paper, β ~ = β n + ∑ i > n w i ξ i \widetilde{\beta}=\beta_n+\sum_{i>n}w_i\xi_i β = β n + ∑ i > n w i ξ i , where ξ i ∼ i i d N ( 0 , 1 ) \xi_i\mathrel{\overset{\mathrm{iid}}{\sim}}\mathcal{N}(0,1) ξ i ∼ iid N ( 0 , 1 ) in this case, so that w i ξ i ∼ N p ( 0 , w i w i ⊤ ) w_i\xi_i\sim\mathcal{N}_p(0,w_iw_i^\top) w i ξ i ∼ N p ( 0 , w i w i ⊤ ) . Since w i w_i w i only depends on the known covariates, the vectors w i ξ i w_i\xi_i w i ξ i , i ≥ 1 i\ge1 i ≥ 1 , are independent, and those with i > n i>n i > n are independent of y 1 : n y_{1:n} y 1 : n . Each partial sum is a finite sum of independent Gaussian vectors, hence ∑ i = n + 1 N w i ξ i ∼ N p ( 0 , Σ n − Σ N ) \sum_{i=n+1}^{N}w_i\xi_i\sim\mathcal{N}_p(0,\Sigma_n-\Sigma_N) ∑ i = n + 1 N w i ξ i ∼ N p ( 0 , Σ n − Σ N ) , with characteristic function exp { − t ⊤ ( Σ n − Σ N ) t / 2 } \exp\{-t^\top(\Sigma_n-\Sigma_N)t/2\} exp { − t ⊤ ( Σ n − Σ N ) t /2 } . As N → ∞ N\to\infty N → ∞ , the partial sums converge in L 2 L_2 L 2 , hence in distribution, and their characteristic functions converge to exp ( − t ⊤ Σ n t / 2 ) \exp(-t^\top\Sigma_nt/2) exp ( − t ⊤ Σ n t /2 ) , because Σ N = ∑ i > N w i w i ⊤ \Sigma_N=\sum_{i>N}w_iw_i^\top Σ N = ∑ i > N w i w i ⊤ is the tail of a convergent series and thus tends to 0 0 0 . Thus the limit ∑ i > n w i ξ i \sum_{i>n}w_i\xi_i ∑ i > n w i ξ i is distributed as N p ( 0 , Σ n ) \mathcal{N}_p(0,\Sigma_n) N p ( 0 , Σ n ) . Therefore, β ~ ∣ y 1 : n ∼ N p ( β n , Σ n ) \widetilde{\beta}\mid y_{1:n}\sim\mathcal{N}_p(\beta_n,\Sigma_n) β ∣ y 1 : n ∼ N p ( β n , Σ n ) for any n ≥ 1 n\ge1 n ≥ 1 and, taking n = 0 n=0 n = 0 , β ~ ∼ N p ( β 0 , Σ 0 ) \widetilde{\beta}\sim\mathcal{N}_p(\beta_0,\Sigma_0) β ∼ N p ( β 0 , Σ 0 ) . The result for σ n 2 = σ 2 / a n \sigma_n^2=\sigma^2/a_n σ n 2 = σ 2 / a n follows as a special case, since Σ n = σ 2 V n \Sigma_n=\sigma^2V_n Σ n = σ 2 V n by Proposition 2.
Additional simulations
Comparison between uncertainty quantification methods with Student-t t t predictive distribution
In this section, we consider a linear regression setting in which the response follows a location-scale Student-t t t distribution with ν = 5 \nu= 5 ν = 5 degrees of freedom and scale parameter ϕ = 2 \phi= 2 ϕ = 2 , both treated as known. All other settings are as in § 5.1 of the paper. For each scenario, we generate 30 datasets. In place of the exact posterior, which is no longer Gaussian, we use the Gaussian distribution N p ( β n , Σ n ) \mathcal{N}_{p}(\boldsymbol{\beta}_{n}, \boldsymbol{\Sigma}_{n}) N p ( β n , Σ n ) with the exact posterior mean and covariance, where σ 2 = ϕ 2 ν / ( ν − 2 ) \sigma^{2} = \phi^{2}\nu/(\nu- 2) σ 2 = ϕ 2 ν / ( ν − 2 ) (APX); when σ n 2 = σ 2 a n − 1 \sigma_{n}^{2} = \sigma^{2}a_{n}^{-1} σ n 2 = σ 2 a n − 1 , this is the Gaussian approximation in Equation (7) of the paper. Its B B B draws provide the Monte Carlo reference for the errors. The results, reported in Table 3 , are consistent with those for the Gaussian case. The errors of PS-1 and PS-2 are comparable with the Monte Carlo errors of APX, and the widths and coverages of the credible intervals from predictive resampling essentially coincide with those of APX. The coverage is nearly nominal when σ n 2 = σ 2 a n − 1 \sigma_{n}^{2} = \sigma^{2}a_{n}^{-1} σ n 2 = σ 2 a n − 1 ; when σ n 2 = σ 2 \sigma_{n}^{2} = \sigma^{2} σ n 2 = σ 2 , the intervals are narrower and the coverage is slightly lower for n = 3 p n = 3p n = 3 p . Finally, PS-2 is about 40 to 50 times faster than PS-1.
Table 3: Comparison of uncertainty quantification from predictive resampling with Student-t t t responses, with Algorithms 1 (PS-1) and 2 (PS-2) of the paper and with B B B draws from the Gaussian distribution N p ( β n , Σ n ) \mathcal{N}_{p}(\beta_{n}, \Sigma_{n}) N p ( β n , Σ n ) (APX): logarithm of the mean squared error of the posterior mean, log ( ∥ β ^ − β n ∥ 2 / p ) \log(\lVert\widehat{\beta}-\beta_{n}\rVert^{2}/p) log (∥ β − β n ∥ 2 / p ) , logarithm of the Frobenius norm of the lower-triangular part of σ − 2 ( Σ ^ − Σ n ) \sigma^{-2}(\widehat{\Sigma}-\Sigma_{n}) σ − 2 ( Σ − Σ n ) , average width and coverage of the 95% intervals for the p p p regression coefficients, and computation time in seconds. For APX, the errors are Monte Carlo errors and the intervals are the corresponding Gaussian ones. Values are averages over the simulated datasets, with standard deviations in parentheses.
Table 3.
Comparison with frequentist additive models
In this section, we compare the proposed method with frequentist additive models (AM), using the same settings as in the simulation study of § 5.3 of the paper. We consider two versions of the additive model, both fitted with the R package mgcv: one with a Gaussian likelihood (AM-n n n ) and one with a location-scale Student-t t t likelihood (AM-t t t ). We also include the Gaussian approximation to the martingale posterior given in Equation (7) of the paper (APX). The methods are evaluated in terms of the mean squared error (MSE) between the estimate of E ( Y ∣ x ) \mathrm{E}(Y \mid x) E ( Y ∣ x ) and its true value on the test set, the average coverage and width of the 95% intervals for E ( Y ∣ x ) \mathrm{E}(Y \mid x) E ( Y ∣ x ) on the test set, and the total computation time. For the frequentist AM, these intervals are confidence intervals rather than credible intervals.
The results are reported in Table 4 . AM-t t t yields the most accurate point estimates in both scenarios. The RLS posterior mean has exactly the same form as the AM-n n n point estimate. The two differ only in how the smoothing parameters are selected: RLS maximizes the LOO-PLL, whereas AM-n n n uses restricted maximum likelihood [41 ] . In both scenarios, all methods have coverage close to the nominal level, and the intervals obtained by predictive resampling are slightly narrower than those of the Gaussian approximation. Finally, our method is faster than mgcv: the Gaussian approximation, which only requires the posterior mean, is more than ten times faster than both AM-t t t and AM-n n n , and RLS-t t t , including predictive resampling, is more than four times faster than AM-n n n .
Table 4. Comparison between our proposed method with Student-t t t predictive distribution, with predictive resampling (RLS-t t t ) and with the Gaussian approximation in Equation (7) of the paper (APX), and frequentist additive models with Student-t t t (AM-t t t ) and Gaussian (AM-n n n ) likelihoods, in terms of out-of-sample MSE, average coverage and width of 95% intervals for E ( Y ∣ x ) E(Y \mid x) E ( Y ∣ x ) , and total computation time in seconds. Values are averages over the simulated datasets, with standard deviations in parentheses.
Additional details on effective sample size in simulations
In this section, we detail how the lengths of the MCMC chains were chosen for the traditional Bayesian additive model and BART in the simulation study of Section 5.3 of the paper. We set the number of martingale posterior draws to B = 5,000 B=5{,}000 B = 5 , 000 . Since Algorithm 2 of the paper runs separate forward chains of length N N N , the martingale posterior draws are independent by construction, given the shared future covariates, so their effective sample size (ESS) equals B B B . The forward length N N N can be chosen as described in Section 3.4 of the paper; Figure 8 shows a sample path of a forward chain for the analysis of Section 6 of the paper. For the competitors, we then chose the chain length and burn-in so that the average ESS of the posterior draws of f ( x ) f(x) f ( x ) across the test set is close to B B B , when possible. For traditional Bayes, five chains of 2,000 draws each, with the first 1,000 discarded as burn-in, yield an ESS close to B B B . BART mixes poorly, and its ESS remains far below B B B also with long chains; we run it for 55,000 iterations, discarding the first 50,000 as burn-in. The results are reported in Table 5 .
Table 5. Average effective sample size for the posterior draws of f ( x ) f(x) f ( x ) across the test set in the simulation study of Section 5.3 of the paper, for our proposed methods with Student-t t t (RLS-t t t ) and Gaussian (RLS-n n n ) predictive distributions, the traditional Bayesian linear model with Student-t t t likelihood (BLM) and BART. Values are averages over the simulated datasets, with standard deviations in parentheses.
Pipeline for selecting the predictive distribution
The validity of inference under the proposed method requires that the predictive distribution adequately reflect the unknown data-generating process. In this section, we describe a pipeline for selecting an appropriate predictive distribution. We illustrate it with the NHANES serum triglycerides dataset analyzed in Section 6 of the paper. Although some choices at each step are specific to this application, the general ideas behind the pipeline apply more broadly.
As an exploratory check of homoscedasticity, we divide the observations of the training set into quartiles of each continuous covariate, each containing between 16 and 26 observations, and compute the variance of log-triglycerides within each quartile; for the income ratio, which is top-coded at 5, the last group consists of the 26 top-coded observations. Ethnicity is not considered, since four of its six categories have fewer than ten observations. The results are reported in Figure 3 . The variance of the response varies little across covariates and quartiles, remaining between 0.15 and 0.35 in all but three cases (0.12 and 0.39 for BMI, 0.44 for age), which are plausibly due to the small number of observations in each quartile. We therefore consider homoscedasticity a reasonable assumption for the response.
Figure 3. Variance of log-triglycerides computed across quartiles of each continuous covariate.
Next, we assess whether a Gaussian predictive distribution is adequate or whether a heavier-tailed distribution is needed. To do so, we fit a frequentist additive model with a Gaussian likelihood using mgcv and compare the quantiles of the resulting residuals with the theoretical quantiles of candidate distributions. As shown in Figure 4 , for the NHANES serum triglycerides dataset, the residual quantiles are more aligned with those of a Student-t t t distribution with 9 degrees of freedom than with those of a Gaussian distribution. We therefore adopt a Student-t t t predictive distribution.
Figure 4. Empirical quantiles of the residuals plotted against the theoretical ones of a Student-t t t distribution with 9 degrees of freedom (left) and a Gaussian distribution (right).
Finally, we consider whether a skew Student-t t t distribution might be more appropriate than a symmetric one. We then maximize the LOO-PLL with P n P_n P n taken to be a skew Student-t t t distribution. The estimated degrees of freedom and scale are 13.65 and 0.36, respectively, very close to the estimates under the symmetric Student-t t t reported in the main paper. The estimated skewness parameter is −0.007, thereby indicating that a symmetric Student-t t t predictive distribution is sufficient.
Analysis of NHANES serum triglycerides dataset: additional figures
This section contains additional figures related to the analysis of NHANES serum triglycerides dataset in § 6 of the paper.
Figure 5. Posterior distributions of the intercept and of the regression coefficients of the ethnicity indicators, with non-Hispanic white as the baseline, obtained with traditional and predictive Bayes.
Figure 6. Estimated covariate-specific effects with pointwise 95% highest posterior density bands, obtained using the traditional Bayesian model, and partial residuals. The bands are truncated to the range of the partial residuals.
Figure 7. Estimated covariate-specific effects with pointwise 95% confidence bands, obtained using a frequentist additive model with Student-t t t likelihood, and partial residuals. The bands are truncated to the range of the partial residuals.
Figure 8. One sample path of the squared Euclidean distance ∥ β m − β n ∥ 2 \lVert\beta_m-\beta_n\rVert^2 ∥ β m − β n ∥ 2 between β m \beta_m β m , m = n + 1 , … , n + 400,000 m=n+1,\ldots,n+400{,}000 m = n + 1 , … , n + 400 , 000 , and the initial β n \beta_n β n .
Computational details
This section describes how the posterior mean, the LOO-PLL of § 4.3 of the paper and its gradient, and the predictive resampling of Algorithm 2 are computed in our implementation. We present them for the additive models of § 4.1 of the paper, with features z i ∈ R r z_i \in\mathbb{R}^r z i ∈ R r and V 0 = ( Λ + ε I r ) − 1 V_0=(\Lambda+\varepsilon I_r)^{-1} V 0 = ( Λ + ε I r ) − 1 ; the linear model is the special case without smoothing parameters.
Posterior mean and leave-one-out prequential log-likelihood
The recursion of Equations (3) –(4) of the paper is not needed on the observed data, since its output is available in closed form. Recalling Equation (8) of the paper, V n − 1 = A V_n^{-1}=A V n − 1 = A and β n = A − 1 Z ⊤ y \beta_n=A^{-1}Z^\top y β n = A − 1 Z ⊤ y , where
A : = Λ + ε I r + Z ⊤ Z , A := \Lambda+\varepsilon I_r+Z^\top Z, A := Λ + ε I r + Z ⊤ Z ,
Z ∈ R n × r Z\in\mathbb{R}^{n\times r} Z ∈ R n × r has rows z 1 ⊤ , … , z n ⊤ z_1^\top,\ldots,z_n^\top z 1 ⊤ , … , z n ⊤ and y = ( y 1 , … , y n ) ⊤ y=(y_1,\ldots,y_n)^\top y = ( y 1 , … , y n ) ⊤ . The matrix Z ⊤ Z Z^\top Z Z ⊤ Z and the vector Z ⊤ y Z^\top y Z ⊤ y do not depend on the hyperparameters, so they are computed once for each dataset. For given smoothing parameters, we compute the Cholesky factorization A = R ⊤ R A=R^\top R A = R ⊤ R , with R R R upper triangular, and obtain β n \beta_n β n by two triangular solves. The leverages needed by the LOO-PLL are h i = z i ⊤ V n z i = ∥ R − ⊤ z i ∥ 2 h_i=z_i^\top V_nz_i=\lVert R^{-\top}z_i\rVert^2 h i = z i ⊤ V n z i = ∥ R − ⊤ z i ∥ 2 , so they follow from one further triangular solve with right-hand side Z ⊤ Z^\top Z ⊤ . The leave-one-out predictions are then z i ⊤ β − i = y i − ( y i − z i ⊤ β n ) / ( 1 − h i ) z_i^\top\beta_{-i}=y_i-(y_i-z_i^\top\beta_n)/(1-h_i) z i ⊤ β − i = y i − ( y i − z i ⊤ β n ) / ( 1 − h i ) , as in § 4.3 of the paper. Each evaluation of the LOO-PLL therefore costs O ( r 3 + n r 2 ) O(r^3+nr^2) O ( r 3 + n r 2 ) operations.
Gradient and maximization
The hyperparameters are selected by maximizing the LOO-PLL over the logarithms of the smoothing parameters λ 1 , … , λ p \lambda_1,\ldots,\lambda_p λ 1 , … , λ p and of the scale and shape parameters of P n P_n P n , with the quasi-Newton routine nlminb of R. For the additive models of § 5.3 and § 6 of the paper, we supply the analytic gradient of the LOO-PLL, so that no finite differences are needed; the linear models of § 5.2 , whose only hyperparameters are the two or three parameters of P n P_n P n , use the finite differences of nlminb. The gradient follows from the chain rule, starting from the differentials of V n V_n V n , β n \beta_n β n and h i h_i h i with respect to log λ j \log\lambda_j log λ j . Since A A A depends on λ j \lambda_j λ j only through the block λ j Λ j \lambda_j\Lambda_j λ j Λ j of Λ \Lambda Λ , the rule d A − 1 = − A − 1 ( d A ) A − 1 \mathrm{d}A^{-1}=-A^{-1}(\mathrm{d}A)A^{-1} d A − 1 = − A − 1 ( d A ) A − 1 for the differential of the inverse ([29 ] ) gives the differentials of V n = A − 1 V_n=A^{-1} V n = A − 1 , of β n = A − 1 Z T y \beta_n=A^{-1}Z^{\mathsf{T}}\mathbf{y} β n = A − 1 Z T y and of h i = z i T A − 1 z i h_i=\mathbf{z}_i^{\mathsf{T}}A^{-1}\mathbf{z}_i h i = z i T A − 1 z i . The derivatives with respect to the parameters of P n P_n P n follow from those of its log-density; for the Student-t t t distribution, the derivative with respect to the degrees of freedom involves the digamma function. All these quantities are obtained from the Cholesky factor R \mathbf{R} R and from A − 1 Z T A^{-1}Z^{\mathsf{T}} A − 1 Z T , so the gradient costs the same order of operations as the LOO-PLL itself.
Predictive resampling
In Algorithm 2 of the paper, the covariates X n + 1 , … , X N \mathbf{X}_{n+1},\ldots,\mathbf{X}_N X n + 1 , … , X N are sampled from the empirical distribution of the observed features, and the weights w i \mathbf{w}_i w i and matrices V i V_i V i are computed once, in O ( N r 2 ) O(Nr^2) O ( N r 2 ) operations. Each of the B B B draws then requires sampling N − n N-n N − n innovations from P ξ P_\xi P ξ and computing a weighted sum, in O ( N r ) O(Nr) O ( N r ) operations, and the draws are computed in parallel.
Stopping the series at N N N omits the variance Σ N \mathbf{\Sigma}_N Σ N of the remainder β ~ − β N = ∑ i > N w i ξ i \widetilde{\beta}-\beta_N=\sum_{i>N}\mathbf{w}_i\xi_i β − β N = ∑ i > N w i ξ i , given the covariates. With σ n 2 = σ 2 / a n \sigma_n^2=\sigma^2/a_n σ n 2 = σ 2 / a n , Section 3.2 of the paper shows that Σ N = σ 2 V N \mathbf{\Sigma}_N=\sigma^2V_N Σ N = σ 2 V N . When V 0 − 1 V_0^{-1} V 0 − 1 is negligible, V N V_N V N is roughly n / N n/N n / N times V n V_n V n , as noted in Section 3.4 of the paper, so n / N n/N n / N gives the order of the omitted fraction of the posterior variance.
The last step of Algorithms 1 and 2 adds to each draw an independent draw from N p ( 0 , σ 2 V N ) N_p(0,\sigma^2V_N) N p ( 0 , σ 2 V N ) , in place of the remainder. Two facts justify it. First, the remainder is approximately Gaussian: it is a weighted sum of infinitely many independent innovations, each with a small weight, and Theorem 2 of the paper, applied with N N N in place of n n n , shows that it is asymptotically Gaussian with covariance Σ N \mathbf{\Sigma}_N Σ N . The non-Gaussian features of the martingale posterior, such as heavy tails, come mainly from the first terms of the series, which have the largest weights and are all simulated. Second, the covariance σ 2 V N \sigma^2V_N σ 2 V N is exact with σ n 2 = σ 2 / a n \sigma_n^2=\sigma^2/a_n σ n 2 = σ 2 / a n , as recalled above. With a constant σ n = σ \sigma_n=\sigma σ n = σ , Lemma A.3 with N N N in place of n n n and s N − = s N + = σ 2 s_N^-=s_N^+=\sigma^2 s N − = s N + = σ 2 gives
σ 2 ( 1 − ϵ N ) V N ⪯ Σ N ⪯ σ 2 V N , ϵ N = sup i > N x i T V i − 1 x i . \sigma^2(1-\epsilon_N)V_N \preceq\mathbf{\Sigma}_N \preceq\sigma^2V_N,\qquad\epsilon_N=\sup_{i>N}\mathbf{x}_i^{\mathsf{T}}V_{i-1}\mathbf{x}_i. σ 2 ( 1 − ϵ N ) V N ⪯ Σ N ⪯ σ 2 V N , ϵ N = i > N sup x i T V i − 1 x i .
The quantity ϵ N \epsilon_N ϵ N can be bounded by observed quantities. For i > N i>N i > N , Equation (4) of the paper gives V i − 1 − 1 = V N − 1 + ∑ k = N + 1 i x k x k T ⪰ V N − 1 V_{i-1}^{-1}=V_N^{-1}+\sum_{k=N+1}^{i}\mathbf{x}_k\mathbf{x}_k^{\mathsf{T}}\succeq V_N^{-1} V i − 1 − 1 = V N − 1 + ∑ k = N + 1 i x k x k T ⪰ V N − 1 , and inversion reverses ⪯ \preceq ⪯ , so V i − 1 ⪯ V N V_{i-1}\preceq V_N V i − 1 ⪯ V N . Hence $\epsilon_N\le max x x ⊤ V N x \max_{\mathbf{x}} \mathbf{x}^{\top}V_N\mathbf{x} max x x ⊤ V N x , where the maximum is over the support of the distribution of the future covariates, that is, over the observed features when they are resampled. Therefore, adding a draw from N p ( 0 , σ 2 V N ) \mathcal{N}_p(0,\sigma^2V_N) N p ( 0 , σ 2 V N ) overstates the covariance of the remainder by at most ϵ N σ 2 V N \epsilon_N\sigma^2V_N ϵ N σ 2 V N , where ϵ N \epsilon_N ϵ N is of order 1 / N 1/N 1/ N and computable.
In all the examples of the paper, we use N = 10 n N = 10n N = 10 n with the correction. We checked this choice against a reference with a much larger N N N , for which the omitted part of the posterior variance is negligible: N = 1,000 n N = 1{,}000n N = 1 , 000 n in the simulation study of § 5.3 of the paper, on five datasets per scenario, N = n + 100,000 N = n + 100{,}000 N = n + 100 , 000 in the two examples of § 5.2 , and N = n + 300,000 N = n + 300{,}000 N = n + 300 , 000 in the application of § 6 . In all cases, the 95% credible intervals are close to the reference ones: in § 5.3 , the average width and coverage of the intervals for f ( x ) f(x) f ( x ) coincide with the reference ones to three decimal places; in § 5.2 , the widths of the intervals for the coefficients are between 0.99 and 1.01 times the reference ones and their endpoints differ by at most 3.5% of the width, also in the first example, whose Student-t t t P n P_n P n has ν ^ = 2.60 \hat{\nu} = 2.60 ν ^ = 2.60 and n = 15 n = 15 n = 15 ; in § 6 , the widths of the intervals for E ( Y ∣ x ) \mathrm{E}(Y \mid x) E ( Y ∣ x ) at the test points are 1.004 times the reference ones on average, and their endpoints differ by at most 2% of the width. Without the correction, the omitted part of the posterior variance is of order n / N = 10 % n/N = 10\% n / N = 10% , and the intervals are narrower, by about 5% in § 5.1 and § 5.2 and 6% to 8% in § 5.3 .