Introduction
Bayesian nonparametric regression has wide applications in areas like machine learning, spatial statistics, and biostatistics. Unfortunately, methods such as dependent mixture models [26 , 30 ] and Gaussian processes (GP, Rasmussen and Williams [ [31 ] ) can have prohibitively expensive computation due to Markov chain Monte Carlo (MCMC) or large matrix inversions. Furthermore, these models have many moving parts, and prior and model elicitation often lacks clear guidance. These issues are further compounded in multi-output regression [3 ] , where the outcome is multivariate.
An alternative MCMC-free approach to Bayesian nonparametric inference for mixture models is based on recursive density updates, pioneered by the predictive recursion method [27 ] and more recently by Hahn et al. [ [16 ] . Advances within the predictive Bayesian literature have demonstrated recursive density updates can even be utilized to obtain full posterior uncertainty through the imputation of unseen future variables [13 , 7 , 12 ] , resulting in the martingale posterior.
Within the single-output regression setting, Fong et al. [ [12 ] , Cappello and Walker [ [9 ] extend the bivariate copula update of Hahn et al. [ [16 ] to conditional density estimation, and Lin and Tokdar [ [25 ] extend predictive recursion to estimate a covariate-dependent mixing density. However, in many cases in regression, we are often primarily interested in the mean or variance function, as opposed to the entire conditional density. While the previously mentioned recursive updates are flexible, they all require numerical integration on a grid if the moment functions are of interest, which is already computationally demanding in the single-output case, and quickly becomes infeasible as the dimension of the outcome grows.
In this paper, we introduce flexible, tractable, and interpretable recursive updates for the covariate-dependent moment functions, with uncertainty quantification through a semiparametric martingale posterior. Our moment updates easily handle heteroscedasticity, heavy-tailed or skewed residuals, and importantly scale very well to the multi-output setting with missing outcomes and covariate-dependent correlations. Our approach achieves competitive or superior statistical performance compared to other recursive approaches and traditional Bayesian nonparametric methods, while being orders of magnitude faster as we do not rely on numerical integration. Hyperparameter tuning is automatic, so our approach can be used off-the-shelf for Bayesian nonparametric regression. Finally, tractability of the moment updates allows thorough study of predictive and frequentist asymptotics.
Methodology for univariate outcomes
In this section, we consider single outcome regression with random covariates to illustrate the main idea, before extending to the multi-output case in § 3 . Let Y ∈ R Y \in\mathbb{R} Y ∈ R denote the outcome and X ∈ X ⊆ R p X \in\mathcal{X} \subseteq\mathbb{R}^{p} X ∈ X ⊆ R p denote the covariates where X \mathcal{X} X is compact. Let ( Y i , X i ) i = 1 , … , n (Y_i, X_i)_{i=1,\ldots,n} ( Y i , X i ) i = 1 , … , n be n n n independently and identically distributed (i.i.d.) copies of ( Y , X ) (Y, X) ( Y , X ) from an unknown distribution P ∗ P^{*} P ∗ with Lebesgue density p ∗ ( y , x ) p^{*}(y,x) p ∗ ( y , x ) . Furthermore, let p X ∗ ( x ) p_{X}^{*}(x) p X ∗ ( x ) denote the marginal density of X X X , and let p ∗ ( y ∣ x ) p^{*}(y \mid x) p ∗ ( y ∣ x ) denote the conditional density of interest which we assume exists for all x ∈ X x \in\mathcal{X} x ∈ X .
Recursive bivariate copula updates
We begin by revisiting a recursive approach for conditional density estimation introduced in Fong et al. [ [12 ] using bivariate copula densities. Whilst not the main focus, our recursive moment updates will be derived from this copula update. The starting point is that the Dirichlet process (DP) mixture is tractable when only a single datum is observed [27 ] . In the covariate-free DP mixture, Hahn et al. [ [16 ] derived the predictive density update in terms of a Gaussian copula density, and proposed extending the n = 1 n=1 n = 1 rule recursively for n > 1 n>1 n > 1 .
The same idea can be carried out for the dependent DP mixture model with a Gaussian kernel and base measure. Let p i ( y ∣ x ) p_i(y \mid x) p i ( y ∣ x ) and u i = P i ( y ∣ x ) u_i=P_i(y \mid x) u i = P i ( y ∣ x ) denote the conditional probability density function (PDF) and cumulative distribution function (CDF) after i i i observations. Fong et al. [ [12 ] suggests the recursive update
p i ( y ∣ x ) = [ 1 − α i ( x , X i ) + α i ( x , X i ) c ρ ( u i − 1 , v i − 1 ) ] p i − 1 ( y ∣ x ) , (1) p_i(y \mid x) = [1-\alpha_i(x,X_i)+\alpha_i(x,X_i)c_{\rho}(u_{i-1},v_{i-1})]p_{i-1}(y \mid x), \tag*{(1)} p i ( y ∣ x ) = [ 1 − α i ( x , X i ) + α i ( x , X i ) c ρ ( u i − 1 , v i − 1 )] p i − 1 ( y ∣ x ) , (1)
where v i − 1 = P i − 1 ( Y i ∣ X i ) v_{i-1}=P_{i-1}(Y_i \mid X_i) v i − 1 = P i − 1 ( Y i ∣ X i ) , and c ρ c_{\rho} c ρ is the bivariate Gaussian copula density:
c ρ ( u , v ) = ϕ 2 ( Φ − 1 ( u ) , Φ − 1 ( v ) ; ρ ) ϕ ( Φ − 1 ( u ) ) ϕ ( Φ − 1 ( v ) ) c_{\rho}(u,v)=\frac{\phi_2(\Phi^{-1}(u),\Phi^{-1}(v);\rho)}{\phi(\Phi^{-1}(u))\phi(\Phi^{-1}(v))} c ρ ( u , v ) = ϕ ( Φ − 1 ( u )) ϕ ( Φ − 1 ( v )) ϕ 2 ( Φ − 1 ( u ) , Φ − 1 ( v ) ; ρ )
and ϕ 2 ( y 1 , y 2 ; ρ ) \phi_2(y_1,y_2;\rho) ϕ 2 ( y 1 , y 2 ; ρ ) is the standard bivariate Gaussian density with correlation ρ \rho ρ , ϕ ( y ) \phi(y) ϕ ( y ) is the standard Gaussian density, Φ − 1 \Phi^{-1} Φ − 1 is the Gaussian quantile function, ρ ∈ ( 0 , 1 ) \rho\in(0,1) ρ ∈ ( 0 , 1 ) controls the smoothness of the density, and the covariate-dependent learning rate α i ( x , X i ) \alpha_i(x,X_i) α i ( x , X i ) determines the information borrowed between covariates.
Covariate-dependent learning rate
We consider a general form of α i ( x , x ′ ) \alpha_i(x,x') α i ( x , x ′ ) based on the ‘conditional’ approach of Fong et al. [ [12 ] . Let s ( x , x ′ ) s(x,x') s ( x , x ′ ) be a similarity function ,
s ( x , x ′ ) = E Ψ [ k ( x ; Ψ ) k ( x ′ ; Ψ ) ] E Ψ [ k ( x ; Ψ ) ] E Ψ [ k ( x ′ ; Ψ ) ] (2) s(x,x') = \frac{E_{\Psi}[k(x;\Psi)k(x';\Psi)]}{E_{\Psi}[k(x;\Psi)]E_{\Psi}[k(x';\Psi)]} \tag*{(2)} s ( x , x ′ ) = E Ψ [ k ( x ; Ψ )] E Ψ [ k ( x ′ ; Ψ )] E Ψ [ k ( x ; Ψ ) k ( x ′ ; Ψ )] (2)
where Ψ ∼ Π \Psi\sim\Pi Ψ ∼ Π , the kernel k ( ⋅ ; ψ ) k(\cdot;\psi) k ( ⋅ ; ψ ) is a PDF for all ψ ∈ supp ( Π ) \psi\in\operatorname{supp}(\Pi) ψ ∈ supp ( Π ) , and we assume that s ( x , x ′ ) s(x,x') s ( x , x ′ ) is well-defined for all ( x , x ′ ) ∈ X × X (x,x') \in\mathcal{X} \times\mathcal{X} ( x , x ′ ) ∈ X × X . We then define
α i ( x , x ′ ) = α i s ( x , x ′ ) 1 − α i + α i s ( x , x ′ ) . \alpha_i(x,x') = \frac{\alpha_i s(x,x')}{1-\alpha_i+\alpha_i s(x,x')}. α i ( x , x ′ ) = 1 − α i + α i s ( x , x ′ ) α i s ( x , x ′ ) .
where 0 < α i < 1 0 < \alpha_i < 1 0 < α i < 1 and α i = O ( i − m ) \alpha_i = O(i^{-m}) α i = O ( i − m ) for some 0.5 < m ≤ 1 0.5 < m \le1 0.5 < m ≤ 1 . In our implementations, we choose the learning rate α i = ( 2 − i − 1 ) ( i + 1 ) − 0.9 \alpha_i = (2-i^{-1})(i+1)^{-0.9} α i = ( 2 − i − 1 ) ( i + 1 ) − 0.9 , which works well in practice. Further discussions on the role of the learning rate can be found in Fortini and Petrone [2020], Agnoletto et al. [2026] and Fong et al. [2023, Appendix E].
Importantly, we have α i ( x , x ′ ) → 0 \alpha_i(x,x') \to0 α i ( x , x ′ ) → 0 if s ( x , x ′ ) → 0 s(x,x') \to0 s ( x , x ′ ) → 0 , so the datum ( Y i , X i ) (Y_i,X_i) ( Y i , X i ) is less relevant for the conditional density at x x x if s ( x , X i ) s(x,X_i) s ( x , X i ) is small, and α i ( x , x ′ ) → 0 \alpha_i(x,x') \to0 α i ( x , x ′ ) → 0 as i → ∞ i \to\infty i → ∞ . Like Antoniano-Villalobos et al. [2014], our model elicitation involves a kernel k ( x ; ψ ) k(x;\psi) k ( x ; ψ ) with a prior for ψ \psi ψ . We now illustrate an example of s ( x , x ′ ) s(x,x') s ( x , x ′ ) .
Example 1. Consider the Gaussian kernel k ( x ; ψ ) = N ( x ; ψ , 1 − τ − 1 ) k(x;\psi) = \mathcal{N}(x;\psi,1-\tau^{-1}) k ( x ; ψ ) = N ( x ; ψ , 1 − τ − 1 ) and Π = N ( 0 , τ − 1 ) \Pi= \mathcal{N}(0,\tau^{-1}) Π = N ( 0 , τ − 1 ) for a fixed τ > 1 \tau> 1 τ > 1 . In this case, we recover s ( x , x ′ ) = c ρ x { P 0 ( x ) , P 0 ( x ′ ) } s(x,x') = c_{\rho_x}\{P_0(x),P_0(x')\} s ( x , x ′ ) = c ρ x { P 0 ( x ) , P 0 ( x ′ )} where P 0 = Φ P_0 = \Phi P 0 = Φ is the normal CDF and ρ x = τ − 1 \rho_x = \tau^{-1} ρ x = τ − 1 , as considered in Fong et al. [ [12 ] .
More generally, a closed form s ( x , x ′ ) s(x,x') s ( x , x ′ ) can be obtained if k ( ⋅ ; ψ ) k(\cdot;\psi) k ( ⋅ ; ψ ) is an exponential family with conjugate prior Π \Pi Π ; we outline more examples in Appendix § 13 .§ 1 . Multivariate covariates can be handled by defining k ( x ; ψ ) = ∏ j = 1 p k ( x ( j ) ; ψ ( j ) ) k(x;\psi) = \prod_{j=1}^{p} k(x^{(j)};\psi^{(j)}) k ( x ; ψ ) = ∏ j = 1 p k ( x ( j ) ; ψ ( j ) ) , and hyperparameters of s ( x , x ′ ) s(x,x') s ( x , x ′ ) , e.g. ρ x \rho_x ρ x for the Gaussian copula, can be tuned in an empirical Bayesian manner using the prequential log-likelihood. In the multivariate case, the bandwidth tuning procedure can also carry out variable selection.
The main property we assert on s ( x , x ′ ) s(x,x') s ( x , x ′ ) is that the family { k ( x ; ψ ) : ψ ∈ supp ( Π ) } \{k(x;\psi):\psi\in\operatorname{supp}(\Pi)\} { k ( x ; ψ ) : ψ ∈ supp ( Π )} is complete [22 ] as defined in Definition 1. Completeness will be necessary for consistency of the recursive moment updates, and is satisfied by many examples including Example 1. Intuitively, this means that s ( x , x ′ ) s(x,x') s ( x , x ′ ) does not have any ‘gaps’ which may impede consistency.
Definition 1. The family { k ( x ; ψ ) : ψ ∈ supp ( Π ) } \{k(x;\psi):\psi\in\operatorname{supp}(\Pi)\} { k ( x ; ψ ) : ψ ∈ supp ( Π )} is complete if for every measurable function f : R p → R f:\mathbb{R}^p \to\mathbb{R} f : R p → R , the condition ∫ f ( x ) k ( x ; ψ ) d x = 0 \int f(x)k(x;\psi)\,dx = 0 ∫ f ( x ) k ( x ; ψ ) d x = 0 for all ψ ∈ supp ( Π ) \psi\in\operatorname{supp}(\Pi) ψ ∈ supp ( Π ) implies f ( x ) = 0 f(x) = 0 f ( x ) = 0 for k ( ⋅ ; ψ ) k(\cdot;\psi) k ( ⋅ ; ψ ) -almost all x x x for every ψ ∈ supp ( Π ) \psi\in\operatorname{supp}(\Pi) ψ ∈ supp ( Π ) .
Recursive moment updates
In nonparametric regression, the mean function E [ Y ∣ X = x ] E[Y \mid X = x] E [ Y ∣ X = x ] is often of primary interest for prediction, or one may be interested in assessing heteroscedasticity through the conditional variance Var ( Y ∣ X = x ) \operatorname{Var}(Y \mid X = x) Var ( Y ∣ X = x ) . A significant drawback of the nonparametric copula approach is that the moment functions must be computed numerically on a grid from the conditional density estimates p ( y ∣ x ) p(y \mid x) p ( y ∣ x ) , which can be very computationally expensive, and quickly becomes completely infeasible in the multi-output setting in the next Section. In addition, the selection of the y y y -grid is non-trivial, especially under strong heteroscedasticity or heavy tails where a very high resolution is needed to cover the range of y y y values. We highlight that a grid over the mixing parameter is also needed for the predictive recursion approach of Lin and Tokdar [2026], which will scale exponentially poorly with outcome dimensionality.
To that end, we introduce a semiparametric recursive update, which can be viewed as a projection of the nonparametric copula update onto a parametric family. Instead of a recursive update for the conditional density, we restrict p i ( y ∣ x ) p_i(y \mid x) p i ( y ∣ x ) to a specific parametric family, and derive tractable recursive updates for the moment functions directly. The moment updates are simple, flexible and interpretable, and are significantly more expedient than the copula update and other Bayesian competitors. Importantly, uncertainty quantification of the moment functions can still be obtained through the martingale posterior, and § 4.2 will illustrate that consistency can still be attained even if the family is misspecified.
Gaussian family
The core idea is to distill the full nonparametric copula update into its first two moments, then project the updated density onto the Gaussian family through moment matching. Let us restrict p i − 1 ( y ∣ x ) p_{i-1}(y \mid x) p i − 1 ( y ∣ x ) to be Gaussian, that is
p i − 1 ( y ∣ x ) = N ( y ; μ i − 1 ( x ) , σ i − 1 2 ( x ) ) . p_{i-1}(y \mid x) = \mathcal{N}\left(y; \mu_{i-1}(x), \sigma_{i-1}^{2}(x)\right). p i − 1 ( y ∣ x ) = N ( y ; μ i − 1 ( x ) , σ i − 1 2 ( x ) ) .
Whilst applying (1) to the above yields a non-Gaussian p i ( y ∣ x ) p_i(y \mid x) p i ( y ∣ x ) , its updated mean and variance can be computed exactly from conjugacy of the predictive CDF with the Gaussian copula. We then use these updated moments to define the Gaussian predictive density for the next step. To derive the moment updates, note that P i − 1 ( y ∣ x ) = Φ { ( y − μ i − 1 ( x ) ) / σ i − 1 ( x ) } P_{i-1}(y \mid x) = \Phi\{(y - \mu_{i-1}(x))/\sigma_{i-1}(x)\} P i − 1 ( y ∣ x ) = Φ {( y − μ i − 1 ( x )) / σ i − 1 ( x )} , and Gaussian conditioning then gives
c ρ ( u i − 1 , v i − 1 ) p i − 1 ( y ∣ x ) = N ( y ; μ i − 1 ( x ) + ρ σ i − 1 ( x ) Z i , ( 1 − ρ 2 ) σ i − 1 2 ( x ) ) , (3) c_{\rho}(u_{i-1}, v_{i-1})p_{i-1}(y \mid x) = \mathcal{N}\left(y; \mu_{i-1}(x) + \rho\sigma_{i-1}(x)Z_i, (1-\rho^2)\sigma_{i-1}^{2}(x)\right), \tag*{(3)} c ρ ( u i − 1 , v i − 1 ) p i − 1 ( y ∣ x ) = N ( y ; μ i − 1 ( x ) + ρ σ i − 1 ( x ) Z i , ( 1 − ρ 2 ) σ i − 1 2 ( x ) ) , (3)
where Z i Z_i Z i is the standardized residual of the form
Z i = Y i − μ i − 1 ( X i ) σ i − 1 ( X i ) . (4) Z_i = \frac{Y_i-\mu_{i-1}(X_i)}{\sigma_{i-1}(X_i)}. \tag*{(4)} Z i = σ i − 1 ( X i ) Y i − μ i − 1 ( X i ) . (4)
Crucially, this gives a tractable recursive update of the k k k -th moment m i ( k ) ( x ) = ∫ y k p i ( y ∣ x ) d y m_i^{(k)}(x)=\int y^k p_i(y\mid x)\,dy m i ( k ) ( x ) = ∫ y k p i ( y ∣ x ) d y , as integrating (1) results in the recursion
m i ( k ) ( x ) = [ 1 − α i ( x , X i ) ] m i − 1 ( k ) ( x ) + α i ( x , X i ) m ~ i ( k ) ( x ) m_i^{(k)}(x) = [1-\alpha_i(x,X_i)]m_{i-1}^{(k)}(x)+\alpha_i(x,X_i)\widetilde{m}_i^{(k)}(x) m i ( k ) ( x ) = [ 1 − α i ( x , X i )] m i − 1 ( k ) ( x ) + α i ( x , X i ) m i ( k ) ( x )
where m ~ i ( k ) \widetilde{m}_i^{(k)} m i ( k ) is the k k k -th moment of the conditional Gaussian given in (3). Evaluating the above for k = 1 , 2 k=1,2 k = 1 , 2 then gives the tractable mean and variance recursion
μ i ( x ) = μ i − 1 ( x ) + α i ( x , X i ) σ i − 1 ( x ) ρ Z i , (5) \mu_i(x)=\mu_{i-1}(x)+\alpha_i(x,X_i)\sigma_{i-1}(x)\rho Z_i, \tag*{(5)} μ i ( x ) = μ i − 1 ( x ) + α i ( x , X i ) σ i − 1 ( x ) ρ Z i , (5)
σ i 2 ( x ) = σ i − 1 2 ( x ) ( 1 + α i ( x , X i ) ρ 2 { [ 1 − α i ( x , X i ) ] Z i 2 − 1 } ) . (6) \sigma_i^2(x)=\sigma_{i-1}^2(x)\left(1+\alpha_i(x,X_i)\rho^2\left\{[1-\alpha_i(x,X_i)]Z_i^2-1\right\}\right). \tag*{(6)} σ i 2 ( x ) = σ i − 1 2 ( x ) ( 1 + α i ( x , X i ) ρ 2 { [ 1 − α i ( x , X i )] Z i 2 − 1 } ) . (6)
The form of the mean update is interpretable, where the residual Z i Z_i Z i drives the update to the mean function, with α i ( x , X i ) \alpha_i(x,X_i) α i ( x , X i ) playing a similar weighting role. The scaling of the residual by σ i − 1 ( x ) \sigma_{i-1}(x) σ i − 1 ( x ) then rescales the normalization by σ i − 1 ( X i ) \sigma_{i-1}(X_i) σ i − 1 ( X i ) at the test point of interest. The variance update is similarly driven by the error term ( Z i 2 − 1 ) (Z_i^2-1) ( Z i 2 − 1 ) . Interestingly, the two updates are heavily coupled, so studying the frequentist consistency will rely on a carefully constructed almost-supermartingale [33 ] . We note that ρ ∈ ( 0 , 1 ) \rho\in(0,1) ρ ∈ ( 0 , 1 ) now governs the influence of a new datum through scaling of the update. Finally, while mean/variance function updates exist in the non-Bayesian literature for estimation, e.g. Stadtmüller and Tsybakov [1995], our above update is sufficient for full Bayesian uncertainty quantification.
Example 2. We demonstrate the Gaussian moment updating scheme following the simulation setup of Antoniano-Villalobos et al. [2014], with p X ∗ = Unif ( − 7 , 7 ) p_X^*=\operatorname{Unif}(-7,7) p X ∗ = Unif ( − 7 , 7 ) and p ∗ ( y ∣ x ) = N ( m ( x ) , s 2 ( x ) ) p^*(y\mid x)=\mathcal{N}(m(x),s^2(x)) p ∗ ( y ∣ x ) = N ( m ( x ) , s 2 ( x )) where
m ( x ) = 5 1 + exp ( − x ) , s ( x ) = 1 4 + exp ( x − 6 3 ) . m(x)=\frac{5}{1+\exp(-x)},\qquad s(x)=\frac{1}{4}+\exp\left(\frac{x-6}{3}\right). m ( x ) = 1 + exp ( − x ) 5 , s ( x ) = 4 1 + exp ( 3 x − 6 ) . Figure 1 illustrates the results with n = 200 n=200 n = 200 simulated observations and B = 1000 B=1000 B = 1000 posterior samples with a truncation of N = n + 4000 N=n+4000 N = n + 4000 . The copula approach requires a grid of 200 y y y -values for numerical accuracy. We see that both the predictive densities and the posterior over μ ( x ) \mu(x) μ ( x ) are similar, with slightly higher posterior uncertainty for the moment update. The key difference is that the copula updates required 180s to run, whereas the moment updates only required 0.4s (on an Apple M2 Pro CPU).
Figure 1. Predictive mean with 95% predictive intervals (Left); Posterior mean with 95% credible intervals for μ ( x ) \mu(x) μ ( x ) (Right).
Predictive resampling for uncertainty
Equipped with the above, it is natural to define our sequence of conditional predictives as the Gaussian density p i ( y ∣ x ) = N ( y ; μ i ( x ) , σ i 2 ( x ) ) p_i(y \mid x) = \mathcal{N}\left(y; \mu_i(x), \sigma_i^2(x)\right) p i ( y ∣ x ) = N ( y ; μ i ( x ) , σ i 2 ( x ) ) with moment updates given by (5) and (6). Under predictive resampling, it is not too hard to see that Z i ∼ i i d N ( 0 , 1 ) Z_i \mathrel{\overset{\mathrm{iid}}{\sim}} \mathcal{N}(0,1) Z i ∼ iid N ( 0 , 1 ) for i ≥ n + 1 i \ge n+1 i ≥ n + 1 , which is similar to the i.i.d. uniform random variables in the copula case. As a result, predictive resampling is computationally very simple, as it just involves applying (5) and (6) to i.i.d. Gaussian noise. Furthermore, the first two raw moments can be shown to be function-valued martingales, and Algorithm 1 illustrates how to sample from the martingale posterior, where P X P_X P X is a chosen design distribution. We briefly discuss the role of P X P_X P X , which was investigated in Sun and Fong [2026]. Since our objects of interest only depend on p ( y ∣ x ) p(y \mid x) p ( y ∣ x ) , we do not require uncertainty in P X P_X P X , so we suggest fixing it as the default choice of the uniform distribution over X \mathcal{X} X if the covariate space is compact, or the Gaussian distribution if X = R p \mathcal{X} = \mathbb{R}^p X = R p . In § 4.1 , we will study the convergence of Algorithm 1 using function-valued martingale theory [29 ] . In addition to proving convergence of our algorithm, we will show that the martingale posterior has support over smooth mean/variance functions under appropriate assumptions on s ( x , x ′ ) s(x,x^\prime) s ( x , x ′ ) .
Algorithm 1: Predictive resampling with recursive moment updates
Estimate p n ( y ∣ x ) p_n(y \mid x) p n ( y ∣ x ) from ( Y 1 : n , X 1 : n ) (Y_{1:n}, X_{1:n}) ( Y 1 : n , X 1 : n ) using Equations (5) and (6)
Draw X n + 1 : N ∼ i i d P X X_{n+1:N} \mathrel{\overset{\mathrm{iid}}{\sim}} P_X X n + 1 : N ∼ iid P X
for i ← n + 1 i \leftarrow n+1 i ← n + 1 to N N N do
Draw Z i ∼ i i d N ( 0 , 1 ) Z_i \mathrel{\overset{\mathrm{iid}}{\sim}} \mathcal{N}(0,1) Z i ∼ iid N ( 0 , 1 )
Update { μ i − 1 ( x ) , σ i − 1 2 ( x ) } ↦ { μ i ( x ) , σ i 2 ( x ) } \{\mu_{i-1}(x), \sigma_{i-1}^2(x)\} \mapsto\{\mu_i(x), \sigma_i^2(x)\} { μ i − 1 ( x ) , σ i − 1 2 ( x )} ↦ { μ i ( x ) , σ i 2 ( x )} using Equations (5) and (6)
end
Return { μ N ( x ) , σ N 2 ( x ) } \{\mu_N(x), \sigma_N^2(x)\} { μ N ( x ) , σ N 2 ( x )}
Heavy-tailed family
A key property of our moment matching framework is that it extends naturally to more general choices of p i ( y ∣ x ) p_i(y \mid x) p i ( y ∣ x ) beyond the Gaussian, which may be more appropriate depending on the setting. To illustrate the idea, we investigate the induced moment updates for a heavy-tailed choice of p i ( y ∣ x ) p_i(y \mid x) p i ( y ∣ x ) . Our proposed parametric family is a transformed Gaussian random variable which is still ‘conjugate’ to the copula, and induces a tractable robust moment update with only a small computational overhead. This is in stark contrast to other heavy-tailed nonparametric regression methods, e.g. GPs with a Student-t likelihood [20 ] , which require approximate inference, although recent generalized Bayes procedures preserve conjugacy [2 , 34 ] .
Let δ > 0 \delta> 0 δ > 0 be a parameter controlling the kurtosis, and let ω = exp ( 1 / δ 2 ) \omega= \exp(1/\delta^2) ω = exp ( 1/ δ 2 ) for shorthand. We define the family p S U ( y ; μ , σ 2 , δ ) p^{\mathrm{SU}}(y; \mu, \sigma^2, \delta) p SU ( y ; μ , σ 2 , δ ) as the random variable generated as
Y = μ + σ [ sinh ( ε δ ) 1 2 ( ω 2 − 1 ) ] , (7) Y = \mu+ \sigma\left[\frac{\sinh\left(\frac{\varepsilon}{\delta}\right)}{\sqrt{\frac{1}{2}(\omega^2 - 1)}}\right], \tag*{(7)} Y = μ + σ 2 1 ( ω 2 − 1 ) sinh ( δ ε ) , (7)
where ε ∼ N ( 0 , 1 ) \varepsilon\sim\mathcal{N}(0,1) ε ∼ N ( 0 , 1 ) , which is a reparametrized Johnson’s SU distribution [19 ] . One can verify that Y Y Y has mean μ \mu μ and variance σ 2 \sigma^2 σ 2 with kurtosis equal to ( ω 4 + 2 ω 2 + 3 ) / 2 (\omega^4 + 2\omega^2 + 3)/2 ( ω 4 + 2 ω 2 + 3 ) /2 , so we can attain the range of kurtosis values of [ 3 , ∞ ) [3,\infty) [ 3 , ∞ ) for δ ∈ ( 0 , ∞ ] \delta\in(0,\infty] δ ∈ ( 0 , ∞ ] . Figure 2 (left) illustrates this family for different values of δ \delta δ , where taking δ → ∞ \delta\to\infty δ → ∞ (i.e. ω → 1 \omega\to1 ω → 1 ) recovers the Gaussian. We then have our main result.
Figure 2. Density p S U ( y ; 0 , 1 , δ ) p^{\mathrm{SU}}(y;0,1,\delta) p SU ( y ; 0 , 1 , δ ) (Left); Ratio of mean update ∣ R δ ( 1 ) ( ε i ) ∣ / ∣ ρ Z i ∣ \lvert R_{\delta}^{(1)}(\varepsilon_i)\rvert/\lvert\rho Z_i\rvert ∣ R δ ( 1 ) ( ε i )∣ / ∣ ρ Z i ∣ (Right)
Proposition 1. Let p i − 1 ( y ∣ x ) = p S U ( y ; μ i − 1 ( x ) , σ i − 1 2 ( x ) , δ ) p_{i-1}(y \mid x) = p^{\mathrm{SU}}(y; \mu_{i-1}(x), \sigma_{i-1}^2(x), \delta) p i − 1 ( y ∣ x ) = p SU ( y ; μ i − 1 ( x ) , σ i − 1 2 ( x ) , δ ) for fixed δ > 0 \delta> 0 δ > 0 . Then the mean and variance function updates corresponding to (1) take the form
μ i ( x ) = μ i − 1 ( x ) + α i ( x , X i ) σ i − 1 ( x ) R δ ( 1 ) ( ε i ) , (8) \mu_i(x) = \mu_{i-1}(x) + \alpha_i(x,X_i)\sigma_{i-1}(x)R_{\delta}^{(1)}(\varepsilon_i), \tag*{(8)} μ i ( x ) = μ i − 1 ( x ) + α i ( x , X i ) σ i − 1 ( x ) R δ ( 1 ) ( ε i ) , (8) σ i 2 ( x ) = σ i − 1 2 ( x ) [ 1 + α i ( x , X i ) { R δ ( 2 ) ( ε i ) − 1 − α i ( x , X i ) R δ ( 1 ) ( ε i ) 2 } ] , (9) \sigma_i^2(x) = \sigma_{i-1}^2(x)\left[1+\alpha_i(x,X_i)\left\{R_{\delta}^{(2)}(\varepsilon_i)-1-\alpha_i(x,X_i)R_{\delta}^{(1)}(\varepsilon_i)^2\right\}\right], \tag*{(9)} σ i 2 ( x ) = σ i − 1 2 ( x ) [ 1 + α i ( x , X i ) { R δ ( 2 ) ( ε i ) − 1 − α i ( x , X i ) R δ ( 1 ) ( ε i ) 2 } ] , (9) where the update terms take the form
R δ ( 1 ) ( ε i ) = ω ( 1 − ρ 2 ) / 2 sinh ( ρ ε i δ ) 1 2 ( ω 2 − 1 ) , R δ ( 2 ) ( ε i ) = ω 2 ( 1 − ρ 2 ) cosh ( 2 ρ ε i δ ) − 1 ω 2 − 1 (10) R_{\delta}^{(1)}(\varepsilon_i)=\frac{\omega^{(1-\rho^2)/2}\sinh\left(\frac{\rho\varepsilon_i}{\delta}\right)}{\sqrt{\frac{1}{2}(\omega^2-1)}}, \qquad R_{\delta}^{(2)}(\varepsilon_i)=\frac{\omega^{2(1-\rho^2)}\cosh\left(\frac{2\rho\varepsilon_i}{\delta}\right)-1}{\omega^2-1} \tag*{(10)} R δ ( 1 ) ( ε i ) = 2 1 ( ω 2 − 1 ) ω ( 1 − ρ 2 ) /2 sinh ( δ ρ ε i ) , R δ ( 2 ) ( ε i ) = ω 2 − 1 ω 2 ( 1 − ρ 2 ) cosh ( δ 2 ρ ε i ) − 1 (10) and ε i = δ sinh − 1 ( 1 2 ( ω 2 − 1 ) Z i ) \varepsilon_i=\delta\sinh^{-1}\left(\sqrt{\frac{1}{2}(\omega^2-1)}Z_i\right) ε i = δ sinh − 1 ( 2 1 ( ω 2 − 1 ) Z i ) where Z i Z_i Z i is the standardized residual as in (4). Furthermore, δ → ∞ \delta\to\infty δ → ∞ recovers the Gaussian update in (5) and (6).
The above update generalizes the Gaussian update: under p i ( y ∣ x ) = p S U ( y ; μ i ( x ) , σ i 2 ( x ) , δ ) p_i(y \mid x)=p^{\mathrm{SU}}(y;\mu_i(x),\sigma_i^2(x),\delta) p i ( y ∣ x ) = p SU ( y ; μ i ( x ) , σ i 2 ( x ) , δ ) , we have ε i ∼ i i d N ( 0 , 1 ) \varepsilon_i \overset{\mathrm{iid}}{\sim} \mathcal{N}(0,1) ε i ∼ iid N ( 0 , 1 ) , so predictive resampling in Algorithm 1 can easily be extended by replacing the moment updates with (8) and (9), and Z i Z_i Z i with ε i \varepsilon_i ε i . A key property of the SU moment update is the increased robustness to outliers. In Figure 2 (right), we plot the ratio of ∣ R δ ( 1 ) ( ε i ) ∣ \lvert R_{\delta}^{(1)}(\varepsilon_i)\rvert ∣ R δ ( 1 ) ( ε i )∣ to the Gaussian counterpart ∣ ρ Z i ∣ \lvert\rho Z_i\rvert ∣ ρ Z i ∣ , which clearly illustrates a reduced sensitivity to outliers for large values of ∣ Z i ∣ \lvert Z_i\rvert ∣ Z i ∣ . For the variance update, the Gaussian counterpart to R δ ( 2 ) ( ε i ) R_{\delta}^{(2)}(\varepsilon_i) R δ ( 2 ) ( ε i ) is 1 − ρ 2 + ρ 2 Z i 2 1-\rho^2+\rho^2Z_i^2 1 − ρ 2 + ρ 2 Z i 2 , where R δ ( 2 ) R_{\delta}^{(2)} R δ ( 2 ) also has a robustifying effect. This is formalized below.
Proposition 2. For fixed ρ ∈ ( 0 , 1 ) \rho\in(0,1) ρ ∈ ( 0 , 1 ) and δ > 0 \delta> 0 δ > 0 and taking ∣ Z i ∣ → ∞ \lvert Z_i\rvert\to\infty ∣ Z i ∣ → ∞ , (10) satisfies ∣ R δ ( 1 ) ( ε i ) ∣ = O ( ∣ Z i ∣ ρ ) \lvert R_{\delta}^{(1)}(\varepsilon_i)\rvert= O(\lvert Z_i\rvert^\rho) ∣ R δ ( 1 ) ( ε i )∣ = O (∣ Z i ∣ ρ ) and R δ ( 2 ) ( ε i ) = O ( ∣ Z i ∣ 2 ρ ) R_{\delta}^{(2)}(\varepsilon_i) = O(\lvert Z_i\rvert^{2\rho}) R δ ( 2 ) ( ε i ) = O (∣ Z i ∣ 2 ρ ) , where the constants depend on ( δ , ρ ) (\delta,\rho) ( δ , ρ ) . Furthermore, for any δ > 0 \delta> 0 δ > 0 and fixed Z i Z_i Z i , we have lim ρ → 1 R δ ( 1 ) ( ε i ) = Z i \lim_{\rho\to1}R_{\delta}^{(1)}(\varepsilon_i)=Z_i lim ρ → 1 R δ ( 1 ) ( ε i ) = Z i and lim ρ → 1 R δ ( 2 ) ( ε i ) = Z i 2 \lim_{\rho\to1}R_{\delta}^{(2)}(\varepsilon_i)=Z_i^2 lim ρ → 1 R δ ( 2 ) ( ε i ) = Z i 2 , which are equivalent to the Gaussian updates at ρ = 1 \rho=1 ρ = 1 .
Proposition 2 confirms our observations in Figure 2 (right), where a smaller value of ρ \rho ρ increases the relative robustness, and δ \delta δ comes into the constant. In practice, we learn δ \delta δ and ρ \rho ρ alongside the hyperparameters of s ( x , x ′ ) s(x,x') s ( x , x ′ ) using a prequential log-likelihood score, which only adds a small computational overhead on top of the Gaussian update. Finally, we highlight that the robustness emerged naturally from the copula update under the assumption of a heavy-tailed predictive density.
Example 3. Consider the same setup as Example 2 except with
p ∗ ( y ∣ x ) = 0.95 N ( m ( x ) , s 2 ( x ) ) + 0.05 N ( m ( x ) , 100 s 2 ( x ) ) , p^*(y\mid x)=0.95\mathcal{N}\left(m(x),s^2(x)\right)+0.05\mathcal{N}\left(m(x),100s^2(x)\right), p ∗ ( y ∣ x ) = 0.95 N ( m ( x ) , s 2 ( x ) ) + 0.05 N ( m ( x ) , 100 s 2 ( x ) ) ,
corresponding to a significant Huber contamination. Figure 3 illustrates the Gaussian versus the SU moment updates. We see that the Gaussian’s mean and variance functions are noticeably influenced by the outliers, whereas the SU update is robust to the outliers (prequentially tuned ρ = 0.52 \rho=0.52 ρ = 0.52 ). The run-times for the Gauss and SU methods are 0.5s and 0.8s respectively, indicating the small overhead for the robust approach. We also note that the credible intervals for the SU update are narrower for the mean.
Figure 3. Predictive mean with 95% predictive intervals (Left); Posterior mean with 95% credible intervals for μ ( x ) \mu(x) μ ( x ) (Right).
General families
A natural generalization is to replace the sinh transformation with other invertible transforms of a Gaussian random variable; we show in Appendix § 10 .§ 1 that the moment updates take the same form as in Proposition 1 but with a transform-specific R ( j ) ( ε i ) R^{(j)}(\varepsilon_i) R ( j ) ( ε i ) . Suitable choices of the transform will preserve tractability, and predictive resampling will still only require i.i.d. Gaussian random variables. In Appendix § 13 .§ 2 , we show generalizations for skewed residuals and the platykurtic setting.
Methodology for multivariate outcomes
We now extend the previous ideas to the multi-output setting. Let Y ∈ R q Y \in\mathbb{R}^{q} Y ∈ R q and let Y ( j ) Y^{(j)} Y ( j ) denote the j j j -th component of Y Y Y . The multi-output setting is where the computational advantage of the moment update is clearest: both the copula update and predictive recursion require numerical integration on a grid and have a computational cost that can potentially scale exponentially with q q q , and methods based on GPs are inflexible, scale poorly, or require complex posterior approximations. We begin with a natural extension of the copula update to the multi-output setting:
p i ( y ∣ x ) = [ 1 − α i ( x , X i ) + α i ( x , X i ) ∏ j = 1 q C ρ ( u i − 1 ( j ) , v i − 1 ( j ) ) ] p i − 1 ( y ∣ x ) , (11) p_i(y \mid x) = \left[1-\alpha_i(x,X_i)+\alpha_i(x,X_i)\prod_{j=1}^{q} C_{\rho}\left(u_{i-1}^{(j)},v_{i-1}^{(j)}\right)\right]p_{i-1}(y \mid x), \tag*{(11)} p i ( y ∣ x ) = [ 1 − α i ( x , X i ) + α i ( x , X i ) j = 1 ∏ q C ρ ( u i − 1 ( j ) , v i − 1 ( j ) ) ] p i − 1 ( y ∣ x ) , (11)
where u i − 1 ( j ) = P i − 1 ( y ( j ) ∣ y ( 1 : j − 1 ) , x ) u_{i-1}^{(j)}=P_{i-1}(y^{(j)}\mid y^{(1:j-1)},x) u i − 1 ( j ) = P i − 1 ( y ( j ) ∣ y ( 1 : j − 1 ) , x ) and v i − 1 ( j ) = P i − 1 ( Y i ( j ) ∣ Y i ( 1 : j − 1 ) , X i ) v_{i-1}^{(j)}=P_{i-1}(Y_i^{(j)}\mid Y_i^{(1:j-1)},X_i) v i − 1 ( j ) = P i − 1 ( Y i ( j ) ∣ Y i ( 1 : j − 1 ) , X i ) . This again follows from the same argument as in Fong et al. [ [12 ] based on the multivariate copula update where the only difference is that y y y can now also be multivariate. We highlight that an ordering of the dimensions of y y y must be specified, but this poses no real concern in practice; we discuss this in § 3.1 .
Following § 2.2 , we let p i − 1 ( y ∣ x ) = N ( y ; μ i − 1 ( x ) , Σ i − 1 ( x ) ) p_{i-1}(y \mid x) = \mathcal{N}(y; \mu_{i-1}(x), \Sigma_{i-1}(x)) p i − 1 ( y ∣ x ) = N ( y ; μ i − 1 ( x ) , Σ i − 1 ( x )) , where μ i − 1 ( x ) \mu_{i-1}(x) μ i − 1 ( x ) and Σ i − 1 ( x ) \Sigma_{i-1}(x) Σ i − 1 ( x ) are now the covariate-dependent mean vector and covariance matrix respectively. Plugging the Gaussian family into (11) yields the multivariate updates:
μ i ( x ) = μ i − 1 ( x ) + α i ( x , X i ) L i − 1 ( x ) ρ Z i , (12) \mu_i(x) = \mu_{i-1}(x) + \alpha_i(x, X_i) L_{i-1}(x) \rho Z_i, \tag*{(12)} μ i ( x ) = μ i − 1 ( x ) + α i ( x , X i ) L i − 1 ( x ) ρ Z i , (12)
Σ i ( x ) = L i − 1 ( x ) ( I q + α i ( x , X i ) ρ 2 { [ 1 − α i ( x , X i ) ] Z i Z i T − I q } ) L i − 1 ( x ) T (13) \Sigma_i(x) = L_{i-1}(x) \left(I_q + \alpha_i(x, X_i)\rho^2 \left\{[1-\alpha_i(x, X_i)]Z_i Z_i^{\mathsf{T}} - I_q\right\}\right) L_{i-1}(x)^{\mathsf{T}} \tag*{(13)} Σ i ( x ) = L i − 1 ( x ) ( I q + α i ( x , X i ) ρ 2 { [ 1 − α i ( x , X i )] Z i Z i T − I q } ) L i − 1 ( x ) T (13)
where L i ( x ) L_i(x) L i ( x ) is the Cholesky factor of Σ i ( x ) \Sigma_i(x) Σ i ( x ) , and the residual is now
Z i = L i − 1 − 1 ( X i ) [ Y i − μ i − 1 ( X i ) ] . (14) Z_i = L_{i-1}^{-1}(X_i) [Y_i - \mu_{i-1}(X_i)]. \tag*{(14)} Z i = L i − 1 − 1 ( X i ) [ Y i − μ i − 1 ( X i )] . (14)
The main difference is that the standard deviation function has now been replaced by the Cholesky factor of the covariance matrix. Algorithm 1 naturally extends to the multi-output case, where we now have that Z i ∼ N ( 0 , I ) Z_i \sim\mathcal{N}(0, I) Z i ∼ N ( 0 , I ) under predictive resampling, and the first two raw moment functions will once again be martingales.
Our recursive updates are highly flexible: they allow for arbitrary covariate-dependent variance and correlation between outcomes, and the updates are consistent under weak conditions and perform very well in practice. This is generally more challenging to attain with traditional GP approaches, where a latent function model must be explicitly assumed. Standard tractable coregionalization models [3 ] cannot attain covariate-dependent residual variance/correlations, and flexible models like [41 , 32 ] can have intractable components and require tuning-heavy approximations.
Example 4. We follow the same setup as Example 2 (for n n n , B B B , etc.) and extend to the q = 2 q = 2 q = 2 case, where the first outcome y ( 1 ) y^{(1)} y ( 1 ) is as before, and the second outcome y ( 2 ) y^{(2)} y ( 2 ) has mean and variance functions m ( − x ) m(-x) m ( − x ) and s 2 ( − x ) s^2(-x) s 2 ( − x ) respectively, and the outcome correlation function between y ( 1 ) y^{(1)} y ( 1 ) and y ( 2 ) y^{(2)} y ( 2 ) is r ( x ) = 0.8 tanh ( x / 2 ) r(x) = 0.8 \tanh(x/2) r ( x ) = 0.8 tanh ( x /2 ) . The recursive moment update required 1.0s to run, which is slightly longer than the single-output case as expected. Since the copula approach would now require a 200 × 200 200 \times200 200 × 200 grid for y ( 1 ) × y ( 2 ) y^{(1)} \times y^{(2)} y ( 1 ) × y ( 2 ) , this would cost on the order of 200 times the run-time of the single-output case, which would amount to roughly 10 hours.
Ordering
The Cholesky preconditioning by L i − 1 − 1 ( X i ) L_{i-1}^{-1}(X_i) L i − 1 − 1 ( X i ) in (14) gives an autoregressive interpretation for Z i Z_i Z i which we outline below.
Proposition 3. Let p i ( y ∣ x ) = N ( y ; μ i ( x ) , Σ i ( x ) ) p_i(y \mid x) = \mathcal{N}(y; \mu_i(x), \Sigma_i(x)) p i ( y ∣ x ) = N ( y ; μ i ( x ) , Σ i ( x )) with μ i \mu_i μ i and Σ i \Sigma_i Σ i following the updates (12) and (13). Then the j j j -th element of the residual takes the form
Z i ( j ) = Y i ( j ) − μ ~ i − 1 ( j ) ( X i ; Y i ( 1 : j − 1 ) ) σ ~ i − 1 ( j ) ( X i ; Y i ( 1 : j − 1 ) ) Z_i^{(j)} = \frac{Y_i^{(j)} - \widetilde{\mu}_{i-1}^{(j)}\left(X_i;Y_i^{(1:j-1)}\right)}{\widetilde{\sigma}_{i-1}^{(j)}\left(X_i;Y_i^{(1:j-1)}\right)} Z i ( j ) = σ i − 1 ( j ) ( X i ; Y i ( 1 : j − 1 ) ) Y i ( j ) − μ i − 1 ( j ) ( X i ; Y i ( 1 : j − 1 ) ) where μ ~ i − 1 ( j ) ( x ; y ( 1 : j − 1 ) ) \widetilde{\mu}_{i-1}^{(j)}(x;y^{(1:j-1)}) μ i − 1 ( j ) ( x ; y ( 1 : j − 1 ) ) and σ ~ i − 1 ( j ) ( x ; y ( 1 : j − 1 ) ) \widetilde{\sigma}_{i-1}^{(j)}(x;y^{(1:j-1)}) σ i − 1 ( j ) ( x ; y ( 1 : j − 1 ) ) are the mean and standard deviation respectively of the conditional density p i − 1 ( y ( j ) ∣ y ( 1 : j − 1 ) , x ) p_{i-1}(y^{(j)} \mid y^{(1:j-1)},x) p i − 1 ( y ( j ) ∣ y ( 1 : j − 1 ) , x ) .
Thus, Z i ( j ) Z_i^{(j)} Z i ( j ) is the standardized residual when sequentially predicting Y i ( j ) Y_i^{(j)} Y i ( j ) given Y i ( 1 : j − 1 ) Y_i^{(1:j-1)} Y i ( 1 : j − 1 ) . In particular, the update for μ i ( j ) \mu_i^{(j)} μ i ( j ) only depends on Y ( 1 : j ) Y^{(1:j)} Y ( 1 : j ) , and one can see that the update for μ i ( j ) \mu_i^{(j)} μ i ( j ) would be unchanged if only Y ( 1 : j ) Y^{(1:j)} Y ( 1 : j ) were observed instead of the full vector Y ( 1 : q ) Y^{(1:q)} Y ( 1 : q ) , akin to the marginal consistency property observed for the multivariate copula update in Fong et al. [ [12 ] . Importantly, we highlight that the ordering is not an assumption on the data generating process (DGP): the outcome covariance matrix Σ i ( x ) \Sigma_i(x) Σ i ( x ) can still be fully dense.
In practice, we do not see a large difference in performance due to ordering, and frequentist consistency is also agnostic to this choice. If there is missing data, it can be helpful to order the outcomes so that the dimensions with the most missing entries come last as we will see later. Another benefit of the Cholesky form of L i ( x ) L_i(x) L i ( x ) is that each update only costs O ( q 2 ) O(q^2) O ( q 2 ) as we can update L i − 1 L_{i-1} L i − 1 to L i L_i L i with a rank-1 Cholesky update, avoiding the potential O ( q 3 ) O(q^3) O ( q 3 ) complexity of a generic matrix square root.
Missing outcomes
Missing outcomes occur very naturally within the multi-output setting, and handling missingness is a popular use case for multi-output regression methods [3 ] . For example, a sensor may only capture a subset of signals at each location, or certain measurements may be much more expensive than others: multi-output regression allows for the prediction of missing outcomes using the learned correlations, which in our case can be covariate-dependent.
We now derive the moment matching updates for the copula update when marginalizing out the missing components. Let O i ⊆ { 1 , … , q } O_i \subseteq\{1,\ldots,q\} O i ⊆ { 1 , … , q } denote the set of observed outputs at observation i i i , and let S i S_i S i be the corresponding selection matrix of size ∣ O i ∣ × q |O_i| \times q ∣ O i ∣ × q . Let M i = { 1 , … , q } ∖ O i M_i = \{1,\ldots,q\} \setminus O_i M i = { 1 , … , q } ∖ O i denote the missingness pattern. We thus have Y i ( O i ) = S i Y i Y_i^{(O_i)} = S_iY_i Y i ( O i ) = S i Y i and Y i ( M i ) = Y i ∖ Y i ( O i ) Y_i^{(M_i)} = Y_i \setminus Y_i^{(O_i)} Y i ( M i ) = Y i ∖ Y i ( O i ) , where we only observe Y i ( O i ) Y_i^{(O_i)} Y i ( O i ) . We assume throughout that the outcomes are missing at random (MAR), which implies that
p i − 1 ( Y i ( M i ) ∣ Y i ( O i ) , O i , X i ) = p i − 1 ( Y i ( M i ) ∣ Y i ( O i ) , X i ) . p_{i-1}\left(Y_i^{(M_i)} \mid Y_i^{(O_i)}, O_i, X_i\right) = p_{i-1}\left(Y_i^{(M_i)} \mid Y_i^{(O_i)}, X_i\right). p i − 1 ( Y i ( M i ) ∣ Y i ( O i ) , O i , X i ) = p i − 1 ( Y i ( M i ) ∣ Y i ( O i ) , X i ) .
Note that the subscript ( i − 1 ) (i-1) ( i − 1 ) indicates additional conditioning on ( Y j , X j ) j = 1 : i − 1 (Y_j, X_j)_{j=1:i-1} ( Y j , X j ) j = 1 : i − 1 . Starting again with the fully observed copula update, we consider the update from p i − 1 ( y ∣ x ) p_{i-1}(y \mid x) p i − 1 ( y ∣ x ) to p i − 1 ( y ∣ x , Y i ( O i ) , X i ) p_{i-1}(y \mid x, Y_i^{(O_i)}, X_i) p i − 1 ( y ∣ x , Y i ( O i ) , X i ) . The key is to write
p i − 1 ( y ∣ x , Y i ( O i ) , X i ) = E Y i ( M i ) [ p i ( y ∣ x ) ∣ Y i ( O i ) , X i ] . p_{i-1}\left(y \mid x, Y_i^{(O_i)}, X_i\right) = \mathbb{E}_{Y_i^{(M_i)}}\left[p_i(y \mid x) \mid Y_i^{(O_i)}, X_i\right]. p i − 1 ( y ∣ x , Y i ( O i ) , X i ) = E Y i ( M i ) [ p i ( y ∣ x ) ∣ Y i ( O i ) , X i ] .
where Y i ( M i ) ∼ p i − 1 ( Y i ( M i ) ∣ Y i ( O i ) , X i ) Y_i^{(M_i)} \sim p_{i-1}(Y_i^{(M_i)} \mid Y_i^{(O_i)}, X_i) Y i ( M i ) ∼ p i − 1 ( Y i ( M i ) ∣ Y i ( O i ) , X i ) , and p i ( y ∣ x ) p_i(y \mid x) p i ( y ∣ x ) denotes the predictive density updated with (11) given the fully observed outcome Y i = ( Y i ( O i ) , Y i ( M i ) ) Y_i = (Y_i^{(O_i)}, Y_i^{(M_i)}) Y i = ( Y i ( O i ) , Y i ( M i ) ) . The k k k -th updated moment m i ( k ) : = ∫ y ⊗ k p i − 1 ( y ∣ x , Y i ( O i ) , X i ) d y m_i^{(k)} := \int y^{\otimes k}p_{i-1}(y \mid x, Y_i^{(O_i)}, X_i)\,dy m i ( k ) := ∫ y ⊗ k p i − 1 ( y ∣ x , Y i ( O i ) , X i ) d y then satisfies
m i ( k ) = [ 1 − α i ( x , X i ) ] m i − 1 ( k ) ( x ) + α i ( x , X i ) E Y i ( M i ) [ m ~ i ( k ) ( x ) ∣ Y i ( O i ) , X i ] (15) m_i^{(k)} = \left[1-\alpha_i(x,X_i)\right]m_{i-1}^{(k)}(x) + \alpha_i(x,X_i)\mathbb{E}_{Y_i^{(M_i)}}\left[\widetilde{m}_i^{(k)}(x) \mid Y_i^{(O_i)}, X_i\right] \tag*{(15)} m i ( k ) = [ 1 − α i ( x , X i ) ] m i − 1 ( k ) ( x ) + α i ( x , X i ) E Y i ( M i ) [ m i ( k ) ( x ) ∣ Y i ( O i ) , X i ] (15)
where m ~ i ( k ) ( x ) \widetilde{m}_i^{(k)}(x) m i ( k ) ( x ) is the moment of the copula kernel as in the single-output case. We now present the general Gaussian moment update under missingness.
Proposition 4. Let p i − 1 ( y ∣ x ) = N ( y ; μ i − 1 ( x ) , Σ i − 1 ( x ) ) p_{i-1}(y \mid x) = \mathcal{N}(y; \mu_{i-1}(x), \Sigma_{i-1}(x)) p i − 1 ( y ∣ x ) = N ( y ; μ i − 1 ( x ) , Σ i − 1 ( x )) . Then (15) corresponds to the mean and variance updates
μ i ( x ) = μ i − 1 ( x ) + α i ( x , X i ) ρ L i − 1 ( x ) Z ‾ i , (16) \mu_i(x) = \mu_{i-1}(x) + \alpha_i(x,X_i)\rho L_{i-1}(x)\overline{Z}_i, \tag*{(16)} μ i ( x ) = μ i − 1 ( x ) + α i ( x , X i ) ρ L i − 1 ( x ) Z i , (16) Σ i ( x ) = L i − 1 ( x ) [ I + α i ( x , X i ) ρ 2 { [ 1 − α i ( x , X i ) ] Z ‾ i Z ‾ i T − Q i } ] L i − 1 ( x ) T (17) \Sigma_i(x) = L_{i-1}(x)\left[I + \alpha_i(x,X_i)\rho^2\left\{\left[1-\alpha_i(x,X_i)\right]\overline{Z}_i\overline{Z}_i^{\mathsf{T}} - Q_i\right\}\right]L_{i-1}(x)^{\mathsf{T}} \tag*{(17)} Σ i ( x ) = L i − 1 ( x ) [ I + α i ( x , X i ) ρ 2 { [ 1 − α i ( x , X i ) ] Z i Z i T − Q i } ] L i − 1 ( x ) T (17) where Q i Q_i Q i is a projection matrix taking the form
Q i = A i T ( A i A i T ) − 1 A i , A i = S i L i − 1 ( X i ) , (18) Q_i = A_i^{\mathsf{T}}(A_iA_i^{\mathsf{T}})^{-1}A_i,\qquad A_i = S_iL_{i-1}(X_i), \tag*{(18)} Q i = A i T ( A i A i T ) − 1 A i , A i = S i L i − 1 ( X i ) , (18) and Z ‾ i = Q i Z i \overline{Z}_i = Q_iZ_i Z i = Q i Z i is the projected residual with Z i Z_i Z i defined in (14).
The above moment updates are simply projected versions of the fully observed updates (12) and (13). Here, Q i Q_i Q i projects the fully observed residual Z i Z_i Z i into the subspace revealed by Y i ( O i ) Y_i^{(O_i)} Y i ( O i ) , and the update only depends on Y i ( O i ) Y_i^{(O_i)} Y i ( O i ) as A i Z i = S i [ Y i − μ i − 1 ( X i ) ] A_iZ_i = S_i[Y_i-\mu_{i-1}(X_i)] A i Z i = S i [ Y i − μ i − 1 ( X i )] . The projection space is particularly clear when the missingness pattern is order-compatible in the sense that O i = { 1 , … , q i } O_i = \{1,\ldots,q_i\} O i = { 1 , … , q i } for some q i ≤ q q_i \leq q q i ≤ q , as the Cholesky form of L i − 1 L_{i-1} L i − 1 gives Q i = [ I q i 0 0 0 ] Q_i = \begin{bmatrix} I_{q_i} & 0 \\ 0 & 0 \end{bmatrix} Q i = [ I q i 0 0 0 ] in this setting. For example, for q = 2 q=2 q = 2 where Y i ( 2 ) Y_i^{(2)} Y i ( 2 ) is missing, we have
μ i ( 2 ) ( x ) = μ i − 1 ( 2 ) ( x ) + α i ( x , X i ) L i − 1 ( 2 , 1 ) ( x ) ρ Z i ( 1 ) , \mu_i^{(2)}(x)=\mu_{i-1}^{(2)}(x)+\alpha_i(x,X_i)L_{i-1}^{(2,1)}(x)\rho Z_i^{(1)}, μ i ( 2 ) ( x ) = μ i − 1 ( 2 ) ( x ) + α i ( x , X i ) L i − 1 ( 2 , 1 ) ( x ) ρ Z i ( 1 ) ,
where Z i ( 1 ) Z_i^{(1)} Z i ( 1 ) is as in Proposition 3, which follows as E Y i ( 2 ) [ Z i ( 2 ) ∣ Y i ( 1 ) , X i ] = 0 E_{Y_i^{(2)}}\left[Z_i^{(2)}\mid Y_i^{(1)},X_i\right]=0 E Y i ( 2 ) [ Z i ( 2 ) ∣ Y i ( 1 ) , X i ] = 0 . The update for μ i ( 1 ) ( x ) \mu_i^{(1)}(x) μ i ( 1 ) ( x ) is then unchanged.
Our proposed moment update under missingness also satisfies an intuitive property on the preservation of conditional distributions.
Proposition 5. Let p i ( y ∣ x ) = N ( y ; μ i ( x ) , Σ i ( x ) ) p_i(y\mid x)=\mathcal{N}\left(y;\mu_i(x),\Sigma_i(x)\right) p i ( y ∣ x ) = N ( y ; μ i ( x ) , Σ i ( x ) ) for all i ≥ 0 i\geq0 i ≥ 0 . Under the moment updates (16) and (17), the conditional densities satisfy
p i ( y ( M i ) ∣ y ( O i ) , x = X i ) = p i − 1 ( y ( M i ) ∣ y ( O i ) , x = X i ) p_i\left(y^{(M_i)}\mid y^{(O_i)},x=X_i\right)=p_{i-1}\left(y^{(M_i)}\mid y^{(O_i)},x=X_i\right) p i ( y ( M i ) ∣ y ( O i ) , x = X i ) = p i − 1 ( y ( M i ) ∣ y ( O i ) , x = X i ) for all y = { y ( M i ) ∪ y ( O i ) } ∈ R q y=\left\{y^{(M_i)}\cup y^{(O_i)}\right\}\in\mathbb{R}^q y = { y ( M i ) ∪ y ( O i ) } ∈ R q . Furthermore, if O i O_i O i is order-compatible, then the above holds with X i X_i X i replaced with any x ∈ X x\in\mathcal{X} x ∈ X .
The above highlights a desirable property of the update: only observing Y i ( O i ) Y_i^{(O_i)} Y i ( O i ) affects the marginal distribution of y ( O i ) y^{(O_i)} y ( O i ) , but not the predictive distribution of y ( M i ) y^{(M_i)} y ( M i ) given y ( O i ) y^{(O_i)} y ( O i ) . We highlight that the marginal distribution of y ( M i ) y^{(M_i)} y ( M i ) can still change, but only due to mixing over the updated marginal of y ( O i ) y^{(O_i)} y ( O i ) . In § 4.2 , we further verify frequentist consistency of the updates under the slightly stricter regime where the outcomes are covariate-dependent MAR, that is where O i O_i O i may depend on X i X_i X i but not Y i ( O i ) Y_i^{(O_i)} Y i ( O i ) . Finally, we briefly discuss predictive resampling - it is natural to carry out predictive resampling with fully observed outcomes for identifiability, corresponding to drawing X i ∼ i i d P X X_i \mathrel{\overset{\mathrm{iid}}{\sim}} P_X X i ∼ iid P X and always setting O i = { 1 , … , q } O_i=\{1,\ldots,q\} O i = { 1 , … , q } .
Generalizations
Generalizations to use other families beyond the Gaussian in the multi-output setting follow naturally from Sections 2.2.3 and 2.2.4, and allow us to capture heavy-tailedness or other properties in the multi-output setting. In the interest of space, we defer further details to Appendix § 13 .2.3.
Practical considerations
We briefly discuss some practical considerations which apply to both the single- and multi-output settings in the fully observed case. In Appendix E.3, we outline similar details for the missing-output updates. As the recursive approaches are not exchangeable, we follow Fong et al. [2023] and average over 10 random permutations of the dataset when estimating μ n ( x ) \mu_n(x) μ n ( x ) and σ n ( x ) \sigma_n(x) σ n ( x ) , averaging on the raw moment scale. For larger n n n , we use 1 or 2 permutations for hyperparameter tuning, but 10 for subsequent fitting. We standardize the outcomes and covariates, and initialize our estimates at μ 0 ( x ) = 0 \mu_0(x) = 0 μ 0 ( x ) = 0 and Σ 0 ( x ) = I q \Sigma_0(x) = I_q Σ 0 ( x ) = I q . All hyperparameters (δ \delta δ , ρ \rho ρ and ρ x \rho_x ρ x ) are tuned by maximizing the prequential log-likelihood ∑ i = 1 n log p i − 1 ( Y i ∣ X i ) \sum_{i=1}^n \log p_{i-1}(Y_i \mid X_i) ∑ i = 1 n log p i − 1 ( Y i ∣ X i ) . Fitting the model involves computing Z 1 : n Z_{1:n} Z 1 : n which costs O ( n 2 ( p + q 2 ) ) O(n^2(p + q^2)) O ( n 2 ( p + q 2 )) , with an additional O ( n ( p + q 2 ) ) O(n(p + q^2)) O ( n ( p + q 2 )) for each prediction. Although the cost grows quadratically with n n n , it is significantly faster than the cubic case for exact GPs (and even variational approximations), and orders of magnitude faster than MCMC approaches for dependent DP mixtures.
Asymptotic theory
Predictive asymptotics
We now consider the theory underlying predictive resampling in our semiparametric martingale posterior. Since p i ( y ∣ x ) p_i(y \mid x) p i ( y ∣ x ) is restricted to be parametric and is no longer a martingale, we do not satisfy the conditionally identically distributed (c.i.d.) property [5 ] . Instead, we will leverage the theory of Banach-valued martingales [29 , 11 ] to directly show convergence of μ i ( ⋅ ) \mu_i(\cdot) μ i ( ⋅ ) and σ i 2 ( ⋅ ) \sigma_i^2(\cdot) σ i 2 ( ⋅ ) , which then drives convergence of the joint predictive density p i ( y , x ) p_i(y, x) p i ( y , x ) .
We consider the univariate case first. As a reminder, we will assume throughout that X \mathcal{X} X is compact. For convenience, we will usually omit the dependence on x x x .
Assumption 1 (Similarity). The similarity function s s s from (2) is well-defined on X × X \mathcal{X} \times\mathcal{X} X × X and satisfies S = sup x ∈ X s ( x , x ) < ∞ S = \sup_{x \in\mathcal{X}} s(x, x) < \infty S = sup x ∈ X s ( x , x ) < ∞ .
Assumption 2 (Bandwidth/learning rate). Let ρ ∈ ( 0 , 1 ) \rho\in(0, 1) ρ ∈ ( 0 , 1 ) and α i = O ( i − m ) \alpha_i = O(i^{-m}) α i = O ( i − m ) where 0.5 < m ≤ 1 0.5 < m \le1 0.5 < m ≤ 1 with 0 < α i < 1 0 < \alpha_i < 1 0 < α i < 1 for all i i i .
Assumption 3 (Initialization). Let μ 0 ( x ) = 0 \mu_0(x) = 0 μ 0 ( x ) = 0 and σ 0 2 ( x ) = 1 \sigma_0^2(x) = 1 σ 0 2 ( x ) = 1 for all x ∈ X x \in\mathcal{X} x ∈ X .
Assumption 4 (Design). Let the covariate distribution for predictive resampling P X P_X P X have compact support X \mathcal{X} X and Lebesgue density p X p_X p X .
The first assumption holds if s ( x , x ′ ) s(x,x') s ( x , x ′ ) is continuous and X \mathcal{X} X is compact, and the second is satisfied as we have α i = ( 2 − i − 1 ) ( i + 1 ) − 0.9 \alpha_i=(2-i^{-1})(i+1)^{-0.9} α i = ( 2 − i − 1 ) ( i + 1 ) − 0.9 . Assumption 3 can easily be relaxed, but this form is convenient. We then have our main convergence result.
Theorem 1. Under Algorithm 1 and Assumptions 1-4, there exist random μ ∞ ∈ L 2 ( X ) \mu_\infty\in L^2(\mathcal{X}) μ ∞ ∈ L 2 ( X ) and σ ∞ 2 ∈ L 2 ( X ) \sigma_\infty^2 \in L^2(\mathcal{X}) σ ∞ 2 ∈ L 2 ( X ) , with σ ∞ 2 ( x ) > 0 \sigma_\infty^2(x)>0 σ ∞ 2 ( x ) > 0 for Lebesgue-almost all x ∈ X x \in\mathcal{X} x ∈ X , such that
∫ X [ μ ∞ ( x ) − μ i ( x ) ] 2 d x → 0 , ∫ X [ σ ∞ 2 ( x ) − σ i 2 ( x ) ] 2 d x → 0 a.s. \int_{\mathcal{X}} [\mu_\infty(x)-\mu_i(x)]^2\,dx \to0,\qquad\int_{\mathcal{X}} [\sigma_\infty^2(x)-\sigma_i^2(x)]^2\,dx \to0 \quad\text{a.s.} ∫ X [ μ ∞ ( x ) − μ i ( x ) ] 2 d x → 0 , ∫ X [ σ ∞ 2 ( x ) − σ i 2 ( x ) ] 2 d x → 0 a.s. Furthermore, we have the total variation convergence
1 2 ∫ X [ ∫ ∣ p i ( y ∣ x ) − N { y ; μ ∞ ( x ) , σ ∞ 2 ( x ) } ∣ d y ] p X ( x ) d x → 0 a.s. \frac{1}{2}\int_{\mathcal{X}}\left[\int\left|p_i(y\mid x)-\mathcal{N}\{y;\mu_\infty(x),\sigma_\infty^2(x)\}\right|\,dy\right]p_{\mathcal{X}}(x)\,dx \to0 \quad\text{a.s.} 2 1 ∫ X [ ∫ p i ( y ∣ x ) − N { y ; μ ∞ ( x ) , σ ∞ 2 ( x )} d y ] p X ( x ) d x → 0 a.s.
The above theorem validates the convergence of predictive resampling, as the mean and variance functions and the predictive density all converge a.s. The total variation result can be viewed as a functional version of the results in Garelli et al. [2026], and moment martingales in the non-regression setting were also investigated by Yung et al. [2025]. We highlight that the above theorem is significantly stronger than previous results for the regression setting, e.g. Fong et al. [2023], which only show a.s. weak convergence of the joint predictive distribution on ( Y , X ) (Y,X) ( Y , X ) . In particular, we have convergence of the conditional moment functions, and the total variation result is much stronger than a.s. weak convergence as it means the limiting measure has a density p ∞ ( y , x ) p_\infty(y,x) p ∞ ( y , x ) , following Berti et al. [2013]. The multi-output version of the above Theorem 1 is very similar and outlined in Appendix C.§ 2.2 .
Under a smoothness condition on s ( x , x ′ ) s(x,x') s ( x , x ′ ) , we can show a stronger result that the martingale posterior has support over smooth mean and variance functions. For a Lipschitz domain D D D and integer r r r , let H r ( D ) = W r , 2 ( D ) H^r(D)=W^{r,2}(D) H r ( D ) = W r , 2 ( D ) denote the Sobolev space, where H r ( D ) H^r(D) H r ( D ) is a Hilbert space with norm ∥ ⋅ ∥ H r \|\cdot\|_{H^r} ∥ ⋅ ∥ H r .
Assumption 5 (Sobolev regularity). Let X = D \mathcal{X}=D X = D , where D ⊂ R p D\subset\mathbb{R}^p D ⊂ R p is a bounded Lipschitz domain. For some integer r ≥ 1 r\ge1 r ≥ 1 , suppose that all partial derivatives of s ( x , x ′ ) s(x,x') s ( x , x ′ ) with respect to its first input x x x , up to and including total order r r r , exist and are jointly continuous in ( x , x ′ ) ∈ X × X (x,x')\in\mathcal{X}\times\mathcal{X} ( x , x ′ ) ∈ X × X .
Theorem 2. Under Algorithm 1 and Assumptions 1-5, there exist random functions μ ∞ ∈ H r ( D ) \mu_\infty\in H^r(D) μ ∞ ∈ H r ( D ) and σ ∞ 2 ∈ H r ( D ) \sigma_\infty^2\in H^r(D) σ ∞ 2 ∈ H r ( D ) , with σ ∞ 2 ( x ) > 0 \sigma_\infty^2(x)>0 σ ∞ 2 ( x ) > 0 for Lebesgue-almost all x x x , such that
∥ μ ∞ − μ i ∥ H r → 0 , ∥ σ ∞ 2 − σ i 2 ∥ H r → 0 a.s. \|\mu_\infty-\mu_i\|_{H^r}\to0,\qquad\|\sigma_\infty^2-\sigma_i^2\|_{H^r}\to0 \quad\text{a.s.} ∥ μ ∞ − μ i ∥ H r → 0 , ∥ σ ∞ 2 − σ i 2 ∥ H r → 0 a.s.
The control of the Sobolev norm follows from the nice property of the Gaussian having finite moments of all orders. We note that r r r controls the smoothness of the resulting Sobolev space, which in turn is driven by the smoothness of s ( x , x ′ ) s(x,x') s ( x , x ′ ) . For r > p / 2 r > p/2 r > p /2 , it is well known that the Sobolev space H r H^r H r can be embedded into (Hölder) continuous spaces [24 ] , e.g. μ ∞ \mu_\infty μ ∞ and σ ∞ 2 \sigma_\infty^2 σ ∞ 2 are equal almost-everywhere to continuous representatives. Furthermore, if r > k + p / 2 r > k + p/2 r > k + p /2 for some positive integer k k k , then functions in H r ( D ) H^r(D) H r ( D ) have representatives with k k k continuous derivatives.
Unsurprisingly, the Gaussian choice of s ( x , x ′ ) s(x,x') s ( x , x ′ ) in Example 1 results in very smooth posterior samples, as we outline in the below corollary.
Corollary 1. Under the assumptions of Theorem 2, for the Gaussian choice of s ( x , x ′ ) s(x,x') s ( x , x ′ ) as in Example 1, posterior samples of μ ∞ \mu_\infty μ ∞ and σ ∞ 2 \sigma_\infty^2 σ ∞ 2 have infinitely differentiable representatives a.s.
We also have the following extension to the heavy-tailed setting, which holds as the SU distribution has well-defined moments despite being heavy-tailed.
Corollary 2. Theorems 1 and 2 both hold under the same assumptions if p i ( y ∣ x ) = p S U ( y ; μ i ( x ) , σ i 2 ( x ) , δ ) p_i(y \mid x) = p^{\mathrm{SU}}(y; \mu_i(x), \sigma_i^2(x), \delta) p i ( y ∣ x ) = p SU ( y ; μ i ( x ) , σ i 2 ( x ) , δ ) for fixed δ > 0 \delta> 0 δ > 0 and the moment updates (8) and (9) are used instead, and the total variation limit is p S U ( y ; μ ∞ ( x ) , σ ∞ 2 ( x ) , δ ) p^{\mathrm{SU}}(y; \mu_\infty(x), \sigma_\infty^2(x), \delta) p SU ( y ; μ ∞ ( x ) , σ ∞ 2 ( x ) , δ ) .
Frequentist asymptotics
In this section, we investigate the frequentist properties of our proposed recursive updates under i.i.d. samples ( Y i , X i ) ∼ i i d P ∗ (Y_i, X_i) \overset{\mathrm{iid}}{\sim} P^* ( Y i , X i ) ∼ iid P ∗ . We let μ ∗ , σ ∗ 2 \mu_*, \sigma_*^2 μ ∗ , σ ∗ 2 and Σ ∗ \Sigma_* Σ ∗ denote the true mean, variance and covariance matrix functions in the respective single and multi-output settings. The main difficulty in establishing consistency is that the mean and variance updates in (5) and (6) are coupled in a nonlinear fashion. Other nonparametric recursive estimates for the mean function [42 , 23 ] rely on a linear update without a data-dependent variance component, and are thus much easier to study compared to our case. Like other proofs for recursive predictive methods [39 , 16 ] , our results rely on the construction of an almost-supermartingale [33 ] to show convergence, followed by a consistency argument. However, due to the semiparametric nature of the update, novel techniques are needed.
Our main finding is that our recursive moment updates are consistent under very weak assumptions on P ∗ P^* P ∗ , even if p ∗ ( y ∣ x ) p^*(y \mid x) p ∗ ( y ∣ x ) is not Gaussian, and this extends to the multivariate case under covariate-dependent MAR. The tradeoff of this generality is that consistency is measured using a norm induced by the similarity function s ( x , x ′ ) s(x,x') s ( x , x ′ ) , rather than in a stronger L 1 L^1 L 1 or L 2 L^2 L 2 sense. For comparison, in the simpler setting of single-output online mean function estimation, Lei et al. [ [23 ] obtain a.s. L 2 L^2 L 2 consistency but under much stronger assumptions than our setting, e.g. requiring a bounded outcome space and the mean function living in a reproducing kernel Hilbert space.
Single-output
We begin with the single-output case before extending to the general multi-output setting. To that end, we state some additional required assumptions.
Assumption 6 (DGP). Let Y ∈ R Y \in\mathbb{R} Y ∈ R , and suppose P X ∗ P_X^* P X ∗ has compact support X \mathcal{X} X and Lebesgue density p X ∗ p_X^* p X ∗ , and E P ∗ [ Y 4 ] < ∞ \mathbb{E}_{P^*}[Y^4] < \infty E P ∗ [ Y 4 ] < ∞ and E P ∗ [ ∣ log σ ∗ 2 ( X ) ∣ ] < ∞ \mathbb{E}_{P^*}[|\log\sigma_*^2(X)|] < \infty E P ∗ [ ∣ log σ ∗ 2 ( X ) ∣ ] < ∞ .
Assumption 7 (Bandwidth/learning rate). Let ρ ∈ ( 0 , 1 ) \rho\in(0,1) ρ ∈ ( 0 , 1 ) and α i = α 0 ( i + 1 ) − 1 \alpha_i = \alpha_0(i+1)^{-1} α i = α 0 ( i + 1 ) − 1 , where 0 < α 0 < min { 1 , ( 2 ρ 2 S ) − 1 } 0 < \alpha_0 < \min\{1,(2\rho^2 S)^{-1}\} 0 < α 0 < min { 1 , ( 2 ρ 2 S ) − 1 } for S S S defined in Assumption 1.
Assumption 8 (Regularity of kernel). The kernel k ( x ; ψ ) k(x;\psi) k ( x ; ψ ) is jointly continuous in ( x , ψ ) (x,\psi) ( x , ψ ) , and inf x ∈ X E Ψ [ k ( x ; Ψ ) ] > 0 \inf_{x\in\mathcal{X}} \mathbb{E}_{\Psi}[k(x;\Psi)] > 0 inf x ∈ X E Ψ [ k ( x ; Ψ )] > 0 .
Assumption 9 (Completeness of kernel). The family { k ( x ; ψ ) : ψ ∈ supp ( Π ) } \{k(x;\psi):\psi\in\operatorname{supp}(\Pi)\} { k ( x ; ψ ) : ψ ∈ supp ( Π )} is complete in the sense of Definition 1 and its support contains X \mathcal{X} X .
A sufficient condition for the integrability of log σ ∗ 2 ( x ) \log\sigma_*^2(x) log σ ∗ 2 ( x ) in Assumption 6 is that σ ∗ 2 ( x ) \sigma_*^2(x) σ ∗ 2 ( x ) is continuous and σ ∗ 2 ( x ) > 0 \sigma_*^2(x) > 0 σ ∗ 2 ( x ) > 0 for all x ∈ X x \in\mathcal{X} x ∈ X where X \mathcal{X} X is compact. We believe that Assumption 7 is stricter than necessary and is a technical artifact in the proof when controlling the almost-supermartingale, and we suspect consistency holds under a more general α i = α 0 ( i − m ) \alpha_i = \alpha_0(i^{-m}) α i = α 0 ( i − m ) for 0.5 < m ≤ 1 0.5 < m \le1 0.5 < m ≤ 1 . Finally, we see that completeness of the kernel will be responsible for ensuring consistency.
For a function f ∈ L 1 ( P X ∗ ) f \in L^1(P_X^*) f ∈ L 1 ( P X ∗ ) , let ∥ ⋅ ∥ s : L 1 ( P X ∗ ) → [ 0 , ∞ ) \|\cdot\|_s : L^1(P_X^*) \to[0,\infty) ∥ ⋅ ∥ s : L 1 ( P X ∗ ) → [ 0 , ∞ ) be defined as
∥ f ∥ s 2 = ∫ f ( x ) f ( x ′ ) s ( x , x ′ ) d P X ∗ ( x ) d P X ∗ ( x ′ ) . \|f\|_s^2 = \int f(x)f(x')s(x,x')\,\mathrm{d}P_X^*(x)\,\mathrm{d}P_X^*(x'). ∥ f ∥ s 2 = ∫ f ( x ) f ( x ′ ) s ( x , x ′ ) d P X ∗ ( x ) d P X ∗ ( x ′ ) .
Next, let us consider the error functions
u n = μ ∗ − μ n σ n , w n = σ ∗ 2 + ( μ ∗ − μ n ) 2 σ n 2 − 1. u_n = \frac{\mu_*-\mu_n}{\sigma_n}, \qquad w_n = \frac{\sigma_*^2+(\mu_*-\mu_n)^2}{\sigma_n^2}-1. u n = σ n μ ∗ − μ n , w n = σ n 2 σ ∗ 2 + ( μ ∗ − μ n ) 2 − 1.
Theorem 3. Under Assumptions 1, 3, 6, 7, the updates (5) and (6) satisfy
∥ u n ∥ s → 0 , ∥ w n ∥ s → 0 P ∗ -a.s. \|u_n\|_s \to0,\qquad\|w_n\|_s \to0 \quad P^*\text{-a.s.} ∥ u n ∥ s → 0 , ∥ w n ∥ s → 0 P ∗ -a.s.
Furthermore, under Assumptions 8 and 9, ∥ ⋅ ∥ s \lVert\cdot\rVert_s ∥ ⋅ ∥ s is a norm on L 1 ( P X ∗ ) L^1(P_X^*) L 1 ( P X ∗ ) and ∥ u n ∥ s = ∥ w n ∥ s = 0 \lVert u_n\rVert_s = \lVert w_n\rVert_s = 0 ∥ u n ∥ s = ∥ w n ∥ s = 0 if and only if μ n ( x ) = μ ∗ ( x ) \mu_n(x) = \mu_*(x) μ n ( x ) = μ ∗ ( x ) and σ n 2 ( x ) = σ ∗ 2 ( x ) \sigma_n^2(x) = \sigma_*^2(x) σ n 2 ( x ) = σ ∗ 2 ( x ) for P X ∗ P_X^* P X ∗ -almost all x x x .
We first highlight that without the completeness condition in Assumption 9, ∥ f ∥ s = 0 \lVert f\rVert_s = 0 ∥ f ∥ s = 0 may not necessarily give f = 0 f = 0 f = 0 for P X ∗ P_X^* P X ∗ -almost all x x x . Furthermore, one can show that the norm ∥ ⋅ ∥ s \lVert\cdot\rVert_s ∥ ⋅ ∥ s is weaker than the more familiar ∥ ⋅ ∥ L 1 ( P X ∗ ) \lVert\cdot\rVert_{L^1(P_X^*)} ∥ ⋅ ∥ L 1 ( P X ∗ ) , due to the smoothing induced by s ( x , x ′ ) s(x,x') s ( x , x ′ ) . In Appendix D.§ 5 , we give further discussion on how the norm ∥ ⋅ ∥ s \lVert\cdot\rVert_s ∥ ⋅ ∥ s is related to weak convergence in the function space. The main technical obstruction to strengthening the convergence is to control high-frequency oscillations in u n u_n u n and w n w_n w n , which is nontrivial due to the nonlinear mean and variance update. Nonetheless, we see that this does not occur in practice, and we leave this for future work. Finally, we note that these weak norm issues vanish if X \mathcal{X} X is finite.
The next corollary extends our consistency result to the generalized moment updates corresponding to the SU distribution given in § 2.2.3 . Note that here we require ρ i → 1 \rho_i \to1 ρ i → 1 so the fixed points of the moment updates are the true moment functions, although in practice we recommend using a fixed prequentially tuned ρ \rho ρ .
Corollary 3. Fix δ > 0 \delta> 0 δ > 0 and let 0 < ρ i < 1 0 < \rho_i < 1 0 < ρ i < 1 satisfy 1 − ρ i = O ( α i ) 1-\rho_i = O(\alpha_i) 1 − ρ i = O ( α i ) . Under the assumptions of Theorem 3, with bandwidth sequence ρ i \rho_i ρ i , and the strengthened condition 0 < α 0 < min { 1 , ( 2 S ) − 1 } 0 < \alpha_0 < \min\{1,(2S)^{-1}\} 0 < α 0 < min { 1 , ( 2 S ) − 1 } , the conclusions of Theorem 3 hold for the moment updates (8) and (9).
Multi-output
Importantly, Theorem 3 can be generalized to the multivariate case with missing data. Here, we consider the q = 2 q = 2 q = 2 case with order-compatible missingness as it is the clearest to present; the general case with any q q q and arbitrary missingness pattern also holds true, which we leave for Appendix D.§ 4 . Let ( Y i , X i , O i ) ∼ i i d P ∗ (Y_i, X_i, O_i) \overset{\mathrm{iid}}{\sim} P^* ( Y i , X i , O i ) ∼ iid P ∗ , and let S i S_i S i be the random selection matrix. We have the multi-output version of Assumption 6.
Assumption 10 (Multi-output). Let Y ∈ R 2 Y \in\mathbb{R}^2 Y ∈ R 2 , O ∈ O = { { 1 } , { 1 , 2 } } O \in\mathcal{O} = \{\{1\},\{1,2\}\} O ∈ O = {{ 1 } , { 1 , 2 }} , and suppose P X ∗ P_X^* P X ∗ has compact support X \mathcal{X} X and Lebesgue density p X ∗ p_X^* p X ∗ . Assume E P ∗ [ ∥ Y ∥ 4 ] < ∞ \mathbb{E}_{P^*}[\lVert Y\rVert^4] < \infty E P ∗ [∥ Y ∥ 4 ] < ∞ and E P ∗ [ ∣ log det Σ ∗ ( X ) ∣ ] < ∞ \mathbb{E}_{P^*}[|\log\det\Sigma^*(X)|] < \infty E P ∗ [ ∣ log det Σ ∗ ( X ) ∣ ] < ∞ . Furthermore, assume covariate-dependent MAR, that is O ⊥ Y ∣ X O \perp Y \mid X O ⊥ Y ∣ X , with P ∗ ( O = { 1 , 2 } ∣ x ) > 0 P^*(O = \{1,2\}\mid x) > 0 P ∗ ( O = { 1 , 2 } ∣ x ) > 0 for P X ∗ P_X^* P X ∗ -almost all x x x .
For a vector-valued function f ∈ L 1 ( P X ∗ ; R q ) f \in L^1(P_X^*;\mathbb{R}^q) f ∈ L 1 ( P X ∗ ; R q ) and a matrix-valued function F ∈ L 1 ( P X ∗ ; R q × q ) F \in L^1(P_X^*;\mathbb{R}^{q \times q}) F ∈ L 1 ( P X ∗ ; R q × q ) , define ∥ f ∥ s 2 = ∑ j = 1 q ∥ f ( j ) ∥ s 2 \lVert f\rVert_s^2 = \sum_{j=1}^q \lVert f^{(j)}\rVert_s^2 ∥ f ∥ s 2 = ∑ j = 1 q ∥ f ( j ) ∥ s 2 and ∥ F ∥ s 2 = ∑ j , k = 1 q ∥ F ( j , k ) ∥ s 2 \lVert F\rVert_s^2 = \sum_{j,k=1}^q \lVert F^{(j,k)}\rVert_s^2 ∥ F ∥ s 2 = ∑ j , k = 1 q ∥ F ( j , k ) ∥ s 2 . Let us define the vector/matrix-valued error functions
u n = L n − 1 ( μ ∗ − μ n ) , W n = L n − 1 [ Σ ∗ + ( μ ∗ − μ n ) ( μ ∗ − μ n ) T ] L n − T − I q , u_n = L_n^{-1}(\mu_*-\mu_n), \qquad W_n = L_n^{-1}\left[\Sigma_*+(\mu_*-\mu_n)(\mu_*-\mu_n)^\mathsf{T}\right]L_n^{-\mathsf{T}}-I_q, u n = L n − 1 ( μ ∗ − μ n ) , W n = L n − 1 [ Σ ∗ + ( μ ∗ − μ n ) ( μ ∗ − μ n ) T ] L n − T − I q ,
and let the projection matrix be Q o = [ 1 0 0 0 ] Q^{o} = \begin{bmatrix} 1 & 0 \\ 0 & 0 \end{bmatrix} Q o = [ 1 0 0 0 ] if o = { 1 } o = \{1\} o = { 1 } , and Q o = I 2 Q^{o} = I_{2} Q o = I 2 if o = { 1 , 2 } o = \{1,2\} o = { 1 , 2 } . Next, we define the projection-averaged versions of u n u_n u n and W n W_n W n :
u ‾ n ( x ) = ∑ o ∈ O p ∗ ( o ∣ x ) [ Q o u n ( x ) ] , W ‾ n ( x ) = ∑ o ∈ O p ∗ ( o ∣ x ) [ Q o W n ( x ) Q o ] \overline{u}_n(x) = \sum_{o \in\mathcal{O}} p^*(o \mid x) \left[Q^{o}u_n(x)\right], \qquad\overline{W}_n(x) = \sum_{o \in\mathcal{O}} p^*(o \mid x) \left[Q^{o}W_n(x)Q^{o}\right] u n ( x ) = o ∈ O ∑ p ∗ ( o ∣ x ) [ Q o u n ( x ) ] , W n ( x ) = o ∈ O ∑ p ∗ ( o ∣ x ) [ Q o W n ( x ) Q o ]
where p ∗ ( o ∣ x ) = P ∗ ( O = o ∣ X = x ) p^*(o \mid x) = P^*(O = o \mid X = x) p ∗ ( o ∣ x ) = P ∗ ( O = o ∣ X = x ) . The above terms are thus averaging the error functions over the possible missingness patterns. We then have the main result.
Theorem 4. Under Assumptions 1, 7, 10, C.1, the updates (16) and (17) satisfy
∥ u ‾ n ∥ s → 0 , ∥ W ‾ n ∥ s → 0 P ∗ -a.s. \lVert\overline{u}_n\rVert_s \to0,\qquad\lVert\overline{W}_n\rVert_s \to0 \quad P^*\text{-a.s.} ∥ u n ∥ s → 0 , ∥ W n ∥ s → 0 P ∗ -a.s. Furthermore, under Assumptions 8 and 9, ∥ ⋅ ∥ s \lVert\cdot\rVert_s ∥ ⋅ ∥ s is a norm on the respective spaces L 1 ( P X ∗ ; R q ) L^1(P_X^*;\mathbb{R}^q) L 1 ( P X ∗ ; R q ) and L 1 ( P X ∗ ; R q × q ) L^1(P_X^*;\mathbb{R}^{q\times q}) L 1 ( P X ∗ ; R q × q ) , and ∥ u ‾ n ∥ s = ∥ W ‾ n ∥ s = 0 \lVert\overline{u}_n\rVert_s = \lVert\overline{W}_n\rVert_s = 0 ∥ u n ∥ s = ∥ W n ∥ s = 0 if and only if μ n ( x ) = μ ∗ ( x ) \mu_n(x) = \mu_*(x) μ n ( x ) = μ ∗ ( x ) and Σ n ( x ) = Σ ∗ ( x ) \Sigma_n(x) = \Sigma_*(x) Σ n ( x ) = Σ ∗ ( x ) for P X ∗ P_X^* P X ∗ -almost all x x x .
In the fully observed case with P ∗ ( O = { 1 , 2 } ∣ x ) = 1 P^*(O = \{1,2\} \mid x) = 1 P ∗ ( O = { 1 , 2 } ∣ x ) = 1 for P X ∗ P_X^* P X ∗ -almost all x x x , it is clear that u ‾ n = u n \overline{u}_n = u_n u n = u n and W ‾ n = W n \overline{W}_n = W_n W n = W n as expected, which is the natural multivariate version of Theorem 3. Otherwise, we note that the measure used for consistency depends on the missingness pattern, where a positive probability of observing O = { 1 , 2 } O = \{1,2\} O = { 1 , 2 } for each x x x ensures that the covariance matrix is identifiable. The extension to larger q q q and general ordering follows naturally, which we outline in Appendix D.4.
Illustrations
In this section, we illustrate the performance of our proposed recursive moment (RM) updates in a range of single and multi-output examples. All recursive updates are implemented in JAX in Python, and all timings are carried out on an Apple M2 Pro CPU. Additional speed-up for predictive resampling may be obtained with parallelization if desired, but it is sufficiently fast already. Throughout, we choose P X P_X P X to be uniform on X \mathcal{X} X as the covariate space is compact in our examples.
Simulations
All simulations are repeated for 20 seeds, and run-times and the following evaluation metrics are reported. We evaluate test data prediction and the mean squared error for point estimates of μ ∗ \mu_* μ ∗ and σ ∗ 2 \sigma_*^2 σ ∗ 2 . To evaluate the uncertainty quantification, we follow Wade and Inácio [2025] and report the empirical coverage defined as Cov μ = ∫ 1 ( μ ∗ ( x ) ∈ C μ ( x ) ) d P X ∗ \operatorname{Cov}_{\mu} = \int1(\mu_*(x) \in C_{\mu}(x)) \, dP^*_X Cov μ = ∫ 1 ( μ ∗ ( x ) ∈ C μ ( x )) d P X ∗ , where C μ ( x ) C_{\mu}(x) C μ ( x ) is the point-wise 95% credible interval for μ ( x ) \mu(x) μ ( x ) , and μ ∗ ( x ) \mu_*(x) μ ∗ ( x ) is the true mean function, where we note that we do not necessarily expect asymptotic calibration. Average interval lengths are defined as Len μ = ∫ ∣ C μ ( x ) ∣ d P X ∗ \operatorname{Len}_{\mu} = \int|C_{\mu}(x)| \, dP^*_X Len μ = ∫ ∣ C μ ( x ) ∣ d P X ∗ , and coverage/length can be evaluated on a grid or by averaging over the held-out test set. We report both quantities for σ 2 ( x ) \sigma^2(x) σ 2 ( x ) as well.
Heavy-tailed First, we cover the Huber-contaminated setting of Example 3. For comparison, we fit the normalized-weights dependent DP (DDP) mixture of Antoniano-Villalobos et al. [2014] with their suggested hyperparameters. The MCMC sampler is based on the Matlab implementation of Wade and Inácio [2025], with a burn-in of 3000 iterations followed by 1000 retained draws with thinning every second iteration. We also compare to the standard GP with the RBF kernel with length-scale optimized through the marginal likelihood, which is implemented in sklearn in Python. Note that coverage for σ 2 ( ⋅ ) \sigma^2(\cdot) σ 2 ( ⋅ ) is not reported for the GP since the noise variance is a fixed constant. Coverage/interval length are estimated on an x x x -grid of size 50, and test log-likelihood is evaluated on a held-out set of size 2000.
Table 1 shows that RM (SU) has the smallest MSE for both μ \mu μ and σ 2 \sigma^2 σ 2 in both settings, with the best or very competitive test prediction. Only the DDP mixture outperforms in prediction, which is unsurprising as the DGP is a Gaussian mixture. The run-times for the RM are also orders of magnitude smaller than the DDP mixture and copula approaches, and even faster than the GP for n = 1000 n = 1000 n = 1000 . RM attains good coverage with reasonable interval lengths comparable to the DDP mixture for μ \mu μ . For σ 2 \sigma^2 σ 2 , RM (SU) produces intervals roughly half as wide as the DDP mixture, with comparable coverage at n = 200 n = 200 n = 200 but worse coverage at n = 1000 n = 1000 n = 1000 . Finally, we notice that the gap between RM (Gauss) and RM (SU) shrinks as n n n increases, and RM (Gauss) is still consistent under non-Gaussian noise.
Table 1. Heavy-tailed contamination simulation. Boldface marks the smallest MSE and run-time and the largest held-out log score for each sample size; standard errors in parentheses.
Table 2. Multivariate simulation. Boldface marks the smallest MSE and run-time and the largest held-out log score for each sample size; standard errors in parentheses.
Multivariate x x x Next, we carry out a multivariate covariate example which extends Example 3 of Wade and Inácio [2025]. We generate n = 600 n = 600 n = 600 observations with X ( j ) ∼ i i d Unif ( − 2 , 2 ) X^{(j)} \stackrel{\mathrm{iid}}{\sim} \operatorname{Unif}(-2, 2) X ( j ) ∼ iid Unif ( − 2 , 2 ) , j = 1 , … , 5 j = 1, \ldots, 5 j = 1 , … , 5 , and p ∗ ( y ∣ x ) = N ( y ; m m v ( x ) , s m v 2 ( x ) ) p^*(y \mid x) = \mathcal{N}(y; m_{\mathrm{mv}}(x), s_{\mathrm{mv}}^{2}(x)) p ∗ ( y ∣ x ) = N ( y ; m mv ( x ) , s mv 2 ( x )) where
t ( x ) = sin ( π x ( 1 ) x ( 2 ) 2 ) , m m v ( x ) = { 1 , t ( x ) ≤ 0 , − 1 , t ( x ) > 0 , s m v ( x ) = { 0.1 , t ( x ) ≤ 0 , 1 , t ( x ) > 0. t(x) = \sin\left(\frac{\pi x^{(1)}x^{(2)}}{2}\right), \qquad m_{\mathrm{mv}}(x) =
\begin{cases}
1, & t(x) \le0,\\
-1, & t(x) > 0,
\end{cases}
\qquad s_{\mathrm{mv}}(x) =
\begin{cases}
0.1, & t(x) \le0,\\
1, & t(x) > 0.
\end{cases} t ( x ) = sin ( 2 π x ( 1 ) x ( 2 ) ) , m mv ( x ) = { 1 , − 1 , t ( x ) ≤ 0 , t ( x ) > 0 , s mv ( x ) = { 0.1 , 1 , t ( x ) ≤ 0 , t ( x ) > 0.
Note that X ( 3 ) X^{(3)} X ( 3 ) , X ( 4 ) X^{(4)} X ( 4 ) , X ( 5 ) X^{(5)} X ( 5 ) are inactive covariates. Here we use similar settings as in Example 3 of Wade and Inácio [2025] for the DDP mixture, with the same normalized weights hyperparameters for each of the 5 covariate dimensions. Coverage and interval length are now estimated on the held-out test set of size 2000 instead of a grid. We do not report the copula results as the memory/run-time requirements are too high due to the larger test set required for predictive resampling.
As RM and GP involve automatic hyperparameter tuning, they can identify that only the first two covariates are relevant. This is much more challenging for the DDP mixture, as the marginal likelihood is intractable and each MCMC run takes a significant amount of time; the spurious covariates are likely responsible for the poor performance of the DDP mixture. Again, we see that the RM approaches are very competitive for estimation and prediction, with RM (Gauss) attaining the best MSE for σ 2 \sigma^{2} σ 2 as the noise is Gaussian but RM (SU) still scoring the best for prediction. The gap in run-time to the DDP case is even more noticeable in this setting due to the multivariate covariates, with RM run-times beating even the GP.
Real data analysis
We now analyze a large-scale spatiotemporal weather dataset within the multi-output regression setting based on the curated WEATHER-5K dataset of Han et al. [2026]. We study the annual average temperature (TMP), dew point (DEW) and sea level pressure (SLP) readings from 1033 weather stations based in the contiguous United States from 2014-2023, resulting in n = 10330 n = 10330 n = 10330 observations. The multi-output regression task is thus to regress the q = 3 q = 3 q = 3 weather outcomes given the weather station’s longitude and latitude, and the year of measurement (p = 3 p = 3 p = 3 ). Our model is particularly suitable here, as the correlations between TMP, DEW and SLP vary noticeably over the US geography, e.g. [15 ] . Further details on data preprocessing can be found in Appendix § 14 .§ 2 .
To evaluate our method, we hold out 20% of the dataset stratified by year as test data. Among the remaining 80% training observations (n = 8264 n = 8264 n = 8264 ), we randomly omit 20% of DEW measurements for weather stations with longitude > − 100 ∘ > -100^\circ > − 10 0 ∘ to simulate covariate-dependent missing outcomes. We tune the hyperparameters of our approach using 1 data permutation, fit to 10 data permutations, and use the output ordering {TMP, SLP, DEW} so the missingness is order-compatible. For predictive resampling, we again use N = n + 4000 N = n + 4000 N = n + 4000 and obtain B = 1000 B = 1000 B = 1000 samples.
As baselines, we compare to a full-rank intrinsic coregionalization model (ICM) implemented in GPflow in Python, and the more flexible GP autoregressive regression model (GPAR, Requeima et al. [ [32 ] ) implemented in PyTorch in Python. For ICM, exact inference is infeasible due to the O ( ( q n ) 3 ) O((qn)^3) O (( q n ) 3 ) complexity, so we only consider sparse variational inference with m m m inducing points. For GPAR, exact inference is possible but costs an expensive O ( q n 3 ) O(qn^3) O ( q n 3 ) , and we also consider approximate inference with inducing points. The authors’ implementation does not allow for variational optimization of the inducing points, so we choose a uniform grid of m ′ = 10 × l × l m' = 10 \times l \times l m ′ = 10 × l × l points for time, longitude and latitude within the support.
Table 3 reports the average log predictive density separately on the 2066 held-out full data points and the 1139 held-out DEW measurements (conditioned on TMP and SLP). We also report run-times, where the fit column corresponds to hyperparameter tuning and fitting, and the total time includes additional method-specific prediction time which we outline shortly. Figure 4 plots the posterior mean of the outcome correlation function r ( x ) r(x) r ( x ) between TMP-DEW and SLP-DEW across varying longitude/latitude in the year 2021. Figure 5 similarly plots the posterior mean and variance of the mean function μ ( x ) \mu(x) μ ( x ) of TMP, where the uncertain regions are driven by a larger heteroscedastic variance.
Figure 4. Posterior mean of the correlation function r ( x ) r(x) r ( x ) between TMP-DEW (left) and SLP-DEW (right) in the year 2021 under the recursive moment updates
Figure 5. Posterior mean (left) and variance (right) of the mean function μ ( x ) \mu(x) μ ( x ) for TMP in the year 2021 under the recursive moment updates
Table 3. Weather data results. ICM is run with 5000 optimizer steps. RM (Independent) plugs in the same hyperparameters as RM (Multi-output), so we do not report run-times. Total run-time includes method-specific prediction time on the 50 × 50 50 \times50 50 × 50 longitude-latitude grid for the year 2021.
We highlight that the RM multi-output approach has the best test log likelihood for the held-out dataset, and is very competitive with the best approach (exact GPAR) for the missing DEW measurements. As a sanity check, the multi-output RM noticeably improves performance compared to 3 independent RM fits using the same hyperparameters. RM’s strong performance is likely due to the ability to capture heterogeneous covariance structure between the outcomes, e.g. Figure 4 . The ICM is unable to capture this structure, and while GPAR is more flexible, it does not allow for a full covariate-dependent outcome covariance structure as the order-dependence is an assumption about the DGP, e.g. the first outcome in the ordering is assumed to have homoscedastic noise. This highlights a stark advantage of our approach: RM is fully flexible and consistent under weak assumptions, whereas GP approaches require complex constructions to attain flexibility. We further note that GP performance is very sensitive to approximation, as we see large deterioration in performance for GPAR as m ′ m' m ′ decreases, and a similar but less drastic trend for ICM.
The main strength however of the RM approach is the rapid computation time. RM’s exact fit time is faster than GP approximations with even a small number of inducing points, and is orders of magnitude faster than the exact GP. For RM’s total run-time, we include the time for predictive resampling on a 50 × 50 50 \times50 50 × 50 longitude-latitude grid for the year 2021 as illustrated in Figure 4 and Figure 5 . Estimating μ ( x ) \mu(x) μ ( x ) for GPAR requires Monte Carlo sampling for each x x x (using 200 samples) on the same longitude-latitude grid in 2021, so this is taken into account for GPAR’s total time. We further note that estimating the posterior variance of μ ( x ) \mu(x) μ ( x ) for GPAR would require additional computation due to the intractable autoregressive structure. On the other hand, ICM prediction is quick after fitting due to the tractable structure, but the tradeoff is a lack of flexibility.
Discussion
In this paper, we have introduced a class of recursive moment updates for Bayesian nonparametric regression, which covers a wide array of settings including heteroscedastic, heavy-tailed and multi-output settings. In addition to being flexible with theoretical guarantees, the approach is expedient and does not require MCMC for uncertainty quantification. An interesting direction for future methodological research is exploration of more complex choices of the similarity function s ( x , x ′ ) s(x, x') s ( x , x ′ ) , perhaps incorporating machine learning or dimension-reduction techniques. Another ambitious extension is to allow covariate-dependent skewness by matching the third moment. Another important direction for future research is a strengthening of the theoretical results, including stronger consistency guarantees and convergence rates, although new technical tools will be needed.
Acknowledgments
Edwin Fong receives funding from the Research Grants Council of Hong Kong through the Early Career Scheme (Grant 27304424) and the General Research Fund (Grant 17306925). The authors thank Joshua Rooijakkers for helpful discussions. ChatGPT Astra was used to assist with coding, proofreading, and reviewing the presentation of technical proofs during the preparation of this manuscript. The authors take full responsibility for the content.
Data availability
All datasets used in this paper are publicly available. Code for reproducing the results can be found at https://github.com/edfong/recursive_moments .
Notation
We begin with some notation choices for the Appendix to aid in proof readability. A repeatedly used term is α i ( x , X i ) \alpha_i(x, X_i) α i ( x , X i ) , which we will write as the shorthand
β i ( x ) = α i ( x , X i ) = α i s ( x , X i ) 1 − α i + α i s ( x , X i ) . \beta_i(x) = \alpha_i(x, X_i) = \frac{\alpha_i s(x, X_i)}{1-\alpha_i+\alpha_i s(x, X_i)}. β i ( x ) = α i ( x , X i ) = 1 − α i + α i s ( x , X i ) α i s ( x , X i ) .
As we will be working with functions, we will sometimes omit the dependence on the test point x x x , e.g. we can write the single-output updates as
μ i = μ i − 1 + β i σ i − 1 ρ Z i , σ i 2 = σ i − 1 2 ( 1 + β i ρ 2 [ ( 1 − β i ) Z i 2 − 1 ] ) . \mu_i = \mu_{i-1} + \beta_i \sigma_{i-1}\rho Z_i,\qquad\sigma_i^2 = \sigma_{i-1}^2\left(1+\beta_i\rho^2\left[(1-\beta_i)Z_i^2-1\right]\right). μ i = μ i − 1 + β i σ i − 1 ρ Z i , σ i 2 = σ i − 1 2 ( 1 + β i ρ 2 [ ( 1 − β i ) Z i 2 − 1 ] ) .
In some cases, we will evaluate a function f f f at both x x x and x ′ x' x ′ , in which case we will use the shorthand f ( x ′ ) = f ′ f(x')=f' f ( x ′ ) = f ′ . Finally, whenever α i \alpha_i α i is observed with no arguments, it will be assumed to be the O ( i − m ) O(i^{-m}) O ( i − m ) scalar sequence and not the function sequence.
We use the notation x ≲ y x \lesssim y x ≲ y to denote the existence of a large constant C C C such that x ≤ C y x \le Cy x ≤ C y . In the cases where an explicit constant is helpful, we will use C C C which may take on different values in different parts of the proofs. Similarly, we use R n R_n R n to denote remainders that may take on different values.
Proposition proofs
Proof of Proposition 1
It is clearer to first present the form of the mean and variance update under a general transformation g λ g_\lambda g λ , before substituting the SU case.
Proposition B.1. Let P ( ⋅ ; μ , σ , λ ) P(\cdot;\mu,\sigma,\lambda) P ( ⋅ ; μ , σ , λ ) for μ ∈ R \mu\in\mathbb{R} μ ∈ R and σ > 0 \sigma>0 σ > 0 denote the family induced by Y = μ + σ g λ ( ϵ ) Y=\mu+\sigma g_\lambda(\epsilon) Y = μ + σ g λ ( ϵ ) with ϵ ∼ N ( 0 , 1 ) \epsilon\sim\mathcal{N}(0,1) ϵ ∼ N ( 0 , 1 ) , where g λ : R → R g_\lambda:\mathbb{R}\to\mathbb{R} g λ : R → R is a strictly increasing bijection satisfying E [ g λ ( ϵ ) ] = 0 \mathbb{E}[g_\lambda(\epsilon)]=0 E [ g λ ( ϵ )] = 0 and E [ g λ ( ϵ ) 2 ] = 1 \mathbb{E}[g_\lambda(\epsilon)^2]=1 E [ g λ ( ϵ ) 2 ] = 1 for all λ ∈ Λ \lambda\in\Lambda λ ∈ Λ . Let
P i − 1 ( y ∣ x ) = P ( y ; μ i − 1 ( x ) , σ i − 1 ( x ) , λ ) P_{i-1}(y\mid x)=P\left(y;\mu_{i-1}(x),\sigma_{i-1}(x),\lambda\right) P i − 1 ( y ∣ x ) = P ( y ; μ i − 1 ( x ) , σ i − 1 ( x ) , λ ) for fixed λ ∈ Λ \lambda\in\Lambda λ ∈ Λ . Then the mean and variance function updates corresponding to the
nonparametric copula update take the form
μ i = μ i − 1 + β i σ i − 1 R λ ( 1 ) ( ε i ) , \mu_i = \mu_{i-1} + \beta_i \sigma_{i-1} R_\lambda^{(1)}(\varepsilon_i), μ i = μ i − 1 + β i σ i − 1 R λ ( 1 ) ( ε i ) ,
σ i 2 = σ i − 1 2 [ 1 + β i { R λ ( 2 ) ( ε i ) − 1 − β i R λ ( 1 ) ( ε i ) 2 } ] , \sigma_i^2 = \sigma_{i-1}^2 \left[ 1 + \beta_i \left\{ R_\lambda^{(2)}(\varepsilon_i) - 1 - \beta_i R_\lambda^{(1)}(\varepsilon_i)^2 \right\} \right], σ i 2 = σ i − 1 2 [ 1 + β i { R λ ( 2 ) ( ε i ) − 1 − β i R λ ( 1 ) ( ε i ) 2 } ] ,
where the update terms take the form
R λ ( j ) ( ε i ) = E W ∼ N ( 0 , 1 ) [ g λ ( 1 − ρ 2 W + ρ ε i ) j ] R_\lambda^{(j)}(\varepsilon_i) = \mathbb{E}_{W \sim\mathcal{N}(0,1)} \left[ g_\lambda\left( \sqrt{1-\rho^2}W + \rho\varepsilon_i \right)^j \right] R λ ( j ) ( ε i ) = E W ∼ N ( 0 , 1 ) [ g λ ( 1 − ρ 2 W + ρ ε i ) j ]
and ε i = g λ − 1 ( Z i ) \varepsilon_i = g_\lambda^{-1}(Z_i) ε i = g λ − 1 ( Z i ) with
Z i = Y i − μ i − 1 ( X i ) σ i − 1 ( X i ) . Z_i = \frac{Y_i - \mu_{i-1}(X_i)}{\sigma_{i-1}(X_i)}. Z i = σ i − 1 ( X i ) Y i − μ i − 1 ( X i ) .
Furthermore, ε i ∼ N ( 0 , 1 ) \varepsilon_i \sim\mathcal{N}(0,1) ε i ∼ N ( 0 , 1 ) under predictive resampling.
Proof. The copula update for u i = P i ( y ∣ x ) u_i = P_i(y \mid x) u i = P i ( y ∣ x ) is
u i = ( 1 − β i ) u i − 1 + β i H ρ ( u i − 1 , v i − 1 ) u_i = (1-\beta_i)u_{i-1} + \beta_i H_\rho(u_{i-1},v_{i-1}) u i = ( 1 − β i ) u i − 1 + β i H ρ ( u i − 1 , v i − 1 )
where v i − 1 = P i − 1 ( Y i ∣ X i ) v_{i-1} = P_{i-1}(Y_i \mid X_i) v i − 1 = P i − 1 ( Y i ∣ X i ) and
H ρ ( u , v ) = Φ { Φ − 1 ( u ) − ρ Φ − 1 ( v ) 1 − ρ 2 } . H_\rho(u,v) = \Phi\left\{ \frac{\Phi^{-1}(u) - \rho\Phi^{-1}(v)}{\sqrt{1-\rho^2}} \right\}. H ρ ( u , v ) = Φ { 1 − ρ 2 Φ − 1 ( u ) − ρ Φ − 1 ( v ) } .
First, we note that
u i − 1 = P ( μ i − 1 + σ i − 1 g λ ( ε ) ≤ y ) = Φ { g λ − 1 ( y − μ i − 1 ( x ) σ i − 1 ( x ) ) } . u_{i-1} = P\left(\mu_{i-1} + \sigma_{i-1}g_\lambda(\varepsilon) \le y\right) = \Phi\left\{ g_\lambda^{-1}\left( \frac{y-\mu_{i-1}(x)}{\sigma_{i-1}(x)} \right) \right\}. u i − 1 = P ( μ i − 1 + σ i − 1 g λ ( ε ) ≤ y ) = Φ { g λ − 1 ( σ i − 1 ( x ) y − μ i − 1 ( x ) ) } .
A similar argument gives v i − 1 = Φ ( ε i ) v_{i-1} = \Phi(\varepsilon_i) v i − 1 = Φ ( ε i ) . Therefore, we have
H ρ ( u i − 1 , v i − 1 ) = Φ { g λ − 1 ( y − μ i − 1 σ i − 1 ) − ρ ε i 1 − ρ 2 } . H_\rho(u_{i-1},v_{i-1}) = \Phi\left\{ \frac{g_\lambda^{-1}\left(\frac{y-\mu_{i-1}}{\sigma_{i-1}}\right) - \rho\varepsilon_i}{\sqrt{1-\rho^2}} \right\}. H ρ ( u i − 1 , v i − 1 ) = Φ ⎩ ⎨ ⎧ 1 − ρ 2 g λ − 1 ( σ i − 1 y − μ i − 1 ) − ρ ε i ⎭ ⎬ ⎫ .
Now let W ∼ N ( 0 , 1 ) W \sim\mathcal{N}(0,1) W ∼ N ( 0 , 1 ) and define
Y ~ i = μ i − 1 + σ i − 1 g λ ( 1 − ρ 2 W + ρ ε i ) . \widetilde{Y}_i = \mu_{i-1} + \sigma_{i-1}g_\lambda\left(\sqrt{1-\rho^2}W+\rho\varepsilon_i\right). Y i = μ i − 1 + σ i − 1 g λ ( 1 − ρ 2 W + ρ ε i ) .
We thus have Pr ( Y ~ i ≤ y ) = H ρ ( u i − 1 , v i − 1 ) \Pr(\widetilde{Y}_i \le y) = H_\rho(u_{i-1},v_{i-1}) Pr ( Y i ≤ y ) = H ρ ( u i − 1 , v i − 1 ) . The mean of P i ( ⋅ ∣ x ) P_i(\cdot\mid x) P i ( ⋅ ∣ x ) is then equal to
μ i = ( 1 − β i ) μ i − 1 ( x ) + β i E [ Y ~ i ] = μ i − 1 + β i σ i − 1 R λ ( 1 ) ( ε i ) . \mu_i = (1-\beta_i)\mu_{i-1}(x) + \beta_i \mathbb{E}\left[\widetilde{Y}_i\right] = \mu_{i-1} + \beta_i\sigma_{i-1}R_\lambda^{(1)}(\varepsilon_i). μ i = ( 1 − β i ) μ i − 1 ( x ) + β i E [ Y i ] = μ i − 1 + β i σ i − 1 R λ ( 1 ) ( ε i ) .
For the conditional second moment function m i ( 2 ) m_i^{(2)} m i ( 2 ) , we similarly have
m i ( 2 ) ( x ) = ( 1 − β i ) m i − 1 ( 2 ) + β i E [ Y ~ i 2 ] = m i − 1 ( 2 ) + β i [ σ i − 1 2 { R λ ( 2 ) ( ε i ) − 1 } + 2 μ i − 1 σ i − 1 R λ ( 1 ) ( ε i ) ] . \begin{aligned}
m_i^{(2)}(x) &= (1-\beta_i)m_{i-1}^{(2)}+\beta_i E\left[\widetilde{Y}_i^2\right] \\
&=m_{i-1}^{(2)}+\beta_i\left[\sigma_{i-1}^2\left\{R_\lambda^{(2)}(\varepsilon_i)-1\right\}+2\mu_{i-1}\sigma_{i-1}R_\lambda^{(1)}(\varepsilon_i)\right].
\end{aligned} m i ( 2 ) ( x ) = ( 1 − β i ) m i − 1 ( 2 ) + β i E [ Y i 2 ] = m i − 1 ( 2 ) + β i [ σ i − 1 2 { R λ ( 2 ) ( ε i ) − 1 } + 2 μ i − 1 σ i − 1 R λ ( 1 ) ( ε i ) ] .
The identity σ i 2 = m i ( 2 ) − μ i 2 \sigma_i^2=m_i^{(2)}-\mu_i^2 σ i 2 = m i ( 2 ) − μ i 2 then gives us the update for the variance. The predictive resampling result follows directly from the invertibility of g λ g_\lambda g λ .
We end with a technical point, and show that finite E [ g λ ( ε ) 2 ] E[g_\lambda(\varepsilon)^2] E [ g λ ( ε ) 2 ] is sufficient to give R λ ( j ) ( ε i ) < ∞ R_\lambda^{(j)}(\varepsilon_i)<\infty R λ ( j ) ( ε i ) < ∞ for j ∈ { 1 , 2 } j\in\{1,2\} j ∈ { 1 , 2 } . One can see that for V i ∼ N ( ρ ε i , 1 − ρ 2 ) V_i\sim\mathcal{N}(\rho\varepsilon_i,1-\rho^2) V i ∼ N ( ρ ε i , 1 − ρ 2 ) , we have
R λ ( 2 ) ( ε i ) = E [ g λ ( V i ) 2 ] = ∫ g λ ( v ) 2 N ( v ; 0 , 1 ) N ( v ; ρ ε i , 1 − ρ 2 ) N ( v ; 0 , 1 ) d v . R_\lambda^{(2)}(\varepsilon_i)=E\left[g_\lambda(V_i)^2\right]=\int g_\lambda(v)^2\mathcal{N}(v;0,1)\frac{\mathcal{N}(v;\rho\varepsilon_i,1-\rho^2)}{\mathcal{N}(v;0,1)}\,dv. R λ ( 2 ) ( ε i ) = E [ g λ ( V i ) 2 ] = ∫ g λ ( v ) 2 N ( v ; 0 , 1 ) N ( v ; 0 , 1 ) N ( v ; ρ ε i , 1 − ρ 2 ) d v .
Since 0 < ρ < 1 0<\rho<1 0 < ρ < 1 , the density ratio is bounded (as a function of v v v ) for each ε i \varepsilon_i ε i , so E [ g λ ( ε ) 2 ] < ∞ E[g_\lambda(\varepsilon)^2]<\infty E [ g λ ( ε ) 2 ] < ∞ implies R λ ( 2 ) ( ε i ) < ∞ R_\lambda^{(2)}(\varepsilon_i)<\infty R λ ( 2 ) ( ε i ) < ∞ . Cauchy-Schwarz then gives R λ ( 1 ) ( ε i ) < ∞ R_\lambda^{(1)}(\varepsilon_i)<\infty R λ ( 1 ) ( ε i ) < ∞ .
Proposition 1 in the main paper then follows from directly calculating R δ ( j ) ( ε i ) R_\delta^{(j)}(\varepsilon_i) R δ ( j ) ( ε i ) , which we demonstrate for j = 1 j=1 j = 1 . Here, for δ > 0 \delta>0 δ > 0 and ω = exp ( 1 / δ 2 ) \omega=\exp(1/\delta^2) ω = exp ( 1/ δ 2 ) , we have
g δ ( ε ) = sinh ( ε δ ) 1 2 ( ω 2 − 1 ) . (19) g_\delta(\varepsilon)=\frac{\sinh\left(\frac{\varepsilon}{\delta}\right)}{\sqrt{\frac{1}{2}(\omega^2-1)}}. \tag*{(19)} g δ ( ε ) = 2 1 ( ω 2 − 1 ) sinh ( δ ε ) . (19)
Next, we note that
E [ sinh ( V i δ ) ] = 1 2 { E [ exp ( V i δ ) ] − E [ exp ( − V i δ ) ] } . E\left[\sinh\left(\frac{V_i}{\delta}\right)\right]=\frac{1}{2}\left\{E\left[\exp\left(\frac{V_i}{\delta}\right)\right]-E\left[\exp\left(-\frac{V_i}{\delta}\right)\right]\right\}. E [ sinh ( δ V i ) ] = 2 1 { E [ exp ( δ V i ) ] − E [ exp ( − δ V i ) ] } .
where V i ∼ N ( ρ ε i , 1 − ρ 2 ) V_i\sim\mathcal{N}(\rho\varepsilon_i,1-\rho^2) V i ∼ N ( ρ ε i , 1 − ρ 2 ) as before. The two expectations correspond to Gaussian moment generating functions evaluated at t = 1 / δ t=1/\delta t = 1/ δ and t = − 1 / δ t=-1/\delta t = − 1/ δ respectively, so
E [ sinh ( V i δ ) ] = 1 2 { exp ( ρ ε i δ + 1 − ρ 2 2 δ 2 ) − exp ( − ρ ε i δ + 1 − ρ 2 2 δ 2 ) } = ω ( 1 − ρ 2 ) / 2 sinh ( ρ ε i δ ) , \begin{aligned}
E\left[\sinh\left(\frac{V_i}{\delta}\right)\right]
&=\frac{1}{2}\left\{\exp\left(\frac{\rho\varepsilon_i}{\delta}+\frac{1-\rho^2}{2\delta^2}\right)-\exp\left(-\frac{\rho\varepsilon_i}{\delta}+\frac{1-\rho^2}{2\delta^2}\right)\right\}\\
&=\omega^{(1-\rho^2)/2}\sinh\left(\frac{\rho\varepsilon_i}{\delta}\right),
\end{aligned} E [ sinh ( δ V i ) ] = 2 1 { exp ( δ ρ ε i + 2 δ 2 1 − ρ 2 ) − exp ( − δ ρ ε i + 2 δ 2 1 − ρ 2 ) } = ω ( 1 − ρ 2 ) /2 sinh ( δ ρ ε i ) ,
which gives the desired form for R δ ( 1 ) ( ε i ) R_\delta^{(1)}(\varepsilon_i) R δ ( 1 ) ( ε i ) . A similar argument using the identity sinh 2 ( x ) = { cosh ( 2 x ) − 1 } / 2 \sinh^2(x)=\{\cosh(2x)-1\}/2 sinh 2 ( x ) = { cosh ( 2 x ) − 1 } /2 gives the stated form for R δ ( 2 ) ( ε i ) R_\delta^{(2)}(\varepsilon_i) R δ ( 2 ) ( ε i ) .
For the δ → ∞ \delta\to\infty δ → ∞ limit, let us write D δ = 1 2 ( ω 2 − 1 ) D_\delta=\sqrt{\frac{1}{2}(\omega^2-1)} D δ = 2 1 ( ω 2 − 1 ) . First, we note that
δ D δ = exp ( 2 / δ 2 ) − 1 2 / δ 2 → 1. \delta D_\delta=\sqrt{\frac{\exp(2/\delta^2)-1}{2/\delta^2}}\to1. δ D δ = 2/ δ 2 exp ( 2/ δ 2 ) − 1 → 1.
Next, for Z i ≠ 0 Z_i \ne0 Z i = 0 , we have
ε i = ( δ D δ ) Z i sinh − 1 ( D δ Z i ) D δ Z i → Z i \varepsilon_i=(\delta D_\delta)Z_i\frac{\sinh^{-1}(D_\delta Z_i)}{D_\delta Z_i}\to Z_i ε i = ( δ D δ ) Z i D δ Z i sinh − 1 ( D δ Z i ) → Z i
following the earlier limit and lim t → 0 sinh − 1 ( t ) / t = 1 \lim_{t\to0}\sinh^{-1}(t)/t=1 lim t → 0 sinh − 1 ( t ) / t = 1 since D δ → 0 D_\delta\to0 D δ → 0 . The Z i = 0 Z_i=0 Z i = 0 case is immediate. Finally, we write
R δ ( 1 ) ( ε i ) = ρ ε i ω ( 1 − ρ 2 ) / 2 1 δ D δ sinh ( ρ ε i / δ ) ρ ε i / δ → ρ Z i R_\delta^{(1)}(\varepsilon_i)=\rho\varepsilon_i\omega^{(1-\rho^2)/2}\frac{1}{\delta D_\delta}\frac{\sinh(\rho\varepsilon_i/\delta)}{\rho\varepsilon_i/\delta}\to\rho Z_i R δ ( 1 ) ( ε i ) = ρ ε i ω ( 1 − ρ 2 ) /2 δ D δ 1 ρ ε i / δ sinh ( ρ ε i / δ ) → ρ Z i
where we have used the previous limits and lim t → 0 sinh ( t ) / t = 1 \lim_{t\to0}\sinh(t)/t=1 lim t → 0 sinh ( t ) / t = 1 as ρ ε i / δ → 0 \rho\varepsilon_i/\delta\to0 ρ ε i / δ → 0 .
For R δ ( 2 ) ( ε i ) R_\delta^{(2)}(\varepsilon_i) R δ ( 2 ) ( ε i ) , we can write
R δ ( 2 ) ( ε i ) = ω 2 ( 1 − ρ 2 ) − 1 ω 2 − 1 + ω 2 ( 1 − ρ 2 ) sinh 2 ( ρ ε i / δ ) D δ 2 R_\delta^{(2)}(\varepsilon_i)=\frac{\omega^{2(1-\rho^2)}-1}{\omega^2-1}+\omega^{2(1-\rho^2)}\frac{\sinh^2(\rho\varepsilon_i/\delta)}{D_\delta^2} R δ ( 2 ) ( ε i ) = ω 2 − 1 ω 2 ( 1 − ρ 2 ) − 1 + ω 2 ( 1 − ρ 2 ) D δ 2 sinh 2 ( ρ ε i / δ )
using the identity cosh ( 2 x ) − 1 = 2 sinh 2 ( x ) \cosh(2x)-1=2\sinh^2(x) cosh ( 2 x ) − 1 = 2 sinh 2 ( x ) . The first term converges to 1 − ρ 2 1-\rho^2 1 − ρ 2 as lim t → 1 ( t a − 1 ) / ( t − 1 ) = a \lim_{t\to1}(t^a-1)/(t-1)=a lim t → 1 ( t a − 1 ) / ( t − 1 ) = a . For the second term, we apply a similar technique:
sinh 2 ( ρ ε i / δ ) D δ 2 = ρ 2 ε i 2 1 ( δ D δ ) 2 { sinh ( ρ ε i / δ ) ρ ε i / δ } 2 → ρ 2 Z i 2 . \frac{\sinh^2(\rho\varepsilon_i/\delta)}{D_\delta^2}=\rho^2\varepsilon_i^2\frac{1}{(\delta D_\delta)^2}\left\{\frac{\sinh(\rho\varepsilon_i/\delta)}{\rho\varepsilon_i/\delta}\right\}^2\to\rho^2 Z_i^2. D δ 2 sinh 2 ( ρ ε i / δ ) = ρ 2 ε i 2 ( δ D δ ) 2 1 { ρ ε i / δ sinh ( ρ ε i / δ ) } 2 → ρ 2 Z i 2 .
Combining this gives R δ ( 2 ) ( ε i ) → 1 − ρ 2 + ρ 2 Z i 2 R_\delta^{(2)}(\varepsilon_i)\to1-\rho^2+\rho^2Z_i^2 R δ ( 2 ) ( ε i ) → 1 − ρ 2 + ρ 2 Z i 2 .
Proof of Proposition 2
In this section, we provide a more precise characterization of the behaviour of R δ ( 1 ) ( ε i ) R_\delta^{(1)}(\varepsilon_i) R δ ( 1 ) ( ε i ) and R δ ( 2 ) ( ε i ) R_\delta^{(2)}(\varepsilon_i) R δ ( 2 ) ( ε i ) as ∣ Z i ∣ → ∞ |Z_i|\to\infty ∣ Z i ∣ → ∞ , from which Proposition 2 follows.
Proposition B.2. For fixed ρ ∈ ( 0 , 1 ) \rho\in(0,1) ρ ∈ ( 0 , 1 ) and δ > 0 \delta>0 δ > 0 and taking ∣ Z i ∣ → ∞ |Z_i|\to\infty ∣ Z i ∣ → ∞ , (10) satisfies
lim ∣ Z i ∣ → ∞ ∣ R δ ( 1 ) ( ε i ) ∣ C δ , ρ ∣ Z i ∣ ρ = 1 , lim ∣ Z i ∣ → ∞ R δ ( 2 ) ( ε i ) ω 1 − ρ 2 C δ , ρ 2 ∣ Z i ∣ 2 ρ = 1 , \lim_{|Z_i|\to\infty}\frac{|R_\delta^{(1)}(\varepsilon_i)|}{C_{\delta,\rho}|Z_i|^\rho}=1,\qquad\lim_{|Z_i|\to\infty}\frac{R_\delta^{(2)}(\varepsilon_i)}{\omega^{1-\rho^2}C_{\delta,\rho}^2|Z_i|^{2\rho}}=1, ∣ Z i ∣ → ∞ lim C δ , ρ ∣ Z i ∣ ρ ∣ R δ ( 1 ) ( ε i ) ∣ = 1 , ∣ Z i ∣ → ∞ lim ω 1 − ρ 2 C δ , ρ 2 ∣ Z i ∣ 2 ρ R δ ( 2 ) ( ε i ) = 1 , where C δ , ρ = 2 ρ − 1 ω ( 1 − ρ 2 ) / 2 D δ ρ − 1 C_{\delta,\rho}=2^{\rho-1}\omega^{(1-\rho^2)/2}D_\delta^{\rho-1} C δ , ρ = 2 ρ − 1 ω ( 1 − ρ 2 ) /2 D δ ρ − 1 .
Proof. We again begin with R δ ( 1 ) ( ε i ) R_\delta^{(1)}(\varepsilon_i) R δ ( 1 ) ( ε i ) first, where the only non-constant terms are
sinh { ρ sinh − 1 ( D δ Z i ) } . \sinh\left\{\rho\sinh^{-1}(D_\delta Z_i)\right\}. sinh { ρ sinh − 1 ( D δ Z i ) } .
Since the above term is odd as a function of Z i Z_i Z i , we have
∣ R δ ( 1 ) ( ε i ) ∣ = ω ( 1 − ρ 2 ) / 2 D δ sinh { ρ sinh − 1 ( D δ ∣ Z i ∣ ) } . \left|R_{\delta}^{(1)}(\varepsilon_i)\right|=\frac{\omega^{(1-\rho^2)/2}}{D_\delta}\sinh\left\{\rho\sinh^{-1}\left(D_\delta\lvert Z_i\rvert\right)\right\}. R δ ( 1 ) ( ε i ) = D δ ω ( 1 − ρ 2 ) /2 sinh { ρ sinh − 1 ( D δ ∣ Z i ∣ ) } .
Now, we have
sinh − 1 ( D δ ∣ Z i ∣ ) = log ( 2 D δ ∣ Z i ∣ ) + o ( 1 ) , \sinh^{-1}\left(D_\delta\lvert Z_i\rvert\right)=\log\left(2D_\delta\lvert Z_i\rvert\right)+o(1), sinh − 1 ( D δ ∣ Z i ∣ ) = log ( 2 D δ ∣ Z i ∣ ) + o ( 1 ) ,
sinh { ρ sinh − 1 ( x ) } = 1 2 exp ( ρ sinh − 1 ( x ) ) [ 1 − exp ( − 2 ρ sinh − 1 ( x ) ) ] . \sinh\left\{\rho\sinh^{-1}(x)\right\}=\frac{1}{2}\exp\left(\rho\sinh^{-1}(x)\right)\left[1-\exp\left(-2\rho\sinh^{-1}(x)\right)\right]. sinh { ρ sinh − 1 ( x ) } = 2 1 exp ( ρ sinh − 1 ( x ) ) [ 1 − exp ( − 2 ρ sinh − 1 ( x ) ) ] .
Combining the above gives
sinh { ρ sinh − 1 ( D δ ∣ Z i ∣ ) } = 2 ρ − 1 D δ ρ ∣ Z i ∣ ρ { 1 + o ( 1 ) } . \sinh\left\{\rho\sinh^{-1}\left(D_\delta\lvert Z_i\rvert\right)\right\}=2^{\rho-1}D_\delta^\rho\lvert Z_i\rvert^\rho\{1+o(1)\}. sinh { ρ sinh − 1 ( D δ ∣ Z i ∣ ) } = 2 ρ − 1 D δ ρ ∣ Z i ∣ ρ { 1 + o ( 1 )} .
This then gives the desired form of C δ , ρ C_{\delta,\rho} C δ , ρ . For R δ ( 2 ) ( ε i ) R_{\delta}^{(2)}(\varepsilon_i) R δ ( 2 ) ( ε i ) , a similar argument gives
cosh { 2 ρ sinh − 1 ( D δ ∣ Z i ∣ ) } = 2 2 ρ − 1 D δ 2 ρ ∣ Z i ∣ 2 ρ { 1 + o ( 1 ) } , \cosh\left\{2\rho\sinh^{-1}\left(D_\delta\lvert Z_i\rvert\right)\right\}=2^{2\rho-1}D_\delta^{2\rho}\lvert Z_i\rvert^{2\rho}\{1+o(1)\}, cosh { 2 ρ sinh − 1 ( D δ ∣ Z i ∣ ) } = 2 2 ρ − 1 D δ 2 ρ ∣ Z i ∣ 2 ρ { 1 + o ( 1 )} ,
from which the result for R δ ( 2 ) ( ε i ) R_{\delta}^{(2)}(\varepsilon_i) R δ ( 2 ) ( ε i ) follows. □ \square □
For the ρ → 1 \rho\to1 ρ → 1 limit with fixed Z i Z_i Z i and δ > 0 \delta>0 δ > 0 , continuity gives
lim ρ → 1 sinh { ρ sinh − 1 ( D δ Z i ) } = D δ Z i \lim_{\rho\to1}\sinh\left\{\rho\sinh^{-1}\left(D_\delta Z_i\right)\right\}=D_\delta Z_i ρ → 1 lim sinh { ρ sinh − 1 ( D δ Z i ) } = D δ Z i
since D δ D_\delta D δ and Z i Z_i Z i are constants. Since lim ρ → 1 ω ( 1 − ρ 2 ) / 2 = 1 \lim_{\rho\to1}\omega^{(1-\rho^2)/2}=1 lim ρ → 1 ω ( 1 − ρ 2 ) /2 = 1 , the D δ D_\delta D δ constants cancel and we obtain lim ρ → 1 R δ ( 1 ) ( ε i ) = Z i \lim_{\rho\to1}R_{\delta}^{(1)}(\varepsilon_i)=Z_i lim ρ → 1 R δ ( 1 ) ( ε i ) = Z i . For R δ ( 2 ) ( ε i ) R_{\delta}^{(2)}(\varepsilon_i) R δ ( 2 ) ( ε i ) , continuity gives
lim ρ → 1 R δ ( 2 ) ( ε i ) = cosh { 2 sinh − 1 ( D δ Z i ) } − 1 ω 2 − 1 \lim_{\rho\to1}R_{\delta}^{(2)}(\varepsilon_i)=\frac{\cosh\left\{2\sinh^{-1}\left(D_\delta Z_i\right)\right\}-1}{\omega^2-1} ρ → 1 lim R δ ( 2 ) ( ε i ) = ω 2 − 1 cosh { 2 sinh − 1 ( D δ Z i ) } − 1
and the identity cosh ( 2 x ) − 1 = 2 sinh 2 ( x ) \cosh(2x)-1=2\sinh^2(x) cosh ( 2 x ) − 1 = 2 sinh 2 ( x ) then gives lim ρ → 1 R δ ( 2 ) ( ε i ) = Z i 2 \lim_{\rho\to1}R_{\delta}^{(2)}(\varepsilon_i)=Z_i^2 lim ρ → 1 R δ ( 2 ) ( ε i ) = Z i 2 .
Proof of Proposition 3
This follows from standard properties of the multivariate Gaussian. The random variable (r.v.) Y ∼ N ( μ , Σ ) Y\sim\mathcal{N}(\mu,\Sigma) Y ∼ N ( μ , Σ ) for arbitrary ( μ , Σ ) (\mu,\Sigma) ( μ , Σ ) may be written as
Y = μ + L Z , Z ∼ N ( 0 , I q ) , Y=\mu+LZ,\qquad Z\sim\mathcal{N}(0,I_q), Y = μ + L Z , Z ∼ N ( 0 , I q ) ,
where L L L is the Cholesky factor of Σ \Sigma Σ , and Z = L − 1 ( Y − μ ) Z = L^{-1}(Y-\mu) Z = L − 1 ( Y − μ ) . Hence, for each j j j ,
Y ( j ) = μ ( j ) + ∑ k = 1 j − 1 L ( j , k ) Z ( k ) + L ( j , j ) Z ( j ) . (20) Y^{(j)} = \mu^{(j)} + \sum_{k=1}^{j-1} L^{(j,k)}Z^{(k)} + L^{(j,j)}Z^{(j)}. \tag*{(20)} Y ( j ) = μ ( j ) + k = 1 ∑ j − 1 L ( j , k ) Z ( k ) + L ( j , j ) Z ( j ) . (20)
Moreover, since L ( 1 : j − 1 , 1 : j − 1 ) L^{(1:j-1,1:j-1)} L ( 1 : j − 1 , 1 : j − 1 ) is lower triangular with positive diagonal, Z ( 1 : j − 1 ) Z^{(1:j-1)} Z ( 1 : j − 1 ) and Y ( 1 : j − 1 ) Y^{(1:j-1)} Y ( 1 : j − 1 ) are deterministic one-to-one functions of each other. Since the components of Z Z Z are i.i.d., it follows that
( Y ( j ) ∣ Y ( 1 : j − 1 ) ) ∼ N ( μ ( j ) + ∑ k = 1 j − 1 L ( j , k ) Z ( k ) , ( L ( j , j ) ) 2 ) . \left(Y^{(j)} \mid Y^{(1:j-1)}\right) \sim\mathcal{N}\left(\mu^{(j)}+\sum_{k=1}^{j-1}L^{(j,k)}Z^{(k)},\left(L^{(j,j)}\right)^2\right). ( Y ( j ) ∣ Y ( 1 : j − 1 ) ) ∼ N ( μ ( j ) + k = 1 ∑ j − 1 L ( j , k ) Z ( k ) , ( L ( j , j ) ) 2 ) .
Therefore, the conditional density p i − 1 ( y ( j ) ∣ y ( 1 : j − 1 ) , x ) p_{i-1}(y^{(j)}\mid y^{(1:j-1)},x) p i − 1 ( y ( j ) ∣ y ( 1 : j − 1 ) , x ) has mean function
μ ~ i − 1 ( j ) ( x ; y ( 1 : j − 1 ) ) = μ i − 1 ( j ) ( x ) + ∑ k = 1 j − 1 L i − 1 ( j , k ) ( x ) z ( k ) , \widetilde{\mu}_{i-1}^{(j)}(x;y^{(1:j-1)})=\mu_{i-1}^{(j)}(x)+\sum_{k=1}^{j-1}L_{i-1}^{(j,k)}(x)z^{(k)}, μ i − 1 ( j ) ( x ; y ( 1 : j − 1 ) ) = μ i − 1 ( j ) ( x ) + k = 1 ∑ j − 1 L i − 1 ( j , k ) ( x ) z ( k ) ,
and standard deviation function
σ ~ i − 1 ( j ) ( x ; y ( 1 : j − 1 ) ) = L i − 1 ( j , j ) ( x ) . \widetilde{\sigma}_{i-1}^{(j)}(x;y^{(1:j-1)})=L_{i-1}^{(j,j)}(x). σ i − 1 ( j ) ( x ; y ( 1 : j − 1 ) ) = L i − 1 ( j , j ) ( x ) .
Evaluating at x = X i x=X_i x = X i and y = Y i y=Y_i y = Y i and rearranging (20) then gives
Z i ( j ) = Y i ( j ) − μ ~ i − 1 ( j ) ( X i ; Y i ( 1 : j − 1 ) ) σ ~ i − 1 ( j ) ( X i ; Y i ( 1 : j − 1 ) ) . Z_i^{(j)}=\frac{Y_i^{(j)}-\widetilde{\mu}_{i-1}^{(j)}(X_i;Y_i^{(1:j-1)})}{\widetilde{\sigma}_{i-1}^{(j)}(X_i;Y_i^{(1:j-1)})}. Z i ( j ) = σ i − 1 ( j ) ( X i ; Y i ( 1 : j − 1 ) ) Y i ( j ) − μ i − 1 ( j ) ( X i ; Y i ( 1 : j − 1 ) ) .
Proof of Proposition 4
For completeness, we begin by deriving the moment update under the fully observed case. The conditional distribution of the multivariate Gaussian from Proposition 3 gives
P i − 1 ( y ( j ) ∣ y ( 1 : j − 1 ) , x ) = Φ ( z ( j ) ) P_{i-1}(y^{(j)}\mid y^{(1:j-1)},x)=\Phi(z^{(j)}) P i − 1 ( y ( j ) ∣ y ( 1 : j − 1 ) , x ) = Φ ( z ( j ) )
where z ( j ) z^{(j)} z ( j ) is the j j j -th component of the vector z = L i − 1 − 1 ( y − μ i − 1 ) z=L_{i-1}^{-1}(y-\mu_{i-1}) z = L i − 1 − 1 ( y − μ i − 1 ) . Similarly, we have ν i − 1 ( j ) = Φ ( Z i ( j ) ) \nu_{i-1}^{(j)}=\Phi(Z_i^{(j)}) ν i − 1 ( j ) = Φ ( Z i ( j ) ) . Since p i − 1 ( y ∣ x ) = ∣ L i − 1 ∣ − 1 ∏ j = 1 q ϕ ( z ( j ) ) p_{i-1}(y\mid x)=\lvert L_{i-1}\rvert^{-1}\prod_{j=1}^{q}\phi(z^{(j)}) p i − 1 ( y ∣ x ) = ∣ L i − 1 ∣ − 1 ∏ j = 1 q ϕ ( z ( j ) ) , the copula-weighted component in (11) is a PDF of the form
p ~ i ( y ∣ x ) = p i − 1 ( y ∣ x ) ∏ j = 1 q c ρ ( Φ ( z ( j ) ) , Φ ( Z i ( j ) ) ) = ∣ L i − 1 ∣ − 1 ∏ j = 1 q ϕ ( z ( j ) ; ρ Z i ( j ) , 1 − ρ 2 ) . \widetilde{p}_i(y \mid x) = p_{i-1}(y \mid x) \prod_{j=1}^{q} c_\rho\left(\Phi(z^{(j)}), \Phi(Z_i^{(j)})\right)
= \lvert L_{i-1}\rvert^{-1} \prod_{j=1}^{q} \phi\left(z^{(j)};\rho Z_i^{(j)},1-\rho^2\right). p i ( y ∣ x ) = p i − 1 ( y ∣ x ) j = 1 ∏ q c ρ ( Φ ( z ( j ) ) , Φ ( Z i ( j ) ) ) = ∣ L i − 1 ∣ − 1 j = 1 ∏ q ϕ ( z ( j ) ; ρ Z i ( j ) , 1 − ρ 2 ) .
where we have used the standard conditional Gaussian identity
ϕ ( z ) c ρ { Φ ( z ) , Φ ( Z i ) } = ϕ ( z ; ρ Z i , 1 − ρ 2 ) . \phi(z)c_\rho\{\Phi(z),\Phi(Z_i)\}=\phi(z;\rho Z_i,1-\rho^2). ϕ ( z ) c ρ { Φ ( z ) , Φ ( Z i )} = ϕ ( z ; ρ Z i , 1 − ρ 2 ) .
If we define
Y ~ i = μ i − 1 + L i − 1 { ρ Z i + 1 − ρ 2 W } , W ∼ N ( 0 , I q ) , \widetilde{Y}_i=\mu_{i-1}+L_{i-1}\left\{\rho Z_i+\sqrt{1-\rho^2}W\right\}, \qquad W\sim\mathcal{N}(0,I_q), Y i = μ i − 1 + L i − 1 { ρ Z i + 1 − ρ 2 W } , W ∼ N ( 0 , I q ) ,
then it is clear that Y ~ i ∼ p ~ i ( ⋅ ∣ x ) \widetilde{Y}_i\sim\widetilde{p}_i(\cdot\mid x) Y i ∼ p i ( ⋅ ∣ x ) through a change of variables argument between y y y and z z z . Following the same moment argument used in Proposition 1, we have
E [ Y ~ i ] = μ i − 1 + d i , E [ Y ~ i Y ~ i T ] = ( 1 − ρ 2 ) Σ i − 1 + [ μ i − 1 + d i ] [ μ i − 1 + d i ] T , E[\widetilde{Y}_i]=\mu_{i-1}+d_i,\qquad E[\widetilde{Y}_i\widetilde{Y}_i^{\mathsf{T}}]=(1-\rho^2)\Sigma_{i-1}+[\mu_{i-1}+d_i][\mu_{i-1}+d_i]^{\mathsf{T}}, E [ Y i ] = μ i − 1 + d i , E [ Y i Y i T ] = ( 1 − ρ 2 ) Σ i − 1 + [ μ i − 1 + d i ] [ μ i − 1 + d i ] T ,
where we write d i ( x ) = ρ L i − 1 ( x ) Z i d_i(x)=\rho L_{i-1}(x)Z_i d i ( x ) = ρ L i − 1 ( x ) Z i . Computing the moments gives
μ i = ( 1 − β i ) μ i − 1 + β i E [ Y ~ i ] = μ i − 1 + β i d i (21) \mu_i=(1-\beta_i)\mu_{i-1}+\beta_i E[\widetilde{Y}_i]=\mu_{i-1}+\beta_i d_i \tag*{(21)} μ i = ( 1 − β i ) μ i − 1 + β i E [ Y i ] = μ i − 1 + β i d i (21)
and for the second moment matrix M i ( 2 ) M_i^{(2)} M i ( 2 ) we have
M i ( 2 ) = ( 1 − β i ) M i − 1 ( 2 ) + β i E [ Y ~ i Y ~ i T ] = M i − 1 ( 2 ) + β i [ μ i − 1 d i T + d i μ i − 1 T + d i d i T − ρ 2 Σ i − 1 ] . (22) \begin{aligned}
M_i^{(2)}&=(1-\beta_i)M_{i-1}^{(2)}+\beta_i E[\widetilde{Y}_i\widetilde{Y}_i^{\mathsf{T}}]\\
&=M_{i-1}^{(2)}+\beta_i\left[\mu_{i-1}d_i^{\mathsf{T}}+d_i\mu_{i-1}^{\mathsf{T}}+d_id_i^{\mathsf{T}}-\rho^2\Sigma_{i-1}\right]. \tag*{(22)}
\end{aligned} M i ( 2 ) = ( 1 − β i ) M i − 1 ( 2 ) + β i E [ Y i Y i T ] = M i − 1 ( 2 ) + β i [ μ i − 1 d i T + d i μ i − 1 T + d i d i T − ρ 2 Σ i − 1 ] . (22)
Now, we take the expectation over ( Y i ( M i ) ∣ Y i ( O i ) , X i ) (Y_i^{(M_i)}\mid Y_i^{(O_i)},X_i) ( Y i ( M i ) ∣ Y i ( O i ) , X i ) . For the realized O i O_i O i , we treat S i S_i S i as a fixed selection matrix and apply Gaussian conditioning under the predictive law p i − 1 ( ⋅ ∣ X i ) p_{i-1}(\cdot\mid X_i) p i − 1 ( ⋅ ∣ X i ) . By MAR, the resulting conditional law given the observed outcomes also applies when conditioning on the missingness pattern, that is
p i − 1 ( y ( M i ) ∣ Y i ( O i ) , O i , X i ) = p i − 1 ( y ( M i ) ∣ Y i ( O i ) , X i ) . p_{i-1}\left(y^{(M_i)}\mid Y_i^{(O_i)},O_i,X_i\right)=p_{i-1}\left(y^{(M_i)}\mid Y_i^{(O_i)},X_i\right). p i − 1 ( y ( M i ) ∣ Y i ( O i ) , O i , X i ) = p i − 1 ( y ( M i ) ∣ Y i ( O i ) , X i ) .
The key is that under the Gaussian assumption for p i − 1 ( ⋅ ∣ x ) p_{i-1}(\cdot\mid x) p i − 1 ( ⋅ ∣ x ) , the conditional density p i − 1 ( Y i ( M i ) ∣ Y i ( O i ) , X i ) p_{i-1}(Y_i^{(M_i)}\mid Y_i^{(O_i)},X_i) p i − 1 ( Y i ( M i ) ∣ Y i ( O i ) , X i ) is Gaussian.
Instead of working with y i ( M i ) y_i^{(M_i)} y i ( M i ) however, we directly consider the conditional moments of Z i Z_i Z i . Under the predictive model, we have Z i ∣ X i ∼ N ( 0 , I q ) Z_i\mid X_i\sim\mathcal{N}(0,I_q) Z i ∣ X i ∼ N ( 0 , I q ) . Next, given X i X_i X i , we note that observing Y i ( O i ) Y_i^{(O_i)} Y i ( O i ) is equivalent to observing
A i Z i = S i [ Y i − μ i − 1 ( X i ) ] . A_i Z_i = S_i[Y_i-\mu_{i-1}(X_i)]. A i Z i = S i [ Y i − μ i − 1 ( X i )] .
A convenient approach is to notice that
[ Z i A i Z i ] ∣ X i ∼ N ( [ 0 q 0 ∣ O i ∣ ] , [ I q A i ⊤ A i A i A i ⊤ ] ) , \begin{bmatrix} Z_i \\ A_iZ_i \end{bmatrix}\mid X_i \sim\mathcal{N}\left(\begin{bmatrix} 0_q \\ 0_{\lvert O_i\rvert} \end{bmatrix},\begin{bmatrix} I_q & A_i^\top\\ A_i & A_iA_i^\top\end{bmatrix}\right), [ Z i A i Z i ] ∣ X i ∼ N ( [ 0 q 0 ∣ O i ∣ ] , [ I q A i A i ⊤ A i A i ⊤ ] ) ,
then Gaussian conditioning gives
Z i ∣ A i Z i , X i ∼ N ( Z ‾ i , I q − Q i ) . Z_i\mid A_iZ_i,X_i\sim\mathcal{N}\left(\overline{Z}_i,I_q-Q_i\right). Z i ∣ A i Z i , X i ∼ N ( Z i , I q − Q i ) .
We now take the conditional expectation of (21) and (22) over Z i ∣ Y i ( O i ) , X i Z_i\mid Y_i^{(O_i)},X_i Z i ∣ Y i ( O i ) , X i . Let
d ‾ i = E [ d i ∣ Y i ( O i ) , X i ] = ρ L i − 1 Z ‾ i . \overline{d}_i=E[d_i\mid Y_i^{(O_i)},X_i]=\rho L_{i-1}\overline{Z}_i. d i = E [ d i ∣ Y i ( O i ) , X i ] = ρ L i − 1 Z i .
We then have the mean update as
μ i = μ i − 1 + β i d ‾ i \mu_i=\mu_{i-1}+\beta_i\overline{d}_i μ i = μ i − 1 + β i d i
and the second moment as
M i ( 2 ) = M i − 1 ( 2 ) + β i [ μ i − 1 d ‾ i ⊤ + d ‾ i μ i − 1 ⊤ + ρ 2 L i − 1 ( I − Q i ) L i − 1 ⊤ + d ‾ i d ‾ i ⊤ − ρ 2 Σ i − 1 ] = M i − 1 ( 2 ) + β i [ μ i − 1 d ‾ i ⊤ + d ‾ i μ i − 1 ⊤ − ρ 2 L i − 1 Q i L i − 1 ⊤ + d ‾ i d ‾ i ⊤ ] . \begin{aligned}
M_i^{(2)}=M_{i-1}^{(2)}+\beta_i\left[\mu_{i-1}\overline{d}_i^\top+\overline{d}_i\mu_{i-1}^\top+\rho^2L_{i-1}(I-Q_i)L_{i-1}^\top+\overline{d}_i\overline{d}_i^\top-\rho^2\Sigma_{i-1}\right] \\
&=M_{i-1}^{(2)}+\beta_i\left[\mu_{i-1}\overline{d}_i^\top+\overline{d}_i\mu_{i-1}^\top-\rho^2L_{i-1}Q_iL_{i-1}^\top+\overline{d}_i\overline{d}_i^\top\right].
\end{aligned} M i ( 2 ) = M i − 1 ( 2 ) + β i [ μ i − 1 d i ⊤ + d i μ i − 1 ⊤ + ρ 2 L i − 1 ( I − Q i ) L i − 1 ⊤ + d i d i ⊤ − ρ 2 Σ i − 1 ] = M i − 1 ( 2 ) + β i [ μ i − 1 d i ⊤ + d i μ i − 1 ⊤ − ρ 2 L i − 1 Q i L i − 1 ⊤ + d i d i ⊤ ] .
Computing Σ i = M i ( 2 ) − μ i μ i ⊤ \Sigma_i=M_i^{(2)}-\mu_i\mu_i^\top Σ i = M i ( 2 ) − μ i μ i ⊤ then gives the form in the main paper.
Proof of Proposition 5
First, we note that under Y ∼ p i − 1 ( ⋅ ∣ x = X i ) Y\sim p_{i-1}(\cdot\mid x=X_i) Y ∼ p i − 1 ( ⋅ ∣ x = X i ) , we can write
Y = μ i − 1 ( X i ) + L i − 1 ( X i ) U i − 1 , U i − 1 ∼ N ( 0 , I q ) . Y=\mu_{i-1}(X_i)+L_{i-1}(X_i)U_{i-1},\quad U_{i-1}\sim\mathcal{N}(0,I_q). Y = μ i − 1 ( X i ) + L i − 1 ( X i ) U i − 1 , U i − 1 ∼ N ( 0 , I q ) .
Next, we note that the observed component of Y Y Y can be written as
Y ( O i ) = S i μ i − 1 ( X i ) + A i U i − 1 , Y^{(O_i)}=S_i\mu_{i-1}(X_i)+A_iU_{i-1}, Y ( O i ) = S i μ i − 1 ( X i ) + A i U i − 1 ,
so conditioning on Y ( O i ) = y ( O i ) Y^{(O_i)}=y^{(O_i)} Y ( O i ) = y ( O i ) is equivalent to conditioning on A i U i − 1 = a A_iU_{i-1}=a A i U i − 1 = a where a : = y ( O i ) − S i μ i − 1 ( X i ) a:=y^{(O_i)}-S_i\mu_{i-1}(X_i) a := y ( O i ) − S i μ i − 1 ( X i ) . Gaussian conditioning with ( U i − 1 , A i U i − 1 ) (U_{i-1},A_iU_{i-1}) ( U i − 1 , A i U i − 1 ) gives
U i − 1 ∣ A i U i − 1 = a ∼ N ( A i T ( A i A i T ) − 1 a , I − Q i ) . U_{i-1}\mid A_iU_{i-1}=a\sim\mathcal{N}\left(A_i^{\mathsf{T}}(A_iA_i^{\mathsf{T}})^{-1}a,I-Q_i\right). U i − 1 ∣ A i U i − 1 = a ∼ N ( A i T ( A i A i T ) − 1 a , I − Q i ) .
Since Y ( M i ) Y^{(M_i)} Y ( M i ) is an affine function of U i − 1 U_{i-1} U i − 1 , the above distribution characterizes p i − 1 ( y ( M i ) ∣ y ( O i ) , x = X i ) p_{i-1}(y^{(M_i)}\mid y^{(O_i)},x=X_i) p i − 1 ( y ( M i ) ∣ y ( O i ) , x = X i ) . Next, Y ∼ p i ( ⋅ ∣ x = X i ) Y\sim p_i(\cdot\mid x=X_i) Y ∼ p i ( ⋅ ∣ x = X i ) can be expressed as a function of the previous time step’s mean/covariance:
Y = μ i − 1 ( X i ) + L i − 1 ( X i ) U i , U i ∼ N ( β i ( X i ) ρ Z ˉ i , C i ) . Y=\mu_{i-1}(X_i)+L_{i-1}(X_i)U_i,\qquad U_i\sim\mathcal{N}\left(\beta_i(X_i)\rho\bar{Z}_i,C_i\right). Y = μ i − 1 ( X i ) + L i − 1 ( X i ) U i , U i ∼ N ( β i ( X i ) ρ Z ˉ i , C i ) .
where C i = I + β i ( X i ) ρ 2 [ { 1 − β i ( X i ) } Z ˉ i Z ˉ i T − Q i ] C_i=I+\beta_i(X_i)\rho^2\left[\{1-\beta_i(X_i)\}\bar{Z}_i\bar{Z}_i^{\mathsf{T}}-Q_i\right] C i = I + β i ( X i ) ρ 2 [ { 1 − β i ( X i )} Z ˉ i Z ˉ i T − Q i ] . We will now use key properties of the projection matrix, that is Q i T = Q i Q_i^{\mathsf{T}}=Q_i Q i T = Q i , Q i Q i = Q i Q_iQ_i=Q_i Q i Q i = Q i , and A i Q i = A i A_iQ_i=A_i A i Q i = A i . Next, we can decompose U i U_i U i into two components:
U i = Q i U i + ϵ i , ϵ i = ( I − Q i ) U i U_i=Q_iU_i+\epsilon_i,\qquad\epsilon_i=(I-Q_i)U_i U i = Q i U i + ϵ i , ϵ i = ( I − Q i ) U i
where we will now show ϵ i ∼ N ( 0 , I − Q i ) \epsilon_i\sim\mathcal{N}(0,I-Q_i) ϵ i ∼ N ( 0 , I − Q i ) and ϵ i ⊥ Q i U i \epsilon_i\perp Q_iU_i ϵ i ⊥ Q i U i . To verify these properties, it is clear that ( Q i U i , ϵ i ) (Q_iU_i,\epsilon_i) ( Q i U i , ϵ i ) are jointly Gaussian, and
E [ ϵ i ] = ( I − Q i ) β i ( X i ) ρ Z ˉ i = 0 , E [ ϵ i ϵ i T ] = ( I − Q i ) C i ( I − Q i ) = ( I − Q i ) ( I − Q i ) = I − Q i , \begin{aligned}
E[\epsilon_i]&=(I-Q_i)\beta_i(X_i)\rho\bar{Z}_i=0,\\
E[\epsilon_i\epsilon_i^{\mathsf{T}}]&=(I-Q_i)C_i(I-Q_i)=(I-Q_i)(I-Q_i)=I-Q_i,
\end{aligned} E [ ϵ i ] E [ ϵ i ϵ i T ] = ( I − Q i ) β i ( X i ) ρ Z ˉ i = 0 , = ( I − Q i ) C i ( I − Q i ) = ( I − Q i ) ( I − Q i ) = I − Q i ,
where we have used C i ( I − Q i ) = I − Q i C_i(I-Q_i)=I-Q_i C i ( I − Q i ) = I − Q i , following Q i Z ˉ i = Z ˉ i Q_i\bar{Z}_i=\bar{Z}_i Q i Z ˉ i = Z ˉ i and Q i Q i = Q i Q_iQ_i=Q_i Q i Q i = Q i . The independence property follows from the joint Gaussianity and
Cov ( Q i U i , ϵ i ) = Q i C i ( I − Q i ) = 0 \operatorname{Cov}(Q_iU_i,\epsilon_i)=Q_iC_i(I-Q_i)=0 Cov ( Q i U i , ϵ i ) = Q i C i ( I − Q i ) = 0
again following the above identity for C i ( I − Q i ) C_i(I-Q_i) C i ( I − Q i ) .
Since A i U i = A i Q i U i A_iU_i=A_iQ_iU_i A i U i = A i Q i U i , we immediately have that A i U i ⊥ ϵ i A_iU_i\perp\epsilon_i A i U i ⊥ ϵ i . Furthermore, Q i U i = A i T ( A i A i T ) − 1 A i U i Q_iU_i=A_i^{\mathsf{T}}(A_iA_i^{\mathsf{T}})^{-1}A_iU_i Q i U i = A i T ( A i A i T ) − 1 A i U i , so we have
U i ∣ A i U i = a ∼ N ( A i T ( A i A i T ) − 1 a , I − Q i ) . U_i\mid A_iU_i=a\sim\mathcal{N}\left(A_i^{\mathsf{T}}(A_iA_i^{\mathsf{T}})^{-1}a,I-Q_i\right). U i ∣ A i U i = a ∼ N ( A i T ( A i A i T ) − 1 a , I − Q i ) .
Since ( U i − 1 ∣ A i U i − 1 = a ) = d ( U i ∣ A i U i = a ) (U_{i-1}\mid A_iU_{i-1}=a)\mathrel{\overset{d}{=}}(U_i\mid A_iU_i=a) ( U i − 1 ∣ A i U i − 1 = a ) = d ( U i ∣ A i U i = a ) , and Y ( M i ) Y^{(M_i)} Y ( M i ) is the same affine function of U i U_i U i , we have p i ( y ( M i ) ∣ y ( O i ) , x = X i ) = p i − 1 ( y ( M i ) ∣ y ( O i ) , x = X i ) p_i(y^{(M_i)}\mid y^{(O_i)},x=X_i)=p_{i-1}(y^{(M_i)}\mid y^{(O_i)},x=X_i) p i ( y ( M i ) ∣ y ( O i ) , x = X i ) = p i − 1 ( y ( M i ) ∣ y ( O i ) , x = X i ) for all y ∈ R q y\in\mathbb{R}^q y ∈ R q .
We now strengthen to the case for a generic x ∈ X x\in\mathcal{X} x ∈ X under order-compatible missingness where O i = { 1 , … , q i } O_i=\{1,\ldots,q_i\} O i = { 1 , … , q i } for some q i ≤ q q_i\le q q i ≤ q . Let A i ( x ) = S i L i − 1 ( x ) A_i(x)=S_iL_{i-1}(x) A i ( x ) = S i L i − 1 ( x ) for a generic x ∈ X x\in\mathcal{X} x ∈ X . In this setting, the Cholesky form of L i − 1 ( x ) L_{i-1}(x) L i − 1 ( x ) gives
Q i ( x ) : = A i ( x ) ⊤ ( A i ( x ) A i ( x ) ⊤ ) − 1 A i ( x ) = [ I q i 0 0 0 ] Q_i(x) := A_i(x)^\top\left(A_i(x)A_i(x)^\top\right)^{-1}A_i(x) =
\begin{bmatrix}
I_{q_i} & 0 \\
0 & 0
\end{bmatrix} Q i ( x ) := A i ( x ) ⊤ ( A i ( x ) A i ( x ) ⊤ ) − 1 A i ( x ) = [ I q i 0 0 0 ]
which is now independent of x x x . Importantly, this holds for Q i = Q i ( X i ) Q_i = Q_i(X_i) Q i = Q i ( X i ) .
The first part of the proof regarding p i − 1 ( ⋅ ∣ x ) p_{i-1}(\cdot\mid x) p i − 1 ( ⋅ ∣ x ) remains unchanged, where we now have a : = y ( O i ) − S i μ i − 1 ( x ) a := y^{(O_i)} - S_i\mu_{i-1}(x) a := y ( O i ) − S i μ i − 1 ( x ) and we are conditioning on A i ( x ) U i − 1 = a A_i(x)U_{i-1}=a A i ( x ) U i − 1 = a where A i ( x ) = S i L i − 1 ( x ) A_i(x)=S_iL_{i-1}(x) A i ( x ) = S i L i − 1 ( x ) . Then, we have
U i − 1 ∣ A i ( x ) U i − 1 = a ∼ N ( A i ( x ) ⊤ ( A i ( x ) A i ( x ) ⊤ ) − 1 a , I − Q i ) U_{i-1} \mid A_i(x)U_{i-1}=a \sim\mathcal{N}\left(A_i(x)^\top\left(A_i(x)A_i(x)^\top\right)^{-1}a, I-Q_i\right) U i − 1 ∣ A i ( x ) U i − 1 = a ∼ N ( A i ( x ) ⊤ ( A i ( x ) A i ( x ) ⊤ ) − 1 a , I − Q i )
where we have used Q i ( x ) = Q i Q_i(x)=Q_i Q i ( x ) = Q i . For the second part of the proof with the updated p i ( ⋅ ∣ x ) p_i(\cdot\mid x) p i ( ⋅ ∣ x ) , we have
Y = μ i − 1 ( x ) + L i − 1 ( x ) U i ( x ) , U i ( x ) ∼ N ( β i ( x ) ρ Z ‾ i , C i ( x ) ) Y=\mu_{i-1}(x)+L_{i-1}(x)U_i(x),\qquad U_i(x)\sim\mathcal{N}\left(\beta_i(x)\rho\overline{Z}_i,C_i(x)\right) Y = μ i − 1 ( x ) + L i − 1 ( x ) U i ( x ) , U i ( x ) ∼ N ( β i ( x ) ρ Z i , C i ( x ) )
where C i ( x ) = I + β i ( x ) ρ 2 [ { 1 − β i ( x ) } Z ‾ i Z ‾ i ⊤ − Q i ] C_i(x)=I+\beta_i(x)\rho^2\left[\{1-\beta_i(x)\}\overline{Z}_i\overline{Z}_i^\top-Q_i\right] C i ( x ) = I + β i ( x ) ρ 2 [ { 1 − β i ( x )} Z i Z i ⊤ − Q i ] . The rest of the proof remains identical, as order-compatibility implies
A i ( x ) Q i = A i ( x ) , Q i ( x ) = Q i . A_i(x)Q_i=A_i(x),\qquad Q_i(x)=Q_i. A i ( x ) Q i = A i ( x ) , Q i ( x ) = Q i .
We can then use the same decomposition U i ( x ) = Q i U i ( x ) + ε i ( x ) U_i(x)=Q_iU_i(x)+\varepsilon_i(x) U i ( x ) = Q i U i ( x ) + ε i ( x ) with ε i ( x ) = ( I − Q i ) U i ( x ) \varepsilon_i(x)=(I-Q_i)U_i(x) ε i ( x ) = ( I − Q i ) U i ( x ) , and thus show that ( U i ( x ) ∣ A i ( x ) U i ( x ) ) = d ( U i − 1 ∣ A i ( x ) U i − 1 ) (U_i(x)\mid A_i(x)U_i(x))\mathrel{\mathop{=}\limits^d}(U_{i-1}\mid A_i(x)U_{i-1}) ( U i ( x ) ∣ A i ( x ) U i ( x )) = d ( U i − 1 ∣ A i ( x ) U i − 1 ) which gives us the result for any x ∈ X x\in\mathcal{X} x ∈ X under order-compatibility.
Predictive asymptotics proofs
Preliminary lemmas
We begin with some intermediate lemmas that will be useful in other proofs.
Lemma C.1. Under Assumption 1 , we have sup x , x ′ ∈ X s ( x , x ′ ) ≤ S \sup_{x,x'\in\mathcal{X}}s(x,x')\le S sup x , x ′ ∈ X s ( x , x ′ ) ≤ S .
Proof. Cauchy-Schwarz applied to the definition of s ( x , x ′ ) s(x,x') s ( x , x ′ ) gives
s ( x , x ′ ) ≤ s ( x , x ) s ( x ′ , x ′ ) ≤ S , s(x,x')\le\sqrt{s(x,x)s(x',x')}\le S, s ( x , x ′ ) ≤ s ( x , x ) s ( x ′ , x ′ ) ≤ S , where S S S is finite by Assumption 1.
Lemma C.2. Under Assumptions 1 and 2 , we have
sup x , x ′ ∈ X α i ( x , x ′ ) ≲ i − m . \sup_{x,x'\in\mathcal{X}} \alpha_i(x,x') \lesssim i^{-m}. x , x ′ ∈ X sup α i ( x , x ′ ) ≲ i − m . Proof. It is clear from Assumption 1 and Lemma C.1 that
sup x , x ′ α i ( x , x ′ ) ≤ α i S 1 − α i . \sup_{x,x'} \alpha_i(x,x') \le\frac{\alpha_i S}{1-\alpha_i}. x , x ′ sup α i ( x , x ′ ) ≤ 1 − α i α i S . Since α i = O ( i − m ) \alpha_i = O(i^{-m}) α i = O ( i − m ) with m > 0 m > 0 m > 0 , we have α i → 0 \alpha_i \to0 α i → 0 , and hence ( 1 − α i ) − 1 = O ( 1 ) (1-\alpha_i)^{-1} = O(1) ( 1 − α i ) − 1 = O ( 1 ) . Since α i < 1 \alpha_i < 1 α i < 1 for all i i i , we can find a sufficiently large constant for all i ≥ n + 1 i \ge n+1 i ≥ n + 1 .
We include another lemma that is useful when managing the variance term.
Lemma C.3. Let Z ∼ N ( 0 , 1 ) Z \sim\mathcal{N}(0,1) Z ∼ N ( 0 , 1 ) , 0 ≤ a < 1 0 \le a < 1 0 ≤ a < 1 and 0 < ρ < 1 0 < \rho< 1 0 < ρ < 1 . Then we have
E [ { 1 + a ρ 2 [ ( 1 − a ) Z 2 − 1 ] } 2 ] ≤ 1. E\left[\left\{1+a\rho^2\left[(1-a)Z^2-1\right]\right\}^2\right]\le1. E [ { 1 + a ρ 2 [ ( 1 − a ) Z 2 − 1 ] } 2 ] ≤ 1. Furthermore, for any z ∈ R z \in\mathbb{R} z ∈ R , we also have
1 + a ρ 2 [ ( 1 − a ) z 2 − 1 ] ≥ 1 − ρ 2 > 0. 1+a\rho^2\left[(1-a)z^2-1\right]\ge1-\rho^2>0. 1 + a ρ 2 [ ( 1 − a ) z 2 − 1 ] ≥ 1 − ρ 2 > 0. Proof. Expanding the square gives
E [ { 1 + a ρ 2 [ ( 1 − a ) Z 2 − 1 ] } 2 ] = 1 − 2 a 2 ρ 2 + a 2 ρ 4 ( 2 − 4 a + 3 a 2 ) E\left[\left\{1+a\rho^2\left[(1-a)Z^2-1\right]\right\}^2\right]
=1-2a^2\rho^2+a^2\rho^4(2-4a+3a^2) E [ { 1 + a ρ 2 [ ( 1 − a ) Z 2 − 1 ] } 2 ] = 1 − 2 a 2 ρ 2 + a 2 ρ 4 ( 2 − 4 a + 3 a 2 ) ≤ 1 + a 2 ρ 4 ( 3 a 2 − 4 a ) \le1+a^2\rho^4(3a^2-4a) ≤ 1 + a 2 ρ 4 ( 3 a 2 − 4 a ) where the second line uses ρ < 1 \rho<1 ρ < 1 . Finally, we have ( 3 a 2 − 4 a ) ≤ 0 (3a^2-4a)\le0 ( 3 a 2 − 4 a ) ≤ 0 for 0 ≤ a < 1 0\le a<1 0 ≤ a < 1 . The second part of the lemma follows by taking z = 0 z=0 z = 0 and again using 0 ≤ a < 1 0\le a<1 0 ≤ a < 1 .
Proof of Theorem 1
We follow the setting of Fong and Yiu [2026, Proposition 1], and in a sense extend the setting of Garelli et al. [2026] to the functional case. Let B = L 2 ( X ) B=L^2(\mathcal{X}) B = L 2 ( X ) denote the Hilbert space where X \mathcal{X} X is equipped with the Lebesgue measure. Let Z i ∼ i i d N ( 0 , 1 ) Z_i \mathrel{\overset{\mathrm{iid}}{\sim}} \mathcal{N}(0,1) Z i ∼ iid N ( 0 , 1 ) be i.i.d. Gaussian r.v.s for i ≥ n + 1 i\ge n+1 i ≥ n + 1 (independent of { X i } i ≥ n + 1 \{X_i\}_{i\ge n+1} { X i } i ≥ n + 1 ), and let { F i } i ≥ n \{\mathcal{F}_i\}_{i\ge n} { F i } i ≥ n be our filtration with F i = σ ( X n + 1 , Z n + 1 , … , X i , Z i ) \mathcal{F}_i=\sigma(X_{n+1},Z_{n+1},\ldots,X_i,Z_i) F i = σ ( X n + 1 , Z n + 1 , … , X i , Z i ) with F n = { ∅ , Ω } \mathcal{F}_n=\{\varnothing,\Omega\} F n = { ∅ , Ω } . Let L 2 ( B ) L^2(B) L 2 ( B ) be the space of Bochner measurable functions with E [ ∥ f ∥ B 2 ] < ∞ E[\|f\|_B^2]<\infty E [ ∥ f ∥ B 2 ] < ∞ and norm ∥ f ∥ L 2 ( B ) 2 = E [ ∥ f ∥ B 2 ] \|f\|_{L^2(B)}^2=E[\|f\|_B^2] ∥ f ∥ L 2 ( B ) 2 = E [ ∥ f ∥ B 2 ] . Unless otherwise stated, (a.s.) convergence will be in the B B B norm.
We begin by proving convergence of the mean and variance functions. For any fixed sequence Z 1 : n Z_{1:n} Z 1 : n , we can see that the estimates μ n \mu_n μ n and σ n 2 \sigma_n^2 σ n 2 lie in B B B given the initialization from Assumption 3. As L 2 ( X ) L^2(\mathcal{X}) L 2 ( X ) is separable, following a similar argument to Fong and Yiu [2026, Lemma A8], the random functions μ i \mu_i μ i and σ i 2 \sigma_i^2 σ i 2 are Bochner F i \mathcal{F}_i F i -measurable and { μ i } i ≥ n \{\mu_i\}_{i\ge n} { μ i } i ≥ n is a B B B -valued martingale (given integrability). As in Fong and Yiu [ [11 ] , it suffices to check pointwise expectations through ‘scalarization’ Pisier [ [29 ] , e.g. by checking E [ μ i + 1 ( x ) ∣ F i ] = μ i ( x ) E[\mu_{i+1}(x) \mid\mathcal{F}_i] = \mu_i(x) E [ μ i + 1 ( x ) ∣ F i ] = μ i ( x ) for all x ∈ X x \in\mathcal{X} x ∈ X followed by a Riesz representation argument to verify the martingale condition.
It thus suffices to verify that { μ i } i ≥ n \{\mu_i\}_{i\ge n} { μ i } i ≥ n is bounded in L 2 ( B ) L^2(B) L 2 ( B ) so we can use Pisier [ [29 ] for convergence. Since μ i 2 ( x ) ≤ m i ( 2 ) ( x ) \mu_i^2(x) \le m_i^{(2)}(x) μ i 2 ( x ) ≤ m i ( 2 ) ( x ) , we immediately have
sup i E [ ∥ μ i ∥ B 2 ] ≤ sup i E [ ∫ X m i ( 2 ) ( x ) d x ] = ∫ X m n ( 2 ) ( x ) d x < ∞ . \sup_i E\left[\|\mu_i\|_B^2\right] \le\sup_i E\left[\int_{\mathcal{X}}m_i^{(2)}(x)\,\mathrm{d}x\right] = \int_{\mathcal{X}}m_n^{(2)}(x)\,\mathrm{d}x < \infty. i sup E [ ∥ μ i ∥ B 2 ] ≤ i sup E [ ∫ X m i ( 2 ) ( x ) d x ] = ∫ X m n ( 2 ) ( x ) d x < ∞.
where m n ( 2 ) ∈ L 1 ( X ) m_n^{(2)} \in L^1(\mathcal{X}) m n ( 2 ) ∈ L 1 ( X ) follows from μ n ∈ L 2 ( X ) \mu_n \in L^2(\mathcal{X}) μ n ∈ L 2 ( X ) and σ n 2 ∈ L 1 ( X ) \sigma_n^2 \in L^1(\mathcal{X}) σ n 2 ∈ L 1 ( X ) , where the latter follows from σ n 2 ∈ L 2 ( X ) \sigma_n^2 \in L^2(\mathcal{X}) σ n 2 ∈ L 2 ( X ) and X \mathcal{X} X having finite Lebesgue measure. As a result, { μ i } i ≥ n \{\mu_i\}_{i\ge n} { μ i } i ≥ n is bounded in L 2 ( B ) L^2(B) L 2 ( B ) , and thus converges a.s. to μ ∞ ∈ B \mu_\infty\in B μ ∞ ∈ B .
The variance function is not a martingale, but can be decomposed into a martingale and a summable term. Specifically, we have
σ i 2 = σ i − 1 2 + β i σ i − 1 2 ρ 2 ( Z i 2 − 1 ) ⏟ Δ M i − β i 2 σ i − 1 2 ρ 2 Z i 2 ⏟ Δ R i = σ n 2 + ∑ j = n + 1 i Δ M j − ∑ j = n + 1 i Δ R j . (23) \sigma_i^2 = \sigma_{i-1}^2 + \underbrace{\beta_i\sigma_{i-1}^2\rho^2(Z_i^2-1)}_{\Delta M_i} - \underbrace{\beta_i^2\sigma_{i-1}^2\rho^2Z_i^2}_{\Delta R_i} = \sigma_n^2 + \sum_{j=n+1}^{i}\Delta M_j - \sum_{j=n+1}^{i}\Delta R_j. \tag*{(23)} σ i 2 = σ i − 1 2 + Δ M i β i σ i − 1 2 ρ 2 ( Z i 2 − 1 ) − Δ R i β i 2 σ i − 1 2 ρ 2 Z i 2 = σ n 2 + j = n + 1 ∑ i Δ M j − j = n + 1 ∑ i Δ R j . (23)
Note that M i = σ n 2 + ∑ j = n + 1 i Δ M j M_i = \sigma_n^2 + \sum_{j=n+1}^{i}\Delta M_j M i = σ n 2 + ∑ j = n + 1 i Δ M j is now a B B B -valued martingale, and R i = ∑ j = n + 1 i Δ R j R_i = \sum_{j=n+1}^{i}\Delta R_j R i = ∑ j = n + 1 i Δ R j is an error term. First, we have
E [ ∥ Δ M i ∥ B 2 ∣ F i − 1 ] ≲ i − 2 m ∥ σ i − 1 2 ∥ B 2 E\left[\|\Delta M_i\|_B^2 \mid\mathcal{F}_{i-1}\right] \lesssim i^{-2m}\|\sigma_{i-1}^2\|_B^2 E [ ∥Δ M i ∥ B 2 ∣ F i − 1 ] ≲ i − 2 m ∥ σ i − 1 2 ∥ B 2
where we have used E [ ( Z i 2 − 1 ) 2 ] = 2 E[(Z_i^2-1)^2] = 2 E [( Z i 2 − 1 ) 2 ] = 2 . Lemma C.3 then gives
sup i ≥ n E [ ∥ σ i 2 ∥ B 2 ] ≤ ∥ σ n 2 ∥ B 2 < ∞ \sup_{i\ge n} E\left[\|\sigma_i^2\|_B^2\right] \le\|\sigma_n^2\|_B^2 < \infty i ≥ n sup E [ ∥ σ i 2 ∥ B 2 ] ≤ ∥ σ n 2 ∥ B 2 < ∞
which gives E [ ∥ Δ M i ∥ B 2 ] = O ( i − 2 m ) E[\|\Delta M_i\|_B^2] = O(i^{-2m}) E [ ∥Δ M i ∥ B 2 ] = O ( i − 2 m ) . This is summable as m > 0.5 m > 0.5 m > 0.5 , so martingale orthogonality gives that { M i } i ≥ n \{M_i\}_{i\ge n} { M i } i ≥ n is bounded in L 2 ( B ) L^2(B) L 2 ( B ) and converges a.s. to M ∞ ∈ B M_\infty\in B M ∞ ∈ B .
For the remainder term, we note that ∥ Δ R i ∥ B ≲ Z i 2 ∥ σ i − 1 2 ∥ B i − 2 m \|\Delta R_i\|_B \lesssim Z_i^2\|\sigma_{i-1}^2\|_B i^{-2m} ∥Δ R i ∥ B ≲ Z i 2 ∥ σ i − 1 2 ∥ B i − 2 m , and a similar argument gives E [ ∥ Δ R i ∥ B ] = O ( i − 2 m ) E[\|\Delta R_i\|_B] = O(i^{-2m}) E [ ∥Δ R i ∥ B ] = O ( i − 2 m ) , so ∑ i = n + 1 ∞ E [ ∥ Δ R i ∥ B ] < ∞ \sum_{i=n+1}^{\infty} E[\|\Delta R_i\|_B] < \infty ∑ i = n + 1 ∞ E [ ∥Δ R i ∥ B ] < ∞ which implies ∑ i = n + 1 ∞ ∥ Δ R i ∥ B < ∞ \sum_{i=n+1}^{\infty}\|\Delta R_i\|_B < \infty ∑ i = n + 1 ∞ ∥Δ R i ∥ B < ∞ a.s. Since B B B is complete and R i R_i R i is a Cauchy sequence a.s., it converges a.s. to R ∞ ∈ B R_\infty\in B R ∞ ∈ B . Together, this implies σ i 2 \sigma_i^2 σ i 2 converges a.s. to $\sigma_\infty^2 \in B.
Finally, we verify the positivity of σ ∞ 2 \sigma_\infty^2 σ ∞ 2 . Following a similar argument to Garelli et al. [ [14 ] , we take the log of the variance update,
log σ i 2 = log σ i − 1 2 + log ( 1 + U i ) , \log\sigma_i^2 = \log\sigma_{i-1}^2 + \log(1+U_i), log σ i 2 = log σ i − 1 2 + log ( 1 + U i ) ,
where we write U i = β i ρ 2 [ ( 1 − β i ) Z i 2 − 1 ] U_i = \beta_i \rho^2 \left[(1-\beta_i)Z_i^2-1\right] U i = β i ρ 2 [ ( 1 − β i ) Z i 2 − 1 ] . The above can be handled in a similar fashion through a decomposition into a martingale and error term. We write
log σ i 2 = log σ i − 1 2 + Δ M i ′ + Δ R i ′ \log\sigma_i^2 = \log\sigma_{i-1}^2 + \Delta M_i' + \Delta R_i' log σ i 2 = log σ i − 1 2 + Δ M i ′ + Δ R i ′
where Δ M i ′ = β i ρ 2 ( Z i 2 − 1 ) \Delta M_i'=\beta_i\rho^2(Z_i^2-1) Δ M i ′ = β i ρ 2 ( Z i 2 − 1 ) and
Δ R i ′ = log ( 1 + U i ) − Δ M i ′ = { log ( 1 + U i ) − U i } − ρ 2 β i 2 Z i 2 . \Delta R_i'=\log(1+U_i)-\Delta M_i'=\{\log(1+U_i)-U_i\}-\rho^2\beta_i^2Z_i^2. Δ R i ′ = log ( 1 + U i ) − Δ M i ′ = { log ( 1 + U i ) − U i } − ρ 2 β i 2 Z i 2 .
We can use the same earlier argument to show the martingale term converges a.s., and log σ n 2 ∈ B \log\sigma_n^2 \in B log σ n 2 ∈ B as 1 − ρ 2 ≤ 1 + U i ≤ 1 + ρ 2 Z i 2 1-\rho^2\leq1+U_i\leq1+\rho^2Z_i^2 1 − ρ 2 ≤ 1 + U i ≤ 1 + ρ 2 Z i 2 as log σ 0 ∈ B \log\sigma_0\in B log σ 0 ∈ B from Assumption 3. It thus suffices to show R i ′ = ∑ j = n + 1 i Δ R j ′ R_i'=\sum_{j=n+1}^{i}\Delta R_j' R i ′ = ∑ j = n + 1 i Δ R j ′ is a Cauchy sequence. The term ρ 2 β i 2 Z i 2 \rho^2\beta_i^2Z_i^2 ρ 2 β i 2 Z i 2 can be handled similarly to Δ R i \Delta R_i Δ R i earlier. Next, for sufficiently large i i i , we have U i ≥ − ρ 2 K i − m ≥ − 1 2 U_i\geq-\rho^2Ki^{-m}\geq-\frac{1}{2} U i ≥ − ρ 2 K i − m ≥ − 2 1 for all x ∈ X x\in\mathcal{X} x ∈ X . Combined with ∣ log ( 1 + x ) − x ∣ ≤ x 2 |\log(1+x)-x|\leq x^2 ∣ log ( 1 + x ) − x ∣ ≤ x 2 for x > − 1 / 2 x>-1/2 x > − 1/2 , we have
∣ log ( 1 + U i ) − U i ∣ ≤ U i 2 ≲ i − 2 m ( 1 + Z i 2 ) 2 . |\log(1+U_i)-U_i|\leq U_i^2\lesssim i^{-2m}(1+Z_i^2)^2. ∣ log ( 1 + U i ) − U i ∣ ≤ U i 2 ≲ i − 2 m ( 1 + Z i 2 ) 2 .
Since the bound does not depend on x x x and ∫ X d x < ∞ \int_{\mathcal{X}}dx<\infty ∫ X d x < ∞ , we have
∥ log ( 1 + U i ) − U i ∥ B ≲ i − 2 m ( 1 + Z i 2 ) 2 . \|\log(1+U_i)-U_i\|_B\lesssim i^{-2m}(1+Z_i^2)^2. ∥ log ( 1 + U i ) − U i ∥ B ≲ i − 2 m ( 1 + Z i 2 ) 2 .
Finally, as E [ ( 1 + Z i 2 ) 2 ] = 6 E[(1+Z_i^2)^2]=6 E [( 1 + Z i 2 ) 2 ] = 6 , we have ∑ i = n + 1 ∞ E [ ∥ Δ R i ′ ∥ B ] < ∞ \sum_{i=n+1}^{\infty}E[\|\Delta R_i'\|_B]<\infty ∑ i = n + 1 ∞ E [ ∥Δ R i ′ ∥ B ] < ∞ giving ∑ i = n + 1 ∞ ∥ Δ R i ′ ∥ B < ∞ \sum_{i=n+1}^{\infty}\|\Delta R_i'\|_B<\infty ∑ i = n + 1 ∞ ∥Δ R i ′ ∥ B < ∞ a.s. So R i ′ R_i' R i ′ is a Cauchy sequence and converges to R ∞ ′ ∈ B R_\infty'\in B R ∞ ′ ∈ B a.s. Together, this implies log σ i 2 \log\sigma_i^2 log σ i 2 converges to ℓ ∞ ∈ B \ell_\infty\in B ℓ ∞ ∈ B a.s.
We now show σ ∞ 2 = exp ( ℓ ∞ ) > 0 \sigma_\infty^2=\exp(\ell_\infty)>0 σ ∞ 2 = exp ( ℓ ∞ ) > 0 for Lebesgue-almost all x ∈ X x\in\mathcal{X} x ∈ X a.s. We will work on the probability-one set on which σ i 2 \sigma_i^2 σ i 2 and log σ i 2 \log\sigma_i^2 log σ i 2 converge in L 2 ( X ) L^2(\mathcal{X}) L 2 ( X ) . Since L 2 ( X ) L^2(\mathcal{X}) L 2 ( X ) convergence implies convergence in (Lebesgue) measure, we have that both σ i 2 → σ ∞ 2 \sigma_i^2\to\sigma_\infty^2 σ i 2 → σ ∞ 2 and log σ i 2 → ℓ ∞ \log\sigma_i^2\to\ell_\infty log σ i 2 → ℓ ∞ in measure. Continuity of exp ( ⋅ ) \exp(\cdot) exp ( ⋅ ) gives σ i 2 → exp ( ℓ ∞ ) \sigma_i^2\to\exp(\ell_\infty) σ i 2 → exp ( ℓ ∞ ) in measure. Finally, uniqueness of limits in measure gives σ ∞ 2 = exp ( ℓ ∞ ) \sigma_\infty^2=\exp(\ell_\infty) σ ∞ 2 = exp ( ℓ ∞ ) Lebesgue-almost everywhere. Since ℓ ∞ ∈ L 2 ( X ) \ell_\infty\in L^2(\mathcal{X}) ℓ ∞ ∈ L 2 ( X ) , it must be finite Lebesgue-almost everywhere, so σ ∞ 2 ( x ) > 0 \sigma_\infty^2(x)>0 σ ∞ 2 ( x ) > 0 for Lebesgue-almost all x ∈ X x\in\mathcal{X} x ∈ X . This gives our desired positivity a.s.
Finally, we tackle the total variation convergence. Again we work on the appropriate probability-one set on which L 2 ( X ) L^2(\mathcal{X}) L 2 ( X ) convergence occurs. Let
D i ( x ) = 1 2 ∫ ∣ N { y ; μ i ( x ) , σ i 2 ( x ) } − N { y ; μ ∞ ( x ) , σ ∞ 2 ( x ) } ∣ d y . D_i(x) = \frac{1}{2} \int\left\lvert\mathcal{N}\{y; \mu_i(x), \sigma_i^2(x)\} - \mathcal{N}\{y; \mu_\infty(x), \sigma_\infty^2(x)\}\right\rvert\,dy. D i ( x ) = 2 1 ∫ N { y ; μ i ( x ) , σ i 2 ( x )} − N { y ; μ ∞ ( x ) , σ ∞ 2 ( x )} d y .
For each x x x , continuity of the Gaussian density and Scheffé’s lemma give D i ( x ) → 0 D_i(x) \to0 D i ( x ) → 0 if ( μ i ( x ) , σ i 2 ( x ) ) → ( μ ∞ ( x ) , σ ∞ 2 ( x ) ) (\mu_i(x), \sigma_i^2(x)) \to(\mu_\infty(x), \sigma_\infty^2(x)) ( μ i ( x ) , σ i 2 ( x )) → ( μ ∞ ( x ) , σ ∞ 2 ( x )) and σ ∞ 2 ( x ) > 0 \sigma_\infty^2(x) > 0 σ ∞ 2 ( x ) > 0 . L 2 ( X ) L^2(\mathcal{X}) L 2 ( X ) convergence of ( μ i , σ i 2 ) (\mu_i, \sigma_i^2) ( μ i , σ i 2 ) gives ( μ i , σ i 2 ) → ( μ ∞ , σ ∞ 2 ) (\mu_i, \sigma_i^2) \to(\mu_\infty, \sigma_\infty^2) ( μ i , σ i 2 ) → ( μ ∞ , σ ∞ 2 ) in P X P_X P X -measure, following Assumption 4. Together with the positivity of σ ∞ 2 \sigma_\infty^2 σ ∞ 2 for P X P_X P X -almost all x x x , we have D i → 0 D_i \to0 D i → 0 in P X P_X P X -measure from the continuity of total variation distance as a function of the Gaussian parameters. Moreover, 0 ≤ D i ≤ 1 0 \le D_i \le1 0 ≤ D i ≤ 1 . Hence, for every ε > 0 \varepsilon> 0 ε > 0 ,
∫ D i ( x ) d P X ( x ) ≤ ε + P X ( D i > ε ) \int D_i(x)\,dP_X(x) \le\varepsilon+ P_X(D_i > \varepsilon) ∫ D i ( x ) d P X ( x ) ≤ ε + P X ( D i > ε )
where P X ( D i > ε ) → 0 P_X(D_i > \varepsilon) \to0 P X ( D i > ε ) → 0 . Letting ε → 0 \varepsilon\to0 ε → 0 gives ∫ D i ( x ) d P X ( x ) → 0 \int D_i(x)\,dP_X(x) \to0 ∫ D i ( x ) d P X ( x ) → 0 as desired.
Proof of Corollary 2 (Theorem 1 part)
Let g δ ( ε ) g_\delta(\varepsilon) g δ ( ε ) be defined as in (19). The proof is largely similar to that of Theorem 1, so we focus on the differences. We replace ρ Z i \rho Z_i ρ Z i with R δ ( 1 ) ( ε i ) R_\delta^{(1)}(\varepsilon_i) R δ ( 1 ) ( ε i ) and ρ 2 ( Z i 2 − 1 ) \rho^2(Z_i^2-1) ρ 2 ( Z i 2 − 1 ) with R δ ( 2 ) ( ε i ) − 1 R_\delta^{(2)}(\varepsilon_i)-1 R δ ( 2 ) ( ε i ) − 1 . Under predictive resampling, ε i ∼ i i d N ( 0 , 1 ) \varepsilon_i \mathrel{\overset{\mathrm{iid}}{\sim}} \mathcal{N}(0,1) ε i ∼ iid N ( 0 , 1 ) , which gives
E [ R δ ( j ) ( ε i ) ∣ F i − 1 , X i ] = E W ′ ∼ N ( 0 , 1 ) [ g δ ( W ′ ) j ] = { 0 if j = 1 , 1 if j = 2 , E\left[R_\delta^{(j)}(\varepsilon_i)\mid\mathcal{F}_{i-1},X_i\right] = E_{W'\sim\mathcal{N}(0,1)}\left[g_\delta(W')^j\right] = \begin{cases} 0 & \text{if } j=1, \\ 1 & \text{if } j=2, \end{cases} E [ R δ ( j ) ( ε i ) ∣ F i − 1 , X i ] = E W ′ ∼ N ( 0 , 1 ) [ g δ ( W ′ ) j ] = { 0 1 if j = 1 , if j = 2 ,
since W ′ = 1 − ρ 2 W + ρ ε i ∼ N ( 0 , 1 ) W'=\sqrt{1-\rho^2}W+\rho\varepsilon_i\sim\mathcal{N}(0,1) W ′ = 1 − ρ 2 W + ρ ε i ∼ N ( 0 , 1 ) . We rely on the following property of g δ g_\delta g δ .
Lemma C.4. For fixed δ > 0 \delta> 0 δ > 0 , the transformation g δ g_\delta g δ satisfies
κ δ : = E W ′ ∼ N ( 0 , 1 ) [ g δ ( W ′ ) 4 ] < ∞ . \kappa_\delta:= E_{W'\sim\mathcal{N}(0,1)}\left[g_\delta(W')^4\right] < \infty. κ δ := E W ′ ∼ N ( 0 , 1 ) [ g δ ( W ′ ) 4 ] < ∞. Furthermore, we have
E [ { R δ ( 1 ) ( ε i ) } 4 ∣ F i − 1 , X i ] ≤ κ δ , E [ { R δ ( 2 ) ( ε i ) } 2 ∣ F i − 1 , X i ] ≤ κ δ . E\left[\left\{R_\delta^{(1)}(\varepsilon_i)\right\}^4\mid\mathcal{F}_{i-1},X_i\right] \le\kappa_\delta,\qquad E\left[\left\{R_\delta^{(2)}(\varepsilon_i)\right\}^2\mid\mathcal{F}_{i-1},X_i\right] \le\kappa_\delta. E [ { R δ ( 1 ) ( ε i ) } 4 ∣ F i − 1 , X i ] ≤ κ δ , E [ { R δ ( 2 ) ( ε i ) } 2 ∣ F i − 1 , X i ] ≤ κ δ . Proof. The Gaussian moment generating function gives κ δ = 1 2 ( ω 4 + 2 ω 2 + 3 ) < ∞ \kappa_\delta=\frac{1}{2}(\omega^4+2\omega^2+3)<\infty κ δ = 2 1 ( ω 4 + 2 ω 2 + 3 ) < ∞ . The remaining bounds follow from Jensen’s inequality, since R δ ( j ) ( ε i ) = E [ g δ ( W ′ ) j ∣ ε i ] R_\delta^{(j)}(\varepsilon_i)=E[g_\delta(W')^j\mid\varepsilon_i] R δ ( j ) ( ε i ) = E [ g δ ( W ′ ) j ∣ ε i ] for W ′ = 1 − ρ 2 W + ρ ε i W'=\sqrt{1-\rho^2}W+\rho\varepsilon_i W ′ = 1 − ρ 2 W + ρ ε i with W ∼ N ( 0 , 1 ) W\sim\mathcal{N}(0,1) W ∼ N ( 0 , 1 ) independently.
Lemma C.5. Let ε ∼ N ( 0 , 1 ) \varepsilon\sim\mathcal{N}(0,1) ε ∼ N ( 0 , 1 ) , 0 ≤ a < 1 0 \le a < 1 0 ≤ a < 1 , 0 < ρ < 1 0 < \rho< 1 0 < ρ < 1 and δ > 0 \delta> 0 δ > 0 . Then there exists a finite C C C such that
E [ ( 1 + a { R δ ( 2 ) ( ε ) − 1 − a R δ ( 1 ) ( ε ) 2 } ) 2 ] ≤ 1 + C a 2 . E\left[\left(1+a\left\{R_{\delta}^{(2)}(\varepsilon)-1-aR_{\delta}^{(1)}(\varepsilon)^2\right\}\right)^2\right] \le1+Ca^2. E [ ( 1 + a { R δ ( 2 ) ( ε ) − 1 − a R δ ( 1 ) ( ε ) 2 } ) 2 ] ≤ 1 + C a 2 . Proof. Jensen’s inequality gives R δ ( 1 ) ( ε ) 2 ≤ R δ ( 2 ) ( ε ) R_{\delta}^{(1)}(\varepsilon)^2 \le R_{\delta}^{(2)}(\varepsilon) R δ ( 1 ) ( ε ) 2 ≤ R δ ( 2 ) ( ε ) , so
0 < 1 − a ≤ 1 + a { R δ ( 2 ) ( ε ) − 1 − a R δ ( 1 ) ( ε ) 2 } ≤ 1 + a { R δ ( 2 ) ( ε ) − 1 } . 0 < 1-a \le1+a\left\{R_{\delta}^{(2)}(\varepsilon)-1-aR_{\delta}^{(1)}(\varepsilon)^2\right\} \le1+a\left\{R_{\delta}^{(2)}(\varepsilon)-1\right\}. 0 < 1 − a ≤ 1 + a { R δ ( 2 ) ( ε ) − 1 − a R δ ( 1 ) ( ε ) 2 } ≤ 1 + a { R δ ( 2 ) ( ε ) − 1 } . Since the above term is positive, we can square and take expectations, where the square of the upper bound has expectation
1 + a 2 E [ { R δ ( 2 ) ( ε ) − 1 } 2 ] ≤ 1 + ( κ δ − 1 ) a 2 , 1+a^2E\left[\left\{R_{\delta}^{(2)}(\varepsilon)-1\right\}^2\right] \le1+(\kappa_{\delta}-1)a^2, 1 + a 2 E [ { R δ ( 2 ) ( ε ) − 1 } 2 ] ≤ 1 + ( κ δ − 1 ) a 2 , where we have used E [ R δ ( 2 ) ( ε ) ] = 1 E[R_{\delta}^{(2)}(\varepsilon)]=1 E [ R δ ( 2 ) ( ε )] = 1 and Lemma C.4.
Lemma C.4 replaces the Gaussian fourth moment bound, whereas Lemma C.5 replaces Lemma C.3. The argument for the convergence μ i → μ ∞ \mu_i \to\mu_\infty μ i → μ ∞ is unchanged.
For the variance, Lemmas C.2 and C.5 give
E [ ∥ σ i 2 ∥ B 2 ∣ F i − 1 ] ≤ ( 1 + C i − 2 m ) ∥ σ i − 1 2 ∥ B 2 E[\|\sigma_i^2\|_B^2 \mid\mathcal{F}_{i-1}] \le(1+Ci^{-2m})\|\sigma_{i-1}^2\|_B^2 E [ ∥ σ i 2 ∥ B 2 ∣ F i − 1 ] ≤ ( 1 + C i − 2 m ) ∥ σ i − 1 2 ∥ B 2
which when iterated gives sup i E [ ∥ σ i 2 ∥ B 2 ] < ∞ \sup_i E[\|\sigma_i^2\|_B^2] < \infty sup i E [ ∥ σ i 2 ∥ B 2 ] < ∞ . The variance decomposition now has
Δ M i = β i σ i − 1 2 { R δ ( 2 ) ( ε i ) − 1 } , Δ R i = β i 2 σ i − 1 2 R δ ( 1 ) ( ε i ) 2 , \Delta M_i=\beta_i\sigma_{i-1}^2\left\{R_{\delta}^{(2)}(\varepsilon_i)-1\right\}, \qquad\Delta R_i=\beta_i^2\sigma_{i-1}^2R_{\delta}^{(1)}(\varepsilon_i)^2, Δ M i = β i σ i − 1 2 { R δ ( 2 ) ( ε i ) − 1 } , Δ R i = β i 2 σ i − 1 2 R δ ( 1 ) ( ε i ) 2 ,
where Lemma C.4 can be used to control the norms to show σ i 2 → σ ∞ 2 \sigma_i^2 \to\sigma_\infty^2 σ i 2 → σ ∞ 2 a.s.
To verify positivity, write
U i = β i { R δ ( 2 ) ( ε i ) − 1 − β i R δ ( 1 ) ( ε i ) 2 } . U_i=\beta_i\left\{R_{\delta}^{(2)}(\varepsilon_i)-1-\beta_iR_{\delta}^{(1)}(\varepsilon_i)^2\right\}. U i = β i { R δ ( 2 ) ( ε i ) − 1 − β i R δ ( 1 ) ( ε i ) 2 } .
As in the Gaussian case, we decompose the log-variance update as
log σ i 2 = log σ i − 1 2 + Δ M i ′ + Δ R i ′ , \log\sigma_i^2=\log\sigma_{i-1}^2+\Delta M_i^\prime+\Delta R_i^\prime, log σ i 2 = log σ i − 1 2 + Δ M i ′ + Δ R i ′ ,
where Δ M i ′ = β i { R δ ( 2 ) ( ε i ) − 1 } \Delta M_i^\prime=\beta_i\{R_{\delta}^{(2)}(\varepsilon_i)-1\} Δ M i ′ = β i { R δ ( 2 ) ( ε i ) − 1 } and
Δ R i ′ = { log ( 1 + U i ) − U i } − β i 2 R δ ( 1 ) ( ε i ) 2 . \Delta R_i^\prime=\{\log(1+U_i)-U_i\}-\beta_i^2R_{\delta}^{(1)}(\varepsilon_i)^2. Δ R i ′ = { log ( 1 + U i ) − U i } − β i 2 R δ ( 1 ) ( ε i ) 2 .
Again, U i ≥ − β i ≥ − 1 / 2 U_i \ge-\beta_i \ge-1/2 U i ≥ − β i ≥ − 1/2 for sufficiently large i i i uniformly in x x x , and the rest of the convergence proof is identical. Finally, for fixed δ > 0 \delta> 0 δ > 0 , the SU density is continuous in ( μ , σ 2 ) (\mu,\sigma^2) ( μ , σ 2 ) whenever σ 2 > 0 \sigma^2 > 0 σ 2 > 0 , so the total variation argument follows.
Finally, we highlight that the entire proof extends to general transformations g λ g_\lambda g λ if analogous versions of Lemmas C.4 and C.5 and continuity of the density hold.
Extension to the multi-output case
For the multi-output setting, we use the Euclidean norm ∥ ⋅ ∥ \|\cdot\| ∥ ⋅ ∥ for vectors and the Frobenius norm ∥ ⋅ ∥ F \|\cdot\|_F ∥ ⋅ ∥ F for matrices. We begin with a statement of the multi-output version of Assumption 3, followed by the multi-output extension of Theorem 1.
Assumption C.1 (Initialization). The initial estimates for the recursion are μ 0 ( x ) = 0 \mu_0(x) = 0 μ 0 ( x ) = 0 and Σ 0 ( x ) = I q \Sigma_0(x) = I_q Σ 0 ( x ) = I q .
Theorem C.1. Consider the multivariate version of Algorithm 1 with updates (12) and (13). Under Assumptions 1, 2, C.1 and 4, there exist random μ ∞ ∈ L 2 ( X ; R q ) \mu_\infty\in L^2(\mathcal{X};\mathbb{R}^q) μ ∞ ∈ L 2 ( X ; R q ) and Σ ∞ ∈ L 2 ( X ; R q × q ) \Sigma_\infty\in L^2(\mathcal{X};\mathbb{R}^{q \times q}) Σ ∞ ∈ L 2 ( X ; R q × q ) , with Σ ∞ ( x ) ≻ 0 \Sigma_\infty(x) \succ0 Σ ∞ ( x ) ≻ 0 for Lebesgue-almost all x ∈ X x \in\mathcal{X} x ∈ X , such that
∫ X ∥ μ ∞ ( x ) − μ i ( x ) ∥ 2 d x → 0 , ∫ X ∥ Σ ∞ ( x ) − Σ i ( x ) ∥ F 2 d x → 0 a.s. \int_{\mathcal{X}} \|\mu_\infty(x) - \mu_i(x)\|^2\,dx \to0,\qquad\int_{\mathcal{X}} \|\Sigma_\infty(x) - \Sigma_i(x)\|_F^2\,dx \to0\quad\text{a.s.} ∫ X ∥ μ ∞ ( x ) − μ i ( x ) ∥ 2 d x → 0 , ∫ X ∥ Σ ∞ ( x ) − Σ i ( x ) ∥ F 2 d x → 0 a.s. Furthermore, we have the total variation convergence
1 2 ∫ X [ ∫ ∣ p i ( y ∣ x ) − N { y ; μ ∞ ( x ) , Σ ∞ ( x ) } ∣ d y ] p X ( x ) d x → 0 a.s. \frac{1}{2}\int_{\mathcal{X}}\left[\int\left|p_i(y \mid x) - \mathcal{N}\{y;\mu_\infty(x),\Sigma_\infty(x)\}\right|\,dy\right]p_X(x)\,dx \to0\quad\text{a.s.} 2 1 ∫ X [ ∫ ∣ p i ( y ∣ x ) − N { y ; μ ∞ ( x ) , Σ ∞ ( x )} ∣ d y ] p X ( x ) d x → 0 a.s.
Proof. Again, the proof is practically identical to that of Theorem 1, so we focus on the differences. We use B B B interchangeably to denote L 2 ( X ; R q ) L^2(\mathcal{X};\mathbb{R}^q) L 2 ( X ; R q ) and L 2 ( X ; R q × q ) L^2(\mathcal{X};\mathbb{R}^{q\times q}) L 2 ( X ; R q × q ) , with respective norms ∥ μ ∥ B 2 = ∫ X ∥ μ ( x ) ∥ 2 d x \|\mu\|_B^2 = \int_{\mathcal{X}}\|\mu(x)\|^2\,dx ∥ μ ∥ B 2 = ∫ X ∥ μ ( x ) ∥ 2 d x for a vector function μ \mu μ , and ∥ Σ ∥ B 2 = ∫ X ∥ Σ ( x ) ∥ F 2 d x \|\Sigma\|_B^2 = \int_{\mathcal{X}}\|\Sigma(x)\|_F^2\,dx ∥Σ ∥ B 2 = ∫ X ∥Σ ( x ) ∥ F 2 d x for a matrix function Σ \Sigma Σ .
We will rely on a few lemmas. The first is for handling the variance increments.
Lemma C.6. Let Z ∼ N ( 0 , I q ) Z \sim\mathcal{N}(0,I_q) Z ∼ N ( 0 , I q ) and Σ \Sigma Σ be a positive definite matrix with Cholesky factor L L L . Then we have
E [ ∥ L ( Z Z T − I ) L T ∥ F 2 ] ≤ ( q + 1 ) ∥ Σ ∥ F 2 . \mathbb{E}\left[\left\|L(ZZ^\mathsf{T}-I)L^\mathsf{T}\right\|_F^2\right] \le(q+1)\|\Sigma\|_F^2. E [ L ( Z Z T − I ) L T F 2 ] ≤ ( q + 1 ) ∥Σ ∥ F 2 .
Proof. The matrix is symmetric so its squared Frobenius norm is the trace of the matrix squared, giving
∥ L ( Z Z T − I ) L T ∥ F 2 = tr { ( L Z Z T L T ) 2 } − 2 tr ( Σ L Z Z T L T ) + tr ( Σ 2 ) . \left\|L(ZZ^{\mathsf{T}}-I)L^{\mathsf{T}}\right\|_{F}^{2}=\operatorname{tr}\left\{\left(LZZ^{\mathsf{T}}L^{\mathsf{T}}\right)^{2}\right\}-2\operatorname{tr}\left(\Sigma LZZ^{\mathsf{T}}L^{\mathsf{T}}\right)+\operatorname{tr}\left(\Sigma^{2}\right). L ( Z Z T − I ) L T F 2 = tr { ( L Z Z T L T ) 2 } − 2 tr ( Σ L Z Z T L T ) + tr ( Σ 2 ) .
The second term has expectation
E [ tr ( Σ L Z Z T L T ) ] = tr ( Σ 2 ) E\left[\operatorname{tr}\left(\Sigma LZZ^{\mathsf{T}}L^{\mathsf{T}}\right)\right]=\operatorname{tr}\left(\Sigma^{2}\right) E [ tr ( Σ L Z Z T L T ) ] = tr ( Σ 2 )
from linearity of the trace. Note that L Z LZ L Z is a vector, so we have tr { ( L Z Z T L T ) 2 } = ( Z T L T L Z ) 2 \operatorname{tr}\left\{\left(LZZ^{\mathsf{T}}L^{\mathsf{T}}\right)^{2}\right\}=\left(Z^{\mathsf{T}}L^{\mathsf{T}}LZ\right)^{2} tr { ( L Z Z T L T ) 2 } = ( Z T L T L Z ) 2 . Furthermore, L Z ∼ N ( 0 , Σ ) LZ\sim\mathcal{N}(0,\Sigma) L Z ∼ N ( 0 , Σ ) , so standard Gaussian quadratic form identities [28 Section 8.2.2] then give
E [ Z T L T L Z ] = tr Σ , Var [ Z T L T L Z ] = 2 ∥ Σ ∥ F 2 , (24) E\left[Z^{\mathsf{T}}L^{\mathsf{T}}LZ\right]=\operatorname{tr}\Sigma,\qquad\operatorname{Var}\left[Z^{\mathsf{T}}L^{\mathsf{T}}LZ\right]=2\|\Sigma\|_{F}^{2}, \tag*{(24)} E [ Z T L T L Z ] = tr Σ , Var [ Z T L T L Z ] = 2∥Σ ∥ F 2 , (24)
which implies E [ ( Z T L T L Z ) 2 ] = 2 tr ( Σ 2 ) + ( tr Σ ) 2 E\left[\left(Z^{\mathsf{T}}L^{\mathsf{T}}LZ\right)^{2}\right]=2\operatorname{tr}\left(\Sigma^{2}\right)+(\operatorname{tr}\Sigma)^{2} E [ ( Z T L T L Z ) 2 ] = 2 tr ( Σ 2 ) + ( tr Σ ) 2 . Together, this gives
E [ ∥ L ( Z Z T − I ) L T ∥ F 2 ] = ( tr Σ ) 2 + tr ( Σ 2 ) ≤ ( q + 1 ) ∥ Σ ∥ F 2 , E\left[\left\|L(ZZ^{\mathsf{T}}-I)L^{\mathsf{T}}\right\|_{F}^{2}\right]=(\operatorname{tr}\Sigma)^{2}+\operatorname{tr}\left(\Sigma^{2}\right)\le(q+1)\|\Sigma\|_{F}^{2}, E [ L ( Z Z T − I ) L T F 2 ] = ( tr Σ ) 2 + tr ( Σ 2 ) ≤ ( q + 1 ) ∥Σ ∥ F 2 ,
where we used ( tr Σ ) 2 = ⟨ Σ , I ⟩ F 2 ≤ q ∥ Σ ∥ F 2 (\operatorname{tr}\Sigma)^{2}=\langle\Sigma,I\rangle_{F}^{2}\le q\|\Sigma\|_{F}^{2} ( tr Σ ) 2 = ⟨ Σ , I ⟩ F 2 ≤ q ∥Σ ∥ F 2 following Cauchy-Schwarz. □
The second is a multivariate version of Lemma C.3 .
Lemma C.7. Let Z ∼ N ( 0 , I q ) Z\sim\mathcal{N}(0,I_q) Z ∼ N ( 0 , I q ) , 0 ≤ a < 1 0\le a<1 0 ≤ a < 1 , 0 < ρ < 1 0<\rho<1 0 < ρ < 1 and let Σ \Sigma Σ be a positive definite matrix with Cholesky factor L L L . Let us define
Σ ′ = L [ I + a ρ 2 { ( 1 − a ) Z Z T − I } ] L T . \Sigma'=L\left[I+a\rho^{2}\left\{(1-a)ZZ^{\mathsf{T}}-I\right\}\right]L^{\mathsf{T}}. Σ ′ = L [ I + a ρ 2 { ( 1 − a ) Z Z T − I } ] L T . Then we have E [ ( tr Σ ′ ) 2 ] ≤ ( tr Σ ) 2 E[(\operatorname{tr}\Sigma')^{2}]\le(\operatorname{tr}\Sigma)^{2} E [( tr Σ ′ ) 2 ] ≤ ( tr Σ ) 2 . Furthermore, for any z ∈ R q z\in\mathbb{R}^{q} z ∈ R q , we have
I + a ρ 2 { ( 1 − a ) z z T − I } ⪰ ( 1 − ρ 2 ) I . I+a\rho^{2}\left\{(1-a)zz^{\mathsf{T}}-I\right\}\succeq(1-\rho^{2})I. I + a ρ 2 { ( 1 − a ) z z T − I } ⪰ ( 1 − ρ 2 ) I .
Proof. By linearity and cyclicity of the trace,
tr Σ ′ = ( 1 − a ρ 2 ) tr Σ + a ρ 2 ( 1 − a ) Z T L T L Z . \operatorname{tr}\Sigma'=(1-a\rho^{2})\operatorname{tr}\Sigma+a\rho^{2}(1-a)Z^{\mathsf{T}}L^{\mathsf{T}}LZ. tr Σ ′ = ( 1 − a ρ 2 ) tr Σ + a ρ 2 ( 1 − a ) Z T L T L Z .
Consequently, (24) gives
E [ ( tr Σ ′ ) 2 ] = Var [ tr Σ ′ ] + E [ tr Σ ′ ] 2 = 2 a 2 ρ 4 ( 1 − a ) 2 ∥ Σ ∥ F 2 + ( 1 − a 2 ρ 2 ) 2 ( tr Σ ) 2 ≤ { 1 − 2 a 2 ρ 2 + a 2 ρ 4 ( 2 − 4 a + 3 a 2 ) } ( tr Σ ) 2 ≤ ( tr Σ ) 2 . \begin{aligned}
E[(\operatorname{tr}\Sigma')^{2}]=\operatorname{Var}[\operatorname{tr}\Sigma']+E[\operatorname{tr}\Sigma']^{2}=2a^{2}\rho^{4}(1-a)^{2}\|\Sigma\|_{F}^{2}+(1-a^{2}\rho^{2})^{2}(\operatorname{tr}\Sigma)^{2} \\
&\le\left\{1-2a^{2}\rho^{2}+a^{2}\rho^{4}(2-4a+3a^{2})\right\}(\operatorname{tr}\Sigma)^{2} \\
&\le(\operatorname{tr}\Sigma)^{2}.
\end{aligned} E [( tr Σ ′ ) 2 ] = Var [ tr Σ ′ ] + E [ tr Σ ′ ] 2 = 2 a 2 ρ 4 ( 1 − a ) 2 ∥Σ ∥ F 2 + ( 1 − a 2 ρ 2 ) 2 ( tr Σ ) 2 ≤ { 1 − 2 a 2 ρ 2 + a 2 ρ 4 ( 2 − 4 a + 3 a 2 ) } ( tr Σ ) 2 ≤ ( tr Σ ) 2 .
The second line uses ∥ Σ ∥ F 2 ≤ ( tr Σ ) 2 \|\Sigma\|_{F}^{2} \le(\operatorname{tr}\Sigma)^{2} ∥Σ ∥ F 2 ≤ ( tr Σ ) 2 for positive semidefinite matrices, and the final inequality follows from the same scalar calculation as in Lemma C.3. The second part of the lemma again follows from setting z = 0 z=0 z = 0 and noting that 0 ≤ a < 1 0 \le a < 1 0 ≤ a < 1 . □ \square □
To begin, we note that Σ i \Sigma_i Σ i is positive definite at every finite step, since
I + β i ρ 2 { ( 1 − β i ) Z i Z i T − I } ⪰ ( 1 − ρ 2 ) I I+\beta_i\rho^2\{(1-\beta_i)Z_iZ_i^{\mathsf T}-I\}\succeq(1-\rho^2)I I + β i ρ 2 {( 1 − β i ) Z i Z i T − I } ⪰ ( 1 − ρ 2 ) I
from the second part of Lemma C.7. The mean and second moment matrix Σ i ( x ) + μ i ( x ) μ i ( x ) T \Sigma_i(x)+\mu_i(x)\mu_i(x)^{\mathsf T} Σ i ( x ) + μ i ( x ) μ i ( x ) T are pointwise martingales, and taking the trace of the latter gives
sup i ≥ n E ∥ μ i ∥ B 2 ≤ ∫ X { ∥ μ n ( x ) ∥ 2 + tr Σ n ( x ) } d x < ∞ , \sup_{i\ge n} E\|\mu_i\|_{B}^{2}\le\int_{\mathcal X}\{\|\mu_n(x)\|^{2}+\operatorname{tr}\Sigma_n(x)\}\,dx<\infty, i ≥ n sup E ∥ μ i ∥ B 2 ≤ ∫ X { ∥ μ n ( x ) ∥ 2 + tr Σ n ( x )} d x < ∞ ,
giving μ i → μ ∞ \mu_i\to\mu_\infty μ i → μ ∞ a.s. Next, we again decompose into Σ i = Σ i − 1 + Δ M i − Δ R i \Sigma_i=\Sigma_{i-1}+\Delta M_i-\Delta R_i Σ i = Σ i − 1 + Δ M i − Δ R i , where
Δ M i = β i ρ 2 L i − 1 ( Z i Z i T − I ) L i − 1 T , Δ R i = β i 2 ρ 2 L i − 1 Z i Z i T L i − 1 T . \Delta M_i=\beta_i\rho^2L_{i-1}(Z_iZ_i^{\mathsf T}-I)L_{i-1}^{\mathsf T},\qquad\Delta R_i=\beta_i^2\rho^2L_{i-1}Z_iZ_i^{\mathsf T}L_{i-1}^{\mathsf T}. Δ M i = β i ρ 2 L i − 1 ( Z i Z i T − I ) L i − 1 T , Δ R i = β i 2 ρ 2 L i − 1 Z i Z i T L i − 1 T .
Lemmas C.2 and C.6 together give E [ ∥ Δ M i ∥ B 2 ∣ F i − 1 ] ≲ i − 2 m ∥ Σ i − 1 ∥ B 2 E[\|\Delta M_i\|_{B}^{2}\mid\mathcal F_{i-1}]\lesssim i^{-2m}\|\Sigma_{i-1}\|_{B}^{2} E [ ∥Δ M i ∥ B 2 ∣ F i − 1 ] ≲ i − 2 m ∥ Σ i − 1 ∥ B 2 . For the remainder, (24) gives
E [ ∥ L Z Z T L T ∥ F 2 ] = 2 ∥ Σ ∥ F 2 + ( tr Σ ) 2 ≤ ( q + 2 ) ∥ Σ ∥ F 2 , E[\|LZZ^{\mathsf T}L^{\mathsf T}\|_{F}^{2}]=2\|\Sigma\|_{F}^{2}+(\operatorname{tr}\Sigma)^{2}\le(q+2)\|\Sigma\|_{F}^{2}, E [ ∥ L Z Z T L T ∥ F 2 ] = 2∥Σ ∥ F 2 + ( tr Σ ) 2 ≤ ( q + 2 ) ∥Σ ∥ F 2 ,
which gives E [ ∥ Δ R i ∥ B 2 ∣ F i − 1 ] ≲ i − 4 m ∥ Σ i − 1 ∥ B 2 E[\|\Delta R_i\|_{B}^{2}\mid\mathcal F_{i-1}]\lesssim i^{-4m}\|\Sigma_{i-1}\|_{B}^{2} E [ ∥Δ R i ∥ B 2 ∣ F i − 1 ] ≲ i − 4 m ∥ Σ i − 1 ∥ B 2 . Next, we bound sup i ≥ n E [ ∥ Σ i ∥ B 2 ] \sup_{i\ge n}E[\|\Sigma_i\|_{B}^{2}] sup i ≥ n E [ ∥ Σ i ∥ B 2 ] . Applying Lemma C.7 conditionally on F i − 1 \mathcal F_{i-1} F i − 1 and X i X_i X i gives
E [ ( tr Σ i ( x ) ) 2 ∣ F i − 1 , X i ] ≤ ( tr Σ i − 1 ( x ) ) 2 . E[(\operatorname{tr}\Sigma_i(x))^{2}\mid\mathcal F_{i-1},X_i]\le(\operatorname{tr}\Sigma_{i-1}(x))^{2}. E [( tr Σ i ( x ) ) 2 ∣ F i − 1 , X i ] ≤ ( tr Σ i − 1 ( x ) ) 2 .
Since ∥ Σ i ( x ) ∥ F 2 ≤ ( tr Σ i ( x ) ) 2 \|\Sigma_i(x)\|_{F}^{2}\le(\operatorname{tr}\Sigma_i(x))^{2} ∥ Σ i ( x ) ∥ F 2 ≤ ( tr Σ i ( x ) ) 2 , integrating and iterating gives
sup i ≥ n E [ ∥ Σ i ∥ B 2 ] ≤ ∫ X ( tr Σ n ( x ) ) 2 d x ≤ q ∥ Σ n ∥ B 2 < ∞ , \sup_{i\ge n}E[\|\Sigma_i\|_{B}^{2}]\le\int_{\mathcal X}(\operatorname{tr}\Sigma_n(x))^{2}\,dx\le q\|\Sigma_n\|_{B}^{2}<\infty, i ≥ n sup E [ ∥ Σ i ∥ B 2 ] ≤ ∫ X ( tr Σ n ( x ) ) 2 d x ≤ q ∥ Σ n ∥ B 2 < ∞ ,
so Σ i → Σ ∞ \Sigma_i\to\Sigma_\infty Σ i → Σ ∞ a.s. Finally, we verify Σ ∞ ≻ 0 \Sigma_\infty\succ0 Σ ∞ ≻ 0 , using log det Σ i \log\det\Sigma_i log det Σ i in place of log σ i 2 \log\sigma_i^2 log σ i 2 . Let
U i = β i ρ 2 { ( 1 − β i ) Z i Z i T − I } , log det Σ i = log det Σ i − 1 + log det ( I + U i ) . U_i=\beta_i\rho^2\{(1-\beta_i)Z_iZ_i^{\mathsf T}-I\},\qquad\log\det\Sigma_i=\log\det\Sigma_{i-1}+\log\det(I+U_i). U i = β i ρ 2 {( 1 − β i ) Z i Z i T − I } , log det Σ i = log det Σ i − 1 + log det ( I + U i ) .
We decompose the update as log det Σ i = log det Σ i − 1 + Δ M i ′ + Δ R i ′ \log\det\Sigma_i=\log\det\Sigma_{i-1}+\Delta M_i'+\Delta R_i' log det Σ i = log det Σ i − 1 + Δ M i ′ + Δ R i ′ , where
Δ M i ′ = β i ρ 2 ( ∥ Z i ∥ 2 − q ) , Δ R i ′ = { log det ( I + U i ) − tr U i } − β i 2 ρ 2 ∥ Z i ∥ 2 . \Delta M_i'=\beta_i\rho^2(\|Z_i\|^{2}-q),\qquad\Delta R_i'=\{\log\det(I+U_i)-\operatorname{tr}U_i\}-\beta_i^2\rho^2\|Z_i\|^{2}. Δ M i ′ = β i ρ 2 ( ∥ Z i ∥ 2 − q ) , Δ R i ′ = { log det ( I + U i ) − tr U i } − β i 2 ρ 2 ∥ Z i ∥ 2 .
Since E [ ∥ Z i ∥ 2 ] = q E[\|Z_i\|^{2}]=q E [ ∥ Z i ∥ 2 ] = q and Var ( ∥ Z i ∥ 2 ) = 2 q \operatorname{Var}(\|Z_i\|^{2})=2q Var ( ∥ Z i ∥ 2 ) = 2 q , the terms Δ M i ′ \Delta M_i' Δ M i ′ and β i 2 ρ 2 ∥ Z i ∥ 2 \beta_i^2\rho^2\|Z_i\|^{2} β i 2 ρ 2 ∥ Z i ∥ 2 are handled easily. For the remaining term, all eigenvalues of U i U_i U i are at least − β i ρ 2 ≥ − 1 / 2 -\beta_i\rho^2\ge-1/2 − β i ρ 2 ≥ − 1/2 for sufficiently large i i i , uniformly in x x x . Applying ∣ log ( 1 + u ) − u ∣ ≤ u 2 |\log(1+u)-u|\le u^2 ∣ log ( 1 + u ) − u ∣ ≤ u 2 to each eigenvalue,
∣ log det ( I + U i ) − tr U i ∣ ≤ ∑ j = 1 q λ j 2 = ∥ U i ∥ F 2 ≲ i − 2 m ( 1 + ∥ Z i ∥ 2 2 ) 2 . \left|\log\det(I+U_i)-\operatorname{tr}U_i\right|\le\sum_{j=1}^{q}\lambda_j^2=\|U_i\|_F^2\lesssim i^{-2m}(1+\|Z_i\|_2^2)^2. ∣ log det ( I + U i ) − tr U i ∣ ≤ j = 1 ∑ q λ j 2 = ∥ U i ∥ F 2 ≲ i − 2 m ( 1 + ∥ Z i ∥ 2 2 ) 2 .
Hence log det Σ i → ℓ ∞ \log\det\Sigma_i\to\ell_\infty log det Σ i → ℓ ∞ a.s. The positivity argument is then identical. Since Σ i → Σ ∞ \Sigma_i\to\Sigma_\infty Σ i → Σ ∞ in Lebesgue measure, Σ ∞ \Sigma_\infty Σ ∞ is symmetric positive semidefinite Lebesgue-almost everywhere. Continuity of the determinant and exponential give det Σ ∞ = exp ( ℓ ∞ ) > 0 \det\Sigma_\infty=\exp(\ell_\infty)>0 det Σ ∞ = exp ( ℓ ∞ ) > 0 Lebesgue-almost everywhere. Thus Σ ∞ ≻ 0 \Sigma_\infty\succ0 Σ ∞ ≻ 0 Lebesgue-almost everywhere a.s. The total variation convergence proof is then identical.
Proof of Theorem 2
The result is primarily driven by the smoothness of α i ( ⋅ , x ′ ) \alpha_i(\cdot,x') α i ( ⋅ , x ′ ) and properties of Sobolev spaces. The proof is similar to that of Theorem 1 with more care in managing norms due to the higher-order derivatives. To that end, we begin with some useful lemmas.
Lemma C.8. Under Assumptions 1, 2 and 5, we have
sup x ′ ∈ X ∥ α i ( ⋅ , x ′ ) ∥ W r , ∞ ≲ i − m . \sup_{x'\in\mathcal{X}}\|\alpha_i(\cdot,x')\|_{W^{r,\infty}}\lesssim i^{-m}. x ′ ∈ X sup ∥ α i ( ⋅ , x ′ ) ∥ W r , ∞ ≲ i − m .
Proof. Since continuous functions on compact spaces are bounded, Assumption 5 gives the existence of a finite S r S_r S r such that sup x ′ ∈ X ∥ s ( ⋅ , x ′ ) ∥ W r , ∞ ≤ S r \sup_{x'\in\mathcal{X}}\|s(\cdot,x')\|_{W^{r,\infty}}\le S_r sup x ′ ∈ X ∥ s ( ⋅ , x ′ ) ∥ W r , ∞ ≤ S r . Next, for 0 < a < 1 0<a<1 0 < a < 1 and s ≥ 0 s\ge0 s ≥ 0 , consider the function
f a ( s ) = s 1 − a + a s . f_a(s)=\frac{s}{1-a+as}. f a ( s ) = 1 − a + a s s .
which satisfies, for 1 ≤ k ≤ r 1\le k\le r 1 ≤ k ≤ r ,
∣ f a ( k ) ( s ) ∣ = k ! ( 1 − a ) a k − 1 ( 1 − a + a s ) k + 1 ≤ k ! ( 1 − a ) k + 1 . \left|f_a^{(k)}(s)\right|=\frac{k!(1-a)a^{k-1}}{(1-a+as)^{k+1}}\le\frac{k!}{(1-a)^{k+1}}. f a ( k ) ( s ) = ( 1 − a + a s ) k + 1 k ! ( 1 − a ) a k − 1 ≤ ( 1 − a ) k + 1 k ! .
Since α i → 0 \alpha_i\to0 α i → 0 and α i < 1 \alpha_i<1 α i < 1 for every i i i , we have inf i ≥ n + 1 ( 1 − α i ) > 0 \inf_{i\ge n+1}(1-\alpha_i)>0 inf i ≥ n + 1 ( 1 − α i ) > 0 . Thus these derivatives are uniformly bounded, as is the k = 0 k=0 k = 0 term, since f α i ( s ) ≤ S r / ( 1 − α i ) f_{\alpha_i}(s)\le S_r/(1-\alpha_i) f α i ( s ) ≤ S r / ( 1 − α i ) . For a multi-index ν \nu ν with 1 ≤ ∣ ν ∣ ≤ r 1\le|\nu|\le r 1 ≤ ∣ ν ∣ ≤ r , Faà di Bruno’s formula expresses the derivative ∂ ν [ f α i ∘ s ( ⋅ , x ′ ) ] \partial^\nu[f_{\alpha_i}\circ s(\cdot,x')] ∂ ν [ f α i ∘ s ( ⋅ , x ′ )] as a finite linear combination of terms
f α i ( k ) { s ( x , x ′ ) } ∏ j = 1 k ∂ ν j s ( x , x ′ ) where ∑ j = 1 k ν j = ν , ∣ ν j ∣ ≥ 1. f_{\alpha_i}^{(k)}\{s(x,x')\}\prod_{j=1}^{k}\partial^{\nu_j}s(x,x')\qquad\text{where}\quad\sum_{j=1}^{k}\nu_j=\nu,\quad|\nu_j|\ge1. f α i ( k ) { s ( x , x ′ )} j = 1 ∏ k ∂ ν j s ( x , x ′ ) where j = 1 ∑ k ν j = ν , ∣ ν j ∣ ≥ 1.
Every factor is uniformly bounded in i i i , x x x and x ′ x' x ′ , so
sup i ≥ n + 1 sup x ′ ∈ X ∥ f α i ∘ s ( ⋅ , x ′ ) ∥ W r , ∞ < ∞ . \sup_{i\ge n+1}\sup_{x'\in\mathcal{X}}\lVert f_{\alpha_i}\circ s(\cdot,x')\rVert_{W^{r,\infty}}<\infty. i ≥ n + 1 sup x ′ ∈ X sup ∥ f α i ∘ s ( ⋅ , x ′ ) ∥ W r , ∞ < ∞.
Multiplying by α i = O ( i − m ) \alpha_i=O(i^{-m}) α i = O ( i − m ) gives the result, since α i ( ⋅ , x ′ ) = α i f α i ∘ s ( ⋅ , x ′ ) \alpha_i(\cdot,x')=\alpha_i f_{\alpha_i}\circ s(\cdot,x') α i ( ⋅ , x ′ ) = α i f α i ∘ s ( ⋅ , x ′ ) .
Another useful lemma is the following.
Lemma C.9. Let f ∈ W r , ∞ ( D ) f\in W^{r,\infty}(D) f ∈ W r , ∞ ( D ) and g ∈ W r , 2 ( D ) g\in W^{r,2}(D) g ∈ W r , 2 ( D ) . Then we have
∥ f g ∥ W r , 2 ≲ ∥ f ∥ W r , ∞ ∥ g ∥ W r , 2 , ∥ f 2 ∥ W r , ∞ ≲ ∥ f ∥ W r , ∞ 2 . \lVert fg\rVert_{W^{r,2}}\lesssim\lVert f\rVert_{W^{r,\infty}}\lVert g\rVert_{W^{r,2}},\qquad\lVert f^2\rVert_{W^{r,\infty}}\lesssim\lVert f\rVert_{W^{r,\infty}}^2. ∥ f g ∥ W r , 2 ≲ ∥ f ∥ W r , ∞ ∥ g ∥ W r , 2 , ∥ f 2 ∥ W r , ∞ ≲ ∥ f ∥ W r , ∞ 2 .
Proof. We follow standard multi-index notation. For each multi-index ν \nu ν with ∣ ν ∣ ≤ r |\nu|\le r ∣ ν ∣ ≤ r , the Leibniz product rule and Hölder’s inequality give
∥ ∂ ν ( f g ) ∥ L 2 ≤ ∑ γ ≤ ν ( ν γ ) ∥ ∂ γ f ∥ L ∞ ∥ ∂ ν − γ g ∥ L 2 ≲ ∥ f ∥ W r , ∞ ∥ g ∥ W r , 2 . \lVert\partial^\nu(fg)\rVert_{L^2}\le\sum_{\gamma\le\nu}\binom{\nu}{\gamma}\lVert\partial^\gamma f\rVert_{L^\infty}\lVert\partial^{\nu-\gamma}g\rVert_{L^2}\lesssim\lVert f\rVert_{W^{r,\infty}}\lVert g\rVert_{W^{r,2}}. ∥ ∂ ν ( f g ) ∥ L 2 ≤ γ ≤ ν ∑ ( γ ν ) ∥ ∂ γ f ∥ L ∞ ∥ ∂ ν − γ g ∥ L 2 ≲ ∥ f ∥ W r , ∞ ∥ g ∥ W r , 2 .
Summing the squares over ∣ ν ∣ ≤ r |\nu|\le r ∣ ν ∣ ≤ r gives the first bound. The same argument with L ∞ L^\infty L ∞ in place of L 2 L^2 L 2 and g = f g=f g = f gives the second.
For any fixed observed sample, the initialization, smoothness of β i \beta_i β i and lower bound on the variance for each i i i give μ n , σ n , σ n 2 ∈ H r ( D ) \mu_n,\sigma_n,\sigma_n^2\in H^r(D) μ n , σ n , σ n 2 ∈ H r ( D ) . First, we show that σ i 2 \sigma_i^2 σ i 2 is bounded in L 2 ( H r ( D ) ) L^2(H^r(D)) L 2 ( H r ( D )) and converges a.s. Following (23) , we decompose
σ i 2 = σ i − 1 2 + Δ M i − Δ R i . \sigma_i^2=\sigma_{i-1}^2+\Delta M_i-\Delta R_i. σ i 2 = σ i − 1 2 + Δ M i − Δ R i .
Using E [ ( Z i 2 − 1 ) 2 ] = 2 E[(Z_i^2-1)^2]=2 E [( Z i 2 − 1 ) 2 ] = 2 and E [ Z i 4 ] = 3 E[Z_i^4]=3 E [ Z i 4 ] = 3 , Lemmas C.8 and C.9 give
E [ ∥ Δ M i ∥ H r 2 ∣ F i − 1 , X i ] ≲ i − 2 m ∥ σ i − 1 2 ∥ H r 2 , E[\lVert\Delta M_i\rVert_{H^r}^2\mid\mathcal{F}_{i-1},X_i]\lesssim i^{-2m}\lVert\sigma_{i-1}^2\rVert_{H^r}^2, E [∥ Δ M i ∥ H r 2 ∣ F i − 1 , X i ] ≲ i − 2 m ∥ σ i − 1 2 ∥ H r 2 ,
E [ ∥ Δ R i ∥ H r 2 ∣ F i − 1 , X i ] ≲ i − 4 m ∥ σ i − 1 2 ∥ H r 2 . (25) E[\lVert\Delta R_i\rVert_{H^r}^2\mid\mathcal{F}_{i-1},X_i]\lesssim i^{-4m}\lVert\sigma_{i-1}^2\rVert_{H^r}^2. \tag*{(25)} E [∥ Δ R i ∥ H r 2 ∣ F i − 1 , X i ] ≲ i − 4 m ∥ σ i − 1 2 ∥ H r 2 . (25)
Expanding the squared norm of the update gives
∥ σ i 2 ∥ H r 2 = ∥ σ i − 1 2 ∥ H r 2 + ∥ Δ M i ∥ H r 2 + ∥ Δ R i ∥ H r 2 + 2 ⟨ σ i − 1 2 , Δ M i ⟩ H r − 2 ⟨ σ i − 1 2 , Δ R i ⟩ H r − 2 ⟨ Δ M i , Δ R i ⟩ H r . \begin{aligned}
\lVert\sigma_i^2\rVert_{H^r}^2={}&\lVert\sigma_{i-1}^2\rVert_{H^r}^2+\lVert\Delta M_i\rVert_{H^r}^2+\lVert\Delta R_i\rVert_{H^r}^2\\
&+2\langle\sigma_{i-1}^2,\Delta M_i\rangle_{H^r}-2\langle\sigma_{i-1}^2,\Delta R_i\rangle_{H^r}-2\langle\Delta M_i,\Delta R_i\rangle_{H^r}.
\end{aligned} ∥ σ i 2 ∥ H r 2 = ∥ σ i − 1 2 ∥ H r 2 + ∥ Δ M i ∥ H r 2 + ∥ Δ R i ∥ H r 2 + 2 ⟨ σ i − 1 2 , Δ M i ⟩ H r − 2 ⟨ σ i − 1 2 , Δ R i ⟩ H r − 2 ⟨ Δ M i , Δ R i ⟩ H r .
Since Δ M i \Delta M_i Δ M i has conditional mean 0 0 0 , the term ⟨ σ i − 1 2 , Δ M i ⟩ H r \langle\sigma_{i-1}^2,\Delta M_i\rangle_{H^r} ⟨ σ i − 1 2 , Δ M i ⟩ H r vanishes under the expectation. Applying Cauchy-Schwarz to the remaining cross terms gives
E [ ∥ σ i 2 ∥ H r 2 ∣ F i − 1 , X i ] ≤ ( 1 + C i − 2 m ) ∥ σ i − 1 2 ∥ H r 2 . (26) E[\lVert\sigma_i^2\rVert_{H^r}^2\mid\mathcal{F}_{i-1},X_i]\le(1+Ci^{-2m})\lVert\sigma_{i-1}^2\rVert_{H^r}^2. \tag*{(26)} E [∥ σ i 2 ∥ H r 2 ∣ F i − 1 , X i ] ≤ ( 1 + C i − 2 m ) ∥ σ i − 1 2 ∥ H r 2 . (26)
Taking expectations and iterating thus gives that σ i 2 \sigma_i^2 σ i 2 is bounded in L 2 ( H r ( D ) ) L^2(H^r(D)) L 2 ( H r ( D )) . Convergence of σ i 2 \sigma_i^2 σ i 2 then follows easily from a similar argument to Theorem 1 based on the convergence of the martingale and Cauchy sequence. Taking expectations in (25) and using the fact that σ i 2 \sigma_i^2 σ i 2 is bounded in L 2 ( H r ( D ) ) L^2(H^r(D)) L 2 ( H r ( D )) gives σ i 2 → σ ∞ 2 \sigma_i^2 \to\sigma_\infty^2 σ i 2 → σ ∞ 2 a.s.
The mean function is already a martingale, so we just need to bound
∥ μ i − μ i − 1 ∥ H r 2 = Z i 2 ρ 2 ∥ β i σ i − 1 ∥ H r 2 ≲ Z i 2 i − 2 m ∥ σ i − 1 ∥ H r 2 . \lVert\mu_i-\mu_{i-1}\rVert_{H^r}^2=Z_i^2\rho^2\lVert\beta_i\sigma_{i-1}\rVert_{H^r}^2\lesssim Z_i^2i^{-2m}\lVert\sigma_{i-1}\rVert_{H^r}^2. ∥ μ i − μ i − 1 ∥ H r 2 = Z i 2 ρ 2 ∥ β i σ i − 1 ∥ H r 2 ≲ Z i 2 i − 2 m ∥ σ i − 1 ∥ H r 2 .
The task is now to show σ i − 1 \sigma_{i-1} σ i − 1 is bounded in L 2 ( H r ( D ) ) L^2(H^r(D)) L 2 ( H r ( D )) . We once again use a Taylor expansion argument and a martingale and residual decomposition. To that end, we require the following intermediate lemma.
Lemma C.10. For z ∈ R z\in\mathbb{R} z ∈ R , 0 ≤ a < 1 0\leq a<1 0 ≤ a < 1 and 0 < ρ < 1 0<\rho<1 0 < ρ < 1 , consider the function
g ( a , z ) = 1 + ρ 2 a [ ( 1 − a ) z 2 − 1 ] . g(a,z)=\sqrt{1+\rho^2a[(1-a)z^2-1]}. g ( a , z ) = 1 + ρ 2 a [( 1 − a ) z 2 − 1 ] . A Taylor expansion around a = 0 a=0 a = 0 gives
g ( a , z ) = 1 + 1 2 ρ 2 a ( z 2 − 1 ) + a 2 q ( a , z ) , max 0 ≤ k ≤ r sup a ∈ [ 0 , 1 ) ∣ ∂ k ∂ a k q ( a , z ) ∣ ≤ P r ( z ) g(a,z)=1+\frac{1}{2}\rho^2a(z^2-1)+a^2q(a,z),\qquad\max_{0\leq k\leq r}\sup_{a\in[0,1)}\left|\frac{\partial^k}{\partial a^k}q(a,z)\right|\leq P_r(z) g ( a , z ) = 1 + 2 1 ρ 2 a ( z 2 − 1 ) + a 2 q ( a , z ) , 0 ≤ k ≤ r max a ∈ [ 0 , 1 ) sup ∂ a k ∂ k q ( a , z ) ≤ P r ( z ) for the polynomial P r ( z ) = C r ( 1 + z 2 ) r + 2 P_r(z)=C_r(1+z^2)^{r+2} P r ( z ) = C r ( 1 + z 2 ) r + 2 , with C r C_r C r depending only on r r r and ρ \rho ρ .
Proof. Let h ( a , z ) = 1 + ρ 2 a [ ( 1 − a ) z 2 − 1 ] h(a,z)=1+\rho^2a[(1-a)z^2-1] h ( a , z ) = 1 + ρ 2 a [( 1 − a ) z 2 − 1 ] . Then h ( a , z ) ≥ 1 − ρ 2 > 0 h(a,z)\geq1-\rho^2>0 h ( a , z ) ≥ 1 − ρ 2 > 0 from Lemma C.3 , and its first two derivatives in a a a are bounded in absolute value by 2 ρ 2 ( 1 + z 2 ) 2\rho^2(1+z^2) 2 ρ 2 ( 1 + z 2 ) , with all higher derivatives being 0. Faà di Bruno’s formula therefore gives
sup a ∈ [ 0 , 1 ] ∣ ∂ a j g ( a , z ) ∣ ≤ C j ( 1 + z 2 ) j , 1 ≤ j ≤ r + 2 , \sup_{a\in[0,1]}\left|\partial_a^j g(a,z)\right|\leq C_j(1+z^2)^j,\qquad1\leq j\leq r+2, a ∈ [ 0 , 1 ] sup ∂ a j g ( a , z ) ≤ C j ( 1 + z 2 ) j , 1 ≤ j ≤ r + 2 ,
since the derivatives of the square-root function are bounded on [ 1 − ρ 2 , ∞ ) [1-\rho^2,\infty) [ 1 − ρ 2 , ∞ ) . For a > 0 a>0 a > 0 , Taylor’s formula with integral remainder gives
q ( a , z ) = a − 2 ∫ 0 a ( a − t ) ( ∂ a 2 g ) ( t , z ) d t = ∫ 0 1 ( 1 − t ) ( ∂ a 2 g ) ( t a , z ) d t . q(a,z)=a^{-2}\int_0^a(a-t)(\partial_a^2g)(t,z)\,\mathrm{d}t=\int_0^1(1-t)(\partial_a^2g)(ta,z)\,\mathrm{d}t. q ( a , z ) = a − 2 ∫ 0 a ( a − t ) ( ∂ a 2 g ) ( t , z ) d t = ∫ 0 1 ( 1 − t ) ( ∂ a 2 g ) ( t a , z ) d t .
The latter form is also valid for a = 0 a=0 a = 0 . Differentiating under the integral for 0 ≤ k ≤ r 0\leq k\leq r 0 ≤ k ≤ r , which is permitted since the integrand and its derivatives are jointly continuous for each z z z , gives
∣ ∂ a k q ( a , z ) ∣ = ∣ ∫ 0 1 ( 1 − t ) t k ( ∂ a k + 2 g ) ( t a , z ) d t ∣ ≤ C r ( 1 + z 2 ) r + 2 , \left|\partial_a^kq(a,z)\right|=\left|\int_0^1(1-t)t^k(\partial_a^{k+2}g)(ta,z)\,\mathrm{d}t\right|\leq C_r(1+z^2)^{r+2}, ∂ a k q ( a , z ) = ∫ 0 1 ( 1 − t ) t k ( ∂ a k + 2 g ) ( t a , z ) d t ≤ C r ( 1 + z 2 ) r + 2 ,
which proves the bound.
Now, we can write
σ i = σ i − 1 + 1 2 ρ 2 β i σ i − 1 ( Z i 2 − 1 ) ⏟ A i + β i 2 σ i − 1 q ( β i , Z i ) ⏟ B i . \sigma_i = \sigma_{i-1} + \underbrace{\frac{1}{2}\rho^2\beta_i\sigma_{i-1}(Z_i^2-1)}_{A_i} + \underbrace{\beta_i^2\sigma_{i-1}q(\beta_i,Z_i)}_{B_i}. σ i = σ i − 1 + A i 2 1 ρ 2 β i σ i − 1 ( Z i 2 − 1 ) + B i β i 2 σ i − 1 q ( β i , Z i ) .
Expanding the squared norm gives
∥ σ i ∥ H r 2 = ∥ σ i − 1 ∥ H r 2 + 2 ⟨ σ i − 1 , A i ⟩ H r + 2 ⟨ σ i − 1 , B i ⟩ H r + ∥ A i ∥ H r 2 + 2 ⟨ A i , B i ⟩ H r + ∥ B i ∥ H r 2 . \|\sigma_i\|_{H^r}^2 = \|\sigma_{i-1}\|_{H^r}^2 + 2\langle\sigma_{i-1}, A_i\rangle_{H^r} + 2\langle\sigma_{i-1}, B_i\rangle_{H^r}
+ \|A_i\|_{H^r}^2 + 2\langle A_i, B_i\rangle_{H^r} + \|B_i\|_{H^r}^2. ∥ σ i ∥ H r 2 = ∥ σ i − 1 ∥ H r 2 + 2 ⟨ σ i − 1 , A i ⟩ H r + 2 ⟨ σ i − 1 , B i ⟩ H r + ∥ A i ∥ H r 2 + 2 ⟨ A i , B i ⟩ H r + ∥ B i ∥ H r 2 .
Faà di Bruno’s formula and Lemmas C.8 and C.10 give ∥ q ( β i , Z i ) ∥ W r , ∞ ≲ P r ( Z i ) \|q(\beta_i,Z_i)\|_{W^{r,\infty}} \lesssim P_r(Z_i) ∥ q ( β i , Z i ) ∥ W r , ∞ ≲ P r ( Z i ) . Thus Lemmas C.8 and C.9 give
∥ A i ∥ H r ≲ i − m ∣ Z i 2 − 1 ∣ ∥ σ i − 1 ∥ H r , ∥ B i ∥ H r ≲ i − 2 m P r ( Z i ) ∥ σ i − 1 ∥ H r . \|A_i\|_{H^r} \lesssim i^{-m}|Z_i^2-1|\|\sigma_{i-1}\|_{H^r}, \qquad\|B_i\|_{H^r} \lesssim i^{-2m}P_r(Z_i)\|\sigma_{i-1}\|_{H^r}. ∥ A i ∥ H r ≲ i − m ∣ Z i 2 − 1∣∥ σ i − 1 ∥ H r , ∥ B i ∥ H r ≲ i − 2 m P r ( Z i ) ∥ σ i − 1 ∥ H r .
The term involving ⟨ σ i − 1 , A i ⟩ H r \langle\sigma_{i-1}, A_i\rangle_{H^r} ⟨ σ i − 1 , A i ⟩ H r has conditional expectation zero. Again, Cauchy-Schwarz bounds the other cross terms, and E [ P r ( Z i ) 2 ] E[P_r(Z_i)^2] E [ P r ( Z i ) 2 ] and E [ ( Z i 2 − 1 ) 2 ] E[(Z_i^2-1)^2] E [( Z i 2 − 1 ) 2 ] are finite because of the Gaussian having all moments. We thus have
E [ ∥ σ i ∥ H r 2 ∣ F i − 1 , X i ] ≤ ( 1 + C i − 2 m ) ∥ σ i − 1 ∥ H r 2 . E[\|\sigma_i\|_{H^r}^2 \mid\mathcal{F}_{i-1},X_i] \le(1+Ci^{-2m})\|\sigma_{i-1}\|_{H^r}^2. E [ ∥ σ i ∥ H r 2 ∣ F i − 1 , X i ] ≤ ( 1 + C i − 2 m ) ∥ σ i − 1 ∥ H r 2 .
Taking expectations and iterating then gives sup i ≥ n E [ ∥ σ i ∥ H r 2 ] < ∞ \sup_{i\ge n}E[\|\sigma_i\|_{H^r}^2]<\infty sup i ≥ n E [ ∥ σ i ∥ H r 2 ] < ∞ . We thus have that μ i \mu_i μ i is bounded in L 2 ( H r ( D ) ) L^2(H^r(D)) L 2 ( H r ( D )) and thus converges to some μ ∞ ∈ H r ( D ) \mu_\infty\in H^r(D) μ ∞ ∈ H r ( D ) a.s. Finally, H r ( D ) H^r(D) H r ( D ) convergence implies L 2 ( D ) L^2(D) L 2 ( D ) convergence, so uniqueness of the latter limit identifies σ ∞ 2 \sigma_\infty^2 σ ∞ 2 with the positive limit in Theorem 1.
Proof of Corollary 1
For the Gaussian similarity in Example 1, s ( x , x ′ ) s(x,x') s ( x , x ′ ) is infinitely differentiable jointly in ( x , x ′ ) (x,x') ( x , x ′ ) . Compactness of X \mathcal{X} X therefore gives Assumption 5 for every integer r ≥ 1 r\ge1 r ≥ 1 . Applying Theorem 2 for each r r r and intersecting the resulting countably many probability-one events, uniqueness of the L 2 ( D ) L^2(D) L 2 ( D ) limits gives
μ ∞ , σ ∞ 2 ∈ ⋂ r = 1 ∞ H r ( D ) a.s. \mu_\infty,\sigma_\infty^2\in\bigcap_{r=1}^{\infty}H^r(D)\quad\text{a.s.} μ ∞ , σ ∞ 2 ∈ r = 1 ⋂ ∞ H r ( D ) a.s.
For each integer k ≥ 0 k\ge0 k ≥ 0 , choose an integer r > k + p / 2 r>k+p/2 r > k + p /2 . Sobolev embedding then gives C k ( D ) C^k(D) C k ( D ) representatives of both limits. For each limit, these representatives agree everywhere on D D D , since they are continuous and agree almost everywhere. Hence both limits have C ∞ ( D ) C^\infty(D) C ∞ ( D ) representatives a.s.
Proof of Corollary 2 (Theorem 2 part)
We follow the proof of Theorem 2, using the moment identities and variance decomposition from the proof of Corollary 2 (Theorem 1 part), with fixed δ > 0 \delta> 0 δ > 0 . Lemmas C.4 , C.8 , and C.9 give the same conditional increment bounds as equation (25) . Since Δ M i \Delta M_i Δ M i still has conditional mean 0 0 0 , the same cross-term argument gives (26) , and iteration yields sup i ≥ n E [ ∥ σ i 2 ∥ H r 2 ] < ∞ \sup_{i \ge n} E[\|\sigma_i^2\|_{H^r}^2] < \infty sup i ≥ n E [ ∥ σ i 2 ∥ H r 2 ] < ∞ . The martingale and Cauchy sequence arguments then give
σ i 2 → σ ∞ 2 a.s. in H r ( D ) . \sigma_i^2 \to\sigma_\infty^2 \quad\text{a.s. in } H^r(D). σ i 2 → σ ∞ 2 a.s. in H r ( D ) .
For the mean, we again need a bound on σ i \sigma_i σ i in L 2 ( H r ( D ) ) L^2(H^r(D)) L 2 ( H r ( D )) . The square-root argument requires higher moments, which follow from the same Jensen argument as Lemma C.4 . For every integer k ≥ 1 k \ge1 k ≥ 1 ,
E [ { R δ ( 2 ) ( ε i ) } k ] ≤ E [ ∣ g δ ( W ′ ) ∣ 2 k ] < ∞ , W ′ ∼ N ( 0 , 1 ) , E\left[\left\{R_\delta^{(2)}(\varepsilon_i)\right\}^k\right] \le E[|g_\delta(W')|^{2k}] < \infty,\qquad W' \sim\mathcal{N}(0,1), E [ { R δ ( 2 ) ( ε i ) } k ] ≤ E [ ∣ g δ ( W ′ ) ∣ 2 k ] < ∞ , W ′ ∼ N ( 0 , 1 ) ,
where the finiteness comes from the existence of the Gaussian moment generating function. Using the definition of β i \beta_i β i and the argument of Lemma C.2 , there exists a positive constant b < 1 b < 1 b < 1 such that β i ( x ) < b \beta_i(x) < b β i ( x ) < b uniformly in i ≥ n + 1 i \ge n+1 i ≥ n + 1 and x ∈ X x \in\mathcal{X} x ∈ X . As in Lemma C.5 , R δ ( 1 ) ( ε ) 2 ≤ R δ ( 2 ) ( ε ) R_\delta^{(1)}(\varepsilon)^2 \le R_\delta^{(2)}(\varepsilon) R δ ( 1 ) ( ε ) 2 ≤ R δ ( 2 ) ( ε ) , so for 0 ≤ a < b 0 \le a < b 0 ≤ a < b ,
h ( a , ε ) : = 1 + a { R δ ( 2 ) ( ε ) − 1 } − a 2 R δ ( 1 ) ( ε ) 2 ≥ 1 − b > 0. (27) h(a,\varepsilon) := 1 + a\left\{R_\delta^{(2)}(\varepsilon)-1\right\} - a^2 R_\delta^{(1)}(\varepsilon)^2 \ge1-b > 0. \tag*{(27)} h ( a , ε ) := 1 + a { R δ ( 2 ) ( ε ) − 1 } − a 2 R δ ( 1 ) ( ε ) 2 ≥ 1 − b > 0. (27)
The two derivatives in a a a of h ( a , ε ) h(a,\varepsilon) h ( a , ε ) are bounded in absolute value by a constant times 1 + R δ ( 2 ) ( ε ) 1+R_\delta^{(2)}(\varepsilon) 1 + R δ ( 2 ) ( ε ) , and all higher derivatives vanish. For g ( a , ε ) = h ( a , ε ) g(a,\varepsilon)=\sqrt{h(a,\varepsilon)} g ( a , ε ) = h ( a , ε ) , the same Taylor and Faà di Bruno argument as in Lemma C.10 therefore gives
g ( a , ε ) = 1 + a 2 { R δ ( 2 ) ( ε ) − 1 } + a 2 q ( a , ε ) , max 0 ≤ k ≤ r sup a ∈ [ 0 , b ) ∣ ∂ a k q ( a , ε ) ∣ ≤ C [ 1 + R δ ( 2 ) ( ε ) ] r + 2 . \begin{aligned}
g(a,\varepsilon) &= 1+\frac{a}{2}\left\{R_\delta^{(2)}(\varepsilon)-1\right\}+a^2q(a,\varepsilon),\\
\max_{0\le k\le r}\sup_{a\in[0,b)}\left|\partial_a^kq(a,\varepsilon)\right| &\le C\left[1+R_\delta^{(2)}(\varepsilon)\right]^{r+2}.
\end{aligned} g ( a , ε ) 0 ≤ k ≤ r max a ∈ [ 0 , b ) sup ∂ a k q ( a , ε ) = 1 + 2 a { R δ ( 2 ) ( ε ) − 1 } + a 2 q ( a , ε ) , ≤ C [ 1 + R δ ( 2 ) ( ε ) ] r + 2 .
The bound has finite second moment by the preceding moment calculation. Since E [ R δ ( 2 ) ( ε i ) − 1 ] = 0 E[R_\delta^{(2)}(\varepsilon_i)-1]=0 E [ R δ ( 2 ) ( ε i ) − 1 ] = 0 , the same squared-norm expansion for σ i \sigma_i σ i , now with R δ ( 2 ) ( ε i ) − 1 R_\delta^{(2)}(\varepsilon_i)-1 R δ ( 2 ) ( ε i ) − 1 in place of ρ 2 ( Z i 2 − 1 ) \rho^2(Z_i^2-1) ρ 2 ( Z i 2 − 1 ) , gives
E [ ∥ σ i ∥ H r 2 ∣ F i − 1 , X i ] ≤ ( 1 + C i − 2 m ) ∥ σ i − 1 ∥ H r 2 . E[\|\sigma_i\|_{H^r}^2\mid\mathcal{F}_{i-1},X_i]\le(1+Ci^{-2m})\|\sigma_{i-1}\|_{H^r}^2. E [ ∥ σ i ∥ H r 2 ∣ F i − 1 , X i ] ≤ ( 1 + C i − 2 m ) ∥ σ i − 1 ∥ H r 2 .
The rest of the proof is identical.
Frequentist asymptotics proofs
Notation and preliminary lemmas
In this section, we will assume throughout that ( Y n , X n ) ∼ i i d P ∗ (Y_n, X_n) \overset{\mathrm{iid}}{\sim} P^* ( Y n , X n ) ∼ iid P ∗ for n ≥ 1 n \ge1 n ≥ 1 , and let F n = σ ( Y 1 , X 1 , … , Y n , X n ) \mathcal{F}_n = \sigma(Y_1, X_1, \ldots, Y_n, X_n) F n = σ ( Y 1 , X 1 , … , Y n , X n ) . We note that by Assumption 6, we have
∣ μ ∗ ( x ) ∣ < ∞ , 0 < σ ∗ 2 ( x ) < ∞ for P X ∗ -almost all x , |\mu_*(x)| < \infty,\qquad0 < \sigma_*^2(x) < \infty\quad\text{for } P_X^*\text{-almost all } x, ∣ μ ∗ ( x ) ∣ < ∞ , 0 < σ ∗ 2 ( x ) < ∞ for P X ∗ -almost all x ,
and μ ∗ ∈ L 2 ( P X ∗ ) \mu_* \in L^2(P_X^*) μ ∗ ∈ L 2 ( P X ∗ ) , σ ∗ 2 ∈ L 1 ( P X ∗ ) \sigma_*^2 \in L^1(P_X^*) σ ∗ 2 ∈ L 1 ( P X ∗ ) . Finiteness and integrability follow from
μ ∗ ( x ) 2 ≤ E [ Y 2 ∣ X = x ] , σ ∗ 2 ( x ) ≤ E [ Y 2 ∣ X = x ] , \mu_*(x)^2 \le E[Y^2 \mid X = x],\qquad\sigma_*^2(x) \le E[Y^2 \mid X = x], μ ∗ ( x ) 2 ≤ E [ Y 2 ∣ X = x ] , σ ∗ 2 ( x ) ≤ E [ Y 2 ∣ X = x ] ,
and E [ Y 2 ] < ∞ E[Y^2] < \infty E [ Y 2 ] < ∞ follows from E [ Y 4 ] < ∞ E[Y^4] < \infty E [ Y 4 ] < ∞ . Strict positivity of σ ∗ 2 ( x ) \sigma_*^2(x) σ ∗ 2 ( x ) P X ∗ P_X^* P X ∗ -almost everywhere follows from E [ ∣ log σ ∗ 2 ( X ) ∣ ] < ∞ E[|\log\sigma_*^2(X)|] < \infty E [ ∣ log σ ∗ 2 ( X ) ∣ ] < ∞ .
Error functions
For the single-output case, a key term is the pointwise (random) error function
r n ( y , x ) = y − μ n ( x ) σ n ( x ) r_n(y, x) = \frac{y - \mu_n(x)}{\sigma_n(x)} r n ( y , x ) = σ n ( x ) y − μ n ( x )
for y ∈ R y \in\mathbb{R} y ∈ R and x ∈ X x \in\mathcal{X} x ∈ X . We then have Z n = r n − 1 ( Y n , X n ) Z_n = r_{n-1}(Y_n, X_n) Z n = r n − 1 ( Y n , X n ) , where Z n Z_n Z n is not necessarily distributed as N ( 0 , 1 ) \mathcal{N}(0, 1) N ( 0 , 1 ) , and the error functions are clearly equal to
u n − 1 ( x ) = E [ Z n ∣ F n − 1 , X n = x ] , w n − 1 ( x ) = E [ Z n 2 − 1 ∣ F n − 1 , X n = x ] . (28) u_{n-1}(x) = E[Z_n \mid\mathcal{F}_{n-1}, X_n = x],\qquad w_{n-1}(x) = E[Z_n^2 - 1 \mid\mathcal{F}_{n-1}, X_n = x]. \tag*{(28)} u n − 1 ( x ) = E [ Z n ∣ F n − 1 , X n = x ] , w n − 1 ( x ) = E [ Z n 2 − 1 ∣ F n − 1 , X n = x ] . (28)
For convenience, we will also define the intermediate term
v n ( x ) = σ ∗ 2 ( x ) σ n 2 ( x ) . v_n(x) = \frac{\sigma_*^2(x)}{\sigma_n^2(x)}. v n ( x ) = σ n 2 ( x ) σ ∗ 2 ( x ) .
Since 0 ≤ β i ( x ) < 1 0 \le\beta_i(x) < 1 0 ≤ β i ( x ) < 1 , the initialization and updates imply that μ n \mu_n μ n and σ n 2 \sigma_n^2 σ n 2 are bounded on X \mathcal{X} X for each n n n , with inf x ∈ X σ n 2 ( x ) ≥ ( 1 − ρ 2 ) n > 0 \inf_{x\in\mathcal{X}} \sigma_n^2(x) \ge(1-\rho^2)^n > 0 inf x ∈ X σ n 2 ( x ) ≥ ( 1 − ρ 2 ) n > 0 . Together with the integrability of μ ∗ \mu_* μ ∗ and σ ∗ 2 \sigma_*^2 σ ∗ 2 established earlier, this gives u n ∈ L 2 ( P X ∗ ) u_n \in L^2(P_X^*) u n ∈ L 2 ( P X ∗ ) and w n ∈ L 1 ( P X ∗ ) w_n \in L^1(P_X^*) w n ∈ L 1 ( P X ∗ ) for every finite n n n P ∗ P^* P ∗ -a.s. All subsequent identities involving these error functions are understood to hold P X ∗ P_X^* P X ∗ -almost everywhere.
Learning rates
Next, we have the following uniform upper bounds to handle β n ( x ) \beta_n(x) β n ( x ) .
Lemma D.1. Under Assumptions 1 and 7, we have
β n 2 ( x ) ≲ α n 2 , ∣ β n ( x ) − α n s ( x , X n ) ∣ ≲ α n 2 \beta_n^2(x) \lesssim\alpha_n^2,\quad|\beta_n(x)-\alpha_n s(x,X_n)| \lesssim\alpha_n^2 β n 2 ( x ) ≲ α n 2 , ∣ β n ( x ) − α n s ( x , X n ) ∣ ≲ α n 2 where the constants are uniform over n n n , x x x and X n X_n X n .
Proof. The first part follows from the same argument as Lemma C.2. For the second,
β n ( x ) − α n s ( x , X n ) = α n 2 s ( x , X n ) { 1 − s ( x , X n ) } 1 − α n + α n s ( x , X n ) . \beta_n(x)-\alpha_n s(x,X_n)=\alpha_n^2\frac{s(x,X_n)\{1-s(x,X_n)\}}{1-\alpha_n+\alpha_n s(x,X_n)}. β n ( x ) − α n s ( x , X n ) = α n 2 1 − α n + α n s ( x , X n ) s ( x , X n ) { 1 − s ( x , X n )} . The denominator is bounded away from 0, and Assumption 1 gives the result.
Similarity norm
Let S \mathcal{S} S denote the integral operator,
( S f ) ( x ) = ∫ f ( x ′ ) s ( x , x ′ ) d P X ∗ ( x ′ ) , (\mathcal{S}f)(x)=\int f(x')s(x,x')\,dP_X^*(x'), ( S f ) ( x ) = ∫ f ( x ′ ) s ( x , x ′ ) d P X ∗ ( x ′ ) ,
and let ⟨ f , g ⟩ s \langle f,g\rangle_s ⟨ f , g ⟩ s denote the bilinear form
⟨ f , g ⟩ s = ∫ f ( x ) g ( x ′ ) s ( x , x ′ ) d P X ∗ ( x ) d P X ∗ ( x ′ ) . \langle f,g\rangle_s=\int f(x)g(x')s(x,x')\,dP_X^*(x)\,dP_X^*(x'). ⟨ f , g ⟩ s = ∫ f ( x ) g ( x ′ ) s ( x , x ′ ) d P X ∗ ( x ) d P X ∗ ( x ′ ) .
For shorthand, we will write L ∗ 1 = L 1 ( P X ∗ ) L_*^1=L^1(P_X^*) L ∗ 1 = L 1 ( P X ∗ ) , with the respective norm ∥ f ∥ L ∗ 1 = ∫ ∣ f ( x ) ∣ d P X ∗ ( x ) \|f\|_{L_*^1}=\int|f(x)|\,dP_X^*(x) ∥ f ∥ L ∗ 1 = ∫ ∣ f ( x ) ∣ d P X ∗ ( x ) . We now demonstrate that the function ∥ ⋅ ∥ s \|\cdot\|_s ∥ ⋅ ∥ s is a norm.
Lemma D.2. Under Assumption 1, ∥ ⋅ ∥ s : L ∗ 1 → R \|\cdot\|_s:L_*^1\to\mathbb{R} ∥ ⋅ ∥ s : L ∗ 1 → R is a seminorm, and
∥ S f ∥ ∞ ≲ ∥ f ∥ s , ∣ ⟨ f , g ⟩ s ∣ ≲ ∥ f ∥ L ∗ 1 ∥ g ∥ L ∗ 1 . \|\mathcal{S}f\|_\infty\lesssim\|f\|_s,\quad|\langle f,g\rangle_s|\lesssim\|f\|_{L_*^1}\|g\|_{L_*^1}. ∥ S f ∥ ∞ ≲ ∥ f ∥ s , ∣ ⟨ f , g ⟩ s ∣ ≲ ∥ f ∥ L ∗ 1 ∥ g ∥ L ∗ 1 . If additionally Assumptions 6, 8 and 9 hold, then ∥ ⋅ ∥ s \|\cdot\|_s ∥ ⋅ ∥ s is a norm.
Proof. Let f f f and g g g be functions in L ∗ 1 L_*^1 L ∗ 1 . We begin with the bilinear form result:
∣ ⟨ f , g ⟩ s ∣ ≤ ∫ ∫ ∣ f ( x ) ∣ ∣ g ( x ′ ) ∣ ∣ s ( x , x ′ ) ∣ d P X ∗ ( x ) d P X ∗ ( x ′ ) ≤ S ∥ f ∥ L ∗ 1 ∥ g ∥ L ∗ 1 |\langle f,g\rangle_s|\leq\int\int|f(x)||g(x')||s(x,x')|\,dP_X^*(x)\,dP_X^*(x')\leq S\|f\|_{L_*^1}\|g\|_{L_*^1} ∣ ⟨ f , g ⟩ s ∣ ≤ ∫∫ ∣ f ( x ) ∣∣ g ( x ′ ) ∣∣ s ( x , x ′ ) ∣ d P X ∗ ( x ) d P X ∗ ( x ′ ) ≤ S ∥ f ∥ L ∗ 1 ∥ g ∥ L ∗ 1
by Lemma C.1. Next, the kernel form of s ( x , x ′ ) s(x,x') s ( x , x ′ ) immediately gives us non-negativity, as Fubini’s theorem gives
∥ f ∥ s 2 = ∫ ( η f ) ( ψ ) 2 d Π ( ψ ) ≥ 0 , ( η f ) ( ψ ) = ∫ f ( x ) k ( x ; ψ ) E Ψ [ k ( x ; Ψ ) ] d P X ∗ ( x ) . (29) \|f\|_{s}^{2}=\int(\eta f)(\psi)^{2}\,d\Pi(\psi)\ge0,\qquad(\eta f)(\psi)=\int\frac{f(x)k(x;\psi)}{E_{\Psi}[k(x;\Psi)]}\,dP_{X}^{*}(x). \tag*{(29)} ∥ f ∥ s 2 = ∫ ( η f ) ( ψ ) 2 d Π ( ψ ) ≥ 0 , ( η f ) ( ψ ) = ∫ E Ψ [ k ( x ; Ψ )] f ( x ) k ( x ; ψ ) d P X ∗ ( x ) . (29)
The scaling property of a norm holds immediately. For the triangle property,
∣ ⟨ f , g ⟩ s ∣ = ∣ ∫ ( η f ) ( ψ ) ( η g ) ( ψ ) d Π ( ψ ) ∣ ≤ ∥ f ∥ s ∥ g ∥ s , |\langle f,g\rangle_{s}|=\left|\int(\eta f)(\psi)(\eta g)(\psi)\,d\Pi(\psi)\right|\le\|f\|_{s}\|g\|_{s}, ∣ ⟨ f , g ⟩ s ∣ = ∫ ( η f ) ( ψ ) ( η g ) ( ψ ) d Π ( ψ ) ≤ ∥ f ∥ s ∥ g ∥ s ,
where we have used Cauchy-Schwarz. Next, we note that
∥ f + g ∥ s 2 = ∥ f ∥ s 2 + ∥ g ∥ s 2 + 2 ⟨ f , g ⟩ s ≤ ∥ f ∥ s 2 + ∥ g ∥ s 2 + 2 ∥ f ∥ s ∥ g ∥ s = ( ∥ f ∥ s + ∥ g ∥ s ) 2 . \begin{aligned}
\|f+g\|_{s}^{2}&=\|f\|_{s}^{2}+\|g\|_{s}^{2}+2\langle f,g\rangle_{s}\\
&\le\|f\|_{s}^{2}+\|g\|_{s}^{2}+2\|f\|_{s}\|g\|_{s}=(\|f\|_{s}+\|g\|_{s})^{2}.
\end{aligned} ∥ f + g ∥ s 2 = ∥ f ∥ s 2 + ∥ g ∥ s 2 + 2 ⟨ f , g ⟩ s ≤ ∥ f ∥ s 2 + ∥ g ∥ s 2 + 2∥ f ∥ s ∥ g ∥ s = ( ∥ f ∥ s + ∥ g ∥ s ) 2 .
Taking the square root then gives the triangle inequality, so ∥ ⋅ ∥ s \|\cdot\|_{s} ∥ ⋅ ∥ s is a seminorm. For the first inequality, we note that
( S f ) ( x ) = ∫ k ( x ; ψ ) E Ψ [ k ( x ; Ψ ) ] ( η f ) ( ψ ) d Π ( ψ ) , (Sf)(x)=\int\frac{k(x;\psi)}{E_{\Psi}[k(x;\Psi)]}(\eta f)(\psi)\,d\Pi(\psi), ( S f ) ( x ) = ∫ E Ψ [ k ( x ; Ψ )] k ( x ; ψ ) ( η f ) ( ψ ) d Π ( ψ ) ,
so Cauchy-Schwarz applied to the above gives
( S f ) ( x ) 2 ≤ ∫ [ k ( x ; ψ ) E Ψ [ k ( x ; Ψ ) ] ] 2 d Π ( ψ ) ∫ ( η f ) ( ψ ′ ) 2 d Π ( ψ ′ ) = s ( x , x ) ∥ f ∥ s 2 . (Sf)(x)^{2}\le\int\left[\frac{k(x;\psi)}{E_{\Psi}[k(x;\Psi)]}\right]^{2}\,d\Pi(\psi)\int(\eta f)(\psi')^{2}\,d\Pi(\psi')=s(x,x)\|f\|_{s}^{2}. ( S f ) ( x ) 2 ≤ ∫ [ E Ψ [ k ( x ; Ψ )] k ( x ; ψ ) ] 2 d Π ( ψ ) ∫ ( η f ) ( ψ ′ ) 2 d Π ( ψ ′ ) = s ( x , x ) ∥ f ∥ s 2 .
Taking the supremum of both sides gives ∥ ( S f ) 2 ∥ ∞ ≤ S ∥ f ∥ s 2 \|(Sf)^{2}\|_{\infty}\le S\|f\|_{s}^{2} ∥ ( S f ) 2 ∥ ∞ ≤ S ∥ f ∥ s 2 .
For the norm result, let ∥ f ∥ s = 0 \|f\|_{s}=0 ∥ f ∥ s = 0 . By (29) , we have ( η f ) ( ψ ) 2 = 0 (\eta f)(\psi)^{2}=0 ( η f ) ( ψ ) 2 = 0 for Π \Pi Π -almost all ψ \psi ψ , which can be strengthened to hold for all ψ ∈ supp ( Π ) \psi\in\operatorname{supp}(\Pi) ψ ∈ supp ( Π ) if η f \eta f η f is continuous. Let ψ j → ψ \psi_j\to\psi ψ j → ψ be some convergent sequence, and note that
∣ f ( x ) k ( x ; ψ j ) E Ψ [ k ( x ; Ψ ) ] ∣ ≤ sup x ∈ X , j ≥ 1 ∣ k ( x ; ψ j ) ∣ inf x ∈ X E Ψ [ k ( x ; Ψ ) ] ∣ f ( x ) ∣ ≲ ∣ f ( x ) ∣ . \left|\frac{f(x)k(x;\psi_j)}{E_{\Psi}[k(x;\Psi)]}\right|\le\frac{\sup_{x\in\mathcal{X},j\ge1}|k(x;\psi_j)|}{\inf_{x\in\mathcal{X}}E_{\Psi}[k(x;\Psi)]}|f(x)|\lesssim|f(x)|. E Ψ [ k ( x ; Ψ )] f ( x ) k ( x ; ψ j ) ≤ inf x ∈ X E Ψ [ k ( x ; Ψ )] sup x ∈ X , j ≥ 1 ∣ k ( x ; ψ j ) ∣ ∣ f ( x ) ∣ ≲ ∣ f ( x ) ∣.
where the denominator is bounded away from 0 by Assumption 8 , and joint continuity of k k k , compactness of X \mathcal{X} X and convergence of ψ j \psi_j ψ j ensure the numerator is finite. Since f ∈ L ∗ 1 f\in L_{*}^{1} f ∈ L ∗ 1 and k ( x ; ψ ) k(x;\psi) k ( x ; ψ ) is jointly continuous, we can apply dominated convergence to show ( η f ) ( ψ j ) → ( η f ) ( ψ ) (\eta f)(\psi_j)\to(\eta f)(\psi) ( η f ) ( ψ j ) → ( η f ) ( ψ ) , so η f \eta f η f is continuous and thus ( η f ) ( ψ ) = 0 (\eta f)(\psi)=0 ( η f ) ( ψ ) = 0 for all ψ ∈ supp ( Π ) \psi\in\operatorname{supp}(\Pi) ψ ∈ supp ( Π ) . By completeness and Tonelli’s theorem, we then have
0 = ∫ [ ∫ R p { 1 ( x ∈ X ) ∣ f ( x ) ∣ p X ∗ ( x ) E Ψ [ k ( x ; Ψ ) ] } k ( x ; ψ ) d x ] d Π ( ψ ) = ∥ f ∥ L ∗ 1 . 0=\int\left[\int_{\mathbb{R}^{p}}\left\{\mathbf{1}(x\in\mathcal{X})\frac{|f(x)|p_{X}^{*}(x)}{E_{\Psi}[k(x;\Psi)]}\right\}k(x;\psi)\,dx\right]d\Pi(\psi)=\|f\|_{L_{*}^{1}}. 0 = ∫ [ ∫ R p { 1 ( x ∈ X ) E Ψ [ k ( x ; Ψ )] ∣ f ( x ) ∣ p X ∗ ( x ) } k ( x ; ψ ) d x ] d Π ( ψ ) = ∥ f ∥ L ∗ 1 .
where completeness ensures the inner integral is 0 for each ψ \psi ψ . We thus have f ( x ) = 0 f(x)=0 f ( x ) = 0 for P X ∗ P_X^* P X ∗ -almost all x x x . The converse direction is immediate.
Proof of Theorem 3
We will now study the convergence of the energy K n = ∫ K n X ( x ) d P X ∗ ( x ) K_n=\int K_n^X(x)\,\mathrm{d}P_X^*(x) K n = ∫ K n X ( x ) d P X ∗ ( x ) where
K n X = 2 KL [ N ( μ ∗ , σ ∗ 2 ) , N ( μ n , σ n 2 ) ] = w n − log v n , K_n^X=2\operatorname{KL}\left[\mathcal{N}\left(\mu_*,\sigma_*^2\right),\mathcal{N}\left(\mu_n,\sigma_n^2\right)\right]=w_n-\log v_n, K n X = 2 KL [ N ( μ ∗ , σ ∗ 2 ) , N ( μ n , σ n 2 ) ] = w n − log v n ,
where KL ( f , g ) \operatorname{KL}(f,g) KL ( f , g ) is the Kullback-Leibler (KL) divergence between the densities f f f and g g g . Note that we are not measuring the KL divergence from the true p ∗ ( y ∣ x ) p^*(y\mid x) p ∗ ( y ∣ x ) . The above choice of K n X K_n^X K n X is especially compatible with our moment updates, allowing us to utilize the almost-supermartingale theory of Robbins and Siegmund [1971]. We note that the factor 2 is for convenience, and that Assumptions 3 and 6 give
K 0 = E [ Y 2 ] − 1 − E [ log σ ∗ 2 ( X ) ] < ∞ . K_0=E[Y^2]-1-E[\log\sigma_*^2(X)]<\infty. K 0 = E [ Y 2 ] − 1 − E [ log σ ∗ 2 ( X )] < ∞.
While we will show the convergence of K n K_n K n to a finite (random) limit P ∗ P^* P ∗ -a.s., it is not necessarily 0. The key term that we will show goes to 0 P ∗ P^* P ∗ -a.s. is
T n = 2 ρ ∥ u n ∥ s 2 + ρ 2 ∥ w n ∥ s 2 , T_n=2\rho\lVert u_n\rVert_s^2+\rho^2\lVert w_n\rVert_s^2, T n = 2 ρ ∥ u n ∥ s 2 + ρ 2 ∥ w n ∥ s 2 ,
where clearly T n ≥ 0 T_n\ge0 T n ≥ 0 from Lemma D.2.
Technical lemmas
In this section, we outline some useful technical lemmas. First, we show a lemma on the growth of r n ( y , x ) r_n(y,x) r n ( y , x ) , which we use to control the residual terms in the almost-supermartingale. The assumptions are slightly stronger than needed but convenient.
Lemma D.3. Under Assumptions 1, 3 and 7, the following holds for all ( y , x ) ∈ R × X (y,x)\in\mathbb{R}\times\mathcal{X} ( y , x ) ∈ R × X :
r n 2 ( y , x ) ≲ [ 1 + r 0 2 ( y , x ) ] ( n + 1 ) α 0 ρ 2 s . r_n^2(y,x)\lesssim[1+r_0^2(y,x)](n+1)^{\alpha_0\rho^2s}. r n 2 ( y , x ) ≲ [ 1 + r 0 2 ( y , x )] ( n + 1 ) α 0 ρ 2 s . Proof. Let r ∈ R r\in\mathbb{R} r ∈ R , z ∈ R z\in\mathbb{R} z ∈ R , 0 < ρ < 1 0<\rho<1 0 < ρ < 1 and 0 ≤ a < 1 0\le a<1 0 ≤ a < 1 . Consider the term
h ( z ) = ( r − ρ a z ) 2 1 + ρ 2 a [ ( 1 − a ) z 2 − 1 ] . h(z)=\frac{(r-\rho az)^2}{1+\rho^2a[(1-a)z^2-1]}. h ( z ) = 1 + ρ 2 a [( 1 − a ) z 2 − 1 ] ( r − ρ a z ) 2 .
Differentiating with respect to z z z gives
h ′ ( z ) = − 2 ρ a ( r − ρ a z ) { 1 − ρ 2 a + ρ ( 1 − a ) r z } { 1 + ρ 2 a [ ( 1 − a ) z 2 − 1 ] } 2 . h'(z) = \frac{-2\rho a(r-\rho az)\left\{1-\rho^{2}a+\rho(1-a)rz\right\}}{\left\{1+\rho^{2}a\left[(1-a)z^{2}-1\right]\right\}^{2}}. h ′ ( z ) = { 1 + ρ 2 a [ ( 1 − a ) z 2 − 1 ] } 2 − 2 ρ a ( r − ρ a z ) { 1 − ρ 2 a + ρ ( 1 − a ) r z } .
For r ≠ 0 r \ne0 r = 0 , we can solve for h ′ ( z ) = 0 h'(z)=0 h ′ ( z ) = 0 and verify that h ( z ) h(z) h ( z ) is maximized at
z ∗ = − 1 − ρ 2 a ρ ( 1 − a ) r z_*=-\frac{1-\rho^{2}a}{\rho(1-a)r} z ∗ = − ρ ( 1 − a ) r 1 − ρ 2 a
which when substituted gives the upper bound
h ( z ) ≤ h ( z ∗ ) = r 2 1 − ρ 2 a + a 1 − a . h(z) \le h(z_*) = \frac{r^{2}}{1-\rho^{2}a}+\frac{a}{1-a}. h ( z ) ≤ h ( z ∗ ) = 1 − ρ 2 a r 2 + 1 − a a .
When r = 0 r=0 r = 0 , we have
h ( z ) = a 1 − a ρ 2 a ( 1 − a ) z 2 1 + ρ 2 a [ ( 1 − a ) z 2 − 1 ] ≤ a 1 − a h(z)=\frac{a}{1-a}\frac{\rho^{2}a(1-a)z^{2}}{1+\rho^{2}a\left[(1-a)z^{2}-1\right]}\le\frac{a}{1-a} h ( z ) = 1 − a a 1 + ρ 2 a [ ( 1 − a ) z 2 − 1 ] ρ 2 a ( 1 − a ) z 2 ≤ 1 − a a
so the upper bound holds for all r ∈ R r\in\mathbb{R} r ∈ R .
Substituting r = r n − 1 ( y , x ) r=r_{n-1}(y,x) r = r n − 1 ( y , x ) , z = Z n z=Z_n z = Z n and a = β n ( x ) a=\beta_n(x) a = β n ( x ) then gives the recursion
r n 2 ( y , x ) ≤ r n − 1 2 ( y , x ) 1 − ρ 2 β n ( x ) + β n ( x ) 1 − β n ( x ) . r_n^{2}(y,x)\le\frac{r_{n-1}^{2}(y,x)}{1-\rho^{2}\beta_n(x)}+\frac{\beta_n(x)}{1-\beta_n(x)}. r n 2 ( y , x ) ≤ 1 − ρ 2 β n ( x ) r n − 1 2 ( y , x ) + 1 − β n ( x ) β n ( x ) .
Iterating the above gives
r n 2 ( y , x ) ≤ r 0 2 ( y , x ) ∏ i = 1 n 1 1 − ρ 2 β i ( x ) + ∑ i = 1 n β i ( x ) 1 − β i ( x ) ∏ j = i + 1 n 1 1 − ρ 2 β j ( x ) . r_n^{2}(y,x)\le r_0^{2}(y,x)\prod_{i=1}^{n}\frac{1}{1-\rho^{2}\beta_i(x)}+\sum_{i=1}^{n}\frac{\beta_i(x)}{1-\beta_i(x)}\prod_{j=i+1}^{n}\frac{1}{1-\rho^{2}\beta_j(x)}. r n 2 ( y , x ) ≤ r 0 2 ( y , x ) i = 1 ∏ n 1 − ρ 2 β i ( x ) 1 + i = 1 ∑ n 1 − β i ( x ) β i ( x ) j = i + 1 ∏ n 1 − ρ 2 β j ( x ) 1 .
Next, the definition of β j ( x ) \beta_j(x) β j ( x ) gives
β j ( x ) 1 − ρ 2 β j ( x ) ≤ β j ( x ) 1 − β j ( x ) ≤ α 0 S j + 1 − α 0 . \frac{\beta_j(x)}{1-\rho^{2}\beta_j(x)}\le\frac{\beta_j(x)}{1-\beta_j(x)}\le\frac{\alpha_0 S}{j+1-\alpha_0}. 1 − ρ 2 β j ( x ) β j ( x ) ≤ 1 − β j ( x ) β j ( x ) ≤ j + 1 − α 0 α 0 S .
The product term then satisfies the following for 0 ≤ i ≤ n 0\le i\le n 0 ≤ i ≤ n :
∏ j = i + 1 n 1 1 − ρ 2 β j ( x ) ≤ exp { ∑ j = i + 1 n ρ 2 β j ( x ) 1 − ρ 2 β j ( x ) } ≤ ( n + 1 i + 1 − α 0 ) α 0 ρ 2 S . \prod_{j=i+1}^{n}\frac{1}{1-\rho^{2}\beta_j(x)}
\le\exp\left\{\sum_{j=i+1}^{n}\frac{\rho^{2}\beta_j(x)}{1-\rho^{2}\beta_j(x)}\right\}
\le\left(\frac{n+1}{i+1-\alpha_0}\right)^{\alpha_0\rho^{2}S}. j = i + 1 ∏ n 1 − ρ 2 β j ( x ) 1 ≤ exp { j = i + 1 ∑ n 1 − ρ 2 β j ( x ) ρ 2 β j ( x ) } ≤ ( i + 1 − α 0 n + 1 ) α 0 ρ 2 S .
Together, this gives
r n 2 ( y , x ) ≲ r 0 2 ( y , x ) ( n + 1 ) α 0 ρ 2 S + α 0 S ( n + 1 ) α 0 ρ 2 S ∑ i = 1 n i − ( 1 + α 0 ρ 2 S ) r_n^2(y,x) \lesssim r_0^2(y,x)(n+1)^{\alpha_0\rho^2S}+\alpha_0S(n+1)^{\alpha_0\rho^2S}\sum_{i=1}^{n}i^{-(1+\alpha_0\rho^2S)} r n 2 ( y , x ) ≲ r 0 2 ( y , x ) ( n + 1 ) α 0 ρ 2 S + α 0 S ( n + 1 ) α 0 ρ 2 S i = 1 ∑ n i − ( 1 + α 0 ρ 2 S )
≲ [ 1 + r 0 2 ( y , x ) ] ( n + 1 ) α 0 ρ 2 S \lesssim[1+r_0^2(y,x)](n+1)^{\alpha_0\rho^2S} ≲ [ 1 + r 0 2 ( y , x )] ( n + 1 ) α 0 ρ 2 S
since ∑ i = 1 ∞ i − ( 1 + α 0 ρ 2 S ) < ∞ \sum_{i=1}^{\infty}i^{-(1+\alpha_0\rho^2S)}<\infty ∑ i = 1 ∞ i − ( 1 + α 0 ρ 2 S ) < ∞ , and Assumption 3 ensures r 0 ( y , x ) r_0(y,x) r 0 ( y , x ) is well-defined.
We first collect bounds on the standardized mean/variance increments. Let
Δ μ = ρ β n Z n , Δ σ 2 = β n ρ 2 { ( 1 − β n ) Z n 2 − 1 } . \Delta_\mu=\rho\beta_n Z_n,\qquad\Delta_{\sigma^2}=\beta_n\rho^2\{(1-\beta_n)Z_n^2-1\}. Δ μ = ρ β n Z n , Δ σ 2 = β n ρ 2 {( 1 − β n ) Z n 2 − 1 } .
so that μ n = μ n − 1 + σ n − 1 Δ μ \mu_n=\mu_{n-1}+\sigma_{n-1}\Delta_\mu μ n = μ n − 1 + σ n − 1 Δ μ and σ n 2 = σ n − 1 2 ( 1 + Δ σ 2 ) \sigma_n^2=\sigma_{n-1}^2(1+\Delta_{\sigma^2}) σ n 2 = σ n − 1 2 ( 1 + Δ σ 2 ) . We then have the following.
Lemma D.4. Under Assumptions 1, 3, 6 and 7, we have
∣ Δ μ ∣ ≲ α n ∣ Z n ∣ , ∣ Δ σ 2 ∣ ≲ α n ( 1 + Z n 2 ) . (30) |\Delta_\mu|\lesssim\alpha_n|Z_n|,\qquad|\Delta_{\sigma^2}|\lesssim\alpha_n(1+Z_n^2). \tag*{(30)} ∣ Δ μ ∣ ≲ α n ∣ Z n ∣ , ∣ Δ σ 2 ∣ ≲ α n ( 1 + Z n 2 ) . (30) Furthermore, we have
∣ E [ Δ μ ∣ F n − 1 ] − α n ρ S u n − 1 ∣ ≲ α n 2 E [ ( 1 + Z n 2 ) 2 ∣ F n − 1 ] , (31) \left|\mathbb{E}[\Delta_\mu\mid\mathcal{F}_{n-1}]-\alpha_n\rho\,\mathcal{S}u_{n-1}\right|\lesssim\alpha_n^2\mathbb{E}[(1+Z_n^2)^2\mid\mathcal{F}_{n-1}], \tag*{(31)} ∣ E [ Δ μ ∣ F n − 1 ] − α n ρ S u n − 1 ∣ ≲ α n 2 E [( 1 + Z n 2 ) 2 ∣ F n − 1 ] , (31) ∣ E [ Δ σ 2 ∣ F n − 1 ] − α n ρ 2 S w n − 1 ∣ ≲ α n 2 E [ ( 1 + Z n 2 ) 2 ∣ F n − 1 ] . (32) \left|\mathbb{E}[\Delta_{\sigma^2}\mid\mathcal{F}_{n-1}]-\alpha_n\rho^2\,\mathcal{S}w_{n-1}\right|\lesssim\alpha_n^2\mathbb{E}[(1+Z_n^2)^2\mid\mathcal{F}_{n-1}]. \tag*{(32)} E [ Δ σ 2 ∣ F n − 1 ] − α n ρ 2 S w n − 1 ≲ α n 2 E [( 1 + Z n 2 ) 2 ∣ F n − 1 ] . (32)
Proof. Since 0 ≤ β n < 1 0\le\beta_n<1 0 ≤ β n < 1 , Lemma D.1 immediately gives (30) and
∣ Δ μ − α n s ( ⋅ , X n ) ρ Z n ∣ ≲ α n 2 ( 1 + Z n 2 ) 2 , |\Delta_\mu-\alpha_ns(\cdot,X_n)\rho Z_n|\lesssim\alpha_n^2(1+Z_n^2)^2, ∣ Δ μ − α n s ( ⋅ , X n ) ρ Z n ∣ ≲ α n 2 ( 1 + Z n 2 ) 2 ,
∣ Δ σ 2 − α n s ( ⋅ , X n ) ρ 2 ( Z n 2 − 1 ) ∣ ≲ α n 2 ( 1 + Z n 2 ) 2 , |\Delta_{\sigma^2}-\alpha_ns(\cdot,X_n)\rho^2(Z_n^2-1)|\lesssim\alpha_n^2(1+Z_n^2)^2, ∣ Δ σ 2 − α n s ( ⋅ , X n ) ρ 2 ( Z n 2 − 1 ) ∣ ≲ α n 2 ( 1 + Z n 2 ) 2 ,
where we used Δ σ 2 = ρ 2 β n ( Z n 2 − 1 ) − ρ 2 β n 2 Z n 2 \Delta_{\sigma^2}=\rho^2\beta_n(Z_n^2-1)-\rho^2\beta_n^2Z_n^2 Δ σ 2 = ρ 2 β n ( Z n 2 − 1 ) − ρ 2 β n 2 Z n 2 for the second inequality. We use the common upper bound ( 1 + Z n 2 ) 2 (1+Z_n^2)^2 ( 1 + Z n 2 ) 2 for convenience. Next, (28) gives
E [ s ( ⋅ , X n ) Z n ∣ F n − 1 ] = S u n − 1 , E [ s ( ⋅ , X n ) ( Z n 2 − 1 ) ∣ F n − 1 ] = S w n − 1 , \mathbb{E}[s(\cdot,X_n)Z_n\mid\mathcal{F}_{n-1}]=\mathcal{S}u_{n-1},\qquad\mathbb{E}[s(\cdot,X_n)(Z_n^2-1)\mid\mathcal{F}_{n-1}]=\mathcal{S}w_{n-1}, E [ s ( ⋅ , X n ) Z n ∣ F n − 1 ] = S u n − 1 , E [ s ( ⋅ , X n ) ( Z n 2 − 1 ) ∣ F n − 1 ] = S w n − 1 ,
and Jensen’s inequality gives (31) and (32).
Next, we show a key technical lemma that controls the change in the error functions w n w_n w n and u n u_n u n .
Lemma D.5. Under Assumptions 1, 3, 6 and 7 , we have
u n − u n − 1 = − Δ μ − 1 2 Δ σ 2 u n − 1 + R n u , u_n-u_{n-1}=-\Delta_\mu-\frac{1}{2}\Delta_{\sigma^2}u_{n-1}+R_n^u, u n − u n − 1 = − Δ μ − 2 1 Δ σ 2 u n − 1 + R n u , w n − w n − 1 = − 2 Δ μ u n − 1 − Δ σ 2 ( 1 + w n − 1 ) + R n w , w_n-w_{n-1}=-2\Delta_\mu u_{n-1}-\Delta_{\sigma^2}(1+w_{n-1})+R_n^w, w n − w n − 1 = − 2 Δ μ u n − 1 − Δ σ 2 ( 1 + w n − 1 ) + R n w , where ∣ R n w ∣ + ∣ R n u ∣ ≲ α n 2 ( 1 + Z n 2 ) 2 ( 1 + ∣ u n − 1 ∣ + ∣ w n − 1 ∣ ) |R_n^w|+|R_n^u|\lesssim\alpha_n^2(1+Z_n^2)^2(1+|u_{n-1}|+|w_{n-1}|) ∣ R n w ∣ + ∣ R n u ∣ ≲ α n 2 ( 1 + Z n 2 ) 2 ( 1 + ∣ u n − 1 ∣ + ∣ w n − 1 ∣ ) . Furthermore ,
∣ u n − u n − 1 ∣ + ∣ w n − w n − 1 ∣ ≲ α n ( 1 + Z n 2 ) ( 1 + ∣ u n − 1 ∣ + ∣ w n − 1 ∣ ) . |u_n-u_{n-1}|+|w_n-w_{n-1}|\lesssim\alpha_n(1+Z_n^2)(1+|u_{n-1}|+|w_{n-1}|). ∣ u n − u n − 1 ∣ + ∣ w n − w n − 1 ∣ ≲ α n ( 1 + Z n 2 ) ( 1 + ∣ u n − 1 ∣ + ∣ w n − 1 ∣ ) .
Proof. We start with w n w_n w n . The updates give
w n − w n − 1 = − 2 Δ μ u n − 1 + Δ μ 2 − Δ σ 2 ( 1 + w n − 1 ) 1 + Δ σ 2 = − 2 Δ μ u n − 1 − Δ σ 2 ( 1 + w n − 1 ) + R n w . (33) \begin{aligned}
w_n-w_{n-1}=\frac{-2\Delta_\mu u_{n-1}+\Delta_\mu^2-\Delta_{\sigma^2}(1+w_{n-1})}{1+\Delta_{\sigma^2}} \\
&=-2\Delta_\mu u_{n-1}-\Delta_{\sigma^2}(1+w_{n-1})+R_n^w. \tag*{(33)}
\end{aligned} w n − w n − 1 = 1 + Δ σ 2 − 2 Δ μ u n − 1 + Δ μ 2 − Δ σ 2 ( 1 + w n − 1 ) = − 2 Δ μ u n − 1 − Δ σ 2 ( 1 + w n − 1 ) + R n w . (33)
where
R n w = Δ μ 2 + Δ σ 2 2 ( 1 + w n − 1 ) + 2 Δ μ Δ σ 2 u n − 1 1 + Δ σ 2 . R_n^w=\frac{\Delta_\mu^2+\Delta_{\sigma^2}^2(1+w_{n-1})+2\Delta_\mu\Delta_{\sigma^2}u_{n-1}}{1+\Delta_{\sigma^2}}. R n w = 1 + Δ σ 2 Δ μ 2 + Δ σ 2 2 ( 1 + w n − 1 ) + 2 Δ μ Δ σ 2 u n − 1 .
Using 1 + Δ σ 2 ≥ 1 − ρ 2 1+\Delta_{\sigma^2}\geq1-\rho^2 1 + Δ σ 2 ≥ 1 − ρ 2 from Lemma C.3 and the triangle inequality gives
∣ R n w ∣ ≤ Δ μ 2 + Δ σ 2 2 ( 1 + ∣ w n − 1 ∣ ) + 2 ∣ Δ μ Δ σ 2 ∣ ∣ u n − 1 ∣ 1 − ρ 2 |R_n^w|\leq\frac{\Delta_\mu^2+\Delta_{\sigma^2}^2(1+|w_{n-1}|)+2|\Delta_\mu\Delta_{\sigma^2}||u_{n-1}|}{1-\rho^2} ∣ R n w ∣ ≤ 1 − ρ 2 Δ μ 2 + Δ σ 2 2 ( 1 + ∣ w n − 1 ∣ ) + 2∣ Δ μ Δ σ 2 ∣∣ u n − 1 ∣
≲ α n 2 ( 1 + Z n 2 ) 2 ( 1 + ∣ u n − 1 ∣ + ∣ w n − 1 ∣ ) , \lesssim\alpha_n^2(1+Z_n^2)^2(1+|u_{n-1}|+|w_{n-1}|), ≲ α n 2 ( 1 + Z n 2 ) 2 ( 1 + ∣ u n − 1 ∣ + ∣ w n − 1 ∣ ) ,
where the last line follows from (30). For the first-difference bound, (33) gives
∣ w n − w n − 1 ∣ ≤ 2 ∣ Δ μ ∣ ∣ u n − 1 ∣ + Δ μ 2 + ∣ Δ σ 2 ∣ ( 1 + ∣ w n − 1 ∣ ) 1 − ρ 2 |w_n-w_{n-1}|\leq\frac{2|\Delta_\mu||u_{n-1}|+\Delta_\mu^2+|\Delta_{\sigma^2}|(1+|w_{n-1}|)}{1-\rho^2} ∣ w n − w n − 1 ∣ ≤ 1 − ρ 2 2∣ Δ μ ∣∣ u n − 1 ∣ + Δ μ 2 + ∣ Δ σ 2 ∣ ( 1 + ∣ w n − 1 ∣ )
≲ α n ( 1 + Z n 2 ) ( 1 + ∣ u n − 1 ∣ + ∣ w n − 1 ∣ ) , \lesssim\alpha_n(1+Z_n^2)(1+|u_{n-1}|+|w_{n-1}|), ≲ α n ( 1 + Z n 2 ) ( 1 + ∣ u n − 1 ∣ + ∣ w n − 1 ∣ ) ,
using (30) and Δ μ 2 ≲ α n 2 Z n 2 ≤ α n Z n 2 \Delta_\mu^2\lesssim\alpha_n^2Z_n^2\leq\alpha_nZ_n^2 Δ μ 2 ≲ α n 2 Z n 2 ≤ α n Z n 2 . Next, for u n u_n u n , we have
u n − u n − 1 = u n − 1 − Δ μ 1 + Δ σ 2 − u n − 1 = − Δ μ − 1 2 Δ σ 2 u n − 1 + R n u , u_n-u_{n-1}=\frac{u_{n-1}-\Delta_\mu}{\sqrt{1+\Delta_{\sigma^2}}}-u_{n-1}=-\Delta_\mu-\frac{1}{2}\Delta_{\sigma^2}u_{n-1}+R_n^u, u n − u n − 1 = 1 + Δ σ 2 u n − 1 − Δ μ − u n − 1 = − Δ μ − 2 1 Δ σ 2 u n − 1 + R n u ,
where
R n u = u n − 1 ( 1 1 + Δ σ 2 − 1 + 1 2 Δ σ 2 ) − Δ μ ( 1 1 + Δ σ 2 − 1 ) . R_n^u=u_{n-1}\left(\frac{1}{\sqrt{1+\Delta_{\sigma^2}}}-1+\frac{1}{2}\Delta_{\sigma^2}\right)-\Delta_\mu\left(\frac{1}{\sqrt{1+\Delta_{\sigma^2}}}-1\right). R n u = u n − 1 ( 1 + Δ σ 2 1 − 1 + 2 1 Δ σ 2 ) − Δ μ ( 1 + Δ σ 2 1 − 1 ) .
::: Taylor’s theorem gives
∣ 1 1 + t − 1 ∣ ≲ ∣ t ∣ , ∣ 1 1 + t − 1 + t 2 ∣ ≲ t 2 \left|\frac{1}{\sqrt{1+t}}-1\right|\lesssim|t|,\qquad\left|\frac{1}{\sqrt{1+t}}-1+\frac{t}{2}\right|\lesssim t^2 1 + t 1 − 1 ≲ ∣ t ∣ , 1 + t 1 − 1 + 2 t ≲ t 2
for t ≥ − ρ 2 > − 1 t\ge-\rho^2>-1 t ≥ − ρ 2 > − 1 , with constants depending on ρ \rho ρ . Thus (30) gives
∣ R n u ∣ ≲ ∣ u n − 1 ∣ ∣ Δ σ 2 ∣ 2 + ∣ Δ μ ∣ ∣ Δ σ 2 ∣ ≲ α n 2 ( 1 + Z n 2 ) ( 1 + ∣ u n − 1 ∣ + ∣ w n − 1 ∣ ) . |R_n^u|\lesssim|u_{n-1}||\Delta_{\sigma^2}|^2+|\Delta_\mu||\Delta_{\sigma^2}|\lesssim\alpha_n^2(1+Z_n^2)(1+|u_{n-1}|+|w_{n-1}|). ∣ R n u ∣ ≲ ∣ u n − 1 ∣∣ Δ σ 2 ∣ 2 + ∣ Δ μ ∣∣ Δ σ 2 ∣ ≲ α n 2 ( 1 + Z n 2 ) ( 1 + ∣ u n − 1 ∣ + ∣ w n − 1 ∣ ) .
Finally, writing
u n − u n − 1 = u n − 1 ( 1 1 + Δ σ 2 − 1 ) − Δ μ 1 + Δ σ 2 u_n-u_{n-1}=u_{n-1}\left(\frac{1}{\sqrt{1+\Delta_{\sigma^2}}}-1\right)-\frac{\Delta_\mu}{\sqrt{1+\Delta_{\sigma^2}}} u n − u n − 1 = u n − 1 ( 1 + Δ σ 2 1 − 1 ) − 1 + Δ σ 2 Δ μ
and using the same Taylor bound and 1 + Δ σ 2 ≥ 1 − ρ 2 1+\Delta_{\sigma^2}\ge1-\rho^2 1 + Δ σ 2 ≥ 1 − ρ 2 gives
∣ u n − u n − 1 ∣ ≲ ∣ u n − 1 ∣ ∣ Δ σ 2 ∣ + ∣ Δ μ ∣ ≲ α n ( 1 + Z n 2 ) ( 1 + ∣ u n − 1 ∣ ) . |u_n-u_{n-1}|\lesssim|u_{n-1}||\Delta_{\sigma^2}|+|\Delta_\mu|\lesssim\alpha_n(1+Z_n^2)(1+|u_{n-1}|). ∣ u n − u n − 1 ∣ ≲ ∣ u n − 1 ∣∣ Δ σ 2 ∣ + ∣ Δ μ ∣ ≲ α n ( 1 + Z n 2 ) ( 1 + ∣ u n − 1 ∣ ) .
Finally, we provide a useful upper bound on the error terms.
Lemma D.6. The error terms satisfy
1 + ∣ u n ∣ + ∣ w n ∣ ≲ 1 + K n X . 1+|u_n|+|w_n|\lesssim1+K_n^X. 1 + ∣ u n ∣ + ∣ w n ∣ ≲ 1 + K n X . Proof. Since log v ≤ v / 2 \log v\le v/2 log v ≤ v /2 for v > 0 v>0 v > 0 , we have
1 + K n X = u n 2 + v n − log v n ≥ u n 2 + v n 2 . 1+K_n^X=u_n^2+v_n-\log v_n\ge u_n^2+\frac{v_n}{2}. 1 + K n X = u n 2 + v n − log v n ≥ u n 2 + 2 v n . Using ∣ u n ∣ ≤ 1 + u n 2 |u_n|\le1+u_n^2 ∣ u n ∣ ≤ 1 + u n 2 and ∣ w n ∣ = ∣ u n 2 + v n − 1 ∣ ≤ 1 + u n 2 + v n |w_n|=|u_n^2+v_n-1|\le1+u_n^2+v_n ∣ w n ∣ = ∣ u n 2 + v n − 1∣ ≤ 1 + u n 2 + v n gives
1 + ∣ u n ∣ + ∣ w n ∣ ≲ 1 + u n 2 + v n ≲ 1 + K n X 1+|u_n|+|w_n|\lesssim1+u_n^2+v_n\lesssim1+K_n^X 1 + ∣ u n ∣ + ∣ w n ∣ ≲ 1 + u n 2 + v n ≲ 1 + K n X where the last inequality uses K n X ≥ 0 K_n^X\ge0 K n X ≥ 0 .
Main proof
We are now ready to show the main convergence result.
Theorem D.1. Under Assumptions 1, 3, 6, 7, the updates (5) and (6) give
K n → K ∞ < ∞ and ∑ n = 1 ∞ α n T n − 1 < ∞ P ∗ -a.s. K_n \to K_\infty< \infty\quad\text{and} \quad\sum_{n=1}^{\infty} \alpha_n T_{n-1} < \infty\quad P^*\text{-a.s.} K n → K ∞ < ∞ and n = 1 ∑ ∞ α n T n − 1 < ∞ P ∗ -a.s.
Proof. Using log ( 1 + t ) ≤ t \log(1+t) \leq t log ( 1 + t ) ≤ t for t > − 1 t > -1 t > − 1 and Lemma D.5, we have
K n X − K n − 1 X = log ( 1 + Δ σ 2 ) + ( w n − w n − 1 ) ≤ Δ σ 2 + ( w n − w n − 1 ) = − 2 Δ μ u n − 1 − Δ σ 2 w n − 1 + R n w . \begin{aligned}
K_n^X - K_{n-1}^X = \log(1+\Delta_{\sigma^2}) + (w_n-w_{n-1}) \\
&\leq\Delta_{\sigma^2} + (w_n-w_{n-1}) = -2\Delta_\mu u_{n-1} - \Delta_{\sigma^2}w_{n-1} + R_n^w.
\end{aligned} K n X − K n − 1 X = log ( 1 + Δ σ 2 ) + ( w n − w n − 1 ) ≤ Δ σ 2 + ( w n − w n − 1 ) = − 2 Δ μ u n − 1 − Δ σ 2 w n − 1 + R n w .
We now take the conditional expectation and integrate over P X ∗ P_X^* P X ∗ . Applying (31) and (32), the leading terms integrate to − α n T n − 1 -\alpha_n T_{n-1} − α n T n − 1 . The combined error is bounded by
C α n 2 E [ ( 1 + Z n 2 ) 2 ∣ F n − 1 ] ∥ 1 + ∣ u n − 1 ∣ + ∣ w n − 1 ∣ ∥ L ∗ 1 , C\alpha_n^2 E[(1+Z_n^2)^2 \mid\mathcal{F}_{n-1}]\|1+|u_{n-1}|+|w_{n-1}|\|_{L_*^1}, C α n 2 E [( 1 + Z n 2 ) 2 ∣ F n − 1 ] ∥1 + ∣ u n − 1 ∣ + ∣ w n − 1 ∣ ∥ L ∗ 1 ,
following Lemmas D.5 and D.4. Next, Lemma D.6 and ∥ K n − 1 X ∥ L ∗ 1 = K n − 1 \|K_{n-1}^X\|_{L_*^1}=K_{n-1} ∥ K n − 1 X ∥ L ∗ 1 = K n − 1 give
E [ K n ∣ F n − 1 ] − K n − 1 ≤ − α n T n − 1 + C α n 2 E [ ( 1 + Z n 2 ) 2 ∣ F n − 1 ] ( 1 + K n − 1 ) . E[K_n\mid\mathcal{F}_{n-1}]-K_{n-1}\leq-\alpha_nT_{n-1}+C\alpha_n^2E[(1+Z_n^2)^2\mid\mathcal{F}_{n-1}](1+K_{n-1}). E [ K n ∣ F n − 1 ] − K n − 1 ≤ − α n T n − 1 + C α n 2 E [( 1 + Z n 2 ) 2 ∣ F n − 1 ] ( 1 + K n − 1 ) .
Next, Lemma D.3 and the initialization from Assumption 3 give
Z n 2 ≲ ( 1 + Y n 2 ) n α 0 ρ 2 S . Z_n^2\lesssim(1+Y_n^2)n^{\alpha_0\rho^2 S}. Z n 2 ≲ ( 1 + Y n 2 ) n α 0 ρ 2 S .
Assumption 6 then gives E [ ( 1 + Y n 2 ) 2 ] < ∞ E[(1+Y_n^2)^2]<\infty E [( 1 + Y n 2 ) 2 ] < ∞ , so
α n 2 E [ ( 1 + Z n 2 ) 2 ∣ F n − 1 ] ≲ n − 2 ( 1 − α 0 ρ 2 S ) , \alpha_n^2 E[(1+Z_n^2)^2\mid\mathcal{F}_{n-1}]\lesssim n^{-2(1-\alpha_0\rho^2 S)}, α n 2 E [( 1 + Z n 2 ) 2 ∣ F n − 1 ] ≲ n − 2 ( 1 − α 0 ρ 2 S ) ,
and under the upper bound on α 0 < min { 1 , 1 / ( 2 ρ 2 S ) } \alpha_0 < \min\{1,1/(2\rho^2S)\} α 0 < min { 1 , 1/ ( 2 ρ 2 S )} from Assumption 7, we have
∑ n = 1 ∞ α n 2 E [ ( 1 + Z n 2 ) 2 ∣ F n − 1 ] < ∞ . \sum_{n=1}^{\infty}\alpha_n^2 E[(1+Z_n^2)^2\mid\mathcal{F}_{n-1}]<\infty. n = 1 ∑ ∞ α n 2 E [( 1 + Z n 2 ) 2 ∣ F n − 1 ] < ∞.
From the seminorm property in Lemma D.2, T n ≥ 0 T_n\geq0 T n ≥ 0 , and clearly K n ≥ 0 K_n\geq0 K n ≥ 0 and K 0 < ∞ K_0<\infty K 0 < ∞ . We can thus apply the almost-supermartingale convergence theorem of Robbins and Siegmund [1971], giving us the final result. □
While the above theorem shows that lim inf n T n = 0 \liminf_n T_n=0 lim inf n T n = 0 P ∗ P^* P ∗ -a.s. since ∑ n = 1 ∞ α n = ∞ \sum_{n=1}^{\infty}\alpha_n=\infty ∑ n = 1 ∞ α n = ∞ , we now show convergence of T n T_n T n in order to strengthen this to T n → 0 T_n\to0 T n → 0 P ∗ P^* P ∗ -a.s.
Theorem D.2. Under Assumptions 1, 3, 6, 7, the updates (5) and (6) give
T n → 0 P ∗ -a.s. T_n\to0 \quad P^*\text{-a.s.} T n → 0 P ∗ -a.s.
Proof. We will use a second almost-supermartingale. Let Δ u : = u n − u n − 1 \Delta_u := u_n-u_{n-1} Δ u := u n − u n − 1 and Δ w : = w n − w n − 1 \Delta_w := w_n-w_{n-1} Δ w := w n − w n − 1 . First, we write
T n − T n − 1 = 4 ρ ⟨ u n − 1 , Δ u ⟩ s + 2 ρ 2 ⟨ w n − 1 , Δ w ⟩ s + 2 ρ ∥ Δ u ∥ s 2 + ρ 2 ∥ Δ w ∥ s 2 . T_n-T_{n-1}=4\rho\langle u_{n-1},\Delta_u\rangle_s+2\rho^2\langle w_{n-1},\Delta_w\rangle_s+2\rho\|\Delta_u\|_s^2+\rho^2\|\Delta_w\|_s^2. T n − T n − 1 = 4 ρ ⟨ u n − 1 , Δ u ⟩ s + 2 ρ 2 ⟨ w n − 1 , Δ w ⟩ s + 2 ρ ∥ Δ u ∥ s 2 + ρ 2 ∥ Δ w ∥ s 2 .
We first handle the remainder terms, starting with R n u R_n^u R n u . Lemmas D.2 and D.5 give
∣ ⟨ u n − 1 , R n u ⟩ s ∣ ≲ α n 2 ( 1 + Z n 2 ) 2 ∥ 1 + ∣ u n − 1 ∣ + ∣ w n − 1 ∣ ∥ L ∗ 1 ∥ u n − 1 ∥ L ∗ 1 \left|\langle u_{n-1},R_n^u\rangle_s\right|\lesssim\alpha_n^2(1+Z_n^2)^2\|1+|u_{n-1}|+|w_{n-1}|\|_{L_*^1}\|u_{n-1}\|_{L_*^1} ∣ ⟨ u n − 1 , R n u ⟩ s ∣ ≲ α n 2 ( 1 + Z n 2 ) 2 ∥1 + ∣ u n − 1 ∣ + ∣ w n − 1 ∣ ∥ L ∗ 1 ∥ u n − 1 ∥ L ∗ 1
≲ α n 2 ( 1 + Z n 2 ) 2 ( 1 + K n − 1 ) 2 , \lesssim\alpha_n^2(1+Z_n^2)^2(1+K_{n-1})^2, ≲ α n 2 ( 1 + Z n 2 ) 2 ( 1 + K n − 1 ) 2 ,
where the second inequality follows from Lemma D.6. Since ∥ Δ u ∥ s 2 = ⟨ Δ u , Δ u ⟩ s \|\Delta_u\|_s^2=\langle\Delta_u,\Delta_u\rangle_s ∥ Δ u ∥ s 2 = ⟨ Δ u , Δ u ⟩ s , the second part of Lemma D.5 gives a similar bound. Handling ⟨ w n − 1 , R n w ⟩ s \langle w_{n-1},R_n^w\rangle_s ⟨ w n − 1 , R n w ⟩ s and ∥ Δ w ∥ s 2 \|\Delta_w\|_s^2 ∥ Δ w ∥ s 2 in the same way gives
∣ ⟨ u n − 1 , R n u ⟩ s ∣ + ∣ ⟨ w n − 1 , R n w ⟩ s ∣ + ∥ Δ u ∥ s 2 + ∥ Δ w ∥ s 2 ≲ α n 2 ( 1 + Z n 2 ) 2 ( 1 + K n − 1 ) 2 . \left|\langle u_{n-1},R_n^u\rangle_s\right|+\left|\langle w_{n-1},R_n^w\rangle_s\right|+\|\Delta_u\|_s^2+\|\Delta_w\|_s^2\lesssim\alpha_n^2(1+Z_n^2)^2(1+K_{n-1})^2. ∣ ⟨ u n − 1 , R n u ⟩ s ∣ + ∣ ⟨ w n − 1 , R n w ⟩ s ∣ + ∥ Δ u ∥ s 2 + ∥ Δ w ∥ s 2 ≲ α n 2 ( 1 + Z n 2 ) 2 ( 1 + K n − 1 ) 2 .
For the leading terms, Lemma D.5 gives
E [ Δ w ∣ F n − 1 ] = − 2 u n − 1 E [ Δ μ ∣ F n − 1 ] − ( 1 + w n − 1 ) E [ Δ σ 2 ∣ F n − 1 ] + E [ R n w ∣ F n − 1 ] , E [ Δ u ∣ F n − 1 ] = − E [ Δ μ ∣ F n − 1 ] − 1 2 u n − 1 E [ Δ σ 2 ∣ F n − 1 ] + E [ R n u ∣ F n − 1 ] . \begin{aligned}
E[\Delta_w\mid\mathcal{F}_{n-1}]&=-2u_{n-1}E[\Delta_\mu\mid\mathcal{F}_{n-1}]-(1+w_{n-1})E[\Delta_{\sigma^2}\mid\mathcal{F}_{n-1}]+E[R_n^w\mid\mathcal{F}_{n-1}],\\
E[\Delta_u\mid\mathcal{F}_{n-1}]&=-E[\Delta_\mu\mid\mathcal{F}_{n-1}]-\frac{1}{2}u_{n-1}E[\Delta_{\sigma^2}\mid\mathcal{F}_{n-1}]+E[R_n^u\mid\mathcal{F}_{n-1}].
\end{aligned} E [ Δ w ∣ F n − 1 ] E [ Δ u ∣ F n − 1 ] = − 2 u n − 1 E [ Δ μ ∣ F n − 1 ] − ( 1 + w n − 1 ) E [ Δ σ 2 ∣ F n − 1 ] + E [ R n w ∣ F n − 1 ] , = − E [ Δ μ ∣ F n − 1 ] − 2 1 u n − 1 E [ Δ σ 2 ∣ F n − 1 ] + E [ R n u ∣ F n − 1 ] .
Substituting (31) and (32) introduces additional errors bounded pointwise by
C α n 2 E [ ( 1 + Z n 2 ) 2 ∣ F n − 1 ] ( 1 + ∣ u n − 1 ∣ + ∣ w n − 1 ∣ ) , C\alpha_n^2E[(1+Z_n^2)^2\mid\mathcal{F}_{n-1}](1+|u_{n-1}|+|w_{n-1}|), C α n 2 E [( 1 + Z n 2 ) 2 ∣ F n − 1 ] ( 1 + ∣ u n − 1 ∣ + ∣ w n − 1 ∣ ) ,
and their inner products with u n − 1 u_{n-1} u n − 1 and w n − 1 w_{n-1} w n − 1 can be controlled as for the other remainders. Handling E [ ( 1 + Z n 2 ) 2 ∣ F n − 1 ] E[(1+Z_n^2)^2\mid\mathcal{F}_{n-1}] E [( 1 + Z n 2 ) 2 ∣ F n − 1 ] as in Theorem D.1, we have
E [ T n ∣ F n − 1 ] − T n − 1 ≤ − 4 α n ρ 2 ∫ ( S u n − 1 ) 2 d P X ∗ − 2 α n ρ 4 ∫ ( 1 + w n − 1 ) ( S w n − 1 ) 2 d P X ∗ − 6 α n ρ 3 ∫ u n − 1 ( S w n − 1 ) ( S u n − 1 ) d P X ∗ + C n − 2 ( 1 − α 0 ρ 2 S ) ( 1 + K n − 1 ) 2 . (34) \begin{aligned}
E[T_n\mid\mathcal{F}_{n-1}]-T_{n-1}
&\le-4\alpha_n\rho^2\int(Su_{n-1})^2\,dP_X^*\\
&\quad-2\alpha_n\rho^4\int(1+w_{n-1})(Sw_{n-1})^2\,dP_X^*\\
&\quad-6\alpha_n\rho^3\int u_{n-1}(Sw_{n-1})(Su_{n-1})\,dP_X^*\\
&\quad+Cn^{-2(1-\alpha_0\rho^2S)}(1+K_{n-1})^2. \tag*{(34)}
\end{aligned} E [ T n ∣ F n − 1 ] − T n − 1 ≤ − 4 α n ρ 2 ∫ ( S u n − 1 ) 2 d P X ∗ − 2 α n ρ 4 ∫ ( 1 + w n − 1 ) ( S w n − 1 ) 2 d P X ∗ − 6 α n ρ 3 ∫ u n − 1 ( S w n − 1 ) ( S u n − 1 ) d P X ∗ + C n − 2 ( 1 − α 0 ρ 2 S ) ( 1 + K n − 1 ) 2 . (34)
The first two terms in the above are non-positive, so we only need summability of the remaining two terms. The last term is handled by Assumption 7 as before and by $K_n\to K ∞ < ∞ K_{\infty} < \infty K ∞ < ∞ P ∗ P^* P ∗ -a.s. from Theorem D.1. For the third term (34) , we have
∣ ∫ u n − 1 ( S w n − 1 ) ( S u n − 1 ) d P X ∗ ∣ ≤ ∥ u n − 1 ∥ L ∗ 1 ∥ S w n − 1 ∥ ∞ ∥ S u n − 1 ∥ ∞ ≤ ( 1 + K n − 1 ) ∥ u n − 1 ∥ s ∥ w n − 1 ∥ s \begin{aligned}
\left|\int u_{n-1}(\mathcal{S}w_{n-1})(\mathcal{S}u_{n-1})\,dP_X^*\right| \le\|u_{n-1}\|_{L^1_*}\|\mathcal{S}w_{n-1}\|_{\infty}\|\mathcal{S}u_{n-1}\|_{\infty} \\
&\le(1+K_{n-1})\|u_{n-1}\|_s\|w_{n-1}\|_s
\end{aligned} ∫ u n − 1 ( S w n − 1 ) ( S u n − 1 ) d P X ∗ ≤ ∥ u n − 1 ∥ L ∗ 1 ∥ S w n − 1 ∥ ∞ ∥ S u n − 1 ∥ ∞ ≤ ( 1 + K n − 1 ) ∥ u n − 1 ∥ s ∥ w n − 1 ∥ s
≲ ( 1 + K n − 1 ) T n − 1 , \lesssim(1+K_{n-1})T_{n-1}, ≲ ( 1 + K n − 1 ) T n − 1 ,
where the second line follows from Lemmas D.6 and D.2, and the last line follows from 2 ∣ a b ∣ ≤ a 2 + b 2 2|ab| \le a^2+b^2 2∣ ab ∣ ≤ a 2 + b 2 . Since ∑ n = 1 ∞ α n T n − 1 < ∞ \sum_{n=1}^{\infty}\alpha_nT_{n-1}<\infty ∑ n = 1 ∞ α n T n − 1 < ∞ and K n → K ∞ < ∞ K_n\to K_{\infty}<\infty K n → K ∞ < ∞ P ∗ P^* P ∗ -a.s., the absolute value of (34) is also summable. We can then apply the almost-supermartingale result again to show T n → T ∞ T_n\to T_{\infty} T n → T ∞ P ∗ P^* P ∗ -a.s. Finally, because lim inf n T n = 0 \liminf_n T_n=0 lim inf n T n = 0 , we must have T n → 0 T_n\to0 T n → 0 P ∗ P^* P ∗ -a.s.
It is clear that T n → 0 T_n\to0 T n → 0 implies ∥ u n ∥ s → 0 \|u_n\|_s\to0 ∥ u n ∥ s → 0 and ∥ w n ∥ s → 0 \|w_n\|_s\to0 ∥ w n ∥ s → 0 . The final statement then follows from ∥ ⋅ ∥ s \|\cdot\|_s ∥ ⋅ ∥ s being a norm from Lemma D.2.
Proof of Corollary 3
For the SU case, the proof is a natural extension of the previous section, where we rely on 1 − ρ i = O ( α i ) 1-\rho_i=O(\alpha_i) 1 − ρ i = O ( α i ) to approximate the SU update with the Gaussian update at ρ = 1 \rho=1 ρ = 1 . The main difference is that
Δ μ = β n R δ ( 1 ) ( ε n ) , Δ σ 2 = β n { R δ ( 2 ) ( ε n ) − β n R δ ( 1 ) ( ε n ) 2 − 1 } \Delta_{\mu}=\beta_n R_{\delta}^{(1)}(\varepsilon_n),\qquad\Delta_{\sigma^2}=\beta_n\left\{R_{\delta}^{(2)}(\varepsilon_n)-\beta_nR_{\delta}^{(1)}(\varepsilon_n)^2-1\right\} Δ μ = β n R δ ( 1 ) ( ε n ) , Δ σ 2 = β n { R δ ( 2 ) ( ε n ) − β n R δ ( 1 ) ( ε n ) 2 − 1 }
Note that both R δ ( 1 ) ( ε n ) R_{\delta}^{(1)}(\varepsilon_n) R δ ( 1 ) ( ε n ) and R δ ( 2 ) ( ε n ) R_{\delta}^{(2)}(\varepsilon_n) R δ ( 2 ) ( ε n ) now depend on a changing ρ n \rho_n ρ n , but we use the same notation as the main paper for convenience. The definitions of r n ( y , x ) r_n(y,x) r n ( y , x ) , u n u_n u n , w n w_n w n , and K n K_n K n all remain identical, and we set ρ = 1 \rho=1 ρ = 1 for T n T_n T n so T n = 2 ∥ u n ∥ s 2 + ∥ w n ∥ s 2 T_n=2\|u_n\|_s^2+\|w_n\|_s^2 T n = 2∥ u n ∥ s 2 + ∥ w n ∥ s 2 .
We begin by showing that Lemma D.3 extends to this setting. First,
r n 2 = ( r n − 1 − Δ μ ) 2 1 + Δ σ 2 r_n^2=\frac{(r_{n-1}-\Delta_{\mu})^2}{1+\Delta_{\sigma^2}} r n 2 = 1 + Δ σ 2 ( r n − 1 − Δ μ ) 2
where we omit the dependence on ( y , x ) (y,x) ( y , x ) . Since R δ ( 1 ) ( ε n ) 2 ≤ R δ ( 2 ) ( ε n ) R_{\delta}^{(1)}(\varepsilon_n)^2\le R_{\delta}^{(2)}(\varepsilon_n) R δ ( 1 ) ( ε n ) 2 ≤ R δ ( 2 ) ( ε n ) ,
r n 2 ≤ ( r n − 1 − a z ) 2 1 + a [ ( 1 − a ) z 2 − 1 ] r_n^2\le\frac{(r_{n-1}-az)^2}{1+a\left[(1-a)z^2-1\right]} r n 2 ≤ 1 + a [ ( 1 − a ) z 2 − 1 ] ( r n − 1 − a z ) 2
where z = R δ ( 1 ) ( ε n ) z=R_{\delta}^{(1)}(\varepsilon_n) z = R δ ( 1 ) ( ε n ) and a = β n a=\beta_n a = β n . The right-hand side is exactly h ( z ) h(z) h ( z ) with ρ = 1 \rho=1 ρ = 1 , and the rest of the proof is identical as it only relies on a < 1 a<1 a < 1 and not on ρ < 1 \rho<1 ρ < 1 , so
r n 2 ( y , x ) ≲ [ 1 + r 0 2 ( y , x ) ] ( n + 1 ) α 0 S . (35) r_n^2(y,x)\lesssim\left[1+r_0^2(y,x)\right](n+1)^{\alpha_0^S}. \tag*{(35)} r n 2 ( y , x ) ≲ [ 1 + r 0 2 ( y , x ) ] ( n + 1 ) α 0 S . (35)
Next, we will need to obtain the equivalent of Lemma D.4. As a reminder, we have Z n = r n − 1 ( Y n , X n ) Z_n = r_{n-1}(Y_n, X_n) Z n = r n − 1 ( Y n , X n ) . The following technical lemma is the main new ingredient.
Lemma D.7. For 0 < ρ n ≤ 1 0 < \rho_n \le1 0 < ρ n ≤ 1 , we have
∣ R δ ( 1 ) ( ε n ) ∣ ≲ ∣ Z n ∣ , R δ ( 2 ) ( ε n ) ≲ 1 + Z n 2 . \left|R_{\delta}^{(1)}(\varepsilon_n)\right| \lesssim|Z_n|,\qquad R_{\delta}^{(2)}(\varepsilon_n) \lesssim1 + Z_n^2. R δ ( 1 ) ( ε n ) ≲ ∣ Z n ∣ , R δ ( 2 ) ( ε n ) ≲ 1 + Z n 2 . Furthermore, we have
∣ R δ ( 1 ) ( ε n ) − Z n ∣ + ∣ R δ ( 2 ) ( ε n ) − Z n 2 ∣ ≲ ( 1 − ρ n ) ( 1 + Z n 2 ) 2 . \left|R_{\delta}^{(1)}(\varepsilon_n) - Z_n\right| + \left|R_{\delta}^{(2)}(\varepsilon_n) - Z_n^2\right| \lesssim(1-\rho_n)(1+Z_n^2)^2. R δ ( 1 ) ( ε n ) − Z n + R δ ( 2 ) ( ε n ) − Z n 2 ≲ ( 1 − ρ n ) ( 1 + Z n 2 ) 2 . Proof. Since ∣ sinh ( ρ n t ) ∣ ≤ ∣ sinh t ∣ |\sinh(\rho_n t)| \le|\sinh t| ∣ sinh ( ρ n t ) ∣ ≤ ∣ sinh t ∣ for any t ∈ R t \in\mathbb{R} t ∈ R and 0 < ρ n ≤ 1 0 < \rho_n \le1 0 < ρ n ≤ 1 , we have
∣ R δ ( 1 ) ( ε n ) ∣ ≤ ω ( 1 − ρ n 2 ) / 2 D δ ∣ D δ Z n ∣ ≤ ω ∣ Z n ∣ , \left|R_{\delta}^{(1)}(\varepsilon_n)\right| \le\frac{\omega^{(1-\rho_n^2)/2}}{D_\delta}|D_\delta Z_n| \le\sqrt{\omega}|Z_n|, R δ ( 1 ) ( ε n ) ≤ D δ ω ( 1 − ρ n 2 ) /2 ∣ D δ Z n ∣ ≤ ω ∣ Z n ∣ , where D δ = 1 2 ( ω 2 − 1 ) D_\delta= \sqrt{\frac{1}{2}(\omega^2-1)} D δ = 2 1 ( ω 2 − 1 ) and ω > 1 \omega> 1 ω > 1 . Similarly, cosh ( 2 ρ n t ) ≤ cosh ( 2 t ) \cosh(2\rho_n t) \le\cosh(2t) cosh ( 2 ρ n t ) ≤ cosh ( 2 t ) and cosh ( 2 t ) = 1 + 2 sinh 2 ( t ) \cosh(2t) = 1 + 2\sinh^2(t) cosh ( 2 t ) = 1 + 2 sinh 2 ( t ) give
R δ ( 2 ) ( ε n ) ≤ ω 2 { 1 + ( ω 2 − 1 ) Z n 2 } − 1 ω 2 − 1 = 1 + ω 2 Z n 2 . R_{\delta}^{(2)}(\varepsilon_n) \le\frac{\omega^2\{1+(\omega^2-1)Z_n^2\}-1}{\omega^2-1} = 1+\omega^2 Z_n^2. R δ ( 2 ) ( ε n ) ≤ ω 2 − 1 ω 2 { 1 + ( ω 2 − 1 ) Z n 2 } − 1 = 1 + ω 2 Z n 2 . For the remaining inequalities, hold Z n Z_n Z n fixed, and differentiate with respect to ρ n \rho_n ρ n :
∂ R δ ( 1 ) ( ε n ) ∂ ρ n = ω ( 1 − ρ n 2 ) / 2 D δ [ cosh ( ρ n ε n δ ) sinh − 1 ( D δ Z n ) − ρ n log ( ω ) sinh ( ρ n ε n δ ) ] . \frac{\partial R_{\delta}^{(1)}(\varepsilon_n)}{\partial\rho_n}
=
\frac{\omega^{(1-\rho_n^2)/2}}{D_\delta}
\left[
\cosh\left(\frac{\rho_n\varepsilon_n}{\delta}\right)\sinh^{-1}(D_\delta Z_n)
-\rho_n\log(\omega)\sinh\left(\frac{\rho_n\varepsilon_n}{\delta}\right)
\right]. ∂ ρ n ∂ R δ ( 1 ) ( ε n ) = D δ ω ( 1 − ρ n 2 ) /2 [ cosh ( δ ρ n ε n ) sinh − 1 ( D δ Z n ) − ρ n log ( ω ) sinh ( δ ρ n ε n ) ] . To upper bound the above magnitude, we have
cosh ( ρ n ε n δ ) ≤ cosh ( ε n δ ) = 1 + D δ 2 Z n 2 ≤ 1 + D δ ∣ Z n ∣ , \cosh\left(\frac{\rho_n\varepsilon_n}{\delta}\right)
\le
\cosh\left(\frac{\varepsilon_n}{\delta}\right)
=
\sqrt{1+D_\delta^2 Z_n^2}
\le1+D_\delta|Z_n|, cosh ( δ ρ n ε n ) ≤ cosh ( δ ε n ) = 1 + D δ 2 Z n 2 ≤ 1 + D δ ∣ Z n ∣ , using cosh { sinh − 1 ( x ) } = 1 + x 2 \cosh\{\sinh^{-1}(x)\} = \sqrt{1+x^2} cosh { sinh − 1 ( x )} = 1 + x 2 . Handling sinh ( ρ n ε n / δ ) \sinh(\rho_n\varepsilon_n/\delta) sinh ( ρ n ε n / δ ) as before, we have
∣ ∂ R δ ( 1 ) ( ε n ) ∂ ρ n ∣ ≲ ( 1 + ∣ Z n ∣ ) ∣ sinh − 1 ( D δ Z n ) ∣ + ∣ Z n ∣ ≲ ( 1 + Z n 2 ) \left|\frac{\partial R_{\delta}^{(1)}(\varepsilon_n)}{\partial\rho_n}\right|
\lesssim(1+|Z_n|)|\sinh^{-1}(D_\delta Z_n)|+|Z_n|
\lesssim(1+Z_n^2) ∂ ρ n ∂ R δ ( 1 ) ( ε n ) ≲ ( 1 + ∣ Z n ∣ ) ∣ sinh − 1 ( D δ Z n ) ∣ + ∣ Z n ∣ ≲ ( 1 + Z n 2 ) where we use ∣ sinh − 1 ( x ) ∣ ≤ ∣ x ∣ |\sinh^{-1}(x)| \le|x| ∣ sinh − 1 ( x ) ∣ ≤ ∣ x ∣ and 2 ∣ Z n ∣ ≤ 1 + Z n 2 2|Z_n| \le1+Z_n^2 2∣ Z n ∣ ≤ 1 + Z n 2 . Next, we have
∂ R δ ( 2 ) ( ε n ) ∂ ρ n = ω 2 ( 1 − ρ n 2 ) ω 2 − 1 [ 2 sinh ( 2 ρ n ε n δ ) sinh − 1 ( D δ Z n ) − 4 ρ n log ( ω ) cosh ( 2 ρ n ε n δ ) ] , \frac{\partial R_{\delta}^{(2)}(\varepsilon_n)}{\partial\rho_n}
=
\frac{\omega^{2(1-\rho_n^2)}}{\omega^2-1}
\left[
2\sinh\left(\frac{2\rho_n\varepsilon_n}{\delta}\right)\sinh^{-1}(D_\delta Z_n)
-4\rho_n\log(\omega)\cosh\left(\frac{2\rho_n\varepsilon_n}{\delta}\right)
\right], ∂ ρ n ∂ R δ ( 2 ) ( ε n ) = ω 2 − 1 ω 2 ( 1 − ρ n 2 ) [ 2 sinh ( δ 2 ρ n ε n ) sinh − 1 ( D δ Z n ) − 4 ρ n log ( ω ) cosh ( δ 2 ρ n ε n ) ] ,
and using ∣ sinh ( 2 ρ n ε n / δ ) ∣ ≤ cosh ( 2 ρ n ε n / δ ) ≤ 1 + ( ω 2 − 1 ) Z n 2 |\sinh(2\rho_n\varepsilon_n/\delta)| \le\cosh(2\rho_n\varepsilon_n/\delta) \le1+(\omega^2-1)Z_n^2 ∣ sinh ( 2 ρ n ε n / δ ) ∣ ≤ cosh ( 2 ρ n ε n / δ ) ≤ 1 + ( ω 2 − 1 ) Z n 2 then gives
∣ ∂ R δ ( 2 ) ( ε n ) ∂ ρ n ∣ ≲ ( 1 + Z n 2 ) { 1 + ∣ sinh − 1 ( D δ Z n ) ∣ } ≲ ( 1 + Z n 2 ) 2 . \left|\frac{\partial R_{\delta}^{(2)}(\varepsilon_n)}{\partial\rho_n}\right| \lesssim(1+Z_n^2)\{1+|\sinh^{-1}(D_\delta Z_n)|\} \lesssim(1+Z_n^2)^2. ∂ ρ n ∂ R δ ( 2 ) ( ε n ) ≲ ( 1 + Z n 2 ) { 1 + ∣ sinh − 1 ( D δ Z n ) ∣ } ≲ ( 1 + Z n 2 ) 2 .
Both derivative bounds hold uniformly for 0 < ρ n ≤ 1 0<\rho_n\le1 0 < ρ n ≤ 1 . At ρ n = 1 \rho_n=1 ρ n = 1 , we have R δ ( 1 ) ( ε n ) = Z n R_{\delta}^{(1)}(\varepsilon_n)=Z_n R δ ( 1 ) ( ε n ) = Z n and R δ ( 2 ) ( ε n ) = Z n 2 R_{\delta}^{(2)}(\varepsilon_n)=Z_n^2 R δ ( 2 ) ( ε n ) = Z n 2 from Proposition 2, so the mean value theorem gives the remaining inequalities.
Based on the above lemma, we immediately have Lemma D.4 with ρ = 1 \rho=1 ρ = 1 , where we have used α n ( 1 − ρ n ) ≲ α n 2 \alpha_n(1-\rho_n)\lesssim\alpha_n^2 α n ( 1 − ρ n ) ≲ α n 2 by assumption. To show Lemma D.5, the argument in (27) ensures a positive b < 1 b<1 b < 1 exists such that 1 + Δ σ 2 ≥ 1 − b 1+\Delta_{\sigma^2}\ge1-b 1 + Δ σ 2 ≥ 1 − b uniformly, so the proof can proceed exactly as before since (30) holds in this case. Finally, Lemma D.6 is unchanged as the terms are the same.
The residual bound (35) and Assumption 6 give
α n 2 E [ ( 1 + Z n 2 ) 2 ∣ F n − 1 ] ≲ n − 2 ( 1 − α 0 S ) . \alpha_n^2 E[(1+Z_n^2)^2\mid\mathcal{F}_{n-1}]\lesssim n^{-2(1-\alpha_0S)}. α n 2 E [( 1 + Z n 2 ) 2 ∣ F n − 1 ] ≲ n − 2 ( 1 − α 0 S ) .
which is summable since α 0 S < 1 / 2 \alpha_0S<1/2 α 0 S < 1/2 by assumption. Also, K 0 < ∞ K_0<\infty K 0 < ∞ under the same assumptions. The proofs of Theorems D.1 and D.2 therefore apply with ρ = 1 \rho=1 ρ = 1 , giving
K n → K ∞ < ∞ , ∑ n = 1 ∞ α n T n − 1 < ∞ , T n → 0 P ∗ -a.s. K_n\to K_\infty<\infty,\qquad\sum_{n=1}^{\infty}\alpha_nT_{n-1}<\infty,\qquad T_n\to0\quad P^*\text{-a.s.} K n → K ∞ < ∞ , n = 1 ∑ ∞ α n T n − 1 < ∞ , T n → 0 P ∗ -a.s.
The final statement follows from Lemma D.2 as before.
Proof of Theorem 4
We begin by presenting a general version of the theorem for the multi-output case with general missingness patterns and dimensionality q q q . The key difference to the order-compatible result in Theorem 4 is that the projection matrix is data-dependent. We begin with a general version of Assumption 10.
Assumption D.1 (Multi-output with general missingness). Let ( Y n , X n , O n ) ∼ i i d P ∗ (Y_n,X_n,O_n)\overset{\mathrm{iid}}{\sim}P^* ( Y n , X n , O n ) ∼ iid P ∗ , where Y n ∈ R q Y_n\in\mathbb{R}^q Y n ∈ R q and O n ∈ O O_n\in\mathcal{O} O n ∈ O , where O \mathcal{O} O is the collection of nonempty subsets of { 1 , … , q } \{1,\ldots,q\} { 1 , … , q } . Suppose P X ∗ P_X^* P X ∗ has compact support X \mathcal{X} X and Lebesgue density p X ∗ p_X^* p X ∗ , E [ ∥ Y ∥ 4 ] < ∞ E[\lVert Y\rVert^4]<\infty E [∥ Y ∥ 4 ] < ∞ and E [ ∣ log det Σ ∗ ( X ) ∣ ] < ∞ E[|\log\det\Sigma_*(X)|]<\infty E [ ∣ log det Σ ∗ ( X ) ∣ ] < ∞ . Furthermore, assume O ⊥ Y ∣ X O\perp Y\mid X O ⊥ Y ∣ X , with P ∗ ( O = { 1 , … , q } ∣ x ) > 0 P^*(O=\{1,\ldots,q\}\mid x)>0 P ∗ ( O = { 1 , … , q } ∣ x ) > 0 for P X ∗ P_X^* P X ∗ -almost all x x x .
As we will need to consider taking expectations over the missing pattern O O O , we will use the superscript o o o to denote the observed versions of appropriate terms. Let S o S^{o} S o be the selection matrix of size ∣ o ∣ × q |o| \times q ∣ o ∣ × q , and write
Q n o ( x ) = A n o ( x ) T ( A n o ( x ) A n o ( x ) T ) − 1 A n o ( x ) , A n o ( x ) = S o L n ( x ) . Q_n^{o}(x) = A_n^{o}(x)^{\mathsf{T}}\left(A_n^{o}(x)A_n^{o}(x)^{\mathsf{T}}\right)^{-1}A_n^{o}(x), \qquad A_n^{o}(x) = S^{o}L_n(x). Q n o ( x ) = A n o ( x ) T ( A n o ( x ) A n o ( x ) T ) − 1 A n o ( x ) , A n o ( x ) = S o L n ( x ) .
We may drop the dependence on x x x and write Q n o Q_n^{o} Q n o and A n o A_n^{o} A n o for shorthand. For the observed values as in (18) in the main paper, we would then have
Q n = Q n − 1 O n ( X n ) , A n = A n − 1 O n ( X n ) . Q_n = Q_{n-1}^{O_n}(X_n), \qquad A_n = A_{n-1}^{O_n}(X_n). Q n = Q n − 1 O n ( X n ) , A n = A n − 1 O n ( X n ) .
We will follow the notation of omitting both o o o and x x x if referring to the observed values. We then write the projected error functions as
u n o ( x ) = Q n o ( x ) u n ( x ) , W n o ( x ) = Q n o ( x ) W n ( x ) Q n o ( x ) . u_n^{o}(x) = Q_n^{o}(x)u_n(x), \qquad W_n^{o}(x) = Q_n^{o}(x)W_n(x)Q_n^{o}(x). u n o ( x ) = Q n o ( x ) u n ( x ) , W n o ( x ) = Q n o ( x ) W n ( x ) Q n o ( x ) .
We highlight that the above error functions are still of size q q q and q × q q \times q q × q respectively, as we are only applying a projection. The averaged error functions are then
u ‾ n ( x ) = ∑ o ∈ O p ∗ ( o ∣ x ) u n o ( x ) , W ‾ n ( x ) = ∑ o ∈ O p ∗ ( o ∣ x ) W n o ( x ) . \overline{u}_n(x) = \sum_{o \in\mathcal{O}} p^{*}(o \mid x)u_n^{o}(x), \qquad\overline{W}_n(x) = \sum_{o \in\mathcal{O}} p^{*}(o \mid x)W_n^{o}(x). u n ( x ) = o ∈ O ∑ p ∗ ( o ∣ x ) u n o ( x ) , W n ( x ) = o ∈ O ∑ p ∗ ( o ∣ x ) W n o ( x ) .
We then have the main theorem for general missingness patterns, with Theorem 4 as the special case when q = 2 q = 2 q = 2 and O ∈ { { 1 } , { 1 , 2 } } O \in\{\{1\}, \{1,2\}\} O ∈ {{ 1 } , { 1 , 2 }} a.s.
Theorem D.3. Under Assumptions 1, 7, C.1 and D.1, the updates (16)–(17) satisfy
∥ u ‾ n ∥ s → 0 , ∥ W ‾ n ∥ s → 0 P X ∗ -a.s. \lVert\overline{u}_n \rVert_s \to0, \qquad\lVert\overline{W}_n \rVert_s \to0 \qquad P_X^{*}\text{-a.s.} ∥ u n ∥ s → 0 , ∥ W n ∥ s → 0 P X ∗ -a.s. Furthermore, under Assumptions 9 and 8, ∥ ⋅ ∥ s \lVert\cdot\rVert_s ∥ ⋅ ∥ s is a norm on the respective vector- and matrix-valued L 1 ( P X ∗ ) L^{1}(P_X^{*}) L 1 ( P X ∗ ) spaces, and ∥ u ‾ n ∥ s = ∥ W ‾ n ∥ s = 0 \lVert\overline{u}_n \rVert_s = \lVert\overline{W}_n \rVert_s = 0 ∥ u n ∥ s = ∥ W n ∥ s = 0 if and only if μ n ( x ) = μ ∗ ( x ) \mu_n(x) = \mu_{*}(x) μ n ( x ) = μ ∗ ( x ) and Σ n ( x ) = Σ ∗ ( x ) \Sigma_n(x) = \Sigma_{*}(x) Σ n ( x ) = Σ ∗ ( x ) for P X ∗ P_X^{*} P X ∗ -almost all x x x .
Proof outline
The proof follows almost exactly the same technique as the univariate case, with additional technical steps in controlling the remainders due to the projection matrices. The learning rate bounds, namely Lemma D.1, can be applied identically. We let ∥ ⋅ ∥ \lVert\cdot\rVert ∥ ⋅ ∥ and ∥ ⋅ ∥ F \lVert\cdot\rVert_F ∥ ⋅ ∥ F denote the Euclidean and Frobenius norms for vectors and matrices respectively, and extend L ∗ 1 = L 1 ( P X ∗ ) L_{*}^{1} = L^{1}(P_X^{*}) L ∗ 1 = L 1 ( P X ∗ ) , ⟨ ⋅ , ⋅ ⟩ s \langle\cdot,\cdot\rangle_s ⟨ ⋅ , ⋅ ⟩ s and S S S component-wise, so Lemma D.2 applies to both types of functions. The residual function is now a vector,
r n ( y , x ) = L n ( x ) − 1 { y − μ n ( x ) } , r_n(y,x) = L_n(x)^{-1}\{y-\mu_n(x)\}, r n ( y , x ) = L n ( x ) − 1 { y − μ n ( x )} ,
which gives Z n = r n − 1 ( Y n , X n ) Z_n=r_{n-1}(Y_n,X_n) Z n = r n − 1 ( Y n , X n ) and Z ‾ n = Q n Z n \overline{Z}_n=Q_nZ_n Z n = Q n Z n . The multivariate equivalents of the expected update terms (28) are
E [ Z ‾ n ∣ F n − 1 , X n = x ] = u ‾ n − 1 ( x ) , E[\overline{Z}_n \mid\mathcal{F}_{n-1},X_n=x]=\overline{u}_{n-1}(x), E [ Z n ∣ F n − 1 , X n = x ] = u n − 1 ( x ) ,
E [ Z ‾ n Z ‾ n T − Q n ∣ F n − 1 , X n = x ] = W ‾ n − 1 ( x ) , E[\overline{Z}_n\overline{Z}_n^{\mathsf{T}}-Q_n \mid\mathcal{F}_{n-1},X_n=x]=\overline{W}_{n-1}(x), E [ Z n Z n T − Q n ∣ F n − 1 , X n = x ] = W n − 1 ( x ) ,
where F n = σ ( Y 1 , X 1 , O 1 , … , Y n , X n , O n ) \mathcal{F}_n=\sigma(Y_1,X_1,O_1,\ldots,Y_n,X_n,O_n) F n = σ ( Y 1 , X 1 , O 1 , … , Y n , X n , O n ) . The above follows from conditioning first on O n O_n O n and using O n ⊥ Y n ∣ X n O_n\perp Y_n\mid X_n O n ⊥ Y n ∣ X n .
Next, we define the general version of the KL divergence terms. For mean vector μ \mu μ and covariance matrix Σ \Sigma Σ , we will use the notation
μ o = S o μ , Σ o = S o Σ S o T , \mu^{o}=S^{o}\mu,\qquad\Sigma^{o}=S^{o}\Sigma S^{o\mathsf{T}}, μ o = S o μ , Σ o = S o Σ S o T ,
which corresponds to choosing the observed coordinates in the mean vector and covariance matrix. Unlike the error function case, the mean vector and covariance matrix have size ∣ o ∣ |o| ∣ o ∣ and ∣ o ∣ × ∣ o ∣ |o|\times|o| ∣ o ∣ × ∣ o ∣ respectively. We then have
K n X , O ( x , o ) = 2 KL [ N { μ ∗ o ( x ) , Σ ∗ o ( x ) } , N { μ n o ( x ) , Σ n o ( x ) } ] , K_n^{X,O}(x,o)=2\operatorname{KL}\left[\mathcal{N}\{\mu_*^{o}(x),\Sigma_*^{o}(x)\},\mathcal{N}\{\mu_n^{o}(x),\Sigma_n^{o}(x)\}\right], K n X , O ( x , o ) = 2 KL [ N { μ ∗ o ( x ) , Σ ∗ o ( x )} , N { μ n o ( x ) , Σ n o ( x )} ] ,
K n X ( x ) = ∑ o ∈ O p ∗ ( o ∣ x ) K n X , O ( x , o ) , K_n^{X}(x)=\sum_{o\in\mathcal{O}}p^*(o\mid x)K_n^{X,O}(x,o), K n X ( x ) = o ∈ O ∑ p ∗ ( o ∣ x ) K n X , O ( x , o ) ,
where K n = ∫ K n X ( x ) d P X ∗ ( x ) K_n=\int K_n^{X}(x)\,dP_X^*(x) K n = ∫ K n X ( x ) d P X ∗ ( x ) as before. Therefore, K n K_n K n is twice the KL divergence averaged over both O O O and X X X , and will serve as the almost-supermartingale. Naturally, we have
T n = 2 ρ ∥ u ‾ n ∥ s 2 + ρ 2 ∥ W ‾ n ∥ s 2 . T_n=2\rho\lVert\overline{u}_n\rVert_s^2+\rho^2\lVert\overline{W}_n\rVert_s^2. T n = 2 ρ ∥ u n ∥ s 2 + ρ 2 ∥ W n ∥ s 2 .
A similar argument to the univariate case shows u n u_n u n and W n W_n W n are suitably integrable, which is preserved by projection/averaging. Likewise, one can verify that K 0 < ∞ K_0<\infty K 0 < ∞ .
Technical lemmas
To begin, we will show the multivariate version of Lemma D.3 to bound the residual.
Lemma D.8. Under Assumptions 1, 7, C.1, the following holds for all ( y , x ) ∈ R q × X (y,x) \in\mathbb{R}^{q} \times\mathcal{X} ( y , x ) ∈ R q × X :
∥ r n ( y , x ) ∥ 2 ≲ [ 1 + ∥ r 0 ( y , x ) ∥ 2 ] ( n + 1 ) α 0 ρ 2 S \lVert r_n(y,x)\rVert^2 \lesssim\left[1+\lVert r_0(y,x)\rVert^2\right](n+1)^{\alpha_0\rho^2 S} ∥ r n ( y , x ) ∥ 2 ≲ [ 1 + ∥ r 0 ( y , x ) ∥ 2 ] ( n + 1 ) α 0 ρ 2 S
Proof. Let Q Q Q be an orthogonal projection matrix, so we have Q ⪯ I Q \preceq I Q ⪯ I . For any z ∈ R q z \in\mathbb{R}^{q} z ∈ R q , 0 < ρ < 1 0 < \rho< 1 0 < ρ < 1 and 0 ≤ a < 1 0 \leq a < 1 0 ≤ a < 1 , we thus have
I + a ρ 2 { ( 1 − a ) z z T − Q } ⪰ ( 1 − a ρ 2 ) I + a ρ 2 ( 1 − a ) z z T . I+a\rho^2\left\{(1-a)zz^{\mathsf{T}}-Q\right\}\succeq(1-a\rho^2)I+a\rho^2(1-a)zz^{\mathsf{T}}. I + a ρ 2 { ( 1 − a ) z z T − Q } ⪰ ( 1 − a ρ 2 ) I + a ρ 2 ( 1 − a ) z z T .
Next, for some r ∈ R q r \in\mathbb{R}^{q} r ∈ R q , the above then gives
Δ = ( r − ρ a z ) T [ I + a ρ 2 { ( 1 − a ) z z T − Q } ] − 1 ( r − ρ a z ) ≤ ( r − ρ a z ) T [ ( 1 − a ρ 2 ) I + a ρ 2 ( 1 − a ) z z T ] − 1 ( r − ρ a z ) = 1 1 − a ρ 2 [ ∥ r − ρ a z ∥ 2 − a ρ 2 ( 1 − a ) ( z T r − ρ a ∥ z ∥ 2 ) 2 1 − a ρ 2 + a ρ 2 ( 1 − a ) ∥ z ∥ 2 ] . (36) \begin{aligned}
\Delta=(r-\rho az)^{\mathsf{T}}\left[I+a\rho^2\left\{(1-a)zz^{\mathsf{T}}-Q\right\}\right]^{-1}(r-\rho az) \\
&\leq(r-\rho az)^{\mathsf{T}}\left[(1-a\rho^2)I+a\rho^2(1-a)zz^{\mathsf{T}}\right]^{-1}(r-\rho az) \\
&=\frac{1}{1-a\rho^2}\left[\lVert r-\rho az\rVert^2-\frac{a\rho^2(1-a)\left(z^{\mathsf{T}}r-\rho a\lVert z\rVert^2\right)^2}{1-a\rho^2+a\rho^2(1-a)\lVert z\rVert^2}\right]. \tag*{(36)}
\end{aligned} Δ = ( r − ρ a z ) T [ I + a ρ 2 { ( 1 − a ) z z T − Q } ] − 1 ( r − ρ a z ) ≤ ( r − ρ a z ) T [ ( 1 − a ρ 2 ) I + a ρ 2 ( 1 − a ) z z T ] − 1 ( r − ρ a z ) = 1 − a ρ 2 1 [ ∥ r − ρ a z ∥ 2 − 1 − a ρ 2 + a ρ 2 ( 1 − a ) ∥ z ∥ 2 a ρ 2 ( 1 − a ) ( z T r − ρ a ∥ z ∥ 2 ) 2 ] . (36)
where we have applied Sherman-Morrison in the last step. Next, for z ≠ 0 z\ne0 z = 0 , we have
∥ r − ρ a z ∥ 2 = ∥ r ∥ 2 − 2 ρ a z T r + ρ 2 a 2 ∥ z ∥ 2 = [ ∥ r ∥ 2 − ( z T r ) 2 ∥ z ∥ 2 ] + ( z T r − ρ a ∥ z ∥ 2 ) 2 ∥ z ∥ 2 . \lVert r-\rho az\rVert^2=\lVert r\rVert^2-2\rho az^{\mathsf{T}}r+\rho^2a^2\lVert z\rVert^2
=\left[\lVert r\rVert^2-\frac{(z^{\mathsf{T}}r)^2}{\lVert z\rVert^2}\right]+\frac{(z^{\mathsf{T}}r-\rho a\lVert z\rVert^2)^2}{\lVert z\rVert^2}. ∥ r − ρ a z ∥ 2 = ∥ r ∥ 2 − 2 ρ a z T r + ρ 2 a 2 ∥ z ∥ 2 = [ ∥ r ∥ 2 − ∥ z ∥ 2 ( z T r ) 2 ] + ∥ z ∥ 2 ( z T r − ρ a ∥ z ∥ 2 ) 2 .
where we have completed the square. Combining the last term with the fraction in the square bracket of (36) gives
Δ ≤ ∥ r ∥ 2 − ( z T r ) 2 ∥ z ∥ 2 1 − a ρ 2 + ( z T r ∥ z ∥ − ρ a ∥ z ∥ ) 2 1 − a ρ 2 + a ρ 2 ( 1 − a ) ∥ z ∥ 2 . \Delta\leq\frac{\lVert r\rVert^2-\frac{(z^{\mathsf{T}}r)^2}{\lVert z\rVert^2}}{1-a\rho^2}
+\frac{\left(\frac{z^{\mathsf{T}}r}{\lVert z\rVert}-\rho a\lVert z\rVert\right)^2}{1-a\rho^2+a\rho^2(1-a)\lVert z\rVert^2}. Δ ≤ 1 − a ρ 2 ∥ r ∥ 2 − ∥ z ∥ 2 ( z T r ) 2 + 1 − a ρ 2 + a ρ 2 ( 1 − a ) ∥ z ∥ 2 ( ∥ z ∥ z T r − ρ a ∥ z ∥ ) 2 .
Note that the second term in the above is exactly h ( ∥ z ∥ ) h(\lVert z\rVert) h (∥ z ∥) from the proof of Lemma D.3, with ( z T r ) / ∥ z ∥ (z^{\mathsf{T}}r)/\lVert z\rVert ( z T r ) / ∥ z ∥ as the scalar, so we have the upper bound
Δ ≤ ∥ r ∥ 2 1 − a ρ 2 + a 1 − a . (37) \Delta\leq\frac{\lVert r\rVert^2}{1-a\rho^2}+\frac{a}{1-a}. \tag*{(37)} Δ ≤ 1 − a ρ 2 ∥ r ∥ 2 + 1 − a a . (37)
For z = 0 z=0 z = 0 , the above upper bound holds immediately.
We can now show the bound. Dropping the dependence on ( y , x ) (y,x) ( y , x ) , we have
r n = L n − 1 L n − 1 ( r n − 1 − ρ β n Z ‾ n ) , r_n=L_n^{-1}L_{n-1}(r_{n-1}-\rho\beta_n\overline{Z}_n), r n = L n − 1 L n − 1 ( r n − 1 − ρ β n Z n ) ,
which gives
r n T r n = ( r n − 1 − ρ β n Z ‾ n ) T [ I + β n ρ 2 { ( 1 − β n ) Z ‾ n Z ‾ n T − Q n } ] − 1 ( r n − 1 − ρ β n Z ‾ n ) . r_n^{\mathsf{T}}r_n=(r_{n-1}-\rho\beta_n\overline{Z}_n)^{\mathsf{T}}
\left[I+\beta_n\rho^2\left\{(1-\beta_n)\overline{Z}_n\overline{Z}_n^{\mathsf{T}}-Q_n\right\}\right]^{-1}
(r_{n-1}-\rho\beta_n\overline{Z}_n). r n T r n = ( r n − 1 − ρ β n Z n ) T [ I + β n ρ 2 { ( 1 − β n ) Z n Z n T − Q n } ] − 1 ( r n − 1 − ρ β n Z n ) .
As a result, we can apply (37) with r = r n − 1 r=r_{n-1} r = r n − 1 , a = β n a=\beta_n a = β n , z = Z ‾ n z=\overline{Z}_n z = Z n and Q = Q n Q=Q_n Q = Q n , giving
∥ r n ∥ 2 ≤ ∥ r n − 1 ∥ 2 1 − ρ 2 β n + β n 1 − β n . \lVert r_n\rVert^2 \le\frac{\lVert r_{n-1}\rVert^2}{1-\rho^2\beta_n}+\frac{\beta_n}{1-\beta_n}. ∥ r n ∥ 2 ≤ 1 − ρ 2 β n ∥ r n − 1 ∥ 2 + 1 − β n β n .
Recursing in the exact same manner as in Lemma D.3 gives us the result.
Next, we define the standardized multivariate increments:
Δ μ = ρ β n Z ‾ n , Δ Σ = β n ρ 2 { ( 1 − β n ) Z ‾ n Z ‾ n T − Q n } . \Delta_\mu=\rho\beta_n\overline{Z}_n,\qquad\Delta_\Sigma=\beta_n\rho^2\left\{(1-\beta_n)\overline{Z}_n\overline{Z}_n^{\mathsf T}-Q_n\right\}. Δ μ = ρ β n Z n , Δ Σ = β n ρ 2 { ( 1 − β n ) Z n Z n T − Q n } .
We then have the following multivariate analogue of Lemma D.4 .
Lemma D.9. Under Assumptions 1, 7, C.1 and D.1, we have
∥ Δ μ ∥ ≲ α n ∥ Z ‾ n ∥ , ∥ Δ Σ ∥ F ≲ α n ( 1 + ∥ Z ‾ n ∥ 2 ) . (38) \lVert\Delta_\mu\rVert\lesssim\alpha_n\lVert\overline{Z}_n\rVert,\qquad\lVert\Delta_\Sigma\rVert_F\lesssim\alpha_n(1+\lVert\overline{Z}_n\rVert^2). \tag*{(38)} ∥ Δ μ ∥ ≲ α n ∥ Z n ∥ , ∥ Δ Σ ∥ F ≲ α n ( 1 + ∥ Z n ∥ 2 ) . (38) Furthermore,
∥ E [ Δ μ ∣ F n − 1 ] − α n ρ S u ‾ n − 1 ∥ ≲ α n 2 E [ ( 1 + ∥ Z ‾ n ∥ 2 ) 2 ∣ F n − 1 ] , (39) \left\lVert E[\Delta_\mu\mid\mathcal{F}_{n-1}]-\alpha_n\rho\mathcal{S}\overline{u}_{n-1}\right\rVert\lesssim\alpha_n^2E[(1+\lVert\overline{Z}_n\rVert^2)^2\mid\mathcal{F}_{n-1}], \tag*{(39)} ∥ E [ Δ μ ∣ F n − 1 ] − α n ρ S u n − 1 ∥ ≲ α n 2 E [( 1 + ∥ Z n ∥ 2 ) 2 ∣ F n − 1 ] , (39) ∥ E [ Δ Σ ∣ F n − 1 ] − α n ρ 2 S W ‾ n − 1 ∥ F ≲ α n 2 E [ ( 1 + ∥ Z ‾ n ∥ 2 ) 2 ∣ F n − 1 ] . (40) \left\lVert E[\Delta_\Sigma\mid\mathcal{F}_{n-1}]-\alpha_n\rho^2\mathcal{S}\overline{W}_{n-1}\right\rVert_F\lesssim\alpha_n^2E[(1+\lVert\overline{Z}_n\rVert^2)^2\mid\mathcal{F}_{n-1}]. \tag*{(40)} E [ Δ Σ ∣ F n − 1 ] − α n ρ 2 S W n − 1 F ≲ α n 2 E [( 1 + ∥ Z n ∥ 2 ) 2 ∣ F n − 1 ] . (40)
Proof. The proof is the same as the univariate case in Lemma D.4 , using ∥ Z ‾ n Z ‾ n T ∥ F = ∥ Z ‾ n ∥ 2 \lVert\overline{Z}_n\overline{Z}_n^{\mathsf T}\rVert_F=\lVert\overline{Z}_n\rVert^2 ∥ Z n Z n T ∥ F = ∥ Z n ∥ 2 and ∥ Q n ∥ F ≤ q \lVert Q_n\rVert_F\le\sqrt{q} ∥ Q n ∥ F ≤ q , since Q n Q_n Q n is an orthogonal projection matrix.
Next, we show a technical lemma that is the main novel step for the multivariate extension. Unlike the univariate case, we need to handle the changing projection matrix for W n o W_n^o W n o and u n o u_n^o u n o which changes with n n n . To that end, define the term
D n : = L n − 1 − 1 L n , D n D n T = I + Δ Σ . D_n:=L_{n-1}^{-1}L_n,\qquad D_nD_n^{\mathsf T}=I+\Delta_\Sigma. D n := L n − 1 − 1 L n , D n D n T = I + Δ Σ .
Furthermore, for a symmetric matrix A A A , let L ( A ) \mathcal{L}(A) L ( A ) denote the lower triangular part of A A A , with the diagonal values halved. The operator L \mathcal{L} L arises in the study of perturbations of Cholesky factorizations, e.g. Stewart [ [37 ] , Drmač et al. [ [10 ] . Heuristically, we have the matrix equivalent of 1 + x ≈ 1 + x / 2 \sqrt{1+x}\approx1+x/2 1 + x ≈ 1 + x /2 as Chol ( I + A ) ≈ I + L ( A ) \operatorname{Chol}(I+A)\approx I+\mathcal{L}(A) Chol ( I + A ) ≈ I + L ( A ) when A A A is small and symmetric.
Lemma D.10. Consider the projection term
Δ Q = Q n o D n − 1 − Q n − 1 o . \Delta_Q=Q_n^oD_n^{-1}-Q_{n-1}^o. Δ Q = Q n o D n − 1 − Q n − 1 o .
We have the expansion
Δ Q = [ L ( Δ Σ ) T − Q n − 1 ∘ Δ Σ + R n D ] Q n − 1 ∘ (41) \Delta_Q = \left[\mathcal{L}(\Delta_\Sigma)^{\mathsf{T}} - Q^{\circ}_{n-1}\Delta_\Sigma+ R^D_n\right]Q^{\circ}_{n-1} \tag*{(41)} Δ Q = [ L ( Δ Σ ) T − Q n − 1 ∘ Δ Σ + R n D ] Q n − 1 ∘ (41)
where ∥ R n D ∥ F ≲ ∥ Δ Σ ∥ F 2 \lVert R^D_n\rVert_F \lesssim\lVert\Delta_\Sigma\rVert_F^2 ∥ R n D ∥ F ≲ ∥ Δ Σ ∥ F 2 . In particular, the above form gives Δ Q Q n − 1 ∘ = Δ Q \Delta_Q Q^{\circ}_{n-1} = \Delta_Q Δ Q Q n − 1 ∘ = Δ Q . Furthermore, we have the inequalities
∥ Q n ∘ D n − 1 ∥ F ≲ 1 , ∥ Δ Q ∥ F ≲ min { 1 , ∥ Δ Σ ∥ F } . \lVert Q^{\circ}_n D_n^{-1}\rVert_F \lesssim1,\qquad\lVert\Delta_Q\rVert_F \lesssim\min\{1,\lVert\Delta_\Sigma\rVert_F\}. ∥ Q n ∘ D n − 1 ∥ F ≲ 1 , ∥ Δ Q ∥ F ≲ min { 1 , ∥ Δ Σ ∥ F } .
Proof. Intuitively, D n D_n D n is the matrix version of 1 + Δ σ 2 \sqrt{1+\Delta_{\sigma}^2} 1 + Δ σ 2 , and we note that D n D_n D n is lower triangular and contains positive diagonal entries, so it is the Cholesky factor of I + Δ Σ I+\Delta_\Sigma I + Δ Σ . Since A n ∘ = A n − 1 ∘ D n A^{\circ}_n=A^{\circ}_{n-1}D_n A n ∘ = A n − 1 ∘ D n , we have
Q n ∘ D n − 1 = A n ∘ T ( A n ∘ A n ∘ T ) − 1 A n ∘ D n − 1 = D n T A n − 1 ∘ − 1 ( Σ n ∘ ) − 1 A n − 1 ∘ . Q^{\circ}_nD_n^{-1} = A_n^{\circ\mathsf{T}}\left(A_n^\circ A_n^{\circ\mathsf{T}}\right)^{-1}A_n^\circ D_n^{-1}
= D_n^{\mathsf{T}}A_{n-1}^{\circ-1}\left(\Sigma_n^\circ\right)^{-1}A_{n-1}^\circ. Q n ∘ D n − 1 = A n ∘ T ( A n ∘ A n ∘ T ) − 1 A n ∘ D n − 1 = D n T A n − 1 ∘− 1 ( Σ n ∘ ) − 1 A n − 1 ∘ .
For the inner inverse, we use the identity A − 1 = B − 1 − A − 1 ( A − B ) B − 1 A^{-1}=B^{-1}-A^{-1}(A-B)B^{-1} A − 1 = B − 1 − A − 1 ( A − B ) B − 1 [28 Section 3.2.5] and Σ n ∘ = Σ n − 1 ∘ + A n − 1 ∘ Δ Σ ( A n − 1 ∘ ) T \Sigma_n^\circ=\Sigma_{n-1}^\circ+A_{n-1}^\circ\Delta_\Sigma(A_{n-1}^\circ)^{\mathsf{T}} Σ n ∘ = Σ n − 1 ∘ + A n − 1 ∘ Δ Σ ( A n − 1 ∘ ) T which gives
( Σ n ∘ ) − 1 = ( Σ n − 1 ∘ ) − 1 − ( Σ n ∘ ) − 1 [ A n − 1 ∘ Δ Σ ( A n − 1 ∘ ) T ] ( Σ n − 1 ∘ ) − 1 . \left(\Sigma_n^\circ\right)^{-1}
=
\left(\Sigma_{n-1}^\circ\right)^{-1}
-
\left(\Sigma_n^\circ\right)^{-1}
\left[A_{n-1}^\circ\Delta_\Sigma(A_{n-1}^\circ)^{\mathsf{T}}\right]
\left(\Sigma_{n-1}^\circ\right)^{-1}. ( Σ n ∘ ) − 1 = ( Σ n − 1 ∘ ) − 1 − ( Σ n ∘ ) − 1 [ A n − 1 ∘ Δ Σ ( A n − 1 ∘ ) T ] ( Σ n − 1 ∘ ) − 1 .
Substituting this into the above then gives
Q n ∘ D n − 1 = D n T Q n − 1 ∘ − [ D n T ( A n − 1 ∘ ) T ( Σ n ∘ ) − 1 A n − 1 ∘ ] Δ Σ Q n − 1 ∘ = [ D n T − Q n ∘ D n − 1 Δ Σ ] Q n − 1 ∘ . (42) \begin{aligned}
Q_n^\circ D_n^{-1}
=
D_n^{\mathsf{T}}Q_{n-1}^\circ
-
\left[D_n^{\mathsf{T}}(A_{n-1}^\circ)^{\mathsf{T}}(\Sigma_n^\circ)^{-1}A_{n-1}^\circ\right]\Delta_\Sigma Q_{n-1}^\circ\\
&=
\left[D_n^{\mathsf{T}}-Q_n^\circ D_n^{-1}\Delta_\Sigma\right]Q_{n-1}^\circ. \tag*{(42)}
\end{aligned} Q n ∘ D n − 1 = D n T Q n − 1 ∘ − [ D n T ( A n − 1 ∘ ) T ( Σ n ∘ ) − 1 A n − 1 ∘ ] Δ Σ Q n − 1 ∘ = [ D n T − Q n ∘ D n − 1 Δ Σ ] Q n − 1 ∘ . (42)
We can then write the above as
Q n ∘ D n − 1 = [ I + L ( Δ Σ ) T − Q n − 1 ∘ Δ Σ + R n D ] Q n − 1 ∘ . Q_n^\circ D_n^{-1}
=
\left[I+\mathcal{L}(\Delta_\Sigma)^{\mathsf{T}}-Q_{n-1}^\circ\Delta_\Sigma+R_n^D\right]Q_{n-1}^\circ. Q n ∘ D n − 1 = [ I + L ( Δ Σ ) T − Q n − 1 ∘ Δ Σ + R n D ] Q n − 1 ∘ .
where R n D = [ D n − I − L ( Δ Σ ) ] T − Δ Q Δ Σ R_n^D=[D_n-I-\mathcal{L}(\Delta_\Sigma)]^{\mathsf{T}}-\Delta_Q\Delta_\Sigma R n D = [ D n − I − L ( Δ Σ ) ] T − Δ Q Δ Σ .
The next step is to bound ∥ R n D ∥ F \lVert R_n^D\rVert_F ∥ R n D ∥ F , which we do separately for the two terms. For the first term, we will use the results of Drmač et al. [1994] for perturbations of the Cholesky factor. Firstly, we highlight that D n D n T = I + Δ Σ ⪰ ( 1 − ρ 2 ) I ≻ 0 D_nD_n^{\mathsf{T}}=I+\Delta_\Sigma\succeq(1-\rho^2)I\succ0 D n D n T = I + Δ Σ ⪰ ( 1 − ρ 2 ) I ≻ 0 , so I + Δ Σ I+\Delta_\Sigma I + Δ Σ is positive definite. Furthermore, Δ Σ \Delta_\Sigma Δ Σ is symmetric, so the setting of Drmač et al. [1994] applies. Since D n D_n D n is the Cholesky factor of I + Δ Σ I+\Delta_\Sigma I + Δ Σ , Drmač et al. [1994, Equation (12)] immediately gives
∥ D n − I ∥ F ≤ 8 q + 2 q ∥ Δ Σ ∥ F ≲ ∥ Δ Σ ∥ F . (43) \lVert D_n-I\rVert_F\leq\sqrt{8q+2\sqrt{q}}\lVert\Delta_\Sigma\rVert_F\lesssim\lVert\Delta_\Sigma\rVert_F. \tag*{(43)} ∥ D n − I ∥ F ≤ 8 q + 2 q ∥ Δ Σ ∥ F ≲ ∥ Δ Σ ∥ F . (43)
Next, the definition of D n D n T D_nD_n^{\mathsf{T}} D n D n T gives
( D n − I ) ( D n − I ) T = Δ Σ − ( D n − I ) − ( D n − I ) T , (D_n-I)(D_n-I)^{\mathsf{T}}=\Delta_{\Sigma}-(D_n-I)-(D_n-I)^{\mathsf{T}}, ( D n − I ) ( D n − I ) T = Δ Σ − ( D n − I ) − ( D n − I ) T ,
and applying L \mathcal{L} L to both sides gives
L { ( D n − I ) ( D n − I ) T } = L ( Δ Σ ) − ( D n − I ) , (44) \mathcal{L}\{(D_n-I)(D_n-I)^{\mathsf{T}}\}=\mathcal{L}(\Delta_{\Sigma})-(D_n-I), \tag*{(44)} L {( D n − I ) ( D n − I ) T } = L ( Δ Σ ) − ( D n − I ) , (44)
where we have used the fact that L ( A + A T ) = A \mathcal{L}(A+A^{\mathsf{T}})=A L ( A + A T ) = A if A A A is lower triangular. Note that this is equivalent to a transposed version of Drmač et al. [ [10 ] with η = 1 \eta=1 η = 1 . Rearranging (44), we have
∥ ( D n − I ) − L ( Δ Σ ) ∥ F = ∥ L { ( D n − I ) ( D n − I ) T } ∥ F ≤ ∥ ( D n − I ) ( D n − I ) T ∥ F ≤ ∥ D n − I ∥ F 2 ≲ ∥ Δ Σ ∥ F 2 \begin{aligned}
\|(D_n-I)-\mathcal{L}(\Delta_{\Sigma})\|_F
&=\|\mathcal{L}\{(D_n-I)(D_n-I)^{\mathsf{T}}\}\|_F\\
&\leq\|(D_n-I)(D_n-I)^{\mathsf{T}}\|_F
\leq\|D_n-I\|_F^2\lesssim\|\Delta_{\Sigma}\|_F^2
\end{aligned} ∥ ( D n − I ) − L ( Δ Σ ) ∥ F = ∥ L {( D n − I ) ( D n − I ) T } ∥ F ≤ ∥ ( D n − I ) ( D n − I ) T ∥ F ≤ ∥ D n − I ∥ F 2 ≲ ∥ Δ Σ ∥ F 2
where the second line follows as L \mathcal{L} L cannot increase the Frobenius norm.
For the second term in R n D R_n^D R n D , substituting (42) gives
∥ Q n o D n − 1 − Q n − 1 o ∥ F ≤ ∥ D n − I ∥ F + ∥ D n − 1 ∥ F ∥ Δ Σ ∥ F , \|Q_n^oD_n^{-1}-Q_{n-1}^o\|_F\leq\|D_n-I\|_F+\|D_n^{-1}\|_F\|\Delta_{\Sigma}\|_F, ∥ Q n o D n − 1 − Q n − 1 o ∥ F ≤ ∥ D n − I ∥ F + ∥ D n − 1 ∥ F ∥ Δ Σ ∥ F ,
where we have used the fact that ∥ Q A ∥ F ≤ ∥ A ∥ F \|QA\|_F\leq\|A\|_F ∥ Q A ∥ F ≤ ∥ A ∥ F if Q Q Q is an orthogonal projection matrix. The term ∥ D n − I ∥ F \|D_n-I\|_F ∥ D n − I ∥ F can be handled again by (43). For the latter term, since D n D n T ⪰ ( 1 − ρ 2 ) I D_nD_n^{\mathsf{T}}\succeq(1-\rho^2)I D n D n T ⪰ ( 1 − ρ 2 ) I , we have
∥ D n − 1 ∥ F 2 = tr { ( D n D n T ) − 1 } ≤ q 1 − ρ 2 \|D_n^{-1}\|_F^2=\operatorname{tr}\{(D_nD_n^{\mathsf{T}})^{-1}\}\leq\frac{q}{1-\rho^2} ∥ D n − 1 ∥ F 2 = tr {( D n D n T ) − 1 } ≤ 1 − ρ 2 q
which gives ∥ Δ Q ∥ F ≲ ∥ Δ Σ ∥ F \|\Delta_Q\|_F\lesssim\|\Delta_{\Sigma}\|_F ∥ Δ Q ∥ F ≲ ∥ Δ Σ ∥ F . Together, the above implies ∥ R n D ∥ F ≲ ∥ Δ Σ ∥ F 2 \|R_n^D\|_F\lesssim\|\Delta_{\Sigma}\|_F^2 ∥ R n D ∥ F ≲ ∥ Δ Σ ∥ F 2 .
The equality then follows from Q n − 1 o Q n − 1 o = Q n − 1 o Q_{n-1}^oQ_{n-1}^o=Q_{n-1}^o Q n − 1 o Q n − 1 o = Q n − 1 o since it is an orthogonal projection matrix. For the remaining inequalities, we note that
∥ Q n o D n − 1 ∥ F ≤ ∥ D n − 1 ∥ F ≲ 1 , \|Q_n^oD_n^{-1}\|_F\leq\|D_n^{-1}\|_F\lesssim1, ∥ Q n o D n − 1 ∥ F ≤ ∥ D n − 1 ∥ F ≲ 1 ,
since Q n o Q_n^o Q n o is an orthogonal projection matrix. The final inequality then follows from the just shown ∥ Δ Q ∥ F ≲ ∥ Δ Σ ∥ F \|\Delta_Q\|_F\lesssim\|\Delta_{\Sigma}\|_F ∥ Δ Q ∥ F ≲ ∥ Δ Σ ∥ F and
∥ Q n o D n − 1 − Q n − 1 o ∥ F ≤ ∥ Q n o D n − 1 ∥ F + ∥ Q n − 1 o ∥ F ≲ 1 , \|Q_n^oD_n^{-1}-Q_{n-1}^o\|_F\leq\|Q_n^oD_n^{-1}\|_F+\|Q_{n-1}^o\|_F\lesssim1, ∥ Q n o D n − 1 − Q n − 1 o ∥ F ≤ ∥ Q n o D n − 1 ∥ F + ∥ Q n − 1 o ∥ F ≲ 1 ,
where ∥ Q n − 1 o ∥ F = ∣ o ∣ ≤ q \|Q_{n-1}^o\|_F=\sqrt{|o|}\leq\sqrt{q} ∥ Q n − 1 o ∥ F = ∣ o ∣ ≤ q as it is an orthogonal projection matrix.
Next, we will show multivariate extensions of Lemma D.5.
Lemma D.11. Under Assumptions 1, 7, C.1 and D.1, we have
u n o − u n − 1 o = [ L ( Δ Σ ) ⊤ − Q n − 1 o Δ Σ ] u n − 1 o − Q n − 1 o Δ μ + R n u , u_n^o-u_{n-1}^o=\left[\mathcal{L}(\Delta_\Sigma)^\top-Q_{n-1}^o\Delta_\Sigma\right]u_{n-1}^o-Q_{n-1}^o\Delta_\mu+R_n^u, u n o − u n − 1 o = [ L ( Δ Σ ) ⊤ − Q n − 1 o Δ Σ ] u n − 1 o − Q n − 1 o Δ μ + R n u , W n o − W n − 1 o = [ L ( Δ Σ ) ⊤ − Q n − 1 o Δ Σ ] W n − 1 o + W n − 1 o [ L ( Δ Σ ) − Δ Σ Q n − 1 o ] − [ Q n − 1 o Δ μ ( u n − 1 o ) ⊤ + u n − 1 o Δ μ ⊤ Q n − 1 o + Q n − 1 o Δ Σ Q n − 1 o ] + R n W , W_n^o-W_{n-1}^o=\left[\mathcal{L}(\Delta_\Sigma)^\top-Q_{n-1}^o\Delta_\Sigma\right]W_{n-1}^o+W_{n-1}^o\left[\mathcal{L}(\Delta_\Sigma)-\Delta_\Sigma Q_{n-1}^o\right]
-\left[Q_{n-1}^o\Delta_\mu(u_{n-1}^o)^\top+u_{n-1}^o\Delta_\mu^\top Q_{n-1}^o+Q_{n-1}^o\Delta_\Sigma Q_{n-1}^o\right]+R_n^W, W n o − W n − 1 o = [ L ( Δ Σ ) ⊤ − Q n − 1 o Δ Σ ] W n − 1 o + W n − 1 o [ L ( Δ Σ ) − Δ Σ Q n − 1 o ] − [ Q n − 1 o Δ μ ( u n − 1 o ) ⊤ + u n − 1 o Δ μ ⊤ Q n − 1 o + Q n − 1 o Δ Σ Q n − 1 o ] + R n W , where the remainders satisfy
∥ R n u ∥ + ∥ R n W ∥ F ≲ α n 2 ( 1 + ∥ Z ‾ n ∥ 2 ) 2 ( 1 + ∥ u n − 1 o ∥ + ∥ W n − 1 o ∥ F ) . \lVert R_n^u\rVert+\lVert R_n^W\rVert_F\lesssim\alpha_n^2\left(1+\lVert\overline{Z}_n\rVert^2\right)^2\left(1+\lVert u_{n-1}^o\rVert+\lVert W_{n-1}^o\rVert_F\right). ∥ R n u ∥ + ∥ R n W ∥ F ≲ α n 2 ( 1 + ∥ Z n ∥ 2 ) 2 ( 1 + ∥ u n − 1 o ∥ + ∥ W n − 1 o ∥ F ) . Furthermore, we have
∥ u n o − u n − 1 o ∥ + ∥ W n o − W n − 1 o ∥ F ≲ α n ( 1 + ∥ Z ‾ n ∥ 2 ) ( 1 + ∥ u n − 1 o ∥ + ∥ W n − 1 o ∥ F ) . \lVert u_n^o-u_{n-1}^o\rVert+\lVert W_n^o-W_{n-1}^o\rVert_F\lesssim\alpha_n\left(1+\lVert\overline{Z}_n\rVert^2\right)\left(1+\lVert u_{n-1}^o\rVert+\lVert W_{n-1}^o\rVert_F\right). ∥ u n o − u n − 1 o ∥ + ∥ W n o − W n − 1 o ∥ F ≲ α n ( 1 + ∥ Z n ∥ 2 ) ( 1 + ∥ u n − 1 o ∥ + ∥ W n − 1 o ∥ F ) .
Proof. We handle the u n o u_n^o u n o case first as it is simpler. We can show that
u n o = Q n o D n − 1 ( u n − 1 − Δ μ ) . u_n^o=Q_n^oD_n^{-1}(u_{n-1}-\Delta_\mu). u n o = Q n o D n − 1 ( u n − 1 − Δ μ ) .
The presence of Q n o D n − 1 Q_n^oD_n^{-1} Q n o D n − 1 is the main technical challenge, and necessitates Lemma D.10 to bound its change. Substituting the expression for Q n o D n − 1 Q_n^oD_n^{-1} Q n o D n − 1 from Lemma D.10 gives the stated expansion for u n o − u n − 1 o u_n^o-u_{n-1}^o u n o − u n − 1 o , where R n u = R n D u n − 1 o − Δ Q Δ μ R_n^u=R_n^D u_{n-1}^o-\Delta_Q\Delta_\mu R n u = R n D u n − 1 o − Δ Q Δ μ and
∥ R n u ∥ ≤ ∥ R n D ∥ F ∥ u n − 1 o ∥ + ∥ Δ Q ∥ F ∥ Δ μ ∥ ≲ α n 2 ( 1 + ∥ Z ‾ n ∥ 2 ) 2 ( 1 + ∥ u n − 1 o ∥ ) \lVert R_n^u\rVert\leq\lVert R_n^D\rVert_F\lVert u_{n-1}^o\rVert+\lVert\Delta_Q\rVert_F\lVert\Delta_\mu\rVert\lesssim\alpha_n^2\left(1+\lVert\overline{Z}_n\rVert^2\right)^2\left(1+\lVert u_{n-1}^o\rVert\right) ∥ R n u ∥ ≤ ∥ R n D ∥ F ∥ u n − 1 o ∥ + ∥ Δ Q ∥ F ∥ Δ μ ∥ ≲ α n 2 ( 1 + ∥ Z n ∥ 2 ) 2 ( 1 + ∥ u n − 1 o ∥ )
from Lemma D.10 and (38). For ∥ u n o − u n − 1 o ∥ \lVert u_n^o-u_{n-1}^o\rVert ∥ u n o − u n − 1 o ∥ , the update and Lemma D.10 give
∥ u n o − u n − 1 o ∥ ≤ ∥ Δ Q ∥ F ∥ u n − 1 o ∥ + ∥ Q n o D n − 1 ∥ F ∥ Δ μ ∥ \lVert u_n^o-u_{n-1}^o\rVert\leq\lVert\Delta_Q\rVert_F\lVert u_{n-1}^o\rVert+\lVert Q_n^oD_n^{-1}\rVert_F\lVert\Delta_\mu\rVert ∥ u n o − u n − 1 o ∥ ≤ ∥ Δ Q ∥ F ∥ u n − 1 o ∥ + ∥ Q n o D n − 1 ∥ F ∥ Δ μ ∥
≲ α n ( 1 + ∥ Z ‾ n ∥ 2 ) ( 1 + ∥ u n − 1 o ∥ ) . \lesssim\alpha_n\left(1+\lVert\overline{Z}_n\rVert^2\right)\left(1+\lVert u_{n-1}^o\rVert\right). ≲ α n ( 1 + ∥ Z n ∥ 2 ) ( 1 + ∥ u n − 1 o ∥ ) .
Next, for W n o − W n − 1 o W_n^o-W_{n-1}^o W n o − W n − 1 o , the definition of W n W_n W n gives
W n = D n − 1 B n D n − ⊤ , B n = W n − 1 − Δ Σ − Δ μ u n − 1 ⊤ − u n − 1 Δ μ ⊤ + Δ μ Δ μ ⊤ . W_n=D_n^{-1}B_nD_n^{-\top},\qquad B_n=W_{n-1}-\Delta_\Sigma-\Delta_\mu u_{n-1}^\top-u_{n-1}\Delta_\mu^\top+\Delta_\mu\Delta_\mu^\top. W n = D n − 1 B n D n − ⊤ , B n = W n − 1 − Δ Σ − Δ μ u n − 1 ⊤ − u n − 1 Δ μ ⊤ + Δ μ Δ μ ⊤ .
Noting that Δ Q Q n − 1 o = Δ Q \Delta_QQ_{n-1}^o=\Delta_Q Δ Q Q n − 1 o = Δ Q , we can write
W n o = ( Q n − 1 o + Δ Q ) Q n − 1 o B n Q n − 1 o ( Q n − 1 o + Δ Q ) ⊤ . W_n^o=(Q_{n-1}^o+\Delta_Q)Q_{n-1}^oB_nQ_{n-1}^o(Q_{n-1}^o+\Delta_Q)^\top. W n o = ( Q n − 1 o + Δ Q ) Q n − 1 o B n Q n − 1 o ( Q n − 1 o + Δ Q ) ⊤ .
Expanding this expression and applying (41) gives the stated expansion for W n o − W n − 1 o W_n^o-W_{n-1}^o W n o − W n − 1 o , where R n W R_n^W R n W contains the terms quadratic or of higher order in ( Δ Σ , Δ μ ) (\Delta_\Sigma,\Delta_\mu) ( Δ Σ , Δ μ ) . Grouping the remainder according to the terms in B n B_n B n gives
∥ R n W ∥ F ≲ ( ∥ R n D ∥ F + ∥ Δ Q ∥ F 2 ) ∥ W n − 1 o ∥ F + ( ( ∥ Δ Q ∥ F + ∥ Δ Q ∥ F 2 ) ( ∥ Δ Σ ∥ F + ∥ Δ μ ∥ ∥ u n − 1 o ∥ ) + ∥ Δ μ ∥ 2 ) ≲ α n 2 ( 1 + ∥ Z ‾ n ∥ 2 ) ( 1 + ∥ u n − 1 o ∥ + ∥ W n − 1 o ∥ F ) . \lVert R_n^W\rVert_F \lesssim\left(\lVert R_n^D\rVert_F+\lVert\Delta Q\rVert_F^2\right)\lVert W_{n-1}^o\rVert_F
+\left(\left(\lVert\Delta Q\rVert_F+\lVert\Delta Q\rVert_F^2\right)\left(\lVert\Delta\Sigma\rVert_F+\lVert\Delta\mu\rVert\lVert u_{n-1}^o\rVert\right)+\lVert\Delta\mu\rVert^2\right)
\lesssim\alpha_n^2\left(1+\lVert\overline{Z}_n\rVert^2\right)\left(1+\lVert u_{n-1}^o\rVert+\lVert W_{n-1}^o\rVert_F\right). ∥ R n W ∥ F ≲ ( ∥ R n D ∥ F + ∥ Δ Q ∥ F 2 ) ∥ W n − 1 o ∥ F + ( ( ∥ Δ Q ∥ F + ∥ Δ Q ∥ F 2 ) ( ∥ ΔΣ ∥ F + ∥ Δ μ ∥ ∥ u n − 1 o ∥ ) + ∥ Δ μ ∥ 2 ) ≲ α n 2 ( 1 + ∥ Z n ∥ 2 ) ( 1 + ∥ u n − 1 o ∥ + ∥ W n − 1 o ∥ F ) .
using Lemma D.10 and (38) . In particular, ∥ Δ Q ∥ F 2 ≲ ∥ Δ Q ∥ F \lVert\Delta Q\rVert_F^2\lesssim\lVert\Delta Q\rVert_F ∥ Δ Q ∥ F 2 ≲ ∥ Δ Q ∥ F since ∥ Δ Q ∥ F ≲ 1 \lVert\Delta Q\rVert_F\lesssim1 ∥ Δ Q ∥ F ≲ 1 .
Finally, the exact update and the same bounds give
∥ W n o − W n − 1 o ∥ F ≲ ∥ Δ Σ ∥ F ( 1 + ∥ W n − 1 o ∥ F ) + ∥ Δ μ ∥ ∥ u n − 1 o ∥ + ∥ Δ μ ∥ 2 ≲ α n ( 1 + ∥ Z ‾ n ∥ 2 ) ( 1 + ∥ u n − 1 o ∥ + ∥ W n − 1 o ∥ F ) , \lVert W_n^o-W_{n-1}^o\rVert_F\lesssim\lVert\Delta\Sigma\rVert_F(1+\lVert W_{n-1}^o\rVert_F)+\lVert\Delta\mu\rVert\lVert u_{n-1}^o\rVert+\lVert\Delta\mu\rVert^2
\lesssim\alpha_n\left(1+\lVert\overline{Z}_n\rVert^2\right)\left(1+\lVert u_{n-1}^o\rVert+\lVert W_{n-1}^o\rVert_F\right), ∥ W n o − W n − 1 o ∥ F ≲ ∥ ΔΣ ∥ F ( 1 + ∥ W n − 1 o ∥ F ) + ∥ Δ μ ∥ ∥ u n − 1 o ∥ + ∥ Δ μ ∥ 2 ≲ α n ( 1 + ∥ Z n ∥ 2 ) ( 1 + ∥ u n − 1 o ∥ + ∥ W n − 1 o ∥ F ) ,
where we also used ∥ Δ μ ∥ 2 = ρ 2 β n 2 ∥ Z ‾ n ∥ 2 ≲ α n ∥ Z ‾ n ∥ 2 \lVert\Delta\mu\rVert^2=\rho^2\beta_n^2\lVert\overline{Z}_n\rVert^2\lesssim\alpha_n\lVert\overline{Z}_n\rVert^2 ∥ Δ μ ∥ 2 = ρ 2 β n 2 ∥ Z n ∥ 2 ≲ α n ∥ Z n ∥ 2 .
Lemma D.12. The error terms satisfy
1 + ∥ u n o ∥ + ∥ W n o ∥ F ≲ 1 + K n X , O . 1+\lVert u_n^o\rVert+\lVert W_n^o\rVert_F\lesssim1+K_n^{X,O}. 1 + ∥ u n o ∥ + ∥ W n o ∥ F ≲ 1 + K n X , O .
Proof. We begin with the multivariate counterpart to 1 + w n = u n 2 + v n 1+w_n=u_n^2+v_n 1 + w n = u n 2 + v n . By definition,
W n o + Q n o = u n o ( u n o ) ⊤ + Q n o L n − 1 Σ ∗ o L n − ⊤ Q n o . W_n^o+Q_n^o=u_n^o(u_n^o)^\top+Q_n^oL_n^{-1}\Sigma_*^oL_n^{-\top}Q_n^o. W n o + Q n o = u n o ( u n o ) ⊤ + Q n o L n − 1 Σ ∗ o L n − ⊤ Q n o .
Taking traces, using cyclicity, ( Q n o ) 2 = Q n o (Q_n^o)^2=Q_n^o ( Q n o ) 2 = Q n o and the definition of Q n o Q_n^o Q n o , gives
tr ( W n o + Q n o ) = ∥ u n o ∥ 2 + tr [ ( Σ n o ) − 1 A n o L n − 1 Σ ∗ o L n − ⊤ ( A n o ) ⊤ ] = ∥ u n o ∥ 2 + tr [ ( Σ n o ) − 1 Σ ∗ o ] . (45) \operatorname{tr}(W_n^o+Q_n^o)=\lVert u_n^o\rVert^2+\operatorname{tr}\left[(\Sigma_n^o)^{-1}A_n^oL_n^{-1}\Sigma_*^oL_n^{-\top}(A_n^o)^\top\right]
=\lVert u_n^o\rVert^2+\operatorname{tr}\left[(\Sigma_n^o)^{-1}\Sigma_*^o\right]. \tag*{(45)} tr ( W n o + Q n o ) = ∥ u n o ∥ 2 + tr [ ( Σ n o ) − 1 A n o L n − 1 Σ ∗ o L n − ⊤ ( A n o ) ⊤ ] = ∥ u n o ∥ 2 + tr [ ( Σ n o ) − 1 Σ ∗ o ] . (45)
where the last equality follows from ( A n o L n − 1 ) = S o (A_n^oL_n^{-1})=S^o ( A n o L n − 1 ) = S o . As in Lemma D.6, applying log v ≤ v / 2 \log v\le v/2 log v ≤ v /2 to the eigenvalues of ( Σ n o ) − 1 Σ ∗ o (\Sigma_n^o)^{-1}\Sigma_*^o ( Σ n o ) − 1 Σ ∗ o gives
K n X , O + ∣ o ∣ ≥ ∥ u n o ∥ 2 + 1 2 tr { ( Σ n o ) − 1 Σ ∗ o } = 1 2 [ ∥ u n o ∥ 2 + tr ( W n o + Q n o ) ] . K_n^{X,O}+|o|\ge\lVert u_n^o\rVert^2+\frac{1}{2}\operatorname{tr}\left\{(\Sigma_n^o)^{-1}\Sigma_*^o\right\}
=\frac{1}{2}\left[\lVert u_n^o\rVert^2+\operatorname{tr}(W_n^o+Q_n^o)\right]. K n X , O + ∣ o ∣ ≥ ∥ u n o ∥ 2 + 2 1 tr { ( Σ n o ) − 1 Σ ∗ o } = 2 1 [ ∥ u n o ∥ 2 + tr ( W n o + Q n o ) ] .
Since W n o + Q n o ⪰ 0 W_n^o+Q_n^o\succeq0 W n o + Q n o ⪰ 0 , its Frobenius norm is upper bounded by its trace, giving
∥ W n o + Q n o − Q n o ∥ F ≤ ∥ W n o + Q n o ∥ F + ∥ Q n o ∥ F ≤ tr ( W n o + Q n o ) + ∣ o ∣ . \lVert W_n^o+Q_n^o-Q_n^o\rVert_F\le\lVert W_n^o+Q_n^o\rVert_F+\lVert Q_n^o\rVert_F\le\operatorname{tr}(W_n^o+Q_n^o)+\sqrt{|o|}. ∥ W n o + Q n o − Q n o ∥ F ≤ ∥ W n o + Q n o ∥ F + ∥ Q n o ∥ F ≤ tr ( W n o + Q n o ) + ∣ o ∣ .
Together with ∥ u n o ∥ ≤ 1 + ∥ u n o ∥ 2 \lVert u_n^o\rVert\le1+\lVert u_n^o\rVert^2 ∥ u n o ∥ ≤ 1 + ∥ u n o ∥ 2 , this gives
1 + ∥ u n o ∥ + ∥ W n o ∥ F ≲ 1 + K n X , O , 1+\lVert u_n^o\rVert+\lVert W_n^o\rVert_F \lesssim1+K_n^{X,O}, 1 + ∥ u n o ∥ + ∥ W n o ∥ F ≲ 1 + K n X , O ,
where we used ∣ o ∣ ≤ q |o|\le q ∣ o ∣ ≤ q and K n X , O ≥ 0 K_n^{X,O}\ge0 K n X , O ≥ 0 .
Main proof
The remainder of the proof follows in a largely similar fashion to the univariate case, so we only outline the main differences. From (45) , we have
K n X , O = log ∣ Σ n o ∣ ∣ Σ o ∣ + tr ( W n o ) , K n X , O − K n − 1 X , O = log ∣ Σ n o ∣ ∣ Σ n − 1 o ∣ + tr ( W n o − W n − 1 o ) . K_n^{X,O}=\log\frac{|\Sigma_n^o|}{|\Sigma_o|}+\operatorname{tr}(W_n^o),\qquad K_n^{X,O}-K_{n-1}^{X,O}=\log\frac{|\Sigma_n^o|}{|\Sigma_{n-1}^o|}+\operatorname{tr}(W_n^o-W_{n-1}^o). K n X , O = log ∣ Σ o ∣ ∣ Σ n o ∣ + tr ( W n o ) , K n X , O − K n − 1 X , O = log ∣ Σ n − 1 o ∣ ∣ Σ n o ∣ + tr ( W n o − W n − 1 o ) .
Next, we have
log ∣ Σ n o ∣ ∣ Σ n − 1 o ∣ = log ∣ ( Σ n − 1 o ) − 1 Σ n o ∣ ≤ tr [ ( Σ n − 1 o ) − 1 ( Σ n o − Σ n − 1 o ) ] \log\frac{|\Sigma_n^o|}{|\Sigma_{n-1}^o|}=\log\left|(\Sigma_{n-1}^o)^{-1}\Sigma_n^o\right|\le\operatorname{tr}\left[(\Sigma_{n-1}^o)^{-1}(\Sigma_n^o-\Sigma_{n-1}^o)\right] log ∣ Σ n − 1 o ∣ ∣ Σ n o ∣ = log ( Σ n − 1 o ) − 1 Σ n o ≤ tr [ ( Σ n − 1 o ) − 1 ( Σ n o − Σ n − 1 o ) ]
where the inequality follows from log ( λ ) ≤ λ − 1 \log(\lambda)\le\lambda-1 log ( λ ) ≤ λ − 1 applied to the eigenvalues of ( Σ n − 1 o ) − 1 Σ n o (\Sigma_{n-1}^o)^{-1}\Sigma_n^o ( Σ n − 1 o ) − 1 Σ n o . Next, since Σ n o − Σ n − 1 o = A n − 1 o Δ Σ ( A n − 1 o ) T \Sigma_n^o-\Sigma_{n-1}^o=A_{n-1}^o\Delta_\Sigma(A_{n-1}^o)^\mathsf{T} Σ n o − Σ n − 1 o = A n − 1 o Δ Σ ( A n − 1 o ) T , we have
tr [ ( Σ n − 1 o ) − 1 ( Σ n o − Σ n − 1 o ) ] = tr [ ( A n − 1 o ) T ( Σ n − 1 o ) − 1 A n − 1 o Δ Σ ] = tr ( Q n − 1 o Δ Σ ) . \operatorname{tr}\left[(\Sigma_{n-1}^o)^{-1}(\Sigma_n^o-\Sigma_{n-1}^o)\right]=\operatorname{tr}\left[(A_{n-1}^o)^\mathsf{T}(\Sigma_{n-1}^o)^{-1}A_{n-1}^o\Delta_\Sigma\right]=\operatorname{tr}(Q_{n-1}^o\Delta_\Sigma). tr [ ( Σ n − 1 o ) − 1 ( Σ n o − Σ n − 1 o ) ] = tr [ ( A n − 1 o ) T ( Σ n − 1 o ) − 1 A n − 1 o Δ Σ ] = tr ( Q n − 1 o Δ Σ ) .
Next, apply Lemma D.11 to tr ( W n o − W n − 1 o ) \operatorname{tr}(W_n^o-W_{n-1}^o) tr ( W n o − W n − 1 o ) . Using the cyclicity of the trace, and the identities L ( Δ Σ ) + L ( Δ Σ ) T = Δ Σ \mathcal{L}(\Delta_\Sigma)+\mathcal{L}(\Delta_\Sigma)^\mathsf{T}=\Delta_\Sigma L ( Δ Σ ) + L ( Δ Σ ) T = Δ Σ and ( Q n − 1 o ) 2 = Q n − 1 o (Q_{n-1}^o)^2=Q_{n-1}^o ( Q n − 1 o ) 2 = Q n − 1 o , we obtain
tr ( W n o − W n − 1 o ) = − tr ( Δ Σ W n − 1 o ) − tr ( Q n − 1 o Δ Σ ) − 2 Δ μ T u n − 1 o + tr ( R n W ) , \operatorname{tr}(W_n^o-W_{n-1}^o)=-\operatorname{tr}(\Delta_\Sigma W_{n-1}^o)-\operatorname{tr}(Q_{n-1}^o\Delta_\Sigma)-2\Delta_\mu^\mathsf{T}u_{n-1}^o+\operatorname{tr}(R_n^W), tr ( W n o − W n − 1 o ) = − tr ( Δ Σ W n − 1 o ) − tr ( Q n − 1 o Δ Σ ) − 2 Δ μ T u n − 1 o + tr ( R n W ) ,
and the inequality on the log determinant ratio gives
K n X , O − K n − 1 X , O ≤ − tr ( Δ Σ W n − 1 o ) − 2 Δ μ T u n − 1 o + tr ( R n W ) . K_n^{X,O}-K_{n-1}^{X,O}\le-\operatorname{tr}(\Delta_\Sigma W_{n-1}^o)-2\Delta_\mu^\mathsf{T}u_{n-1}^o+\operatorname{tr}(R_n^W). K n X , O − K n − 1 X , O ≤ − tr ( Δ Σ W n − 1 o ) − 2 Δ μ T u n − 1 o + tr ( R n W ) .
The other difference is averaging the error bounds over o o o . Lemma D.12 gives
∑ o ∈ O p ∗ ( o ∣ x ) { 1 + ∥ u n − 1 o ( x ) ∥ + ∥ W n − 1 o ( x ) ∥ F } ≲ 1 + K n − 1 X ( x ) . (46) \sum_{o\in\mathcal{O}}p^*(o\mid x)\left\{1+\lVert u_{n-1}^o(x)\rVert+\lVert W_{n-1}^o(x)\rVert_F\right\}\lesssim1+K_{n-1}^X(x). \tag*{(46)} o ∈ O ∑ p ∗ ( o ∣ x ) { 1 + ∥ u n − 1 o ( x )∥ + ∥ W n − 1 o ( x ) ∥ F } ≲ 1 + K n − 1 X ( x ) . (46)
We now take the conditional expectation of K n X , O − K n − 1 X , O K_n^{X,O}-K_{n-1}^{X,O} K n X , O − K n − 1 X , O and integrate over o o o and x x x . (39) and (40) give the leading term as − α n T n − 1 -\alpha_n T_{n-1} − α n T n − 1 . The remainders are handled exactly as in Theorem D.1, additionally using ∣ tr ( R n W ) ∣ ≤ q ∥ R n W ∥ F |\operatorname{tr}(R_n^W)|\le\sqrt{q}\lVert R_n^W\rVert_F ∣ tr ( R n W ) ∣ ≤ q ∥ R n W ∥ F , Lemma D.9 and (46) . Also, ∥ Z ‾ n ∥ ≤ ∥ Z n ∥ \lVert\overline{Z}_n \rVert\le\lVert Z_n \rVert ∥ Z n ∥ ≤ ∥ Z n ∥ and Lemma D.8 give the same moment bound:
α n 2 E [ ( 1 + ∥ Z n ∥ 2 ) 2 ∣ F n − 1 ] ≲ n − 2 ( 1 − α 0 ρ 2 S ) . (47) \alpha_n^2 E[(1+\lVert Z_n\rVert^2)^2\mid\mathcal{F}_{n-1}]\lesssim n^{-2(1-\alpha_0\rho^2 S)}. \tag*{(47)} α n 2 E [( 1 + ∥ Z n ∥ 2 ) 2 ∣ F n − 1 ] ≲ n − 2 ( 1 − α 0 ρ 2 S ) . (47)
Here K n , T n ≥ 0 K_n,T_n\ge0 K n , T n ≥ 0 , and we can show K 0 < ∞ K_0<\infty K 0 < ∞ under Assumptions C.1 and D.1. The same almost-supermartingale argument therefore gives K n → K ∞ < ∞ K_n\to K_\infty<\infty K n → K ∞ < ∞ and ∑ n = 1 ∞ α n T n − 1 < ∞ \sum_{n=1}^{\infty}\alpha_nT_{n-1}<\infty ∑ n = 1 ∞ α n T n − 1 < ∞ P ∗ P^* P ∗ -a.s.
The convergence of T n T_n T n follows the same strategy as Theorem D.2, but with u ‾ n \overline{u}_n u n and W ‾ n \overline{W}_n W n , and applying Jensen’s inequality gives
∥ u ‾ n − u ‾ n − 1 ∥ ≤ ∑ o ∈ O p ∗ ( o ∣ x ) ∥ u n o − u n − 1 o ∥ \lVert\overline{u}_n-\overline{u}_{n-1}\rVert\le\sum_{o\in\mathcal{O}}p^*(o\mid x)\lVert u_n^o-u_{n-1}^o\rVert ∥ u n − u n − 1 ∥ ≤ o ∈ O ∑ p ∗ ( o ∣ x ) ∥ u n o − u n − 1 o ∥
and likewise for W ‾ n \overline{W}_n W n . The squared increments and remainder inner products have the same bounds, by Lemmas D.2 and D.11 , together with (46) and (47) . It remains to handle the leading terms. Taking conditional expectations in Lemma D.11 replaces Δ μ \Delta_\mu Δ μ and Δ Σ \Delta_\Sigma Δ Σ by α n ρ S u ‾ n − 1 \alpha_n\rho S\overline{u}_{n-1} α n ρS u n − 1 and α n ρ 2 S W ‾ n − 1 \alpha_n\rho^2S\overline{W}_{n-1} α n ρ 2 S W n − 1 with controllable remainders. Dropping the non-positive leading terms, we have
E [ T n ∣ F n − 1 ] − T n − 1 ≤ ∫ ∑ o ∈ O p ∗ ( o ∣ x ) h n ( x , o ) d P X o ( x ) + C n − 2 ( 1 − α 0 ρ 2 S ) ( 1 + K n − 1 ) 2 E[T_n\mid\mathcal{F}_{n-1}]-T_{n-1}\le\int\sum_{o\in\mathcal{O}}p^*(o\mid x)h_n(x,o)\,\mathrm{d}P_X^o(x)+Cn^{-2(1-\alpha_0\rho^2 S)}(1+K_{n-1})^2 E [ T n ∣ F n − 1 ] − T n − 1 ≤ ∫ o ∈ O ∑ p ∗ ( o ∣ x ) h n ( x , o ) d P X o ( x ) + C n − 2 ( 1 − α 0 ρ 2 S ) ( 1 + K n − 1 ) 2
where h n ( x , o ) h_n(x,o) h n ( x , o ) takes the form
h n = 4 α n ρ 3 ( u n − 1 o ) ⊤ [ L ( S W ‾ n − 1 ) − 2 ( S W ‾ n − 1 ) Q n − 1 o ] S u ‾ n − 1 + 4 α n ρ 4 tr [ ( S W ‾ n − 1 ) { L ( S W ‾ n − 1 ) ⊤ − Q n − 1 o ( S W ‾ n − 1 ) } W n − 1 o ] . \begin{aligned}
h_n={}&4\alpha_n\rho^3(u_{n-1}^o)^\top\left[\mathcal{L}(S\overline{W}_{n-1})-2(S\overline{W}_{n-1})Q_{n-1}^o\right]S\overline{u}_{n-1}\\
&+4\alpha_n\rho^4\operatorname{tr}\left[(S\overline{W}_{n-1})\{\mathcal{L}(S\overline{W}_{n-1})^\top-Q_{n-1}^o(S\overline{W}_{n-1})\}W_{n-1}^o\right].
\end{aligned} h n = 4 α n ρ 3 ( u n − 1 o ) ⊤ [ L ( S W n − 1 ) − 2 ( S W n − 1 ) Q n − 1 o ] S u n − 1 + 4 α n ρ 4 tr [ ( S W n − 1 ) { L ( S W n − 1 ) ⊤ − Q n − 1 o ( S W n − 1 )} W n − 1 o ] .
The first term is the multivariate analogue of the mixed term in (34) , while the second collects the remaining covariance contributions. Neither term has a definite sign, so we bound both in absolute value. Again, ∥ L ( A ) ∥ F ≤ ∥ A ∥ F \lVert\mathcal{L}(A)\rVert_F\le\lVert A\rVert_F ∥ L ( A ) ∥ F ≤ ∥ A ∥ F and ∥ Q n − 1 o A ∥ F ≤ ∥ A ∥ F \lVert Q_{n-1}^oA\rVert_F\le\lVert A\rVert_F ∥ Q n − 1 o A ∥ F ≤ ∥ A ∥ F , and Cauchy-Schwarz applied to the Frobenius inner product (trace term) gives
∣ h n ∣ ≲ α n ( ∥ u n − 1 o ∥ ∥ S W ‾ n − 1 ∥ F ∥ S u ‾ n − 1 ∥ + ∥ S W ‾ n − 1 ∥ F 2 ∥ W n − 1 o ∥ F ) ≲ α n ( ∥ u n − 1 o ∥ + ∥ W n − 1 o ∥ F ) ( ∥ S u ‾ n − 1 ∥ 2 + ∥ S W ‾ n − 1 ∥ F 2 ) . \begin{aligned}
\lvert h_n\rvert&\lesssim\alpha_n\left(\lVert u_{n-1}^o\rVert\lVert S\overline{W}_{n-1}\rVert_F\lVert S\overline{u}_{n-1}\rVert+\lVert S\overline{W}_{n-1}\rVert_F^2\lVert W_{n-1}^o\rVert_F\right)\\
&\lesssim\alpha_n\left(\lVert u_{n-1}^o\rVert+\lVert W_{n-1}^o\rVert_F\right)\left(\lVert S\overline{u}_{n-1}\rVert^2+\lVert S\overline{W}_{n-1}\rVert_F^2\right).
\end{aligned} ∣ h n ∣ ≲ α n ( ∥ u n − 1 o ∥ ∥ S W n − 1 ∥ F ∥ S u n − 1 ∥ + ∥ S W n − 1 ∥ F 2 ∥ W n − 1 o ∥ F ) ≲ α n ( ∥ u n − 1 o ∥ + ∥ W n − 1 o ∥ F ) ( ∥ S u n − 1 ∥ 2 + ∥ S W n − 1 ∥ F 2 ) .
where the second line follows from 2 ∣ a b ∣ ≤ a 2 + b 2 2\lvert ab\rvert\le a^2+b^2 2 ∣ ab ∣ ≤ a 2 + b 2 . Finally, Lemma D.2 implies ∥ S u ‾ n − 1 ∥ 2 ≲ ∥ u ‾ n − 1 ∥ s 2 \lVert S\overline{u}_{n-1}\rVert^2\lesssim\lVert\overline{u}_{n-1}\rVert_s^2 ∥ S u n − 1 ∥ 2 ≲ ∥ u n − 1 ∥ s 2 and ∥ S W ‾ n − 1 ∥ F 2 ≲ ∥ W ‾ n − 1 ∥ s 2 \lVert S\overline{W}_{n-1}\rVert_F^2\lesssim\lVert\overline{W}_{n-1}\rVert_s^2 ∥ S W n − 1 ∥ F 2 ≲ ∥ W n − 1 ∥ s 2 , which with Lemma D.12 gives
∣ h n ∣ ≲ α n ( 1 + K n − 1 X , O ) ( ∥ u ‾ n − 1 ∥ s 2 + ∥ W ‾ n − 1 ∥ s 2 ) . \lvert h_n\rvert\lesssim\alpha_n(1+K_{n-1}^{X,O})\left(\lVert\overline{u}_{n-1}\rVert_s^2+\lVert\overline{W}_{n-1}\rVert_s^2\right). ∣ h n ∣ ≲ α n ( 1 + K n − 1 X , O ) ( ∥ u n − 1 ∥ s 2 + ∥ W n − 1 ∥ s 2 ) .
Integrating over o o o and x x x gives
E [ T n ∣ F n − 1 ] − T n − 1 ≤ C α n ( 1 + K n − 1 ) T n − 1 + C n − 2 ( 1 − α 0 ρ 2 S ) ( 1 + K n − 1 ) 2 . E[T_n \mid\mathcal{F}_{n-1}] - T_{n-1} \le C\alpha_n(1+K_{n-1})T_{n-1} + Cn^{-2(1-\alpha_0\rho^2S)}(1+K_{n-1})^2. E [ T n ∣ F n − 1 ] − T n − 1 ≤ C α n ( 1 + K n − 1 ) T n − 1 + C n − 2 ( 1 − α 0 ρ 2 S ) ( 1 + K n − 1 ) 2 .
The conclusion T n → 0 T_n \to0 T n → 0 P ∗ P^* P ∗ -a.s. now follows exactly as in Theorem D.2.
Lemma D.2 applies component-wise under Assumptions 8 and 9. If T n = 0 T_n=0 T n = 0 , then u ˉ n = 0 \bar{u}_n=0 u ˉ n = 0 and W ‾ n = 0 \overline{W}_n=0 W n = 0 for P X ∗ P_X^* P X ∗ -almost all x x x , giving
0 = u n T u ˉ n = ∑ o ∈ O p ∗ ( o ∣ x ) ∥ u n o ∥ 2 , 0 = tr ( W n W ‾ n ) = ∑ o ∈ O p ∗ ( o ∣ x ) ∥ W n o ∥ F 2 . 0=u_n^\mathsf{T}\bar{u}_n=\sum_{o\in\mathcal{O}}p^*(o\mid x)\lVert u_n^o\rVert^2,\qquad
0=\operatorname{tr}(W_n\overline{W}_n)=\sum_{o\in\mathcal{O}}p^*(o\mid x)\lVert W_n^o\rVert_F^2. 0 = u n T u ˉ n = o ∈ O ∑ p ∗ ( o ∣ x ) ∥ u n o ∥ 2 , 0 = tr ( W n W n ) = o ∈ O ∑ p ∗ ( o ∣ x ) ∥ W n o ∥ F 2 .
The fully observed pattern with projection I q I_q I q has positive probability by Assumption D.1, so u n = W n = 0 u_n=W_n=0 u n = W n = 0 gives the final statement. The converse is immediate.
Norm strength
We now discuss the strength of the norm ∥ ⋅ ∥ s \lVert\cdot\rVert_s ∥ ⋅ ∥ s , focusing on the single-output case for convenience. Since ∥ S f ∥ ∞ ≲ ∥ f ∥ s \lVert Sf\rVert_\infty\lesssim\lVert f\rVert_s ∥ S f ∥ ∞ ≲ ∥ f ∥ s from Lemma D.2, we have thus shown that the smoothed error functions S u n Su_n S u n and S w n Sw_n S w n uniformly converge to 0 0 0 P ∗ P^* P ∗ -a.s. However, the similarity norm is weaker than the L 1 ( P X ∗ ) L^1(P_X^*) L 1 ( P X ∗ ) norm (e.g. ∥ f ∥ s ≲ ∥ f ∥ L ∗ 1 \lVert f\rVert_s\lesssim\lVert f\rVert_{L^1_*} ∥ f ∥ s ≲ ∥ f ∥ L ∗ 1 ), as we may have oscillatory functions which converge to 0 0 0 in ∥ ⋅ ∥ s \lVert\cdot\rVert_s ∥ ⋅ ∥ s but not in ∥ ⋅ ∥ L ∗ 1 \lVert\cdot\rVert_{L^1_*} ∥ ⋅ ∥ L ∗ 1 . A concrete example is f n ( x ) = sin ( 2 π n x ) f_n(x)=\sin(2\pi nx) f n ( x ) = sin ( 2 π n x ) with P X ∗ = Unif [ 0 , 1 ] P_X^*=\operatorname{Unif}[0,1] P X ∗ = Unif [ 0 , 1 ] . Since s ( x , x ′ ) ≤ S s(x,x')\le S s ( x , x ′ ) ≤ S from Lemma C.1, we have s ( x , ⋅ ) ∈ L ∗ 1 s(x,\cdot)\in L^1_* s ( x , ⋅ ) ∈ L ∗ 1 , so the Riemann-Lebesgue lemma gives for each x x x
( S f n ) ( x ) = ∫ 0 1 s ( x , x ′ ) sin ( 2 π n x ′ ) d x ′ → 0 (Sf_n)(x)=\int_0^1s(x,x')\sin(2\pi nx')\,dx'\to0 ( S f n ) ( x ) = ∫ 0 1 s ( x , x ′ ) sin ( 2 π n x ′ ) d x ′ → 0
Thus, ∥ f n ∥ s 2 ≤ ∫ 0 1 ∣ ( S f n ) ( x ) ∣ d x → 0 \lVert f_n\rVert_s^2\le\int_0^1|(Sf_n)(x)|\,dx\to0 ∥ f n ∥ s 2 ≤ ∫ 0 1 ∣ ( S f n ) ( x ) ∣ d x → 0 since ∣ S f n ∣ ≤ S |Sf_n|\le S ∣ S f n ∣ ≤ S is dominated. However, since ∥ f n ∥ L ∗ 1 = 2 / π \lVert f_n\rVert_{L^1_*}=2/\pi ∥ f n ∥ L ∗ 1 = 2/ π , we have convergence in ∥ ⋅ ∥ s \lVert\cdot\rVert_s ∥ ⋅ ∥ s but not ∥ ⋅ ∥ L ∗ 1 \lVert\cdot\rVert_{L^1_*} ∥ ⋅ ∥ L ∗ 1 . Nonetheless, we do not see u n u_n u n and w n w_n w n exhibiting this oscillatory behaviour in practice.
Convergence in the similarity norm can imply weak convergence in the function space under additional conditions. Let L ∗ 2 = L 2 ( P X ∗ ) L^2_*=L^2(P_X^*) L ∗ 2 = L 2 ( P X ∗ ) and f n , h ∈ L ∗ 2 f_n,h\in L^2_* f n , h ∈ L ∗ 2 . We say that f n f_n f n converges weakly to h h h in L ∗ 2 L^2_* L ∗ 2 , or f n ⇀ h f_n\rightharpoonup h f n ⇀ h in L ∗ 2 L^2_* L ∗ 2 , if
∫ f n ( x ) g ( x ) d P X ∗ ( x ) ⟶ ∫ h ( x ) g ( x ) d P X ∗ ( x ) \int f_n(x)g(x)\,dP_X^*(x)\longrightarrow\int h(x)g(x)\,dP_X^*(x) ∫ f n ( x ) g ( x ) d P X ∗ ( x ) ⟶ ∫ h ( x ) g ( x ) d P X ∗ ( x )
for every g ∈ L ∗ 2 g\in L^2_* g ∈ L ∗ 2 . We then have the following lemma.
Lemma D.13. Suppose Assumptions 1, 6, 8 and 9 hold. If sup n ∥ f n ∥ L ∗ 2 < ∞ \sup_n\lVert f_n\rVert_{L^2_*}<\infty sup n ∥ f n ∥ L ∗ 2 < ∞ , then ∥ f n ∥ s → 0 \lVert f_n\rVert_s\to0 ∥ f n ∥ s → 0 implies f n ⇀ 0 f_n\rightharpoonup0 f n ⇀ 0 in L ∗ 2 L^2_* L ∗ 2 .
Proof. Every subsequence is bounded in L ∗ 2 L^2_* L ∗ 2 and thus contains a further weakly convergent subsequence, say f n j ⇀ h f_{n_j} \rightharpoonup h f n j ⇀ h . As s ( x , x ′ ) s(x,x') s ( x , x ′ ) is uniformly bounded and thus s ( x , ⋅ ) ∈ L ∗ 2 s(x,\cdot) \in L^2_* s ( x , ⋅ ) ∈ L ∗ 2 , we may use it as a test function. For each x x x , we have
( S h ) ( x ) = lim j → ∞ ∫ f n j ( x ′ ) s ( x , x ′ ) d P X ∗ ( x ′ ) = lim j → ∞ ( S f n j ) ( x ) = 0 , (Sh)(x) = \lim_{j\to\infty} \int f_{n_j}(x')s(x,x')\,dP^*_X(x') = \lim_{j\to\infty}(Sf_{n_j})(x) = 0, ( S h ) ( x ) = j → ∞ lim ∫ f n j ( x ′ ) s ( x , x ′ ) d P X ∗ ( x ′ ) = j → ∞ lim ( S f n j ) ( x ) = 0 ,
where the last equality follows from ∥ S f n j ∥ ∞ ≲ ∥ f n j ∥ s → 0 \|Sf_{n_j}\|_\infty\lesssim\|f_{n_j}\|_s \to0 ∥ S f n j ∥ ∞ ≲ ∥ f n j ∥ s → 0 . Hence ∥ h ∥ s 2 = ∫ h ( S h ) d P X ∗ = 0 \|h\|^2_s = \int h(Sh)\,dP^*_X = 0 ∥ h ∥ s 2 = ∫ h ( S h ) d P X ∗ = 0 , and Lemma D.2 gives h = 0 h=0 h = 0 , since L ∗ 2 ⊂ L ∗ 1 L^2_* \subset L^1_* L ∗ 2 ⊂ L ∗ 1 . Every subsequence thus has a further subsequence converging weakly to 0, proving f n ⇀ 0 f_n \rightharpoonup0 f n ⇀ 0 in L ∗ 2 L^2_* L ∗ 2 .
Under the setting of Theorem 3 including Assumptions 8 and 9, we have K n X = u n 2 + v n − 1 − log v n ≥ u n 2 K^X_n = u_n^2 + v_n - 1 - \log v_n \ge u_n^2 K n X = u n 2 + v n − 1 − log v n ≥ u n 2 , so sup n ∥ u n ∥ L ∗ 2 2 ≤ sup n K n < ∞ \sup_n \|u_n\|^2_{L^2_*} \le\sup_n K_n < \infty sup n ∥ u n ∥ L ∗ 2 2 ≤ sup n K n < ∞ P ∗ P^* P ∗ -a.s. by Theorem D.1. Together with ∥ u n ∥ s → 0 \|u_n\|_s \to0 ∥ u n ∥ s → 0 , this gives u n ⇀ 0 u_n \rightharpoonup0 u n ⇀ 0 in L ∗ 2 L^2_* L ∗ 2 P ∗ P^* P ∗ -a.s. For w n w_n w n however, the existing bounds give sup n ∥ w n ∥ L ∗ 1 < ∞ \sup_n \|w_n\|_{L^1_*} < \infty sup n ∥ w n ∥ L ∗ 1 < ∞ P ∗ P^* P ∗ -a.s., but neither uniform L ∗ 2 L^2_* L ∗ 2 boundedness nor uniform integrability has been established. Thus, weak convergence of w n w_n w n in L ∗ 2 L^2_* L ∗ 2 , or even in L ∗ 1 L^1_* L ∗ 1 , would require further analysis. Finally, even if weak convergence for both u n u_n u n and w n w_n w n was shown, this would not directly translate into convergence statements about μ n \mu_n μ n and σ n 2 \sigma_n^2 σ n 2 , since squaring or taking reciprocals generally need not preserve weak convergence.
Additional Methodological Details
Similarity functions
The mixture construction in (2) also accommodates discrete covariates by taking k ( x ; ψ ) k(x;\psi) k ( x ; ψ ) to be a probability mass function. We give two examples below.
Example 5. For x ∈ { 1 , … , K } x \in\{1,\ldots,K\} x ∈ { 1 , … , K } , let k ( x ; ψ ) = ψ x k(x;\psi)=\psi_x k ( x ; ψ ) = ψ x and Ψ ∼ Dirichlet ( ϕ μ 1 , … , ϕ μ K ) \Psi\sim\operatorname{Dirichlet}(\phi\mu_1,\ldots,\phi\mu_K) Ψ ∼ Dirichlet ( ϕ μ 1 , … , ϕ μ K ) , where ϕ > 0 \phi>0 ϕ > 0 , μ k > 0 \mu_k>0 μ k > 0 and ∑ k μ k = 1 \sum_k \mu_k=1 ∑ k μ k = 1 . The Dirichlet moments give
s ( x , x ′ ) = 1 − ρ x + ρ x 1 ( x = x ′ ) μ x , ρ x = 1 ϕ + 1 . s(x,x') = 1-\rho_x+\rho_x\frac{\mathbf{1}(x=x')}{\mu_x}, \qquad\rho_x = \frac{1}{\phi+1}. s ( x , x ′ ) = 1 − ρ x + ρ x μ x 1 ( x = x ′ ) , ρ x = ϕ + 1 1 .
The components μ k \mu_k μ k can be set to empirical means, with smoothing if any categories have zero counts. Larger ρ x \rho_x ρ x thus places more weight on matching categories, especially those that are rare, and we may tune ρ x \rho_x ρ x prequentially.
Example 6. For counts x ∈ N 0 x \in\mathbb{N}_0 x ∈ N 0 , let k ( x ; ψ ) = e − ψ ψ x / x ! k(x;\psi)=e^{-\psi}\psi^x/x! k ( x ; ψ ) = e − ψ ψ x / x ! and Ψ ∼ Gamma ( ϕ μ , ϕ ) \Psi\sim\operatorname{Gamma}(\phi\mu,\phi) Ψ ∼ Gamma ( ϕ μ , ϕ ) , with mean μ > 0 \mu> 0 μ > 0 and rate ϕ > 0 \phi> 0 ϕ > 0 . Conjugacy gives
s ( x , x ′ ) = Γ ( ϕ μ ) Γ ( ϕ μ + x + x ′ ) Γ ( ϕ μ + x ) Γ ( ϕ μ + x ′ ) ( ϕ + 1 ) 2 ϕ μ + x + x ′ ϕ ϕ μ ( ϕ + 2 ) ϕ μ + x + x ′ . s(x,x') = \frac{\Gamma(\phi\mu)\Gamma(\phi\mu+x+x')}{\Gamma(\phi\mu+x)\Gamma(\phi\mu+x')} \frac{(\phi+1)^{2\phi\mu+x+x'}}{\phi^{\phi\mu}(\phi+2)^{\phi\mu+x+x'}}. s ( x , x ′ ) = Γ ( ϕ μ + x ) Γ ( ϕ μ + x ′ ) Γ ( ϕ μ ) Γ ( ϕ μ + x + x ′ ) ϕ ϕ μ ( ϕ + 2 ) ϕ μ + x + x ′ ( ϕ + 1 ) 2 ϕ μ + x + x ′ .
We may set μ \mu μ to the empirical mean and tune ϕ \phi ϕ prequentially.
Both kernel families (and the Gaussian example) are complete following Lehmann and Romano [2006, Theorem 4.3.1]. Finally, product kernels with independent mixing parameters give a product of coordinate-wise similarities. This allows continuous, categorical and count covariates to be combined in the same update.
General transformations
As discussed in the main paper, one may be interested in other transforms of the Gaussian random variable to obtain different families for p i ( y ∣ x ) p_i(y \mid x) p i ( y ∣ x ) , which then induce different moment updates. Our framework is quite general, as Proposition B.1 allows for general transformations. We now briefly discuss two choices beyond the symmetric SU family introduced in the main paper.
Skewed residuals
For skewed residuals, use the full SU distribution of Johnson [ [19 ] which generalizes the case in the main paper. Let λ = ( s , δ ) \lambda=(s,\delta) λ = ( s , δ ) with s ∈ R s \in\mathbb{R} s ∈ R and δ > 0 \delta> 0 δ > 0 , and define
g s , δ ( ϵ ) = sinh ( ϵ − s δ ) + ω sinh ( s δ ) D s , δ , D s , δ 2 = ω − 1 2 { 2 ω sinh 2 ( s δ ) + ω + 1 } . g_{s,\delta}(\epsilon) = \frac{\sinh\left(\frac{\epsilon-s}{\delta}\right) + \sqrt{\omega}\sinh\left(\frac{s}{\delta}\right)}{D_{s,\delta}}, \qquad D_{s,\delta}^{2} = \frac{\omega-1}{2}\left\{2\omega\sinh^{2}\left(\frac{s}{\delta}\right)+\omega+1\right\}. g s , δ ( ϵ ) = D s , δ sinh ( δ ϵ − s ) + ω sinh ( δ s ) , D s , δ 2 = 2 ω − 1 { 2 ω sinh 2 ( δ s ) + ω + 1 } .
and ω = e 1 / δ 2 \omega=e^{1/\delta^{2}} ω = e 1/ δ 2 as before. The parameter s s s introduces skewness, while δ \delta δ controls tail weight, and s = 0 s=0 s = 0 recovers (7) . We highlight that while the SU distribution cannot attain all admissible skewness-kurtosis pairs, it can attain the full skewness range as long as ω \omega ω grows appropriately. In practice, we may prequentially tune ( s , δ ) (s,\delta) ( s , δ ) jointly. The terms R s , δ ( 1 ) R_{s,\delta}^{(1)} R s , δ ( 1 ) and R s , δ ( 2 ) R_{s,\delta}^{(2)} R s , δ ( 2 ) also have similar closed forms:
R s , δ ( 1 ) ( ϵ i ) = ω ( 1 − ρ 2 ) / 2 sinh ( ρ ϵ i − s δ ) + ω sinh ( s δ ) D s , δ , R_{s,\delta}^{(1)}(\epsilon_i) = \frac{\omega^{(1-\rho^2)/2}\sinh\left(\frac{\rho\epsilon_i-s}{\delta}\right)+\sqrt{\omega}\sinh\left(\frac{s}{\delta}\right)}{D_{s,\delta}}, R s , δ ( 1 ) ( ϵ i ) = D s , δ ω ( 1 − ρ 2 ) /2 sinh ( δ ρ ϵ i − s ) + ω sinh ( δ s ) ,
R s , δ ( 2 ) ( ϵ i ) = [ R s , δ ( 1 ) ( ϵ i ) ] 2 + ( ω 1 − ρ 2 − 1 ) [ ω 1 − ρ 2 cosh ( 2 ( ρ ϵ i − s ) δ ) + 1 ] 2 D s , δ 2 . R_{s,\delta}^{(2)}(\epsilon_i) = \left[R_{s,\delta}^{(1)}(\epsilon_i)\right]^2 + \frac{\left(\omega^{1-\rho^2}-1\right)\left[\omega^{1-\rho^2}\cosh\left(\frac{2(\rho\epsilon_i-s)}{\delta}\right)+1\right]}{2D_{s,\delta}^{2}}. R s , δ ( 2 ) ( ϵ i ) = [ R s , δ ( 1 ) ( ϵ i ) ] 2 + 2 D s , δ 2 ( ω 1 − ρ 2 − 1 ) [ ω 1 − ρ 2 cosh ( δ 2 ( ρ ϵ i − s ) ) + 1 ] .
where ϵ i = s + δ sinh − 1 { D s , δ Z i − ω sinh ( s δ ) } \epsilon_i=s+\delta\sinh^{-1}\left\{D_{s,\delta}Z_i-\sqrt{\omega}\sinh\left(\frac{s}{\delta}\right)\right\} ϵ i = s + δ sinh − 1 { D s , δ Z i − ω sinh ( δ s ) } .
Platykurtic residuals
Consider the signed power transform [8 ]
g a ( ε ) = sign ( ε ) ∣ ε ∣ a C a , C a 2 = 2 a Γ ( a + 1 / 2 ) π , a > 0. g_a(\varepsilon) = \frac{\operatorname{sign}(\varepsilon)|\varepsilon|^a}{C_a}, \qquad C_a^2 = \frac{2^a\Gamma(a+1/2)}{\sqrt{\pi}}, \qquad a > 0. g a ( ε ) = C a sign ( ε ) ∣ ε ∣ a , C a 2 = π 2 a Γ ( a + 1/2 ) , a > 0.
This is again standardized, with skewness 0 and kurtosis equal to
E [ g a ( ε ) 4 ] = π Γ ( 2 a + 1 / 2 ) Γ ( a + 1 / 2 ) 2 . E[g_a(\varepsilon)^4] = \sqrt{\pi}\,\frac{\Gamma(2a+1/2)}{\Gamma(a+1/2)^2}. E [ g a ( ε ) 4 ] = π Γ ( a + 1/2 ) 2 Γ ( 2 a + 1/2 ) .
The range 0 < a < 1 0 < a < 1 0 < a < 1 gives platykurtic residuals, where a = 1 a = 1 a = 1 returns the Gaussian case. The terms R a ( 1 ) R_a^{(1)} R a ( 1 ) and R a ( 2 ) R_a^{(2)} R a ( 2 ) also have the forms:
R a ( 1 ) ( ε i ) = C a + 1 2 / 2 C a 2 ρ ε i ( 1 − ρ 2 ) ( a − 1 ) / 2 1 F 1 ( 1 − a 2 ; 3 2 ; − ρ 2 ε i 2 2 ( 1 − ρ 2 ) ) , R_a^{(1)}(\varepsilon_i) = \frac{C_{a+1}^{2}/2}{C_a^2}\rho\varepsilon_i(1-\rho^2)^{(a-1)/2}\,{}_1F_1\left(\frac{1-a}{2};\frac{3}{2};-\frac{\rho^2\varepsilon_i^2}{2(1-\rho^2)}\right), R a ( 1 ) ( ε i ) = C a 2 C a + 1 2 /2 ρ ε i ( 1 − ρ 2 ) ( a − 1 ) /2 1 F 1 ( 2 1 − a ; 2 3 ; − 2 ( 1 − ρ 2 ) ρ 2 ε i 2 ) ,
R a ( 2 ) ( ε i ) = ( 1 − ρ 2 ) a 1 F 1 ( − a ; 1 2 ; − ρ 2 ε i 2 2 ( 1 − ρ 2 ) ) , R_a^{(2)}(\varepsilon_i) = (1-\rho^2)^a\,{}_1F_1\left(-a;\frac{1}{2};-\frac{\rho^2\varepsilon_i^2}{2(1-\rho^2)}\right), R a ( 2 ) ( ε i ) = ( 1 − ρ 2 ) a 1 F 1 ( − a ; 2 1 ; − 2 ( 1 − ρ 2 ) ρ 2 ε i 2 ) ,
where 1 F 1 {}_1F_1 1 F 1 denotes the confluent hypergeometric function and ε i = sign ( Z i ) ( C a ∣ Z i ∣ ) 1 / a \varepsilon_i = \operatorname{sign}(Z_i)(C_a|Z_i|)^{1/a} ε i = sign ( Z i ) ( C a ∣ Z i ∣ ) 1/ a .
Multi-output
For the multi-output SU extension, we consider the family of p i ( y ∣ x ) p_i(y \mid x) p i ( y ∣ x ) induced by
Y = μ ( x ) + L ( x ) g δ ( ε ) , ε ∼ N ( 0 , I q ) , Y = \mu(x) + L(x)g_\delta(\varepsilon), \qquad\varepsilon\sim\mathcal{N}(0,I_q), Y = μ ( x ) + L ( x ) g δ ( ε ) , ε ∼ N ( 0 , I q ) ,
where g δ g_\delta g δ applies g δ g_\delta g δ as defined in (19) component-wise to the vector ε \varepsilon ε and L ( x ) L(x) L ( x ) is a Cholesky factor. The standardized SU components are independent, so E [ Y ∣ x ] = μ ( x ) E[Y \mid x] = \mu(x) E [ Y ∣ x ] = μ ( x ) and Var ( Y ∣ x ) = L ( x ) L ( x ) T \operatorname{Var}(Y \mid x) = L(x)L(x)^\mathsf{T} Var ( Y ∣ x ) = L ( x ) L ( x ) T .
We now outline the multivariate moment updates by extending Proposition 1. Let Z i Z_i Z i be as in (14) and set ε i ( j ) = g δ − 1 ( Z i ( j ) ) \varepsilon^{(j)}_i = g_\delta^{-1}(Z^{(j)}_i) ε i ( j ) = g δ − 1 ( Z i ( j ) ) . The triangular form implies that conditioning on Y ( 1 : j − 1 ) Y^{(1:j-1)} Y ( 1 : j − 1 ) is equivalent to conditioning on ε ( 1 : j − 1 ) \varepsilon^{(1:j-1)} ε ( 1 : j − 1 ) . As a result, the conditional CDFs in (11) are u i − 1 ( j ) = Φ ( ε ( j ) ) u^{(j)}_{i-1} = \Phi(\varepsilon^{(j)}) u i − 1 ( j ) = Φ ( ε ( j ) ) and v i − 1 ( j ) = Φ ( ε i ( j ) ) v^{(j)}_{i-1} = \Phi(\varepsilon^{(j)}_i) v i − 1 ( j ) = Φ ( ε i ( j ) ) , so the copula-weighted component is the distribution of the random vector
Y ~ i = μ i − 1 + L i − 1 g δ ( 1 − ρ 2 W + ρ ε i ) , W ∼ N ( 0 , I q ) . \widetilde{Y}_i = \mu_{i-1} + L_{i-1}g_\delta\left(\sqrt{1-\rho^2}W+\rho\varepsilon_i\right), \qquad W \sim\mathcal{N}(0,I_q). Y i = μ i − 1 + L i − 1 g δ ( 1 − ρ 2 W + ρ ε i ) , W ∼ N ( 0 , I q ) .
Using the univariate terms in (10) , write
R i = ( R δ ( 1 ) ( ε i ( j ) ) ) j = 1 q , V i = Diag { ( R δ ( 2 ) ( ε i ( j ) ) − R δ ( 1 ) ( ε i ( j ) ) 2 ) j = 1 q } . R_i = \left(R_\delta^{(1)}(\varepsilon_i^{(j)})\right)_{j=1}^{q}, \qquad V_i = \operatorname{Diag}\left\{\left(R_\delta^{(2)}(\varepsilon_i^{(j)})-R_\delta^{(1)}(\varepsilon_i^{(j)})^2\right)_{j=1}^{q}\right\}. R i = ( R δ ( 1 ) ( ε i ( j ) ) ) j = 1 q , V i = Diag { ( R δ ( 2 ) ( ε i ( j ) ) − R δ ( 1 ) ( ε i ( j ) ) 2 ) j = 1 q } .
Suppressing dependence on x x x , one can then show that the moment updates are
μ i = μ i − 1 + β i L i − 1 R i , Σ i = L i − 1 [ I q + β i { ( 1 − β i ) R i R i T − ( I q − V i ) } ] L i − 1 T . \begin{aligned}
\mu_i &= \mu_{i-1} + \beta_i L_{i-1}R_i,\qquad\Sigma_i = L_{i-1}\left[I_q + \beta_i\left\{(1-\beta_i)R_iR_i^\mathsf{T} - (I_q - V_i)\right\}\right]L_{i-1}^\mathsf{T}.
\end{aligned} μ i = μ i − 1 + β i L i − 1 R i , Σ i = L i − 1 [ I q + β i { ( 1 − β i ) R i R i T − ( I q − V i ) } ] L i − 1 T .
The term V i V_i V i arises from computing the expectation of the outer product g δ ( ε i ′ ) g δ ( ε i ′ ) T \mathbf{g}_{\delta}(\varepsilon_i')\mathbf{g}_{\delta}(\varepsilon_i')^\mathsf{T} g δ ( ε i ′ ) g δ ( ε i ′ ) T where ε i ′ = 1 − ρ 2 W + ρ ε i \varepsilon_i' = \sqrt{1-\rho^2}W+\rho\varepsilon_i ε i ′ = 1 − ρ 2 W + ρ ε i under W ∼ N ( 0 , I q ) W\sim\mathcal{N}(0,I_q) W ∼ N ( 0 , I q ) . As a sanity check, we recover the Gaussian update as δ → ∞ \delta\to\infty δ → ∞ since R i → ρ Z i R_i\to\rho Z_i R i → ρ Z i , and V i → ( 1 − ρ 2 ) I V_i\to(1-\rho^2)I V i → ( 1 − ρ 2 ) I .
Under order-compatible missingness, where O i = { 1 , … , r } O_i=\{1,\ldots,r\} O i = { 1 , … , r } , the remaining latent variables satisfy ε i ( r + 1 : q ) ∣ Y i ( 1 : r ) , X i ∼ N ( 0 , I q − r ) \varepsilon_i^{(r+1:q)}\mid Y_i^{(1:r)},X_i\sim\mathcal{N}(0,I_{q-r}) ε i ( r + 1 : q ) ∣ Y i ( 1 : r ) , X i ∼ N ( 0 , I q − r ) under the predictive model. Averaging the raw moment updates as in (15) then amounts to computing, for each j > r j>r j > r , E [ R i ( j ) ∣ Y i ( 1 : r ) , X i ] = 0 E\left[R_i^{(j)}\mid Y_i^{(1:r)},X_i\right]=0 E [ R i ( j ) ∣ Y i ( 1 : r ) , X i ] = 0 , and E [ V i ( j , j ) + ( R i ( j ) ) 2 ∣ Y i ( 1 : r ) , X i ] = 1 E\left[V_i^{(j,j)}+\left(R_i^{(j)}\right)^2\mid Y_i^{(1:r)},X_i\right]=1 E [ V i ( j , j ) + ( R i ( j ) ) 2 ∣ Y i ( 1 : r ) , X i ] = 1 in the above updates. This will result in closed form moment updates that are similar to the Gaussian case. For arbitrary missingness patterns, this simplification need not hold, and conditional integration under the SU family is required which may not be tractable. A potential approximation is to apply a similar projection by Q i Q_i Q i of the residuals as the Gaussian case, which we leave for future work.
Implementation details
We begin by discussing a few implementation details that apply to all the recursive moment updates. As discussed in the main paper, for prediction, we average over permutations on the raw moment scale, that is we average μ n \mu_n μ n and Σ n + μ n μ n T \Sigma_n+\mu_n\mu_n^\mathsf{T} Σ n + μ n μ n T before recovering the covariance. Predictive resampling then carries forward from the averaged moment functions. To avoid calculating s ( x , X i ) s(x,X_i) s ( x , X i ) repeatedly for i ≥ n + 1 i\ge n+1 i ≥ n + 1 , we use the same covariate sequence X n + 1 : N ∼ i i d P X X_{n+1:N}\overset{\mathrm{iid}}{\sim}P_X X n + 1 : N ∼ iid P X for each predictive resampling trajectory; see [2026] for further discussion of this approach.
Next, we discuss hyperparameter optimization. For the Gaussian-copula similarity, we use a separate bandwidth for each covariate with s ( x , x ′ ) = ∏ j = 1 p c ρ x ( j ) { Φ ( x ( j ) ) , Φ ( x ′ ( j ) ) } s(x,x')=\prod_{j=1}^{p}c_{\rho_x^{(j)}}\{\Phi(x^{(j)}),\Phi(x'^{(j)})\} s ( x , x ′ ) = ∏ j = 1 p c ρ x ( j ) { Φ ( x ( j ) ) , Φ ( x ′ ( j ) )} , where ρ x ( j ) = 0 \rho_x^{(j)}=0 ρ x ( j ) = 0 removes the contribution of covariate j j j . For the outcome, we use a single bandwidth ρ \rho ρ shared across coordinates in the multi-output case. The bandwidths ( ρ , ρ x ) (\rho,\rho_x) ( ρ , ρ x ) are selected jointly by maximizing the prequential log-likelihood averaged over permutations, and for the symmetric SU model, we also optimize δ \delta δ jointly with ( ρ , ρ x ) (\rho,\rho_x) ( ρ , ρ x ) . When there are missing outcomes, each summand in the prequential log-likelihood is the marginal predictive density of the observed coordinates instead. For optimization, we use L-BFGS-B with gradients obtained by automatic differentiation.
We conclude with a discussion on algorithmic implementation. In an ideal setting, evaluating the prequential log-likelihood only requires n ( n − 1 ) / 2 n(n-1)/2 n ( n − 1 ) /2 moment updates, but this would entail dynamically-sized arrays which are not supported by JAX’s scan operation. For larger datasets, a compromise is to evaluate the prequential objective in blocks, and discard the earlier blocks whose prequential contributions have already been evaluated, avoiding unnecessary subsequent updates. For the multi-output model, we store the mean components and the q ( q + 1 ) / 2 q(q + 1)/2 q ( q + 1 ) /2 lower-triangular entries of L i ( x ) L_i(x) L i ( x ) as arrays during updating, and only reconstruct Σ i ( x ) \Sigma_i(x) Σ i ( x ) afterwards. The Gaussian covariance update (13) involves computing the Cholesky factorization of a term I + c z z T I + czz^{\mathsf{T}} I + cz z T then scaling, allowing L i − 1 L_{i-1} L i − 1 to be updated to L i L_i L i in O ( q 2 ) O(q^2) O ( q 2 ) operations per observation and x x x -value, e.g. Hiptmair and Gradinaru [ [18 ] . This cost also holds under order-compatible missingness for (17) , since Q i Q_i Q i is diagonal so again we just have a rank-one Cholesky update. For arbitrary missingness patterns, Q i Q_i Q i is generally dense, and standard projection and factorization routines can incur a worst-case O ( q 3 ) O(q^3) O ( q 3 ) cost, although low-rank methods reduce this when few outputs are observed or missing. We leave the investigation of more efficient algorithms or approximations in the large q q q setting for future work.
Additional experimental details
Simulation details
In this section, we include additional details on the simulations. For the DDP baselines, we follow the notation of Antoniano-Villalobos et al. [ [4 ] for the hyperparameters. For the heavy-tailed settings, we use the same prior hyperparameters as the simulation study of Antoniano-Villalobos et al. [ [4 ] , with:
β 0 = ( 5 / 2 , 5 / 8 ) T , C − 1 = Diag ( 4 , 1 / 4 ) , α 1 = 1 , α 2 = 1 , μ 0 = 0 , c = 1 / 8 , a = 1 , b = 1. \begin{aligned}
\beta_0 &= (5/2, 5/8)^{\mathsf{T}}, \quad C^{-1} = \operatorname{Diag}(4, 1/4), \quad\alpha_1 = 1, \quad\alpha_2 = 1, \\
\mu_0 &= 0, \quad c = 1/8, \quad a = 1, \quad b = 1.
\end{aligned} β 0 μ 0 = ( 5/2 , 5/8 ) T , C − 1 = Diag ( 4 , 1/4 ) , α 1 = 1 , α 2 = 1 , = 0 , c = 1/8 , a = 1 , b = 1.
For the multivariate setting, we use similar settings to Wade and Inácio [ [40 ] , but extend it to 5 covariates:
β 0 = 0 6 , C − 1 = Var ( Y ) ‾ 0.0405 ⋅ Diag ( 10 , 1 , 1 , 1 , 1 , 1 ) , α 1 = 2 , α 2 = 0.0405 , μ 0 = 0 5 , c = ( 1 / 36 ) ⋅ 1 5 , a = 2 ⋅ 1 5 , b = 0.0375 ⋅ 1 5 . \begin{aligned}
\beta_0 &= \mathbf{0}_6, \quad C^{-1} = \frac{\overline{\operatorname{Var}(Y)}}{0.0405} \cdot\operatorname{Diag}(10, 1, 1, 1, 1, 1), \quad\alpha_1 = 2, \quad\alpha_2 = 0.0405, \\
\mu_0 &= \mathbf{0}_5, \quad c = (1/36) \cdot\mathbf{1}_5, \quad a = 2 \cdot\mathbf{1}_5, \quad b = 0.0375 \cdot\mathbf{1}_5.
\end{aligned} β 0 μ 0 = 0 6 , C − 1 = 0.0405 Var ( Y ) ⋅ Diag ( 10 , 1 , 1 , 1 , 1 , 1 ) , α 1 = 2 , α 2 = 0.0405 , = 0 5 , c = ( 1/36 ) ⋅ 1 5 , a = 2 ⋅ 1 5 , b = 0.0375 ⋅ 1 5 .
The Matlab implementation of Wade and Inácio [ [40 ] can be found here: https://github.com/sarawade/BayesXMix/tree/main/NormalizedWeights . For evaluating the MSE of the DDP approach, we use the posterior mean for estimating μ ∗ \mu_* μ ∗ . For σ ∗ 2 \sigma_*^2 σ ∗ 2 , we use the posterior median as the variance posterior is heavy-tailed and the posterior mean performs poorly. As the DDP runs are slow and must be carried out on a cluster, the run-times are for a single run on the reported local machine.
For the single-output GP in the simulations, we use a radial basis function (RBF) kernel with one length scale per dimension, and optimize the marginal likelihood to select the length scale as well as the noise variance and kernel amplitude, using a standard sklearn implementation. Here, the posterior mean is used as the point estimate for μ ∗ \mu_{\ast} μ ∗ , and the optimized noise variance is used for estimating σ ∗ 2 \sigma_{\ast}^{2} σ ∗ 2 .
For hyperparameter tuning in the recursive moment updates, we use 10 permutations for all examples with n = 200 n = 200 n = 200 , and 2 permutations for the larger examples with n = 600 n = 600 n = 600 and 1000. For actual fitting and prediction, 10 permutations are used in all cases. For predictive resampling, we set P X P_X P X to be uniform on the known compact X \mathcal{X} X , i.e. [ − 7 , 7 ] [-7, 7] [ − 7 , 7 ] for the univariate covariate simulation and [ − 2 , 2 ] 5 [-2, 2]^5 [ − 2 , 2 ] 5 for the multivariate covariate simulation. Predictive resampling uses N = n + 4000 N = n + 4000 N = n + 4000 and B = 1000 B = 1000 B = 1000 throughout. We naturally use the posterior means of μ ∗ \mu_{\ast} μ ∗ and σ ∗ 2 \sigma_{\ast}^{2} σ ∗ 2 as the point estimates for calculating the MSE. Note that while the posterior mean of μ ∗ \mu_{\ast} μ ∗ equals the fitted mean μ n \mu_n μ n due to the martingale, this is not necessarily the case for σ ∗ 2 \sigma_{\ast}^{2} σ ∗ 2 . For test prediction, we use the fitted predictive density p n ( y ∣ x ) p_n(y \mid x) p n ( y ∣ x ) , which can be computed without predictive resampling.
Weather dataset details
We begin by outlining the key data preprocessing steps to obtain our curated dataset from the WEATHER-5K dataset of Han et al. [2026]. We extracted the 1033 weather stations from the 48 contiguous US states by matching the weather station codes obtained from the Integrated Surface Database of the National Oceanic and Atmospheric Administration [35 ] . For each station and year, we averaged all TMP/DEW/SLP hourly measurements from the WEATHER-5K dataset to obtain the full dataset. To produce Figure 4 , Figure 5 and Figure 6 , country outlines were obtained from the Natural Earth website [21 ] .
Figure 6. Posterior mean of the mean function μ ( x ) \mu(x) μ ( x ) for TMP in the year 2021 for exact GPAR (left) and ICM with m = 256 m = 256 m = 256 (right).
For GPAR, we use the package implementation with both linear and nonlinear dependencies on preceding outputs. Since GPAR is order-dependent, we use the ordering (TMP, SLP, DEW), so prediction of DEW given the other outputs uses the final GP. We use 200 Monte Carlo samples to evaluate the predictive mean function, which is shown in Figure 6 (left) for TMP in the year 2021. We attribute the apparent roughness to Monte Carlo variation, and highlight that it is non-trivial to produce the GPAR equivalent of Figure 4 as moments of the joint output distribution generally lack closed forms.
For ICM, we use an RBF kernel with a separate length scale for each input dimension and a full-rank coregionalization matrix. Each output has its own Gaussian observation noise variance. Following the construction in the GPflow documentation, we augment the inputs with an output index and retain only observed responses, using SwitchedLikelihood for the output-specific likelihoods. For inference, we use the SVGP class for sparse variational inference which optimizes the variational parameters, hyperparameters and continuous coordinates of the m m m inducing inputs. The posterior mean of TMP for the ICM with m = 256 m = 256 m = 256 is shown in Figure 6 (right), which is quite similar to the RM approach.
Finally, we outline the evaluation metrics in Table 3 . Let D o b s \mathcal{D}_{\mathrm{obs}} D obs denote the training data, p n p_n p n denote each method’s predictive density, and I t e s t \mathcal{I}_{\mathrm{test}} I test and I D E W \mathcal{I}_{\mathrm{DEW}} I DEW denote the indices of the held-out full data points and DEW measurements respectively, with sizes n t e s t n_{\mathrm{test}} n test and n D E W n_{\mathrm{DEW}} n DEW . For the fully held-out data points, we compute
1 n t e s t ∑ i ∈ I t e s t log p n ( Y i ∣ X i , D o b s ) . \frac{1}{n_{\mathrm{test}}}\sum_{i\in\mathcal{I}_{\mathrm{test}}}\log p_n(Y_i\mid X_i,\mathcal{D}_{\mathrm{obs}}). n test 1 i ∈ I test ∑ log p n ( Y i ∣ X i , D obs ) .
For predicting DEW, writing Y ( 1 : 2 ) = ( T M P , S L P ) Y^{(1:2)} = (\mathrm{TMP},\mathrm{SLP}) Y ( 1 : 2 ) = ( TMP , SLP ) and Y ( 3 ) = D E W Y^{(3)} = \mathrm{DEW} Y ( 3 ) = DEW , we compute
1 n D E W ∑ i ∈ I D E W log p n ( Y i ( 3 ) ∣ Y i ( 1 : 2 ) , X i , D o b s ) . \frac{1}{n_{\mathrm{DEW}}}\sum_{i\in\mathcal{I}_{\mathrm{DEW}}}\log p_n\left(Y_i^{(3)}\mid Y_i^{(1:2)},X_i,\mathcal{D}_{\mathrm{obs}}\right). n DEW 1 i ∈ I DEW ∑ log p n ( Y i ( 3 ) ∣ Y i ( 1 : 2 ) , X i , D obs ) .
The methods incorporate the observed outputs differently for predicting DEW. For GPAR, ( X i , Y i ( 1 : 2 ) ) (X_i,Y_i^{(1:2)}) ( X i , Y i ( 1 : 2 ) ) enter as inputs to the final layer GP. Similarly, for RM (Multi-output), we apply the Gaussian conditional formulas to the joint p n ( y ∣ x ) p_n(y\mid x) p n ( y ∣ x ) , conditioning Y i ( 3 ) Y_i^{(3)} Y i ( 3 ) on Y i ( 1 : 2 ) Y_i^{(1:2)} Y i ( 1 : 2 ) . For ICM, conditional on the latent functions, the observation noises are independent. As Y i ( 1 : 2 ) Y_i^{(1:2)} Y i ( 1 : 2 ) is already included in D o b s \mathcal{D}_{\mathrm{obs}} D obs , their information enters through the posterior and no additional conditioning on Y i ( 1 : 2 ) Y_i^{(1:2)} Y i ( 1 : 2 ) is needed. Finally, RM (Independent) predicts Y i ( 3 ) Y_i^{(3)} Y i ( 3 ) independently given X i X_i X i .