Introduction
Construction of highly multivariate cross-covariances is of great contemporary interest in disparate applications, including environmental sciences, where recent technological advances have enabled measurements of multiple correlated variables over a spatial domain, and joint modeling is desired. One example is NASA’s MERRA-2 reanalysis, which incorporates observations from satellite instruments and provides global outputs of atmospheric variables at multiple vertical levels ([21 ] ). Statistical models have been used to study dependence in MERRA-2 fields ([42 , 56 , 45 ] ), and jointly predicting multiple MERRA-2 fields is also of interest ([26 ] ).
However, flexible approaches to multivariate cross-covariances ([24 , 3 ] ) unfortunately do not efficiently scale to a number of variables beyond single digits while retaining flexibility for the individual processes. As evident from the comprehensive review by Genton and Kleiber [22 ] , a key reason is that the traditional approaches construct a fully dense cross-covariance, and there is no notion of sparsity , in a suitable sense to be defined, that enables statistical estimation with respect to three main challenges: managing computation, reducing memory usage, and handling model parsimony and interpretability. The estimation of a fully dense highly multivariate cross-covariance is infeasible both computationally and statistically. In particular, likelihood estimation requires the repeated inversion of a matrix of size n p × n p np \times np n p × n p , where p p p is the number of variables or processes, and n n n is the average number of data locations per process, incurring a nominal computational cost of O ( n 3 p 3 ) O(n^{3}p^{3}) O ( n 3 p 3 ) floating point operations per likelihood evaluation, without further simplifications. Furthermore, memory storage of O ( n 2 p 2 ) O(n^{2}p^{2}) O ( n 2 p 2 ) is needed. Finally, the number of parameters grows quadratically in p p p , complicating numerical optimization of the likelihood for large p p p , and proposing a model that is both flexible and valid can be challenging. The large n n n problem has traditionally received attention in the Gaussian process (GP) literature ([50 , 15 , 28 ] ), as have the problems with asymmetry (see, e.g., [52 ] (Li and Zhang 2011)) or nonstationarity (see, e.g., [29 ] ) in cross-covariances; whereas the focus on the large p p p problem is still at a relatively nascent stage, which we review below.
A simple and traditional large p p p approach is the linear model of coregionalization ([25 ] ), which leads to quite restrictive smoothness properties of the component processes, in that the smoothness of any component defaults to that of the roughest ([22 ] ). More flexibly, Dey et al. [17 ] introduced a notion of stitching processes together at specific spatial locations, where conditional independence is enforced between different processes at other locations using an iterative proportional scaling approach ([47 ] ). Similarly, Peruzzi [40 ] uses the joint dependence at a reference set of locations to extend the process to the entire spatial domain. A potential disadvantage of reference set-based approaches is that estimates, as well as out-of-sample prediction, may depend on the reference set used for construction, including its ordering, i.e., the reference set becomes part of the model construction. If all model parameters are held fixed but two different reference sets are used, then the predictive distributions at a new location are generally different. Peruzzi [41 ] imposes sparsity post hoc via the graphical LASSO ([20 ] ) after full spatial estimation of the cross-covariance in a two-stage plug-in approach. Krock et al. [30 ] directly model a sparse precision matrix for latent basis coefficients. In this approach, one does not model the covariance functions parametrically, making covariance properties like process smoothness and tail decay unavailable, and prediction at unobserved spatial locations is not considered.
One suitable framework for introducing sparsity is the notion of process-level conditional independence, as introduced by Brillinger [8 ] and Dahlhaus [13 ] , which essentially amounts to sparsity in the inverse spectral density matrix. Approaches such as Dey et al. [17 ] attempt to model this sparsity from the spatial domain . In contrast, we develop a multivariate modeling approach in the spectral domain , directly targeting a sparse inverse spectral density, which results in a reference-free construction. Further, we use penalized likelihood inference to jointly estimate spatial dependence and sparsity in process dependence, avoiding a two-stage plug-in approach (e.g., Peruzzi, [41 ] ). Moreover, our modeling approach separately parameterizes the multivariate dependence, modeled via a sparse graphical model, and the process-specific covariance parameters. Our approach also enables process-level conditional independence for processes with different range parameters, as well as processes with different classes of covariance functions, a noted gap in Peruzzi [41 ] . Inference proceeds via a random Fourier features approach [43 , 18 , 19 , 38 ] , and a computationally-efficient stochastic approximation expectation–maximization (SAEM) scheme [16 ] for penalized maximum likelihood inference is developed.
In simulations, we consider both when the processes are observed at the same locations (the colocated setting) and when there are at least some locations not observed for all processes (the misaligned setting). Asymptotic results suggest that in the colocated setting, leveraging additional processes for prediction of a single process may not yield improvements in prediction performance [54 , 33 ] . However, many practical settings may have misaligned observations. While works in highly multivariate spatial processes have traditionally focused on the colocated setting [40 , 17 , 30 ] , we demonstrate improved performance under misalignment. In addition, we examine quantities that may be of interest for the combined processes (e.g., the sum of the processes). For example, practitioners may be interested in aerosol column mass, obtained by summing the weighted mixing ratios of aerosol components across atmospheric layers, or the total toxic load, calculated as the sum of toxicity levels for different chemicals at each location. In this case, accurately capturing dependence between variables is necessary for appropriate uncertainty quantification of the predictions. We show improvements over existing approaches in terms of estimation, prediction accuracy, and predictive uncertainty quantification. We demonstrate this methodology in an application to the MERRA-2 aerosol data set [21 ] . We find substantial improvements in prediction for our approach in the misaligned setting. Such misalignment can arise when MERRA-2 fields are combined with observations at different locations or spatial resolutions [9 , 55 ] . This demonstrates the benefits of modeling the underlying multivariate spatial process directly, rather than relying on independent univariate analyses.
Preliminaries and Prior Works
Let { Y ∘ ( s ) ∈ R p , s ∈ R d } \{Y^{\circ}(s) \in\mathbb{R}^{p}, s \in\mathbb{R}^{d}\} { Y ∘ ( s ) ∈ R p , s ∈ R d } be a multivariate spatial process with output dimension p p p and input dimension d d d . We refer to the entries { Y j ∘ ( s ) ∈ R , s ∈ R d } \{Y^{\circ}_{j}(s) \in\mathbb{R}, s \in\mathbb{R}^{d}\} { Y j ∘ ( s ) ∈ R , s ∈ R d } of Y ∘ ( s ) Y^{\circ}(s) Y ∘ ( s ) for j = 1 , … , p j = 1, \ldots, p j = 1 , … , p as individual or marginal processes. A principal consideration for Gaussian processes is the covariance structure. In particular, if we assume E [ Y ∘ ( s ) ] = μ ( s ) \mathbb{E}[Y^{\circ}(s)] = \boldsymbol{\mu}(s) E [ Y ∘ ( s )] = μ ( s ) for some μ ( s ) ∈ R p \boldsymbol{\mu}(s) \in\mathbb{R}^{p} μ ( s ) ∈ R p , and the processes are second-order, the multivariate covariance function is:
E [ ( Y ∘ ( s ) − μ ( s ) ) ( Y ∘ ( t ) − μ ( t ) ) T ] = C ( s , t ; θ ) , \mathbb{E}\left[\left(Y^{\circ}(s)-\boldsymbol{\mu}(s)\right)\left(Y^{\circ}(t)-\boldsymbol{\mu}(t)\right)^{\mathsf{T}}\right] = C(s,t;\boldsymbol{\theta}), E [ ( Y ∘ ( s ) − μ ( s ) ) ( Y ∘ ( t ) − μ ( t ) ) T ] = C ( s , t ; θ ) ,
depending on parameters θ \boldsymbol{\theta} θ . The R p × p \mathbb{R}^{p \times p} R p × p -valued covariance function C ( s , t ; θ ) C(s,t;\boldsymbol{\theta}) C ( s , t ; θ ) describes dependence both between and within processes at two locations. Let C ( s , t ; θ ) = [ C j k ( s , t ; θ ) ] j , k = 1 p C(s,t;\boldsymbol{\theta}) = [C_{jk}(s,t;\boldsymbol{\theta})]_{j,k=1}^{p} C ( s , t ; θ ) = [ C j k ( s , t ; θ ) ] j , k = 1 p , and we refer to C j j ( s , t ; θ ) C_{jj}(s,t;\boldsymbol{\theta}) C j j ( s , t ; θ ) as the marginal covariance function of process j j j and C j k ( s , t ; θ ) C_{jk}(s,t;\boldsymbol{\theta}) C j k ( s , t ; θ ) as the cross-covariance function between processes j j j and k k k . We say that { Y ∘ ( s ) } \{Y^{\circ}(s)\} { Y ∘ ( s )} is second-order stationary (or henceforth, stationary) if E [ Y ∘ ( s ) ] = μ \mathbb{E}[Y^{\circ}(s)] = \boldsymbol{\mu} E [ Y ∘ ( s )] = μ and C ( s , t ; θ ) = C ( t − s ; θ ) C(s,t;\boldsymbol{\theta}) = C(t-s;\boldsymbol{\theta}) C ( s , t ; θ ) = C ( t − s ; θ ) , so that the covariance function C ( ⋅ , ⋅ ; θ ) C(\cdot,\cdot;\boldsymbol{\theta}) C ( ⋅ , ⋅ ; θ ) only depends on the lag between t t t and s s s through t − s t-s t − s [11 ] . Throughout, we assume stationarity and that μ = 0 p \boldsymbol{\mu} = \mathbf{0}_{p} μ = 0 p .
In the stationary case, many multivariate covariances admit a spectral representation:
C ( h ; θ ) = ∫ R d e i h T x f ( x ; θ ) d x C(h;\theta) = \int_{\mathbb{R}^{d}} e^{i h^{\mathsf{T}} x} f(x;\theta) \, dx C ( h ; θ ) = ∫ R d e i h T x f ( x ; θ ) d x
where f ( x ; θ ) = [ f j k ( x ; θ ) ] j , k = 1 p \mathbf{f}(x;\theta) = [f_{jk}(x;\theta)]_{j,k=1}^{p} f ( x ; θ ) = [ f j k ( x ; θ ) ] j , k = 1 p is a matrix-valued spectral density and i i i is the imaginary unit. If f ( x ; θ ) \mathbf{f}(x;\theta) f ( x ; θ ) exists, the validity of the covariance C ( h ; θ ) C(h;\theta) C ( h ; θ ) may be shown by ensuring f ( x ; θ ) \mathbf{f}(x;\theta) f ( x ; θ ) is Hermitian and positive semidefinite for all x ∈ R d x \in\mathbb{R}^{d} x ∈ R d ; see, for example, § 2.1 of Yarger et al. [53 ] . The spectral density representation is a special case of multivariate Bochner’s theorem, ensuring multivariate validity when spectral densities exist. In this paper, we present an approach that accommodates any stationary marginal covariance function with a spectral density, including mixed families of processes.
A key motivation for our work is the notion of process-level conditional independence ([13 ] ). Let Y − j k ∘ ( s ) ∈ R p − 2 \mathbf{Y}^{\circ}_{-jk}(s) \in\mathbb{R}^{p-2} Y − j k ∘ ( s ) ∈ R p − 2 be all entries of Y ∘ ( s ) \mathbf{Y}^{\circ}(s) Y ∘ ( s ) except marginal processes j j j and k k k . Dahlhaus [13 ] defines process-level conditional independence for fixed j ≠ k j \ne k j = k as Y j ∘ ( s 1 ) ⊥ ⊥ Y k ∘ ( s 2 ) ∣ { Y − j k ∘ ( s ) , s ∈ R d } \mathbf{Y}^{\circ}_{j}(s_{1}) \perp\!\!\!\perp\mathbf{Y}^{\circ}_{k}(s_{2}) \mid\{\mathbf{Y}^{\circ}_{-jk}(s),s \in\mathbb{R}^{d}\} Y j ∘ ( s 1 ) ⊥ ⊥ Y k ∘ ( s 2 ) ∣ { Y − j k ∘ ( s ) , s ∈ R d } for all s 1 , s 2 ∈ R d s_{1},s_{2} \in\mathbb{R}^{d} s 1 , s 2 ∈ R d , and shows that this process-level conditional independence holds whenever the j , k j,k j , k entry of the inverse spectral density matrix f ( x ; θ ) − 1 \mathbf{f}(x;\theta)^{-1} f ( x ; θ ) − 1 is 0 for all x ∈ R d x \in\mathbb{R}^{d} x ∈ R d . While conceptually appealing, one cannot practically condition on all { Y − j k ∘ ( s ) , s ∈ R d } \{\mathbf{Y}^{\circ}_{-jk}(s),s \in\mathbb{R}^{d}\} { Y − j k ∘ ( s ) , s ∈ R d } for a direct spatial domain construction. However, by directly modeling in the spectral domain, process-level conditional independence enables computational tractability when the conditional independence graph of the processes is sparse. The key bridge between our spectral construction to the spatial domain estimation and computation is provided by random Fourier features (see, e.g., Rahimi and Recht [43 ] ).
Existing Approaches for Highly Multivariate Cross-covariances
Although already briefly summarized in § 1 , we review relevant approaches for modeling highly multivariate spatial data with a bit more technical detail. First, Cressie and ZammitMangion (2016) construct a multivariate covariance one process at a time. Each subsequent variable depends on earlier variables across space, with a separate covariance for its remaining unexplained variation. The paper further uses a potentially-sparse directed acyclic graph to express dependence among variables. Alternatively, the stitching approach of Dey et al. [17 ] uses a finite set of locations S \mathcal{S} S of size ∣ S ∣ |\mathcal{S}| ∣ S ∣ , referred to as reference or stitching locations, and a desired multivariate covariance function C ( h ; θ ) C(h; \theta) C ( h ; θ ) . They construct p 2 p^2 p 2 covariance matrices of size ∣ S ∣ × ∣ S ∣ |\mathcal{S}| \times|\mathcal{S}| ∣ S ∣ × ∣ S ∣ , denoted by M j k ( S , S ) M_{jk}(\mathcal{S}, \mathcal{S}) M j k ( S , S ) , for the reference locations such that conditional independence holds where desired for the spatial domain, and the covariances and cross-covariances match with M j k ( S , S ) = C j k ( S , S ) M_{jk}(\mathcal{S}, \mathcal{S}) = C_{jk}(\mathcal{S}, \mathcal{S}) M j k ( S , S ) = C j k ( S , S ) . The multivariate covariance is defined by:
M j k ( s 1 , s 2 ) = C j j ( s 1 , S ) C j j ( S , S ) − 1 M j k ( S , S ) C k k ( S , S ) − 1 C k k ( S , s 2 ) . (1) M_{jk}(s_1, s_2) = C_{jj}(s_1, \mathcal{S}) C_{jj}(\mathcal{S}, \mathcal{S})^{-1} M_{jk}(\mathcal{S}, \mathcal{S}) C_{kk}(\mathcal{S}, \mathcal{S})^{-1} C_{kk}(\mathcal{S}, s_2). \tag*{(1)} M j k ( s 1 , s 2 ) = C j j ( s 1 , S ) C j j ( S , S ) − 1 M j k ( S , S ) C k k ( S , S ) − 1 C k k ( S , s 2 ) . (1)
Essentially, the processes are stitched together at the reference locations S \mathcal{S} S , then independently extended to all locations. This approach encourages sparsity in the inverse cross-spectral density matrix through a sparse underlying graph G G G . However, the out-of-sample predictive variance computed in this manner at a new location s n + 1 ∗ s^*_{n+1} s n + 1 ∗ is dependent on the choice of reference set S \mathcal{S} S . The sample locations and reference locations S \mathcal{S} S may or may not be the same. Dey et al. [17 ] propose a chromatic Gibbs sampler for Bayesian estimation.
More recently, Peruzzi [40 ] proposed a similar approach with a few notable differences. Most importantly, they introduce an efficient likelihood estimation scheme that is computationally scalable to the setting where both n n n and p p p are large. The proposed model is similar to Dey et al. [17 ] in (1) , using reference locations S \mathcal{S} S with the choice of: M j k ( S , S ) = L j L k T M_{jk}(\mathcal{S}, \mathcal{S}) = L_j L_k^{\mathsf{T}} M j k ( S , S ) = L j L k T , where L j L_j L j is the lower-triangular Cholesky decomposition of C j j ( S , S ) C_{jj}(\mathcal{S}, \mathcal{S}) C j j ( S , S ) . Therefore, the model has the same dependence on the reference location set S \mathcal{S} S or its ordering. Peruzzi [41 ] examines spectral density structures that induce process-level conditional independence without a reference set. However, they note that their proposed framework does not accommodate general multivariate Matérn models with process-specific scale parameters. Whether these parameters can encode process-level conditional independence remains an open question that this paper addresses. For inference, Peruzzi [41 ] uses a two-stage plug-in approach.
Krock et al. [30 ] is conceptually the most similar to our work, in that they model directly in the frequency domain, obviating the need for a reference set. The p × p p \times p p × p inverse spectral density matrix f ( x ; θ ) − 1 \mathbf{f}(x;\boldsymbol{\theta})^{-1} f ( x ; θ ) − 1 is estimated directly and nonparametrically across x x x using the graphical LASSO ([20 ] ) and the fused graphical LASSO ([14 ] ) penalties. This handles nonstationarities in highly multivariate data efficiently. On the other hand, process-level conditional independence is not specifically encouraged, in favor of piecewise conditional independence in the spectral domain.
Approximating Cross-covariances by Random Fourier Features
We outline the model construction and key ideas of our paper by illustrating the approximation of a multivariate cross-covariance following Emery et al. [19 ] , where a full derivation is provided. We drop θ \boldsymbol{\theta} θ in the f ( x ; θ ) \mathbf{f}(x;\boldsymbol{\theta}) f ( x ; θ ) notation for simplified development. Applying exp ( i h T x ) = cos ( h T x ) + i sin ( h T x ) \exp(\mathrm{i}h^{\mathsf{T}}x) = \cos(h^{\mathsf{T}}x) + \mathrm{i}\sin(h^{\mathsf{T}}x) exp ( i h T x ) = cos ( h T x ) + i sin ( h T x ) , the overall goal is to approximate the integral:
E [ Y ∘ ( s ) Y ∘ ( s + h ) T ] = ∫ R d { cos ( h T x ) ℜ [ f ( x ) ] − sin ( h T x ) ℑ [ f ( x ) ] } d x , (2) \mathbb{E}\left[Y^{\circ}(s)Y^{\circ}(s+h)^{\mathsf{T}}\right] = \int_{\mathbb{R}^{d}} \left\{\cos(h^{\mathsf{T}}x)\Re[\mathbf{f}(x)] - \sin(h^{\mathsf{T}}x)\Im[\mathbf{f}(x)]\right\}\,\mathrm{d}x, \tag*{(2)} E [ Y ∘ ( s ) Y ∘ ( s + h ) T ] = ∫ R d { cos ( h T x ) ℜ [ f ( x )] − sin ( h T x ) ℑ [ f ( x )] } d x , (2)
using a set of random features in the spectral domain, where ℜ ( z ) \Re(z) ℜ ( z ) and ℑ ( z ) \Im(z) ℑ ( z ) are the real and imaginary parts of z z z , respectively. First consider a proposal probability density function g ( x ) g(x) g ( x ) with g ( x ) > 0 g(x) > 0 g ( x ) > 0 wherever f \mathbf{f} f is nonzero, and simulate X ℓ , j ∼ g ( ⋅ ) X_{\ell,j} \sim g(\cdot) X ℓ , j ∼ g ( ⋅ ) , ψ ℓ , j ∼ U [ 0 , 2 π ] \psi_{\ell,j} \sim U[0,2\pi] ψ ℓ , j ∼ U [ 0 , 2 π ] for j = 1 , … , p j = 1,\ldots,p j = 1 , … , p and ℓ = 1 , … , L \ell= 1,\ldots,L ℓ = 1 , … , L independently. Here, L L L indicates the size of the Fourier feature approximation and U U U denotes the uniform density. Emery et al. [19 ] then construct:
Y ∘ ( s ) = 1 L ∑ ℓ = 1 L ∑ j = 1 p a j ( X ℓ , j ) cos ( ⟨ X ℓ , j , s ⟩ + ψ ℓ , j ) + b j ( X ℓ , j ) sin ( ⟨ X ℓ , j , s ⟩ + ψ ℓ , j ) . (3) Y^{\circ}(s) = \frac{1}{\sqrt{L}}\sum_{\ell=1}^{L}\sum_{j=1}^{p} a_j(X_{\ell,j})\cos\left(\langle X_{\ell,j},s\rangle+ \psi_{\ell,j}\right) + b_j(X_{\ell,j})\sin\left(\langle X_{\ell,j},s\rangle+ \psi_{\ell,j}\right). \tag*{(3)} Y ∘ ( s ) = L 1 ℓ = 1 ∑ L j = 1 ∑ p a j ( X ℓ , j ) cos ( ⟨ X ℓ , j , s ⟩ + ψ ℓ , j ) + b j ( X ℓ , j ) sin ( ⟨ X ℓ , j , s ⟩ + ψ ℓ , j ) . (3)
The two terms of (3) correspond to those in (2) . With particular choices of a j ( x ) ∈ R p \mathbf{a}_j(x) \in\mathbb{R}^{p} a j ( x ) ∈ R p and b j ( x ) ∈ R p \mathbf{b}_j(x) \in\mathbb{R}^{p} b j ( x ) ∈ R p , Emery et al. [19 ] show that (3) has covariance (2) in expectation. The vectors a j ( x ) a_j(x) a j ( x ) and b j ( x ) b_j(x) b j ( x ) encode both the dependence relations between processes and information about the individual spectral densities. In particular, we let a j ( x ) a_j(x) a j ( x ) and b j ( x ) b_j(x) b j ( x ) denote the j j j -th columns of the matrices A ( x ) ∈ R p × p \mathbf{A}(x) \in\mathbb{R}^{p \times p} A ( x ) ∈ R p × p and B ( x ) ∈ R p × p \mathbf{B}(x) \in\mathbb{R}^{p \times p} B ( x ) ∈ R p × p , which in turn are the real and imaginary Cholesky components of the (weighted) spectral density matrix:
2 f ( x ) g ( x ) = ( A ( x ) + i B ( x ) ) ( A ( x ) + i B ( x ) ) ∗ , (4) \frac{2f(x)}{g(x)} = \left(\mathbf{A}(x) + \mathrm{i}\mathbf{B}(x)\right)\left(\mathbf{A}(x) + \mathrm{i}\mathbf{B}(x)\right)^{*}, \tag*{(4)} g ( x ) 2 f ( x ) = ( A ( x ) + i B ( x ) ) ( A ( x ) + i B ( x ) ) ∗ , (4)
where M ∗ M^{*} M ∗ denotes the conjugate transpose of the matrix M M M . The density g ( ⋅ ) g(\cdot) g ( ⋅ ) is thus a proposal in the spirit of importance sampling in order to target the matrix-valued integral
∫ R d e i ⟨ x , h ⟩ f ( x ) d x . \int_{\mathbb{R}^{d}} e^{\mathrm{i}\langle x,h\rangle} f(x)\,dx. ∫ R d e i ⟨ x , h ⟩ f ( x ) d x .
Emery and Arroyo [18 ] extend this work to nonstationary models, and Arroyo and Emery [5 ] describe the computational algorithm in more detail. This approach for approximate simulation has an advantage over exact circulant embedding approaches (e.g., [39 ] ) by providing simulations at arbitrary spatial locations.
In this paper’s context, the algorithm’s issue is the computational cost of O ( L p 4 + L p 2 n ) O(Lp^{4} + Lp^{2}n) O ( L p 4 + L p 2 n ) floating point operations, where O ( ⋅ ) O(\cdot) O ( ⋅ ) denotes the general order, and n n n is the number of simulation locations. This is prohibitive for large p p p , needing the Cholesky decomposition of the dense p × p p \times p p × p matrix 2 f ( X ℓ , j ) / g ( X ℓ , j ) 2f(X_{\ell,j})/g(X_{\ell,j}) 2 f ( X ℓ , j ) / g ( X ℓ , j ) , for each of the L p Lp L p summands in (3) .
Simulation and Likelihood via Fourier Features
Simulating Highly Multivariate Processes
We first propose an efficient simulation approach for highly multivariate GPs using random Fourier features. Our framework follows a similar, but subtly different construction to Emery et al. [19 ] . Let X ℓ X_{\ell} X ℓ and ψ ℓ \psi_{\ell} ψ ℓ be simulated as X ℓ , j X_{\ell,j} X ℓ , j and ψ ℓ , j \psi_{\ell,j} ψ ℓ , j in (3) , the matrices A ( x ) \mathbf{A}(x) A ( x ) and B ( x ) \mathbf{B}(x) B ( x ) be given by (4) , and each Z ℓ ∼ i i d N ( 0 p , I p ) Z_{\ell} \overset{\mathrm{iid}}{\sim} \mathcal{N}(\mathbf{0}_{p}, \mathbf{I}_{p}) Z ℓ ∼ iid N ( 0 p , I p ) . Our approach then constructs:
Y ∘ ( s ) = 1 L ∑ ℓ = 1 L A ( X ℓ ) Z ℓ cos ( ⟨ s , X ℓ ⟩ + ψ ℓ ) + 1 L ∑ ℓ = 1 L B ( X ℓ ) Z ℓ sin ( ⟨ s , X ℓ ⟩ + ψ ℓ ) . Y^{\circ}(s) = \frac{1}{\sqrt{L}} \sum_{\ell=1}^{L} \mathbf{A}(X_{\ell})Z_{\ell}\cos\left(\langle s,X_{\ell}\rangle+ \psi_{\ell}\right) + \frac{1}{\sqrt{L}} \sum_{\ell=1}^{L} \mathbf{B}(X_{\ell})Z_{\ell}\sin\left(\langle s,X_{\ell}\rangle+ \psi_{\ell}\right). Y ∘ ( s ) = L 1 ℓ = 1 ∑ L A ( X ℓ ) Z ℓ cos ( ⟨ s , X ℓ ⟩ + ψ ℓ ) + L 1 ℓ = 1 ∑ L B ( X ℓ ) Z ℓ sin ( ⟨ s , X ℓ ⟩ + ψ ℓ ) .
This construction replaces a sum over the columns of A ( X ℓ , j ) \mathbf{A}(X_{\ell,j}) A ( X ℓ , j ) and B ( X ℓ , j ) \mathbf{B}(X_{\ell,j}) B ( X ℓ , j ) in (3) with the matrix-vector products A ( X ℓ ) Z ℓ \mathbf{A}(X_\ell)Z_\ell A ( X ℓ ) Z ℓ and B ( X ℓ ) Z ℓ \mathbf{B}(X_\ell)Z_\ell B ( X ℓ ) Z ℓ . Naïvely, this reduces the number of Choleskys needed from L p Lp L p in (3) to L L L . Importantly, we need not construct Y ∘ ( s ) Y^{\circ}(s) Y ∘ ( s ) using A ( X ℓ ) \mathbf{A}(X_\ell) A ( X ℓ ) and B ( X ℓ ) \mathbf{B}(X_\ell) B ( X ℓ ) directly. Instead, we may backsolve against the Cholesky of the matrix g ( X ℓ ) f ( X ℓ ) − 1 / 2 g(X_\ell)f(X_\ell)^{-1/2} g ( X ℓ ) f ( X ℓ ) − 1/2 , where f ( x ) − 1 f(x)^{-1} f ( x ) − 1 is potentially sparse, encoding process-level conditional independence.
We thus formulate f ( x ) f(x) f ( x ) so that sparsity is directly interpretable. In particular, we choose:
f ( x ) = h ( x ) σ h ( x ) T , h ( x ) = diag ( f j ( x ) , j = 1 , … , p ) 1 / 2 , (5) f(x) = h(x)\boldsymbol{\sigma}h(x)^{\mathsf{T}}, \qquad h(x) = \operatorname{diag}(f_j(x),j=1,\ldots,p)^{1/2}, \tag*{(5)} f ( x ) = h ( x ) σ h ( x ) T , h ( x ) = diag ( f j ( x ) , j = 1 , … , p ) 1/2 , (5)
for a positive-definite matrix σ ∈ R p × p \boldsymbol{\sigma} \in\mathbb{R}^{p \times p} σ ∈ R p × p and unit-integral marginal spectral density functions { f j ( x ) , j = 1 , … , p } \{f_j(x),j=1,\ldots,p\} { f j ( x ) , j = 1 , … , p } . Since ∫ R d f j ( x ) d x = 1 \int_{\mathbb{R}^d} f_j(x)\,\mathrm{d}x = 1 ∫ R d f j ( x ) d x = 1 , all marginal process variances reside in σ \boldsymbol{\sigma} σ , and there is no scaling ambiguity between h ( x ) h(x) h ( x ) and σ \boldsymbol{\sigma} σ . In this setting, the Cholesky decomposition of f ( x ) − 1 f(x)^{-1} f ( x ) − 1 may be computed directly from the Cholesky of σ − 1 \boldsymbol{\sigma}^{-1} σ − 1 . Simulation thus requires computing a Cholesky decomposition once. Specifying process-level conditional independence is simple through zeros in σ − 1 \boldsymbol{\sigma}^{-1} σ − 1 , and validity is immediate ([53 ] ). For the Matérn model, this spectrally-separable model is similar to the parsimonious model of Gneiting et al. [24 ] . Terres et al. [48 ] , Bolin and Wallin [6 ] , Yarger et al. [53 ] , Guinness et al. [27 ] , and Peruzzi [41 ] propose similar spectrally-separable constructions of multivariate models. We give a detailed algorithm in Supplementary Section S.1, where we construct W ℓ : = A ( X ℓ ) Z ℓ ∼ N ( 0 p , 2 f ( X ℓ ) / g ( X ℓ ) ) W_\ell:= \mathbf{A}(X_\ell)Z_\ell\sim\mathcal{N}(\mathbf{0}_p,2f(X_\ell)/g(X_\ell)) W ℓ := A ( X ℓ ) Z ℓ ∼ N ( 0 p , 2 f ( X ℓ ) / g ( X ℓ )) . A key advance of the present work is to move from spectral domain process construction to spatial domain penalized likelihood estimation using random Fourier features as a bridge from the two domains.
Likelihood and Penalty for Highly Multivariate Processes
We next model the observed data by adding a nugget to the spatial process Y ∘ ( s ) Y^{\circ}(s) Y ∘ ( s ) . Defining the coefficients as c i , ℓ = L − 1 / 2 cos ( ⟨ s i , X ℓ ⟩ + ψ ℓ ) c_{i,\ell}=L^{-1/2}\cos(\langle s_i,X_\ell\rangle+\psi_\ell) c i , ℓ = L − 1/2 cos (⟨ s i , X ℓ ⟩ + ψ ℓ ) , we express the observation at spatial location s i s_i s i as Y ( s i ) = ∑ ℓ = 1 L c i , ℓ W ℓ + ϵ i \mathbf{Y}(s_i)=\sum_{\ell=1}^{L}c_{i,\ell}\mathbf{W}_{\ell}+\epsilon_i Y ( s i ) = ∑ ℓ = 1 L c i , ℓ W ℓ + ϵ i , where ϵ i ∼ N ( 0 p , d ) \epsilon_i\sim\mathcal{N}(\mathbf{0}_p,\mathbf{d}) ϵ i ∼ N ( 0 p , d ) represents the nugget vector with d = diag ( d 1 , … , d p ) \mathbf{d}=\operatorname{diag}(d_1,\ldots,d_p) d = diag ( d 1 , … , d p ) (Equation (5) is real-valued so B ( x ) = 0 p × p \mathbf{B}(x)=\mathbf{0}_{p\times p} B ( x ) = 0 p × p ). With the exception of Proposition 1, we condition on the realized features { ( X ℓ , ψ ℓ ) } \{(X_\ell,\psi_\ell)\} {( X ℓ , ψ ℓ )} from § 3.1 throughout all subsequent likelihood and expectation evaluations, leading to the following marginal log-likelihood of Y \mathbf{Y} Y after integrating out W \mathbf{W} W :
L ( θ ∣ Y ) = − 1 2 { n p log ( 2 π ) + log ( det [ Σ L ( θ ) + D ( θ ) ] ) + Y T [ Σ L ( θ ) + D ( θ ) ] − 1 Y } \mathcal{L}(\boldsymbol{\theta}\mid\mathbf{Y})=-\frac{1}{2}\left\{np\log(2\pi)+\log\left(\det\left[\boldsymbol{\Sigma}_L(\boldsymbol{\theta})+\mathbf{D}(\boldsymbol{\theta})\right]\right)+\mathbf{Y}^{\mathsf{T}}\left[\boldsymbol{\Sigma}_L(\boldsymbol{\theta})+\mathbf{D}(\boldsymbol{\theta})\right]^{-1}\mathbf{Y}\right\} L ( θ ∣ Y ) = − 2 1 { n p log ( 2 π ) + log ( det [ Σ L ( θ ) + D ( θ ) ] ) + Y T [ Σ L ( θ ) + D ( θ ) ] − 1 Y }
where [ Σ L ( θ ) ] i j = ∑ ℓ = 1 L c i , ℓ c j , ℓ Σ W ℓ [\boldsymbol{\Sigma}_L(\boldsymbol{\theta})]_{ij}=\sum_{\ell=1}^{L}c_{i,\ell}c_{j,\ell}\boldsymbol{\Sigma}_{W_\ell} [ Σ L ( θ ) ] ij = ∑ ℓ = 1 L c i , ℓ c j , ℓ Σ W ℓ with Σ W ℓ = 2 f ( X ℓ ) / g ( X ℓ ) ∈ R p × p \boldsymbol{\Sigma}_{W_\ell}=2f(X_\ell)/g(X_\ell)\in\mathbb{R}^{p\times p} Σ W ℓ = 2 f ( X ℓ ) / g ( X ℓ ) ∈ R p × p , D ( θ ) \mathbf{D}(\boldsymbol{\theta}) D ( θ ) is an n p × n p np\times np n p × n p diagonal matrix with entries { d j } j = 1 p \{d_j\}_{j=1}^{p} { d j } j = 1 p appropriately repeated along the diagonal, and Y = { Y ( s i ) } i = 1 n ∈ R n p \mathbf{Y}=\{\mathbf{Y}(s_i)\}_{i=1}^{n}\in\mathbb{R}^{np} Y = { Y ( s i ) } i = 1 n ∈ R n p . Here, θ \boldsymbol{\theta} θ includes σ − 1 \boldsymbol{\sigma}^{-1} σ − 1 , d \mathbf{d} d , and any marginal covariance parameters in h ( x ) \mathbf{h}(x) h ( x ) . For L ≪ n L\ll n L ≪ n , the Woodbury identity can reduce computational cost.
Even with suitable computational complexity, numerical optimization of the marginal log-likelihood is complicated by the high dimensionality of the parameter space, as σ − 1 \boldsymbol{\sigma}^{-1} σ − 1 contains p ( p + 1 ) / 2 = O ( p 2 ) p(p+1)/2=O(p^2) p ( p + 1 ) /2 = O ( p 2 ) parameters. To tackle this problem, we add a graphical LASSO penalty ([20 ] ) to the log-likelihood. In particular, consider:
L P ( θ ∣ Y ) = L ( θ ∣ Y ) − λ ∑ 1 ≤ j ≠ k ≤ p ∣ η j k ∣ = L ( θ ∣ Y ) − P ( θ ) , \mathcal{L}_{P}(\boldsymbol{\theta}\mid\mathbf{Y})=\mathcal{L}(\boldsymbol{\theta}\mid\mathbf{Y})-\lambda\sum_{1\leq j\ne k\leq p}|\eta_{jk}|=\mathcal{L}(\boldsymbol{\theta}\mid\mathbf{Y})-P(\boldsymbol{\theta}), L P ( θ ∣ Y ) = L ( θ ∣ Y ) − λ 1 ≤ j = k ≤ p ∑ ∣ η j k ∣ = L ( θ ∣ Y ) − P ( θ ) ,
where σ − 1 = [ η j k ] j , k = 1 p \boldsymbol{\sigma}^{-1}=[\eta_{jk}]_{j,k=1}^{p} σ − 1 = [ η j k ] j , k = 1 p and λ > 0 \lambda>0 λ > 0 is a tuning parameter. In this case, η j k = 0 \eta_{jk}=0 η j k = 0 directly corresponds to process-level conditional independence for processes j j j and k k k .
An immediate challenge is that standard graphical LASSO methodology is not directly applicable as W \mathbf{W} W is unobserved. The matrix σ \boldsymbol{\sigma} σ , defined in the spectral domain, does not necessarily correspond to the covariance matrix of Y ( s ) \mathbf{Y}(s) Y ( s ) , and the marginal precision of Y \mathbf{Y} Y may be dense even when σ − 1 \boldsymbol{\sigma}^{-1} σ − 1 is sparse. To address estimation with this penalty, we propose an SAEM algorithm ([51 , 16 ] ) in the following section that bridges the spectral domain construction and spatial domain estimation.
Inference Using Stochastic Approximation EM
For simplicity, the following derivations use colocated observations. Under misalignment, sums are restricted to observed entries and n n n is replaced by the process-specific sample sizes n j n_j n j . We formulate the penalized complete-data log-likelihood as follows:
L P ( θ ∣ Y , W ) = log ( f ( Y ∣ W , θ ) ) + log ( f ( W ∣ θ ) ) − λ ∑ 1 ≤ j ≠ k ≤ p ∣ η j k ∣ = C − n 2 log ∣ d ∣ − 1 2 ∑ i = 1 n ( Y ( s i ) − ∑ ℓ = 1 L c i , ℓ W ℓ ) T d − 1 ( Y ( s i ) − ∑ ℓ = 1 L c i , ℓ W ℓ ) − L 2 log ∣ σ ∣ \begin{aligned}
\mathcal{L}_P(\boldsymbol{\theta}\mid\mathbf{Y},\mathbf{W})=\log(f(\mathbf{Y}\mid\mathbf{W},\boldsymbol{\theta}))+\log(f(\mathbf{W}\mid\boldsymbol{\theta}))-\lambda\sum_{1\leq j\ne k\leq p}|\eta_{jk}| \\
&=C-\frac{n}{2}\log|\mathbf{d}|-\frac{1}{2}\sum_{i=1}^{n}\left(\mathbf{Y}(s_i)-\sum_{\ell=1}^{L}c_{i,\ell}\mathbf{W}_{\ell}\right)^{\mathsf{T}}\mathbf{d}^{-1}\left(\mathbf{Y}(s_i)-\sum_{\ell=1}^{L}c_{i,\ell}\mathbf{W}_{\ell}\right)-\frac{L}{2}\log|\boldsymbol{\sigma}|
\end{aligned} L P ( θ ∣ Y , W ) = log ( f ( Y ∣ W , θ )) + log ( f ( W ∣ θ )) − λ 1 ≤ j = k ≤ p ∑ ∣ η j k ∣ = C − 2 n log ∣ d ∣ − 2 1 i = 1 ∑ n ( Y ( s i ) − ℓ = 1 ∑ L c i , ℓ W ℓ ) T d − 1 ( Y ( s i ) − ℓ = 1 ∑ L c i , ℓ W ℓ ) − 2 L log ∣ σ ∣
− ∑ ℓ = 1 L log ∣ h ( X ℓ ) ∣ − 1 4 ∑ ℓ = 1 L g ( X ℓ ) W ℓ T h ( X ℓ ) − 1 σ − 1 h ( X ℓ ) − 1 W ℓ − λ ∑ 1 ≤ j ≠ k ≤ p ∣ η j k ∣ , (6) -\sum_{\ell=1}^{L}\log|\mathbf{h}(X_\ell)|-\frac{1}{4}\sum_{\ell=1}^{L}g(X_\ell)\mathbf{W}_{\ell}^{\mathsf{T}}\mathbf{h}(X_\ell)^{-1}\boldsymbol{\sigma}^{-1}\mathbf{h}(X_\ell)^{-1}\mathbf{W}_{\ell}-\lambda\sum_{1\leq j\ne k\leq p}|\eta_{jk}|, \tag*{(6)} − ℓ = 1 ∑ L log ∣ h ( X ℓ ) ∣ − 4 1 ℓ = 1 ∑ L g ( X ℓ ) W ℓ T h ( X ℓ ) − 1 σ − 1 h ( X ℓ ) − 1 W ℓ − λ 1 ≤ j = k ≤ p ∑ ∣ η j k ∣ , (6)
where C C C is a constant independent of θ \boldsymbol{\theta} θ . At step m m m , the E step of an EM approach requires:
Q ( θ ∣ θ ^ ( m − 1 ) ) = E [ L P ( θ ∣ Y , W ) ∣ Y , θ ^ ( m − 1 ) ] . Q\left(\boldsymbol{\theta}\mid\hat{\boldsymbol{\theta}}^{(m-1)}\right)=\mathbb{E}\left[\mathcal{L}_P(\boldsymbol{\theta}\mid\mathbf{Y},\mathbf{W})\mid\mathbf{Y},\hat{\boldsymbol{\theta}}^{(m-1)}\right]. Q ( θ ∣ θ ^ ( m − 1 ) ) = E [ L P ( θ ∣ Y , W ) ∣ Y , θ ^ ( m − 1 ) ] .
where θ ^ ( m − 1 ) \hat{\boldsymbol{\theta}}^{(m-1)} θ ^ ( m − 1 ) are the estimates from the previous iteration, while the M step maximizes Q ( θ ∣ θ ^ ( m − 1 ) ) Q\left(\boldsymbol{\theta}\mid\hat{\boldsymbol{\theta}}^{(m-1)}\right) Q ( θ ∣ θ ^ ( m − 1 ) ) over the parameters θ \boldsymbol{\theta} θ to form the new estimate θ ^ ( m ) \hat{\boldsymbol{\theta}}^{(m)} θ ^ ( m ) . However, computing the multivariate normal conditional distribution W ∣ Y , θ ^ ( m − 1 ) \mathbf{W}\mid\mathbf{Y},\hat{\boldsymbol{\theta}}^{(m-1)} W ∣ Y , θ ^ ( m − 1 ) directly requires solving against a matrix of size n p np n p , the size of the original computational problem. To ameliorate this challenge, we propose a Gibbs sampler-based E step, detailed next.
Markov Chain Monte Carlo E step
For a fixed EM step m m m , the Monte Carlo E step proceeds as follows. For a Monte Carlo sample of size T T T , we simulate W ( τ ) ∼ W ∣ Y , θ ^ ( m − 1 ) \mathbf{W}^{(\tau)}\sim\mathbf{W}\mid\mathbf{Y},\hat{\boldsymbol{\theta}}^{(m-1)} W ( τ ) ∼ W ∣ Y , θ ^ ( m − 1 ) for τ = 1 , 2 , … , T \tau=1,2,\ldots,T τ = 1 , 2 , … , T , suppressing the dependence of W ( τ ) \mathbf{W}^{(\tau)} W ( τ ) on m m m for simplicity. The estimate of Q ( θ ∣ θ ^ ( m − 1 ) ) Q\left(\boldsymbol{\theta}\mid\hat{\boldsymbol{\theta}}^{(m-1)}\right) Q ( θ ∣ θ ^ ( m − 1 ) ) is then:
Q ^ ( θ ∣ θ ^ ( m − 1 ) ) = 1 T ∑ τ = 1 T L P ( θ ∣ Y , W ( τ ) ) . \hat{Q}\left(\boldsymbol{\theta}\mid\hat{\boldsymbol{\theta}}^{(m-1)}\right)=\frac{1}{T}\sum_{\tau=1}^{T}\mathcal{L}_P\left(\boldsymbol{\theta}\mid\mathbf{Y},\mathbf{W}^{(\tau)}\right). Q ^ ( θ ∣ θ ^ ( m − 1 ) ) = T 1 τ = 1 ∑ T L P ( θ ∣ Y , W ( τ ) ) .
To generate samples of W ( τ ) \mathbf{W}^{(\tau)} W ( τ ) efficiently, we use Gibbs sampling. At the m m m -th iteration, we initialize the chain at the final Gibbs state from iteration m − 1 m-1 m − 1 and then perform $T sequential Gibbs sweeps. After setting an initial W ( 0 ) \mathbf{W}^{(0)} W ( 0 ) , in each step τ \tau τ , we update each W ℓ \mathbf{W}_{\ell} W ℓ sequentially: for ℓ = 1 , … , L \ell= 1,\ldots,L ℓ = 1 , … , L , we simulate W ℓ ( τ ) ∣ W − ℓ ( τ , p r e v ) , Y \mathbf{W}_{\ell}^{(\tau)} \mid\mathbf{W}_{-\ell}^{(\tau,\mathrm{prev})},\mathbf{Y} W ℓ ( τ ) ∣ W − ℓ ( τ , prev ) , Y where W − ℓ ( τ , p r e v ) = { W ℓ ′ ( τ ) , ℓ ′ < ℓ } ∪ { W ℓ ′ ( τ − 1 ) , ℓ ′ > ℓ } \mathbf{W}_{-\ell}^{(\tau,\mathrm{prev})} = \{\mathbf{W}_{\ell'}^{(\tau)},\ell' < \ell\} \cup\{\mathbf{W}_{\ell'}^{(\tau-1)},\ell' > \ell\} W − ℓ ( τ , prev ) = { W ℓ ′ ( τ ) , ℓ ′ < ℓ } ∪ { W ℓ ′ ( τ − 1 ) , ℓ ′ > ℓ } is the most recent value for all other ℓ ′ ≠ ℓ \ell' \ne\ell ℓ ′ = ℓ . This process continues for τ = 1 , … , T \tau= 1,\ldots,T τ = 1 , … , T . Samples from the distribution W ℓ ( τ ) ∣ W − ℓ ( τ , p r e v ) , Y \mathbf{W}_{\ell}^{(\tau)} \mid\mathbf{W}_{-\ell}^{(\tau,\mathrm{prev})},\mathbf{Y} W ℓ ( τ ) ∣ W − ℓ ( τ , prev ) , Y can be generated based on the Cholesky decomposition of size p p p via the Woodbury matrix identity. We detail the form of the multivariate Gaussian conditional distribution for W ℓ ( τ ) \mathbf{W}_{\ell}^{(\tau)} W ℓ ( τ ) in Supplementary Section S.3.
Conditional Maximization Steps
We detail the updates for the parameters σ − 1 \boldsymbol{\sigma}^{-1} σ − 1 , { d j } j = 1 p \{d_j\}_{j=1}^{p} { d j } j = 1 p , and ι \boldsymbol{\iota} ι , which denotes the finite-dimensional spatial parameters included in h ( x ) \boldsymbol{h}(x) h ( x ) . We perform three conditional maximization steps in an alternating fashion [37 ] :
CM1. (The precision matrix ). Since Y ( s ) ∣ W , θ \mathbf{Y}(s) \mid\mathbf{W},\boldsymbol{\theta} Y ( s ) ∣ W , θ in Equation (6) is only a function of the nugget d \boldsymbol{d} d , the update for σ \boldsymbol{\sigma} σ is:
σ ( m ) = arg min σ ≻ 0 { L 2 log ∣ σ ∣ + 1 4 T ∑ τ = 1 T ∑ ℓ = 1 L g ( X ℓ ) W ℓ ( τ ) T h ( X ℓ ) − 1 σ − 1 h ( X ℓ ) − 1 W ℓ ( τ ) + λ ∑ 1 ≤ j ≠ k ≤ p ∣ η j k ∣ } . \boldsymbol{\sigma}^{(m)} = \underset{\boldsymbol{\sigma} \succ0}{\arg\min}\left\{\frac{L}{2}\log|\boldsymbol{\sigma}|+\frac{1}{4T}\sum_{\tau=1}^{T}\sum_{\ell=1}^{L}g(X_{\ell})\mathbf{W}_{\ell}^{(\tau)\mathsf{T}}\boldsymbol{h}(X_{\ell})^{-1}\boldsymbol{\sigma}^{-1}\boldsymbol{h}(X_{\ell})^{-1}\mathbf{W}_{\ell}^{(\tau)}+\lambda\sum_{1\leq j\ne k\leq p}|\eta_{jk}|\right\}. σ ( m ) = σ ≻ 0 arg min ⎩ ⎨ ⎧ 2 L log ∣ σ ∣ + 4 T 1 τ = 1 ∑ T ℓ = 1 ∑ L g ( X ℓ ) W ℓ ( τ ) T h ( X ℓ ) − 1 σ − 1 h ( X ℓ ) − 1 W ℓ ( τ ) + λ 1 ≤ j = k ≤ p ∑ ∣ η j k ∣ ⎭ ⎬ ⎫ .
which can be recognized as a graphical LASSO problem with an estimated second-moment matrix ∑ ℓ = 1 L g ( X ℓ ) h ( X ℓ ) − 1 U ‾ m , ℓ h ( X ℓ ) − 1 / ( 2 L ) \sum_{\ell=1}^{L}g(X_{\ell})\boldsymbol{h}(X_{\ell})^{-1}\overline{\mathbf{U}}_{m,\ell}\boldsymbol{h}(X_{\ell})^{-1}/(2L) ∑ ℓ = 1 L g ( X ℓ ) h ( X ℓ ) − 1 U m , ℓ h ( X ℓ ) − 1 / ( 2 L ) and a penalty parameter ρ L = 2 λ / L \rho_L = 2\lambda/L ρ L = 2 λ / L , where U ‾ m , ℓ = ∑ τ = 1 T W ℓ ( τ ) W ℓ ( τ ) T / T \overline{\mathbf{U}}_{m,\ell} = \sum_{\tau=1}^{T}\mathbf{W}_{\ell}^{(\tau)}\mathbf{W}_{\ell}^{(\tau)\mathsf{T}}/T U m , ℓ = ∑ τ = 1 T W ℓ ( τ ) W ℓ ( τ ) T / T is the Monte Carlo estimate of the conditional expectation of the complete-data sufficient statistic.
CM2. (The measurement error variance ). Holding other parameters fixed, d \boldsymbol{d} d is updated using the log f ( Y ∣ W , θ ) \log f(\mathbf{Y}\mid\mathbf{W},\boldsymbol{\theta}) log f ( Y ∣ W , θ ) term. The update for the j j j -th entry of d \boldsymbol{d} d at iteration m m m is
d ^ j ( m ) = 1 n T ∑ τ = 1 T ∑ i = 1 n ( Y j ( s i ) − ∑ ℓ = 1 L c i , ℓ W ℓ , j ( τ ) ) 2 , \widehat{d}_{j}^{(m)}=\frac{1}{nT}\sum_{\tau=1}^{T}\sum_{i=1}^{n}\left(Y_j(s_i)-\sum_{\ell=1}^{L}c_{i,\ell}W_{\ell,j}^{(\tau)}\right)^2, d j ( m ) = n T 1 τ = 1 ∑ T i = 1 ∑ n ( Y j ( s i ) − ℓ = 1 ∑ L c i , ℓ W ℓ , j ( τ ) ) 2 ,
where W ℓ , j ( τ ) W_{\ell,j}^{(\tau)} W ℓ , j ( τ ) is the j j j -th entry of W ℓ ( τ ) \mathbf{W}_{\ell}^{(\tau)} W ℓ ( τ ) .
CM3. (Parameters governing spatial covariance ). The spatial parameters ι \iota ι governing the individual processes through the spectral densities { f j ( x ) } \{f_j(x)\} { f j ( x )} are only related to log ( f ( W ∣ θ ) ) \log(f(\mathbf{W}\mid\boldsymbol{\theta})) log ( f ( W ∣ θ )) . Holding ι k \iota_k ι k , k ≠ j k\ne j k = j fixed and cycling through j = 1 , … , p j=1,\ldots,p j = 1 , … , p , yields:
ι ^ j ( m ) = arg max ι j { − ∑ ℓ = 1 L log ∣ h ( X ℓ ) ∣ − 1 4 ∑ ℓ = 1 L g ( X ℓ ) ( η j j U ‾ m , ℓ ) j j f j ( X ℓ ) + 2 ∑ k ≠ j η j k ( U ‾ m , ℓ ) j k f k ( X ℓ ) f j ( X ℓ ) ) } . \widehat{\iota}_j^{(m)}=\underset{\iota_j}{\arg\max}\left\{-\sum_{\ell=1}^{L}\log\left\lvert h(X_\ell)\right\rvert-\frac{1}{4}\sum_{\ell=1}^{L}g(X_\ell)\left(\eta_{jj}\frac{\overline{U}_{m,\ell})_{jj}}{f_j(X_\ell)}+2\sum_{k\ne j}\eta_{jk}\frac{(\overline{U}_{m,\ell})_{jk}}{\sqrt{f_k(X_\ell)f_j(X_\ell)}}\right)\right\}. ι j ( m ) = ι j arg max ⎩ ⎨ ⎧ − ℓ = 1 ∑ L log ∣ h ( X ℓ ) ∣ − 4 1 ℓ = 1 ∑ L g ( X ℓ ) η j j f j ( X ℓ ) U m , ℓ ) j j + 2 k = j ∑ η j k f k ( X ℓ ) f j ( X ℓ ) ( U m , ℓ ) j k ⎭ ⎬ ⎫ .
This coordinatewise approach also accommodates more flexible spatial models, such as those with nonstationarity ([18 ] ).
Stochastic Approximation Updates
To ensure convergence, the MCEM algorithm requires the Monte Carlo sample size T T T to increase over iterations. One alternative approach is the stochastic approximation EM (SAEM) algorithm, which does not require T T T to increase with iterations by averaging E step estimates ([16 ] ). We consider the stochastic approximation expectation conditional maximization (SAECM) algorithm. In the m m m -th iteration, let u ℓ ( m ) \mathbf{u}_{\ell}^{(m)} u ℓ ( m ) and d ^ S A ( m ) \widehat{d}_{SA}^{(m)} d S A ( m ) denote the stochastic approximation estimates of the complete-data sufficient statistic. Define
Q ^ S A ( m ) ( θ ) = ( 1 − γ m ) Q ^ S A ( m − 1 ) ( θ ) + γ m Q ^ ( θ ∣ θ ^ ( m − 1 ) ) , \widehat{Q}_{SA}^{(m)}(\boldsymbol{\theta})=(1-\gamma_m)\widehat{Q}_{SA}^{(m-1)}(\boldsymbol{\theta})+\gamma_m\widehat{Q}\left(\boldsymbol{\theta}\mid\widehat{\boldsymbol{\theta}}^{(m-1)}\right), Q S A ( m ) ( θ ) = ( 1 − γ m ) Q S A ( m − 1 ) ( θ ) + γ m Q ( θ ∣ θ ( m − 1 ) ) ,
where { γ m } \{\gamma_m\} { γ m } is a nonincreasing sequence with γ 1 = 1 \gamma_1=1 γ 1 = 1 . Then, for a set of parameters θ = ( θ 1 , … , θ k ) \boldsymbol{\theta}=(\theta_1,\ldots,\theta_k) θ = ( θ 1 , … , θ k ) , SAECM cycles through parameters: θ ^ j ( m ) = arg max θ j Q ^ S A ( m ) ( θ j ) \widehat{\theta}_j^{(m)}=\arg\max_{\theta_j}\widehat{Q}_{SA}^{(m)}(\theta_j) θ j ( m ) = arg max θ j Q S A ( m ) ( θ j ) , for j = 1 , … , k j=1,\ldots,k j = 1 , … , k with the rest of θ \boldsymbol{\theta} θ fixed at ( θ ^ 1 ( m ) , … , θ ^ j − 1 ( m ) , θ ^ j + 1 ( m − 1 ) , … , θ ^ k ( m − 1 ) ) (\widehat{\theta}_1^{(m)},\ldots,\widehat{\theta}_{j-1}^{(m)},\widehat{\theta}_{j+1}^{(m-1)},\ldots,\widehat{\theta}_k^{(m-1)}) ( θ 1 ( m ) , … , θ j − 1 ( m ) , θ j + 1 ( m − 1 ) , … , θ k ( m − 1 ) ) . For γ m < 1 \gamma_m<1 γ m < 1 , the procedure uses information from previous steps to form the new estimates. Therefore, we update the sufficient statistic estimates using u ℓ ( m ) = ( 1 − γ m ) u ℓ ( m − 1 ) + γ m U ‾ m , ℓ \mathbf{u}_{\ell}^{(m)}=(1-\gamma_m)\mathbf{u}_{\ell}^{(m-1)}+\gamma_m\overline{\mathbf{U}}_{m,\ell} u ℓ ( m ) = ( 1 − γ m ) u ℓ ( m − 1 ) + γ m U m , ℓ , and
d ^ S A ( m ) = ( 1 − γ m ) d ^ S A ( m − 1 ) + γ m d ^ ( m ) . \widehat{d}_{SA}^{(m)}=(1-\gamma_m)\widehat{d}_{SA}^{(m-1)}+\gamma_m\widehat{d}^{(m)}. d S A ( m ) = ( 1 − γ m ) d S A ( m − 1 ) + γ m d ( m ) .
The remaining parameters are updated by conditional maximization using u ℓ ( m ) \mathbf{u}_{\ell}^{(m)} u ℓ ( m ) .
Prediction at Unobserved Locations
In order to predict Y ( s ) \mathbf{Y}(s) Y ( s ) at a new location s ∗ s_* s ∗ , we sample the conditional distribution of ( W ( τ ) ∼ W ∣ Y , θ ^ ) (\mathbf{W}^{(\tau)} \sim\mathbf{W} \mid\mathbf{Y}, \widehat{\boldsymbol{\theta}}) ( W ( τ ) ∼ W ∣ Y , θ ) , similar to the E step of the algorithm with the final estimated parameters θ ^ \widehat{\boldsymbol{\theta}} θ . We then use the prediction Y ^ ( s ∗ ) = L − 1 / 2 ∑ ℓ = 1 L W ^ ℓ cos ( ⟨ s ∗ , X ℓ ⟩ + ψ ℓ ) \widehat{\mathbf{Y}}(s_*) = L^{-1/2} \sum_{\ell=1}^{L} \widehat{\mathbf{W}}_\ell\cos(\langle s_*, X_\ell\rangle+ \psi_\ell) Y ( s ∗ ) = L − 1/2 ∑ ℓ = 1 L W ℓ cos (⟨ s ∗ , X ℓ ⟩ + ψ ℓ ) , where W ^ ℓ = ( 1 / T ) ∑ τ = 1 T W ℓ ( τ ) \widehat{\mathbf{W}}_\ell= (1/T) \sum_{\tau=1}^{T} \mathbf{W}^{(\tau)}_\ell W ℓ = ( 1/ T ) ∑ τ = 1 T W ℓ ( τ ) . The predictive variance matrix is formed as:
Var ~ ( Y ( s ∗ ) ∣ Y , θ ^ ) = 1 L ∑ ℓ = 1 L ∑ ℓ ′ = 1 L [ ( 1 T − 1 ∑ τ = 1 T ( W ℓ ( τ ) − W ^ ℓ ) ( W ℓ ′ ( τ ) − W ^ ℓ ′ ) T ) × cos ( ⟨ s ∗ , X ℓ ⟩ + ψ ℓ ) cos ( ⟨ s ∗ , X ℓ ′ ⟩ + ψ ℓ ′ ) ] + d ^ . \widetilde{\operatorname{Var}}(\mathbf{Y}(s_*) \mid\mathbf{Y}, \widehat{\boldsymbol{\theta}}) = \frac{1}{L} \sum_{\ell=1}^{L} \sum_{\ell'=1}^{L} \left[ \left( \frac{1}{T-1} \sum_{\tau=1}^{T} \left(\mathbf{W}^{(\tau)}_\ell- \widehat{\mathbf{W}}_\ell\right)\left(\mathbf{W}^{(\tau)}_{\ell'} - \widehat{\mathbf{W}}_{\ell'}\right)^{\mathsf{T}} \right) \times\cos(\langle s_*, X_\ell\rangle+ \psi_\ell)\cos(\langle s_*, X_{\ell'}\rangle+ \psi_{\ell'}) \right] + \widehat{\mathbf{d}}. Var ( Y ( s ∗ ) ∣ Y , θ ) = L 1 ℓ = 1 ∑ L ℓ ′ = 1 ∑ L [ ( T − 1 1 τ = 1 ∑ T ( W ℓ ( τ ) − W ℓ ) ( W ℓ ′ ( τ ) − W ℓ ′ ) T ) × cos (⟨ s ∗ , X ℓ ⟩ + ψ ℓ ) cos (⟨ s ∗ , X ℓ ′ ⟩ + ψ ℓ ′ ) ] + d .
Implementation Details
We next discuss some algorithmic details. To select ρ L \rho_L ρ L , information criteria or cross-validation can be used. In our experiments, the Akaike information criterion (AIC) [1 ] tends to select denser precision estimates, whereas the Bayesian information criterion (BIC) [46 ] yields sparser estimates. We select ρ L \rho_L ρ L among nonzero penalties by BIC and denote the selected fit Penalized hereafter. The BIC is defined as BIC ( ρ L ) = − 2 L ( θ ^ ρ L ∣ Y ) + k ln n \operatorname{BIC}(\rho_L) = -2\mathcal{L}(\widehat{\boldsymbol{\theta}}_{\rho_L} \mid\mathbf{Y}) + k \ln n BIC ( ρ L ) = − 2 L ( θ ρ L ∣ Y ) + k ln n , where k = ∑ 1 ≤ i < j ≤ p 1 ( η ^ i j ≠ 0 ) k = \sum_{1 \le i < j \le p} \mathbb{1}(\widehat{\eta}_{ij} \ne0) k = ∑ 1 ≤ i < j ≤ p 1 ( η ij = 0 ) counts the selected edges. The remaining spatial parameters shift BIC ( ρ L ) \operatorname{BIC}(\rho_L) BIC ( ρ L ) by a common constant and do not affect the selected ρ L \rho_L ρ L .
Under misalignment, since each process is observed at a different subset of locations, the random feature matrix no longer admits the simple Kronecker structure available under colocated observations. We handle the misalignment using an observation mask together with the n × L n \times L n × L Fourier feature matrix. The required matrix products are evaluated directly from the feature values at the observed locations, without explicitly constructing a full matrix.
Computational Complexity
Let n ˉ \bar{n} n ˉ be the average number of observations over processes. For each EM iteration, L L L relevant Cholesky blocks must be computed, so the complexity with precomputed random features, T T T Gibbs sweeps, and residuals updated after each Gibbs draw is O ( L p 3 + ( T + J ) L p 2 + T L n ˉ p ) O(Lp^3 + (T + J)Lp^2 + TL\bar{n}p) O ( L p 3 + ( T + J ) L p 2 + T L n ˉ p ) plus the cost of the graphical-LASSO update. Here, J J J bounds the number of CM3 objective evaluations per process. The p 3 p^3 p 3 and p 2 p^2 p 2 parts may be reduced appropriately when the sparsity of σ − 1 \sigma^{-1} σ − 1 can be exploited. Each iteration avoids the O ( n ˉ 3 p 3 ) O(\bar{n}^3p^3) O ( n ˉ 3 p 3 ) factorization of the full observation covariance and the potential O ( L 3 p 3 ) O(L^3p^3) O ( L 3 p 3 ) factorization arising from marginalizing out W \mathbf{W} W . Likelihood evaluation conditional on the features costs at most O ( p ˉ n ˉ L 2 + p ˉ n ˉ L + L 3 p 3 ) O(\bar{p}\bar{n}L^2 + \bar{p}\bar{n}L + L^3p^3) O ( p ˉ n ˉ L 2 + p ˉ n ˉ L + L 3 p 3 ) . For prediction, generating T T T posterior draws costs O { L p 3 + T L p 2 + T L n ˉ p } O\{Lp^3 + TLp^2 + TL\bar{n}p\} O { L p 3 + T L p 2 + T L n ˉ p } .
Theoretical Results
We start by proving in Proposition 1 that the unconditional distribution of Y \mathbf{Y} Y converges to a multivariate Gaussian distribution with the target covariance function as L → ∞ L \to\infty L → ∞ , providing asymptotic support for our random Fourier features approximation. Proposition 2 establishes convergence to the stationary set of the penalized random feature likelihood for fixed L L L . Proposition 3 establishes convergence to the stationary set of the exact penalized GP likelihood along suitable sequences L k → ∞ L_k \to\infty L k → ∞ and m k → ∞ m_k \to\infty m k → ∞ . Explicit assumptions and detailed proofs are deferred to Supplementary Section S.4.
Proposition 1 (The unconditional distribution of Y L \mathbf{Y}_L Y L ). Consider { Y L ( s ) } s ∈ R d \{Y_L(s)\}_{s\in\mathbb{R}^d} { Y L ( s ) } s ∈ R d simulated as in § 3.1 with a measurement error with variance d \mathbf{d} d . Let [ Σ ] i j = C ( s i , s j ) ∈ R p × p [\mathbf{\Sigma}]_{ij} = C(s_i,s_j) \in\mathbb{R}^{p\times p} [ Σ ] ij = C ( s i , s j ) ∈ R p × p and 1 ( ⋅ ) \mathbb{1}(\cdot) 1 ( ⋅ ) be the indicator function. Then,
(a) For each L ∈ { 1 , 2 , … } L \in\{1,2,\ldots\} L ∈ { 1 , 2 , … } , the unconditional moments after averaging over X ℓ , ψ ℓ , W ℓ X_\ell,\psi_\ell,W_\ell X ℓ , ψ ℓ , W ℓ are E [ Y L ( s ) ] = 0 p \mathbb{E}[Y_L(s)] = \mathbf{0}_p E [ Y L ( s )] = 0 p and Cov ( Y L ( s i ) , Y L ( s j ) ; θ ) = C ( s i , s j ) + d 1 { i = j } \operatorname{Cov}(Y_L(s_i),Y_L(s_j);\theta) = C(s_i,s_j) + \mathbf{d}\mathbb{1}\{i=j\} Cov ( Y L ( s i ) , Y L ( s j ) ; θ ) = C ( s i , s j ) + d 1 { i = j } .
(b) For fixed finite collections s 1 , … , s n s_1,\ldots,s_n s 1 , … , s n , the stacked vector Y L = ( Y L ( s 1 ) ⊤ , … , Y L ( s n ) ⊤ ) ⊤ \mathbf{Y}_L = (\mathbf{Y}_L(s_1)^\top,\ldots,\mathbf{Y}_L(s_n)^\top)^\top Y L = ( Y L ( s 1 ) ⊤ , … , Y L ( s n ) ⊤ ) ⊤ converges in distribution to N ( 0 , Σ + D ) \mathcal{N}(\mathbf{0},\mathbf{\Sigma}+\mathbf{D}) N ( 0 , Σ + D ) as L → ∞ L \to\infty L → ∞ .
Proposition 1 can be proved using the laws of total expectation and covariance and the multivariate central limit theorem. It demonstrates that our model correctly recovers the multivariate Gaussian distribution with the target covariance C ( ⋅ , ⋅ ; θ ) C(\cdot,\cdot;\boldsymbol{\theta}) C ( ⋅ , ⋅ ; θ ) asymptotically.
We next establish the convergence results for the algorithm. Let u ( m ) u^{(m)} u ( m ) be the stochastic-approximation iterate of the complete-data sufficient statistic for θ \boldsymbol{\theta} θ at the m m m -th iteration, and θ ^ ( u ( m ) ) \hat{\boldsymbol{\theta}}(u^{(m)}) θ ^ ( u ( m ) ) denote the fixed point of the CM cycle with u u u held fixed. To simplify the notation, let ℓ g ( θ ) : = L ( θ ∣ Y ) \ell_g(\boldsymbol{\theta}) := \mathcal{L}(\boldsymbol{\theta} \mid Y) ℓ g ( θ ) := L ( θ ∣ Y ) denote the marginal random feature log-likelihood under the proposal distribution g g g , and ℓ g , P ( θ ) : = ℓ g ( θ ) − P ( θ ) \ell_{g,P}(\boldsymbol{\theta}) := \ell_g(\boldsymbol{\theta}) - P(\boldsymbol{\theta}) ℓ g , P ( θ ) := ℓ g ( θ ) − P ( θ ) . We denote the stationary set of ℓ g , P ( θ ) \ell_{g,P}(\boldsymbol{\theta}) ℓ g , P ( θ ) as Θ g , P ⋆ = { θ ⋆ ∈ Θ : 0 ∈ ∂ ∘ ℓ g , P ( θ ⋆ ) } \Theta^\star_{g,P} = \{\boldsymbol{\theta}^\star\in\Theta: 0 \in\partial^\circ\ell_{g,P}(\boldsymbol{\theta}^\star)\} Θ g , P ⋆ = { θ ⋆ ∈ Θ : 0 ∈ ∂ ∘ ℓ g , P ( θ ⋆ )} , where ∂ ∘ \partial^\circ ∂ ∘ is the Clarke subdifferential [10 ] . For a point x x x and a nonempty set A A A in a finite-dimensional Euclidean space, write dist ( x , A ) = inf a ∈ A ∥ x − a ∥ \operatorname{dist}(x,A) = \inf_{a\in A}\|x-a\| dist ( x , A ) = inf a ∈ A ∥ x − a ∥ . Matrix components are vectorized when evaluating the norm.
Proposition 2 (Fixed-L L L convergence with a Gibbs E step). Fix n n n , L L L , and the realized random features. Under Assumptions A1 to A7, the SAECM algorithm satisfies, almost surely,
dist ( θ ^ ( u ( m ) ) , Θ g , P ⋆ ) ⟶ 0 , (7) \operatorname{dist}\left(\hat{\boldsymbol{\theta}}(u^{(m)}),\Theta^\star_{g,P}\right) \longrightarrow0, \tag*{(7)} dist ( θ ^ ( u ( m ) ) , Θ g , P ⋆ ) ⟶ 0 , (7) dist ( θ ( m ) , Θ g , P ⋆ ) ⟶ 0. (8) \operatorname{dist}\left(\boldsymbol{\theta}^{(m)},\Theta^\star_{g,P}\right) \longrightarrow0. \tag*{(8)} dist ( θ ( m ) , Θ g , P ⋆ ) ⟶ 0. (8)
Proposition 2 shows that, for fixed n n n and L L L , the parameter estimates from our SAECM algorithm with a Gibbs E step approach the stationary set of the random-feature-approximated penalized likelihood as the iterations increase, providing a convergence guarantee for the computational procedure. This is similar to the result of Kuhn and Lavielle [31 ] , who further note that, under additional conditions on the stochastic perturbations, convergence to local minima or saddle points can be avoided.
Next, we present a convergence result for the algorithm based on the random feature approximation to the target covariance function. Let ℓ \ell ℓ denote the exact GP log-likelihood and Θ P ⋆ = { θ ∈ Θ : ∇ ℓ ( θ ) ∈ ∂ P ( θ ) } \Theta^\star_P = \{\theta\in\Theta: \nabla\ell(\theta) \in\partial P(\theta)\} Θ P ⋆ = { θ ∈ Θ : ∇ ℓ ( θ ) ∈ ∂ P ( θ )} its penalized stationary set.
Proposition 3 (Convergence to the stationary set of the exact penalized GP likelihood).
Fix n n n , p p p . Suppose Assumptions A 8 A8 A 8 and A 9 A9 A 9 hold. Suppose further that, almost surely with respect to the random features, Assumptions A 1 A1 A 1 to A 7 A7 A 7 hold for all sufficiently large L L L . Then there exist deterministic sequences L k → ∞ L_k \to\infty L k → ∞ and m k → ∞ m_k \to\infty m k → ∞ such that, almost surely,
dist ( θ ^ L k ( u L k ( m k ) ) , Θ P ⋆ ) ⟶ 0 , (9) \operatorname{dist}\left(\hat{\theta}_{L_k}\left(u_{L_k}^{(m_k)}\right), \Theta^\star_P\right) \longrightarrow0, \tag*{(9)} dist ( θ ^ L k ( u L k ( m k ) ) , Θ P ⋆ ) ⟶ 0 , (9) dist ( θ L k ( m k ) , Θ P ⋆ ) ⟶ 0. (10) \operatorname{dist}\left(\theta_{L_k}^{(m_k)}, \Theta^\star_P\right) \longrightarrow0. \tag*{(10)} dist ( θ L k ( m k ) , Θ P ⋆ ) ⟶ 0. (10)
To extend Proposition 2 from fixed L L L to L → ∞ L \to\infty L → ∞ , additional regularity and containment assumptions are required (Assumptions A 8 A8 A 8 and A 9 A9 A 9 ). Then, uniform score convergence can be applied to establish convergence to the stationary set along suitably chosen sequences L k → ∞ L_k \to\infty L k → ∞ and m k → ∞ m_k \to\infty m k → ∞ , thereby proving Proposition 3. It shows that, along these sequences, not only does the covariance approximation converge, but the parameter estimates also approach the stationary set of the exact penalized log-likelihood with the target covariance.
Numerical Experiments
We consider a simulation design consisting of p = 50 p = 50 p = 50 processes, with a sparse data-generating σ − 1 \sigma^{-1} σ − 1 . The data are simulated using the algorithm proposed in Supplementary Section S.1 with L = 100,000 L = 100{,}000 L = 100 , 000 , which allows processes from different covariance families, with each marginal process represented by a potentially different covariance function. In particular, we consider three families: Matérn, squared-exponential (SE), and confluent hypergeometric (CH).
The Matérn model ([36 ] ) has covariance function and spectral density:
C M ( h ; ν , ϕ , σ ) = σ 2 1 − ν Γ ( ν ) ( ∥ h ∥ ϕ ) ν K ν ( ∥ h ∥ ϕ ) ; f M ( x ; ν , ϕ , σ ) = σ Γ ( ν + d 2 ) π d / 2 Γ ( ν ) ϕ 2 ν ( ϕ − 2 + ∥ x ∥ 2 ) − ν − d 2 , C_M(h;\nu,\phi,\sigma)=\sigma\frac{2^{1-\nu}}{\Gamma(\nu)}\left(\frac{\lVert h\rVert}{\phi}\right)^\nu K_\nu\left(\frac{\lVert h\rVert}{\phi}\right); \quad f_M(x;\nu,\phi,\sigma)=\sigma\frac{\Gamma\left(\nu+\frac{d}{2}\right)}{\pi^{d/2}\Gamma(\nu)\phi^{2\nu}}\left(\phi^{-2}+\lVert x\rVert^2\right)^{-\nu-\frac{d}{2}}, C M ( h ; ν , ϕ , σ ) = σ Γ ( ν ) 2 1 − ν ( ϕ ∥ h ∥ ) ν K ν ( ϕ ∥ h ∥ ) ; f M ( x ; ν , ϕ , σ ) = σ π d /2 Γ ( ν ) ϕ 2 ν Γ ( ν + 2 d ) ( ϕ − 2 + ∥ x ∥ 2 ) − ν − 2 d ,
where K ν ( ⋅ ) K_\nu(\cdot) K ν ( ⋅ ) is the modified Bessel function of the second kind, ν \nu ν is a smoothness parameter, ϕ \phi ϕ is a range parameter, and σ \sigma σ is a variance parameter. The SE covariance function can be derived from a Matérn model by letting ν → ∞ \nu\to\infty ν → ∞ with appropriate range rescaling. Its covariance function and spectral density are
C G ( h ; ϕ , σ ) = σ exp ( − ∥ h ∥ 2 2 ϕ 2 ) ; f G ( x ; ϕ , σ ) = σ ( ϕ 2 π ) d exp ( − ϕ 2 ∥ x ∥ 2 2 ) . C_G(h;\phi,\sigma)=\sigma\exp\left(-\frac{\lVert h\rVert^2}{2\phi^2}\right); \quad f_G(x;\phi,\sigma)=\sigma\left(\frac{\phi}{\sqrt{2\pi}}\right)^d\exp\left(-\frac{\phi^2\lVert x\rVert^2}{2}\right). C G ( h ; ϕ , σ ) = σ exp ( − 2 ϕ 2 ∥ h ∥ 2 ) ; f G ( x ; ϕ , σ ) = σ ( 2 π ϕ ) d exp ( − 2 ϕ 2 ∥ x ∥ 2 ) .
The CH model ([52 ] (Ma and Bhadra 2023)) additionally allows flexible control of covariance tail decay. Its covariance function and spectral density are
C C H ( h ; ν , α , β , σ ) = σ Γ ( ν + α ) Γ ( ν ) U ( α , 1 − ν , ∥ h ∥ 2 2 β 2 ) , C_{CH}(h;\nu,\alpha,\beta,\sigma)=\sigma\frac{\Gamma(\nu+\alpha)}{\Gamma(\nu)}\mathcal{U}\left(\alpha,1-\nu,\frac{\lVert h\rVert^2}{2\beta^2}\right), C C H ( h ; ν , α , β , σ ) = σ Γ ( ν ) Γ ( ν + α ) U ( α , 1 − ν , 2 β 2 ∥ h ∥ 2 ) ,
f C H ( x ; ν , α , β , σ ) = σ Γ ( ν + d 2 ) β d ( 2 π ) d / 2 B ( α , ν ) U ( ν + d 2 , 1 − α + d 2 , β 2 ∥ x ∥ 2 2 ) ; α > d 2 , f_{CH}(x;\nu,\alpha,\beta,\sigma)=\sigma\frac{\Gamma\left(\nu+\frac{d}{2}\right)\beta^d}{(2\pi)^{d/2}B(\alpha,\nu)}\mathcal{U}\left(\nu+\frac{d}{2},1-\alpha+\frac{d}{2},\frac{\beta^2\lVert x\rVert^2}{2}\right); \quad\alpha>\frac{d}{2}, f C H ( x ; ν , α , β , σ ) = σ ( 2 π ) d /2 B ( α , ν ) Γ ( ν + 2 d ) β d U ( ν + 2 d , 1 − α + 2 d , 2 β 2 ∥ x ∥ 2 ) ; α > 2 d ,
where B ( a , b ) B(a,b) B ( a , b ) is the beta function and U ( a , b , z ) \mathcal{U}(a,b,z) U ( a , b , z ) is the confluent hypergeometric function of the second kind. The CH covariance combines Matérn-like smoothness control with flexible polynomial tail behavior that can model either short or long range dependence. The parameter ν \nu ν controls smoothness, β \beta β plays a role similar to that of ϕ \phi ϕ in the Matérn covariance, and α \alpha α controls the covariance tail decay and the degree of long range dependence.
Throughout the simulations, we use a mixed-family setting in which 20 of the 50 processes are randomly assigned to the Matérn covariance family, 20 to the CH covariance family, and 10 to the SE covariance family. The parameters are randomly drawn from uniform distributions. For the Matérn and SE families, we use the inverse-range parameterization a = 1 / ϕ a=1/\phi a = 1/ ϕ . The Matérn models have a ∼ U [ 2.5 , 3.5 ] a\sim\mathrm{U}[2.5,3.5] a ∼ U [ 2.5 , 3.5 ] and ν ∼ U [ 1.5 , 2.5 ] \nu\sim\mathrm{U}[1.5,2.5] ν ∼ U [ 1.5 , 2.5 ] . The CH models have α ∼ U [ 1.25 , 1.75 ] \alpha\sim\mathrm{U}[1.25,1.75] α ∼ U [ 1.25 , 1.75 ] , β ∼ U [ 0.6 , 0.9 ] \beta\sim\mathrm{U}[0.6,0.9] β ∼ U [ 0.6 , 0.9 ] , and ν ∼ U [ 1.5 , 2.5 ] \nu\sim\mathrm{U}[1.5,2.5] ν ∼ U [ 1.5 , 2.5 ] , whereas the SE models have a ∼ U [ 1.75 , 2.25 ] a \sim\operatorname{U}[1.75, 2.25] a ∼ U [ 1.75 , 2.25 ] . There are n o = 500 n_o = 500 n o = 500 observed locations and n t = 125 n_t = 125 n t = 125 test locations, resulting in 25,000 values. We take d = 2 d = 2 d = 2 , with locations sampled uniformly over [ 0 , 5 ] × [ 0 , 5 ] [0, 5] \times[0, 5] [ 0 , 5 ] × [ 0 , 5 ] .
We assume a randomly generated precision structure among the processes. The diagonal entries are set to ( σ − 1 ) j j = 2 (\boldsymbol{\sigma}^{-1})_{jj} = 2 ( σ − 1 ) j j = 2 . For the off-diagonal entries, we consider two separate scenarios: a positive off-diagonal setting, in which 39 randomly selected upper-triangular entries are set to 0.8, and a negative off-diagonal setting, in which 48 randomly selected upper-triangular entries are set to − 0.8 -0.8 − 0.8 . All remaining off-diagonal entries are set to 0, and the corresponding lower-triangular entries are determined by symmetry (see Figure 1 ). These settings yield variance ranges of [ 0.5 , 4.34 ] [0.5, 4.34] [ 0.5 , 4.34 ] for the positive off-diagonal case and [ 0.5 , 3.41 ] [0.5, 3.41] [ 0.5 , 3.41 ] for the negative off-diagonal case. The nugget variance is set to 0.1 for all processes and is assumed to remain identical across processes during parameter estimation, while the process-specific covariance parameters are estimated separately. Results for a tridiagonal precision matrix are provided in Supplementary Section S.5.
Figure 1. True correlation and precision matrices and entrywise median precision estimates across 15 replicates for colocated simulations.
Previous literature suggests that, for several multivariate approaches, modeling processes independently offers predictive performance similar to cokriging when data are colocated [54 , 33 ] . Because observations of a target process at training locations already provide substantial information, colocated data from correlated processes may offer limited or no improvement to the prediction. Consequently, we also consider a misaligned setting where a randomly chosen 40% of the observed locations for each process are missing. In this setting, observations from correlated processes at locations where the target process is unobserved can provide additional predictive information, highlighting the benefit of multivariate modeling over independent process estimation. We evaluate these approaches based on parameter estimation, the predictive performance of individual and summed processes, and the associated uncertainty quantification.
We evaluate our method using both the graphical LASSO without a penalty (Dense) and the graphical LASSO with a penalty selected using the BIC (Penalized). We compare our method with kriging under the assumption that the processes are independent (Ind), the latent model in the inside-out cross-covariance approach (IOX, [40 ] ), and the linear model of coregionalization (LMC). For IOX, we consider both colocated and misaligned settings using Matérn marginals because the implementation does not support fitting CH marginals, although the IOX construction itself allows different marginal covariance families.
For LMC, full-rank deterministic maximum likelihood estimation is challenging when p p p is large, and the covariance family of each latent process must be determined because we model processes using multiple covariance families. Therefore, given a predetermined number of latent processes, q q q , we construct the loading matrix from a rank-q q q approximation to the sample covariance matrix and determine the covariance function of each latent process through AIC. Then the spatial parameters and the nugget are estimated through maximum likelihood estimation. We set the number of latent processes to q = 10 q = 10 q = 10 , which explains around 70% of the variance in the population covariance matrix, while increasing q q q to 15 adds only 6–7 percentage points at greater computational cost. We omit the basis graphical LASSO approach proposed by Krock et al. [30 ] from our prediction comparison because the paper does not develop a procedure for prediction at new locations. Their application uses an empirical orthogonal basis defined at the observed locations, and extension of this basis to new locations is neither discussed in the paper nor implemented in the supplied example code. Further implementation details of the benchmark methods are given in Supplementary Section S.5.2. All results use AMD EPYC 7662 “Rome” processors with 128 cores and 256 GB of memory per node. Computer code is available at: https://github.com/bychen713/SSGGP/ .
Estimation Results
We present the estimation results in Table 1 and Figure 1 . Our method produces lower bias than the Ind approach for most parameters. Figure 1 displays the elementwise median of the precision-matrix estimates from the proposed method across the 15 replicates. Both the Dense and Penalized approaches capture the nonzero patterns in the true precision matrix. The results for the misaligned observations are qualitatively similar, although they generally exhibit larger biases and standard deviations (Supplementary Table S3).
Table 1. Parameter estimation for colocated observations. Entries are bias (SD) across 15 replicates, averaged over the processes in each family. Bold indicates the smallest absolute bias.
Complete timing results for all settings are provided in Supplementary Table S1. SAECM requires a sufficiently large L L L to approximate the covariance functions well when p p p is large, and the E step requires repeated O ( p 3 ) O(p^3) O ( p 3 ) computations. Furthermore, CM3 requires sequential optimization of the spatial parameters, so both the Penalized and Dense fits take approximately 2–3.5 hours, with the Penalized fits slightly faster.
Prediction of Colocated Observations
We evaluate the ℓ 2 \ell_2 ℓ 2 norm of the residuals, the root mean squared error (RMSE) of the sum of processes, and the coverage of the predictive intervals for both individual processes and the sum of the processes across the 15 replicates. PI lengths refer to predictive interval lengths for individual processes and their sum. We demonstrate in the simulation that cokriging outperforms the Ind approach in the prediction of the sums. We present these results in Table 2 . All metrics are computed over all test locations and, when calculated per process, averaged across all processes. Letting n t n_t n t denote the number of test locations, the average ℓ 2 \ell_2 ℓ 2 norm of the residuals is calculated as:
Table 2. Prediction summaries for colocated simulations. Entries are mean (SD) across 15 replicates. Pos/Neg denote positive/negative precision off-diagonals. Bold indicates coverage closest to 95% and the lowest mean for other metrics.
Norm = 1 n t ∑ i = 1 n t ∑ j = 1 p ( Y j ( s i ) − Y ^ j ( s i ) ) 2 . \operatorname{Norm} = \frac{1}{n_t}\sum_{i=1}^{n_t}\sqrt{\sum_{j=1}^{p}\left(Y_j(s_i)-\widehat{Y}_j(s_i)\right)^2}. Norm = n t 1 i = 1 ∑ n t j = 1 ∑ p ( Y j ( s i ) − Y j ( s i ) ) 2 .
For the sum of processes, we calculate the RMSE as:
RMSE s u m = [ 1 n t ∑ i = 1 n t ( ∑ j = 1 p Y j ( s i ) − ∑ j = 1 p Y ^ j ( s i ) ) 2 ] 1 2 . \operatorname{RMSE}_{\mathrm{sum}} = \left[\frac{1}{n_t}\sum_{i=1}^{n_t}\left(\sum_{j=1}^{p}Y_j(s_i)-\sum_{j=1}^{p}\widehat{Y}_j(s_i)\right)^2\right]^{\frac{1}{2}}. RMSE sum = n t 1 i = 1 ∑ n t ( j = 1 ∑ p Y j ( s i ) − j = 1 ∑ p Y j ( s i ) ) 2 2 1 .
For individual-process prediction based on colocated observations, our approach outperforms all other methods under both precision matrix settings. The Penalized approach performs slightly better than the Dense approach on average, particularly in prediction norm, which could be attributed to the reduced number of false positive edges while retaining true edges. The Ind approach performs similarly as it works with the correctly specified marginal model. Our approach also demonstrates superior performance in the sum RMSE, as it estimates correlation between processes, unlike independent prediction.
The results for prediction uncertainty are also shown in Table 2 . The coverage rates for individual processes are similar across all methods and are all close to the nominal rate. We also report the average predictive interval lengths, with our approach generally providing shorter intervals and coverage close to the nominal level. The coverage rate for the sum under cokriging is closer to the nominal rate than that under the Ind approach, especially in the negative off-diagonal setting. The Penalized approach also slightly undercovers the sum in the negative off-diagonal setting, possibly because penalization attenuates estimated cross-process dependence. This difference between cokriging and Ind can be explained by the variance decomposition, Var ( ∑ j = 1 p Y j ( s ) ) = ∑ j = 1 p Var ( Y j ( s ) ) + 2 ∑ k < j Cov ( Y j ( s ) , Y k ( s ) ) \operatorname{Var}\left(\sum_{j=1}^{p}Y_j(s)\right)=\sum_{j=1}^{p}\operatorname{Var}(Y_j(s))+2\sum_{k<j}\operatorname{Cov}(Y_j(s),Y_k(s)) Var ( ∑ j = 1 p Y j ( s ) ) = ∑ j = 1 p Var ( Y j ( s )) + 2 ∑ k < j Cov ( Y j ( s ) , Y k ( s )) . Because the Ind approach ignores the second term, its predictive intervals are narrower in the negative off-diagonal setting and wider in the positive off-diagonal setting. The pattern becomes clearer in the results of the tridiagonal precision setting in Supplementary Section S.5.
Low-rank approximations such as the LMC cannot achieve comparable performance because the true processes have a full-rank dependence structure. Its wide predictive intervals provide good coverage results but are less meaningful given the significantly larger norm. IOX is implemented under a full-rank specification. However, the current implementation of IOX does not support the CH marginal covariance functions and is therefore misspecified in this setting, which may contribute to its slightly higher norm and RMSE. To separate the effect of this misspecification from the method’s performance, Supplementary Section S.5.7 presents a setting where every marginal process has a Matérn covariance function. In both settings, IOX does not estimate the sparsity of the precision matrix and shows slight overcoverage.
Prediction of Misaligned Observations
For the simulation of misaligned observations, removing 40% of the observations leads to 300 points per process, resulting in a total of 15,000 observed values across all processes. Parameter estimates are obtained as described in § 4.5 and are similar to those obtained for the colocated observations (Supplementary Table S3). The prediction results are reported in Table 3 . Overall, our approach achieves the best performance in most measures. The improvement over the Ind approach is even greater here because our method can utilize information from correlated processes at locations where the target process is unobserved. While LMC has the best coverage rate for the sum of the processes, it has wider predictive intervals and the worst point-prediction performance.
Table 3: Prediction summaries for misaligned simulations. Entries are mean (SD) across 15 replicates. Pos/Neg denote positive/negative precision off-diagonals. Bold indicates coverage closest to 95% and the lowest mean for other metrics.
Table 3.
Sparsity Recovery
Although our results have mainly focused on prediction and estimation performances so far, graph selection is a useful outcome of our methodology, and we evaluate the selection results using the precision-recall curves for the estimated precision matrix in Figure 2 . The Dense approach is a baseline that does not perform selection and thus has recall one. The Penalized approach retains a large proportion of the true edges and substantially reduces the false positive rate (FPR) relative to the Dense approach (Supplementary Table S2). However, the figure shows a trade-off between precision and recall across penalties. When the penalty is large, only a few off-diagonal entries with strong signals are selected, resulting in low recall and high precision. As the penalty decreases, more off-diagonal entries are selected, including false positives. The resulting loss of precision is exacerbated by the imbalance between true and absent edges, since only 39 of the 1,225 possible edges are nonzero in the positive off-diagonal sign setting, and only 48 are nonzero in the negative off-diagonal sign setting. At the BIC-selected penalties, precision remains low, suggesting that likelihood-based selection may retain spurious edges alongside true dependencies that improve model fit. The LMC and IOX approaches do not directly perform selection, so their results are excluded.
Figure 2. Precision–recall curves for precision-matrix edge recovery in colocated and misaligned simulations, averaged across 15 replicates.
Analysis of MERRA-2 Data
We consider the aerosol data set from the Modern-Era Retrospective Analysis for Research and Applications, Version 2 (MERRA-2) for our data analysis. MERRA-2 is a global atmospheric reanalysis developed by NASA’s Global Modeling and Assimilation Office that incorporates aerosol modeling and assimilates aerosol optical depth observations, with data available from 1980 onward [21 ] . It combines satellite observations of aerosol optical depth with simulated aerosol transport and physical processes, providing output on a regular grid with a resolution of 0.625 ∘ 0.625^\circ 0.62 5 ∘ longitude by 0.5 ∘ 0.5^\circ 0. 5 ∘ latitude.
We use the three-hourly instantaneous aerosol mixing-ratio product on native model levels, M2I3NVAER version 5.12.4 [23 ] .
Ten aerosol components, three dust sizes, two sea-salt sizes, hydrophobic and hydrophilic black carbon, hydrophobic and hydrophilic organic carbon, and sulfate, are selected from four vertical layers numbered 50, 52, 55, and 62, with regional-mean mid-layer pressures of approximately 506, 582, 688, and 843 hPa. We treat each component–layer combination as a distinct spatial process, resulting in p = 40 p = 40 p = 40 processes. We focus on the region bounded by longitudes 95 ∘ W 95^\circ\mathrm{W} 9 5 ∘ W and 65 ∘ W 65^\circ\mathrm{W} 6 5 ∘ W and latitudes 25 ∘ N 25^\circ\mathrm{N} 2 5 ∘ N and 50 ∘ N 50^\circ\mathrm{N} 5 0 ∘ N at 00 UTC on 15 July 2022, yielding 2,499 grid cells with complete data. For each process and fold, we apply a logarithmic transformation using an offset equal to half its smallest positive observed value and subtract a spatially varying mean estimated from training observations in the fold by local linear regression with a bandwidth of 1,000 kilometers. We then divide the residuals by their process-specific standard deviations computed from the same observations. The first, twentieth, and fortieth processes, corresponding to DU001 at layer 50, SS005 at layer 62, and SO4 at layer 62, are plotted in Figure 3 , with the resulting processed values shown below each map.
Figure 3. MERRA-2 data for colocated fold 1. Left to right: dust DU001 at 506 hPa, sea salt SS005 at 843 hPa, and sulfate SO4 at 843 hPa. Top: mixing ratios at all 2,499 original grid cells. Bottom: preprocessed values at the 700 sampled locations.
For prediction assessment, we construct three folds, each containing n = 500 n = 500 n = 500 training locations and 200 non-overlapping held-out test locations. In the colocated design, all 40 processes are observed at each training location. In the misaligned design, we mask 40% of the training responses separately for each process, leaving 300 observed values per process. The nugget is estimated separately for each process, and other implementation details are described in Supplementary Section S.6.
Figure 4 shows the entrywise median of precision matrices from the three data folds. There is a clear pattern of negative off-diagonal entries within the dust variables and among the black carbon and the organic carbon, especially at corresponding vertical levels. DU001 and DU003 represent the finest and intermediate-sized dust bins, respectively, and their negative precision entries suggest positive partial associations, possibly reflecting shared emission and transport. DU005, the coarsest dust bin, shows weaker associations with the other dust bins, possibly because larger particles settle faster [35 ] . Several black and organic carbon variables also show positive partial associations, consistent with shared combustion sources [7 , 44 ] .
Figure 4. MERRA-2 sample precision matrices and SAECM precision estimates from colocated observations, shown as entrywise medians across three folds.
The estimation results are shown in Supplementary Section S.6 and Figure 4 . In Figure 4 , the Penalized estimate is on a different scale from the Dense estimate, but both capture the main precision structure among the dust and carbon variables. Table 4 shows the prediction results for all approaches. When the data are colocated, all approaches have similar individual-process coverage and sum RMSE, although LMC has a larger prediction norm. The Ind approach has substantially lower coverage for the sum of the processes. This is expected because several variables are positively correlated, and ignoring their dependence results in underestimation of the predictive variance of the sum. When the data are misaligned, IOX, Penalized, and Dense have similar prediction errors and outperform the Ind approach in individual-process prediction, with IOX providing sum coverage closest to 95%. LMC has the largest prediction norm but a lower sum RMSE than Ind. Prediction norms increase under misalignment for all methods, whereas sum RMSE is essentially unchanged for Penalized and Dense.
Table 4. MERRA-2 prediction results for colocated and misaligned observations. Entries are mean (SD) across three folds. Bold indicates coverage closest to 95% and the lowest mean for other metrics.
Discussion
We introduce a new penalized likelihood estimation methodology for highly multivariate Gaussian processes that defines the model in the spectral domain, where process-level conditional independence is most naturally interpretable, while inference can proceed in the spatial domain using random Fourier features as the key connecting bridge. Unlike previous works using spatial domain constructions (e.g., Dey et al., [17 ] ; Peruzzi, [40 ] ), our approach does not require a reference set. Also unlike previous works using spectral constructions (e.g., Terres et al., [48 ] ), our approach obviates the need for Whittle likelihood-based estimation, and targets maximizing the penalized observed data likelihood.
By separating the covariance parameters and the spatial parameters in the spectral domain, our covariance function is valid by construction and easily allows different marginal covariance families with spectral densities in finite-dimensional parametric families. Furthermore, the combination of the SAECM algorithm with the graphical LASSO method provides sparsity in the precision estimate, establishing process-level conditional independence estimation. Our construction allows process-specific Matérn marginal scales while retaining this graph interpretation, at the expense of cross-covariances that are generally not Matérn, thereby partially addressing the modeling challenge noted by Peruzzi [41 ] .
Our simulation and data analyses demonstrate comparable or better individual-process predictive performance relative to benchmarks under colocated observations, with more pronounced improvements under misalignment, particularly under heterogeneous marginal covariance families. The consideration of process dependence provides additional information when the data are not fully observed, and further improves prediction and inference regarding the functionals of the process. This approach can be readily applied to environmental studies including atmospheric science (Gelaro et al., [21 ] ) or oceanography (Argo, [4 ] ).
Although we work with stationary covariance functions for demonstrations, future work can consider extensions to nonstationary (Emery and Arroyo, [18 ] ; Ton et al., [49 ] ) or spatio-temporal covariance functions (Allard et al., [2 ] ). Extension to stationary multivariate spatio-temporal processes is conceptually straightforward, provided suitable joint spatiotemporal marginal spectral densities are available. Accommodating frequency-dependent partial coherence [30 ] is another promising direction.
Supplementary Material
The Supplementary Material contains mathematical proofs, additional technical details and numerical results.
Data Availability Statement
The MERRA-2 data analyzed in this study are available from NASA’s Goddard Earth Sciences Data and Information Services Center under product M2I3NVAER, version 5.12.4 (doi:10.5067/LTVB4GPCOTK2). Code for the simulations and data analysis is available at https://github.com/bychen713/SSGGP .
Supplementary Material to
Penalized Likelihood Inference in Highly Multivariate Sparse Spectral Graphical Gaussian Processes
Example Simulation Details
We summarize the described approach in Algorithm S1 for f ( x ) \mathbf{f}(x) f ( x ) as in (5) , which constructs
W ℓ ∼ N ( 0 p , 2 f ( X ℓ ) / g ( X ℓ ) ) W_{\ell} \sim\mathcal{N}(\mathbf{0}_{p},2f(X_{\ell})/g(X_{\ell})) W ℓ ∼ N ( 0 p , 2 f ( X ℓ ) / g ( X ℓ )) . We provide an example in Figure S1 .
Figure S1. Simulated fields for the first 21 processes in replicate 1 of the random-precision setting with positive off-diagonal entries.
Data: L L L , s ∈ R n × d \mathbf{s} \in\mathbb{R}^{n \times d} s ∈ R n × d , σ − 1 ∈ R p × p \boldsymbol{\sigma}^{-1} \in\mathbb{R}^{p \times p} σ − 1 ∈ R p × p , { f j ( x ) } j = 1 p \{f_j(x)\}_{j=1}^{p} { f j ( x ) } j = 1 p , g ( x ) g(x) g ( x ) .
Result: Simulated multivariate process Y ( s ) ∈ R n × p \mathbf{Y}(\mathbf{s}) \in\mathbb{R}^{n \times p} Y ( s ) ∈ R n × p at locations s \mathbf{s} s .
Initialize Y ( s ) \mathbf{Y}(\mathbf{s}) Y ( s ) ’s entries to 0 0 0 , and compute the Cholesky factor of σ − 1 = Λ Λ T \boldsymbol{\sigma}^{-1} = \boldsymbol{\Lambda}\boldsymbol{\Lambda}^{\mathsf{T}} σ − 1 = Λ Λ T .
for ℓ = 1 , 2 , … , L \ell= 1,2,\ldots,L ℓ = 1 , 2 , … , L
Simulate X ℓ ∼ g ( x ) X_{\ell} \sim g(x) X ℓ ∼ g ( x ) , ψ ℓ ∼ U [ 0 , 2 π ] \psi_{\ell} \sim\mathrm{U}[0,2\pi] ψ ℓ ∼ U [ 0 , 2 π ] , and Z ℓ ∼ N ( 0 p , I p ) \mathbf{Z}_{\ell} \sim\mathcal{N}(\mathbf{0}_{p},\mathbf{I}_{p}) Z ℓ ∼ N ( 0 p , I p ) .
Compute M ℓ = ( g ( X ℓ ) / 2 ) 1 / 2 Λ T h ( X ℓ ) − 1 = Λ T diag ( g ( X ℓ ) 2 f j ( X ℓ ) , j = 1 , … , p ) \mathbf{M}_{\ell} = (g(X_{\ell})/2)^{1/2}\boldsymbol{\Lambda}^{\mathsf{T}}\mathbf{h}(X_{\ell})^{-1} = \boldsymbol{\Lambda}^{\mathsf{T}}\operatorname{diag}\left(\sqrt{\frac{g(X_{\ell})}{2f_j(X_{\ell})}},j=1,\ldots,p\right) M ℓ = ( g ( X ℓ ) /2 ) 1/2 Λ T h ( X ℓ ) − 1 = Λ T diag ( 2 f j ( X ℓ ) g ( X ℓ ) , j = 1 , … , p ) .
Compute W ℓ \mathbf{W}_{\ell} W ℓ by solving M ℓ W ℓ = Z ℓ \mathbf{M}_{\ell}\mathbf{W}_{\ell} = \mathbf{Z}_{\ell} M ℓ W ℓ = Z ℓ .
for i = 1 , … , n i = 1,\ldots,n i = 1 , … , n
Set Y ( s i ) = Y ( s i ) + c i , ℓ W ℓ \mathbf{Y}(\mathbf{s}_i) = \mathbf{Y}(\mathbf{s}_i) + c_{i,\ell}\mathbf{W}_{\ell} Y ( s i ) = Y ( s i ) + c i , ℓ W ℓ , where c i , ℓ = 1 L cos ( ⟨ X ℓ , s i ⟩ + ψ ℓ ) c_{i,\ell} = \frac{1}{\sqrt{L}}\cos(\langle X_{\ell},\mathbf{s}_i\rangle+\psi_{\ell}) c i , ℓ = L 1 cos (⟨ X ℓ , s i ⟩ + ψ ℓ ) .
Algorithm S1: Simulation of Y ( s ) \mathbf{Y}(\mathbf{s}) Y ( s ) for f ( x ) ∈ R p × p \mathbf{f}(x) \in\mathbb{R}^{p \times p} f ( x ) ∈ R p × p as in (5) .
The computational bottlenecks are the Cholesky decomposition of σ − 1 \boldsymbol{\sigma}^{-1} σ − 1 which is O ( p 3 ) O(p^3) O ( p 3 ) for dense σ − 1 \boldsymbol{\sigma}^{-1} σ − 1 , the creation of { M ℓ } ℓ = 1 L \{\mathbf{M}_{\ell}\}_{\ell=1}^{L} { M ℓ } ℓ = 1 L and { W ℓ } ℓ = 1 L \{\mathbf{W}_{\ell}\}_{\ell=1}^{L} { W ℓ } ℓ = 1 L by backsolving which is O ( L p 2 ) O(Lp^2) O ( L p 2 ) , and the inner loop of constructing Y ( s ) \mathbf{Y}(\mathbf{s}) Y ( s ) has overall complexity O ( L p n + L d n ) O(Lpn+Ldn) O ( L p n + L d n ) . Furthermore, the cubic and quadratic terms of p p p can be effectively reduced with sparsity, greatly reducing the computational complexity for large p p p .
Marginal Likelihood Calculation
In the following, we suppress the notation on θ \theta θ , using the matrices Σ L \boldsymbol{\Sigma}_{L} Σ L , D D D , and:
Σ W = block-diagonal ( Σ W ℓ ℓ ′ , ℓ , ℓ ′ = 1 , … , L ) ∈ R L p × L p , A = [ c i , ℓ ] i = 1 , ℓ = 1 n , L ⊗ I p ∈ R n p × L p \boldsymbol{\Sigma}_{W} = \operatorname{block\text{-}diagonal}\left(\boldsymbol{\Sigma}_{W_{\ell\ell'}},\ \ell,\ell'=1,\ldots,L\right) \in\mathbb{R}^{Lp \times Lp}, \qquad A = \left[c_{i,\ell}\right]_{i=1,\ell=1}^{n,L} \otimes I_p \in\mathbb{R}^{np \times Lp} Σ W = block - diagonal ( Σ W ℓ ℓ ′ , ℓ , ℓ ′ = 1 , … , L ) ∈ R L p × L p , A = [ c i , ℓ ] i = 1 , ℓ = 1 n , L ⊗ I p ∈ R n p × L p
where c i , ℓ = L − 1 / 2 cos ( ⟨ s i , X ℓ ⟩ + ψ ℓ ) c_{i,\ell} = L^{-1/2}\cos\left(\langle s_i, X_{\ell}\rangle+ \psi_{\ell}\right) c i , ℓ = L − 1/2 cos ( ⟨ s i , X ℓ ⟩ + ψ ℓ ) .
Likelihood calculation without simulated W W W
By integrating out W W W , the random feature covariance can be expressed: Σ L = A Σ W A T \boldsymbol{\Sigma}_{L} = A\boldsymbol{\Sigma}_{W}A^{\mathsf{T}} Σ L = A Σ W A T .
Then we can apply the Woodbury identity for the likelihood calculation:
[ A Σ W A T + D ] − 1 = D − 1 − D − 1 A ( Σ W − 1 + A T D − 1 A ) − 1 A T D − 1 . \left[A\boldsymbol{\Sigma}_{W}A^{\mathsf{T}} + D\right]^{-1} = D^{-1} - D^{-1}A\left(\boldsymbol{\Sigma}_{W}^{-1} + A^{\mathsf{T}}D^{-1}A\right)^{-1}A^{\mathsf{T}}D^{-1}. [ A Σ W A T + D ] − 1 = D − 1 − D − 1 A ( Σ W − 1 + A T D − 1 A ) − 1 A T D − 1 .
det [ A Σ W A T + D ] = det [ Σ W − 1 + A T D − 1 A ] det [ Σ W ] det [ D ] . \det\left[A\boldsymbol{\Sigma}_{W}A^{\mathsf{T}} + D\right] = \det\left[\boldsymbol{\Sigma}_{W}^{-1} + A^{\mathsf{T}}D^{-1}A\right]\det\left[\boldsymbol{\Sigma}_{W}\right]\det\left[D\right]. det [ A Σ W A T + D ] = det [ Σ W − 1 + A T D − 1 A ] det [ Σ W ] det [ D ] .
The calculation is dominated by the Cholesky factorization of the L p × L p Lp \times Lp L p × L p matrix Σ W − 1 + A T D − 1 A \boldsymbol{\Sigma}_{W}^{-1} + A^{\mathsf{T}}D^{-1}A Σ W − 1 + A T D − 1 A , which requires O ( L 3 p 3 ) O(L^{3}p^{3}) O ( L 3 p 3 ) .
Likelihood calculation with simulated W W W
Directly following our approach for simulation above for f ( x ) ∈ R n p × p f(x) \in\mathbb{R}^{np \times p} f ( x ) ∈ R n p × p , we take Φ = [ c i , ℓ W ℓ ] i = 1 , ℓ = 1 n p , L ∈ R n p × L \boldsymbol{\Phi} = [c_{i,\ell}\boldsymbol{W}_{\ell}]_{i=1,\ell=1}^{np,L} \in\mathbb{R}^{np \times L} Φ = [ c i , ℓ W ℓ ] i = 1 , ℓ = 1 n p , L ∈ R n p × L . The Monte Carlo estimate of the marginal log-likelihood function based on Φ ( θ ) \boldsymbol{\Phi}(\theta) Φ ( θ ) is then:
L ~ ( θ ∣ Y ) = − 1 2 { n p log ( 2 π ) + log ( det [ Σ ~ L ( θ ) + D ] ) + Y T ( Σ ~ L ( θ ) + D ) − 1 Y } . \widetilde{\mathcal{L}}(\theta\mid\boldsymbol{Y}) = -\frac{1}{2}\left\{np\log(2\pi) + \log\left(\det\left[\widetilde{\boldsymbol{\Sigma}}_{L}(\theta) + \boldsymbol{D}\right]\right) + \boldsymbol{Y}^{\mathsf{T}}\left(\widetilde{\boldsymbol{\Sigma}}_{L}(\theta) + \boldsymbol{D}\right)^{-1}\boldsymbol{Y}\right\}. L ( θ ∣ Y ) = − 2 1 { n p log ( 2 π ) + log ( det [ Σ L ( θ ) + D ] ) + Y T ( Σ L ( θ ) + D ) − 1 Y } .
where Σ ~ L ( θ ) = Φ ( θ ) Φ ( θ ) T \widetilde{\boldsymbol{\Sigma}}_{L}(\theta) = \boldsymbol{\Phi}(\theta)\boldsymbol{\Phi}(\theta)^{\mathsf{T}} Σ L ( θ ) = Φ ( θ ) Φ ( θ ) T is a Monte Carlo estimate of the covariance Σ L ( θ ) \boldsymbol{\Sigma}_{L}(\theta) Σ L ( θ ) , obtained by simulating the latent variables W = { W ℓ } \boldsymbol{W} = \{\boldsymbol{W}_{\ell}\} W = { W ℓ } rather than integrating them out analytically, and E W [ Σ ~ L ( θ ; W ) ] = Σ L ( θ ) \mathbb{E}_{W}[\widetilde{\boldsymbol{\Sigma}}_{L}(\theta;W)] = \boldsymbol{\Sigma}_{L}(\theta) E W [ Σ L ( θ ; W )] = Σ L ( θ ) . To compute the likelihood, we need to find the determinant of Σ ~ L + D \widetilde{\boldsymbol{\Sigma}}_{L} + \boldsymbol{D} Σ L + D and solve against Σ ~ L + D \widetilde{\boldsymbol{\Sigma}}_{L} + \boldsymbol{D} Σ L + D . The Woodbury matrix identity and the matrix determinant lemma, respectively, yield:
[ Σ ~ L + D ] − 1 = [ Φ Φ T + D ] − 1 = D − 1 − D − 1 Φ ( I L + Φ T D − 1 Φ ) − 1 Φ T D − 1 , det [ Σ ~ L + D ] = det [ Φ Φ T + D ] = det [ I L + Φ T D − 1 Φ ] det [ D ] . (11) \begin{aligned}
\left[\widetilde{\boldsymbol{\Sigma}}_{L}+\boldsymbol{D}\right]^{-1} &= \left[\boldsymbol{\Phi}\boldsymbol{\Phi}^{\mathsf{T}}+\boldsymbol{D}\right]^{-1} = \boldsymbol{D}^{-1}-\boldsymbol{D}^{-1}\boldsymbol{\Phi}\left(\boldsymbol{I}_{L}+\boldsymbol{\Phi}^{\mathsf{T}}\boldsymbol{D}^{-1}\boldsymbol{\Phi}\right)^{-1}\boldsymbol{\Phi}^{\mathsf{T}}\boldsymbol{D}^{-1},\\
\det\left[\widetilde{\boldsymbol{\Sigma}}_{L}+\boldsymbol{D}\right] &= \det\left[\boldsymbol{\Phi}\boldsymbol{\Phi}^{\mathsf{T}}+\boldsymbol{D}\right] = \det\left[\boldsymbol{I}_{L}+\boldsymbol{\Phi}^{\mathsf{T}}\boldsymbol{D}^{-1}\boldsymbol{\Phi}\right]\det[\boldsymbol{D}]. \tag*{(11)}
\end{aligned} [ Σ L + D ] − 1 det [ Σ L + D ] = [ Φ Φ T + D ] − 1 = D − 1 − D − 1 Φ ( I L + Φ T D − 1 Φ ) − 1 Φ T D − 1 , = det [ Φ Φ T + D ] = det [ I L + Φ T D − 1 Φ ] det [ D ] . (11)
This reduces the necessary Cholesky decomposition from size n p np n p to L L L . We summarize the overall algorithm for likelihood evaluation in Algorithm S2. The computational complexity for simulation is O ( p 3 + L p 2 + L p n + L n d ) O(p^{3} + Lp^{2} + Lpn + Lnd) O ( p 3 + L p 2 + L p n + L n d ) , but the likelihood evaluation has additional terms. In particular, forming the matrix I L + Φ T D − 1 Φ \boldsymbol{I}_{L} + \boldsymbol{\Phi}^{\mathsf{T}}\boldsymbol{D}^{-1}\boldsymbol{\Phi} I L + Φ T D − 1 Φ has complexity O ( L 2 p + L 2 n ) O(L^{2}p + L^{2}n) O ( L 2 p + L 2 n ) , and computing its Cholesky factor Ψ \boldsymbol{\Psi} Ψ has complexity O ( L 3 ) O(L^{3}) O ( L 3 ) . The additional complexity is thus directly controllable by the Fourier approximation size L L L , and remains less (in p p p ) than the O ( L p 3 ) O(Lp^{3}) O ( L p 3 ) in Krock et al. [30 ] for estimation.
This approach also lessens the memory burden of the likelihood evaluation, which would be O ( p 2 n 2 ) O(p^{2}n^{2}) O ( p 2 n 2 ) for the n p × n p np \times np n p × n p matrix Σ ~ L + D \widetilde{\boldsymbol{\Sigma}}_{L} + \boldsymbol{D} Σ L + D . Our proposed methodology for computation of the likelihood circumvents direct formation of Σ ~ L + D \widetilde{\boldsymbol{\Sigma}}_{L} + \boldsymbol{D} Σ L + D and has a memory burden of size O ( p 2 + L 2 + L p + L n + n p ) O(p^{2} + L^{2} + Lp + Ln + np) O ( p 2 + L 2 + L p + L n + n p ) . We outline the sources for these terms: the p 2 p^{2} p 2 for σ − 1 \boldsymbol{\sigma}^{-1} σ − 1 and Λ \boldsymbol{\Lambda} Λ , L 2 L^{2} L 2 for I L + Φ T D − 1 Φ \boldsymbol{I}_{L} + \boldsymbol{\Phi}^{\mathsf{T}}\boldsymbol{D}^{-1}\boldsymbol{\Phi} I L + Φ T D − 1 Φ and Ψ \boldsymbol{\Psi} Ψ , L p Lp L p for { W ℓ } ℓ = 1 L \{\boldsymbol{W}_{\ell}\}_{\ell=1}^{L} { W ℓ } ℓ = 1 L , L n Ln L n for { c i , ℓ } \{c_{i,\ell}\} { c i , ℓ } , and n p np n p for Y \boldsymbol{Y} Y . The largest terms Data: Y ( s ) ∈ R n × p Y(s) \in\mathbb{R}^{n \times p} Y ( s ) ∈ R n × p , L L L , s ∈ R n × d s \in\mathbb{R}^{n \times d} s ∈ R n × d , σ − 1 ∈ R p × p \boldsymbol{\sigma}^{-1} \in\mathbb{R}^{p \times p} σ − 1 ∈ R p × p , { f j ( x ) } j = 1 p \{f_j(x)\}_{j=1}^{p} { f j ( x ) } j = 1 p , d = diag ( { d j } j = 1 p ) d = \operatorname{diag}(\{d_j\}_{j=1}^{p}) d = diag ({ d j } j = 1 p ) , g ( x ) g(x) g ( x ) .
Result: Likelihood evaluation for parameters σ − 1 \boldsymbol{\sigma}^{-1} σ − 1 , { d j } j = 1 p \{d_j\}_{j=1}^{p} { d j } j = 1 p , and those in { f j ( x ) } j = 1 p \{f_j(x)\}_{j=1}^{p} { f j ( x ) } j = 1 p .
Compute the Cholesky factor of σ − 1 = Λ Λ T \boldsymbol{\sigma}^{-1} = \boldsymbol{\Lambda}\boldsymbol{\Lambda}^{\mathsf{T}} σ − 1 = Λ Λ T .
Compute W ℓ \boldsymbol{W}_{\ell} W ℓ and c i , ℓ c_{i,\ell} c i , ℓ as in Algorithm S1 for all ℓ = 1 , 2 , … , L \ell= 1, 2, \ldots, L ℓ = 1 , 2 , … , L .
for ℓ 1 = 1 , 2 , … , L \ell_1 = 1, 2, \ldots, L ℓ 1 = 1 , 2 , … , L
for ℓ 2 = 1 , 2 , … , L \ell_2 = 1, 2, \ldots, L ℓ 2 = 1 , 2 , … , L
Compute ( I L + Φ T D − 1 Φ ) ℓ 1 , ℓ 2 = 1 ( ℓ 1 = ℓ 2 ) + W ℓ 1 T d − 1 W ℓ 2 ∑ i = 1 n c i , ℓ 1 c i , ℓ 2 (I_L + \boldsymbol{\Phi}^{\mathsf{T}}\boldsymbol{D}^{-1}\boldsymbol{\Phi})_{\ell_1,\ell_2} = \mathbb{1}(\ell_1 = \ell_2) + \boldsymbol{W}_{\ell_1}^{\mathsf{T}}d^{-1}\boldsymbol{W}_{\ell_2}\sum_{i=1}^{n} c_{i,\ell_1}c_{i,\ell_2} ( I L + Φ T D − 1 Φ ) ℓ 1 , ℓ 2 = 1 ( ℓ 1 = ℓ 2 ) + W ℓ 1 T d − 1 W ℓ 2 ∑ i = 1 n c i , ℓ 1 c i , ℓ 2 .
Compute the Cholesky factor I L + Φ T D − 1 Φ = Ψ Ψ T I_L + \boldsymbol{\Phi}^{\mathsf{T}}\boldsymbol{D}^{-1}\boldsymbol{\Phi} = \boldsymbol{\Psi}\boldsymbol{\Psi}^{\mathsf{T}} I L + Φ T D − 1 Φ = Ψ Ψ T .
Compute the likelihood using Ψ \boldsymbol{\Psi} Ψ and (S1).
Algorithm S2: Likelihood evaluation when f ( x ) f(x) f ( x ) has form (5).
are again directly controllable by L L L or by sparsity in σ − 1 \boldsymbol{\sigma}^{-1} σ − 1 .
Conditional Distribution for Gibbs Sampling
First, we introduce some notation. With ⊗ \otimes ⊗ being the standard Kronecker product, define
A ℓ = [ c i , ℓ ] i = 1 n ⊗ I p ∈ R n p × p , W − ℓ = [ W ℓ ′ ] ℓ ′ ≠ ℓ L ∈ R ( L − 1 ) p A_{\ell} = [c_{i,\ell}]_{i=1}^{n} \otimes I_p \in\mathbb{R}^{np \times p}, \qquad W_{-\ell} = [W_{\ell'}]_{\ell' \ne\ell}^{L} \in\mathbb{R}^{(L-1)p} A ℓ = [ c i , ℓ ] i = 1 n ⊗ I p ∈ R n p × p , W − ℓ = [ W ℓ ′ ] ℓ ′ = ℓ L ∈ R ( L − 1 ) p
A − ℓ = [ c i , ℓ ′ ] i = 1 , ℓ ′ ≠ ℓ n , L ⊗ I p ∈ R n p × ( L − 1 ) p , D = I n ⊗ d . A_{-\ell} = [c_{i,\ell'}]_{i=1,\ell' \ne\ell}^{n,L} \otimes I_p \in\mathbb{R}^{np \times(L-1)p}, \qquad D = I_n \otimes d. A − ℓ = [ c i , ℓ ′ ] i = 1 , ℓ ′ = ℓ n , L ⊗ I p ∈ R n p × ( L − 1 ) p , D = I n ⊗ d .
With this notation, we have the following result.
Proposition 4. The distribution of W ℓ ∣ W − ℓ , Y , θ W_{\ell} \mid W_{-\ell}, Y, \theta W ℓ ∣ W − ℓ , Y , θ is multivariate Gaussian with:
E [ W ℓ ∣ W − ℓ , Y , θ ] = Σ W ℓ [ I p − A ℓ T D − 1 A ℓ ( Σ W ℓ − 1 + A ℓ T D − 1 A ℓ ) − 1 ] [ A ℓ T D − 1 ( Y − A − ℓ W − ℓ ) ] , \mathbb{E}[W_{\ell} \mid W_{-\ell}, Y, \theta] = \Sigma_{W_{\ell}}\left[I_p - A_{\ell}^{\mathsf{T}}D^{-1}A_{\ell}(\Sigma_{W_{\ell}}^{-1} + A_{\ell}^{\mathsf{T}}D^{-1}A_{\ell})^{-1}\right]\left[A_{\ell}^{\mathsf{T}}D^{-1}(Y - A_{-\ell}W_{-\ell})\right], E [ W ℓ ∣ W − ℓ , Y , θ ] = Σ W ℓ [ I p − A ℓ T D − 1 A ℓ ( Σ W ℓ − 1 + A ℓ T D − 1 A ℓ ) − 1 ] [ A ℓ T D − 1 ( Y − A − ℓ W − ℓ ) ] , Var [ W ℓ ∣ W − ℓ , Y , θ ] = ( Σ W ℓ − 1 + A ℓ T D − 1 A ℓ ) − 1 . \operatorname{Var}[W_{\ell} \mid W_{-\ell}, Y, \theta] = (\Sigma_{W_{\ell}}^{-1} + A_{\ell}^{\mathsf{T}}D^{-1}A_{\ell})^{-1}. Var [ W ℓ ∣ W − ℓ , Y , θ ] = ( Σ W ℓ − 1 + A ℓ T D − 1 A ℓ ) − 1 .
Therefore, we can simulate from this distribution using a matrix decomposition of dimension p p p . We study the computation in a bit more detail. Notice that
A ℓ T D − 1 A ℓ = ( [ c i , ℓ ] i = 1 n ⊗ I p ) T ( I n ⊗ d − 1 ) ( [ c i , ℓ ] i = 1 n ⊗ I p ) = [ ∑ i = 1 n c i , ℓ 2 ] d − 1 , \mathbf{A}_{\ell}^{\mathsf{T}}\mathbf{D}^{-1}\mathbf{A}_{\ell} = ([c_{i,\ell}]_{i=1}^{n} \otimes I_{p})^{\mathsf{T}}(I_{n} \otimes d^{-1})([c_{i,\ell}]_{i=1}^{n} \otimes I_{p}) = \left[\sum_{i=1}^{n} c_{i,\ell}^{2}\right]d^{-1}, A ℓ T D − 1 A ℓ = ([ c i , ℓ ] i = 1 n ⊗ I p ) T ( I n ⊗ d − 1 ) ([ c i , ℓ ] i = 1 n ⊗ I p ) = [ i = 1 ∑ n c i , ℓ 2 ] d − 1 ,
a p × p p \times p p × p diagonal matrix whose computation only has a computational complexity of O ( n p ) O(np) O ( n p ) .
Furthermore, the matrices Σ W ℓ − 1 \boldsymbol{\Sigma}_{\mathbf{W}_{\ell}}^{-1} Σ W ℓ − 1 (which can be formed directly from σ − 1 \boldsymbol{\sigma}^{-1} σ − 1 ) and Σ W ℓ − 1 + A ℓ T D − 1 A ℓ \boldsymbol{\Sigma}_{\mathbf{W}_{\ell}}^{-1}+\mathbf{A}_{\ell}^{\mathsf{T}}\mathbf{D}^{-1}\mathbf{A}_{\ell} Σ W ℓ − 1 + A ℓ T D − 1 A ℓ have the same sparsity as σ − 1 \boldsymbol{\sigma}^{-1} σ − 1 . Thus, process-level conditional independence relations result in computational improvements even within the Gibbs sampling algorithm of the E step. We note that similar expressions for the conditional expectation may be formed, though we have found the above most computationally efficient in practice. The proof is as follows.
Proof of Proposition 4. Let
Σ W − ℓ = block-diagonal ( Σ W ℓ ′ , ℓ ′ ≠ ℓ ) ∈ R ( L − 1 ) p × ( L − 1 ) p . \boldsymbol{\Sigma}_{\mathbf{W}_{-\ell}} = \operatorname{block\text{-}diagonal}(\boldsymbol{\Sigma}_{\mathbf{W}_{\ell'}}, \ell' \ne\ell) \in\mathbb{R}^{(L-1)p \times(L-1)p}. Σ W − ℓ = block - diagonal ( Σ W ℓ ′ , ℓ ′ = ℓ ) ∈ R ( L − 1 ) p × ( L − 1 ) p .
We have the joint distribution
[ W ℓ W − ℓ Y ] ∣ θ ∼ N ( [ 0 0 0 ] , [ Σ W ℓ 0 Σ W ℓ A ℓ T 0 Σ W − ℓ Σ W − ℓ A − ℓ T A ℓ Σ W ℓ A − ℓ Σ W − ℓ A Σ W A T + D ] ) . \begin{bmatrix}
\mathbf{W}_{\ell}\\
\mathbf{W}_{-\ell}\\
\mathbf{Y}
\end{bmatrix}
\mid\theta\sim\mathcal{N}\left(
\begin{bmatrix}
\mathbf{0}\\
\mathbf{0}\\
\mathbf{0}
\end{bmatrix},
\begin{bmatrix}
\boldsymbol{\Sigma}_{\mathbf{W}_{\ell}} & \mathbf{0} & \boldsymbol{\Sigma}_{\mathbf{W}_{\ell}}\mathbf{A}_{\ell}^{\mathsf{T}}\\
\mathbf{0} & \boldsymbol{\Sigma}_{\mathbf{W}_{-\ell}} & \boldsymbol{\Sigma}_{\mathbf{W}_{-\ell}}\mathbf{A}_{-\ell}^{\mathsf{T}}\\
\mathbf{A}_{\ell}\boldsymbol{\Sigma}_{\mathbf{W}_{\ell}} & \mathbf{A}_{-\ell}\boldsymbol{\Sigma}_{\mathbf{W}_{-\ell}} & \mathbf{A}\boldsymbol{\Sigma}_{\mathbf{W}}\mathbf{A}^{\mathsf{T}}+\mathbf{D}
\end{bmatrix}
\right). W ℓ W − ℓ Y ∣ θ ∼ N 0 0 0 , Σ W ℓ 0 A ℓ Σ W ℓ 0 Σ W − ℓ A − ℓ Σ W − ℓ Σ W ℓ A ℓ T Σ W − ℓ A − ℓ T A Σ W A T + D .
We begin with the variance. Let
Ω = [ Σ W − ℓ Σ W − ℓ A − ℓ T A − ℓ Σ W − ℓ A Σ W A T + D ] − 1 = [ Ω 11 Ω 21 T Ω 21 Ω 22 ] . \boldsymbol{\Omega} =
\begin{bmatrix}
\boldsymbol{\Sigma}_{\mathbf{W}_{-\ell}} & \boldsymbol{\Sigma}_{\mathbf{W}_{-\ell}}\mathbf{A}_{-\ell}^{\mathsf{T}}\\
\mathbf{A}_{-\ell}\boldsymbol{\Sigma}_{\mathbf{W}_{-\ell}} & \mathbf{A}\boldsymbol{\Sigma}_{\mathbf{W}}\mathbf{A}^{\mathsf{T}}+\mathbf{D}
\end{bmatrix}^{-1}
=
\begin{bmatrix}
\boldsymbol{\Omega}_{11} & \boldsymbol{\Omega}_{21}^{\mathsf{T}}\\
\boldsymbol{\Omega}_{21} & \boldsymbol{\Omega}_{22}
\end{bmatrix}. Ω = [ Σ W − ℓ A − ℓ Σ W − ℓ Σ W − ℓ A − ℓ T A Σ W A T + D ] − 1 = [ Ω 11 Ω 21 Ω 21 T Ω 22 ] .
Based on properties of the multivariate normal distribution,
Var [ W ℓ ∣ W − ℓ , Y , θ ] = Σ W ℓ − ( 0 Σ W ℓ A ℓ T ) Ω ( 0 A ℓ Σ W ℓ ) = Σ W ℓ − Σ W ℓ A ℓ T Ω 22 A ℓ Σ W ℓ . \operatorname{Var}[\mathbf{W}_{\ell} \mid\mathbf{W}_{-\ell},\mathbf{Y},\theta] = \boldsymbol{\Sigma}_{\mathbf{W}_{\ell}} - \left( \mathbf{0}\ \boldsymbol{\Sigma}_{\mathbf{W}_{\ell}}\mathbf{A}_{\ell}^{\mathsf{T}} \right)\boldsymbol{\Omega}
\begin{pmatrix}
\mathbf{0}\\
\mathbf{A}_{\ell}\boldsymbol{\Sigma}_{\mathbf{W}_{\ell}}
\end{pmatrix}
= \boldsymbol{\Sigma}_{\mathbf{W}_{\ell}}-\boldsymbol{\Sigma}_{\mathbf{W}_{\ell}}\mathbf{A}_{\ell}^{\mathsf{T}}\boldsymbol{\Omega}_{22}\mathbf{A}_{\ell}\boldsymbol{\Sigma}_{\mathbf{W}_{\ell}}. Var [ W ℓ ∣ W − ℓ , Y , θ ] = Σ W ℓ − ( 0 Σ W ℓ A ℓ T ) Ω ( 0 A ℓ Σ W ℓ ) = Σ W ℓ − Σ W ℓ A ℓ T Ω 22 A ℓ Σ W ℓ .
We next compute Ω 22 \Omega_{22} Ω 22 using the Schur complement:
Ω 22 = [ A Σ W A T + D − A − ℓ Σ W − ℓ Σ W − ℓ − 1 Σ W − ℓ A − ℓ T ] − 1 = [ A Σ W A T + D − A − ℓ Σ W − ℓ A − ℓ T ] − 1 . \Omega_{22} = \left[A\Sigma_W A^{\mathsf{T}} + D - A_{-\ell}\Sigma_{W_{-\ell}}\Sigma_{W_{-\ell}}^{-1}\Sigma_{W_{-\ell}}A_{-\ell}^{\mathsf{T}}\right]^{-1}
= \left[A\Sigma_W A^{\mathsf{T}} + D - A_{-\ell}\Sigma_{W_{-\ell}}A_{-\ell}^{\mathsf{T}}\right]^{-1}. Ω 22 = [ A Σ W A T + D − A − ℓ Σ W − ℓ Σ W − ℓ − 1 Σ W − ℓ A − ℓ T ] − 1 = [ A Σ W A T + D − A − ℓ Σ W − ℓ A − ℓ T ] − 1 .
We may decompose A Σ W A T = A ℓ Σ W ℓ A ℓ T + A − ℓ Σ W − ℓ A − ℓ T A\Sigma_W A^{\mathsf{T}} = A_{\ell}\Sigma_{W_{\ell}}A_{\ell}^{\mathsf{T}} + A_{-\ell}\Sigma_{W_{-\ell}}A_{-\ell}^{\mathsf{T}} A Σ W A T = A ℓ Σ W ℓ A ℓ T + A − ℓ Σ W − ℓ A − ℓ T , which results in Ω 22 = [ A ℓ Σ W ℓ A ℓ T + D ] − 1 \Omega_{22} = \left[A_{\ell}\Sigma_{W_{\ell}}A_{\ell}^{\mathsf{T}} + D\right]^{-1} Ω 22 = [ A ℓ Σ W ℓ A ℓ T + D ] − 1 . This leads to the variance
Var [ W ℓ ∣ W − ℓ , Y , θ ] = Σ W ℓ − Σ W ℓ A ℓ T [ A ℓ Σ W ℓ A ℓ T + D ] − 1 A ℓ Σ W ℓ = [ Σ W ℓ − 1 + A ℓ T D − 1 A ℓ ] − 1 . \operatorname{Var}[W_{\ell} \mid W_{-\ell},Y,\theta] = \Sigma_{W_{\ell}} - \Sigma_{W_{\ell}}A_{\ell}^{\mathsf{T}}\left[A_{\ell}\Sigma_{W_{\ell}}A_{\ell}^{\mathsf{T}} + D\right]^{-1}A_{\ell}\Sigma_{W_{\ell}}
= \left[\Sigma_{W_{\ell}}^{-1} + A_{\ell}^{\mathsf{T}}D^{-1}A_{\ell}\right]^{-1}. Var [ W ℓ ∣ W − ℓ , Y , θ ] = Σ W ℓ − Σ W ℓ A ℓ T [ A ℓ Σ W ℓ A ℓ T + D ] − 1 A ℓ Σ W ℓ = [ Σ W ℓ − 1 + A ℓ T D − 1 A ℓ ] − 1 .
where the last line follows from the Woodbury matrix identity.
The expectation follows in a similar manner:
E [ W ℓ ∣ W − ℓ , Y , θ ] = ( 0 Σ W ℓ A ℓ T ) Ω ( W − ℓ Y ) = Σ W ℓ A ℓ T Ω 21 W − ℓ + Σ W ℓ A ℓ T Ω 22 Y . \mathbb{E}[W_{\ell} \mid W_{-\ell},Y,\theta] = \begin{pmatrix} 0 & \Sigma_{W_{\ell}}A_{\ell}^{\mathsf{T}} \end{pmatrix}\Omega
\begin{pmatrix} W_{-\ell} \\ Y \end{pmatrix}
= \Sigma_{W_{\ell}}A_{\ell}^{\mathsf{T}}\Omega_{21}W_{-\ell} + \Sigma_{W_{\ell}}A_{\ell}^{\mathsf{T}}\Omega_{22}Y. E [ W ℓ ∣ W − ℓ , Y , θ ] = ( 0 Σ W ℓ A ℓ T ) Ω ( W − ℓ Y ) = Σ W ℓ A ℓ T Ω 21 W − ℓ + Σ W ℓ A ℓ T Ω 22 Y .
Using blockwise inversion, we have Ω 21 = − Ω 22 A − ℓ Σ W − ℓ Σ W − ℓ − 1 = − Ω 22 A − ℓ \Omega_{21} = -\Omega_{22}A_{-\ell}\Sigma_{W_{-\ell}}\Sigma_{W_{-\ell}}^{-1} = -\Omega_{22}A_{-\ell} Ω 21 = − Ω 22 A − ℓ Σ W − ℓ Σ W − ℓ − 1 = − Ω 22 A − ℓ , which gives
E [ W ℓ ∣ W − ℓ , Y , θ ] = Σ W ℓ A ℓ T Ω 22 [ Y − A − ℓ W − ℓ ] = Σ W ℓ A ℓ T [ A ℓ Σ W ℓ A ℓ T + D ] − 1 [ Y − A − ℓ W − ℓ ] . \mathbb{E}[W_{\ell} \mid W_{-\ell},Y,\theta] = \Sigma_{W_{\ell}}A_{\ell}^{\mathsf{T}}\Omega_{22}\left[Y-A_{-\ell}W_{-\ell}\right]
= \Sigma_{W_{\ell}}A_{\ell}^{\mathsf{T}}\left[A_{\ell}\Sigma_{W_{\ell}}A_{\ell}^{\mathsf{T}}+D\right]^{-1}\left[Y-A_{-\ell}W_{-\ell}\right]. E [ W ℓ ∣ W − ℓ , Y , θ ] = Σ W ℓ A ℓ T Ω 22 [ Y − A − ℓ W − ℓ ] = Σ W ℓ A ℓ T [ A ℓ Σ W ℓ A ℓ T + D ] − 1 [ Y − A − ℓ W − ℓ ] .
Applying the Woodbury matrix identity:
[ A ℓ Σ W ℓ A ℓ T + D ] − 1 = D − 1 − D − 1 A ℓ [ Σ W ℓ − 1 + A ℓ T D − 1 A ℓ ] − 1 A ℓ T D − 1 , \left[A_{\ell}\Sigma_{W_{\ell}}A_{\ell}^{\mathsf{T}}+D\right]^{-1}
= D^{-1}-D^{-1}A_{\ell}\left[\Sigma_{W_{\ell}}^{-1}+A_{\ell}^{\mathsf{T}}D^{-1}A_{\ell}\right]^{-1}A_{\ell}^{\mathsf{T}}D^{-1}, [ A ℓ Σ W ℓ A ℓ T + D ] − 1 = D − 1 − D − 1 A ℓ [ Σ W ℓ − 1 + A ℓ T D − 1 A ℓ ] − 1 A ℓ T D − 1 ,
the conditional expectation is
E [ W ℓ ∣ W − ℓ , Y , θ ] = Σ W ℓ A ℓ T [ D − 1 − D − 1 A ℓ [ Σ W ℓ − 1 + A ℓ T D − 1 A ℓ ] − 1 A ℓ T D − 1 ] \mathbb{E}[W_{\ell} \mid W_{-\ell},Y,\theta] = \Sigma_{W_{\ell}}A_{\ell}^{\mathsf{T}}\left[D^{-1}-D^{-1}A_{\ell}\left[\Sigma_{W_{\ell}}^{-1}+A_{\ell}^{\mathsf{T}}D^{-1}A_{\ell}\right]^{-1}A_{\ell}^{\mathsf{T}}D^{-1}\right] E [ W ℓ ∣ W − ℓ , Y , θ ] = Σ W ℓ A ℓ T [ D − 1 − D − 1 A ℓ [ Σ W ℓ − 1 + A ℓ T D − 1 A ℓ ] − 1 A ℓ T D − 1 ]
[ Y − A − ℓ W − ℓ ] . \left[Y-A_{-\ell}W_{-\ell}\right]. [ Y − A − ℓ W − ℓ ] .
The expression in the proposition then follows from factoring and rearranging.
Proofs of Propositions
Proof of prop1
Let θ p r o p \boldsymbol{\theta}_{\mathrm{prop}} θ prop be the parameters of the proposal distribution in the importance sampling. We have the following hierarchical structure:
X ℓ ∼ i . i . d . g ( ⋅ ∣ θ p r o p ) , X_{\ell} \mathrel{\overset{\mathrm{i.i.d.}}{\sim}} g(\cdot\mid\boldsymbol{\theta}_{\mathrm{prop}}), X ℓ ∼ i.i.d. g ( ⋅ ∣ θ prop ) ,
W ℓ ∣ X ℓ , θ p r o p , θ ∼ N ( 0 , 2 f ( X ℓ ) g ( X ℓ ) ) = N ( 0 p , Σ W ℓ ) , \mathbf{W}_{\ell} \mid X_{\ell},\boldsymbol{\theta}_{\mathrm{prop}},\boldsymbol{\theta} \sim\mathcal{N}\left(0,2\frac{f(X_{\ell})}{g(X_{\ell})}\right)=\mathcal{N}\left(\mathbf{0}_{p},\boldsymbol{\Sigma}_{W_{\ell}}\right), W ℓ ∣ X ℓ , θ prop , θ ∼ N ( 0 , 2 g ( X ℓ ) f ( X ℓ ) ) = N ( 0 p , Σ W ℓ ) ,
Y L ( s ) ∣ W , d , { X ℓ , ψ ℓ } ℓ = 1 L ∼ N ( ∑ ℓ c ℓ W ℓ , d ) . Y_{L}(s) \mid\mathbf{W},\mathbf{d},\{X_{\ell},\psi_{\ell}\}_{\ell=1}^{L} \sim\mathcal{N}\left(\sum_{\ell}c_{\ell}\mathbf{W}_{\ell},\mathbf{d}\right). Y L ( s ) ∣ W , d , { X ℓ , ψ ℓ } ℓ = 1 L ∼ N ( ℓ ∑ c ℓ W ℓ , d ) .
where c ℓ = 1 L cos ( ⟨ s , X ℓ ⟩ + ψ ℓ ) c_{\ell}=\frac{1}{\sqrt{L}}\cos(\langle s,X_{\ell}\rangle+\psi_{\ell}) c ℓ = L 1 cos (⟨ s , X ℓ ⟩ + ψ ℓ ) and ψ ℓ ∼ i . i . d . U [ 0 , 2 π ] \psi_{\ell}\mathrel{\overset{\mathrm{i.i.d.}}{\sim}}U[0,2\pi] ψ ℓ ∼ i.i.d. U [ 0 , 2 π ] . In matrix form:
W ∣ { X ℓ } ℓ = 1 L , θ p r o p , θ ∼ N ( 0 , Σ W ) , \mathbf{W}\mid\{X_{\ell}\}_{\ell=1}^{L},\boldsymbol{\theta}_{\mathrm{prop}},\boldsymbol{\theta}\sim\mathcal{N}(\mathbf{0},\boldsymbol{\Sigma}_{W}), W ∣ { X ℓ } ℓ = 1 L , θ prop , θ ∼ N ( 0 , Σ W ) ,
Y L ∣ W , d , { X ℓ , ψ ℓ } ℓ = 1 L ∼ N ( A W , D ) . Y_{L}\mid\mathbf{W},\mathbf{d},\{X_{\ell},\psi_{\ell}\}_{\ell=1}^{L}\sim\mathcal{N}(A\mathbf{W},D). Y L ∣ W , d , { X ℓ , ψ ℓ } ℓ = 1 L ∼ N ( A W , D ) .
where D = I n ⊗ d D=I_{n}\otimes\mathbf{d} D = I n ⊗ d , and Σ W = blockdiag { Σ W ℓ } \boldsymbol{\Sigma}_{W}=\operatorname{blockdiag}\{\boldsymbol{\Sigma}_{W_{\ell}}\} Σ W = blockdiag { Σ W ℓ } . By integrating out W \mathbf{W} W , we have
Y L ∣ d , { X ℓ , ψ ℓ } ℓ = 1 L ∼ N ( 0 , A Σ W A T + D ) . Y_{L}\mid\mathbf{d},\{X_{\ell},\psi_{\ell}\}_{\ell=1}^{L}\sim\mathcal{N}(\mathbf{0},A\boldsymbol{\Sigma}_{W}A^{\mathsf{T}}+D). Y L ∣ d , { X ℓ , ψ ℓ } ℓ = 1 L ∼ N ( 0 , A Σ W A T + D ) .
The first two moments of Y L Y_{L} Y L can be calculated by the law of total expectation and variance:
E [ Y ] = E { X ℓ , ψ ℓ } ℓ = 1 L [ E [ Y ∣ { X ℓ , ψ ℓ } ℓ = 1 L ] ] = 0 , \mathbb{E}[\mathbf{Y}]=\mathbb{E}_{\{X_{\ell},\psi_{\ell}\}_{\ell=1}^{L}}\left[\mathbb{E}\left[\mathbf{Y}\mid\{X_{\ell},\psi_{\ell}\}_{\ell=1}^{L}\right]\right]=0, E [ Y ] = E { X ℓ , ψ ℓ } ℓ = 1 L [ E [ Y ∣ { X ℓ , ψ ℓ } ℓ = 1 L ] ] = 0 ,
Var ( Y ) = Var { X ℓ , ψ ℓ } ℓ = 1 L ( E [ Y ∣ { X ℓ , ψ ℓ } ℓ = 1 L ] ) + E { X ℓ , ψ ℓ } ℓ = 1 L [ Var ( Y ∣ { X ℓ , ψ ℓ } ℓ = 1 L ) ] = 0 + E { X ℓ , ψ ℓ } ℓ = 1 L [ A Σ W A T + D ] . \begin{aligned}
\operatorname{Var}(\mathbf{Y})={}&\operatorname{Var}_{\{X_{\ell},\psi_{\ell}\}_{\ell=1}^{L}}\left(\mathbb{E}\left[\mathbf{Y}\mid\{X_{\ell},\psi_{\ell}\}_{\ell=1}^{L}\right]\right)\\
&+\mathbb{E}_{\{X_{\ell},\psi_{\ell}\}_{\ell=1}^{L}}\left[\operatorname{Var}\left(\mathbf{Y}\mid\{X_{\ell},\psi_{\ell}\}_{\ell=1}^{L}\right)\right]\\
={}&0+\mathbb{E}_{\{X_{\ell},\psi_{\ell}\}_{\ell=1}^{L}}\left[A\boldsymbol{\Sigma}_{W}A^{\mathsf{T}}+D\right].
\end{aligned} Var ( Y ) = = Var { X ℓ , ψ ℓ } ℓ = 1 L ( E [ Y ∣ { X ℓ , ψ ℓ } ℓ = 1 L ] ) + E { X ℓ , ψ ℓ } ℓ = 1 L [ Var ( Y ∣ { X ℓ , ψ ℓ } ℓ = 1 L ) ] 0 + E { X ℓ , ψ ℓ } ℓ = 1 L [ A Σ W A T + D ] .
Each p × p p \times p p × p block of E { X ℓ , ψ ℓ } ℓ = 1 L [ A Σ W A ⊤ ] \mathbb{E}_{\{X_\ell,\psi_\ell\}_{\ell=1}^{L}}[A\Sigma_W A^{\top}] E { X ℓ , ψ ℓ } ℓ = 1 L [ A Σ W A ⊤ ] is:
∑ ℓ E [ 2 L cos ( ⟨ s 1 , X ℓ ⟩ + ψ ℓ ) cos ( ⟨ s 2 , X ℓ ⟩ + ψ ℓ ) f ( X ℓ ) g ( X ℓ ) ] = 1 L ∑ ℓ E X ℓ [ E ψ ℓ [ { cos ( ⟨ s 1 − s 2 , X ℓ ⟩ ) + cos ( ⟨ s 1 + s 2 , X ℓ ⟩ + 2 ψ ℓ ) } f ( X ℓ ) g ( X ℓ ) | X ℓ ] ] = 1 L ∑ ℓ E X ℓ [ cos ( ⟨ s 1 − s 2 , X ℓ ⟩ ) f ( X ℓ ) g ( X ℓ ) ] = C ( s 1 , s 2 ) . \begin{aligned}
\sum_{\ell}\mathbb{E}\left[\frac{2}{L}\cos(\langle s_1,X_\ell\rangle+\psi_\ell)\cos(\langle s_2,X_\ell\rangle+\psi_\ell)\frac{f(X_\ell)}{g(X_\ell)}\right]
&=\frac{1}{L}\sum_{\ell}\mathbb{E}_{X_\ell}\left[\mathbb{E}_{\psi_\ell}\left[\left\{\cos(\langle s_1-s_2,X_\ell\rangle)+\cos(\langle s_1+s_2,X_\ell\rangle+2\psi_\ell)\right\}\frac{f(X_\ell)}{g(X_\ell)}\middle|X_\ell\right]\right]\\
&=\frac{1}{L}\sum_{\ell}\mathbb{E}_{X_\ell}\left[\cos(\langle s_1-s_2,X_\ell\rangle)\frac{f(X_\ell)}{g(X_\ell)}\right]\\
&=C(s_1,s_2).
\end{aligned} ℓ ∑ E [ L 2 cos (⟨ s 1 , X ℓ ⟩ + ψ ℓ ) cos (⟨ s 2 , X ℓ ⟩ + ψ ℓ ) g ( X ℓ ) f ( X ℓ ) ] = L 1 ℓ ∑ E X ℓ [ E ψ ℓ [ { cos (⟨ s 1 − s 2 , X ℓ ⟩) + cos (⟨ s 1 + s 2 , X ℓ ⟩ + 2 ψ ℓ ) } g ( X ℓ ) f ( X ℓ ) X ℓ ] ] = L 1 ℓ ∑ E X ℓ [ cos (⟨ s 1 − s 2 , X ℓ ⟩) g ( X ℓ ) f ( X ℓ ) ] = C ( s 1 , s 2 ) .
Express Y L ( s ) = ∑ ℓ c ℓ ( s ) W ℓ + ϵ = 1 L ∑ ℓ cos ( ⟨ s , X ℓ ⟩ + ψ ℓ ) W ℓ + ϵ ≡ 1 L ∑ ℓ V ℓ ( s ) + ϵ Y_L(s)=\sum_\ell c_\ell(s)W_\ell+\epsilon=\frac{1}{\sqrt{L}}\sum_\ell\cos(\langle s,X_\ell\rangle+\psi_\ell)W_\ell+\epsilon\equiv\frac{1}{\sqrt{L}}\sum_\ell V_\ell(s)+\epsilon Y L ( s ) = ∑ ℓ c ℓ ( s ) W ℓ + ϵ = L 1 ∑ ℓ cos (⟨ s , X ℓ ⟩ + ψ ℓ ) W ℓ + ϵ ≡ L 1 ∑ ℓ V ℓ ( s ) + ϵ . Since E ∥ W ℓ ∥ 2 = 2 tr ( σ ) < ∞ \mathbb{E}\lVert W_\ell\rVert^2=2\operatorname{tr}(\sigma)<\infty E ∥ W ℓ ∥ 2 = 2 tr ( σ ) < ∞ and the cosine factors are bounded, the finite-dimensional summands have finite second moments. The i.i.d. random vectors V ℓ ( s ) V_\ell(s) V ℓ ( s ) are not normally distributed, but by the multivariate central limit theorem, Y L → N ( 0 , Σ + D ) Y_L\to\mathcal{N}(0,\Sigma+D) Y L → N ( 0 , Σ + D ) as L → ∞ L\to\infty L → ∞ .
Preliminaries for the proofs of Propositions 2 and 3
With θ = ( σ − 1 , d , ι ) \theta=(\sigma^{-1},d,\iota) θ = ( σ − 1 , d , ι ) , the penalized complete-data log-likelihood has an exponential form
L P ( θ ∣ Y , W ) = − ψ ( θ ) + ⟨ U ( Y , W ) , ϕ ( θ ) ⟩ − P ( θ ) . \mathcal{L}_P(\theta\mid Y,W)=-\psi(\theta)+\langle U(Y,W),\phi(\theta)\rangle-P(\theta). L P ( θ ∣ Y , W ) = − ψ ( θ ) + ⟨ U ( Y , W ) , ϕ ( θ )⟩ − P ( θ ) .
where U ( Y , W ) U(Y,W) U ( Y , W ) is the complete-data sufficient statistic, ϕ ( θ ) = ( ϕ σ − 1 , ϕ d ) \phi(\theta)=(\phi_{\sigma^{-1}},\phi_d) ϕ ( θ ) = ( ϕ σ − 1 , ϕ d ) and ψ ( θ ) \psi(\theta) ψ ( θ ) are known functions of θ \theta θ .
U ( Y , W ) = ( ( W ℓ W ℓ ⊤ ) ℓ = 1 L , ( ∑ i = 1 n ( Y j ( s i ) − ∑ ℓ = 1 L c i , ℓ W ℓ j ) 2 ) j = 1 p ) = ( U σ − 1 , U d ) . U(Y,W)=\left(\left(W_\ell W_\ell^{\top}\right)_{\ell=1}^{L},\left(\sum_{i=1}^{n}\left(Y_j(s_i)-\sum_{\ell=1}^{L}c_{i,\ell}W_{\ell j}\right)^2\right)_{j=1}^{p}\right)=(U_{\sigma^{-1}},U_d). U ( Y , W ) = ( W ℓ W ℓ ⊤ ) ℓ = 1 L , i = 1 ∑ n ( Y j ( s i ) − ℓ = 1 ∑ L c i , ℓ W ℓ j ) 2 j = 1 p = ( U σ − 1 , U d ) .
ϕ ( θ ) = ( ( − g ( X ℓ ) 4 h ι ( X ℓ ) − 1 σ − 1 h ι ( X ℓ ) − 1 ) ℓ = 1 L , ( − 1 2 d j ) j = 1 p ) . \phi(\theta)=\left(\left(-\frac{g(X_\ell)}{4}h_\iota(X_\ell)^{-1}\sigma^{-1}h_\iota(X_\ell)^{-1}\right)_{\ell=1}^{L},\left(-\frac{1}{2d_j}\right)_{j=1}^{p}\right). ϕ ( θ ) = ( ( − 4 g ( X ℓ ) h ι ( X ℓ ) − 1 σ − 1 h ι ( X ℓ ) − 1 ) ℓ = 1 L , ( − 2 d j 1 ) j = 1 p ) .
ψ ( θ ) = n 2 log ∣ d ∣ − L 2 log det σ − 1 + ∑ ℓ = 1 L log ∣ h ι ( X ℓ ) ∣ . \psi(\theta)=\frac{n}{2}\log|d|-\frac{L}{2}\log\det\sigma^{-1}+\sum_{\ell=1}^{L}\log|h_\iota(X_\ell)|. ψ ( θ ) = 2 n log ∣ d ∣ − 2 L log det σ − 1 + ℓ = 1 ∑ L log ∣ h ι ( X ℓ ) ∣.
For a sufficient statistic u u u , let L P ( θ ∣ u ) \mathcal{L}_P(\theta\mid u) L P ( θ ∣ u ) be the complete-data log-likelihood with U ( Y , W ) U(Y,W) U ( Y , W ) replaced by u u u . Parameter ι \iota ι enters f ( Y , W ; θ ) f(Y,W;\theta) f ( Y , W ; θ ) only through h ι ( X ℓ ) h_\iota(X_\ell) h ι ( X ℓ ) in ϕ σ − 1 \phi_{\sigma^{-1}} ϕ σ − 1 and ψ \psi ψ , sharing U σ − 1 U_{\sigma^{-1}} U σ − 1 and has no sufficient statistic of its own. Additionally, let u ˉ ( θ ) = E θ [ U ( Y , W ) ∣ Y ] \bar{u}(\theta)=\mathbb{E}_{\theta}[U(Y,W)\mid Y] u ˉ ( θ ) = E θ [ U ( Y , W ) ∣ Y ] be the conditional mean of the complete-data sufficient statistic given the observed data, and U ‾ m = 1 B ∑ b = 1 B U ( Y , W m , b ) \overline{U}_m = \frac{1}{B}\sum_{b=1}^{B} U(\mathbf{Y},\mathbf{W}_{m,b}) U m = B 1 ∑ b = 1 B U ( Y , W m , b ) its sample estimate. For notational simplicity, we write the block averages without burn-in. The same convergence arguments apply when a fixed number B 0 < B B_0 < B B 0 < B of initial sweeps is discarded and the averages are taken over the remaining B − B 0 B - B_0 B − B 0 states. Here, θ ′ = T ( u , θ ) \boldsymbol{\theta}' = \mathcal{T}(u,\boldsymbol{\theta}) θ ′ = T ( u , θ ) is the updated parameter after one iteration, T ( u , θ ^ ( u ) ) = θ ^ ( u ) \mathcal{T}(u,\widehat{\boldsymbol{\theta}}(u)) = \widehat{\boldsymbol{\theta}}(u) T ( u , θ ( u )) = θ ( u ) .
In the Gibbs algorithm, let π θ = p g ( w ∣ Y , θ ) \pi_{\boldsymbol{\theta}} = p_g(w \mid\mathbf{Y},\boldsymbol{\theta}) π θ = p g ( w ∣ Y , θ ) be the fixed-L L L posterior distribution of the latent values, so u ‾ ( θ ) = π θ U \overline{u}(\boldsymbol{\theta}) = \pi_{\boldsymbol{\theta}}U u ( θ ) = π θ U , and let P θ \mathbf{P}_{\boldsymbol{\theta}} P θ denote the Markov kernel of one complete Gibbs scan targeting π θ \pi_{\boldsymbol{\theta}} π θ , so
W m , 0 = W m − 1 , B , W m , b ∣ F m , b − 1 ∼ P θ ( m − 1 ) ( W m , b − 1 , ⋅ ) , b = 1 , … , B . (12) \begin{aligned}
\mathbf{W}_{m,0} &= \mathbf{W}_{m-1,B},\\
\mathbf{W}_{m,b} \mid\mathcal{F}_{m,b-1} &\sim\mathbf{P}_{\boldsymbol{\theta}^{(m-1)}}(\mathbf{W}_{m,b-1},\cdot), \qquad b=1,\ldots,B. \tag*{(12)}
\end{aligned} W m , 0 W m , b ∣ F m , b − 1 = W m − 1 , B , ∼ P θ ( m − 1 ) ( W m , b − 1 , ⋅ ) , b = 1 , … , B . (12)
where { F m , b } \{\mathcal{F}_{m,b}\} { F m , b } denote the filtration of the algorithm recording the initialization and algorithmic history through the b b b -th Gibbs sweep of iteration m m m . Write F m − 1 = F m , 0 \mathcal{F}_{m-1} = \mathcal{F}_{m,0} F m − 1 = F m , 0 . The SAECM recursion is
u ( m ) = ( 1 − γ m ) u ( m − 1 ) + γ m U ‾ m , θ ( m ) = T ( u ( m ) , θ ( m − 1 ) ) . (13) \begin{aligned}
u^{(m)} &= (1-\gamma_m)u^{(m-1)}+\gamma_m\overline{U}_m,\\
\boldsymbol{\theta}^{(m)} &= \mathcal{T}(u^{(m)},\boldsymbol{\theta}^{(m-1)}). \tag*{(13)}
\end{aligned} u ( m ) θ ( m ) = ( 1 − γ m ) u ( m − 1 ) + γ m U m , = T ( u ( m ) , θ ( m − 1 ) ) . (13)
Proof of prop2
Assumptions
Let Θ \Theta Θ denote the open interior parameter space and let U \mathcal{U} U denote the open domain of admissible averaged sufficient statistics. The latter is a domain for the SA averages. Individual raw statistics U ( Y , w ) U(\mathbf{Y},w) U ( Y , w ) need not belong to U \mathcal{U} U . Define a graphical-lasso stratum to be a parameter region with fixed zeros and fixed signs of the nonzero precision entries, and the derivatives on a stratum use its free coordinates.
Assumption A1 (Fixed-L L L regularity). Assume ψ \psi ψ , ϕ \phi ϕ , and u ˉ \bar{u} u ˉ are C 2 C^2 C 2 on Θ \Theta Θ , and T \mathcal{T} T and θ ^ \hat{\theta} θ ^ corresponding to each graphical-LASSO stratum in Θ \Theta Θ are C 2 C^2 C 2 . For every compact K ⊂ Θ K \subset\Theta K ⊂ Θ , p g ( Y , w ; ⋅ ) p_g(\boldsymbol{Y},w;\cdot) p g ( Y , w ; ⋅ ) is C 1 C^1 C 1 , and
∫ sup θ ∈ K ∥ ∇ θ p g ( Y , w ; θ ) ∥ d w < ∞ . \int\sup_{\theta\in K}\lVert\nabla_\theta p_g(\boldsymbol{Y},w;\theta)\rVert\,dw < \infty. ∫ θ ∈ K sup ∥ ∇ θ p g ( Y , w ; θ )∥ d w < ∞. Hence Fisher’s identity applies.
Assumption A2 (Moment bound for the Gibbs E step). Let π θ ( d w ) = p g ( w ∣ Y ; θ ) d w \pi_\theta(dw)=p_g(w\mid\boldsymbol{Y};\theta)\,dw π θ ( d w ) = p g ( w ∣ Y ; θ ) d w denote the finite-L L L posterior distribution of W = ( W 1 T , … , W L T ) T W=(W_1^\mathsf{T},\ldots,W_L^\mathsf{T})^\mathsf{T} W = ( W 1 T , … , W L T ) T , and let P θ P_\theta P θ be the transition kernel obtained by one complete Gibbs sweep, π θ P θ = π θ \pi_\theta P_\theta=\pi_\theta π θ P θ = π θ . If one stochastic E step performs B B B complete Gibbs sweeps, denote its endpoint kernel by Π θ = P θ B \Pi_\theta=P_\theta^B Π θ = P θ B .
Set V ( w ) = 1 + ∥ w ∥ 2 V(w)=1+\lVert w\rVert^2 V ( w ) = 1 + ∥ w ∥ 2 . For every compact set K ⊆ Θ K\subseteq\Theta K ⊆ Θ , there exist constants κ K ∈ ( 0 , 1 ) \kappa_K\in(0,1) κ K ∈ ( 0 , 1 ) and b K < ∞ b_K<\infty b K < ∞ such that ( Π θ V 2 ) ( w ) = E θ [ V ( W + ) 2 ∣ W 0 = w ] ≤ κ K V ( w ) 2 + b K (\Pi_\theta V^2)(w)=\mathbb{E}_\theta[V(W^+)^2\mid W_0=w]\leq\kappa_KV(w)^2+b_K ( Π θ V 2 ) ( w ) = E θ [ V ( W + ) 2 ∣ W 0 = w ] ≤ κ K V ( w ) 2 + b K . The initial Gibbs state is deterministic or satisfies E [ V 2 ( W 0 , B ) ] < ∞ \mathbb{E}[V^2(W_{0,B})]<\infty E [ V 2 ( W 0 , B )] < ∞ .
Assumption A3 (Step sizes). The gains satisfy 0 < γ m ≤ 1 0<\gamma_m\leq1 0 < γ m ≤ 1 , are nonincreasing, and
γ m → 0 , ∑ m γ m = ∞ , ∑ m γ m 2 < ∞ , γ m − q γ m → 1 for every fixed q . (14) \gamma_m\to0,\qquad\sum_m\gamma_m=\infty,\qquad\sum_m\gamma_m^2<\infty,\qquad\frac{\gamma_{m-q}}{\gamma_m}\to1\quad\text{for every fixed }q. \tag*{(14)} γ m → 0 , m ∑ γ m = ∞ , m ∑ γ m 2 < ∞ , γ m γ m − q → 1 for every fixed q . (14)
Assumption A4 (Almost-sure compactness). Almost surely, there exist compact sets K u ⊂ U K_u\subset\mathcal{U} K u ⊂ U and K θ ⊂ Θ K_\theta\subset\Theta K θ ⊂ Θ such that
{ ( u ( m ) , θ ( m ) ) : m ≥ 0 } ‾ ⊆ K u × K θ . \overline{\{(u^{(m)},\theta^{(m)}):m\geq0\}}\subseteq K_u\times K_\theta. {( u ( m ) , θ ( m ) ) : m ≥ 0 } ⊆ K u × K θ .
Assumption A5 (Finite-cycle contraction). The selected CM updates are exact, well-defined, and interior in their respective free-coordinate domains. For every compact K u ⊂ U K_u\subset\mathcal{U} K u ⊂ U and K θ ⊂ Θ K_\theta\subset\Theta K θ ⊂ Θ , there exist a compact K ~ θ ⊂ Θ \widetilde{K}_\theta\subset\Theta K θ ⊂ Θ , q ≥ 1 q\geq1 q ≥ 1 , and κ T ∈ ( 0 , 1 ) \kappa_{\mathcal{T}}\in(0,1) κ T ∈ ( 0 , 1 ) such that
K θ ⊂ K ~ θ , T ( K u × K ~ θ ) ⊂ K ~ θ , K_\theta\subset\widetilde{K}_\theta,\qquad\mathcal{T}(K_u\times\widetilde{K}_\theta)\subset\widetilde{K}_\theta, K θ ⊂ K θ , T ( K u × K θ ) ⊂ K θ , and, writing T u ( θ ) = T ( u , θ ) \mathcal{T}_u(\theta)=\mathcal{T}(u,\theta) T u ( θ ) = T ( u , θ ) ,
∥ T u q ( θ ) − θ ^ ( u ) ∥ ≤ κ T ∥ θ − θ ^ ( u ) ∥ , ( u , θ ) ∈ K u × K ~ θ . \lVert\mathcal{T}_u^q(\theta)-\hat{\theta}(u)\rVert\leq\kappa_{\mathcal{T}}\lVert\theta-\hat{\theta}(u)\rVert,\qquad(u,\theta)\in K_u\times\widetilde{K}_\theta. ∥ T u q ( θ ) − θ ^ ( u )∥ ≤ κ T ∥ θ − θ ^ ( u )∥ , ( u , θ ) ∈ K u × K θ .
Moreover, T \mathcal{T} T , θ ^ \hat{\theta} θ ^ , and u ˉ \bar{u} u ˉ are Lipschitz on K u × K ~ θ K_u \times\widetilde{K}_{\theta} K u × K θ , K u K_u K u , and K ~ θ \widetilde{K}_{\theta} K θ , respectively, with finite constants C T C_{\mathcal{T}} C T , C θ ^ C_{\hat{\theta}} C θ ^ , and C u ˉ C_{\bar{u}} C u ˉ .
Assumption A6 (Eventual graphical-LASSO pattern stability). Let M ζ \mathcal{M}_{\zeta} M ζ denote the parameter region with a fixed zero pattern and fixed signs for the nonzero entries of σ − 1 \boldsymbol{\sigma}^{-1} σ − 1 , and let L u : = { u : θ ^ ( u ) ∈ Θ g , P ⋆ } \mathcal{L}_u := \{u : \hat{\theta}(u) \in\Theta_{g,P}^{\star}\} L u := { u : θ ^ ( u ) ∈ Θ g , P ⋆ } . For each stratum, let K ζ , L K_{\zeta,L} K ζ , L be a deterministic compact set contained in the deterministic open set N ζ , L ⊂ U \mathcal{N}_{\zeta,L} \subset\mathcal{U} N ζ , L ⊂ U . Almost surely, there exist ζ \zeta ζ and m 0 < ∞ m_0 < \infty m 0 < ∞ such that
K t a i l , L : = { u ( m ) : m ≥ m 0 } ⊆ K ζ , L ⊂ N ζ , L . K_{\mathrm{tail},L} := \{u^{(m)} : m \ge m_0\} \subseteq K_{\zeta,L} \subset\mathcal{N}_{\zeta,L}. K tail , L := { u ( m ) : m ≥ m 0 } ⊆ K ζ , L ⊂ N ζ , L . For every sufficiently large m m m , both θ ( m ) \theta^{(m)} θ ( m ) and θ ^ ( u ( m ) ) \hat{\theta}(u^{(m)}) θ ^ ( u ( m ) ) belong to M ζ \mathcal{M}_{\zeta} M ζ , and θ ^ ( u ) ∈ M ζ \hat{\theta}(u) \in\mathcal{M}_{\zeta} θ ^ ( u ) ∈ M ζ throughout N ζ , L \mathcal{N}_{\zeta,L} N ζ , L . On this neighborhood, V P ( u ) : = − ℓ g , P ( θ ^ ( u ) ) V_P(u) := -\ell_{g,P}(\hat{\theta}(u)) V P ( u ) := − ℓ g , P ( θ ^ ( u )) is C 2 C^2 C 2 , with bounded first and second derivatives on a neighborhood of K ζ , L K_{\zeta,L} K ζ , L . Define Z L : = K t a i l , L ∩ L u Z_L := K_{\mathrm{tail},L} \cap\mathcal{L}_u Z L := K tail , L ∩ L u .
Assumption A7 (Local curvature and inactive-coordinate optimality). Let F F F denote the free coordinates of the eventual graphical LASSO pattern. For every u ∈ N ζ , L u \in\mathcal{N}_{\zeta,L} u ∈ N ζ , L , the Hessian H ( u ) = ∇ θ F θ F 2 L P ( θ ^ ( u ) ∣ u ) \mathcal{H}(u) = \nabla^2_{\theta_F \theta_F} \mathcal{L}_P(\hat{\theta}(u) \mid u) H ( u ) = ∇ θ F θ F 2 L P ( θ ^ ( u ) ∣ u ) is negative semidefinite. Moreover, whenever ∇ θ F ℓ g , P ( θ ^ ( u ) ) = 0 \nabla_{\theta_F}\ell_{g,P}(\hat{\theta}(u)) = 0 ∇ θ F ℓ g , P ( θ ^ ( u )) = 0 , every inactive off-diagonal precision entry satisfies
∣ ∂ ∂ η j k ℓ g ( θ ^ ( u ) ) ∣ ≤ 2 λ . \left|\frac{\partial}{\partial\eta_{jk}}\ell_g(\hat{\theta}(u))\right| \le2\lambda. ∂ η j k ∂ ℓ g ( θ ^ ( u )) ≤ 2 λ .
Lemmas
The SAECM update is decomposed as follows:
u ( m ) = u ( m − 1 ) + γ m { h ( u ( m − 1 ) ) + ε m + R m } , (15) u^{(m)} = u^{(m-1)} + \gamma_m\{h(u^{(m-1)}) + \varepsilon_m + R_m\}, \tag*{(15)} u ( m ) = u ( m − 1 ) + γ m { h ( u ( m − 1 ) ) + ε m + R m } , (15)
where h ( u ) = u ˉ ( θ ^ ( u ) ) − u h(u) = \bar{u}(\hat{\theta}(u)) - u h ( u ) = u ˉ ( θ ^ ( u )) − u is the mean field, ε m = U ‾ m − u ˉ ( θ ( m − 1 ) ) \varepsilon_m = \overline{U}_m - \bar{u}(\theta^{(m-1)}) ε m = U m − u ˉ ( θ ( m − 1 ) ) is the error term, and R m = u ˉ ( θ ( m − 1 ) ) − u ˉ ( θ ^ ( u ( m − 1 ) ) ) R_m = \bar{u}(\theta^{(m-1)}) - \bar{u}(\hat{\theta}(u^{(m-1)})) R m = u ˉ ( θ ( m − 1 ) ) − u ˉ ( θ ^ ( u ( m − 1 ) )) is the tracking remainder. Then, Theorem 1 shows the tracking remainder is summable, Theorem 2 shows the noise can be controlled, and Theorem 3 provides the mean field descent identity. Together these imply convergence to the stationary set.
The proofs of Lemmas 1 and 2 rely on conditional-moment and martingale-difference bounds for the Gibbs noise. These bounds have deterministic constants only on a fixed compact set depending on the entire realized path by Assumption A4. Therefore, it is required to create a localized process on a countable deterministic compact exhaustion, which does not depend on the future event to avoid breaking the martingale-difference property. Then, almost surely, such localization has no exit, and hence coincides with the original recursion.
Let K u , j × K θ , j K_{u,j} \times K_{\theta,j} K u , j × K θ , j , j ≥ 1 j \ge1 j ≥ 1 , be a deterministic compact exhaustion of U × Θ \mathcal{U} \times\Theta U × Θ , with each member contained in the interior of the next. Define the exit time
τ j = inf { m ≥ 1 : ( u ( m ) , θ ( m ) ) ∉ K u , j × K θ , j } . \tau_j = \inf\{m \ge1 : (u^{(m)}, \theta^{(m)}) \notin K_{u,j} \times K_{\theta,j}\}. τ j = inf { m ≥ 1 : ( u ( m ) , θ ( m ) ) ∈ / K u , j × K θ , j } .
For each j j j , define a localized process that follows the original recursion until τ j \tau_j τ j . After an attempted exit, hold ( u , θ ) (u, \theta) ( u , θ ) fixed and continue the Gibbs endpoints under the corresponding fixed kernel, so ( u , θ ) (u, \theta) ( u , θ ) remains in the compact set. When { τ j = ∞ } \{\tau_j = \infty\} { τ j = ∞ } , the extension coincides with the original recursion. By Assumption A4, P ( ⋃ j ≥ 1 { ( u ( m ) , θ ( m ) ) ∈ K u , j × K θ , j for all m } ) = 1 \mathbb{P}\left(\bigcup_{j \ge1}\{(u^{(m)}, \theta^{(m)}) \in K_{u,j} \times K_{\theta,j}\text{ for all }m\}\right) = 1 P ( ⋃ j ≥ 1 {( u ( m ) , θ ( m ) ) ∈ K u , j × K θ , j for all m } ) = 1 . Thus, it is enough to prove the proposition for each localized process.
Fix a localization index j j j , write τ = τ j \tau= \tau_j τ = τ j , K u = K u , j K_u = K_{u,j} K u = K u , j and K = K θ , j K = K_{\theta,j} K = K θ , j , and let all quantities in Theorems 1 and 2 refer to its localized process.
Lemma 1 (Localized moments and finite-cycle tracking). Define
Δ m = ∥ u ( m ) − u ( m − 1 ) ∥ , e m τ = 1 { m < τ } ∥ θ ( m ) − θ ^ ( u ( m ) ) ∥ . \Delta_m = \left\|u^{(m)} - u^{(m-1)}\right\|,\qquad e_m^\tau= \mathbf{1}_{\{m<\tau\}}\left\|\theta^{(m)} - \hat{\theta}(u^{(m)})\right\|. Δ m = u ( m ) − u ( m − 1 ) , e m τ = 1 { m < τ } θ ( m ) − θ ^ ( u ( m ) ) . Then, almost surely,
Δ m → 0 , ∑ m γ m Δ m < ∞ , e m τ → 0 , ∑ m γ m e m − 1 τ < ∞ . (16) \begin{aligned}
\Delta_m &\to0,\qquad\sum_m \gamma_m\Delta_m < \infty,\\
e_m^\tau&\to0,\qquad\sum_m \gamma_m e_{m-1}^\tau< \infty. \tag*{(16)}
\end{aligned} Δ m e m τ → 0 , m ∑ γ m Δ m < ∞ , → 0 , m ∑ γ m e m − 1 τ < ∞. (16) In addition,
sup m E V 2 ( W m , B ) < ∞ , max 1 ≤ b ≤ B sup m E ∥ U ( Y , W m , b ) ∥ 2 < ∞ , \sup_m \mathbb{E}V^2(W_{m,B}) < \infty,\qquad\max_{1\le b\le B}\sup_m \mathbb{E}\left\|U(Y,W_{m,b})\right\|^2 < \infty, m sup E V 2 ( W m , B ) < ∞ , 1 ≤ b ≤ B max m sup E ∥ U ( Y , W m , b ) ∥ 2 < ∞ , ∑ m γ m ∥ θ ( m ) − θ ( m − 1 ) ∥ V ( W m , B ) < ∞ almost surely . (17) \sum_m \gamma_m\left\|\theta^{(m)}-\theta^{(m-1)}\right\|V(W_{m,B}) < \infty\quad\text{almost surely}. \tag*{(17)} m ∑ γ m θ ( m ) − θ ( m − 1 ) V ( W m , B ) < ∞ almost surely . (17)
Proof. Taking expectation over the past, the inequality in Assumption A2 gives
E [ V 2 ( W m , B ) ] ≤ κ K E [ V 2 ( W m − 1 , B ) ] + b K . \mathbb{E}\left[V^{2}(\mathbf{W}_{m,B})\right] \le\kappa_{K}\mathbb{E}\left[V^{2}(\mathbf{W}_{m-1,B})\right] + b_{K}. E [ V 2 ( W m , B ) ] ≤ κ K E [ V 2 ( W m − 1 , B ) ] + b K .
Iterating this scalar inequality gives
sup m E V 2 ( W m , B ) ≤ E V 2 ( W 0 , B ) + b K 1 − κ K < ∞ . \sup_{m}\mathbb{E}V^{2}(\mathbf{W}_{m,B}) \le\mathbb{E}V^{2}(\mathbf{W}_{0,B}) + \frac{b_{K}}{1-\kappa_{K}} < \infty. m sup E V 2 ( W m , B ) ≤ E V 2 ( W 0 , B ) + 1 − κ K b K < ∞.
The finite Gibbs steps have bounded coefficients and fourth moments, so the second moment of U ‾ m \overline{U}_{m} U m is bounded. By Markov inequality and Assumption A3,
∑ m P ( γ m ∥ U ‾ m ∥ > ε ) ≤ ε − 2 ∑ m γ m 2 E ∥ U ‾ m ∥ 2 < ∞ , \sum_{m}\mathbb{P}\left(\gamma_{m}\left\|\overline{U}_{m}\right\|>\varepsilon\right) \le\varepsilon^{-2}\sum_{m}\gamma_{m}^{2}\mathbb{E}\left\|\overline{U}_{m}\right\|^{2}<\infty, m ∑ P ( γ m U m > ε ) ≤ ε − 2 m ∑ γ m 2 E U m 2 < ∞ ,
and therefore γ m ∥ U ‾ m ∥ → 0 \gamma_{m}\left\|\overline{U}_{m}\right\|\to0 γ m U m → 0 almost surely by Borel–Cantelli lemma. Next, we plug in the SAEM update into Δ m \Delta_{m} Δ m and apply triangle inequality:
Δ m ≤ γ m ∥ U ‾ m − u ( m − 1 ) ∥ ≤ γ m ∥ U ‾ m ∥ + γ m ∥ u ( m − 1 ) ∥ → 0 , \Delta_{m}\le\gamma_{m}\left\|\overline{U}_{m}-u^{(m-1)}\right\|\le\gamma_{m}\left\|\overline{U}_{m}\right\|+\gamma_{m}\left\|u^{(m-1)}\right\|\to0, Δ m ≤ γ m U m − u ( m − 1 ) ≤ γ m U m + γ m u ( m − 1 ) → 0 ,
by localization and boundedness of u ( m − 1 ) u^{(m-1)} u ( m − 1 ) . Moreover,
∑ m γ m Δ m ≤ ∑ m γ m 2 { ∥ U ‾ m ∥ + sup u ∈ K u ∥ u ∥ } . \sum_{m}\gamma_{m}\Delta_{m}\le\sum_{m}\gamma_{m}^{2}\left\{\left\|\overline{U}_{m}\right\|+\sup_{u\in K_{u}}\left\|u\right\|\right\}. m ∑ γ m Δ m ≤ m ∑ γ m 2 { U m + u ∈ K u sup ∥ u ∥ } .
The expectation of the right hand side is finite, proving (S6).
Tracking the frozen-statistic CM fixed point. Compare q q q actual CM cycles with q q q cycles at the frozen statistic and use the Lipschitz bounds in Assumption A5
e m + q τ ≤ κ T e m τ + C ∑ k = m + 1 m + q Δ k . e^{\tau}_{m+q}\le\kappa_{T}e^{\tau}_{m}+C\sum_{k=m+1}^{m+q}\Delta_{k}. e m + q τ ≤ κ T e m τ + C k = m + 1 ∑ m + q Δ k .
Since Δ m → 0 \Delta_{m}\to0 Δ m → 0 and ∑ m γ m Δ m < ∞ \sum_{m}\gamma_{m}\Delta_{m}<\infty ∑ m γ m Δ m < ∞ , e m τ → 0 e^{\tau}_{m}\to0 e m τ → 0 , ∑ m γ m e m − 1 τ < ∞ \sum_{m}\gamma_{m}e^{\tau}_{m-1}<\infty ∑ m γ m e m − 1 τ < ∞ by geometric summation.
Weighted parameter movement. The final bound is needed in Lemma 2 because a change in the Gibbs kernel is multiplied there by the possibly unbounded endpoint W m , B \mathbf{W}_{m,B} W m , B . Since
∥ θ ( m ) − θ ( m − 1 ) ∥ ≤ e m τ + e m − 1 τ + C θ ^ Δ m , \left\|\theta^{(m)}-\theta^{(m-1)}\right\|\le e^{\tau}_{m}+e^{\tau}_{m-1}+C_{\hat{\theta}}\Delta_{m}, θ ( m ) − θ ( m − 1 ) ≤ e m τ + e m − 1 τ + C θ ^ Δ m ,
square the tracking recursion and choose the Young-inequality constant so that the coefficient of ( ε m τ ) 2 (\varepsilon_m^\tau)^2 ( ε m τ ) 2 is strictly below one. Using E Δ m 2 = O ( γ m 2 ) \mathbb{E}\Delta_m^2 = O(\gamma_m^2) E Δ m 2 = O ( γ m 2 ) and the fixed-lag gain condition then gives, for some r < 1 r < 1 r < 1 , E ( ε m τ ) 2 = O ( γ m 2 ) + O ( r m ) \mathbb{E}(\varepsilon_m^\tau)^2 = O(\gamma_m^2)+O(r^m) E ( ε m τ ) 2 = O ( γ m 2 ) + O ( r m ) . Consequently, E ∥ θ ( m ) − θ ( m − 1 ) ∥ 2 = O ( γ m 2 ) + O ( r m ) \mathbb{E}\left\|\theta^{(m)}-\theta^{(m-1)}\right\|^2 = O(\gamma_m^2)+O(r^m) E θ ( m ) − θ ( m − 1 ) 2 = O ( γ m 2 ) + O ( r m ) .
By Cauchy–Schwarz and the uniform bound on E V 2 ( W m , B ) \mathbb{E}V^2(W_{m,B}) E V 2 ( W m , B ) ,
E [ ∥ θ ( m ) − θ ( m − 1 ) ∥ V ( W m , B ) ] ≤ E [ ∥ θ ( m ) − θ ( m − 1 ) ∥ 2 ] 1 / 2 E [ V 2 ( W m , B ) ] 1 / 2 ≤ C { γ m + r m / 2 } . \begin{aligned}
\mathbb{E}\left[\left\|\theta^{(m)}-\theta^{(m-1)}\right\|V(W_{m,B})\right] \le\mathbb{E}\left[\left\|\theta^{(m)}-\theta^{(m-1)}\right\|^2\right]^{1/2}\mathbb{E}\left[V^2(W_{m,B})\right]^{1/2} \\
&\le C\{\gamma_m+r^{m/2}\}.
\end{aligned} E [ θ ( m ) − θ ( m − 1 ) V ( W m , B ) ] ≤ E [ θ ( m ) − θ ( m − 1 ) 2 ] 1/2 E [ V 2 ( W m , B ) ] 1/2 ≤ C { γ m + r m /2 } .
Therefore,
∑ m γ m E [ ∥ θ ( m ) − θ ( m − 1 ) ∥ V ( W m , B ) ] < ∞ . \sum_m\gamma_m\mathbb{E}\left[\left\|\theta^{(m)}-\theta^{(m-1)}\right\|V(W_{m,B})\right]<\infty. m ∑ γ m E [ θ ( m ) − θ ( m − 1 ) V ( W m , B ) ] < ∞.
Tonelli’s theorem now yields
∑ m γ m ∥ θ ( m ) − θ ( m − 1 ) ∥ V ( W m , B ) < ∞ , almost surely. \sum_m\gamma_m\left\|\theta^{(m)}-\theta^{(m-1)}\right\|V(W_{m,B})<\infty,\qquad\text{almost surely.} m ∑ γ m θ ( m ) − θ ( m − 1 ) V ( W m , B ) < ∞ , almost surely.
Lemma 2 (Persistent-Gibbs noise control). Almost surely,
∑ m γ m ε m converges , (18) \sum_m\gamma_m\varepsilon_m\ \text{converges}, \tag*{(18)} m ∑ γ m ε m converges , (18) ∑ m γ m 2 ∥ ε m ∥ 2 < ∞ . (19) \sum_m\gamma_m^2\|\varepsilon_m\|^2<\infty. \tag*{(19)} m ∑ γ m 2 ∥ ε m ∥ 2 < ∞. (19) Moreover, for every bounded globally Lipschitz matrix function A A A ,
∑ m γ m A ( u ( m − 1 ) ) ε m converges almost surely. (20) \sum_m\gamma_m A(u^{(m-1)})\varepsilon_m\ \text{converges almost surely.} \tag*{(20)} m ∑ γ m A ( u ( m − 1 ) ) ε m converges almost surely. (20)
Proof. Starting a retained block at W 0 = w W_0=w W 0 = w , let μ θ = E π θ [ W ] \mu_\theta=\mathbb{E}_{\pi_\theta}[W] μ θ = E π θ [ W ] denote the posterior mean.
Define
B θ ( w ) = E θ [ 1 B ∑ b = 1 B U ( Y , W b ) ∣ W 0 = w ] − u ˉ ( θ ) , B_\theta(w)=\mathbb{E}_\theta\left[\left.\frac{1}{B}\sum_{b=1}^{B}U(Y,W_b)\right|W_0=w\right]-\bar{u}(\theta), B θ ( w ) = E θ [ B 1 b = 1 ∑ B U ( Y , W b ) W 0 = w ] − u ˉ ( θ ) ,
which is a quadratic polynomial. If we let z = w − μ θ z=w-\mu_\theta z = w − μ θ , then one scalar component can be written as B θ ( w ) = c θ + b θ ⊤ z + z ⊤ A θ z B_\theta(w)=c_\theta+b_\theta^\top z+z^\top A_\theta z B θ ( w ) = c θ + b θ ⊤ z + z ⊤ A θ z . The endpoint of the Gibbs block satisfies
W B − μ θ = J θ z + ϵ θ , ϵ θ ∼ N ( 0 , Ω θ ) , W_B-\mu_\theta=J_\theta z+\epsilon_\theta,\qquad\epsilon_\theta\sim\mathcal{N}(0,\Omega_\theta), W B − μ θ = J θ z + ϵ θ , ϵ θ ∼ N ( 0 , Ω θ ) ,
where J θ = G θ B J_\theta= G_\theta^B J θ = G θ B and G θ G_\theta G θ is the affine transition of conditional Gaussian. Since the transition maps quadratic functions to quadratic functions, we define χ θ ( w ) = β θ ⊤ z + z ⊤ S θ z \chi_\theta(w) = \beta_\theta^\top z + z^\top S_\theta z χ θ ( w ) = β θ ⊤ z + z ⊤ S θ z satisfying the Poisson equation
χ θ − Π θ χ θ = B θ . (21) \chi_\theta- \Pi_\theta\chi_\theta= B_\theta. \tag*{(21)} χ θ − Π θ χ θ = B θ . (21)
Matching the linear and quadratic coefficients gives
( I − J θ ) ⊤ β θ = b θ , S θ − J θ ⊤ S θ J θ = A θ . (I - J_\theta)^\top\beta_\theta= b_\theta,\qquad S_\theta- J_\theta^\top S_\theta J_\theta= A_\theta. ( I − J θ ) ⊤ β θ = b θ , S θ − J θ ⊤ S θ J θ = A θ .
Substituting w = μ θ + t v w = \mu_\theta+ tv w = μ θ + t v into the drift inequality in Assumption A2, dividing by t 4 t^4 t 4 , and letting t → ∞ t \to\infty t → ∞ give ∥ J θ ∥ o p ≤ κ K 1 / 4 < 1 \lVert J_\theta\rVert_{\mathrm{op}} \le\kappa_K^{1/4} < 1 ∥ J θ ∥ op ≤ κ K 1/4 < 1 , so the two coefficient equations have unique solutions. Here, ∥ ⋅ ∥ o p \lVert\cdot\rVert_{\mathrm{op}} ∥ ⋅ ∥ op denotes the operator norm induced by the Euclidean norm. The constant term agrees because B θ B_\theta B θ is centered, implying c θ = − tr ( S θ Ω θ ) c_\theta= -\operatorname{tr}(S_\theta\Omega_\theta) c θ = − tr ( S θ Ω θ ) .
For a vector-valued function f f f , define ∥ f ∥ V = sup w ∥ f ( w ) ∥ / V ( w ) \lVert f\rVert_V = \sup_w \lVert f(w)\rVert/V(w) ∥ f ∥ V = sup w ∥ f ( w )∥ / V ( w ) . μ θ \mu_\theta μ θ , J θ J_\theta J θ , b θ b_\theta b θ , and A θ A_\theta A θ are Lipschitz in θ \theta θ on the localization compact. Since sup θ ∥ J θ ∥ o p < 1 \sup_\theta\lVert J_\theta\rVert_{\mathrm{op}} < 1 sup θ ∥ J θ ∥ op < 1 , β θ \beta_\theta β θ and S θ S_\theta S θ are uniformly bounded and Lipschitz. Since V V V dominates quadratic growth, there is a uniform C χ < ∞ C_\chi< \infty C χ < ∞ such that
∥ χ θ ∥ V + ∥ Π θ χ θ ∥ V ≤ C χ , (22) \lVert\chi_\theta\rVert_V + \lVert\Pi_\theta\chi_\theta\rVert_V \le C_\chi, \tag*{(22)} ∥ χ θ ∥ V + ∥ Π θ χ θ ∥ V ≤ C χ , (22)
∥ χ θ − χ ϑ ∥ V ≤ C χ ∥ θ − ϑ ∥ . (23) \lVert\chi_\theta- \chi_\vartheta\rVert_V \le C_\chi\lVert\theta-\vartheta\rVert. \tag*{(23)} ∥ χ θ − χ ϑ ∥ V ≤ C χ ∥ θ − ϑ ∥ . (23)
The vector-valued result follows by applying this construction component-wise.
Let χ m = χ θ ( m − 1 ) \chi_m = \chi_{\theta^{(m-1)}} χ m = χ θ ( m − 1 ) ,
ε m = D m + χ m ( W m , B ) − Π θ ( m − 1 ) χ m ( W m − 1 , B ) ⏟ M m + χ m ( W m − 1 , B ) − χ m ( W m , B ) ⏟ C m , (24) \varepsilon_m = \underbrace{D_m + \chi_m(W_{m,B}) - \Pi_{\theta^{(m-1)}}\chi_m(W_{m-1,B})}_{M_m} + \underbrace{\chi_m(W_{m-1,B}) - \chi_m(W_{m,B})}_{C_m}, \tag*{(24)} ε m = M m D m + χ m ( W m , B ) − Π θ ( m − 1 ) χ m ( W m − 1 , B ) + C m χ m ( W m − 1 , B ) − χ m ( W m , B ) , (24)
where D m = U ‾ m − E [ U ‾ m ∣ F m − 1 ] D_m = \overline{U}_m - \mathbb{E}[\overline{U}_m\mid\mathcal{F}_{m-1}] D m = U m − E [ U m ∣ F m − 1 ] . The first term M m M_m M m is a martingale difference. The uniform bound sup m E ∥ U ‾ m ∥ 2 < ∞ \sup_m\mathbb{E}\lVert\overline{U}_m\rVert^2 < \infty sup m E ∥ U m ∥ 2 < ∞ , together with (S12), gives it a uniformly bounded second moment. Hence ∑ m γ m M m \sum_m\gamma_mM_m ∑ m γ m M m converges almost surely by the martingale convergence theorem. The same series also converges after multiplying a bounded predictable matrix A ( u ( m − 1 ) ) A(u^{(m-1)}) A ( u ( m − 1 ) ) .
To handle C m C_m C m , we let B m = γ m A ( u ( m − 1 ) ) B_m=\gamma_m A(u^{(m-1)}) B m = γ m A ( u ( m − 1 ) ) . Summation by parts yields
∑ m = 1 N B m C m = B 1 χ 1 ( W 0 , B ) − B N χ N ( W N , B ) + ∑ m = 1 N − 1 { B m + 1 χ m + 1 ( W m , B ) − B m χ m ( W m , B ) } . (25) \sum_{m=1}^{N} B_m C_m = B_1\chi_1(W_{0,B}) - B_N\chi_N(W_{N,B}) + \sum_{m=1}^{N-1}\{B_{m+1}\chi_{m+1}(W_{m,B})-B_m\chi_m(W_{m,B})\}. \tag*{(25)} m = 1 ∑ N B m C m = B 1 χ 1 ( W 0 , B ) − B N χ N ( W N , B ) + m = 1 ∑ N − 1 { B m + 1 χ m + 1 ( W m , B ) − B m χ m ( W m , B )} . (25)
B N χ N ( W N , B ) → 0 B_N\chi_N(W_{N,B})\to0 B N χ N ( W N , B ) → 0 since B N B_N B N is bounded and by (S12), ∥ B N χ N ( W N , B ) ∥ ≤ C γ N V ( W N , B ) \lVert B_N\chi_N(W_{N,B})\rVert\le C\gamma_N V(W_{N,B}) ∥ B N χ N ( W N , B )∥ ≤ C γ N V ( W N , B ) , and ∑ N γ N 2 E V 2 ( W N , B ) < ∞ \sum_N\gamma_N^2\mathbb{E}V^2(W_{N,B})<\infty ∑ N γ N 2 E V 2 ( W N , B ) < ∞ , then Borel–Cantelli applies.
For the internal difference in (S15), boundedness and Lipschitz continuity of A A A , together with (S13), give
∥ B m + 1 χ m + 1 ( W m , B ) − B m χ m ( W m , B ) ∥ ≤ C [ ∣ γ m + 1 − γ m ∣ + γ m Δ m + γ m ∥ θ ( m ) − θ ( m − 1 ) ∥ ] V ( W m , B ) . \begin{aligned}
\lVert B_{m+1}\chi_{m+1}(W_{m,B})-B_m\chi_m(W_{m,B})\rVert
&\le C\left[|\gamma_{m+1}-\gamma_m|+\gamma_m\Delta_m+\gamma_m\left\lVert\theta^{(m)}-\theta^{(m-1)}\right\rVert\right]V(W_{m,B}).
\end{aligned} ∥ B m + 1 χ m + 1 ( W m , B ) − B m χ m ( W m , B )∥ ≤ C [ ∣ γ m + 1 − γ m ∣ + γ m Δ m + γ m θ ( m ) − θ ( m − 1 ) ] V ( W m , B ) .
The first series converges since γ m \gamma_m γ m is nonincreasing and E V ( W m , B ) \mathbb{E}V(W_{m,B}) E V ( W m , B ) is bounded,
E ∑ m ∣ γ m + 1 − γ m ∣ V ( W m , B ) ≤ sup m E V ( W m , B ) ∑ m ( γ m − γ m + 1 ) = γ 1 sup m E V ( W m , B ) < ∞ . \mathbb{E}\sum_m|\gamma_{m+1}-\gamma_m|V(W_{m,B})\le\sup_m\mathbb{E}V(W_{m,B})\sum_m(\gamma_m-\gamma_{m+1})
=\gamma_1\sup_m\mathbb{E}V(W_{m,B})<\infty. E m ∑ ∣ γ m + 1 − γ m ∣ V ( W m , B ) ≤ m sup E V ( W m , B ) m ∑ ( γ m − γ m + 1 ) = γ 1 m sup E V ( W m , B ) < ∞.
The second series converges by applying Cauchy–Schwarz and the result of Lemma 1.
E ∑ m γ m Δ m V ( W m , B ) ≤ ∑ m γ m ( E Δ m 2 ) 1 / 2 ( E V 2 ( W m , B ) ) 1 / 2 ≤ C ∑ m γ m 2 < ∞ . \mathbb{E}\sum_m\gamma_m\Delta_mV(W_{m,B})\le\sum_m\gamma_m(\mathbb{E}\Delta_m^2)^{1/2}(\mathbb{E}V^2(W_{m,B}))^{1/2}
\le C\sum_m\gamma_m^2<\infty. E m ∑ γ m Δ m V ( W m , B ) ≤ m ∑ γ m ( E Δ m 2 ) 1/2 ( E V 2 ( W m , B ) ) 1/2 ≤ C m ∑ γ m 2 < ∞.
The third series is finite almost surely by (S7). Hence
∑ m ∥ B m + 1 χ m + 1 ( W m , B ) − B m χ m ( W m , B ) ∥ < ∞ . \sum_m\lVert B_{m+1}\chi_{m+1}(W_{m,B})-B_m\chi_m(W_{m,B})\rVert<\infty. m ∑ ∥ B m + 1 χ m + 1 ( W m , B ) − B m χ m ( W m , B )∥ < ∞.
Together with B N χ N ( W N , B ) → 0 B_N\chi_N(W_{N,B})\to0 B N χ N ( W N , B ) → 0 , we have ∑ m B m C m \sum_m B_mC_m ∑ m B m C m converges almost surely, which proves (S8) and (S10).
Finally, Theorem 1 and the boundedness of u ˉ \bar{u} u ˉ on the localization compact give
sup m E ∥ ε m ∥ 2 ≤ 2 sup m E ∥ U ‾ m ∥ 2 + 2 sup θ ∈ K ∥ u ˉ ( θ ) ∥ 2 < ∞ . \sup_m\mathbb{E}\lVert\varepsilon_m\rVert^2\le2\sup_m\mathbb{E}\lVert\overline{U}_m\rVert^2+2\sup_{\theta\in K}\lVert\bar{u}(\theta)\rVert^2<\infty. m sup E ∥ ε m ∥ 2 ≤ 2 m sup E ∥ U m ∥ 2 + 2 θ ∈ K sup ∥ u ˉ ( θ ) ∥ 2 < ∞.
Consequently,
E [ ∑ m γ m 2 ∥ ε m ∥ 2 ] = ∑ m γ m 2 E ∥ ε m ∥ 2 ≤ C ∑ m γ m 2 < ∞ , \mathbb{E}\left[\sum_m \gamma_m^2 \lVert\varepsilon_m\rVert^2\right] = \sum_m \gamma_m^2 \mathbb{E}\lVert\varepsilon_m\rVert^2 \le C\sum_m \gamma_m^2 < \infty, E [ m ∑ γ m 2 ∥ ε m ∥ 2 ] = m ∑ γ m 2 E ∥ ε m ∥ 2 ≤ C m ∑ γ m 2 < ∞ ,
so, ∑ m γ m 2 ∥ ε m ∥ 2 < ∞ \sum_m \gamma_m^2 \lVert\varepsilon_m\rVert^2 < \infty ∑ m γ m 2 ∥ ε m ∥ 2 < ∞ almost surely, proving (S9).
Lemma 3 (Lyapunov calculation). On the stabilized stratum,
⟨ ∇ V P ( u ) , h ( u ) ⟩ ≤ 0 , (26) \langle\nabla V_P(u),h(u)\rangle\le0, \tag*{(26)} ⟨ ∇ V P ( u ) , h ( u )⟩ ≤ 0 , (26) with equality if and only if θ ^ ( u ) ∈ Θ g , P ⋆ \widehat{\theta}(u) \in\Theta^\star_{g,P} θ ( u ) ∈ Θ g , P ⋆ .
Proof. Work in the free coordinates F F F . On the identified stratum, every inactive coordinate of θ ^ ( u ) \widehat{\theta}(u) θ ( u ) is identically zero, so the composite V P = − ℓ g , P ∘ θ ^ V_P=-\ell_{g,P}\circ\widehat{\theta} V P = − ℓ g , P ∘ θ is differentiated only through the free coordinates. Let B ( u ) = D u θ ^ F ( u ) B(u)=D_u\widehat{\theta}_F(u) B ( u ) = D u θ F ( u ) , G ( u ) = D θ F ϕ ( θ ^ ( u ) ) G(u)=D_{\theta_F}\phi(\widehat{\theta}(u)) G ( u ) = D θ F ϕ ( θ ( u )) . The fixed-point condition is
− ∇ θ F ψ ( θ ^ ) + G ( u ) T u − ∇ θ F P ( θ ^ ) = 0. (27) -\nabla_{\theta_F}\psi(\widehat{\theta})+G(u)^{\mathsf{T}}u-\nabla_{\theta_F}P(\widehat{\theta})=0. \tag*{(27)} − ∇ θ F ψ ( θ ) + G ( u ) T u − ∇ θ F P ( θ ) = 0. (27)
Differentiate in u u u ,
H ( u ) B ( u ) + G ( u ) T = 0. (28) \mathcal{H}(u)B(u)+G(u)^{\mathsf{T}}=0. \tag*{(28)} H ( u ) B ( u ) + G ( u ) T = 0. (28)
The Fisher identity and (S17) give
∇ θ F ℓ g , P ( θ ^ ) = G ( u ) T h ( u ) . (29) \nabla_{\theta_F}\ell_{g,P}(\widehat{\theta})=G(u)^{\mathsf{T}}h(u). \tag*{(29)} ∇ θ F ℓ g , P ( θ ) = G ( u ) T h ( u ) . (29)
Consequently, ∇ V P ( u ) = − B T ∇ θ F ℓ g , P ( θ ^ ) = B T H B h \nabla V_P(u)=-B^{\mathsf{T}}\nabla_{\theta_F}\ell_{g,P}(\widehat{\theta})=B^{\mathsf{T}}\mathcal{H}Bh ∇ V P ( u ) = − B T ∇ θ F ℓ g , P ( θ ) = B T H B h , and hence
⟨ ∇ V P ( u ) , h ( u ) ⟩ = ( B h ) T H ( B h ) ≤ 0. \langle\nabla V_P(u),h(u)\rangle=(Bh)^{\mathsf{T}}\mathcal{H}(Bh)\le0. ⟨ ∇ V P ( u ) , h ( u )⟩ = ( B h ) T H ( B h ) ≤ 0.
As H \mathcal{H} H is symmetric and negative semidefinite, equality holds if and only if H B h = 0 \mathcal{H}Bh=0 H B h = 0 . By (S18), this is equivalent to G T h = 0 G^{\mathsf{T}}h=0 G T h = 0 , and hence ∇ θ F ℓ g , P ( θ ^ ) = 0 \nabla_{\theta_F}\ell_{g,P}(\widehat{\theta})=0 ∇ θ F ℓ g , P ( θ ) = 0 by (S19). The inactive-coordinate condition in Assumption A7 then gives full Clarke stationarity. Conversely, full Clarke stationarity gives G T h = 0 G^{\mathsf{T}}h=0 G T h = 0 , hence H B h = 0 \mathcal{H}Bh=0 H B h = 0 , and therefore equality.
Final proof
Proof of Proposition 2. Since ∇ V P T \nabla V_P^{\mathsf{T}} ∇ V P T is bounded and Lipschitz on K ζ , L K_{\zeta,L} K ζ , L , let A ζ A_\zeta A ζ be a bounded, globally Lipschitz extension that agrees with ∇ V P T \nabla V_P^{\mathsf{T}} ∇ V P T on K ζ , L K_{\zeta,L} K ζ , L . Apply Theorems 1 and 2 to every deterministic localization and stratum. Since there are countably many localizations and finitely many strata, the resulting estimates hold on a common probability-one event.
Fix a realization in this event. By Assumption A4, some localization has τ = ∞ \tau=\infty τ = ∞ , and by Assumption A6, the trajectory eventually remains in some stratum ζ \zeta ζ . For such localization, the original and localized process agree at every iteration.
On the stabilized tail, write the recursion as (S5). By Theorem 1,
∑ m γ m ∥ R m ∥ ≤ C u ˉ ∑ m γ m ϵ m − 1 τ < ∞ . \sum_m \gamma_m \lVert R_m\rVert\le C_{\bar u}\sum_m \gamma_m\epsilon_{m-1}^{\tau}<\infty. m ∑ γ m ∥ R m ∥ ≤ C u ˉ m ∑ γ m ϵ m − 1 τ < ∞.
By Theorem 2 , ∑ m γ m ϵ m \sum_m\gamma_m\epsilon_m ∑ m γ m ϵ m converges, ∑ m γ m 2 ∥ ϵ m ∥ 2 < ∞ \sum_m\gamma_m^2\lVert\epsilon_m\rVert^2<\infty ∑ m γ m 2 ∥ ϵ m ∥ 2 < ∞ . Applying the weighted conclusion of that lemma to A ζ A_\zeta A ζ , which equals ∇ V P T \nabla V_P^{\mathsf{T}} ∇ V P T on K ζ , L K_{\zeta,L} K ζ , L , gives ∑ m γ m ∇ V P ( u ( m − 1 ) ) T ϵ m \sum_m\gamma_m\nabla V_P(u^{(m-1)})^{\mathsf{T}}\epsilon_m ∑ m γ m ∇ V P ( u ( m − 1 ) ) T ϵ m converges.
By Theorem 1, ∥ u ( m ) − u ( m − 1 ) ∥ → 0 \lVert u^{(m)}-u^{(m-1)}\rVert\to0 ∥ u ( m ) − u ( m − 1 ) ∥ → 0 , so the Taylor-expansion segments eventually remain in the region where V P V_P V P is C 2 C^2 C 2 . Define q ( u ) = − ⟨ ∇ V P ( u ) , h ( u ) ⟩ q(u)=-\langle\nabla V_P(u),h(u)\rangle q ( u ) = − ⟨ ∇ V P ( u ) , h ( u )⟩ . By Theorem 3, q ≥ 0 q\ge0 q ≥ 0 , and its zero set on K t a i l , L K_{\mathrm{tail},L} K tail , L is Z L Z_L Z L . Taylor expansion of V P V_P V P along (S5) gives
V P ( u ( m ) ) − V P ( u ( m − 1 ) ) = − γ m q ( u ( m − 1 ) ) + γ m ∇ V P ( u ( m − 1 ) ) T ϵ m + γ m ∇ V P ( u ( m − 1 ) ) T R m + O ( γ m 2 { 1 + ∥ ϵ m ∥ 2 + ∥ R m ∥ 2 } ) . \begin{aligned}
V_P(u^{(m)})-V_P(u^{(m-1)})={}&-\gamma_mq(u^{(m-1)})\\
&+\gamma_m\nabla V_P(u^{(m-1)})^{\mathsf{T}}\epsilon_m+\gamma_m\nabla V_P(u^{(m-1)})^{\mathsf{T}}R_m\\
&+O\left(\gamma_m^2\{1+\lVert\epsilon_m\rVert^2+\lVert R_m\rVert^2\}\right).
\end{aligned} V P ( u ( m ) ) − V P ( u ( m − 1 ) ) = − γ m q ( u ( m − 1 ) ) + γ m ∇ V P ( u ( m − 1 ) ) T ϵ m + γ m ∇ V P ( u ( m − 1 ) ) T R m + O ( γ m 2 { 1 + ∥ ϵ m ∥ 2 + ∥ R m ∥ 2 } ) .
The preceding bounds make every perturbation series on the right convergent. Since V P V_P V P is bounded on the compact trajectory, ∑ m γ m q ( u ( m − 1 ) ) < ∞ \sum_m\gamma_mq(u^{(m-1)})<\infty ∑ m γ m q ( u ( m − 1 ) ) < ∞ . This, together with ∑ m γ m = ∞ \sum_m\gamma_m=\infty ∑ m γ m = ∞ implies lim inf m q ( u ( m ) ) = 0 \liminf_mq(u^{(m)})=0 lim inf m q ( u ( m ) ) = 0 . If q ( u ( m ) ) q(u^{(m)}) q ( u ( m ) ) did not converge to zero, there would be infinitely many crossings between two fixed positive levels of q q q , each contributing a fixed positive amount to the sum by uniform continuity of q q q , boundedness of h h h , and the preceding noise bounds, contradicting its finiteness. Hence q ( u ( m ) ) → 0 q(u^{(m)}) \to0 q ( u ( m ) ) → 0 . By compactness and continuity, every limit point belongs to Z L \mathcal{Z}_L Z L , so Z L ≠ ∅ \mathcal{Z}_L \ne\varnothing Z L = ∅ and dist ( u ( m ) , Z L ) → 0 \operatorname{dist}(u^{(m)},\mathcal{Z}_L) \to0 dist ( u ( m ) , Z L ) → 0 .
For every u ⋆ ∈ Z L u_\star\in\mathcal{Z}_L u ⋆ ∈ Z L , θ ^ ( u ⋆ ) ∈ Θ g , P ⋆ \hat{\theta}(u_\star) \in\Theta^\star_{g,P} θ ^ ( u ⋆ ) ∈ Θ g , P ⋆ . The Lipschitz property of θ ^ \hat{\theta} θ ^ gives
dist ( θ ^ ( u ( m ) ) , Θ g , P ⋆ ) ≤ C θ ^ dist ( u ( m ) , Z L ) → 0 , \operatorname{dist}\left(\hat{\theta}(u^{(m)}),\Theta^\star_{g,P}\right) \le C_{\hat{\theta}}\operatorname{dist}(u^{(m)},\mathcal{Z}_L) \to0, dist ( θ ^ ( u ( m ) ) , Θ g , P ⋆ ) ≤ C θ ^ dist ( u ( m ) , Z L ) → 0 ,
proving (7) . Finally,
dist ( θ ( m ) , Θ g , P ⋆ ) ≤ ϵ m τ + dist ( θ ^ ( u ( m ) ) , Θ g , P ⋆ ) → 0 , \operatorname{dist}\left(\theta^{(m)},\Theta^\star_{g,P}\right) \le\epsilon_m^\tau+\operatorname{dist}\left(\hat{\theta}(u^{(m)}),\Theta^\star_{g,P}\right) \to0, dist ( θ ( m ) , Θ g , P ⋆ ) ≤ ϵ m τ + dist ( θ ^ ( u ( m ) ) , Θ g , P ⋆ ) → 0 ,
by Theorem 1, proving (8) . Since the preceding choices exist on a probability-one event, the proposition follows. □
Proof of prop3
Additional assumptions
Assumption A8 (Uniform-in-L L L random Fourier features (RFF) regularity near K ⋆ K^\star K ⋆ ). All values of L L L use the first L L L terms of one infinite i.i.d. sequence { ξ ℓ } ℓ ≥ 1 \{\xi_\ell\}_{\ell\ge1} { ξ ℓ } ℓ ≥ 1 , ξ ℓ = ( X ℓ , ψ ℓ ) \xi_\ell=(X_\ell,\psi_\ell) ξ ℓ = ( X ℓ , ψ ℓ ) . There are deterministic compact convex sets K ⋆ K^\star K ⋆ and K + K^+ K + satisfying K ⋆ ⊆ int ( K + ) K^\star\subseteq\operatorname{int}(K^+) K ⋆ ⊆ int ( K + ) and K + ⊆ int ( Θ ) K^+ \subseteq\operatorname{int}(\Theta) K + ⊆ int ( Θ ) , where int ( S ) \operatorname{int}(S) int ( S ) denotes the interior of S S S , with the following properties.
(1) The spectral matrix is real, symmetric, and even by construction. For X ∼ g X \sim g X ∼ g , the map θ ↦ f ( X ; θ ) \theta\mapsto f(X;\theta) θ ↦ f ( X ; θ ) is C 2 C^2 C 2 on a neighborhood of K + K^+ K + . The penalty is the same fixed function P P P , with the same coefficient λ > 0 \lambda>0 λ > 0 , for every L L L and for the exact target-GP likelihood.
(2) Let K i j ( ( x , ψ ) ; θ ) = 2 cos ( s i ⊤ x + ψ ) cos ( s j ⊤ x + ψ ) g ( x ) f ( x ; θ ) \mathcal{K}_{ij}((x,\psi);\theta)=\frac{2\cos(s_i^\top x+\psi)\cos(s_j^\top x+\psi)}{g(x)}f(x;\theta) K ij (( x , ψ ) ; θ ) = g ( x ) 2 c o s ( s i ⊤ x + ψ ) c o s ( s j ⊤ x + ψ ) f ( x ; θ ) . There is a nonnegative random variable M ( ξ 1 ) M(\xi_1) M ( ξ 1 ) , with E M ( ξ 1 ) < ∞ \mathbb{E}M(\xi_1)<\infty E M ( ξ 1 ) < ∞ , such that for every i , j i,j i , j ,
sup θ ∈ K + { ∥ K i j ( ξ 1 ; θ ) ∥ + ∥ ∇ θ K i j ( ξ 1 ; θ ) ∥ + ∥ ∇ θ 2 K i j ( ξ 1 ; θ ) ∥ } ≤ M ( ξ 1 ) , \sup_{\theta\in K^+}\left\{\lVert\mathcal{K}_{ij}(\xi_1;\theta)\rVert+\lVert\nabla_\theta\mathcal{K}_{ij}(\xi_1;\theta)\rVert+\lVert\nabla_\theta^2\mathcal{K}_{ij}(\xi_1;\theta)\rVert\right\}\le M(\xi_1), θ ∈ K + sup { ∥ K ij ( ξ 1 ; θ )∥ + ∥ ∇ θ K ij ( ξ 1 ; θ )∥ + ∥ ∇ θ 2 K ij ( ξ 1 ; θ )∥ } ≤ M ( ξ 1 ) , almost surely.
(3) The nugget map D ( θ ) D(\theta) D ( θ ) is C 1 C^{1} C 1 on a neighborhood of K ⋆ K^\star K ⋆ , and there exists d 0 > 0 d_0 > 0 d 0 > 0 such that
D ( θ ) ⪰ d 0 I n p , θ ∈ K ⋆ . D(\theta) \succeq d_0 I_{np}, \qquad\theta\in K^\star. D ( θ ) ⪰ d 0 I n p , θ ∈ K ⋆ .
Assumption A9 (Uniform-in-L L L containment of stationary sets). For the deterministic compact set K ⋆ K^\star K ⋆ in Assumption A8,
∅ ≠ Θ P ⋆ ⊂ K ⋆ , \varnothing\ne\Theta^\star_P \subset K^\star, ∅ = Θ P ⋆ ⊂ K ⋆ ,
and
P { ∃ L 0 < ∞ such that ∅ ≠ Θ g , L , P ⋆ ⊂ K ⋆ for every L ≥ L 0 } = 1. \mathbb{P}\left\{\exists L_0 < \infty\text{ such that } \varnothing\ne\Theta^\star_{g,L,P} \subset K^\star\text{ for every } L \ge L_0\right\} = 1. P { ∃ L 0 < ∞ such that ∅ = Θ g , L , P ⋆ ⊂ K ⋆ for every L ≥ L 0 } = 1.
Thus K ⋆ K^\star K ⋆ contains the exact stationary set and, almost surely, all sufficiently large-L L L RFF stationary sets.
Lemmas
Lemma 4 (Uniform convergence of RFF scores). Under Assumption A8, almost surely,
sup θ ∈ K ⋆ ∥ ∇ θ ℓ g , L ( θ ) − ∇ θ ℓ ( θ ) ∥ ⟶ 0. (30) \sup_{\theta\in K^\star} \left\|\nabla_\theta\ell_{g,L}(\theta) - \nabla_\theta\ell(\theta)\right\| \longrightarrow0. \tag*{(30)} θ ∈ K ⋆ sup ∥ ∇ θ ℓ g , L ( θ ) − ∇ θ ℓ ( θ ) ∥ ⟶ 0. (30)
Proof. We first establish the uniform law used below. Let a ( z , θ ) a(z,\theta) a ( z , θ ) be a finite-dimensional vector or matrix that is continuously differentiable in θ \theta θ on a neighborhood of K ⋆ K^\star K ⋆ , and suppose
sup θ ∈ K ⋆ { ∥ a ( ξ 1 , θ ) ∥ + ∥ ∇ θ a ( ξ 1 , θ ) ∥ } ≤ M ( ξ 1 ) , E M ( ξ 1 ) < ∞ . \sup_{\theta\in K^\star} \left\{\|a(\xi_1,\theta)\| + \|\nabla_\theta a(\xi_1,\theta)\|\right\} \le M(\xi_1), \qquad\mathbb{E}M(\xi_1) < \infty. θ ∈ K ⋆ sup { ∥ a ( ξ 1 , θ ) ∥ + ∥ ∇ θ a ( ξ 1 , θ ) ∥ } ≤ M ( ξ 1 ) , E M ( ξ 1 ) < ∞.
Then,
sup θ ∈ K ⋆ ∥ 1 L ∑ ℓ = 1 L a ( ξ ℓ , θ ) − E a ( ξ 1 , θ ) ∥ ⟶ 0 , almost surely . (31) \sup_{\theta\in K^\star} \left\| \frac{1}{L}\sum_{\ell=1}^{L} a(\xi_\ell,\theta) - \mathbb{E}a(\xi_1,\theta)\right\| \longrightarrow0, \qquad\text{almost surely}. \tag*{(31)} θ ∈ K ⋆ sup L 1 ℓ = 1 ∑ L a ( ξ ℓ , θ ) − E a ( ξ 1 , θ ) ⟶ 0 , almost surely . (31)
For a finite δ \delta δ -net { θ 1 , … , θ N } \{\theta_1,\ldots,\theta_N\} { θ 1 , … , θ N } of K ⋆ K^\star K ⋆ , the mean-value theorem and the triangle inequality give:
sup θ ∈ K ⋆ ∥ 1 L ∑ ℓ = 1 L a ( ξ ℓ , θ ) − E a ( ξ 1 , θ ) ∥ ≤ max 1 ≤ r ≤ N ∥ 1 L ∑ ℓ = 1 L a ( ξ ℓ , θ r ) − E a ( ξ 1 , θ r ) ∥ + δ { 1 L ∑ ℓ = 1 L M ( ξ ℓ ) + E M ( ξ 1 ) } . (32) \begin{aligned}
\sup_{\theta\in K^\star} \left\| \frac{1}{L}\sum_{\ell=1}^{L} a(\xi_\ell,\theta) - \mathbb{E}a(\xi_1,\theta)\right\|
&\le\max_{1\le r\le N}\left\| \frac{1}{L}\sum_{\ell=1}^{L} a(\xi_\ell,\theta_r) - \mathbb{E}a(\xi_1,\theta_r)\right\| \\
&\quad+ \delta\left\{\frac{1}{L}\sum_{\ell=1}^{L}M(\xi_\ell)+\mathbb{E}M(\xi_1)\right\}. \tag*{(32)}
\end{aligned} θ ∈ K ⋆ sup L 1 ℓ = 1 ∑ L a ( ξ ℓ , θ ) − E a ( ξ 1 , θ ) ≤ 1 ≤ r ≤ N max L 1 ℓ = 1 ∑ L a ( ξ ℓ , θ r ) − E a ( ξ 1 , θ r ) + δ { L 1 ℓ = 1 ∑ L M ( ξ ℓ ) + E M ( ξ 1 ) } . (32)
The strong law of large numbers, followed by δ ↓ 0 \delta\downarrow0 δ ↓ 0 , proves (S21).
Applying (S21) to K i j \mathcal{K}_{ij} K ij and its first derivatives gives uniform convergence of the covariance matrices and their first derivatives. The nugget lower bound keeps ( Σ L + D ) ( θ ) (\boldsymbol{\Sigma}_L+\boldsymbol{D})(\boldsymbol{\theta}) ( Σ L + D ) ( θ ) and ( Σ + D ) ( θ ) (\boldsymbol{\Sigma}+\boldsymbol{D})(\boldsymbol{\theta}) ( Σ + D ) ( θ ) positive definite on K ⋆ K^\star K ⋆ . Because the matrix inversion and the log-determinant are uniformly continuous, the uniform convergence of the covariance matrices and their derivatives implies uniform convergence of the Gaussian log-likelihoods and their scores.
Lemma 5 (Outer convergence of stationary sets). Under Assumptions A 8 A8 A 8 and A 9 A9 A 9 and the fixed common-penalty convention above,
Δ L : = { sup θ ∈ Θ g , L , P ⋆ dist ( θ , Θ P ⋆ ) , Θ g , L , P ⋆ ≠ ∅ , 0 , Θ g , L , P ⋆ = ∅ , ⟶ 0 almost surely . (33) \Delta_L := \begin{cases}
\sup_{\boldsymbol{\theta}\in\Theta^\star_{g,L,P}} \operatorname{dist}(\boldsymbol{\theta},\Theta^\star_P), & \Theta^\star_{g,L,P}\ne\varnothing,\\
0, & \Theta^\star_{g,L,P}=\varnothing,
\end{cases}
\longrightarrow0 \quad\textit{almost surely}. \tag*{(33)} Δ L := { sup θ ∈ Θ g , L , P ⋆ dist ( θ , Θ P ⋆ ) , 0 , Θ g , L , P ⋆ = ∅ , Θ g , L , P ⋆ = ∅ , ⟶ 0 almost surely . (33) Proof. Focus on a probability-one event on which (S20) and Assumption A 9 A9 A 9 hold. If (S22) failed, there exists ε > 0 \varepsilon>0 ε > 0 , L j → ∞ L_j\to\infty L j → ∞ , and θ j ∈ Θ g , L j , P ⋆ \boldsymbol{\theta}_j\in\Theta^\star_{g,L_j,P} θ j ∈ Θ g , L j , P ⋆ such that dist ( θ j , Θ P ⋆ ) ≥ ε \operatorname{dist}(\boldsymbol{\theta}_j,\Theta^\star_P)\ge\varepsilon dist ( θ j , Θ P ⋆ ) ≥ ε . By compactness, there exists a subsequence θ j k \boldsymbol{\theta}_{j_k} θ j k converging to θ ∞ ∈ K ⋆ \boldsymbol{\theta}_\infty\in K^\star θ ∞ ∈ K ⋆ . We relabel ( L j k , θ j k ) (L_{j_k},\boldsymbol{\theta}_{j_k}) ( L j k , θ j k ) as ( L j , θ j ) (L_j,\boldsymbol{\theta}_j) ( L j , θ j ) . The stationary condition states that there are some z j ∈ ∂ P ( θ j ) \boldsymbol{z}_j\in\partial P(\boldsymbol{\theta}_j) z j ∈ ∂ P ( θ j ) , so z j = ∇ ℓ g , L j ( θ j ) \boldsymbol{z}_j=\nabla\ell_{g,L_j}(\boldsymbol{\theta}_j) z j = ∇ ℓ g , L j ( θ j ) . Uniform score convergence and continuity of ∇ ℓ \nabla\ell ∇ ℓ imply z j → ∇ ℓ ( θ ∞ ) \boldsymbol{z}_j\to\nabla\ell(\boldsymbol{\theta}_\infty) z j → ∇ ℓ ( θ ∞ ) . Since P P P is continuous and convex, passing to the limit in z j ∈ ∂ P ( θ j ) \boldsymbol{z}_j\in\partial P(\boldsymbol{\theta}_j) z j ∈ ∂ P ( θ j ) gives ∇ ℓ ( θ ∞ ) ∈ ∂ P ( θ ∞ ) \nabla\ell(\boldsymbol{\theta}_\infty)\in\partial P(\boldsymbol{\theta}_\infty) ∇ ℓ ( θ ∞ ) ∈ ∂ P ( θ ∞ ) . Therefore, θ ∞ ∈ Θ P ⋆ \boldsymbol{\theta}_\infty\in\Theta^\star_P θ ∞ ∈ Θ P ⋆ contradicts dist ( θ j , Θ P ⋆ ) ≥ ε \operatorname{dist}(\boldsymbol{\theta}_j,\Theta^\star_P)\ge\varepsilon dist ( θ j , Θ P ⋆ ) ≥ ε , and proves the lemma.
Final proof
Proof. For each L L L , let G L G_L G L be the random-feature event on which Assumptions A 1 A1 A 1 to A 7 A7 A 7 hold. By hypothesis, almost surely G L G_L G L holds for all sufficiently large L L L , and on G L G_L G L , Proposition 2 ensures Θ g , L , P ⋆ ≠ ∅ \Theta^\star_{g,L,P}\ne\varnothing Θ g , L , P ⋆ = ∅ . Set D L , m = 0 D_{L,m}=0 D L , m = 0 off G L G_L G L , and on G L G_L G L let
D L , m = max { dist ( θ ^ L ( u L ( m ) ) , Θ g , L , P ⋆ ) , dist ( θ L ( m ) , Θ g , L , P ⋆ ) } . D_{L,m}=\max\left\{\operatorname{dist}\left(\widehat{\boldsymbol{\theta}}_L(\boldsymbol{u}^{(m)}_L),\Theta^\star_{g,L,P}\right),\operatorname{dist}\left(\boldsymbol{\theta}^{(m)}_L,\Theta^\star_{g,L,P}\right)\right\}. D L , m = max { dist ( θ L ( u L ( m ) ) , Θ g , L , P ⋆ ) , dist ( θ L ( m ) , Θ g , L , P ⋆ ) } .
By Theorem 5, Δ L → 0 \Delta_L \to0 Δ L → 0 almost surely and hence in probability. For every fixed L L L , Proposition 2 on G L G_L G L gives D L , m → 0 D_{L,m} \to0 D L , m → 0 conditionally almost surely given the random features and therefore unconditionally in probability by bounded convergence.
We construct deterministic L k → ∞ L_k \to\infty L k → ∞ and m k → ∞ m_k \to\infty m k → ∞ to combine these two limits. Choose them increasingly so that
P ( Δ L k > k − 1 ) < 2 − k − 1 , P ( D L k , m k > k − 1 ) < 2 − k − 1 . \mathbb{P}(\Delta_{L_k} > k^{-1}) < 2^{-k-1}, \qquad\mathbb{P}(D_{L_k,m_k} > k^{-1}) < 2^{-k-1}. P ( Δ L k > k − 1 ) < 2 − k − 1 , P ( D L k , m k > k − 1 ) < 2 − k − 1 .
Then ∑ k = 1 ∞ P ( Δ L k + D L k , m k > 2 k − 1 ) < ∞ \sum_{k=1}^{\infty} \mathbb{P}(\Delta_{L_k} + D_{L_k,m_k} > 2k^{-1}) < \infty ∑ k = 1 ∞ P ( Δ L k + D L k , m k > 2 k − 1 ) < ∞ . The Borel–Cantelli lemma therefore gives Δ L k + D L k , m k → 0 \Delta_{L_k} + D_{L_k,m_k} \to0 Δ L k + D L k , m k → 0 almost surely. Since G L k G_{L_k} G L k holds eventually, on it
dist ( x , Θ P ∗ ) ≤ dist ( x , Θ g , L , P ∗ ) + Δ L , \operatorname{dist}(x,\Theta_P^*) \le\operatorname{dist}(x,\Theta_{g,L,P}^*) + \Delta_L, dist ( x , Θ P ∗ ) ≤ dist ( x , Θ g , L , P ∗ ) + Δ L ,
applying the inequality with x = θ ^ L k ( u L k ( m k ) ) x=\hat{\theta}_{L_k}(u_{L_k}^{(m_k)}) x = θ ^ L k ( u L k ( m k ) ) and x = θ L k ( m k ) x=\theta_{L_k}^{(m_k)} x = θ L k ( m k ) proves (9) and (10) . □ \square □
Additional Technical Details and Simulation Results
We provide additional implementation details and numerical results in this section.
Implementation details
We first give additional details on the implementation of our simulation study. For algorithm initialization, one may simply set σ − 1 \sigma^{-1} σ − 1 to a diagonal matrix of inverse empirical variances or, when processes are colocated, use [ ( 1 / n ) ∑ i = 1 n Y ( s i ) Y ( s i ) ⊤ ] − 1 [(1/n)\sum_{i=1}^{n}\mathbf{Y}(s_i)\mathbf{Y}(s_i)^\top]^{-1} [( 1/ n ) ∑ i = 1 n Y ( s i ) Y ( s i ) ⊤ ] − 1 as a sensible but crude initialization. The spatial parameters governing { f j ( x ) } j = 1 p \{f_j(x)\}_{j=1}^{p} { f j ( x ) } j = 1 p and the nugget parameter d d d may be initialized using independent-process fits. In addition, there are a number of tuning parameters and choices in the algorithm.
Under the SAECM framework, we found that estimates were generally robust to the number of Monte Carlo samples T T T , and increasing L L L generally results in better accuracy up to a point, at the expense of computation. Therefore, we use different numbers of random features and Monte Carlo sample sizes for estimation (L = 20,000 L = 20{,}000 L = 20 , 000 , T = 30 T = 30 T = 30 ), likelihood evaluation (L l l = 1,000 L_{\mathrm{ll}} = 1{,}000 L ll = 1 , 000 ), and prediction (L p r e d = 50,000 L_{\mathrm{pred}} = 50{,}000 L pred = 50 , 000 , T p r e d = 10,000 T_{\mathrm{pred}} = 10{,}000 T pred = 10 , 000 ). The first T / 2 T/2 T /2 sweeps are discarded as burn-in. We take γ m = 200 / ( 199 + m ) \gamma_m = 200/(199 + m) γ m = 200/ ( 199 + m ) and conduct 15 replicates, each with 500 SAECM steps or until the change in the Monte Carlo complete-data penalized log-likelihood falls below 10 − 3 10^{-3} 1 0 − 3 . For choice of the proposal distribution g ( ⋅ ) g(\cdot) g ( ⋅ ) , we refer the reader to Arroyo and Emery [5 ] . Conceptually, g ( ⋅ ) g(\cdot) g ( ⋅ ) should have heavier tails than each of the diagonal elements in f ( x ) \boldsymbol{f}(x) f ( x ) , so we choose a Matérn spectral density as the proposal with a p r o p = 3 a_{\mathrm{prop}} = 3 a prop = 3 and ν p r o p = 1.25 \nu_{\mathrm{prop}} = 1.25 ν prop = 1.25 as the parameters. We consider the penalty grid { 0.0003 , 0.0006 , 0.001 , 0.003 , 0.006 , 0.012 , 0.025 , 0.05 , 0.1 } \{0.0003, 0.0006, 0.001, 0.003, 0.006, 0.012, 0.025, 0.05, 0.1\} { 0.0003 , 0.0006 , 0.001 , 0.003 , 0.006 , 0.012 , 0.025 , 0.05 , 0.1 } , together with an unpenalized dense fit. BIC uses a sample size of n = 500 n = 500 n = 500 for colocated observations and n = 300 n = 300 n = 300 for misaligned observations.
Benchmark Implementation Details
We describe the detailed steps of LMC and IOX in the following. First, the loading matrix is estimated from the sample covariance after subtracting the pilot nugget, which is estimated from the marginal variogram. Then, the largest q q q eigenvalues are retained, while the rest are set to zero. For each eigenvector, we project the data onto its direction and fit a univariate Gaussian process under each of the three covariance families. The family for each process is selected using AIC. Finally, given the truncated loading matrix and the selected family for each latent process, maximum likelihood estimation is used to estimate the spatial parameters and a shared nugget. In the misaligned setting, the loading matrix is formed by calculating the pairwise covariances between processes using overlapping locations and is then truncated to rank q q q . Because the misaligned data cannot be projected onto the eigenspace, we perform weighted least squares to regress the covariogram on the eigenvectors and estimate the correlogram of each latent process. Each covariance function is then fitted to the estimated correlogram, and the best is selected by AIC, subject to the constraint that each family is represented by at least one latent process. Once the covariance functions are determined, maximum likelihood estimation can be performed.
The IOX approach is implemented using the spiox package (version 0.8.1) from its GitHub repository. As in Peruzzi [40 ] , we run 10,000 MCMC iterations. The first 7,500 are discarded as burn-in. Starting values are obtained from spiox::autostart(). The number of Vecchia neighbors is set to 15. We construct predictive intervals from the 2.5th and 97.5th percentiles of the posterior predictive draws. The upper bound of ν \nu ν is raised to 2.9 to cover the smoothness values in our simulation settings.
Sensitivity analysis and convergence diagnostics
We provide a sensitivity analysis of L L L and a convergence diagnostic of the simulation in this section. In the sensitivity analysis, we simulate data under different n n n and p p p and compare the random-feature approximated likelihood with a reference log-likelihood based on 100,000 features for the mixed-family process. We use tridiagonal precision with negative off-diagonal entries and otherwise the same parameters as in the main-text simulations. The results are shown in Figure S2 . The y y y -axis shows the absolute difference between each approximate likelihood and its reference likelihood, divided by n p np n p , and the x x x -axis shows L L L . We compare both the marginal likelihood obtained by integrating out W W W , denoted by ℓ L , i n t \ell_{L,\mathrm{int}} ℓ L , int , and the approximate likelihood calculated using Algorithm S2, denoted by ℓ L , s i m \ell_{L,\mathrm{sim}} ℓ L , sim . The plots show similar patterns across different p p p and n n n . The absolute error of ℓ L , i n t \ell_{L,\mathrm{int}} ℓ L , int is smaller than that of ℓ L , s i m \ell_{L,\mathrm{sim}} ℓ L , sim when L L L is small. Above L = 20,000 L = 20{,}000 L = 20 , 000 , the error of ℓ L , i n t \ell_{L,\mathrm{int}} ℓ L , int levels off, showing limited improvement from increasing L L L , while the error of ℓ L , s i m \ell_{L,\mathrm{sim}} ℓ L , sim continues to decrease slightly. These results support our choice of L = 20,000 L = 20{,}000 L = 20 , 000 for estimation.
Figure S2. Mean absolute log-likelihood error divided by n p np n p for mixed-family simulations with tridiagonal precision and negative off-diagonal entries, across different values of n n n , L L L , and p p p . Results are averaged across 15 replicates. Error bars indicate ± 1 \pm1 ± 1 SD.
Figure S3 shows trace plots of the penalized observed log-likelihood based on L = 10,000 L = 10{,}000 L = 10 , 000 , the Frobenius norm of the precision matrix, the microergodic parameters, and the nugget variances for the colocated random-precision simulation in the main text. For process j j j , the CH microergodic parameter is calculated by σ j j 2 ν j Γ ( ν j + α j ) β j 2 ν j Γ ( α j ) \frac{\sigma_{jj}^{2}\nu_j\Gamma(\nu_j+\alpha_j)}{\beta_j^{2\nu_j}\Gamma(\alpha_j)} β j 2 ν j Γ ( α j ) σ j j 2 ν j Γ ( ν j + α j ) . In our analysis, the penalized observed log-likelihood and microergodic parameters continue to drift slightly through 1,000 iterations. However, predictive performance changes little between 500 and 1,000 iterations, so we report the 500-iteration results. The nugget continues to drift, although the magnitude of the drift is small. The Frobenius norm stabilizes early.
Figure S3. SAECM trace diagnostics for 15 Penalized fits in the colocated mixed-family random-precision setting. Dark blue curves show means, blue bands show ± 1 \pm1 ± 1 SD, and thin curves show individual replicates. “ll” denotes log-likelihood.
Timing results
Table S1 reports estimation times for the simulation studies. Ind is generally the fastest approach because it fits each process separately without estimating cross-covariances. LMC is relatively fast in most settings but substantially slower in the random-precision misaligned settings, where it uses more latent processes and requires a more costly likelihood calculation. The all-Matérn fits also involve more covariance parameters and require more optimization steps than the mixed-family fits, which predominantly select SE latent covariance functions. The reported SAECM and IOX times correspond to 500 and 10,000 iterations. We fit nine positive-penalty candidates and a separate Dense benchmark, with each candidate fit run in parallel. Each SAECM fit used 32 threads, while each IOX and LMC fit used 4 threads and each Ind fit was allocated 8 CPU cores. These timings reflect the computational time of the procedures used, rather than a comparison at matched estimation accuracy.
Table S1: Estimation times per fit in minutes. Entries are mean (SD) across 15 replicates. Col/Mis denote colocated/misaligned observations, and Pos/Neg denote positive/negative precision off-diagonals.
Table S1.
Selection results
For completeness, we provide additional selection results below for § 6.4 . Table S2 reports the true positive rate (TPR) and false positive rate (FPR) for selection.
Table S2. True-positive and false-positive rates for precision-matrix edge recovery in mixed-family random-precision simulations, averaged across 15 replicates. Col/Mis denote colocated/misaligned observations, and Pos/Neg denote positive/negative precision off-diagonals.
Additional results for mixed-family settings
We present additional simulation results for misaligned estimation and tridiagonal precision settings in this section. Table S3 shows the estimation results for the mixed-family covariance model in the misaligned setting. Compared with the colocated results in Table 1 , all approaches generally have larger biases, and the Ind approach provides better estimates for some parameters. Overall, our approach still provides better estimation performance.
Table S3 : Parameter estimation for mixed-family random-precision simulations with misaligned observations. Entries are bias (SD) across 15 replicates, averaged over the processes in each family. Bold indicates the smallest absolute bias.
Table S3.
In the tridiagonal setting, the diagonal entries of the precision matrices are set to 2. The off-diagonal values are set to 1 to induce alternating signs in the correlation structure, and to − 1 -1 − 1 to produce uniformly positive correlations, as illustrated in Figure S4 . The resulting variance for each process ranges from 0.98 to 12.75.
Figure S4. True correlation and precision matrices and entrywise median precision estimates across 15 replicates for mixed-family tridiagonal-precision simulations.
We use the same hyperparameters as in the random-precision setting, with the number of Monte Carlo steps T = 30 T = 30 T = 30 , and LMC q = 6 q = 6 q = 6 to account for 90% of total variance. The estimation results are shown in Figure S4 and Table S4 . Among all compared approaches, our method yields estimates with generally lower bias. Prediction results are provided in
Table S4. Parameter estimation for mixed-family tridiagonal-precision simulations with colocated observations. Entries are bias (SD) across 15 replicates, averaged over the processes in each family. Bold indicates the smallest absolute bias.
Table S5 . Both the Penalized and Dense approaches capture the true precision signals, and the Penalized fit produces sparse estimates that more closely match the true underlying structure.
Table S5. Prediction summaries for mixed-family tridiagonal-precision simulations with colocated and misaligned observations. Entries are means (SDs) across 15 replicates. Col/Mis denote colocated/misaligned observations, and Pos/Neg denote positive/negative precision off-diagonals. Bold indicates coverage closest to 95%, or the lowest mean for other metrics.
The predictive performance for the tridiagonal precision structure is similar to the results obtained for the random structure. When the data are colocated, our approach provides slightly better predictive performance for individual processes than other approaches. It also provides better coverage for the sum of processes than Ind. In the misaligned setting, our method outperforms the Ind approach for both individual processes and the sum of processes.
Multivariate Matérn Process
We present the estimation and prediction results in Table S6 and Table S7 when the data are generated from a multivariate Matérn process with a = 3 a = 3 a = 3 and ν = 1.5 \nu= 1.5 ν = 1.5 to address the gap where IOX is misspecified for the marginal process, as in the main text. Under this setting, the cross-process parameters are a i j = 3 a_{ij} = 3 a ij = 3 and ν i j = 1.5 \nu_{ij} = 1.5 ν ij = 1.5 for i ≠ j i \ne j i = j and the covariance is separable before adding the nugget. We implemented the latent model that separately models the spatial process and the nugget described in Peruzzi [40 ] since it aligns with our model formulation.
Table S6: All-Matérn random-precision simulations: mean (SD) across 15 replicates, with Bias rows averaged over processes. Col/Mis: colocated/misaligned; Pos/Neg: positive/negative precision off-diagonals. Bold: smallest absolute bias, coverage closest to 95%, or lowest mean otherwise.
Table S6.
Table S7. All-Matérn tridiagonal-precision simulations: mean (SD) across 15 replicates, with Bias rows averaged over processes. Col/Mis: colocated/misaligned; Pos/Neg: positive/negative precision off-diagonals. Bold: smallest absolute bias, coverage closest to 95%, or lowest mean otherwise.
For random precision matrices, IOX has larger absolute bias in a a a and smaller absolute bias in ν \nu ν under colocated observations, with the pattern reversed under misalignment. For tridiagonal precision matrices, Penalized has smaller absolute biases in both parameters than IOX. For random precision matrices, Penalized and IOX have comparable prediction norms and sum RMSE. For tridiagonal precision matrices, IOX achieves the lowest prediction norm and sum RMSE under most settings, and its individual-process and sum coverages are slightly above the nominal rate. The Penalized approach has near-nominal sum coverage under positive off-diagonals but undercovers under negative off-diagonals. LMC has near-nominal sum coverage under negative off-diagonals. Overall, the correctly specified Matérn comparison shows similar performance for Penalized and IOX.
Additional MERRA-2 Analysis Results
We provide parameter estimation results for MERRA-2 analysis in Table S8 . The proposal parameters a p r o p a_{\mathrm{prop}} a prop and ν p r o p \nu_{\mathrm{prop}} ν prop are the median fitted a a a and minimum fitted ν \nu ν from the independent process fits. Different numbers of random features and Monte Carlo sample sizes are used for estimation (L = 20,000 L = 20{,}000 L = 20 , 000 , T = 40 T = 40 T = 40 ), likelihood evaluation (L l l = 1,000 L_{\mathrm{ll}} = 1{,}000 L ll = 1 , 000 ), and prediction (L p r e d = 30,000 L_{\mathrm{pred}} = 30{,}000 L pred = 30 , 000 , T p r e d = 20,000 T_{\mathrm{pred}} = 20{,}000 T pred = 20 , 000 ). The first T / 2 T/2 T /2 sweeps are discarded as burn-in. In addition to an unpenalized dense fit, we consider the penalty grid { 0.0001 , 0.00025 , 0.0004 , 0.00055 , 0.0007 , 0.00085 , 0.001 , 0.00125 , 0.00175 , 0.0025 , 0.003 , 0.005 , 0.01 , 0.02 } \{0.0001, 0.00025, 0.0004, 0.00055, 0.0007, 0.00085, 0.001, 0.00125, 0.00175, 0.0025, 0.003, 0.005, 0.01, 0.02\} { 0.0001 , 0.00025 , 0.0004 , 0.00055 , 0.0007 , 0.00085 , 0.001 , 0.00125 , 0.00175 , 0.0025 , 0.003 , 0.005 , 0.01 , 0.02 } , each with 700 iterations. We take γ m = 30 / ( 29 + m ) \gamma_m = 30/(29 + m) γ m = 30/ ( 29 + m ) . BIC uses n = 500 n = 500 n = 500 for colocated observations and n = 300 n = 300 n = 300 for misaligned observations. In the misaligned setting, σ − 1 \boldsymbol{\sigma}^{-1} σ − 1 is initialized at the diagonal matrix of inverse independent-fit variances.
Table S8. Parameter estimation for MERRA-2. Entries are medians across three folds of the within-fold means over processes.