Introduction
Gaussian processes (GPs) are a non-parametric fitting tool, which makes them very useful at modelling signals we cannot describe explicitly. They have therefore seen widespread use across many areas of astronomy, such as modelling stellar activity (Aigrain et al. [3 ] ; Rajpaul et al. [97 ] ), asteroseismology and stellar granulation (Brewer and Stello 2009; Foreman-Mackey et al. [42 ] ), brown dwarf variability (Vos et al. [124 ] ), gravitational lensing (Press et al. [94 ] ), pulsar timing delays (Haasteren and Vallisneri [120 ] ; Susobhanan and van Haasteren [111 ] ), atmospheric retrievals (Rotman et al. [99 ] ), supernova light curves (Lochner et al. [80 ] ), and many more (see Aigrain and Foreman-Mackey [2 ] for a review). Outside of astronomy, they have also seen very diverse applications, including their original application for geostatistics (Krige [77 ] ), and later uses within epidemiology (Diggle et al. [33 ] ), ecology (Dovers et al. [35 ] ), genetics (Rakitsch et al. [98 ] ; Velten et al. [121 ] ), and neuroimaging (Cai et al. [20 ] ).
This work has mainly been motivated by their applications for modelling correlated noise in exoplanet transit time-series, originally introduced in Gibson et al. [48 ] . Noise which is correlated in time, often appearing as ‘bumps’ in the light curves, can be referred to as either correlated noise, red noise, or systematics, while noise which is independent in time (e.g. due to photon noise) is referred to as ‘white noise’. Correlated noise may arise from a variety of sources, typically split between instrumental sources (e.g. pointing instabilities, thermal breathing), and astrophysical sources (e.g. stellar granulation, asteroseismology and stellar activity). The key reason why we may want to joint-fit transit observations with both a deterministic transit model and a GP for modelling systematics, is that the GP can marginalise over an infinite space of possible systematics models and weight towards the best-fitting ones, which propagates the uncertainty introduced by complex noise sources into our measured transit depth uncertainties. Gibson [47 ] demonstrated using simulated data that GPs can produce robust results despite contamination from a wide variety of systematics models, and are often more robust than common alternatives, such as marginalising over a set of detrending vectors.
While Gaussian processes are typically applied to one-dimensional datasets (e.g. a time-series of flux values), there are many cases where it is useful to apply them to 2D datasets, modelling correlations across two input dimensions, such as wavelength and time. Despite being a standard approach within exoplanet atmospheres, fitting transit light curves one wavelength band at a time ignores the presence of wavelength-correlated noise, which can arise from effects such as detector readout noise or stellar granulation (e.g. Schlawin et al. [105 ] ; Gordon et al. [53 ] ). Fortune et al. [44 ] demonstrated using simulated data that simultaneously fitting full spectroscopic time-series with a 2D Gaussian process can produce more robust transmission spectra in the presence of both wavelength- and time-correlated systematics. This could enable us to reliably analyse spectroscopic time-series at higher spectroscopic-resolution, while accounting for wavelength correlation due to instrumental effects such as interpixel capacitance, and 1/f readout noise (e.g. Schlawin et al. [105 ] ; Ahrer et al. [1 ] ). In the case of exoplanet detection, multi-band transit fitting (Gordon et al. [53 ] ), as well as joint-fitting radial velocities with multiple stellar activity indicators (Rajpaul et al. [97 ] ), have been shown to more reliably recover exoplanet signals in the presence of stellar variability. Additional uses of GPs applied to multi-dimensional datasets include joint-fitting pixel light curves for eclipse photometry (Fortune et al. [45 ] ), modelling pulsar profile variations (Keith et al. [71 ] ), and fitting multi-band supernova light curves (Villar et al. [122 ] ).
Unfortunately, GPs can be expensive to scale to large datasets, especially data with multiple dimensions. In the most general case, GP log-likelihood evaluations scale as O ( N 3 ) \mathcal{O}(N^{3}) O ( N 3 ) in runtime and O ( N 2 ) \mathcal{O}(N^{2}) O ( N 2 ) in memory for N N N datapoints. Fortunately, plenty of optimisations exist with significantly improved scaling in both runtime and memory, which either approximate log-likelihood calculations, or use a restricted set of kernel functions which have special mathematical structure. A popular example of the latter is CELERITE, a GP optimisation developed to fit one-dimensional inputs (Foreman-Mackey et al. [42 ] ), which has been used extensively for modelling systematics in transit light curves (e.g. Espinoza et al. [40 ] ; Barros et al. [10 ] ; Alderson et al. [4 ] ), as well as for modelling stellar activity, granulation and oscillations (e.g. Angus et al. [8 ] ; Morris et al. [89 ] ). In fact, CELERITE is a special case of a broader class of GP optimisations which scale linearly on 1D (sortable) inputs. Quasiseparable kernels, as implemented in TINYGP (Foreman-Mackey et al. [43 ] ), S+LEAF matrices from the package SPLEAF (Delisle et al. [30 , 31 ] ), and state-space models (Hartikainen and Särkkä [59 ] ; Jordán et al. [68 ] ) implemented in SMOLGP (Rubenzahl et al. [101 ] ), can all be viewed as generalisations of CELERITE which can implement all CELERITE kernels, plus exact implementations of Matérn half-integer kernels, GP derivatives, and banded covariance matrices.
Various GP optimisations have been introduced for 2D ‘rectangular’ datasets, such as a spectroscopic time-series, images, parallel time-series of stellar activity indicators, etc. The various 2D GP optimisations introduced include extensions of CELERITE (Gordon et al. [53 ] ; Keith et al. [71 ] ), extensions of state-space models (Särkkä et al. [104 ] ), and various approximations including inducing point formulations (e.g. Snelson and Ghahramani [108 ] ; Wilson and Nickisch [130 ] ) and HODLR (Ambikasaran et al. [7 ] ), not all of which require a complete rectangular dataset. Some very flexible approaches work by exploiting Kronecker product algebra, which use clever linear transformations to split up covariance matrix decomposition into separate covariance matrices in each dimension, such as in work by Saatchi [102 ] , Rakitsch et al. [98 ] , and Brunisma et al. (2020). These algorithms do tend to be restricted to complete rectangular data without missing values, but non-uniform spacing in each dimension is allowed. In this work, we expand on such optimisations to include a much broader range of kernel functions, as well as identifying which algorithms can be combined with common 1D GP optimisations, including CELERITE, to achieve excellent scaling in one or multiple dimensions, while log-likelihood calculations remain exact.
These optimisations could be relevant to a wide range of exoplanet applications, including transit spectroscopy at both low- and high-resolution, as well as extreme precision radial velocities, and high-contrast imaging. In the discussion section, we demonstrate the close connections between these methods and popular data-driven algorithms like PCA and SysRem, but implemented as a likelihood-based approach which marginalises over components, can include smoothing priors on components, and features a principled treatment of noise. Of relevance outside of astronomy, where 2D GPs may instead be referred to as spatiotemporal GPs, multi-output GPs, or multi-task GPs, we provide a new, exact optimisation for the Linear Mixing Model, which combined with Gordon et al. [53 ] and quasiseparable kernels, extends the Orthogonal Instantaneous Linear Mixing Model (OILMM) from Bruinsma et al. [19 ] to drop the orthogonality requirement for the mixing matrix, while maintaining a linear runtime scaling in the number of datapoints.
Each optimisation identified has been implemented and validated in a new update to our open-source Python package LUAS, with tutorials and extensive documentation available1 .
Methods
Introduction to Gaussian processes
Gaussian processes provide a very versatile method for calculating the mean and uncertainty of deterministic model parameters, given a dataset containing complex noise sources. As an example, suppose that we have a single transit light curve with time-correlated noise, where the data can be described as:
y = μ ( θ , t ) + f s y s + ε (1) \mathbf{y} = \boldsymbol{\mu}(\boldsymbol{\theta},\mathbf{t}) + \mathbf{f}_{\mathrm{sys}} + \boldsymbol{\varepsilon} \tag*{(1)} y = μ ( θ , t ) + f sys + ε (1)
where μ ( θ , t ) \boldsymbol{\mu}(\boldsymbol{\theta},\mathbf{t}) μ ( θ , t ) is an analytical model of the transit light curve observed at times t \mathbf{t} t , and parameterised by model parameters θ \boldsymbol{\theta} θ , f s y s \mathbf{f}_{\mathrm{sys}} f sys is an unknown flux time-series due to some form of systematics (e.g. stellar variability), and ε \boldsymbol{\varepsilon} ε is the observed white noise arising from Poisson processes like photon counting statistics, with ε ∼ N ( 0 , Σ ) \boldsymbol{\varepsilon} \sim\mathcal{N}(\mathbf{0},\boldsymbol{\Sigma}) ε ∼ N ( 0 , Σ ) , where the covariance matrix describing the white noise Σ \boldsymbol{\Sigma} Σ is diagonal. μ ( θ , t ) \boldsymbol{\mu}(\boldsymbol{\theta},\mathbf{t}) μ ( θ , t ) is typically referred to as the ‘mean function’, and for transits is often modelled following Mandel and Agol [85 ] . For any fixed mean function and systematics model, the likelihood function (the probability of the data given the model) is simply the probability of generating ε \boldsymbol{\varepsilon} ε from Σ \boldsymbol{\Sigma} Σ , which is given by the Gaussian probability density function:
p ( D ∣ θ , t , f s y s ) = 1 ( 2 π ) N ∣ Σ ∣ exp ( − 1 2 ε T Σ − 1 ε ) (2) p(\mathbf{D} \mid\boldsymbol{\theta},\mathbf{t},\mathbf{f}_{\mathrm{sys}}) = \frac{1}{\sqrt{(2\pi)^{N} \lvert\boldsymbol{\Sigma} \rvert}} \exp\left(-\frac{1}{2}\boldsymbol{\varepsilon}^{T}\boldsymbol{\Sigma}^{-1}\boldsymbol{\varepsilon}\right) \tag*{(2)} p ( D ∣ θ , t , f sys ) = ( 2 π ) N ∣ Σ ∣ 1 exp ( − 2 1 ε T Σ − 1 ε ) (2)
which after taking the logarithm (useful for numerical stability) yields
log p ( D ∣ θ , t , f s y s ) = − 1 2 ε T Σ − 1 ε − 1 2 log ∣ Σ ∣ − N 2 log 2 π . (3) \log p(\mathbf{D} \mid\boldsymbol{\theta},\mathbf{t},\mathbf{f}_{\mathrm{sys}}) = -\frac{1}{2}\boldsymbol{\varepsilon}^{T}\boldsymbol{\Sigma}^{-1}\boldsymbol{\varepsilon} - \frac{1}{2}\log\lvert\boldsymbol{\Sigma} \rvert- \frac{N}{2}\log2\pi. \tag*{(3)} log p ( D ∣ θ , t , f sys ) = − 2 1 ε T Σ − 1 ε − 2 1 log ∣ Σ ∣ − 2 N log 2 π . (3)
The systematics model may in reality describe the host star flux varying in time due to chaotic processes such as stellar granulation, it could describe variations in the throughput of the telescope, or it could be a combination of multiple processes varying over time which we may have no way to analytically model. To span this full space of models, we could fit for every element of f s y s f_{\mathrm{sys}} f sys , introducing N N N free parameters for N N N datapoints, but doing this without any constraints would completely overfit our data. So we add a prior constraint on f s y s f_{\mathrm{sys}} f sys which penalises complexity, normally we impose that it should vary with some amplitude h h h , and it should have a correlation timescale it changes over, with nearby points in time having similar values, and widely separated points in time being almost fully independent. We can implement this by choosing a multivariate Gaussian prior f s y s ∼ N ( 0 , K s y s ) f_{\mathrm{sys}} \sim\mathcal{N}(0, K_{\mathrm{sys}}) f sys ∼ N ( 0 , K sys ) , where the covariance matrix K s y s K_{\mathrm{sys}} K sys is calculated using a kernel function which describes the covariance between each pair of points. For example, a common choice of kernel function is the exponential kernel
k ℓ ( t , t ′ ) = h 2 exp ( − ∣ t − t ′ ∣ ℓ t ) , (4) k_{\ell}(t,t') = h^2 \exp\left(-\frac{|t-t'|}{\ell_t}\right), \tag*{(4)} k ℓ ( t , t ′ ) = h 2 exp ( − ℓ t ∣ t − t ′ ∣ ) , (4)
where the covariance between two points decays exponentially with their separation in time, parameterised by the timescale ℓ t \ell_t ℓ t . The kernel function can be thought of as a ‘smoothness prior’ on our systematics model f s y s f_{\mathrm{sys}} f sys , i.e. whatever process is the cause of these systematics, we assume that it is continuous, varying over some characteristic time scale ℓ t \ell_t ℓ t , and with an average amplitude h h h .
Now that we have this smoothing prior on f s y s f_{\mathrm{sys}} f sys , we can freely fit N N N parameters to N N N datapoints. However, if we optimised f s y s f_{\mathrm{sys}} f sys to its best-fit values and then kept it fixed, we would essentially be assuming that we have perfectly modelled out the systematics without any error, but the presence of white noise introduces uncertainty in the exact systematics model. Instead, we marginalise over all of the parameter space f s y s ∈ R N f_{\mathrm{sys}} \in\mathbb{R}^{N} f sys ∈ R N , weighting towards the best-fitting models. This could be performed by marginalising all N N N parameters of f s y s f_{\mathrm{sys}} f sys in an MCMC, while simultaneously inferring our model parameters θ \boldsymbol{\theta} θ . Thankfully, GPs offer a much more efficient approach, since f s y s f_{\mathrm{sys}} f sys consists of linear parameters with the multivariate Gaussian prior p ( f s y s ) = N ( 0 , K s y s ) p(f_{\mathrm{sys}}) = \mathcal{N}(0, K_{\mathrm{sys}}) p ( f sys ) = N ( 0 , K sys ) , we can exactly marginalise over it analytically (e.g. Luger et al. 2017). The result of integrating Equation (3) over all values of f s y s f_{\mathrm{sys}} f sys results in the marginal log-likelihood
log p ( D ∣ θ , t ) = − 1 2 r T K − 1 r − 1 2 log ∣ K ∣ − N 2 log ( 2 π ) , (5) \log p(D|\boldsymbol{\theta},t) = -\frac{1}{2}\boldsymbol{r}^{T}K^{-1}\boldsymbol{r} - \frac{1}{2}\log|K| - \frac{N}{2}\log(2\pi), \tag*{(5)} log p ( D ∣ θ , t ) = − 2 1 r T K − 1 r − 2 1 log ∣ K ∣ − 2 N log ( 2 π ) , (5)
where we have written our GP covariance matrix as K = K s y s + Σ K = K_{\mathrm{sys}} + \Sigma K = K sys + Σ , and our data residuals as r = y − μ ( θ , t ) \boldsymbol{r} = \boldsymbol{y} - \boldsymbol{\mu}(\boldsymbol{\theta},t) r = y − μ ( θ , t ) . Combining this marginalised log-likelihood with Bayesian inference methods (e.g. MCMC, Nested Sampling) allows us to propagate the uncertainty introduced by the presence of unknown systematics into our transit model parameters θ \boldsymbol{\theta} θ , within a statistically rigorous and interpretable framework, and without sampling a huge number of free parameters.
If the constraint on the systematics model f s y s f_{\mathrm{sys}} f sys is desired, it is described by the GP predictive mean and predictive covariance f s y s ∼ N ( μ f s y s , K f s y s ) f_{\mathrm{sys}} \sim\mathcal{N}(\boldsymbol{\mu}_{f_{\mathrm{sys}}}, K_{f_{\mathrm{sys}}}) f sys ∼ N ( μ f sys , K f sys ) , as given by the equations:
μ f s y s = K s y s K − 1 r (6) \boldsymbol{\mu}_{f_{\mathrm{sys}}} = K_{\mathrm{sys}}K^{-1}\boldsymbol{r} \tag*{(6)} μ f sys = K sys K − 1 r (6)
K f s y s = K s y s − K s y s T K − 1 K s y s . (7) K_{f_{\mathrm{sys}}} = K_{\mathrm{sys}} - K_{\mathrm{sys}}^{T}K^{-1}K_{\mathrm{sys}}. \tag*{(7)} K f sys = K sys − K sys T K − 1 K sys . (7)
This can be used to visualise the most strongly weighted systematics models the GP is fitting, and in fact the distribution of systematics can even be predicted for unseen observations using similar formulae (see Appendix § 13 ). The reason why we can get a constraint on unseen data is because the correlation described by the kernel function restricts how much the systematics can deviate for predicted locations close to the observed data, which should give a robust predictive distribution if our kernel functions are an appropriate model of the systematics. The predictive mean and covariance on unseen data can be used to perform cross-validation tests, which can help to validate the accuracy of a GP model.
Implementing GP Inference
Performing inference with GPs typically involves calculating the log-likelihood function in Equation (5) many times (∼ 10 5 − 10 6 \sim10^{5} - 10^{6} ∼ 1 0 5 − 1 0 6 times for MCMC inference), which requires calculating both r T K − 1 r \boldsymbol{r}^{T}K^{-1}\boldsymbol{r} r T K − 1 r , and the log-determinant log ∣ K ∣ \log|K| log ∣ K ∣ each time. There are various approaches to do this depending on the problem, and methods like Cholesky decomposition can avoid explicitly calculating K − 1 K^{-1} K − 1 .
Using Cholesky Decomposition
A common approach is to calculate the Cholesky decomposition K = L L T K = LL^{T} K = L L T , where L L L is a lower triangular matrix. Since K − 1 = L − T L − 1 K^{-1} = L^{-T}L^{-1} K − 1 = L − T L − 1 , we can calculate:
r T K − 1 r = r T ( L − 1 ) T L − 1 r , (8) \boldsymbol{r}^{T}K^{-1}\boldsymbol{r} = \boldsymbol{r}^{T}(L^{-1})^{T}L^{-1}\boldsymbol{r}, \tag*{(8)} r T K − 1 r = r T ( L − 1 ) T L − 1 r , (8)
= r ~ T r ~ , (9) = \tilde{\boldsymbol{r}}^{T}\tilde{\boldsymbol{r}}, \tag*{(9)} = r ~ T r ~ , (9)
where r ~ = L − 1 r \tilde{\boldsymbol{r}} = L^{-1}\boldsymbol{r} r ~ = L − 1 r . We can use the fact that L L L is lower triangular to efficiently solve the linear system L r ~ = r L\tilde{\boldsymbol{r}} = \boldsymbol{r} L r ~ = r using an algorithm known as forward-substitution, which scales as O ( N 2 ) \mathcal{O}(N^{2}) O ( N 2 ) for a dense triangular solve. The log-determinant is simply calculated as log ∣ K ∣ = ∑ i 2 log ( L i i ) \log|K| = \sum_i 2\log(L_{ii}) log ∣ K ∣ = ∑ i 2 log ( L ii ) .
Using Eigendecomposition
Cholesky decomposition is not the only matrix decomposition that can be used for GP log-likelihood calculations. We could instead calculate the eigendecomposition K = Q Λ Q T K = Q\Lambda Q^{T} K = Q Λ Q T , where Q Q Q is a matrix of eigenvectors of K K K , and Λ \Lambda Λ is a diagonal matrix formed using the corresponding eigenvalues of those eigenvectors. While typically more expensive than Cholesky decomposition (see benchmarking in Figure 1), eigendecomposition can be used to calculate the log-likelihood in a similar way
r T K − 1 r = r T ( Λ − 1 2 Q T ) T ( Λ − 1 2 Q T ) r , (10) \boldsymbol{r}^{T}K^{-1}\boldsymbol{r} = \boldsymbol{r}^{T}(\Lambda^{-\frac{1}{2}}Q^{T})^{T}(\Lambda^{-\frac{1}{2}}Q^{T})\boldsymbol{r}, \tag*{(10)} r T K − 1 r = r T ( Λ − 2 1 Q T ) T ( Λ − 2 1 Q T ) r , (10)
= r ~ T r ~ , (11) = \tilde{\boldsymbol{r}}^{T}\tilde{\boldsymbol{r}}, \tag*{(11)} = r ~ T r ~ , (11)
now with r ~ = Λ − 1 2 Q T r \tilde{\boldsymbol{r}} = \Lambda^{-\frac{1}{2}}Q^{T}\boldsymbol{r} r ~ = Λ − 2 1 Q T r , and the log-determinant is simply log ∣ K ∣ = ∑ i log Λ i i \log|K| = \sum_i \log\Lambda_{ii} log ∣ K ∣ = ∑ i log Λ ii .
Using Scalable 1D Approaches
Despite their performance differences, Cholesky decomposition and eigendecomposition both scale as O ( N 3 ) \mathcal{O}(N^{3}) O ( N 3 ) in runtime and O ( N 2 ) \mathcal{O}(N^{2}) O ( N 2 ) in memory for a general covariance matrix. These approaches can always be used for any valid covariance matrix, regardless of kernel function choice or the dimensions of the dataset. However, if we build the covariance matrix using CELERITE terms, the Cholesky decomposition can be computed in O ( N J 2 ) \mathcal{O}(NJ^{2}) O ( N J 2 ) time, forward substitution can be applied in O ( N J ) \mathcal{O}(NJ) O ( N J ) time, and the memory cost also scales as O ( N J ) \mathcal{O}(NJ) O ( N J ) (Foreman-Mackey et al. 2017), where J J J refers to the number of CELERITE terms in the kernel function, which can be expressed as
k ( t , t ′ ) = ∑ j = 1 J e − c j ∣ t − t ′ ∣ [ a j cos ( d j ∣ t − t ′ ∣ ) + b j sin ( d j ∣ t − t ′ ∣ ) ] . (12) k(t,t') = \sum_{j=1}^{J} e^{-c_j|t-t'|}\left[a_j\cos(d_j|t-t'|) + b_j\sin(d_j|t-t'|)\right]. \tag*{(12)} k ( t , t ′ ) = j = 1 ∑ J e − c j ∣ t − t ′ ∣ [ a j cos ( d j ∣ t − t ′ ∣ ) + b j sin ( d j ∣ t − t ′ ∣ ) ] . (12)
CELERITE can implement the exponential kernel and the simple harmonic oscillator kernel, which can be very useful for modelling stellar activity [42 , 8 , 89 ] .
Figure 1. Runtime to decompose matrices using an M4 MacBook Air and an NVIDIA A100 GPU for eigendecomposition (using JAX.NUMPY.LINALG.EIGH), general Cholesky decomposition (using JAX.SCIPY.LINALG.CHOLESKY), and quasiseparable Cholesky decomposition of an exponential kernel (using TINYGP). The GPU is typically slower for small matrices, but outperforms the CPU for larger matrices by exploiting more parallel workers.
For our work, we use the package TINYGP, which implements a generalisation of CELERITE [43 ] , using the algebra of quasiseparable matrices [38 ] , and state-space models [e.g. 59 , 68 ] . It can implement all CELERITE kernels with the same runtime and memory scaling, but it also features numerically stable, exact implementations of the Matérn-3/2 and Matérn-5/2 kernels, and banded covariance matrices [43 ] . It can also calculate GP derivatives following the framework of state-space models, similar to Rubenzahl et al. [101 ] . While similar extensions have also been implemented in the package S+LEAF [30 , 31 ] , the quasiseparable kernels in TINYGP are implemented entirely within JAX, which combined with our JAX-based implementation of LUAS, allows for efficient auto-differentiation and GPU hardware acceleration [16 ] . This is useful for scaling to problems involving large datasets, and for performing high-dimensional parameter inference using efficient gradient-based methods. For this work, we will use J J J to refer to the quasiseparable rank, where the exponential kernel is quasiseparable rank 1, the Matérn-3/2 kernel is rank 2, and the kernel in Equation (12) actually has rank 2 J 2J 2 J . Summing kernels will sum their quasiseparable ranks, while multiplying kernels multiplies their ranks. We note that some quasiseparable kernels theoretically follow a O ( N J 3 ) \mathcal{O}(NJ^3) O ( N J 3 ) runtime scaling and O ( N J 2 ) \mathcal{O}(NJ^2) O ( N J 2 ) memory scaling, such as high-order Matérn kernels and wide banded matrices, but these are not typically used in practice, and sums of arbitrary quasiseparable kernels will still tend towards CELERITE scalings by exploiting block diagonal transition matrices [43 ] , so we use the scalings associated with CELERITE from Foreman-Mackey et al. [42 ] for calculations in this work.
Unfortunately, while all these methods have excellent scaling, they depend on the sort order of their inputs, and cannot add or
multiply kernels whose inputs have a different sort order. In practice, this makes them challenging to extend to 2D datasets, for example, if wavelength and time are kernel inputs on a spectroscopic time-series, it is not possible to sort all datapoints into chronological order, while never decreasing in wavelength. Creative approaches do exist to avoid this problem [e.g. 104 , 53 , 71 ] , but they can come at significant computational cost for large datasets.
Using Any Matrix Square Root
Both the Cholesky factor L \mathbf{L} L , and the term Q Λ 1 2 \mathbf{Q}\mathbf{\Lambda}^{\frac{1}{2}} Q Λ 2 1 , are distinct examples of matrix square roots K 1 2 \mathbf{K}^{\frac{1}{2}} K 2 1 of K \mathbf{K} K , where we will use the notation that a matrix square root satisfies
K = K 1 2 ( K 1 2 ) T , (13) \mathbf{K} = \mathbf{K}^{\frac{1}{2}}\left(\mathbf{K}^{\frac{1}{2}}\right)^{\mathrm{T}}, \tag*{(13)} K = K 2 1 ( K 2 1 ) T , (13)
and a matrix inverse square root satisfies
K − 1 = ( K − 1 2 ) T K − 1 2 , (14) \mathbf{K}^{-1} = \left(\mathbf{K}^{-\frac{1}{2}}\right)^{\mathrm{T}}\mathbf{K}^{-\frac{1}{2}}, \tag*{(14)} K − 1 = ( K − 2 1 ) T K − 2 1 , (14)
where for our purposes, we never require the matrix square root to be symmetric, but this notation does vary across the literature. By multiplying these two equations together, we see that if K 1 2 \mathbf{K}^{\frac{1}{2}} K 2 1 is invertible, then its inverse must be a matrix inverse square root K − 1 2 \mathbf{K}^{-\frac{1}{2}} K − 2 1 .
We introduce this concept to notate a general framework to calculate r T K − 1 r \mathbf{r}^{\mathrm{T}}\mathbf{K}^{-1}\mathbf{r} r T K − 1 r in the GP log-likelihood, where we first multiply our data residuals by any matrix inverse square root r ~ = K − 1 2 r \tilde{\mathbf{r}} = \mathbf{K}^{-\frac{1}{2}}\mathbf{r} r ~ = K − 2 1 r , followed by squaring and summing the results to calculate r ~ T r ~ \tilde{\mathbf{r}}^{\mathrm{T}}\tilde{\mathbf{r}} r ~ T r ~ . The log-determinant can often be calculated easily once a matrix square root is calculated, as noted for both the Cholesky decomposition and the eigendecomposition.
In addition, if we want to take a draw from the GP covariance matrix K \mathbf{K} K , which is equivalent to sampling from the prior GP places on the space of systematics models f s y s f_{\mathrm{sys}} f sys , we simply generate a random normal vector z ∼ N ( 0 , I N ) \mathbf{z} \sim\mathcal{N}(0,\mathbf{I}_N) z ∼ N ( 0 , I N ) and multiply by any matrix square root r = K 1 2 z \mathbf{r} = \mathbf{K}^{\frac{1}{2}}\mathbf{z} r = K 2 1 z , which will result in r \mathbf{r} r having covariance K \mathbf{K} K , which can be shown from the definition of covariance K = E ( ( r − r ˉ ) ( r − r ˉ ) T ) \mathbf{K} = \mathbb{E}((\mathbf{r}-\bar{\mathbf{r}})(\mathbf{r}-\bar{\mathbf{r}})^{\mathrm{T}}) K = E (( r − r ˉ ) ( r − r ˉ ) T ) . This can even be applied to sampling from the GP predictive covariance matrix (see Appendix § 13 ). Matrix square roots can therefore perform all of the typical calculations GPs are used for. We use this generalisation of factorising matrices into arbitrary matrix square roots to introduce modularity in our GP optimisations, scalable approaches using CELERITE and TINYGP can be used for calculations when a linear runtime scaling when appropriate, but if an arbitrary covariance model is needed, general Cholesky decomposition or eigendecomposition can alternatively be used.
Introduction to 2D Gaussian processes
Consider the example of a spectroscopic time-series of a transiting exoplanet, containing N λ N_{\lambda} N λ wavelengths and N t N_t N t time points. We would like to recover the transit depth at each wavelength, accounting for some unknown, two-dimensional systematics model F s y s F_{\mathrm{sys}} F sys , i.e.
Y = M ( θ , λ , t ) + F s y s + E λ , t , (15) \mathbf{Y} = \mathbf{M}(\boldsymbol{\theta},\boldsymbol{\lambda},\mathbf{t}) + \mathbf{F}_{\mathrm{sys}} + \mathbf{E}_{\lambda,t}, \tag*{(15)} Y = M ( θ , λ , t ) + F sys + E λ , t , (15)
where our mean function M ( θ , λ , t ) \mathbf{M}(\boldsymbol{\theta},\boldsymbol{\lambda},\mathbf{t}) M ( θ , λ , t ) now produces a ( N λ , N t ) (N_{\lambda},N_t) ( N λ , N t ) 2D array of transit light curves parameterised by θ \boldsymbol{\theta} θ (e.g. including transit depths at every wavelength), our systematics model F s y s F_{\mathrm{sys}} F sys is an unknown 2D array, and E λ , t E_{\lambda,t} E λ , t is a 2D array of independent white noise, which may be homoscedastic or heteroscedastic. Similar to the 1D case, we want to marginalise over F s y s F_{\mathrm{sys}} F sys by computing a marginal log-likelihood that avoids us needing to freely fit for every parameter in F s y s F_{\mathrm{sys}} F sys (which could require millions of free parameters!).
Note that typical transit spectroscopy analyses fit each spectroscopic light curve separately, and assume that the systematics in each light curve are independent (i.e. each row of F s y s F_{\mathrm{sys}} F sys is statistically independent). However, effects such as readout noise and various sources of stellar activity typically do not occur in isolation for each wavelength (e.g. [105 , 89 ] ). Uncertainties in our mean function parameters θ \boldsymbol{\theta} θ will not be correctly propagated if F s y s F_{\mathrm{sys}} F sys is not marginalised over accurately, typically if the process which produces F s y s F_{\mathrm{sys}} F sys is correlated in wavelength, then the uncertainty introduced to the transit depths will have a similar correlation in wavelength (Holmberg and Madhusudhan [64 ] ; Fortune et al. [44 ] ). By joint-fitting the full dataset with a 2D GP, we can recover our transit depths reliably despite correlations in both wavelength and time. To help distinguish different noise sources, we divide them into a few useful scenarios.
White noise
We can expect some level of white noise in any observation, such as from photon noise. White noise has no correlation in time or wavelength, and is included by adding to the diagonal of the covariance matrix. Noise in real observations is almost always heteroscedastic, but an important distinction for this work is whether the amplitude of the white noise can be expressed in a separable way in each dimension, or if the amplitude of white noise must be specified explicitly for each datapoint. Mathematically we can separate these cases as
k s e p a r a b l e ( λ , λ ′ , t , t ′ ) = [ σ 2 ( λ ) δ λ λ ′ ] [ σ 2 ( t ) δ t t ′ ] , (16) k_{\mathrm{separable}}(\lambda,\lambda',t,t') = [\sigma^2(\lambda)\delta_{\lambda\lambda'}][\sigma^2(t)\delta_{tt'}], \tag*{(16)} k separable ( λ , λ ′ , t , t ′ ) = [ σ 2 ( λ ) δ λ λ ′ ] [ σ 2 ( t ) δ t t ′ ] , (16)
k n o n - s e p a r a b l e ( λ , λ ′ , t , t ′ ) = σ 2 ( λ , t ) δ λ λ ′ δ t t ′ , (17) k_{\mathrm{non\text{-}separable}}(\lambda,\lambda',t,t') = \sigma^2(\lambda,t)\delta_{\lambda\lambda'}\delta_{tt'}, \tag*{(17)} k non - separable ( λ , λ ′ , t , t ′ ) = σ 2 ( λ , t ) δ λ λ ′ δ t t ′ , (17)
where each Kronecker delta δ i j = 1 \delta_{ij}=1 δ ij = 1 if i = j i=j i = j and zero otherwise. The separable form produces structure in the covariance matrix on rectangular datasets which is important for our Kronecker product approach. Alternative 2D GP optimisations exist which can model non-separable noise amplitudes ([104 , 53 ] ), but we will exclusively focus on optimisations for the separable case.
Measurement Covariance
In addition to white noise, we may also have noise which is uncorrelated in one of the dimensions, but correlated in the other. The case of time-correlated noise which is uncorrelated in wavelength is already an existing model, when time-correlated 1D GPs are fit to each spectroscopic light curve independently (which we return to in § 2.8 ). For this section, we will focus on wavelength-correlated noise which is independent for every time point. For example, each spectrum we measure may include some wavelength correlation when being read out, such as due to residual 1/f readout noise in JWST data (e.g. [105 , 1 ] ). See Figure 2 for a simulated example of measurement covariance. In this case, our kernel function may be of the form
Figure 2. Simulated spectroscopic time-series of an exoplanet eclipse with measurement covariance added following the kernel function given in Equation (19) . Note the vertical streaks in the residuals which resemble real JWST/NIRCam data reductions, such as in Ahrer et al. [1 ] . Despite simulating with very low white noise, the white light curve appears particularly noisy (compare with Figure 3 , Figure 4 , Figure 5 which have the same amplitude of white noise), a consequence of how wavelength correlations bin down gradually compared to white noise.
k ( λ , λ ′ , t , t ′ ) = [ s λ ( λ , λ ′ ) ] [ σ 2 ( t ) δ t t ′ ] , (18) k(\lambda,\lambda',t,t') = [s_{\lambda}(\lambda,\lambda')][\sigma^2(t)\delta_{tt'}], \tag*{(18)} k ( λ , λ ′ , t , t ′ ) = [ s λ ( λ , λ ′ )] [ σ 2 ( t ) δ t t ′ ] , (18)
where the wavelength-correlation given by the kernel function s λ ( λ , λ ′ ) s_{\lambda}(\lambda,\lambda') s λ ( λ , λ ′ ) models the measurement covariance , but this wavelength-correlation is independent for each time point (i.e. each integration has an independent draw of readout noise). The square brackets highlight that this kernel function is also fully separable as a product of kernel functions in wavelength and time, important for the optimisations in this work. Within the notation of Equation (15) , this noise can be thought of as adding independent wavelength-correlated noise draws to each column of F s y s F_{\mathrm{sys}} F sys . Equation 18 includes a function σ ( t ) \sigma(t) σ ( t ) , which can be used to vary the amplitude of the measurement covariance in time, useful if the exposure time varies across the time-series.
Since both measurement covariance and white noise are independent in time, it is natural to include both terms within the kernel function s λ ( λ , λ ′ ) s_{\lambda}(\lambda,\lambda') s λ ( λ , λ ′ ) . For example, if readout noise can be described by an exponential kernel in wavelength, then we may have
s λ ( λ , λ ′ ) = σ 2 ( λ ) δ λ λ ′ + h r e a d o u t 2 exp ( − ∣ λ − λ ′ ∣ l r e a d o u t ) . (19) s_{\lambda}(\lambda,\lambda') = \sigma^2(\lambda)\delta_{\lambda\lambda'} + h_{\mathrm{readout}}^2 \exp\left(-\frac{\lvert\lambda-\lambda'\rvert}{l_{\mathrm{readout}}}\right). \tag*{(19)} s λ ( λ , λ ′ ) = σ 2 ( λ ) δ λ λ ′ + h readout 2 exp ( − l readout ∣ λ − λ ′ ∣ ) . (19)
Alternatively, rather than parameterising using kernel functions, we could instead build the measurement covariance matrix S λ \mathbf{S}_{\lambda} S λ directly using data from darks or from the autocovariance of the data residuals (e.g. [105 , 64 ] ). However, parameterising measurement covariance in terms of kernel functions allows us to quantify the strength of each term, which could be important for comparing reduction choices, and it avoids conditioning on the best-fit residuals.
For the optimisations in this work, we will assume that the covariance matrices produced by s λ ( λ , λ ′ ) s_{\lambda}(\lambda,\lambda') s λ ( λ , λ ′ ) and s t ( t , t ′ ) s_t(t,t') s t ( t , t ′ ) are both positive definite. This will always be true if non-zero white noise is included in these terms plus any valid kernel terms, since the diagonal covariance matrix describing the white noise is guaranteed to be positive definite, and adding valid kernel terms to a positive definite matrix will always maintain positive definiteness. We label kernels as s λ s_{\lambda} s λ and s t s_t s t to highlight that these must produce positive definite terms, while any kernels labelled as k λ k_{\lambda} k λ or k t k_t k t can be either positive definite or positive semi-definite, which all kernels must satisfy by definition.
Low-Rank Kernel Terms
Some systematics sources may follow a Gaussian process in time, with only their amplitude varying in wavelength, i.e. they have the same shape in all light curves. Common-mode systematics are an example, where systematics have both the same shape and the same amplitude across all light curves. Stellar granulation has been suggested to approximately follow this model with a wavelength-dependent amplitude (e.g. [53 , 89 ] ). Figure 3 simulates a simple stellar granulation signal following an exponential kernel function in time (see Kallinger et al. [70 ] for relevant time kernels), and a wavelength-dependence from Morris et al. [89 ] for a T e f f = 6000 T_{\mathrm{eff}}=6000 T eff = 6000 K star, scaled to 80 ppm at 0.4 μ \mu μ m. We can describe such a model with the kernel
Figure 3. Same as Figure 2 but injecting a stellar granulation signal following the kernel in Equation (20) . The wavelength-dependence α ( λ ) \alpha(\lambda) α ( λ ) follows the model in Morris et al. [89 ] with an arbitrary scaling, while the time-dependence is stochastic and follows an exponential kernel in time. The systematics have the same shape in all light curves, but their amplitude is largest at optical wavelengths and gradually weakens into near-infrared wavelengths.
k ( λ , λ ′ , t , t ′ ) = [ α ( λ ) α ( λ ′ ) ] [ k t ( t , t ′ ) ] , (20) k(\lambda,\lambda',t,t') = [\alpha(\lambda)\alpha(\lambda')][k_t(t,t')], \tag*{(20)} k ( λ , λ ′ , t , t ′ ) = [ α ( λ ) α ( λ ′ )] [ k t ( t , t ′ )] , (20)
where α ( λ ) \alpha(\lambda) α ( λ ) is a deterministic function we can use to describe the amplitude in wavelength. Alternatively, we could freely fit for the amplitude at each wavelength in an amplitude vector α λ \boldsymbol{\alpha}_{\lambda} α λ , rather than expecting it to follow a physically-motivated model. While current data analyses may attempt to remove these systematics before spectroscopic light curve fitting (e.g. performing a common-mode correction), we can use such a kernel to joint-fit these systematics and propagate the uncertainty induced into the transit spectrum (Fortune et al. [44 ] ).
We will refer to these low-rank kernel terms as inputs (or just low-rank terms for short), since the wavelength covariance matrix built using k λ ( λ , λ ′ ) = α ( λ ) α ( λ ′ ) k_{\lambda}(\lambda,\lambda')=\alpha(\lambda)\alpha(\lambda') k λ ( λ , λ ′ ) = α ( λ ) α ( λ ′ ) only has a single non-zero eigenvalue. It is also equivalent to a low-rank systematics model, of the form F s y s = α v t T \mathbf{F}_{\mathrm{sys}}=\boldsymbol{\alpha}\mathbf{v}_{t}^{T} F sys = α v t T , where v t ∼ N ( 0 , K t ) \mathbf{v}_{t}\sim\mathcal{N}(\mathbf{0},K_{t}) v t ∼ N ( 0 , K t ) is a 1D GP in time. In fact, we will show in § 6.2 that this connects to a probabilistic interpretation of a single PCA component, with a GP smoothing prior placed on its time component.
We might also consider a low-rank kernel term with the dimensions flipped, i.e.
k ( λ , λ ′ , t , t ′ ) = [ k λ ( λ , λ ′ ) ] [ β ( t ) β ( t ′ ) ] . (21) k(\lambda,\lambda',t,t')=\left[k_{\lambda}(\lambda,\lambda')\right]\left[\beta(t)\beta(t')\right]. \tag*{(21)} k ( λ , λ ′ , t , t ′ ) = [ k λ ( λ , λ ′ ) ] [ β ( t ) β ( t ′ ) ] . (21)
This type of kernel could be useful if fitting a systematic with a fixed shape in time, but with an unknown shape in wavelength. This could be most useful for analytically marginalising out baseline or detrending parameters that are fit for each light curve. As an example, modelling initial detector settling in spectroscopic time-series could be a use case, where the time-dependence could be explicitly modelled as a decaying exponential with some settling time length scale l r a m p l_{\mathrm{ramp}} l ramp , i.e.
β ( t ) = exp ( − t − t 0 l r a m p ) (22) \beta(t)=\exp\left(-\frac{t-t_{0}}{l_{\mathrm{ramp}}}\right) \tag*{(22)} β ( t ) = exp ( − l ramp t − t 0 ) (22)
which is a deterministic function, not to be confused with the exponential kernel function. The wavelength-dependence could be treated as stochastic and independent for each wavelength, i.e.
k λ ( λ , λ ′ ) = σ r a m p 2 δ λ λ ′ . (23) k_{\lambda}(\lambda,\lambda')=\sigma_{\mathrm{ramp}}^{2}\delta_{\lambda\lambda'}. \tag*{(23)} k λ ( λ , λ ′ ) = σ ramp 2 δ λ λ ′ . (23)
Such a model would be exactly equivalent to freely marginalising over an additive settling ramp model to every spectroscopic light curve independently, with a Gaussian prior on its amplitude N ( 0 , σ r a m p ) \mathcal{N}(0,\sigma_{\mathrm{ramp}}) N ( 0 , σ ramp ) , and a shared decay length scale l r a m p l_{\mathrm{ramp}} l ramp . By adding this to the GP covariance matrix, we save ourselves fitting N λ f r e e N_{\lambda}^{\mathrm{free}} N λ free parameters, instead marginalising over their values analytically. Alternatively, we could use a kernel function in wavelength, if a smoothing prior is desired (see Figure 4 as a simulated example). Imposing a constraint that the settling amplitude varies smoothly in wavelength may help to improve constraints if pushing to analyse data at higher spectroscopic resolution, as freely fitting a ramp to each pixel channel could be poorly constrained. Some caveats of this approach are that anomalous ramps in individual pixels may be poorly modelled, and that this ramp model would have to be applied additively rather than being multiplicative (although if the transit model is close to unity at the beginning of the observation then this may be a negligible distinction). Other possible parameters which could be marginalised out include linear baseline terms, such as an offset or slope at each wavelength, or fitting regularised detrending vectors to each light curve, such as the y y y -offset of the spectrum, or the FWHM of the spectral trace.
Figure 4. Analogous plot to Figure 3 but with the kernel function flipped, injecting a low-rank term whose amplitude follows a deterministic exponential decay in time, but treating the amplitude in wavelength as a GP following a squared exponential kernel. An example use case is fitting spectroscopic light curves with wavelength-dependent settling, imposing that the settling amplitude varies gradually between wavelengths as a GP, but still explicitly defining how the settling varies in time within the function β ( t ) \beta(t) β ( t ) . Adding this term to the GP covariance matrix allows us to analytically marginalise out the wavelength-dependent settling amplitude.
2D Kernel Terms
The final scenario we consider is systematics which change shape as a function of both time-separation and wavelength-separation. We focus on the case when the kernel function is formed from multiplying together separate kernel functions in wavelength and time:
k ( λ , λ ′ , t , t ′ ) = k λ ( λ , λ ′ ) k t ( t , t ′ ) . (24) k(\lambda,\lambda',t,t')=k_{\lambda}(\lambda,\lambda')k_{t}(t,t'). \tag*{(24)} k ( λ , λ ′ , t , t ′ ) = k λ ( λ , λ ′ ) k t ( t , t ′ ) . (24)
which we will refer to as a 2D kernel term. For example, we could have a squared exponential kernel with both wavelength and time as inputs
k ( λ , λ ′ , t , t ′ ) = h 2 exp ( − 1 2 [ ∣ λ − λ ′ ∣ 2 l λ 2 + ∣ t − t ′ ∣ 2 l t 2 ] ) (25) k(\lambda,\lambda',t,t')=h^{2}\exp\left(-\frac{1}{2}\left[\frac{|\lambda-\lambda'|^{2}}{l_{\lambda}^{2}}+\frac{|t-t'|^{2}}{l_{t}^{2}}\right]\right) \tag*{(25)} k ( λ , λ ′ , t , t ′ ) = h 2 exp ( − 2 1 [ l λ 2 ∣ λ − λ ′ ∣ 2 + l t 2 ∣ t − t ′ ∣ 2 ] ) (25)
= [ h 2 exp ( − ∣ λ − λ ′ ∣ 2 2 l λ 2 ) ] [ exp ( − ∣ t − t ′ ∣ 2 2 l t 2 ) ] , (26) =\left[h^{2}\exp\left(-\frac{|\lambda-\lambda'|^{2}}{2l_{\lambda}^{2}}\right)\right]\left[\exp\left(-\frac{|t-t'|^{2}}{2l_{t}^{2}}\right)\right], \tag*{(26)} = [ h 2 exp ( − 2 l λ 2 ∣ λ − λ ′ ∣ 2 ) ] [ exp ( − 2 l t 2 ∣ t − t ′ ∣ 2 ) ] , (26)
which can be viewed as the product of separate wavelength and time squared exponential kernels. We can also just pick arbitrary kernels in wavelength and time to multiply together, e.g. a squared exponential in wavelength times a Matérn-5/2 kernel in time. Note that 2D kernel terms can produce a full-rank F s y s \mathbf{F}_{\mathrm{sys}} F sys systematics model, they are not to be confused with the model F s y s = u λ v t T \mathbf{F}_{\mathrm{sys}}=\mathbf{u}_{\lambda}\mathbf{v}_{t}^{T} F sys = u λ v t T , where both u λ \mathbf{u}_{\lambda} u λ and v t \mathbf{v}_{t} v t represent independent GPs in wavelength and time.
In the limit as the wavelength length scale l λ → 0 l_{\lambda}\to0 l λ → 0 , these 2D kernels become equivalent to time-correlated systematics which are statistically independent for each wavelength. Figure 5 visualises this by simulating systematics with the 2D kernel using a very short wavelength length scale. This model is therefore similar to fitting 1D GPs to each spectroscopic light curve separately, just with a shared time length scale (see the approach as compared directly in Fortune et al. [44 ] ). We could even have a wavelength-dependent height scale h 2 → h ( λ ) h ( λ ′ ) h^{2}\to h(\lambda)h(\lambda') h 2 → h ( λ ) h ( λ ′ ) , if we want to account for specific wavelengths having larger systematics amplitudes.
Figure 5. Similar to Figure 2 , Figure 3 , and Figure 4 , but with systematics simulated using a 2D kernel term (a squared exponential in wavelength times an exponential kernel in time), with a short wavelength length scale to simulate systematics that are close to independent for each light curve. This is similar to what is currently modelled when fitting a GP to each spectroscopic light curve independently. The height scale could also be varied in wavelength, in case there are specific wavelengths with strong systematics.
Combining All Terms: The EXTENDEDLUASKERNEL
The primary focus of this work is a family of seven GP optimisations, which are all fully spanned by the EXTENDEDLUASKERNEL, which we define as
k ( λ , λ ′ , t , t ′ ) = s λ ( λ , λ ′ ) s t ( t , t ′ ) + k λ ; 2 D ( λ , λ ′ ) k t ; 2 D ( t , t ′ ) + ∑ i = 1 N α α i ( λ ) α i ( λ ′ ) k t i ( t , t ′ ) + ∑ j = 1 N β k λ j ( λ , λ ′ ) β j ( t ) β j ( t ′ ) , (27) \begin{aligned}
k(\lambda,\lambda',t,t')={}&s_{\lambda}(\lambda,\lambda')s_{t}(t,t')+k_{\lambda;2\mathrm{D}}(\lambda,\lambda')k_{t;2\mathrm{D}}(t,t')\\
&+\sum_{i=1}^{N_{\alpha}}\alpha_{i}(\lambda)\alpha_{i}(\lambda')k_{\mathrm{ti}}(t,t')+\sum_{j=1}^{N_{\beta}}k_{\lambda j}(\lambda,\lambda')\beta_{j}(t)\beta_{j}(t'), \tag*{(27)}
\end{aligned} k ( λ , λ ′ , t , t ′ ) = s λ ( λ , λ ′ ) s t ( t , t ′ ) + k λ ; 2 D ( λ , λ ′ ) k t ; 2 D ( t , t ′ ) + i = 1 ∑ N α α i ( λ ) α i ( λ ′ ) k ti ( t , t ′ ) + j = 1 ∑ N β k λj ( λ , λ ′ ) β j ( t ) β j ( t ′ ) , (27)
where s λ s_{\lambda} s λ , s t s_{t} s t , k λ ; 2 D k_{\lambda;2\mathrm{D}} k λ ; 2 D , k t ; 2 D k_{t;2\mathrm{D}} k t ; 2 D , k λ j k_{\lambda j} k λj , k t i k_{\mathrm{ti}} k ti can be arbitrary kernel functions in wavelength and time, and α i ( λ ) \alpha_{i}(\lambda) α i ( λ ) and β j ( t ) \beta_{j}(t) β j ( t ) are explicit user-defined functions (they may just build vectors of free parameters). The scaling in runtime and memory depends on which (if any) of these terms are non-zero, and which kernel terms can utilise GP optimisations, e.g. if they are quasiseparable. LUAS will automatically select from these optimisations based on the kernel function input by the user, although the user can also select the desired optimisation for cases where multiple are applicable.
Implementing 2D GP Inference
General approaches like Cholesky decomposition or eigendecomposition quickly become intractable beyond N ∼ 10 3 N\sim10^{3} N ∼ 1 0 3 datapoints, which is severely limiting for most 2D datasets. In fact, simply storing the covariance matrix K K K in memory can be prohibitive, e.g. a 2D dataset of shape ( 10 3 , 10 3 ) (10^{3},10^{3}) ( 1 0 3 , 1 0 3 ) has a ( 10 6 , 10 6 ) (10^{6},10^{6}) ( 1 0 6 , 1 0 6 ) covariance matrix, requiring 8 terabytes of memory at double precision. Methods which are scalable in both runtime and memory are needed to implement 2D GP inference for most datasets. Thankfully, there are already many existing approaches for this which we build from, with the most relevant for our work being CELERITE-based approaches, and Kronecker product algebra approaches.
Existing CELERITE-based Approaches
Simply multiplying together CELERITE kernels in wavelength and time is generally invalid, since the kernels must share the same input sort order2 . However, low-rank kernel terms avoid this problem by only having the kernel applied to inputs along a single dimension. This was used in Gordon et al. [53 ] to implement kernels of the form:
k ( λ , λ ′ , t , t ′ ) = σ 2 ( λ , t ) δ λ λ ′ δ t t ′ + ∑ i = 1 N α α i ( λ ) α i ( λ ′ ) k t i ( t , t ′ ) . (28) k(\lambda,\lambda',t,t')=\sigma^{2}(\lambda,t)\delta_{\lambda\lambda'}\delta_{tt'}+\sum_{i=1}^{N_{\alpha}}\alpha_i(\lambda)\alpha_i(\lambda')k_{t_i}(t,t'). \tag*{(28)} k ( λ , λ ′ , t , t ′ ) = σ 2 ( λ , t ) δ λ λ ′ δ t t ′ + i = 1 ∑ N α α i ( λ ) α i ( λ ′ ) k t i ( t , t ′ ) . (28)
It is equivalent to treating the 2D dataset as a single long time-series which can be fit using CELERITE kernels, first sorting all datapoints in chronological order, and then applying CELERITE while letting the white noise amplitude and kernel height scales vary for each point. Note that the white noise amplitude does not need to be separable in wavelength and time (distinguished in § 2.5 ). This approach naturally extends to quasiseparable kernels in TINYGP with the same scaling as in § 2.2.3 , but for N = N λ N t N=N_{\lambda}N_t N = N λ N t datapoints. For N α N_{\alpha} N α quasiseparable kernels, we get a total quasiseparable rank of J = ∑ i N α J k t i = N α J ˉ k t J=\sum_i^{N_{\alpha}}J_{k_{t_i}}=N_{\alpha}\bar{J}_{k_t} J = ∑ i N α J k t i = N α J ˉ k t . This gives a total scaling of O ( N α 2 N λ N t J ˉ k t 2 ) O(N_{\alpha}^{2}N_{\lambda}N_t\bar{J}_{k_t}^{2}) O ( N α 2 N λ N t J ˉ k t 2 ) in runtime, and O ( N α N λ N t J ˉ k t ) O(N_{\alpha}N_{\lambda}N_t\bar{J}_{k_t}) O ( N α N λ N t J ˉ k t ) in memory. We will demonstrate how this can be further optimised with Kronecker product algebra using our MULTILOWRANKKERNEL.
If we instead have the kernel function
k ( λ , λ ′ , t , t ′ ) = σ 2 ( λ , t ) δ λ λ ′ δ t t ′ + k λ ( λ , λ ′ ) k t ( t , t ′ ) , (29) k(\lambda,\lambda',t,t')=\sigma^{2}(\lambda,t)\delta_{\lambda\lambda'}\delta_{tt'}+k_{\lambda}(\lambda,\lambda')k_t(t,t'), \tag*{(29)} k ( λ , λ ′ , t , t ′ ) = σ 2 ( λ , t ) δ λ λ ′ δ t t ′ + k λ ( λ , λ ′ ) k t ( t , t ′ ) , (29)
we can express the wavelength covariance matrix built by k λ ( λ , λ ′ ) k_{\lambda}(\lambda,\lambda') k λ ( λ , λ ′ ) as a sum of low-rank terms (e.g. using its eigendecomposition K λ = ∑ i N λ λ i v i v i T K_{\lambda}=\sum_i^{N_{\lambda}}\lambda_i\mathbf{v}_i\mathbf{v}_i^{T} K λ = ∑ i N λ λ i v i v i T ), and then use Equation (28) with N α = N λ N_{\alpha}=N_{\lambda} N α = N λ , to decompose the matrix with a runtime scaling of O ( N λ 3 N t ) O(N_{\lambda}^{3}N_t) O ( N λ 3 N t ) . Accounting for measurement covariance is not explored in Gordon et al. [53 ] , but adding a LEAF matrix as introduced in Delisle et al. [30 ] , or adding a banded covariance matrix as implemented within TINYGP, could add arbitrary measurement covariance with a similar O ( N λ 2 N t ) O(N_{\lambda}^{2}N_t) O ( N λ 2 N t ) runtime scaling. While these optimisations are significantly faster than general matrix decomposition, they can still be prohibitive when N λ N_{\lambda} N λ is large, which is a common case for many JWST spectroscopic time-series.
Kronecker Product Algebra
Our work combines various Kronecker product algebra methods, particularly inspired by the work of Saatchi [102 ] and Rakitsch et al. [98 ] , to extend scalable 1D GP optimisations to 2D datasets for rectangular datasets. These are exact optimisations which can make the analysis of 2D datasets containing millions of datapoints tractable on a laptop. So far we have expressed our GP models in terms of kernel functions, but these optimisations are best derived by describing the covariance matrix of the GP in terms of its linear algebra structure. When we have a kernel term which is separable in each dimension, such as
k ( λ , λ ′ , t , t ′ ) = k λ ( λ , λ ′ ) k t ( t , t ′ ) , (30) k(\lambda,\lambda',t,t')=k_{\lambda}(\lambda,\lambda')k_t(t,t'), \tag*{(30)} k ( λ , λ ′ , t , t ′ ) = k λ ( λ , λ ′ ) k t ( t , t ′ ) , (30)
and if our dataset lies on a complete 2D grid of input locations, e.g. we have flux values observed at wavelengths λ \lambda λ and times t t t without any missing values, the covariance matrix this builds will be a Kronecker product of separate covariance matrices in each dimension:
( K λ ) i j = k λ ( λ i , λ j ) (31) (K_{\lambda})_{ij}=k_{\lambda}(\lambda_i,\lambda_j) \tag*{(31)} ( K λ ) ij = k λ ( λ i , λ j ) (31)
( K t ) i j = k t ( t i , t j ) (32) (K_t)_{ij}=k_t(t_i,t_j) \tag*{(32)} ( K t ) ij = k t ( t i , t j ) (32)
K = K λ ⊗ K t . (33) K=K_{\lambda}\otimes K_t. \tag*{(33)} K = K λ ⊗ K t . (33)
The Kronecker product operation multiplies each element in the left matrix times the full matrix on the right, for example:
A ⊗ B = [ a 11 a 12 a 13 a 21 a 22 a 23 a 31 a 32 a 33 ] ⊗ B = [ a 11 B a 12 B a 13 B a 21 B a 22 B a 23 B a 31 B a 32 B a 33 B ] . (34) A\otimes B=
\begin{bmatrix}
a_{11}&a_{12}&a_{13}\\
a_{21}&a_{22}&a_{23}\\
a_{31}&a_{32}&a_{33}
\end{bmatrix}
\otimes B=
\begin{bmatrix}
a_{11}B&a_{12}B&a_{13}B\\
a_{21}B&a_{22}B&a_{23}B\\
a_{31}B&a_{32}B&a_{33}B
\end{bmatrix}. \tag*{(34)} A ⊗ B = a 11 a 21 a 31 a 12 a 22 a 32 a 13 a 23 a 33 ⊗ B = a 11 B a 21 B a 31 B a 12 B a 22 B a 32 B a 13 B a 23 B a 33 B . (34)
For a spectroscopic time-series with N λ N_{\lambda} N λ wavelengths and N t N_t N t time points, this Kronecker product would form a ( N λ N t , N λ N t ) (N_{\lambda}N_t,N_{\lambda}N_t) ( N λ N t , N λ N t ) matrix. Since this full covariance matrix could be prohibitively expensive to store in memory, methods that only require storing individual covariance matrices in each dimension can have significantly better scaling.
For the low-rank kernel terms in § 2.7 , we can represent the covariance matrix they build using vector outer products. For example, our kernel term
k ( λ , λ ′ , t , t ′ ) = [ α ( λ ) α ( λ ′ ) ] [ k t ( t , t ′ ) ] , (35) k(\lambda,\lambda',t,t')=[\alpha(\lambda)\alpha(\lambda')][k_t(t,t')], \tag*{(35)} k ( λ , λ ′ , t , t ′ ) = [ α ( λ ) α ( λ ′ )] [ k t ( t , t ′ )] , (35)
builds a covariance matrix of the form
K = α α T ⊗ K t , (36) K=\boldsymbol{\alpha}\boldsymbol{\alpha}^{T}\otimes K_t, \tag*{(36)} K = α α T ⊗ K t , (36)
and the kernel term
k ( λ , λ ′ , t , t ′ ) = [ k λ ( λ , λ ′ ) ] [ β ( t ) β ( t ′ ) ] , (37) k(\lambda,\lambda',t,t')=[k_{\lambda}(\lambda,\lambda')][\beta(t)\beta(t')], \tag*{(37)} k ( λ , λ ′ , t , t ′ ) = [ k λ ( λ , λ ′ )] [ β ( t ) β ( t ′ )] , (37)
can be written as
K = K λ ⊗ β β T , (38) K=K_{\lambda}\otimes\boldsymbol{\beta}\boldsymbol{\beta}^{T}, \tag*{(38)} K = K λ ⊗ β β T , (38)
where the elements of these vectors are α i = α ( λ i ) \alpha_i=\alpha(\lambda_i) α i = α ( λ i ) , and β i = β ( t i ) \beta_i=\beta(t_i) β i = β ( t i ) . Many useful properties of Kronecker product algebra are laid out similarly for B . These properties demonstrate that Kronecker product algebra can dramatically simplify calculating the inverse and determinant of a large covariance matrix of the form A ⊗ B A\otimes B A ⊗ B , by breaking up the calculation into simpler decompositions of the individual matrices A A A and B B B .
Finally, Saatchi [102 ] also present an efficient algorithm for performing matrix multiplication of a Kronecker product matrix times a vector, for an arbitrary N A N B N_A N_B N A N B long vector c :
[ A ⊗ B ] c = vec ( A C B T ) , (39) [A\otimes B]\mathbf{c}=\operatorname{vec}(A C B^{T}), \tag*{(39)} [ A ⊗ B ] c = vec ( A C B T ) , (39)
where we reshape c \mathbf{c} c into a ( N A , N B ) (N_A,N_B) ( N A , N B ) matrix with C i j = c ( i − 1 ) N B + j C_{ij}=c_{(i-1)N_B+j} C ij = c ( i − 1 ) N B + j and the vec() operator reverses this, such that vec ( C ) = c \operatorname{vec}(C)=\mathbf{c} vec ( C ) = c . This trick reduces what would be a runtime scaling of O ( N A 2 N B 2 ) O(N_A^{2}N_B^{2}) O ( N A 2 N B 2 ) , down to O ( N A 2 N B + N A N B 2 ) O(N_A^{2}N_B+N_A N_B^{2}) O ( N A 2 N B + N A N B 2 ) in the worst case, and even faster if we can exploit special structure in the matrices A A A and/or B B B for fast matrix multiplications (e.g. for covariance matrices which are diagonal, low-rank, or built using CELERITE kernels). We make frequent use of this algorithm, and in our implementation we typically use the ( N λ , N t ) (N_{\lambda},N_t) ( N λ , N t ) residual array R \mathbf{R} R , rather than a N λ N t N_{\lambda}N_t N λ N t -long vector r \mathbf{r} r .
Results
We will build up to the ExtendedLUASKernel in Equation (27) by adding terms one-at-a-time. In practice, these more restricted optimisations are likely to be more useful for typical analyses, since they scale more efficiently, and most datasets likely won’t require the full flexibility of the ExtendedLUASKernel.
SeparableKernel: Accounting for Measurement Covariance
Suppose we have any kernel which is fully separable in each dimension
k ( λ , λ ′ , t , t ′ ) = s λ ( λ , λ ′ ) s t ( t , t ′ ) , (40) k(\lambda,\lambda',t,t') = s_{\lambda}(\lambda,\lambda')s_t(t,t'), \tag*{(40)} k ( λ , λ ′ , t , t ′ ) = s λ ( λ , λ ′ ) s t ( t , t ′ ) , (40)
which could be useful to account for both white noise and measurement covariance (e.g. following Equations (18) and (19) ). When evaluated on a 2D grid of input locations, this can be written as a single Kronecker product of positive definite covariance matrices in each dimension:
K = Σ λ ⊗ Σ t . (41) \mathbf{K} = \boldsymbol{\Sigma}_{\lambda} \otimes\boldsymbol{\Sigma}_t. \tag*{(41)} K = Σ λ ⊗ Σ t . (41)
Saatchi (2011) introduces an efficient approach to decompose this matrix, which we reformulate in terms of arbitrary matrix square roots. We factor our covariance matrix into a product of terms using Kronecker product algebra
K = Σ λ 1 2 ( Σ λ 1 2 ) T ⊗ Σ t 1 2 ( Σ t 1 2 ) T , (42) \mathbf{K} = \boldsymbol{\Sigma}_{\lambda}^{\frac{1}{2}}(\boldsymbol{\Sigma}_{\lambda}^{\frac{1}{2}})^{T} \otimes\boldsymbol{\Sigma}_{t}^{\frac{1}{2}}(\boldsymbol{\Sigma}_{t}^{\frac{1}{2}})^{T}, \tag*{(42)} K = Σ λ 2 1 ( Σ λ 2 1 ) T ⊗ Σ t 2 1 ( Σ t 2 1 ) T , (42)
= [ Σ λ 1 2 ⊗ Σ t 1 2 ] [ ( Σ λ 1 2 ) T ⊗ ( Σ t 1 2 ) T ] , (43) = [\boldsymbol{\Sigma}_{\lambda}^{\frac{1}{2}} \otimes\boldsymbol{\Sigma}_{t}^{\frac{1}{2}}][(\boldsymbol{\Sigma}_{\lambda}^{\frac{1}{2}})^{T} \otimes(\boldsymbol{\Sigma}_{t}^{\frac{1}{2}})^{T}], \tag*{(43)} = [ Σ λ 2 1 ⊗ Σ t 2 1 ] [( Σ λ 2 1 ) T ⊗ ( Σ t 2 1 ) T ] , (43)
= [ Σ λ 1 2 ⊗ Σ t 1 2 ] [ I N λ ⊗ I N t ] [ ( Σ λ 1 2 ) T ⊗ ( Σ t 1 2 ) T ] , (44) = [\boldsymbol{\Sigma}_{\lambda}^{\frac{1}{2}} \otimes\boldsymbol{\Sigma}_{t}^{\frac{1}{2}}][\mathbf{I}_{N_{\lambda}} \otimes\mathbf{I}_{N_t}][(\boldsymbol{\Sigma}_{\lambda}^{\frac{1}{2}})^{T} \otimes(\boldsymbol{\Sigma}_{t}^{\frac{1}{2}})^{T}], \tag*{(44)} = [ Σ λ 2 1 ⊗ Σ t 2 1 ] [ I N λ ⊗ I N t ] [( Σ λ 2 1 ) T ⊗ ( Σ t 2 1 ) T ] , (44)
where this factorisation includes a redundant identity matrix term labelled K ~ \widetilde{\mathbf{K}} K , which will be a useful concept for later optimisations. We can invert the covariance matrix by inverting the individual terms:
K − 1 = [ ( Σ λ − 1 2 ) T ⊗ ( Σ t − 1 2 ) T ] [ I N λ ⊗ I N t ] [ Σ λ − 1 2 ⊗ Σ t − 1 2 ] , (45) \mathbf{K}^{-1} = [(\boldsymbol{\Sigma}_{\lambda}^{-\frac{1}{2}})^{T} \otimes(\boldsymbol{\Sigma}_{t}^{-\frac{1}{2}})^{T}][\mathbf{I}_{N_{\lambda}} \otimes\mathbf{I}_{N_t}][\boldsymbol{\Sigma}_{\lambda}^{-\frac{1}{2}} \otimes\boldsymbol{\Sigma}_{t}^{-\frac{1}{2}}], \tag*{(45)} K − 1 = [( Σ λ − 2 1 ) T ⊗ ( Σ t − 2 1 ) T ] [ I N λ ⊗ I N t ] [ Σ λ − 2 1 ⊗ Σ t − 2 1 ] , (45)
The Transformed Log-Likelihood
We make use of an idea from Rakitsch et al. (2013), where we interpret this log-likelihood calculation as taking the data residuals r = y − μ \mathbf{r} = \mathbf{y} - \boldsymbol{\mu} r = y − μ , performing the linear transformation r ~ = K t r a n s f − 1 r \widetilde{\mathbf{r}} = \mathbf{K}_{\mathrm{transf}}^{-1}\mathbf{r} r = K transf − 1 r using Equation (39) , and then calculating the standard GP log-likelihood with the transformed residuals r ~ \widetilde{\mathbf{r}} r and transformed covariance matrix K ~ \widetilde{\mathbf{K}} K
log p ( D ∣ θ ) = − 1 2 r T K t r a n s f − T K ~ − 1 K t r a n s f − 1 r − 1 2 log ∣ K t r a n s f K ~ K t r a n s f T ∣ − N 2 log ( 2 π ) , (46) \log p(\mathcal{D} \mid\theta) = -\frac{1}{2}\mathbf{r}^{T}\mathbf{K}_{\mathrm{transf}}^{-T}\widetilde{\mathbf{K}}^{-1}\mathbf{K}_{\mathrm{transf}}^{-1}\mathbf{r} - \frac{1}{2}\log\left|\mathbf{K}_{\mathrm{transf}}\widetilde{\mathbf{K}}\mathbf{K}_{\mathrm{transf}}^{T}\right| - \frac{N}{2}\log(2\pi), \tag*{(46)} log p ( D ∣ θ ) = − 2 1 r T K transf − T K − 1 K transf − 1 r − 2 1 log K transf K K transf T − 2 N log ( 2 π ) , (46)
log p ( D ∣ θ ) = − log ∣ K t r a n s f ∣ + [ − 1 2 r ~ T K ~ − 1 r ~ − 1 2 log ∣ K ~ ∣ − N 2 log ( 2 π ) ] ⏟ transformed log-likelihood , (47) \log p(\mathcal{D} \mid\theta) = -\log\left|\mathbf{K}_{\mathrm{transf}}\right| + \underbrace{\left[-\frac{1}{2}\widetilde{\mathbf{r}}^{T}\widetilde{\mathbf{K}}^{-1}\widetilde{\mathbf{r}} - \frac{1}{2}\log\left|\widetilde{\mathbf{K}}\right| - \frac{N}{2}\log(2\pi)\right]}_{\text{transformed log-likelihood}}, \tag*{(47)} log p ( D ∣ θ ) = − log ∣ K transf ∣ + transformed log-likelihood [ − 2 1 r T K − 1 r − 2 1 log K − 2 N log ( 2 π ) ] , (47)
where we introduce the transformed log-likelihood , which differs from the normal GP log-likelihood by log ∣ K t r a n s f ∣ \log|\mathbf{K}_{\mathrm{transf}}| log ∣ K transf ∣ . For the SeparableKernel, we have
log ∣ K t r a n s f ∣ = N t 2 log ∣ Σ λ ∣ + N λ 2 log ∣ Σ t ∣ (48) \log|\mathbf{K}_{\mathrm{transf}}| = \frac{N_t}{2}\log|\boldsymbol{\Sigma}_{\lambda}| + \frac{N_{\lambda}}{2}\log|\boldsymbol{\Sigma}_{t}| \tag*{(48)} log ∣ K transf ∣ = 2 N t log ∣ Σ λ ∣ + 2 N λ log ∣ Σ t ∣ (48)
and in fact this will be similar for many of the optimisations introduced, since additional orthogonal transformations in K t r a n s f \mathbf{K}_{\mathrm{transf}} K transf will not affect its determinant. For the SeparableKernel, K ~ \widetilde{\mathbf{K}} K is just the identity matrix, so our transformed log-likelihood reduces to a simple least-squares likelihood on the transformed residuals. We can think of the transformation as ‘whitening’ the data residuals; if measurement covariance is present and is modelled accurately, the effect would be to transform the residuals to remove any correlations, resulting in white noise with unit variance.
Computational Scaling
We note that since the transformation of the residuals can be performed with any matrix inverse square roots, CELERITE or quasi-separable covariance matrices can be combined with the Saatchi (2011) approach to achieve a linear runtime scaling. If both Σ λ \boldsymbol{\Sigma}_{\lambda} Σ λ and Σ t \boldsymbol{\Sigma}_{t} Σ t are quasiseparable, it costs O ( N λ J λ 2 + N t J t 2 ) \mathcal{O}(N_{\lambda}J_{\lambda}^{2} + N_tJ_t^{2}) O ( N λ J λ 2 + N t J t 2 ) to decompose both matrices, and O ( N λ N t ( J λ + J t ) ) \mathcal{O}(N_{\lambda}N_t(J_{\lambda} + J_t)) O ( N λ N t ( J λ + J t )) to transform the residuals. As an example, this could be applied for measurement covariance described by Equations (18) and (19) , an approach which is benchmarked in Figure G8 . Even for a large spectroscopic time-series of N λ = 1024 N_{\lambda} = 1024 N λ = 1024 wavelengths and N t = 16384 N_t = 16384 N t = 16384 time points, we recover a log-likelihood runtime of just 27 ms on CPU. For reference, using JAXOPLANET to model transit light curves for the same dataset size took 0.2 seconds on CPU (see Figure G2 ). Our memory scaling is also linear, scaling as O ( N λ J λ + N t J t + N λ N t ) \mathcal{O}(N_{\lambda}J_{\lambda} + N_tJ_t + N_{\lambda}N_t) O ( N λ J λ + N t J t + N λ N t ) .
Figure G2. Transit light curve runtime cost using JAXOPLANET on an M4 MacBook Air, which approaches approximately 10-20 ns per datapoint for larger datasets, more expensive than many of the GP optimisations presented in this work.
Figure G8. Measurement covariance which is in the form of a rank-1 quasiseparable matrix can be efficiently scaled to very large datasets even on a MacBook, as it has a linear scaling with the number of datapoints ∼ O ( N λ J λ 2 + N λ N t J Σ λ ) \sim O(N_{\lambda}J_{\lambda}^{2}+N_{\lambda}N_{t}J_{\Sigma\lambda}) ∼ O ( N λ J λ 2 + N λ N t J Σ λ ) , where J Σ λ J_{\Sigma\lambda} J Σ λ is the quasiseparable rank of the measurement covariance matrix.
If we instead have arbitrary matrices Σ λ \boldsymbol{\Sigma}_{\lambda} Σ λ and Σ t \boldsymbol{\Sigma}_{t} Σ t , then we can simply use the existing Saatchi (2011) approach with general Cholesky decomposition applied to both matrices. If we are just accounting for white noise and measurement covariance, then Σ t \boldsymbol{\Sigma}_{t} Σ t must be diagonal and is therefore trivial to decompose. In this case, we get a runtime scaling of O ( N λ 3 + N t N λ 2 ) \mathcal{O}(N_{\lambda}^{3} + N_tN_{\lambda}^{2}) O ( N λ 3 + N t N λ 2 ) and memory scaling of O ( N λ 2 + N λ N t ) \mathcal{O}(N_{\lambda}^{2} + N_{\lambda}N_t) O ( N λ 2 + N λ N t ) for any arbitrary measurement covariance Σ λ \boldsymbol{\Sigma}_{\lambda} Σ λ . The log-likelihood calculation for a ( N λ , N t ) = ( 1024 , 16384 ) (N_{\lambda},N_t) = (1024,16384) ( N λ , N t ) = ( 1024 , 16384 ) dataset took 0.1 seconds with this model, see Figure G9 for more benchmarking.
Figure G9. For an arbitrary measurement covariance matrix, the runtime scaling is O ( N λ 3 + N λ 2 N t ) O(N_{\lambda}^{3}+N_{\lambda}^{2}N_{t}) O ( N λ 3 + N λ 2 N t ) with the SEPARABLEKERNEL, so it runs slower for a large number of wavelengths relative to Figure G8 with quasiseparable measurement covariance.
LUASKernel: Measurement Covariance Plus a 2D Kernel Term
Rakitsch et al. (2013) identified an approach used within computational biology, which can fit any kernel function of the form
k ( λ , λ ′ , t , t ′ ) = s λ ( λ , λ ′ ) s t ( t , t ′ ) + k λ ( λ , λ ′ ) k t ( t , t ′ ) (49) k(\lambda,\lambda',t,t') = s_{\lambda}(\lambda,\lambda')s_t(t,t') + k_{\lambda}(\lambda,\lambda')k_t(t,t') \tag*{(49)} k ( λ , λ ′ , t , t ′ ) = s λ ( λ , λ ′ ) s t ( t , t ′ ) + k λ ( λ , λ ′ ) k t ( t , t ′ ) (49)
on a rectangular dataset, which corresponds to a covariance matrix which is a sum of two Kronecker products
K = Σ λ ⊗ Σ t + K λ ⊗ K t . (50) \mathbf{K} = \boldsymbol{\Sigma}_{\lambda} \otimes\boldsymbol{\Sigma}_t + \mathbf{K}_{\lambda} \otimes\mathbf{K}_t. \tag*{(50)} K = Σ λ ⊗ Σ t + K λ ⊗ K t . (50)
This increase in complexity in allowing two separate time covariance matrices is extremely useful for simultaneously accounting for effects which are independent in time, such as white noise and measurement covariance, in addition to various sources of time-correlated systematics. This approach was adapted and applied to transit spectroscopy in Fortney et al. (2024), motivating the creation of the original LUAS package, which is why we label it as the LUASKernel. The optimisation works by performing simultaneous diagonalisation on pairs of matrices, which we first describe in general terms.
Simultaneous Diagonalisation
Suppose we have two symmetric matrices Σ \mathbf{\Sigma} Σ and K \mathbf{K} K , where at least Σ \mathbf{\Sigma} Σ is positive definite. Can we perform the following transformation
Σ + K = S [ D Σ + D K ] S T (51) \mathbf{\Sigma}+\mathbf{K}=\mathbf{S}[\mathbf{D}_{\Sigma}+\mathbf{D}_{K}]\mathbf{S}^{T} \tag*{(51)} Σ + K = S [ D Σ + D K ] S T (51)
using some invertible matrix S \mathbf{S} S , such that both D Σ \mathbf{D}_{\Sigma} D Σ and D K \mathbf{D}_{K} D K are diagonal?
In fact, we can always construct S \mathbf{S} S . First, we use a matrix inverse square root of Σ \mathbf{\Sigma} Σ to transform K \mathbf{K} K as follows:
K ′ = Σ − 1 2 K ( Σ − 1 2 ) T (52) \mathbf{K}^{\prime}=\mathbf{\Sigma}^{-\frac{1}{2}}\mathbf{K}(\mathbf{\Sigma}^{-\frac{1}{2}})^{T} \tag*{(52)} K ′ = Σ − 2 1 K ( Σ − 2 1 ) T (52)
where Σ − 1 2 \mathbf{\Sigma}^{-\frac{1}{2}} Σ − 2 1 must exist because Σ \mathbf{\Sigma} Σ is positive definite. This is followed by computing the eigendecomposition K ′ = Q K ′ Λ K ′ Q K ′ T \mathbf{K}^{\prime}=\mathbf{Q}_{K^{\prime}}\mathbf{\Lambda}_{K^{\prime}}\mathbf{Q}_{K^{\prime}}^{T} K ′ = Q K ′ Λ K ′ Q K ′ T , which doesn’t require K ′ \mathbf{K}^{\prime} K ′ to be positive definite. S \mathbf{S} S is then calculated as S = Σ 1 2 Q K ′ \mathbf{S}=\mathbf{\Sigma}^{\frac{1}{2}}\mathbf{Q}_{K^{\prime}} S = Σ 2 1 Q K ′ , which we can verify solves the problem for D Σ = I N \mathbf{D}_{\Sigma}=\mathbf{I}_{N} D Σ = I N and D K = Λ K ′ \mathbf{D}_{K}=\mathbf{\Lambda}_{K^{\prime}} D K = Λ K ′
Σ + K = Σ 1 2 Q K ′ [ I N + Λ K ′ ] ( Σ 1 2 Q K ′ ) T (53) \mathbf{\Sigma}+\mathbf{K}=\mathbf{\Sigma}^{\frac{1}{2}}\mathbf{Q}_{K^{\prime}}[\mathbf{I}_{N}+\mathbf{\Lambda}_{K^{\prime}}](\mathbf{\Sigma}^{\frac{1}{2}}\mathbf{Q}_{K^{\prime}})^{T} \tag*{(53)} Σ + K = Σ 2 1 Q K ′ [ I N + Λ K ′ ] ( Σ 2 1 Q K ′ ) T (53)
= Σ 1 2 Q K ′ Q K ′ T ( Σ 1 2 ) T + Σ 1 2 Q K ′ Λ K ′ Q K ′ T ( Σ 1 2 ) T (54) =\mathbf{\Sigma}^{\frac{1}{2}}\mathbf{Q}_{K^{\prime}}\mathbf{Q}_{K^{\prime}}^{T}(\mathbf{\Sigma}^{\frac{1}{2}})^{T}+\mathbf{\Sigma}^{\frac{1}{2}}\mathbf{Q}_{K^{\prime}}\mathbf{\Lambda}_{K^{\prime}}\mathbf{Q}_{K^{\prime}}^{T}(\mathbf{\Sigma}^{\frac{1}{2}})^{T} \tag*{(54)} = Σ 2 1 Q K ′ Q K ′ T ( Σ 2 1 ) T + Σ 2 1 Q K ′ Λ K ′ Q K ′ T ( Σ 2 1 ) T (54)
= Σ 1 2 ( Σ 1 2 ) T + Σ 1 2 K ′ ( Σ 1 2 ) T (55) =\mathbf{\Sigma}^{\frac{1}{2}}(\mathbf{\Sigma}^{\frac{1}{2}})^{T}+\mathbf{\Sigma}^{\frac{1}{2}}\mathbf{K}^{\prime}(\mathbf{\Sigma}^{\frac{1}{2}})^{T} \tag*{(55)} = Σ 2 1 ( Σ 2 1 ) T + Σ 2 1 K ′ ( Σ 2 1 ) T (55)
= Σ + K (56) =\mathbf{\Sigma}+\mathbf{K} \tag*{(56)} = Σ + K (56)
We see that simultaneous diagonalisation can always be performed on two symmetric matrices, provided that at least one of them is positive definite [65 ] . In the case that K \mathbf{K} K is positive semi-definite, then K ′ \mathbf{K}^{\prime} K ′ will also be positive semi-definite, which will simply result in some of the eigenvalues in Λ K ′ \mathbf{\Lambda}_{K^{\prime}} Λ K ′ equalling zero.
The LUASKERNEL Factorisation
The method in Rakitsch et al. [98 ] performs a simultaneous diagonalisation in both dimensions. We use the inverse matrix square roots of Σ λ \mathbf{\Sigma}_{\lambda} Σ λ and Σ t \mathbf{\Sigma}_{t} Σ t to perform the transformations
K λ ′ = Σ λ − 1 2 K λ ( Σ λ − 1 2 ) T , (57) \mathbf{K}^{\prime}_{\lambda}=\mathbf{\Sigma}_{\lambda}^{-\frac{1}{2}}\mathbf{K}_{\lambda}(\mathbf{\Sigma}_{\lambda}^{-\frac{1}{2}})^{T}, \tag*{(57)} K λ ′ = Σ λ − 2 1 K λ ( Σ λ − 2 1 ) T , (57)
K t ′ = Σ t − 1 2 K t ( Σ t − 1 2 ) T , (58) \mathbf{K}^{\prime}_{t}=\mathbf{\Sigma}_{t}^{-\frac{1}{2}}\mathbf{K}_{t}(\mathbf{\Sigma}_{t}^{-\frac{1}{2}})^{T}, \tag*{(58)} K t ′ = Σ t − 2 1 K t ( Σ t − 2 1 ) T , (58)
where we’ve assumed Σ λ \mathbf{\Sigma}_{\lambda} Σ λ and Σ t \mathbf{\Sigma}_{t} Σ t are both positive definite to do this, while K λ \mathbf{K}_{\lambda} K λ and K t \mathbf{K}_{t} K t can either be positive semi-definite or positive definite.
Next, we calculate the eigendecompositions of these transformed covariance matrices:
K λ ′ = Q K λ ′ Λ K λ ′ Q K λ ′ T (59) \mathbf{K}^{\prime}_{\lambda}=\mathbf{Q}_{K^{\prime}_{\lambda}}\mathbf{\Lambda}_{K^{\prime}_{\lambda}}\mathbf{Q}_{K^{\prime}_{\lambda}}^{T} \tag*{(59)} K λ ′ = Q K λ ′ Λ K λ ′ Q K λ ′ T (59)
K t ′ = Q K t ′ Λ K t ′ Q K t ′ T (60) \mathbf{K}^{\prime}_{t}=\mathbf{Q}_{K^{\prime}_{t}}\mathbf{\Lambda}_{K^{\prime}_{t}}\mathbf{Q}_{K^{\prime}_{t}}^{T} \tag*{(60)} K t ′ = Q K t ′ Λ K t ′ Q K t ′ T (60)
and we can then factorise the overall covariance matrix as
K = Σ λ ⊗ Σ t + K λ ⊗ K t = [ Σ λ 1 2 Q K λ ′ ⊗ Σ t 1 2 Q K t ′ ] ⏟ K t r a n s f [ I N λ ⊗ I N t + Λ K λ ′ ⊗ Λ K t ′ ] ⏟ K ~ ∗ [ Q K λ ′ T ( Σ λ 1 2 ) T ⊗ Q K t ′ T ( Σ t 1 2 ) T ] ⏟ K t r a n s f T . (61) \begin{aligned}
\mathbf{K}=\mathbf{\Sigma}_{\lambda}\otimes\mathbf{\Sigma}_{t}+\mathbf{K}_{\lambda}\otimes\mathbf{K}_{t} \\
&=\underbrace{[\mathbf{\Sigma}_{\lambda}^{\frac{1}{2}}\mathbf{Q}_{K^{\prime}_{\lambda}}\otimes\mathbf{\Sigma}_{t}^{\frac{1}{2}}\mathbf{Q}_{K^{\prime}_{t}}]}_{\mathbf{K}_{\mathrm{transf}}}\underbrace{[\mathbf{I}_{N_{\lambda}}\otimes\mathbf{I}_{N_{t}}+\mathbf{\Lambda}_{K^{\prime}_{\lambda}}\otimes\mathbf{\Lambda}_{K^{\prime}_{t}}]}_{\widetilde{\mathbf{K}}}^{*}\underbrace{[\mathbf{Q}_{K^{\prime}_{\lambda}}^{T}(\mathbf{\Sigma}_{\lambda}^{\frac{1}{2}})^{T}\otimes\mathbf{Q}_{K^{\prime}_{t}}^{T}(\mathbf{\Sigma}_{t}^{\frac{1}{2}})^{T}]}_{\mathbf{K}_{\mathrm{transf}}^{T}}. \tag*{(61)}
\end{aligned} K = Σ λ ⊗ Σ t + K λ ⊗ K t = K transf [ Σ λ 2 1 Q K λ ′ ⊗ Σ t 2 1 Q K t ′ ] ∗ [ I N λ ⊗ I N t + Λ K λ ′ ⊗ Λ K t ′ ] K transf T [ Q K λ ′ T ( Σ λ 2 1 ) T ⊗ Q K t ′ T ( Σ t 2 1 ) T ] . (61)
Similar to the SEPARABLEKERNEL, we can calculate our log-likelihood by first transforming our residuals vector as
r ~ = K t r a n s f − 1 r = [ Q K λ ′ T Σ λ − 1 2 ⊗ Q K t ′ T Σ t − 1 2 ] r , (62) \widetilde{\mathbf{r}}=\mathbf{K}_{\mathrm{transf}}^{-1}\mathbf{r}=[\mathbf{Q}_{K^{\prime}_{\lambda}}^{T}\mathbf{\Sigma}_{\lambda}^{-\frac{1}{2}}\otimes\mathbf{Q}_{K^{\prime}_{t}}^{T}\mathbf{\Sigma}_{t}^{-\frac{1}{2}}]\mathbf{r}, \tag*{(62)} r = K transf − 1 r = [ Q K λ ′ T Σ λ − 2 1 ⊗ Q K t ′ T Σ t − 2 1 ] r , (62)
Computational Scaling
The LUASKERNEL has a runtime scaling of O ( N λ 3 + N t 3 + N λ N t ( N λ + N t ) ) \mathcal{O}(N_{\lambda}^{3}+N_{t}^{3}+N_{\lambda}N_{t}(N_{\lambda}+N_{t})) O ( N λ 3 + N t 3 + N λ N t ( N λ + N t )) , and a memory scaling of O ( N λ 2 + N t 2 + N λ N t ) \mathcal{O}(N_{\lambda}^{2}+N_{t}^{2}+N_{\lambda}N_{t}) O ( N λ 2 + N t 2 + N λ N t ) . While it has a nice level of flexibility, capable of being used with any valid kernel functions, the need to eigendecompose two matrices of shape ( N λ , N λ ) (N_{\lambda},N_{\lambda}) ( N λ , N λ ) and ( N t , N t ) (N_{t},N_{t}) ( N t , N t ) can be a major limitation, restricting us from easily scaling beyond ∼ 1000 − 4000 \sim1000-4000 ∼ 1000 − 4000 points in either dimension. This can be seen from the benchmarking in Figure G10 and Figure G11 for CPU and GPU respectively. As we highlight in Figure 1, eigendecomposition can be significantly slower than general Cholesky decomposition - despite both scaling as O ( N 3 ) \mathcal{O}(N^{3}) O ( N 3 ) - while Cholesky decompositions using CELERITE and quasiseparable matrices are dramatically faster. Thus, the LUASKERNEL is typically bottlenecked by the eigendecomposition of the larger of the two covariance matrices.
Figure G10. With a runtime scaling of O ( N λ 3 + N t 3 + N λ N t ( N λ + N t ) ) \mathcal{O}(N_{\lambda}^{3} + N_{t}^{3} + N_{\lambda}N_{t}(N_{\lambda} + N_{t})) O ( N λ 3 + N t 3 + N λ N t ( N λ + N t )) , the runtime cost of the LUASKERNEL is dominated by the eigendecomposition of covariance matrices in both dimensions. By comparing with Figure 1, we can note that the top row of runtimes is only marginally larger than the cost of eigendecomposition of the time covariance matrix alone, and this remains the bottleneck even for hundreds of wavelengths.
Figure G11. Since eigendecomposition can be performed faster on GPU (see Figure 1), the LUASKERNEL can be scaled to larger datasets than on CPU (shown in Figure G10 ). We see that the GPU runtimes are often significantly faster, excluding the smallest datasets tested.
Compared with Rakitsch et al. [98 ] , the only further optimisation we introduce with the LUASKERNEL is that we generalise to any matrix square root to decompose Σ λ \mathbf{\Sigma}_{\lambda} Σ λ and Σ t \mathbf{\Sigma}_{t} Σ t (e.g. Cholesky decomposition, CELERITE), rather than specifically using eigendecomposition, which can roughly halve the runtime cost.
LUASLASHKERNEL: Extending 1D GP Optimisations to 2D
To avoid the cost of eigendecomposition in both dimensions, an alternative approach is to perform the simultaneous diagonalisation in just one of the dimensions:
K = Σ λ ⊗ Σ t + K λ ⊗ K t , (63) \mathbf{K}=\mathbf{\Sigma}_{\lambda}\otimes\mathbf{\Sigma}_{t}+\mathbf{K}_{\lambda}\otimes\mathbf{K}_{t}, \tag*{(63)} K = Σ λ ⊗ Σ t + K λ ⊗ K t , (63)
K = [ Σ λ 1 2 Q K λ ′ ⊗ I N t ] ⏟ K t r a n s f [ I N λ ⊗ Σ t + Λ K λ ′ ⊗ K t ] ⏟ K ~ [ Q K λ ′ T ( Σ λ 1 2 ) T ⊗ I N t ] , (64) \mathbf{K}=\underbrace{[\mathbf{\Sigma}_{\lambda}^{\frac{1}{2}}\mathbf{Q}_{K^{\prime}_{\lambda}}\otimes\mathbf{I}_{N_{t}}]}_{\mathbf{K}_{\mathrm{transf}}}\underbrace{[\mathbf{I}_{N_{\lambda}}\otimes\mathbf{\Sigma}_{t}+\mathbf{\Lambda}_{K^{\prime}_{\lambda}}\otimes\mathbf{K}_{t}]}_{\widetilde{\mathbf{K}}}[\mathbf{Q}_{K^{\prime}_{\lambda}}^{T}(\mathbf{\Sigma}_{\lambda}^{\frac{1}{2}})^{T}\otimes\mathbf{I}_{N_{t}}], \tag*{(64)} K = K transf [ Σ λ 2 1 Q K λ ′ ⊗ I N t ] K [ I N λ ⊗ Σ t + Λ K λ ′ ⊗ K t ] [ Q K λ ′ T ( Σ λ 2 1 ) T ⊗ I N t ] , (64)
The transformed covariance matrix K ~ \widetilde{\mathbf{K}} K is now block-diagonal
K ~ = [ Σ t + λ K λ ′ ; 1 K t 0 ⋯ 0 0 Σ t + λ K λ ′ ; 2 K t ⋯ 0 ⋮ ⋮ ⋱ ⋮ 0 0 ⋯ Σ t + λ K λ ′ ; N λ K t ] , (65) \widetilde{\mathbf{K}}=
\begin{bmatrix}
\mathbf{\Sigma}_{t}+\lambda_{K^{\prime}_{\lambda};1}\mathbf{K}_{t} & 0 & \cdots& 0 \\
0 & \mathbf{\Sigma}_{t}+\lambda_{K^{\prime}_{\lambda};2}\mathbf{K}_{t} & \cdots& 0 \\
\vdots& \vdots& \ddots& \vdots\\
0 & 0 & \cdots& \mathbf{\Sigma}_{t}+\lambda_{K^{\prime}_{\lambda};N_{\lambda}}\mathbf{K}_{t}
\end{bmatrix}, \tag*{(65)} K = Σ t + λ K λ ′ ; 1 K t 0 ⋮ 0 0 Σ t + λ K λ ′ ; 2 K t ⋮ 0 ⋯ ⋯ ⋱ ⋯ 0 0 ⋮ Σ t + λ K λ ′ ; N λ K t , (65)
which means our transformed log-likelihood is equivalent to separately decomposing each light curve in parallel
log p ( r ∣ K ) = − log ∣ K t r a n s f ∣ + ∑ i = 1 N λ [ − 1 2 r ~ i T ( Σ t + λ K λ ′ ; i K t ) − 1 r ~ i − 1 2 log ∣ Σ t + λ K λ ′ ; i K t ∣ − N t 2 log ( 2 π ) ] , (66) \log p(\mathbf{r} \mid\mathbf{K}) = -\log\lvert\mathbf{K}_{\mathrm{transf}} \rvert+ \sum_{i=1}^{N_{\lambda}} \left[ -\frac{1}{2}\tilde{\mathbf{r}}_{i}^{T}\left(\mathbf{\Sigma}_{t}+\lambda_{\mathbf{K}'_{\lambda};i}\mathbf{K}_{t}\right)^{-1}\tilde{\mathbf{r}}_{i} -\frac{1}{2}\log\left\lvert\mathbf{\Sigma}_{t}+\lambda_{\mathbf{K}'_{\lambda};i}\mathbf{K}_{t}\right\rvert-\frac{N_{t}}{2}\log(2\pi) \right], \tag*{(66)} log p ( r ∣ K ) = − log ∣ K transf ∣ + i = 1 ∑ N λ [ − 2 1 r ~ i T ( Σ t + λ K λ ′ ; i K t ) − 1 r ~ i − 2 1 log Σ t + λ K λ ′ ; i K t − 2 N t log ( 2 π ) ] , (66)
log p ( r ∣ K ) = − N t log ∣ Σ λ 1 2 ∣ + ∑ i = 1 N λ log p ( r ~ i ∣ Σ t + λ K λ ′ ; i K t ) , (67) \log p(\mathbf{r} \mid\mathbf{K}) = -N_{t}\log\left\lvert\mathbf{\Sigma}_{\lambda}^{\frac{1}{2}}\right\rvert+ \sum_{i=1}^{N_{\lambda}}\log p\left(\tilde{\mathbf{r}}_{i}\mid\mathbf{\Sigma}_{t}+\lambda_{\mathbf{K}'_{\lambda};i}\mathbf{K}_{t}\right), \tag*{(67)} log p ( r ∣ K ) = − N t log Σ λ 2 1 + i = 1 ∑ N λ log p ( r ~ i ∣ Σ t + λ K λ ′ ; i K t ) , (67)
where we’ve labelled the i th i^{\text{th}} i th transformed light curve as r ~ i \tilde{\mathbf{r}}_{i} r ~ i . Transforming the residuals with K t r a n s f \mathbf{K}_{\mathrm{transf}} K transf therefore decouples a dense 2D GP into a sum of independent 1D GPs. The covariance matrices K t \mathbf{K}_{t} K t and Σ t \mathbf{\Sigma}_{t} Σ t are completely unmodified by this transformation, with the exception that K t \mathbf{K}_{t} K t gets multiplied by a scalar and is then added to Σ t \mathbf{\Sigma}_{t} Σ t (which is often diagonal). Therefore, a wide range of 1D GP optimisations will be valid to decompose each block, including CELERITE, quasiseparable kernels, and state-space models, as well as approximate methods like HODLR and inducing point methods.
Computational Scaling
Our transformation cost with K t r a n s f − 1 \mathbf{K}_{\mathrm{transf}}^{-1} K transf − 1 is now O ( N λ 3 + N λ 2 N t ) \mathcal{O}(N_{\lambda}^{3}+N_{\lambda}^{2}N_{t}) O ( N λ 3 + N λ 2 N t ) including decomposition, avoiding the O ( N t 3 ) \mathcal{O}(N_{t}^{3}) O ( N t 3 ) eigendecomposition of the time covariance matrix. Instead, we may need to perform as many as N λ N_{\lambda} N λ separate ( N t , N t ) (N_{t},N_{t}) ( N t , N t ) matrix decompositions, one for every non-zero eigenvalue λ K λ ′ ; i \lambda_{\mathbf{K}'_{\lambda};i} λ K λ ′ ; i , which naively should scale even worse. However, we are not limited to eigendecomposition, so we can decompose using faster methods, and each of these decompositions can be performed in parallel.
Suppose that K λ \mathbf{K}_{\lambda} K λ has R λ R_{\lambda} R λ non-zero eigenvalues, then the transformed matrix K λ ′ \mathbf{K}'_{\lambda} K λ ′ will have the same number (since they are congruent). If the runtime cost of each decomposition is O ( C t ) \mathcal{O}(C_{t}) O ( C t ) , then the decomposition of all blocks should scale linearly as O ( R λ C t ) \mathcal{O}(R_{\lambda}C_{t}) O ( R λ C t ) . For example, if we have a 2D kernel term K λ ⊗ K t \mathbf{K}_{\lambda}\otimes\mathbf{K}_{t} K λ ⊗ K t with a full-rank wavelength covariance matrix K λ \mathbf{K}_{\lambda} K λ (i.e. R λ = N λ R_{\lambda}=N_{\lambda} R λ = N λ ), and a quasiseparable rank of J K t J_{\mathbf{K}_{t}} J K t for the time covariance matrix, then we get a scaling of O ( N λ N t J K t 2 ) \mathcal{O}(N_{\lambda}N_{t}J_{\mathbf{K}_{t}}^{2}) O ( N λ N t J K t 2 ) for the decomposition of all blocks, linear in the number of datapoints. Including the cost of transforming the residuals with any arbitrary measurement covariance matrix Σ λ \mathbf{\Sigma}_{\lambda} Σ λ , we get a total runtime scaling of O ( N λ 3 + N λ 2 N t + N λ N t J K t 2 ) \mathcal{O}(N_{\lambda}^{3}+N_{\lambda}^{2}N_{t}+N_{\lambda}N_{t}J_{\mathbf{K}_{t}}^{2}) O ( N λ 3 + N λ 2 N t + N λ N t J K t 2 ) . The memory cost is only ∼ O ( N λ 2 + N λ N t J K t ) \sim\mathcal{O}(N_{\lambda}^{2}+N_{\lambda}N_{t}J_{\mathbf{K}_{t}}) ∼ O ( N λ 2 + N λ N t J K t ) .
Note that we can also parallelise the transformation of the residuals and the decomposition of each block, so for T ≤ min ( N λ , N t ) T\leq\min(N_{\lambda},N_{t}) T ≤ min ( N λ , N t ) parallel workers, the runtime scaling can reduce to O ( N λ 2 + N λ N t ( J Σ λ + J K t 2 ) T ) \mathcal{O}\left(N_{\lambda}^{2}+\frac{N_{\lambda}N_{t}(J_{\mathbf{\Sigma}_{\lambda}}+J_{\mathbf{K}_{t}}^{2})}{T}\right) O ( N λ 2 + T N λ N t ( J Σ λ + J K t 2 ) ) , which suggests that GPUs should be particularly fast for this optimisation. See Figure G12 and Figure G13 for log-likelihood runtimes comparing CPU and GPU runtimes (for arbitrary Σ λ \mathbf{\Sigma}_{\lambda} Σ λ ), on CPU the performance can be comparable to transit light curve modelling (Figure G2 ), and was always faster when N λ ≤ 64 N_{\lambda}\leq64 N λ ≤ 64 and N t ≥ 1024 N_{t}\geq1024 N t ≥ 1024 . The GPU is faster than the CPU for larger datasets, and achieves a 61 ms log-likelihood runtime for ( N λ , N t ) = ( 1024 , 16384 ) (N_{\lambda},N_{t})=(1024,16384) ( N λ , N t ) = ( 1024 , 16384 ) .
Figure G12. In comparison to the LUASKERNEL in Figure G10 , the LUASLASRACHKERNEL can achieve dramatically better scaling in N t N_t N t when the time kernel is quasiseparable, with a runtime scaling of O ( N λ 3 + N λ 2 N t / T + N λ N t K T / T ) \mathcal{O}(N_{\lambda}^{3} + N_{\lambda}^{2}N_{t}/T + N_{\lambda}N_{t}K_{T}/T) O ( N λ 3 + N λ 2 N t / T + N λ N t K T / T ) for T T T parallel workers. As we still must perform an eigendecomposition of a wavelength covariance matrix, runtimes for high N λ N_{\lambda} N λ and low N t N_{t} N t are similar to the LUASKERNEL.
Figure G13. Compared to Figure G12 , running the LUASLASRACHKERNEL on GPU can also perform significantly faster on large datasets, as GPUs optimise the eigendecomposition of the wavelength covariance matrix, as well as parallelising the decomposition with many parallel workers T T T .
Note that this is not just limited to CELERITE and other quasiseparable kernels, even approximations like HODLR (Ambikasaran et al. [7 ] ) could be used to extend to a more general range of time kernels, later discussed in § 6.10 . In fact, due to the difference in runtime cost between eigendecomposition and general Cholesky decomposition (Figure 1), the LUASLASRACHKERNEL with general Cholesky decompositions of R λ R_{\lambda} R λ time covariance matrices can outperform a single eigendecomposition using the LUASKERNEL when R λ R_{\lambda} R λ is small. Due to the similar flexibility of this optimisation to the LUASKERNEL, named after the Irish word for speed ‘luas’, we refer to this optimisation as the LUASLASRACHKERNEL, from the Irish ‘luas lasrach’3 , meaning ‘lightning speed’.
MultiLowRankKernel: Summing Low-Rank Kernel Terms
Suppose we have a combination of multiple low-rank kernel terms, potentially in the presence of white noise or measurement covariance
k ( λ , λ ′ , t , t ′ ) = s λ ( λ , λ ′ ) σ 2 ( t ) δ t t ′ + ∑ i = 1 N α α i ( λ ) α i ( λ ′ ) k t i ( t , t ′ ) (68) k(\lambda,\lambda',t,t')=s_{\lambda}(\lambda,\lambda')\sigma^{2}(t)\delta_{tt'}+\sum_{i=1}^{N_{\alpha}}\alpha_{i}(\lambda)\alpha_{i}(\lambda')k_{t_{i}}(t,t') \tag*{(68)} k ( λ , λ ′ , t , t ′ ) = s λ ( λ , λ ′ ) σ 2 ( t ) δ t t ′ + i = 1 ∑ N α α i ( λ ) α i ( λ ′ ) k t i ( t , t ′ ) (68)
corresponding to the covariance matrix
K = Σ λ ⊗ D t + ∑ i = 1 N α α i α i T ⊗ K t ; i (69) \mathbf{K}=\mathbf{\Sigma}_{\lambda}\otimes\mathbf{D}_{t}+\sum_{i=1}^{N_{\alpha}}\alpha_{i}\alpha_{i}^{T}\otimes\mathbf{K}_{t;i} \tag*{(69)} K = Σ λ ⊗ D t + i = 1 ∑ N α α i α i T ⊗ K t ; i (69)
for a rectangular dataset.
As described in § 2.7 , Gordon et al. (2020) already introduced an optimisation for this kernel in the case where Σ λ \mathbf{\Sigma}_{\lambda} Σ λ is diagonal and k t i k_{t_{i}} k t i are CELERITE kernels. This is a particularly important kernel to optimise, which we focus on extensively in § 6 , as it can be viewed as a generalisation of PCA/SysRem, it is a physically-motivated model of stellar granulation (Gordon et al. [53 ] ; Morris et al. [89 ] ), can model common-mode systematics, and this model is also used within machine learning, where it may be referred to as the linear mixing model. To understand how we implement the MULTILOWRANKKERNEL, we begin with the simplest case of N α = 1 N_{\alpha}=1 N α = 1 .
The Case of N α = 1 N_{\alpha}=1 N α = 1
We can already decompose N α = 1 N_{\alpha}=1 N α = 1 , as it is a special case of the LUASLASRACHKERNEL with K λ = α λ α λ T \mathbf{K}_{\lambda}=\alpha_{\lambda}\alpha_{\lambda}^{T} K λ = α λ α λ T . First we factor out Σ λ \mathbf{\Sigma}_{\lambda} Σ λ
K = Σ λ ⊗ D t + α λ α λ T ⊗ K t (70) \mathbf{K}=\mathbf{\Sigma}_{\lambda}\otimes\mathbf{D}_{t}+\alpha_{\lambda}\alpha_{\lambda}^{T}\otimes\mathbf{K}_{t} \tag*{(70)} K = Σ λ ⊗ D t + α λ α λ T ⊗ K t (70)
K = [ Σ λ 1 2 ⊗ I N t ] [ I N λ ⊗ D t + α λ ′ α λ ′ T ⊗ K t ] [ Σ λ 1 2 ] T ⊗ I N t (71) \mathbf{K}=\left[\mathbf{\Sigma}_{\lambda}^{\frac{1}{2}}\otimes\mathbf{I}_{N_{t}}\right]\left[\mathbf{I}_{N_{\lambda}}\otimes\mathbf{D}_{t}+\alpha'_{\lambda}\alpha_{\lambda}^{\prime T}\otimes\mathbf{K}_{t}\right]\left[\mathbf{\Sigma}_{\lambda}^{\frac{1}{2}}\right]^{T}\otimes\mathbf{I}_{N_{t}} \tag*{(71)} K = [ Σ λ 2 1 ⊗ I N t ] [ I N λ ⊗ D t + α λ ′ α λ ′ T ⊗ K t ] [ Σ λ 2 1 ] T ⊗ I N t (71)
where we define the whitened wavelength component α λ ′ = Σ λ − 1 2 α λ \alpha'_{\lambda}=\mathbf{\Sigma}_{\lambda}^{-\frac{1}{2}}\alpha_{\lambda} α λ ′ = Σ λ − 2 1 α λ , which is applying the same initial simultaneous diagonalisation step as before K λ ′ = Σ λ − 1 2 K λ ( Σ λ − 1 2 ) T \mathbf{K}'_{\lambda}=\mathbf{\Sigma}_{\lambda}^{-\frac{1}{2}}\mathbf{K}_{\lambda}\left(\mathbf{\Sigma}_{\lambda}^{-\frac{1}{2}}\right)^{T} K λ ′ = Σ λ − 2 1 K λ ( Σ λ − 2 1 ) T .
However, instead of performing a general O ( N λ 3 ) \mathcal{O}(N_{\lambda}^{3}) O ( N λ 3 ) eigendecomposition on K λ ′ = α λ ′ α λ ′ T \mathbf{K}'_{\lambda}=\alpha'_{\lambda}\alpha_{\lambda}^{\prime T} K λ ′ = α λ ′ α λ ′ T , a valid eigendecomposition can be calculated using the Householder matrix
H α λ ′ = I N λ − 2 u ^ u ^ T (72) \mathbf{H}_{\alpha'_{\lambda}}=\mathbf{I}_{N_{\lambda}}-2\hat{\mathbf{u}}\hat{\mathbf{u}}^{T} \tag*{(72)} H α λ ′ = I N λ − 2 u ^ u ^ T (72)
where u ^ \hat{\mathbf{u}} u ^ is calculated as
u = α λ ′ ∣ α λ ′ ∣ ± e ^ i (73) \mathbf{u}=\frac{\alpha'_{\lambda}}{\lvert\alpha'_{\lambda}\rvert}\pm\hat{\mathbf{e}}_{i} \tag*{(73)} u = ∣ α λ ′ ∣ α λ ′ ± e ^ i (73)
u ^ = u ∣ u ∣ (74) \hat{\mathbf{u}}=\frac{\mathbf{u}}{\lvert\mathbf{u}\rvert} \tag*{(74)} u ^ = ∣ u ∣ u (74)
where we are free to choose the sign of ± \pm ± to maximise numerical stability (e.g. if α λ ′ = e ^ i \alpha'_{\lambda}=\hat{\mathbf{e}}_{i} α λ ′ = e ^ i we use this sign choice to avoid u = 0 \mathbf{u}=0 u = 0 ).
This Householder matrix has a number of useful properties, which include:
H α λ ′ e ^ i = ∓ α λ ′ ∣ α λ ′ ∣ (75) \mathbf{H}_{\alpha'_{\lambda}}\hat{\mathbf{e}}_i = \mp\frac{\boldsymbol{\alpha}'_{\lambda}}{\lvert\boldsymbol{\alpha}'_{\lambda}\rvert} \tag*{(75)} H α λ ′ e ^ i = ∓ ∣ α λ ′ ∣ α λ ′ (75)
H α λ ′ H α λ ′ T = I N λ (76) \mathbf{H}_{\alpha'_{\lambda}}\mathbf{H}_{\alpha'_{\lambda}}^{T} = \mathbf{I}_{N_{\lambda}} \tag*{(76)} H α λ ′ H α λ ′ T = I N λ (76)
α λ ′ α λ ′ T = H α λ ′ [ ∣ α λ ′ ∣ 2 e ^ i e ^ i T ] H α λ ′ T (77) \boldsymbol{\alpha}'_{\lambda}\boldsymbol{\alpha}_{\lambda}^{\prime T} = \mathbf{H}_{\alpha'_{\lambda}}\left[\lvert\boldsymbol{\alpha}'_{\lambda}\rvert^2\hat{\mathbf{e}}_i\hat{\mathbf{e}}_i^T\right]\mathbf{H}_{\alpha'_{\lambda}}^T \tag*{(77)} α λ ′ α λ ′ T = H α λ ′ [ ∣ α λ ′ ∣ 2 e ^ i e ^ i T ] H α λ ′ T (77)
So H α λ ′ \mathbf{H}_{\alpha'_{\lambda}} H α λ ′ maps the basis vector e ^ i \hat{\mathbf{e}}_i e ^ i to our desired transformed vector α λ ′ \boldsymbol{\alpha}'_{\lambda} α λ ′ (potentially with a sign flip), H α λ ′ \mathbf{H}_{\alpha'_{\lambda}} H α λ ′ is unitary, and it diagonalises α λ ′ α λ ′ T \boldsymbol{\alpha}'_{\lambda}\boldsymbol{\alpha}_{\lambda}^{\prime T} α λ ′ α λ ′ T . Since H α λ ′ \mathbf{H}_{\alpha'_{\lambda}} H α λ ′ is just a rank-1 update to an identity matrix, we can perform the matrix multiplication H α λ ′ R \mathbf{H}_{\alpha'_{\lambda}}\mathbf{R} H α λ ′ R in linear time O ( N λ N t ) \mathcal{O}(N_{\lambda}N_t) O ( N λ N t ) , and because H α λ ′ \mathbf{H}_{\alpha'_{\lambda}} H α λ ′ is completely defined by the N λ N_{\lambda} N λ -long vector u \mathbf{u} u , each Householder matrix has a linear O ( N λ ) \mathcal{O}(N_{\lambda}) O ( N λ ) memory scaling.
Factorising Equation (71) further using the Householder matrix H α λ ′ \mathbf{H}_{\alpha'_{\lambda}} H α λ ′ with e ^ i = e ^ 1 \hat{\mathbf{e}}_i=\hat{\mathbf{e}}_1 e ^ i = e ^ 1 , we get
K = [ Σ λ 1 2 H α λ ′ ⊗ I N t ] ⏟ K t r a n s f [ I N λ ⊗ D t + ∣ α λ ′ ∣ 2 e ^ 1 e ^ 1 T ⊗ K t ] ⏟ K ~ [ H α λ ′ T Σ λ 1 2 T ⊗ I N t ] ⏟ K t r a n s f T (78) \mathbf{K} =
\underbrace{\left[\boldsymbol{\Sigma}_{\lambda}^{\frac{1}{2}}\mathbf{H}_{\alpha'_{\lambda}}\otimes\mathbf{I}_{N_t}\right]}_{\mathbf{K}_{\mathrm{transf}}}
\underbrace{\left[\mathbf{I}_{N_{\lambda}}\otimes\mathbf{D}_t+\lvert\boldsymbol{\alpha}'_{\lambda}\rvert^2\hat{\mathbf{e}}_1\hat{\mathbf{e}}_1^T\otimes\mathbf{K}_t\right]}_{\widetilde{\mathbf{K}}}
\underbrace{\left[\mathbf{H}_{\alpha'_{\lambda}}^T\boldsymbol{\Sigma}_{\lambda}^{\frac{1}{2}T}\otimes\mathbf{I}_{N_t}\right]}_{\mathbf{K}_{\mathrm{transf}}^T} \tag*{(78)} K = K transf [ Σ λ 2 1 H α λ ′ ⊗ I N t ] K [ I N λ ⊗ D t + ∣ α λ ′ ∣ 2 e ^ 1 e ^ 1 T ⊗ K t ] K transf T [ H α λ ′ T Σ λ 2 1 T ⊗ I N t ] (78)
where K ~ \widetilde{\mathbf{K}} K can be expressed as the block matrix
K ~ = [ A B C D ] = [ D t + ∣ α λ ′ ∣ 2 K t 0 0 I N λ − 1 ⊗ D t ] , (79) \widetilde{\mathbf{K}}=
\begin{bmatrix}
\mathbf{A} & \mathbf{B}\\
\mathbf{C} & \mathbf{D}
\end{bmatrix}
=
\begin{bmatrix}
\mathbf{D}_t+\lvert\boldsymbol{\alpha}'_{\lambda}\rvert^2\mathbf{K}_t & \mathbf{0}\\
\mathbf{0} & \mathbf{I}_{N_{\lambda}-1}\otimes\mathbf{D}_t
\end{bmatrix}, \tag*{(79)} K = [ A C B D ] = [ D t + ∣ α λ ′ ∣ 2 K t 0 0 I N λ − 1 ⊗ D t ] , (79)
which is diagonal apart from the ( N t , N t ) (N_t,N_t) ( N t , N t ) A \mathbf{A} A block. In the highly optimised scenario where both K t \mathbf{K}_t K t and Σ λ \boldsymbol{\Sigma}_{\lambda} Σ λ are quasiseparable, we achieve a log-likelihood runtime scaling of O ( N t J K t 2 + N λ J Σ λ 2 + N λ N t ( 1 + J Σ λ ) ) \mathcal{O}(N_tJ_{\mathbf{K}_t}^2+N_{\lambda}J_{\boldsymbol{\Sigma}_{\lambda}}^2+N_{\lambda}N_t(1+J_{\boldsymbol{\Sigma}_{\lambda}})) O ( N t J K t 2 + N λ J Σ λ 2 + N λ N t ( 1 + J Σ λ )) .
Extending to N α > 1 N_{\alpha}>1 N α > 1 Using Householder QR
Now let’s consider the general case of N α ≥ 1 N_{\alpha}\geq1 N α ≥ 1 low-rank kernel terms. We start off with a similar trick as before, where we whiten all α λ ; i \alpha_{\lambda;i} α λ ; i components to factor out measurement covariance:
K = [ Σ λ 1 2 ⊗ I N t ] [ I N λ ⊗ D t + ∑ i = 1 N α α λ ; i ′ α λ ; i ′ T ⊗ K t ; i ] [ Σ λ 1 2 T ⊗ I N t ] (80) \mathbf{K}=
\left[\boldsymbol{\Sigma}_{\lambda}^{\frac{1}{2}}\otimes\mathbf{I}_{N_t}\right]
\left[\mathbf{I}_{N_{\lambda}}\otimes\mathbf{D}_t+\sum_{i=1}^{N_{\alpha}}\boldsymbol{\alpha}'_{\lambda;i}\boldsymbol{\alpha}_{\lambda;i}^{\prime T}\otimes\mathbf{K}_{t;i}\right]
\left[\boldsymbol{\Sigma}_{\lambda}^{\frac{1}{2}T}\otimes\mathbf{I}_{N_t}\right] \tag*{(80)} K = [ Σ λ 2 1 ⊗ I N t ] [ I N λ ⊗ D t + i = 1 ∑ N α α λ ; i ′ α λ ; i ′ T ⊗ K t ; i ] [ Σ λ 2 1 T ⊗ I N t ] (80)
with α λ ; i ′ = Σ λ − 1 2 α λ ; i \boldsymbol{\alpha}'_{\lambda;i}=\boldsymbol{\Sigma}_{\lambda}^{-\frac{1}{2}}\boldsymbol{\alpha}_{\lambda;i} α λ ; i ′ = Σ λ − 2 1 α λ ; i . Our transformed covariance matrix K ~ \widetilde{\mathbf{K}} K is now in a form that can be handled by the optimisation in Gordon et al. [53 ] for CELERITE/quasiseparable kernels in O ( N α 2 N λ N t J K t 2 ) \mathcal{O}(N_{\alpha}^2N_{\lambda}N_tJ_{\mathbf{K}_t}^2) O ( N α 2 N λ N t J K t 2 ) runtime, since we’ve diagonalised the measurement covariance. However, in the case that N α < N λ N_{\alpha}<N_{\lambda} N α < N λ , we introduce an additional transformation which can further reduce runtime costs.
We can make use of successive Householder transformations to make the problem more sparse and faster to compute. In fact, this process is equivalent to calculating the QR decomposition of the ( N λ , N α ) (N_{\lambda},N_{\alpha}) ( N λ , N α ) matrix A λ ′ \mathbf{A}'_{\lambda} A λ ′ , which we form using α λ ; i ′ \boldsymbol{\alpha}'_{\lambda;i} α λ ; i ′ as column vectors
A λ ′ = [ α λ ; 1 ′ ⋯ α λ ; N α ′ ] = Q A λ ′ [ R A λ ′ 0 ] (81) \mathbf{A}'_{\lambda}=
\begin{bmatrix}
\boldsymbol{\alpha}'_{\lambda;1} & \cdots& \boldsymbol{\alpha}'_{\lambda;N_{\alpha}}
\end{bmatrix}
=
\mathbf{Q}_{\mathbf{A}'_{\lambda}}
\begin{bmatrix}
\mathbf{R}_{\mathbf{A}'_{\lambda}}\\
\mathbf{0}
\end{bmatrix} \tag*{(81)} A λ ′ = [ α λ ; 1 ′ ⋯ α λ ; N α ′ ] = Q A λ ′ [ R A λ ′ 0 ] (81)
where Q A λ ′ \mathbf{Q}_{\mathbf{A}'_{\lambda}} Q A λ ′ is an ( N λ , N λ ) (N_{\lambda},N_{\lambda}) ( N λ , N λ ) orthonormal matrix and R A λ ′ \mathbf{R}_{\mathbf{A}'_{\lambda}} R A λ ′ is ( N α , N α ) (N_{\alpha},N_{\alpha}) ( N α , N α ) and upper-triangular. By using the Householder approach to calculating the QR decomposition, Q A λ ′ \mathbf{Q}_{\mathbf{A}'_{\lambda}} Q A λ ′ will be equal to a product of N α N_{\alpha} N α Householder matrices, and its inverse is simply the Householder matrices multiplied in reverse order
Q A λ ′ = H 1 H 2 … H N α , (82) \mathbf{Q}_{\mathbf{A}'_{\lambda}}=\mathbf{H}_1\mathbf{H}_2\ldots\mathbf{H}_{N_{\alpha}}, \tag*{(82)} Q A λ ′ = H 1 H 2 … H N α , (82)
Q A λ ′ T = H N α H N α − 1 … H 1 , (83) \mathbf{Q}_{\mathbf{A}'_{\lambda}}^T=\mathbf{H}_{N_{\alpha}}\mathbf{H}_{N_{\alpha-1}}\ldots\mathbf{H}_1, \tag*{(83)} Q A λ ′ T = H N α H N α − 1 … H 1 , (83)
Representing Q A λ ′ \mathbf{Q}_{\mathbf{A}'_{\lambda}} Q A λ ′ as a stack of Householder transformations has a number of advantages when N α < N λ N_{\alpha}<N_{\lambda} N α < N λ , it allows us to transform our data residuals in linear O ( N α N λ N t ) \mathcal{O}(N_{\alpha}N_{\lambda}N_t) O ( N α N λ N t ) by performing N α N_{\alpha} N α successive Householder transformations, and it comes with a cheap memory cost of O ( N α N λ ) \mathcal{O}(N_{\alpha}N_{\lambda}) O ( N α N λ ) to represent Q A λ ′ \mathbf{Q}_{\mathbf{A}'_{\lambda}} Q A λ ′ . The QR decomposition is cheap to calculate, with a runtime scaling of ∼ O ( N α 2 N λ ) \sim\mathcal{O}(N_{\alpha}^2N_{\lambda}) ∼ O ( N α 2 N λ ) for a ( N λ , N α ) (N_{\lambda},N_{\alpha}) ( N λ , N α ) matrix (Goubet and Van Loon 2013).
The MultiLowRankKernel Factorisation
Using our QR decomposition of A λ ′ \mathbf{A}'_{\lambda} A λ ′ in Equation (81) , we can now factorise our MultiLowRankKernel covariance matrix:
K = Σ λ ⊗ D t + ∑ i = 1 N α α λ ; i α λ ; i T ⊗ K t ; i (84) \mathbf{K}=\boldsymbol{\Sigma}_{\lambda}\otimes\mathbf{D}_t+\sum_{i=1}^{N_{\alpha}}\boldsymbol{\alpha}_{\lambda;i}\boldsymbol{\alpha}_{\lambda;i}^T\otimes\mathbf{K}_{t;i} \tag*{(84)} K = Σ λ ⊗ D t + i = 1 ∑ N α α λ ; i α λ ; i T ⊗ K t ; i (84)
= [ Σ λ 1 2 Q A λ ′ ⊗ I N t ] [ I N λ ⊗ D t + ∑ i = 1 N α [ R A λ ′ e ^ i 0 ] [ R A λ ′ e ^ i 0 ] T ⊗ K t ; i ] [ Σ λ 1 2 T ⊗ I N t ] (85) =
\left[\boldsymbol{\Sigma}_{\lambda}^{\frac{1}{2}}\mathbf{Q}_{\mathbf{A}'_{\lambda}}\otimes\mathbf{I}_{N_t}\right]
\left[\mathbf{I}_{N_{\lambda}}\otimes\mathbf{D}_t+\sum_{i=1}^{N_{\alpha}}
\begin{bmatrix}
\mathbf{R}_{\mathbf{A}'_{\lambda}}\hat{\mathbf{e}}_i\\
\mathbf{0}
\end{bmatrix}
\begin{bmatrix}
\mathbf{R}_{\mathbf{A}'_{\lambda}}\hat{\mathbf{e}}_i\\
\mathbf{0}
\end{bmatrix}^{T}
\otimes\mathbf{K}_{t;i}\right]
\left[\boldsymbol{\Sigma}_{\lambda}^{\frac{1}{2}T}\otimes\mathbf{I}_{N_t}\right] \tag*{(85)} = [ Σ λ 2 1 Q A λ ′ ⊗ I N t ] [ I N λ ⊗ D t + i = 1 ∑ N α [ R A λ ′ e ^ i 0 ] [ R A λ ′ e ^ i 0 ] T ⊗ K t ; i ] [ Σ λ 2 1 T ⊗ I N t ] (85)
Re-labelling the N α N_{\alpha} N α -long vectors a i = R A λ ′ e ^ i \mathbf{a}_i=\mathbf{R}_{\mathbf{A}'_{\lambda}}\hat{\mathbf{e}}_i a i = R A λ ′ e ^ i , we write K ~ \widetilde{\mathbf{K}} K as the block matrix
K ~ = [ I N α ⊗ D t + ∑ i = 1 N α a i a i T ⊗ K t ; i 0 0 I N λ − N α ⊗ D t ] (86) \widetilde{\mathbf{K}}=
\begin{bmatrix}
\mathbf{I}_{N_{\alpha}}\otimes\mathbf{D}_t+\sum_{i=1}^{N_{\alpha}}\mathbf{a}_i\mathbf{a}_i^T\otimes\mathbf{K}_{t;i} & \mathbf{0}\\
\mathbf{0} & \mathbf{I}_{N_{\lambda}-N_{\alpha}}\otimes\mathbf{D}_t
\end{bmatrix} \tag*{(86)} K = [ I N α ⊗ D t + ∑ i = 1 N α a i a i T ⊗ K t ; i 0 0 I N λ − N α ⊗ D t ] (86)
where our B \mathbf{B} B and C \mathbf{C} C blocks consist fully of zeros, and our D \mathbf{D} D block is fully diagonal. The A \mathbf{A} A block is still in the same form as K ~ \widetilde{\mathbf{K}} K in Equation (80) , before transforming with the QR decomposition. However, we have exploited sparsity to reduce the problem from decomposing a ( N λ N t , N λ N t ) (N_{\lambda}N_t,N_{\lambda}N_t) ( N λ N t , N λ N t ) matrix, down to a ( N α N t , N α N t ) (N_{\alpha}N_t,N_{\alpha}N_t) ( N α N t , N α N t ) matrix. This QR decomposition transformation has an interesting interpretation, as it is closely related to PCA, where we are projecting the data residuals onto separate systematics components which can have different kernel functions. This is explained in more detail in § 6.1.1 , and visualised in Figure 7 .
Figure 7. The MultiLowRankKernel can separate low-rank components based on different time correlations, while PCA can struggle to distinguish them when they have a similar amplitude. The left plot shows simulated systematics generated with a sum of N α = 2 N_\alpha= 2 N α = 2 low-rank kernel terms, an exponential time kernel with a constant 100 ppm amplitude in wavelength, and a simple harmonic oscillator time kernel with wavelength amplitudes α λ \alpha_\lambda α λ varying linearly from 172 ppm to -172 ppm (i.e. it is anti-correlated between the top and bottom light curves, and zero in the middle light curve). The time-dependence of the injected signals is overplotted in grey in both the middle and right plots (with a rescaling applied): the two largest PCA components recover a mixture of these two signals as they have similar amplitudes (middle plot), while the best-fit MultiLowRankKernel model fits for the wavelength-dependence of these components and separates their time-dependence almost exactly (right plot), visualised using the transformed residuals r ~ = K t r a n s f − 1 r \tilde{\mathbf{r}} = \mathbf{K}_{\mathrm{transf}}^{-1}\mathbf{r} r ~ = K transf − 1 r , as this projects each systematics component onto separate light curves.
Computational Scaling
If our time kernels are valid quasiseparable kernels, then the method in Gordon et al. [53 ] can decompose the A \mathbf{A} A block of K ~ \widetilde{\mathbf{K}} K , rather than be applied to the full K ~ \widetilde{\mathbf{K}} K matrix. The log-likelihood runtime scaling works out to O ( N α 3 N t J K t 2 + N α N λ N t ) \mathcal{O}(N_{\alpha}^3N_tJ_{\mathbf{K}_t}^2+N_{\alpha}N_{\lambda}N_t) O ( N α 3 N t J K t 2 + N α N λ N t ) when Σ λ \boldsymbol{\Sigma}_{\lambda} Σ λ is diagonal, with an extra O ( N λ J Σ λ 2 + N λ N t J Σ λ ) \mathcal{O}(N_{\lambda}J_{\Sigma_{\lambda}}^{2}+N_{\lambda}N_{t}J_{\Sigma_{\lambda}}) O ( N λ J Σ λ 2 + N λ N t J Σ λ ) runtime cost if Σ λ \Sigma_{\lambda} Σ λ is quasiseparable with rank J Σ λ J_{\Sigma_{\lambda}} J Σ λ , or an extra O ( N λ 3 + N λ 2 N t ) \mathcal{O}(N_{\lambda}^{3}+N_{\lambda}^{2}N_{t}) O ( N λ 3 + N λ 2 N t ) cost for arbitrary Σ λ \Sigma_{\lambda} Σ λ . The memory scaling will be at most ∼ O ( N λ 2 + N λ 2 N t J ˉ K t + N λ N t ) \sim\mathcal{O}(N_{\lambda}^{2}+N_{\lambda}^{2}N_{t}\bar{J}_{K_{t}}+N_{\lambda}N_{t}) ∼ O ( N λ 2 + N λ 2 N t J ˉ K t + N λ N t ) for arbitrary Σ λ \Sigma_{\lambda} Σ λ , but this can be reduced to ∼ O ( N λ J Σ λ + N λ 2 N t J ˉ K t + N λ N t ) \sim\mathcal{O}(N_{\lambda}J_{\Sigma_{\lambda}}+N_{\lambda}^{2}N_{t}\bar{J}_{K_{t}}+N_{\lambda}N_{t}) ∼ O ( N λ J Σ λ + N λ 2 N t J ˉ K t + N λ N t ) with quasiseparable Σ λ \Sigma_{\lambda} Σ λ .
We make an improvement to the existing runtime scaling of O ( N α 2 N λ N t J ˉ K t 2 ) \mathcal{O}(N_{\alpha}^{2}N_{\lambda}N_{t}\bar{J}_{K_{t}}^{2}) O ( N α 2 N λ N t J ˉ K t 2 ) from applying the Gordon et al. (2020) method without the QR decomposition, and we have also generalised it to include measurement covariance, although we cannot deal with non-separable noise (see § 2.5 ). Not only do we achieve a better runtime scaling, but even in the case of N α = 1 N_{\alpha}=1 N α = 1 and J ˉ K t = 1 \bar{J}_{K_{t}}=1 J ˉ K t = 1 , benchmarking both approaches in Figures G14 and G15 found that the MultiLowRankKernel can reduce runtimes by over an order of magnitude compared to Gordon et al. (2020). For example, with ( N λ , N t ) = ( 1024 , 16384 ) (N_{\lambda},N_{t})=(1024,16384) ( N λ , N t ) = ( 1024 , 16384 ) we achieve a log-likelihood runtime of just 29 ms on CPU, compared to 0.6 s with the Gordon et al. (2020) approach implemented using TinyGP. Increasing the MultiLowRankKernel to N α = 4 , 8 , 16 N_{\alpha}=4,8,16 N α = 4 , 8 , 16 is benchmarked in Figure G18 , Figure G19 and Figure G20 , demonstrating highly scalable runtimes even for multiple components with different time kernels.
Figure G18. Comparing with Figure G15 where N α = 1 N_{\alpha}=1 N α = 1 , increasing to N α = 4 N_{\alpha}=4 N α = 4 results in a relatively minor runtime increase, particularly for large N λ N_{\lambda} N λ , consistent with its runtime scaling as O ( N α 3 N t J ˉ κ 2 + N α N λ N t ) \mathcal{O}(N_{\alpha}^{3}N_{\mathrm{t}}\bar{J}_{\kappa}^{2}+N_{\alpha}N_{\lambda}N_{\mathrm{t}}) O ( N α 3 N t J ˉ κ 2 + N α N λ N t ) , where the fact the quasiseparable decomposition does not scale with N λ N_{\lambda} N λ mitigates the cubic scaling with N α N_{\alpha} N α .
Figure G19. Increasing further to N α = 8 N_{\alpha}=8 N α = 8 gives a further increase in runtimes relative to Figures G15 and Figure G18 for N α = 1 , 4 N_{\alpha}=1,4 N α = 1 , 4 respectively, most significant when N λ N_{\lambda} N λ is small and the O ( N α 3 N t J ˉ κ 2 ) \mathcal{O}(N_{\alpha}^{3}N_{\mathrm{t}}\bar{J}_{\kappa}^{2}) O ( N α 3 N t J ˉ κ 2 ) term dominates.
Figure G20. Increasing to N α = 16 N_{\alpha}=16 N α = 16 , we can see the cost of the O ( N α 3 N t J ˉ κ 2 ) \mathcal{O}(N_{\alpha}^{3}N_{\mathrm{t}}\bar{J}_{\kappa}^{2}) O ( N α 3 N t J ˉ κ 2 ) term dominates, as the runtime cost is almost constant between N λ = 16 → 256 N_{\lambda}=16\to256 N λ = 16 → 256 . Note that when N λ = 8 , 16 N_{\lambda}=8,16 N λ = 8 , 16 , our method reduces to the Gordon et al. [53 ] method with the addition of measurement covariance, as the transformations to exploit sparsity only work when N λ > N α N_{\lambda}>N_{\alpha} N λ > N α .
Unlike Gordon et al. (2020), our approach even optimises the case where K t ; i \mathbf{K}_{t;i} K t ; i are arbitrary covariance matrices with cubic runtime decompositions, as we reduce the dense covariance matrix decomposition from a runtime scaling of O ( N λ 3 N t 3 ) \mathcal{O}(N_{\lambda}^{3}N_{t}^{3}) O ( N λ 3 N t 3 ) , down to ∼ O ( N α 3 N t 3 ) \sim\mathcal{O}(N_{\alpha}^{3}N_{t}^{3}) ∼ O ( N α 3 N t 3 ) by using a general Cholesky decomposition of the A block in Equation 94. We have benchmarked this case for N α = 1 N_{\alpha}=1 N α = 1 on both CPU and GPU in Figures G16 and G17 respectively. Note that a similar optimisation has already been independently derived in Bruinsma et al. (2020) for machine learning applications, but they do not handle measurement covariance, and to achieve a linear scaling in N t N_{t} N t they require orthogonal α λ ; i \boldsymbol{\alpha}_{\lambda;i} α λ ; i components, which we avoid by combining with the Gordon et al. (2020) method and Householder QR. Restricting to orthogonal components can improve runtimes further however, as we discuss in the next subsection.
Improved Scaling with Orthogonality
If the whitened vectors α λ ; i ′ = Σ λ − 1 2 α λ ; i \boldsymbol{\alpha}'_{\lambda;i}=\Sigma_{\lambda}^{-\frac{1}{2}}\boldsymbol{\alpha}_{\lambda;i} α λ ; i ′ = Σ λ − 2 1 α λ ; i are assumed to be orthogonal, then the matrix R α ′ R_{\boldsymbol{\alpha}'} R α ′ from the QR decomposition will be diagonal, which results in K ~ \tilde{\mathbf{K}} K ~ becoming fully block-diagonal. This is equivalent to the LUASLASRACHKERNEL, but where we vary each time covariance matrix K t \mathbf{K}_{t} K t within the diagonal blocks of K ~ \tilde{\mathbf{K}} K ~ from Equation 73. We derive a similar black-box log-likelihood as the LUASLASRACHKERNEL for the decoupled terms:
log p ( r ∣ K ) = − N t log ∣ Σ λ 1 2 ∣ + ∑ i = 1 N α log p ( r ~ i , D t + ∣ α λ ; i ′ ∣ 2 K t ; i ) + ∑ i = N α + 1 N λ log p ( r ~ i , D t ) , (87) \log p(\mathbf{r}\mid\mathbf{K})=-N_{t}\log\left|\Sigma_{\lambda}^{\frac{1}{2}}\right|+\sum_{i=1}^{N_{\alpha}}\log p(\tilde{\mathbf{r}}_{i},\mathbf{D}_{t}+\left|\boldsymbol{\alpha}'_{\lambda;i}\right|^{2}\mathbf{K}_{t;i})+\sum_{i=N_{\alpha}+1}^{N_{\lambda}}\log p(\tilde{\mathbf{r}}_{i},\mathbf{D}_{t}), \tag*{(87)} log p ( r ∣ K ) = − N t log Σ λ 2 1 + i = 1 ∑ N α log p ( r ~ i , D t + α λ ; i ′ 2 K t ; i ) + i = N α + 1 ∑ N λ log p ( r ~ i , D t ) , (87)
where each of the 1D log-likelihood terms can use any desired GP optimisation on the transformed residuals. This orthogonal case essentially reduces to the GP optimisation introduced in Bruinsma et al. (2020), although we have further generalised it to include measurement covariance.
Orthogonal components produce a more efficient runtime scaling of O ( N α N t J ˉ K t 2 + N α N λ N t ) \mathcal{O}(N_{\alpha}N_{t}\bar{J}_{K_{t}}^{2}+N_{\alpha}N_{\lambda}N_{t}) O ( N α N t J ˉ K t 2 + N α N λ N t ) for diagonal Σ λ \Sigma_{\lambda} Σ λ , plus the same additional costs as § 4.3 if Σ λ \Sigma_{\lambda} Σ λ is instead quasiseparable or arbitrary. Note that orthogonality allows for a linear rather than cubic scaling in N α N_{\alpha} N α , which can make a difference with many components, but if N λ N t N_{\lambda}N_{t} N λ N t is large then they can both be bottlenecked by the same O ( N α N λ N t ) \mathcal{O}(N_{\alpha}N_{\lambda}N_{t}) O ( N α N λ N t ) cost to transform the residuals, or from additional costs due to measurement covariance. This can be seen by examining the benchmarking plots in Figure G20 and Figure G21 , which compare running N α = 16 N_{\alpha}=16 N α = 16 components without and with the orthogonality constraint respectively, and show that the greatest benefit of orthogonality is when N t ≫ N λ N_{t}\gg N_{\lambda} N t ≫ N λ . However, we note that the relative cost to calculate gradients of the log-likelihood was also cheaper with orthogonality, as discussed in Section G.1. One other difference worth noting, specifically for CELERITE and quasiseparable kernels, is that this decoupling of each time kernel with orthogonality means that each kernel k t ; i \mathbf{k}_{t;i} k t ; i has no requirement to share the same input sort order, e.g. allowing us to joint-fit low-rank CELERITE kernels with different input regressors along the time axis (e.g. FWHM, centroiding position).
Figure G21. If the α λ ; i \alpha_{\lambda;i} α λ ; i vectors are orthogonal (after a whitening transformation), then the runtime scaling is instead O ( N α N t J ˉ κ 2 + N α N λ N t ) \mathcal{O}(N_{\alpha}N_{\mathrm{t}}\bar{J}_{\kappa}^{2}+N_{\alpha}N_{\lambda}N_{\mathrm{t}}) O ( N α N t J ˉ κ 2 + N α N λ N t ) , which performs significantly faster when N λ N_{\lambda} N λ is small and N α N_{\alpha} N α large compared with Figure G20 . When N α = N λ N_{\alpha}=N_{\lambda} N α = N λ , this is equivalent to the LUASLASRACHKERNEL with different time kernel functions for each eigenvector of the wavelength covariance matrix. Note we must exclude N λ = 8 N_{\lambda}=8 N λ = 8 as we cannot have 16 orthogonal vectors in this scenario.
DoubleMultiLowRankKernel: Low-Rank Terms in Both Dimensions
We can extend these tricks to deal with low-rank terms which have kernel functions in different dimensions, corresponding to kernel functions of the form
k ( λ , λ ′ , t , t ′ ) = σ 2 ( λ ) σ 2 ( t ) δ λ λ ′ δ t t ′ + ∑ i = 1 N α α i ( λ ) α i ( λ ′ ) k t ; i ( t , t ′ ) + ∑ j = 1 N β k λ ; j ( λ , λ ′ ) β j ( t ) β j ( t ′ ) , (88) k(\lambda,\lambda',t,t')=\sigma^{2}(\lambda)\sigma^{2}(t)\delta_{\lambda\lambda'}\delta_{tt'}+\sum_{i=1}^{N_{\alpha}}\alpha_{i}(\lambda)\alpha_{i}(\lambda')k_{t;i}(t,t')+\sum_{j=1}^{N_{\beta}}k_{\lambda;j}(\lambda,\lambda')\beta_{j}(t)\beta_{j}(t'), \tag*{(88)} k ( λ , λ ′ , t , t ′ ) = σ 2 ( λ ) σ 2 ( t ) δ λ λ ′ δ t t ′ + i = 1 ∑ N α α i ( λ ) α i ( λ ′ ) k t ; i ( t , t ′ ) + j = 1 ∑ N β k λ ; j ( λ , λ ′ ) β j ( t ) β j ( t ′ ) , (88)
In this case, the factorisation we use is similar to the MultiLowRankKernel, but with a QR decomposition in both dimensions:
K = D λ ⊗ D t + ∑ i = 1 N α α λ ; i α λ ; i T ⊗ K t ; i + ∑ j = 1 N β K λ ; j ⊗ β t ; j β t ; j T = [ D λ 1 2 Q A ′ ⊗ D t 1 2 Q B ′ ] ⏟ K t r a n s f [ I N λ N t + ∑ i = 1 N α [ a i 0 ] [ a i 0 ] T ⊗ K t ; i ′ + ∑ j = 1 N β K λ ; j ′ ⊗ [ b j 0 ] [ b j 0 ] T ] (89) \begin{aligned}
\mathbf{K}=\mathbf{D}_{\lambda}\otimes\mathbf{D}_{t}+\sum_{i=1}^{N_{\alpha}}\boldsymbol{\alpha}_{\lambda;i}\boldsymbol{\alpha}_{\lambda;i}^{T}\otimes\mathbf{K}_{t;i}+\sum_{j=1}^{N_{\beta}}\mathbf{K}_{\lambda;j}\otimes\boldsymbol{\beta}_{t;j}\boldsymbol{\beta}_{t;j}^{T} \\
&=\underbrace{\left[\mathbf{D}_{\lambda}^{\frac{1}{2}}\mathbf{Q}_{A'}\otimes\mathbf{D}_{t}^{\frac{1}{2}}\mathbf{Q}_{B'}\right]}_{\mathbf{K}_{\mathrm{transf}}}
\left[\mathbf{I}_{N_{\lambda}N_{t}}+\sum_{i=1}^{N_{\alpha}}\begin{bmatrix}\mathbf{a}_{i}\\0\end{bmatrix}\begin{bmatrix}\mathbf{a}_{i}\\0\end{bmatrix}^{T}\otimes\mathbf{K}'_{t;i}+\sum_{j=1}^{N_{\beta}}\mathbf{K}'_{\lambda;j}\otimes\begin{bmatrix}\mathbf{b}_{j}\\0\end{bmatrix}\begin{bmatrix}\mathbf{b}_{j}\\0\end{bmatrix}^{T}\right] \tag*{(89)}
\end{aligned} K = D λ ⊗ D t + i = 1 ∑ N α α λ ; i α λ ; i T ⊗ K t ; i + j = 1 ∑ N β K λ ; j ⊗ β t ; j β t ; j T = K transf [ D λ 2 1 Q A ′ ⊗ D t 2 1 Q B ′ ] I N λ N t + i = 1 ∑ N α [ a i 0 ] [ a i 0 ] T ⊗ K t ; i ′ + j = 1 ∑ N β K λ ; j ′ ⊗ [ b j 0 ] [ b j 0 ] T (89)
∗ [ Q A ′ T D λ 1 2 ⊗ Q B ′ T D t 1 2 ] ⏟ K t r a n s f T (90) \qquad\qquad*\underbrace{\left[\mathbf{Q}_{A'}^{T}\mathbf{D}_{\lambda}^{\frac{1}{2}}\otimes\mathbf{Q}_{B'}^{T}\mathbf{D}_{t}^{\frac{1}{2}}\right]}_{\mathbf{K}_{\mathrm{transf}}^{T}} \tag*{(90)} ∗ K transf T [ Q A ′ T D λ 2 1 ⊗ Q B ′ T D t 2 1 ] (90)
where we’ve transformed all of the K λ ; j \mathbf{K}_{\lambda;j} K λ ; j and K t ; i \mathbf{K}_{t;i} K t ; i matrices following
K λ ; j ′ = Q A ′ T D λ − 1 2 K λ ; j D λ − 1 2 Q A ′ (91) \mathbf{K}'_{\lambda;j}=\mathbf{Q}_{A'}^{T}\mathbf{D}_{\lambda}^{-\frac{1}{2}}\mathbf{K}_{\lambda;j}\mathbf{D}_{\lambda}^{-\frac{1}{2}}\mathbf{Q}_{A'} \tag*{(91)} K λ ; j ′ = Q A ′ T D λ − 2 1 K λ ; j D λ − 2 1 Q A ′ (91)
K t ; i ′ = Q B ′ T D t − 1 2 K t ; i D t − 1 2 Q B ′ (92) \mathbf{K}'_{t;i}=\mathbf{Q}_{B'}^{T}\mathbf{D}_{t}^{-\frac{1}{2}}\mathbf{K}_{t;i}\mathbf{D}_{t}^{-\frac{1}{2}}\mathbf{Q}_{B'} \tag*{(92)} K t ; i ′ = Q B ′ T D t − 2 1 K t ; i D t − 2 1 Q B ′ (92)
which can be performed while maintaining quasiseparability if we exclude measurement covariance (D λ \mathbf{D}_{\lambda} D λ and D t \mathbf{D}_{t} D t must be diagonal), which is demonstrated in Appendix § 9 .§ 1 . Just note that each of the N α N_{\alpha} N α Householder transformations in Q A ′ \mathbf{Q}_{A'} Q A ′ (and N β N_{\beta} N β transformations in Q B ′ \mathbf{Q}_{B'} Q B ′ ) increases the quasiseparable rank of the transformed covariance matrix by 2. Therefore, all quasiseparable ranks get shifted from J K λ ; j → J K λ ; j + 2 N α J_{K_{\lambda;j}}\rightarrow J_{K_{\lambda;j}}+2N_{\alpha} J K λ ; j → J K λ ; j + 2 N α and J K t ; i → J K t ; i + 2 N β J_{K_{t;i}}\rightarrow J_{K_{t;i}}+2N_{\beta} J K t ; i → J K t ; i + 2 N β . Similar to the MULTILOWRANKKERNEL, the K ~ \widetilde{\mathbf{K}} K term can be written as the block matrix:
[ ( I N α ⊗ I N t + ∑ i = 1 N α a i a i T ⊗ K t ; i ′ + ∑ j = 1 N β K λ ; j , A ′ ⊗ [ b j 0 ] [ b j 0 ] T ) ∑ j = 1 N β K λ ; j , B ′ ⊗ [ b j 0 ] [ b j 0 ] T ∑ j = 1 N β K λ ; j , C ′ ⊗ [ b j 0 ] [ b j 0 ] T I N λ − N α ⊗ I N t + ∑ j = 1 N β K λ ; j , D ′ ⊗ [ b j 0 ] [ b j 0 ] T ] (93) \begin{bmatrix}
\left(\mathbf{I}_{N_{\alpha}}\otimes\mathbf{I}_{N_t}+\sum_{i=1}^{N_{\alpha}}\mathbf{a}_i\mathbf{a}_i^{T}\otimes\mathbf{K}'_{t;i}+\sum_{j=1}^{N_{\beta}}\mathbf{K}'_{\lambda;j,A}\otimes\begin{bmatrix}\mathbf{b}_j\\\mathbf{0}\end{bmatrix}\begin{bmatrix}\mathbf{b}_j\\\mathbf{0}\end{bmatrix}^{T}\right) & \sum_{j=1}^{N_{\beta}}\mathbf{K}'_{\lambda;j,B}\otimes\begin{bmatrix}\mathbf{b}_j\\\mathbf{0}\end{bmatrix}\begin{bmatrix}\mathbf{b}_j\\\mathbf{0}\end{bmatrix}^{T}\\
\sum_{j=1}^{N_{\beta}}\mathbf{K}'_{\lambda;j,C}\otimes\begin{bmatrix}\mathbf{b}_j\\\mathbf{0}\end{bmatrix}\begin{bmatrix}\mathbf{b}_j\\\mathbf{0}\end{bmatrix}^{T} & \mathbf{I}_{N_{\lambda}-N_{\alpha}}\otimes\mathbf{I}_{N_t}+\sum_{j=1}^{N_{\beta}}\mathbf{K}'_{\lambda;j,D}\otimes\begin{bmatrix}\mathbf{b}_j\\\mathbf{0}\end{bmatrix}\begin{bmatrix}\mathbf{b}_j\\\mathbf{0}\end{bmatrix}^{T} \tag*{(93)}
\end{bmatrix} ( I N α ⊗ I N t + ∑ i = 1 N α a i a i T ⊗ K t ; i ′ + ∑ j = 1 N β K λ ; j , A ′ ⊗ [ b j 0 ] [ b j 0 ] T ) ∑ j = 1 N β K λ ; j , C ′ ⊗ [ b j 0 ] [ b j 0 ] T ∑ j = 1 N β K λ ; j , B ′ ⊗ [ b j 0 ] [ b j 0 ] T I N λ − N α ⊗ I N t + ∑ j = 1 N β K λ ; j , D ′ ⊗ [ b j 0 ] [ b j 0 ] T (93)
where we can split up the decomposition of K ~ \widetilde{\mathbf{K}} K using formulae for 2x2 block matrices. This works by first decomposing the A block, calculating the term C A − 1 B \mathbf{C}\mathbf{A}^{-1}\mathbf{B} C A − 1 B , and then decomposing D − C A − 1 B \mathbf{D}-\mathbf{C}\mathbf{A}^{-1}\mathbf{B} D − C A − 1 B , all described in detail in Appendix § 9 .§ 2 .
Computational Scaling
In Appendix § 9 , we derive a runtime scaling of O ( N α 3 N t ( J ˉ K t + 3 N β ) 2 + N β 3 N λ ( J ˉ K λ + 3 N α ) 2 + N λ N t ( N α + N β ) ) \mathcal{O}(N_{\alpha}^{3}N_t(\bar{J}_{K_t}+3N_{\beta})^{2}+N_{\beta}^{3}N_{\lambda}(\bar{J}_{K_{\lambda}}+3N_{\alpha})^{2}+N_{\lambda}N_t(N_{\alpha}+N_{\beta})) O ( N α 3 N t ( J ˉ K t + 3 N β ) 2 + N β 3 N λ ( J ˉ K λ + 3 N α ) 2 + N λ N t ( N α + N β )) assuming all K λ ; j \mathbf{K}_{\lambda;j} K λ ; j and K t ; i \mathbf{K}_{t;i} K t ; i are quasiseparable, with mean quasiseparable ranks of J ˉ K λ \bar{J}_{K_{\lambda}} J ˉ K λ and J ˉ K t \bar{J}_{K_t} J ˉ K t respectively. The memory scaling for this case is ∼ O ( N α 2 N t ( J ˉ K t + 3 N β ) + N β 2 N λ ( J ˉ K λ + 3 N α ) + N λ N t ) \sim\mathcal{O}(N_{\alpha}^{2}N_t(\bar{J}_{K_t}+3N_{\beta})+N_{\beta}^{2}N_{\lambda}(\bar{J}_{K_{\lambda}}+3N_{\alpha})+N_{\lambda}N_t) ∼ O ( N α 2 N t ( J ˉ K t + 3 N β ) + N β 2 N λ ( J ˉ K λ + 3 N α ) + N λ N t ) . See Figure G22 , Figure G23 , Figure G24 and Figure G25 for benchmarking in the cases of ( N α , N β ) = ( 1 , 1 ) , ( 1 , 4 ) , ( 4 , 1 ) , ( 4 , 4 ) (N_{\alpha},N_{\beta})=(1,1),(1,4),(4,1),(4,4) ( N α , N β ) = ( 1 , 1 ) , ( 1 , 4 ) , ( 4 , 1 ) , ( 4 , 4 ) , respectively. For ( N α , N β ) = ( 1 , 1 ) (N_{\alpha},N_{\beta})=(1,1) ( N α , N β ) = ( 1 , 1 ) and ( N λ , N t ) = ( 1024 , 16384 ) (N_{\lambda},N_t)=(1024,16384) ( N λ , N t ) = ( 1024 , 16384 ) , we get a log-likelihood runtime of 80 ms on CPU, so this kernel remains viable even for large datasets. However, this runtime increases to 0.2-0.3 seconds when N α = 4 N_{\alpha}=4 N α = 4 or N β = 4 N_{\beta}=4 N β = 4 , and if both N α = N β = 4 N_{\alpha}=N_{\beta}=4 N α = N β = 4 , then it increases to 2 seconds to calculate, as the runtime can grow significantly if the product N α N β N_{\alpha}N_{\beta} N α N β increases.
Figure G22. With low-rank quasiseparable kernel terms in both dimensions, we can still achieve a linear runtime scaling with the number of datapoints using the DOUBLEMULTILOWRANKKERNEL, with a runtime scaling of O ( N α 3 N t ( J ˉ K t + 3 N β ) 2 + N β 3 N λ ( J ˉ K λ + 3 N α ) 2 + N λ N t ( N α + N β ) ) \mathcal{O}(N_\alpha^3N_t(\bar{J}_{K_t}+3N_\beta)^2+N_\beta^3N_\lambda(\bar{J}_{K_\lambda}+3N_\alpha)^2+N_\lambda N_t(N_\alpha+N_\beta)) O ( N α 3 N t ( J ˉ K t + 3 N β ) 2 + N β 3 N λ ( J ˉ K λ + 3 N α ) 2 + N λ N t ( N α + N β )) . Currently we can only maintain quasiseparability in both dimensions if we exclude measurement covariance, so heterogeneous white noise is modelled for this optimisation.
Figure G23. Increasing from N β = 1 N_\beta=1 N β = 1 in Figure G22 to N β = 4 N_\beta=4 N β = 4 is quite cheap when N α N_\alpha N α is small, where N β N_\beta N β corresponds to the kernel term which has its vector outer product in the longer axis for most of these data shapes.
Figure G24. In contrast to Figure G23 , having more kernel terms which have their vector outer product in the shorter axis scales less efficiently, particularly for a small number of wavelengths.
Figure G25. The DOUBLEMULTILOWRANKKERNEL scales poorly with the product of N α N β N_\alpha N_\beta N α N β , we see that log-likelihood calculations are expensive even for small datasets when N α = N β = 4 N_\alpha=N_\beta=4 N α = N β = 4 , and our GPU runtimes were typically slower or similar for this kernel.
LUASPLUSMULTILOWRANKKERNEL: Combining 2D Kernels with Low-Rank Kernel Terms
We can keep extending to more complex kernels, with K ~ \widetilde{\mathbf{K}} K becoming an increasingly complex 2x2 block matrix. Consider the case where we may have measurement covariance, a full-rank 2D kernel term, and a sum of low-rank terms, such as:
k ( λ , λ ′ , t , t ′ ) = s λ ( λ , λ ′ ) s t ( t , t ′ ) + k λ ; 2 D ( λ , λ ′ ) k t ; 2 D ( t , t ′ ) + ∑ i = 1 N α α i ( λ ) α i ( λ ′ ) k t ; i ( t , t ′ ) . (94) k(\lambda,\lambda',t,t')=s_{\lambda}(\lambda,\lambda')s_t(t,t')+k_{\lambda;2D}(\lambda,\lambda')k_{t;2D}(t,t')+\sum_{i=1}^{N_{\alpha}}\alpha_i(\lambda)\alpha_i(\lambda')k_{t;i}(t,t'). \tag*{(94)} k ( λ , λ ′ , t , t ′ ) = s λ ( λ , λ ′ ) s t ( t , t ′ ) + k λ ; 2 D ( λ , λ ′ ) k t ; 2 D ( t , t ′ ) + i = 1 ∑ N α α i ( λ ) α i ( λ ′ ) k t ; i ( t , t ′ ) . (94)
which corresponds to covariance matrices of the form
K = Σ λ ⊗ Σ t + K λ ; 2 D ⊗ K t ; 2 D + ∑ i = 1 N α α λ ; i α λ ; i T ⊗ K t ; i . (95) \mathbf{K}=\mathbf{\Sigma}_{\lambda}\otimes\mathbf{\Sigma}_t+\mathbf{K}_{\lambda;2D}\otimes\mathbf{K}_{t;2D}+\sum_{i=1}^{N_{\alpha}}\boldsymbol{\alpha}_{\lambda;i}\boldsymbol{\alpha}_{\lambda;i}^{T}\otimes\mathbf{K}_{t;i}. \tag*{(95)} K = Σ λ ⊗ Σ t + K λ ; 2 D ⊗ K t ; 2 D + i = 1 ∑ N α α λ ; i α λ ; i T ⊗ K t ; i . (95)
In this scenario, our LUASPLUSMULTILOWRANKKERNEL optimisation can still exploit quasiseparability, but just in the wavelength dimension (i.e. Σ λ \mathbf{\Sigma}_{\lambda} Σ λ and K λ ; 2 D \mathbf{K}_{\lambda;2D} K λ ; 2 D must be quasiseparable/diagonal), in which case the runtime scales linearly with N λ N_{\lambda} N λ . However, if the dimensions of the low-rank terms are flipped, i.e.
k ( λ , λ ′ , t , t ′ ) = s λ ( λ , λ ′ ) s t ( t , t ′ ) + k λ ; 2 D ( λ , λ ′ ) k t ; 2 D ( t , t ′ ) + ∑ j = 1 N β k λ ; j ( λ , λ ′ ) β j ( t ) β j ( t ′ ) (96) k(\lambda,\lambda',t,t')=s_{\lambda}(\lambda,\lambda')s_t(t,t')+k_{\lambda;2D}(\lambda,\lambda')k_{t;2D}(t,t')+\sum_{j=1}^{N_{\beta}}k_{\lambda;j}(\lambda,\lambda')\beta_j(t)\beta_j(t') \tag*{(96)} k ( λ , λ ′ , t , t ′ ) = s λ ( λ , λ ′ ) s t ( t , t ′ ) + k λ ; 2 D ( λ , λ ′ ) k t ; 2 D ( t , t ′ ) + j = 1 ∑ N β k λ ; j ( λ , λ ′ ) β j ( t ) β j ( t ′ ) (96)
then we can instead scale linearly with N t N_t N t , provided Σ t \mathbf{\Sigma}_t Σ t and K t ; 2 D \mathbf{K}_{t;2D} K t ; 2 D are quasiseparable (or diagonal). The algorithm is the same in either case, we can simply transpose our residuals array R \mathbf{R} R before calculating to switch between them.
For the case in Equation (95) , our transformation performs a simultaneous diagonalisation in the time dimension of Σ t \mathbf{\Sigma}_t Σ t and K t ; 2 D \mathbf{K}_{t;2D} K t ; 2 D , similar to the LUASKERNEL. In the wavelength dimension, we perform a similar transformation to the MULTILOWRANKKERNEL, without any whitening transformation. We form the matrix A λ \mathbf{A}_{\lambda} A λ using α λ ; i \boldsymbol{\alpha}_{\lambda;i} α λ ; i as column vectors, and instead of a QR decomposition, we perform Householder QL decomposition, where we instead project to a lower-triangular matrix L A \mathbf{L}_{A} L A , i.e.
A λ = [ α λ ; 1 ⋯ α λ ; N α ] = Q A λ [ 0 L A λ ] (97) \mathbf{A}_{\lambda}=[\boldsymbol{\alpha}_{\lambda;1}\ \cdots\ \boldsymbol{\alpha}_{\lambda;N_{\alpha}}]=\mathbf{Q}_{A_{\lambda}}\begin{bmatrix}\mathbf{0}\\\mathbf{L}_{A_{\lambda}}\end{bmatrix} \tag*{(97)} A λ = [ α λ ; 1 ⋯ α λ ; N α ] = Q A λ [ 0 L A λ ] (97)
which creates sparsity at the top of the vectors, as it happens to simplify the calculation of C A − 1 B \mathbf{C}\mathbf{A}^{-1}\mathbf{B} C A − 1 B . We now label a i = L A λ e ^ i \mathbf{a}_i=\mathbf{L}_{A_{\lambda}}\hat{\mathbf{e}}_i a i = L A λ e ^ i , and factorise K \mathbf{K} K as follows
K = [ Q A λ ⊗ Σ t 1 2 Q K t ; 2 D ′ ] ⏟ K t r a n s f [ Σ λ ′ ⊗ I N t + K λ ; 2 D ′ ⊗ K t ; 2 D ′ + ∑ i = 1 N α [ a i 0 ] [ a i 0 ] T ⊗ K t ; i ′ ] ⏟ K ~ [ Q A λ T ⊗ Q K t ; 2 D ′ T ( Σ t 1 2 ) T ] ⏟ K t r a n s f T . (98) \mathbf{K}=\underbrace{[\mathbf{Q}_{A_{\lambda}}\otimes\mathbf{\Sigma}_t^{\frac{1}{2}}\mathbf{Q}_{K'_{t;2D}}]}_{\mathbf{K}_{\mathrm{transf}}}
\underbrace{\left[\mathbf{\Sigma}'_{\lambda}\otimes\mathbf{I}_{N_t}+\mathbf{K}'_{\lambda;2D}\otimes\mathbf{K}'_{t;2D}+\sum_{i=1}^{N_{\alpha}}\begin{bmatrix}\mathbf{a}_i\\\mathbf{0}\end{bmatrix}\begin{bmatrix}\mathbf{a}_i\\\mathbf{0}\end{bmatrix}^{T}\otimes\mathbf{K}'_{t;i}\right]}_{\widetilde{\mathbf{K}}}
\underbrace{[\mathbf{Q}_{A_{\lambda}}^{T}\otimes\mathbf{Q}_{K'_{t;2D}}^{T}(\mathbf{\Sigma}_t^{\frac{1}{2}})^{T}]}_{\mathbf{K}_{\mathrm{transf}}^{T}}. \tag*{(98)} K = K transf [ Q A λ ⊗ Σ t 2 1 Q K t ; 2 D ′ ] K [ Σ λ ′ ⊗ I N t + K λ ; 2 D ′ ⊗ K t ; 2 D ′ + i = 1 ∑ N α [ a i 0 ] [ a i 0 ] T ⊗ K t ; i ′ ] K transf T [ Q A λ T ⊗ Q K t ; 2 D ′ T ( Σ t 2 1 ) T ] . (98)
where the QL transformation has transformed the wavelength covariance matrices as
Σ λ ′ = Q A λ T Σ λ Q A λ (99) \mathbf{\Sigma}'_{\lambda}=\mathbf{Q}_{A_{\lambda}}^{T}\mathbf{\Sigma}_{\lambda}\mathbf{Q}_{A_{\lambda}} \tag*{(99)} Σ λ ′ = Q A λ T Σ λ Q A λ (99)
K λ ; 2 D ′ = Q A λ T K λ ; 2 D Q A λ (100) \mathbf{K}'_{\lambda;2D}=\mathbf{Q}_{A_{\lambda}}^{T}\mathbf{K}_{\lambda;2D}\mathbf{Q}_{A_{\lambda}} \tag*{(100)} K λ ; 2 D ′ = Q A λ T K λ ; 2 D Q A λ (100)
which will increase each of their quasiseparable ranks by 2 N α 2N_{\alpha} 2 N α if they are quasiseparable (see Appendix § 9 .§ 1 ). Each of the time covariance matrices K t ; i \mathbf{K}_{t;i} K t ; i are transformed as
K t ; i ′ = Q K t ; 2 D ′ T Σ t − 1 2 K t ; i ( Σ t − 1 2 ) T Q K t ; 2 D ′ (101) \mathbf{K}'_{t;i}=\mathbf{Q}_{K'_{t;2D}}^{T}\mathbf{\Sigma}_t^{-\frac{1}{2}}\mathbf{K}_{t;i}(\mathbf{\Sigma}_t^{-\frac{1}{2}})^{T}\mathbf{Q}_{K'_{t;2D}} \tag*{(101)} K t ; i ′ = Q K t ; 2 D ′ T Σ t − 2 1 K t ; i ( Σ t − 2 1 ) T Q K t ; 2 D ′ (101)
by the simultaneous diagonalisation of the other two time covariance matrices Σ t \mathbf{\Sigma}_t Σ t and K t ; 2 D \mathbf{K}_{t;2D} K t ; 2 D . Similar to the DOUBLEMULTILOWRANKKERNEL, we can break the decomposition of K ~ \widetilde{\mathbf{K}} K up as a 2x2 block matrix, where we can already decompose the A block using either the LUASKERNEL or LUASRANKKERNEL, depending on whether quasiseparable kernels are used. Once we can calculate C A − 1 B \mathbf{C}\mathbf{A}^{-1}\mathbf{B} C A − 1 B , general Cholesky decomposition is used to decompose D − C A − 1 B \mathbf{D}-\mathbf{C}\mathbf{A}^{-1}\mathbf{B} D − C A − 1 B , which introduces a scaling of O ( N α 3 N t 3 ) \mathcal{O}(N_{\alpha}^{3}N_t^{3}) O ( N α 3 N t 3 ) and tends to bottleneck this optimisation. See Appendix B for more details on the decomposition, including the calculation of C A − 1 B \mathbf{C}\mathbf{A}^{-1}\mathbf{B} C A − 1 B .
Computational Scaling
In Appendix B, we derive an overall runtime scaling of O ( ( 1 + N α 3 ) N t 3 + N λ N t 2 + N λ N t ( J λ + J K λ + 4 N α ) 2 ) \mathcal{O}((1+N_{\alpha}^{3})N_t^{3}+N_{\lambda}N_t^{2}+N_{\lambda}N_t(J_{\lambda}+J_{K_{\lambda}}+4N_{\alpha})^{2}) O (( 1 + N α 3 ) N t 3 + N λ N t 2 + N λ N t ( J λ + J K λ + 4 N α ) 2 ) for the kernel function in Equation (94) assuming all wavelength kernels are quasiseparable, which is linear in N λ N_{\lambda} N λ . If we instead use the LUASKERNEL in our decomposition of K ~ \widetilde{\mathbf{K}} K , then we do not require any quasiseparable kernels, but our decomposition scales as O ( N α 3 + ( 1 + N α 3 ) N t 3 + N λ N t ( N λ + N t + N α 2 ) ) \mathcal{O}(N_{\alpha}^{3} + (1 + N_{\alpha}^{3})N_{t}^{3} + N_{\lambda}N_{t}(N_{\lambda} + N_{t} + N_{\alpha}^{2})) O ( N α 3 + ( 1 + N α 3 ) N t 3 + N λ N t ( N λ + N t + N α 2 )) , cubic in both dimensions. However, if N t ≫ N λ N_{t} \gg N_{\lambda} N t ≫ N λ then this difference is negligible, and we present benchmarking with this LUASKERNEL approach in Figure G28 and Figure G29 for N α = 1 N_{\alpha}=1 N α = 1 on CPU and GPU respectively.
Figure G28. Compared to Figure G26 , we see that the runtimes are significantly more expensive when we set N α = 1 N_{\alpha}=1 N α = 1 instead of N β = 1 N_{\beta}=1 N β = 1 . This is to be expected, because the algorithm requires an eigendecomposition in the time dimension in this case. We have used the LUASKERNEL method to decompose in this scenario (see Appendix B.§ 1 ), which results in a runtime scaling of ∼ O ( ( 1 + N α 3 ) N t 3 + N t 3 + N λ N t ( N t + N λ + N α 2 ) ) \sim\mathcal{O}((1+N_{\alpha}^{3})N_{t}^{3}+N_{t}^{3}+N_{\lambda}N_{t}(N_{t}+N_{\lambda}+N_{\alpha}^{2})) ∼ O (( 1 + N α 3 ) N t 3 + N t 3 + N λ N t ( N t + N λ + N α 2 )) , but doesn’t require any quasiseparable kernels in either dimension. It has a similar runtime to the LUASKERNEL in Figure G10 , both primarily bottlenecked by the time covariance matrix eigendecomposition.
Figure G29. Compared to Figure G28 , running this kernel on GPU scales more efficiently to larger datasets, similar to the LUASKERNEL on GPU (Figure G11 ).
Alternatively, for the kernel function in Equation (96) we recover a runtime scaling of O ( ( 1 + N β 3 ) N λ 3 + N λ 2 N t + N λ N t ( J k t + 2 N β ) 2 ) \mathcal{O}((1 + N_{\beta}^{3})N_{\lambda}^{3} + N_{\lambda}^{2}N_{t} + N_{\lambda}N_{t}(J_{k_{t}} + 2N_{\beta})^{2}) O (( 1 + N β 3 ) N λ 3 + N λ 2 N t + N λ N t ( J k t + 2 N β ) 2 ) with quasiseparable k t ; 2 D k_{t;2D} k t ; 2 D and where we’ve assumed s t s_{t} s t is diagonal. This has a linear scaling in N t N_{t} N t , which can be seen from benchmarking in Figure G26 and Figure G27 for N β = 1 N_{\beta}=1 N β = 1 and N β = 4 N_{\beta}=4 N β = 4 respectively. For N β = 1 N_{\beta}=1 N β = 1 , fitting a large ( N λ , N t ) = ( 1024 , 16384 ) (N_{\lambda},N_{t})=(1024,16384) ( N λ , N t ) = ( 1024 , 16384 ) dataset had a log-likelihood runtime of 1.1 seconds on CPU.
Figure G26. Even for the more complex scenario of arbitrary measurement covariance, a 2D kernel term with a quasiseparable time kernel, and a low-rank term, we can use the LUASPLUSMULTILOWRANKKERNEL to scale linearly with the number of time points, provided the low-rank term has its vector outer product in time. We achieve a runtime scaling of O ( ( 1 + N β 3 ) N β 3 + N λ 2 N t + N λ N t ( J K t + 2 N β 2 ) ) \mathcal{O}((1+N_{\beta}^{3})N_{\beta}^{3}+N_{\lambda}^{2}N_{t}+N_{\lambda}N_{t}(J K_{t}+2N_{\beta}^{2})) O (( 1 + N β 3 ) N β 3 + N λ 2 N t + N λ N t ( J K t + 2 N β 2 )) in this case.
Figure G27. Compared to Figure G26 , increasing from N β = 1 N_{\beta}=1 N β = 1 to N β = 4 N_{\beta}=4 N β = 4 produces a further increase in runtime, but the linear scaling with the number of time points remains.
ExtendedLUASKERNEL: Putting it All Together
We finally arrive at the most flexible of kernel functions in this work:
k ( λ , λ ′ , t , t ′ ) = s λ ( λ , λ ′ ) s t ( t , t ′ ) + k λ ; 2 D ( λ , λ ′ ) k t ; 2 D ( t , t ′ ) + ∑ i = 1 N α α i ( λ ) α i ( λ ′ ) k t ; i ( t , t ′ ) + ∑ j = 1 N β k λ ; j ( λ , λ ′ ) β j ( t ) β j ( t ′ ) (102) k(\lambda,\lambda',t,t') = s_{\lambda}(\lambda,\lambda')s_{t}(t,t') + k_{\lambda;2D}(\lambda,\lambda')k_{t;2D}(t,t') + \sum_{i=1}^{N_{\alpha}}\alpha_{i}(\lambda)\alpha_{i}(\lambda')k_{t;i}(t,t') + \sum_{j=1}^{N_{\beta}}k_{\lambda;j}(\lambda,\lambda')\beta_{j}(t)\beta_{j}(t') \tag*{(102)} k ( λ , λ ′ , t , t ′ ) = s λ ( λ , λ ′ ) s t ( t , t ′ ) + k λ ; 2 D ( λ , λ ′ ) k t ; 2 D ( t , t ′ ) + i = 1 ∑ N α α i ( λ ) α i ( λ ′ ) k t ; i ( t , t ′ ) + j = 1 ∑ N β k λ ; j ( λ , λ ′ ) β j ( t ) β j ( t ′ ) (102)
which builds covariance matrices of the form
K = Σ λ ⊗ Σ t + K λ ; 2 D ⊗ K t ; 2 D + ∑ i = 1 N α α λ ; i α λ ; i T ⊗ K t ; i + ∑ j = 1 N β K λ ; j ⊗ β t ; j β t ; j T (103) \mathbf{K} = \mathbf{\Sigma}_{\lambda} \otimes\mathbf{\Sigma}_{t} + \mathbf{K}_{\lambda;2D} \otimes\mathbf{K}_{t;2D} + \sum_{i=1}^{N_{\alpha}}\boldsymbol{\alpha}_{\lambda;i}\boldsymbol{\alpha}_{\lambda;i}^{T} \otimes\mathbf{K}_{t;i} + \sum_{j=1}^{N_{\beta}}\mathbf{K}_{\lambda;j} \otimes\boldsymbol{\beta}_{t;j}\boldsymbol{\beta}_{t;j}^{T} \tag*{(103)} K = Σ λ ⊗ Σ t + K λ ; 2 D ⊗ K t ; 2 D + i = 1 ∑ N α α λ ; i α λ ; i T ⊗ K t ; i + j = 1 ∑ N β K λ ; j ⊗ β t ; j β t ; j T (103)
At this point, we abandon any attempt to exploit quasiseparability, and our scaling is dominated by O ( N α 3 N t 3 + N β 3 N λ 3 ) \mathcal{O}(N_{\alpha}^{3}N_{t}^{3} + N_{\beta}^{3}N_{\lambda}^{3}) O ( N α 3 N t 3 + N β 3 N λ 3 ) matrix decompositions. We perform a whitening transformation followed by a QL decomposition in both dimensions, factorising as follows:
K = [ Σ λ 1 2 Q A λ , Σ t 1 2 Q B t ] ⏟ K t r a n s f ∗ [ I N λ ⊗ I N t + K λ ; 2 D ′ ⊗ K t ; 2 D ′ + ∑ i = 1 N α [ 0 a i ] [ 0 a i ] T ⊗ K t ; i ′ + ∑ j = 1 N β K λ ; j ′ ⊗ [ 0 b j ] [ 0 b j ] T ] \mathbf{K} = \underbrace{\left[\mathbf{\Sigma}_{\lambda}^{\frac{1}{2}}\mathbf{Q}_{A_{\lambda}},\mathbf{\Sigma}_{t}^{\frac{1}{2}}\mathbf{Q}_{B_{t}}\right]}_{\mathbf{K}_{\mathrm{transf}}} \ast\left[\mathbf{I}_{N_{\lambda}} \otimes\mathbf{I}_{N_{t}} + \mathbf{K}'_{\lambda;2D} \otimes\mathbf{K}'_{t;2D} + \sum_{i=1}^{N_{\alpha}}\begin{bmatrix}\mathbf{0}\\ \mathbf{a}_{i}\end{bmatrix}\begin{bmatrix}\mathbf{0}\\ \mathbf{a}_{i}\end{bmatrix}^{T} \otimes\mathbf{K}'_{t;i} + \sum_{j=1}^{N_{\beta}}\mathbf{K}'_{\lambda;j} \otimes\begin{bmatrix}\mathbf{0}\\ \mathbf{b}_{j}\end{bmatrix}\begin{bmatrix}\mathbf{0}\\ \mathbf{b}_{j}\end{bmatrix}^{T}\right] K = K transf [ Σ λ 2 1 Q A λ , Σ t 2 1 Q B t ] ∗ I N λ ⊗ I N t + K λ ; 2 D ′ ⊗ K t ; 2 D ′ + i = 1 ∑ N α [ 0 a i ] [ 0 a i ] T ⊗ K t ; i ′ + j = 1 ∑ N β K λ ; j ′ ⊗ [ 0 b j ] [ 0 b j ] T
∗ [ Q A λ T , ( Σ λ 1 2 ) T Q B t T , ( Σ t 1 2 ) T ] ⏟ K t r a n s f T (104) \ast\underbrace{\left[\mathbf{Q}_{A_{\lambda}}^{T},(\mathbf{\Sigma}_{\lambda}^{\frac{1}{2}})^{T} \quad\mathbf{Q}_{B_{t}}^{T},(\mathbf{\Sigma}_{t}^{\frac{1}{2}})^{T}\right]}_{\mathbf{K}_{\mathrm{transf}}^{T}} \tag*{(104)} ∗ K transf T [ Q A λ T , ( Σ λ 2 1 ) T Q B t T , ( Σ t 2 1 ) T ] (104)
with K λ ; 2 D ′ \mathbf{K}'_{\lambda;2D} K λ ; 2 D ′ and all K λ ; j ′ \mathbf{K}'_{\lambda;j} K λ ; j ′ transformed as:
K λ ; 2 D ′ = Q A λ T Σ λ − 1 2 K λ ; 2 D ( Σ λ − 1 2 ) T Q A λ (105) \mathbf{K}'_{\lambda;2D} = \mathbf{Q}_{A_{\lambda}}^{T}\mathbf{\Sigma}_{\lambda}^{-\frac{1}{2}}\mathbf{K}_{\lambda;2D}(\mathbf{\Sigma}_{\lambda}^{-\frac{1}{2}})^{T}\mathbf{Q}_{A_{\lambda}} \tag*{(105)} K λ ; 2 D ′ = Q A λ T Σ λ − 2 1 K λ ; 2 D ( Σ λ − 2 1 ) T Q A λ (105)
K λ ; j ′ = Q A λ T Σ λ − 1 2 K λ ; j ( Σ λ − 1 2 ) T Q A λ (106) \mathbf{K}'_{\lambda;j} = \mathbf{Q}_{A_{\lambda}}^{T}\mathbf{\Sigma}_{\lambda}^{-\frac{1}{2}}\mathbf{K}_{\lambda;j}(\mathbf{\Sigma}_{\lambda}^{-\frac{1}{2}})^{T}\mathbf{Q}_{A_{\lambda}} \tag*{(106)} K λ ; j ′ = Q A λ T Σ λ − 2 1 K λ ; j ( Σ λ − 2 1 ) T Q A λ (106)
and similarly for K t ; 2 D ′ \mathbf{K}'_{t;2D} K t ; 2 D ′ and all K t ; i ′ \mathbf{K}'_{t;i} K t ; i ′ .
The decomposition of K ~ \widetilde{\mathbf{K}} K makes use of the LUASPLUSMULTILOWRANKKERNEL to decompose the A block of K ~ \widetilde{\mathbf{K}} K , while calculation of C A − 1 B \mathbf{C}\mathbf{A}^{-1}\mathbf{B} C A − 1 B is performed by splitting the A, B and C blocks into their own 2x2 block matrices and exploiting sparsity. The D − C A − 1 B \mathbf{D}-\mathbf{C}\mathbf{A}^{-1}\mathbf{B} D − C A − 1 B term is decomposed as a general ( N α N t , N α N t ) (N_{\alpha}N_{t},N_{\alpha}N_{t}) ( N α N t , N α N t ) matrix. See Appendix C for the full details of the decomposition.
Computational Scaling
In Appendix C, the runtime scaling is worked out to ∼ O ( ( 1 + N α 3 ) N t 3 + ( 1 + N β 3 ) N λ 3 + N α N β 2 N λ 2 N t + N λ N t 2 ) \sim\mathcal{O}((1 + N_{\alpha}^{3})N_{t}^{3} + (1 + N_{\beta}^{3})N_{\lambda}^{3} + N_{\alpha}N_{\beta}^{2}N_{\lambda}^{2}N_{t} + N_{\lambda}N_{t}^{2}) ∼ O (( 1 + N α 3 ) N t 3 + ( 1 + N β 3 ) N λ 3 + N α N β 2 N λ 2 N t + N λ N t 2 ) , with a memory scaling of ∼ O ( N α 2 N t 2 + N β 2 N λ 2 + N α N β N λ N t ) \sim\mathcal{O}(N_{\alpha}^{2}N_{t}^{2} + N_{\beta}^{2}N_{\lambda}^{2} + N_{\alpha}N_{\beta}N_{\lambda}N_{t}) ∼ O ( N α 2 N t 2 + N β 2 N λ 2 + N α N β N λ N t ) . It has a similar runtime to the LUASKERNEL when N α N_{\alpha} N α and N β N_{\beta} N β are both small (compare Figure G10 and Figure G30 ), which means it could certainly help to add flexibility to analyses where the LUASKERNEL has already been applied [44 , 45 ] . Similar to the LUASKERNEL, this optimisation is significantly faster on GPUs, as seen by comparing Figure G30 and Figure G31 . We also demonstrate GPU runtimes with ( N α , N β ) = ( 1 , 4 ) , ( 4 , 1 ) (N_{\alpha},N_{\beta})=(1,4),(4,1) ( N α , N β ) = ( 1 , 4 ) , ( 4 , 1 ) in Figure G32 and Figure G33 respectively.
Figure G30. The EXTENDEDLUASKERNEL scales less efficiently to larger datasets, particularly because it cannot exploit quasiseparable kernels in either dimension, with a runtime scaling of ∼ O ( ( 1 + N α 3 ) N t 3 + ( 1 + N β 3 ) N λ 3 + N α N β 2 N λ 2 N t + N t N λ 2 ) \sim\mathcal{O}\left((1+N_\alpha^3)N_t^3+(1+N_\beta^3)N_\lambda^3+N_\alpha N_\beta^2N_\lambda^2N_t+N_tN_\lambda^2\right) ∼ O ( ( 1 + N α 3 ) N t 3 + ( 1 + N β 3 ) N λ 3 + N α N β 2 N λ 2 N t + N t N λ 2 ) . It is worth noting however that for the case N α = N β = 1 N_\alpha=N_\beta=1 N α = N β = 1 , we still achieve runtimes similar to the LUASKERNEL (Figure G10 ) as well as the LUASPLUSMULTILOWRANKKERNEL with N α = 1 N_\alpha=1 N α = 1 (Figure G28 ), so we can add these extra low-rank terms with minimal extra computational expense.
Figure G31. Re-running the same kernel in Figure G30 on GPU, the calculation still appears to be bottlenecked by the eigendecomposition of a time covariance matrix (comparing with Figure 1), but this is much faster on the GPU.
Figure G32. Remaining on GPU, we see that we can increase to N β = 4 N_\beta=4 N β = 4 while keeping N α = 1 N_\alpha=1 N α = 1 with limited change in runtime from Figure G31 , likely because we are still bottlenecked by the same time covariance matrix eigendecomposition.
Figure G33. However, increasing to N α = 4 N_\alpha=4 N α = 4 while keeping N β = 1 N_\beta=1 N β = 1 is much more expensive, even on the GPU. In fact, for small N λ N_\lambda N λ this can sometimes be out-performed by a general Cholesky decomposition of the full covariance matrix (Figure G7 ), although it is much more efficient for larger N λ N_\lambda N λ . This expensive calculation is mostly due to the ( N α N t , N α N t ) (N_\alpha N_t,N_\alpha N_t) ( N α N t , N α N t ) Cholesky decomposition, combined with the additional costs due to the complex algorithm (as shown in Appendix C).
Summary and Further Calculations
We also include calculation details for the GP predictive mean and covariance in Appendix § 13 , which can exploit any of these optimisations, provided not only our input observed points lie on a grid, but also the combined observed and predicted points lie on a grid. This allows us to visualise draws from the 2D GP model conditioned on the data, and also to perform cross-validation using any of these models, e.g. leaving out particular wavelengths and/or time points. We also extend the SEPARABLEKERNEL, LUASKERNEL and LUASRASKERNEL to n n n -dimensional datasets in Appendix § 14 , and benchmark log-likelihood calculations on datacubes for both CPU and GPU in Figure G34 –Figure G37 . We demonstrate the high numerical stability of our algorithms in Appendix § 15 . Finally, Appendix § 16 shows how benchmarking was performed, Figure G1 summarises the runtime scaling for the different optimisations introduced, and the cost of gradient calculations is explained in Appendix § 16 .§ 1 .
Figure G34. Arbitrary, separable 3D kernel function implemented using the LUASKERNEL for a third data dimension with a fixed size of N m = 64 N_m = 64 N m = 64 . Runtimes are relatively slow on CPU, with a runtime scaling of O ( N m 3 + N λ 3 + N t 3 + N m N λ N t ( N m + N λ + N t ) ) O(N_m^3 + N_\lambda^3 + N_t^3 + N_mN_\lambda N_t(N_m + N_\lambda+ N_t)) O ( N m 3 + N λ 3 + N t 3 + N m N λ N t ( N m + N λ + N t )) , limited both by the cost of eigendecomposition and the large matrix multiplications, compared with the same model in Figure G36 .
Figure G37. The same as Figure G36 but using the LUASLASRACHKERNEL with an exponential kernel in time, also on GPU. The runtimes are largely comparable although the scaling with the number of time points is slightly better. We reach datasets with 2 29 2^{29} 2 29 datapoints (i.e. 4 GB) for the largest datacubes benchmarked.
Discussion
Comparison to Existing Methods
Principal Component Analysis (PCA)
PCA is a classic approach to fit, understand, compress, and clean data, which can be calculated using the Singular Value Decomposition (SVD) of a 2D data array. A PCA model of a spectroscopic time-series can be written as
M P C A = U λ Λ S V D V t T , (107) \mathbf{M}_{\mathrm{PCA}} = \mathbf{U}_{\lambda}\mathbf{\Lambda}_{\mathrm{SVD}}\mathbf{V}_{t}^{T}, \tag*{(107)} M PCA = U λ Λ SVD V t T , (107)
where assuming we use N α N_{\alpha} N α PCA components, U λ \mathbf{U}_{\lambda} U λ is ( N λ , N α ) (N_{\lambda},N_{\alpha}) ( N λ , N α ) , Λ S V D \mathbf{\Lambda}_{\mathrm{SVD}} Λ SVD is ( N α , N α ) (N_{\alpha},N_{\alpha}) ( N α , N α ) and diagonal, and V t \mathbf{V}_{t} V t is ( N t , N α ) (N_{t},N_{\alpha}) ( N t , N α ) . Both U λ \mathbf{U}_{\lambda} U λ and V t \mathbf{V}_{t} V t consist of orthonormal vectors, so they both satisfy U λ T U λ = V t T V t = I N α \mathbf{U}_{\lambda}^{T}\mathbf{U}_{\lambda}=\mathbf{V}_{t}^{T}\mathbf{V}_{t}=\mathbf{I}_{N_{\alpha}} U λ T U λ = V t T V t = I N α . We can alternatively express our PCA model in Equation (107) as a sum of outer products, i.e.
M P C A = ∑ i = 1 N α λ i u λ ; i v t ; i T (108) \mathbf{M}_{\mathrm{PCA}} = \sum_{i=1}^{N_{\alpha}}\lambda_{i}\mathbf{u}_{\lambda;i}\mathbf{v}_{t;i}^{T} \tag*{(108)} M PCA = i = 1 ∑ N α λ i u λ ; i v t ; i T (108)
where a classic result of PCA is that it produces the best low-rank model to minimise the sum of squared residuals from the data, i.e. it minimises the loss function
S 2 = ∑ i = 1 N λ ∑ j = 1 N t ( R i j − ∑ k = 1 N α λ k u i k v j k ) 2 σ 2 , (109) S^{2} = \sum_{i=1}^{N_{\lambda}}\sum_{j=1}^{N_{t}}\frac{\left(\mathbf{R}_{ij}-\sum_{k=1}^{N_{\alpha}}\lambda_{k}u_{ik}v_{jk}\right)^{2}}{\sigma^{2}}, \tag*{(109)} S 2 = i = 1 ∑ N λ j = 1 ∑ N t σ 2 ( R ij − ∑ k = 1 N α λ k u ik v j k ) 2 , (109)
for uniform white noise. Tipping and Bishop [116 ] demonstrate that PCA can be converted to a likelihood-based method, which they refer to as Probabilistic Principal Component Analysis (PPCA). Consider fitting a 2D dataset with a GP log-likelihood using the covariance matrix:
K = σ 2 I N λ ⊗ I N t + ∑ i = 1 N α α λ ; i α λ ; i T ⊗ I N t , (110) \mathbf{K} = \sigma^{2}\mathbf{I}_{N_\lambda} \otimes\mathbf{I}_{N_t} + \sum_{i=1}^{N_\alpha} \boldsymbol{\alpha}_{\lambda;i}\boldsymbol{\alpha}_{\lambda;i}^{T} \otimes\mathbf{I}_{N_t}, \tag*{(110)} K = σ 2 I N λ ⊗ I N t + i = 1 ∑ N α α λ ; i α λ ; i T ⊗ I N t , (110)
where we assume uniform white noise, a mean function of zero, and we freely fit for all of the elements within each vector α λ ; i \boldsymbol{\alpha}_{\lambda;i} α λ ; i . This model is equivalent to the MULTILOWRANKKERNEL with uncorrelated time kernels k t ; i = δ t t ′ k_{t;i} = \delta_{tt'} k t ; i = δ t t ′ . Alternatively, it is equivalent to measurement covariance, as it reduces to a single Kronecker term in the form of the SEPARABLEKERNEL
K = [ σ 2 I N λ + ∑ i = 1 N α α λ ; i α λ ; i T ] ⏟ Σ λ ⊗ I N t . (111) \mathbf{K} = \underbrace{\left[\sigma^{2}\mathbf{I}_{N_\lambda} + \sum_{i=1}^{N_\alpha} \boldsymbol{\alpha}_{\lambda;i}\boldsymbol{\alpha}_{\lambda;i}^{T}\right]}_{\Sigma_\lambda} \otimes\mathbf{I}_{N_t}. \tag*{(111)} K = Σ λ [ σ 2 I N λ + i = 1 ∑ N α α λ ; i α λ ; i T ] ⊗ I N t . (111)
Tipping and Bishop (1999) derive the best-fit values of α λ ; i \boldsymbol{\alpha}_{\lambda;i} α λ ; i analytically for exactly this covariance model. The maximum GP log-likelihood occurs at
α λ ; i = U λ [ Λ λ − σ 2 I N α ] 1 2 W e ^ i , (112) \boldsymbol{\alpha}_{\lambda;i} = \mathbf{U}_\lambda\left[\mathbf{\Lambda}_\lambda- \sigma^{2}\mathbf{I}_{N_\alpha}\right]^{\frac{1}{2}}\mathbf{W}\hat{\mathbf{e}}_i, \tag*{(112)} α λ ; i = U λ [ Λ λ − σ 2 I N α ] 2 1 W e ^ i , (112)
where U λ \mathbf{U}_\lambda U λ and Λ λ \mathbf{\Lambda}_\lambda Λ λ come from the eigendecomposition of the sample wavelength covariance of the data. Equivalently, U λ \mathbf{U}_\lambda U λ match the SVD components in Equation (107) , while the eigenvalues are related by Λ λ = Λ S V D 2 / N t \mathbf{\Lambda}_\lambda= \mathbf{\Lambda}_{\mathrm{SVD}}^{2}/N_t Λ λ = Λ SVD 2 / N t . W \mathbf{W} W is an arbitrary ( N α , N α ) (N_\alpha,N_\alpha) ( N α , N α ) unitary matrix satisfying W W T = I N α \mathbf{W}\mathbf{W}^{T} = \mathbf{I}_{N_\alpha} W W T = I N α , which appears due to a degeneracy in our model, which can be seen if Equation (112) is plugged into Equation (111) . Since W \mathbf{W} W has no impact on our model, α λ ; i \boldsymbol{\alpha}_{\lambda;i} α λ ; i matching the principal PCA components in U λ \mathbf{U}_\lambda U λ maximises the log-likelihood when we weight each component by ( λ i − σ 2 ) (\lambda_i-\sigma^2) ( λ i − σ 2 ) .
The GP mean for this model is
μ ∗ = U λ [ I N α − σ 2 Λ λ − 1 ] U λ T R , (113) \boldsymbol{\mu}_* = \mathbf{U}_\lambda\left[\mathbf{I}_{N_\alpha} - \sigma^{2}\mathbf{\Lambda}_\lambda^{-1}\right]\mathbf{U}_\lambda^{T}\mathbf{R}, \tag*{(113)} μ ∗ = U λ [ I N α − σ 2 Λ λ − 1 ] U λ T R , (113)
regardless of W \mathbf{W} W , while the PCA model in Equation (107) can equivalently be calculated as
M P C A = U λ U λ T R . (114) \mathbf{M}_{\mathrm{PCA}} = \mathbf{U}_\lambda\mathbf{U}_\lambda^{T}\mathbf{R}. \tag*{(114)} M PCA = U λ U λ T R . (114)
We see that our GP predictive mean approaches the PCA model in the limit as σ → 0 \sigma\to0 σ → 0 . Also in this limit, the predictive covariance goes to zero, which means the GP probability distribution becomes fixed at the GP mean. Therefore, PCA is equivalent to the GP model in Equation (110) , using its best-fit hyperparameters, in the limit of negligible white noise. When we include white noise, this GP model can be interpreted as a likelihood-based version of PCA which downweights components in the presence of white noise; analytically marginalises over the components in precisely one of the two dimensions; and can be applied to the data residuals after the subtraction of a mean function.
Gaussian Process PCA (GPPCA)
Note that Equation (110) can be generalised to
K = σ 2 I N λ ⊗ I N t + ∑ i = 1 N α α λ ; i α λ ; i T ⊗ K t ; i (115) \mathbf{K} = \sigma^{2}\mathbf{I}_{N_\lambda} \otimes\mathbf{I}_{N_t} + \sum_{i=1}^{N_\alpha} \boldsymbol{\alpha}_{\lambda;i}\boldsymbol{\alpha}_{\lambda;i}^{T} \otimes\mathbf{K}_{t;i} \tag*{(115)} K = σ 2 I N λ ⊗ I N t + i = 1 ∑ N α α λ ; i α λ ; i T ⊗ K t ; i (115)
which is simply the MULTILOWRANKKERNEL, in this case with uniform white noise. This is equivalent to PPCA, but with the option to place smoothness priors v t , i ∼ N ( 0 , K t ; i ) v_{t,i} \sim\mathcal{N}(0,\mathbf{K}_{t;i}) v t , i ∼ N ( 0 , K t ; i ) on the time-components. It can be referred to as Gaussian Process PCA, or Generalised Probabilistic PCA (GPPCA) (Gu and Shen (2018)), and this link between PCA and GPs is also made in Bruinsma et al. (2020) for their Orthogonal Instantaneous Linear Mixing Model (OILMM). Both of these works provide GP optimisations for this model, which become equivalent to the MULTILOWRANKKERNEL in the case of uniform white noise and orthogonal α λ ; i \boldsymbol{\alpha}_{\lambda;i} α λ ; i vectors. The optimisations in our work extend further, able to handle heterogeneous noise, measurement covariance, and non-orthogonal α λ ; i \boldsymbol{\alpha}_{\lambda;i} α λ ; i vectors, which make it more equivalent to factor analysis models, described in the following sections.
We argue for three potential advantages of GPPCA over ordinary PCA:
GPPCA is a likelihood-based approach that can be joint-fit with a mean function describing physical signals in the data
It marginalises over a space of models analytically
We can specify “smoothness priors” on components, which may be physically-motivated
The first advantage could be particularly useful compared to current approaches which apply PCA as a pre-cleaning step, such as ground-based high-resolution transit spectroscopy, where pre-cleaning using PCA or SysRem has the ability to remove some of the exoplanet signal before it is ever fit (e.g. Birkby et al. (2013); Gibson et al. (2022); Maguire et al. (2024)). Our optimisations could permit efficient joint-inference of a modelled exoplanet signal combined with a GPPCA systematics model applied to the residuals of the data, potentially leading to less signal loss.
The second advantage could also help avoid signal loss when data cleaning, since marginalising over a space of models may make it less likely for the model to subtract out the exact exoplanet signal, since the best-fit model may be overfitting the data. It also allows us to marginalise over the uncertainty in the systematics correction, propagating this uncertainty into the atmospheric constraints in a Bayesian framework.
The third option could be useful for more closely modelling real physical signals. For example, PCA is sometimes used in astronomy to separate spectra which vary as continuous, time-correlated signals (e.g. modelling brown dwarf variability; Wang et al. (2026); Schrader et al. (2026)), despite PCA not being weighted towards fitting smooth signals at all. This gives us an option to weight towards physically-motivated time-correlations, including with periodic kernels, or damped-harmonic oscillator kernels. It could even help disentangle two signals with the same amplitude but different time-correlations, which PCA cannot do (see § 6.6 ).
Factor Analysis
Factor analysis, as described by Bartholomew (1987), generalises Probabilistic PCA to heterogeneous white noise4
K = Σ + ∑ i = 1 N α α λ ; i α λ ; i T ⊗ I N t , (116) \mathbf{K} = \mathbf{\Sigma} + \sum_{i=1}^{N_\alpha} \boldsymbol{\alpha}_{\lambda;i}\boldsymbol{\alpha}_{\lambda;i}^{T} \otimes\mathbf{I}_{N_t}, \tag*{(116)} K = Σ + i = 1 ∑ N α α λ ; i α λ ; i T ⊗ I N t , (116)
where Σ \mathbf{\Sigma} Σ is diagonal, but not necessarily uniform or separable. Note the similarity with the popular data cleaning algorithm SysRem (Tamuz et al. (2005)), which also generalises PCA to non-uniform white noise. However, factor analysis is analogous to the PPCA generalisation of PCA, as it also marginalises over components, makes use of mean functions, and downweights components in the presence of noise.
If the noise Σ \boldsymbol{\Sigma} Σ is not separable in each dimension, there is no analytical solution that immediately solves for the best principal components, like SVD does for PCA, with iterative methods normally used to find the optimal α λ ; i \boldsymbol{\alpha}_{\lambda;i} α λ ; i . We also do not expect to recover orthogonal vectors α λ ; i \boldsymbol{\alpha}_{\lambda;i} α λ ; i for our best-fit solution. If Σ = Σ λ ⊗ I t \boldsymbol{\Sigma}=\boldsymbol{\Sigma}_{\lambda}\otimes\mathbf{I}_{t} Σ = Σ λ ⊗ I t is separable in each dimension on a grid of inputs, we can factor in the low-rank terms similar to Equation (110) , and model the full covariance matrix K \mathbf{K} K as a single Kronecker product using the SEPARABLEKERNEL.
Gaussian Process Factor Analysis (GPFA)
Similar to how GPPCA extends PCA to include a smoothness prior on time components, GP factor analysis makes the same generalisation for factor analysis, this time with heterogeneous noise present. It is equivalent to modelling using the covariance matrix:
K = Σ + ∑ i = 1 N α α λ ; i α λ ; i T ⊗ K t ; i (117) \mathbf{K}=\boldsymbol{\Sigma}+\sum_{i=1}^{N_{\alpha}}\boldsymbol{\alpha}_{\lambda;i}\boldsymbol{\alpha}_{\lambda;i}^{T}\otimes\mathbf{K}_{t;i} \tag*{(117)} K = Σ + i = 1 ∑ N α α λ ; i α λ ; i T ⊗ K t ; i (117)
Introduced in Yu et al. (2008) to model time-series of multiple neurons in macaques, the MULTILOWRANKKERNEL generalises it further to include measurement covariance, provided Σ \boldsymbol{\Sigma} Σ is separable in each dimension Σ = Σ λ ⊗ D t \boldsymbol{\Sigma}=\boldsymbol{\Sigma}_{\lambda}\otimes\mathbf{D}_{t} Σ = Σ λ ⊗ D t . While previous optimisations (e.g. Gu and Shen 2018; Bruinsma et al. 2020) have been restricted to orthogonal components and/or homogeneous noise, we can handle non-orthogonal components and measurement covariance while still being exact and scalable. We summarise the links between the various data-driven methods introduced and two of our optimisations in Figure 6 .
Figure 6. Comparison between different data-driven algorithms, with methods gaining in generality moving right and down. Therefore the LUASL...KERNEL and MULTILOWRANKKERNEL can be applied to any model to the left of them, provided the noise is separable, although the SEPARABLEKERNEL can more efficiently handle Probabilistic PCA (PPCA) and Factor Analysis (FA). Note that both the LUASL...KERNEL and MULTILOWRANKKERNEL can account for heterogeneous noise, the difference is the LUASL...KERNEL is limited to a shared kernel function between all components, and those components come from eigendecomposition and therefore are all orthogonal, while the MULTILOWRANKKERNEL can have non-orthogonal components each with different kernel functions. Plot inspired by Bruinsma et al. [19 ] , who present a model similar to GPPCA (Gu and Shen 2018).
In practice, heterogeneous noise and measurement covariance are very common in astronomy, therefore GP factor analysis may be even more relevant than GPPCA. Astronomical examples equivalent to (or similar to) GP factor analysis include various approaches which attempt to disentangle different components of spectra, such as the use of ‘multidimensional GPs’ within radial velocities to identify stellar activity components (e.g. Rajpaul et al. 2015; Barragán et al. 2022), distinguishing spectroscopic binaries (Czekala et al. 2017), or Heteroscedastic Matrix Factorisation (HMF), introduced to distinguish galactic spectra at different redshifts (Tsalmantza and Hogg 2012), although for HMF the smoothing priors are merely inspired by GP priors. Not all of these examples necessarily have complete, rectangular datasets and separable kernels in each dimension, required for the MULTILOWRANKKERNEL optimisation to be used, but we discuss applications to radial velocities in § 6.12 .
Analytical Marginalisation over Separable Linear Parameters
Consider a similar model to PCA or factor analysis, but where we have a model for building the wavelength components α ( λ i ) = α λ ; i \boldsymbol{\alpha}(\lambda_i)=\boldsymbol{\alpha}_{\lambda;i} α ( λ i ) = α λ ; i , rather than freely fitting for their elements. In this case, our GP covariance matrix becomes equivalent to analytically marginalising out time components with Gaussian priors, convenient for reducing parameters to fit. For example, consider the GP model of a 2D dataset:
Y ∼ N ( M ′ + α λ ω t T , Σ ) , (118) \mathbf{Y}\sim\mathcal{N}\left(\mathbf{M}^{\prime}+\boldsymbol{\alpha}_{\lambda}\boldsymbol{\omega}_{t}^{T},\boldsymbol{\Sigma}\right), \tag*{(118)} Y ∼ N ( M ′ + α λ ω t T , Σ ) , (118)
where M = M ′ + α λ ω t T \mathbf{M}=\mathbf{M}^{\prime}+\boldsymbol{\alpha}_{\lambda}\boldsymbol{\omega}_{t}^{T} M = M ′ + α λ ω t T is our mean function model, equal to a vector outer product α λ ω t T \boldsymbol{\alpha}_{\lambda}\boldsymbol{\omega}_{t}^{T} α λ ω t T plus some remaining model M ′ \mathbf{M}^{\prime} M ′ . Suppose we want a Gaussian prior on ω t ∼ N ( 0 , K t ) \boldsymbol{\omega}_{t}\sim\mathcal{N}(\mathbf{0},\mathbf{K}_{t}) ω t ∼ N ( 0 , K t ) , where K t \mathbf{K}_{t} K t may or may not be diagonal, and our noise covariance is modelled by Σ \boldsymbol{\Sigma} Σ , which may be a sum of Kronecker products. Rather than fitting for the values of ω t \boldsymbol{\omega}_{t} ω t , we can analytically marginalise them out. Following Luger et al. (2017), we can switch to the (marginal) GP log-likelihood:
y ∼ N ( M ′ , Σ + α λ α λ T ⊗ K t ) , (119) \mathbf{y}\sim\mathcal{N}\left(\mathbf{M}^{\prime},\boldsymbol{\Sigma}+\boldsymbol{\alpha}_{\lambda}\boldsymbol{\alpha}_{\lambda}^{T}\otimes\mathbf{K}_{t}\right), \tag*{(119)} y ∼ N ( M ′ , Σ + α λ α λ T ⊗ K t ) , (119)
where we’ve essentially moved some of our mean function into the GP covariance matrix. This new covariance matrix is equivalent to the MULTILOWRANKKERNEL with N α = 1 N_{\alpha}=1 N α = 1 when the covariance matrix Σ \boldsymbol{\Sigma} Σ can be expressed as a single Kronecker product, or the LUASPLUSMULTILOWRANKKERNEL when Σ \boldsymbol{\Sigma} Σ is a sum of two Kronecker products. The reverse of this process is also valid, any low-rank terms in our GP kernel function α λ α λ T ⊗ K t \boldsymbol{\alpha}_{\lambda}\boldsymbol{\alpha}_{\lambda}^{T}\otimes\mathbf{K}_{t} α λ α λ T ⊗ K t could instead be marginalised over via MCMC sampling by adding a low-rank term α λ ω t T \boldsymbol{\alpha}_{\lambda}\boldsymbol{\omega}_{t}^{T} α λ ω t T to the mean function and setting the prior ω t ∼ N ( 0 , K t ) \boldsymbol{\omega}_{t}\sim\mathcal{N}(\mathbf{0},\mathbf{K}_{t}) ω t ∼ N ( 0 , K t ) . Naturally, the analytical marginalisation approach can be more computationally favourable if N t N_t N t is very large, as it avoids freely-sampling thousands of parameters. That said, gradient-based inference can be highly scalable, so sampling low-rank kernel terms by moving them to the mean function may be more efficient if it permits a faster GP optimisation, particularly if the kernel function can also be approximated using a small number of low-rank components (see § 6.9 ).
If we want distinct Gaussian priors on multiple linear parameters, and also to marginalise out parameters in both dimensions, e.g.
Y = μ ( θ ) + ∑ i = 1 N α α λ ; i ω t ; i T + ∑ j = 1 N β ω λ ; j β t ; j T + E λ t , (120) \mathbf{Y}=\boldsymbol{\mu}(\theta)+\sum_{i=1}^{N_{\alpha}}\boldsymbol{\alpha}_{\lambda;i}\boldsymbol{\omega}_{t;i}^{T}+\sum_{j=1}^{N_{\beta}}\boldsymbol{\omega}_{\lambda;j}\boldsymbol{\beta}_{t;j}^{T}+\mathbf{E}_{\lambda t}, \tag*{(120)} Y = μ ( θ ) + i = 1 ∑ N α α λ ; i ω t ; i T + j = 1 ∑ N β ω λ ; j β t ; j T + E λ t , (120)
with ω t ; i ∼ N ( 0 , K t ; i ) \boldsymbol{\omega}_{t;i}\sim\mathcal{N}(\mathbf{0},\mathbf{K}_{t;i}) ω t ; i ∼ N ( 0 , K t ; i ) , ω λ ; j ∼ N ( 0 , K λ ; j ) \boldsymbol{\omega}_{\lambda;j}\sim\mathcal{N}(\mathbf{0},\mathbf{K}_{\lambda;j}) ω λ ; j ∼ N ( 0 , K λ ; j ) , and some 2D noise draw E λ , t ∼ N ( 0 , Σ ) \mathbf{E}_{\lambda,t}\sim\mathcal{N}(\mathbf{0},\boldsymbol{\Sigma}) E λ , t ∼ N ( 0 , Σ ) , then marginalising over all ω \boldsymbol{\omega} ω is equivalent to the GP model
y ∼ N ( μ , Σ + ∑ i = 1 N α α λ ; i α λ ; i T ⊗ K t ; i + ∑ j = 1 N β K λ ; j ⊗ β t ; j β t ; j T ) , (121) \mathbf{y}\sim\mathcal{N}\left(\boldsymbol{\mu},\boldsymbol{\Sigma}+\sum_{i=1}^{N_{\alpha}}\boldsymbol{\alpha}_{\lambda;i}\boldsymbol{\alpha}_{\lambda;i}^{T}\otimes\mathbf{K}_{t;i}+\sum_{j=1}^{N_{\beta}}\mathbf{K}_{\lambda;j}\otimes\boldsymbol{\beta}_{t;j}\boldsymbol{\beta}_{t;j}^{T}\right), \tag*{(121)} y ∼ N μ , Σ + i = 1 ∑ N α α λ ; i α λ ; i T ⊗ K t ; i + j = 1 ∑ N β K λ ; j ⊗ β t ; j β t ; j T , (121)
which is the DOUBLEMULTILOWRANKKERNEL when Σ \boldsymbol{\Sigma} Σ is diagonal and separable. If Σ \boldsymbol{\Sigma} Σ is the sum of two Kronecker products, i.e. if there are also wavelength- and time-correlated systematics we are marginalising over, then we can instead use the EXTENDEDLUASKERNEL.
Analytic marginalisation can be useful when our mean function μ ( θ ) \boldsymbol{\mu}(\theta) μ ( θ ) contains linear nuisance parameters, as we can remove them from the mean function and analytically marginalise them out, reducing the number of free parameters. As a simple example, ( β t ) i = 1 (\beta_t)_i=1 ( β t ) i = 1 and K λ = c 2 I N λ \mathbf{K}_{\lambda}=c^2\mathbf{I}_{N_{\lambda}} K λ = c 2 I N λ would allow us to marginalise over independent additive offsets to each light curve, which each have a Gaussian prior with standard deviation c c c .
Blind Signal Separation (BSS)
Blind Signal Separation (BSS) refers to algorithms which distinguish different components out of an observed mixture of signals, with no a priori information about the content of the signals, and limited assumptions about its structure. A classic example is the ‘cocktail party’ problem. For our purposes, it could be trying to distinguish the wavelength-dependence of instrumental systematics apart from stellar variability. PCA and factor analysis may be considered examples of BSS algorithms, the goal of the method is what’s different: the emphasis is on separating and distinguishing each component.
Independent Component Analysis (ICA) is strongly associated with BSS and has been used within exoplanet research before [126 , 88 ] . It works by assuming signals present in the data are non-Gaussian, optimising individual components to maximise some measure of non-Gaussianity, e.g. the kurtosis of the signal amplitude over time [69 ] . In contrast, methods like PPCA, GPPCA, factor analysis, and GP factor analysis, all assume a mixture of Gaussian signals. As noted in Equation (112) , PPCA components are degenerate up to an arbitrary orthonormal transformation labelled W W W (and factor analysis has a similar degeneracy), which limits their ability to separate components. In fact, PCA only distinguishes each component by maximising the variance of each successive component, so a mixture of signals with similar amplitudes will be poorly distinguished by PCA.
Unlike PCA or factor analysis, GPPCA and GP factor analysis can uniquely distinguish components with similar amplitudes if they have different time covariance. We demonstrate this in Figure 7 on a simulated dataset with two low-rank kernel terms, fitting for each wavelength component initialised from random normal vectors, optimising using Limited-Memory BFGS, and correctly extracting the injected time-varying signals. Depending on the kernel function, we can distinguish signals based on whether they vary smoothly or sharply in time, whether they are periodic, or whether they correlate with some other choice of input regressor. For example, the wavelength-dependence of instrumental systematics could be distinguished from stellar variability if they vary on different timescales. In fact, this is already similar to an existing method of BSS introduced by Belouchrani et al. [13 ] called Second-Order Blind Identification (SOBI). SOBI calculates autocovariance matrices with different amounts of time lag applied when cross-correlating the residuals, to distinguish components which are correlated on different timescales. They prove that components can be distinguished based on different time covariance up to a sign flip (or complex phase shift), provided each time kernel has a different length scale or shape.
Interestingly, the transformed residuals R ~ \tilde{\mathbf{R}} R ~ that the MULTILOWRANKKERNEL calculates to simplify log-likelihood calculations can also be used to visualise each component in time, as highlighted in Figure 7 . This is a consequence of the transformed residuals being distributed according to the transformed covariance matrix R ~ ∼ N ( 0 , K ~ ) \tilde{\mathbf{R}} \sim\mathcal{N}(0,\tilde{\mathbf{K}}) R ~ ∼ N ( 0 , K ~ ) , provided we are modelling the overall covariance matrix correctly. For orthogonal α λ , i \boldsymbol{\alpha}_{\lambda,i} α λ , i , their transformed covariance matrix is block-diagonal, thus the transformed residuals isolate statistically independent components. The MULTILOWRANKKERNEL can thus be viewed as a method of BSS in a Bayesian framework, for signals which can be modelled as Gaussian processes in one dimension. We note that even when their wavelength-dependence can be distinguished, if α λ , i \boldsymbol{\alpha}_{\lambda,i} α λ , i are non-orthogonal, the projection to calculate their time components is not unique, the QR-decomposition used by the MULTILOWRANKKERNEL will just be one choice of orthonormal basis for the set of vectors, and the transformed components will not be statistically independent. For Figure 7 , the components were simulated to be orthogonal so that we can visualise each component separately.
Fourier-based GP Optimisations
Suppose we have data on a regularly-sampled 2D input grid, where the input grid has periodic boundary conditions (i.e. the endpoints become correlated with each other), and we have uniform white noise. This is a regime where Fourier-based GP optimisations can be performed exactly, and it has an interesting connection with the Rakitsch et al. [98 ] method implemented by the LUASKERNEL.
In this case, our covariance matrix can be written as:
K = σ 2 I N λ ⊗ I N t + K λ ⊗ K t , (122) \mathbf{K} = \sigma^2 \mathbf{I}_{N_{\lambda}} \otimes\mathbf{I}_{N_t} + \mathbf{K}_{\lambda} \otimes\mathbf{K}_t, \tag*{(122)} K = σ 2 I N λ ⊗ I N t + K λ ⊗ K t , (122)
One consequence of our assumptions is that both K λ \mathbf{K}_{\lambda} K λ and K t \mathbf{K}_t K t will be symmetric circulant matrices, i.e. they are characterised by a single row of values which get cycled to the right one row at a time, for example if N λ = 6 N_{\lambda}=6 N λ = 6 , then we can express K λ \mathbf{K}_{\lambda} K λ as
K λ = ( c 0 c 1 c 2 c 3 c 2 c 1 c 1 c 0 c 1 c 2 c 3 c 2 c 2 c 1 c 0 c 1 c 2 c 3 c 3 c 2 c 1 c 0 c 1 c 2 c 2 c 3 c 2 c 1 c 0 c 1 c 1 c 2 c 3 c 2 c 1 c 0 ) (123) \mathbf{K}_{\lambda} =
\begin{pmatrix}
c_0 & c_1 & c_2 & c_3 & c_2 & c_1 \\
c_1 & c_0 & c_1 & c_2 & c_3 & c_2 \\
c_2 & c_1 & c_0 & c_1 & c_2 & c_3 \\
c_3 & c_2 & c_1 & c_0 & c_1 & c_2 \\
c_2 & c_3 & c_2 & c_1 & c_0 & c_1 \\
c_1 & c_2 & c_3 & c_2 & c_1 & c_0
\end{pmatrix} \tag*{(123)} K λ = c 0 c 1 c 2 c 3 c 2 c 1 c 1 c 0 c 1 c 2 c 3 c 2 c 2 c 1 c 0 c 1 c 2 c 3 c 3 c 2 c 1 c 0 c 1 c 2 c 2 c 3 c 2 c 1 c 0 c 1 c 1 c 2 c 3 c 2 c 1 c 0 (123)
One property of circulant matrices is that their eigenvectors are given by Fourier modes, i.e. we can express the eigenvector matrix analytically as:
( Q K λ ) j k = 1 N λ ω ( j − 1 ) ( k − 1 ) , (124) (\mathbf{Q}_{K_\lambda})_{jk} = \frac{1}{\sqrt{N_\lambda}}\omega^{(j-1)(k-1)}, \tag*{(124)} ( Q K λ ) j k = N λ 1 ω ( j − 1 ) ( k − 1 ) , (124)
ω = exp ( 2 π i N λ ) (125) \omega= \exp\left(\frac{2\pi i}{N_\lambda}\right) \tag*{(125)} ω = exp ( N λ 2 π i ) (125)
where ω \omega ω is a root of unity. The eigenvalues can also be efficiently calculated, and are related to the kernel functions power spectrum. The interesting connection here is that multiplying a vector by this eigenvector matrix is exactly equivalent to the inverse Discrete Fourier Transform (DFT), and multiplying by its conjugate transpose Q K λ † = W D F T ; λ \mathbf{Q}_{K_\lambda}^{\dagger} = \mathbf{W}_{\mathrm{DFT};\lambda} Q K λ † = W DFT ; λ applies the DFT to the data residuals along the wavelength dimension. If both K λ \mathbf{K}_\lambda K λ and K t \mathbf{K}_t K t are circulant, then the LuaSKernel will transform residuals as
K t r a n s f − 1 = Q K λ † ⊗ Q K t † = W D F T ; λ ⊗ W D F T ; t (126) \mathbf{K}_{\mathrm{transf}}^{-1} = \mathbf{Q}_{K_\lambda}^{\dagger}\otimes\mathbf{Q}_{K_t}^{\dagger} = \mathbf{W}_{\mathrm{DFT};\lambda}\otimes\mathbf{W}_{\mathrm{DFT};t} \tag*{(126)} K transf − 1 = Q K λ † ⊗ Q K t † = W DFT ; λ ⊗ W DFT ; t (126)
where our transformation of the residuals is exactly the 2D DFT, and can be performed in O ( N λ N t log ( N λ N t ) ) \mathcal{O}(N_\lambda N_t\log(N_\lambda N_t)) O ( N λ N t log ( N λ N t )) runtime using Fast Fourier Transforms (FFTs) applied along both wavelength and time axes. Note that while we have included the eigenvector matrix in a complex form to be consistent with the FFT, all the phase information is lost when we calculate the amplitude of the residuals, and there are specific FFT optimisations for real signals.
Not only is the transformation using the 2D FFT faster to perform than for arbitrary eigenvector matrices, but we also avoid any need to perform eigendecomposition as we know it analytically, removing the main bottleneck of the LuaSKernel. Simply performing the 2D FFT on our residuals diagonalises our log-likelihood, and our transformed log-likelihood becomes a simple weighted sum of squares in Fourier space. So the LuaSKernel becomes equivalent to the Fourier-based approach under common assumptions made for exact Fourier-based GP optimisations, regular sampling, homogeneous noise, and periodic boundaries. The LuaSKernel is an exact generalisation for heterogeneous noise, irregular sampling, and non-periodic bounds, while the LuaSKernel extends further by combining with fast GP optimisations to achieve linear runtime scaling in the number of time points.
Further Extensions
Fourier-based Optimisations in Individual Dimensions
There are cases where we can mix Fourier methods with our optimisations, which can remain exact when we have circulant covariance matrices in at least one of the dimensions. For example, consider a covariance matrix of the form
K = Σ λ ⊗ I N t + K λ ⊗ K t ; p e r i o d i c (127) \mathbf{K} = \boldsymbol{\Sigma}_\lambda\otimes\mathbf{I}_{N_t} + \mathbf{K}_\lambda\otimes\mathbf{K}_{t;\mathrm{periodic}} \tag*{(127)} K = Σ λ ⊗ I N t + K λ ⊗ K t ; periodic (127)
= [ Σ λ 1 2 Q K λ † ⊗ W D F T ; t † ] [ I N λ ⊗ I N t + Λ K λ † ⊗ Λ K t † ] (128) = \left[\boldsymbol{\Sigma}_\lambda^{\frac{1}{2}}\mathbf{Q}_{K_\lambda}^{\dagger}\otimes\mathbf{W}_{\mathrm{DFT};t}^{\dagger}\right]\left[\mathbf{I}_{N_\lambda}\otimes\mathbf{I}_{N_t}+\mathbf{\Lambda}_{K_\lambda}^{\dagger}\otimes\mathbf{\Lambda}_{K_t}^{\dagger}\right] \tag*{(128)} = [ Σ λ 2 1 Q K λ † ⊗ W DFT ; t † ] [ I N λ ⊗ I N t + Λ K λ † ⊗ Λ K t † ] (128)
∗ [ Q K λ † ( Σ λ 1 2 ) † ⊗ W D F T ; t ] . \quad* \left[\mathbf{Q}_{K_\lambda}^{\dagger}\left(\boldsymbol{\Sigma}_\lambda^{\frac{1}{2}}\right)^{\dagger}\otimes\mathbf{W}_{\mathrm{DFT};t}\right]. ∗ [ Q K λ † ( Σ λ 2 1 ) † ⊗ W DFT ; t ] .
where K t ; p e r i o d i c \mathbf{K}_{t;\mathrm{periodic}} K t ; periodic is built using periodic kernels (plus a constant diagonal if desired), and its eigenvector matrix exactly corresponds to a DFT matrix when we have regular time-sampling observed over an integer number of periods, since the matrix is circulant (see § 6.7 ). We can transform our residuals using K t r a n s f − 1 = Q K λ T Σ λ − 1 2 ⊗ W D F T ; t \mathbf{K}_{\mathrm{transf}}^{-1} = \mathbf{Q}_{K_\lambda}^{T}\boldsymbol{\Sigma}_\lambda^{-\frac{1}{2}}\otimes\mathbf{W}_{\mathrm{DFT};t} K transf − 1 = Q K λ T Σ λ − 2 1 ⊗ W DFT ; t , which can be performed using a Fast Fourier Transform along the time-dimension, combined with a simultaneous diagonalisation in wavelength. The log-likelihood would have a runtime scaling of O ( N λ 3 + N λ 2 N t + N λ N t log N t ) \mathcal{O}(N_\lambda^3 + N_\lambda^2N_t + N_\lambda N_t\log N_t) O ( N λ 3 + N λ 2 N t + N λ N t log N t ) , close to linear with the number of time points. Other Fourier-based methods could be explored in future work for the case of non-uniform time-sampling, or non-periodic time kernels, although these approaches are often approximate.
Low-Rank Approximations of Covariance Matrices
We have frequently distinguished between 2D kernel terms (i.e. the product of two kernel functions) and low-rank kernel terms (see § 2.4 ). On a grid of inputs, 2D kernel terms produce Kronecker product covariance matrices
K 2 D = K λ ; 2 D ⊗ K t ; 2 D (129) \mathbf{K}_{\mathrm{2D}} = \mathbf{K}_{\lambda;\mathrm{2D}} \otimes\mathbf{K}_{t;\mathrm{2D}} \tag*{(129)} K 2D = K λ ; 2D ⊗ K t ; 2D (129)
where both K λ ; 2 D \mathbf{K}_{\lambda;\mathrm{2D}} K λ ; 2D and K t ; 2 D \mathbf{K}_{t;\mathrm{2D}} K t ; 2D may be full-rank. In contrast, optimisations like MULTILOWRANKKERNEL focus on sums of low-rank kernel terms, i.e.
K l o w − r a n k = ∑ i α λ ; i α λ ; i T ⊗ K t ; i . (130) \mathbf{K}_{\mathrm{low-rank}} = \sum_i \boldsymbol{\alpha}_{\lambda;i}\boldsymbol{\alpha}_{\lambda;i}^{\mathsf{T}} \otimes\mathbf{K}_{t;i}. \tag*{(130)} K low − rank = i ∑ α λ ; i α λ ; i T ⊗ K t ; i . (130)
Re-writing the 2D kernel term using the eigendecomposition of K λ ; 2 D \mathbf{K}_{\lambda;\mathrm{2D}} K λ ; 2D reveals the connection between these:
K 2 D = ∑ i Rank ( K λ ) λ i v λ ; i v λ ; i T ⊗ K t ; 2 D . (131) \mathbf{K}_{\mathrm{2D}} = \sum_{i}^{\operatorname{Rank}(\mathbf{K}_{\lambda})} \lambda_i \mathbf{v}_{\lambda;i}\mathbf{v}_{\lambda;i}^{\mathsf{T}} \otimes\mathbf{K}_{t;\mathrm{2D}}. \tag*{(131)} K 2D = i ∑ Rank ( K λ ) λ i v λ ; i v λ ; i T ⊗ K t ; 2D . (131)
While the 2D kernel terms are flexible enough to handle full-rank covariance matrices, for smooth kernel functions like the squared exponential and rational quadratic kernels, the number of significant eigenvectors can often be very low. Figure 8 highlights this for the squared exponential kernel, measuring the number of eigenvalues > 10 − 12 > 10^{-12} > 1 0 − 12 as a function of length scale, and the number of input points. This means we can often closely approximate the squared exponential kernel using a small number of low-rank terms. For example, benchmarking shows that even N α = 16 N_{\alpha} = 16 N α = 16 can be highly scalable, even without orthogonal components (see Figure G20 and Figure G21 ). This trick could be particularly important to avoid a mixture of a 2D kernel term and low-rank kernel terms as in the LUASPLUSMULTILOWRANKKERNEL, and the EXTENDEDLUASKERNEL, as they scale less efficiently than the MULTILOWRANKKERNEL. There are also low-rank approximations to kernels which avoid eigendecomposition altogether (Solin and Särkkä 2014; Hensman et al. 2017), which may provide significant runtime improvements if the optimisation is bottlenecked by the cubic cost of eigendecomposition.
Figure 8. Left: Number of eigenvalues λ > 10 − 12 \lambda> 10^{-12} λ > 1 0 − 12 for a covariance matrix built using a squared exponential kernel on N N N regularly sampled input locations, with h = 1 h = 1 h = 1 and length scale given relative to the full input range. Longer length scales result in very few significant eigenvectors. Right: Nine most significant eigenvectors for the case of N = 256 N = 256 N = 256 and length scale equal to the input range. The square root of the eigenvalue λ \sqrt{\lambda} λ gives the standard deviation of the Gaussian prior placed on that eigenvector by the GP log-likelihood.
Other GP Optimisations
Given the generality of many of the optimisations in this work, particularly highlighted by Equation (67) for the LUASLASRACHKERNEL, we can consider combining it with a wide range of existing 1D GP optimisations, each of which might offer computational or flexibility benefits. Our optimisations have been implemented within LUAS in a modular fashion, wrapping different 1D GP optimisations in a common COVTYPE class which defines how to efficiently calculate matrix square roots, inverse square roots, inverses, and matrix multiplications. This modularity means that any existing JAX implementations of 1D GP optimisations can be imported and wrapped without re-writing each optimisation.
While TINYGP implements a flexible range of quasiseparable kernels, Rubenzahl et al. (2026) have introduced the JAX-based package SMOLGP to extend it to a more general class of state-space models using Kalman filtering. They used this to implement efficient integrals of GPs, useful for combining simultaneous observations with different exposure times. However, if this increased flexibility is not needed, benchmarking did not recover faster inference runtimes than quasiseparable kernels from TINYGP (Rubenzahl et al. 2026). While Kalman filtering does not explicitly factorise the covariance matrix as a matrix square root, if it can be formulated as one then it could be better combined with more of our optimisations, but it can already be combined with the LUASLASRACHKERNEL using Equation (67) .
Hierarchical Off-Diagonal Low-Rank (HODLR) is a black box GP approximation which can perform almost linear time inference O ( N log N ) \mathcal{O}(N \log N) O ( N log N ) with very limited restrictions on the choice of kernel functions (Ambikasaran and Darve 2013; Ambikasaran et al. 2015). HODLR works by factoring a covariance matrix into an exact block diagonal component, combined with a series of low-rank updates which approximate regions away from the diagonal. The approximation can often be tuned to reach close to machine precision (Ambikasaran et al. 2015), although higher accuracy naturally comes at increased computational cost. While HODLR can be applied to multi-dimensional inputs directly, its computational performance is fastest for 1D inputs, while optimisations like the LUASLASRACHKERNEL can not only reduce 2D GPs to separate 1D GP likelihoods, but can also reduce the problem size by exploiting sparsity. It could be used as an alternative to quasiseparable kernels, particularly when the quasiseparable structure is too restrictive.
There are plenty of other approximate methods, popular within machine learning, which may be complementary to our optimisations. Methods such as inducing point approximations estimate the GP log-likelihood by evaluating the GP on a reduced set of input locations, where these locations can be optimised to improve approximation accuracy (Titsias 2009). KISS-GP uses
inducing points on a grid to exploit Kronecker product optimisations for data which does not need to lie on a grid (Wilson and Nickisch [130 ] ). There are also approximations which are optimised for GPUs, which can exploit fast matrix-vector multiplications to iterate towards increasingly accurate approximations of the GP log-likelihood (e.g. [46 , 92 ] ). Combined with hardware acceleration, these methods present many options to be combined with our optimisations, in order to apply GP inference to very large datasets.
Applications
We list potential applications primarily focused on exoplanet observations. Just note, the main limitations that could affect these applications are the need for complete, rectangular datasets (i.e. no missing data), and separable noise (see § 2.5 ). These can be minor limitations for many spectroscopic time-series observations, as they possess a rectangular structure where occasional outliers can be clipped and replaced. The separable noise assumption could limit the use of rescaling noise estimates from data pipelines, but still allows us to fit white noise amplitudes in a data-driven way as separate functions of wavelength and time.
Transiting Exoplanets with JWST
The primary motivation for this work was to scale 2D GPs for the analysis of large spectroscopic time-series of transiting exoplanets. As data analyses typically assume the noise in each light curve is independent, this can result in unreliable results when wavelength-correlations are present. Unaccounted for wavelength-correlations can lead to inaccurate constraints from atmospheric retrievals, previously demonstrated both for transiting exoplanets ([44 ] (Ih and Kempton 2021)), and for directly-imaged planets ([55 , 90 ] ). Rather than inferring wavelength-correlations when performing atmospheric retrievals ([99 ] ), recovering wavelength-correlations at the level of fitting the spectroscopic time-series can exploit much more information, and avoids degeneracies with inaccurate atmospheric models. Recovering wavelength-correlations from the spectroscopic time-series using 2D GPs has already been demonstrated for ground-based low-resolution transmission spectroscopy, utilising the LUASKERNEL optimisation ([44 ] ), but the computational scaling of this method limits its applications to larger JWST datasets.
All four JWST instruments have shown at least some evidence of wavelength-correlated noise, due to effects such as 1 / f 1/f 1/ f noise and spectral trace changes ([1 , 64 , 12 , 54 , 36 ] ). There is also evidence that these wavelength-correlations can be reduction-dependent ([96 ] ), which implies that we cannot accurately compare the results of different reduction choices unless we can account for their differences in wavelength-correlations. JWST analyses have frequently found that different reduction and analysis choices can result in transit spectra that are offset from each other ([100 ] ), have different slopes ([73 , 22 ] ), or potentially even different features ([72 , 106 ] ). In future, we hope to constrain the role wavelength-covariance could play in such differences, as wavelength-covariance has already been shown to affect offsets and slopes of transmission spectra from VLT/FORS2 ([44 ] ).
One perpetual challenge within the field is the choice of wavelength-binning. This is the source of ongoing debate, since coarse wavelength bins should be more resilient to wavelength-correlations on short length scales ([12 ] ), yet reducing the wavelength resolution of transit spectra can weaken constraints from atmospheric retrievals ([28 ] ). This problem only arises because current analyses are treating light curves as independent, our optimisations offer a principled solution - we can fit at full wavelength (and time) resolution, while propagating the impact of wavelength-correlations to maintain robust results.
Even with perfectly stable instruments, at the photometric precision of JWST, astrophysical red noise from stellar granulation and asteroseismology can become a noise floor ([125 , 129 ] ). These signals are also chromatic ([83 , 89 ] ), which could introduce further degeneracy with atmospheric constraints, but accounting for this wavelength-dependence can also help to improve our constraints ([53 ] ). Even our own Sun, which is relatively quiet in terms of stellar activity, still exhibits a time-correlated signal of ∼ 100 \sim100 ∼ 100 ppm in optical light, largely due to solar granulation and oscillations ([89 ] ). This signal is large enough to pose a significant barrier to many of the subtle variations in transit light curve geometry we may hope to measure, due to effects such as oblateness, limb-asymmetries, eclipse mapping, exo-moons, etc. GP models have long been one of the most frequently used tools to model stellar granulation, asteroseismology and stellar activity, in part due to their connection to physically-motivated stochastic processes (e.g. [42 , 57 ] (Brewer and Stello 2009)). Viewed from a stellar astrophysics perspective, these optimisations could help us learn more about stellar variability, in combination with studying exoplanets.
Ground-based High-Resolution Spectroscopic Time-Series
Time-series observations of transiting exoplanets at high spectroscopic resolution can be used to distinguish the Doppler-shifted atmospheric signal of an orbiting exoplanet, relative to the approximately stationary spectra from the host star’s emission, and the Earth’s own atmospheric absorption (known as tellurics). Typically these observations require a significant amount of data cleaning prior to analysis, which often includes subtracting a number of PCA or SysRem ([113 ] ) components from the reduced data (e.g. [29 , 14 ] ). An active area of research is whether this data cleaning can be improved so as to reduce the amount of exoplanet signal lost ([87 , 84 ] ), particularly for slower moving targets when the Doppler shift is significantly reduced ([21 ] ).
While GPs have been explored already for data cleaning, this was performed after reducing each order to 1D spectra ([87 ] ), rather than fitting a 2D GP directly to each order. Combining 2D GPs with existing likelihood-based high-resolution retrieval approaches ([18 , 49 ] ) could be a promising method for reducing exoplanet signal loss. As highlighted in Figure 6 , PCA and SysRem are closely linked to 2D GP kernels such as the LUASLASRACKERNEL and MULTILOWRANKKERNEL, but the GP approaches offer a number of potential advantages, where we could modify these approaches with smoothing priors on the time and/or wavelength components, and marginalise over a space of systematics models rather than subtracting off a single best-fit model. Perhaps the most important change would be the ability to efficiently joint-fit the systematics model and an exoplanet signal, applying the cleaning to the data residuals, rather than to the data which still contains the exoplanet signal.
We leave the application of this to future work, but to give a runtime estimate based on our benchmarking, Figure G20 tests a GP kernel using the MULTILOWRANKKERNEL with N α = 16 N_{\alpha}=16 N α = 16 , which has flexibility equivalent to 16 components of PCA or SysRem, and with GP priors on components, e.g. if we want to weight some components to vary over particular timescales or wavelength scales. If we have a single high-resolution spectroscopic order with N λ = 1024 N_{\lambda}=1024 N λ = 1024 and N t = 128 N_{t}=128 N t = 128 , we achieve a 1.5 ms log-likelihood runtime on an M4 MacBook Air. This model also has linear runtime scaling with the number of datapoints, with a runtime scaling of ∼ O ( N α N t J ˉ 2 + N α N λ N t ) \sim\mathcal{O}(N_{\alpha}N_{t}\bar{J}^{2}+N_{\alpha}N_{\lambda}N_{t}) ∼ O ( N α N t J ˉ 2 + N α N λ N t ) , and joint-fits of multiple spectral orders could be parallelised across cores on a CPU or GPU.
Extreme Precision Radial Velocities (EPRV)
Extreme precision radial velocity (EPRV) analyses have been utilising 2D GPs - referred to as multidimensional GPs in the field - for many years (e.g. [97 , 51 , 9 ] ), where they have typically been used to joint-fit stellar radial velocity signals along with a number of activity indicators. Existing GP optimisations have already heavily optimised these analyses when joint-fitting a small number of activity indicators (e.g. [53 , 30 , 31 ] ). Since these existing analyses are normally in the regime where there are at least as many time-correlated low-rank terms as there are parallel time-series (i.e. N α ≥ N λ N_{\alpha}\geq N_{\lambda} N α ≥ N λ ), the MULTILOWRANKKERNEL cannot provide a runtime advantage over these existing optimisations through its sparsity exploiting transformations (see § 4 ).
However, our optimisations do offer some advantages if extending these models in certain ways. Accounting for measurement covariance between activity indicators has been discussed before ([51 ] ), and is a flexibility improvement our optimisations offer at virtually no extra cost (see Figure G9 ). Measurement covariance could be beneficial to include since the activity indicators are often derived from the same spectra as the radial velocities, and thus are exposed to the same noise draw. Joint-fitting increasing numbers of parallel time-series can scale more efficiently with the MULTILOWRANKKERNEL for the same number of kernel terms, e.g. compare the runtimes in Figure G14 for the existing Gordon et al. [53 ] method for just N α = 1 N_{\alpha}=1 N α = 1 low-rank kernel term, with Figure G18 , Figure G19 and Figure G20 with N α = 4 N_{\alpha}=4 N α = 4 , 8 and 16 low-rank kernel terms, respectively. This could be very useful as the identification of new activity indicators ([86 ] ; [24 ] ), or fitting RVs from multiple wavelength regions ([134 , 78 ] ), are active areas of research. Even fitting systematics directly to the high-resolution spectroscopic time-series with 2D GPs could be a possibility, as discussed in the previous section.
Fitting Imaging Time-series at the Pixel-Level
Another area of interest has been the push to make use of, or even fit, the individual pixel time-series, such as in JWST time-series ([45 , 23 , 54 ] ), although there is a long history of this for analyses of Spitzer ([110 , 32 , 88 ] ), Kepler/K2 ([127 , 82 ] ; [93 ] ), and TESS ([60 ] ). These methods vary in their methodology, (e.g. using GPs, PCA, ICA), but as demonstrated in § 6.1 , these methods have links to highly-scalable GP optimisations. Fortune et al. [45 ] introduced the pixel-fitting method, where pixel time-series from JWST/MIRI photometric data are joint-fit using the LUASKERNEL, but the exact kernel function used could now be implemented with LUASLASHRACKERNEL to provide linear scaling with the number of time points, rather than cubic scaling before. Comparing the runtimes for a ( 64 , 1024 ) (64, 1024) ( 64 , 1024 ) dataset, similar to the dataset in Fortune et al. [45 ] , using benchmarking in Figure G10 and Figure G12 , this switch could reduce log-likelihood runtimes from 74 ms to 0.9 ms on an M4 MacBook Air. Alternatively, more flexible kernels such as the EXTENDEDLUASKERNEL could be used to simultaneously model multiple systematics sources with different timescales, as previously all systematics sources were limited to sharing the same timescale.
Extending this approach to joint-fitting pixel light curves from spectroscopic time-series would be a novel approach, replacing spectral trace extraction with a Bayesian inference approach, where we joint-model the changes in shape of the spectral trace, detector readout noise, and possible background contaminants. This could be combined with simultaneously inferring our astrophysical model parameters directly from the pixel time-series (similar to Fortune et al. [45 ] ), or alternatively spectroscopic time-series could be inferred from the pixel time-series and fit separately. This approach could be very useful for NIRCam Grism time-series, where correlated readout noise is particularly challenging to separate from the spectral trace due to the fast read direction on the detector ([105 ] ). Recently, Qiu et al. [95 ] have suggested a related approach, using multidimensional GPs to account for readout noise from CCD detectors, using it to infer the covariance matrix of uncertainty in stellar spectra.
Other Applications in Astronomy
GPs are ubiquitous across astronomy, and even multidimensional GPs have already been used for a range of applications. Keith et al. [71 ] apply a 2D GP approximation based on CELERITE to fit pulsar profile timing variations, useful for probing the ISM. Optimisations such as the LUASLASHRACKERNEL could be used for the same kernel function and would be exact and significantly more scalable. Multidimensional AGN time variability has also been recently suggested to benefit from a 2D GP model which exactly corresponds to the MULTILOWRANKKERNEL ([27 ] ), provided the observations lie on a grid of datapoints. Spectroscopic time-series of brown dwarf variability present another potential application, analogous to the current use of PCA to identify spectroscopic components of variability, but with GP priors on the time components (see § 6.2 ).
Image reconstruction with interferometric data often uses some form of regularisation to place smoothing priors on the images being recovered, such as to calculate the first image of a black hole ([41 ] ). A 2D GP can be used as a regulariser within a principled statistical framework, weighting towards more physically plausible images with an explicit, interpretable prior. 2D GPs have already been optimised for image reconstruction by combining variational inference with Fourier-based GP optimisations ([75 ] ; [37 ] ), and are often applied within a Bayesian framework known as Information Field Theory (Enßlin [39 ] ). Our optimisations provide alternative approaches to scale GPs to large images, which are exact and flexible, and have connections with existing Fourier-based optimisations (see § 6.8.1 ).
Machine Learning Applications
Related 2D GP models have been in use across fields such as geostatistics, computational biology, signal-processing, and machine learning, a summary of which is provided in Bruinsma et al. [19 ] . Multi-task GPs, also known as multi-output GPs, are regression models which recover the covariance between different features/tasks over multiple samples, helping to understand features which are correlated, while also being useful for predicting unseen features (Bonilla et al. [15 ] ). The covariance matrices used in these models can be in the same form as the LUAS KERNEL:
K = Σ f e a t u r e ⊗ I N s a m p l e + K f e a t u r e ⊗ K s a m p l e (132) \mathbf{K} = \mathbf{\Sigma}_{\mathrm{feature}} \otimes\mathbf{I}_{N_{\mathrm{sample}}} + \mathbf{K}_{\mathrm{feature}} \otimes\mathbf{K}_{\mathrm{sample}} \tag*{(132)} K = Σ feature ⊗ I N sample + K feature ⊗ K sample (132)
A simple example of this in Bonilla et al. [15 ] is to predict student exam scores from features about the school and students, with observations made for multiple students each year. The correlation between each feature and the student’s exam score is modelled within the feature covariance matrix K f e a t u r e \mathbf{K}_{\mathrm{feature}} K feature , and the correlation between different samples/years is modelled within the sample covariance matrix K s a m p l e \mathbf{K}_{\mathrm{sample}} K sample . There can be additional measurement covariance between features in the same sample (e.g. correlations between features for a given student), modelled by the Σ f e a t u r e \mathbf{\Sigma}_{\mathrm{feature}} Σ feature matrix, which motivated the work in Rakitsch et al. [98 ] . Within the machine learning literature, our LUASLASRACHKERNEL could be viewed as combining the work of Rakitsch et al. [98 ] to handle measurement covariance between features, while still utilising fast 1D GP optimisations for correlations between samples, similar to Bruinsma et al. [19 ] .
Linear Mixing Models are another form of 2D GP popular within machine learning, which are essentially the GP factor analysis/MULTILOWRANKKERNEL model focused on in § 6.4 , which is particularly useful as a method of dimensionality reduction (e.g. Yu et al. [133 ] ; Wu et al. 2022). There are many papers which focus on optimising them, although these have so far required approximations (Teh et al. [114 ] ; Alie et al. [5 ] ), scale less efficiently (Gordon et al. [53 ] ; Sarkka and Hartikainen [103 ] ), or have more stringent restrictions on the choice of covariance matrices to remain exact (Bruinsma et al. [19 ] ; Truffinet et al. [118 ] ). Our MULTILOWRANKKERNEL is both exact and particularly fast at dimensionality reduction, as it scales very efficiently when the number of components to extract is much smaller than the dimensions of the dataset.
Geostatistics
Geostatistics often makes use of multidimensional GPs to model random fields over the Earth’s surface. This draws from the original application of a GP, to predict the concentrations of metal ores over a given region, using measurements taken over a limited set of locations (Krige [77 ] ). It has since been applied to many remote sensing problems, such as interpolating over gaps in satellite imagery (Zhang et al. [135 ] ), and modelling environmental variables such as groundwater salinity (Cui et al. [25 ] ). One popular GP model in use is called the Linear Model of Coregionalisation, which can be expressed as
K = ∑ i = 1 L B q i ; q ⊗ K l o c a t i o n ; i (133) \mathbf{K} = \sum_{i=1}^{L} \mathbf{B}_{q_i;q} \otimes\mathbf{K}_{\mathrm{location};i} \tag*{(133)} K = i = 1 ∑ L B q i ; q ⊗ K location ; i (133)
where B q i ; q \mathbf{B}_{q_i;q} B q i ; q are covariance matrices between N v N_{\mathrm{v}} N v variables measured at every location, and K l o c a t i o n ; i \mathbf{K}_{\mathrm{location};i} K location ; i model correlations between N l o c N_{\mathrm{loc}} N loc different locations; while the sum over L L L can handle different correlation structures over multiple distance scales. We note that this simplifies to the SEPARABLEKERNEL when L = 1 L=1 L = 1 (although existing optimisations in the field exist for this scenario), and the LUASKERNEL or LUASLASRACHKERNEL when L = 2 L=2 L = 2 . Due to the typically non-grid 2D locations of each measurement, CELERITE-based optimisations may not easily be applied since they require sortable inputs, but combining the LUASLASRACHKERNEL with approximate methods such as HODLR (Ambikasaran et al. [7 ] ) or inducing points could efficiently handle popular kernel choices such as the exponential and squared exponential kernel on 2D non-grid inputs.
Spatiotemporal GPs, as the name suggests, can be used to model data with dimensions of space and time, and are closely linked to methods in geostatistics. Their applications include epidemiology (Diggle et al. [33 ] ), geophysical modelling (Svendsen et al. [112 ] ), and even weather forecasting (e.g. Donnelly et al. [34 ] ; Kisi et al. [74 ] ). Optimisations such as the LUASLASRACHKERNEL and MULTILOWRANKKERNEL could be useful for optimising these approaches and scaling to large datasets, while maintaining exact inference with highly flexible kernel functions. Compared with popular machine learning approaches like deep neural networks, GPs can be more interpretable, and produce predictions with more principled uncertainty quantification, crucial for many of these domains where risk mitigation is important.
Summary and Conclusions
This work has introduced a selection of new, exact, highly scalable GP optimisations focused on two-dimensional rectangular datasets. These optimisations combine Kronecker product algebra with fast GP optimisations, including CELERITE, to permit inference on millions of datapoints across a flexible range of kernel choices and models. We’ve demonstrated links between these methods and popular algorithms like PCA, SysRem, and 2D Fast Fourier Transforms. These optimisations have an exciting range of applications which we will explore in future work, some of which include:
Efficiently accounting for wavelength- and time-correlated systematics in exoplanet transit spectroscopy
Data-cleaning of ground-based high-resolution observations within a Bayesian framework
Scaling Extreme Precision Radial Velocity techniques to larger datasets
Inference applied directly to pixel time-series to accurately model detector systematics
Gaussian process PCA to identify and distinguish smooth time components in datasets
Fast analytical marginalisation over linear parameters with Gaussian priors which are separable in each dimension
Data Availability
Our algorithms are implemented within the open-source Python package LUAS, which is available here . Documentation and tutorials are located on readthedocs .
Acknowledgements
The authors wish to acknowledge the Irish Centre for High-End Computing (ICHEC) for the provision of computational facilities and support. We acknowledge and pay respect to the traditional owners of the land on which Macquarie University is situated, upon whose unceded, sovereign, ancestral lands we work. We pay respects to their Ancestors and descendants, who continue cultural and spiritual connections to Country.
Large Language Models (LLMs) were used to search for relevant literature, particularly to study related work outside the authors’ domains of expertise, such as within machine learning, geostatistics, and signal processing. LLMs also identified useful literature in order to sample the GP predictive covariance, and to improve numerical stability. They aided in formatting some figures and LaTeX equations. While coding assistants were helpful for some testing and documentation of code, their use beyond this was limited to experimenting with algorithms and answering questions about implementations. LLMs did not develop the GP optimisations introduced in this work, nor write the text in this work, and the implementations of these algorithms were written from scratch with minimal LLM input beyond basic autocomplete coding assistance.
We are grateful to the developers of JAX, NumPy, SciPy, Matplotlib, TinyGP, Pandas, IPython, and JAXoplanet ([16 , 58 , 123 , 66 , 43 , 91 , 61 ] (The pandas development team 2020)), as these packages were used extensively in this work.
The DoubleMultiLowRankKernel Decomposition
If a matrix such as K λ ; i \mathbf{K}_{\lambda;i} K λ ; i is semiseparable or quasiseparable, we would like to maintain this property for K λ ; i ′ \mathbf{K}'_{\lambda;i} K λ ; i ′ , defined as
K λ ; i ′ = Q A T D λ − 1 2 K λ ; i D λ − 1 2 Q A , (134) \mathbf{K}'_{\lambda;i}=\mathbf{Q}_{A}^{\mathsf{T}}\mathbf{D}_{\lambda}^{-\frac{1}{2}}\mathbf{K}_{\lambda;i}\mathbf{D}_{\lambda}^{-\frac{1}{2}}\mathbf{Q}_{A}, \tag*{(134)} K λ ; i ′ = Q A T D λ − 2 1 K λ ; i D λ − 2 1 Q A , (134)
where D λ \mathbf{D}_{\lambda} D λ is diagonal and Q A \mathbf{Q}_{A} Q A consists of a product of Householder matrices. We show this for semiseparable matrices, but this naturally extends to the quasiseparable matrices which TINYGP implements without modification, since they are a more flexible class of matrices which include semiseparable matrices as a special case.
Semiseparable matrices can be represented as:
K = D + tril ( U V T ) + triu ( V U T ) (135) \mathbf{K}=\mathbf{D}+\operatorname{tril}(\mathbf{U}\mathbf{V}^{\mathsf{T}})+\operatorname{triu}(\mathbf{V}\mathbf{U}^{\mathsf{T}}) \tag*{(135)} K = D + tril ( U V T ) + triu ( V U T ) (135)
where D \mathbf{D} D is diagonal and U , V \mathbf{U},\mathbf{V} U , V are both of shape ( N , J ) (N,J) ( N , J ) , where J J J is the semiseparable rank. tril ( ) \operatorname{tril}() tril ( ) and triu ( ) \operatorname{triu}() triu ( ) are operators which zero out any elements which do not lie in the strictly lower/upper triangular part of the matrix.
Transformations with diagonal matrices, such as D λ − 1 2 K λ ; i D λ − 1 2 \mathbf{D}_{\lambda}^{-\frac{1}{2}}\mathbf{K}_{\lambda;i}\mathbf{D}_{\lambda}^{-\frac{1}{2}} D λ − 2 1 K λ ; i D λ − 2 1 , can be easily implemented while maintaining semiseparability, by transforming as:
D ′ = D λ − 1 2 D D λ − 1 2 (136) \mathbf{D}'=\mathbf{D}_{\lambda}^{-\frac{1}{2}}\mathbf{D}\mathbf{D}_{\lambda}^{-\frac{1}{2}} \tag*{(136)} D ′ = D λ − 2 1 D D λ − 2 1 (136)
U ′ = D λ − 1 2 U (137) \mathbf{U}'=\mathbf{D}_{\lambda}^{-\frac{1}{2}}\mathbf{U} \tag*{(137)} U ′ = D λ − 2 1 U (137)
V ′ = D λ − 1 2 V , (138) \mathbf{V}'=\mathbf{D}_{\lambda}^{-\frac{1}{2}}\mathbf{V}, \tag*{(138)} V ′ = D λ − 2 1 V , (138)
which has no effect on the semiseparable rank J J J , i.e. we don’t need to add any columns to U \mathbf{U} U or V \mathbf{V} V . We will show that the transformations using Q A ′ \mathbf{Q}_{A}' Q A ′ and Q B ′ \mathbf{Q}_{B}' Q B ′ , which equal a product of N α N_{\alpha} N α and N β N_{\beta} N β Householder matrices respectively, will increase the semiseparable ranks of each matrix by 2 N α 2N_{\alpha} 2 N α and 2 N β 2N_{\beta} 2 N β respectively.
Multiplication of a matrix on the left and right by a Householder matrix is equivalent to adding/subtracting three vector outer products
K H = ( I − 2 u u T ) K ( I − 2 u u T ) (139) \mathbf{K}\mathbf{H}=(\mathbf{I}-2\mathbf{u}\mathbf{u}^{\mathsf{T}})\mathbf{K}(\mathbf{I}-2\mathbf{u}\mathbf{u}^{\mathsf{T}}) \tag*{(139)} KH = ( I − 2 u u T ) K ( I − 2 u u T ) (139)
However, we can express our three outer products as just two symmetric outer products by defining:
p 1 = − K u + 2 ( u ⋅ K u ) u , (140) \mathbf{p}_{1}=-\mathbf{K}\mathbf{u}+2(\mathbf{u}\cdot\mathbf{K}\mathbf{u})\mathbf{u}, \tag*{(140)} p 1 = − Ku + 2 ( u ⋅ Ku ) u , (140)
p 2 = K u , (141) \mathbf{p}_{2}=\mathbf{K}\mathbf{u}, \tag*{(141)} p 2 = Ku , (141)
where we see that
p 1 p 1 T u ⋅ K u = K u ( K u ) T u ⋅ K u − 2 u ( K u ) T − 2 ( K u ) u T + 4 ( u ⋅ K u ) u u T , (142) \frac{\mathbf{p}_{1}\mathbf{p}_{1}^{\mathsf{T}}}{\mathbf{u}\cdot\mathbf{K}\mathbf{u}}=
\frac{\mathbf{K}\mathbf{u}(\mathbf{K}\mathbf{u})^{\mathsf{T}}}{\mathbf{u}\cdot\mathbf{K}\mathbf{u}}-2\mathbf{u}(\mathbf{K}\mathbf{u})^{\mathsf{T}}-2(\mathbf{K}\mathbf{u})\mathbf{u}^{\mathsf{T}}+4(\mathbf{u}\cdot\mathbf{K}\mathbf{u})\mathbf{u}\mathbf{u}^{\mathsf{T}}, \tag*{(142)} u ⋅ Ku p 1 p 1 T = u ⋅ Ku Ku ( Ku ) T − 2 u ( Ku ) T − 2 ( Ku ) u T + 4 ( u ⋅ Ku ) u u T , (142)
− p 2 p 2 T u ⋅ K u = − K u ( K u ) T u ⋅ K u , (143) -\frac{\mathbf{p}_{2}\mathbf{p}_{2}^{\mathsf{T}}}{\mathbf{u}\cdot\mathbf{K}\mathbf{u}}=-\frac{\mathbf{K}\mathbf{u}(\mathbf{K}\mathbf{u})^{\mathsf{T}}}{\mathbf{u}\cdot\mathbf{K}\mathbf{u}}, \tag*{(143)} − u ⋅ Ku p 2 p 2 T = − u ⋅ Ku Ku ( Ku ) T , (143)
⟹ K H = K + p 1 p 1 T u ⋅ K u − p 2 p 2 T u ⋅ K u . (144) \Longrightarrow\mathbf{K}\mathbf{H}=\mathbf{K}+\frac{\mathbf{p}_{1}\mathbf{p}_{1}^{\mathsf{T}}}{\mathbf{u}\cdot\mathbf{K}\mathbf{u}}-\frac{\mathbf{p}_{2}\mathbf{p}_{2}^{\mathsf{T}}}{\mathbf{u}\cdot\mathbf{K}\mathbf{u}}. \tag*{(144)} ⟹ KH = K + u ⋅ Ku p 1 p 1 T − u ⋅ Ku p 2 p 2 T . (144)
To maintain semiseparability, we now must show that these low-rank updates can be written in semiseparable form{sup}e.
We could alternatively handle low-rank updates using the Sherman-Morrison-Woodbury Formula (e.g. Golub and Van Loan (2013)), but expressing these transformed
Low-Rank Updates to Semiseparable Matrices
If we have a symmetric, low-rank update K → K + c p p T \mathbf{K} \rightarrow\mathbf{K} + c\mathbf{p}\mathbf{p}^{\mathsf{T}} K → K + c p p T , this can be added to a semiseparable matrix in the form of Equation (135) , e.g. by appending p \mathbf{p} p as an extra column of U \mathbf{U} U , and c p c\mathbf{p} c p as an extra column of V \mathbf{V} V . We must also add the diagonal of c p p T c\mathbf{p}\mathbf{p}^{\mathsf{T}} c p p T to the diagonal matrix D \mathbf{D} D . For both semiseparable and quasiseparable cases, each rank-1 update increases the rank of the matrices from J → J + 1 J \rightarrow J + 1 J → J + 1 . Since each Householder transformation H K H \mathbf{H}\mathbf{K}\mathbf{H} HKH requires a rank-2 update, if we have to apply this transformation successively for a stack of N α N_{\alpha} N α Householder matrices, this will increase the rank of any quasiseparable matrix K \mathbf{K} K from J → J + 2 N α J \rightarrow J + 2N_{\alpha} J → J + 2 N α .
Decomposing K ~ \widetilde{\mathbf{K}} K for the DOUBLEMULTILOWRANKKERNEL
Our transformed covariance matrix is in the form of a 2x2 block matrix:
K ~ = [ ( I N α ⊗ I N t + ∑ i = 1 N β a i a i T ⊗ K τ ; i + ′ ) ∑ j = 1 N β [ K λ ; j ′ ] B ⊗ [ b j 0 ] [ b j 0 ] T ∑ j = 1 N β [ K λ ; j ′ ] C ⊗ [ b j 0 ] [ b j 0 ] T ( I N λ − N α ⊗ I N t + ∑ j = 1 N β [ K λ ; j ′ ] D ⊗ [ b j 0 ] [ b j 0 ] T ) ] (145) \widetilde{\mathbf{K}} =
\begin{bmatrix}
\left(
\mathbf{I}_{N_{\alpha}} \otimes\mathbf{I}_{N_t}
+ \sum_{i=1}^{N_{\beta}} \mathbf{a}_i\mathbf{a}_i^{\mathsf{T}} \otimes\mathbf{K}'_{\tau;i+}
\right) &
\sum_{j=1}^{N_{\beta}} [\mathbf{K}'_{\lambda;j}]_B \otimes
\begin{bmatrix}\mathbf{b}_j\\0\end{bmatrix}
\begin{bmatrix}\mathbf{b}_j\\0\end{bmatrix}^{\mathsf{T}}
\\
\sum_{j=1}^{N_{\beta}} [\mathbf{K}'_{\lambda;j}]_C \otimes
\begin{bmatrix}\mathbf{b}_j\\0\end{bmatrix}
\begin{bmatrix}\mathbf{b}_j\\0\end{bmatrix}^{\mathsf{T}}
&
\left(
\mathbf{I}_{N_{\lambda}-N_{\alpha}} \otimes\mathbf{I}_{N_t}
+ \sum_{j=1}^{N_{\beta}} [\mathbf{K}'_{\lambda;j}]_D \otimes
\begin{bmatrix}\mathbf{b}_j\\0\end{bmatrix}
\begin{bmatrix}\mathbf{b}_j\\0\end{bmatrix}^{\mathsf{T}}
\right)
\end{bmatrix} \tag*{(145)} K = ( I N α ⊗ I N t + ∑ i = 1 N β a i a i T ⊗ K τ ; i + ′ ) ∑ j = 1 N β [ K λ ; j ′ ] C ⊗ [ b j 0 ] [ b j 0 ] T ∑ j = 1 N β [ K λ ; j ′ ] B ⊗ [ b j 0 ] [ b j 0 ] T ( I N λ − N α ⊗ I N t + ∑ j = 1 N β [ K λ ; j ′ ] D ⊗ [ b j 0 ] [ b j 0 ] T ) (145)
For the MULTILOWRANKKERNEL, K ~ \widetilde{\mathbf{K}} K could be expressed as a simpler block diagonal matrix (Equation (86) ), while K ~ \widetilde{\mathbf{K}} K in Equation (145) has non-zero B and C blocks. We can use the following formula for a general symmetric 2x2 block matrix
K ~ − 1 2 = [ A − 1 2 0 − ( D − C A − 1 B ) − 1 2 C A − 1 ( D − C A − 1 B ) − 1 2 ] (146) \widetilde{\mathbf{K}}^{-\frac{1}{2}} =
\begin{bmatrix}
\mathbf{A}^{-\frac{1}{2}} & \mathbf{0}\\
-(\mathbf{D}-\mathbf{C}\mathbf{A}^{-1}\mathbf{B})^{-\frac{1}{2}}\mathbf{C}\mathbf{A}^{-1} & (\mathbf{D}-\mathbf{C}\mathbf{A}^{-1}\mathbf{B})^{-\frac{1}{2}}
\end{bmatrix} \tag*{(146)} K − 2 1 = [ A − 2 1 − ( D − C A − 1 B ) − 2 1 C A − 1 0 ( D − C A − 1 B ) − 2 1 ] (146)
known as the block Cholesky factorisation (Golub and Van Loan [52 ] ). Splitting our transformed residuals between the first N α N_{\alpha} N α light curves r ~ A \widetilde{\mathbf{r}}_A r A and the remaining light curves r ~ D \widetilde{\mathbf{r}}_D r D , we can calculate:
K ~ − 1 2 r ~ = [ A − 1 2 0 − ( D − C A − 1 B ) − 1 2 C A − 1 ( D − C A − 1 B ) − 1 2 ] [ r ~ A r ~ D ] = [ A − 1 2 r ~ A ( D − C A − 1 B ) − 1 2 ( − C A − 1 r ~ A + r ~ D ) ] . (147) \widetilde{\mathbf{K}}^{-\frac{1}{2}}\widetilde{\mathbf{r}} =
\begin{bmatrix}
\mathbf{A}^{-\frac{1}{2}} & \mathbf{0}\\
-(\mathbf{D}-\mathbf{C}\mathbf{A}^{-1}\mathbf{B})^{-\frac{1}{2}}\mathbf{C}\mathbf{A}^{-1} & (\mathbf{D}-\mathbf{C}\mathbf{A}^{-1}\mathbf{B})^{-\frac{1}{2}}
\end{bmatrix}
\begin{bmatrix}
\widetilde{\mathbf{r}}_A\\
\widetilde{\mathbf{r}}_D
\end{bmatrix}
=
\begin{bmatrix}
\mathbf{A}^{-\frac{1}{2}}\widetilde{\mathbf{r}}_A\\
(\mathbf{D}-\mathbf{C}\mathbf{A}^{-1}\mathbf{B})^{-\frac{1}{2}}(-\mathbf{C}\mathbf{A}^{-1}\widetilde{\mathbf{r}}_A+\widetilde{\mathbf{r}}_D)
\end{bmatrix}. \tag*{(147)} K − 2 1 r = [ A − 2 1 − ( D − C A − 1 B ) − 2 1 C A − 1 0 ( D − C A − 1 B ) − 2 1 ] [ r A r D ] = [ A − 2 1 r A ( D − C A − 1 B ) − 2 1 ( − C A − 1 r A + r D ) ] . (147)
Once we have K ~ − 1 2 r ~ \widetilde{\mathbf{K}}^{-\frac{1}{2}}\widetilde{\mathbf{r}} K − 2 1 r , we simply square and sum it to get r ~ T K ~ − 1 r ~ \widetilde{\mathbf{r}}^{\mathsf{T}}\widetilde{\mathbf{K}}^{-1}\widetilde{\mathbf{r}} r T K − 1 r for our transformed log-likelihood. The multiplication by C is cheap as it is very sparse, we can simply multiply by all N α N β 2 ( N λ − N α ) N_{\alpha}N_{\beta}^{2}(N_{\lambda}-N_{\alpha}) N α N β 2 ( N λ − N α ) non-zero values in O ( N α N β 2 ( N λ − N α ) ) \mathcal{O}(N_{\alpha}N_{\beta}^{2}(N_{\lambda}-N_{\alpha})) O ( N α N β 2 ( N λ − N α )) time. The only matrices we need to decompose are A and D − C A − 1 B \mathbf{D}-\mathbf{C}\mathbf{A}^{-1}\mathbf{B} D − C A − 1 B . The log-determinant of a 2x2 block matrix is given by:
log ∣ K ~ ∣ = log ∣ A ∣ + log ∣ D − C A − 1 B ∣ , (148) \log\lvert\widetilde{\mathbf{K}}\rvert= \log\lvert\mathbf{A}\rvert+ \log\lvert\mathbf{D}-\mathbf{C}\mathbf{A}^{-1}\mathbf{B}\rvert, \tag*{(148)} log ∣ K ∣ = log ∣ A ∣ + log ∣ D − C A − 1 B ∣ , (148)
so we can also use our decompositions of A and D − C A − 1 B \mathbf{D}-\mathbf{C}\mathbf{A}^{-1}\mathbf{B} D − C A − 1 B to calculate the log-determinant of K ~ \widetilde{\mathbf{K}} K in our transformed log-likelihood.
Decomposing the A block
Starting with the ( N α N t , N α N t ) (N_{\alpha}N_t,N_{\alpha}N_t) ( N α N t , N α N t ) A block
[ K ~ ] A = I N α ⊗ I N t + ∑ i = 1 N β a i a i T ⊗ K τ ; i ′ + ∑ j = 1 N β [ K λ ; j ′ ] A ⊗ [ b j 0 ] [ b j 0 ] T , (149) [\widetilde{\mathbf{K}}]_A =
\mathbf{I}_{N_{\alpha}}\otimes\mathbf{I}_{N_t}
+\sum_{i=1}^{N_{\beta}}\mathbf{a}_i\mathbf{a}_i^{\mathsf{T}}\otimes\mathbf{K}'_{\tau;i}
+\sum_{j=1}^{N_{\beta}}[\mathbf{K}'_{\lambda;j}]_A\otimes
\begin{bmatrix}\mathbf{b}_j\\0\end{bmatrix}
\begin{bmatrix}\mathbf{b}_j\\0\end{bmatrix}^{\mathsf{T}}, \tag*{(149)} [ K ] A = I N α ⊗ I N t + i = 1 ∑ N β a i a i T ⊗ K τ ; i ′ + j = 1 ∑ N β [ K λ ; j ′ ] A ⊗ [ b j 0 ] [ b j 0 ] T , (149)
note that the first two terms are in the form of the MULTILOWRANKKERNEL, where K τ ; i ′ \mathbf{K}'_{\tau;i} K τ ; i ′ can be transformed following Appendix § 9 .§ 1 to remain quasiseparable. However, there is the addition of the ( N α , N α ) (N_{\alpha},N_{\alpha}) ( N α , N α ) A block of the transformed wavelength covariance matrices [ K λ ; j ′ ] A [\mathbf{K}'_{\lambda;j}]_A [ K λ ; j ′ ] A in a Kronecker product with sparse matrices within the common framework of semiseparable and quasiseparable matrices.
vectors which have at most N β N_{\beta} N β non-zero values within b j \mathbf{b}_j b j . If we flip the order of the Kronecker product (which is actually the form that the Gordon et al. [53 ] method will decompose it in anyway), then these terms are entirely concentrated in the top left ( N α N β , N α N β ) (N_{\alpha}N_{\beta},N_{\alpha}N_{\beta}) ( N α N β , N α N β ) of the A block of K ~ \widetilde{\mathbf{K}} K . This flip in Kronecker product is not an issue for the calculations, provided we are careful to unravel our residuals array R Fortran-style instead of C-style for matrix-vector products.
This small update to the top left corner of the A block can be performed using a ( N α N β ) (N_{\alpha}N_{\beta}) ( N α N β ) -wide banded matrix, which is useful since banded matrices are valid quasiseparable matrices already implemented in TINYGP, and all quasiseparable matrices are closed under addition (Eidelman and Gohberg [38 ] ). This update simply has the cost of increasing the quasiseparable rank of A by another N α N β N_{\alpha}N_{\beta} N α N β , for a total rank of J = N α ( J ~ K + 3 N β ) J=N_{\alpha}(\widetilde{J}_{K}+3N_{\beta}) J = N α ( J K + 3 N β ) . This allows us to decompose A as a fully quasiseparable matrix, scaling as O ( N α 2 N t ( J ~ K + 3 N β ) 2 ) \mathcal{O}(N_{\alpha}^{2}N_t(\widetilde{J}_{K}+3N_{\beta})^{2}) O ( N α 2 N t ( J K + 3 N β ) 2 ) in runtime.
Calculating C A − 1 B \mathbf{C}\mathbf{A}^{-1}\mathbf{B} C A − 1 B
With A already decomposed, we must calculate C A − 1 B \mathbf{C}\mathbf{A}^{-1}\mathbf{B} C A − 1 B , where each matrix-vector product A − 1 2 \mathbf{A}^{-\frac{1}{2}} A − 2 1 scales as O ( N α 2 N t ( J ~ K + 3 N β ) ) \mathcal{O}(N_{\alpha}^{2}N_t(\widetilde{J}_{K}+3N_{\beta})) O ( N α 2 N t ( J K + 3 N β )) . Naturally, we can ignore columns in B which consist entirely of zeros, which already limits us to just ( N λ − N α ) N β (N_{\lambda}-N_{\alpha})N_{\beta} ( N λ − N α ) N β non-zero column vectors. However, we can reduce the number of matrix-vector products even further by noticing that B consists of only N α N β N_{\alpha}N_{\beta} N α N β non-zero rows. If we swap the order of the Kronecker products for B, labelling it as B ~ \widetilde{\mathbf{B}} B , then these non-zero rows are restricted to the first N α N β N_{\alpha}N_{\beta} N α N β rows of B ~ \widetilde{\mathbf{B}} B . We see that rather than needing each of A − 1 \mathbf{A}^{-1} A − 1 , we only need to know with its own ( N α N β , N α N β ) (N_{\alpha}N_{\beta},N_{\alpha}N_{\beta}) ( N α N β , N α N β ) A block
\begin{bmatrix}
\widetilde{\mathbf{B}}^{\mathsf{T}} & \mathbf{0}
\end{bmatrix}
\begin{bmatrix}
[\mathbf{A}^{-1}]_A & [\mathbf{A}^{-1}]_B\\
[\mathbf{A}^{-1}]_C & [\mathbf{A}^{-1}]_D
\end{bmatrix}
\begin{bmatrix}
\widetilde{\mathbf{B}}\\
\mathbf{0}
\end{bmatrix}
=
\begin{bmatrix}
\widetilde{\mathbf{B}}^{\mathsf{T}}[\mathbf{A}^{-1}]_A\widetilde{\mathbf{B}} & \mathbf{0}\\
\mathbf{0} & \mathbf{0}
\end{bmatrix}. \tag*{(150)}\end{bmatrix}
We want to avoid explicitly computing all of A − 1 \mathbf{A}^{-1} A − 1 , and instead just calculate [ A − 1 ] A [\mathbf{A}^{-1}]_A [ A − 1 ] A . To do this, we can multiply our representation of A − 1 2 \mathbf{A}^{-\frac{1}{2}} A − 2 1 by the first N α N β N_{\alpha}N_{\beta} N α N β basis vectors, and evaluate each dot product
( A − 1 ) ( i , j ) = ( A − 1 2 e ^ i ) T ( A − 1 2 e ^ j ) . (151) (\mathbf{A}^{-1})_{(i,j)}=(\mathbf{A}^{-\frac{1}{2}}\hat{\mathbf{e}}_i)^{\mathsf{T}}(\mathbf{A}^{-\frac{1}{2}}\hat{\mathbf{e}}_j). \tag*{(151)} ( A − 1 ) ( i , j ) = ( A − 2 1 e ^ i ) T ( A − 2 1 e ^ j ) . (151)
We can then calculate a matrix square root of [ A − 1 ] A [\mathbf{A}^{-1}]_A [ A − 1 ] A , so that we can calculate B ~ ′ = ( [ A − 1 ] A 1 2 ) T B ~ \widetilde{\mathbf{B}}' = ([\mathbf{A}^{-1}]_A^{\frac{1}{2}})^{\mathsf{T}}\widetilde{\mathbf{B}} B ′ = ([ A − 1 ] A 2 1 ) T B , where B ~ ′ T B ~ ′ \widetilde{\mathbf{B}}'^{\mathsf{T}}\widetilde{\mathbf{B}}' B ′ T B ′ includes all the non-zero values of C A − 1 B \mathbf{C}\mathbf{A}^{-1}\mathbf{B} C A − 1 B . Since the non-zero values of B ~ ′ \widetilde{\mathbf{B}}' B ′ lie in N α N β N_{\alpha}N_{\beta} N α N β rows, we only need to make a rank-( N α N β ) (N_{\alpha}N_{\beta}) ( N α N β ) update to the matrix D, which can maintain its quasiseparability. The most significant runtime cost from this is the N α N β N_{\alpha}N_{\beta} N α N β matrix-vector products of each basis vector, scaling as O ( N α 3 N β N t ( J ~ K + 3 N β ) ) \mathcal{O}(N_{\alpha}^{3}N_{\beta}N_t(\widetilde{J}_{K}+3N_{\beta})) O ( N α 3 N β N t ( J K + 3 N β )) in runtime, which is completely dominated by the previous scaling for decomposing the A block.
Decomposing D − C A − 1 B \mathbf{D}-\mathbf{C}\mathbf{A}^{-1}\mathbf{B} D − C A − 1 B
Since C A − 1 B \mathbf{C}\mathbf{A}^{-1}\mathbf{B} C A − 1 B is a symmetric, rank-N α N β N_{\alpha}N_{\beta} N α N β matrix, we can maintain D − C A − 1 B \mathbf{D}-\mathbf{C}\mathbf{A}^{-1}\mathbf{B} D − C A − 1 B as a semiseparable matrix provided that D is semiseparable (as shown in Section A.1), and similarly for quasiseparable matrices. By doing so, we increase its rank by N α N β N_{\alpha}N_{\beta} N α N β . The D block already has a quasiseparable rank of N β ( J ~ K + 2 N α ) N_{\beta}(\widetilde{J}_{K}+2N_{\alpha}) N β ( J K + 2 N α ) due to the sum of N β N_{\beta} N β terms which have each been transformed by N α N_{\alpha} N α Householder transformations, so we get a total rank of N β ( J ~ K + 3 N α ) N_{\beta}(\widetilde{J}_{K}+3N_{\alpha}) N β ( J K + 3 N α ) . Note that unlike the A block, TINYGP will actually decompose the D block in the Kronecker product order it is written in from Equation (145) , as it must be sorted into the order of its wavelength kernel inputs, so we must be careful to re-order our B ~ ′ T B ~ ′ \widetilde{\mathbf{B}}'^{\mathsf{T}}\widetilde{\mathbf{B}}' B ′ T B ′ → B ~ ′ B ~ ′ T \rightarrow\widetilde{\mathbf{B}}'\widetilde{\mathbf{B}}'^{\mathsf{T}} → B ′ B ′ T , so that the non-zero elements of C A − 1 B \mathbf{C}\mathbf{A}^{-1}\mathbf{B} C A − 1 B match this sort order.
Note that due to the sparsity of both the D block and B ~ \widetilde{\mathbf{B}} B , we can split D − C A − 1 B \mathbf{D}-\mathbf{C}\mathbf{A}^{-1}\mathbf{B} D − C A − 1 B into its own 2x2 block matrix, as it can be expressed as:
D − C A − 1 B = [ ( I N λ − N α ⊗ I N β + ∑ j = 1 N β [ K λ ; j ′ ] D ⊗ b j b j T − B ~ ′ B ~ ′ T ) 0 0 I N λ − N α ⊗ I N t − N β ] . (152) \mathbf{D}-\mathbf{C}\mathbf{A}^{-1}\mathbf{B} =
\begin{bmatrix}
\left(
\mathbf{I}_{N_{\lambda}-N_{\alpha}}\otimes\mathbf{I}_{N_{\beta}}
+\sum_{j=1}^{N_{\beta}}[\mathbf{K}'_{\lambda;j}]_D\otimes\mathbf{b}_j\mathbf{b}_j^{\mathsf{T}}
-\widetilde{\mathbf{B}}'\widetilde{\mathbf{B}}'^{\mathsf{T}}
\right) & \mathbf{0}\\
\mathbf{0} & \mathbf{I}_{N_{\lambda}-N_{\alpha}}\otimes\mathbf{I}_{N_t-N_{\beta}}
\end{bmatrix}. \tag*{(152)} D − C A − 1 B = [ ( I N λ − N α ⊗ I N β + ∑ j = 1 N β [ K λ ; j ′ ] D ⊗ b j b j T − B ′ B ′ T ) 0 0 I N λ − N α ⊗ I N t − N β ] . (152)
We only need to decompose the quasiseparable ( N λ − N α ) N β , ( N λ − N α ) N β (N_{\lambda}-N_{\alpha})N_{\beta},(N_{\lambda}-N_{\alpha})N_{\beta} ( N λ − N α ) N β , ( N λ − N α ) N β A block of this, which scales as O ( N β 3 N λ ( J ~ K + 3 N α ) 2 ) \mathcal{O}(N_{\beta}^{3}N_{\lambda}(\widetilde{J}_{K}+3N_{\alpha})^{2}) O ( N β 3 N λ ( J K + 3 N α ) 2 ) .
матриces within the common framework of semiseparable and quasiseparable matrices simple derivations and implementation
Total Computational Cost
We now have everything we need to multiply our transformed residuals by the matrix inverse square root of K ~ \widetilde{K} K using Equation (147) , and our decompositions of A A A and D − C A − 1 B D-CA^{-1}B D − C A − 1 B allow us to cheaply calculate the block matrix determinant from Equation (148) . Our total runtime scaling to decompose K ~ \widetilde{K} K works out to O ( N λ 3 N t ( J ‾ K t + 3 N β 2 ) + N β 3 N λ ( J ‾ K λ + 3 N α 2 ) ) \mathcal{O}(N_{\lambda}^{3}N_{t}(\overline{J}_{K_{t}}+3N_{\beta}^{2})+N_{\beta}^{3}N_{\lambda}(\overline{J}_{K_{\lambda}}+3N_{\alpha}^{2})) O ( N λ 3 N t ( J K t + 3 N β 2 ) + N β 3 N λ ( J K λ + 3 N α 2 )) , as the decomposition of the A A A and D − C A − 1 B D-CA^{-1}B D − C A − 1 B blocks dominate. There is an additional cost of O ( N λ N t ( N α + N β ) ) \mathcal{O}(N_{\lambda}N_{t}(N_{\alpha}+N_{\beta})) O ( N λ N t ( N α + N β )) to transform the residuals with N α + N β N_{\alpha}+N_{\beta} N α + N β Householder transformations. Note that the scaling is completely linear in N λ N_{\lambda} N λ and N t N_{t} N t , although it can scale quickly if the product N α N β N_{\alpha}N_{\beta} N α N β is large.
Decomposing for the LuasPlusMultiLowRankKernel
We have reduced K ~ \widetilde{K} K to the block matrix
K ~ = [ [ Σ λ ′ ] A ⊗ I N t + [ K λ ; 2 D ′ ] A ⊗ Λ t ; 2 D K [ Σ λ ′ ] B ⊗ I N t + [ K λ ; 2 D ′ ] B ⊗ Λ t ; 2 D K [ Σ λ ′ ] C ⊗ I N t + [ K λ ; 2 D ′ ] C ⊗ Λ t ; 2 D K [ Σ λ ′ ] D ⊗ I N t + [ K λ ; 2 D ′ ] D ⊗ Λ t ; 2 D K + ∑ i = 1 N α a i a i T ⊗ K t ; i ] (153) \widetilde{K}=
\begin{bmatrix}
[\Sigma'_{\lambda}]_{A}\otimes I_{N_{t}}+[\mathit{K}'_{\lambda;2D}]_{A}\otimes\Lambda^{K}_{t;2D} &
[\Sigma'_{\lambda}]_{B}\otimes I_{N_{t}}+[\mathit{K}'_{\lambda;2D}]_{B}\otimes\Lambda^{K}_{t;2D}\\
[\Sigma'_{\lambda}]_{C}\otimes I_{N_{t}}+[\mathit{K}'_{\lambda;2D}]_{C}\otimes\Lambda^{K}_{t;2D} &
[\Sigma'_{\lambda}]_{D}\otimes I_{N_{t}}+[\mathit{K}'_{\lambda;2D}]_{D}\otimes\Lambda^{K}_{t;2D}+\sum_{i=1}^{N_{\alpha}}a_i a_i^{T}\otimes K_{t;i}
\end{bmatrix} \tag*{(153)} K = [ [ Σ λ ′ ] A ⊗ I N t + [ K λ ; 2 D ′ ] A ⊗ Λ t ; 2 D K [ Σ λ ′ ] C ⊗ I N t + [ K λ ; 2 D ′ ] C ⊗ Λ t ; 2 D K [ Σ λ ′ ] B ⊗ I N t + [ K λ ; 2 D ′ ] B ⊗ Λ t ; 2 D K [ Σ λ ′ ] D ⊗ I N t + [ K λ ; 2 D ′ ] D ⊗ Λ t ; 2 D K + ∑ i = 1 N α a i a i T ⊗ K t ; i ] (153)
with the individual covariance matrices transformed as described in § 4.7 , where now the D D D block is ( N α N t , N α N t ) (N_{\alpha}N_{t},N_{\alpha}N_{t}) ( N α N t , N α N t ) , instead of the A A A block as in Appendix § 9 .§ 2 . We have two options to decompose the A A A block, either with the LuasLasrachKernel, or the LuasKernel. The LuasLasrachKernel approach can permit linear runtime scaling along one of the dimensions when combined with quasiseparable kernels, while the LuasKernel is more general and is not limited to quasiseparable kernels, but typically cannot achieve linear scaling in either dimension.
Decomposing A with the LuasKernel
In this case, we fully simultaneously diagonalise the A A A block, following the Rakitsch et al. (2013) trick implemented as the LuasKernel. This allows us to write the matrix inverse square root of A A A as:
A − 1 2 = [ I N λ − N α ⊗ I N t + Λ λ ; 2 D K ⊗ Λ t ; 2 D K ] − 1 2 [ Q λ ; 2 D T [ Σ λ ′ ] A − 1 2 ⊗ I N t ] (154) A^{-\frac{1}{2}}=\left[I_{N_{\lambda}-N_{\alpha}}\otimes I_{N_{t}}+\Lambda^{K}_{\lambda;2D}\otimes\Lambda^{K}_{t;2D}\right]^{-\frac{1}{2}}\left[Q^{T}_{\lambda;2D}[\Sigma'_{\lambda}]_{A}^{-\frac{1}{2}}\otimes I_{N_{t}}\right] \tag*{(154)} A − 2 1 = [ I N λ − N α ⊗ I N t + Λ λ ; 2 D K ⊗ Λ t ; 2 D K ] − 2 1 [ Q λ ; 2 D T [ Σ λ ′ ] A − 2 1 ⊗ I N t ] (154)
= D A − 1 2 [ W T ⊗ I N t ] (155) =D_{A}^{-\frac{1}{2}}[W^{T}\otimes I_{N_{t}}] \tag*{(155)} = D A − 2 1 [ W T ⊗ I N t ] (155)
where we’ve relabelled using D A D_{A} D A and W W W to simplify calculations.
We can multiply by one of the two terms in the B B B block as
A − 1 2 [ Σ λ ′ ] B ⊗ I N t = D A − 1 2 [ W T [ Σ λ ′ ] B ⊗ I N t ] . (156) A^{-\frac{1}{2}}[\Sigma'_{\lambda}]_{B}\otimes I_{N_{t}}=D_{A}^{-\frac{1}{2}}[W^{T}[\Sigma'_{\lambda}]_{B}\otimes I_{N_{t}}]. \tag*{(156)} A − 2 1 [ Σ λ ′ ] B ⊗ I N t = D A − 2 1 [ W T [ Σ λ ′ ] B ⊗ I N t ] . (156)
which can be performed efficiently since W T [ Σ λ ′ ] B W^{T}[\Sigma'_{\lambda}]_{B} W T [ Σ λ ′ ] B is a ( N λ − N α , N λ − N α ) (N_{\lambda}-N_{\alpha},N_{\lambda}-N_{\alpha}) ( N λ − N α , N λ − N α ) matrix multiplied by a ( N λ − N α , N α ) (N_{\lambda}-N_{\alpha},N_{\alpha}) ( N λ − N α , N α ) matrix, scaling as O ( N α ( N λ − N α ) 2 ) \mathcal{O}(N_{\alpha}(N_{\lambda}-N_{\alpha})^{2}) O ( N α ( N λ − N α ) 2 ) in runtime cost. Since both terms are diagonal in the time dimension, we can treat this as N t N_{t} N t independent calculations, and store our result as a ( N λ − N α , N α , N t ) (N_{\lambda}-N_{\alpha},N_{\alpha},N_{t}) ( N λ − N α , N α , N t ) tensor. We can add to it the same calculation for the second term in the B B B matrix [ K λ ; 2 D ′ ] B ⊗ Λ t ; 2 D K [\mathit{K}'_{\lambda;2D}]_{B}\otimes\Lambda^{K}_{t;2D} [ K λ ; 2 D ′ ] B ⊗ Λ t ; 2 D K . We can then contract the tensor with itself along the first axis, scaling as O ( N α ( N λ − N α ) N t ) \mathcal{O}(N_{\alpha}(N_{\lambda}-N_{\alpha})N_{t}) O ( N α ( N λ − N α ) N t ) in runtime, to yield a ( N α , N α , N t ) (N_{\alpha},N_{\alpha},N_{t}) ( N α , N α , N t ) tensor which includes all non-zero elements of C A − 1 B CA^{-1}B C A − 1 B .
Decomposing A with the LuasLasrachKernel
Note that A A A , B B B and C C C are already block diagonal in the time dimension, and so the calculation of C A − 1 B CA^{-1}B C A − 1 B can be split up as follows
C A − 1 B = ∑ i = 1 N t ( [ Σ λ ′ ] B + λ K t ; 2 D [ K λ ; 2 D ′ ] B ) T ( [ Σ λ ′ ] A + λ K t ; 2 D [ K λ ; 2 D ′ ] A ) − 1 ( [ Σ λ ′ ] B + λ K t ; 2 D [ K λ ; 2 D ′ ] B ) ⊗ e ^ i e ^ i T (157) CA^{-1}B=\sum_{i=1}^{N_{t}}\left([\Sigma'_{\lambda}]_{B}+\lambda_{K_{t;2D}}[\mathit{K}'_{\lambda;2D}]_{B}\right)^{T}\left([\Sigma'_{\lambda}]_{A}+\lambda_{K_{t;2D}}[\mathit{K}'_{\lambda;2D}]_{A}\right)^{-1}\left([\Sigma'_{\lambda}]_{B}+\lambda_{K_{t;2D}}[\mathit{K}'_{\lambda;2D}]_{B}\right)\otimes\hat{e}_{i}\hat{e}_{i}^{T} \tag*{(157)} C A − 1 B = i = 1 ∑ N t ( [ Σ λ ′ ] B + λ K t ; 2 D [ K λ ; 2 D ′ ] B ) T ( [ Σ λ ′ ] A + λ K t ; 2 D [ K λ ; 2 D ′ ] A ) − 1 ( [ Σ λ ′ ] B + λ K t ; 2 D [ K λ ; 2 D ′ ] B ) ⊗ e ^ i e ^ i T (157)
where the ( N α N t , N α N t ) (N_{\alpha}N_{t},N_{\alpha}N_{t}) ( N α N t , N α N t ) C A − 1 B CA^{-1}B C A − 1 B only contains N α 2 N t N_{\alpha}^{2}N_{t} N α 2 N t non-zero elements since [ Σ λ ′ ] B [\Sigma'_{\lambda}]_{B} [ Σ λ ′ ] B and [ K λ ; 2 D ′ ] B [\mathit{K}'_{\lambda;2D}]_{B} [ K λ ; 2 D ′ ] B only have N α N_{\alpha} N α columns.
Similar to the LuasLasrachKernel, if Σ λ \Sigma_{\lambda} Σ λ and K λ ; 2 D K_{\lambda;2D} K λ ; 2 D are quasiseparable, then it can be much faster to decompose them than eigendecomposition. Each block has a quasiseparable rank of J Σ λ + J K λ ; 2 D + 4 N α J_{\Sigma_{\lambda}}+J_{K_{\lambda;2D}}+4N_{\alpha} J Σ λ + J K λ ; 2 D + 4 N α after applying N α N_{\alpha} N α Householder transformations to both Σ λ \Sigma_{\lambda} Σ λ and K λ ; 2 D K_{\lambda;2D} K λ ; 2 D . Decomposing N t N_{t} N t separate blocks, where each matrix is ( N λ − N α , N λ − N α ) (N_{\lambda}-N_{\alpha},N_{\lambda}-N_{\alpha}) ( N λ − N α , N λ − N α ) , scales as O ( ( N λ − N α ) ( J Σ λ + J K λ + 4 N α ) ) \mathcal{O}((N_{\lambda}-N_{\alpha})(J_{\Sigma_{\lambda}}+J_{K_{\lambda}}+4N_{\alpha})) O (( N λ − N α ) ( J Σ λ + J K λ + 4 N α )) in runtime.
Afterwards, the C A − 1 B CA^{-1}B C A − 1 B calculation requires N α N t N_{\alpha}N_{t} N α N t matrix-vector products of each of these blocks, scaling as O ( N α ( N λ − N α ) N t ( J Σ λ + J K λ + 4 N α ) ) \mathcal{O}(N_{\alpha}(N_{\lambda}-N_{\alpha})N_{t}(J_{\Sigma_{\lambda}}+J_{K_{\lambda}}+4N_{\alpha})) O ( N α ( N λ − N α ) N t ( J Σ λ + J K λ + 4 N α )) , which is dominated by the previous scaling to decompose the quasiseparable blocks. Similar to Appendix B.§ 1 , the result of this can be stored as a ( N λ − N α , N α , N t ) (N_{\lambda}-N_{\alpha},N_{\alpha},N_{t}) ( N λ − N α , N α , N t ) tensor, which we contract along the first axis to recover a ( N α , N α , N t ) (N_{\alpha},N_{\alpha},N_{t}) ( N α , N α , N t ) tensor which contains all non-zero values in C A − 1 B CA^{-1}B C A − 1 B .
Decomposing D − CA⁻¹B
The remaining calculation is the same regardless of the approach used to calculate decompose A A A and calculate C A − 1 B CA^{-1}B C A − 1 B . We simply build the ( N α N t , N α N t ) (N_{\alpha}N_{t},N_{\alpha}N_{t}) ( N α N t , N α N t ) D D D block and subtract C A − 1 B CA^{-1}B C A − 1 B . We can then decompose D − C A − 1 B D-CA^{-1}B D − C A − 1 B using general Cholesky decomposition because we cannot exploit any quasiseparability, which scales as O ( N α 3 N t 3 ) \mathcal{O}(N_{\alpha}^{3}N_{t}^{3}) O ( N α 3 N t 3 ) in runtime, and costs O ( N α 2 N t 2 ) \mathcal{O}(N_{\alpha}^{2}N_{t}^{2}) O ( N α 2 N t 2 ) in memory.
Total Computational Cost
We recover an overall runtime scaling dominated by O ( ( 1 + N α 3 ) N λ 3 + N λ N t 2 + N λ N t ( J Σ λ + J K λ + 4 N α ) 2 ) \mathcal{O}((1+N_{\alpha}^{3})N_{\lambda}^{3}+N_{\lambda}N_{t}^{2}+N_{\lambda}N_{t}(J_{\Sigma_{\lambda}}+J_{K_{\lambda}}+4N_{\alpha})^{2}) O (( 1 + N α 3 ) N λ 3 + N λ N t 2 + N λ N t ( J Σ λ + J K λ + 4 N α ) 2 ) if we exploit quasiseparability in the wavelength dimension, linear in N λ N_{\lambda} N λ . If we do not use any quasiseparable kernels, we still recover a scaling of O ( ( 1 + N α 3 ) N λ 3 + N λ 3 + N λ N t ( N λ + N α 2 ) ) \mathcal{O}((1+N_{\alpha}^{3})N_{\lambda}^{3}+N_{\lambda}^{3}+N_{\lambda}N_{t}(N_{\lambda}+N_{\alpha}^{2})) O (( 1 + N α 3 ) N λ 3 + N λ 3 + N λ N t ( N λ + N α 2 )) , for which benchmarking is presented in Figure G28 and Figure G29 on CPU and GPU respectively.
The memory scaling for the quasiseparable case is dominated by building the D − C A − 1 B D-CA^{-1}B D − C A − 1 B matrix, and the decomposition of the N t N_{t} N t quasiseparable blocks of [ K ~ ] λ [\widetilde{K}]_{\lambda} [ K ] λ , working out to ∼ O ( N α 2 N t 2 + N λ N t ( J Σ λ + J K λ + 4 N α ) ) \sim\mathcal{O}(N_{\alpha}^{2}N_{t}^{2}+N_{\lambda}N_{t}(J_{\Sigma_{\lambda}}+J_{K_{\lambda}}+4N_{\alpha})) ∼ O ( N α 2 N t 2 + N λ N t ( J Σ λ + J K λ + 4 N α )) which is also linear in N λ N_{\lambda} N λ . The more general case using the LuasKernel builds full wavelength covariance matrices, and has a memory scaling of O ( N λ 2 + N α 2 N t 2 + N α N λ N t ) \mathcal{O}(N_{\lambda}^{2}+N_{\alpha}^{2}N_{t}^{2}+N_{\alpha}N_{\lambda}N_{t}) O ( N λ 2 + N α 2 N t 2 + N α N λ N t ) , including the tensor stored in Appendix B.§ 1 .
Alternatively, we can flip the kernel dimensions around and recover a runtime scaling linear in the number of time points, provided we exploit quasiseparability in the time dimension, which Figure G26 and Figure G27 demonstrate.
Decomposing for the ExtendedLuasKernel
We can write K ~ \widetilde{K} K as the block matrix
K ~ = [ ( I N λ − N α ⊗ I N t + [ K λ ; 2 D ′ ] A ⊗ K t ; 2 D ′ + ∑ j = 1 N β [ K λ ; j ′ ] A ⊗ [ b j 0 b j 0 ] [ b j 0 b j 0 ] T ) ( [ K λ ; 2 D ′ ] B ⊗ K t ; 2 D ′ + ∑ j = 1 N β [ K λ ; j ′ ] B ⊗ [ b j 0 b j 0 ] [ b j 0 b j 0 ] T ) ( [ K λ ; 2 D ′ ] C ⊗ K t ; 2 D ′ + ∑ j = 1 N β [ K λ ; j ′ ] C ⊗ [ b j 0 b j 0 ] [ b j 0 b j 0 ] T ) ( I N α ⊗ I N t + [ K λ ; 2 D ′ ] D ⊗ K t ; 2 D ′ + ∑ i = 1 N α a i a i T ⊗ K t ; i ′ + ∑ j = 1 N β [ K λ ; j ′ ] D ⊗ [ b j 0 b j 0 ] [ b j 0 b j 0 ] T ) ] (158) \widetilde{K}=
\begin{bmatrix}
\left(I_{N_{\lambda}-N_{\alpha}}\otimes I_{N_{t}}+[\mathit{K}'_{\lambda;2D}]_{A}\otimes K'_{t;2D}+\sum_{j=1}^{N_{\beta}}[\mathit{K}'_{\lambda;j}]_{A}\otimes\begin{bmatrix}b^{0}_{j}\\ b^{0}_{j}\end{bmatrix}\begin{bmatrix}b^{0}_{j}\\ b^{0}_{j}\end{bmatrix}^{T}\right) &
\left([\mathit{K}'_{\lambda;2D}]_{B}\otimes K'_{t;2D}+\sum_{j=1}^{N_{\beta}}[\mathit{K}'_{\lambda;j}]_{B}\otimes\begin{bmatrix}b^{0}_{j}\\ b^{0}_{j}\end{bmatrix}\begin{bmatrix}b^{0}_{j}\\ b^{0}_{j}\end{bmatrix}^{T}\right)\\
\left([\mathit{K}'_{\lambda;2D}]_{C}\otimes K'_{t;2D}+\sum_{j=1}^{N_{\beta}}[\mathit{K}'_{\lambda;j}]_{C}\otimes\begin{bmatrix}b^{0}_{j}\\ b^{0}_{j}\end{bmatrix}\begin{bmatrix}b^{0}_{j}\\ b^{0}_{j}\end{bmatrix}^{T}\right) &
\left(I_{N_{\alpha}}\otimes I_{N_{t}}+[\mathit{K}'_{\lambda;2D}]_{D}\otimes K'_{t;2D}+\sum_{i=1}^{N_{\alpha}}a_i a_i^{T}\otimes K'_{t;i}+\sum_{j=1}^{N_{\beta}}[\mathit{K}'_{\lambda;j}]_{D}\otimes\begin{bmatrix}b^{0}_{j}\\ b^{0}_{j}\end{bmatrix}\begin{bmatrix}b^{0}_{j}\\ b^{0}_{j}\end{bmatrix}^{T}\right)
\end{bmatrix} \tag*{(158)} K = ( I N λ − N α ⊗ I N t + [ K λ ; 2 D ′ ] A ⊗ K t ; 2 D ′ + ∑ j = 1 N β [ K λ ; j ′ ] A ⊗ [ b j 0 b j 0 ] [ b j 0 b j 0 ] T ) ( [ K λ ; 2 D ′ ] C ⊗ K t ; 2 D ′ + ∑ j = 1 N β [ K λ ; j ′ ] C ⊗ [ b j 0 b j 0 ] [ b j 0 b j 0 ] T ) ( [ K λ ; 2 D ′ ] B ⊗ K t ; 2 D ′ + ∑ j = 1 N β [ K λ ; j ′ ] B ⊗ [ b j 0 b j 0 ] [ b j 0 b j 0 ] T ) ( I N α ⊗ I N t + [ K λ ; 2 D ′ ] D ⊗ K t ; 2 D ′ + ∑ i = 1 N α a i a i T ⊗ K t ; i ′ + ∑ j = 1 N β [ K λ ; j ′ ] D ⊗ [ b j 0 b j 0 ] [ b j 0 b j 0 ] T ) (158)
where the A A A block is simply in the form of the LuasPlusMultiLowRankKernel and can be decomposed readily. We make use of the LuasKernel method of decomposing it, as it simplifies the C A − 1 B CA^{-1}B C A − 1 B calculation, which gives a runtime scaling of ∼ O ( ( 1 + N β 3 ) N λ 2 + N λ 3 + N α N t ( N λ + N β 2 ) ) \sim\mathcal{O}((1+N_{\beta}^{3})N_{\lambda}^{2}+N_{\lambda}^{3}+N_{\alpha}N_{t}(N_{\lambda}+N_{\beta}^{2})) ∼ O (( 1 + N β 3 ) N λ 2 + N λ 3 + N α N t ( N λ + N β 2 )) , while the D − C A − 1 B D-CA^{-1}B D − C A − 1 B matrix will be decomposed using Cholesky factorisation with a runtime scaling of O ( N α 3 N t 3 ) \mathcal{O}(N_{\alpha}^{3}N_{t}^{3}) O ( N α 3 N t 3 ) .
Calculation of C A − 1 B \mathrm{CA}^{-1}\mathrm{B} CA − 1 B
We further subdivide the A, B and C blocks into their own 2x2 block matrices, this time splitting the ( N t , N t ) (N_t,N_t) ( N t , N t ) time covariance matrices as [ N t − N β , N β ] [N_t-N_\beta,N_\beta] [ N t − N β , N β ] :
[ K ~ ] A = [ ( I N λ − N α ⊗ I N t − N β + [ K λ ; 2 D ′ ] A ⊗ [ K t ; 2 D ′ ] A ) [ K λ ; 2 D ′ ] A ⊗ [ K t ; 2 D ′ ] B [ K λ ; 2 D ′ ] A ⊗ [ K t ; 2 D ′ ] C ( I N λ − N α ⊗ I N β + [ K λ ; 2 D ′ ] A ⊗ [ K t ; 2 D ′ ] D + ∑ j = 1 N ρ [ K λ ; j ′ ] A ⊗ b j b j T ) ] . (159) [\widetilde{\mathrm{K}}]_{\mathrm{A}}=
\begin{bmatrix}
\left(\mathrm{I}_{N_\lambda-N_\alpha}\otimes\mathrm{I}_{N_t-N_\beta}+[\mathrm{K}'_{\lambda;2\mathrm{D}}]_{\mathrm{A}}\otimes[\mathrm{K}'_{t;2\mathrm{D}}]_{\mathrm{A}}\right) & [\mathrm{K}'_{\lambda;2\mathrm{D}}]_{\mathrm{A}}\otimes[\mathrm{K}'_{t;2\mathrm{D}}]_{\mathrm{B}}\\
[\mathrm{K}'_{\lambda;2\mathrm{D}}]_{\mathrm{A}}\otimes[\mathrm{K}'_{t;2\mathrm{D}}]_{\mathrm{C}} & \left(\mathrm{I}_{N_\lambda-N_\alpha}\otimes\mathrm{I}_{N_\beta}+[\mathrm{K}'_{\lambda;2\mathrm{D}}]_{\mathrm{A}}\otimes[\mathrm{K}'_{t;2\mathrm{D}}]_{\mathrm{D}}+\sum_{j=1}^{N_\rho}[\mathrm{K}'_{\lambda;j}]_{\mathrm{A}}\otimes\mathbf{b}_j\mathbf{b}_j^{\mathsf{T}}\right)
\end{bmatrix}. \tag*{(159)} [ K ] A = ( I N λ − N α ⊗ I N t − N β + [ K λ ; 2 D ′ ] A ⊗ [ K t ; 2 D ′ ] A ) [ K λ ; 2 D ′ ] A ⊗ [ K t ; 2 D ′ ] C [ K λ ; 2 D ′ ] A ⊗ [ K t ; 2 D ′ ] B ( I N λ − N α ⊗ I N β + [ K λ ; 2 D ′ ] A ⊗ [ K t ; 2 D ′ ] D + ∑ j = 1 N ρ [ K λ ; j ′ ] A ⊗ b j b j T ) . (159)
Note that since we are splitting the second term in the Kronecker product, the A, B and C blocks will actually be mixed between each other rather than fully separated, but the block matrix inversion formulae are still valid.
Then write the B block, let us first label the following matrices as
B D ; 1 = [ K λ ; 2 D ′ ] B ⊗ I N β (160) \mathrm{B}_{\mathrm{D};1}=[\mathrm{K}'_{\lambda;2\mathrm{D}}]_{\mathrm{B}}\otimes\mathrm{I}_{N_\beta} \tag*{(160)} B D ; 1 = [ K λ ; 2 D ′ ] B ⊗ I N β (160)
B D ; 2 = ∑ j = 1 N β [ K λ ; j ′ ] B ⊗ b j b j T (161) \mathrm{B}_{\mathrm{D};2}=\sum_{j=1}^{N_\beta}[\mathrm{K}'_{\lambda;j}]_{\mathrm{B}}\otimes\mathbf{b}_j\mathbf{b}_j^{\mathsf{T}} \tag*{(161)} B D ; 2 = j = 1 ∑ N β [ K λ ; j ′ ] B ⊗ b j b j T (161)
and we then write the B block as the sum of 2x2 block matrices
[ K ~ ] B = [ [ K λ ; 2 D ′ ] B ⊗ I N t − N β 0 0 B D ; 1 ] [ I N α ⊗ K t ; 2 D ′ ] + [ 0 0 0 B D ; 2 ] (162) [\widetilde{\mathrm{K}}]_{\mathrm{B}}=
\begin{bmatrix}
[\mathrm{K}'_{\lambda;2\mathrm{D}}]_{\mathrm{B}}\otimes\mathrm{I}_{N_t-N_\beta} & 0\\
0 & \mathrm{B}_{\mathrm{D};1}
\end{bmatrix}
[\mathrm{I}_{N_\alpha}\otimes\mathrm{K}'_{t;2\mathrm{D}}]
+
\begin{bmatrix}
0&0\\
0&\mathrm{B}_{\mathrm{D};2}
\end{bmatrix} \tag*{(162)} [ K ] B = [ [ K λ ; 2 D ′ ] B ⊗ I N t − N β 0 0 B D ; 1 ] [ I N α ⊗ K t ; 2 D ′ ] + [ 0 0 0 B D ; 2 ] (162)
Using this form for the B block, we split up the C A − 1 B \mathrm{CA}^{-1}\mathrm{B} CA − 1 B calculation into separate terms of the form
[ C A 0 0 C D ] [ [ A − 1 ] A [ A − 1 ] B [ A − 1 ] C [ A − 1 ] D ] [ B A 0 0 B D ] = [ C A [ A − 1 ] A B A C A [ A − 1 ] B B D C D [ A − 1 ] C B A C D [ A − 1 ] D B D ] (163) \begin{bmatrix}
\mathrm{C}_{\mathrm{A}}&0\\
0&\mathrm{C}_{\mathrm{D}}
\end{bmatrix}
\begin{bmatrix}
[\mathrm{A}^{-1}]_{\mathrm{A}}&[\mathrm{A}^{-1}]_{\mathrm{B}}\\
[\mathrm{A}^{-1}]_{\mathrm{C}}&[\mathrm{A}^{-1}]_{\mathrm{D}}
\end{bmatrix}
\begin{bmatrix}
\mathrm{B}_{\mathrm{A}}&0\\
0&\mathrm{B}_{\mathrm{D}}
\end{bmatrix}
=
\begin{bmatrix}
\mathrm{C}_{\mathrm{A}}[\mathrm{A}^{-1}]_{\mathrm{A}}\mathrm{B}_{\mathrm{A}}&\mathrm{C}_{\mathrm{A}}[\mathrm{A}^{-1}]_{\mathrm{B}}\mathrm{B}_{\mathrm{D}}\\
\mathrm{C}_{\mathrm{D}}[\mathrm{A}^{-1}]_{\mathrm{C}}\mathrm{B}_{\mathrm{A}}&\mathrm{C}_{\mathrm{D}}[\mathrm{A}^{-1}]_{\mathrm{D}}\mathrm{B}_{\mathrm{D}} \tag*{(163)}
\end{bmatrix} [ C A 0 0 C D ] [ [ A − 1 ] A [ A − 1 ] C [ A − 1 ] B [ A − 1 ] D ] [ B A 0 0 B D ] = [ C A [ A − 1 ] A B A C D [ A − 1 ] C B A C A [ A − 1 ] B B D C D [ A − 1 ] D B D ] (163)
and add everything together as
C A − 1 B = [ I N α ⊗ K t ; 2 D ′ ] [ C A [ A − 1 ] A B A C A [ A − 1 ] B B D ; 1 C D ; 1 [ A − 1 ] C B A C D ; 1 [ A − 1 ] D B D ; 1 ] [ I N α ⊗ K t ; 2 D ′ ] + [ I N α ⊗ K t ; 2 D ′ ] [ 0 C A [ A − 1 ] B B D ; 2 0 C D ; 1 [ A − 1 ] D B D ; 2 ] + [ 0 0 C D ; 2 [ A − 1 ] C B A C D ; 2 [ A − 1 ] D B D ; 1 ] [ I N α ⊗ K t ; 2 D ′ ] + [ 0 0 0 C D ; 2 [ A − 1 ] D B D ; 2 ] . (164) \begin{aligned}
\mathrm{CA}^{-1}\mathrm{B}={}&[\mathrm{I}_{N_\alpha}\otimes\mathrm{K}'_{t;2\mathrm{D}}]
\begin{bmatrix}
\mathrm{C}_{\mathrm{A}}[\mathrm{A}^{-1}]_{\mathrm{A}}\mathrm{B}_{\mathrm{A}}&\mathrm{C}_{\mathrm{A}}[\mathrm{A}^{-1}]_{\mathrm{B}}\mathrm{B}_{\mathrm{D};1}\\
\mathrm{C}_{\mathrm{D};1}[\mathrm{A}^{-1}]_{\mathrm{C}}\mathrm{B}_{\mathrm{A}}&\mathrm{C}_{\mathrm{D};1}[\mathrm{A}^{-1}]_{\mathrm{D}}\mathrm{B}_{\mathrm{D};1}
\end{bmatrix}
[\mathrm{I}_{N_\alpha}\otimes\mathrm{K}'_{t;2\mathrm{D}}]\\
&+[\mathrm{I}_{N_\alpha}\otimes\mathrm{K}'_{t;2\mathrm{D}}]
\begin{bmatrix}
0&\mathrm{C}_{\mathrm{A}}[\mathrm{A}^{-1}]_{\mathrm{B}}\mathrm{B}_{\mathrm{D};2}\\
0&\mathrm{C}_{\mathrm{D};1}[\mathrm{A}^{-1}]_{\mathrm{D}}\mathrm{B}_{\mathrm{D};2}
\end{bmatrix}\\
&+\begin{bmatrix}
0&0\\
\mathrm{C}_{\mathrm{D};2}[\mathrm{A}^{-1}]_{\mathrm{C}}\mathrm{B}_{\mathrm{A}}&\mathrm{C}_{\mathrm{D};2}[\mathrm{A}^{-1}]_{\mathrm{D}}\mathrm{B}_{\mathrm{D};1}
\end{bmatrix}
[\mathrm{I}_{N_\alpha}\otimes\mathrm{K}'_{t;2\mathrm{D}}]\\
&+\begin{bmatrix}
0&0\\
0&\mathrm{C}_{\mathrm{D};2}[\mathrm{A}^{-1}]_{\mathrm{D}}\mathrm{B}_{\mathrm{D};2}
\end{bmatrix}. \tag*{(164)}
\end{aligned} CA − 1 B = [ I N α ⊗ K t ; 2 D ′ ] [ C A [ A − 1 ] A B A C D ; 1 [ A − 1 ] C B A C A [ A − 1 ] B B D ; 1 C D ; 1 [ A − 1 ] D B D ; 1 ] [ I N α ⊗ K t ; 2 D ′ ] + [ I N α ⊗ K t ; 2 D ′ ] [ 0 0 C A [ A − 1 ] B B D ; 2 C D ; 1 [ A − 1 ] D B D ; 2 ] + [ 0 C D ; 2 [ A − 1 ] C B A 0 C D ; 2 [ A − 1 ] D B D ; 1 ] [ I N α ⊗ K t ; 2 D ′ ] + [ 0 0 0 C D ; 2 [ A − 1 ] D B D ; 2 ] . (164)
where we can build the ( N α N t , N α N t ) (N_\alpha N_t,N_\alpha N_t) ( N α N t , N α N t ) inner block matrices in memory, and after this we can perform the required matrix multiplications by the [ I N α ⊗ K t ; 2 D ′ ] [\mathrm{I}_{N_\alpha}\otimes\mathrm{K}'_{t;2\mathrm{D}}] [ I N α ⊗ K t ; 2 D ′ ] terms in O ( N α 3 N t 3 ) \mathcal{O}(N_\alpha^3N_t^3) O ( N α 3 N t 3 ) time. Therefore, we just need to be able to build these inner block matrix terms to calculate C A − 1 B \mathrm{CA}^{-1}\mathrm{B} CA − 1 B . Using the 2x2 block matrix inverse formula for A = [ K ~ ] A \mathrm{A}=[\widetilde{\mathrm{K}}]_{\mathrm{A}} A = [ K ] A , we can calculate each of these terms as
[ C A A A − 1 B A + ( C A A A − 1 A B ) [ A − 1 ] D ( A A T A A − 1 B A ) − ( C A A A − 1 A B ) [ A − 1 ] D B D ; i − C D ; i [ A − 1 ] D ( A A T A A − 1 B A ) C D ; i [ A − 1 ] D B D ; i ] (165) \begin{bmatrix}
\mathrm{C}_{\mathrm{A}}\mathrm{A}_{\mathrm{A}}^{-1}\mathrm{B}_{\mathrm{A}}+\left(\mathrm{C}_{\mathrm{A}}\mathrm{A}_{\mathrm{A}}^{-1}\mathrm{A}_{\mathrm{B}}\right)[\mathrm{A}^{-1}]_{\mathrm{D}}\left(\mathrm{A}_{\mathrm{A}}^{\mathsf{T}}\mathrm{A}_{\mathrm{A}}^{-1}\mathrm{B}_{\mathrm{A}}\right)-\left(\mathrm{C}_{\mathrm{A}}\mathrm{A}_{\mathrm{A}}^{-1}\mathrm{A}_{\mathrm{B}}\right)[\mathrm{A}^{-1}]_{\mathrm{D}}\mathrm{B}_{\mathrm{D};i}\\
-\mathrm{C}_{\mathrm{D};i}[\mathrm{A}^{-1}]_{\mathrm{D}}\left(\mathrm{A}_{\mathrm{A}}^{\mathsf{T}}\mathrm{A}_{\mathrm{A}}^{-1}\mathrm{B}_{\mathrm{A}}\right)\qquad\mathrm{C}_{\mathrm{D};i}[\mathrm{A}^{-1}]_{\mathrm{D}}\mathrm{B}_{\mathrm{D};i} \tag*{(165)}
\end{bmatrix} [ C A A A − 1 B A + ( C A A A − 1 A B ) [ A − 1 ] D ( A A T A A − 1 B A ) − ( C A A A − 1 A B ) [ A − 1 ] D B D ; i − C D ; i [ A − 1 ] D ( A A T A A − 1 B A ) C D ; i [ A − 1 ] D B D ; i ] (165)
where we have from Equation (159) :
A A = I N λ − N α ⊗ I N t + [ K λ ; 2 D ′ ] A ⊗ [ K t ; 2 D ′ ] A , (166) \mathrm{A}_{\mathrm{A}}=\mathrm{I}_{N_\lambda-N_\alpha}\otimes\mathrm{I}_{N_t}+[\mathrm{K}'_{\lambda;2\mathrm{D}}]_{\mathrm{A}}\otimes[\mathrm{K}'_{t;2\mathrm{D}}]_{\mathrm{A}}, \tag*{(166)} A A = I N λ − N α ⊗ I N t + [ K λ ; 2 D ′ ] A ⊗ [ K t ; 2 D ′ ] A , (166)
A B = [ K λ ; 2 D ′ ] A ⊗ [ K t ; 2 D ′ ] B , (167) \mathrm{A}_{\mathrm{B}}=[\mathrm{K}'_{\lambda;2\mathrm{D}}]_{\mathrm{A}}\otimes[\mathrm{K}'_{t;2\mathrm{D}}]_{\mathrm{B}}, \tag*{(167)} A B = [ K λ ; 2 D ′ ] A ⊗ [ K t ; 2 D ′ ] B , (167)
A C = [ K λ ; 2 D ′ ] A ⊗ [ K t ; 2 D ′ ] C , (168) \mathrm{A}_{\mathrm{C}}=[\mathrm{K}'_{\lambda;2\mathrm{D}}]_{\mathrm{A}}\otimes[\mathrm{K}'_{t;2\mathrm{D}}]_{\mathrm{C}}, \tag*{(168)} A C = [ K λ ; 2 D ′ ] A ⊗ [ K t ; 2 D ′ ] C , (168)
A D = I N λ − N α ⊗ I N β + [ K λ ; 2 D ′ ] A ⊗ [ K t ; 2 D ′ ] D + ∑ j = 1 N β [ K λ ; j ′ ] A ⊗ b j b j T (169) \mathrm{A}_{\mathrm{D}}=\mathrm{I}_{N_\lambda-N_\alpha}\otimes\mathrm{I}_{N_\beta}+[\mathrm{K}'_{\lambda;2\mathrm{D}}]_{\mathrm{A}}\otimes[\mathrm{K}'_{t;2\mathrm{D}}]_{\mathrm{D}}+\sum_{j=1}^{N_\beta}[\mathrm{K}'_{\lambda;j}]_{\mathrm{A}}\otimes\mathbf{b}_j\mathbf{b}_j^{\mathsf{T}} \tag*{(169)} A D = I N λ − N α ⊗ I N β + [ K λ ; 2 D ′ ] A ⊗ [ K t ; 2 D ′ ] D + j = 1 ∑ N β [ K λ ; j ′ ] A ⊗ b j b j T (169)
[ A − 1 ] D = ( A D − A C A A − 1 A B ) − 1 (170) [\mathrm{A}^{-1}]_{\mathrm{D}}=(\mathrm{A}_{\mathrm{D}}-\mathrm{A}_{\mathrm{C}}\mathrm{A}_{\mathrm{A}}^{-1}\mathrm{A}_{\mathrm{B}})^{-1} \tag*{(170)} [ A − 1 ] D = ( A D − A C A A − 1 A B ) − 1 (170)
where [ A − 1 ] D [\mathrm{A}^{-1}]_{\mathrm{D}} [ A − 1 ] D is calculated following Appendix B, as it is also needed for the LUASPLUSMULTILOWRANKKERNEL calculation.
The C A A A − 1 B A \mathrm{C}_{\mathrm{A}}\mathrm{A}_{\mathrm{A}}^{-1}\mathrm{B}_{\mathrm{A}} C A A A − 1 B A Term
The C A A A − 1 B A \mathrm{C}_{\mathrm{A}}\mathrm{A}_{\mathrm{A}}^{-1}\mathrm{B}_{\mathrm{A}} C A A A − 1 B A term in Equation (164) can be calculated by simultaneously diagonalising A A \mathrm{A}_{\mathrm{A}} A A in both dimensions, similar to the LUASKERNEL. To simplify the notation, we will write its diagonal matrix of eigenvalues as
Λ A A = I N λ − N α ⊗ I N t − N β + Λ [ K λ ; 2 D ′ ] A ⊗ Λ [ K t ; 2 D ′ ] A (171) \boldsymbol{\Lambda}_{\mathrm{A}_{\mathrm{A}}}=\mathrm{I}_{N_\lambda-N_\alpha}\otimes\mathrm{I}_{N_t-N_\beta}+\boldsymbol{\Lambda}_{[\mathrm{K}'_{\lambda;2\mathrm{D}}]_{\mathrm{A}}}\otimes\boldsymbol{\Lambda}_{[\mathrm{K}'_{t;2\mathrm{D}}]_{\mathrm{A}}} \tag*{(171)} Λ A A = I N λ − N α ⊗ I N t − N β + Λ [ K λ ; 2 D ′ ] A ⊗ Λ [ K t ; 2 D ′ ] A (171)
and we define the ( N λ − N α , N α ) (N_\lambda-N_\alpha,N_\alpha) ( N λ − N α , N α ) matrix:
V B = [ K λ ; 2 D ′ ] B . (172) \mathrm{V}_{\mathrm{B}}=[\mathrm{K}'_{\lambda;2\mathrm{D}}]_{\mathrm{B}}. \tag*{(172)} V B = [ K λ ; 2 D ′ ] B . (172)
We can then efficiently compute C A A A − 1 B A \mathrm{C}_{\mathrm{A}}\mathrm{A}_{\mathrm{A}}^{-1}\mathrm{B}_{\mathrm{A}} C A A A − 1 B A as
C A A A − 1 B A = [ V B T ⊗ I N t − N β ] [ Q [ K λ ; 2 D ′ ] A ⊗ Q [ K t ; 2 D ′ ] A Λ A A − 1 \mathrm{C}_{\mathrm{A}}\mathrm{A}_{\mathrm{A}}^{-1}\mathrm{B}_{\mathrm{A}}=[\mathrm{V}_{\mathrm{B}}^{\mathsf{T}}\otimes\mathrm{I}_{N_t-N_\beta}][\mathrm{Q}_{[\mathrm{K}'_{\lambda;2\mathrm{D}}]_{\mathrm{A}}}\otimes\mathrm{Q}_{[\mathrm{K}'_{t;2\mathrm{D}}]_{\mathrm{A}}}\boldsymbol{\Lambda}_{\mathrm{A}_{\mathrm{A}}}^{-1} C A A A − 1 B A = [ V B T ⊗ I N t − N β ] [ Q [ K λ ; 2 D ′ ] A ⊗ Q [ K t ; 2 D ′ ] A Λ A A − 1
[ Q [ K λ ; 2 D ′ ] A T ⊗ Q [ K t ; 2 D ′ ] A T ] [ V B ⊗ I N t − N β ] (173) [\mathrm{Q}_{[\mathrm{K}'_{\lambda;2\mathrm{D}}]_{\mathrm{A}}}^{\mathsf{T}}\otimes\mathrm{Q}_{[\mathrm{K}'_{t;2\mathrm{D}}]_{\mathrm{A}}}^{\mathsf{T}}][\mathrm{V}_{\mathrm{B}}\otimes\mathrm{I}_{N_t-N_\beta}] \tag*{(173)} [ Q [ K λ ; 2 D ′ ] A T ⊗ Q [ K t ; 2 D ′ ] A T ] [ V B ⊗ I N t − N β ] (173)
= [ I N α ⊗ Q [ K t ; 2 D ′ ] A ] [ V B T ⊗ I N t − N β ] Λ A A − 1 [ V B ⊗ I N t − N β ] [ I N α ⊗ Q [ K t ; 2 D ′ ] A T ] (174) =[\mathrm{I}_{N_\alpha}\otimes\mathrm{Q}_{[\mathrm{K}'_{t;2\mathrm{D}}]_{\mathrm{A}}}][\mathrm{V}_{\mathrm{B}}^{\mathsf{T}}\otimes\mathrm{I}_{N_t-N_\beta}]\boldsymbol{\Lambda}_{\mathrm{A}_{\mathrm{A}}}^{-1}[\mathrm{V}_{\mathrm{B}}\otimes\mathrm{I}_{N_t-N_\beta}][\mathrm{I}_{N_\alpha}\otimes\mathrm{Q}_{[\mathrm{K}'_{t;2\mathrm{D}}]_{\mathrm{A}}}^{\mathsf{T}}] \tag*{(174)} = [ I N α ⊗ Q [ K t ; 2 D ′ ] A ] [ V B T ⊗ I N t − N β ] Λ A A − 1 [ V B ⊗ I N t − N β ] [ I N α ⊗ Q [ K t ; 2 D ′ ] A T ] (174)
= [ I N α ⊗ Q [ K t ; 2 D ′ ] A ] [ ∑ i = 1 N t − N β V B T Λ A A ; i − 1 V B ⊗ e ^ i e ^ i T ] [ I N α ⊗ Q [ K t ; 2 D ′ ] A T ] (175) =[\mathrm{I}_{N_\alpha}\otimes\mathrm{Q}_{[\mathrm{K}'_{t;2\mathrm{D}}]_{\mathrm{A}}}]
\left[
\sum_{i=1}^{N_t-N_\beta}\mathrm{V}_{\mathrm{B}}^{\mathsf{T}}\boldsymbol{\Lambda}_{\mathrm{A}_{\mathrm{A};i}}^{-1}\mathrm{V}_{\mathrm{B}}\otimes\hat{\mathbf{e}}_i\hat{\mathbf{e}}_i^{\mathsf{T}}
\right]
[\mathrm{I}_{N_\alpha}\otimes\mathrm{Q}_{[\mathrm{K}'_{t;2\mathrm{D}}]_{\mathrm{A}}}^{\mathsf{T}}] \tag*{(175)} = [ I N α ⊗ Q [ K t ; 2 D ′ ] A ] i = 1 ∑ N t − N β V B T Λ A A ; i − 1 V B ⊗ e ^ i e ^ i T [ I N α ⊗ Q [ K t ; 2 D ′ ] A T ] (175)
Although M 1 \mathrm{M}_1 M 1 is block diagonal, the simplest approach for performing the outer matrix multiplications is simply to build it as a full ( N α ( N t − N β ) , N α ( N t − N β ) ) (N_\alpha(N_t-N_\beta),N_\alpha(N_t-N_\beta)) ( N α ( N t − N β ) , N α ( N t − N β )) matrix, costing O ( N α 2 N t 2 ) \mathcal{O}(N_\alpha^2N_t^2) O ( N α 2 N t 2 ) in memory. The dominant runtime cost of this calculation is the simultaneous eigendecomposition of both matrices, scaling as O ( N λ 3 + N t 3 + N λ N t ( N λ + N t ) ) \mathcal{O}(N_\lambda^3+N_t^3+N_\lambda N_t(N_\lambda+N_t)) O ( N λ 3 + N t 3 + N λ N t ( N λ + N t )) , with the calculation of M 1 \mathrm{M}_1 M 1 scaling as O ( N α 2 N λ N t ) \mathcal{O}(N_\alpha^2N_\lambda N_t) O ( N α 2 N λ N t ) in runtime, and the O ( N α 3 N t 3 ) \mathcal{O}(N_\alpha^3N_t^3) O ( N α 3 N t 3 ) matrix multiplications of the outer terms will be later factored out by the B Q \mathrm{B}_{\mathrm{Q}} B Q term in Equation (182) .
The A B T A A − 1 B A \mathrm{A}_{\mathrm{B}}^{\mathsf{T}}\mathrm{A}_{\mathrm{A}}^{-1}\mathrm{B}_{\mathrm{A}} A B T A A − 1 B A and C A A A − 1 A B \mathrm{C}_{\mathrm{A}}\mathrm{A}_{\mathrm{A}}^{-1}\mathrm{A}_{\mathrm{B}} C A A A − 1 A B Terms
Factoring using the eigendecompositions of [ K λ ; 2 D ′ ] A [\mathrm{K}'_{\lambda;2\mathrm{D}}]_{\mathrm{A}} [ K λ ; 2 D ′ ] A and [ K t ; 2 D ′ ] B [\mathrm{K}'_{t;2\mathrm{D}}]_{\mathrm{B}} [ K t ; 2 D ′ ] B we have already calculated:
A B T A A − 1 B A = ( [ K λ ; 2 D ′ ] A ⊗ [ K t ; 2 D ′ ] B ) T ( I N λ − N α ⊗ I N t + [ K λ ; 2 D ′ ] A ⊗ [ K t ; 2 D ′ ] A ) − 1 [ V B ⊗ I N t − N β ] (176) \mathrm{A}_{\mathrm{B}}^{\mathsf{T}}\mathrm{A}_{\mathrm{A}}^{-1}\mathrm{B}_{\mathrm{A}}=
\left([\mathrm{K}'_{\lambda;2\mathrm{D}}]_{\mathrm{A}}\otimes[\mathrm{K}'_{t;2\mathrm{D}}]_{\mathrm{B}}\right)^{\mathsf{T}}
\left(\mathrm{I}_{N_\lambda-N_\alpha}\otimes\mathrm{I}_{N_t}+[\mathrm{K}'_{\lambda;2\mathrm{D}}]_{\mathrm{A}}\otimes[\mathrm{K}'_{t;2\mathrm{D}}]_{\mathrm{A}}\right)^{-1}
[\mathrm{V}_{\mathrm{B}}\otimes\mathrm{I}_{N_t-N_\beta}] \tag*{(176)} A B T A A − 1 B A = ( [ K λ ; 2 D ′ ] A ⊗ [ K t ; 2 D ′ ] B ) T ( I N λ − N α ⊗ I N t + [ K λ ; 2 D ′ ] A ⊗ [ K t ; 2 D ′ ] A ) − 1 [ V B ⊗ I N t − N β ] (176)
= [ Q [ K λ ; 2 D ′ ] A Λ [ K λ ; 2 D ′ ] A ⊗ [ K t ; 2 D ′ ] B T Q [ K t ; 2 D ′ ] A Λ A A − 1 [ V B ⊗ I N t − N β ] [ I N α ⊗ Q [ K t ; 2 D ′ ] A T ] (177) =[\mathrm{Q}_{[\mathrm{K}'_{\lambda;2\mathrm{D}}]_{\mathrm{A}}}\boldsymbol{\Lambda}_{[\mathrm{K}'_{\lambda;2\mathrm{D}}]_{\mathrm{A}}}\otimes[\mathrm{K}'_{t;2\mathrm{D}}]_{\mathrm{B}}^{\mathsf{T}}\mathrm{Q}_{[\mathrm{K}'_{t;2\mathrm{D}}]_{\mathrm{A}}}\boldsymbol{\Lambda}_{\mathrm{A}_{\mathrm{A}}}^{-1}[\mathrm{V}_{\mathrm{B}}\otimes\mathrm{I}_{N_t-N_\beta}][\mathrm{I}_{N_\alpha}\otimes\mathrm{Q}_{[\mathrm{K}'_{t;2\mathrm{D}}]_{\mathrm{A}}}^{\mathsf{T}}] \tag*{(177)} = [ Q [ K λ ; 2 D ′ ] A Λ [ K λ ; 2 D ′ ] A ⊗ [ K t ; 2 D ′ ] B T Q [ K t ; 2 D ′ ] A Λ A A − 1 [ V B ⊗ I N t − N β ] [ I N α ⊗ Q [ K t ; 2 D ′ ] A T ] (177)
= [ Q [ K λ ; 2 D ′ ] A Λ [ K λ ; 2 D ′ ] A ⊗ [ K t ; 2 D ′ ] B T Q [ K t ; 2 D ′ ] A ] [ ∑ i = 1 N t − N β A A ; i − 1 V B T ⊗ e ^ i e ^ i T ] [ I N α ⊗ Q [ K t ; 2 D ′ ] A T ] (178) =[\mathrm{Q}_{[\mathrm{K}'_{\lambda;2\mathrm{D}}]_{\mathrm{A}}}\boldsymbol{\Lambda}_{[\mathrm{K}'_{\lambda;2\mathrm{D}}]_{\mathrm{A}}}\otimes[\mathrm{K}'_{t;2\mathrm{D}}]_{\mathrm{B}}^{\mathsf{T}}\mathrm{Q}_{[\mathrm{K}'_{t;2\mathrm{D}}]_{\mathrm{A}}}]
\left[
\sum_{i=1}^{N_t-N_\beta}\mathrm{A}_{\mathrm{A};i}^{-1}\mathrm{V}_{\mathrm{B}}^{\mathsf{T}}\otimes\hat{\mathbf{e}}_i\hat{\mathbf{e}}_i^{\mathsf{T}}
\right]
[\mathrm{I}_{N_\alpha}\otimes\mathrm{Q}_{[\mathrm{K}'_{t;2\mathrm{D}}]_{\mathrm{A}}}^{\mathsf{T}}] \tag*{(178)} = [ Q [ K λ ; 2 D ′ ] A Λ [ K λ ; 2 D ′ ] A ⊗ [ K t ; 2 D ′ ] B T Q [ K t ; 2 D ′ ] A ] i = 1 ∑ N t − N β A A ; i − 1 V B T ⊗ e ^ i e ^ i T [ I N α ⊗ Q [ K t ; 2 D ′ ] A T ] (178)
= [ ∑ i = 1 N t − N β Q [ K λ ; 2 D ′ ] A Λ [ K λ ; 2 D ′ ] A A A ; i − 1 V B T ⊗ [ K t ; 2 D ′ ] B T Q [ K t ; 2 D ′ ] A e ^ i e ^ i T ] [ I N α ⊗ Q [ K t ; 2 D ′ ] A T ] (179) =
\left[
\sum_{i=1}^{N_t-N_\beta}\mathrm{Q}_{[\mathrm{K}'_{\lambda;2\mathrm{D}}]_{\mathrm{A}}}\boldsymbol{\Lambda}_{[\mathrm{K}'_{\lambda;2\mathrm{D}}]_{\mathrm{A}}}\mathrm{A}_{\mathrm{A};i}^{-1}\mathrm{V}_{\mathrm{B}}^{\mathsf{T}}\otimes[\mathrm{K}'_{t;2\mathrm{D}}]_{\mathrm{B}}^{\mathsf{T}}\mathrm{Q}_{[\mathrm{K}'_{t;2\mathrm{D}}]_{\mathrm{A}}}\hat{\mathbf{e}}_i\hat{\mathbf{e}}_i^{\mathsf{T}}
\right]
[\mathrm{I}_{N_\alpha}\otimes\mathrm{Q}_{[\mathrm{K}'_{t;2\mathrm{D}}]_{\mathrm{A}}}^{\mathsf{T}}] \tag*{(179)} = i = 1 ∑ N t − N β Q [ K λ ; 2 D ′ ] A Λ [ K λ ; 2 D ′ ] A A A ; i − 1 V B T ⊗ [ K t ; 2 D ′ ] B T Q [ K t ; 2 D ′ ] A e ^ i e ^ i T [ I N α ⊗ Q [ K t ; 2 D ′ ] A T ] (179)
where M 2 \mathrm{M}_2 M 2 is ( ( N λ − N α ) N β , N α ( N t − N β ) ) ((N_\lambda-N_\alpha)N_\beta,N_\alpha(N_t-N_\beta)) (( N λ − N α ) N β , N α ( N t − N β )) , which contributes a memory scaling of O ( N α N β N λ N t ) \mathcal{O}(N_\alpha N_\beta N_\lambda N_t) O ( N α N β N λ N t ) .
Combining Terms to Calculate C A − 1 B \mathrm{CA}^{-1}\mathrm{B} CA − 1 B
For consistency, we will label:
M 3 = B D ; 1 = [ K λ ; 2 D ′ ] B ⊗ I N β (180) \mathrm{M}_3=\mathrm{B}_{\mathrm{D};1}=[\mathrm{K}'_{\lambda;2\mathrm{D}}]_{\mathrm{B}}\otimes\mathrm{I}_{N_\beta} \tag*{(180)} M 3 = B D ; 1 = [ K λ ; 2 D ′ ] B ⊗ I N β (180)
M 4 = B D ; 2 = ∑ j = 1 N β [ K λ ; j ′ ] B ⊗ b j b j T (181) \mathrm{M}_4=\mathrm{B}_{\mathrm{D};2}=\sum_{j=1}^{N_\beta}[\mathrm{K}'_{\lambda;j}]_{\mathrm{B}}\otimes\mathbf{b}_j\mathbf{b}_j^{\mathsf{T}} \tag*{(181)} M 4 = B D ; 2 = j = 1 ∑ N β [ K λ ; j ′ ] B ⊗ b j b j T (181)
where M 3 \mathrm{M}_3 M 3 and M 4 \mathrm{M}_4 M 4 are both ( N λ − N α ) N β , N α N β (N_\lambda-N_\alpha)N_\beta,N_\alpha N_\beta ( N λ − N α ) N β , N α N β matrices and can be directly calculated in O ( N α N β 2 N λ ) \mathcal{O}(N_\alpha N_\beta^2N_\lambda) O ( N α N β 2 N λ ) runtime and memory.
We will also define
B Q = [ Q [ K t ; 2 D ′ ] A 0 0 I N β ] (182) \mathrm{B}_{\mathrm{Q}}=
\begin{bmatrix}
\mathrm{Q}_{[\mathrm{K}'_{t;2\mathrm{D}}]_{\mathrm{A}}}&0\\
0&\mathrm{I}_{N_\beta}
\end{bmatrix} \tag*{(182)} B Q = [ Q [ K t ; 2 D ′ ] A 0 0 I N β ] (182)
which we use to deal with the extra factor which was factored out of our expressions for M 1 \mathrm{M}_1 M 1 and M 2 \mathrm{M}_2 M 2 in Equations (175) and (179) , but not for M 3 \mathrm{M}_3 M 3 or M 4 \mathrm{M}_4 M 4 . We can now rewrite entirely in terms of precomputed terms:
C A − 1 B = [ I N α ⊗ K r ; 2 D T B Q ] [ M 1 + M 2 T [ A − 1 ] D M 2 − M 2 T [ A − 1 ] D M 3 − M 3 T [ A − 1 ] D M 2 + M 3 T [ A − 1 ] D M 3 ] [ I N α ⊗ B Q T K r ; 2 D T ] + [ I N α ⊗ K r ; 2 D T B Q ] [ 0 − M 2 T [ A − 1 ] D M 4 0 + M 3 T [ A − 1 ] D M 4 ] + [ 0 0 − M 4 T [ A − 1 ] D M 2 M 4 T [ A − 1 ] D M 3 ] [ I N α ⊗ B Q T K r ; 2 D T ] + [ 0 0 0 M 4 T [ A − 1 ] D M 4 ] (183) \begin{aligned}
\mathbf{C}\mathbf{A}^{-1}\mathbf{B} =
[\mathbf{I}_{N_\alpha}\otimes\mathbf{K}^{\mathsf{T}}_{r;2D}\mathbf{B}_{\mathcal{Q}}]
\begin{bmatrix}
\mathbf{M}_{1}+\mathbf{M}_{2}^{\mathsf{T}}[\mathbf{A}^{-1}]_{\mathsf{D}}\mathbf{M}_{2}-\mathbf{M}_{2}^{\mathsf{T}}[\mathbf{A}^{-1}]_{\mathsf{D}}\mathbf{M}_{3} &
-\mathbf{M}_{3}^{\mathsf{T}}[\mathbf{A}^{-1}]_{\mathsf{D}}\mathbf{M}_{2}+\mathbf{M}_{3}^{\mathsf{T}}[\mathbf{A}^{-1}]_{\mathsf{D}}\mathbf{M}_{3}
\end{bmatrix}
[\mathbf{I}_{N_\alpha}\otimes\mathbf{B}_{\mathcal{Q}}^{\mathsf{T}}\mathbf{K}_{r;2D}^{\mathsf{T}}] \\
&+[\mathbf{I}_{N_\alpha}\otimes\mathbf{K}_{r;2D}^{\mathsf{T}}\mathbf{B}_{\mathcal{Q}}]
\begin{bmatrix}
\mathbf{0}-\mathbf{M}_{2}^{\mathsf{T}}[\mathbf{A}^{-1}]_{\mathsf{D}}\mathbf{M}_{4}\\
\mathbf{0}+\mathbf{M}_{3}^{\mathsf{T}}[\mathbf{A}^{-1}]_{\mathsf{D}}\mathbf{M}_{4}
\end{bmatrix} \\
&+ \begin{bmatrix}
\mathbf{0} & \mathbf{0}\\
-\mathbf{M}_{4}^{\mathsf{T}}[\mathbf{A}^{-1}]_{\mathsf{D}}\mathbf{M}_{2} &
\mathbf{M}_{4}^{\mathsf{T}}[\mathbf{A}^{-1}]_{\mathsf{D}}\mathbf{M}_{3}
\end{bmatrix}
[\mathbf{I}_{N_\alpha}\otimes\mathbf{B}_{\mathcal{Q}}^{\mathsf{T}}\mathbf{K}_{r;2D}^{\mathsf{T}}] \\
&+\begin{bmatrix}
\mathbf{0} & \mathbf{0}\\
\mathbf{0} & \mathbf{M}_{4}^{\mathsf{T}}[\mathbf{A}^{-1}]_{\mathsf{D}}\mathbf{M}_{4}
\end{bmatrix} \tag*{(183)}
\end{aligned} C A − 1 B = [ I N α ⊗ K r ; 2 D T B Q ] [ M 1 + M 2 T [ A − 1 ] D M 2 − M 2 T [ A − 1 ] D M 3 − M 3 T [ A − 1 ] D M 2 + M 3 T [ A − 1 ] D M 3 ] [ I N α ⊗ B Q T K r ; 2 D T ] + [ I N α ⊗ K r ; 2 D T B Q ] [ 0 − M 2 T [ A − 1 ] D M 4 0 + M 3 T [ A − 1 ] D M 4 ] + [ 0 − M 4 T [ A − 1 ] D M 2 0 M 4 T [ A − 1 ] D M 3 ] [ I N α ⊗ B Q T K r ; 2 D T ] + [ 0 0 0 M 4 T [ A − 1 ] D M 4 ] (183)
In practice, we can first compute M i ′ = [ A − 1 2 ] D M i \mathbf{M}'_{i}=[\mathbf{A}^{-\frac{1}{2}}]_{\mathsf{D}}\mathbf{M}_{i} M i ′ = [ A − 2 1 ] D M i for i = 2 , 3 , 4 i=2,3,4 i = 2 , 3 , 4 , and then cheaply compute each term M i T [ A − 1 ] D M j = M i ′ T M j ′ \mathbf{M}_{i}^{\mathsf{T}}[\mathbf{A}^{-1}]_{\mathsf{D}}\mathbf{M}_{j}=\mathbf{M}_{i}^{\prime\mathsf{T}}\mathbf{M}'_{j} M i T [ A − 1 ] D M j = M i ′ T M j ′ afterwards. The cost of the matrix inverse square root calculations are O ( N α N β 2 N λ 2 N t ) \mathcal{O}(N_{\alpha}N_{\beta}^{2}N_{\lambda}^{2}N_{t}) O ( N α N β 2 N λ 2 N t ) for M 2 ′ \mathbf{M}'_{2} M 2 ′ , which dominates over the O ( N α N β 3 N λ 2 ) \mathcal{O}(N_{\alpha}N_{\beta}^{3}N_{\lambda}^{2}) O ( N α N β 3 N λ 2 ) scaling for M 3 ′ \mathbf{M}'_{3} M 3 ′ and M 4 ′ \mathbf{M}'_{4} M 4 ′ .
Total Computational Cost
None of the terms in the calculation of C A − 1 B \mathbf{C}\mathbf{A}^{-1}\mathbf{B} C A − 1 B adds an additional runtime scaling term which dominates except for the O ( N α N β 2 N λ 2 N t ) \mathcal{O}(N_{\alpha}N_{\beta}^{2}N_{\lambda}^{2}N_{t}) O ( N α N β 2 N λ 2 N t ) calculation of M 2 ′ \mathbf{M}'_{2} M 2 ′ . Adding this to the cost to decompose A \mathbf{A} A and D − C A − 1 B \mathbf{D}-\mathbf{C}\mathbf{A}^{-1}\mathbf{B} D − C A − 1 B yields a O ( ( 1 + N α 3 ) N t 3 + ( 1 + N β 3 ) N λ 3 + N α N β 2 N λ 2 N t + N λ N t 2 ) \mathcal{O}((1+N_{\alpha}^{3})N_{t}^{3}+(1+N_{\beta}^{3})N_{\lambda}^{3}+N_{\alpha}N_{\beta}^{2}N_{\lambda}^{2}N_{t}+N_{\lambda}N_{t}^{2}) O (( 1 + N α 3 ) N t 3 + ( 1 + N β 3 ) N λ 3 + N α N β 2 N λ 2 N t + N λ N t 2 ) runtime scaling. Our total memory scaling is dominated by building [ K ~ ] D [\widetilde{\mathbf{K}}]_{\mathsf{D}} [ K ] D , A D \mathbf{A}_{\mathsf{D}} A D , and M 2 \mathbf{M}_{2} M 2 in memory, working out to O ( N α 2 N t 2 + N β 2 N λ 2 + N α N β N λ N t ) \mathcal{O}(N_{\alpha}^{2}N_{t}^{2}+N_{\beta}^{2}N_{\lambda}^{2}+N_{\alpha}N_{\beta}N_{\lambda}N_{t}) O ( N α 2 N t 2 + N β 2 N λ 2 + N α N β N λ N t ) in total.
GP Predictive Mean and Covariance Calculation
To calculate the distribution of models the GP is fitting to the data, we need the GP predictive mean and predictive covariance
μ p r e d = μ ∗ + K ∗ K − 1 ( y − μ ) (184) \boldsymbol{\mu}_{\mathrm{pred}}=\boldsymbol{\mu}_{*}+\mathbf{K}_{*}\mathbf{K}^{-1}(\mathbf{y}-\boldsymbol{\mu}) \tag*{(184)} μ pred = μ ∗ + K ∗ K − 1 ( y − μ ) (184)
K p r e d = K ∗ ∗ − K ∗ T K − 1 K ∗ . (185) \mathbf{K}_{\mathrm{pred}}=\mathbf{K}_{**}-\mathbf{K}_{*}^{\mathsf{T}}\mathbf{K}^{-1}\mathbf{K}_{*}. \tag*{(185)} K pred = K ∗∗ − K ∗ T K − 1 K ∗ . (185)
The GP mean is already straight-forward to calculate given that we can calculate the log-likelihood for each optimisation, as we can already calculate K − 1 r = ( K − 1 2 ) T K − 1 2 r \mathbf{K}^{-1}\mathbf{r}=(\mathbf{K}^{-\frac{1}{2}})^{\mathsf{T}}\mathbf{K}^{-\frac{1}{2}}\mathbf{r} K − 1 r = ( K − 2 1 ) T K − 2 1 r efficiently. Multiplication by K ∗ \mathbf{K}_{*} K ∗ is simple if the predicted points also lie on a grid, in which case it equals a sum of Kronecker products, for which matrix-vector multiplications can be performed efficiently (Saatchi [102 ] ).
The GP predictive covariance is more challenging to calculate, there’s no obvious factorisation which allows us to efficiently decompose K p r e d \mathbf{K}_{\mathrm{pred}} K pred directly for all optimisations. However, we can exploit our existing optimisations provided that not only do the observed inputs lie on a grid, but also the union of the observed and predicted inputs lie on a grid. For example, we could predict values at unseen times or wavelengths, or even predict at both unseen times and unseen wavelengths, but the region we train on must always be a grid, and the combination of points we train and predict on must be a grid, as visualised in Figure D1 . We can also handle the case of both the training points and the predicted points being on a grid, useful for upsampling or downsampling a dataset, by utilising a trick shown in Figure D2 .
Figure D1. GP regression and prediction requires the training points y \mathbf{y} y to lie on a complete input grid, which is satisfied in the left and middle plots, but not the rightmost plot. The prediction locations y ∗ \mathbf{y}_* y ∗ are not required to lie on a grid, but we do require the combined training and prediction locations to lie on a grid, which all plots satisfy.
Figure D2. We can always train on a grid of locations and predict values on a different input grid, even though the union of these points doesn’t form a grid directly. We do this by predicting at additional locations, so that the union of trained and predicted points forms a complete grid. We can then simply discard these additional predicted values.
For this method, we first concatenate our observed and predicted input locations together as X f u l l = ( [ λ , λ ∗ ] , [ t , t ∗ ] ) \mathbf{X}_{\mathrm{full}}=([\boldsymbol{\lambda},\boldsymbol{\lambda}_{*}],[\mathbf{t},\mathbf{t}_{*}]) X full = ([ λ , λ ∗ ] , [ t , t ∗ ]) and construct the covariance matrix using any of the previously mentioned kernel functions, which will have the block structure
K f u l l = [ K K ∗ K ∗ T K ∗ ∗ ] . (186) \mathbf{K}_{\mathrm{full}}=
\begin{bmatrix}
\mathbf{K} & \mathbf{K}_{*}\\
\mathbf{K}_{*}^{\mathsf{T}} & \mathbf{K}_{**}
\end{bmatrix}. \tag*{(186)} K full = [ K K ∗ T K ∗ K ∗∗ ] . (186)
The covariance matrix for the trained points K \mathbf{K} K , and the covariance matrix on the union of trained and predicted points K f u l l \mathbf{K}_{\mathrm{full}} K full , both lie on complete grids and therefore calculations with them can be optimised using any of the GP optimisations in this work. This means that we can perform cross-validation and sample from the GP predictive distribution efficiently, provided that we can perform these calculations using decompositions of the matrices K \mathbf{K} K and K f u l l \mathbf{K}_{\mathrm{full}} K full .
Note that the inverse of K f u l l \mathbf{K}_{\mathrm{full}} K full is given by the formula for the inverse of a 2 × 2 2\times2 2 × 2 block matrix is
K f u l l − 1 = [ K − 1 + K − 1 K ∗ K p r e d − 1 K ∗ T K − 1 − K − 1 K ∗ K p r e d − 1 − K p r e d − 1 K ∗ T K − 1 K p r e d − 1 ] , (187) \mathbf{K}_{\mathrm{full}}^{-1}=
\begin{bmatrix}
\mathbf{K}^{-1}+\mathbf{K}^{-1}\mathbf{K}_{*}\mathbf{K}_{\mathrm{pred}}^{-1}\mathbf{K}_{*}^{\mathsf{T}}\mathbf{K}^{-1} &
-\mathbf{K}^{-1}\mathbf{K}_{*}\mathbf{K}_{\mathrm{pred}}^{-1}\\
-\mathbf{K}_{\mathrm{pred}}^{-1}\mathbf{K}_{*}^{\mathsf{T}}\mathbf{K}^{-1} &
\mathbf{K}_{\mathrm{pred}}^{-1}
\end{bmatrix}, \tag*{(187)} K full − 1 = [ K − 1 + K − 1 K ∗ K pred − 1 K ∗ T K − 1 − K pred − 1 K ∗ T K − 1 − K − 1 K ∗ K pred − 1 K pred − 1 ] , (187)
where the D \mathbf{D} D block of K f u l l − 1 \mathbf{K}_{\mathrm{full}}^{-1} K full − 1 matches the inverse of K p r e d \mathbf{K}_{\mathrm{pred}} K pred as defined in Equation (185) . This allows us to multiply vectors by K p r e d − 1 \mathbf{K}_{\mathrm{pred}}^{-1} K pred − 1 by decomposing K f u l l \mathbf{K}_{\mathrm{full}} K full .
Cross-Validation Calculation
Suppose we want to perform cross-validation using our GP model, where we favour GP predictive models which maximise the probability of drawing the unseen data, i.e. maximise the log-probability:
log p ( y ∗ ∣ μ p r e d , K p r e d ) = − 1 2 r ∗ T K p r e d − 1 r ∗ − 1 2 log ∣ K p r e d ∣ − N ∗ 2 log 2 π (188) \log p(\mathbf{y}_{*}\mid\boldsymbol{\mu}_{\mathrm{pred}},\mathbf{K}_{\mathrm{pred}})=-\frac{1}{2}\mathbf{r}_{*}^{\mathsf{T}}\mathbf{K}_{\mathrm{pred}}^{-1}\mathbf{r}_{*}-\frac{1}{2}\log\lvert\mathbf{K}_{\mathrm{pred}}\rvert-\frac{N_{*}}{2}\log2\pi \tag*{(188)} log p ( y ∗ ∣ μ pred , K pred ) = − 2 1 r ∗ T K pred − 1 r ∗ − 2 1 log ∣ K pred ∣ − 2 N ∗ log 2 π (188)
where the residuals in our predictions are calculated as r ∗ = y ∗ − μ p r e d \mathbf{r}_{*}=\mathbf{y}_{*}-\boldsymbol{\mu}_{\mathrm{pred}} r ∗ = y ∗ − μ pred , where we use our GP predictive mean from Equation (184) to calculate predictions on unseen points using the training data y \mathbf{y} y .
When zero-pad the unseen residuals and multiply by the inverse of K f u l l \mathbf{K}_{\mathrm{full}} K full in Equation (187)
\begin{bmatrix}\mathbf{0} & \mathbf{r}_{*}^{\mathsf{T}}\end{bmatrix}
\mathbf{K}_{\mathrm{full}}^{-1}
\begin{bmatrix}\mathbf{0}\\\mathbf{r}_{*}\end{bmatrix}
=\mathbf{r}_{*}^{\mathsf{T}}\mathbf{K}_{\mathrm{pred}}^{-1}\mathbf{r}_{*}, \tag*{(189)}\end{bmatrix}
We can express this zero-padding of the unseen residuals as a matrix multiplication using the non-square ( N + N ∗ , N ∗ ) (N+N_{*},N_{*}) ( N + N ∗ , N ∗ ) matrix
P ∗ = [ 0 I N ∗ ] (190) \mathbf{P}_{*}=
\begin{bmatrix}
\mathbf{0}\\
\mathbf{I}_{N_{*}}
\end{bmatrix} \tag*{(190)} P ∗ = [ 0 I N ∗ ] (190)
which allows us to define a (non-square) matrix inverse square root of K p r e d \mathbf{K}_{\mathrm{pred}} K pred as K p r e d − 1 2 = K f u l l − 1 2 P ∗ \mathbf{K}_{\mathrm{pred}}^{-\frac{1}{2}}=\mathbf{K}_{\mathrm{full}}^{-\frac{1}{2}}\mathbf{P}_{*} K pred − 2 1 = K full − 2 1 P ∗ , which satisfies K p r e d − 1 = ( K p r e d − 1 2 ) T K p r e d − 1 2 \mathbf{K}_{\mathrm{pred}}^{-1}=(\mathbf{K}_{\mathrm{pred}}^{-\frac{1}{2}})^{\mathsf{T}}\mathbf{K}_{\mathrm{pred}}^{-\frac{1}{2}} K pred − 1 = ( K pred − 2 1 ) T K pred − 2 1 . For the determinant of the GP predictive covariance ∣ K p r e d ∣ \lvert\mathbf{K}_{\mathrm{pred}}\rvert ∣ K pred ∣ , smaller values of which correspond to more confident predictions on the unseen data, we note our formula for the determinant of a block matrix from Equation (148) , which for K f u l l \mathbf{K}_{\mathrm{full}} K full yields:
log ∣ K f u l l ∣ = log ∣ K ∣ + log ∣ K ∗ ∗ − K ∗ T K − 1 K ∗ ∣ (191) \log\lvert\mathbf{K}_{\mathrm{full}}\rvert=\log\lvert\mathbf{K}\rvert+\log\lvert\mathbf{K}_{**}-\mathbf{K}_{*}^{\mathsf{T}}\mathbf{K}^{-1}\mathbf{K}_{*}\rvert \tag*{(191)} log ∣ K full ∣ = log ∣ K ∣ + log ∣ K ∗∗ − K ∗ T K − 1 K ∗ ∣ (191)
log ∣ K f u l l ∣ = log ∣ K ∣ + log ∣ K p r e d ∣ (192) \log\lvert\mathbf{K}_{\mathrm{full}}\rvert=\log\lvert\mathbf{K}\rvert+\log\lvert\mathbf{K}_{\mathrm{pred}}\rvert \tag*{(192)} log ∣ K full ∣ = log ∣ K ∣ + log ∣ K pred ∣ (192)
So using any of the GP optimisations presented, we can calculate the log determinant log ∣ K ∣ \log\lvert\mathbf{K}\rvert log ∣ K ∣ using the trained inputs X \mathbf{X} X , calculate the log determinant ∣ K f u l l ∣ \lvert\mathbf{K}_{\mathrm{full}}\rvert ∣ K full ∣ by using all inputs X f u l l \mathbf{X}_{\mathrm{full}} X full , and then the difference gives the GP predictive covariance log determinant log ∣ K p r e d ∣ \log\lvert\mathbf{K}_{\mathrm{pred}}\rvert log ∣ K pred ∣ .
Our cross-validation likelihood can thus be written as:
log p ( y ∗ ∣ μ p r e d , K p r e d ) = − 1 2 [ 0 r ∗ T ] K f u l l − 1 [ 0 r ∗ ] − 1 2 log ∣ K f u l l ∣ − N ∗ 2 log 2 π + 1 2 log ∣ K ∣ (193) \log p(\mathbf{y}_{*}\mid\boldsymbol{\mu}_{\mathrm{pred}},\mathbf{K}_{\mathrm{pred}})=-\frac{1}{2}\begin{bmatrix}\mathbf{0}&\mathbf{r}_{*}^{\mathsf{T}}\end{bmatrix}\mathbf{K}_{\mathrm{full}}^{-1}\begin{bmatrix}\mathbf{0}\\\mathbf{r}_{*}\end{bmatrix}-\frac{1}{2}\log\lvert\mathbf{K}_{\mathrm{full}}\rvert-\frac{N_{*}}{2}\log2\pi+\frac{1}{2}\log\lvert\mathbf{K}\rvert \tag*{(193)} log p ( y ∗ ∣ μ pred , K pred ) = − 2 1 [ 0 r ∗ T ] K full − 1 [ 0 r ∗ ] − 2 1 log ∣ K full ∣ − 2 N ∗ log 2 π + 2 1 log ∣ K ∣ (193)
Sampling from the Predictive Covariance
If we want to take samples from K p r e d \mathbf{K}_{\mathrm{pred}} K pred , we can use a corresponding non-square matrix square root for K p r e d \mathbf{K}_{\mathrm{pred}} K pred , equivalent to a sampling method known as Matheron’s rule (e.g. Wilson et al. [131 ] ). The matrix square root can be expressed as:
K p r e d 1 2 = [ − K ∗ T K − 1 I N ∗ ] K f u l l 1 2 (194) \mathbf{K}_{\mathrm{pred}}^{\frac{1}{2}}=
\begin{bmatrix}
-\mathbf{K}_{*}^{\mathsf{T}}\mathbf{K}^{-1} & \mathbf{I}_{N_{*}}
\end{bmatrix}
\mathbf{K}_{\mathrm{full}}^{\frac{1}{2}} \tag*{(194)} K pred 2 1 = [ − K ∗ T K − 1 I N ∗ ] K full 2 1 (194)
which we can validate as a matrix square root by multiplying it out:
K p r e d 1 2 ( K p r e d 1 2 ) T = [ K K ∗ − K ∗ T K − 1 I N ∗ ] [ K − 1 K ∗ K T K − 1 K ∗ ∗ ] (195) \mathbf{K}_{\mathrm{pred}}^{\frac{1}{2}}\left(\mathbf{K}_{\mathrm{pred}}^{\frac{1}{2}}\right)^{T}
=
\begin{bmatrix}
\mathbf{K} & \mathbf{K}_* \\
-\mathbf{K}_*^{T}\mathbf{K}^{-1} & \mathbf{I}_{N_*}
\end{bmatrix}
\begin{bmatrix}
\mathbf{K}^{-1}\mathbf{K}_* \\
\mathbf{K}^{T}\mathbf{K}^{-1} & \mathbf{K}_{**}
\end{bmatrix} \tag*{(195)} K pred 2 1 ( K pred 2 1 ) T = [ K − K ∗ T K − 1 K ∗ I N ∗ ] [ K − 1 K ∗ K T K − 1 K ∗∗ ] (195)
= K ∗ ∗ − K ∗ T K − 1 K ∗ (196) = \mathbf{K}_{**}-\mathbf{K}_*^{T}\mathbf{K}^{-1}\mathbf{K}_* \tag*{(196)} = K ∗∗ − K ∗ T K − 1 K ∗ (196)
Therefore, we can draw samples from K p r e d \mathbf{K}_{\mathrm{pred}} K pred by taking a random normal vector matching the shape of the combined training and prediction locations ( N λ + N λ , ∗ , N t + N t , ∗ ) (N_{\lambda}+N_{\lambda,*},N_t+N_{t,*}) ( N λ + N λ , ∗ , N t + N t , ∗ ) , and multiplying by K p r e d 1 2 \mathbf{K}_{\mathrm{pred}}^{\frac{1}{2}} K pred 2 1 in Equation (194) . This procedure still produces valid draws from K p r e d \mathbf{K}_{\mathrm{pred}} K pred despite the non-square matrix square root, as can be verified using the definition of the sample covariance:
E [ ( r − r ˉ ) ( r − r ˉ ) T ] = K p r e d 1 2 E [ ( z − z ˉ ) ( z − z ˉ ) T ] ( K p r e d 1 2 ) T (197) \mathbb{E}\left[(\mathbf{r}-\bar{\mathbf{r}})(\mathbf{r}-\bar{\mathbf{r}})^{T}\right]
=
\mathbf{K}_{\mathrm{pred}}^{\frac{1}{2}}\mathbb{E}\left[(\mathbf{z}-\bar{\mathbf{z}})(\mathbf{z}-\bar{\mathbf{z}})^{T}\right]\left(\mathbf{K}_{\mathrm{pred}}^{\frac{1}{2}}\right)^{T} \tag*{(197)} E [ ( r − r ˉ ) ( r − r ˉ ) T ] = K pred 2 1 E [ ( z − z ˉ ) ( z − z ˉ ) T ] ( K pred 2 1 ) T (197)
= K p r e d 1 2 I N + N ∗ ( K p r e d 1 2 ) T (198) =
\mathbf{K}_{\mathrm{pred}}^{\frac{1}{2}}\mathbf{I}_{N+N_*}\left(\mathbf{K}_{\mathrm{pred}}^{\frac{1}{2}}\right)^{T} \tag*{(198)} = K pred 2 1 I N + N ∗ ( K pred 2 1 ) T (198)
= K p r e d (199) = \mathbf{K}_{\mathrm{pred}} \tag*{(199)} = K pred (199)
Extending to Higher Dimensions
We can implement the SEPARABLEKERNEL, LUASKERNEL, and LUASLASRACHKERNEL for arbitrary numbers of dimensions d d d with minimal modification. The SEPARABLEKERNEL simply follows the algorithms in Saatchi [102 ] , decomposing each of the d d d covariance matrices in the Kronecker product one-by-one. We simply extend this by allowing each matrix to be decomposed by any matrix square root, rather than specifically Cholesky decomposition. The data residuals are now a d d d -dim tensor, and can be transformed using algorithms in Saatchi [102 ] . The LUASKERNEL is extended similarly, we just perform the same simultaneous diagonalisation trick looping over all dimensions, applying eigendecompositions in each dimension. The transformation of the residuals then follows the same algorithm. For 3D datacubes, both methods yield a scaling of O ( N m 3 + N λ 3 + N t 3 + N m N λ N t ( N m + N λ + N t ) ) \mathcal{O}(N_m^3+N_{\lambda}^3+N_t^3+N_mN_{\lambda}N_t(N_m+N_{\lambda}+N_t)) O ( N m 3 + N λ 3 + N t 3 + N m N λ N t ( N m + N λ + N t )) for some third dimension of size N m N_m N m . See Figure G34 and Figure G36 for benchmarking examples on both CPU and GPU for the LUASKERNEL with N m = 64 N_m=64 N m = 64 .
Figure G36. The same 3D kernel function as Figure G34 but implemented on GPU. The runtime cost is comparable in runtime to the 2D case in Figure G11 despite having 64 × 64\times 64 × as many datapoints to fit in the additional dimension. However, for the largest datacubes benchmarked the large matrix multiplications required to transform the residuals dominate the calculations.
We can also extend the LUASLASRACHKERNEL to arbitrary dimensions with a modification to the LUASKERNEL approach. We perform simultaneous diagonalisation on all dimensions except for one of them, which still creates a block diagonal matrix, although potentially a very large one. For example, suppose we take a set of N m N_m N m spectroscopic time-series of the same shape (for example N m N_m N m different spectral orders), of shape ( N m , N λ , N t ) (N_m,N_{\lambda},N_t) ( N m , N λ , N t ) , and we have some 3D kernel function consisting of two Kronecker products:
K = Σ m ⊗ Σ λ ⊗ Σ t + K m ⊗ K λ ⊗ K t (200) \mathbf{K}=\boldsymbol{\Sigma}_m\otimes\boldsymbol{\Sigma}_{\lambda}\otimes\boldsymbol{\Sigma}_t+\mathbf{K}_m\otimes\mathbf{K}_{\lambda}\otimes\mathbf{K}_t \tag*{(200)} K = Σ m ⊗ Σ λ ⊗ Σ t + K m ⊗ K λ ⊗ K t (200)
We decompose the matrix using the transformation:
K = [ Σ m 1 2 Q K m ⊗ Σ λ 1 2 Q K λ ⊗ I N t ] [ I N m ⊗ I N λ ⊗ Σ t + Λ K m ⊗ Λ K λ ⊗ K t ] (201) \mathbf{K}=
\left[\boldsymbol{\Sigma}_m^{\frac{1}{2}}\mathbf{Q}_{K_m}\otimes\boldsymbol{\Sigma}_{\lambda}^{\frac{1}{2}}\mathbf{Q}_{K_{\lambda}}\otimes\mathbf{I}_{N_t}\right]
\left[\mathbf{I}_{N_m}\otimes\mathbf{I}_{N_{\lambda}}\otimes\boldsymbol{\Sigma}_t+\mathbf{\Lambda}_{K_m}\otimes\mathbf{\Lambda}_{K_{\lambda}}\otimes\mathbf{K}_t\right] \tag*{(201)} K = [ Σ m 2 1 Q K m ⊗ Σ λ 2 1 Q K λ ⊗ I N t ] [ I N m ⊗ I N λ ⊗ Σ t + Λ K m ⊗ Λ K λ ⊗ K t ] (201)
The block diagonal transformed covariance matrix K ~ \widetilde{\mathbf{K}} K is now similar to the result in Eq. (200) , but with the eigenvalues of all other dimensions stacked together:
K ~ = I N m N λ ⊗ Σ t + ( Λ K m ⊗ Λ K λ ) ⊗ K t (202) \widetilde{\mathbf{K}}=\mathbf{I}_{N_mN_{\lambda}}\otimes\boldsymbol{\Sigma}_t+\left(\boldsymbol{\Lambda}_{K_m}\otimes\boldsymbol{\Lambda}_{K_{\lambda}}\right)\otimes\mathbf{K}_t \tag*{(202)} K = I N m N λ ⊗ Σ t + ( Λ K m ⊗ Λ K λ ) ⊗ K t (202)
The number of dense blocks to decompose in K ~ \widetilde{\mathbf{K}} K will be equal to the product of the number of non-zero eigenvalues for each dimension we eigendecompose. Our example yields a scaling of O ( N m 3 + N λ 3 + N m N λ N t ( N m + N λ + ( J K + J Σ ) 2 ) ) \mathcal{O}(N_m^3+N_{\lambda}^3+N_mN_{\lambda}N_t(N_m+N_{\lambda}+(J_K+J_{\Sigma})^2)) O ( N m 3 + N λ 3 + N m N λ N t ( N m + N λ + ( J K + J Σ ) 2 )) when K \mathbf{K} K and Σ \boldsymbol{\Sigma} Σ are quasiseparable/diagonal. Since this scaling is very similar to the LUASKERNEL, but with a linear scaling with the number of time points, the LUASLASRACHKERNEL optimisation only scales more efficiently if the other dimensions N m N_m N m and N λ N_{\lambda} N λ are relatively small, or if the kernels in those other dimensions are low-rank. Figures G35 and G37 benchmark the LUASLASRACHKERNEL on 3D datacubes for N m = 64 N_m=64 N m = 64 on both CPU and GPU, which gives comparable performance to the LUASKERNEL for the same dataset shapes, but which does perform faster for higher numbers of time points.
Numerical Stability
The numerical stability of these algorithms was very high from our testing, although we still recommend double precision for any of these optimisations. We compared the calculations from each of our algorithms benchmarked in Appendix G to Cholesky decomposition of the full constructed covariance matrix for N λ = 16 N_\lambda= 16 N λ = 16 and N t = 128 N_t = 128 N t = 128 . We computed both the log-likelihood and the gradients of the log-likelihood with respect to the hyperparameters (assuming zero mean function). We kept the kernel height scales fixed to unity, and randomly draw white noise amplitudes in wavelength with unit mean and a standard deviation of 0.1. We found the relative difference in the log-likelihood values were < 10 × < 10\times < 10 × machine precision (ε d o u b l e ∼ 2.2 × 10 − 16 \varepsilon_{\mathrm{double}} \sim2.2 \times10^{-16} ε double ∼ 2.2 × 1 0 − 16 ) compared to building the full covariance matrix and decomposing using Cholesky decomposition. The gradients automatically calculated by JAX autodiff were also typically very accurate, with relative errors on the order of 10 − 12 10^{-12} 1 0 − 12 . The only case where we needed to specify a more stable way for JAX to perform the gradient calculation was when taking the gradient with respect to eigendecomposition, because this gradient calculation depends on a matrix of the form
F i j = 1 λ i − λ j , i ≠ j (203) F_{ij} = \frac{1}{\lambda_i-\lambda_j},\ i\ne j \tag*{(203)} F ij = λ i − λ j 1 , i = j (203)
which becomes unstable if any eigenvalues are close or approximately equal (Liao et al. [79 ] ). Such a situation occurs frequently, so we use a solution described in Liao et al. [79 ] and approximate the gradient calculation by approximating F F F as:
F i j ≈ λ i − λ j ( λ i − λ j ) 2 + ε , i ≠ j (204) F_{ij} \approx\frac{\lambda_i-\lambda_j}{(\lambda_i-\lambda_j)^2+\varepsilon},\ i\ne j \tag*{(204)} F ij ≈ ( λ i − λ j ) 2 + ε λ i − λ j , i = j (204)
where ε \varepsilon ε is a small constant, which we set to ε = 10 − 16 \varepsilon= 10^{-16} ε = 1 0 − 16 during our testing. Note that ε \varepsilon ε will be negligible except in the scenario when λ i − λ j ≈ 0 \lambda_i-\lambda_j \approx0 λ i − λ j ≈ 0 , where it is needed for stability. This approximation only slightly affects the gradient calculation, with the largest relative errors we recovered on the order of 10 − 8 10^{-8} 1 0 − 8 , while the log-likelihood value is completely unaffected and remains exact. Note that many applications of gradients, such as gradient-based optimisers, and MCMC samplers like Hamiltonian Monte Carlo and No U-Turn Sampling, do not require exact gradients in order for inference to remain exact (Hoffman and Gelman [63 ] ).
Benchmarking
In this section, we’ve provided log-likelihood runtimes of the LUAS implementations of these new GP optimisations at the time of writing. Calculations were performed on both an M4 MacBook Air (10 cores), as well as on an NVIDIA A100-SXM4-40GB GPU. All calculations were performed at double precision, as calculations may not be numerically stable at single precision. Therefore, the use of a GPU with a high rate of floating point operations per second at double precision (FP64 FLOPS) is crucial, as it varies dramatically between GPUs, and is often not supported at all. We calculate these runtimes for a range of 2D dataset shapes, reporting both the log-likelihood runtime, as well as the runtime per number of datapoints (which should be constant for fully linear scaling). Calculations were performed on the CPU using JAX version 0.10.1, SciPy version 1.17.1 and a recent commit of tinygp (Commit f3466f0; see pull request #269 for details). The GPU environment had JAX version 0.9.0 for CUDA 12, SciPy version 1.17.0 and the same commit of tinygp (which features GPU optimisation for quasiseparable kernels). We exclude the one-time compile cost of jit-compilation, typically on the order of 0.1 - 10 seconds on the MacBook, and use the method BLOCK_UNTIL_READY to avoid issues with asynchronous dispatch in JAX.
Runtimes were taken as the median over at least 10 runs. If 10 runs completed in less than 10 seconds, the number of runs was multiplied by 10 and run again, with this process terminating after greater than 10 seconds were exceeded, or at 10,000 runs. For the MacBook, background processes would occasionally cause significantly longer runtimes over durations of minutes, so a best of three separate runs was used to avoid these outliers. The GPU runtimes were significantly more consistent, so only a single run was performed for each.
These calculations were performed assuming zero mean, but for many applications there is a mean function we also calculate for each likelihood calculation, such as building a model of transit light curves. For many of these new GP optimisations, transit light curve modelling would be the bottleneck if running on CPU, so the runtimes for transit light curve generation using JAX-OPLANET (Hattori et al. [61 ] ) with quadratic limb darkening are included in Figure G2 for reference. We also tested the popular transit modelling package BATMAN (Kreidberg [76 ] ), which yielded similar runtimes, typically within a factor of two of these values. Eclipse light curve modelling can be performed significantly faster in either case, as spectroscopic eclipse light curves are typically assumed to share the same planetary parameters (such as radius ratio R p / R ∗ R_p/R_\ast R p / R ∗ ) without any limb darkening, which means that they all have the same shape but scaled by the planet-to-star flux ratio f p / f ∗ f_p/f_\ast f p / f ∗ . This means that we can simply generate a single transit light curve and scale it for each light curve, resulting in significant computational savings if the number of wavelengths N λ N_\lambda N λ is high, as shown in Figure G4 using JAXOPLANET. Since both problems are easily parallelised, these calculations can be performed dramatically faster on GPU for large datasets, as shown in Figure G3 and Figure G5 . The cost of a general covariance matrix decomposition using Cholesky factorisation is shown in Figure G6 and Figure G7 on CPU and GPU respectively, but quickly becomes intractable for larger datasets. These are compared to a wide range of new GP optimisations we introduce, on both CPU and GPU, which are arranged and linked to in Figure G1, which summarises each methods runtime scaling.
Figure G3. Compared to Figure G2 , transit modelling with JAXOPLANET can scale significantly better for larger datasets on the A100 GPU, at which point it outperforms any of the GP optimisations in this work, even when the GP calculations are also run on a GPU.
Figure G4. Eclipse light curve modelling can be performed significantly faster by generating a single transit light curve (also with JAXOPLANET), and scaling it by f p / f ⋆ f_{\mathrm{p}}/f_{\star} f p / f ⋆ for each light curve. The runtime cost approaches approximately 2 nanoseconds per datapoint for larger datasets on the MacBook, which has a similar runtime cost to some of the fastest GP optimisations implemented for this work.
Figure G5. Compared to Figure G4 , eclipse modelling with this optimised approach is trivial on the GPU and is dominated by a fixed bottleneck for these dataset sizes.
Figure G6. Decomposition of any general covariance matrix using Cholesky decomposition scales as O ( N λ 3 N t 3 ) O(N_{\lambda}^{3}N_{t}^{3}) O ( N λ 3 N t 3 ) in runtime, which becomes prohibitively expensive even for relatively small datasets, particularly on CPU.
Figure G7. Compared to Figure G6 , running on GPU allows us to scale to slightly larger datasets for any general covariance matrix on 2D data, but due to the expensive memory scaling of O ( N λ 2 N t 2 ) O(N_{\lambda}^{2}N_{t}^{2}) O ( N λ 2 N t 2 ) required to store the covariance matrix, we were unable to decompose matrices with a total size larger than ( 16384 , 16384 ) (16384, 16384) ( 16384 , 16384 ) .
Gradient Calculations
We use automatic differentiation in JAX for gradient calculations, where the relative increase in computational cost when including gradient calculations was typically found to be between ∼ 1 \sim1 ∼ 1 –10 × 10\times 10 × for all the log-likelihood runtimes. Specifically for the benchmarking calculations, if the log-likelihood runtime was < 1 <1 < 1 second for a given dataset shape between Figure G8 –Figure G32 , we also benchmarked calculating both the log-likelihood and its gradients for all hyperparameters. Averaging across all dataset shapes tested for each optimisation (excluding Figure G14 which implements the existing Gordon et al. 2020 method in tinygp), the mean relative cost of including gradients was 4.5 × 4.5\times 4.5 × , while the median relative cost was 3.6 × 3.6\times 3.6 × . Both the mean and median relative cost were always < 10 × <10\times < 10 × the log-likelihood cost, except for the MultiLowRankKernel with N α = 16 N_\alpha= 16 N α = 16 in Figure G20 , which averaged between 15–16× \times × more expensive for both mean and median averages. This is likely due to the large number of parameters to calculate the gradients with respect to (the vectors α λ , i \boldsymbol{\alpha}_{\lambda,i} α λ , i alone contribute N α N λ N_\alpha N_\lambda N α N λ free parameters). Interestingly, the MultiLowRankKernel with N α = 16 N_\alpha= 16 N α = 16 orthogonal components in Figure G21 performs significantly better, with a < 5 × <5\times < 5 × performance hit relative to its log-likelihood calculation. Regardless, no individual dataset shape tested ever cost > 20 × >20\times > 20 × the computational cost across all figures, even for cases where there are > 10 , 000 >10,000 > 10 , 000 hyperparameters to calculate gradients for.
Figure 1 (PDF p. 32). Summary of the runtime scaling for different GP optimisations introduced in this work. The particular kernel optimised is the sum of the kernel term of that row (e.g. arbitrary measurement covariance), plus some number of low-rank terms based on the column. The total runtime scaling is found by adding together all terms which have arrows pointing toward it, where the scaling for the DOUBLEMULTILOWRANKKERNEL assumes arbitrary components for the MULTILOWRANKKERNEL. Note that terms in the right-most column either have both low-rank time kernels and low-rank wavelength kernels (the DOUBLEMULTILOWRANKKERNEL and EXTENDEDLUASKERNEL), or just low-rank wavelength kernels added to it (the LUASPLUSMULTILOWRANKKERNEL), depending on how many arrows connect toward it. Different benchmarking plots are linked to for most optimisations.
Figure 14. For comparison, we benchmarked the Gordon et al. (2020) optimisation for a low-rank kernel term implemented within TINYGP. The right plot clearly shows the linear scaling of this algorithm O ( N λ N t J 2 ) \mathcal{O}(N_\lambda N_t J^2) O ( N λ N t J 2 ) , with a cost of about 30 ns/point for likelihood calculations on CPU. Unlike our optimisations, this model does not account for measurement covariance, however, it can account for white noise amplitudes which are not separable in each dimension (see § 2.5 ).
Figure 15. Compared to the Gordon et al. (2020) optimisation in Figure G14, our MULTILOWRANKKERNEL optimisation achieves significantly faster runtimes for a similar kernel function. While both algorithms achieve a linear runtime scaling, our algorithm only requires a couple floating point operations per datapoint, and therefore costs just 1-2 ns per datapoint for larger datasets. We also include quasi-separable measurement covariance with this kernel, flexibility not introduced in Gordon et al. (2020), but we cannot handle non-separable white noise amplitudes across the dataset, which the Gordon et al. (2020) method can handle.
Figure 16. We can also extend this kernel to include any time kernel, not just quasi-separable kernels. We also include measurement covariance, but assume it is quasi-separable. With a scaling of O ( N α N t 3 + N α N λ N t ) \mathcal{O}(N_\alpha N_t^3 + N_\alpha N_\lambda N_t) O ( N α N t 3 + N α N λ N t ) , the runtime cost is dominated by the Cholesky factorisation of a ( N α N t , N α N t ) (N_\alpha N_t, N_\alpha N_t) ( N α N t , N α N t ) matrix on CPU, which can be seen by comparing these values with Figure 1.
Figure 17. Compared to the CPU calculations in Figure G16, the GPU can more efficiently model an arbitrary low-rank kernel term even with a more general arbitrary measurement covariance. The runtimes are similarly dominated by the ( N α N t , N α N t ) (N_\alpha N_t, N_\alpha N_t) ( N α N t , N α N t ) matrix Cholesky decomposition, but this is much faster on GPU (Figure 1).
Figure G35. The same as Figure G34 but using the LUASLASRACHKERNEL with an exponential kernel in time. The O ( N m 3 + N λ 3 + N m N λ N t ( N m + N λ + ( J K t + J Σ t ) 2 ) ) O(N_m^3 + N_\lambda^3 + N_mN_\lambda N_t(N_m + N_\lambda+ (J_{K_t} + J_{\Sigma_t})^2)) O ( N m 3 + N λ 3 + N m N λ N t ( N m + N λ + ( J K t + J Σ t ) 2 )) runtime scaling is linear in the number of time points, but this runtime advantage is limited to quasiseparable kernels in the longest dimension, while the LUASKERNEL can use any time kernel.