WorksMathematics
Empirical Variational Bayes Factorization for Longitudinal Matrix Data
Abstract
Collecting high-dimensional data over time for multiple individuals is increasingly common in biomedical research, particularly in studies of disease progression or molecular time-course experiments. Often data are missing or the timepoints are irregularly spaced across individuals. There is a lack of flexible methodology to explore latent patterns of change over time and account for missing data. We propose an empirical variational Bayes (EVB) approach to factorize high-dimensional longitudinal data, in which the underlying factors have temporal correlation over time. The EVB approach is computationally efficient and accommodates flexible covariance structures over time with automatic determination of all hyper-parameters (including the underlying rank, i.e., number of factors). Simulations demonstrate that the approach captures the underlying signal more precisely than other strategies, and performs well with respect to missing data imputation under different scenarios. An application to a gene time-course data for multiple sclerosis treatment provides a factorization with interpretable components over time and improved imputation accuracy for missing data.
keywords: Dimension reduction; Gaussian process; Low-rank approximation; Temporal correlation; Variational Bayes
Introduction
Matrix factorization methods have long been critical to analyze and summarize high-dimensional data for diverse tasks such as latent structure inference, data compression, and missing data imputation. Recently, empirical variational Bayes has been shown to be a powerful, flexible, and efficient approach to low-rank matrix factorization. Under a variational Bayes approach the true posterior for a Bayesian model is approximated by a variational distribution with simplified assumptions [1], under EVB the model hyperparameters are also empirically determined. Previous work describes a simple EVB approach to matrix factorization that results in an efficient closed-form low-rank solution [2]. Empirical Bayes matrix factorization flexibly incorporates different forms of priors, allowing novel combinations of classical models such as sparse matrix factorization and non-negative matrix factorization [3]. In the context of multiple linked matrices, EVB has been used to decompose signal into low-rank approximations for shared and unshared structure [4].
Advancements in biotechnology have enabled high-throughput data collection (e.g., for “omics” or imaging modalities) from multiple timepoints, motivating longitudinal studies. However, in most cases it is difficult to obtain complete data, often caused by patient unavailability or dropout. This can lead to several issues, such as biased results and a reduction in statistical power, compromising the validity of the study and the reliability of its findings. As a motivating example, consider a study that investigates genomic responses of multiple sclerosis (MS) patients to the genetic therapy recombinant human interferon beta (rIFN) [5]. The study involves analyzing a gene time course dataset that measures 76 gene expression levels in 53 subjects, at 7 timepoints over a 2-year followup period. However, 10.5% of the data are missing across all timepoints, limiting the validity of inference.
Several classes of methods have been developed to apply factorization models to longitudinal data settings. One popular area of research are tensor factorization methods [6].
The tensor framework generalizes the two-way matrix to higher-order arrays. Tensor factorization methods enjoy many desirable properties of matrix factorization, such as low rank representation of the data and interpretable additive components. They can conveniently accommodate longitudinally collected high-dimensional matrices by adding time as a third dimension, forming a third-order tensor array. Tensor factorization methods have been developed specifically to accommodate temporal structure, by incorporating functional representations or smoothing along the temporal dimension [7, 8]. Other tensor methods have been developed for imputation in a longitudinal setting using a tensor factorization [9]. However, treating time as an additional dimension within a traditional tensor framework assumes that timepoints are aligned across individuals.
For matrices, a related class of methods incorporate auxiliary side information on rows or columns to inform matrix factorization. For example, [10] considers additional matrices of covariates for the rows and columns, and borrows information to factorize all simultaneously. Variational Bayes has enabled computationally efficient posterior approximation for matrix factorization under structured Bayesian priors that incorporate side information for rows or columns [11]; however, this requires model-specific hyperparameters that are not empirically determined. More recently, [12] describe an empirical variational Bayes approach in which auxiliary covariates inform the latent factor means. While time can be incorporated as a covariate in such methods to capture mean trends over time, they do not inherently accommodate within-individual correlation structures over time that are common for longitudinal studies.
A small but growing number of matrix factorization models are explicitly designed for longitudinal settings. For example, [13] use a factor analysis approach and models the scores using a multivariate linear mixed model. This induces correlation within individual observations, but limits flexibility to model the correlation to the specific form under a linear mixed model. Bayesian methods, on the other hand, can deal with more complex correlation structures by naturally incorporating flexible priors for the factors. For example, [14] uses a factor model for random effects within a mixed modeling framework, where the scores are employed with an infinite mixture prior. However, this work centered on the task of clustering instead of signal estimation or imputation, which is the focus of this paper. Alternatively, [15] propose a longitudinal factor model for disease progression from high-dimensional data, but assume a transition between two latent states over time rather than a more general correlation structure.
We propose an empirical variation Bayes factor model for high-dimensional longitudinal data that simultaneously addresses estimation and imputation, even when timepoints are inconsistently measured. Our contribution has four main components. First, we introduce a general repeated measures model for high-dimensional data using latent factors that flexibly accommodates a range of correlation structures over time. Second, we use an EVB framework for estimation that does not require any user-specified tuning parameters, and that achieves automatic rank selection—an important advantage since manual rank selection can be difficult or computationally prohibitive for large datasets. Third, we develop a novel empirical Bayes random process model (EBRPM), which extends prior work to allow efficient computation and unified estimation of both priors and variationally approximated posteriors. This formulation leverages empirical Bayes principles to maximize efficiency while retaining the ability to recover complex signals in high-dimensional settings. Fourth, our framework naturally handles missing data, including both entrywise and blockwise missingness, by retaining a key Bayesian property: the sharing of information across individuals to estimate a common prior distribution. This feature is especially important for blockwise missingness, where data for an individual may be entirely absent at a timepoint and imputation without borrowing information and temporal correlation is impossible.
Methods
Model Construction
Here we describe our model for aligned measurement times across all subjects; extensions to scenarios with irregularly measured timepoints across subjects are straightforward, and are described in § 10.1 of the Supplementary Materials.
Assume there are subjects, each with variables measured at timepoints. We define the rank model as
where are the observed data at each timepoint, , are factor matrices and is an error matrix at time . Following standard terminology, we refer to as the loading matrix, as the score matrix, and as the rank/number of factors. The model can be expanded as the sum rank-1 components, via the outer products between columns of and ,
where and denote the -th column of and , respectively. Following a Bayesian framework, we give independent normal priors to each column of , and a joint normal prior to , the corresponding entries of across timepoints:
where captures component-specific temporal correlation in the factor scores, as detailed in § 2.4. The use of normal priors with factor-specific variances can be considered an automatic relevance determination (ARD) prior [16, 17, 18, 19].
Empirical Variational Bayes Estimation
We begin by describing the estimation algorithm for the rank 1 model, and generalize to the multiple rank scenario in § 2.3. In a standard empirical Bayes framework, model estimation proceeds by first obtaining point estimates for the prior parameters , , , e.g., by maximizing the marginal log-likelihood under estimated priors :
then performing inference on , via the posterior distribution
Approximate Inference
The integrals involved in both (4) and (5) are intractable due to marginalizing over latent variables. Therefore, we must resort to approximate inference techniques. Markov Chain Monte Carlo (MCMC) [20] is the most common technique where approximate samples are drawn from the true posterior, however it often suffers from high computational demands for large-scale problems. Instead, we propose a computationally attractive variational Bayes solution for our setting.
In variational Bayes, posterior estimation is posed as an optimization problem, where we search for a factor distribution that gives the smallest KL divergence with the true posterior:
[21]. This is done by maximizing the evidence lower bound (ELBO),
which is a lower bound for the log marginal likelihood, as the KL divergence is non-negative. Since is constant with respect to ,
However, without further constraints on , this optimization is difficult. We restrict the space of factor distributions to the mean field variational family,
Under this factorization assumption, we develop a coordinate ascent algorithm to solve for given by Algorithm 1. To allow for modeling of flexible correlation structure, we define a simpler, more general empirical Bayes model with optimization in the following section that can be used to solve step 4 and step 5 of Algorithm 1.
Algorithm 1 Coordinate Ascent Model Estimation with Rank-1 Model
Require: Initialize , , , ,
repeat
until convergence criterion met
Return
Optimizing via Empirical Bayes Random Process Means (EBRPM) Model
In this section, we define the Empirical Bayes Random Process Means Model (EBRPM). This model is closely related to the proposed model, in the sense that there is an equivalence with respect to or when the other is fixed. Suppose that we have noisy observations of true values , each can potentially be of different dimension:
The ’s are known diagonal matrices reflecting independent noise and are generated from a random process. Here we focus on modeling correlation within the ’s and restrict our prior family for to zero-mean Gaussian processes,
since there are many well studied covariance functions and this distribution generally have good computational properties. We are interested in estimating the ’s and via an empirical Bayes approach, and this is performed in two steps:
Estimate by type II maximum likelihood:
Calculate the posteriors for ’s using the estimated prior from the first step:
This model is a special case of the empirical Bayes multivariate normal means model (EBMNM) [22] with multivariate Normal priors when for all realized at the same timepoints, and the empirical Bayes normal means model (EBNM) [3] with normal priors when for all . Following the EBNM notation, we summarize the estimation procedure as a function, with an input of and output , where , . In Theorem 2.1 we establish that steps 4 and 5 of Algorithm 1 can be solved by an EBRPM problem.
Theorem 2.1. Step 4 and 5 of Algorithm 1 are equivalent to the following EBRPM problems:
and
Where , , represent the first and second moments of under .
The proof of Theorem 2.1 is provided in Section S2 of the Supplementary Materials, and additional computational details are provided in § 10.3 and § 10.4.
Extension to Rank Model
For the rank model (), we simply extend the variational approximation by setting factor distributions for all as in (2). is then maximized over and Algorithm 1 is modified accordingly to iteratively update each factor component while keeping all others constant. The update steps for each , follows Theorem 2.1, with replaced by . We summarize the estimation procedure in Algorithm 2.
Algorithm 2 Extension to Rank Model
Require: Initialize , , , , ,
repeat
for do
end for
until convergence criterion met
Return ,
When is large, this algorithm can become computationally intensive, as all factors are updated at each iteration. To alleviate this issue, we implement a greedy algorithm, where rank-1 models are sequentially fitted to the residuals from the sum of previous estimated models, effectively adding one column to the factor matrices at a time, as described in Algorithm 3.
Algorithm 3 Greedy Algorithm for Initialization
Require: Initialize and for
repeat
Set for
Perform Algorithm 1 with for
Set , for
Set , , for
until convergence criterion met
Return for
In practice, Algorithm 2 often achieves significantly better fit, given that it optimizes over a larger space. Combining both by using the greedy algorithm as an initialization for the coordinate ascent algorithm results in significantly less iterations required to achieve convergence, thus reducing computational cost without sacrificing model fit.
Covariance Functions
Several covariance functions for the Gaussian process prior of have been developed for a wide range of scenarios. We implement several common covariance functions, as follows:
where denotes the set of positive definite matrices. We consider the model with independence over time as a benchmark to quantify the performance gain of modeling covariance. This is a special case of the model described in [2] and [4], which has a closed form analytic solution to the empirical variational Bayes approximation. Thus we opt to use their code for faster implementation. Covariance (17) is a free structure with a different covariance for each pair of timepoints. It is impractical at highly irregular timepoints, but is the most flexible structure for the regular timepoint model, and is likely to perform well when the true model structure is unclear. The white noise and exchangeable covariances (14) and (15) have a closed form solution for their parameters in the optimization sub-step (10), while the other covariance structures may be estimated via gradient-based techniques (see § 10.3 of the Supplementary Materials for more details).
The choice of covariance structure may depend on context and nature of the data. For example the RBF covariance (16) accounts for distance between timepoints and is well-motivated when observations are irregularly spaced over time. Otherwise, our results show that the free covariance model is generally a robust choice to misspecification of the true covariance structure (see simulations in § 3 and Section S6.2), and is a flexible option. Alternatively, cross-validation of imputation accuracy may be used to select a covariance structure, as done for our application in § 4. The objective function itself is not recommended as a general criteria to select covariance, as the empirical estimation will always favor the free model because it subsumes the other models.
Model Dimension Selection
Variational Bayes matrix factorization methods have the advantage of automatically selecting the rank concurrently with the estimation framework [3, 17]. This comes from the fact that can be viewed as a balance between model fit and prior resemblence [21]:
Let , and a larger model , . It is clear that when , . So when the decrease in expected squared residuals is less than the penalty incurred by the KL divergence between the added factor distributions and their priors, cannot increase and prefers the lower rank model, equivalent to optimizing towards point masses at zero. Thus, setting large initially, the backfit algorithm “prunes” the model by setting redundant factors to zero.
Further, rank sparsity is also achieved through the empirical Bayes step, which can minimize the prior density to a point mass at zero. As noted in [3] and [17], it is possible that the marginal likelihood is maximized for at zero, and the following lemma from [3] shows that is guaranteed to be maximized at if the marginal likelihood is, which can be applied to our model as well.
Lemma. For a rank 1 model, if
then
In our context, this means that , are maximized at . This justifies the procedure of the greedy algorithm, as columns will be added to the factor matrices until eventually adding a column of zeros which terminates the algorithm.
Identifiability
As in other matrix factorization models, the loading and score components are only identifiable up to a scaling factor, and permutations [3, 23]. Specifically, for any , rescaling and , together with , , leaves both and the estimated signal invariant. This is generally not an issue as signal estimation and imputation are unaffected. A scale convention can be imposed by normalizing each loading estimate to norm 1 and adjust the scale of the corresponding scores accordingly.
Missing Data Imputation
Here we consider incomplete data and the task of imputing missing elements. Denote the indices of missing entries as and , we propose an Expectation-Maximization type approach, summarized in Algorithm 4.
Algorithm 4 Imputation
Set
repeat
Perform the complete data algorithm on and denote for
Impute ,
until for some small
Return
This approach is analogous to previous methodologies including [4] and [24]. Note that step 4 and 5 are respectively the “maximization” and “expectation” steps, as the former maximizes according to Theorem 2.1 and Algorithm 2, and the latter replaces the missing entries with the expected value under current estimates of the factor distributions.
This algorithm is justified by Theorem 2.2, which shows the imputation algorithm optimizes over missing entries as well as the latent factor distributions and hyperparameters.
Theorem 2.2. Algorithm 4 iteratively maximizes over , , , ,
The proof of Theorem 2.2 is given in Section S2 of the Supplementary Materials. Compared to the related imputation algorithm in [4], we do not require pre-normalizing the data by estimating first, but rather incorporate it into the optimization algorithm. This also avoids handling differently under entrywise and blockwise missingness.
Simulations
We conduct two simulation studies to assess both signal recovery and imputation accuracy. We use the regular timepoints model in all simulations to allow for comparisons with competing methods that may not accommodate irregular timepoints.
Simulation study 1: Complete data signal recovery
Here we assess our model’s ability to estimate underlying structure for complete data, and illustrate aspects of the different covariance functions. We generate matrices of size for 5 regularly spaced timepoints with true rank 3 according to our model description, under each of , , and covariance structures with varying signal-to-noise ratios (SNR) and correlation levels. We also use a block covariance structure that none of the covariance functions correctly capture to assess misspecification, where an correlation is given independently for different subsets of timepoints. See Section S5 of the Supplementary Materials for complete details on the data generation for each scenario. We implement our model using the three non-trivial covariance functions in § 2.4 (EC, RBF, Free), and without correlation using the EVB implementation in [4]. We also consider the general tensor decomposition method TEMPTED that doesn’t explicitly model covariance between timepoints but rather treats time as a continuous variable [8]. Accuracy is measured using the relative squared error:
where is the low-rank signal: . Table 1 shows the results for all methods. For each covariance structure, the correctly specified model consistently performs the best, as one would reasonably expect. When SNR is high, the model can perform well without modeling correlation, even when the true underlying structure is highly correlated. However, the margin of improvement in estimation accuracy is more significant under low SNR settings, demonstrating that the borrowing of information is effective when the data itself has low information. The TEMPTED method does not perform in a comparable range in this experiment as it does not assume a correlated low rank matrix structure. In the misspecified scenario, the free covariance model performs consistently the best across SNR ratios and correlation level, since it can be considered as correctly specified but overparametrized, which leads to more robust estimation at the cost of lower efficiency. Lastly, we note that in all simulation repetitions, all models selected the true rank.
| Method | Low SNR Low Corr | Low SNR High Corr | Medium SNR Low Corr | Medium SNR High Corr | High SNR Low Corr | High SNR High Corr |
| EC | 0.239 (0.007) | 0.184 (0.005) | 0.042 (0.001) | 0.043 (0.001) | 0.012 (0.000) | 0.011 (0.000) |
| RBF | 0.213 (0.006) | 0.139 (0.005) | 0.040 (0.001) | 0.029 (0.001) | 0.011 (0.000) | 0.008 (0.000) |
| Free | 0.218 (0.006) | 0.149 (0.005) | 0.040 (0.001) | 0.030 (0.001) | 0.011 (0.000) | 0.008 (0.000) |
| EVB | 0.251 (0.007) | 0.251 (0.008) | 0.042 (0.001) | 0.047 (0.002) | 0.012 (0.000) | 0.012 (0.000) |
| TEMPTED | 0.509 (0.023) | 0.515 (0.018) | 0.110 (0.002) | 0.080 (0.003) | 0.073 (0.001) | 0.016 (0.001) |
| EC | 0.244 (0.007) | 0.181 (0.003) | 0.045 (0.001) | 0.040 (0.001) | 0.012 (0.000) | 0.011 (0.000) |
| RBF | 0.248 (0.007) | 0.231 (0.003) | 0.045 (0.001) | 0.042 (0.001) | 0.012 (0.000) | 0.011 (0.000) |
| Free | 0.247 (0.007) | 0.189 (0.003) | 0.045 (0.001) | 0.040 (0.001) | 0.012 (0.000) | 0.011 (0.000) |
| EVB | 0.254 (0.007) | 0.251 (0.005) | 0.045 (0.001) | 0.043 (0.001) | 0.012 (0.000) | 0.012 (0.000) |
| TEMPTED | 0.647 (0.018) | 0.639 (0.017) | 0.247 (0.004) | 0.109 (0.003) | 0.208 (0.003) | 0.062 (0.002) |
| EC | 0.243 (0.024) | 0.046 (0.005) | 0.015 (0.001) | |||
| RBF | 0.242 (0.024) | 0.046 (0.005) | 0.015 (0.001) | |||
| Free | 0.203 (0.022) | 0.040 (0.005) | 0.013 (0.001) | |||
| EVB | 0.248 (0.026) | 0.047 (0.006) | 0.015 (0.001) | |||
| TEMPTED | 0.536 (0.080) | 0.089 (0.009) | 0.050 (0.006) | |||
| EC | 0.250 (0.006) | 0.232 (0.004) | 0.045 (0.001) | 0.044 (0.001) | 0.012 (0.000) | 0.011 (0.000) |
| RBF | 0.250 (0.006) | 0.225 (0.004) | 0.045 (0.001) | 0.043 (0.001) | 0.012 (0.000) | 0.011 (0.000) |
| Free | 0.252 (0.006) | 0.203 (0.004) | 0.045 (0.001) | 0.042 (0.001) | 0.012 (0.000) | 0.011 (0.000) |
| EVB | 0.256 (0.006) | 0.245 (0.005) | 0.045 (0.001) | 0.044 (0.001) | 0.012 (0.000) | 0.011 (0.000) |
| TEMPTED | 0.669 (0.018) | 0.551 (0.012) | 0.260 (0.003) | 0.111 (0.002) | 0.227 (0.003) | 0.068 (0.002) |
Table 1. RSE for complete data estimation
Simulation study 2: Missing data imputation
Here we evaluate the imputation performance of our model. We consider both entrywise and columnwise missingness, where the latter refers to entire observations missing at a timepoint. Data are generated as in § 3.1, then 10% of the data are randomly set to be missing. In addition to the methods in experiment 1, we compare our model against two imputation algorithms - BAMITA, which is a Bayesian tensor imputation model [9], as well as MissForest, a non-parametric, random forest-based imputation method that doesn’t assume low rank latent structure [25]. We evaluate imputation accuracy by restricting to the for missing entries only.
We note that TEMPTED cannot handle entrywise missing data so it is excluded from the setting. The results for entrywise missingness are shown in Table 2. The patterns closely mirror the complete data estimation results from Table 1, where the correctly specified covariance model consistently performs the best, especially in low SNR settings. Alternative imputation methods perform markedly worse across all scenarios, with BAMITA performing substantially worse in high SNR settings, due to the strong signal indicating a non-tensor data structure. MissForest exhibits substantially higher errors potentially due to not assuming low rank structure or an explicit correlation structure across time.
| RBF Scenario | ||||||
| Method | Low SNR: Low Corr | Low SNR: High Corr | Medium SNR: Low Corr | Medium SNR: High Corr | High SNR: Low Corr | High SNR: High Corr |
| EC | 0.280 (0.010) | 0.212 (0.007) | 0.051 (0.001) | 0.049 (0.002) | 0.013 (0.000) | 0.013 (0.001) |
| RBF | 0.249 (0.009) | 0.161 (0.006) | 0.048 (0.001) | 0.033 (0.001) | 0.013 (0.000) | 0.009 (0.000) |
| Free | 0.255 (0.009) | 0.187 (0.009) | 0.049 (0.001) | 0.034 (0.001) | 0.013 (0.000) | 0.009 (0.000) |
| EVB | 0.295 (0.010) | 0.295 (0.008) | 0.052 (0.001) | 0.054 (0.002) | 0.014 (0.000) | 0.013 (0.001) |
| BAMITA | 0.390 (0.012) | 0.272 (0.009) | 0.115 (0.002) | 0.053 (0.002) | 0.075 (0.001) | 0.013 (0.001) |
| MissForest | 0.802 (0.018) | 0.810 (0.015) | 0.211 (0.004) | 0.223 (0.004) | 0.089 (0.002) | 0.089 (0.003) |
| EC Scenario | ||||||
| Method | Low SNR: Low Corr | Low SNR: High Corr | Medium SNR: Low Corr | Medium SNR: High Corr | High SNR: Low Corr | High SNR: High Corr |
| EC | 0.287 (0.012) | 0.199 (0.006) | 0.051 (0.002) | 0.048 (0.001) | 0.014 (0.000) | 0.013 (0.000) |
| RBF | 0.290 (0.013) | 0.246 (0.006) | 0.051 (0.002) | 0.051 (0.002) | 0.014 (0.000) | 0.014 (0.000) |
| Free | 0.291 (0.012) | 0.211 (0.007) | 0.051 (0.002) | 0.048 (0.002) | 0.014 (0.000) | 0.013 (0.000) |
| EVB | 0.296 (0.013) | 0.291 (0.009) | 0.052 (0.002) | 0.053 (0.002) | 0.014 (0.000) | 0.014 (0.000) |
| BAMITA | 0.528 (0.013) | 0.314 (0.008) | 0.240 (0.004) | 0.103 (0.002) | 0.206 (0.005) | 0.065 (0.002) |
| MissForest | 0.800 (0.019) | 0.804 (0.020) | 0.209 (0.006) | 0.210 (0.006) | 0.088 (0.003) | 0.088 (0.003) |
| Free Scenario | ||||||
| Method | Low SNR | Medium SNR | High SNR | |||
| Single Condition | Single Condition | Single Condition | ||||
| EC | 0.332 (0.027) | 0.051 (0.004) | 0.015 (0.001) | |||
| RBF | 0.331 (0.028) | 0.051 (0.004) | 0.015 (0.001) | |||
| Free | 0.283 (0.040) | 0.043 (0.004) | 0.014 (0.001) | |||
| EVB | 0.339 (0.029) | 0.051 (0.004) | 0.015 (0.001) | |||
| BAMITA | 0.401 (0.040) | 0.082 (0.007) | 0.051 (0.006) | |||
| MissForest | 0.883 (0.052) | 0.215 (0.014) | 0.098 (0.005) | |||
| Block Scenario | ||||||
| Method | Low SNR: Low Corr | Low SNR: High Corr | Medium SNR: Low Corr | Medium SNR: High Corr | High SNR: Low Corr | High SNR: High Corr |
| EC | 0.286 (0.005) | 0.277 (0.007) | 0.051 (0.001) | 0.054 (0.001) | 0.013 (0.000) | 0.014 (0.000) |
| RBF | 0.287 (0.005) | 0.266 (0.006) | 0.051 (0.001) | 0.053 (0.001) | 0.013 (0.000) | 0.014 (0.000) |
| Free | 0.288 (0.005) | 0.245 (0.006) | 0.051 (0.002) | 0.050 (0.001) | 0.013 (0.000) | 0.013 (0.000) |
| EVB | 0.291 (0.005) | 0.295 (0.008) | 0.051 (0.001) | 0.054 (0.001) | 0.013 (0.000) | 0.014 (0.000) |
| BAMITA | 0.517 (0.007) | 0.363 (0.009) | 0.255 (0.005) | 0.111 (0.002) | 0.212 (0.004) | 0.066 (0.001) |
| MissForest | 0.795 (0.010) | 0.800 (0.014) | 0.209 (0.005) | 0.222 (0.004) | 0.086 (0.003) | 0.090 (0.002) |
Table 2. RSE for entrywise missingness imputation
Table 3 summarizes the results for columnwise missingness imputation. The task becomes much more difficult, as imputation completely relies on shared information between timepoints. This is shown by the results of low correlation scenarios under both the and correlation structure, where vary little over SNR levels. Even for the correct model, is significantly worse than in estimation or entrywise imputation. All alternative methods other than BAMITA have , which is the of null imputing with 0. It is also worth noting that misspecification of the correlation structure is more costly, as shown by the relatively worse performance of our model in misspecified scenarios compared to estimation and entrywise imputation. Only the most flexible free covariance could still achieve an well below 1. On the other hand, it is remarkable that the proposed method performs relatively well even under the lowest SNR under high correlation, recovering about 70% of true structure in both the and scenarios. In all repetitions, all models selected the true rank.
| Method | Low SNR Low Corr | Low SNR High Corr | Medium SNR Low Corr | Medium SNR High Corr | High SNR Low Corr | High SNR High Corr |
| RBF Scenario | ||||||
| EC | 0.880 (0.071) | 0.352 (0.037) | 0.843 (0.057) | 0.291 (0.052) | 0.847 (0.022) | 0.282 (0.038) |
| RBF | 0.627 (0.061) | 0.234 (0.033) | 0.462 (0.050) | 0.083 (0.014) | 0.449 (0.063) | 0.042 (0.009) |
| Free | 0.670 (0.066) | 0.264 (0.042) | 0.476 (0.050) | 0.089 (0.015) | 0.462 (0.068) | 0.044 (0.008) |
| EVB | 1.000 (0.000) | 1.000 (0.000) | 1.000 (0.000) | 1.000 (0.000) | 1.000 (0.000) | 1.000 (0.000) |
| BAMITA | 0.874 (0.082) | 0.350 (0.061) | 0.652 (0.105) | 0.124 (0.021) | 0.900 (0.233) | 0.059 (0.013) |
| MissForest | 11.205 (1.150) | 10.368 (1.577) | 3.430 (0.432) | 3.392 (0.346) | 2.443 (0.319) | 2.279 (0.219) |
| TEMPTED | 2.005 (0.401) | 1.978 (0.132) | 1.643 (0.204) | 1.776 (0.197) | 1.508 (0.139) | 1.908 (0.300) |
| EC Scenario | ||||||
| EC | 0.928 (0.030) | 0.335 (0.027) | 0.931 (0.036) | 0.278 (0.033) | 0.916 (0.018) | 0.254 (0.027) |
| RBF | 0.976 (0.018) | 0.380 (0.048) | 0.973 (0.028) | 0.557 (0.064) | 0.970 (0.020) | 0.506 (0.050) |
| Free | 0.964 (0.028) | 0.373 (0.028) | 0.975 (0.064) | 0.294 (0.039) | 0.932 (0.025) | 0.263 (0.032) |
| EVB | 1.000 (0.000) | 1.000 (0.000) | 1.000 (0.000) | 1.000 (0.000) | 1.000 (0.000) | 1.000 (0.000) |
| BAMITA | 1.325 (0.163) | 0.451 (0.033) | 2.723 (0.255) | 0.516 (0.069) | 5.835 (1.340) | 0.812 (0.220) |
| MissForest | 9.877 (1.417) | 10.730 (1.043) | 3.556 (0.396) | 3.282 (0.224) | 2.459 (0.164) | 2.249 (0.213) |
| TEMPTED | 1.723 (0.216) | 2.172 (0.317) | 1.519 (0.140) | 1.811 (0.188) | 1.500 (0.162) | 1.856 (0.273) |
| Free Scenario | ||||||
| EC | 0.999 (0.014) | 0.970 (0.042) | 0.986 (0.032) | |||
| RBF | 0.981 (0.037) | 0.971 (0.035) | 0.976 (0.040) | |||
| Free | 0.662 (0.196) | 0.364 (0.126) | 0.318 (0.133) | |||
| EVB | 1.000 (0.000) | 1.000 (0.000) | 1.000 (0.000) | |||
| BAMITA | 0.807 (0.222) | 0.477 (0.147) | 0.673 (0.415) | |||
| MissForest | 11.321 (2.702) | 3.653 (0.504) | 2.353 (0.165) | |||
| TEMPTED | 2.041 (0.358) | 1.793 (0.183) | 1.784 (0.234) | |||
| Block Scenario | ||||||
| EC | 0.979 (0.029) | 0.866 (0.071) | 0.981 (0.018) | 0.843 (0.054) | 0.991 (0.020) | 0.868 (0.028) |
| RBF | 0.983 (0.020) | 0.697 (0.063) | 0.972 (0.023) | 0.651 (0.058) | 0.988 (0.015) | 0.690 (0.044) |
| Free | 0.999 (0.045) | 0.549 (0.041) | 0.990 (0.047) | 0.405 (0.040) | 0.969 (0.030) | 0.418 (0.056) |
| EVB | 1.000 (0.000) | 1.000 (0.000) | 1.000 (0.000) | 1.000 (0.000) | 1.000 (0.000) | 1.000 (0.000) |
| BAMITA | 1.416 (0.123) | 0.700 (0.053) | 3.010 (0.381) | 0.633 (0.071) | 5.545 (1.837) | 0.985 (0.125) |
| MissForest | 10.182 (0.997) | 11.127 (1.703) | 3.275 (0.332) | 3.389 (0.325) | 2.384 (0.167) | 2.209 (0.134) |
| TEMPTED | 1.801 (0.163) | 2.093 (0.332) | 1.616 (0.155) | 1.686 (0.241) | 1.564 (0.112) | 26.49 (77.14) |
Table 3. RSE of blockwise imputation for all methods under varying simulated conditions, with the best performing method highlighted in bold font.
Gene Time Course Data Application
We apply our method to a set of gene time course data described in [5]. This study explored methods to classify genetic response of multiple sclerosis patients to recombinant human interferon beta (rIFN) treatment. rIFN was given to 53 patients and expression levels of 76 genes were measured at 7 timepoints (baseline and follow-ups at 3, 6, 9, 12, 18, 24 months after treatment). Most patients had some percentage of missingness, ranging from 1 – 6 entire timepoints missing and additional partial missingness at observed timepoints. Overall, 10.5% of all data were missing. We first use this dataset to evaluate the empirical imputation performance of our model. Similar to the experiments in § 3.2, we apply our model with the three non-trivial covariance functions, EVB under independence, BAMITA, and MissForest to the data. TEMPTED is omitted as it cannot handle entrywise missingness. We take the complete indices and perform 10, 3, 2-fold cross validation based on the missing proportion, repeated 10 times over different random seeds. Figure 1 shows the results.

Figure 1. Analysis results for the IFN data application: Imputation accuracy for all methods from cross validation.
In the comparisons, the EC covariance model performed the best across both entrywise and columnwise missingness. The independent model performed comparably in the entrywise missing scenarios, but had in columnwise missingness, consistent with simulation results. All alternative methods performed worse than the independent model, highlighting the advantage of our proposed model in columnwise imputation.
As the covariance model performed the best, we refer to the results produced by this model for interpretation. We estimated a total of 14 components; the top 7 ordered by variance explained are shown in Figure 2a, along with the estimated correlation in each component’s scores. The first component explains about 35% variance, while the next 6 each explains less than 10% variance each. Both the third and seventh component exhibits high correlation between sample scores at different timepoints. We plot individual score trajectories in Figure 2b, colored by manually labeled good/poor responders from the original study. The first component, which is the major contributor, exhibits similar patterns for most individuals, indicating a mean effect of subjects. The seventh component, on the other hand, shows a clear clustering effect in good and poor responders, with good responders at the bottom and poor responders at the top. The individual scores have little variation between time points, which indicates that the two types of responders can be distinguished without inferring from temporal patterns. This is consistent with the results in [5], where no group*time interaction effects were discovered; as well as [26], where distinction between the two groups were not driven by factors that vary over time. More generally, the EC covariance and accompanying score trajectories suggest some clear subject effects but no other temporal trends.
Figure 2. (a) Variance explained and estimated correlation () for the first 7 components. (b) Individual trajectories for the first and seventh component scores. Figure 2: Analysis results for the IFN data application.
To further investigate the biological relevance of Component 7, we performed a gene-set enrichment analysis of its loadings using GO Biological Processes via WebGestalt [27]. While no processes were significant at the FDR level, the top enriched processes and their nominal significance is shown in Figure 3. The most significant and top downregulated process, implying higher levels for good responders, was response to transforming growth factor beta; this is consistent with prior literature associating higher TGF- levels with better treatment response [28].

Figure 3. Gene set enrichment analysis for loadings of Component 7 in the IFN factorization
Discussion
In this paper, we introduce a latent matrix factorization method for high dimensional longitudinal data that captures temporal correlation by flexibly modeling the covariance structure in latent factor scores. The additive nature of our model enables decomposition of data into static and temporal factors, enhancing interpretability. The model employs a computationally efficient empirical variational Bayes estimation algorithm that automatically estimates hyperparameters and selects model dimension by optimizing an objective function balancing model fit and complexity. Through extensive simulation studies and a real world data application, the proposed method demonstrates superior estimation and imputation accuracy compared to existing approaches across diverse scenarios.
While we focus on single-source data, multimodal datasets combining data from multiple sites are increasingly common in biomedical studies, such as collaborative neurodegenerative disease research that collects imaging, clinical, and genetic modalities [29, 30, 31]. Extensions of our method to multimodal matrix factorization—where some components are restricted to specific cohorts or data modalities—is an interesting direction for future research, as the full rank and greedy estimation algorithms do not straightforwardly extend to such settings.
A known limitation of variational Bayes methods is uncertainty quantification. While point estimates generally perform well and have good properties, variational Bayes methods typically underestimate marginal posterior variances [32, 33]. To address this limitation, our method can serve as initialization for the associated fully Bayesian model, offering several advantages including initialization near the posterior mode and automatic rank selection, which are challenging for the fully Bayesian approach to achieve alone. A related Bayesian model is described in [34], which illustrates the benefits of accounting for uncertainty in imputation and prediction. However, standard MCMC strategies such as Gibbs sampling would likely face substantial computational challenges, necessitating the development of more efficient sampling algorithms. As our model imputes values given the fit to observed data, it implicitly assumes data are missing at random; extensions to scenarios where the missing mechanism is not random but known (i.e., a limit of detection) is another direction of future work.
Code & Data Availability
Documented R code to perform the proposed method, including an R package, and the IFN dataset analyzed in § 4 is available at https://github.com/andyljk/Longitudinal_Factorization. Additionally, a supplementary zipped folder includes code to perform all the simulations and data analysis in the manuscript.
Supplementary Materials
A supplementary document available online includes proofs, details for the method with irregular timepoints, several additional simulation studies, additional computational details, and further details for the real data application.
References
Supplementary Materials for “Empirical Variational Bayes Factorization for Longitudinal Matrix Data”
Jiakun Li; Eric F. Lock
Supplementary Materials
Irregular Timepoint Model
In this section, we provide a description of the key differences of the extended model that handles inconsistency in measured timepoints across individuals. For the individual, assume each with variables measured at time . The outcome for individual , variable at time is modeled as
Note that the matrix product representation in (1) is not available in this setting, since does not have consistent measures at each timepoint. Priors are prescribed to each of similar to (3), where
and . The prior covariance matrix has dimension matching the number of measurements from individual . We assume all prior covariance are parametrized by a common set of parameters as described in § 2.4, and denote estimates of as for convenience. When the measurements are highly irregular, modeling the noise variance separately at each timepoints becomes difficult. In this situation, a reasonable modification is to set for all .
Denoting and , we maximize the same objective function using algorithm 2. For the rank 1 model, it can be shown following the proof of Theorem 2.1, step 4, 5 of algorithm 1 are equivalent to solving the following
and
Note that each is of potentially different dimension, which fits in the framework of the EBRPM model by design. For estimation of the multiple rank model, steps 5, 6 of algorithm 2 and step 3 of algorithm 3 are modified accordingly to accommodate the timepoint structure.
Proofs
Theorem 2.1
We prove Theorem 2.1, where we stated the equivalence between step 4, 5 of algorithm 1 and the EBRPM problem. The proof is similar to Proposition 5 in [3], which can be viewed as a univariate case of our model. We first introduce the following lemma as an intermediate step.
Lemma S1. For , let , , and be known. Recall the EBRPM first estimates the prior variance function as
then uses this estimate to calculate the posterior . The solution to the EBRPM problem is equivalent to the solution of the following objective:
where
where ,
Proof. To demonstrate equivalence, we need to show that and . Note that
Since with equality iff , we see that
when , so then
Lastly we show that has the expanded form, which can be done by straightforwardly factorizing the joint distribution:
We now prove Theorem 2.1.
Proof. In step 4 of Algorithm 1, we treat all quantities other than as fixed. The objective function is then
Where ,
Then equation (12) follows from Lemma S1. Similarly, with respect to for step 5, the objective function becomes
Where , ,
and equation (13) follows from Lemma S1.
Theorem 2.2
Proof. From algorithm 2, it can be seen that the M steps of algorithm 4 each partially maximize the ELBO over , holding all other quantities fixed. So it leaves to show that the E step maximizes the ELBO over the missing entries . The ELBO can be decomposed into
holding all other quantities fixed, with respect to is proportional to
where is a constant
Thus it can be seen that is maximized at , for all .
Solving the EBRPM problem
Here, we provide the details for implementing the two step solution to solve the EBRPM problem described in § 2.2.2. From the model described in equation (8), 9, the first step is to estimate by maximizing the marginal likelihood. We restrict to the case where is parametrized by some parameters , and estimate instead, which is the case that all covariance functions discussed fall into. Below, we calculate the analytic form of the gradients and several well-documented gradient-based optimization methods can be used. For convenience, let , the marginal log likelihood is given by:
It can be shown that the matrix derivative of inverse matrices and log determinant of positive definite matrices are given by
Where denotes the elementwise derivative matrix. Thus the gradient is given by
Where . Once (and ) is obtained, the posterior distribution for is given by
ELBO Calculation
In the CAVI algorithm described in section 2.2.2, each update step to the parameters does not explicitly compute the objective function itself. Being able to calculate the objective function could be beneficial for monitoring the convergence of the optimization algorithm, especially since can be highly nonconcave. Here, we provide the procedure for computing using the same decomposition as in equation (S13). The first term is straightforward to compute, given the factor distributions . The other two terms can be identified as and , each ’th term from either sums can be found as follows from the EBRPM subproblem:
This calculation is straightforward to implement, as the only additional quantity required is the optimized log likelihood value from the EBRPM problem, which is usually not difficult to obtain.
Simulations: Data Generation
In the simulation section, we set the sample size , variable dimension , and number of timepoints to be complete data are generated according to the prior described in (3): For , draw the factor matrices , , compute the underlying structure and where entries of are standard normal draws. For the covariance structures (15), (16), we set for the three SNR we consider and fix at 1. This gives the low, medium, high SNR settings the SNR of 0.12, 0.75, 3 respectively, as the total SNR is the sum of each single rank component’s variance. In low and high correlation levels, we set for the structure and for the structure respectively. The free covariance structure (17) is generated as , where has standard normal entries. The block covariance is defined as a block diagonal covariance matrix where are 2-by-2 and 3-by-3 covariances respectively.
Additional Simulations
Irregular timepoints
Settings
In this experiment we investigate our model’s performance under irregular timepoints. The data dimensions are set to , , , consistent with § 3. The loadings and scores are generated as described in section S5. For subject , each score vector is drawn from the rbf covariance kernel. Parameters of the kernel are set to scale and length-scale for all of the components. The noise variance is set to . Each setting is repeated 100 times.
We used two designs for the visit times (Figure S1).
Figure S1. Visit times for 120 subjects in one draw from each design.
Jittered. Each subject has five planned fixed timepoints at times . Each visit is then offset by a small amount of noise . We then randomly remove 7% of the visits. After each visit, the subject drops out with probability 0.03.
Completely irregular. The number of timepoints is generated from a negative binomial distribution with size 2.8 and mean 4 then added 1, upper bounded at 15 timepoints. The time of the first visit is randomly generated using a LogNormal distribution, followed by the time between visits, randomly drawn from a Gamma distribution.
For each dataset, 10% of all visits are held out, one visit for each selected subject. Each subject is guaranteed to have at least one timepoint, so we only randomly sample from subjects with two or more timepoints. We fitted two types of model:
Irregular time model. The irregular model uses the true visit times of each subject. We used the EC and RBF priors.
Regular time model with binned times. Observed timepoints are rounded to the
nearest whole time unit. If multiple visits are in one time bin, we take the average.
We then fitted the aligned model with the EC, Free and RBF priors.
Figure S1 provides a visualization of the generated timepoints from a single dataset. To assess performance, we report the relative squared error at the observed visits and at the held-out visits. We also report the selected rank and the fit time.
Results
Table S1 and Table S2 provides a summary of the RSE performance on observed visits and heldout visits. In the jittered time design, both the irregular time model and the regular time model fitted with binned timepoints perform comparably across all SNR and correlation strength settings. However, in the irregular time setting, fitting the regular time model fails severely, especially when SNR is low. The irregular time models selected the true rank in all simulation replicates across all settings. On the other hand, while the binned model selected the true rank in all jittered time settings, Table S3 shows that it was unable to accurately estimate the rank in all completely irregular time settings with . Table S3 gives the settings where the binned models failed. These results demonstrate that correctly accounting for the irregular timepoints is essential and that fitting the regular time model with approximated timepoints results in poor model fit and incorrect model structure.
| Design | EC irregular | RBF irregular | EC binned | Free binned | RBF binned | ||
| Jittered | 1 | 3 | 0.012 | 0.009 | 0.012 | 0.010 | 0.009 |
| Jittered | 1 | 1 | 0.012 | 0.012 | 0.012 | 0.012 | 0.012 |
| Jittered | 0.5 | 3 | 0.044 | 0.032 | 0.045 | 0.034 | 0.033 |
| Jittered | 0.5 | 1 | 0.047 | 0.045 | 0.047 | 0.048 | 0.046 |
| Jittered | 0.2 | 3 | 0.200 | 0.159 | 0.212 | 0.166 | 0.161 |
| Jittered | 0.2 | 1 | 0.251 | 0.232 | 0.255 | 0.248 | 0.238 |
| Irregular | 1 | 3 | 0.012 | 0.008 | 0.024 | 0.014 | 0.016 |
| Irregular | 1 | 1 | 0.012 | 0.011 | 0.039 | 0.032 | 0.040 |
| Irregular | 0.5 | 3 | 0.044 | 0.028 | 0.143 | 0.046 | 0.053 |
| Irregular | 0.5 | 1 | 0.046 | 0.040 | 0.110 | 0.076 | 0.105 |
| Irregular | 0.2 | 3 | 0.212 | 0.147 | 0.838 | 0.276 | 0.675 |
| Irregular | 0.2 | 1 | 0.245 | 0.203 | 0.995 | 0.460 | 0.925 |
Table S1. Mean RSE at the observed visits.
| Design | EC irregular | RBF irregular | EC binned | Free binned | RBF binned | ||
| Jittered | 1 | 3 | 0.276 | 0.040 | 0.280 | 0.044 | 0.043 |
| Jittered | 1 | 1 | 0.844 | 0.432 | 0.842 | 0.465 | 0.440 |
| Jittered | 0.5 | 3 | 0.297 | 0.082 | 0.297 | 0.088 | 0.084 |
| Jittered | 0.5 | 1 | 0.841 | 0.474 | 0.840 | 0.530 | 0.486 |
| Jittered | 0.2 | 3 | 0.365 | 0.238 | 0.365 | 0.252 | 0.242 |
| Jittered | 0.2 | 1 | 0.861 | 0.616 | 0.861 | 0.669 | 0.623 |
| Irregular | 1 | 3 | 0.338 | 0.065 | 0.255 | 0.136 | 0.083 |
| Irregular | 1 | 1 | 0.812 | 0.359 | 0.604 | 0.565 | 0.457 |
| Irregular | 0.5 | 3 | 0.332 | 0.095 | 0.274 | 0.158 | 0.132 |
| Irregular | 0.5 | 1 | 0.784 | 0.380 | 0.630 | 0.625 | 0.488 |
| Irregular | 0.2 | 3 | 0.410 | 0.238 | 0.856 | 0.371 | 0.702 |
| Irregular | 0.2 | 1 | 0.826 | 0.524 | 0.998 | 0.786 | 0.959 |
Table S2. RSE at the held-out visits.
| EC binned | Free binned | RBF binned | |
| 3 | 0 (0–1) | 76 (2–3) | 0 (0–2) |
| 1 | 0 (0–1) | 39 (1–3) | 0 (0–1) |
Table S3. Number of replicates (out of 100) with the correct rank for the binned models, completely irregular design, . The range of the selected rank is included in parentheses.
Misspecified Settings
In this section, we investigate the robustness of our model to several ways in which data generation can misalign with the model. Unless stated otherwise, we use , , 5 timepoints, , , 10% entrywise or columnwise missingness and 100 replicates per setting, as in the main simulations.
Tensor data structure
Let denote the outer product. Data are generated from a rank-3 CP tensor model,
where and have entries, has entries to control the SNR, and the random errors terms are iid distributed . Figure S2 shows the results under completely observed data, 10% random entrywise missingness, and 10% random columnwise missingness. In all three settings, we observe that while the EC and RBF models perform worse than the correctly specified BAMITA model, the Free covariance model performs comparably across all SNR and observed/missing data settings. Indeed, the tensor structure is very similar to the Free covariance model, as each frontal slice of the tensor can be viewed as a matrix observation at a single timepoint. Under this angle, the flexible Free covariance structure model is able to capture covariance that is not structured smoothly over time, demonstrating robustness to such misspecifications.

Figure S2. RSE for data generated from the CP tensor model.
Mixture of temporal & static factors
Data are generated from our model with rank : two static factors, whose scores are the same at every timepoint for each subject, and two temporal factors, whose scores follow the RBF covariance with . Each factor has score variance , so the total signal matches the rank-3 settings. From Figure S3, we observe that our model performs similar to the correctly specified setting, and captures the true signal of the model.

Figure S3. RSE for data with two static and two temporal factors.
Non-Gaussian loadings
Data are generated as in the main simulations from § 3, with the loading-generating distribution replaced by the centered Gamma(1, 1) distribution. Each setting is compared with the same setting under loadings generated from a standard normal distribution. Figure S4 shows that the models under non-Gaussian loadings perform similarly to the correctly specified Gaussian loading setting, and thus misspecification of the loading distribution does not result in poor model fit of the EVBLM model.

Figure S4. RSE with centered gamma loadings (faded) and Gaussian loadings (solid). Rows give the true covariance of the scores.
Scalability
In this experiment, we investigate the scalability of our model. We benchmark the runtime to fit our model by varying each of the number of subjects, variables, timepoints, and missingness. Data are generated from our model with the rbf covariance with parameters , , and . We establish a reference setting of variables, subjects, timepoints and rank 3. We scale one of the dimensions each at a time: from 100 to 20000 variables; 50 to 5000 subjects; 3 to 50 timepoints; rank 1 to 20; and the total entries at a timepoint () from to , which gives to entries; and 10%, 30% and 50% of the entries or of the visits missing. Each setting is repeated 100 times. The model is fitted with the ec, free and rbf covariances, with automatic rank selection. All timings were obtained on a single node with two 64-core AMD EPYC 7763 processors and 512 GB of RAM, using R 4.3.0 with single-threaded OpenBLAS, with eight fits running concurrently.
Figure S5 shows the median runtime per model fit. The log computing time is linear in log growth of each of , , and . The consistent behavior is expected as these parameters all scale the total number of matrix entries linearly. This is also observed in the first panel on the second row of Figure S5, where we scale the parameters by the product of , , . All three covariance models perform comparably when scaling and . The free covariance model takes longer to fit with more timepoints, as there are parameters to estimate, whereas the parameters of EC and rbf are constant with respect to timepoints. In the missingness scenarios, the computing time depends on the mode of missingness. Entrywise missingness scales the log computing time linearly similar to the previous settings, because more missing entries require more iterations of the imputation step before the ELBO converges. On the other hand, computing time stays relatively stable in the blockwise missingness scenario, and even faster when 50% of the visits are missing.
Figure S5. Median computing time per fit (100 replicates) as each dimension increases, with the other dimensions fixed at 1000 variables, 200 subjects, 5 timepoints and rank 3. Both axes are on a log scale.
IFN Data Application
Preprocessing
As described in the original study [5], expression of pre-selected genes was measured in PBMC RNA by one-step kinetic RT-PCR (ABI 7900), with triplicate reactions on 384-well plates that included positive and negative controls and a calibration curve. Absolute expression values were obtained by interpolation from a calibration curve spanning five orders of magnitude (built from an in vitro transcribed GAPDH clone), and the triplicate means were divided by the value of a positive-control RNA (thymus) on the same plate to correct for plate-to-plate variability. Housekeeping-gene normalization was deliberately not used; instead, equal RNA input was ensured through fluorescence-based RNA quantitation, dilution of all samples to a common concentration, and automated liquid handling. The original authors did not apply a formal batch correction beyond this plate-level normalization. We used the publicly released, normalized expression values (Dataset S1), which were not log-scaled, applied a natural log transformation, and then centered each gene to have mean zero across its observed values. No further processing was performed.
Number of components
The greedy algorithm adds one component at a time and stops when a new component does not increase the ELBO; backfitting then updates all components jointly. With the EC covariance this selects 13 components on the full data. Table S4 shows the number selected across bootstrap resamples (§ 10.7.4).
| Number of components | 12 | 13 | 14 | 15 | 16 | 17 | |||||
| Count | 10 | 191 | 521 | 253 | 24 | 1 | |||||
| Matched position | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 13 |
| Count | 3 | 36 | 46 | 104 | 173 | 407 | 162 | 40 | 16 | 9 | 4 |
Table S4. Bootstrap results for the IFN data (): number of components selected, and position of the component matched to full-data component 7.
Responder Difference Significance Test
Let be the posterior mean score of component 7 for subject at visit , with the visit time in years from baseline, and the indicator that subject is a good responder. We fit the linear mixed model
where is a random subject intercept. Here is the difference between good and poor responders at baseline, and is the change in that difference per year; if , the baseline difference holds at every visit. The model uses all estimated scores, including those at missing visits, and is fit by REML with the nlme package.
The estimates are (SE 0.054, ) and per year (SE 0.0041, ). Good responders have higher component 7 scores, and we find no evidence that the difference changes over time. These -values treat the estimated scores as observed and do not account for component 7 having been chosen after inspecting the score trajectories; a Bonferroni correction over the 13 components gives for .
Stability across resamples
The EVBLM initialization is deterministic, so refitting the full data returns the same components; we therefore assess stability over resamples of subjects. We drew bootstrap samples of the 53 subjects with replacement and refit EVBLM with the same settings (EC covariance, greedy selection followed by backfitting). Because components with similar variance explained can change order between fits, the component matched to full-data component 7 is the one whose loading vector has the largest absolute correlation with the full-data loading of component 7. Its scores are multiplied by the sign of that correlation, and model (S22) is refit to them.
The median absolute loading correlation of the matched component is 0.84 (interquartile range 0.72 to 0.90), and it exceeds 0.8 in 61% of resamples. The matched component is at position 5 to 8 in 85% of resamples. Across resamples, has median 0.352 (2.5% and 97.5% quantiles 0.122 and 0.473) and in 96.8% of resamples; has median 0.000 (quantiles and 0.019) and in 10.6%.
References
- [1]Charles W Fox and Stephen J Roberts. A tutorial on variational Bayesian inference. Artificial intelligence review, 38(2):85–95, 2012.
- [2]S. Nakajima, M. Sugiyama, S. D. Babacan, and R. T omioka. Global analytic solution of fully-observed variational Bayesian matrix factorization. Journal of Machine Learning Research, 14(1):1–37, 2013.
- [3]W. Wang and M. Stephens. Empirical Bayes matrix factorization. Journal of Machine Learning Research, 22(120):1–40, 2021.
- [4]E. F. Lock. Empirical Bayes linked matrix decomposition. Machine Learning, 113:7451–7477, 2024.DOI
- [5]S. E. Baranzini, P. Mousavi, J. Rio, S. J. Caillier, A. Stillman, P. Villoslada, M. M. Wyatt, M. Comabella, L. D. Greller, R. Somogyi, X. Montalban, and J. O. Oksenberg. Transcription-based prediction of response to IFNβ using supervised computational methods. PloS Biology, 3(1):e2, 2004.pubmed.ncbi.nlm.nih.gov/15630474
- [6]Tamara G Kolda and Brett W Bader. Tensor decompositions and applications. SIAM review, 51(3):455–500, 2009.DOI
- [7]Leying Guan. Smooth and probabilistic PARAFAC model with auxiliary covariates. Journal of Computational and Graphical Statistics, 33(2):538–550, 2024.arxiv.org/abs/2104.05184
- [8]P. Shi, C. Martino, R. Han, and et al. TEMPTED: time-informed dimensionality reduction for longitudinal microbiome studies. Genome Biology, 25(317):891–921, 2024. doi: https://doi.org/10.1186/s13059-024-03453-x.
- [9]Z. Jiang, G. Li, and E. F. Lock. BAMITA: Bayesian multiple imputation for tensor arrays. Biostatistics, 26(1), 2025.arxiv.org/abs/2410.23412
- [10]Ajit P Singh and Geoffrey J Gordon. Relational learning via collective matrix factorization. In Proceedings of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 650–658, 2008.
- [11]Mehmet Gonen and Samuel Kaski. Kernelized Bayesian matrix factorization. IEEE Transactions on Pattern Analysis and Machine Intelligence, 36(10):2047–2060, 2014.DOI
- [12]Z. Wang, F. Zhang, C. Zheng, X. Hu, M. Cai, and C. Yang. MFAI : A scalable Bayesian matrix factorization approach to leveraging auxiliary information. Jour- nal of Computational and Graphical Statistics, 33(4):1339–1349, 2024. doi: https: //doi.org/10.1080/10618600.2024.2319160.DOI
- [13]X. An, Q. Yang, and P.M. Bentler. A latent factor linear mixed model for high-dimensional longitudinal data analysis. Statistics in Medicine, 32(24):4229–4239, 2013.DOI
- [14]Z. Lu and N.K. Chandra. A sparse factor model for clustering high-dimensional longitudinal data. Statistics in Medicine, 43(19):3633–3648, 2024.DOI
- [15]Qinxia Wang and Yuanjia Wang. Multilayer exponential family factor models for integrative analysis and learning disease progression. Biostatistics, 25(1):203–219, 2024.
- [16]C. M. Bishop. Bayesian PCA. In Advances in Neural Information Processing Systems 11, pages 382–388. MIT Press, 1999.
- [17]C. M. Bishop. Variational principal components. In Ninth International Conference on Artificial Neural Networks, 1(470):509–514, 1999.DOI
- [18]Q. Zhao, L. Zhang, and A. Cichocki. Bayesian CP factorization of incomplete tensors with automatic rank determination. IEEE Transactions on Pattern Analysis and Machine Intelligence, 37(9):1751–1763, 2015. doi: 10.1109/TPAMI.2015.2392756.DOI
- [19]S. Nakajima, R. Tomioka, M. Sugiyama, and S. D. Babacan. Condition for perfect dimensionality recovery by variational Bayesian PCA. Journal of Machine Learning Research, 16(114):3757–3811, 2015.
- [20]R. M. Neal. Probabilistic inference using Markov chain Monte Carlo methods. Technical Reports, Department of Computer Science:University of Toronto, 1993.
- [21]D. M. Blei, A. Kucukelbir, and J. D. McAuliffe. Variational inference: a review for statisticians. Journal of the American Statistical Association, 112(518):859–877, 2017.
- [22]Y. Yang, P. Carbonetto, D. Gerard, and M. Stephens. Improved Methods for Empirical Bayes Multivariate Multiple Testing and Effect Size Estimation. arXiv preprint arXiv: 2406.08784, 2024.arxiv.org/abs/2406.08784
- [23]Patrik O. Hoyer. Non-negative matrix factorization with sparseness constraints. Journal of Machine Learning Research, 5:1457–1469, 2004. URL https://www.jmlr.org/papers/v5/hoyer04a.html.DOI
- [24]R. Mazumder, T. Hastie, and R. Tibshirani. Spectral regularization algorithms for learning large incomplete matrices. Journal of Machine Learning Research, 11(80):2287–2322, 2010.pubmed.ncbi.nlm.nih.gov/21552465
- [25]D. J. Stekhoven and P Bühlmann. Missforest: non-parametric missing value imputation for mixed-type data. Bioinformatics, 28(1):112–128, 2012.
- [26]B. Guo, L. E. Eberly, P. G. Henry, C. Lenglet, and E. F. Lock. Multiway sparse distance weighted discrimination. Journal of Computational and Graphical Statistics, 32(2):730–743, 2022. doi: https://doi.org/10.1080/10618600.2022.2099404.
- [27]John M Elizarraras, Yuxing Liao, Zhiaο Shi, Qian Zhu, Alexander R Pico, and Bing Zhang. Webgestalt 2024: faster gene set analysis and new support for metabolomics and multi-omics. Nucleic acids research, 52(W1):W415–W421, 2024.DOI
- [28]Angele Cucci, Pierangelo Barbero, Marinella Clerico, Bruno Ferrero, Elisabetta Versino, Giulia Contessa, Stefania Demercanti, Emanuela Viglietta, Alessandra Di Liberto, Alessandra Giai Vai, et al. Pro-inflammatory cytokine and chemokine mrna blood level in multiple sclerosis is related to treatment response and interferon-beta dose. Journal of Neuroimmunology, 226(1-2):150–157, 2010.
- [29]Nellie Georgiou-Karistianis, Louise A. Corben, Eric F. Lock, Helena Bujalka, Isaac Adanyeguh, Manuela Corti, Dinesh K. Deelchand, Martin B. Delatycki, Imis Dogan, Jennifer Farmer, Marcondes C. França Jr., Anthony S. Gabay, William Gaetz, Ian H. Harding, James Joers, Michelle A. Lax, Jiakun Li, David R. Lynch, Thomas H. Mareci, Alberto R. M. Martinez, Massimo Pandolfo, Marina Papoutsi, Richard G. Parker, Kathrin Reetz, Thiago J.R. Rezende, Timothy P. Roberts, Sandro Romanzetti, David A. Rudko, Susmita Saha, Jörg B. Schulz, Sub H. Subramony, Veena G. Supramaniam, Christophe Lenglet, and Pierre-Gilles Henry. Neuroimaging biomarkers for friedreich ataxia: A cross-sectional analysis of the track-fa study. Annals of Neurology, 98(2): 386–397, 2025.DOI
- [30]H.S.H. Oh, D.Y. Urey, and L. et al. Karlsson. A cerebrospinal fluid synaptic protein biomarker for prediction of cognitive resilience versus decline in alzheimer’s disease. Nature Medicine, 31:1592–1603, 2025.
- [31]Sarah J Tabrizi, Rachael I Scahill, Alexandra Durr, Raymund AC Roos, Blair R Leavitt, Rebecca Jones, G Bernhard Landwehrmeyer, Nick C Fox, Hans Johnson, Stephen L Hicks, Christopher Kennard, David Craufurd, Chris Frost, Douglas R Langbehn, Ralf Reilmann, and Julie C Stout. Biological and clinical changes in premanifest and early stage huntington’s disease in the track-hd study: the 12-month longitudinal analysis. The Lancet Neurology, 10(1):31–42, 2011.DOI
- [32]B Wang and M Titterington. Inadequacy of interval estimates corresponding to variational bayesian approximations. Workshop on Artificial Intelligence and Statistics, pages 373–380, 2005.
- [33]Richard Turner, M. Sahani, D. Barber, Ali Cemgil, and S. Chiappa. Two problems with variational expectation maximisation for time-series models. Bayesian Time Series Models, pages 109–130, 2011.DOI
- [34]Sarah Samorodnitsky, Chris H. Wendt, and Eric F. Lock. Bayesian simultaneous factor- ization and prediction using multi-omic data. Computational Statistics & Data Analysis, 197, 2024.


