WorksMathematics
Advancing mixed-effects random forests for predictions based on multi-cohort data
Abstract
Pooling longitudinal cohorts can extend age coverage and improve long-term health predictions, but cohort heterogeneity complicate estimation and prediction. Mixed-effects machine learning (ME-ML) combines flexible machine learning methods with mixed-effect models to account for correlated observations, yet these ME-ML methods are limited in their applicability in complex settings such as multi-cohort data. We propose a multi-cohort mixed-effects random forest (MERFmulti-cohort) for improving predictions of individual health trajectories beyond the observed period. The method extends standard MERF through (i) better separation of population-level and subject-specific variation, (ii) flexibility in correcting systematic discrepancies in fixed-effects estimates to improve covariance parameter estimation, and (iii) flexible specification of random-effects and residual variance-covariance structures. We evaluate MERFmulti-cohort against linear regression, linear mixed models, random forests, and standard MERF in simulations and four real-data scenarios predicting BMI z-scores in children and adolescents. In simulations, MERFmulti-cohort improves covariance parameter estimation across several covariance structures while achieving prediction accuracy comparable to standard MERF. Predictive gains are most obvious when residual variances vary across study waves. In real-data applications, MERFmulti-cohort improves predictive accuracy relative to MERF by flexibly capturing complex patterns and accommodating cohort heterogeneity. Pooling similar cohorts covering extended age coverage can improve future individual-level predictions if the heterogeneity in multi-cohort data is adequately accounted for.
Introduction
Early prediction of chronic diseases is crucial for informing personalized interventions that promote health and well-being across the life course. Although observing developmental processes over time is ideal for understanding mechanisms of diseases with long latency periods, in practice many cohort studies cover only a limited age range, making it difficult to predict (markers of) diseases long before an outbreak.
Pooling multiple cohorts spanning different but overlapping stages in life could extend the observation periods [1, 2]. We anticipate that sharing information across cohorts may be an efficient way to improve the predictions of individual-level health trajectories. However, joint analysis of multi-cohort data inevitably adds complexity because of the heterogeneity across studies and the hierarchical data structure (repeated measurements nested within subjects, subjects nested within cohorts). So far, there is a striking lack of analytical techniques that can adequately predict health outcomes over extended age spans based on multi-cohort data.
In predictive modeling, machine learning (ML) methods such as random forests (RF) [3] are widely applied because of their flexibility in capturing non-linear relationships and interactions in complex data. However, they assume that observations are independent and identically distributed (i.i.d). In single-cohort settings, this assumption is already violated because repeated observations are nested within subjects, and subjects may be further nested within larger units (e.g., families or clinics). (Generalized) Linear Mixed Models (LMMs) are widely used in epidemiology to accommodate longitudinal, clustered data but rely on parametric assumptions and require predefined interactions among predictors. Mixed-effects machine learning (ME-ML) methods, which combine (Generalized) LMMs with ML, have shown improved prediction performance in single-cohort settings by leveraging the strengths of both approaches while mitigating limitations [4]. However, these ME-ML methods remain underexplored in multi-cohort settings with more complex hierarchical structures.
To fill this gap, we propose multi-cohort mixed-effects random forests (), a new approach that combines LMMs and RF to improve the predictions of individual-level health trajectories beyond the actual individual measurement period based on multi-cohort data. The novelty of our method lies in extending MERF to accommodate complex covariance structures to allow more flexibility in modeling multi-cohort data.
The structure of this paper is as follows. In § 2, we present a motivating example. In § 3, we introduce the general ME-ML framework and our approach. In § 4, we present simulation studies that assess the performance of the proposed method. In § 5, we describes the experimental setup for illustrating our approach using a real-data example of predicting BMI z-scores in children based on either single-cohort or multi-cohort data. § 6 presents the empirical findings and interpretation of the results. § 7 provides a discussion and conclusions.
Motivating Example
For illustration, we consider the task of predicting health outcomes (e.g., BMI z-scores) at future ages in children from information collected in cohort studies, although the proposed approach is more generally applicable to longitudinal health outcomes across later life stages. Suppose we have one target cohort and aim to predict health outcomes for its subjects at ages beyond the period covered in that cohort. We anticipate that augmenting the training data of the target cohort with data from other cohorts covering extended age spans could improve the prediction of individual-level health outcome trajectories in the target cohort. Therefore, we aim to investigate whether the accuracy of individual-level health outcome predictions can be improved by complementing data from the target cohort with data from other cohorts, particularly in cases where the target cohort is limited in the age range covered.
One challenge when working with multi-cohort data is measurement heterogeneity, which arises when different measurement tools are used in different studies or even across waves within the same study [2]. Although the variables are often harmonized using simple algorithmic transformations (e.g., combining categories, recoding), this does not fully resolve the measurement differences, which still need to be considered in modeling. Additionally, multi-cohort data have at least three hierarchical levels (i.e., repeated measurements nested within subjects, and subjects nested within cohorts). The hierarchical structure can be even more complex when additional units, such as clinics or families, exist between the subject and cohort levels. Such multi-level hierarchical structure introduces multiple sources of variation that need to be accounted for in the analysis. Therefore, advanced analytical techniques are required to accommodate the complexities of multi-cohort data.
Methods
Mixed-effects Models
(Generalized) Linear Mixed Models (LMMs) were developed for data with correlated observations. For continuous outcomes, they can be formulated as [5]:
where is the vector (vector of BMI z-scores of subject in our motivating example), is the number of measurements taken on subject , is the matrix of predictors collected on subject , is the vector of fixed effects parameters, is a design matrix, is a vector of unobserved random effects for subject with , and is the vector of normally distributed random errors with .
By choosing suitable covariance structures and for the residuals and random effects, LMMs can account for dependencies in correlated data.
Mixed-effects Machine Learning (ME-ML)
ME-ML approaches are built on the idea of integrating the random effects structure of LMMs into flexible ML methods and are typically formulated as [4, 6]:
The fixed effects part in LMMs is replaced by a more general unknown function to capture complex patterns. In theory, the unknown function could be modeled by any ML approach. ME-ML approaches typically alternate between estimating the fixed effects with a ML model, and estimating the random effects with a conventional LMM until a certain prespecified convergence criterion is reached.
Mixed-effects Random Forests (MERF)
Among the broad class of ME-ML methods, we focus particularly on MERF because RFs are applicable to a wide range of prediction problems and are generally recognized for their accuracy with few parameters to tune [7]. Comprehensive reviews of existing MERF approaches are provided by Hu and Szymczak [8] and Capitaine et al. [9]. In general, most MERF approaches follow the formulation [10]:
where denotes the RF for the fixed effects, is a design matrix, is a vector of unobserved random effects for subject with , and is the vector of normally distributed random errors with . MERF estimates the fixed effects using a random forest and alternates between estimating the fixed effects and predicting the random effects with a conventional LMM. In the first iteration, random effects are initialized to be 0. At iteration (), the RF is fitted to the outcome after removing the current estimated random contribution
giving
The predictions from are subsequently treated as the fixed effects contribution in the LMM:
This formulation has two limitations which make it too restrictive in multi-cohort settings.
First, RF could capture the pattern of the true fixed-effects function well, but systematic discrepancies in location and/or scale may still remain in the fixed effect predictions. Let denote the true fixed effect function for measurement of subject in cohort . The true data-generating model could be written as
At each MERF iteration, the RF predictions enter the LMM directly as the fixed effect component, so that the LMM is fitted to the residuals after subtracting from the outcome. The fixed effect function estimation error is
Then the RF residual is
Hence, unless is negligible, the RF residuals which are subsequently used to estimate the random effect and residual covariance structures contain not only the intended components , but also systematic error in the estimated fixed-effect function .
When data from multiple cohorts are pooled, this issue can be particularly relevant for estimating parameters of the random effects and residuals variance-covariance matrix (i.e., and ). For example, suppose cohort 1 covers ages 5-15 years and cohort 2 ages 6-25 years. The pooled sample then contains more information in the overlapping age range 6-15 years than at the upper end of the age distribution (>15 years), where only cohort 2 contributes. If the true regression function varies nonlinearly with age, the fixed-function estimation error may therefore differ across the age range. In such a setting we may have, for example, for measurements in cohort 1 with more age support, but deviates from 0 systematically for later-age measurements in cohort 2. The fixed effect function estimation error then has both cohort and longitudinal structure which may contaminate the estimation of the and .
Second, previous MERF approaches are limited in the flexibility of their covariance structures for and since they were mainly applied to settings with rather simple correlation structures (e.g., clustered data, single cohorts). Most existing approaches assume homogeneity of the within-subject residual variance and assume that no serial correlation exists between the within-subject residuals (i.e., ). As mentioned in our motivating example, when working with multi-cohort data, the measurement heterogeneity should be considered in modeling. Additionally, empirical studies [11, 12, 13] suggest that serial correlation sometimes remains even after accounting for random intercepts/slopes. In such cases, the assumption of no correlation between the within-subject residuals is too restrictive and a more flexible correlation structure for the residuals is required.
To adapt MERF to settings requiring more complex variance-covariance structures, Capitaine et al. [9] proposed stochastic mixed effects random forests (SMERF) and stochastic random effects expectation maximization forests (SREEMforest). Both SMERF and SREEMforest extend MERF by incorporating a stochastic serial correlation effects which allows the covariance structure to vary over time:
where is a stochastic process (e.g., Ornstein-Uhlenbeck process) used to model serial correlation for individual .
While useful in many longitudinal settings, the MERF approach suggested by Capitaine et al. [9] still presents methodological limitations that restrict its applicability to multi-cohort data. The stochastic process requires a sufficient number of measurements per subject to obtain stable estimates. In many cohort studies, however, each subject has only a limited number of measurements despite large sample sizes. Additionally, the estimated parameters in the variance-covariance matrix for random effects are assumed to be constant across all subjects (i.e., ) in the MERF approach by Capitaine et al. [9]. In multi-cohort settings, some cohorts might have greater between-subject variability when the sample is from a more heterogeneous population. In such cases, ignoring the heterogeneity in the variance-covariance matrix will either overestimate between-subject variability in the less heterogeneous cohorts or underestimate it in the more heterogeneous cohorts, which can lead to poor prediction of individual-level outcome trajectories [14].
Mixed-effects Random Forest for Multi-cohort Data (MERF multi-cohort)
Our MERF generalizes MERF in three respects: (i) we use subject-level rather than observation-level out-of-bag (OOB) prediction to reduce the extent to which subject-specific variation is absorbed by the RF; (ii) we relax the assumption that the RF estimate enters the mixed-effects model directly with unit coefficient by introducing an affine calibration of the fixed effect contribution; (iii) we allow more flexible variance–covariance structures for both random effects and residuals estimated as parts of the LMM.
Subject-level cross-fitting of the RF component
In standard RF with repeated measurements, an observation can be OOB for a particular tree while other observations from the same subject are still in-bag, which can make the observation-level OOB prediction error too optimistic for longitudinal data [8].
To reduce the subject-specific variation absorbed by the RF, we construct RF predictions using -fold cross-fitting at the subject level. are mutually exclusive sets of subjects in the training data. For a subject , the RF prediction is obtained from a forest trained using subjects outside .
In this way, no observation from subject contributes to the RF used to construct that subject’s fixed effect prediction, which provides a better separation between population-level variations used to estimate the fixed effect component and the subject-specific variations subsequently estimated through the random effects.
Calibrated fixed effect component
At iteration , let
denote the RF predictions for subject . We propose the MERF approach with calibrated fixed effect component
where
Conditional on a given RF prediction, the original MERF is nested within the proposed model as the restricted case when and .
The calibration does not introduce additional nonlinear structure, it only provides more flexibility in correcting the systematic location and scale discrepancy in the RF estimate.
Flexible random effect and residual covariance structures
To account for between-cohort variability, cohort-level random effects can be specified in the LMM to directly model the hierarchical structure (measurement nested within subject and subject nested within cohort ):
where denotes the cohort containing subject . is the cohort-level random effects vector, is the subject-level random effects vector, is the within-subject error.
To account for variation across cohorts in the magnitude and structure of between-subject heterogeneity, for cohorts with possibly different random effect structures per cohort (e.g., random intercept in one cohort, random intercept and random slope of age in other cohorts), we use an indicator to denote subject in cohort . Let denote the design vector associated with the random effects for measurement of subject and is the vector of random effects for subject in cohort . The subject-level random effects component for subject in cohort is:
where
This allows each cohort to have its own set of parameters in the variance-covariance matrix for subject-level random effects. This way, the estimated variance parameters are not constrained to be equal, and different random effects structures can also be specified for each cohort. The random effects for different cohorts are uncorrelated and therefore the variance-covariance matrix of subject-level random effects is block-diagonal across cohorts:
The choices are not restricted to the specification presented here. In principle, more complex hierarchical or cross-classified data structures could also be accommodated through the specification of the random effects structure depending on the specific data settings.
To account for measurement heterogeneity across different study waves and the within-subject temporal correlation not captured by random effects, we assume the residual variance-covariance matrix to be
where
Here () denotes the study wave associated with observation of subject , and denotes the residual variance for wave . Therefore contains the study wave-specific residual standard deviations, and is a correlation matrix with continuous autoregressive model of order 1 (CAR(1)) structure:
where is the correlation parameter which indicates how quickly correlation to decrease as time interval between measurements increases. This allows the correlation between two observations decreases in magnitude as the time interval increases and accommodates the irregular measurement intervals typical for multi-cohort data (since measurement schedules across cohorts are not aligned in general).
The greater flexibility in the specification of the random effects structure and residual variance–covariance structure helps to better capture between- and within-subject variation in multi-cohort data.
Implementation of MERF multi-cohort
The proposed MERF algorithm is shown in Table 1. The random effects are initialized to be 0 at the beginning since both fixed effects and random effects are unknown. The adjusted outcome is constructed by subtracting the current random effects prediction from the original outcome, which is intended to contain mainly the population-level variation for the RF to capture. As detailed in 3.4.1, we fit RF using subject-level folds to the adjusted outcome and estimate the fixed effect component using subject-level OOB RF predictions.
| Algorithm: Multi-cohort Mixed-effects Random Forest | |
| Input | Longitudinal data |
| Output | Estimated random forest , calibration parameters and , covariance parameter estimates , and random-effect predictions |
| Initialize | |
| Iterate through the following steps until converge:<br><br>(a) Construct the adjusted outcome<br><br><br><br>(b) Estimate the fixed effect component using subject-level cross-fitting<br>For each subject-level fold , fit RF to the adjusted outcomes using subjects outside fold . For observations belonging to fold , obtain .<br><br>(c) Fit the LMM<br><br>where<br><br>with<br><br>estimate , , and the covariance parameters .<br>(d) Update the random effect predictions<br><br>where<br><br>(e) Assess convergence<br>Compute the relative change in the log-likelihood<br><br>Stop when<br><br>or when the prespecified maximum number of iterations is reached. |
Table 1. MERF algorithm
Given RF predictions, calibration parameters ( and ), and the covariance parameters contained in and are jointly estimated within the LMM. Maximum likelihood rather than restricted maximum likelihood is used during the iterative procedure because the fixed effect components changes across iterations. The random effects predictions are subsequently updated based on the LMM parameter estimates. This provides greater flexibility in the specification of the random effects structure and residual variance–covariance structure. In principle, a wide range of variance–covariance structures typical of LMMs can be considered in MERF depending on the data characteristics and the underlying scientific question. Additionally, the likelihood evaluated during the LMM fit is used to monitor iteration to iteration stabilization. At each iteration, denotes the log-likelihood of the fitted LMM evaluated at the estimated coefficients. The relative change in the log-likelihood is then calculated to assess convergence. The alternation between estimating fixed effects by RF and predicting random effects by LMM continues until the relative change in the log-likelihood is less than a prespecified tolerance value or the prespecified maximum number of iterations is reached. In similar EM-type alternating procedures for ME-ML [[15, 16]], tolerance values between and are used to track the change of absolute/relative change in the log-likelihood to balance stability and efficiency. The specific tolerance value used as default in our implementation is with maximum 25 iterations. The R code implementing the proposed method is available in the online Supplementary Material.
Prediction
After convergence, fixed effect predictions for new subjects (i.e., subjects with no previous observations seen during model training) are obtained from the final RF and calibration parameters,
When only subject-level random effects are included, for subject with previous observations included in the training set, the prediction consists of the fixed effect predictions and the BLUP of the subject-specific random effects:
When cohort-level random effects are included in addition to subject-level random effects, for subject with previous observations included in the training set, the prediction additionally incorporates the BLUP of the cohort-specific random effects:
where denotes the cohort k containing subject .
Simulation Study
Simulation Design
In Section 5, we apply the proposed method to predict BMI z-score trajectories because BMI is easy to measure and widely available in many cohort studies. However, the proposed MERF could also be applied to other health outcomes that are more difficult or costly to measure, where leveraging previous measurements to obtain future Notes: denotes centered age; denotes the cohort k containing subject i; denotes subject-level random effects; and denotes cohort-level random effects. denotes the subject-level random-effects variance–covariance matrix; denotes a cohort-specific subject-level random-effects variance–covariance matrix; and and denote the cohort- and subject-level random-effects variance–covariance matrices, respectively. The common random intercept standard deviation in the simulation study is . The cohort-specific random intercept standard deviations are and for cohort 1 and cohort 2. The cohort-level random intercept standard deviation is .
predictions may be particularly beneficial. For example, measurement of fasting insulin requires blood collection under fasting status which is challenging in children.
We simulated longitudinal data from two child cohorts, with repeated measurements of fasting insulin concentrations on the natural-log scale as outcome. To mimic the design of real multi-cohort data, the simulated two-cohort data have:
different design features (e.g., number of subjects, age coverage at baseline, measurement-wave schedules);
irregular actual measurement times around planned measurement-waves;
different numbers of repeated measurements within cohorts due to dropout;
differences in covariate distribution.
For subject in cohort at measurement , the outcome (fasting insulin concentration on the natural-log scale) was generated as
where denotes the age of subject from cohort at measurement , is a nonlinear fixed effects function of age and covariates (details provided in Appendix § 11), denotes cohort-level random effects, denotes subject-level random effects, and denotes the within-subject residual error.
As illustrated in Figure 1, our simulation study was conducted under different data-generating process (DGP) settings, with varying specifications of the random effects and residual structures, as shown in Table 2 and Table 3. The design features, cohort-specific covariate distributions, and the true fixed-effect function are kept identical across all DGP settings. For each DGP setting, 20 longitudinal datasets (each consisting of two cohorts and ) were simulated. Cohort consists of 8,000 subjects with baseline ages uniformly distributed between 4.5 and 7 years, while Cohort consists of 15,000 subjects with baseline ages uniformly distributed between 2 and 10 years. Four measurement waves are scheduled at 0, 3, 6, and 11 years after baseline for and at 0, 2, 6, and 13 years after baseline for . Within each simulated dataset, three training scenarios were constructed to assess prediction of outcomes after age 14 in the target cohort (). The test set consisted of 50% of observations at age 14 or older and was kept identical across the three training scenarios. In scenario 1, the training set included only observations from before age 14. In scenario 2, the training set additionally included the remaining 50% of the observations from at age 14 or older Note: Cohort C1 consists of 8,000 subjects with baseline ages uniformly distributed between 4.5 and 7 years, while Cohort C2 consists of 15,000 subjects with baseline ages uniformly distributed between 2 and 10 years. Four measurement waves are scheduled at 0, 3, 6, and 11 years after baseline for C1 and at 0, 2, 6, and 13 years after baseline for C2. Actual measurement ages are obtained by adding independent uniform deviations between −0.5 and 0.5 years to the scheduled ages and are rounded to one decimal place. In both cohorts subjects have a dropout probability of 0.30 at each follow-up wave.
| Random effects structure | Random effects contribution | Variance–covariance structure |
| Subject-level random intercept with the same variance across cohorts | ||
| Subject-level random intercept with cohort-specific variance | ||
| Random intercepts at both the cohort and subject levels |
Table 2. Random effects structures considered in different DGP settings

Figure 1. Simulation design
| Residual structure | Variance–covariance structure |
| Constant residual variance and conditionally independent residuals within subjects | |
| Study wave-specific residual variances and conditionally independent residuals within subjects | |
| Constant residual variance with continuous-time AR(1) correlation within subjects | |
| Study wave-specific residual variances with continuous-time AR(1) correlation within subjects |
Table 3. Notes: The constant residual variance in the simulation study is . () denotes the study wave associated with observation of subject , and denotes the residual variance for wave in cohort . In the simulation study, there are 4 study waves within each cohort . is the correlation parameter which indicates how quickly correlation decreases as time interval between measurements increases ( in the simulation study).
(after assigning observations to the test set). In scenario 3, the training set combined observations from C1 before age 14 with observations from the second cohort C2, thereby extending the age range being covered in the training data.
Within each training scenario, we considered baseline methods, linear mixed models (LMMs), and mixed-effects random forest (MERF) approaches. The baseline methods include prediction using the subject’s most recent observation or the mean of the subject’s previous observations, linear models with correctly specified or mis-specified fixed effects (LM and LM_misspec, respectively), and random forest (RF). Similarly, for the LMMs, we considered models with correctly specified or mis-specified fixed effects (LMM and LMM_misspec, respectively). For MERF, the standard MERF [10] assumed the same random-effects variance–covariance matrix for all subjects, i.e., , and independent residuals with constant variance, i.e., . In contrast, MERF allowed the random effects and residual variance–covariance structures to correspond to the true DGP.
We first evaluated these models under the DGP setting with the simplest variance–covariance structure, consisting of subject-level random intercepts with a common variance across cohorts and homoscedastic independent residuals. This setting is compatible with the covariance assumptions of standard MERF [10] and therefore provides a benchmark for assessing whether the additional flexibility of affects performance when a more complex variance–covariance structure is not required. We then considered DGP settings with increasingly complex variance–covariance structures to assess whether allowing more flexible random effect and residual structures improves prediction performance.
Prediction performance was evaluated by the mean absolute error (MAE) on the same test set within each simulation iteration. For each method and training scenario, we report the mean of the test-set MAE across 20 simulation iterations together with its Monte Carlo standard error (MCSE). Since the true covariance parameters are known under the simulation DGP, we additionally examined recovery of random effects (co)variance and residual variance. Recovery of covariance parameters are evaluated using the mean of the parameter estimates across 20 simulation iterations together with its Monte Carlo standard error (MCSE).
Simulation Results
Under the simple DGP setting
The simple DGP provides a setting in which the standard MERF covariance assumptions are met. Table 4 reports the mean and SD of the test-set MAE, together with the corresponding Monte Carlo standard error (MCSE) in 20 simulation iterations, for all candidate methods across three training scenarios. Table 5 summarizes parameter recovery of the subject-level random intercept standard deviation and residual variance for the models incorporating random effects.
| Method | Scenario 1 mean (MCSE) | Scenario 2 mean (MCSE) | Scenario 3 mean (MCSE) |
| Mean | 1.068 (0.006) | 1.068 (0.006) | 1.068 (0.006) |
| Previous | 1.323 (0.007) | 1.323 (0.007) | 1.323 (0.007) |
| LM | 0.911 (0.003) | 0.910 (0.003) | 0.909 (0.003) |
| 0.989 (0.003) | 0.981 (0.003) | 0.977 (0.003) | |
| RF | 0.933 (0.004) | 0.917 (0.003) | 0.911 (0.003) |
| LMM | 0.647 (0.003) | 0.645 (0.003) | 0.645 (0.003) |
| 0.764 (0.004) | 0.754 (0.004) | 0.750 (0.004) | |
| MERF | 0.691 (0.004) | 0.660 (0.003) | 0.650 (0.003) |
| 0.696 (0.004) | 0.663 (0.003) | 0.651 (0.002) |
Table 4. Prediction performance under the simple DGP across the three training scenarios. Notes: mean denotes the mean test-set mean absolute error (MAE) across 20 simulation iterations, with Monte Carlo standard errors (MCSE) in parentheses.
| Parameter | Truth | Method | Scenario 1 mean (MCSE) | Scenario 2 mean (MCSE) | Scenario 3 mean (MCSE) |
| 0.800 | LMM | 0.800 (0.004) | 0.800 (0.004) | 0.800 (0.004) | |
| 0.800 | 0.801 (0.004) | 0.801 (0.003) | 0.799 (0.002) | ||
| 0.800 | MERF | 0.756 (0.004) | 0.769 (0.005) | 0.776 (0.003) | |
| 0.800 | 0.801 (0.004) | 0.802 (0.004) | 0.804 (0.002) | ||
| 0.490 | LMM | 0.495 (0.002) | 0.496 (0.002) | 0.495 (0.001) | |
| 0.490 | 0.709 (0.002) | 0.711 (0.002) | 0.822 (0.002) | ||
| 0.490 | MERF | 0.554 (0.003) | 0.544 (0.004) | 0.532 (0.002) | |
| 0.490 | 0.522 (0.002) | 0.522 (0.002) | 0.520 (0.001) |
Table 5. Recovery of variance components under the simple DGP. Notes: mean and MCSE denote the mean of the parameter estimates across 20 simulation iterations and corresponding Monte Carlo standard error.
Under the simple DGP, shows predictive performance comparable to that of standard MERF across all three training scenarios. Within each training scenario, the differences in mean test-set MAEs between and MERF are small relative to the between-repetition variation and decreased as the amount of training information increased.
A clearer difference is observed in the recovery of the covariance parameters. Standard MERF systematically underestimates the subject-level random-intercept standard deviation compared with the true value. In contrast, provides estimates closer to the truth. Similarly, standard MERF overestimates the residual variance across the three scenarios, whereas the corresponding estimates from are consistently closer to the true value. These results suggest that the proposed estimation procedure with subject-level cross-fitting and affine calibration of fixed effect improves the separation between the fixed effect component and the stochastic components, while retaining predictive performance comparable to standard MERF.
The correctly specified LMM provides the best predictive performance and the most accurate recovery of the covariance parameters. This is expected because the functional form of the fixed effects used to generate the data is explicitly specified as fixed effects in LMM. However, the true fixed-effects function which may contain complex nonlinearities and interactions are unknown in practice. This limitation is illustrated by , which shows larger prediction error than across all three training scenarios. Although recovers the random intercept standard deviation well, it overestimates the residual variance across three scenarios. This shows that mis specification of the fixed effect component can leave population-level variation unexplained, which may subsequently be attributed to the residual component.
Under more complex DGP settings
We next evaluate the methods under DGP settings with increasingly complex between- and within-subject covariance structures, including cohort-specific random-effects variances, cohort-level random effects, study wave-specific residual variances, and continuous-time AR(1) residual correlation. These settings were used to assess whether prediction performance and estimation of the covariance parameters can be improved by when the covariance assumptions in standard MERF are violated.
Under the simple DGP, standard MERF [10] was implemented using the LongituRF package [17]. The improved recovery of the covariance parameters by relative to standard MERF [10] is consistent with the aim of the proposed subject-level cross-fitting and affine calibration procedures. Under the more complex DGP settings, to assess whether allowing more flexible variance-covariance structures improves covariance parameter recovery, we compared with a MERF fitted using the same proposed estimation procedure, including subject-level cross-fitting and affine calibration, but retaining the same random effects and residual covariance structures as standard MERF. Thus, under more complex DGP settings, and MERF differed only in their variance-covariance structure specification, allowing the effect of flexibility in the specification of the random effects structure and residual variance–covariance structure to be assessed.
Prediction performance under the more complex DGP settings is summarized in Table 6. Table 7 and Table 8 summarize the recovery of variance-covariance parameters across the three training scenarios under different DGP settings with complex random effects structures and residual structures respectively.
| DGP setting | Method | Scenario 1 mean (MCSE) | Scenario 2 mean (MCSE) | Scenario 3 mean (MCSE) |
| Random intercepts at both the cohort and subject levels; constant residual variance and conditionally independent residuals within subjects | Mean | 1.057 (0.004) | 1.057 (0.004) | 1.057 (0.004) |
| Random intercepts at both the cohort and subject levels; constant residual variance and conditionally independent residuals within subjects | Previous | 1.306 (0.007) | 1.306 (0.007) | 1.306 (0.007) |
| Random intercepts at both the cohort and subject levels; constant residual variance and conditionally independent residuals within subjects | LM | 0.912 (0.005) | 0.911 (0.005) | 0.971 (0.018) |
| Random intercepts at both the cohort and subject levels; constant residual variance and conditionally independent residuals within subjects | LM | 0.983 (0.006) | 0.975 (0.005) | 1.056 (0.024) |
| Random intercepts at both the cohort and subject levels; constant residual variance and conditionally independent residuals within subjects | RF | 0.933 (0.004) | 0.913 (0.005) | 1.031 (0.035) |
| Random intercepts at both the cohort and subject levels; constant residual variance and conditionally independent residuals within subjects | LMM | only one cohort | only one cohort | 0.640 (0.003) |
| Random intercepts at both the cohort and subject levels; constant residual variance and conditionally independent residuals within subjects | LMM | only one cohort | only one cohort | 0.742 (0.004) |
| Random intercepts at both the cohort and subject levels; constant residual variance and conditionally independent residuals within subjects | MERF | 0.685 (0.003) | 0.658 (0.003) | 0.651 (0.003) |
| Random intercepts at both the cohort and subject levels; constant residual variance and conditionally independent residuals within subjects | MERF | only one cohort | only one cohort | 0.645 (0.003) |
| Subject-level random intercepts with cohort-specific variance; constant residual variance and conditionally independent residuals within subjects | Mean | 1.051 (0.008) | 1.051 (0.008) | 1.051 (0.008) |
| Subject-level random intercepts with cohort-specific variance; constant residual variance and conditionally independent residuals within subjects | Previous | 1.310 (0.008) | 1.310 (0.008) | 1.310 (0.008) |
| Subject-level random intercepts with cohort-specific variance; constant residual variance and conditionally independent residuals within subjects | LM | 0.911 (0.004) | 0.910 (0.004) | 0.910 (0.004) |
| Subject-level random intercepts with cohort-specific variance; constant residual variance and conditionally independent residuals within subjects | LM | 0.987 (0.004) | 0.978 (0.004) | 0.975 (0.004) |
| Subject-level random intercepts with cohort-specific variance; constant residual variance and conditionally independent residuals within subjects | RF | 0.934 (0.005) | 0.916 (0.005) | 0.914 (0.005) |
| Subject-level random intercepts with cohort-specific variance; constant residual variance and conditionally independent residuals within subjects | LMM | only one cohort | only one cohort | 0.636 (0.003) |
| Subject-level random intercepts with cohort-specific variance; constant residual variance and conditionally independent residuals within subjects | LMM | only one cohort | only one cohort | 0.742 (0.004) |
| Subject-level random intercepts with cohort-specific variance; constant residual variance and conditionally independent residuals within subjects | MERF | 0.690 (0.005) | 0.658 (0.004) | 0.647 (0.003) |
| Subject-level random intercepts with cohort-specific variance; constant residual variance and conditionally independent residuals within subjects | MERF | only one cohort | only one cohort | 0.646 (0.003) |
| Subject-level random intercepts; study wave-specific residual variances and conditionally independent residuals within subjects | Mean | 0.887 (0.005) | 0.887 (0.005) | 0.887 (0.005) |
| Subject-level random intercepts; study wave-specific residual variances and conditionally independent residuals within subjects | Previous | 1.070 (0.007) | 1.070 (0.007) | 1.070 (0.007) |
| Subject-level random intercepts; study wave-specific residual variances and conditionally independent residuals within subjects | LM | 0.764 (0.003) | 0.763 (0.003) | 0.762 (0.003) |
| Subject-level random intercepts; study wave-specific residual variances and conditionally independent residuals within subjects | LM | 0.848 (0.004) | 0.837 (0.004) | 0.834 (0.004) |
| Subject-level random intercepts; study wave-specific residual variances and conditionally independent residuals within subjects | RF | 0.778 (0.004) | 0.763 (0.004) | 0.760 (0.004) |
| Subject-level random intercepts; study wave-specific residual variances and conditionally independent residuals within subjects | LMM | 0.328 (0.002) | 0.323 (0.002) | 0.326 (0.002) |
| Subject-level random intercepts; study wave-specific residual variances and conditionally independent residuals within subjects | LMM | 0.530 (0.003) | 0.500 (0.003) | 0.493 (0.003) |
| Subject-level random intercepts; study wave-specific residual variances and conditionally independent residuals within subjects | MERF | 0.426 (0.004) | 0.381 (0.002) | 0.363 (0.002) |
| Subject-level random intercepts; study wave-specific residual variances and conditionally independent residuals within subjects | MERF | 0.403 (0.004) | 0.355 (0.002) | 0.333 (0.002) |
| Subject-level random intercepts; constant residual variance with continuous-time AR(1) correlation within subjects | Mean | 1.068 (0.004) | 1.068 (0.004) | 1.068 (0.004) |
| Subject-level random intercepts; constant residual variance with continuous-time AR(1) correlation within subjects | Previous | 1.316 (0.006) | 1.316 (0.006) | 1.316 (0.006) |
| Subject-level random intercepts; constant residual variance with continuous-time AR(1) correlation within subjects | LM | 0.916 (0.005) | 0.915 (0.005) | 0.914 (0.005) |
| Subject-level random intercepts; constant residual variance with continuous-time AR(1) correlation within subjects | LM | 0.994 (0.005) | 0.985 (0.005) | 0.983 (0.005) |
| Subject-level random intercepts; constant residual variance with continuous-time AR(1) correlation within subjects | RF | 0.940 (0.005) | 0.920 (0.005) | 0.915 (0.005) |
| Subject-level random intercepts; constant residual variance with continuous-time AR(1) correlation within subjects | LMM | 0.650 (0.003) | 0.649 (0.003) | 0.648 (0.003) |
| Subject-level random intercepts; constant residual variance with continuous-time AR(1) correlation within subjects | LMM | 0.769 (0.003) | 0.760 (0.003) | 0.754 (0.003) |
| Subject-level random intercepts; constant residual variance with continuous-time AR(1) correlation within subjects | MERF | 0.702 (0.005) | 0.669 (0.002) | 0.659 (0.003) |
| Subject-level random intercepts; constant residual variance with continuous-time AR(1) correlation within subjects | MERF | 0.700 (0.005) | 0.668 (0.003) | 0.658 (0.003) |
Table 6. Prediction performance under the more complex DGP settings across the three training scenarios. Notes: values are the mean test-set mean absolute error (MAE) across 20 simulation repetitions, with the Monte Carlo standard error (MCSE) in parentheses. Cohort level random effects and cohort-specific random effect variances could only be estimated with training data from 2 cohorts.
| DGP setting | Parameter | Truth | Method | Scenario 1 mean (MCSE) | Scenario 2 mean (MCSE) | Scenario 3 mean (MCSE) |
| Random intercepts at both the cohort and subject levels; constant residual variance and conditionally independent residuals within subjects | 0.200 | LMM | only one cohort | only one cohort | 0.105 (0.037) | |
| only one cohort | only one cohort | 0.106 (0.035) | ||||
| MERF | – | – | – | |||
| MERF_{\mathrm{multi\mathchar`-cohort}} | only one cohort | only one cohort | 0.104 (0.035) | |||
| 0.800 | LMM | only one cohort | only one cohort | 0.800 (0.002) | ||
| only one cohort | only one cohort | 0.791 (0.003) | ||||
| MERF | 0.801 (0.005) | 0.799 (0.005) | 0.885 (0.003) | |||
| MERF_{\mathrm{multi\mathchar`-cohort}} | only one cohort | only one cohort | 0.801 (0.002) | |||
| 0.490 | LMM | only one cohort | only one cohort | 0.495 (0.001) | ||
| only one cohort | only one cohort | 0.825 (0.002) | ||||
| MERF | 0.522 (0.002) | 0.523 (0.002) | 0.526 (0.001) | |||
| MERF_{\mathrm{multi\mathchar`-cohort}} | only one cohort | only one cohort | 0.526 (0.001) | |||
| Subject-level random intercepts with cohort-specific variance; constant residual variance and conditionally independent residuals within subjects | 0.800 | LMM | only one cohort | only one cohort | 0.804 (0.004) | |
| only one cohort | only one cohort | 0.755 (0.005) | ||||
| MERF | 0.806 (0.005) | 0.806 (0.005) | 1.260 (0.003) | |||
| MERF_{\mathrm{multi\mathchar`-cohort}} | only one cohort | only one cohort | 0.799 (0.004) | |||
| 1.500 | LMM | only one cohort | only one cohort | 1.500 (0.004) | ||
| only one cohort | only one cohort | 1.510 (0.004) | ||||
| MERF | 0.806 (0.005) | 0.806 (0.005) | 1.260 (0.003) | |||
| MERF_{\mathrm{multi\mathchar`-cohort}} | only one cohort | only one cohort | 1.510 (0.004) | |||
| 0.490 | LMM | only one cohort | only one cohort | 0.495 (0.001) | ||
| only one cohort | only one cohort | 0.825 (0.002) | ||||
| MERF | 0.522 (0.002) | 0.523 (0.002) | 0.524 (0.001) | |||
| MERF_{\mathrm{multi\mathchar`-cohort}} | only one cohort | only one cohort | 0.525 (0.001) |
Table 7. Recovery of variance-covariance parameters under complex random effects DGP settings across the three training scenarios
| DGP setting | Parameter | Truth | Method | Scenario 1 mean (MCSE) | Scenario 2 mean (MCSE) | Scenario 3 mean (MCSE) |
| Subject-level random intercepts; study wave-specific residual variances and conditionally independent residuals within subjects | 0.800 | LMM | 0.800 (0.004) | 0.800 (0.004) | 0.798 (0.002) | |
| LMM | 0.799 (0.004) | 0.800 (0.004) | 0.801 (0.002) | |||
| MERF | 0.881 (0.005) | 0.891 (0.005) | 0.851 (0.002) | |||
| MERF | 0.809 (0.004) | 0.811 (0.004) | 0.810 (0.002) | |||
| 0.640 | LMM | 0.641 (0.003) | 0.641 (0.003) | 0.642 (0.003) | ||
| LMM | 0.852 (0.006) | 0.849 (0.005) | 0.874 (0.006) | |||
| MERF | 0.351 (0.007) | 0.328 (0.006) | 0.260 (0.003) | |||
| MERF | 0.667 (0.003) | 0.668 (0.003) | 0.649 (0.004) | |||
| 0.160 | LMM | 0.160 (0.001) | 0.160 (0.001) | 0.160 (0.001) | ||
| LMM | 0.371 (0.004) | 0.367 (0.004) | 0.391 (0.004) | |||
| MERF | 0.351 (0.007) | 0.328 (0.006) | 0.260 (0.003) | |||
| MERF | 0.191 (0.001) | 0.189 (0.001) | 0.175 (0.001) | |||
| 0.090 | LMM | 0.092 (0.001) | 0.093 (0.001) | 0.095 (0.001) | ||
| LMM | 0.317 (0.003) | 0.330 (0.003) | 0.326 (0.004) | |||
| MERF | 0.351 (0.007) | 0.328 (0.006) | 0.260 (0.003) | |||
| MERF | 0.143 (0.002) | 0.135 (0.002) | 0.110 (0.001) | |||
| 0.090 | LMM | – | 0.111 (0.002) | – | ||
| LMM | – | 0.321 (0.006) | – | |||
| MERF | – | 0.328 (0.006) | – | |||
| MERF | – | 0.141 (0.003) | – | |||
| 0.250 | LMM | – | – | 0.248 (0.001) | ||
| LMM | – | – | 0.589 (0.003) | |||
| MERF | – | – | 0.260 (0.003) | |||
| MERF | – | – | 0.285 (0.002) | |||
| 0.250 | LMM | – | – | 0.254 (0.001) | ||
| LMM | – | – | 0.591 (0.003) | |||
| MERF | – | – | 0.260 (0.003) | |||
| MERF | – | – | 0.285 (0.002) | |||
| 0.090 | LMM | – | – | 0.107 (0.001) | ||
| LMM | – | – | 0.529 (0.004) | |||
| MERF | – | – | 0.260 (0.003) | |||
| MERF | – | – | 0.150 (0.002) | |||
| 0.090 | LMM | – | – | 0.099 (0.001) | ||
| LMM | – | – | 0.503 (0.004) | |||
| MERF | – | – | 0.260 (0.003) | |||
| MERF | – | – | 0.152 (0.002) | |||
| Subject-level random intercepts; constant residual variance with continuous-time AR(1) correlation within subjects | 0.800 | LMM | 0.808 (0.004) | 0.798 (0.004) | 0.802 (0.002) | |
| LMM | 0.805 (0.005) | 0.800 (0.004) | 0.789 (0.004) | |||
| MERF | 0.855 (0.004) | 0.847 (0.004) | 0.857 (0.003) | |||
| MERF | 0.807 (0.004) | 0.801 (0.004) | 0.805 (0.003) | |||
| 0.490 | LMM | 0.486 (0.002) | 0.496 (0.002) | 0.496 (0.001) | ||
| LMM | 0.701 (0.005) | 0.707 (0.003) | 0.836 (0.002) | |||
| MERF | 0.474 (0.001) | 0.483 (0.001) | 0.478 (0.001) | |||
| MERF | 0.501 (0.003) | 0.506 (0.002) | 0.506 (0.001) |
Table 8. Recovery of variance–covariance parameters under complex residual DGP settings across the three training scenarios.
Across the more complex DGP settings, the main advantage of over MERF is the improvement in estimating the variance-covariance component rather than lower prediction error.
When random intercepts at both the cohort and subject levels are present, the comparison between and MERF is available in Scenario 3 where the training data contains more than one cohort which allows the hierarchical structure to be estimated. Including a random effect for cluster membership is usually not recommended when there are very few clusters (e.g., fewer than five as a rule of thumb [18, 19, 20]) because the between-cluster variance is estimated from too little information, often leading to imprecise variance component estimates. In our setting with two-cohort data, recovers the subject-level random intercept variance as well as the correctly specified LMM. The cohort-level random intercept variance is poorly recovered in all models. This may also explain why incorporating cohort-level random-effect predictions resulted in no obvious improvement in predictive performance.
When the subject-level random intercept variability differed between cohorts, the advantage of the more flexible covariance specification is more apparent in estimating random effect variances than in improving prediction performance. In Scenario 3, MERF and achieved nearly identical predictive performance. , which allows the subject-level random-intercept variance to differ by cohort, provides the random intercept variance close to the true value. In contrast, MERF imposes a single common random-intercept variance and therefore removes information about how between-subject heterogeneity differs between cohorts.
The benefit of the more flexible covariance specification is more obvious when residual variances differed across study waves. This setting is relevant to multi-cohort data analyses because outcome measurements may differ across assay methods and laboratories, potentially introducing measurement heterogeneity across measurement occasions. In this setting, consistently achieves better predictive performance than MERF and provides estimates of the subject-level random intercept variance that are closer to the true value. When residual heteroscedasticity is ignored, MERF overestimates random intercept variance in all three scenarios. This indicates that part of the wave-specific residual variation may instead be attributed to between-subject heterogeneity.
Under continuous-time AR(1) residual correlation, and MERF show similar predictive performance, but provides estimates of the subject-level random intercept variance and residual variance closer to the true value. MERF assumes conditionally independent residuals and therefore does not consider the additional within-subject serial residual correlation. MERF overestimates the random intercept variance across scenarios, which suggests part of the unmodeled residual dependence being attributed to the random effect component.
Compared with the mis-specified LMM, consistently achieves better prediction performance in all complex DGP settings. Additionally, the mis-specified LMM tends to overestimate the residual variance, suggesting that population-level variation not captured by the fixed effects is partly retained in the stochastic component of the model. In contrast, allows the fixed effect function to be estimated flexibly without requiring nonlinearities and interactions to be specified in advance. This flexibility is particularly relevant in real-data applications, where the functional form of the fixed effect is generally unknown.
Although pooling cohorts increases sample size and extends age coverage, the additional cohort may also introduce heterogeneity in the population distribution and variance-covariance structure. When such heterogeneity is not adequately represented by the fitted model, the additional data may provide little benefit and, in some settings, result in poorer prediction performance. Under the DGP containing both cohort- and subject-level random intercepts, where the prediction errors of LM, , and RF increases in Scenario 3 after training data in Scenario 1 further incorporates data from the additional cohort. can use the additional cohort while explicitly modeling the hierarchical structure in multi-cohort data. Under the DGP with both cohort- and subject-level random intercepts, MERF achieves a mean MAE of 0.645 in Scenario 3, compared with 0.685 for MERF in Scenario 1. Similar patterns are observed under the other complex DGP settings, with MERF in Scenario 3 achieving better prediction performance than models trained under Scenario 1 and achieving similar performance as models trained under Scenario 2. MERF provides a framework for incorporating information from other cohort and extend observation period while allowing the random effects and residual variance-covariance structures to accommodate the heterogeneity introduced by the additional cohort.
Notes: values are the mean parameter estimates across 20 simulation repetitions, with Monte Carlo standard error (MCSE) in parentheses. denotes the cohort-level random-intercept standard deviation. and denote the subject-level random-intercept and random-slope standard deviations, respectively, in cohort . denotes the corresponding intercept–slope covariance. and denotes the residual variance. Entries are reported as "–" when the cohort-level random intercept standard deviation parameter is not estimated by the MERF because MERF does not support nested random effects.
Notes: values are the mean parameter estimates across 20 simulation repetitions, with Monte Carlo standard error (MCSE) in parentheses. denotes the subject-level random-intercept standard deviation, denotes the common residual variance, denotes the study wave-specific residual variance for wave in cohort . Entries are reported as "–" when the study wave-specific residual variance parameter is not estimated by the fitted model because training data do not include observation from wave in cohort .
Application to Real Data
Harmonized data of two children cohorts will be used as a real-data example to illustrate our approach. We will predict age- and sex-specific BMI z-scores [21] based on comprehensive information on demographics, lifestyle, and early-life factors. A detailed description of the data sources and preprocessing are provided in the online Supplementary Material A.
The target cohort in our example is the ABCD [22] cohort, which consists of 8266 children residing in Amsterdam who were followed up since pregnancy. Three follow-ups (W0, W1 and W2) took place at around the ages of 6 years, 12 years and 16 years, respectively. As second cohort we consider the multi-center population-based IDEFICS/I.Family cohort [23] covering eight European countries (Belgium, Cyprus, Estonia, Germany, Hungary, Italy, Spain, and Sweden). Compared with ABCD, the IDEFICS/I.Family study includes a larger sample (16229 children at the baseline wave W0) drawn from a more heterogeneous population. As shown in Figure 2, the age coverage in the ABCD study includes gaps, whereas IDEFICS/I.Family spans a broader age range. In the IDEFICS/I.Family cohort, the baseline examination wave (W0) included children aged 2.0–9.9 years. Three subsequent follow-up waves were conducted approximately 2 (W1), 6 (W2), and 13 (W3) years after baseline spanning data from childhood to early adulthood. These different cohort designs enable us to investigate the benefits of using multi-cohort rather than single-cohort data to enhance the accuracy of individual-level predictions, particularly in cases where single cohorts are limited in the age range covered.
Figure 2. Age coverage in IDEFICS/I.Family cohort and ABCD cohort
In our illustrative example, we predict BMI after age 14 for ABCD participants who have previous measurements before age 14. To assess the benefit of additional data from the second cohort, we design four scenarios of forecasting BMI z-score in participants seen in training using information from single-cohort data as shown in Figure 3 and multi-cohort data with additional measurements at later ages as shown in Figure 4.

Figure 3. Single-cohort scenarios

Figure 4. Multi-cohort scenarios
Scenario 1 mimics the common situation that longitudinal data are only available up to a certain age. ABCD observations are used only up to age 14 as training data. ABCD observations at ages older than 14 years (from 1730 subjects with measurements before age 14) are randomly split into validation (865 subjects) and test data (865 subjects). Scenario 2 represents the ideal situation that follow-up data taken at older ages from the same cohort are available in a subset. We randomly split the validation set from scenario 1 into five folds: each time 1/5 of the validation data is added to the training set while the other 4/5 is kept as validation data. In scenario 3, we augment the ABCD training set from scenario 1 with data from the IDEFICS/I.Family cohort, but to reduce heterogeneity we include only 3943 IDEFICS/I.Family participants from countries with similar demographic profile to the Netherlands (specifically, Germany and Belgium, considered “central EU”). In scenario 4, data from all IDEFICS/I.Family participants are added to the ABCD training set from scenario 1, which introduces greater heterogeneity in the training data.
Within each scenario, we consider the models listed in Table 9. For both LMM and , as the true DGP is unknown, we estimate different models with varying variance-covariance structures and select the one that best fits the validation data, using mean absolute error (MAE) as evaluation metric. In contrast, for standard MERF [10], we include the subject-level random effects and use the covariance structure assuming independent residuals within-subject and constant residual variance, because alternative structures are not supported by the available implementation. After selecting an appropriate structure, we compare the predictive performance of using the most recent outcome measurements before age 14 as predictions, LM, LMM, RF and standard MERF approach [10] against the proposed . The comparison is based on the test data that are held-out and kept the same across scenarios.
| Model | Description |
| Previous | For subject , the prediction at time , , is the latest available measurement . |
| LM | Linear model with . |
| LMM | Linear mixed model with as fixed effects, with a random intercept and random slope for age for each subject, implemented in the nlme package [24]. |
| RF | Random forest implemented in the ranger package [25], using the same set of features with default hyperparameters. |
| MERF | Mixed-effects random forest proposed by Hajjem et al. [10], implemented in the LongituRF package [17], using the same set of features and the same hyperparameters as RF. |
| MERF | Our proposed MERF, using the ranger package [25] for estimating the fixed-effects function and the nlme package [24] for estimating the random effects, with the same set of features and the same random-forest hyperparameters as RF. |
Table 9. Models considered for the evaluation and comparison
Results and Interpretation
The results of the model selection procedure for the variance–covariance structures are provided in the online Supplementary Material B, including the candidate structures evaluated and their corresponding performance on the validation data. The selected variance–covariance structure (with smallest mean absolute error on validation data) for each model are reported in the online Supplementary Material B.
Table 10 presents the performance of models without random effects (LM and RF), the standard mixed-effects model (LMM), previous MERF approach [10] and MERF, evaluated under four scenarios.
In scenario 1 and scenario 2, LMM outperforms LM, and similarly both MERF and MERF outperform RF. This suggests that in single cohort scenarios the prediction of future observations for a given subject is improved by considering the random effect associated with that subject.
Within each scenario under investigation, the LMM outperforms the LM, and similarly MERF achieves better predictive performance than RF and MERF, indicating that models accounting for a hierarchical structure in the data through more flexible covariance structures have better predictive performance than models that assume oversimplified covariance structures.
Across all four scenarios, the baseline method shows identical performance, as the test set is fixed and the baseline prediction relies only on the most recent observed outcome measurement for each subject. Except for the baseline method, prediction performance across all other models improves when adding future data from the same cohort. In both, scenarios 3 and 4, all models show lower prediction performance compared with scenario 2. However, in scenario 3 (training data complemented with data from a similar population), the decrease in predictive performance is minor.
Benefits of MERF multi-cohort in scenario with heterogeneous data
The advantage of MERF over MERF becomes more evident in multi-cohort scenarios, demonstrating the benefit of allowing more flexible covariance structures for random effects and residuals in multi-cohort settings, which are not supported in MERF.
MERF and LMM show comparable performance in single-cohort scenarios and in the multi-cohort scenario involving similar cohorts. While the predictive performance of MERF remains relatively stable across all scenarios, the advantage of MERF is most evident in scenario 4, where the additional cohort is more heterogeneous. This suggests that, while both MERF and LMM can capture complex correlation structures, MERF provides greater flexibility in modeling nonlinear relationships and complex interactions.
Benefits of additional data from external cohorts
While increasing the amount of training data is often expected to improve model performance, this improvement is not guaranteed when cohort heterogeneity is ignored. Our results suggest that the optimal scenario for predicting future BMI trajectories is to have observations at later ages from a subset of participants within the same cohort. However, in many epidemiological studies, measurements are often only available up to certain ages, which restricts the ability to predict trajectories at later ages. In our results, predictive performance of MERF decreases in scenarios 3 and 4 compared with scenario 2, but remains better than in scenario 1. This indicates that heterogeneity in multi-cohort data makes prediction more difficult, but, if the heterogeneity is adequately accounted for by the model, additional data from other cohorts may improve extrapolation to later ages by extending the observed period beyond the target cohort’s age range.
Despite these promising results, it should be noted that the observed differences in predictive performance across methods are relatively small, and the incremental gain over the baseline method from incorporating additional predictors and more complex modeling is limited. A possible explanation is that the current evaluation is based on two European child cohorts with moderate heterogeneity, and the application considers tabular data with a relatively small number of predictors and an outcome with limited temporal variability.
| Scenario | Previous | LM | LMM | RF | MERF | MERF |
| 1 | 0.729 | 0.687 | 0.635 | 0.681 | 0.645 | 0.635 |
| 2 | 0.729 | 0.652 | 0.585 | 0.663 | 0.626 | 0.605 |
| 3 | 0.729 | 0.655 | 0.598 | 0.664 | 0.660 | 0.613 |
| 4 | 0.729 | 0.742 | 0.704 | 0.663 | 0.675 | 0.619 |
Table 10. Model performance evaluated using mean absolute error on the test set across four scenarios
Scenario 1: Training data include ABCD observations only up to age 14 (ABCD).
Scenario 2: Training data from Scenario 1 are augmented with additional observations from ABCD subjects older than 14 years who are not included in the test set (ABCD).
Scenario 3: Training data from Scenario 1 are complemented with data from a similar population (ABCD + IDEFICS/I.Family).
Scenario 4: Training data from Scenario 1 are complemented with data from an additional cohort that differs from the target cohort in its demographic profile (ABCD + IDEFICS/I.Family).
Discussion and Conclusion
In this study, we extend the previous MERF approach [10] to multi-cohort settings and generalize MERF in three respects: (i) subject-level rather than observation-level out-of-bag (OOB) predictions are used as fixed effect estimates to better separate population-level and subject-specific variation; (ii) affine calibration of the fixed effect contribution provides more flexibility in correcting the systematic discrepancy in the fixed effect estimate to improve covariance estimation; (iii) flexible variance–covariance structures for both random effects and residuals are estimated as parts of the LMM.
The simulation results demonstrate that the proposed MERF retains predictive performance comparable to standard MERF under simple covariance structure, while providing more accurate estimates of the variance-covariance components. Improvements in predictive performance and covariance parameter estimation are observed in some settings with complex covariance structure. The benefits of allowing flexible covariance structures are more apparent under residual heteroscedasticity, which may arise in multi-cohort data from differences in measurement procedures and tools.
We demonstrate MERF for predicting individual-level BMI z-score trajectories using pooled data from two children cohorts. MERF provides greater flexibility in modeling the covariance structure in the analysis of complex multi-cohort data, which is limited in previous MERF approaches. At the same time, MERF enables flexible modeling of fixed effects with non-linear patterns and complex interactions. This combined flexibility in modeling both the random effects and fixed effects components allows MERF to better leverage larger and more heterogeneous training data.
Our proposed approach separates the estimation of fixed and random effects. As a result, our proposed approach is not restricted to RF and in principle could be extended to other ML methods. This separation also allows the use of established explainability methods for the ML component, such as variable importance scores used for RF by Ngufor et al. [4], which may help to retain interpretability despite the use of flexible ML models. Compared to standard MERF [10], our new approach could have more correct attribution of random effects and fixed effects, which may improve the interpretation. MERF is currently limited to continuous outcomes. Future work could extend the approach by integrating generalized mixed-effects models into RF, enabling the prediction of outcomes with non-normal distributions.
Existing studies do not provide a rigorous convergence analysis of the iterative ME-ML estimation algorithm itself. Instead of analyzing the iterative procedure directly, Capitaine et al. [9] derive a result on the asymptotic prediction bias of the fitted MERF predictor by expressing the prediction bias as a linear transformation of the RF estimation bias and then invoking the RF consistency result of Scornet et al. [26]. Similar to the approach of Capitaine et al. [9], we derive in the Appendix that the asymptotic prediction bias of the fitted MERF predictor is inherited from the RF component when the variance components are considered as known. However, extending the asymptotic result of Capitaine et al. [9] to the proposed MERF requires further investigation because existing RF consistency results do not directly apply to the present setting, which involves heterogeneous covariance structures across cohorts and potentially remaining temporal dependence after including the random effects.
Additionally, model evaluation in longitudinal prediction settings that leverage previous measurements to predict future outcomes for the same subjects is challenging. Standard cross-validation strategies are not straightforward in such settings because random partitioning of the data at the observation level may break temporal dependencies and lead to information leakage between training and validation sets, whereas partitioning at the subject level removes prior measurements of the same subjects from the training data. Therefore, model performance is evaluated based on their MAE calculated on a heldout test set that preserves the temporal structure of the observations. Future work could further investigate how to quantify the uncertainty of the estimated prediction performance in such settings, for example by adapting advanced techniques such as the approach proposed by Rink and Brannath [27] to these settings.
Overall, our study highlights the potential of ME-ML approaches for predicting individual-level health trajectories in multi-cohort studies by combining ML methods with an explicit modeling of complex hierarchical structures and cohort heterogeneity through flexible random effects and residual covariance structures. However, further evaluation is required to assess the performance of the proposed method in a wider range of data settings with varying numbers of cohorts, stronger heterogeneity across cohorts, outcomes showing greater temporal variability, stronger nonlinear relationships, and more complex data structures that cannot be easily modeled using LMM, where its advantages are expected to be more pronounced.
Acknowledgments
This work was funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) - 459360854 as part of the Research Unit "Lifespan AI: From Longitudinal Data to Lifespan Inference in Health” (DFG FOR 5347), University of Bremen (http://lifespanai.de).
Conflicts of Interest
The authors declare no conflicts of interest.
Supporting Information
The R scripts for the simulation study can be downloaded from an Open Science OSF project accompanying this manuscript. See: https://osf.io/c39tv/overview. Data from the ABCD is not publicly available and can be obtained upon application to the respective cohort study. Access to and use of IDEFICS/I.Family cohort data and biosamples can be obtained via an electronic application portal at https://www.bips-institut.de/en/research/research-data-portal. R scripts for the real data example are provided in the OSF project. Online Supplementary Materials A and B, referenced in Section 3.3, Section 5 and Section 6, are available with this paper and are also provided in the OSF project.
Simulation
The simulated fasting-insulin trajectories are based on the qualitative shape of the transient pubertal pattern: insulin sensitivity decreases after pubertal onset, reaches its lowest level around mid-puberty, and then recovers toward pre-pubertal levels by late puberty [28].
Baseline and time-varying covariates
Let denote subjects, measurement occasions, cohort membership, and the age of subject from cohort at measurement . To allow distribution drift between cohorts, cohort-specific distributional differences in the covariates are introduced through
Four baseline covariates were generated once per subject and kept constant across repeated measurements. Sex was generated according to
The continuous baseline covariates and were jointly generated as
where . The third continuous baseline covariate was generated from
Two time-varying covariates were generated with continuous-time within-subject correlation. For
we assumed
where
with .
For the second time-varying covariate,
we assumed
with
and
where .
True fixed effects function
The fixed-effect function included a smooth pubertal component to represent the transient increase in insulin resistance observed during puberty [28]. The Gaussian form and its location and scale parameters were specified for simulation purposes to provide a smooth approximation to this transient pattern rather than to reproduce an empirically estimated age-specific trajectory.
together with linear effects of baseline and time-varying covariates, a nonlinear adiposity effect, where steeper increase in fasting insulin is observed among subjects with overweight/obesity
and interactions between baseline and time-varying covariates. Specifically,
Here, denotes sex, to denote baseline covariates, and and denote time-varying covariates.
Derivation of the asymptotic bias of the fitted predictor
Consider the as a special case in the semi-parametric stochastic mixed effects model (11) in Section 3.3 proposed by Capitaine et al. [9], where the stochastic process is not considered and flexible covariance structures for random effects and residuals are allowed:
where
Then we derive the asymptotic bias of the fitted predictor in the following way similar to the approach of Capitaine et al. [9] assuming the variance-covariance components are known.
The marginal covariance matrix of is
The fixed-effects component is estimated using random forests, yielding the estimator .
The fitted predictor is defined as
where is the BLUP estimator of the random effects expressed as:
Substituting (25) into (24) gives
Define
Then
Taking expectations yields
From model (22),
because
Moreover, define the random forest bias by
Hence,
Substituting into (31):
Since
we obtain
Using (28), this becomes
If the RF estimator becomes asymptotically unbiased, then the mean fit of MERF prediction also tends to the true mean behavior . Thus, under some regularity conditions on the random forest
then
This is the step where Capitaine et al. [9] invoke the consistency result of Scornet et al. [26], which do not directly apply to multi-cohort setting due to the presence of heterogeneous covariance structures across cohorts, and the possibly remaining temporal dependence after including the random effects.
References
References
- [1]Katrina L Kezios, M. Maria Glymour, and Adina Zeki Al Hazzouri. An introduction to longitudinal synthetic cohorts for studying the life course drivers of health outcomes and inequalities in older age. Current Epidemiology Reports, 12(1):2, 2025.DOI
- [2]Sascha Zuber, Laura Bechtiger, Julien Stéphane Bodelet, Marta Golin, Jens Heumann, Jung Hyun Kim, Matthias Klee, Jure Mur, Jennie Noll, Stacey Voll, Patrick O’Keefe, Annekatrin Steinhoff, Ulf Zölitz, Graciela Muniz-Terrera, Lilly Shanahan, Michael J Shanahan, and Scott M Hofer. An integrative approach for the analysis of risk and health across the life course: challenges, innovations, and opportunities for life course research. Discover Social Science and Health, 3(1):14, 2023.
- [3]Leo Breiman. Random forests. Machine Learning, 45(1):5–32, 2001. doi: 10.1023/A:1010933404324.DOI
- [4]Che Ngufor, Holly Van Houten, Brian S. Caffo, Nilay D. Shah, and Rozalina G. McCoy. Mixed effect machine learning: A framework for predicting longitudinal change in hemoglobin A1c. Journal of Biomedical Informatics, 89:56–67, 2019. ISSN 1532-0464. doi: https://doi.org/10.1016/j.jbi.2018.09.001.
- [5]David A. Harville. Maximum likelihood approaches to variance component estimation and to related problems. Journal of the American Statistical Association, 72(358):320–338, 1977. doi: 10.1080/01621459.1977.10480998.DOI
- [6]Pascal Kilian, Sangbeak Ye, and Augustin Kelava. Mixed effects in machine learning – a flexible mixedML framework to add random effects to supervised machine learning regression. Transactions on Machine Learning Research, 2023. URL https://openreview.net/forum?id=MKZyHtmfWH.
- [7]Gérard Biau and Erwan Scornet. A random forest guided tour. TEST, 25(2):197–227, 2016. doi: 10.1007/s11749-016-0481-7.
- [8]Jianchang Hu and Silke Symczak. A review on longitudinal data analysis with random forest. Briefings in Bioinformatics, 24(2):2, 2023. doi: 10.1093/bib/bbad002.DOI
- [9]Louis Capitaine, Robin Genuer, and Rodolphe Thiébaut. Random forests for high-dimensional longitudinal data. Statistical Methods in Medical Research, 30(1):166–184, 2021. doi: 10.1177/0962280220946080.DOI
- [10]Ahlem Hajjem, François Bellavance, and Denis Larocque. Mixed-effects random forest for clustered data. Journal of Statistical Computation and Simulation, 84(6):1313–1328, 2014. doi: 10.1080/00949655.2012.741599.
- [11]Laura M. Grajeda, Andrada Ivanescu, Mayuko Saito, Ciprian Crainiceanu, Devan Jaganath, Jean E. Gilman, Robert H. and Crabtree, Dermott Kelleher, Lilia Cabrera, Vitaliano Cama, and William Checkley. Modelling subject-specific childhood growth using linear mixed-effect models with cubic regression splines. Emerging Themes in Epidemiology, 13(1):1, 2016. doi: 10.1186/s12982-015-0038-3.DOI
- [12]Ruotong Liu, Xiang Qi, Qian Huo, and Bei Wu. Trajectory of cognitive decline across different racial/ethnic groups: The role of edentulism. Research on Aging, 47(5-6):297–307, 2025. doi: 10.1177/01640275251315850.DOI
- [13]Prashanthi Vemuri, Timothy G. Lesnick, Scott A. Przybelski, Mary Machulda, David S. Knopman, Michelle M. Mielke, Rosebud O. Roberts, Yonas E. Geda, Walter A. Rocca, Ronald C. Petersen, and Jr Jack, Clifford R. Association of lifetime intellectual enrichment with cognitive decline in the older population. JAMA Neurology, 71(8):1017–1024, 2014. doi: 10.1001/jamaneurol.2014.963.DOI
- [14]Michael J. Daniels and Yan D. Zhao. Modelling the random effects covariance matrix in longitudinal data. Statistics in Medicine, 22(10):1631–1647, 2003. doi: https://doi.org/10.1002/sim.1470.
- [15]Rebecca J Sela and Jeffrey S Simonoff. RE-EM trees: a data mining approach for longitudinal and clustered data. Machine Learning, 86(2):169–207, 2012. doi: 10.1007/s10994-011-5258-3.DOI
- [16]Patrick Krennmair and Timo Schmid. Flexible domain prediction using mixed effects random forests. Journal of the Royal Statistical Society Series C: Applied Statistics, 71(5):1865–1894, 2022. doi: 10.1111/rssc.12600.DOI
- [17]Louis Capitaine. LongituRF: Random Forests for Longitudinal Data, 2024. R package version 0.9.
- [18]Benjamin M. Bolker, Mollie E. Brooks, Connie J. Clark, Shane W. Geange, John R. Poulsen, M. Henry H. Stevens, and Jada-Simone S. White. Generalized linear mixed models: a practical guide for ecology and evolution. Trends in Ecology & Evolution, 24(3):127–135, 2009.DOI
- [19]Benjamin M Bolker. Linear and generalized linear mixed models. In Ecological Statistics: Contemporary theory and application, pages 309–333. Oxford University Press, 2015.DOI
- [20]Dylan G. E. Gomes. Should i use fixed effects or random effects when i have fewer than five levels of a grouping factor in a mixed-effects model? PeerJ, 10:e12794, 2022. doi: 10.7717/peerj.12794.DOI
- [21]T. J. Cole and T. Lobstein. Extended international (IOTF) body mass index cut-offs for thinness, overweight and obesity. Paediatric Obesity, 7(4):284–294, 2012. doi: https://doi.org/10.1111/j.2047-6310.2012.00064.x.
- [22]Manon van Eijsden, Tanja GM Vrijkotte, Reinoud JBJ Gemke, and Marcel F van der Wal. Cohort profile: The amsterdam born children and their development (abcd) study. International Journal of Epidemiology, 40(5): 1176–1186, 2011. doi: 10.1093/ije/dyq128.DOI
- [23]W. Ahrens, A. Siani, R. Adan, S. De Henauw, G. Eiben, W. Gwozdz, A. Hebestreit, M. Hunsberger, J. Kaprio, V. Krogh, L. Lissner, D. Molnár, L. A. Moreno, A. Page, C. Picó, L. Reisch, R. M. Smith, M. Tornaritis, T. Veidebaum, G. Williams, H. Pohlabeln, I. Pigeot, and on behalf of the I.Family consortium. Cohort profile: The transition from childhood to adolescence in european children—how I.Family extends the IDEFICS cohort. International Journal of Epidemiology, 46(5):1394–1395, 2017. doi: 10.1093/ije/dyw317.DOI
- [24]José Pinheiro, Douglas Bates, and R Core Team. nlme: Linear and Nonlinear Mixed Effects Models, 2025. URL https://svn.r-project.org/R-packages/trunk/nlme/. R package version 3.1-168.
- [25]Marvin N Wright and Andreas Ziegler. Ranger: A fast implementation of random forests for high dimensional data in c++ and R. Journal of Statistical Software, 77(1):1–17, 2017. doi: 10.18637/jss.v077.i01.DOI
- [26]Erwan Scornet, Gérard Biau, and Jean-Philippe Vert. Consistency of random forests. The Annals of Statistics, 43(4):1716–1741, 2015. doi: 10.1214/15-AOS1321. URL https://www.jstor.org/stable/43556658.DOI
- [27]Pascal Rink and Werner Brannath. Post-selection confidence bounds for prediction performance. Machine Learning, 114(82):82, 2025.arxiv.org/abs/2210.13206
- [28]Antoinette Moran, David R. Jr. Jacobs, Julia Steinberger, Ching Ping Hong, Ronald Prineas, Russell Luepker, and Alan R. Sinaiko. Insulin resistance during puberty: Results from clamp studies in 357 children. Diabetes, 48(10):2039–2044, 1999. doi: 10.2337/diabetes.48.10.2039. URL https://doi.org/10.2337/diabetes.48.10.2039.
