WorksMathematics
Unifying Data-Adaptive Longitudinal Treatment-Effect Subgroup Discovery and Pretreatment Membership Prediction
Abstract
Precision medicine seeks to match patients with treatments from which they are most likely to benefit. A key step toward this goal is to learn treatment-effect heterogeneity from longitudinal study data. This requires discovering subgroups with distinct treatment-effect trajectories and predicting subgroup membership for new patients from pretreatment information. Existing methods do not fully unify these tasks. Covariate-informed membership models typically prespecify the subgroup number or covariance structure, whereas data-adaptive discovery methods often require prediction in a separate stage. We propose a data-adaptive semiparametric mixture-of-experts framework that unifies longitudinal treatment-effect subgroup discovery and pretreatment membership prediction. Subgroup-specific varying-coefficient functions characterize treatment-effect trajectories, while flexible covariance functions account for within-subject dependence. A covariate-dependent gating model links membership to pretreatment characteristics. A penalty on global component weights removes redundant subgroups, while separate gating coefficients allow patient-specific membership probabilities among retained subgroups. A blockwise generalized expectation–maximization algorithm estimates all model components under a single penalized observed-data objective. We establish consistent subgroup-number selection, convergence rates for the functional estimators, and asymptotic normality for the finite-dimensional parameters. Simulations show reliable subgroup-number selection and more accurate conditional membership prediction than a two-step procedure. Applications to the SANAD epilepsy study and the COMPETE migraine trial identify distinct temporal treatment-effect patterns.
Introduction
A central goal of precision medicine is to use pretreatment information to guide treatment selection for new patients. Conventional analyses often summarize treatment benefit through a population-average effect, which may obscure important differences across patients [11, 15]. Subgroup analysis of longitudinal outcomes can characterize this heterogeneity by identifying groups with distinct treatment-effect trajectories. To translate outcome-defined subgroups to new patients, membership must be predictable from information available before treatment. This calls for a unified model that learns the subgroup number and treatment-effect trajectories from study outcomes while predicting membership from pretreatment covariates.
Longitudinal follow-up provides a richer basis for studying treatment-effect heterogeneity than a single endpoint. Repeated measurements reveal when treatment responses emerge, how they evolve, and whether they persist, and may also improve the precision of treatment comparisons ([7, 22] (Collins 2006)). These advantages have motivated a substantial literature on statistical methods for longitudinal treatment-response analysis. At the same time, longitudinal data introduce within-subject dependence and irregular observation times, while treatment-response trajectories may be nonlinear. In heterogeneous populations, both treatment-effect trajectories and within-subject covariance structures may vary across latent subgroups, requiring flexible subgroup-specific modeling of both features.
Existing methods fall into two broad, partly overlapping strands. Covariate-defined approaches construct treatment-sensitive subgroups from pretreatment characteristics. Threshold and change-plane models define subgroup boundaries through one or more covariates, with extensions for dropout, time-varying treatment effects, and floor or ceiling effects ([5, 26, 27, 6]). Tree-based methods recursively partition the covariate space and accommodate longitudinal or multivariate outcomes ([16, 28, 31]). Outcome-driven approaches instead infer latent heterogeneity from repeated responses. Random-effects mixtures and structured mixed-effects models identify subgroups with distinct treatment-response trajectories; some also model membership probabilities using pretreatment covariates ([30, 25, 29]).
More recent work has pursued data-adaptive subgroup discovery and membership prediction, although often separately. Zhang et al. [32] proposed a semiparametric longitudinal treatment-effect model that selects the subgroup number and estimates subgroup-specific covariance functions. Related methods include sparse mixture regression for multivariate functional responses ([33]) and fusion-based clustering of longitudinal profiles ([34]). For nonlongitudinal outcomes, Shen and He [24] jointly identify a treatment-sensitive subgroup and construct predictive scores for membership under a structured mixture model. Chen et al. [3] considered a scalar outcome with a multivariate grouping vector, allowing coefficient-specific subgroup structures; their procedure first detects groups by regularization and then uses a support vector machine to recover covariate-space partitions for predicting future membership.
Taken together, these methods do not jointly address data-adaptive subgroup discovery and pretreatment membership prediction. Methods that incorporate pretreatment covariates often prespecify the subgroup number or within-subject covariance structure. The threshold and change-plane methods cited above, as well as Shen and Qu [25], assume two subgroups. Wen et al. [29] select the number of classes by comparing prespecified candidate models and retain a parametric model for within-subject dependence. In contrast, the data-adaptive longitudinal methods discussed above do not estimate a pretreatment membership model jointly with subgroup discovery. Zhang et al. [32], for example, fit a membership model only after discovering the subgroups; in a nonlongitudinal setting, Chen et al. [3] similarly recover covariate-space partitions with a support vector machine after detecting coefficient groups. Because pretreatment covariates do not contribute to the discovery stage, these procedures treat estimated membership labels or responses as fixed during prediction, so discovery errors can propagate to the prediction stage.
We address this gap with a data-adaptive semiparametric mixture-of-experts model that jointly discovers longitudinal treatment-effect subgroups and predicts membership from pretreatment covariates. Within each subgroup, varying-coefficient functions characterize the treatment-effect trajectory, while a flexible covariance function captures within-subject dependence. A covariate-dependent gating model relates subgroup membership to pretreatment characteristics. Beginning with a prespecified upper bound on the number of subgroups, we penalize global component weights to eliminate redundant groups, while separate gating coefficients allow membership probabilities among the retained groups to vary across patients. A blockwise generalized expectation–maximization algorithm jointly estimates all model components under a single penalized observed-data objective.
For study participants, posterior membership probabilities combine information from pretreatment covariates and longitudinal outcomes; for future patients, the fitted gating model produces membership probabilities from pretreatment covariates alone.
The framework differs from existing methods in three respects. First, because the gating model enters the observed-data likelihood, pretreatment covariates contribute directly to subgroup discovery rather than being used only to predict subgroup labels estimated in a separate stage. Second, separating global component weights from covariate-dependent gating coefficients enables subgroup-number selection and patient-specific membership modeling within the same objective. A redundant component can therefore be eliminated globally without forcing membership probabilities among the retained groups to be constant across patients. Third, the longitudinal model allows both treatment-effect trajectories and covariance functions to vary across subgroups without prespecifying a parametric correlation structure. We establish selection consistency for the subgroup number, convergence rates for the functional estimators, and asymptotic normality of the finite-dimensional estimators.
The remainder of the paper is organized as follows. Section 2 develops the proposed model and estimation procedure, including tuning-parameter selection and prospective membership prediction. Section 3 establishes subgroup-selection consistency and the asymptotic properties of the estimators. Section 4 evaluates finite-sample performance through simulations. Section 5 illustrates the method using data from the COMPETE migraine trial and the SANAD epilepsy study. The final section discusses the implications, limitations, and directions for future research.
Model and estimation
Consider independent subjects, with subject observed at times , . Let and denote the corresponding longitudinal response and covariate vector. Define , , . Let denote treatment assignment, with indicating the treatment of interest and the comparator. Let contain pretreatment covariates used to predict subgroup membership for new subjects. The vectors and may share baseline covariates. The observed data for subject are .
Let denote the latent subgroup membership of subject , where is the true but unknown number of subgroups. Conditional on , we assume the subgroup-specific varying-coefficient model
where represents the time-varying covariate effects, and represents the time-varying treatment effects in subgroup . For a fixed covariate profile , the corresponding treatment contrast is .
Within-subject dependence in subgroup is represented by a mean-zero Gaussian process with covariance function . Its Karhunen–Loève expansion is [1, 10], where is the th eigenfunction of . The eigenfunctions are orthonormal, satisfying , and the scores are independent across , with . We order the eigenvalues so that and assume . In practice, we retain the first terms and define and . Let ; then . The measurement errors satisfy and are independent of . Let denote the matrix with th row . Under the rank- approximation, the covariance matrix of conditional on is .
Because is unknown, we embed the true model in an over-specified working model with candidate subgroups. Under this working model, , and we relate subgroup membership to pretreatment covariates through
where , , and . The global weights determine which components remain active, whereas the gating coefficients allow membership probabilities among active components to vary with pretreatment covariates. In particular, removes subgroup for every covariate profile, whereas variation in represents subject-specific membership rather than global subgroup removal. Let denote the true active set, so that . For identifiability, we exclude an intercept from and set . The global weights satisfy . If for every , the gating model reduces to constant mixing proportions, with for all ; in general, however, need not equal the marginal prevalence of subgroup .
Penalized estimation
Let denote the subgroup-specific mean vector, where . Under the rank- covariance approximation, the subgroup-conditional density is , where denotes the -variate Gaussian density. Marginalizing over gives the observed-data density .
We approximate the nonparametric functions using B-spline basis expansions [23]. Define the Hölder class
where is a nonnegative integer, , , and . Assume that and . Let be a -dimensional B-spline basis on . We use for and , where and .
Let . The constraints ensure orthonormality. For an orthonormalized spline basis, .
Let be the matrix whose th row is . Set , , and . Let denote the vector formed by stacking and . The approximated mean and covariance are then , and . Substituting these approximations into the mixture density, the observed-data log-likelihood is
where collects the spline coefficients, eigenvalues, measurement-error variances, global weights, and gating coefficients.
To remove redundant candidate subgroups, we apply an adapted logarithmic penalty to the global weights [9],
where controls the degree of subgroup selection and ensures that the penalty remains finite at zero. The penalized log-likelihood is
The estimator maximizes (4) subject to the weight-simplex constraint, the reference restriction , and the covariance and orthonormality constraints. The estimated number of subgroups is .
Estimation algorithm
We estimate using a blockwise ascent procedure that combines EM-type posterior weighting with direct optimization of the penalized observed-data objective . Each iteration first computes the posterior subgroup probabilities and then updates the longitudinal and gating parameters in separate blocks. The longitudinal block updates the mean and covariance parameters, whereas the gating block jointly updates the global weights and covariate effects. Candidate updates are accepted only when they do not decrease beyond a numerical tolerance. This structure accommodates the covariance and weight-simplex constraints while retaining a common objective for all model components.
Let denote the latent subgroup indicator. Treating as missing data, the complete-data log-likelihood is . Given the current estimate , the E-step computes the posterior responsibilities
These probabilities combine pretreatment information through with longitudinal evidence through . They serve as subject- and subgroup-specific weights in the subsequent longitudinal updates. For notational simplicity, iteration superscripts are suppressed below, and unspecified quantities on the right-hand sides are evaluated at the current iterate.
For fixed responsibilities and covariance matrices, the mean-function coefficients are updated by posterior-weighted generalized least squares,
We next update the covariance structure using the GLS residuals . Let denote the th column of . With the remaining covariance parameters fixed, the covariance score equation gives the unconstrained proposal
For the eigenvalue update, let . With the remaining covariance parameters fixed, a Fisher-scoring proposal for is
Similarly, with the functional covariance parameters fixed, a Fisher-scoring proposal for the measurement-error variance is
The proposed eigenfunction coefficients are orthonormalized by QR decomposition in the orthonormal spline basis. The resulting eigenvalue–eigenfunction pairs are ordered by decreasing eigenvalue, and each eigenfunction is given a consistent sign across iterations. We then apply backtracking between the current and proposed covariance parameters, reimposing these constraints at each trial step. A trial step is accepted only if it does not decrease beyond the numerical tolerance; otherwise, the current covariance parameters are retained.
Holding the subgroup-conditional densities fixed, we update the global weights and gating coefficients by increasing
subject to the weight-simplex constraint and . This is the restriction of to the gating parameters with the longitudinal parameters held fixed.
For the gating updates, define . Unlike the fixed E-step responsibility , varies with the candidate gating parameters. The relevant derivatives are
We update the global weights and nonreference gating coefficients by constrained Newton-type optimization of , subject to the weight-simplex constraint, the reference restriction , and any specified coordinate bounds. A backtracking line search is used to ensure that each accepted step does not decrease . After each update, components with weights at or below the numerical threshold are removed, and the remaining weights are renormalized. The active component with the largest weight is then selected as the reference, and the gating coefficients are recentered accordingly.
Algorithm 1 summarizes the procedure for fixed .
Algorithm 1 Blockwise estimation for fixed
Input: Data , , tuning parameters, and initial values.
Output: and the active set .
Construct the orthonormal spline basis and design matrices.
for each initialization do
Initialize model parameters and the active subgroup set.
repeat
Compute the posterior responsibilities .
Update the mean parameters using (5).
Update the covariance parameters using (6)–(8) and backtracking.
Update the gating parameters by constrained optimization of (9).
Prune inactive components and update the active subgroup set.
Compute
and the blockwise first-order residual .
until and , or the iteration limit is reached
Store the fitted parameters and objective value.
Retain the fit with the largest attained value of .
return and .
Tuning-parameter selection
The tuning parameters are the spline dimension , covariance rank , and penalty parameter . For each candidate configuration, we compute , where is the unpenalized observed-data log-likelihood. For a fitted model with active subgroups, the nominal parameter count is
The terms correspond to the varying-coefficient functions, orthonormal eigenfunctions, eigenvalues, measurement-error variances, global weights, and nonreference gating coefficients. We select the configuration with the smallest BIC.
Subgroup membership estimation and prediction
For a subject in the training sample, subgroup membership is inferred from
These posterior probabilities incorporate both pretreatment covariates and the observed longitudinal response. For a new subject with pretreatment covariates , only pretreatment information is available. The fitted gating model then gives
where . The subgroup with the largest probability gives a prediction, whereas the full probability vector quantifies assignment uncertainty.
Asymptotic properties
Let , where , , and . The estimated covariance function is
Let and denote the Euclidean and norms, respectively. The working parameter space is
with the weight-simplex, reference, and orthonormality constraints specified in § 2, where .
For , define the distance between two -component parameter values, after aligning their component labels, by
A subscript 0 denotes a true parameter or function. Without loss of generality, order the global weights so that and . The following conditions are used to establish the asymptotic properties of .
(A1) Let denote the spline-knot sequence. Then for , and the ratio of the largest to the smallest knot spacing is bounded.
(A2) The parameters associated with the active components lie in the interior of their corresponding parameter spaces.
(A3) , where .
(A4) For each active subgroup, .
(A5) The matrix is finite and positive definite, where . Here , stacks the nonreference gating coefficients, and . The score is defined in the proof of Theorem 3.
Condition (A1) excludes highly unbalanced knot sequences and is standard in spline analysis [17]. Condition (A2) excludes degenerate active-component parameters [2]. Condition (A3) controls the growth of the covariance rank, which is typically small in practice. Condition (A4) ensures that each covariance operator is trace class [8]. Condition (A5) ensures that the asymptotic covariance matrix is well defined [18].
Theorem 1. Suppose Conditions (A1)–(A5) hold, , , and . Then as .
Theorem 2. Suppose Conditions – hold, , and . Then
Here is the smoothness index defined in (3), and the knot sequence grows at rate . Increasing enlarges the covariance parameter space and slows the convergence rate. If , so that is fixed, then
In particular, choosing yields , the optimal rate for univariate nonparametric function estimation.
Theorem 3. Suppose Conditions – hold with , , , and . For ,
where .
In particular,
Here is the block corresponding to . Because , this result also determines the asymptotic distribution of the full vector of active global weights.
Simulation study
We evaluate the finite-sample performance of the proposed joint method through Monte Carlo simulations. The primary comparator is a two-step procedure that first identifies longitudinal subgroups and then relates estimated membership to pretreatment covariates Zhang et al. [32]. We also include an oracle estimator based on the true subgroup labels and a latent class mixed-model benchmark (HLME) Lin et al. [14]. We assess subgroup-number selection, longitudinal parameter estimation, training membership recovery, and pretreatment membership prediction for new subjects.
Simulation design
Each replication includes a training sample with and an independent test sample with . The true subgroup number is , and the candidate upper bound is . We use independent replications for each setting. Treatment is generated as , and . Conditional on , observation times are sampled independently from and then ordered within subject.
The longitudinal covariate vector is , where , , and . Its third component is
The scalar is generated once per subject. The coefficients are independent across subjects and basis indices. The pretreatment predictors are generated independently as . Define , , and . The latent subgroup label is generated according to
with . Because the linear predictor is symmetric about zero, both subgroups have marginal prevalence 0.5. Under the proposed parameterization, the true gating parameters are , , and .
Conditional on , the longitudinal response follows
where and are mutually independent. We set , , and . Let . The subgroup-specific coefficient functions are
and , with .
Both penalized procedures use the same orthonormalized cubic B-spline basis, with interior knots at 0.25, 0.50, and 0.75 (), and covariance rank . We use for the joint estimator and for the first stage of the two-step procedure. Joint-model gating coefficients are constrained to , whereas the second-stage logistic coefficients are unconstrained. Each longitudinal fit is allowed at most 200 iterations.
Competing methods and implementation
The joint estimator fits the longitudinal and membership models simultaneously using Algorithm 1. The two-step comparator first fits a longitudinal mixture with constant component weights. It then treats the posterior membership probabilities as fixed soft responses in a multinomial logistic regression on the centered pretreatment predictors. The second stage does not update the longitudinal parameters. For training classification, the fitted gating model is combined with the fixed component densities. Predictions for new subjects use the gating model alone. This implementation uses soft-label logistic regression rather than the posterior-log-odds regression of Zhang et al. [32].
The oracle estimator fits two groups using the true training labels. HLME fits two latent classes using hlme from the R package lcmm ([21]). It uses the same spline mean design and pretreatment membership predictors. Its random-effects model uses the true cosine and sine functions, a diagonal covariance, a class-specific proportional factor, and a common residual variance. Thus, HLME is supplied with the true subgroup number and covariance shapes; it neither estimates eigenfunctions nor selects the subgroup number.
Evaluation criteria
We report the number of replications that correctly identify out of 200 (RCI). Training membership recovery is measured by IIC, the proportion correctly classified after label alignment. For new subjects, we report classification accuracy (ACC), Brier score, negative log predictive probability (NLP), and , where is defined in (12). Higher ACC and lower Brier score, NLP, and indicate better prediction. All test predictions use pretreatment covariates alone. Parameter results are reported as RMSE (bias), with signed bias for scalar parameters. For a function evaluated at equally spaced grid points, functional bias is . Functional RMSE averages squared errors over grid points and replications.
Simulation results
Tables 1 and 2 summarize the results for and . Each table compares the oracle estimator, the fixed-two-group HLME benchmark, and both penalized procedures initialized at . The comparison involved 3,200 fits across 400 paired training and test datasets, and all fits completed without execution errors. All oracle and HLME fits met their numerical convergence criteria. Returned estimates from the penalized procedures that did not meet the numerical convergence criterion are retained in the summaries.
For , the joint estimator selected two subgroups in 196, 194, and 192 replications for , respectively. The corresponding counts for the two-step procedure were 196, 180, and 191. For , these counts were 200, 200, and 198 for the joint estimator and 193, 194, and 195 for the two-step procedure. Thus, the joint estimator had equal or higher correct-selection counts in every configuration considered, indicating that the global-weight penalty effectively removed redundant candidate subgroups across the three choices of .
Across the candidate upper bounds, the joint estimator’s errors in the longitudinal parameters were stable. Its varying-coefficient RMSEs were consistently smaller than those of the two-step procedure, and its eigenfunction RMSEs were also generally smaller. This pattern is consistent with more accurate recovery of the longitudinal component when pretreatment covariates inform subgroup learning.
Among fits selecting two subgroups, the joint estimator achieved IIC values of 95.42–95.44% for and approximately 95.61% for . The corresponding ranges for the two-step procedure were 67.17–69.36% and 65.08–65.88%. Test-set classification accuracy ranged from 90.48–90.59% for the joint estimator and 79.60–81.78% for the two-step procedure. The corresponding Brier score ranges were 0.068–0.069 and 0.182–0.194, and test-set ranged from 0.034 to 0.042 and from 0.331 to 0.350, respectively. Bias in the gating slopes also showed substantially less attenuation under the joint estimator. Thus, the improvement was evident in training-set membership recovery, test-set classification, and probability estimation. These comparisons remain conditional on selecting two subgroups and use method-specific eligible replications.
The correctly specified HLME benchmark had IIC values of 95.53% and 95.70% for and , respectively. Its prediction performance was close to that of the joint estimator. HLME often had smaller covariance and gating-parameter RMSEs, reflecting the additional subgroup-number and covariance-shape information supplied to this benchmark. Accordingly, the joint estimator did not outperform HLME for every parameter. The oracle estimator provided a known-membership reference for longitudinal estimation. Conditional selection and Monte Carlo variation can produce small departures from a strict empirical RMSE ordering.
For and , the average estimated eigenfunctions and varying-coefficient functions followed their true shapes. Figure 1 displays the four subgroup-specific eigenfunctions, and Figure 2 displays the six coefficient functions for each subgroup. Solid curves denote the true functions, and dashed curves denote averages across replications. The pointwise bands summarize empirical variation across the 196 replications selecting two subgroups after label and eigenfunction alignment. They are neither standard errors of the Monte Carlo averages nor simultaneous confidence bands. Variation was greater near the boundaries for several coefficient functions.

Figure 1. Estimated eigenfunctions for and candidate upper bound . The first row displays and ; the second displays and . Solid curves are the true functions, and dashed curves are averages across replications. Gray areas are pointwise 95% Monte Carlo variability bands, constructed as the average estimate times the pointwise empirical standard deviation across the 196 replications (of 200) that selected two subgroups. Curves and band boundaries are evaluated at 200 equally spaced points on , including both endpoints. These bands describe sampling variation across replications.

Figure 2. Estimated varying coefficients for both subgroups, with and candidate upper bound . The top two rows show subgroup 1, and the bottom two rows show subgroup 2. Within each subgroup, the first row displays , , and ; the second shows , , and . Solid curves, dashed curves, and gray areas denote the true functions, averages across replications, and pointwise 95% Monte Carlo variability bands, respectively, evaluated at 200 equally spaced points on , using the same 196 replications and construction as in Figure 1.
| Parameter | Oracle | HLME () | Proposed | Two-step | Proposed | Two-step | Proposed | Two-step |
| / | ||||||||
| / | ||||||||
| / | ||||||||
| / | ||||||||
| / | ||||||||
| / | ||||||||
| / | ||||||||
| / | ||||||||
| RCI | / | / | 196/200 | 196/200 | 194/200 | 180/200 | 192/200 | 191/200 |
| IIC | 100.00% | 95.53% | 95.42% | 67.17% | 95.44% | 69.36% | 95.43% | 68.63% |
| Test ACC | / | 90.51% | 90.48% | 79.60% | 90.51% | 80.31% | 90.53% | 80.17% |
| Brier | / | 0.068 | 0.069 | 0.188 | 0.068 | 0.182 | 0.068 | 0.184 |
| NLP | / | 0.220 | 0.224 | 0.561 | 0.223 | 0.546 | 0.222 | 0.550 |
| RMSE | / | 0.034 | 0.042 | 0.341 | 0.041 | 0.331 | 0.041 | 0.334 |
Table 1. Simulation results for (200 replications per method).
Note. Parameter entries are RMSE (bias). Scalar bias is signed; functional bias is the root integrated squared bias. RCI uses all 200 replications. Parameter errors, IIC, and test metrics use method-specific fits selecting two subgroups, irrespective of numerical convergence.
The oracle uses known training labels. HLME fixes two groups and the cosine and sine random-effect functions; unavailable quantities are marked “/”. Test prediction uses pretreatment covariates only. Proposed: , ; two-step: .
| Parameter | Oracle | HLME () | Proposed () | Two-step () | Proposed () | Two-step () | Proposed () | Two-step () |
| / | ||||||||
| / | ||||||||
| / | ||||||||
| / | ||||||||
| / | ||||||||
| / | ||||||||
| / | ||||||||
| / | ||||||||
| RCI | / | / | 200/200 | 193/200 | 200/200 | 194/200 | 198/200 | 195/200 |
| IIC | 100.00% | 95.70% | 95.61% | 65.88% | 95.61% | 65.70% | 95.61% | 65.08% |
| Test ACC | / | 90.65% | 90.56% | 81.78% | 90.55% | 81.78% | 90.59% | 81.01% |
| Brier | / | 0.067 | 0.068 | 0.191 | 0.068 | 0.192 | 0.068 | 0.194 |
| NLP | / | 0.219 | 0.221 | 0.568 | 0.221 | 0.569 | 0.221 | 0.573 |
| RMSE | / | 0.028 | 0.034 | 0.346 | 0.034 | 0.347 | 0.034 | 0.350 |
Table 2. Simulation results for (200 replications per method). Note. Parameter entries are RMSE (bias). Scalar bias is signed; functional bias is the root integrated squared bias. RCI uses all 200 replications. Parameter errors, IIC, and test metrics use method-specific fits selecting two subgroups, irrespective of numerical convergence. The oracle uses known training labels. HLME fixes two groups and the cosine and sine random-effect functions; unavailable quantities are marked “/”. Test prediction uses pretreatment covariates only. Proposed: , ; two-step: .
Applications to clinical trial data
COMPETE migraine trial
We illustrate the proposed mixture-of-experts model using the COMPETE randomized clinical trial of migraine prevention in patients with patent foramen ovale. COMPETE was a multicenter, randomized, active-controlled study of antithrombotic agents versus metoprolol [12]. We compare rivaroxaban with metoprolol, the active control. The complete-case analysis included 469 participants, of whom 236 received rivaroxaban and 233 received metoprolol. The primary trial report evaluated the 12-week responder outcome [13].
The longitudinal outcome was the monthly number of migraine days, transformed as , and measured at baseline and weeks 4, 8, and 12. Follow-up time was rescaled to . We set for rivaroxaban and for metoprolol. The standardized baseline migraine-attack frequency was included as a prognostic covariate. The longitudinal model was
In this model, is the subgroup-specific treatment-effect function, approximated by cubic B-splines. The term represents the leading functional principal component of within-subject variation.
We used two standardized pretreatment domains from the Migraine-Specific Quality of Life Questionnaire to model subgroup membership. The Role Restrictive (MSQ-RR) and Emotional Function (MSQ-EF) domains measure limitations in daily activities and emotional functioning, respectively. Let , without an intercept. The gating model was
where . We initialized candidate components and set . The fitted model retained two subgroups. Table 3 reports the parameter estimates and bootstrap summaries.
| Parameter | Subgroup 1: Estimate | Subgroup 1: SD | Subgroup 1: 95% CI | Subgroup 2: Estimate | Subgroup 2: SD | Subgroup 2: 95% CI |
| Longitudinal and covariance parameters | ||||||
| 0.365 | 0.108 | (0.191, 0.616) | 0.635 | 0.108 | (0.384, 0.809) | |
| 1.988 | 0.071 | (1.890, 2.168) | 2.176 | 0.042 | (2.073, 2.217) | |
| 0.322 | 0.088 | (0.188, 0.533) | 0.713 | 0.058 | (0.606, 0.832) | |
| 0.233 | 0.034 | (0.147, 0.279) | 0.029 | 0.007 | (0.012, 0.040) | |
| 0.548 | 0.072 | (0.434, 0.736) | 0.504 | 0.056 | (0.413, 0.630) | |
| Gating coefficients relative to subgroup 1 | ||||||
| Coefficient | Estimate | SD | 95% CI | |||
| (MSQ-RR) | 0.446 | 0.232 | (0.024, 0.936) | |||
| (MSQ-EF) | -0.268 | 0.225 | (−0.723, 0.157) |
Table 3. Parameter estimates for the two identified subgroups in the COMPETE migraine trial. Note. Point estimates use the full sample. Longitudinal and covariance SDs and CIs use 200 nonstratified patient-level warm-start bootstrap replications. Gating SDs and CIs use 200 patient-level bootstrap replications stratified by the original posterior-modal subgroups (162 and 307 patients); all 200 fits converged. The gating intervals are bootstrap means ±1.96 bootstrap SDs. Gating coefficients are log-odds coefficients for subgroup 2 relative to subgroup 1 per one-SD increase in the standardized MSQ score.
The fitted global weights were and . Relative to subgroup 1, the estimated subgroup 2 gating coefficients were 0.446 for MSQ-RR and for MSQ-EF. Their bootstrap SDs were 0.232 and 0.225, respectively. The corresponding bootstrap-mean-centered normal 95% intervals were (0.024, 0.936) and (, 0.157). Thus, higher MSQ-RR was associated with greater odds of subgroup 2 membership, whereas the association with MSQ-EF was less precisely estimated.
Figure 3(a) shows two distinct temporal treatment-effect patterns. Subgroup 2 exhibited an early-plateau pattern, with estimated effects of , , and at weeks 4, 8, and 12. Subgroup 1 exhibited a delayed-strengthening pattern, with corresponding estimates of , , and . The estimated curves crossed during follow-up, showing that their ordering changed over time.

Figure 3. Estimated treatment-effect trajectories and leading eigenfunctions for the COMPETE migraine trial. Panel (a) shows for subgroup 1 (red, delayed-strengthening pattern) and subgroup 2 (blue, early-plateau pattern). Panels (b) and (c) show and , respectively. Curves are fitted estimates; shaded regions are pointwise 95% confidence intervals based on 200 patient-level warm-start bootstrap resamples. Follow-up is expressed in weeks. Negative treatment-effect values indicate a lower transformed migraine-day outcome under rivaroxaban than under metoprolol.
Because the outcome was measured on the scale, negative estimates indicate a lower conditional mean under rivaroxaban than under metoprolol. The estimated heterogeneity therefore concerned both the magnitude and timing of the treatment contrast. A population-average curve would not distinguish an early response approaching a plateau from a contrast that strengthened later.
Figure 3(b,Figure 3c) shows the estimated leading eigenfunctions. Their distinct shapes describe subgroup-specific patterns of within-subject temporal variation beyond the estimated mean treatment effects.
For a new patient, the fitted gating model gives the subgroup 2 probability
The scores and are standardized using the analysis-sample means and standard deviations, and . The subgroup with the larger probability gives a hard prediction, whereas retaining both probabilities quantifies assignment uncertainty.
SANAD epilepsy study
The Standard and New Antiepileptic Drugs (SANAD) study was a randomized trial of standard and newer antiepileptic drugs [19 b]. Following Wei et al. [27], we analyzed patients receiving lamotrigine (LTG) or carbamazepine (CBZ). The outcome was the log-transformed Liverpool Adverse Events Profile (AEP), for which higher scores indicate a greater burden of adverse effects. The complete-case dataset contained 1,720 observations from 540 patients, including 262 assigned to LTG and 278 assigned to CBZ. Observation times were divided by the maximum observed follow-up time, giving .
For each subject, the first recorded anxiety (ANX) and depression (DEP) scores were standardized and used as subject-level predictors. The model included constant prognostic coefficients and a time-varying treatment effect. Conditional on , the fitted longitudinal model was
where for LTG and for CBZ. The same ANX and DEP scores entered the gating model in (2). We used cubic B-splines with interior knots at 0.153 and 0.312 and set . Starting from with , the algorithm converged to two subgroups after 25 iterations.
Table 4 summarizes the fitted parameters. The global weights were and . These are membership probabilities at the centered predictor profile, not marginal subgroup proportions. ANX and DEP had positive associations with log-AEP in both subgroups, with larger estimates in subgroup 2. The estimated latent-process and measurement-error variances were also larger in subgroup 2.
| Parameter | Estimate | SD | 95% CI | Estimate | SD | 95% CI |
| 0.292 | 0.105 | (0.087, 0.497) | 0.708 | 0.105 | (0.503, 0.913) | |
| 3.849 | 0.043 | (3.766, 3.933) | 3.641 | 0.041 | (3.561, 3.721) | |
| (ANX) | 0.049 | 0.020 | (0.010, 0.088) | 0.127 | 0.021 | (0.086, 0.169) |
| (DEP) | 0.045 | 0.014 | (0.018, 0.073) | 0.097 | 0.022 | (0.055, 0.139) |
| 0.006 | 0.003 | (0.000, 0.012) | 0.025 | 0.003 | (0.018, 0.031) | |
| 0.012 | 0.006 | (0.001, 0.023) | 0.043 | 0.007 | (0.029, 0.056) | |
| Gating coefficients: subgroup 2 relative to subgroup 1 | ||||||
| Coefficient | Estimate | SD | 95% CI | |||
| (ANX) | -0.993 | 0.473 | (, ) | |||
| (DEP) | -0.433 | 0.330 | (, 0.213) |
Table 4. Parameter estimates for the two identified subgroups in the SANAD epilepsy study.
Of 200 subject-level bootstrap replications, 186 selected two subgroups and were used for interval estimation; seven of these did not meet the numerical convergence criterion. Scalar intervals used a normal approximation, whereas function intervals used pointwise percentiles. Before summarization, subgroup labels were aligned with the fitted model and gating coefficients were recentered to the same reference subgroup.
Note. ANX and DEP are standardized first-recorded subject-level scores. SDs and normal-approximation 95% CIs use the 186 bootstrap replications (of 200) that selected two subgroups; 179 met the numerical convergence criterion; the other seven fits are retained. Gating coefficients are recentered to subgroup 1 after label alignment.
Figure 4(a) shows the subgroup-specific treatment contrasts. The estimated remained close to zero, and its pointwise interval included zero throughout follow-up. In subgroup 2, the estimated contrast was negative during the middle of follow-up, returned toward zero, and decreased again near the end. Its pointwise interval lay below zero over approximately [0.25, 0.43] and [0.87, 1.00] on the normalized time scale. Negative contrasts indicate a lower conditional mean log-AEP under LTG than under CBZ. The late estimates were less precise, as indicated by the widening intervals. Panels (b) and (c) show the corresponding eigenfunctions. Subgroup 1 had a late upswing, whereas subgroup 2 was comparatively stable during most of the later follow-up.

Figure 4. Estimated treatment-effect trajectories and leading eigenfunctions for the SANAD epilepsy study. Panel (a) displays (upper, red) and (lower, blue) with separate vertical scales. Panels (b) and (c) display and . Curves are fitted estimates, and shaded regions are pointwise 95% bootstrap percentile intervals based on the 186 replications (of 200) that selected two subgroups. Eigenfunction signs are aligned to the fitted functions. Time is normalized to .
The gating estimates for subgroup 2 relative to subgroup 1 were for ANX and for DEP. The reference-aligned 95% intervals were and , respectively. Higher ANX was therefore associated with lower odds of subgroup 2 membership. The association with DEP was less precisely estimated. Although the fitted gating model uses these two scores alone, their pretreatment availability has not been established. Using the fitted model at treatment assignment would require confirming that both measurements precede treatment.
Discussion
In this article, we proposed a data-adaptive semiparametric mixture-of-experts model that unifies longitudinal treatment-effect subgroup discovery and pretreatment membership prediction. Unlike the two-step strategy, the pretreatment membership model enters the observed-data likelihood and directly contributes to subgroup discovery. Separate global component weights and covariate-dependent gating coefficients allow redundant components to be removed while retaining patient-specific membership probabilities. The model also accommodates subgroup-specific treatment-effect trajectories and covariance functions. Under regularity conditions, it consistently selects the subgroup number, with established convergence rates and asymptotic normality. Across candidate upper bounds, simulations showed reliable subgroup-number selection and generally more accurate recovery of the longitudinal components and membership probabilities than the two-step procedure. These comparisons were conditional on selecting two subgroups; the correctly specified HLME benchmark, supplied with the true subgroup number and covariance shapes, performed better for some parameters.
The applications illustrate the value of studying temporal treatment-effect heterogeneity together with pretreatment membership prediction. In the SANAD study, the comparison between lamotrigine and carbamazepine was close to zero in one subgroup but favored lamotrigine during specific periods in the other. In the COMPETE trial, the analysis distinguished an early-plateau pattern from a delayed-strengthening pattern in the comparison of rivaroxaban with metoprolol. Such temporal differences would be obscured by a population-average trajectory. The COMPETE analysis also translated pretreatment characteristics into membership probabilities for new patients. Retaining these probabilities instead of a hard assignment preserves uncertainty when subgroup separation is incomplete. These clinical findings remain exploratory and require independent validation.
Several limitations remain. Because the latent subgroups are defined by the full longitudinal distribution, they may differ in prognosis or within-subject covariance as well as in treatment response and need not represent distinct treatment-sensitivity phenotypes. The working model assumes Gaussian component outcomes, a finite-rank covariance representation, smooth coefficient functions, and a multinomial-logit gating model. The penalized mixture objective is nonconvex, and both applications used complete-case data. Future work could accommodate sparse or nonlinear gating effects, non-Gaussian or multivariate longitudinal outcomes, informative dropout, and subgroup-selection uncertainty. External validation is essential because prediction requires the learned subgroup structure to transport to the target population. Finally, accurate membership prediction does not establish an optimal treatment rule; decision-oriented methods must also account for membership uncertainty and the consequences of incorrect treatment assignment.
References
- [1]Robert B. Ash and Melvyn F. Gardner. Topics in Stochastic Processes: Probability and Mathematical Statistics: A Series of Monographs and Textbooks, volume 27. Academic press, 2014.
- [2]Kani Chen and Xingwei Tong. Varying coefficient transformation models with censored data. Biometrika, 97(4):969–976, 2010.DOI
- [3]Lu Chen, Xuerong Chen, Xinzhou Guo, and Yi Li. Subgroup identification and membership prediction. Biometrics, 82(3):ujag122, 2026. doi: 10.1093/biomtc/ujag122.DOI
- [5]Xinyi Ge, Yingwei Peng, and Dongsheng Tu. A threshold linear mixed model for identification of treatment-sensitive subsets in a clinical trial based on longitudinal outcomes and a continuous covariate. Statistical Methods in Medical Research, 29(10):2919–2931, 2020. doi: 10.1177/0962280220912772.DOI
- [6]Xinyi Ge, Yingwei Peng, and Dongsheng Tu. A threshold mixed-effects Tobit model for treatment-sensitive subgroup identification based on longitudinal measures with floor and ceiling effects and a continuous covariate. Journal of Statistical Computation and Simulation, 94(11):2544–2563, 2024. doi: 10.1080/00949655.2024.2344126.DOI
- [7]Robert D. Gibbons, Donald Hedeker, and Stephen DuToit. Advances in analysis of longitudinal data. Annual Review of Clinical Psychology, 6:79–107, 2010. doi: 10.1146/annurev.clinpsy.032408.153550.DOI
- [8]Peter Hall and Joel L Horowitz. Methodology and convergence rates for functional linear regression. The Annals of Statistics, 2007.
- [9]Tao Huang, Heng Peng, and Kun Zhang. Model selection for Gaussian mixture models. Statistica Sinica, 27(1):147–169, 2017. doi: 10.5705/ss.2014.105.DOI
- [10]Gareth M James, Trevor J Hastie, and Catherine A Sugar. Principal component models for sparse functional data. Biometrika, 87(3):587–602, 2000.DOI
- [11]David M. Kent, Ewout Steyerberg, and David van Klaveren. Personalized evidence based medicine: predictive approaches to heterogeneous treatment effects. BMJ, 363:k4245, 2018. doi: 10.1136/bmj.k4245.DOI
- [12]Ziping Li and colleagues. Study design and rationale of compete: Comparison of the effect of medication therapy in alleviating migraine with patent foramen ovale. American Heart Journal, 269:1–7, 2024. doi: 10.1016/j.ahj.2023.12.011.DOI
- [13]Ziping Li and colleagues. Antithrombotic treatment for migraine in patients with patent foramen ovale: multicentre, randomised, active controlled, open label trial. The BMJ, 394:e100103, 2026. doi: 10.1136/bmj-2026-100103.DOI
- [14]Haiqun Lin, Charles E. McCulloch, Bruce W. Turnbull, Elizabeth H. Slate, and Larry C. Clark. A latent class mixed model for analysing biomarker trajectories with irregularly scheduled observations. Statistics in Medicine, 19(10):1303–1318, 2000. doi: 10.1002/(SICI)1097-0258(20000530)19:10<1303::AID-SIM424>3.0.CO;2-E.doi.org/10.1002/(sici)1097-0258(20000530)19:10<1303::aid-sim424>3.0.co;2-e
- [15]Ilya Lipkovich, Alex Dmitrienko, and Ralph B. D’Agostino, Sr. Tutorial in biostatistics: data-driven subgroup identification and analysis in clinical trials. Statistics in Medicine, 36(1):136–196, 2017. doi: 10.1002/sim.7064.DOI
- [16]Wei-Yin Loh, Haoda Fu, Michael Man, Victoria Champion, and Menggang Yu. Identification of subgroups with differential treatment effects for longitudinal and multiresponse variables. Statistics in Medicine, 35(26):4837–4855, 2016. doi: 10.1002/sim.7020.DOI
- [17]Minggen Lu, Ying Zhang, and Jian Huang. Semiparametric estimation methods for panel count data using monotone b-splines. Journal of the American Statistical Association, 104(487):1060–1070, 2009.
- [18]Ling Ma, Tao Hu, and Jianguo Sun. Sieve maximum likelihood regression analysis of dependent current status data. Biometrika, 102(3):731–738, 2015.DOI
- [19]AG Marson, R Appleton, GA Baker, DW Chadwick, J Doughty, B Eaton, C Gamble, A Jacoby, P Shackley, DF Smith, et al. A randomised controlled trial examining the longer-term out- comes of standard versus new antiepileptic drugs. the sanad trial. HEALTH TECHNOLOGY ASSESSMENT-SOUTHAMPTON-, 11(37), 2007a.pubmed.ncbi.nlm.nih.gov/17903391
- [21]Cécile Proust-Lima, Viviane Philipps, and Benoit Liquet. Estimation of extended mixed models using latent classes and latent processes: The R package lcmm. Journal of Statistical Software, 78(2):1–56, 2017. doi: 10.18637/jss.v078.i02.
- [22]Patrick Schober and Thomas R. Vetter. Repeated measures designs and analysis of longitudinal data: If at first you do not succeed—try, try again. Anesthesia & Analgesia, 127(2):569–575, 2018. doi: 10.1213/ANE.00000000000003511.DOI
- [23]Larry Schumaker. Spline functions: basic theory. Cambridge university press, 2007.
- [24]Juan Shen and Xuming He. Inference for subgroup analysis with a structured logistic-normal mixture model. Journal of the American Statistical Association, 110(509):303–312, 2015. doi: 10.1080/01621459.2014.894763.DOI
- [25]Juan Shen and Annie Qu. Subgroup analysis based on structured mixed-effects models for longitudinal data. Journal of Biopharmaceutical Statistics, 30(4):607–622, 2020. doi: 10.1080/10543406.2020. 1730867.DOI
- [26]Kecheng Wei, Huichen Zhu, Guoyou Qin, Zhongyi Zhu, and Dongsheng Tu. Multiply robust subgroup analysis based on a single-index threshold linear marginal model for longitudinal data with dropouts. Statistics in Medicine, 41(15):2822–2839, 2022. doi: 10.1002/sim.9386.DOI
- [27]Kecheng Wei, Guoyou Qin, and Zhongyi Zhu. Subgroup analysis for longitudinal data based on a partial linear varying coefficient model with a change plane. Statistics in Medicine, 42(20): 3716–3731, 2023. doi: 10.1002/sim.9827.DOI
- [28]Yishu Wei, Lei Liu, Xiaogang Su, Lihui Zhao, and Hongmei Jiang. Precision medicine: Subgroup identification in longitudinal trajectories. Statistical Methods in Medical Research, 29(9):2603–2616, 2020. doi: 10.1177/0962280220904114.DOI
- [29]Chun-Che Wen, Nathaniel Baker, Rajib Paul, Elizabeth Hill, Kelly Hunt, Hong Li, Kevin Gray, and Brian Neelon. A Bayesian beta-binomial piecewise growth mixture model for longitudinal overdispersed nominal data. Statistical Methods in Medical Research, 33(10):1859–1876, 2024. doi: 10.1177/09622802241279109.DOI
- [30]Weichun Xu and Donald Hedeker. A random-effects mixture model for classifying treatment response in longitudinal clinical trials. Journal of Biopharmaceutical Statistics, 11(4):253–273, 2001. doi: 10.1081/BIP-120008848.DOI
- [31]Zhichen Xu, Jimin Ding, Xiaogang Su, Guoqiao Wang, Ke Xie, Yining Zhao, Chengjie Xiong, Eric McDade, Lon S. Schneider, and Lei Liu. Subgroup identification via interaction tree and mixed model for repeated measures with application to Alzheimer’s disease. Biometrics, 82(2):ujag104, 2026. doi: 10.1093/biomtc/ujag104.DOI
- [32]Pu Zhang, Xinsheng Zhang, Jiao Jin, and Jiakun Jiang. Varying treatment effects in subgroups: A unified framework for longitudinal data analysis. Statistics in Medicine, 45(10–12):e70600, 2026. doi: 10.1002/sim.70600.DOI
- [33]Qingzhi Zhong and Xinyuan Song. Sparse functional varying-coefficient mixture regression. Journal of Multivariate Analysis, 206:105383, 2025. doi: 10.1016/j.jmva.2024.105383.DOI
- [34]Xiaolu Zhu and Annie Qu. Cluster analysis of longitudinal profiles with subgroups. Electronic Journal of Statistics, 12(1):171–193, 2018. doi: 10.1214/17-EJS1389.DOI