Introduction
Deciphering how complex exposures exert their effects through intermediary processes is a cornerstone of modern epidemiology. In the NHANES data considered here, for example, lifestyle characteristics may directly influence glycohemoglobin levels, but might also affect blood biomarkers which in turn cause these variations. However, the continued emergence of large-scale, high-dimensional datasets poses new challenges to established mediation analysis methods. Analyses conducted on biological data are particularly affected, as molecular profiling can yield hundreds or thousands of correlated molecular measurements, only a subset of which may be relevant to the mediating process linking an exposure to an outcome. Latent variable models have been introduced to extend classical mediation models to these settings [1 , 7 , 12 ] (Chén, Crainiceanu, Ogburn, Caffo, Wager, and Lindquist 2017). This approach brings a number of improvements over non-latent mediation techniques [20 ] (Zhu, He, Ma, Hu, Li, and Li 2024) (e.g. based on OLS, LASSO or ElasticNet): noise decomposition, information compression, interpretability and robustness of the estimates.
However, existing latent mediation approaches face several limitations. Stepwise methods estimate latent scores separately from subsets of the observed data and use the resulting scores in subsequent regressions on remaining observed variables, potentially inferring latent variables that are irrelevant to mediation systems [12 ] (Chén et al. 2017). Joint approaches avoid this issue by estimating latent variables along with their relation to observed variables, but may be overparameterized, prone to overfitting, limited in scalability, and typically assume univariate exposures, univariate outcomes, and the conditional independence among manifest variables given the latent mediators whereas biological measurement are generally correlated [1 , 7 ] . To address these limitations, we introduce the Latent Mediation (LaMe) framework, which jointly models multivariate exposures, outcomes and high-dimensional manifest variables through a small number of latent mediators without assuming diagonality of noise covariances, while explicitly accounting for direct exposure-outcome effects.
This approach makes three main methodological contributions. First, LaMe is novel in that it infers latent mediators from a joint probabilistic model with multivariate exposures and outcomes, high-dimensional manifest variables, and non-diagonal residual covariance matrices. As a result, it avoids biases associated with estimating latent scores separately from the rest of the system as in stepwise procedures and estimates all its parameters based on all available information rather
This paper has been submitted for consideration for publication in Biometrics than subsets of observed variables. We characterize when the representation imposes a genuine restriction on cross-block relationships, distinguishing dimension reduction from a reparameterization of unrestricted regression. Second, LaMe connects joint probabilistic estimation methods [1 , 7 ] with representations in which latent mediators are exactly recoverable from the manifest measurements [12 , 21 ] (Chén et al. 2017). Under this explicit restriction and suitable rank conditions, the aggregate natural direct and indirect effects are statistically identified from the observed data. Throughout this article, identifiability refers to statistical identification from the observed-data distribution, rather than causal identification of natural effects. We also introduce a score to assess which manifest variables are associated with exposures and informative about outcomes through the latent mediators, independently of how the latent variables are represented. Third, we derive closed-form EM updates for two equivalent likelihood factorizations, allowing computational efficiency to be improved when manifest measurements and exposures have substantially different dimensions. Simulations assess parameter recovery, effect recovery under the exact recoverability assumption, and feature ranking; real-world applications assess predictive performance and the compactness of the learned representations. These results position LaMe as a useful methodological option.
Background
Let N > 1 N > 1 N > 1 be the number of observations and, for each observation, let I ≥ 1 I \ge1 I ≥ 1 be the number of exposures, J ≥ 1 J \ge1 J ≥ 1 the number of manifest variables, and L ≥ 1 L \ge1 L ≥ 1 the number of outcomes. Then, let X ∈ R N × I X \in\mathbb{R}^{N \times I} X ∈ R N × I denote the exposure matrix, M ∈ R N × J M \in\mathbb{R}^{N \times J} M ∈ R N × J the manifest variables matrix and Y ∈ R N × L Y \in\mathbb{R}^{N \times L} Y ∈ R N × L the outcome matrix.
Non-latent models
In their simplest form, linear mediation models are specified by the following set of structural causal equations (see Figure 1 -left):
Figure 1. Non-latent (left-panel) and latent (middle-panel) mediation systems, and overview of several associated inference procedures (right-panel) — (left and middle panels) Variables X X X , M M M , S S S and Y Y Y are specified in the ovals, structural coefficients matrices β \beta β are shown in rounded rectangles, and the system’s dimensions are given in white squares. Arrows represent causal relations between variables. For variables with a causal path to the outcome, squares’ color represents the total causal effect of a unit intervention on the corresponding variable on the outcome Y Y Y under the model specified in Eq. (2) –in this case, the sum of all causal paths leading to Y Y Y ; in the latent graph, the colors of the manifest-variable squares represent their conditional associations with Y Y Y , not causal effects. For both squares and arrows, blue represents a negative relation, red represents a positive relation, and color intensity represents its strength. (right-panel) Common optimization approaches to mediation systems and the order in which they infer parameters. Alt text: three-panel schematic comparing non-latent and latent mediation systems and their inference strategies. Exposures influence outcomes directly and through latent mediators; red and blue arrows indicate positive and negative relationships. The right panel groups methods by two-step, three-step, or fully joint estimation.
X = ε x M = X β ( x m ) + ε m Y = X β ( x y ) + M β ( m y ) + ε y (1) \begin{aligned}
X &= \varepsilon_x\\
M &= X\beta^{(xm)}+\varepsilon_m\\
Y &= X\beta^{(xy)}+M\beta^{(my)}+\varepsilon_y \tag*{(1)}
\end{aligned} X M Y = ε x = X β ( x m ) + ε m = X β ( x y ) + M β ( m y ) + ε y (1)
The random vectors ε x , n \varepsilon_{x,n} ε x , n , ε m , n \varepsilon_{m,n} ε m , n and ε y , n \varepsilon_{y,n} ε y , n corresponding to the n t h n^{\mathrm{th}} n th observation are usually assumed to be mean-zero Gaussians and mutually independent. Formally, for each observation n n n , ε x , n ∼ N ( 0 , Σ x ) \varepsilon_{x,n}\sim\mathcal{N}(0,\Sigma_x) ε x , n ∼ N ( 0 , Σ x ) , ε m , n ∼ N ( 0 , Σ m ∣ x ) \varepsilon_{m,n}\sim\mathcal{N}(0,\Sigma_{m\mid x}) ε m , n ∼ N ( 0 , Σ m ∣ x ) and ε y , n ∼ N ( 0 , Σ y ∣ x , m ) \varepsilon_{y,n}\sim\mathcal{N}(0,\Sigma_{y\mid x,m}) ε y , n ∼ N ( 0 , Σ y ∣ x , m ) ; these terms account for noise and/or exogenous variables. Coefficient matrices are β ( x m ) ∈ R I × J \beta^{(xm)}\in\mathbb{R}^{I\times J} β ( x m ) ∈ R I × J , β ( x y ) ∈ R I × L \beta^{(xy)}\in\mathbb{R}^{I\times L} β ( x y ) ∈ R I × L , β ( m y ) ∈ R J × L \beta^{(my)}\in\mathbb{R}^{J\times L} β ( m y ) ∈ R J × L and noise covariances are Σ m ∣ x ∈ R J × J \Sigma_{m\mid x}\in\mathbb{R}^{J\times J} Σ m ∣ x ∈ R J × J , Σ y ∣ x , m ∈ R L × L \Sigma_{y\mid x,m}\in\mathbb{R}^{L\times L} Σ y ∣ x , m ∈ R L × L . We consider that all variables are centered. As shown in Web Appendix 6, the inclusion of confounders C C C between X X X , M M M and Y Y Y is straightforward and is therefore omitted here for clarity.
According to recent surveys [17 , 20 ] (Zhu et al. 2024), most high-dimensional mediation analyses published in the literature rely on non-latent linear mediation models, under which the parameters of Eq. (1) are estimated using sequential regression procedures combined with penalization. Classical mediation analysis is based on ordinary least squares (OLS) estimation of β ( x m ) \beta^{(xm)} β ( x m ) , β ( m y ) \beta^{(my)} β ( m y ) and β ( x y ) \beta^{(xy)} β ( x y ) , while extensions to high-dimensional settings incorporate regularization to select the relevant mediators or improve estimation or prediction accuracy. Typical examples include the LASSO, Ridge and ElasticNet regressions. Another example is the Pathway LASSO model (Zhao and Luo, 2022), which introduces a nonlinear penalty targeting entire indirect paths rather than individual coefficients. These approaches allow for straightforward coefficient estimation, quickly converge to global optima, and provide readily interpretable results. On the other hand, they heavily rely on hyperparameter tuning and may yield unstable and selection-inconsistent estimates in the presence of strongly correlated mediators or high-dimensional regimes, where mediator selection can become effectively arbitrary [18 ] (Ye, Zhu, and Coffman 2021).
Latent models
Latent mediators.
Structural equations. To overcome some of these limitations, latent models have been suggested, which assume the existence of unobserved variables, denoted by S ∈ R N × K S\in\mathbb{R}^{N\times K} S ∈ R N × K , where K K K is the number of latent dimensions. Following the latent mediation formulations of [1 , 7 ] , LaMe is specified by the following structural equations (see Figure 1 -middle):
X = ε x S = X β ( x s ) + ε s M = S β ( s m ) + ε m Y = X β ( x y ) + S β ( s y ) + ε y (2) \begin{aligned}
X &= \varepsilon_x\\
S &= X\beta^{(xs)}+\varepsilon_s\\
M &= S\beta^{(sm)}+\varepsilon_m\\
Y &= X\beta^{(xy)}+S\beta^{(sy)}+\varepsilon_y \tag*{(2)}
\end{aligned} X S M Y = ε x = X β ( x s ) + ε s = S β ( s m ) + ε m = X β ( x y ) + S β ( sy ) + ε y (2)
The coefficient matrices of the model are β ( x s ) ∈ R I × K \beta^{(xs)}\in\mathbb{R}^{I\times K} β ( x s ) ∈ R I × K , β ( s m ) ∈ R K × J \beta^{(sm)}\in\mathbb{R}^{K\times J} β ( s m ) ∈ R K × J , β ( x y ) ∈ R I × L \beta^{(xy)}\in\mathbb{R}^{I\times L} β ( x y ) ∈ R I × L , β ( s y ) ∈ R K × L \beta^{(sy)}\in\mathbb{R}^{K\times L} β ( sy ) ∈ R K × L . The noise vectors ε x , n \varepsilon_{x,n} ε x , n , ε s , n \varepsilon_{s,n} ε s , n , ε m , n \varepsilon_{m,n} ε m , n and ε y , n \varepsilon_{y,n} ε y , n corresponding to the n t h n^{\mathrm{th}} n th observation are assumed to be mean-zero Gaussians and mutually independent. Formally, for each observation n n n , ε x , n ∼ N ( 0 , Σ x ) \varepsilon_{x,n}\sim\mathcal{N}(0,\Sigma_x) ε x , n ∼ N ( 0 , Σ x ) , ε s , n ∼ N ( 0 , Σ s ) \varepsilon_{s,n}\sim\mathcal{N}(0,\Sigma_s) ε s , n ∼ N ( 0 , Σ s ) , ε m , n ∼ N ( 0 , Σ m ∣ s ) \varepsilon_{m,n}\sim\mathcal{N}(0,\Sigma_{m\mid s}) ε m , n ∼ N ( 0 , Σ m ∣ s ) and ε y , n ∼ N ( 0 , Σ y ∣ x , s ) \varepsilon_{y,n}\sim\mathcal{N}(0,\Sigma_{y\mid x,s}) ε y , n ∼ N ( 0 , Σ y ∣ x , s ) , for some noise covariance matrices Σ x ∈ R I × I \Sigma_x\in\mathbb{R}^{I\times I} Σ x ∈ R I × I , Σ s ∈ R K × K \Sigma_s\in\mathbb{R}^{K\times K} Σ s ∈ R K × K , Σ m ∣ s ∈ R J × J \Sigma_{m\mid s}\in\mathbb{R}^{J\times J} Σ m ∣ s ∈ R J × J , Σ y ∣ x , s ∈ R L × L \Sigma_{y\mid x,s}\in\mathbb{R}^{L\times L} Σ y ∣ x , s ∈ R L × L .
To simplify the exposition, we omit observed confounders (say C C C ) from the subsequent derivations. As shown in Web Appendix 6, the inclusion of confounders between X X X , S S S and Y Y Y is straightforward and requires only replacing the exposure variable with a composite variable ξ : = [ X , C ] \xi:= [X,C] ξ := [ X , C ] , and appropriately partitioning the estimated coefficients corresponding to X X X and C C C .
Natural effects. Under sequential ignorability, namely { Y ( x , s ) , S ( x ′ ) } ⊥ X ∣ C = c \{Y(x,s),S(x')\}\perp X\mid C=c { Y ( x , s ) , S ( x ′ )} ⊥ X ∣ C = c and Y ( x , s ) ⊥ S ( x ′ ) ∣ X = x ′ , C = c Y(x,s)\perp S(x')\mid X=x',C=c Y ( x , s ) ⊥ S ( x ′ ) ∣ X = x ′ , C = c where Y ( x , s ) Y(x,s) Y ( x , s ) and S ( x ′ ) S(x') S ( x ′ ) denote the counterfactual outcome and mediator under the corresponding interventions, the natural direct and indirect effects associated with a one-unit increase in the exposures are given by [1 , 7 ] :
N D E = β ( x y ) ; N I E = β ( x s ) β ( s y ) , (3) NDE=\beta^{(xy)}\quad;\qquad NIE=\beta^{(xs)}\beta^{(sy)}, \tag*{(3)} N D E = β ( x y ) ; N I E = β ( x s ) β ( sy ) , (3)
Stepwise approaches.
Stepwise approaches to latent mediation generally define S S S as a deterministic linear combination of manifest variables. These methods typically derive latent scores from an initial model, for instance from Principal Component Analysis (PCA), Partial Least Squares (PLS) or their variants; then, these scores enter as observed mediators in the subsequent estimation of a non-latent mediation model [4 , 11 , 21 ] (Hu, DeSimone, and Wang 2024). However, most existing stepwise approaches do not use all available data to derive their latent scores. For instance, approaches based on PCA optimize S S S to explain the variance of M M M , and approaches based on PLS optimize S S S to explain the covariance between X X X and M M M , or M M M and Y Y Y , regardless of whether the resulting scores best characterize mediation between X X X and Y Y Y . Conversely, a correctly specified model accounting for the joint variance and covariance of ( X , M , Y ) (X, M, Y) ( X , M , Y ) does not require latent scores to be selected using only a subset of the observed relationships.
Joint modeling
A joint approach that maximizes a likelihood corresponding to the full structural model may therefore be preferable.
The latent approaches of [12 ] (Chén et al. 2017) both consider an univariate exposure and outcome, and both estimate the first latent direction and the associated path coefficients from a joint objective based on all observed data. In (Chén et al., 2017), the model extracts subsequent directions through approximate deflation while keeping earlier directions fixed. Consequently, the retained directions generally differ from those estimated by maximizing the joint likelihood of a full multivariate latent model; in particular, the sequential objective does not include cross-score residual covariances. Finally, the authors remark that inference may become cumbersome when J ≫ 1 J \gg1 J ≫ 1 without resorting to additional data augmentation techniques. The model introduced in [12 ] extends this approach beyond linear relationships to model S S S as an arbitrary deterministic function of M M M . However, this extension is limited to a single latent variable. Moreover, score selection and the estimated direct–indirect decomposition are sensitive to the scale of Y Y Y , as shown in Web Appendix 9. Finally, standardization of the univariate score is imposed algorithmically rather than incorporated into the objective function, complicating the analytical characterization of the resulting estimator.
Joint probabilistic formulations, on the other hand, explicitly account for invariance to latent rescaling. In [1 ] , the authors formulate a joint probabilistic latent model with a single latent mediator. The approach was later extended by [7 ] to handle multivariate latent mediators. In that work, the authors assume a univariate exposure and outcome, and constrain the conditional noise covariance of manifest variables given latent mediators Σ m ∣ s \Sigma_{m|s} Σ m ∣ s to be diagonal. In this setting, however, we show in Web Appendix 10 that the non-regularized and correctly specified model from [7 ] is equivalent to Factor Analysis of M M M followed by linear regressions on the resulting S S S when K > 1 K > 1 K > 1 . Additionally, the diagonal constraint on Σ m ∣ s \Sigma_{m|s} Σ m ∣ s imposes conditional independence of manifest variables given the latent mediators, which is not necessarily true when M M M represents biomarkers. Finally, Condition 1 in [7 ] is required for identification of the effects, but the proposed model does not guarantee that this condition holds.
Methods
We develop LaMe to extend existing approaches by jointly modeling all observed variables, allowing non-diagonal noise covariance matrices and multivariate exposures, outcomes and latent information bottleneck. Under the exact recoverability assumption for S S S , the framework also permits identification of natural effects.
We first detail the exact recoverability and information bottleneck assumptions made by our approach, then we develop a general inference procedure for an auxiliary model that does not assume exact recoverability, and we finally show in Proposition 1 that exact recoverability can be recovered through a likelihood-equivalent reparametrization of the model post-estimation. This proposition further establishes that the parametric representations of natural direct and indirect effects are statistically identified under this condition.
Assumptions
Exact recoverability
Following score-based mediation approaches, which define latent mediators as linear combinations of manifest variables [12 , 21 ] (Chén et al. 2017), we assume the same kind of deterministic relationship holds within our probabilistic framework. Specifically, we will work under what we call the exact recoverability assumption, under which S = E [ S ∣ M ] : = M β ( m s ) S = \mathbb{E}[S|M] := M\beta^{(\mathrm{ms})} S = E [ S ∣ M ] := M β ( ms ) . We emphasize that this equality does not imply that the measurements M M M are causes of S S S ; they remain the manifest variables causally influenced by S S S , with S S S causally influenced by X X X . This requirement is equivalent to:
Σ m ∣ s β ( m s ) = 0 (4) \Sigma_{m|s}\beta^{(\mathrm{ms})} = 0 \tag*{(4)} Σ m ∣ s β ( ms ) = 0 (4)
where β ( m s ) = ( ( β ( s m ) ) T Σ s β ( s m ) + Σ m ∣ s ) − 1 ( β ( s m ) ) T ( Σ s ∣ x + ( β ( x s ) ) T Σ x β ( x s ) ) \beta^{(\mathrm{ms})} = \left((\beta^{(\mathrm{sm})})^{\mathsf{T}}\Sigma_s\beta^{(\mathrm{sm})}+\Sigma_{m|s}\right)^{-1}(\beta^{(\mathrm{sm})})^{\mathsf{T}}\left(\Sigma_{s|x}+(\beta^{(\mathrm{xs})})^{\mathsf{T}}\Sigma_x\beta^{(\mathrm{xs})}\right) β ( ms ) = ( ( β ( sm ) ) T Σ s β ( sm ) + Σ m ∣ s ) − 1 ( β ( sm ) ) T ( Σ s ∣ x + ( β ( xs ) ) T Σ x β ( xs ) ) .
We point out that the exact-recoverability approximation becomes increasingly accurate as the number of manifest variables J J J grows, making it well suited to high-dimensional settings. To see this, consider the difference between the unconstrained latent variable S ′ S' S ′ in Eq. (2) and its exactly recoverable counterpart S ∗ = M β ( m s ) S^* = M\beta^{(\mathrm{ms})} S ∗ = M β ( ms ) . Assuming that additional manifest variables provide non-vanishing, non-redundant information about S S S , the measurement precision β ( s m ) Σ m ∣ s − 1 ( β ( s m ) ) T \beta^{(\mathrm{sm})}\Sigma_{m|s}^{-1}(\beta^{(\mathrm{sm})})^{\mathsf{T}} β ( sm ) Σ m ∣ s − 1 ( β ( sm ) ) T grows linearly with J J J . Then, we show in Web Appendix 7 that E [ ∥ S ∗ − S ′ ∥ 2 ] = O ( J − 1 ) \mathbb{E}[\lVert S^* - S'\rVert^2] = \mathcal{O}(J^{-1}) E [∥ S ∗ − S ′ ∥ 2 ] = O ( J − 1 ) , meaning that S ′ → J → ∞ S ∗ S' \xrightarrow{J\to\infty} S^* S ′ J → ∞ S ∗ . Thus, as the number of informative manifest variables increases, the latent variable S S S of the unconstrained model in Eq. (2) asymptotically converges to the latent mediator that is exactly recoverable from M M M . Although exact recoverability is an idealization for finite J J J , it therefore becomes a reasonable approximation when sufficiently many informative manifest variables are available, supporting its use in high-dimensional settings.
We show in Proposition 1 below that the joint model specified in Eq. (2) can always be formulated to satisfy exact recoverability.
Information bottleneck
Introducing low-dimensional latent variables addresses several limitations of non-latent least-squares mediation models. When the number of observations is finite, sampling noise can introduce spurious association patterns; by imposing a low-rank structure on the cross-covariance between X X X and M M M , and between M M M and Y Y Y , latent models alleviate this by forcing the model to focus on the most relevant covariation patterns, typically preventing overfitting the noise and improving robustness and generalization of structural parameters β ( x s ) \beta^{(xs)} β ( x s ) , β ( s m ) \beta^{(sm)} β ( s m ) , β ( x y ) \beta^{(xy)} β ( x y ) , β ( s y ) \beta^{(sy)} β ( sy ) . In this sense, the latent representation acts as an information bottleneck between observed variables [5 ] .
Likewise, information bottlenecks can suppress weaker associations between observed variables, allowing the system to be approximated by a simplified representation that only captures its most defining patterns of covariation, which is a recurring objective of dimension-reduction approaches to mediation analysis.
Under the assumptions from Eq. (2) (linear relations, full noise covariances and mutually independent noises), a necessary and sufficient condition for an information bottleneck can be formulated as (see Web Appendix 8 for details):
K < rank ( Cov ( X , M ) Cov ( Y , M ∣ X ) ) K < \operatorname{rank}\begin{pmatrix} \operatorname{Cov}(X,M) \\ \operatorname{Cov}(Y,M \mid X) \end{pmatrix} K < rank ( Cov ( X , M ) Cov ( Y , M ∣ X ) )
where
rank ( Cov ( X , M ) Cov ( Y , M ∣ X ) ) ≤ min ( I + L , J ) , (5) \operatorname{rank}\begin{pmatrix} \operatorname{Cov}(X,M) \\ \operatorname{Cov}(Y,M \mid X) \end{pmatrix} \le\min(I+L,J), \tag*{(5)} rank ( Cov ( X , M ) Cov ( Y , M ∣ X ) ) ≤ min ( I + L , J ) , (5)
where Cov ( ⋅ , ⋅ ) \operatorname{Cov}(\cdot,\cdot) Cov ( ⋅ , ⋅ ) denotes an observed covariance. Conversely, K ≥ min ( J , I + L ) K \ge\min(J,I+L) K ≥ min ( J , I + L ) is a sufficient condition ensuring that every possible cross-covariance matrix can be represented exactly. Then, the observable regression coefficients between X X X , M M M and Y Y Y estimated from the latent model in Eq. (2) are a reparametrization of those estimated by a non-latent OLS model, as shown in Web Appendix 8, limiting the interest of the approach. Note that assuming a diagonal conditional noise covariance Σ m ∣ s \Sigma_{m\mid s} Σ m ∣ s does not affect the condition above. In practice, the latent dimension K K K can be selected via cross-validation or information criteria such as AIC or BIC to balance model complexity and goodness-of-fit.
LaMe model
Likelihood
Starting from Eq. (2) , θ = { β ( x s ) , β ( s m ) , β ( x y ) , β ( s y ) , Σ x , Σ s ∣ x , Σ m ∣ s , Σ y ∣ x , s } \theta= \{\beta^{(xs)}, \beta^{(sm)}, \beta^{(xy)}, \beta^{(sy)}, \Sigma_x, \Sigma_{s\mid x}, \Sigma_{m\mid s}, \Sigma_{y\mid x,s}\} θ = { β ( x s ) , β ( s m ) , β ( x y ) , β ( sy ) , Σ x , Σ s ∣ x , Σ m ∣ s , Σ y ∣ x , s } denote the full set of model parameters, and p ( θ ) p(\theta) p ( θ ) its prior distribution. Assuming that mean-zero Gaussian noises are mutually independent in Eq. (2) , a natural factorization of the joint distribution is
p ( X , S , M , Y , θ ) = p ( X ∣ θ ) p ( S ∣ X , θ ) p ( M ∣ S , θ ) p ( Y ∣ X , S , θ ) p ( θ ) (6) p(X,S,M,Y,\theta) = p(X\mid\theta)p(S\mid X,\theta)p(M\mid S,\theta)p(Y\mid X,S,\theta)p(\theta) \tag*{(6)} p ( X , S , M , Y , θ ) = p ( X ∣ θ ) p ( S ∣ X , θ ) p ( M ∣ S , θ ) p ( Y ∣ X , S , θ ) p ( θ ) (6)
Recall that all the distributions above are Gaussian, with parameters specified as, for example, p ( S ∣ X , θ ) = N ( S ; X β ( x s ) , Σ s ∣ x ) p(S\mid X,\theta)=\mathcal{N}(S;X\beta^{(xs)},\Sigma_{s\mid x}) p ( S ∣ X , θ ) = N ( S ; X β ( x s ) , Σ s ∣ x ) . In the following derivations, we assume a flat prior p ( θ ) p(\theta) p ( θ ) on all parameters, so that p ( X , S , M , Y , θ ) ∝ p ( X , S , M , Y ∣ θ ) p(X,S,M,Y,\theta)\propto p(X,S,M,Y\mid\theta) p ( X , S , M , Y , θ ) ∝ p ( X , S , M , Y ∣ θ ) . Other priors that regularize the parameters, together with an extension to log-normal variables, are discussed in Web Appendix 13.
The latent nature of S S S prevents direct optimization of (6). We can, however, compute the incomplete likelihood by marginalizing the complete likelihood over all possible values of S S S . Using the complete likelihood expression (6), a standard Gaussian integration formula and grouping terms together, we find
p ( X , M , Y ∣ θ ) = ∫ S e − 1 2 ⟨ S T , S ⟩ A + ⟨ S , B ⟩ + C p ( S , X , M , Y ∣ θ ) d S = e 1 2 ⟨ B T , B , A − 1 ⟩ e C ( 2 π ) N K 2 ∣ A ∣ − N 2 (7) p(X,M,Y\mid\theta)=\int_S\frac{e^{-\frac{1}{2}\langle S^T,S\rangle_A+\langle S,B\rangle+C}}{p(S,X,M,Y\mid\theta)}\,dS=e^{\frac{1}{2}\langle B^T,B,A^{-1}\rangle}e^C(2\pi)^{\frac{NK}{2}}\lvert A\rvert^{-\frac{N}{2}} \tag*{(7)} p ( X , M , Y ∣ θ ) = ∫ S p ( S , X , M , Y ∣ θ ) e − 2 1 ⟨ S T , S ⟩ A + ⟨ S , B ⟩ + C d S = e 2 1 ⟨ B T , B , A − 1 ⟩ e C ( 2 π ) 2 N K ∣ A ∣ − 2 N (7)
where A : = Σ s ∣ x − 1 + β ( s m ) Σ m ∣ s − 1 ( β ( s m ) ) T + β ( s y ) Σ y ∣ x , s − 1 ( β ( s y ) ) T A:=\Sigma_{s\mid x}^{-1}+\beta^{(sm)}\Sigma_{m\mid s}^{-1}(\beta^{(sm)})^T+\beta^{(sy)}\Sigma_{y\mid x,s}^{-1}(\beta^{(sy)})^T A := Σ s ∣ x − 1 + β ( s m ) Σ m ∣ s − 1 ( β ( s m ) ) T + β ( sy ) Σ y ∣ x , s − 1 ( β ( sy ) ) T
and B : = X β ( x s ) Σ s ∣ x − 1 + M Σ m ∣ s − 1 ( β ( s m ) ) T + ( Y − X β ( x y ) ) Σ y ∣ x , s − 1 ( β ( s y ) ) T B:=X\beta^{(xs)}\Sigma_{s\mid x}^{-1}+M\Sigma_{m\mid s}^{-1}(\beta^{(sm)})^T+(Y-X\beta^{(xy)})\Sigma_{y\mid x,s}^{-1}(\beta^{(sy)})^T B := X β ( x s ) Σ s ∣ x − 1 + M Σ m ∣ s − 1 ( β ( s m ) ) T + ( Y − X β ( x y ) ) Σ y ∣ x , s − 1 ( β ( sy ) ) T .
Above, ∣ A ∣ \lvert A\rvert ∣ A ∣ denotes the determinant of A A A and ⟨ A , B ⟩ : = ∑ i , j A i j B i j \langle A,B\rangle:=\sum_{i,j}A_{ij}B_{ij} ⟨ A , B ⟩ := ∑ i , j A ij B ij is the Frobenius product. The expression of the normalization term C C C is provided in Web Appendix 12.
The expectation-maximization algorithm then first computes the expectation of latent variables given the current parameter estimates θ ( t ) \theta^{(t)} θ ( t ) (E-step), and then derives the new optimal parameter set θ ( t + 1 ) \theta^{(t+1)} θ ( t + 1 ) given the expectation of latent variables computed in the previous step (M-step). This procedure is iterated until convergence.
E-step
Given the parameter estimates at the t t h t^{\mathrm{th}} t th iteration, θ ( t ) \theta^{(t)} θ ( t ) , the E-step computes the conditional distribution of the latent variables S S S and evaluates expectations of functions of S S S with respect to this distribution.
We show in Web Appendix 11 that, for any candidate distribution q θ ( t ) ( S ) q_{\theta^{(t)}}(S) q θ ( t ) ( S ) , the observed log-likelihood is lower-bounded as:
log p ( X , M , Y ∣ θ ) ≥ ∫ S q θ ( t ) ( S ) log p ( S , X , M , Y ∣ θ ) q θ ( t ) ( S ) d S (8) \log p(X,M,Y\mid\theta) \ge\int_S q_{\theta^{(t)}}(S)\log\frac{p(S,X,M,Y\mid\theta)}{q_{\theta^{(t)}}(S)}\,dS \tag*{(8)} log p ( X , M , Y ∣ θ ) ≥ ∫ S q θ ( t ) ( S ) log q θ ( t ) ( S ) p ( S , X , M , Y ∣ θ ) d S (8)
Then, it can be shown that this lower-bound is equal to log p ( X , M , Y ∣ θ ( t ) ) \log p(X,M,Y\mid\theta^{(t)}) log p ( X , M , Y ∣ θ ( t ) ) , and therefore maximal, when:
q θ ( t ) ( S ) = p ( S ∣ X , M , Y , θ ( t ) ) = p ( S , X , M , Y ∣ θ ( t ) ) p ( X , M , Y ∣ θ ( t ) ) (9) q_{\theta^{(t)}}(S)=p(S\mid X,M,Y,\theta^{(t)})=\frac{p(S,X,M,Y\mid\theta^{(t)})}{p(X,M,Y\mid\theta^{(t)})} \tag*{(9)} q θ ( t ) ( S ) = p ( S ∣ X , M , Y , θ ( t ) ) = p ( X , M , Y ∣ θ ( t ) ) p ( S , X , M , Y ∣ θ ( t ) ) (9)
A full derivation of this result can be found in Web Appendix 11.
We finally use q θ ( t ) ( S ) q_{\theta^{(t)}}(S) q θ ( t ) ( S ) to compute the moments of S S S under the current parameter estimates before proceeding to the M-step, recalling that N N N is the number of observations:
S ‾ ( t ) = B A − 1 ; S T S ‾ ( t ) = N A − 1 + A − 1 B T B A − 1 (10) \overline{S}^{(t)}=BA^{-1}\quad;\quad\overline{S^TS}^{(t)}=NA^{-1}+A^{-1}B^TBA^{-1} \tag*{(10)} S ( t ) = B A − 1 ; S T S ( t ) = N A − 1 + A − 1 B T B A − 1 (10)
M M M -step
The M-step then computes the parameter set θ ( t + 1 ) \theta^{(t+1)} θ ( t + 1 ) that maximizes the lower-bound in Eq. (8) with q θ ( t ) ( S ) q_{\theta^{(t)}}(S) q θ ( t ) ( S ) from Eq. (9) . Taking β ( s m ) \beta^{(sm)} β ( s m ) as an illustration:
∂ β ( s m ) ∫ S q θ ( t ) ( S ) log p ( S , X , M , Y ∣ θ ) d S = − S T S ‾ ( t ) β ( s m ) Σ m ∣ s − 1 + S ‾ ( t ) T M Σ m ∣ s − 1 = 0 ⇔ β ^ ( s m ) = ( S T S ‾ ( t ) ) − 1 S ‾ ( t ) T M (11) \begin{aligned}
\partial_{\beta^{(sm)}}\int_S q_{\theta^{(t)}}(S)\log p(S,X,M,Y\mid\theta)dS=-\overline{S^T S}^{(t)}\beta^{(sm)}\Sigma_{m\mid s}^{-1}+\overline{S}^{(t)T}M\Sigma_{m\mid s}^{-1}=0 \\
&\Leftrightarrow\ \hat{\beta}^{(sm)}=\left(\overline{S^T S}^{(t)}\right)^{-1}\overline{S}^{(t)T}M \tag*{(11)}
\end{aligned} ∂ β ( s m ) ∫ S q θ ( t ) ( S ) log p ( S , X , M , Y ∣ θ ) d S = − S T S ( t ) β ( s m ) Σ m ∣ s − 1 + S ( t ) T M Σ m ∣ s − 1 = 0 ⇔ β ^ ( s m ) = ( S T S ( t ) ) − 1 S ( t ) T M (11)
where S ‾ ( t ) \overline{S}^{(t)} S ( t ) and S T S ‾ ( t ) \overline{S^T S}^{(t)} S T S ( t ) are obtained from Eq. (10) . Detailed derivations for other parameters are provided in Web Appendix 12.
The EM algorithm alternates between the E-step, where S ‾ ( t ) \overline{S}^{(t)} S ( t ) and S T S ‾ ( t ) \overline{S^T S}^{(t)} S T S ( t ) are computed given the current parameter estimates θ ( t ) \theta^{(t)} θ ( t ) , and the M-step, where the parameter estimates are updated to θ ( t + 1 ) \theta^{(t+1)} θ ( t + 1 ) given S ‾ ( t ) \overline{S}^{(t)} S ( t ) and S T S ‾ ( t ) \overline{S^T S}^{(t)} S T S ( t ) , until convergence of the incomplete likelihood. The EM algorithm guarantees a non-decreasing likelihood and convergence to a stationary point (Neal and Hinton, [14 ] ). Note that although we consider all parameters θ \theta θ to be random variables, the EM optimization algorithm yields point estimates rather than their distribution.
At this point, S S S may not be exactly recoverable from M M M as assumed in Eq. (4) . In the next section, we show how to satisfy this assumption through a likelihood-equivalent reparametrization of the EM solution.
Identifiability
The parametric representation of the observed moments recalled in Web Appendix 8 shows that the observed likelihood in Eq. (7) is invariant to some transformations of the latent space; the exact recoverability property limits these to invertible linear transformations, which makes natural effects identifiable.
Proposition 1 (Statistical identification of the natural effects). The natural direct and indirect effect through S S S as defined in Eq. (3) , namely β ( x y ) \beta^{(xy)} β ( x y ) and β ( x s ) β ( s y ) \beta^{(xs)}\beta^{(sy)} β ( x s ) β ( sy ) , are uniquely identified from the joint distribution of ( X , M , Y ) (X,M,Y) ( X , M , Y ) under the exact recoverability assumption S = E [ S ∣ M ] S=\mathbb{E}[S\mid M] S = E [ S ∣ M ] when the bottleneck condition from Eq. (5) is satisfied.
Proof. [(Proposition 1)]
Likelihood-equivalent reparametrizations. Let H M ∈ R K × K H_M\in\mathbb{R}^{K\times K} H M ∈ R K × K and H Y ∈ R K × K H_Y\in\mathbb{R}^{K\times K} H Y ∈ R K × K denote invertible matrices. Then, we define the elements of a set of transformed parameters denoted by θ ∗ \theta^* θ ∗ expressed in terms of θ \theta θ , H M H_M H M and H Y H_Y H Y as:
β ∗ ( x s ) = β ( x s ) H M ; Σ s ∣ x ∗ = H M T Σ s ∣ x H Y β ∗ ( s m ) = H M − 1 β ( s m ) ; Σ m ∣ s ∗ = Σ m ∣ s + ( β ( s m ) T Σ s ∣ x β ( s m ) − β ( s m ) T Σ s ∣ x ∗ β ( s m ) ) β ∗ ( s y ) = H Y − 1 β ( s y ) ; Σ y ∣ x , s ∗ = Σ y ∣ x , s + ( β ( s y ) T Σ s ∣ x β ( s y ) − β ( s y ) T Σ s ∣ x ∗ β ( s y ) ) β ∗ ( x y ) = β ( x y ) + β ( x s ) β ( s y ) − β ∗ ( x s ) β ∗ ( s y ) ; Σ x ∗ = Σ x (12) \begin{aligned}
\beta_*^{(xs)}&=\beta^{(xs)}H_M \quad;& \Sigma_{s\mid x}^*&=H_M^T\Sigma_{s\mid x}H_Y\\
\beta_*^{(sm)}&=H_M^{-1}\beta^{(sm)} \quad;& \Sigma_{m\mid s}^*&=\Sigma_{m\mid s}+\left(\beta^{(sm)T}\Sigma_{s\mid x}\beta^{(sm)}-\beta^{(sm)T}\Sigma_{s\mid x}^*\beta^{(sm)}\right)\\
\beta_*^{(sy)}&=H_Y^{-1}\beta^{(sy)} \quad;& \Sigma_{y\mid x,s}^*&=\Sigma_{y\mid x,s}+\left(\beta^{(sy)T}\Sigma_{s\mid x}\beta^{(sy)}-\beta^{(sy)T}\Sigma_{s\mid x}^*\beta^{(sy)}\right)\\
\beta_*^{(xy)}&=\beta^{(xy)}+\beta^{(xs)}\beta^{(sy)}-\beta_*^{(xs)}\beta_*^{(sy)} \quad;& \Sigma_x^*&=\Sigma_x \tag*{(12)}
\end{aligned} β ∗ ( x s ) β ∗ ( s m ) β ∗ ( sy ) β ∗ ( x y ) = β ( x s ) H M ; = H M − 1 β ( s m ) ; = H Y − 1 β ( sy ) ; = β ( x y ) + β ( x s ) β ( sy ) − β ∗ ( x s ) β ∗ ( sy ) ; Σ s ∣ x ∗ Σ m ∣ s ∗ Σ y ∣ x , s ∗ Σ x ∗ = H M T Σ s ∣ x H Y = Σ m ∣ s + ( β ( s m ) T Σ s ∣ x β ( s m ) − β ( s m ) T Σ s ∣ x ∗ β ( s m ) ) = Σ y ∣ x , s + ( β ( sy ) T Σ s ∣ x β ( sy ) − β ( sy ) T Σ s ∣ x ∗ β ( sy ) ) = Σ x (12)
Finally, in the absence of additional assumptions, the relation P ( X , M , Y ∣ θ ) = P ( X , M , Y ∣ θ ∗ ) P(X,M,Y\mid\theta)=P(X,M,Y\mid\theta^*) P ( X , M , Y ∣ θ ) = P ( X , M , Y ∣ θ ∗ ) holds. We demonstrate in Web Appendix 14 that no other likelihood-equivalent reparametrizations exist when the bottleneck condition from Eq. (5) is satisfied.
Parametrizations with exact recoverability. From the definition in Eq. (4) and noting that Σ m ∗ = Σ m \Sigma_{m}^*=\Sigma_m Σ m ∗ = Σ m (see Web Appendix 15), we derive an expression of H M H Y − 1 H_MH_Y^{-1} H M H Y − 1 such that they define a reparametrization that satisfies the exact recoverability assumption, meaning Σ m ∣ s ∗ Σ s ∣ m − 1 ( β ∗ ( s m ) ) T Σ s ∗ = 0 \Sigma_{m\mid s}^*\Sigma_{s\mid m}^{-1}(\beta_*^{(sm)})^T\Sigma_s^*=0 Σ m ∣ s ∗ Σ s ∣ m − 1 ( β ∗ ( s m ) ) T Σ s ∗ = 0 , as:
H M H Y − 1 = ( ( β ( s m ) Σ m − 1 β ( s m ) T ) − 1 − ( β ( x s ) T Σ x β ( x s ) ) − 1 ) − 1 Σ s ∣ x H_MH_Y^{-1}=\left(\left(\beta^{(sm)}\Sigma_m^{-1}\beta^{(sm)T}\right)^{-1}-\left(\beta^{(xs)T}\Sigma_x\beta^{(xs)}\right)^{-1}\right)^{-1}\Sigma_{s\mid x} H M H Y − 1 = ( ( β ( s m ) Σ m − 1 β ( s m ) T ) − 1 − ( β ( x s ) T Σ x β ( x s ) ) − 1 ) − 1 Σ s ∣ x
Note that the relation above is defined as long as β ( s m ) Σ m − 1 β ( s m ) T \beta^{(sm)}\Sigma_m^{-1}\beta^{(sm)T} β ( s m ) Σ m − 1 β ( s m ) T is non-singular, which is necessarily true when the bottleneck condition from Eq. (5) is satisfied. We demonstrate in Web Appendix 15 that the expression above also guarantees the positive semi-definiteness of all transformed noise covariances Σ s ∣ x ∗ \Sigma_{s\mid x}^* Σ s ∣ x ∗ , Σ m ∣ s ∗ \Sigma_{m\mid s}^* Σ m ∣ s ∗ and Σ y ∣ x , s ∗ \Sigma_{y\mid x,s}^* Σ y ∣ x , s ∗ . Therefore, any choice of an invertible H M H_M H M , along with the corresponding H Y H_Y H Y from the equation above, guarantees the assumption from Eq. (4) is satisfied.
Invariance of natural effects under exact recoverability. Note that β ( s m ) Σ m − 1 β ( s m ) T = Σ s − 1 \beta^{(sm)}\Sigma_m^{-1}\beta^{(sm)T}=\Sigma_s^{-1} β ( s m ) Σ m − 1 β ( s m ) T = Σ s − 1 (see Web Appendix 15) when the assumption in Eq. (4) is verified. Injecting it in the expression above yields H M H Y − 1 = I H_MH_Y^{-1}=I H M H Y − 1 = I . Identification of the natural effects coefficients under the exact recoverability assumption immediately follows:
β ∗ ( x y ) = β ( x y ) + β ( x s ) ( I − H M H Y − 1 ) β ( s y ) = β ( x y ) , β ∗ ( x s ) β ∗ ( s y ) = β ( x s ) H M H Y − 1 β ( s y ) = β ( x s ) β ( s y ) \beta_*^{(xy)}=\beta^{(xy)}+\beta^{(xs)}(I-H_MH_Y^{-1})\beta^{(sy)}=\beta^{(xy)},\qquad\beta_*^{(xs)}\beta_*^{(sy)}=\beta^{(xs)}H_MH_Y^{-1}\beta^{(sy)}=\beta^{(xs)}\beta^{(sy)} β ∗ ( x y ) = β ( x y ) + β ( x s ) ( I − H M H Y − 1 ) β ( sy ) = β ( x y ) , β ∗ ( x s ) β ∗ ( sy ) = β ( x s ) H M H Y − 1 β ( sy ) = β ( x s ) β ( sy )
We point out that assumptions other than exact recoverability may also make the NDE and NIE identifiable, although they may not provide similar convergence guarantees when J → ∞ J\to\infty J → ∞ ; for instance, alternative choices may allow S S S to be a noisy function of M M M rather than deterministic. This question lies outside the scope of the present article and is left for future work.
Alternative factorization
The causal model specified in Eq. (2) implies two conditional independence relationships, M ⊥ X ∣ S M \perp X \mid S M ⊥ X ∣ S and Y ⊥ M ∣ X , S Y \perp M \mid X, S Y ⊥ M ∣ X , S , which allow the joint distribution of ( X , S , M , Y ) (X, S, M, Y) ( X , S , M , Y ) to be factorized equivalently in two ways, denoted LaMe-vX and LaMe-vM hereafter:
p ( X , S , M , Y ∣ θ ) = p ( X ∣ θ ) p ( S ∣ X , θ ) p ( M ∣ S , θ ) p ( Y ∣ X , S , θ ) (LaMe-vX) = p ( M ∣ θ ) p ( S ∣ M , θ ) p ( X ∣ S , θ ) p ( Y ∣ X , S , θ ) (LaMe-vM) (13) \begin{aligned}
p(X,S,M,Y \mid\theta) &= p(X \mid\theta)p(S \mid X,\theta)p(M \mid S,\theta)p(Y \mid X,S,\theta) &&\text{(LaMe-vX)}\\
&= p(M \mid\theta)p(S \mid M,\theta)p(X \mid S,\theta)p(Y \mid X,S,\theta) &&\text{(LaMe-vM)} \tag*{(13)}
\end{aligned} p ( X , S , M , Y ∣ θ ) = p ( X ∣ θ ) p ( S ∣ X , θ ) p ( M ∣ S , θ ) p ( Y ∣ X , S , θ ) = p ( M ∣ θ ) p ( S ∣ M , θ ) p ( X ∣ S , θ ) p ( Y ∣ X , S , θ ) (LaMe-vX) (LaMe-vM) (13)
Both factorizations describe the same family of joint distributions and can therefore be used interchangeably for parameter estimation, with parameters from either representation recoverable from the other. Note that this reformulation is not necessarily equivalent in the presence of regularization, orthogonality, or structural constraints—for instance, a diagonal conditional noise covariance in Derkach et al. [7 ] prevents such equivalence. However, it holds for any reparametrization based on Eq. (12) , including those assuming exact recoverability.
A first remark is that alternative factorizations may not correspond to any relevant causal system. For instance, the coefficient β ( s x ) \beta^{(sx)} β ( s x ) in LaMe-vM has no causal interpretation in a mediation analysis setting; however, as shown in Web Appendix 16, it can be used to retrieve the relevant causal path coefficient β ( x s ) \beta^{(xs)} β ( x s ) . Second, structural parameters differ depending on the factorization. Third, optimizing the joint model using one factorization instead of the other can make the EM algorithm orders of magnitude faster and the estimation more accurate when J ≫ I J \gg I J ≫ I : using the LaMe-vM factorization, Σ x ∣ s \Sigma_{x \mid s} Σ x ∣ s and Σ s ∣ m \Sigma_{s \mid m} Σ s ∣ m must be updated (O ( N I ( K + I ) + N K ( K + J ) ) O(NI(K+I)+NK(K+J)) O ( N I ( K + I ) + N K ( K + J )) ) at each iteration whereas Σ m \Sigma_m Σ m needs to be updated (O ( N J 2 ) O(NJ^2) O ( N J 2 ) ) only once. We numerically support this result in Web Appendix 19.
The full table of the update equations for both models is available in Web Appendix 12.
Results
For both the LaMe-vX and LaMe-vM factorizations (see Eq. (13) ), we stop the EM loop once the relative change in the log-likelihood remains below 10 − 5 10^{-5} 1 0 − 5 for ten consecutive iterations. This procedure is repeated 50 times with different random initializations, and the run with the highest likelihood is selected. We consider both unregularized models and models with an ℓ 1 \ell_1 ℓ 1 penalty on the regression coefficients linking S S S and M M M .
Baselines
We evaluate LaMe against OLS, OLS with ℓ 1 \ell_1 ℓ 1 regularizer (LASSO ), OLS with ℓ 1 \ell_1 ℓ 1 and ℓ 2 \ell_2 ℓ 2 regularizers (ElasticNet ), PCA on the manifest variables M M M followed by OLS or LASSO on the resulting ( X , S , Y ) (X,S,Y) ( X , S , Y ) latent mediation system (PCA+OLS and PCA+LASSO ), PLS on X → M X \to M X → M followed by OLS on [ X , S ] → Y [X,S] \to Y [ X , S ] → Y (brackets denote column-wise concatenation) (PLS(X,M)+OLS ), PLS on X → [ M , Y ] X \to[M,Y] X → [ M , Y ] followed by OLS on X → Y r e s X \to Y_{\mathrm{res}} X → Y res (PLS(X,M,Y)+OLS ), OLS on the direct path X → Y X \to Y X → Y followed by PLS on the indirect path X → [ M , Y r e s ] X \to[M,Y_{\mathrm{res}}] X → [ M , Y res ] (OLS+PLS(X,M,Y) ) (where Y r e s Y_{\mathrm{res}} Y res are the residuals of Y Y Y that are not explained by the model used in the first step), an over-parameterized version of LaMe-vX (OVP-LaMe ), and an adaptation of the latent mediation model described in Nath et al. [12 ] (Joint Alternate Optimization - Gaussian case JAOG ). A discussion of our adaptation of Nath et al. [12 ] (originally designed for univariate latent mediators) and why we consider OVP-LaMe rather than the model of Derkach et al. [7 ] is provided in Web Appendix 18.
All experiments were run on an Intel Core Ultra 7 165U CPU; a scalability comparison is provided in Web Appendix 19.
Synthetic data
Data generation.
Synthetic variables X X X , M M M , Y Y Y are generated according to the linear-Gaussian structural equations specified by Eq. (2) when data are assumed to stem from a latent model, and according to Eq. (1) when we assume the absence of a shared latent structure. The dimensions I I I , J J J , L L L and possibly K t r u e K_{\mathrm{true}} K true are specified for each experiment below. Noise covariances are rescaled so the following signal-to-noise ratios (SNR) S N R ( S ∣ X ) SNR(S \mid X) S N R ( S ∣ X ) , S N R ( M ∣ S ) SNR(M \mid S) S N R ( M ∣ S ) , S N R ( Y ∣ X , S ) SNR(Y \mid X,S) S N R ( Y ∣ X , S ) (in the latent case) and S N R ( M ∣ X ) SNR(M \mid X) S N R ( M ∣ X ) , S N R ( Y ∣ X , M ) SNR(Y \mid X,M) S N R ( Y ∣ X , M ) (in the non-latent case) attain a given value (e.g. S N R ( M ∣ S ) = var ( S β ( s m ) ) var ( N ( 0 , Σ m ∣ s ) ) = 25 100 SNR(M \mid S)=\frac{\operatorname{var}(S\beta^{(sm)})}{\operatorname{var}(N(0,\Sigma_{m \mid s}))}=\frac{25}{100} S N R ( M ∣ S ) = var ( N ( 0 , Σ m ∣ s )) var ( S β ( s m ) ) = 100 25 means 20% of the variations in M M M are explained by the signal from S S S , and 80% are due to random noise; see Eq. (2) and Web Appendix 17).
Recovering underlying parameters.
To compare LaMe against baselines that do not assume the existence of latent mediators (e.g. OLS and OVP-LaMe), we first evaluate the capacity of each model to recover the true marginal association between observed variables X X X and M M M (β ( x m ) = ∇ X E [ M ∣ X ] \beta^{(xm)}=\nabla_X\mathbb{E}[M \mid X] β ( x m ) = ∇ X E [ M ∣ X ] ) and between M M M and Y Y Y (β ( m y ) = ∇ M E [ Y ∣ M ] \beta^{(my)}=\nabla_M\mathbb{E}[Y \mid M] β ( m y ) = ∇ M E [ Y ∣ M ] ). Not only do these coefficients provide a common basis for comparing models, they are also involved in the computation of the natural effects under the exact recoverability assumption (i.e. N I E = β ( x s ) H M H Y − 1 β ( s y ) = β ( x m ) β ( m y ) NIE=\beta^{(xs)}H_MH_Y^{-1}\beta^{(sy)}=\beta^{(xm)}\beta^{(my)} N I E = β ( x s ) H M H Y − 1 β ( sy ) = β ( x m ) β ( m y ) where H M H Y − 1 H_MH_Y^{-1} H M H Y − 1 are derived from Proposition 1). A detailed assessment of LaMe’s capacity to retrieve causal natural effects is developed in the next subsection. We compare the models’ estimates with the corresponding parameters used for data generation, ( β t r u e ( x m ) (\beta_{\mathrm{true}}^{(xm)} ( β true ( x m ) and β t r u e ( m y ) ) \beta_{\mathrm{true}}^{(my)}) β true ( m y ) ) , using the mean absolute error across all coefficients as the evaluation metric.
We present results under a challenging configuration with I = 100 I=100 I = 100 , J = 100 J=100 J = 100 , L = 1 L=1 L = 1 , signal-to-noise ratios S N R ( M ∣ S ) = SNR(M \mid S)= S N R ( M ∣ S ) =
S N R ( Y ∣ X , S ) = 0.1 SNR(Y|X,S) = 0.1 S N R ( Y ∣ X , S ) = 0.1 and S N R ( S ∣ X ) = 1 SNR(S|X) = 1 S N R ( S ∣ X ) = 1 . Data generated from the latent mediation system in Eq. (2) uses K t r u e = 10 K_{\mathrm{true}} = 10 K true = 10 latent dimensions. For all latent models (OVP-LaMe aside), K K K is set to K t r u e K_{\mathrm{true}} K true when data are generated under a latent model. When data are generated under a non-latent model, we set K = min ( J , I + L ) 2 K = \frac{\min(J,I+L)}{2} K = 2 m i n ( J , I + L ) to test latent models’ ability to compress a full-rank covariance matrix between X X X and M M M , Y Y Y through a matrix of half this rank. Regularized models (LASSO, ElasticNet, Pathway LASSO, PCA+LASSO) are not considered because the data in this experiment were generated without sparse coefficients.
Figure 2 reports the mean absolute error between the inferred ( β ( x m ) , β ( m y ) ) (\beta^{(xm)}, \beta^{(my)}) ( β ( x m ) , β ( m y ) ) and the true ( β t r u e ( x m ) , β t r u e ( m y ) ) (\beta_{\mathrm{true}}^{(xm)}, \beta_{\mathrm{true}}^{(my)}) ( β true ( x m ) , β true ( m y ) ) , for varying numbers of observations N N N , and for data generated under latent and non-latent models. First, LaMe consistently outperforms all baselines in the latent case (Figure 2 -left). In particular, because K t r u e ≪ min ( J , I + L ) K_{\mathrm{true}} \ll\min(J, I + L) K true ≪ min ( J , I + L ) , LaMe reduces estimation error through its low-rank restriction where OLS cannot. Second, and as expected, OVP-LaMe performs similarly to OLS, confirming that the violation of Eq. (5) allows OVP-LaMe to converge to a solution equivalent to the OLS solution. In particular, OVP-LaMe is prone to overfitting, just as OLS, when data are generated under a latent model. In the non-latent case, LaMe outperforms latent baselines and performs nearly as well as the non-latent models, despite the constraint K = min ( J , I + L ) 2 K = \frac{\min(J,I+L)}{2} K = 2 m i n ( J , I + L ) .
Figure 2. LaMe accurately retrieves underlying parameters — LaMe and baselines when I = 100 I = 100 I = 100 , J = 100 J = 100 J = 100 , L = 1 L = 1 L = 1 , and a signal-to-noise ratio from S S S to M M M and Y Y Y equal to 0.1 for varying N N N . (Top) Mean absolute error of the estimated coefficients β ( x m ) \beta^{(xm)} β ( x m ) relative to the true coefficients β t r u e ( x m ) \beta_{\mathrm{true}}^{(xm)} β true ( x m ) | (Bottom) Mean absolute error of the estimated coefficients β ( m y ) \beta^{(my)} β ( m y ) relative to the true coefficients β t r u e ( m y ) \beta_{\mathrm{true}}^{(my)} β true ( m y ) | (left) Data generated using a latent process (K = 10 K = 10 K = 10 , see Eq. (2) ) with the true K K K provided to all latent models | (right) Data generated using a non-latent process (see Eq. (1) ) with K = min ( J , I + L ) 2 K = \frac{\min(J,I+L)}{2} K = 2 m i n ( J , I + L ) for all latent models except for OVP-LaMe where K = I + J + L 2 = 100 K = \frac{I+J+L}{2} = 100 K = 2 I + J + L = 100 . Alt text: on latent data, LaMe’s estimates decrease with N N N toward smaller mean absolute errors on the true parameters whereas other models do not for N ∈ [ 1000 ; 10000 ] N \in[1000; 10000] N ∈ [ 1000 ; 10000 ] ; on non-latent data, estimates from OLS, OVP-LaMe, LaMe and JAOG decrease with N N N toward a smaller mean absolute error on β t r u e ( m y ) \beta_{\mathrm{true}}^{(my)} β true ( m y ) whereas other models do not for N ∈ [ 1000 ; 10000 ] N \in[1000; 10000] N ∈ [ 1000 ; 10000 ] .
Additional results for { I = 50 , J = 200 } \{I = 50, J = 200\} { I = 50 , J = 200 } , as well as for varying signal-to-noise ratios are presented and discussed in Web Appendix 21.
Causal effect recovery
We recall from Eq. (3) that N I E = β ( x s ) β ( s y ) NIE = \beta^{(xs)}\beta^{(sy)} N I E = β ( x s ) β ( sy ) , N D E = β ( x y ) NDE = \beta^{(xy)} N D E = β ( x y ) and T E = β ( x s ) β ( s y ) + β ( x y ) TE = \beta^{(xs)}\beta^{(sy)} + \beta^{(xy)} T E = β ( x s ) β ( sy ) + β ( x y ) ([1 , 7 ] ). We observe that LaMe correctly retrieves the ground-truth natural effects coefficients overall in Figure 3 , when optimized with N = 1 , 000 N = 1,000 N = 1 , 000 observations on data generated under the exact recoverability assumption (see Proposition 1 ) with I = 10 I = 10 I = 10 , J = 50 J = 50 J = 50 , K = 5 K = 5 K = 5 and L = 1 L = 1 L = 1 , unless specified otherwise. The normalization using the true total effect provides a shared comparison ground while avoiding artificial errors when either the NDE or NIE coefficient is small. Figure 3 -top in particular shows that natural effects estimation error is stable when K K K is larger than the true number of latent dimensions K t r u e K_{\mathrm{true}} K true used for data generation. When K < K t r u e K < K_{\mathrm{true}} K < K true however, the NDE and NIE estimates are not accurate, as the under-parameterized model must express effects through a smaller number of latent variables. Taken together, these findings hint at a possible model selection strategy based on parameter stability rather than information criteria. Estimation error also decreases over the sample sizes considered. Finally, note that the direct path ensures that TE is equal to the OLS regression coefficient X X X and Y Y Y , namely T E = ( X T X ) − 1 X T Y TE = (X^{T}X)^{-1}X^{T}Y T E = ( X T X ) − 1 X T Y , explaining why it does not depend on K K K . Figure 3 -bottom shows that LaMe estimates the NDE, NIE and TE coefficients more accurately when the norm of the NDE coefficient matrix exceeds that of the NIE coefficient matrix: in this case, most of the observed covariance between X X X and Y Y Y is expressed through the NDE, whose estimation does not require a fine approximation of the unobserved variable S S S . Conversely, this explains why the relative error on the NDE, NIE and TE coefficients increases when the norm of the NIE coefficient is larger than that of the NDE. Overall, LaMe accurately retrieves the true natural effects across a range of direct and indirect effect magnitudes, supporting the applicability of the LaMe model to various settings.
Figure 3. LaMe accurately recovers the natural effects — Root mean squared error (RMSE) of estimated natural effect coefficients, normalized by the norm of the true total-effect coefficient, averaged over 100 runs, along with the associated standard deviation, for LaMe optimized with N = 1 000 N = 1\,000 N = 1 000 observations on data generated with I = 10 I = 10 I = 10 , J = 50 J = 50 J = 50 , K = 5 K = 5 K = 5 and L = 1 L = 1 L = 1 , unless specified otherwise. (Top) Varying the number of observations and the number of latent variables used for estimation when the true number of latent variables used for data generation is fixed at K t r u e = 5 K_{\mathrm{true}} = 5 K true = 5 | (Bottom) Varying the values of NIE and NDE coefficients norm used for data generation | Root mean squared error relative to the true total effect between the true and estimated (Left) natural direct effect, (Middle) natural indirect effect and (Right) total effect. Alt text: Heatmap representing the RMSE of LaMe with respect to the true NIE, NDE and TE path coefficients relative to the true TE. The RMSE of NIE, NDE and TE decreases with N N N , with K − K t r u e K - K_{\mathrm{true}} K − K true (plateauing once K ≥ K t r u e K \ge K_{\mathrm{true}} K ≥ K true ) and with ∥ N D E ^ − N D E t r u e ∥ 2 ∥ N D E t r u e ∥ 2 \frac{\lVert\widehat{\mathrm{NDE}}-\mathrm{NDE}_{\mathrm{true}}\rVert^2}{\lVert\mathrm{NDE}_{\mathrm{true}}\rVert^2} ∥ NDE true ∥ 2 ∥ NDE − NDE true ∥ 2 , with a relative RMSE remaining below 0.1 in most cases.
Retrieving the manifest variables involved in mediation
We show here that LaMe efficiently identifies the entries of M M M that are both influenced by X X X through S S S and informative about Y Y Y through S S S given X X X . These are not observed mediators, since there is no direct path between M M M and X X X or Y Y Y ; M M M is a descendant of the latent mediator rather than a causal intermediate on the indirect path. However, they are the closest analogue of observed mediating variables and are used as such in [7 ] ; for simplicity, we refer to them as mediating manifest features. Building on the definition proposed in [7 ] , we define a global mediating manifest feature M j M_j M j as the j t h j^{\mathrm{th}} j th entry of M M M under the exact recoverability assumption such that both:
∑ i = 1 I ∣ C i j X M ∣ : = ∑ i = 1 I ∣ ∑ k = 1 K β i k ( X S ) β k j ( S M ) ∣ ≠ 0 and ∑ l = 1 L ∣ C j l M Y ∣ : = ∑ l = 1 L ∣ ∑ k = 1 K β j k ( M S ) β k l ( S Y ) ∣ ≠ 0 \sum_{i=1}^{I} \lvert C_{ij}^{XM} \rvert:= \sum_{i=1}^{I} \left\lvert\sum_{k=1}^{K} \beta_{ik}^{(XS)} \beta_{kj}^{(SM)} \right\rvert\ne0
\quad\text{and}\quad
\sum_{l=1}^{L} \lvert C_{jl}^{MY} \rvert:= \sum_{l=1}^{L} \left\lvert\sum_{k=1}^{K} \beta_{jk}^{(MS)} \beta_{kl}^{(SY)} \right\rvert\ne0 i = 1 ∑ I ∣ C ij X M ∣ := i = 1 ∑ I k = 1 ∑ K β ik ( X S ) β k j ( S M ) = 0 and l = 1 ∑ L ∣ C j l M Y ∣ := l = 1 ∑ L k = 1 ∑ K β j k ( M S ) β k l ( S Y ) = 0
A mediating manifest feature is therefore an entry of M M M , noted M j M_j M j , such that C j : = ∑ i = 1 I ∑ l = 1 L ∣ C i j X M ∣ ∣ C j l M Y ∣ ≠ 0 C_j := \sum_{i=1}^{I} \sum_{l=1}^{L} \lvert C_{ij}^{XM} \rvert\lvert C_{jl}^{MY} \rvert\ne0 C j := ∑ i = 1 I ∑ l = 1 L ∣ C ij X M ∣ ∣ C j l M Y ∣ = 0 . This new definition is intended to generalize that proposed by [7 ] , in particular to make it invariant to linear transformations of the latent space and to be robust to cases where manifest variables are correlated (see Web Appendix 20).
We generate data from Eq. (2) such that there are exactly J = 10 J = 10 J = 10 observed mediating manifest variables M j M_j M j with C j ≠ 0 C_j \ne0 C j = 0 and Δ J ∈ [ 1 , 990 ] \Delta J \in[1, 990] Δ J ∈ [ 1 , 990 ] non-mediating manifest variables with C j = 0 C_j = 0 C j = 0 . In practice, we generate full-rank parameters (which ensures that C j ≠ 0 C_j \ne0 C j = 0 ) for the system with J J J mediating manifest variables, expand β ( s m ) \beta^{(sm)} β ( s m ) and Σ m ∣ s \Sigma_{m \mid s} Σ m ∣ s with Δ J \Delta J Δ J columns (and rows for Σ m ∣ s \Sigma_{m \mid s} Σ m ∣ s ) such that C j = 0 C_j = 0 C j = 0 , and generate 5 000 observations according to the resulting set of parameters to train LaMe. We then evaluate whether the model accurately distinguishes the Δ J \Delta J Δ J manifest variables with a true C j = 0 C_j = 0 C j = 0 from the J J J manifest variables with C j ≠ 0 C_j \ne0 C j = 0 using the area under the ROC curve (AUC-ROC), the Average Precision (AP) and the maximum F1 across classification thresholds (Max-F1). These metrics estimate how well the inferred C j C_j C j (continuous value) distinguishes the nonzero entries in C ( t r u e ) C^{(true)} C ( t r u e ) (binary value based on its null/non-null entries) without the need to specify a threshold between scores associated with nonzero and near-zero values of C j C_j C j . For AUC-ROC, AP and Max-F1, a value of 1 means the model achieves perfect separation between mediating and non-mediating variables in M M M .
The results presented in Figure 4 show that LaMe consistently distinguishes the ten true mediating manifest features even in situations where these represent only 1% of the total number of manifest variables.
Figure 4. LaMe accurately recovers the observed “mediating” set of manifest variables in M M M among a large set of “non-mediating” candidates – Data are generated from Eq. (2) with I = 10 I = 10 I = 10 , J = 10 + Δ J J = 10 + \Delta J J = 10 + Δ J , K = 2 K = 2 K = 2 , L = 1 L = 1 L = 1 , N = 5 000 N = 5\,000 N = 5 000 without confounding, where Δ J ∈ [ 1 , 990 ] \Delta J \in[1, 990] Δ J ∈ [ 1 , 990 ] is the number of features in M M M that do not contribute to mediation mechanisms (that are either not influenced by X X X , or not informative about Y Y Y given X X X ). The shaded area represents the standard deviation over 50 datasets generated under the same parameter setting. The AUC-ROC, AP and Max-F1 all range from 0 (worst) to 1 (best), and quantify how well the model recovers the true set of mediating manifest features among all possible candidates. Alt text: The three metrics remain close to 1 over the whole range of candidate mediators, with standard deviations around 0.05 across all experiments.
Real-world data
Data. The NHANES dataset ([13 ] ) examines whether participants’ (N = 2,710 N = 2{,}710 N = 2 , 710 ) glycohemoglobin levels (Y Y Y ) are a direct consequence of lifestyle habits (I = 14 I = 14 I = 14 ; hours of sleep, alcohol intake, sunscreen use, etc.) or are mediated by biomarkers (J = 25 J = 25 J = 25 ; blood levels of lead, insulin, cholesterol, etc.).
The EPIC-CAN dataset (International Agency for Research on Cancer, 2024) examines whether the age of a cancer event (N = 4,993 N = 4{,}993 N = 4 , 993 ) is directly influenced by socio-demographic and lifestyle features (I = 33 I = 33 I = 33 ; weight, height, gender, age, smoking habits, physical activity index, diet, etc.) or whether this influence is mediated by blood levels of a high-dimensional collection of candidate proteomic features measured at inclusion in the study (J = 379 J = 379 J = 379 ).
The CRIME dataset ([16 ] ) examines whether cities’ (N = 1,993 N = 1{,}993 N = 1 , 993 ) crime rates (output Y Y Y ) are a direct consequence of population characteristics (I = 67 I = 67 I = 67 representing the number of inhabitants, unemployment rate, percentage of divorce, ethnicity, etc.) or are mediated by city-level contextual factors (J = 33 J = 33 J = 33 representing median rent, public transportation, number of vacant houses, etc.).
The PISA-I dataset comes from the PISA corpus ([15 ] ). It examines whether students’ (N = 52,028 N = 52{,}028 N = 52 , 028 ) mathematics proficiency score is a direct consequence of their home environment (I = 27 I = 27 I = 27 ; indexes quantifying home possessions, availability of electronic devices, mental health, body image, etc.) or is mediated by the school environment (J = 20 J = 20 J = 20 ; indexes quantifying the sense of belonging, safety perception, school quality, etc.).
The complete list of variables is provided in Web Appendix 24. The applications evaluate whether LaMe can summarize exposure–biomarker–outcome associations using compact latent representations while retaining predictive performance; these applications are not intended to provide substantive causal mediation analyses. In particular, for EPIC-CAN, our analysis focused on individuals who developed cancer during follow-up, precluding a straightforward causal interpretation of the fitted relationships. We therefore do not report natural direct and indirect effects for these applications, and instead assess predictive performance, goodness of fit, and the ability to summarize observed relationships through parsimonious latent representations.
Method. We empirically observe that most models reach a plateau in predictive performance (the addition of new data no longer improves predictive accuracy) when trained on 80% of the data, and converge to similar fitted solutions. We instead report the results for a small-training-sample stress test by using an outer 5-fold cross validation procedure: we train the models on 20% of the available data, and evaluate them on the 80% remaining observations; this is iterated five times with different training data. The results from the usual 5-fold cross validation are provided in Web Appendix 22. Note that this procedure serves for unbiased evaluation only, and is not involved in any hyper-parameter tuning: test data are systematically isolated from the training procedure.
We optimize LaMe for K ∈ [ 1 , 2 , 3 , 4 , 5 , 7 , 10 , 15 , 20 , 25 ] K \in[1, 2, 3, 4, 5, 7, 10, 15, 20, 25] K ∈ [ 1 , 2 , 3 , 4 , 5 , 7 , 10 , 15 , 20 , 25 ] and select the version with the lowest Bayesian Information Criterion within each fold; for PCA, K K K has been chosen using scikit-learn’s maximum likelihood estimator for the number of components; for PLS and JAOG, K K K has been chosen to maximize their respective scoring function on a validation set (10% randomly sampled from the training set).
Results. We report our results for LaMe and latent baselines in Table 1 . LaMe achieves predictive performance comparable to that of the strongest latent baselines, sometimes improving on them, while using parsimonious representations. Visualizations of the inferred systems are provided in Web Appendix 23. In particular, these experiments show that LaMe achieves competitive predictive performance with compact latent representations. These results support a favorable trade-off between predictive performance and latent dimensionality under the stated model-selection procedures. On the other hand, over-parameterized models such as OVP-LaMe converge to solutions that are equivalent to the OLS solution. A similar case arises for PCA-based methods here, for which the maximum likelihood estimator selected models with large K K K —thus forfeiting any low-dimensional covariance structure. For regularized LaMe on CRIME, and for NHANES and EPIC-CAN, LaMe uses a single latent dimension where other latent approaches used many more to achieve comparable performance in terms of prediction and goodness-of-fit.
Table 1. Out-of-sample and in-sample performance metrics of LaMe (version X and M, with and without ℓ1 regularization, assuming full noise covariances) and all baseline models. The last lines report the average improvement of models’ metrics over OLS across all datasets –hence the null line for OLS. NLPD is the Negative Log-Predictive Density, R 2 R^{2} R 2 the fraction of explained variance and AIC the Akaike Information Criterion. When applicable, the number of latent variables is reported as K K K . Errors correspond to the standard error across five folds. The colour scale indicates relative performance within each dataset and metric ; (orange=best).
Conclusion
We have introduced LaMe as a general framework for joint estimation of multivariate latent mediation systems that makes the conditions for dimension reduction and identification of the effect decomposition explicit. The simulations demonstrate gains from low-rank estimation when the data contain shared latent structure and recovery of natural effects under the stipulated measurement model. They also show that accurate estimation of the total effect can coexist with persistent errors in its direct and indirect components when too few latent dimensions are fitted. The real-world applications complement these findings by demonstrating competitive prediction with compact latent representations. The present evidence supports LaMe as a scalable, parsimonious and accurate option for high-dimensional mediation analysis across a wide variety of situations.
Possible directions for future work include incorporating mixed data types such as categorical exposures, jointly inferring the number of latent components K K K (e.g. through point-process priors), relaxing linearity assumptions and extending the framework to survival models with censored data.
Acknowledgments. This work was supported by a grant from the French National Research Agency as part of the “Investissements d’Avenir ExcellencES” program from France 2030 (SHAPE-Med@Lyon; [ANR-22-EXES-0012]). Funding for grant [IIG_FULL_2022_013] was obtained from Wereld Kanker Onderzoek Fonds (WKOF) as part of the World Cancer Research Fund International grant programme. Where authors are identified as personnel of the International Agency for Research on Cancer/World Health Organization, the authors alone are responsible for the views expressed in this article and they do not necessarily represent the decisions, policy, or views of the International Agency for Research on Cancer/World Health Organization.
Conflicts of interest. The authors declare no conflict of interest.
Data availability statement. The NHANES August 2021–August 2023 data are publicly available at https://wwwn.cdc.gov/nchs/nhanes/continuousnhanes/default.aspx?Cycle=2021-2023. The CRIME dataset (DOI: 10.24432/C53W3X) is publicly available at https://doi.org/10.24432/C53W3X. The PISA-I data were derived from the PISA 2022 database, publicly available at https://www.oecd.org/en/data/datasets/pisa-2022-database.html. The data underlying EPIC-CAN is based on data from the European Prospective Investigation into Cancer and Nutrition (EPIC) and cannot be shared publicly due to privacy considerations. EPIC data are available by application to the EPIC Steering Committee (https://epic.iarc.fr/access/).
References
Supplementary Materials. Web Appendices are available with this paper at the Biometrics website on Oxford Academic. The Python code for performing the simulations and data analysis in this paper is available at https://github.com/GaelPouxMedard/LaMe-repr/releases/tag/v1.1.0-paper. Code is permanently archived in Zenodo at https://doi.org/10.5281/zenodo.23264883.
Observed confounding between exposures, mediators and outcomes
Based on [7 ] , observed confounding C C C enters latent mediation systems as follows:
C = ε c X = C β ( c x ) + ε x S = X β ( x s ) + C β ( c s ) + ε s M = S β ( s m ) + ε m Y = X β ( x y ) + S β ( s y ) + C β ( c y ) + ε y (14) \begin{aligned}
C &= \varepsilon_c \\
X &= C\beta^{(cx)} + \varepsilon_x \\
S &= X\beta^{(xs)} + C\beta^{(cs)} + \varepsilon_s \\
M &= S\beta^{(sm)} + \varepsilon_m \\
Y &= X\beta^{(xy)} + S\beta^{(sy)} + C\beta^{(cy)} + \varepsilon_y \tag*{(14)}
\end{aligned} C X S M Y = ε c = C β ( c x ) + ε x = X β ( x s ) + C β ( cs ) + ε s = S β ( s m ) + ε m = X β ( x y ) + S β ( sy ) + C β ( cy ) + ε y (14)
Observed confounding C C C enters the non-latent mediation system as follows:
C = ε c X = C β ( c x ) + ε x M = X β ( x m ) + C β ( c m ) + ε m Y = X β ( x y ) + M β ( m y ) + C β ( c y ) + ε y (15) \begin{aligned}
C &= \varepsilon_c \\
X &= C\beta^{(cx)} + \varepsilon_x \\
M &= X\beta^{(xm)} + C\beta^{(cm)} + \varepsilon_m \\
Y &= X\beta^{(xy)} + M\beta^{(my)} + C\beta^{(cy)} + \varepsilon_y \tag*{(15)}
\end{aligned} C X M Y = ε c = C β ( c x ) + ε x = X β ( x m ) + C β ( c m ) + ε m = X β ( x y ) + M β ( m y ) + C β ( cy ) + ε y (15)
Here, we demonstrate that observed exposures X X X and observed confounders C C C can be jointly represented as a single composite observed variable ξ : = [ X , C ] \xi:= [X,C] ξ := [ X , C ] from a likelihood perspective. As such, the entire technical demonstration in the main text can be conducted by considering ξ = [ X ] \xi= [X] ξ = [ X ] without loss of generality. Modelling confounding simply requires replacing exposures by their augmented counterpart ξ \xi ξ of dimension I + dim ( C ) I + \dim(C) I + dim ( C ) , and performing the separation step below after parameters estimation.
For latent systems (Eq. (14) ), we start from any joint distribution p ( X , S , M , Y , C ) p(X,S,M,Y,C) p ( X , S , M , Y , C ) along with the conditional independence relationships corresponding to confounded LaMe (see Eq. (14) ), namely M ⊥ ( X , C ) ∣ S M \perp(X,C) \mid S M ⊥ ( X , C ) ∣ S and Y ⊥ M ∣ ( X , C , S ) Y \perp M \mid(X,C,S) Y ⊥ M ∣ ( X , C , S ) . Developing p ( X , S , M , Y , C ) p(X,S,M,Y,C) p ( X , S , M , Y , C ) using the chain rule, we show that X X X and C C C can be treated jointly as a single observed variable from a statistical standpoint:
p ( X , S , M , Y , C ) = p ( X , C ) p ( S ∣ X , C ) p ( M ∣ S ) p ( Y ∣ X , C , S ) : = p ( ξ ) p ( S ∣ ξ ) p ( M ∣ S ) p ( Y ∣ ξ , S ) , (16) \begin{aligned}
p(X,S,M,Y,C) \\
= p(X,C)p(S\mid X,C)p(M\mid S)p(Y\mid X,C,S) \\
:= p(\xi)p(S\mid\xi)p(M\mid S)p(Y\mid\xi,S), \tag*{(16)}
\end{aligned} p ( X , S , M , Y , C ) = p ( X , C ) p ( S ∣ X , C ) p ( M ∣ S ) p ( Y ∣ X , C , S ) := p ( ξ ) p ( S ∣ ξ ) p ( M ∣ S ) p ( Y ∣ ξ , S ) , (16)
where we defined ξ : = [ X , C ] \xi:= [X,C] ξ := [ X , C ] as a composite observed variable. The last factorization is identical in form to that of unconfounded LaMe-vX in the main text, with ξ \xi ξ replacing X X X ; maximum likelihood estimates for all these parameters are therefore obtained from the same inference procedure, treating ξ \xi ξ as an augmented observed exposure variable of dimension I + dim ( C ) I + \dim(C) I + dim ( C ) .
For non-latent systems (Eq. (15) ), the same demonstration immediately follows from the chain rule noting that p ( ξ , M , Y ) = p ( ξ ) p ( M ∣ ξ ) p ( Y ∣ ξ , M ) p(\xi,M,Y) = p(\xi)p(M\mid\xi)p(Y\mid\xi,M) p ( ξ , M , Y ) = p ( ξ ) p ( M ∣ ξ ) p ( Y ∣ ξ , M ) .
The path coefficients from X X X and C C C to the mediator and the outcome are then obtained by appropriate slicing of the coefficients associated with ξ \xi ξ e.g. in the latent case β ( s ξ ) = [ β ( x s ) , β ( c s ) ] T \beta^{(s\xi)} = [\beta^{(xs)},\beta^{(cs)}]^{T} β ( s ξ ) = [ β ( x s ) , β ( cs ) ] T and β ( ξ y ) = [ β ( x y ) , β ( c y ) ] T \beta^{(\xi y)} = [\beta^{(xy)},\beta^{(cy)}]^{T} β ( ξ y ) = [ β ( x y ) , β ( cy ) ] T . In both latent and non-latent cases, the path coefficient from C C C to X X X is obtained from the off-diagonal blocks in the inferred covariance of ξ \xi ξ , namely Σ ξ \Sigma_\xi Σ ξ . After appropriate slicing of Σ ξ \Sigma_\xi Σ ξ in terms of exposures X X X and confounders C C C :
Σ ξ = ( Σ x Σ x c Σ c x Σ c ) , (17) \Sigma_\xi=
\begin{pmatrix}
\Sigma_x & \Sigma_{xc} \\
\Sigma_{cx} & \Sigma_c
\end{pmatrix}, \tag*{(17)} Σ ξ = ( Σ x Σ c x Σ x c Σ c ) , (17)
then the path coefficient from C C C to X X X is β ( c x ) = Σ c − 1 Σ c x \beta^{(cx)} = \Sigma_c^{-1}\Sigma_{cx} β ( c x ) = Σ c − 1 Σ c x .
Exact recoverability error
We recall that:
β ( m s ) = ( ( β ( s m ) ) T Σ s β ( s m ) + Σ m ∣ s ) − 1 ( β ( s m ) ) T ( Σ s ∣ x + ( β ( x s ) ) T Σ x β ( x s ) ) \beta^{(ms)} = \left((\beta^{(sm)})^{T}\Sigma_s\beta^{(sm)}+\Sigma_{m\mid s}\right)^{-1}(\beta^{(sm)})^{T}\left(\Sigma_{s\mid x}+(\beta^{(xs)})^{T}\Sigma_x\beta^{(xs)}\right) β ( m s ) = ( ( β ( s m ) ) T Σ s β ( s m ) + Σ m ∣ s ) − 1 ( β ( s m ) ) T ( Σ s ∣ x + ( β ( x s ) ) T Σ x β ( x s ) )
β ( m s ) = Σ m − 1 ( β ( s m ) ) T Σ s . \beta^{(ms)}=\Sigma_m^{-1}(\beta^{(sm)})^{T}\Sigma_s. β ( m s ) = Σ m − 1 ( β ( s m ) ) T Σ s .
The expected difference between S S S and S ∗ S^* S ∗ is zero following the tower property of nested expectations:
E [ S ∗ − S ] = E [ E [ S ∗ − S ∣ M ] ] = 0 \begin{aligned}
\mathbb{E}[S^*-S]=\mathbb{E}[\mathbb{E}[S^*-S\mid M]] \\
&=0
\end{aligned} E [ S ∗ − S ] = E [ E [ S ∗ − S ∣ M ]] = 0
The variance of the difference is:
Var ( S ∗ − S ) = Var ( S ) + ( 1 − 2 ) Var ( E [ S ∣ M ] ) = Σ s − ( β ( m s ) ) T Σ m β ( m s ) = Σ s ( I − β ( s m ) β ( m s ) ) \begin{aligned}
\operatorname{Var}(S^*-S)=\operatorname{Var}(S)+(1-2)\operatorname{Var}(\mathbb{E}[S\mid M]) \\
&=\Sigma_s-(\beta^{(ms)})^{T}\Sigma_m\beta^{(ms)} \\
&=\Sigma_s(I-\beta^{(sm)}\beta^{(ms)})
\end{aligned} Var ( S ∗ − S ) = Var ( S ) + ( 1 − 2 ) Var ( E [ S ∣ M ]) = Σ s − ( β ( m s ) ) T Σ m β ( m s ) = Σ s ( I − β ( s m ) β ( m s ) )
From the definition of β ( m s ) \beta^{(ms)} β ( m s ) :
β ( m s ) = ( ( β ( s m ) ) T Σ s β ( s m ) + Σ m ∣ s ) − 1 ( β ( s m ) ) T Σ s = Σ m ∣ s − 1 ( ( β ( s m ) ) T Σ s − ( β ( s m ) ) T Σ s β ( s m ) β ( m s ) ) = Σ m ∣ s − 1 ( β ( s m ) ) T Σ s ( I − β ( s m ) β ( m s ) ) = Σ m ∣ s − 1 ( β ( s m ) ) T Var ( S ∗ − S ) \begin{aligned}
\beta^{(ms)}=\left((\beta^{(sm)})^{T}\Sigma_s\beta^{(sm)}+\Sigma_{m\mid s}\right)^{-1}(\beta^{(sm)})^{T}\Sigma_s \\
&=\Sigma_{m\mid s}^{-1}\left((\beta^{(sm)})^{T}\Sigma_s-(\beta^{(sm)})^{T}\Sigma_s\beta^{(sm)}\beta^{(ms)}\right) \\
&=\Sigma_{m\mid s}^{-1}(\beta^{(sm)})^{T}\Sigma_s(I-\beta^{(sm)}\beta^{(ms)}) \\
&=\Sigma_{m\mid s}^{-1}(\beta^{(sm)})^{T}\operatorname{Var}(S^*-S)
\end{aligned} β ( m s ) = ( ( β ( s m ) ) T Σ s β ( s m ) + Σ m ∣ s ) − 1 ( β ( s m ) ) T Σ s = Σ m ∣ s − 1 ( ( β ( s m ) ) T Σ s − ( β ( s m ) ) T Σ s β ( s m ) β ( m s ) ) = Σ m ∣ s − 1 ( β ( s m ) ) T Σ s ( I − β ( s m ) β ( m s ) ) = Σ m ∣ s − 1 ( β ( s m ) ) T Var ( S ∗ − S )
Therefore:
Var ( S ∗ − S ) = Σ s ( I − β ( s m ) β ( m s ) ) = Σ s ( I − β ( s m ) Σ m ∣ s − 1 ( β ( s m ) ) T Var ( S ∗ − S ) ) = ( Σ s − 1 + β ( s m ) Σ m ∣ s − 1 ( β ( s m ) ) T ) − 1 \begin{aligned}
\operatorname{Var}(S^*-S)=\Sigma_s(I-\beta^{(sm)}\beta^{(ms)}) \\
&=\Sigma_s\left(I-\beta^{(sm)}\Sigma_{m\mid s}^{-1}(\beta^{(sm)})^{T}\operatorname{Var}(S^*-S)\right) \\
&=\left(\Sigma_s^{-1}+\beta^{(sm)}\Sigma_{m\mid s}^{-1}(\beta^{(sm)})^{T}\right)^{-1}
\end{aligned} Var ( S ∗ − S ) = Σ s ( I − β ( s m ) β ( m s ) ) = Σ s ( I − β ( s m ) Σ m ∣ s − 1 ( β ( s m ) ) T Var ( S ∗ − S ) ) = ( Σ s − 1 + β ( s m ) Σ m ∣ s − 1 ( β ( s m ) ) T ) − 1
The result from the main text follows:
E [ ∥ S ∗ − S ∥ 2 ] = Tr ( Var ( S ∗ − S ) ) + ∥ E [ S ∗ − S ] ∥ 2 = Tr ( ( Σ s − 1 + β ( s m ) Σ m ∣ s − 1 ( β ( s m ) ) T ) − 1 ) + 0 = O ( J − 1 ) , (18) \begin{aligned}
\mathbb{E}[\lVert S^*-S\rVert^2]=\operatorname{Tr}(\operatorname{Var}(S^*-S))+\lVert\mathbb{E}[S^*-S]\rVert^2 \\
&=\operatorname{Tr}\left(\left(\Sigma_s^{-1}+\beta^{(sm)}\Sigma_{m\mid s}^{-1}(\beta^{(sm)})^{T}\right)^{-1}\right)+0 \\
&=O(J^{-1}), \tag*{(18)}
\end{aligned} E [∥ S ∗ − S ∥ 2 ] = Tr ( Var ( S ∗ − S )) + ∥ E [ S ∗ − S ] ∥ 2 = Tr ( ( Σ s − 1 + β ( s m ) Σ m ∣ s − 1 ( β ( s m ) ) T ) − 1 ) + 0 = O ( J − 1 ) , (18)
Bottleneck condition
In the most general setting, we assume that I ≥ 1 I\ge1 I ≥ 1 , J ≥ 1 J\ge1 J ≥ 1 , K ≥ 1 K\ge1 K ≥ 1 , L ≥ 1 L\ge1 L ≥ 1 , that all noise covariance matrices are full, and that variables are centred on their average value so that intercepts can be omitted. We show that under these conditions, the optimal solution of a model that violates the bottleneck condition K < min ( I + L , J ) K<\min(I+L,J) K < min ( I + L , J ) corresponds to a reparametrization of the Ordinary Least Squares (OLS) optimum.
It has already been established that in the fully unconstrained case, OLS is the maximum likelihood estimator for a linear model of the observed variables, and that it exactly matches the first and second population moments [8 ] . Although the latent system defined in the main text assumes that linear relations between observed variables are structured by a latent variable S S S , the relations between observed variables remain linear. Therefore, whenever the latent model can attain (a given reparametrization of) the OLS solution, this solution also corresponds to a maximum likelihood estimator of the latent model.
We therefore derive the value of K K K for which the latent model will necessarily produce a solution equivalent to the OLS solution, while using its own parametric representation. Equivalence here means that the two models reproduce exactly the same population moments through their respective parametrizations. Here, the latent model is identified from the population moments Σ X X , Σ M M , Σ Y Y , Σ X M , Σ X Y , Σ M Y \Sigma_{XX}, \Sigma_{MM}, \Sigma_{YY}, \Sigma_{XM}, \Sigma_{XY}, \Sigma_{MY} Σ X X , Σ M M , Σ Y Y , Σ X M , Σ X Y , Σ M Y by the following system of equations:
Σ X X = Σ x Σ M M = Σ m ∣ s + ( β ( s m ) ) T Σ s ∣ x β ( s m ) + ( β ( x s ) β ( s m ) ) T Σ x β ( x s ) β ( s m ) Σ Y Y = Σ y ∣ x , s + ( β ( s y ) ) T Σ s ∣ x β ( s y ) + ( β ( x s ) β ( s y ) + β ( x y ) ) T Σ x ( β ( x s ) β ( s y ) + β ( x y ) ) Σ X Y = Σ x ( β ( x s ) β ( s y ) + β ( x y ) ) Σ X M = Σ x β ( x s ) β ( s m ) Σ M Y = ( β ( s m ) ) T Σ s ∣ x β ( s y ) + ( β ( x s ) β ( s m ) ) T Σ x ( β ( x s ) β ( s y ) + β ( x y ) ) (19) \begin{aligned}
\Sigma_{XX} &= \Sigma_x \\
\Sigma_{MM} &= \Sigma_{m\mid s} + (\beta^{(sm)})^T\Sigma_{s\mid x}\beta^{(sm)} + (\beta^{(xs)}\beta^{(sm)})^T\Sigma_x\beta^{(xs)}\beta^{(sm)} \\
\Sigma_{YY} &= \Sigma_{y\mid x,s} + (\beta^{(sy)})^T\Sigma_{s\mid x}\beta^{(sy)} + (\beta^{(xs)}\beta^{(sy)}+\beta^{(xy)})^T\Sigma_x(\beta^{(xs)}\beta^{(sy)}+\beta^{(xy)}) \\
\Sigma_{XY} &= \Sigma_x(\beta^{(xs)}\beta^{(sy)}+\beta^{(xy)}) \\
\Sigma_{XM} &= \Sigma_x\beta^{(xs)}\beta^{(sm)} \\
\Sigma_{MY} &= (\beta^{(sm)})^T\Sigma_{s\mid x}\beta^{(sy)} + (\beta^{(xs)}\beta^{(sm)})^T\Sigma_x(\beta^{(xs)}\beta^{(sy)}+\beta^{(xy)}) \tag*{(19)}
\end{aligned} Σ X X Σ M M Σ Y Y Σ X Y Σ X M Σ M Y = Σ x = Σ m ∣ s + ( β ( s m ) ) T Σ s ∣ x β ( s m ) + ( β ( x s ) β ( s m ) ) T Σ x β ( x s ) β ( s m ) = Σ y ∣ x , s + ( β ( sy ) ) T Σ s ∣ x β ( sy ) + ( β ( x s ) β ( sy ) + β ( x y ) ) T Σ x ( β ( x s ) β ( sy ) + β ( x y ) ) = Σ x ( β ( x s ) β ( sy ) + β ( x y ) ) = Σ x β ( x s ) β ( s m ) = ( β ( s m ) ) T Σ s ∣ x β ( sy ) + ( β ( x s ) β ( s m ) ) T Σ x ( β ( x s ) β ( sy ) + β ( x y ) ) (19)
The OLS solution is reachable when all these equations can be exactly satisfied simultaneously. The equations involving Σ X X \Sigma_{XX} Σ X X , Σ M M \Sigma_{MM} Σ M M , Σ Y Y \Sigma_{YY} Σ Y Y , and Σ X Y \Sigma_{XY} Σ X Y are straightforward, as these population moments are always reachable from the latent model’s parameters in the fully unconstrained case: Σ X X \Sigma_{XX} Σ X X , Σ M M \Sigma_{MM} Σ M M and Σ Y Y \Sigma_{YY} Σ Y Y can always be exactly modelled by the full matrices Σ x \Sigma_x Σ x , Σ m ∣ s \Sigma_{m\mid s} Σ m ∣ s and Σ y ∣ x , s \Sigma_{y\mid x,s} Σ y ∣ x , s ; Σ X Y \Sigma_{XY} Σ X Y can always be modelled exactly by β ( x y ) \beta^{(xy)} β ( x y ) because its contribution to Σ Y Y \Sigma_{YY} Σ Y Y can always be compensated by the full matrix Σ y ∣ x , s \Sigma_{y\mid x,s} Σ y ∣ x , s , and because it does not appear in Σ M Y \Sigma_{MY} Σ M Y once Σ X Y \Sigma_{XY} Σ X Y is known. The rank constraint on the joint system Eq. (19) induced by the value of K K K can therefore be expressed in terms of Σ X M \Sigma_{XM} Σ X M and Σ M Y \Sigma_{MY} Σ M Y only.
Replacing the expression for Σ X M \Sigma_{XM} Σ X M in Σ M Y \Sigma_{MY} Σ M Y , then Σ M Y = ( β ( s m ) ) T Σ s ∣ x β ( s y ) + ( Σ X M ) T Σ X X − 1 Σ X Y \Sigma_{MY} = (\beta^{(sm)})^T\Sigma_{s\mid x}\beta^{(sy)} + (\Sigma_{XM})^T\Sigma_{XX}^{-1}\Sigma_{XY} Σ M Y = ( β ( s m ) ) T Σ s ∣ x β ( sy ) + ( Σ X M ) T Σ X X − 1 Σ X Y ; the above joint system then reduces to:
C : = ( Σ X X − 1 Σ X M Σ M Y − Σ X Y T Σ X X − 1 Σ X M ) = ( β ( x s ) ( β ( s y ) ) T Σ s ∣ x ) β ( s m ) (20) C := \begin{pmatrix}
\Sigma_{XX}^{-1}\Sigma_{XM} \\
\Sigma_{MY}-\Sigma_{XY}^T\Sigma_{XX}^{-1}\Sigma_{XM}
\end{pmatrix}
=
\begin{pmatrix}
\beta^{(xs)} \\
(\beta^{(sy)})^T\Sigma_{s\mid x}
\end{pmatrix}
\beta^{(sm)} \tag*{(20)} C := ( Σ X X − 1 Σ X M Σ M Y − Σ X Y T Σ X X − 1 Σ X M ) = ( β ( x s ) ( β ( sy ) ) T Σ s ∣ x ) β ( s m ) (20)
The matrix of population moments has I + L I+L I + L rows and J J J columns; its rank is therefore upper-bounded by min ( J , I + L ) \min(J,I+L) min ( J , I + L ) . Its parametric representation instead has rank upper-bounded by min ( J , I + L , K ) \min(J,I+L,K) min ( J , I + L , K ) . Therefore, a sufficient condition for it to be represented exactly by the latent model is K ≥ min ( I + L , J ) K \ge\min(I+L,J) K ≥ min ( I + L , J ) . Conversely, a necessary but not sufficient condition for an approximate solution is K < min ( J , I + L ) K < \min(J,I+L) K < min ( J , I + L ) . It follows that the globally optimal solution of an unconstrained latent linear model with K ≥ min ( J , I + L ) K \ge\min(J,I+L) K ≥ min ( J , I + L ) must necessarily be a reparametrization of the optimal OLS solution.
Note that the exact condition on K K K for the latent model to be able to reparametrize the OLS solution is K ≥ rank ( C ) K \ge\operatorname{rank}(C) K ≥ rank ( C ) , with C C C defined above; however, the sufficient condition based on rank upper bounds is clearer and sufficient for our demonstration, and is the only one we use in the main text.
Dependence of S on the scale of the outcome in sum-of-variance objectives
Let θ \theta θ denote the set of parameters to estimate. Omitting the intercepts for simplicity, the joint objective of the model in [12 ] is:
arg min θ ∥ Y − X β ( x y ) − S β ( s y ) ∥ 2 + ∥ S − X β ( x s ) ∥ 2 \underset{\theta}{\arg\min}\ \lVert Y-X\beta^{(xy)}-S\beta^{(sy)}\rVert^2+\lVert S-X\beta^{(xs)}\rVert^2 θ arg min ∥ Y − X β ( x y ) − S β ( sy ) ∥ 2 + ∥ S − X β ( x s ) ∥ 2
where S = Φ ( M ) S=\Phi(M) S = Φ ( M )
After profiling S S S , this can equivalently be written as an inference problem on S S S directly:
arg min S = Φ ( M ) Var ( Y ∣ X , S ) + Var ( S ∣ X ) \underset{S=\Phi(M)}{\arg\min}\ \operatorname{Var}(Y\mid X,S)+\operatorname{Var}(S\mid X) S = Φ ( M ) arg min Var ( Y ∣ X , S ) + Var ( S ∣ X )
The sensitivity of the optimal representation S S S to the scale of Y Y Y is immediate, since the transformation Y → c Y Y\to cY Y → c Y makes Var ( Y ∣ X , S ) → c 2 Var ( Y ∣ X , S ) \operatorname{Var}(Y\mid X,S)\to c^2\operatorname{Var}(Y\mid X,S) Var ( Y ∣ X , S ) → c 2 Var ( Y ∣ X , S ) :
arg min S = Φ ( M ) Var ( S ∣ X ) + Var ( Y ∣ X , S ) ≠ arg min S = Φ ( M ) Var ( S ∣ X ) + c 2 Var ( Y ∣ X , S ) \underset{S=\Phi(M)}{\arg\min}\ \operatorname{Var}(S\mid X)+\operatorname{Var}(Y\mid X,S)\ne\underset{S=\Phi(M)}{\arg\min}\ \operatorname{Var}(S\mid X)+c^2\operatorname{Var}(Y\mid X,S) S = Φ ( M ) arg min Var ( S ∣ X ) + Var ( Y ∣ X , S ) = S = Φ ( M ) arg min Var ( S ∣ X ) + c 2 Var ( Y ∣ X , S )
More importantly, this implies that NIE ^ ( c Y ) ≠ c NIE ^ ( Y ) \widehat{\operatorname{NIE}}(cY)\ne c\widehat{\operatorname{NIE}}(Y) NIE ( c Y ) = c NIE ( Y ) , where NIE ^ ( Y ) \widehat{\operatorname{NIE}}(Y) NIE ( Y ) denotes the estimated natural indirect effect associated with a one-unit increase in the exposures; the NIE also depends on the scale of Y Y Y .
Constrained LVMA is a factor analysis model
We show that under the assumptions of the latent linear model from [7 ] , namely Σ m ∣ s = diag ( σ m 1 ∣ s , … , σ m J ∣ s ) \Sigma_{m\mid s}=\operatorname{diag}(\sigma_{m_1\mid s},\ldots,\sigma_{m_J\mid s}) Σ m ∣ s = diag ( σ m 1 ∣ s , … , σ m J ∣ s ) , I = 1 I=1 I = 1 and L = 1 L=1 L = 1 , a correctly specified LVMA model (full rank coefficient matrices and positive semi-definite covariances) recovers the same latent subspace as a Factor Analysis (FA) model for M M M whenever the latent space dimension is larger than K = 1 K=1 K = 1 , that is large enough not to constrain the population cross-covariances of the system (see Sec. 8).
Although the authors in [7 ] work under the assumption that Σ s ∣ x = I \Sigma_{s\mid x}=I Σ s ∣ x = I , we chose a representation where the unconditional variance Σ s = I \Sigma_s=I Σ s = I , without loss of generality. These formulations are equivalent due to the usual invariance of linear latent-variable models to rescaling and rotations of the latent space; the system can be equivalently defined in terms of Σ s = I \Sigma_s = I Σ s = I or Σ s ∣ x = I \Sigma_{s|x} = I Σ s ∣ x = I after applying the appropriate transformations of the latent space. Here, we choose the convention where Σ s = I \Sigma_s = I Σ s = I to simplify the derivations below. Then, the model implies the following population moment equations:
Σ X X = Σ x (21) \Sigma_{XX} = \Sigma_x \tag*{(21)} Σ X X = Σ x (21)
Σ M M = Σ m ∣ s + ( β ( s m ) ) T ( Σ s ∣ x + ( β ( x s ) ) T Σ x β ( x s ) ) ⏟ Σ s = I β ( s m ) = Σ m ∣ s + ( β ( s m ) ) T β ( s m ) (22) \begin{aligned}
\Sigma_{MM} &= \Sigma_{m|s} + (\beta^{(sm)})^T \underbrace{\left(\Sigma_{s|x} + (\beta^{(xs)})^T \Sigma_x \beta^{(xs)}\right)}_{\Sigma_s=I} \beta^{(sm)} \\
&= \Sigma_{m|s} + (\beta^{(sm)})^T \beta^{(sm)} \tag*{(22)}
\end{aligned} Σ M M = Σ m ∣ s + ( β ( s m ) ) T Σ s = I ( Σ s ∣ x + ( β ( x s ) ) T Σ x β ( x s ) ) β ( s m ) = Σ m ∣ s + ( β ( s m ) ) T β ( s m ) (22)
Σ Y Y = Σ y ∣ x , s + ( β ( s y ) ) T Σ s ∣ x β ( s y ) + ( ( β ( x s ) ) T β ( s y ) + β ( x y ) ) T Σ x ( ( β ( x s ) ) T β ( s y ) + β ( x y ) ) (23) \Sigma_{YY} = \Sigma_{y|x,s} + (\beta^{(sy)})^T \Sigma_{s|x}\beta^{(sy)} + \left((\beta^{(xs)})^T\beta^{(sy)} + \beta^{(xy)}\right)^T \Sigma_x \left((\beta^{(xs)})^T\beta^{(sy)} + \beta^{(xy)}\right) \tag*{(23)} Σ Y Y = Σ y ∣ x , s + ( β ( sy ) ) T Σ s ∣ x β ( sy ) + ( ( β ( x s ) ) T β ( sy ) + β ( x y ) ) T Σ x ( ( β ( x s ) ) T β ( sy ) + β ( x y ) ) (23)
Σ X Y = Σ x ( ( β ( x s ) ) T β ( s y ) + β ( x y ) ) (24) \Sigma_{XY} = \Sigma_x\left((\beta^{(xs)})^T\beta^{(sy)} + \beta^{(xy)}\right) \tag*{(24)} Σ X Y = Σ x ( ( β ( x s ) ) T β ( sy ) + β ( x y ) ) (24)
Σ X M = Σ x β ( x s ) β ( s m ) (25) \Sigma_{XM} = \Sigma_x\beta^{(xs)}\beta^{(sm)} \tag*{(25)} Σ X M = Σ x β ( x s ) β ( s m ) (25)
Σ M Y = ( β ( s m ) ) T Σ s ∣ x β ( s y ) + ( ( β ( x s ) β ( s m ) ) T Σ x ( ( β ( x s ) ) T β ( s y ) + β ( x y ) ) ) (26) \Sigma_{MY} = (\beta^{(sm)})^T\Sigma_{s|x}\beta^{(sy)} + \left((\beta^{(xs)}\beta^{(sm)})^T\Sigma_x\left((\beta^{(xs)})^T\beta^{(sy)} + \beta^{(xy)}\right)\right) \tag*{(26)} Σ M Y = ( β ( s m ) ) T Σ s ∣ x β ( sy ) + ( ( β ( x s ) β ( s m ) ) T Σ x ( ( β ( x s ) ) T β ( sy ) + β ( x y ) ) ) (26)
Equation (22) is exactly the equation for an FA model applied to M M M , which uniquely identifies the latent subspace. The cross-moment equations can then determine the other population parameters according to this subspace without altering it. In particular, the FA equation can be solved independently from the rest of the system when the model is correctly specified.
As shown in § 9 , K < rank ( C ) K < \operatorname{rank}(C) K < rank ( C ) induce low-rank constraints, where rank ( C ) ≤ min ( I + L , J ) \operatorname{rank}(C) \leq\min(I+L,J) rank ( C ) ≤ min ( I + L , J ) , meaning the model can only approximate the population moments Var ( M ) \operatorname{Var}(M) Var ( M ) , Cov ( X , M ) \operatorname{Cov}(X,M) Cov ( X , M ) and Cov ( M , Y ) \operatorname{Cov}(M,Y) Cov ( M , Y ) through joint optimization. Therefore, Eq. (22) cannot be solved analytically in isolation: its estimation results from a minimization problem defined by Eq. (22) , Eq. (25) and Eq. (26) , and the resulting latent space no longer matches with that of an FA solution.
In [7 ] however, K > I = 1 K > I = 1 K > I = 1 is large enough that the model is not rank-constrained. Then, β ( s m ) \beta^{(sm)} β ( s m ) and Σ m ∣ s \Sigma_{m|s} Σ m ∣ s can be obtained analytically from Eq. (22) as the solution of a FA model, under suitable identification conditions [2 ] . Once these are known, deriving a closed-form solution for β ( x s ) \beta^{(xs)} β ( x s ) , β ( s y ) \beta^{(sy)} β ( sy ) , β ( x y ) \beta^{(xy)} β ( x y ) , Σ x \Sigma_x Σ x , Σ y ∣ x , s \Sigma_{y|x,s} Σ y ∣ x , s is straightforward from the remaining population moment equations, provided the model remains correctly specified.
We support this observation numerically in Figure 5 -left, where the blue curve shows that the model in [7 ] recovers exactly the same solution as a simple FA model on M M M when K ≥ K t r u e = rank ( C ) K \geq K_{true} = \operatorname{rank}(C) K ≥ K t r u e = rank ( C ) , and starts to yield different solutions when K < K t r u e K < K_{true} K < K t r u e . The orange curve shows that the same model with unconstrained noise covariances does not yield the same solution as a FA model, regardless of the rank of C C C . This further demonstrates the importance of choosing K K K based on the system’s dimensions in a way that guarantees the model does not reduce to either FA or ordinary least squares problems. For exhaustiveness, we also report results when both models are incorrectly specified, that is, when the true Σ m ∣ s \Sigma_{m|s} Σ m ∣ s used for data generation is full, in Figure 5 -right; results are similar.
Figure 5. LVMA is similar to a factor analysis — (Left) Data generated using a diagonal Σ m ∣ s \Sigma_{m|s} Σ m ∣ s ; (Right) Data generated using a full Σ m ∣ s \Sigma_{m|s} Σ m ∣ s . We report the MSE between the covariance of M M M explained by S S S from a factor analysis model and the same quantity estimated from one of two latent models, one with diagonal conditional covariance Σ m ∣ s \Sigma_{m|s} Σ m ∣ s , and one with full conditional covariance Σ m ∣ s \Sigma_{m|s} Σ m ∣ s . We generate N = 1000 N = 1000 N = 1000 data points with I = J = 50 I = J = 50 I = J = 50 , L = 1 L = 1 L = 1 and K t r u e = rank ( C ) K_{true} = \operatorname{rank}(C) K t r u e = rank ( C ) (see Eq. (20) ) denoting the varying rank of the system used for data generation. We fit the LVMA model with K = 25 K = 25 K = 25 . Note that I > 1 I > 1 I > 1 was not considered in LVMA, although it provides more informative visualizations for our demonstration. The dashed black line indicates when K t r u e = K K_{true} = K K t r u e = K . The shaded region indicates the standard deviation over 50 runs. This shows that forcing Σ m ∣ s \Sigma_{m|s} Σ m ∣ s to be diagonal makes the latent model equivalent to a factor analysis model when K ≥ K t r u e K \geq K_{true} K ≥ K t r u e .
E-step developments
Jensen's inequality
Here, we derive an expression for a lower bound for log p ( X , M , Y ∣ θ ) \log p(X,M,Y|\theta) log p ( X , M , Y ∣ θ ) . Let q ( S ) q(S) q ( S ) denote any distribution of S S S :
log p ( X , M , Y ∣ θ ) = log p ( X , M , Y ∣ θ ) ∫ S q ( S ) d S ⏟ = 1 = ∫ S q ( S ) log p ( X , M , Y ∣ θ ) d S = ∫ S q ( S ) log ( q ( S ) p ( S , X , M , Y ∣ θ ) q ( S ) p ( S , X , M , Y ∣ θ ) p ( X , M , Y ∣ θ ) ) d S = ∫ S q ( S ) log p ( S , X , M , Y ∣ θ ) q ( S ) d S + ∫ S q ( S ) log q ( S ) p ( X , M , Y ∣ θ ) p ( S , X , M , Y ∣ θ ) d S = ∫ S q ( S ) log p ( S , X , M , Y ∣ θ ) q ( S ) d S + ∫ S q ( S ) log q ( S ) p ( S ∣ X , M , Y , θ ) d S ≥ ∫ S q ( S ) log p ( S , X , M , Y ∣ θ ) q ( S ) d S (27) \begin{aligned}
\log p(X,M,Y|\theta) &= \log p(X,M,Y|\theta)\underbrace{\int_{S}q(S)dS}_{=1} \\
&= \int_{S}q(S)\log p(X,M,Y|\theta)dS \\
&= \int_{S}q(S)\log\left(\frac{q(S)p(S,X,M,Y|\theta)}{q(S)p(S,X,M,Y|\theta)}p(X,M,Y|\theta)\right)dS \\
&= \int_{S}q(S)\log\frac{p(S,X,M,Y|\theta)}{q(S)}dS+\int_{S}q(S)\log\frac{q(S)p(X,M,Y|\theta)}{p(S,X,M,Y|\theta)}dS \\
&= \int_{S}q(S)\log\frac{p(S,X,M,Y|\theta)}{q(S)}dS+\int_{S}q(S)\log\frac{q(S)}{p(S|X,M,Y,\theta)}dS \\
&\ge\int_{S}q(S)\log\frac{p(S,X,M,Y|\theta)}{q(S)}dS \tag*{(27)}
\end{aligned} log p ( X , M , Y ∣ θ ) = log p ( X , M , Y ∣ θ ) = 1 ∫ S q ( S ) d S = ∫ S q ( S ) log p ( X , M , Y ∣ θ ) d S = ∫ S q ( S ) log ( q ( S ) p ( S , X , M , Y ∣ θ ) q ( S ) p ( S , X , M , Y ∣ θ ) p ( X , M , Y ∣ θ ) ) d S = ∫ S q ( S ) log q ( S ) p ( S , X , M , Y ∣ θ ) d S + ∫ S q ( S ) log p ( S , X , M , Y ∣ θ ) q ( S ) p ( X , M , Y ∣ θ ) d S = ∫ S q ( S ) log q ( S ) p ( S , X , M , Y ∣ θ ) d S + ∫ S q ( S ) log p ( S ∣ X , M , Y , θ ) q ( S ) d S ≥ ∫ S q ( S ) log q ( S ) p ( S , X , M , Y ∣ θ ) d S (27)
where the Kullback-Leibler divergence is K L ( q θ , X , M , Y ( ⋅ ) ∣ ∣ p θ ( ⋅ ∣ X , M , Y ) ) : = ∫ S q ( S ) log q θ , o ( s ) p θ ( s ∣ o ) d S ≥ 0 KL(q_{\theta,X,M,Y}(\cdot)||p_{\theta}(\cdot|X,M,Y)):=\int_{S}q(S)\log\frac{q_{\theta,o}(s)}{p_{\theta}(s|o)}dS\ge0 K L ( q θ , X , M , Y ( ⋅ ) ∣∣ p θ ( ⋅ ∣ X , M , Y )) := ∫ S q ( S ) log p θ ( s ∣ o ) q θ , o ( s ) d S ≥ 0 and q ( S ) q(S) q ( S ) is any distribution of S S S .
Alternatively, using Jensen’s inequality (E [ log x ] ≤ log E [ x ] \mathbb{E}[\log x]\leq\log\mathbb{E}[x] E [ log x ] ≤ log E [ x ] ) directly:
log p ( X , M , Y ∣ θ ) = log ∫ S q ( S ) p ( S , X , M , Y ∣ θ ) q ( S ) d S ⏟ log E [ p ( S , X , M , Y ∣ θ ) q ( S ) ] ≥ ∫ S q ( S ) log p ( S , X , M , Y ∣ θ ) q ( S ) d S ⏟ E [ log p ( S , X , M , Y ∣ θ ) q ( S ) ] \log p(X,M,Y|\theta)=\log\underbrace{\int_{S}q(S)\frac{p(S,X,M,Y|\theta)}{q(S)}dS}_{\log\mathbb{E}\left[\frac{p(S,X,M,Y|\theta)}{q(S)}\right]}\geq\underbrace{\int_{S}q(S)\log\frac{p(S,X,M,Y|\theta)}{q(S)}dS}_{\mathbb{E}\left[\log\frac{p(S,X,M,Y|\theta)}{q(S)}\right]} log p ( X , M , Y ∣ θ ) = log l o g E [ q ( S ) p ( S , X , M , Y ∣ θ ) ] ∫ S q ( S ) q ( S ) p ( S , X , M , Y ∣ θ ) d S ≥ E [ l o g q ( S ) p ( S , X , M , Y ∣ θ ) ] ∫ S q ( S ) log q ( S ) p ( S , X , M , Y ∣ θ ) d S
Lower-bound maximization
Here, we demonstrate that the lower bound in Eq. (27) is equal to log p ( X , M , Y ∣ θ ) \log p(X,M,Y|\theta) log p ( X , M , Y ∣ θ ) when q ( S ) = q θ , X , M , Y ( S ) = p ( S , X , M , Y ∣ θ ) p ( X , M , Y ∣ θ ) q(S)=q_{\theta,X,M,Y}(S)=\frac{p(S,X,M,Y|\theta)}{p(X,M,Y|\theta)} q ( S ) = q θ , X , M , Y ( S ) = p ( X , M , Y ∣ θ ) p ( S , X , M , Y ∣ θ ) –it is maximized.
∫ S q θ , X , M , Y ( S ) log p ( S , X , M , Y ∣ θ ) q θ , X , M , Y ( S ) d S = ∫ S p ( S , X , M , Y ∣ θ ) p ( X , M , Y ∣ θ ) log p ( S , X , M , Y ∣ θ ) p ( S , X , M , Y ∣ θ ) p ( X , M , Y ∣ θ ) d S = ∫ S p ( S , X , M , Y ∣ θ ) p ( X , M , Y ∣ θ ) log p ( X , M , Y ∣ θ ) d S = log p ( X , M , Y ∣ θ ) p ( X , M , Y ∣ θ ) ∫ S p ( S , X , M , Y ∣ θ ) d S ⏟ : = p ( X , M , Y ∣ θ ) = log p ( X , M , Y ∣ θ ) (28) \begin{aligned}
\int_{S}q_{\theta,X,M,Y}(S)\log\frac{p(S,X,M,Y|\theta)}{q_{\theta,X,M,Y}(S)}dS
&=\int_{S}\frac{p(S,X,M,Y|\theta)}{p(X,M,Y|\theta)}\log\frac{p(S,X,M,Y|\theta)}{\frac{p(S,X,M,Y|\theta)}{p(X,M,Y|\theta)}}dS \\
&=\int_{S}\frac{p(S,X,M,Y|\theta)}{p(X,M,Y|\theta)}\log p(X,M,Y|\theta)dS \\
&=\frac{\log p(X,M,Y|\theta)}{p(X,M,Y|\theta)}\underbrace{\int_{S}p(S,X,M,Y|\theta)dS}_{:=p(X,M,Y|\theta)} \\
&=\log p(X,M,Y|\theta) \tag*{(28)}
\end{aligned} ∫ S q θ , X , M , Y ( S ) log q θ , X , M , Y ( S ) p ( S , X , M , Y ∣ θ ) d S = ∫ S p ( X , M , Y ∣ θ ) p ( S , X , M , Y ∣ θ ) log p ( X , M , Y ∣ θ ) p ( S , X , M , Y ∣ θ ) p ( S , X , M , Y ∣ θ ) d S = ∫ S p ( X , M , Y ∣ θ ) p ( S , X , M , Y ∣ θ ) log p ( X , M , Y ∣ θ ) d S = p ( X , M , Y ∣ θ ) log p ( X , M , Y ∣ θ ) := p ( X , M , Y ∣ θ ) ∫ S p ( S , X , M , Y ∣ θ ) d S = log p ( X , M , Y ∣ θ ) (28)
The equality naturally follows (because a lower bound cannot be larger than the bounded quantity):
log p ( X , M , Y ∣ θ ) = ∫ S q θ , X , M , Y ( S ) log p ( S , X , M , Y ∣ θ ) q θ , X , M , Y ( S ) d S (29) \log p(X,M,Y|\theta)=\int_{S}q_{\theta,X,M,Y}(S)\log\frac{p(S,X,M,Y|\theta)}{q_{\theta,X,M,Y}(S)}dS \tag*{(29)} log p ( X , M , Y ∣ θ ) = ∫ S q θ , X , M , Y ( S ) log q θ , X , M , Y ( S ) p ( S , X , M , Y ∣ θ ) d S (29)
when q θ , X , M , Y ( S ) = p ( S , X , M , Y ∣ θ ) p ( X , M , Y ∣ θ ) q_{\theta,X,M,Y}(S)=\frac{p(S,X,M,Y|\theta)}{p(X,M,Y|\theta)} q θ , X , M , Y ( S ) = p ( X , M , Y ∣ θ ) p ( S , X , M , Y ∣ θ )
Full M-step derivation
Intercepts are omitted for notational clarity; they can be included in the usual way by augmenting the relevant design matrix with a column of ones.
LaMe-vX
The solutions for the LaMe-vX model are summarized below:
β ( x s ) = ( X T X ) − 1 X T S ˉ β ( s m ) = ( S T S ) − 1 S T M β ( x y ) = ( X T ( I − P S ) X ) − 1 X T ( I − P S ) Y β ( s y ) = ( S ˉ T S ˉ ) − 1 S ˉ T ( Y − X β ( x y ) ) (30) \begin{aligned}
\beta^{(xs)}&=(X^{T}X)^{-1}X^{T}\bar{S}
&\qquad\beta^{(sm)}&=(S^{T}S)^{-1}S^{T}M \\
\beta^{(xy)}&=\left(X^{T}(I-P_{S})X\right)^{-1}X^{T}(I-P_{S})Y
&\qquad\beta^{(sy)}&=(\bar{S}^{T}\bar{S})^{-1}\bar{S}^{T}(Y-X\beta^{(xy)}) \tag*{(30)}
\end{aligned} β ( x s ) β ( x y ) = ( X T X ) − 1 X T S ˉ = ( X T ( I − P S ) X ) − 1 X T ( I − P S ) Y β ( s m ) β ( sy ) = ( S T S ) − 1 S T M = ( S ˉ T S ˉ ) − 1 S ˉ T ( Y − X β ( x y ) ) (30)
Σ x = 1 N X T X (31) \Sigma_x = \frac{1}{N}X^T X \tag*{(31)} Σ x = N 1 X T X (31)
Σ s ∣ x = 1 N ( S T S ‾ + ( S ‾ − X β ( x s ) ) T ( S ‾ − X β ( x s ) ) − S ‾ T S ‾ ) \Sigma_{s|x} = \frac{1}{N}\left(\overline{S^T S}+\left(\overline{S}-X\beta^{(xs)}\right)^T\left(\overline{S}-X\beta^{(xs)}\right)-\overline{S}^T\overline{S}\right) Σ s ∣ x = N 1 ( S T S + ( S − X β ( x s ) ) T ( S − X β ( x s ) ) − S T S )
Σ m ∣ s = 1 N ( ( β ( s m ) ) T ( S T S ) ‾ β ( s m ) + ( M − S ‾ β ( s m ) ) T ( M − S ‾ β ( s m ) ) − ( S ‾ β ( s m ) ) T ( S ‾ β ( s m ) ) ) \Sigma_{m|s} = \frac{1}{N}\left((\beta^{(sm)})^T\overline{(S^T S)}\beta^{(sm)}+\left(M-\overline{S}\beta^{(sm)}\right)^T\left(M-\overline{S}\beta^{(sm)}\right)-\left(\overline{S}\beta^{(sm)}\right)^T\left(\overline{S}\beta^{(sm)}\right)\right) Σ m ∣ s = N 1 ( ( β ( s m ) ) T ( S T S ) β ( s m ) + ( M − S β ( s m ) ) T ( M − S β ( s m ) ) − ( S β ( s m ) ) T ( S β ( s m ) ) )
Σ y ∣ x , s = 1 N ( ( β ( s y ) ) T ( S T S ) ‾ β ( s y ) + ( Y − X β ( x y ) − S ‾ β ( s y ) ) T ( Y − X β ( x y ) − S ‾ β ( s y ) ) − ( S ‾ β ( s y ) ) T ( S ‾ β ( s y ) ) ) \Sigma_{y|x,s} = \frac{1}{N}\left((\beta^{(sy)})^T\overline{(S^T S)}\beta^{(sy)}+\left(Y-X\beta^{(xy)}-\overline{S}\beta^{(sy)}\right)^T\left(Y-X\beta^{(xy)}-\overline{S}\beta^{(sy)}\right)-\left(\overline{S}\beta^{(sy)}\right)^T\left(\overline{S}\beta^{(sy)}\right)\right) Σ y ∣ x , s = N 1 ( ( β ( sy ) ) T ( S T S ) β ( sy ) + ( Y − X β ( x y ) − S β ( sy ) ) T ( Y − X β ( x y ) − S β ( sy ) ) − ( S β ( sy ) ) T ( S β ( sy ) ) )
where P S : = S ‾ ( S T S ‾ ) − 1 S ‾ T P_S := \overline{S}\left(\overline{S^T S}\right)^{-1}\overline{S}^T P S := S ( S T S ) − 1 S T is the estimated latent projection-like matrix.
A A A , B B B , C C C .
p ( S , X , M , Y ∣ θ ) : = e − 1 2 ⟨ S T S , A ⟩ + ⟨ S , B ⟩ + C p(S,X,M,Y|\theta) := e^{-\frac{1}{2}\langle S^T S,A\rangle+\langle S,B\rangle+C} p ( S , X , M , Y ∣ θ ) := e − 2 1 ⟨ S T S , A ⟩ + ⟨ S , B ⟩ + C
A : = Σ s ∣ x − 1 + β ( s m ) Σ m ∣ s − 1 ( β ( s m ) ) T + β ( s y ) Σ y ∣ x , s − 1 ( β ( s y ) ) T A := \Sigma_{s|x}^{-1}+\beta^{(sm)}\Sigma_{m|s}^{-1}(\beta^{(sm)})^T+\beta^{(sy)}\Sigma_{y|x,s}^{-1}(\beta^{(sy)})^T A := Σ s ∣ x − 1 + β ( s m ) Σ m ∣ s − 1 ( β ( s m ) ) T + β ( sy ) Σ y ∣ x , s − 1 ( β ( sy ) ) T
B : = Y Σ y ∣ x , s − 1 ( β ( s y ) ) T + X β ( x s ) Σ s ∣ x − 1 − X β ( x y ) Σ y ∣ x , s − 1 ( β ( s y ) ) T + M Σ m ∣ s − 1 ( β ( s m ) ) T B := Y\Sigma_{y|x,s}^{-1}(\beta^{(sy)})^T+X\beta^{(xs)}\Sigma_{s|x}^{-1}-X\beta^{(xy)}\Sigma_{y|x,s}^{-1}(\beta^{(sy)})^T+M\Sigma_{m|s}^{-1}(\beta^{(sm)})^T B := Y Σ y ∣ x , s − 1 ( β ( sy ) ) T + X β ( x s ) Σ s ∣ x − 1 − X β ( x y ) Σ y ∣ x , s − 1 ( β ( sy ) ) T + M Σ m ∣ s − 1 ( β ( s m ) ) T
C : = − 1 2 ⟨ X T X , ( Σ x − 1 + β ( x s ) Σ s ∣ x − 1 ( β ( x s ) ) T + β ( x y ) Σ y ∣ x , s − 1 ( β ( x y ) ) T ) ⟩ − 1 2 ⟨ M T M , Σ m ∣ s − 1 ⟩ − 1 2 ⟨ Y T Y , Σ y ∣ x , s − 1 ⟩ + ⟨ X T Y , β ( x y ) Σ y ∣ x , s − 1 ⟩ − N 2 log ( ( 2 π ) I + J + K + L ∣ Σ x ∣ ∣ Σ s ∣ x ∣ ∣ Σ m ∣ s ∣ ∣ Σ y ∣ x , s ∣ ) \begin{aligned}
C := {}&-\frac{1}{2}\langle X^T X,\left(\Sigma_x^{-1}+\beta^{(xs)}\Sigma_{s|x}^{-1}(\beta^{(xs)})^T+\beta^{(xy)}\Sigma_{y|x,s}^{-1}(\beta^{(xy)})^T\right)\rangle\\
&-\frac{1}{2}\langle M^T M,\Sigma_{m|s}^{-1}\rangle-\frac{1}{2}\langle Y^T Y,\Sigma_{y|x,s}^{-1}\rangle\\
&+\langle X^T Y,\beta^{(xy)}\Sigma_{y|x,s}^{-1}\rangle-\frac{N}{2}\log\left((2\pi)^{I+J+K+L}|\Sigma_x||\Sigma_{s|x}||\Sigma_{m|s}||\Sigma_{y|x,s}|\right)
\end{aligned} C := − 2 1 ⟨ X T X , ( Σ x − 1 + β ( x s ) Σ s ∣ x − 1 ( β ( x s ) ) T + β ( x y ) Σ y ∣ x , s − 1 ( β ( x y ) ) T ) ⟩ − 2 1 ⟨ M T M , Σ m ∣ s − 1 ⟩ − 2 1 ⟨ Y T Y , Σ y ∣ x , s − 1 ⟩ + ⟨ X T Y , β ( x y ) Σ y ∣ x , s − 1 ⟩ − 2 N log ( ( 2 π ) I + J + K + L ∣ Σ x ∣∣ Σ s ∣ x ∣∣ Σ m ∣ s ∣∣ Σ y ∣ x , s ∣ )
Let ℓ ˉ θ \bar{\ell}_\theta ℓ ˉ θ denote the expected incomplete log-likelihood (see the main text):
ℓ ˉ θ = ∫ S q θ , X , M , Y ( S ) log p ( S , X , M , Y ∣ θ ) q θ , X , M , Y ( S ) d S \bar{\ell}_\theta=\int_S q_{\theta,X,M,Y}(S)\log\frac{p(S,X,M,Y|\theta)}{q_{\theta,X,M,Y}(S)}\,dS ℓ ˉ θ = ∫ S q θ , X , M , Y ( S ) log q θ , X , M , Y ( S ) p ( S , X , M , Y ∣ θ ) d S
x → s x\to s x → s . For β ( x s ) \beta^{(xs)} β ( x s ) , Σ s ∣ x \Sigma_{s|x} Σ s ∣ x :
∂ β ( x s ) ℓ ˉ θ = ∑ i N ∫ S q θ , x i , m i , y i ( s i ) ∂ β ( x s ) ( − 1 2 ( s i − x i β ( x s ) ) Σ s ∣ x − 1 ( s i − x i β ( x s ) ) T ) d s i = 0 = ∑ i N ∫ S q θ , x i , m i , y i ( s i ) ∂ β ( x s ) ( ( s i − x i β ( x s ) ) Σ s ∣ x − 1 ( s i − x i β ( x s ) ) T ) d s i = 0 = ∑ i N ∫ S q θ , x i , m i , y i ( s i ) ( ( − x i T ) ( s i − x i β ( x s ) ) Σ s ∣ x − 1 ) d s i = 0 \begin{aligned}
\partial_{\beta^{(xs)}}\bar{\ell}_\theta
={}&\sum_{i}^{N}\int_S q_{\theta,x_i,m_i,y_i}(s_i)\partial_{\beta^{(xs)}}\left(-\frac{1}{2}(s_i-x_i\beta^{(xs)})\Sigma_{s|x}^{-1}(s_i-x_i\beta^{(xs)})^T\right)\,ds_i=0\\
={}&\sum_{i}^{N}\int_S q_{\theta,x_i,m_i,y_i}(s_i)\partial_{\beta^{(xs)}}\left((s_i-x_i\beta^{(xs)})\Sigma_{s|x}^{-1}(s_i-x_i\beta^{(xs)})^T\right)\,ds_i=0\\
={}&\sum_{i}^{N}\int_S q_{\theta,x_i,m_i,y_i}(s_i)\left((-x_i^T)(s_i-x_i\beta^{(xs)})\Sigma_{s|x}^{-1}\right)\,ds_i=0
\end{aligned} ∂ β ( x s ) ℓ ˉ θ = = = i ∑ N ∫ S q θ , x i , m i , y i ( s i ) ∂ β ( x s ) ( − 2 1 ( s i − x i β ( x s ) ) Σ s ∣ x − 1 ( s i − x i β ( x s ) ) T ) d s i = 0 i ∑ N ∫ S q θ , x i , m i , y i ( s i ) ∂ β ( x s ) ( ( s i − x i β ( x s ) ) Σ s ∣ x − 1 ( s i − x i β ( x s ) ) T ) d s i = 0 i ∑ N ∫ S q θ , x i , m i , y i ( s i ) ( ( − x i T ) ( s i − x i β ( x s ) ) Σ s ∣ x − 1 ) d s i = 0
⇔ β ( x s ) = ( ∑ i N x i T x i ) − 1 ∑ i N x i T s i ‾ = ( X T X ) − 1 X T S ‾ \begin{aligned}
\Leftrightarrow\ \beta^{(xs)}
&=\left(\sum_i^N x_i^T x_i\right)^{-1}\sum_i^N x_i^T\overline{s_i}\\
&=(X^T X)^{-1}X^T\overline{S}
\end{aligned} ⇔ β ( x s ) = ( i ∑ N x i T x i ) − 1 i ∑ N x i T s i = ( X T X ) − 1 X T S
∂ Σ s ∣ x ℓ ˉ θ = ∑ i N ∫ S q θ , x i , m i , y i ( s i ) ∂ Σ s ∣ x ( − 1 2 ( s i − x i β ( x s ) ) Σ s ∣ x − 1 ( s i − x i β ( x s ) ) T − 1 2 ln ( ∣ Σ s ∣ x ∣ ) ) d s i = 0 = ∑ i N ∫ S q θ , x i , m i , y i ( s i ) ( Σ s ∣ x − 1 ( s i − x i β ( x s ) ) T ( s i − x i β ( x s ) ) Σ s ∣ x − 1 − Σ s ∣ x − 1 ) d s i = 0 \begin{aligned}
\partial_{\Sigma_{s|x}}\bar{\ell}_\theta
={}&\sum_i^N\int_S q_{\theta,x_i,m_i,y_i}(s_i)\partial_{\Sigma_{s|x}}\left(-\frac{1}{2}(s_i-x_i\beta^{(xs)})\Sigma_{s|x}^{-1}(s_i-x_i\beta^{(xs)})^T-\frac{1}{2}\ln(|\Sigma_{s|x}|)\right)\,ds_i=0\\
={}&\sum_i^N\int_S q_{\theta,x_i,m_i,y_i}(s_i)\left(\Sigma_{s|x}^{-1}(s_i-x_i\beta^{(xs)})^T(s_i-x_i\beta^{(xs)})\Sigma_{s|x}^{-1}-\Sigma_{s|x}^{-1}\right)\,ds_i=0
\end{aligned} ∂ Σ s ∣ x ℓ ˉ θ = = i ∑ N ∫ S q θ , x i , m i , y i ( s i ) ∂ Σ s ∣ x ( − 2 1 ( s i − x i β ( x s ) ) Σ s ∣ x − 1 ( s i − x i β ( x s ) ) T − 2 1 ln ( ∣ Σ s ∣ x ∣ ) ) d s i = 0 i ∑ N ∫ S q θ , x i , m i , y i ( s i ) ( Σ s ∣ x − 1 ( s i − x i β ( x s ) ) T ( s i − x i β ( x s ) ) Σ s ∣ x − 1 − Σ s ∣ x − 1 ) d s i = 0
⇔ Σ s ∣ x = 1 N ∑ i N ∫ S q θ , x i , m i , y i ( s i ) ( ( s i − x i β ( x s ) ) T ( s i − x i β ( x s ) ) ) d s i = 1 N ∑ i N s i T s i ‾ − s i ‾ T ( x i β ( x s ) ) − ( x i β ( x s ) ) T s i ‾ + ( x i β ( x s ) ) T ( x i β ( x s ) ) = 1 N ( S T S ‾ − S ‾ T X β ( x s ) − ( X β ( x s ) ) T S ‾ + ( X β ( x s ) ) T ( X β ( x s ) ) ) = 1 N ( S T S ‾ + ( S ‾ − X β ( x s ) ) T ( S ‾ − X β ( x s ) ) − S ‾ T S ‾ ) \begin{aligned}
\Leftrightarrow\ \Sigma_{s|x}
&=\frac{1}{N}\sum_i^N\int_S q_{\theta,x_i,m_i,y_i}(s_i)\left((s_i-x_i\beta^{(xs)})^T(s_i-x_i\beta^{(xs)})\right)\,ds_i\\
&=\frac{1}{N}\sum_i^N\overline{s_i^T s_i}-\overline{s_i}^T(x_i\beta^{(xs)})-(x_i\beta^{(xs)})^T\overline{s_i}+(x_i\beta^{(xs)})^T(x_i\beta^{(xs)})\\
&=\frac{1}{N}\left(\overline{S^T S}-\overline{S}^T X\beta^{(xs)}-(X\beta^{(xs)})^T\overline{S}+(X\beta^{(xs)})^T(X\beta^{(xs)})\right)\\
&=\frac{1}{N}\left(\overline{S^T S}+\left(\overline{S}-X\beta^{(xs)}\right)^T\left(\overline{S}-X\beta^{(xs)}\right)-\overline{S}^T\overline{S}\right)
\end{aligned} ⇔ Σ s ∣ x = N 1 i ∑ N ∫ S q θ , x i , m i , y i ( s i ) ( ( s i − x i β ( x s ) ) T ( s i − x i β ( x s ) ) ) d s i = N 1 i ∑ N s i T s i − s i T ( x i β ( x s ) ) − ( x i β ( x s ) ) T s i + ( x i β ( x s ) ) T ( x i β ( x s ) ) = N 1 ( S T S − S T X β ( x s ) − ( X β ( x s ) ) T S + ( X β ( x s ) ) T ( X β ( x s ) ) ) = N 1 ( S T S + ( S − X β ( x s ) ) T ( S − X β ( x s ) ) − S T S )
s → m s \to m s → m
For β ( s m ) \beta^{(sm)} β ( s m ) , Σ m ∣ s \Sigma_{m\mid s} Σ m ∣ s :
∂ β ( s m ) ℓ ˉ θ = ∑ i N ∫ S q θ , x i , m i , y i ( s i ) ( − 2 s i T ) ( m i − s i β ( s m ) ) Σ m ∣ s − 1 d s i = 0 ⇔ β ( s m ) = ( ∑ i N s ˉ i T s ˉ i ) − 1 ∑ i N s ˉ i T m i = ( S ˉ T S ˉ ) − 1 S ˉ T M \begin{aligned}
\partial_{\beta^{(sm)}}\bar{\ell}_{\theta}
= \sum_{i}^{N}\int_{S}q_{\theta,x_i,m_i,y_i}(s_i)\left(-2s_i^{T}\right)\left(m_i-s_i\beta^{(sm)}\right)\Sigma_{m\mid s}^{-1}\,ds_i=0 \\
&\Leftrightarrow\ \beta^{(sm)}
=\left(\sum_{i}^{N}\bar{s}_i^{T}\bar{s}_i\right)^{-1}\sum_{i}^{N}\bar{s}_i^{T}m_i
=\left(\bar{S}^{T}\bar{S}\right)^{-1}\bar{S}^{T}M
\end{aligned} ∂ β ( s m ) ℓ ˉ θ = i ∑ N ∫ S q θ , x i , m i , y i ( s i ) ( − 2 s i T ) ( m i − s i β ( s m ) ) Σ m ∣ s − 1 d s i = 0 ⇔ β ( s m ) = ( i ∑ N s ˉ i T s ˉ i ) − 1 i ∑ N s ˉ i T m i = ( S ˉ T S ˉ ) − 1 S ˉ T M
∂ Σ m ∣ s ℓ ˉ θ = ∑ i N ∫ S q θ , x i , m i , y i ( s i ) ( − Σ m ∣ s − 1 ( m i − s i β ( s m ) ) T ( m i − s i β ( s m ) ) Σ m ∣ s − 1 + Σ m ∣ s − 1 ) d s i = 0 ⇔ Σ m ∣ s = 1 N ∑ i N ∫ S q θ , x i , m i , y i ( s i ) ( m i − s i β ( s m ) ) T ( m i − s i β ( s m ) ) d s i = 1 N ∑ i N m i T m i − m i T ( s ˉ i β ( s m ) ) − ( s ˉ i β ( s m ) ) T m i + ∫ S q θ , x i , m i , y i ( s i ) ( s i β ( s m ) ) T ( s i β ( s m ) ) d s = 1 N ∑ i N m i T m i − m i T ( s ˉ i β ( s m ) ) − ( s ˉ i β ( s m ) ) T m i + ( β ( s m ) ) T S ˉ i T S ˉ i β ( s m ) = 1 N ( M T M − M T S ˉ β ( s m ) − ( S ˉ β ( s m ) ) T M + ( β ( s m ) ) T ( S ˉ T S ˉ ) β ( s m ) ) = 1 N ( ( β ( s m ) ) T ( S ˉ T S ˉ ) β ( s m ) + ( M − S ˉ β ( s m ) ) T ( M − S ˉ β ( s m ) ) − ( S ˉ β ( s m ) ) T ( S ˉ β ( s m ) ) ) \begin{aligned}
\partial_{\Sigma_{m\mid s}}\bar{\ell}_{\theta}
=\sum_{i}^{N}\int_{S}q_{\theta,x_i,m_i,y_i}(s_i)\left(-\Sigma_{m\mid s}^{-1}\left(m_i-s_i\beta^{(sm)}\right)^{T}\left(m_i-s_i\beta^{(sm)}\right)\Sigma_{m\mid s}^{-1}+\Sigma_{m\mid s}^{-1}\right)\,ds_i=0 \\
&\Leftrightarrow\ \Sigma_{m\mid s}
=\frac{1}{N}\sum_{i}^{N}\int_{S}q_{\theta,x_i,m_i,y_i}(s_i)\left(m_i-s_i\beta^{(sm)}\right)^{T}\left(m_i-s_i\beta^{(sm)}\right)\,ds_i \\
&=\frac{1}{N}\sum_{i}^{N}m_i^{T}m_i-m_i^{T}\left(\bar{s}_i\beta^{(sm)}\right)-\left(\bar{s}_i\beta^{(sm)}\right)^{T}m_i+\int_{S}q_{\theta,x_i,m_i,y_i}(s_i)\left(s_i\beta^{(sm)}\right)^{T}\left(s_i\beta^{(sm)}\right)\,ds \\
&=\frac{1}{N}\sum_{i}^{N}m_i^{T}m_i-m_i^{T}\left(\bar{s}_i\beta^{(sm)}\right)-\left(\bar{s}_i\beta^{(sm)}\right)^{T}m_i+\left(\beta^{(sm)}\right)^{T}\bar{S}_i^{T}\bar{S}_i\beta^{(sm)} \\
&=\frac{1}{N}\left(M^{T}M-M^{T}\bar{S}\beta^{(sm)}-\left(\bar{S}\beta^{(sm)}\right)^{T}M+\left(\beta^{(sm)}\right)^{T}\left(\bar{S}^{T}\bar{S}\right)\beta^{(sm)}\right) \\
&=\frac{1}{N}\left(\left(\beta^{(sm)}\right)^{T}\left(\bar{S}^{T}\bar{S}\right)\beta^{(sm)}+\left(M-\bar{S}\beta^{(sm)}\right)^{T}\left(M-\bar{S}\beta^{(sm)}\right)-\left(\bar{S}\beta^{(sm)}\right)^{T}\left(\bar{S}\beta^{(sm)}\right)\right)
\end{aligned} ∂ Σ m ∣ s ℓ ˉ θ = i ∑ N ∫ S q θ , x i , m i , y i ( s i ) ( − Σ m ∣ s − 1 ( m i − s i β ( s m ) ) T ( m i − s i β ( s m ) ) Σ m ∣ s − 1 + Σ m ∣ s − 1 ) d s i = 0 ⇔ Σ m ∣ s = N 1 i ∑ N ∫ S q θ , x i , m i , y i ( s i ) ( m i − s i β ( s m ) ) T ( m i − s i β ( s m ) ) d s i = N 1 i ∑ N m i T m i − m i T ( s ˉ i β ( s m ) ) − ( s ˉ i β ( s m ) ) T m i + ∫ S q θ , x i , m i , y i ( s i ) ( s i β ( s m ) ) T ( s i β ( s m ) ) d s = N 1 i ∑ N m i T m i − m i T ( s ˉ i β ( s m ) ) − ( s ˉ i β ( s m ) ) T m i + ( β ( s m ) ) T S ˉ i T S ˉ i β ( s m ) = N 1 ( M T M − M T S ˉ β ( s m ) − ( S ˉ β ( s m ) ) T M + ( β ( s m ) ) T ( S ˉ T S ˉ ) β ( s m ) ) = N 1 ( ( β ( s m ) ) T ( S ˉ T S ˉ ) β ( s m ) + ( M − S ˉ β ( s m ) ) T ( M − S ˉ β ( s m ) ) − ( S ˉ β ( s m ) ) T ( S ˉ β ( s m ) ) )
x , s → y x,s \to y x , s → y
For β ( x y ) \beta^{(xy)} β ( x y ) , β ( s y ) \beta^{(sy)} β ( sy ) :
∂ β ( x y ) ℓ ˉ θ = ∑ i N ∫ S q θ , x i , m i , y i ( s i ) ( − 2 x i T ( Y i − x i β ( x y ) − s i β ( s y ) ) Σ y ∣ x , s − 1 ) d s i = 0 ⇔ β ( x y ) = ( ∑ i N x i T x i ) − 1 ( ∑ i N x i T Y i − x ˉ i T s ˉ i β ( s y ) ) = ( X T X ) − 1 X T Y − ( X T X ) − 1 ( X T S ˉ ) β ( s y ) \begin{aligned}
\partial_{\beta^{(xy)}}\bar{\ell}_{\theta}
=\sum_{i}^{N}\int_{S}q_{\theta,x_i,m_i,y_i}(s_i)\left(-2x_i^{T}\left(Y_i-x_i\beta^{(xy)}-s_i\beta^{(sy)}\right)\Sigma_{y\mid x,s}^{-1}\right)\,ds_i=0 \\
&\Leftrightarrow\ \beta^{(xy)}
=\left(\sum_{i}^{N}x_i^{T}x_i\right)^{-1}\left(\sum_{i}^{N}x_i^{T}Y_i-\bar{x}_i^{T}\bar{s}_i\beta^{(sy)}\right) \\
&=\left(X^{T}X\right)^{-1}X^{T}Y-\left(X^{T}X\right)^{-1}\left(X^{T}\bar{S}\right)\beta^{(sy)}
\end{aligned} ∂ β ( x y ) ℓ ˉ θ = i ∑ N ∫ S q θ , x i , m i , y i ( s i ) ( − 2 x i T ( Y i − x i β ( x y ) − s i β ( sy ) ) Σ y ∣ x , s − 1 ) d s i = 0 ⇔ β ( x y ) = ( i ∑ N x i T x i ) − 1 ( i ∑ N x i T Y i − x ˉ i T s ˉ i β ( sy ) ) = ( X T X ) − 1 X T Y − ( X T X ) − 1 ( X T S ˉ ) β ( sy )
∂ β ( s y ) ℓ ˉ θ = ∑ i N ∫ S q θ , x i , m i , y i ( s i ) ( − 2 s i T ( Y i − x i β ( x y ) − s i β ( s y ) ) Σ y ∣ x , s − 1 ) d s i = 0 ⇔ β ( s y ) = ( ∑ i N s ˉ i T s ˉ i ) − 1 ( ∑ i N s ˉ i T Y i − s ˉ i T x i β ( x y ) ) = ( S ˉ T S ˉ ) − 1 S ˉ T Y − ( S ˉ T S ˉ ) − 1 ( S ˉ T X ) β ( x y ) \begin{aligned}
\partial_{\beta^{(sy)}}\bar{\ell}_{\theta}
=\sum_{i}^{N}\int_{S}q_{\theta,x_i,m_i,y_i}(s_i)\left(-2s_i^{T}\left(Y_i-x_i\beta^{(xy)}-s_i\beta^{(sy)}\right)\Sigma_{y\mid x,s}^{-1}\right)\,ds_i=0 \\
&\Leftrightarrow\ \beta^{(sy)}
=\left(\sum_{i}^{N}\bar{s}_i^{T}\bar{s}_i\right)^{-1}\left(\sum_{i}^{N}\bar{s}_i^{T}Y_i-\bar{s}_i^{T}x_i\beta^{(xy)}\right) \\
&=\left(\bar{S}^{T}\bar{S}\right)^{-1}\bar{S}^{T}Y-\left(\bar{S}^{T}\bar{S}\right)^{-1}\left(\bar{S}^{T}X\right)\beta^{(xy)}
\end{aligned} ∂ β ( sy ) ℓ ˉ θ = i ∑ N ∫ S q θ , x i , m i , y i ( s i ) ( − 2 s i T ( Y i − x i β ( x y ) − s i β ( sy ) ) Σ y ∣ x , s − 1 ) d s i = 0 ⇔ β ( sy ) = ( i ∑ N s ˉ i T s ˉ i ) − 1 ( i ∑ N s ˉ i T Y i − s ˉ i T x i β ( x y ) ) = ( S ˉ T S ˉ ) − 1 S ˉ T Y − ( S ˉ T S ˉ ) − 1 ( S ˉ T X ) β ( x y )
Thus:
( X T X X T S ˉ S ˉ T X S ˉ T S ˉ ) ( β ( x y ) β ( s y ) ) = ( X T Y S ˉ T Y ) \begin{pmatrix}
X^{T}X & X^{T}\bar{S}\\
\bar{S}^{T}X & \bar{S}^{T}\bar{S}
\end{pmatrix}
\begin{pmatrix}
\beta^{(xy)}\\
\beta^{(sy)}
\end{pmatrix}
=
\begin{pmatrix}
X^{T}Y\\
\bar{S}^{T}Y
\end{pmatrix} ( X T X S ˉ T X X T S ˉ S ˉ T S ˉ ) ( β ( x y ) β ( sy ) ) = ( X T Y S ˉ T Y )
The solution is Θ = A − 1 r \Theta=A^{-1}r Θ = A − 1 r .
Finally for Σ y ∣ x , s \Sigma_{y|x,s} Σ y ∣ x , s :
∂ Σ y ∣ x , s ℓ ˉ θ = ∑ i N ∫ S q θ , x i , m i , y i ( s i ) ( Σ y ∣ x , s − 1 ( Y i − x i β ( x y ) − s i β ( s y ) ) T ( Y i − x i β ( x y ) − s i β ( s y ) ) Σ y ∣ x , s − 1 − Σ y ∣ x , s − 1 ) d s i = 0 ⇔ Σ y ∣ x , s = 1 N ∑ i N ∫ S q θ , x i , m i , y i ( s i ) ( Y i − x i β ( x y ) − s i β ( s y ) ) ( Y i − x i β ( x y ) − s i β ( s y ) ) T d s i = 1 N ( ( Y − X β ( x y ) − S ‾ β ( s y ) ) T ( Y − X β ( x y ) − S ‾ β ( s y ) ) − ( S β ( s y ) ) T ( S β ( s y ) ) ) + 1 N ( β ( s y ) ) T S T S β ( s y ) \begin{aligned}
\frac{\partial_{\Sigma_{y|x,s}} \bar{\ell}_{\theta}}{} =
\sum_{i}^{N}\int_{S} q_{\theta,x_i,m_i,y_i}(s_i)\left(\Sigma_{y|x,s}^{-1}(Y_i-x_i\beta^{(xy)}-s_i\beta^{(sy)})^{T}(Y_i-x_i\beta^{(xy)}-s_i\beta^{(sy)})\Sigma_{y|x,s}^{-1}-\Sigma_{y|x,s}^{-1}\right)\,ds_i
=0 \\
&\Leftrightarrow\Sigma_{y|x,s}=\frac{1}{N}\sum_{i}^{N}\int_{S}q_{\theta,x_i,m_i,y_i}(s_i)(Y_i-x_i\beta^{(xy)}-s_i\beta^{(sy)})(Y_i-x_i\beta^{(xy)}-s_i\beta^{(sy)})^{T}\,ds_i \\
&=\frac{1}{N}\left((Y-X\beta^{(xy)}-\overline{S}\beta^{(sy)})^{T}(Y-X\beta^{(xy)}-\overline{S}\beta^{(sy)})-(S\beta^{(sy)})^{T}(S\beta^{(sy)})\right)
+\frac{1}{N}(\beta^{(sy)})^{T}S^{T}S\beta^{(sy)}
\end{aligned} ∂ Σ y ∣ x , s ℓ ˉ θ = i ∑ N ∫ S q θ , x i , m i , y i ( s i ) ( Σ y ∣ x , s − 1 ( Y i − x i β ( x y ) − s i β ( sy ) ) T ( Y i − x i β ( x y ) − s i β ( sy ) ) Σ y ∣ x , s − 1 − Σ y ∣ x , s − 1 ) d s i = 0 ⇔ Σ y ∣ x , s = N 1 i ∑ N ∫ S q θ , x i , m i , y i ( s i ) ( Y i − x i β ( x y ) − s i β ( sy ) ) ( Y i − x i β ( x y ) − s i β ( sy ) ) T d s i = N 1 ( ( Y − X β ( x y ) − S β ( sy ) ) T ( Y − X β ( x y ) − S β ( sy ) ) − ( S β ( sy ) ) T ( S β ( sy ) ) ) + N 1 ( β ( sy ) ) T S T S β ( sy )
LaMe-vM
A A A , B B B , C C C .
p ( S , X , M , Y ∣ θ ) : = e − 1 2 ⟨ S T S , A ⟩ + ⟨ S , B ⟩ + C p(S,X,M,Y\mid\theta):=e^{-\frac{1}{2}\langle S^{T}S,A\rangle+\langle S,B\rangle+C} p ( S , X , M , Y ∣ θ ) := e − 2 1 ⟨ S T S , A ⟩ + ⟨ S , B ⟩ + C
A : = Σ s ∣ m − 1 + β ( s x ) Σ x ∣ s − 1 ( β ( s x ) ) T + β ( s y ) Σ y ∣ x , s − 1 ( β ( s y ) ) T A:=\Sigma_{s|m}^{-1}+\beta^{(sx)}\Sigma_{x|s}^{-1}(\beta^{(sx)})^{T}+\beta^{(sy)}\Sigma_{y|x,s}^{-1}(\beta^{(sy)})^{T} A := Σ s ∣ m − 1 + β ( s x ) Σ x ∣ s − 1 ( β ( s x ) ) T + β ( sy ) Σ y ∣ x , s − 1 ( β ( sy ) ) T
B : = M β ( m s ) Σ s ∣ m − 1 + X Σ x ∣ s − 1 ( β ( s x ) ) T − X β ( x y ) Σ y ∣ x , s − 1 ( β ( s y ) ) T + Y Σ y ∣ x , s − 1 ( β ( s y ) ) T B:=M\beta^{(ms)}\Sigma_{s|m}^{-1}+X\Sigma_{x|s}^{-1}(\beta^{(sx)})^{T}-X\beta^{(xy)}\Sigma_{y|x,s}^{-1}(\beta^{(sy)})^{T}+Y\Sigma_{y|x,s}^{-1}(\beta^{(sy)})^{T} B := M β ( m s ) Σ s ∣ m − 1 + X Σ x ∣ s − 1 ( β ( s x ) ) T − X β ( x y ) Σ y ∣ x , s − 1 ( β ( sy ) ) T + Y Σ y ∣ x , s − 1 ( β ( sy ) ) T
C : = − 1 2 ⟨ X T X , ( Σ x ∣ s − 1 + β ( x y ) Σ y ∣ x , s − 1 ( β ( x y ) ) T ) ⟩ C:=-\frac{1}{2}\left\langle X^{T}X,\left(\Sigma_{x|s}^{-1}+\beta^{(xy)}\Sigma_{y|x,s}^{-1}(\beta^{(xy)})^{T}\right)\right\rangle C := − 2 1 ⟨ X T X , ( Σ x ∣ s − 1 + β ( x y ) Σ y ∣ x , s − 1 ( β ( x y ) ) T ) ⟩
− 1 2 ⟨ M T M , ( Σ m − 1 + β ( m s ) Σ s ∣ m − 1 ( β ( m s ) ) T ) ⟩ − 1 2 ⟨ Y T Y , Σ y ∣ x , s − 1 ⟩ + ⟨ X T Y , β ( x y ) Σ y ∣ x , s − 1 ⟩ − N 2 log ( ( 2 π ) I + J + K + L ∣ Σ m ∣ ∣ Σ s ∣ m ∣ ∣ Σ x ∣ s ∣ ∣ Σ y ∣ x , s ∣ ) \begin{aligned}
-\frac{1}{2}\left\langle M^{T}M,\left(\Sigma_{m}^{-1}+\beta^{(ms)}\Sigma_{s|m}^{-1}(\beta^{(ms)})^{T}\right)\right\rangle-\frac{1}{2}\langle Y^{T}Y,\Sigma_{y|x,s}^{-1}\rangle\\
&+\langle X^{T}Y,\beta^{(xy)}\Sigma_{y|x,s}^{-1}\rangle-\frac{N}{2}\log\left((2\pi)^{I+J+K+L}|\Sigma_m||\Sigma_{s|m}||\Sigma_{x|s}||\Sigma_{y|x,s}|\right)
\end{aligned} − 2 1 ⟨ M T M , ( Σ m − 1 + β ( m s ) Σ s ∣ m − 1 ( β ( m s ) ) T ) ⟩ − 2 1 ⟨ Y T Y , Σ y ∣ x , s − 1 ⟩ + ⟨ X T Y , β ( x y ) Σ y ∣ x , s − 1 ⟩ − 2 N log ( ( 2 π ) I + J + K + L ∣ Σ m ∣∣ Σ s ∣ m ∣∣ Σ x ∣ s ∣∣ Σ y ∣ x , s ∣ )
m → s m\to s m → s . For β ( m s ) \beta^{(ms)} β ( m s ) , Σ s ∣ m \Sigma_{s|m} Σ s ∣ m :
∂ β ( m s ) ℓ ˉ θ ( O ) = ∑ i N ∫ S q θ , o i ( s i ) ∂ β ( m s ) ( − 1 2 ( s i − m i β ( m s ) ) Σ s ∣ m − 1 ( s i − m i β ( m s ) ) T ) d s i = 0 = ∑ i N ∫ S q θ , o i ( s i ) ( ( m i T ) ( s i − m i β ( m s ) ) Σ s ∣ m − 1 ) d s i = 0 = ( ∑ i N m i T m i ) β ( m s ) − ∑ i N m i T ( s i ‾ ) = 0 ⇔ β ( m s ) = ( ∑ i N m i T m i ) − 1 ∑ i N m i T ( s i ‾ ) = ( M T M ) − 1 M T S ‾ \begin{aligned}
\partial_{\beta^{(ms)}}\bar{\ell}_{\theta}(O)=\sum_{i}^{N}\int_{S}q_{\theta,o_i}(s_i)\partial_{\beta^{(ms)}}\left(-\frac{1}{2}(s_i-m_i\beta^{(ms)})\Sigma_{s|m}^{-1}(s_i-m_i\beta^{(ms)})^{T}\right)\,ds_i=0 \\
&=\sum_{i}^{N}\int_{S}q_{\theta,o_i}(s_i)\left((m_i^{T})(s_i-m_i\beta^{(ms)})\Sigma_{s|m}^{-1}\right)\,ds_i=0 \\
&=\left(\sum_{i}^{N}m_i^{T}m_i\right)\beta^{(ms)}-\sum_{i}^{N}m_i^{T}(\overline{s_i})=0 \\
&\Leftrightarrow\beta^{(ms)}=\left(\sum_{i}^{N}m_i^{T}m_i\right)^{-1}\sum_{i}^{N}m_i^{T}(\overline{s_i}) \\
&=(M^{T}M)^{-1}M^{T}\overline{S}
\end{aligned} ∂ β ( m s ) ℓ ˉ θ ( O ) = i ∑ N ∫ S q θ , o i ( s i ) ∂ β ( m s ) ( − 2 1 ( s i − m i β ( m s ) ) Σ s ∣ m − 1 ( s i − m i β ( m s ) ) T ) d s i = 0 = i ∑ N ∫ S q θ , o i ( s i ) ( ( m i T ) ( s i − m i β ( m s ) ) Σ s ∣ m − 1 ) d s i = 0 = ( i ∑ N m i T m i ) β ( m s ) − i ∑ N m i T ( s i ) = 0 ⇔ β ( m s ) = ( i ∑ N m i T m i ) − 1 i ∑ N m i T ( s i ) = ( M T M ) − 1 M T S
∂ Σ s ∣ m ℓ ˉ θ ( O ) = ∑ i N ∫ S q θ , o i ( s i ) ∂ Σ s ∣ m ( − ( s i − m i β ( m s ) ) Σ s ∣ m − 1 ( s i − m i β ( m s ) ) T − ln ( ∣ Σ s ∣ m ∣ ) ) d s i = 0 = ∑ i N ∫ S q θ , o i ( s i ) ( − Σ s ∣ m − 1 ( s i − m i β ( m s ) ) T ( s i − m i β ( m s ) ) Σ s ∣ m − 1 + Σ s ∣ m − 1 ) d s i = 0 ⇔ Σ s ∣ m = 1 N ∑ i N ∫ S q θ , o i ( s i ) ( ( s i − m i β ( m s ) ) T ( s i − m i β ( m s ) ) ) d s i = 1 N ∑ i N s i T s i ‾ + ( s i ‾ − m i β ( m s ) ) T ( s i ‾ − m i β ( m s ) ) − s i ‾ T s i ‾ = 1 N ( S T S + ( S ‾ − M β ( m s ) ) T ( S ‾ − M β ( m s ) ) − S ‾ T S ‾ ) \begin{aligned}
\partial_{\Sigma_{s|m}}\bar{\ell}_{\theta}(O)=\sum_{i}^{N}\int_{S}q_{\theta,o_i}(s_i)\partial_{\Sigma_{s|m}}\left(-(s_i-m_i\beta^{(ms)})\Sigma_{s|m}^{-1}(s_i-m_i\beta^{(ms)})^{T}-\ln(|\Sigma_{s|m}|)\right)\,ds_i=0 \\
&=\sum_{i}^{N}\int_{S}q_{\theta,o_i}(s_i)\left(-\Sigma_{s|m}^{-1}(s_i-m_i\beta^{(ms)})^{T}(s_i-m_i\beta^{(ms)})\Sigma_{s|m}^{-1}+\Sigma_{s|m}^{-1}\right)\,ds_i=0 \\
&\Leftrightarrow\Sigma_{s|m}=\frac{1}{N}\sum_{i}^{N}\int_{S}q_{\theta,o_i}(s_i)\left((s_i-m_i\beta^{(ms)})^{T}(s_i-m_i\beta^{(ms)})\right)\,ds_i \\
&=\frac{1}{N}\sum_{i}^{N}\overline{s_i^{T}s_i}+(\overline{s_i}-m_i\beta^{(ms)})^{T}(\overline{s_i}-m_i\beta^{(ms)})-\overline{s_i}^{T}\overline{s_i} \\
&=\frac{1}{N}\left(S^{T}S+(\overline{S}-M\beta^{(ms)})^{T}(\overline{S}-M\beta^{(ms)})-\overline{S}^{T}\overline{S}\right)
\end{aligned} ∂ Σ s ∣ m ℓ ˉ θ ( O ) = i ∑ N ∫ S q θ , o i ( s i ) ∂ Σ s ∣ m ( − ( s i − m i β ( m s ) ) Σ s ∣ m − 1 ( s i − m i β ( m s ) ) T − ln ( ∣ Σ s ∣ m ∣ ) ) d s i = 0 = i ∑ N ∫ S q θ , o i ( s i ) ( − Σ s ∣ m − 1 ( s i − m i β ( m s ) ) T ( s i − m i β ( m s ) ) Σ s ∣ m − 1 + Σ s ∣ m − 1 ) d s i = 0 ⇔ Σ s ∣ m = N 1 i ∑ N ∫ S q θ , o i ( s i ) ( ( s i − m i β ( m s ) ) T ( s i − m i β ( m s ) ) ) d s i = N 1 i ∑ N s i T s i + ( s i − m i β ( m s ) ) T ( s i − m i β ( m s ) ) − s i T s i = N 1 ( S T S + ( S − M β ( m s ) ) T ( S − M β ( m s ) ) − S T S )
s → x s \to x s → x
For β ( s x ) \beta^{(sx)} β ( s x ) , Σ x ∣ s \Sigma_{x|s} Σ x ∣ s :
∂ β ( s x ) ℓ ˉ θ ( O ) = ∑ i N ∫ S q θ , o i ( s i ) ( − 2 s i T ) ( x i − s i β ( s x ) ) Σ x ∣ s − 1 d s i = 0 \partial_{\beta^{(sx)}}\bar{\ell}_{\theta}(O)=\sum_{i}^{N}\int_{S}q_{\theta,o_i}(s_i)(-2s_i^{T})(x_i-s_i\beta^{(sx)})\Sigma_{x|s}^{-1}\,ds_i=0 ∂ β ( s x ) ℓ ˉ θ ( O ) = i ∑ N ∫ S q θ , o i ( s i ) ( − 2 s i T ) ( x i − s i β ( s x ) ) Σ x ∣ s − 1 d s i = 0
⟺ β ( s x ) = ( ∑ i N s ˉ i T s ˉ i ) − 1 ∑ i N s ˉ i T ( x i ) = ( S ˉ T S ˉ ) − 1 S ˉ T X \begin{aligned}
\Longleftrightarrow\ \beta^{(sx)}=\left(\sum_{i}^{N}\bar{s}_i^{T}\bar{s}_i\right)^{-1}\sum_{i}^{N}\bar{s}_i^{T}(x_i) \\
&=\left(\bar{S}^{T}\bar{S}\right)^{-1}\bar{S}^{T}X
\end{aligned} ⟺ β ( s x ) = ( i ∑ N s ˉ i T s ˉ i ) − 1 i ∑ N s ˉ i T ( x i ) = ( S ˉ T S ˉ ) − 1 S ˉ T X
∂ Σ x ∣ s ℓ ˉ θ ( O ) = ∑ i N ∫ S q θ , o i ( s i ) ( − Σ x ∣ s − 1 ( x i − s i β ( s x ) ) T ( x i − s i β ( s x ) ) Σ x ∣ s − 1 + Σ x ∣ s − 1 ) d s i = 0 \partial_{\Sigma_{x|s}}\bar{\ell}_{\theta}(O)=\sum_{i}^{N}\int_{S}q_{\theta,o_i}(s_i)\left(-\Sigma_{x|s}^{-1}(x_i-s_i\beta^{(sx)})^{T}(x_i-s_i\beta^{(sx)})\Sigma_{x|s}^{-1}+\Sigma_{x|s}^{-1}\right)\,ds_i=0 ∂ Σ x ∣ s ℓ ˉ θ ( O ) = i ∑ N ∫ S q θ , o i ( s i ) ( − Σ x ∣ s − 1 ( x i − s i β ( s x ) ) T ( x i − s i β ( s x ) ) Σ x ∣ s − 1 + Σ x ∣ s − 1 ) d s i = 0
⟺ Σ x ∣ s = 1 N ∑ i N ∫ S q θ , o i ( s i ) ( x i − s i β ( s x ) ) T ( x i − s i β ( s x ) ) d s i = 1 N ∑ i N ( ( β ( s x ) ) T s ˉ i T s ˉ i β ( s x ) + ( x i − s ˉ i β ( s x ) ) T ( x i − s ˉ i β ( s x ) ) − ( s ˉ i β ( s x ) ) T ( s ˉ i β ( s x ) ) ) = 1 N ( ( β ( s x ) ) T ( S ˉ T S ˉ ) β ( s x ) + ( X − S ˉ β ( s x ) ) T ( X − S ˉ β ( s x ) ) − ( S ˉ β ( s x ) ) T ( S ˉ β ( s x ) ) ) \begin{aligned}
\Longleftrightarrow\ \Sigma_{x|s}=\frac{1}{N}\sum_{i}^{N}\int_{S}q_{\theta,o_i}(s_i)(x_i-s_i\beta^{(sx)})^{T}(x_i-s_i\beta^{(sx)})\,ds_i \\
&=\frac{1}{N}\sum_{i}^{N}\left((\beta^{(sx)})^{T}\bar{s}_i^{T}\bar{s}_i\beta^{(sx)}+(x_i-\bar{s}_i\beta^{(sx)})^{T}(x_i-\bar{s}_i\beta^{(sx)})-(\bar{s}_i\beta^{(sx)})^{T}(\bar{s}_i\beta^{(sx)})\right) \\
&=\frac{1}{N}\left((\beta^{(sx)})^{T}(\bar{S}^{T}\bar{S})\beta^{(sx)}+(X-\bar{S}\beta^{(sx)})^{T}(X-\bar{S}\beta^{(sx)})-(\bar{S}\beta^{(sx)})^{T}(\bar{S}\beta^{(sx)})\right)
\end{aligned} ⟺ Σ x ∣ s = N 1 i ∑ N ∫ S q θ , o i ( s i ) ( x i − s i β ( s x ) ) T ( x i − s i β ( s x ) ) d s i = N 1 i ∑ N ( ( β ( s x ) ) T s ˉ i T s ˉ i β ( s x ) + ( x i − s ˉ i β ( s x ) ) T ( x i − s ˉ i β ( s x ) ) − ( s ˉ i β ( s x ) ) T ( s ˉ i β ( s x ) ) ) = N 1 ( ( β ( s x ) ) T ( S ˉ T S ˉ ) β ( s x ) + ( X − S ˉ β ( s x ) ) T ( X − S ˉ β ( s x ) ) − ( S ˉ β ( s x ) ) T ( S ˉ β ( s x ) ) )
x , s → y x,s \to y x , s → y
For β ( x y ) \beta^{(xy)} β ( x y ) , β ( s y ) \beta^{(sy)} β ( sy ) :
∂ β ( x y ) ℓ ˉ θ ( O ) = ∑ i N ∫ S q θ , o i ( s i ) ( − 2 x i T ( Y i − x i β ( x y ) − s i β ( s y ) ) Σ y ∣ x , s − 1 ) d s i = 0 \partial_{\beta^{(xy)}}\bar{\ell}_{\theta}(O)=\sum_{i}^{N}\int_{S}q_{\theta,o_i}(s_i)\left(-2x_i^{T}(Y_i-x_i\beta^{(xy)}-s_i\beta^{(sy)})\Sigma_{y|x,s}^{-1}\right)\,ds_i=0 ∂ β ( x y ) ℓ ˉ θ ( O ) = i ∑ N ∫ S q θ , o i ( s i ) ( − 2 x i T ( Y i − x i β ( x y ) − s i β ( sy ) ) Σ y ∣ x , s − 1 ) d s i = 0
⟺ β ( x y ) = ( ∑ i N x i T x i ) − 1 ( ∑ i N x i T Y i − x i T s ˉ i β ( s y ) ) = ( X T X ) − 1 X T Y − ( X T X ) − 1 ( X T S ˉ ) β ( s y ) \begin{aligned}
\Longleftrightarrow\ \beta^{(xy)}=\left(\sum_{i}^{N}x_i^{T}x_i\right)^{-1}\left(\sum_{i}^{N}x_i^{T}Y_i-x_i^{T}\bar{s}_i\beta^{(sy)}\right) \\
&=\left(X^{T}X\right)^{-1}X^{T}Y-\left(X^{T}X\right)^{-1}\left(X^{T}\bar{S}\right)\beta^{(sy)}
\end{aligned} ⟺ β ( x y ) = ( i ∑ N x i T x i ) − 1 ( i ∑ N x i T Y i − x i T s ˉ i β ( sy ) ) = ( X T X ) − 1 X T Y − ( X T X ) − 1 ( X T S ˉ ) β ( sy )
∂ β ( s y ) ℓ ˉ θ ( O ) = ∑ i N ∫ S q θ , o i ( s i ) ( − 2 s i T ( Y i − x i β ( x y ) − s i β ( s y ) ) Σ y ∣ x , s − 1 ) d s i = 0 \partial_{\beta^{(sy)}}\bar{\ell}_{\theta}(O)=\sum_{i}^{N}\int_{S}q_{\theta,o_i}(s_i)\left(-2s_i^{T}(Y_i-x_i\beta^{(xy)}-s_i\beta^{(sy)})\Sigma_{y|x,s}^{-1}\right)\,ds_i=0 ∂ β ( sy ) ℓ ˉ θ ( O ) = i ∑ N ∫ S q θ , o i ( s i ) ( − 2 s i T ( Y i − x i β ( x y ) − s i β ( sy ) ) Σ y ∣ x , s − 1 ) d s i = 0
⟺ β ( s y ) = ( ∑ i N s ˉ i T s i ) − 1 ( ∑ i N s ˉ i T Y i − s ˉ i T x i β ( x y ) ) = ( S ˉ T S ˉ ) − 1 S ˉ T Y − ( S ˉ T S ˉ ) − 1 ( S ˉ T X ) β ( x y ) \begin{aligned}
\Longleftrightarrow\ \beta^{(sy)}=\left(\sum_{i}^{N}\bar{s}_i^{T}s_i\right)^{-1}\left(\sum_{i}^{N}\bar{s}_i^{T}Y_i-\bar{s}_i^{T}x_i\beta^{(xy)}\right) \\
&=\left(\bar{S}^{T}\bar{S}\right)^{-1}\bar{S}^{T}Y-\left(\bar{S}^{T}\bar{S}\right)^{-1}\left(\bar{S}^{T}X\right)\beta^{(xy)}
\end{aligned} ⟺ β ( sy ) = ( i ∑ N s ˉ i T s i ) − 1 ( i ∑ N s ˉ i T Y i − s ˉ i T x i β ( x y ) ) = ( S ˉ T S ˉ ) − 1 S ˉ T Y − ( S ˉ T S ˉ ) − 1 ( S ˉ T X ) β ( x y )
Thus:
( X T X X T S ˉ S ˉ T X S ˉ T S ˉ ) ( β ( x y ) β ( s y ) ) = ( X T Y S ˉ T Y ) \begin{pmatrix}
X^{T}X & X^{T}\bar{S}\\
\bar{S}^{T}X & \bar{S}^{T}\bar{S}
\end{pmatrix}
\begin{pmatrix}
\beta^{(xy)}\\
\beta^{(sy)}
\end{pmatrix}
=
\begin{pmatrix}
X^{T}Y\\
\bar{S}^{T}Y
\end{pmatrix} ( X T X S ˉ T X X T S ˉ S ˉ T S ˉ ) ( β ( x y ) β ( sy ) ) = ( X T Y S ˉ T Y )
The solution is Θ = A − 1 r \Theta=A^{-1}r Θ = A − 1 r .
Finally for Σ y ∣ x , s \Sigma_{y|x,s} Σ y ∣ x , s :
∂ Σ y ∣ x , s ℓ ˉ θ ( O ) = ∑ i N ∫ S q θ , o i ( s i ) ( Σ y ∣ x , s − 1 ( Y i − x i β ( x y ) − s i β ( s y ) ) T ( Y i − x i β ( x y ) − s i β ( s y ) ) Σ y ∣ x , s − 1 − Σ y ∣ x , s − 1 ) d s i = 0 \begin{aligned}
\partial_{\Sigma_{y|x,s}}\bar{\ell}_{\theta}(O)
=\sum_{i}^{N}\int_{S}q_{\theta,o_i}(s_i)\left(\Sigma_{y|x,s}^{-1}(Y_i-x_i\beta^{(xy)}-s_i\beta^{(sy)})^{T}(Y_i-x_i\beta^{(xy)}-s_i\beta^{(sy)})\Sigma_{y|x,s}^{-1}-\Sigma_{y|x,s}^{-1}\right)\,ds_i \\
&=0
\end{aligned} ∂ Σ y ∣ x , s ℓ ˉ θ ( O ) = i ∑ N ∫ S q θ , o i ( s i ) ( Σ y ∣ x , s − 1 ( Y i − x i β ( x y ) − s i β ( sy ) ) T ( Y i − x i β ( x y ) − s i β ( sy ) ) Σ y ∣ x , s − 1 − Σ y ∣ x , s − 1 ) d s i = 0
⟺ Σ y ∣ x , s = 1 N ∑ i N ∫ S q θ , o i ( s i ) ( Y i − x i β ( x y ) − s i β ( s y ) ) T ( Y i − x i β ( x y ) − s i β ( s y ) ) d s i = 1 N ( ( Y − X β ( x y ) − S ˉ β ( s y ) ) T ( Y − X β ( x y ) − S ˉ β ( s y ) ) ) \begin{aligned}
\Longleftrightarrow\ \Sigma_{y|x,s}=\frac{1}{N}\sum_{i}^{N}\int_{S}q_{\theta,o_i}(s_i)(Y_i-x_i\beta^{(xy)}-s_i\beta^{(sy)})^{T}(Y_i-x_i\beta^{(xy)}-s_i\beta^{(sy)})\,ds_i \\
&=\frac{1}{N}\left((Y-X\beta^{(xy)}-\bar{S}\beta^{(sy)})^{T}(Y-X\beta^{(xy)}-\bar{S}\beta^{(sy)})\right)
\end{aligned} ⟺ Σ y ∣ x , s = N 1 i ∑ N ∫ S q θ , o i ( s i ) ( Y i − x i β ( x y ) − s i β ( sy ) ) T ( Y i − x i β ( x y ) − s i β ( sy ) ) d s i = N 1 ( ( Y − X β ( x y ) − S ˉ β ( sy ) ) T ( Y − X β ( x y ) − S ˉ β ( sy ) ) )
− 1 N ( ( S ˉ β ( s y ) ) T ( S ˉ β ( s y ) ) + ( β ( s y ) ) T S ˉ T S ˉ β ( s y ) ) -\frac{1}{N}\left((\bar{S}\beta^{(sy)})^{T}(\bar{S}\beta^{(sy)})+(\beta^{(sy)})^{T}\bar{S}^{T}\bar{S}\beta^{(sy)}\right) − N 1 ( ( S ˉ β ( sy ) ) T ( S ˉ β ( sy ) ) + ( β ( sy ) ) T S ˉ T S ˉ β ( sy ) )
Model’s features and remarks
Rotational invariance
Latent mediation models exhibit rotational invariance: for any invertible linear transformation Q ∈ R K × K Q \in\mathbb{R}^{K \times K} Q ∈ R K × K , the sets ( S , β ( x s ) , β ( s m ) , β ( s y ) , Σ s ∣ x ) (S, \beta^{(xs)}, \beta^{(sm)}, \beta^{(sy)}, \Sigma_{s|x}) ( S , β ( x s ) , β ( s m ) , β ( sy ) , Σ s ∣ x ) and ( S Q , β ( x s ) Q , Q − 1 β ( s m ) , Q − 1 β ( s y ) , Q T Σ s ∣ x Q ) (SQ, \beta^{(xs)}Q, Q^{-1}\beta^{(sm)}, Q^{-1}\beta^{(sy)}, Q^{T}\Sigma_{s|x}Q) ( S Q , β ( x s ) Q , Q − 1 β ( s m ) , Q − 1 β ( sy ) , Q T Σ s ∣ x Q ) induce exactly the same likelihood. Although ordering conventions can fix a canonical rotation post-hoc (e.g. in PCA), the above rotational invariance remains. Note that, despite the name, rotational invariance is not restricted to orthogonal rotations; any invertible linear transform Q Q Q used in the transformations above leaves the likelihood unchanged.
This implies that latent components (and their corresponding parameters) are not uniquely defined. Importantly, once the noise-signal invariance has been fixed by choosing a canonical representation, this non-identifiability is confined to the latent representation itself. The induced mappings between observed variables, such as the path coefficients β ( x m ) = β ( x s ) β ( s m ) \beta^{(xm)} = \beta^{(xs)}\beta^{(sm)} β ( x m ) = β ( x s ) β ( s m ) , remain identifiable. Consequently, the marginalized mediation model
p ( X , M , Y ) = ∫ S p ( S , X , M , Y ) d S p(X, M, Y) = \int_{S} p(S, X, M, Y)\,\mathrm{d}S p ( X , M , Y ) = ∫ S p ( S , X , M , Y ) d S
remains identifiable.
Regularized versions
Regularization can be introduced by changing the prior p ( θ ) p(\theta) p ( θ ) on the parameters in the log-likelihood, which amounts to adding penalty terms. Here we present ℓ 1 \ell_1 ℓ 1 and ℓ 2 \ell_2 ℓ 2 regularizations, which both modify the definition of C C C in the E-step as well as the optimization problem solved in the M-step. We describe below how such penalties can be applied to any coefficient matrix β ( u v ) ∈ { β ( x s ) , β ( s m ) , β ( s y ) , β ( x y ) } \beta^{(uv)} \in\{\beta^{(xs)}, \beta^{(sm)}, \beta^{(sy)}, \beta^{(xy)}\} β ( uv ) ∈ { β ( x s ) , β ( s m ) , β ( sy ) , β ( x y ) } :
ℓ 2 \ell_2 ℓ 2 (Ridge ) regularization corresponds to placing a zero-mean Gaussian prior with covariance Σ v ∣ u \Sigma_{v|u} Σ v ∣ u on β ( u v ) \beta^{(uv)} β ( uv ) , which amounts to setting p ( β ( u v ) ) ∝ e − λ 2 2 ⟨ ( β ( u v ) ) T β ( u v ) , Σ v ∣ u − 1 ⟩ p(\beta^{(uv)}) \propto e^{-\frac{\lambda_2}{2}\langle(\beta^{(uv)})^T\beta^{(uv)},\Sigma_{v|u}^{-1}\rangle} p ( β ( uv ) ) ∝ e − 2 λ 2 ⟨( β ( uv ) ) T β ( uv ) , Σ v ∣ u − 1 ⟩ . As a result, the term C C C in the log-likelihood becomes C r e g = C − λ 2 2 ⟨ ( β ( u v ) ) T β ( u v ) , Σ v ∣ u − 1 ⟩ C_{\mathrm{reg}} = C - \frac{\lambda_2}{2}\langle(\beta^{(uv)})^T\beta^{(uv)},\Sigma_{v|u}^{-1}\rangle C reg = C − 2 λ 2 ⟨( β ( uv ) ) T β ( uv ) , Σ v ∣ u − 1 ⟩ , and the update equations become β r e g ( u v ) = ( U T U + λ 2 I ) − 1 U T V \beta_{\mathrm{reg}}^{(uv)} = (U^T U + \lambda_2 I)^{-1}U^T V β reg ( uv ) = ( U T U + λ 2 I ) − 1 U T V with corresponding updates of the noise covariance matrices. Importantly, the M-step remains available in closed form under ℓ 2 \ell_2 ℓ 2 regularization. Note that this is equivalent to simply adding a penalty term − λ 2 2 ⟨ ( β ( u v ) ) T β ( u v ) , Σ v ∣ u − 1 ⟩ -\frac{\lambda_2}{2}\langle(\beta^{(uv)})^T\beta^{(uv)},\Sigma_{v|u}^{-1}\rangle − 2 λ 2 ⟨( β ( uv ) ) T β ( uv ) , Σ v ∣ u − 1 ⟩ to the log-likelihood.
ℓ 1 \ell_1 ℓ 1 (LASSO ) regularization corresponds to placing a zero-mean Laplace prior on β ( u v ) \beta^{(uv)} β ( uv ) , which amounts to setting p ( β ( u v ) ) ∝ e − λ 1 ∑ i j ∣ β i j ( u v ) ∣ p(\beta^{(uv)}) \propto e^{-\lambda_1 \sum_{ij}|\beta_{ij}^{(uv)}|} p ( β ( uv ) ) ∝ e − λ 1 ∑ ij ∣ β ij ( uv ) ∣ . As a result, the term C C C in the log-likelihood becomes C r e g = C − λ 1 ∑ i j ∣ β i j ( u v ) ∣ C_{\mathrm{reg}} = C - \lambda_1 \sum_{ij}|\beta_{ij}^{(uv)}| C reg = C − λ 1 ∑ ij ∣ β ij ( uv ) ∣ . In contrast to the ℓ 2 \ell_2 ℓ 2 case, the update equations have no analytical solution and must be solved numerically. Notably, ℓ 1 \ell_1 ℓ 1 regularization can partially break rotational invariance by favouring sparse orientations of the latent space. Note that this is equivalent to simply adding a penalty term − λ 1 ∑ i j ∣ β i j ( u v ) ∣ -\lambda_1 \sum_{ij}|\beta_{ij}^{(uv)}| − λ 1 ∑ ij ∣ β ij ( uv ) ∣ to the log-likelihood.
Log-normally distributed variables
Real-world data are not always well approximated by Gaussian distributions. When outcomes are strictly positive and exhibit right skewness, as in survival analysis for example, the log-normal distribution can be a more appropriate modelling choice. The LaMe framework naturally accommodates log-normally distributed variables through a simple reparametrization. Let Z ∈ { X , M , Y } Z \in\{X, M, Y\} Z ∈ { X , M , Y } denote a strictly positive variable assumed to follow a log-normal distribution. The associated transformations are then Z → log Z Z \to\log Z Z → log Z throughout the model, and log p ( X , M , Y ∣ θ ) → log p ( X , M , Y ∣ θ ) − ∑ i , j log Z i j \log p(X, M, Y|\theta) \to\log p(X, M, Y|\theta) - \sum_{i,j}\log Z_{ij} log p ( X , M , Y ∣ θ ) → log p ( X , M , Y ∣ θ ) − ∑ i , j log Z ij , where the summation runs over all the elements of Z Z Z .
Likelihood-equivalent reparametrizations
Starting from the observed moments parametrization from § 9 , we allow for arbitrary reparametrizations of the parameters related to S S S , namely ( β ( x s ) , β ( s m ) , β ( s y ) , Σ s ∣ x ) (\beta^{(xs)}, \beta^{(sm)}, \beta^{(sy)}, \Sigma_{s|x}) ( β ( x s ) , β ( s m ) , β ( sy ) , Σ s ∣ x ) , noted ( β ∗ ( x s ) , β ∗ ( s m ) , β ∗ ( s y ) , Σ s ∣ x ∗ ) (\beta_*^{(xs)}, \beta_*^{(sm)}, \beta_*^{(sy)}, \Sigma^*_{s|x}) ( β ∗ ( x s ) , β ∗ ( s m ) , β ∗ ( sy ) , Σ s ∣ x ∗ ) . These transformed parameters describe every possible reparametrization, including translations and changes of rank. Remaining parameters β ( x y ) \beta^{(xy)} β ( x y ) , Σ x \Sigma_x Σ x , Σ m ∣ s \Sigma_{m|s} Σ m ∣ s and Σ y ∣ s , x \Sigma_{y|s,x} Σ y ∣ s , x must be transformed accordingly to preserve the observed moments.
Now, we assume that the bottleneck assumption holds (see § 9 ), meaning:
rank ( C ) = rank ( ( Σ X Y − 1 Σ X M Σ M Y T − Σ X Y T Σ X Y − 1 Σ X M ) ) ≥ rank ( ( β ( x s ) β ( s m ) ( β ( s y ) ) T Σ s ∣ x β ( s m ) ) ) = K (32) \operatorname{rank}(C) = \operatorname{rank}\left(\begin{pmatrix} \Sigma_{XY}^{-1}\Sigma_{XM} \\ \Sigma_{MY}^{T} - \Sigma_{XY}^{T}\Sigma_{XY}^{-1}\Sigma_{XM} \end{pmatrix}\right) \ge\operatorname{rank}\left(\begin{pmatrix} \beta^{(xs)}\beta^{(sm)} \\ (\beta^{(sy)})^{T}\Sigma_{s|x}\beta^{(sm)} \end{pmatrix}\right) = K \tag*{(32)} rank ( C ) = rank ( ( Σ X Y − 1 Σ X M Σ M Y T − Σ X Y T Σ X Y − 1 Σ X M ) ) ≥ rank ( ( β ( x s ) β ( s m ) ( β ( sy ) ) T Σ s ∣ x β ( s m ) ) ) = K (32)
This constraint means that all latent dimensions K K K are used to approximate the observed moments in C C C . Therefore, both factors ( β ( x s ) , ( β ( s y ) ) T Σ s ∣ x ) T (\beta^{(xs)},(\beta^{(sy)})^{T}\Sigma_{s|x})^{T} ( β ( x s ) , ( β ( sy ) ) T Σ s ∣ x ) T and β ( s m ) \beta^{(sm)} β ( s m ) above must have rank K K K . Any other factorization of the same C C C through K K K dimensions must consequently differ only by an invertible transformation.
In particular, β ∗ ( s m ) \beta_*^{(sm)} β ∗ ( s m ) must also have rank K K K from the condition above. It follows that ( β ∗ ( s m ) ( β ∗ ( s m ) ) T ) (\beta_*^{(sm)}(\beta_*^{(sm)})^{T}) ( β ∗ ( s m ) ( β ∗ ( s m ) ) T ) is invertible, meaning R : = ( β ∗ ( s m ) ) T ( β ∗ ( s m ) ( β ∗ ( s m ) ) T ) − 1 R := (\beta_*^{(sm)})^{T}(\beta_*^{(sm)}(\beta_*^{(sm)})^{T})^{-1} R := ( β ∗ ( s m ) ) T ( β ∗ ( s m ) ( β ∗ ( s m ) ) T ) − 1 exists and satisfies β ∗ ( s m ) R = I K \beta_*^{(sm)}R = I_K β ∗ ( s m ) R = I K . Therefore, right-multiplying the parametric approximation of C C C by R R R and reminding that likelihood equivalence implies that this representation is preserved by equivalent transformations, then:
( β ( x s ) ( β ( s y ) ) T Σ s ∣ x ) β ( s m ) R = ( β ∗ ( x s ) ( β ∗ ( s y ) ) T Σ s ∣ x ∗ ) β ∗ ( s m ) R = ( β ∗ ( x s ) ( β ∗ ( s y ) ) T Σ s ∣ x ∗ ) \begin{pmatrix}
\beta^{(xs)} \\
(\beta^{(sy)})^{T}\Sigma_{s|x}
\end{pmatrix}
\beta^{(sm)}R
=
\begin{pmatrix}
\beta_*^{(xs)} \\
(\beta_*^{(sy)})^{T}\Sigma^*_{s|x}
\end{pmatrix}
\beta_*^{(sm)}R
=
\begin{pmatrix}
\beta_*^{(xs)} \\
(\beta_*^{(sy)})^{T}\Sigma^*_{s|x}
\end{pmatrix} ( β ( x s ) ( β ( sy ) ) T Σ s ∣ x ) β ( s m ) R = ( β ∗ ( x s ) ( β ∗ ( sy ) ) T Σ s ∣ x ∗ ) β ∗ ( s m ) R = ( β ∗ ( x s ) ( β ∗ ( sy ) ) T Σ s ∣ x ∗ )
Defining H M = β ( s m ) R H_M = \beta^{(sm)}R H M = β ( s m ) R , which must therefore be an invertible K × K K \times K K × K matrix to preserve the rank K K K of the transformed expression of C C C , it follows parametrizations preserving observed moments when the bottleneck condition is satisfied are:
β ∗ ( x s ) = β ( x s ) H M \beta_*^{(xs)} = \beta^{(xs)}H_M β ∗ ( x s ) = β ( x s ) H M
β ∗ ( s m ) = H M − 1 β ( s m ) \beta_*^{(sm)} = H_M^{-1}\beta^{(sm)} β ∗ ( s m ) = H M − 1 β ( s m )
( β ∗ ( s y ) T Σ s ∣ x ∗ = ( β ( s y ) T Σ s ∣ x H M (\beta_*^{(sy)T}\Sigma^*_{s|x} = (\beta^{(sy)T}\Sigma_{s|x}H_M ( β ∗ ( sy ) T Σ s ∣ x ∗ = ( β ( sy ) T Σ s ∣ x H M
The last equation preserved the covariance between M M M and Y Y Y given X X X , and defines a condition up to an invertible linear transformation K × K K \times K K × K matrix noted H Y H_Y H Y :
Σ s ∣ x ∗ β ∗ ( s y ) = H M T Σ s ∣ x β ( s y ) \Sigma^*_{s|x}\beta_*^{(sy)} = H_M^T\Sigma_{s|x}\beta^{(sy)} Σ s ∣ x ∗ β ∗ ( sy ) = H M T Σ s ∣ x β ( sy )
Since Σ s ∣ x ∗ \Sigma^*_{s|x} Σ s ∣ x ∗ must remain positive-definite, we can invert it, which yields the transformation of β ( s y ) \beta^{(sy)} β ( sy ) implied by the conservation of observed moments parametrization:
β ∗ ( s y ) = ( Σ s ∣ x ∗ ) − 1 H M T Σ s ∣ x β ( s y ) \beta_*^{(sy)} = (\Sigma^*_{s|x})^{-1}H_M^T\Sigma_{s|x}\beta^{(sy)} β ∗ ( sy ) = ( Σ s ∣ x ∗ ) − 1 H M T Σ s ∣ x β ( sy )
Noting that covariances are positive-definite and H M H_M H M is invertible, H Y H_Y H Y is also guaranteed to be invertible. Finally, from the definition of H Y H_Y H Y :
Σ s ∣ x ∗ = H M T Σ s ∣ x H Y \Sigma^*_{s|x} = H_M^T\Sigma_{s|x}H_Y Σ s ∣ x ∗ = H M T Σ s ∣ x H Y
Remaining expressions for the transformed parameters follows as in the main text.
Supporting derivations for identifiability
Transformed covariances remain positive semi-definite
We recall:
H Y H M − 1 = Σ s ∣ x − 1 ( ( β ( s m ) Σ m − 1 ( β ( s m ) ) T ) − 1 − ( β ( x s ) T Σ x β ( x s ) ) ) H_YH_M^{-1} = \Sigma^{-1}_{s|x}\left((\beta^{(sm)}\Sigma_m^{-1}(\beta^{(sm)})^T)^{-1} - (\beta^{(xs)T}\Sigma_x\beta^{(xs)})\right) H Y H M − 1 = Σ s ∣ x − 1 ( ( β ( s m ) Σ m − 1 ( β ( s m ) ) T ) − 1 − ( β ( x s ) T Σ x β ( x s ) ) )
Σ s ∣ x ∗ = H M T Σ s ∣ x H Y \Sigma^*_{s|x} = H_M^T\Sigma_{s|x}H_Y Σ s ∣ x ∗ = H M T Σ s ∣ x H Y
Σ m ∣ s ∗ = Σ m ∣ s + ( β ( s m ) T Σ s ∣ x β ( s m ) − ( β ( s m ) T Σ s ∣ x ∗ β ∗ ( s m ) \Sigma^*_{m|s} = \Sigma_{m|s} + (\beta^{(sm)T}\Sigma_{s|x}\beta^{(sm)} - (\beta^{(sm)T}\Sigma^*_{s|x}\beta_*^{(sm)} Σ m ∣ s ∗ = Σ m ∣ s + ( β ( s m ) T Σ s ∣ x β ( s m ) − ( β ( s m ) T Σ s ∣ x ∗ β ∗ ( s m )
Σ y ∣ x , s ∗ = Σ y ∣ x , s + ( β ( s y ) T Σ s ∣ x β ( s y ) − ( β ∗ ( s y ) T Σ s ∣ x ∗ β ∗ ( s y ) \Sigma^*_{y|x,s} = \Sigma_{y|x,s} + (\beta^{(sy)T}\Sigma_{s|x}\beta^{(sy)} - (\beta_*^{(sy)T}\Sigma^*_{s|x}\beta_*^{(sy)} Σ y ∣ x , s ∗ = Σ y ∣ x , s + ( β ( sy ) T Σ s ∣ x β ( sy ) − ( β ∗ ( sy ) T Σ s ∣ x ∗ β ∗ ( sy )
The equations above guarantee positive semi-definite conditional covariances after transformation. Note that in general, A T G A ⪰ 0 A^TGA \succeq0 A T G A ⪰ 0 when G ⪰ 0 G \succeq0 G ⪰ 0 for any transformation of A A A .
For convenience of notation, note that the noise covariance of M M M given X X X is Σ m ∣ x = Σ m − ( β ( x s ) β ( s m ) T Σ x β ( x s ) β ( s m ) \Sigma_{m|x} = \Sigma_m - (\beta^{(xs)}\beta^{(sm)T}\Sigma_x\beta^{(xs)}\beta^{(sm)} Σ m ∣ x = Σ m − ( β ( x s ) β ( s m ) T Σ x β ( x s ) β ( s m ) . Therefore:
H Y H M − 1 = Σ s ∣ x − 1 ( ( β ( s m ) Σ m − 1 ( β ( s m ) ) T ) − 1 − ( β ( x s ) T Σ x β ( x s ) ) = Σ s ∣ x − 1 ( β ( s m ) Σ m ∣ x − 1 ( β ( s m ) T ) − 1 \begin{aligned}
H_YH_M^{-1} = \Sigma^{-1}_{s|x}\left((\beta^{(sm)}\Sigma_m^{-1}(\beta^{(sm)})^T)^{-1} - (\beta^{(xs)T}\Sigma_x\beta^{(xs)}\right) \\
&= \Sigma^{-1}_{s|x}(\beta^{(sm)}\Sigma^{-1}_{m|x}(\beta^{(sm)T})^{-1}
\end{aligned} H Y H M − 1 = Σ s ∣ x − 1 ( ( β ( s m ) Σ m − 1 ( β ( s m ) ) T ) − 1 − ( β ( x s ) T Σ x β ( x s ) ) = Σ s ∣ x − 1 ( β ( s m ) Σ m ∣ x − 1 ( β ( s m ) T ) − 1
H M H Y − 1 = β ( s m ) Σ m ∣ x − 1 ( β ( s m ) ) T Σ s ∣ x H_MH_Y^{-1} = \beta^{(sm)}\Sigma^{-1}_{m|x}(\beta^{(sm)})^T\Sigma_{s|x} H M H Y − 1 = β ( s m ) Σ m ∣ x − 1 ( β ( s m ) ) T Σ s ∣ x
The result is immediate since Σ s ∣ x ∗ \Sigma^*_{s|x} Σ s ∣ x ∗ since Σ m ∣ x ⪰ 0 \Sigma_{m|x} \succeq0 Σ m ∣ x ⪰ 0 :
Σ s ∣ x ∗ = H M T ( β ( s m ) Σ m ∣ x − 1 ( β ( s m ) ) T ) − 1 H M \Sigma^*_{s|x} = H_M^T(\beta^{(sm)}\Sigma^{-1}_{m|x}(\beta^{(sm)})^T)^{-1}H_M Σ s ∣ x ∗ = H M T ( β ( s m ) Σ m ∣ x − 1 ( β ( s m ) ) T ) − 1 H M
Developing Σ m ∣ s ∗ \Sigma^*_{m|s} Σ m ∣ s ∗ , we can show it also is a form A T G A ⪰ 0 A^TGA \succeq0 A T G A ⪰ 0 :
Σ m ∣ s ∗ = Σ m ∣ x − ( β ( s m ) ) T ( ( β ( s m ) Σ m ∣ x − 1 ( β ( s m ) ) T ) − 1 ) β ( s m ) ⪰ 0 = ( I − ( β ( s m ) ) T ( ( β ( s m ) Σ m ∣ x − 1 ( β ( s m ) ) T ) − 1 ) β ( s m ) Σ m ∣ x − 1 ) × Σ m ∣ x × ( I − ( β ( s m ) ) T ( ( β ( s m ) Σ m ∣ x − 1 ( β ( s m ) ) T ) − 1 ) β ( s m ) Σ m ∣ x − 1 ) T \begin{aligned}
\Sigma^*_{m|s} = \Sigma_{m|x} - (\beta^{(sm)})^T((\beta^{(sm)}\Sigma^{-1}_{m|x}(\beta^{(sm)})^T)^{-1})\beta^{(sm)} \succeq0 \\
&= (I - (\beta^{(sm)})^T((\beta^{(sm)}\Sigma^{-1}_{m|x}(\beta^{(sm)})^T)^{-1})\beta^{(sm)}\Sigma^{-1}_{m|x}) \\
&\times\Sigma_{m|x} \\
&\times(I - (\beta^{(sm)})^T((\beta^{(sm)}\Sigma^{-1}_{m|x}(\beta^{(sm)})^T)^{-1})\beta^{(sm)}\Sigma^{-1}_{m|x})^T
\end{aligned} Σ m ∣ s ∗ = Σ m ∣ x − ( β ( s m ) ) T (( β ( s m ) Σ m ∣ x − 1 ( β ( s m ) ) T ) − 1 ) β ( s m ) ⪰ 0 = ( I − ( β ( s m ) ) T (( β ( s m ) Σ m ∣ x − 1 ( β ( s m ) ) T ) − 1 ) β ( s m ) Σ m ∣ x − 1 ) × Σ m ∣ x × ( I − ( β ( s m ) ) T (( β ( s m ) Σ m ∣ x − 1 ( β ( s m ) ) T ) − 1 ) β ( s m ) Σ m ∣ x − 1 ) T
Finally, let ξ : = ( Σ s ∣ x − 1 − β ( s m ) Σ m ∣ x − 1 ( β ( s m ) ) T ) \xi:= (\Sigma^{-1}_{s|x} - \beta^{(sm)}\Sigma^{-1}_{m|x}(\beta^{(sm)})^{T}) ξ := ( Σ s ∣ x − 1 − β ( s m ) Σ m ∣ x − 1 ( β ( s m ) ) T ) and note that:
ξ = ξ Σ s ∣ x ξ T + β ( s m ) Σ m ∣ x − 1 Σ m ∣ s Σ s ∣ m − 1 ( β ( s m ) ) T ⇔ ξ ⪰ 0 \xi= \xi\Sigma_{s|x}\xi^{T} + \beta^{(sm)}\Sigma^{-1}_{m|x}\Sigma_{m|s}\Sigma^{-1}_{s|m}(\beta^{(sm)})^{T}
\Leftrightarrow\xi\succeq0 ξ = ξ Σ s ∣ x ξ T + β ( s m ) Σ m ∣ x − 1 Σ m ∣ s Σ s ∣ m − 1 ( β ( s m ) ) T ⇔ ξ ⪰ 0
Then positive semi-definiteness of Σ y ∣ x , s ∗ \Sigma^{*}_{y|x,s} Σ y ∣ x , s ∗ immediately follows:
Σ y ∣ x , s ∗ = Σ y ∣ x , s + ( β ( s y ) ) T Σ s ∣ x ( Σ s ∣ x − 1 − β ( s m ) Σ m ∣ x − 1 ( β ( s m ) ) T ) Σ s ∣ x β ( s y ) = Σ y ∣ x , s + ( β ( s y ) ) T Σ s ∣ x ξ Σ s ∣ x β ( s y ) \begin{aligned}
\Sigma^{*}_{y|x,s} = \Sigma_{y|x,s} + (\beta^{(sy)})^{T}\Sigma_{s|x}(\Sigma^{-1}_{s|x} - \beta^{(sm)}\Sigma^{-1}_{m|x}(\beta^{(sm)})^{T})\Sigma_{s|x}\beta^{(sy)} \\
&= \Sigma_{y|x,s} + (\beta^{(sy)})^{T}\Sigma_{s|x}\xi\Sigma_{s|x}\beta^{(sy)}
\end{aligned} Σ y ∣ x , s ∗ = Σ y ∣ x , s + ( β ( sy ) ) T Σ s ∣ x ( Σ s ∣ x − 1 − β ( s m ) Σ m ∣ x − 1 ( β ( s m ) ) T ) Σ s ∣ x β ( sy ) = Σ y ∣ x , s + ( β ( sy ) ) T Σ s ∣ x ξ Σ s ∣ x β ( sy )
Invariance of parametric covariance
Σ m ∗ = Σ m ∣ s ∗ + ( β ( s m ) ) T Σ s ∗ β ( s m ) = Σ m ∣ s + ( β ( s m ) ) T Σ s ∣ x β ( s m ) − ( β ( s m ) ) T Σ s ∣ x ∗ β ( s m ) + ( β ( s m ) ) T ( Σ s ∣ x ∗ + ( β ( x s ) ) T Σ x ∗ β ( x s ) ) β ( s m ) = Σ m ∣ s + ( β ( s m ) ) T Σ s ∣ x β ( s m ) + ( β ( x s ) β ( s m ) ) T Σ x ∗ β ( x s ) β ( s m ) = Σ m ∣ s + ( β ( s m ) ) T Σ s ∣ x β ( s m ) + ( β ( x s ) H M H − 1 β ( s m ) ) T Σ x ∗ β ( x s ) H M H − 1 β ( s m ) = Σ m ∣ s + ( β ( s m ) ) T ( Σ s ∣ x + ( β ( x s ) ) T Σ x ∗ β ( x s ) ) β ( s m ) = Σ m ∣ s + ( β ( s m ) ) T ( Σ s ∣ x + ( β ( x s ) ) T Σ x β ( x s ) ) β ( s m ) = Σ m ∣ s + ( β ( s m ) ) T Σ s β ( s m ) = Σ m \begin{aligned}
\Sigma^{*}_{m} &= \Sigma^{*}_{m|s} + (\beta^{(sm)})^{T}\Sigma^{*}_{s}\beta^{(sm)} \\
&= \Sigma_{m|s} + (\beta^{(sm)})^{T}\Sigma_{s|x}\beta^{(sm)} - (\beta^{(sm)})^{T}\Sigma^{*}_{s|x}\beta^{(sm)} + (\beta^{(sm)})^{T}(\Sigma^{*}_{s|x} + (\beta^{(xs)})^{T}\Sigma^{*}_{x}\beta^{(xs)})\beta^{(sm)} \\
&= \Sigma_{m|s} + (\beta^{(sm)})^{T}\Sigma_{s|x}\beta^{(sm)} + (\beta^{(xs)}\beta^{(sm)})^{T}\Sigma^{*}_{x}\beta^{(xs)}\beta^{(sm)} \\
&= \Sigma_{m|s} + (\beta^{(sm)})^{T}\Sigma_{s|x}\beta^{(sm)} + (\beta^{(xs)}HMH^{-1}\beta^{(sm)})^{T}\Sigma^{*}_{x}\beta^{(xs)}HMH^{-1}\beta^{(sm)} \\
&= \Sigma_{m|s} + (\beta^{(sm)})^{T}(\Sigma_{s|x} + (\beta^{(xs)})^{T}\Sigma^{*}_{x}\beta^{(xs)})\beta^{(sm)} \\
&= \Sigma_{m|s} + (\beta^{(sm)})^{T}(\Sigma_{s|x} + (\beta^{(xs)})^{T}\Sigma_{x}\beta^{(xs)})\beta^{(sm)} \\
&= \Sigma_{m|s} + (\beta^{(sm)})^{T}\Sigma_{s}\beta^{(sm)} \\
&= \Sigma_{m}
\end{aligned} Σ m ∗ = Σ m ∣ s ∗ + ( β ( s m ) ) T Σ s ∗ β ( s m ) = Σ m ∣ s + ( β ( s m ) ) T Σ s ∣ x β ( s m ) − ( β ( s m ) ) T Σ s ∣ x ∗ β ( s m ) + ( β ( s m ) ) T ( Σ s ∣ x ∗ + ( β ( x s ) ) T Σ x ∗ β ( x s ) ) β ( s m ) = Σ m ∣ s + ( β ( s m ) ) T Σ s ∣ x β ( s m ) + ( β ( x s ) β ( s m ) ) T Σ x ∗ β ( x s ) β ( s m ) = Σ m ∣ s + ( β ( s m ) ) T Σ s ∣ x β ( s m ) + ( β ( x s ) H M H − 1 β ( s m ) ) T Σ x ∗ β ( x s ) H M H − 1 β ( s m ) = Σ m ∣ s + ( β ( s m ) ) T ( Σ s ∣ x + ( β ( x s ) ) T Σ x ∗ β ( x s ) ) β ( s m ) = Σ m ∣ s + ( β ( s m ) ) T ( Σ s ∣ x + ( β ( x s ) ) T Σ x β ( x s ) ) β ( s m ) = Σ m ∣ s + ( β ( s m ) ) T Σ s β ( s m ) = Σ m
Relation under exact recoverability
Σ m ∣ s Σ m − 1 ( β ( s m ) ) T Σ s = 0 ⇔ ( Σ m − ( β ( s m ) ) T Σ s β ( s m ) ) Σ m − 1 ( β ( s m ) ) T Σ s = 0 ⇔ ( β ( s m ) ) T Σ s − ( β ( s m ) ) T Σ s β ( s m ) Σ m − 1 ( β ( s m ) ) T Σ s = 0 ⇔ ( β ( s m ) ) T Σ s ( I − β ( s m ) Σ m − 1 ( β ( s m ) ) T Σ s ) = 0 ⇔ β ( s m ) Σ m − 1 ( β ( s m ) ) T Σ s = I \begin{aligned}
\Sigma_{m|s}\Sigma^{-1}_{m}(\beta^{(sm)})^{T}\Sigma_{s} = 0 \\
&\Leftrightarrow(\Sigma_{m} - (\beta^{(sm)})^{T}\Sigma_{s}\beta^{(sm)})\Sigma^{-1}_{m}(\beta^{(sm)})^{T}\Sigma_{s} = 0 \\
&\Leftrightarrow(\beta^{(sm)})^{T}\Sigma_{s} - (\beta^{(sm)})^{T}\Sigma_{s}\beta^{(sm)}\Sigma^{-1}_{m}(\beta^{(sm)})^{T}\Sigma_{s} = 0 \\
&\Leftrightarrow(\beta^{(sm)})^{T}\Sigma_{s}(I - \beta^{(sm)}\Sigma^{-1}_{m}(\beta^{(sm)})^{T}\Sigma_{s}) = 0 \\
&\Leftrightarrow\beta^{(sm)}\Sigma^{-1}_{m}(\beta^{(sm)})^{T}\Sigma_{s} = I
\end{aligned} Σ m ∣ s Σ m − 1 ( β ( s m ) ) T Σ s = 0 ⇔ ( Σ m − ( β ( s m ) ) T Σ s β ( s m ) ) Σ m − 1 ( β ( s m ) ) T Σ s = 0 ⇔ ( β ( s m ) ) T Σ s − ( β ( s m ) ) T Σ s β ( s m ) Σ m − 1 ( β ( s m ) ) T Σ s = 0 ⇔ ( β ( s m ) ) T Σ s ( I − β ( s m ) Σ m − 1 ( β ( s m ) ) T Σ s ) = 0 ⇔ β ( s m ) Σ m − 1 ( β ( s m ) ) T Σ s = I
Inverting the model’s parameters
Parameter estimates for any factorization of the joint linear Gaussian distribution p ( X , S , M , Y ) p(X, S, M, Y) p ( X , S , M , Y ) can be recovered from the corresponding covariance representation. For instance, if the optimal parameter set θ v M \theta_{vM} θ v M is obtained from the LaMe-vM factorization, we can easily derive the equivalent parameter set θ v X = f ( θ v M ) \theta_{vX} = f(\theta_{vM}) θ v X = f ( θ v M ) used in the LaMe-vX factorization. For illustration, consider the expectation of X X X under the LaMe-vM factorization E [ X ∣ M ] = M β ( m s ) β ( s x ) : = M β ( m x ) \mathbb{E}[X|M] = M\beta^{(ms)}\beta^{(sx)} := M\beta^{(mx)} E [ X ∣ M ] = M β ( m s ) β ( s x ) := M β ( m x ) , which has no causal interpretation. Instead, under the LaMe-vX factorization, E [ M ∣ X ] = X β ( x m ) \mathbb{E}[M|X] = X\beta^{(xm)} E [ M ∣ X ] = X β ( x m ) depends on the causal path coefficient β ( x m ) \beta^{(xm)} β ( x m ) , and can be recovered as follows [3 ] :
β ( x m ) = Σ x x − 1 Σ x m = Σ x x − 1 Σ m x T = ( ( β ( s x ) ) T ( β ( m s ) ) T Σ m β ( m s ) β ( s x ) + ( β ( s x ) ) T Σ s ∣ m β ( s x ) + Σ x ∣ s ) − 1 × ( Σ m β ( m s ) β ( s x ) ) T (33) \begin{aligned}
\beta^{(xm)} &= \Sigma^{-1}_{xx}\Sigma_{xm} \\
&= \Sigma^{-1}_{xx}\Sigma^{T}_{mx} \\
&= \left((\beta^{(sx)})^{T}(\beta^{(ms)})^{T}\Sigma_{m}\beta^{(ms)}\beta^{(sx)} + (\beta^{(sx)})^{T}\Sigma_{s|m}\beta^{(sx)} + \Sigma_{x|s}\right)^{-1} \\
&\qquad\times\left(\Sigma_{m}\beta^{(ms)}\beta^{(sx)}\right)^{T} \tag*{(33)}
\end{aligned} β ( x m ) = Σ xx − 1 Σ x m = Σ xx − 1 Σ m x T = ( ( β ( s x ) ) T ( β ( m s ) ) T Σ m β ( m s ) β ( s x ) + ( β ( s x ) ) T Σ s ∣ m β ( s x ) + Σ x ∣ s ) − 1 × ( Σ m β ( m s ) β ( s x ) ) T (33)
For any generic unconditioned linear coefficient β ( a b ) \beta^{(ab)} β ( ab ) , the following relation holds:
β ( a b ∣ a ) = Σ a a − 1 Σ a b (34) \beta^{(ab|a)} = \Sigma^{-1}_{aa}\Sigma_{ab} \tag*{(34)} β ( ab ∣ a ) = Σ aa − 1 Σ ab (34)
When conditioning on additional variables, the coefficient reads:
β ( a b ∣ a , c ) = ( Σ a a − Σ a c Σ c c − 1 Σ c a ) − 1 ( Σ a b − Σ a c Σ c c − 1 Σ c b ) (35) \beta^{(ab|a,c)} = (\Sigma_{aa} - \Sigma_{ac}\Sigma^{-1}_{cc}\Sigma_{ca})^{-1}(\Sigma_{ab} - \Sigma_{ac}\Sigma^{-1}_{cc}\Sigma_{cb}) \tag*{(35)} β ( ab ∣ a , c ) = ( Σ aa − Σ a c Σ cc − 1 Σ c a ) − 1 ( Σ ab − Σ a c Σ cc − 1 Σ c b ) (35)
Finally, we recall below all covariances and cross-covariances needed to compute the regression parameters in various models and factorizations. Note that in the main text, we defined the shorthand notation β ( m y ) : = β ( m y ∣ m , x ) = ( Σ m m − Σ m x Σ x x − 1 Σ x m ) − 1 ( Σ m y − Σ m x Σ x x − 1 Σ x y ) \beta^{(my)} := \beta^{(my|m,x)} = (\Sigma_{mm} - \Sigma_{mx}\Sigma^{-1}_{xx}\Sigma_{xm})^{-1}(\Sigma_{my} - \Sigma_{mx}\Sigma^{-1}_{xx}\Sigma_{xy}) β ( m y ) := β ( m y ∣ m , x ) = ( Σ mm − Σ m x Σ xx − 1 Σ x m ) − 1 ( Σ m y − Σ m x Σ xx − 1 Σ x y ) and β ( x m ) : = β ( x m ∣ x ) = Σ x x − 1 Σ x m \beta^{(xm)} := \beta^{(xm|x)} = \Sigma^{-1}_{xx}\Sigma_{xm} β ( x m ) := β ( x m ∣ x ) = Σ xx − 1 Σ x m .
Non-latent model
Standard mediation models can be expressed as:
p ( x , m , y ) = p ( y ∣ x , m ) p ( m ∣ x ) p ( x ) p(x,m,y)=p(y\mid x,m)p(m\mid x)p(x) p ( x , m , y ) = p ( y ∣ x , m ) p ( m ∣ x ) p ( x )
We have:
Σ x x = Σ x Σ m m = ( β ( x m ) ) T Σ x x β ( x m ) + Σ m Σ m x = ( β ( x m ) ) T Σ x x Σ x m = Σ x x β ( x m ) Σ m y = ( β ( x m ) ) T Σ x β ( x y ) + Σ m m β ( m y ) Σ x y = Σ x x ( β ( x y ) + β ( x m ) β ( m y ) ) \begin{aligned}
\Sigma_{xx} &= \Sigma_x \\
\Sigma_{mm} &= (\beta^{(xm)})^{T}\Sigma_{xx}\beta^{(xm)}+\Sigma_m \\
\Sigma_{mx} &= (\beta^{(xm)})^{T}\Sigma_{xx} \\
\Sigma_{xm} &= \Sigma_{xx}\beta^{(xm)} \\
\Sigma_{my} &= (\beta^{(xm)})^{T}\Sigma_x\beta^{(xy)}+\Sigma_{mm}\beta^{(my)} \\
\Sigma_{xy} &= \Sigma_{xx}\left(\beta^{(xy)}+\beta^{(xm)}\beta^{(my)}\right)
\end{aligned} Σ xx Σ mm Σ m x Σ x m Σ m y Σ x y = Σ x = ( β ( x m ) ) T Σ xx β ( x m ) + Σ m = ( β ( x m ) ) T Σ xx = Σ xx β ( x m ) = ( β ( x m ) ) T Σ x β ( x y ) + Σ mm β ( m y ) = Σ xx ( β ( x y ) + β ( x m ) β ( m y ) )
Latent model
Latent mediation models can be expressed as:
p ( x , m , y , s ) = p ( y ∣ x , s ) p ( s ∣ x ) p ( m ∣ s ) p ( x ) = p ( y ∣ x , s ) p ( s ∣ m ) p ( x ∣ s ) p ( m ) p(x,m,y,s)=p(y\mid x,s)p(s\mid x)p(m\mid s)p(x)=p(y\mid x,s)p(s\mid m)p(x\mid s)p(m) p ( x , m , y , s ) = p ( y ∣ x , s ) p ( s ∣ x ) p ( m ∣ s ) p ( x ) = p ( y ∣ x , s ) p ( s ∣ m ) p ( x ∣ s ) p ( m )
The equations for inverting the parameters remain the same, but covariance matrices are now expressed as:
Model vX.
Σ x x = Σ x Σ s s = Σ s ∣ x + ( β ( x s ) ) T Σ x x β ( x s ) Σ m m = Σ m ∣ s + ( β ( s m ) ) T Σ s s β ( s m ) Σ x m = Σ x x β ( x s ) β ( s m ) Σ m x = ( β ( s m ) ) T ( β ( x s ) ) T Σ x x Σ m y = ( β ( s m ) ) T ( β ( x s ) ) T Σ x x β ( x y ) + ( β ( s m ) ) T Σ s s β ( s y ) Σ x y = Σ x x ( β ( x y ) + β ( x s ) β ( s y ) ) \begin{aligned}
\Sigma_{xx} &= \Sigma_x \\
\Sigma_{ss} &= \Sigma_{s\mid x}+(\beta^{(xs)})^{T}\Sigma_{xx}\beta^{(xs)} \\
\Sigma_{mm} &= \Sigma_{m\mid s}+(\beta^{(sm)})^{T}\Sigma_{ss}\beta^{(sm)} \\
\Sigma_{xm} &= \Sigma_{xx}\beta^{(xs)}\beta^{(sm)} \\
\Sigma_{mx} &= (\beta^{(sm)})^{T}(\beta^{(xs)})^{T}\Sigma_{xx} \\
\Sigma_{my} &= (\beta^{(sm)})^{T}(\beta^{(xs)})^{T}\Sigma_{xx}\beta^{(xy)}+(\beta^{(sm)})^{T}\Sigma_{ss}\beta^{(sy)} \\
\Sigma_{xy} &= \Sigma_{xx}\left(\beta^{(xy)}+\beta^{(xs)}\beta^{(sy)}\right)
\end{aligned} Σ xx Σ ss Σ mm Σ x m Σ m x Σ m y Σ x y = Σ x = Σ s ∣ x + ( β ( x s ) ) T Σ xx β ( x s ) = Σ m ∣ s + ( β ( s m ) ) T Σ ss β ( s m ) = Σ xx β ( x s ) β ( s m ) = ( β ( s m ) ) T ( β ( x s ) ) T Σ xx = ( β ( s m ) ) T ( β ( x s ) ) T Σ xx β ( x y ) + ( β ( s m ) ) T Σ ss β ( sy ) = Σ xx ( β ( x y ) + β ( x s ) β ( sy ) )
Model vM.
Σ m m = Σ m Σ s s = Σ s ∣ m + ( β ( m s ) ) T Σ m m β ( m s ) Σ x x = Σ x ∣ s + ( β ( s x ) ) T Σ s s β ( s x ) Σ m x = Σ m m β ( m s ) β ( s x ) Σ x m = ( β ( s x ) ) T ( β ( m s ) ) T Σ m m Σ m y = Σ m m β ( m s ) ( β ( s x ) β ( x y ) + β ( s y ) ) Σ x y = Σ x x β ( x y ) + ( β ( s x ) ) T Σ s s β ( s y ) \begin{aligned}
\Sigma_{mm} &= \Sigma_m \\
\Sigma_{ss} &= \Sigma_{s\mid m}+(\beta^{(ms)})^{T}\Sigma_{mm}\beta^{(ms)} \\
\Sigma_{xx} &= \Sigma_{x\mid s}+(\beta^{(sx)})^{T}\Sigma_{ss}\beta^{(sx)} \\
\Sigma_{mx} &= \Sigma_{mm}\beta^{(ms)}\beta^{(sx)} \\
\Sigma_{xm} &= (\beta^{(sx)})^{T}(\beta^{(ms)})^{T}\Sigma_{mm} \\
\Sigma_{my} &= \Sigma_{mm}\beta^{(ms)}\left(\beta^{(sx)}\beta^{(xy)}+\beta^{(sy)}\right) \\
\Sigma_{xy} &= \Sigma_{xx}\beta^{(xy)}+(\beta^{(sx)})^{T}\Sigma_{ss}\beta^{(sy)}
\end{aligned} Σ mm Σ ss Σ xx Σ m x Σ x m Σ m y Σ x y = Σ m = Σ s ∣ m + ( β ( m s ) ) T Σ mm β ( m s ) = Σ x ∣ s + ( β ( s x ) ) T Σ ss β ( s x ) = Σ mm β ( m s ) β ( s x ) = ( β ( s x ) ) T ( β ( m s ) ) T Σ mm = Σ mm β ( m s ) ( β ( s x ) β ( x y ) + β ( sy ) ) = Σ xx β ( x y ) + ( β ( s x ) ) T Σ ss β ( sy )
Signal-to-noise ratios
LaMe
LaMe-vX
By definition, signal-to-noise ratios are:
SNR ( S ∣ X ) = Tr ( ( β ( x s ) ) T Σ x β ( x s ) ) Tr ( Σ s ∣ x ) = ∥ Σ x 1 2 β ( x s ) ∥ F 2 ∥ Σ s ∣ x 1 2 ∥ F 2 \operatorname{SNR}(S|X)=\frac{\operatorname{Tr}\left((\beta^{(xs)})^{T}\Sigma_x\beta^{(xs)}\right)}{\operatorname{Tr}(\Sigma_{s|x})}
=\frac{\left\|\Sigma_x^{\frac{1}{2}}\beta^{(xs)}\right\|_{F}^{2}}{\left\|\Sigma_{s|x}^{\frac{1}{2}}\right\|_{F}^{2}} SNR ( S ∣ X ) = Tr ( Σ s ∣ x ) Tr ( ( β ( x s ) ) T Σ x β ( x s ) ) = Σ s ∣ x 2 1 F 2 Σ x 2 1 β ( x s ) F 2
SNR ( M ∣ S ) = Tr ( ( β ( s m ) ) T ( ( β ( x s ) ) T Σ x β ( x s ) + Σ s ∣ x ) β ( s m ) ) Tr ( Σ m ∣ s ) = ∥ ( ( β ( x s ) ) T Σ x β ( x s ) + Σ s ∣ x ) 1 2 β ( s m ) ∥ F 2 ∥ Σ m ∣ s 1 2 ∥ F 2 \operatorname{SNR}(M|S)=\frac{\operatorname{Tr}\left((\beta^{(sm)})^{T}\left((\beta^{(xs)})^{T}\Sigma_x\beta^{(xs)}+\Sigma_{s|x}\right)\beta^{(sm)}\right)}{\operatorname{Tr}(\Sigma_{m|s})}
=\frac{\left\|\left((\beta^{(xs)})^{T}\Sigma_x\beta^{(xs)}+\Sigma_{s|x}\right)^{\frac{1}{2}}\beta^{(sm)}\right\|_{F}^{2}}{\left\|\Sigma_{m|s}^{\frac{1}{2}}\right\|_{F}^{2}} SNR ( M ∣ S ) = Tr ( Σ m ∣ s ) Tr ( ( β ( s m ) ) T ( ( β ( x s ) ) T Σ x β ( x s ) + Σ s ∣ x ) β ( s m ) ) = Σ m ∣ s 2 1 F 2 ( ( β ( x s ) ) T Σ x β ( x s ) + Σ s ∣ x ) 2 1 β ( s m ) F 2
SNR ( Y ∣ X , S ) = Tr ( ( β ( x y ) + β ( x s ) β ( s y ) ) T Σ x ( β ( x y ) + β ( x s ) β ( s y ) ) + ( β ( s y ) ) T Σ s ∣ x β ( s y ) ) Tr ( Σ y ∣ x , s ) = ∥ Σ x 1 2 ( β ( x y ) + β ( x s ) β ( s y ) ) ∥ F 2 + ∥ Σ s ∣ x 1 2 β ( s y ) ∥ F 2 ∥ Σ y ∣ x , s 1 2 ∥ F 2 \begin{aligned}
\operatorname{SNR}(Y|X,S)
&=\frac{\operatorname{Tr}\left((\beta^{(xy)}+\beta^{(xs)}\beta^{(sy)})^{T}\Sigma_x(\beta^{(xy)}+\beta^{(xs)}\beta^{(sy)})+(\beta^{(sy)})^{T}\Sigma_{s|x}\beta^{(sy)}\right)}{\operatorname{Tr}(\Sigma_{y|x,s})}\\
&=\frac{\left\|\Sigma_x^{\frac{1}{2}}(\beta^{(xy)}+\beta^{(xs)}\beta^{(sy)})\right\|_{F}^{2}+\left\|\Sigma_{s|x}^{\frac{1}{2}}\beta^{(sy)}\right\|_{F}^{2}}{\left\|\Sigma_{y|x,s}^{\frac{1}{2}}\right\|_{F}^{2}}
\end{aligned} SNR ( Y ∣ X , S ) = Tr ( Σ y ∣ x , s ) Tr ( ( β ( x y ) + β ( x s ) β ( sy ) ) T Σ x ( β ( x y ) + β ( x s ) β ( sy ) ) + ( β ( sy ) ) T Σ s ∣ x β ( sy ) ) = Σ y ∣ x , s 2 1 F 2 Σ x 2 1 ( β ( x y ) + β ( x s ) β ( sy ) ) F 2 + Σ s ∣ x 2 1 β ( sy ) F 2
The scaling factor can be obtained by simple algebra so that Σ A ′ = s A Σ A \Sigma'_A=s_A\Sigma_A Σ A ′ = s A Σ A so that the signal-to-noise ratio of A A A given its parent nodes is R A R_A R A , for A ∈ [ X , S , M , Y ] A\in[X,S,M,Y] A ∈ [ X , S , M , Y ] . Thus introducing s m s_m s m , s y s_y s y and s s s_s s s :
SNR ( S ∣ X ) : = R s = ∥ Σ x 1 2 β ( x s ) ∥ F 2 s s ∥ Σ s ∣ x 1 2 ∥ F 2 \operatorname{SNR}(S|X):=R_s=\frac{\left\|\Sigma_x^{\frac{1}{2}}\beta^{(xs)}\right\|_{F}^{2}}{s_s\left\|\Sigma_{s|x}^{\frac{1}{2}}\right\|_{F}^{2}} SNR ( S ∣ X ) := R s = s s Σ s ∣ x 2 1 F 2 Σ x 2 1 β ( x s ) F 2
⟺ s s = ∥ Σ x 1 2 β ( x s ) ∥ F 2 R s ∥ Σ s ∣ x 1 2 ∥ F 2 \Longleftrightarrow s_s=\frac{\left\|\Sigma_x^{\frac{1}{2}}\beta^{(xs)}\right\|_{F}^{2}}{R_s\left\|\Sigma_{s|x}^{\frac{1}{2}}\right\|_{F}^{2}} ⟺ s s = R s Σ s ∣ x 2 1 F 2 Σ x 2 1 β ( x s ) F 2
SNR ( M ∣ S ) : = R m = ∥ ( ( β ( x s ) ) T Σ x β ( x s ) + s s Σ s ∣ x ) 1 2 β ( s m ) ∥ F 2 s m ∥ Σ m ∣ s 1 2 ∥ F 2 \operatorname{SNR}(M|S):=R_m=\frac{\left\|\left((\beta^{(xs)})^{T}\Sigma_x\beta^{(xs)}+s_s\Sigma_{s|x}\right)^{\frac{1}{2}}\beta^{(sm)}\right\|_{F}^{2}}{s_m\left\|\Sigma_{m|s}^{\frac{1}{2}}\right\|_{F}^{2}} SNR ( M ∣ S ) := R m = s m Σ m ∣ s 2 1 F 2 ( ( β ( x s ) ) T Σ x β ( x s ) + s s Σ s ∣ x ) 2 1 β ( s m ) F 2
⟺ s m = ∥ ( ( β ( x s ) ) T Σ x β ( x s ) + s s Σ s ∣ x ) 1 2 β ( s m ) ∥ F 2 R m ∥ Σ m ∣ s 1 2 ∥ F 2 \Longleftrightarrow s_m=\frac{\left\|\left((\beta^{(xs)})^{T}\Sigma_x\beta^{(xs)}+s_s\Sigma_{s|x}\right)^{\frac{1}{2}}\beta^{(sm)}\right\|_{F}^{2}}{R_m\left\|\Sigma_{m|s}^{\frac{1}{2}}\right\|_{F}^{2}} ⟺ s m = R m Σ m ∣ s 2 1 F 2 ( ( β ( x s ) ) T Σ x β ( x s ) + s s Σ s ∣ x ) 2 1 β ( s m ) F 2
SNR ( Y ∣ X , S ) : = R y = ∥ Σ x 1 2 ( β ( x y ) + β ( x s ) β ( s y ) ) ∥ F 2 + s s ∥ Σ s ∣ x 1 2 β ( s y ) ∥ F 2 s y ∥ Σ y ∣ x , s 1 2 ∥ F 2 \operatorname{SNR}(Y|X,S):=R_y=\frac{\left\|\Sigma_x^{\frac{1}{2}}(\beta^{(xy)}+\beta^{(xs)}\beta^{(sy)})\right\|_{F}^{2}+s_s\left\|\Sigma_{s|x}^{\frac{1}{2}}\beta^{(sy)}\right\|_{F}^{2}}{s_y\left\|\Sigma_{y|x,s}^{\frac{1}{2}}\right\|_{F}^{2}} SNR ( Y ∣ X , S ) := R y = s y Σ y ∣ x , s 2 1 F 2 Σ x 2 1 ( β ( x y ) + β ( x s ) β ( sy ) ) F 2 + s s Σ s ∣ x 2 1 β ( sy ) F 2
⟺ s y = ∥ Σ x 1 2 ( β ( x y ) + β ( x s ) β ( s y ) ) ∥ F 2 + s s ∥ Σ s ∣ x 1 2 β ( s y ) ∥ F 2 R y ∥ Σ y ∣ x , s 1 2 ∥ F 2 \Longleftrightarrow s_y=\frac{\left\|\Sigma_x^{\frac{1}{2}}(\beta^{(xy)}+\beta^{(xs)}\beta^{(sy)})\right\|_{F}^{2}+s_s\left\|\Sigma_{s|x}^{\frac{1}{2}}\beta^{(sy)}\right\|_{F}^{2}}{R_y\left\|\Sigma_{y|x,s}^{\frac{1}{2}}\right\|_{F}^{2}} ⟺ s y = R y Σ y ∣ x , s 2 1 F 2 Σ x 2 1 ( β ( x y ) + β ( x s ) β ( sy ) ) F 2 + s s Σ s ∣ x 2 1 β ( sy ) F 2
LaMe-vM
By definition, signal-to-noise ratios are:
SNR ( S ∣ M ) = Tr ( ( β ( m s ) ) T Σ m β ( m s ) ) Tr ( Σ s ∣ m ) = ∥ Σ m 1 2 β ( m s ) ∥ F 2 ∥ Σ s ∣ m 1 2 ∥ F 2 \operatorname{SNR}(S|M)=\frac{\operatorname{Tr}\left((\beta^{(ms)})^{T}\Sigma_m\beta^{(ms)}\right)}{\operatorname{Tr}(\Sigma_{s|m})}
=\frac{\left\|\Sigma_m^{\frac{1}{2}}\beta^{(ms)}\right\|_{F}^{2}}{\left\|\Sigma_{s|m}^{\frac{1}{2}}\right\|_{F}^{2}} SNR ( S ∣ M ) = Tr ( Σ s ∣ m ) Tr ( ( β ( m s ) ) T Σ m β ( m s ) ) = Σ s ∣ m 2 1 F 2 Σ m 2 1 β ( m s ) F 2
SNR ( X ∣ S ) = Tr ( ( β ( s x ) ) T ( ( β ( m s ) ) T Σ m β ( m s ) + Σ s ∣ m ) β ( s x ) ) Tr ( Σ x ∣ s ) = ∥ ( ( β ( m s ) ) T Σ m β ( m s ) + Σ s ∣ m ) 1 2 β ( s x ) ∥ F 2 ∥ Σ x ∣ s 1 2 ∥ F 2 \operatorname{SNR}(X|S)=\frac{\operatorname{Tr}\left((\beta^{(sx)})^{T}\left((\beta^{(ms)})^{T}\Sigma_m\beta^{(ms)}+\Sigma_{s|m}\right)\beta^{(sx)}\right)}{\operatorname{Tr}(\Sigma_{x|s})}
=\frac{\left\|\left((\beta^{(ms)})^{T}\Sigma_m\beta^{(ms)}+\Sigma_{s|m}\right)^{\frac{1}{2}}\beta^{(sx)}\right\|_{F}^{2}}{\left\|\Sigma_{x|s}^{\frac{1}{2}}\right\|_{F}^{2}} SNR ( X ∣ S ) = Tr ( Σ x ∣ s ) Tr ( ( β ( s x ) ) T ( ( β ( m s ) ) T Σ m β ( m s ) + Σ s ∣ m ) β ( s x ) ) = Σ x ∣ s 2 1 F 2 ( ( β ( m s ) ) T Σ m β ( m s ) + Σ s ∣ m ) 2 1 β ( s x ) F 2
SNR ( Y ∣ X , S ) = Tr ( ( β ( s y ) ) T Σ s β ( s y ) + ( β ( x y ) ) T Σ x β ( x y ) + 2 ( β ( s x ) β ( x y ) ) T Σ s β ( s y ) ) Tr ( Σ y ∣ x , s ) = ∥ ( ( β ( m s ) ) T Σ m β ( m s ) + Σ s ∣ m ) 1 2 ( β ( s x ) β ( x y ) + β ( s y ) ) ∥ F 2 + ∥ Σ x ∣ s 1 2 β ( x y ) ∥ F 2 ∥ Σ y ∣ x , s 1 2 ∥ F 2 \begin{aligned}
\operatorname{SNR}(Y|X,S)
&=\frac{\operatorname{Tr}\left((\beta^{(sy)})^{T}\Sigma_s\beta^{(sy)}+(\beta^{(xy)})^{T}\Sigma_x\beta^{(xy)}+2(\beta^{(sx)}\beta^{(xy)})^{T}\Sigma_s\beta^{(sy)}\right)}{\operatorname{Tr}(\Sigma_{y|x,s})}\\
&=\frac{\left\|\left((\beta^{(ms)})^{T}\Sigma_m\beta^{(ms)}+\Sigma_{s|m}\right)^{\frac{1}{2}}(\beta^{(sx)}\beta^{(xy)}+\beta^{(sy)})\right\|_{F}^{2}+\left\|\Sigma_{x|s}^{\frac{1}{2}}\beta^{(xy)}\right\|_{F}^{2}}{\left\|\Sigma_{y|x,s}^{\frac{1}{2}}\right\|_{F}^{2}}
\end{aligned} SNR ( Y ∣ X , S ) = Tr ( Σ y ∣ x , s ) Tr ( ( β ( sy ) ) T Σ s β ( sy ) + ( β ( x y ) ) T Σ x β ( x y ) + 2 ( β ( s x ) β ( x y ) ) T Σ s β ( sy ) ) = Σ y ∣ x , s 2 1 F 2 ( ( β ( m s ) ) T Σ m β ( m s ) + Σ s ∣ m ) 2 1 ( β ( s x ) β ( x y ) + β ( sy ) ) F 2 + Σ x ∣ s 2 1 β ( x y ) F 2
where the unconditional variances are Σ s = ( ( β ( m s ) ) T Σ m β ( m s ) + Σ s ∣ m ) \Sigma_s=((\beta^{(ms)})^{T}\Sigma_m\beta^{(ms)}+\Sigma_{s|m}) Σ s = (( β ( m s ) ) T Σ m β ( m s ) + Σ s ∣ m ) and Σ x = Σ x ∣ s + ( β ( s x ) ) T Σ s β ( s x ) \Sigma_x=\Sigma_{x|s}+(\beta^{(sx)})^{T}\Sigma_s\beta^{(sx)} Σ x = Σ x ∣ s + ( β ( s x ) ) T Σ s β ( s x ) in the LaMe-vM parametrization.
The scaling factor can be obtained by simple algebra so that Σ A ′ = s A Σ A \Sigma'_A=s_A\Sigma_A Σ A ′ = s A Σ A so that the signal-to-noise ratio of A A A given its parent nodes is R A R_A R A , for A ∈ [ X , S , M , Y ] A \in[X,S,M,Y] A ∈ [ X , S , M , Y ] . Thus introducing s x s_x s x , s y s_y s y and s s s_s s s :
SNR ( S ∣ M ) : = R s = ∥ Σ m 1 2 β ( m s ) ∥ F 2 s s ∥ Σ s ∣ m 1 2 ∥ F 2 ⟺ s s = ∥ Σ m 1 2 β ( m s ) ∥ F 2 R s ∥ Σ s ∣ m 1 2 ∥ F 2 \operatorname{SNR}(S\mid M) := R_s = \frac{\left\lVert\Sigma_{m}^{\frac{1}{2}}\beta^{(ms)} \right\rVert_F^2}{s_s\left\lVert\Sigma_{s\mid m}^{\frac{1}{2}} \right\rVert_F^2} \Longleftrightarrow s_s = \frac{\left\lVert\Sigma_{m}^{\frac{1}{2}}\beta^{(ms)} \right\rVert_F^2}{R_s\left\lVert\Sigma_{s\mid m}^{\frac{1}{2}} \right\rVert_F^2} SNR ( S ∣ M ) := R s = s s Σ s ∣ m 2 1 F 2 Σ m 2 1 β ( m s ) F 2 ⟺ s s = R s Σ s ∣ m 2 1 F 2 Σ m 2 1 β ( m s ) F 2
SNR ( X ∣ S ) : = R x = ∥ ( ( β ( m s ) ) T Σ m β ( m s ) + s s Σ s ∣ m ) 1 2 β ( s x ) ∥ F 2 s x ∥ Σ x ∣ s 1 2 ∥ F 2 \operatorname{SNR}(X\mid S) := R_x = \frac{\left\lVert\left((\beta^{(ms)})^{T}\Sigma_m\beta^{(ms)} + s_s\Sigma_{s\mid m}\right)^{\frac{1}{2}}\beta^{(sx)} \right\rVert_F^2}{s_x\left\lVert\Sigma_{x\mid s}^{\frac{1}{2}} \right\rVert_F^2} SNR ( X ∣ S ) := R x = s x Σ x ∣ s 2 1 F 2 ( ( β ( m s ) ) T Σ m β ( m s ) + s s Σ s ∣ m ) 2 1 β ( s x ) F 2
⟺ s x = ∥ ( ( β ( m s ) ) T Σ m β ( m s ) + s s Σ s ∣ m ) 1 2 β ( s x ) ∥ F 2 R x ∥ Σ x ∣ s 1 2 ∥ F 2 \Longleftrightarrow s_x = \frac{\left\lVert\left((\beta^{(ms)})^{T}\Sigma_m\beta^{(ms)} + s_s\Sigma_{s\mid m}\right)^{\frac{1}{2}}\beta^{(sx)} \right\rVert_F^2}{R_x\left\lVert\Sigma_{x\mid s}^{\frac{1}{2}} \right\rVert_F^2} ⟺ s x = R x Σ x ∣ s 2 1 F 2 ( ( β ( m s ) ) T Σ m β ( m s ) + s s Σ s ∣ m ) 2 1 β ( s x ) F 2
SNR ( Y ∣ X , S ) : = R y = ∥ Σ s ∣ s 1 2 ( s s ) ( β ( s x ) β ( x y ) + β ( s y ) ) ∥ F 2 + s x ∥ Σ x ∣ s 1 2 β ( x y ) ∥ F 2 s y ∥ Σ y ∣ x , s 1 2 ∥ F 2 \operatorname{SNR}(Y\mid X,S) := R_y = \frac{\left\lVert\Sigma_{s\mid s}^{\frac{1}{2}}(s_s)(\beta^{(sx)}\beta^{(xy)}+\beta^{(sy)}) \right\rVert_F^2+s_x\left\lVert\Sigma_{x\mid s}^{\frac{1}{2}}\beta^{(xy)} \right\rVert_F^2}{s_y\left\lVert\Sigma_{y\mid x,s}^{\frac{1}{2}} \right\rVert_F^2} SNR ( Y ∣ X , S ) := R y = s y Σ y ∣ x , s 2 1 F 2 Σ s ∣ s 2 1 ( s s ) ( β ( s x ) β ( x y ) + β ( sy ) ) F 2 + s x Σ x ∣ s 2 1 β ( x y ) F 2
⟺ s y = ∥ Σ s ∣ s 1 2 ( s s ) ( β ( s x ) β ( x y ) + β ( s y ) ) ∥ F 2 + s x ∥ Σ x ∣ s 1 2 β ( x y ) ∥ F 2 R y ∥ Σ y ∣ x , s 1 2 ∥ F 2 = ∥ ( ( β ( m s ) ) T Σ m β ( m s ) + s s Σ s ∣ m ) 1 2 ( β ( s x ) β ( x y ) + β ( s y ) ) ∥ F 2 + s x ∥ Σ x ∣ s 1 2 β ( x y ) ∥ F 2 R y ∥ Σ y ∣ x , s 1 2 ∥ F 2 \begin{aligned}
\Longleftrightarrow s_y = \frac{\left\lVert\Sigma_{s\mid s}^{\frac{1}{2}}(s_s)(\beta^{(sx)}\beta^{(xy)}+\beta^{(sy)}) \right\rVert_F^2+s_x\left\lVert\Sigma_{x\mid s}^{\frac{1}{2}}\beta^{(xy)} \right\rVert_F^2}{R_y\left\lVert\Sigma_{y\mid x,s}^{\frac{1}{2}} \right\rVert_F^2} \\
&= \frac{\left\lVert\left((\beta^{(ms)})^{T}\Sigma_m\beta^{(ms)}+s_s\Sigma_{s\mid m}\right)^{\frac{1}{2}}(\beta^{(sx)}\beta^{(xy)}+\beta^{(sy)}) \right\rVert_F^2+s_x\left\lVert\Sigma_{x\mid s}^{\frac{1}{2}}\beta^{(xy)} \right\rVert_F^2}{R_y\left\lVert\Sigma_{y\mid x,s}^{\frac{1}{2}} \right\rVert_F^2}
\end{aligned} ⟺ s y = R y Σ y ∣ x , s 2 1 F 2 Σ s ∣ s 2 1 ( s s ) ( β ( s x ) β ( x y ) + β ( sy ) ) F 2 + s x Σ x ∣ s 2 1 β ( x y ) F 2 = R y Σ y ∣ x , s 2 1 F 2 ( ( β ( m s ) ) T Σ m β ( m s ) + s s Σ s ∣ m ) 2 1 ( β ( s x ) β ( x y ) + β ( sy ) ) F 2 + s x Σ x ∣ s 2 1 β ( x y ) F 2
where Σ s s ( s s ) = ( β ( m s ) ) T Σ m β ( m s ) + s s Σ s ∣ m \Sigma_{ss}(s_s)=(\beta^{(ms)})^{T}\Sigma_m\beta^{(ms)}+s_s\Sigma_{s\mid m} Σ ss ( s s ) = ( β ( m s ) ) T Σ m β ( m s ) + s s Σ s ∣ m and Σ x x ( s x , s s ) = ( β ( s x ) ) T ( β ( m s ) ) T Σ m β ( m s ) β ( s x ) + ( β ( s x ) ) T s s Σ s ∣ m β ( s x ) + s x Σ x ∣ s \Sigma_{xx}(s_x,s_s)=(\beta^{(sx)})^{T}(\beta^{(ms)})^{T}\Sigma_m\beta^{(ms)}\beta^{(sx)}+(\beta^{(sx)})^{T}s_s\Sigma_{s\mid m}\beta^{(sx)}+s_x\Sigma_{x\mid s} Σ xx ( s x , s s ) = ( β ( s x ) ) T ( β ( m s ) ) T Σ m β ( m s ) β ( s x ) + ( β ( s x ) ) T s s Σ s ∣ m β ( s x ) + s x Σ x ∣ s .
Non-latent models
By definition, signal-to-noise ratios are:
SNR ( M ∣ X ) = Tr ( ( β ( x m ) ) T Σ x β ( x m ) ) Tr ( Σ m ∣ x ) = ∥ Σ x 1 2 β ( x m ) ∥ F 2 ∥ Σ m ∣ x 1 2 ∥ F 2 \operatorname{SNR}(M\mid X)=\frac{\operatorname{Tr}((\beta^{(xm)})^{T}\Sigma_x\beta^{(xm)})}{\operatorname{Tr}(\Sigma_{m\mid x})}=\frac{\left\lVert\Sigma_x^{\frac{1}{2}}\beta^{(xm)} \right\rVert_F^2}{\left\lVert\Sigma_{m\mid x}^{\frac{1}{2}} \right\rVert_F^2} SNR ( M ∣ X ) = Tr ( Σ m ∣ x ) Tr (( β ( x m ) ) T Σ x β ( x m ) ) = Σ m ∣ x 2 1 F 2 Σ x 2 1 β ( x m ) F 2
SNR ( Y ∣ X , M ) = Tr ( ( ( β ( x y ) + β ( x m ) β ( m y ) ) T Σ x ( β ( x y ) + β ( x m ) β ( m y ) ) + ( β ( m y ) ) T Σ m ∣ x β ( m y ) ) Tr ( Σ y ∣ x , m ) = ∥ Σ x 1 2 ( β ( x y ) + β ( x m ) β ( m y ) ) ∥ F 2 + ∥ Σ m ∣ x 1 2 β ( m y ) ∥ F 2 ∥ Σ y ∣ x , m 1 2 ∥ F 2 \begin{aligned}
\operatorname{SNR}(Y\mid X,M)=\frac{\operatorname{Tr}(((\beta^{(xy)}+\beta^{(xm)}\beta^{(my)})^{T}\Sigma_x(\beta^{(xy)}+\beta^{(xm)}\beta^{(my)})+(\beta^{(my)})^{T}\Sigma_{m\mid x}\beta^{(my)})}{\operatorname{Tr}(\Sigma_{y\mid x,m})} \\
&=\frac{\left\lVert\Sigma_x^{\frac{1}{2}}(\beta^{(xy)}+\beta^{(xm)}\beta^{(my)}) \right\rVert_F^2+\left\lVert\Sigma_{m\mid x}^{\frac{1}{2}}\beta^{(my)} \right\rVert_F^2}{\left\lVert\Sigma_{y\mid x,m}^{\frac{1}{2}} \right\rVert_F^2}
\end{aligned} SNR ( Y ∣ X , M ) = Tr ( Σ y ∣ x , m ) Tr ((( β ( x y ) + β ( x m ) β ( m y ) ) T Σ x ( β ( x y ) + β ( x m ) β ( m y ) ) + ( β ( m y ) ) T Σ m ∣ x β ( m y ) ) = Σ y ∣ x , m 2 1 F 2 Σ x 2 1 ( β ( x y ) + β ( x m ) β ( m y ) ) F 2 + Σ m ∣ x 2 1 β ( m y ) F 2
The scaling factor can be obtained by simple algebra so that Σ A ′ = s A Σ A \Sigma'_A=s_A\Sigma_A Σ A ′ = s A Σ A so that the signal-to-noise ratio of A A A given its parent nodes is R A R_A R A , for A ∈ [ X , M , Y ] A \in[X,M,Y] A ∈ [ X , M , Y ] . Thus introducing s m s_m s m and s y s_y s y :
SNR ( M ∣ X ) : = R m = ∥ Σ x 1 2 β ( x m ) ∥ F 2 s m ∥ Σ m ∣ x 1 2 ∥ F 2 \operatorname{SNR}(M\mid X):=R_m=\frac{\left\lVert\Sigma_x^{\frac{1}{2}}\beta^{(xm)} \right\rVert_F^2}{s_m\left\lVert\Sigma_{m\mid x}^{\frac{1}{2}} \right\rVert_F^2} SNR ( M ∣ X ) := R m = s m Σ m ∣ x 2 1 F 2 Σ x 2 1 β ( x m ) F 2
⟺ s m = ∥ Σ x 1 2 β ( x m ) ∥ F 2 R m ∥ Σ m ∣ x 1 2 ∥ F 2 \Longleftrightarrow s_m=\frac{\left\lVert\Sigma_x^{\frac{1}{2}}\beta^{(xm)} \right\rVert_F^2}{R_m\left\lVert\Sigma_{m\mid x}^{\frac{1}{2}} \right\rVert_F^2} ⟺ s m = R m Σ m ∣ x 2 1 F 2 Σ x 2 1 β ( x m ) F 2
SNR ( Y ∣ X , M ) : = R y = ∥ Σ x 1 2 ( β ( x y ) + β ( x m ) β ( m y ) ) ∥ F 2 + s m ∥ Σ m ∣ x 1 2 β ( m y ) ∥ F 2 s y ∥ Σ y ∣ x , m 1 2 ∥ F 2 \operatorname{SNR}(Y\mid X,M):=R_y=\frac{\left\lVert\Sigma_x^{\frac{1}{2}}(\beta^{(xy)}+\beta^{(xm)}\beta^{(my)}) \right\rVert_F^2+s_m\left\lVert\Sigma_{m\mid x}^{\frac{1}{2}}\beta^{(my)} \right\rVert_F^2}{s_y\left\lVert\Sigma_{y\mid x,m}^{\frac{1}{2}} \right\rVert_F^2} SNR ( Y ∣ X , M ) := R y = s y Σ y ∣ x , m 2 1 F 2 Σ x 2 1 ( β ( x y ) + β ( x m ) β ( m y ) ) F 2 + s m Σ m ∣ x 2 1 β ( m y ) F 2
⟺ s y = ∥ Σ x 1 2 ( β ( x y ) + β ( x m ) β ( m y ) ) ∥ F 2 + s m ∥ Σ m ∣ x 1 2 β ( m y ) ∥ F 2 R y ∥ Σ y ∣ x , m 1 2 ∥ F 2 \Longleftrightarrow s_y=\frac{\left\lVert\Sigma_x^{\frac{1}{2}}(\beta^{(xy)}+\beta^{(xm)}\beta^{(my)}) \right\rVert_F^2+s_m\left\lVert\Sigma_{m\mid x}^{\frac{1}{2}}\beta^{(my)} \right\rVert_F^2}{R_y\left\lVert\Sigma_{y\mid x,m}^{\frac{1}{2}} \right\rVert_F^2} ⟺ s y = R y Σ y ∣ x , m 2 1 F 2 Σ x 2 1 ( β ( x y ) + β ( x m ) β ( m y ) ) F 2 + s m Σ m ∣ x 2 1 β ( m y ) F 2
Baselines implementation details
JAOG. We extended Nath et al., [12 ] ’s model to handle multivariate latent variables (K ≥ 1 K \ge1 K ≥ 1 ). In the article, the authors posit a deterministic dimension reduction model S = φ ( M ) S=\varphi(M) S = φ ( M ) , whose output, S S S , is used in a standard LASSO mediation model. To do so, the authors define an optimization target both for inferring the parameters of φ ( M ) \varphi(M) φ ( M ) and for inferring the structural parameters of the mediation model. Adapting their notation, we define E = Y − X β ( x y ) E=Y-X\beta^{(xy)} E = Y − X β ( x y ) and H = X β ( x s ) H=X\beta^{(xs)} H = X β ( x s ) . The target of the multivariate extension of Nath et al., [12 ] then becomes ∥ φ ( M ) − ( E ( β ( s y ) ) T + H ) ( β ( s y ) ( β ( s y ) ) T + I K ) − 1 ∥ 2 \left\lVert\varphi(M)-\left(E(\beta^{(sy)})^{T}+H\right)\left(\beta^{(sy)}(\beta^{(sy)})^{T}+I_K\right)^{-1}\right\rVert^2 φ ( M ) − ( E ( β ( sy ) ) T + H ) ( β ( sy ) ( β ( sy ) ) T + I K ) − 1 2 . Finally, we choose φ ( M ) \varphi(M) φ ( M ) to be a multivariate linear Gaussian regression model so that Nath et al., [12 ] ’s approach is the closest to the data generation process used in our experiments.
LVMA. We considered a direct comparison with [7 ] , whose model assumes a diagonal manifest residual covariance Σ m ∣ s \Sigma_{m\mid s} Σ m ∣ s , whereas LaMe and other baselines impose the perfect-measurement restriction Σ m ∣ s β ( m s ) = 0 \Sigma_{m\mid s}\beta^{(ms)} = 0 Σ m ∣ s β ( m s ) = 0 for effect identification. In preliminary experiments, an adaptation accommodating multivariate exposures and outcomes performed poorly when the data-generating process violated the diagonal residual-covariance assumption. Such a comparison would therefore combine differences in latent dimensionality with differences in covariance restrictions, making it difficult to isolate the contribution of dimension reduction. Moreover, a direct benchmark of [7 ] would require extending its scalar exposure-outcome formulation while preserving its covariance restrictions, penalization, and model-selection procedure. The resulting comparison would depend on additional methodological choices beyond the scope of the present evaluation. We therefore avoid attributing the performance of a substantially modified implementation to the original method. Instead, we use an overparametrized LaMe comparator with K = I + J + L 2 K = \frac{I+J+L}{2} K = 2 I + J + L , satisfying K ≥ min ( I + L , J ) K \ge\min(I+L, J) K ≥ min ( I + L , J ) . This comparator retains the same covariance flexibility while removing the rank restriction responsible for compression, which closely relates to the point we discussed about [7 ] in the background section.
All remaining baselines were implemented using scikit-learn.
Scalability
In Figure 6 , we plot the normalized running time of LaMe and the baseline methods for a system with I = 100 I = 100 I = 100 , J = 100 J = 100 J = 100 , L = 1 L = 1 L = 1 , N = 3000 N = 3000 N = 3000 , unless specified otherwise. The models are run for a full inference procedure, meaning LaMe’s EM algorithm is run 100 times, LASSO, PLS and PCA select their hyperparameters by cross-validation, etc. Refer to the main text for more details.
Figure 6. Scalability of the models — For a system with I = 100 I = 100 I = 100 , J = 100 J = 100 J = 100 , L = 1 L = 1 L = 1 , N = 3000 N = 3000 N = 3000 , with one full inference cycle per model. Each curve is normalized by its smallest entry to show how running time evolves as a function of N N N (left) and J J J (right). α \alpha α denotes the power-law exponent.
We see that LaMe scales similarly to other methods with respect to N N N , and better than most with respect to J J J . Pathway LASSO seems to scale well, but has the longest absolute convergence time. This figure also highlights the benefits of choosing the LaMe-vM factorization over the LaMe-vX factorization when J J J is large. Absolute running times must be interpreted with caution due to their dependence on experimental parameters (number of runs, number of cross-validation folds, convergence criterion, etc.). Overall, this further supports the versatility of the LaMe framework, which can handle a range of situations efficiently and accurately.
Correlation and identifiability issues in retrieving the observed measurement set
In [7 ] , the authors define the mediating set between exposure X i X_i X i and outcome Y l Y_l Y l as: M L V M A i l : = { M j ∣ C i j l L V M A : = ∑ k β i k ( x s ) β k j ( s m ) β k l ( s y ) ≠ 0 } \mathcal{M}^{il}_{LVMA}:=\{M_j\mid C^{LVMA}_{ijl}:=\sum_k\beta^{(xs)}_{ik}\beta^{(sm)}_{kj}\beta^{(sy)}_{kl}\ne0\} M L V M A i l := { M j ∣ C ij l L V M A := ∑ k β ik ( x s ) β k j ( s m ) β k l ( sy ) = 0 } , where C i j l L V M A C^{LVMA}_{ijl} C ij l L V M A is a shorthand notation. This definition, however, does not account for candidate mediators’ correlation and non-identifiability, which raises interpretation issues.
Correlation matters for interpretation, because β k j ( s m ) ≠ 0 \beta^{(sm)}_{kj}\ne0 β k j ( s m ) = 0 in M L V M A i l \mathcal{M}^{il}_{LVMA} M L V M A i l does not necessarily imply that M j M_j M j informs latent variables S S S acting as mediators between X i X_i X i and Y l Y_l Y l . For instance, β k j ( s m ) ≠ 0 \beta^{(sm)}_{kj}\ne0 β k j ( s m ) = 0 could reflect variation in M j M_j M j along a latent component associated with X i X_i X i , without implying that this component contributes to Y l Y_l Y l . For M j M_j M j to be interpreted as a mediator, a proper definition should therefore account for both a non-zero β ( x s ) β ( s m ) \beta^{(xs)}\beta^{(sm)} β ( x s ) β ( s m ) (M M M varies with X X X through S S S ) and a non-zero β ( m s ) β ( s y ) \beta^{(ms)}\beta^{(sy)} β ( m s ) β ( sy ) (M M M informs Y Y Y through S S S ); the expression of β ( m s ) \beta^{(ms)} β ( m s ) in particular involves Σ m ∣ s \Sigma_{m\mid s} Σ m ∣ s , thus accounting for correlation between manifest variables.
The non-identifiability issue arises in two cases when rotational and scaling invariance exists. Scaling invariance can be removed by forcing Σ s ∣ x = I \Sigma_{s\mid x}=I Σ s ∣ x = I , as in [7 ] ; rotational invariance however, remains. Rotational invariance states that ( β ( x s ) , β ( s m ) , β ( s y ) , θ ( I ) ) (\beta^{(xs)},\beta^{(sm)},\beta^{(sy)},\theta(I)) ( β ( x s ) , β ( s m ) , β ( sy ) , θ ( I )) and ( β ( x s ) R , R T β ( s m ) , R T β ( s y ) , θ ( R ) ) (\beta^{(xs)}R,R^{T}\beta^{(sm)},R^{T}\beta^{(sy)},\theta(R)) ( β ( x s ) R , R T β ( s m ) , R T β ( sy ) , θ ( R )) define an equivalent reparametrization of the latent mediation model, where θ ( R ) \theta(R) θ ( R ) denotes the set of other parameters related to S S S after an orthogonal rotation R R R . The issue arises because M L V M A i l \mathcal{M}^{il}_{LVMA} M L V M A i l is not invariant to these likelihood-equivalent parametrizations: factorizing C i j l L V M A = ∣ β i ⋅ ( x s ) diag ( β ⋅ j ( s m ) ) β ⋅ l ( s y ) ∣ ≠ ∣ β i ⋅ ( x s ) R ⋅ diag ( R T β ⋅ j ( s m ) ) ⋅ R T β ⋅ l ( s y ) ∣ C^{LVMA}_{ijl}=|\beta^{(xs)}_{i\cdot}\operatorname{diag}(\beta^{(sm)}_{\cdot j})\beta^{(sy)}_{\cdot l}|\ne|\beta^{(xs)}_{i\cdot}R\cdot\operatorname{diag}(R^{T}\beta^{(sm)}_{\cdot j})\cdot R^{T}\beta^{(sy)}_{\cdot l}| C ij l L V M A = ∣ β i ⋅ ( x s ) diag ( β ⋅ j ( s m ) ) β ⋅ l ( sy ) ∣ = ∣ β i ⋅ ( x s ) R ⋅ diag ( R T β ⋅ j ( s m ) ) ⋅ R T β ⋅ l ( sy ) ∣ shows that R R R does not cancel out. As such, mediating biomarkers can be arbitrarily added or removed from the set M L V M A i l \mathcal{M}^{il}_{LVMA} M L V M A i l through likelihood-equivalent rotations of the latent space, hence the need for a rotationally invariant definition of the “mediating” manifest set of variables.
Additional synthetic experiments
We explore in Figure 7 how models perform when there is an imbalance between the dimensionality of X X X and M M M . Our results show that models perform similarly as in the balanced case, although with a notable increase in accuracy of two-step latent approaches (PCA, PLS) for latent data. Increasing J J J means the mediators carry more information about the whole system, thus alleviating PCA’s sequential estimation.
Figure 7. LaMe accurately retrieves underlying parameters — LaMe and baselines when I = 50 I = 50 I = 50 , J = 200 J = 200 J = 200 , L = 1 L = 1 L = 1 , and a signal-to-noise ratio from S S S to Y Y Y and M M M equal to 0.1. (Top ) Mean absolute error over all parameters in β i n f ( x m ) \beta_{\mathrm{inf}}^{(xm)} β inf ( x m ) and β t r u e ( x m ) \beta_{\mathrm{true}}^{(xm)} β true ( x m ) ; (Bottom ) Mean absolute error over all parameters β i n f ( m y ) \beta_{\mathrm{inf}}^{(my)} β inf ( m y ) and β t r u e ( m y ) \beta_{\mathrm{true}}^{(my)} β true ( m y ) ; (left ) data are generated using a latent process (K = 10 K = 10 K = 10 ) with the true K K K provided to all latent models; (right ) data are generated using a non-latent process with K = min ( I + L , J ) 2 K = \frac{\min(I+L,J)}{2} K = 2 m i n ( I + L , J ) for all latent models except for OVP-LaMe where K = I + J 2 = 100 K = \frac{I+J}{2} = 100 K = 2 I + J = 100 .
In Figure 9 , we compare models against extreme signal-to-noise ratios from S S S to M M M and Y Y Y on latent data. We use cosine similarity as a normalized measure for absolute comparison. Overall, we observe a sharp decrease in accuracy for β ( m y ) \beta^{(my)} β ( m y ) once S N R ( Y ∣ S ) < 3 % SNR(Y\mid S)<3\% S N R ( Y ∣ S ) < 3% whereas β ( x m ) \beta^{(xm)} β ( x m ) estimates remain accurate for even smaller S N R ( M ∣ S ) SNR(M\mid S) S N R ( M ∣ S ) –which can be explained by a greater amount of information available due to J ≫ L J\gg L J ≫ L . Overall, LaMe still outperforms baselines and retains its accuracy for lower values of S N R ( M , Y ∣ S ) SNR(M,Y\mid S) S N R ( M , Y ∣ S ) .
Figure 9. LaMe accurately retrieves underlying parameters over a range of signal-to-noise ratios — LaMe and baselines when I = 100 I = 100 I = 100 , J = 100 J = 100 J = 100 , K = 10 K = 10 K = 10 , L = 1 L = 1 L = 1 , and a varying signal-to-noise ratio from S S S to Y Y Y and M M M . (left) Cosine similarity between β inf ( x m ) \beta_{\inf}^{(xm)} β i n f ( x m ) and \beta_{\true}^{(xm)} ; (right) Cosine similarity between β inf ( m y ) \beta_{\inf}^{(my)} β i n f ( m y ) and \beta_{\true}^{(my)} ; data are generated using a latent process (K = 10 K = 10 K = 10 ).
Figure 8 shows that LaMe’s performance is unaffected by the dimension of M M M once S N R ( M ∣ S ) SNR(M\mid S) S N R ( M ∣ S ) is fixed. We use cosine similarity as a normalized measure for absolute comparison.
Figure 8. LaMe’s parameter retrieval in challenging situations — LaMe’s cosine similarity when I = 100 I = 100 I = 100 , J = 100 J = 100 J = 100 , K = 10 K = 10 K = 10 , L = 1 L = 1 L = 1 , and S N R ( Y ∣ S ) = S N R ( M ∣ S ) = 0.1 SNR(Y|S) = SNR(M|S) = 0.1 S N R ( Y ∣ S ) = S N R ( M ∣ S ) = 0.1 , unless specified otherwise, with a varying number of observations N N N (top ) with respect to S N R ( M ∣ S ) = S N R ( Y ∣ S ) SNR(M|S) = SNR(Y|S) S N R ( M ∣ S ) = S N R ( Y ∣ S ) (bottom ) with respect to the number of mediators J J J ; a dataset is generated once for J = 400 J = 400 J = 400 , and models are trained on subsets of M M M , which can cause S N R ( Y ∣ S ) SNR(Y|S) S N R ( Y ∣ S ) , S N R ( M ∣ S ) SNR(M|S) S N R ( M ∣ S ) to fluctuate around their target value across different J J J (left ) between estimated β ( x m ) \beta^{(xm)} β ( x m ) and β t r u e ( x m ) \beta_{\mathrm{true}}^{(xm)} β true ( x m ) | (right ) between estimated β ( m y ) \beta^{(my)} β ( m y ) and β t r u e ( m y ) \beta_{\mathrm{true}}^{(my)} β true ( m y ) .
Real-world data - Standard 5-fold CV
The results for a standard 5-fold cross-validation procedure are provided in Table 2 . The data are split into 80% training data and 20% test data with no overlap between testing samples. We stress that PCA+LASSO’s overall good predictive performances are consistently achieved at the cost of high-dimensional latent mediators, whereas models such as LaMe or JAOG seem to efficiently compress the relevant information into a smaller number of latent variables.
Table 2. Out-of-sample and in-sample performance metrics of LaMe (version X and M, with and without ℓ 1 \ell1 ℓ 1 regularization, assuming full noise covariances) and all baseline models. The last lines report the average improvement of models’ metrics over OLS across all datasets –hence the null line for OLS. NLPD is the Negative Log-Predictive Density, R 2 R^2 R 2 the fraction of explained variance and AIC the Akaike Information Criterion. When applicable, the number of latent variables is reported as K K K . Errors correspond to the standard error across five folds. The colour scale indicates relative performance within each dataset and metric ; (orange=best).
Full mediation graphs
Figure 10. LaMe’s results on real-world data — (A) Schematic illustration of the dataset and of the mediated system at hand. (B,C) Results from the LaMe-vX model, assuming either the causal structural model with latent mediators with K K K chosen for the lowest BIC (B) or without latent mediators (C). Arrows represent the standardized structural coefficients β ~ ( u v ) = Σ u 1 / 2 β ( u v ) Σ v − 1 / 2 \tilde{\beta}^{(uv)} = \Sigma_{u}^{1/2}\beta^{(uv)}\Sigma_{v}^{-1/2} β ~ ( uv ) = Σ u 1/2 β ( uv ) Σ v − 1/2 ; for variables with a causal path to the outcome, squares’ colour represents the total causal effect of a unit intervention on the corresponding variable on the outcome Y Y Y under the model specified with latent mediators–in this case, the sum of all causal paths leading to Y Y Y ; in the latent graph, mediator colours represent marginal associations with Y Y Y , not causal effects. For both squares and arrows, blue represents a negative relation, red represents a positive relation, and colour intensity represents its strength with extremal values at ± 0.25 \pm0.25 ± 0.25 s.d. Annotations of (C) match those in (B). (D) Plot of the predicted values of Y Y Y and entries of M M M (the three with highest R 2 R^2 R 2 ) against their true values, along with the associated coefficient of determination R 2 R^2 R 2 .
Figure 11. Full graphs for the CRIME dataset
Figure 12. Full graphs for the NHANES dataset
Figure 13. Full graphs for the PISA-I dataset
Figure 14. Full graphs for the EPIC-CAN dataset
Real-world data labels
List of all source variables considered in the proposed real-world datasets. Not all variables are being used in the main experiments, some being removed to avoid collinearity, reduce noise and/or due to irrelevance regarding the task at hand.
PISA-I
24.1.1 X. ST003D02T = Birth month 1-12 , ST004D01T = Gender 1-2 , ST001D01T = class level , STUBMI = Body Mass Index (BMI) , MISCED = Mother’s level of education (ISCED) , FISCED = Father’s level of education (ISCED) , HISCED = Highest level of education of parents (ISCED) , PAREDINT = Index highest parental education (international years of schooling scale) , BMMJ1 = Mother’s occupational status (ISEI) based on 4-digit human coded ISCO , BFMJ2 = Father’s occupational status (ISEI) based on 4-digit human coded ISCO , HISEI = Highest parental occupational status (ISEI) based on 4-digit human coded ISCO , ESCS = Index of economic, social and cultural status , FAMSUP = Family support (WLE) , FAMSUPSL = Family support for self-directed learning (WLE) , CURSUPP = Current parental/guardian Support (WLE) , SOCCON = Social Connections: Ease of Communication About Worries and Concerns (WLE) , SO-CONPA = Social connection to Parents (WLE) , ICTRES = ICT Resources (WLE) , HOMEPOS = Home possessions (WLE) , ICTHOME = ICT availability outside of school (WLE) , ICTAVHOM = Availability and Usage of ICT at Home , ICTQUAL = Quality of access to ICT (WLE) , ICTEFFIC = Self-efficacy in digital competencies (WLE) , ICTWKDY = Frequency of ICT activity (weekday) (WLE) , ICTWKEND = Frequency of ICT activity (weekend) (WLE) , PARINVOL = Parental Involvement (WLE) , PQMIMPP = Parent Attitudes Toward Mathematics (WLE) , PQMCA R = Mathematics Career (WLE) , PAREXPT = Parent’s expectations in child’s career , ATTIMMP = Parents’ Attitudes towards Immigrants (WLE) , EXERPRAC = Exercise or practice a sport before or after school , WORK-PAY = Working for pay before or after school , WORKHOME = Working in household/take care of family members before or after school , PERS EVAGR = Perseverance (agreement) (WLE) , CURI OAGR = Curiosity (agreement) (WLE) , COOPAGR = Cooperation (agreement) (WLE) , EMPATAGR = Empathy (agreement) (WLE) , ASSERAGR = Assertiveness (agreement) (WLE) , STRESAGR = Stress resistance (agreement) (WLE) , EMOCOAGR = Emotional control (agreement) (WLE) , GROSAGR = Growth Mindset (WLE) , INFOSEEK = Information seeking about future careers (WLE) , PROBSELF = Problems with self-directed learning (WLE) , FEELLAH = Feelings about learning at home (WLE) , SDLEFF = Self-directed learning self-efficacy (WLE) , ACCESSFP = Access to money and financial products, sources of money (WLE) , FLFAMILY = Parental involvement in matters of FL (WLE) , FLCONFIN = Confidence about financial matters (WLE) , FLCONICT = Confidence about financial matters using digital devices (WLE) , ACCESSFA = Access to money and financial products, financial activities (WLE) , ATTCONFM = Attitudes towards and confidence about financial matters (WLE) , FRINFLFM = Friends’ influence on financial matters (WLE) , FCFMLRTY = Familiarity with concepts of finance , OPENART = Openness to Art and Reflection TBD (WLE) , IMAGINE = Imagination and Adventurousness TBD (WLE) , CREATHEM = Creative Home Environment (WLE) , CREAT ACT = Participation in creative activities outside of school (WLE) , CREATOPN = Creativity and Openness to Intellect (WLE) , CREATOR = Openness to creativity: Other’s report (WLE) , CREATEFF = Creative self-efficacy (WLE) , CREATFAM = Creative peers and family environment (WLE) , CREATOOS = Creative Activities outside of school (WLE) , CREATOP = Creativity and Openness to Intellect TBD (WLE) , ICTDISTR = Distress from online content and cyberbullying , ANXMAT = Mathematics Anxiety (WLE) , BODYIMA = Body Image (WLE) , LIFESAT = Students’ Life Satisfaction across Domains (WLE) , PSYCHSYM = Psychosomatic Symptoms (WLE) , EXPWB = Experienced Well-being (Previous Day) (WLE) 24.1.2 M. REPEAT = Grade repetition, MISSSC = Missing school for more than 3 months, SKIPPING = Skipping classes or days of school, TARDYSD = Arriving late for school stricter definition, DURECEC = Duration in early childhood education and care, STUDYHMW = Studying for school or homework before or after school, EXPECEDU = Highest expected educational level, RELATST = Quality of student-teacher relationships (WLE), BELONG = Sense of belonging (WLE), BULLIED = Being bullied (WLE), FEELSAFE = Feeling safe (WLE), SCHRISK = School safety risks (WLE), SCHSUST = School actions/activities to sustain learning (WLE), LEARRES = Types of learning resources used while school was closed (WLE), ICTSCH = ICT availability at school (WLE), PQSCHOOL = School quality (WLE), ICTAVSCH = Availability and Usage of ICT at School, PASCHPOL = School Policies for Parental Involvement (WLE), MATHPREF = Prefers maths, MATHEASE = Ease at maths, MATHMOT = Motivation in maths, CREATSCH = Creative school and class environment (WLE), CREATAS = Creative Activities at school (WLE), ICTSUBJ = Subject-related ICT Use During Lessons (WLE), ICTENQ = Use of ICT in enquiry-based learning activities (WLE), ICTFEED = Support or feedback via ICT (WLE), ICTOUT = Use of ICT for school activities outside of the classroom (WLE), ICTREG = Views of regulated ICT use in school (WLE), ICTINFO = Students’ practices regarding online information (WLE), FLSCHOOL = Financial education in school lessons (WLE), FLMULTSB = Financial education in school lessons, multiple subjects (WLE), COGACRCO = Cognitive activation in mathematics: Foster reasoning Version B (WLE), COGACMCO = Cognitive activation in mathematics: Encourage mathematical thinking Version B (WLE), EXPOFA = Exposure to Formal and Applied Mathematics Tasks (WLE), EXPO21ST = Exposure to Mathematical Reasoning and 21st century mathematics tasks (WLE), MATHEFF = Mathematics self-efficacy: formal and applied mathematics - response options reversed in 2022 (WLE), MATHEF21 = Mathematics self-efficacy: mathematical reasoning and 21st century skills (WLE), FAMCON = Subjective familiarity with mathematics concepts (WLE), MATHPERS = Effort and Persistence in Mathematics (WLE), TEACHSUP = Mathematics Teacher Support (WLE), DISCLIM = Disciplinary climate in mathematics (WLE)
24.1.3 Y. MATH = Mathematics proficiency score
NHANES
24.2.1 X. RIAGENDR = Gender, RIDAGEYR = Age at screening, DMDEDUC2 = Education level [1-5], AUQ054 = General condition of hearing (audiometry) [1-6] from [excellent-deaf], ALQ130 = Avg number alcoholic drinks/day/past 12 mos [1-15], ALQ280 = Number times 8+ drinks in 1 day/past 12 mos, DEQ034A = Stay in the shade when sunny outside [1-5] from [always-never] in the shadows, DEQ034C = Wear a long sleeved shirt [1-5] from [always-never], DEQ034D = Use sunscreen [1-5] from [always-never], ECD070A = Weight at birth, pounds, HUQ010 = General health condition [1-5] from [good-poor], HOD051 = Number of rooms in home, INDFMMPI = Family monthly poverty level index, KIQ481 = How many times wake up and urinate?, OCQ180 = Number of hours worked in the last week, OHQ845 = Rate the health of your teeth and gums [1-5] from [good-poor], PAD680 = Minutes sedentary activity/day (sitting at school, work, bus, etc), RXQ050 = Number of prescription medicines taken last 30 days [1-5], SLD012 = Sleep hours - weekdays or workdays, SLD013 = Sleep hours - weekends, SMQ020 = Smoked at least 100 cigarettes in life, SMD460 = Number of people who live here smoke tobacco? [0-2], WHD010 = Current self-reported height (inches) [40-83], WHD020 = Current self-reported weight (pounds) [60-530], WHD050 = Self-reported weight - 1 yr ago (pounds) [60-530], FNQ450 = >17yo - Difficulty communicating [1-4] from [no difficulty-cannot at all], FNQ460 = >17yo - Difficulty remembering things/concentrating [1-4] from [no difficulty-cannot at all], FNQ530 = >17yo - How often very sad/depressed [1-5] from [daily-never]
24.2.2 M. LBXAGP = alpha-1-acid glycoprotein (g/L), LBXTC = Total Cholesterol (mg/dL), LBXFER = Ferritin(ng/mL), LBDFOL = RBC folate (ng/mL), LBXHSCRP = HS C-Reactive Protein (mg/L) - Linked to risk of CVD, LBXIN = Insulin (uU/mL), LBXBPPB = Blood lead (ug/dL), LBXBCD = Blood cadmium (ug/L), LBXTHG = Blood mercury, total (ug/L), LBXBSE = Blood selenium (ug/L), LBXBMN = Blood manganese (ug/L), LBXSF1SI = 5-Methyl-tetrahydrofolate (nmol/L), LBXSF2SI = Folic acid (nmol/L), LBXSF3SI = 5-Formyl-tetrahydrofolate (nmol/L), LBXSF4SI = Tetrahydrofolate (nmol/L), LBXSF5SI = 5,10-Methenyl-tetrahydrofolate (nmol/L), LBXSF6SI = Mefox oxidation product (nmol/L), LBX17H = 17alpha-hydroxyprogesterone (ng/dL), LBXAND = Androstenedione (ng/dL), LBXAMH = Anti-Mullerian hormone (ng/mL), LBXDH E = DHEAS (µg/dL), LBXEST = Estradiol (pg/mL), LBXESO = Estrone (ng/dL), LBXFSH = Follicle Stimulating Hormone (mIU/mL), LBXL UH = Luteinizing Hormone (mIU/mL), LBXPG4 = Progesterone (ng/dL), LBXTST = Testosterone, total (ng/dL), LBXSHBG = SHBG (nmol/L), LBXTFR = Transferrin receptor (mg/L), LBXVIDMS = 25OHD2+25OHD3 (nmol/L), LBXHA = Hepatitis A antibody [1,2] for [positive,negative], LBXHBS = Hepatitis B surface antibody [1,2] for [positive,negative], LBXGLU = Fasting Blood/Plasma Glucose (mg/dL)
24.2.3 Y. LBXGH = Glycohemoglobin (%) - Glucose in blood (%)
EPIC-CAN
Acronyms: whr: waist to height ratio, rmed: Mediterranean diet, alc: alcohol, _c: category, pa: physical activity, aggr: aggregated, qe_us: USDA, hli: Healthy Lifestyle Index
24.3.1 X. age , sex, whr_c , height_c , weight_c , waist_c , bmi_c , l_school , mrmed_score , mmed_score_sx , rmed_score , mrmed_score , pa_mets , pa_index , alc_re , alc_re_c , alc_lifetime , alc_pattern , dur_smok , dur_smok_c , n_cigret_aggr , n_cigret_lifetime , smoke_intensity , pack_years , qe_us_alc , qe_us_chocdf , qe_us_energy , qe_us_fat , qe_us_fibtg , qe_us_procnt , hli_dietscore , hli_pamets , hli_smoke , hli_alcohol , hli_bmic , hli_score , systol , diastol
24.3.2 M. Protein identifiers
24.3.3 Y. Age at cancer event
CRIME
24.4.1 X. population population for community, householdsize mean people per household, racepctblack percentage of population that is african american, racePctWhite percentage of population that is caucasian, racePctAsian percentage of population that is of asian heritage, racePctHisp percentage of population that is of hispanic heritage, agePct12t21 percentage of population that is 12-21 in age, agePct12t29 percentage of population that is 12-29 in age, agePct16t24 percentage of population that is 16-24 in age, agePct65up percentage of population that is 65 and over in age, numbUrban number of people living in areas classified as urban, pctUrban percentage of people living in areas classified as urban, PctForeignBorn percent of people foreign born, PctBornSameState percent of people born in the same state as currently living, PctSameHouse85 percent of people living in the same house as in 1985 (5 years before), PctSameCity85 percent of people living in the same city as in 1985 (5 years before), PctSameState85 percent of people living in the same state as in 1985 (5 years before), MalePctDivorce percentage of males who are divorced, MalePctNevMarr percentage of males who have never married, FemalePctDiv percentage of females who are divorced, TotalPctDiv percentage of population who are divorced, PersPerFam mean number of people per family, PctFam2Par percentage of families (with kids) that are headed by two parents, PctKids2Par percentage of kids in family housing with two parents, PctYoungKids2Par percent of kids 4 and under in two parent households, PctTeen2Par percent of kids age 12-17 in two parent households, PctWorkMomYoungKids percentage of moms of kids 6 and under in labor force, PctWorkMom percentage of moms of kids under 18 in labor force, NumIlleg number of kids born to never married, NumImmig total number of people known to be foreign born, PctImmigRecent percentage of immigrants who immigrated within last 3 years, PctImmigRec5 percentage of immigrants who immigrated within last 5 years, PctImmigRec8 percentage of immigrants who immigrated within last 8 years, PctImmigRec10 percentage of immigrants who immigrated within last 10 years, PctRecentImmig percent of population who have immigrated within the last 3 years, PctRecImmig5 percent of population who have immigrated within the last 5 years, PctRecImmig8 percent of population who have immigrated within the last 8 years, PctRecImmig10 percent of population who have immigrated within the last 10 years, PctSpeakEnglOnly percent of people who speak only English, PctNotSpeakEnglWell percent of people who do not speak English well, PctLess9thGrade percentage of people 25 and over with less than a 9th grade education, PctNotHSGrad percentage of people 25 and over that are not high school graduates, PctBSorMore percentage of people 25 and over with a bachelor’s degree or higher education, medIncome median household income, pctWWage percentage of households with wage or salary income in 1989, pctWFarmSelf percentage of households with farm or self employment income in 1989, pctWInvInc percentage of households with investment / rent income in 1989, pctWSocSec percentage of households with social security income in 1989, pctWPubAsst percentage of households with public assistance income in 1989, pctWRetire percentage of households with retirement income in 1989, medFamInc median family income (differs from household income for non-family households), perCapInc per capita income, whitePerCap per capita income for caucasians, blackPerCap per capita income for african americans, indianPerCap per capita income for native americans, AsianPerCap per capita income for people with asian heritage, OtherPerCap per capita income for people with ’other’ heritage, HispPerCap per capita income for people with hispanic heritage, NumUnderPov number of people under the poverty level, PctPopUnderPov percentage of people under the poverty level, PctUnemployed percentage of people 16 and over, in the labor force, and unemployed, PctEmploy percentage of people 16 and over who are employed, PctEmplManu percentage of people 16 and over who are employed in manufacturing, PctEmplProfServ percentage of people 16 and over who are employed in professional services, PctOccupManu percentage of people 16 and over who are employed in manufacturing, PctOccupMgmtProf percentage of people 16 and over who are employed in management or professional occupations
24.4.2 M. LemasSwornFT number of sworn full time police officers, LemasSwFTPerPop sworn full time police officers per 100K population, LemasSwFTFieldOps number of sworn full time police officers in field operations (on the street as opposed to administrative etc), LemasSwFTFieldPerPop sworn full time police officers in field operations (on the street as opposed to administrative etc) per 100K population , LemasTotalReq total requests for police , LemasTotReqPerPop total requests for police per 100K population , PolicReqPerOffic total requests for police per police officer , PolicPerPop police officers per 100K population , RacialMatchCommPol a measure of the racial match between the community and the police force. High values indicate proportions in community and police force are similar , PctPolicWhite percent of police that are caucasian , PctPolicBlack percent of police that are african american , PctPolicHisp percent of police that are hispanic , PctPolicAsian percent of police that are asian , PctPolicMinor percent of police that are minority of any kind , OfficAssgnDrugUnits number of officers assigned to special drug units , NumKindsDrugsSeiz number of different kinds of drugs seized , PolicAveOTWorked police average overtime worked , PolicCars number of police cars , PolicOperBudg police operating budget , LemasPctPolicOnPatr percent of sworn full time police officers on patrol , LemasGangUnitDeploy gang unit deployed (numeric - decimal - but really ordinal - 0 means NO, 1 means YES, 0.5 means Part Time) , LemasPctOfficDrugUn percent of officers assigned to drug units , PolicBudgPerPop police operating budget per population , PctLargHouseFam percent of family households that are large (6 or more) , PctLargHouseOccup percent of all occupied households that are large (6 or more people) , PersPerOccupHous mean persons per household , PersPerOwnOccHous mean persons per owner occupied household , PersPerRentOccHous mean persons per rental household , PctPersOwnOccup percent of people in owner occupied households , PctPersDenseHous percent of persons in dense housing (more than 1 person per room) , PctHousLess3BR percent of housing units with less than 3 bedrooms , MedNumBR median number of bedrooms , HousVacant number of vacant households , PctHousOccup percent of housing occupied , PctHousOwnOcc percent of households owner occupied , PctVacantBoarded percent of vacant housing that is boarded up , PctVacMore6Mos percent of vacant housing that has been vacant more than 6 months , MedYrHousBuilt median year housing units built , PctHousNoPhone percent of occupied housing units without phone (in 1990, this was rare!) , PctWOFullPlumb percent of housing without complete plumbing facilities , OwnOccLowQuart owner occupied housing - lower quartile value , OwnOccMedVal owner occupied housing - median value , OwnOccHiQuart owner occupied housing - upper quartile value , RentLowQ rental housing - lower quartile rent , RentMedian rental housing - median rent (Census variable H32B from file STF1A) , RentHighQ rental housing - upper quartile rent , MedRent median gross rent (Census variable H43A from file STF3A - includes utilities) , MedRentPctHousInc median gross rent as a percentage of household income , MedOwnCostPctInc median owners cost as a percentage of household income - for owners with a mortgage , MedOwnCostPctIncNoMtg median owners cost as a percentage of household income - for owners without a mortgage , NumInShelters number of people in homeless shelters , NumStreet number of homeless people counted in the street , LandArea land area in square miles , PopDens population density in persons per square mile , PctUsePubTrans percent of people using public transit for commuting
Y. ViolentCrimesPerPop
ViolentCrimesPerPop total number of violent crimes per 100K population GOAL attribute (to be predicted)