betaFeedback

WorksMathematics

Deep Mixture Models: A Statistical Review with R

Abstract

Deep mixture models extend finite mixture models through hierarchical latent factor structures with shared mixture parameters. This focused review examines deep Gaussian mixture models (DGMMs), robust deep mixture models (RDMMs), deep skew-tt mixture models (DStMMs), and mixed deep Gaussian mixture models (MDGMMs) within a common statistical framework. We compare their pathway distributions, measurement models, and latent architectures, highlighting the implications for likelihood evaluation and inference. For continuous measurements, we discuss Gaussian conditional calculations and the scale-mixture methods underlying Student-tt and skew-tt pathways. For mixed measurements, we consider response-specific likelihoods and numerical integration for discrete variables. Alongside the review, we introduce MixedDeepMM, an R package providing a unified interface for fitting and evaluating deep mixture models with Gaussian, Student-tt, and skew-tt kernels, supporting continuous and mixed measurements and single- and two-head architectures. We illustrate its use on the UCI Wine and Cryotherapy datasets, reporting clustering measures with standard errors across repeated fits and visualising consensus partitions. For mixed data, Gower silhouette widths complement external clustering agreement measures. We conclude by discussing computational challenges, model selection, and uncertainty assessment.

Introduction

Finite mixture models are widely used for density estimation, classification, and modelling population heterogeneity [1, 2, 3, 4]. In high dimensions, however, unrestricted component covariances can be costly: a Gaussian mixture with kk components and pp variables requires kp(p+1)/2kp(p + 1)/2 covariance parameters alone. Parsimonious covariance models and mixtures of factor analyzers (MFAs) address this problem by representing within-component dependence through lower-dimensional latent factors [5, 6, 7, 8].

Hierarchical factor mixtures extend MFAs by introducing successive mixture layers between the observations and deeper latent factors. Deep mixtures of factor analyzers and deep Gaussian mixture models (DGMMs) yield flexible marginal densities while sharing layer-specific parameters across latent pathways [9, 10]. The hierarchy supports density estimation, clustering, and lower-dimensional representation; variational and sparse extensions address high-dimensional applications [11]. Two directions broaden this framework. For continuous data, robust deep mixture models (RDMMs) introduce a shared scale variable to produce Student-tt pathways, while deep skew-tt mixture models (DStMMs) also accommodate asymmetry [12, 13, 14]. For heterogeneous data, mixed deep Gaussian mixture models (MDGMMs) combine latent Gaussian factors with measurement distributions for binary, ordinal, nominal, and count responses, using either a common latent head or separate continuous and discrete heads [15, 16, 17, 18]. Although these models share a factor-mixture foundation, they differ in pathway distributions, observation models, and computational requirements. Figure 1 provides a schematic overview of these model families, highlighting their distinctions in response types, pathway kernels, and latent architectures.

Hierarchical factor-mixture model relationships

Figure 1. Schematic relationships among the reviewed deep mixture specifications. Pathway kernels govern tail behaviour and asymmetry; mixed-response formulations also distinguish the common-head and two-head latent architectures.

This focused review compares DGMMs, RDMMs, DStMMs, and MDGMMs using common notation for latent pathways, observed likelihoods, and parameter estimation. We also show how response type, pathway kernel, and latent architecture fit within a single statistical formulation. The MixedDeepMM R package illustrates the corresponding model specifications through the UCI Wine and UCI Cryotherapy datasets; its main functions are summarised in Appendix § 6.

The remaining parts are organised as follows. § 2 introduces the model families, § 3 develops the common likelihood and inferential framework, and § 4 presents the data illustrations. § 5 discusses their connections and remaining challenges.

Established deep mixture models

Gaussian factor-mixture hierarchies

Let yi∈Rp\mathbf{y}_i \in\mathbb{R}^p, i=1,…,ni = 1,\ldots,n, denote independent observations. An MFA has density

f(yi)=∑a=1kπaϕp(yi;ηa,ΛaΛaT+Ψa),πa>0,∑a=1kπa=1.(1)f(\mathbf{y}_i) = \sum_{a=1}^{k} \pi_a \phi_p\left(\mathbf{y}_i; \boldsymbol{\eta}_a, \boldsymbol{\Lambda}_a\boldsymbol{\Lambda}_a^{\mathsf{T}} + \boldsymbol{\Psi}_a\right), \qquad\pi_a > 0,\qquad\sum_{a=1}^{k} \pi_a = 1. \tag*{(1)}

Here Λa\boldsymbol{\Lambda}_a is a p×qp \times q loading matrix with q<pq < p, Ψa\boldsymbol{\Psi}_a is usually a positive diagonal uniqueness matrix, and ϕp(⋅;μ,Σ)\phi_p(\cdot; \boldsymbol{\mu}, \boldsymbol{\Sigma}) denotes the multivariate normal density. Component aa admits the factor representation yi=ηa+Λazi+ϵia\mathbf{y}_i = \boldsymbol{\eta}_a + \boldsymbol{\Lambda}_a\mathbf{z}_i + \boldsymbol{\epsilon}_{ia}, where zi∼Nq(0,I)\mathbf{z}_i \sim\mathcal{N}_q(\mathbf{0}, \mathbf{I}) and ϵia∼Np(0,Ψa)\boldsymbol{\epsilon}_{ia} \sim\mathcal{N}_p(\mathbf{0}, \boldsymbol{\Psi}_a). This representation permits tractable conditional calculations even when the marginal density is multimodal.

A DGMM extends this representation to hh layers. Set zi(0)=yi\mathbf{z}_i^{(0)} = \mathbf{y}_i and r0=p>r1>⋯>rhr_0 = p > r_1 > \cdots> r_h. The pathway Si=(Si1,…,Sih)\mathbf{S}_i = (S_{i1}, \ldots, S_{ih}) records the component at each layer, with Siℓ∈{1,…,kℓ}S_{i\ell} \in\{1, \ldots, k_\ell\}. Under independent layer allocations, πs=∏ℓ=1hπsℓ(ℓ)\pi_{\mathbf{s}} = \prod_{\ell=1}^{h} \pi_{s_\ell}^{(\ell)}. Conditional on a pathway,

zi(ℓ−1)=ηsℓ(ℓ)+Λsℓ(ℓ)zi(ℓ)+ϵi,sℓ(ℓ),ϵi,sℓ(ℓ)∼Nrℓ−1(0,Ψsℓ(ℓ)),(2)\mathbf{z}_i^{(\ell-1)} = \boldsymbol{\eta}_{s_\ell}^{(\ell)} + \boldsymbol{\Lambda}_{s_\ell}^{(\ell)}\mathbf{z}_i^{(\ell)} + \boldsymbol{\epsilon}_{i,s_\ell}^{(\ell)}, \qquad\boldsymbol{\epsilon}_{i,s_\ell}^{(\ell)} \sim\mathcal{N}_{r_{\ell-1}}\left(\mathbf{0}, \boldsymbol{\Psi}_{s_\ell}^{(\ell)}\right), \tag*{(2)}

where disturbances are independent and zi(h)∼Nrh(0,I)\mathbf{z}_i^{(h)} \sim\mathcal{N}_{r_h}(\mathbf{0}, \mathbf{I}). Define As(0)=Ip\mathbf{A}_{\mathbf{s}}^{(0)} = \mathbf{I}_p and As(ℓ)=Λs1(1)⋯Λsℓ(ℓ)\mathbf{A}_{\mathbf{s}}^{(\ell)} = \boldsymbol{\Lambda}_{s_1}^{(1)} \cdots\boldsymbol{\Lambda}_{s_\ell}^{(\ell)}. Integrating through the hierarchy gives the exact pathway moments

μs=∑ℓ=1hAs(ℓ−1)ηsℓ(ℓ),Σs=As(h)As(h)T+∑ℓ=1hAs(ℓ−1)Ψsℓ(ℓ)As(ℓ−1)T.(3)\begin{aligned} \boldsymbol{\mu}_{\mathbf{s}} &= \sum_{\ell=1}^{h} \mathbf{A}_{\mathbf{s}}^{(\ell-1)} \boldsymbol{\eta}_{s_\ell}^{(\ell)},\\ \boldsymbol{\Sigma}_{\mathbf{s}} &= \mathbf{A}_{\mathbf{s}}^{(h)}\mathbf{A}_{\mathbf{s}}^{(h)\mathsf{T}} + \sum_{\ell=1}^{h} \mathbf{A}_{\mathbf{s}}^{(\ell-1)}\boldsymbol{\Psi}_{s_\ell}^{(\ell)}\mathbf{A}_{\mathbf{s}}^{(\ell-1)\mathsf{T}}. \tag*{(3)} \end{aligned}

The induced observed density is

fDGMM(y)=∑s∈Sπsϕp(y;μs,Σs),∣S∣=K=∏ℓ=1hkℓ.(4)f_{\mathrm{DGMM}}(\mathbf{y}) = \sum_{\mathbf{s}\in\mathcal{S}} \pi_{\mathbf{s}}\phi_p(\mathbf{y}; \boldsymbol{\mu}_{\mathbf{s}}, \boldsymbol{\Sigma}_{\mathbf{s}), \qquad|\mathcal{S}| = K = \prod_{\ell=1}^{h} k_\ell.} \tag*{(4)}

Although the hierarchy induces KK pathway densities, it shares their transition parameters. Marginalising the posterior pathway probabilities yields cluster memberships at each layer.

Robust and asymmetric continuous pathways

Normal scale mixtures connect Gaussian and heavy-tailed pathway distributions [12, 19]. Let Ui=Wi−1U_i = W_i^{-1} denote a shared precision. In an RDMM, conditional on Si=sS_i = \mathbf{s},

Ui∼gamma⁡(νs/2,νs/2),yi∣(Ui=u,s)∼Np(μs,Σs/u).(5)U_i \sim\operatorname{gamma}(\nu_{\mathbf{s}}/2,\nu_{\mathbf{s}}/2), \qquad\mathbf{y}_i \mid(U_i = u,\mathbf{s}) \sim\mathcal{N}_p(\boldsymbol{\mu}_{\mathbf{s}},\boldsymbol{\Sigma}_{\mathbf{s}}/u). \tag*{(5)}

The gamma distribution is parameterised by shape and rate. A common uu scales the deepest factor and the disturbance covariances throughout the hierarchy, yielding the marginal Student-tt density f(yi∣s)=tp(yi;μs,Σs,νs)f(\mathbf{y}_i \mid s) = t_p(\mathbf{y}_i; \boldsymbol{\mu}_s, \boldsymbol{\Sigma}_s, \nu_s), where Σs\boldsymbol{\Sigma}_s is a scale matrix. This shared precision is essential: independent layer-specific scales generally do not yield an exact multivariate Student-tt marginal. The Gaussian case is recovered as νs→∞\nu_s \to\infty [13].

DStMMs additionally represent asymmetry through the normal mean–variance construction

yi∣(Ui=u,s)∼Np(μs+αsu,Σsu),αs=∑ℓ=1hAs(ℓ)δsℓ(ℓ).(6)\mathbf{y}_i \mid(U_i=u,s) \sim\mathcal{N}_p\left(\boldsymbol{\mu}_s+\frac{\boldsymbol{\alpha}_s}{u},\frac{\boldsymbol{\Sigma}_s}{u}\right), \qquad\boldsymbol{\alpha}_s=\sum_{\ell=1}^{h}\mathbf{A}_s^{(\ell)}\boldsymbol{\delta}_{s_\ell}^{(\ell)}. \tag*{(6)}

With the gamma precision in Eq. (5), this construction yields a generalised-hyperbolic skew-tt pathway [14], one of several multivariate skew-tt parameterisations [20, 21]. Setting αs=0\boldsymbol{\alpha}_s=0 recovers the symmetric Student-tt case; for νs>2\nu_s>2, the pathway mean is μs+νsαs/(νs−2)\boldsymbol{\mu}_s+\nu_s\boldsymbol{\alpha}_s/(\nu_s-2). Related computational methods arise in generalised hyperbolic mixtures [22].

Mixed-response deep mixtures

For mixed observations, partition yi=(yiC,yiD)\mathbf{y}_i=(\mathbf{y}_i^C,\mathbf{y}_i^D) into continuous and discrete responses. Their conditional measurement distributions are linked through continuous latent factors. A generic conditional-independence representation is

p(yi∣ziC,ziD)=∏j∈Cfj(yij∣ziC)∏j∈Dgj(yij∣ziD),(7)p(\mathbf{y}_i \mid\mathbf{z}_i^C,\mathbf{z}_i^D)=\prod_{j\in\mathcal{C}}f_j(y_{ij}\mid\mathbf{z}_i^C)\prod_{j\in\mathcal{D}}g_j(y_{ij}\mid\mathbf{z}_i^D), \tag*{(7)}

where Gaussian distributions may be used for continuous variables and gjg_j is chosen according to the discrete response type [15, 23]. Typical choices include Bernoulli-logit models for binary responses, Poisson-log models for counts, threshold models for ordinal responses, and multinomial logits for nominal responses. These links allow different measurement scales to enter a joint latent-variable model.

MDGMMs use a common latent head (M1DGMM) or separate continuous and discrete heads connected by a shared deeper tail (M2DGMM) [18]. In either case, the observed likelihood can be written schematically as

p(yi)=∑sπs∫p(yi∣z,s)p(z∣s) dz.(8)p(\mathbf{y}_i)=\sum_s\pi_s\int p(\mathbf{y}_i\mid\mathbf{z},s)p(\mathbf{z}\mid s)\,d\mathbf{z}. \tag*{(8)}

The latent dimensions and integrand depend on the architecture. Unlike the Gaussian continuous-data case, response-specific discrete likelihood terms usually prevent analytic marginalisation.

Unified framework and likelihood-based inference

Model dimensions and statistical specification

The specifications can be organised along three dimensions:

D∈{continuous,mixed},F∈{Gaussian,t,skew-t},A∈{single head,two head}.\begin{aligned} D &\in\{\text{continuous},\text{mixed}\},\\ F &\in\{\text{Gaussian},t,\text{skew-}t\},\\ A &\in\{\text{single head},\text{two head}\}. \end{aligned}

The architecture dimension applies to mixed-response specifications: a single-head model uses a common latent representation, whereas a two-head model connects continuous and discrete representations through a shared tail. For continuous-only observations, the implementation uses the single-head structure.

Let Si=sS_i=s denote the mixture pathway, ZiZ_i the collection of continuous latent factors, and UiU_i an optional precision variable. Conditional on Zi=zZ_i=z, write zCz_C and zDz_D for the latent coordinates associated with the two response blocks. For continuous-only data, the discrete block yiDy_i^D is empty. Denote the family-specific precision distribution by gF(u∣s)g_F(u\mid s) and the latent hierarchy by hF(z∣u,s)h_F(z\mid u,s). For Gaussian kernels, Ui=1U_i=1; Student-tt kernels use a gamma precision, and the skew-tt construction additionally introduces a displacement proportional to Ui−1U_i^{-1}. The joint model then factorises as

p(yi,z,u,s;θ)=πsgF(u∣s)hF(z∣u,s)fC(yiC∣zC,u,s)fD(yiD∣zD,s).(9)p(y_i,z,u,s;\theta)=\pi_s g_F(u\mid s)h_F(z\mid u,s)f_C(y_i^C\mid z_C,u,s)f_D(y_i^D\mid z_D,s). \tag*{(9)}

Here fCf_C is the conditional Gaussian measurement density and fDf_D is the product of response-specific discrete measurement probabilities. In a single-head model, zCz_C and zDz_D coincide; in a two-head model, they are linked through a deeper shared factor. Marginalising the unobserved variables gives

L(θ)=∏i=1n{∑s∈Sπs∬fC(yiC∣zC,u,s)fD(yiD∣zD,s)hF(z∣u,s)gF(u∣s) dz du}.(10)L(\theta)=\prod_{i=1}^{n}\left\{\sum_{s\in\mathcal{S}}\pi_s\iint f_C(y_i^C\mid z_C,u,s)f_D(y_i^D\mid z_D,s)h_F(z\mid u,s)g_F(u\mid s)\,dz\,du\right\}. \tag*{(10)}

For Gaussian kernels, integration over uu is replaced by evaluation at u=1u=1. With continuous measurements, marginalisation produces the pathway densities in § 2; with discrete measurements, the response likelihood remains inside the latent integral and typically requires numerical approximation.

For discrete variable jj, let ηij=β0j+βjTzD\eta_{ij}=\beta_{0j}+\boldsymbol{\beta}_j^{T}z_D be a conditional linear predictor. Common links include

Pr⁡(Yij=1∣zD)=logit⁡−1(ηij),E(Yij∣zD)=exp⁡(ηij)for counts.(11)\Pr(Y_{ij}=1\mid z_D)=\operatorname{logit}^{-1}(\eta_{ij}),\qquad\mathbb{E}(Y_{ij}\mid z_D)=\exp(\eta_{ij})\quad\text{for counts}. \tag*{(11)}

Ordinal variables may use ordered cumulative-link thresholds, whereas nominal variables can use multinomial logits relative to a reference category. Identification constraints depend on the response family.

Table 1 cross-classifies the reviewed families and the corresponding mixed-response specifications. In the mixed Student-tt and skew-tt settings, response-specific measurement links are combined with heavy-tailed or asymmetric latent kernels. These combinations describe model specifications in the common framework; they are not identified with the Gaussian-only MDGMM of the original publication.

ResponseKernelArchitectureModel family
ContinuousGaussianSingle-headDGMM [10]
ContinuousStudent-ttSingle-headRDMM [13]
ContinuousSkew-ttSingle-headDStMM [14]
MixedGaussianSingle- or two-headMDGMM [18]
MixedStudent-ttSingle- or two-headRobust mixed-response
MixedSkew-ttSingle- or two-headAsymmetric mixed-response

Table 1. Model families and mixed-response specifications in the unified framework.

Estimation strategy and the EM objective

The families share an expectation–maximisation (EM) framework [24] but differ in the conditional moments required by the E-step. Let Hi=(Si,Zi,Ui)\boldsymbol{H}_i = (S_i, Z_i, U_i) collect the missing variables, with UiU_i omitted in the Gaussian case. For continuous responses, Gaussian, Student-tt, and skew-tt kernels lead to Gaussian, gamma, and generalised inverse Gaussian (GIG) calculations, respectively. Mixed-response models generally require numerical approximation, such as Monte Carlo EM (MCEM) [25], and response-specific optimisation in the M-step. The following subsections detail these differences.

Observed likelihood and pathway posterior probabilities

For the continuous model families, let qs(yi;θ)q_s(\boldsymbol{y}_i; \theta) denote the Gaussian, Student-tt, or generalised-hyperbolic skew-tt pathway density. Their log likelihood and posterior pathway probabilities are

ℓ(θ)=∑i=1nlog⁡{∑s∈Sπsqs(yi;θ)},τis=πsqs(yi;θ)∑t∈Sπtqt(yi;θ).(12)\ell(\theta) = \sum_{i=1}^{n} \log\left\{\sum_{s\in\mathcal{S}} \pi_s q_s(\boldsymbol{y}_i;\theta)\right\}, \qquad\tau_{is} = \frac{\pi_s q_s(\boldsymbol{y}_i;\theta)}{\sum_{t\in\mathcal{S}} \pi_t q_t(\boldsymbol{y}_i;\theta)}. \tag*{(12)}

For numerical stability, set ais=log⁡πs+log⁡qs(yi)a_{is} = \log\pi_s + \log q_s(\boldsymbol{y}_i) and evaluate log⁡∑sexp⁡(ais)=mi+log⁡∑sexp⁡(ais−mi)\log\sum_s \exp(a_{is}) = m_i + \log\sum_s \exp(a_{is} - m_i), where mi=max⁡saism_i = \max_s a_{is}. Marginalising the pathway responsibilities gives layer-specific weights τi,ℓ,a=∑s:sℓ=aτis\tau_{i,\ell,a} = \sum_{s:s_\ell=a} \tau_{is} and the mixing-proportion update

π^a(ℓ)=1n∑i=1nτi,ℓ,a.(13)\hat{\pi}^{(\ell)}_a = \frac{1}{n}\sum_{i=1}^{n}\tau_{i,\ell,a}. \tag*{(13)}

For the complete latent-variable model, the EM objective is

Q(θ∣θ(t))=∑i=1nEθ(t)[log⁡p(yi,Hi;θ)∣yi].(14)Q(\boldsymbol{\theta}\mid\boldsymbol{\theta}^{(t)})=\sum_{i=1}^{n}\mathbb{E}_{\boldsymbol{\theta}^{(t)}}\left[\log p(\boldsymbol{y}_i,\boldsymbol{H}_i;\boldsymbol{\theta})\mid\boldsymbol{y}_i\right]. \tag*{(14)}

The expectations in Eq. (14) depend on the pathway kernel and observation model [26].

Gaussian conditional moments and factor updates

For a fixed DGMM pathway, stack the latent factors as Zi=((zi(1))T,…,(zi(h))T)T\boldsymbol{Z}_i=((\boldsymbol{z}_i^{(1)})^{\mathsf{T}},\ldots,(\boldsymbol{z}_i^{(h)})^{\mathsf{T}})^{\mathsf{T}}. Since (yiT,ZiT)T(\boldsymbol{y}_i^{\mathsf{T}},\boldsymbol{Z}_i^{\mathsf{T}})^{\mathsf{T}} is jointly normal, the pathway-specific moments mZ,s\boldsymbol{m}_{Z,s}, CZy,sC_{Zy,s}, and CZZ,sC_{ZZ,s} give

mis=E(Zi∣yi,s)=mZ,s+CZy,sΣs−1(yi−μs),Vs=Var⁡(Zi∣yi,s)=CZZ,s−CZy,sΣs−1CyZ,s.(15)\begin{aligned} \boldsymbol{m}_{is}&=\mathbb{E}(\boldsymbol{Z}_i\mid\boldsymbol{y}_i,s)=\boldsymbol{m}_{Z,s}+C_{Zy,s}\boldsymbol{\Sigma}_s^{-1}(\boldsymbol{y}_i-\boldsymbol{\mu}_s),\\ V_s&=\operatorname{Var}(\boldsymbol{Z}_i\mid\boldsymbol{y}_i,s)=C_{ZZ,s}-C_{Zy,s}\boldsymbol{\Sigma}_s^{-1}C_{yZ,s}. \tag*{(15)} \end{aligned}

The corresponding second moment is E(ZiZiT∣yi,s)=Vs+mismisT\mathbb{E}(\boldsymbol{Z}_i\boldsymbol{Z}_i^{\mathsf{T}}\mid\boldsymbol{y}_i,s)=V_s+\boldsymbol{m}_{is}\boldsymbol{m}_{is}^{\mathsf{T}}. Unlike the collapsed pathway likelihood, updating shared factor transitions also requires posterior cross-moments between adjacent layers.

For layer ℓ\ell and component aa, define xi(ℓ)=(1,(zi(ℓ))T)T\boldsymbol{x}_i^{(\ell)}=(1,(\boldsymbol{z}_i^{(\ell)})^{\mathsf{T}})^{\mathsf{T}} and Ba(ℓ)=[ηa(ℓ),Λa(ℓ)]B_a^{(\ell)}=[\boldsymbol{\eta}_a^{(\ell)},\Lambda_a^{(\ell)}]. Let Eis(⋅)E_{is}(\cdot) denote expectation conditional on (yi,Si=s)(\boldsymbol{y}_i,S_i=s) at the current parameter values. Weighted least squares yields

B^a(ℓ)={∑i,s:sℓ=aτisEis(zi(ℓ−1)xi(ℓ)T)}{∑i,s:sℓ=aτisEis(xi(ℓ)xi(ℓ)T)}−1.(16)\hat{B}_a^{(\ell)}= \left\{\sum_{i,s:s_\ell=a}\tau_{is}E_{is}\left(\boldsymbol{z}_i^{(\ell-1)}\boldsymbol{x}_i^{(\ell)\mathsf{T}}\right)\right\} \left\{\sum_{i,s:s_\ell=a}\tau_{is}E_{is}\left(\boldsymbol{x}_i^{(\ell)}\boldsymbol{x}_i^{(\ell)\mathsf{T}}\right)\right\}^{-1}. \tag*{(16)}

The diagonal uniqueness update is the diagonal of the weighted expected residual cross-product, divided by ∑i,s:sℓ=aτis\sum_{i,s:s_\ell=a}\tau_{is}. Loading or uniqueness restrictions require constrained maximisation. Gaussian conditioning or smoothing provides the necessary moments, although ill-conditioned factor blocks may require regularisation.

Shared-scale inference for robust pathways

Let dis=(yi−μs)TΣs−1(yi−μs)d_{is}=(\boldsymbol{y}_i-\boldsymbol{\mu}_s)^{\mathsf{T}}\boldsymbol{\Sigma}_s^{-1}(\boldsymbol{y}_i-\boldsymbol{\mu}_s) be the squared Mahalanobis distance. The normal–gamma representation in Eq. (5) gives

Ui∣(yi,s)∼gamma⁡(νs+p2,νs+dis2),E(Ui∣yi,s)=νs+pνs+dis.(17)U_i\mid(\boldsymbol{y}_i,s)\sim\operatorname{gamma}\left(\frac{\nu_s+p}{2},\frac{\nu_s+d_{is}}{2}\right),\qquad\mathbb{E}(U_i\mid\boldsymbol{y}_i,s)=\frac{\nu_s+p}{\nu_s+d_{is}}. \tag*{(17)}

The corresponding log-precision moment is

E(log⁡Ui∣yi,s)=ψ(νs+p2)−log⁡(νs+dis2).(18)\mathbb{E}(\log U_i\mid\boldsymbol{y}_i,s)=\psi\left(\frac{\nu_s+p}{2}\right)-\log\left(\frac{\nu_s+d_{is}}{2}\right). \tag*{(18)}

Here ψ\psi denotes the digamma function. For the factor-transition updates, the Gaussian sufficient statistics in Eq. (16) become Eis(Uizi(ℓ−1)xi(ℓ)T)\mathbb{E}_{is}(U_i z_i^{(\ell-1)} x_i^{(\ell)\mathsf{T}}) and Eis(Uixi(ℓ)xi(ℓ)T)\mathbb{E}_{is}(U_i x_i^{(\ell)} x_i^{(\ell)\mathsf{T}}). The weighted normal-equation structure is retained, but its moments are evaluated under the joint latent posterior.

Maximising the expected gamma log density gives the following stationarity equation for a degrees-of-freedom parameter ν\nu shared across a specified set of pathways:

log⁡(ν/2)−ψ(ν/2)+1+∑i,sτis{Eis(log⁡Ui)−Eis(Ui)}∑i,sτis=0.(19)\log(\nu/2)-\psi(\nu/2)+1+\frac{\sum_{i,s}\tau_{is}\{\mathbb{E}_{is}(\log U_i)-\mathbb{E}_{is}(U_i)\}}{\sum_{i,s}\tau_{is}}=0. \tag*{(19)}

The sums in Eq. (19) cover the pathways sharing ν\nu. Bounded scalar optimisation is useful when effective component sizes are small. Observations with large Mahalanobis distances receive lower expected precisions, which underlies the robust-mixture behaviour [12, 27]. Estimation of the full RDMM hierarchy also involves stochastic or Monte Carlo steps [13]. Although the general model permits pathway-specific νs\nu_s, MixedDeepMM estimates one common ν\nu across pathways for parsimony and stability.

Generalised inverse Gaussian computations for skewed pathways

The skew-tt construction in Eq. (6) is conditionally Gaussian, but the posterior precision is no longer gamma. Define

χs=αsTΣs−1αs,ωis=νs+dis,λs=νs+p2.(20)\chi_s=\boldsymbol{\alpha}_s^{\mathsf{T}}\boldsymbol{\Sigma}_s^{-1}\boldsymbol{\alpha}_s,\qquad\omega_{is}=\nu_s+d_{is},\qquad\lambda_s=\frac{\nu_s+p}{2}. \tag*{(20)}

Combining the conditional Gaussian likelihood with the gamma prior gives

p(u∣yi,s)∝uλs−1exp⁡{−12(ωisu+χsu)},u>0.(21)p(u\mid\boldsymbol{y}_i,s)\propto u^{\lambda_s-1}\exp\left\{-\frac{1}{2}\left(\omega_{is}u+\frac{\chi_s}{u}\right)\right\},\qquad u>0. \tag*{(21)}

Thus Ui∣(yi,s)∼GIG⁡(λs,χs,ωis)U_i\mid(\boldsymbol{y}_i,s)\sim\operatorname{GIG}(\lambda_s,\chi_s,\omega_{is}), with the parameterisation defined by Eq. (21). Its conditional power moments are

E(Uir∣yi,s)=(χsωis)r/2Kλs+r(χsωis)Kλs(χsωis),χs>0.(22)\mathbb{E}(U_i^r\mid\boldsymbol{y}_i,s)=\left(\frac{\chi_s}{\omega_{is}}\right)^{r/2}\frac{K_{\lambda_s+r}\left(\sqrt{\chi_s\omega_{is}}\right)}{K_{\lambda_s}\left(\sqrt{\chi_s\omega_{is}}\right)},\qquad\chi_s>0. \tag*{(22)}

Here KλK_\lambda denotes the modified Bessel function of the second kind. The moments for r=1r=1 and r=−1r=-1, together with E(log⁡Ui∣yi,s)\mathbb{E}(\log U_i\mid\boldsymbol{y}_i,s), enter parameter estimation. When χs=0\chi_s=0, the posterior reduces to gamma; numerical implementations should use this limit rather than a singular Bessel ratio. Log-scale evaluation or specialised GIG routines improve stability [14, 22].

The Ui−1U_i^{-1} displacement introduces additional scale-weighted cross-moments in transition updates. Moreover, different layer-wise displacements may yield the same collapsed αs\alpha_s, making constraints and initialisation important. Consequently, skew-tt estimation cannot be obtained by merely appending a skewness coefficient to the Gaussian EM updates.

Latent integration for mixed measurements

For a mixed-response pathway, the latent posterior is

p(z∣yi,s)∝p(yi∣z,s)p(z∣s).(23)p(z \mid y_i, s) \propto p(y_i \mid z, s)p(z \mid s). \tag*{(23)}

Discrete measurement terms generally preclude the Gaussian conditioning result in Eq. (15). The latent integrals can instead be evaluated numerically or approximated by importance sampling, posterior simulation, Laplace approximation, or stochastic EM. For example, drawing zim∼qi(z)z_{im} \sim q_i(z) gives the self-normalised importance approximation

E^{h(Zi)∣yi,s}=∑m=1Mwimsh(zim)∑m=1Mwims,wims=p(yi∣zim,s)p(zim∣s)qi(zim).(24)\widehat{\mathbb{E}}\{h(Z_i) \mid y_i, s\} = \frac{\sum_{m=1}^{M} w_{ims}h(z_{im})}{\sum_{m=1}^{M} w_{ims}}, \qquad w_{ims} = \frac{p(y_i \mid z_{im},s)p(z_{im} \mid s)}{q_i(z_{im})}. \tag*{(24)}

MCEM and related stochastic methods offer alternative approximations to the E-step [18, 25, 28]. For mixed Student-tt and skew-tt kernels, the latent integration also includes the precision variable.

Clustering assessment and model-selection considerations

A cluster assignment must be associated with a specified level of the hierarchy: a complete pathway, an individual latent mixture layer, or the shared tail of a two-head architecture. Once this level is fixed, external class agreement can be assessed by the adjusted Rand index (ARI) [29] and the misclassification rate (MR) after optimal label matching. Neither measure is used to fit the unsupervised models in the examples below.

For mixed observations, we additionally use the average Gower silhouette width. Gower dissimilarity accommodates variable-specific measurement types: continuous and count variables contribute range-scaled numerical differences, binary and nominal variables contribute categorical mismatches, and ordinal variables retain their ordering. Given a partition and the predictor-based Gower dissimilarities, let a(i)a(i) denote the average dissimilarity of observation ii to its assigned cluster, and let b(i)b(i) denote the smallest average dissimilarity to another cluster. Its silhouette width is

s(i)=b(i)−a(i)max⁡{a(i),b(i)}ands‾=1n∑i=1ns(i).(25)s(i) = \frac{b(i) - a(i)}{\max\{a(i), b(i)\}} \qquad\text{and} \qquad\overline{s} = \frac{1}{n}\sum_{i=1}^{n}s(i). \tag*{(25)}

Larger values indicate greater separation under the specified Gower dissimilarity. The same predictor-based distance is used to evaluate each Cryotherapy partition; the withheld outcome labels are not part of the distance calculation. The silhouette is undefined when a fitted partition contains fewer than two occupied clusters.

Likelihood-based criteria provide a complementary summary of model fit. In particular, BIC =−2ℓ+dlog⁡n= -2\ell+ d\log n, where dd is the number of estimated parameters [30]. The applications report BIC descriptively, without treating it as a direct measure of clustering agreement. Automatic selection of component numbers, factor dimensions, and head architecture is left for future investigation.

Illustrative analyses of two real datasets

Two publicly available UCI datasets are used: Wine, with continuous chemical measurements, and Cryotherapy, with continuous, binary, nominal, and count predictors. All response or class labels are withheld from fitting and retained solely for external evaluation. The analyses focus on ARI and MR, supplemented by a Gower silhouette for Cryotherapy and BIC as a secondary likelihood-based measure.

For each configuration, R=20R = 20 fits are conducted with one initialisation per seed. Results are summarised by the run mean mˉ=R−1∑r=1Rmr\bar{m} = R^{-1}\sum_{r=1}^{R}m_r and its standard error SE⁡(mˉ)=sd⁡(m1,…,mR)/R\operatorname{SE}(\bar{m}) = \operatorname{sd}(m_1,\ldots,m_R)/\sqrt{R}. Reported standard errors therefore quantify variability across initialisations on a fixed dataset; they are not bootstrap standard errors for population-level inference. Each model’s visualised partition is constructed from the RR fitted partitions using the co-assignment frequencies

Cij=1R∑r=1RI{gi(r)=gj(r)},(26)C_{ij} = \frac{1}{R}\sum_{r=1}^{R}\mathbb{I}\left\{g_i^{(r)} = g_j^{(r)}\right\}, \tag*{(26)}

where gi(r)g_i^{(r)} is the cluster label assigned to observation ii in run rr. Average-linkage hierarchical clustering of 1−Cij1-C_{ij}, cut at the prespecified number of clusters, gives a consensus partition. This construction is invariant to permutations of the cluster labels across runs. The consensus partition is used for visualisation, while the tables summarise the individual runs; its agreement measures need not equal the corresponding run averages.

Continuous measurements: UCI Wine

The UCI Wine dataset contains 178 observations from three wine classes, measured on 13 continuous chemical variables. All predictors are standardised to zero mean and unit sample standard deviation. The wine-class labels are excluded from estimation and are used only to assess the fitted clusters. We consider the Gaussian DGMM, Student-tt RDMM, and skew-tt DStMM with a two-layer, single-head structure, component counts k=(3,2)k = (3, 2), and latent dimensions r=(4,2)r = (4, 2). For the Student-tt and skew-tt fits, a common degrees-of-freedom parameter ν\nu is estimated, starting from ν=8\nu= 8. Clustering is evaluated at the first mixture layer using maximum-posterior-probability assignments. All runs produced three occupied clusters.

Table 2 shows that the run-averaged ARI and MR values for DGMM and RDMM do not vary across repeated fits to the reported precision. The DStMM results show some variability between runs. Figure 2 displays the associated consensus partitions in the first two principal components; the graphical agreement with the observed classes is descriptive and does not replace the external evaluation measures. The likelihood-based values are included to document fit characteristics, rather than to determine the ordering of models by clustering performance.

ModelARIMRBICν^\hat{\nu}
DGMM0.9149 ±\pm 0.00000.0281 ±\pm 0.00005652.31 ±\pm 0.78—
RDMM0.9832 ±\pm 0.00000.0056 ±\pm 0.00005538.40 ±\pm 0.0020.81 ±\pm 0.00
DStMM0.9583 ±\pm 0.00410.0140 ±\pm 0.00145562.77 ±\pm 1.8019.51 ±\pm 0.14

Table 2. UCI Wine clustering results (mean ±\pm SE). ARI and MR compare predicted clusters with the withheld class labels.

Figure 2. Principal component projection of the standardised UCI Wine predictors. Panel (A) shows the observed wine classes; panels (B)–(D) show the DGMM, RDMM, and DStMM consensus partitions constructed from the fitted partitions. All panels use identical PCA coordinates and scales. Cluster identifiers and colours are specific to each partition.

Mixed measurements: UCI Cryotherapy

The UCI Cryotherapy dataset contains 90 observations and a binary treatment-outcome label. Its six predictors include three continuous variables (age, treatment time, and affected area), one count variable (number of warts), one binary variable (sex), and one nominal variable (wart type). Continuous predictors are standardised; discrete predictors retain their respective measurement types. The treatment outcome is withheld from fitting and from the Gower distance calculation.

We compare Gaussian, Student-tt, and skew-tt kernels under single-head (Single) and two-head (Double) architectures, retaining the shorthand DGMM, RDMM, and DStMM for the respective kernel families. The single-head configurations use k=(2,2)k=(2,2) and r=(2,1)r=(2,1), with clustering evaluated at the first common mixture layer. The two-head configurations use k=(2,2,2)k=(2,2,2), two-dimensional continuous and discrete heads, and a one-dimensional shared tail; clustering is evaluated at the shared-tail layer. All configurations yielded two occupied clusters. Table 3 reports run-averaged Gower silhouette widths, ARI, MR, BIC, and estimated degrees of freedom.

ModelGower silhouetteARIMRBICν^\hat{\nu}
DGMM (Single)0.1749 ±\pm 0.01220.0463 ±\pm 0.01370.4033 ±\pm 0.01531484.17 ±\pm 5.04—
RDMM (Single)0.1597 ±\pm 0.00290.0560 ±\pm 0.01050.3822 ±\pm 0.01181545.19 ±\pm 0.963.12 ±\pm 0.01
DStMM (Single)0.1595 ±\pm 0.00330.0758 ±\pm 0.01340.3639 ±\pm 0.01231548.87 ±\pm 1.103.37 ±\pm 0.02
DGMM (Double)0.2441 ±\pm 0.00380.1191 ±\pm 0.00560.3233 ±\pm 0.00381508.85 ±\pm 1.67—
RDMM (Double)0.2395 ±\pm 0.00390.1129 ±\pm 0.00780.3289 ±\pm 0.00571509.59 ±\pm 1.7919.26 ±\pm 2.00
DStMM (Double)0.1967 ±\pm 0.00280.2646 ±\pm 0.01930.2428 ±\pm 0.01011540.80 ±\pm 2.355.43 ±\pm 0.14

Table 3. UCI Cryotherapy clustering results (mean ±\pm SE). The Gower silhouette is computed from the predictors alone; ARI and MR use the withheld treatment outcomes.

In Table 3, the two-head architecture is associated with higher mean Gower silhouette widths and ARI values than its single-head counterpart for each of the three kernel families. The DGMM and RDMM two-head silhouettes are similar, whereas the DStMM two-head partition shows a different relationship between internal separation and external class agreement. The values illustrate that the Gower silhouette and agreement with withheld outcomes describe different aspects of the fitted partition.

Figure 3 further shows differences in the spatial distribution of the consensus clusters. The DGMM (Single) consensus contains 87 and 3 observations, compared with 51 and 39 in the DStMM (Double) consensus. These are properties of the consensus partitions, not run averages. Together, the results illustrate the sensitivity of mixed-response clustering to the latent architecture and pathway kernel without implying that a single criterion provides a complete assessment.

Figure 3. Gower principal coordinates analysis (PCoA) of the Cryotherapy predictors, showing the observed treatment-outcome classes and six model-specific consensus partitions. Pairwise co-assignment frequencies define each consensus partition. All seven scatter panels use the same predictor-based Gower PCoA coordinates and axis ranges; the eighth panel contains the legend. Colours denote the two classes or the two predicted clusters, whose numerical identifiers are arbitrary.

Discussion

The families considered in this review share a hierarchical factor-mixture basis but differ in their assumptions about the latent pathway distributions and their observation models. Gaussian hierarchies support direct conditional calculations; Student-tt and skew-tt pathways require additional precision-related expectations; and mixed

measurements introduce response-specific likelihoods that typically require numerical integration. A unified notation makes these distinctions explicit without treating the Gaussian, robust, asymmetric, and mixed-response specifications as interchangeable. The Wine and Cryotherapy analyses illustrate complementary aspects of model evaluation. In the continuous-data example, the three fitted model families produce three-group partitions that can be compared directly against withheld class labels. For mixed measurements, the Gower silhouette provides an additional predictor-based description of cluster separation, while the ARI and MR evaluate agreement with an external treatment outcome. The two quantities need not vary together, and the consensus visualisations should not be conflated with mean metrics across individual fits. The observed differences are descriptive findings for the chosen datasets and configurations, not general rankings of the kernel families. Several problems remain open. Inference for mixed observations can be computationally demanding, especially when non-Gaussian pathway kernels introduce further latent variables. Parameter constraints, component occupancy, and stability across initialisations warrant careful attention. Future work will consider systematic, reproducible selection of component numbers, latent dimensions, and single- or two-head architecture, potentially combining likelihood-based criteria, internal cluster validation, and stability diagnostics. Broader simulation studies, sensitivity analyses, and applications to other measurement structures would help assess how these choices affect clustering and computational cost.

MixedDeepMM version 0.1.0: R interface

The MixedDeepMM R package provides a common interface for Gaussian, Student-tt, and skew-tt deep mixtures with continuous and mixed measurements. Model specifications are controlled by the pathway family, the latent architecture, and the layer dimensions. The principal functions are listed in Table 4.

FunctionPurpose
mixeddeepmm()Fits a specified Gaussian, Student-tt, or skew-tt hierarchy.
summary(), logLik()Summarise model estimates and fitted likelihood.
predict()Obtains layer-specific posterior cluster probabilities.
mixeddeepmm_ari(), mixeddeepmm_mr()Evaluate agreement with external class labels.
mixeddeepmm_gower_distance(), mixeddeepmm_gower_silhouette()Compute mixed-data dissimilarities and the corresponding silhouette.

Table 4. Principal functions in MixedDeepMM.

After installing the versioned source package, the following illustrates the model specification for prepared predictors X and a named measurement-type vector types. The external outcome label is not passed to the fitting function.

For continuous measurements, heads="single" and layers=list(k=c(3L,2L), r=c(4L,2L)) provide the Wine specification (without a types argument). Changing family to "gaussian" or "skew-t" selects the corresponding pathway kernel. The Gower silhouette is evaluated from the predictors, whereas ARI and MR use withheld external labels.

library(MixedDeepMM)
fit <- mixeddeepmm(
  data = X, types = types, family = "t", heads = "two",
  layers = list(k = c(2L, 2L, 2L),
                r = list(continuous = 2L,
                          discrete = 2L, tail = 1L)),
  control = list(seed = 42L, nstart = 1L,
                 nu = 8, estimate_nu = TRUE)
)
post <- predict(fit, type = "layer",
                cluster_level = "shared_tail")
clusters <- max.col(post, ties.method = "first")

Funding

Funding. This work was supported by the Australian Research Council [DP230101671].

Competing interests

Competing interests. The authors declare that they have no competing interests.

Data availability

Data availability. The UCI Wine and UCI Cryotherapy datasets are publicly available from the UCI Machine Learning Repository. The datasets, processed data files, and accompanying example implementation and analysis code are also publicly available in the GitHub repository at https://github.com/wujrtudou/MixedDeepMMcases.

Code availability

Code availability. The analyses use the MixedDeepMM R implementation described in Appendix § 6.

Author contributions

Author contributions. Jinran Wu: Writing – review & editing, Writing – original draft, Visualization, Software, Methodology, Formal analysis, Conceptualization. Geoffrey J. McLachlan: Supervision, Investigation, Conceptualization, Methodology, Writing – review & editing, Funding acquisition.

References

  1. [1]Titterington, D.M., Smith, A.F.M., Makov, U.E.: Statistical Analysis of Finite Mixture Distributions. Wiley, Chichester (1985)
  2. [2]McLachlan, G.J., Peel, D.: Finite Mixture Models. Wiley, New York (2000)DOI
  3. [3]Fraley, C., Raftery, A.E.: Model-based clustering, discriminant analysis, and density estimation. Journal of the American Statistical Association 97(458), 611–631 (2002)
  4. [4]McLachlan, G.J., Lee, S.X., Rathnayake, S.I.: Finite mixture models. Annual Review of Statistics and its Application 6(1), 355–378 (2019)
  5. [5]Celeux, G., Govaert, G.: Gaussian parsimonious clustering models. Pattern Recognition 28(5), 781–793 (1995)DOI
  6. [6]Ghahramani, Z., Hinton, G.E.: The EM algorithm for mixtures of factor analyzers. Technical Report CRG-TR-96-1, University of Toronto (1996). Revised February 1997
  7. [7]McLachlan, G.J., Peel, D., Bean, R.W.: Modelling high-dimensional data by mixtures of factor analyzers. Computational Statistics & Data Analysis 41(3–4), 379–388 (2003)DOI
  8. [8]Baek, J., McLachlan, G.J., Flack, L.K.: Mixtures of factor analyzers with common factor loadings: Applications to the clustering and visualization of high-dimensional data. IEEE Transactions on Pattern Analysis and Machine Intelligence 32(7), 1298–1309 (2010)DOI
  9. [9]Tang, Y., Salakhutdinov, R., Hinton, G.E.: Deep mixtures of factor analyzers. In: Proceedings of the 29th International Conference on Machine Learning, pp. 505–512 (2012)arxiv.org/abs/1206.4635
  10. [10]Viroli, C., McLachlan, G.J.: Deep Gaussian mixture models. Statistics and Computing 29(1), 43–51 (2019)
  11. [11]Kock, L., Klein, N., Nott, D.J.: Variational inference and sparsity in high-dimensional deep Gaussian mixture models. Statistics and Computing 32 (2022)
  12. [12]Peel, D., McLachlan, G.J.: Robust mixture modelling using the t distribution. Statistics and Computing 10(4), 339–348 (2000)
  13. [13]Wu, J., McLachlan, G.J.: Robust deep mixture models. arXiv preprint arXiv:2608.01613 (2026)arxiv.org/abs/2608.01613
  14. [14]Wu, J., Wang, Y.-G., McLachlan, G.J.: Deep skew-t mixture models. arXiv preprint arXiv:2609.00773 (2026)arxiv.org/abs/2609.00773
  15. [15]Moustaki, I., Knott, M.: Generalized latent trait models. Psychometrika 65(3), 391–411 (2000)DOI
  16. [16]Gollini, I., Murphy, T.B.: Mixture of latent trait analyzers for model-based clustering of categorical data. Statistics and Computing 24(4), 569–588 (2014)
  17. [17]McParland, D., Gormley, I.C.: Model based clustering for mixed data: clustMD. Advances in Data Analysis and Classification 10(2), 155–169 (2016)
  18. [18]Fuchs, R., Pommeret, D., Viroli, C.: Mixed deep Gaussian mixture model: A clustering model for mixed datasets. Advances in Data Analysis and Classification 16(1), 31–53 (2022)DOI
  19. [19]Andrews, D.F., Mallows, C.L.: Scale mixtures of normal distributions. Journal of the Royal Statistical Society: Series B 36(1), 99–102 (1974)DOI
  20. [20]Azzalini, A., Capitanio, A.: Distributions generated by perturbation of symmetry with emphasis on a multivariate skew t-distribution. Journal of the Royal Statistical Society: Series B 65(2), 367–389 (2003)
  21. [21]Lee, S.X., McLachlan, G.J.: Finite mixtures of multivariate skew t-distributions: Some recent and new results. Statistics and Computing 24, 181–202 (2014)
  22. [22]Browne, R.P., McNicholas, P.D.: A mixture of generalized hyperbolic distributions. Canadian Journal of Statistics 43(2), 176–198 (2015)DOI
  23. [23]McCullagh, P., Nelder, J.A.: Generalized Linear Models, 2nd edn. Chapman & Hall, London (1989)DOI
  24. [24]McLachlan, G.J., Krishnan, T.: The EM Algorithm and Extensions. John Wiley & Sons, New York (2008)
  25. [25]Wei, G.C.G., Tanner, M.A.: A Monte Carlo implementation of the EM algorithm and the poor man’s data augmentation algorithms. Journal of the American Statistical Association 85(411), 699–704 (1990)
  26. [26]Dempster, A.P., Laird, N.M., Rubin, D.B.: Maximum likelihood from incomplete data via the EM algorithm. Journal of the Royal Statistical Society: Series B 39(1), 1–38 (1977)DOI
  27. [27]Lange, K.L., Little, R.J.A., Taylor, J.M.G.: Robust statistical modeling using the t distribution. Journal of the American Statistical Association 84(408), 881–896 (1989)
  28. [28]Nielsen, S.F.: The stochastic EM algorithm: Estimation and asymptotic results. Bernoulli 6(3), 457–489 (2000)DOI
  29. [29]Hubert, L., Arabie, P.: Comparing partitions. Journal of Classification 2, 193–218 (1985)DOI
  30. [30]Schwarz, G.: Estimating the dimension of a model. The Annals of Statistics 6(2), 461–464 (1978)DOI

Paper details

vlnc:W_fzrmyq8h
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Mathematics

Not recorded for this paper

View statement
Looking for related papers…
Contents