betaFeedback

WorksMathematics

When Is Membership Graded? Identifiability and Resolution Limits of a Graded Gaussian Mixture Model

Abstract

Mixed-membership models describe each unit as a graded blend of latent types. Before the blend is interpreted, the data should be able to rule out the simpler story in which every unit belongs to a single type. We ask when they can, in a scalar graded Gaussian mixture model, in which each observation is a Beta-weighted convex combination of independent Gaussian draws from two types, the mechanism behind functional mixed-membership models and the normal compositional model. The model is identifiable, unequal variances included, but the identifying information sits in the far tails of the marginal. For the symmetric model we give a closed-form hard-membership mixture that matches the graded marginal in five cumulants, and a Le Cam bound turns it into a certified sample size below which no test separates graded from hard membership with summed error under one half. The threshold is governed by the concentration cc relative to c0≈d2c_0\approx d^2, where dd is the type half-separation in within-type standard deviations. At d=3d=3 it is about 2,600 observations at c=3c=3 and 77,000 at c=6c=6. Most of the difficulty comes from learning cc. On 100 simulated datasets with graded membership, a correctly specified posterior under the default priors of a published Bayesian sampler for this model class reports hard membership every time, a weak prior gives intervals for the concentration spanning orders of magnitude, and the sampler's own answers depend on where it starts.

Introduction

Mixed-membership models give each unit a vector of weights over latent types in place of a single label. For discrete data the idea is standard ([21, 3, 7, 1]). For continuous measurements, a common construction writes each observation as a convex combination of independent draws, one from each type. This is the mechanism of the functional and multivariate mixed-membership models of Marco et al. [18] and of the normal compositional model used in hyperspectral unmixing ([22, 6, 5]). In both settings the weights are the quantity of scientific interest. In unmixing they are the abundances of materials within a pixel. In the EEG studies of Marco et al. [18], a child’s membership measures how far a peak in the alpha band has emerged above an aperiodic 1/f1/f background.

Before asking where a unit lies between two types, one can ask whether units lie between types at all. The alternative is that every unit belongs to exactly one type and the population is an ordinary two-component mixture. We ask when data can tell these two stories apart, and find that for many parameter values they cannot at the sample sizes one usually has. We work in the simplest model in which the question is well posed: a scalar observation

X=UY1+(1−U)Y2,Yk∼N(μk,σk2),U∼Beta⁡(a,b),(1)X = UY_1 + (1-U)Y_2, \qquad Y_k \sim\mathcal{N}(\mu_k,\sigma_k^2), \qquad U \sim\operatorname{Beta}(a,b), \tag*{(1)}

with UU, Y1Y_1 and Y2Y_2 independent. We call this the graded Gaussian mixture model (G2M2). The concentration c=a+bc = a + b sets how graded the population is. As c→0c \to0 the weights pile up at 0 and 1, and the law of XX becomes a two-component Gaussian mixture with hard memberships. As cc grows, the memberships concentrate around their mean and every unit becomes nearly the same blend. When the types’ draws are independent, any one-dimensional linear projection of the multivariate models cited above has exactly the form (1).

The model is identifiable, unequal within-type variances included (§ 4). The proof, however, reads the parameters off the moment generating function M(z)=EezXM(z) = \mathbb{E}e^{zX} as z→∞z \to\infty, one per order of the expansion, that is, from the far tails of the density, where a sample of moderate size has almost no observations. G2M2 is identifiable and weakly identified at the same time, and most of this paper measures how weak.

The main result concerns graded versus hard membership (§ 5). For the symmetric model we write down, in closed form, a two-component Gaussian mixture with hard memberships whose first five cumulants equal those of the graded marginal (Figure 1). A Le Cam argument turns the construction into a certified sample size, below which every test of graded against hard membership has type-I and type-II errors summing to at least 12\frac{1}{2}. The certified size depends on cc and on dd, the half-distance between the type means measured in within-type standard deviations, and it changes character at c0(d)=4d4/(4d2+1)≈d2c_0(d) = 4d^4/(4d^2 + 1) \approx d^2. At d=3d = 3 it is 2,605 observations when c=3c = 3 and 77,025 when c=6c = 6, and for c≥c0c \ge c_0 it exceeded 6×1046 \times10^4 in every case we computed. Away from c=0c = 0, where graded and hard membership coincide, the more graded the population, the more data one needs to show that it is graded at all.

Figure 1. Two stories, one marginal (d=3d = 3). First two panels: the graded marginal (solid) and its hard-membership look-alike from Proposition 6 (dashed) at c=3c = 3 and c=6c = 6. The two agree in their first five cumulants, and below the certified sizes shown no test separates them with summed error under 12\frac{1}{2} (Theorem 7). Third panel: the types behind each story, at ±3\pm3 with standard deviation 1 for the graded model, against ±1.14\pm1.14 with standard deviation 1.26 (c=3c = 3) and ±0.65\pm0.65 with standard deviation 1.20 (c=6c = 6) for the look-alikes. Right: the membership laws, Beta⁡(c/2,c/2)\operatorname{Beta}(c/2,c/2) against mass 12\frac{1}{2} at each of 0 and 1.

Contributions

  • Identifiability with unequal variances, from a tail expansion that also shows why identification is weak (§ 4).

  • A closed-form hard-membership look-alike matching five cumulants, certified sample sizes, with the sample size needed for reliable testing pinned down to within a factor of 2.6, the

regime boundary c0(d)c_0(d), and a corollary that cc cannot be estimated below the certified size (§ 5); and the matching statement at the Gaussian end (§ 6).

  • Evidence that learning cc causes most of the difficulty: it inflates Cramér–Rao sample sizes for the separation by a factor of 29 at c=1c = 1 and by more than 10410^4 for 4.5≤c≤64.5 \le c \le6 (§ 7).

  • Evidence that in practice the prior settles the question: under the default priors of a published sampler for this model class, a correctly specified posterior reports hard membership on all 100 simulated datasets with graded membership, and the sampler’s own answers depend on where it starts (§ 8).

Model and Related Work

The graded Gaussian mixture

Given U=uU = u, the observation in (1) is Gaussian,

X∣U=u∼N(uμ1+(1−u)μ2,u2σ12+(1−u)2σ22).(2)X \mid U = u \sim\mathcal{N}\left(u\mu_1 + (1-u)\mu_2, u^2\sigma_1^2 + (1-u)^2\sigma_2^2\right). \tag*{(2)}

The conditional mean is linear in the membership and the conditional variance is quadratic. The quadratic variance distinguishes averaging independent draws from selecting one of them, as a finite mixture does. We write c=a+bc = a + b for the concentration and π=a/c\pi= a/c for the prior mean membership. As c→0c \to0 with π\pi fixed, UU converges in distribution to a Bernoulli(π\pi) variable, and the law of XX converges to the hard-membership mixture πN(μ1,σ12)+(1−π)N(μ2,σ22)\pi\mathcal{N}(\mu_1,\sigma_1^2) + (1-\pi)\mathcal{N}(\mu_2,\sigma_2^2). Hard membership is therefore a boundary point of the graded model, not a separate model.

The symmetric model. Much of the analysis uses type means ±s\pm s, a common within-type standard deviation tt and a symmetric membership law, a=b=ra = b = r, so that c=2rc = 2r. Writing W=2U−1W = 2U - 1 and Z∼N(0,1)Z \sim\mathcal{N}(0,1) independent of WW,

X=sW+t(1+W2)/2 Z.(3)X = sW + t\sqrt{(1+W^2)/2}\,Z. \tag*{(3)}

We write θ=(s,r,t)\theta= (s,r,t), fθf_\theta for the density of XX, vv for its variance, κk\kappa_k for its kkth cumulant and γ2=κ4/v2\gamma_2 = \kappa_4/v^2 for its excess kurtosis. The separation-to-noise ratio is d=s/td = s/t, so the type means lie 2d2d within-type standard deviations apart. Because f(s,r,t)(x)=t−1f(d,r,1)(x/t)f_{(s,r,t)}(x) = t^{-1}f_{(d,r,1)}(x/t), the divergence between fθf_\theta and any comparison law built from θ\theta in a scale-equivariant way (its moment-matched Gaussian, or the look-alike of § 5) depends on θ\theta only through (d,c)(d,c). All thresholds below are reported as functions of (d,c)(d,c).

Related work

Instances of the model. Marco et al. [18] introduced functional mixed-membership models in which each observation is a convex combination of dependent latent Gaussian features (their term for types), so that the conditional covariance is quadratic in the memberships; Marco et al. [19] treat the high-dimensional multivariate case. Both are implemented in the BayesFMMM package, which we use in § 8. G2M2 is their model with one-dimensional observations, two features and no cross-covariance between them. They point out an exact rescaling problem: given the memberships, shrinking them into a sub-simplex and extrapolating the feature means leaves the likelihood unchanged. They resolve it by requiring at least one unit to belong entirely to each feature (separability), and their consistency results condition on known memberships. Under a Beta membership law a rescaled membership is no longer Beta distributed, so the exact rescaling is unavailable. The weak direction we study is its closest analogue inside the model: a path along which the low-order moments of the marginal stay nearly fixed, because raising cc, which compresses memberships toward 12\frac{1}{2}, is offset by moving the type means apart.

In hyperspectral imaging, the normal compositional model treats each pixel as a convex combination of random endmember spectra [22, 6], with Bayesian estimation developed by Eches et al. [5], Halimi et al. [12] and Zhou et al. [30]. Woodbridge et al. [26] consider estimating KK Gaussians from convex combinations of their realizations, which is our setting with KK types. They note that Bayesian estimators for this model are sensitive to their priors, for which our results give a quantitative reason.

Two related models do not contain G2M2: the continuous mixed-membership model of Gruhl and Erosheva [10] draws each outcome from a single type, and the partial-membership model of Heller et al. [15] gives a precision-weighted conditional law.

Identifiability and resolution. Finite mixtures of Gaussians are identifiable up to relabeling [23, 28] but can be weakly identified: Ho and Nguyen [16] show that location–scale Gaussian mixtures are estimated slowly because derivatives in the mean and in the variance are linked by the heat equation, and Nguyen [20] and Heinrich and Kahn [14] relate estimation rates to the strength of identifiability. Testing the number of components is non-regular because the null lies on the boundary of the parameter space [9, 4]. Hard membership is such a boundary point of G2M2, and a singularity in the sense of Watanabe [25]. Andrews and Cheng [2] treat inference when the strength of identification varies over the parameter space. For mixed membership, Yurochkin et al. [29] give geometric estimators for the Dirichlet simplex nest, in which noise is additive and does not scale with the membership, and Gu et al. [11] establish identifiability of grade-of-membership models for multivariate categorical data. Neither covers draw averaging. Identifiability does not settle how much data a problem needs. Hardt and Price [13] show that estimating the parameters of a mixture of two univariate Gaussians to constant additive error takes Θ(σ12)\Theta(\sigma^{12}) samples, where σ2\sigma^2 is the mixture variance; their lower bound adds Gaussian noise to two mixtures that agree in their first five moments. Wu and Yang [27] use moment matching to prove optimal rates for Gaussian mixtures. Our look-alike is a construction of the same kind, but across two model classes: a graded marginal and a hard mixture that agree through the fifth cumulant. The lower bounds use the two-point method of Le Cam [17, 24].

Identifiability

The moment generating function M(z)=EezXM(z)=\mathbb{E}e^{zX} is finite on R\mathbb{R} because UU is bounded. Conditioning on UU in (2) writes M(z)M(z) as a Beta integral of exp⁡{φz(u)}\exp\{\varphi_z(u)\}, with φz\varphi_z quadratic in uu. For large zz the integral concentrates at the endpoint of [0,1][0,1] attached to the type with the larger variance (or, if the variances are equal, the larger mean), and a Laplace-type expansion at that endpoint gives the following.

Lemma 1 (Tail of the moment generating function). In model (1) with (μ1,σ1)≠(μ2,σ2)(\mu_1,\sigma_1)\ne(\mu_2,\sigma_2), label the types so that (σ1,μ1)(\sigma_1,\mu_1) exceeds (σ2,μ2)(\sigma_2,\mu_2) lexicographically, and let Δ=μ1−μ2\Delta=\mu_1-\mu_2. As z→+∞z\to+\infty,

log⁡M(z)=12σ12z2+μ1z−2blog⁡z+C0+C1z+C2z2+O(z−3),\log M(z)=\frac{1}{2}\sigma_1^2z^2+\mu_1z-2b\log z+C_0+\frac{C_1}{z}+\frac{C_2}{z^2}+O(z^{-3}),

where C0=log⁡{Γ(b)/B(a,b)}−blog⁡σ12C_0 = \log\{\Gamma(b)/B(a,b)\} - b\log\sigma_1^2, C1=−bΔ/σ12C_1 = -b\Delta/\sigma_1^2, and

C2=bΔ22σ14+b(b+1)(σ12+σ22)2σ14−(a−1)bσ12.C_2 = \frac{b\Delta^2}{2\sigma_1^4} + \frac{b(b+1)(\sigma_1^2+\sigma_2^2)}{2\sigma_1^4} - \frac{(a-1)b}{\sigma_1^2}.

The proof (Appendix A) substitutes u=1−τ/Bzu = 1-\tau/B_z with Bz=σ12z2+ΔzB_z = \sigma_1^2z^2+\Delta z. This rescaling absorbs the z2z^2 term of φz\varphi_z into the Gamma kernel e−ττb−1e^{-\tau}\tau^{b-1}, and a second-order expansion in the manner of Watson’s lemma follows. The coefficients were also checked numerically at 50-digit precision (Appendix A).

References

  1. [1]Edoardo M. Airoldi, David M. Blei, Stephen E. Fienberg, and Eric P. Xing. Mixed membership stochastic blockmodels. Journal of Machine Learning Research, 9:1981–2014, 2008.
  2. [2]Donald W. K. Andrews and Xu Cheng. Estimation and inference with weak, semi-strong, and strong identification. Econometrica, 80(5):2153–2211, 2012.
  3. [3]David M. Blei, Andrew Y. Ng, and Michael I. Jordan. Latent Dirichlet allocation. Journal of Machine Learning Research, 3:993–1022, 2003.
  4. [4]Hanfeng Chen and Jiahua Chen. The likelihood ratio test for homogeneity in finite mixture models. The Canadian Journal of Statistics, 29(2):201–215, 2001.DOI
  5. [5]Olivier Eches, Nicolas Dobigeon, Corinne Mailhes, and Jean-Yves Tourneret. Bayesian estimation of linear mixtures using the normal compositional model. application to hyperspectral imagery. IEEE Transactions on Image Processing, 19(6):1403–1413, 2010.DOI
  6. [6]Michael T. Eismann and David W. J. Stein. Stochastic mixture modeling. In Chein-I Chang, editor, Hyperspectral Data Exploitation: Theory and Applications, pages 107–148. Wiley, 2007.DOI
  7. [7]Elena Erosheva, Stephen Fienberg, and John Lafferty. Mixed-membership models of scientific publications. Proceedings of the National Academy of Sciences, 101(suppl 1):5220–5227, 2004.
  8. [8]Daniel Foreman-Mackey, David W. Hogg, Dustin Lang, and Jonathan Goodman. emcee: The MCMC hammer. Publications of the Astronomical Society of the Pacific, 125(925):306–312, 2013.
  9. [9]Jayanta K. Ghosh and Pranab K. Sen. On the asymptotic performance of the log likelihood ratio statistic for the mixture model and related results. In Lucien M. Le Cam and Richard A. Olshen, editors, Proceedings of the Berkeley Conference in Honor of Jerzy Neyman and Jack Kiefer, volume 2, pages 789–806. Wadsworth, 1985.
  10. [10]Jonathan Gruhl and Elena A. Erosheva. A tale of two (types of) memberships: Comparing mixed and partial membership with a continuous data example. In Edoardo M. Airoldi, David M. Blei, Elena A. Erosheva, and Stephen E. Fienberg, editors, Handbook of Mixed Membership Models and Their Applications, pages 15–38. Chapman & Hall/CRC, 2014.
  11. [11]Yuqi Gu, Elena A. Erosheva, Gongjun Xu, and David B. Dunson. Dimension-grouped mixed membership models for multivariate categorical data. Journal of Machine Learning Research, 24(88):1–49, 2023.
  12. [12]Abderrahim Halimi, Nicolas Dobigeon, and Jean-Yves Tourneret. Unsupervised unmixing of hyper-spectral images accounting for endmember variability. IEEE Transactions on Image Processing, 24(12):4904–4917, 2015.
  13. [13]Moritz Hardt and Eric Price. Tight bounds for learning a mixture of two Gaussians. In Proceedings of the 47th Annual ACM Symposium on Theory of Computing (STOC), pages 753–760, 2015.
  14. [14]Philippe Heinrich and Jonas Kahn. Strong identifiability and optimal minimax rates for finite mixture estimation. The Annals of Statistics, 46(6A):2844–2870, 2018.
  15. [15]Katherine A. Heller, Sinead Williamson, and Zoubin Ghahramani. Statistical models for partial membership. In Proceedings of the 25th International Conference on Machine Learning, pages 392–399, 2008.
  16. [16]Nhat Ho and XuanLong Nguyen. Convergence rates of parameter estimation for some weakly identifiable finite mixtures. The Annals of Statistics, 44(6):2726–2755, 2016.
  17. [17]Lucien Le Cam. Convergence of estimates under dimensionality restrictions. The Annals of Statistics, 1(1):38–53, 1973.
  18. [18]Nicholas Marco, Damla Şentürk, Shafali Jeste, Charlotte DiStefano, Abigail Dickinson, and Donatello Telesca. Functional mixed membership models. Journal of Computational and Graphical Statistics, 33(4):1139–1149, 2024a. doi: 10.1080/10618600.2024.2304633.DOI
  19. [19]Nicholas Marco, Damla Şentürk, Shafali Jeste, Charlotte C. DiStefano, Abigail Dickinson, and Donatello Telesca. Flexible regularized estimation in high-dimensional mixed membership models. Computational Statistics & Data Analysis, 194:107931, 2024b. doi: 10.1016/j.csda.2024.107931.
  20. [20]XuanLong Nguyen. Convergence of latent mixing measures in finite and infinite mixture models. The Annals of Statistics, 41(1):370–400, 2013.
  21. [21]Jonathan K. Pritchard, Matthew Stephens, and Peter Donnelly. Inference of population structure using multilocus genotype data. Genetics, 155(2):945–959, 2000.
  22. [22]David Stein. Application of the normal compositional model to the analysis of hyperspectral imagery. In IEEE Workshop on Advances in Techniques for Analysis of Remotely Sensed Data, pages 44–51, 2003.DOI
  23. [23]Henry Teicher. Identifiability of finite mixtures. The Annals of Mathematical Statistics, 34(4):1265–1269, 1963.DOI
  24. [24]Alexandre B. Tsybakov. Introduction to Nonparametric Estimation. Springer, New York, 2009.
  25. [25]Sumio Watanabe. Algebraic Geometry and Statistical Learning Theory. Cambridge University Press, 2009.
  26. [26]Yonatan Woodbridge, Uri Okun, Gal Elidan, and Ami Wiesel. Unmixing K-Gaussians with application to hyperspectral imaging. IEEE Transactions on Geoscience and Remote Sensing, 57(9):7281–7293, 2019.DOI
  27. [27]Yihong Wu and Pengkun Yang. Optimal estimation of Gaussian mixtures via denoised method of moments. The Annals of Statistics, 48(4):1981–2007, 2020.arxiv.org/abs/1807.07237
  28. [28]Sidney J. Yakowitz and John D. Spragins. On the identifiability of finite mixtures. The Annals of Mathematical Statistics, 39(1):209–214, 1968.DOI
  29. [29]Mikhail Yurochkin, Aritra Guha, Yuekai Sun, and XuanLong Nguyen. Dirichlet simplex nest and geometric inference. In Proceedings of the 36th International Conference on Machine Learning, volume 97 of PMLR, pages 7262–7271, 2019.arxiv.org/abs/1905.11009
  30. [30]Yuan Zhou, Anand Rangarajan, and Paul D. Gader. A Gaussian mixture model representation of endmember variability in hyperspectral unmixing. IEEE Transactions on Image Processing, 27(5):2242–2256, 2018.DOI

Paper details

vlnc:W_bp99nwcy
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Mathematics

generative AI tools

View statement
Looking for related papers…
Contents