Introduction
Mixed-membership models give each unit a vector of weights over latent types in place of a single label. For discrete data the idea is standard ([21 , 3 , 7 , 1 ] ). For continuous measurements, a common construction writes each observation as a convex combination of independent draws, one from each type. This is the mechanism of the functional and multivariate mixed-membership models of Marco et al. [18 ] and of the normal compositional model used in hyperspectral unmixing ([22 , 6 , 5 ] ). In both settings the weights are the quantity of scientific interest. In unmixing they are the abundances of materials within a pixel. In the EEG studies of Marco et al. [18 ] , a child’s membership measures how far a peak in the alpha band has emerged above an aperiodic 1 / f 1/f 1/ f background.
Before asking where a unit lies between two types, one can ask whether units lie between types at all. The alternative is that every unit belongs to exactly one type and the population is an ordinary two-component mixture. We ask when data can tell these two stories apart, and find that for many parameter values they cannot at the sample sizes one usually has. We work in the simplest model in which the question is well posed: a scalar observation
X = U Y 1 + ( 1 − U ) Y 2 , Y k ∼ N ( μ k , σ k 2 ) , U ∼ Beta ( a , b ) , (1) X = UY_1 + (1-U)Y_2, \qquad Y_k \sim\mathcal{N}(\mu_k,\sigma_k^2), \qquad U \sim\operatorname{Beta}(a,b), \tag*{(1)} X = U Y 1 + ( 1 − U ) Y 2 , Y k ∼ N ( μ k , σ k 2 ) , U ∼ Beta ( a , b ) , (1)
with U U U , Y 1 Y_1 Y 1 and Y 2 Y_2 Y 2 independent. We call this the graded Gaussian mixture model (G2M2). The concentration c = a + b c = a + b c = a + b sets how graded the population is. As c → 0 c \to0 c → 0 the weights pile up at 0 and 1, and the law of X X X becomes a two-component Gaussian mixture with hard memberships. As c c c grows, the memberships concentrate around their mean and every unit becomes nearly the same blend. When the types’ draws are independent, any one-dimensional linear projection of the multivariate models cited above has exactly the form (1).
The model is identifiable, unequal within-type variances included (§ 4 ). The proof, however, reads the parameters off the moment generating function M ( z ) = E e z X M(z) = \mathbb{E}e^{zX} M ( z ) = E e z X as z → ∞ z \to\infty z → ∞ , one per order of the expansion, that is, from the far tails of the density, where a sample of moderate size has almost no observations. G2M2 is identifiable and weakly identified at the same time, and most of this paper measures how weak.
The main result concerns graded versus hard membership (§ 5 ). For the symmetric model we write down, in closed form, a two-component Gaussian mixture with hard memberships whose first five cumulants equal those of the graded marginal (Figure 1 ). A Le Cam argument turns the construction into a certified sample size, below which every test of graded against hard membership has type-I and type-II errors summing to at least 1 2 \frac{1}{2} 2 1 . The certified size depends on c c c and on d d d , the half-distance between the type means measured in within-type standard deviations, and it changes character at c 0 ( d ) = 4 d 4 / ( 4 d 2 + 1 ) ≈ d 2 c_0(d) = 4d^4/(4d^2 + 1) \approx d^2 c 0 ( d ) = 4 d 4 / ( 4 d 2 + 1 ) ≈ d 2 . At d = 3 d = 3 d = 3 it is 2,605 observations when c = 3 c = 3 c = 3 and 77,025 when c = 6 c = 6 c = 6 , and for c ≥ c 0 c \ge c_0 c ≥ c 0 it exceeded 6 × 10 4 6 \times10^4 6 × 1 0 4 in every case we computed. Away from c = 0 c = 0 c = 0 , where graded and hard membership coincide, the more graded the population, the more data one needs to show that it is graded at all.
Figure 1. Two stories, one marginal (d = 3 d = 3 d = 3 ). First two panels: the graded marginal (solid) and its hard-membership look-alike from Proposition 6 (dashed) at c = 3 c = 3 c = 3 and c = 6 c = 6 c = 6 . The two agree in their first five cumulants, and below the certified sizes shown no test separates them with summed error under 1 2 \frac{1}{2} 2 1 (Theorem 7 ). Third panel: the types behind each story, at ± 3 \pm3 ± 3 with standard deviation 1 for the graded model, against ± 1.14 \pm1.14 ± 1.14 with standard deviation 1.26 (c = 3 c = 3 c = 3 ) and ± 0.65 \pm0.65 ± 0.65 with standard deviation 1.20 (c = 6 c = 6 c = 6 ) for the look-alikes. Right: the membership laws, Beta ( c / 2 , c / 2 ) \operatorname{Beta}(c/2,c/2) Beta ( c /2 , c /2 ) against mass 1 2 \frac{1}{2} 2 1 at each of 0 and 1.
Contributions
Identifiability with unequal variances, from a tail expansion that also shows why identification is weak (§ 4 ).
A closed-form hard-membership look-alike matching five cumulants, certified sample sizes, with the sample size needed for reliable testing pinned down to within a factor of 2.6, the
regime boundary c 0 ( d ) c_0(d) c 0 ( d ) , and a corollary that c c c cannot be estimated below the certified size (§ 5 ); and the matching statement at the Gaussian end (§ 6 ).
Evidence that in practice the prior settles the question: under the default priors of a published sampler for this model class, a correctly specified posterior reports hard membership on all 100 simulated datasets with graded membership, and the sampler’s own answers depend on where it starts (§ 8 ).
Model and Related Work
The graded Gaussian mixture
Given U = u U = u U = u , the observation in (1) is Gaussian,
X ∣ U = u ∼ N ( u μ 1 + ( 1 − u ) μ 2 , u 2 σ 1 2 + ( 1 − u ) 2 σ 2 2 ) . (2) X \mid U = u \sim\mathcal{N}\left(u\mu_1 + (1-u)\mu_2, u^2\sigma_1^2 + (1-u)^2\sigma_2^2\right). \tag*{(2)} X ∣ U = u ∼ N ( u μ 1 + ( 1 − u ) μ 2 , u 2 σ 1 2 + ( 1 − u ) 2 σ 2 2 ) . (2)
The conditional mean is linear in the membership and the conditional variance is quadratic. The quadratic variance distinguishes averaging independent draws from selecting one of them, as a finite mixture does. We write c = a + b c = a + b c = a + b for the concentration and π = a / c \pi= a/c π = a / c for the prior mean membership. As c → 0 c \to0 c → 0 with π \pi π fixed, U U U converges in distribution to a Bernoulli(π \pi π ) variable, and the law of X X X converges to the hard-membership mixture π N ( μ 1 , σ 1 2 ) + ( 1 − π ) N ( μ 2 , σ 2 2 ) \pi\mathcal{N}(\mu_1,\sigma_1^2) + (1-\pi)\mathcal{N}(\mu_2,\sigma_2^2) π N ( μ 1 , σ 1 2 ) + ( 1 − π ) N ( μ 2 , σ 2 2 ) . Hard membership is therefore a boundary point of the graded model, not a separate model.
The symmetric model. Much of the analysis uses type means ± s \pm s ± s , a common within-type standard deviation t t t and a symmetric membership law, a = b = r a = b = r a = b = r , so that c = 2 r c = 2r c = 2 r . Writing W = 2 U − 1 W = 2U - 1 W = 2 U − 1 and Z ∼ N ( 0 , 1 ) Z \sim\mathcal{N}(0,1) Z ∼ N ( 0 , 1 ) independent of W W W ,
X = s W + t ( 1 + W 2 ) / 2 Z . (3) X = sW + t\sqrt{(1+W^2)/2}\,Z. \tag*{(3)} X = s W + t ( 1 + W 2 ) /2 Z . (3)
We write θ = ( s , r , t ) \theta= (s,r,t) θ = ( s , r , t ) , f θ f_\theta f θ for the density of X X X , v v v for its variance, κ k \kappa_k κ k for its k k k th cumulant and γ 2 = κ 4 / v 2 \gamma_2 = \kappa_4/v^2 γ 2 = κ 4 / v 2 for its excess kurtosis. The separation-to-noise ratio is d = s / t d = s/t d = s / t , so the type means lie 2 d 2d 2 d within-type standard deviations apart. Because f ( s , r , t ) ( x ) = t − 1 f ( d , r , 1 ) ( x / t ) f_{(s,r,t)}(x) = t^{-1}f_{(d,r,1)}(x/t) f ( s , r , t ) ( x ) = t − 1 f ( d , r , 1 ) ( x / t ) , the divergence between f θ f_\theta f θ and any comparison law built from θ \theta θ in a scale-equivariant way (its moment-matched Gaussian, or the look-alike of § 5 ) depends on θ \theta θ only through ( d , c ) (d,c) ( d , c ) . All thresholds below are reported as functions of ( d , c ) (d,c) ( d , c ) .
Related work
Instances of the model. Marco et al. [18 ] introduced functional mixed-membership models in which each observation is a convex combination of dependent latent Gaussian features (their term for types), so that the conditional covariance is quadratic in the memberships; Marco et al. [19 ] treat the high-dimensional multivariate case. Both are implemented in the BayesFMMM package, which we use in § 8 . G2M2 is their model with one-dimensional observations, two features and no cross-covariance between them. They point out an exact rescaling problem: given the memberships, shrinking them into a sub-simplex and extrapolating the feature means leaves the likelihood unchanged. They resolve it by requiring at least one unit to belong entirely to each feature (separability), and their consistency results condition on known memberships. Under a Beta membership law a rescaled membership is no longer Beta distributed, so the exact rescaling is unavailable. The weak direction we study is its closest analogue inside the model: a path along which the low-order moments of the marginal stay nearly fixed, because raising c c c , which compresses memberships toward 1 2 \frac{1}{2} 2 1 , is offset by moving the type means apart.
In hyperspectral imaging, the normal compositional model treats each pixel as a convex combination of random endmember spectra [22 , 6 ] , with Bayesian estimation developed by Eches et al. [5 ] , Halimi et al. [12 ] and Zhou et al. [30 ] . Woodbridge et al. [26 ] consider estimating K K K Gaussians from convex combinations of their realizations, which is our setting with K K K types. They note that Bayesian estimators for this model are sensitive to their priors, for which our results give a quantitative reason.
Two related models do not contain G2M2: the continuous mixed-membership model of Gruhl and Erosheva [10 ] draws each outcome from a single type, and the partial-membership model of Heller et al. [15 ] gives a precision-weighted conditional law.
Identifiability and resolution. Finite mixtures of Gaussians are identifiable up to relabeling [23 , 28 ] but can be weakly identified: Ho and Nguyen [16 ] show that location–scale Gaussian mixtures are estimated slowly because derivatives in the mean and in the variance are linked by the heat equation, and Nguyen [20 ] and Heinrich and Kahn [14 ] relate estimation rates to the strength of identifiability. Testing the number of components is non-regular because the null lies on the boundary of the parameter space [9 , 4 ] . Hard membership is such a boundary point of G2M2, and a singularity in the sense of Watanabe [25 ] . Andrews and Cheng [2 ] treat inference when the strength of identification varies over the parameter space. For mixed membership, Yurochkin et al. [29 ] give geometric estimators for the Dirichlet simplex nest, in which noise is additive and does not scale with the membership, and Gu et al. [11 ] establish identifiability of grade-of-membership models for multivariate categorical data. Neither covers draw averaging. Identifiability does not settle how much data a problem needs. Hardt and Price [13 ] show that estimating the parameters of a mixture of two univariate Gaussians to constant additive error takes Θ ( σ 12 ) \Theta(\sigma^{12}) Θ ( σ 12 ) samples, where σ 2 \sigma^2 σ 2 is the mixture variance; their lower bound adds Gaussian noise to two mixtures that agree in their first five moments. Wu and Yang [27 ] use moment matching to prove optimal rates for Gaussian mixtures. Our look-alike is a construction of the same kind, but across two model classes: a graded marginal and a hard mixture that agree through the fifth cumulant. The lower bounds use the two-point method of Le Cam [17 , 24 ] .
Identifiability
The moment generating function M ( z ) = E e z X M(z)=\mathbb{E}e^{zX} M ( z ) = E e z X is finite on R \mathbb{R} R because U U U is bounded. Conditioning on U U U in (2) writes M ( z ) M(z) M ( z ) as a Beta integral of exp { φ z ( u ) } \exp\{\varphi_z(u)\} exp { φ z ( u )} , with φ z \varphi_z φ z quadratic in u u u . For large z z z the integral concentrates at the endpoint of [ 0 , 1 ] [0,1] [ 0 , 1 ] attached to the type with the larger variance (or, if the variances are equal, the larger mean), and a Laplace-type expansion at that endpoint gives the following.
Lemma 1 (Tail of the moment generating function). In model (1) with ( μ 1 , σ 1 ) ≠ ( μ 2 , σ 2 ) (\mu_1,\sigma_1)\ne(\mu_2,\sigma_2) ( μ 1 , σ 1 ) = ( μ 2 , σ 2 ) , label the types so that ( σ 1 , μ 1 ) (\sigma_1,\mu_1) ( σ 1 , μ 1 ) exceeds ( σ 2 , μ 2 ) (\sigma_2,\mu_2) ( σ 2 , μ 2 ) lexicographically, and let Δ = μ 1 − μ 2 \Delta=\mu_1-\mu_2 Δ = μ 1 − μ 2 . As z → + ∞ z\to+\infty z → + ∞ ,
log M ( z ) = 1 2 σ 1 2 z 2 + μ 1 z − 2 b log z + C 0 + C 1 z + C 2 z 2 + O ( z − 3 ) , \log M(z)=\frac{1}{2}\sigma_1^2z^2+\mu_1z-2b\log z+C_0+\frac{C_1}{z}+\frac{C_2}{z^2}+O(z^{-3}), log M ( z ) = 2 1 σ 1 2 z 2 + μ 1 z − 2 b log z + C 0 + z C 1 + z 2 C 2 + O ( z − 3 ) ,
where C 0 = log { Γ ( b ) / B ( a , b ) } − b log σ 1 2 C_0 = \log\{\Gamma(b)/B(a,b)\} - b\log\sigma_1^2 C 0 = log { Γ ( b ) / B ( a , b )} − b log σ 1 2 , C 1 = − b Δ / σ 1 2 C_1 = -b\Delta/\sigma_1^2 C 1 = − b Δ/ σ 1 2 , and
C 2 = b Δ 2 2 σ 1 4 + b ( b + 1 ) ( σ 1 2 + σ 2 2 ) 2 σ 1 4 − ( a − 1 ) b σ 1 2 . C_2 = \frac{b\Delta^2}{2\sigma_1^4} + \frac{b(b+1)(\sigma_1^2+\sigma_2^2)}{2\sigma_1^4} - \frac{(a-1)b}{\sigma_1^2}. C 2 = 2 σ 1 4 b Δ 2 + 2 σ 1 4 b ( b + 1 ) ( σ 1 2 + σ 2 2 ) − σ 1 2 ( a − 1 ) b .
The proof (Appendix A) substitutes u = 1 − τ / B z u = 1-\tau/B_z u = 1 − τ / B z with B z = σ 1 2 z 2 + Δ z B_z = \sigma_1^2z^2+\Delta z B z = σ 1 2 z 2 + Δ z . This rescaling absorbs the z 2 z^2 z 2 term of φ z \varphi_z φ z into the Gamma kernel e − τ τ b − 1 e^{-\tau}\tau^{b-1} e − τ τ b − 1 , and a second-order expansion in the manner of Watson’s lemma follows. The coefficients were also checked numerically at 50-digit precision (Appendix A).
Proposition 2 (Identifiability). Among parameter vectors of model (1) with ( μ 1 , σ 1 ) ≠ ( μ 2 , σ 2 ) (\mu_1,\sigma_1) \ne(\mu_2,\sigma_2) ( μ 1 , σ 1 ) = ( μ 2 , σ 2 ) , the law of X X X determines ( μ 1 , μ 2 , σ 1 , σ 2 , a , b ) (\mu_1,\mu_2,\sigma_1,\sigma_2,a,b) ( μ 1 , μ 2 , σ 1 , σ 2 , a , b ) up to the label swap ( μ 1 , μ 2 , σ 1 , σ 2 , a , b ) ↦ ( μ 2 , μ 1 , σ 2 , σ 1 , b , a ) (\mu_1,\mu_2,\sigma_1,\sigma_2,a,b) \mapsto(\mu_2,\mu_1,\sigma_2,\sigma_1,b,a) ( μ 1 , μ 2 , σ 1 , σ 2 , a , b ) ↦ ( μ 2 , μ 1 , σ 2 , σ 1 , b , a ) . In particular c c c is identified, and π \pi π is identified up to π ↦ 1 − π \pi\mapsto1-\pi π ↦ 1 − π .
Proof sketch. The label swap preserves the law because 1 − U ∼ Beta ( b , a ) 1-U \sim\operatorname{Beta}(b,a) 1 − U ∼ Beta ( b , a ) , and exactly one labeling satisfies the ordering in Lemma 1. Equal laws give equal coefficients in the asymptotic scale z 2 z^2 z 2 , z z z , log z \log z log z , 1 1 1 , z − 1 z^{-1} z − 1 , z − 2 z^{-2} z − 2 , which determine in turn σ 1 \sigma_1 σ 1 , μ 1 \mu_1 μ 1 , b b b , then a a a (since a ↦ log B ( a , b ) a \mapsto\log B(a,b) a ↦ log B ( a , b ) is strictly decreasing), Δ \Delta Δ , and finally σ 2 2 \sigma_2^2 σ 2 2 , which enters C 2 C_2 C 2 linearly with positive coefficient. □
The excluded case ( μ 1 , σ 1 ) = ( μ 2 , σ 2 ) (\mu_1,\sigma_1)=(\mu_2,\sigma_2) ( μ 1 , σ 1 ) = ( μ 2 , σ 2 ) is a pure scale mixture. Proposition 2 does not rule out that a law in its class also arises from the excluded case. Within the symmetric model (3), a separate one-term expansion covers the excluded case as well: f θ = f θ ′ f_\theta=f_{\theta'} f θ = f θ ′ if and only if ( ∣ s ∣ , r , t ) = ( ∣ s ′ ∣ , r ′ , t ′ ) (|s|,r,t)=(|s'|,r',t') ( ∣ s ∣ , r , t ) = ( ∣ s ′ ∣ , r ′ , t ′ ) , for all s ∈ R s\in\mathbb{R} s ∈ R (Appendix A).
Remark 3 (An identification ladder). The parameters enter the tail of log M \log M log M at successively weaker orders, from σ 1 \sigma_1 σ 1 at z 2 z^2 z 2 down to the smaller within-type variance at z − 2 z^{-2} z − 2 , and every step takes z → ∞ z\to\infty z → ∞ . For roughly Gaussian data with variance v v v the empirical moment generating function has relative variance of order e z 2 v / n e^{z^2v}/n e z 2 v / n , so a sample of size n n n sees M ( z ) M(z) M ( z ) only for z z z up to about ( log n ) / v \sqrt{(\log n)/v} ( log n ) / v , far from where the expansion separates the parameters.
Graded Versus Hard Membership
This section works in the symmetric model (3). Proofs are in Appendix B.
Cumulants and the sign of the kurtosis
Lemma 4 (Cumulants). All odd cumulants of f θ f_\theta f θ vanish, and
v = s 2 + ( r + 1 ) t 2 2 r + 1 , κ 4 = − 6 s 4 + 12 r s 2 t 2 + 3 r t 4 ( 2 r + 1 ) 2 ( 2 r + 3 ) . v = \frac{s^2+(r+1)t^2}{2r+1}, \qquad\kappa_4 = \frac{-6s^4+12rs^2t^2+3rt^4}{(2r+1)^2(2r+3)}. v = 2 r + 1 s 2 + ( r + 1 ) t 2 , κ 4 = ( 2 r + 1 ) 2 ( 2 r + 3 ) − 6 s 4 + 12 r s 2 t 2 + 3 r t 4 .
The variance is constant along the curves s 2 + ( r + 1 ) t 2 = ( 2 r + 1 ) v s^2+(r+1)t^2=(2r+1)v s 2 + ( r + 1 ) t 2 = ( 2 r + 1 ) v , so the second moment alone cannot separate separation, noise and concentration. The fourth cumulant can, and its sign decides what a hard-membership impostor must look like.
Proposition 5 (Range and sign of the excess kurtosis). For every θ \theta θ , − 2 < γ 2 < 3 -2<\gamma_2<3 − 2 < γ 2 < 3 . Moreover γ 2 < 0 \gamma_2<0 γ 2 < 0 if and only if c < c 0 ( d ) c<c_0(d) c < c 0 ( d ) , where
c 0 ( d ) = 4 d 4 4 d 2 + 1 = d 2 − 1 4 + O ( d − 2 ) , c_0(d)=\frac{4d^4}{4d^2+1}=d^2-\frac{1}{4}+O(d^{-2}), c 0 ( d ) = 4 d 2 + 1 4 d 4 = d 2 − 4 1 + O ( d − 2 ) , and γ 2 = 0 \gamma_2=0 γ 2 = 0 at c = c 0 ( d ) c=c_0(d) c = c 0 ( d ) .
Below c 0 c_0 c 0 the marginal is platykurtic, like a two-component mixture with separated means. Above it the marginal is leptokurtic, like a scale mixture. At d = 3 d=3 d = 3 , c 0 = 8.76 c_0=8.76 c 0 = 8.76 . Numerically sup θ γ 2 ≈ 0.24 \sup_\theta\gamma_2\approx0.24 sup θ γ 2 ≈ 0.24 , so the upper bound of 3 is far from attained; it is the bound that keeps the look-alike below well defined.
A hard-membership look-alike
Proposition 6 (Look-alike). Let v v v and γ 2 \gamma_2 γ 2 be the variance and excess kurtosis of f θ f_\theta f θ . Define a two-component Gaussian mixture G θ G_\theta G θ as follows. If γ 2 < 0 \gamma_2<0 γ 2 < 0 ,
G θ = 1 2 N ( s G , t G 2 ) + 1 2 N ( − s G , t G 2 ) . G_\theta=\frac{1}{2}\mathcal{N}(s_G,t_G^2)+\frac{1}{2}\mathcal{N}(-s_G,t_G^2). G θ = 2 1 N ( s G , t G 2 ) + 2 1 N ( − s G , t G 2 ) . *with s G 2 = v − γ 2 / 2 s_G^2=v\sqrt{-\gamma_2}/2 s G 2 = v − γ 2 /2 and t G 2 = v − s G 2 t_G^2=v-s_G^2 t G 2 = v − s G 2 . If γ 2 = 0 \gamma_2=0 γ 2 = 0 , G θ = N ( 0 , v ) G_\theta=\mathcal{N}(0,v) G θ = N ( 0 , v ) . If γ 2 > 0 \gamma_2>0 γ 2 > 0 ,
G θ = 1 2 N ( 0 , v + ) + 1 2 N ( 0 , v − ) , v ± = v ( 1 ± γ 2 / 3 ) . G_\theta=\frac{1}{2}\mathcal{N}(0,v_+)+\frac{1}{2}\mathcal{N}(0,v_-),\qquad v_\pm=v(1\pm\sqrt{\gamma_2/3}). G θ = 2 1 N ( 0 , v + ) + 2 1 N ( 0 , v − ) , v ± = v ( 1 ± γ 2 /3 ) . Then G θ G_\theta G θ is well defined for every θ \theta θ , and κ k ( G θ ) = κ k ( f θ ) \kappa_k(G_\theta)=\kappa_k(f_\theta) κ k ( G θ ) = κ k ( f θ ) for k = 1 , … , 5 k=1,\ldots,5 k = 1 , … , 5 . For γ 2 < 0 \gamma_2<0 γ 2 < 0 , G θ G_\theta G θ is the hard-membership limit (c → 0 c\to0 c → 0 ) of the symmetric model with half-separation s G s_G s G and noise t G t_G t G : every unit belongs to exactly one type.
The match is less remarkable than it may sound: both laws are symmetric, so odd cumulants vanish, and a symmetric two-component mixture has exactly two free parameters with which to match κ 2 \kappa_2 κ 2 and κ 4 \kappa_4 κ 4 . What matters is that this is enough: the moment-matched Gaussian first differs from f θ f_\theta f θ in κ 4 \kappa_4 κ 4 , and G θ G_\theta G θ only in κ 6 \kappa_6 κ 6 . All numbers below use exact divergences.
Certified sample sizes
Theorem 7 (Graded against hard). Let n h a r d ( θ ) = 1 / { 2 KL ( f θ ∥ G θ ) } n_{\mathrm{hard}}(\theta)=1/\{2\operatorname{KL}(f_\theta\mathbin\Vert G_\theta)\} n hard ( θ ) = 1/ { 2 KL ( f θ ∥ G θ )} , which is positive and finite and depends on θ \theta θ only through ( d , c ) (d,c) ( d , c ) . For every n n n and every test ϕ \phi ϕ based on n n n i.i.d. observations,
f θ ⊗ n ( ϕ = 1 ) + G θ ⊗ n ( ϕ = 0 ) ≥ 1 − 1 2 n / n h a r d ( θ ) . f_\theta^{\otimes n}(\phi=1)+G_\theta^{\otimes n}(\phi=0)\ge1-\frac{1}{2}\sqrt{n/n_{\mathrm{hard}}(\theta)}. f θ ⊗ n ( ϕ = 1 ) + G θ ⊗ n ( ϕ = 0 ) ≥ 1 − 2 1 n / n hard ( θ ) . In particular, for n ≤ n h a r d ( θ ) n\le n_{\mathrm{hard}}(\theta) n ≤ n hard ( θ ) the two error probabilities sum to at least 1 2 \frac{1}{2} 2 1 .
The proof is Pinsker’s inequality for the product measures, and it holds for any fixed comparison law. A certified sample size is a value of 1 / ( 2 KL ) 1/(2\operatorname{KL}) 1/ ( 2 KL ) for an explicit comparison law; we write n G a u s s ( θ ) n_{\mathrm{Gauss}}(\theta) n Gauss ( θ ) for the one against N ( 0 , v ) \mathcal{N}(0,v) N ( 0 , v ) , and use numerically optimized two-component mixtures where they are closer than the closed form.
Tightness. For the pair ( f θ , G θ ) (f_\theta,G_\theta) ( f θ , G θ ) , with squared Hellinger distance H 2 = 1 2 ∫ ( f θ − G θ ) 2 H^2=\frac{1}{2}\int(\sqrt{f_\theta}-\sqrt{G_\theta})^2 H 2 = 2 1 ∫ ( f θ − G θ ) 2 , every test has summed error at least 1 2 ( 1 − H 2 ) 2 n \frac{1}{2}(1-H^2)^{2n} 2 1 ( 1 − H 2 ) 2 n and the likelihood-ratio test at most ( 1 − H 2 ) n (1-H^2)^n ( 1 − H 2 ) n (Appendix B), so the sample size at which summed error 0.05 becomes achievable is pinned down to within a factor of 2.6. At d = 3 d=3 d = 3 , H 2 / KL ∈ [ 0.250 , 0.252 ] H^2/\operatorname{KL}\in[0.250,0.252] H 2 / KL ∈ [ 0.250 , 0.252 ] for c ∈ { 1 , 2 , 3 , 4 , 6 , 8 } c\in\{1,2,3,4,6,8\} c ∈ { 1 , 2 , 3 , 4 , 6 , 8 } , which places it between about 9 n h a r d 9\,n_{\mathrm{hard}} 9 n hard and 24 n h a r d 24\,n_{\mathrm{hard}} 24 n hard . Against the composite family of all hard mixtures testing can only be harder, so the certified size remains a lower bound there.
Estimating the concentration. The testing bound transfers to estimation, because the hard look-alike is a limit of graded models with vanishing concentration.
Corollary 8. Let γ 2 ( θ ) < 0 \gamma_2(\theta)<0 γ 2 ( θ ) < 0 and n ≤ n h a r d ( θ ) n\le n_{\mathrm{hard}}(\theta) n ≤ n hard ( θ ) . For every estimator c ^ \hat c c ^ and every η > 0 \eta>0 η > 0 there is ϵ > 0 \epsilon>0 ϵ > 0 such that
max θ ′ ∈ { θ , ( s G , ϵ / 2 , t G ) } P θ ′ ( ∣ c ^ − c ( θ ′ ) ∣ ≥ ( c − ϵ ) / 2 ) ≥ 1 4 − η . \max_{\theta'\in\{\theta,(s_G,\epsilon/2,t_G)\}}\mathbb{P}_{\theta'}\left(|\hat c-c(\theta')|\ge(c-\epsilon)/2\right)\ge\frac{1}{4}-\eta. θ ′ ∈ { θ , ( s G , ϵ /2 , t G )} max P θ ′ ( ∣ c ^ − c ( θ ′ ) ∣ ≥ ( c − ϵ ) /2 ) ≥ 4 1 − η .
No procedure can tell concentration c c c from concentration near zero with probability better than 3 4 \frac{3}{4} 4 3 under both.
Table 1 gives the numbers at d = 3 d = 3 d = 3 . For c ≤ 6 c \le6 c ≤ 6 , the certified size against hard membership is 15 to 36 times that against a single Gaussian. Near c 0 c_0 c 0 the Gaussian itself becomes a good impostor, because γ 2 \gamma_2 γ 2 passes through zero, and n G a u s s n_{\mathrm{Gauss}} n Gauss jumps to 53,363 at c = 8 c = 8 c = 8 . Section 5 returns to that end of the model.
Table 1. Certified sample sizes 1 / ( 2 K L ) 1/(2KL) 1/ ( 2 K L ) at d = 3 d = 3 d = 3 . The look-alike column gives the means and, in parentheses, the standard deviations of G θ G_\theta G θ ; at c = 12 > c 0 ( 3 ) c = 12 > c_0(3) c = 12 > c 0 ( 3 ) it is the equal-mean scale mixture. “Optimized” uses a numerically optimized two-component mixture and is never smaller than the closed form. n G a u s s n_{\mathrm{Gauss}} n Gauss is against N ( 0 , v ) \mathcal{N}(0,v) N ( 0 , v ) .
The regime map
Figure 2 extends the table to a grid over ( d , c ) (d,c) ( d , c ) . The certified size is U-shaped in c c c . As c → 0 c \to0 c → 0 it diverges, because the graded model itself converges to a hard mixture; this end is benign, since there graded membership is nearly hard. Away from it the certified size rises steeply with c c c ; for d ≥ 1 d \ge1 d ≥ 1 the minimum lies at or below the smallest computed concentration, c = 0.5 c = 0.5 c = 0.5 . The boundary c 0 ( d ) ≈ d 2 c_0(d) \approx d^2 c 0 ( d ) ≈ d 2 separates two regimes. To its left, well-separated types at modest concentration have small certified sizes: at d = 6 d = 6 d = 6 , types twelve within-type standard deviations apart, the certified size at c = 6 c = 6 c = 6 is 687. To its right, every computed cell exceeds 6 × 10 4 6 \times10^4 6 × 1 0 4 , and most exceed 10 6 10^6 1 0 6 . When the types are close, the question is hard throughout: for d ≤ 1.5 d \le1.5 d ≤ 1.5 no computed cell falls below 1.4 × 10 4 1.4 \times10^4 1.4 × 1 0 4 .
Figure 2. Certified sample size below which no test separates graded from hard membership with summed error under 1 2 \frac{1}{2} 2 1 , across separation-to-noise ratio d = s / t d=s/t d = s / t and concentration c c c . Each cell reports 1 / ( 2 KL ( f θ ∥ G ) ) 1/(2\operatorname{KL}(f_\theta\Vert G)) 1/ ( 2 KL ( f θ ∥ G )) for the closer, in divergence, of the closed-form look-alike and a numerically optimized two-component mixture. Red staircase: c = c 0 ( d ) = 4 d 4 / ( 4 d 2 + 1 ) ≈ d 2 c=c_0(d)=4d^4/(4d^2+1)\approx d^2 c = c 0 ( d ) = 4 d 4 / ( 4 d 2 + 1 ) ≈ d 2 . In the hatched cells to its right, γ 2 ≥ 0 \gamma_2\ge0 γ 2 ≥ 0 and the look-alike is an equal-mean scale mixture, so the comparison is with one type of heterogeneous variance rather than two separated types. Every computed cell with c ≥ c 0 ( d ) c\ge c_0(d) c ≥ c 0 ( d ) exceeds 6 × 10 4 6\times10^4 6 × 1 0 4 . KL values carry quadrature error below 10 − 15 10^{-15} 1 0 − 15 . In cells marked * the divergence is below 10 − 6 10^{-6} 1 0 − 6 and the numerical search for the closest mixture is unreliable: each value is still a valid lower bound, but these cells need not be monotone in c c c and should not be compared with one another.
Asymmetric truths. Theorem 7 applies to any truth and any comparison law. For three asymmetric truths at d = 3 d = 3 d = 3 (π = 0.3 \pi= 0.3 π = 0.3 , σ 1 = 1.5 \sigma_1 = 1.5 σ 1 = 1.5 , or both), certified sizes against numerically optimized hard mixtures range from 1,009 to 4,312 at c = 3 c = 3 c = 3 and from 4,462 to 34,218 at c = 6 c = 6 c = 6 , about 40 to 125 times the corresponding Gaussian thresholds (Table 5).
The Gaussian End
Section 4 concerned the end of the model where c → 0 c \to0 c → 0 . At the other end, a large concentration pulls every membership toward 1 2 \frac{1}{2} 2 1 and the marginal approaches a Gaussian. By Lemma 4, the set of symmetric configurations with a given variance v v v and noise level t t t contains the point ( s r , r , t ) (s_r,r,t) ( s r , r , t ) for every r r r at which
s r 2 = r ( 2 v − t 2 ) + ( v − t 2 ) ≥ 0. s_r^2 = r(2v-t^2) + (v-t^2) \ge0. s r 2 = r ( 2 v − t 2 ) + ( v − t 2 ) ≥ 0.
When t 2 < 2 v t^2 < 2v t 2 < 2 v this curve is unbounded, and s r → ∞ s_r \to\infty s r → ∞ as r → ∞ r \to\infty r → ∞ . Along it the marginal converges to N ( 0 , v ) \mathcal{N}(0,v) N ( 0 , v ) .
Proposition 9 (The Gaussian end). Fix θ \theta θ with variance v v v and t 2 < 2 v t^2<2v t 2 < 2 v , and let θ k = ( s r k , r k , t ) \theta_k=(sr_k,r_k,t) θ k = ( s r k , r k , t ) with r k → ∞ r_k\to\infty r k → ∞ . Then f θ k → N ( 0 , v ) f_{\theta_k}\to N(0,v) f θ k → N ( 0 , v ) in total variation, and for every n n n and every test ϕ \phi ϕ ,
lim inf k → ∞ [ f θ ⊗ n ( ϕ = 1 ) + f θ k ⊗ n ( ϕ = 0 ) ] ≥ 1 − 1 2 n n G a u s s ( θ ) . \liminf_{k\to\infty}\left[f_\theta^{\otimes n}(\phi=1)+f_{\theta_k}^{\otimes n}(\phi=0)\right]\ge1-\frac{1}{2}\sqrt{\frac{n}{n_{\mathrm{Gauss}}(\theta)}}. k → ∞ lim inf [ f θ ⊗ n ( ϕ = 1 ) + f θ k ⊗ n ( ϕ = 0 ) ] ≥ 1 − 2 1 n Gauss ( θ ) n . The proof (Appendix B) combines a Beta central limit theorem, Scheffé’s lemma and Pinsker’s inequality. For n ≤ n G a u s s n\le n_{\mathrm{Gauss}} n ≤ n Gauss , no test can exclude the unbounded part of the curve with summed error below 1 2 \frac{1}{2} 2 1 . How fast the marginal approaches the Gaussian depends on the path.
Proposition 10 (Two regimes for the kurtosis). (a) Along the curve above, with ρ = t 2 / v ∈ ( 0 , 2 ) \rho=t^2/v\in(0,2) ρ = t 2 / v ∈ ( 0 , 2 ) fixed, γ 2 = 3 ( 2 − ρ ) ( 3 ρ − 2 ) / ( 4 r ) + O ( r − 2 ) \gamma_2=3(2-\rho)(3\rho-2)/(4r)+O(r^{-2}) γ 2 = 3 ( 2 − ρ ) ( 3 ρ − 2 ) / ( 4 r ) + O ( r − 2 ) . (b) At fixed ( s , t ) (s,t) ( s , t ) , γ 2 = ( 6 d 2 + 3 / 2 ) / r 2 + O ( r − 3 ) \gamma_2=(6d^2+3/2)/r^2+O(r^{-3}) γ 2 = ( 6 d 2 + 3/2 ) / r 2 + O ( r − 3 ) as r → ∞ r\to\infty r → ∞ .
With the heuristic KL ( f θ ∥ N ( 0 , v ) ) ≈ γ 2 2 / 48 \operatorname{KL}(f_\theta\Vert N(0,v))\approx\gamma_2^2/48 KL ( f θ ∥ N ( 0 , v )) ≈ γ 2 2 /48 , accurate to 4% for r ≥ 5 r\ge5 r ≥ 5 along such curves, n G a u s s n_{\mathrm{Gauss}} n Gauss grows like c 2 c^2 c 2 along a curve of fixed variance (unless ρ = 2 3 \rho=\frac{2}{3} ρ = 3 2 ) and like c 4 c^4 c 4 at fixed ( s , t ) (s,t) ( s , t ) , where it is also non-monotone, peaking near c 0 ( d ) c_0(d) c 0 ( d ) (Table 1). Neither power law holds over practical ranges of c c c .
Throughout Table 1 the hard end binds (n h a r d > n G a u s s n_{\mathrm{hard}}>n_{\mathrm{Gauss}} n hard > n Gauss ). The Gaussian end matters in practice because a sampler started far from the truth tends to settle there (Section 7).
Supplying the Concentration
In applications c c c is usually learned along with everything else. To see how much of the difficulty that causes, we computed Cramér–Rao bounds for the full two-type model at the symmetric truth (μ = ± 3 \mu= \pm3 μ = ± 3 , σ 1 = σ 2 = 1 \sigma_1 = \sigma_2 = 1 σ 1 = σ 2 = 1 , π = 1 2 \pi= \frac{1}{2} π = 2 1 ), with μ 1 \mu_1 μ 1 , μ 2 \mu_2 μ 2 , σ 1 \sigma_1 σ 1 , σ 2 \sigma_2 σ 2 and π \pi π free and c c c either supplied or learned. Table 2 reports, for each target, the sample size at which the bound on the standard deviation equals a stated tolerance.
Table 2. Cramér–Rao sample sizes at d = 3 d = 3 d = 3 : the n n n at which the bound on the standard deviation equals the tolerance. Tolerances are 0.3 (10%) for the half-separation, 10% for σ 1 \sigma_1 σ 1 , 0.05 for π \pi π and 0.1 for log c \log c log c . The last column is the ratio of the two half-separation columns, computed from unrounded bounds.
With c c c supplied, the separation is cheap: 44 to 1,408 observations bring the bound on its standard deviation down to 10%. Learning c c c multiplies these sizes by 29 at c = 1 c = 1 c = 1 and by about 17,000 at c = 6 c = 6 c = 6 ; on a finer grid the factor peaks at about 54,000 near c = 5.5 c = 5.5 c = 5.5 (Appendix § 13 ). Supplying c c c removes the curve of § 6 , since at fixed r r r and v v v the relation s 2 + ( r + 1 ) t 2 = ( 2 r + 1 ) v s^2 + (r + 1)t^2 = (2r + 1)v s 2 + ( r + 1 ) t 2 = ( 2 r + 1 ) v bounds s s s .
Something weak remains. At c = 6 c = 6 c = 6 the weakest Fisher direction, in coordinates ( μ 1 , μ 2 , log σ 1 , log σ 2 , logit π ) (\mu_1, \mu_2, \log\sigma_1, \log\sigma_2, \operatorname{logit} \pi) ( μ 1 , μ 2 , log σ 1 , log σ 2 , logit π ) , is about ( 0.64 , 0.64 , − 0.08 , 0.08 , − 0.42 ) (0.64, 0.64, -0.08, 0.08, -0.42) ( 0.64 , 0.64 , − 0.08 , 0.08 , − 0.42 ) : it shifts both means together, moves the log-variances in opposite directions and changes π \pi π , leaving the separation almost unchanged. This is why σ 1 \sigma_1 σ 1 and π \pi π are expensive at c = 6 c = 6 c = 6 (225,012 and 1.6 × 10 6 1.6 \times10^6 1.6 × 1 0 6 observations) although the separation is not. These columns are non-monotone in c c c ; on a finer grid both peak near c = 6.5 c = 6.5 c = 6.5 .
A check against fitted estimates. We fitted the full model, with the true c c c supplied, to ten datasets per cell at n ∈ { 10 3 , 10 4 } n \in\{10^3, 10^4\} n ∈ { 1 0 3 , 1 0 4 } by maximizing the marginal likelihood from many starts (Appendices § 14 and § 16 ). At n = 10 4 n = 10^4 n = 1 0 4 the across-dataset standard deviation of s ^ \hat{s} s ^ is within the sampling band of the bound for 0.25 ≤ c ≤ 4 0.25 \le c \le4 0.25 ≤ c ≤ 4 (ratios 0.71 to 1.27), but about 2.6 times the bound at c = 6 c = 6 c = 6 and 8, where a few fits sit on other local maxima. For c ≥ 10 c \ge10 c ≥ 10 the likelihood has competing maxima within about a nat of one another, collapsed, separated and point-mass configurations, and the fit depends on the number of starts. At n = 10 3 n = 10^3 n = 1 0 3 , four to eight of ten datasets had a rival maximum within one nat whose half-separation differed by more than 0.5. There the local bound no longer describes the estimator; this is the weak identification of § 5 and § 6 seen from the estimator’s side.
Existing Inference
We studied the published sampler for this model class, BayesFMMM (Marco et al., [18 ] ), on data from the symmetric model with d = 3 d = 3 d = 3 , n = 2000 n = 2000 n = 2000 and c ∈ { 3 , 6 } c \in\{3, 6\} c ∈ { 3 , 6 } , ten datasets per value of c c c . With one-dimensional observations its noise variance is absorbed into the feature covariance, so G2M2 lies inside the model it fits, although scalar data are far from the functional and high-dimensional settings it was built for. We ask two questions: what a correct posterior under the package’s default priors reports, and whether the package’s sampler finds it. Configurations are compared by the exact marginal log-likelihood, not the conditional one the sampler prints.
The prior decides. To answer the first question we sampled the same model with memberships integrated out by quadrature, using an ensemble sampler (Foreman-Mackey et al., [8 ] ). Each dataset was run from the truth and from the configuration where BayesFMMM’s default start settles, under two priors: the package’s priors on the means and on α 3 \alpha_3 α 3 , which is exponential with mean 0.1, and a weak prior (Appendix § 15 ). On 100 datasets, 50 per value of c c c , the package’s priors put the posterior at the hard end every time and from both starts. The pooled median of α 3 \alpha_3 α 3 is 0.03 to 0.09, every 95% interval excludes the true concentration, and the half-separation is between about 0.1 and 1.2, against 3 in truth. Yet the best hard configuration fits about as well as the truth: the median log-likelihood difference is between − 0.7 -0.7 − 0.7 and + 2.0 +2.0 + 2.0 nats across values of c c c and starts. Under the weak prior no dataset goes hard and the intervals cover the truth (all for α 3 \alpha_3 α 3 , 192 of 200 for s s s ), but their upper limits for α 3 \alpha_3 α 3 range from 5.5 to 907. At n = 2000 n = 2000 n = 2000 the data do not determine whether membership is graded; the priors do (Figure 3 ). With the package’s priors this persists far beyond the certified sizes: at n = 120,000 n = 120{,}000 n = 120 , 000 for c = 3 c = 3 c = 3 and n = 10 6 n = 10^{6} n = 1 0 6 for c = 6 c = 6 c = 6 the intervals still exclude the true concentration and separation. A rough accounting attributes this to the priors placing the truth tens of nats below a hard configuration, while the data’s evidence grows only like n K L n\mathrm{KL} n KL (Appendix § 15 ).
Figure 3. Where the posterior puts its mass (dataset 1 at each c c c ; n = 2000 n = 2000 n = 2000 , d = 3 d = 3 d = 3 ). Gray: reference posterior with memberships integrated out, under a weak prior. Orange: the same under BayesFMMM’s priors on the means and on the concentration, α 3 ∼ Exp ( 10 ) \alpha_3 \sim\operatorname{Exp}(10) α 3 ∼ Exp ( 10 ) . Each pools two runs, started at the truth and at the configuration in which BayesFMMM’s default start settles; at c = 3 c = 3 c = 3 the gray cloud at small s s s and large α 3 \alpha_3 α 3 comes from the collapsed start, which mixes incompletely. Dotted: the weak curve, the symmetric configuration closest in divergence to the truth at each concentration; thick where it lies within one nat of the truth. Red and blue: BayesFMMM draws from the second half of its long runs, from its default and truth-initialized starts. Star: truth.
The sampler. We then ran BayesFMMM itself under its default priors, from its default initialization and from the truth, in short runs (2,000 iterations, one initialization, ten datasets per c c c ) and in long runs on five of them (50,000 iterations, multiple starts, tempered transitions; Appendix § 15 ). From its default start the sampler settles near zero separation, with memberships uncorrelated with the true ones; longer runs change its concentration but not its separation (Table 3 ). The two starts gave disjoint 95% intervals for s s s on all 20 short-run and all 10 long-run datasets (split-R ^ \hat{R} R ^ 12 to 26). Tempered transitions hot enough to cross between the configurations were essentially never accepted with 2,000 latent memberships. Nor is the truth start a calibrated reference: over 50,000 iterations it drifts toward smaller s s s and α 3 \alpha_3 α 3 along the weak curve, the symmetric configurations closest in divergence to the truth at each concentration (Figure 3 ), and its intervals stop covering the truth.
Table 3. BayesFMMM at d = 3 d = 3 d = 3 , n = 2000 n = 2000 n = 2000 , default priors, two starting points. Short runs: 2,000 iterations, one initialization try, ten datasets per c c c . Long runs: 50,000 iterations, multiple starts, tempered transitions, five of the same datasets. “Gap” is the marginal log-likelihood of the truth minus that of the best of 12 posterior draws, memberships integrated out; the predicted gap to the Gaussian end, n KL ( f θ ∥ N ( 0 , v ) ) n\operatorname{KL}(f_\theta\Vert\mathcal{N}(0,v)) n KL ( f θ ∥ N ( 0 , v )) , is 10.3 at c = 3 c = 3 c = 3 and 0.47 at c = 6 c = 6 c = 6 . Coverage is of the nominal 95% interval.
The likelihood gaps separate sampler failure from genuine ambiguity. The default-start configuration is close to N ( 0 , v ) \mathcal{N}(0,v) N ( 0 , v ) , so its gap to the truth should be near n KL ( f θ ∥ N ( 0 , v ) ) n\operatorname{KL}(f_\theta\Vert\mathcal{N}(0,v)) n KL ( f θ ∥ N ( 0 , v )) . That is 10.3 nats at c = 3 c = 3 c = 3 , where the observed medians are 8.9 and 10.9 and every gap is positive, so the sampler fails to reach a better fit. At c = 6 c = 6 c = 6 it is 0.47 and the observed median is 0.4, since n = 2000 n = 2000 n = 2000 is close to n G a u s s = 2,139 n_{\mathrm{Gauss}} = 2{,}139 n Gauss = 2 , 139 : the data cannot choose.
Discussion and Limitations
G2M2 is identifiable, yet over much of its parameter space the data cannot tell graded from hard membership at the sample sizes one usually has. A practical consequence is a diagnostic. Before reporting that units are blends, compute the certified size against the best-fitting hard mixture over the configurations the data cannot exclude, such as posterior draws or fits within a nat of the best; a point estimate is unreliable in exactly this regime (§ 7 ). If n n n is below it for any of them, the data cannot rule out hard membership, and a conclusion of gradedness rests on the prior or the sampler’s starting point.
Limitations. Our results are for scalar data and two types. When the types’ draws are independent, any one-dimensional linear projection of a multivariate model of this kind is exactly a G2M2, so the thresholds bound what any single projection can show. With P P P features sharing one membership, however, features in which the two types have equal means still reveal each unit’s variance factor U 2 + ( 1 − U ) 2 U^2 + (1 - U)^2 U 2 + ( 1 − U ) 2 , and we expect this to weaken the ambiguity as P P P grows; we have not quantified how fast. The closed-form thresholds are for the symmetric model; for asymmetric truths the certificates are numerical. The case ( μ 1 , σ 1 ) = ( μ 2 , σ 2 ) (\mu_1, \sigma_1) = (\mu_2, \sigma_2) ( μ 1 , σ 1 ) = ( μ 2 , σ 2 ) is treated only in its symmetric instance, and for K ≥ 3 K \ge3 K ≥ 3 , scalar data constrain a membership only through two conditional moments, so different memberships can induce the same law, an exact obstruction we do not treat. Finally, § 8 evaluates one sampler, and at large n n n its collapsed-start reference chains do not mix.
Code availability. Code for all computations, and the outputs behind every table and figure, are available from the authors; a documented public repository is in preparation.
Use of generative AI
In this work, we used generative AI tools for formulating precise mathematical claims consistent with the standard terminology used in the literature, writing the proof and double-checking notations and consistencies in the proof, identifying gaps in the proof, designing and providing feedback for the suite of experiments as well as suggesting additional ablations, helping analyze the results and identifying typos / non-material algebraic errors in the proof.
We have not used generative AI tools for supplying the main mathematical ideas of the proof (all ideas including key mathematical ideas in the proof belonged to the authors), result interpretations, main coding tasks (other than identifying bugs).
Additionally, tasks such as cleaning and reformatting datasets, assisting with translation are not applicable to this work.
Additionally, we used generative AI tools to create plotting and graphing code suitable for latex given experimental results, suggesting the suites of experiments and the range of parameters to run them over, drafting parts of the paper as well as grammar correction, performing literature search in order to discover other works that have been done, summarizing other papers related to our works, analyzing our paper versus those other works in order to ensure we do not overclaim novelty, brainstorming next experiment ideas given those other works, grammar and style correction.
We have reviewed all AI-assisted work. We take responsibility for the final content of this work, including text, claims, or artifacts produced with the aid of generative AI.
Proofs for Section 3
The symmetric model
The symmetric model needs only a one-term expansion, which also covers the case s = 0 s=0 s = 0 excluded from Proposition 2 .
Lemma 11 (Tail of the moment generating function). Let s ≥ 0 s \ge0 s ≥ 0 , r > 0 r > 0 r > 0 , t > 0 t > 0 t > 0 , write M θ ( z ) = E e z X M_\theta(z)=\mathbb{E}e^{zX} M θ ( z ) = E e z X and A θ ( z ) = s z + t 2 z 2 / 2 A_\theta(z)=sz+t^2z^2/2 A θ ( z ) = sz + t 2 z 2 /2 . As z → + ∞ z\to+\infty z → + ∞ ,
log M θ ( z ) = A θ ( z ) − r log A θ ( z ) + log Γ ( 2 r ) 2 r Γ ( r ) + 1 { s = 0 } log 2 + o ( 1 ) . \log M_\theta(z)=A_\theta(z)-r\log A_\theta(z)+\log\frac{\Gamma(2r)}{2^r\Gamma(r)}+\mathbb{1}_{\{s=0\}}\log2+o(1). log M θ ( z ) = A θ ( z ) − r log A θ ( z ) + log 2 r Γ ( r ) Γ ( 2 r ) + 1 { s = 0 } log 2 + o ( 1 ) .
Proof. M θ M_\theta M θ is finite on R \mathbb{R} R because ∣ W ∣ ≤ 1 |W|\le1 ∣ W ∣ ≤ 1 . The density of W W W is p r ( w ) = C r ( 1 − w 2 ) r − 1 p_r(w)=C_r(1-w^2)^{r-1} p r ( w ) = C r ( 1 − w 2 ) r − 1 on [ − 1 , 1 ] [-1,1] [ − 1 , 1 ] with C r = Γ ( 2 r ) / ( 2 2 r − 1 Γ ( r ) 2 ) C_r=\Gamma(2r)/(2^{2r-1}\Gamma(r)^2) C r = Γ ( 2 r ) / ( 2 2 r − 1 Γ ( r ) 2 ) , and conditioning on W W W gives M θ ( z ) = E exp { φ z ( W ) } M_\theta(z)=\mathbb{E}\exp\{\varphi_z(W)\} M θ ( z ) = E exp { φ z ( W )} with φ z ( w ) = s z w + 1 4 t 2 z 2 ( 1 + w 2 ) \varphi_z(w)=szw+\frac{1}{4}t^2z^2(1+w^2) φ z ( w ) = sz w + 4 1 t 2 z 2 ( 1 + w 2 ) . Substituting w = 1 − y w=1-y w = 1 − y gives φ z ( 1 − y ) = A − s z y − 1 4 t 2 z 2 y ( 2 − y ) \varphi_z(1-y)=A-szy-\frac{1}{4}t^2z^2y(2-y) φ z ( 1 − y ) = A − sz y − 4 1 t 2 z 2 y ( 2 − y ) , where A = A θ ( z ) A=A_\theta(z) A = A θ ( z ) , so
M θ ( z ) = C r e A J ( z ) , J ( z ) = ∫ 0 2 e − s z y − 1 4 t 2 z 2 y ( 2 − y ) ( y ( 2 − y ) ) r − 1 d y . M_\theta(z)=C_re^AJ(z),\qquad J(z)=\int_0^2e^{-szy-\frac{1}{4}t^2z^2y(2-y)}(y(2-y))^{r-1}\,dy. M θ ( z ) = C r e A J ( z ) , J ( z ) = ∫ 0 2 e − sz y − 4 1 t 2 z 2 y ( 2 − y ) ( y ( 2 − y ) ) r − 1 d y .
Fix δ ∈ ( 0 , 1 ) \delta\in(0,1) δ ∈ ( 0 , 1 ) and split J = J L + J M + J R J=J_L+J_M+J_R J = J L + J M + J R over [ 0 , δ ] [0,\delta] [ 0 , δ ] , [ δ , 2 − δ ] [\delta,2-\delta] [ δ , 2 − δ ] , [ 2 − δ , 2 ] [2-\delta,2] [ 2 − δ , 2 ] . Middle. There y ( 2 − y ) > δ y(2-y)>\delta y ( 2 − y ) > δ , so J M ≤ e − t 2 z 2 δ / 4 ∫ 0 2 ( y ( 2 − y ) ) r − 1 d y = O ( e − t 2 z 2 δ / 4 ) J_M\le e^{-t^2z^2\delta/4}\int_0^2(y(2-y))^{r-1}\,dy=O(e^{-t^2z^2\delta/4}) J M ≤ e − t 2 z 2 δ /4 ∫ 0 2 ( y ( 2 − y ) ) r − 1 d y = O ( e − t 2 z 2 δ /4 ) . Left. On [ 0 , δ ] [0,\delta] [ 0 , δ ] , 2 y ( 1 − δ / 2 ) ≤ y ( 2 − y ) ≤ 2 y 2y(1-\delta/2)\le y(2-y)\le2y 2 y ( 1 − δ /2 ) ≤ y ( 2 − y ) ≤ 2 y , so the exponent lies between − A y -Ay − A y and − ( 1 − δ / 2 ) A y -(1-\delta/2)Ay − ( 1 − δ /2 ) A y ; also ( y ( 2 − y ) ) r − 1 = ( 2 y ) r − 1 ( 1 − y / 2 ) r − 1 (y(2-y))^{r-1}=(2y)^{r-1}(1-y/2)^{r-1} ( y ( 2 − y ) ) r − 1 = ( 2 y ) r − 1 ( 1 − y /2 ) r − 1 with ( 1 − y / 2 ) r − 1 ∈ [ m δ , M δ ] (1-y/2)^{r-1}\in[m_\delta,M_\delta] ( 1 − y /2 ) r − 1 ∈ [ m δ , M δ ] , where m δ , M δ → 1 m_\delta,M_\delta\to1 m δ , M δ → 1 as δ → 0 \delta\to0 δ → 0 . Since ∫ 0 δ e − B y y r − 1 d y = Γ ( r ) B − r ( 1 + o ( 1 ) ) \int_0^\delta e^{-By}y^{r-1}\,dy=\Gamma(r)B^{-r}(1+o(1)) ∫ 0 δ e − B y y r − 1 d y = Γ ( r ) B − r ( 1 + o ( 1 )) as B → ∞ B\to\infty B → ∞ ,
m δ 2 r − 1 Γ ( r ) ≤ lim inf z → ∞ A r J L ≤ lim sup z → ∞ A r J L ≤ M δ 2 r − 1 Γ ( r ) ( 1 − δ / 2 ) − r . m_\delta2^{r-1}\Gamma(r)\le\liminf_{z\to\infty}A^rJ_L\le\limsup_{z\to\infty}A^rJ_L\le M_\delta2^{r-1}\Gamma(r)(1-\delta/2)^{-r}. m δ 2 r − 1 Γ ( r ) ≤ z → ∞ lim inf A r J L ≤ z → ∞ lim sup A r J L ≤ M δ 2 r − 1 Γ ( r ) ( 1 − δ /2 ) − r .
Right. With y = 2 − u y=2-u y = 2 − u , the exponent is − 2 s z + s z u − 1 4 t 2 z 2 u ( 2 − u ) ≤ − 2 s z − B ′ u -2sz+szu-\frac{1}{4}t^2z^2u(2-u)\le-2sz-B'u − 2 sz + sz u − 4 1 t 2 z 2 u ( 2 − u ) ≤ − 2 sz − B ′ u on [ 0 , δ ] [0,\delta] [ 0 , δ ] , where B ′ = ( 1 − δ / 2 ) t 2 z 2 / 2 − s z → ∞ B'=(1-\delta/2)t^2z^2/2-sz\to\infty B ′ = ( 1 − δ /2 ) t 2 z 2 /2 − sz → ∞ . Hence A r J R = O ( e − 2 s z ) → 0 A^rJ_R=O(e^{-2sz})\to0 A r J R = O ( e − 2 sz ) → 0 if s > 0 s>0 s > 0 . If s = 0 s=0 s = 0 , the integrand is invariant under y ↦ 2 − y y\mapsto2-y y ↦ 2 − y , so J R = J L J_R=J_L J R = J L . Letting z → ∞ z\to\infty z → ∞ and then δ → 0 \delta\to0 δ → 0 gives A r J ( z ) → 2 r − 1 Γ ( r ) ( 1 + 1 { s = 0 } ) A^rJ(z)\to2^{r-1}\Gamma(r)(1+\mathbb{1}_{\{s=0\}}) A r J ( z ) → 2 r − 1 Γ ( r ) ( 1 + 1 { s = 0 } ) . Finally, C r 2 r − 1 Γ ( r ) = Γ ( 2 r ) / ( 2 r Γ ( r ) ) C_r2^{r-1}\Gamma(r)=\Gamma(2r)/(2^r\Gamma(r)) C r 2 r − 1 Γ ( r ) = Γ ( 2 r ) / ( 2 r Γ ( r )) . □
Proposition 12 (Identifiability, symmetric model). For θ = ( s , r , t ) \theta=(s,r,t) θ = ( s , r , t ) and θ ′ = ( s ′ , r ′ , t ′ ) \theta'=(s',r',t') θ ′ = ( s ′ , r ′ , t ′ ) in R × ( 0 , ∞ ) × ( 0 , ∞ ) \mathbb{R}\times(0,\infty)\times(0,\infty) R × ( 0 , ∞ ) × ( 0 , ∞ ) , f θ = f θ ′ f_\theta=f_{\theta'} f θ = f θ ′ if and only if ( ∣ s ∣ , r , t ) = ( ∣ s ′ ∣ , r ′ , t ′ ) (|s|,r,t)=(|s'|,r',t') ( ∣ s ∣ , r , t ) = ( ∣ s ′ ∣ , r ′ , t ′ ) .
Proof. Because W = d − W W\mathrel{\mathop{=}^{d}}-W W = d − W , we have f ( s , r , t ) = f ( − s , r , t ) f_{(s,r,t)}=f_{(-s,r,t)} f ( s , r , t ) = f ( − s , r , t ) , so it suffices to take s , s ′ ≥ 0 s,s'\ge0 s , s ′ ≥ 0 . If f θ = f θ ′ f_\theta=f_{\theta'} f θ = f θ ′ , then M θ = M θ ′ M_\theta=M_{\theta'} M θ = M θ ′ on R \mathbb{R} R . By Lemma 11, each of the following limits depends on M θ M_\theta M θ alone: t 2 = lim z → ∞ 2 log M θ ( z ) / z 2 t^2=\lim_{z\to\infty}2\log M_\theta(z)/z^2 t 2 = lim z → ∞ 2 log M θ ( z ) / z 2 ; s = lim z → ∞ ( log M θ ( z ) − t 2 z 2 / 2 ) / z s=\lim_{z\to\infty}(\log M_\theta(z)-t^2z^2/2)/z s = lim z → ∞ ( log M θ ( z ) − t 2 z 2 /2 ) / z , since r log A θ ( z ) = O ( log z ) r\log A_\theta(z)=O(\log z) r log A θ ( z ) = O ( log z ) ; r = − 1 2 lim z → ∞ ( log M θ ( z ) − t 2 z 2 / 2 − s z ) / log z r=-\frac{1}{2}\lim_{z\to\infty}(\log M_\theta(z)-t^2z^2/2-sz)/\log z r = − 2 1 lim z → ∞ ( log M θ ( z ) − t 2 z 2 /2 − sz ) / log z , since log A θ ( z ) = 2 log z + log ( t 2 / 2 ) + o ( 1 ) \log A_\theta(z)=2\log z+\log(t^2/2)+o(1) log A θ ( z ) = 2 log z + log ( t 2 /2 ) + o ( 1 ) . Hence ( s , r , t ) = ( s ′ , r ′ , t ′ ) (s,r,t)=(s',r',t') ( s , r , t ) = ( s ′ , r ′ , t ′ ) . □
The two-type model
Proof of Lemma 1. Given U = u U=u U = u , X ∼ N ( μ 2 + u Δ , σ 1 2 u 2 + σ 2 2 ( 1 − u ) 2 ) X\sim\mathcal{N}(\mu_2+u\Delta,\sigma_1^2u^2+\sigma_2^2(1-u)^2) X ∼ N ( μ 2 + u Δ , σ 1 2 u 2 + σ 2 2 ( 1 − u ) 2 ) , so, with p ( u ) = u a − 1 ( 1 − u ) b − 1 / B ( a , b ) p(u)=u^{a-1}(1-u)^{b-1}/B(a,b) p ( u ) = u a − 1 ( 1 − u ) b − 1 / B ( a , b ) the density of U U U , M ( z ) = E e φ z ( U ) M(z)=\mathbb{E}e^{\varphi_z(U)} M ( z ) = E e φ z ( U ) with φ z ( u ) = z ( μ 2 + u Δ ) + 1 2 z 2 ( σ 1 2 u 2 + σ 2 2 ( 1 − u ) 2 ) \varphi_z(u)=z(\mu_2+u\Delta)+\frac{1}{2}z^2(\sigma_1^2u^2+\sigma_2^2(1-u)^2) φ z ( u ) = z ( μ 2 + u Δ ) + 2 1 z 2 ( σ 1 2 u 2 + σ 2 2 ( 1 − u ) 2 ) . Put T ( z ) = μ 1 z + 1 2 σ 1 2 z 2 = φ z ( 1 ) T(z)=\mu_1z+\frac{1}{2}\sigma_1^2z^2=\varphi_z(1) T ( z ) = μ 1 z + 2 1 σ 1 2 z 2 = φ z ( 1 ) , B z = σ 1 2 z 2 + Δ z B_z=\sigma_1^2z^2+\Delta z B z = σ 1 2 z 2 + Δ z , q = 1 2 ( σ 1 2 + σ 2 2 ) ≤ σ 1 2 q=\frac{1}{2}(\sigma_1^2+\sigma_2^2)\le\sigma_1^2 q = 2 1 ( σ 1 2 + σ 2 2 ) ≤ σ 1 2 , and δ = 1 2 \delta=\frac{1}{2} δ = 2 1 .
Away from $u=1. For u ∈ [ 0 , 1 − δ ] u\in[0,1-\delta] u ∈ [ 0 , 1 − δ ] , φ z ( u ) − T ( z ) = − ( 1 − u ) [ z Δ + 1 2 z 2 ( σ 1 2 ( 1 + u ) − σ 2 2 ( 1 − u ) ) ] \varphi_z(u)-T(z)=-(1-u)[z\Delta+\frac{1}{2}z^2(\sigma_1^2(1+u)-\sigma_2^2(1-u))] φ z ( u ) − T ( z ) = − ( 1 − u ) [ z Δ + 2 1 z 2 ( σ 1 2 ( 1 + u ) − σ 2 2 ( 1 − u ))] . If σ 1 > σ 2 \sigma_1>\sigma_2 σ 1 > σ 2 , the bracket is at least z Δ + 1 2 z 2 ( σ 1 2 − σ 2 2 ) ≥ 1 4 ( σ 1 2 − σ 2 2 ) z 2 z\Delta+\frac{1}{2}z^2(\sigma_1^2-\sigma_2^2)\ge\frac{1}{4}(\sigma_1^2-\sigma_2^2)z^2 z Δ + 2 1 z 2 ( σ 1 2 − σ 2 2 ) ≥ 4 1 ( σ 1 2 − σ 2 2 ) z 2 for large z z z . If σ 1 = σ 2 \sigma_1=\sigma_2 σ 1 = σ 2 , it is at least z Δ z\Delta z Δ with Δ > 0 \Delta>0 Δ > 0 . Either way ∫ 0 1 − δ e φ z ( u ) − T ( z ) p ( u ) d u ≤ e − c z 2 \int_0^{1-\delta}e^{\varphi_z(u)-T(z)}p(u)\,du\le e^{-cz^2} ∫ 0 1 − δ e φ z ( u ) − T ( z ) p ( u ) d u ≤ e − c z 2 for some c > 0 c>0 c > 0 and large z z z , which is O ( z − k ) O(z^{-k}) O ( z − k ) relative to B z − b B_z^{-b} B z − b for every k k k . Near u = 1 u=1 u = 1 . With y = 1 − u y=1-u y = 1 − u , φ z ( 1 − y ) − T ( z ) = − B z y + q z 2 y 2 \varphi_z(1-y)-T(z)=-B_z y+q_z^2y^2 φ z ( 1 − y ) − T ( z ) = − B z y + q z 2 y 2 and p ( 1 − y ) = y b − 1 ( 1 − y ) a − 1 / B ( a , b ) p(1-y)=y^{b-1}(1-y)^{a-1}/B(a,b) p ( 1 − y ) = y b − 1 ( 1 − y ) a − 1 / B ( a , b ) . Substituting y = τ / B z y=\tau/B_z y = τ / B z ,
∫ 0 δ e − B z y + q z 2 y 2 y b − 1 ( 1 − y ) a − 1 d y = B z − b ∫ 0 δ B z e − τ τ b − 1 g z ( τ ) d τ , g z ( τ ) = e κ z τ 2 ( 1 − τ B z ) a − 1 . \int_0^\delta e^{-B_z y+q_z^2y^2}y^{b-1}(1-y)^{a-1}\,dy=B_z^{-b}\int_0^{\delta B_z}e^{-\tau}\tau^{b-1}g_z(\tau)\,d\tau,\qquad g_z(\tau)=e^{\kappa_z\tau^2}\left(1-\frac{\tau}{B_z}\right)^{a-1}. ∫ 0 δ e − B z y + q z 2 y 2 y b − 1 ( 1 − y ) a − 1 d y = B z − b ∫ 0 δ B z e − τ τ b − 1 g z ( τ ) d τ , g z ( τ ) = e κ z τ 2 ( 1 − B z τ ) a − 1 .
with κ z = q z 2 / B z 2 = O ( z − 2 ) \kappa_z=q_z^2/B_z^2=O(z^{-2}) κ z = q z 2 / B z 2 = O ( z − 2 ) and 1 / B z = O ( z − 2 ) 1/B_z=O(z^{-2}) 1/ B z = O ( z − 2 ) . On τ ≤ z \tau\le\sqrt{z} τ ≤ z , Taylor expansion gives g z ( τ ) = 1 + κ z τ 2 − ( a − 1 ) τ / B z + R z ( τ ) g_z(\tau)=1+\kappa_z\tau^2-(a-1)\tau/B_z+R_z(\tau) g z ( τ ) = 1 + κ z τ 2 − ( a − 1 ) τ / B z + R z ( τ ) with ∣ R z ( τ ) ∣ ≤ C z − 4 ( 1 + τ 4 ) \lvert R_z(\tau)\rvert\le Cz^{-4}(1+\tau^4) ∣ R z ( τ )∣ ≤ C z − 4 ( 1 + τ 4 ) . On z < τ ≤ δ B z \sqrt{z}<\tau\le\delta B_z z < τ ≤ δ B z , κ z τ ≤ q δ z 2 / B z → q δ / σ 1 2 ≤ 1 2 \kappa_z\tau\le q\delta z^2/B_z\to q\delta/\sigma_1^2\le\frac12 κ z τ ≤ q δ z 2 / B z → q δ / σ 1 2 ≤ 2 1 , so e − τ + κ z τ 2 ≤ e − τ / 2 e^{-\tau+\kappa_z\tau^2}\le e^{-\tau/2} e − τ + κ z τ 2 ≤ e − τ /2 for large z z z , and ( 1 − τ / B z ) a − 1 ≤ max ( 1 , 2 1 − a ) (1-\tau/B_z)^{a-1}\le\max(1,2^{1-a}) ( 1 − τ / B z ) a − 1 ≤ max ( 1 , 2 1 − a ) ; this range contributes O ( e − z / 4 ) O(e^{-\sqrt{z}/4}) O ( e − z /4 ) , as do the tails beyond z \sqrt{z} z of the integrals of the Taylor terms. Therefore
∫ 0 δ B z e − τ τ b − 1 g z ( τ ) d τ = Γ ( b ) [ 1 + κ z b ( b + 1 ) − ( a − 1 ) b B z ] + O ( z − 4 ) = Γ ( b ) [ 1 + K z 2 + O ( z − 3 ) ] , \int_0^{\delta B_z}e^{-\tau}\tau^{b-1}g_z(\tau)\,d\tau=\Gamma(b)\left[1+\kappa_z b(b+1)-\frac{(a-1)b}{B_z}\right]+O(z^{-4})=\Gamma(b)\left[1+\frac{K}{z^2}+O(z^{-3})\right], ∫ 0 δ B z e − τ τ b − 1 g z ( τ ) d τ = Γ ( b ) [ 1 + κ z b ( b + 1 ) − B z ( a − 1 ) b ] + O ( z − 4 ) = Γ ( b ) [ 1 + z 2 K + O ( z − 3 ) ] ,
with K = b ( b + 1 ) ( σ 1 2 + σ 2 2 ) 2 σ 1 4 − ( a − 1 ) b σ 1 2 K=\frac{b(b+1)(\sigma_1^2+\sigma_2^2)}{2\sigma_1^4}-\frac{(a-1)b}{\sigma_1^2} K = 2 σ 1 4 b ( b + 1 ) ( σ 1 2 + σ 2 2 ) − σ 1 2 ( a − 1 ) b , using κ z = q σ 1 4 z 2 + O ( z − 3 ) \kappa_z=\frac{q}{\sigma_1^4z^2}+O(z^{-3}) κ z = σ 1 4 z 2 q + O ( z − 3 ) and B z − 1 = 1 σ 1 2 z 2 + O ( z − 3 ) B_z^{-1}=\frac{1}{\sigma_1^2z^2}+O(z^{-3}) B z − 1 = σ 1 2 z 2 1 + O ( z − 3 ) .
Assembly. log M ( z ) = T ( z ) − b log B z + log Γ ( b ) B ( a , b ) + K z 2 + O ( z − 3 ) \log M(z)=T(z)-b\log B_z+\log\frac{\Gamma(b)}{B(a,b)}+\frac{K}{z^2}+O(z^{-3}) log M ( z ) = T ( z ) − b log B z + log B ( a , b ) Γ ( b ) + z 2 K + O ( z − 3 ) , and − b log B z = − 2 b log z − b log σ 1 2 − b Δ σ 1 2 z + b Δ 2 2 σ 1 4 z 2 + O ( z − 3 ) -b\log B_z=-2b\log z-b\log\sigma_1^2-\frac{b\Delta}{\sigma_1^2z}+\frac{b\Delta^2}{2\sigma_1^4z^2}+O(z^{-3}) − b log B z = − 2 b log z − b log σ 1 2 − σ 1 2 z b Δ + 2 σ 1 4 z 2 b Δ 2 + O ( z − 3 ) .
Numerical check. The expansion was checked with mpmath at 50 digits for three parameter sets, including μ 1 = μ 2 \mu_1=\mu_2 μ 1 = μ 2 with σ 1 ≠ σ 2 \sigma_1\ne\sigma_2 σ 1 = σ 2 . After subtracting the expansion through orders 0, 1 and 2, the residuals shrink by factors of 2, 4 and 8 per doubling of z z z (by 16 at order 2 when Δ = 0 \Delta=0 Δ = 0 , where the odd terms vanish).
Proof of Proposition 2. The statement compares parameter vectors within the class ( μ 1 , σ 1 ) ≠ ( μ 2 , σ 2 ) (\mu_1,\sigma_1)\ne(\mu_2,\sigma_2) ( μ 1 , σ 1 ) = ( μ 2 , σ 2 ) . Whether a law in this class can also arise from the excluded case, where both endpoints of [ 0 , 1 ] [0,1] [ 0 , 1 ] contribute to the tail of M M M , is not settled by the expansion, and we do not claim it; within the symmetric family, Proposition 12 covers that case (s = 0 s=0 s = 0 ). The label swap preserves the law because 1 − U ∼ Beta ( b , a ) 1-U\sim\operatorname{Beta}(b,a) 1 − U ∼ Beta ( b , a ) , and exactly one of the two labelings satisfies ( σ 1 , μ 1 ) ≻ ( σ 2 , μ 2 ) (\sigma_1,\mu_1)\succ(\sigma_2,\mu_2) ( σ 1 , μ 1 ) ≻ ( σ 2 , μ 2 ) . Let ϑ , ϑ ′ \vartheta,\vartheta' ϑ , ϑ ′ both satisfy this ordering and induce the same law. Then M = M ′ M=M' M = M ′ on R \mathbb{R} R , and since z 2 , z , log z , 1 , z − 1 , z − 2 z^2,z,\log z,1,z^{-1},z^{-2} z 2 , z , log z , 1 , z − 1 , z − 2 form an asymptotic scale, the coefficients in Lemma 1 agree:
the z 2 z^2 z 2 coefficient gives σ 1 = σ 1 ′ \sigma_1=\sigma'_1 σ 1 = σ 1 ′ ;
the z z z coefficient gives μ 1 = μ 1 ′ \mu_1=\mu'_1 μ 1 = μ 1 ′ ;
the log z \log z log z coefficient gives b = b ′ b=b' b = b ′ ;
C 0 = C 0 ′ C_0=C'_0 C 0 = C 0 ′ gives B ( a , b ) = B ( a ′ , b ′ ) B(a,b)=B(a',b') B ( a , b ) = B ( a ′ , b ′ ) , hence a = a ′ a=a' a = a ′ , because ∂ a log B ( a , b ) = ψ ( a ) − ψ ( a + b ) < 0 \partial_a\log B(a,b)=\psi(a)-\psi(a+b)<0 ∂ a log B ( a , b ) = ψ ( a ) − ψ ( a + b ) < 0 ;
C 1 = C 1 ′ C_1=C'_1 C 1 = C 1 ′ gives Δ = Δ ′ \Delta=\Delta' Δ = Δ ′ , hence μ 2 = μ 2 ′ \mu_2=\mu'_2 μ 2 = μ 2 ′ ;
C 2 = C 2 ′ C_2=C'_2 C 2 = C 2 ′ gives σ 2 2 = σ 2 ′ 2 \sigma_2^2=\sigma_2'^2 σ 2 2 = σ 2 ′2 , because σ 2 2 \sigma_2^2 σ 2 2 enters C 2 C_2 C 2 linearly with coefficient b ( b + 1 ) / ( 2 σ 1 4 ) > 0 b(b+1)/(2\sigma_1^4)>0 b ( b + 1 ) / ( 2 σ 1 4 ) > 0 and every other term already agrees.
Proofs for Sections 4 and 5
Proof of Lemma 4. W W W is symmetric and W 2 ∼ Beta ( 1 2 , r ) W^2 \sim\operatorname{Beta}\left(\frac{1}{2},r\right) W 2 ∼ Beta ( 2 1 , r ) , so m 2 : = E W 2 = 1 / ( 2 r + 1 ) m_2 := \mathbb{E}W^2 = 1/(2r+1) m 2 := E W 2 = 1/ ( 2 r + 1 ) and m 4 : = E W 4 = 3 / ( ( 2 r + 1 ) ( 2 r + 3 ) ) m_4 := \mathbb{E}W^4 = 3/((2r+1)(2r+3)) m 4 := E W 4 = 3/ (( 2 r + 1 ) ( 2 r + 3 )) . Conditioning on W W W , with μ = s W \mu=sW μ = s W and σ 2 = t 2 ( 1 + W 2 ) / 2 \sigma^2=t^2(1+W^2)/2 σ 2 = t 2 ( 1 + W 2 ) /2 , E X 2 = E [ μ 2 + σ 2 ] \mathbb{E}X^2=\mathbb{E}[\mu^2+\sigma^2] E X 2 = E [ μ 2 + σ 2 ] and E X 4 = E [ μ 4 + 6 μ 2 σ 2 + 3 σ 4 ] \mathbb{E}X^4=\mathbb{E}[\mu^4+6\mu^2\sigma^2+3\sigma^4] E X 4 = E [ μ 4 + 6 μ 2 σ 2 + 3 σ 4 ] . Substituting gives κ 4 = E X 4 − 3 v 2 = s 4 ( m 4 − 3 m 2 2 ) + 3 s 2 t 2 ( m 4 − m 2 2 ) + 3 4 t 4 ( m 4 − m 2 2 ) \kappa_4=\mathbb{E}X^4-3v^2=s^4(m_4-3m_2^2)+3s^2t^2(m_4-m_2^2)+\frac{3}{4}t^4(m_4-m_2^2) κ 4 = E X 4 − 3 v 2 = s 4 ( m 4 − 3 m 2 2 ) + 3 s 2 t 2 ( m 4 − m 2 2 ) + 4 3 t 4 ( m 4 − m 2 2 ) , and m 4 − m 2 2 = 4 r / ( ( 2 r + 1 ) 2 ( 2 r + 3 ) ) m_4-m_2^2=4r/((2r+1)^2(2r+3)) m 4 − m 2 2 = 4 r / (( 2 r + 1 ) 2 ( 2 r + 3 )) , m 4 − 3 m 2 2 = − 6 / ( ( 2 r + 1 ) 2 ( 2 r + 3 ) ) m_4-3m_2^2=-6/((2r+1)^2(2r+3)) m 4 − 3 m 2 2 = − 6/ (( 2 r + 1 ) 2 ( 2 r + 3 )) . Odd cumulants vanish by the symmetry X = d − X X \mathrel{\mathop=^{d}} -X X = d − X .
Proof of Proposition 5. Lower bound: E X 4 ≥ ( E X 2 ) 2 \mathbb{E}X^4\geq(\mathbb{E}X^2)^2 E X 4 ≥ ( E X 2 ) 2 with equality only if ∣ X ∣ |X| ∣ X ∣ is a.s. constant, which is impossible since f θ f_\theta f θ is a density; hence γ 2 = E X 4 / v 2 − 3 > − 2 \gamma_2=\mathbb{E}X^4/v^2-3>-2 γ 2 = E X 4 / v 2 − 3 > − 2 . Upper bound: drop the − 6 s 4 -6s^4 − 6 s 4 term and use ( s 2 + ( r + 1 ) t 2 ) 2 ≥ max { 4 ( r + 1 ) s 2 t 2 , ( r + 1 ) 2 t 4 } (s^2+(r+1)t^2)^2\geq\max\{4(r+1)s^2t^2,(r+1)^2t^4\} ( s 2 + ( r + 1 ) t 2 ) 2 ≥ max { 4 ( r + 1 ) s 2 t 2 , ( r + 1 ) 2 t 4 } on the two remaining terms separately: γ 2 ≤ 3 r ( 2 r + 3 ) ( r + 1 ) + 3 r ( 2 r + 3 ) ( r + 1 ) 2 < 3 2 + 3 2 = 3 \gamma_2\leq\frac{3r}{(2r+3)(r+1)}+\frac{3r}{(2r+3)(r+1)^2}<\frac{3}{2}+\frac{3}{2}=3 γ 2 ≤ ( 2 r + 3 ) ( r + 1 ) 3 r + ( 2 r + 3 ) ( r + 1 ) 2 3 r < 2 3 + 2 3 = 3 . (Numerically, sup θ γ 2 ≈ 0.24 \sup_\theta\gamma_2\approx0.24 sup θ γ 2 ≈ 0.24 .) Sign: the numerator of κ 4 \kappa_4 κ 4 is negative iff r ( 12 s 2 + 3 t 4 ) < 6 s 4 r(12s^2+3t^4)<6s^4 r ( 12 s 2 + 3 t 4 ) < 6 s 4 , i.e. r < 2 d 4 / ( 4 d 2 + 1 ) r<2d^4/(4d^2+1) r < 2 d 4 / ( 4 d 2 + 1 ) .
Proof of Proposition 6. Well-definedness is Proposition 5: t G 2 > 0 ⟺ γ 2 > − 2 t_G^2>0\Longleftrightarrow\gamma_2>-2 t G 2 > 0 ⟺ γ 2 > − 2 and v ( 1 − γ 2 / 3 ) > 0 ⟺ γ 2 < 3 v(1-\sqrt{\gamma_2/3})>0\Longleftrightarrow\gamma_2<3 v ( 1 − γ 2 /3 ) > 0 ⟺ γ 2 < 3 . Case γ 2 < 0 \gamma_2<0 γ 2 < 0 : write Y = s G B + t G Z Y=s_GB+t_GZ Y = s G B + t G Z with B B B uniform on { ± 1 } \{\pm1\} { ± 1 } independent of Z Z Z . Cumulants add over independent summands, κ k ( t G Z ) = 0 \kappa_k(t_GZ)=0 κ k ( t G Z ) = 0 for k ≥ 3 k\geq3 k ≥ 3 , and B B B has κ 2 = 1 \kappa_2=1 κ 2 = 1 , κ 4 = E B 4 − 3 = − 2 \kappa_4=\mathbb{E}B^4-3=-2 κ 4 = E B 4 − 3 = − 2 and vanishing odd cumulants. Hence κ 2 ( Y ) = s G 2 + t G 2 = v \kappa_2(Y)=s_G^2+t_G^2=v κ 2 ( Y ) = s G 2 + t G 2 = v and κ 4 ( Y ) = − 2 s G 4 = γ 2 v 2 = κ 4 ( θ ) \kappa_4(Y)=-2s_G^4=\gamma_2v^2=\kappa_4(\theta) κ 4 ( Y ) = − 2 s G 4 = γ 2 v 2 = κ 4 ( θ ) ; odd cumulants vanish on both sides. Case γ 2 > 0 \gamma_2>0 γ 2 > 0 : Y = σ Z Y=\sigma Z Y = σ Z with σ 2 \sigma^2 σ 2 uniform on { v ( 1 ± γ 2 / 3 ) } \{v(1\pm\sqrt{\gamma_2/3})\} { v ( 1 ± γ 2 /3 )} has κ 2 = E σ 2 = v \kappa_2=\mathbb{E}\sigma^2=v κ 2 = E σ 2 = v and κ 4 = 3 Var ( σ 2 ) = γ 2 v 2 \kappa_4=3\operatorname{Var}(\sigma^2)=\gamma_2v^2 κ 4 = 3 Var ( σ 2 ) = γ 2 v 2 . The limit statement follows from W → ± 1 W\to\pm1 W → ± 1 in distribution as r → 0 r\to0 r → 0 and ( 1 + W 2 ) / 2 → 1 (1+W^2)/2\to1 ( 1 + W 2 ) /2 → 1 .
Proof of Theorem 7. The minimal error sum over tests equals 1 − TV ( f θ ⊗ n , G θ ⊗ n ) 1-\operatorname{TV}(f_\theta^{\otimes n},G_\theta^{\otimes n}) 1 − TV ( f θ ⊗ n , G θ ⊗ n ) . By Pinsker’s inequality and additivity of KL over products, TV ( f θ ⊗ n , G θ ⊗ n ) ≤ n KL ( f θ ∥ G θ ) / 2 = 1 2 n / n h a r d ( θ ) \operatorname{TV}(f_\theta^{\otimes n},G_\theta^{\otimes n})\leq\sqrt{n\operatorname{KL}(f_\theta\Vert G_\theta)/2}=\frac{1}{2}\sqrt{n/n_{\mathrm{hard}}(\theta)} TV ( f θ ⊗ n , G θ ⊗ n ) ≤ n KL ( f θ ∥ G θ ) /2 = 2 1 n / n hard ( θ ) . n h a r d n_{\mathrm{hard}} n hard is positive because KL ( f θ ∥ G θ ) < ∞ \operatorname{KL}(f_\theta\Vert G_\theta)<\infty KL ( f θ ∥ G θ ) < ∞ : log ( f θ / G θ ) \log(f_\theta/G_\theta) log ( f θ / G θ ) grows at most quadratically in x x x , because both are finite or continuous Gaussian location-scale mixtures with variances bounded away from zero, and f θ f_\theta f θ has Gaussian tails. n h a r d n_{\mathrm{hard}} n hard is finite because KL ( f θ ∥ G θ ) > 0 \operatorname{KL}(f_\theta\Vert G_\theta)>0 KL ( f θ ∥ G θ ) > 0 , that is, f θ ≠ G θ f_\theta\neq G_\theta f θ = G θ : by Lemma 11, log M θ ( z ) \log M_\theta(z) log M θ ( z ) contains the term − r log A θ ( z ) ∼ − 2 r log z -r\log A_\theta(z)\sim-2r\log z − r log A θ ( z ) ∼ − 2 r log z as z → ∞ z\to\infty z → ∞ , whereas the log moment generating function of every candidate G θ G_\theta G θ is a quadratic in z z z plus a bounded term ( t G 2 z 2 / 2 + s G z − log 2 + o ( 1 ) , v z 2 / 2 , or v + z 2 / 2 − log 2 + o ( 1 ) ) (t_G^2z^2/2+s_Gz-\log2+o(1), vz^2/2, \text{or }v_+z^2/2-\log2+o(1)) ( t G 2 z 2 /2 + s G z − log 2 + o ( 1 ) , v z 2 /2 , or v + z 2 /2 − log 2 + o ( 1 )) . Dependence on ( d , c ) (d,c) ( d , c ) only follows from the scaling identity in Section 2, since G θ G_\theta G θ is built from ( v , γ 2 ) (v,\gamma_2) ( v , γ 2 ) in a scale-equivariant way.
Proposition 13 (Tightness for a fixed pair). Let H 2 = H 2 ( f θ , G θ ) = 1 2 ∫ ( f θ − G θ ) 2 H^2=H^2(f_\theta,G_\theta)=\frac{1}{2}\int(\sqrt{f_\theta}-\sqrt{G_\theta})^2 H 2 = H 2 ( f θ , G θ ) = 2 1 ∫ ( f θ − G θ ) 2 . Every test ϕ \phi ϕ based on n n n observations satisfies
f θ ⊗ n ( ϕ = 1 ) + G θ ⊗ n ( ϕ = 0 ) ≥ 1 2 ( 1 − H 2 ) 2 n , f_\theta^{\otimes n}(\phi=1)+G_\theta^{\otimes n}(\phi=0)\geq\frac{1}{2}(1-H^2)^{2n}, f θ ⊗ n ( ϕ = 1 ) + G θ ⊗ n ( ϕ = 0 ) ≥ 2 1 ( 1 − H 2 ) 2 n ,
and the likelihood-ratio test ϕ ⋆ = 1 { ∏ i G θ ( X i ) > ∏ i f θ ( X i ) } \phi^\star=\mathbb{1}\{\prod_iG_\theta(X_i)>\prod_i f_\theta(X_i)\} ϕ ⋆ = 1 { ∏ i G θ ( X i ) > ∏ i f θ ( X i )} satisfies
f θ ⊗ n ( ϕ ⋆ = 1 ) + G θ ⊗ n ( ϕ ⋆ = 0 ) ≤ ( 1 − H 2 ) n . f_\theta^{\otimes n}(\phi^\star=1)+G_\theta^{\otimes n}(\phi^\star=0)\leq(1-H^2)^n. f θ ⊗ n ( ϕ ⋆ = 1 ) + G θ ⊗ n ( ϕ ⋆ = 0 ) ≤ ( 1 − H 2 ) n .
Consequently, the smallest n n n at which some test has summed error at most ϵ < 1 2 \epsilon<\frac{1}{2} ϵ < 2 1 lies between log ( 1 / ( 2 ϵ ) ) / { − 2 log ( 1 − H 2 ) } \log(1/(2\epsilon))/\{-2\log(1-H^2)\} log ( 1/ ( 2 ϵ )) / { − 2 log ( 1 − H 2 )} and log ( 1 / ϵ ) / { − log ( 1 − H 2 ) } \log(1/\epsilon)/\{-\log(1-H^2)\} log ( 1/ ϵ ) / { − log ( 1 − H 2 )} , which to first order in H 2 H^2 H 2 are log ( 1 / ( 2 ϵ ) ) / ( 2 H 2 ) \log(1/(2\epsilon))/(2H^2) log ( 1/ ( 2 ϵ )) / ( 2 H 2 ) and log ( 1 / ϵ ) / H 2 \log(1/\epsilon)/H^2 log ( 1/ ϵ ) / H 2 .
Proof. The minimal error sum over tests is 1 − TV ( f θ ⊗ n , G θ ⊗ n ) = ∫ min ( f θ ⊗ n , G θ ⊗ n ) 1-\operatorname{TV}(f_\theta^{\otimes n},G_\theta^{\otimes n})=\int\min(f_\theta^{\otimes n},G_\theta^{\otimes n}) 1 − TV ( f θ ⊗ n , G θ ⊗ n ) = ∫ min ( f θ ⊗ n , G θ ⊗ n ) . Le Cam’s inequality [24 Lemma 2.3] bounds it below by 1 2 ( ∫ f θ ⊗ n G θ ⊗ n ) 2 = 1 2 ( 1 − H 2 ) 2 n \frac{1}{2}\left(\int\sqrt{f_\theta^{\otimes n}G_\theta^{\otimes n}}\right)^2=\frac{1}{2}(1-H^2)^{2n} 2 1 ( ∫ f θ ⊗ n G θ ⊗ n ) 2 = 2 1 ( 1 − H 2 ) 2 n . The error sum of ϕ ⋆ \phi^\star ϕ ⋆ equals ∫ min ( f θ ⊗ n , G θ ⊗ n ) ≤ ∫ f θ ⊗ n G θ ⊗ n = ( 1 − H 2 ) n \int\min(f_\theta^{\otimes n},G_\theta^{\otimes n})\leq\int\sqrt{f_\theta^{\otimes n}G_\theta^{\otimes n}}=(1-H^2)^n ∫ min ( f θ ⊗ n , G θ ⊗ n ) ≤ ∫ f θ ⊗ n G θ ⊗ n = ( 1 − H 2 ) n . Solving each bound for n n n at error ϵ \epsilon ϵ gives the two ends.
At ϵ = 0.05 \epsilon= 0.05 ϵ = 0.05 the two ends differ by the factor log 20 / ( 1 2 log 10 ) ≈ 2.6 \log20/(\frac{1}{2}\log10) \approx2.6 log 20/ ( 2 1 log 10 ) ≈ 2.6 , whatever H 2 H^2 H 2 . Numerically, for the closed-form look-alike at d = 3 d = 3 d = 3 , H 2 / K L ∈ [ 0.250 , 0.252 ] H^2/\mathrm{KL} \in[0.250, 0.252] H 2 / KL ∈ [ 0.250 , 0.252 ] for c ∈ { 1 , 2 , 3 , 4 , 6 , 8 } c \in\{1, 2, 3, 4, 6, 8\} c ∈ { 1 , 2 , 3 , 4 , 6 , 8 } , so the two ends are about 9.1 n h a r d 9.1\,n_{\mathrm{hard}} 9.1 n hard and 24 n h a r d 24\,n_{\mathrm{hard}} 24 n hard (Table 4).
Lemma 14 (Continuity at hard membership). For γ 2 ( θ ) < 0 \gamma_2(\theta) < 0 γ 2 ( θ ) < 0 and θ ϵ = ( s G , ϵ / 2 , t G ) \theta_\epsilon= (s_G, \epsilon/2, t_G) θ ϵ = ( s G , ϵ /2 , t G ) , K L ( f θ ∥ f θ ϵ ) → K L ( f θ ∥ G θ ) \mathrm{KL}(f_\theta\Vert f_{\theta_\epsilon}) \to\mathrm{KL}(f_\theta\Vert G_\theta) KL ( f θ ∥ f θ ϵ ) → KL ( f θ ∥ G θ ) as ϵ ↓ 0 \epsilon\downarrow0 ϵ ↓ 0 .
Proof. Under θ ϵ \theta_\epsilon θ ϵ , given U = u U = u U = u the density of X X X at x x x is g x ( u ) = φ ( x ; s G ( 2 u − 1 ) , t G 2 ( u 2 + ( 1 − u ) 2 ) ) g_x(u) = \varphi(x; s_G(2u - 1), t_G^2(u^2 + (1-u)^2)) g x ( u ) = φ ( x ; s G ( 2 u − 1 ) , t G 2 ( u 2 + ( 1 − u ) 2 )) , with mean in [ − s G , s G ] [-s_G, s_G] [ − s G , s G ] and variance in [ t G 2 / 2 , t G 2 ] [t_G^2/2, t_G^2] [ t G 2 /2 , t G 2 ] . For fixed x x x , u ↦ g x ( u ) u \mapsto g_x(u) u ↦ g x ( u ) is bounded and continuous on [ 0 , 1 ] [0,1] [ 0 , 1 ] , and Beta ( ϵ / 2 , ϵ / 2 ) ⇒ 1 2 δ 0 + 1 2 δ 1 \operatorname{Beta}(\epsilon/2,\epsilon/2) \Rightarrow\frac{1}{2}\delta_0 + \frac{1}{2}\delta_1 Beta ( ϵ /2 , ϵ /2 ) ⇒ 2 1 δ 0 + 2 1 δ 1 , so f θ ϵ ( x ) → 1 2 g x ( 0 ) + 1 2 g x ( 1 ) = G θ ( x ) f_{\theta_\epsilon}(x) \to\frac{1}{2}g_x(0) + \frac{1}{2}g_x(1) = G_\theta(x) f θ ϵ ( x ) → 2 1 g x ( 0 ) + 2 1 g x ( 1 ) = G θ ( x ) pointwise. The variance bounds give, uniformly in ϵ \epsilon ϵ , ( 2 π t G 2 ) − 1 / 2 exp { − ( ∣ x ∣ + s G ) 2 / t G 2 } ≤ f θ ϵ ( x ) ≤ ( π t G 2 ) − 1 / 2 (2\pi t_G^2)^{-1/2}\exp\{-(|x| + s_G)^2/t_G^2\} \le f_{\theta_\epsilon}(x) \le(\pi t_G^2)^{-1/2} ( 2 π t G 2 ) − 1/2 exp { − ( ∣ x ∣ + s G ) 2 / t G 2 } ≤ f θ ϵ ( x ) ≤ ( π t G 2 ) − 1/2 , so ∣ log f θ ϵ ( x ) ∣ ≤ C + ( ∣ x ∣ + s G ) 2 / t G 2 |\log f_{\theta_\epsilon}(x)| \le C + (|x| + s_G)^2/t_G^2 ∣ log f θ ϵ ( x ) ∣ ≤ C + ( ∣ x ∣ + s G ) 2 / t G 2 , which is f θ f_\theta f θ -integrable since f θ f_\theta f θ has a finite second moment. Dominated convergence gives ∫ f θ log f θ ϵ → ∫ f θ log G θ \int f_\theta\log f_{\theta_\epsilon} \to\int f_\theta\log G_\theta ∫ f θ log f θ ϵ → ∫ f θ log G θ ; ∫ f θ log f θ \int f_\theta\log f_\theta ∫ f θ log f θ is finite. □
Proof of Corollary 8. Le Cam’s two-point reduction: for θ 0 , θ 1 \theta_0,\theta_1 θ 0 , θ 1 with ∣ c ( θ 0 ) − c ( θ 1 ) ∣ ≥ 2 δ |c(\theta_0) - c(\theta_1)| \ge2\delta ∣ c ( θ 0 ) − c ( θ 1 ) ∣ ≥ 2 δ , any estimator has max i P θ i ( ∣ c ^ − c ( θ i ) ∣ ≥ δ ) ≥ 1 2 { 1 − T V ( f θ 0 ⊗ n , f θ 1 ⊗ n ) } \max_i P_{\theta_i}(|\widehat{c} - c(\theta_i)| \ge\delta) \ge\frac{1}{2}\{1 - \mathrm{TV}(f_{\theta_0}^{\otimes n}, f_{\theta_1}^{\otimes n})\} max i P θ i ( ∣ c − c ( θ i ) ∣ ≥ δ ) ≥ 2 1 { 1 − TV ( f θ 0 ⊗ n , f θ 1 ⊗ n )} . Take θ 0 = θ \theta_0 = \theta θ 0 = θ , θ 1 = ( s G , ϵ / 2 , t G ) \theta_1 = (s_G,\epsilon/2,t_G) θ 1 = ( s G , ϵ /2 , t G ) and δ = ( c − ϵ ) / 2 \delta= (c - \epsilon)/2 δ = ( c − ϵ ) /2 . By Pinsker, T V ≤ n K L ( f θ ∥ f θ 1 ) / 2 \mathrm{TV} \le\sqrt{n\mathrm{KL}(f_\theta\Vert f_{\theta_1})/2} TV ≤ n KL ( f θ ∥ f θ 1 ) /2 , and by Lemma 14 this tends to n K L ( f θ ∥ G θ ) / 2 = 1 2 n / n h a r d ≤ 1 2 \sqrt{n\mathrm{KL}(f_\theta\Vert G_\theta)/2} = \frac{1}{2}\sqrt{n/n_{\mathrm{hard}}} \le\frac{1}{2} n KL ( f θ ∥ G θ ) /2 = 2 1 n / n hard ≤ 2 1 as ϵ → 0 \epsilon\to0 ϵ → 0 . □
Proof of Proposition 9. Write s k = s r k s_k = s_{r_k} s k = s r k . Since t 2 < 2 v t^2 < 2v t 2 < 2 v , s k 2 = r k ( 2 v − t 2 ) + ( v − t 2 ) > 0 s_k^2 = r_k(2v - t^2) + (v - t^2) > 0 s k 2 = r k ( 2 v − t 2 ) + ( v − t 2 ) > 0 for large k k k . Let W k = 2 U k − 1 W_k = 2U_k - 1 W k = 2 U k − 1 with U k ∼ Beta ( r k , r k ) U_k \sim\operatorname{Beta}(r_k,r_k) U k ∼ Beta ( r k , r k ) . By the central limit theorem for symmetric Beta laws, 2 r k + 1 W k ⇒ N ( 0 , 1 ) \sqrt{2r_k + 1}W_k \Rightarrow N(0,1) 2 r k + 1 W k ⇒ N ( 0 , 1 ) , and s k 2 / ( 2 r k + 1 ) → v − t 2 / 2 s_k^2/(2r_k + 1) \to v - t^2/2 s k 2 / ( 2 r k + 1 ) → v − t 2 /2 , so s k W k ⇒ N ( 0 , v − t 2 / 2 ) s_kW_k \Rightarrow N(0,v - t^2/2) s k W k ⇒ N ( 0 , v − t 2 /2 ) while W k 2 → 0 W_k^2 \to0 W k 2 → 0 in probability. By Slutsky’s lemma, ( s k W k , W k 2 ) ⇒ ( N , 0 ) (s_kW_k,W_k^2) \Rightarrow(N,0) ( s k W k , W k 2 ) ⇒ ( N , 0 ) jointly, with N ∼ N ( 0 , v − t 2 / 2 ) N \sim N(0,v - t^2/2) N ∼ N ( 0 , v − t 2 /2 ) . For fixed x x x , the map ( m , w ) ↦ φ ( x ; m , t 2 ( 1 + w ) / 2 ) (m,w) \mapsto\varphi(x;m,t^2(1+w)/2) ( m , w ) ↦ φ ( x ; m , t 2 ( 1 + w ) /2 ) is bounded and continuous on R × [ 0 , 1 ] \mathbb{R} \times[0,1] R × [ 0 , 1 ] , so by the definition of weak convergence (the portmanteau theorem) f θ k ( x ) = E φ ( x ; s k W k , t 2 ( 1 + W k 2 ) / 2 ) → E φ ( x ; N , t 2 / 2 ) = φ ( x ; 0 , v ) f_{\theta_k}(x) = \mathbb{E}\,\varphi(x;s_kW_k,t^2(1+W_k^2)/2) \to\mathbb{E}\,\varphi(x;N,t^2/2) = \varphi(x;0,v) f θ k ( x ) = E φ ( x ; s k W k , t 2 ( 1 + W k 2 ) /2 ) → E φ ( x ; N , t 2 /2 ) = φ ( x ; 0 , v ) . Scheffé’s lemma upgrades pointwise convergence of densities to total variation. Finally, T V ( f θ ⊗ n , f θ k ⊗ n ) ≤ T V ( f θ ⊗ n , N ( 0 , v ) ⊗ n ) + n T V ( N ( 0 , v ) , f θ k ) \mathrm{TV}(f_{\theta}^{\otimes n},f_{\theta_k}^{\otimes n}) \le\mathrm{TV}(f_{\theta}^{\otimes n},N(0,v)^{\otimes n}) + n\mathrm{TV}(N(0,v),f_{\theta_k}) TV ( f θ ⊗ n , f θ k ⊗ n ) ≤ TV ( f θ ⊗ n , N ( 0 , v ) ⊗ n ) + n TV ( N ( 0 , v ) , f θ k ) ; the first term is bounded by Pinsker as in Theorem 7 , and the second tends to 0 0 0 for fixed n n n . □
Proof of Proposition 10. From Lemma 4, γ 2 = − 6 s 4 + 12 r s 2 t 2 + 3 r t 4 ( 2 r + 3 ) ( s 2 + ( r + 1 ) t 2 ) 2 \gamma_2 = \frac{-6s^4 + 12rs^2t^2 + 3rt^4}{(2r+3)(s^2+(r+1)t^2)^2} γ 2 = ( 2 r + 3 ) ( s 2 + ( r + 1 ) t 2 ) 2 − 6 s 4 + 12 r s 2 t 2 + 3 r t 4 . (a) Substitute s 2 = r v ( 2 − ρ ) + v ( 1 − ρ ) s^2 = rv(2-\rho)+v(1-\rho) s 2 = r v ( 2 − ρ ) + v ( 1 − ρ ) : the numerator is 6 r 2 v 2 ( 2 − ρ ) ( 3 ρ − 2 ) + O ( r ) 6r^2v^2(2-\rho)(3\rho-2)+O(r) 6 r 2 v 2 ( 2 − ρ ) ( 3 ρ − 2 ) + O ( r ) and the denominator is 8 r 3 v 2 + O ( r 2 ) 8r^3v^2+O(r^2) 8 r 3 v 2 + O ( r 2 ) . (b) The numerator is r ( 12 s 2 t 2 + 3 t 4 ) + O ( 1 ) r(12s^2t^2+3t^4)+O(1) r ( 12 s 2 t 2 + 3 t 4 ) + O ( 1 ) and the denominator is 2 r 3 t 4 + O ( r 2 ) 2r^3t^4+O(r^2) 2 r 3 t 4 + O ( r 2 ) . □
Numerical Methods and Additional Tables
Densities and divergences. Graded densities are computed by Gauss–Jacobi quadrature in the membership, with 100 to 160 nodes. Divergences are computed by Simpson’s rule on grids of 2,401 to 4,001 points, with quadrature error in KL below 10 − 15 10^{-15} 1 0 − 15 . For the optimized look-alikes we minimized K L ( f θ ∥ G ) \mathrm{KL}(f_\theta\Vert G) KL ( f θ ∥ G ) over two-component Gaussian mixtures G G G , parameterized by the logit of the weight, the two means and the two log standard deviations, with the Nelder–Mead method in SciPy. For the ( d , c ) (d,c) ( d , c ) grid of Figure 2 there were 11 starts (symmetric hard mixtures at ± { 0.6 , 1.0 , 1.4 } s G \pm\{0.6,1.0,1.4\}s_G ± { 0.6 , 1.0 , 1.4 } s G and eight equal-mean scale mixtures), with tolerances 10 − 8 10^{-8} 1 0 − 8 in the parameters and 10 − 15 10^{-15} 1 0 − 15 in the objective and at most 3,000 evaluations; the reported value uses whichever of the best search and the closed form has the smaller divergence. For the asymmetric truths there were 24 starts and at most 2,500 evaluations. We did not check convergence flags. This does not affect validity: each reported value is the divergence of an explicit mixture, so a suboptimal search only makes the certificate conservative. Where KL falls below about 10 − 6 10^{-6} 1 0 − 6 the search is less reliable, which explains the non-monotone entries in the γ 2 ≥ 0 \gamma_2 \ge0 γ 2 ≥ 0 region.
Cramér–Rao bounds. Table 2 uses the Fisher information of the full two-type model at the symmetric truth. The density is computed with 120 Gauss–Jacobi nodes in the membership on a grid of 2,401 points over [ − 12 , 12 ] [-12,12] [ − 12 , 12 ] ; its derivatives are central differences with step 10 − 4 10^{-4} 1 0 − 4 in the coordinates ( μ 1 , μ 2 , log σ 1 , log σ 2 , logit π , log c ) (\mu_1,\mu_2,\log\sigma_1,\log\sigma_2,\operatorname{logit}\pi,\log c) ( μ 1 , μ 2 , log σ 1 , log σ 2 , logit π , log c ) , and the information is integrated by Simpson’s rule on the same grid. The non-monotonicity in c c c was checked against changes in step size, number of quadrature nodes and grid width. At c = 5 c=5 c = 5 and c = 7 c=7 c = 7 the σ 1 \sigma_1 σ 1 column is 40,001 and 177,014, and the π \pi π column is 149,883 and 2.26 × 10 6 2.26 \times10^6 2.26 × 1 0 6 . The bounds were also computed on a finer grid ( c = 0.1 , 0.25 , (c=0.1, 0.25, ( c = 0.1 , 0.25 , then 0.5 to 16 in steps of 0.5, and 18, 20, 24). On the finer grid the fixed-c c c bounds for σ 1 \sigma_1 σ 1 and π \pi π both peak near c = 6.5 c=6.5 c = 6.5 , and the learned-c c c bound for the half-separation peaks near c = 5.5 c=5.5 c = 5.5 , where it is about 54,000 times the fixed-c c c bound, before falling. At c = 0.1 c=0.1 c = 0.1 and 0.25 the Beta density is singular at both endpoints; Gauss–Jacobi quadrature integrates the singularity exactly, and the bounds change by at most 6 × 10 − 4 6 \times10^{-4} 6 × 1 0 − 4 in relative terms between 160 and 400 nodes, across step sizes and with a wider grid. With c c c learned, the bound for the marginal variance equals ( 2 + γ 2 ) v 2 (2+\gamma^2)v^2 ( 2 + γ 2 ) v 2 , the variance of the sample variance, to four digits at every c c c on the grid; with c c c supplied it equals it for c ≥ 4 c \ge4 c ≥ 4 and is up to about 2% smaller below. As a check between local and global calculations, the learned-c c c half-separation sizes are about half of the sample sizes at which the best configuration with a 10% change in separation reaches a divergence of one nat.
Table 4: Tightness of Theorem 7 at d = 3 d=3 d = 3 for the closed-form look-alike. The certified size is n h a r d = 1 / ( 2 K L ) n_{\mathrm{hard}}=1/(2\mathrm{KL}) n hard = 1/ ( 2 KL ) . The last two columns bracket the smallest n n n at which some test has summed error at most 0.05 (Proposition 13): no test achieves it below the first, and the likelihood-ratio test achieves it at the second.
Table 4.
Table 5: Certified sizes 1 / ( 2 K L ) 1/(2\mathrm{KL}) 1/ ( 2 KL ) for asymmetric truths with μ 1 = 3 \mu_1=3 μ 1 = 3 , μ 2 = − 3 \mu_2=-3 μ 2 = − 3 , against the numerically optimized two-component mixture (hard) and the moment-matched Gaussian.
Table 5.
The Fixed-Concentration Sweep
Protocol. For each c ∈ { 0.1 , 0.25 , 0.5 , 1 , 2 , 3 , 4 , 6 , 8 , 10 , 15 , 20 } c \in\{0.1,0.25,0.5,1,2,3,4,6,8,10,15,20\} c ∈ { 0.1 , 0.25 , 0.5 , 1 , 2 , 3 , 4 , 6 , 8 , 10 , 15 , 20 } and n ∈ { 10 3 , 10 4 } n \in\{10^3,10^4\} n ∈ { 1 0 3 , 1 0 4 } we drew ten datasets from the symmetric model with s = 3 s=3 s = 3 , t = 1 t=1 t = 1 (seeds 67–76), standardized them, and fitted the full two-type model, with μ 1 , μ 2 , σ 1 , σ 2 \mu_1,\mu_2,\sigma_1,\sigma_2 μ 1 , μ 2 , σ 1 , σ 2 and π \pi π free and c c c fixed at its true value, as described in Appendix § 16 : EM from six starts followed by a quasi-Newton polish of the marginal likelihood from the three best endpoints, and for c ≥ 6 c \ge6 c ≥ 6 EM from forty starts with the ten best distinct endpoints polished. Refits in fresh processes reproduce every fit exactly. Estimates are transformed back to the original scale. We report s ^ = ∣ μ ^ 1 − μ ^ 2 ∣ / 2 \hat{s} = |\hat{\mu}_1-\hat{\mu}_2|/2 s ^ = ∣ μ ^ 1 − μ ^ 2 ∣/2 , the number of fits in which one type’s standard deviation falls below 0.01 in standardized units (a point-mass type), and the variance v ^ \hat{v} v ^ implied by the fitted parameters through
v = ∑ k σ k 2 ( m ˉ k 2 + m ˉ k ( 1 − m ˉ k ) c + 1 ) + m ˉ 1 m ˉ 2 ( μ 1 − μ 2 ) 2 c + 1 , m ˉ = ( π , 1 − π ) , v = \sum_k \sigma_k^2 \left(\bar{m}_k^2 + \frac{\bar{m}_k(1-\bar{m}_k)}{c+1}\right) + \frac{\bar{m}_1\bar{m}_2(\mu_1-\mu_2)^2}{c+1}, \qquad\bar{m} = (\pi,1-\pi), v = k ∑ σ k 2 ( m ˉ k 2 + c + 1 m ˉ k ( 1 − m ˉ k ) ) + c + 1 m ˉ 1 m ˉ 2 ( μ 1 − μ 2 ) 2 , m ˉ = ( π , 1 − π ) ,
which does not assume that the fit stayed symmetric.
Table 6. The Gaussian end along the fixed-( s , t ) (s,t) ( s , t ) family s = 3 s=3 s = 3 , t = 1 t=1 t = 1 . The column 1 / χ 2 1/\chi^2 1/ χ 2 is a chi-squared counterpart of n G a u s s = 1 / ( 2 K L ( f θ ∥ N ( 0 , v ) ) ) n_{\mathrm{Gauss}}=1/(2\mathrm{KL}(f_\theta\Vert\mathcal{N}(0,v))) n Gauss = 1/ ( 2 KL ( f θ ∥ N ( 0 , v ))) , which is shown in Table 1. c = 8.76 c=8.76 c = 8.76 is c 0 ( 3 ) c_0(3) c 0 ( 3 ) , where γ 2 = 0 \gamma_2=0 γ 2 = 0 .
Table 7. Fixed-c c c sweep, ten datasets per cell (six starts for c ≤ 4 c \le4 c ≤ 4 , forty for c ≥ 6 c \ge6 c ≥ 6 ). sd: across-dataset standard deviation of s ^ \hat{s} s ^ ; CRB: Cramér–Rao standard deviation of the half-separation; #: fits with s ^ < 1.5 \hat{s}<1.5 s ^ < 1.5 ; pm: point-mass fits; v ^ \hat{v} v ^ : relative across-dataset standard deviation of v ^ \hat{v} v ^ divided by ( 2 + γ 2 ) / n \sqrt{(2+\gamma_2)/n} ( 2 + γ 2 ) / n . † \dagger † : start-dependent cell, in which forty starts changed several fits relative to six; its values are the best available, not certified maxima. For ten datasets the 95% sampling band of sd/CRB is [0.55, 1.45].
Where the bound describes the fits. For 0.25 ≤ c ≤ 4 0.25 \le c \le4 0.25 ≤ c ≤ 4 the spread of s ^ \hat{s} s ^ matches the bound at both sample sizes, with ratios between 0.71 and 1.46. At c = 0.1 c=0.1 c = 0.1 , n = 10 4 n=10^4 n = 1 0 4 it is below the bound (ratio 0.47). This is a property of the ten datasets, not of the estimator: on them the naive estimator (half the difference between the means of the positive and the negative observations) also has standard deviation 0.0063, and on 80 further datasets the EM alone gives 0.0106 against a bound of 0.0115, with blocks of ten datasets ranging from 0.0064 to 0.0137.
Figure 4. Fixed-c c c sweep against Cramér–Rao bounds (ten datasets per cell, truth s = 3 s = 3 s = 3 , t = 1 t = 1 t = 1 ; top row n = 10 3 n = 10^{3} n = 1 0 3 , bottom row n = 10 4 n = 10^{4} n = 1 0 4 ). Points are individual fits (six starts for c ≤ 4 c \le4 c ≤ 4 , forty for c ≥ 6 c \ge6 c ≥ 6 ); red marks s ^ < 1.5 \hat{s} < 1.5 s ^ < 1.5 , open squares mark point-mass fits, and open triangles at the top edge mark values beyond the plotted range. Black lines are medians. Left: s ^ \hat{s} s ^ with the band 3 ± 1.96 3 \pm1.96 3 ± 1.96 sd from the bound. Middle: log ( σ ^ max / σ ^ min ) \log(\hat{\sigma}_{\max}/\hat{\sigma}_{\min}) log ( σ ^ m a x / σ ^ m i n ) , the absolute value of log σ ^ 1 − log σ ^ 2 \log\hat{\sigma}_{1} - \log\hat{\sigma}_{2} log σ ^ 1 − log σ ^ 2 , against 0.674 0.674 0.674 sd (dashed, the median of a folded normal) and 1.96 1.96 1.96 sd (dotted) from the bound. Right: v ^ / v − 1 \hat{v}/v - 1 v ^ / v − 1 with the band ± 1.96 ( 2 + γ 2 ) / n \pm1.96\sqrt{(2+\gamma^{2})/n} ± 1.96 ( 2 + γ 2 ) / n .
Where it does not. At c = 6 c = 6 c = 6 and 8 8 8 and n = 10 4 n = 10^{4} n = 1 0 4 , forty starts reproduce the six-start fits exactly, and the spread is 2.59 and 2.55 times the bound (1.45 and 1.51 for the robust, interquartile estimate): a few fits sit on other local maxima. For c ≥ 10 c \ge10 c ≥ 10 , and at c = 6 c = 6 c = 6 with n = 10 3 n = 10^{3} n = 1 0 3 , the fits depend on the number of starts. Relative to six starts, forty changed five to seven of ten fits at n = 10 3 n = 10^{3} n = 1 0 3 and two of ten at n = 10 4 n = 10^{4} n = 1 0 4 , by up to 2.0 nats. The competing maxima are qualitatively different configurations: collapsed (s ^ ≤ 0.5 \hat{s} \le0.5 s ^ ≤ 0.5 ), separated (s ^ ≈ 3 \hat{s} \approx3 s ^ ≈ 3 ) and point-mass (s ^ \hat{s} s ^ of 7 to 10 with one standard deviation near zero). The fraction of datasets with a rival maximum within one nat of the kept one, whose s ^ \hat{s} s ^ differs from it by more than 0.5, is
Table 4 (PDF p. 21).
At c = 20 c = 20 c = 20 and n = 10 3 n = 10^{3} n = 1 0 3 , the number of collapsed fits grows with the number of starts: none with six, three with twenty and five with forty. The collapsed and separated maxima differ by less than a nat.
In 14 of the 240 six-start fits one type’s standard deviation drifts toward zero. In these fits the Beta exponent at that type’s pure end exceeds one, so the likelihood stays bounded and becomes flat in that direction (Appendix § 16 .§ 1 ). Such point-mass configurations belong to the weakly identified set, and some are the highest maximum found, by 1.06 and 1.56 nats in two datasets.
Variances. For c ≤ 4 c \le4 c ≤ 4 the median of ∣ log σ ^ 1 − log σ ^ 2 ∣ |\log\hat{\sigma}_1-\log\hat{\sigma}_2| ∣ log σ ^ 1 − log σ ^ 2 ∣ is between 0.39 and 1.85 times the folded-normal prediction 0.674 sd. At c = 6 c=6 c = 6 and 8, excluding point-mass fits, it is 0.23 to 0.55 times the prediction. That shortfall is what one expects of a local bound where the likelihood is far from quadratic. Along the weakest Fisher direction at c = 6 c=6 c = 6 , at the displacement where the quadratic approximation reaches one nat, the exact divergence is 2.7 × 10 3 2.7 \times10^3 2.7 × 1 0 3 times larger at n = 10 3 n=10^3 n = 1 0 3 and 9.2 × 10 3 9.2 \times10^3 9.2 × 1 0 3 times larger at n = 10 4 n=10^4 n = 1 0 4 . The exact one-nat displacement corresponds to ∣ Δ log ( σ 1 / σ 2 ) ∣ |\Delta\log(\sigma_1/\sigma_2)| ∣Δ log ( σ 1 / σ 2 ) ∣ of about 0.42 and 0.27, against 4.2 and 1.3 from the quadratic approximation. This is a line search along one direction, not a full profile. The implied marginal variance is estimated as well as a second moment can be in every cell: its relative spread is between 0.50 and 1.40 times ( 2 + γ 2 ) / n \sqrt{(2+\gamma_2)/n} ( 2 + γ 2 ) / n .
Details of the Inference Experiments
BayesFMMM. We used the multivariate pipeline of the R package (BMVMMM_Nu_Z_multiple_try, then BMVMMM_Theta_est, then BMVMMM_warm_start) with one-dimensional observations (P = 1 P=1 P = 1 ), two features (K = 2 K=2 K = 2 ) and M = 2 = K P M=2=KP M = 2 = K P eigencomponents, so the covariance representation is exact. The default priors are Z i ∼ Dir ( α 3 π ) Z_i \sim\operatorname{Dir}(\alpha_3\pi) Z i ∼ Dir ( α 3 π ) for the memberships, α 3 ∼ Exp ( rate 10 ) \alpha_3 \sim\operatorname{Exp}(\operatorname{rate}\ 10) α 3 ∼ Exp ( rate 10 ) , π ∼ Dir ( 1 ) \pi\sim\operatorname{Dir}(1) π ∼ Dir ( 1 ) , an inverse-gamma( 10 , 1 ) (10,1) ( 10 , 1 ) prior on the variance of the feature means (so the means are marginally t 20 t_{20} t 20 with scale 0.316), and σ 2 ∼ \sigma^2 \sim σ 2 ∼ inverse-gamma( 1 , 1 ) (1,1) ( 1 , 1 ) for the noise. With one-dimensional observations, σ 2 \sigma^2 σ 2 is exactly redundant with the feature covariance: since the memberships sum to one, σ 2 = σ 2 ( Z i 1 + Z i 2 ) 2 \sigma^2=\sigma^2(Z_{i1}+Z_{i2})^2 σ 2 = σ 2 ( Z i 1 + Z i 2 ) 2 , so only C + σ 2 11 T C+\sigma^2\mathbf{1}\mathbf{1}^{\mathsf{T}} C + σ 2 1 1 T is identified, and G2M2 corresponds to this matrix being diagonal. The truth-initialized runs overwrite the warm-start objects with the true values. The package’s membership rescaling step, which enforces the separability condition of Marco et al. [18 ] , is not exported, and we did not use it.
Short runs (ten datasets per c c c , seeds 1–10) used the package defaults: one initialization try, 2,000 iterations with the second half kept, and no tempered transitions (the defaults set the number of tempering rungs to one). A run takes about 9 seconds. Weakening the prior on the feature means (inverse-gamma( 2 , 20 ) (2,20) ( 2 , 20 ) ) did not change the default-start result, so the collapse is not driven by that prior.
Long runs (seeds 1–5 of the same datasets) used 50,000 iterations with the second half kept, run in segments of 5,000 iterations with exact restarts from the saved state. The default start used the multiple-start algorithm with ten tries of 500 iterations, followed by BMVMMM_Theta_est with five tries. Tempering is available only in BMVMMM_warm_start, through a geometric ladder of N τ N_\tau N τ rungs from inverse temperature 1 to β N τ \beta N_\tau β N τ , applied once every given number of iterations. In test runs at c = 6 c=6 c = 6 , the log acceptance probabilities of the first tempered transitions were between − 597 -597 − 597 and − 47 -47 − 47 for 50 rungs down to β = 0.3 \beta=0.3 β = 0.3 , between − 368 -368 − 368 and − 89 -89 − 89 for 200 rungs down to 0.3, and between − 262 -262 − 262 and − 1.8 -1.8 − 1.8 for 50 rungs down to 0.8: with 2,000 latent memberships, any ladder hot enough to move between configurations is essentially never accepted, as Marco et al. [18 ] anticipate. We used 50 rungs down to β = 0.8 \beta=0.8 β = 0.8 , one tempered transition every 500 iterations (100 per chain); acceptance rates were 0% to 13%. Split-R ^ \hat{R} R ^ between the two starts was 12 to 26 for s s s and 1.7 to 5.2 for log α 3 \log\alpha_3 log α 3 . The two starts’ 95% intervals for α 3 \alpha_3 α 3 were disjoint on five of five datasets at c = 3 c=3 c = 3 and two of five at c = 6 c=6 c = 6 , because the truth-started chains drift toward smaller α 3 \alpha_3 α 3 . A chain took 150 to 200 seconds.
Reference posterior. The reference model is the same marginal model with memberships integrated out by quadrature on a logit grid with atoms at the endpoints. For speed, the sampler evaluates the likelihood on a 400-bin histogram of the data; every log-likelihood we report is recomputed exactly, without binning, with 128-node Gauss–Jacobi quadrature. Its seven global parameters are the two means, the effective covariance through a Cholesky factor, α 3 \alpha_3 α 3 and π \pi π . Both prior settings use C e f f = L L T + 10 − 6 I C_{\mathrm{eff}} = LL^{\mathsf{T}} + 10^{-6}I C eff = L L T + 1 0 − 6 I with L L L lower triangular, half-normal priors with scale 3 on the diagonal entries of L L L (sampled on the log scale), N ( 0 , 3 2 ) \mathcal{N}(0,3^2) N ( 0 , 3 2 ) on the off-diagonal entry, π ∼ Uniform ( 0.02 , 0.98 ) \pi\sim\operatorname{Uniform}(0.02,0.98) π ∼ Uniform ( 0.02 , 0.98 ) (sampled on the logit scale) and α 3 \alpha_3 α 3 restricted to [ 10 − 3 , 10 4 ] [10^{-3},10^4] [ 1 0 − 3 , 1 0 4 ] . The weak setting puts N ( 0 , 10 2 ) \mathcal{N}(0,10^2) N ( 0 , 1 0 2 ) on the means and N ( log 4 , 1.5 2 ) \mathcal{N}(\log4,1.5^2) N ( log 4 , 1. 5 2 ) on log α 3 \log\alpha_3 log α 3 ; the matched setting replaces these with BayesFMMM’s t 20 ( 0 , 0.316 ) t_{20}(0,0.316) t 20 ( 0 , 0.316 ) and Exp ( 10 ) \operatorname{Exp}(10) Exp ( 10 ) , and does not reproduce the package’s shrinkage prior on the covariance factors or its prior on σ 2 \sigma^2 σ 2 . Each of 100 datasets (50 per value of c c c : the 20 of the sampler runs and 80 new ones) was run under both settings from two starts, the truth and the configuration in which BayesFMMM’s default start settles, with emcee using 28 walkers for 2,000 steps and the second half kept (400 runs; all runs seeded).
Under the matched setting the two starts agree on α 3 \alpha_3 α 3 (split-R ^ ≤ 1.15 \hat{R} \le1.15 R ^ ≤ 1.15 ), the posterior probability of α 3 < 0.1 \alpha_3 < 0.1 α 3 < 0.1 is at least 0.49 in every run, and the 95% upper limits for α 3 \alpha_3 α 3 are 0.13 to 0.50. In 48 of the 200 runs more than 1% (at most 3.5%) of the draws of α 3 \alpha_3 α 3 fall below 1.5 × 10 − 3 1.5 \times10^{-3} 1.5 × 1 0 − 3 , near the support bound; this affects the lower 2.5% quantile, not the medians or upper limits. The hard posterior has two modes at c = 3 c = 3 c = 3 : nearly identical types (s ≈ 0.1 s \approx0.1 s ≈ 0.1 to 0.5; 28 truth-start and 48 collapsed-start runs) and closer but distinct types (s ≈ 1.0 s \approx1.0 s ≈ 1.0 to 1.2; the other 24 runs). At c = 6 c = 6 c = 6 the types are nearly identical (s ≈ 0.09 s \approx0.09 s ≈ 0.09 to 0.25) from both starts. The fit of the hard configuration is measured by the exact log-likelihood of the best posterior draw with α 3 < 0.1 \alpha_3 < 0.1 α 3 < 0.1 relative to the truth: medians +2.03 and −0.71 nats at c = 3 c = 3 c = 3 (truth and collapsed starts) and +1.66 and +1.64 at c = 6 c = 6 c = 6 . Part of a positive value is the usual advantage of seven free parameters over the true values. We do not summarize by the coordinate-wise posterior median, which lies off the curved ridge of the posterior and fits 4 to 26 nats worse than the truth. An unbinned likelihood gives the same result on five datasets per value of c c c : the posterior is hard on all ten, and over all saved draws the binned and exact log-likelihoods relative to the truth differ by at most 0.15 nats.
Under the weak setting no run goes hard; every interval for α 3 \alpha_3 α 3 covers the truth, as do 192 of the 200 intervals for s s s . At c = 3 c = 3 c = 3 the 2,000-step runs from the two starts mix incompletely (split-R ^ \hat{R} R ^ up to 1.44); on the ten original datasets, 10,000 steps with 56 walkers give split-R ^ ≤ 1.07 \hat{R} \le1.07 R ^ ≤ 1.07 for both log α 3 \log\alpha_3 log α 3 and s s s , and every interval covers the truth.
Larger samples. We repeated the comparison at c = 3 c = 3 c = 3 for n ∈ { 10 4 , 3 × 10 4 , 6 × 10 4 , 1.2 × 10 5 } n \in\{10^4, 3 \times10^4, 6 \times10^4, 1.2 \times10^5\} n ∈ { 1 0 4 , 3 × 1 0 4 , 6 × 1 0 4 , 1.2 × 1 0 5 } and at c = 6 c = 6 c = 6 for n ∈ { 7.7 × 10 4 , 3 × 10 5 , 10 6 } n \in\{7.7 \times10^4, 3 \times10^5, 10^6\} n ∈ { 7.7 × 1 0 4 , 3 × 1 0 5 , 1 0 6 } , five new datasets per n n n , with 2,000 histogram bins for n ≥ 3 × 10 4 n \ge3 \times10^4 n ≥ 3 × 1 0 4 (400 and 2,000 bins agree within Monte Carlo error at n = 6 × 10 4 n = 6 \times10^4 n = 6 × 1 0 4 ). At these sizes the chains started at the collapsed configuration stay near it under both priors (split-R ^ \hat{R} R ^ up to 2.3 for log α 3 \log\alpha_3 log α 3 ), so we report the truth-started chains. Under the package’s priors the median of α 3 \alpha_3 α 3 rises only slowly with n n n : at c = 3 c = 3 c = 3 it is about 0.05 up to n = 3 × 10 4 n = 3 \times10^4 n = 3 × 1 0 4 , about 0.1 at 6 × 10 4 6 \times10^4 6 × 1 0 4 and 0.25 to 0.36 at 1.2 × 10 5 1.2 \times10^5 1.2 × 1 0 5 ; at c = 6 c = 6 c = 6 and n = 10 6 n = 10^6 n = 1 0 6 it is 1.0 to 1.8. The half-separation stays between 1.2 and 1.5, and no interval covers the true concentration or separation. The data do turn against the hard configuration: at c = 3 c = 3 c = 3 the best hard draw goes from +1.3 nats relative to the truth at n = 2 , 000 n = 2,000 n = 2 , 000 to −5.9 nats at n = 1.2 × 10 5 n = 1.2 \times10^5 n = 1.2 × 1 0 5 , and at c = 6 c = 6 c = 6 , n = 10 6 n = 10^6 n = 1 0 6 the chains no longer visit the hard region. Under the weak prior no run goes hard, and the truth-started intervals cover the truth in 77 of 80 cases. A rough accounting explains the persistence. Relative to a typical hard configuration (means ± 0.15 \pm0.15 ± 0.15 , α 3 = 0.05 \alpha_3 = 0.05 α 3 = 0.05 ), the package’s priors place the truth about 61 nats lower at c = 3 c = 3 c = 3 and 90 nats lower at c = 6 c = 6 c = 6 , about 36 of them from the prior on the means, while the data’s evidence against the closest hard look-alike grows like n K L n\mathrm{KL} n KL , with K L = 1.5 × 10 − 4 \mathrm{KL} = 1.5 \times10^{-4} KL = 1.5 × 1 0 − 4 at c = 3 c = 3 c = 3 and 5.9 × 10 − 6 5.9 \times10^{-6} 5.9 × 1 0 − 6 at c = 6 c = 6 c = 6 . The two balance only near n ≈ 4 × 10 5 n \approx4 \times10^{5} n ≈ 4 × 1 0 5 and 1.5 × 10 7 1.5 \times10^{7} 1.5 × 1 0 7 . The certified sizes concern testing two fixed laws; they do not say when data overcome a prior that sits tens of nats away from the truth. In applications with 33 or 50 dimensions per unit, Marco et al. [19 ] found their fits insensitive to the prior on α 3 \alpha_3 α 3 ; we expect the ambiguity, and with it the prior’s leverage, to be weaker there (§ 9 ).
Figure 3 shows the four runs for dataset 1 at each value of c c c .
Fitting the Fixed-Concentration Model
Each fit in Appendix D maximizes the marginal likelihood of the full model, with c c c fixed, in two stages. An EM algorithm generates candidate starting points, and a quasi-Newton method then maximizes the marginal likelihood from the best of them. Only the second stage determines the reported fits, so the EM needs to explore well, not to be exact. All fits are reproducible: refits in fresh processes return identical values.
Procedure
Data and parameters. The data are standardized by their sample mean and standard deviation, and the estimates are mapped back at the end. The free parameters are μ 1 \mu_1 μ 1 , μ 2 \mu_2 μ 2 , σ 1 \sigma_1 σ 1 , σ 2 \sigma_2 σ 2 and the prior mean membership π \pi π .
Stage 1: starting points. The EM represents memberships on M = 256 M = 256 M = 256 nodes placed at the quantiles ( m − 1 2 ) / M (m - \frac{1}{2})/M ( m − 2 1 ) / M of Beta ( c π , c ( 1 − π ) ) \operatorname{Beta}(c\pi, c(1-\pi)) Beta ( c π , c ( 1 − π )) , each with prior weight 1 / M 1/M 1/ M , so the posterior weight of node m m m for observation i i i is proportional to ϕ ( x i ; p m μ 1 + ( 1 − p m ) μ 2 , p m 2 σ 1 2 + ( 1 − p m ) 2 σ 2 2 ) \phi\left(x_i; p_m\mu_1 + (1-p_m)\mu_2, p_m^2\sigma_1^2 + (1-p_m)^2\sigma_2^2\right) ϕ ( x i ; p m μ 1 + ( 1 − p m ) μ 2 , p m 2 σ 1 2 + ( 1 − p m ) 2 σ 2 2 ) . The M-step alternates between the exact mean update of Proposition 15 and iteratively reweighted least squares for the variances with weights g m / V m 2 g_m/V_m^2 g m / V m 2 (Propositions 16 and 18 ), and sets π \pi π to the average posterior membership; a variance step that would make a variance negative is rejected and the previous value kept. The update of π \pi π is a moment rule rather than the exact EM step, and the iteration does not check that the likelihood increases; both simplifications are harmless for an initializer. The EM stops when the relative change in its log-likelihood falls below 10 − 5 10^{-5} 1 0 − 5 , or after 50 iterations. It is run from the k k k -means solution (ten initializations, seeded by the dataset’s seed) and from five perturbations of it: the means shifted by N ( 0 , 0.5 2 ) \mathcal{N}(0, 0.5^2) N ( 0 , 0. 5 2 ) in standardized units, the log standard deviations by N ( 0 , 0.4 2 ) \mathcal{N}(0, 0.4^2) N ( 0 , 0. 4 2 ) , and π \pi π drawn from a Dirichlet distribution with mean equal to the cluster proportions and concentration 10. For c ≥ 6 c \ge6 c ≥ 6 we also ran 39 perturbations.
Stage 2: maximization. From the three highest-likelihood EM endpoints, or for c ≥ 6 c \ge6 c ≥ 6 the ten best endpoints of the forty-start run whose separations differ by more than 10 − 2 10^{-2} 1 0 − 2 , we maximize the marginal log-likelihood directly with L-BFGS-B over (\mu_1,\mu_2,\log\sigma_1,\log\sigma_2,\logit\pi) , with c c c fixed. The likelihood is computed with 128-node Gauss–Jacobi quadrature in the membership, with nodes and weights recomputed for the current Beta ( c π , c ( 1 − π ) ) \operatorname{Beta}(c\pi,c(1-\pi)) Beta ( c π , c ( 1 − π )) at every evaluation; for a fixed number of nodes the result is smooth in the parameters, and gradients are taken by finite differences. It agrees with 256 nodes to 1.2 × 10 − 11 1.2 \times10^{-11} 1.2 × 1 0 − 11 in relative terms at every fit. (The EM’s quantile-midpoint rule is accurate only to about 10 − 5 10^{-5} 1 0 − 5 at large c c c , enough for starting points but not for locating an optimum on a flat ridge.) The bounds, in standardized units, are ∣ μ k ∣ < 20 |\mu_k| < 20 ∣ μ k ∣ < 20 , σ k ∈ [ 10 − 3 , 10 ] \sigma_k \in[10^{-3},10] σ k ∈ [ 1 0 − 3 , 10 ] and |\logit\pi| \le12 . A run that stops without success restarts from its own endpoint, at most twice, and the best maximized point is kept. This stage matters: at n = 10 4 n = 10^{4} n = 1 0 4 and 2 ≤ c ≤ 8 2 \le c \le8 2 ≤ c ≤ 8 it gained a median of 1.0 to 2.4 nats over the best EM endpoint, and up to 4.45. Nine runs, all at c ≤ 0.5 c \le0.5 c ≤ 0.5 , ended on a line-search stop; in each, the three runs agreed to 10 − 3 10^{-3} 1 0 − 3 in separation and within 0.01 nats. The largest gradient at a kept point corresponds to a displacement of order 10 − 5 10^{-5} 1 0 − 5 in s ^ \hat{s} s ^ .
The likelihood has no global maximum. As for Gaussian mixtures with unequal variances, the likelihood is unbounded. Suppose the Beta exponent at a type’s pure end is at most one; for type 1 this is c ( 1 − π ) ≤ 1 c(1-\pi) \le1 c ( 1 − π ) ≤ 1 , which a free π \pi π allows at any c c c . Put that type’s mean at an observation and let its standard deviation tend to zero. The density at that observation then diverges, because near the pure end it behaves like ∫ 0 y b − 2 d y \int_0 y^{b-2}\,\mathrm{d}y ∫ 0 y b − 2 d y with b = c ( 1 − π ) ≤ 1 b = c(1-\pi) \le1 b = c ( 1 − π ) ≤ 1 , while the densities at the other observations stay bounded below. The reported fits are therefore the highest interior local maxima reached from the starts, as is standard for mixtures. The point-mass fits of Appendix D are different: there the exponent exceeds one, the likelihood stays bounded, and it is flat as the standard deviation shrinks.
The M-step of the initializer
For completeness we record the properties of the two M-step blocks. Fix the nodes p m p_m p m and the posterior weights γ i m \gamma_{im} γ im , write σ 2 = ( σ 1 2 , σ 2 2 ) T \sigma^2 = (\sigma_1^2,\sigma_2^2)^{\mathsf{T}} σ 2 = ( σ 1 2 , σ 2 2 ) T , u ( m ) = ( p m , 1 − p m ) T u^{(m)} = (p_m,1-p_m)^{\mathsf{T}} u ( m ) = ( p m , 1 − p m ) T , q ( m ) = ( p m 2 , ( 1 − p m ) 2 ) T q^{(m)} = (p_m^2,(1-p_m)^2)^{\mathsf{T}} q ( m ) = ( p m 2 , ( 1 − p m ) 2 ) T , V m = q ( m ) T σ 2 V_m = q^{(m)\mathsf{T}}\sigma^2 V m = q ( m ) T σ 2 and e i ( m ) ( μ ) = ( x i − u ( m ) T μ ) 2 e_i^{(m)}(\mu) = (x_i-u^{(m)\mathsf{T}}\mu)^2 e i ( m ) ( μ ) = ( x i − u ( m ) T μ ) 2 . Up to constants, the expected complete-data log-likelihood in ( μ , σ 2 ) (\mu,\sigma^2) ( μ , σ 2 ) is
Q ( μ , σ 2 ) = − 1 2 ∑ i , m γ i m [ log V m + e i ( m ) ( μ ) V m ] . Q(\mu,\sigma^2) = -\frac{1}{2}\sum_{i,m}\gamma_{im}\left[\log V_m + \frac{e_i^{(m)}(\mu)}{V_m}\right]. Q ( μ , σ 2 ) = − 2 1 i , m ∑ γ im [ log V m + V m e i ( m ) ( μ ) ] .
It depends on the data only through the node sums g m = ∑ i γ i m g_m = \sum_i\gamma_{im} g m = ∑ i γ im , h m = ∑ i γ i m x i h_m = \sum_i\gamma_{im}x_i h m = ∑ i γ im x i and ρ m = ∑ i γ i m x i 2 \rho_m = \sum_i\gamma_{im}x_i^2 ρ m = ∑ i γ im x i 2 , since d m ( μ ) : = ∑ i γ i m e i ( m ) ( μ ) = ρ m − 2 ( u ( m ) T μ ) h m + g m ( u ( m ) T μ ) 2 d_m(\mu) := \sum_i\gamma_{im}e_i^{(m)}(\mu) = \rho_m - 2(u^{(m)\mathsf{T}}\mu)h_m + g_m(u^{(m)\mathsf{T}}\mu)^2 d m ( μ ) := ∑ i γ im e i ( m ) ( μ ) = ρ m − 2 ( u ( m ) T μ ) h m + g m ( u ( m ) T μ ) 2 . Write e ˉ m = d m / g m \bar{e}_m = d_m/g_m e ˉ m = d m / g m for the posterior-weighted mean squared residual at node m m m .
Proposition 15 (Mean block). Fix σ 2 \sigma^2 σ 2 with positive entries. Then μ ↦ Q \mu\mapsto Q μ ↦ Q is a concave quadratic with Hessian − A -A − A , where A = ∑ m ( g m / V m ) u ( m ) u ( m ) T A = \sum_m(g_m/V_m)u^{(m)}u^{(m)\mathsf{T}} A = ∑ m ( g m / V m ) u ( m ) u ( m ) T , and its stationary points solve A μ = b A\mu= b A μ = b with b = ∑ m ( h m / V m ) u ( m ) b = \sum_m(h_m/V_m)u^{(m)} b = ∑ m ( h m / V m ) u ( m ) . Since γ i m > 0 \gamma_{im} > 0 γ im > 0 for all i , m i,m i , m , A A A is positive definite as soon as two nodes are distinct, and μ = A − 1 b \mu= A^{-1}b μ = A − 1 b is the unique maximizer of Q Q Q in μ \mu μ .
Proof. ∂ μ Q = ∑ i , m γ i m u ( m ) ( x i − u ( m ) T μ ) / V m = b − A μ \partial_\mu Q = \sum_{i,m}\gamma_{im}u^{(m)}(x_i-u^{(m)\mathsf{T}}\mu)/V_m = b-A\mu ∂ μ Q = ∑ i , m γ im u ( m ) ( x i − u ( m ) T μ ) / V m = b − A μ , and z T A z = ∑ m ( g m / V m ) ( z T u ( m ) ) 2 z^{\mathsf{T}}Az = \sum_m(g_m/V_m)(z^{\mathsf{T}}u^{(m)})^2 z T A z = ∑ m ( g m / V m ) ( z T u ( m ) ) 2 vanishes only if z z z is orthogonal to every u ( m ) u^{(m)} u ( m ) .
Proposition 16 (Variance block: the weights are forced). Fix μ \mu μ and let σ 2 \sigma^2 σ 2 have positive entries. Then
∂ Q ∂ σ k 2 = 1 2 ∑ m q k ( m ) g m ( e ˉ m − V m ) V m 2 . \frac{\partial Q}{\partial\sigma_k^2} = \frac{1}{2}\sum_m q_k^{(m)}\frac{g_m(\bar{e}_m-V_m)}{V_m^2}. ∂ σ k 2 ∂ Q = 2 1 m ∑ q k ( m ) V m 2 g m ( e ˉ m − V m ) . so σ 2 \sigma^2 σ 2 is a stationary point of Q Q Q if and only if it is a stationary point, at frozen weights, of the weighted least-squares criterion ∑ m ω m ( e ˉ m − q ( m ) T σ 2 ) 2 \sum_m\omega_m(\bar{e}_m-q^{(m)\mathsf{T}}\sigma^2)^2 ∑ m ω m ( e ˉ m − q ( m ) T σ 2 ) 2 with ω m = g m / V m 2 \omega_m = g_m/V_m^2 ω m = g m / V m 2 . No other weight function has this property: if ω m = g m ψ ( V m ) \omega_m = g_m\psi(V_m) ω m = g m ψ ( V m ) for a positive ψ \psi ψ , the normal equations agree with the stationarity condition for all data and all node configurations only if ψ ( V ) ∝ V − 2 \psi(V) \propto V^{-2} ψ ( V ) ∝ V − 2 .
Proof. Q Q Q sees σ 2 \sigma^2 σ 2 only through V m V_m V m , and ∂ V [ − 1 2 ( g log V + d / V ) ] = 1 2 ( d − g V ) / V 2 \partial_V[-\frac{1}{2}(g\log V+d/V)] = \frac{1}{2}(d-gV)/V^2 ∂ V [ − 2 1 ( g log V + d / V )] = 2 1 ( d − g V ) / V 2 ; the chain rule and d m = g m e ˉ m d_m = g_m\bar{e}_m d m = g m e ˉ m give the score. The normal equations of the weighted criterion are ∑ m ω m q k ( m ) ( e ˉ m − V m ) = 0 \sum_m\omega_mq_k^{(m)}(\bar{e}_m-V_m)=0 ∑ m ω m q k ( m ) ( e ˉ m − V m ) = 0 , which coincide with the score equations exactly when ω m ∝ g m / V m 2 \omega_m \propto g_m/V_m^2 ω m ∝ g m / V m 2 with a constant common to all nodes; since the nodes realize a continuum of values of V V V , this forces ψ ( V ) ∝ V − 2 \psi(V) \propto V^{-2} ψ ( V ) ∝ V − 2 .
The block is an exact Gamma regression: given u u u , e ( u ; μ ) ∼ V σ ( u ) χ 1 2 e(u;\mu)\sim V\sigma(u)\chi_1^2 e ( u ; μ ) ∼ V σ ( u ) χ 1 2 , whose log-density in its mean V V V is − 1 2 log V − e / ( 2 V ) -\frac{1}{2}\log V-e/(2V) − 2 1 log V − e / ( 2 V ) up to terms free of V V V , and summing against γ i m \gamma_{im} γ im reproduces Q Q Q . The link is the identity, which is not canonical for the Gamma family, so Q Q Q is not globally concave in σ 2 \sigma^2 σ 2 .
Proposition 17 (Where the block is concave). The Hessian of Q Q Q in σ 2 \sigma^2 σ 2 is
H = 1 2 ∑ m q ( m ) q ( m ) T g m ( V m − 2 e ˉ m ) / V m 3 . H=\frac{1}{2}\sum_m q^{(m)}q^{(m)\mathsf{T}}g_m(V_m-2\bar e_m)/V_m^3. H = 2 1 m ∑ q ( m ) q ( m ) T g m ( V m − 2 e ˉ m ) / V m 3 . Hence Q Q Q is concave on the convex set { σ 2 : V m < 2 e ˉ m for every m } \{\sigma^2:V_m<2\bar e_m\text{ for every }m\} { σ 2 : V m < 2 e ˉ m for every m } , strictly so when the q ( m ) q^{(m)} q ( m ) span R 2 \mathbb{R}^2 R 2 , and then a stationary point in that set is its unique maximizer there.
Proof. Differentiate the score once more; on the set every factor V m − 2 e ˉ m V_m-2\bar e_m V m − 2 e ˉ m is negative. □
Proposition 18 (The variance update is Fisher scoring). Let
T ( σ 2 ) = [ ∑ m ω m q ( m ) q ( m ) T ] − 1 ∑ m ω m q ( m ) e ˉ m . T(\sigma^2)=\left[\sum_m\omega_mq^{(m)}q^{(m)\mathsf{T}}\right]^{-1}\sum_m\omega_mq^{(m)}\bar e_m. T ( σ 2 ) = [ m ∑ ω m q ( m ) q ( m ) T ] − 1 m ∑ ω m q ( m ) e ˉ m . with ω m = g m / V m 2 \omega_m=g_m/V_m^2 ω m = g m / V m 2 evaluated at σ 2 \sigma^2 σ 2 , let I = 1 2 ∑ m ω m q ( m ) q ( m ) T \mathcal{I}=\frac{1}{2}\sum_m\omega_mq^{(m)}q^{(m)\mathsf{T}} I = 2 1 ∑ m ω m q ( m ) q ( m ) T and let U = ∇ σ 2 Q U=\nabla_{\sigma^2}Q U = ∇ σ 2 Q . Then T ( σ 2 ) = σ 2 + I − 1 U T(\sigma^2)=\sigma^2+\mathcal{I}^{-1}U T ( σ 2 ) = σ 2 + I − 1 U . Its fixed points are the stationary points of Q Q Q in σ 2 \sigma^2 σ 2 , T ( σ 2 ) − σ 2 T(\sigma^2)-\sigma^2 T ( σ 2 ) − σ 2 is an ascent direction whenever U ≠ 0 U\ne0 U = 0 , and near a fixed point the iteration converges linearly at rate equal to the spectral radius of I − 1 R \mathcal{I}^{-1}R I − 1 R , with R = ∑ m q ( m ) q ( m ) T g m ( V m − e ˉ m ) / V m 3 R=\sum_mq^{(m)}q^{(m)\mathsf{T}}g_m(V_m-\bar e_m)/V_m^3 R = ∑ m q ( m ) q ( m ) T g m ( V m − e ˉ m ) / V m 3 .
Proof. I σ 2 + U = 1 2 ∑ m ω m q ( m ) V m + 1 2 ∑ m ω m q ( m ) ( e ˉ m − V m ) = 1 2 ∑ m ω m q ( m ) e ˉ m \mathcal{I}\sigma^2+U=\frac{1}{2}\sum_m\omega_mq^{(m)}V_m+\frac{1}{2}\sum_m\omega_mq^{(m)}(\bar e_m-V_m)=\frac{1}{2}\sum_m\omega_mq^{(m)}\bar e_m I σ 2 + U = 2 1 ∑ m ω m q ( m ) V m + 2 1 ∑ m ω m q ( m ) ( e ˉ m − V m ) = 2 1 ∑ m ω m q ( m ) e ˉ m ; multiply by I − 1 \mathcal{I}^{-1} I − 1 . I \mathcal{I} I is positive definite when the q ( m ) q^{(m)} q ( m ) span R 2 \mathbb{R}^2 R 2 . Since H = − I + R H=-\mathcal{I}+R H = − I + R , differentiating T T T at a fixed point, where U = 0 U=0 U = 0 , gives D T = I − 1 R DT=\mathcal{I}^{-1}R D T = I − 1 R . □
The rate is driven by the node-wise mismatch e ˉ m − V m \bar e_m-V_m e ˉ m − V m , which has conditional mean zero at a correctly specified configuration, so the iteration is fast where the model fits. A unit scoring step is not guaranteed to keep the variances positive; the implementation rejects such steps (Appendix F.1).