betaFeedback

WorksMathematics

Improved estimation of differential entropy of a multivariate Gaussian distribution

Abstract

We consider estimation of entropy of a multivariate variate normal distribution with unknown mean vector and unknown covariance matrix. Using a decision-theoretic approach, we study the problem under a general location-invariant loss function that is strictly convex and minimized at zero. We first characterize the class of affine equivariant estimators and obtain the best affine equivariant estimator. We further show that the best affine equivariant estimator is inadmissible under the general loss. Exploiting information in the sample mean, we derive Stein-type improved estimators, including an extension to the case of a mean matrix. Under the linex loss, we obtain a smooth Brewster–Zidek-type estimator that dominates the best affine equivariant estimator, and we show that this estimator is generalized Bayes under a suitable non-informative prior.

Introduction

In information theory, a well-known measure of uncertainty of a probability distribution is the entropy. Shannon used the term entropy in 1948 within the communication theory and denotes the average quantity of information generated by stochastic source data. The method for evaluating thermodynamic properties like entropy is an important problem. It is especially applied in the fields of molecular biology, chemistry, and molecular physics [3]. Let YY be a random variable with probability density function Λ(y∣θ)\Lambda(y|\theta). Shannon’s entropy is defined as

H(θ)=−E(ln⁡Λ(Y∣θ))H(\theta) = -E(\ln\Lambda(Y|\theta))

Shannon’s entropy is used in various areas of science and technology, such as molecular bio-sciences, hydrology, and software reliability. In the literature, several authors studied the estimation of entropy in non-parametric problems. However, very few authors have studied the estimation of entropy using parametric inference techniques, mainly using the decision-theoretic approach. Over the past two decades, some researchers have used a decision-theoretic approach to estimate the entropy of various continuous probability distributions. [6] studied the estimation of entropy of a multivariate normal distribution with respect to squared error loss. They have calculated the bias and risk values of the maximum likelihood estimator (MLE) and best affine equivariant estimator (BAEE). [6] proved the inadmissibility of BAEE by proposing improved estimators. To prove this, they have obtained improved [15] and [1] type estimators. The authors also demonstrated that the [1] type improved estimators are generalized Bayes. [4] have considered the estimation of entropy of several exponential distributions with common scale parameters and different location parameters under the squared error loss function. They have proposed sufficient under which BAEE is inadmissible. As a consequence, the authors have obtained an improved estimator. Estimation Renyi entropy of several exponential populations has been studied by [5] with respect to the squared error loss function. They have derived UMVUE. The authors proposed sufficient conditions for improvement over affine and scale equivariant estimators. As an application, they derived estimators that dominate UMVUE and MLE. [10] studied the estimation entropy of an exponential distribution under a general location invariant loss function. The authors prove the inadmissibility of BAEE by proposing various improved estimators. Improved estimation entropy of an exponential population based on doubly censored sample considered by [9]. They have proposed various estimators that dominate BAEE. Improved estimation of the entropy of a scale mixture of exponential distribution has been discussed by [11]. They have obtained estimators that dominate BAEE. Applying these results, the authors derived improved estimators for the entropy of a multivariate Lomax distribution. Recently [7] discuss the problem of finding improved estimators selected entropy for two gamma distributions. For this problem, they have considered the squared error loss function. Several shrinkage estimators are obtained that dominate naive estimators. From the above literature review, we have seen that there is not much work for estimating the entropy of multivariate distribution except [6].

In this paper, we consider the problem of estimating entropy of a multivariate normal distribution. Let X1,X2,…,XnX_1, X_2, \ldots, X_n be a random sample form a Np(η,Σ)N_p(\eta, \boldsymbol{\Sigma}) having density

f(x;η,Σ)=1(2π)p/2∣Σ∣1/2exp⁡(−12(x−η)tΣ−1(x−η)).(1)f(x; \eta, \boldsymbol{\Sigma}) = \frac{1}{(2\pi)^{p/2}|\boldsymbol{\Sigma}|^{1/2}} \exp\left(-\frac{1}{2}(x-\eta)^{t}\boldsymbol{\Sigma}^{-1}(x-\eta)\right). \tag*{(1)}

The entropy of a pp-variate normal random variable XX with a probability density function fη,Σ(x)f_{\eta,\boldsymbol{\Sigma}}(x), where η∈Rp\eta\in\mathbb{R}^p is the mean vector and Σ\boldsymbol{\Sigma} is the p×pp \times p positive definite covariance matrix, is provided by

Hp(Σ)=E(−ln⁡fη,Σ(X))=p2[1+ln⁡(2π)]+ln⁡∣Σ∣2.\begin{aligned} H_p(\boldsymbol{\Sigma}) &= E\left(-\ln f_{\eta,\boldsymbol{\Sigma}}(X)\right) \\ &= \frac{p}{2}[1+\ln(2\pi)] + \frac{\ln|\boldsymbol{\Sigma}|}{2}. \end{aligned}

So estimating Hp(Σ)H_p(\boldsymbol{\Sigma}) is equivalent to estimating θ=ln⁡∣Σ∣\theta= \ln|\boldsymbol{\Sigma}| of multivariate normal distribution. A complete sufficient statistics is

Xˉ=1m∑i=1mXi,X=mXˉ and S=∑i=1m(Xi−Xˉ)(Xi−Xˉ)t.\bar{X} = \frac{1}{m}\sum_{i=1}^{m} X_i,\quad X = \sqrt{m}\bar{X}\ \text{and}\ S = \sum_{i=1}^{m}(X_i-\bar{X})(X_i-\bar{X})^t.

Then we have

X∼Np(μ,Σ)S∼Wp(m−1,Σ),where μ=mη, n=m−1.(2)X \sim N_p(\mu,\boldsymbol{\Sigma})\quad S \sim W_p(m-1,\boldsymbol{\Sigma}),\quad\text{where }\mu= \sqrt{m}\eta,\ n=m-1. \tag*{(2)}

are independently distributed, here Wp(n,Σ)W_p(n,\boldsymbol{\Sigma}) denotes the p-variate Wishart distribution with m−1m-1 degrees of freedom and covariance matrix Σ\boldsymbol{\Sigma}. In this paper we will consider the estimation of θ=ln⁡∣Σ∣\theta= \ln|\boldsymbol{\Sigma}| with respect to location invariant loss function L(δ−θ)L(\delta-\theta). Assume that the loss function L(t)L(t) under consideration satisfies the following conditions (A1) L(t)L(t) is twice differentiable and strictly convex function in R\mathbb{R}, where R\mathbb{R} denotes the set of real numbers. Further, L(t)L(t) is minimized at t=0t=0 with L(0)=0L(0)=0.

(A2) Integrals involving L(t)L(t) are finite and differentiation under the integral sign is permissible.

The remaining sections of this paper are arranged as follows. In § 2, we derive an estimator that improves upon the BAEE. Moreover we have obtain a generalized Bayes estimator which coincides with BAEE under an improper prior. Furthermore in § 3, we obtained a improved estimator using Stein type approach. Further we have extended the problem of estimation of entropy for matrix variate normal distribution in § 4. In § 5 we have obtained a smooth improved estimator for linex loss function. Lastly, we give a conclusion in § 6.

Best affine equivariant estimator

In this section, we derive BAEE and its properties. Consider the affine group AlnAl_n that is the collection of pairs (A,b)(A,b), where AA is a p×pp \times p non singular matrix, b∈Rpb \in\mathbb{R}^p. The group AlpAl_p act on the X=Rp×Sp+\mathcal{X}=\mathbb{R}^p \times S_p^+ in the following manner

(A,b)(X,S)=(AX+b,ASAt)(A,b)(μ,Σ)=(Aμ+b,AΣAt),(A,b)(X,S)=(AX+b,ASA^t) \qquad(A,b)(\mu,\Sigma)=(A\mu+b,A\Sigma A^t),

where Sp+S_p^+ set of all p×pp \times p positive definite matrix. The estimation problem is invariant under the affine group AlpAl_p. Under this group action we have ln⁡∣Σ∣→ln⁡∣Σ∣+ln⁡∣A∣2\ln|\Sigma| \to\ln|\Sigma|+\ln|A|^2 for all p×pp \times p non-singular real matrices AA and all b∈Rpb \in\mathbb{R}^p. An estimator δ(X,S)\delta(X,S) is called AlnAl_n-equivariant or affine equivariant estimator if δ(X,S)\delta(X,S) satisfies

δ(AX+b,ASAt)=δ(X,S)+ln⁡∣A∣2\delta(AX+b,ASA^t)=\delta(X,S)+\ln|A|^2

Lemma 2.1 The form of an affine equivariant estimator of θ\theta is given as δc(X,S)=ln⁡∣S∣−c\delta_c(X,S)=\ln|S|-c for some real constant cc.

Proof. An affine equivariant estimator satisfies

δ(AX+b,ASAt)=δ(X,S)+ln⁡∣A∣2.(3)\delta(AX+b,ASA^t)=\delta(X,S)+\ln|A|^2. \tag*{(3)}

We have SS is a positive definite matrix so there exist a positive definite matrix S1/2S^{1/2} such that S=S1/2S1/2S=S^{1/2}S^{1/2}. We take b=−AXb=-AX and A=HS−1/2A=HS^{-1/2} in (3), where HH is an p×pp \times p orthogonal matrix and S−1/2S^{-1/2} is the inverse of S1/2S^{1/2}. Then we get the form of any affine equivariant estimator as

δc(X,S)=ln⁡∣S∣+c\delta_c(X,S)=\ln|S|+c

Remark 2.1 If c=pln⁡nc=p\ln n, then we get the MLE of ln⁡∣Σ∣\ln|\Sigma| as δml(X,S)=ln⁡∣1nS∣\delta_{ml}(X,S)=\ln|\frac{1}{n}S| is mle of ln⁡Σ\ln\Sigma. So the MLE belongs to the class of affine equivariant estimator. The following theorem gives the best affine equivariant estimator of ln⁡∣Σ∣\ln|\Sigma|.

Theorem 2.2 Under the location invariant loss function L(δ−θ)L(\delta-\theta) the best affine equivariant estimator is given as δ0(X,S)=ln⁡∣S∣+c0\delta_0(X,S)=\ln|S|+c_0, where c0c_0 is the solution of

E[L′(ln⁡V+c0)]=0,(4)E\left[L'(\ln V+c_0)\right]=0, \tag*{(4)}

with V=∣S∣∣Σ∣∼∏i=1pχm−i2V=\frac{|S|}{|\Sigma|}\sim\prod_{i=1}^{p}\chi^2_{m-i}.

Proof. The risk of δc(X,S)\delta_c(X,S) is obtained as

R(c)=EL(ln⁡∣S∣+c−ln⁡∣Σ∣)=EL(ln⁡∣SΣ∣+c)=EL(ln⁡∣SΣ∣+c).R(c)=EL(\ln|S|+c-\ln|\Sigma|)=EL\left(\ln\left|\frac{S}{\Sigma}\right|+c\right)=EL\left(\ln\left|\frac{S}{\Sigma}\right|+c\right).

Since, the L(t)L(t) is a strictly bowl-shaped loss function so L(t)L(t) is convex. This proves the result.

Example 2.1 Under the quadratic loss function L1(t)=t2L_1(t)=t^2, the BAEE of ln⁡∣Σ∣\ln|\Sigma| is given by

δ01(X,S)=ln⁡∣S∣+c01,\delta_{01}(X,S)=\ln|S|+c_{01},

where c01=−pln⁡2−∑i=1pψ(m−i2)c_{01}=-p\ln2-\sum_{i=1}^{p}\psi\left(\frac{m-i}{2}\right). Moreover, δ01(X,S)\delta_{01}(X,S) is also an unbiased estimator.

Example 2.2 For the linex loss function L2(t)=eat−at−1L_2(t)=e^{at}-at-1, a∈R/{0}a\in\mathbb{R}/\{0\}, the BAEE of ln⁡∣Σ∣\ln|\Sigma| is given by

δ02(X,S)=ln⁡∣S∣+c02,\delta_{02}(X,S)=\ln|S|+c_{02},

where c02=−pln⁡2−1a∑i=1pln⁡(Γ(m−i2+a)Γ(m−i2))c_{02}=-p\ln2-\frac{1}{a}\sum_{i=1}^{p}\ln\left(\frac{\Gamma\left(\frac{m-i}{2}+a\right)}{\Gamma\left(\frac{m-i}{2}\right)}\right).

Remark 2.2 The best affine equivariant estimator δ01\delta_{01} is also an unbiased estimator of ln⁡∣Σ∣\ln|\Sigma|, which is rare statistical phenomenon. It follows that the BAEE δ01\delta_{01} is better alternative to δ02\delta_{02} in terms of bias.

References

  1. [1]Brewster, J. F. and Zidek, J. [1974]. Improving on equivariant estimators, The Annals of Statistics. 2(1), 21–38.DOI
  2. [2]Iliopoulos, G. and Kourouklis, S. [1999]. Improving on the best affine equivariant estimator of the ratio of generalized variances, Journal of multivariate analysis. 68(2),176–192.DOI
  3. [3]Jusuf, S., Loll, P. J. and Axelsen, P. H. [2002]. The role of configurational entropy in biochemical cooperativity, Journal of the American Chemical Society. 124(14), 3490–3491.pubmed.ncbi.nlm.nih.gov/11929222
  4. [4]Kayal, S. and Kumar, S. [2013]. Estimation of the shannon’s entropy of several shifted exponential populations, Statistics and Probability Letters. 83(4), 1127–1135.DOI
  5. [5]Kayal, S., Kumar, S., Vellaisamy, P. et al. [2015]. Estimating the Rényi entropy of several exponential populations, Brazilian Journal of Probability and Statistics. 29(1), 94–111.DOI
  6. [6]Misra, N., Singh, H. and Demchuk, E. [2005]. Estimation of the entropy of a multivariate normal distribution, Journal of multivariate analysis. 92(2), 324–342.
  7. [7]Misra, N. et al. [2023]. Shrinkage estimators dominating some naive estimators of the selected entropy, IEEE Transactions on Information Theory. 70(532–549).arxiv.org/abs/2209.10330
  8. [8]Muirhead, R. J. [2009]. Aspects of multivariate statistical theory, John Wiley & Sons.
  9. [9]Patra, L. K. [2020]. On the improved estimation of a function of the scale parameter of an exponential distribution based on doubly censored sample, Journal of Applied Statistics. 47(9), 1637–1651.DOI
  10. [10]Patra, L. K., Kayal, S. and Kumar, S. [2020]. Estimating a function of scale parameter of an exponential population with unknown location under general loss function, Statistical Papers. 61(6), 2511–2527.
  11. [11]Petropoulos, C., Kanta Patra, L. and Kumar, S. [2020]. Improved estimators of the entropy in scale mixture of exponential distributions, Brazilian Journal of Probability and Statistics. 34(3), 580–593.DOI
  12. [12]Rohatgi, V. K. and Saleh, A. M. E. [2015]. An introduction to probability and statistics, John Wiley & Sons.
  13. [13]Sarkar, S. K. [1991]. Stein-type improvements of confidence intervals for the generalized variance, Annals of the Institute of Statistical Mathematics. 43, 369–375.DOI
  14. [14]Shorrock, R. and Zidek, J. [1976]. An improved estimator of the generalized variance, The Annals of Statistics. pp. 629–638.DOI
  15. [15]Stein, C. (1964). Inadmissibility of the usual estimator for the variance of a normal distribution with unknown mean, Annals of the Institute of Statistical Mathematics. 16(1), 155–160.DOI
  16. [16]Xiaoqian, S. and Wankai, P. [2000]. Improved estimation of the generalized precision under the entropy loss, Acta Mathematicae Applicatae Sinica. 16, 162–170.

Paper details

vlnc:W_sfx5qj3f
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY-SA 4.0
Usage

views · 8

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Mathematics

Not recorded for this paper

View statement
Looking for related papers…
Contents