betaFeedback

WorksMathematics

Bivariate Renyi Tsallis Divergence

Abstract

In this paper, a bivariate extension of the differential Renyi Tsallis divergence is introduced for comparing two absolutely continuous bivariate probability distributions. Fundamental properties of the proposed divergence are studied, including non negativity, pseudo additivity under component wise independence, and invariance under monotone and nonsingular linear transformations. Relationships with Renyi, Tsallis, alpha, Bhattacharyya, and Hellinger divergence measures are also established. A sensitivity analysis is performed to compare the Renyi, Tsallis and Renyi Tsallis divergences under the Marshall Olkin and Gumbel type bivariate exponential models, taking the magnitude of Kendall's rank correlation coefficient as a common scale for dependence. The proposed measure is further developed for bivariate order statistics, with particular emphasis on the joint distribution of the minimum and maximum order statistics for several lifetime distributions, including exponential, Weibull, Pareto, and Rayleigh models. When the underlying distributions are unknown, a nonparametric estimator is constructed using bivariate Gaussian kernel density estimators with plug in bandwidth selection. The proposed estimator is applied to a real life dataset and bootstrap resampling is used to assess estimation uncertainty. The results show the applicability of the proposed divergence for comparing bivariate distributions and indicate that its finite sample performance depends on the order parameter and the distributions being compared.

Introduction

The notion of entropy was first introduced by Clausius [4] as a measure of disorder in thermodynamics. This concept was later extended to information theory by Shannon [15] to measure the average uncertainty in a random variable. For a continuous non-negative random variable XX with probability density function fX(x)f_X(x), the differential entropy is defined as,

H(X)=−∫0∞f(x)log⁡f(x) dx=−EX[log⁡f(X)].(1)H(X)=-\int_{0}^{\infty} f(x)\log f(x)\,dx=-E_X[\log f(X)]. \tag*{(1)}

Among the several generalizations of entropy measures, Rényi entropy introduced by Rényi [14] is one of the most important measures. For a continuous random variable XX with probability density function fX(x)f_X(x) the Rényi entropy of order α\alpha is given by,

Hα(X)=11−αlog⁡(∫0∞f(x)α dx)=11−αlog⁡(EX[f(X)α−1]),α>0, α≠1.(2)H_{\alpha}(X)=\frac{1}{1-\alpha}\log\left(\int_{0}^{\infty}f(x)^{\alpha}\,dx\right)=\frac{1}{1-\alpha}\log\left(E_X\left[f(X)^{\alpha-1}\right]\right),\quad\alpha>0,\ \alpha\ne1. \tag*{(2)}

Tsallis [16] introduced another generalization called Tsallis entropy of order α\alpha. Tsallis entropy is widely used for analyzing non extensive data. For a continuous non-negative random variable XX with probability density function fX(x)f_X(x), the Tsallis entropy is defined as follows,

Tα(X)=1α−1[1−∫0∞f(x)α dx]=1α−1[1−EX(f(X)α−1)],α>0, α≠1.(3)T_{\alpha}(X)=\frac{1}{\alpha-1}\left[1-\int_{0}^{\infty}f(x)^{\alpha}\,dx\right]=\frac{1}{\alpha-1}\left[1-E_X\left(f(X)^{\alpha-1}\right)\right],\quad\alpha>0,\ \alpha\ne1. \tag*{(3)}

As α→1\alpha\to1, both equation (1.2) and (1.3) converge to equation (1.1).

Apart from entropy measures Kullback and Leibler [11] introduced the Kullback–Leibler (KL) divergence, provides a useful measure for comparing two probability distributions FX(⋅)F_X(\cdot) and GY(⋅)G_Y(\cdot), and is denoted by dKL(FX,GY)d_{KL}(F_X,G_Y). It is a measure of the inefficiency of assuming that the distribution is gY(x)g_Y(x) when the true distribution is fX(x)f_X(x). Let XX and YY be two continuous non-negative random variables with probability density functions fX(x)f_X(x) and gY(x)g_Y(x), respectively. Then the Kullback–Leibler (KL) divergence is defined as,

dKL(X,Y)=∫0∞f(x)log⁡f(x)g(x) dx=EX[log⁡(f(X)g(X))].(4)d_{KL}(X,Y)=\int_{0}^{\infty} f(x)\log\frac{f(x)}{g(x)}\,dx=E_X\left[\log\left(\frac{f(X)}{g(X)}\right)\right]. \tag*{(4)}

Ebrahimi et al. [8] developed the bivariate case of Kullback–Leibler divergence and studied its properties. Let X=(X1,X2)\mathbf{X}=(X_1,X_2) and Y=(Y1,Y2)\mathbf{Y}=(Y_1,Y_2) be two non-negative bivariate random vectors with absolutely continuous joint pdfs fX(x1,x2)f_X(x_1,x_2) and gY(x1,x2)g_Y(x_1,x_2) and the corresponding cdfs FX(x1,x2)F_X(x_1,x_2) and GY(x1,x2)G_Y(x_1,x_2) respectively. Then the bivariate KL divergence is defined as,

dKL(FX,GY)=∫0∞∫0∞fX(x1,x2)log⁡(fX(x1,x2)gY(x1,x2)) dx1 dx2=EX[log⁡(fX(X1,X2)gY(X1,X2))].(5)d_{KL}(F_X,G_Y)=\int_{0}^{\infty}\int_{0}^{\infty}f_X(x_1,x_2)\log\left(\frac{f_X(x_1,x_2)}{g_Y(x_1,x_2)}\right)\,dx_1\,dx_2 =\mathit{E}_X\left[\log\left(\frac{f_X(X_1,X_2)}{g_Y(X_1,X_2)}\right)\right]. \tag*{(5)}

For more information one can refer Contreras-Reyes and Arellano-Valle [5], Zhang et al. [18], Rafflesia Chackochan et al. [13]. Several generalizations of the KL divergence exist in the literature. Among them, the Rényi and Tsallis divergence measures of order α\alpha are the most studied. The Rényi divergence of order α\alpha between two continuous bivariate distributions FXF_X and GYG_Y is defined as

dRα(FX,GY)=1α−1[log⁡∫0∞∫0∞fXα(x1,x2)gY1−α(x1,x2) dx1 dx2]=1α−1[log⁡EX(fX(X1,X2)gY(X1,X2))α−1],0<α≠1<∞.(6)d_R^\alpha(F_X,G_Y)=\frac{1}{\alpha-1}\left[\log\int_{0}^{\infty}\int_{0}^{\infty}f_X^\alpha(x_1,x_2)g_Y^{1-\alpha}(x_1,x_2)\,dx_1\,dx_2\right] =\frac{1}{\alpha-1}\left[\log E_X\left(\frac{f_X(X_1,X_2)}{g_Y(X_1,X_2)}\right)^{\alpha-1}\right],\qquad0<\alpha\ne1<\infty. \tag*{(6)}

Similarly, the Tsallis divergence of order α\alpha is defined as

dTα(FX,GY)=1α−1[∫0∞∫0∞fXα(x1,x2)gY1−α(x1,x2) dx1 dx2−1]=1α−1[EX(fX(X1,X2)gY(X1,X2))α−1−1],0<α≠1<∞.(7)d_T^\alpha(F_X,G_Y)=\frac{1}{\alpha-1}\left[\int_{0}^{\infty}\int_{0}^{\infty}f_X^\alpha(x_1,x_2)g_Y^{1-\alpha}(x_1,x_2)\,dx_1\,dx_2-1\right] =\frac{1}{\alpha-1}\left[E_X\left(\frac{f_X(X_1,X_2)}{g_Y(X_1,X_2)}\right)^{\alpha-1}-1\right],\qquad0<\alpha\ne1<\infty. \tag*{(7)}

References

  1. [1]Aharon Ben-Tal, Adi Ben-Israel, and Marc Teboulle. Certainty equivalents and information measures: Duality and extremal principles. Journal of Mathematical Analysis and Applications, 157(1):211–236, 1991. doi: 10.1016/0022-247X(91)90140-N.DOI
  2. [3]Andrzej Cichocki, Rafal Zdunek, Anh Huy Phan, and Shun-ichi Amari. Nonnegative Matrix and Tensor Factorizations: Applications to Exploratory Multi-Way Data Analysis and Blind Source Separation. John Wiley & Sons, Chichester (UK), 2009.DOI
  3. [4]R. Clausius. On different forms of the fundamental equations of the mechanical theory of heat and their convenience for application. In The Second Law of Thermodynamics, pages 162–193. 1865.
  4. [5]Javier E. Contreras-Reyes and Reinaldo B. Arellano-Valle. Kullback–leibler divergence measure for multivariate skew-normal distributions. Entropy, 14(9):1606–1626, 2012. doi: 10.3390/e14091606.
  5. [6]S. S. Dragomir and V. Gluscevic. Some inequalities for the Kullback–Leibler and χ²-distances in information theory and applications. RGMIA Research Report Collection, 3(2):199–210, 2000.
  6. [7]Tarn Duong. Using ks for bivariate kernel density estimation, 2007. R package documentation.
  7. [8]Nader Ebrahimi, S. N. U. A. Kirmani, and Ehsan S. Soofi. Multivariate dynamic information. Journal of Multivariate Analysis, 98(2):328–349, 2007. doi: 10.1016/j.jmva.2005.08.004.
  8. [9]Ernst Hellinger. Die Orthogonalinvarianten Quadratischer Formen von Unendlichvielen Variabeln. W. Fr. Kaestner, Leipzig (DE), 1907.
  9. [10]A. Khammar and H. A. Noughabi. On differential Renyi’s–Tsallis divergence measure and its applications. Journal of Statistical Computation and Simulation, 92(15):3109–3128, 2022. doi: 10.1080/00949655.2022.2043417.DOI
  10. [11]S. Kullback and R. A. Leibler. On information and sufficiency. The Annals of Mathematical Statistics, 22(1):79–86, 1951. doi: 10.1214/aoms/1177729694.DOI
  11. [12]Michael D. Perlman. Jensen’s inequality for a convex vector-valued function on an infinite-dimensional space. Journal of Multivariate Analysis, 4(1):52–65, 1974. doi: 10.1016/0047-259X(74)90020-8.DOI
  12. [13]M. Rafflesia Chackochan, P. G. Sankaran, and N. Unnikrishnan Nair. Bivariate Kullback–Leibler divergence. Communications in Statistics – Theory and Methods, 55(1):292–312, 2026.
  13. [14]A. Renyi. On measures of entropy and information. In J. Neyman, editor, Proceedings of the Fourth Berkeley Symposium on Mathematical Statistics and Probability, Volume 1: Contributions to the Theory of Statistics, pages 547–562, Berkeley (CA), 1961. University of California Press.
  14. [15]C. E. Shannon. A mathematical theory of communication. The Bell System Technical Journal, 27(3):379–423, 1948. doi: 10.1002/j.1538-7305.1948.tb01338.x.DOI
  15. [16]C. Tsallis. Possible generalization of Boltzmann–Gibbs statistics. Journal of Statistical Physics, 52(1–2):479–487, 1988. doi: 10.1007/BF01016429.DOI
  16. [17]L. Wondie and S. Kumar. A joint representation of Renyi’s and Tsallis’s entropy with application in coding theory. International Journal of Mathematics and Mathematical Sciences, 2017:2683293, 2017. doi: 10.1155/2017/2683293.DOI
  17. [18]Yufeng Zhang, Jialu Pan, Kenli Li, Wanwei Liu, Zhenbang Chen, Xinwang Liu, and Ji Wang. On the properties of Kullback–Leibler divergence between multivariate Gaussian distributions. In Advances in Neural Information Processing Systems, volume 36, pages 58152–58165, 2023.arxiv.org/abs/2102.05485

Paper details

vlnc:W_47qp7688
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC0-1.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Mathematics

Not recorded for this paper

View statement
Looking for related papers…
Contents