Introduction
Exponential families constitute one of the most important classes of statistical models. Their broad applicability stems from a combination of analytical and statistical properties, including tractable likelihoods, finite-dimensional sufficient statistics, and the existence of natural parametrizations in terms of both natural and expectation parameters [14 ] (Barndorff-Nielsen 2014). They encompass many classical distributions, including Gaussian, Poisson, Bernoulli, Binomial, Gamma, and Beta distributions, and play a central role in statistical inference, information geometry, and machine learning.
An important connection between regular exponential families and convex analysis is provided by Bregman divergences. It is well established that, under standard regularity and minimality assumptions, the log-density of a regular exponential-family distribution can be expressed in terms of a Bregman divergence generated by the convex conjugate of the log-partition function [4 ] . This representation provides a geometric viewpoint of exponential families and has proved useful well beyond statistical interpretation. In particular, Bregman divergences constitute fundamental tools in optimization and statistical learning, where they underlie generalized projection methods, mirror descent, Bregman proximal algorithms, and a variety of estimation and learning procedures [9 , 6 ] . The Bregman representation of an exponential-family distribution also endows its data space with a natural geometry, with the sufficient statistics playing the role of geometric coordinates.
This representation, however, reveals an important asymmetry between the two arguments of the Bregman divergence. In the canonical representation, the first argument corresponds to the sufficient statistics, whereas the second one is associated with the expectation parameter. Convexity with respect to the first argument is guaranteed by the convexity of the Bregman generator. In contrast, convexity with respect to the second argument is not generally guaranteed. Thus, although the Bregman divergence naturally describes the geometry of the data space, its reverse form does not necessarily define an ordinary Bregman geometry on the expectation-parameter space. This difficulty is particularly pronounced when conducting asymptotically exact data augmentation [27 ] and deriving splitting-based Monte Carlo algorithms [26 ] with non-Euclidean and, in particular, Bregman kernels. With respect to these two tasks, respectively, it has recently been overcome by augmenting the statistical model with the same divergence evaluated with swapped arguments [15 ] or, equivalently, by replacing the Bregman divergences by symmetrized counterparts defined as Jensen-Bregman divergences [13 ] . More generally and beyond these purely modeling and algorithmic workarounds, this divergence asymmetry raises a more fundamental question: can a reverse Bregman divergence itself be represented as an ordinary Bregman divergence after a suitable transformation of the coordinates? Then, can the Bregman geometry underlying a regular exponential-family distribution be transferred to its parameter space through an appropriate dual coordinate system? From an information geometry perspective, this question can be formulated as the search for a geometric structure underlying conformal divergences as studied by Amari and Nagaoka [2 ] and Nock et al. [21 ] .
The objective of this paper is to answer this question. A sound, albeit incomplete, response is naturally provided by leveraging the role of Legendre duality in the geometry of exponential families [23 ] . However, the term duality , used hereafter and underlying the main rationale of this work, refers to a property intended to go beyond the sole Legendre duality. Thus, starting from the primal Bregman representation of regular exponential-family distributions, we build on a projective characterization providing necessary and sufficient conditions for the existence of such dual Bregman representations. We further establish a general construction of the corresponding dual coordinates and Bregman generators, together with normalization and uniqueness results. Among admissible representations, we exhibit two particular models, distinguished by the form of an associated scaling function and, consequently, by the resulting Bregman generator and parameter coordinates. The first one, referred to as the β \beta β -scaled dual Bregman representation, recovers the self-evident representation induced by Legendre duality in exponential families. In this case, the resulting geometry is determined by the gradient of the original Bregman generator together with a scaled version of its Legendre conjugate. The second class, referred to as the α \alpha α -scaled dual Bregman representation, relies on a different transformed coordinate system and an associated Bregman generator. Both constructions arise as particular instances of the general projective framework. These two constructions there fore provide distinct but equivalent geometric descriptions of the same exponential-family distribution. In particular, different parametrizations of the distribution can induce different Bregman geometries on its parameter space while remaining equivalent at the level of the underlying probability model.
An important application of the proposed framework concerns conjugate priors for exponential families. By applying the two dual Bregman representations previously identified to the conjugate distributions associated with regular exponential families, we obtain alternative geometric representations of these priors. These representations differ from the primal Bregman formulation of conjugate priors established by Agarwal and Daumé III (2010) in both their coordinate systems and associated Bregman generators, while retaining the defining conjugacy property under Bayesian updating. In this sense, the proposed framework does not modify the underlying probabilistic structure of conjugate priors, but provides alternative geometric descriptions of the same distributions. The resulting framework also recovers, as particular cases, dual Bregman representations associated with several classical statistical models and their conjugate priors. These examples illustrate how apparently different models can share common geometric structures and provide concrete interpretations of the proposed constructions.
Finally, the proposed framework also leads to consequences beyond the sole construction of these alternative Bregman representations. It is well known that relative smoothness generalizes the classical Lipschitz-gradient condition by measuring the variation of a function relative to a reference convex function, and has become an important notion in modern Bregman-based optimization [7 , 18 ] . We show that the geometries induced by the β \beta β - and α \alpha α -scaled models identified in this work provide natural reference geometries for establishing relative smoothness of the negative log-density associated with the original, primal Bregman representation. These results are extended to separable multidimensional models, thereby covering a broad class of statistical models with component-wise structure. For the geometries induced by both models, we derive explicit relative-smoothness bounds, with sharp constants in the scalar and separable settings. These results establish a direct connection between the geometry of exponential-family distributions and the convex-analytic notion of relative smoothness, which is central to modern Bregman-based optimization.
Addressing Poisson inverse problems leads to useful illustrations in which the proposed geometric framework provides a principled interpretation of choices that have appeared in previous optimization and statistical-learning approaches.
The main contributions of this work can be summarized as follows:
Building on the projective characterization stated in Theorem 2, which provides necessary and sufficient conditions for a reverse Bregman divergence to admit an ordinary Bregman representation in transformed coordinates, Theorem 3 clarifies the normalization freedom and the associated uniqueness properties. Assuming the existence of admissible dual Bregman representations, Proposition 4 then derives reconstruction formulas of such representations;
Theorem 4 then establishes the existence and uniqueness of dual Bregman representations for regular minimal exponential families, with explicit construction formulas given in Theorem 5. Theorem 6 specifies how these dual representations translate to the log-density of regular exponential family. Theorem 7 identifies two classes of admissible dual representations, corresponding to the so-called β \beta β - and α \alpha α -scaled constructions, as particular instances of the general projective framework;
Starting from Proposition 5, which establishes the primal representation of posterior distributions resulting from conjugate priors, Theorems 8 and 9 show that the β \beta β - and α \alpha α -scaled dual Bregman representations can be instantiated to derive as alternatives of these priors, while preserving conjugacy properties;
Table 1. Dual Bregman representations of classical exponential family models. Fixed parameters appear in gray. Whenever well-identified, the divergences resulting from the α \alpha α -scaled or β \beta β -scaled generators h h h are specified in Appendix C.
We apply the proposed framework to Poisson inverse problems to recover several results from the literature. In particular, we show that Bregman gradient iterations under the geometries induced by the α \alpha α -scaled and β \beta β -scaled model recover the NoLips ([7 ] ) and the expectation-maximization ([24 ] ) algorithms, respectively.
The remainder of the paper is organized as follows. § 2 recalls the necessary background on Legendre duality, Bregman geometry, relative smoothness and regular exponential families. § 3 states the projective characterization of dual Bregman representations and establishes their general construction, normalization and uniqueness properties. In particular, we instantiate these representations to exhibit dual representations of prior distributions required in a Bayesian framework and show that they naturally recover expected conjugacy properties. We also derive these representations for several classical exponential-family distributions. § 4 shows that these representations also provide suitable reference geometries to ensure relative smoothness properties. As an illustrative example, § 5 considers the case of Poisson inverse problems. Finally, § 6 concludes the paper and discusses possible extensions.
Background and preliminaries
Legendre duality
Notions of Legendre duality are recalled below following the terminology adopted by Rockafellar [23 ] .
Definition 1 (Legendre function). Let f : X ⊆ R n → R ∪ { + ∞ } f : \mathcal{X} \subseteq\mathbb{R}^{n} \to\mathbb{R} \cup\{+\infty\} f : X ⊆ R n → R ∪ { + ∞ } be a proper, lower semicontinuous, convex function . The function f f f is said to be essentially smooth if it is differentiable throughout int ( dom f ) \operatorname{int}(\operatorname{dom} f) int ( dom f ) and ∥ ∇ f ( x k ) ∥ → + ∞ \lVert\nabla f(x_k) \rVert\to+\infty ∥ ∇ f ( x k )∥ → + ∞ whenever x k ∈ int ( dom f ) x_k \in\operatorname{int}(\operatorname{dom} f) x k ∈ int ( dom f ) converges to a boundary point of dom f \operatorname{dom} f dom f . Moreover, if f f f is strictly convex on int ( dom f ) \operatorname{int}(\operatorname{dom} f) int ( dom f ) , it is called a Legendre function .
Definition 2 (Fenchel conjugate). Let f f f be a Legendre function with X = int ( dom f ) \mathcal{X} = \operatorname{int}(\operatorname{dom} f) X = int ( dom f ) . The Fenchel conjugate of f f f is defined for θ ∈ dom f ∗ \theta\in\operatorname{dom} f^{*} θ ∈ dom f ∗ by
f ∗ ( θ ) = sup x ∈ dom f { ⟨ θ , x ⟩ − f ( x ) } . f^{*}(\theta) = \sup_{\mathbf{x} \in\operatorname{dom} f} \left\lbrace\langle\theta,\mathbf{x}\rangle- f(\mathbf{x}) \right\rbrace. f ∗ ( θ ) = x ∈ dom f sup { ⟨ θ , x ⟩ − f ( x ) } . The associated dual coordinate is
θ = ∇ f ( x ) . \theta= \nabla f(\mathbf{x}). θ = ∇ f ( x ) .
A Legendre function and its conjugate satisfy the following duality property.
Proposition 1 (Legendre duality). The function f : X ⊆ R n → R ∪ { + ∞ } f:\mathcal{X}\subseteq\mathbb{R}^{n}\to\mathbb{R}\cup\{+\infty\} f : X ⊆ R n → R ∪ { + ∞ } is a Legendre function if and only if its Fenchel conjugate f ∗ f^{*} f ∗ is also a Legendre function. Moreover, its gradient ∇ f : int ( dom f ) → int ( dom f ∗ ) \nabla f:\operatorname{int}(\operatorname{dom} f)\to\operatorname{int}(\operatorname{dom} f^{*}) ∇ f : int ( dom f ) → int ( dom f ∗ ) is a bijection and satisfies the identity
∇ f ∗ = ( ∇ f ) − 1 \nabla f^{*}=(\nabla f)^{-1} ∇ f ∗ = ( ∇ f ) − 1 or, equivalently, ∀ x ∈ X , θ = ∇ f ( x ) ⇔ x = ∇ f ∗ ( θ ) \forall x\in\mathcal{X},\ \theta=\nabla f(x)\Leftrightarrow x=\nabla f^{*}(\theta) ∀ x ∈ X , θ = ∇ f ( x ) ⇔ x = ∇ f ∗ ( θ ) . Moreover, one has
f ( x ) + f ∗ ( ∇ f ( x ) ) = ⟨ x , ∇ f ( x ) ⟩ . f(x)+f^{*}(\nabla f(x))=\langle x,\nabla f(x)\rangle. f ( x ) + f ∗ ( ∇ f ( x )) = ⟨ x , ∇ f ( x )⟩ . Thus, for a Legendre function, the gradient establishes a one-to-one correspondence between the respective interiors of the effective domains of f f f and f ∗ f^{*} f ∗ . These properties are central in the construction of divergences underlying the Bregman geometry.
Bregman geometry
Definition 3 (Bregman divergences). ([9 ] ) Let f : X ⊆ R n → R ∪ { + ∞ } f:\mathcal{X}\subseteq\mathbb{R}^{n}\to\mathbb{R}\cup\{+\infty\} f : X ⊆ R n → R ∪ { + ∞ } be a proper convex function that is differentiable on int ( dom f ) \operatorname{int}(\operatorname{dom} f) int ( dom f ) . For y ∈ dom f y\in\operatorname{dom} f y ∈ dom f and x ∈ int ( dom f ) x\in\operatorname{int}(\operatorname{dom} f) x ∈ int ( dom f ) , the Bregman divergence induced by f f f is defined as
d f ( y , x ) = f ( y ) − f ( x ) − ⟨ ∇ f ( x ) , y − x ⟩ . d_f(y,x)=f(y)-f(x)-\langle\nabla f(x),y-x\rangle. d f ( y , x ) = f ( y ) − f ( x ) − ⟨ ∇ f ( x ) , y − x ⟩ . When f f f is strictly convex, d f ( y , x ) ≥ 0 d_f(y,x)\geq0 d f ( y , x ) ≥ 0 , with
d f ( y , x ) = 0 ⟺ y = x . d_f(y,x)=0\ \Longleftrightarrow\ y=x. d f ( y , x ) = 0 ⟺ y = x .
It is well-known that the Bregman divergences satisfy the duality identity recalled in the following proposition.
Proposition 2 (Duality of Bregman divergences). ([9 ] ) Let f f f be a Legendre function. For every x , y ∈ int ( dom f ) x,y\in\operatorname{int}(\operatorname{dom} f) x , y ∈ int ( dom f ) , one has
d f ( y , x ) = d f ∗ ( ∇ f ( x ) , ∇ f ( y ) ) . (1) d_f(y,x)=d_{f^{*}}(\nabla f(x),\nabla f(y)). \tag*{(1)} d f ( y , x ) = d f ∗ ( ∇ f ( x ) , ∇ f ( y )) . (1)
Relative smoothness
Relative smoothness generalizes the definition of Lipschitz continuity of the gradient by replacing the Euclidean quadratic distance with a Bregman divergence ([7 ] ).
Definition 4 (Relative smoothness). Let h h h be a Legendre function and let g g g be a convex function, differentiable on int ( dom h ) \operatorname{int}(\operatorname{dom} h) int ( dom h ) , with dom h ⊆ dom g \operatorname{dom} h\subseteq\operatorname{dom} g dom h ⊆ dom g , We say that the function g g g is L L L -smooth relative to h h h if ∃ L > 0 \exists L>0 ∃ L > 0 such that the function L h − g Lh-g L h − g is convex on int ( dom h ) \operatorname{int}(\operatorname{dom} h) int ( dom h ) .
Interestingly, this so-called Lipschitz-like/convexity condition (LC) yields a Bregman descent lemma, as stated below.
Proposition 3. ([7 Lemma 1] ) Let g g g be L L L -smooth relative to h h h . Then, for all x , x ′ ∈ int ( dom h ) x,x'\in\operatorname{int}(\operatorname{dom} h) x , x ′ ∈ int ( dom h ) , one has
g ( x ′ ) ≤ g ( x ) + ⟨ ∇ g ( x ) , x ′ − x ⟩ + L d h ( x ′ , x ) g(x')\leq g(x)+\langle\nabla g(x),x'-x\rangle+L d_h(x',x) g ( x ′ ) ≤ g ( x ) + ⟨ ∇ g ( x ) , x ′ − x ⟩ + L d h ( x ′ , x ) where d h d_h d h is the Bregman divergence induced by the generator h h h .
While the Lipschitz continuity of the gradient imposes a Euclidean curvature bound, relative smoothness allows the Euclidean geometry to be replaced by a geometry adapted to the problem. In particular, when designing descent-based algorithms, the conventional yet restrictive gradient Lipschitz condition can be lifted by requiring instead that the curvature of the objective be controlled relative to that of a suitable reference function. For instance, one of the main algorithmic consequences of relative smoothness can be formulated as follows: for any γ > 0 \gamma> 0 γ > 0 such that γ ≤ 1 / L \gamma\le1/L γ ≤ 1/ L , the Bregman gradient update Auslender and Teboulle, [3 ]
x + ∈ arg min u { ⟨ ∇ g ( x ) , u − x ⟩ + 1 γ d h ( u , x ) } \mathbf{x}^{+} \in\underset{\mathbf{u}}{\arg\min}\left\{\langle\nabla g(\mathbf{x}),\mathbf{u}-\mathbf{x}\rangle+ \frac{1}{\gamma}d_h(\mathbf{u},\mathbf{x})\right\} x + ∈ u arg min { ⟨ ∇ g ( x ) , u − x ⟩ + γ 1 d h ( u , x ) }
satisfies
g ( x + ) ≤ g ( x ) − ( 1 γ − L ) d h ( x + , x ) . g(\mathbf{x}^{+}) \le g(\mathbf{x})-\left(\frac{1}{\gamma}-L\right)d_h(\mathbf{x}^{+},\mathbf{x}). g ( x + ) ≤ g ( x ) − ( γ 1 − L ) d h ( x + , x ) .
In particular, when γ < 1 / L \gamma< 1/L γ < 1/ L , the objective decreases monotonically along the iterates, while γ = 1 / L \gamma= 1/L γ = 1/ L guarantees non-increase. This result is the Bregman analogue of the classical descent guarantee for gradient descent under a Lipschitz-gradient assumption: the Euclidean quadratic term L 2 ∥ u − x ∥ 2 \frac{L}{2}\|\mathbf{u}-\mathbf{x}\|^2 2 L ∥ u − x ∥ 2 is replaced by the more general Bregman divergence L d h ( u , x ) Ld_h(\mathbf{u},\mathbf{x}) L d h ( u , x ) . Consequently, identifying a suitable reference geometry h h h and establishing a relative-smoothness constant L L L directly provides a valid step-size range for Bregman gradient or proximal-gradient methods. For instance, this formalism has proved particularly useful to solve estimation problems with Poisson distributed data, where it provides Bregman descent inequalities without requiring globally Lipschitz gradients. For Poisson linear inverse problems, Burg’s entropy provides a suitable reference function h h h satisfying the relative-smoothness condition [7 ] . This geometry has subsequently been exploited in Poisson imaging problems [16 , 25 , 12 ] and Kullback-Leibler regression [10 ] . These examples illustrate the importance of choosing a reference geometry adapted to the statistical model. In this work, we investigate such geometric structures in the broader setting of regular minimal exponential families, whose Bregman representations provide the starting point for our analysis.
Regular exponential families
In this paper, we consider regular exponential families whose natural parametrization is defined below. Let ω \boldsymbol{\omega} ω denote an observation and let y = T ( ω ) ∈ R n \mathbf{y}=T(\boldsymbol{\omega})\in\mathbb{R}^{n} y = T ( ω ) ∈ R n be its sufficient statistic.
Definition 5 (Natural parametrization). An exponential family admits the natural parametrization
p ( y ; θ ) = exp ( ⟨ θ , y ⟩ − ψ ( θ ) ) p 0 ( y ) , (2) p(\mathbf{y};\boldsymbol{\theta})=\exp\left(\langle\boldsymbol{\theta},\mathbf{y}\rangle-\psi(\boldsymbol{\theta})\right)p_0(\mathbf{y}), \tag*{(2)} p ( y ; θ ) = exp ( ⟨ θ , y ⟩ − ψ ( θ ) ) p 0 ( y ) , (2) where θ ∈ Θ ⊆ R n \boldsymbol{\theta}\in\Theta\subseteq\mathbb{R}^{n} θ ∈ Θ ⊆ R n is the natural parameter, p 0 p_0 p 0 is a parameter-independent base density with respect to a reference measure μ \mu μ (e.g., Lebesgue or counting measure), and
ψ ( θ ) = log ∫ exp ( ⟨ θ , y ⟩ ) p 0 ( y ) μ ( d y ) \psi(\boldsymbol{\theta})=\log\int\exp\left(\langle\boldsymbol{\theta},\mathbf{y}\rangle\right)p_0(\mathbf{y})\,\mu(\mathrm{d}\mathbf{y}) ψ ( θ ) = log ∫ exp ( ⟨ θ , y ⟩ ) p 0 ( y ) μ ( d y ) is the log-partition function. The natural parameter space is
Θ = dom ψ = { θ ∈ R n : ψ ( θ ) < + ∞ } . \Theta=\operatorname{dom}\psi=\left\{\boldsymbol{\theta}\in\mathbb{R}^{n}:\psi(\boldsymbol{\theta})<+\infty\right\}. Θ = dom ψ = { θ ∈ R n : ψ ( θ ) < + ∞ } . The family is regular when Θ \Theta Θ is open, and minimal when no nonzero linear combination of the components of y \mathbf{y} y is constant almost surely under the family.
Throughout the paper, we assume that the exponential family is regular and minimal. Under these assumptions, the following standard properties hold [4 , 23 ] (Barndorff-Nielsen 2014):
(E1) The log-partition function ψ \psi ψ is closed, proper and convex. Consequently, setting f = ψ ∗ f = \psi^{*} f = ψ ∗ gives f ∗ = ψ f^{*} = \psi f ∗ = ψ .
(E2) The function ψ \psi ψ belongs to C ∞ ( Θ ) C^{\infty}(\Theta) C ∞ ( Θ ) , and
∇ 2 ψ ( θ ) = Cov θ ( y ) ≻ 0 , θ ∈ Θ . \nabla^{2}\psi(\theta) = \operatorname{Cov}_{\theta}(y) \succ0,\qquad\theta\in\Theta. ∇ 2 ψ ( θ ) = Cov θ ( y ) ≻ 0 , θ ∈ Θ.
In particular, ψ \psi ψ is strictly convex on Θ \Theta Θ .
(E3) The function ψ \psi ψ is essentially smooth and therefore of Legendre type. Its gradient is a C ∞ C^{\infty} C ∞ diffeomorphism from Θ \Theta Θ onto X = int ( dom f ) \mathcal{X} = \operatorname{int}(\operatorname{dom} f) X = int ( dom f ) , the interior of the convex support of the sufficient statistic.
The conjugate generator f = ψ ∗ f = \psi^{*} f = ψ ∗ is also of Legendre type and belongs to C ∞ ( X ) C^{\infty}(\mathcal{X}) C ∞ ( X ) , with ∇ 2 f ≻ 0 \nabla^{2}f \succ0 ∇ 2 f ≻ 0 on X \mathcal{X} X . The natural and expectation parameters are related by
x = ∇ ψ ( θ ) = E θ [ y ] , \mathbf{x} = \nabla\psi(\theta) = \mathbb{E}_{\theta}[\mathbf{y}], x = ∇ ψ ( θ ) = E θ [ y ] ,
and conversely,
θ = ∇ f ( x ) . \theta= \nabla f(\mathbf{x}). θ = ∇ f ( x ) .
Thus,
∇ ψ : Θ ↦ X , ∇ f : X ↦ Θ \nabla\psi: \Theta\mapsto\mathcal{X},\qquad\nabla f : \mathcal{X} \mapsto\Theta ∇ ψ : Θ ↦ X , ∇ f : X ↦ Θ
are mutually inverse C ∞ C^{\infty} C ∞ diffeomorphisms. Moreover, for θ = ∇ f ( x ) \theta= \nabla f(\mathbf{x}) θ = ∇ f ( x ) ,
∇ 2 f ( x ) = [ ∇ 2 ψ ( θ ) ] − 1 = [ Cov θ ( y ) ] − 1 . \nabla^{2}f(\mathbf{x}) = \left[\nabla^{2}\psi(\theta)\right]^{-1} = \left[\operatorname{Cov}_{\theta}(\mathbf{y})\right]^{-1}. ∇ 2 f ( x ) = [ ∇ 2 ψ ( θ ) ] − 1 = [ Cov θ ( y ) ] − 1 .
Finally, let ℓ ( y ; x ) \ell(\mathbf{y};\mathbf{x}) ℓ ( y ; x ) denote the log-density log p ( y ; θ ) \log p(\mathbf{y};\theta) log p ( y ; θ ) hereafter expressed in expectation parametrization
ℓ ( y ; x ) = ⟨ ∇ f ( x ) , y ⟩ − ψ ( ∇ f ( x ) ) + log p 0 ( y ) . \ell(\mathbf{y};\mathbf{x}) = \langle\nabla f(\mathbf{x}),\mathbf{y}\rangle- \psi(\nabla f(\mathbf{x})) + \log p_{0}(\mathbf{y}). ℓ ( y ; x ) = ⟨ ∇ f ( x ) , y ⟩ − ψ ( ∇ f ( x )) + log p 0 ( y ) .
This set of properties provides the regularity, strict convexity, and Legendre duality required in the sequel. They also establish the correspondence between the natural and expectation parametrizations, which underlies the Bregman representation of exponential-family distributions recalled next.
Theorem 1. (Banerjee et al., [4 ] ) Let p ( y ; θ ) p(\mathbf{y};\theta) p ( y ; θ ) denote the natural parametrization of the density of the regular exponential family. Under the expectation parametrization, let X ⊆ R n \mathcal{X} \subseteq\mathbb{R}^{n} X ⊆ R n denote the mean-parameter space and let x ∈ X \mathbf{x} \in\mathcal{X} x ∈ X denote the expectation parameter. Then, the log-density ℓ ( y ; x ) \ell(\mathbf{y};\mathbf{x}) ℓ ( y ; x ) is uniquely expressed as
ℓ ( y ; x ) = − d f ( y , x ) + ϕ f ( y ) (3) \ell(\mathbf{y};\mathbf{x}) = -d_{f}(\mathbf{y},\mathbf{x}) + \phi_{f}(\mathbf{y}) \tag*{(3)} ℓ ( y ; x ) = − d f ( y , x ) + ϕ f ( y ) (3) where f : X → R f : \mathcal{X} \to\mathbb{R} f : X → R is the Legendre conjugate of the cumulant function, i.e., f = ψ ∗ f = \psi^{*} f = ψ ∗ , d f d_{f} d f denotes the associated Bregman divergence and the function ϕ f \phi_{f} ϕ f is independent of the parameter x \mathbf{x} x .
This theorem states that the log-density of a regular exponential family distribution can be expressed under the expectation parametrization through a Bregman divergence fully determined by the convex generator f f f . In what follows, this expression will be referred to as the primal Bregman representation of the distribution. For several classical distributions from the regular exponential families, Appendix § 9 reports the Legendre generator f f f and the associated divergence d f d_f d f for their primal Bregman representations, as well as the minimal sufficient statistic y \mathbf{y} y for an observation ω \omega ω and the expectation parameter x \mathbf{x} x .
Dual Bregman representations of regular exponential families
As recalled in Theorem 1, the log-density of a regular exponential-family distribution admits the primal Bregman representation (3) . While this representation provides a natural geometry for the sufficient statistics, it does not directly extend to the expectation-parameter space. Indeed, Bregman divergences are generally asymmetric: although d f ( y , x ) d_f(\mathbf{y},\mathbf{x}) d f ( y , x ) is convex in its first argument, convexity with respect to the second argument is not guaranteed. Consequently, the primal representation does not, in general, yield an ordinary Bregman divergence with the expectation parameter in the first argument. This may cause severe issues when conducting statistical inference of the parameter x ∈ X \mathbf{x} \in\mathcal{X} x ∈ X within a frequentist or Bayesian framework. This observation naturally raises the first central question addressed in this paper.
(Q) Can the reverse Bregman divergence defining a regular exponential family distribution be expressed as a potentially scaled ordinary Bregman divergence in a suitable coordinate system?
Formally, this question can be answered by identifying a strictly convex function h h h defined on a dual domain T \mathcal{T} T together with a diffeomorphism τ : X → T \tau:\mathcal{X}\to\mathcal{T} τ : X → T , such that
d f ( y , x ) = g ( y ) d h ( τ ( x ) , τ ( y ) ) (4) d_f(\mathbf{y},\mathbf{x})=g(\mathbf{y})d_h\bigl(\tau(\mathbf{x}),\tau(\mathbf{y})\bigr) \tag*{(4)} d f ( y , x ) = g ( y ) d h ( τ ( x ) , τ ( y ) ) (4)
where g : X → R + ∗ g:\mathcal{X}\to\mathbb{R}_{+}^{*} g : X → R + ∗ is a positive scaling function that does not depend on x \mathbf{x} x . If such a representation exists, then the regular exponential family distribution allows the geometry of the parameter space to be also characterized by an ordinary Bregman divergence, although expressed in a dual coordinate system. Yet, to appreciate the scope of the objective pursued in this paper, it is worth emphasizing that the terms dual and duality used hereafter are not restricted to the notion of Legendre duality. Rather, Legendre duality will be shown to be recovered as a particular instance within the broader class of dual representations characterized by our approach.
Remark 1 (Connection with conformal divergences). The scaled divergence appearing in the right-hand side of (4) coincides with the τ \tau τ -conformal divergence d h , g τ ( x , y ) d_{h,g}^{\tau}(\mathbf{x},\mathbf{y}) d h , g τ ( x , y ) which generalizes the notion of ordinary Bregman divergence [21 ] . By denoting ξ = ∇ h ∘ τ \boldsymbol{\xi}=\nabla h\circ\tau ξ = ∇ h ∘ τ and using the Bregman-Legendre duality identity (see Proposition 2), this identity can be rewritten as
d f ( y , x ) = g ( y ) d h ∗ ( ∇ h ∘ τ ( y ) , ∇ h ∘ τ ( x ) ) = g ( y ) d h ∗ ( ξ ( y ) , ξ ( x ) ) = d h ∗ , g ξ ( y , x ) . \begin{aligned}
d_f(\mathbf{y},\mathbf{x}) &= g(\mathbf{y})d_{h_*}\bigl(\nabla h\circ\tau(\mathbf{y}),\nabla h\circ\tau(\mathbf{x})\bigr) \\
&= g(\mathbf{y})d_{h_*}\bigl(\boldsymbol{\xi}(\mathbf{y}),\boldsymbol{\xi}(\mathbf{x})\bigr) \\
&= d_{h_*,g}^{\boldsymbol{\xi}}(\mathbf{y},\mathbf{x}).
\end{aligned} d f ( y , x ) = g ( y ) d h ∗ ( ∇ h ∘ τ ( y ) , ∇ h ∘ τ ( x ) ) = g ( y ) d h ∗ ( ξ ( y ) , ξ ( x ) ) = d h ∗ , g ξ ( y , x ) .
In other words, looking for the strictly convex function h h h and the diffeomorphism τ \tau τ such that the identity (4) holds amounts to identifying a ( τ , ξ ) (\tau,\xi) ( τ , ξ ) -geometric structure underlying conformal counterparts of the primal Bregman representation d f ( y , x ) d_f(y,x) d f ( y , x ) [2 ] .
We now turn to the projective characterization of dual Bregman representations, which provides the theoretical foundation for the constructive results developed in the sequel.
Projective characterization of dual Bregman representations
The following theorem gives necessary and sufficient conditions for a reverse Bregman divergence to admit a scaled ordinary Bregman divergence in a suitable coordinate system.
Theorem 2. Let X , T ⊂ R n \mathcal{X},\mathcal{T} \subset\mathbb{R}^{n} X , T ⊂ R n be nonempty open convex sets. Suppose that f ∈ C 2 ( X ) f \in C^{2}(\mathcal{X}) f ∈ C 2 ( X ) and h ∈ C 2 ( T ) h \in C^{2}(\mathcal{T}) h ∈ C 2 ( T ) are strictly convex, τ : X → T \tau:\mathcal{X}\to\mathcal{T} τ : X → T is a C 1 C^{1} C 1 diffeomorphism, and g : X → ( 0 , ∞ ) g:\mathcal{X}\to(0,\infty) g : X → ( 0 , ∞ ) is a positive function, not assumed differentiable. Write ξ = ∇ h ∘ τ \xi=\nabla h\circ\tau ξ = ∇ h ∘ τ . Then the identity
d f ( y , x ) = g ( y ) d h ( τ ( x ) , τ ( y ) ) (5) d_f(y,x)=g(y)d_h\left(\tau(x),\tau(y)\right) \tag*{(5)} d f ( y , x ) = g ( y ) d h ( τ ( x ) , τ ( y ) ) (5) holds for all x , y ∈ X x,y\in\mathcal{X} x , y ∈ X if and only if there exist A ∈ R n × n A\in\mathbb{R}^{n\times n} A ∈ R n × n , b , c , ζ ∈ R n \mathbf{b},\mathbf{c},\boldsymbol{\zeta}\in\mathbb{R}^{n} b , c , ζ ∈ R n and d , κ ∈ R d,\kappa\in\mathbb{R} d , κ ∈ R such that
g ( x ) = c T x + d > 0 , (6) g(x)=\mathbf{c}^{\mathsf{T}}x+d>0, \tag*{(6)} g ( x ) = c T x + d > 0 , (6) ξ ( x ) = A x + b g ( x ) , M = ( A b c T d ) , det M ≠ 0 , f ( x ) = g ( x ) h ∗ ( ξ ( x ) ) + ζ T x + κ . (7) \begin{aligned}
\xi(x)&=\frac{A x+\mathbf{b}}{g(x)}, \qquad M=\begin{pmatrix} A & \mathbf{b} \\ \mathbf{c}^{\mathsf{T}} & d \end{pmatrix}, \qquad\det M\ne0,\\
f(x)&=g(x)h^{*}(\xi(x))+\boldsymbol{\zeta}^{\mathsf{T}}x+\kappa. \tag*{(7)}
\end{aligned} ξ ( x ) f ( x ) = g ( x ) A x + b , M = ( A c T b d ) , det M = 0 , = g ( x ) h ∗ ( ξ ( x )) + ζ T x + κ . (7) Moreover, whenever these equivalent conditions hold, the Hessians satisfy
∇ 2 f ( x ) = g ( x ) J τ ( x ) T ∇ 2 h ( τ ( x ) ) J τ ( x ) . \nabla^{2}f(x)=g(x)J_{\tau}(x)^{\mathsf{T}}\nabla^{2}h(\tau(x))J_{\tau}(x). ∇ 2 f ( x ) = g ( x ) J τ ( x ) T ∇ 2 h ( τ ( x )) J τ ( x ) .
Proof. See Appendix A.1
The weighting function g g g fixes the last row ( c T , d ) (\mathbf{c}^{\mathsf{T}},d) ( c T , d ) of M M M . For a given weight g g g , the choice of the remaining rows ( A , b ) (A,\mathbf{b}) ( A , b ) of M M M will be referred to as the gauge in the sequel.
The projective characterization determines the admissible representations up to certain changes of coordinates and affine terms. Before deriving explicit reconstruction formulas, we clarify this gauge freedom and adopt a convenient normalization. The following corollary also establishes the corresponding uniqueness property.
Corollary 3 (Normalization and uniqueness). Assume that the hypotheses of Theorem 2 hold. Let ( h 0 , τ 0 ) (h_0,\tau_0) ( h 0 , τ 0 ) be an admissible representation associated with the affine weight
g ( x ) = c T x + d > 0. g(\mathbf{x}) = \mathbf{c}^{\mathsf{T}}\mathbf{x} + d > 0. g ( x ) = c T x + d > 0. Then the following properties hold.
i) Normalization. There exist a ∈ R n \mathbf{a} \in\mathbb{R}^{n} a ∈ R n and κ 1 ∈ R \kappa_1 \in\mathbb{R} κ 1 ∈ R such that
τ ~ ( x ) = τ 0 ( x ) + a , h ~ ( t ) = h 0 ( t − a ) − κ 1 \widetilde{\tau}(\mathbf{x}) = \tau_0(\mathbf{x}) + \mathbf{a}, \qquad\widetilde{h}(\mathbf{t}) = h_0(\mathbf{t} - \mathbf{a}) - \kappa_1 τ ( x ) = τ 0 ( x ) + a , h ( t ) = h 0 ( t − a ) − κ 1 define an admissible representation with the same projective matrix M \mathbf{M} M , the same map ξ \xi ξ , and with ζ = 0 \zeta= 0 ζ = 0 and κ = 0 \kappa= 0 κ = 0 in (7) . The associated Bregman divergence is unchanged.
ii) Uniqueness up to affine equivalence. If ( h , τ ) (h,\tau) ( h , τ ) is another admissible representation associated with the same weight g g g , then there exist an invertible matrix G ∈ R n × n \mathbf{G} \in\mathbb{R}^{n \times n} G ∈ R n × n , vectors w , v ∈ R n \mathbf{w},\mathbf{v} \in\mathbb{R}^{n} w , v ∈ R n , and a scalar κ 0 ∈ R \kappa_0 \in\mathbb{R} κ 0 ∈ R such that
τ ( x ) = G − T τ 0 ( x ) + v \tau(\mathbf{x}) = \mathbf{G}^{-\mathsf{T}}\tau_0(\mathbf{x}) + \mathbf{v} τ ( x ) = G − T τ 0 ( x ) + v and
h ( t ) = h 0 ( G T ( t − v ) ) + ⟨ w , t − v ⟩ + κ 0 . h(\mathbf{t}) = h_0\left(\mathbf{G}^{\mathsf{T}}(\mathbf{t} - \mathbf{v})\right) + \langle\mathbf{w},\mathbf{t} - \mathbf{v}\rangle+ \kappa_0. h ( t ) = h 0 ( G T ( t − v ) ) + ⟨ w , t − v ⟩ + κ 0 . Conversely, every pair of this form defines an admissible representation with the same weight g g g . In particular,
d h ( τ ( x ) , τ ( y ) ) = d h 0 ( τ 0 ( x ) , τ 0 ( y ) ) . d_h(\tau(\mathbf{x}),\tau(\mathbf{y})) = d_{h_0}(\tau_0(\mathbf{x}),\tau_0(\mathbf{y})). d h ( τ ( x ) , τ ( y )) = d h 0 ( τ 0 ( x ) , τ 0 ( y )) .
Proof. See Appendix § 8 .§ 2 .
Exploiting the normalization property stated by Theorem 3 i), we may and do assume throughout the remainder of this section that ζ = 0 \zeta= 0 ζ = 0 and κ = 0 \kappa= 0 κ = 0 while defining f f f in (7) .
The projective characterization also provides a way to explicitly recover the dual coordinates and their associated Bregman generator from the projective coefficients, as established in the following proposition.
Proposition 4 (Block-matrix reconstruction). Assume that the hypotheses and conclusions of the projective characterization established by Theorem 2 hold. In particular, let
g ( x ) = c T x + d > 0 , ξ ( x ) = ∇ h ( τ ( x ) ) = A x + b g ( x ) , M = ( A b c T d ) (8) g(\mathbf{x}) = \mathbf{c}^{\mathsf{T}}\mathbf{x} + d > 0,\qquad\xi(\mathbf{x}) = \nabla h(\tau(\mathbf{x})) = \frac{\mathbf{A}\mathbf{x} + \mathbf{b}}{g(\mathbf{x})},\qquad\mathbf{M} = \begin{pmatrix} \mathbf{A} & \mathbf{b} \\ \mathbf{c}^{\mathsf{T}} & d \end{pmatrix} \tag*{(8)} g ( x ) = c T x + d > 0 , ξ ( x ) = ∇ h ( τ ( x )) = g ( x ) Ax + b , M = ( A c T b d ) (8) such that det M ≠ 0 \det\mathbf{M} \ne0 det M = 0 . Then
( τ ( x ) − h ( τ ( x ) ) ) = M − T ( ∇ f ( x ) − F ( x ) ) (9) \begin{pmatrix} \tau(\mathbf{x}) \\ -h(\tau(\mathbf{x})) \end{pmatrix} = \mathbf{M}^{-\mathsf{T}}\begin{pmatrix} \nabla f(\mathbf{x}) \\ -F(\mathbf{x}) \tag*{(9)} \end{pmatrix} ( τ ( x ) − h ( τ ( x )) ) = M − T ( ∇ f ( x ) − F ( x ) ) (9) with F ( x ) = f ∗ ( ∇ f ( x ) ) F(\mathbf{x}) = f^*(\nabla f(\mathbf{x})) F ( x ) = f ∗ ( ∇ f ( x )) . Equivalently, when
M − T = ( P q r T s ) , \mathbf{M}^{-\mathsf{T}} = \begin{pmatrix} \mathbf{P} & \mathbf{q} \\ \mathbf{r}^{\mathsf{T}} & s \end{pmatrix}, M − T = ( P r T q s ) , then
τ ( x ) = P ∇ f ( x ) − q F ( x ) and h ( τ ( x ) ) = − r T ∇ f ( x ) + s F ( x ) . (10) \tau(\mathbf{x}) = \mathbf{P}\nabla f(\mathbf{x}) - \mathbf{q}F(\mathbf{x}) \quad\text{and} \quad h(\tau(\mathbf{x})) = -\mathbf{r}^{\mathsf{T}}\nabla f(\mathbf{x}) + sF(\mathbf{x}). \tag*{(10)} τ ( x ) = P ∇ f ( x ) − q F ( x ) and h ( τ ( x )) = − r T ∇ f ( x ) + s F ( x ) . (10)
Proof. See Appendix A.3.
It is worth noting that the reconstruction formulas stated in Proposition 4 require the hypotheses of Theorem 2 to be satisfied, thereby presupposing the existence of an admissible dual Bregman representation. Going beyond these results, the following theorem establishes both the existence and uniqueness of such representations when f f f is the expectation-parameter generator of a regular minimal exponential family.
Theorem 4 (Existence and uniqueness of dual Bregman representations). Let f = ψ ∗ f = \psi^{*} f = ψ ∗ be the expectation-parameter generator of a regular minimal exponential family, where ψ \psi ψ denotes its log-partition function, and let X = int ( dom f ) \mathcal{X} = \operatorname{int}(\operatorname{dom} f) X = int ( dom f ) . Let M \mathbf{M} M be an invertible block matrix of the form introduced in (8) , and let
g ( x ) = c T x + d > 0 , ξ ( x ) = A x + b g ( x ) , x ∈ X . g(\mathbf{x}) = \mathbf{c}^{\mathsf{T}}\mathbf{x} + d > 0, \qquad\xi(\mathbf{x}) = \frac{\mathbf{A}\mathbf{x} + \mathbf{b}}{g(\mathbf{x})}, \qquad\mathbf{x} \in\mathcal{X}. g ( x ) = c T x + d > 0 , ξ ( x ) = g ( x ) Ax + b , x ∈ X . Then there exists a unique pair ( h , τ ) (h,\tau) ( h , τ ) satisfying the conditions of Theorem 2 with this M \mathbf{M} M , ζ = 0 \zeta= 0 ζ = 0 and κ = 0 \kappa= 0 κ = 0 , where τ \tau τ is a C ∞ C^{\infty} C ∞ diffeomorphism from X \mathcal{X} X onto a nonempty open convex set T \mathcal{T} T , and h h h is a closed convex function of Legendre type with dom h = T \operatorname{dom} h = \mathcal{T} dom h = T , C ∞ C^{\infty} C ∞ on T \mathcal{T} T with ∇ 2 h ≻ 0 \nabla^{2}h \succ0 ∇ 2 h ≻ 0 . In particular,
∇ h ∘ τ = ξ \nabla h \circ\tau= \xi ∇ h ∘ τ = ξ and, for x , y ∈ X \mathbf{x}, \mathbf{y} \in\mathcal{X} x , y ∈ X ,
d f ( y , x ) = g ( y ) d h ( τ ( x ) , τ ( y ) ) . (11) d_f(\mathbf{y},\mathbf{x}) = g(\mathbf{y})d_h\bigl(\tau(\mathbf{x}),\tau(\mathbf{y})\bigr). \tag*{(11)} d f ( y , x ) = g ( y ) d h ( τ ( x ) , τ ( y ) ) . (11) Proof. See Appendix A.4.
The following corollary provides explicit formulas for the dual representation established in Theorem 4.
Corollary 5 (Explicit construction). Let ( h , τ ) (h,\tau) ( h , τ ) be the pair exhibited in Theorem 4. Then, the map τ \tau τ and h ∘ τ h \circ\tau h ∘ τ are explicitly given by (10) with
h ( t ) = sup x ∈ X { ⟨ ξ ( x ) , t ⟩ − f ( x ) g ( x ) } , (12) h(\mathbf{t}) = \sup_{\mathbf{x}\in\mathcal{X}} \left\{\langle\xi(\mathbf{x}),\mathbf{t}\rangle- \frac{f(\mathbf{x})}{g(\mathbf{x})}\right\}, \tag*{(12)} h ( t ) = x ∈ X sup { ⟨ ξ ( x ) , t ⟩ − g ( x ) f ( x ) } , (12) Moreover, we have int ( dom h ∗ ) = ξ ( X ) \operatorname{int}(\operatorname{dom} h^{*}) = \xi(\mathcal{X}) int ( dom h ∗ ) = ξ ( X ) on which h ∗ h^{*} h ∗ is given by
h ∗ ( u ) = ( q T u + s ) f ( P T u + r q T u + s ) (13) h^{*}(\mathbf{u}) = \bigl(\mathbf{q}^{\mathsf{T}}\mathbf{u} + s\bigr)f\left(\frac{\mathbf{P}^{\mathsf{T}}\mathbf{u} + \mathbf{r}}{\mathbf{q}^{\mathsf{T}}\mathbf{u} + s}\right) \tag*{(13)} h ∗ ( u ) = ( q T u + s ) f ( q T u + s P T u + r ) (13) and
∇ h ∗ = τ ∘ ξ − 1 . \nabla h^{*} = \tau\circ\xi^{-1}. ∇ h ∗ = τ ∘ ξ − 1 . Finally, for every t ∈ T \mathbf{t} \in\mathcal{T} t ∈ T , the value σ = h ( t ) \sigma= h(\mathbf{t}) σ = h ( t ) is the unique solution of
ψ ( A T t − c σ ) = d σ − b T t . (14) \psi(\mathbf{A}^{\mathsf{T}}\mathbf{t} - \mathbf{c}\sigma) = d\sigma- \mathbf{b}^{\mathsf{T}}\mathbf{t}. \tag*{(14)} ψ ( A T t − c σ ) = d σ − b T t . (14) Proof. See Appendix A.5.
Therefore, Theorem 4 provides an affirmative answer to the question (Q). Indeed, it establishes the existence of dual Bregman representations in transformed coordinates for every invertible projective matrix whose associated affine scaling function is positive on the mean-parameter space. Moreover, the result goes beyond the requirements of question (Q), since the constructed Bregman generator h h h is not merely convex but Legendre. Besides, it appears that the regularity assumptions on the exponential family play an important role in ensuring the global properties of the constructed representation. The following remark clarifies their scope.
Remark 3 (Role of the assumptions). The proof of Theorem 4 relies on the fact that f f f is closed, proper, of Legendre type, and C 2 C^2 C 2 with ∇ 2 f ≻ 0 \nabla^2 f \succ0 ∇ 2 f ≻ 0 on X \mathcal{X} X , together with the openness of dom f ∗ \operatorname{dom} f^* dom f ∗ . Under these assumptions, the construction remains valid with C 1 C^1 C 1 regularity for τ \tau τ and C 2 C^2 C 2 regularity for h h h , without requiring f f f to be C ∞ C^\infty C ∞ .
The openness of the natural parameter space Θ = dom f ∗ \Theta= \operatorname{dom} f^* Θ = dom f ∗ , which is the regularity assumption for the exponential family, is essential in the proof of the global properties: the convexity of T \mathcal{T} T , the implicit normalization identity (14) , and the Legendre property of h h h .
Without regularity, τ \tau τ remains a C 1 C^1 C 1 diffeomorphism onto the open set τ ( X ) \tau(\mathcal{X}) τ ( X ) , and h h h is convex on R n \mathbb{R}^n R n with ∇ 2 h ≻ 0 \nabla^2 h \succ0 ∇ 2 h ≻ 0 on τ ( X ) \tau(\mathcal{X}) τ ( X ) . The identity (11) still holds, but τ ( X ) \tau(\mathcal{X}) τ ( X ) need not be convex when n ≥ 2 n \ge2 n ≥ 2 .
The equality τ ( X ) = dom h \tau(\mathcal{X}) = \operatorname{dom} h τ ( X ) = dom h and the Legendre property can indeed fail, even for n = 1 n = 1 n = 1 , as shown in Appendix C.8 which discusses the case of the non-regular inverse Gaussian family with known shape parameter (Barndorff-Nielsen, 2014, Ch. 8).
The following corollary now translates the results established above when deriving the log-density of regular exponential family.
Corollary 6 (Dual representations of regular exponential family density). Let f = ψ ∗ f = \psi^* f = ψ ∗ be the expectation-parameter generator of a regular minimal exponential family, where ψ \psi ψ denotes its log-partition function. Let Y \mathcal{Y} Y denote the support of the sufficient statistic, and let p 0 p_0 p 0 be the base density of the exponential family. Let ( h , τ ) (h,\tau) ( h , τ ) be the pair exhibited in Theorem 4 and associated with the block matrix
M = ( A b c T d ) . \mathbf{M} = \begin{pmatrix} \mathbf{A} & \mathbf{b} \\ \mathbf{c}^{\mathsf{T}} & d \end{pmatrix}. M = ( A c T b d ) . For every x ∈ X \mathbf{x} \in\mathcal{X} x ∈ X and y ∈ Y y \in\mathcal{Y} y ∈ Y , the log-density admits the dual exponential representation
ℓ ( y ; x ) = ⟨ τ ( x ) , A y + b ⟩ − g ( y ) h ( τ ( x ) ) + log p 0 ( y ) . (15) \ell(y;\mathbf{x}) = \langle\tau(\mathbf{x}), \mathbf{A}y + \mathbf{b} \rangle- g(y)h(\tau(\mathbf{x})) + \log p_0(y). \tag*{(15)} ℓ ( y ; x ) = ⟨ τ ( x ) , A y + b ⟩ − g ( y ) h ( τ ( x )) + log p 0 ( y ) . (15) Moreover, Eq. (14) evaluated at t = τ ( x ) t = \tau(\mathbf{x}) t = τ ( x ) coincides with the normalization condition of this dual parametrization. Finally, for y ∈ X y \in\mathcal{X} y ∈ X , the log-density also admits the dual Bregman representation
ℓ ( y ; x ) = − g ( y ) d h ( τ ( x ) , τ ( y ) ) + ϕ f ( y ) . (16) \ell(y;\mathbf{x}) = -g(y)d_h(\tau(\mathbf{x}),\tau(y)) + \phi_f(y). \tag*{(16)} ℓ ( y ; x ) = − g ( y ) d h ( τ ( x ) , τ ( y )) + ϕ f ( y ) . (16) Proof. See Appendix A.6.
Remark 4 (Exponential vs. Bregman representations). Unlike the dual exponential representation, the dual Bregman representation requires y ∈ X y \in\mathcal{X} y ∈ X . Observations on the boundary of the mean-parameter space, as may occur for discrete families, are covered by the dual exponential representation.
Remark 5 (Graphs of ψ \psi ψ and h h h ). The construction admits a geometric interpretation in terms of graphs. Define D = diag ( I , − 1 ) \mathbf{D} = \operatorname{diag}(\mathbf{I}, -1) D = diag ( I , − 1 ) and, for x ∈ X \mathbf{x} \in\mathcal{X} x ∈ X , θ = ∇ f ( x ) ∈ Θ \boldsymbol{\theta} = \nabla f(\mathbf{x}) \in\Theta θ = ∇ f ( x ) ∈ Θ and t = τ ( x ) ∈ T \mathbf{t} = \tau(\mathbf{x}) \in\mathcal{T} t = τ ( x ) ∈ T , so that ψ ( θ ) = f ∗ ( ∇ f ( x ) ) \psi(\boldsymbol{\theta}) = f^*(\nabla f(\mathbf{x})) ψ ( θ ) = f ∗ ( ∇ f ( x )) . Since D 2 = I \mathbf{D}^2 = \mathbf{I} D 2 = I , the identity (9) in Proposition 4 reads
\begin{pmatrix} \mathbf{t} \\ h(\mathbf{t}) \end{pmatrix}
= \mathbf{D}\mathbf{M}^{-\mathsf{T}}\mathbf{D}
\begin{pmatrix} \boldsymbol{\theta} \\ \psi(\boldsymbol{\theta}) \end{pmatrix}. \tag*{(17)}\end{pmatrix}
As x \mathbf{x} x ranges over X \mathcal{X} X , θ \boldsymbol{\theta} θ ranges over Θ \Theta Θ (Rockafellar, [23 ] ) and t \mathbf{t} t over T \mathcal{T} T . Thus the graph of h h h on T \mathcal{T} T is the image of the graph of ψ \psi ψ on Θ \Theta Θ under the invertible linear map D M − T D \mathbf{D}\mathbf{M}^{-\mathsf{T}}\mathbf{D} D M − T D .
The log-density pairs these graph points with the statistics D ( y ; 1 ) = ( y ; − 1 ) \mathbf{D}(y;1)=(y;-1) D ( y ; 1 ) = ( y ; − 1 ) . For y ∈ Y y\in\mathcal{Y} y ∈ Y , the log-density (2) expressed in natural parameter can be rewritten as
log p ( y ; θ ) = ⟨ ( θ ψ ( θ ) ) , ( y − 1 ) ⟩ + log p 0 ( y ) . \log p(y;\boldsymbol{\theta})=\left\langle\begin{pmatrix}\boldsymbol{\theta}\\ \psi(\boldsymbol{\theta})\end{pmatrix},\begin{pmatrix}y\\ -1\end{pmatrix}\right\rangle+\log p_0(y). log p ( y ; θ ) = ⟨ ( θ ψ ( θ ) ) , ( y − 1 ) ⟩ + log p 0 ( y ) .
From (15) with t = τ ( x ) \mathbf{t}=\tau(\mathbf{x}) t = τ ( x ) , this log-density also admits the dual exponential representation
log p ( y ; θ ) = ⟨ ( t h ( t ) ) , D M D ( y − 1 ) ⟩ + log p 0 ( y ) \log p(y;\boldsymbol{\theta})=\left\langle\begin{pmatrix}\mathbf{t}\\ h(\mathbf{t})\end{pmatrix},\mathbf{DMD}\begin{pmatrix}y\\ -1\end{pmatrix}\right\rangle+\log p_0(y) log p ( y ; θ ) = ⟨ ( t h ( t ) ) , DMD ( y − 1 ) ⟩ + log p 0 ( y )
Graph points and statistics are thus transformed contragrediently, by D M − ⊤ D \mathbf{DM}^{-\top}\mathbf{D} DM − ⊤ D and D M D \mathbf{DMD} DMD , which preserves the log-likelihood.
The preceding results establish a general framework for constructing dual Bregman representations of regular exponential families. We now consider two particularly simple choices of the projective coefficients, leading to the so-called β \beta β -scaled and α \alpha α -scaled representations. These constructions will also provide the reference geometries used later to study conjugate priors and relative smoothness.
Corollary 7 (β \beta β - and α \alpha α -scaled dual models). Let f = ψ ∗ f=\psi^* f = ψ ∗ be the expectation-parameter generator of a regular minimal exponential family, where ψ \psi ψ denotes its log-partition function, and let X = int ( dom f ) \mathcal{X}=\operatorname{int}(\operatorname{dom} f) X = int ( dom f ) . Let α , β > 0 \alpha,\beta>0 α , β > 0 . The following are two particular instances of the dual Bregman representation (11) .
i) β \beta β -scaled dual model. In arbitrary dimension, the constant weight
g ( y ) = 1 β g(y)=\frac{1}{\beta} g ( y ) = β 1 gives
τ = ∇ f , T = int ( dom f ∗ ) , h = β f ∗ . \tau=\nabla f,\qquad\mathcal{T}=\operatorname{int}(\operatorname{dom} f^*),\qquad h=\beta f^*. τ = ∇ f , T = int ( dom f ∗ ) , h = β f ∗ . Consequently, for x , y ∈ X \mathbf{x},\mathbf{y}\in\mathcal{X} x , y ∈ X ,
ℓ ( y ; x ) = − 1 β d h ( τ ( x ) , τ ( y ) ) + ϕ f ( y ) . \ell(\mathbf{y};\mathbf{x})=-\frac{1}{\beta}d_h(\tau(\mathbf{x}),\tau(\mathbf{y}))+\phi_f(\mathbf{y}). ℓ ( y ; x ) = − β 1 d h ( τ ( x ) , τ ( y )) + ϕ f ( y ) . ii) α \alpha α -scaled dual model. Suppose that n = 1 n=1 n = 1 and X ⊆ ( 0 , ∞ ) \mathcal{X}\subseteq(0,\infty) X ⊆ ( 0 , ∞ ) . The weight
g ( y ) = y α g(y)=\frac{y}{\alpha} g ( y ) = α y gives
τ ( x ) = x f ′ ( x ) − f ( x ) = f ∗ ( f ′ ( x ) ) , T = f ∗ ( int ( dom f ∗ ) ) , \tau(x)=xf'(x)-f(x)=f^*(f'(x)),\qquad\mathcal{T}=f^*(\operatorname{int}(\operatorname{dom} f^*)), τ ( x ) = x f ′ ( x ) − f ( x ) = f ∗ ( f ′ ( x )) , T = f ∗ ( int ( dom f ∗ )) , and
h ( t ) = − α [ f ∗ ] − 1 ( t ) , t ∈ T . h(t)=-\alpha[f^*]^{-1}(t),\qquad t\in\mathcal{T}. h ( t ) = − α [ f ∗ ] − 1 ( t ) , t ∈ T . Consequently, for x , y ∈ X x,y\in\mathcal{X} x , y ∈ X ,
ℓ ( y ; x ) = − y α d h ( τ ( x ) , τ ( y ) ) + ϕ f ( y ) . \ell(y;x)=-\frac{y}{\alpha}d_h(\tau(x),\tau(y))+\phi_f(y). ℓ ( y ; x ) = − α y d h ( τ ( x ) , τ ( y )) + ϕ f ( y ) . In the first case, τ \tau τ is strictly monotone, whereas in the second case, τ \tau τ is strictly increasing. Both dual generators are of Legendre type under the assumptions of Theorem 4.
Proof. See Appendix § 8 .7.
The two scaled models illustrate how the projective construction yields different, yet equivalent, dual Bregman representations. This corollary also shows that this construction encompasses the classical Legendre duality as a particular instance of the β \beta β -scaled dual representation with β = 1 \beta=1 β = 1 . We next investigate the implications of these representations while expressing conjugate prior distributions within a Bayesian framework.
Dual representations of conjugate priors
When conducting parameter inference in a statistical model induced by a distribution belonging to the exponential family, Theorem 1 shows that maximizing the likelihood function is equivalent to minimizing the Bregman divergence generated by f f f . Interestingly, this primal Bregman representation also proves particularly useful to derive conjugate prior distributions widely adopted in Bayesian inference. Indeed, suppose now that the prior information on the parameter x \mathbf{x} x is governed by a positive precision parameter β > 0 \beta> 0 β > 0 together with a location parameter α / β ∈ X \alpha/\beta\in\mathcal{X} α / β ∈ X . For a given likelihood function, a prior distribution p ( x ∣ α , β ) p(\mathbf{x} \mid\alpha,\beta) p ( x ∣ α , β ) is said to be conjugate when it preserves the analytical form of the posterior distribution after observing1 y \mathbf{y} y . It appears that this conjugacy systematically emerges when resorting to a primal Bregman representation of the prior distribution, i.e., by considering a log-prior on expectation coordinates of the form (Agarwal and Daumé III, [1 ] )
log p ( x ∣ α , β ) = − β d f ( α β , x ) + φ f ( α , β ) (18) \log p(\mathbf{x}\mid\alpha,\beta)=-\beta\,d_f\left(\frac{\alpha}{\beta},\mathbf{x}\right)+\varphi_f(\alpha,\beta) \tag*{(18)} log p ( x ∣ α , β ) = − β d f ( β α , x ) + φ f ( α , β ) (18)
where φ f \varphi_f φ f denotes a normalization potential independent of x \mathbf{x} x . Throughout this section, prior and posterior densities in expectation coordinates are understood with respect to Lebesgue measure. The hyperparameters are assumed to be such that the prior density is normalizable. The conjugate nature of such prior models is confirmed by the following proposition.
Proposition 5. Consider the Bayesian model defined by the log-likelihood
ℓ ( y ; x ) = − d f ( y , x ) + ϕ f ( y ) \ell(\mathbf{y};\mathbf{x})=-d_f(\mathbf{y},\mathbf{x})+\phi_f(\mathbf{y}) ℓ ( y ; x ) = − d f ( y , x ) + ϕ f ( y ) and the Bregman representation of its conjugate prior
log p ( x ∣ α , β ) = − β d f ( α β , x ) + φ f ( α , β ) . \log p(\mathbf{x}\mid\alpha,\beta)=-\beta\,d_f\left(\frac{\alpha}{\beta},\mathbf{x}\right)+\varphi_f(\alpha,\beta). log p ( x ∣ α , β ) = − β d f ( β α , x ) + φ f ( α , β ) . Then the log-posterior can be written as
log p ( x ∣ y , α , β ) = − ( 1 + β ) d f ( y + α 1 + β , x ) + Ψ f ( y , α , β ) (19) \log p(\mathbf{x}\mid\mathbf{y},\alpha,\beta)=-(1+\beta)d_f\left(\frac{\mathbf{y}+\alpha}{1+\beta},\mathbf{x}\right)+\Psi_f(\mathbf{y},\alpha,\beta) \tag*{(19)} log p ( x ∣ y , α , β ) = − ( 1 + β ) d f ( 1 + β y + α , x ) + Ψ f ( y , α , β ) (19) where Ψ f ( y , α , β ) \Psi_f(\mathbf{y},\alpha,\beta) Ψ f ( y , α , β ) is independent of x \mathbf{x} x .
Proof. See Appendix § 8 .8.
This property shows that the posterior (19) retains the same parametric form (18) as the prior with updated hyperparameters
α + = y + α , β + = 1 + β , \alpha^{+}=\mathbf{y}+\alpha,\qquad\beta^{+}=1+\beta, α + = y + α , β + = 1 + β ,
which is the defining property of conjugacy in the present parametrization.
The dual Bregman representations of regular exponential family distributions identified in § 3.1 are particularly useful for deriving dual representations of these conjugate priors, as alternatives to the primal Bregman representation (18) . Indeed, after explicitly introducing the location parameter
z = α β (20) \mathbf{z}=\frac{\alpha}{\beta} \tag*{(20)} z = β α (20)
the primal Bregman representation (18) exactly matches the forms inducing α \alpha α - and β \beta β -scaled dual models identified in § 3.1 , leading to the two following corollaries.
Corollary 8. Let f = ψ ∗ f=\psi^* f = ψ ∗ be the expectation-parameter generator of a regular minimal exponential family, where ψ \psi ψ denotes its log-partition function, and let X = int ( dom f ) \mathcal{X}=\operatorname{int}(\operatorname{dom} f) X = int ( dom f ) , T = int ( dom f ∗ ) \mathcal{T}=\operatorname{int}(\operatorname{dom} f^*) T = int ( dom f ∗ ) . For any ( x , z ) ∈ X 2 (x,z)\in\mathcal{X}^2 ( x , z ) ∈ X 2 and β > 0 \beta>0 β > 0 , consider the Bregman representation of the log-likelihood associated with a regular exponential family distribution
ℓ ( y ; x ) = − d f ( y , x ) + ϕ f ( y ) \ell(\mathbf{y};\mathbf{x})=-d_f(\mathbf{y},\mathbf{x})+\phi_f(\mathbf{y}) ℓ ( y ; x ) = − d f ( y , x ) + ϕ f ( y ) and the Bregman representation of its conjugate prior
log p ( x ∣ z , β ) = − β d f ( z , x ) + φ f ( β z , β ) . (21) \log p(\mathbf{x}\mid\mathbf{z},\beta)=-\beta d_f(\mathbf{z},\mathbf{x})+\varphi_f(\beta\mathbf{z},\beta). \tag*{(21)} log p ( x ∣ z , β ) = − β d f ( z , x ) + φ f ( β z , β ) . (21) Then, this conjugate prior admits the equivalent dual Bregman representation
log p ( x ∣ z , β ) = − d h ( τ ( x ) , τ ( z ) ) + φ f ( β z , β ) (22) \log p(\mathbf{x}\mid\mathbf{z},\beta)=-d_h(\tau(\mathbf{x}),\tau(\mathbf{z}))+\varphi_f(\beta\mathbf{z},\beta) \tag*{(22)} log p ( x ∣ z , β ) = − d h ( τ ( x ) , τ ( z )) + φ f ( β z , β ) (22) with
τ = ∇ f : X → T and h = β f ∗ . \tau=\nabla f:\mathcal{X}\to\mathcal{T}\qquad\text{and}\qquad h=\beta f^*. τ = ∇ f : X → T and h = β f ∗ . Moreover the resulting log-posterior can be written as
log p ( x ∣ y , z , β ) = − 1 + β β d h ( τ ( x ) , τ ( y + β z 1 + β ) ) + Ψ h ( y , z , β ) (23) \log p(\mathbf{x}\mid\mathbf{y},\mathbf{z},\beta)=-\frac{1+\beta}{\beta}d_h\left(\tau(\mathbf{x}),\tau\left(\frac{\mathbf{y}+\beta\mathbf{z}}{1+\beta}\right)\right)+\Psi_h(\mathbf{y},\mathbf{z},\beta) \tag*{(23)} log p ( x ∣ y , z , β ) = − β 1 + β d h ( τ ( x ) , τ ( 1 + β y + β z ) ) + Ψ h ( y , z , β ) (23) where Ψ h ( y , z , β ) \Psi_h(\mathbf{y},\mathbf{z},\beta) Ψ h ( y , z , β ) is independent of x \mathbf{x} x .
Proof. The existence of the dual representation (22) of the prior follows directly from applying Theorem 7 to the conjugate prior (21) . Similarly, the expression of the log-posterior distribution (23) follows from applying Theorem 7 to the log-posterior (19) derived in Proposition 5. □
It is worth noting that Theorem 8 does not introduce a new family of conjugate priors associated with likelihoods derived from the regular exponential family. Rather, it provides a systematic way to derive Bregman representations of those priors, as an alternative of the representation already identified by Agarwal and Daumé III [1 ] . Equation (23) shows that the posterior retains the same parametric form as the dual formulation (22) of the prior with updated hyperparameters τ ( z + ) \tau(\mathbf{z}^+) τ ( z + ) and β + / β \beta^+/\beta β + / β where z + = α + / β + \mathbf{z}^+=\boldsymbol{\alpha}^+/\beta^+ z + = α + / β + . Hence, Theorem 8 shows that the conjugacy property also emerges from the dual representations of the likelihood and the prior. Similar properties can also be exhibited when considering a conjugate prior of the form of an α \alpha α -scaled model.
Corollary 9. Let f = ψ ∗ f=\psi^* f = ψ ∗ be the expectation-parameter generator of a regular minimal exponential family, where ψ \psi ψ denotes its log-partition function, and let X = int ( dom f ) ⊆ R + \mathcal{X}=\operatorname{int}(\operatorname{dom} f)\subseteq\mathbb{R}_+ X = int ( dom f ) ⊆ R + , T = f ∗ ( int ( dom f ∗ ) ) \mathcal{T}=f^*(\operatorname{int}(\operatorname{dom} f^*)) T = f ∗ ( int ( dom f ∗ )) . For any ( x , z ) ∈ X 2 (x,z)\in\mathcal{X}^2 ( x , z ) ∈ X 2 and α > 0 \alpha>0 α > 0 , consider the Bregman representation of the log-likelihood associated with a regular exponential family distribution
ℓ ( y ; x ) = − d f ( y , x ) + ϕ f ( y ) \ell(\mathbf{y};\mathbf{x})=-d_f(\mathbf{y},\mathbf{x})+\phi_f(\mathbf{y}) ℓ ( y ; x ) = − d f ( y , x ) + ϕ f ( y ) and the Bregman representation of its conjugate prior
log p ( x ∣ z , α ) = − α z d f ( z , x ) + φ f ( α , α z ) . (24) \log p(\mathbf{x}\mid\mathbf{z},\alpha)=-\frac{\alpha}{z}d_f(\mathbf{z},\mathbf{x})+\varphi_f\left(\alpha,\frac{\alpha}{z}\right). \tag*{(24)} log p ( x ∣ z , α ) = − z α d f ( z , x ) + φ f ( α , z α ) . (24) Then, this conjugate prior admits the equivalent dual Bregman representation
log p ( x ∣ z , α ) = − d h ( τ ( x ) , τ ( z ) ) + φ f ( α , α z ) \log p(\mathbf{x}\mid\mathbf{z},\alpha)=-d_h(\tau(\mathbf{x}),\tau(\mathbf{z}))+\varphi_f\left(\alpha,\frac{\alpha}{z}\right) log p ( x ∣ z , α ) = − d h ( τ ( x ) , τ ( z )) + φ f ( α , z α )
where
τ : X → T and h : T → R ∪ { + ∞ } x ↦ f ∗ ( f ′ ( x ) ) t ↦ − α [ f ∗ ] − 1 ( t ) . \begin{aligned}
\tau&: \mathcal{X} \to\mathcal{T} && \text{and} && h : \mathcal{T} \to\mathbb{R} \cup\{+\infty\} \\
x &\mapsto f^{*}(f'(x)) &&&& t \mapsto-\alpha[f^{*}]^{-1}(t).
\end{aligned} τ x : X → T ↦ f ∗ ( f ′ ( x )) and h : T → R ∪ { + ∞ } t ↦ − α [ f ∗ ] − 1 ( t ) .
Then τ \tau τ is a strictly increasing diffeomorphism and h h h is twice continuously differentiable and strictly convex on T \mathcal{T} T . Moreover, the log-posterior can be written as
log p ( x ∣ y , z , α ) = − y + α α d h ( τ ( x ) , τ ( y + α 1 + α z ) ) + Ψ h ( y , z , α ) (25) \log p(x \mid y,z,\alpha) = -\frac{y+\alpha}{\alpha}d_h\left(\tau(x),\tau\left(\frac{y+\alpha}{1+\frac{\alpha}{z}}\right)\right)+\Psi_h(y,z,\alpha) \tag*{(25)} log p ( x ∣ y , z , α ) = − α y + α d h ( τ ( x ) , τ ( 1 + z α y + α ) ) + Ψ h ( y , z , α ) (25)
where Ψ h ( y , z , α ) \Psi_h(y,z,\alpha) Ψ h ( y , z , α ) is independent of x x x .
Proof. The existence of the dual representation of the prior easily follows from the application of Theorem 7 to the conjugate prior (24). The expression of the log-posterior distribution (25) results from the application of Theorem 7 applied to the log-posterior (19) derived in Proposition 5.
Theorem 9 shows that the scaling factor α / z \alpha/z α / z involved in the primal Bregman representation of the α \alpha α -scaled prior model can be absorbed into the coordinate mapping τ \tau τ and the convex generator h h h .
As expected, these results confirm that prior conjugacy is systematically ensured by first identifying the Bregman generator f f f associated with the likelihood and then adopting the β \beta β - or α \alpha α -scaled dual Bregman representations induced by ( h , τ ) (h,\tau) ( h , τ ) . Contrary to the primal Bregman representation of conjugate priors introduced by Agarwal and Daumé III [1 ] , the proposed representations rely on ordinary forms of Bregman divergences in dual coordinates, characterizing the geometry of the parameter space. Besides, this geometric characterization is independent of the particular probability distribution and relies only on the convex geometry induced by the likelihood generator. This suggests that conjugate Bayesian models may be organized according to their underlying Bregman geometry, rather than solely according to the form of their probability distributions.
Dual representations of classical exponential family distributions
The general construction established in Theorem 4 provides a systematic way of deriving dual Bregman representations for regular minimal exponential families. The β \beta β - and α \alpha α -scaled constructions of Theorem 7 are particular instances of this projective framework. Table 1 illustrates these two constructions for several classical exponential family models, with α = β = 1 \alpha=\beta=1 α = β = 1 for simplicity. The corresponding calculations are detailed in Appendix § 10 . The inverse-Gaussian family is also included as a non-regular example, illustrating the limitations of the global Legendre conclusions when the regularity assumption is not met.
The examples reported in this table reveal several structural properties of the proposed geometric framework. First, they highlight that several apparently different exponential-family models exhibit closely related dual Bregman geometries. This occurs whenever the primal Bregman representations of their densities are induced by the same convex generator f f f , or by generators differing only by a positive scaling factor. For instance, the Exponential and Gamma distributions with known shape parameter both induce (scaled versions of) the same Itakura-Saito geometry, while the Bernoulli and Binomial distributions exhibit closely related entropic geometries.
The α \alpha α -scaled construction of Theorem 7 provides explicit dual Bregman representations for one-dimensional regular minimal exponential families with expectation-parameter domain X ⊆ ( 0 , ∞ ) \mathcal{X}\subseteq(0,\infty) X ⊆ ( 0 , ∞ ) . For the corresponding models in Table 1 , the dual generator h h h is of Legendre type, consistently with Theorem 4.
The inverse-Gaussian family provides a contrasting example. Although this family is non-regular, it admits an α \alpha α -scaled dual Bregman representation. However, its generator is not essentially smooth on the transformed domain τ ( X ) = ( − ∞ , 0 ) \tau(\mathcal{X})=(-\infty,0) τ ( X ) = ( − ∞ , 0 ) . This illustrates the distinction between the existence of a dual representation and the stronger Legendre properties guaranteed by regularity.
These examples also illustrate how different scalings give rise to distinct dual Bregman representations of the same underlying likelihood divergence. In particular, the α \alpha α -scaled construction of Theorem 7 expresses a conformally weighted Bregman divergence as an ordinary Bregman divergence in transformed coordinates, through a suitable dual mapping τ \tau τ and generator h h h . For the Poisson distribution, this construction yields an Itakura–Saito geometry, whereas the β \beta β -scaled construction involves the exponential generator in natural coordinates. These representations are related by the coordinate transformations and conformal weights established in the general projective framework. This illustrates how the geometric representation of conjugacy depends on the chosen scaling, without changing the underlying statistical model.
Finally, the Gaussian, Gamma, and Beta examples with multiple unknown parameters demonstrate that the general projective framework is not restricted to one-dimensional exponential families. In particular, the β \beta β -scaled construction extends directly to multi-dimensional models, even when the corresponding Legendre mappings cannot be expressed in elementary closed form. For the Gamma distribution with unknown shape and rate parameters and the Beta distribution with two unknown shape parameters, the relevant dual coordinate mappings generally require numerical evaluation. This computational requirement does not affect the validity of the dual Bregman representation or the underlying Legendre properties.
Remark 6. (Application to conjugate priors) Adopting the dual Bregman representations proposed in § 3.1 and specifically instantiated to derive conjugate priors in
Section 3.2 recovers well-established conjugate models, as emphasized in Appendix § 10 whenever appropriate.
These examples demonstrate that the Bregman geometry associated with regular exponential families is not merely a model-specific representation, but reflects a common convex-analytic structure across these families. This raises another natural question: beyond providing an explicit ordinary Bregman formulation of regular exponential family log-density, do these β \beta β - and α \alpha α -scaled dual Bregman representations also induce a suitable geometry for analyzing the regularity of this log-density? We address this question in the next section through the prism of relative smoothness.
Relative smoothness of regular exponential families in dual Bregman geometries
This section investigates whether the dual Bregman geometry identified in § 3 also provides a natural reference geometry for relative smoothness. More precisely, we characterize the relative smoothness of the negative log-likelihood associated with the β \beta β - and α \alpha α -scaled models exhibited in § 3.1 . This establishes a connection between the geometry induced by the dual Bregman representations of regular exponential family distributions and their regularity properties required by Bregman-based optimization methods.
Geometry induced by β \beta β -scaled dual Bregman representations
We first investigate the geometry induced by the β \beta β -scaled ( h , τ ) (h,\tau) ( h , τ ) -dual representations identified in Theorem 7. Recall that, under such a parametrization, the dual coordinates are defined by t = τ ( x ) = ∇ f ( x ) \mathbf{t}=\tau(\mathbf{x})=\nabla f(\mathbf{x}) t = τ ( x ) = ∇ f ( x ) and the dual Bregman generator is h = β f ∗ h=\beta f^* h = β f ∗ . Interestingly, under such a geometry, the negative log-likelihood associated with a regular exponential family distribution satisfies the Lipschitz-like/convexity condition enounced in Definition 4.
Proposition 6. Consider the Bregman representation ℓ ( y ; x ) = − d f ( y , x ) + ϕ f ( y ) \ell(\mathbf{y};\mathbf{x})=-d_f(\mathbf{y},\mathbf{x})+\phi_f(\mathbf{y}) ℓ ( y ; x ) = − d f ( y , x ) + ϕ f ( y ) of the log-likelihood associated with a regular exponential family distribution. Let λ ( y ; ⋅ ) \lambda(\mathbf{y};\cdot) λ ( y ; ⋅ ) denote the negative log-likelihood expressed in dual coordinates induced by a β \beta β -scaled dual Bregman representation, i.e.,
λ ( y ; t ) = − ℓ ( y ; τ − 1 ( t ) ) with t = τ ( x ) = ∇ f ( x ) . (26) \lambda(\mathbf{y};\mathbf{t})=-\ell(\mathbf{y};\tau^{-1}(\mathbf{t})) \quad\text{with} \quad\mathbf{t}=\tau(\mathbf{x})=\nabla f(\mathbf{x}). \tag*{(26)} λ ( y ; t ) = − ℓ ( y ; τ − 1 ( t )) with t = τ ( x ) = ∇ f ( x ) . (26) For β > 0 \beta>0 β > 0 , define h ( t ) = β f ∗ ( t ) h(\mathbf{t})=\beta f^*(\mathbf{t}) h ( t ) = β f ∗ ( t ) . Then the function λ ( y ; ⋅ ) \lambda(\mathbf{y};\cdot) λ ( y ; ⋅ ) is L L L -smooth relative to h h h for every L ≥ 1 β L\geq\frac{1}{\beta} L ≥ β 1 .
Proof. See Appendix § 8 .9
Remark 7. Since f ∗ f^* f ∗ is strictly convex on T \mathcal{T} T , the threshold L = 1 / β L=1/\beta L = 1/ β is sharp: for L > 1 / β L>1/\beta L > 1/ β , the function L h − λ Lh-\lambda L h − λ is strictly convex, whereas for L = 1 / β L=1/\beta L = 1/ β it is affine. For L < 1 / β L<1/\beta L < 1/ β , the function L h − λ Lh-\lambda L h − λ is strictly concave.
This result shows that, for a given negative log-likelihood, a reference function h h h leading to a suitable reference geometry can be systematically exhibited by resorting to the β \beta β -scaled dual Bregman representation exhibited in Theorem 7. Then, as a specific instance of Proposition 3, it yields the Bregman descent lemma in dual coordinates
λ ( y ; t ′ ) ≤ λ ( y ; t ) + ⟨ ∇ λ ( y ; t ) , t ′ − t ⟩ + L d h ( t ′ , t ) \lambda(\mathbf{y};\mathbf{t}')\leq\lambda(\mathbf{y};\mathbf{t})+\langle\nabla\lambda(\mathbf{y};\mathbf{t}),\mathbf{t}'-\mathbf{t}\rangle+L\,d_h(\mathbf{t}',\mathbf{t}) λ ( y ; t ′ ) ≤ λ ( y ; t ) + ⟨ ∇ λ ( y ; t ) , t ′ − t ⟩ + L d h ( t ′ , t )
for all t , t ′ ∈ T t,t' \in\mathcal{T} t , t ′ ∈ T and every L ≥ 1 / β L \ge1/\beta L ≥ 1/ β . Moreover, whenever the natural and expectation parameters coincide, such as for Gaussian location model with unit variance, this descent lemma also holds in expectation coordinates. Proposition 6 also admits a component-wise formulation, in the case of a Bregman representation of the log-likelihood induced by a separable generator, as stated in the following corollary.
Corollary 10. Consider the Bregman representation ℓ ( y ; x ) = − d f ( y , x ) + ϕ f ( y ) \ell(y;x)=-d_f(y,x)+\phi_f(y) ℓ ( y ; x ) = − d f ( y , x ) + ϕ f ( y ) of the log-likelihood associated with a regular exponential family distribution. Assume the underlying generator f f f is separable, i.e.,
f ( x ) = ∑ i = 1 n f i ( x i ) . f(\mathbf{x})=\sum_{i=1}^{n} f_i(x_i). f ( x ) = i = 1 ∑ n f i ( x i ) . Let λ ( y ; t ) \lambda(\mathbf{y};\mathbf{t}) λ ( y ; t ) denote the negative log-likelihood expressed in dual coordinates t = τ ( x ) \mathbf{t}=\tau(\mathbf{x}) t = τ ( x ) induced by a component-wise β \beta β -scaled dual Bregman representation, i.e., with
t = ( t 1 , … , t n ) \mathbf{t}=(t_1,\ldots,t_n) t = ( t 1 , … , t n ) τ ( x ) = ( τ 1 ( x 1 ) , … , τ n ( x n ) ) \tau(\mathbf{x})=(\tau_1(x_1),\ldots,\tau_n(x_n)) τ ( x ) = ( τ 1 ( x 1 ) , … , τ n ( x n )) τ i ( x i ) = f i ′ ( x i ) . \tau_i(x_i)=f_i'(x_i). τ i ( x i ) = f i ′ ( x i ) . For β = ( β 1 , … , β n ) ⊤ \boldsymbol{\beta}=(\beta_1,\ldots,\beta_n)^\top β = ( β 1 , … , β n ) ⊤ and β i > 0 \beta_i>0 β i > 0 , let define h s e p ( t ) = ∑ i = 1 n β i f i ∗ ( t i ) h_{\mathrm{sep}}(\mathbf{t})=\sum_{i=1}^{n}\beta_i f_i^*(t_i) h sep ( t ) = ∑ i = 1 n β i f i ∗ ( t i ) . Then the function λ ( y ; ⋅ ) \lambda(\mathbf{y};\cdot) λ ( y ; ⋅ ) is L L L -smooth relative to h s e p h_{\mathrm{sep}} h sep for every L ≥ ∥ β − 1 ∥ ∞ L\ge\|\boldsymbol{\beta}^{-1}\|_\infty L ≥ ∥ β − 1 ∥ ∞ where the division in β − 1 \boldsymbol{\beta}^{-1} β − 1 should be understood componentwise.
Proof. The proof easily follows from the convexity of nonnegative linear combination of convex functions, combined with the proof of Proposition 6.
Remark 8. Since the negative log-likelihood associated with a regular exponential family distribution and its primal Bregman divergence only differs by the constant − ϕ f ( y ) -\phi_f(y) − ϕ f ( y ) , this property also provides a reference geometry for the Bregman divergence itself, yet expressed in the dual coordinates. In other words, the function t ↦ d f ( y ; τ − 1 ( t ) ) t\mapsto d_f(y;\tau^{-1}(t)) t ↦ d f ( y ; τ − 1 ( t )) is L L L -smooth relative to h h h for the same range of L L L values.
Geometry induced by α \alpha α -scaled dual Bregman representations
We now consider the geometry induced by the α \alpha α -scaled ( τ , h ) (\tau,h) ( τ , h ) -dual Bregman representations introduced in Theorem 7. Under such a parametrization, contrary to the previous one, the dual coordinates t t t do not, in general, coincide with the natural parameter of the regular exponential family model. Instead, they write as t = τ ( x ) = f ∗ ( f ′ ( x ) ) t=\tau(x)=f^*(f'(x)) t = τ ( x ) = f ∗ ( f ′ ( x )) while the dual generator is h ( t ) = − α ( f ∗ ) − 1 ( t ) h(t)=-\alpha(f^*)^{-1}(t) h ( t ) = − α ( f ∗ ) − 1 ( t ) . Again, adopting such a geometry ensures relative smoothness of the log-likelihood expressed in dual coordinates, as stated above.
Proposition 7. Consider a regular exponential family with one-dimensional expectation-parameter domain X ⊂ ( 0 , ∞ ) \mathcal{X}\subset(0,\infty) X ⊂ ( 0 , ∞ ) , and let
ℓ ( y ; x ) = − d f ( y , x ) + ϕ f ( y ) \ell(y;x)=-d_f(y,x)+\phi_f(y) ℓ ( y ; x ) = − d f ( y , x ) + ϕ f ( y ) be its Bregman log-likelihood representation.
Let λ ( y ; ⋅ ) \lambda(y;\cdot) λ ( y ; ⋅ ) denote the negative log-likelihood expressed in dual coordinates induced by an α \alpha α -scaled dual Bregman representation, i.e.,
λ ( y ; t ) = − ℓ ( y ; τ − 1 ( t ) ) with t = τ ( x ) = f ∗ ( f ′ ( x ) ) . (27) \lambda(y;t)=-\ell(y;\tau^{-1}(t)) \quad\text{with} \quad t=\tau(x)=f^*(f'(x)). \tag*{(27)} λ ( y ; t ) = − ℓ ( y ; τ − 1 ( t )) with t = τ ( x ) = f ∗ ( f ′ ( x )) . (27) For α > 0 \alpha>0 α > 0 , define h ( t ) = − α f ′ ( τ − 1 ( t ) ) h(t)=-\alpha f'(\tau^{-1}(t)) h ( t ) = − α f ′ ( τ − 1 ( t )) . If the function h h h is a Legendre function, then
if y ≠ 0 y \ne0 y = 0 , the function λ ( y ; ⋅ ) \lambda(y;\cdot) λ ( y ; ⋅ ) is L L L -smooth relative to h h h for every L ≥ y α L \ge\frac{y}{\alpha} L ≥ α y .
if y = 0 y = 0 y = 0 , for any ϵ > 0 \epsilon> 0 ϵ > 0 , the function λ ( y ; ⋅ ) \lambda(y;\cdot) λ ( y ; ⋅ ) is L L L -smooth relative to h h h for every L ≥ ϵ L \ge\epsilon L ≥ ϵ .
Proof. See Appendix A.10
Remark 9. Since h h h is strictly convex on int ( dom h ) \operatorname{int}(\operatorname{dom} h) int ( dom h ) , the threshold L = y / α L = y/\alpha L = y / α is sharp: for L > y / α L > y/\alpha L > y / α , the function L h − λ Lh - \lambda L h − λ is strictly convex, whereas for L = y / α L = y/\alpha L = y / α it is affine. For L < y / α L < y/\alpha L < y / α , it is strictly concave.
The corresponding Bregman descent lemma in the transformed coordinates reads
λ ( y ; t ′ ) ≤ λ ( y ; t ) + ( t ′ − t ) λ ′ ( y ; t ) + L d h ( t ′ , t ) , (28) \lambda(y;t') \le\lambda(y;t) + (t' - t)\lambda'(y;t) + L d_h(t',t), \tag*{(28)} λ ( y ; t ′ ) ≤ λ ( y ; t ) + ( t ′ − t ) λ ′ ( y ; t ) + L d h ( t ′ , t ) , (28)
for all t , t ′ ∈ int ( dom h ) t,t' \in\operatorname{int}(\operatorname{dom} h) t , t ′ ∈ int ( dom h ) and every L ≥ y / α L \ge y/\alpha L ≥ y / α . Moreover, whenever τ = Id \tau= \operatorname{Id} τ = Id , the primal and dual coordinates coincide and this descent lemma also holds in expectation coordinates. An archetypal example is the Poisson model, which will be deeply investigated in § 5 . This particular example also reveals important features when Proposition 7 extends to a separable multidimensional model, as stated in the following corollary (see Remark 10).
Corollary 11. Consider the Bregman representation ℓ ( y ; x ) = − d f ( y , x ) + ϕ f ( y ) \ell(y;x) = -d_f(y,x) + \phi_f(y) ℓ ( y ; x ) = − d f ( y , x ) + ϕ f ( y ) of the log-likelihood associated with a regular exponential family distribution. Assume the underlying generator f f f is separable, i.e.,
f ( x ) = ∑ i = 1 n f i ( x i ) . (29) f(\mathbf{x}) = \sum_{i=1}^{n} f_i(x_i). \tag*{(29)} f ( x ) = i = 1 ∑ n f i ( x i ) . (29) Assume moreover that each component has an expectation-parameter domain X i ⊂ ( 0 , ∞ ) \mathcal{X}_i \subset(0,\infty) X i ⊂ ( 0 , ∞ ) . Let λ ( y ; t ) \lambda(y;t) λ ( y ; t ) denote the negative log-likelihood expressed in dual coordinates t = τ ( x ) \mathbf{t} = \tau(\mathbf{x}) t = τ ( x ) induced by a component-wise α \alpha α -scaled dual Bregman representation, i.e., with
t = ( t 1 , … , t n ) (30) \mathbf{t} = (t_1,\ldots,t_n) \tag*{(30)} t = ( t 1 , … , t n ) (30) τ ( x ) = ( τ 1 ( x 1 ) , … , τ n ( x n ) ) (31) \tau(\mathbf{x}) = (\tau_1(x_1),\ldots,\tau_n(x_n)) \tag*{(31)} τ ( x ) = ( τ 1 ( x 1 ) , … , τ n ( x n )) (31) τ i ( x i ) = f i ∗ ( f i ′ ( x i ) ) . (32) \tau_i(x_i) = f_i^*(f_i'(x_i)). \tag*{(32)} τ i ( x i ) = f i ∗ ( f i ′ ( x i )) . (32) For α = ( α 1 , … , α n ) ⊤ \boldsymbol{\alpha} = (\alpha_1,\ldots,\alpha_n)^\top α = ( α 1 , … , α n ) ⊤ and α i > 0 \alpha_i > 0 α i > 0 , define h s e p ( t ) = ∑ i = 1 n h i ( t i ) h_{\mathrm{sep}}(\mathbf{t}) = \sum_{i=1}^{n} h_i(t_i) h sep ( t ) = ∑ i = 1 n h i ( t i ) with h i ( t i ) = − α i f i ′ ( τ i − 1 ( t i ) ) h_i(t_i) = -\alpha_i f_i'(\tau_i^{-1}(t_i)) h i ( t i ) = − α i f i ′ ( τ i − 1 ( t i )) . If all functions h i h_i h i are Legendre functions then
if ∥ y ∥ ∞ ≠ 0 \lVert\mathbf{y} \rVert_\infty\ne0 ∥ y ∥ ∞ = 0 , the function λ ( y ; ⋅ ) \lambda(\mathbf{y};\cdot) λ ( y ; ⋅ ) is L L L -smooth relative to h s e p h_{\mathrm{sep}} h sep for every L ≥ ∥ y / α ∥ ∞ L \ge\lVert\mathbf{y}/\boldsymbol{\alpha} \rVert_\infty L ≥ ∥ y / α ∥ ∞ where the division by α \boldsymbol{\alpha} α should be understood componentwise,
if ∥ y ∥ ∞ = 0 \lVert\mathbf{y} \rVert_\infty= 0 ∥ y ∥ ∞ = 0 , for any ϵ > 0 \epsilon> 0 ϵ > 0 , the function λ ( y ; ⋅ ) \lambda(\mathbf{y};\cdot) λ ( y ; ⋅ ) is L L L -smooth relative to h s e p h_{\mathrm{sep}} h sep for every L ≥ ϵ L \ge\epsilon L ≥ ϵ ,
Moreover, the threshold L min = ∥ y / α ∥ ∞ L_{\min} = \lVert\mathbf{y}/\boldsymbol{\alpha} \rVert_\infty L m i n = ∥ y / α ∥ ∞ is sharp.
Proof. See Appendix A.11
Remark 10. For Poisson linear inverse problems, Bauschke et al. [7 ] established the relative-smoothness condition using the Burg entropy as reference function, with a sufficient constant L ≥ ∥ y ∥ 1 L \ge\lVert\mathbf{y} \rVert_1 L ≥ ∥ y ∥ 1 . For the separable model considered above with α i = α = 1 \alpha_i = \alpha= 1 α i = α = 1 , the componentwise structure instead leads to the bound L ≥ ∥ y ∥ ∞ L \ge\lVert\mathbf{y} \rVert_\infty L ≥ ∥ y ∥ ∞ . These bounds arise from different model structures and reference geometries: the result enounced by Bauschke et al. [7 ] is associated to a Poisson inverse problem which involves a linear operator coupling the components whereas the result above exploits a separable structure. Theorem 11 may prove useful when deriving splitting-based iterative schemes, where the introduction of auxiliary variables can restore a componentwise structure in conditional models [27 , 15 ] .
Remark 11. Since the negative log-likelihood − ℓ ( y ; x ) -\ell(\mathbf{y};\mathbf{x}) − ℓ ( y ; x ) associated with a regular exponential family distribution and its primal Bregman divergence only differs by the constant − ϕ f ( y ) -\phi_f(\mathbf{y}) − ϕ f ( y ) , this property also provides a reference geometry for the Bregman divergence itself, yet expressed in the dual coordinates. In other words, the function t ↦ d f ( y ; τ − 1 ( t ) ) \mathbf{t} \mapsto d_f(\mathbf{y};\tau^{-1}(\mathbf{t})) t ↦ d f ( y ; τ − 1 ( t )) is L L L -smooth relative to h h h for the same range of L L L values.
Discussion: from dual Bregman representation to relative smoothness
The results established in this section establish a direct connection between dual Bregman representations and relative smoothness in regular exponential families. For instance, this connection is particularly relevant to address Poisson inverse problems, where the negative log-likelihood typically has a gradient that is not globally Lipschitz on its natural domain. Bauschke et al. [7 ] showed that this issue can be overcome by replacing the classical Lipschitz-gradient condition with a Bregman descent condition. On an ad hoc basis, they identified the Burg entropy as one appropriate reference function to ensure the relative smoothness. This framework has subsequently been exploited in Poisson imaging, including Bregman plug-and-play methods [16 ] and mirror-descent-based deep equilibrium models [12 ] . It will be discussed more deeply in § 5 .
Conversely, Proposition 6 and Proposition 7 offer a different perspective on this quest for an appropriate reference geometry. More precisely, for regular exponential family distributions, a reference geometry ensuring relative smoothness of the negative log-likelihood in an appropriate coordinate system arises directly from the dual Bregman representation of this distribution. In the β \beta β -scaled formulation, passing to the natural coordinates t = ∇ f ( x ) \mathbf{t} = \nabla f(\mathbf{x}) t = ∇ f ( x ) and using the generator h = β f ∗ h = \beta f^* h = β f ∗ yields relative smoothness of the negative log-likelihood with respect to the same Bregman geometry. Likewise, the α \alpha α -scaled formulation induces its own transformed coordinate system and Bregman generator, which also provide a reference geometry for relative smoothness. Thus, in both representations, the geometry arising from the dual Bregman representation of the density is naturally adapted to its own regularity. Interestingly, within a Bayesian context, the dual Bregman representation of conjugate priors discussed in § 3.2 also reveals an intrinsic compatibility between the geometry of this prior and the geometry of the likelihood.
The resulting relative smoothness constants are explicitly determined by the scaling function and depend on the corresponding coordinate representation. In the β \beta β -scaled formulation, the sharp threshold is L min = 1 / β L_{\min} = 1/\beta L m i n = 1/ β and L min = ∥ β − 1 ∥ ∞ L_{\min} = \lVert\beta^{-1}\rVert_{\infty} L m i n = ∥ β − 1 ∥ ∞ in the separable multidimensional setting. In the α \alpha α -scaled formulation, the dual coordinates t = τ ( x ) \mathbf{t} = \tau(\mathbf{x}) t = τ ( x ) yield the sharp threshold L min = y / α L_{\min} = y/\alpha L m i n = y / α in the scalar case and L min = ∥ y / α ∥ ∞ L_{\min} = \lVert\mathbf{y}/\alpha\rVert_{\infty} L m i n = ∥ y / α ∥ ∞ in the separable multidimensional setting. Hence, the two conjugate parametrizations lead to distinct, yet explicitly characterized, Bregman geometries in which the negative log-likelihood admits a quantitative descent control.
Finally, these results have a direct computational interpretation. Relative smoothness yields a Bregman descent inequality, which provides a basic estimate underlying Bregman and mirror-based first-order methods [8 , 7 ] . The geometry induced by the dual Bregman representation is therefore not only relevant:VEVENT to the characterization of the parameter space, as stated in § 3 , but also provides a natural framework for controlling the associated likelihood. In this sense, these representations connect the statistical structure of regular exponential family models with the convex-analytic regularity properties that underlie Bregman-based computational methods. This connection is illustrated in the next section by addressing Poisson linear inverse problems in the proposed framework.
Application to Poisson linear inverse problems
To illustrate the relevance of the proposed geometrical framework, we propose to address Poisson linear inverse problems from the scope of the dual Bregman representations derived in § 3 and further explored in § 4 . In this context, the data consists in observed counts y ∈ N m \mathbf{y} \in\mathbb{N}^{m} y ∈ N m related to an unknown parameter x ∈ R n \mathbf{x} \in\mathbb{R}^{n} x ∈ R n to be recovered through the degradation model
y i ∼ P ( [ K x ] i ) , i = 1 , … , m , y_i \sim\mathcal{P}([\mathbf{K}\mathbf{x}]_i), \qquad i=1,\ldots,m, y i ∼ P ([ Kx ] i ) , i = 1 , … , m ,
where K ∈ R + m × n \mathbf{K} \in\mathbb{R}_{+}^{m \times n} K ∈ R + m × n is a forward operator satisfying K 1 n > 0 \mathbf{K}\mathbf{1}_n > 0 K 1 n > 0 . Adopting a penalized maximum likelihood paradigm, the problem consists in solving the following minimization problem
min x ∈ R + n J ( x ) ≜ − ℓ ( y ; K x ) + r ( x ) (33) \underset{\mathbf{x}\in\mathbb{R}_{+}^{n}}{\min}\ \mathcal{J}(\mathbf{x}) \triangleq-\ell(\mathbf{y};\mathbf{K}\mathbf{x})+r(\mathbf{x}) \tag*{(33)} x ∈ R + n min J ( x ) ≜ − ℓ ( y ; Kx ) + r ( x ) (33)
where − ℓ ( y ; ⋅ ) -\ell(\mathbf{y};\cdot) − ℓ ( y ; ⋅ ) is the Poisson negative log-likelihood and r ( ⋅ ) r(\cdot) r ( ⋅ ) is a regularization function. This problem is particularly relevant for the present framework since the Poisson distribution is naturally associated with non-Euclidean geometries. Indeed, Theorem 1 states that the Poisson negative log-likelihood admits the primal Bregman representation
− ℓ ( y ; K x ) = d f ( y , K x ) − ϕ f ( y ) -\ell(\mathbf{y};\mathbf{K}\mathbf{x})=d_f(\mathbf{y},\mathbf{K}\mathbf{x})-\phi_f(\mathbf{y}) − ℓ ( y ; Kx ) = d f ( y , Kx ) − ϕ f ( y )
where the Bregman generator is the negative Boltzmann-Shannon entropy function f ( x ) = x log x − x f(x)=x\log x-x f ( x ) = x log x − x on dom f = R + \operatorname{dom}f=\mathbb{R}_{+} dom f = R + (with 0 log 0 = 0 0\log0=0 0 log 0 = 0 ) and X = R + ∗ \mathcal{X}=\mathbb{R}_{+}^{*} X = R + ∗ , inducing the generalized Kullback-Leibler divergence (see Table 3 ). In particular, when K = I \mathbf{K}=\mathbf{I} K = I , according to Propositions 6 and 7, the dual Bregman representations identified in § 3 provide two different reference geometries for establishing relative smoothness of the negative log-likelihood expressed in dual coordinate systems. To appreciate the different relative-smoothness structures induced by the two resulting geometries, we analyze the properties of the Bregman proximal gradient method (BPGM) as studied by Bauschke et al. [7 ] where the regularization denoted by r ( ⋅ ) r(\cdot) r ( ⋅ ) is chosen as the ℓ 1 \ell_1 ℓ 1 -norm sparsity-promoting penalization
Table 3. Primal Bregman representations of classical regular exponential family models. Fixed parameters appear in gray.
r ( x ) = μ ∥ x ∥ 1 = μ ∑ i = 1 n x i . r(\mathbf{x})=\mu\lVert\mathbf{x}\rVert_1=\mu\sum_{i=1}^{n}x_i. r ( x ) = μ ∥ x ∥ 1 = μ i = 1 ∑ n x i .
When h h h is a reference function ensuring that the negative log-likelihood satisfies the LC condition in the primal domain, BPGM minimizes a composite functional such as in (33) by iterating the updating rule
x + = arg min u > 0 { ⟨ − ∇ ℓ ( y ; K x ) , u − x ⟩ + r ( u ) + 1 δ d h ( u , x ) } (34) \mathbf{x}^{+}=\underset{\mathbf{u}>0}{\operatorname{arg\,min}}\left\{\langle-\nabla\ell(\mathbf{y};\mathbf{K}\mathbf{x}),\mathbf{u}-\mathbf{x}\rangle+r(\mathbf{u})+\frac{1}{\delta}d_h(\mathbf{u},\mathbf{x})\right\} \tag*{(34)} x + = u > 0 arg min { ⟨ − ∇ ℓ ( y ; Kx ) , u − x ⟩ + r ( u ) + δ 1 d h ( u , x ) } (34)
where δ > 0 \delta>0 δ > 0 denotes the stepsize. In what follows, we propose to instantiate the BPGM in the dual coordinate system when these dual coordinates and the reference functions are derived from the α \alpha α - or β \beta β -scaled dual Bregman representation of the Poisson distribution, i.e., with the two generators h h h specified in Table 1 . To easily assess the impact of the chosen geometry, we will first consider the case K = I \mathbf{K} = \mathbf{I} K = I which perfectly matches the framework introduced in this paper. By removing the coupling between coordinates, this simple setting will also allow one to derive the solution of the problem (33) explicitly. It will enable the explicit characterization of the induced iterations, relative-curvature constants, and convergence rates of the algorithms. Then, we will explore the case K ≠ I \mathbf{K} \ne\mathbf{I} K = I , which sheds striking light on some existing results from the literature.
Geometry induced by the α \alpha α -scaled dual Bregman representation
For the Poisson distribution, the α \alpha α -scaled dual Bregman representation with α = 1 \alpha= 1 α = 1 yields (see Table 1 and Section C.2)
τ ( x ) = x , h ( t ) = − ∑ j = 1 n log t j \tau(\mathbf{x}) = \mathbf{x}, \qquad h(\mathbf{t}) = -\sum_{j=1}^{n} \log t_j τ ( x ) = x , h ( t ) = − j = 1 ∑ n log t j
and the Itakura-Saito divergence
d h ( u , t ) = ∑ j = 1 n [ u j t j − log ( u j t j ) − 1 ] . d_h(\mathbf{u},\mathbf{t}) = \sum_{j=1}^{n} \left[\frac{u_j}{t_j} - \log\left(\frac{u_j}{t_j}\right) - 1\right]. d h ( u , t ) = j = 1 ∑ n [ t j u j − log ( t j u j ) − 1 ] .
Interestingly, the choice of the Burg entropy as a suitable Legendre function h h h was already considered in several works from the literature to derive BPGM for solving Poisson inverse problems [7 , 25 , 10 ] . However it is worth noting that, in those works, this choice was never properly motivated but rather resulted from an ad hoc approach, driven by the search of one possible reference geometry ensuring the LC condition. In contrast, in our work, this reference geometry naturally emerges from a principled construction, as a direct consequence of the dual Bregman representation exhibited in Theorem 7 and underlying Proposition 7.
Case K = I \mathbf{K} = \mathbf{I} K = I
We first assume that
y j > 0 , j = 1 , … , n . y_j > 0, \qquad j = 1,\ldots,n. y j > 0 , j = 1 , … , n .
The particular case of zero-count will be discussed later (see Remark 13 ). When K = I \mathbf{K} = \mathbf{I} K = I , the objective function writes
J ( x ) = ∑ j = 1 n [ ( 1 + μ ) x j − y j log x j ] \mathcal{J}(\mathbf{x}) = \sum_{j=1}^{n} \left[(1+\mu)x_j - y_j \log x_j\right] J ( x ) = j = 1 ∑ n [ ( 1 + μ ) x j − y j log x j ]
and its unique minimizer is
x ⋆ = 1 1 + μ y . (35) \mathbf{x}^{\star} = \frac{1}{1+\mu}\mathbf{y}. \tag*{(35)} x ⋆ = 1 + μ 1 y . (35)
Since τ = Id \tau= \operatorname{Id} τ = Id , according to Proposition 7, the Burg entropy h h h provides a suitable reference geometry to ensure relative smoothness of the Poisson negative log-likelihood function in the primal and the dual coordinate systems equivalently. Thus, the BPGM iteration (34) can be implemented in the primal domain and, given that ∇ u d h ( u , x ) = x − 1 − u − 1 \nabla_{\mathbf{u}}d_h(\mathbf{u},\mathbf{x}) = \mathbf{x}^{-1} - \mathbf{u}^{-1} ∇ u d h ( u , x ) = x − 1 − u − 1 , the first-order optimality condition, leads to the following multiplicative update rules
x + = x 1 + δ ( ( μ + 1 ) x − y ) (36) \mathbf{x}^{+} = \frac{\mathbf{x}}{1+\delta((\mu+1)\mathbf{x}-\mathbf{y})} \tag*{(36)} x + = 1 + δ (( μ + 1 ) x − y ) x (36)
where the division should be understood component-wise. Assume now that
0 < δ ≤ 1 y max (37) 0 < \delta\le\frac{1}{y_{\max}} \tag*{(37)} 0 < δ ≤ y m a x 1 (37)
with y max = ∥ y ∥ ∞ y_{\max} = \lVert y\rVert_{\infty} y m a x = ∥ y ∥ ∞ . Since y j > 0 y_j > 0 y j > 0 for every j j j , this implies
0 ≤ 1 − δ y j < 1 (38) 0 \le1-\delta y_j < 1 \tag*{(38)} 0 ≤ 1 − δ y j < 1 (38)
and the denominator in (36) satisfies
1 + δ ( ( 1 + μ ) x j − y j ) > 0 1+\delta((1+\mu)x_j-y_j)>0 1 + δ (( 1 + μ ) x j − y j ) > 0
for every x j > 0 x_j>0 x j > 0 . This condition guarantees that the BPGM iterations are geometrically compliant, i.e., they are well defined and remain in the positive orthant for every initialization x 0 ∈ R + + n \mathbf{x}^{0}\in\mathbb{R}^{n}_{++} x 0 ∈ R ++ n . To further analyze this iterative scheme which is component-wise separable, consider the reciprocal variable
v j = 1 x j . v_j=\frac{1}{x_j}. v j = x j 1 .
By using (35) , the updating rule (36) becomes
v j + = v j ⋆ + ( 1 − δ y j ) ( v j − v j ⋆ ) . (39) v_j^{+}=v_j^{\star}+(1-\delta y_j)(v_j-v_j^{\star}). \tag*{(39)} v j + = v j ⋆ + ( 1 − δ y j ) ( v j − v j ⋆ ) . (39)
and, after k k k iterations,
v j k = v j ⋆ + ( 1 − δ y j ) k ( v j 0 − v j ⋆ ) . (40) v_j^{k}=v_j^{\star}+(1-\delta y_j)^{k}(v_j^{0}-v_j^{\star}). \tag*{(40)} v j k = v j ⋆ + ( 1 − δ y j ) k ( v j 0 − v j ⋆ ) . (40)
Moreover, let define
q α = 1 − δ y min q_{\alpha}=1-\delta y_{\min} q α = 1 − δ y m i n
with y min = min j y j > 0 y_{\min}=\min_j y_j>0 y m i n = min j y j > 0 . Since the stepsize δ \delta δ satisfies (38) , we have
0 ≤ 1 − δ y j ≤ 1 − δ y min < 1 0\le1-\delta y_j\le1-\delta y_{\min}<1 0 ≤ 1 − δ y j ≤ 1 − δ y m i n < 1
and it follows from (40) that
∥ v k − v ⋆ ∥ ∞ ≤ q α k ∥ v 0 − v ⋆ ∥ ∞ . \lVert\mathbf{v}^{k}-\mathbf{v}^{\star}\rVert_{\infty}\le q_{\alpha}^{k}\lVert\mathbf{v}^{0}-\mathbf{v}^{\star}\rVert_{\infty}. ∥ v k − v ⋆ ∥ ∞ ≤ q α k ∥ v 0 − v ⋆ ∥ ∞ .
Thus the iteration converges globally and linearly in the reciprocal coordinates, with exact worst-case contraction factor q α q_{\alpha} q α . At the largest stepsize δ = 1 y max \delta=\frac{1}{y_{\max}} δ = y m a x 1 allowed by (37) , the exact worst-case contraction factor is
q I S = 1 − y min y max . (41) q_{\mathrm{IS}}=1-\frac{y_{\min}}{y_{\max}}. \tag*{(41)} q IS = 1 − y m a x y m i n . (41)
The continuity of the reciprocal mapping towards the original domain on ( 0 , ∞ ) (0,\infty) ( 0 , ∞ ) gives lim k → + ∞ x k = x ⋆ \lim_{k\to+\infty}\mathbf{x}^{k}=\mathbf{x}^{\star} lim k → + ∞ x k = x ⋆ for any x 0 ∈ R + + n \mathbf{x}^{0}\in\mathbb{R}^{n}_{++} x 0 ∈ R ++ n . Hence the convergence statement in the original variables follows directly from the exact linear convergence established in the reciprocal coordinates.
Remark 12. This result also admits a natural interpretation in terms of relative geometry. Indeed,
∇ 2 J ( x ) = Diag ( y j x j 2 ) , \nabla^{2}\mathcal{J}(\mathbf{x})=\operatorname{Diag}\left(\frac{y_j}{x_j^{2}}\right), ∇ 2 J ( x ) = Diag ( x j 2 y j ) ,
whereas the Burg generator induced by the α \alpha α -scaled dual Bregman representation satisfies
∇ 2 h ( x ) = Diag ( 1 x j 2 ) . \nabla^2 h(\mathbf{x}) = \operatorname{Diag}\left(\frac{1}{x_j^2}\right). ∇ 2 h ( x ) = Diag ( x j 2 1 ) .
Hence the two Hessians exhibit the same dependence on the variable x j x_j x j , and differ only through the coefficients y j y_j y j . Consequently,
y min ∇ 2 h ( x ) ⪯ ∇ 2 J ( x ) ⪯ y max ∇ 2 h ( x ) . (42) y_{\min}\nabla^2 h(\mathbf{x}) \preceq\nabla^2 \mathcal{J}(\mathbf{x}) \preceq y_{\max}\nabla^2 h(\mathbf{x}). \tag*{(42)} y m i n ∇ 2 h ( x ) ⪯ ∇ 2 J ( x ) ⪯ y m a x ∇ 2 h ( x ) . (42)
The exact relative strong-convexity and relative-smoothness constants are thus
μ α = y min and L α = y max . \mu_\alpha= y_{\min}\quad\text{and}\quad L_\alpha= y_{\max}. μ α = y m i n and L α = y m a x .
Introducing the corresponding relative condition number
κ α = L α μ α = y max y min , \kappa_\alpha= \frac{L_\alpha}{\mu_\alpha} = \frac{y_{\max}}{y_{\min}}, κ α = μ α L α = y m i n y m a x ,
the exact contraction factor obtained in (41) can equivalently be written as
q α = 1 − μ α L α = 1 − 1 κ α . q_\alpha= 1 - \frac{\mu_\alpha}{L_\alpha} = 1 - \frac{1}{\kappa_\alpha}. q α = 1 − L α μ α = 1 − κ α 1 .
This is the familiar dependence of a gradient-type contraction factor on a condition number, here expressed in the relative Burg geometry rather than in the Euclidean geometry. In particular, κ α = 1 \kappa_\alpha= 1 κ α = 1 corresponds to perfect relative conditioning and gives q α = 0 q_\alpha= 0 q α = 0 , whereas the convergence becomes progressively slower as κ α \kappa_\alpha κ α increases. It is worth emphasizing that, in the present case with K = I \mathbf{K} = \mathbf{I} K = I , this expression is not obtained by applying a generic convergence bound. The contraction factor follows directly from the exact scalar recursion (39) . The relative-curvature analysis shows, independently, that this exact factor can be expressed as 1 − μ α / L α 1-\mu_\alpha/L_\alpha 1 − μ α / L α .
Case K not identity
When K ≠ I \mathbf{K} \ne\mathbf{I} K = I , the linear operator couples the components of x \mathbf{x} x in the likelihood function, which makes the divergence d f ( y , K x ) d_f(\mathbf{y},\mathbf{Kx}) d f ( y , Kx ) not separable. Thus the assumptions required by Proposition 7 or Theorem 11 are not met and these statements are not sufficient to ensure relative smoothness of the Poisson negative log-likelihood. However, as already highlighted (see Remark 10), Bauschke et al. [7 ] filled this gap by stating the following property.
Lemma 1. ([7 Lemma 7] ) Consider the Bregman representation ℓ ( y ; K x ) = − d f ( y ; K x ) + ϕ f ( y ) \ell(\mathbf{y};\mathbf{Kx})=-d_f(\mathbf{y};\mathbf{Kx})+\phi_f(\mathbf{y}) ℓ ( y ; Kx ) = − d f ( y ; Kx ) + ϕ f ( y ) of the log-likelihood associated with a Poisson linear inverse problem. Let λ ~ ( y ; x ) \tilde{\lambda}(\mathbf{y};\mathbf{x}) λ ~ ( y ; x ) denote the negative log-likelihood function, i.e.,
λ ~ ( y ; x ) = d f ( y ; K x ) − ϕ f ( y ) . \tilde{\lambda}(\mathbf{y};\mathbf{x})=d_f(\mathbf{y};\mathbf{Kx})-\phi_f(\mathbf{y}). λ ~ ( y ; x ) = d f ( y ; Kx ) − ϕ f ( y ) . Let define h ( x ) = − ∑ i = 1 n log x i h(\mathbf{x})=-\sum_{i=1}^{n}\log x_i h ( x ) = − ∑ i = 1 n log x i . Then the function λ ~ ( y ; x ) \tilde{\lambda}(\mathbf{y};\mathbf{x}) λ ~ ( y ; x ) is L L L -smooth relative to h h h for every L ≥ L B = ∥ y ∥ 1 L\geq L_{\mathrm{B}}=\lVert\mathbf{y}\rVert_1 L ≥ L B = ∥ y ∥ 1 .
This proposition shows that the reference geometry identified by applying Proposition 7 and corresponding to the case K = I \mathbf{K}=\mathbf{I} K = I remains suitable to ensure relative smoothness of the negative log-likelihood in the case K ≠ I \mathbf{K}\ne\mathbf{I} K = I , yet with a different constant L B L_{\mathrm{B}} L B . The BPGM iterations (34) can thus be derived in this case to generalize (36) . They write
x + = x 1 + δ x ⊙ ( μ I + s − K T y K x ) \mathbf{x}^{+}=\frac{\mathbf{x}}{1+\delta\mathbf{x}\odot\left(\boldsymbol{\mu}\mathbf{I}+\mathbf{s}-\mathbf{K}^{\mathsf{T}}\frac{\mathbf{y}}{\mathbf{Kx}}\right)} x + = 1 + δ x ⊙ ( μ I + s − K T Kx y ) x
where s = ( s 1 , … , s n ) \mathbf{s}=(s_1,\ldots,s_n) s = ( s 1 , … , s n ) with s j = ∑ i = 1 m k i , j s_j=\sum_{i=1}^{m}k_{i,j} s j = ∑ i = 1 m k i , j . As expected, this iterative scheme perfectly matches the so-called NoLips algorithm derived by Bauschke et al. [7 ] to solve (33) . In other words, even when K ≠ I \mathbf{K}\ne\mathbf{I} K = I , the α \alpha α -scaled Bregman representation recovers the Itakura-Saito geometry and the corresponding NoLips update. Besides, given the convexity and smoothness characterization (42) , when y > 0 \mathbf{y}>0 y > 0 , recall that the exact relative constants in the identity case are
μ α = y min , L α = y max . \mu_{\alpha}=y_{\min},\qquad L_{\alpha}=y_{\max}. μ α = y m i n , L α = y m a x .
In particular, y max ≤ ∥ y ∥ 1 y_{\max}\leq\lVert\mathbf{y}\rVert_1 y m a x ≤ ∥ y ∥ 1 , so the general relative-smoothness bound L B L_{\mathrm{B}} L B identified by Bauschke et al. [7 ] in the general case can be sharpened, in the identity case, to the exact constant y max y_{\max} y m a x .
Geometry induced by the β \beta β -scaled dual Bregman representation
We now consider the geometry induced by the β \beta β -scaled dual Bregman representation as discussed in § 4.1 . More precisely, as in Theorem 10, we consider a separable generator h h h parametrized by non-negative weights β j > 0 \beta_j>0 β j > 0 , which leads to (see Table 1 and Section C.2)
τ ( x ) = log x , h ( t ) = ∑ j = 1 n β j e t j . \tau(\mathbf{x})=\log\mathbf{x},\qquad h(\mathbf{t})=\sum_{j=1}^{n}\beta_j e^{t_j}. τ ( x ) = log x , h ( t ) = j = 1 ∑ n β j e t j .
and the generalized Kullback-Leibler divergence KL ( y ∥ x ) \operatorname{KL}(\mathbf{y}\|\mathbf{x}) KL ( y ∥ x ) expressed in the dual coordinates
d h ( u , t ) = ∑ j = 1 n β j [ e y j − e t j − e t j ( u j − t j ) ] . d_h(\mathbf{u},\mathbf{t})=\sum_{j=1}^{n}\beta_j\left[e^{y_j}-e^{t_j}-e^{t_j}(u_j-t_j)\right]. d h ( u , t ) = j = 1 ∑ n β j [ e y j − e t j − e t j ( u j − t j ) ] .
Several choices of the weights β = ( β 1 , … , β n ) T \boldsymbol{\beta}=(\beta_1,\ldots,\beta_n)^{\mathsf{T}} β = ( β 1 , … , β n ) T will be discussed along this section.
Case K = I
We first assume that
y j > 0 , j = 1 , … , n . y_j > 0,\qquad j=1,\ldots,n. y j > 0 , j = 1 , … , n .
The particular case of zero-count will be discussed later (see Remark 15). In the identity case, the objective function (33) expressed in the dual coordinates is
J ~ ( t ) ≜ J ( e t ) = λ ( y ; t ) + r ( e t ) (43) \widetilde{\mathcal{J}}(\mathbf{t}) \triangleq\mathcal{J}(e^{\mathbf{t}})
= \lambda(\mathbf{y};\mathbf{t}) + r(e^{\mathbf{t}}) \tag*{(43)} J ( t ) ≜ J ( e t ) = λ ( y ; t ) + r ( e t ) (43)
where λ ( y ; t ) \lambda(\mathbf{y};\mathbf{t}) λ ( y ; t ) has been defined in (26) as the negative log-likelihood expressed in the dual coordinates, which leads to
J ~ ( t ) = ∑ j = 1 n [ ( 1 + μ ) e t j − y j t j ] = ( 1 + μ ) h ( t ) − ⟨ y , t ⟩ . (44) \widetilde{\mathcal{J}}(\mathbf{t}) = \sum_{j=1}^{n}\left[(1+\mu)e^{t_j}-y_jt_j\right] = (1+\mu)h(\mathbf{t})-\langle\mathbf{y},\mathbf{t}\rangle. \tag*{(44)} J ( t ) = j = 1 ∑ n [ ( 1 + μ ) e t j − y j t j ] = ( 1 + μ ) h ( t ) − ⟨ y , t ⟩ . (44)
According to Theorem 10, the generator h h h provides a suitable reference geometry to ensure relative smoothness of the Poisson negative log-likelihood function in the dual coordinate system. Under the choice β j = β = 1 \beta_j=\beta=1 β j = β = 1 , the BPGM iteration (34) can thus be rewritten in this dual domain as
t + = arg min u { ⟨ ∇ λ ( y ; t ) , u − t ⟩ + r ( τ − 1 ( u ) ) + 1 δ d h ( u , t ) } . (45) \mathbf{t}^{+}=\underset{\mathbf{u}}{\operatorname{arg\,min}}\left\{\langle\nabla\lambda(\mathbf{y};\mathbf{t}),\mathbf{u}-\mathbf{t}\rangle+r(\tau^{-1}(\mathbf{u}))+\frac{1}{\delta}d_h(\mathbf{u},\mathbf{t})\right\}. \tag*{(45)} t + = u arg min { ⟨ ∇ λ ( y ; t ) , u − t ⟩ + r ( τ − 1 ( u )) + δ 1 d h ( u , t ) } . (45)
This iteration expressed in the dual domain has been recently proposed by Modrzyk et al. [20 ] as the right Bregman proximal gradient. The first-order optimality condition leads to the following updating rule
e t + = 1 − δ 1 + μ δ e t + δ 1 + μ δ y . e^{\mathbf{t}^{+}}=\frac{1-\delta}{1+\mu\delta}e^{\mathbf{t}}+\frac{\delta}{1+\mu\delta}\mathbf{y}. e t + = 1 + μ δ 1 − δ e t + 1 + μ δ δ y .
or, in the primal domain,
x + = 1 − δ 1 + μ δ x + δ 1 + μ δ y . (46) \mathbf{x}^{+}=\frac{1-\delta}{1+\mu\delta}\mathbf{x}+\frac{\delta}{1+\mu\delta}\mathbf{y}. \tag*{(46)} x + = 1 + μ δ 1 − δ x + 1 + μ δ δ y . (46)
Note that the positivity of the primal variable x \mathbf{x} x is ensured by construction. By using (35) , this update can be rewritten as
x + = x ⋆ + 1 − δ 1 + μ δ ( x − x ⋆ ) \mathbf{x}^{+}=\mathbf{x}^{\star}+\frac{1-\delta}{1+\mu\delta}(\mathbf{x}-\mathbf{x}^{\star}) x + = x ⋆ + 1 + μ δ 1 − δ ( x − x ⋆ )
and, after k k k iterations,
x k = x ⋆ + ( 1 − δ 1 + μ δ ) k ( x 0 − x ⋆ ) . \mathbf{x}^{k}=\mathbf{x}^{\star}+\left(\frac{1-\delta}{1+\mu\delta}\right)^{k}(\mathbf{x}^{0}-\mathbf{x}^{\star}). x k = x ⋆ + ( 1 + μ δ 1 − δ ) k ( x 0 − x ⋆ ) .
Let define
q β = 1 − δ 1 + μ δ q_{\beta}=\frac{1-\delta}{1+\mu\delta} q β = 1 + μ δ 1 − δ
with
0 < δ ≤ 1. (47) 0<\delta\leq1. \tag*{(47)} 0 < δ ≤ 1. (47)
Then
∥ x k − x ⋆ ∥ ∞ ≤ q β k ∥ x 0 − x ⋆ ∥ ∞ . \left\|\mathbf{x}^{k}-\mathbf{x}^{\star}\right\|_{\infty}\leq q_{\beta}^{k}\left\|\mathbf{x}^{0}-\mathbf{x}^{\star}\right\|_{\infty}. x k − x ⋆ ∞ ≤ q β k x 0 − x ⋆ ∞ .
Thus the iteration converges globally and linearly in the primal domain, with exact contraction factor q β q_{\beta} q β . At the largest stepsize δ = 1 \delta=1 δ = 1 allowed by (47) , the contraction factor vanishes and x k = x ⋆ x^k=x^\star x k = x ⋆ , for every k ≥ 1 k\ge1 k ≥ 1 . Thus, when K = I K=I K = I , the BPGM implemented in the dual coordinates reaches the minimizer in a single step. This one-step property follows from the exact geometric matching in (44) : using the dual coordinates t = log x t=\log x t = log x , the nonlinear part of the objective function is driven by h ( t ) h(t) h ( t ) , while the remaining term is affine.
Remark 14. This result also admits a direct interpretation in terms of relative smoothness in the natural coordinates. Indeed, from (44) , we have
∇ 2 J ~ ( t ) = ( 1 + μ ) ∇ 2 h ( t ) . \nabla^2\widetilde{J}(t)=(1+\mu)\nabla^2h(t). ∇ 2 J ( t ) = ( 1 + μ ) ∇ 2 h ( t ) .
Thus, the objective function in the dual coordinate system is both relatively strongly convex and relatively smooth with respect to h h h , with exact constants
μ β = L β = 1 + μ . \mu_\beta=L_\beta=1+\mu. μ β = L β = 1 + μ .
The corresponding relative condition number is therefore
κ β = L β μ β = 1. \kappa_\beta=\frac{L_\beta}{\mu_\beta}=1. κ β = μ β L β = 1.
This perfect relative conditioning reflects the exact geometric matching between the objective and the reference geometry induced by the Bregman generator h h h . As in the α \alpha α -scaled case, this property is not stated after merely applying a generic convergence bound. The contraction factor follows directly from the exact recursion (46) ; the relative-curvature analysis shows, independently, that the one-step behavior corresponds to perfect conditioning in the natural β \beta β -scaled geometry.
Remark 15 (Case of zero-count). For y j = 0 y_j=0 y j = 0 , the BPGM iteration (46) reduces to
x j + = 1 − δ 1 + μ δ x j x_j^+=\frac{1-\delta}{1+\mu\delta}x_j x j + = 1 + μ δ 1 − δ x j
and, after k k k iterations,
x j k = ( 1 − δ 1 + μ δ ) k x j 0 . x_j^k=\left(\frac{1-\delta}{1+\mu\delta}\right)^k x_j^0. x j k = ( 1 + μ δ 1 − δ ) k x j 0 .
Hence, for 0 < δ < 1 0<\delta<1 0 < δ < 1 the iterates remain strictly positive and converge geometrically to the boundary. At the limiting stepsize δ = 1 \delta=1 δ = 1 , the formal primal recursion gives x j + = 0 x_j^+=0 x j + = 0 in a single iteration. However, this value does not correspond to a finite BPGM iterate in the dual domain. Indeed, when y j = 0 y_j=0 y j = 0 and δ = 1 \delta=1 δ = 1 , the corresponding dual subproblem has no finite minimizer: its infimum is approached as t j → − ∞ t_j\to-\infty t j → − ∞ . Since x j = e t j x_j=e^{t_j} x j = e t j , this limiting behavior corresponds to x j → 0 x_j\to0 x j → 0 in the primal domain. Hence, the one-step statement at δ = 1 \delta=1 δ = 1 should be understood as a limiting statement in the primal representation, rather than as an attained finite BPGM iterate in the dual domain.
Case K not identity
When K ≠ I K\ne I K = I , the negative log-likelihood is not separable with respect to the parameter components. This coupling is not compatible with the assumptions required by Theorem 10, which cannot ensure relative smoothness of the negative log-likelihood with respect to the Bregman generator h h h . However, we can show that this generator still provides a suitable reference geometry, as stated in the following lemma which extends the result of Theorem 10 for the Poisson distribution specifically.
Lemma 2. Consider the Bregman representation ℓ ( y ; K x ) = − d f ( y ; K x ) + ϕ f ( y ) \ell(\mathbf{y};\mathbf{Kx})=-d_f(\mathbf{y};\mathbf{Kx})+\phi_f(\mathbf{y}) ℓ ( y ; Kx ) = − d f ( y ; Kx ) + ϕ f ( y ) of the log-likelihood associated with a Poisson linear inverse problem. Let λ ˘ ( y ; t ) \breve{\lambda}(\mathbf{y};\mathbf{t}) λ ˘ ( y ; t ) denote the negative log-likelihood function expressed in the dual coordinates t = log ( x ) \mathbf{t}=\log(\mathbf{x}) t = log ( x ) induced by a component-wise β \beta β -scaled dual Bregman representation, i.e.,
λ ˘ ( y ; t ) = d f ( y ; K e t ) − ϕ f ( y ) . \breve{\lambda}(\mathbf{y};\mathbf{t})=d_f(\mathbf{y};\mathbf{K}e^{\mathbf{t}})-\phi_f(\mathbf{y}). λ ˘ ( y ; t ) = d f ( y ; K e t ) − ϕ f ( y ) . For β = ( β 1 , … , β n ) T \boldsymbol{\beta}=(\beta_1,\ldots,\beta_n)^\mathsf{T} β = ( β 1 , … , β n ) T and β i > 0 \beta_i>0 β i > 0 , let define h ( t ) = ∑ i = 1 n β i e t i h(\mathbf{t})=\sum_{i=1}^{n}\beta_i e^{t_i} h ( t ) = ∑ i = 1 n β i e t i . Set s = ( s 1 , … , s n ) T = K T 1 \mathbf{s}=(s_1,\ldots,s_n)^\mathsf{T}=\mathbf{K}^\mathsf{T}\mathbf{1} s = ( s 1 , … , s n ) T = K T 1 , i.e., s j = ∑ i = 1 m k i , j s_j=\sum_{i=1}^{m}k_{i,j} s j = ∑ i = 1 m k i , j , with s j > 0 s_j>0 s j > 0 for every j = 1 , … , n j=1,\ldots,n j = 1 , … , n . Then the function λ ˘ ( y ; t ) \breve{\lambda}(\mathbf{y};\mathbf{t}) λ ˘ ( y ; t ) is L L L -smooth relative to h h h for every L ≥ L inf L\geq L_{\inf} L ≥ L i n f with
L inf = { 1 when β = s , ∥ K ∥ 1 when β = 1. L_{\inf}=
\begin{cases}
1 & \text{when }\boldsymbol{\beta}=\mathbf{s},\\
\lVert\mathbf{K}\rVert_1 & \text{when }\boldsymbol{\beta}=\mathbf{1}.
\end{cases} L i n f = { 1 ∥ K ∥ 1 when β = s , when β = 1 . Proof. See Appendix § 8 .12
In other words, Theorem 10 still provides a suitable reference geometry for the Poisson distribution, despite the linear coupling of the parameter components. This result can thus lead to two different valid BPGM iteration schemes, depending on the choice of the weights. Whatever this choice, the BPGM update (45) with
∇ λ ˘ ( y ; t ) = e t ⊙ K T ( 1 − y K e t ) \nabla\breve{\lambda}(\mathbf{y};\mathbf{t})=e^{\mathbf{t}}\odot\mathbf{K}^\mathsf{T}\left(\mathbf{1}-\frac{\mathbf{y}}{\mathbf{K}e^{\mathbf{t}}}\right) ∇ λ ˘ ( y ; t ) = e t ⊙ K T ( 1 − K e t y )
writes
e t + = e t ⊙ β − δ K T ( 1 − y K e t ) β + δ μ 1 . e^{\mathbf{t}^{+}}=e^{\mathbf{t}}\odot\frac{\boldsymbol{\beta}-\delta\mathbf{K}^\mathsf{T}\left(\mathbf{1}-\frac{\mathbf{y}}{\mathbf{K}e^{\mathbf{t}}}\right)}{\boldsymbol{\beta}+\delta\mu\mathbf{1}}. e t + = e t ⊙ β + δ μ 1 β − δ K T ( 1 − K e t y ) .
or, in the primal domain,
x + = x ⊙ β − δ ( K T ( 1 − y K x ) ) β + δ μ 1 . (48) \mathbf{x}^{+}=\mathbf{x}\odot\frac{\boldsymbol{\beta}-\delta\left(\mathbf{K}^\mathsf{T}\left(\mathbf{1}-\frac{\mathbf{y}}{\mathbf{Kx}}\right)\right)}{\boldsymbol{\beta}+\delta\mu\mathbf{1}}. \tag*{(48)} x + = x ⊙ β + δ μ 1 β − δ ( K T ( 1 − Kx y ) ) . (48)
This BPGM update is well defined provided that the right-hand side of (48) belongs to ( R + ) n (\mathbb{R}_+)^n ( R + ) n . Interestingly, when μ = 0 \mu=0 μ = 0 (i.e., in the absence of regularization), this update simplifies to
x + = x ⊙ [ 1 − δ β ⊙ K T ( 1 − y K x ) ] . \mathbf{x}^{+}=\mathbf{x}\odot\left[\mathbf{1}-\frac{\delta}{\boldsymbol{\beta}}\odot\mathbf{K}^\mathsf{T}\left(\mathbf{1}-\frac{\mathbf{y}}{\mathbf{Kx}}\right)\right]. x + = x ⊙ [ 1 − β δ ⊙ K T ( 1 − Kx y ) ] .
Now, let assume that this iteration is implemented with the choice β = s \boldsymbol{\beta}=\mathbf{s} β = s and associated canonical stepsize δ = 1 \delta=1 δ = 1 , this iteration then reduces to
x + = x s ⊙ ( K T y K x ) . (49) \mathbf{x}^{+}=\frac{\mathbf{x}}{\mathbf{s}}\odot\left(\mathbf{K}^\mathsf{T}\frac{\mathbf{y}}{\mathbf{Kx}}\right). \tag*{(49)} x + = s x ⊙ ( K T Kx y ) . (49)
A similar update would be have obtained with the alternative choice β = 1 \boldsymbol{\beta}=\mathbf{1} β = 1 and using an anisotropic stepsize, i.e., δ i = 1 s i \delta_i=\frac{1}{s_i} δ i = s i 1 or, equivalently, a diagonal preconditioning matrix
P = diag ( 1 s 1 , … , 1 s n ) . \mathbf{P}=\operatorname{diag}\left(\frac{1}{s_1},\ldots,\frac{1}{s_n}\right). P = diag ( s 1 1 , … , s n 1 ) .
With both choices, the step (49) coincides with the expectation-maximization (EM) iteration [24 ] , or equivalently with the Richardson-Lucy iteration [19 , 22 ] , for Poisson data. Thus, the adopted β \beta β -scaled dual geometry turns BPGM expressed in the dual domain into a multiplicative iterative scheme in the primal domain and recovers the EM/Richardson-Lucy iteration. Connections between EM and Bregman or mirror-descent methods have also been investigated, for instance, by Kunstner et al. [17 ] and Clerc et al. [11 ] . It is also interesting to compare this construction with the so-called right BPGM proposed by Modrzyk et al. [20 ] and applied to Poisson linear inverse problems. In their approach, the vector s = K T 1 \mathbf{s}=\mathbf{K}^{\mathsf{T}}\mathbf{1} s = K T 1 also serves to define a weighted reference (negative-entropy) geometry, akin to the separable construction of β \beta β -scaled models proposed in Theorem 10. Their approach leads, in the unregularized Poisson linear inverse problem, to
x k + 1 = ( 1 − δ ) x k + δ x k s ⊙ K T y K x k \mathbf{x}^{k+1}=(1-\delta)\mathbf{x}^{k}+\delta\frac{\mathbf{x}^{k}}{\mathbf{s}}\odot\frac{\mathbf{K}^{\mathsf{T}}\mathbf{y}}{\mathbf{K}\mathbf{x}^{k}} x k + 1 = ( 1 − δ ) x k + δ s x k ⊙ K x k K T y
which reduces to the EM/Richardson-Lucy iteration for δ = 1 \delta=1 δ = 1 . Hence, both approaches recover the same EM/Richardson-Lucy iteration, despite different starting points. Modrzyk et al. [20 ] devise a new variant of BPGM to solve the problem (33) in the primal domain and leverages the Legendre duality to reformulate the iterations with respect to the dual variables. On the contrary, we apply the canonical instance of BPGM to solve the problem (33) in the dual domain induced by the β \beta β -scaled Bregman representation.
Summary of the case K = I \mathbf{K}=\mathbf{I} K = I
For K = I \mathbf{K}=\mathbf{I} K = I and y > 0 \mathbf{y}>0 y > 0 , the two derivations of BPGM iterations associated with the two geometries lead to explicit iterations with distinct convergence properties. Adopting the geometry induced by the α \alpha α -scaled dual Bregman representation, BPGM yields the Burg/Itakura-Saito iterations in the primal domain. Introducing the reciprocal variable v j = 1 / x j v_j=1/x_j v j = 1/ x j yields the exact affine error recursion
v j + − v j ∗ = ( 1 − δ y j ) ( v j − v j ∗ ) . v_j^+-v_j^*=(1-\delta y_j)(v_j-v_j^*). v j + − v j ∗ = ( 1 − δ y j ) ( v j − v j ∗ ) .
Hence, for 0 < δ ≤ 1 / y max 0<\delta\le1/y_{\max} 0 < δ ≤ 1/ y m a x , the iteration is globally linearly convergent in reciprocal coordinates, with exact worst-case coordinate-wise contraction factor
q α = 1 − δ y min . q_\alpha=1-\delta y_{\min}. q α = 1 − δ y m i n .
At the largest simple globally positive stepsize δ = 1 / y max \delta=1/y_{\max} δ = 1/ y m a x , this becomes
q α = 1 − y min y max . q_\alpha=1-\frac{y_{\min}}{y_{\max}}. q α = 1 − y m a x y m i n .
This exact rate is consistent with the relative geometry of the identity problem, for which the relative strong-convexity and smoothness constants are respectively y min y_{\min} y m i n and y max y_{\max} y m a x . Adopting the geometry induced by the β \beta β -scaled dual Bregman representation leads to a BPGM iteration in dual coordinates. The objective function is rewritten as
J ~ ( t ) = ( 1 + μ ) h ( t ) − ⟨ y , t ⟩ \widetilde{\mathcal{J}}(\mathbf{t})=(1+\mu)h(\mathbf{t})-\langle\mathbf{y},\mathbf{t}\rangle J ( t ) = ( 1 + μ ) h ( t ) − ⟨ y , t ⟩
where the nonlinear part of this objective function matches the Bregman generator h ( t ) = ∑ j = 1 n e t j h(\mathbf{t})=\sum_{j=1}^{n}e^{t_j} h ( t ) = ∑ j = 1 n e t j of the β \beta β -scaled dual representation exactly: the objective function and the reference function share the same geometry, since
∇ 2 J ~ ( t ) = ( 1 + μ ) ∇ 2 h ( t ) . \nabla^2\widetilde{\mathcal{J}}(\mathbf{t})=(1+\mu)\nabla^2h(\mathbf{t}). ∇ 2 J ( t ) = ( 1 + μ ) ∇ 2 h ( t ) .
The resulting BPGM can be mapped back to the original variable of interest,
x + = 1 − δ 1 + μ δ x + + δ 1 + μ δ y , \mathbf{x}^+=\frac{1-\delta}{1+\mu\delta}\mathbf{x}^+ + \frac{\delta}{1+\mu\delta}\mathbf{y}, x + = 1 + μ δ 1 − δ x + + 1 + μ δ δ y ,
and satisfies the exact global error recursion
x + − x ⋆ = 1 − δ 1 + μ δ ( x − x ⋆ ) . \mathbf{x}^{+}-\mathbf{x}^{\star}=\frac{1-\delta}{1+\mu\delta}(\mathbf{x}-\mathbf{x}^{\star}). x + − x ⋆ = 1 + μ δ 1 − δ ( x − x ⋆ ) .
At the canonical stepsize δ = 1 \delta=1 δ = 1 , the contraction factor vanishes and the minimizer is reached in a single iteration:
x + = y 1 + μ = x ⋆ . \mathbf{x}^{+}=\frac{\mathbf{y}}{1+\mu}=\mathbf{x}^{\star}. x + = 1 + μ y = x ⋆ .
The identity case shows that the two geometries induced by the proposed dual Bregman representations provide substantially different convergence dynamics and that a given geometry does not determine a unique algorithmic implementation. In particular, the one-step convergence property of the β \beta β -scaled construction results from the derivation of BPGM in the dual coordinates together with the exact matching of the objective and the reference function. Table 2 summarizes these findings, highlighting the distinct updates, convergence properties, and limiting-step behavior of the BPGM in the α \alpha α -scaled and β \beta β -scaled geometries. This table also specifies the particular behaviors of the two algorithms when the observations lie on the boundary of the corresponding parameter space, i.e., y i = 0 y_i=0 y i = 0 .
Table 2. Summary of the main convergence properties of the BPGM iterations implemented in the α \alpha α -scaled and β \beta β -scaled geometries.
Conclusion
In this paper, we developed a geometric framework for devising dual Bregman representations of distributions from regular exponential families. Building upon the primal Bregman representation of these distributions, we showed that the arguments involved in the Bregman divergence can be swapped after passing to a new coordinate system. Building on a projective characterization of such representations, we established a general construction of the dual coordinates and their associated Bregman generators, together with normalization and uniqueness results. We also identified two particular forms of admissible dual representations. One of them, referred to as the β \beta β -scaled dual Bregman representation, is directly induced by Legendre duality, with the gradient of the density generator defining the dual coordinates and its scaled Legendre conjugate defining the corresponding Bregman geometry. A second family of models, referred to as the α \alpha α -scaled dual Bregman representation, adopts explicit yet distinct coordinates and Bregman geometry. Both constructions arise as particular instances of the general projective framework. One particular application of these representations was the derivation of alternative forms of conjugate priors within a Bayesian inference context. The proposed framework was also instantiated on several classical statistical models, showing that different exponential-family models may share the same Bregman geometry, up to a positive scaling.
Beyond the construction of these dual Bregman representations, we established a direct connection between these representations and relative smoothness. Precisely, the Bregman geometries induced by these representations provide natural reference geometries for the associated negative log-density, with explicit relative smoothness constants determined by the conjugate scaling. In the scalar and separable settings, the corresponding constants are sharp.
These results connect statistical modeling, convex duality, and Bregman-based regularity within a common geometric framework. Beyond their theoretical interest, the geometries induced by these representations may provide useful reference structures for the development and analysis of Bregman- and mirror-based computational methods for exponential-family models. This was illustrated by addressing Poisson linear inverse problems. The derivation of Bregman proximal gradient methods under the two geometries induced by the dual representations recovers popular algorithms from the literature, shedding new light on their constructions and properties.
Several directions deserve further investigation. Extending the present framework beyond regular exponential families and exploiting the induced geometries for the design and analysis of Bayesian computational algorithms constitute promising directions for future research.
Acknowledgements
The authors would like to thank Pr. Ismaël Castillo (LPSM, Sorbonne Université, France) for his helpful comments and suggestions, and Dr. Emmanuel Soubies (IRIT, Univ. Toulouse, France) for stimulating discussions regarding this work and for pointing out the reference [20 ] .
M. D. Fall acknowledges partial financial support from the BACKUP project (ANR-23-CE40-0018-01). N. Dobigeon acknowledges partial financial support from the Artificial and Natural Intelligence Toulouse Institute (ANITI), funded by the France 2030 program under the grant agreement ANR-23-IACL-0002.
Proofs
Proof of Theorem 2
We establish two elementary identities before proving the theorem.
Lemma 3 (Projective Jacobian). Let X ⊂ R n \mathcal{X} \subset\mathbb{R}^{n} X ⊂ R n be open, and let g ( x ) = c T x + d > 0 g(\mathbf{x}) = \mathbf{c}^{\mathsf{T}}\mathbf{x} + d > 0 g ( x ) = c T x + d > 0 and ξ ( x ) = ( A x + b ) / g ( x ) \boldsymbol{\xi}(\mathbf{x}) = (A\mathbf{x} + \mathbf{b})/g(\mathbf{x}) ξ ( x ) = ( A x + b ) / g ( x ) , where M = ( A b c T d ) M = \begin{pmatrix} A & \mathbf{b} \\ \mathbf{c}^{\mathsf{T}} & d \end{pmatrix} M = ( A c T b d ) . Then
J ξ ( x ) = A − ξ ( x ) c T g ( x ) , det J ξ ( x ) = det M g ( x ) n + 1 . (50) J_{\boldsymbol{\xi}}(\mathbf{x}) = \frac{A - \boldsymbol{\xi}(\mathbf{x})\mathbf{c}^{\mathsf{T}}}{g(\mathbf{x})}, \qquad\det J_{\boldsymbol{\xi}}(\mathbf{x}) = \frac{\det M}{g(\mathbf{x})^{n+1}}. \tag*{(50)} J ξ ( x ) = g ( x ) A − ξ ( x ) c T , det J ξ ( x ) = g ( x ) n + 1 det M . (50) If det M ≠ 0 \det M \ne0 det M = 0 , ξ \boldsymbol{\xi} ξ is a smooth diffeomorphism onto its open image.
Proof. The first identity is the quotient rule. Right-multiplication of M M M by ( I x 0 1 ) \begin{pmatrix} I & \mathbf{x} \\ 0 & 1 \end{pmatrix} ( I 0 x 1 ) does not change its determinant, and produces the last column ( g ( x ) ξ ( x ) , g ( x ) ) T (g(\mathbf{x})\boldsymbol{\xi}(\mathbf{x}),g(\mathbf{x}))^{\mathsf{T}} ( g ( x ) ξ ( x ) , g ( x ) ) T . Subtracting ξ ( x ) \boldsymbol{\xi}(\mathbf{x}) ξ ( x ) times the final row from the first n n n rows gives
det M = g ( x ) det ( A − ξ ( x ) c T ) , \det M = g(\mathbf{x}) \det(A - \boldsymbol{\xi}(\mathbf{x})\mathbf{c}^{\mathsf{T}}), det M = g ( x ) det ( A − ξ ( x ) c T ) , which proves (50) . If ξ ( x ) = ξ ( y ) = v \boldsymbol{\xi}(\mathbf{x}) = \boldsymbol{\xi}(\mathbf{y}) = \mathbf{v} ξ ( x ) = ξ ( y ) = v , then M ( x , 1 ) T = g ( x ) ( v , 1 ) T M(\mathbf{x},1)^{\mathsf{T}} = g(\mathbf{x})(\mathbf{v},1)^{\mathsf{T}} M ( x , 1 ) T = g ( x ) ( v , 1 ) T and likewise for y \mathbf{y} y . Invertibility of M M M implies that ( x , 1 ) T (\mathbf{x},1)^{\mathsf{T}} ( x , 1 ) T and ( y , 1 ) T (\mathbf{y},1)^{\mathsf{T}} ( y , 1 ) T are proportional; comparison of the last entries yields x = y \mathbf{x} = \mathbf{y} x = y . The inverse function theorem and injectivity give the asserted smooth inverse on the image.
Lemma 4 (Perspective–Bregman identity). Suppose X , T ⊂ R n \mathcal{X}, \mathcal{T} \subset\mathbb{R}^{n} X , T ⊂ R n are open convex sets, h ∈ C 2 ( T ) h \in C^{2}(\mathcal{T}) h ∈ C 2 ( T ) has positive-definite Hessian, g > 0 g > 0 g > 0 and g ξ g\boldsymbol{\xi} g ξ are affine on X \mathcal{X} X , and ξ ( X ) ⊂ ∇ h ( T ) \boldsymbol{\xi}(\mathcal{X}) \subset\nabla h(\mathcal{T}) ξ ( X ) ⊂ ∇ h ( T ) . Set τ = ( ∇ h ) − 1 ∘ ξ \tau= (\nabla h)^{-1} \circ\boldsymbol{\xi} τ = ( ∇ h ) − 1 ∘ ξ and ρ = g h ∗ ∘ ξ \rho= gh^{*} \circ\boldsymbol{\xi} ρ = g h ∗ ∘ ξ . Then ρ ∈ C 2 ( X ) \rho\in C^{2}(\mathcal{X}) ρ ∈ C 2 ( X ) and
d ρ ( y , x ) = g ( y ) d h ( τ ( x ) , τ ( y ) ) . (51) d_{\rho}(\mathbf{y},\mathbf{x}) = g(\mathbf{y})d_{h}(\tau(\mathbf{x}),\tau(\mathbf{y})). \tag*{(51)} d ρ ( y , x ) = g ( y ) d h ( τ ( x ) , τ ( y )) . (51) Proof. Because T \mathcal{T} T is convex and ∇ 2 h ≻ 0 \nabla^{2}h \succ0 ∇ 2 h ≻ 0 , the gradient ∇ h \nabla h ∇ h is injective with everywhere invertible derivative and hence a diffeomorphism onto its image. For t ∈ T \mathbf{t} \in\mathcal{T} t ∈ T , the supporting-hyperplane inequality implies that the supremum defining h ∗ h^{*} h ∗ at v = ∇ h ( t ) \mathbf{v} = \nabla h(\mathbf{t}) v = ∇ h ( t ) is attained at t \mathbf{t} t . Consequently, on ∇ h ( T ) \nabla h(\mathcal{T}) ∇ h ( T ) ,
h ∗ ( v ) = v T t − h ( t ) , ∇ h ∗ ( v ) = t . h^{*}(\mathbf{v}) = \mathbf{v}^{\mathsf{T}}\mathbf{t} - h(\mathbf{t}), \qquad\nabla h^{*}(\mathbf{v}) = \mathbf{t}. h ∗ ( v ) = v T t − h ( t ) , ∇ h ∗ ( v ) = t . These local conjugacy relations require no essential smoothness. Write g ξ = A x + b g\boldsymbol{\xi} = A\mathbf{x} + \mathbf{b} g ξ = A x + b and ∇ g = c \nabla g = \mathbf{c} ∇ g = c . Differentiating ρ \rho ρ and using Fenchel equality,
∇ ρ = c h ∗ ( ξ ) + g J ξ T τ = c ( ξ T τ − h ( τ ) ) + ( A − ξ c T ) T τ = A T τ − c h ( τ ) . \begin{aligned}
\nabla\rho&= \mathbf{c}h^{*}(\boldsymbol{\xi}) + gJ_{\boldsymbol{\xi}}^{\mathsf{T}}\tau\\
&= \mathbf{c}(\boldsymbol{\xi}^{\mathsf{T}}\tau- h(\tau)) + (A - \boldsymbol{\xi}\mathbf{c}^{\mathsf{T}})^{\mathsf{T}}\tau\\
&= A^{\mathsf{T}}\tau- \mathbf{c}h(\tau).
\end{aligned} ∇ ρ = c h ∗ ( ξ ) + g J ξ T τ = c ( ξ T τ − h ( τ )) + ( A − ξ c T ) T τ = A T τ − c h ( τ ) . The affine relations imply A ( y − x ) = g ( y ) ξ ( y ) − g ( x ) ξ ( x ) A(\mathbf{y} - \mathbf{x}) = g(\mathbf{y})\boldsymbol{\xi}(\mathbf{y}) - g(\mathbf{x})\boldsymbol{\xi}(\mathbf{x}) A ( y − x ) = g ( y ) ξ ( y ) − g ( x ) ξ ( x ) and c T ( y − x ) = g ( y ) − g ( x ) \mathbf{c}^{\mathsf{T}}(\mathbf{y} - \mathbf{x}) = g(\mathbf{y}) - g(\mathbf{x}) c T ( y − x ) = g ( y ) − g ( x ) . Substitution into the definition of d ρ d_{\rho} d ρ gives
d ρ ( y , x ) = g ( y ) [ h ∗ ( ξ ( y ) ) − h ∗ ( ξ ( x ) ) − τ ( x ) T ( ξ ( y ) − ξ ( x ) ) ] = g ( y ) d h ∗ ( ξ ( y ) , ξ ( x ) ) = g ( y ) d h ( τ ( x ) , τ ( y ) ) , \begin{aligned}
d_{\rho}(\mathbf{y},\mathbf{x}) &= g(\mathbf{y})[h^{*}(\boldsymbol{\xi}(\mathbf{y})) - h^{*}(\boldsymbol{\xi}(\mathbf{x})) - \tau(\mathbf{x})^{\mathsf{T}}(\boldsymbol{\xi}(\mathbf{y}) - \boldsymbol{\xi}(\mathbf{x}))] \\
&= g(\mathbf{y})d_{h^{*}}(\boldsymbol{\xi}(\mathbf{y}),\boldsymbol{\xi}(\mathbf{x})) \\
&= g(\mathbf{y})d_{h}(\tau(\mathbf{x}),\tau(\mathbf{y})),
\end{aligned} d ρ ( y , x ) = g ( y ) [ h ∗ ( ξ ( y )) − h ∗ ( ξ ( x )) − τ ( x ) T ( ξ ( y ) − ξ ( x ))] = g ( y ) d h ∗ ( ξ ( y ) , ξ ( x )) = g ( y ) d h ( τ ( x ) , τ ( y )) , where the last equality follows by expanding the two divergences and applying Fenchel equality at both points.
We are now ready to prove Theorem 2.
Necessity — Differentiating (5) with respect to x x x while holding y y y fixed yields
∇ 2 f ( x ) ( y − x ) = g ( y ) J τ ( x ) T [ ξ ( y ) − ξ ( x ) ] . (52) \nabla^2 f(\mathbf{x})(\mathbf{y}-\mathbf{x})=g(\mathbf{y})J_{\tau}(\mathbf{x})^{\mathsf{T}}[\boldsymbol{\xi}(\mathbf{y})-\boldsymbol{\xi}(\mathbf{x})]. \tag*{(52)} ∇ 2 f ( x ) ( y − x ) = g ( y ) J τ ( x ) T [ ξ ( y ) − ξ ( x )] . (52)
This step does not differentiate g g g . As J τ J_{\tau} J τ is invertible, throughout the proof, define K ( x ) = J τ ( x ) − T ∇ 2 f ( x ) \mathbf{K}(\mathbf{x})=J_{\tau}(\mathbf{x})^{-\mathsf{T}}\nabla^2 f(\mathbf{x}) K ( x ) = J τ ( x ) − T ∇ 2 f ( x ) . Then
g ( y ) [ ξ ( y ) − ξ ( x ) ] = K ( x ) ( y − x ) . (53) g(\mathbf{y})[\boldsymbol{\xi}(\mathbf{y})-\boldsymbol{\xi}(\mathbf{x})]=\mathbf{K}(\mathbf{x})(\mathbf{y}-\mathbf{x}). \tag*{(53)} g ( y ) [ ξ ( y ) − ξ ( x )] = K ( x ) ( y − x ) . (53)
Strict convexity of differentiable h h h on the convex set T \mathcal{T} T implies injectivity of ∇ h \nabla h ∇ h : the strict supporting-hyperplane inequalities at two distinct points imply strict monotonicity of its gradient. Hence ξ = ∇ h ∘ τ \boldsymbol{\xi}=\nabla h\circ\tau ξ = ∇ h ∘ τ is injective. Choose x 1 ≠ x 2 \mathbf{x}_1\ne\mathbf{x}_2 x 1 = x 2 . Subtracting (53) at these base points gives
g ( y ) [ ξ ( x 2 ) − ξ ( x 1 ) ] = K ( x 1 ) ( y − x 1 ) − K ( x 2 ) ( y − x 2 ) . (54) g(\mathbf{y})[\boldsymbol{\xi}(\mathbf{x}_2)-\boldsymbol{\xi}(\mathbf{x}_1)]=\mathbf{K}(\mathbf{x}_1)(\mathbf{y}-\mathbf{x}_1)-\mathbf{K}(\mathbf{x}_2)(\mathbf{y}-\mathbf{x}_2). \tag*{(54)} g ( y ) [ ξ ( x 2 ) − ξ ( x 1 )] = K ( x 1 ) ( y − x 1 ) − K ( x 2 ) ( y − x 2 ) . (54)
The right-hand side is affine in y \mathbf{y} y , and the fixed vector multiplying g ( y ) g(\mathbf{y}) g ( y ) is nonzero. Taking a nonzero component shows that g ( y ) = c T y + d g(\mathbf{y})=\mathbf{c}^{\mathsf{T}}\mathbf{y}+d g ( y ) = c T y + d is affine. Returning to (53) with x \mathbf{x} x fixed shows that g ( y ) ξ ( y ) = A y + b g(\mathbf{y})\boldsymbol{\xi}(\mathbf{y})=\mathbf{A}\mathbf{y}+\mathbf{b} g ( y ) ξ ( y ) = Ay + b is affine as well.
To establish nonsingularity of M \mathbf{M} M , suppose instead that it is singular. There is a nonzero ( u , v ) (\mathbf{u},v) ( u , v ) with u T A + v c T = 0 T \mathbf{u}^{\mathsf{T}}\mathbf{A}+v\mathbf{c}^{\mathsf{T}}=\mathbf{0}^{\mathsf{T}} u T A + v c T = 0 T and u T b + v d = 0 \mathbf{u}^{\mathsf{T}}\mathbf{b}+vd=0 u T b + v d = 0 . Since g > 0 g>0 g > 0 , necessarily u ≠ 0 \mathbf{u}\ne\mathbf{0} u = 0 . It follows that u T ξ ( x ) + v = 0 \mathbf{u}^{\mathsf{T}}\boldsymbol{\xi}(\mathbf{x})+v=0 u T ξ ( x ) + v = 0 for all x \mathbf{x} x . Thus the image of ξ \boldsymbol{\xi} ξ lies in a proper affine hyperplane, contradicting invariance of domain, because ξ \boldsymbol{\xi} ξ is a continuous injection on the open set X ⊂ R n \mathcal{X}\subset\mathbb{R}^n X ⊂ R n . Therefore det M ≠ 0 \det\mathbf{M}\ne0 det M = 0 .
Lemma 3 now shows that J ξ J_{\xi} J ξ is invertible. From J ξ ( x ) = ∇ 2 h ( τ ( x ) ) J τ ( x ) J_{\xi}(\mathbf{x})=\nabla^2h(\tau(\mathbf{x}))J_{\tau}(\mathbf{x}) J ξ ( x ) = ∇ 2 h ( τ ( x )) J τ ( x ) and invertibility of J τ J_{\tau} J τ , the Hessian of h h h is invertible at every point of T \mathcal{T} T . It is positive semidefinite by convexity, and therefore positive definite. Since g g g is now known to be affine, we may differentiate (52) with respect to y \mathbf{y} y and set y = x \mathbf{y}=\mathbf{x} y = x . The derivative of g g g multiplies a vanishing difference on the diagonal, giving
∇ 2 f ( x ) = g ( x ) J τ ( x ) T ∇ 2 h ( τ ( x ) ) J τ ( x ) ≻ 0. \nabla^2 f(\mathbf{x})=g(\mathbf{x})J_{\tau}(\mathbf{x})^{\mathsf{T}}\nabla^2h(\tau(\mathbf{x}))J_{\tau}(\mathbf{x})\succ0. ∇ 2 f ( x ) = g ( x ) J τ ( x ) T ∇ 2 h ( τ ( x )) J τ ( x ) ≻ 0.
Finally, Lemma 4 applies to ρ = g h ∗ ∘ ξ \rho=gh^*\circ\boldsymbol{\xi} ρ = g h ∗ ∘ ξ and yields d ρ = d f d\rho=df d ρ = df . Differentiating this equality with respect to y \mathbf{y} y gives ∇ ( f − ρ ) ( y ) = ∇ ( f − ρ ) ( x ) \nabla(f-\rho)(\mathbf{y})=\nabla(f-\rho)(\mathbf{x}) ∇ ( f − ρ ) ( y ) = ∇ ( f − ρ ) ( x ) for all points. Hence f − ρ f-\rho f − ρ is affine on the connected set X \mathcal{X} X , proving (7) .
Sufficiency — Conversely, suppose (6) –(7) hold under the initial hypotheses. Lemma 3 and the invertibility of J τ J_{\tau} J τ imply that ∇ 2 h \nabla^2h ∇ 2 h is positive definite. The perspective identity then gives d g h ∗ ∘ ξ = g d h d_{gh^*\circ\xi}=g\,dh d g h ∗ ∘ ξ = g d h ; the affine term in (7) has zero Bregman divergence. This proves (5) .
Proof of Theorem 3
i) Normalization. Let ( h 0 , τ 0 ) (h_0,\tau_0) ( h 0 , τ 0 ) be an admissible representation associated with the projective matrix
M = ( A b c T d ) , M − T = ( P q r T s ) . \mathbf{M}=\begin{pmatrix}\mathbf{A}&\mathbf{b}\\ \mathbf{c}^{\mathsf{T}}&d\end{pmatrix},\qquad
\mathbf{M}^{-\mathsf{T}}=\begin{pmatrix}\mathbf{P}&\mathbf{q}\\ \mathbf{r}^{\mathsf{T}}&s\end{pmatrix}. M = ( A c T b d ) , M − T = ( P r T q s ) .
By Theorem 2, there exist ζ ∈ R n \boldsymbol{\zeta}\in\mathbb{R}^n ζ ∈ R n and κ ∈ R \kappa\in\mathbb{R} κ ∈ R such that
f ( x ) = g ( x ) h 0 ∗ ( ξ ( x ) ) + ζ T x + κ (55) f(\mathbf{x})=g(\mathbf{x})h_0^*(\boldsymbol{\xi}(\mathbf{x}))+\boldsymbol{\zeta}^{\mathsf{T}}\mathbf{x}+\kappa \tag*{(55)} f ( x ) = g ( x ) h 0 ∗ ( ξ ( x )) + ζ T x + κ (55)
where
g ( x ) = c T x + d , ξ ( x ) = A x + b g ( x ) . g(\mathbf{x})=\mathbf{c}^{\mathsf{T}}\mathbf{x}+d,\qquad
\boldsymbol{\xi}(\mathbf{x})=\frac{\mathbf{A}\mathbf{x}+\mathbf{b}}{g(\mathbf{x})}. g ( x ) = c T x + d , ξ ( x ) = g ( x ) Ax + b .
Define
a = P ζ + q κ , κ 1 = r T ζ + s κ . \mathbf{a}=\mathbf{P}\zeta+q\kappa,\qquad\kappa_1=\mathbf{r}^{\mathsf{T}}\zeta+s\kappa. a = P ζ + q κ , κ 1 = r T ζ + s κ .
and consider
τ ~ ( x ) = τ 0 ( x ) + a , h ~ ( t ) = h 0 ( t − a ) − κ 1 . \widetilde{\tau}(\mathbf{x})=\tau_0(\mathbf{x})+\mathbf{a},\qquad\widetilde{h}(\mathbf{t})=h_0(\mathbf{t}-\mathbf{a})-\kappa_1. τ ( x ) = τ 0 ( x ) + a , h ( t ) = h 0 ( t − a ) − κ 1 .
The Fenchel conjugate of h ~ \widetilde{h} h satisfies
h ~ ∗ ( u ) = h 0 ∗ ( u ) + ⟨ a , u ⟩ + κ 1 . (56) \widetilde{h}^{*}(\mathbf{u})=h_0^{*}(\mathbf{u})+\langle\mathbf{a},\mathbf{u}\rangle+\kappa_1. \tag*{(56)} h ∗ ( u ) = h 0 ∗ ( u ) + ⟨ a , u ⟩ + κ 1 . (56)
Since
( a κ 1 ) = M − T ( ζ κ ) , \begin{pmatrix}\mathbf{a}\\\kappa_1\end{pmatrix}
=\mathbf{M}^{-\mathsf{T}}
\begin{pmatrix}\zeta\\\kappa\end{pmatrix}, ( a κ 1 ) = M − T ( ζ κ ) ,
we have
( ζ κ ) = M T ( a κ 1 ) . \begin{pmatrix}\zeta\\\kappa\end{pmatrix}
=\mathbf{M}^{\mathsf{T}}
\begin{pmatrix}\mathbf{a}\\\kappa_1\end{pmatrix}. ( ζ κ ) = M T ( a κ 1 ) .
Consequently,
ζ = A T a + c κ 1 , κ = b T a + d κ 1 . \zeta=\mathbf{A}^{\mathsf{T}}\mathbf{a}+c\kappa_1,\qquad\kappa=\mathbf{b}^{\mathsf{T}}\mathbf{a}+d\kappa_1. ζ = A T a + c κ 1 , κ = b T a + d κ 1 .
It follows that
g ( x ) ( ⟨ a , ξ ( x ) ⟩ + κ 1 ) = ⟨ a , A x + b ⟩ + κ 1 ( c T x + d ) = ζ T x + κ . g(\mathbf{x})(\langle\mathbf{a},\xi(\mathbf{x})\rangle+\kappa_1)
=\langle\mathbf{a},\mathbf{A}\mathbf{x}+\mathbf{b}\rangle+\kappa_1(\mathbf{c}^{\mathsf{T}}\mathbf{x}+d)
=\zeta^{\mathsf{T}}\mathbf{x}+\kappa. g ( x ) (⟨ a , ξ ( x )⟩ + κ 1 ) = ⟨ a , Ax + b ⟩ + κ 1 ( c T x + d ) = ζ T x + κ .
Combining this identity with (55) and (56) yields
f ( x ) = g ( x ) h ~ ∗ ( ξ ( x ) ) . f(\mathbf{x})=g(\mathbf{x})\widetilde{h}^{*}(\xi(\mathbf{x})). f ( x ) = g ( x ) h ∗ ( ξ ( x )) .
Thus the affine terms in the projective representation have been eliminated. Moreover,
∇ h ~ ( τ ~ ( x ) ) = ∇ h 0 ( τ 0 ( x ) ) = ξ ( x ) \nabla\widetilde{h}(\widetilde{\tau}(\mathbf{x}))=\nabla h_0(\tau_0(\mathbf{x}))=\xi(\mathbf{x}) ∇ h ( τ ( x )) = ∇ h 0 ( τ 0 ( x )) = ξ ( x )
so the projective matrix and the map ξ \xi ξ remain unchanged. Finally, translations of the argument and additive constants do not affect Bregman divergences. Therefore,
d h ~ ( τ ~ ( x ) , τ ~ ( y ) ) = d h 0 ( τ 0 ( x ) , τ 0 ( y ) ) . d_{\widetilde{h}}(\widetilde{\tau}(\mathbf{x}),\widetilde{\tau}(\mathbf{y}))
=d_{h_0}(\tau_0(\mathbf{x}),\tau_0(\mathbf{y})). d h ( τ ( x ) , τ ( y )) = d h 0 ( τ 0 ( x ) , τ 0 ( y )) .
This establishes the normalization property.
ii) Uniqueness up to affine equivalence. Let ( h , τ ) (h,\tau) ( h , τ ) and ( h 0 , τ 0 ) (h_0,\tau_0) ( h 0 , τ 0 ) be two admissible representations associated with the same weight g ( x ) = c T x + d g(\mathbf{x})=\mathbf{c}^{\mathsf{T}}\mathbf{x}+d g ( x ) = c T x + d . By part i) , we may first work with their normalised forms, for which the affine terms ζ \zeta ζ and κ \kappa κ vanish. Let M \mathbf{M} M and M 0 \mathbf{M}_0 M 0 denote the corresponding projective matrices. Since the two representations have the same weight g g g on the open set X \mathcal{X} X , their matrices have the same last row ( c T , d ) (\mathbf{c}^{\mathsf{T}},d) ( c T , d ) . It follows that
M M 0 − 1 = ( G w 0 T 1 ) \mathbf{M}\mathbf{M}_0^{-1}
=
\begin{pmatrix}
\mathbf{G} & \mathbf{w}\\
\mathbf{0}^{\mathsf{T}} & 1
\end{pmatrix} M M 0 − 1 = ( G 0 T w 1 )
for some G ∈ R n × n \mathbf{G}\in\mathbb{R}^{n\times n} G ∈ R n × n and w ∈ R n \mathbf{w}\in\mathbb{R}^{n} w ∈ R n . Moreover,
det G = det M det M 0 ≠ 0 , \det\mathbf{G}=\frac{\det\mathbf{M}}{\det\mathbf{M}_0}\ne0, det G = det M 0 det M = 0 ,
so G G G is invertible. By Proposition 4, the two normalised representations satisfy
( τ ( x ) − h ( τ ( x ) ) ) = M − T ( ∇ f ( x ) − F ( x ) ) \begin{pmatrix} \tau(\mathbf{x}) \\ -h(\tau(\mathbf{x})) \end{pmatrix}
=
\mathbf{M}^{-\mathsf{T}}
\begin{pmatrix} \nabla f(\mathbf{x}) \\ -F(\mathbf{x}) \end{pmatrix} ( τ ( x ) − h ( τ ( x )) ) = M − T ( ∇ f ( x ) − F ( x ) )
and
( τ 0 ( x ) − h 0 ( τ 0 ( x ) ) ) = M 0 − T ( ∇ f ( x ) − F ( x ) ) . \begin{pmatrix} \tau_0(\mathbf{x}) \\ -h_0(\tau_0(\mathbf{x})) \end{pmatrix}
=
\mathbf{M}_0^{-\mathsf{T}}
\begin{pmatrix} \nabla f(\mathbf{x}) \\ -F(\mathbf{x}) \end{pmatrix}. ( τ 0 ( x ) − h 0 ( τ 0 ( x )) ) = M 0 − T ( ∇ f ( x ) − F ( x ) ) .
Consequently,
( τ ( x ) − h ( τ ( x ) ) ) = ( M M 0 − 1 ) − T ( τ 0 ( x ) − h 0 ( τ 0 ( x ) ) ) . \begin{pmatrix} \tau(\mathbf{x}) \\ -h(\tau(\mathbf{x})) \end{pmatrix}
=
(\mathbf{M}\mathbf{M}_0^{-1})^{-\mathsf{T}}
\begin{pmatrix} \tau_0(\mathbf{x}) \\ -h_0(\tau_0(\mathbf{x})) \end{pmatrix}. ( τ ( x ) − h ( τ ( x )) ) = ( M M 0 − 1 ) − T ( τ 0 ( x ) − h 0 ( τ 0 ( x )) ) .
A direct computation gives
( M M 0 − 1 ) − T = ( G − T 0 − w T G − T 1 ) . (\mathbf{M}\mathbf{M}_0^{-1})^{-\mathsf{T}}
=
\begin{pmatrix}
\mathbf{G}^{-\mathsf{T}} & 0 \\
-\mathbf{w}^{\mathsf{T}}\mathbf{G}^{-\mathsf{T}} & 1
\end{pmatrix}. ( M M 0 − 1 ) − T = ( G − T − w T G − T 0 1 ) .
Hence
τ ( x ) = G − T τ 0 ( x ) (57) \tau(\mathbf{x})=\mathbf{G}^{-\mathsf{T}}\tau_0(\mathbf{x}) \tag*{(57)} τ ( x ) = G − T τ 0 ( x ) (57)
and
h ( τ ( x ) ) = h 0 ( τ 0 ( x ) ) + ⟨ w , τ ( x ) ⟩ . h(\tau(\mathbf{x}))=h_0(\tau_0(\mathbf{x}))+\langle\mathbf{w},\tau(\mathbf{x})\rangle. h ( τ ( x )) = h 0 ( τ 0 ( x )) + ⟨ w , τ ( x )⟩ .
Since τ 0 ( x ) = G T τ ( x ) \tau_0(\mathbf{x})=\mathbf{G}^{\mathsf{T}}\tau(\mathbf{x}) τ 0 ( x ) = G T τ ( x ) , we obtain
h ( t ) = h 0 ( G T t ) + ⟨ w , t ⟩ . (58) h(\mathbf{t})=h_0(\mathbf{G}^{\mathsf{T}}\mathbf{t})+\langle\mathbf{w},\mathbf{t}\rangle. \tag*{(58)} h ( t ) = h 0 ( G T t ) + ⟨ w , t ⟩ . (58)
Undoing the translations introduced in part i ) i) i ) yields the general expressions
τ ( x ) = G − T τ 0 ( x ) + v \tau(\mathbf{x})=\mathbf{G}^{-\mathsf{T}}\tau_0(\mathbf{x})+\mathbf{v} τ ( x ) = G − T τ 0 ( x ) + v
and
h ( t ) = h 0 ( G T ( t − v ) ) + ⟨ w , t − v ⟩ + κ 0 , h(\mathbf{t})=h_0\left(\mathbf{G}^{\mathsf{T}}(\mathbf{t}-\mathbf{v})\right)+\langle\mathbf{w},\mathbf{t}-\mathbf{v}\rangle+\kappa_0, h ( t ) = h 0 ( G T ( t − v ) ) + ⟨ w , t − v ⟩ + κ 0 ,
for suitable v , w ∈ R n \mathbf{v},\mathbf{w}\in\mathbb{R}^{n} v , w ∈ R n and κ 0 ∈ R \kappa_0\in\mathbb{R} κ 0 ∈ R .
Conversely, suppose that ( h , τ ) (h,\tau) ( h , τ ) is obtained from ( h 0 , τ 0 ) (h_0,\tau_0) ( h 0 , τ 0 ) by such an invertible affine change of coordinates and an affine modification of the generator. Then τ \tau τ remains a diffeomorphism onto an open convex set, and h h h remains strictly convex. Furthermore, Bregman divergences are invariant under invertible affine coordinate changes accompanied by the corresponding transformation of their generators. The additional affine term in h h h does not affect its Bregman divergence. Thus
d h ( τ ( x ) , τ ( y ) ) = d h 0 ( τ 0 ( x ) , τ 0 ( y ) ) . d_h(\tau(\mathbf{x}),\tau(\mathbf{y}))=d_{h_0}(\tau_0(\mathbf{x}),\tau_0(\mathbf{y})). d h ( τ ( x ) , τ ( y )) = d h 0 ( τ 0 ( x ) , τ 0 ( y )) .
The original dual representation therefore remains valid with the same weight g g g . This proves uniqueness up to affine equivalence.
Proof of Proposition 4
By Fenchel equality, h ∗ ( ξ ) = ⟨ ξ , τ ⟩ − h ( τ ) h^*(\boldsymbol{\xi})=\langle\boldsymbol{\xi},\boldsymbol{\tau}\rangle-h(\boldsymbol{\tau}) h ∗ ( ξ ) = ⟨ ξ , τ ⟩ − h ( τ ) . The normalized perspective identity gives
f ( x ) = ( A x + b ) T τ ( x ) − g ( x ) h ( τ ( x ) ) . (59) f(\mathbf{x})=(\mathbf{A}\mathbf{x}+\mathbf{b})^{\mathsf{T}}\tau(\mathbf{x})-g(\mathbf{x})h(\tau(\mathbf{x})). \tag*{(59)} f ( x ) = ( Ax + b ) T τ ( x ) − g ( x ) h ( τ ( x )) . (59)
Differentiating the perspective identity, and using ∇ h ∗ ( ξ ( x ) ) = τ ( x ) \nabla h^*(\boldsymbol{\xi}(\mathbf{x}))=\tau(\mathbf{x}) ∇ h ∗ ( ξ ( x )) = τ ( x ) , yields
∇ f ( x ) = A T τ ( x ) − c h ( τ ( x ) ) . (60) \nabla f(\mathbf{x})=\mathbf{A}^{\mathsf{T}}\tau(\mathbf{x})-\mathbf{c}\,h(\tau(\mathbf{x})). \tag*{(60)} ∇ f ( x ) = A T τ ( x ) − c h ( τ ( x )) . (60)
Subtracting (59) from x ⊤ ∇ f ( x ) \mathbf{x}^{\top}\nabla f(\mathbf{x}) x ⊤ ∇ f ( x ) now gives
f ∗ ( ∇ f ( x ) ) = d h ( τ ( x ) ) − b ⊤ τ ( x ) . (61) f^{*}(\nabla f(\mathbf{x}))=d\,h(\tau(\mathbf{x}))-\mathbf{b}^{\top}\tau(\mathbf{x}). \tag*{(61)} f ∗ ( ∇ f ( x )) = d h ( τ ( x )) − b ⊤ τ ( x ) . (61)
Consequently,
M ⊤ ( τ ( x ) − h ( τ ( x ) ) ) = ( ∇ f ( x ) − f ∗ ( ∇ f ( x ) ) ) . \mathbf{M}^{\top}
\begin{pmatrix}
\tau(\mathbf{x})\\
-h(\tau(\mathbf{x}))
\end{pmatrix}
=
\begin{pmatrix}
\nabla f(\mathbf{x})\\
-f^{*}(\nabla f(\mathbf{x}))
\end{pmatrix}. M ⊤ ( τ ( x ) − h ( τ ( x )) ) = ( ∇ f ( x ) − f ∗ ( ∇ f ( x )) ) .
Invertibility of M \mathbf{M} M proves (9) ; block multiplication proves (10) .
Proof of Theorem 3
By the regularity and minimality assumptions, f = ψ ∗ f=\psi^{*} f = ψ ∗ is closed, proper, and of Legendre type. Moreover, f ∈ C ∞ ( X ) f\in C^{\infty}(\mathcal{X}) f ∈ C ∞ ( X ) , ∇ 2 f ≻ 0 \nabla^{2}f\succ0 ∇ 2 f ≻ 0 on X \mathcal{X} X , and Θ = dom ψ \Theta=\operatorname{dom}\psi Θ = dom ψ is open. The Legendre correspondence gives mutually inverse diffeomorphisms
∇ f : X ⟶ Θ , ∇ ψ : Θ ⟶ X . \nabla f:\mathcal{X}\longrightarrow\Theta,\qquad\nabla\psi:\Theta\longrightarrow\mathcal{X}. ∇ f : X ⟶ Θ , ∇ ψ : Θ ⟶ X .
Throughout the proof, write
M − ⊤ = ( P q r ⊤ s ) , F ( x ) = ⟨ ∇ f ( x ) , x ⟩ − f ( x ) . \mathbf{M}^{-\top}=
\begin{pmatrix}
\mathbf{P} & \mathbf{q}\\
\mathbf{r}^{\top} & s
\end{pmatrix},
\qquad
F(\mathbf{x})=\langle\nabla f(\mathbf{x}),\mathbf{x}\rangle-f(\mathbf{x}). M − ⊤ = ( P r ⊤ q s ) , F ( x ) = ⟨ ∇ f ( x ) , x ⟩ − f ( x ) .
Then
F ( x ) = ψ ( ∇ f ( x ) ) . F(\mathbf{x})=\psi(\nabla f(\mathbf{x})). F ( x ) = ψ ( ∇ f ( x )) .
Define
τ ( x ) = P ∇ f ( x ) − q F ( x ) , η ( x ) = s F ( x ) − r ⊤ ∇ f ( x ) , \tau(\mathbf{x})=\mathbf{P}\nabla f(\mathbf{x})-\mathbf{q}F(\mathbf{x}),
\qquad
\eta(\mathbf{x})=sF(\mathbf{x})-\mathbf{r}^{\top}\nabla f(\mathbf{x}), τ ( x ) = P ∇ f ( x ) − q F ( x ) , η ( x ) = s F ( x ) − r ⊤ ∇ f ( x ) ,
and set
T = τ ( X ) . \mathcal{T}=\tau(\mathcal{X}). T = τ ( X ) .
The function h h h is defined on R n \mathbb{R}^{n} R n by
h ( t ) = sup x ∈ X { ⟨ ξ ( x ) , t ⟩ − f ( x ) g ( x ) } . h(\mathbf{t})=\sup_{\mathbf{x}\in\mathcal{X}}\left\{\langle\xi(\mathbf{x}),\mathbf{t}\rangle-\frac{f(\mathbf{x})}{g(\mathbf{x})}\right\}. h ( t ) = x ∈ X sup { ⟨ ξ ( x ) , t ⟩ − g ( x ) f ( x ) } .
By the definition of M − ⊤ \mathbf{M}^{-\top} M − ⊤ ,
A ⊤ τ ( x ) − c η ( x ) = ∇ f ( x ) , \mathbf{A}^{\top}\tau(\mathbf{x})-c\eta(\mathbf{x})=\nabla f(\mathbf{x}), A ⊤ τ ( x ) − cη ( x ) = ∇ f ( x ) ,
b ⊤ τ ( x ) − d η ( x ) = − F ( x ) . (62) \mathbf{b}^{\top}\tau(\mathbf{x})-d\eta(\mathbf{x})=-F(\mathbf{x}). \tag*{(62)} b ⊤ τ ( x ) − d η ( x ) = − F ( x ) . (62)
i) Identification of h ∘ τ h\circ\tau h ∘ τ . For x ∈ X \mathbf{x}\in\mathcal{X} x ∈ X , let
m x ( t ) = ⟨ ξ ( x ) , t ⟩ − f ( x ) g ( x ) . m_{\mathbf{x}}(\mathbf{t})=\langle\xi(\mathbf{x}),\mathbf{t}\rangle-\frac{f(\mathbf{x})}{g(\mathbf{x})}. m x ( t ) = ⟨ ξ ( x ) , t ⟩ − g ( x ) f ( x ) .
Thus h = sup x ∈ X m x h=\sup_{\mathbf{x}\in\mathcal{X}}m_{\mathbf{x}} h = sup x ∈ X m x . For any x , x 0 ∈ X \mathbf{x},\mathbf{x}_{0}\in\mathcal{X} x , x 0 ∈ X , the identities (62) , evaluated at x 0 \mathbf{x}_{0} x 0 , yield
⟨ A x + b , τ ( x 0 ) ⟩ − η ( x 0 ) g ( x ) = ⟨ ∇ f ( x 0 ) , x ⟩ − F ( x 0 ) . (63) \langle\mathbf{A}\mathbf{x}+\mathbf{b},\tau(\mathbf{x}_{0})\rangle-\eta(\mathbf{x}_{0})g(\mathbf{x})
=
\langle\nabla f(\mathbf{x}_{0}),\mathbf{x}\rangle-F(\mathbf{x}_{0}). \tag*{(63)} ⟨ Ax + b , τ ( x 0 )⟩ − η ( x 0 ) g ( x ) = ⟨ ∇ f ( x 0 ) , x ⟩ − F ( x 0 ) . (63)
Using F ( x 0 ) = ⟨ ∇ f ( x 0 ) , x 0 ⟩ − f ( x 0 ) F(\mathbf{x}_{0})=\langle\nabla f(\mathbf{x}_{0}),\mathbf{x}_{0}\rangle-f(\mathbf{x}_{0}) F ( x 0 ) = ⟨ ∇ f ( x 0 ) , x 0 ⟩ − f ( x 0 ) , we obtain
m x ( τ ( x 0 ) ) = η ( x 0 ) − f ( x ) − f ( x 0 ) − ⟨ ∇ f ( x 0 ) , x − x 0 ⟩ g ( x ) = η ( x 0 ) − d f ( x , x 0 ) g ( x ) . (64) \begin{aligned}
m_{\mathbf{x}}(\tau(\mathbf{x}_{0}))=\eta(\mathbf{x}_{0})-
\frac{f(\mathbf{x})-f(\mathbf{x}_{0})-\langle\nabla f(\mathbf{x}_{0}),\mathbf{x}-\mathbf{x}_{0}\rangle}{g(\mathbf{x})} \\
&=\eta(\mathbf{x}_{0})-\frac{d_{f}(\mathbf{x},\mathbf{x}_{0})}{g(\mathbf{x})}. \tag*{(64)}
\end{aligned} m x ( τ ( x 0 )) = η ( x 0 ) − g ( x ) f ( x ) − f ( x 0 ) − ⟨ ∇ f ( x 0 ) , x − x 0 ⟩ = η ( x 0 ) − g ( x ) d f ( x , x 0 ) . (64)
Since f f f is strictly convex on X \mathcal{X} X and g > 0 g>0 g > 0 , the right-hand side is bounded above by η ( x 0 ) \eta(x_0) η ( x 0 ) , with equality if and only if x = x 0 \mathbf{x}=\mathbf{x}_0 x = x 0 . Consequently, for x 0 ∈ X x_0\in\mathcal{X} x 0 ∈ X ,
h ( τ ( x 0 ) ) = η ( x 0 ) . (65) h(\tau(x_0))=\eta(x_0). \tag*{(65)} h ( τ ( x 0 )) = η ( x 0 ) . (65)
The supremum defining h ( τ ( x 0 ) ) h(\tau(x_0)) h ( τ ( x 0 )) is attained uniquely at x = x 0 \mathbf{x}=\mathbf{x}_0 x = x 0 . In particular, τ \tau τ is injective: if τ ( x 1 ) = τ ( x 2 ) \tau(x_1)=\tau(x_2) τ ( x 1 ) = τ ( x 2 ) , then both x 1 x_1 x 1 and x 2 x_2 x 2 maximize the same expression, hence x 1 = x 2 x_1=x_2 x 1 = x 2 . Furthermore, since m x 0 ≤ h m_{x_0}\leq h m x 0 ≤ h and m x 0 m_{x_0} m x 0 touches h h h at τ ( x 0 ) \tau(x_0) τ ( x 0 ) ,
ξ ( x 0 ) ∈ ∂ h ( τ ( x 0 ) ) . (66) \xi(x_0)\in\partial h(\tau(x_0)). \tag*{(66)} ξ ( x 0 ) ∈ ∂ h ( τ ( x 0 )) . (66)
Fenchel-Young equality then gives
h ∗ ( ξ ( x 0 ) ) = f ( x 0 ) g ( x 0 ) . (67) h^*(\xi(x_0))=\frac{f(x_0)}{g(x_0)}. \tag*{(67)} h ∗ ( ξ ( x 0 )) = g ( x 0 ) f ( x 0 ) . (67)
ii) Diffeomorphism τ \tau τ . Since ∇ F ( x ) = ∇ 2 f ( x ) x \nabla F(\mathbf{x})=\nabla^2 f(\mathbf{x})\mathbf{x} ∇ F ( x ) = ∇ 2 f ( x ) x , differentiating the definition of τ \tau τ gives
J τ ( x ) = ( P − q x T ) ∇ 2 f ( x ) . (68) J_\tau(\mathbf{x})=(\mathbf{P}-\mathbf{q}\mathbf{x}^{\mathsf T})\nabla^2 f(\mathbf{x}). \tag*{(68)} J τ ( x ) = ( P − q x T ) ∇ 2 f ( x ) . (68)
The projective matrix identities give
det ( P − q x T ) = g ( x ) det M . \det(\mathbf{P}-\mathbf{q}\mathbf{x}^{\mathsf T})=\frac{g(\mathbf{x})}{\det\mathbf{M}}. det ( P − q x T ) = det M g ( x ) .
Therefore
det J τ ( x ) = g ( x ) det ∇ 2 f ( x ) det M ≠ 0. \det J_\tau(\mathbf{x})=\frac{g(\mathbf{x})\det\nabla^2 f(\mathbf{x})}{\det\mathbf{M}}\neq0. det J τ ( x ) = det M g ( x ) det ∇ 2 f ( x ) = 0.
Thus τ \tau τ is a local C ∞ C^\infty C ∞ diffeomorphism. Since it is injective by the identification of h ∘ τ h\circ\tau h ∘ τ in part i) above, it is a C ∞ C^\infty C ∞ diffeomorphism from X \mathcal{X} X onto the nonempty open set T = τ ( X ) \mathcal{T}=\tau(\mathcal{X}) T = τ ( X ) .
iii) Convexity of T \mathcal{T} T . As a supremum of affine functions, h h h is convex on R n \mathbb{R}^n R n . Consequently, dom h \operatorname{dom}h dom h is convex. By (65) , T ⊆ dom h \mathcal{T}\subseteq\operatorname{dom}h T ⊆ dom h . We prove the reverse inclusion. Let t ∈ dom h \mathbf{t}\in\operatorname{dom}h t ∈ dom h and set
σ = h ( t ) , θ = A T t − c σ . \sigma=h(\mathbf{t}),\qquad\boldsymbol{\theta}=\mathbf{A}^{\mathsf T}\mathbf{t}-\mathbf{c}\sigma. σ = h ( t ) , θ = A T t − c σ .
By the definition of h h h , for every x ∈ X \mathbf{x}\in\mathcal{X} x ∈ X ,
⟨ ξ ( x ) , t ⟩ − f ( x ) g ( x ) ≤ σ . \langle\xi(\mathbf{x}),\mathbf{t}\rangle-\frac{f(\mathbf{x})}{g(\mathbf{x})}\leq\sigma. ⟨ ξ ( x ) , t ⟩ − g ( x ) f ( x ) ≤ σ .
Multiplying by g ( x ) > 0 g(\mathbf{x})>0 g ( x ) > 0 gives
⟨ θ , x ⟩ − f ( x ) ≤ d σ − b T t , for all x ∈ X . \langle\boldsymbol{\theta},\mathbf{x}\rangle-f(\mathbf{x})\leq d\sigma-\mathbf{b}^{\mathsf T}\mathbf{t},\qquad\text{for all }\mathbf{x}\in\mathcal{X}. ⟨ θ , x ⟩ − f ( x ) ≤ d σ − b T t , for all x ∈ X .
That is, since f f f is closed and X = int dom f \mathcal{X}=\operatorname{int}\operatorname{dom}f X = int dom f ([23 Thm. 7.5] ), and using f ∗ = ψ f^*=\psi f ∗ = ψ ,
ψ ( θ ) ≤ d σ − b T t (69) \psi(\boldsymbol{\theta})\leq d\sigma-\mathbf{b}^{\mathsf T}\mathbf{t} \tag*{(69)} ψ ( θ ) ≤ d σ − b T t (69)
In particular, θ ∈ Θ \boldsymbol{\theta}\in\Theta θ ∈ Θ . We claim that equality holds in (69) . Otherwise, since Θ \Theta Θ is open and ψ \psi ψ is continuous on Θ \Theta Θ , there exists ε > 0 \varepsilon>0 ε > 0 sufficiently small such that
ψ ( θ + ε c ) ≤ d ( σ − ε ) − b T t . \psi(\boldsymbol{\theta}+\varepsilon\mathbf{c})\leq d(\sigma-\varepsilon)-\mathbf{b}^{\mathsf T}\mathbf{t}. ψ ( θ + ε c ) ≤ d ( σ − ε ) − b T t .
By conjugacy, this implies
⟨ θ + ε c , x ⟩ − f ( x ) ≤ d ( σ − ε ) − b T t \langle\boldsymbol{\theta}+\varepsilon\mathbf{c},\mathbf{x}\rangle-f(\mathbf{x})\leq d(\sigma-\varepsilon)-\mathbf{b}^{\mathsf T}\mathbf{t} ⟨ θ + ε c , x ⟩ − f ( x ) ≤ d ( σ − ε ) − b T t
for every x ∈ X \mathbf{x} \in\mathcal{X} x ∈ X . Rearranging and dividing by g ( x ) > 0 g(\mathbf{x}) > 0 g ( x ) > 0 gives
m x ( t ) ≤ σ − ε . m_{\mathbf{x}}(t) \le\sigma- \varepsilon. m x ( t ) ≤ σ − ε .
Taking the supremum over x \mathbf{x} x would imply
h ( t ) ≤ σ − ε , h(t) \le\sigma- \varepsilon, h ( t ) ≤ σ − ε ,
which contradicts σ = h ( t ) \sigma= h(t) σ = h ( t ) . Hence
ψ ( θ ) = d σ − b T t . (70) \psi(\boldsymbol{\theta}) = d\sigma- \mathbf{b}^{\mathsf{T}}t. \tag*{(70)} ψ ( θ ) = d σ − b T t . (70)
Set x = ∇ ψ ( θ ) ∈ X \mathbf{x} = \nabla\psi(\boldsymbol{\theta}) \in\mathcal{X} x = ∇ ψ ( θ ) ∈ X . By Legendre duality, one has ∇ f ( x ) = θ \nabla f(\mathbf{x}) = \boldsymbol{\theta} ∇ f ( x ) = θ and F ( x ) = ψ ( θ ) F(\mathbf{x}) = \psi(\boldsymbol{\theta}) F ( x ) = ψ ( θ ) . It follows that
M − T ( t − σ ) = ( ∇ f ( x ) − F ( x ) ) . \mathbf{M}^{-\mathsf{T}}
\begin{pmatrix}
t\\
-\sigma
\end{pmatrix}
=
\begin{pmatrix}
\nabla f(\mathbf{x})\\
-F(\mathbf{x})
\end{pmatrix}. M − T ( t − σ ) = ( ∇ f ( x ) − F ( x ) ) .
Multiplying by M − T \mathbf{M}^{-\mathsf{T}} M − T gives
t = τ ( x ) , σ = η ( x ) . t = \tau(\mathbf{x}), \qquad\sigma= \eta(\mathbf{x}). t = τ ( x ) , σ = η ( x ) .
Thus t ∈ T t \in\mathcal{T} t ∈ T , proving
dom h = T . (71) \operatorname{dom} h = \mathcal{T}. \tag*{(71)} dom h = T . (71)
Since dom h \operatorname{dom} h dom h is convex and T \mathcal{T} T is open by part (b), T \mathcal{T} T is a nonempty open convex set.
iv) Regularity and strict convexity of h h h . By part i), one has h = η ∘ τ − 1 h = \eta\circ\tau^{-1} h = η ∘ τ − 1 on T \mathcal{T} T . Hence h h h is differentiable on T \mathcal{T} T . Since a differentiable convex function has a singleton subdifferential, Eq. (66) implies
∇ h ( τ ( x ) ) = ξ ( x ) . (72) \nabla h(\tau(\mathbf{x})) = \boldsymbol{\xi}(\mathbf{x}). \tag*{(72)} ∇ h ( τ ( x )) = ξ ( x ) . (72)
Therefore ∇ h = ξ ∘ τ − 1 \nabla h = \boldsymbol{\xi} \circ\tau^{-1} ∇ h = ξ ∘ τ − 1 on T \mathcal{T} T , which shows that h ∈ C 2 ( T ) h \in C^2(\mathcal{T}) h ∈ C 2 ( T ) . Differentiating (72) gives
∇ 2 h ( τ ( x ) ) J τ ( x ) = J ξ ( x ) . \nabla^2 h(\tau(\mathbf{x}))J_{\tau}(\mathbf{x}) = J_{\boldsymbol{\xi}}(\mathbf{x}). ∇ 2 h ( τ ( x )) J τ ( x ) = J ξ ( x ) .
The projective identities yield
J ξ ( x ) = 1 g ( x ) ( P − q x T ) − T . J_{\boldsymbol{\xi}}(\mathbf{x}) = \frac{1}{g(\mathbf{x})}(\mathbf{P} - \mathbf{q}\mathbf{x}^{\mathsf{T}})^{-\mathsf{T}}. J ξ ( x ) = g ( x ) 1 ( P − q x T ) − T .
Combining this with (68) , we obtain
∇ 2 h ( τ ( x ) ) = 1 g ( x ) ( P − q x T ) − T [ ∇ 2 f ( x ) ] − 1 ( P − q x T ) − 1 . (73) \nabla^2 h(\tau(\mathbf{x})) =
\frac{1}{g(\mathbf{x})}(\mathbf{P} - \mathbf{q}\mathbf{x}^{\mathsf{T}})^{-\mathsf{T}}
[\nabla^2 f(\mathbf{x})]^{-1}
(\mathbf{P} - \mathbf{q}\mathbf{x}^{\mathsf{T}})^{-1}. \tag*{(73)} ∇ 2 h ( τ ( x )) = g ( x ) 1 ( P − q x T ) − T [ ∇ 2 f ( x ) ] − 1 ( P − q x T ) − 1 . (73)
Since g ( x ) > 0 g(\mathbf{x}) > 0 g ( x ) > 0 , ∇ 2 f ( x ) ≻ 0 \nabla^2 f(\mathbf{x}) \succ0 ∇ 2 f ( x ) ≻ 0 , and P − q x T \mathbf{P} - \mathbf{q}\mathbf{x}^{\mathsf{T}} P − q x T is invertible,
∇ 2 h ( t ) ≻ 0 , t ∈ T . \nabla^2 h(t) \succ0, \qquad t \in\mathcal{T}. ∇ 2 h ( t ) ≻ 0 , t ∈ T .
Thus h h h is strictly convex on the convex set T \mathcal{T} T .
v) Conformal identity. By parts i)–iv), the pair ( h , τ ) (h,\tau) ( h , τ ) satisfies the regularity and convexity assumptions of Theorem 2. Moreover, Eq. (72) gives
ξ = ∇ h ∘ τ , \boldsymbol{\xi} = \nabla h \circ\tau, ξ = ∇ h ∘ τ ,
while Eq. (67) gives
f ( x ) = g ( x ) h ∗ ( ξ ( x ) ) . f(\mathbf{x}) = g(\mathbf{x})h^{*}(\boldsymbol{\xi}(\mathbf{x})). f ( x ) = g ( x ) h ∗ ( ξ ( x )) .
Thus the projective representation holds with no additional affine term. Applying Theorem 2, we obtain, for x , y ∈ X x,y \in\mathcal{X} x , y ∈ X ,
d f ( y , x ) = g ( y ) d h ( τ ( x ) , τ ( y ) ) . (74) d_f(\mathbf{y},\mathbf{x})=g(\mathbf{y})d_h(\tau(\mathbf{x}),\tau(\mathbf{y})). \tag*{(74)} d f ( y , x ) = g ( y ) d h ( τ ( x ) , τ ( y )) . (74)
vi) Legendre property. Since h h h is a supremum of continuous affine functions, it is convex and lower semicontinuous on R n \mathbb{R}^n R n . By part i) , it is finite on the nonempty set T \mathcal{T} T , and is therefore proper.
By part iii) ,
dom h = int ( dom h ) = T . \operatorname{dom} h=\operatorname{int}(\operatorname{dom} h)=\mathcal{T}. dom h = int ( dom h ) = T .
By part iv) , h h h is differentiable and strictly convex on T \mathcal{T} T . Since
dom ∂ h ⊆ dom h = T , \operatorname{dom}\partial h\subseteq\operatorname{dom} h=\mathcal{T}, dom ∂ h ⊆ dom h = T ,
strict convexity on T \mathcal{T} T implies essential strict convexity.
We now verify essential smoothness. Let ( t k ) (t_k) ( t k ) be a sequence in T \mathcal{T} T converging to t ˉ ∈ ∂ T \bar{t}\in\partial\mathcal{T} t ˉ ∈ ∂ T . Since dom h = T \operatorname{dom} h=\mathcal{T} dom h = T is open,
h ( t ˉ ) = + ∞ . h(\bar{t})=+\infty. h ( t ˉ ) = + ∞.
By lower semicontinuity,
h ( t k ) ⟶ + ∞ . h(t_k)\longrightarrow+\infty. h ( t k ) ⟶ + ∞.
Suppose, by contradiction, that ∥ ∇ h ( t k ) ∥ \|\nabla h(t_k)\| ∥∇ h ( t k ) ∥ does not tend to infinity. Then there exists a subsequence, still denoted ( t k ) (t_k) ( t k ) , along which the gradients remain bounded. Fix t 0 ∈ T t_0\in\mathcal{T} t 0 ∈ T . The convexity inequality gives
h ( t 0 ) ≥ h ( t k ) + ⟨ ∇ h ( t k ) , t 0 − t k ⟩ . h(t_0)\ge h(t_k)+\langle\nabla h(t_k),t_0-t_k\rangle. h ( t 0 ) ≥ h ( t k ) + ⟨ ∇ h ( t k ) , t 0 − t k ⟩ .
Hence
h ( t k ) ≤ h ( t 0 ) + ∥ ∇ h ( t k ) ∥ ∥ t k − t 0 ∥ . h(t_k)\le h(t_0)+\|\nabla h(t_k)\|\,\|t_k-t_0\|. h ( t k ) ≤ h ( t 0 ) + ∥∇ h ( t k ) ∥ ∥ t k − t 0 ∥.
The right-hand side remains bounded along the subsequence, contradicting h ( t k ) → + ∞ h(t_k)\to+\infty h ( t k ) → + ∞ . Therefore
∥ ∇ h ( t k ) ∥ ⟶ + ∞ . \|\nabla h(t_k)\|\longrightarrow+\infty. ∥∇ h ( t k ) ∥ ⟶ + ∞.
Thus h h h is essentially smooth. Together with essential strict convexity, this proves that h h h is of Legendre type.
vii) Smoothness. Since f ∈ C ∞ ( X ) f\in C^\infty(\mathcal{X}) f ∈ C ∞ ( X ) , both ∇ f \nabla f ∇ f and
F ( x ) = ⟨ ∇ f ( x ) , x ⟩ − f ( x ) F(\mathbf{x})=\langle\nabla f(\mathbf{x}),\mathbf{x}\rangle-f(\mathbf{x}) F ( x ) = ⟨ ∇ f ( x ) , x ⟩ − f ( x )
are C ∞ C^\infty C ∞ on X \mathcal{X} X . Consequently, τ \tau τ and η \eta η are C ∞ C^\infty C ∞ . By part ii) , J τ J_\tau J τ is invertible everywhere on X \mathcal{X} X , and τ − 1 \tau^{-1} τ − 1 is C ∞ C^\infty C ∞ on T \mathcal{T} T . Finally, by part i) ,
h = η ∘ τ − 1 on T , h=\eta\circ\tau^{-1}\quad\text{on }\mathcal{T}, h = η ∘ τ − 1 on T ,
so
h ∈ C ∞ ( T ) . h\in C^\infty(\mathcal{T}). h ∈ C ∞ ( T ) .
viii) Uniqueness. If ( h 1 , τ 1 ) (h_1,\tau_1) ( h 1 , τ 1 ) is another such pair, Proposition 4 gives τ 1 = τ \tau_1=\tau τ 1 = τ and h 1 ∘ τ = h ∘ τ h_1\circ\tau=h\circ\tau h 1 ∘ τ = h ∘ τ , so h 1 = h h_1=h h 1 = h on T \mathcal{T} T , and both equal + ∞ +\infty + ∞ outside dom h 1 = dom h = T \operatorname{dom} h_1=\operatorname{dom} h=\mathcal{T} dom h 1 = dom h = T .
Proof of Corollary 2
By Theorem 4, τ \tau τ is a bijection from X \mathcal{X} X onto T \mathcal{T} T , h h h is a closed proper convex function of Legendre type with dom h = T \operatorname{dom} h = \mathcal{T} dom h = T , and ( h , τ ) (h,\tau) ( h , τ ) satisfies the conditions of Theorem 2, so that Proposition 4 applies. In particular, it gives τ = P ∇ f − q F \tau= \mathbf{P}\nabla f - \mathbf{q}F τ = P ∇ f − q F and
A T τ − c h ∘ τ = ∇ f , b T τ − d h ∘ τ = − F , (75) \mathbf{A}^{\mathsf{T}}\tau-c\,h\circ\tau=\nabla f,\qquad\mathbf{b}^{\mathsf{T}}\tau-d\,h\circ\tau=-F, \tag*{(75)} A T τ − c h ∘ τ = ∇ f , b T τ − d h ∘ τ = − F , (75)
the two block rows of
M T ( τ ; − h ∘ τ ) = ( ∇ f ; − F ) \mathbf{M}^{\mathsf{T}}(\tau;-h\circ\tau)=(\nabla f;-F) M T ( τ ; − h ∘ τ ) = ( ∇ f ; − F )
i) Expression of h ∗ ( u ) h^*(u) h ∗ ( u ) . Since h h h is of Legendre type, Rockafellar [23 ] shows that h ∗ h^* h ∗ is of Legendre type, that ∇ h \nabla h ∇ h is a bijection from T \mathcal{T} T onto int ( dom h ∗ ) \operatorname{int}(\operatorname{dom}h^*) int ( dom h ∗ ) , and that ∇ h ∗ = ( ∇ h ) − 1 \nabla h^*=(\nabla h)^{-1} ∇ h ∗ = ( ∇ h ) − 1 . Since τ \tau τ is a bijection from X \mathcal{X} X onto T \mathcal{T} T and ∇ h ∘ τ = ξ \nabla h\circ\tau=\xi ∇ h ∘ τ = ξ , ξ \xi ξ is a bijection from X \mathcal{X} X onto ∇ h ( T ) = int ( dom h ∗ ) \nabla h(\mathcal{T})=\operatorname{int}(\operatorname{dom}h^*) ∇ h ( T ) = int ( dom h ∗ ) , and ∇ h ∗ = τ ∘ ξ − 1 \nabla h^*=\tau\circ\xi^{-1} ∇ h ∗ = τ ∘ ξ − 1 there. Finally, let u = ξ ( x ) \mathbf{u}=\xi(\mathbf{x}) u = ξ ( x ) . By (8) ,
g ( x ) ( u 1 ) = M ( x 1 ) g(\mathbf{x})\begin{pmatrix}\mathbf{u}\\1\end{pmatrix}
=\mathbf{M}\begin{pmatrix}\mathbf{x}\\1\end{pmatrix} g ( x ) ( u 1 ) = M ( x 1 )
and M − 1 = ( M − T ) T \mathbf{M}^{-1}=(\mathbf{M}^{-\mathsf{T}})^{\mathsf{T}} M − 1 = ( M − T ) T has block rows ( P T r ) (\mathbf{P}^{\mathsf{T}}\ \mathbf{r}) ( P T r ) and ( q T s ) (\mathbf{q}^{\mathsf{T}}\ \mathbf{s}) ( q T s ) , so
( x 1 ) = g ( x ) ( P T u + r q T u + s ) . \begin{pmatrix}\mathbf{x}\\1\end{pmatrix}
=g(\mathbf{x})\begin{pmatrix}\mathbf{P}^{\mathsf{T}}\mathbf{u}+\mathbf{r}\\\mathbf{q}^{\mathsf{T}}\mathbf{u}+s\end{pmatrix}. ( x 1 ) = g ( x ) ( P T u + r q T u + s ) .
The inverse projective identities give
q T ξ ( x ) + s = 1 g ( x ) , \mathbf{q}^{\mathsf{T}}\xi(\mathbf{x})+s=\frac{1}{g(\mathbf{x})}, q T ξ ( x ) + s = g ( x ) 1 ,
and
x = P T ξ ( x ) + r q T ξ ( x ) + s . \mathbf{x}=\frac{\mathbf{P}^{\mathsf{T}}\xi(\mathbf{x})+\mathbf{r}}{\mathbf{q}^{\mathsf{T}}\xi(\mathbf{x})+s}. x = q T ξ ( x ) + s P T ξ ( x ) + r .
Substituting into (7) , we obtain, for every u ∈ ξ ( X ) \mathbf{u}\in\xi(\mathcal{X}) u ∈ ξ ( X ) ,
h ∗ ( u ) = ( q T u + s ) f ( P T u + r q T u + s ) . (76) h^*(\mathbf{u})=(\mathbf{q}^{\mathsf{T}}\mathbf{u}+s)f\left(\frac{\mathbf{P}^{\mathsf{T}}\mathbf{u}+\mathbf{r}}{\mathbf{q}^{\mathsf{T}}\mathbf{u}+s}\right). \tag*{(76)} h ∗ ( u ) = ( q T u + s ) f ( q T u + s P T u + r ) . (76)
which is (13) .
ii) Expression of h ( t ) h(\mathbf{t}) h ( t ) . Since h h h is closed, proper and convex, h = h ∗ ∗ h=h^{**} h = h ∗∗ (Rockafellar, [23 ] ), that is, for t ∈ R n \mathbf{t}\in\mathbb{R}^n t ∈ R n ,
h ( t ) = sup u ( ⟨ u , t ⟩ − h ∗ ( u ) ) . h(\mathbf{t})=\sup_{\mathbf{u}}\left(\langle\mathbf{u},\mathbf{t}\rangle-h^*(\mathbf{u})\right). h ( t ) = u sup ( ⟨ u , t ⟩ − h ∗ ( u ) ) .
The function u ↦ h ∗ ( u ) − ⟨ u , t ⟩ \mathbf{u}\mapsto h^*(\mathbf{u})-\langle\mathbf{u},\mathbf{t}\rangle u ↦ h ∗ ( u ) − ⟨ u , t ⟩ is closed, proper and convex, so its values on dom h ∗ \operatorname{dom}h^* dom h ∗ are limits of its values on int ( dom h ∗ ) \operatorname{int}(\operatorname{dom}h^*) int ( dom h ∗ ) along segments (Rockafellar, [23 ] ), and the supremum may be taken over int ( dom h ∗ ) = ξ ( X ) \operatorname{int}(\operatorname{dom}h^*)=\xi(\mathcal{X}) int ( dom h ∗ ) = ξ ( X ) . By part i),
h ( t ) = sup x ∈ X ( ⟨ ξ ( x ) , t ⟩ − h ∗ ( ξ ( x ) ) ) = sup x ∈ X ( ⟨ ξ ( x ) , t ⟩ − f ( x ) g ( x ) ) , h(\mathbf{t})=\sup_{\mathbf{x}\in\mathcal{X}}\left(\langle\xi(\mathbf{x}),\mathbf{t}\rangle-h^*(\xi(\mathbf{x}))\right)
=\sup_{\mathbf{x}\in\mathcal{X}}\left(\langle\xi(\mathbf{x}),\mathbf{t}\rangle-\frac{f(\mathbf{x})}{g(\mathbf{x})}\right), h ( t ) = x ∈ X sup ( ⟨ ξ ( x ) , t ⟩ − h ∗ ( ξ ( x )) ) = x ∈ X sup ( ⟨ ξ ( x ) , t ⟩ − g ( x ) f ( x ) ) ,
which is (12) . In particular, h h h is a supremum of affine functions, namely the conjugate of the restriction of h ∗ h^* h ∗ to ξ ( X ) \xi(\mathcal{X}) ξ ( X ) , given there by (13) .
iii) Implicit equation. Let t ∈ T \mathbf{t} \in\mathcal{T} t ∈ T . There exists a unique x ∈ X \mathbf{x} \in\mathcal{X} x ∈ X such that t = τ ( x ) \mathbf{t} = \tau(\mathbf{x}) t = τ ( x ) . The block identities (75) therefore give
A T t − c h ( t ) = ∇ f ( x ) \mathbf{A}^{\mathsf{T}}\mathbf{t}-c h(\mathbf{t})=\nabla f(\mathbf{x}) A T t − c h ( t ) = ∇ f ( x )
and
d h ( t ) − b T t = F ( x ) = ψ ( ∇ f ( x ) ) . d h(\mathbf{t})-\mathbf{b}^{\mathsf{T}}\mathbf{t}=F(\mathbf{x})=\psi(\nabla f(\mathbf{x})). d h ( t ) − b T t = F ( x ) = ψ ( ∇ f ( x )) .
Consequently,
ψ ( A T t − c h ( t ) ) = d h ( t ) − b T t . \psi(\mathbf{A}^{\mathsf{T}}\mathbf{t}-c h(\mathbf{t}))=d h(\mathbf{t})-\mathbf{b}^{\mathsf{T}}\mathbf{t}. ψ ( A T t − c h ( t )) = d h ( t ) − b T t .
Thus σ = h ( t ) \sigma=h(\mathbf{t}) σ = h ( t ) solves
ψ ( A T t − c σ ) = d σ − b T t . (77) \psi(\mathbf{A}^{\mathsf{T}}\mathbf{t}-c\sigma)=d\sigma-\mathbf{b}^{\mathsf{T}}\mathbf{t}. \tag*{(77)} ψ ( A T t − c σ ) = d σ − b T t . (77)
Conversely, suppose σ ∈ R \sigma\in\mathbb{R} σ ∈ R satisfies (77) . Since the right-hand side is finite,
θ = A T t − c σ ∈ Θ . \boldsymbol{\theta}=\mathbf{A}^{\mathsf{T}}\mathbf{t}-c\sigma\in\Theta. θ = A T t − c σ ∈ Θ.
Set x ′ = ∇ ψ ( θ ) ∈ X \mathbf{x}'=\nabla\psi(\boldsymbol{\theta})\in\mathcal{X} x ′ = ∇ ψ ( θ ) ∈ X . Then
∇ f ( x ′ ) = A T t − c σ \nabla f(\mathbf{x}')=\mathbf{A}^{\mathsf{T}}\mathbf{t}-c\sigma ∇ f ( x ′ ) = A T t − c σ
and
F ( x ′ ) = ψ ( θ ) = d σ − b T t . F(\mathbf{x}')=\psi(\boldsymbol{\theta})=d\sigma-\mathbf{b}^{\mathsf{T}}\mathbf{t}. F ( x ′ ) = ψ ( θ ) = d σ − b T t .
Therefore
M T ( t − σ ) = ( ∇ f ( x ′ ) − F ( x ′ ) ) . \mathbf{M}^{\mathsf{T}}
\begin{pmatrix}
\mathbf{t}\\
-\sigma
\end{pmatrix}
=
\begin{pmatrix}
\nabla f(\mathbf{x}')\\
-F(\mathbf{x}')
\end{pmatrix}. M T ( t − σ ) = ( ∇ f ( x ′ ) − F ( x ′ ) ) .
Multiplication by M − T \mathbf{M}^{-\mathsf{T}} M − T yields
t = τ ( x ′ ) , − σ = − h ( τ ( x ′ ) ) . \mathbf{t}=\tau(\mathbf{x}'),\qquad-\sigma=-h(\tau(\mathbf{x}')). t = τ ( x ′ ) , − σ = − h ( τ ( x ′ )) .
Thus,
σ = h ( τ ( x ′ ) ) = h ( t ) . \sigma=h(\tau(\mathbf{x}'))=h(\mathbf{t}). σ = h ( τ ( x ′ )) = h ( t ) .
This proves uniqueness.
Proof of Corollary 3
Proof of Theorem 6
Let t = τ ( x ) \mathbf{t}=\tau(\mathbf{x}) t = τ ( x ) and θ = ∇ f ( x ) \boldsymbol{\theta}=\nabla f(\mathbf{x}) θ = ∇ f ( x ) . The reconstruction identities give
θ = A T t − c h ( t ) , f ∗ ( θ ) = d h ( t ) − b T t . \boldsymbol{\theta}=\mathbf{A}^{\mathsf{T}}\mathbf{t}-c h(\mathbf{t}),\qquad f^*(\boldsymbol{\theta})=d h(\mathbf{t})-\mathbf{b}^{\mathsf{T}}\mathbf{t}. θ = A T t − c h ( t ) , f ∗ ( θ ) = d h ( t ) − b T t .
Substituting into the natural exponential-family log-density yields
log p ( y ; θ ) = ⟨ θ , y ⟩ − f ∗ ( θ ) + log p 0 ( y ) = ⟨ t , A y + b ⟩ − ( c T y + d ) h ( t ) + log p 0 ( y ) . \begin{aligned}
\log p(\mathbf{y};\boldsymbol{\theta})&=\langle\boldsymbol{\theta},\mathbf{y}\rangle-f^*(\boldsymbol{\theta})+\log p_0(\mathbf{y})\\
&=\langle\mathbf{t},\mathbf{A}\mathbf{y}+\mathbf{b}\rangle-(c^{\mathsf{T}}\mathbf{y}+d)h(\mathbf{t})+\log p_0(\mathbf{y}).
\end{aligned} log p ( y ; θ ) = ⟨ θ , y ⟩ − f ∗ ( θ ) + log p 0 ( y ) = ⟨ t , Ay + b ⟩ − ( c T y + d ) h ( t ) + log p 0 ( y ) .
This proves (15) . The normalization statement follows from (14) . Finally, for y ∈ X \mathbf{y}\in\mathcal{X} y ∈ X , combining (3) with (11) gives (16) .
Proof of Theorem 7
i) For the β \beta β -scaled model, choose
M = ( I n 0 0 1 / β ) , M − T = ( I n 0 0 β ) . \mathbf{M} = \begin{pmatrix} I_n & 0 \\ 0 & 1/\beta\end{pmatrix}, \qquad\mathbf{M}^{-\mathsf{T}} = \begin{pmatrix} I_n & 0 \\ 0 & \beta\end{pmatrix}. M = ( I n 0 0 1/ β ) , M − T = ( I n 0 0 β ) .
The reconstruction formulas give τ = ∇ f \tau= \nabla f τ = ∇ f and h = β f ∗ h = \beta f^* h = β f ∗ .
ii) For the α \alpha α -scaled model, choose
M = ( 0 − 1 1 / α 0 ) , M − T = ( 0 − 1 α 0 ) . \mathbf{M} = \begin{pmatrix} 0 & -1 \\ 1/\alpha& 0 \end{pmatrix}, \qquad\mathbf{M}^{-\mathsf{T}} = \begin{pmatrix} 0 & -1 \\ \alpha& 0 \end{pmatrix}. M = ( 0 1/ α − 1 0 ) , M − T = ( 0 α − 1 0 ) .
The reconstruction formulas give
τ ( x ) = x f ′ ( x ) − f ( x ) = f ∗ ( f ′ ( x ) ) , h ( τ ( x ) ) = − α f ′ ( x ) . \tau(x) = xf'(x) - f(x) = f^*(f'(x)), \qquad h(\tau(x)) = -\alpha f'(x). τ ( x ) = x f ′ ( x ) − f ( x ) = f ∗ ( f ′ ( x )) , h ( τ ( x )) = − α f ′ ( x ) .
Since f ′ ′ ( x ) > 0 f''(x) > 0 f ′′ ( x ) > 0 and x > 0 x > 0 x > 0 ,
τ ′ ( x ) = x f ′ ′ ( x ) > 0. \tau'(x) = xf''(x) > 0. τ ′ ( x ) = x f ′′ ( x ) > 0.
Furthermore, f ∗ f^* f ∗ is strictly increasing on f ′ ( X ) = int ( dom f ∗ ) = dom f ∗ f'(X) = \operatorname{int}(\operatorname{dom} f^*) = \operatorname{dom} f^* f ′ ( X ) = int ( dom f ∗ ) = dom f ∗ , because
( f ∗ ) ′ ( f ′ ( x ) ) = x > 0. (f^*)'(f'(x)) = x > 0. ( f ∗ ) ′ ( f ′ ( x )) = x > 0.
Thus f ∗ f^* f ∗ is invertible and yields the stated expression for h h h .
The two log-density representations follow from Theorem 6. The Legendre properties follow from Theorem 4.
Proof of Proposition 5
We start by deriving the general form of the log-posterior in the following lemma.
Lemma 5. Consider the Bayesian model defined by the log-likelihood
ℓ ( y ; x ) = − d f ( y , x ) + ϕ f ( y ) (78) \ell(\mathbf{y};\mathbf{x}) = -d_f(\mathbf{y},\mathbf{x}) + \phi_f(\mathbf{y}) \tag*{(78)} ℓ ( y ; x ) = − d f ( y , x ) + ϕ f ( y ) (78) and the log-prior
log p ( x ∣ α , β ) = − β d f ( α β , x ) + φ f ( α , β ) . (79) \log p(\mathbf{x}\mid\alpha,\beta) = -\beta d_f\left(\frac{\alpha}{\beta},\mathbf{x}\right) + \varphi_f(\alpha,\beta). \tag*{(79)} log p ( x ∣ α , β ) = − β d f ( β α , x ) + φ f ( α , β ) . (79) Then the log-posterior can be written as
log p ( x ∣ y , α , β ) = − d f ( y , x ) − β d f ( α β , x ) + ψ f ( y , α , β ) , (80) \log p(\mathbf{x}\mid\mathbf{y},\alpha,\beta) = -d_f(\mathbf{y},\mathbf{x}) - \beta d_f\left(\frac{\alpha}{\beta},\mathbf{x}\right) + \psi_f(\mathbf{y},\alpha,\beta), \tag*{(80)} log p ( x ∣ y , α , β ) = − d f ( y , x ) − β d f ( β α , x ) + ψ f ( y , α , β ) , (80) where the term ψ f ( y , α , β ) \psi_f(\mathbf{y},\alpha,\beta) ψ f ( y , α , β ) does not depend on the parameter x \mathbf{x} x .
Proof. We want to show that combining the log prior
log p ( x ∣ α , β ) = − β d f ( α β , x ) + φ f ( α , β ) (81) \log p(\mathbf{x}\mid\alpha,\beta) = -\beta d_f\left(\frac{\alpha}{\beta},\mathbf{x}\right) + \varphi_f(\alpha,\beta) \tag*{(81)} log p ( x ∣ α , β ) = − β d f ( β α , x ) + φ f ( α , β ) (81)
with the log-likelihood (78) gives the log-posterior
log p ( x ∣ y , α , β ) = ℓ ( y ; x ) + log p ( x ∣ α , β ) = − d f ( y , x ) − β d f ( α β , x ) + ψ f ( y , α , β ) (82) \begin{aligned}
\log p(\mathbf{x}\mid\mathbf{y},\alpha,\beta) &= \ell(\mathbf{y};\mathbf{x}) + \log p(\mathbf{x}\mid\alpha,\beta) \\
&= -d_f(\mathbf{y},\mathbf{x}) - \beta d_f\left(\frac{\alpha}{\beta},\mathbf{x}\right) + \psi_f(\mathbf{y},\alpha,\beta) \tag*{(82)}
\end{aligned} log p ( x ∣ y , α , β ) = ℓ ( y ; x ) + log p ( x ∣ α , β ) = − d f ( y , x ) − β d f ( β α , x ) + ψ f ( y , α , β ) (82)
where the term ψ f ( y , α , β ) \psi_f(\mathbf{y},\alpha,\beta) ψ f ( y , α , β ) does not depend on the parameter x \mathbf{x} x . Starting from the likelihood and prior,
p ( y ∣ x ) = exp ( − d f ( y , x ) + ϕ f ( y ) ) , p(\mathbf{y}|\mathbf{x})=\exp\left(-d_f(\mathbf{y},\mathbf{x})+\phi_f(\mathbf{y})\right), p ( y ∣ x ) = exp ( − d f ( y , x ) + ϕ f ( y ) ) ,
p ( x ∣ α , β ) = exp ( − β d f ( α β , x ) + φ f ( α , β ) ) , p(\mathbf{x}|\alpha,\beta)=\exp\left(-\beta d_f\left(\frac{\alpha}{\beta},\mathbf{x}\right)+\varphi_f(\alpha,\beta)\right), p ( x ∣ α , β ) = exp ( − β d f ( β α , x ) + φ f ( α , β ) ) ,
applying Bayes’ rule gives
p ( x ∣ y , α , β ) = p ( y ∣ x ) p ( x ∣ α , β ) p ( y ∣ α , β ) = exp ( − d f ( y , x ) − β d f ( α β , x ) + ϕ f ( y ) + φ f ( α , β ) ) p ( y ∣ α , β ) . \begin{aligned}
p(\mathbf{x}|\mathbf{y},\alpha,\beta)=\frac{p(\mathbf{y}|\mathbf{x})p(\mathbf{x}|\alpha,\beta)}{p(\mathbf{y}|\alpha,\beta)} \\
&=\frac{\exp\left(-d_f(\mathbf{y},\mathbf{x})-\beta d_f\left(\frac{\alpha}{\beta},\mathbf{x}\right)+\phi_f(\mathbf{y})+\varphi_f(\alpha,\beta)\right)}{p(\mathbf{y}|\alpha,\beta)}.
\end{aligned} p ( x ∣ y , α , β ) = p ( y ∣ α , β ) p ( y ∣ x ) p ( x ∣ α , β ) = p ( y ∣ α , β ) exp ( − d f ( y , x ) − β d f ( β α , x ) + ϕ f ( y ) + φ f ( α , β ) ) .
Taking the logarithm yields
log p ( x ∣ y , α , β ) = − d f ( y , x ) − β d f ( α β , x ) + ϕ f ( y ) + φ f ( α , β ) − log p ( y ∣ α , β ) . \log p(\mathbf{x}|\mathbf{y},\alpha,\beta)=-d_f(\mathbf{y},\mathbf{x})-\beta d_f\left(\frac{\alpha}{\beta},\mathbf{x}\right)+\phi_f(\mathbf{y})+\varphi_f(\alpha,\beta)-\log p(\mathbf{y}|\alpha,\beta). log p ( x ∣ y , α , β ) = − d f ( y , x ) − β d f ( β α , x ) + ϕ f ( y ) + φ f ( α , β ) − log p ( y ∣ α , β ) .
Hence,
log p ( x ∣ y , α , β ) = − d f ( y , x ) − β d f ( α β , x ) + ψ f ( y , α , β ) , \log p(\mathbf{x}|\mathbf{y},\alpha,\beta)=-d_f(\mathbf{y},\mathbf{x})-\beta d_f\left(\frac{\alpha}{\beta},\mathbf{x}\right)+\psi_f(\mathbf{y},\alpha,\beta), log p ( x ∣ y , α , β ) = − d f ( y , x ) − β d f ( β α , x ) + ψ f ( y , α , β ) ,
where
ψ f ( y , α , β ) = ϕ f ( y ) + φ f ( α , β ) − log p ( y ∣ α , β ) . \psi_f(\mathbf{y},\alpha,\beta)=\phi_f(\mathbf{y})+\varphi_f(\alpha,\beta)-\log p(\mathbf{y}|\alpha,\beta). ψ f ( y , α , β ) = ϕ f ( y ) + φ f ( α , β ) − log p ( y ∣ α , β ) .
Furthermore, the marginal likelihood is
p ( y ∣ α , β ) = ∫ X p ( y ∣ x ) p ( x ∣ α , β ) d x = exp ( ϕ f ( y ) + φ f ( α , β ) ) ∫ X exp ( − d f ( y , x ) − β d f ( α β , x ) ) d x . \begin{aligned}
p(\mathbf{y}|\alpha,\beta)=\int_{\mathcal{X}}p(\mathbf{y}|\mathbf{x})p(\mathbf{x}|\alpha,\beta)\,d\mathbf{x} \\
&=\exp\left(\phi_f(\mathbf{y})+\varphi_f(\alpha,\beta)\right)\int_{\mathcal{X}}\exp\left(-d_f(\mathbf{y},\mathbf{x})-\beta d_f\left(\frac{\alpha}{\beta},\mathbf{x}\right)\right)\,d\mathbf{x}.
\end{aligned} p ( y ∣ α , β ) = ∫ X p ( y ∣ x ) p ( x ∣ α , β ) d x = exp ( ϕ f ( y ) + φ f ( α , β ) ) ∫ X exp ( − d f ( y , x ) − β d f ( β α , x ) ) d x .
Substituting this expression into ψ f ( y , α , β ) \psi_f(\mathbf{y},\alpha,\beta) ψ f ( y , α , β ) gives
ψ f ( y , α , β ) = − log ∫ X exp ( − d f ( y , x ) − β d f ( α β , x ) ) d x . \psi_f(\mathbf{y},\alpha,\beta)=-\log\int_{\mathcal{X}}\exp\left(-d_f(\mathbf{y},\mathbf{x})-\beta d_f\left(\frac{\alpha}{\beta},\mathbf{x}\right)\right)\,d\mathbf{x}. ψ f ( y , α , β ) = − log ∫ X exp ( − d f ( y , x ) − β d f ( β α , x ) ) d x .
Lemma 5 shows that the log-posterior can be written as a weighted sum of Bregman divergences with a common generator. We can now show that it can be rewritten as a single Bregman divergence. Indeed, using the definition of the Bregman divergence,
d f ( y , x ) + β d f ( α β , x ) = f ( y ) − f ( x ) − ⟨ y − x , ∇ f ( x ) ⟩ + β [ f ( α β ) − f ( x ) − ⟨ α β − x , ∇ f ( x ) ⟩ ] = f ( y ) + β f ( α β ) − ( 1 + β ) f ( x ) − ⟨ y + α − ( 1 + β ) x , ∇ f ( x ) ⟩ \begin{aligned}
d_f(\mathbf{y},\mathbf{x})+\beta d_f\left(\frac{\alpha}{\beta},\mathbf{x}\right)=f(\mathbf{y})-f(\mathbf{x})-\langle\mathbf{y}-\mathbf{x},\nabla f(\mathbf{x})\rangle+\beta\left[f\left(\frac{\alpha}{\beta}\right)-f(\mathbf{x})-\left\langle\frac{\alpha}{\beta}-\mathbf{x},\nabla f(\mathbf{x})\right\rangle\right] \\
&=f(\mathbf{y})+\beta f\left(\frac{\alpha}{\beta}\right)-(1+\beta)f(\mathbf{x})-\langle\mathbf{y}+\alpha-(1+\beta)\mathbf{x},\nabla f(\mathbf{x})\rangle
\end{aligned} d f ( y , x ) + β d f ( β α , x ) = f ( y ) − f ( x ) − ⟨ y − x , ∇ f ( x )⟩ + β [ f ( β α ) − f ( x ) − ⟨ β α − x , ∇ f ( x ) ⟩ ] = f ( y ) + β f ( β α ) − ( 1 + β ) f ( x ) − ⟨ y + α − ( 1 + β ) x , ∇ f ( x )⟩
Since X \mathcal{X} X is convex and y , α β ∈ dom ( f ) \mathbf{y},\frac{\alpha}{\beta}\in\operatorname{dom}(f) y , β α ∈ dom ( f ) , their weighted average
y 1 + β + β 1 + β ( α β ) = y + α 1 + β ≔ α + β + \frac{\mathbf{y}}{1+\beta}+\frac{\beta}{1+\beta}\left(\frac{\alpha}{\beta}\right)=\frac{\mathbf{y}+\alpha}{1+\beta}\coloneqq\frac{\alpha^{+}}{\beta^{+}} 1 + β y + 1 + β β ( β α ) = 1 + β y + α : = β + α +
also belongs to dom ( f ) \operatorname{dom}(f) dom ( f ) . Thus
d f ( y , x ) + β d f ( α β , x ) = β + [ f ( α + β + ) − f ( x ) − ⟨ α + β + − x , ∇ f ( x ) ⟩ ] + f ( y ) + β f ( α β ) − β + f ( α + β + ) = β + d f ( α + β + , x ) + f ( y ) + β f ( α β ) − β + f ( α + β + ) . \begin{aligned}
d_f(\mathbf{y},\mathbf{x})+\beta d_f\left(\frac{\alpha}{\beta},\mathbf{x}\right)=\beta^{+}\left[f\left(\frac{\alpha^{+}}{\beta^{+}}\right)-f(\mathbf{x})-\left\langle\frac{\alpha^{+}}{\beta^{+}}-\mathbf{x},\nabla f(\mathbf{x})\right\rangle\right]+f(\mathbf{y})+\beta f\left(\frac{\alpha}{\beta}\right)-\beta^{+}f\left(\frac{\alpha^{+}}{\beta^{+}}\right) \\
&=\beta^{+}d_f\left(\frac{\alpha^{+}}{\beta^{+}},\mathbf{x}\right)+f(\mathbf{y})+\beta f\left(\frac{\alpha}{\beta}\right)-\beta^{+}f\left(\frac{\alpha^{+}}{\beta^{+}}\right).
\end{aligned} d f ( y , x ) + β d f ( β α , x ) = β + [ f ( β + α + ) − f ( x ) − ⟨ β + α + − x , ∇ f ( x ) ⟩ ] + f ( y ) + β f ( β α ) − β + f ( β + α + ) = β + d f ( β + α + , x ) + f ( y ) + β f ( β α ) − β + f ( β + α + ) .
Finally, plugging this expansion into the expression of the log-posterior
log p ( x ∣ y , α , β ) = − d f ( y , x ) − β d f ( α β , x ) + ψ f ( y , α , β ) (83) \log p(\mathbf{x}\mid\mathbf{y},\alpha,\beta)=-d_f(\mathbf{y},\mathbf{x})-\beta d_f\left(\frac{\alpha}{\beta},\mathbf{x}\right)+\psi_f(\mathbf{y},\alpha,\beta) \tag*{(83)} log p ( x ∣ y , α , β ) = − d f ( y , x ) − β d f ( β α , x ) + ψ f ( y , α , β ) (83)
exhibited in Lemma 5, then it follows
log p ( x ∣ y , α , β ) = − β + d f ( α + β + , x ) + Ψ f ( y , α , β ) \log p(\mathbf{x}\mid\mathbf{y},\alpha,\beta)=-\beta^{+}d_f\left(\frac{\alpha^{+}}{\beta^{+}},\mathbf{x}\right)+\Psi_f(\mathbf{y},\alpha,\beta) log p ( x ∣ y , α , β ) = − β + d f ( β + α + , x ) + Ψ f ( y , α , β )
where
Ψ f ( y , α , β ) = ψ f ( y , α , β ) − f ( y ) − β f ( α β ) + β + f ( α + β + ) \Psi_f(\mathbf{y},\alpha,\beta)=\psi_f(\mathbf{y},\alpha,\beta)-f(\mathbf{y})-\beta f\left(\frac{\alpha}{\beta}\right)+\beta^{+}f\left(\frac{\alpha^{+}}{\beta^{+}}\right) Ψ f ( y , α , β ) = ψ f ( y , α , β ) − f ( y ) − β f ( β α ) + β + f ( β + α + )
is independent of x \mathbf{x} x .
Proof of Proposition 6
In natural coordinates, the negative log-density associated with a distribution belonging to the regular exponential family writes (see Definition 5)
λ ( y ; t ) = ψ ( t ) − ⟨ t , y ⟩ − log p 0 ( y ) . (84) \lambda(\mathbf{y};\mathbf{t})=\psi(\mathbf{t})-\langle\mathbf{t},\mathbf{y}\rangle-\log p_0(\mathbf{y}). \tag*{(84)} λ ( y ; t ) = ψ ( t ) − ⟨ t , y ⟩ − log p 0 ( y ) . (84)
According to Proposition 1, this log-density admits a Bregman representation once ψ = f ∗ \psi=f^{*} ψ = f ∗ with t = ∇ f ( x ) \mathbf{t}=\nabla f(\mathbf{x}) t = ∇ f ( x ) . Consider the Legendre function h = β f ∗ h=\beta f^{*} h = β f ∗ and dual coordinates t \mathbf{t} t associated with the β \beta β -scaled dual Bregman representation identified in Theorem 7. Then
ζ ( t ) = L h ( t ) − λ ( y ; t ) (85) \zeta(\mathbf{t})=Lh(\mathbf{t})-\lambda(\mathbf{y};\mathbf{t}) \tag*{(85)} ζ ( t ) = L h ( t ) − λ ( y ; t ) (85)
= L β f ∗ ( t ) − f ∗ ( t ) + ⟨ t , y ⟩ + log p 0 ( y ) (86) =L\beta f^{*}(\mathbf{t})-f^{*}(\mathbf{t})+\langle\mathbf{t},\mathbf{y}\rangle+\log p_0(\mathbf{y}) \tag*{(86)} = L β f ∗ ( t ) − f ∗ ( t ) + ⟨ t , y ⟩ + log p 0 ( y ) (86)
= ( L β − 1 ) f ∗ ( t ) + ⟨ t , y ⟩ + log p 0 ( y ) . (87) =(L\beta-1)f^{*}(\mathbf{t})+\langle\mathbf{t},\mathbf{y}\rangle+\log p_0(\mathbf{y}). \tag*{(87)} = ( L β − 1 ) f ∗ ( t ) + ⟨ t , y ⟩ + log p 0 ( y ) . (87)
Since f ∗ f^{*} f ∗ is convex and the remaining terms are affine in t \mathbf{t} t , the function ζ \zeta ζ is convex on T \mathcal{T} T whenever L β − 1 ≥ 0 L\beta-1\geq0 L β − 1 ≥ 0 . Hence, λ ( y ; ⋅ ) \lambda(\mathbf{y};\cdot) λ ( y ; ⋅ ) is L L L -smooth relative to h h h for every L ≥ 1 / β L\geq1/\beta L ≥ 1/ β .
Proof of Proposition 7
In natural coordinates, the negative log-density associated with a distribution belonging to the regular exponential family writes (see Definition 5)
λ ( y ; t ) = ψ ( t ) − t y − log p 0 ( y ) . (88) \lambda(\mathbf{y};t)=\psi(t)-ty-\log p_0(y). \tag*{(88)} λ ( y ; t ) = ψ ( t ) − t y − log p 0 ( y ) . (88)
According to Proposition 1, this log-density admits a Bregman representation once ψ = f ∗ \psi=f^{*} ψ = f ∗ with t = f ′ ( x ) t=f'(x) t = f ′ ( x ) . Hence, the negative log-likelihood writes
− ℓ ( y ; x ) = f ∗ ( f ′ ( x ) ) − f ′ ( x ) y − log p 0 ( y ) . (89) -\ell(y;x)=f^{*}(f'(x))-f'(x)y-\log p_0(y). \tag*{(89)} − ℓ ( y ; x ) = f ∗ ( f ′ ( x )) − f ′ ( x ) y − log p 0 ( y ) . (89)
and
− ∂ ∂ x ℓ ( y ; x ) = x f ′ ′ ( x ) − y f ′ ′ ( x ) = ( x − y ) f ′ ′ ( x ) . -\frac{\partial}{\partial x}\ell(y;x)=xf''(x)-yf''(x)=(x-y)f''(x). − ∂ x ∂ ℓ ( y ; x ) = x f ′′ ( x ) − y f ′′ ( x ) = ( x − y ) f ′′ ( x ) .
Moreover, the dual coordinates induced by the α \alpha α -scaled Bregman representation identified in Theorem 7 are defined as t = τ ( x ) t=\tau(x) t = τ ( x ) with
τ ( x ) = f ∗ ( f ′ ( x ) ) = x f ′ ( x ) − f ( x ) , \tau(x)=f^*(f'(x))=xf'(x)-f(x), τ ( x ) = f ∗ ( f ′ ( x )) = x f ′ ( x ) − f ( x ) ,
which yields
d τ ( x ) d x = x f ′ ′ ( x ) , \frac{d\tau(x)}{dx}=xf''(x), d x d τ ( x ) = x f ′′ ( x ) ,
and
d x d t = 1 x f ′ ′ ( x ) (90) \frac{dx}{dt}=\frac{1}{xf''(x)} \tag*{(90)} d t d x = x f ′′ ( x ) 1 (90)
By defining λ ( y ; t ) = − ℓ ( y ; τ − 1 ( t ) ) \lambda(y;t)=-\ell(y;\tau^{-1}(t)) λ ( y ; t ) = − ℓ ( y ; τ − 1 ( t )) , it follows that
λ ′ ( y ; t ) = − ∂ ∂ x ℓ ( y ; x ) d x d t = x − y x = 1 − y x , \lambda'(y;t)=-\frac{\partial}{\partial x}\ell(y;x)\frac{dx}{dt}=\frac{x-y}{x}=1-\frac{y}{x}, λ ′ ( y ; t ) = − ∂ x ∂ ℓ ( y ; x ) d t d x = x x − y = 1 − x y ,
Differentiating once more gives
λ ′ ′ ( y ; t ) = y x 3 f ′ ′ ( x ) (91) \lambda''(y;t)=\frac{y}{x^3f''(x)} \tag*{(91)} λ ′′ ( y ; t ) = x 3 f ′′ ( x ) y (91)
By setting h ( t ) = − α f ′ ( τ − 1 ( t ) ) h(t)=-\alpha f'(\tau^{-1}(t)) h ( t ) = − α f ′ ( τ − 1 ( t )) and t = τ ( x ) t=\tau(x) t = τ ( x ) as in Theorem 7, one has
h ′ ( t ) = − α d f ′ ( x ) d x d x d t , h'(t)=-\alpha\frac{df'(x)}{dx}\frac{dx}{dt}, h ′ ( t ) = − α d x d f ′ ( x ) d t d x ,
and, plugging (90) , it yields
h ′ ( t ) = − α x , h'(t)=-\frac{\alpha}{x}, h ′ ( t ) = − x α ,
and
h ′ ′ ( t ) = α x 3 f ′ ′ ( x ) (92) h''(t)=\frac{\alpha}{x^3f''(x)} \tag*{(92)} h ′′ ( t ) = x 3 f ′′ ( x ) α (92)
Consider the function
ζ ( t ) = L h ( t ) − λ ( y ; t ) . (93) \zeta(t)=Lh(t)-\lambda(y;t). \tag*{(93)} ζ ( t ) = L h ( t ) − λ ( y ; t ) . (93)
Then, using (91) and (92)
ζ ′ ′ ( t ) = ( L − y α ) h ′ ′ ( t ) (94) \zeta''(t)=\left(L-\frac{y}{\alpha}\right)h''(t) \tag*{(94)} ζ ′′ ( t ) = ( L − α y ) h ′′ ( t ) (94)
Since h h h is convex, h ′ ′ ( t ) ≥ 0 h''(t)\ge0 h ′′ ( t ) ≥ 0 . Therefore,
if y ≠ 0 y\ne0 y = 0 , ζ \zeta ζ is convex whenever L ≥ y α L\ge\frac{y}{\alpha} L ≥ α y . Thus, the function λ ( y ; ⋅ ) \lambda(y;\cdot) λ ( y ; ⋅ ) is L L L -smooth relative to h h h for every L ≥ y / α L\ge y/\alpha L ≥ y / α .
if y = 0 y=0 y = 0 , for any ϵ > 0 \epsilon>0 ϵ > 0 , the function λ ( y ; ⋅ ) \lambda(y;\cdot) λ ( y ; ⋅ ) is L L L -smooth relative to h h h for every L ≥ ϵ L\ge\epsilon L ≥ ϵ .
Proof of Theorem 11
Let define
t = τ ( x ) . \mathbf{t}=\tau(\mathbf{x}). t = τ ( x ) .
For each component,
t i = τ i ( x i ) = x i f i ′ ( x i ) − f i ( x i ) , t_i=\tau_i(x_i)=x_i f_i'(x_i)-f_i(x_i), t i = τ i ( x i ) = x i f i ′ ( x i ) − f i ( x i ) ,
so that
d t i d x i = x i f i ′ ′ ( x i ) , d x i d t i = 1 x i f i ′ ′ ( x i ) . \frac{dt_i}{dx_i}=x_i f_i''(x_i),\qquad\frac{dx_i}{dt_i}=\frac{1}{x_i f_i''(x_i)}. d x i d t i = x i f i ′′ ( x i ) , d t i d x i = x i f i ′′ ( x i ) 1 .
Since the model is separable, the negative log-likelihood can be written as
ℓ ˉ ( y ; x ) = ∑ i = 1 n [ f i ∗ ( f i ′ ( x i ) ) − y i f i ′ ( x i ) ] − ϕ f ( y ) , \bar{\ell}(\mathbf{y};\mathbf{x})=\sum_{i=1}^{n}\left[f_i^*(f_i'(x_i))-y_i f_i'(x_i)\right]-\phi_f(\mathbf{y}), ℓ ˉ ( y ; x ) = i = 1 ∑ n [ f i ∗ ( f i ′ ( x i )) − y i f i ′ ( x i ) ] − ϕ f ( y ) ,
where ϕ f ( y ) \phi_f(\mathbf{y}) ϕ f ( y ) does not depend on x \mathbf{x} x . Hence,
∂ ∂ x i ℓ ˉ ( y ; x ) = ( x i − y i ) f i ′ ′ ( x i ) . \frac{\partial}{\partial x_i}\bar{\ell}(\mathbf{y};\mathbf{x})=(x_i-y_i)f_i''(x_i). ∂ x i ∂ ℓ ˉ ( y ; x ) = ( x i − y i ) f i ′′ ( x i ) .
Therefore,
∂ λ ∂ t i = ∂ ℓ ˉ ∂ x i d x i d t i = ( x i − y i ) f i ′ ′ ( x i ) x i f i ′ ′ ( x i ) = 1 − y i x i . \frac{\partial\lambda}{\partial t_i}=\frac{\partial\bar{\ell}}{\partial x_i}\frac{dx_i}{dt_i}=\frac{(x_i-y_i)f_i''(x_i)}{x_i f_i''(x_i)}=1-\frac{y_i}{x_i}. ∂ t i ∂ λ = ∂ x i ∂ ℓ ˉ d t i d x i = x i f i ′′ ( x i ) ( x i − y i ) f i ′′ ( x i ) = 1 − x i y i .
Differentiating once more with respect to t i t_i t i gives
∂ 2 λ ∂ t i 2 = y i x i 2 d x i d t i = y i x i 3 f i ′ ( x i ) . \frac{\partial^2\lambda}{\partial t_i^2}=\frac{y_i}{x_i^2}\frac{dx_i}{dt_i}=\frac{y_i}{x_i^3 f_i'(x_i)}. ∂ t i 2 ∂ 2 λ = x i 2 y i d t i d x i = x i 3 f i ′ ( x i ) y i .
On the other hand, by the one-dimensional α \alpha α -scaled construction,
h i ′ ( t i ) = − α i x i , h_i'(t_i)=-\frac{\alpha_i}{x_i}, h i ′ ( t i ) = − x i α i ,
and therefore
h i ′ ′ ( t i ) = α i x i 2 d x i d t i = α i x i 3 f i ′ ′ ( x i ) . h_i''(t_i)=\frac{\alpha_i}{x_i^2}\frac{dx_i}{dt_i}=\frac{\alpha_i}{x_i^3 f_i''(x_i)}. h i ′′ ( t i ) = x i 2 α i d t i d x i = x i 3 f i ′′ ( x i ) α i .
Hence,
∂ 2 λ ∂ t i 2 = y i α i h i ′ ′ ( t i ) . \frac{\partial^2\lambda}{\partial t_i^2}=\frac{y_i}{\alpha_i}h_i''(t_i). ∂ t i 2 ∂ 2 λ = α i y i h i ′′ ( t i ) .
Since h s e p h_{\mathrm{sep}} h sep is separable and λ ( y ; ⋅ ) \lambda(\mathbf{y};\cdot) λ ( y ; ⋅ ) is separable in the transformed coordinates, all mixed second derivatives vanish. Consequently,
∇ 2 ( L h s e p − λ ( y ; ⋅ ) ) = Diag [ ( L − y i α i ) h i ′ ′ ( t i ) ] i = 1 n . \nabla^2\left(Lh_{\mathrm{sep}}-\lambda(\mathbf{y};\cdot)\right)=\operatorname{Diag}\left[\left(L-\frac{y_i}{\alpha_i}\right)h_i''(t_i)\right]_{i=1}^{n}. ∇ 2 ( L h sep − λ ( y ; ⋅ ) ) = Diag [ ( L − α i y i ) h i ′′ ( t i ) ] i = 1 n .
Since each h i h_i h i is convex, h i ′ ′ ( t i ) ≥ 0 h_i''(t_i)\ge0 h i ′′ ( t i ) ≥ 0 . Thus,
L ≥ max 1 ≤ i ≤ n y i α i . L\ge\max_{1\le i\le n}\frac{y_i}{\alpha_i}. L ≥ 1 ≤ i ≤ n max α i y i .
Hence, the function λ ( y ; ⋅ ) \lambda(\mathbf{y};\cdot) λ ( y ; ⋅ ) is L L L -smooth relative to h s e p h_{\mathrm{sep}} h sep for every
L ≥ ∥ y ∥ ∞ ∥ α ∥ ∞ . L\ge\frac{\|\mathbf{y}\|_\infty}{\|\boldsymbol{\alpha}\|_\infty}. L ≥ ∥ α ∥ ∞ ∥ y ∥ ∞ .
We now show that, since each h i h_i h i is strictly convex, the threshold L min = ∥ y / α ∥ ∞ L_{\min}=\lVert y/\alpha\rVert_{\infty} L m i n = ∥ y / α ∥ ∞ is sharp. Indeed, let i ⋆ ∈ argmax 1 ≤ i ≤ n y i α i i^\star\in\operatorname*{argmax}_{1\le i\le n}\frac{y_i}{\alpha_i} i ⋆ ∈ argmax 1 ≤ i ≤ n α i y i . If L < ∥ y / α ∥ ∞ L<\lVert y/\alpha\rVert_{\infty} L < ∥ y / α ∥ ∞ , then L − y i ⋆ α i ⋆ < 0 L-\frac{y_{i^\star}}{\alpha_{i^\star}}<0 L − α i ⋆ y i ⋆ < 0 . Choose t \mathbf{t} t and t ′ \mathbf{t}' t ′ that differ only in the i ⋆ i^\star i ⋆ -th coordinate. Since h i ⋆ h_{i^\star} h i ⋆ is strictly convex, d h i ⋆ ( t i ⋆ , t i ⋆ ′ ) > 0 d_{h_{i^\star}}(t_{i^\star},t'_{i^\star})>0 d h i ⋆ ( t i ⋆ , t i ⋆ ′ ) > 0 whenever t i ⋆ ≠ t i ⋆ ′ t_{i^\star}\ne t'_{i^\star} t i ⋆ = t i ⋆ ′ . Moreover, since the two points differ only in the i ⋆ i^\star i ⋆ -th coordinate, d h j ( t j , t j ′ ) = 0 d_{h_j}(t_j,t'_j)=0 d h j ( t j , t j ′ ) = 0 , j ≠ i ⋆ j\ne i^\star j = i ⋆ . Therefore,
d ζ ( t , t ′ ) = ( L − y i ⋆ α i ⋆ ) d h i ⋆ ( t i ⋆ , t i ⋆ ′ ) < 0. d_{\zeta}(\mathbf{t},\mathbf{t}')=\left(L-\frac{y_{i^\star}}{\alpha_{i^\star}}\right)d_{h_{i^\star}}(t_{i^\star},t'_{i^\star})<0. d ζ ( t , t ′ ) = ( L − α i ⋆ y i ⋆ ) d h i ⋆ ( t i ⋆ , t i ⋆ ′ ) < 0.
Hence, ζ \zeta ζ is not convex. Thus, no L < ∥ y / α ∥ ∞ L<\lVert y/\alpha\rVert_{\infty} L < ∥ y / α ∥ ∞ can satisfy the relative smoothness condition, which proves that the threshold is sharp. At the threshold L min = ∥ y / α ∥ ∞ L_{\min}=\lVert y/\alpha\rVert_{\infty} L m i n = ∥ y / α ∥ ∞ , the coefficient associated with each component satisfying y i α i = ∥ y / α ∥ ∞ \frac{y_i}{\alpha_i}=\lVert y/\alpha\rVert_{\infty} α i y i = ∥ y / α ∥ ∞ vanishes, whereas it remains strictly positive for every component satisfying y i α i < ∥ y / α ∥ ∞ \frac{y_i}{\alpha_i}<\lVert y/\alpha\rVert_{\infty} α i y i < ∥ y / α ∥ ∞ . Consequently, ζ \zeta ζ is convex but not strictly convex at the threshold.
Proof of Lemma 2
Let λ ˘ ( y ; t ) \breve{\lambda}(y;t) λ ˘ ( y ; t ) the negative log-likelihood function associated with a Poisson linear inverse problem expressed in the dual coordinates t = log ( x ) t=\log(x) t = log ( x ) induced by a component-wise β \beta β -scaled dual Bregman representation, i.e.,
λ ˘ ( y ; t ) = − ℓ ( y ; K e t ) (95) \breve{\lambda}(y;t)=-\ell(y;K\mathrm{e}^{t}) \tag*{(95)} λ ˘ ( y ; t ) = − ℓ ( y ; K e t ) (95)
= d f ( y ; K e t ) − ϕ f ( y ) . (96) = d_f(y;K\mathrm{e}^{t})-\phi_f(y). \tag*{(96)} = d f ( y ; K e t ) − ϕ f ( y ) . (96)
where the operator K ∈ R m × n K\in\mathbb{R}^{m\times n} K ∈ R m × n has non negative entries: k i j ≥ 0 k_{ij}\ge0 k ij ≥ 0 for 1 ≤ i ≤ m 1\le i\le m 1 ≤ i ≤ m , 1 ≤ j ≤ n 1\le j\le n 1 ≤ j ≤ n . Let denote by s j s_j s j the sum of its j t h j^{\mathrm{th}} j th column: s j = ∑ i = 1 m k i j s_j=\sum_{i=1}^{m}k_{ij} s j = ∑ i = 1 m k ij . Consider the intensity model, for all 1 ≤ i ≤ m 1\le i\le m 1 ≤ i ≤ m
u i ( t ) = [ K exp t ] i = ∑ j = 1 n k i j e t j (97) u_i(t)=[K\exp t]_i=\sum_{j=1}^{n}k_{ij}\mathrm{e}^{t_j} \tag*{(97)} u i ( t ) = [ K exp t ] i = j = 1 ∑ n k ij e t j (97)
such that the objective function λ ˘ ( y ; t ) \breve{\lambda}(y;t) λ ˘ ( y ; t ) writes
λ ˘ ( y ; t ) = ∑ i = 1 m ( y i log y i − y i log u i ( t ) + u i ( t ) − y i ) − ϕ f ( y ) (98) \breve{\lambda}(y;t)=\sum_{i=1}^{m}\left(y_i\log y_i-y_i\log u_i(t)+u_i(t)-y_i\right)-\phi_f(y) \tag*{(98)} λ ˘ ( y ; t ) = i = 1 ∑ m ( y i log y i − y i log u i ( t ) + u i ( t ) − y i ) − ϕ f ( y ) (98)
For β j > 0 \beta_j>0 β j > 0 , let define h s e p ( t ) = ∑ j = 1 n β j e t j h_{\mathrm{sep}}(t)=\sum_{j=1}^{n}\beta_j\mathrm{e}^{t_j} h sep ( t ) = ∑ j = 1 n β j e t j . Since λ ˘ ( y ; ⋅ ) \breve{\lambda}(y;\cdot) λ ˘ ( y ; ⋅ ) and h h h are C 2 C^2 C 2 on R n \mathbb{R}^n R n , the convexity of L h s e p − λ ˘ ( y ; ⋅ ) Lh_{\mathrm{sep}}-\breve{\lambda}(y;\cdot) L h sep − λ ˘ ( y ; ⋅ ) is warranted for any L > 0 L>0 L > 0 such that for any v ∈ R n \mathbf{v}\in\mathbb{R}^n v ∈ R n and t ∈ R n \mathbf{t}\in\mathbb{R}^n t ∈ R n ,
v T ∇ 2 λ ˘ ( y ; t ) v ≤ L v T ∇ 2 h s e p ( t ) v (99) \mathbf{v}^{\mathsf{T}}\nabla^2\breve{\lambda}(y;\mathbf{t})\mathbf{v}\le L\mathbf{v}^{\mathsf{T}}\nabla^2h_{\mathrm{sep}}(\mathbf{t})\mathbf{v} \tag*{(99)} v T ∇ 2 λ ˘ ( y ; t ) v ≤ L v T ∇ 2 h sep ( t ) v (99)
h h h -Hessian quadratic form. Since h s e p ( t ) h_{\mathrm{sep}}(\mathbf{t}) h sep ( t ) is separable, its Hessian is diagonal
∇ 2 h s e p ( t ) = diag ( β 1 e t 1 , … , β n e t n ) . (100) \nabla^2h_{\mathrm{sep}}(\mathbf{t})=\operatorname{diag}\left(\beta_1\mathrm{e}^{t_1},\ldots,\beta_n\mathrm{e}^{t_n}\right). \tag*{(100)} ∇ 2 h sep ( t ) = diag ( β 1 e t 1 , … , β n e t n ) . (100)
For any vector v ∈ R n \mathbf{v}\in\mathbb{R}^n v ∈ R n ,
v T ∇ 2 h s e p ( t ) v = ∑ j = 1 n β j v j 2 e t j (101) \mathbf{v}^{\mathsf{T}}\nabla^2h_{\mathrm{sep}}(\mathbf{t})\mathbf{v}=\sum_{j=1}^{n}\beta_jv_j^2\mathrm{e}^{t_j} \tag*{(101)} v T ∇ 2 h sep ( t ) v = j = 1 ∑ n β j v j 2 e t j (101)
λ ~ \tilde{\lambda} λ ~ -Hessian quadratic form. Differentiating λ ~ ( y ; t ) \tilde{\lambda}(\mathbf{y};\mathbf{t}) λ ~ ( y ; t ) twice with respect to t j t_j t j and t k t_k t k yields the quadratic form, for any v ∈ R n \mathbf{v} \in\mathbb{R}^n v ∈ R n ,
v T ∇ 2 λ ~ ( y ; t ) v = ∑ j = 1 n v j 2 e t j ∑ i = 1 m k i j − ∑ j = 1 n v j e t j ∑ i = 1 m y i k i j u i ( t ) + ∑ i = 1 m y i u i ( t ) 2 ( ∑ j = 1 n k i j v j exp t j ) 2 = ∑ j = 1 n s j v j 2 e t j + ∑ i = 1 m y i [ ( ∑ j = 1 n k i j e t j u i ( t ) v j ) 2 − ∑ j = 1 n k i j e t j u i ( t ) v j 2 ] (102) \begin{aligned}
\mathbf{v}^{\mathsf{T}}\nabla^2\tilde{\lambda}(\mathbf{y};\mathbf{t})\mathbf{v}
= \sum_{j=1}^{n}v_j^2e^{t_j}\sum_{i=1}^{m}k_{ij}
-\sum_{j=1}^{n}v_j e^{t_j}\sum_{i=1}^{m}\frac{y_i k_{ij}}{u_i(\mathbf{t})}
+\sum_{i=1}^{m}\frac{y_i}{u_i(\mathbf{t})^2}\left(\sum_{j=1}^{n}k_{ij}v_j\exp t_j\right)^2 \\
&= \sum_{j=1}^{n}s_jv_j^2e^{t_j}
+\sum_{i=1}^{m}y_i\left[\left(\sum_{j=1}^{n}\frac{k_{ij}e^{t_j}}{u_i(\mathbf{t})}v_j\right)^2-\sum_{j=1}^{n}\frac{k_{ij}e^{t_j}}{u_i(\mathbf{t})}v_j^2\right] \tag*{(102)}
\end{aligned} v T ∇ 2 λ ~ ( y ; t ) v = j = 1 ∑ n v j 2 e t j i = 1 ∑ m k ij − j = 1 ∑ n v j e t j i = 1 ∑ m u i ( t ) y i k ij + i = 1 ∑ m u i ( t ) 2 y i ( j = 1 ∑ n k ij v j exp t j ) 2 = j = 1 ∑ n s j v j 2 e t j + i = 1 ∑ m y i ( j = 1 ∑ n u i ( t ) k ij e t j v j ) 2 − j = 1 ∑ n u i ( t ) k ij e t j v j 2 (102)
λ ~ \tilde{\lambda} λ ~ -upper bound using Jensen’s inequality. For each row i i i , define convex weights λ i j = k i j e t j u i ( t ) \lambda_{ij}=\frac{k_{ij}e^{t_j}}{u_i(\mathbf{t})} λ ij = u i ( t ) k ij e t j . Since k i j ≥ 0 k_{ij}\ge0 k ij ≥ 0 and u i ( t ) = ∑ j = 1 n k i j e t j u_i(\mathbf{t})=\sum_{j=1}^{n}k_{ij}e^{t_j} u i ( t ) = ∑ j = 1 n k ij e t j , the weights satisfy λ i j ≥ 0 \lambda_{ij}\ge0 λ ij ≥ 0 and ∑ j = 1 n λ i j = 1 \sum_{j=1}^{n}\lambda_{ij}=1 ∑ j = 1 n λ ij = 1 . Thus, by Jensen’s Inequality,
( ∑ j = 1 n λ i j v j ) 2 ≤ ∑ j = 1 n λ i j v j 2 (103) \left(\sum_{j=1}^{n}\lambda_{ij}v_j\right)^2\le\sum_{j=1}^{n}\lambda_{ij}v_j^2 \tag*{(103)} ( j = 1 ∑ n λ ij v j ) 2 ≤ j = 1 ∑ n λ ij v j 2 (103)
Since y i ≥ 0 y_i\ge0 y i ≥ 0 , multiplying by y i y_i y i and summing over i i i yields
v T ∇ 2 λ ~ ( y ; t ) v = ∑ j = 1 n s j v j 2 e t j + ∑ i = 1 m y i [ ( ∑ j = 1 n λ i j v j ) 2 − ∑ j = 1 n λ i j v j 2 ] ≤ ∑ j = 1 n s j v j 2 e t j . (104) \mathbf{v}^{\mathsf{T}}\nabla^2\tilde{\lambda}(\mathbf{y};\mathbf{t})\mathbf{v}
=\sum_{j=1}^{n}s_jv_j^2e^{t_j}
+\sum_{i=1}^{m}y_i\left[\left(\sum_{j=1}^{n}\lambda_{ij}v_j\right)^2-\sum_{j=1}^{n}\lambda_{ij}v_j^2\right]
\le\sum_{j=1}^{n}s_jv_j^2e^{t_j}. \tag*{(104)} v T ∇ 2 λ ~ ( y ; t ) v = j = 1 ∑ n s j v j 2 e t j + i = 1 ∑ m y i ( j = 1 ∑ n λ ij v j ) 2 − j = 1 ∑ n λ ij v j 2 ≤ j = 1 ∑ n s j v j 2 e t j . (104)
Single weight. Consider the choice β j = β = 1 \beta_j=\beta=1 β j = β = 1 for 1 ≤ j ≤ n 1\le j\le n 1 ≤ j ≤ n , such that
h s e p ( t ) = h 1 ( t ) = ∑ j = 1 n e t j (105) h_{\mathrm{sep}}(\mathbf{t})=h_1(\mathbf{t})=\sum_{j=1}^{n}e^{t_j} \tag*{(105)} h sep ( t ) = h 1 ( t ) = j = 1 ∑ n e t j (105)
Then, (101) writes
v T ∇ 2 h 1 ( t ) v = ∑ j = 1 n v j 2 e t j (106) \mathbf{v}^{\mathsf{T}}\nabla^2h_1(\mathbf{t})\mathbf{v}=\sum_{j=1}^{n}v_j^2e^{t_j} \tag*{(106)} v T ∇ 2 h 1 ( t ) v = j = 1 ∑ n v j 2 e t j (106)
Bounding each column sum s j s_j s j by the maximum column sum ∥ K ∥ 1 = max 1 ≤ j ≤ n s j \lVert K\rVert_1=\max_{1\le j\le n}s_j ∥ K ∥ 1 = max 1 ≤ j ≤ n s j in (104) , we get for all v ∈ R n \mathbf{v}\in\mathbb{R}^n v ∈ R n ,
v T ∇ 2 λ ~ ( y ; t ) v ≤ ∥ K ∥ 1 ∑ j = 1 n v j 2 e t j = ∥ K ∥ 1 v T ∇ 2 h 1 ( t ) v (107) \mathbf{v}^{\mathsf{T}}\nabla^2\tilde{\lambda}(\mathbf{y};\mathbf{t})\mathbf{v}
\le\lVert K\rVert_1\sum_{j=1}^{n}v_j^2e^{t_j}
=\lVert K\rVert_1\mathbf{v}^{\mathsf{T}}\nabla^2h_1(\mathbf{t})\mathbf{v} \tag*{(107)} v T ∇ 2 λ ~ ( y ; t ) v ≤ ∥ K ∥ 1 j = 1 ∑ n v j 2 e t j = ∥ K ∥ 1 v T ∇ 2 h 1 ( t ) v (107)
Thus, the function λ ~ ( y ; t ) \tilde{\lambda}(\mathbf{y};\mathbf{t}) λ ~ ( y ; t ) is L L L -smooth relative to h 1 h_1 h 1 for every L ≥ ∥ K ∥ 1 L\ge\lVert K\rVert_1 L ≥ ∥ K ∥ 1 .
Multiple weights. If we choose β j = s j \beta_j=s_j β j = s j for 1 ≤ j ≤ n 1\le j\le n 1 ≤ j ≤ n , such that
h s e p ( t ) = h s ( t ) = ∑ j = 1 n s j e t j (108) h_{\mathrm{sep}}(\mathbf{t})=h_s(\mathbf{t})=\sum_{j=1}^{n}s_je^{t_j} \tag*{(108)} h sep ( t ) = h s ( t ) = j = 1 ∑ n s j e t j (108)
Then, (101) writes
v T ∇ 2 h s ( t ) v = ∑ j = 1 n s j v j 2 e t j (109) \mathbf{v}^{\mathsf{T}}\nabla^2h_s(\mathbf{t})\mathbf{v}=\sum_{j=1}^{n}s_jv_j^2e^{t_j} \tag*{(109)} v T ∇ 2 h s ( t ) v = j = 1 ∑ n s j v j 2 e t j (109)
and, from (104) , we immediately get that for all v ∈ R n \mathbf{v}\in\mathbb{R}^n v ∈ R n ,
v T ∇ 2 λ ~ ( y ; t ) v ≤ v T ∇ 2 h s ( t ) v . (110) \mathbf{v}^{\mathsf{T}}\nabla^2\tilde{\lambda}(\mathbf{y};\mathbf{t})\mathbf{v}
\le\mathbf{v}^{\mathsf{T}}\nabla^2h_s(\mathbf{t})\mathbf{v}. \tag*{(110)} v T ∇ 2 λ ~ ( y ; t ) v ≤ v T ∇ 2 h s ( t ) v . (110)
Thus, the function λ ~ ( y ; t ) \tilde{\lambda}(\mathbf{y};\mathbf{t}) λ ~ ( y ; t ) is L L L -smooth relative to h s h_s h s for every L ≥ 1 L\ge1 L ≥ 1 .
Primal Bregman representations of classical exponential family distributions
Table 3 summarizes the primal Bregman representations of several exponential family distributions widely encountered in the literature. For each model, it reports the minimal sufficient statistic, the expectation parameter, the corresponding Legendre generator f f f , and the associated Bregman divergence d f d_f d f whenever it admits a convenient explicit expression in expectation coordinates. These geometries provide the starting point for the dual Bregman representations derived in § 3 .
For the Gamma model with unknown shape and rate parameters and the Beta model with both shape parameters unknown, the Legendre conjugate f ∗ f^* f ∗ of the generator (i.e., log-partition function) is explicit in natural coordinates, whereas the generator itself f f f does not admit a simple closed-form expression in expectation coordinates. For the Gaussian model with unknown mean and variance, the Gamma model with unknown shape and rate, and the Beta model with both shape parameters unknown, a single observed sufficient statistic lies on the boundary of the corresponding mean-parameter space. Accordingly, the standard finite Bregman divergence d f ( y , x ) d_f(y,x) d f ( y , x ) is not reported in the last column.
Dual Bregman representations of classical exponential family distributions: detailed derivations
Gaussian distribution with known variance
β \beta β -scaled dual Bregman representation – Consider the Gaussian model with known variance σ 2 > 0 \sigma^2>0 σ 2 > 0 . In the expectation parametrization, the parameter is the mean x ∈ R x\in\mathbb{R} x ∈ R , and the associated convex generator is
f ( x ) = x 2 2 σ 2 , x ∈ R . f(x)=\frac{x^2}{2\sigma^2},\qquad x\in\mathbb{R}. f ( x ) = 2 σ 2 x 2 , x ∈ R .
Its Legendre conjugate is
f ∗ ( t ) = σ 2 t 2 2 , t ∈ R , f^{*}(t) = \frac{\sigma^{2}t^{2}}{2}, \qquad t \in\mathbb{R}, f ∗ ( t ) = 2 σ 2 t 2 , t ∈ R ,
while the dual coordinate mapping is
τ ( x ) = f ′ ( x ) = x σ 2 . \tau(x) = f'(x) = \frac{x}{\sigma^{2}}. τ ( x ) = f ′ ( x ) = σ 2 x .
By Theorem 7, the resulting dual Bregman representation of the log-density writes
d f ( z , x ) = d h ( τ ( x ) , τ ( z ) ) (111) d_f(z,x) = d_h(\tau(x),\tau(z)) \tag*{(111)} d f ( z , x ) = d h ( τ ( x ) , τ ( z )) (111)
= σ 2 2 ( x σ 2 − z σ 2 ) 2 = ( x − z ) 2 2 σ 2 . (112) = \frac{\sigma^{2}}{2}\left(\frac{x}{\sigma^{2}}-\frac{z}{\sigma^{2}}\right)^{2} = \frac{(x-z)^{2}}{2\sigma^{2}}. \tag*{(112)} = 2 σ 2 ( σ 2 x − σ 2 z ) 2 = 2 σ 2 ( x − z ) 2 . (112)
Consequently, the primal and dual representations of the density share the same quadratic Bregman geometry, up to the fixed scaling determined by the known variance σ 2 \sigma^{2} σ 2 . In this case, the primal and dual generators are both quadratic, reflecting the self-dual character of the Gaussian geometry under Legendre transformation.
α \alpha α -scaled dual Bregman representation – The alternative representation of Theorem 7 does not apply here, since its assumptions require a positive expectation-parameter domain X ⊆ R + ∗ \mathcal{X} \subseteq\mathbb{R}_{+}^{*} X ⊆ R + ∗ , whereas the Gaussian mean parameter ranges over R \mathbb{R} R .
Poisson distribution
β \beta β -scaled dual Bregman representation – Consider the Poisson model, for which the expectation parameter is the Poisson intensity itself. The corresponding Bregman generator is
f ( x ) = x log x − x , x ∈ R + ∗ . f(x) = x \log x - x, \qquad x \in\mathbb{R}_{+}^{*}. f ( x ) = x log x − x , x ∈ R + ∗ .
The corresponding Legendre conjugate is
f ∗ ( t ) = e t f^{*}(t) = e^{t} f ∗ ( t ) = e t
and the dual coordinate mapping is
τ ( x ) = f ′ ( x ) = log x . \tau(x) = f'(x) = \log x. τ ( x ) = f ′ ( x ) = log x .
By Theorem 7, the resulting dual Bregman representation of the log-density writes
d f ( z , x ) = d h ( τ ( x ) , τ ( z ) ) = x − z − z log ( x z ) , (113) d_f(z,x) = d_h(\tau(x),\tau(z)) = x-z-z\log\left(\frac{x}{z}\right), \tag*{(113)} d f ( z , x ) = d h ( τ ( x ) , τ ( z )) = x − z − z log ( z x ) , (113)
which can be identified as the generalized Kullback–Leibler divergence d K L ( z ∥ x ) d_{\mathrm{KL}}(z\Vert x) d KL ( z ∥ x ) .
α \alpha α -scaled dual Bregman representation – Adopting the alternative parametrization of Theorem 7 leads to a different dual geometry defined by
τ ( x ) = x , h ( t ) = − log t , \tau(x) = x, \qquad h(t) = -\log t, τ ( x ) = x , h ( t ) = − log t ,
leading to
1 z d f ( z , x ) = d h ( τ ( x ) , τ ( z ) ) = x z − log ( x z ) − 1 , (114) \frac{1}{z}d_f(z,x) = d_h(\tau(x),\tau(z)) = \frac{x}{z} - \log\left(\frac{x}{z}\right) - 1, \tag*{(114)} z 1 d f ( z , x ) = d h ( τ ( x ) , τ ( z )) = z x − log ( z x ) − 1 , (114)
which can be identified as the Itakura-Saito divergence d I S ( x , z ) d_{\mathrm{IS}}(x,z) d IS ( x , z ) . Thus, the alternative parametrization replaces the generalized Kullback-Leibler geometry by the Itakura-Saito geometry while preserving the geometric structure.
Exponential distribution
β \beta β -scaled dual Bregman representation – Considering the Exponential model, the expectation parameter belongs to R + ∗ \mathbb{R}_{+}^{*} R + ∗ and the associated generator is
f ( x ) = − log x . f(x)=-\log x. f ( x ) = − log x .
The corresponding gradient mapping is
τ ( x ) = f ′ ( x ) = − 1 x , \tau(x)=f'(x)=-\frac{1}{x}, τ ( x ) = f ′ ( x ) = − x 1 ,
while the Legendre conjugate reads
f ∗ ( t ) = − 1 − log ( − t ) , t < 0. f^{*}(t)=-1-\log(-t),\qquad t<0. f ∗ ( t ) = − 1 − log ( − t ) , t < 0.
Theorem 7 therefore yields
h ( t ) = − 1 − log ( − t ) , h(t)=-1-\log(-t), h ( t ) = − 1 − log ( − t ) ,
and
d h ( τ ( x ) , τ ( z ) ) = z x − log ( z x ) − 1 , d_h(\tau(x),\tau(z))=\frac{z}{x}-\log\left(\frac{z}{x}\right)-1, d h ( τ ( x ) , τ ( z )) = x z − log ( x z ) − 1 ,
which is the reverse Itakura-Saito divergence.
α \alpha α -scaled dual Bregman representation – For the alternative parametrization,
τ ( x ) = log x − 1 , h ( t ) = e − t − 1 , \tau(x)=\log x-1,\qquad h(t)=e^{-t-1}, τ ( x ) = log x − 1 , h ( t ) = e − t − 1 ,
which gives
d h ( τ ( x ) , τ ( z ) ) = 1 x − 1 z − 1 z log ( z x ) . d_h(\tau(x),\tau(z))=\frac{1}{x}-\frac{1}{z}-\frac{1}{z}\log\left(\frac{z}{x}\right). d h ( τ ( x ) , τ ( z )) = x 1 − z 1 − z 1 log ( x z ) .
It is worth noting that the convex generator f ( x ) = − log x f(x)=-\log x f ( x ) = − log x also arises for other exponential-family models, including the centered Laplace and Weibull (with known shape parameter k k k ) distributions, when expressed in the expectation coordinate associated with their sufficient statistic (y = ∣ ω ∣ y=|\omega| y = ∣ ω ∣ and y = ω k y=\omega^k y = ω k respectively). These models therefore inherit the same dual Bregman geometry in the corresponding expectation parametrization.
Gamma distribution with known shape parameter
β \beta β -scaled dual Bregman representation – We now consider the Gamma distribution with known shape parameter k > 0 k>0 k > 0 . The expectation parameter belongs to R + ∗ \mathbb{R}_{+}^{*} R + ∗ and the associated Bregman generator is
f ( x ) = − k log x , x > 0. f(x)=-k\log x,\qquad x>0. f ( x ) = − k log x , x > 0.
Consequently, the dual coordinates are given by
τ ( x ) = f ′ ( x ) = − k x , \tau(x)=f'(x)=-\frac{k}{x}, τ ( x ) = f ′ ( x ) = − x k ,
while the Legendre conjugate is
f ∗ ( t ) = − k log ( − t ) + k log k − k , t < 0. f^{*}(t)=-k\log(-t)+k\log k-k,\qquad t<0. f ∗ ( t ) = − k log ( − t ) + k log k − k , t < 0.
By Theorem 7, h = f ∗ h=f^{*} h = f ∗ . But since additive constants do not affect Bregman divergences, we may equivalently take
h ( t ) = − k log ( − t ) , h(t)=-k\log(-t), h ( t ) = − k log ( − t ) ,
and the corresponding divergence
d h ( τ ( x ) , τ ( z ) ) = k ( z x − log ( z x ) − 1 ) . d_h(\tau(x),\tau(z))=k\left(\frac{z}{x}-\log\left(\frac{z}{x}\right)-1\right). d h ( τ ( x ) , τ ( z )) = k ( x z − log ( x z ) − 1 ) .
α \alpha α -scaled dual Bregman representation – For the alternative parametrization, τ ( x ) = k log x − k \tau(x)=k\log x-k τ ( x ) = k log x − k , h ( t ) = k exp ( − t + k k ) h(t)=k\exp\left(-\frac{t+k}{k}\right) h ( t ) = k exp ( − k t + k ) ,
d h ( τ ( x ) , τ ( z ) ) = k ( 1 x − 1 z + 1 z log x z ) . (115) d_h(\tau(x),\tau(z))=k\left(\frac{1}{x}-\frac{1}{z}+\frac{1}{z}\log\frac{x}{z}\right). \tag*{(115)} d h ( τ ( x ) , τ ( z )) = k ( x 1 − z 1 + z 1 log z x ) . (115)
Hence, the canonical dual geometry of the Gamma family is described by a scaled reverse Itakura-Saito divergence. This example illustrates that exponential-family models whose convex generators differ only by a positive scaling factor share the same underlying Bregman geometry, up to the corresponding rescaling of the divergence.
Bernoulli distribution
β \beta β -scaled dual Bregman representation – Consider now the Bernoulli model. The expectation parameter is the success probability x ∈ ( 0 , 1 ) x \in(0,1) x ∈ ( 0 , 1 ) and the corresponding convex generator is
f ( x ) = x log x + ( 1 − x ) log ( 1 − x ) . f(x)=x\log x+(1-x)\log(1-x). f ( x ) = x log x + ( 1 − x ) log ( 1 − x ) .
The associated mapping is the logit transformation
τ ( x ) = log x − log ( 1 − x ) = log x 1 − x , \tau(x)=\log x-\log(1-x)=\log\frac{x}{1-x}, τ ( x ) = log x − log ( 1 − x ) = log 1 − x x ,
with Legendre conjugate
f ∗ ( t ) = log ( 1 + e t ) , f^*(t)=\log(1+e^t), f ∗ ( t ) = log ( 1 + e t ) ,
Application of Theorem 7 gives
d f ( z , x ) = d h ( τ ( x ) , τ ( z ) ) = − z log x − ( 1 − z ) log ( 1 − x ) + z log z + ( 1 − z ) log ( 1 − z ) , d_f(z,x)=d_h(\tau(x),\tau(z))
=-z\log x-(1-z)\log(1-x)+z\log z+(1-z)\log(1-z), d f ( z , x ) = d h ( τ ( x ) , τ ( z )) = − z log x − ( 1 − z ) log ( 1 − x ) + z log z + ( 1 − z ) log ( 1 − z ) ,
which is the binary cross-entropy, up to an additive constant independent of x x x .
α \alpha α -scaled dual Bregman representation – For the alternative α \alpha α -scaled parametrization, one has
τ ( x ) = − log ( 1 − x ) \tau(x)=-\log(1-x) τ ( x ) = − log ( 1 − x )
and
h ( t ) = − log ( e t − 1 ) h(t)=-\log(e^t-1) h ( t ) = − log ( e t − 1 )
yielding
1 z d f ( z , x ) = d h ( τ ( x ) , τ ( z ) ) = − log x − 1 − z z log ( 1 − x ) + log z + 1 − z z log ( 1 − z ) . \frac{1}{z}d_f(z,x)=d_h(\tau(x),\tau(z))
=-\log x-\frac{1-z}{z}\log(1-x)+\log z+\frac{1-z}{z}\log(1-z). z 1 d f ( z , x ) = d h ( τ ( x ) , τ ( z )) = − log x − z 1 − z log ( 1 − x ) + log z + z 1 − z log ( 1 − z ) .
This example illustrates how the proposed construction recovers the geometry underlying the classical Beta-Bernoulli conjugate model adopted in Bayesian inference within the same general framework.
Binomial distribution with know trial parameter
β \beta β -scaled dual Bregman representation – Considering the Binomial distribution with a fixed number of trials k ∈ N ∗ k \in\mathbb{N}^{*} k ∈ N ∗ , the expectation parameter is the mean number of successes, x ∈ ( 0 , k ) x \in(0,k) x ∈ ( 0 , k ) , and the associated convex generator is
f ( x ) = x log x + ( k − x ) log ( k − x ) . f(x) = x \log x + (k-x)\log(k-x). f ( x ) = x log x + ( k − x ) log ( k − x ) .
Its gradient is
τ ( x ) = log x − log ( k − x ) , \tau(x) = \log x-\log(k-x), τ ( x ) = log x − log ( k − x ) ,
The Legendre conjugate is
f ∗ ( t ) = k log ( 1 + e t ) − k log k . f^{*}(t) = k\log(1+e^{t})-k\log k. f ∗ ( t ) = k log ( 1 + e t ) − k log k .
The conjugate generator is therefore
h ( t ) = k log ( 1 + e t ) , h(t) = k\log(1+e^{t}), h ( t ) = k log ( 1 + e t ) ,
which immediately yields
d h ( τ ( x ) , τ ( z ) ) = − z log x − ( k − z ) log ( k − x ) + z log z + ( k − z ) log ( k − z ) . d_h(\tau(x),\tau(z)) = -z\log x-(k-z)\log(k-x)+z\log z+(k-z)\log(k-z). d h ( τ ( x ) , τ ( z )) = − z log x − ( k − z ) log ( k − x ) + z log z + ( k − z ) log ( k − z ) .
As in the Bernoulli case, the divergence corresponds to the Binomial cross-entropy, up to an additive constant independent of the unknown parameter.
α \alpha α -scaled dual Bregman representation – For the alternative parametrization,
τ ( x ) = − k log ( k − x ) , \tau(x) = -k\log(k-x), τ ( x ) = − k log ( k − x ) ,
and
h ( t ) = − log ( k e t / k − 1 ) , h(t) = -\log\left(ke^{t/k}-1\right), h ( t ) = − log ( k e t / k − 1 ) ,
leading to
d h ( τ ( x ) , τ ( z ) ) = − log x − k − z z log ( k − x ) + log z + k − z z log ( k − z ) . d_h(\tau(x),\tau(z)) = -\log x-\frac{k-z}{z}\log(k-x)+\log z+\frac{k-z}{z}\log(k-z). d h ( τ ( x ) , τ ( z )) = − log x − z k − z log ( k − x ) + log z + z k − z log ( k − z ) .
This example shows that the same geometric construction extends naturally from the Bernoulli to the Binomial family, with the number of trials k k k entering explicitly into the corresponding generators and coordinate mappings.
Negative Binomial distribution with known number of failure
β \beta β -scaled dual Bregman representation – Consider now the Negative Binomial distribution with a known number of failures r r r . The corresponding generator is
f ( x ) = x log x − ( r + x ) log ( r + x ) , f(x) = x\log x-(r+x)\log(r+x), f ( x ) = x log x − ( r + x ) log ( r + x ) ,
defined on R + ∗ \mathbb{R}_{+}^{*} R + ∗ . The associated gradient mapping is
τ ( x ) = log x − log ( r + x ) , \tau(x) = \log x-\log(r+x), τ ( x ) = log x − log ( r + x ) ,
while its Legendre conjugate is
f ∗ ( t ) = r log r − r log ( 1 − e t ) , t < 0. f^{*}(t) = r\log r-r\log(1-e^{t}), \quad t<0. f ∗ ( t ) = r log r − r log ( 1 − e t ) , t < 0.
By Theorem 7,
h ( t ) = − r log ( 1 − e t ) h(t)=-r\log(1-e^t) h ( t ) = − r log ( 1 − e t )
and the induced divergence becomes
d h ( τ ( x ) , τ ( z ) ) = − z log x + ( r + z ) log ( r + x ) + z log z − ( r + z ) log ( r + z ) . d_h(\tau(x),\tau(z))=-z\log x+(r+z)\log(r+x)+z\log z-(r+z)\log(r+z). d h ( τ ( x ) , τ ( z )) = − z log x + ( r + z ) log ( r + x ) + z log z − ( r + z ) log ( r + z ) .
α \alpha α -scaled dual Bregman representation – For the alternative parametrization,
τ ( x ) = r log ( r + x ) , \tau(x)=r\log(r+x), τ ( x ) = r log ( r + x ) ,
and
h ( t ) = t r − log ( e t / r − r ) , h(t)=\frac{t}{r}-\log\left(e^{t/r}-r\right), h ( t ) = r t − log ( e t / r − r ) ,
yielding
d h ( τ ( x ) , τ ( z ) ) = − log x + r + z z log ( r + x ) + log z − r + z z log ( r + z ) . d_h(\tau(x),\tau(z))=-\log x+\frac{r+z}{z}\log(r+x)+\log z-\frac{r+z}{z}\log(r+z). d h ( τ ( x ) , τ ( z )) = − log x + z r + z log ( r + x ) + log z − z r + z log ( r + z ) .
This example shows that the same Legendre-dual construction applies to the Negative Binomial family, thereby extending the proposed geometric representation to overdispersed count models.
Inverse-Gaussian distribution with known shape parameter
The Inverse-Gaussian model with known shape λ \lambda λ is a classical non-regular family whose log-partition function is essentially smooth but Θ = ( − ∞ , 0 ] \Theta=(-\infty,0] Θ = ( − ∞ , 0 ] is not open (Barndorff-Nielsen (2014, Ch. 8)). According to Remark 3, all conclusions of Theorem 7 do not apply. Nevertheless, expressions of ( h , τ ) (h,\tau) ( h , τ ) can still be derived for the β \beta β - and α \alpha α -scaled models. Its expectation parameter is the mean x ∈ R + ∗ x\in\mathbb{R}^{*}_{+} x ∈ R + ∗ and the corresponding convex generator is
f ( x ) = λ 2 x , x ∈ R + ∗ . f(x)=\frac{\lambda}{2x},\qquad x\in\mathbb{R}^{*}_{+}. f ( x ) = 2 x λ , x ∈ R + ∗ .
β \beta β -scaled dual Bregman representation – The Legendre conjugate of f f f writes
h ( t ) = f ∗ ( t ) = − − 2 λ t h(t)=f^*(t)=-\sqrt{-2\lambda t} h ( t ) = f ∗ ( t ) = − − 2 λ t
while the dual coordinate mapping is
τ ( x ) = f ′ ( x ) = − λ 2 x 2 . \tau(x)=f'(x)=-\frac{\lambda}{2x^2}. τ ( x ) = f ′ ( x ) = − 2 x 2 λ .
Application of Theorem 7 gives
d f ( z , x ) = d h ( τ ( x ) , τ ( z ) ) = λ ( x − z ) 2 2 x 2 z . \begin{aligned}
d_f(z,x)&=d_h(\tau(x),\tau(z))\\
&=\frac{\lambda(x-z)^2}{2x^2z}.
\end{aligned} d f ( z , x ) = d h ( τ ( x ) , τ ( z )) = 2 x 2 z λ ( x − z ) 2 .
Note that h h h is of Legendre type since f f f is Legendre, but the equality dom h = τ ( X ) \operatorname{dom}h=\tau(\mathcal{X}) dom h = τ ( X ) fails: τ ( X ) = int Θ = ( − ∞ , 0 ) \tau(\mathcal{X})=\operatorname{int}\Theta=(-\infty,0) τ ( X ) = int Θ = ( − ∞ , 0 ) misses the boundary point 0 0 0 of dom h = Θ = ( − ∞ , 0 ] \operatorname{dom}h=\Theta=(-\infty,0] dom h = Θ = ( − ∞ , 0 ] .
α \alpha α -scaled dual Bregman representation – For the alternative α \alpha α -scaled parametrization, one has
τ ( x ) = − λ x . (116) \tau(x)=-\frac{\lambda}{x}. \tag*{(116)} τ ( x ) = − x λ . (116)
So τ ( X ) = ( − ∞ , 0 ) \tau(\mathcal{X}) = (-\infty, 0) τ ( X ) = ( − ∞ , 0 ) , and from Theorem 5
h ( t ) = sup x > 0 ( − t x − λ 2 x 2 ) = { t 2 2 λ , t < 0 , 0 , t ≥ 0. h(t) = \sup_{x>0}\left(-\frac{t}{x}-\frac{\lambda}{2x^2}\right) =
\begin{cases}
\frac{t^2}{2\lambda}, & t < 0,\\
0, & t \ge0.
\end{cases} h ( t ) = x > 0 sup ( − x t − 2 x 2 λ ) = { 2 λ t 2 , 0 , t < 0 , t ≥ 0.
Note that for t < 0 t < 0 t < 0 the supremum is attained at x = − λ / t = τ − 1 ( t ) x = -\lambda/t = \tau^{-1}(t) x = − λ / t = τ − 1 ( t ) . For t ≥ 0 t \ge0 t ≥ 0 , it is approached as x → ∞ x \to\infty x → ∞ , that is, as the natural parameter f ′ ( x ) = − λ / ( 2 x 2 ) f'(x) = -\lambda/(2x^2) f ′ ( x ) = − λ / ( 2 x 2 ) tends to the boundary point 0 0 0 of Θ \Theta Θ , where f ∗ ( 0 ) = 0 f^*(0) = 0 f ∗ ( 0 ) = 0 is finite. This boundary point, absent in the regular case, is why dom h = R \operatorname{dom} h = \mathbb{R} dom h = R is larger than τ ( X ) \tau(\mathcal{X}) τ ( X ) . Moreover, the restriction of h h h to τ ( X ) \tau(\mathcal{X}) τ ( X ) is not essentially smooth (hence not Legendre), since h ′ ( t ) → 0 h'(t) \to0 h ′ ( t ) → 0 as t ↑ 0 t \uparrow0 t ↑ 0 .
Finally, ( h , τ ) (h,\tau) ( h , τ ) expressions yield
1 z d f ( z , x ) = d h ( τ ( x ) , τ ( z ) ) = λ 2 ( 1 x − 1 z ) 2 . \begin{aligned}
\frac{1}{z}d_f(z,x) = d_h(\tau(x),\tau(z)) \\
&= \frac{\lambda}{2}\left(\frac{1}{x}-\frac{1}{z}\right)^2.
\end{aligned} z 1 d f ( z , x ) = d h ( τ ( x ) , τ ( z )) = 2 λ ( x 1 − z 1 ) 2 .
Gaussian distribution
β \beta β -scaled dual Bregman representation – The proposed dual representation of the log-density is not restricted to exponential families defined by a one-dimensional parameter and naturally extends to distributions with vector-parameter. To illustrate this extension, we consider the Gaussian distribution with unknown mean and variance. The minimal sufficient statistic is
y = ( ω , ω 2 ) , \mathbf{y} = (\omega,\omega^2), y = ( ω , ω 2 ) ,
and the corresponding expectation parameter x = ( x 1 , x 2 ) ∈ X \mathbf{x} = (x_1,x_2) \in\mathcal{X} x = ( x 1 , x 2 ) ∈ X is
x = ( E [ ω ] , E [ ω 2 ] ) \mathbf{x} = \left(\mathbb{E}[\omega],\mathbb{E}[\omega^2]\right) x = ( E [ ω ] , E [ ω 2 ] )
where the parameter space is
X = { ( x 1 , x 2 ) ∈ R × R + : x 2 > x 1 2 } . \mathcal{X} = \left\{(x_1,x_2) \in\mathbb{R} \times\mathbb{R}_+ : x_2 > x_1^2\right\}. X = { ( x 1 , x 2 ) ∈ R × R + : x 2 > x 1 2 } .
The associated generator is
f ( x ) = − 1 2 log ( x 2 − x 1 2 ) . f(\mathbf{x}) = -\frac{1}{2}\log(x_2-x_1^2). f ( x ) = − 2 1 log ( x 2 − x 1 2 ) .
whose Legendre conjugate is
f ∗ ( t ) = − t 1 2 4 t 2 − 1 2 log ( − 2 t 2 ) − 1 2 , t 2 < 0. (117) f^*(\mathbf{t}) = -\frac{t_1^2}{4t_2} - \frac{1}{2}\log(-2t_2) - \frac{1}{2}, \qquad t_2 < 0. \tag*{(117)} f ∗ ( t ) = − 4 t 2 t 1 2 − 2 1 log ( − 2 t 2 ) − 2 1 , t 2 < 0. (117)
By Theorem 7, the dual representation of the log-density writes
d f ( z , x ) = d h ( τ ( x ) , τ ( z ) ) = 1 2 [ ( x 1 − z 1 ) 2 x 2 − x 1 2 + z 2 − z 1 2 x 2 − x 1 2 + log x 2 − x 1 2 z 2 − z 1 2 − 1 ] . \begin{aligned}
d_f(\mathbf{z},\mathbf{x}) = d_h(\tau(\mathbf{x}),\tau(\mathbf{z})) \\
&= \frac{1}{2}\left[\frac{(x_1-z_1)^2}{x_2-x_1^2}+\frac{z_2-z_1^2}{x_2-x_1^2}+\log\frac{x_2-x_1^2}{z_2-z_1^2}-1\right].
\end{aligned} d f ( z , x ) = d h ( τ ( x ) , τ ( z )) = 2 1 [ x 2 − x 1 2 ( x 1 − z 1 ) 2 + x 2 − x 1 2 z 2 − z 1 2 + log z 2 − z 1 2 x 2 − x 1 2 − 1 ] .
In the usual parametrization
( μ , σ 2 ) = ( x 1 , x 2 − x 1 2 ) . (\mu,\sigma^2) = (x_1,x_2-x_1^2). ( μ , σ 2 ) = ( x 1 , x 2 − x 1 2 ) .
the resulting Bregman divergence has the characteristic quadratic-over-variance structure underlying the classical Normal-Inverse-Gamma conjugate model adopted in Bayesian inference.
Gamma distribution
β \beta β -scaled dual Bregman representation – Another example of the extension to distributions with vector-valued parameter is the Gamma distribution when both the shape a > 0 a > 0 a > 0 and rate b > 0 b > 0 b > 0 are unknown. A minimal sufficient statistic is
y = ( log ω , ω ) . \mathbf{y} = (\log\omega,\omega). y = ( log ω , ω ) .
The corresponding expectation parameter x = ( x 1 , x 2 ) ∈ X \mathbf{x} = (x_1,x_2) \in\mathcal{X} x = ( x 1 , x 2 ) ∈ X is
x = ( E [ log ω ] , E [ ω ] ) \mathbf{x} = (\mathbb{E}[\log\omega],\mathbb{E}[\omega]) x = ( E [ log ω ] , E [ ω ])
with
x 1 = ψ 0 ( a ) − log b , x 2 = a b , x_1 = \psi_0(a) - \log b,\qquad x_2 = \frac{a}{b}, x 1 = ψ 0 ( a ) − log b , x 2 = b a ,
where ψ 0 \psi_0 ψ 0 denotes the digamma function and the expectation-parameter space is defined as
X = { ( x 1 , x 2 ) ∈ R × R + ∗ : x 1 < log x 2 } . \mathcal{X} = \{(x_1,x_2) \in\mathbb{R} \times\mathbb{R}_{+}^{*} : x_1 < \log x_2\}. X = {( x 1 , x 2 ) ∈ R × R + ∗ : x 1 < log x 2 } .
The natural coordinates t = ( t 1 , t 2 ) \mathbf{t} = (t_1,t_2) t = ( t 1 , t 2 ) are
t 1 = a − 1 > − 1 , t 2 = − b < 0. t_1 = a - 1 > -1,\qquad t_2 = -b < 0. t 1 = a − 1 > − 1 , t 2 = − b < 0.
The Legendre conjugate of the likelihood generator f f f , which coincides with the log-partition function, is
f ∗ ( t ) = log Γ ( t 1 + 1 ) − ( t 1 + 1 ) log ( − t 2 ) , t 1 > − 1 , t 2 < 0 , (118) f^{*}(\mathbf{t}) = \log\Gamma(t_1 + 1) - (t_1 + 1)\log(-t_2),\qquad t_1 > -1,\quad t_2 < 0, \tag*{(118)} f ∗ ( t ) = log Γ ( t 1 + 1 ) − ( t 1 + 1 ) log ( − t 2 ) , t 1 > − 1 , t 2 < 0 , (118)
with gradient
∇ f ∗ ( t ) = ( ψ 0 ( t 1 + 1 ) − log ( − t 2 ) , − t 1 + 1 t 2 ) T . \nabla f^{*}(\mathbf{t}) = \left(\psi_0(t_1 + 1) - \log(-t_2),-\frac{t_1 + 1}{t_2}\right)^{T}. ∇ f ∗ ( t ) = ( ψ 0 ( t 1 + 1 ) − log ( − t 2 ) , − t 2 t 1 + 1 ) T .
The likelihood generator f f f and the dual coordinate mapping τ = ∇ f \tau= \nabla f τ = ∇ f cannot be expressed with elementary closed forms in expectation coordinates, since inversion of the mean map requires solving a (scalar) nonlinear equation involving the digamma function. Indeed, since b = a / x 2 b = a/x_2 b = a / x 2 , the natural coordinates t = ( t 1 , t 2 ) \mathbf{t} = (t_1,t_2) t = ( t 1 , t 2 ) can be rewritten as functions of x ∈ X \mathbf{x} \in\mathcal{X} x ∈ X as
t 1 = a ( x ) − 1 , t 2 = − a ( x ) x 2 t_1 = a(\mathbf{x}) - 1,\qquad t_2 = -\frac{a(\mathbf{x})}{x_2} t 1 = a ( x ) − 1 , t 2 = − x 2 a ( x )
where a ( x ) > 0 a(\mathbf{x}) > 0 a ( x ) > 0 denotes the unique solution of
ψ 0 ( a ) − log a = x 1 − log x 2 . (119) \psi_0(a) - \log a = x_1 - \log x_2. \tag*{(119)} ψ 0 ( a ) − log a = x 1 − log x 2 . (119)
Hence,
τ ( x ) = ( a ( x ) − 1 , − a ( x ) x 2 ) T \tau(\mathbf{x}) = \left(a(\mathbf{x}) - 1,-\frac{a(\mathbf{x})}{x_2}\right)^{T} τ ( x ) = ( a ( x ) − 1 , − x 2 a ( x ) ) T
and, by Legendre duality f ( x ) = ⟨ t , x ⟩ − f ∗ ( t ) f(\mathbf{x}) = \langle\mathbf{t},\mathbf{x}\rangle- f^{*}(\mathbf{t}) f ( x ) = ⟨ t , x ⟩ − f ∗ ( t ) ,
f ( x ) = ( a ( x ) − 1 ) x 1 − a ( x ) − log Γ ( a ( x ) ) + a ( x ) log ( a ( x ) x 2 ) . f(\mathbf{x}) = (a(\mathbf{x}) - 1)x_1 - a(\mathbf{x}) - \log\Gamma(a(\mathbf{x})) + a(\mathbf{x})\log\left(\frac{a(\mathbf{x})}{x_2}\right). f ( x ) = ( a ( x ) − 1 ) x 1 − a ( x ) − log Γ ( a ( x )) + a ( x ) log ( x 2 a ( x ) ) .
Thus, both τ ( x ) \tau(\mathbf{x}) τ ( x ) and f ( x ) f(\mathbf{x}) f ( x ) can be evaluated once the scalar equation (119) defining a ( x ) a(\mathbf{x}) a ( x ) has been solved. Theorem 7 gives
h ( t ) = f ∗ ( t ) = log Γ ( t 1 + 1 ) − ( t 1 + 1 ) log ( − t 2 ) , h(\mathbf{t}) = f^{*}(\mathbf{t}) = \log\Gamma(t_1 + 1) - (t_1 + 1)\log(-t_2), h ( t ) = f ∗ ( t ) = log Γ ( t 1 + 1 ) − ( t 1 + 1 ) log ( − t 2 ) ,
and consequently
d h ( τ ( x ) , τ ( z ) ) = d f ∗ ( τ ( x ) , τ ( z ) ) = d f ( z , x ) . d_h(\tau(x),\tau(z)) = d_{f^*}(\tau(x),\tau(z)) = d_f(z,x). d h ( τ ( x ) , τ ( z )) = d f ∗ ( τ ( x ) , τ ( z )) = d f ( z , x ) .
Although d h d_h d h can be evaluated explicitly once the corresponding natural coordinates are known, its evaluation from expectation coordinates requires the numerical computation of τ ( x ) \tau(x) τ ( x ) and τ ( z ) \tau(z) τ ( z ) .
α \alpha α -scaled dual Bregman representation – The alternative representation of Theorem 7 does not apply directly to this model, since that result is restricted to one-dimensional positive expectation parameters.
Beta distribution
β \beta β -scaled dual Bregman representation – We finally consider the Beta distribution when both shape parameters a a a and b b b are unknown. A minimal sufficient statistic is
y = ( log ω , log ( 1 − ω ) ) , ω ∈ ( 0 , 1 ) . \mathbf{y} = (\log\omega, \log(1-\omega)), \qquad\omega\in(0,1). y = ( log ω , log ( 1 − ω )) , ω ∈ ( 0 , 1 ) .
The corresponding expectation parameter is
x = ( x 1 , x 2 ) = ( E [ log ω ] , E [ log ( 1 − ω ) ] ) . \mathbf{x} = (x_1,x_2) = (\mathbb{E}[\log\omega], \mathbb{E}[\log(1-\omega)]). x = ( x 1 , x 2 ) = ( E [ log ω ] , E [ log ( 1 − ω )]) .
For the Beta distribution,
x 1 = ψ 0 ( a ) − ψ 0 ( a + b ) , x 2 = ψ 0 ( b ) − ψ 0 ( a + b ) , x_1 = \psi_0(a) - \psi_0(a+b), \qquad x_2 = \psi_0(b) - \psi_0(a+b), x 1 = ψ 0 ( a ) − ψ 0 ( a + b ) , x 2 = ψ 0 ( b ) − ψ 0 ( a + b ) ,
where ψ 0 \psi_0 ψ 0 denotes the digamma function. The expectation-parameter space is 2
X = ran ∇ f ∗ ⊂ ( R ∗ ) 2 . \mathcal{X} = \operatorname{ran} \nabla f^* \subset(\mathbb{R}^*)^2. X = ran ∇ f ∗ ⊂ ( R ∗ ) 2 .
A useful necessary condition for membership in X \mathcal{X} X follows from Jensen’s inequality. Indeed,
e x 1 = exp ( E [ log ω ] ) < E [ ω ] , e x 2 = exp ( E [ log ( 1 − ω ) ] ) < E [ 1 − ω ] , e^{x_1} = \exp(\mathbb{E}[\log\omega]) < \mathbb{E}[\omega], \qquad e^{x_2} = \exp(\mathbb{E}[\log(1-\omega)]) < \mathbb{E}[1-\omega], e x 1 = exp ( E [ log ω ]) < E [ ω ] , e x 2 = exp ( E [ log ( 1 − ω )]) < E [ 1 − ω ] ,
and therefore
e x 1 + e x 2 < 1. e^{x_1} + e^{x_2} < 1. e x 1 + e x 2 < 1.
The natural coordinates are
t = ( t 1 , t 2 ) , t 1 = a − 1 > − 1 , t 2 = b − 1 > − 1. \mathbf{t} = (t_1,t_2), \qquad t_1 = a-1 > -1, \qquad t_2 = b-1 > -1. t = ( t 1 , t 2 ) , t 1 = a − 1 > − 1 , t 2 = b − 1 > − 1.
The Legendre conjugate of the likelihood generator f f f , which coincides with the log-partition function, is
f ∗ ( t ) = log Γ ( t 1 + 1 ) + log Γ ( t 2 + 1 ) − log Γ ( t 1 + t 2 + 2 ) , (120) f^*(\mathbf{t}) = \log\Gamma(t_1+1) + \log\Gamma(t_2+1) - \log\Gamma(t_1+t_2+2), \tag*{(120)} f ∗ ( t ) = log Γ ( t 1 + 1 ) + log Γ ( t 2 + 1 ) − log Γ ( t 1 + t 2 + 2 ) , (120)
with
int ( dom f ∗ ) = ( − 1 , + ∞ ) 2 . \operatorname{int}(\operatorname{dom} f^*) = (-1,+\infty)^2. int ( dom f ∗ ) = ( − 1 , + ∞ ) 2 .
Its gradient is
∇ f ∗ ( t ) = ( ψ 0 ( t 1 + 1 ) − ψ 0 ( t 1 + t 2 + 2 ) , ψ 0 ( t 2 + 1 ) − ψ 0 ( t 1 + t 2 + 2 ) ) . \nabla f^*(\mathbf{t}) = \left(\psi_0(t_1+1)-\psi_0(t_1+t_2+2), \psi_0(t_2+1)-\psi_0(t_1+t_2+2)\right). ∇ f ∗ ( t ) = ( ψ 0 ( t 1 + 1 ) − ψ 0 ( t 1 + t 2 + 2 ) , ψ 0 ( t 2 + 1 ) − ψ 0 ( t 1 + t 2 + 2 ) ) .
The likelihood generator f f f and the dual coordinate mapping τ = ∇ f \tau= \nabla f τ = ∇ f do not admit simple closed-form expressions directly in expectation coordinates. More precisely, for x ∈ X x \in\mathcal{X} x ∈ X , let ( a ( x ) , b ( x ) ) (a(x), b(x)) ( a ( x ) , b ( x )) denote the unique solution in ( 0 , + ∞ ) 2 (0, +\infty)^2 ( 0 , + ∞ ) 2 of
{ ψ 0 ( a ) − ψ 0 ( a + b ) = x 1 , ψ 0 ( b ) − ψ 0 ( a + b ) = x 2 . \begin{cases}
\psi_0(a) - \psi_0(a+b) = x_1,\\
\psi_0(b) - \psi_0(a+b) = x_2.
\end{cases} { ψ 0 ( a ) − ψ 0 ( a + b ) = x 1 , ψ 0 ( b ) − ψ 0 ( a + b ) = x 2 .
Existence and uniqueness follow from the Legendre property of f ∗ f^* f ∗ , since ∇ f ∗ \nabla f^* ∇ f ∗ is a bijection from the natural-parameter space onto X \mathcal{X} X . Hence,
τ ( x ) = ∇ f ( x ) = ( ∇ f ∗ ) − 1 ( x ) = ( a ( x ) − 1 , b ( x ) − 1 ) , \tau(\mathbf{x}) = \nabla f(\mathbf{x}) = \left(\nabla f^*\right)^{-1}(\mathbf{x}) = (a(\mathbf{x}) - 1, b(\mathbf{x}) - 1), τ ( x ) = ∇ f ( x ) = ( ∇ f ∗ ) − 1 ( x ) = ( a ( x ) − 1 , b ( x ) − 1 ) ,
which must generally be evaluated numerically.
Similarly, Legendre duality gives
f ( x ) = ⟨ t , x ⟩ − f ∗ ( t ) , f(\mathbf{x}) = \langle\mathbf{t}, \mathbf{x} \rangle- f^*(\mathbf{t}), f ( x ) = ⟨ t , x ⟩ − f ∗ ( t ) ,
and therefore
f ( x ) = ( a − 1 ) x 1 + ( b − 1 ) x 2 − log Γ ( a ) − log Γ ( b ) + log Γ ( a + b ) , f(\mathbf{x}) = (a-1)x_1 + (b-1)x_2 - \log\Gamma(a) - \log\Gamma(b) + \log\Gamma(a+b), f ( x ) = ( a − 1 ) x 1 + ( b − 1 ) x 2 − log Γ ( a ) − log Γ ( b ) + log Γ ( a + b ) ,
where
a = a ( x ) , b = b ( x ) . a = a(\mathbf{x}), \qquad b = b(\mathbf{x}). a = a ( x ) , b = b ( x ) .
Thus, f f f is explicit once ( a ( x ) , b ( x ) ) (a(\mathbf{x}), b(\mathbf{x})) ( a ( x ) , b ( x )) has been determined. For β = 1 \beta= 1 β = 1 , Theorem 7 gives
h ( t ) = f ∗ ( t ) = log Γ ( t 1 + 1 ) + log Γ ( t 2 + 1 ) − log Γ ( t 1 + t 2 + 2 ) , h(\mathbf{t}) = f^*(\mathbf{t}) = \log\Gamma(t_1+1) + \log\Gamma(t_2+1) - \log\Gamma(t_1+t_2+2), h ( t ) = f ∗ ( t ) = log Γ ( t 1 + 1 ) + log Γ ( t 2 + 1 ) − log Γ ( t 1 + t 2 + 2 ) ,
and consequently
d h ( τ ( x ) , τ ( z ) ) = d f ∗ ( τ ( x ) , τ ( z ) ) = d f ( z , x ) . d_h(\tau(\mathbf{x}), \tau(\mathbf{z})) = d_{f^*}(\tau(\mathbf{x}), \tau(\mathbf{z})) = d_f(\mathbf{z}, \mathbf{x}). d h ( τ ( x ) , τ ( z )) = d f ∗ ( τ ( x ) , τ ( z )) = d f ( z , x ) .
Although the divergence is explicit in natural coordinates, its evaluation in expectation coordinates requires the numerical computation of τ ( x ) \tau(\mathbf{x}) τ ( x ) and τ ( z ) \tau(\mathbf{z}) τ ( z ) .
α \alpha α -scaled dual Bregman representation – The alternative representation of Theorem 7 does not apply directly to this model, since that result is restricted to one-dimensional positive expectation parameters.