Introduction
Gibbs probability measures play an important role in statistical learning theory and information theory, in part due to their versatility for modeling different learning scenarios [7 , 8 , 9 , 10 , 11 , 12 , 13 , 14 , 15 , 16 ] . For instance, under suitable conditions, Gibbs measures describe the long-run behavior of stochastic-gradient-based learning algorithms with time-invariant learning rates [17 ] . Moreover, Gibbs algorithms are represented by data-dependent Gibbs probability measures on the set of models. Interestingly, these algorithms provide a benchmark for the analysis of the generalization error of learning algorithms [18 , 19 , 20 , 21 , 22 ] . Gibbs probability measures on the set of datasets also characterize worst-case data-generating probability distributions for a fixed model [23 ] . More generally, Gibbs probability measures naturally arise as solutions to the minimization or maximization of the expectation of an objective function, such as the empirical risk, subject to a relative entropy regularization. In such a case, the relative entropy is defined with respect to a reference measure, and a regularization factor controls the tradeoff between the contribution of the objective function and that of the regularization. Similarly, under a specific choice of regularization factor, the same Gibbs probability measure also solves the corresponding minimization or maximization problem subject to a relative entropy constraint [15 , 24 , 25 , 26 , 27 ] . These variational characterizations relate an objective function, a reference measure and a regularization factor to the resulting Gibbs probability measures. This relation allows jointly studying the effects of the regularization factor and the reference measure on the resulting Gibbs algorithm [10 , 12 , 15 ] . It also motivates the study of operations that transform or combine Gibbs probability measures, with the aim of determining how the objective functions, reference measures, and regularization factors affect the resulting measure and its associated optimization problems. The present work studies three such operations: renormalization, normalized log-linear combinations, and nesting. The analysis makes explicit how these operations act on the objective functions, reference measures, and regularization factors, and establishes the associated optimization problems and connections between the operations. Geometric representations are also developed when the reference measures are probability measures. These results are specialized to supervised machine learning, where connections with weighted empirical risk are established. Particular emphasis is placed on exact machine unlearning and centralized-equivalent performance, in the sense of training on the aggregation of all local datasets in decentralized and federated learning systems.
Yaiza Bermudez is with the Centre Inria d’Université Côte d’Azur, INRIA, 06902 Sophia Antipolis, France (e-mail: yaiza.bermudez@inria.fr).
Samir M. Perlaza is with the College of Computing, University Mohammed VI Polytechnic University (UM6P), Ben Guerir, 43150, Morocco; also with the Centre Inria d’Université Côte d’Azur, INRIA, 06902 Sophia Antipolis, France; also with the Department of Electrical and Computer Engineering at Princeton University, Princeton, NJ 08544 USA; and also with the GAATI Mathematics Laboratory at the University of French Polynesia, 98702 Faaa, French Polynesia (e-mail: samir.perlaza@um6p.ma).
Iñaki Esnaola is with the School of Electrical and Electronic Engineering, University of Sheffield, Sheffield S1 3JD, U.K.; and also with the Department of Electrical and Computer Engineering, Princeton University, Princeton, NJ 08544 USA (e-mail: esnaola@sheffield.ac.uk).
Parts of this work have been presented at IEEE International Symposium on Information Theory (ISIT), Guangzhou, China, Jun., 2026 [1 , 2 ] ; at the IEEE Information Theory Workshop (ITW), Tempe, Arizona, USA, Nov. 2026 [3 ] ; and appear as Inria Technical Reports in [4 , 5 , 6 ] .
Operations on Gibbs Probability Measures
The first operation, referred to as renormalization , consists of normalizing a power of the Radon–Nikodym derivative of a probability measure. This operation is frequently encountered in statistical physics and information theory, and the resulting measure is often referred to as an escort distribution. See for instance [28 , 29 , 30 , 31 ] and references therein. In this work, the focus is on the renormalization of a given Gibbs probability measure, in the particular case in which the measure used for the normalization is allowed to differ from the reference measure of the given measure. In this work, Theorem 4 shows that the resulting measure is a Gibbs probability measure with the same objective function, a modified regularization factor, and a possibly different reference measure. These two changes can be determined independently, and the change of reference measure may restrict the support of the resulting Gibbs measure. Hence, renormalization describes how a Gibbs probability measure can be transformed through changes in its regularization factor and reference measure.
The second operation, referred to as normalized log-linear combination , extends renormalization to several probability measures. More specifically, this operation consists of normalizing a product of powers of Radon–Nikodym derivatives. Related constructions have been studied beyond Gibbs measures in [28 ] , [32 ] , [33 ] . In this work, Theorem 7 provides an explicit expression for the normalized log-linear combination of several Gibbs probability measures. The resulting measure is a Gibbs probability measure whose objective function is a linear combination of the original objective functions, with coefficients determined by the powers used in the operation and the regularization factors. This explicit expression is complemented by Theorem 8, which establishes equivalent formulations for the resulting measure and its role on some specific regularized and constrained optimization problems.
The third operation, referred to as nesting , consists of constructing a Gibbs probability measure whose reference measure is itself a Gibbs probability measure. This construction also appears in sequential updating of Gibbs posteriors in [15 ] and [34 ] . In this work, Theorem 11 shows that sequentially nesting several Gibbs probability measures leads to another Gibbs probability measure whose objective function is a linear combination of the objective functions of the original measures. Theorem 12 establishes the corresponding equivalent variational characterizations of such a resulting measure. The connection with normalized log-linear combinations is then made explicit by Corollary 13, which establishes the equivalence between the two operations under specific conditions.
These operations also admit geometric representations when the reference measures are probability measures. These representations build on the geometry of escort distributions and relative entropy [24 ] , [28 ] . In particular, variations of the parameters of renormalizations and normalized log-linear combinations are shown to induce exponential geodesics. For nested Gibbs probability measures, identities relating relative entropies to differences of expectations admit Pythagorean interpretations. Lemma 15 establishes such an identity for multiple nested Gibbs probability measures.
Supervised Machine Learning
In this work, when the operations described above are applied in the context of supervised machine learning, Gibbs probability measures (algorithms) are defined using empirical risks associated with training datasets as objective functions. Normalized log-linear combinations and nesting then yield Gibbs probability measures associated with weighted empirical risks on the aggregation of the corresponding training datasets. For normalized log-linear combinations, the weights are determined by the coefficients of the combination and the regularization factors of the Gibbs algorithms being combined. Alternatively, for nested Gibbs algorithms, the weights are determined by the regularization factors, and the dependence on previously used datasets is carried by the data-dependent reference measures. Thus, the parameters of the corresponding operation determine the weights assigned to the data points within the empirical risk with respect to local datasets. These parameters can, for instance, be chosen such that the resulting Gibbs probability measure is identical to the measure obtained as the unique solution to an empirical risk minimization with relative entropy regularization (ERM-RER) with respect to the aggregation of the local datasets. Corollary 30 establishes this result in a decentralized learning setting through a sequence of nested Gibbs algorithms. In such a decentralized setting, each client uses the Gibbs probability measure received from the previous client as its reference measure, and the regularization factors are chosen according to the local dataset sizes. Alternatively, in a federated learning setting, such a result is established by Corollary 35. In this case, local Gibbs probability measures are combined at a central server through a normalized log-linear combination. Interestingly, nesting and normalized log-linear combinations yield the same Gibbs probability measure as centralized training, thereby establishing the condition for centralized performance guarantees in decentralized and federated learning systems, respectively. Nonetheless, the connection with weighted empirical risks is not restricted to reproducing centralized training. More generally, these operations provide a way to control the weight assigned to each data point through their parameters and its membership to a specific dataset. In particular, the contribution of selected data points can be canceled through a suitable choice of these parameters, which leads to applications to exact machine unlearning.
In this work, the term machine unlearning refers to the task of removing the influence of part of the training dataset from a learning algorithm while seeking to avoid retraining from scratch [35 , 36 , 37 ] . Early formulations of data deletion in learning compare the updated algorithm with the algorithm trained exclusively upon the retained dataset [38 ] , [39 ] . In exact unlearning , the probability distributions on the models induced by these two algorithms are required to be identical. Other formulations allow approximate removal, subject to guarantees on the discrepancy between these distributions [40 ] , [41 ] . This work studies exact unlearning of datasets from a given Gibbs algorithm in centralized learning, as well as in decentralized and federated learning systems. In the decentralized and federated case, clients collaborate without explicitly exchanging their local datasets [42 ] , [43 ] .
The connection of nesting and normalized log-linear combinations with weighted empirical risks provides two mechanisms for exact machine unlearning. In both cases, the parameters of the operation are chosen such that the contribution of the data points to be unlearned is canceled. Hence, the remaining objective is the empirical risk on the retained dataset, up to a scaling accounted for by the regularization factor.
In this context, the first mechanism uses nesting and consists of maximizing the expected empirical risk on the dataset to be unlearned with relative entropy regularization with respect to the Gibbs algorithm from which data must be unlearnt. Corollary 27 shows that a particular choice of the regularization factors yields the same Gibbs probability measure as training exclusively upon the retained dataset. Only the dataset to be unlearned is explicitly used in this optimization problem, while the dependence on the retained dataset is carried by the reference measure. Together with Corollary 30, this result provides a mechanism for exact unlearning in decentralized learning, where the contribution of a local dataset is canceled from the given Gibbs probability measure.
The second mechanism uses a normalized log-linear combination of the Gibbs probability measure trained upon the full training dataset and a Gibbs probability measure trained upon the dataset to be unlearned. Corollary 34 establishes conditions on the coefficients of the combination and the regularization factors under which the resulting measure achieves exact unlearning. Alternatively, the unlearning application to federated learning is established by Corollary 37.
Organization of the Paper
The remainder of the paper is organized as follows. Section II introduces the notation, Gibbs conditional probability measures, and their variational characterizations. Section III studies renormalization, normalized log-linear combinations, and nesting, together with their associated optimization problems and the connections between these operations. Section IV develops their geometric representations when the reference measures are probability measures. Section V specializes these results to supervised machine learning, and Section VI presents their applications to exact machine unlearning. Finally, Section VII concludes the paper.
Preliminaries
This section describes the notation used in this work and formally introduces Gibbs probability measures and Gibbs conditional probability measures. More importantly, the connection between such measures and a particular class of optimization problems involving relative entropies is thoroughly presented.
Notation
Given a set X ⊆ R d \mathcal{X} \subseteq\mathbb{R}^{d} X ⊆ R d , for some d ∈ N d \in\mathbb{N} d ∈ N , the Borel σ \sigma σ -field defined on X \mathcal{X} X is denoted by B ( X ) \mathcal{B}(\mathcal{X}) B ( X ) . The set of probability measures that can be defined on the measurable space ( X , B ( X ) ) (\mathcal{X},\mathcal{B}(\mathcal{X})) ( X , B ( X )) is denoted by Δ ( X ) \Delta(\mathcal{X}) Δ ( X ) . Given a σ \sigma σ -finite measure Q Q Q on ( X , B ( X ) ) (\mathcal{X},\mathcal{B}(\mathcal{X})) ( X , B ( X )) , the set of probability measures in Δ ( X ) \Delta(\mathcal{X}) Δ ( X ) that are absolutely continuous with respect to Q Q Q is denoted by Δ Q ( X ) \Delta_{Q}(\mathcal{X}) Δ Q ( X ) . Given a measure P ∈ Δ Q ( X ) P \in\Delta_{Q}(\mathcal{X}) P ∈ Δ Q ( X ) , the notation P ≪ Q P \ll Q P ≪ Q stands for “the measure P P P is absolutely continuous with respect to Q Q Q ”. Thus,
Δ Q ( X ) ≜ { P ∈ Δ ( X ) : P ≪ Q } . (1) \Delta_{Q}(\mathcal{X}) \triangleq\{P \in\Delta(\mathcal{X}) : P \ll Q\}. \tag*{(1)} Δ Q ( X ) ≜ { P ∈ Δ ( X ) : P ≪ Q } . (1)
The Radon-Nikodym derivative of P ∈ Δ Q ( X ) P \in\Delta_{Q}(\mathcal{X}) P ∈ Δ Q ( X ) with respect to Q Q Q is a map X → [ 0 , + ∞ ) \mathcal{X} \to[0,+\infty) X → [ 0 , + ∞ ) , denoted by d P d Q \frac{\mathrm{d}P}{\mathrm{d}Q} d Q d P . See for instance [44 Section 2.2] . Using this notation, the relative entropy or Kullback-Leibler (KL) divergence is defined as follows.
Definition 1 (Relative Entropy). Given a probability measure P P P and a σ \sigma σ -finite measure Q Q Q , both on the same measurable space, with P ≪ Q P \ll Q P ≪ Q the relative entropy of P P P with respect to Q Q Q is
D ( P ∥ Q ) ≜ ∫ d P d Q ( x ) log ( d P d Q ( x ) ) d Q ( x ) . (2) D(P \parallel Q) \triangleq\int\frac{\mathrm{d}P}{\mathrm{d}Q}(x)\log\left(\frac{\mathrm{d}P}{\mathrm{d}Q}(x)\right)\mathrm{d}Q(x). \tag*{(2)} D ( P ∥ Q ) ≜ ∫ d Q d P ( x ) log ( d Q d P ( x ) ) d Q ( x ) . (2)
Note that when Q Q Q in Definition 1 is not a probability measure, the relative entropy might be negative. In particular, when Q Q Q is the corresponding Lebesgue measure, the relative entropy D ( P ∥ Q ) D(P \parallel Q) D ( P ∥ Q ) becomes negative Shannon’s differential entropy [45 Section 8.3] .
In this work, conditional probability measures are distinguished from probability measures in the following sense.
Definition 2 (Conditional Probability). A family P Y ∣ X ≜ ( P Y ∣ X = x ) x ∈ X P_{Y\mid X} \triangleq(P_{Y\mid X=x})_{x\in\mathcal{X}} P Y ∣ X ≜ ( P Y ∣ X = x ) x ∈ X of elements of Δ ( Y ) \Delta(\mathcal{Y}) Δ ( Y ) indexed by X \mathcal{X} X is said to be a conditional probability measure if, for all measurable sets B ∈ B ( Y ) \mathcal{B} \in\mathcal{B}(\mathcal{Y}) B ∈ B ( Y ) , the map
X ⟶ [ 0 , 1 ] x ⟼ P Y ∣ X = x ( B ) (3) \begin{aligned}
\mathcal{X} &\longrightarrow[0,1]\\
x &\longmapsto P_{Y\mid X=x}(\mathcal{B}) \tag*{(3)}
\end{aligned} X x ⟶ [ 0 , 1 ] ⟼ P Y ∣ X = x ( B ) (3) is Borel measurable. The set of all such conditional probability measures is denoted by Δ ( Y ∣ X ) \Delta(\mathcal{Y}\mid\mathcal{X}) Δ ( Y ∣ X ) .
In a nutshell, while P Y ∣ X P_{Y\mid X} P Y ∣ X is a conditional probability measure, the measure P Y ∣ X = x P_{Y\mid X=x} P Y ∣ X = x , obtained by conditioning on a given x ∈ X x \in\mathcal{X} x ∈ X , is a probability measure.
Gibbs Conditional Probability Measures
Consider two sets X ⊂ R d 1 \mathcal{X} \subset\mathbb{R}^{d_1} X ⊂ R d 1 and Y ⊂ R d 2 \mathcal{Y} \subset\mathbb{R}^{d_2} Y ⊂ R d 2 , for some given integers d 1 d_1 d 1 and d 2 d_2 d 2 . Gibbs conditional probability measures in △ ( Y ∣ X ) \triangle(\mathcal{Y}\mid\mathcal{X}) △ ( Y ∣ X ) are parametrized by a Borel measurable function
h : X × Y → R ; (4) h:\mathcal{X}\times\mathcal{Y}\to\mathbb{R}; \tag*{(4)} h : X × Y → R ; (4)
a σ \sigma σ -finite measure Q Q Q on the measurable space ( Y , B ( Y ) ) (\mathcal{Y},\mathcal{B}(\mathcal{Y})) ( Y , B ( Y )) ; and a real λ ∈ R ∖ { 0 } \lambda\in\mathbb{R}\setminus\{0\} λ ∈ R ∖ { 0 } . Using these parameters, Gibbs conditional probability measures are defined as follows.
Definition 3 (Gibbs Conditional Probability Measures). Given a Borel measurable function h : X × Y → R h:\mathcal{X}\times\mathcal{Y}\to\mathbb{R} h : X × Y → R ; a σ \sigma σ -finite measure Q Q Q on the measurable space ( Y , B ( Y ) ) (\mathcal{Y},\mathcal{B}(\mathcal{Y})) ( Y , B ( Y )) ; and a real λ ∈ R ∖ { 0 } \lambda\in\mathbb{R}\setminus\{0\} λ ∈ R ∖ { 0 } , the conditional probability measure P Y ∣ X ( h , Q , λ ) ∈ △ ( Y ∣ X ) P^{(h,Q,\lambda)}_{Y\mid X}\in\triangle(\mathcal{Y}\mid\mathcal{X}) P Y ∣ X ( h , Q , λ ) ∈ △ ( Y ∣ X ) is said to be an ( h , Q , λ ) (h,Q,\lambda) ( h , Q , λ ) -Gibbs conditional probability measure if
∀ x ∈ X , 0 < ∫ exp ( − 1 λ h ( x , y ) ) d Q ( y ) < + ∞ ; (5) \forall x\in\mathcal{X},\ 0<\int\exp\left(-\frac{1}{\lambda}h(x,y)\right)dQ(y)<+\infty; \tag*{(5)} ∀ x ∈ X , 0 < ∫ exp ( − λ 1 h ( x , y ) ) d Q ( y ) < + ∞ ; (5) and for all ( x , y ) ∈ X × supp Q (x,y)\in\mathcal{X}\times\operatorname{supp}Q ( x , y ) ∈ X × supp Q ,
d P Y ∣ X = x ( h , Q , λ ) d Q ( y ) = exp ( − 1 λ h ( x , y ) ) ∫ exp ( − 1 λ h ( x , y ~ ) ) d Q ( y ~ ) . (6) \frac{dP^{(h,Q,\lambda)}_{Y\mid X=x}}{dQ}(y)=\frac{\exp\left(-\frac{1}{\lambda}h(x,y)\right)}{\int\exp\left(-\frac{1}{\lambda}h(x,\widetilde{y})\right)dQ(\widetilde{y})}. \tag*{(6)} d Q d P Y ∣ X = x ( h , Q , λ ) ( y ) = ∫ exp ( − λ 1 h ( x , y ) ) d Q ( y ) exp ( − λ 1 h ( x , y ) ) . (6)
While P Y ∣ X ( h , Q , λ ) P^{(h,Q,\lambda)}_{Y\mid X} P Y ∣ X ( h , Q , λ ) is an ( h , Q , λ ) (h,Q,\lambda) ( h , Q , λ ) -Gibbs conditional probability measure, the measure P Y ∣ X = x ( h , Q , λ ) P^{(h,Q,\lambda)}_{Y\mid X=x} P Y ∣ X = x ( h , Q , λ ) , obtained by conditioning on a given x ∈ X x\in\mathcal{X} x ∈ X , is an ( h , Q , λ ) (h,Q,\lambda) ( h , Q , λ ) -Gibbs probability measure.
The relevance of ( h , Q , λ ) (h,Q,\lambda) ( h , Q , λ ) -Gibbs probability measures stems from the fact that under some conditions, they are the unique solutions to particular optimization problems involving relative entropy. The following lemma highlights such a fact.
Lemma 1. When λ > 0 \lambda>0 λ > 0 , the ( h , Q , λ ) (h,Q,\lambda) ( h , Q , λ ) -Gibbs probability measure P Y ∣ X = x ( h , Q , λ ) P^{(h,Q,\lambda)}_{Y\mid X=x} P Y ∣ X = x ( h , Q , λ ) in (6) is the unique solution, if it exists, to
min P ∈ △ Q ( Y ) ∫ h ( x , y ) d P ( y ) + λ D ( P ∥ Q ) . (7) \min_{P\in\triangle_Q(\mathcal{Y})}\int h(x,y)dP(y)+\lambda D(P\parallel Q). \tag*{(7)} P ∈ △ Q ( Y ) min ∫ h ( x , y ) d P ( y ) + λ D ( P ∥ Q ) . (7) Alternatively, when λ < 0 \lambda<0 λ < 0 , such a measure P Y ∣ X = x ( h , Q , λ ) P^{(h,Q,\lambda)}_{Y\mid X=x} P Y ∣ X = x ( h , Q , λ ) is the unique solution, if it exists, to
max P ∈ △ Q ( Y ) ∫ h ( x , y ) d P ( y ) + λ D ( P ∥ Q ) . (8) \max_{P\in\triangle_Q(\mathcal{Y})}\int h(x,y)dP(y)+\lambda D(P\parallel Q). \tag*{(8)} P ∈ △ Q ( Y ) max ∫ h ( x , y ) d P ( y ) + λ D ( P ∥ Q ) . (8)
Proof: The proof is immediate from [27 Lemma 1] .
Two optimization problems that are also closely related to the probability measure P Y ∣ X = x ( h , Q , λ ) P^{(h,Q,\lambda)}_{Y\mid X=x} P Y ∣ X = x ( h , Q , λ ) in (6) are the following:
min P ∈ △ Q ( Y ) ∫ h ( x , y ) d P ( y ) s.t. D ( P ∥ Q ) ≤ γ ; (9) \begin{aligned}
\min_{P\in\triangle_Q(\mathcal{Y})}\quad&\int h(x,y)dP(y)\\
\text{s.t.}\quad&D(P\parallel Q)\leq\gamma; \tag*{(9)}
\end{aligned} P ∈ △ Q ( Y ) min s.t. ∫ h ( x , y ) d P ( y ) D ( P ∥ Q ) ≤ γ ; (9)
and
max P ∈ △ Q ( Y ) ∫ h ( x , y ) d P ( y ) s.t. D ( P ∥ Q ) ≤ γ , (10) \begin{aligned}
\max_{P\in\triangle_Q(\mathcal{Y})}\quad&\int h(x,y)dP(y)\\
\text{s.t.}\quad&D(P\parallel Q)\leq\gamma, \tag*{(10)}
\end{aligned} P ∈ △ Q ( Y ) max s.t. ∫ h ( x , y ) d P ( y ) D ( P ∥ Q ) ≤ γ , (10)
for some γ ∈ R \gamma\in\mathbb{R} γ ∈ R . The following lemma formalizes these relations.
Lemma 2 ([27 Lemma 4] ). Assume that the optimization problem in (9) (respectively, in (10)) admits a solution. Consider the ( h , Q , λ ) (h,Q,\lambda) ( h , Q , λ ) -Gibbs probability measure P Y ∣ X = x ( h , Q , λ ) P^{(h,Q,\lambda)}_{Y\mid X=x} P Y ∣ X = x ( h , Q , λ ) in (6) and assume that λ > 0 \lambda>0 λ > 0 (respectively, λ < 0 \lambda<0 λ < 0 ) is chosen such that
D ( P Y ∣ X = x ( h , Q , λ ) ∥ Q ) = γ . (11) D\left(P^{(h,Q,\lambda)}_{Y\mid X=x}\parallel Q\right)=\gamma. \tag*{(11)} D ( P Y ∣ X = x ( h , Q , λ ) ∥ Q ) = γ . (11) Then, the ( h , Q , λ ) (h,Q,\lambda) ( h , Q , λ ) -Gibbs probability measure P Y ∣ X = x ( h , Q , λ ) P^{(h,Q,\lambda)}_{Y\mid X=x} P Y ∣ X = x ( h , Q , λ ) is the unique solution to (9) (respectively, to (10)).
A property of the ( h , Q , λ ) (h,Q,\lambda) ( h , Q , λ ) -Gibbs probability measure P Y ∣ X = x ( h , Q , λ ) P^{(h,Q,\lambda)}_{Y\mid X=x} P Y ∣ X = x ( h , Q , λ ) in (6) that plays an important role in this work concerns the variation of the relative entropy D ( P Y ∣ X = x ( h , Q , λ ) ∥ Q ) D\left(P^{(h,Q,\lambda)}_{Y\mid X=x}\parallel Q\right) D ( P Y ∣ X = x ( h , Q , λ ) ∥ Q ) due to changes in the parameter λ \lambda λ . The following lemma characterizes such a variation.
Lemma 3. Consider the ( h , Q , λ ) (h,Q,\lambda) ( h , Q , λ ) -Gibbs conditional probability measure, P Y ∣ X ( h , Q , λ ) ∈ Δ ( Y ∣ X ) P^{(h,Q,\lambda)}_{Y\mid X} \in\Delta(\mathcal{Y}\mid\mathcal{X}) P Y ∣ X ( h , Q , λ ) ∈ Δ ( Y ∣ X ) , in Definition 3. Then, given a fixed x ∈ X x \in\mathcal{X} x ∈ X , the map
λ ↦ D ( P Y ∣ X = x ( h , Q , λ ) ∥ Q ) (12) \lambda\mapsto D\left(P^{(h,Q,\lambda)}_{Y\mid X=x}\middle\|Q\right) \tag*{(12)} λ ↦ D ( P Y ∣ X = x ( h , Q , λ ) Q ) (12) is nonincreasing on all intervals within ( 0 , + ∞ ) (0,+\infty) ( 0 , + ∞ ) , and nondecreasing on all intervals within ( − ∞ , 0 ) (-\infty,0) ( − ∞ , 0 ) , for which D ( P Y ∣ X = x ( h , Q , λ ) ∥ Q ) < + ∞ D\left(P^{(h,Q,\lambda)}_{Y\mid X=x}\middle\|Q\right)<+\infty D ( P Y ∣ X = x ( h , Q , λ ) Q ) < + ∞ . Moreover, if the function y ↦ h ( x , y ) y\mapsto h(x,y) y ↦ h ( x , y ) , with x x x fixed, is not constant almost surely with respect to Q Q Q , then the monotonicity is strict.
Proof: The proof is presented in Appendix A.
Main Results
In this section three operations on Gibbs probability measures are thoroughly studied: renormalization, normalized log-linear combinations and nesting.
Renormalized Gibbs Probability Measures
The normalization of a power of a density of a probability measure, which is often referred to as renormalization , is classical in statistical physics and information theory, and is not limited to Gibbs measures. See for instance [28 , 29 , 30 , 31 ] , [33 ] , [46 , 47 , 48 , 49 , 50 ] and references therein.
Definition 4 (Renormalization). Consider an α ∈ R ∖ { 0 } \alpha\in\mathbb{R}\setminus\{0\} α ∈ R ∖ { 0 } , two σ \sigma σ -finite measures Q 1 Q_1 Q 1 and Q 2 Q_2 Q 2 , and a probability measure P P P , all on the same measurable space with P ≪ Q 1 P\ll Q_1 P ≪ Q 1 and Q 2 ≪ Q 1 Q_2\ll Q_1 Q 2 ≪ Q 1 . The ( α , Q 1 , Q 2 ) (\alpha,Q_1,Q_2) ( α , Q 1 , Q 2 ) -renormalization of P P P is a probability measure, denoted by R R R , such that for all y ∈ supp Q 2 y\in\operatorname{supp}Q_2 y ∈ supp Q 2 ,
d R d Q 2 ( y ) = ( d P d Q 1 ( y ) ) α ∫ ( d P d Q 1 ( y ~ ) ) α d Q 2 ( y ~ ) , (13) \frac{\mathrm{d}R}{\mathrm{d}Q_2}(y)=\frac{\left(\frac{\mathrm{d}P}{\mathrm{d}Q_1}(y)\right)^\alpha}{\int\left(\frac{\mathrm{d}P}{\mathrm{d}Q_1}(\widetilde{y})\right)^\alpha\mathrm{d}Q_2(\widetilde{y})}, \tag*{(13)} d Q 2 d R ( y ) = ∫ ( d Q 1 d P ( y ) ) α d Q 2 ( y ) ( d Q 1 d P ( y ) ) α , (13) provided that 0 < ∫ ( d P d Q 1 ( y ) ) α d Q 2 ( y ) < + ∞ 0<\int\left(\frac{\mathrm{d}P}{\mathrm{d}Q_1}(y)\right)^\alpha\mathrm{d}Q_2(y)<+\infty 0 < ∫ ( d Q 1 d P ( y ) ) α d Q 2 ( y ) < + ∞ .
The focus of this subsection is on the renormalization of Gibbs probability measures. More specifically, given a σ \sigma σ -finite measure Q 1 Q_1 Q 1 and the ( h , Q , λ ) (h,Q,\lambda) ( h , Q , λ ) -Gibbs probability measure P Y ∣ X = x ( h , Q , λ ) P^{(h,Q,\lambda)}_{Y\mid X=x} P Y ∣ X = x ( h , Q , λ ) in (6), the ( α , Q , Q 1 ) (\alpha,Q,Q_1) ( α , Q , Q 1 ) -renormalization of P Y ∣ X = x ( h , Q , λ ) P^{(h,Q,\lambda)}_{Y\mid X=x} P Y ∣ X = x ( h , Q , λ ) , denoted by R 1 R_1 R 1 , satisfies for all y ∈ supp Q 1 y\in\operatorname{supp}Q_1 y ∈ supp Q 1 ,
d R 1 d Q 1 ( y ) = ( d P Y ∣ X = x ( h , Q , λ ) d Q ( y ) ) α ∫ ( d P Y ∣ X = x ( h , Q , λ ) d Q ( y ~ ) ) α d Q 1 ( y ~ ) . (16) \frac{\mathrm{d}R_1}{\mathrm{d}Q_1}(y)=\frac{\left(\frac{\mathrm{d}P^{(h,Q,\lambda)}_{Y\mid X=x}}{\mathrm{d}Q}(y)\right)^\alpha}{\int\left(\frac{\mathrm{d}P^{(h,Q,\lambda)}_{Y\mid X=x}}{\mathrm{d}Q}(\widetilde{y})\right)^\alpha\mathrm{d}Q_1(\widetilde{y})}. \tag*{(16)} d Q 1 d R 1 ( y ) = ∫ ( d Q d P Y ∣ X = x ( h , Q , λ ) ( y ) ) α d Q 1 ( y ) ( d Q d P Y ∣ X = x ( h , Q , λ ) ( y ) ) α . (16)
The probability measure R 1 R_1 R 1 in (16) is referred to as the “α \alpha α -escort measure of P Y ∣ X = x ( h , Q , λ ) P^{(h,Q,\lambda)}_{Y\mid X=x} P Y ∣ X = x ( h , Q , λ ) relative to Q 1 Q_1 Q 1 ”. The reason why the measure Q Q Q is not taken into account within such an appellation, is made clear by the following theorem, which shows that the explicit expression of the Radon-Nikodym derivative of R 1 R_1 R 1 with respect to Q 1 Q_1 Q 1 does not depend on Q Q Q .
Theorem 4. The ( α , Q , Q 1 ) (\alpha,Q,Q_1) ( α , Q , Q 1 ) -renormalization of the Gibbs probability measure P Y ∣ X = x ( h , Q , λ ) P^{(h,Q,\lambda)}_{Y\mid X=x} P Y ∣ X = x ( h , Q , λ ) in (6), namely the measure R 1 R_1 R 1 in (16), satisfies for all y ∈ supp Q 1 y\in\operatorname{supp}Q_1 y ∈ supp Q 1 ,
d R 1 d Q 1 ( y ) = exp ( − α λ h ( x , y ) ) ∫ exp ( − α λ h ( x , y ~ ) ) d Q 1 ( y ~ ) = d P Y ∣ X = x ( h , Q 1 , λ / α ) d Q 1 ( y ) . (17) \frac{\mathrm{d}R_1}{\mathrm{d}Q_1}(y)=\frac{\exp\left(-\frac{\alpha}{\lambda}h(x,y)\right)}{\int\exp\left(-\frac{\alpha}{\lambda}h(x,\widetilde{y})\right)\mathrm{d}Q_1(\widetilde{y})}=\frac{\mathrm{d}P^{(h,Q_1,\lambda/\alpha)}_{Y\mid X=x}}{\mathrm{d}Q_1}(y). \tag*{(17)} d Q 1 d R 1 ( y ) = ∫ exp ( − λ α h ( x , y ) ) d Q 1 ( y ) exp ( − λ α h ( x , y ) ) = d Q 1 d P Y ∣ X = x ( h , Q 1 , λ / α ) ( y ) . (17)
Proof. The proof is presented in Appendix B.
Theorem 4 shows that the ( α , Q , Q 1 ) (\alpha,Q,Q_1) ( α , Q , Q 1 ) -renormalization of the ( h , Q , λ ) (h,Q,\lambda) ( h , Q , λ ) -Gibbs probability measure P Y ∣ X = x ( h , Q , λ ) P^{(h,Q,\lambda)}_{Y\mid X=x} P Y ∣ X = x ( h , Q , λ ) in (6) is identical to the ( h , Q 1 , λ α ) (h,Q_1,\frac{\lambda}{\alpha}) ( h , Q 1 , α λ ) -Gibbs probability measure P Y ∣ X = x ( h , Q 1 , λ α ) P^{(h,Q_1,\frac{\lambda}{\alpha})}_{Y\mid X=x} P Y ∣ X = x ( h , Q 1 , α λ ) in (17). More specifically, D ( R 1 ∥ P Y ∣ X = x ( h , Q 1 , λ α ) ) = 0 D\left(R_1\middle\|P^{(h,Q_1,\frac{\lambda}{\alpha})}_{Y\mid X=x}\right)=0 D ( R 1 P Y ∣ X = x ( h , Q 1 , α λ ) ) = 0 . In a nutshell, the α \alpha α -escort measure of a Gibbs measure relative to a given reference measure is also a Gibbs probability measure.
Renormalization has a twofold effect on the Gibbs measure P Y ∣ X = x ( h , Q , λ ) P^{(h,Q,\lambda)}_{Y\mid X=x} P Y ∣ X = x ( h , Q , λ ) in (6): (i) the regularization factor is changed from λ \lambda λ to λ α \frac{\lambda}{\alpha} α λ ; and (ii) the reference measure is changed from Q Q Q to Q 1 Q_1 Q 1 . The second effect might imply a concentration of the measure given that Q 1 ≪ Q Q_1\ll Q Q 1 ≪ Q . More specifically, if supp Q 1 ⊂ supp Q \operatorname{supp}Q_1\subset\operatorname{supp}Q supp Q 1 ⊂ supp Q , then, supp P Y ∣ X = x ( h , Q 1 , λ α ) = supp Q 1 ⊂ supp Q = supp P Y ∣ X = x ( h , Q , λ ) \operatorname{supp}P^{(h,Q_1,\frac{\lambda}{\alpha})}_{Y\mid X=x}=\operatorname{supp}Q_1\subset\operatorname{supp}Q=\operatorname{supp}P^{(h,Q,\lambda)}_{Y\mid X=x} supp P Y ∣ X = x ( h , Q 1 , α λ ) = supp Q 1 ⊂ supp Q = supp P Y ∣ X = x ( h , Q , λ ) . Interestingly, both effects are not necessarily simultaneously observed, which follows from the fact that α \alpha α and Q 1 Q_1 Q 1 can be chosen independently. For instance, if Q 1 Q_1 Q 1 is chosen to be identical to Q Q Q , then only the change of the regularization factor is observed. Alternatively, if α = 1 \alpha=1 α = 1 , only the change of reference measure is observed. These observations reveal a structural property of Gibbs probability measures, which can be leveraged to characterize the variation of the solution to the optimization problems in (7) , (8) , (9), and (10) due to changes in the regularization factor λ \lambda λ and the reference measure Q Q Q . The following corollary, which follows from Lemma 1 and Lemma 2, describes this characterization.
Corollary 5. When λ α > 0 \frac{\lambda}{\alpha}>0 α λ > 0 , the ( α , Q , Q 1 ) (\alpha,Q,Q_1) ( α , Q , Q 1 ) -renormalization of the Gibbs probability measure P Y ∣ X = x ( h , Q , λ ) P^{(h,Q,\lambda)}_{Y\mid X=x} P Y ∣ X = x ( h , Q , λ ) in (6), namely the measure R 1 R_1 R 1 in (16), is the unique solution, if it exists, to the following optimization problems:
min P ∈ △ Q 1 ( Y ) ∫ h ( x , y ) d P ( y ) + λ α D ( P ∥ Q 1 ) ; (18) \min_{P\in\triangle_{Q_1}(\mathcal{Y})}\int h(x,y)\,dP(y)+\frac{\lambda}{\alpha}D(P\mathbin\Vert Q_1); \tag*{(18)} P ∈ △ Q 1 ( Y ) min ∫ h ( x , y ) d P ( y ) + α λ D ( P ∥ Q 1 ) ; (18) min P ∈ △ Q 1 ( Y ) ∫ h ( x , y ) d P ( y ) s.t. D ( P ∥ Q 1 ) ≤ D ( P Y ∣ X = x ( h , Q 1 , λ α ) ∥ Q 1 ) . (19) \begin{aligned}
\min_{P\in\triangle_{Q_1}(\mathcal{Y})}\quad&\int h(x,y)\,dP(y)\\
\text{s.t.}\quad&D(P\mathbin\Vert Q_1)\leq D\left(P^{(h,Q_1,\frac{\lambda}{\alpha})}_{Y\mid X=x}\mathbin\Vert Q_1\right). \tag*{(19)}
\end{aligned} P ∈ △ Q 1 ( Y ) min s.t. ∫ h ( x , y ) d P ( y ) D ( P ∥ Q 1 ) ≤ D ( P Y ∣ X = x ( h , Q 1 , α λ ) ∥ Q 1 ) . (19) Alternatively, when λ α < 0 \frac{\lambda}{\alpha}<0 α λ < 0 , the measure R 1 R_1 R 1 in (16) is the unique solution, if it exists, to the following optimization problems
max P ∈ △ Q 1 ( Y ) ∫ h ( x , y ) d P ( y ) + λ α D ( P ∥ Q 1 ) ; (20) \max_{P\in\triangle_{Q_1}(\mathcal{Y})}\int h(x,y)\,dP(y)+\frac{\lambda}{\alpha}D(P\mathbin\Vert Q_1); \tag*{(20)} P ∈ △ Q 1 ( Y ) max ∫ h ( x , y ) d P ( y ) + α λ D ( P ∥ Q 1 ) ; (20) max P ∈ △ Q 1 ( Y ) ∫ h ( x , y ) d P ( y ) s.t. D ( P ∥ Q 1 ) ≤ D ( P Y ∣ X = x ( h , Q 1 , λ α ) ∥ Q 1 ) . (21) \begin{aligned}
\max_{P\in\triangle_{Q_1}(\mathcal{Y})}\quad&\int h(x,y)\,dP(y)\\
\text{s.t.}\quad&D(P\mathbin\Vert Q_1)\leq D\left(P^{(h,Q_1,\frac{\lambda}{\alpha})}_{Y\mid X=x}\mathbin\Vert Q_1\right). \tag*{(21)}
\end{aligned} P ∈ △ Q 1 ( Y ) max s.t. ∫ h ( x , y ) d P ( y ) D ( P ∥ Q 1 ) ≤ D ( P Y ∣ X = x ( h , Q 1 , α λ ) ∥ Q 1 ) . (21)
Corollary 5 fully characterizes the effect of changing either the regularization factor λ \lambda λ or the reference measure Q Q Q in the optimization problems (7) , (8) , (9), and (10), which is a central question in statistical learning theory. See, for instance, [1 ] , [2 ] , [12 ] , [15 ] , and references therein. In a nutshell, if the solution to the optimization problem in (7) is known, an ( α , Q , Q 1 ) (\alpha,Q,Q_1) ( α , Q , Q 1 ) -renormalization of such a solution leads to the solution to the optimization problem in (18) in which the regularization factor is λ α \frac{\lambda}{\alpha} α λ and the reference measure is Q 1 Q_1 Q 1 .
Normalized Log-Linear Combinations of Gibbs Probability Measures
The normalization of the product of powers of densities of probability measures, which is often referred to as normalized log-linear combination of probability measures , is a generalization of renormalization (Section III-A). Such a generalization, which is not specific to Gibbs probability measures, has been studied in [28 ] , [32 ] , [33 ] , and references therein. The definition of such an operation is as follows:
Definition 5 (Normalized Log-Linear Combination of Measures). Consider a vector α = ( α 1 , α 2 , … , α K ) ∈ R K \alpha=(\alpha_1,\alpha_2,\ldots,\alpha_K)\in\mathbb{R}^{K} α = ( α 1 , α 2 , … , α K ) ∈ R K , such that for all i ∈ { 1 , 2 , … , K } i\in\{1,2,\ldots,K\} i ∈ { 1 , 2 , … , K } , α i ∈ R ∖ { 0 } \alpha_i\in\mathbb{R}\setminus\{0\} α i ∈ R ∖ { 0 } . Let P 1 , P 2 , … , P K P_1,P_2,\ldots,P_K P 1 , P 2 , … , P K be probability measures, let Q 1 , Q 2 , … , Q K Q_1,Q_2,\ldots,Q_K Q 1 , Q 2 , … , Q K be σ \sigma σ -finite measures, and let Q Q Q be a σ \sigma σ -finite measure, all on the same measurable space. Assume that for all i ∈ { 1 , 2 , … , K } i\in\{1,2,\ldots,K\} i ∈ { 1 , 2 , … , K } , P i ≪ Q i P_i\ll Q_i P i ≪ Q i and Q ≪ Q i Q\ll Q_i Q ≪ Q i . Then, the normalized log-linear combination of P 1 , P 2 , … , P K P_1,P_2,\ldots,P_K P 1 , P 2 , … , P K with respect to Q Q Q is a probability measure, denoted by S K S_K S K , such that for all y ∈ supp Q y\in\operatorname{supp}Q y ∈ supp Q ,
d S K d Q ( y ) = ∏ i = 1 K ( d P i d Q i ( y ) ) α i ∫ ∏ i = 1 K ( d P i d Q i ( y ~ ) ) α i d Q ( y ~ ) , (22) \frac{dS_K}{dQ}(y)=\frac{\prod_{i=1}^{K}\left(\frac{dP_i}{dQ_i}(y)\right)^{\alpha_i}}{\int\prod_{i=1}^{K}\left(\frac{dP_i}{dQ_i}(\tilde{y})\right)^{\alpha_i}\,dQ(\tilde{y})}, \tag*{(22)} d Q d S K ( y ) = ∫ ∏ i = 1 K ( d Q i d P i ( y ~ ) ) α i d Q ( y ~ ) ∏ i = 1 K ( d Q i d P i ( y ) ) α i , (22)
provided that 0 < ∫ ∏ i = 1 K ( d P i d Q i ( y ~ ) ) α i d Q ( y ~ ) < + ∞ 0<\int\prod_{i=1}^{K}\left(\frac{dP_i}{dQ_i}(\tilde{y})\right)^{\alpha_i}\,dQ(\tilde{y})<+\infty 0 < ∫ ∏ i = 1 K ( d Q i d P i ( y ~ ) ) α i d Q ( y ~ ) < + ∞ .
Note that if K = 1 K = 1 K = 1 , the probability measure S K S_K S K in (22) is the ( α 1 , Q 1 , Q ) (\alpha_1,Q_1,Q) ( α 1 , Q 1 , Q ) -renormalization of P 1 P_1 P 1 , which highlights the generalization with respect to renormalization. The following corollary, which follows from Definition 5, reinforces this view.
Corollary 6. Consider the normalized log-linear combination S K S_K S K in (22) under the assumptions that for all i ∈ { 2 , 3 , … , K } i \in\{2,3,\ldots,K\} i ∈ { 2 , 3 , … , K } and for all y ∈ supp Q y \in\operatorname{supp} Q y ∈ supp Q ,
d P i d Q i ( y ) = d P 1 d Q 1 ( y ) ; and (23) \frac{\mathrm{d}P_i}{\mathrm{d}Q_i}(y) = \frac{\mathrm{d}P_1}{\mathrm{d}Q_1}(y); \quad\text{and} \tag*{(23)} d Q i d P i ( y ) = d Q 1 d P 1 ( y ) ; and (23) α ≜ ∑ i = 1 K α i ≠ 0. (24) \alpha\triangleq\sum_{i=1}^{K} \alpha_i \ne0. \tag*{(24)} α ≜ i = 1 ∑ K α i = 0. (24) Then, the normalized log-linear combination S K S_K S K is identical to the ( α , Q 1 , Q ) (\alpha,Q_1,Q) ( α , Q 1 , Q ) -renormalization of P 1 P_1 P 1 .
Remark 2. In the case in which K = 2 K = 2 K = 2 , with Q 1 = Q 2 = Q Q_1 = Q_2 = Q Q 1 = Q 2 = Q and α 1 = 1 − α 2 = α ∈ ( 0 , 1 ) \alpha_1 = 1 - \alpha_2 = \alpha\in(0,1) α 1 = 1 − α 2 = α ∈ ( 0 , 1 ) , for all y ∈ supp Q y \in\operatorname{supp} Q y ∈ supp Q the equality in (22) reduces to
d S 2 d Q ( y ) = ( d P 1 d Q ( y ) ) α ( d P 2 d Q ( y ) ) 1 − α ∫ ( d P 1 d Q ( y ~ ) ) α ( d P 2 d Q ( y ~ ) ) 1 − α d Q ( y ~ ) , (25) \frac{\mathrm{d}S_2}{\mathrm{d}Q}(y) = \frac{\left(\frac{\mathrm{d}P_1}{\mathrm{d}Q}(y)\right)^\alpha\left(\frac{\mathrm{d}P_2}{\mathrm{d}Q}(y)\right)^{1-\alpha}}{\int\left(\frac{\mathrm{d}P_1}{\mathrm{d}Q}(\widetilde{y})\right)^\alpha\left(\frac{\mathrm{d}P_2}{\mathrm{d}Q}(\widetilde{y})\right)^{1-\alpha} \mathrm{d}Q(\widetilde{y})}, \tag*{(25)} d Q d S 2 ( y ) = ∫ ( d Q d P 1 ( y ) ) α ( d Q d P 2 ( y ) ) 1 − α d Q ( y ) ( d Q d P 1 ( y ) ) α ( d Q d P 2 ( y ) ) 1 − α , (25)
where the denominator admits several connections with classical quantities in information theory. More specifically, the constant
∫ ( d P 1 d Q ( y ) ) α ( d P 2 d Q ( y ) ) 1 − α d Q ( y ) (26) \int\left(\frac{\mathrm{d}P_1}{\mathrm{d}Q}(y)\right)^\alpha\left(\frac{\mathrm{d}P_2}{\mathrm{d}Q}(y)\right)^{1-\alpha} \mathrm{d}Q(y) \tag*{(26)} ∫ ( d Q d P 1 ( y ) ) α ( d Q d P 2 ( y ) ) 1 − α d Q ( y ) (26)
is the Hellinger integral of order α \alpha α involving P 1 P_1 P 1 and P 2 P_2 P 2 , see for instance [51 ] . Such an integral plays a key role in the definition of the Rényi divergence [52 ] , [53 ] between the measures P 1 P_1 P 1 and P 2 P_2 P 2 , which is defined as follows
D α ( P 1 ∥ P 2 ) ≜ 1 α − 1 log ∫ ( d P 1 d Q ( y ) ) α ( d P 2 d Q ( y ) ) 1 − α d Q ( y ) . (27) D_\alpha(P_1\mathbin\Vert P_2) \triangleq\frac{1}{\alpha- 1} \log\int\left(\frac{\mathrm{d}P_1}{\mathrm{d}Q}(y)\right)^\alpha\left(\frac{\mathrm{d}P_2}{\mathrm{d}Q}(y)\right)^{1-\alpha} \mathrm{d}Q(y). \tag*{(27)} D α ( P 1 ∥ P 2 ) ≜ α − 1 1 log ∫ ( d Q d P 1 ( y ) ) α ( d Q d P 2 ( y ) ) 1 − α d Q ( y ) . (27)
The Hellinger integral involving the measures P 1 P_1 P 1 and P 2 P_2 P 2 also appears in the definition of the Chernoff coefficient,
inf α ∈ [ 0 , 1 ] ∫ ( d P 1 d Q ( y ) ) α ( d P 2 d Q ( y ) ) 1 − α d Q ( y ) , (28) \inf_{\alpha\in[0,1]} \int\left(\frac{\mathrm{d}P_1}{\mathrm{d}Q}(y)\right)^\alpha\left(\frac{\mathrm{d}P_2}{\mathrm{d}Q}(y)\right)^{1-\alpha} \mathrm{d}Q(y), \tag*{(28)} α ∈ [ 0 , 1 ] inf ∫ ( d Q d P 1 ( y ) ) α ( d Q d P 2 ( y ) ) 1 − α d Q ( y ) , (28)
and the Chernoff information [54 ] ,
− log inf α ∈ [ 0 , 1 ] ∫ ( d P 1 d Q ( y ) ) α ( d P 2 d Q ( y ) ) 1 − α d Q ( y ) . (29) -\log\inf_{\alpha\in[0,1]} \int\left(\frac{\mathrm{d}P_1}{\mathrm{d}Q}(y)\right)^\alpha\left(\frac{\mathrm{d}P_2}{\mathrm{d}Q}(y)\right)^{1-\alpha} \mathrm{d}Q(y). \tag*{(29)} − log α ∈ [ 0 , 1 ] inf ∫ ( d Q d P 1 ( y ) ) α ( d Q d P 2 ( y ) ) 1 − α d Q ( y ) . (29)
In the case in which α = 1 2 \alpha= \frac{1}{2} α = 2 1 , the quantity in (26) is the Bhattacharyya coefficient induced by P 1 P_1 P 1 and P 2 P_2 P 2 ; and
− log ∫ ( d P 1 d Q ( y ) ) 1 2 ( d P 2 d Q ( y ) ) 1 2 d Q ( y ) , (30) -\log\int\left(\frac{\mathrm{d}P_1}{\mathrm{d}Q}(y)\right)^{\frac{1}{2}} \left(\frac{\mathrm{d}P_2}{\mathrm{d}Q}(y)\right)^{\frac{1}{2}} \mathrm{d}Q(y), \tag*{(30)} − log ∫ ( d Q d P 1 ( y ) ) 2 1 ( d Q d P 2 ( y ) ) 2 1 d Q ( y ) , (30)
is the Bhattacharyya distance between P 1 P_1 P 1 and P 2 P_2 P 2 , see for instance [55 ] . Finally, it is worth noting that in the case in which Q ≪ P 2 Q \ll P_2 Q ≪ P 2 , the integral in (26) satisfies
∫ ( d P 1 d Q ( y ) ) α ( d P 2 d Q ( y ) ) 1 − α d Q ( y ) = ∫ exp ( α log d P 1 d P 2 ( y ) ) d P 2 ( y ) , (31) \int\left(\frac{\mathrm{d}P_1}{\mathrm{d}Q}(y)\right)^\alpha\left(\frac{\mathrm{d}P_2}{\mathrm{d}Q}(y)\right)^{1-\alpha} \mathrm{d}Q(y) = \int\exp\left(\alpha\log\frac{\mathrm{d}P_1}{\mathrm{d}P_2}(y)\right) \mathrm{d}P_2(y), \tag*{(31)} ∫ ( d Q d P 1 ( y ) ) α ( d Q d P 2 ( y ) ) 1 − α d Q ( y ) = ∫ exp ( α log d P 2 d P 1 ( y ) ) d P 2 ( y ) , (31)
which is also the moment generating function, evaluated at α \alpha α , of the log-likelihood ratio log d P 1 d P 2 ( y ) \log\frac{\mathrm{d}P_1}{\mathrm{d}P_2}(y) log d P 2 d P 1 ( y ) , when y y y is sampled from P 2 P_2 P 2 . These observations are important as they allow, in some specific cases, to write the right-hand side of (25) in terms of information measures, including the Rényi entropy, among others.
In the following, the focus is exclusively on the normalized log-linear combination of K K K Gibbs probability measures, which are defined using the following parameters. For all i ∈ { 1 , 2 , … , K } i \in\{1,2,\ldots,K\} i ∈ { 1 , 2 , … , K } , consider a Borel measurable function
h i : X × Y → R ; (32) h_i:\mathcal{X} \times\mathcal{Y} \to\mathbb{R}; \tag*{(32)} h i : X × Y → R ; (32)
a real λ i ∈ R ∖ { 0 } \lambda_i \in\mathbb{R} \setminus\{0\} λ i ∈ R ∖ { 0 } ; and a σ \sigma σ -finite measure Q i Q_i Q i on ( Y , B ( Y ) ) (\mathcal{Y},\mathcal{B}(\mathcal{Y})) ( Y , B ( Y )) . Hence, the ( h i , Q i , λ i ) (h_i,Q_i,\lambda_i) ( h i , Q i , λ i ) -Gibbs conditional probability measures of interest are
P Y ∣ X ( h i , Q i , λ i ) ∈ △ ( Y ∣ X ) . (33) P_{Y\mid X}^{(h_i,Q_i,\lambda_i)} \in\triangle(\mathcal{Y}\mid\mathcal{X}). \tag*{(33)} P Y ∣ X ( h i , Q i , λ i ) ∈ △ ( Y ∣ X ) . (33)
For a fixed vector ( x 1 , x 2 , … , x K ) ∈ X K (x_1,x_2,\ldots,x_K) \in\mathcal{X}^K ( x 1 , x 2 , … , x K ) ∈ X K , the normalized log-linear combination of the Gibbs probability measures P Y ∣ X = x 1 ( h 1 , Q 1 , λ 1 ) , P Y ∣ X = x 2 ( h 2 , Q 2 , λ 2 ) , … , P Y ∣ X = x K ( h K , Q K , λ K ) P^{(h_1,Q_1,\lambda_1)}_{Y|X=x_1}, P^{(h_2,Q_2,\lambda_2)}_{Y|X=x_2},\ldots,P^{(h_K,Q_K,\lambda_K)}_{Y|X=x_K} P Y ∣ X = x 1 ( h 1 , Q 1 , λ 1 ) , P Y ∣ X = x 2 ( h 2 , Q 2 , λ 2 ) , … , P Y ∣ X = x K ( h K , Q K , λ K ) , with respect to some given σ \sigma σ -finite measure Q Q Q , satisfies for all y ∈ supp Q y \in\operatorname{supp} Q y ∈ supp Q ,
d S K d Q ( y ) = ∏ i = 1 K ( d P Y ∣ X = x i ( h i , Q i , λ i ) d Q i ( y ) ) α i ∫ ∏ i = 1 K ( d P Y ∣ X = x i ( h i , Q i , λ i ) d Q i ( y ~ ) ) α i d Q ( y ~ ) . (34) \frac{\mathrm{d}S_K}{\mathrm{d}Q}(y)=\frac{\prod_{i=1}^{K}\left(\frac{\mathrm{d}P^{(h_i,Q_i,\lambda_i)}_{Y|X=x_i}}{\mathrm{d}Q_i}(y)\right)^{\alpha_i}}{\int\prod_{i=1}^{K}\left(\frac{\mathrm{d}P^{(h_i,Q_i,\lambda_i)}_{Y|X=x_i}}{\mathrm{d}Q_i}(\tilde{y})\right)^{\alpha_i}\mathrm{d}Q(\tilde{y})}. \tag*{(34)} d Q d S K ( y ) = ∫ ∏ i = 1 K ( d Q i d P Y ∣ X = x i ( h i , Q i , λ i ) ( y ~ ) ) α i d Q ( y ~ ) ∏ i = 1 K ( d Q i d P Y ∣ X = x i ( h i , Q i , λ i ) ( y ) ) α i . (34)
In this section, the vector ( x 1 , x 2 , … , x K ) ∈ X K (x_1,x_2,\ldots,x_K) \in\mathcal{X}^K ( x 1 , x 2 , … , x K ) ∈ X K used in (34) remains fixed. Its role becomes apparent in § 5 . The following theorem provides an explicit expression for the Radon–Nikodym derivative d S K d Q \frac{\mathrm{d}S_K}{\mathrm{d}Q} d Q d S K in (34) . For this purpose, consider the function
h ^ β : { X K × Y ⟶ R ( x , y ) ⟼ ∑ i = 1 K β i h i ( x i , y ) , (35) \widehat{h}_{\boldsymbol{\beta}}:
\begin{cases}
\mathcal{X}^K \times\mathcal{Y} \longrightarrow\mathbb{R}\\
(\boldsymbol{x},y)\longmapsto\sum_{i=1}^{K}\beta_i h_i(x_i,y),
\end{cases} \tag*{(35)} h β : { X K × Y ⟶ R ( x , y ) ⟼ ∑ i = 1 K β i h i ( x i , y ) , (35)
where β = ( β 1 , β 2 , … , β K ) \boldsymbol{\beta}=(\beta_1,\beta_2,\ldots,\beta_K) β = ( β 1 , β 2 , … , β K ) is a vector of parameters, h i h_i h i is the Borel measurable function in (32) , and x = ( x 1 , x 2 , … , x K ) ∈ X K \boldsymbol{x}=(x_1,x_2,\ldots,x_K)\in\mathcal{X}^K x = ( x 1 , x 2 , … , x K ) ∈ X K . Using this notation, the following theorem introduces the main result of this subsection.
Theorem 7. The normalized log-linear combination S K S_K S K in (34) , satisfies for all y ∈ supp Q y \in\operatorname{supp} Q y ∈ supp Q ,
d S K d Q ( y ) = exp ( − ∑ i = 1 K α i λ i h i ( x i , y ) ) ∫ exp ( − ∑ i = 1 K α i λ i h i ( x i , y ~ ) ) d Q ( y ~ ) (36) \frac{\mathrm{d}S_K}{\mathrm{d}Q}(y)=\frac{\exp\left(-\sum_{i=1}^{K}\frac{\alpha_i}{\lambda_i}h_i(x_i,y)\right)}{\int\exp\left(-\sum_{i=1}^{K}\frac{\alpha_i}{\lambda_i}h_i(x_i,\tilde{y})\right)\mathrm{d}Q(\tilde{y})} \tag*{(36)} d Q d S K ( y ) = ∫ exp ( − ∑ i = 1 K λ i α i h i ( x i , y ~ ) ) d Q ( y ~ ) exp ( − ∑ i = 1 K λ i α i h i ( x i , y ) ) (36) = d P Y ∣ X = x ( h ^ β , Q , 1 ) d Q ( y ) , (37) =\frac{\mathrm{d}P^{(\widehat{h}_{\boldsymbol{\beta}},Q,1)}_{Y|X=\boldsymbol{x}}}{\mathrm{d}Q}(y), \tag*{(37)} = d Q d P Y ∣ X = x ( h β , Q , 1 ) ( y ) , (37) where x = ( x 1 , x 2 , … , x K ) \boldsymbol{x}=(x_1,x_2,\ldots,x_K) x = ( x 1 , x 2 , … , x K ) ; β = ( α 1 λ 1 , α 2 λ 2 , … , α K λ K ) \boldsymbol{\beta}=\left(\frac{\alpha_1}{\lambda_1},\frac{\alpha_2}{\lambda_2},\ldots,\frac{\alpha_K}{\lambda_K}\right) β = ( λ 1 α 1 , λ 2 α 2 , … , λ K α K ) ; the function h ^ β \widehat{h}_{\boldsymbol{\beta}} h β is defined in (35) ; and the measure P Y ∣ X = x ( h ^ β , Q , 1 ) P^{(\widehat{h}_{\boldsymbol{\beta}},Q,1)}_{Y|X=\boldsymbol{x}} P Y ∣ X = x ( h β , Q , 1 ) is an ( h ^ β , Q , 1 ) (\widehat{h}_{\boldsymbol{\beta}},Q,1) ( h β , Q , 1 ) -Gibbs probability measure in △ ( Y ) \triangle(\mathcal{Y}) △ ( Y ) .
Proof. The proof is presented in Appendix C. ■
Theorem 7 unveils two important observations concerning the effect of log-linearly combining Gibbs probability measures. On one hand, the normalized log-linear combination S K S_K S K in (34) is identical to the ( h ^ β , Q , 1 ) (\widehat{h}_{\boldsymbol{\beta}},Q,1) ( h β , Q , 1 ) -Gibbs probability measure P Y ∣ X = x ( h ^ β , Q , 1 ) P^{(\widehat{h}_{\boldsymbol{\beta}},Q,1)}_{Y|X=\boldsymbol{x}} P Y ∣ X = x ( h β , Q , 1 ) in (37) . Interestingly, the objective function h ^ β \widehat{h}_{\boldsymbol{\beta}} h β parametrizing the Gibbs conditional probability measure P Y ∣ X ( h ^ β , Q , 1 ) P^{(\widehat{h}_{\boldsymbol{\beta}},Q,1)}_{Y|X} P Y ∣ X ( h β , Q , 1 ) is the linear combination of the functions h 1 , h 2 , … , h K h_1,h_2,\ldots,h_K h 1 , h 2 , … , h K parametrizing the Gibbs conditional probability measures P Y ∣ X ( h 1 , Q 1 , λ 1 ) , P Y ∣ X ( h 2 , Q 2 , λ 2 ) , … , P Y ∣ X ( h K , Q K , λ K ) P^{(h_1,Q_1,\lambda_1)}_{Y|X},P^{(h_2,Q_2,\lambda_2)}_{Y|X},\ldots,P^{(h_K,Q_K,\lambda_K)}_{Y|X} P Y ∣ X ( h 1 , Q 1 , λ 1 ) , P Y ∣ X ( h 2 , Q 2 , λ 2 ) , … , P Y ∣ X ( h K , Q K , λ K ) in (33) , with weights given by β 1 = α 1 λ 1 \beta_1=\frac{\alpha_1}{\lambda_1} β 1 = λ 1 α 1 , β 2 = α 2 λ 2 , … , β K = α K λ K \beta_2=\frac{\alpha_2}{\lambda_2},\ldots,\beta_K=\frac{\alpha_K}{\lambda_K} β 2 = λ 2 α 2 , … , β K = λ K α K , respectively. On the other hand, for all i ∈ { 1 , 2 , … , K } i\in\{1,2,\ldots,K\} i ∈ { 1 , 2 , … , K } , it holds that Q ≪ Q i Q\ll Q_i Q ≪ Q i and thus,
supp S K = supp Q ⊆ supp Q i = supp P Y ∣ X = x i ( h i , Q i , λ i ) . (38) \operatorname{supp} S_K=\operatorname{supp} Q\subseteq\operatorname{supp} Q_i=\operatorname{supp}P^{(h_i,Q_i,\lambda_i)}_{Y|X=x_i}. \tag*{(38)} supp S K = supp Q ⊆ supp Q i = supp P Y ∣ X = x i ( h i , Q i , λ i ) . (38)
Essentially, the choice of the reference measure Q Q Q might imply a concentration of the support of S K S_K S K on a support contained within the intersection of the supports of all the Gibbs probability measures taking part in the normalized log-linear combination. The following theorem further characterizes the identity in (37) through equivalent formulations involving the Radon–Nikodym derivative, the relative entropy, and some optimization problems involving relative entropies.
Theorem 8. Given the measure S K S_K S K in (34) and the measure P Y ∣ X = x ( h ^ β , Q , 1 ) P^{(\widehat{h}_{\boldsymbol{\beta}},Q,1)}_{Y|X=\boldsymbol{x}} P Y ∣ X = x ( h β , Q , 1 ) in (37) , the following statements are equivalent:
(i) For all y ∈ supp Q y\in\operatorname{supp} Q y ∈ supp Q ,
d S K d P Y ∣ X = x ( h ^ β , Q , 1 ) ( y ) = 1 ; (39) \frac{\mathrm{d}S_K}{\mathrm{d}P^{(\widehat{h}_{\boldsymbol{\beta}},Q,1)}_{Y|X=\boldsymbol{x}}}(y)=1; \tag*{(39)} d P Y ∣ X = x ( h β , Q , 1 ) d S K ( y ) = 1 ; (39)
(ii)
D ( S K ∥ P Y ∣ X = x ( h β , Q , 1 ) ) = 0 ; (40) D\left(S_K \,\middle\|\, P^{(h_{\boldsymbol{\beta},Q,1})}_{Y\mid X=x}\right)=0; \tag*{(40)} D ( S K P Y ∣ X = x ( h β , Q , 1 ) ) = 0 ; (40)
(iii) The probability measure S K S_K S K is the unique solution, if it exists, to the following optimization problem
min P ∈ △ Q ( Y ) ∫ ( ∑ i = 1 K α i λ i h i ( x i , y ) ) d P ( y ) + D ( P ∥ Q ) ; (41) \min_{P\in\triangle_Q(\mathcal{Y})}\int\left(\sum_{i=1}^{K}\frac{\alpha_i}{\lambda_i}h_i(x_i,y)\right)\mathrm{d}P(y)+D(P\parallel Q); \tag*{(41)} P ∈ △ Q ( Y ) min ∫ ( i = 1 ∑ K λ i α i h i ( x i , y ) ) d P ( y ) + D ( P ∥ Q ) ; (41)
(iv) The probability measure S K S_K S K is the unique solution, if it exists, to the following optimization problem
min P ∈ △ Q ( Y ) ∫ ( ∑ i = 1 K α i λ i h i ( x i , y ) ) d P ( y ) \min_{P\in\triangle_Q(\mathcal{Y})}\int\left(\sum_{i=1}^{K}\frac{\alpha_i}{\lambda_i}h_i(x_i,y)\right)\mathrm{d}P(y) P ∈ △ Q ( Y ) min ∫ ( i = 1 ∑ K λ i α i h i ( x i , y ) ) d P ( y )
s.t. D ( P ∥ Q ) ≤ D ( P Y ∣ X = x ( h β , Q , 1 ) ∥ Q ) . (42) \text{s.t.}\quad D(P\parallel Q)\leq D\left(P^{(h_{\boldsymbol{\beta},Q,1})}_{Y\mid X=x}\middle\|Q\right). \tag*{(42)} s.t. D ( P ∥ Q ) ≤ D ( P Y ∣ X = x ( h β , Q , 1 ) Q ) . (42)
Proof: The proof is presented in Appendix D.
An interesting observation from Theorem 8 is that the optimization problems in (41) and (42) do not depend on the σ \sigma σ -finite measures Q 1 , Q 2 , … , Q K Q_1,Q_2,\ldots,Q_K Q 1 , Q 2 , … , Q K , but exclusively on the σ \sigma σ -finite measure Q Q Q .
Nested Gibbs Probability Measures
Given a Gibbs probability measure, constructing a new Gibbs probability measure by using the given measure as the reference measure is known as nesting Gibbs probability measures . The given Gibbs measure is then said to be nested within the resulting Gibbs measure, yielding a nested Gibbs probability measure . Nested Gibbs measures form a special class of Gibbs measures with important applications in statistical learning theory as shown in the remaining of this work. A formal definition is as follows.
Definition 6 (Nested Gibbs Probability Measures). An ( h , Q , λ ) (h,Q,\lambda) ( h , Q , λ ) -Gibbs probability measure P Y ∣ X = x ( h , Q , λ ) ∈ △ ( Y ) P^{(h,Q,\lambda)}_{Y\mid X=x}\in\triangle(\mathcal{Y}) P Y ∣ X = x ( h , Q , λ ) ∈ △ ( Y ) , of the form in (6), is said to be a nested Gibbs probability measure if Q Q Q is a Gibbs probability measure.
For a fixed vector ( x 1 , x 2 ) ∈ X 2 (x_1,x_2)\in\mathcal{X}^2 ( x 1 , x 2 ) ∈ X 2 , consider the ( h 1 , Q 1 , λ 1 ) (h_1,Q_1,\lambda_1) ( h 1 , Q 1 , λ 1 ) -Gibbs probability measure P Y ∣ X = x 1 ( h 1 , Q 1 , λ 1 ) P^{(h_1,Q_1,\lambda_1)}_{Y\mid X=x_1} P Y ∣ X = x 1 ( h 1 , Q 1 , λ 1 ) and the ( h 2 , Q 2 , λ 2 ) (h_2,Q_2,\lambda_2) ( h 2 , Q 2 , λ 2 ) -Gibbs probability measure P Y ∣ X = x 2 ( h 2 , Q 2 , λ 2 ) P^{(h_2,Q_2,\lambda_2)}_{Y\mid X=x_2} P Y ∣ X = x 2 ( h 2 , Q 2 , λ 2 ) obtained by conditioning the Gibbs conditional probability measures in (33) . In the following, the measure P Y ∣ X = x 1 ( h 1 , Q 1 , λ 1 ) P^{(h_1,Q_1,\lambda_1)}_{Y\mid X=x_1} P Y ∣ X = x 1 ( h 1 , Q 1 , λ 1 ) is said to be nested within P Y ∣ X = x 2 ( h 2 , Q 2 , λ 2 ) P^{(h_2,Q_2,\lambda_2)}_{Y\mid X=x_2} P Y ∣ X = x 2 ( h 2 , Q 2 , λ 2 ) , as Q 2 Q_2 Q 2 is chosen to be identical to P Y ∣ X = x 1 ( h 1 , Q 1 , λ 1 ) P^{(h_1,Q_1,\lambda_1)}_{Y\mid X=x_1} P Y ∣ X = x 1 ( h 1 , Q 1 , λ 1 ) . This construction yields the ( h 2 , P Y ∣ X = x 1 ( h 1 , Q 1 , λ 1 ) , λ 2 ) (h_2,P^{(h_1,Q_1,\lambda_1)}_{Y\mid X=x_1},\lambda_2) ( h 2 , P Y ∣ X = x 1 ( h 1 , Q 1 , λ 1 ) , λ 2 ) -Gibbs probability measure, denoted by
P Y ∣ X = x 2 ( h 2 , P Y ∣ X = x 1 ( h 1 , Q 1 , λ 1 ) , λ 2 ) ∈ △ ( Y ) . (43) P^{(h_2,P^{(h_1,Q_1,\lambda_1)}_{Y\mid X=x_1},\lambda_2)}_{Y\mid X=x_2}\in\triangle(\mathcal{Y}). \tag*{(43)} P Y ∣ X = x 2 ( h 2 , P Y ∣ X = x 1 ( h 1 , Q 1 , λ 1 ) , λ 2 ) ∈ △ ( Y ) . (43)
Such a measure is a nested Gibbs probability measure in the sense of Definition 6. Note that P Y ∣ X = x 1 ( h 1 , Q 1 , λ 1 ) P^{(h_1,Q_1,\lambda_1)}_{Y\mid X=x_1} P Y ∣ X = x 1 ( h 1 , Q 1 , λ 1 ) and Q 1 Q_1 Q 1 are mutually absolutely continuous [15 Lemma 3] . Then, △ P Y ∣ X = x 1 ( h 1 , Q 1 , λ 1 ) ( Y ) = △ Q 1 ( Y ) \triangle_{P^{(h_1,Q_1,\lambda_1)}_{Y\mid X=x_1}}(\mathcal{Y})=\triangle_{Q_1}(\mathcal{Y}) △ P Y ∣ X = x 1 ( h 1 , Q 1 , λ 1 ) ( Y ) = △ Q 1 ( Y ) . Using this observation, the following corollary, which follows from Lemma 1 and Lemma 2, unveils an important connection between P Y ∣ X = x 2 ( h 2 , P Y ∣ X = x 1 ( h 1 , Q 1 , λ 1 ) , λ 2 ) P^{(h_2,P^{(h_1,Q_1,\lambda_1)}_{Y\mid X=x_1},\lambda_2)}_{Y\mid X=x_2} P Y ∣ X = x 2 ( h 2 , P Y ∣ X = x 1 ( h 1 , Q 1 , λ 1 ) , λ 2 ) and particular optimization problems involving relative entropies.
Corollary 9. When λ 2 > 0 \lambda_2>0 λ 2 > 0 , the nested Gibbs probability measure P Y ∣ X = x 2 ( h 2 , P Y ∣ X = x 1 ( h 1 , Q 1 , λ 1 ) , λ 2 ) P^{(h_2,P^{(h_1,Q_1,\lambda_1)}_{Y\mid X=x_1},\lambda_2)}_{Y\mid X=x_2} P Y ∣ X = x 2 ( h 2 , P Y ∣ X = x 1 ( h 1 , Q 1 , λ 1 ) , λ 2 ) in (43) is the unique solution, if it exists, to the following optimization problems:
min P ∈ △ Q 1 ( Y ) ∫ h 2 ( x 2 , y ) d P ( y ) + λ 2 D ( P ∥ P Y ∣ X = x 1 ( h 1 , Q 1 , λ 1 ) ) ; (44) \min_{P\in\triangle_{Q_1}(\mathcal{Y})}\int h_2(x_2,y)\mathrm{d}P(y)+\lambda_2D\left(P\middle\|P^{(h_1,Q_1,\lambda_1)}_{Y\mid X=x_1}\right); \tag*{(44)} P ∈ △ Q 1 ( Y ) min ∫ h 2 ( x 2 , y ) d P ( y ) + λ 2 D ( P P Y ∣ X = x 1 ( h 1 , Q 1 , λ 1 ) ) ; (44) min P ∈ △ Q 1 ( Y ) ∫ h 2 ( x 2 , y ) d P ( y ) s.t. D ( P ∥ P Y ∣ X = x 1 ( h 1 , Q 1 , λ 1 ) ) ≤ D ( P Y ∣ X = x 2 ( h 2 , P Y ∣ X = x 1 ( h 1 , Q 1 , λ 1 ) , λ 2 ) ∥ P Y ∣ X = x 1 ( h 1 , Q 1 , λ 1 ) ) . (45) \begin{aligned}
\min_{P\in\triangle_{Q_1}(\mathcal{Y})}\quad&\int h_2(x_2,y)\mathrm{d}P(y)\\
\text{s.t.}\quad&D\left(P\middle\|P^{(h_1,Q_1,\lambda_1)}_{Y\mid X=x_1}\right)\leq D\left(P^{(h_2,P^{(h_1,Q_1,\lambda_1)}_{Y\mid X=x_1},\lambda_2)}_{Y\mid X=x_2}\middle\|P^{(h_1,Q_1,\lambda_1)}_{Y\mid X=x_1}\right). \tag*{(45)}
\end{aligned} P ∈ △ Q 1 ( Y ) min s.t. ∫ h 2 ( x 2 , y ) d P ( y ) D ( P P Y ∣ X = x 1 ( h 1 , Q 1 , λ 1 ) ) ≤ D ( P Y ∣ X = x 2 ( h 2 , P Y ∣ X = x 1 ( h 1 , Q 1 , λ 1 ) , λ 2 ) P Y ∣ X = x 1 ( h 1 , Q 1 , λ 1 ) ) . (45)
Alternatively, when λ 2 < 0 \lambda_{2}<0 λ 2 < 0 , the measure P Y ∣ X = x 2 ( h 2 , P Y ∣ X = x 1 ( h 1 , Q 1 , λ 1 ) , λ 2 ) P^{(h_{2},P_{Y\mid X=x_{1}}^{(h_{1},Q_{1},\lambda_{1})},\lambda_{2})}_{Y\mid X=x_{2}} P Y ∣ X = x 2 ( h 2 , P Y ∣ X = x 1 ( h 1 , Q 1 , λ 1 ) , λ 2 ) in (43) is the unique solution, if it exists, to the following optimization problems
max P ∈ △ Q 1 ( Y ) ∫ h 2 ( x 2 , y ) d P ( y ) + λ 2 D ( P ∥ P Y ∣ X = x 1 ( h 1 , Q 1 , λ 1 ) ) ; (46) \max_{P\in\triangle_{Q_{1}}(\mathcal{Y})}\int h_{2}(x_{2},y)\mathrm{d}P(y)+\lambda_{2}D\left(P\mathbin{\|}P_{Y\mid X=x_{1}}^{(h_{1},Q_{1},\lambda_{1})}\right); \tag*{(46)} P ∈ △ Q 1 ( Y ) max ∫ h 2 ( x 2 , y ) d P ( y ) + λ 2 D ( P ∥ P Y ∣ X = x 1 ( h 1 , Q 1 , λ 1 ) ) ; (46)
max P ∈ △ Q 1 ( Y ) ∫ h 2 ( x 2 , y ) d P ( y ) s.t. D ( P ∥ P Y ∣ X = x 1 ( h 1 , Q 1 , λ 1 ) ) ≤ D ( P Y ∣ X = x 2 ( h 2 , P Y ∣ X = x 1 ( h 1 , Q 1 , λ 1 ) , λ 2 ) ∥ P Y ∣ X = x 1 ( h 1 , Q 1 , λ 1 ) ) . (47) \begin{aligned}
\max_{P\in\triangle_{Q_{1}}(\mathcal{Y})}\quad&\int h_{2}(x_{2},y)\mathrm{d}P(y)\\
\text{s.t.}\quad&D\left(P\mathbin{\|}P_{Y\mid X=x_{1}}^{(h_{1},Q_{1},\lambda_{1})}\right)\leq D\left(P_{Y\mid X=x_{2}}^{(h_{2},P_{Y\mid X=x_{1}}^{(h_{1},Q_{1},\lambda_{1})},\lambda_{2})}\mathbin{\|}P_{Y\mid X=x_{1}}^{(h_{1},Q_{1},\lambda_{1})}\right). \tag*{(47)}
\end{aligned} P ∈ △ Q 1 ( Y ) max s.t. ∫ h 2 ( x 2 , y ) d P ( y ) D ( P ∥ P Y ∣ X = x 1 ( h 1 , Q 1 , λ 1 ) ) ≤ D ( P Y ∣ X = x 2 ( h 2 , P Y ∣ X = x 1 ( h 1 , Q 1 , λ 1 ) , λ 2 ) ∥ P Y ∣ X = x 1 ( h 1 , Q 1 , λ 1 ) ) . (47)
The following theorem provides an explicit expression for the Radon-Nikodym derivative of the measure P Y ∣ X = x 2 ( h 2 , P Y ∣ X = x 1 ( h 1 , Q 1 , λ 1 ) , λ 2 ) P^{(h_{2},P_{Y\mid X=x_{1}}^{(h_{1},Q_{1},\lambda_{1})},\lambda_{2})}_{Y\mid X=x_{2}} P Y ∣ X = x 2 ( h 2 , P Y ∣ X = x 1 ( h 1 , Q 1 , λ 1 ) , λ 2 ) in (43) with respect to the measure Q 1 Q_{1} Q 1 .
Theorem 10. The nested Gibbs probability measure P Y ∣ X = x 2 ( h 2 , P Y ∣ X = x 1 ( h 1 , Q 1 , λ 1 ) , λ 2 ) P^{(h_{2},P_{Y\mid X=x_{1}}^{(h_{1},Q_{1},\lambda_{1})},\lambda_{2})}_{Y\mid X=x_{2}} P Y ∣ X = x 2 ( h 2 , P Y ∣ X = x 1 ( h 1 , Q 1 , λ 1 ) , λ 2 ) in (43) satisfies for all y ∈ supp Q 1 y\in\operatorname{supp}Q_{1} y ∈ supp Q 1 ,
d P Y ∣ X = x 2 ( h 2 , P Y ∣ X = x 1 ( h 1 , Q 1 , λ 1 ) , λ 2 ) d Q 1 ( y ) = exp ( − 1 λ 1 h 1 ( x 1 , y ) − 1 λ 2 h 2 ( x 2 , y ) ) ∫ exp ( − 1 λ 1 h 1 ( x 1 , y ~ ) − 1 λ 2 h 2 ( x 2 , y ~ ) ) d Q 1 ( y ~ ) (48) \frac{\mathrm{d}P^{(h_{2},P_{Y\mid X=x_{1}}^{(h_{1},Q_{1},\lambda_{1})},\lambda_{2})}_{Y\mid X=x_{2}}}{\mathrm{d}Q_{1}}(y)=\frac{\exp\left(-\frac{1}{\lambda_{1}}h_{1}(x_{1},y)-\frac{1}{\lambda_{2}}h_{2}(x_{2},y)\right)}{\int\exp\left(-\frac{1}{\lambda_{1}}h_{1}(x_{1},\widetilde{y})-\frac{1}{\lambda_{2}}h_{2}(x_{2},\widetilde{y})\right)\mathrm{d}Q_{1}(\widetilde{y})} \tag*{(48)} d Q 1 d P Y ∣ X = x 2 ( h 2 , P Y ∣ X = x 1 ( h 1 , Q 1 , λ 1 ) , λ 2 ) ( y ) = ∫ exp ( − λ 1 1 h 1 ( x 1 , y ) − λ 2 1 h 2 ( x 2 , y ) ) d Q 1 ( y ) exp ( − λ 1 1 h 1 ( x 1 , y ) − λ 2 1 h 2 ( x 2 , y ) ) (48) = d P Y ∣ X = x ( h ^ β , Q 1 , 1 ) d Q 1 ( y ) , (49) =\frac{\mathrm{d}P^{(\widehat{h}_{\boldsymbol{\beta}},Q_{1},1)}_{Y\mid X=x}}{\mathrm{d}Q_{1}}(y), \tag*{(49)} = d Q 1 d P Y ∣ X = x ( h β , Q 1 , 1 ) ( y ) , (49) where x = ( x 1 , x 2 ) x=(x_{1},x_{2}) x = ( x 1 , x 2 ) ; β = ( 1 λ 1 , 1 λ 2 ) \boldsymbol{\beta}=\left(\frac{1}{\lambda_{1}},\frac{1}{\lambda_{2}}\right) β = ( λ 1 1 , λ 2 1 ) ; the function h ^ β \widehat{h}_{\boldsymbol{\beta}} h β is defined in (35) ; and the measure P Y ∣ X = x ( h ^ β , Q 1 , 1 ) P^{(\widehat{h}_{\boldsymbol{\beta}},Q_{1},1)}_{Y\mid X=x} P Y ∣ X = x ( h β , Q 1 , 1 ) is an ( h ^ β , Q 1 , 1 ) (\widehat{h}_{\boldsymbol{\beta}},Q_{1},1) ( h β , Q 1 , 1 ) -Gibbs probability measure in △ ( Y ) \triangle(\mathcal{Y}) △ ( Y ) .
Proof. The proof is presented in Appendix E.
Theorem 10 shows that the nested Gibbs probability measure P Y ∣ X = x 2 ( h 2 , P Y ∣ X = x 1 ( h 1 , Q 1 , λ 1 ) , λ 2 ) P^{(h_{2},P_{Y\mid X=x_{1}}^{(h_{1},Q_{1},\lambda_{1})},\lambda_{2})}_{Y\mid X=x_{2}} P Y ∣ X = x 2 ( h 2 , P Y ∣ X = x 1 ( h 1 , Q 1 , λ 1 ) , λ 2 ) in (43) is identical to the ( h ^ β , Q 1 , 1 ) (\widehat{h}_{\boldsymbol{\beta}},Q_{1},1) ( h β , Q 1 , 1 ) -Gibbs probability measure P Y ∣ X = x ( h ^ β , Q 1 , 1 ) P^{(\widehat{h}_{\boldsymbol{\beta}},Q_{1},1)}_{Y\mid X=x} P Y ∣ X = x ( h β , Q 1 , 1 ) in (49) . Hence, the nested Gibbs probability measure in (43) is itself a Gibbs probability measure parametrized by the function h ^ β \widehat{h}_{\boldsymbol{\beta}} h β in (35) , with β = ( 1 λ 1 , 1 λ 2 ) \boldsymbol{\beta}=\left(\frac{1}{\lambda_{1}},\frac{1}{\lambda_{2}}\right) β = ( λ 1 1 , λ 2 1 ) , which is a linear combination of the functions h 1 h_{1} h 1 and h 2 h_{2} h 2 , with coefficients 1 λ 1 \frac{1}{\lambda_{1}} λ 1 1 and 1 λ 2 \frac{1}{\lambda_{2}} λ 2 1 , respectively. This observation motivates considering a recursive choice of Gibbs reference measures to form new nested Gibbs measures. For a fixed vector ( x 1 , x 2 , … , x K ) ∈ X K (x_{1},x_{2},\ldots,x_{K})\in\mathcal{X}^{K} ( x 1 , x 2 , … , x K ) ∈ X K , consider the ( h k , Q k , λ k ) (h_{k},Q_{k},\lambda_{k}) ( h k , Q k , λ k ) -Gibbs probability measure, with k ∈ { 1 , 2 , … , K } k\in\{1,2,\ldots,K\} k ∈ { 1 , 2 , … , K } ,
P Y ∣ X = x k ( h k , Q k , λ k ) ∈ △ ( Y ) , (50) P^{(h_{k},Q_{k},\lambda_{k})}_{Y\mid X=x_{k}}\in\triangle(\mathcal{Y}), \tag*{(50)} P Y ∣ X = x k ( h k , Q k , λ k ) ∈ △ ( Y ) , (50)
where the function h k h_{k} h k is defined in (32) ; λ k ∈ R ∖ { 0 } \lambda_{k}\in\mathbb{R}\setminus\{0\} λ k ∈ R ∖ { 0 } ; and the reference measure Q k Q_{k} Q k is recursively defined as
Q k = { Q 1 if k = 1 , P Y ∣ X = x k − 1 ( h k − 1 , Q k − 1 , λ k − 1 ) if k ≥ 2 , (51) Q_{k}=
\begin{cases}
Q_{1} & \text{if } k=1,\\
P^{(h_{k-1},Q_{k-1},\lambda_{k-1})}_{Y\mid X=x_{k-1}} & \text{if } k\geq2,
\end{cases} \tag*{(51)} Q k = { Q 1 P Y ∣ X = x k − 1 ( h k − 1 , Q k − 1 , λ k − 1 ) if k = 1 , if k ≥ 2 , (51)
for some given σ \sigma σ -finite measure Q 1 Q_{1} Q 1 .
For all k ∈ { 2 , … , K } k\in\{2,\ldots,K\} k ∈ { 2 , … , K } , the measure Q k Q_{k} Q k in (51) is a Gibbs probability measure. Hence, the ( h k , Q k , λ k ) (h_{k},Q_{k},\lambda_{k}) ( h k , Q k , λ k ) -Gibbs probability measure P Y ∣ X = x k ( h k , Q k , λ k ) P^{(h_{k},Q_{k},\lambda_{k})}_{Y\mid X=x_{k}} P Y ∣ X = x k ( h k , Q k , λ k ) is a nested Gibbs probability measure in the sense of Definition 6. The following theorem provides an explicit expression for the Radon-Nikodym derivative of the ( h K , Q K , λ K ) (h_{K},Q_{K},\lambda_{K}) ( h K , Q K , λ K ) -Gibbs probability measure P Y ∣ X = x K ( h K , Q K , λ K ) P^{(h_{K},Q_{K},\lambda_{K})}_{Y\mid X=x_{K}} P Y ∣ X = x K ( h K , Q K , λ K ) in (50) with respect to the measure Q 1 Q_{1} Q 1 .
Theorem 11. The ( h K , Q K , λ K ) (h_{K},Q_{K},\lambda_{K}) ( h K , Q K , λ K ) -Gibbs probability measure P Y ∣ X = x K ( h K , Q K , λ K ) P^{(h_{K},Q_{K},\lambda_{K})}_{Y\mid X=x_{K}} P Y ∣ X = x K ( h K , Q K , λ K ) in (50) satisfies for all y ∈ supp Q 1 y\in\operatorname{supp}Q_{1} y ∈ supp Q 1 ,
d P Y ∣ X = x K ( h K , Q K , λ K ) d Q 1 ( y ) = exp ( − ∑ i = 1 K 1 λ i h i ( x i , y ) ) ∫ exp ( − ∑ i = 1 K 1 λ i h i ( x i , y ~ ) ) d Q 1 ( y ~ ) (52) \frac{\mathrm{d}P^{(h_{K},Q_{K},\lambda_{K})}_{Y\mid X=x_{K}}}{\mathrm{d}Q_{1}}(y)=\frac{\exp\left(-\sum_{i=1}^{K}\frac{1}{\lambda_{i}}h_{i}(x_{i},y)\right)}{\int\exp\left(-\sum_{i=1}^{K}\frac{1}{\lambda_{i}}h_{i}(x_{i},\widetilde{y})\right)\mathrm{d}Q_{1}(\widetilde{y})} \tag*{(52)} d Q 1 d P Y ∣ X = x K ( h K , Q K , λ K ) ( y ) = ∫ exp ( − ∑ i = 1 K λ i 1 h i ( x i , y ) ) d Q 1 ( y ) exp ( − ∑ i = 1 K λ i 1 h i ( x i , y ) ) (52) = d P Y ∣ X = x ( h ^ β , Q 1 , 1 ) d Q 1 ( y ) , (53) =\frac{\mathrm{d}P^{(\widehat{h}_{\boldsymbol{\beta}},Q_{1},1)}_{Y\mid X=x}}{\mathrm{d}Q_{1}}(y), \tag*{(53)} = d Q 1 d P Y ∣ X = x ( h β , Q 1 , 1 ) ( y ) , (53)
where x = ( x 1 , x 2 , … , x K ) \boldsymbol{x}=(x_1,x_2,\ldots,x_K) x = ( x 1 , x 2 , … , x K ) ; β = ( 1 λ 1 , 1 λ 2 , … , 1 λ K ) \boldsymbol{\beta}=\left(\frac{1}{\lambda_1},\frac{1}{\lambda_2},\ldots,\frac{1}{\lambda_K}\right) β = ( λ 1 1 , λ 2 1 , … , λ K 1 ) ; the function h ^ β \widehat{h}_{\boldsymbol{\beta}} h β is defined in (35) ; and the measure P Y ∣ X = x ( h ^ β , Q 1 , 1 ) P_{Y\mid X=\boldsymbol{x}}^{(\widehat{h}_{\boldsymbol{\beta}},Q_1,1)} P Y ∣ X = x ( h β , Q 1 , 1 ) is an ( h ^ β , Q 1 , 1 ) (\widehat{h}_{\boldsymbol{\beta}},Q_1,1) ( h β , Q 1 , 1 ) -Gibbs probability measure in △ ( Y ) \triangle(\mathcal{Y}) △ ( Y ) .
Proof: The proof is presented in Appendix F.
Theorem 11 shows that the nested Gibbs probability measure P Y ∣ X = x K ( h K , Q K , λ K ) P_{Y\mid X=x_K}^{(h_K,Q_K,\lambda_K)} P Y ∣ X = x K ( h K , Q K , λ K ) in (50) is identical to the ( h ^ β , Q 1 , 1 ) (\widehat{h}_{\boldsymbol{\beta}},Q_1,1) ( h β , Q 1 , 1 ) -Gibbs probability measure P Y ∣ X = x ( h ^ β , Q 1 , 1 ) P_{Y\mid X=\boldsymbol{x}}^{(\widehat{h}_{\boldsymbol{\beta}},Q_1,1)} P Y ∣ X = x ( h β , Q 1 , 1 ) in (53) . Although the reference measures in (51) are defined recursively, the Radon–Nikodym derivative in (52) depends on the functions h 1 , h 2 , … , h K h_1,h_2,\ldots,h_K h 1 , h 2 , … , h K only through a specific linear combination, i.e., h ^ β ( x , y ) = ∑ i = 1 K 1 λ i h i ( x i , y ) \widehat{h}_{\boldsymbol{\beta}}(\boldsymbol{x},y)=\sum_{i=1}^{K}\frac{1}{\lambda_i}h_i(x_i,y) h β ( x , y ) = ∑ i = 1 K λ i 1 h i ( x i , y ) . The following theorem further characterizes these observations through equivalent formulations involving the Radon–Nikodym derivative, the relative entropy, and some optimization problems involving relative entropies.
Theorem 12. Given the measure P Y ∣ X = x K ( h K , Q K , λ K ) P_{Y\mid X=x_K}^{(h_K,Q_K,\lambda_K)} P Y ∣ X = x K ( h K , Q K , λ K ) in (50) and the measure P Y ∣ X = x ( h ^ β , Q 1 , 1 ) P_{Y\mid X=\boldsymbol{x}}^{(\widehat{h}_{\boldsymbol{\beta}},Q_1,1)} P Y ∣ X = x ( h β , Q 1 , 1 ) in (53) , the following statements are equivalent:
(i) For all y ∈ supp Q 1 y\in\operatorname{supp}Q_1 y ∈ supp Q 1 ,
d P Y ∣ X = x K ( h K , Q K , λ K ) d P Y ∣ X = x ( h ^ β , Q 1 , 1 ) ( y ) = 1 ; (54) \frac{\mathrm{d}P_{Y\mid X=x_K}^{(h_K,Q_K,\lambda_K)}}{\mathrm{d}P_{Y\mid X=\boldsymbol{x}}^{(\widehat{h}_{\boldsymbol{\beta}},Q_1,1)}}(y)=1; \tag*{(54)} d P Y ∣ X = x ( h β , Q 1 , 1 ) d P Y ∣ X = x K ( h K , Q K , λ K ) ( y ) = 1 ; (54) (ii)
D ( P Y ∣ X = x K ( h K , Q K , λ K ) ∥ P Y ∣ X = x ( h ^ β , Q 1 , 1 ) ) = 0 ; (55) D\left(P_{Y\mid X=x_K}^{(h_K,Q_K,\lambda_K)}\middle\|P_{Y\mid X=\boldsymbol{x}}^{(\widehat{h}_{\boldsymbol{\beta}},Q_1,1)}\right)=0; \tag*{(55)} D ( P Y ∣ X = x K ( h K , Q K , λ K ) P Y ∣ X = x ( h β , Q 1 , 1 ) ) = 0 ; (55) (iii) The probability measure P Y ∣ X = x K ( h K , Q K , λ K ) P_{Y\mid X=x_K}^{(h_K,Q_K,\lambda_K)} P Y ∣ X = x K ( h K , Q K , λ K ) is the unique solution, if it exists, to the following optimization problem
min P ∈ △ Q 1 ( Y ) ∫ ( ∑ i = 1 K 1 λ i h i ( x i , y ) ) d P ( y ) + D ( P ∥ Q 1 ) ; (56) \min_{P\in\triangle_{Q_1}(\mathcal{Y})}\int\left(\sum_{i=1}^{K}\frac{1}{\lambda_i}h_i(x_i,y)\right)\mathrm{d}P(y)+D(P\mathbin\Vert Q_1); \tag*{(56)} P ∈ △ Q 1 ( Y ) min ∫ ( i = 1 ∑ K λ i 1 h i ( x i , y ) ) d P ( y ) + D ( P ∥ Q 1 ) ; (56) (iv) The measure P Y ∣ X = x K ( h K , Q K , λ K ) P_{Y\mid X=x_K}^{(h_K,Q_K,\lambda_K)} P Y ∣ X = x K ( h K , Q K , λ K ) is the unique solution, if it exists, to the following optimization problem
min P ∈ △ Q 1 ( Y ) ∫ ( ∑ i = 1 K 1 λ i h i ( x i , y ) ) d P ( y ) s.t. D ( P ∥ Q 1 ) ≤ D ( P Y ∣ X = x ( h ^ β , Q 1 , 1 ) ∥ Q 1 ) . (57) \begin{aligned}
\min_{P\in\triangle_{Q_1}(\mathcal{Y})}\quad&\int\left(\sum_{i=1}^{K}\frac{1}{\lambda_i}h_i(x_i,y)\right)\mathrm{d}P(y)\\
\text{s.t.}\quad&D(P\mathbin\Vert Q_1)\leq D\left(P_{Y\mid X=\boldsymbol{x}}^{(\widehat{h}_{\boldsymbol{\beta}},Q_1,1)}\middle\|Q_1\right). \tag*{(57)}
\end{aligned} P ∈ △ Q 1 ( Y ) min s.t. ∫ ( i = 1 ∑ K λ i 1 h i ( x i , y ) ) d P ( y ) D ( P ∥ Q 1 ) ≤ D ( P Y ∣ X = x ( h β , Q 1 , 1 ) Q 1 ) . (57) Proof: The proof is presented in Appendix G.
A key observation from Theorem 12 is that the optimization problems in (56) and (57) coincide with the optimization problems in (41) and (42) , respectively, when α 1 = α 2 = … = α K = 1 \alpha_1=\alpha_2=\ldots=\alpha_K=1 α 1 = α 2 = … = α K = 1 and the measures Q Q Q and Q 1 Q_1 Q 1 are identical. Therefore, an equivalence relation between nesting Gibbs probability measures and normalized log-linear combinations can be established as follows.
Corollary 13. Consider K K K Gibbs probability measures denoted by P Y ∣ X = x 1 ( h 1 , Q ^ 1 , λ 1 ) P_{Y\mid X=x_1}^{(h_1,\widehat{Q}_1,\lambda_1)} P Y ∣ X = x 1 ( h 1 , Q 1 , λ 1 ) , P Y ∣ X = x 2 ( h 2 , Q ^ 2 , λ 2 ) P_{Y\mid X=x_2}^{(h_2,\widehat{Q}_2,\lambda_2)} P Y ∣ X = x 2 ( h 2 , Q 2 , λ 2 ) , … \ldots … , P Y ∣ X = x K ( h K , Q ^ K , λ K ) P_{Y\mid X=x_K}^{(h_K,\widehat{Q}_K,\lambda_K)} P Y ∣ X = x K ( h K , Q K , λ K ) , where for all i ∈ { 1 , 2 , … , K } i\in\{1,2,\ldots,K\} i ∈ { 1 , 2 , … , K } , the function h i h_i h i , the nonzero real λ i \lambda_i λ i , and the vector x i x_i x i are the same as in (50); and the σ \sigma σ -finite measures Q ^ 1 , Q ^ 2 , … , Q ^ K \widehat{Q}_1,\widehat{Q}_2,\ldots,\widehat{Q}_K Q 1 , Q 2 , … , Q K are arbitrary measures, with Q ^ 1 \widehat{Q}_1 Q 1 identical to Q 1 Q_1 Q 1 in (50). Consider a normalized log-linear combination of such K K K Gibbs probability measures, denoted by T ∈ △ Q 1 ( Y ) T\in\triangle_{Q_1}(\mathcal{Y}) T ∈ △ Q 1 ( Y ) , such that for all y ∈ supp Q 1 y\in\operatorname{supp}Q_1 y ∈ supp Q 1 ,
d T d Q 1 ( y ) = ∏ i = 1 K d P Y ∣ X = x i ( h i , Q ^ i , λ i ) d Q ^ i ( y ) ∫ ∏ i = 1 K d P Y ∣ X = x i ( h i , Q ^ i , λ i ) d Q ^ i ( y ~ ) d Q 1 ( y ~ ) . (58) \frac{\mathrm{d}T}{\mathrm{d}Q_1}(y)=\frac{\prod_{i=1}^{K}\frac{\mathrm{d}P_{Y\mid X=x_i}^{(h_i,\widehat{Q}_i,\lambda_i)}}{\mathrm{d}\widehat{Q}_i}(y)}{\int\prod_{i=1}^{K}\frac{\mathrm{d}P_{Y\mid X=x_i}^{(h_i,\widehat{Q}_i,\lambda_i)}}{\mathrm{d}\widehat{Q}_i}(\widetilde{y})\,\mathrm{d}Q_1(\widetilde{y})}. \tag*{(58)} d Q 1 d T ( y ) = ∫ ∏ i = 1 K d Q i d P Y ∣ X = x i ( h i , Q i , λ i ) ( y ) d Q 1 ( y ) ∏ i = 1 K d Q i d P Y ∣ X = x i ( h i , Q i , λ i ) ( y ) . (58) Consider also the nested Gibbs probability measure P Y ∣ X = x K ( h K , Q K , λ K ) P_{Y\mid X=x_K}^{(h_K,Q_K,\lambda_K)} P Y ∣ X = x K ( h K , Q K , λ K ) in (50). Then,
D ( T ∥ P Y ∣ X = x K ( h K , Q K , λ K ) ) = 0. (59) D\left(T\middle\|P_{Y\mid X=x_K}^{(h_K,Q_K,\lambda_K)}\right)=0. \tag*{(59)} D ( T P Y ∣ X = x K ( h K , Q K , λ K ) ) = 0. (59) An interesting observation from Corollary 13 is that, as long as the measures Q ^ 1 \widehat{Q}_1 Q 1 and Q 1 Q_1 Q 1 are identical, the measures Q ^ 2 , Q ^ 3 , … , Q ^ K \widehat{Q}_2,\widehat{Q}_3,\ldots,\widehat{Q}_K Q 2 , Q 3 , … , Q K do not influence the normalized log-linear combination T T T in (58) , provided that they satisfy the conditions in Definition 5.
Geometric Representation
This section provides geometric representations of the renormalized, log-linearly combined, and nested Gibbs probability measures studied in Section III, under the assumption that the reference measures are probability measures. These representations use the connection established by Lemma 2 between the Gibbs probability measure in (6) and the optimization problems in (9) and (10). In particular, variations of some of the parameters of renormalizations and normalized log-linear combinations of Gibbs measures are associated with information geometric objects, e.g., exponential geodesics1 , which admit special Pythagorean interpretations.
Consider the ( h k , Q k , λ k ) (h_k,Q_k,\lambda_k) ( h k , Q k , λ k ) -Gibbs probability measures P Y ∣ X = x k ( h k , Q k , λ k ) P^{(h_k,Q_k,\lambda_k)}_{Y|X=x_k} P Y ∣ X = x k ( h k , Q k , λ k ) in (6), with k ∈ { 1 , 2 , … , K } k \in\{1,2,\ldots,K\} k ∈ { 1 , 2 , … , K } . For all k ∈ { 1 , 2 , … , K } k \in\{1,2,\ldots,K\} k ∈ { 1 , 2 , … , K } , let
0 ≤ γ k ≜ D ( P Y ∣ X = x k ( h k , Q k , λ k ) ∥ Q k ) < + ∞ . (60) 0 \le\gamma_k \triangleq D\left(P^{(h_k,Q_k,\lambda_k)}_{Y|X=x_k}\middle\| Q_k\right)<+\infty. \tag*{(60)} 0 ≤ γ k ≜ D ( P Y ∣ X = x k ( h k , Q k , λ k ) Q k ) < + ∞. (60)
The reference measure Q k Q_k Q k and γ k \gamma_k γ k in (59) determine the following subset of △ Q k ( Y ) \triangle_{Q_k}(\mathcal{Y}) △ Q k ( Y ) :
G Q k , γ k ≜ { P ∈ △ Q k ( Y ) : D ( P ∥ Q k ) ≤ γ k } . (61) \mathcal{G}_{Q_k,\gamma_k} \triangleq\left\{P \in\triangle_{Q_k}(\mathcal{Y}):D(P\parallel Q_k)\le\gamma_k\right\}. \tag*{(61)} G Q k , γ k ≜ { P ∈ △ Q k ( Y ) : D ( P ∥ Q k ) ≤ γ k } . (61)
In general, throughout this section, the notation G Q , γ \mathcal{G}_{Q,\gamma} G Q , γ denotes a set of the form in (60), where the relative entropy is with respect to the reference measure Q Q Q and the upper bound on the relative entropy is γ \gamma γ . The set G Q k , γ k \mathcal{G}_{Q_k,\gamma_k} G Q k , γ k in (60) can be interpreted as a relative-entropy neighborhood around the reference measure Q k Q_k Q k . Hence, from Lemma 2, the measure P Y ∣ X = x k ( h k , Q k , λ k ) P^{(h_k,Q_k,\lambda_k)}_{Y|X=x_k} P Y ∣ X = x k ( h k , Q k , λ k ) is the unique minimizer when λ k > 0 \lambda_k>0 λ k > 0 , or the unique maximizer when λ k < 0 \lambda_k<0 λ k < 0 , of the expectation ∫ h k ( x k , y ) d P ( y ) \int h_k(x_k,y)\,dP(y) ∫ h k ( x k , y ) d P ( y ) , when P ∈ G Q k , γ k P\in\mathcal{G}_{Q_k,\gamma_k} P ∈ G Q k , γ k . This observation plays a central role in the remainder of this section.
Renormalized Gibbs Probability Measures
For a fixed x ∈ X x\in\mathcal{X} x ∈ X , consider the Gibbs probability measure P Y ∣ X = x ( h , Q , λ ) P^{(h,Q,\lambda)}_{Y|X=x} P Y ∣ X = x ( h , Q , λ ) in (6) and its ( α , Q , Q 1 ) (\alpha,Q,Q_1) ( α , Q , Q 1 ) -renormalization R 1 R_1 R 1 in (16). Under the assumption that Q Q Q and Q 1 Q_1 Q 1 are probability measures, consider the reals
γ ≜ D ( P Y ∣ X = x ( h , Q , λ ) ∥ Q ) ; (62) \gamma\triangleq D\left(P^{(h,Q,\lambda)}_{Y|X=x}\middle\| Q\right); \tag*{(62)} γ ≜ D ( P Y ∣ X = x ( h , Q , λ ) Q ) ; (62)
η 1 ≜ D ( R 1 ∥ Q 1 ) . (63) \eta_1 \triangleq D(R_1\parallel Q_1). \tag*{(63)} η 1 ≜ D ( R 1 ∥ Q 1 ) . (63)
The measures P Y ∣ X = x ( h , Q , λ ) P^{(h,Q,\lambda)}_{Y|X=x} P Y ∣ X = x ( h , Q , λ ) and R 1 R_1 R 1 are such that P Y ∣ X = x ( h , Q , λ ) ∈ G Q , γ P^{(h,Q,\lambda)}_{Y|X=x}\in\mathcal{G}_{Q,\gamma} P Y ∣ X = x ( h , Q , λ ) ∈ G Q , γ and R 1 ∈ G Q 1 , η 1 R_1\in\mathcal{G}_{Q_1,\eta_1} R 1 ∈ G Q 1 , η 1 , with G Q , γ \mathcal{G}_{Q,\gamma} G Q , γ and G Q 1 , η 1 \mathcal{G}_{Q_1,\eta_1} G Q 1 , η 1 being both sets of the form in (60). From Theorem 4, the measure R 1 R_1 R 1 is identical to the measure P Y ∣ X = x ( h , Q 1 , λ α ) P^{(h,Q_1,\frac{\lambda}{\alpha})}_{Y|X=x} P Y ∣ X = x ( h , Q 1 , α λ ) in (17). Hence, Lemma 2 implies that P Y ∣ X = x ( h , Q , λ ) P^{(h,Q,\lambda)}_{Y|X=x} P Y ∣ X = x ( h , Q , λ ) and R 1 R_1 R 1 optimize the expectation ∫ h ( x , y ) d P ( y ) \int h(x,y)dP(y) ∫ h ( x , y ) d P ( y ) , when P P P is searched within G Q , γ \mathcal{G}_{Q,\gamma} G Q , γ and G Q 1 , η 1 \mathcal{G}_{Q_1,\eta_1} G Q 1 , η 1 , respectively. The former is a minimizer when λ > 0 \lambda>0 λ > 0 and a maximizer when λ < 0 \lambda<0 λ < 0 . The latter is a minimizer when λ α > 0 \frac{\lambda}{\alpha}>0 α λ > 0 and a maximizer when λ α < 0 \frac{\lambda}{\alpha}<0 α λ < 0 . Thus, renormalization preserves the expectation being optimized, while replacing the reference measure Q Q Q by Q 1 Q_1 Q 1 and the Gibbs parameter λ \lambda λ by λ α \frac{\lambda}{\alpha} α λ . Figure 1 provides a geometric representation of this operation.
Figure 1. Geometric representation of the ( α , Q , Q 1 ) (\alpha,Q,Q_1) ( α , Q , Q 1 ) -renormalization of P Y ∣ X = x ( h , Q , λ ) P^{(h,Q,\lambda)}_{Y|X=x} P Y ∣ X = x ( h , Q , λ ) in (6). The original Gibbs probability measure P Y ∣ X = x ( h , Q , λ ) P^{(h,Q,\lambda)}_{Y|X=x} P Y ∣ X = x ( h , Q , λ ) lies on the boundary of G Q , γ \mathcal{G}_{Q,\gamma} G Q , γ , whereas the renormalized probability measure P Y ∣ X = x ( h , Q 1 , λ α ) P^{(h,Q_1,\frac{\lambda}{\alpha})}_{Y|X=x} P Y ∣ X = x ( h , Q 1 , α λ ) in (17) lies on the boundary of G Q 1 , η 1 \mathcal{G}_{Q_1,\eta_1} G Q 1 , η 1 . The reals γ \gamma γ and η 1 \eta_1 η 1 are defined in (61) and (62), respectively. The purple arrow represents the mapping from P Y ∣ X = x ( h , Q , λ ) P^{(h,Q,\lambda)}_{Y|X=x} P Y ∣ X = x ( h , Q , λ ) to P Y ∣ X = x ( h , Q 1 , λ α ) P^{(h,Q_1,\frac{\lambda}{\alpha})}_{Y|X=x} P Y ∣ X = x ( h , Q 1 , α λ ) , which is induced by the renormalization operation.
A deeper insight is obtained by fixing h h h , x x x , λ \lambda λ , Q Q Q , and Q 1 Q_1 Q 1 and letting α \alpha α vary. From (16) and Theorem 4, the Radon–Nikodym derivative of the renormalized probability measure R 1 R_1 R 1 in (16), or similarly P Y ∣ X = x ( h , Q 1 , λ α ) P^{(h,Q_1,\frac{\lambda}{\alpha})}_{Y|X=x} P Y ∣ X = x ( h , Q 1 , α λ ) in (17), with respect to Q 1 Q_1 Q 1 , satisfies for all y ∈ supp Q 1 y\in\operatorname{supp}Q_1 y ∈ supp Q 1 ,
log d P Y ∣ X = x ( h , Q 1 , λ α ) d Q 1 ( y ) = α log ( d P Y ∣ X = x ( h , Q , λ ) d Q ( y ) ) − log ( ∫ ( d P Y ∣ X = x ( h , Q , λ ) d Q ( y ~ ) ) α d Q 1 ( y ~ ) ) . (64) \log\frac{dP^{(h,Q_1,\frac{\lambda}{\alpha})}_{Y|X=x}}{dQ_1}(y)=\alpha\log\left(\frac{dP^{(h,Q,\lambda)}_{Y|X=x}}{dQ}(y)\right)-\log\left(\int\left(\frac{dP^{(h,Q,\lambda)}_{Y|X=x}}{dQ}(\widetilde{y})\right)^\alpha dQ_1(\widetilde{y})\right). \tag*{(64)} log d Q 1 d P Y ∣ X = x ( h , Q 1 , α λ ) ( y ) = α log d Q d P Y ∣ X = x ( h , Q , λ ) ( y ) − log ∫ d Q d P Y ∣ X = x ( h , Q , λ ) ( y ) α d Q 1 ( y ) . (64)
The expression in (63) implies that, as α \alpha α varies, the Gibbs probability measures P Y ∣ X = x ( h , Q 1 , λ α ) P_{Y\mid X=x}^{(h,Q_1,\frac{\lambda}{\alpha})} P Y ∣ X = x ( h , Q 1 , α λ ) in (17) describe an exponential geodesic generated by the Radon–Nikodym derivative d P Y ∣ X = x ( h , Q , λ ) d Q \frac{dP_{Y\mid X=x}^{(h,Q,\lambda)}}{dQ} d Q d P Y ∣ X = x ( h , Q , λ ) in (6), up to a normalization term with respect to Q 1 Q_1 Q 1 ; see [58 ] . In particular, from [15 Lemma 5] , it follows that this family of probability measures admits a continuous extension to α = 0 \alpha=0 α = 0 , which corresponds to the measure Q 1 Q_1 Q 1 . At α = 1 \alpha=1 α = 1 , the corresponding measure is the ( h , Q 1 , λ ) (h,Q_1,\lambda) ( h , Q 1 , λ ) -Gibbs probability measure P Y ∣ X = x ( h , Q 1 , λ ) P_{Y\mid X=x}^{(h,Q_1,\lambda)} P Y ∣ X = x ( h , Q 1 , λ ) . For any other value of α \alpha α , the corresponding measure represents the ( α , Q , Q 1 ) (\alpha,Q,Q_1) ( α , Q , Q 1 ) -renormalization of the Gibbs probability measure P Y ∣ X = x ( h , Q , λ ) P_{Y\mid X=x}^{(h,Q,\lambda)} P Y ∣ X = x ( h , Q , λ ) , i.e., the measure P Y ∣ X = x ( h , Q 1 , λ α ) P_{Y\mid X=x}^{(h,Q_1,\frac{\lambda}{\alpha})} P Y ∣ X = x ( h , Q 1 , α λ ) in (17). Using these observations, such a renormalization is shown to exhibit different interpretations depending on whether Q 1 = Q Q_1=Q Q 1 = Q or Q 1 ≠ Q Q_1\ne Q Q 1 = Q . Both cases are separately studied.
First, when Q 1 = Q Q_1=Q Q 1 = Q , the probability measure P Y ∣ X = x ( h , Q 1 , λ α ) P_{Y\mid X=x}^{(h,Q_1,\frac{\lambda}{\alpha})} P Y ∣ X = x ( h , Q 1 , α λ ) in (17) at α = 1 \alpha=1 α = 1 coincides with the Gibbs probability measure P Y ∣ X = x ( h , Q , λ ) P_{Y\mid X=x}^{(h,Q,\lambda)} P Y ∣ X = x ( h , Q , λ ) in (6). Hence, as α \alpha α varies the renormalized measure P Y ∣ X = x ( h , Q 1 , λ α ) P_{Y\mid X=x}^{(h,Q_1,\frac{\lambda}{\alpha})} P Y ∣ X = x ( h , Q 1 , α λ ) in (17) traces out the exponential geodesic described above passing through Q 1 Q_1 Q 1 at α = 0 \alpha=0 α = 0 and P Y ∣ X = x ( h , Q 1 , λ ) P_{Y\mid X=x}^{(h,Q_1,\lambda)} P Y ∣ X = x ( h , Q 1 , λ ) at α = 1 \alpha=1 α = 1 . Note also that the inclusion between G Q , γ G_{Q,\gamma} G Q , γ and G Q , η 1 G_{Q,\eta_1} G Q , η 1 is determined by the ordering of γ \gamma γ and η 1 \eta_1 η 1 . For α > 0 \alpha>0 α > 0 , the parameters λ \lambda λ and λ α \frac{\lambda}{\alpha} α λ have the same sign. From Lemma 3, the relative entropy is nonincreasing with respect to the Gibbs parameter on positive intervals and nondecreasing on negative intervals. Consequently, for either sign of λ \lambda λ ,
G Q , γ ⊆ G Q , η 1 , if α > 1 ; (65) G_{Q,\gamma}\subseteq G_{Q,\eta_1},\qquad\text{if }\alpha>1; \tag*{(65)} G Q , γ ⊆ G Q , η 1 , if α > 1 ; (65)
G Q , η 1 ⊆ G Q , γ , if 0 < α < 1. (66) G_{Q,\eta_1}\subseteq G_{Q,\gamma},\qquad\text{if }0<\alpha<1. \tag*{(66)} G Q , η 1 ⊆ G Q , γ , if 0 < α < 1. (66)
At α = 1 \alpha=1 α = 1 , the two measures coincide, and hence G Q , γ = G Q , η 1 G_{Q,\gamma}=G_{Q,\eta_1} G Q , γ = G Q , η 1 . For α < 0 \alpha<0 α < 0 , the parameters λ \lambda λ and λ α \frac{\lambda}{\alpha} α λ have opposite signs. The monotonicity properties in Lemma 3 apply separately on positive and negative intervals and therefore do not provide an ordering of γ \gamma γ and η 1 \eta_1 η 1 in this case. The neighborhoods remain ordered by inclusion, but the direction of inclusion depends on which of these relative entropies, γ \gamma γ and η 1 \eta_1 η 1 , is larger.
Figure 2 provides a geometric representation of the two possible orderings of γ \gamma γ and η 1 \eta_1 η 1 .
Figure 2. Geometric representation of the ( α , Q , Q ) (\alpha,Q,Q) ( α , Q , Q ) -renormalization of P Y ∣ X = x ( h , Q , λ ) P_{Y\mid X=x}^{(h,Q,\lambda)} P Y ∣ X = x ( h , Q , λ ) in (6). The solid purple curve represents the exponential geodesic generated by the measure P Y ∣ X = x ( h , Q 1 , λ α ) P_{Y\mid X=x}^{(h,Q_1,\frac{\lambda}{\alpha})} P Y ∣ X = x ( h , Q 1 , α λ ) in (17) as α \alpha α varies. The panels show (a) γ < η 1 \gamma<\eta_1 γ < η 1 , with G Q , γ ⊆ G Q , η 1 G_{Q,\gamma}\subseteq G_{Q,\eta_1} G Q , γ ⊆ G Q , η 1 , and (b) η 1 < γ \eta_1<\gamma η 1 < γ , with G Q , η 1 ⊆ G Q , γ G_{Q,\eta_1}\subseteq G_{Q,\gamma} G Q , η 1 ⊆ G Q , γ . The sets G Q , γ G_{Q,\gamma} G Q , γ and G Q , η 1 G_{Q,\eta_1} G Q , η 1 are of the form in (60), with γ \gamma γ and η 1 \eta_1 η 1 in (61) and (62), respectively.
On the other hand, when Q 1 ≠ Q Q_1\ne Q Q 1 = Q , the intersection between the sets G Q , γ G_{Q,\gamma} G Q , γ and G Q 1 , η 1 G_{Q_1,\eta_1} G Q 1 , η 1 might be empty, as shown in Figure 1 . Nonetheless, the same interpretation of the exponential geodesic described above holds. As α \alpha α varies, the renormalized probability measure P Y ∣ X = x ( h , Q 1 , λ α ) P_{Y\mid X=x}^{(h,Q_1,\frac{\lambda}{\alpha})} P Y ∣ X = x ( h , Q 1 , α λ ) in (17) traces out this geodesic, passing through Q 1 Q_1 Q 1 at α = 0 \alpha=0 α = 0 , by continuous extension, and P Y ∣ X = x ( h , Q 1 , λ ) P_{Y\mid X=x}^{(h,Q_1,\lambda)} P Y ∣ X = x ( h , Q 1 , λ ) at α = 1 \alpha=1 α = 1 , as shown in Figure 3 , where η 2 \eta_2 η 2 is defined as follows:
Figure 3. Geometric representation of the ( α , Q , Q 1 ) (\alpha,Q,Q_1) ( α , Q , Q 1 ) -renormalization of P Y ∣ X = x ( h , Q , λ ) P^{(h,Q,\lambda)}_{Y\mid X=x} P Y ∣ X = x ( h , Q , λ ) in (6) , with Q 1 ≠ Q Q_1\ne Q Q 1 = Q . The solid purple curve represents the exponential geodesic generated by the measure P Y ∣ X = x ( h , Q 1 , α / λ ) P^{(h,Q_1,\alpha/\lambda)}_{Y\mid X=x} P Y ∣ X = x ( h , Q 1 , α / λ ) in (17) when α \alpha α varies. The panels show (a) η 1 < η 2 \eta_1<\eta_2 η 1 < η 2 , with G Q 1 , η 1 ⊆ G Q 1 , η 2 G_{Q_1,\eta_1}\subseteq G_{Q_1,\eta_2} G Q 1 , η 1 ⊆ G Q 1 , η 2 , and (b) η 1 > η 2 \eta_1>\eta_2 η 1 > η 2 , with G Q 1 , η 2 ⊆ G Q 1 , η 1 G_{Q_1,\eta_2}\subseteq G_{Q_1,\eta_1} G Q 1 , η 2 ⊆ G Q 1 , η 1 . The sets G Q , γ G_{Q,\gamma} G Q , γ , G Q 1 , η 1 G_{Q_1,\eta_1} G Q 1 , η 1 , and G Q 1 , η 2 G_{Q_1,\eta_2} G Q 1 , η 2 are of the form in (61) , with γ \gamma γ , η 1 \eta_1 η 1 , and η 2 \eta_2 η 2 defined in (62) , (63) , and (67) , respectively.
η 2 ≜ D ( P Y ∣ X = x ( h , Q 1 , λ ) ∥ Q 1 ) . (67) \eta_2 \triangleq D\left(P_{Y\mid X=x}^{(h,Q_1,\lambda)}\middle\|Q_1\right). \tag*{(67)} η 2 ≜ D ( P Y ∣ X = x ( h , Q 1 , λ ) Q 1 ) . (67)
Interestingly, the Gibbs probability measure P Y ∣ X = x ( h , Q , λ ) P_{Y\mid X=x}^{(h,Q,\lambda)} P Y ∣ X = x ( h , Q , λ ) in (6), as well as the reference measure Q Q Q , do not necessarily belong to such an exponential geodesic. Finally, the inclusion relations between the sets G Q 1 , η 1 G_{Q_1,\eta_1} G Q 1 , η 1 and G Q 1 , η 2 G_{Q_1,\eta_2} G Q 1 , η 2 in (60) follow from the same argument as in the case Q 1 = Q Q_1=Q Q 1 = Q , with γ \gamma γ replaced by η 2 \eta_2 η 2 in (66). Figure 3 provides a geometric representation of the two possible orderings of η 1 \eta_1 η 1 and η 2 \eta_2 η 2 .
Normalized Log-Linear Combinations of Gibbs Probability Measures
For K = 2 K=2 K = 2 and a fixed x = ( x 1 , x 2 ) ∈ X 2 x=(x_1,x_2)\in\mathcal{X}^2 x = ( x 1 , x 2 ) ∈ X 2 , consider the Gibbs probability measures P Y ∣ X = x 1 ( h 1 , Q 1 , λ 1 ) P_{Y\mid X=x_1}^{(h_1,Q_1,\lambda_1)} P Y ∣ X = x 1 ( h 1 , Q 1 , λ 1 ) and P Y ∣ X = x 2 ( h 2 , Q 2 , λ 2 ) P_{Y\mid X=x_2}^{(h_2,Q_2,\lambda_2)} P Y ∣ X = x 2 ( h 2 , Q 2 , λ 2 ) in (6). These measures satisfy, with equality, the relative entropy constraints defining the sets G Q 1 , γ 1 G_{Q_1,\gamma_1} G Q 1 , γ 1 and G Q 2 , γ 2 G_{Q_2,\gamma_2} G Q 2 , γ 2 in (60), respectively, where
γ 1 \gamma_1 γ 1 and γ 2 \gamma_2 γ 2 are defined in (60) . The normalized log-linear combination of the densities of these measures with respect to Q Q Q and coefficients ( α 1 , α 2 ) ∈ R 2 (\alpha_1,\alpha_2)\in\mathbb{R}^2 ( α 1 , α 2 ) ∈ R 2 , coincides, from Theorem 7 , with the Gibbs probability measure P Y ∣ X = x ( h ^ β , Q , 1 ) P^{(\widehat{h}_{\boldsymbol{\beta}},Q,1)}_{Y\mid X=x} P Y ∣ X = x ( h β , Q , 1 ) in (37) , where β = ( α 1 λ 1 , α 2 λ 2 ) \boldsymbol{\beta}=\left(\frac{\alpha_1}{\lambda_1},\frac{\alpha_2}{\lambda_2}\right) β = ( λ 1 α 1 , λ 2 α 2 ) and h ^ β \widehat{h}_{\boldsymbol{\beta}} h β is the linear combination in (35) .
For a fixed pair ( α 1 , α 2 ) ∈ ( R ∖ { 0 } ) 2 (\alpha_1,\alpha_2)\in(\mathbb{R}\setminus\{0\})^2 ( α 1 , α 2 ) ∈ ( R ∖ { 0 } ) 2 , and under the assumption that Q Q Q is a probability measure, consider the real
γ 0 ≜ D ( P Y ∣ X = x ( h ^ β , Q , 1 ) ∥ Q ) . (68) \gamma_0\triangleq D\left(P^{(\widehat{h}_{\boldsymbol{\beta}},Q,1)}_{Y\mid X=x}\mathbin{\|}Q\right). \tag*{(68)} γ 0 ≜ D ( P Y ∣ X = x ( h β , Q , 1 ) ∥ Q ) . (68)
From Theorem 8, it follows that the probability measure P Y ∣ X = x ( h ^ β , Q , 1 ) P^{(\widehat{h}_{\boldsymbol{\beta}},Q,1)}_{Y\mid X=x} P Y ∣ X = x ( h β , Q , 1 ) in (37) is the unique minimizer of the expectation ∫ h ^ β ( x , y ) d P ( y ) \int\widehat{h}_{\boldsymbol{\beta}}(x,y)\,\mathrm{d}P(y) ∫ h β ( x , y ) d P ( y ) when P P P is searched within G Q , γ 0 G_{Q,\gamma_0} G Q , γ 0 in (61) . Beyond this observation, the normalized log-linear combination P Y ∣ X = x ( h ^ β , Q , 1 ) P^{(\widehat{h}_{\boldsymbol{\beta}},Q,1)}_{Y\mid X=x} P Y ∣ X = x ( h β , Q , 1 ) also admits a geometric interpretation obtained by fixing h 1 h_1 h 1 , h 2 h_2 h 2 , x 1 x_1 x 1 , x 2 x_2 x 2 , λ 1 \lambda_1 λ 1 , λ 2 \lambda_2 λ 2 , Q 1 Q_1 Q 1 , Q 2 Q_2 Q 2 , and Q Q Q , and varying the coefficients α 1 \alpha_1 α 1 and α 2 \alpha_2 α 2 . From (37) , the Radon–Nikodym derivative of P Y ∣ X = x ( h ^ β , Q , 1 ) P^{(\widehat{h}_{\boldsymbol{\beta}},Q,1)}_{Y\mid X=x} P Y ∣ X = x ( h β , Q , 1 ) with respect to Q Q Q satisfies for all y ∈ supp Q y\in\operatorname{supp}Q y ∈ supp Q ,
log d P Y ∣ X = x ( h ^ β , Q , 1 ) d Q ( y ) = α 1 log ( d P Y ∣ X = x 1 ( h 1 , Q 1 , λ 1 ) d Q 1 ( y ) ) + α 2 log ( d P Y ∣ X = x 2 ( h 2 , Q 2 , λ 2 ) d Q 2 ( y ) ) − log ( ∫ ( d P Y ∣ X = x 1 ( h 1 , Q 1 , λ 1 ) d Q 1 ( y ~ ) ) α 1 ( d P Y ∣ X = x 2 ( h 2 , Q 2 , λ 2 ) d Q 2 ( y ~ ) ) α 2 d Q ( y ~ ) ) . (69) \log\frac{\mathrm{d}P^{(\widehat{h}_{\boldsymbol{\beta}},Q,1)}_{Y\mid X=x}}{\mathrm{d}Q}(y)=\alpha_1\log\left(\frac{\mathrm{d}P^{(h_1,Q_1,\lambda_1)}_{Y\mid X=x_1}}{\mathrm{d}Q_1}(y)\right)+\alpha_2\log\left(\frac{\mathrm{d}P^{(h_2,Q_2,\lambda_2)}_{Y\mid X=x_2}}{\mathrm{d}Q_2}(y)\right)-\log\left(\int\left(\frac{\mathrm{d}P^{(h_1,Q_1,\lambda_1)}_{Y\mid X=x_1}}{\mathrm{d}Q_1}(\widetilde{y})\right)^{\alpha_1}\left(\frac{\mathrm{d}P^{(h_2,Q_2,\lambda_2)}_{Y\mid X=x_2}}{\mathrm{d}Q_2}(\widetilde{y})\right)^{\alpha_2}\,\mathrm{d}Q(\widetilde{y})\right). \tag*{(69)} log d Q d P Y ∣ X = x ( h β , Q , 1 ) ( y ) = α 1 log d Q 1 d P Y ∣ X = x 1 ( h 1 , Q 1 , λ 1 ) ( y ) + α 2 log d Q 2 d P Y ∣ X = x 2 ( h 2 , Q 2 , λ 2 ) ( y ) − log ∫ d Q 1 d P Y ∣ X = x 1 ( h 1 , Q 1 , λ 1 ) ( y ) α 1 d Q 2 d P Y ∣ X = x 2 ( h 2 , Q 2 , λ 2 ) ( y ) α 2 d Q ( y ) . (69)
= − α 1 λ 1 h 1 ( x 1 , y ) − α 2 λ 2 h 2 ( x 2 , y ) − log ∫ exp ( − α 1 λ 1 h 1 ( x 1 , y ~ ) − α 2 λ 2 h 2 ( x 2 , y ~ ) ) d Q ( y ~ ) . (70) =\frac{-\alpha_1}{\lambda_1}h_1(x_1,y)-\frac{\alpha_2}{\lambda_2}h_2(x_2,y)-\log\int\exp\left(-\frac{\alpha_1}{\lambda_1}h_1(x_1,\widetilde{y})-\frac{\alpha_2}{\lambda_2}h_2(x_2,\widetilde{y})\right)\,\mathrm{d}Q(\widetilde{y}). \tag*{(70)} = λ 1 − α 1 h 1 ( x 1 , y ) − λ 2 α 2 h 2 ( x 2 , y ) − log ∫ exp ( − λ 1 α 1 h 1 ( x 1 , y ) − λ 2 α 2 h 2 ( x 2 , y ) ) d Q ( y ) . (70)
Hence, as α 1 \alpha_1 α 1 and α 2 \alpha_2 α 2 vary over values for which the normalization term in (69) is finite, the Gibbs probability measure P Y ∣ X = x ( h ^ β , Q , 1 ) P^{(\widehat{h}_{\boldsymbol{\beta}},Q,1)}_{Y\mid X=x} P Y ∣ X = x ( h β , Q , 1 ) in (37) forms an exponential family with natural parameter ( − α 1 λ 1 , − α 2 λ 2 ) ∈ R 2 \left(-\frac{\alpha_1}{\lambda_1},-\frac{\alpha_2}{\lambda_2}\right)\in\mathbb{R}^2 ( − λ 1 α 1 , − λ 2 α 2 ) ∈ R 2 . For the ease of presentation, consider α 1 = α \alpha_1=\alpha α 1 = α and α 2 = 1 − α \alpha_2=1-\alpha α 2 = 1 − α , for some α ∈ ( 0 , 1 ) \alpha\in(0,1) α ∈ ( 0 , 1 ) . The expression in (69) implies that, as α \alpha α varies, the Gibbs probability measures P Y ∣ X = x ( h ^ β , Q , 1 ) P^{(\widehat{h}_{\boldsymbol{\beta}},Q,1)}_{Y\mid X=x} P Y ∣ X = x ( h β , Q , 1 ) in (37) describe a segment of an exponential geodesic, up to a normalization with respect to Q Q Q . Note that at α = 0 \alpha=0 α = 0 , the probability measure P Y ∣ X = x ( h ^ β , Q , 1 ) P^{(\widehat{h}_{\boldsymbol{\beta}},Q,1)}_{Y\mid X=x} P Y ∣ X = x ( h β , Q , 1 ) in (37) is identical to the ( h 2 , Q , λ 2 ) (h_2,Q,\lambda_2) ( h 2 , Q , λ 2 ) -Gibbs probability measure P Y ∣ X = x 2 ( h 2 , Q , λ 2 ) P^{(h_2,Q,\lambda_2)}_{Y\mid X=x_2} P Y ∣ X = x 2 ( h 2 , Q , λ 2 ) . Alternatively, at α = 1 \alpha=1 α = 1 , such a measure is identical to the ( h 1 , Q , λ 1 ) (h_1,Q,\lambda_1) ( h 1 , Q , λ 1 ) -Gibbs probability measure P Y ∣ X = x 1 ( h 1 , Q , λ 1 ) P^{(h_1,Q,\lambda_1)}_{Y\mid X=x_1} P Y ∣ X = x 1 ( h 1 , Q , λ 1 ) . These endpoints are understood as continuous extensions of (69), taking into account that Definition 5 assumes that the coefficients, α 1 \alpha_1 α 1 and α 2 \alpha_2 α 2 , are nonzero.
When Q 1 = Q 2 = Q Q_1=Q_2=Q Q 1 = Q 2 = Q , these endpoints coincide with the original Gibbs probability measures P Y ∣ X = x 2 ( h 2 , Q 2 , λ 2 ) P^{(h_2,Q_2,\lambda_2)}_{Y\mid X=x_2} P Y ∣ X = x 2 ( h 2 , Q 2 , λ 2 ) and P Y ∣ X = x 1 ( h 1 , Q 1 , λ 1 ) P^{(h_1,Q_1,\lambda_1)}_{Y\mid X=x_1} P Y ∣ X = x 1 ( h 1 , Q 1 , λ 1 ) , respectively. Hence, the exponential geodesic obtained by varying α \alpha α over [ 0 , 1 ] [0,1] [ 0 , 1 ] connects the two original Gibbs probability measures. Figure 4 provides a geometric representation of this case.
Figure 4. Geometric representation of the normalized log-linear combination P Y ∣ X = x ( h β , Q , 1 ) P_{Y\mid X=\mathbf{x}}^{(h_{\boldsymbol{\beta},Q,1})} P Y ∣ X = x ( h β , Q , 1 ) in (37), with Q 1 = Q 2 = Q Q_1=Q_2=Q Q 1 = Q 2 = Q and β = ( α λ 1 , 1 − α λ 2 ) \boldsymbol{\beta}=\left(\frac{\alpha}{\lambda_1},\frac{1-\alpha}{\lambda_2}\right) β = ( λ 1 α , λ 2 1 − α ) , where 0 < α < 1 0<\alpha<1 0 < α < 1 . The solid purple curve represents the exponential segment obtained by varying α \alpha α over [ 0 , 1 ] [0,1] [ 0 , 1 ] . Its endpoints are P Y ∣ X = x 2 ( h 2 , Q , λ 2 ) P_{Y\mid X=x_2}^{(h_2,Q,\lambda_2)} P Y ∣ X = x 2 ( h 2 , Q , λ 2 ) at α = 0 \alpha=0 α = 0 and P Y ∣ X = x 1 ( h 1 , Q , λ 1 ) P_{Y\mid X=x_1}^{(h_1,Q,\lambda_1)} P Y ∣ X = x 1 ( h 1 , Q , λ 1 ) at α = 1 \alpha=1 α = 1 , and it passes through the normalized log-linear combination at the chosen value of α \alpha α . The neighborhoods G Q , γ 1 G_{Q,\gamma_1} G Q , γ 1 , G Q , γ 2 G_{Q,\gamma_2} G Q , γ 2 , and G Q , γ 0 G_{Q,\gamma_0} G Q , γ 0 are defined by (60), with γ 1 \gamma_1 γ 1 and γ 2 \gamma_2 γ 2 given in (59) and γ 0 \gamma_0 γ 0 given in (67). Under the illustrated ordering γ 1 < γ 2 \gamma_1<\gamma_2 γ 1 < γ 2 , the panels show (a) γ 0 < γ 1 < γ 2 \gamma_0<\gamma_1<\gamma_2 γ 0 < γ 1 < γ 2 , (b) γ 1 < γ 0 < γ 2 \gamma_1<\gamma_0<\gamma_2 γ 1 < γ 0 < γ 2 , and (c) γ 1 < γ 2 < γ 0 \gamma_1<\gamma_2<\gamma_0 γ 1 < γ 2 < γ 0 .
When Q 1 Q_1 Q 1 , Q 2 Q_2 Q 2 , and Q Q Q are different, the Gibbs probability measures P Y ∣ X = x 1 ( h 1 , Q 1 , λ 1 ) P^{(h_1,Q_1,\lambda_1)}_{Y\mid X=x_1} P Y ∣ X = x 1 ( h 1 , Q 1 , λ 1 ) and P Y ∣ X = x 2 ( h 2 , Q 2 , λ 2 ) P^{(h_2,Q_2,\lambda_2)}_{Y\mid X=x_2} P Y ∣ X = x 2 ( h 2 , Q 2 , λ 2 ) at the endpoints of the exponential geodesic do not, in general, coincide with the original Gibbs probability measures P Y ∣ X = x 1 ( h 1 , Q 1 , λ 1 ) P^{(h_1,Q_1,\lambda_1)}_{Y\mid X=x_1} P Y ∣ X = x 1 ( h 1 , Q 1 , λ 1 ) and P Y ∣ X = x 2 ( h 2 , Q 2 , λ 2 ) P^{(h_2,Q_2,\lambda_2)}_{Y\mid X=x_2} P Y ∣ X = x 2 ( h 2 , Q 2 , λ 2 ) , respectively. Consider the values
η 1 ≜ D ( P Y ∣ X = x 1 ( h 1 , Q 1 , λ 1 ) ∥ Q ) ; (71) \eta_1\triangleq D\left(P^{(h_1,Q_1,\lambda_1)}_{Y\mid X=x_1}\mathbin{\|}Q\right); \tag*{(71)} η 1 ≜ D ( P Y ∣ X = x 1 ( h 1 , Q 1 , λ 1 ) ∥ Q ) ; (71)
η 2 ≜ D ( P Y ∣ X = x 2 ( h 2 , Q , λ 2 ) ∥ Q ) . (72) \eta_2 \triangleq D\left(P_{Y\mid X=x_2}^{(h_2,Q,\lambda_2)}\middle\|Q\right). \tag*{(72)} η 2 ≜ D ( P Y ∣ X = x 2 ( h 2 , Q , λ 2 ) Q ) . (72)
The endpoint measures therefore satisfy, with equality, the relative entropy constraints defining G Q , η 1 G_{Q,\eta_1} G Q , η 1 and G Q , η 2 G_{Q,\eta_2} G Q , η 2 in (60), respectively. Hence, varying α \alpha α over [ 0 , 1 ] [0,1] [ 0 , 1 ] describes the exponential geodesic connecting P Y ∣ X = x 2 ( h 2 , Q , λ 2 ) P_{Y\mid X=x_2}^{(h_2,Q,\lambda_2)} P Y ∣ X = x 2 ( h 2 , Q , λ 2 ) (α = 0 \alpha=0 α = 0 ) and P Y ∣ X = x 1 ( h 1 , Q , λ 1 ) P_{Y\mid X=x_1}^{(h_1,Q,\lambda_1)} P Y ∣ X = x 1 ( h 1 , Q , λ 1 ) (α = 1 \alpha=1 α = 1 ). Figure 5 provides a geometric representation of this case.
Figure 5. Geometric representation of the normalized log-linear combination P Y ∣ X = x ( h β , Q , 1 ) P_{Y\mid X=\mathbf{x}}^{(h_{\boldsymbol{\beta},Q,1})} P Y ∣ X = x ( h β , Q , 1 ) in (37), with β = ( α / λ 1 , ( 1 − α ) / λ 2 ) \boldsymbol{\beta}=(\alpha/\lambda_1,(1-\alpha)/\lambda_2) β = ( α / λ 1 , ( 1 − α ) / λ 2 ) and 0 < α < 1 0<\alpha<1 0 < α < 1 . The solid purple curve represents the exponential geodesic obtained by varying α \alpha α over [ 0 , 1 ] [0,1] [ 0 , 1 ] . Its endpoints are P Y ∣ X = x 2 ( h 2 , Q , λ 2 ) P_{Y\mid X=x_2}^{(h_2,Q,\lambda_2)} P Y ∣ X = x 2 ( h 2 , Q , λ 2 ) at α = 0 \alpha=0 α = 0 and P Y ∣ X = x 1 ( h 1 , Q , λ 1 ) P_{Y\mid X=x_1}^{(h_1,Q,\lambda_1)} P Y ∣ X = x 1 ( h 1 , Q , λ 1 ) at α = 1 \alpha=1 α = 1 . The original Gibbs probability measures P Y ∣ X = x 1 ( h 1 , Q 1 , λ 1 ) P_{Y\mid X=x_1}^{(h_1,Q_1,\lambda_1)} P Y ∣ X = x 1 ( h 1 , Q 1 , λ 1 ) and P Y ∣ X = x 2 ( h 2 , Q 2 , λ 2 ) P_{Y\mid X=x_2}^{(h_2,Q_2,\lambda_2)} P Y ∣ X = x 2 ( h 2 , Q 2 , λ 2 ) are represented separately on the boundaries of G Q 1 , γ 1 G_{Q_1,\gamma_1} G Q 1 , γ 1 and G Q 2 , γ 2 G_{Q_2,\gamma_2} G Q 2 , γ 2 , respectively, and do not, in general, belong to the exponential geodesic. The sets G Q , η 1 G_{Q,\eta_1} G Q , η 1 , G Q , η 2 G_{Q,\eta_2} G Q , η 2 , and G Q , γ 0 G_{Q,\gamma_0} G Q , γ 0 contain the two endpoints and the normalized log-linear combination, respectively, with η 1 \eta_1 η 1 , η 2 \eta_2 η 2 , and γ 0 \gamma_0 γ 0 defined in (70), (71), and (67).
Nested Gibbs Probability Measures
Consider the recursively nested Gibbs probability measures P Y ∣ X = x k ( h k , Q k , λ k ) P_{Y\mid X=x_k}^{(h_k,Q_k,\lambda_k)} P Y ∣ X = x k ( h k , Q k , λ k ) in (50) with k ∈ { 1 , 2 , … , K } k\in\{1,2,\ldots,K\} k ∈ { 1 , 2 , … , K } and the recursive choice of the reference measures in (50b), namely, Q k + 1 = P Y ∣ X = x k ( h k , Q k , λ k ) Q_{k+1}=P_{Y\mid X=x_k}^{(h_k,Q_k,\lambda_k)} Q k + 1 = P Y ∣ X = x k ( h k , Q k , λ k ) for k ∈ { 1 , 2 , … , K − 1 } k\in\{1,2,\ldots,K-1\} k ∈ { 1 , 2 , … , K − 1 } . In this case, it suffices to assume that Q 1 Q_1 Q 1 is a probability measure. In addition to the values γ k \gamma_k γ k in (59), consider the real
η ≜ D ( P Y ∣ X = x K ( h K , Q K , λ K ) ∥ Q 1 ) , (73) \eta\triangleq D\left(P_{Y\mid X=x_K}^{(h_K,Q_K,\lambda_K)}\middle\|Q_1\right), \tag*{(73)} η ≜ D ( P Y ∣ X = x K ( h K , Q K , λ K ) Q 1 ) , (73)
and, for all i ∈ { 1 , 2 , … , K − 1 } i\in\{1,2,\ldots,K-1\} i ∈ { 1 , 2 , … , K − 1 } , consider also the reals
δ i ≜ 1 λ i ( ∫ h i ( x i , y ) d P Y ∣ X = x K ( h K , Q K , λ K ) ( y ) − ∫ h i ( x i , y ) d Q i + 1 ( y ) ) , (74) \delta_i \triangleq\frac{1}{\lambda_i}\left(\int h_i(x_i,y)\,\mathrm{d}P_{Y\mid X=x_K}^{(h_K,Q_K,\lambda_K)}(y)-\int h_i(x_i,y)\,\mathrm{d}Q_{i+1}(y)\right), \tag*{(74)} δ i ≜ λ i 1 ( ∫ h i ( x i , y ) d P Y ∣ X = x K ( h K , Q K , λ K ) ( y ) − ∫ h i ( x i , y ) d Q i + 1 ( y ) ) , (74)
where both expectations are assumed to be finite.
For K = 2 K=2 K = 2 , from Lemma 2, it follows that the Gibbs probability measure P Y ∣ X = x 1 ( h 1 , Q 1 , λ 1 ) P_{Y\mid X=x_1}^{(h_1,Q_1,\lambda_1)} P Y ∣ X = x 1 ( h 1 , Q 1 , λ 1 ) and the nested Gibbs probability measure P Y ∣ X = x 2 ( h 2 , Q 2 , λ 2 ) P_{Y\mid X=x_2}^{(h_2,Q_2,\lambda_2)} P Y ∣ X = x 2 ( h 2 , Q 2 , λ 2 ) are optimizers (minimizer or maximizer depending on the sign of λ 1 \lambda_1 λ 1 and λ 2 \lambda_2 λ 2 , respectively) of the expectations ∫ h 1 ( x 1 , y ) d P 1 ( y ) \int h_1(x_1,y)\mathrm{d}P_1(y) ∫ h 1 ( x 1 , y ) d P 1 ( y ) and ∫ h 2 ( x 2 , y ) d P 2 ( y ) \int h_2(x_2,y)\mathrm{d}P_2(y) ∫ h 2 ( x 2 , y ) d P 2 ( y ) , when P 1 P_1 P 1 and P 2 P_2 P 2 are respectively searched within the neighborhoods G Q 1 , γ 1 \mathcal{G}_{Q_1,\gamma_1} G Q 1 , γ 1 and G Q 2 , γ 2 \mathcal{G}_{Q_2,\gamma_2} G Q 2 , γ 2 both of the form in (61) . Similarly, using Lemma 2, the measure P Y ∣ X = x ( h ^ β , Q 1 , 1 ) P_{Y\mid\mathbf{X}=\mathbf{x}}^{(\widehat{h}_{\boldsymbol{\beta}},Q_1,1)} P Y ∣ X = x ( h β , Q 1 , 1 ) in (49) , which coincides with the nested Gibbs probability measure P Y ∣ X = x 2 ( h 2 , Q 2 , λ 2 ) P_{Y\mid X=x_2}^{(h_2,Q_2,\lambda_2)} P Y ∣ X = x 2 ( h 2 , Q 2 , λ 2 ) , can be identified as a minimizer of the expectation ∫ h ^ β ( ( x 1 , x 2 ) , y ) d P 3 ( y ) \int\widehat{h}_{\boldsymbol{\beta}}((x_1,x_2),y)\mathrm{d}P_3(y) ∫ h β (( x 1 , x 2 ) , y ) d P 3 ( y ) , when P 3 P_3 P 3 is searched within the neighborhood G Q 1 , η \mathcal{G}_{Q_1,\eta} G Q 1 , η . Consequently,
P Y ∣ X = x ( h ^ β , Q 1 , 1 ) = arg min P ∈ G Q 1 , η ∫ h ^ β ( ( x 1 , x 2 ) , y ) d P ( y ) (75) P_{Y\mid\mathbf{X}=\mathbf{x}}^{(\widehat{h}_{\boldsymbol{\beta}},Q_1,1)}=\underset{P\in\mathcal{G}_{Q_1,\eta}}{\arg\min}\int\widehat{h}_{\boldsymbol{\beta}}((x_1,x_2),y)\mathrm{d}P(y) \tag*{(75)} P Y ∣ X = x ( h β , Q 1 , 1 ) = P ∈ G Q 1 , η arg min ∫ h β (( x 1 , x 2 ) , y ) d P ( y ) (75)
= { arg min P ∈ G Q 2 , γ 2 ∫ h 2 ( x 2 , y ) d P ( y ) , λ 2 > 0 ; arg max P ∈ G Q 2 , γ 2 ∫ h 2 ( x 2 , y ) d P ( y ) , λ 2 < 0 , (76) =\begin{cases}
\underset{P\in\mathcal{G}_{Q_2,\gamma_2}}{\arg\min}\int h_2(x_2,y)\mathrm{d}P(y), & \lambda_2>0;\\
\underset{P\in\mathcal{G}_{Q_2,\gamma_2}}{\arg\max}\int h_2(x_2,y)\mathrm{d}P(y), & \lambda_2<0,
\end{cases} \tag*{(76)} = ⎩ ⎨ ⎧ P ∈ G Q 2 , γ 2 arg min ∫ h 2 ( x 2 , y ) d P ( y ) , P ∈ G Q 2 , γ 2 arg max ∫ h 2 ( x 2 , y ) d P ( y ) , λ 2 > 0 ; λ 2 < 0 , (76)
= P Y ∣ X = x 2 ( h 2 , Q 2 , λ 2 ) . (77) =P_{Y\mid X=x_2}^{(h_2,Q_2,\lambda_2)}. \tag*{(77)} = P Y ∣ X = x 2 ( h 2 , Q 2 , λ 2 ) . (77)
The measures P Y ∣ X = x 1 ( h 1 , Q 1 , λ 1 ) P_{Y\mid X=x_1}^{(h_1,Q_1,\lambda_1)} P Y ∣ X = x 1 ( h 1 , Q 1 , λ 1 ) , P Y ∣ X = x 2 ( h 2 , Q 2 , λ 2 ) P_{Y\mid X=x_2}^{(h_2,Q_2,\lambda_2)} P Y ∣ X = x 2 ( h 2 , Q 2 , λ 2 ) , and P Y ∣ X = x ( h ^ β , Q 1 , 1 ) P_{Y\mid\mathbf{X}=\mathbf{x}}^{(\widehat{h}_{\boldsymbol{\beta}},Q_1,1)} P Y ∣ X = x ( h β , Q 1 , 1 ) can be placed within the sets G Q 1 , γ 1 \mathcal{G}_{Q_1,\gamma_1} G Q 1 , γ 1 , G Q 2 , γ 2 \mathcal{G}_{Q_2,\gamma_2} G Q 2 , γ 2 , and G Q 1 , η \mathcal{G}_{Q_1,\eta} G Q 1 , η , respectively, as in Figure 6 .
Figure 6. Geometric representation of nested Gibbs probability measures illustrating (a) G Q 1 , γ 1 ⊆ G Q 1 , η \mathcal{G}_{Q_1,\gamma_1} \subseteq\mathcal{G}_{Q_1,\eta} G Q 1 , γ 1 ⊆ G Q 1 , η and (b) G Q 1 , η ⊆ G Q 1 , γ 1 \mathcal{G}_{Q_1,\eta} \subseteq\mathcal{G}_{Q_1,\gamma_1} G Q 1 , η ⊆ G Q 1 , γ 1 . The dotted neighborhood G Q 2 , γ 2 \mathcal{G}_{Q_2,\gamma_2} G Q 2 , γ 2 has reference measure Q 2 Q_2 Q 2 in both cases.
Beyond this geometrical representation, it is important to highlight that γ 1 \gamma_1 γ 1 and γ 2 \gamma_2 γ 2 in (60) , and η \eta η in (73) exhibit an important relation as shown by the following lemma.
Lemma 14. The measures P Y ∣ X = x 1 ( h 1 , Q 1 , λ 1 ) P_{Y\mid X=x_1}^{(h_1,Q_1,\lambda_1)} P Y ∣ X = x 1 ( h 1 , Q 1 , λ 1 ) in (33) , P Y ∣ X = x 2 ( h 2 , Q 2 , λ 2 ) P_{Y\mid X=x_2}^{(h_2,Q_2,\lambda_2)} P Y ∣ X = x 2 ( h 2 , Q 2 , λ 2 ) in (43) , and P Y ∣ X = x ( h ^ β , Q 1 , 1 ) P_{Y\mid\mathbf{X}=\mathbf{x}}^{(\widehat{h}_{\boldsymbol{\beta}},Q_1,1)} P Y ∣ X = x ( h β , Q 1 , 1 ) in (49) satisfy
η + 1 λ 1 ( ∫ h 1 ( x 1 , y ) d P Y ∣ X = x 2 ( h 2 , Q 2 , λ 2 ) ( y ) − ∫ h 1 ( x 1 , y ) d P Y ∣ X = x 1 ( h 1 , Q 1 , λ 1 ) ( y ) ) = γ 1 + γ 2 , (78) \eta+\frac{1}{\lambda_1}\left(\int h_1(x_1,y)\mathrm{d}P_{Y\mid X=x_2}^{(h_2,Q_2,\lambda_2)}(y)-\int h_1(x_1,y)\mathrm{d}P_{Y\mid X=x_1}^{(h_1,Q_1,\lambda_1)}(y)\right)=\gamma_1+\gamma_2, \tag*{(78)} η + λ 1 1 ( ∫ h 1 ( x 1 , y ) d P Y ∣ X = x 2 ( h 2 , Q 2 , λ 2 ) ( y ) − ∫ h 1 ( x 1 , y ) d P Y ∣ X = x 1 ( h 1 , Q 1 , λ 1 ) ( y ) ) = γ 1 + γ 2 , (78) where γ 1 \gamma_1 γ 1 and γ 2 \gamma_2 γ 2 are defined in (60) , and η \eta η in (73) .
Proof. The proof follows from [27 Lemma 3] .
Note that under the assumption that Q 1 Q_1 Q 1 is a probability measure, both γ 1 \gamma_1 γ 1 and γ 2 \gamma_2 γ 2 are nonnegative, and thus, the left-hand side of (78) is nonnegative as well. These observations lead to interpreting (78) from the perspective of the converse Pythagorean theorem [59 Proposition 48] . From such a perspective, a right-angle triangle can be formed as shown in Figure 7 . This provides yet another intuition on the structure governing the change in the sizes of the neighborhoods G Q 1 , γ 1 \mathcal{G}_{Q_1,\gamma_1} G Q 1 , γ 1 , G Q 2 , γ 2 \mathcal{G}_{Q_2,\gamma_2} G Q 2 , γ 2 , and G Q 1 , η \mathcal{G}_{Q_1,\eta} G Q 1 , η ; and the variation of the expectation of h 1 ( x 1 , y ) h_1(x_1,y) h 1 ( x 1 , y ) when y y y is sampled from a measure that varies from P Y ∣ X = x 1 ( h 1 , Q 1 , λ 1 ) P_{Y\mid X=x_1}^{(h_1,Q_1,\lambda_1)} P Y ∣ X = x 1 ( h 1 , Q 1 , λ 1 ) to P Y ∣ X = x 2 ( h 2 , Q 2 , λ 2 ) P_{Y\mid X=x_2}^{(h_2,Q_2,\lambda_2)} P Y ∣ X = x 2 ( h 2 , Q 2 , λ 2 ) .
Figure 7. Pythagorean interpretation of Lemma 14.
The arguments above can be extended to the case in which K K K Gibbs probability measures are recursively nested. The corresponding identity is given by the following lemma.
Lemma 15. The nested Gibbs probability measures P Y ∣ X = x 1 ( h 1 , Q 1 , λ 1 ) P_{Y\mid X=x_1}^{(h_1,Q_1,\lambda_1)} P Y ∣ X = x 1 ( h 1 , Q 1 , λ 1 ) , P Y ∣ X = x 2 ( h 2 , Q 2 , λ 2 ) P_{Y\mid X=x_2}^{(h_2,Q_2,\lambda_2)} P Y ∣ X = x 2 ( h 2 , Q 2 , λ 2 ) , … \ldots … , P Y ∣ X = x K ( h K , Q K , λ K ) P_{Y\mid X=x_K}^{(h_K,Q_K,\lambda_K)} P Y ∣ X = x K ( h K , Q K , λ K ) in (50) satisfy
∑ k = 1 K D ( P Y ∣ X = x k ( h k , Q k , λ k ) ∥ Q k ) = ∑ i = 1 K − 1 δ i + D ( P Y ∣ X = x K ( h K , Q K , λ K ) ∥ Q 1 ) , (79) \sum_{k=1}^{K}D\left(P_{Y\mid X=x_k}^{(h_k,Q_k,\lambda_k)}\middle\|Q_k\right)=\sum_{i=1}^{K-1}\delta_i+D\left(P_{Y\mid X=x_K}^{(h_K,Q_K,\lambda_K)}\middle\|Q_1\right), \tag*{(79)} k = 1 ∑ K D ( P Y ∣ X = x k ( h k , Q k , λ k ) Q k ) = i = 1 ∑ K − 1 δ i + D ( P Y ∣ X = x K ( h K , Q K , λ K ) Q 1 ) , (79)
where δ i \delta_i δ i is defined in (74) .
Proof: The proof is presented in Appendix § 16 .
In Lemma 15, the sum ∑ k = 1 K D ( P Y ∣ X = x k ( h k , Q k , λ k ) ∥ Q k ) \sum_{k=1}^{K}D\left(P_{Y\mid X=x_k}^{(h_k,Q_k,\lambda_k)}\middle\|Q_k\right) ∑ k = 1 K D ( P Y ∣ X = x k ( h k , Q k , λ k ) Q k ) can be viewed as the length of a path from the reference measure Q 1 Q_1 Q 1 to the final nested Gibbs probability measure P Y ∣ X = x K ( h K , Q K , λ K ) P_{Y\mid X=x_K}^{(h_K,Q_K,\lambda_K)} P Y ∣ X = x K ( h K , Q K , λ K ) , passing successively through the reference measures Q 2 , Q 3 , … , Q K Q_2,Q_3,\ldots,Q_K Q 2 , Q 3 , … , Q K . Figure 8 provides a geometric representation of this path for K = 4 K=4 K = 4 .
Figure 8. Geometric representation of recursively nested Gibbs probability measures illustrating (a) G Q 1 , γ 1 ⊆ G Q 1 , η \mathcal{G}_{Q_1,\gamma_1} \subseteq\mathcal{G}_{Q_1,\eta} G Q 1 , γ 1 ⊆ G Q 1 , η and (b) G Q 1 , η ⊆ G Q 1 , γ 1 \mathcal{G}_{Q_1,\eta} \subseteq\mathcal{G}_{Q_1,\gamma_1} G Q 1 , η ⊆ G Q 1 , γ 1 .
An equivalent form of the identity in (79) is
γ 1 + ∑ i = 2 K γ i = ∑ i = 1 K − 1 δ i + η . (80) \gamma_1+\sum_{i=2}^{K}\gamma_i=\sum_{i=1}^{K-1}\delta_i+\eta. \tag*{(80)} γ 1 + i = 2 ∑ K γ i = i = 1 ∑ K − 1 δ i + η . (80)
Under the assumption that Q 1 Q_1 Q 1 is a probability measure, the reals γ 1 , γ 2 , … , γ K \gamma_1,\gamma_2,\ldots,\gamma_K γ 1 , γ 2 , … , γ K are nonnegative. Consequently, from (80) , it follows that
∑ i = 1 K − 1 δ i + η ≥ 0 , (81) \sum_{i=1}^{K-1}\delta_i+\eta\ge0, \tag*{(81)} i = 1 ∑ K − 1 δ i + η ≥ 0 , (81)
which allows interpreting (80) from the perspective of the converse Pythagorean theorem [59 Proposition 48] . Figure 9 shows the corresponding right-angle triangle.
Figure 9. Pythagorean interpretation of Lemma 15.
Gibbs Measures in Supervised Machine Learning
This section studies normalized log-linear combinations of Gibbs probability measures (Definition 5) and nested Gibbs probability measures (Definition 6) in the context of supervised machine learning. The first subsection introduces a formal formulation of such a problem and establishes the notation used throughout the section. The following subsections leverage the results in Section III to characterize the solutions to optimizations of weighted empirical risks, subject to relative entropy regularizations, as results of log-linear combinations of Gibbs measures and/or nesting multiple Gibbs probability measures.
The Problem of Supervised Machine Learning
Let X X X and Y Y Y be sets of patterns and labels, respectively. Let M ⊆ R d \mathcal{M} \subseteq\mathbb{R}^{d} M ⊆ R d , with d ∈ N d \in\mathbb{N} d ∈ N , denote a set of models. A data point ( x , y ) (x,y) ( x , y ) is an element of the set Z ≜ X × Y \mathcal{Z} \triangleq X \times Y Z ≜ X × Y . A training dataset z z z consists of a tuple of n ∈ N n \in\mathbb{N} n ∈ N data points, explicitly written as
z ≜ ( ( x 1 , y 1 ) , ( x 2 , y 2 ) , … , ( x n , y n ) ) ∈ Z n . (82) z \triangleq((x_{1},y_{1}), (x_{2},y_{2}),\ldots,(x_{n},y_{n})) \in\mathcal{Z}^{n}. \tag*{(82)} z ≜ (( x 1 , y 1 ) , ( x 2 , y 2 ) , … , ( x n , y n )) ∈ Z n . (82)
Given a model θ ∈ M \theta\in\mathcal{M} θ ∈ M , the loss induced by such a model on a data point ( x , y ) ∈ Z (x,y) \in\mathcal{Z} ( x , y ) ∈ Z is denoted by ℓ ( x , y , θ ) \ell(x,y,\theta) ℓ ( x , y , θ ) , where
ℓ : Z × M → [ 0 , + ∞ ) (83) \ell: \mathcal{Z} \times\mathcal{M} \to[0,+\infty) \tag*{(83)} ℓ : Z × M → [ 0 , + ∞ ) (83)
is referred to as the loss function , which is assumed to be Borel measurable. The empirical risk induced by a model with respect to an n n n -length dataset is determined by the function
L n : { Z n × M → [ 0 , + ∞ ) ( z , θ ) ↦ 1 n ∑ i = 1 n ℓ ( x i , y i , θ ) . (84) L_{n} :
\begin{cases}
\mathcal{Z}^{n} \times\mathcal{M} \to[0,+\infty) \\
(z,\theta) \mapsto\frac{1}{n}\sum_{i=1}^{n}\ell(x_{i},y_{i},\theta).
\end{cases} \tag*{(84)} L n : { Z n × M → [ 0 , + ∞ ) ( z , θ ) ↦ n 1 ∑ i = 1 n ℓ ( x i , y i , θ ) . (84)
The expected empirical risk with respect to a fixed dataset z z z of the form in (81), when models are sampled from a given probability measure, is determined by the functional
R z : { △ ( M ) → [ 0 , + ∞ ) P ↦ ∫ L n ( z , θ ) d P ( θ ) . (85) R_{z} :
\begin{cases}
\triangle(\mathcal{M}) \to[0,+\infty) \\
P \mapsto\int L_{n}(z,\theta)\,\mathrm{d}P(\theta).
\end{cases} \tag*{(85)} R z : { △ ( M ) → [ 0 , + ∞ ) P ↦ ∫ L n ( z , θ ) d P ( θ ) . (85)
In this work, a supervised learning algorithm, trained upon n n n -length datasets, is represented by a conditional probability measure in △ ( M ∣ Z n ) \triangle(\mathcal{M}\mid\mathcal{Z}^{n}) △ ( M ∣ Z n ) as defined hereunder.
Definition 7 (Algorithm). A conditional probability measure P Θ ∣ Z ∈ △ ( M ∣ Z n ) P_{\Theta\mid Z} \in\triangle(\mathcal{M}\mid\mathcal{Z}^{n}) P Θ ∣ Z ∈ △ ( M ∣ Z n ) is said to represent a supervised machine learning algorithm with n n n -length training datasets.
The instance of such an algorithm trained upon the dataset z z z in (81) is denoted by P Θ ∣ Z = z P_{\Theta\mid Z=z} P Θ ∣ Z = z , which is simply a probability measure in △ ( M ) \triangle(\mathcal{M}) △ ( M ) . Given a σ \sigma σ -finite measure Q Q Q defined on the set M \mathcal{M} M and a real λ ∈ R ∖ { 0 } \lambda\in\mathbb{R}\setminus\{0\} λ ∈ R ∖ { 0 } , a central class of algorithms known as Gibbs algorithms, are represented by ( L n , Q , λ ) (L_{n},Q,\lambda) ( L n , Q , λ ) -Gibbs conditional probability measures (Definition 3). An example of such a class of algorithms can be denoted by
P Θ ∣ Z ( L n , Q , λ ) ∈ △ ( M ∣ Z n ) . (86) P_{\Theta\mid Z}^{(L_{n},Q,\lambda)} \in\triangle(\mathcal{M}\mid\mathcal{Z}^{n}). \tag*{(86)} P Θ ∣ Z ( L n , Q , λ ) ∈ △ ( M ∣ Z n ) . (86)
Given a fixed dataset z ∈ Z n z \in\mathcal{Z}^{n} z ∈ Z n , the instance of the Gibbs algorithm in (85) obtained by training it upon the dataset z z z is represented by the measure
P Θ ∣ Z = z ( L n , Q , λ ) ∈ △ Q ( M ) , (87) P_{\Theta\mid Z=z}^{(L_{n},Q,\lambda)} \in\triangle_{Q}(\mathcal{M}), \tag*{(87)} P Θ ∣ Z = z ( L n , Q , λ ) ∈ △ Q ( M ) , (87)
whose Radon-Nikodym derivative with respect to Q Q Q satisfies
d P Θ ∣ Z = z ( L n , Q , λ ) d Q ( θ ) = exp ( − 1 λ L n ( z , θ ) ) ∫ exp ( − 1 λ L n ( z , ν ) ) d Q ( ν ) . (88) \frac{\mathrm{d}P_{\Theta\mid Z=z}^{(L_n,Q,\lambda)}}{\mathrm{d}Q}(\theta)=\frac{\exp\left(-\frac{1}{\lambda}L_n(z,\theta)\right)}{\int\exp\left(-\frac{1}{\lambda}L_n(z,\nu)\right)\mathrm{d}Q(\nu)}. \tag*{(88)} d Q d P Θ ∣ Z = z ( L n , Q , λ ) ( θ ) = ∫ exp ( − λ 1 L n ( z , ν ) ) d Q ( ν ) exp ( − λ 1 L n ( z , θ ) ) . (88)
The Gibbs probability measure P Θ ∣ Z = z ( L n , Q , λ ) P_{\Theta\mid Z=z}^{(L_n,Q,\lambda)} P Θ ∣ Z = z ( L n , Q , λ ) in (86) admits two complementary characterizations in terms of empirical-risk optimization problems. The first characterization consists of an empirical-risk optimization (minimization or maximization) with relative-entropy regularization:
min P ∈ △ Q ( M ) R z ( P ) + λ D ( P ∥ Q ) , with λ > 0 ; (89) \min_{P\in\triangle_Q(\mathcal{M})} R_z(P)+\lambda D(P\mathbin\Vert Q), \qquad\text{with }\lambda>0; \tag*{(89)} P ∈ △ Q ( M ) min R z ( P ) + λ D ( P ∥ Q ) , with λ > 0 ; (89)
max P ∈ △ Q ( M ) R z ( P ) + λ D ( P ∥ Q ) , with λ < 0 , (90) \max_{P\in\triangle_Q(\mathcal{M})} R_z(P)+\lambda D(P\mathbin\Vert Q), \qquad\text{with }\lambda<0, \tag*{(90)} P ∈ △ Q ( M ) max R z ( P ) + λ D ( P ∥ Q ) , with λ < 0 , (90)
where R z R_z R z is defined in (85) . These problems have been thoroughly studied in [15 ] , [20 ] , [26 ] . The following lemma establishes their connection with Gibbs algorithms.
Lemma 16. Assume that the optimization problem in (87) (respectively, in (88)) admits a solution. Then, if λ > 0 \lambda>0 λ > 0 (respectively, if λ < 0 \lambda<0 λ < 0 ), the Gibbs probability measure P Θ ∣ Z = z ( L n , Q , λ ) P_{\Theta\mid Z=z}^{(L_n,Q,\lambda)} P Θ ∣ Z = z ( L n , Q , λ ) in (86) is the unique solution to (87) (respectively, to (88)).
Proof. The proof is immediate from Lemma 1.
The optimization problems in (87) and (88) provide a classical method for constructing Gibbs algorithms [15 ] . The minimization is motivated by building an algorithm that minimizes a linear combination of the expectation of the empirical risk and a relative entropy term. The first term represents the objective of searching for probability measures concentrated on the minimizers of the empirical risk with respect to the training dataset z z z , i.e., the set arg min θ ∈ M L n ( z , θ ) \arg\min_{\theta\in\mathcal{M}} L_n(z,\theta) arg min θ ∈ M L n ( z , θ ) . The second term represents the aim of constraining such a search to measures that have small relative entropy with respect to the reference measure Q Q Q . That is, measures that “look like” or are “close to” the reference measure, in the sense of relative entropy. This suggests that λ \lambda λ deals with the trade-off between the solution being driven by the training data, for small ∣ λ ∣ \lvert\lambda\rvert ∣ λ ∣ , or by the reference measure, for large ∣ λ ∣ \lvert\lambda\rvert ∣ λ ∣ . Interestingly, this justifies why Q Q Q is often seen as a prior in the existing literature. See [7 ] , [15 ] , [19 ] and references therein. While the minimization in (87) concentrates most of the attention in the existing literature, the maximization in (88) is shown later in this work to be important for machine unlearning.
The second characterization consists of an empirical-risk optimization (minimization or maximization) subject to a relative entropy constraint:
min P ∈ △ Q ( M ) R z ( P ) (91) \min_{P\in\triangle_Q(\mathcal{M})} R_z(P) \tag*{(91)} P ∈ △ Q ( M ) min R z ( P ) (91)
s.t. D ( P ∥ Q ) ≤ γ , (92) \text{s.t.}\quad D(P\mathbin\Vert Q)\leq\gamma, \tag*{(92)} s.t. D ( P ∥ Q ) ≤ γ , (92)
and
max P ∈ △ Q ( M ) R z ( P ) (93) \max_{P\in\triangle_Q(\mathcal{M})} R_z(P) \tag*{(93)} P ∈ △ Q ( M ) max R z ( P ) (93)
s.t. D ( P ∥ Q ) ≤ γ , (94) \text{s.t.}\quad D(P\mathbin\Vert Q)\leq\gamma, \tag*{(94)} s.t. D ( P ∥ Q ) ≤ γ , (94)
where γ ∈ R \gamma\in\mathbb{R} γ ∈ R . These problems have been thoroughly studied in [27 ] . The following lemma establishes their connection with Gibbs algorithms.
Lemma 17. Assume λ > 0 \lambda>0 λ > 0 (respectively, λ < 0 \lambda<0 λ < 0 ) is such that the ( L n , Q , λ ) (L_n,Q,\lambda) ( L n , Q , λ ) Gibbs probability measure P Θ ∣ Z = z ( L n , Q , λ ) P_{\Theta\mid Z=z}^{(L_n,Q,\lambda)} P Θ ∣ Z = z ( L n , Q , λ ) in (86) satisfies
D ( P Θ ∣ Z = z ( L n , Q , λ ) ∥ Q ) = γ , (95) D\left(P_{\Theta\mid Z=z}^{(L_n,Q,\lambda)}\mathbin\Vert Q\right)=\gamma, \tag*{(95)} D ( P Θ ∣ Z = z ( L n , Q , λ ) ∥ Q ) = γ , (95) for some given γ ∈ R \gamma\in\mathbb{R} γ ∈ R . Then, such a measure P Θ ∣ Z = z ( L n , Q , λ ) P_{\Theta\mid Z=z}^{(L_n,Q,\lambda)} P Θ ∣ Z = z ( L n , Q , λ ) is the unique solution to (89) (respectively, to (90)).
Proof. The proof is immediate from Lemma 2.
When Q Q Q is a probability measure, the optimization problems in (89) and (90) respectively consist in the minimization and maximization of the empirical risk within a relative entropy neighborhood G Q , γ \mathcal{G}_{Q,\gamma} G Q , γ , with γ \gamma γ in (91), around the reference measure Q Q Q . Hence, the ( L n , Q , λ ) (L_n,Q,\lambda) ( L n , Q , λ ) Gibbs probability measure P Θ ∣ Z = z ( L n , Q , λ ) P_{\Theta\mid Z=z}^{(L_n,Q,\lambda)} P Θ ∣ Z = z ( L n , Q , λ ) in (86) is the minimizer (λ > 0 \lambda>0 λ > 0 ) or the maximizer (λ < 0 \lambda<0 λ < 0 ), of the expected empirical risk R z R_z R z within G Q , γ \mathcal{G}_{Q,\gamma} G Q , γ .
Normalized Log-Linear Combination of Gibbs Algorithms
This subsection leverages the results on normalized log-linear combinations of Gibbs probability measures presented in Section III-B. In the supervised learning setting, these results reveal a direct connection between normalized log-linear combinations of Gibbs algorithms and weighted empirical risks. Consider two datasets
z 1 ≜ ( ( x 1 , 1 , y 1 , 1 ) , ( x 1 , 2 , y 1 , 2 ) , … , ( x 1 , n 1 , y 1 , n 1 ) ) ∈ Z n 1 (96) \boldsymbol{z}_1 \triangleq((x_{1,1},y_{1,1}),(x_{1,2},y_{1,2}),\ldots,(x_{1,n_1},y_{1,n_1}))\in\mathcal{Z}^{n_1} \tag*{(96)} z 1 ≜ (( x 1 , 1 , y 1 , 1 ) , ( x 1 , 2 , y 1 , 2 ) , … , ( x 1 , n 1 , y 1 , n 1 )) ∈ Z n 1 (96)
z 2 ≜ ( ( x 2 , 1 , y 2 , 1 ) , ( x 2 , 2 , y 2 , 2 ) , … , ( x 2 , n 2 , y 2 , n 2 ) ) ∈ Z n 2 , (97) \boldsymbol{z}_2 \triangleq((x_{2,1},y_{2,1}),(x_{2,2},y_{2,2}),\ldots,(x_{2,n_2},y_{2,n_2}))\in\mathcal{Z}^{n_2}, \tag*{(97)} z 2 ≜ (( x 2 , 1 , y 2 , 1 ) , ( x 2 , 2 , y 2 , 2 ) , … , ( x 2 , n 2 , y 2 , n 2 )) ∈ Z n 2 , (97)
with n 1 n_1 n 1 and n 2 n_2 n 2 data points, respectively. Let λ 1 ∈ R ∖ { 0 } \lambda_1\in\mathbb{R}\setminus\{0\} λ 1 ∈ R ∖ { 0 } and λ 2 ∈ R ∖ { 0 } \lambda_2\in\mathbb{R}\setminus\{0\} λ 2 ∈ R ∖ { 0 } be two given reals; and let Q 1 Q_1 Q 1 and Q 2 Q_2 Q 2 be two given σ \sigma σ -finite measures. Consider also an ( L n 1 , Q 1 , λ 1 ) (L_{n_1},Q_1,\lambda_1) ( L n 1 , Q 1 , λ 1 ) -Gibbs probability measure denoted by
P Θ ∣ Z = z 1 ( L n 1 , Q 1 , λ 1 ) ∈ △ Q 1 ( M ) (98) P_{\Theta\mid Z=z_1}^{(L_{n_1},Q_1,\lambda_1)}\in\triangle_{Q_1}(\mathcal{M}) \tag*{(98)} P Θ ∣ Z = z 1 ( L n 1 , Q 1 , λ 1 ) ∈ △ Q 1 ( M ) (98)
and an ( L n 2 , Q 2 , λ 2 ) (L_{n_2},Q_2,\lambda_2) ( L n 2 , Q 2 , λ 2 ) -Gibbs probability measure denoted by
P Θ ∣ Z = z 2 ( L n 2 , Q 2 , λ 2 ) ∈ △ Q 2 ( M ) , (99) P_{\Theta\mid Z=z_2}^{(L_{n_2},Q_2,\lambda_2)}\in\triangle_{Q_2}(\mathcal{M}), \tag*{(99)} P Θ ∣ Z = z 2 ( L n 2 , Q 2 , λ 2 ) ∈ △ Q 2 ( M ) , (99)
where the functions L n 1 L_{n_1} L n 1 and L n 2 L_{n_2} L n 2 are defined as in (84) . Given α 1 ∈ R ∖ { 0 } \alpha_1\in\mathbb{R}\setminus\{0\} α 1 ∈ R ∖ { 0 } , α 2 ∈ R ∖ { 0 } \alpha_2\in\mathbb{R}\setminus\{0\} α 2 ∈ R ∖ { 0 } and a σ \sigma σ -finite measure Q Q Q on M \mathcal{M} M , such that Q ≪ Q 1 Q\ll Q_1 Q ≪ Q 1 and Q ≪ Q 2 Q\ll Q_2 Q ≪ Q 2 , let the normalized log-linear combination of the Gibbs probability measures in (98) and (99) with respect to Q Q Q be denoted by S 2 ∈ △ ( M ) S_2\in\triangle(\mathcal{M}) S 2 ∈ △ ( M ) , such that for all θ ∈ supp Q \theta\in\operatorname{supp}Q θ ∈ supp Q ,
d S 2 d Q ( θ ) = ( d P Θ ∣ Z = z 1 ( L n 1 , Q 1 , λ 1 ) d Q 1 ( θ ) ) α 1 ( d P Θ ∣ Z = z 2 ( L n 2 , Q 2 , λ 2 ) d Q 2 ( θ ) ) α 2 ∫ ( d P Θ ∣ Z = z 1 ( L n 1 , Q 1 , λ 1 ) d Q 1 ( ν ) ) α 1 ( d P Θ ∣ Z = z 2 ( L n 2 , Q 2 , λ 2 ) d Q 2 ( ν ) ) α 2 d Q ( ν ) . (100) \frac{\mathrm{d}S_2}{\mathrm{d}Q}(\theta)=\frac{\left(\frac{\mathrm{d}P_{\Theta\mid Z=z_1}^{(L_{n_1},Q_1,\lambda_1)}}{\mathrm{d}Q_1}(\theta)\right)^{\alpha_1}\left(\frac{\mathrm{d}P_{\Theta\mid Z=z_2}^{(L_{n_2},Q_2,\lambda_2)}}{\mathrm{d}Q_2}(\theta)\right)^{\alpha_2}}{\int\left(\frac{\mathrm{d}P_{\Theta\mid Z=z_1}^{(L_{n_1},Q_1,\lambda_1)}}{\mathrm{d}Q_1}(\nu)\right)^{\alpha_1}\left(\frac{\mathrm{d}P_{\Theta\mid Z=z_2}^{(L_{n_2},Q_2,\lambda_2)}}{\mathrm{d}Q_2}(\nu)\right)^{\alpha_2}\mathrm{d}Q(\nu)}. \tag*{(100)} d Q d S 2 ( θ ) = ∫ ( d Q 1 d P Θ ∣ Z = z 1 ( L n 1 , Q 1 , λ 1 ) ( ν ) ) α 1 ( d Q 2 d P Θ ∣ Z = z 2 ( L n 2 , Q 2 , λ 2 ) ( ν ) ) α 2 d Q ( ν ) ( d Q 1 d P Θ ∣ Z = z 1 ( L n 1 , Q 1 , λ 1 ) ( θ ) ) α 1 ( d Q 2 d P Θ ∣ Z = z 2 ( L n 2 , Q 2 , λ 2 ) ( θ ) ) α 2 . (100)
For ease of presentation, consider a positive integer m ∈ N m\in\mathbb{N} m ∈ N , and the function
L ‾ m : { R m × Z m × M ⟶ R ( ω , z , θ ) ⟼ ∑ t = 1 m ω t ℓ ( x t , y t , θ ) . (101) \overline{L}_m:\left\{\begin{aligned}
\mathbb{R}^m\times\mathcal{Z}^m\times\mathcal{M}&\longrightarrow\mathbb{R}\\
(\boldsymbol{\omega},\boldsymbol{z},\theta)&\longmapsto\sum_{t=1}^{m}\omega_t\ell(x_t,y_t,\theta).
\end{aligned}\right. \tag*{(101)} L m : ⎩ ⎨ ⎧ R m × Z m × M ( ω , z , θ ) ⟶ R ⟼ t = 1 ∑ m ω t ℓ ( x t , y t , θ ) . (101)
where the vector ω ∈ R m \boldsymbol{\omega}\in\mathbb{R}^m ω ∈ R m is of the form ω = ( ω 1 , ω 2 , … , ω m ) \boldsymbol{\omega}=(\omega_1,\omega_2,\ldots,\omega_m) ω = ( ω 1 , ω 2 , … , ω m ) ; the m m m -length dataset z \boldsymbol{z} z is of the form in (82) ; and the function ℓ \ell ℓ is defined in (83) . Note that L ‾ m ( ω , z , θ ) \overline{L}_m(\boldsymbol{\omega},\boldsymbol{z},\theta) L m ( ω , z , θ ) is a weighted empirical risk. More specifically, each data point ( x t , y t ) (x_t,y_t) ( x t , y t ) in the dataset z \boldsymbol{z} z is associated with a weight ω t \omega_t ω t , with t ∈ { 1 , 2 , … , m } t\in\{1,2,\ldots,m\} t ∈ { 1 , 2 , … , m } . Note that, if for all t ∈ { 1 , 2 , … , m } t\in\{1,2,\ldots,m\} t ∈ { 1 , 2 , … , m } , ω t = 1 m \omega_t=\frac{1}{m} ω t = m 1 , the function L ‾ m \overline{L}_m L m in (101) satisfies that L ‾ m ( ω , z , θ ) = L m ( z , θ ) \overline{L}_m(\boldsymbol{\omega},\boldsymbol{z},\theta)=L_m(\boldsymbol{z},\theta) L m ( ω , z , θ ) = L m ( z , θ ) , where the function L m L_m L m is the empirical risk function in (84) .
Let the aggregation of the datasets z 1 \boldsymbol{z}_1 z 1 and z 2 \boldsymbol{z}_2 z 2 in (92) be denoted by
z 0 ≜ ( z 1 , z 2 ) (102) \boldsymbol{z}_0\triangleq(\boldsymbol{z}_1,\boldsymbol{z}_2) \tag*{(102)} z 0 ≜ ( z 1 , z 2 ) (102)
= ( ( x 0 , 1 , y 0 , 1 ) , ( x 0 , 2 , y 0 , 2 ) , … , ( x 0 , n 0 , y 0 , n 0 ) ) ∈ Z n 0 , (103) =((x_{0,1},y_{0,1}),(x_{0,2},y_{0,2}),\ldots,(x_{0,n_0},y_{0,n_0}))\in\mathcal{Z}^{n_0}, \tag*{(103)} = (( x 0 , 1 , y 0 , 1 ) , ( x 0 , 2 , y 0 , 2 ) , … , ( x 0 , n 0 , y 0 , n 0 )) ∈ Z n 0 , (103)
where
n 0 ≜ n 1 + n 2 . (104) n_0\triangleq n_1+n_2. \tag*{(104)} n 0 ≜ n 1 + n 2 . (104)
Using this notation, Theorem 7 leads to an explicit expression for the measure S 2 S_2 S 2 in (100) , as shown by the following lemma.
Lemma 18. The normalized log-linear combination S 2 S_2 S 2 in (100) satisfies for all θ ∈ supp Q \theta\in\operatorname{supp}Q θ ∈ supp Q ,
d S 2 d Q ( θ ) = exp ( − L ‾ n 0 ( ω , z 0 , θ ) ) ∫ exp ( − L ‾ n 0 ( ω , z 0 , ν ) ) d Q ( ν ) , (105) \frac{\mathrm{d}S_2}{\mathrm{d}Q}(\theta)=\frac{\exp(-\overline{L}_{n_0}(\boldsymbol{\omega},\boldsymbol{z}_0,\theta))}{\int\exp(-\overline{L}_{n_0}(\boldsymbol{\omega},\boldsymbol{z}_0,\nu))\mathrm{d}Q(\nu)}, \tag*{(105)} d Q d S 2 ( θ ) = ∫ exp ( − L n 0 ( ω , z 0 , ν )) d Q ( ν ) exp ( − L n 0 ( ω , z 0 , θ )) , (105) where the function L ‾ n 0 \overline{L}_{n_0} L n 0 is defined in (101) ; the dataset z 0 \boldsymbol{z}_0 z 0 is defined in (97); and the vector ω = ( ω 1 , ω 2 , … , ω n 0 ) \boldsymbol{\omega}=(\omega_1,\omega_2,\ldots,\omega_{n_0}) ω = ( ω 1 , ω 2 , … , ω n 0 ) satisfies, for all t ∈ { 1 , 2 , … , n 0 } t\in\{1,2,\ldots,n_0\} t ∈ { 1 , 2 , … , n 0 } ,
ω t = { α 1 n 1 λ 1 if t ≤ n 1 , α 2 n 2 λ 2 if n 1 < t ≤ n 0 . (106) \omega_t=\begin{cases}
\frac{\alpha_1}{n_1\lambda_1} & \text{if }t\le n_1,\\
\frac{\alpha_2}{n_2\lambda_2} & \text{if }n_1<t\le n_0.
\end{cases} \tag*{(106)} ω t = { n 1 λ 1 α 1 n 2 λ 2 α 2 if t ≤ n 1 , if n 1 < t ≤ n 0 . (106) Proof: The proof follows from Theorem 7 by observing that
L ‾ n 0 ( ω , z 0 , θ ) = α 1 λ 1 L n 1 ( z 1 , θ ) + α 2 λ 2 L n 2 ( z 2 , θ ) , (107) \overline{L}_{n_0}(\boldsymbol{\omega},\boldsymbol{z}_0,\theta)=\frac{\alpha_1}{\lambda_1}L_{n_1}(\boldsymbol{z}_1,\theta)+\frac{\alpha_2}{\lambda_2}L_{n_2}(\boldsymbol{z}_2,\theta), \tag*{(107)} L n 0 ( ω , z 0 , θ ) = λ 1 α 1 L n 1 ( z 1 , θ ) + λ 2 α 2 L n 2 ( z 2 , θ ) , (107)
which completes the proof.
Lemma 18 shows that the normalized log-linear combination of the two Gibbs probability measures in (98) and (99) is itself a Gibbs probability measure whose objective function is a weighted empirical risk over the aggregated dataset Z 0 \mathcal{Z}_0 Z 0 . More precisely, for all i ∈ { 1 , 2 } i \in\{1,2\} i ∈ { 1 , 2 } , the contribution associated with the dataset Z i \mathcal{Z}_i Z i is jointly determined by the coefficient α i \alpha_i α i of the normalized log-linear combination and the regularization factor λ i \lambda_i λ i of the corresponding Gibbs probability measure. This Gibbs representation also provides an optimization interpretation of the normalized log-linear combination. From Lemma 1, the probability measure S 2 S_2 S 2 in (100) is the unique solution to the weighted empirical-risk minimization with relative-entropy regularization:
min P ∈ △ Q ( M ) [ L ‾ n 0 ( ω , z 0 , θ ) d P ( θ ) + D ( P ∥ Q ) ] (108) \min_{P\in\triangle_Q(\mathcal{M})}\left[\overline{L}_{n_0}(\boldsymbol{\omega},\boldsymbol{z}_0,\theta)\mathrm{d}P(\theta)+D(P\mathbin\Vert Q)\right] \tag*{(108)} P ∈ △ Q ( M ) min [ L n 0 ( ω , z 0 , θ ) d P ( θ ) + D ( P ∥ Q ) ] (108)
where the vector of weights ω \boldsymbol{\omega} ω satisfies (106) .
For some fixed m ∈ N m \in\mathbb{N} m ∈ N and ω ∈ R m \boldsymbol{\omega}\in\mathbb{R}^m ω ∈ R m , let the notation L ‾ m ( ω , ⋅ , ⋅ ) \overline{L}_m(\boldsymbol{\omega},\cdot,\cdot) L m ( ω , ⋅ , ⋅ ) , with the function L ‾ m \overline{L}_m L m defined as in (101) , identify the mapping
L ‾ m ( ω , ⋅ , ⋅ ) : { Z m × M ⟶ R ( z , θ ) ⟼ ∑ t = 1 m ω t ℓ ( x t , y t , θ ) . (109) \overline{L}_m(\boldsymbol{\omega},\cdot,\cdot):
\begin{cases}
\mathcal{Z}^m\times\mathcal{M}\longrightarrow\mathbb{R}\\
(\boldsymbol{z},\theta)\longmapsto\sum_{t=1}^{m}\omega_t\ell(x_t,y_t,\theta).
\end{cases} \tag*{(109)} L m ( ω , ⋅ , ⋅ ) : { Z m × M ⟶ R ( z , θ ) ⟼ ∑ t = 1 m ω t ℓ ( x t , y t , θ ) . (109)
Using this notation, Lemma 18 and Definition 3 imply that the probability measure S 2 S_2 S 2 in (100) is identical to an ( L ‾ n 0 ( ω , ⋅ , ⋅ ) , Q , 1 ) (\overline{L}_{n_0}(\boldsymbol{\omega},\cdot,\cdot),Q,1) ( L n 0 ( ω , ⋅ , ⋅ ) , Q , 1 ) -Gibbs probability measure denoted by
P Θ ∣ Z = z 0 ( L ‾ n 0 ( ω , ⋅ , ⋅ ) , Q , 1 ) ∈ △ Q ( M ) , (110) P_{\Theta\mid Z=z_0}^{(\overline{L}_{n_0}(\boldsymbol{\omega},\cdot,\cdot),Q,1)}\in\triangle_Q(\mathcal{M}), \tag*{(110)} P Θ ∣ Z = z 0 ( L n 0 ( ω , ⋅ , ⋅ ) , Q , 1 ) ∈ △ Q ( M ) , (110)
which, from Lemma 1 is the unique solution to (108) . The following corollary formalizes this observation.
Corollary 19. The probability measure S 2 S_2 S 2 in (100) and the ( L ‾ n 0 ( ω , ⋅ , ⋅ ) , Q , 1 ) (\overline{L}_{n_0}(\boldsymbol{\omega},\cdot,\cdot),Q,1) ( L n 0 ( ω , ⋅ , ⋅ ) , Q , 1 ) -Gibbs probability measure P Θ ∣ Z = z 0 ( L ‾ n 0 ( ω , ⋅ , ⋅ ) , Q , 1 ) P_{\Theta\mid Z=z_0}^{(\overline{L}_{n_0}(\boldsymbol{\omega},\cdot,\cdot),Q,1)} P Θ ∣ Z = z 0 ( L n 0 ( ω , ⋅ , ⋅ ) , Q , 1 ) in (110) satisfy
D ( S 2 ∥ P Θ ∣ Z = z 0 ( L ‾ n 0 ( ω , ⋅ , ⋅ ) , Q , 1 ) ) = 0. (111) D\left(S_2\mathbin\Vert P_{\Theta\mid Z=z_0}^{(\overline{L}_{n_0}(\boldsymbol{\omega},\cdot,\cdot),Q,1)}\right)=0. \tag*{(111)} D ( S 2 ∥ P Θ ∣ Z = z 0 ( L n 0 ( ω , ⋅ , ⋅ ) , Q , 1 ) ) = 0. (111)
Lemma 18 reveals that the weight associated to each data point on the measure S 2 S_2 S 2 in (105) depends on whether it is part of z 1 \boldsymbol{z}_1 z 1 or z 2 \boldsymbol{z}_2 z 2 . This naturally raises the question of whether the parameters of the normalized log-linear combination can be chosen such that all data points exhibit the same weight in the weighted empirical risk in (108) .
Lemma 20. Consider a real λ 0 ∈ R ∖ { 0 } \lambda_0\in\mathbb{R}\setminus\{0\} λ 0 ∈ R ∖ { 0 } . Assume also that the regularization factors λ 1 \lambda_1 λ 1 and λ 2 \lambda_2 λ 2 of the Gibbs probability measures in (100) satisfy
λ 1 = n 0 λ 0 α 1 n 1 ; λ 2 = n 0 λ 0 α 2 n 2 . (112) \lambda_1=\frac{n_0\lambda_0\alpha_1}{n_1};\qquad\lambda_2=\frac{n_0\lambda_0\alpha_2}{n_2}. \tag*{(112)} λ 1 = n 1 n 0 λ 0 α 1 ; λ 2 = n 2 n 0 λ 0 α 2 . (112) Then, the probability measure S 2 S_2 S 2 in (100) satisfies, for all θ ∈ supp Q \theta\in\operatorname{supp}Q θ ∈ supp Q ,
d S 2 d Q ( θ ) = exp ( − 1 λ 0 L ‾ n 0 ( z 0 , θ ) ) ∫ exp ( − 1 λ 0 L ‾ n 0 ( z 0 , ν ) ) d Q ( ν ) (113) \frac{\mathrm{d}S_2}{\mathrm{d}Q}(\theta)=\frac{\exp\left(-\frac{1}{\lambda_0}\overline{L}_{n_0}(\boldsymbol{z}_0,\theta)\right)}{\int\exp\left(-\frac{1}{\lambda_0}\overline{L}_{n_0}(\boldsymbol{z}_0,\nu)\right)\mathrm{d}Q(\nu)} \tag*{(113)} d Q d S 2 ( θ ) = ∫ exp ( − λ 0 1 L n 0 ( z 0 , ν ) ) d Q ( ν ) exp ( − λ 0 1 L n 0 ( z 0 , θ ) ) (113) = d P Θ ∣ Z = z 0 ( L ‾ n 0 , Q , λ 0 ) d Q ( θ ) , (114) =\frac{\mathrm{d}P_{\Theta\mid Z=z_0}^{(\overline{L}_{n_0},Q,\lambda_0)}}{\mathrm{d}Q}(\theta), \tag*{(114)} = d Q d P Θ ∣ Z = z 0 ( L n 0 , Q , λ 0 ) ( θ ) , (114) where the probability measure P Θ ∣ Z = z 0 ( L ‾ n 0 , Q , λ 0 ) P_{\Theta\mid Z=z_0}^{(\overline{L}_{n_0},Q,\lambda_0)} P Θ ∣ Z = z 0 ( L n 0 , Q , λ 0 ) is an ( L ‾ n 0 , Q , λ 0 ) (\overline{L}_{n_0},Q,\lambda_0) ( L n 0 , Q , λ 0 ) -Gibbs probability measure, with the function L ‾ n 0 \overline{L}_{n_0} L n 0 defined in (84) .
Proof. The proof follows by observing that under the choice of λ 1 \lambda_1 λ 1 and λ 2 \lambda_2 λ 2 in (112) , the entries of the vector ω = ( ω 1 , ω 2 , … , ω n 0 ) \boldsymbol{\omega}=(\omega_1,\omega_2,\ldots,\omega_{n_0}) ω = ( ω 1 , ω 2 , … , ω n 0 ) in (106) satisfy, for all t ∈ { 1 , 2 , … , n 0 } t\in\{1,2,\ldots,n_0\} t ∈ { 1 , 2 , … , n 0 } ,
ω t = 1 n 0 λ 0 . (115) \omega_t=\frac{1}{n_0\lambda_0}. \tag*{(115)} ω t = n 0 λ 0 1 . (115)
Therefore, for all θ ∈ M \theta\in\mathcal{M} θ ∈ M ,
L ‾ n 0 ( ω , z 0 , θ ) = ∑ t = 1 n 0 1 n 0 λ 0 ℓ ( x 0 , t , y 0 , t , θ ) (116) \overline{L}_{n_0}(\boldsymbol{\omega},\boldsymbol{z}_0,\theta)=\sum_{t=1}^{n_0}\frac{1}{n_0\lambda_0}\ell(x_{0,t},y_{0,t},\theta) \tag*{(116)} L n 0 ( ω , z 0 , θ ) = t = 1 ∑ n 0 n 0 λ 0 1 ℓ ( x 0 , t , y 0 , t , θ ) (116)
= 1 λ 0 L ‾ n 0 ( z 0 , θ ) , (117) =\frac{1}{\lambda_0}\overline{L}_{n_0}(\boldsymbol{z}_0,\theta), \tag*{(117)} = λ 0 1 L n 0 ( z 0 , θ ) , (117)
where the function L ‾ n 0 \overline{L}_{n_0} L n 0 is defined in (101) . The equality in (113) follows from Lemma 18, and the equality in (114) follows from Definition 3, which completes the proof.
Lemma 20 shows a particular choice of the regularization factors λ 1 \lambda_1 λ 1 and λ 2 \lambda_2 λ 2 for which the normalized log-linear combination S 2 S_2 S 2 in (100) is identical to the ( L n 0 , Q , λ 0 ) (L_{n_0},Q,\lambda_0) ( L n 0 , Q , λ 0 ) -Gibbs probability measure trained upon the aggregated dataset z 0 z_0 z 0 . Under this choice, the weighted empirical risk associated with S 2 S_2 S 2 satisfies
L ˉ n 0 ( ω , z 0 , θ ) = 1 λ 0 L n 0 ( z 0 , θ ) . (118) \bar{L}_{n_0}(\boldsymbol{\omega},z_0,\theta)=\frac{1}{\lambda_0}L_{n_0}(z_0,\theta). \tag*{(118)} L ˉ n 0 ( ω , z 0 , θ ) = λ 0 1 L n 0 ( z 0 , θ ) . (118)
Hence, the optimization problem in (108) is equivalent to
{ min P ∈ △ Q ( M ) ∫ L n 0 ( z 0 , θ ) d P ( θ ) + λ 0 D ( P ∥ Q ) , if λ 0 > 0 , max P ∈ △ Q ( M ) ∫ L n 0 ( z 0 , θ ) d P ( θ ) + λ 0 D ( P ∥ Q ) , if λ 0 < 0. (119) \left\{
\begin{aligned}
\min_{P\in\triangle_Q(\mathcal{M})}\int L_{n_0}(z_0,\theta)\,dP(\theta)+\lambda_0D(P\mathbin\Vert Q),&\quad\text{if }\lambda_0>0,\\
\max_{P\in\triangle_Q(\mathcal{M})}\int L_{n_0}(z_0,\theta)\,dP(\theta)+\lambda_0D(P\mathbin\Vert Q),&\quad\text{if }\lambda_0<0.
\end{aligned}
\right. \tag*{(119)} ⎩ ⎨ ⎧ P ∈ △ Q ( M ) min ∫ L n 0 ( z 0 , θ ) d P ( θ ) + λ 0 D ( P ∥ Q ) , P ∈ △ Q ( M ) max ∫ L n 0 ( z 0 , θ ) d P ( θ ) + λ 0 D ( P ∥ Q ) , if λ 0 > 0 , if λ 0 < 0. (119)
where equivalence follows by multiplying the objective function in (108) by λ 0 \lambda_0 λ 0 . Thus, the choice of λ 1 \lambda_1 λ 1 and λ 2 \lambda_2 λ 2 in Lemma 20 makes all the data points in the aggregated dataset z 0 z_0 z 0 contribute uniformly to the empirical risk and yields the same Gibbs probability measure as that obtained by training directly upon z 0 z_0 z 0 with reference measure Q Q Q and regularization factor λ 0 \lambda_0 λ 0 .
When the datasets z 1 z_1 z 1 and z 2 z_2 z 2 share common data points, the contributions induced by the two Gibbs probability measures are combined. The following lemma makes this effect explicit.
Lemma 21. Assume that the datasets z 1 z_1 z 1 and z 2 z_2 z 2 in the normalized log-linear combination S 2 S_2 S 2 in (100) satisfy
z 1 = ( z 1 , 1 , z 1 , 2 ) ∈ Z n 1 , 1 × Z n 1 , 2 , and (120) z_1=(z_{1,1},z_{1,2})\in\mathcal{Z}^{n_{1,1}}\times\mathcal{Z}^{n_{1,2}},\quad\text{and} \tag*{(120)} z 1 = ( z 1 , 1 , z 1 , 2 ) ∈ Z n 1 , 1 × Z n 1 , 2 , and (120) z 2 = ( z 2 , 1 , z 1 , 2 ) ∈ Z n 2 , 1 × Z n 1 , 2 , (121) z_2=(z_{2,1},z_{1,2})\in\mathcal{Z}^{n_{2,1}}\times\mathcal{Z}^{n_{1,2}}, \tag*{(121)} z 2 = ( z 2 , 1 , z 1 , 2 ) ∈ Z n 2 , 1 × Z n 1 , 2 , (121) and
n 1 = n 1 , 1 + n 1 , 2 ; and (122) n_1=n_{1,1}+n_{1,2};\quad\text{and} \tag*{(122)} n 1 = n 1 , 1 + n 1 , 2 ; and (122) n 2 = n 2 , 1 + n 1 , 2 , (123) n_2=n_{2,1}+n_{1,2}, \tag*{(123)} n 2 = n 2 , 1 + n 1 , 2 , (123) where all the data points common to z 1 z_1 z 1 and z 2 z_2 z 2 are in z 1 , 2 z_{1,2} z 1 , 2 . Then, the ( L ˉ n 0 ( ω , ⋅ , ⋅ ) , Q , 1 ) (\bar{L}_{n_0}(\boldsymbol{\omega},\cdot,\cdot),Q,1) ( L ˉ n 0 ( ω , ⋅ , ⋅ ) , Q , 1 ) -Gibbs probability measure in (110) is such that the vector ω = ( ω 1 , ω 2 , … , ω n 0 ) \boldsymbol{\omega}=(\omega_1,\omega_2,\ldots,\omega_{n_0}) ω = ( ω 1 , ω 2 , … , ω n 0 ) satisfies, for all t ∈ { 1 , 2 , … , n 0 } t\in\{1,2,\ldots,n_0\} t ∈ { 1 , 2 , … , n 0 } ,
ω t = { α 1 n 1 λ 1 , if t ≤ n 1 , 1 , α 1 n 1 λ 1 + α 2 n 2 λ 2 , if n 1 , 1 < t ≤ n 1 , α 2 n 2 λ 2 , if n 1 < t ≤ n 1 + n 2 , 1 , 0 , if n 1 + n 2 , 1 < t ≤ n 0 . (124) \omega_t=
\begin{cases}
\dfrac{\alpha_1}{n_1\lambda_1},&\text{if }t\le n_{1,1},\\
\dfrac{\alpha_1}{n_1\lambda_1}+\dfrac{\alpha_2}{n_2\lambda_2},&\text{if }n_{1,1}<t\le n_1,\\
\dfrac{\alpha_2}{n_2\lambda_2},&\text{if }n_1<t\le n_1+n_{2,1},\\
0,&\text{if }n_1+n_{2,1}<t\le n_0.
\end{cases} \tag*{(124)} ω t = ⎩ ⎨ ⎧ n 1 λ 1 α 1 , n 1 λ 1 α 1 + n 2 λ 2 α 2 , n 2 λ 2 α 2 , 0 , if t ≤ n 1 , 1 , if n 1 , 1 < t ≤ n 1 , if n 1 < t ≤ n 1 + n 2 , 1 , if n 1 + n 2 , 1 < t ≤ n 0 . (124)
Proof. Given a fixed model θ ∈ M \theta\in\mathcal{M} θ ∈ M , the empirical risk functions L n 1 L_{n_1} L n 1 and L n 2 L_{n_2} L n 2 in (84) satisfy
L n 1 ( z 1 , θ ) = n 1 , 1 n 1 L n 1 , 1 ( z 1 , 1 , θ ) + n 1 , 2 n 1 L n 1 , 2 ( z 1 , 2 , θ ) ; (125) L_{n_1}(z_1,\theta)=\frac{n_{1,1}}{n_1}L_{n_{1,1}}(z_{1,1},\theta)+\frac{n_{1,2}}{n_1}L_{n_{1,2}}(z_{1,2},\theta); \tag*{(125)} L n 1 ( z 1 , θ ) = n 1 n 1 , 1 L n 1 , 1 ( z 1 , 1 , θ ) + n 1 n 1 , 2 L n 1 , 2 ( z 1 , 2 , θ ) ; (125)
L n 2 ( z 2 , θ ) = n 2 , 1 n 2 L n 2 , 1 ( z 2 , 1 , θ ) + n 1 , 2 n 2 L n 1 , 2 ( z 1 , 2 , θ ) , (126) L_{n_2}(z_2,\theta)=\frac{n_{2,1}}{n_2}L_{n_{2,1}}(z_{2,1},\theta)+\frac{n_{1,2}}{n_2}L_{n_{1,2}}(z_{1,2},\theta), \tag*{(126)} L n 2 ( z 2 , θ ) = n 2 n 2 , 1 L n 2 , 1 ( z 2 , 1 , θ ) + n 2 n 1 , 2 L n 1 , 2 ( z 1 , 2 , θ ) , (126)
which follows from [12 Lemma 4] . Moreover, the function L ˉ n 0 \bar{L}_{n_0} L ˉ n 0 in (101) satisfies
L ˉ n 0 ( ω , z 0 , θ ) = α 1 λ 1 L n 1 ( z 1 , θ ) + α 2 λ 2 L n 2 ( z 2 , θ ) = α 1 n 1 , 1 n 1 λ 1 L n 1 , 1 ( z 1 , 1 , θ ) + n 1 , 2 α 1 n 1 λ 1 L n 1 , 2 ( z 1 , 2 , θ ) + α 2 n 2 , 1 n 2 λ 2 L n 2 , 1 ( z 2 , 1 , θ ) + n 1 , 2 α 2 n 2 λ 2 L n 1 , 2 ( z 1 , 2 , θ ) = ∑ t = 1 n 1 , 1 α 1 n 1 λ 1 ℓ ( x 0 , t , y 0 , t , θ ) + ∑ t = n 1 , 1 + 1 n 1 ( α 1 n 1 λ 1 + α 2 n 2 λ 2 ) ℓ ( x 0 , t , y 0 , t , θ ) + ∑ t = n 1 + 1 n 1 + n 2 , 1 α 2 n 2 λ 2 ℓ ( x 0 , t , y 0 , t , θ ) + ∑ t = n 1 + n 2 , 1 + 1 n 0 0 ℓ ( x 0 , t , y 0 , t , θ ) = ∑ t = 1 n 0 ω t ℓ ( x 0 , t , y 0 , t , θ ) , (127) \begin{aligned}
\bar{L}_{n_0}(\boldsymbol{\omega},z_0,\theta)
&=\frac{\alpha_1}{\lambda_1}L_{n_1}(z_1,\theta)+\frac{\alpha_2}{\lambda_2}L_{n_2}(z_2,\theta)\\
&=\frac{\alpha_1n_{1,1}}{n_1\lambda_1}L_{n_{1,1}}(z_{1,1},\theta)+\frac{n_{1,2}\alpha_1}{n_1\lambda_1}L_{n_{1,2}}(z_{1,2},\theta)\\
&\quad+\frac{\alpha_2n_{2,1}}{n_2\lambda_2}L_{n_{2,1}}(z_{2,1},\theta)+\frac{n_{1,2}\alpha_2}{n_2\lambda_2}L_{n_{1,2}}(z_{1,2},\theta)\\
&=\sum_{t=1}^{n_{1,1}}\frac{\alpha_1}{n_1\lambda_1}\ell(x_{0,t},y_{0,t},\theta)
+\sum_{t=n_{1,1}+1}^{n_1}\left(\frac{\alpha_1}{n_1\lambda_1}+\frac{\alpha_2}{n_2\lambda_2}\right)\ell(x_{0,t},y_{0,t},\theta)\\
&\quad+\sum_{t=n_1+1}^{n_1+n_{2,1}}\frac{\alpha_2}{n_2\lambda_2}\ell(x_{0,t},y_{0,t},\theta)
+\sum_{t=n_1+n_{2,1}+1}^{n_0}0\ell(x_{0,t},y_{0,t},\theta)\\
&=\sum_{t=1}^{n_0}\omega_t\ell(x_{0,t},y_{0,t},\theta), \tag*{(127)}
\end{aligned} L ˉ n 0 ( ω , z 0 , θ ) = λ 1 α 1 L n 1 ( z 1 , θ ) + λ 2 α 2 L n 2 ( z 2 , θ ) = n 1 λ 1 α 1 n 1 , 1 L n 1 , 1 ( z 1 , 1 , θ ) + n 1 λ 1 n 1 , 2 α 1 L n 1 , 2 ( z 1 , 2 , θ ) + n 2 λ 2 α 2 n 2 , 1 L n 2 , 1 ( z 2 , 1 , θ ) + n 2 λ 2 n 1 , 2 α 2 L n 1 , 2 ( z 1 , 2 , θ ) = t = 1 ∑ n 1 , 1 n 1 λ 1 α 1 ℓ ( x 0 , t , y 0 , t , θ ) + t = n 1 , 1 + 1 ∑ n 1 ( n 1 λ 1 α 1 + n 2 λ 2 α 2 ) ℓ ( x 0 , t , y 0 , t , θ ) + t = n 1 + 1 ∑ n 1 + n 2 , 1 n 2 λ 2 α 2 ℓ ( x 0 , t , y 0 , t , θ ) + t = n 1 + n 2 , 1 + 1 ∑ n 0 0 ℓ ( x 0 , t , y 0 , t , θ ) = t = 1 ∑ n 0 ω t ℓ ( x 0 , t , y 0 , t , θ ) , (127)
where the dataset z 0 z_0 z 0 is defined in (97), and the vector ω = ( ω 1 , ω 2 , … , ω n 0 ) \boldsymbol{\omega}=(\omega_1,\omega_2,\ldots,\omega_{n_0}) ω = ( ω 1 , ω 2 , … , ω n 0 ) satisfies (124) , which completes the proof.
A fundamental observation from Lemma 21 is that, under the decomposition of the datasets in (113), the weight assigned to a data point depends on whether it appears exclusively in one of the two datasets; or whether it is common to both datasets. More specifically, for all i ∈ { 1 , 2 } i\in\{1,2\} i ∈ { 1 , 2 } , the data points in z i , 1 z_{i,1} z i , 1 , which are exclusively in the dataset z i z_i z i , exhibit weight α i n i λ i \frac{\alpha_i}{n_i\lambda_i} n i λ i α i in (124) , and thus their contribution is exclusively controlled by α i \alpha_i α i and the parameters of the Gibbs probability measure P Θ ∣ Z = z i ( L n i , Q , λ i ) P^{(L_{n_i},Q,\lambda_i)}_{\Theta\mid Z=z_i} P Θ ∣ Z = z i ( L n i , Q , λ i ) in (93) or (94). In contrast, the data points in z 1 , 2 \mathcal{z}_{1,2} z 1 , 2 , which are common to the datasets z 1 \mathcal{z}_1 z 1 and z 2 \mathcal{z}_2 z 2 , exhibit weight α 1 n 1 λ 1 + α 2 n 2 λ 2 \frac{\alpha_1}{n_1\lambda_1}+\frac{\alpha_2}{n_2\lambda_2} n 1 λ 1 α 1 + n 2 λ 2 α 2 in (124) . Hence, their contribution to the weighted empirical risk is jointly determined by the coefficients α 1 \alpha_1 α 1 and α 2 \alpha_2 α 2 of the normalized log-linear combination S 2 S_2 S 2 in (100) and by the regularization factors λ 1 \lambda_1 λ 1 and λ 2 \lambda_2 λ 2 of the Gibbs probability measures in (98) and (99) , respectively. This observation plays a crucial role in the remainder of this work. Essentially, the weights of data points that appear in both training datasets, namely z 1 \mathcal{z}_1 z 1 and z 2 \mathcal{z}_2 z 2 can be made equal to zero by choosing the parameters such that α 1 n 1 λ 1 + α 2 n 2 λ 2 = 0 \frac{\alpha_1}{n_1\lambda_1}+\frac{\alpha_2}{n_2\lambda_2}=0 n 1 λ 1 α 1 + n 2 λ 2 α 2 = 0 . Note that the weights α i n i λ i \frac{\alpha_i}{n_i\lambda_i} n i λ i α i of data points that appear exclusively on one training dataset, with i ∈ { 1 , 2 } i \in\{1,2\} i ∈ { 1 , 2 } , cannot be made equal to zero. Interestingly, a zero-weight associated to a data point eliminates its effect on the training process, which resonates with the objective of exact machine unlearning, as discussed later in this work.
Nested Gibbs Algorithms with Data-dependent Reference Measures
This subsection leverages the results on nested Gibbs probability measures presented in Section III-C in the context of supervised machine learning. In particular, in Section III-C, an equivalence between nesting Gibbs probability measures and log-linear combinations of Gibbs measures was established under some specific choice of parameters (Corollary 13). Consequently, under such conditions, all comments in § 5 -B also apply to nested Gibbs probability measures in supervised machine learning. Nonetheless, for completeness, some of those comments are revisited when new relevant observations arise.
In the following, the measure P Θ ∣ Z = z 1 ( L n 1 , Q 1 , λ 1 ) P_{\Theta|\mathcal{Z}=\mathcal{z}_1}^{(L_{n_1,Q_1,\lambda_1})} P Θ∣ Z = z 1 ( L n 1 , Q 1 , λ 1 ) in (98) is nested within P Θ ∣ Z = z 2 ( L n 2 , Q 2 , λ 2 ) P_{\Theta|\mathcal{Z}=\mathcal{z}_2}^{(L_{n_2,Q_2,\lambda_2})} P Θ∣ Z = z 2 ( L n 2 , Q 2 , λ 2 ) in (99) by choosing Q 2 = P Θ ∣ Z = z 1 ( L n 1 , Q 1 , λ 1 ) Q_2=P_{\Theta|\mathcal{Z}=\mathcal{z}_1}^{(L_{n_1,Q_1,\lambda_1})} Q 2 = P Θ∣ Z = z 1 ( L n 1 , Q 1 , λ 1 ) . This yields the nested Gibbs probability measure denoted by
P_{\Theta|\mathcal{Z}=\mathcal{z}_2}^{\left(L_{n_2,P_{\Theta|\mathcal{Z}=\mathcal{z}_1}^{(L_{n_1,Q_1,\lambda_1})},\lambda_2\right)} \in\triangle_{Q_1}(\mathcal{M}),} \tag*{(128)}
which represents a nested Gibbs algorithm. A central observation is that the nested Gibbs probability measure P_{\Theta|\mathcal{Z}=\mathcal{z}_2}^{\left(L_{n_2,P_{\Theta|\mathcal{Z}=\mathcal{z}_1}^{(L_{n_1,Q_1,\lambda_1})},\lambda_2\right)}} in (128) depends explicitly on the training dataset z 2 \mathcal{z}_2 z 2 , whereas its dependence on the training dataset z 1 \mathcal{z}_1 z 1 is carried by the data-dependent reference measure P Θ ∣ Z = z 1 ( L n 1 , Q 1 , λ 1 ) P_{\Theta|\mathcal{Z}=\mathcal{z}_1}^{(L_{n_1,Q_1,\lambda_1})} P Θ∣ Z = z 1 ( L n 1 , Q 1 , λ 1 ) in (98) . The following corollary, which follows from Lemma 18 and Corollary 13, reinforces this observation.
Corollary 22. The nested Gibbs probability measure P_{\Theta|\mathcal{Z}=\mathcal{z}_2}^{\left(L_{n_2,P_{\Theta|\mathcal{Z}=\mathcal{z}_1}^{(L_{n_1,Q_1,\lambda_1})},\lambda_2\right)}} in (128) satisfies, for all θ ∈ supp Q 1 \theta\in\operatorname{supp} Q_1 θ ∈ supp Q 1 ,
\frac{\mathrm{d}P_{\Theta|\mathcal{Z}=\mathcal{z}_2}^{\left(L_{n_2,P_{\Theta|\mathcal{Z}=\mathcal{z}_1}^{(L_{n_1,Q_1,\lambda_1})},\lambda_2\right)}}{\mathrm{d}Q_1}(\theta)=\frac{\exp\left(-\bar{L}_{n_0}(\boldsymbol{\omega},\mathcal{z}_0,\theta)\right)}{\int\exp\left(-\bar{L}_{n_0}(\boldsymbol{\omega},\mathcal{z}_0,\vartheta)\right)\mathrm{d}Q_1(\vartheta)}=\frac{\mathrm{d}P_{\Theta|\mathcal{Z}=\mathcal{z}_0}^{(\bar{L}_{n_0}(\boldsymbol{\omega},\cdot,\cdot),Q_1,1)}}{\mathrm{d}Q_1}(\theta),} \tag*{(129)}
where the function L ˉ n 0 \bar{L}_{n_0} L ˉ n 0 is defined in (101) ; the dataset z 0 \mathcal{z}_0 z 0 is defined in (97); the vector ω = ( ω 1 , ω 2 , … , ω n 0 ) \boldsymbol{\omega}=(\omega_1,\omega_2,\ldots,\omega_{n_0}) ω = ( ω 1 , ω 2 , … , ω n 0 ) satisfies, for all t ∈ { 1 , 2 , … , n 0 } t \in\{1,2,\ldots,n_0\} t ∈ { 1 , 2 , … , n 0 } ,
ω t = { 1 n 1 λ 1 if t ≤ n 1 , 1 n 2 λ 2 if n 1 < t ≤ n 0 ; (130) \omega_t =
\begin{cases}
\frac{1}{n_1\lambda_1} & \text{if } t \le n_1,\\
\frac{1}{n_2\lambda_2} & \text{if } n_1 < t \le n_0;
\end{cases} \tag*{(130)} ω t = { n 1 λ 1 1 n 2 λ 2 1 if t ≤ n 1 , if n 1 < t ≤ n 0 ; (130) and P Θ ∣ Z = z 0 ( L ˉ n 0 ( ω , ⋅ , ⋅ ) , Q 1 , 1 ) P_{\Theta|\mathcal{Z}=\mathcal{z}_0}^{(\bar{L}_{n_0}(\boldsymbol{\omega},\cdot,\cdot),Q_1,1)} P Θ∣ Z = z 0 ( L ˉ n 0 ( ω , ⋅ , ⋅ ) , Q 1 , 1 ) is an ( L ˉ n 0 ( ω , ⋅ , ⋅ ) , Q 1 , 1 ) (\bar{L}_{n_0}(\boldsymbol{\omega},\cdot,\cdot),Q_1,1) ( L ˉ n 0 ( ω , ⋅ , ⋅ ) , Q 1 , 1 ) Gibbs probability measure.
Corollary 22 shows that the nested Gibbs probability measure P_{\Theta|\mathcal{Z}=\mathcal{z}_2}^{\left(L_{n_2,P_{\Theta|\mathcal{Z}=\mathcal{z}_1}^{(L_{n_1,Q_1,\lambda_1})},\lambda_2\right)}} in (128) is itself a Gibbs probability measure whose objective function is a weighted empirical risk, with weights ω 1 , ω 2 , … , ω n 0 \omega_1,\omega_2,\ldots,\omega_{n_0} ω 1 , ω 2 , … , ω n 0 that satisfy (130) , with respect to the aggregated dataset z 0 \mathcal{z}_0 z 0 , i.e., the function L ˉ n 0 ( ω , ⋅ , ⋅ ) \bar{L}_{n_0}(\boldsymbol{\omega},\cdot,\cdot) L ˉ n 0 ( ω , ⋅ , ⋅ ) . In particular, the data points within z 1 \mathcal{z}_1 z 1 are assigned weight 1 n 1 λ 1 \frac{1}{n_1\lambda_1} n 1 λ 1 1 , whereas those within z 2 \mathcal{z}_2 z 2 are assigned weight 1 n 2 λ 2 \frac{1}{n_2\lambda_2} n 2 λ 2 1 . Hence, although the dependence on the dataset z 1 \mathcal{z}_1 z 1 is carried by the reference measure in the nested Gibbs algorithm, the contribution of z 1 \mathcal{z}_1 z 1 appears explicitly in the weighted empirical risk. Beyond this important observation, Corollary 22 also implies the following identity.
Corollary 23. The probability measures P_{\Theta|\mathcal{Z}=\mathcal{z}_2}^{\left(L_{n_2,P_{\Theta|\mathcal{Z}=\mathcal{z}_1}^{(L_{n_1,Q_1,\lambda_1})},\lambda_2\right)}} in (128) and P Θ ∣ Z = z 0 ( L ˉ n 0 ( ω , ⋅ , ⋅ ) , Q 1 , 1 ) P_{\Theta|\mathcal{Z}=\mathcal{z}_0}^{(\bar{L}_{n_0}(\boldsymbol{\omega},\cdot,\cdot),Q_1,1)} P Θ∣ Z = z 0 ( L ˉ n 0 ( ω , ⋅ , ⋅ ) , Q 1 , 1 ) in (129) satisfy
D\left(P_{\Theta|\mathcal{Z}=\mathcal{z}_2}^{\left(L_{n_2,P_{\Theta|\mathcal{Z}=\mathcal{z}_1}^{(L_{n_1,Q_1,\lambda_1})},\lambda_2\right)} \middle\| P_{\Theta|\mathcal{Z}=\mathcal{z}_0}^{(\bar{L}_{n_0}(\boldsymbol{\omega},\cdot,\cdot),Q_1,1)}\right)=0.} \tag*{(131)}
Corollary 23 and Lemma 16 allow establishing connections between the nested Gibbs probability measure P Θ ∣ Z = z 2 ( L ‾ n 2 , P Θ ∣ Z = z 1 ( L n 1 , Q 1 , λ 1 ) , λ 2 ) P_{\Theta\mid Z=z_2}^{\left(\underline{L}_{n_2},P_{\Theta\mid Z=z_1}^{(L_{n_1},\mathcal{Q}_1,\lambda_1)},\lambda_2\right)} P Θ ∣ Z = z 2 ( L n 2 , P Θ ∣ Z = z 1 ( L n 1 , Q 1 , λ 1 ) , λ 2 ) in (128) and two different optimization problems involving a relative entropy. On one hand, from Lemma 16, it follows that the nested Gibbs probability measure P Θ ∣ Z = z 2 ( L ‾ n 2 , P Θ ∣ Z = z 1 ( L n 1 , Q 1 , λ 1 ) , λ 2 ) P_{\Theta\mid Z=z_2}^{\left(\underline{L}_{n_2},P_{\Theta\mid Z=z_1}^{(L_{n_1},\mathcal{Q}_1,\lambda_1)},\lambda_2\right)} P Θ ∣ Z = z 2 ( L n 2 , P Θ ∣ Z = z 1 ( L n 1 , Q 1 , λ 1 ) , λ 2 ) in (128) is the unique solution, if it exists, to
{ min P ∈ Q 1 ( M ) ∫ L ‾ n 2 ( z 2 , θ ) d P ( θ ) + λ 2 D ( P ∥ P Θ ∣ Z = z 1 ( L n 1 , Q 1 , λ 1 ) ) if λ 2 > 0 , max P ∈ Q 1 ( M ) ∫ L ‾ n 2 ( z 2 , θ ) d P ( θ ) + λ 2 D ( P ∥ P Θ ∣ Z = z 1 ( L n 1 , Q 1 , λ 1 ) ) if λ 2 < 0. (132) \left\{
\begin{aligned}
\min_{P\in\mathcal{Q}_1(\mathcal{M})}\ &\int\underline{L}_{n_2}(z_2,\theta)\,dP(\theta)+\lambda_2D\left(P\middle\|P_{\Theta\mid Z=z_1}^{(L_{n_1},\mathcal{Q}_1,\lambda_1)}\right) && \text{if }\lambda_2>0,\\
\max_{P\in\mathcal{Q}_1(\mathcal{M})}\ &\int\underline{L}_{n_2}(z_2,\theta)\,dP(\theta)+\lambda_2D\left(P\middle\|P_{\Theta\mid Z=z_1}^{(L_{n_1},\mathcal{Q}_1,\lambda_1)}\right) && \text{if }\lambda_2<0.
\end{aligned}
\right. \tag*{(132)} ⎩ ⎨ ⎧ P ∈ Q 1 ( M ) min P ∈ Q 1 ( M ) max ∫ L n 2 ( z 2 , θ ) d P ( θ ) + λ 2 D ( P P Θ ∣ Z = z 1 ( L n 1 , Q 1 , λ 1 ) ) ∫ L n 2 ( z 2 , θ ) d P ( θ ) + λ 2 D ( P P Θ ∣ Z = z 1 ( L n 1 , Q 1 , λ 1 ) ) if λ 2 > 0 , if λ 2 < 0. (132)
On the other hand, Lemma 16 shows that the measure P Θ ∣ Z = z 0 ( L ‾ n 0 ( ω , ⋅ , ⋅ ) , Q 1 , 1 ) P_{\Theta\mid Z=z_0}^{(\underline{L}_{n_0}(\omega,\cdot,\cdot),\mathcal{Q}_1,1)} P Θ ∣ Z = z 0 ( L n 0 ( ω , ⋅ , ⋅ ) , Q 1 , 1 ) in (129) is the unique solution, if it exists, to
min P ∈ Q 1 ( M ) ∫ [ L ‾ n 0 ( ω , z 0 , θ ) d P ( θ ) + D ( P ∥ Q 1 ) ] . (133) \min_{P\in\mathcal{Q}_1(\mathcal{M})}\int\left[\underline{L}_{n_0}(\omega,z_0,\theta)\,dP(\theta)+D(P\parallel Q_1)\right]. \tag*{(133)} P ∈ Q 1 ( M ) min ∫ [ L n 0 ( ω , z 0 , θ ) d P ( θ ) + D ( P ∥ Q 1 ) ] . (133)
where the vector of weights ω \omega ω satisfies (130) . Hence, from the fact that the nested Gibbs probability measure P Θ ∣ Z = z 2 ( L ‾ n 2 , P Θ ∣ Z = z 1 ( L n 1 , Q 1 , λ 1 ) , λ 2 ) P_{\Theta\mid Z=z_2}^{\left(\underline{L}_{n_2},P_{\Theta\mid Z=z_1}^{(L_{n_1},\mathcal{Q}_1,\lambda_1)},\lambda_2\right)} P Θ ∣ Z = z 2 ( L n 2 , P Θ ∣ Z = z 1 ( L n 1 , Q 1 , λ 1 ) , λ 2 ) in (128) is identical to the ( L ‾ n 0 ( ω , ⋅ , ⋅ ) , Q 1 , 1 ) \left(\underline{L}_{n_0}(\omega,\cdot,\cdot),\mathcal{Q}_1,1\right) ( L n 0 ( ω , ⋅ , ⋅ ) , Q 1 , 1 ) -Gibbs probability measure P Θ ∣ Z = z 0 ( L ‾ n 0 ( ω , ⋅ , ⋅ ) , Q 1 , 1 ) P_{\Theta\mid Z=z_0}^{(\underline{L}_{n_0}(\omega,\cdot,\cdot),\mathcal{Q}_1,1)} P Θ ∣ Z = z 0 ( L n 0 ( ω , ⋅ , ⋅ ) , Q 1 , 1 ) in (129) , as established by Corollary 23, it follows that
{ P Θ ∣ Z = z 2 ( L ‾ n 2 , P Θ ∣ Z = z 1 ( L n 1 , Q 1 , λ 1 ) , λ 2 ) } = { arg min P ∈ Q 1 ( M ) ∫ L ‾ n 2 ( z 2 , θ ) d P ( θ ) + λ 2 D ( P ∥ P Θ ∣ Z = z 1 ( L n 1 , Q 1 , λ 1 ) ) if λ 2 > 0 , arg max P ∈ Q 1 ( M ) ∫ L ‾ n 2 ( z 2 , θ ) d P ( θ ) + λ 2 D ( P ∥ P Θ ∣ Z = z 1 ( L n 1 , Q 1 , λ 1 ) ) if λ 2 < 0. (134) \left\{
P_{\Theta\mid Z=z_2}^{\left(\underline{L}_{n_2},P_{\Theta\mid Z=z_1}^{(L_{n_1},\mathcal{Q}_1,\lambda_1)},\lambda_2\right)}
\right\}
=
\left\{
\begin{aligned}
\operatorname*{arg\,min}_{P\in\mathcal{Q}_1(\mathcal{M})}\ &\int\underline{L}_{n_2}(z_2,\theta)\,dP(\theta)+\lambda_2D\left(P\middle\|P_{\Theta\mid Z=z_1}^{(L_{n_1},\mathcal{Q}_1,\lambda_1)}\right) && \text{if }\lambda_2>0,\\
\operatorname*{arg\,max}_{P\in\mathcal{Q}_1(\mathcal{M})}\ &\int\underline{L}_{n_2}(z_2,\theta)\,dP(\theta)+\lambda_2D\left(P\middle\|P_{\Theta\mid Z=z_1}^{(L_{n_1},\mathcal{Q}_1,\lambda_1)}\right) && \text{if }\lambda_2<0.
\end{aligned}
\right. \tag*{(134)} ⎩ ⎨ ⎧ P Θ ∣ Z = z 2 ( L n 2 , P Θ ∣ Z = z 1 ( L n 1 , Q 1 , λ 1 ) , λ 2 ) ⎭ ⎬ ⎫ = ⎩ ⎨ ⎧ P ∈ Q 1 ( M ) arg min P ∈ Q 1 ( M ) arg max ∫ L n 2 ( z 2 , θ ) d P ( θ ) + λ 2 D ( P P Θ ∣ Z = z 1 ( L n 1 , Q 1 , λ 1 ) ) ∫ L n 2 ( z 2 , θ ) d P ( θ ) + λ 2 D ( P P Θ ∣ Z = z 1 ( L n 1 , Q 1 , λ 1 ) ) if λ 2 > 0 , if λ 2 < 0. (134)
= arg min P ∈ Q 1 ( M ) ∫ [ L ‾ n 0 ( ω , z 0 , θ ) d P ( θ ) + D ( P ∥ Q 1 ) ] (135) = \operatorname*{arg\,min}_{P\in\mathcal{Q}_1(\mathcal{M})}\int\left[\underline{L}_{n_0}(\omega,z_0,\theta)\,dP(\theta)+D(P\parallel Q_1)\right] \tag*{(135)} = P ∈ Q 1 ( M ) arg min ∫ [ L n 0 ( ω , z 0 , θ ) d P ( θ ) + D ( P ∥ Q 1 ) ] (135)
= { P Θ ∣ Z = z 0 ( L ‾ n 0 ( ω , ⋅ , ⋅ ) , Q 1 , 1 ) } . (136) = \left\{P_{\Theta\mid Z=z_0}^{(\underline{L}_{n_0}(\omega,\cdot,\cdot),\mathcal{Q}_1,1)}\right\}. \tag*{(136)} = { P Θ ∣ Z = z 0 ( L n 0 ( ω , ⋅ , ⋅ ) , Q 1 , 1 ) } . (136)
which provides two equivalent interpretations of the nested Gibbs probability measure P Θ ∣ Z = z 2 ( L ‾ n 2 , P Θ ∣ Z = z 1 ( L n 1 , Q 1 , λ 1 ) , λ 2 ) P_{\Theta\mid Z=z_2}^{\left(\underline{L}_{n_2},P_{\Theta\mid Z=z_1}^{(L_{n_1},\mathcal{Q}_1,\lambda_1)},\lambda_2\right)} P Θ ∣ Z = z 2 ( L n 2 , P Θ ∣ Z = z 1 ( L n 1 , Q 1 , λ 1 ) , λ 2 ) in (128) . On one hand, it is the solution to an empirical-risk optimization with respect to the dataset z 2 z_2 z 2 , with a regularization by a relative entropy with respect to P Θ ∣ Z = z 1 ( L n 1 , Q 1 , λ 1 ) P_{\Theta\mid Z=z_1}^{(L_{n_1},\mathcal{Q}_1,\lambda_1)} P Θ ∣ Z = z 1 ( L n 1 , Q 1 , λ 1 ) . On the other hand, it is the solution to the minimization of a weighted empirical-risk with respect to the aggregated dataset z 0 z_0 z 0 , with a regularization by relative entropy with respect to the reference measure Q 1 Q_1 Q 1 .
The following corollary, which follows from Corollary 22, considers a particular choice of the regularization factors λ 1 \lambda_1 λ 1 and λ 2 \lambda_2 λ 2 that leads to a case in which all the data points in the aggregated dataset z 0 z_0 z 0 exhibit the same weight in the weighted empirical risk in (133) .
Corollary 24. Consider a real λ 0 ∈ R ∖ { 0 } \lambda_0\in\mathbb{R}\setminus\{0\} λ 0 ∈ R ∖ { 0 } . Assume that the regularization factors λ 1 \lambda_1 λ 1 and λ 2 \lambda_2 λ 2 satisfy
λ 1 = n 0 λ 0 n 1 and λ 2 = n 0 λ 0 n 2 . (137) \lambda_1=\frac{n_0\lambda_0}{n_1}
\qquad\text{and}\qquad
\lambda_2=\frac{n_0\lambda_0}{n_2}. \tag*{(137)} λ 1 = n 1 n 0 λ 0 and λ 2 = n 2 n 0 λ 0 . (137) Then, the nested Gibbs probability measure P Θ ∣ Z = z 2 ( L ‾ n 2 , P Θ ∣ Z = z 1 ( L n 1 , Q 1 , λ 1 ) , λ 2 ) P_{\Theta\mid Z=z_2}^{\left(\underline{L}_{n_2},P_{\Theta\mid Z=z_1}^{(L_{n_1},\mathcal{Q}_1,\lambda_1)},\lambda_2\right)} P Θ ∣ Z = z 2 ( L n 2 , P Θ ∣ Z = z 1 ( L n 1 , Q 1 , λ 1 ) , λ 2 ) in (128) satisfies, for all θ ∈ supp Q 1 \theta\in\operatorname{supp}\mathcal{Q}_1 θ ∈ supp Q 1 ,
d P Θ ∣ Z = z 2 ( L ‾ n 2 , P Θ ∣ Z = z 1 ( L n 1 , Q 1 , λ 1 ) , λ 2 ) d Q 1 ( θ ) = exp ( − 1 λ 0 L ‾ n 0 ( z 0 , θ ) ) ∫ exp ( − 1 λ 0 L ‾ n 0 ( z 0 , ν ) ) d Q 1 ( ν ) (138) \frac{dP_{\Theta\mid Z=z_2}^{\left(\underline{L}_{n_2},P_{\Theta\mid Z=z_1}^{(L_{n_1},\mathcal{Q}_1,\lambda_1)},\lambda_2\right)}}{d\mathcal{Q}_1}(\theta)
=
\frac{\exp\left(-\frac{1}{\lambda_0}\underline{L}_{n_0}(z_0,\theta)\right)}
{\int\exp\left(-\frac{1}{\lambda_0}\underline{L}_{n_0}(z_0,\nu)\right)d\mathcal{Q}_1(\nu)} \tag*{(138)} d Q 1 d P Θ ∣ Z = z 2 ( L n 2 , P Θ ∣ Z = z 1 ( L n 1 , Q 1 , λ 1 ) , λ 2 ) ( θ ) = ∫ exp ( − λ 0 1 L n 0 ( z 0 , ν ) ) d Q 1 ( ν ) exp ( − λ 0 1 L n 0 ( z 0 , θ ) ) (138) = d P Θ ∣ Z = z 0 ( L ‾ n 0 , Q 1 , λ 0 ) d Q 1 ( θ ) . (139) =
\frac{dP_{\Theta\mid Z=z_0}^{(\underline{L}_{n_0},\mathcal{Q}_1,\lambda_0)}}{d\mathcal{Q}_1}(\theta). \tag*{(139)} = d Q 1 d P Θ ∣ Z = z 0 ( L n 0 , Q 1 , λ 0 ) ( θ ) . (139) where the probability measure P Θ ∣ Z = z 0 ( L ‾ n 0 , Q 1 , λ 0 ) P_{\Theta\mid Z=z_0}^{(\underline{L}_{n_0},\mathcal{Q}_1,\lambda_0)} P Θ ∣ Z = z 0 ( L n 0 , Q 1 , λ 0 ) is an ( L ‾ n 0 , Q 1 , λ 0 ) (\underline{L}_{n_0},\mathcal{Q}_1,\lambda_0) ( L n 0 , Q 1 , λ 0 ) -Gibbs probability measure, with the function L ‾ n 0 \underline{L}_{n_0} L n 0 defined in (84) .
Corollary 24 shows a particular choice of the regularization factors λ 1 \lambda_1 λ 1 and λ 2 \lambda_2 λ 2 for which the nested Gibbs probability measure in (128) is identical to the ( L n 0 , Q 1 , λ 0 ) (L_{n_0},Q_1,\lambda_0) ( L n 0 , Q 1 , λ 0 ) -Gibbs probability measure trained upon the aggregated dataset z 0 \mathbf{z}_0 z 0 . Under this choice, the weighted empirical risk associated with the nested Gibbs probability measure satisfies for all θ ∈ M \theta\in\mathcal{M} θ ∈ M ,
L ˉ n 0 ( ω , z 0 , θ ) = 1 λ 0 L n 0 ( z 0 , θ ) . (140) \bar{L}_{n_0}(\boldsymbol{\omega},\mathbf{z}_0,\theta)=\frac{1}{\lambda_0}L_{n_0}(\mathbf{z}_0,\theta). \tag*{(140)} L ˉ n 0 ( ω , z 0 , θ ) = λ 0 1 L n 0 ( z 0 , θ ) . (140)
Hence, the choice of λ 1 \lambda_1 λ 1 and λ 2 \lambda_2 λ 2 in (137) makes all the data points in the aggregated dataset z 0 \mathbf{z}_0 z 0 contribute uniformly to the empirical risk and yields the same Gibbs probability measure as that obtained by training directly upon z 0 \mathbf{z}_0 z 0 with reference measure Q 1 Q_1 Q 1 and regularization factor λ 0 \lambda_0 λ 0 .
The following corollary, which follows from Corollary 22, highlights the fact that when datasets share common data points, those common data points exhibit weights that are dependent on a larger number of parameters, when compared to data points that appear exclusively in one of the two datasets.
Corollary 25. Assume that the datasets z 1 \mathbf{z}_1 z 1 and z 2 \mathbf{z}_2 z 2 satisfy (113), where all the data points common to z 1 \mathbf{z}_1 z 1 and z 2 \mathbf{z}_2 z 2 are in z 1 , 2 \mathbf{z}_{1,2} z 1 , 2 . Then, the nested Gibbs probability measure P Θ ∣ Z = z 2 ( L n 2 , P Θ ∣ Z = z 1 ( L n 1 , Q 1 , λ 1 ) , λ 2 ) P_{\Theta\mid\mathbf{Z}=\mathbf{z}_2}^{(L_{n_2},P_{\Theta\mid\mathbf{Z}=\mathbf{z}_1}^{(L_{n_1},Q_1,\lambda_1)},\lambda_2)} P Θ ∣ Z = z 2 ( L n 2 , P Θ ∣ Z = z 1 ( L n 1 , Q 1 , λ 1 ) , λ 2 ) in (128) is such that the vector ω = ( ω 1 , ω 2 , … , ω n 0 ) \boldsymbol{\omega}=(\omega_1,\omega_2,\ldots,\omega_{n_0}) ω = ( ω 1 , ω 2 , … , ω n 0 ) satisfies, for all t ∈ { 1 , 2 , … , n 0 } t \in\{1,2,\ldots,n_0\} t ∈ { 1 , 2 , … , n 0 } ,
ω t = { 1 n 1 λ 1 if t ≤ n 1 , 1 , 1 n 1 λ 1 + 1 n 2 λ 2 if n 1 , 1 < t ≤ n 1 , 1 n 2 λ 2 if n 1 < t ≤ n 1 + n 2 , 1 , 0 if n 1 + n 2 , 1 < t ≤ n 0 . (141) \omega_t =
\begin{cases}
\frac{1}{n_1\lambda_1} & \text{if } t \le n_{1,1},\\
\frac{1}{n_1\lambda_1}+\frac{1}{n_2\lambda_2} & \text{if } n_{1,1}<t\le n_1,\\
\frac{1}{n_2\lambda_2} & \text{if } n_1<t\le n_1+n_{2,1},\\
0 & \text{if } n_1+n_{2,1}<t\le n_0.
\end{cases} \tag*{(141)} ω t = ⎩ ⎨ ⎧ n 1 λ 1 1 n 1 λ 1 1 + n 2 λ 2 1 n 2 λ 2 1 0 if t ≤ n 1 , 1 , if n 1 , 1 < t ≤ n 1 , if n 1 < t ≤ n 1 + n 2 , 1 , if n 1 + n 2 , 1 < t ≤ n 0 . (141)
Corollary 25 highlights a fundamental observation. In particular, the weight associated to common data points in the weighted empirical-risk optimization with relative-entropy regularization problem in (133) is determined by the regularization factors λ 1 \lambda_1 λ 1 and λ 2 \lambda_2 λ 2 . That is, 1 n 1 λ 1 + 1 n 2 λ 2 \frac{1}{n_1\lambda_1}+\frac{1}{n_2\lambda_2} n 1 λ 1 1 + n 2 λ 2 1 . Alternatively, the weights of data points that appear exclusively on dataset i i i , with i ∈ { 1 , 2 } i \in\{1,2\} i ∈ { 1 , 2 } , are determined by λ i \lambda_i λ i , that is, 1 n i λ i \frac{1}{n_i\lambda_i} n i λ i 1 . Hence, nesting causes the contributions associated with a data point appearing in both training datasets to accumulate in the resulting weighted empirical risk.
More importantly, the weight of data points appearing in both datasets can be set to satisfy
1 n 1 λ 1 + 1 n 2 λ 2 = 0 , (142) \frac{1}{n_1\lambda_1}+\frac{1}{n_2\lambda_2}=0, \tag*{(142)} n 1 λ 1 1 + n 2 λ 2 1 = 0 , (142)
whereas the weights of data points appearing exclusively on one dataset z i \mathbf{z}_i z i satisfy 1 n i λ i ≠ 0 \frac{1}{n_i\lambda_i}\ne0 n i λ i 1 = 0 . Thus, the regularization factors can be chosen not only to modulate the contribution of specific data points, but also to cancel the contribution of data points common to both datasets. This property is leveraged in the following section in the context of exact machine unlearning.
Exact Machine Unlearning
This section leverages the properties of nested Gibbs probability measures and normalized log-linear combinations of Gibbs probability measures established in Section III, together with their weighted empirical-risk interpretations in supervised machine learning developed in § 5 , to study exact machine unlearning. In particular, § 5 -B shows that, for normalized log-linear combinations, the weights of the resulting weighted empirical risk are jointly determined by the coefficients of the combination and the regularization factors. Alternatively, § 5 -C shows that nesting Gibbs algorithms induces a weighted empirical risk whose weights are exclusively determined by the corresponding regularization factors. In both cases, exact unlearning is achieved by choosing the corresponding parameters such that the contribution of the dataset to be unlearned is associated with a zero weight.
Consider the datasets z 1 ∈ Z n 1 \mathbf{z}_1\in\mathcal{Z}^{n_1} z 1 ∈ Z n 1 and z 2 ∈ Z n 2 \mathbf{z}_2\in\mathcal{Z}^{n_2} z 2 ∈ Z n 2 in (92), and let their aggregation be the dataset z 0 ∈ Z n 0 ‘ i n x r e f ‘ ( 97 ) ‘ , w i t h \mathbf{z}_0\in\mathcal{Z}^{n_0}` in {xref}`(97)`, with z 0 ∈ Z n 0 ‘ in x r e f ‘ ( 97 ) ‘ , w i t h n_0=n_1+n_2$. Assume that the dataset z 1 \mathbf{z}_1 z 1 is the dataset to be unlearned and that the dataset z 2 \mathbf{z}_2 z 2 is the dataset to be retained. Consider also the ( L n 0 , Q , λ 0 ) (L_{n_0},Q,\lambda_0) ( L n 0 , Q , λ 0 ) -Gibbs probability measure
P Θ ∣ Z = z 0 ( L n 0 , Q , λ 0 ) ∈ Δ Q ( M ) , (143) P_{\Theta\mid\mathbf{Z}=\mathbf{z}_0}^{(L_{n_0},Q,\lambda_0)}\in\Delta_Q(\mathcal{M}), \tag*{(143)} P Θ ∣ Z = z 0 ( L n 0 , Q , λ 0 ) ∈ Δ Q ( M ) , (143)
with λ 0 > 0 \lambda_0>0 λ 0 > 0 , which represents a Gibbs algorithm trained upon the aggregated dataset z 0 \mathbf{z}_0 z 0 .
The objective is to transform the ( L n 0 , Q , λ 0 ) (L_{n_0},Q,\lambda_0) ( L n 0 , Q , λ 0 ) -Gibbs probability measure P Θ ∣ Z = z 0 ( L n 0 , Q , λ 0 ) P_{\Theta\mid\mathbf{Z}=\mathbf{z}_0}^{(L_{n_0},Q,\lambda_0)} P Θ ∣ Z = z 0 ( L n 0 , Q , λ 0 ) into an ( L n 2 , Q , λ 2 ) (L_{n_2},Q,\lambda_2) ( L n 2 , Q , λ 2 ) -Gibbs probability measure
P Θ ∣ Z = z 2 ( L n 2 , Q , λ 2 ) ∈ Δ Q ( M ) , (144) P_{\Theta\mid\mathbf{Z}=\mathbf{z}_2}^{(L_{n_2},Q,\lambda_2)}\in\Delta_Q(\mathcal{M}), \tag*{(144)} P Θ ∣ Z = z 2 ( L n 2 , Q , λ 2 ) ∈ Δ Q ( M ) , (144)
for some λ 2 > 0 \lambda_2>0 λ 2 > 0 , which represents a Gibbs algorithm trained exclusively upon the retained dataset z 2 \mathbf{z}_2 z 2 . This is referred to as exact unlearning and is defined for this particular setting as follows.
Definition 8 (Exact unlearning). Consider the algorithm represented by the ( L n 0 , Q , λ 0 ) (L_{n_0},Q,\lambda_0) ( L n 0 , Q , λ 0 ) -Gibbs probability measure P Θ ∣ Z = z 0 ( L n 0 , Q , λ 0 ) P_{\Theta\mid Z=z_0}^{(L_{n_0},Q,\lambda_0)} P Θ ∣ Z = z 0 ( L n 0 , Q , λ 0 ) in (143) , and the datasets z 1 z_1 z 1 and z 2 z_2 z 2 to be unlearned and retained, respectively. An algorithm P ∈ △ Q ( M ) P \in\triangle_Q(\mathcal{M}) P ∈ △ Q ( M ) is said to achieve exact unlearning of z 1 z_1 z 1 from P Θ ∣ Z = z 0 ( L n 0 , Q , λ 0 ) P_{\Theta\mid Z=z_0}^{(L_{n_0},Q,\lambda_0)} P Θ ∣ Z = z 0 ( L n 0 , Q , λ 0 ) if, for all θ ∈ supp Q \theta\in\operatorname{supp} Q θ ∈ supp Q ,
d P d P Θ ∣ Z = z 2 ( L n 2 , Q , λ 2 ) ( θ ) = 1 , (145) \frac{\mathrm{d}P}{\mathrm{d}P_{\Theta\mid Z=z_2}^{(L_{n_2},Q,\lambda_2)}}(\theta)=1, \tag*{(145)} d P Θ ∣ Z = z 2 ( L n 2 , Q , λ 2 ) d P ( θ ) = 1 , (145) where the probability measure P Θ ∣ Z = z 2 ( L n 2 , Q , λ 2 ) P_{\Theta\mid Z=z_2}^{(L_{n_2},Q,\lambda_2)} P Θ ∣ Z = z 2 ( L n 2 , Q , λ 2 ) is defined in (144) , for some λ 2 > 0 \lambda_2>0 λ 2 > 0 .
The equality in (145) implies that the measures P P P and P Θ ∣ Z = z 2 ( L n 2 , Q , λ 2 ) P_{\Theta\mid Z=z_2}^{(L_{n_2},Q,\lambda_2)} P Θ ∣ Z = z 2 ( L n 2 , Q , λ 2 ) are identical. See [60 Theorem 3] . In the following, two mechanisms for achieving exact unlearning are studied. The first relies on nested Gibbs algorithms, whereas the second relies on normalized log-linear combinations of Gibbs algorithms.
Exact Unlearning with Nested Gibbs Algorithms
This subsection leverages the results on nested Gibbs algorithms developed in Sections III-C and V-C to study unlearning in both centralized and decentralized learning systems.
Centralized Learning Systems
This section shows that exact unlearning of z 1 z_1 z 1 from P Θ ∣ Z = z 0 ( L n 0 , Q , λ 0 ) P_{\Theta\mid Z=z_0}^{(L_{n_0},Q,\lambda_0)} P Θ ∣ Z = z 0 ( L n 0 , Q , λ 0 ) in (143) is achieved by solving the optimization problem
max P ∈ △ Q ( M ) R z 1 ( P ) + λ 1 D ( P ∥ P Θ ∣ Z = z 0 ( L n 0 , Q , λ 0 ) ) , (146) \max_{P\in\triangle_Q(\mathcal{M})} R_{z_1}(P)+\lambda_1D\left(P\middle\|P_{\Theta\mid Z=z_0}^{(L_{n_0},Q,\lambda_0)}\right), \tag*{(146)} P ∈ △ Q ( M ) max R z 1 ( P ) + λ 1 D ( P P Θ ∣ Z = z 0 ( L n 0 , Q , λ 0 ) ) , (146)
for some λ 1 < 0 \lambda_1<0 λ 1 < 0 , where the functional R z 1 R_{z_1} R z 1 is defined in (85) . This setting is associated to a central learning system, capable of using all the dataset z 0 z_0 z 0 for training, that receives the request to unlearn a particular dataset z 1 z_1 z 1 . From Corollary 9, the unique solution to (146) , if it exists, is the nested Gibbs probability measure
P Θ ∣ Z = z 1 ( L n 1 , P Θ ∣ Z = z 0 ( L n 0 , Q , λ 0 ) , λ 1 ) ∈ △ Q ( M ) . (147) P_{\Theta\mid Z=z_1}^{\left(L_{n_1},P_{\Theta\mid Z=z_0}^{(L_{n_0},Q,\lambda_0)},\lambda_1\right)}\in\triangle_Q(\mathcal{M}). \tag*{(147)} P Θ ∣ Z = z 1 ( L n 1 , P Θ ∣ Z = z 0 ( L n 0 , Q , λ 0 ) , λ 1 ) ∈ △ Q ( M ) . (147)
The following corollary of Lemma 21 makes explicit the reweighting induced by the use of the Gibbs probability measure P Θ ∣ Z = z 0 ( L n 0 , Q , λ 0 ) P_{\Theta\mid Z=z_0}^{(L_{n_0},Q,\lambda_0)} P Θ ∣ Z = z 0 ( L n 0 , Q , λ 0 ) in (143) as reference measure.
Corollary 26. The ( L n 1 , P Θ ∣ Z = z 0 ( L n 0 , Q , λ 0 ) , λ 1 ) \left(L_{n_1},P_{\Theta\mid Z=z_0}^{(L_{n_0},Q,\lambda_0)},\lambda_1\right) ( L n 1 , P Θ ∣ Z = z 0 ( L n 0 , Q , λ 0 ) , λ 1 ) -Gibbs probability measure P Θ ∣ Z = z 1 ( L n 1 , P Θ ∣ Z = z 0 ( L n 0 , Q , λ 0 ) , λ 1 ) P_{\Theta\mid Z=z_1}^{\left(L_{n_1},P_{\Theta\mid Z=z_0}^{(L_{n_0},Q,\lambda_0)},\lambda_1\right)} P Θ ∣ Z = z 1 ( L n 1 , P Θ ∣ Z = z 0 ( L n 0 , Q , λ 0 ) , λ 1 ) in (147) , with λ 0 > 0 \lambda_0>0 λ 0 > 0 and λ 1 ∈ R ∖ { 0 } \lambda_1\in\mathbb{R}\setminus\{0\} λ 1 ∈ R ∖ { 0 } , satisfies, for all θ ∈ supp Q \theta\in\operatorname{supp}Q θ ∈ supp Q ,
d P Θ ∣ Z = z 1 ( L n 1 , P Θ ∣ Z = z 0 ( L n 0 , Q , λ 0 ) , λ 1 ) d Q ( θ ) = exp ( − ( 1 λ 1 + n 1 n 0 λ 0 ) L n 1 ( z 1 , θ ) − n 2 n 0 λ 0 L n 2 ( z 2 , θ ) ) ∫ exp ( − ( 1 λ 1 + n 1 n 0 λ 0 ) L n 1 ( z 1 , ν ) − n 2 n 0 λ 0 L n 2 ( z 2 , ν ) ) d Q ( ν ) . (148) \frac{\mathrm{d}P_{\Theta\mid Z=z_1}^{\left(L_{n_1},P_{\Theta\mid Z=z_0}^{(L_{n_0},Q,\lambda_0)},\lambda_1\right)}}{\mathrm{d}Q}(\theta)=\frac{\exp\left(-\left(\frac{1}{\lambda_1}+\frac{n_1}{n_0\lambda_0}\right)L_{n_1}(z_1,\theta)-\frac{n_2}{n_0\lambda_0}L_{n_2}(z_2,\theta)\right)}{\int\exp\left(-\left(\frac{1}{\lambda_1}+\frac{n_1}{n_0\lambda_0}\right)L_{n_1}(z_1,\nu)-\frac{n_2}{n_0\lambda_0}L_{n_2}(z_2,\nu)\right)\mathrm{d}Q(\nu)}. \tag*{(148)} d Q d P Θ ∣ Z = z 1 ( L n 1 , P Θ ∣ Z = z 0 ( L n 0 , Q , λ 0 ) , λ 1 ) ( θ ) = ∫ exp ( − ( λ 1 1 + n 0 λ 0 n 1 ) L n 1 ( z 1 , ν ) − n 0 λ 0 n 2 L n 2 ( z 2 , ν ) ) d Q ( ν ) exp ( − ( λ 1 1 + n 0 λ 0 n 1 ) L n 1 ( z 1 , θ ) − n 0 λ 0 n 2 L n 2 ( z 2 , θ ) ) . (148)
Corollary 26 shows that the use of the Gibbs probability measure P Θ ∣ Z = z 0 ( L n 0 , Q , λ 0 ) P_{\Theta\mid Z=z_0}^{(L_{n_0},Q,\lambda_0)} P Θ ∣ Z = z 0 ( L n 0 , Q , λ 0 ) as reference measure induces a weighted combination of the empirical risks associated with the datasets z 1 z_1 z 1 and z 2 z_2 z 2 . More precisely, the coefficient of the empirical risk induced by the dataset to be unlearned is 1 λ 1 + n 1 n 0 λ 0 \frac{1}{\lambda_1}+\frac{n_1}{n_0\lambda_0} λ 1 1 + n 0 λ 0 n 1 , whereas the coefficient of the empirical risk induced by the dataset to be retained is n 2 n 0 λ 0 \frac{n_2}{n_0\lambda_0} n 0 λ 0 n 2 . Hence, exact unlearning is obtained by choosing λ 1 \lambda_1 λ 1 such that the first coefficient is zero in (148) . The following corollary shows that a specific choice of λ 1 \lambda_1 λ 1 and λ 2 \lambda_2 λ 2 makes the ( L n 1 , P Θ ∣ Z = z 0 ( L n 0 , Q , λ 0 ) , λ 1 ) \left(L_{n_1},P_{\Theta\mid Z=z_0}^{(L_{n_0},Q,\lambda_0)},\lambda_1\right) ( L n 1 , P Θ ∣ Z = z 0 ( L n 0 , Q , λ 0 ) , λ 1 ) -Gibbs probability measure in (147) identical to the probability measure P Θ ∣ Z = z 2 ( L n 2 , Q , λ 2 ) P_{\Theta\mid Z=z_2}^{(L_{n_2},Q,\lambda_2)} P Θ ∣ Z = z 2 ( L n 2 , Q , λ 2 ) in (144) , in the sense of [60 Theorem 3] .
Corollary 27. Assume that
λ 1 = − n 0 n 1 λ 0 ; λ 2 = n 0 n 2 λ 0 , (149) \lambda_1=-\frac{n_0}{n_1}\lambda_0;\qquad\lambda_2=\frac{n_0}{n_2}\lambda_0, \tag*{(149)} λ 1 = − n 1 n 0 λ 0 ; λ 2 = n 2 n 0 λ 0 , (149) for some λ 0 > 0 \lambda_0>0 λ 0 > 0 . Then, the ( L n 1 , P Θ ∣ Z = z 0 ( L n 0 , Q , λ 0 ) , λ 1 ) \left(L_{n_1},P_{\Theta\mid Z=z_0}^{(L_{n_0},Q,\lambda_0)},\lambda_1\right) ( L n 1 , P Θ ∣ Z = z 0 ( L n 0 , Q , λ 0 ) , λ 1 ) -Gibbs probability measure in (147) and the ( L n 2 , Q , λ 2 ) (L_{n_2},Q,\lambda_2) ( L n 2 , Q , λ 2 ) -Gibbs probability measure in (144) satisfy, for all θ ∈ supp Q \theta\in\operatorname{supp}Q θ ∈ supp Q ,
d P Θ ∣ Z = z 1 ( L n 1 , P Θ ∣ Z = z 0 ( L n 0 , Q , λ 0 ) , λ 1 ) d P Θ ∣ Z = z 2 ( L n 2 , Q , λ 2 ) ( θ ) = 1. (150) \frac{\mathrm{d}P_{\Theta\mid Z=z_1}^{\left(L_{n_1},P_{\Theta\mid Z=z_0}^{(L_{n_0},Q,\lambda_0)},\lambda_1\right)}}{\mathrm{d}P_{\Theta\mid Z=z_2}^{(L_{n_2},Q,\lambda_2)}}(\theta)=1. \tag*{(150)} d P Θ ∣ Z = z 2 ( L n 2 , Q , λ 2 ) d P Θ ∣ Z = z 1 ( L n 1 , P Θ ∣ Z = z 0 ( L n 0 , Q , λ 0 ) , λ 1 ) ( θ ) = 1. (150)
Corollary 27 shows that the probability measure P Θ ∣ Z = z 0 ( L n 0 , Q , λ 0 ) P_{\Theta\mid Z=z_0}^{(L_{n_0},Q,\lambda_0)} P Θ ∣ Z = z 0 ( L n 0 , Q , λ 0 ) in (136) can be transformed into P Θ ∣ Z = z 2 ( L n 2 , Q , λ 2 ) P_{\Theta\mid Z=z_2}^{(L_{n_2},Q,\lambda_2)} P Θ ∣ Z = z 2 ( L n 2 , Q , λ 2 ) in (137) via the optimization problem in (139), achieving exact unlearning in the sense of Definition 8. An important observation is that only the dataset to be unlearned, i.e., z 1 z_1 z 1 , is explicitly used in the optimization problem in (139). The dependence on the dataset to be retained, i.e., z 2 z_2 z 2 , is carried by the reference measure P Θ ∣ Z = z 0 ( L n 0 , Q , λ 0 ) P_{\Theta\mid Z=z_0}^{(L_{n_0},Q,\lambda_0)} P Θ ∣ Z = z 0 ( L n 0 , Q , λ 0 ) , which represents the Gibbs algorithm from which z 1 z_1 z 1 is to be unlearned. Hence, unlearning z 1 z_1 z 1 is achieved by training a new Gibbs algorithm upon the dataset to be unlearned while using the original Gibbs algorithm P Θ ∣ Z = z 0 ( L n 0 , Q , λ 0 ) P_{\Theta\mid Z=z_0}^{(L_{n_0},Q,\lambda_0)} P Θ ∣ Z = z 0 ( L n 0 , Q , λ 0 ) as reference measure. Under the scaling in (142), the resulting nested Gibbs probability measure coincides with the Gibbs probability measure that would have been obtained by training exclusively upon the retained dataset z 2 z_2 z 2 .
Another consequence of Corollary 27, together with the uniqueness of the solutions implied by Lemma 16, is
arg max P ∈ △ Q ( M ) R z 1 ( P ) − n 0 n 1 λ 0 D ( P ∥ P Θ ∣ Z = z 0 ( L n 0 , Q , λ 0 ) ) = arg min P ∈ △ Q ( M ) R z 2 ( P ) + n 0 n 2 λ 0 D ( P ∥ Q ) . (151) \begin{aligned}
\underset{P\in\triangle_Q(\mathcal{M})}{\arg\max}\;R_{z_1}(P)-\frac{n_0}{n_1}\lambda_0D\left(P\mathbin{\|}P_{\Theta\mid Z=z_0}^{(L_{n_0},Q,\lambda_0)}\right)
&=\underset{P\in\triangle_Q(\mathcal{M})}{\arg\min}\;R_{z_2}(P)+\frac{n_0}{n_2}\lambda_0D(P\mathbin{\|}Q). \tag*{(151)}
\end{aligned} P ∈ △ Q ( M ) arg max R z 1 ( P ) − n 1 n 0 λ 0 D ( P ∥ P Θ ∣ Z = z 0 ( L n 0 , Q , λ 0 ) ) = P ∈ △ Q ( M ) arg min R z 2 ( P ) + n 2 n 0 λ 0 D ( P ∥ Q ) . (151)
From the perspective of Lemma 17, when Q Q Q is a probability measure, the equality in (144) admits a geometric representation in terms of the relative-entropy neighborhoods in (60). Under the assumption in (142), consider the values
γ 1 ≜ D ( P Θ ∣ Z = z 0 ( L n 0 , Q , λ 0 ) ∥ Q ) , (152) \gamma_1\triangleq D\left(P_{\Theta\mid Z=z_0}^{(L_{n_0},Q,\lambda_0)}\mathbin{\|}Q\right), \tag*{(152)} γ 1 ≜ D ( P Θ ∣ Z = z 0 ( L n 0 , Q , λ 0 ) ∥ Q ) , (152)
γ 2 ≜ D ( P Θ ∣ Z = z 1 ( L n 1 , P Θ ∣ Z = z 0 ( L n 0 , Q , λ 0 ) , λ 1 ) ∥ P Θ ∣ Z = z 0 ( L n 0 , Q , λ 0 ) ) , (153) \gamma_2\triangleq D\left(P_{\Theta\mid Z=z_1}^{(L_{n_1},P_{\Theta\mid Z=z_0}^{(L_{n_0},Q,\lambda_0)},\lambda_1)}\mathbin{\|}P_{\Theta\mid Z=z_0}^{(L_{n_0},Q,\lambda_0)}\right), \tag*{(153)} γ 2 ≜ D ( P Θ ∣ Z = z 1 ( L n 1 , P Θ ∣ Z = z 0 ( L n 0 , Q , λ 0 ) , λ 1 ) ∥ P Θ ∣ Z = z 0 ( L n 0 , Q , λ 0 ) ) , (153)
η ≜ D ( P Θ ∣ Z = z 2 ( L n 2 , Q , λ 2 ) ∥ Q ) . (154) \eta\triangleq D\left(P_{\Theta\mid Z=z_2}^{(L_{n_2},Q,\lambda_2)}\mathbin{\|}Q\right). \tag*{(154)} η ≜ D ( P Θ ∣ Z = z 2 ( L n 2 , Q , λ 2 ) ∥ Q ) . (154)
Consider the sets G Q , γ 1 G_{Q,\gamma_1} G Q , γ 1 , G Q , η G_{Q,\eta} G Q , η , and G P Θ ∣ Z = z 0 ( L n 0 , Q , λ 0 ) , γ 2 G_{P_{\Theta\mid Z=z_0}^{(L_{n_0},Q,\lambda_0)},\gamma_2} G P Θ ∣ Z = z 0 ( L n 0 , Q , λ 0 ) , γ 2 defined as in (60). Using this notation, it follows from Corollary 27 that the nested Gibbs probability measure in (140) is identical to the Gibbs probability measure in (137). Hence, the equality in (144) implies that the probability measure that minimizes the expected empirical risk with respect to the dataset to be retained within G Q , η G_{Q,\eta} G Q , η is identical to the probability measure that maximizes the expected empirical risk with respect to the dataset to be unlearned within G P Θ ∣ Z = z 0 ( L n 0 , Q , λ 0 ) , γ 2 G_{P_{\Theta\mid Z=z_0}^{(L_{n_0},Q,\lambda_0)},\gamma_2} G P Θ ∣ Z = z 0 ( L n 0 , Q , λ 0 ) , γ 2 . More precisely,
{ P Θ ∣ Z = z 2 ( L n 2 , Q , λ 2 ) } = arg min P ∈ G Q , η R z 2 ( P ) (155) \left\{P_{\Theta\mid Z=z_2}^{(L_{n_2},Q,\lambda_2)}\right\}
=\underset{P\in G_{Q,\eta}}{\arg\min}\;R_{z_2}(P) \tag*{(155)} { P Θ ∣ Z = z 2 ( L n 2 , Q , λ 2 ) } = P ∈ G Q , η arg min R z 2 ( P ) (155)
= arg max P ∈ G P Θ ∣ Z = z 0 ( L n 0 , Q , λ 0 ) , γ 2 R z 1 ( P ) (156) =\underset{P\in G_{P_{\Theta\mid Z=z_0}^{(L_{n_0},Q,\lambda_0)},\gamma_2}}{\arg\max}\;R_{z_1}(P) \tag*{(156)} = P ∈ G P Θ ∣ Z = z 0 ( L n 0 , Q , λ 0 ) , γ 2 arg max R z 1 ( P ) (156)
= { P Θ ∣ Z = z 1 ( L n 1 , P Θ ∣ Z = z 0 ( L n 0 , Q , λ 0 ) , λ 1 ) } . (157) =\left\{P_{\Theta\mid Z=z_1}^{(L_{n_1},P_{\Theta\mid Z=z_0}^{(L_{n_0},Q,\lambda_0)},\lambda_1)}\right\}. \tag*{(157)} = { P Θ ∣ Z = z 1 ( L n 1 , P Θ ∣ Z = z 0 ( L n 0 , Q , λ 0 ) , λ 1 ) } . (157)
Figure 10 provides a geometric representation of (148). The figure shows the sets G Q , γ 1 G_{Q,\gamma_1} G Q , γ 1 , G Q , η G_{Q,\eta} G Q , η , and G P Θ ∣ Z = z 0 ( L n 0 , Q , λ 0 ) , γ 2 G_{P_{\Theta\mid Z=z_0}^{(L_{n_0},Q,\lambda_0)},\gamma_2} G P Θ ∣ Z = z 0 ( L n 0 , Q , λ 0 ) , γ 2 . The original Gibbs probability measure P Θ ∣ Z = z 0 ( L n 0 , Q , λ 0 ) P_{\Theta\mid Z=z_0}^{(L_{n_0},Q,\lambda_0)} P Θ ∣ Z = z 0 ( L n 0 , Q , λ 0 ) in (136) is the minimizer of the expected empirical risk with respect to z 0 z_0 z 0 within G Q , γ 1 G_{Q,\gamma_1} G Q , γ 1 and is the reference measure used to form the set G P Θ ∣ Z = z 0 ( L n 0 , Q , λ 0 ) , γ 2 G_{P_{\Theta\mid Z=z_0}^{(L_{n_0},Q,\lambda_0)},\gamma_2} G P Θ ∣ Z = z 0 ( L n 0 , Q , λ 0 ) , γ 2 . The nested Gibbs probability measure in (140) and the Gibbs probability measure in (137) are represented by the same point, taking into account that they are identical under the assumption in (142). Note that in this representation, it has been assumed that γ 1 < η \gamma_1<\eta γ 1 < η , which justifies the inclusion G Q , γ 1 ⊆ G Q , η G_{Q,\gamma_1}\subseteq G_{Q,\eta} G Q , γ 1 ⊆ G Q , η .
Finally, the parameter λ 0 > 0 \lambda_0>0 λ 0 > 0 remains free and can be further optimized, for instance with the objective of improving the generalization properties of the resulting Gibbs algorithm. See, for instance, [7 , 18 , 19 , 20 , 23 ] and the references therein.
Decentralized Learning Systems
The centralized exact-unlearning mechanism in the preceding subsection requires the availability of a Gibbs probability measure trained upon the entire dataset from which a subset is to be unlearned. In a decentralized learning system, the local datasets are distributed among the clients and such a probability measure is not obtained by directly training upon their aggregation. The purpose of this subsection is first to show that the nested Gibbs construction developed in Section V-C can produce the same Gibbs probability measure as in a centralized system through a peer-to-peer communication protocol without explicitly aggregating the local datasets. Once this centralized-equivalent Gibbs probability measure is available, the exact-unlearning mechanism established in Corollary 27 can be applied to the dataset of any client to be unlearned.
Figure 10. Geometric representation of exact unlearning under the assumption in (142), with γ 1 < η \gamma_1 < \eta γ 1 < η . The neighborhoods G Q , γ 1 G_{Q,\gamma_1} G Q , γ 1 , G Q , η G_{Q,\eta} G Q , η , and G P Θ ∣ Z = z 0 ( L n 0 , Q , λ 0 ) , γ 2 G_{P_{\Theta\mid Z=z_0}^{(L_{n_0},Q,\lambda_0)},\gamma_2} G P Θ ∣ Z = z 0 ( L n 0 , Q , λ 0 ) , γ 2 are defined as in (60), with γ 1 \gamma_1 γ 1 , γ 2 \gamma_2 γ 2 , and η \eta η in (145), (146), and (147), respectively. The Gibbs probability measure in (137) and the nested Gibbs probability measure in (140) coincide and satisfy (148).
Consider a decentralized learning system in which K ∈ N K \in\mathbb{N} K ∈ N clients collaboratively tune their learning algorithms by communicating with each other. Assume that all clients use the same model space M ⊆ R d \mathcal{M} \subseteq\mathbb{R}^{d} M ⊆ R d , for some d ∈ N d \in\mathbb{N} d ∈ N , and the same data space Z ≜ X × Y \mathcal{Z} \triangleq\mathcal{X} \times\mathcal{Y} Z ≜ X × Y . For all k ∈ { 1 , 2 , … , K } k \in\{1,2,\ldots,K\} k ∈ { 1 , 2 , … , K } , let the training dataset available at client k k k be denoted by
z k ≜ ( ( x k , 1 , y k , 1 ) , ( x k , 2 , y k , 2 ) , … , ( x k , n k , y k , n k ) ) ∈ Z n k , (158) z_k \triangleq\left((x_{k,1},y_{k,1}),(x_{k,2},y_{k,2}),\ldots,(x_{k,n_k},y_{k,n_k})\right) \in\mathcal{Z}^{n_k}, \tag*{(158)} z k ≜ ( ( x k , 1 , y k , 1 ) , ( x k , 2 , y k , 2 ) , … , ( x k , n k , y k , n k ) ) ∈ Z n k , (158)
for some n k ∈ N n_k \in\mathbb{N} n k ∈ N . The aggregation of all local datasets is denoted by
z 0 ≜ ( z 1 , z 2 , … , z K ) (159) z_0 \triangleq(z_1,z_2,\ldots,z_K) \tag*{(159)} z 0 ≜ ( z 1 , z 2 , … , z K ) (159)
= ( ( x 0 , 1 , y 0 , 1 ) , ( x 0 , 2 , y 0 , 2 ) , … , ( x 0 , n 0 , y 0 , n 0 ) ) ∈ Z n 0 , (160) = \left((x_{0,1},y_{0,1}),(x_{0,2},y_{0,2}),\ldots,(x_{0,n_0},y_{0,n_0})\right) \in\mathcal{Z}^{n_0}, \tag*{(160)} = ( ( x 0 , 1 , y 0 , 1 ) , ( x 0 , 2 , y 0 , 2 ) , … , ( x 0 , n 0 , y 0 , n 0 ) ) ∈ Z n 0 , (160)
where
n 0 ≜ ∑ k = 1 K n k . (161) n_0 \triangleq\sum_{k=1}^{K} n_k. \tag*{(161)} n 0 ≜ k = 1 ∑ K n k . (161)
Consider, for all k ∈ { 1 , 2 , … , K } k \in\{1,2,\ldots,K\} k ∈ { 1 , 2 , … , K } , the ( L n k , Q k , λ k ) (L_{n_k},Q_k,\lambda_k) ( L n k , Q k , λ k ) -Gibbs probability measure
P Θ ∣ Z = z k ( L n k , Q k , λ k ) ∈ △ ( M ) , (162) P_{\Theta\mid Z=z_k}^{(L_{n_k},Q_k,\lambda_k)} \in\triangle(\mathcal{M}), \tag*{(162)} P Θ ∣ Z = z k ( L n k , Q k , λ k ) ∈ △ ( M ) , (162)
with λ k > 0 \lambda_k > 0 λ k > 0 , which represents the Gibbs algorithm of client k k k trained upon its local dataset z k z_k z k in (158) . The corresponding reference measure Q k Q_k Q k is specified recursively as
Q k = { Q 1 , if k = 1 , P Θ ∣ Z = z k − 1 ( L n k − 1 , Q k − 1 , λ k − 1 ) , if k ≥ 2 , (163) Q_k =
\begin{cases}
Q_1, & \text{if } k=1,\\
P_{\Theta\mid Z=z_{k-1}}^{(L_{n_{k-1}},Q_{k-1},\lambda_{k-1})}, & \text{if } k \ge2,
\end{cases} \tag*{(163)} Q k = { Q 1 , P Θ ∣ Z = z k − 1 ( L n k − 1 , Q k − 1 , λ k − 1 ) , if k = 1 , if k ≥ 2 , (163)
which implies that, for all k ∈ { 1 , 2 , … , K − 1 } k \in\{1,2,\ldots,K-1\} k ∈ { 1 , 2 , … , K − 1 } , the Gibbs probability measure produced by client k k k becomes the reference measure used by client k + 1 k+1 k + 1 . More specifically, from Definition 3, for all k ∈ { 1 , 2 , … , K } k \in\{1,2,\ldots,K\} k ∈ { 1 , 2 , … , K } and for all θ ∈ supp Q 1 \theta\in\operatorname{supp} Q_1 θ ∈ supp Q 1 , it follows that
d P Θ ∣ Z = z k ( L n k , Q k , λ k ) d Q k ( θ ) = exp ( − 1 λ k L n k ( z k , θ ) ) ∫ exp ( − 1 λ k L n k ( z k , ν ) ) d Q k ( ν ) . (164) \frac{\mathrm{d}P_{\Theta\mid Z=z_k}^{(L_{n_k},Q_k,\lambda_k)}}{\mathrm{d}Q_k}(\theta)
=
\frac{\exp\left(-\frac{1}{\lambda_k}L_{n_k}(z_k,\theta)\right)}
{\int\exp\left(-\frac{1}{\lambda_k}L_{n_k}(z_k,\nu)\right)\mathrm{d}Q_k(\nu)}. \tag*{(164)} d Q k d P Θ ∣ Z = z k ( L n k , Q k , λ k ) ( θ ) = ∫ exp ( − λ k 1 L n k ( z k , ν ) ) d Q k ( ν ) exp ( − λ k 1 L n k ( z k , θ ) ) . (164)
which shows that the ( L n k , Q k , λ k ) (L_{n_k},Q_k,\lambda_k) ( L n k , Q k , λ k ) -Gibbs probability measure in (155) is a nested Gibbs probability measure in the sense of Definition 6. From Lemma 16, for all k ∈ { 1 , 2 , … , K } k \in\{1,2,\ldots,K\} k ∈ { 1 , 2 , … , K } , such a Gibbs probability measure is the unique solution to the optimization problem
min P ∈ △ Q k ( M ) R z k ( P ) + λ k D ( P ∥ Q k ) . (165) \min_{P \in\triangle_{Q_k}(\mathcal{M})} R_{z_k}(P) + \lambda_k D(P \mathbin{\|} Q_k). \tag*{(165)} P ∈ △ Q k ( M ) min R z k ( P ) + λ k D ( P ∥ Q k ) . (165)
where the functional R z k R_{z_k} R z k is defined in (85) . Equivalently, from Lemma 17, it is also the unique solution to
min P ∈ △ Q k ( M ) R z k ( P ) s.t. D ( P ∥ Q k ) ≤ D ( P Θ ∣ Z = z k ( L n k , Q k , λ k ) ∥ Q k ) . (166) \min_{P\in\triangle_{\mathcal{Q}_k}(\mathcal{M})} R_{z_k}(P)
\quad\text{s.t.}\quad D(P\parallel Q_k)\leq D\left(P_{\Theta\mid Z=z_k}^{(L_{n_k},Q_k,\lambda_k)}\parallel Q_k\right). \tag*{(166)} P ∈ △ Q k ( M ) min R z k ( P ) s.t. D ( P ∥ Q k ) ≤ D ( P Θ ∣ Z = z k ( L n k , Q k , λ k ) ∥ Q k ) . (166)
Such a nested Gibbs probability measure is obtained through the forward–backward peer-to-peer communication protocol depicted in Figure 11 and described hereunder.
Figure 11. Nested structure induced by the forward–backward peer-to-peer communication protocol: for all k ∈ { 1 , 2 , … , K − 1 } k \in\{1,2,\ldots,K-1\} k ∈ { 1 , 2 , … , K − 1 } , the Gibbs probability measure produced by client k k k becomes the reference measure used by client k + 1 k+1 k + 1 .
a) Communication protocol: Figure 11 depicts the forward–backward peer-to-peer communication protocol used in this section. In the forward direction (blue arrows), for all k ∈ { 1 , 2 , … , K − 1 } k \in\{1,2,\ldots,K-1\} k ∈ { 1 , 2 , … , K − 1 } , client k k k transmits to client k + 1 k+1 k + 1 the ( L n k , Q k , λ k ) (L_{n_k},Q_k,\lambda_k) ( L n k , Q k , λ k ) -Gibbs probability measure P Θ ∣ Z = z k ( L n k , Q k , λ k ) P_{\Theta\mid Z=z_k}^{(L_{n_k},Q_k,\lambda_k)} P Θ ∣ Z = z k ( L n k , Q k , λ k ) in (155), obtained as the solution to the optimization problem in (165) . Client k + 1 k+1 k + 1 adopts this transmitted measure as its reference measure Q k + 1 Q_{k+1} Q k + 1 in (165) . This choice induces a nested structure of the reference measure, as Q k + 1 = P Θ ∣ Z = z k ( L n k , Q k , λ k ) Q_{k+1}=P_{\Theta\mid Z=z_k}^{(L_{n_k},Q_k,\lambda_k)} Q k + 1 = P Θ ∣ Z = z k ( L n k , Q k , λ k ) , for all k ∈ { 1 , 2 , … , K − 1 } k \in\{1,2,\ldots,K-1\} k ∈ { 1 , 2 , … , K − 1 } . This is formalized later in Corollary 28 . The backward direction (red arrows) disseminates the final Gibbs probability measure. More specifically, once client K K K has computed its ( L n K , Q K , λ K ) (L_{n_K},Q_K,\lambda_K) ( L n K , Q K , λ K ) -Gibbs probability measure P Θ ∣ Z = z K ( L n K , Q K , λ K ) P_{\Theta\mid Z=z_K}^{(L_{n_K},Q_K,\lambda_K)} P Θ ∣ Z = z K ( L n K , Q K , λ K ) in (155), this measure is transmitted back along the chain, from client K K K to client K − 1 K-1 K − 1 , then from client K − 1 K-1 K − 1 to client K − 2 K-2 K − 2 , and so on until client 1. This backward transmission provides all clients with access to the same final Gibbs algorithm.
b) Nested algorithms: The following corollary is a direct consequence of Theorem 11, applied to the nested structure induced by (163) .
Corollary 28. For all k ∈ { 1 , 2 , … , K } k \in\{1,2,\ldots,K\} k ∈ { 1 , 2 , … , K } , let the probability measure P Θ ∣ Z = z k ( L n k , Q k , λ k ) P_{\Theta\mid Z=z_k}^{(L_{n_k},Q_k,\lambda_k)} P Θ ∣ Z = z k ( L n k , Q k , λ k ) be defined as in (155), with the reference measures satisfying (163) . Then, for all θ ∈ supp Q 1 \theta\in\operatorname{supp} Q_1 θ ∈ supp Q 1 ,
d P Θ ∣ Z = z k ( L n k , Q k , λ k ) d Q 1 ( θ ) = exp ( − ∑ j = 1 k 1 λ j L n j ( z j , θ ) ) ∫ exp ( − ∑ j = 1 k 1 λ j L n j ( z j , ν ) ) d Q 1 ( ν ) . (167) \frac{\mathrm{d}P_{\Theta\mid Z=z_k}^{(L_{n_k},Q_k,\lambda_k)}}{\mathrm{d}Q_1}(\theta)
=
\frac{\exp\left(-\sum_{j=1}^{k}\frac{1}{\lambda_j}L_{n_j}(z_j,\theta)\right)}
{\int\exp\left(-\sum_{j=1}^{k}\frac{1}{\lambda_j}L_{n_j}(z_j,\nu)\right)\mathrm{d}Q_1(\nu)}. \tag*{(167)} d Q 1 d P Θ ∣ Z = z k ( L n k , Q k , λ k ) ( θ ) = ∫ exp ( − ∑ j = 1 k λ j 1 L n j ( z j , ν ) ) d Q 1 ( ν ) exp ( − ∑ j = 1 k λ j 1 L n j ( z j , θ ) ) . (167)
The choice of reference measure Q k Q_k Q k in (163) induces a nested structure. Under this structure, the training performed by client k k k uses only its local dataset, while the influence of the previous clients’ datasets is carried through the reference measure Q k Q_k Q k . Corollary 28 makes this dependence explicit: although the reference measures are recursively defined, the Radon–Nikodym derivative in (167) depends on the datasets z 1 , z 2 , … , z k z_1,z_2,\ldots,z_k z 1 , z 2 , … , z k through the weighted sum of their empirical risks, with coefficients 1 λ 1 , 1 λ 2 , … , 1 λ k \frac{1}{\lambda_1},\frac{1}{\lambda_2},\ldots,\frac{1}{\lambda_k} λ 1 1 , λ 2 1 , … , λ k 1 , respectively. Moreover, from Theorem 12, the probability measure P Θ ∣ Z = z k ( L n k , Q k , λ k ) P_{\Theta\mid Z=z_k}^{(L_{n_k},Q_k,\lambda_k)} P Θ ∣ Z = z k ( L n k , Q k , λ k ) in (155) is the unique solution, if it exists, to the following optimization problem:
min P ∈ △ Q 1 ( M ) ∫ ( ∑ j = 1 k 1 λ j L n j ( z j , θ ) ) d P ( θ ) + D ( P ∥ Q 1 ) . (168) \min_{P\in\triangle_{\mathcal{Q}_1}(\mathcal{M})}
\int\left(\sum_{j=1}^{k}\frac{1}{\lambda_j}L_{n_j}(z_j,\theta)\right)\mathrm{d}P(\theta)+D(P\parallel Q_1). \tag*{(168)} P ∈ △ Q 1 ( M ) min ∫ ( j = 1 ∑ k λ j 1 L n j ( z j , θ ) ) d P ( θ ) + D ( P ∥ Q 1 ) . (168)
Hence, the probability measure P Θ ∣ Z = z k ( L n k , Q k , λ k ) P_{\Theta\mid Z=z_k}^{(L_{n_k},Q_k,\lambda_k)} P Θ ∣ Z = z k ( L n k , Q k , λ k ) in (155) admits two equivalent representations. On one hand, it is obtained by training client k k k exclusively upon its local dataset z k z_k z k , using the Gibbs probability measure produced by the preceding client as reference measure. On the other hand, it is the solution to the weighted empirical-risk optimization in (168) , which depends explicitly on the datasets of all clients from 1 to k k k and uses the initial reference measure Q 1 Q_1 Q 1 . The following corollary formalizes this observation.
Corollary 29. Under the assumption that, for all k ∈ { 1 , 2 , … , K } k \in\{1,2,\ldots,K\} k ∈ { 1 , 2 , … , K } , the measure Q k Q_k Q k satisfies (163) , the solutions to the optimization problems in (165) and (168) are unique and coincide with the ( L n k , Q k , λ k ) (L_{n_k},Q_k,\lambda_k) ( L n k , Q k , λ k ) -Gibbs probability measure P Θ ∣ Z = z k ( L n k , Q k , λ k ) P_{\Theta\mid Z=z_k}^{(L_{n_k},Q_k,\lambda_k)} P Θ ∣ Z = z k ( L n k , Q k , λ k ) in (167) .
Given k > 1 k>1 k > 1 , the optimization problem in (165) depends exclusively on the local training dataset z k z_k z k . While the reference measure Q k Q_k Q k in (163) depends on the training datasets of the k − 1 k-1 k − 1 previous clients, the dependence is not direct. Solving (165) requires access to the probability measure Q k ∈ △ ( M ) Q_k \in\triangle(\mathcal{M}) Q k ∈ △ ( M ) , but not to the training datasets of all previous clients. This is because, for fixed training datasets, Q k Q_k Q k is simply a probability measure on the model space. In contrast, the optimization problem in (168) depends on the training datasets of client j j j , for all j ∈ { 1 , 2 , … , k } j \in\{1,2,\ldots,k\} j ∈ { 1 , 2 , … , k } , while the reference measure Q 1 Q_1 Q 1 does not depend on any training dataset. The fact that problems (165) and (168) share the same solution unveils an important observation: providing client k k k with a reference measure Q k Q_k Q k of the form in (163) reproduces the effect of having access to the training datasets of the k − 1 k-1 k − 1 previous clients, without transmitting those datasets.
c) Centralized-performance guarantee: An important observation is that a strategic choice of λ 1 , λ 2 , … , λ K \lambda_1,\lambda_2,\ldots,\lambda_K λ 1 , λ 2 , … , λ K in (167) can lead to the same Gibbs probability measure as in a setting in which the training datasets of all clients are available at all clients. This describes a decentralized system whose distributed nature does not prevent achieving the same Gibbs algorithm that one would have obtained if all the training data were available at all clients. The following corollary formalizes this observation.
Corollary 30. Assume that, for all k ∈ { 1 , 2 , … , K } k \in\{1,2,\ldots,K\} k ∈ { 1 , 2 , … , K } ,
λ k = n 0 λ 0 n k , (169) \lambda_k=\frac{n_0\lambda_0}{n_k}, \tag*{(169)} λ k = n k n 0 λ 0 , (169) for some λ 0 > 0 \lambda_0>0 λ 0 > 0 . Consider the measure Q K Q_K Q K satisfying (163) . Then, for all θ ∈ supp Q 1 \theta\in\operatorname{supp} Q_1 θ ∈ supp Q 1 , it follows that
d P Θ ∣ Z = z K ( L n K , Q K , λ K ) d P Θ ∣ Z = z 0 ( L n 0 , Q 1 , λ 0 ) ( θ ) = 1 , (170) \frac{\mathrm{d}P_{\Theta\mid Z=z_K}^{(L_{n_K},Q_K,\lambda_K)}}{\mathrm{d}P_{\Theta\mid Z=z_0}^{(L_{n_0},Q_1,\lambda_0)}}(\theta)=1, \tag*{(170)} d P Θ ∣ Z = z 0 ( L n 0 , Q 1 , λ 0 ) d P Θ ∣ Z = z K ( L n K , Q K , λ K ) ( θ ) = 1 , (170) where the probability measure P Θ ∣ Z = z K ( L n K , Q K , λ K ) P_{\Theta\mid Z=z_K}^{(L_{n_K},Q_K,\lambda_K)} P Θ ∣ Z = z K ( L n K , Q K , λ K ) is defined in (167) , and the measure P Θ ∣ Z = z 0 ( L n 0 , Q 1 , λ 0 ) P_{\Theta\mid Z=z_0}^{(L_{n_0},Q_1,\lambda_0)} P Θ ∣ Z = z 0 ( L n 0 , Q 1 , λ 0 ) satisfies, for all θ ∈ supp Q 1 \theta\in\operatorname{supp} Q_1 θ ∈ supp Q 1 ,
d P Θ ∣ Z = z 0 ( L n 0 , Q 1 , λ 0 ) d Q 1 ( θ ) = exp ( − 1 λ 0 L n 0 ( z 0 , θ ) ) ∫ exp ( − 1 λ 0 L n 0 ( z 0 , ν ) ) d Q 1 ( ν ) . (171) \frac{\mathrm{d}P_{\Theta\mid Z=z_0}^{(L_{n_0},Q_1,\lambda_0)}}{\mathrm{d}Q_1}(\theta)=\frac{\exp\left(-\frac{1}{\lambda_0}L_{n_0}(z_0,\theta)\right)}{\int\exp\left(-\frac{1}{\lambda_0}L_{n_0}(z_0,\nu)\right)\mathrm{d}Q_1(\nu)}. \tag*{(171)} d Q 1 d P Θ ∣ Z = z 0 ( L n 0 , Q 1 , λ 0 ) ( θ ) = ∫ exp ( − λ 0 1 L n 0 ( z 0 , ν ) ) d Q 1 ( ν ) exp ( − λ 0 1 L n 0 ( z 0 , θ ) ) . (171)
Corollary 30 establishes the centralized-performance guarantee required for exact unlearning: under the choice in (169) , the last nested Gibbs probability measure is identical to the Gibbs probability measure that would have been obtained by training directly upon z 0 z_0 z 0 . The relevance of this identity is further highlighted by the following observations. Under the assumptions of Corollary 30, the equality in (169) together with [12 Lemma 4] allow rewriting the optimization problem in (168) as
min P ∈ △ ( M ) [ ∫ L n 0 ( z 0 , θ ) d P ( θ ) + λ 0 D ( P ∥ Q 1 ) ] , (172) \min_{P\in\triangle(\mathcal{M})}\left[\int L_{n_0}(z_0,\theta)\mathrm{d}P(\theta)+\lambda_0D(P\mathbin\Vert Q_1)\right], \tag*{(172)} P ∈ △ ( M ) min [ ∫ L n 0 ( z 0 , θ ) d P ( θ ) + λ 0 D ( P ∥ Q 1 ) ] , (172)
which requires access to the training datasets of all clients. Interestingly, from Lemma 16, it follows that the probability measure P Θ ∣ Z = z 0 ( L n 0 , Q 1 , λ 0 ) P_{\Theta\mid Z=z_0}^{(L_{n_0},Q_1,\lambda_0)} P Θ ∣ Z = z 0 ( L n 0 , Q 1 , λ 0 ) in (171) is the unique solution to (172) .
In the following, assume that Q 1 Q_1 Q 1 is a probability measure and that the regularization factors satisfy (169) . For all k ∈ { 1 , 2 , … , K } k \in\{1,2,\ldots,K\} k ∈ { 1 , 2 , … , K } , consider the values
γ k ≜ D ( P Θ ∣ Z = z k ( L n k , Q k , λ k ) ∥ Q k ) < + ∞ , (173) \gamma_k \triangleq D\left(P_{\Theta\mid Z=z_k}^{(L_{n_k},Q_k,\lambda_k)}\mathbin\Vert Q_k\right)<+\infty, \tag*{(173)} γ k ≜ D ( P Θ ∣ Z = z k ( L n k , Q k , λ k ) ∥ Q k ) < + ∞ , (173)
and the corresponding neighborhoods
G Q k , γ k ≜ { P ∈ △ Q k ( M ) : D ( P ∥ Q k ) ≤ γ k } , (174) \mathcal{G}_{Q_k,\gamma_k}\triangleq\left\{P\in\triangle_{Q_k}(\mathcal{M}):D(P\mathbin\Vert Q_k)\leq\gamma_k\right\}, \tag*{(174)} G Q k , γ k ≜ { P ∈ △ Q k ( M ) : D ( P ∥ Q k ) ≤ γ k } , (174)
which are reminiscent of the set in (60). Taking into account that λ k > 0 \lambda_k>0 λ k > 0 , Lemma 17 implies that the Gibbs probability measure P Θ ∣ Z = z k ( L n k , Q k , λ k ) P_{\Theta\mid Z=z_k}^{(L_{n_k},Q_k,\lambda_k)} P Θ ∣ Z = z k ( L n k , Q k , λ k ) in (155) is the unique minimizer of the expected empirical risk R z k R_{z_k} R z k in (84) within G Q k , γ k \mathcal{G}_{Q_k,\gamma_k} G Q k , γ k .
Similarly, consider the finite relative entropy
η ≜ D ( P Θ ∣ Z = z 0 ( L n 0 , Q 1 , λ 0 ) ∥ Q 1 ) < + ∞ . (175) \eta\triangleq D\left(P_{\Theta\mid Z=z_0}^{(L_{n_0},Q_1,\lambda_0)}\mathbin\Vert Q_1\right)<+\infty. \tag*{(175)} η ≜ D ( P Θ ∣ Z = z 0 ( L n 0 , Q 1 , λ 0 ) ∥ Q 1 ) < + ∞. (175)
Taking into account that λ 0 > 0 \lambda_0>0 λ 0 > 0 , Lemma 17 identifies the probability measure P Θ ∣ Z = z 0 ( L n 0 , Q 1 , λ 0 ) P_{\Theta\mid Z=z_0}^{(L_{n_0},Q_1,\lambda_0)} P Θ ∣ Z = z 0 ( L n 0 , Q 1 , λ 0 ) in (171) as the unique solution to
min P ∈ △ Q 1 ( M ) R z 0 ( P ) (176) \min_{P\in\triangle_{Q_1}(\mathcal{M})} R_{z_0}(P) \tag*{(176)} P ∈ △ Q 1 ( M ) min R z 0 ( P ) (176)
s.t. D ( P ∥ Q 1 ) ≤ η . (177) \text{s.t.}\quad D(P\mathbin\Vert Q_1)\leq\eta. \tag*{(177)} s.t. D ( P ∥ Q 1 ) ≤ η . (177)
that is, the minimizer of the expected empirical risk with respect to the aggregated dataset z 0 z_0 z 0 within G Q 1 , η \mathcal{G}_{Q_1,\eta} G Q 1 , η .
From (170) , the probability measure P Θ ∣ Z = z 0 ( L n 0 , Q 1 , λ 0 ) P_{\Theta\mid Z=z_0}^{(\mathcal{L}_{n_0,Q_1,\lambda_0})} P Θ ∣ Z = z 0 ( L n 0 , Q 1 , λ 0 ) is identical to the Gibbs probability measure P Θ ∣ Z = z K ( L n K , Q K , λ K ) P_{\Theta\mid Z=z_K}^{(\mathcal{L}_{n_K,Q_K,\lambda_K})} P Θ ∣ Z = z K ( L n K , Q K , λ K ) of the form in (155). Hence, after K − 1 K - 1 K − 1 forward messages (blue arrows in Figure 11 ), client K K K obtains the same Gibbs probability measure as a centralized system in which all training datasets are available at client K K K . The backward dissemination (red arrows in Figure 11 ) then provides each client with access to such a Gibbs algorithm.
Figure 12 provides a geometric representation of this recursive nesting for K = 4 K = 4 K = 4 . The points Q k + 1 Q_{k+1} Q k + 1 represent
Figure 12. Geometric representation of recursively nested Gibbs probability measures for K = 4 K = 4 K = 4 , under the assumption in (169) . The neighborhoods G Q k , γ k \mathcal{G}_{Q_k,\gamma_k} G Q k , γ k , with k ∈ { 1 , 2 , … , 4 } k \in\{1, 2, \ldots, 4\} k ∈ { 1 , 2 , … , 4 } , and G Q 1 , η \mathcal{G}_{Q_1,\eta} G Q 1 , η are defined as in (61) , with γ k \gamma_k γ k and η \eta η in (173) and (175) , respectively. The points Q k + 1 Q_{k+1} Q k + 1 represent the Gibbs probability measures in (178) . The final point Q 5 Q_5 Q 5 in (179) coincides with the Gibbs probability measure in (171) obtained by training upon the aggregated dataset z 0 z_0 z 0 .
Q k + 1 = P Θ ∣ Z = z k ( L n k , Q k , λ k ) , k ∈ { 1 , 2 , … , K } , (178) Q_{k+1} = P_{\Theta\mid Z=z_k}^{(\mathcal{L}_{n_k,Q_k,\lambda_k})}, \qquad k \in\{1, 2, \ldots, K\}, \tag*{(178)} Q k + 1 = P Θ ∣ Z = z k ( L n k , Q k , λ k ) , k ∈ { 1 , 2 , … , K } , (178)
where the notation Q K + 1 Q_{K+1} Q K + 1 is also used for the final nested Gibbs probability measure. For each k ∈ { 1 , 2 , … , K − 1 } k \in\{1, 2, \ldots, K - 1\} k ∈ { 1 , 2 , … , K − 1 } , the measure obtained by client k k k becomes the reference measure for client k + 1 k + 1 k + 1 , yielding the sequence Q 1 , Q 2 , … , Q K + 1 Q_1, Q_2, \ldots, Q_{K+1} Q 1 , Q 2 , … , Q K + 1 . The figure shows the corresponding neighborhoods G Q k , γ k \mathcal{G}_{Q_k,\gamma_k} G Q k , γ k in (174) , together with G Q 1 , η \mathcal{G}_{Q_1,\eta} G Q 1 , η , where η \eta η is defined in (175) .
For K = 4 K = 4 K = 4 , the final Gibbs probability measures satisfies
Q 5 = P Θ ∣ Z = z 4 ( L n 4 , Q 4 , λ 4 ) (179) Q_5 = P_{\Theta\mid Z=z_4}^{(\mathcal{L}_{n_4,Q_4,\lambda_4})} \tag*{(179)} Q 5 = P Θ ∣ Z = z 4 ( L n 4 , Q 4 , λ 4 ) (179)
= P Θ ∣ Z = z 0 ( L n 0 , Q 1 , λ 0 ) , (180) = P_{\Theta\mid Z=z_0}^{(\mathcal{L}_{n_0,Q_1,\lambda_0})}, \tag*{(180)} = P Θ ∣ Z = z 0 ( L n 0 , Q 1 , λ 0 ) , (180)
which follows from the equality in (170) . Thus, the measure Q 5 Q_5 Q 5 in (179) represents both the minimizer of R z 4 R_{z_4} R z 4 within G Q 4 , γ 4 \mathcal{G}_{Q_4,\gamma_4} G Q 4 , γ 4 and the minimizer of R z 0 R_{z_0} R z 0 within G Q 1 , η \mathcal{G}_{Q_1,\eta} G Q 1 , η .
Beyond this geometric representation, Lemma 15 relates the reals γ k \gamma_k γ k in (173) and η \eta η in (175) to the variation of the expected empirical risks associated with the local datasets, as shown by the following corollary.
Corollary 31. Under the assumptions of Corollary 30, the reals γ 1 , γ 2 , ⋯ , γ K \gamma_1, \gamma_2, \cdots, \gamma_K γ 1 , γ 2 , ⋯ , γ K in (164) and η \eta η in (166) satisfy
∑ k = 1 K γ k − η = 1 n 0 λ 0 ∑ i = 1 K − 1 n i ( R z i ( P Θ ∣ Z = z 0 ( L n 0 , Q 1 , λ 0 ) ) − R z i ( P Θ ∣ Z = z i ( L n i , Q i , λ i ) ) ) , (181) \sum_{k=1}^{K}\gamma_k - \eta= \frac{1}{n_0\lambda_0}\sum_{i=1}^{K-1} n_i \left(R_{z_i}\left(P_{\Theta\mid Z=z_0}^{(\mathcal{L}_{n_0,Q_1,\lambda_0})}\right) - R_{z_i}\left(P_{\Theta\mid Z=z_i}^{(\mathcal{L}_{n_i,Q_i,\lambda_i})}\right)\right), \tag*{(181)} k = 1 ∑ K γ k − η = n 0 λ 0 1 i = 1 ∑ K − 1 n i ( R z i ( P Θ ∣ Z = z 0 ( L n 0 , Q 1 , λ 0 ) ) − R z i ( P Θ ∣ Z = z i ( L n i , Q i , λ i ) ) ) , (181) where R z i R_{z_i} R z i is defined in (84).
For all i ∈ { 1 , 2 , … , K − 1 } i \in\{1, 2, \ldots, K - 1\} i ∈ { 1 , 2 , … , K − 1 } , the difference on the right-hand side of (181) compares the expected empirical risk on z i z_i z i with respect to the Gibbs probability measure P Θ ∣ Z = z 0 ( L n 0 , Q 1 , λ 0 ) P_{\Theta\mid Z=z_0}^{(\mathcal{L}_{n_0,Q_1,\lambda_0})} P Θ ∣ Z = z 0 ( L n 0 , Q 1 , λ 0 ) in (162), trained upon the aggregated dataset z 0 z_0 z 0 , with that under the Gibbs probability measure P Θ ∣ Z = z i ( L n i , Q i , λ i ) P_{\Theta\mid Z=z_i}^{(\mathcal{L}_{n_i,Q_i,\lambda_i})} P Θ ∣ Z = z i ( L n i , Q i , λ i ) in (155), obtained by client i i i .
The following lemma expresses (181) in terms of the expected empirical risk on the aggregated dataset and the expected empirical risks on the local datasets.
Lemma 32. Under the assumptions of Corollary 31, assume also that the expected empirical risks below are finite. Then,
∑ k = 1 K γ k − η = 1 λ 0 ( R z 0 ( P Θ ∣ Z = z 0 ( L n 0 , Q 1 , λ 0 ) ) − ∑ i = 1 K n i n 0 R z i ( P Θ ∣ Z = z i ( L n i , Q i , λ i ) ) ) . (182) \sum_{k=1}^{K}\gamma_k-\eta=\frac{1}{\lambda_0}\left(R_{\boldsymbol{z}_0}\left(P_{\Theta\mid\boldsymbol{Z}=\boldsymbol{z}_0}^{(L_{n_0},Q_1,\lambda_0)}\right)-\sum_{i=1}^{K}\frac{n_i}{n_0}R_{\boldsymbol{z}_i}\left(P_{\Theta\mid\boldsymbol{Z}=\boldsymbol{z}_i}^{(L_{n_i},Q_i,\lambda_i)}\right)\right). \tag*{(182)} k = 1 ∑ K γ k − η = λ 0 1 ( R z 0 ( P Θ ∣ Z = z 0 ( L n 0 , Q 1 , λ 0 ) ) − i = 1 ∑ K n 0 n i R z i ( P Θ ∣ Z = z i ( L n i , Q i , λ i ) ) ) . (182)
Proof: The proof is presented in Appendix I
Lemma 32 shows that the difference ∑ k = 1 K γ k − η \sum_{k=1}^{K}\gamma_k-\eta ∑ k = 1 K γ k − η is 1 λ 0 \frac{1}{\lambda_0} λ 0 1 times the difference between the expected empirical risk on the aggregated dataset z 0 \boldsymbol{z}_0 z 0 with respect to P Θ ∣ Z = z 0 ( L n 0 , Q 1 , λ 0 ) P_{\Theta\mid\boldsymbol{Z}=\boldsymbol{z}_0}^{(L_{n_0},Q_1,\lambda_0)} P Θ ∣ Z = z 0 ( L n 0 , Q 1 , λ 0 ) in (171) and the weighted sum of the expected empirical risks on the local datasets z i \boldsymbol{z}_i z i with respect to P Θ ∣ Z = z i ( L n i , Q i , λ i ) P_{\Theta\mid\boldsymbol{Z}=\boldsymbol{z}_i}^{(L_{n_i},Q_i,\lambda_i)} P Θ ∣ Z = z i ( L n i , Q i , λ i ) in (155), with weights n i n 0 \frac{n_i}{n_0} n 0 n i . In particular, when these quantities are equal, ∑ k = 1 K γ k = η \sum_{k=1}^{K}\gamma_k=\eta ∑ k = 1 K γ k = η , the training error R z 0 ( P Θ ∣ Z = z 0 ( L n 0 , Q 1 , λ 0 ) ) R_{\boldsymbol{z}_0}\left(P_{\Theta\mid\boldsymbol{Z}=\boldsymbol{z}_0}^{(L_{n_0},Q_1,\lambda_0)}\right) R z 0 ( P Θ ∣ Z = z 0 ( L n 0 , Q 1 , λ 0 ) ) , which is induced by the Gibbs algorithm trained upon the aggregation of all data, i.e., P Θ ∣ Z = z 0 ( L n 0 , Q 1 , λ 0 ) P_{\Theta\mid\boldsymbol{Z}=\boldsymbol{z}_0}^{(L_{n_0},Q_1,\lambda_0)} P Θ ∣ Z = z 0 ( L n 0 , Q 1 , λ 0 ) , is identical to the convex combination of the training error induced by Gibbs algorithms P Θ ∣ Z = z i ( L n i , Q i , λ i ) P_{\Theta\mid\boldsymbol{Z}=\boldsymbol{z}_i}^{(L_{n_i},Q_i,\lambda_i)} P Θ ∣ Z = z i ( L n i , Q i , λ i ) , trained exclusively on local training datasets z i \boldsymbol{z}_i z i , with i ∈ { 1 , 2 , … , K } i\in\{1,2,\ldots,K\} i ∈ { 1 , 2 , … , K } .
d) Exact unlearning: The centralized exact-unlearning result in Corollary 27 requires the availability of the Gibbs probability measure trained upon the aggregated dataset z 0 \boldsymbol{z}_0 z 0 , namely P Θ ∣ Z = z 0 ( L n 0 , Q 1 , λ 0 ) P_{\Theta\mid\boldsymbol{Z}=\boldsymbol{z}_0}^{(L_{n_0},Q_1,\lambda_0)} P Θ ∣ Z = z 0 ( L n 0 , Q 1 , λ 0 ) in (171) . In the decentralized setting, such a Gibbs probability measure can be obtained without explicit data aggregation by using the nested structure induced by the recursive choice of the reference measures in (163) . More specifically, under the assumptions in (169) , Corollary 30 shows that the final Gibbs probability measure obtained through the forward pass of the communication protocol coincides with the Gibbs probability measure that would have been obtained by training upon the aggregated dataset z 0 \boldsymbol{z}_0 z 0 .
Once the Gibbs probability measure in (171) is available, Corollary 27 can be applied in a decentralized manner. Fix j ∈ { 1 , 2 , … , K } j\in\{1,2,\ldots,K\} j ∈ { 1 , 2 , … , K } and consider the deletion request of client j j j , that is, the dataset z j ∈ Z n j \boldsymbol{z}_j\in\mathcal{Z}^{n_j} z j ∈ Z n j is to be unlearned from the Gibbs probability measure P Θ ∣ Z = z 0 ( L n 0 , Q 1 , λ 0 ) P_{\Theta\mid\boldsymbol{Z}=\boldsymbol{z}_0}^{(L_{n_0},Q_1,\lambda_0)} P Θ ∣ Z = z 0 ( L n 0 , Q 1 , λ 0 ) . To do so, client j j j solves the following optimization problem:
max P ∈ △ Q 1 ( M ) R z j ( P ) − n 0 n j λ 0 D ( P ∥ P Θ ∣ Z = z 0 ( L n 0 , Q 1 , λ 0 ) ) . (183) \max_{P\in\triangle_{Q_1}(\mathcal{M})}R_{\boldsymbol{z}_j}(P)-\frac{n_0}{n_j}\lambda_0D\left(P\middle\|P_{\Theta\mid\boldsymbol{Z}=\boldsymbol{z}_0}^{(L_{n_0},Q_1,\lambda_0)}\right). \tag*{(183)} P ∈ △ Q 1 ( M ) max R z j ( P ) − n j n 0 λ 0 D ( P P Θ ∣ Z = z 0 ( L n 0 , Q 1 , λ 0 ) ) . (183)
where the functional R z j R_{\boldsymbol{z}_j} R z j is defined in (85) . This optimization problem requires access only to the dataset z j \boldsymbol{z}_j z j and to the Gibbs probability measure in (171) , and does not require explicit access to the datasets of the retained clients.
From Corollary 27, solving (183) admits an exact-unlearning interpretation: the contribution of the dataset z j \boldsymbol{z}_j z j is canceled, and the updated probability measure coincides with the Gibbs probability measure that would have been obtained by training only upon the datasets of the retained clients. Moreover, such a retained-data Gibbs probability measure can itself be obtained in a decentralized manner by applying again the nested decentralized construction above, but only over the retained clients.
Hence, provided that the Gibbs probability measure associated with the aggregated dataset z 0 \boldsymbol{z}_0 z 0 can be obtained in a decentralized manner through the nested structure induced by (163) , the exact-unlearning mechanism established above extends naturally to decentralized learning systems. More specifically, this extension preserves the same operational advantages: the datasets remain local, the communication protocol only requires transmitting Gibbs probability measures, and exact unlearning is achieved without retraining from scratch upon the retained clients.
The same principle extends to multiple deletion requests of whole local datasets. If J ⊂ { 1 , … , K } \mathcal{J}\subset\{1,\ldots,K\} J ⊂ { 1 , … , K } denotes the set of clients whose training dataset is to be forgotten, the corresponding updates can be performed sequentially, each time using as reference measure the Gibbs probability measure obtained after the previous update. Hence, the exact-unlearning mechanism in Section VI-A1 extends to multiple deletion requests in the decentralized setting of whole or part of local training datasets, provided that the Gibbs probability measure used at each update is obtained through the corresponding nested decentralized construction.
Exact Unlearning with Normalized Log-Linear Combinations
This subsection leverages the results on normalized log-linear combinations of Gibbs algorithms developed in Section V-B. As shown therein, the contribution of a data point to the weighted empirical risk associated with a normalized log-linear combination is jointly determined by the coefficients of the combination and the regularization factors of the corresponding Gibbs probability measures.
Centralized Learning Systems
1) Centralized Learning Systems: Consider the ( L n 1 , Q , λ 1 ) (L_{n_1},Q,\lambda_1) ( L n 1 , Q , λ 1 ) -Gibbs probability measure
P Θ ∣ Z = z 1 ( L n 1 , Q , λ 1 ) ∈ △ Q ( M ) . (184) P_{\Theta\mid\boldsymbol{Z}=\boldsymbol{z}_1}^{(L_{n_1},Q,\lambda_1)}\in\triangle_Q(\mathcal{M}). \tag*{(184)} P Θ ∣ Z = z 1 ( L n 1 , Q , λ 1 ) ∈ △ Q ( M ) . (184)
which represents a Gibbs algorithm exclusively trained upon the dataset to be unlearned z 1 \boldsymbol{z}_1 z 1 .
Consider the normalized log-linear combination of P Θ ∣ Z = z 0 ( L n 0 , Q , λ 0 ) P^{(L_{n_0},Q,\lambda_0)}_{\Theta\mid Z=z_0} P Θ ∣ Z = z 0 ( L n 0 , Q , λ 0 ) in (143) and P Θ ∣ Z = z 1 ( L n 1 , Q , λ 1 ) P^{(L_{n_1},Q,\lambda_1)}_{\Theta\mid Z=z_1} P Θ ∣ Z = z 1 ( L n 1 , Q , λ 1 ) in (184) , with coefficients α 0 ∈ R ∖ { 0 } \alpha_0 \in\mathbb{R}\setminus\{0\} α 0 ∈ R ∖ { 0 } and α 1 ∈ R ∖ { 0 } \alpha_1 \in\mathbb{R}\setminus\{0\} α 1 ∈ R ∖ { 0 } , respectively, denoted by S 2 S_2 S 2 . That is, for all θ ∈ supp Q \theta\in\operatorname{supp} Q θ ∈ supp Q ,
d S 2 d Q ( θ ) = ( d P Θ ∣ Z = z 0 ( L n 0 , Q , λ 0 ) d Q ( θ ) ) α 0 ( d P Θ ∣ Z = z 1 ( L n 1 , Q , λ 1 ) d Q ( θ ) ) α 1 ∫ ( d P Θ ∣ Z = z 0 ( L n 0 , Q , λ 0 ) d Q ( v ) ) α 0 ( d P Θ ∣ Z = z 1 ( L n 1 , Q , λ 1 ) d Q ( v ) ) α 1 d Q ( v ) . (185) \frac{\mathrm{d}S_2}{\mathrm{d}Q}(\theta)=\frac{\left(\frac{\mathrm{d}P^{(L_{n_0},Q,\lambda_0)}_{\Theta\mid Z=z_0}}{\mathrm{d}Q}(\theta)\right)^{\alpha_0}\left(\frac{\mathrm{d}P^{(L_{n_1},Q,\lambda_1)}_{\Theta\mid Z=z_1}}{\mathrm{d}Q}(\theta)\right)^{\alpha_1}}{\int\left(\frac{\mathrm{d}P^{(L_{n_0},Q,\lambda_0)}_{\Theta\mid Z=z_0}}{\mathrm{d}Q}(v)\right)^{\alpha_0}\left(\frac{\mathrm{d}P^{(L_{n_1},Q,\lambda_1)}_{\Theta\mid Z=z_1}}{\mathrm{d}Q}(v)\right)^{\alpha_1}\mathrm{d}Q(v)}. \tag*{(185)} d Q d S 2 ( θ ) = ∫ ( d Q d P Θ ∣ Z = z 0 ( L n 0 , Q , λ 0 ) ( v ) ) α 0 ( d Q d P Θ ∣ Z = z 1 ( L n 1 , Q , λ 1 ) ( v ) ) α 1 d Q ( v ) ( d Q d P Θ ∣ Z = z 0 ( L n 0 , Q , λ 0 ) ( θ ) ) α 0 ( d Q d P Θ ∣ Z = z 1 ( L n 1 , Q , λ 1 ) ( θ ) ) α 1 . (185)
The following corollary of Lemma 21 makes explicit the reweighting induced by this normalized log-linear combination.
Corollary 33. The normalized log-linear combination S 2 S_2 S 2 in (185) satisfies, for all θ ∈ supp Q \theta\in\operatorname{supp} Q θ ∈ supp Q ,
d S 2 d Q ( θ ) = exp ( − ( α 0 n 1 n 0 λ 0 + α 1 λ 1 ) L n 1 ( z 1 , θ ) − α 0 n 2 n 0 λ 0 L n 2 ( z 2 , θ ) ) ∫ exp ( − ( α 0 n 1 n 0 λ 0 + α 1 λ 1 ) L n 1 ( z 1 , v ) − α 0 n 2 n 0 λ 0 L n 2 ( z 2 , v ) ) d Q ( v ) . (186) \frac{\mathrm{d}S_2}{\mathrm{d}Q}(\theta)=\frac{\exp\left(-\left(\frac{\alpha_0n_1}{n_0\lambda_0}+\frac{\alpha_1}{\lambda_1}\right)L_{n_1}(z_1,\theta)-\frac{\alpha_0n_2}{n_0\lambda_0}L_{n_2}(z_2,\theta)\right)}{\int\exp\left(-\left(\frac{\alpha_0n_1}{n_0\lambda_0}+\frac{\alpha_1}{\lambda_1}\right)L_{n_1}(z_1,v)-\frac{\alpha_0n_2}{n_0\lambda_0}L_{n_2}(z_2,v)\right)\mathrm{d}Q(v)}. \tag*{(186)} d Q d S 2 ( θ ) = ∫ exp ( − ( n 0 λ 0 α 0 n 1 + λ 1 α 1 ) L n 1 ( z 1 , v ) − n 0 λ 0 α 0 n 2 L n 2 ( z 2 , v ) ) d Q ( v ) exp ( − ( n 0 λ 0 α 0 n 1 + λ 1 α 1 ) L n 1 ( z 1 , θ ) − n 0 λ 0 α 0 n 2 L n 2 ( z 2 , θ ) ) . (186)
Corollary 33 shows that the normalized log-linear combination S 2 S_2 S 2 induces a linear combination of the empirical risks associated with the datasets z 1 z_1 z 1 and z 2 z_2 z 2 . More precisely, the coefficient of the empirical risk induced by the dataset to be unlearned is
α 0 n 1 n 0 λ 0 + α 1 λ 1 , (187) \frac{\alpha_0n_1}{n_0\lambda_0}+\frac{\alpha_1}{\lambda_1}, \tag*{(187)} n 0 λ 0 α 0 n 1 + λ 1 α 1 , (187)
whereas the coefficient of the empirical risk induced by the dataset to be retained is α 0 n 2 n 0 λ 0 \frac{\alpha_0n_2}{n_0\lambda_0} n 0 λ 0 α 0 n 2 . Hence, exact unlearning is obtained by choosing the parameters such that the first coefficient is zero. The following theorem shows that a joint choice of λ 1 \lambda_1 λ 1 and λ 2 \lambda_2 λ 2 , α 0 \alpha_0 α 0 and α 1 \alpha_1 α 1 , makes the normalized log-linear combination S 2 S_2 S 2 in (185) be identical to the probability measure P Θ ∣ Z = z 2 ( L n 2 , Q , λ 2 ) P^{(L_{n_2},Q,\lambda_2)}_{\Theta\mid Z=z_2} P Θ ∣ Z = z 2 ( L n 2 , Q , λ 2 ) in (144) , in the sense of Definition 8.
Corollary 34. Assume that
λ 1 α 1 = n 0 α 0 n 1 λ 0 , and (188) \frac{\lambda_1}{\alpha_1}=\frac{n_0}{\alpha_0n_1}\lambda_0,\quad\text{and} \tag*{(188)} α 1 λ 1 = α 0 n 1 n 0 λ 0 , and (188) λ 2 = n 0 α 0 n 2 λ 0 , (189) \lambda_2=\frac{n_0}{\alpha_0n_2}\lambda_0, \tag*{(189)} λ 2 = α 0 n 2 n 0 λ 0 , (189) with λ 0 > 0 \lambda_0>0 λ 0 > 0 and λ 2 > 0 \lambda_2>0 λ 2 > 0 . Then, the normalized log-linear combination S 2 S_2 S 2 in (185) and the ( L n 2 , Q , λ 2 ) (L_{n_2},Q,\lambda_2) ( L n 2 , Q , λ 2 ) -Gibbs probability measure P Θ ∣ Z = z 2 ( L n 2 , Q , λ 2 ) P^{(L_{n_2},Q,\lambda_2)}_{\Theta\mid Z=z_2} P Θ ∣ Z = z 2 ( L n 2 , Q , λ 2 ) in (144) satisfy, for all θ ∈ supp Q \theta\in\operatorname{supp} Q θ ∈ supp Q ,
d S 2 d P Θ ∣ Z = z 2 ( L n 2 , Q , λ 2 ) ( θ ) = 1. (190) \frac{\mathrm{d}S_2}{\mathrm{d}P^{(L_{n_2},Q,\lambda_2)}_{\Theta\mid Z=z_2}}(\theta)=1. \tag*{(190)} d P Θ ∣ Z = z 2 ( L n 2 , Q , λ 2 ) d S 2 ( θ ) = 1. (190)
Corollary 34 shows that the probability measure P Θ ∣ Z = z 0 ( L n 0 , Q , λ 0 ) P^{(L_{n_0},Q,\lambda_0)}_{\Theta\mid Z=z_0} P Θ ∣ Z = z 0 ( L n 0 , Q , λ 0 ) in (143) can be transformed into P Θ ∣ Z = z 2 ( L n 2 , Q , λ 2 ) P^{(L_{n_2},Q,\lambda_2)}_{\Theta\mid Z=z_2} P Θ ∣ Z = z 2 ( L n 2 , Q , λ 2 ) in (144) through a normalized log-linear combination with the Gibbs probability measure P Θ ∣ Z = z 1 ( L n 1 , Q , λ 1 ) P^{(L_{n_1},Q,\lambda_1)}_{\Theta\mid Z=z_1} P Θ ∣ Z = z 1 ( L n 1 , Q , λ 1 ) in (184) , achieving exact unlearning in the sense of Definition 8. An important observation is that, besides the original Gibbs probability measure, only the Gibbs probability measure trained upon the dataset to be unlearned, i.e., dataset z 1 z_1 z 1 , is involved in the normalized log-linear combination. The dependence on the dataset to be retained, i.e., dataset z 2 z_2 z 2 , is carried by the original Gibbs probability measure P Θ ∣ Z = z 0 ( L n 0 , Q , λ 0 ) P^{(L_{n_0},Q,\lambda_0)}_{\Theta\mid Z=z_0} P Θ ∣ Z = z 0 ( L n 0 , Q , λ 0 ) . Under the choice in (188) –(189) , the contribution associated with z 1 z_1 z 1 is eliminated due to a zero weight. Alternatively, the coefficient associated with z 2 z_2 z 2 becomes 1 λ 2 \frac{1}{\lambda_2} λ 2 1 . Consequently, the resulting probability measure coincides with the Gibbs probability measure that would have been obtained by training exclusively upon the retained dataset z 2 z_2 z 2 .
Unlike the nested construction, in which the cancellation of the contribution associated with z 1 z_1 z 1 is controlled by the regularization factor of the nested Gibbs probability measure, the normalized log-linear combination provides an additional degree of freedom. More specifically, from (188) , the cancellation is jointly determined by the coefficient α 1 \alpha_1 α 1 and the parameter λ 1 \lambda_1 λ 1 of the Gibbs probability measure trained upon z 1 z_1 z 1 . Interestingly, from (189) , it follows that the parameter α 0 \alpha_0 α 0 controls both the coefficient of the retained dataset, which allows achieving unlearning; and also determines the parameter λ 2 \lambda_2 λ 2 of the resulting Gibbs algorithm obtained after exact unlearning.
Federated Learning
Consider the same collection of K K K local datasets z 1 , z 2 , … , z K z_1,z_2,\ldots,z_K z 1 , z 2 , … , z K in (158) , with aggregation z 0 z_0 z 0 in (159) . Consider a federated learning system in which the K K K clients collaboratively tune their local learning algorithms by communicating with a common server. For all k ∈ { 1 , 2 , … , K } k \in\{1,2,\ldots,K\} k ∈ { 1 , 2 , … , K } , consider the ( L n k , Q k , λ k ) (L_{n_k},Q_k,\lambda_k) ( L n k , Q k , λ k ) -Gibbs probability measure
P Θ ∣ Z = z k ( L n k , Q k , λ k ) ∈ △ Q k ( M ) , (191) P_{\Theta\mid Z=z_k}^{(L_{n_k},Q_k,\lambda_k)} \in\triangle_{Q_k}(\mathcal{M}), \tag*{(191)} P Θ ∣ Z = z k ( L n k , Q k , λ k ) ∈ △ Q k ( M ) , (191)
with λ k > 0 \lambda_k>0 λ k > 0 , which represents the Gibbs algorithm of client k k k trained upon its local dataset z k z_k z k .
a) Communication protocol: Consider the one-shot federated learning system depicted in Figure 13 . In the first communication phase, the clients transmit their local Gibbs algorithms to the server. During the second phase, the server aggregates these local algorithms by means of a normalized log-linear combination and broadcasts the resulting probability measure to all clients.
Figure 13. One-shot client–server communication system. For all k ∈ { 1 , 2 , … , K } k \in\{1,2,\ldots,K\} k ∈ { 1 , 2 , … , K } , the Gibbs algorithm of client k k k , P Θ ∣ Z = z k ( L n k , Q k , λ k ) P_{\Theta\mid Z=z_k}^{(L_{n_k},Q_k,\lambda_k)} P Θ ∣ Z = z k ( L n k , Q k , λ k ) , is defined in (191) , and the probability measure representing the aggregation at the server, S K S_K S K , is defined in (192) .
b) Normalized log-linear combination: The server constructs a probability measure S K ∈ △ Q ( M ) S_K \in\triangle_Q(\mathcal{M}) S K ∈ △ Q ( M ) such that, for all θ ∈ supp Q \theta\in\operatorname{supp} Q θ ∈ supp Q ,
d S K d Q ( θ ) = ∏ k = 1 K ( d P Θ ∣ Z = z k ( L n k , Q k , λ k ) d Q k ( θ ) ) α k ∫ ∏ k = 1 K ( d P Θ ∣ Z = z k ( L n k , Q k , λ k ) d Q k ( ν ) ) α k d Q ( ν ) (192) \frac{\mathrm{d}S_K}{\mathrm{d}Q}(\theta)=\frac{\prod_{k=1}^{K}\left(\frac{\mathrm{d}P_{\Theta\mid Z=z_k}^{(L_{n_k},Q_k,\lambda_k)}}{\mathrm{d}Q_k}(\theta)\right)^{\alpha_k}}{\int\prod_{k=1}^{K}\left(\frac{\mathrm{d}P_{\Theta\mid Z=z_k}^{(L_{n_k},Q_k,\lambda_k)}}{\mathrm{d}Q_k}(\nu)\right)^{\alpha_k}\mathrm{d}Q(\nu)} \tag*{(192)} d Q d S K ( θ ) = ∫ ∏ k = 1 K ( d Q k d P Θ ∣ Z = z k ( L n k , Q k , λ k ) ( ν ) ) α k d Q ( ν ) ∏ k = 1 K ( d Q k d P Θ ∣ Z = z k ( L n k , Q k , λ k ) ( θ ) ) α k (192)
= exp ( − ∑ k = 1 K α k λ k L n k ( z k , θ ) ) ∫ exp ( − ∑ k = 1 K α k λ k L n k ( z k , ν ) ) d Q ( ν ) , (193) =\frac{\exp\left(-\sum_{k=1}^{K}\frac{\alpha_k}{\lambda_k}L_{n_k}(z_k,\theta)\right)}{\int\exp\left(-\sum_{k=1}^{K}\frac{\alpha_k}{\lambda_k}L_{n_k}(z_k,\nu)\right)\mathrm{d}Q(\nu)}, \tag*{(193)} = ∫ exp ( − ∑ k = 1 K λ k α k L n k ( z k , ν ) ) d Q ( ν ) exp ( − ∑ k = 1 K λ k α k L n k ( z k , θ ) ) , (193)
where, for all k ∈ { 1 , 2 , … , K } k \in\{1,2,\ldots,K\} k ∈ { 1 , 2 , … , K } , it is assumed that Q ≪ Q k Q \ll Q_k Q ≪ Q k . The equality in (193) follows from Theorem 7 . Moreover, from Lemma 1, the probability measure S K S_K S K is the unique solution to
min P ∈ △ Q ( M ) ( ∫ ( ∑ k = 1 K α k λ k L n k ( z k , θ ) ) d P ( θ ) + D ( P ∥ Q ) ) . (194) \min_{P\in\triangle_Q(\mathcal{M})}\left(\int\left(\sum_{k=1}^{K}\frac{\alpha_k}{\lambda_k}L_{n_k}(z_k,\theta)\right)\mathrm{d}P(\theta)+D(P\parallel Q)\right). \tag*{(194)} P ∈ △ Q ( M ) min ( ∫ ( k = 1 ∑ K λ k α k L n k ( z k , θ ) ) d P ( θ ) + D ( P ∥ Q ) ) . (194)
Hence, although each client trains exclusively upon its local dataset, the normalized log-linear combination performed at the server induces a weighted empirical risk that depends on the datasets of all clients.
c) Centralized-performance guarantee: A particular choice of the regularization factors and the coefficients of the normalized log-linear combination makes the weighted empirical risk in (194) be identical to the empirical risk induced by the aggregation of all local datasets. The following corollary formalizes this observation.
Corollary 35. Assume that, for all k ∈ { 1 , 2 , … , K } k \in\{1,2,\ldots,K\} k ∈ { 1 , 2 , … , K } ,
λ k = 1 n k and α k = 1 n 0 λ 0 , (195) \lambda_k=\frac{1}{n_k}\quad\text{and}\quad\alpha_k=\frac{1}{n_0\lambda_0}, \tag*{(195)} λ k = n k 1 and α k = n 0 λ 0 1 , (195) for some λ 0 > 0 \lambda_0>0 λ 0 > 0 . Then, for all θ ∈ supp Q \theta\in\operatorname{supp}Q θ ∈ supp Q ,
d S K d Q ( θ ) = exp ( − 1 λ 0 L n 0 ( z 0 , θ ) ) ∫ exp ( − 1 λ 0 L n 0 ( z 0 , ν ) ) d Q ( ν ) (196) \frac{\mathrm{d}S_K}{\mathrm{d}Q}(\theta)=\frac{\exp\left(-\frac{1}{\lambda_0}L_{n_0}(z_0,\theta)\right)}{\int\exp\left(-\frac{1}{\lambda_0}L_{n_0}(z_0,\nu)\right)\mathrm{d}Q(\nu)} \tag*{(196)} d Q d S K ( θ ) = ∫ exp ( − λ 0 1 L n 0 ( z 0 , ν ) ) d Q ( ν ) exp ( − λ 0 1 L n 0 ( z 0 , θ ) ) (196) = d P Θ ∣ Z = z 0 ( L n 0 , Q , λ 0 ) d Q ( θ ) . (197) =\frac{\mathrm{d}P_{\Theta\mid Z=z_0}^{(L_{n_0},Q,\lambda_0)}}{\mathrm{d}Q}(\theta). \tag*{(197)} = d Q d P Θ ∣ Z = z 0 ( L n 0 , Q , λ 0 ) ( θ ) . (197)
Corollary 35 shows that the normalized log-linear combination of the K K K Gibbs algorithms independently obtained by the clients is identical to the Gibbs probability measure that would have been obtained by training directly upon the aggregated dataset z 0 z_0 z 0 . More generally, the relevant condition to observe (196) is for all k ∈ { 1 , 2 , … , K } k\in\{1,2,\ldots,K\} k ∈ { 1 , 2 , … , K } ,
α k λ k = n k n 0 λ 0 . (198) \frac{\alpha_k}{\lambda_k}=\frac{n_k}{n_0\lambda_0}. \tag*{(198)} λ k α k = n 0 λ 0 n k . (198)
The particular choice in Corollary 35 allows each client to choose λ k \lambda_k λ k based only on the size of its local dataset, whereas the server uses the same coefficient α k \alpha_k α k for all clients. Hence, the Gibbs probability measure required as the starting point of the centralized exact-unlearning mechanism is available at the server without explicitly aggregating the local datasets.
d) Exact unlearning: Once the Gibbs probability measure P Θ ∣ Z = z 0 ( L n 0 , Q , λ 0 ) P_{\Theta\mid Z=z_0}^{(L_{n_0},Q,\lambda_0)} P Θ ∣ Z = z 0 ( L n 0 , Q , λ 0 ) is available through the normalized log-linear combination in Corollary 35, the exact-unlearning mechanism developed in Section VI-B can be applied in the federated setting. Fix j ∈ { 1 , 2 , … , K } j\in\{1,2,\ldots,K\} j ∈ { 1 , 2 , … , K } and consider a deletion request from client j j j , that is, the dataset z j ∈ Z n j z_j\in\mathcal{Z}^{n_j} z j ∈ Z n j is to be unlearned from the algorithm P Θ ∣ Z = z 0 ( L n 0 , Q , λ 0 ) P_{\Theta\mid Z=z_0}^{(L_{n_0},Q,\lambda_0)} P Θ ∣ Z = z 0 ( L n 0 , Q , λ 0 ) in (196) . Denote by
z − j ≜ ( z 1 , … , z j − 1 , z j + 1 , … , z K ) ∈ Z n − j , (199) z_{-j}\triangleq(z_1,\ldots,z_{j-1},z_{j+1},\ldots,z_K)\in\mathcal{Z}^{n_{-j}}, \tag*{(199)} z − j ≜ ( z 1 , … , z j − 1 , z j + 1 , … , z K ) ∈ Z n − j , (199)
the aggregation of the datasets to be retained, where
n − j ≜ n 0 − n j . (200) n_{-j}\triangleq n_0-n_j. \tag*{(200)} n − j ≜ n 0 − n j . (200)
The subscript − j -j − j in (199) indicates that the dataset of client j j j is excluded from the aggregation of all datasets, i.e., z 0 z_0 z 0 in (159) .
Consider the normalized log-linear combination of P Θ ∣ Z = z 0 ( L n 0 , Q , λ 0 ) P_{\Theta\mid Z=z_0}^{(L_{n_0},Q,\lambda_0)} P Θ ∣ Z = z 0 ( L n 0 , Q , λ 0 ) , or simply S K S_K S K for short, in (196) and P Θ ∣ Z = z j ( L n j , Q j , λ j ) P_{\Theta\mid Z=z_j}^{(L_{n_j},Q_j,\lambda_j)} P Θ ∣ Z = z j ( L n j , Q j , λ j ) of the form in (86), with coefficients 1 1 1 and − α j -\alpha_j − α j , respectively, denoted by S ~ K , j ∈ △ Q ( M ) \widetilde{S}_{K,j}\in\triangle_Q(\mathcal{M}) S K , j ∈ △ Q ( M ) . That is, for all θ ∈ supp Q \theta\in\operatorname{supp}Q θ ∈ supp Q ,
d S ~ K , j d Q ( θ ) = d P Θ ∣ Z = z 0 ( L n 0 , Q , λ 0 ) d Q ( θ ) ( d P Θ ∣ Z = z j ( L n j , Q j , λ j ) d Q j ( θ ) ) − α j ∫ d P Θ ∣ Z = z 0 ( L n 0 , Q , λ 0 ) d Q ( ν ) ( d P Θ ∣ Z = z j ( L n j , Q j , λ j ) d Q j ( ν ) ) − α j d Q ( ν ) . (201) \frac{\mathrm{d}\widetilde{S}_{K,j}}{\mathrm{d}Q}(\theta)=\frac{\dfrac{\mathrm{d}P_{\Theta\mid Z=z_0}^{(L_{n_0},Q,\lambda_0)}}{\mathrm{d}Q}(\theta)\left(\dfrac{\mathrm{d}P_{\Theta\mid Z=z_j}^{(L_{n_j},Q_j,\lambda_j)}}{\mathrm{d}Q_j}(\theta)\right)^{-\alpha_j}}{\int\dfrac{\mathrm{d}P_{\Theta\mid Z=z_0}^{(L_{n_0},Q,\lambda_0)}}{\mathrm{d}Q}(\nu)\left(\dfrac{\mathrm{d}P_{\Theta\mid Z=z_j}^{(L_{n_j},Q_j,\lambda_j)}}{\mathrm{d}Q_j}(\nu)\right)^{-\alpha_j}\mathrm{d}Q(\nu)}. \tag*{(201)} d Q d S K , j ( θ ) = ∫ d Q d P Θ ∣ Z = z 0 ( L n 0 , Q , λ 0 ) ( ν ) d Q j d P Θ ∣ Z = z j ( L n j , Q j , λ j ) ( ν ) − α j d Q ( ν ) d Q d P Θ ∣ Z = z 0 ( L n 0 , Q , λ 0 ) ( θ ) d Q j d P Θ ∣ Z = z j ( L n j , Q j , λ j ) ( θ ) − α j . (201)
The following corollary of Theorem 7 makes explicit the reweighting induced by this normalized log-linear combination.
Corollary 36. The normalized log-linear combination S ~ K , j \widetilde{S}_{K,j} S K , j in (201) satisfies, for all θ ∈ supp Q \theta\in\operatorname{supp}Q θ ∈ supp Q ,
d S ~ K , j d Q ( θ ) = exp ( − ( n j n 0 λ 0 − α j λ j ) L n j ( z j , θ ) − n − j n 0 λ 0 L n − j ( z − j , θ ) ) ∫ exp ( − ( n j n 0 λ 0 − α j λ j ) L n j ( z j , ν ) − n − j n 0 λ 0 L n − j ( z − j , ν ) ) d Q ( ν ) . (202) \frac{\mathrm{d}\widetilde{S}_{K,j}}{\mathrm{d}Q}(\theta)=\frac{\exp\left(-\left(\frac{n_j}{n_0\lambda_0}-\frac{\alpha_j}{\lambda_j}\right)L_{n_j}(z_j,\theta)-\frac{n_{-j}}{n_0\lambda_0}L_{n_{-j}}(z_{-j},\theta)\right)}{\int\exp\left(-\left(\frac{n_j}{n_0\lambda_0}-\frac{\alpha_j}{\lambda_j}\right)L_{n_j}(z_j,\nu)-\frac{n_{-j}}{n_0\lambda_0}L_{n_{-j}}(z_{-j},\nu)\right)\mathrm{d}Q(\nu)}. \tag*{(202)} d Q d S K , j ( θ ) = ∫ exp ( − ( n 0 λ 0 n j − λ j α j ) L n j ( z j , ν ) − n 0 λ 0 n − j L n − j ( z − j , ν ) ) d Q ( ν ) exp ( − ( n 0 λ 0 n j − λ j α j ) L n j ( z j , θ ) − n 0 λ 0 n − j L n − j ( z − j , θ ) ) . (202) where the dataset z − j z_{-j} z − j is defined in (199) and n − j n_{-j} n − j is defined in (200) .
Corollary 36 shows that the normalized log-linear combination S ~ K , j \widetilde{S}_{K,j} S K , j in (201) induces a linear combination of the empirical risks associated with the datasets z j z_j z j in (158) and z − j z_{-j} z − j in (199) . More precisely, the coefficient of the empirical risk induced by the dataset to be unlearned is
n j n 0 λ 0 − α j λ j . (203) \frac{n_j}{n_0\lambda_0}-\frac{\alpha_j}{\lambda_j}. \tag*{(203)} n 0 λ 0 n j − λ j α j . (203)
whereas the coefficient of the empirical risk induced by the dataset to be retained is n − j n 0 λ 0 \frac{n_{-j}}{n_0\lambda_0} n 0 λ 0 n − j . Under the condition in (198) ,
n j n 0 λ 0 − α j λ j = 0. (204) \frac{n_j}{n_0\lambda_0}-\frac{\alpha_j}{\lambda_j}=0. \tag*{(204)} n 0 λ 0 n j − λ j α j = 0. (204)
Hence, the contribution associated with the dataset z j z_j z j is eliminated due to a zero weight. The following corollary shows that the normalized log-linear combination S ~ K , j \widetilde{S}_{K,j} S K , j is identical to the Gibbs probability measure trained exclusively upon the retained dataset z − j z_{-j} z − j in (200) , in the sense of Definition 8.
Corollary 37. Assume that the condition in (198) holds, and define
λ − j ≜ n 0 n − j λ 0 , (205) \lambda_{-j}\triangleq\frac{n_0}{n_{-j}}\lambda_0, \tag*{(205)} λ − j ≜ n − j n 0 λ 0 , (205) where n − j n_{-j} n − j is defined in (200) . Then, the normalized log-linear combination S ~ K , j \widetilde{S}_{K,j} S K , j in (201) and the ( L n − j , Q , λ − j ) (L_{n_{-j}},Q,\lambda_{-j}) ( L n − j , Q , λ − j ) -Gibbs probability measure P Θ ∣ Z = z − j ( L n − j , Q , λ − j ) P_{\Theta\mid Z=z_{-j}}^{(L_{n_{-j}},Q,\lambda_{-j})} P Θ ∣ Z = z − j ( L n − j , Q , λ − j ) , with z − j z_{-j} z − j in (199) , satisfy, for all θ ∈ supp Q \theta\in\operatorname{supp}Q θ ∈ supp Q ,
d S ~ K , j d P Θ ∣ Z = z − j ( L n − j , Q , λ − j ) ( θ ) = 1. (206) \frac{\mathrm{d}\widetilde{S}_{K,j}}{\mathrm{d}P_{\Theta\mid Z=z_{-j}}^{(L_{n_{-j}},Q,\lambda_{-j})}}(\theta)=1. \tag*{(206)} d P Θ ∣ Z = z − j ( L n − j , Q , λ − j ) d S K , j ( θ ) = 1. (206)
Corollary 37 shows that the normalized log-linear combination S ~ K , j \widetilde{S}_{K,j} S K , j in (201) achieves exact unlearning of the dataset z j z_j z j . Under (198) , the coefficient associated with the empirical risk induced by z j z_j z j is zero, as shown in (204) , whereas the coefficient associated with the empirical risk induced by z − j z_{-j} z − j satisfies
n − j n 0 λ 0 = 1 λ − j . (207) \frac{n_{-j}}{n_0\lambda_0}=\frac{1}{\lambda_{-j}}. \tag*{(207)} n 0 λ 0 n − j = λ − j 1 . (207)
Consequently, the resulting probability measure S ~ K , j \widetilde{S}_{K,j} S K , j is identical to the Gibbs probability measure P Θ ∣ Z = z − j ( L n − j , Q , λ − j ) P_{\Theta\mid Z=z_{-j}}^{(L_{n_{-j}},Q,\lambda_{-j})} P Θ ∣ Z = z − j ( L n − j , Q , λ − j ) that would have been obtained by training exclusively upon the retained dataset z − j z_{-j} z − j , with regularization factor λ − j \lambda_{-j} λ − j in (205) . An important observation is that, once P Θ ∣ Z = z 0 ( L n 0 , Q , λ 0 ) P_{\Theta\mid Z=z_0}^{(L_{n_0},Q,\lambda_0)} P Θ ∣ Z = z 0 ( L n 0 , Q , λ 0 ) and the Gibbs probability measure of client j j j , P Θ ∣ Z = z j ( L n j , Q j , λ j ) P_{\Theta\mid Z=z_j}^{(L_{n_j},Q_j,\lambda_j)} P Θ ∣ Z = z j ( L n j , Q j , λ j ) , are available at the server, the normalized log-linear combination in (201) does not require explicit access to the datasets of the retained clients. The dependence on the dataset to be retained, i.e., dataset z − j z_{-j} z − j , is carried by the original Gibbs probability measure P Θ ∣ Z = z 0 ( L n 0 , Q , λ 0 ) P_{\Theta\mid Z=z_0}^{(L_{n_0},Q,\lambda_0)} P Θ ∣ Z = z 0 ( L n 0 , Q , λ 0 ) . Hence, the same centralized exact-unlearning mechanism developed above is implemented in the federated system using the probability measures sent by the clients and the probability measure obtained at the server.
Discussion
The results in this section establish exact machine unlearning through two closely related mechanisms. With nested Gibbs algorithms, the Gibbs probability measure trained upon the original dataset is used as a data-dependent reference measure, and the regularization factor associated with the dataset to be unlearned is chosen such that its contribution to the resulting weighted empirical risk vanishes. This mechanism extends to decentralized learning because the Gibbs probability measure associated with the aggregation of the local datasets can itself be obtained through a sequence of nested Gibbs algorithms. With normalized log-linear combinations, the contributions of the Gibbs probability measures are combined directly through their Radon–Nikodym derivatives. Exact unlearning is obtained by assigning a compensating contribution to the Gibbs probability measure associated with the dataset to be unlearned. In the federated setting, this allows the contribution of a given client to be canceled directly from the normalized log-linear combination produced at the server.
Hence, although nesting and normalized log-linear combinations operate differently on Gibbs probability measures, their exact-unlearning interpretations rely on the same property established in § 5 : the parameters of the corresponding construction determine the weights of the training data points, and exact unlearning is achieved when the weights associated with the data points to be unlearned are set to zero.
Conclusion
Renormalization, normalized log-linear combinations, and nesting of Gibbs probability measures have been studied. Under suitable conditions, the resulting measures have been shown to be Gibbs probability measures with new objective functions, reference measures, and regularization factors. More specifically, renormalization preserves the objective function while allowing changes in the regularization factor and reference measure. In contrast, normalized log-linear combinations and nesting lead to objective functions consisting of linear combinations of the original objective functions. Using these observations, an equivalence between nesting and normalized log-linear combinations is established. Geometric representations complement these results when the reference measures are probability measures, establishing connections with exponential geodesics in information geometry and interpreting the constrained optimization problems through relative entropy neighborhoods.
In supervised machine learning, normalized log-linear combinations and nesting of Gibbs probability measures yield Gibbs probability measures representing Gibbs algorithms associated with weighted empirical risks with respect to aggregated training datasets. This connection provides a way to control the contribution of each data point to the resulting Gibbs algorithm through the parameters of the corresponding operations. For instance, these parameters can be chosen such that the resulting algorithm is identical to the Gibbs algorithm obtained by training upon the aggregation of all local datasets, which has been referred to as centralized learning or centralized-equivalent performance . That is, the case in which local datasets are aggregated to serve as the training dataset. Consequently, nesting and normalized log-linear combinations have been shown to achieve ccentralized-equivalent performance in decentralized and federated learning systems, respectively.
The same control of the weights provides two mechanisms for exact machine unlearning of specific datasets from Gibbs algorithms. For nesting, this is achieved by maximizing the expected empirical risk with respect to the dataset to be unlearned subject to a regularization by relative entropy with respect to the Gibbs algorithm from which data must be unlearnt. Alternatively, for normalized log-linear combinations, the algorithm from which data must be unlearnt is combined with a Gibbs algorithm trained exclusively on the dataset to be unlearned. In both cases, a suitable choice of the parameters yields the same Gibbs algorithm as if it would be training exclusively upon the retained dataset. Together with the constructions that yield the centralized-equivalent performance, these results on (centralized) unlearning establish exact unlearning of a client’s dataset in decentralized and federated learning systems. Thus, centralized-equivalent performance and exact machine unlearning are obtained as particular applications of the control of training-data contributions through operations on Gibbs algorithms.
References
Appendix A: Proof of Lemma 3
For notational convenience, consider the ( h , Q , λ ) (h,Q,\lambda) ( h , Q , λ ) -Gibbs probability measure, P Y ∣ X = x ( h , Q , λ ) ∈ Δ ( Y ) P^{(h,Q,\lambda)}_{Y\mid X=x} \in\Delta(\mathcal{Y}) P Y ∣ X = x ( h , Q , λ ) ∈ Δ ( Y ) and define the function
K h , Q , x : { R → R t ↦ log ( ∫ exp ( t h ( x , y ) ) d Q ( y ) ) , (208) K_{h,Q,x}: \begin{cases}
\mathbb{R} \to\mathbb{R} \\
t \mapsto\log\left(\int\exp\left(t h(x,y)\right)\mathrm{d}Q(y)\right),
\end{cases} \tag*{(208)} K h , Q , x : { R → R t ↦ log ( ∫ exp ( t h ( x , y ) ) d Q ( y ) ) , (208)
which is a differentiable infinitely many times [27 Lemma 15] in the interior of { t ∈ R : ∫ exp ( t h ( x , y ) ) d Q ( y ) < + ∞ } \left\{t \in\mathbb{R} : \int\exp\left(t h(x,y)\right)\mathrm{d}Q(y) < +\infty\right\} { t ∈ R : ∫ exp ( t h ( x , y ) ) d Q ( y ) < + ∞ } . From [27 Lemma 2] , it follows that
D ( P Y ∣ X = x ( h , Q , λ ) ∥ Q ) = − 1 λ ∫ h ( x , y ) d P Y ∣ X = x ( h , Q , λ ) ( y ) − K h , Q , x ( − 1 λ ) . (209) D\left(P^{(h,Q,\lambda)}_{Y\mid X=x}\mathbin{\|}Q\right) = -\frac{1}{\lambda}\int h(x,y)\mathrm{d}P^{(h,Q,\lambda)}_{Y\mid X=x}(y) - K_{h,Q,x}\left(-\frac{1}{\lambda}\right). \tag*{(209)} D ( P Y ∣ X = x ( h , Q , λ ) ∥ Q ) = − λ 1 ∫ h ( x , y ) d P Y ∣ X = x ( h , Q , λ ) ( y ) − K h , Q , x ( − λ 1 ) . (209)
The n n n -th derivative of the function K h , Q , x K_{h,Q,x} K h , Q , x in (198) is denoted by K h , Q , x ( n ) K^{(n)}_{h,Q,x} K h , Q , x ( n ) . Hence, from [15 Lemma 17] , the first derivative of the function K h , Q , x K_{h,Q,x} K h , Q , x , evaluated at − 1 λ -\frac{1}{\lambda} − λ 1 , satisfies
K h , Q , x ( 1 ) ( − 1 λ ) = ∫ h ( x , y ) d P Y ∣ X = x ( h , Q , λ ) ( y ) . (210) K^{(1)}_{h,Q,x}\left(-\frac{1}{\lambda}\right)=\int h(x,y)\,\mathrm{d}P^{(h,Q,\lambda)}_{Y\mid X=x}(y). \tag*{(210)} K h , Q , x ( 1 ) ( − λ 1 ) = ∫ h ( x , y ) d P Y ∣ X = x ( h , Q , λ ) ( y ) . (210)
Hence, combining (199) and (200) yields
D ( P Y ∣ X = x ( h , Q , λ ) ∥ Q ) = − K h , Q , x ( − 1 λ ) − 1 λ K h , Q , x ( 1 ) ( − 1 λ ) . (211) D\left(P^{(h,Q,\lambda)}_{Y\mid X=x}\mathbin{\|}Q\right)=-K_{h,Q,x}\left(-\frac{1}{\lambda}\right)-\frac{1}{\lambda}K^{(1)}_{h,Q,x}\left(-\frac{1}{\lambda}\right). \tag*{(211)} D ( P Y ∣ X = x ( h , Q , λ ) ∥ Q ) = − K h , Q , x ( − λ 1 ) − λ 1 K h , Q , x ( 1 ) ( − λ 1 ) . (211)
The derivative of the right-hand side of (201) with respect to λ \lambda λ yields
d d λ D ( P Y ∣ X = x ( h , Q , λ ) ∥ Q ) = − 1 λ 3 K h , Q , x ( 2 ) ( − 1 λ ) . (212) \frac{\mathrm{d}}{\mathrm{d}\lambda}D\left(P^{(h,Q,\lambda)}_{Y\mid X=x}\mathbin{\|}Q\right)=-\frac{1}{\lambda^3}K^{(2)}_{h,Q,x}\left(-\frac{1}{\lambda}\right). \tag*{(212)} d λ d D ( P Y ∣ X = x ( h , Q , λ ) ∥ Q ) = − λ 3 1 K h , Q , x ( 2 ) ( − λ 1 ) . (212)
Moreover, from [15 Lemma 17] , the second derivative of the function K h , Q , x K_{h,Q,x} K h , Q , x , evaluated at − 1 λ -\frac{1}{\lambda} − λ 1 , satisfies
K h , Q , x ( 2 ) ( − 1 λ ) = ∫ ( h ( x , y ) − K h , Q , x ( 1 ) ( − 1 λ ) ) 2 d P Y ∣ X = x ( h , Q , λ ) ( y ) ≥ 0. (213) K^{(2)}_{h,Q,x}\left(-\frac{1}{\lambda}\right)=\int\left(h(x,y)-K^{(1)}_{h,Q,x}\left(-\frac{1}{\lambda}\right)\right)^2\mathrm{d}P^{(h,Q,\lambda)}_{Y\mid X=x}(y)\ge0. \tag*{(213)} K h , Q , x ( 2 ) ( − λ 1 ) = ∫ ( h ( x , y ) − K h , Q , x ( 1 ) ( − λ 1 ) ) 2 d P Y ∣ X = x ( h , Q , λ ) ( y ) ≥ 0. (213)
Therefore, (202) and (203) imply that
λ > 0 ⇒ d d λ D ( P Y ∣ X = x ( h , Q , λ ) ∥ Q ) ≤ 0 , (214) \lambda>0\Rightarrow\frac{\mathrm{d}}{\mathrm{d}\lambda}D\left(P^{(h,Q,\lambda)}_{Y\mid X=x}\mathbin{\|}Q\right)\le0, \tag*{(214)} λ > 0 ⇒ d λ d D ( P Y ∣ X = x ( h , Q , λ ) ∥ Q ) ≤ 0 , (214)
λ < 0 ⇒ d d λ D ( P Y ∣ X = x ( h , Q , λ ) ∥ Q ) ≥ 0. (215) \lambda<0\Rightarrow\frac{\mathrm{d}}{\mathrm{d}\lambda}D\left(P^{(h,Q,\lambda)}_{Y\mid X=x}\mathbin{\|}Q\right)\ge0. \tag*{(215)} λ < 0 ⇒ d λ d D ( P Y ∣ X = x ( h , Q , λ ) ∥ Q ) ≥ 0. (215)
This proves that the mapping
λ ↦ D ( P Y ∣ X = x ( h , Q , λ ) ∥ Q ) (216) \lambda\mapsto D\left(P^{(h,Q,\lambda)}_{Y\mid X=x}\mathbin{\|}Q\right) \tag*{(216)} λ ↦ D ( P Y ∣ X = x ( h , Q , λ ) ∥ Q ) (216)
is nonincreasing on all intervals within ( 0 , + ∞ ) (0,+\infty) ( 0 , + ∞ ) , and nondecreasing on all intervals within ( − ∞ , 0 ) (-\infty,0) ( − ∞ , 0 ) , for which
D ( P Y ∣ X = x ( h , Q , λ ) ∥ Q ) < + ∞ . (217) D\left(P^{(h,Q,\lambda)}_{Y\mid X=x}\mathbin{\|}Q\right)<+\infty. \tag*{(217)} D ( P Y ∣ X = x ( h , Q , λ ) ∥ Q ) < + ∞. (217)
The proof continues by identifying the conditions under which K h , Q , x ( 2 ) ( − 1 λ ) K^{(2)}_{h,Q,x}\left(-\frac{1}{\lambda}\right) K h , Q , x ( 2 ) ( − λ 1 ) in (202) is nonzero. That is, when the inequalities in (204) and (205) are strict. For doing so, the notion of separable functions [15 Definition 5] is introduced.
Definition 9 (Separable Functions). Given a fixed x x x , the function h h h in (4) is said to be separable with respect to a probability measure P ∈ △ ( Y ) P\in\triangle(\mathcal{Y}) P ∈ △ ( Y ) , if there exists two subsets
A − = { y ∈ Y : h ( x , y ) > c } ; and (218) \mathcal{A}_{-}=\{y\in\mathcal{Y}:h(x,y)>c\};\ \text{and} \tag*{(218)} A − = { y ∈ Y : h ( x , y ) > c } ; and (218) A + = { y ∈ Y : h ( x , y ) < c } , (219) \mathcal{A}_{+}=\{y\in\mathcal{Y}:h(x,y)<c\}, \tag*{(219)} A + = { y ∈ Y : h ( x , y ) < c } , (219) that are nonnegligible with respect to P P P , for some c ∈ R c\in\mathbb{R} c ∈ R .
Note that if the function h h h in (4) is nonseparable with respect to a measure P P P , it is a constant almost surely with respect to P P P . That is,
P ( { y ∈ Y : h ( x , y ) = c } ) = 1 , (220) P(\{y\in\mathcal{Y}:h(x,y)=c\})=1, \tag*{(220)} P ({ y ∈ Y : h ( x , y ) = c }) = 1 , (220)
for some c ∈ R c\in\mathbb{R} c ∈ R . Using this observation, the following lemma characterizes the conditions under which K h , Q , x ( 2 ) ( − 1 λ ) K^{(2)}_{h,Q,x}\left(-\frac{1}{\lambda}\right) K h , Q , x ( 2 ) ( − λ 1 ) in (202) is nonzero.
Lemma 38. The value K h , Q , x ( 2 ) ( − 1 λ ) K^{(2)}_{h,Q,x}\left(-\frac{1}{\lambda}\right) K h , Q , x ( 2 ) ( − λ 1 ) in (203) satisfies
K h , Q , x ( 2 ) ( − 1 λ ) = 0 , (221) K^{(2)}_{h,Q,x}\left(-\frac{1}{\lambda}\right)=0, \tag*{(221)} K h , Q , x ( 2 ) ( − λ 1 ) = 0 , (221) if and only if the function h h h is nonseparable with respect to Q Q Q .
Proof. The proof uses the following sets:
A − = { y ∈ Y : h ( x , y ) > c } ; (222) \mathcal{A}_{-}=\{y\in\mathcal{Y}:h(x,y)>c\}; \tag*{(222)} A − = { y ∈ Y : h ( x , y ) > c } ; (222)
A 0 = { y ∈ Y : h ( x , y ) = c } ; and (223) \mathcal{A}_{0}=\{y\in\mathcal{Y}:h(x,y)=c\};\ \text{and} \tag*{(223)} A 0 = { y ∈ Y : h ( x , y ) = c } ; and (223)
A + = { y ∈ Y : h ( x , y ) < c } , (224) \mathcal{A}_{+}=\{y\in\mathcal{Y}:h(x,y)<c\}, \tag*{(224)} A + = { y ∈ Y : h ( x , y ) < c } , (224)
with
c ≜ ∫ h ( x , y ) d P Y ∣ X = x ( h , Q , λ ) ( y ) (225) c \triangleq\int h(x,y)\,\mathrm{d}P^{(h,Q,\lambda)}_{Y|X=x}(y) \tag*{(225)} c ≜ ∫ h ( x , y ) d P Y ∣ X = x ( h , Q , λ ) ( y ) (225)
= K h , Q , x ( 1 ) ( − 1 λ ) . (226) =K^{(1)}_{h,Q,x}\left(-\frac{1}{\lambda}\right). \tag*{(226)} = K h , Q , x ( 1 ) ( − λ 1 ) . (226)
The main argument of the proof consists of the following equalities:
K h , Q , x ( 2 ) ( − 1 λ ) = ∫ ( h ( x , y ) − K h , Q , x ( 1 ) ( − 1 λ ) ) 2 d P Y ∣ X = x ( h , Q , λ ) ( y ) = ∫ A − ( h ( x , y ) − K h , Q , x ( 1 ) ( − 1 λ ) ) 2 d P Y ∣ X = x ( h , Q , λ ) ( y ) + ∫ A 0 ( h ( x , y ) − K h , Q , x ( 1 ) ( − 1 λ ) ) 2 d P Y ∣ X = x ( h , Q , λ ) ( y ) (227) \begin{aligned}
K^{(2)}_{h,Q,x}\left(-\frac{1}{\lambda}\right)=\int\left(h(x,y)-K^{(1)}_{h,Q,x}\left(-\frac{1}{\lambda}\right)\right)^2\mathrm{d}P^{(h,Q,\lambda)}_{Y|X=x}(y) \\
&=\int_{\mathcal{A}_{-}}\left(h(x,y)-K^{(1)}_{h,Q,x}\left(-\frac{1}{\lambda}\right)\right)^2\mathrm{d}P^{(h,Q,\lambda)}_{Y|X=x}(y)+\int_{\mathcal{A}_{0}}\left(h(x,y)-K^{(1)}_{h,Q,x}\left(-\frac{1}{\lambda}\right)\right)^2\mathrm{d}P^{(h,Q,\lambda)}_{Y|X=x}(y) \tag*{(227)}
\end{aligned} K h , Q , x ( 2 ) ( − λ 1 ) = ∫ ( h ( x , y ) − K h , Q , x ( 1 ) ( − λ 1 ) ) 2 d P Y ∣ X = x ( h , Q , λ ) ( y ) = ∫ A − ( h ( x , y ) − K h , Q , x ( 1 ) ( − λ 1 ) ) 2 d P Y ∣ X = x ( h , Q , λ ) ( y ) + ∫ A 0 ( h ( x , y ) − K h , Q , x ( 1 ) ( − λ 1 ) ) 2 d P Y ∣ X = x ( h , Q , λ ) ( y ) (227)
+ ∫ A + ( h ( x , y ) − K h , Q , x ( 1 ) ( − 1 λ ) ) 2 d P Y ∣ X = x ( h , Q , λ ) ( y ) (228) +\int_{\mathcal{A}_{+}}\left(h(x,y)-K^{(1)}_{h,Q,x}\left(-\frac{1}{\lambda}\right)\right)^2\mathrm{d}P^{(h,Q,\lambda)}_{Y|X=x}(y) \tag*{(228)} + ∫ A + ( h ( x , y ) − K h , Q , x ( 1 ) ( − λ 1 ) ) 2 d P Y ∣ X = x ( h , Q , λ ) ( y ) (228)
= ∫ A − ( h ( x , y ) − K h , Q , x ( 1 ) ( − 1 λ ) ) 2 d P Y ∣ X = x ( h , Q , λ ) ( y ) + ∫ A + ( h ( x , y ) − K h , Q , x ( 1 ) ( − 1 λ ) ) 2 d P Y ∣ X = x ( h , Q , λ ) ( y ) , (229) =\int_{\mathcal{A}_{-}}\left(h(x,y)-K^{(1)}_{h,Q,x}\left(-\frac{1}{\lambda}\right)\right)^2\mathrm{d}P^{(h,Q,\lambda)}_{Y|X=x}(y)+\int_{\mathcal{A}_{+}}\left(h(x,y)-K^{(1)}_{h,Q,x}\left(-\frac{1}{\lambda}\right)\right)^2\mathrm{d}P^{(h,Q,\lambda)}_{Y|X=x}(y), \tag*{(229)} = ∫ A − ( h ( x , y ) − K h , Q , x ( 1 ) ( − λ 1 ) ) 2 d P Y ∣ X = x ( h , Q , λ ) ( y ) + ∫ A + ( h ( x , y ) − K h , Q , x ( 1 ) ( − λ 1 ) ) 2 d P Y ∣ X = x ( h , Q , λ ) ( y ) , (229)
where the equality in (213) follows from [15 Lemma 17] ; the equality in (214) follows from the definition of the sets in (212); and the equality in (215) follows from observing that
∫ A 0 ( h ( x , y ) − K h , Q , x ( 1 ) ( − 1 λ ) ) 2 d P Y ∣ X = x ( h , Q , λ ) ( y ) = 0. (230) \int_{\mathcal{A}_{0}}\left(h(x,y)-K^{(1)}_{h,Q,x}\left(-\frac{1}{\lambda}\right)\right)^2\mathrm{d}P^{(h,Q,\lambda)}_{Y|X=x}(y)=0. \tag*{(230)} ∫ A 0 ( h ( x , y ) − K h , Q , x ( 1 ) ( − λ 1 ) ) 2 d P Y ∣ X = x ( h , Q , λ ) ( y ) = 0. (230)
The proof is divided into two parts. In the first part, it is shown that if K h , Q , x ( 2 ) ( − 1 λ ) = 0 K^{(2)}_{h,Q,x}\left(-\frac{1}{\lambda}\right)=0 K h , Q , x ( 2 ) ( − λ 1 ) = 0 , then the function h h h is nonseparable with respect to Q Q Q , given x x x . The second part shows that if the function h h h is nonseparable with respect to Q Q Q , given x x x , then K h , Q , x ( 2 ) ( − 1 λ ) = 0 K^{(2)}_{h,Q,x}\left(-\frac{1}{\lambda}\right)=0 K h , Q , x ( 2 ) ( − λ 1 ) = 0 .
The first part is as follows. Assume that
K h , Q , x ( 2 ) ( − 1 λ ) = 0 , (231) K^{(2)}_{h,Q,x}\left(-\frac{1}{\lambda}\right)=0, \tag*{(231)} K h , Q , x ( 2 ) ( − λ 1 ) = 0 , (231)
which, together with (215), imply that
0 = ∫ A − ( h ( x , y ) − K h , Q , x ( 1 ) ( − 1 λ ) ) 2 d P Y ∣ X = x ( h , Q , λ ) ( y ) + ∫ A + ( h ( x , y ) − K h , Q , x ( 1 ) ( − 1 λ ) ) 2 d P Y ∣ X = x ( h , Q , λ ) ( y ) . (232) 0=\int_{\mathcal{A}_{-}}\left(h(x,y)-K^{(1)}_{h,Q,x}\left(-\frac{1}{\lambda}\right)\right)^2\mathrm{d}P^{(h,Q,\lambda)}_{Y|X=x}(y)+\int_{\mathcal{A}_{+}}\left(h(x,y)-K^{(1)}_{h,Q,x}\left(-\frac{1}{\lambda}\right)\right)^2\mathrm{d}P^{(h,Q,\lambda)}_{Y|X=x}(y). \tag*{(232)} 0 = ∫ A − ( h ( x , y ) − K h , Q , x ( 1 ) ( − λ 1 ) ) 2 d P Y ∣ X = x ( h , Q , λ ) ( y ) + ∫ A + ( h ( x , y ) − K h , Q , x ( 1 ) ( − λ 1 ) ) 2 d P Y ∣ X = x ( h , Q , λ ) ( y ) . (232)
Note that the integrands in (218) are nonnegative. Thus, from [44 Theorem 1.6.6b] , it follows that
P Y ∣ X = x ( h , Q , λ ) ( { y ∈ Y : h ( x , y ) − K h , Q , x ( 1 ) ( − 1 λ ) = 0 } ) = 1. (233) P^{(h,Q,\lambda)}_{Y|X=x}\left(\left\{y\in\mathcal{Y}:h(x,y)-K^{(1)}_{h,Q,x}\left(-\frac{1}{\lambda}\right)=0\right\}\right)=1. \tag*{(233)} P Y ∣ X = x ( h , Q , λ ) ( { y ∈ Y : h ( x , y ) − K h , Q , x ( 1 ) ( − λ 1 ) = 0 } ) = 1. (233)
Finally, from the fact that the measures P Y ∣ X = x ( h , Q , λ ) P^{(h,Q,\lambda)}_{Y|X=x} P Y ∣ X = x ( h , Q , λ ) and Q Q Q are mutually absolutely continuous [15 Lemma 3] , it holds that
Q ( { y ∈ Y : h ( x , y ) − K h , Q , x ( 1 ) ( − 1 λ ) = 0 } ) = 1. (234) Q\left(\left\{y\in\mathcal{Y}:h(x,y)-K^{(1)}_{h,Q,x}\left(-\frac{1}{\lambda}\right)=0\right\}\right)=1. \tag*{(234)} Q ( { y ∈ Y : h ( x , y ) − K h , Q , x ( 1 ) ( − λ 1 ) = 0 } ) = 1. (234)
Thus, Q ( A 0 ) = 1 Q(\mathcal{A}_{0})=1 Q ( A 0 ) = 1 and shows that the function h h h is nonseparable with respect to Q Q Q , given x x x . This completes the first part of the proof.
The second part of the proof is as follows. Consider that, given x x x , the function h h h is nonseparable with respect to Q Q Q . Then,
Q ( { y ∈ Y : h ( x , y ) = ∫ h ( x , y ) d P Y ∣ X = x ( h , Q , λ ) ( y ) } ) = 1 , (235) Q\left(\left\{y\in\mathcal{Y}:h(x,y)=\int h(x,y)\mathrm{d}P^{(h,Q,\lambda)}_{Y|X=x}(y)\right\}\right)=1, \tag*{(235)} Q ( { y ∈ Y : h ( x , y ) = ∫ h ( x , y ) d P Y ∣ X = x ( h , Q , λ ) ( y ) } ) = 1 , (235)
which, from the fact that the measures P Y ∣ X = x ( h , Q , λ ) P^{(h,Q,\lambda)}_{Y|X=x} P Y ∣ X = x ( h , Q , λ ) and Q Q Q are mutually absolutely continuous [15 Lemma 3] , implies
1 = Q ( { y ∈ Y : h ( x , y ) = ∫ h ( x , y ) d P Y ∣ X = x ( h , Q , λ ) ( y ) } ) (236) 1=Q\left(\left\{y\in\mathcal{Y}:h(x,y)=\int h(x,y)\mathrm{d}P^{(h,Q,\lambda)}_{Y|X=x}(y)\right\}\right) \tag*{(236)} 1 = Q ( { y ∈ Y : h ( x , y ) = ∫ h ( x , y ) d P Y ∣ X = x ( h , Q , λ ) ( y ) } ) (236)
= P Y ∣ X = x ( h , Q , λ ) ( { y ∈ Y : h ( x , y ) = ∫ h ( x , y ) d P Y ∣ X = x ( h , Q , λ ) ( y ) } ) . (237) =P^{(h,Q,\lambda)}_{Y|X=x}\left(\left\{y\in\mathcal{Y}:h(x,y)=\int h(x,y)\mathrm{d}P^{(h,Q,\lambda)}_{Y|X=x}(y)\right\}\right). \tag*{(237)} = P Y ∣ X = x ( h , Q , λ ) ( { y ∈ Y : h ( x , y ) = ∫ h ( x , y ) d P Y ∣ X = x ( h , Q , λ ) ( y ) } ) . (237)
Hence, considering that the sets A − \mathcal{A}_{-} A − , A 0 \mathcal{A}_{0} A 0 , and A + \mathcal{A}_{+} A + form a partition of Y \mathcal{Y} Y , it follows that
P Y ∣ X = x ( h , Q , λ ) ( A − ) = 0 ; (238) P^{(h,Q,\lambda)}_{Y\mid X=x}(\mathcal{A}_{-})=0; \tag*{(238)} P Y ∣ X = x ( h , Q , λ ) ( A − ) = 0 ; (238)
P Y ∣ X = x ( h , Q , λ ) ( A + ) = 0 ; (239) P^{(h,Q,\lambda)}_{Y\mid X=x}(\mathcal{A}_{+})=0; \tag*{(239)} P Y ∣ X = x ( h , Q , λ ) ( A + ) = 0 ; (239)
P Y ∣ X = x ( h , Q , λ ) ( A 0 ) = 1. (240) P^{(h,Q,\lambda)}_{Y\mid X=x}(\mathcal{A}_{0})=1. \tag*{(240)} P Y ∣ X = x ( h , Q , λ ) ( A 0 ) = 1. (240)
Plugging (224) in (215), it follows that K h , Q , x ( 2 ) ( − 1 λ ) = 0 K^{(2)}_{h,Q,x}\left(-\frac{1}{\lambda}\right)=0 K h , Q , x ( 2 ) ( − λ 1 ) = 0 , which completes the second part of the proof. ■
Finally, from Lemma 38 and the equality in (202), under the assumption that the function h h h is separable, it holds that
λ > 0 ⟹ d d λ D ( P Y ∣ X = x ( h , Q , λ ) ∥ Q ) < 0 , (241) \lambda>0\Longrightarrow\frac{\mathrm{d}}{\mathrm{d}\lambda}D\left(P^{(h,Q,\lambda)}_{Y\mid X=x}\mathbin{\|}Q\right)<0, \tag*{(241)} λ > 0 ⟹ d λ d D ( P Y ∣ X = x ( h , Q , λ ) ∥ Q ) < 0 , (241)
λ < 0 ⟹ d d λ D ( P Y ∣ X = x ( h , Q , λ ) ∥ Q ) > 0 , (242) \lambda<0\Longrightarrow\frac{\mathrm{d}}{\mathrm{d}\lambda}D\left(P^{(h,Q,\lambda)}_{Y\mid X=x}\mathbin{\|}Q\right)>0, \tag*{(242)} λ < 0 ⟹ d λ d D ( P Y ∣ X = x ( h , Q , λ ) ∥ Q ) > 0 , (242)
which implies strict monotonicity. This completes the proof.
Appendix B: Proof of Theorem 4
From Definition 4, the ( α , Q , Q 1 ) (\alpha,Q,Q_{1}) ( α , Q , Q 1 ) -renormalization of the ( h , Q , λ ) (h,Q,\lambda) ( h , Q , λ ) -Gibbs probability measure P Y ∣ X = x ( h , Q , λ ) P^{(h,Q,\lambda)}_{Y\mid X=x} P Y ∣ X = x ( h , Q , λ ) in (6), denoted by R 1 R_{1} R 1 in (16), satisfies, for all y ∈ supp Q 1 y\in\operatorname{supp}Q_{1} y ∈ supp Q 1 ,
d R 1 d Q 1 ( y ) = ( d P Y ∣ X = x ( h , Q , λ ) d Q ( y ) ) α ∫ ( d P Y ∣ X = x ( h , Q , λ ) d Q ( y ~ ) ) α d Q 1 ( y ~ ) (243) \frac{\mathrm{d}R_{1}}{\mathrm{d}Q_{1}}(y)=\frac{\left(\frac{\mathrm{d}P^{(h,Q,\lambda)}_{Y\mid X=x}}{\mathrm{d}Q}(y)\right)^{\alpha}}{\int\left(\frac{\mathrm{d}P^{(h,Q,\lambda)}_{Y\mid X=x}}{\mathrm{d}Q}(\widetilde{y})\right)^{\alpha}\mathrm{d}Q_{1}(\widetilde{y})} \tag*{(243)} d Q 1 d R 1 ( y ) = ∫ ( d Q d P Y ∣ X = x ( h , Q , λ ) ( y ) ) α d Q 1 ( y ) ( d Q d P Y ∣ X = x ( h , Q , λ ) ( y ) ) α (243)
= ( exp ( − 1 λ h ( x , y ) − K h , Q , x ( − 1 λ ) ) ) α ∫ ( exp ( − 1 λ h ( x , y ~ ) − K h , Q , x ( − 1 λ ) ) ) α d Q 1 ( y ~ ) (244) =\frac{\left(\exp\left(-\frac{1}{\lambda}h(x,y)-K_{h,Q,x}\left(-\frac{1}{\lambda}\right)\right)\right)^{\alpha}}{\int\left(\exp\left(-\frac{1}{\lambda}h(x,\widetilde{y})-K_{h,Q,x}\left(-\frac{1}{\lambda}\right)\right)\right)^{\alpha}\mathrm{d}Q_{1}(\widetilde{y})} \tag*{(244)} = ∫ ( exp ( − λ 1 h ( x , y ) − K h , Q , x ( − λ 1 ) ) ) α d Q 1 ( y ) ( exp ( − λ 1 h ( x , y ) − K h , Q , x ( − λ 1 ) ) ) α (244)
= exp ( − α λ h ( x , y ) ) exp ( α K h , Q , x ( − 1 λ ) ) ∫ exp ( − α λ h ( x , y ~ ) ) d Q 1 ( y ~ ) exp ( α K h , Q , x ( − 1 λ ) ) (245) =\frac{\exp\left(-\frac{\alpha}{\lambda}h(x,y)\right)\exp\left(\alpha K_{h,Q,x}\left(-\frac{1}{\lambda}\right)\right)}{\int\exp\left(-\frac{\alpha}{\lambda}h(x,\widetilde{y})\right)\mathrm{d}Q_{1}(\widetilde{y})\exp\left(\alpha K_{h,Q,x}\left(-\frac{1}{\lambda}\right)\right)} \tag*{(245)} = ∫ exp ( − λ α h ( x , y ) ) d Q 1 ( y ) exp ( α K h , Q , x ( − λ 1 ) ) exp ( − λ α h ( x , y ) ) exp ( α K h , Q , x ( − λ 1 ) ) (245)
= exp ( − α λ h ( x , y ) ) ∫ exp ( − α λ h ( x , y ~ ) ) d Q 1 ( y ~ ) (246) =\frac{\exp\left(-\frac{\alpha}{\lambda}h(x,y)\right)}{\int\exp\left(-\frac{\alpha}{\lambda}h(x,\widetilde{y})\right)\mathrm{d}Q_{1}(\widetilde{y})} \tag*{(246)} = ∫ exp ( − λ α h ( x , y ) ) d Q 1 ( y ) exp ( − λ α h ( x , y ) ) (246)
= d P Y ∣ X = x ( h , Q 1 , λ / α ) d Q 1 ( y ) , (247) =\frac{\mathrm{d}P^{(h,Q_{1},\lambda/\alpha)}_{Y\mid X=x}}{\mathrm{d}Q_{1}}(y), \tag*{(247)} = d Q 1 d P Y ∣ X = x ( h , Q 1 , λ / α ) ( y ) , (247)
where the function K h , Q , x K_{h,Q,x} K h , Q , x is defined in (198); and the equalities in (228) and (231) follow from Definition 3. This completes the proof.
Appendix C: Proof of Theorem 7
From (6), for all y ∈ supp Q y\in\operatorname{supp}Q y ∈ supp Q , it follows that
∏ i = 1 K ( d P Y ∣ X = x ( h i , Q i , λ i ) d Q i ( y ) ) α i = ∏ i = 1 K ( exp ( − 1 λ i h i ( x i , y ) ) ∫ exp ( − 1 λ i h i ( x i , y ~ ) ) d Q i ( y ~ ) ) α i (248) \prod_{i=1}^{K}\left(\frac{\mathrm{d}P^{(h_{i},Q_{i},\lambda_{i})}_{Y\mid X=x}}{\mathrm{d}Q_{i}}(y)\right)^{\alpha_{i}}=\prod_{i=1}^{K}\left(\frac{\exp\left(-\frac{1}{\lambda_{i}}h_{i}(x_{i},y)\right)}{\int\exp\left(-\frac{1}{\lambda_{i}}h_{i}(x_{i},\widetilde{y})\right)\mathrm{d}Q_{i}(\widetilde{y})}\right)^{\alpha_{i}} \tag*{(248)} i = 1 ∏ K d Q i d P Y ∣ X = x ( h i , Q i , λ i ) ( y ) α i = i = 1 ∏ K ∫ exp ( − λ i 1 h i ( x i , y ) ) d Q i ( y ) exp ( − λ i 1 h i ( x i , y ) ) α i (248)
= ∏ i = 1 K ( exp ( − 1 λ i h i ( x i , y ) − log ( ∫ exp ( − 1 λ i h i ( x i , y ~ ) ) d Q i ( y ~ ) ) ) ) α i (249) =\prod_{i=1}^{K}\left(\exp\left(-\frac{1}{\lambda_{i}}h_{i}(x_{i},y)-\log\left(\int\exp\left(-\frac{1}{\lambda_{i}}h_{i}(x_{i},\widetilde{y})\right)\mathrm{d}Q_{i}(\widetilde{y})\right)\right)\right)^{\alpha_{i}} \tag*{(249)} = i = 1 ∏ K ( exp ( − λ i 1 h i ( x i , y ) − log ( ∫ exp ( − λ i 1 h i ( x i , y ) ) d Q i ( y ) ) ) ) α i (249)
= ∏ i = 1 K exp ( − α i λ i h i ( x i , y ) − α i log ( ∫ exp ( − 1 λ i h i ( x i , y ~ ) ) d Q i ( y ~ ) ) ) (250) =\prod_{i=1}^{K}\exp\left(-\frac{\alpha_{i}}{\lambda_{i}}h_{i}(x_{i},y)-\alpha_{i}\log\left(\int\exp\left(-\frac{1}{\lambda_{i}}h_{i}(x_{i},\widetilde{y})\right)\mathrm{d}Q_{i}(\widetilde{y})\right)\right) \tag*{(250)} = i = 1 ∏ K exp ( − λ i α i h i ( x i , y ) − α i log ( ∫ exp ( − λ i 1 h i ( x i , y ) ) d Q i ( y ) ) ) (250)
= exp ( − ∑ i = 1 K α i λ i h i ( x i , y ) ) exp ( − ∑ i = 1 K α i log ( ∫ exp ( − 1 λ i h i ( x i , y ~ ) ) d Q i ( y ~ ) ) ) (251) = \exp\left(-\sum_{i=1}^{K}\frac{\alpha_i}{\lambda_i}h_i(x_i,y)\right)\exp\left(-\sum_{i=1}^{K}\alpha_i\log\left(\int\exp\left(-\frac{1}{\lambda_i}h_i(x_i,\widetilde{y})\right)\mathrm{d}Q_i(\widetilde{y})\right)\right) \tag*{(251)} = exp ( − i = 1 ∑ K λ i α i h i ( x i , y ) ) exp ( − i = 1 ∑ K α i log ( ∫ exp ( − λ i 1 h i ( x i , y ) ) d Q i ( y ) ) ) (251)
where the equality in (234) follows from the power rule of the exponential function; and the equality in (235) follows from the product rule of the exponential function. Using (235) in (34) yields
d S K d Q ( y ) = ∏ i = 1 K ( d P Y ∣ X = x i ( h i , Q i , λ i ) d Q i ( y ) ) α i ∫ ∏ i = 1 K ( d P Y ∣ X = x i ( h i , Q i , λ i ) d Q i ( y ~ ) ) α i d Q ( y ~ ) (252) \frac{\mathrm{d}S_K}{\mathrm{d}Q}(y)=\frac{\prod_{i=1}^{K}\left(\frac{\mathrm{d}P_{Y\mid X=x_i}^{(h_i,Q_i,\lambda_i)}}{\mathrm{d}Q_i}(y)\right)^{\alpha_i}}{\int\prod_{i=1}^{K}\left(\frac{\mathrm{d}P_{Y\mid X=x_i}^{(h_i,Q_i,\lambda_i)}}{\mathrm{d}Q_i}(\widetilde{y})\right)^{\alpha_i}\mathrm{d}Q(\widetilde{y})} \tag*{(252)} d Q d S K ( y ) = ∫ ∏ i = 1 K ( d Q i d P Y ∣ X = x i ( h i , Q i , λ i ) ( y ) ) α i d Q ( y ) ∏ i = 1 K ( d Q i d P Y ∣ X = x i ( h i , Q i , λ i ) ( y ) ) α i (252)
= exp ( − ∑ i = 1 K α i λ i h i ( x i , y ) ) exp ( − ∑ i = 1 K α i log ( ∫ exp ( − 1 λ i h i ( x i , y ~ ) ) d Q i ( y ~ ) ) ) ∫ exp ( − ∑ i = 1 K α i λ i h i ( x i , y ~ ) ) exp ( − ∑ i = 1 K α i log ( ∫ exp ( − 1 λ i h i ( x i , y ~ ~ ) ) d Q i ( y ~ ~ ) ) ) d Q ( y ~ ) (253) =\frac{\exp\left(-\sum_{i=1}^{K}\frac{\alpha_i}{\lambda_i}h_i(x_i,y)\right)\exp\left(-\sum_{i=1}^{K}\alpha_i\log\left(\int\exp\left(-\frac{1}{\lambda_i}h_i(x_i,\widetilde{y})\right)\mathrm{d}Q_i(\widetilde{y})\right)\right)}{\int\exp\left(-\sum_{i=1}^{K}\frac{\alpha_i}{\lambda_i}h_i(x_i,\widetilde{y})\right)\exp\left(-\sum_{i=1}^{K}\alpha_i\log\left(\int\exp\left(-\frac{1}{\lambda_i}h_i(x_i,\widetilde{\widetilde{y}})\right)\mathrm{d}Q_i(\widetilde{\widetilde{y}})\right)\right)\mathrm{d}Q(\widetilde{y})} \tag*{(253)} = ∫ exp ( − ∑ i = 1 K λ i α i h i ( x i , y ) ) exp ( − ∑ i = 1 K α i log ( ∫ exp ( − λ i 1 h i ( x i , y ) ) d Q i ( y ) ) ) d Q ( y ) exp ( − ∑ i = 1 K λ i α i h i ( x i , y ) ) exp ( − ∑ i = 1 K α i log ( ∫ exp ( − λ i 1 h i ( x i , y ) ) d Q i ( y ) ) ) (253)
= exp ( − ∑ i = 1 K α i λ i h i ( x i , y ) ) exp ( − ∑ i = 1 K α i log ( ∫ exp ( − 1 λ i h i ( x i , y ~ ) ) d Q i ( y ~ ) ) ) ∫ exp ( − ∑ i = 1 K α i λ i h i ( x i , y ~ ) ) d Q ( y ~ ) exp ( − ∑ i = 1 K α i log ( ∫ exp ( − 1 λ i h i ( x i , y ~ ~ ) ) d Q i ( y ~ ~ ) ) ) = exp ( − ∑ i = 1 K α i λ i h i ( x i , y ) ) ∫ exp ( − ∑ i = 1 K α i λ i h i ( x i , y ~ ) ) d Q ( y ~ ) , (254) \begin{aligned}
=\frac{\exp\left(-\sum_{i=1}^{K}\frac{\alpha_i}{\lambda_i}h_i(x_i,y)\right)\exp\left(-\sum_{i=1}^{K}\alpha_i\log\left(\int\exp\left(-\frac{1}{\lambda_i}h_i(x_i,\widetilde{y})\right)\mathrm{d}Q_i(\widetilde{y})\right)\right)}{\int\exp\left(-\sum_{i=1}^{K}\frac{\alpha_i}{\lambda_i}h_i(x_i,\widetilde{y})\right)\mathrm{d}Q(\widetilde{y})\exp\left(-\sum_{i=1}^{K}\alpha_i\log\left(\int\exp\left(-\frac{1}{\lambda_i}h_i(x_i,\widetilde{\widetilde{y}})\right)\mathrm{d}Q_i(\widetilde{\widetilde{y}})\right)\right)} \\
&=\frac{\exp\left(-\sum_{i=1}^{K}\frac{\alpha_i}{\lambda_i}h_i(x_i,y)\right)}{\int\exp\left(-\sum_{i=1}^{K}\frac{\alpha_i}{\lambda_i}h_i(x_i,\widetilde{y})\right)\mathrm{d}Q(\widetilde{y})}, \tag*{(254)}
\end{aligned} = ∫ exp ( − ∑ i = 1 K λ i α i h i ( x i , y ) ) d Q ( y ) exp ( − ∑ i = 1 K α i log ( ∫ exp ( − λ i 1 h i ( x i , y ) ) d Q i ( y ) ) ) exp ( − ∑ i = 1 K λ i α i h i ( x i , y ) ) exp ( − ∑ i = 1 K α i log ( ∫ exp ( − λ i 1 h i ( x i , y ) ) d Q i ( y ) ) ) = ∫ exp ( − ∑ i = 1 K λ i α i h i ( x i , y ) ) d Q ( y ) exp ( − ∑ i = 1 K λ i α i h i ( x i , y ) ) , (254)
{#eq:239}
where the equality in (238) follows from the fact that
exp ( − ∑ i = 1 K α i log ( ∫ exp ( − 1 λ i h i ( x i , y ~ ) ) d Q i ( y ~ ) ) ) (255) \exp\left(-\sum_{i=1}^{K}\alpha_i\log\left(\int\exp\left(-\frac{1}{\lambda_i}h_i(x_i,\widetilde{y})\right)\mathrm{d}Q_i(\widetilde{y})\right)\right) \tag*{(255)} exp ( − i = 1 ∑ K α i log ( ∫ exp ( − λ i 1 h i ( x i , y ) ) d Q i ( y ) ) ) (255)
does not depend on y ~ \widetilde{y} y ; and the equality in (239) follows by simplifying the common multiplicative factor in (240).
From the definition of β \boldsymbol{\beta} β in the statement of the theorem, it holds that
β = ( α 1 λ 1 , α 2 λ 2 , … , α K λ K ) . (256) \boldsymbol{\beta}=\left(\frac{\alpha_1}{\lambda_1},\frac{\alpha_2}{\lambda_2},\ldots,\frac{\alpha_K}{\lambda_K}\right). \tag*{(256)} β = ( λ 1 α 1 , λ 2 α 2 , … , λ K α K ) . (256)
From the definition of h ^ β \widehat{h}_{\boldsymbol{\beta}} h β in (35), for all y ∈ supp Q y\in\operatorname{supp}Q y ∈ supp Q , it follows that
h ^ β ( x , y ) = ∑ i = 1 K α i λ i h i ( x i , y ) , (257) \widehat{h}_{\boldsymbol{\beta}}(\boldsymbol{x},y)=\sum_{i=1}^{K}\frac{\alpha_i}{\lambda_i}h_i(x_i,y), \tag*{(257)} h β ( x , y ) = i = 1 ∑ K λ i α i h i ( x i , y ) , (257)
where x = ( x 1 , x 2 , … , x K ) \boldsymbol{x}=(x_1,x_2,\ldots,x_K) x = ( x 1 , x 2 , … , x K ) . Therefore, from Definition 3, for all y ∈ supp Q y\in\operatorname{supp}Q y ∈ supp Q ,
d P Y ∣ X = x ( h ^ β , Q , 1 ) d Q ( y ) = exp ( − h ^ β ( x , y ) ) ∫ exp ( − h ^ β ( x , y ~ ) ) d Q ( y ~ ) (258) \frac{\mathrm{d}P_{Y\mid X=\boldsymbol{x}}^{(\widehat{h}_{\boldsymbol{\beta}},Q,1)}}{\mathrm{d}Q}(y)=\frac{\exp\left(-\widehat{h}_{\boldsymbol{\beta}}(\boldsymbol{x},y)\right)}{\int\exp\left(-\widehat{h}_{\boldsymbol{\beta}}(\boldsymbol{x},\widetilde{y})\right)\mathrm{d}Q(\widetilde{y})} \tag*{(258)} d Q d P Y ∣ X = x ( h β , Q , 1 ) ( y ) = ∫ exp ( − h β ( x , y ) ) d Q ( y ) exp ( − h β ( x , y ) ) (258)
= exp ( − ∑ i = 1 K α i λ i h i ( x i , y ) ) ∫ exp ( − ∑ i = 1 K α i λ i h i ( x i , y ~ ) ) d Q ( y ~ ) , (259) =\frac{\exp\left(-\sum_{i=1}^{K}\frac{\alpha_i}{\lambda_i}h_i(x_i,y)\right)}{\int\exp\left(-\sum_{i=1}^{K}\frac{\alpha_i}{\lambda_i}h_i(x_i,\widetilde{y})\right)\mathrm{d}Q(\widetilde{y})}, \tag*{(259)} = ∫ exp ( − ∑ i = 1 K λ i α i h i ( x i , y ) ) d Q ( y ) exp ( − ∑ i = 1 K λ i α i h i ( x i , y ) ) , (259)
where the equality in (244) follows from (242). Combining (239) and (244) yields
d S K d Q ( y ) = d P Y ∣ X = x ( h ^ β , Q , 1 ) d Q ( y ) , (260) \frac{\mathrm{d}S_K}{\mathrm{d}Q}(y)=\frac{\mathrm{d}P_{Y\mid X=\boldsymbol{x}}^{(\widehat{h}_{\boldsymbol{\beta}},Q,1)}}{\mathrm{d}Q}(y), \tag*{(260)} d Q d S K ( y ) = d Q d P Y ∣ X = x ( h β , Q , 1 ) ( y ) , (260)
which completes the proof of (36) and (37).
Appendix D: Proof of Theorem 8
From Theorem 7 , the normalized log-linear combination S K S_K S K in (34) is identical to the ( h ^ β , Q , 1 ) (\widehat{h}_{\beta}, Q, 1) ( h β , Q , 1 ) -Gibbs probability measure P Y ∣ X = x ( h ^ β , Q , 1 ) P_{Y\mid X=x}^{(\widehat{h}_{\beta},Q,1)} P Y ∣ X = x ( h β , Q , 1 ) in (37) . Moreover, taking into consideration that P Y ∣ X = x ( h ^ β , Q , 1 ) P_{Y\mid X=x}^{(\widehat{h}_{\beta},Q,1)} P Y ∣ X = x ( h β , Q , 1 ) is a Gibbs probability measure with reference measure Q Q Q , from [15 Lemma 3] , it follows that
Q ≪ P Y ∣ X = x ( h ^ β , Q , 1 ) ≪ Q . (261) Q \ll P_{Y\mid X=x}^{(\widehat{h}_{\beta},Q,1)} \ll Q. \tag*{(261)} Q ≪ P Y ∣ X = x ( h β , Q , 1 ) ≪ Q . (261)
Thus, from [60 Theorem 2] , the Radon–Nikodym derivative d Q d P Y ∣ X = x ( h ^ β , Q , 1 ) \frac{\mathrm{d}Q}{\mathrm{d}P_{Y\mid X=x}^{(\widehat{h}_{\beta},Q,1)}} d P Y ∣ X = x ( h β , Q , 1 ) d Q is well defined and, from the multiplicative inverse of the Radon–Nikodym derivative [60 Theorem 5] , for all y ∈ supp Q y \in\operatorname{supp} Q y ∈ supp Q , it follows that
d Q d P Y ∣ X = x ( h ^ β , Q , 1 ) ( y ) = ( d P Y ∣ X = x ( h ^ β , Q , 1 ) d Q ( y ) ) − 1 . (262) \frac{\mathrm{d}Q}{\mathrm{d}P_{Y\mid X=x}^{(\widehat{h}_{\beta},Q,1)}}(y) = \left(\frac{\mathrm{d}P_{Y\mid X=x}^{(\widehat{h}_{\beta},Q,1)}}{\mathrm{d}Q}(y)\right)^{-1}. \tag*{(262)} d P Y ∣ X = x ( h β , Q , 1 ) d Q ( y ) = d Q d P Y ∣ X = x ( h β , Q , 1 ) ( y ) − 1 . (262)
Therefore, from Theorem 7 and the equality in (262) , for all y ∈ supp Q y \in\operatorname{supp} Q y ∈ supp Q ,
d S K d P Y ∣ X = x ( h ^ β , Q , 1 ) ( y ) = d S K d Q ( y ) d Q d P Y ∣ X = x ( h ^ β , Q , 1 ) ( y ) (263) \frac{\mathrm{d}S_K}{\mathrm{d}P_{Y\mid X=x}^{(\widehat{h}_{\beta},Q,1)}}(y) = \frac{\mathrm{d}S_K}{\mathrm{d}Q}(y)\frac{\mathrm{d}Q}{\mathrm{d}P_{Y\mid X=x}^{(\widehat{h}_{\beta},Q,1)}}(y) \tag*{(263)} d P Y ∣ X = x ( h β , Q , 1 ) d S K ( y ) = d Q d S K ( y ) d P Y ∣ X = x ( h β , Q , 1 ) d Q ( y ) (263)
= d S K d Q ( y ) ( d P Y ∣ X = x ( h ^ β , Q , 1 ) d Q ( y ) ) − 1 (264) = \frac{\mathrm{d}S_K}{\mathrm{d}Q}(y)\left(\frac{\mathrm{d}P_{Y\mid X=x}^{(\widehat{h}_{\beta},Q,1)}}{\mathrm{d}Q}(y)\right)^{-1} \tag*{(264)} = d Q d S K ( y ) d Q d P Y ∣ X = x ( h β , Q , 1 ) ( y ) − 1 (264)
= 1 , (265) = 1, \tag*{(265)} = 1 , (265)
where the equality in (263) follows from the chain rule of the Radon–Nikodym derivative [60 Theorem 4] ; the equality in (264) follows from (262) ; and the equality in (265) follows from Theorem 7 . This proves (i).
Assume that (i) holds. Then,
D ( S K ∥ P Y ∣ X = x ( h ^ β , Q , 1 ) ) = ∫ log ( d S K d P Y ∣ X = x ( h ^ β , Q , 1 ) ( y ) ) d S K ( y ) (266) D\left(S_K \mathbin{\|} P_{Y\mid X=x}^{(\widehat{h}_{\beta},Q,1)}\right) = \int\log\left(\frac{\mathrm{d}S_K}{\mathrm{d}P_{Y\mid X=x}^{(\widehat{h}_{\beta},Q,1)}}(y)\right)\mathrm{d}S_K(y) \tag*{(266)} D ( S K ∥ P Y ∣ X = x ( h β , Q , 1 ) ) = ∫ log d P Y ∣ X = x ( h β , Q , 1 ) d S K ( y ) d S K ( y ) (266)
= 0 , (267) = 0, \tag*{(267)} = 0 , (267)
which proves (ii). Conversely, if (ii) holds, then from [15 Theorem 1] it follows that
d S K d P Y ∣ X = x ( h ^ β , Q , 1 ) ( y ) = 1. (268) \frac{\mathrm{d}S_K}{\mathrm{d}P_{Y\mid X=x}^{(\widehat{h}_{\beta},Q,1)}}(y) = 1. \tag*{(268)} d P Y ∣ X = x ( h β , Q , 1 ) d S K ( y ) = 1. (268)
Therefore, (i) and (ii) are equivalent.
Assume next that (i) holds. Then S K S_K S K is identical to the ( h ^ β , Q , 1 ) (\widehat{h}_{\beta}, Q, 1) ( h β , Q , 1 ) -Gibbs probability measure P Y ∣ X = x ( h ^ β , Q , 1 ) P_{Y\mid X=x}^{(\widehat{h}_{\beta},Q,1)} P Y ∣ X = x ( h β , Q , 1 ) . Hence, from Lemma 1, it is the unique solution, if it exists, to the optimization problem in (41) . This proves that (i) implies (iii).
Assume that (iii) holds. From Lemma 1, the unique solution, if it exists, to the optimization problem in (41) is precisely the ( h ^ β , Q , 1 ) (\widehat{h}_{\beta}, Q, 1) ( h β , Q , 1 ) -Gibbs probability measure P Y ∣ X = x ( h ^ β , Q , 1 ) P_{Y\mid X=x}^{(\widehat{h}_{\beta},Q,1)} P Y ∣ X = x ( h β , Q , 1 ) . Hence,
S K = P Y ∣ X = x ( h ^ β , Q , 1 ) . (269) S_K = P_{Y\mid X=x}^{(\widehat{h}_{\beta},Q,1)}. \tag*{(269)} S K = P Y ∣ X = x ( h β , Q , 1 ) . (269)
Thus,
D ( S K ∥ Q ) = D ( P Y ∣ X = x ( h ^ β , Q , 1 ) ∥ Q ) . (270) D(S_K \mathbin{\|} Q) = D\left(P_{Y\mid X=x}^{(\widehat{h}_{\beta},Q,1)} \mathbin{\|} Q\right). \tag*{(270)} D ( S K ∥ Q ) = D ( P Y ∣ X = x ( h β , Q , 1 ) ∥ Q ) . (270)
Taking into account that P Y ∣ X = x ( h ^ β , Q , 1 ) P_{Y\mid X=x}^{(\widehat h_{\beta},\mathcal{Q},1)} P Y ∣ X = x ( h β , Q , 1 ) is a ( h ^ β , Q , 1 ) (\widehat h_{\beta},\mathcal{Q},1) ( h β , Q , 1 ) -Gibbs probability measure, Lemma 2 implies that it is the unique solution, if it exists, to the optimization problem in (42). Therefore, S K S_K S K is the unique solution, if it exists, to the same optimization problem. This proves that ( i i i ) (iii) ( iii ) implies ( i v ) (iv) ( i v ) .
Assume finally that ( i v ) (iv) ( i v ) holds. From Lemma 2, the unique solution, if it exists, to the optimization problem in (42) is precisely the ( h ^ β , Q , 1 ) (\widehat h_{\beta},\mathcal{Q},1) ( h β , Q , 1 ) -Gibbs probability measure P Y ∣ X = x ( h ^ β , Q , 1 ) P_{Y\mid X=x}^{(\widehat h_{\beta},\mathcal{Q},1)} P Y ∣ X = x ( h β , Q , 1 ) . Hence,
S K = P Y ∣ X = x ( h ^ β , Q , 1 ) . (271) S_K=P_{Y\mid X=x}^{(\widehat h_{\beta},\mathcal{Q},1)}. \tag*{(271)} S K = P Y ∣ X = x ( h β , Q , 1 ) . (271)
Together with the absolute-continuity relation established above and the explicit Radon–Nikodym derivatives in Theorem 7 , it follows that, for all y ∈ supp Q y\in\operatorname{supp}\mathcal{Q} y ∈ supp Q ,
d S K d P Y ∣ X = x ( h ^ β , Q , 1 ) ( y ) = 1. (272) \frac{\mathrm{d}S_K}{\mathrm{d}P_{Y\mid X=x}^{(\widehat h_{\beta},\mathcal{Q},1)}}(y)=1. \tag*{(272)} d P Y ∣ X = x ( h β , Q , 1 ) d S K ( y ) = 1. (272)
This proves that ( i v ) (iv) ( i v ) implies ( i ) (i) ( i ) .
Therefore, ( i ) (i) ( i ) , ( i i ) (ii) ( ii ) , ( i i i ) (iii) ( iii ) , and ( i v ) (iv) ( i v ) are equivalent. This concludes the proof.
Appendix E: Proof of Theorem 10
From Definition 3, for all y ∈ supp Q 1 y\in\operatorname{supp}\mathcal{Q}_1 y ∈ supp Q 1 , it follows that
d P Y ∣ X = x 2 ( h 2 , P Y ∣ X = x 1 ( h 1 , Q 1 , λ 1 ) , λ 2 ) d P Y ∣ X = x 1 ( h 1 , Q 1 , λ 1 ) ( y ) = exp ( − 1 λ 2 h 2 ( x 2 , y ) ) ∫ exp ( − 1 λ 2 h 2 ( x 2 , y ~ ) ) d P Y ∣ X = x 1 ( h 1 , Q 1 , λ 1 ) ( y ~ ) (273) \frac{\mathrm{d}P_{Y\mid X=x_2}^{(h_2,P_{Y\mid X=x_1}^{(h_1,\mathcal{Q}_1,\lambda_1)},\lambda_2)}}{\mathrm{d}P_{Y\mid X=x_1}^{(h_1,\mathcal{Q}_1,\lambda_1)}}(y)
=
\frac{\exp\left(-\frac{1}{\lambda_2}h_2(x_2,y)\right)}
{\int\exp\left(-\frac{1}{\lambda_2}h_2(x_2,\widetilde y)\right)\mathrm{d}P_{Y\mid X=x_1}^{(h_1,\mathcal{Q}_1,\lambda_1)}(\widetilde y)} \tag*{(273)} d P Y ∣ X = x 1 ( h 1 , Q 1 , λ 1 ) d P Y ∣ X = x 2 ( h 2 , P Y ∣ X = x 1 ( h 1 , Q 1 , λ 1 ) , λ 2 ) ( y ) = ∫ exp ( − λ 2 1 h 2 ( x 2 , y ) ) d P Y ∣ X = x 1 ( h 1 , Q 1 , λ 1 ) ( y ) exp ( − λ 2 1 h 2 ( x 2 , y ) ) (273)
= exp ( − 1 λ 2 h 2 ( x 2 , y ) ) ∫ exp ( − 1 λ 2 h 2 ( x 2 , y ~ ) ) d P Y ∣ X = x 1 ( h 1 , Q 1 , λ 1 ) d Q 1 ( y ~ ) d Q 1 ( y ~ ) (274) =
\frac{\exp\left(-\frac{1}{\lambda_2}h_2(x_2,y)\right)}
{\int\exp\left(-\frac{1}{\lambda_2}h_2(x_2,\widetilde y)\right)\frac{\mathrm{d}P_{Y\mid X=x_1}^{(h_1,\mathcal{Q}_1,\lambda_1)}}{\mathrm{d}\mathcal{Q}_1}(\widetilde y)\mathrm{d}\mathcal{Q}_1(\widetilde y)} \tag*{(274)} = ∫ exp ( − λ 2 1 h 2 ( x 2 , y ) ) d Q 1 d P Y ∣ X = x 1 ( h 1 , Q 1 , λ 1 ) ( y ) d Q 1 ( y ) exp ( − λ 2 1 h 2 ( x 2 , y ) ) (274)
= exp ( − 1 λ 2 h 2 ( x 2 , y ) ) ∫ exp ( − 1 λ 1 h 1 ( x 1 , y ~ ) ) d Q 1 ( y ~ ) ∫ exp ( − 1 λ 2 h 2 ( x 2 , y ~ ) ) exp ( − 1 λ 1 h 1 ( x 1 , y ~ ) ) d Q 1 ( y ~ ) . (275) =
\frac{\exp\left(-\frac{1}{\lambda_2}h_2(x_2,y)\right)\int\exp\left(-\frac{1}{\lambda_1}h_1(x_1,\widetilde y)\right)\mathrm{d}\mathcal{Q}_1(\widetilde y)}
{\int\exp\left(-\frac{1}{\lambda_2}h_2(x_2,\widetilde y)\right)\exp\left(-\frac{1}{\lambda_1}h_1(x_1,\widetilde y)\right)\mathrm{d}\mathcal{Q}_1(\widetilde y)}. \tag*{(275)} = ∫ exp ( − λ 2 1 h 2 ( x 2 , y ) ) exp ( − λ 1 1 h 1 ( x 1 , y ) ) d Q 1 ( y ) exp ( − λ 2 1 h 2 ( x 2 , y ) ) ∫ exp ( − λ 1 1 h 1 ( x 1 , y ) ) d Q 1 ( y ) . (275)
From [15 Lemma 3] and [60 Theorem 2] the Radon–Nikodym derivative d Q 1 d P Y ∣ X = x 1 ( h 1 , Q 1 , λ 1 ) \frac{\mathrm{d}\mathcal{Q}_1}{\mathrm{d}P_{Y\mid X=x_1}^{(h_1,\mathcal{Q}_1,\lambda_1)}} d P Y ∣ X = x 1 ( h 1 , Q 1 , λ 1 ) d Q 1 is well defined and from [60 Theorem 4] , for all y ∈ supp Q 1 y\in\operatorname{supp}\mathcal{Q}_1 y ∈ supp Q 1 , it follows that
d P Y ∣ X = x 2 ( h 2 , P Y ∣ X = x 1 ( h 1 , Q 1 , λ 1 ) , λ 2 ) d P Y ∣ X = x 1 ( h 1 , Q 1 , λ 1 ) ( y ) = d P Y ∣ X = x 2 ( h 2 , P Y ∣ X = x 1 ( h 1 , Q 1 , λ 1 ) , λ 2 ) d Q 1 ( y ) d Q 1 d P Y ∣ X = x 1 ( h 1 , Q 1 , λ 1 ) ( y ) . (276) \frac{\mathrm{d}P_{Y\mid X=x_2}^{(h_2,P_{Y\mid X=x_1}^{(h_1,\mathcal{Q}_1,\lambda_1)},\lambda_2)}}{\mathrm{d}P_{Y\mid X=x_1}^{(h_1,\mathcal{Q}_1,\lambda_1)}}(y)
=
\frac{\mathrm{d}P_{Y\mid X=x_2}^{(h_2,P_{Y\mid X=x_1}^{(h_1,\mathcal{Q}_1,\lambda_1)},\lambda_2)}}{\mathrm{d}\mathcal{Q}_1}(y)
\frac{\mathrm{d}\mathcal{Q}_1}{\mathrm{d}P_{Y\mid X=x_1}^{(h_1,\mathcal{Q}_1,\lambda_1)}}(y). \tag*{(276)} d P Y ∣ X = x 1 ( h 1 , Q 1 , λ 1 ) d P Y ∣ X = x 2 ( h 2 , P Y ∣ X = x 1 ( h 1 , Q 1 , λ 1 ) , λ 2 ) ( y ) = d Q 1 d P Y ∣ X = x 2 ( h 2 , P Y ∣ X = x 1 ( h 1 , Q 1 , λ 1 ) , λ 2 ) ( y ) d P Y ∣ X = x 1 ( h 1 , Q 1 , λ 1 ) d Q 1 ( y ) . (276)
Combining (275) and (276) yields for all y ∈ supp Q 1 y\in\operatorname{supp}\mathcal{Q}_1 y ∈ supp Q 1 ,
d P Y ∣ X = x 2 ( h 2 , P Y ∣ X = x 1 ( h 1 , Q 1 , λ 1 ) , λ 2 ) d Q 1 ( y ) = ( d Q 1 d P Y ∣ X = x 1 ( h 1 , Q 1 , λ 1 ) ( y ) ) − 1 exp ( − 1 λ 2 h 2 ( x 2 , y ) ) ∫ exp ( − 1 λ 1 h 1 ( x 1 , y ~ ) ) d Q 1 ( y ~ ) ∫ exp ( − 1 λ 2 h 2 ( x 2 , y ~ ) − 1 λ 1 h 1 ( x 1 , y ~ ) ) d Q 1 ( y ~ ) = d P Y ∣ X = x 1 ( h 1 , Q 1 , λ 1 ) d Q 1 ( y ) exp ( − 1 λ 2 h 2 ( x 2 , y ) ) ∫ exp ( − 1 λ 1 h 1 ( x 1 , y ~ ) ) d Q 1 ( y ~ ) ∫ exp ( − 1 λ 2 h 2 ( x 2 , y ~ ) − 1 λ 1 h 1 ( x 1 , y ~ ) ) d Q 1 ( y ~ ) = exp ( − 1 λ 1 h 1 ( x 1 , y ) ) ∫ exp ( − 1 λ 1 h 1 ( x 1 , y ~ ) ) d Q 1 ( y ~ ) . (277) \begin{aligned}
\frac{\mathrm{d}P_{Y\mid X=x_2}^{(h_2,P_{Y\mid X=x_1}^{(h_1,\mathcal{Q}_1,\lambda_1)},\lambda_2)}}{\mathrm{d}\mathcal{Q}_1}(y)
&=
\left(\frac{\mathrm{d}\mathcal{Q}_1}{\mathrm{d}P_{Y\mid X=x_1}^{(h_1,\mathcal{Q}_1,\lambda_1)}}(y)\right)^{-1}
\frac{\exp\left(-\frac{1}{\lambda_2}h_2(x_2,y)\right)\int\exp\left(-\frac{1}{\lambda_1}h_1(x_1,\widetilde y)\right)\mathrm{d}\mathcal{Q}_1(\widetilde y)}
{\int\exp\left(-\frac{1}{\lambda_2}h_2(x_2,\widetilde y)-\frac{1}{\lambda_1}h_1(x_1,\widetilde y)\right)\mathrm{d}\mathcal{Q}_1(\widetilde y)}
\\
&=
\frac{\mathrm{d}P_{Y\mid X=x_1}^{(h_1,\mathcal{Q}_1,\lambda_1)}}{\mathrm{d}\mathcal{Q}_1}(y)
\frac{\exp\left(-\frac{1}{\lambda_2}h_2(x_2,y)\right)\int\exp\left(-\frac{1}{\lambda_1}h_1(x_1,\widetilde y)\right)\mathrm{d}\mathcal{Q}_1(\widetilde y)}
{\int\exp\left(-\frac{1}{\lambda_2}h_2(x_2,\widetilde y)-\frac{1}{\lambda_1}h_1(x_1,\widetilde y)\right)\mathrm{d}\mathcal{Q}_1(\widetilde y)}
\\
&=
\frac{\exp\left(-\frac{1}{\lambda_1}h_1(x_1,y)\right)}
{\int\exp\left(-\frac{1}{\lambda_1}h_1(x_1,\widetilde y)\right)\mathrm{d}\mathcal{Q}_1(\widetilde y)}. \tag*{(277)}
\end{aligned} d Q 1 d P Y ∣ X = x 2 ( h 2 , P Y ∣ X = x 1 ( h 1 , Q 1 , λ 1 ) , λ 2 ) ( y ) = d P Y ∣ X = x 1 ( h 1 , Q 1 , λ 1 ) d Q 1 ( y ) − 1 ∫ exp ( − λ 2 1 h 2 ( x 2 , y ) − λ 1 1 h 1 ( x 1 , y ) ) d Q 1 ( y ) exp ( − λ 2 1 h 2 ( x 2 , y ) ) ∫ exp ( − λ 1 1 h 1 ( x 1 , y ) ) d Q 1 ( y ) = d Q 1 d P Y ∣ X = x 1 ( h 1 , Q 1 , λ 1 ) ( y ) ∫ exp ( − λ 2 1 h 2 ( x 2 , y ) − λ 1 1 h 1 ( x 1 , y ) ) d Q 1 ( y ) exp ( − λ 2 1 h 2 ( x 2 , y ) ) ∫ exp ( − λ 1 1 h 1 ( x 1 , y ) ) d Q 1 ( y ) = ∫ exp ( − λ 1 1 h 1 ( x 1 , y ) ) d Q 1 ( y ) exp ( − λ 1 1 h 1 ( x 1 , y ) ) . (277)
exp ( − 1 λ 2 h 2 ( x 2 , y ) ) ∫ exp ( − 1 λ 1 h 1 ( x 1 , y ~ ) ) d Q 1 ( y ~ ) ∫ exp ( − 1 λ 2 h 2 ( x 2 , y ~ ) − 1 λ 1 h 1 ( x 1 , y ~ ) ) d Q 1 ( y ~ ) (278) \frac{\exp\left(-\frac{1}{\lambda_2}h_2(x_2,y)\right)\int\exp\left(-\frac{1}{\lambda_1}h_1(x_1,\widetilde{y})\right)\,dQ_1(\widetilde{y})}{\int\exp\left(-\frac{1}{\lambda_2}h_2(x_2,\widetilde{y})-\frac{1}{\lambda_1}h_1(x_1,\widetilde{y})\right)\,dQ_1(\widetilde{y})} \tag*{(278)} ∫ exp ( − λ 2 1 h 2 ( x 2 , y ) − λ 1 1 h 1 ( x 1 , y ) ) d Q 1 ( y ) exp ( − λ 2 1 h 2 ( x 2 , y ) ) ∫ exp ( − λ 1 1 h 1 ( x 1 , y ) ) d Q 1 ( y ) (278)
= exp ( − 1 λ 1 h 1 ( x 1 , y ) − 1 λ 2 h 2 ( x 2 , y ) ) ∫ exp ( − 1 λ 1 h 1 ( x 1 , y ~ ) − 1 λ 2 h 2 ( x 2 , y ~ ) ) d Q 1 ( y ~ ) (279) =\frac{\exp\left(-\frac{1}{\lambda_1}h_1(x_1,y)-\frac{1}{\lambda_2}h_2(x_2,y)\right)}{\int\exp\left(-\frac{1}{\lambda_1}h_1(x_1,\widetilde{y})-\frac{1}{\lambda_2}h_2(x_2,\widetilde{y})\right)\,dQ_1(\widetilde{y})} \tag*{(279)} = ∫ exp ( − λ 1 1 h 1 ( x 1 , y ) − λ 2 1 h 2 ( x 2 , y ) ) d Q 1 ( y ) exp ( − λ 1 1 h 1 ( x 1 , y ) − λ 2 1 h 2 ( x 2 , y ) ) (279)
= exp ( − h ^ β ( x , y ) ) ∫ exp ( − h ^ β ( x , y ~ ) ) d Q 1 ( y ~ ) (280) =\frac{\exp\left(-\widehat{h}_{\boldsymbol{\beta}}(x,y)\right)}{\int\exp\left(-\widehat{h}_{\boldsymbol{\beta}}(x,\widetilde{y})\right)\,dQ_1(\widetilde{y})} \tag*{(280)} = ∫ exp ( − h β ( x , y ) ) d Q 1 ( y ) exp ( − h β ( x , y ) ) (280)
= d P Y ∣ X = x ( h ^ β , Q 1 , 1 ) d Q 1 ( y ) , (281) =\frac{dP^{(\widehat{h}_{\boldsymbol{\beta}},Q_1,1)}_{Y\mid X=x}}{dQ_1}(y), \tag*{(281)} = d Q 1 d P Y ∣ X = x ( h β , Q 1 , 1 ) ( y ) , (281)
where the equality in (263) follows from the multiplicative inverse of the Radon-Nikodym derivative [60 Theorem 5] ; the equality in (264) follows from Definition 3; the equality in (266) follows from the definition of the function h ^ β \widehat{h}_{\boldsymbol{\beta}} h β in (35) with x = ( x 1 , x 2 ) x=(x_1,x_2) x = ( x 1 , x 2 ) and β = ( 1 λ 1 , 1 λ 2 ) \boldsymbol{\beta}=\left(\frac{1}{\lambda_1},\frac{1}{\lambda_2}\right) β = ( λ 1 1 , λ 2 1 ) ; and the equality in (267) follows from Definition 3. This completes the proof.
Appendix F: Proof of Theorem 11
The proof of (51) is done by induction on K ∈ N K\in\mathbb{N} K ∈ N .
Base cases (K = 1 K=1 K = 1 and K = 2 K=2 K = 2 ):
When K = 1 K=1 K = 1 , from (50b), the reference measure is Q 1 Q_1 Q 1 . Hence, from Definition 3, for all y ∈ supp Q 1 y\in\operatorname{supp}Q_1 y ∈ supp Q 1 , it follows that
d P Y ∣ X = x 1 ( h 1 , Q 1 , λ 1 ) d Q 1 ( y ) = exp ( − 1 λ 1 h 1 ( x 1 , y ) ) ∫ exp ( − 1 λ 1 h 1 ( x 1 , y ~ ) ) d Q 1 ( y ~ ) . (282) \frac{dP^{(h_1,Q_1,\lambda_1)}_{Y\mid X=x_1}}{dQ_1}(y)=\frac{\exp\left(-\frac{1}{\lambda_1}h_1(x_1,y)\right)}{\int\exp\left(-\frac{1}{\lambda_1}h_1(x_1,\widetilde{y})\right)\,dQ_1(\widetilde{y})}. \tag*{(282)} d Q 1 d P Y ∣ X = x 1 ( h 1 , Q 1 , λ 1 ) ( y ) = ∫ exp ( − λ 1 1 h 1 ( x 1 , y ) ) d Q 1 ( y ) exp ( − λ 1 1 h 1 ( x 1 , y ) ) . (282)
Thus, the equality holds for K = 1 K=1 K = 1 .
When K = 2 K=2 K = 2 , from (50b), the reference measure Q 2 Q_2 Q 2 satisfies Q 2 = P Y ∣ X = x 1 ( h 1 , Q 1 , λ 1 ) Q_2=P^{(h_1,Q_1,\lambda_1)}_{Y\mid X=x_1} Q 2 = P Y ∣ X = x 1 ( h 1 , Q 1 , λ 1 ) . Therefore, from Theorem 10, for all y ∈ supp Q 1 y\in\operatorname{supp}Q_1 y ∈ supp Q 1 , it follows that
d P Y ∣ X = x 2 ( h 2 , Q 2 , λ 2 ) d Q 1 ( y ) = exp ( − 1 λ 1 h 1 ( x 1 , y ) − 1 λ 2 h 2 ( x 2 , y ) ) ∫ exp ( − 1 λ 1 h 1 ( x 1 , y ~ ) − 1 λ 2 h 2 ( x 2 , y ~ ) ) d Q 1 ( y ~ ) . (283) \frac{dP^{(h_2,Q_2,\lambda_2)}_{Y\mid X=x_2}}{dQ_1}(y)=\frac{\exp\left(-\frac{1}{\lambda_1}h_1(x_1,y)-\frac{1}{\lambda_2}h_2(x_2,y)\right)}{\int\exp\left(-\frac{1}{\lambda_1}h_1(x_1,\widetilde{y})-\frac{1}{\lambda_2}h_2(x_2,\widetilde{y})\right)\,dQ_1(\widetilde{y})}. \tag*{(283)} d Q 1 d P Y ∣ X = x 2 ( h 2 , Q 2 , λ 2 ) ( y ) = ∫ exp ( − λ 1 1 h 1 ( x 1 , y ) − λ 2 1 h 2 ( x 2 , y ) ) d Q 1 ( y ) exp ( − λ 1 1 h 1 ( x 1 , y ) − λ 2 1 h 2 ( x 2 , y ) ) . (283)
Thus, the equality holds for K = 2 K=2 K = 2 .
Induction hypothesis: Assume that (51) holds for K ≥ 2 K\ge2 K ≥ 2 , that is, for all y ∈ supp Q 1 y\in\operatorname{supp}Q_1 y ∈ supp Q 1 ,
d P Y ∣ X = x K ( h K , Q K , λ K ) d Q 1 ( y ) = exp ( − ∑ i = 1 K 1 λ i h i ( x i , y ) ) ∫ exp ( − ∑ i = 1 K 1 λ i h i ( x i , y ~ ) ) d Q 1 ( y ~ ) . (284) \frac{dP^{(h_K,Q_K,\lambda_K)}_{Y\mid X=x_K}}{dQ_1}(y)=\frac{\exp\left(-\sum_{i=1}^{K}\frac{1}{\lambda_i}h_i(x_i,y)\right)}{\int\exp\left(-\sum_{i=1}^{K}\frac{1}{\lambda_i}h_i(x_i,\widetilde{y})\right)\,dQ_1(\widetilde{y})}. \tag*{(284)} d Q 1 d P Y ∣ X = x K ( h K , Q K , λ K ) ( y ) = ∫ exp ( − ∑ i = 1 K λ i 1 h i ( x i , y ) ) d Q 1 ( y ) exp ( − ∑ i = 1 K λ i 1 h i ( x i , y ) ) . (284)
Induction step (K → K + 1 K\to K+1 K → K + 1 ): It remains to prove (51) for K + 1 K+1 K + 1 . Consider the Gibbs probability measure P Y ∣ X = x K + 1 ( h K + 1 , Q K + 1 , λ K + 1 ) P^{(h_{K+1},Q_{K+1},\lambda_{K+1})}_{Y\mid X=x_{K+1}} P Y ∣ X = x K + 1 ( h K + 1 , Q K + 1 , λ K + 1 ) and its reference measure Q K + 1 Q_{K+1} Q K + 1 satisfying (50b). From Definition 3, for all y ∈ supp Q K + 1 y\in\operatorname{supp}Q_{K+1} y ∈ supp Q K + 1 , it follows that
d P Y ∣ X = x K + 1 ( h K + 1 , Q K + 1 , λ K + 1 ) d Q K + 1 ( y ) = exp ( − 1 λ K + 1 h K + 1 ( x K + 1 , y ) ) ∫ exp ( − 1 λ K + 1 h K + 1 ( x K + 1 , y ~ ) ) d Q K + 1 ( y ~ ) , (285) \frac{dP^{(h_{K+1},Q_{K+1},\lambda_{K+1})}_{Y\mid X=x_{K+1}}}{dQ_{K+1}}(y)=\frac{\exp\left(-\frac{1}{\lambda_{K+1}}h_{K+1}(x_{K+1},y)\right)}{\int\exp\left(-\frac{1}{\lambda_{K+1}}h_{K+1}(x_{K+1},\widetilde{y})\right)\,dQ_{K+1}(\widetilde{y})}, \tag*{(285)} d Q K + 1 d P Y ∣ X = x K + 1 ( h K + 1 , Q K + 1 , λ K + 1 ) ( y ) = ∫ exp ( − λ K + 1 1 h K + 1 ( x K + 1 , y ) ) d Q K + 1 ( y ) exp ( − λ K + 1 1 h K + 1 ( x K + 1 , y ) ) , (285)
The proof follows from the fact that Gibbs probability measures and their reference measures are mutually absolutely continuous [15 Lemma 3] , which yields
Q 1 ≪ Q 2 ≪ … ≪ Q K − 1 ≪ Q K ≪ Q K + 1 ≪ Q 1 . (286) Q_1\ll Q_2\ll\ldots\ll Q_{K-1}\ll Q_K\ll Q_{K+1}\ll Q_1. \tag*{(286)} Q 1 ≪ Q 2 ≪ … ≪ Q K − 1 ≪ Q K ≪ Q K + 1 ≪ Q 1 . (286)
Thus, the measures Q K + 1 Q_{K+1} Q K + 1 and Q 1 Q_1 Q 1 are mutually absolutely continuous and from [15 Lemma 3] and [60 Theorem 2] , the Radon–Nikodym derivatives d Q K + 1 d Q 1 \frac{dQ_{K+1}}{dQ_1} d Q 1 d Q K + 1 and d Q 1 d Q K + 1 \frac{dQ_1}{dQ_{K+1}} d Q K + 1 d Q 1 are well defined. From the chain rule of the Radon–Nikodym derivative [60 Theorem 4] , for all y ∈ supp Q 1 y \in\operatorname{supp} Q_1 y ∈ supp Q 1 , it follows that
d P Y ∣ X = x K + 1 ( h K + 1 , Q K + 1 , λ K + 1 ) d Q K + 1 ( y ) = d P Y ∣ X = x K + 1 ( h K + 1 , Q K + 1 , λ K + 1 ) d Q 1 ( y ) d Q 1 d Q K + 1 ( y ) . (287) \frac{dP_{Y\mid X=x_{K+1}}^{(h_{K+1},Q_{K+1},\lambda_{K+1})}}{dQ_{K+1}}(y)=\frac{dP_{Y\mid X=x_{K+1}}^{(h_{K+1},Q_{K+1},\lambda_{K+1})}}{dQ_1}(y)\frac{dQ_1}{dQ_{K+1}}(y). \tag*{(287)} d Q K + 1 d P Y ∣ X = x K + 1 ( h K + 1 , Q K + 1 , λ K + 1 ) ( y ) = d Q 1 d P Y ∣ X = x K + 1 ( h K + 1 , Q K + 1 , λ K + 1 ) ( y ) d Q K + 1 d Q 1 ( y ) . (287)
Moreover, from the multiplicative inverse of the Radon–Nikodym derivative [60 Theorem 5] , by combining (285) and (287) , for all y ∈ supp Q 1 y \in\operatorname{supp} Q_1 y ∈ supp Q 1 , it follows that
d P Y ∣ X = x K + 1 ( h K + 1 , Q K + 1 , λ K + 1 ) d Q 1 ( y ) = d Q K + 1 d Q 1 ( y ) exp ( − 1 λ K + 1 h K + 1 ( x K + 1 , y ) ) ∫ exp ( − 1 λ K + 1 h K + 1 ( x K + 1 , y ~ ) ) d Q K + 1 ( y ~ ) (288) \frac{dP_{Y\mid X=x_{K+1}}^{(h_{K+1},Q_{K+1},\lambda_{K+1})}}{dQ_1}(y)=\frac{dQ_{K+1}}{dQ_1}(y)\frac{\exp\left(-\frac{1}{\lambda_{K+1}}h_{K+1}(x_{K+1},y)\right)}{\int\exp\left(-\frac{1}{\lambda_{K+1}}h_{K+1}(x_{K+1},\widetilde y)\right)dQ_{K+1}(\widetilde y)} \tag*{(288)} d Q 1 d P Y ∣ X = x K + 1 ( h K + 1 , Q K + 1 , λ K + 1 ) ( y ) = d Q 1 d Q K + 1 ( y ) ∫ exp ( − λ K + 1 1 h K + 1 ( x K + 1 , y ) ) d Q K + 1 ( y ) exp ( − λ K + 1 1 h K + 1 ( x K + 1 , y ) ) (288)
= d Q K + 1 d Q 1 ( y ) exp ( − 1 λ K + 1 h K + 1 ( x K + 1 , y ) ) ∫ exp ( − 1 λ K + 1 h K + 1 ( x K + 1 , y ~ ) ) d Q K + 1 d Q 1 ( y ~ ) d Q 1 ( y ~ ) (289) =\frac{dQ_{K+1}}{dQ_1}(y)\frac{\exp\left(-\frac{1}{\lambda_{K+1}}h_{K+1}(x_{K+1},y)\right)}{\int\exp\left(-\frac{1}{\lambda_{K+1}}h_{K+1}(x_{K+1},\widetilde y)\right)\frac{dQ_{K+1}}{dQ_1}(\widetilde y)dQ_1(\widetilde y)} \tag*{(289)} = d Q 1 d Q K + 1 ( y ) ∫ exp ( − λ K + 1 1 h K + 1 ( x K + 1 , y ) ) d Q 1 d Q K + 1 ( y ) d Q 1 ( y ) exp ( − λ K + 1 1 h K + 1 ( x K + 1 , y ) ) (289)
= d P Y ∣ X = x K ( h K , Q K , λ K ) d Q 1 ( y ) exp ( − 1 λ K + 1 h K + 1 ( x K + 1 , y ) ) ∫ exp ( − 1 λ K + 1 h K + 1 ( x K + 1 , y ~ ) ) d P Y ∣ X = x K ( h K , Q K , λ K ) d Q 1 ( y ~ ) d Q 1 ( y ~ ) (290) =\frac{dP_{Y\mid X=x_K}^{(h_K,Q_K,\lambda_K)}}{dQ_1}(y)\frac{\exp\left(-\frac{1}{\lambda_{K+1}}h_{K+1}(x_{K+1},y)\right)}{\int\exp\left(-\frac{1}{\lambda_{K+1}}h_{K+1}(x_{K+1},\widetilde y)\right)\frac{dP_{Y\mid X=x_K}^{(h_K,Q_K,\lambda_K)}}{dQ_1}(\widetilde y)dQ_1(\widetilde y)} \tag*{(290)} = d Q 1 d P Y ∣ X = x K ( h K , Q K , λ K ) ( y ) ∫ exp ( − λ K + 1 1 h K + 1 ( x K + 1 , y ) ) d Q 1 d P Y ∣ X = x K ( h K , Q K , λ K ) ( y ) d Q 1 ( y ) exp ( − λ K + 1 1 h K + 1 ( x K + 1 , y ) ) (290)
= exp ( − ∑ i = 1 K 1 λ i h i ( x i , y ) ) ∫ exp ( − ∑ i = 1 K 1 λ i h i ( x i , y ~ ) ) d Q 1 ( y ~ ) exp ( − 1 λ K + 1 h K + 1 ( x K + 1 , y ) ) ∫ exp ( − 1 λ K + 1 h K + 1 ( x K + 1 , y ~ ) ) exp ( − ∑ i = 1 K 1 λ i h i ( x i , y ~ ) ) ∫ exp ( − ∑ i = 1 K 1 λ i h i ( x i , y ~ ) ) d Q 1 ( y ~ ) d Q 1 ( y ~ ) (291) =\frac{\frac{\exp\left(-\sum_{i=1}^{K}\frac{1}{\lambda_i}h_i(x_i,y)\right)}{\int\exp\left(-\sum_{i=1}^{K}\frac{1}{\lambda_i}h_i(x_i,\widetilde y)\right)dQ_1(\widetilde y)}}{\frac{\exp\left(-\frac{1}{\lambda_{K+1}}h_{K+1}(x_{K+1},y)\right)}{\int\exp\left(-\frac{1}{\lambda_{K+1}}h_{K+1}(x_{K+1},\widetilde y)\right)\frac{\exp\left(-\sum_{i=1}^{K}\frac{1}{\lambda_i}h_i(x_i,\widetilde y)\right)}{\int\exp\left(-\sum_{i=1}^{K}\frac{1}{\lambda_i}h_i(x_i,\widetilde y)\right)dQ_1(\widetilde y)}dQ_1(\widetilde y)}} \tag*{(291)} = ∫ e x p ( − λ K + 1 1 h K + 1 ( x K + 1 , y ) ) ∫ e x p ( − ∑ i = 1 K λ i 1 h i ( x i , y ) ) d Q 1 ( y ) e x p ( − ∑ i = 1 K λ i 1 h i ( x i , y ) ) d Q 1 ( y ) e x p ( − λ K + 1 1 h K + 1 ( x K + 1 , y ) ) ∫ e x p ( − ∑ i = 1 K λ i 1 h i ( x i , y ) ) d Q 1 ( y ) e x p ( − ∑ i = 1 K λ i 1 h i ( x i , y ) ) (291)
= exp ( − ∑ i = 1 K + 1 1 λ i h i ( x i , y ) ) ∫ exp ( − ∑ i = 1 K + 1 1 λ i h i ( x i , y ~ ) ) d Q 1 ( y ~ ) , (292) =\frac{\exp\left(-\sum_{i=1}^{K+1}\frac{1}{\lambda_i}h_i(x_i,y)\right)}{\int\exp\left(-\sum_{i=1}^{K+1}\frac{1}{\lambda_i}h_i(x_i,\widetilde y)\right)dQ_1(\widetilde y)}, \tag*{(292)} = ∫ exp ( − ∑ i = 1 K + 1 λ i 1 h i ( x i , y ) ) d Q 1 ( y ) exp ( − ∑ i = 1 K + 1 λ i 1 h i ( x i , y ) ) , (292)
where the equality in (289) follows from a change of measure of the Radon–Nikodym derivative [60 Theorem 2] ; the equality in (290) follows from the definition of Q K + 1 Q_{K+1} Q K + 1 in (51) ; the equality in (291) follows from the induction hypothesis in (284) and the equality in (292) follows by simplifying the common normalizing factor. Thus, (51) holds for K + 1 K+1 K + 1 . This concludes the proof of (51).
It remains to prove (52). From the definition of β \boldsymbol{\beta} β in the statement of Theorem 11 and the definition of the function h ^ β \widehat h_{\boldsymbol{\beta}} h β in (35) , for all y ∈ supp Q 1 y \in\operatorname{supp} Q_1 y ∈ supp Q 1 , it follows that
h ^ β ( x , y ) = ∑ i = 1 K 1 λ i h i ( x i , y ) . (293) \widehat h_{\boldsymbol{\beta}}(\boldsymbol{x},y)=\sum_{i=1}^{K}\frac{1}{\lambda_i}h_i(x_i,y). \tag*{(293)} h β ( x , y ) = i = 1 ∑ K λ i 1 h i ( x i , y ) . (293)
Therefore, from Definition 3,
d P Y ∣ X = x ( h ^ β , Q 1 , 1 ) d Q 1 ( y ) = exp ( − h ^ β ( x , y ) ) ∫ exp ( − h ^ β ( x , y ~ ) ) d Q 1 ( y ~ ) \frac{dP_{Y\mid X=\boldsymbol{x}}^{(\widehat h_{\boldsymbol{\beta}},Q_1,1)}}{dQ_1}(y)=\frac{\exp\left(-\widehat h_{\boldsymbol{\beta}}(\boldsymbol{x},y)\right)}{\int\exp\left(-\widehat h_{\boldsymbol{\beta}}(\boldsymbol{x},\widetilde y)\right)dQ_1(\widetilde y)} d Q 1 d P Y ∣ X = x ( h β , Q 1 , 1 ) ( y ) = ∫ exp ( − h β ( x , y ) ) d Q 1 ( y ) exp ( − h β ( x , y ) )
exp ( − ∑ i = 1 K 1 λ i h i ( x i , y ) ) ∫ exp ( − ∑ i = 1 K 1 λ i h i ( x i , y ~ ) ) d Q 1 ( y ~ ) (294) \frac{\exp\left(-\sum_{i=1}^{K}\frac{1}{\lambda_i}h_i(x_i,y)\right)}{\int\exp\left(-\sum_{i=1}^{K}\frac{1}{\lambda_i}h_i(x_i,\widetilde{y})\right)\,dQ_1(\widetilde{y})} \tag*{(294)} ∫ exp ( − ∑ i = 1 K λ i 1 h i ( x i , y ) ) d Q 1 ( y ) exp ( − ∑ i = 1 K λ i 1 h i ( x i , y ) ) (294)
Combining (51) and (280) yields
d P Y ∣ X = x K ( h K , Q K , λ K ) d Q 1 ( y ) = d P Y ∣ X = x ( h ^ β , Q 1 , 1 ) d Q 1 ( y ) , (295) \frac{dP_{Y\mid X=x_K}^{(h_K,Q_K,\lambda_K)}}{dQ_1}(y)=\frac{dP_{Y\mid X=x}^{(\widehat{h}_{\boldsymbol{\beta},Q_1,1})}}{dQ_1}(y), \tag*{(295)} d Q 1 d P Y ∣ X = x K ( h K , Q K , λ K ) ( y ) = d Q 1 d P Y ∣ X = x ( h β , Q 1 , 1 ) ( y ) , (295)
which proves (52). This completes the proof.
Proof of Theorem 12
From [15 Lemma 3] , the nested Gibbs probability measure P Y ∣ X = x K ( h K , Q K , λ K ) P_{Y\mid X=x_K}^{(h_K,Q_K,\lambda_K)} P Y ∣ X = x K ( h K , Q K , λ K ) in (50) is mutually absolutely continuous with its reference measure Q K Q_K Q K . Moreover, from (50b) and [15 Lemma 3] , it follows that
P Y ∣ X = x K ( h K , Q K , λ K ) ≪ Q 1 ≪ Q 2 ≪ ⋯ ≪ Q K − 1 ≪ Q K ≪ P Y ∣ X = x K ( h K , Q K , λ K ) . (296) P_{Y\mid X=x_K}^{(h_K,Q_K,\lambda_K)}\ll Q_1\ll Q_2\ll\cdots\ll Q_{K-1}\ll Q_K\ll P_{Y\mid X=x_K}^{(h_K,Q_K,\lambda_K)}. \tag*{(296)} P Y ∣ X = x K ( h K , Q K , λ K ) ≪ Q 1 ≪ Q 2 ≪ ⋯ ≪ Q K − 1 ≪ Q K ≪ P Y ∣ X = x K ( h K , Q K , λ K ) . (296)
Moreover, taking into account that P Y ∣ X = x ( h ^ β , Q 1 , 1 ) P_{Y\mid X=x}^{(\widehat{h}_{\boldsymbol{\beta},Q_1,1})} P Y ∣ X = x ( h β , Q 1 , 1 ) in (52) is a Gibbs probability measure with reference measure Q 1 Q_1 Q 1 , from [15 Lemma 3] , it also follows that
Q 1 ≪ P Y ∣ X = x ( h ^ β , Q 1 , 1 ) ≪ Q 1 . (297) Q_1\ll P_{Y\mid X=x}^{(\widehat{h}_{\boldsymbol{\beta},Q_1,1})}\ll Q_1. \tag*{(297)} Q 1 ≪ P Y ∣ X = x ( h β , Q 1 , 1 ) ≪ Q 1 . (297)
Thus, from [60 Theorem 2] , the Radon–Nikodym derivative
d Q 1 d P Y ∣ X = x ( h ^ β , Q 1 , 1 ) \frac{dQ_1}{dP_{Y\mid X=x}^{(\widehat{h}_{\boldsymbol{\beta},Q_1,1})}} d P Y ∣ X = x ( h β , Q 1 , 1 ) d Q 1
is well defined and from the multiplicative inverse of the Radon–Nikodym derivative [60 Theorem 5] , for all y ∈ supp Q 1 y\in\operatorname{supp}Q_1 y ∈ supp Q 1 , it follows that
d Q 1 d P Y ∣ X = x ( h ^ β , Q 1 , 1 ) ( y ) = ( d P Y ∣ X = x ( h ^ β , Q 1 , 1 ) d Q 1 ( y ) ) − 1 . (298) \frac{dQ_1}{dP_{Y\mid X=x}^{(\widehat{h}_{\boldsymbol{\beta},Q_1,1})}}(y)=\left(\frac{dP_{Y\mid X=x}^{(\widehat{h}_{\boldsymbol{\beta},Q_1,1})}}{dQ_1}(y)\right)^{-1}. \tag*{(298)} d P Y ∣ X = x ( h β , Q 1 , 1 ) d Q 1 ( y ) = d Q 1 d P Y ∣ X = x ( h β , Q 1 , 1 ) ( y ) − 1 . (298)
Therefore, from Theorem 11 and the equality in (284), for all y ∈ supp Q 1 y\in\operatorname{supp}Q_1 y ∈ supp Q 1 ,
d P Y ∣ X = x K ( h K , Q K , λ K ) d P Y ∣ X = x ( h ^ β , Q 1 , 1 ) ( y ) = d P Y ∣ X = x K ( h K , Q K , λ K ) d Q 1 ( y ) d Q 1 d P Y ∣ X = x ( h ^ β , Q 1 , 1 ) ( y ) (299) \frac{dP_{Y\mid X=x_K}^{(h_K,Q_K,\lambda_K)}}{dP_{Y\mid X=x}^{(\widehat{h}_{\boldsymbol{\beta},Q_1,1})}}(y)
=
\frac{dP_{Y\mid X=x_K}^{(h_K,Q_K,\lambda_K)}}{dQ_1}(y)
\frac{dQ_1}{dP_{Y\mid X=x}^{(\widehat{h}_{\boldsymbol{\beta},Q_1,1})}}(y) \tag*{(299)} d P Y ∣ X = x ( h β , Q 1 , 1 ) d P Y ∣ X = x K ( h K , Q K , λ K ) ( y ) = d Q 1 d P Y ∣ X = x K ( h K , Q K , λ K ) ( y ) d P Y ∣ X = x ( h β , Q 1 , 1 ) d Q 1 ( y ) (299)
= 1 , (300) =1, \tag*{(300)} = 1 , (300)
where the equality in (285) follows from the chain rule of the Radon–Nikodym derivative [60 Theorem 4] ; and the equality in (286) follows from Theorem 11 . This proves ( i ) (i) ( i ) .
Assume that ( i ) (i) ( i ) holds. Then,
D ( P Y ∣ X = x K ( h K , Q K , λ K ) ∥ P Y ∣ X = x ( h ^ β , Q 1 , 1 ) ) = ∫ log ( d P Y ∣ X = x K ( h K , Q K , λ K ) d P Y ∣ X = x ( h ^ β , Q 1 , 1 ) ( y ) ) d P Y ∣ X = x K ( h K , Q K , λ K ) ( y ) = 0 , (301) D\left(P_{Y\mid X=x_K}^{(h_K,Q_K,\lambda_K)}\middle\Vert P_{Y\mid X=x}^{(\widehat{h}_{\boldsymbol{\beta},Q_1,1})}\right)
=
\int\log\left(\frac{dP_{Y\mid X=x_K}^{(h_K,Q_K,\lambda_K)}}{dP_{Y\mid X=x}^{(\widehat{h}_{\boldsymbol{\beta},Q_1,1})}}(y)\right)dP_{Y\mid X=x_K}^{(h_K,Q_K,\lambda_K)}(y)
=0, \tag*{(301)} D ( P Y ∣ X = x K ( h K , Q K , λ K ) P Y ∣ X = x ( h β , Q 1 , 1 ) ) = ∫ log d P Y ∣ X = x ( h β , Q 1 , 1 ) d P Y ∣ X = x K ( h K , Q K , λ K ) ( y ) d P Y ∣ X = x K ( h K , Q K , λ K ) ( y ) = 0 , (301)
which proves ( i i ) (ii) ( ii ) . Conversely, if ( i i ) (ii) ( ii ) holds, then from [15 Theorem 1] , it follows that
d P Y ∣ X = x K ( h K , Q K , λ K ) d P Y ∣ X = x ( h β ^ , Q 1 , 1 ) ( y ) = 1. (302) \frac{\mathrm{d}P_{Y\mid X=x_K}^{(h_K,Q_K,\lambda_K)}}{\mathrm{d}P_{Y\mid X=x}^{(\widehat{h_\beta},Q_1,1)}}(y)=1. \tag*{(302)} d P Y ∣ X = x ( h β , Q 1 , 1 ) d P Y ∣ X = x K ( h K , Q K , λ K ) ( y ) = 1. (302)
Therefore, ( i ) (i) ( i ) and ( i i ) (ii) ( ii ) are equivalent.
Assume next that ( i ) (i) ( i ) holds. Then P Y ∣ X = x K ( h K , Q K , λ K ) P_{Y\mid X=x_K}^{(h_K,Q_K,\lambda_K)} P Y ∣ X = x K ( h K , Q K , λ K ) is identical to the ( h β ^ , Q 1 , 1 ) (\widehat{h_\beta},Q_1,1) ( h β , Q 1 , 1 ) -Gibbs probability measure
P Y ∣ X = x ( h β ^ , Q 1 , 1 ) . P_{Y\mid X=x}^{(\widehat{h_\beta},Q_1,1)}. P Y ∣ X = x ( h β , Q 1 , 1 ) .
Hence, from Lemma 1, it is the unique solution, if it exists, to the optimization problem in (55). This proves that ( i ) (i) ( i ) implies ( i i i ) (iii) ( iii ) .
Assume that ( i i i ) (iii) ( iii ) holds. From Lemma 1, the unique solution, if it exists, to the optimization problem in (55) is precisely the ( h β ^ , Q 1 , 1 ) (\widehat{h_\beta},Q_1,1) ( h β , Q 1 , 1 ) -Gibbs probability measure
P Y ∣ X = x ( h β ^ , Q 1 , 1 ) . P_{Y\mid X=x}^{(\widehat{h_\beta},Q_1,1)}. P Y ∣ X = x ( h β , Q 1 , 1 ) .
Hence,
P Y ∣ X = x K ( h K , Q K , λ K ) = P Y ∣ X = x ( h β ^ , Q 1 , 1 ) . (303) P_{Y\mid X=x_K}^{(h_K,Q_K,\lambda_K)}=P_{Y\mid X=x}^{(\widehat{h_\beta},Q_1,1)}. \tag*{(303)} P Y ∣ X = x K ( h K , Q K , λ K ) = P Y ∣ X = x ( h β , Q 1 , 1 ) . (303)
Thus,
D ( P Y ∣ X = x K ( h K , Q K , λ K ) ∥ Q 1 ) = D ( P Y ∣ X = x ( h β ^ , Q 1 , 1 ) ∥ Q 1 ) . (304) D\left(P_{Y\mid X=x_K}^{(h_K,Q_K,\lambda_K)}\mathbin{\|}Q_1\right)=D\left(P_{Y\mid X=x}^{(\widehat{h_\beta},Q_1,1)}\mathbin{\|}Q_1\right). \tag*{(304)} D ( P Y ∣ X = x K ( h K , Q K , λ K ) ∥ Q 1 ) = D ( P Y ∣ X = x ( h β , Q 1 , 1 ) ∥ Q 1 ) . (304)
Taking into account that P Y ∣ X = x ( h β ^ , Q 1 , 1 ) P_{Y\mid X=x}^{(\widehat{h_\beta},Q_1,1)} P Y ∣ X = x ( h β , Q 1 , 1 ) is a ( h β ^ , Q 1 , 1 ) (\widehat{h_\beta},Q_1,1) ( h β , Q 1 , 1 ) -Gibbs probability measure, Lemma 2 implies that it is the unique solution, if it exists, to the optimization problem in (56). Therefore, P Y ∣ X = x K ( h K , Q K , λ K ) P_{Y\mid X=x_K}^{(h_K,Q_K,\lambda_K)} P Y ∣ X = x K ( h K , Q K , λ K ) is the unique solution, if it exists, to the same optimization problem. This proves that ( i i i ) (iii) ( iii ) implies ( i v ) (iv) ( i v ) .
Finally, assume that ( i v ) (iv) ( i v ) holds. From Lemma 2, the unique solution, if it exists, to the optimization problem in (56) is precisely the ( h β ^ , Q 1 , 1 ) (\widehat{h_\beta},Q_1,1) ( h β , Q 1 , 1 ) -Gibbs probability measure
P Y ∣ X = x ( h β ^ , Q 1 , 1 ) . P_{Y\mid X=x}^{(\widehat{h_\beta},Q_1,1)}. P Y ∣ X = x ( h β , Q 1 , 1 ) .
Hence,
P Y ∣ X = x K ( h K , Q K , λ K ) = P Y ∣ X = x ( h β ^ , Q 1 , 1 ) . (305) P_{Y\mid X=x_K}^{(h_K,Q_K,\lambda_K)}=P_{Y\mid X=x}^{(\widehat{h_\beta},Q_1,1)}. \tag*{(305)} P Y ∣ X = x K ( h K , Q K , λ K ) = P Y ∣ X = x ( h β , Q 1 , 1 ) . (305)
Therefore, for all y ∈ supp Q 1 y\in\operatorname{supp}Q_1 y ∈ supp Q 1 ,
d P Y ∣ X = x K ( h K , Q K , λ K ) d P Y ∣ X = x ( h β ^ , Q 1 , 1 ) ( y ) = 1. (306) \frac{\mathrm{d}P_{Y\mid X=x_K}^{(h_K,Q_K,\lambda_K)}}{\mathrm{d}P_{Y\mid X=x}^{(\widehat{h_\beta},Q_1,1)}}(y)=1. \tag*{(306)} d P Y ∣ X = x ( h β , Q 1 , 1 ) d P Y ∣ X = x K ( h K , Q K , λ K ) ( y ) = 1. (306)
This proves that ( i v ) (iv) ( i v ) implies ( i ) (i) ( i ) .
Therefore, ( i ) (i) ( i ) , ( i i ) (ii) ( ii ) , ( i i i ) (iii) ( iii ) , and ( i v ) (iv) ( i v ) are equivalent. This concludes the proof.
Proof Of Lemma 15
For all i ∈ { 1 , 2 , … , K − 1 } i\in\{1,2,\ldots,K-1\} i ∈ { 1 , 2 , … , K − 1 } , using [27 Lemma 3] with the Gibbs probability measures P Y ∣ X = x i ( h i , Q i , λ i ) P_{Y\mid X=x_i}^{(h_i,Q_i,\lambda_i)} P Y ∣ X = x i ( h i , Q i , λ i ) and P Y ∣ X = x K ( h K , Q K , λ K ) P_{Y\mid X=x_K}^{(h_K,Q_K,\lambda_K)} P Y ∣ X = x K ( h K , Q K , λ K ) , it follows that
1 λ i ( ∫ h i ( x i , y ) d P Y ∣ X = x K ( h K , Q K , λ K ) ( y ) − ∫ h i ( x i , y ) d Q i + 1 ( y ) + D ( P Y ∣ X = x K ( h K , Q K , λ K ) ∥ Q i ) ) = D ( Q i + 1 ∥ Q i ) + D ( P Y ∣ X = x K ( h K , Q K , λ K ) ∥ Q i + 1 ) . (307) \begin{aligned}
\frac{1}{\lambda_i}\biggl(&\int h_i(x_i,y)\,\mathrm{d}P_{Y\mid X=x_K}^{(h_K,Q_K,\lambda_K)}(y)-\int h_i(x_i,y)\,\mathrm{d}Q_{i+1}(y)+D\left(P_{Y\mid X=x_K}^{(h_K,Q_K,\lambda_K)}\mathbin{\|}Q_i\right)\biggr)\\
&=D(Q_{i+1}\mathbin{\|}Q_i)+D\left(P_{Y\mid X=x_K}^{(h_K,Q_K,\lambda_K)}\mathbin{\|}Q_{i+1}\right). \tag*{(307)}
\end{aligned} λ i 1 ( ∫ h i ( x i , y ) d P Y ∣ X = x K ( h K , Q K , λ K ) ( y ) − ∫ h i ( x i , y ) d Q i + 1 ( y ) + D ( P Y ∣ X = x K ( h K , Q K , λ K ) ∥ Q i ) ) = D ( Q i + 1 ∥ Q i ) + D ( P Y ∣ X = x K ( h K , Q K , λ K ) ∥ Q i + 1 ) . (307)
where Q i + 1 = P Y ∣ X = x i ( h i , Q i , λ i ) Q_{i+1}=P_{Y\mid X=x_i}^{(h_i,Q_i,\lambda_i)} Q i + 1 = P Y ∣ X = x i ( h i , Q i , λ i ) from (50b). Using the definition of δ i \delta_i δ i in (73), the equality in (293) can be written as
D ( P Y ∣ X = x K ( h K , Q K , λ K ) ∥ Q i ) = D ( P Y ∣ X = x i ( h i , Q i , λ i ) ∥ Q i ) − δ i + D ( P Y ∣ X = x K ( h K , Q K , λ K ) ∥ Q i + 1 ) . (308) D\left(P_{Y\mid X=x_K}^{(h_K,Q_K,\lambda_K)}\mathbin{\|}Q_i\right)=D\left(P_{Y\mid X=x_i}^{(h_i,Q_i,\lambda_i)}\mathbin{\|}Q_i\right)-\delta_i+D\left(P_{Y\mid X=x_K}^{(h_K,Q_K,\lambda_K)}\mathbin{\|}Q_{i+1}\right). \tag*{(308)} D ( P Y ∣ X = x K ( h K , Q K , λ K ) ∥ Q i ) = D ( P Y ∣ X = x i ( h i , Q i , λ i ) ∥ Q i ) − δ i + D ( P Y ∣ X = x K ( h K , Q K , λ K ) ∥ Q i + 1 ) . (308)
Hence, from (308) , it follows that
D ( P Y ∣ X = x K ( h K , Q K , λ K ) ∥ Q 1 ) = D ( P Y ∣ X = x 1 ( h 1 , Q 1 , λ 1 ) ∥ Q 1 ) − δ 1 + D ( P Y ∣ X = x K ( h K , Q K , λ K ) ∥ Q 2 ) (309) D\left(P^{(h_K,Q_K,\lambda_K)}_{Y\mid X=x_K}\mathbin{\|}Q_1\right)=D\left(P^{(h_1,Q_1,\lambda_1)}_{Y\mid X=x_1}\mathbin{\|}Q_1\right)-\delta_1+D\left(P^{(h_K,Q_K,\lambda_K)}_{Y\mid X=x_K}\mathbin{\|}Q_2\right) \tag*{(309)} D ( P Y ∣ X = x K ( h K , Q K , λ K ) ∥ Q 1 ) = D ( P Y ∣ X = x 1 ( h 1 , Q 1 , λ 1 ) ∥ Q 1 ) − δ 1 + D ( P Y ∣ X = x K ( h K , Q K , λ K ) ∥ Q 2 ) (309)
= D ( P Y ∣ X = x 1 ( h 1 , Q 1 , λ 1 ) ∥ Q 1 ) − δ 1 + D ( P Y ∣ X = x 2 ( h 2 , Q 2 , λ 2 ) ∥ Q 2 ) − δ 2 + D ( P Y ∣ X = x K ( h K , Q K , λ K ) ∥ Q 3 ) (310) =D\left(P^{(h_1,Q_1,\lambda_1)}_{Y\mid X=x_1}\mathbin{\|}Q_1\right)-\delta_1+D\left(P^{(h_2,Q_2,\lambda_2)}_{Y\mid X=x_2}\mathbin{\|}Q_2\right)-\delta_2+D\left(P^{(h_K,Q_K,\lambda_K)}_{Y\mid X=x_K}\mathbin{\|}Q_3\right) \tag*{(310)} = D ( P Y ∣ X = x 1 ( h 1 , Q 1 , λ 1 ) ∥ Q 1 ) − δ 1 + D ( P Y ∣ X = x 2 ( h 2 , Q 2 , λ 2 ) ∥ Q 2 ) − δ 2 + D ( P Y ∣ X = x K ( h K , Q K , λ K ) ∥ Q 3 ) (310)
= ∑ i = 1 2 ( D ( P Y ∣ X = x i ( h i , Q i , λ i ) ∥ Q i ) − δ i ) + D ( P Y ∣ X = x 3 ( h 3 , Q 3 , λ 3 ) ∥ Q 3 ) − δ 3 + D ( P Y ∣ X = x K ( h K , Q K , λ K ) ∥ Q 4 ) (311) =\sum_{i=1}^{2}\left(D\left(P^{(h_i,Q_i,\lambda_i)}_{Y\mid X=x_i}\mathbin{\|}Q_i\right)-\delta_i\right)+D\left(P^{(h_3,Q_3,\lambda_3)}_{Y\mid X=x_3}\mathbin{\|}Q_3\right)-\delta_3+D\left(P^{(h_K,Q_K,\lambda_K)}_{Y\mid X=x_K}\mathbin{\|}Q_4\right) \tag*{(311)} = i = 1 ∑ 2 ( D ( P Y ∣ X = x i ( h i , Q i , λ i ) ∥ Q i ) − δ i ) + D ( P Y ∣ X = x 3 ( h 3 , Q 3 , λ 3 ) ∥ Q 3 ) − δ 3 + D ( P Y ∣ X = x K ( h K , Q K , λ K ) ∥ Q 4 ) (311)
⋮ = ∑ i = 1 K − 2 ( D ( P Y ∣ X = x i ( h i , Q i , λ i ) ∥ Q i ) − δ i ) + D ( P Y ∣ X = x K − 1 ( h K − 1 , Q K − 1 , λ K − 1 ) ∥ Q K − 1 ) − δ K − 1 + D ( P Y ∣ X = x K ( h K , Q K , λ K ) ∥ Q K ) (312) \begin{aligned}
\vdots\\
&=\sum_{i=1}^{K-2}\left(D\left(P^{(h_i,Q_i,\lambda_i)}_{Y\mid X=x_i}\mathbin{\|}Q_i\right)-\delta_i\right)+D\left(P^{(h_{K-1},Q_{K-1},\lambda_{K-1})}_{Y\mid X=x_{K-1}}\mathbin{\|}Q_{K-1}\right)-\delta_{K-1}
+D\left(P^{(h_K,Q_K,\lambda_K)}_{Y\mid X=x_K}\mathbin{\|}Q_K\right) \tag*{(312)}
\end{aligned} ⋮ = i = 1 ∑ K − 2 ( D ( P Y ∣ X = x i ( h i , Q i , λ i ) ∥ Q i ) − δ i ) + D ( P Y ∣ X = x K − 1 ( h K − 1 , Q K − 1 , λ K − 1 ) ∥ Q K − 1 ) − δ K − 1 + D ( P Y ∣ X = x K ( h K , Q K , λ K ) ∥ Q K ) (312)
= ∑ k = 1 K D ( P Y ∣ X = x k ( h k , Q k , λ k ) ∥ Q k ) − ∑ i = 1 K − 1 δ i , (313) =\sum_{k=1}^{K}D\left(P^{(h_k,Q_k,\lambda_k)}_{Y\mid X=x_k}\mathbin{\|}Q_k\right)-\sum_{i=1}^{K-1}\delta_i, \tag*{(313)} = k = 1 ∑ K D ( P Y ∣ X = x k ( h k , Q k , λ k ) ∥ Q k ) − i = 1 ∑ K − 1 δ i , (313)
where the equality in (310) follows by applying (308) to the term D ( P Y ∣ X = x K ( h K , Q K , λ K ) ∥ Q 2 ) D\left(P^{(h_K,Q_K,\lambda_K)}_{Y\mid X=x_K}\mathbin{\|}Q_2\right) D ( P Y ∣ X = x K ( h K , Q K , λ K ) ∥ Q 2 ) in (309) ; the equality in (311) follows by applying (308) to the term D ( P Y ∣ X = x K ( h K , Q K , λ K ) ∥ Q 3 ) D\left(P^{(h_K,Q_K,\lambda_K)}_{Y\mid X=x_K}\mathbin{\|}Q_3\right) D ( P Y ∣ X = x K ( h K , Q K , λ K ) ∥ Q 3 ) in (310) and rearranging the terms; the equality in (312) follows by successively applying (308) to the last relative entropy until the application with i = K − 1 i=K-1 i = K − 1 , which yields D ( P Y ∣ X = x K ( h K , Q K , λ K ) ∥ Q K ) D\left(P^{(h_K,Q_K,\lambda_K)}_{Y\mid X=x_K}\mathbin{\|}Q_K\right) D ( P Y ∣ X = x K ( h K , Q K , λ K ) ∥ Q K ) as the last relative entropy; and the equality in (313) follows by rearranging the terms. This completes the proof.
Appendix I: Proof Of Lemma 32
Taking into consideration that z 0 z_0 z 0 is the aggregation of the K K K local datasets, for all θ ∈ M \theta\in\mathcal{M} θ ∈ M , it follows from (84) that
∑ i = 1 K − 1 n i L n i ( z i , θ ) = n 0 L n 0 ( z 0 , θ ) − n K L n K ( z K , θ ) . (314) \sum_{i=1}^{K-1}n_i\mathcal{L}_{n_i}(z_i,\theta)=n_0\mathcal{L}_{n_0}(z_0,\theta)-n_K\mathcal{L}_{n_K}(z_K,\theta). \tag*{(314)} i = 1 ∑ K − 1 n i L n i ( z i , θ ) = n 0 L n 0 ( z 0 , θ ) − n K L n K ( z K , θ ) . (314)
Hence,
∑ i = 1 K − 1 n i R z i ( P Θ ∣ Z = z 0 ( L n 0 , Q 1 , λ 0 ) ) = ∫ ( ∑ i = 1 K − 1 n i L n i ( z i , θ ) ) d P Θ ∣ Z = z 0 ( L n 0 , Q 1 , λ 0 ) ( θ ) (315) \sum_{i=1}^{K-1}n_iR_{z_i}\left(P^{(L_{n_0},Q_1,\lambda_0)}_{\Theta\mid Z=z_0}\right)=\int\left(\sum_{i=1}^{K-1}n_i\mathcal{L}_{n_i}(z_i,\theta)\right)dP^{(L_{n_0},Q_1,\lambda_0)}_{\Theta\mid Z=z_0}(\theta) \tag*{(315)} i = 1 ∑ K − 1 n i R z i ( P Θ ∣ Z = z 0 ( L n 0 , Q 1 , λ 0 ) ) = ∫ ( i = 1 ∑ K − 1 n i L n i ( z i , θ ) ) d P Θ ∣ Z = z 0 ( L n 0 , Q 1 , λ 0 ) ( θ ) (315)
= n 0 R z 0 ( P Θ ∣ Z = z 0 ( L n 0 , Q 1 , λ 0 ) ) − n K R z K ( P Θ ∣ Z = z 0 ( L n 0 , Q 1 , λ 0 ) ) (316) =n_0R_{z_0}\left(P^{(L_{n_0},Q_1,\lambda_0)}_{\Theta\mid Z=z_0}\right)-n_KR_{z_K}\left(P^{(L_{n_0},Q_1,\lambda_0)}_{\Theta\mid Z=z_0}\right) \tag*{(316)} = n 0 R z 0 ( P Θ ∣ Z = z 0 ( L n 0 , Q 1 , λ 0 ) ) − n K R z K ( P Θ ∣ Z = z 0 ( L n 0 , Q 1 , λ 0 ) ) (316)
= n 0 R z 0 ( P Θ ∣ Z = z 0 ( L n 0 , Q 1 , λ 0 ) ) − n K R z K ( P Θ ∣ Z = z K ( L n K , Q K , λ K ) ) , (317) =n_0R_{z_0}\left(P^{(L_{n_0},Q_1,\lambda_0)}_{\Theta\mid Z=z_0}\right)-n_KR_{z_K}\left(P^{(L_{n_K},Q_K,\lambda_K)}_{\Theta\mid Z=z_K}\right), \tag*{(317)} = n 0 R z 0 ( P Θ ∣ Z = z 0 ( L n 0 , Q 1 , λ 0 ) ) − n K R z K ( P Θ ∣ Z = z K ( L n K , Q K , λ K ) ) , (317)
where the equality in (315) follows from (85) and the linearity of the integral; the equality in (316) follows from (314) ; and the equality in (317) follows from (170) . Substituting (317) into (181) yields
∑ k = 1 K γ k − η = 1 λ 0 ( R z 0 ( P Θ ∣ Z = z 0 ( L n 0 , Q 1 , λ 0 ) ) − ∑ i = 1 K n i n 0 R z i ( P Θ ∣ Z = z i ( L n i , Q i , λ i ) ) ) , (318) \sum_{k=1}^{K}\gamma_k-\eta=\frac{1}{\lambda_0}\left(R_{z_0}\left(P^{(L_{n_0},Q_1,\lambda_0)}_{\Theta\mid Z=z_0}\right)-\sum_{i=1}^{K}\frac{n_i}{n_0}R_{z_i}\left(P^{(L_{n_i},Q_i,\lambda_i)}_{\Theta\mid Z=z_i}\right)\right), \tag*{(318)} k = 1 ∑ K γ k − η = λ 0 1 ( R z 0 ( P Θ ∣ Z = z 0 ( L n 0 , Q 1 , λ 0 ) ) − i = 1 ∑ K n 0 n i R z i ( P Θ ∣ Z = z i ( L n i , Q i , λ i ) ) ) , (318)
which completes the proof.