betaFeedback

WorksMathematics

A Unified Framework for Training-Dataset Aggregation and Exact Machine Unlearning via Gibbs Probability Measures

Abstract

In this paper, three operations on Gibbs probability measures are studied: renormalization, normalized log-linear combination, and nesting. The resulting measures are shown to be Gibbs probability measures with explicit expressions for their objective functions, reference measures, and regularization factors. These measures are further characterized as unique solutions to optimization problems involving linear combinations of the original objective functions, subject to either relative entropy regularization or relative entropy constraints. When the reference measures are probability measures, geometric interpretations of these three operations are established involving Pythagorean identities and exponential geodesics.A central result is the equivalence between nesting and normalized log-linear combinations of Gibbs probability measures. These results are then applied to supervised machine learning with Gibbs algorithms, which are represented by Gibbs probability measures. In this context, either through nesting or log-linear combinations, Gibbs algorithms trained on local datasets are used to form new Gibbs algorithms trained upon the aggregation of all local training datasets. Interestingly, the parameters of such operations, determine the weights of each training data point on the aggregated dataset. %Operations on local Gibbs probability measures are shown to control the contributions of training data. This observation is leveraged for achieving: (a) exact machine unlearning in centralized, decentralized, and federated learning systems; and (b) centralized-equivalent performance, in the sense of producing algorithms trained on the aggregation of all local datasets in decentralized and federated learning .

Introduction

Gibbs probability measures play an important role in statistical learning theory and information theory, in part due to their versatility for modeling different learning scenarios [7, 8, 9, 10, 11, 12, 13, 14, 15, 16]. For instance, under suitable conditions, Gibbs measures describe the long-run behavior of stochastic-gradient-based learning algorithms with time-invariant learning rates [17]. Moreover, Gibbs algorithms are represented by data-dependent Gibbs probability measures on the set of models. Interestingly, these algorithms provide a benchmark for the analysis of the generalization error of learning algorithms [18, 19, 20, 21, 22]. Gibbs probability measures on the set of datasets also characterize worst-case data-generating probability distributions for a fixed model [23]. More generally, Gibbs probability measures naturally arise as solutions to the minimization or maximization of the expectation of an objective function, such as the empirical risk, subject to a relative entropy regularization. In such a case, the relative entropy is defined with respect to a reference measure, and a regularization factor controls the tradeoff between the contribution of the objective function and that of the regularization. Similarly, under a specific choice of regularization factor, the same Gibbs probability measure also solves the corresponding minimization or maximization problem subject to a relative entropy constraint [15, 24, 25, 26, 27]. These variational characterizations relate an objective function, a reference measure and a regularization factor to the resulting Gibbs probability measures. This relation allows jointly studying the effects of the regularization factor and the reference measure on the resulting Gibbs algorithm [10, 12, 15]. It also motivates the study of operations that transform or combine Gibbs probability measures, with the aim of determining how the objective functions, reference measures, and regularization factors affect the resulting measure and its associated optimization problems. The present work studies three such operations: renormalization, normalized log-linear combinations, and nesting. The analysis makes explicit how these operations act on the objective functions, reference measures, and regularization factors, and establishes the associated optimization problems and connections between the operations. Geometric representations are also developed when the reference measures are probability measures. These results are specialized to supervised machine learning, where connections with weighted empirical risk are established. Particular emphasis is placed on exact machine unlearning and centralized-equivalent performance, in the sense of training on the aggregation of all local datasets in decentralized and federated learning systems.

Yaiza Bermudez is with the Centre Inria d’Université Côte d’Azur, INRIA, 06902 Sophia Antipolis, France (e-mail: [email protected]).

Samir M. Perlaza is with the College of Computing, University Mohammed VI Polytechnic University (UM6P), Ben Guerir, 43150, Morocco; also with the Centre Inria d’Université Côte d’Azur, INRIA, 06902 Sophia Antipolis, France; also with the Department of Electrical and Computer Engineering at Princeton University, Princeton, NJ 08544 USA; and also with the GAATI Mathematics Laboratory at the University of French Polynesia, 98702 Faaa, French Polynesia (e-mail: [email protected]).

Iñaki Esnaola is with the School of Electrical and Electronic Engineering, University of Sheffield, Sheffield S1 3JD, U.K.; and also with the Department of Electrical and Computer Engineering, Princeton University, Princeton, NJ 08544 USA (e-mail: [email protected]).

Parts of this work have been presented at IEEE International Symposium on Information Theory (ISIT), Guangzhou, China, Jun., 2026 [1, 2]; at the IEEE Information Theory Workshop (ITW), Tempe, Arizona, USA, Nov. 2026 [3]; and appear as Inria Technical Reports in [4, 5, 6].

Operations on Gibbs Probability Measures

The first operation, referred to as renormalization, consists of normalizing a power of the Radon–Nikodym derivative of a probability measure. This operation is frequently encountered in statistical physics and information theory, and the resulting measure is often referred to as an escort distribution. See for instance [28, 29, 30, 31] and references therein. In this work, the focus is on the renormalization of a given Gibbs probability measure, in the particular case in which the measure used for the normalization is allowed to differ from the reference measure of the given measure. In this work, Theorem 4 shows that the resulting measure is a Gibbs probability measure with the same objective function, a modified regularization factor, and a possibly different reference measure. These two changes can be determined independently, and the change of reference measure may restrict the support of the resulting Gibbs measure. Hence, renormalization describes how a Gibbs probability measure can be transformed through changes in its regularization factor and reference measure.

The second operation, referred to as normalized log-linear combination, extends renormalization to several probability measures. More specifically, this operation consists of normalizing a product of powers of Radon–Nikodym derivatives. Related constructions have been studied beyond Gibbs measures in [28], [32], [33]. In this work, Theorem 7 provides an explicit expression for the normalized log-linear combination of several Gibbs probability measures. The resulting measure is a Gibbs probability measure whose objective function is a linear combination of the original objective functions, with coefficients determined by the powers used in the operation and the regularization factors. This explicit expression is complemented by Theorem 8, which establishes equivalent formulations for the resulting measure and its role on some specific regularized and constrained optimization problems.

The third operation, referred to as nesting, consists of constructing a Gibbs probability measure whose reference measure is itself a Gibbs probability measure. This construction also appears in sequential updating of Gibbs posteriors in [15] and [34]. In this work, Theorem 11 shows that sequentially nesting several Gibbs probability measures leads to another Gibbs probability measure whose objective function is a linear combination of the objective functions of the original measures. Theorem 12 establishes the corresponding equivalent variational characterizations of such a resulting measure. The connection with normalized log-linear combinations is then made explicit by Corollary 13, which establishes the equivalence between the two operations under specific conditions.

These operations also admit geometric representations when the reference measures are probability measures. These representations build on the geometry of escort distributions and relative entropy [24], [28]. In particular, variations of the parameters of renormalizations and normalized log-linear combinations are shown to induce exponential geodesics. For nested Gibbs probability measures, identities relating relative entropies to differences of expectations admit Pythagorean interpretations. Lemma 15 establishes such an identity for multiple nested Gibbs probability measures.

Supervised Machine Learning

In this work, when the operations described above are applied in the context of supervised machine learning, Gibbs probability measures (algorithms) are defined using empirical risks associated with training datasets as objective functions. Normalized log-linear combinations and nesting then yield Gibbs probability measures associated with weighted empirical risks on the aggregation of the corresponding training datasets. For normalized log-linear combinations, the weights are determined by the coefficients of the combination and the regularization factors of the Gibbs algorithms being combined. Alternatively, for nested Gibbs algorithms, the weights are determined by the regularization factors, and the dependence on previously used datasets is carried by the data-dependent reference measures. Thus, the parameters of the corresponding operation determine the weights assigned to the data points within the empirical risk with respect to local datasets. These parameters can, for instance, be chosen such that the resulting Gibbs probability measure is identical to the measure obtained as the unique solution to an empirical risk minimization with relative entropy regularization (ERM-RER) with respect to the aggregation of the local datasets. Corollary 30 establishes this result in a decentralized learning setting through a sequence of nested Gibbs algorithms. In such a decentralized setting, each client uses the Gibbs probability measure received from the previous client as its reference measure, and the regularization factors are chosen according to the local dataset sizes. Alternatively, in a federated learning setting, such a result is established by Corollary 35. In this case, local Gibbs probability measures are combined at a central server through a normalized log-linear combination. Interestingly, nesting and normalized log-linear combinations yield the same Gibbs probability measure as centralized training, thereby establishing the condition for centralized performance guarantees in decentralized and federated learning systems, respectively. Nonetheless, the connection with weighted empirical risks is not restricted to reproducing centralized training. More generally, these operations provide a way to control the weight assigned to each data point through their parameters and its membership to a specific dataset. In particular, the contribution of selected data points can be canceled through a suitable choice of these parameters, which leads to applications to exact machine unlearning.

In this work, the term machine unlearning refers to the task of removing the influence of part of the training dataset from a learning algorithm while seeking to avoid retraining from scratch [35, 36, 37]. Early formulations of data deletion in learning compare the updated algorithm with the algorithm trained exclusively upon the retained dataset [38], [39]. In exact unlearning, the probability distributions on the models induced by these two algorithms are required to be identical. Other formulations allow approximate removal, subject to guarantees on the discrepancy between these distributions [40], [41]. This work studies exact unlearning of datasets from a given Gibbs algorithm in centralized learning, as well as in decentralized and federated learning systems. In the decentralized and federated case, clients collaborate without explicitly exchanging their local datasets [42], [43].

The connection of nesting and normalized log-linear combinations with weighted empirical risks provides two mechanisms for exact machine unlearning. In both cases, the parameters of the operation are chosen such that the contribution of the data points to be unlearned is canceled. Hence, the remaining objective is the empirical risk on the retained dataset, up to a scaling accounted for by the regularization factor.

In this context, the first mechanism uses nesting and consists of maximizing the expected empirical risk on the dataset to be unlearned with relative entropy regularization with respect to the Gibbs algorithm from which data must be unlearnt. Corollary 27 shows that a particular choice of the regularization factors yields the same Gibbs probability measure as training exclusively upon the retained dataset. Only the dataset to be unlearned is explicitly used in this optimization problem, while the dependence on the retained dataset is carried by the reference measure. Together with Corollary 30, this result provides a mechanism for exact unlearning in decentralized learning, where the contribution of a local dataset is canceled from the given Gibbs probability measure.

The second mechanism uses a normalized log-linear combination of the Gibbs probability measure trained upon the full training dataset and a Gibbs probability measure trained upon the dataset to be unlearned. Corollary 34 establishes conditions on the coefficients of the combination and the regularization factors under which the resulting measure achieves exact unlearning. Alternatively, the unlearning application to federated learning is established by Corollary 37.

Organization of the Paper

The remainder of the paper is organized as follows. Section II introduces the notation, Gibbs conditional probability measures, and their variational characterizations. Section III studies renormalization, normalized log-linear combinations, and nesting, together with their associated optimization problems and the connections between these operations. Section IV develops their geometric representations when the reference measures are probability measures. Section V specializes these results to supervised machine learning, and Section VI presents their applications to exact machine unlearning. Finally, Section VII concludes the paper.

Preliminaries

This section describes the notation used in this work and formally introduces Gibbs probability measures and Gibbs conditional probability measures. More importantly, the connection between such measures and a particular class of optimization problems involving relative entropies is thoroughly presented.

Notation

Given a set X⊆Rd\mathcal{X} \subseteq\mathbb{R}^{d}, for some d∈Nd \in\mathbb{N}, the Borel σ\sigma-field defined on X\mathcal{X} is denoted by B(X)\mathcal{B}(\mathcal{X}). The set of probability measures that can be defined on the measurable space (X,B(X))(\mathcal{X},\mathcal{B}(\mathcal{X})) is denoted by Δ(X)\Delta(\mathcal{X}). Given a σ\sigma-finite measure QQ on (X,B(X))(\mathcal{X},\mathcal{B}(\mathcal{X})), the set of probability measures in Δ(X)\Delta(\mathcal{X}) that are absolutely continuous with respect to QQ is denoted by ΔQ(X)\Delta_{Q}(\mathcal{X}). Given a measure P∈ΔQ(X)P \in\Delta_{Q}(\mathcal{X}), the notation P≪QP \ll Q stands for “the measure PP is absolutely continuous with respect to QQ”. Thus,

ΔQ(X)≜{P∈Δ(X):P≪Q}.(1)\Delta_{Q}(\mathcal{X}) \triangleq\{P \in\Delta(\mathcal{X}) : P \ll Q\}. \tag*{(1)}

The Radon-Nikodym derivative of P∈ΔQ(X)P \in\Delta_{Q}(\mathcal{X}) with respect to QQ is a map X→[0,+∞)\mathcal{X} \to[0,+\infty), denoted by dPdQ\frac{\mathrm{d}P}{\mathrm{d}Q}. See for instance [44 Section 2.2]. Using this notation, the relative entropy or Kullback-Leibler (KL) divergence is defined as follows.

Definition 1 (Relative Entropy). Given a probability measure PP and a σ\sigma-finite measure QQ, both on the same measurable space, with P≪QP \ll Q the relative entropy of PP with respect to QQ is

D(P∥Q)≜∫dPdQ(x)log⁡(dPdQ(x))dQ(x).(2)D(P \parallel Q) \triangleq\int\frac{\mathrm{d}P}{\mathrm{d}Q}(x)\log\left(\frac{\mathrm{d}P}{\mathrm{d}Q}(x)\right)\mathrm{d}Q(x). \tag*{(2)}

Note that when QQ in Definition 1 is not a probability measure, the relative entropy might be negative. In particular, when QQ is the corresponding Lebesgue measure, the relative entropy D(P∥Q)D(P \parallel Q) becomes negative Shannon’s differential entropy [45 Section 8.3].

In this work, conditional probability measures are distinguished from probability measures in the following sense.

Definition 2 (Conditional Probability). A family PY∣X≜(PY∣X=x)x∈XP_{Y\mid X} \triangleq(P_{Y\mid X=x})_{x\in\mathcal{X}} of elements of Δ(Y)\Delta(\mathcal{Y}) indexed by X\mathcal{X} is said to be a conditional probability measure if, for all measurable sets B∈B(Y)\mathcal{B} \in\mathcal{B}(\mathcal{Y}), the map

X⟶[0,1]x⟼PY∣X=x(B)(3)\begin{aligned} \mathcal{X} &\longrightarrow[0,1]\\ x &\longmapsto P_{Y\mid X=x}(\mathcal{B}) \tag*{(3)} \end{aligned}

is Borel measurable. The set of all such conditional probability measures is denoted by Δ(Y∣X)\Delta(\mathcal{Y}\mid\mathcal{X}).

In a nutshell, while PY∣XP_{Y\mid X} is a conditional probability measure, the measure PY∣X=xP_{Y\mid X=x}, obtained by conditioning on a given x∈Xx \in\mathcal{X}, is a probability measure.

Gibbs Conditional Probability Measures

Consider two sets X⊂Rd1\mathcal{X} \subset\mathbb{R}^{d_1} and Y⊂Rd2\mathcal{Y} \subset\mathbb{R}^{d_2}, for some given integers d1d_1 and d2d_2. Gibbs conditional probability measures in △(Y∣X)\triangle(\mathcal{Y}\mid\mathcal{X}) are parametrized by a Borel measurable function

h:X×Y→R;(4)h:\mathcal{X}\times\mathcal{Y}\to\mathbb{R}; \tag*{(4)}

a σ\sigma-finite measure QQ on the measurable space (Y,B(Y))(\mathcal{Y},\mathcal{B}(\mathcal{Y})); and a real λ∈R∖{0}\lambda\in\mathbb{R}\setminus\{0\}. Using these parameters, Gibbs conditional probability measures are defined as follows.

Definition 3 (Gibbs Conditional Probability Measures). Given a Borel measurable function h:X×Y→Rh:\mathcal{X}\times\mathcal{Y}\to\mathbb{R}; a σ\sigma-finite measure QQ on the measurable space (Y,B(Y))(\mathcal{Y},\mathcal{B}(\mathcal{Y})); and a real λ∈R∖{0}\lambda\in\mathbb{R}\setminus\{0\}, the conditional probability measure PY∣X(h,Q,λ)∈△(Y∣X)P^{(h,Q,\lambda)}_{Y\mid X}\in\triangle(\mathcal{Y}\mid\mathcal{X}) is said to be an (h,Q,λ)(h,Q,\lambda)-Gibbs conditional probability measure if

∀x∈X, 0<∫exp⁡(−1λh(x,y))dQ(y)<+∞;(5)\forall x\in\mathcal{X},\ 0<\int\exp\left(-\frac{1}{\lambda}h(x,y)\right)dQ(y)<+\infty; \tag*{(5)}

and for all (x,y)∈X×supp⁡Q(x,y)\in\mathcal{X}\times\operatorname{supp}Q,

dPY∣X=x(h,Q,λ)dQ(y)=exp⁡(−1λh(x,y))∫exp⁡(−1λh(x,y~))dQ(y~).(6)\frac{dP^{(h,Q,\lambda)}_{Y\mid X=x}}{dQ}(y)=\frac{\exp\left(-\frac{1}{\lambda}h(x,y)\right)}{\int\exp\left(-\frac{1}{\lambda}h(x,\widetilde{y})\right)dQ(\widetilde{y})}. \tag*{(6)}

While PY∣X(h,Q,λ)P^{(h,Q,\lambda)}_{Y\mid X} is an (h,Q,λ)(h,Q,\lambda)-Gibbs conditional probability measure, the measure PY∣X=x(h,Q,λ)P^{(h,Q,\lambda)}_{Y\mid X=x}, obtained by conditioning on a given x∈Xx\in\mathcal{X}, is an (h,Q,λ)(h,Q,\lambda)-Gibbs probability measure.

The relevance of (h,Q,λ)(h,Q,\lambda)-Gibbs probability measures stems from the fact that under some conditions, they are the unique solutions to particular optimization problems involving relative entropy. The following lemma highlights such a fact.

Lemma 1. When λ>0\lambda>0, the (h,Q,λ)(h,Q,\lambda)-Gibbs probability measure PY∣X=x(h,Q,λ)P^{(h,Q,\lambda)}_{Y\mid X=x} in (6) is the unique solution, if it exists, to

min⁡P∈△Q(Y)∫h(x,y)dP(y)+λD(P∥Q).(7)\min_{P\in\triangle_Q(\mathcal{Y})}\int h(x,y)dP(y)+\lambda D(P\parallel Q). \tag*{(7)}

Alternatively, when λ<0\lambda<0, such a measure PY∣X=x(h,Q,λ)P^{(h,Q,\lambda)}_{Y\mid X=x} is the unique solution, if it exists, to

max⁡P∈△Q(Y)∫h(x,y)dP(y)+λD(P∥Q).(8)\max_{P\in\triangle_Q(\mathcal{Y})}\int h(x,y)dP(y)+\lambda D(P\parallel Q). \tag*{(8)}

Proof: The proof is immediate from [27 Lemma 1].

Two optimization problems that are also closely related to the probability measure PY∣X=x(h,Q,λ)P^{(h,Q,\lambda)}_{Y\mid X=x} in (6) are the following:

min⁡P∈△Q(Y)∫h(x,y)dP(y)s.t.D(P∥Q)≤γ;(9)\begin{aligned} \min_{P\in\triangle_Q(\mathcal{Y})}\quad&\int h(x,y)dP(y)\\ \text{s.t.}\quad&D(P\parallel Q)\leq\gamma; \tag*{(9)} \end{aligned}

and

max⁡P∈△Q(Y)∫h(x,y)dP(y)s.t.D(P∥Q)≤γ,(10)\begin{aligned} \max_{P\in\triangle_Q(\mathcal{Y})}\quad&\int h(x,y)dP(y)\\ \text{s.t.}\quad&D(P\parallel Q)\leq\gamma, \tag*{(10)} \end{aligned}

for some γ∈R\gamma\in\mathbb{R}. The following lemma formalizes these relations.

Lemma 2 ([27 Lemma 4]). Assume that the optimization problem in (9) (respectively, in (10)) admits a solution. Consider the (h,Q,λ)(h,Q,\lambda)-Gibbs probability measure PY∣X=x(h,Q,λ)P^{(h,Q,\lambda)}_{Y\mid X=x} in (6) and assume that λ>0\lambda>0 (respectively, λ<0\lambda<0) is chosen such that

D(PY∣X=x(h,Q,λ)∥Q)=γ.(11)D\left(P^{(h,Q,\lambda)}_{Y\mid X=x}\parallel Q\right)=\gamma. \tag*{(11)}

Then, the (h,Q,λ)(h,Q,\lambda)-Gibbs probability measure PY∣X=x(h,Q,λ)P^{(h,Q,\lambda)}_{Y\mid X=x} is the unique solution to (9) (respectively, to (10)).

A property of the (h,Q,λ)(h,Q,\lambda)-Gibbs probability measure PY∣X=x(h,Q,λ)P^{(h,Q,\lambda)}_{Y\mid X=x} in (6) that plays an important role in this work concerns the variation of the relative entropy D(PY∣X=x(h,Q,λ)∥Q)D\left(P^{(h,Q,\lambda)}_{Y\mid X=x}\parallel Q\right) due to changes in the parameter λ\lambda. The following lemma characterizes such a variation.

References

  1. [1]Y. Bermudez, S. M. Perlaza, and I. Esnaola, “Machine unlearning for Gibbs supervised learning algorithms,” in Proceedings of the IEEE International Symposium on Information Theory (ISIT), Guangzhou, China, Jun. 2026.
  2. [2]——, “Decentralized machine learning with centralized performance guarantees via Gibbs algorithms,” in Proceedings of the IEEE International Symposium on Information Theory (ISIT), Guangzhou, China, Jun. 2026.
  3. [3]——, “Equivalence between nested Gibbs measures and log- linear combinations of Gibbs measures,” in Proceedings of the IEEE Information Theory Workshop (ITW), Tempe AZ, USA, Nov. 2026, pp. 1–6.arxiv.org/abs/2609.19988
  4. [4]——, “Decentralized machine learning with centralized performance guarantees via Gibbs algorithms,” INRIA, Centre Inria d’Université Côte d’Azur, Sophia Antipolis, France, Tech. Rep. RR-9608, Jan. 2026.
  5. [5]——, “Machine unlearning for Gibbs supervised learning algorithms,” INRIA, Centre Inria d’Université Côte d’Azur, Sophia Antipolis, France, Tech. Rep. RR-9610, Jan. 2026.
  6. [6]——, “Equivalence between nested Gibbs measures and log-linear combinations of Gibbs measures,” INRIA, Centre Inria d’Université Côte d’Azur, Sophia Antipolis, France, Tech. Rep. RR-9613, May 2026.arxiv.org/abs/2609.19988
  7. [7]A. Xu and M. Raginsky, “Information-theoretic analysis of generalization capability of learning algorithms,” in Proceedings of the International Conference on Neural Information Processing Systems (NeurIPS), vol. 30, Long Beach, CA, USA, Dec. 2017, pp. 2521–2530.arxiv.org/abs/1705.07809
  8. [8]D. Russo and J. Zou, “Controlling bias in adaptive data analysis using information theory,” in Proceedings of the 19th International Conference on Artificial Intelligence and Statistics, vol. 51, Cadiz, Spain, May 2016, pp. 1232–1240.
  9. [9]M. Avella Medina, J. L. Montiel Olea, C. Rush, and A. Velez, “On the robustness to misspecification of α-posteriors and their variational approximations,” Journal of Machine Learning Research, vol. 23, no. 147, pp. 1–51, 2022.DOI
  10. [10]Y. Bu, “Towards optimal inverse systems in the Gibbs algorithm,” in Proceedings of the IEEE International Symposium on Information Theory (ISIT), Athens, Greece, Jul. 2024, pp. 2257–2262.DOI
  11. [11]Y. Bu, H. V. Tetali, G. Aminian, M. Rodrigues, and G. W. Wornell, “On the generalization error of meta learning for the Gibbs algorithm,” in Proceedings of the IEEE International Symposium on Information Theory (ISIT), Taipei, Taiwan, Jun. 2023, pp. 2488–2493.arxiv.org/abs/2304.14332
  12. [12]S. M. Perlaza, I. Esnaola, G. Bisson, and H. V. Poor, “On the validation of Gibbs algorithms: Training datasets, test datasets and their aggregation,” in Proceedings of the IEEE International Symposium on Information Theory (ISIT), Taipei, Taiwan, Jun. 2023, pp. 328–333.arxiv.org/abs/2306.12380
  13. [13]O. Catoni, PAC-Bayesian Supervised Classification: The Thermodynamics of Statistical Learning, 1st ed., ser. IMS Lecture Notes–Monograph Series. Beachwood, OH, USA: Institute of Mathematical Statistics, 2007, vol. 56.DOI
  14. [14]R. Ray, M. A. Medina, and C. Rush, “Asymptotics for power posterior mean estimation,” in Proceedings of the 59th Annual Allerton Conference on Communication, Control, and Computing (Allerton), Monticello, IL, USA, Sep. 2023, pp. 1–8.DOI
  15. [15]S. M. Perlaza, G. Bisson, I. Esnaola, A. Jean-Marie, and S. Rini, “Empirical risk minimization with relative entropy regularization,” IEEE Transactions on Information Theory, vol. 70, no. 7, pp. 5122–5161, Jul. 2024.
  16. [16]B. Rodríguez-Gálvez, “An information-theoretic approach to generalization theory,” Ph.D. dissertation, KTH Royal Institute of Technology, 2024.
  17. [17]W. Azizian, F. Lutzeier, J. Malick, and P. Mertikopoulos, “What is the long-run distribution of stochastic gradient descent? A large deviations analysis,” in Proceedings of the International Conference on Machine Learning (ICML), Vienna, Austria, Jul. 2024, pp. 2168–2229.arxiv.org/abs/2406.09241
  18. [18]G. Aminian, Y. Bu, L. Toni, M. Rodrigues, and G. Wornell, “An exact characterization of the generalization error for the Gibbs algorithm,” in Proceedings of the International Conference on Neural Information Processing Systems (NeurIPS), vol. 34, Virtual Event, Dec. 2021, pp. 8106–8118.
  19. [19]G. Aminian, Y. Bu, L. Toni, M. R. D. Rodrigues, and G. W. Wornell, “Information-theoretic characterizations of generalization error for the Gibbs algorithm,” IEEE Transactions on Information Theory, vol. 70, no. 1, pp. 632–655, Jan. 2024.arxiv.org/abs/2210.09864
  20. [20]S. M. Perlaza and X. Zou, “The method of gaps: Exact expressions for the generalization error of supervised learning algorithms,” To appear in IEEE Transactions on Information Theory, 2026.arxiv.org/abs/2411.12030
  21. [21]S. M. Perlaza, “Corrections to “The method of gaps: Exact expressions for the generalization error of supervised learning algorithms”,” Submitted to the IEEE Transactions on Information Theory, 2026.arxiv.org/abs/2610.26245
  22. [22]X. Zou, S. M. Perlaza, I. Esnaola, and E. Altman, “Generalization analysis of machine learning algorithms via the worst-case data-generating probability measure,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 38, no. 15, Vancouver, Canada, Feb. 2024, pp. 17271–17279.
  23. [23]X. Zou, S. M. Perlaza, I. Esnaola, A. Altman, and H. V. Poor, “The worst-case data-generating probability measure in statistical learning,” IEEE Journal on Selected Areas in Information Theory, vol. 5, pp. 175–189, Apr. 2024.
  24. [24]I. Csiszár, “I-divergence geometry of probability distributions and minimization problems,” The Annals of Probability, vol. 3, no. 1, pp. 146–158, Feb. 1975.DOI
  25. [25]F. Daunas, I. Esnaola, S. M. Perlaza, and H. V. Poor, “Equivalence of empirical risk minimization to regularization on the family of f-divergences,” in Proceedings of the IEEE International Symposium on Information Theory (ISIT), Athens, Greece, Jul. 2024, pp. 759–764.
  26. [26]——, “Asymmetry of the relative entropy in the regularization of empirical risk minimization,” IEEE Transactions on Information Theory, vol. 71, no. 8, pp. 6198–6226, Aug. 2025.
  27. [27]S. M. Perląza and G. Bisson, “Variations on the expectation due to changes in the probability measure,” Entropy, vol. 27, no. 8:865, pp. 1–20, Aug. 2025.arxiv.org/abs/2502.02887
  28. [28]J.-F. Bercher, “A simple probabilistic construction yielding generalized entropies and divergences, escort distributions and q-Gaussians,” Physica A: Statistical Mechanics and its Applications, vol. 391, no. 19, pp. 4460–4469, Oct. 2012.
  29. [29]C. Beck and F. Schlögl, Thermodynamics of Chaotic Systems: An Introduction, ser. Cambridge Nonlinear Science Series. Cambridge, UK: Cambridge University Press, 1993, vol. 4.
  30. [30]S. Abe, “Geometry of escort distributions,” Physical Review E, vol. 68, no. 3, p. 031101, Sep. 2003.DOI
  31. [31]A. Ohara, H. Matsuzoe, and S. Amari, “A dually flat structure on the space of escort distributions,” Journal of Physics: Conference Series, vol. 201, no. 1, p. 012012, Dec. 2010.DOI
  32. [32]T. Lahiri, T. Javidi, and A. D. Sarwate, “Social learning and distributed hypothesis testing,” IEEE Transactions on Information Theory, vol. 64, no. 9, pp. 6161–6179, Sep. 2018.
  33. [33]A. R. Asadi, “Hierarchical maximum entropy via the renormalization group,” arXiv preprint arXiv:2509.01424, Sep. 2025.arxiv.org/abs/2509.01424
  34. [34]P. G. Bissiri, C. C. Holmes, and S. G. Walker, “A general framework for updating belief distributions,” Journal of the Royal Statistical Society: Series B (Statistical Methodology), vol. 78, no. 5, pp. 1103–1130, 2016.
  35. [35]T. T. Nguyen, T. T. Huynh, Z. Ren, P. L. Nguyen, A. W.-C. Liew, H. Yin, and Q. V. H. Nguyen, “A survey of machine unlearning,” ACM Transactions on Intelligent Systems and Technology, vol. 16, no. 5, pp. 1–46, Oct. 2025.arxiv.org/abs/2209.02299
  36. [36]S. Sai, U. Mittal, V. Chamola, K. Huang, I. Spinelli, S. Scardapane, Z. Tan, and A. Hussain, “Machine un-learning: An overview of techniques, applications, and future directions,” Cognitive Computation, vol. 16, no. 2, pp. 482–506, Mar. 2024.DOI
  37. [37]J. Xu, Z. Wu, C. Wang, and X. Jia, “Machine unlearning: Solutions and challenges,” IEEE Transactions on Emerging Topics in Computational Intelligence, vol. 8, no. 3, pp. 2150–2168, Jun. 2024.DOI
  38. [38]A. Ginart, M. Guan, G. Valiant, and J. Y. Zou, “Making AI forget you: Data deletion in machine learning,” in Proceedings of the International Conference on Neural Information Processing Systems (NeurIPS), vol. 32. Vancouver, Canada: Curran Associates, Inc., 2019.arxiv.org/abs/1907.05012
  39. [39]Y. Cao and J. Yang, “Towards making systems forget with machine unlearning,” in Proceedings of the IEEE Symposium on Security and Privacy, San Jose, CA, USA, May 2015, pp. 463–480.DOI
  40. [40]C. Guo, T. Goldstein, E. P. Schwarzer, and L. Van Der Maaten, “Certified data removal from machine learning models,” in Proceedings of the 37th International Conference on Machine Learning (ICML), vol. 119, Virtual Event, Jul. 2020, pp. 3832–3842.arxiv.org/abs/1911.03030
  41. [41]L. Bourtoule, V. Chandrasekaran, C. A. Choquette-Choo, H. Jia, A. Travers, B. Zhang, D. Lie, and N. Papernot, “Machine unlearning,” in Proceedings of the IEEE Symposium on Security and Privacy (SP), San Francisco, CA, USA, May 2021, pp. 141–159.
  42. [42]P. Kairouz, B. H. McMahan, B. Avent, A. Bellet, M. Bennis, A. N. Bhagoji, K. Bonawitz, Z. Charles, G. Cormode, R. Cummings, R. G. L. d’Oliveira, S. E. Rouayheb, D. Evans, J. Gardner, Z. Garrett, A. Gascón, B. Ghazi, P. Gibbons, M. Gruteser, Z. Harchaoui, C. He, L. He, Z. Huo, B. Hutchinson, J. Hsu, M. Jaggi, T. Javidi, G. Joshi, M. Khodak, J. Konečný, A. Korolova, F. Koushanfar, S. Koyejo, T. Lepoint, Y. Liu, P. Mittal, M. Mohri, R. Nock, A. Özȩür, P. Pagh, M. Raykova, H. Qi, D. Ramage, R. Raskar, D. Song, W. Song, S. U. Stich, Z. Sun, A. T. Suresh, F. Tramer, P. Vepakomma, J. Wang, L. Xiong, Z. Xu, Q. Yang, F. X. Yu, H. Yu, and S. Zhao, “Advances and open problems in federated learning,” Foundations and Trends in Machine Learning, vol. 14, no. 1–2, pp. 1–210, 2021.arxiv.org/abs/1912.04977
  43. [43]H. B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. Agüera y Arcas, “Communication-efficient learning of deep networks from decentralized data,” in Proceedings of the 20th International Conference on Artificial Intelligence and Statistics (AISTATS), ser. Proceedings of Machine Learning Research, vol. 54, Apr. 2017, pp. 1273–1282.arxiv.org/abs/1602.05629
  44. [44]R. B. Ash and C. A. Doleans-Dade, Probability and Measure Theory, 2nd ed. Burlington, MA, USA: Academic Press, 2000.
  45. [45]R. B. Ash, Information Theory, 1st ed. Mineola, NY, USA: Dover Publications, 1990.
  46. [46]K. G. Wilson, “The renormalization group and critical phenomena,” Reviews of Modern Physics, vol. 55, no. 3, pp. 583–600, Jul. 1983.DOI
  47. [47]A. Chhabra and R. V. Jensen, “Direct determination of the f(α) singularity spectrum,” Physical Review Letters, vol. 62, no. 12, pp. 1327–1330, Mar. 1989.pubmed.ncbi.nlm.nih.gov/10039645
  48. [48]J.-F. Bercher, “Source coding with escort distributions and Rényi entropy bounds,” Physics Letters A, vol. 373, no. 36, pp. 3235–3238, Aug. 2009.DOI
  49. [49]C. Tsallis, Introduction to Nonextensive Statistical Mechanics: Approaching a Complex World, 1st ed. New York, NY, USA: Springer, 2009.
  50. [50]S. Abe and G. B. Bagci, “Necessity of q-expectation value in nonextensive statistical mechanics,” Physical Review E, vol. 71, no. 1, p. 016139, Jan. 2005.pubmed.ncbi.nlm.nih.gov/15697690
  51. [51]E. Hellinger, “Neue begründung der theorie quadratischer formen von unendlichvielen veränderlichen,” Journal für die reine und angewandte Mathematik, vol. 136, pp. 210–271, 1909.DOI
  52. [52]A. Rényi, “On measures of entropy and information,” in Proceedings of the Fourth Berkeley Symposium on Mathematical Statistics and Probability, vol. 1, Berkeley, CA, USA, 1961, pp. 547–561.
  53. [53]T. van Erven and P. Harremoës, “Rényi divergence and kullback–leibler divergence,” IEEE Transactions on Information Theory, vol. 60, no. 7, pp. 3797–3820, 2014.
  54. [54]H. Chernoff, “A measure of asymptotic efficiency for tests of a hypothesis based on the sum of observations,” The Annals of Mathematical Statistics, vol. 23, no. 4, pp. 493–507, 1952.DOI
  55. [55]A. Bhattacharyya, “On a measure of divergence between two statistical populations defined by their probability distributions,” Bulletin of the Calcutta Mathematical Society, vol. 35, pp. 99–109, 1943.
  56. [56]F. Nielsen, “What is an information projection?” Notices of the American Mathematical Society, vol. 65, no. 3, pp. 321–324, 2018.DOI
  57. [57]S.-i. Amari, Information Geometry and Its Applications, ser. Applied Mathematical Sciences. Springer Japan, 2016, vol. 194.
  58. [58]F. Nielsen, “An elementary introduction to information geometry,” Entropy, vol. 22, no. 10, p. 1100, 2020.DOI
  59. [59]Euclid, The Thirteen Books of Euclid’s Elements, 2nd ed., T. L. Heath, Ed. New York, NY, USA: Dover Publications, Inc., 1956.
  60. [60]Y. Bermudez, G. Bisson, I. Esnaola, and S. M. Per laza, “Proofs for folklore theorem on the Radon-Nikodym derivative,” INRIA, Centre Inria d’Université Côte d’Azur, Sophia Antipolis, France, Tech. Rep. RR-9591, Jul. 2025.arxiv.org/abs/2501.18374

Paper details

vlnc:W_6wpk4ztu
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Mathematics

Not recorded for this paper

View statement
Looking for related papers…
Contents