betaFeedback

WorksMathematics

Transport, Entropy, and the Cost of Dependence: Maximum Conditional Entropy under Transport Constraints

Abstract

Maximum-entropy inference under Wasserstein constraints can be computationally challenging due to the underlying optimization over transport couplings. We investigate a tractable alternative based on maximizing conditional entropy under an average transport constraint. On Euclidean spaces, this formulation admits an explicit generalized-Gaussian additive-noise solution, providing a feasible approximation to the Wasserstein-constrained problem without solving an optimal transport problem. We establish quantitative bounds on the resulting optimality gap by showing that the two formulations admit a common feasible-marginal representation, with their discrepancy characterized by the mutual information retained between the nominal and perturbed variables. For discretely supported nominal distributions, we further obtain an exact decomposition of the gap into a marginal-entropy gain and a weighted Jensen-Shannon divergence.

Introduction

Selecting a probability distribution from incomplete information is a fundamental problem in statistical inference and statistical physics. When the available information does not uniquely determine a distribution, the maximum entropy principle (MEP) provides a systematic criterion for selecting the least informative distribution consistent with that information [2, 3, 12, 13, 23, 24]. Traditionally, the available information is expressed through moment or expectation constraints, leading to Gibbs-type distributions. In many settings, however, uncertainty is naturally described relative to a prescribed nominal distribution, with admissible distributions restricted to a neighborhood of that nominal law. Optimal transport provides a geometric framework for describing such neighborhoods through the Wasserstein distance, which quantifies the minimum cost required to transport one probability distribution into another [22, 27].

This motivates the study of maximum-entropy inference under Wasserstein constraints. Given a nominal distribution PX\mathbb{P}_{X}, the objective is to identify a maximally uncertain distribution PV\mathbb{P}_{V} whose Wasserstein distance from PX\mathbb{P}_{X} does not exceed a prescribed radius. Such a formulation combines the maximum entropy principle with a geometric description of distributional uncertainty. However, directly solving the resulting optimization problem can be challenging. The Wasserstein constraint is defined through an optimization over transport couplings, and characterizing the entropy-maximizing distribution generally requires solving a variational problem involving the induced marginal distribution and the associated transport geometry. This raises a natural question: can Wasserstein-constrained maximum-entropy inference be approximated through a computationally simpler formulation, while retaining quantitative guarantees on the loss of optimality?

One alternative is to impose the transport budget directly on the stochastic transition generating the perturbed variable. Given a nominal random variable XX, one seeks a conditional distribution PV∣X\mathbb{P}_{V\mid X} maximizing the conditional entropy of VV given XX, subject to a bound on the expected transport cost between XX and VV. This formulation is closely related to the maximum-caliber principle, originally proposed by Jaynes [14], which extends maximum-entropy reasoning to stochastic trajectories and dynamical laws subject to constraints on transition or path observables [10, 21, 25]. We refer to this formulation as the average-transport maximum entropy principle (A-MEP). Unlike the Wasserstein constraint, which involves minimizing the transport cost over all couplings between two distributions, the average-transport constraint is evaluated directly under the joint law induced by the perturbation kernel. This distinction leads to a particularly tractable optimization problem, on Euclidean spaces, A-MEP admits an explicit generalized-Gaussian additive-noise solution, including Gaussian and Laplace perturbations as special cases.

In contrast, Wasserstein-constrained maximum-entropy inference imposes the transport budget only on the marginal distribution induced by the perturbation. This distribution-level perspective is related to Wasserstein variational principles, including the Jordan–Kinderlehrer–Otto scheme [15], in which probability distributions are selected through variational problems involving Wasserstein geometry. We refer to the corresponding formulation as the Wasserstein maximum entropy principle (W-MEP). Importantly, the coupling certifying Wasserstein proximity need not coincide with the joint law generated by the entropy-bearing transition kernel. Consequently, although A-MEP and W-MEP maximize the same conditional-entropy objective, they impose different restrictions on the stochastic relationship between the nominal and perturbed variables.

The computational simplicity of A-MEP makes it an attractive surrogate for W-MEP. Indeed, every perturbation kernel satisfying the average-transport constraint induces a marginal distribution lying within the corresponding Wasserstein neighborhood of the nominal law. Thus, the explicit A-MEP solution provides a feasible construction for W-MEP without requiring the computation of an optimal transport coupling or the solution of the Wasserstein-constrained variational problem. Nevertheless, the two formulations need not yield the same optimal value or the same entropy-maximizing marginal distribution. Feasibility alone is therefore insufficient to establish the quality of the approximation. The central question addressed in this paper is: how accurately can the tractable A-MEP formulation approximate W-MEP, and how can the resulting loss of optimality be quantified?

We address this question through three main contributions:

  1. First, we characterize the optimal solutions of both formulations. On Euclidean spaces, A-MEP admits an explicit generalized-Gaussian additive-noise solution, providing a computationally tractable construction of a feasible distribution for W-MEP. For W-MEP, we derive a dual characterization in terms of a Kantorovich potential and show that the entropy-maximizing conditional kernel can be chosen independently of the nominal realization.

  2. Second, we establish that both formulations generate the same set of attainable marginal distributions and admit representations over a common feasible set. This reveals that their discrepancy is governed by the mutual information retained between the nominal and perturbed variables. We use this characterization to derive bounds on the entropy gap, including an explicit upper bound that provides a quantitative optimality guarantee for the A-MEP surrogate.

  3. Third, for finitely supported nominal distributions, we obtain finite-mixture representations and an exact decomposition of the entropy gap into a marginal-entropy gain and a weighted Jensen–Shannon divergence. A symmetric two-point example illustrates that the gap can be strict and that the formulations can yield different entropy-maximizing distributions.

.

Related connections between entropy and optimal transport arise in regularized optimal transport and the Schrödinger problem [4, 17, 20], while entropy-based selection over Wasserstein balls has also been studied through minimum-cross-entropy formulations [26]. These works differ from the question considered here. Rather than introducing an entropy regularization of the transport problem or studying relative-entropy projections onto Wasserstein neighborhoods, we investigate whether a maximum-entropy problem under an average transport constraint can serve as a tractable approximation to its Wasserstein-constrained counterpart. Our results establish that this approximation can be characterized and bounded through information-theoretic quantities, providing a connection between computational tractability, transport geometry, and the dependence retained by stochastic perturbations.

Problem setting

Throughout the paper, all random objects are defined on a common probability space (Ω,F,P)(\Omega,\mathcal{F},\mathbb{P}). Let (X,∥⋅∥)(\mathcal{X},\lVert\cdot\rVert) be a Polish space with Borel σ\sigma-algebra B(X)\mathcal{B}(\mathcal{X}), and let X,V:(Ω,F)→(X,B(X))X,V:(\Omega,\mathcal{F})\to(\mathcal{X},\mathcal{B}(\mathcal{X})) be random variables. Denote by P(X)\mathcal{P}(\mathcal{X}) the set of all Borel probability measures on (X,B(X))(\mathcal{X},\mathcal{B}(\mathcal{X})), and by Pp(X)\mathcal{P}_p(\mathcal{X}) the set of probability measures on X\mathcal{X} with finite pp-th moment,

Pp(X)={P∈P(X):∫X∥x∥p dP(x)<∞}.\mathcal{P}_p(\mathcal{X}) = \left\{ \mathbb{P} \in\mathcal{P}(\mathcal{X}) : \int_{\mathcal{X}} \lVert x \rVert^p \, \mathrm{d}\mathbb{P}(x) < \infty\right\}.

For a given a nominal distribution PX∈Pp(X)\mathbb{P}_X \in\mathcal{P}_p(\mathcal{X}), a perturbation is described by a stochastic kernel PV∣X\mathbb{P}_{V\mid X} from X\mathcal{X} to X\mathcal{X}. Together, PX\mathbb{P}_X and PV∣X\mathbb{P}_{V\mid X} induce the joint law

PX,V:=PX⊗PV∣X,\mathbb{P}_{X,V} := \mathbb{P}_X \otimes\mathbb{P}_{V\mid X},

defined by

PX,V(B×C)=∫BPV∣X=x(C) dPX(x),B,C∈B(X),\mathbb{P}_{X,V}(B \times C) = \int_B \mathbb{P}_{V\mid X=x}(C) \, \mathrm{d}\mathbb{P}_X(x), \qquad B,C \in\mathcal{B}(\mathcal{X}),

and its marginal with respect to VV, denoted by PV\mathbb{P}_V, satisfies

PV(C)=∫XPV∣X=x(C) dPX(x),C∈B(X).\mathbb{P}_V(C) = \int_{\mathcal{X}} \mathbb{P}_{V\mid X=x}(C) \, \mathrm{d}\mathbb{P}_X(x), \qquad C \in\mathcal{B}(\mathcal{X}).

We write

Kp(X)={PV∣X stochastic kernels on X:∫X∫X∥v∥p dPV∣X(v) dPX(x)<∞},\mathcal{K}_p(\mathcal{X}) = \left\{ \mathbb{P}_{V\mid X}\ \text{stochastic kernels on }\mathcal{X} : \int_{\mathcal{X}}\int_{\mathcal{X}} \lVert v \rVert^p \, \mathrm{d}\mathbb{P}_{V\mid X}(v) \, \mathrm{d}\mathbb{P}_X(x) < \infty\right\},

for the set of perturbation kernels whose induced marginal belongs to Pp(X)\mathcal{P}_p(\mathcal{X}). The pp-Wasserstein distance between probability measures PX,PV∈Pp(X)\mathbb{P}_X,\mathbb{P}_V \in\mathcal{P}_p(\mathcal{X}) is defined by

Wp(PX,PV)=(inf⁡π∈Π(PX,PV)∫X×X∥x−v∥p dπ(x,v))1/p.W_p(\mathbb{P}_X,\mathbb{P}_V) = \left( \inf_{\pi\in\Pi(\mathbb{P}_X,\mathbb{P}_V)} \int_{\mathcal{X}\times\mathcal{X}} \lVert x-v \rVert^p \, \mathrm{d}\pi(x,v) \right)^{1/p}.

where Π(PX,PV)\Pi(\mathbb{P}_X,\mathbb{P}_V) denotes the set of all couplings of PX\mathbb{P}_X and PV\mathbb{P}_V. A distinction used throughout the paper is that between the perturbation-induced joint law PX,V=PX⊗PV∣X\mathbb{P}_{X,V} = \mathbb{P}_X \otimes\mathbb{P}_{V\mid X} and a generic transport coupling of the same marginals π∈Π(PX,PV)\pi\in\Pi(\mathbb{P}_X,\mathbb{P}_V). Since X\mathcal{X} is a standard Borel space, such a coupling admits a disintegration [16 Thm. 8.5] of the form

dπ(x,v)=dPX(x) dQV∣X=x(v),\mathrm{d}\pi(x,v) = \mathrm{d}\mathbb{P}_X(x) \, \mathrm{d}\mathbb{Q}_{V\mid X=x}(v),

for some stochastic kernel QV∣X\mathbb{Q}_{V\mid X}. In general, this QV∣X\mathbb{Q}_{V\mid X} need not coincide with the perturbation kernel PV∣X\mathbb{P}_{V\mid X} even though both kernels induce the same marginal PV\mathbb{P}_V. Thus, PX,V\mathbb{P}_{X,V} is the coupling generated by the perturbation, whereas π\pi is a generic coupling used in the transport problem.

Entropy and mutual information. Entropy is not defined on an arbitrary Polish space without specifying additional structure. We therefore follow the standard information-theoretic convention of using HPX,V(V)H_{\mathbb{P}_{X,V}}(V) for Shannon entropy on finite or countable spaces and hPX,V(V)h_{\mathbb{P}_{X,V}}(V) for differential entropy on Euclidean spaces [2]. Mutual information is denoted by IPX,V(X;V)I_{\mathbb{P}_{X,V}}(X;V) in both settings. Throughout, the subscript identifies the joint probability law under which the corresponding information-theoretic quantity is evaluated; marginal and conditional laws are understood to be induced by this joint law. Thus, for example, hPX,V(V)h_{\mathbb{P}_{X,V}}(V) denotes the differential entropy of the VV-marginal of PX,V\mathbb{P}_{X,V}, while hPX,V(V∣X)h_{\mathbb{P}_{X,V}}(V\mid X) denotes the corresponding conditional differential entropy. For instance, if X\mathcal{X} is finite or countable, the Shannon entropy of V∼PVV \sim\mathbb{P}_V is

HPX,V(V):=−∑v∈XPV(v)log⁡PV(v).H_{\mathbb{P}_{X,V}}(V) := -\sum_{v \in\mathcal{X}} \mathbb{P}_V(v) \log\mathbb{P}_V(v).

If instead X⊆Rd\mathcal{X} \subseteq\mathbb{R}^d and PV\mathbb{P}_V admits a density pVp_V, its differential entropy is

hPX,V(V):=−∫XpV(v)log⁡pV(v) dv,h_{\mathbb{P}_{X,V}}(V) := -\int_{\mathcal{X}} p_V(v) \log p_V(v) \, \mathrm{d}v,

whenever the integral is well defined. Analogously, we also define the conditional entropy, in the discrete setting,

HPX,V(V∣X)=−∑x,vPX,V(x,v)log⁡PV∣X=x(v),H_{\mathbb{P}_{X,V}}(V\mid X) = -\sum_{x,v} \mathbb{P}_{X,V}(x,v) \log\mathbb{P}_{V\mid X=x}(v),

as well as in the Euclidean setting, if PV∣X=xP_{V\mid X=x} admits a density pV∣X=x(v)p_{V\mid X=x}(v) for PXP_X-almost every xx,

hPX,V(V∣X)=−∫X×Xlog⁡pV∣X=x(v) dPX,V(x,v).h_{P_X,V}(V\mid X)=-\int_{\mathcal{X}\times\mathcal{X}}\log p_{V\mid X=x}(v)\,\mathrm{d}P_{X,V}(x,v).

Furthermore, the mutual information is

IPX,V(X;V):=KL⁡(PX,V∥PX⊗PV).I_{P_X,V}(X;V):=\operatorname{KL}(P_{X,V}\mathbin{\|}P_X\otimes P_V).

with the Kullback–Leibler divergence,

KL⁡(P∥Q)={∫Ωlog⁡(dPdQ) dP,if P≪Q and log⁡(dPdQ)∈L1(P),∞,otherwise.\operatorname{KL}(P\mathbin{\|}Q)= \begin{cases} \int_{\Omega}\log\left(\frac{\mathrm{d}P}{\mathrm{d}Q}\right)\,\mathrm{d}P, & \text{if }P\ll Q\text{ and }\log\left(\frac{\mathrm{d}P}{\mathrm{d}Q}\right)\in L^1(P),\\ \infty, & \text{otherwise.} \end{cases}

Whenever the relevant quantities are well defined, the entropy decomposition

hPX,V(V∣X)=hPX,V(V)−IPX,V(X;V),h_{P_X,V}(V\mid X)=h_{P_X,V}(V)-I_{P_X,V}(X;V),

holds in the discrete and continuous Euclidean setting, respectively.

Problem setup

We now introduce the two maximum-entropy problems studied in this paper.

Maximum entropy under average transport constraint (A-MEP). Under the average-transport formulation, both entropy and transport cost are evaluated under the joint law generated by the perturbation kernel,

Aζ:=sup⁡PV∣X∈Kp(X)hPX,V(V∣X)s.t.EX,V[∥x−v∥pp]≤ζp.(1)A_{\zeta}:=\sup_{P_{V\mid X}\in\mathcal{K}_p(\mathcal{X})}h_{P_X,V}(V\mid X) \quad\text{s.t.}\quad \mathbb{E}_{X,V}\left[\|x-v\|_p^p\right]\leq\zeta^p. \tag*{(1)}

Maximum entropy under Wasserstein constraint (W-MEP). Under the Wasserstein formulation, the problem maximizes the same conditional entropy, but constrains only the marginal distribution induced by the perturbation kernel,

References

  1. [1]Boyd, S., Vandenberghe, L.: Convex Optimization. Cambridge University Press, USA (2004)
  2. [2]Cover, T.M., Thomas, J.A.: Elements of Information Theory (Wiley Series in Telecommunications and Signal Processing). Wiley-Interscience, USA (2006)
  3. [3]Csiszár, I.: i-divergence geometry of probability distributions and minimization problems. The Annals of Probability 3(1), 146–158 (1975). DOI 10.1214/aop/1176996454DOI
  4. [4]Cuturi, M.: Sinkhorn distances: Lightspeed computation of optimal transport. In: C. Burges, L. Bottou, M. Welling, Z. Ghahramani, K. Weinberger (eds.) Advances in Neural Information Processing Systems, vol. 26. Curran Associates, Inc. (2013)
  5. [5]Dembo, A., Zeitouni, O.: Large Deviations Techniques and Applications. Applications of Mathematics. Springer, New York (1998)
  6. [6]Donsker, M.D., Varadhan, S.R.S.: Asymptotic evaluation of certain markov process expectations for large time. Communications on Pure and Applied Mathematics 28(1), 1–47 (1975)DOI
  7. [7]Donsker, M.D., Varadhan, S.R.S.: Asymptotic evaluation of certain markov process expectations for large time, IV. Communications on Pure and Applied Mathematics 36(2), 183–212 (1983)DOI
  8. [8]Dupuis, P., Ellis, R.S.: A Weak Convergence Approach to the Theory of Large Deviations. Wiley, New York (1997)
  9. [9]Folland, G.B.: Real Analysis: Modern Techniques and Their Applications, 2nd edn. John Wiley & Sons, New York (1999)
  10. [10]Ge, H., Pressé, S., Ghosh, K., Dill, K.A.: Markov processes follow from the principle of maximum caliber. The Journal of Chemical Physics 136(6), 064,108 (2012). DOI 10.1063/1.3681941DOI
  11. [11]Gibbs, J.W.: Elementary Principles in Statistical Mechanics. Charles Scribner’s Sons, New York (1902)
  12. [12]Jaynes, E.T.: Information theory and statistical mechanics. Physical Review 106(4), 620–630 (1957). DOI 10.1103/PhysRev.106.620DOI
  13. [13]Jaynes, E.T.: Information theory and statistical mechanics ii. Physical Review 108(2), 171–190 (1957). DOI 10.1103/PhysRev.108.171DOI
  14. [14]Jaynes, E.T.: The minimum entropy production principle. Annual Review of Physical Chemistry 31, 579–601 (1980). DOI 10.1146/annurev.pc.31.100180.003051DOI
  15. [15]Jordan, R., Kinderlehrer, D., Otto, F.: The variational formulation of the fokker–planck equation. SIAM Journal on Mathematical Analysis 29(1), 1–17 (1998). DOI 10.1137/S0036141096303359
  16. [16]Kallenberg, O.: Foundations of Modern Probability, 3 edn. Springer Nature Switzerland AG (2021)
  17. [17]Léonard, C.: A survey of the schrödinger problem and some of its connections with optimal transport. Discrete & Continuous Dynamical Systems - A 34(4), 1533–1574 (2014)DOI
  18. [18]Lin, J.: Divergence measures based on the shannon entropy. IEEE Transactions on Information Theory 37(1), 145–151 (1991)DOI
  19. [19]Naor, A., Romik, D.: Projecting the surface measure of the sphere of ℓⁿₚ. Annales de l’Institut Henri Poincare (B) Probability and Statistics 39(2), 241–261 (2003). DOI https://doi.org/10.1016/S0246-0203(02)00008-0
  20. [20]Peyré, G., Cuturi, M.: Computational Optimal Transport. Foundations and Trends in Machine Learning. Now Publishers (2019)arxiv.org/abs/1803.00567
  21. [21]Pressé, S., Ghosh, K., Lee, J., Dill, K.A.: Principles of maximum entropy and maximum caliber in statistical physics. Reviews of Modern Physics 85(3), 1115–1141 (2013). DOI 10.1103/RevModPhys.85.1115DOI
  22. [22]Santambrogio, F.: Optimal Transport for Applied Mathematicians: Calculus of Variations, PDEs, and Modeling, Progress in Nonlinear Differential Equations and Their Applications, vol. 87. Birkhäuser, Cham (2015). DOI 10.1007/978-3-319-20828-2
  23. [23]Shore, J.E., Johnson, R.W.: Axiomatic derivation of the principle of maximum entropy and the principle of minimum cross-entropy. IEEE Transactions on Information Theory 26(1), 26–37 (1980)DOI
  24. [24]Skilling, J.: The axioms of maximum entropy. In: J. Skilling (ed.) Maximum Entropy and Bayesian Methods, pp. 173–187. Kluwer Academic (1988)DOI
  25. [25]Stock, G., Ghosh, K., Dill, K.A.: Maximum caliber: A variational approach applied to two-state dynamics. The Journal of Chemical Physics 128(19), 194102 (2008). DOI 10.1063/1.2918345DOI
  26. [26]Vargas, L.F., Velasco, M.: Minimum cross-entropy distributions on Wasserstein balls and their applications (2021). Given a prior probability density p on a compact set K we characterize the probability distribution qδ on K contained in a Wasserstein ball Bδ(μ) centered in a given discrete measure μ for which the relative-entropy H(q,p) achieves its minimum. This characterization gives us an algorithm for computing such distributions efficientlyarxiv.org/abs/2106.03226
  27. [27]Villani, C.: Optimal Transport: Old and New, Grundlehren der mathematischen Wissenschaften, vol. 338. Springer, Berlin, Heidelberg (2008). DOI 10.1007/978-3-540-71050-9
  28. [28]Wainwright, M.J., Jordan, M.I.: Graphical models, exponential families, and variational inference. Foundations and Trends in Machine Learning 1(1–2), 1–305 (2008)
  29. [29]Wang, X.: Volumes of generalized unit balls. Mathematics Magazine 78(5), 390–395 (2005)DOI

Paper details

vlnc:W_xxmsf6cz
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 7

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Mathematics

Not recorded for this paper

View statement
Looking for related papers…
Contents