betaFeedback

WorksMathematics

Mutual Information Estimation via Triangular Transport Maps: Finite Sample Guarantees and Sample Allocation

Abstract

We study finite sample guarantees for transport based mutual information estimation. Mutual information can be estimated by first learning the conditional and marginal densities and then evaluating their log-density ratio by Monte Carlo. Given training samples, we use triangular transport maps to learn these densities and then compute the mutual information estimate on an independent batch of evaluation samples. For conditional density estimation using triangular transport maps, we establish non-asymptotic guarantees over realizable Hölder classes by viewing the conditional density as an ordinary density with respect to the product of the law of the conditioning variable and Lebesgue measure. This allows standard density estimation techniques to be applied directly to the conditional density estimation. These results yield finite sample guarantees for the resulting mutual information estimator and characterize how a fixed sample budget should be allocated between training and evaluation in order to minimize the mean squared error. Numerical experiments support the predicted training and evaluation error rates and illustrate the resulting sample allocation tradeoff.

Introduction

Mutual information (MI) measures statistical dependence and quantifies how much observing one random variable reduces uncertainty about another. It plays an important role across science and engineering, including material science [1], remote sensing [4], clinical trials [16], and optimal experimental design [13, 3, 8]. For random vectors X∈RdxX \in\mathbb{R}^{d_x} and Y∈RdyY \in\mathbb{R}^{d_y} with joint law μX,Y\mu_{X,Y} admitting a Lebesgue density, write πX,πY\pi_X,\pi_Y for the marginal densities and πY∣X,πX∣Y\pi_{Y \mid X},\pi_{X \mid Y} for the conditional densities. Then

I(X;Y)=EμX,Y[log⁡πY∣X(Y∣X)πY(Y)]=EμX,Y[log⁡πX∣Y(X∣Y)πX(X)].(1)I(X;Y) = \mathbb{E}_{\mu_{X,Y}}\left[\log\frac{\pi_{Y \mid X}(Y \mid X)}{\pi_Y(Y)}\right] = \mathbb{E}_{\mu_{X,Y}}\left[\log\frac{\pi_{X \mid Y}(X \mid Y)}{\pi_X(X)}\right]. \tag*{(1)}

In Bayesian inference, XX is the parameter and YY the observable, so these two representations correspond to the likelihood-to-marginal and posterior-to-prior density ratios, respectively. Although the definition is simple, estimating MI can be difficult when the relevant densities or expectations are analytically intractable [8] (Li et al. 2026).

A variety of MI estimators have been proposed. Nested Monte Carlo methods require likelihood evaluations and can converge slowly at finite sample sizes [19, 8]. When only samples are available, commonly used alternatives include kk-nearest-neighbor estimators [11] and variational methods such as MINE [2], NWJ [15], and related bounds [18]. Another class of approaches estimates the densities appearing in the MI density ratio and then evaluates the resulting plug-in estimator [7]. Flexible implementations include conditional normalizing flows [17, 6] and Knothe–Rosenblatt (KR) rearrangements [10, 5] (Li et al. 2026).

Transport maps are particularly attractive for this purpose because they represent complicated target distributions through invertible transformations to simple reference measures [14, 20]. With a triangular KR structure, conditional distributions are naturally encoded in blocks of the map, making the construction well suited to MI estimation, which requires both conditional and marginal densities. Moreover, transport maps can be fitted directly from samples using likelihood-based procedures. Despite this flexibility, finite-sample guarantees for the resulting MI estimators remain limited.

For split-sample transport-based MI estimation, Li et al. (2026) derived asymptotic guarantees with NN training and MM independent evaluation samples. Under in-class parametric assumptions and asymptotic normality, the bias is O(N−1)O(N^{-1}), while the variance and mean-squared error (MSE) are O(M−1+N−2)O(M^{-1}+N^{-2}). For a total budget T=N+MT=N+M, their asymptotically optimal allocation satisfies M/N≍T1/3M/N\asymp T^{1/3}. These results, however, do not provide finite-sample guarantees and rely on asymptotic properties of the fitted density parameters. Separately, Wang and Marzouk [22] established non-asymptotic, minimax-optimal density-estimation rates over Hölder classes, together with convergence rates for triangular transport maps. These results provide the tools for finite-sample transport analysis, but do not directly address conditional transport estimation, MI estimation, or the allocation of samples between training and evaluation.

Contribution

We establish finite-sample guarantees for an estimator that learns πY∣X\pi_{Y\mid X} and πY\pi_Y from NN shared training samples and evaluates their fitted log-density ratio using MM independent evaluation samples, complementing the asymptotic results of Li et al. (2026). Our analysis separates density-estimation error from Monte Carlo fluctuation, converting Kullback-Leibler (KL) divergence bounds into high-probability and MSE guarantees for MI.

For KR maps, we analyze conditional density estimation by treating the conditional density as an ordinary density with respect to the product of the conditioning-variable distribution and Lebesgue measure. This yields non-asymptotic conditional-density rates over realizable Hölder classes without requiring the conditioning variable itself to admit a density. Combining the conditional and marginal density-estimation rates gives explicit high-probability and MSE guarantees for the resulting MI estimator. We then minimize the MSE upper bound under T=N+MT=N+M and show that sublinear training suffices for O(T−1)O(T^{-1}) MSE. Numerical experiments examine the training and evaluation errors and compare the prescribed allocations.

Section § 2 defines the estimator. § 3–§ 5 develop the finite-sample guarantees and sample-allocation results, followed by numerical experiments in § 6 and conclusions in § 7. The appendices provide proofs and additional experimental details.

Problem setup and preliminaries

Let X∈X⊂RdxX \in\mathcal{X} \subset\mathbb{R}^{d_x} have an arbitrary Borel probability law μX\mu_X, and suppose that, conditional on X=xX = x, Y∈Y⊂RdyY \in\mathcal{Y} \subset\mathbb{R}^{d_y} admits a density πY∣X(⋅∣x)\pi_{Y\mid X}(\cdot\mid x) with respect to Lebesgue measure. The joint law can therefore be written as

μX,Y(dx,dy)=μX(dx)πY∣X(y∣x) dy.(2)\mu_{X,Y}(dx,dy) = \mu_X(dx)\pi_{Y\mid X}(y \mid x)\,dy. \tag*{(2)}

The corresponding marginal law is μY(dy)=πY(y) dy\mu_Y(dy) = \pi_Y(y)\,dy, where πY(y)=∫XπY∣X(y∣x) μX(dx)\pi_Y(y) = \int_{\mathcal{X}} \pi_{Y\mid X}(y \mid x)\,\mu_X(dx). For densities p,qp, q with respect to a common dominating measure ν\nu, write DKL(p∥q):=∫plog⁡(p/q) dνD_{\mathrm{KL}}(p\|q) := \int p\log(p/q)\,d\nu for their KL divergence. Assuming integrability of the log-density ratio, the mutual information reads

I(X;Y)=EμX,Y[log⁡πY∣X(Y∣X)πY(Y)](3)I(X;Y) = \mathbb{E}_{\mu_{X,Y}}\left[\log\frac{\pi_{Y\mid X}(Y\mid X)}{\pi_Y(Y)}\right] \tag*{(3)}
=∫XDKL(πY∣X(⋅∣x) ∥ πY) μX(dx).(4)= \int_{\mathcal{X}} D_{\mathrm{KL}}\bigl(\pi_{Y\mid X}(\cdot\mid x)\,\|\,\pi_Y\bigr)\,\mu_X(dx). \tag*{(4)}

Thus, estimating I(X;Y)I(X;Y) amounts to estimating the conditional density πY∣X\pi_{Y\mid X} and the marginal density πY\pi_Y. Importantly, this formulation does not require XX itself to admit a Lebesgue density. We abbreviate I=I(X;Y)I = I(X;Y) throughout. All logarithms are natural. The symbols ≲\lesssim and ≍\asymp denote one- and two-sided bounds up to positive constants independent of the sample sizes.

Conditional transport representation

We represent the conditional and marginal densities using triangular transport maps. For a map SS that pushes a target distribution to a known reference density η\eta, the density induced by SS is qS(z)=η(S(z))∣det⁡∇S(z)∣q_S(z) = \eta(S(z))\lvert\det\nabla S(z)\rvert. For the exact population transport S0S_0 associated with a target density π\pi, this construction satisfies qS0=πq_{S_0} = \pi. Throughout, we use the term KR map to refer to a monotone triangular transport map with a prescribed variable ordering.

For conditional density estimation, we leave xx fixed and transport only the yy-coordinates. For convenience, we work on the unit cube Q=[0,1]Q = [0,1], obtained by rescaling bounded coordinatewise supports when necessary, and take the reference density ηY\eta_Y to be supported on QdyQ^{d_y}. Write y1:j=(y1,…,yj)y_{1:j} = (y_1,\ldots,y_j) and y<j=y1:j−1y_{<j} = y_{1:j-1}, with y<1y_{<1} the empty vector.

With the variable ordering (X,Y)(X,Y), a full KR map, when it exists, can be partitioned blockwise as

S(x,y)=[SX(x),SY∣X(x,y)T]T,(5)S(x,y) = \left[S_X(x), S_{Y\mid X}(x,y)^\mathsf{T}\right]^\mathsf{T}, \tag*{(5)}

where the lower block has the triangular form

SY∣X(x,y)=[S1(x,y1),…,Sdy(x,y)]T.S_{Y\mid X}(x,y) = \left[S_1(x,y_1),\ldots,S_{d_y}(x,y)\right]^\mathsf{T}.

Our construction works directly with SY∣XS_{Y\mid X} and therefore does not require estimation of the leading block SXS_X or a density model for XX. Each component SjS_j depends only on (x,y1:j)(x,y_{1:j}) and is strictly increasing in yjy_j for fixed (x,y<j)(x,y_{<j}). For each jj and fixed (x,y<j)(x,y_{<j}), the map t↦Sj(x,y<j,t)t \mapsto S_j(x,y_{<j},t) is a bijection of QQ onto itself. Consequently, for every fixed xx, y↦SY∣X(x,y)y \mapsto S_{Y\mid X}(x,y) is a bijection of QdyQ^{d_y} onto itself. The conditional density induced by the conditional KR block is

qSY∣X(y∣x)=ηY(SY∣X(x,y))∣det⁡∇ySY∣X(x,y)∣=ηY(SY∣X(x,y))∏j=1dy∂yjSj(x,y1:j).(6)\begin{aligned} q_{S_{Y\mid X}}(y\mid x) &= \eta_Y\bigl(S_{Y\mid X}(x,y)\bigr)\left\lvert\det\nabla_y S_{Y\mid X}(x,y)\right\rvert\\ &= \eta_Y\bigl(S_{Y\mid X}(x,y)\bigr)\prod_{j=1}^{d_y}\partial_{y_j}S_j(x,y_{1:j}). \tag*{(6)} \end{aligned}

where the second equality follows from the triangular structure of ∇ySY∣X\nabla_y S_{Y\mid X}. Since SY∣X(x,⋅)S_{Y\mid X}(x,\cdot) maps QdyQ^{d_y} bijectively onto itself, a change of variables gives ∫QdyqSY∣X(y∣x) dy=1\int_{Q^{d_y}} q_{S_{Y\mid X}}(y\mid x)\,dy=1 for every xx.

Given NN i.i.d. training pairs {(Xi,Yi)}i=1N\{(X_i,Y_i)\}_{i=1}^N from μX,Y\mu_{X,Y}, we fit a maximum likelihood estimator over an admissible class SY∣X\mathcal{S}_{Y\mid X} of conditional KR maps:

References

  1. [1]R. Baptista, L. Cao, J. Chen, O. Ghattas, F. Li, Y. M. Marzouk, and J. T. Oden. Bayesian model calibration for block copolymer self-assembly: Likelihood-free inference and expected information gain computation via measure transport. Journal of Computational Physics, 503: 112844, 2024. ISSN 0021-9991. doi: https://doi.org/10.1016/j.jcp.2024.112844. URL https://www.sciencedirect.com/science/article/pii/S0021999124000937.
  2. [2]M. I. Belghazi, A. Baratin, S. Rajeshwar, S. Ozair, Y. Bengio, A. Courville, and D. Hjelm. Mutual information neural estimation. In J. Dy and A. Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 531–540. PMLR, 10–15 Jul 2018. URL https://proceedings.mlr.press/v80/belghazi18a.html.
  3. [3]K. Chaloner and I. Verdinelli. Bayesian Experimental Design: A Review. Statistical Science, 10(3): 273 – 304, 1995. doi: 10.1214/ss/1177009939. URL https://doi.org/10.1214/ss/1177009939.
  4. [4]H.-m. Chen, M. Arora, and P. Varshney. Mutual information based image registration for remote sensing data. Electrical Engineering and Computer Science, 24, 09 2003. doi: 10.1080/0143116031000117047.DOI
  5. [5]T. Cui, K. Koval, R. Herzog, and R. Scheichl. Subspace accelerated measure transport methods for fast and scalable sequential experimental design, with application to photoacoustic imaging, 2025. URL https://arxiv.org/abs/2502.20086.
  6. [6]J. Dong, C. Jacobsen, M. Khalloufi, M. Akram, W. Liu, K. Duraisamy, and X. Huan. Variational Bayesian optimal experimental design with normalizing flows. arXiv preprint arXiv:2404.13056, 2024.DOI
  7. [7]A. Foster, M. Jankowiak, E. Bingham, P. Horsfall, Y. W. Teh, T. Rainforth, and N. Goodman. Variational Bayesian optimal experimental design. In Advances in Neural Information Processing Systems, volume 32, 2019.arxiv.org/abs/1903.05480
  8. [8]X. Huan, J. Jagalur, and Y. Marzouk. Optimal experimental design: Formulations and computations. Acta Numerica, 33:715–840, 2024.DOI
  9. [9]A. K. H. Kim and R. J. Samworth. Global rates of convergence in log-concave density estimation. The Annals of Statistics, 44(6):2756–2779, 2016. doi: 10.1214/16-AOS1480. URL https://arxiv.org/abs/1404.2298v2.
  10. [10]K. Koval, R. Herzog, and R. Scheichl. Tractable optimal experimental design using transport maps. ArXiv, abs/2401.07971, 2024.arxiv.org/abs/2401.07971
  11. [11]A. Kraskov, H. Stögbauer, and P. Grassberger. Estimating mutual information. Phys. Rev. E, 69:066138, Jun 2004. doi: 10.1103/PhysRevE.69.066138. URL https://link.aps.org/doi/10.1103/PhysRevE.69.066138.DOI
  12. [13]D. V. Lindley. On a Measure of the Information Provided by an Experiment. The Annals of Mathematical Statistics, 27(4):986 – 1005, 1956. doi: 10.1214/aoms/1177728069.DOI
  13. [14]Y. Marzouk, T. Moselhy, M. Parno, and A. Spantini. Sampling via Measure Transport: An Introduction, pages 785–825. Springer International Publishing, Cham, 2016.
  14. [15]X. Nguyen, M. Wainwright, and M. Jordan. Estimating divergence functionals and the likelihood ratio by convex risk minimization. Information Theory, IEEE Transactions on, 56:5847 – 5861, 12 2010. doi: 10.1109/TIT.2010.2068870.DOI
  15. [16]K. Ogungbenro, A. Dokoumetzidis, and L. Aarons. Application of optimal design methodologies in clinical pharmacology experiments. Pharmaceutical Statistics, 8(3):239–252, 2009. doi: https://doi.org/10.1002/pst.354.
  16. [17]R. Orozco, A. Siahkoohi, G. Rizzuti, T. van Leeuwen, and F. J. Herrmann. Adjoint operators enable fast and amortized machine learning based Bayesian uncertainty quantification. In Medical Imaging 2023: Image Processing, volume 12464, page 124641L. International Society for Optics and Photonics, SPIE, 2023.DOI
  17. [18]B. Poole, S. Ozair, A. van den Oord, A. A. Alemi, and G. Tucker. On variational bounds of mutual information. In ICML, 2019.arxiv.org/abs/1905.06922
  18. [19]T. Rainforth, R. Cornish, H. Yang, and A. Warrington. On nesting Monte Carlo estimators. In International Conference on Machine Learning, pages 4267–4276. PMLR, 2018.arxiv.org/abs/1709.06181
  19. [20]A. Spantini, D. Bigoni, and Y. Marzouk. Inference via low-dimensional couplings. Journal of Machine Learning Research, 19(66):1–71, 2018.arxiv.org/abs/1703.06131
  20. [21]A. W. van der Vaart and J. A. Wellner. Weak Convergence and Empirical Processes: With Applications to Statistics. Springer Series in Statistics. Springer, 1996.
  21. [22]S. Wang and Y. Marzouk. On minimax density estimation via measure transport. arXiv:2207.10231, 2022.arxiv.org/abs/2207.10231

Paper details

vlnc:W_h77k7zfw
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Mathematics

generative AI tools

View statement
Looking for related papers…
Contents