Introduction
Mutual information (MI) measures statistical dependence and quantifies how much observing one random variable reduces uncertainty about another. It plays an important role across science and engineering, including material science [1 ] , remote sensing [4 ] , clinical trials [16 ] , and optimal experimental design [13 , 3 , 8 ] . For random vectors X ∈ R d x X \in\mathbb{R}^{d_x} X ∈ R d x and Y ∈ R d y Y \in\mathbb{R}^{d_y} Y ∈ R d y with joint law μ X , Y \mu_{X,Y} μ X , Y admitting a Lebesgue density, write π X , π Y \pi_X,\pi_Y π X , π Y for the marginal densities and π Y ∣ X , π X ∣ Y \pi_{Y \mid X},\pi_{X \mid Y} π Y ∣ X , π X ∣ Y for the conditional densities. Then
I ( X ; Y ) = E μ X , Y [ log π Y ∣ X ( Y ∣ X ) π Y ( Y ) ] = E μ X , Y [ log π X ∣ Y ( X ∣ Y ) π X ( X ) ] . (1) I(X;Y) = \mathbb{E}_{\mu_{X,Y}}\left[\log\frac{\pi_{Y \mid X}(Y \mid X)}{\pi_Y(Y)}\right]
= \mathbb{E}_{\mu_{X,Y}}\left[\log\frac{\pi_{X \mid Y}(X \mid Y)}{\pi_X(X)}\right]. \tag*{(1)} I ( X ; Y ) = E μ X , Y [ log π Y ( Y ) π Y ∣ X ( Y ∣ X ) ] = E μ X , Y [ log π X ( X ) π X ∣ Y ( X ∣ Y ) ] . (1)
In Bayesian inference, X X X is the parameter and Y Y Y the observable, so these two representations correspond to the likelihood-to-marginal and posterior-to-prior density ratios, respectively. Although the definition is simple, estimating MI can be difficult when the relevant densities or expectations are analytically intractable [8 ] (Li et al. 2026).
A variety of MI estimators have been proposed. Nested Monte Carlo methods require likelihood evaluations and can converge slowly at finite sample sizes [19 , 8 ] . When only samples are available, commonly used alternatives include k k k -nearest-neighbor estimators [11 ] and variational methods such as MINE [2 ] , NWJ [15 ] , and related bounds [18 ] . Another class of approaches estimates the densities appearing in the MI density ratio and then evaluates the resulting plug-in estimator [7 ] . Flexible implementations include conditional normalizing flows [17 , 6 ] and Knothe–Rosenblatt (KR) rearrangements [10 , 5 ] (Li et al. 2026).
Transport maps are particularly attractive for this purpose because they represent complicated target distributions through invertible transformations to simple reference measures [14 , 20 ] . With a triangular KR structure, conditional distributions are naturally encoded in blocks of the map, making the construction well suited to MI estimation, which requires both conditional and marginal densities. Moreover, transport maps can be fitted directly from samples using likelihood-based procedures. Despite this flexibility, finite-sample guarantees for the resulting MI estimators remain limited.
For split-sample transport-based MI estimation, Li et al. (2026) derived asymptotic guarantees with N N N training and M M M independent evaluation samples. Under in-class parametric assumptions and asymptotic normality, the bias is O ( N − 1 ) O(N^{-1}) O ( N − 1 ) , while the variance and mean-squared error (MSE) are O ( M − 1 + N − 2 ) O(M^{-1}+N^{-2}) O ( M − 1 + N − 2 ) . For a total budget T = N + M T=N+M T = N + M , their asymptotically optimal allocation satisfies M / N ≍ T 1 / 3 M/N\asymp T^{1/3} M / N ≍ T 1/3 . These results, however, do not provide finite-sample guarantees and rely on asymptotic properties of the fitted density parameters. Separately, Wang and Marzouk [22 ] established non-asymptotic, minimax-optimal density-estimation rates over Hölder classes, together with convergence rates for triangular transport maps. These results provide the tools for finite-sample transport analysis, but do not directly address conditional transport estimation, MI estimation, or the allocation of samples between training and evaluation.
Contribution
We establish finite-sample guarantees for an estimator that learns π Y ∣ X \pi_{Y\mid X} π Y ∣ X and π Y \pi_Y π Y from N N N shared training samples and evaluates their fitted log-density ratio using M M M independent evaluation samples, complementing the asymptotic results of Li et al. (2026). Our analysis separates density-estimation error from Monte Carlo fluctuation, converting Kullback-Leibler (KL) divergence bounds into high-probability and MSE guarantees for MI.
For KR maps, we analyze conditional density estimation by treating the conditional density as an ordinary density with respect to the product of the conditioning-variable distribution and Lebesgue measure. This yields non-asymptotic conditional-density rates over realizable Hölder classes without requiring the conditioning variable itself to admit a density. Combining the conditional and marginal density-estimation rates gives explicit high-probability and MSE guarantees for the resulting MI estimator. We then minimize the MSE upper bound under T = N + M T=N+M T = N + M and show that sublinear training suffices for O ( T − 1 ) O(T^{-1}) O ( T − 1 ) MSE. Numerical experiments examine the training and evaluation errors and compare the prescribed allocations.
Section § 2 defines the estimator. § 3 –§ 5 develop the finite-sample guarantees and sample-allocation results, followed by numerical experiments in § 6 and conclusions in § 7 . The appendices provide proofs and additional experimental details.
Problem setup and preliminaries
Let X ∈ X ⊂ R d x X \in\mathcal{X} \subset\mathbb{R}^{d_x} X ∈ X ⊂ R d x have an arbitrary Borel probability law μ X \mu_X μ X , and suppose that, conditional on X = x X = x X = x , Y ∈ Y ⊂ R d y Y \in\mathcal{Y} \subset\mathbb{R}^{d_y} Y ∈ Y ⊂ R d y admits a density π Y ∣ X ( ⋅ ∣ x ) \pi_{Y\mid X}(\cdot\mid x) π Y ∣ X ( ⋅ ∣ x ) with respect to Lebesgue measure. The joint law can therefore be written as
μ X , Y ( d x , d y ) = μ X ( d x ) π Y ∣ X ( y ∣ x ) d y . (2) \mu_{X,Y}(dx,dy) = \mu_X(dx)\pi_{Y\mid X}(y \mid x)\,dy. \tag*{(2)} μ X , Y ( d x , d y ) = μ X ( d x ) π Y ∣ X ( y ∣ x ) d y . (2)
The corresponding marginal law is μ Y ( d y ) = π Y ( y ) d y \mu_Y(dy) = \pi_Y(y)\,dy μ Y ( d y ) = π Y ( y ) d y , where π Y ( y ) = ∫ X π Y ∣ X ( y ∣ x ) μ X ( d x ) \pi_Y(y) = \int_{\mathcal{X}} \pi_{Y\mid X}(y \mid x)\,\mu_X(dx) π Y ( y ) = ∫ X π Y ∣ X ( y ∣ x ) μ X ( d x ) . For densities p , q p, q p , q with respect to a common dominating measure ν \nu ν , write D K L ( p ∥ q ) : = ∫ p log ( p / q ) d ν D_{\mathrm{KL}}(p\|q) := \int p\log(p/q)\,d\nu D KL ( p ∥ q ) := ∫ p log ( p / q ) d ν for their KL divergence. Assuming integrability of the log-density ratio, the mutual information reads
I ( X ; Y ) = E μ X , Y [ log π Y ∣ X ( Y ∣ X ) π Y ( Y ) ] (3) I(X;Y) = \mathbb{E}_{\mu_{X,Y}}\left[\log\frac{\pi_{Y\mid X}(Y\mid X)}{\pi_Y(Y)}\right] \tag*{(3)} I ( X ; Y ) = E μ X , Y [ log π Y ( Y ) π Y ∣ X ( Y ∣ X ) ] (3)
= ∫ X D K L ( π Y ∣ X ( ⋅ ∣ x ) ∥ π Y ) μ X ( d x ) . (4) = \int_{\mathcal{X}} D_{\mathrm{KL}}\bigl(\pi_{Y\mid X}(\cdot\mid x)\,\|\,\pi_Y\bigr)\,\mu_X(dx). \tag*{(4)} = ∫ X D KL ( π Y ∣ X ( ⋅ ∣ x ) ∥ π Y ) μ X ( d x ) . (4)
Thus, estimating I ( X ; Y ) I(X;Y) I ( X ; Y ) amounts to estimating the conditional density π Y ∣ X \pi_{Y\mid X} π Y ∣ X and the marginal density π Y \pi_Y π Y . Importantly, this formulation does not require X X X itself to admit a Lebesgue density. We abbreviate I = I ( X ; Y ) I = I(X;Y) I = I ( X ; Y ) throughout. All logarithms are natural. The symbols ≲ \lesssim ≲ and ≍ \asymp ≍ denote one- and two-sided bounds up to positive constants independent of the sample sizes.
Conditional transport representation
We represent the conditional and marginal densities using triangular transport maps. For a map S S S that pushes a target distribution to a known reference density η \eta η , the density induced by S S S is q S ( z ) = η ( S ( z ) ) ∣ det ∇ S ( z ) ∣ q_S(z) = \eta(S(z))\lvert\det\nabla S(z)\rvert q S ( z ) = η ( S ( z )) ∣ det ∇ S ( z )∣ . For the exact population transport S 0 S_0 S 0 associated with a target density π \pi π , this construction satisfies q S 0 = π q_{S_0} = \pi q S 0 = π . Throughout, we use the term KR map to refer to a monotone triangular transport map with a prescribed variable ordering.
For conditional density estimation, we leave x x x fixed and transport only the y y y -coordinates. For convenience, we work on the unit cube Q = [ 0 , 1 ] Q = [0,1] Q = [ 0 , 1 ] , obtained by rescaling bounded coordinatewise supports when necessary, and take the reference density η Y \eta_Y η Y to be supported on Q d y Q^{d_y} Q d y . Write y 1 : j = ( y 1 , … , y j ) y_{1:j} = (y_1,\ldots,y_j) y 1 : j = ( y 1 , … , y j ) and y < j = y 1 : j − 1 y_{<j} = y_{1:j-1} y < j = y 1 : j − 1 , with y < 1 y_{<1} y < 1 the empty vector.
With the variable ordering ( X , Y ) (X,Y) ( X , Y ) , a full KR map, when it exists, can be partitioned blockwise as
S ( x , y ) = [ S X ( x ) , S Y ∣ X ( x , y ) T ] T , (5) S(x,y) = \left[S_X(x), S_{Y\mid X}(x,y)^\mathsf{T}\right]^\mathsf{T}, \tag*{(5)} S ( x , y ) = [ S X ( x ) , S Y ∣ X ( x , y ) T ] T , (5)
where the lower block has the triangular form
S Y ∣ X ( x , y ) = [ S 1 ( x , y 1 ) , … , S d y ( x , y ) ] T . S_{Y\mid X}(x,y) = \left[S_1(x,y_1),\ldots,S_{d_y}(x,y)\right]^\mathsf{T}. S Y ∣ X ( x , y ) = [ S 1 ( x , y 1 ) , … , S d y ( x , y ) ] T .
Our construction works directly with S Y ∣ X S_{Y\mid X} S Y ∣ X and therefore does not require estimation of the leading block S X S_X S X or a density model for X X X . Each component S j S_j S j depends only on ( x , y 1 : j ) (x,y_{1:j}) ( x , y 1 : j ) and is strictly increasing in y j y_j y j for fixed ( x , y < j ) (x,y_{<j}) ( x , y < j ) . For each j j j and fixed ( x , y < j ) (x,y_{<j}) ( x , y < j ) , the map t ↦ S j ( x , y < j , t ) t \mapsto S_j(x,y_{<j},t) t ↦ S j ( x , y < j , t ) is a bijection of Q Q Q onto itself. Consequently, for every fixed x x x , y ↦ S Y ∣ X ( x , y ) y \mapsto S_{Y\mid X}(x,y) y ↦ S Y ∣ X ( x , y ) is a bijection of Q d y Q^{d_y} Q d y onto itself. The conditional density induced by the conditional KR block is
q S Y ∣ X ( y ∣ x ) = η Y ( S Y ∣ X ( x , y ) ) ∣ det ∇ y S Y ∣ X ( x , y ) ∣ = η Y ( S Y ∣ X ( x , y ) ) ∏ j = 1 d y ∂ y j S j ( x , y 1 : j ) . (6) \begin{aligned}
q_{S_{Y\mid X}}(y\mid x) &= \eta_Y\bigl(S_{Y\mid X}(x,y)\bigr)\left\lvert\det\nabla_y S_{Y\mid X}(x,y)\right\rvert\\
&= \eta_Y\bigl(S_{Y\mid X}(x,y)\bigr)\prod_{j=1}^{d_y}\partial_{y_j}S_j(x,y_{1:j}). \tag*{(6)}
\end{aligned} q S Y ∣ X ( y ∣ x ) = η Y ( S Y ∣ X ( x , y ) ) det ∇ y S Y ∣ X ( x , y ) = η Y ( S Y ∣ X ( x , y ) ) j = 1 ∏ d y ∂ y j S j ( x , y 1 : j ) . (6)
where the second equality follows from the triangular structure of ∇ y S Y ∣ X \nabla_y S_{Y\mid X} ∇ y S Y ∣ X . Since S Y ∣ X ( x , ⋅ ) S_{Y\mid X}(x,\cdot) S Y ∣ X ( x , ⋅ ) maps Q d y Q^{d_y} Q d y bijectively onto itself, a change of variables gives ∫ Q d y q S Y ∣ X ( y ∣ x ) d y = 1 \int_{Q^{d_y}} q_{S_{Y\mid X}}(y\mid x)\,dy=1 ∫ Q d y q S Y ∣ X ( y ∣ x ) d y = 1 for every x x x .
Given N N N i.i.d. training pairs { ( X i , Y i ) } i = 1 N \{(X_i,Y_i)\}_{i=1}^N {( X i , Y i ) } i = 1 N from μ X , Y \mu_{X,Y} μ X , Y , we fit a maximum likelihood estimator over an admissible class S Y ∣ X \mathcal{S}_{Y\mid X} S Y ∣ X of conditional KR maps:
S ^ Y ∣ X ∈ argmin S Y ∣ X ∈ S Y ∣ X { − 1 N ∑ i = 1 N log q S Y ∣ X ( Y i ∣ X i ) } . (7) \widehat{S}_{Y\mid X}\in\underset{S_{Y\mid X}\in\mathcal{S}_{Y\mid X}}{\operatorname{argmin}}\left\{-\frac{1}{N}\sum_{i=1}^{N}\log q_{S_{Y\mid X}}(Y_i\mid X_i)\right\}. \tag*{(7)} S Y ∣ X ∈ S Y ∣ X ∈ S Y ∣ X argmin { − N 1 i = 1 ∑ N log q S Y ∣ X ( Y i ∣ X i ) } . (7)
We denote the resulting conditional density by q ^ Y ∣ X = q S ^ Y ∣ X \widehat{q}_{Y\mid X}=q_{\widehat{S}_{Y\mid X}} q Y ∣ X = q S Y ∣ X . The precise regularity and monotonicity conditions defining S Y ∣ X \mathcal{S}_{Y\mid X} S Y ∣ X are given in § 4 .
The marginal density π Y \pi_Y π Y is estimated analogously by fitting a d y d_y d y -dimensional KR map S ^ Y \widehat{S}_Y S Y to { Y i } i = 1 N \{Y_i\}_{i=1}^N { Y i } i = 1 N , yielding q ^ Y = q S ^ Y \widehat{q}_Y=q_{\widehat{S}_Y} q Y = q S Y . Thus, the same training sample is used to construct both density estimators appearing in the mutual information estimator.
Split-sample estimator
We now use the fitted conditional and marginal densities to estimate mutual information. Let D t r a i n = { ( X i , Y i ) } i = 1 N D_{\mathrm{train}}=\{(X_i,Y_i)\}_{i=1}^N D train = {( X i , Y i ) } i = 1 N denote the training sample used to construct q ^ Y ∣ X \widehat{q}_{Y\mid X} q Y ∣ X and q ^ Y \widehat{q}_Y q Y , and let D e v a l = { ( X j ′ , Y j ′ ) } j = 1 M D_{\mathrm{eval}}=\{(X'_j,Y'_j)\}_{j=1}^M D eval = {( X j ′ , Y j ′ ) } j = 1 M be an i.i.d. evaluation sample from the same joint law μ X , Y \mu_{X,Y} μ X , Y , independent of D t r a i n D_{\mathrm{train}} D train . The split-sample estimator is
I ^ N , M : = 1 M ∑ j = 1 M [ log q ^ Y ∣ X ( Y j ′ ∣ X j ′ ) − log q ^ Y ( Y j ′ ) ] . (8) \widehat{I}_{N,M}:=\frac{1}{M}\sum_{j=1}^{M}\left[\log\widehat{q}_{Y\mid X}(Y'_j\mid X'_j)-\log\widehat{q}_Y(Y'_j)\right]. \tag*{(8)} I N , M := M 1 j = 1 ∑ M [ log q Y ∣ X ( Y j ′ ∣ X j ′ ) − log q Y ( Y j ′ ) ] . (8)
This is the plug-in analogue of (3) , with the densities fitted using N N N training samples and the resulting log-density ratio evaluated using M M M independent samples. Write q ^ = ( q ^ Y ∣ X , q ^ Y ) \widehat{q}=(\widehat{q}_{Y\mid X},\widehat{q}_Y) q = ( q Y ∣ X , q Y ) for the fitted density pair; its dependence on N N N is implicit unless written as q ^ N \widehat{q}_N q N . Unconditional expectations and probabilities average over all relevant training and evaluation randomness; subscripts train and eval specify one source when needed. In particular, MSE ( I ^ N , M ) : = E [ ( I ^ N , M − I ) 2 ] \operatorname{MSE}(\widehat{I}_{N,M}):=\mathbb{E}[(\widehat{I}_{N,M}-I)^2] MSE ( I N , M ) := E [( I N , M − I ) 2 ] .
Generic plug-in guarantees
In this section, we study the finite-sample error of the mutual information estimator in a generic setting, without restricting the conditional and marginal density estimators to any particular form. The goal is to separate the error arising from density estimation from that arising from Monte Carlo evaluation. We first identify the density estimation quantities that enter the plug-in error, and then combine them with concentration of the independent evaluation sample. The resulting guarantees apply to arbitrary normalized conditional and marginal density estimators satisfying the assumptions below. In § 4 , we specialize these results to the conditional and marginal KR estimators introduced above and establish the required density estimation bounds. For the fitted pair q ^ \widehat{q} q , define the conditional and marginal KL divergence
E X ( q ^ ) : = E X ∼ μ X [ D K L ( π Y ∣ X ( ⋅ ∣ X ) ∥ q ^ Y ∣ X ( ⋅ ∣ X ) ) ] , (9) \mathcal{E}_X(\widehat{q}):=\mathbb{E}_{X\sim\mu_X}\left[\mathcal{D}_{\mathrm{KL}}\left(\pi_{Y\mid X}(\cdot\mid X)\,\middle\|\,\widehat{q}_{Y\mid X}(\cdot\mid X)\right)\right], \tag*{(9)} E X ( q ) := E X ∼ μ X [ D KL ( π Y ∣ X ( ⋅ ∣ X ) q Y ∣ X ( ⋅ ∣ X ) ) ] , (9)
E Y ( q ^ ) : = D K L ( π Y ∥ q ^ Y ) . (10) \mathcal{E}_Y(\widehat{q}):=\mathcal{D}_{\mathrm{KL}}(\pi_Y\|\widehat{q}_Y). \tag*{(10)} E Y ( q ) := D KL ( π Y ∥ q Y ) . (10)
These quantities measure the errors of the two density estimators entering the log-density ratio. Since the estimators are learned from the training sample, both risks remain random through the training data. As shown below, their difference determines the conditional bias of the MI estimator, while their individual high-probability bounds control the magnitude of this bias. We suppress the argument q ^ \hat{q} q ^ in E C \mathcal{E}_C E C and E Y \mathcal{E}_Y E Y when it is unambiguous.
For a fresh pair ( X , Y ) ∼ μ X , Y (X,Y) \sim\mu_{X,Y} ( X , Y ) ∼ μ X , Y independent of training, define the fitted log-density ratio and its population moments, whenever finite, by
φ ^ ( x , y ) : = log q ^ Y ∣ X ( y ∣ x ) − log q ^ Y ( y ) , g ( q ^ ) : = E [ φ ^ ( X , Y ) ∣ q ^ ] , V ( q ^ ) : = Var ( φ ^ ( X , Y ) ∣ q ^ ) . (11) \begin{aligned}
\hat{\varphi}(x,y) &:= \log\hat{q}_{Y \mid X}(y \mid x) - \log\hat{q}_Y(y),\\
g(\hat{q}) &:= \mathbb{E}[\hat{\varphi}(X,Y) \mid\hat{q}],\\
V(\hat{q}) &:= \operatorname{Var}(\hat{\varphi}(X,Y) \mid\hat{q}). \tag*{(11)}
\end{aligned} φ ^ ( x , y ) g ( q ^ ) V ( q ^ ) := log q ^ Y ∣ X ( y ∣ x ) − log q ^ Y ( y ) , := E [ φ ^ ( X , Y ) ∣ q ^ ] , := Var ( φ ^ ( X , Y ) ∣ q ^ ) . (11)
Assumptions
The analysis requires two ingredients. First, a tail condition controls the Monte Carlo fluctuation of the average fitted log-density ratio. Second, high-probability KL-risk bounds control the error introduced by estimating the two densities.
Assumption 3.1 (Sub-Gaussianity). There is a deterministic constant K ≥ 0 K \ge0 K ≥ 0 such that, conditionally on the fitted pair, for every t ∈ R t \in\mathbb{R} t ∈ R ,
E [ e t ( φ ^ ( X , Y ) − g ( q ^ ) ) ∣ q ^ ] ≤ e t 2 K 2 / 2 . \mathbb{E}\left[e^{t(\hat{\varphi}(X,Y)-g(\hat{q}))} \mid\hat{q}\right] \le e^{t^2K^2/2}. E [ e t ( φ ^ ( X , Y ) − g ( q ^ )) ∣ q ^ ] ≤ e t 2 K 2 /2 .
While Assumption 3.1 controls evaluation error for fixed fitted densities, we also need to control how accurately those densities approximate the true conditional and marginal densities.
Assumption 3.2 (Density estimation error). Let ε C ( N , δ ) \varepsilon_C(N,\delta) ε C ( N , δ ) and ε Y ( N , δ ) \varepsilon_Y(N,\delta) ε Y ( N , δ ) denote deterministic upper bounds on the conditional and marginal KL divergence, respectively, as functions of the training sample size N N N and confidence level δ \delta δ . We assume that, for every δ ∈ ( 0 , 1 ) \delta\in(0,1) δ ∈ ( 0 , 1 ) ,
Pr ( E C ( q ^ ) > ε C ( N , δ ) ) ≤ δ , \Pr(\mathcal{E}_C(\hat{q}) > \varepsilon_C(N,\delta)) \le\delta, Pr ( E C ( q ^ ) > ε C ( N , δ )) ≤ δ , Pr ( E Y ( q ^ ) > ε Y ( N , δ ) ) ≤ δ . (12) \Pr(\mathcal{E}_Y(\hat{q}) > \varepsilon_Y(N,\delta)) \le\delta. \tag*{(12)} Pr ( E Y ( q ^ ) > ε Y ( N , δ )) ≤ δ . (12)
The risk bounds are non-increasing in both N N N and δ \delta δ .
The functions ε C \varepsilon_C ε C and ε Y \varepsilon_Y ε Y are left unspecified in this generic analysis. They encode the statistical accuracy of the chosen density estimators. Uniform upper and positive lower bounds on the estimated densities imply Assumption 3.1 by Hoeffding’s lemma (Lemma A.1). In § 4 , we verify these conditions for the KR estimators and derive explicit choices of ε C \varepsilon_C ε C and ε Y \varepsilon_Y ε Y .
Bias identity and finite-sample bound
We now connect the density-estimation errors above to the MI estimation error. The key observation is that, conditional on the fitted densities, the bias of the plug-in estimator is exactly the difference between the marginal and conditional KL divergences.
Lemma 3.1 (Conditional plug-in bias identity). Suppose log q ^ Y ∣ X ( Y ∣ X ) π Y ∣ X ( Y ∣ X ) ∈ L 1 ( μ X , Y ) \log\frac{\hat{q}_{Y \mid X}(Y \mid X)}{\pi_{Y \mid X}(Y \mid X)} \in L^1(\mu_{X,Y}) log π Y ∣ X ( Y ∣ X ) q ^ Y ∣ X ( Y ∣ X ) ∈ L 1 ( μ X , Y ) and log q ^ Y ( Y ) π Y ( Y ) ∈ L 1 ( μ Y ) \log\frac{\hat{q}_Y(Y)}{\pi_Y(Y)} \in L^1(\mu_Y) log π Y ( Y ) q ^ Y ( Y ) ∈ L 1 ( μ Y ) . Then
E [ I ^ N , M − I ∣ q ^ ] = E Y ( q ^ ) − E C ( q ^ ) . (13) \mathbb{E}[\hat{I}_{N,M} - I \mid\hat{q}] = \mathcal{E}_Y(\hat{q}) - \mathcal{E}_C(\hat{q}). \tag*{(13)} E [ I ^ N , M − I ∣ q ^ ] = E Y ( q ^ ) − E C ( q ^ ) . (13) The identity holds for any Borel probability measure μ X \mu_X μ X .
Thus, the plug-in error has two distinct sources. The first is the fluctuation of the finite evaluation sample around its conditional mean, and the second is the error in the fitted densities themselves. In particular,
I ^ N , M − I = I ^ N , M − g ( q ^ ) ⏟ A M + E Y ( q ^ ) − E C ( q ^ ) ⏟ B ( q ^ ) , (14) \widehat{I}_{N,M} - I = \underbrace{\widehat{I}_{N,M} - g(\widehat{q})}_{A_M} + \underbrace{\mathcal{E}_Y(\widehat{q}) - \mathcal{E}_C(\widehat{q})}_{B(\widehat{q})}, \tag*{(14)} I N , M − I = A M I N , M − g ( q ) + B ( q ) E Y ( q ) − E C ( q ) , (14)
with the bias identity proved in Appendix § 9 .§ 1 . In particular, B ( q ^ ) = g ( q ^ ) − I B(\widehat{q}) = g(\widehat{q}) - I B ( q ) = g ( q ) − I . The dependence of A M A_M A M on N N N is suppressed. Here A M A_M A M is a Monte Carlo error controlled by the evaluation sample size M M M , whereas B ( q ^ ) B(\widehat{q}) B ( q ) is a training-dependent plug-in error controlled by the density-estimation accuracy and hence by N N N . This decomposition is the basis of both the finite-sample bound below and the sample-allocation analysis in § 5 .
Theorem 3.2 (General non-asymptotic bound). Under Assumptions 3.1–3.2, for every δ ∈ ( 0 , 1 ) \delta\in(0,1) δ ∈ ( 0 , 1 ) , with probability at least 1 − δ 1-\delta 1 − δ ,
∣ I ^ N , M − I ∣ ≤ K 2 log ( 6 / δ ) M + ε C ( N , δ / 3 ) + ε Y ( N , δ / 3 ) . (15) \left|\widehat{I}_{N,M} - I\right| \le K\sqrt{\frac{2\log(6/\delta)}{M}} + \varepsilon_C(N,\delta/3) + \varepsilon_Y(N,\delta/3). \tag*{(15)} I N , M − I ≤ K M 2 log ( 6/ δ ) + ε C ( N , δ /3 ) + ε Y ( N , δ /3 ) . (15)
Theorem 3.2 makes the two sources of error explicit. The first term is the Monte Carlo contribution and decays with the evaluation sample size M M M . The remaining terms are the conditional and marginal density-estimation errors and decay with the training sample size N N N . Consequently, once explicit rates for ε C \varepsilon_C ε C and ε Y \varepsilon_Y ε Y are available, the theorem immediately yields a finite-sample MI error bound and exposes the tradeoff between training and evaluation samples. The proof details are given in Appendix § 9 . The same decomposition also yields a corresponding mean-squared error bound,
MSE ( I ^ N , M ) ≤ K 2 M + E [ B ( q ^ ) 2 ] . (16) \operatorname{MSE}(\widehat{I}_{N,M}) \le\frac{K^2}{M} + \mathbb{E}\left[B(\widehat{q})^2\right]. \tag*{(16)} MSE ( I N , M ) ≤ M K 2 + E [ B ( q ) 2 ] . (16)
Conditional density estimation and MI rates via KR maps
In this section, we first derive a non-asymptotic convergence rate for a conditional density estimator constructed using KR maps. We then show that the marginal density inherits the required regularity and establish the corresponding marginal density-estimation rate. Finally, we combine these results with the generic guarantees of § 3 to obtain finite-sample convergence bounds for the resulting transport-based mutual information estimator. For the conditional density analysis, we adapt the proof strategy of Wang and Marzouk [22 ] and restrict our attention to Hölder classes on the unit cube. Throughout, we set Q = [ 0 , 1 ] Q = [0,1] Q = [ 0 , 1 ] , X = Q d x \mathcal{X} = Q^{d_x} X = Q d x , and Y = Q d y \mathcal{Y} = Q^{d_y} Y = Q d y , with d x , d y ≥ 1 d_x,d_y \ge1 d x , d y ≥ 1 , and let d = d x + d y d = d_x + d_y d = d x + d y . All regularity assumptions on the conditional density are imposed jointly in ( x , y ) (x,y) ( x , y ) .
Regularity and the conditional estimator
We now specify the regularity assumptions and the conditional transport class used in the convergence analysis.
Definition 4.1 (Hölder class). For α > 0 \alpha> 0 α > 0 , let m α = ⌊ α ⌋ − 1 m_\alpha= \lfloor\alpha\rfloor- 1 m α = ⌊ α ⌋ − 1 and β α = α − m α ∈ ( 0 , 1 ] \beta_\alpha= \alpha- m_\alpha\in(0,1] β α = α − m α ∈ ( 0 , 1 ] . On a compact domain Ω \Omega Ω , write D ν D^\nu D ν for the derivative with multi-index ν \nu ν , ∥ ⋅ ∥ ∞ \|\cdot\|_\infty ∥ ⋅ ∥ ∞ for the supremum norm, and ∥ ⋅ ∥ \|\cdot\| ∥ ⋅ ∥ for the Euclidean norm. Then the Hölder class H α ( Ω ) = C m α , β α ( Ω ) \mathcal{H}^\alpha(\Omega) = \mathcal{C}^{m_\alpha,\beta_\alpha}(\Omega) H α ( Ω ) = C m α , β α ( Ω )
consists of functions with continuous derivatives through order m α m_\alpha m α and finite norm ∥ f ∥ H α : = ∑ ∣ ν ∣ ≤ m α ∥ D ν f ∥ ∞ + ∑ ∣ ν ∣ = m α sup z ≠ z ′ ∣ D ν f ( z ) − D ν f ( z ′ ) ∣ ∥ z − z ′ ∥ β α \|f\|_{\mathcal{H}^\alpha} := \sum_{|\nu|\le m_\alpha}\|D^\nu f\|_\infty+ \sum_{|\nu|=m_\alpha}\sup_{z\ne z'} \frac{|D^\nu f(z)-D^\nu f(z')|}{\|z-z'\|^{\beta_\alpha}} ∥ f ∥ H α := ∑ ∣ ν ∣ ≤ m α ∥ D ν f ∥ ∞ + ∑ ∣ ν ∣ = m α sup z = z ′ ∥ z − z ′ ∥ β α ∣ D ν f ( z ) − D ν f ( z ′ ) ∣ . Derivatives extend continuously to the boundary. At integer α \alpha α , derivatives of order α − 1 \alpha-1 α − 1 are taken to be Lipschitz. For a vector-valued function f = ( f 1 , … , f k ) f=(f_1,\ldots,f_k) f = ( f 1 , … , f k ) , use the componentwise norm ∥ f ∥ H α : = ( ∑ j = 1 k ∥ f j ∥ H α 2 ) 1 / 2 \|f\|_{\mathcal{H}^\alpha}:=\left(\sum_{j=1}^k\|f_j\|_{\mathcal{H}^\alpha}^2\right)^{1/2} ∥ f ∥ H α := ( ∑ j = 1 k ∥ f j ∥ H α 2 ) 1/2 .
Assumption 4.1 (Conditional regularity). The conditional density satisfies ∥ π Y ∣ X ∥ H α ( Q d y ) ≤ L 0 \|\pi_{Y\mid X}\|_{\mathcal{H}^\alpha(Q^{d_y})}\le L_0 ∥ π Y ∣ X ∥ H α ( Q d y ) ≤ L 0 and 0 < c ≤ π Y ∣ X ( y ∣ x ) ≤ B < ∞ 0<c\le\pi_{Y\mid X}(y\mid x)\le B<\infty 0 < c ≤ π Y ∣ X ( y ∣ x ) ≤ B < ∞ for some α > d / 2 \alpha>d/2 α > d /2 , with ∫ Q d y π Y ∣ X ( y ∣ x ) d y = 1 \int_{Q^{d_y}}\pi_{Y\mid X}(y\mid x)\,dy=1 ∫ Q d y π Y ∣ X ( y ∣ x ) d y = 1 for every x ∈ Q d x x\in Q^{d_x} x ∈ Q d x .
Assumption 4.2 (Reference density). The reference density is η Y ( u ) = ∏ j = 1 d y η Y , j ( u j ) \eta_Y(u)=\prod_{j=1}^{d_y}\eta_{Y,j}(u_j) η Y ( u ) = ∏ j = 1 d y η Y , j ( u j ) on Q d y Q^{d_y} Q d y , where each factor is a probability density satisfying ∥ η Y , j ∥ H α ( Q ) ≤ L η \|\eta_{Y,j}\|_{\mathcal{H}^\alpha(Q)}\le L_\eta ∥ η Y , j ∥ H α ( Q ) ≤ L η and 0 < c η ≤ η Y , j ≤ B η < ∞ 0<c_\eta\le\eta_{Y,j}\le B_\eta<\infty 0 < c η ≤ η Y , j ≤ B η < ∞ . We write G j ( t ) = ∫ t η Y , j ( u ) d u G_j(t)=\int^t\eta_{Y,j}(u)\,du G j ( t ) = ∫ t η Y , j ( u ) d u for its cumulative distribution function (CDF).
Definition 4.2 (Conditional KR map class). For L ≥ 1 L\ge1 L ≥ 1 and 0 < c m o n ≤ 1 0<c_{\mathrm{mon}}\le1 0 < c mon ≤ 1 , let S Y ∣ X ( α , L , c m o n ) \mathcal{S}_{Y\mid X}(\alpha,L,c_{\mathrm{mon}}) S Y ∣ X ( α , L , c mon ) denote the class of conditional triangular maps introduced in § 2.1 such that, for every j j j :
(i) S j ( x , y < j , ⋅ ) S_j(x,y_{<j},\cdot) S j ( x , y < j , ⋅ ) is an increasing bijection of Q Q Q onto itself for every fixed ( x , y < j ) (x,y_{<j}) ( x , y < j ) ;
(ii) c m o n ≤ ∂ y j S j ≤ L c_{\mathrm{mon}}\le\partial_{y_j}S_j\le L c mon ≤ ∂ y j S j ≤ L ;
(iii) ∥ S j ∥ H α ( Q d x + j ) ≤ L \|S_j\|_{\mathcal{H}^\alpha(Q^{d_x+j})}\le L ∥ S j ∥ H α ( Q d x + j ) ≤ L and ∥ ∂ y j S j ∥ H α ( Q d x + j ) ≤ L \|\partial_{y_j}S_j\|_{\mathcal{H}^\alpha(Q^{d_x+j})}\le L ∥ ∂ y j S j ∥ H α ( Q d x + j ) ≤ L .
We write Q Y ∣ X = { q S Y ∣ X : S Y ∣ X ∈ S Y ∣ X ( α , L , c m o n ) } \mathcal{Q}_{Y\mid X}=\{q_{S_{Y\mid X}}:S_{Y\mid X}\in\mathcal{S}_{Y\mid X}(\alpha,L,c_{\mathrm{mon}})\} Q Y ∣ X = { q S Y ∣ X : S Y ∣ X ∈ S Y ∣ X ( α , L , c mon )} , where q S Y ∣ X q_{S_{Y\mid X}} q S Y ∣ X is defined in (6) .
The regularity imposed on the diagonal derivatives ∂ y j S j \partial_{y_j}S_j ∂ y j S j , in addition to that of the map components, ensures that the induced density remains α \alpha α -Hölder. This follows the transport density-estimation framework of Wang and Marzouk [22 ] . We use the maximum conditional-likelihood estimator defined in (7) , with S Y ∣ X ∈ S Y ∣ X ( α , L , c m o n ) S_{Y\mid X}\in\mathcal{S}_{Y\mid X}(\alpha,L,c_{\mathrm{mon}}) S Y ∣ X ∈ S Y ∣ X ( α , L , c mon ) , and denote the resulting density by q ^ Y ∣ X = q S ^ Y ∣ X \hat q_{Y\mid X}=q_{\hat S_{Y\mid X}} q ^ Y ∣ X = q S ^ Y ∣ X . Existence and measurability of the minimizer are discussed in Appendix § 10 .
Non-asymptotic conditional and marginal density rates
In this section, we establish non-asymptotic density-estimation guarantees for the conditional and marginal KL divergence E C \mathcal{E}_C E C and E Y \mathcal{E}_Y E Y appearing in § 3 .
Theorem 4.3 (Conditional MLE rate). Under Assumptions 4.1 4.1 4.1 and 4.2 4.2 4.2 , choose L L L sufficiently large and c m o n > 0 c_{\mathrm{mon}}>0 c mon > 0 sufficiently small so that S Y ∣ X \mathcal{S}_{Y\mid X} S Y ∣ X contains the true conditional KR map. Let a = 2 α 2 α + d a=\frac{2\alpha}{2\alpha+d} a = 2 α + d 2 α . There exist constants C 1 , C 2 > 0 C_1,C_2>0 C 1 , C 2 > 0 , depending only on α , d x , d y , c , B , L 0 , L , c m o n , L η , c η , B η \alpha,d_x,d_y,c,B,L_0,L,c_{\mathrm{mon}},L_\eta,c_\eta,B_\eta α , d x , d y , c , B , L 0 , L , c mon , L η , c η , B η and not on μ X \mu_X μ X , such that, for every δ ∈ ( 0 , 1 ) \delta\in(0,1) δ ∈ ( 0 , 1 ) ,
Pr ( E C ( q ^ ) > C 1 [ N − a + log ( 1 / δ ) N ] ) ≤ δ , (17) \Pr\left(\mathcal{E}_C(\hat q)>C_1\left[N^{-a}+\frac{\log(1/\delta)}{N}\right]\right)\le\delta, \tag*{(17)} Pr ( E C ( q ^ ) > C 1 [ N − a + N log ( 1/ δ ) ] ) ≤ δ , (17) E [ E C ( q ^ ) ] ≤ C 2 N − a . (18) \mathbb{E}[\mathcal{E}_C(\hat q)]\le C_2N^{-a}. \tag*{(18)} E [ E C ( q ^ )] ≤ C 2 N − a . (18)
The key reduction is to view conditional density estimation as ordinary density estimation with respect to the dominating measure λ ( d x , d y ) = μ X ( d x ) ⊗ d y \lambda(dx,dy)=\mu_X(dx)\otimes dy λ ( d x , d y ) = μ X ( d x ) ⊗ d y . Viewed as functions of ( x , y ) (x,y) ( x , y ) , π Y ∣ X ( y ∣ x ) \pi_{Y\mid X}(y\mid x) π Y ∣ X ( y ∣ x ) and q S Y ∣ X ( y ∣ x ) q_{S_{Y\mid X}}(y\mid x) q S Y ∣ X ( y ∣ x ) are probability densities with respect to λ \lambda λ , so the conditional MLE coincides with ordinary density MLE over the corresponding class of H α \mathcal{H}^\alpha H α -densities. Let L ( S Y ∣ X ) = − E μ X , π Y ∣ X [ log q S Y ∣ X ( Y ∣ X ) ] \mathcal{L}(S_{Y\mid X})=-\mathbb{E}_{\mu_X,\pi_{Y\mid X}}[\log q_{S_{Y\mid X}}(Y\mid X)] L ( S Y ∣ X ) = − E μ X , π Y ∣ X [ log q S Y ∣ X ( Y ∣ X )] , and let S 0 , Y ∣ X S_{0,Y\mid X} S 0 , Y ∣ X denote the true conditional KR map. Then
L ( S Y ∣ X ) − L ( S 0 , Y ∣ X ) = E X ∼ μ X [ D K L ( π Y ∣ X ( ⋅ ∣ X ) ∥ q S Y ∣ X ( ⋅ ∣ X ) ) ] . (19) \begin{aligned}
\mathcal{L}(S_{Y\mid X})-\mathcal{L}(S_{0,Y\mid X})
&=\mathbb{E}_{X\sim\mu_X}\left[\mathcal{D}_{\mathrm{KL}}\left(\pi_{Y\mid X}(\cdot\mid X)\,\|\,q_{S_{Y\mid X}}(\cdot\mid X)\right)\right]. \tag*{(19)}
\end{aligned} L ( S Y ∣ X ) − L ( S 0 , Y ∣ X ) = E X ∼ μ X [ D KL ( π Y ∣ X ( ⋅ ∣ X ) ∥ q S Y ∣ X ( ⋅ ∣ X ) ) ] . (19)
At S ^ Y ∣ X = S ^ Y ∣ X \widehat{S}_{Y\mid X}=\widehat{S}_{Y\mid X} S Y ∣ X = S Y ∣ X , this excess risk equals E C ( q ^ ) \mathcal{E}_{C}(\widehat{q}) E C ( q ) , the conditional KL divergence used in the generic MI analysis. The remaining argument adapts the density-MLE proof strategy of Wang and Marzouk [22 ] . The induced conditional densities form a uniformly bounded Hölder class with bracketing entropy of order ρ − d / α \rho^{-d/\alpha} ρ − d / α at bracket width ρ > 0 \rho>0 ρ > 0 . Since α > d / 2 \alpha>d/2 α > d /2 , the corresponding entropy integral gives a Hellinger critical radius of order N − α / ( 2 α + d ) N^{-\alpha/(2\alpha+d)} N − α / ( 2 α + d ) . Uniform upper and positive lower bounds on the densities then convert squared Hellinger distance into KL divergence, yielding Theorem 4.3 . The resulting leading KL rate N − 2 α / ( 2 α + d ) N^{-2\alpha/(2\alpha+d)} N − 2 α / ( 2 α + d ) agrees with the minimax-optimal rate established by Wang and Marzouk [22 ] for ordinary d d d -dimensional KR density estimation. Our conditional result adapts their argument by viewing the conditional density as a density with respect to μ X ⊗ d y \mu_X\otimes dy μ X ⊗ d y . The marginal rate below follows directly from their ordinary KR result in dimension d y d_y d y . The formal reduction, realizability argument, and entropy bounds are given in Appendix § 10 .§ 2 .
We next obtain the corresponding rate for the marginal estimator. No additional regularity assumption on π Y \pi_Y π Y is needed, since its smoothness and positivity follow directly from those of π Y ∣ X \pi_{Y\mid X} π Y ∣ X .
Lemma 4.4 (Induced regularity of the marginal). Under Assumption 4.1, π Y ∈ H α ( Q d y ) \pi_Y\in H^\alpha(Q^{d_y}) π Y ∈ H α ( Q d y ) , ∥ π Y ∥ H α ≤ L 0 \|\pi_Y\|_{H^\alpha}\le L_0 ∥ π Y ∥ H α ≤ L 0 , and c ≤ π Y ≤ B c\le\pi_Y\le B c ≤ π Y ≤ B .
Lemma 4.4 allows us to apply the ordinary KR density-estimation result of Wang and Marzouk [22 ] directly in dimension d y d_y d y . Let S Y ( α , L Y , c m o n , Y ) S_Y(\alpha,L_Y,c_{\mathrm{mon},Y}) S Y ( α , L Y , c mon , Y ) denote the d y d_y d y -dimensional analogue of Definition 4.2, with constants chosen so that it contains the true marginal KR map. The marginal estimator is fitted to the same training observations { Y i } i = 1 N \{Y_i\}_{i=1}^{N} { Y i } i = 1 N .
Lemma 4.5 (Marginal MLE rate). Let S ^ Y \widehat{S}_Y S Y be a maximum-likelihood map over S Y S_Y S Y based on { Y i } i = 1 N \{Y_i\}_{i=1}^{N} { Y i } i = 1 N , let q ^ Y = q S ^ Y \widehat{q}_Y=q_{\widehat{S}_Y} q Y = q S Y be its induced density, and let b = 2 α / ( 2 α + d y ) b=2\alpha/(2\alpha+d_y) b = 2 α / ( 2 α + d y ) . Under Assumptions 4.1 and 4.2, there exist constants C Y , 1 , C Y , 2 > 0 C_{Y,1},C_{Y,2}>0 C Y , 1 , C Y , 2 > 0 such that, for every δ ∈ ( 0 , 1 ) \delta\in(0,1) δ ∈ ( 0 , 1 ) ,
Pr ( E Y ( q ^ ) > C Y , 1 [ N − b + log ( 1 / δ ) N ] ) ≤ δ , (20) \Pr\left(\mathcal{E}_Y(\widehat{q})>C_{Y,1}\left[N^{-b}+\frac{\log(1/\delta)}{N}\right]\right)\le\delta, \tag*{(20)} Pr ( E Y ( q ) > C Y , 1 [ N − b + N log ( 1/ δ ) ] ) ≤ δ , (20) E [ E Y ( q ^ ) ] ≤ C Y , 2 N − b . (21) \mathbb{E}[\mathcal{E}_Y(\widehat{q})]\le C_{Y,2}N^{-b}. \tag*{(21)} E [ E Y ( q )] ≤ C Y , 2 N − b . (21)
The proof of Lemma 4.4 and the specialization of the ordinary KR rate are given in Appendix § 10 . Finally, the high-probability bounds above yield the second moments needed for the MSE analysis. Integrating the corresponding KL tails gives E [ E C ( q ^ ) 2 ] ≤ C C , 3 N − 2 a \mathbb{E}[\mathcal{E}_C(\widehat{q})^2]\le C_{C,3}N^{-2a} E [ E C ( q ) 2 ] ≤ C C , 3 N − 2 a and E [ E Y ( q ^ ) 2 ] ≤ C Y , 3 N − 2 b \mathbb{E}[\mathcal{E}_Y(\widehat{q})^2]\le C_{Y,3}N^{-2b} E [ E Y ( q ) 2 ] ≤ C Y , 3 N − 2 b for suitable constants C C , 3 , C Y , 3 > 0 C_{C,3},C_{Y,3}>0 C C , 3 , C Y , 3 > 0 . The tail-integration argument is given in Lemma B.12 of Appendix § 10 .§ 5 .
Consequences for mutual information
We now combine the conditional and marginal density-estimation rates with the generic results of § 3 . Theorem 4.3 and Lemma 4.5 provide the bounds E C \mathcal{E}_C E C and E Y \mathcal{E}_Y E Y in Assumption 3.2, while the uniform upper and positive lower bounds on the fitted transport densities imply Assumption 3.1.
Corollary 4.6 (Finite-sample MI bounds). Under Assumptions 4.1 and 4.2, fit the conditional and marginal estimators of Theorem 4.3 and Lemma 4.5 using the same N N N training pairs. Let [ c ‾ , B ‾ ] [\underline{c},\overline{B}] [ c , B ] and [ c ‾ Y , B ‾ Y ] [\underline{c}_Y,\overline{B}_Y] [ c Y , B Y ] denote uniform lower and upper bounds for the fitted conditional and marginal densities, respectively, and define K = 1 2 log ( B ‾ B ‾ Y c ‾ c ‾ Y ) K=\frac{1}{2}\log\left(\frac{\overline{B}\,\overline{B}_Y}{\underline{c}\,\underline{c}_Y}\right) K = 2 1 log ( c c Y B B Y ) . Let a = 2 α / ( 2 α + d x + d y ) a=2\alpha/(2\alpha+d_x+d_y) a = 2 α / ( 2 α + d x + d y ) and b = 2 α / ( 2 α + d y ) b=2\alpha/(2\alpha+d_y) b = 2 α / ( 2 α + d y ) , so that a < b a<b a < b . For the split-sample estimator I ^ N , M \widehat{I}_{N,M} I N , M in (8) , the following hold:
For every δ ∈ ( 0 , 1 ) \delta\in(0,1) δ ∈ ( 0 , 1 ) , with probability at least 1 − δ 1-\delta 1 − δ ,
∣ I ^ N , M − I ∣ ≤ K 2 log ( 6 / δ ) M + C 1 [ N − a + log ( 3 / δ ) N ] + C Y , 1 [ N − b + log ( 3 / δ ) N ] . (22) \left|\widehat{I}_{N,M}-I\right| \le K\sqrt{\frac{2\log(6/\delta)}{M}}+C_1\left[N^{-a}+\frac{\log(3/\delta)}{N}\right]+C_{Y,1}\left[N^{-b}+\frac{\log(3/\delta)}{N}\right]. \tag*{(22)} I N , M − I ≤ K M 2 log ( 6/ δ ) + C 1 [ N − a + N log ( 3/ δ ) ] + C Y , 1 [ N − b + N log ( 3/ δ ) ] . (22)
There exist constants C ′ , C ′ ′ > 0 C^{\prime}, C^{\prime\prime}>0 C ′ , C ′′ > 0 such that
E [ ( I ^ N , M − I ) 2 ] ≤ C ′ M + C ′ ′ N − 2 a . (23) \mathbb{E}\left[(\widehat{I}_{N,M}-I)^2\right]\leq\frac{C^{\prime}}{M}+C^{\prime\prime}N^{-2a}. \tag*{(23)} E [ ( I N , M − I ) 2 ] ≤ M C ′ + C ′′ N − 2 a . (23)
The proof is given in Appendix § 10 .§ 6 . Since a < b a<b a < b , the conditional density-estimation error decays more slowly than the marginal error, so the training contribution to the MI MSE is governed by N − 2 a N^{-2a} N − 2 a . Thus, up to constants, E [ ( I ^ N , M − I ) 2 ] ≲ M − 1 + N − 2 a \mathbb{E}\left[(\widehat{I}_{N,M}-I)^2\right]\lesssim M^{-1}+N^{-2a} E [ ( I N , M − I ) 2 ] ≲ M − 1 + N − 2 a . The two terms respectively capture the Monte Carlo evaluation error, controlled by M M M , and the density-estimation error, controlled by N N N . This tradeoff forms the basis of the sample-allocation analysis in the next section.
Optimal sample allocation
The MSE bound (23) separates the effects of the training and evaluation sample sizes as E [ ( I ^ N , M − I ) 2 ] ≤ C ′ / M + C ′ ′ N − 2 a \mathbb{E}\left[(\widehat{I}_{N,M}-I)^2\right]\leq C^{\prime}/M+C^{\prime\prime}N^{-2a} E [ ( I N , M − I ) 2 ] ≤ C ′ / M + C ′′ N − 2 a . Under a fixed total sample budget T = N + M T=N+M T = N + M , we optimize this upper bound over the allocation between training and evaluation samples. Here T T T counts samples only; the computational cost of fitting the transport maps is not modeled.
Corollary 5.1 (Allocation minimizing the MSE upper bound). Let a = 2 α / ( 2 α + d x + d y ) ∈ ( 1 / 2 , 1 ) a=2\alpha/(2\alpha+d_x+d_y)\in(1/2,1) a = 2 α / ( 2 α + d x + d y ) ∈ ( 1/2 , 1 ) and C ′ , C ′ ′ > 0 C^{\prime},C^{\prime\prime}>0 C ′ , C ′′ > 0 . Under N + M = T N+M=T N + M = T , let ( N ⋆ , M ⋆ ) (N^\star,M^\star) ( N ⋆ , M ⋆ ) minimize C ′ / M + C ′ ′ N − 2 a C^{\prime}/M+C^{\prime\prime}N^{-2a} C ′ / M + C ′′ N − 2 a . Then, as T → ∞ T\to\infty T → ∞ ,
N ⋆ ≍ T 2 / ( 2 a + 1 ) , M ⋆ T → 1. (24) N^\star\asymp T^{2/(2a+1)},\qquad\frac{M^\star}{T}\to1. \tag*{(24)} N ⋆ ≍ T 2/ ( 2 a + 1 ) , T M ⋆ → 1. (24) At this allocation, E [ ( I ^ N ⋆ , M ⋆ − I ) 2 ] ≲ T − 1 \mathbb{E}\left[(\widehat{I}_{N^\star,M^\star}-I)^2\right]\lesssim T^{-1} E [ ( I N ⋆ , M ⋆ − I ) 2 ] ≲ T − 1 . The smaller training order N ≍ T 1 / ( 2 a ) N\asymp T^{1/(2a)} N ≍ T 1/ ( 2 a ) , with M = T − N M=T-N M = T − N , also attains the T − 1 T^{-1} T − 1 MSE rate, but generally does not minimize the upper bound.
The proof is given in Appendix § 10 .§ 7 . The distinction between the two training orders follows from minimizing the bound rather than simply balancing its two terms. In particular, the first-order condition is C ′ / ( T − N ⋆ ) 2 = 2 a C ′ ′ ( N ⋆ ) − ( 2 a + 1 ) C^{\prime}/(T-N^\star)^2=2aC^{\prime\prime}(N^\star)^{-(2a+1)} C ′ / ( T − N ⋆ ) 2 = 2 a C ′′ ( N ⋆ ) − ( 2 a + 1 ) . Since a > 1 / 2 a>1/2 a > 1/2 , this implies N ⋆ = o ( T ) N^\star=o(T) N ⋆ = o ( T ) and hence M ⋆ / T → 1 M^\star/T\to1 M ⋆ / T → 1 . More precisely, N ⋆ ∼ ( 2 a C ′ ′ / C ′ ) 1 / ( 2 a + 1 ) T 2 / ( 2 a + 1 ) N^\star\sim(2aC^{\prime\prime}/C^{\prime})^{1/(2a+1)}T^{2/(2a+1)} N ⋆ ∼ ( 2 a C ′′ / C ′ ) 1/ ( 2 a + 1 ) T 2/ ( 2 a + 1 ) ; thus the multiplicative constant is problem-dependent, and the asymptotic scaling does not provide a constant-free finite-budget prescription.
For the setting used in the numerical experiments, d x = d y = 1 d_x=d_y=1 d x = d y = 1 and α = 2 \alpha=2 α = 2 , so a = 2 / 3 a=2/3 a = 2/3 . Corollary 5.1 then gives the bound-minimizing training order N ⋆ ≍ T 6 / 7 N^\star\asymp T^{6/7} N ⋆ ≍ T 6/7 , while the smaller order N ≍ T 3 / 4 N\asymp T^{3/4} N ≍ T 3/4 is already sufficient to attain the T − 1 T^{-1} T − 1 MSE rate. In both cases, M / T → 1 M/T\to1 M / T → 1 . More generally, the assumption α > ( d x + d y ) / 2 \alpha>(d_x+d_y)/2 α > ( d x + d y ) /2 implies 2 a > 1 2a>1 2 a > 1 , allowing the T − 1 T^{-1} T − 1 rate to be achieved with a sublinear training budget. For comparison, the parametric bound O ( M − 1 + N − 2 ) O(M^{-1}+N^{-2}) O ( M − 1 + N − 2 ) of Li et al. (2026) yields the minimizing training order T 2 / 3 T^{2/3} T 2/3 . As α → ∞ \alpha\to\infty α → ∞ with the dimensions fixed, a → 1 a\to1 a → 1 and our exponent 2 / ( 2 a + 1 ) 2/(2a+1) 2/ ( 2 a + 1 ) likewise approaches 2 / 3 2/3 2/3 . The O ( T − 1 ) O(T^{-1}) O ( T − 1 ) rate matches the canonical Monte Carlo MSE rate and is the fastest rate permitted by the M − 1 M^{-1} M − 1 evaluation term when its variance is bounded away from zero.
Numerical experiments
We use numerical experiments to examine the density-estimation rates, the training–evaluation error decomposition, and the sample-allocation behavior predicted by the theory. All experiments use the same synthetic distribution with d x = d y = 1 d_x=d_y=1 d x = d y = 1 and X ∼ Unif [ 0 , 1 ] X\sim\operatorname{Unif}[0,1] X ∼ Unif [ 0 , 1 ] . Define h ( t ) = ( 2 t − 1 ) ∣ 2 t − 1 ∣ h(t)=(2t-1)|2t-1| h ( t ) = ( 2 t − 1 ) ∣2 t − 1∣ and w ( x ) = ( 1 + h ( x ) ) / 2 w(x)=(1+h(x))/2 w ( x ) = ( 1 + h ( x )) /2 , so that ∫ 0 1 h ( t ) d t = 0 \int_0^1 h(t)\,\mathrm{d}t=0 ∫ 0 1 h ( t ) d t = 0 and ∫ 0 1 w ( x ) d x = 1 / 2 \int_0^1 w(x)\,\mathrm{d}x=1/2 ∫ 0 1 w ( x ) d x = 1/2 . For x , y ∈ [ 0 , 1 ] x,y\in[0,1] x , y ∈ [ 0 , 1 ] , let π Y ∣ X ( y ∣ x ) = 1 + 0.6 w ( x ) h ( y ) \pi_{Y\mid X}(y\mid x)=1+0.6w(x)h(y) π Y ∣ X ( y ∣ x ) = 1 + 0.6 w ( x ) h ( y ) . Since ∫ 0 1 h ( y ) d y = 0 \int_0^1 h(y)\,\mathrm{d}y=0 ∫ 0 1 h ( y ) d y = 0 , this is a normalized conditional density for every x x x . Because X X X is uniform, the joint Lebesgue density is π X , Y ( x , y ) = π Y ∣ X ( y ∣ x ) \pi_{X,Y}(x,y)=\pi_{Y\mid X}(y\mid x) π X , Y ( x , y ) = π Y ∣ X ( y ∣ x ) , while integration over x x x gives π Y ( y ) = 1 + 0.3 h ( y ) \pi_Y(y)=1+0.3h(y) π Y ( y ) = 1 + 0.3 h ( y ) . On [ 0 , 1 ] 2 [0,1]^2 [ 0 , 1 ] 2 , 0.4 ≤ π Y ∣ X ( y ∣ x ) ≤ 1.6 0.4\leq\pi_{Y\mid X}(y\mid x)\leq1.6 0.4 ≤ π Y ∣ X ( y ∣ x ) ≤ 1.6 , and the density is C 1 , 1 C^{1,1} C 1 , 1 but not C 2 C^2 C 2 , corresponding to α = 2 \alpha=2 α = 2 under Definition 4.1. The true value of MI is obtained using numerical integration. Sampling and implementation details are given in Appendix C.1.
We fit conditional and marginal transport estimators using the same N N N training pairs. In the present one-dimensional setting, the derivatives of the KR maps are parameterized by positive, normalized quadratic B-splines and fitted by maximum likelihood. Further implementation details are given in Appendix C. Recall the decomposition I ^ N , M − I = A M + B ( q ^ ) \widehat{I}_{N,M}-I=A_M+B(\widehat{q}) I N , M − I = A M + B ( q ) , where A M A_M A M is the Monte Carlo evaluation error and B ( q ^ ) = E Y ( q ^ ) − E C ( q ^ ) B(\widehat{q})=\mathcal{E}_Y(\widehat{q})-\mathcal{E}_C(\widehat{q}) B ( q ) = E Y ( q ) − E C ( q ) is the training-dependent plug-in error. With V ( q ^ ) V(\widehat{q}) V ( q ) defined in (11) , the independence of the training and evaluation samples gives E [ ( I ^ N , M − I ) 2 ] = E [ B ( q ^ ) 2 ] + E [ V ( q ^ ) ] / M \mathbb{E}[(\widehat{I}_{N,M}-I)^2]=\mathbb{E}[B(\widehat{q})^2]+\mathbb{E}[V(\widehat{q})]/M E [( I N , M − I ) 2 ] = E [ B ( q ) 2 ] + E [ V ( q )] / M . We examine these two contributions separately before studying their allocation under a fixed total budget.
Training and evaluation error
For this example, α = 2 \alpha=2 α = 2 and d x = d y = 1 d_x=d_y=1 d x = d y = 1 , so a = 2 α / ( 2 α + d x + d y ) = 2 / 3 a=2\alpha/(2\alpha+d_x+d_y)=2/3 a = 2 α / ( 2 α + d x + d y ) = 2/3 and b = 2 α / ( 2 α + d y ) = 4 / 5 b=2\alpha/(2\alpha+d_y)=4/5 b = 2 α / ( 2 α + d y ) = 4/5 . The theory therefore predicts conditional and marginal KL rates of N − 2 / 3 N^{-2/3} N − 2/3 and N − 4 / 5 N^{-4/5} N − 4/5 , respectively, and a squared plug-in error of order N − 4 / 3 N^{-4/3} N − 4/3 .
For each training sample size N N N , we repeat the fitting procedure on 30 independent training datasets. Figure 1 (a) and Figure 1 (b) show that the expected conditional and marginal KL divergence follow the predicted N − 2 / 3 N^{-2/3} N − 2/3 and N − 4 / 5 N^{-4/5} N − 4/5 rates, while E [ B ( q ^ ) 2 ] \mathbb{E}[B(\widehat{q})^2] E [ B ( q ) 2 ] exhibits the predicted N − 4 / 3 N^{-4/3} N − 4/3 scaling. These results support the training-error term N − 2 a N^{-2a} N − 2 a in (23) .
Figure 1. Training- and evaluation-sample convergence. Panels (a)–(b) vary the training sample size N N N . Panel (a) shows the expected conditional and marginal KL divergence, averaged over 30 independent training datasets for each N N N , and panel (b) shows the corresponding squared plug-in error E [ B ( q ^ ) 2 ] \mathbb{E}[B(\hat{q})^2] E [ B ( q ^ ) 2 ] . Panels (c)–(d) fix the training sample size at N = 5000 N = 5000 N = 5000 , hold the fitted densities fixed, and vary the evaluation sample size M M M . Panel (c) shows the Monte Carlo evaluation error, while panel (d) shows the MSE relative to the true MI and its limiting plug-in floor. Error bars or shading denote 95% pointwise bootstrap intervals.
We next fix the training sample size at N = 5000 N=5000 N = 5000 and hold the resulting fitted densities fixed while varying the evaluation sample size M M M . Conditional on q ^ \widehat{q} q , the evaluation error satisfies E [ A q ^ 2 ] = V ( q ^ ) / M \mathbb{E}[A_{\widehat{q}}^2]=V(\widehat{q})/M E [ A q 2 ] = V ( q ) / M , while the MSE relative to the true MI is E [ ( I ^ N , M − I ) 2 ∣ q ^ ] = B ( q ^ ) 2 + V ( q ^ ) / M \mathbb{E}[(\widehat{I}_{N,M}-I)^2\mid\widehat{q}]=B(\widehat{q})^2+V(\widehat{q})/M E [( I N , M − I ) 2 ∣ q ] = B ( q ) 2 + V ( q ) / M . Figure 1 (c) and Figure 1 (d) confirm the resulting M − 1 M^{-1} M − 1 decay and show that, as M M M increases, the total error approaches the fixed plug-in floor B ( q ^ ) 2 B(\widehat{q})^2 B ( q ) 2 .
Together, the four panels empirically support the two components of the finite-sample MSE bound MSE ( I ^ N , M ) ≲ N − 4 / 3 + M − 1 \operatorname{MSE}(\widehat{I}_{N,M})\lesssim N^{-4/3}+M^{-1} MSE ( I N , M ) ≲ N − 4/3 + M − 1 , which motivates the sample-allocation experiment below.
Sample allocation under a fixed budget
We next consider a fixed total sample budget T = N + M T=N+M T = N + M . For each allocation and each T T T , we repeat the training procedure R = 30 R=30 R = 30 times using independent training datasets. For the fitted pair q ^ ( r ) \widehat{q}^{(r)} q ( r ) from repetition r r r , we compute B ( q ^ ( r ) ) B(\widehat{q}^{(r)}) B ( q ( r ) ) and V ( q ^ ( r ) ) V(\widehat{q}^{(r)}) V ( q ( r ) ) by numerical quadrature against the known joint density. The evaluation-averaged MSE is then estimated by
R ^ T ( N ) = 1 R ∑ r = 1 R [ B ( q ^ ( r ) ) 2 + V ( q ^ ( r ) ) M ] . (25) \widehat{R}_T(N)=\frac{1}{R}\sum_{r=1}^{R}\left[B(\widehat{q}^{(r)})^2+\frac{V(\widehat{q}^{(r)})}{M}\right]. \tag*{(25)} R T ( N ) = R 1 r = 1 ∑ R [ B ( q ( r ) ) 2 + M V ( q ( r ) ) ] . (25)
Thus, no evaluation datasets are generated in this experiment; the expectation over an independent evaluation sample is computed directly by quadrature. For this example, a = 2 / 3 a=2/3 a = 2/3 . Corollary 5.1 therefore gives the upper bound minimizing scaling N ≍ T 6 / 7 N \asymp T^{6/7} N ≍ T 6/7 , while N ≍ T 3 / 4 N \asymp T^{3/4} N ≍ T 3/4 is sufficient to attain the T − 1 T^{-1} T − 1 MSE rate. We compare these two theoretically motivated allocations over nine budgets from T = 2000 T = 2000 T = 2000 to T = 10 6 T = 10^6 T = 1 0 6 . Specifically, for γ ∈ { 6 / 7 , 3 / 4 } \gamma\in\{6/7, 3/4\} γ ∈ { 6/7 , 3/4 } , we use N γ ( T ) = round [ 1000 ( T / 2000 ) γ ] N_\gamma(T) = \operatorname{round}[1000(T/2000)^\gamma] N γ ( T ) = round [ 1000 ( T /2000 ) γ ] , with M = T − N γ ( T ) M = T - N_\gamma(T) M = T − N γ ( T ) . Both schemes are normalized to use N = 1000 N = 1000 N = 1000 training samples at T = 2000 T = 2000 T = 2000 , so they differ only in their scaling exponent. No multiplicative constants are tuned using the observed risk.
Figure 2 shows the resulting tradeoff. The T 6 / 7 T^{6/7} T 6/7 scaling allocates more samples to training and achieves a smaller squared plug-in error. Over the tested budget range, its total MSE also decreases faster than that of the T 3 / 4 T^{3/4} T 3/4 scaling, with the two allocations coinciding at T = 2000 T = 2000 T = 2000 . Although the T 3 / 4 T^{3/4} T 3/4 scaling leaves more samples for evaluation, its Monte Carlo contribution is not uniformly smaller because the fitted variance also depends on the training sample size. The T 6 / 7 T^{6/7} T 6/7 allocation is obtained by minimizing this finite-sample upper bound, whereas T 3 / 4 T^{3/4} T 3/4 is only required to achieve the same asymptotic O ( T − 1 ) O(T^{-1}) O ( T − 1 ) rate. Accordingly, the theory predicts that the larger training allocation can yield a smaller finite-budget error even though both allocations attain the same asymptotic rate. The numerical results exhibit precisely this tradeoff over the budgets considered.
Figure 2. Sample allocation under a fixed total budget. Panel (a) shows the empirical squared plug-in error E [ B ( q ^ ) 2 ] \mathbb{E}[B(\hat{q})^2] E [ B ( q ^ ) 2 ] , panel (b) shows the Monte Carlo evaluation contribution E [ V ( q ^ ) ] / M \mathbb{E}[V(\hat{q})]/M E [ V ( q ^ )] / M , and panel (c) shows the MSE E [ ( I ^ N , M − I ) 2 ] \mathbb{E}[(\hat{I}_{N,M} - I)^2] E [( I ^ N , M − I ) 2 ] for the N ∝ T 6 / 7 N \propto T^{6/7} N ∝ T 6/7 and N ∝ T 3 / 4 N \propto T^{3/4} N ∝ T 3/4 allocations. Shading denotes 95% pointwise paired training-bootstrap intervals.
Conclusion and future work
We developed a finite-sample analysis of transport-based mutual information estimation using a split-sample procedure. The error is decomposed into a density-estimation component and a Monte Carlo evaluation component. For KR maps over realizable Hölder classes, we obtain conditional and marginal KL rates of N − 2 α / ( 2 α + d x + d y ) N^{-2\alpha/(2\alpha+d_x+d_y)} N − 2 α / ( 2 α + d x + d y ) and N − 2 α / ( 2 α + d y ) N^{-2\alpha/(2\alpha+d_y)} N − 2 α / ( 2 α + d y ) , respectively, by treating the conditional density as an ordinary density with respect to the conditioning-variable distribution and Lebesgue measure. These results yield an MSE bound of order M − 1 + N − 4 α / ( 2 α + d x + d y ) M^{-1}+N^{-4\alpha/(2\alpha+d_x+d_y)} M − 1 + N − 4 α / ( 2 α + d x + d y ) with the conditional density-estimation term dominating the marginal term. Under a fixed budget T = N + M T=N+M T = N + M , the analysis further characterizes how samples should be allocated between training and evaluation. The allocation minimizing the MSE upper bound uses a sublinear number of training samples, while an even smaller sublinear training allocation is sufficient to attain the O ( T − 1 ) O(T^{-1}) O ( T − 1 ) MSE rate. Numerical experiments support the predicted training and evaluation error rates and illustrate the resulting allocation tradeoff.
AI use statement
In this work, we used generative AI tools to help develop and refine theoretical formulations, formulate mathematical claims, assist with the development and presentation of proofs, provide feedback on research methodology and numerical experiments. Additionally, we used generative AI tools to assist with writing and editing parts of the manuscript and improving readability and organization. We take responsibility for the final content of this work, including text, claims, or artifacts produced with the aid of generative AI.
Proofs for the plug-in analysis
Proof of Lemma 3.1
Proof. Conditionally on the fitted densities, independence of the evaluation sample and the disintegration (2) give
E [ I ^ N , M − I ∣ q ^ ] = ∫ X ∫ Y π Y ∣ X ( y ∣ x ) log q ^ Y ∣ X ( y ∣ x ) π Y ∣ X ( y ∣ x ) d y μ X ( d x ) − ∫ Y π Y ( y ) log q ^ Y ( y ) π Y ( y ) d y \mathbb{E}[\widehat{I}_{N,M}-I\mid\widehat{q}]=\int_{\mathcal{X}}\int_{\mathcal{Y}}\pi_{Y\mid X}(y\mid x)\log\frac{\widehat{q}_{Y\mid X}(y\mid x)}{\pi_{Y\mid X}(y\mid x)}\,dy\,\mu_X(dx)-\int_{\mathcal{Y}}\pi_Y(y)\log\frac{\widehat{q}_Y(y)}{\pi_Y(y)}\,dy E [ I N , M − I ∣ q ] = ∫ X ∫ Y π Y ∣ X ( y ∣ x ) log π Y ∣ X ( y ∣ x ) q Y ∣ X ( y ∣ x ) d y μ X ( d x ) − ∫ Y π Y ( y ) log π Y ( y ) q Y ( y ) d y
which is exactly (13) . No density of μ X \mu_X μ X with respect to Lebesgue measure is used.
Proof of Theorem 3.2
Proof. Adding and subtracting the conditional mean g ( q ^ ) = E [ I ^ N , M ∣ q ^ ] g(\widehat{q})=\mathbb{E}[\widehat{I}_{N,M}\mid\widehat{q}] g ( q ) = E [ I N , M ∣ q ] , and using Lemma 3.1 to identify g ( q ^ ) − I = E Y ( q ^ ) − E C ( q ^ ) g(\widehat{q})-I=\mathcal{E}_Y(\widehat{q})-\mathcal{E}_C(\widehat{q}) g ( q ) − I = E Y ( q ) − E C ( q ) , gives I ^ N , M − I = A M + B ( q ^ ) \widehat{I}_{N,M}-I=A_M+B(\widehat{q}) I N , M − I = A M + B ( q ) as in (14) .
Conditionally on D t r a i n D_{\mathrm{train}} D train , the fitted log ratio is fixed and the evaluation samples are i.i.d. from μ X , Y \mu_{X,Y} μ X , Y . For K > 0 K>0 K > 0 , Assumption 3.1 gives
Pr ( ∣ A M ∣ > t ∣ q ^ ) ≤ 2 exp ( − M t 2 2 K 2 ) . \Pr(|A_M|>t\mid\widehat{q})\leq2\exp\left(-\frac{Mt^2}{2K^2}\right). Pr ( ∣ A M ∣ > t ∣ q ) ≤ 2 exp ( − 2 K 2 M t 2 ) .
For K > 0 K>0 K > 0 , taking t = K 2 log ( 6 / δ ) / M t=K\sqrt{2\log(6/\delta)/M} t = K 2 log ( 6/ δ ) / M makes this probability at most δ / 3 \delta/3 δ /3 . If K = 0 K=0 K = 0 , the centered summand is zero almost surely and the same conclusion holds directly.
For the plug-in term, Lemma 3.1 and nonnegativity of KL give
∣ B ( q ^ ) ∣ ≤ E C ( q ^ ) + E Y ( q ^ ) . (26) |B(\widehat{q})|\leq\mathcal{E}_C(\widehat{q})+\mathcal{E}_Y(\widehat{q}). \tag*{(26)} ∣ B ( q ) ∣ ≤ E C ( q ) + E Y ( q ) . (26)
To combine these bounds, define
E 1 = { ∣ A M ∣ ≤ K 2 log ( 6 / δ ) M } , E 2 = { E C ( q ^ ) ≤ ε C ( N , δ / 3 ) } , E 3 = { E Y ( q ^ ) ≤ ε Y ( N , δ / 3 ) } . \begin{aligned}
E_1&=\left\{|A_M|\leq K\sqrt{\frac{2\log(6/\delta)}{M}}\right\},\\
E_2&=\left\{\mathcal{E}_C(\widehat{q})\leq\varepsilon_C(N,\delta/3)\right\},\\
E_3&=\left\{\mathcal{E}_Y(\widehat{q})\leq\varepsilon_Y(N,\delta/3)\right\}.
\end{aligned} E 1 E 2 E 3 = { ∣ A M ∣ ≤ K M 2 log ( 6/ δ ) } , = { E C ( q ) ≤ ε C ( N , δ /3 ) } , = { E Y ( q ) ≤ ε Y ( N , δ /3 ) } .
Each complement has probability at most δ / 3 \delta/3 δ /3 , by the conditional tail bound above and Assumption 3.2. Therefore Pr ( E 1 ∩ E 2 ∩ E 3 ) ≥ 1 − δ \Pr(E_1\cap E_2\cap E_3)\geq1-\delta Pr ( E 1 ∩ E 2 ∩ E 3 ) ≥ 1 − δ , without requiring independence between the fitted densities. On this event, the triangle inequality and (26) give (15) .
A sufficient condition for sub-Gaussian evaluation
The following lemma verifies the evaluation-tail assumption from uniform bounds on the fitted densities.
Lemma A.1 (Sufficient condition for sub-Gaussianity). Suppose the estimated densities satisfy the uniform bounds
0 < c ℓ ≤ q ^ Y ∣ X ( y ∣ x ) ≤ c u , 0 < c ℓ ′ ≤ q ^ Y ( y ) ≤ c u ′ . (27) 0<c_{\ell}\leq\widehat{q}_{Y\mid X}(y\mid x)\leq c_u,\qquad0<c'_{\ell}\leq\widehat{q}_Y(y)\leq c'_u. \tag*{(27)} 0 < c ℓ ≤ q Y ∣ X ( y ∣ x ) ≤ c u , 0 < c ℓ ′ ≤ q Y ( y ) ≤ c u ′ . (27)
for all ( x , y ) (x,y) ( x , y ) in the support X × Y \mathcal{X}\times\mathcal{Y} X × Y , almost surely over the training data. Then φ ^ ( X , Y ) = log q ^ Y ∣ X ( Y ∣ X ) − log q ^ Y ( Y ) \hat{\varphi}(X,Y)=\log\hat{q}_{Y\mid X}(Y\mid X)-\log\hat{q}_Y(Y) φ ^ ( X , Y ) = log q ^ Y ∣ X ( Y ∣ X ) − log q ^ Y ( Y ) is bounded and Assumption 3.1 holds with
K = 1 2 [ log ( c u / c ℓ ) − log ( c ℓ / c u ′ ) ] = 1 2 log c u c u ′ c ℓ c ℓ ′ . K=\frac{1}{2}\left[\log(c_u/c_\ell)-\log(c_\ell/c'_u)\right]=\frac{1}{2}\log\frac{c_uc'_u}{c_\ell c'_\ell}. K = 2 1 [ log ( c u / c ℓ ) − log ( c ℓ / c u ′ ) ] = 2 1 log c ℓ c ℓ ′ c u c u ′ .
Proof. Under (27) , the log-density ratio satisfies log ( c ℓ / c u ′ ) ≤ φ ^ ( x , y ) ≤ log ( c u / c ℓ ′ ) \log(c_\ell/c'_u)\le\hat{\varphi}(x,y)\le\log(c_u/c'_\ell) log ( c ℓ / c u ′ ) ≤ φ ^ ( x , y ) ≤ log ( c u / c ℓ ′ ) for all ( x , y ) (x,y) ( x , y ) . Centering preserves this interval’s length 2 K 2K 2 K . Hoeffding’s lemma gives the moment-generating-function bound e t 2 K 2 / 2 e^{t^2K^2/2} e t 2 K 2 /2 for every t ∈ R t\in\mathbb{R} t ∈ R , as required. For K > 0 K>0 K > 0 and any t > 0 t>0 t > 0 , it also gives
Pr ( ∣ 1 M ∑ j = 1 M φ ^ ( X j ′ , Y j ′ ) − g ( q ^ ) ∣ > t | q ^ ) ≤ 2 exp ( − M t 2 2 K 2 ) . \Pr\left(\left|\frac{1}{M}\sum_{j=1}^{M}\hat{\varphi}(X'_j,Y'_j)-g(\hat{q})\right|>t\ \middle|\ \hat{q}\right)\le2\exp\left(-\frac{Mt^2}{2K^2}\right). Pr ( M 1 j = 1 ∑ M φ ^ ( X j ′ , Y j ′ ) − g ( q ^ ) > t q ^ ) ≤ 2 exp ( − 2 K 2 M t 2 ) .
When K = 0 K=0 K = 0 , the log ratio is constant and the centered average is zero almost surely.
Note that for the map class of Definition 4.2, the positive reference density and uniform monotonicity give a positive lower bound on each induced density. The reference and derivative upper bounds give the corresponding upper bound. Thus Assumption 3.1 follows from the explicitly imposed map-class constraints; smoothness of the true density alone would not suffice.
Proofs for conditional KR estimation
Throughout this section, we abbreviate the conditional maps S Y ∣ X S_{Y\mid X} S Y ∣ X and S 0 , Y ∣ X S_{0,Y\mid X} S 0 , Y ∣ X by S S S and S 0 S_0 S 0 , respectively, and write q S = q S Y ∣ X q_S=q_{S_{Y\mid X}} q S = q S Y ∣ X . Thus S S S denotes only the lower block in this section, not the full map in (5) .
Conditional likelihood and population risk
The following results justify the dominating-measure reduction used in § 4.2 .
Lemma B.1 (Base-measure reduction). Define the finite measure on Q d Q^d Q d
λ ( d x , d y ) : = μ X ( d x ) ⊗ d y . (28) \lambda(dx,dy):=\mu_X(dx)\otimes dy. \tag*{(28)} λ ( d x , d y ) := μ X ( d x ) ⊗ d y . (28) Then:
for every S ∈ S Y ∣ X S\in\mathcal{S}_{Y\mid X} S ∈ S Y ∣ X , the function r S ( x , y ) : = q S ( y ∣ x ) r_S(x,y):=q_S(y\mid x) r S ( x , y ) := q S ( y ∣ x ) is a probability density with respect to λ \lambda λ ;
the true conditional density is the λ \lambda λ -density of μ X , Y \mu_{X,Y} μ X , Y , that is,
r 0 ( x , y ) : = π Y ∣ X ( y ∣ x ) = d μ X , Y d λ ( x , y ) ; r_0(x,y):=\pi_{Y\mid X}(y\mid x)=\frac{d\mu_{X,Y}}{d\lambda}(x,y); r 0 ( x , y ) := π Y ∣ X ( y ∣ x ) = d λ d μ X , Y ( x , y ) ; the conditional MLE (7) coincides with the ordinary MLE over the class of λ \lambda λ -densities R : = { r S : S ∈ S Y ∣ X } \mathcal{R}:=\{r_S:S\in\mathcal{S}_{Y\mid X}\} R := { r S : S ∈ S Y ∣ X } based on the i.i.d. sample ( X i , Y i ) ∼ μ X , Y (X_i,Y_i)\sim\mu_{X,Y} ( X i , Y i ) ∼ μ X , Y .
Proof. Fix x ∈ X x \in\mathcal{X} x ∈ X . By Definition 4.2(i)–(ii), the map y ↦ S ( x , y ) y \mapsto S(x,y) y ↦ S ( x , y ) is a triangular bijection of Q d y \mathcal{Q}^{d_y} Q d y onto itself, strictly increasing in each component’s last argument, with Jacobian determinant det ∇ y S = ∏ j ∂ y j S j > 0 \det\nabla_y S = \prod_j \partial_{y_j}S_j > 0 det ∇ y S = ∏ j ∂ y j S j > 0 . The change of variables u = S ( x , y ) u = S(x,y) u = S ( x , y ) gives
∫ Q d y q S ( y ∣ x ) d y = ∫ Q d y η Y ( S ( x , y ) ) det ∇ y S d y = ∫ Q d y η Y ( u ) d u = 1. \begin{aligned}
\int_{\mathcal{Q}^{d_y}} q_S(y \mid x)\,dy
&= \int_{\mathcal{Q}^{d_y}} \eta_Y(S(x,y))\det\nabla_y S\,dy \\
&= \int_{\mathcal{Q}^{d_y}} \eta_Y(u)\,du = 1.
\end{aligned} ∫ Q d y q S ( y ∣ x ) d y = ∫ Q d y η Y ( S ( x , y )) det ∇ y S d y = ∫ Q d y η Y ( u ) d u = 1.
Consequently, r S ≥ 0 r_S \ge0 r S ≥ 0 and
∫ Q d r S d λ = ∫ X [ ∫ Y q S ( y ∣ x ) d y ] μ X ( d x ) = ∫ X 1 μ X ( d x ) = 1. \begin{aligned}
\int_{\mathcal{Q}^d} r_S\,d\lambda
&= \int_{\mathcal{X}}\left[\int_{\mathcal{Y}}q_S(y\mid x)\,dy\right]\mu_X(dx) \\
&= \int_{\mathcal{X}}1\,\mu_X(dx)=1.
\end{aligned} ∫ Q d r S d λ = ∫ X [ ∫ Y q S ( y ∣ x ) d y ] μ X ( d x ) = ∫ X 1 μ X ( d x ) = 1.
Normalization holds for every probability measure μ X \mu_X μ X because the inner integral equals one pointwise in x x x . For any Borel A ⊆ Q d A \subseteq\mathcal{Q}^d A ⊆ Q d , the disintegration (2) gives
μ X , Y ( A ) = ∫ X ∫ Y 1 A ( x , y ) π Y ∣ X ( y ∣ x ) d y μ X ( d x ) = ∫ A π Y ∣ X ( y ∣ x ) λ ( d x , d y ) . \begin{aligned}
\mu_{X,Y}(A)
&= \int_{\mathcal{X}}\int_{\mathcal{Y}}\mathbf{1}_A(x,y)\pi_{Y\mid X}(y\mid x)\,dy\,\mu_X(dx) \\
&= \int_A \pi_{Y\mid X}(y\mid x)\,\lambda(dx,dy).
\end{aligned} μ X , Y ( A ) = ∫ X ∫ Y 1 A ( x , y ) π Y ∣ X ( y ∣ x ) d y μ X ( d x ) = ∫ A π Y ∣ X ( y ∣ x ) λ ( d x , d y ) .
Thus μ X , Y ≪ λ \mu_{X,Y} \ll\lambda μ X , Y ≪ λ with density r 0 = π Y ∣ X r_0=\pi_{Y\mid X} r 0 = π Y ∣ X , bounded in [ c , B ] [c,B] [ c , B ] by Assumption 4.1. Finally, log r S ( X i , Y i ) = log q S ( Y i ∣ X i ) \log r_S(X_i,Y_i)=\log q_S(Y_i\mid X_i) log r S ( X i , Y i ) = log q S ( Y i ∣ X i ) , so the two optimization problems have identical objectives.
Remark B.2 (The role of μ X \mu_X μ X ). The relevant representation is with respect to μ X ⊗ d y \mu_X\otimes dy μ X ⊗ d y , not Lebesgue measure on Q d \mathcal{Q}^d Q d . Although q S ( y ∣ x ) q_S(y\mid x) q S ( y ∣ x ) also integrates to one against d x d y dx\,dy d x d y on the unit cube, that density generally does not describe the observed joint law. With the correct dominating measure, the likelihood and its population risk match the conditional estimation problem. The measure μ X \mu_X μ X is never estimated, and the entropy bound in Lemma B.7 is uniform over all probability measures μ X \mu_X μ X .
Under this representation, the excess population risk is exactly the conditional KL divergence.
Lemma B.3 (Exact population identity). Let L ( S ) : = − E μ X , Y [ log q S ( Y ∣ X ) ] \mathcal{L}(S):=-\mathbb{E}_{\mu_{X,Y}}[\log q_S(Y\mid X)] L ( S ) := − E μ X , Y [ log q S ( Y ∣ X )] denote the population risk, and let S 0 ∈ S Y ∣ X S_0\in\mathcal{S}_{Y\mid X} S 0 ∈ S Y ∣ X satisfy q S 0 = π Y ∣ X q_{S_0}=\pi_{Y\mid X} q S 0 = π Y ∣ X . Then, for every S ∈ S Y ∣ X S\in\mathcal{S}_{Y\mid X} S ∈ S Y ∣ X ,
L ( S ) − L ( S 0 ) = E X ∼ μ X [ D K L ( π Y ∣ X ( ⋅ ∣ X ) ∥ q S ( ⋅ ∣ X ) ) ] = D K L ( r 0 ∥ r S ) λ , (29) \begin{aligned}
\mathcal{L}(S)-\mathcal{L}(S_0)
&= \mathbb{E}_{X\sim\mu_X}\left[D_{KL}\left(\pi_{Y\mid X}(\cdot\mid X)\,\middle\|\,q_S(\cdot\mid X)\right)\right] \\
&= D_{KL}(r_0\|r_S)_\lambda, \tag*{(29)}
\end{aligned} L ( S ) − L ( S 0 ) = E X ∼ μ X [ D K L ( π Y ∣ X ( ⋅ ∣ X ) q S ( ⋅ ∣ X ) ) ] = D K L ( r 0 ∥ r S ) λ , (29) where the last expression denotes KL divergence between the λ \lambda λ -densities r 0 r_0 r 0 and r S r_S r S .
Proof. Using (2) ,
L ( S ) − L ( S 0 ) = E μ X , Y [ log π Y ∣ X ( Y ∣ X ) q S ( Y ∣ X ) ] = ∫ X [ ∫ Y π Y ∣ X ( y ∣ x ) log π Y ∣ X ( y ∣ x ) q S ( y ∣ x ) d y ] μ X ( d x ) . \mathcal{L}(S)-\mathcal{L}(S_0)
= \mathbb{E}_{\mu_X,Y}\left[\log\frac{\pi_{Y\mid X}(Y\mid X)}{q_S(Y\mid X)}\right]
= \int_{\mathcal{X}}\left[\int_{\mathcal{Y}}\pi_{Y\mid X}(y\mid x)\log\frac{\pi_{Y\mid X}(y\mid x)}{q_S(y\mid x)}\,dy\right]\mu_X(dx). L ( S ) − L ( S 0 ) = E μ X , Y [ log q S ( Y ∣ X ) π Y ∣ X ( Y ∣ X ) ] = ∫ X [ ∫ Y π Y ∣ X ( y ∣ x ) log q S ( y ∣ x ) π Y ∣ X ( y ∣ x ) d y ] μ X ( d x ) .
The inner integral is the conditional KL divergence. By Lemma B.1 , the same double integral equals D K L ( r 0 ∥ r S ) λ D_{\mathrm{KL}}(r_0\Vert r_S)_\lambda D KL ( r 0 ∥ r S ) λ . All quantities are finite because both densities are bounded above and away from zero (Assumption 4.1 and Lemma B.6).
Thus conditional likelihood targets exactly the KL term entering the plug-in bias.
Proof of Theorem 4.3
Existence of the estimator. We first establish the existence of a measurable minimizer S ^ Y ∣ X \widehat{S}_{Y\mid X} S Y ∣ X . The component functions and their diagonal derivatives form uniformly bounded, equicontinuous families. The Hölder and monotonicity constraints are closed under their joint uniform convergence, and the log-likelihood is continuous because the induced densities are uniformly bounded away from zero. Compactness therefore gives a minimizer, and the measurable maximum theorem supplies a measurable selection. This is the same compactness argument used for the unpenalized Hölder-class MLE in Wang and Marzouk [22 ] .
Lemmas B.1 and B.3 identify the conditional likelihood with an ordinary density likelihood and its excess population risk with the conditional KL divergence. It remains to establish realizability and entropy bounds, then convert Hellinger concentration into conditional KL control. We begin by verifying that the true conditional density is represented by a map in the prescribed class.
Lemma B.4 (Realizability). Under Assumptions 4.1 and 4.2, there exists S 0 ∈ S Y ∣ X ( α , L , c m o n ) S_0 \in S_{Y\mid X}(\alpha,L,c_{\mathrm{mon}}) S 0 ∈ S Y ∣ X ( α , L , c mon ) with q S 0 ( ⋅ ∣ x ) = π Y ∣ X ( ⋅ ∣ x ) q_{S_0}(\cdot\mid x)=\pi_{Y\mid X}(\cdot\mid x) q S 0 ( ⋅ ∣ x ) = π Y ∣ X ( ⋅ ∣ x ) for every x x x , where
L = L ( α , d , c , B , L 0 , L η , c η , B η ) < ∞ , c m o n = c m o n ( c , B , c η , B η ) > 0. L=L(\alpha,d,c,B,L_0,L_\eta,c_\eta,B_\eta)<\infty,\qquad c_{\mathrm{mon}}=c_{\mathrm{mon}}(c,B,c_\eta,B_\eta)>0. L = L ( α , d , c , B , L 0 , L η , c η , B η ) < ∞ , c mon = c mon ( c , B , c η , B η ) > 0.
Proof. The construction is the Knothe–Rosenblatt rearrangement applied to π Y ∣ X ( ⋅ ∣ x ) \pi_{Y\mid X}(\cdot\mid x) π Y ∣ X ( ⋅ ∣ x ) with x x x carried along as a parameter. Define the partial marginals of the conditional density, for j = 0 , 1 , … , d y j=0,1,\ldots,d_y j = 0 , 1 , … , d y ,
p ˉ j ( x , y 1 : j ) : = ∫ Q d y − j π Y ∣ X ( y 1 : j , u ∣ x ) d u , (30) \bar{p}_j(x,y_{1:j}):=\int_{\mathcal{Q}^{d_y-j}}\pi_{Y\mid X}(y_{1:j},u\mid x)\,du, \tag*{(30)} p ˉ j ( x , y 1 : j ) := ∫ Q d y − j π Y ∣ X ( y 1 : j , u ∣ x ) d u , (30)
with the conventions p ˉ d y = π Y ∣ X \bar{p}_{d_y}=\pi_{Y\mid X} p ˉ d y = π Y ∣ X and p ˉ 0 ≡ 1 \bar{p}_0\equiv1 p ˉ 0 ≡ 1 , and set
p j ( y j ∣ x , y < j ) : = p ˉ j ( x , y 1 : j ) p ˉ j − 1 ( x , y 1 : j − 1 ) , j = 1 , … , d y . (31) p_j(y_j\mid x,y_{<j}):=\frac{\bar{p}_j(x,y_{1:j})}{\bar{p}_{j-1}(x,y_{1:j-1})},\qquad j=1,\ldots,d_y. \tag*{(31)} p j ( y j ∣ x , y < j ) := p ˉ j − 1 ( x , y 1 : j − 1 ) p ˉ j ( x , y 1 : j ) , j = 1 , … , d y . (31)
(a) Bounds. Since π Y ∣ X ∈ [ c , B ] \pi_{Y\mid X}\in[c,B] π Y ∣ X ∈ [ c , B ] and the domain of integration in (30) has unit volume, c ≤ p ˉ j ≤ B c\leq\bar{p}_j\leq B c ≤ p ˉ j ≤ B for every j j j , and hence
0 < c B ≤ p j ( y j ∣ x , y < j ) ≤ B c < ∞ . (32) 0<\frac{c}{B}\leq p_j(y_j\mid x,y_{<j})\leq\frac{B}{c}<\infty. \tag*{(32)} 0 < B c ≤ p j ( y j ∣ x , y < j ) ≤ c B < ∞. (32)
(b) Regularity. Integration over a subset of coordinates does not increase Hölder norms: for any multi-index β \beta β in the retained variables ( x , y 1 : j ) (x,y_{1:j}) ( x , y 1 : j ) with ∣ β ∣ ≤ m α |\beta| \le m_\alpha ∣ β ∣ ≤ m α , differentiation under the integral sign (justified by continuity of the integrand and compactness of the domain) gives D β p ˉ j = ∫ D β π Y ∣ X d u D^\beta\bar{p}_j = \int D^\beta\pi_{Y\mid X}\,du D β p ˉ j = ∫ D β π Y ∣ X d u , whence ∥ D β p ˉ j ∥ ∞ ≤ ∥ D β π Y ∣ X ∥ ∞ \|D^\beta\bar{p}_j\|_\infty\le\|D^\beta\pi_{Y\mid X}\|_\infty ∥ D β p ˉ j ∥ ∞ ≤ ∥ D β π Y ∣ X ∥ ∞ and the β α \beta_\alpha β α -Hölder seminorms of the top-order derivatives obey the same inequality. Thus ∥ p ˉ j ∥ H α ≤ L 0 \|\bar{p}_j\|_{\mathcal{H}^\alpha} \le L_0 ∥ p ˉ j ∥ H α ≤ L 0 . Since p ˉ j − 1 ≥ c > 0 \bar{p}_{j-1} \ge c > 0 p ˉ j − 1 ≥ c > 0 and H α ( Ω ) \mathcal{H}^\alpha(\Omega) H α ( Ω ) is an algebra on a bounded convex domain that is stable under division by elements bounded away from zero, (31) gives ∥ p j ∥ H α ≤ C a ( α , d , c , L 0 ) \|p_j\|_{\mathcal{H}^\alpha} \le C_a(\alpha,d,c,L_0) ∥ p j ∥ H α ≤ C a ( α , d , c , L 0 ) .
(c) The map. Define the conditional KR map by
F j ( t ; x , y < j ) : = ∫ 0 t p j ( u ∣ x , y < j ) d u , S 0 , j ( x , y 1 : j ) : = G j − 1 ( F j ( y j ; x , y < j ) ) . (33) F_j(t;x,y_{<j}) := \int_0^t p_j(u\mid x,y_{<j})\,du,\qquad S_{0,j}(x,y_{1:j}) := G_j^{-1}\bigl(F_j(y_j;x,y_{<j})\bigr). \tag*{(33)} F j ( t ; x , y < j ) := ∫ 0 t p j ( u ∣ x , y < j ) d u , S 0 , j ( x , y 1 : j ) := G j − 1 ( F j ( y j ; x , y < j ) ) . (33)
By (32) , t ↦ F j ( t ; ⋅ ) t \mapsto F_j(t;\cdot) t ↦ F j ( t ; ⋅ ) is a strictly increasing bijection of [ 0 , 1 ] [0,1] [ 0 , 1 ] onto [ 0 , 1 ] [0,1] [ 0 , 1 ] with ∂ t F j ∈ [ c / B , B / c ] \partial_tF_j \in[c/B,B/c] ∂ t F j ∈ [ c / B , B / c ] . By Assumption 4.2, G j G_j G j is a strictly increasing bijection of [ 0 , 1 ] [0,1] [ 0 , 1 ] onto [ 0 , 1 ] [0,1] [ 0 , 1 ] with G j ′ = η Y , j ∈ [ c η , B η ] G_j'=\eta_{Y,j}\in[c_\eta,B_\eta] G j ′ = η Y , j ∈ [ c η , B η ] , so G j − 1 G_j^{-1} G j − 1 is a bijection of [ 0 , 1 ] [0,1] [ 0 , 1 ] with ( G j − 1 ) ′ = 1 / η Y , j ( G j − 1 ) ∈ [ B η − 1 , c η − 1 ] (G_j^{-1})'=1/\eta_{Y,j}(G_j^{-1})\in[B_\eta^{-1},c_\eta^{-1}] ( G j − 1 ) ′ = 1/ η Y , j ( G j − 1 ) ∈ [ B η − 1 , c η − 1 ] and ∥ G j − 1 ∥ H α + 1 ( [ 0 , 1 ] ) ≤ C b ( α , c η , B η , ∥ η Y , j ∥ H α ) \|G_j^{-1}\|_{\mathcal{H}^{\alpha+1}([0,1])}\le C_b(\alpha,c_\eta,B_\eta,\|\eta_{Y,j}\|_{\mathcal{H}^\alpha}) ∥ G j − 1 ∥ H α + 1 ([ 0 , 1 ]) ≤ C b ( α , c η , B η , ∥ η Y , j ∥ H α ) . It is here that Assumption 4.2 is used in an essential way. Hence S 0 , j S_{0,j} S 0 , j maps [ 0 , 1 ] [0,1] [ 0 , 1 ] onto [ 0 , 1 ] [0,1] [ 0 , 1 ] in its last argument and is an increasing bijection, giving Definition 4.2(i).
(d) Diagonal derivative. By the chain rule and ( G j − 1 ) ′ ( s ) = 1 / η Y , j ( G j − 1 ( s ) ) (G_j^{-1})'(s)=1/\eta_{Y,j}(G_j^{-1}(s)) ( G j − 1 ) ′ ( s ) = 1/ η Y , j ( G j − 1 ( s )) ,
∂ y j S 0 , j ( x , y 1 : j ) = p j ( y j ∣ x , y < j ) η Y , j ( S 0 , j ( x , y 1 : j ) ) ∈ [ c B B η , B c c η ] . (34) \partial_{y_j}S_{0,j}(x,y_{1:j})=\frac{p_j(y_j\mid x,y_{<j})}{\eta_{Y,j}(S_{0,j}(x,y_{1:j}))}\in\left[\frac{c}{BB_\eta},\frac{B}{cc_\eta}\right]. \tag*{(34)} ∂ y j S 0 , j ( x , y 1 : j ) = η Y , j ( S 0 , j ( x , y 1 : j )) p j ( y j ∣ x , y < j ) ∈ [ B B η c , c c η B ] . (34)
which gives Definition 4.2(ii) with c m o n = c / ( B B η ) c_{\mathrm{mon}}=c/(BB_\eta) c mon = c / ( B B η ) . For the regularity in (iii): F j ∈ H α F_j\in\mathcal{H}^\alpha F j ∈ H α jointly (indeed H α + 1 \mathcal{H}^{\alpha+1} H α + 1 in y j y_j y j ) by the regularity of p j p_j p j and integration; composition with the H α + 1 \mathcal{H}^{\alpha+1} H α + 1 , Lipschitz function G j − 1 G_j^{-1} G j − 1 preserves H α \mathcal{H}^\alpha H α ; and (34) exhibits ∂ y j S 0 , j \partial_{y_j}S_{0,j} ∂ y j S 0 , j as a quotient of an H α \mathcal{H}^\alpha H α function by η Y , j ∘ S 0 , j ∈ H α \eta_{Y,j}\circ S_{0,j}\in\mathcal{H}^\alpha η Y , j ∘ S 0 , j ∈ H α bounded below by c η c_\eta c η , hence itself H α \mathcal{H}^\alpha H α . Collecting the constants yields S 0 ∈ S Y ∣ X ( α , L , c m o n ) S_0\in\mathcal{S}_{Y\mid X}(\alpha,L,c_{\mathrm{mon}}) S 0 ∈ S Y ∣ X ( α , L , c mon ) for L L L as stated.
(e) Correctness. Finally, by construction and telescoping,
q S 0 ( y ∣ x ) = ∏ j = 1 d y η Y , j ( S 0 , j ) ⋅ ∂ y j S 0 , j = ∏ j = 1 d y p j ( y j ∣ x , y < j ) = ∏ j = 1 d y p ˉ j p ˉ j − 1 = p ˉ d y ( x , y ) = π Y ∣ X ( y ∣ x ) . \begin{aligned}
q_{S_0}(y\mid x)=\prod_{j=1}^{d_y}\eta_{Y,j}(S_{0,j})\cdot\partial_{y_j}S_{0,j}
=\prod_{j=1}^{d_y}p_j(y_j\mid x,y_{<j}) \\
&=\prod_{j=1}^{d_y}\frac{\bar{p}_j}{\bar{p}_{j-1}}=\bar{p}_{d_y}(x,y)=\pi_{Y\mid X}(y\mid x).
\end{aligned} q S 0 ( y ∣ x ) = j = 1 ∏ d y η Y , j ( S 0 , j ) ⋅ ∂ y j S 0 , j = j = 1 ∏ d y p j ( y j ∣ x , y < j ) = j = 1 ∏ d y p ˉ j − 1 p ˉ j = p ˉ d y ( x , y ) = π Y ∣ X ( y ∣ x ) .
Remark B.5. Wang and Marzouk [22 ] is the conditional analogue of Lemma B.4: positivity and Hölder smoothness of the target density transfer to the KR map through partial integration, division, and composition with the reference CDF. The variable x x x is a passive parameter throughout, so the construction imposes no regularity on its law.
To control the complexity of the fitted-density class, we also need uniform bounds for every candidate map.
Lemma B.6 (Uniform regularity of the induced class). Under Assumption 4.2, every q S ∈ Q Y ∣ X q_S\in\mathcal{Q}_{Y\mid X} q S ∈ Q Y ∣ X satisfies
0 < c ‾ : = c η d y c m o n d y ≤ q S ( y ∣ x ) ≤ B η d y L d y = : B ‾ < ∞ , (35) 0<\underline{c}:=c_\eta^{d_y}c_{\mathrm{mon}}^{d_y}\le q_S(y\mid x)\le B_\eta^{d_y}L^{d_y}=:\overline{B}<\infty, \tag*{(35)} 0 < c := c η d y c mon d y ≤ q S ( y ∣ x ) ≤ B η d y L d y =: B < ∞ , (35)
and ∥ q S ∥ H α ( Q d ) ≤ C q \|q_S\|_{\mathcal{H}^\alpha(\mathcal{Q}^d)}\le C_q ∥ q S ∥ H α ( Q d ) ≤ C q for a constant C q = C q ( α , d , L , c m o n , c η , B η , ∥ η Y ∥ H α ) C_q=C_q(\alpha,d,L,c_{\mathrm{mon}},c_\eta,B_\eta,\|\eta_Y\|_{\mathcal{H}^\alpha}) C q = C q ( α , d , L , c mon , c η , B η , ∥ η Y ∥ H α ) . Consequently
Q Y ∣ X ⊆ G : = { g ∈ H α ( Q d ) : ∥ g ∥ H α ≤ C q , c ‾ ≤ g ≤ B ‾ } . (36) \mathcal{Q}_{Y\mid X}\subseteq\mathcal{G}:=\left\{g\in\mathcal{H}^\alpha(\mathcal{Q}^d):\|g\|_{\mathcal{H}^\alpha}\le C_q,\ \underline{c}\le g\le\overline{B}\right\}. \tag*{(36)} Q Y ∣ X ⊆ G := { g ∈ H α ( Q d ) : ∥ g ∥ H α ≤ C q , c ≤ g ≤ B } . (36)
Proof. The two-sided bound (35) is immediate from (6) : the product structure in Assumption 4.2 gives η Y ( S ( x , y ) ) ∈ [ c η d y , B η d y ] \eta_Y(S(x,y)) \in[c_{\eta}^{d_y}, B_{\eta}^{d_y}] η Y ( S ( x , y )) ∈ [ c η d y , B η d y ] , while each of the d y d_y d y diagonal factors lies in [ c m o n , L ] [c_{\mathrm{mon}},L] [ c mon , L ] by Definition 4.2(ii). For the Hölder bound: S ∈ H α S \in\mathcal{H}^{\alpha} S ∈ H α with values in the compact set Q d y Q^{d_y} Q d y and is Lipschitz (its entries have bounded first derivatives), so η Y ∘ S ∈ H α \eta_Y \circ S \in\mathcal{H}^{\alpha} η Y ∘ S ∈ H α with norm controlled by ∥ η Y ∥ H α \|\eta_Y\|_{\mathcal{H}^{\alpha}} ∥ η Y ∥ H α and ∥ S ∥ H α ≤ L d y \|S\|_{\mathcal{H}^{\alpha}} \le L\sqrt{d_y} ∥ S ∥ H α ≤ L d y ; each factor ∂ y j S j ∈ H α \partial_{y_j}S_j \in\mathcal{H}^{\alpha} ∂ y j S j ∈ H α with norm ≤ L \le L ≤ L by Definition 4.2(iii); and H α ( Q d ) \mathcal{H}^{\alpha}(Q^d) H α ( Q d ) is a Banach algebra on the bounded convex domain Q d Q^d Q d , so the product of the d y + 1 d_y+1 d y + 1 factors lies in H α \mathcal{H}^{\alpha} H α with norm bounded by a constant depending only on the listed quantities.
Lemma B.6 is the conditional counterpart of Wang and Marzouk [22 ] . Its role is to reduce the entropy computation for the map class S Y ∣ X \mathcal{S}_{Y\mid X} S Y ∣ X to an entropy computation for a bounded Hölder ball of densities , which is classical.
We use this uniform regularity to bound the bracketing entropy. For a class F \mathcal{F} F of functions and a (semi)norm ∥ ⋅ ∥ \|\cdot\| ∥ ⋅ ∥ , let N [ ] ( ρ , F , ∥ ⋅ ∥ ) N_{[]}(\rho,\mathcal{F},\|\cdot\|) N [ ] ( ρ , F , ∥ ⋅ ∥ ) denote the ρ \rho ρ -bracketing number and H [ ] = log N [ ] H_{[]}=\log N_{[]} H [ ] = log N [ ] the bracketing entropy.
Lemma B.7 (Bracketing entropy of the induced class). Let G \mathcal{G} G be as in (36) and let λ = μ X ⊗ d y \lambda=\mu_X\otimes dy λ = μ X ⊗ d y . Then
H [ ] ( ρ , G , L 2 ( λ ) ) ≤ H [ ] ( ρ , G , ∥ ⋅ ∥ ∞ ) ≤ A ρ − d / α , 0 < ρ ≤ 1 , (37) H_{[]}(\rho,\mathcal{G},L^2(\lambda)) \le H_{[]}(\rho,\mathcal{G},\|\cdot\|_\infty) \le A\rho^{-d/\alpha}, \qquad0<\rho\le1, \tag*{(37)} H [ ] ( ρ , G , L 2 ( λ )) ≤ H [ ] ( ρ , G , ∥ ⋅ ∥ ∞ ) ≤ A ρ − d / α , 0 < ρ ≤ 1 , (37) where A = A ( α , d , C q ) A=A(\alpha,d,C_q) A = A ( α , d , C q ) does not depend on μ X \mu_X μ X . The same bound, with a different constant, holds for the midpoint square-root class
A ˉ : = { g + r 0 2 : g ∈ G } , (38) \bar{\mathcal{A}}:=\left\{\sqrt{\frac{g+r_0}{2}}:g\in\mathcal{G}\right\}, \tag*{(38)} A ˉ := { 2 g + r 0 : g ∈ G } , (38) and there exists a constant A 1 > 0 A_1>0 A 1 > 0 , independent of μ X \mu_X μ X , such that
H [ ] ( ρ , A ˉ , L 2 ( λ ) ) ≤ A 1 ρ − d / α , 0 < ρ ≤ 1 , (39) H_{[]}(\rho,\bar{\mathcal{A}},L^2(\lambda))\le A_1\rho^{-d/\alpha}, \qquad0<\rho\le1, \tag*{(39)} H [ ] ( ρ , A ˉ , L 2 ( λ )) ≤ A 1 ρ − d / α , 0 < ρ ≤ 1 , (39)
Proof. (a) Sup-norm entropy. By the classical entropy estimate of Kolmogorov–Tikhomirov for Hölder balls on a bounded domain (see also van der Vaart and Wellner, [21 ] ), the ball { g : ∥ g ∥ H α ( Q d ) ≤ C q } \{g:\|g\|_{\mathcal{H}^{\alpha}(Q^d)}\le C_q\} { g : ∥ g ∥ H α ( Q d ) ≤ C q } satisfies log N ( ρ , ⋅ , ∥ ⋅ ∥ ∞ ) ≤ A 0 ρ − d / α \log N(\rho,\cdot,\|\cdot\|_\infty)\le A_0\rho^{-d/\alpha} log N ( ρ , ⋅ , ∥ ⋅ ∥ ∞ ) ≤ A 0 ρ − d / α . A ρ / 2 \rho/2 ρ /2 -net in sup norm induces ρ \rho ρ -brackets [ g k − ρ / 2 , g k + ρ / 2 ] [g_k-\rho/2,g_k+\rho/2] [ g k − ρ /2 , g k + ρ /2 ] , so H [ ] ( ρ , G , ∥ ⋅ ∥ ∞ ) ≤ A 0 ( ρ / 2 ) − d / α H_{[]}(\rho,\mathcal{G},\|\cdot\|_\infty)\le A_0(\rho/2)^{-d/\alpha} H [ ] ( ρ , G , ∥ ⋅ ∥ ∞ ) ≤ A 0 ( ρ /2 ) − d / α .
(b) Transfer to L 2 ( λ ) L^2(\lambda) L 2 ( λ ) , uniformly in μ X \mu_X μ X . The sup-norm bound transfers to L 2 ( λ ) L^2(\lambda) L 2 ( λ ) uniformly in μ X \mu_X μ X : for any measurable f , g f,g f , g on Q d Q^d Q d ,
∥ f − g ∥ L 2 ( λ ) 2 = ∫ X μ X ( d x ) ∫ Q d y ∣ f ( x , y ) − g ( x , y ) ∣ 2 d y ≤ ∣ Q d y ∣ ⋅ ∥ f − g ∥ ∞ 2 ⋅ μ X ( X ) = ∥ f − g ∥ ∞ 2 , \|f-g\|_{L^2(\lambda)}^2=\int_{\mathcal{X}}\mu_X(dx)\int_{Q^{d_y}}|f(x,y)-g(x,y)|^2\,dy\le|Q^{d_y}|\cdot\|f-g\|_\infty^2\cdot\mu_X(\mathcal{X})=\|f-g\|_\infty^2, ∥ f − g ∥ L 2 ( λ ) 2 = ∫ X μ X ( d x ) ∫ Q d y ∣ f ( x , y ) − g ( x , y ) ∣ 2 d y ≤ ∣ Q d y ∣ ⋅ ∥ f − g ∥ ∞ 2 ⋅ μ X ( X ) = ∥ f − g ∥ ∞ 2 ,
using ∣ Q d y ∣ = 1 |Q^{d_y}|=1 ∣ Q d y ∣ = 1 and μ X ( X ) = 1 \mu_X(\mathcal{X})=1 μ X ( X ) = 1 . Hence any ρ \rho ρ -bracket in sup norm is a ρ \rho ρ -bracket in L 2 ( λ ) L^2(\lambda) L 2 ( λ ) , and (37) follows. Crucially, the only property of μ X \mu_X μ X used is that it is a probability measure: no density, smoothness, or positivity enters. This is the precise sense in which the direct conditional argument is free of assumptions on the law of X X X .
(c) Midpoint square roots. By Lemma B.6 and Assumption 4.1, every ( g + r 0 ) / 2 (g+r_0)/2 ( g + r 0 ) /2 with g ∈ G g\in\mathcal{G} g ∈ G takes values in the compact interval [ ( c ‾ + c ) / 2 , ( B ‾ + B ) / 2 ] ⊂ ( 0 , ∞ ) [(\underline{c}+c)/2,(\overline{B}+B)/2]\subset(0,\infty) [( c + c ) /2 , ( B + B ) /2 ] ⊂ ( 0 , ∞ ) and belongs to a uniformly bounded Hölder ball. Composition with z ↦ z z\mapsto\sqrt{z} z ↦ z therefore places A ˉ \bar{\mathcal{A}} A ˉ in a Hölder ball of radius depending only on ( C q , L 0 , c ‾ , B ‾ , c , B , α , d ) (C_q,L_0,\underline{c},\overline{B},c,B,\alpha,d) ( C q , L 0 , c , B , c , B , α , d ) . Moreover,
∥ g 1 + r 0 2 − g 2 + r 0 2 ∥ ∞ ≤ 1 4 c ‾ c ‾ ∥ g 1 − g 2 ∥ ∞ , \left\|\sqrt{\frac{g_1+r_0}{2}}-\sqrt{\frac{g_2+r_0}{2}}\right\|_\infty\le\frac{1}{4\sqrt{\underline{c}\,\underline{c}}}\|g_1-g_2\|_\infty, 2 g 1 + r 0 − 2 g 2 + r 0 ∞ ≤ 4 c c 1 ∥ g 1 − g 2 ∥ ∞ ,
so the same sup-norm entropy estimate and the preceding norm comparison give (39) .
The entropy bound determines a critical radius for Hellinger concentration of the MLE.
Lemma B.8 (Entropy integral). For δ > 0 \delta>0 δ > 0 , define the global midpoint square-root class and its entropy integral by
A : = { r + r 0 2 : r ∈ R } , (40) \mathcal{A}:=\left\{\sqrt{\frac{r+r_0}{2}}:r\in\mathcal{R}\right\}, \tag*{(40)} A := { 2 r + r 0 : r ∈ R } , (40) J ( δ ) : = δ + ∫ 0 δ H [ ] ( u , A , L 2 ( λ ) ) d u . (41) J(\delta):=\delta+\int_0^\delta\sqrt{H_{[]}(u,\mathcal{A},L^2(\lambda))}\,\mathrm{d}u. \tag*{(41)} J ( δ ) := δ + ∫ 0 δ H [ ] ( u , A , L 2 ( λ )) d u . (41) If α > d / 2 \alpha>d/2 α > d /2 , then J ( δ ) < ∞ J(\delta)<\infty J ( δ ) < ∞ for all 0 < δ ≤ 1 0<\delta\le1 0 < δ ≤ 1 and
J ( δ ) ≤ δ + A 1 1 − d / ( 2 α ) δ 1 − d / ( 2 α ) ≤ C e n t δ 1 − d / ( 2 α ) , (42) J(\delta)\le\delta+\frac{\sqrt{A_1}}{1-d/(2\alpha)}\delta^{1-d/(2\alpha)}\le C_{\mathrm{ent}}\delta^{1-d/(2\alpha)}, \tag*{(42)} J ( δ ) ≤ δ + 1 − d / ( 2 α ) A 1 δ 1 − d / ( 2 α ) ≤ C ent δ 1 − d / ( 2 α ) , (42) where C e n t = 1 + A 1 / ( 1 − d / ( 2 α ) ) C_{\mathrm{ent}}=1+\sqrt{A_1}/(1-d/(2\alpha)) C ent = 1 + A 1 / ( 1 − d / ( 2 α )) is independent of μ X \mu_X μ X .
Proof. Since R ⊆ G \mathcal{R}\subseteq\mathcal{G} R ⊆ G , we have A ⊆ A ‾ \mathcal{A}\subseteq\overline{\mathcal{A}} A ⊆ A . Lemma B.7 therefore gives
H [ ] ( u , A , L 2 ( λ ) ) ≤ A 1 u − d / ( 2 α ) . \sqrt{H_{[]}(u,\mathcal{A},L^2(\lambda))}\le\sqrt{A_1}\,u^{-d/(2\alpha)}. H [ ] ( u , A , L 2 ( λ )) ≤ A 1 u − d / ( 2 α ) . The right-hand side is integrable at the origin when α > d / 2 \alpha>d/2 α > d /2 . Direct integration and δ ≤ δ 1 − d / ( 2 α ) \delta\le\delta^{1-d/(2\alpha)} δ ≤ δ 1 − d / ( 2 α ) for 0 < δ ≤ 1 0<\delta\le1 0 < δ ≤ 1 give (42) .
The condition α > d / 2 \alpha>d/2 α > d /2 is sufficient for this global Hölder-entropy argument. When α ≤ d / 2 \alpha\le d/2 α ≤ d /2 , the displayed entropy upper bound is not integrable at the origin, so this proof does not yield the stated rate; this observation alone does not imply that the actual entropy integral diverges or that no alternative argument is possible.
Proposition B.9 (MLE concentration at the critical radius). Let λ \lambda λ be a σ \sigma σ -finite measure on a standard Borel space Z \mathcal{Z} Z , let R \mathcal{R} R be a class of λ \lambda λ -densities containing the true density r 0 > 0 r_0>0 r 0 > 0 λ \lambda λ -almost everywhere, and let r ^ \hat r r ^ be a measurable MLE based on N N N i.i.d. observations. Set h λ ( r , r ′ ) = ( 1 2 ∫ ( r − r ′ ) 2 d λ ) 1 / 2 h_\lambda(r,r')=\left(\frac{1}{2}\int(\sqrt r-\sqrt{r'})^2\,\mathrm{d}\lambda\right)^{1/2} h λ ( r , r ′ ) = ( 2 1 ∫ ( r − r ′ ) 2 d λ ) 1/2 and define J J J from the global class in (40) –(41) . If δ N ∈ ( 0 , 1 ] \delta_N\in(0,1] δ N ∈ ( 0 , 1 ] satisfies
N δ N 2 ≥ c 0 J ( δ N ) , (43) \sqrt{N}\,\delta_N^2\ge c_0J(\delta_N), \tag*{(43)} N δ N 2 ≥ c 0 J ( δ N ) , (43) for a suitable absolute constant c 0 c_0 c 0 , then there exist absolute constants c 1 , c 2 > 0 c_1,c_2>0 c 1 , c 2 > 0 such that, for every R ≥ δ N R\ge\delta_N R ≥ δ N ,
Pr ( h λ ( r ^ , r 0 ) > R ) ≤ c 1 exp ( − c 2 N R 2 ) . (44) \Pr\left(h_\lambda(\hat r,r_0)>R\right)\le c_1\exp(-c_2NR^2). \tag*{(44)} Pr ( h λ ( r ^ , r 0 ) > R ) ≤ c 1 exp ( − c 2 N R 2 ) . (44) Proof. We use a sufficient-condition consequence of ?, Theorem 7.4, as reproduced in Theorem 17 of Kim and Samworth [9 ] . That formulation uses global Hellinger brackets of the midpoint densities, equivalently L 2 L^2 L 2 brackets of their square roots, and the truncated integral
max { δ , ∫ δ 2 / 2 13 δ H [ ] ( u , A , L 2 ( λ ) ) d u } ≤ J ( δ ) . \max\left\{\delta,\int_{\delta^2/2^{13}}^\delta\sqrt{H_{[]}(u,\mathcal{A},L^2(\lambda))}\,\mathrm{d}u\right\}\le J(\delta). max { δ , ∫ δ 2 / 2 13 δ H [ ] ( u , A , L 2 ( λ )) d u } ≤ J ( δ ) . For Lebesgue densities, its conclusion, after converting its Hellinger convention to h λ h_\lambda h λ , is bounded by
2 13 / 2 ∑ s = 0 ∞ exp ( − 2 2 s N R 2 2 27 ) , R ≥ δ N , 2^{13/2}\sum_{s=0}^{\infty}\exp\left(-\frac{2^{2s}NR^2}{2^{27}}\right),\qquad R\ge\delta_N, 2 13/2 s = 0 ∑ ∞ exp ( − 2 27 2 2 s N R 2 ) , R ≥ δ N ,
when c 0 = 2 16 c_0 = 2^{16} c 0 = 2 16 suffices in (43) . The radius term in J \mathcal{J} J ensures N δ N 2 ≥ 2 32 N\delta_N^2 \ge2^{32} N δ N 2 ≥ 2 32 , so this series is at most twice its first term. This gives (44) with absolute constants.
To justify the extension from Lebesgue densities to the stated dominating measure, let P 0 = r 0 d λ P_0 = r_0\,d\lambda P 0 = r 0 d λ and choose a measurable map τ : [ 0 , 1 ] → Z \tau: [0,1] \to\mathcal{Z} τ : [ 0 , 1 ] → Z sending the uniform law to P 0 P_0 P 0 ; such a map exists because Z \mathcal{Z} Z is standard Borel. For each r ∈ R r \in\mathcal{R} r ∈ R , define the Lebesgue density
q r ( u ) : = r ( τ ( u ) ) r 0 ( τ ( u ) ) . q_r(u) := \frac{r(\tau(u))}{r_0(\tau(u))}. q r ( u ) := r 0 ( τ ( u )) r ( τ ( u )) .
Then ∫ 0 1 q r ( u ) d u = 1 \int_0^1 q_r(u)\,du = 1 ∫ 0 1 q r ( u ) d u = 1 , q r 0 = 1 q_{r_0} = 1 q r 0 = 1 , and h d u ( q r , 1 ) = h λ ( r , r 0 ) h_{du}(q_r,1) = h_\lambda(r,r_0) h d u ( q r , 1 ) = h λ ( r , r 0 ) . For independent uniform U i U_i U i , the sample τ ( U i ) \tau(U_i) τ ( U i ) has law P 0 P_0 P 0 , and
∑ i log q r ( U i ) = ∑ i log r ( τ ( U i ) ) − ∑ i log r 0 ( τ ( U i ) ) . \sum_i \log q_r(U_i) = \sum_i \log r(\tau(U_i)) - \sum_i \log r_0(\tau(U_i)). i ∑ log q r ( U i ) = i ∑ log r ( τ ( U i )) − i ∑ log r 0 ( τ ( U i )) .
The second term does not depend on r r r , so the fitted density corresponds to an MLE in the transformed class. Brackets also transfer: the order-preserving map f ↦ ( f / r 0 ) ∘ τ f \mapsto(f/\sqrt{r_0}) \circ\tau f ↦ ( f / r 0 ) ∘ τ sends the midpoint square roots to ( q r + 1 ) / 2 \sqrt{(q_r+1)/2} ( q r + 1 ) /2 and preserves bracket widths, since
∫ 0 1 ∣ f ( τ ( u ) ) − g ( τ ( u ) ) ∣ 2 r 0 ( τ ( u ) ) d u = ∫ ∣ f − g ∣ 2 d λ . \int_0^1 \frac{\lvert f(\tau(u))-g(\tau(u))\rvert^2}{r_0(\tau(u))}\,du = \int\lvert f-g\rvert^2\,d\lambda. ∫ 0 1 r 0 ( τ ( u )) ∣ f ( τ ( u )) − g ( τ ( u )) ∣ 2 d u = ∫ ∣ f − g ∣ 2 d λ .
Thus the transformed bracketing entropy is no greater than the original one, and the Lebesgue-density theorem applies. This proves the proposition without assuming a Lebesgue density for μ X \mu_X μ X .
Equivalently, one may bracket { ( r + r 0 ) / ( 2 r 0 ) : r ∈ R } \{\sqrt{(r+r_0)/(2r_0)} : r \in\mathcal{R}\} { ( r + r 0 ) / ( 2 r 0 ) : r ∈ R } in L 2 ( P 0 ) L^2(P_0) L 2 ( P 0 ) . The division by r 0 \sqrt{r_0} r 0 is essential for this isometry; raw square-root densities must instead be bracketed in L 2 ( λ ) L^2(\lambda) L 2 ( λ ) . No localization is needed for the global Hölder bound used here.
Lemma B.10 (Solving for δ N \delta_N δ N ). Under the conditions of Theorem 4.3 , there are constants D ≥ 1 D \ge1 D ≥ 1 and N 0 N_0 N 0 , independent of μ X \mu_X μ X , such that
δ N = D N − α / ( 2 α + d ) \delta_N = D N^{-\alpha/(2\alpha+d)} δ N = D N − α / ( 2 α + d ) belongs to ( 0 , 1 ] (0,1] ( 0 , 1 ] and satisfies (43) for every N ≥ N 0 N \ge N_0 N ≥ N 0 .
Proof. By (42) , it suffices that N δ N 1 + d / ( 2 α ) ≥ c 0 C E n t \sqrt{N}\,\delta_N^{1+d/(2\alpha)} \ge c_0 C_{\mathrm{Ent}} N δ N 1 + d / ( 2 α ) ≥ c 0 C Ent . Substituting the stated radius makes the left-hand side equal to D 1 + d / ( 2 α ) D^{1+d/(2\alpha)} D 1 + d / ( 2 α ) . Choose D ≥ 1 D \ge1 D ≥ 1 large enough that this is at least c 0 C E n t c_0 C_{\mathrm{Ent}} c 0 C Ent , and then choose N 0 N_0 N 0 so that δ N ≤ 1 \delta_N \le1 δ N ≤ 1 for N ≥ N 0 N \ge N_0 N ≥ N 0 . The corresponding power of N N N is
1 / 2 1 + d / ( 2 α ) = α 2 α + d . \frac{1/2}{1+d/(2\alpha)} = \frac{\alpha}{2\alpha+d}. 1 + d / ( 2 α ) 1/2 = 2 α + d α .
Combining Proposition B.9 with Lemma B.10, and writing a = 2 α / ( 2 α + d ) a = 2\alpha/(2\alpha+d) a = 2 α / ( 2 α + d ) , yields, for N ≥ N 0 N \ge N_0 N ≥ N 0 ,
Pr ( h 2 ( r ^ , r 0 ) > t 2 D 2 N − a ) ≤ c 1 exp ( − c 2 t 2 D 2 N 1 − a ) , t ≥ 1. (45) \Pr\left(h^2(\hat r,r_0) > t^2D^2N^{-a}\right) \le c_1 \exp\left(-c_2t^2D^2N^{1-a}\right), \qquad t \ge1. \tag*{(45)} Pr ( h 2 ( r ^ , r 0 ) > t 2 D 2 N − a ) ≤ c 1 exp ( − c 2 t 2 D 2 N 1 − a ) , t ≥ 1. (45)
Here D D D and N 0 N_0 N 0 depend on the fixed class parameters, while c 1 , c 2 c_1,c_2 c 1 , c 2 can be chosen absolute. If the threshold exceeds one, the event is empty.
It remains to convert Hellinger concentration into conditional KL control. Hellinger control does not in general imply KL control—KL can be infinite while Hellinger distance is small. Uniform boundedness of the likelihood ratio, which Lemma B.6 supplies, closes the gap.
Lemma B.11 (Bounded-ratio KL–Hellinger comparison). Let p , q p,q p , q be probability densities with respect to a measure λ \lambda λ such that m ≤ q / p ≤ m ˉ m \le q/p \le\bar{m} m ≤ q / p ≤ m ˉ λ \lambda λ -a.e. for constants 0 < m ≤ 1 ≤ m ˉ < ∞ 0 < m \le1 \le\bar{m} < \infty 0 < m ≤ 1 ≤ m ˉ < ∞ . Then
D K L ( p ∥ q ) ≤ 2 C m , m ˉ h λ 2 ( p , q ) , C m , m ˉ : = ( 1 + m ˉ ) 2 2 m 2 . (46) D_{\mathrm{KL}}(p\mathbin\Vert q) \le2C_{m,\bar{m}}h_{\lambda}^{2}(p,q), \qquad C_{m,\bar{m}} := \frac{(1+\sqrt{\bar{m}})^2}{2m^2}. \tag*{(46)} D KL ( p ∥ q ) ≤ 2 C m , m ˉ h λ 2 ( p , q ) , C m , m ˉ := 2 m 2 ( 1 + m ˉ ) 2 . (46)
Proof. Write t : = q / p t := q/p t := q / p and ψ ( t ) : = t − 1 − log t ≥ 0 \psi(t) := t-1-\log t \ge0 ψ ( t ) := t − 1 − log t ≥ 0 . Because p p p and q q q are both probability densities with respect to λ \lambda λ ,
∫ p ( t − 1 ) d λ = ∫ ( q − p ) d λ = 1 − 1 = 0. (47) \int p(t-1)\,d\lambda= \int(q-p)\,d\lambda= 1-1=0. \tag*{(47)} ∫ p ( t − 1 ) d λ = ∫ ( q − p ) d λ = 1 − 1 = 0. (47)
This centering identity is what makes the argument work; it lets us insert the linear term for free:
D K L ( p ∥ q ) = ∫ p log p q d λ = ∫ p ( − log t ) d λ x r e f ‘ ( 47 ) ‘ = ∫ p [ t − 1 − log t ] d λ = ∫ p ψ ( t ) d λ . D_{\mathrm{KL}}(p\mathbin\Vert q) = \int p\log\frac{p}{q}\,d\lambda= \int p(-\log t)\,d\lambda\mathrel{{xref}`(47)`=} \int p[t-1-\log t]\,d\lambda= \int p\psi(t)\,d\lambda. D KL ( p ∥ q ) = ∫ p log q p d λ = ∫ p ( − log t ) d λ x r e f ‘ ( 47 ) ‘ = ∫ p [ t − 1 − log t ] d λ = ∫ p ψ ( t ) d λ .
Next we compare ψ \psi ψ with ϕ ( t ) : = ( t − 1 ) 2 \phi(t):=(\sqrt{t}-1)^2 ϕ ( t ) := ( t − 1 ) 2 on [ m , m ˉ ] [m,\bar{m}] [ m , m ˉ ] . By Taylor’s theorem with Lagrange remainder about t = 1 t=1 t = 1 , using ψ ( 1 ) = ψ ′ ( 1 ) = 0 \psi(1)=\psi'(1)=0 ψ ( 1 ) = ψ ′ ( 1 ) = 0 and ψ ′ ′ ( s ) = s − 2 \psi''(s)=s^{-2} ψ ′′ ( s ) = s − 2 , there is ξ \xi ξ between 1 1 1 and t t t with
ψ ( t ) = ( t − 1 ) 2 2 ξ 2 ≤ ( t − 1 ) 2 2 m 2 , \psi(t)=\frac{(t-1)^2}{2\xi^2}\le\frac{(t-1)^2}{2m^2}, ψ ( t ) = 2 ξ 2 ( t − 1 ) 2 ≤ 2 m 2 ( t − 1 ) 2 ,
since ξ ≥ min ( 1 , t ) ≥ m \xi\ge\min(1,t)\ge m ξ ≥ min ( 1 , t ) ≥ m . Factoring ( t − 1 ) 2 = ( t − 1 ) 2 ( t + 1 ) 2 (t-1)^2=(\sqrt{t}-1)^2(\sqrt{t}+1)^2 ( t − 1 ) 2 = ( t − 1 ) 2 ( t + 1 ) 2 and using t ≤ m ˉ t\le\bar{m} t ≤ m ˉ ,
ψ ( t ) ≤ ( t + 1 ) 2 2 m 2 ϕ ( t ) ≤ ( 1 + m ˉ ) 2 2 m 2 ϕ ( t ) = C m , m ˉ ϕ ( t ) . \psi(t)\le\frac{(\sqrt{t}+1)^2}{2m^2}\phi(t)\le\frac{(1+\sqrt{\bar{m}})^2}{2m^2}\phi(t)=C_{m,\bar{m}}\phi(t). ψ ( t ) ≤ 2 m 2 ( t + 1 ) 2 ϕ ( t ) ≤ 2 m 2 ( 1 + m ˉ ) 2 ϕ ( t ) = C m , m ˉ ϕ ( t ) .
Therefore
D K L ( p ∥ q ) ≤ C m , m ˉ ∫ p ( q p − 1 ) 2 d λ = C m , m ˉ ∫ ( q − p ) 2 d λ = 2 C m , m ˉ h λ 2 ( p , q ) . \begin{aligned}
D_{\mathrm{KL}}(p\mathbin\Vert q)&\le C_{m,\bar{m}}\int p\left(\sqrt{\frac{q}{p}}-1\right)^2\,d\lambda\\
&=C_{m,\bar{m}}\int(\sqrt{q}-\sqrt{p})^2\,d\lambda=2C_{m,\bar{m}}h_{\lambda}^{2}(p,q).
\end{aligned} D KL ( p ∥ q ) ≤ C m , m ˉ ∫ p ( p q − 1 ) 2 d λ = C m , m ˉ ∫ ( q − p ) 2 d λ = 2 C m , m ˉ h λ 2 ( p , q ) .
Proof of Theorem 4.3 . First take N ≥ N 0 N\ge N_0 N ≥ N 0 . By Lemma B.1 , q ^ Y ∣ X \hat{q}_{Y\mid X} q ^ Y ∣ X corresponds to the ordinary MLE r ^ = r S ^ Y ∣ X \hat{r}=r_{\hat{S}_{Y\mid X}} r ^ = r S ^ Y ∣ X over the class R \mathcal{R} R of λ \lambda λ -densities, and Lemma B.4 places the truth r 0 r_0 r 0 in R \mathcal{R} R . Lemmas B.6 and B.7 bound the global midpoint square-root entropy in L 2 ( λ ) L^2(\lambda) L 2 ( λ ) uniformly in μ X \mu_X μ X . Lemmas B.8 and B.10 and Proposition B.9 then give (45) with δ N = D N − α / ( 2 α + d ) \delta_N=DN^{-\alpha/(2\alpha+d)} δ N = D N − α / ( 2 α + d ) .
Assumption 4.1 and Lemma B.6 give the uniform ratio bounds
m : = c B ≤ r ^ r 0 ≤ B ‾ c = : m ˉ λ -a.e. , m:=\frac{c}{B}\le\frac{\hat{r}}{r_0}\le\frac{\overline{B}}{c}=:\bar{m}\qquad\lambda\text{-a.e.}, m := B c ≤ r 0 r ^ ≤ c B =: m ˉ λ -a.e. ,
almost surely over the training data, so Lemma B.11 applies with these m , m ˉ m,\bar{m} m , m ˉ (shrinking m m m and enlarging m ˉ \bar{m} m ˉ if necessary so that m ≤ 1 ≤ m ˉ m\le1\le\bar{m} m ≤ 1 ≤ m ˉ ). Combining with Lemma B.3 ,
E X ∼ μ X [ D K L ( π Y ∣ X ( ⋅ ∣ X ) ∥ q ^ Y ∣ X ( ⋅ ∣ X ) ) ] = D K L ( r 0 ∥ r ^ ) λ ≤ 2 C m , m ˉ h λ 2 ( r ^ , r 0 ) . (48) \mathbb{E}_{X\sim\mu_X}\left[D_{\mathrm{KL}}\left(\pi_{Y\mid X}(\cdot\mid X)\mathbin\Vert\hat{q}_{Y\mid X}(\cdot\mid X)\right)\right]=D_{\mathrm{KL}}(r_0\mathbin\Vert\hat{r})_{\lambda}\le2C_{m,\bar{m}}h_{\lambda}^{2}(\hat{r},r_0). \tag*{(48)} E X ∼ μ X [ D KL ( π Y ∣ X ( ⋅ ∣ X ) ∥ q ^ Y ∣ X ( ⋅ ∣ X ) ) ] = D KL ( r 0 ∥ r ^ ) λ ≤ 2 C m , m ˉ h λ 2 ( r ^ , r 0 ) . (48)
(i) High probability. Choose
t 2 : = max { 1 , [ log ( c 1 / δ ) ] + c 2 N δ N 2 } , [ u ] + : = max { u , 0 } . t^2:=\max\left\{1,\frac{[\log(c_1/\delta)]_+}{c_2N\delta_N^2}\right\},\qquad[u]_+:=\max\{u,0\}. t 2 := max { 1 , c 2 N δ N 2 [ log ( c 1 / δ ) ] + } , [ u ] + := max { u , 0 } .
Apply (44) with R = t δ N R=t\delta_N R = t δ N . Its right-hand side is at most δ \delta δ , so with probability at least 1 − δ 1-\delta 1 − δ ,
h λ 2 ( r ^ , r 0 ) ≤ t 2 δ N 2 ≤ δ N 2 + [ log ( c 1 / δ ) ] + c 2 N . h_\lambda^2(\hat r,r_0) \le t^2\delta_N^2 \le\delta_N^2+\frac{[\log(c_1/\delta)]_+}{c_2N}. h λ 2 ( r ^ , r 0 ) ≤ t 2 δ N 2 ≤ δ N 2 + c 2 N [ log ( c 1 / δ ) ] + .
Substituting into (48) , using δ N 2 ≍ N − 2 α / ( 2 α + d ) \delta_N^2\asymp N^{-2\alpha/(2\alpha+d)} δ N 2 ≍ N − 2 α / ( 2 α + d ) , and absorbing the constant-order term N − 1 [ log c 1 ] + N^{-1}[\log c_1]_+ N − 1 [ log c 1 ] + into N − 2 α / ( 2 α + d ) N^{-2\alpha/(2\alpha+d)} N − 2 α / ( 2 α + d ) gives (17) .
(ii) In expectation. Integrating the tail bound (45) gives
E [ h λ 2 ( r ^ , r 0 ) ] ≤ δ N 2 + ∫ δ N 2 ∞ Pr ( h λ 2 > s ) d s ≤ δ N 2 + δ N 2 ∫ 1 ∞ c 1 e − c 2 N δ N 2 u d u = δ N 2 ( 1 + c 1 e − c 2 N δ N 2 c 2 N δ N 2 ) ≤ C δ N 2 . \begin{aligned}
\mathbb{E}[h_\lambda^2(\hat r,r_0)] \le\delta_N^2+\int_{\delta_N^2}^{\infty}\Pr(h_\lambda^2>s)\,\mathrm{d}s \le\delta_N^2+\delta_N^2\int_1^\infty c_1e^{-c_2N\delta_N^2u}\,\mathrm{d}u \\
&=\delta_N^2\left(1+\frac{c_1e^{-c_2N\delta_N^2}}{c_2N\delta_N^2}\right)\le C\delta_N^2.
\end{aligned} E [ h λ 2 ( r ^ , r 0 )] ≤ δ N 2 + ∫ δ N 2 ∞ Pr ( h λ 2 > s ) d s ≤ δ N 2 + δ N 2 ∫ 1 ∞ c 1 e − c 2 N δ N 2 u d u = δ N 2 ( 1 + c 2 N δ N 2 c 1 e − c 2 N δ N 2 ) ≤ C δ N 2 .
where the last step uses N δ N 2 = D 2 N d / ( 2 α + d ) ≥ 1 N\delta_N^2=D^2N^{d/(2\alpha+d)}\ge1 N δ N 2 = D 2 N d / ( 2 α + d ) ≥ 1 . Combining with (48) and taking expectations yields (18) .
Finally, E C ≤ log ( B / c ) \mathcal{E}_C\le\log(B/c) E C ≤ log ( B / c ) almost surely by the uniform density bounds. Enlarging C 1 , C 2 C_1,C_2 C 1 , C 2 to cover the finitely many sample sizes 1 ≤ N < N 0 1\le N<N_0 1 ≤ N < N 0 proves both statements for all N ≥ 1 N\ge1 N ≥ 1 . These enlarged constants remain independent of μ X \mu_X μ X .
Proof of Lemma 4.4
Proof. The regularity argument in the proof of Lemma B.4 applies with integration against the probability measure μ X \mu_X μ X in place of Lebesgue measure on a unit-volume set: D β π Y = ∫ D β π Y ∣ X μ X ( d x ) D^\beta\pi_Y=\int D^\beta\pi_{Y\mid X}\,\mu_X(\mathrm{d}x) D β π Y = ∫ D β π Y ∣ X μ X ( d x ) for ∣ β ∣ ≤ m α |\beta|\le m_\alpha ∣ β ∣ ≤ m α , so all derivative sup-norms and the top-order Hölder seminorm are bounded by those of π Y ∣ X \pi_{Y\mid X} π Y ∣ X . The pointwise bounds follow since μ X \mu_X μ X is a probability measure.
Proof of Lemma 4.5
Proof. Lemma 4.4 gives π Y ∈ H α ( Q d y ) \pi_Y\in H^\alpha(Q^{d_y}) π Y ∈ H α ( Q d y ) with the required uniform upper and lower bounds. Apply the realizability, regularity, entropy, and KL–Hellinger arguments of Section B.2 to the ordinary density-estimation problem with dominating measure d y \mathrm{d}y d y and total dimension d y d_y d y : realizability and induced-density regularity are the corresponding d y d_y d y -dimensional KR statements, the Hölder bracketing entropy is O ( ρ − d y / α ) O(\rho^{-d_y/\alpha}) O ( ρ − d y / α ) , and Proposition B.9 gives the critical radius N − α / ( 2 α + d y ) N^{-\alpha/(2\alpha+d_y)} N − α / ( 2 α + d y ) . The KL–Hellinger comparison then yields (20) –(21) . The entropy integral is finite because α > ( d x + d y ) / 2 > d y / 2 \alpha>(d_x+d_y)/2>d_y/2 α > ( d x + d y ) /2 > d y /2 .
KL second moments
Lemma B.12 (Second moments from exponential tails). If a nonnegative random variable Z Z Z satisfies Pr ( Z > u + v t ) ≤ e − t \Pr(Z>u+vt)\le e^{-t} Pr ( Z > u + v t ) ≤ e − t for every t > 0 t>0 t > 0 , with deterministic u , v > 0 u,v>0 u , v > 0 , then
E [ Z 2 ] ≤ u 2 + 2 u v + 2 v 2 . (49) \mathbb{E}[Z^2]\le u^2+2uv+2v^2. \tag*{(49)} E [ Z 2 ] ≤ u 2 + 2 uv + 2 v 2 . (49)
In particular, the tail bounds in Theorem 4.3 and Lemma 4.5 imply the second-moment bounds stated in § 4.2 . No independence between the two fitted densities is needed.
Proof. For a nonnegative random variable, Tonelli’s theorem gives E [ Z 2 ] = 2 ∫ 0 ∞ Pr ( Z > s ) d s \mathbb{E}[Z^2] = 2\int_0^\infty\Pr(Z > s)\,\mathrm{d}s E [ Z 2 ] = 2 ∫ 0 ∞ Pr ( Z > s ) d s . Splitting at u u u and substituting s = u + v t s = u + vt s = u + v t in the remaining integral yields
E [ Z 2 ] ≤ u 2 + 2 v ∫ 0 ∞ ( u + v t ) e − t d t = u 2 + 2 u v + 2 v 2 . \mathbb{E}[Z^2] \le u^2 + 2v\int_0^\infty(u+vt)e^{-t}\,\mathrm{d}t
= u^2 + 2uv + 2v^2. E [ Z 2 ] ≤ u 2 + 2 v ∫ 0 ∞ ( u + v t ) e − t d t = u 2 + 2 uv + 2 v 2 .
Take δ = e − t \delta= e^{-t} δ = e − t in (17) . The tail condition holds for Z = E C Z = \mathcal{E}_C Z = E C , u = C 1 N − a u = C_1N^{-a} u = C 1 N − a , and v = C 1 / N v = C_1/N v = C 1 / N , so
E [ E C 2 ] ≤ C 1 2 ( N − 2 a + 2 N − a − 1 + 2 N − 2 ) ≤ 5 C 1 2 N − 2 a , \mathbb{E}[\mathcal{E}_C^2] \le C_1^2\left(N^{-2a} + 2N^{-a-1} + 2N^{-2}\right)
\le5C_1^2N^{-2a}, E [ E C 2 ] ≤ C 1 2 ( N − 2 a + 2 N − a − 1 + 2 N − 2 ) ≤ 5 C 1 2 N − 2 a ,
using a < 1 a < 1 a < 1 and N ≥ 1 N \ge1 N ≥ 1 . Applying the same argument to (20) , with b < 1 b < 1 b < 1 , proves the marginal bound. No joint event or independence between the two risks is used.
Proof of Corollary 4.6
Proof. The reference bounds and diagonal monotonicity constraints give uniform fitted-density bounds. Lemma Lemma A.1 then gives Assumption 3.1 with the stated K K K . Apply Theorem 3.2 using the KL tails in Theorem 4.3 and Lemma 4.5 , each at failure probability δ / 3 \delta/3 δ /3 . This proves the high-probability bound.
For the MSE bound, decompose I ^ N , M − I = A M + B ( q ^ ) \widehat{I}_{N,M} - I = A_M + B(\widehat{q}) I N , M − I = A M + B ( q ) into the centered evaluation term and the conditional bias. Since E [ A M ∣ q ^ ] = 0 \mathbb{E}[A_M \mid\widehat{q}] = 0 E [ A M ∣ q ] = 0 and B ( q ^ ) B(\widehat{q}) B ( q ) is training-data-measurable, E [ A M B ( q ^ ) ] = 0 \mathbb{E}[A_MB(\widehat{q})] = 0 E [ A M B ( q )] = 0 , so
E [ ( I ^ N , M − I ) 2 ] = E [ A M 2 ] + E [ B ( q ^ ) 2 ] . \mathbb{E}[(\widehat{I}_{N,M} - I)^2] = \mathbb{E}[A_M^2] + \mathbb{E}[B(\widehat{q})^2]. E [( I N , M − I ) 2 ] = E [ A M 2 ] + E [ B ( q ) 2 ] .
Sub-Gaussianity gives E [ A M 2 ] ≤ K 2 / M \mathbb{E}[A_M^2] \le K^2/M E [ A M 2 ] ≤ K 2 / M . Both KL risks are nonnegative, so the bias identity and Lemma B.12 give
E [ B ( q ^ ) 2 ] = E [ ( E Y − E C ) 2 ] ≤ E [ E Y 2 ] + E [ E C 2 ] ≤ C Y , 3 N − 2 b + C C , 3 N − 2 a ≤ C ′ ′ N − 2 a , \mathbb{E}[B(\widehat{q})^2] = \mathbb{E}[(\mathcal{E}_Y - \mathcal{E}_C)^2]
\le\mathbb{E}[\mathcal{E}_Y^2] + \mathbb{E}[\mathcal{E}_C^2]
\le C_{Y,3}N^{-2b} + C_{C,3}N^{-2a}
\le C''N^{-2a}, E [ B ( q ) 2 ] = E [( E Y − E C ) 2 ] ≤ E [ E Y 2 ] + E [ E C 2 ] ≤ C Y , 3 N − 2 b + C C , 3 N − 2 a ≤ C ′′ N − 2 a ,
because b > a b > a b > a . This proves the MSE bound, without independence between the fitted densities.
Proof of Corollary 5.1
Proof. Write the displayed upper bound, including its positive constants, as
F T ( M ) : = C ′ M + C ′ ′ ( T − M ) − 2 a , 0 < M < T . F_T(M) := \frac{C'}{M} + C''(T-M)^{-2a}, \qquad0 < M < T. F T ( M ) := M C ′ + C ′′ ( T − M ) − 2 a , 0 < M < T .
It diverges at both endpoints and is strictly convex. Its unique critical point satisfies
C ′ M 2 = 2 a C ′ ′ ( T − M ) − 2 a − 1 . \frac{C'}{M^2} = 2aC''(T-M)^{-2a-1}. M 2 C ′ = 2 a C ′′ ( T − M ) − 2 a − 1 .
Equivalently,
N = ( 2 a C ′ ′ C ′ ) 1 / ( 2 a + 1 ) M 2 / ( 2 a + 1 ) . N = \left(\frac{2aC''}{C'}\right)^{1/(2a+1)} M^{2/(2a+1)}. N = ( C ′ 2 a C ′′ ) 1/ ( 2 a + 1 ) M 2/ ( 2 a + 1 ) .
Since a > 1 / 2 a > 1/2 a > 1/2 , the exponent 2 / ( 2 a + 1 ) 2/(2a + 1) 2/ ( 2 a + 1 ) is less than one. Thus N = o ( M ) N = o(M) N = o ( M ) , M / T → 1 M/T \to1 M / T → 1 , and
N ∼ ( 2 a C ′ ′ C ′ ) 1 / ( 2 a + 1 ) T 2 / ( 2 a + 1 ) . N \sim\left(\frac{2aC''}{C'}\right)^{1/(2a+1)} T^{2/(2a+1)}. N ∼ ( C ′ 2 a C ′′ ) 1/ ( 2 a + 1 ) T 2/ ( 2 a + 1 ) .
At this allocation, M − 1 ≍ T − 1 M^{-1} \asymp T^{-1} M − 1 ≍ T − 1 and N − 2 a ≍ T − 4 a / ( 2 a + 1 ) = o ( T − 1 ) N^{-2a} \asymp T^{-4a/(2a+1)} = o(T^{-1}) N − 2 a ≍ T − 4 a / ( 2 a + 1 ) = o ( T − 1 ) , proving the MSE claim in Corollary 5.1 . Alternatively, any allocation satisfying N ≳ T 1 / ( 2 a ) N \gtrsim T^{1/(2a)} N ≳ T 1/ ( 2 a ) and T − N ≍ T T - N \asymp T T − N ≍ T attains the O ( T − 1 ) O(T^{-1}) O ( T − 1 ) MSE rate. Among polynomially sublinear allocations, N ≍ T 1 / ( 2 a ) N \asymp T^{1/(2a)} N ≍ T 1/ ( 2 a ) is the smallest sufficient training order. Since 1 / ( 2 a ) < 2 / ( 2 a + 1 ) 1/(2a) < 2/(2a + 1) 1/ ( 2 a ) < 2/ ( 2 a + 1 ) for a > 1 / 2 a > 1/2 a > 1/2 , this order is smaller than the bound-minimizing allocation N ⋆ ≍ T 2 / ( 2 a + 1 ) N^\star\asymp T^{2/(2a+1)} N ⋆ ≍ T 2/ ( 2 a + 1 ) . Rounding to positive integer sample sizes leaves these asymptotic orders unchanged.
Experimental details
This appendix supplies the implementation details for the experiments in § 6 .
Fixed finite-smoothness benchmark
The data-generating conditional, joint, and marginal densities are specified in § 6 . In this experimental appendix, bare B B B , g g g , and V V V abbreviate B ( q ^ ) B(\hat{q}) B ( q ^ ) , g ( q ^ ) g(\hat{q}) g ( q ^ ) , and V ( q ^ ) V(\hat{q}) V ( q ^ ) for the fitted pair under discussion. The generating function h ( t ) = ( 2 t − 1 ) ∣ 2 t − 1 ∣ h(t) = (2t - 1)|2t - 1| h ( t ) = ( 2 t − 1 ) ∣2 t − 1∣ has continuous first derivative h ′ ( t ) = 4 ∣ 2 t − 1 ∣ h'(t) = 4|2t - 1| h ′ ( t ) = 4∣2 t − 1∣ , which is Lipschitz, while its second derivative jumps from − 8 -8 − 8 to 8 8 8 at t = 1 / 2 t = 1/2 t = 1/2 . Thus the conditional and marginal densities have the stated C 1 , 1 C^{1,1} C 1 , 1 regularity, with only one curvature breakpoint per coordinate. Their ranges are respectively [ 0.4 , 1.6 ] [0.4, 1.6] [ 0.4 , 1.6 ] and [ 0.7 , 1.3 ] [0.7, 1.3] [ 0.7 , 1.3 ] . Writing
H ( y ) = ∫ 0 y h ( t ) d t = ∣ 2 y − 1 ∣ 3 − 1 6 , H(y) = \int_0^y h(t)\,\mathrm{d}t = \frac{|2y - 1|^3 - 1}{6}, H ( y ) = ∫ 0 y h ( t ) d t = 6 ∣2 y − 1 ∣ 3 − 1 ,
the exact conditional CDF is y + 0.6 w ( x ) H ( y ) y + 0.6w(x)H(y) y + 0.6 w ( x ) H ( y ) and the marginal CDF is y + 0.3 H ( y ) y + 0.3H(y) y + 0.3 H ( y ) . We sample independent joint pairs by rejection from the uniform distribution on [ 0 , 1 ] 2 [0, 1]^2 [ 0 , 1 ] 2 , with envelope 1.6. The numerical MI reference is I = 0.0019084 I = 0.0019084 I = 0.0019084 nats.
Spline estimators and numerical controls
Let { b i ( K ) } i = 1 K + 2 \{b_i^{(K)}\}_{i=1}^{K+2} { b i ( K ) } i = 1 K + 2 denote the quadratic B-spline basis on K K K uniform cells with clamped endpoints. The fitted densities are
q ^ Y ∣ X ( y ∣ x ) = ∑ i , j c i j b i ( K C ) ( x ) b j ( K C ) ( y ) , K C = ⌈ 2 N 1 / 6 ⌉ , \hat{q}_{Y|X}(y \mid x) = \sum_{i,j} c_{ij}b_i^{(K_C)}(x)b_j^{(K_C)}(y), \qquad K_C = \lceil2N^{1/6}\rceil, q ^ Y ∣ X ( y ∣ x ) = i , j ∑ c ij b i ( K C ) ( x ) b j ( K C ) ( y ) , K C = ⌈ 2 N 1/6 ⌉ ,
q ^ Y ( y ) = ∑ j d j b j ( K Y ) ( y ) , K Y = ⌈ 2 N 1 / 5 ⌉ . \hat{q}_Y(y) = \sum_j d_jb_j^{(K_Y)}(y), \qquad K_Y = \lceil2N^{1/5}\rceil. q ^ Y ( y ) = j ∑ d j b j ( K Y ) ( y ) , K Y = ⌈ 2 N 1/5 ⌉ .
With ω j ( K ) = ∫ 0 1 b j ( K ) ( y ) d y \omega_j^{(K)} = \int_0^1 b_j^{(K)}(y)\,\mathrm{d}y ω j ( K ) = ∫ 0 1 b j ( K ) ( y ) d y , we impose ∑ j c i j ω j ( K C ) = 1 \sum_j c_{ij}\omega_j^{(K_C)} = 1 ∑ j c ij ω j ( K C ) = 1 for every row and ∑ j d j ω j ( K Y ) = 1 \sum_j d_j\omega_j^{(K_Y)} = 1 ∑ j d j ω j ( K Y ) = 1 . All coefficients lie in [ 0.25 , 1.75 ] [0.25, 1.75] [ 0.25 , 1.75 ] . Partition of unity then guarantees the same density bounds. Linear constraints on derivative coefficients bound every first partial derivative by 10 and every pure or mixed second partial derivative by 100, wherever the latter exists. These bounds are fixed as N N N increases. Integrating the fitted densities in y y y gives increasing conditional and marginal maps from [ 0 , 1 ] [0, 1] [ 0 , 1 ] onto itself. The uniform knot grids at different resolutions are not nested, so approximation error need not change monotonically with the cell count.
We minimize empirical negative log likelihood over these convex coefficient sets. Conditional fits use SLSQP or constrained Newton steps with sparse quadratic subproblems solved by OSQP. Marginal fits use SLSQP. The objective and feasible set are unchanged between the conditional solvers. A linear-program certificate must give an original-scale global objective gap at most 10 − 6 10^{-6} 1 0 − 6 , and actual constraint violation must not exceed 10 − 7 10^{-7} 1 0 − 7 . Failed fits are not silently discarded or replaced, and negative plug-in MI estimates are not clipped.
Composite Gauss–Legendre quadrature splits at { 0 , 1 / 2 , 1 } \{0,1/2,1\} { 0 , 1/2 , 1 } and all fitted knots. Paired KL risks, g g g , and V V V use orders 24/48, requiring agreement within 10 − 9 10^{-9} 1 0 − 9 . The independent true-MI reference uses orders 48/96. These are numerical convergence diagnostics, not rigorous integration error bounds. In Experiment 1, the largest quadrature discrepancy was 3.47 × 10 − 16 3.47 \times10^{-16} 3.47 × 1 0 − 16 , and the direct residual in g − I = E Y − E C g-I=\mathcal{E}_Y-\mathcal{E}_C g − I = E Y − E C was at most 2.78 × 10 − 17 2.78 \times10^{-17} 2.78 × 1 0 − 17 . The finite-dimensional sieve and nonzero optimization tolerance mean that the experiments are not literal realizations of the infinite-dimensional MLE in Theorem 4.3 .
Sampling and uncertainty
Experiment 1: Training convergence
For each of 13 training sizes from N = 100 N=100 N = 100 to N = 10 6 N=10^6 N = 1 0 6 , we fit R = 30 R=30 R = 30 independent density pairs and report
R ^ C ( N ) = 1 R ∑ r = 1 R E C ( q ^ ( r ) ) , \widehat{R}_C(N)=\frac{1}{R}\sum_{r=1}^{R}\mathcal{E}_C(\widehat{q}^{(r)}), R C ( N ) = R 1 r = 1 ∑ R E C ( q ( r ) ) ,
R ^ Y ( N ) = 1 R ∑ r = 1 R E Y ( q ^ ( r ) ) , \widehat{R}_Y(N)=\frac{1}{R}\sum_{r=1}^{R}\mathcal{E}_Y(\widehat{q}^{(r)}), R Y ( N ) = R 1 r = 1 ∑ R E Y ( q ( r ) ) ,
R ^ B ( N ) = 1 R ∑ r = 1 R [ g ( q ^ ( r ) ) − I ] 2 . (50) \widehat{R}_B(N)=\frac{1}{R}\sum_{r=1}^{R}[g(\widehat{q}^{(r)})-I]^2. \tag*{(50)} R B ( N ) = R 1 r = 1 ∑ R [ g ( q ( r ) ) − I ] 2 . (50)
Knot-aware quadrature supplies the population risks and g g g , so no finite-M M M evaluation noise enters these measurements. We also check g − I = E Y − E C g-I=\mathcal{E}_Y-\mathcal{E}_C g − I = E Y − E C for every fitted pair. The training sizes are N ∈ { 100 , 200 , 500 , 1000 , 2000 , 5000 , 10000 , 20000 , 50000 , 100000 , 200000 , 500000 , 1000000 } N\in\{100,200,500,1000,2000,5000,10000,20000,50000,100000,200000,500000,1000000\} N ∈ { 100 , 200 , 500 , 1000 , 2000 , 5000 , 10000 , 20000 , 50000 , 100000 , 200000 , 500000 , 1000000 } and the two densities share their training dataset, giving 390 fitted pairs in total. Matching fits at the same sample size and repetition are reused across experiments, without selection by observed error. For each fit, quadrature supplies the vector ( E C , E Y , B 2 ) (\mathcal{E}_C,\mathcal{E}_Y,B^2) ( E C , E Y , B 2 ) . We resample this vector jointly within each N N N , using 2000 bootstrap replicates, to obtain 95% pointwise percentile intervals for the mean risks. Log-log slopes are least-squares slopes of log mean risk against log N N N , using either all 13 sizes or the final decade 10 5 ≤ N ≤ 10 6 10^5\leq N\leq10^6 1 0 5 ≤ N ≤ 1 0 6 . Their bootstrap intervals use the same resampled mean risks and quantify finite-repetition uncertainty, not uncertainty in an asymptotic exponent.
Experiment 2: Fixed-fit evaluation
For Figure 1 (c)–(Figure 1 d), we fix the N = 5000 N=5000 N = 5000 density pair from training repetition 0, chosen without selection by error. For each M M M on the same 13-value grid used for N N N above, we generate 200 independent evaluation datasets, independently of training and of the datasets at other M M M values. The densities are not refitted. With I ^ N , M ( s ) \widehat{I}_{N,M}^{(s)} I N , M ( s ) denoting evaluation repetition s s s , the centered empirical risk is
R ^ A ( M ∣ q ^ ) = 1 200 ∑ s = 1 200 [ I ^ N , M ( s ) − g ( q ^ ) ] 2 . (51) \widehat{R}_{A}(M \mid\widehat{q}) = \frac{1}{200} \sum_{s=1}^{200} \left[\widehat{I}^{(s)}_{N,M} - g(\widehat{q})\right]^2. \tag*{(51)} R A ( M ∣ q ) = 200 1 s = 1 ∑ 200 [ I N , M ( s ) − g ( q ) ] 2 . (51)
The center g ( q ^ ) g(\widehat{q}) g ( q ) is computed by quadrature, not from these evaluation repetitions. Replacing g ( q ^ ) g(\widehat{q}) g ( q ) by I I I gives the empirical total MSE. Centered and total squared errors from the same repetition are resampled together, using 2000 bootstrap replicates within each M M M , to obtain 95% pointwise percentile intervals conditional on this fit. The reference curves are V ( q ^ ) / M V(\widehat{q})/M V ( q ) / M and B ( q ^ ) 2 + V ( q ^ ) / M B(\widehat{q})^2 + V(\widehat{q})/M B ( q ) 2 + V ( q ) / M , with the moments computed by quadrature. The total empirical MSE is measured directly; it need not equal the empirical centered MSE plus B ( q ^ ) 2 B(\widehat{q})^2 B ( q ) 2 for finitely many repetitions.
Experiment 3: Prescribed sample allocations
For Figure 2 , the total sample sizes are T ∈ { 2000 , 5000 , 10000 , 20000 , 50000 , 100000 , 200000 , 500000 , 1000000 } T \in\{2000, 5000, 10000, 20000, 50000, 100000, 200000, 500000, 1000000\} T ∈ { 2000 , 5000 , 10000 , 20000 , 50000 , 100000 , 200000 , 500000 , 1000000 } . We use exactly the two allocation rules specified in § 6.2 , with 30 independent training repetitions at each required N N N . The same fitted pairs are used whenever the rules request the same N N N ; in particular, they coincide at T = 2000 T = 2000 T = 2000 . Quadrature supplies ( B ( q ^ ) 2 , V ( q ^ ) ) (B(\widehat{q})^2, V(\widehat{q})) ( B ( q ) 2 , V ( q )) for each fitted pair. For each N N N , we jointly resample these pairs 2000 times and compute the mean squared plug-in error, the mean variance divided by T − N T - N T − N , and their sum as in (25) . The 2.5% and 97.5% bootstrap quantiles give the pointwise bands. Thus pairing preserves the dependence between the two error components; no evaluation datasets are generated for this comparison. Each T − 1 T^{-1} T − 1 reference is anchored to the corresponding bound-scaling component at T = 2000 T = 2000 T = 2000 and is not a fitted rate or a theorem constant.