Introduction
Personalized decision-making from observational data is a problem faced across medicine, public policy, education, and economics. Given historical records of treatments assigned and outcomes observed, the goal is to learn a policy π : X → { 0 , 1 } \pi: \mathcal{X} \to\{0,1\} π : X → { 0 , 1 } that prescribes treatment as a function of an individual’s covariates so as to maximize population welfare. The growing availability of large observational datasets has driven substantial methodological interest in this problem at the intersection of causal inference and statistical learning theory. Kitagawa and Tetenov [ [21 ] , Athey and Wager [ [2 ] introduced the empirical welfare maximization framework, and designed policy learning methods with regret O ( n − 1 / 2 ) O(n^{-1/2}) O ( n − 1/2 ) , which is also statistically optimal.
A central premise of these results, however, is that the distribution P 0 P_0 P 0 generating the observational data coincides with the distribution under which the learnt policy will be deployed. In practice this assumption is rarely satisfied. Welfare programs are designed in one set of counties and deployed in others Kleinberg et al. [ [22 ] . Clinical treatment rules are derived from one patient population and applied to another, and so on. In each case, the deployment distribution can differ from the training Preprint. distribution along covariate marginals. A policy that is optimal under P 0 P_0 P 0 may incur arbitrarily poor performance under such shifts. This concern about distribution shift has received substantial attention in the broader machine-learning literature on out-of-distribution generalization Liu et al. [ [26 ] but has been largely overlooked in the theoretical foundations of policy learning. Indeed, the bulk of theoretical work on policy learning from observational data [21 , 2 , 45 , 19 ] characterizes the regret of an estimated policy under the same distribution that generated the training data. While such guarantees are valuable, they cannot certify performance under deployment-time distributional shift, and they leave open the question of how to learn policies under distributional shifts.
A natural framework for this problem is distributionally robust optimization (DRO), in which the learner optimizes against the worst-case loss over a neighbourhood of distributions, typically defined by an f f f -divergence ball or a Wasserstein ball around the data-generating distribution. DRO has been studied extensively in operations research and statistics [24 , 25 , 42 ] (Duchi and Namkoong 2021) and provides a principled way to certify performance against bounded distributional shift. However, this literature has focused primarily on supervised learning objectives such as classification and regression. Its application to policy learning from observational data — where the objective involves unobserved counterfactual outcomes and must be reconstructed via an estimator (e.g. doubly-robust scores Kennedy [ [19 ] ) — has been taken up only recently [18 , 1 , 20 , 44 ] , and the finite-sample rates available in that literature are all of order O ~ ( n − 1 / 2 ) \widetilde{O}(n^{-1/2}) O ( n − 1/2 ) . The doubly-robust correction has remained unanalyzed.
Our Contributions: In this paper we bring these two strands of work together. We study the distributionally-robust policy learning problem where the goal is to learn a policy that maximizes the worst-case advantage over an f f f -divergence ball of radius ρ \rho ρ around the data-generating distribution. Our algorithms combine doubly-robust scores with a tractable DRO solver, and analyze their regret under three commonly used ambiguity sets: KL, χ 2 \chi^2 χ 2 , and CVaR. Finite-sample regret guarantees of order O ~ ( n − 1 / 2 ) \widetilde{O}(n^{-1/2}) O ( n − 1/2 ) for distributionally robust policy learning from observational data have been obtained before, notably by Kallus et al. [ [18 ] for KL and general f f f -divergence ambiguity sets and by Wang et al. [ [44 ] under concept drift; our first result recovers such a rate in a unified treatment of the KL, χ 2 \chi^2 χ 2 , and CVaR sets. Our main contribution is to go beyond the O ~ ( n − 1 / 2 ) \widetilde{O}(n^{-1/2}) O ( n − 1/2 ) barrier: to our knowledge this is the first work to obtain fast rates of convergence for any distributionally robust learning problem under a wide-margin condition Tsybakov [ [43 ] . Our precise contributions are the following.
We propose an Algorithm that combines doubly-robust scores with an inner DRO solver to learn optimal distributionally-robust policies for KL, χ 2 \chi^2 χ 2 , and CVaR ambiguity sets. We show that the resulting policy π ^ n \hat{\pi}_n π ^ n attains the regret bound of O ( n − ( β + γ ) ) + O ~ ( d / n ) O(n^{-(\beta+\gamma)}) + \widetilde{O}(\sqrt{d/n}) O ( n − ( β + γ ) ) + O ( d / n ) , under standard assumptions on the nuisance estimators1 and a smooth-parametrized policy class with dimension d d d . The first term decays at the doubly-robust rate n − ( β + γ ) n^{-(\beta+\gamma)} n − ( β + γ ) and the second at O ~ ( n − 1 / 2 ) \widetilde{O}(n^{-1/2}) O ( n − 1/2 ) .
We then propose a new algorithm that exploits the Lagrangian dual of the inner DRO problem to perform sample-splitting bias correction. Under a Tsybakov-style wide-margin condition on the treatment-effect contrast, we establish a fast rate of O ~ ( n − ( 1 − α ) / ( 2 + α ) ) \widetilde{O}(n^{-(1-\alpha)/(2+\alpha)}) O ( n − ( 1 − α ) / ( 2 + α ) ) for KL-divergence and χ 2 \chi^2 χ 2 -divergence. As α \alpha α varies in [ 0 , ∞ ) [0,\infty) [ 0 , ∞ ) the rate can interpolate between O ~ ( n − 1 / 2 ) \widetilde{O}(n^{-1/2}) O ( n − 1/2 ) and O ~ ( n − 1 ) \widetilde{O}(n^{-1}) O ( n − 1 ) at α → ∞ \alpha\to\infty α → ∞ . To our knowledge this is the first guarantee for any distributionally-robust learning problem under a margin condition.
We carry out experiments2 on a semi-synthetic dataset calibrated to the California GAIN welfare-to-work program with a deliberate shift between training and test counties. The results illustrate that distributional robustness yields meaningful improvements over standard empirical welfare maximization in the shifted setting, while remaining competitive when no shift is present.
Policy learning from observational data. The problem of choosing a treatment rule from sample data goes back to Manski [ [29 ] , who framed it as a statistical decision problem and studied minimax-regret rules. The empirical welfare maximization framework was introduced by Kitagawa and Tetenov [ [21 ] , who established O ~ ( n − 1 / 2 ) \widetilde{O}(n^{-1/2}) O ( n − 1/2 ) regret bounds and fast rates of order O ~ ( n − ( 1 + α ) / ( 2 + α ) ) \widetilde{O}(n^{-(1+\alpha)/(2+\alpha)}) O ( n − ( 1 + α ) / ( 2 + α ) ) under a margin condition. Athey and Wager [ [2 ] extended this analysis to the observational setting by incorporating doubly-robust scores into the welfare objective, demonstrating that the same O ~ ( n − 1 / 2 ) \widetilde{O}(n^{-1/2}) O ( n − 1/2 ) rate holds under product nuisance errors of order o ( n − 1 / 2 ) o(n^{-1/2}) o ( n − 1/2 ) . Zhou et al. [ [45 ] considered multi-action and multi-armed extensions, and Kennedy [ [19 ] developed optimal doubly-robust estimators for the heterogeneous treatment effect itself. None of these works addresses distributional shift, and the regret bounds they establish are with respect to the training distribution.
Distributionally robust optimization. The DRO literature has developed a rich toolkit for solving worst-case stochastic optimization problems over f f f -divergence balls and Wasserstein balls. Duchi and Namkoong [ [9 ] established a duality between χ 2 \chi^2 χ 2 -divergence DRO and variance-regularized empirical risk minimization which we also exploit. The convex duality that makes f f f -divergence ambiguity sets tractable, and which underlies the dual reformulation our second algorithm exploits, is due to Ben-Tal et al. [ [5 ] ; Duchi and Namkoong [2021] provided uniform convergence guarantees for f f f -divergence DRO over general hypothesis classes, and Duchi et al. [2021] characterised the asymptotics of the plug-in DRO value through a generalized empirical likelihood lens, which explains why its bias is Θ ( n − 1 / 2 ) \Theta(n^{-1/2}) Θ ( n − 1/2 ) . Namkoong and Duchi [ [34 ] and Levy et al. [ [25 ] developed efficient large-scale stochastic methods for DRO, the latter also giving matching lower bounds on the bias of empirical-plug-in estimators of the DRO value. A parallel line of work replaces the divergence ball with an optimal-transport ball [33 , 6 ] ; Sinha et al. [ [42 ] studied Wasserstein DRO with applications to adversarial robustness in supervised learning. The CVaR ambiguity set, in particular, has a long history in risk-sensitive optimization [38 ] . While this literature provides both algorithmic templates and statistical foundations for solving the inner DRO problem, the resulting bounds are typically of order O ~ ( n − 1 / 2 ) \widetilde{O}(n^{-1/2}) O ( n − 1/2 ) , and we are aware of no prior work that establishes a fast rate for any DRO problem under a margin-style condition.
Distributionally-robust causal inference. A small but growing body of work has studied robustness to distributional shift specifically in the causal-inference setting. Mo et al. [ [32 ] consider distributionally-robust treatment assignment under covariate shift, where the test-time covariate marginal lies in a χ 2 \chi^2 χ 2 ball around the training marginal, and analyze a plug-in estimator. Kallus and Zhou [ [17 ] study robustness to unmeasured confounding via sensitivity analysis, which can be viewed as DRO over the propensity score rather than the covariate distribution. Si et al. [ [41 ] consider distributionally-robust policy evaluation in offline contextual bandit settings. Qi et al. [ [35 ] estimate individualized decision rules that control the average lower tail of the outcome distribution, a CVaR-type criterion. Closest to our setting, Kallus et al. [ [18 ] develop doubly-robust estimators for distributionally-robust off-policy evaluation and learning under KL ambiguity sets, and obtain an O ( n − 1 / 2 ) O(n^{-1/2}) O ( n − 1/2 ) regret rate with nonparametrically estimated propensities; Wang et al. [ [44 ] study robust policy learning under concept drift , where only the conditional law of the outcome given the covariates is perturbed, and establish a sub-optimality gap of order O ( n − 1 / 2 ) O(n^{-1/2}) O ( n − 1/2 ) together with a matching lower bound. However, they only consider KL-constrained ambiguity sets and make additional assumptions regarding the asymptotic estimation rate of empirical risk minimizer. In econometrics, Adjaho and Christensen [ [1 ] study externally valid policy choice, characterising which welfare-maximizing policies remain robust to shifts between the experimental and target populations using optimal-transport ambiguity sets, and Kido [ [20 ] analyzes policy learning over Wasserstein balls. Compared to these works, our contribution is to (i) treat the KL, χ 2 \chi^2 χ 2 , and CVaR ambiguity sets for policy learning (rather than evaluation) from observational data with cross-fitted doubly-robust scores in a single framework, and (ii) establish the first fast rates in this setting under a wide-margin condition. We emphasise that the n − 1 / 2 n^{-1/2} n − 1/2 lower bound of Wang et al. [ [44 ] is a worst-case statement over policy classes and distributions; the margin condition we impose (Assumption 6) is precisely what lets us move past it, exactly as the Tsybakov condition does in classification.
Group and subpopulation robustness. When the ambiguity set is indexed by the mixture proportions of k k k known subgroups rather than by a divergence ball over the whole covariate distribution, the DRO objective reduces to a worst-case-over-groups criterion. This formulation was popularised in machine learning by Hashimoto et al. [ [15 ] , who use a χ 2 \chi^2 χ 2 ball to control the risk of latent minority groups without demographic labels, and by Sagawa et al. [ [39 ] , who optimize the worst-group training loss directly and show that regularization is essential for worst-group generalization. Duchi et al. [ [12 ] give a convex procedure with finite-sample guarantees that controls the worst-case loss over all subpopulations above a given size, and Sahoo et al. [ [40 ] study learning under Γ \Gamma Γ -biased sampling, where the training sample over- or under-represents groups relative to the deployment population, reducing the problem to an augmented convex risk minimization via the CVaR duality of Rockafellar et al. [ [38 ] . Martinez et al. [ [30 ] treat the worst-group criterion as a multi-objective problem and characterise minimax-Pareto-optimal classifiers. All of this work concerns supervised losses. Our group-robust extension (Equation (3) ) complements this line by combining group-level ambiguity with cross-fitted doubly-robust score estimation for the causal objective, and by giving a regret bound of order O ~ ( k log k / n ) \widetilde{O}(\sqrt{k\log k/n}) O ( k log k / n ) whose nuisance-error term decomposes along the per-group sample sizes n g n_g n g .
Margin conditions and fast rates. The Tsybakov margin condition originates in classification, with foundational contributions by Mammen and Tsybakov [ [28 ] and Tsybakov [ [43 ] , and was developed further in the analyses of Massart and Nédélec [ [31 ] (ERM regret under margin via weighted empirical processes), Audibert and Tsybakov [ [3 ] (plug-in classifiers achieving fast and super-fast rates), and Koltchinskii [ [23 ] (local Rademacher complexities and oracle inequalities). Kitagawa and Tetenov [ [21 ] adapted the condition to policy learning to obtain fast regret rates in the empirical-welfare-maximization framework. Luedtke and Van Der Laan [ [27 ] invoke a closely related margin assumption in their work on inference for the value of the optimal treatment rule, where it ensures pathwise differentiability of the value parameter and enables n \sqrt{n} n -consistent inference. Our Assumption 6 follows the Kitagawa–Tetenov formulation, applied to the distributionally-robust treatment-effect contrast under the worst-case distribution.
Doubly-Robust Methods in Causal Inference. Doubly-robust estimation has its origins in the semiparametric statistics and missing-data literature, where Robins et al. [ [37 ] introduced the augmented inverse propensity weighted (AIPW) estimator and showed that it remains consistent if either the outcome model or the propensity model is correctly specified. Bang and Robins [ [4 ] extended this idea to causal inference under the no-unmeasured-confounders assumption and formalised the doubly-robust property as a desirable design criterion when both nuisance components are necessarily approximated. The modern double/debiased machine learning framework of Chernozhukov et al. [ [8 ] reformulated this property through the lens of Neyman-orthogonal moment conditions, showing that combining orthogonal scores with cross-fitting allows arbitrary first-stage machine-learning estimators of the nuisances — converging at slow nonparametric rates — to be plugged in while preserving n − 1 / 2 n^{-1/2} n − 1/2 -consistency and asymptotic normality of the target parameter. Kennedy [ [19 ] subsequently developed optimal doubly-robust estimators specifically for the heterogeneous treatment effect τ ( x ) \tau(x) τ ( x ) , characterising the rates achievable for this conditional functional. In parallel, the contextual-bandit and off-policy-learning literature has adapted these techniques to the sequential decision-making setting: Dudík et al. [ [13 ] introduced the doubly-robust estimator for contextual-bandit policy evaluation and learning, and Dudík et al. [ [14 ] provided a refined statistical analysis. The policy-learning theory of Athey and Wager [ [2 ] that we build on combines the AIPW score of Robins et al. [ [37 ] with empirical welfare maximization in the style of Kitagawa and Tetenov [ [21 ] to obtain O ( n − 1 / 2 ) O(n^{-1/2}) O ( n − 1/2 ) regret bounds under product-rate nuisance error o ( n − 1 / 2 ) o(n^{-1/2}) o ( n − 1/2 ) . Our work extends this line by replacing the empirical welfare objective with a distributionally-robust one while preserving the same doubly-robust structure: Theorem 1 shows that the regret of π ^ \hat{\pi} π ^ inherits the standard doubly-robust decomposition, with a nuisance-product term decaying at n − ( β + γ ) n^{-(\beta+\gamma)} n − ( β + γ ) and a statistical term at O ~ ( n − 1 / 2 ) \widetilde{O}(n^{-1/2}) O ( n − 1/2 ) .
Setting
We assume access to observational data D = { ( X i , Y i , W i ) } i = 1 n \mathcal{D} = \{(X_i,Y_i,W_i)\}_{i=1}^{n} D = {( X i , Y i , W i ) } i = 1 n where X i X_i X i is co-variate of an individual, W i W_i W i is the treatment, Y i Y_i Y i is the outcome. The goal of policy learning is to learn a policy π ∈ Π : X → { 0 , 1 } \pi\in\Pi: \mathcal{X} \to\{0,1\} π ∈ Π : X → { 0 , 1 } which maximizes its value. Following Kitagawa and Tetenov [ [21 ] , Athey and Wager [ [2 ] we consider the value of a policy to be the utility relative to treating no one.
V ( π ) = E X i ∼ P 0 [ Y i ( π ( X i ) ) − Y i ( 0 ) ] V(\pi) = \mathbb{E}_{X_i \sim P_0}[Y_i(\pi(X_i)) - Y_i(0)] V ( π ) = E X i ∼ P 0 [ Y i ( π ( X i )) − Y i ( 0 )]
Let us now write τ ( X i ) = E [ Y i ( 1 ) − Y i ( 0 ) ∣ X i ] \tau(X_i)=\mathbb{E}[Y_i(1)-Y_i(0)\mid X_i] τ ( X i ) = E [ Y i ( 1 ) − Y i ( 0 ) ∣ X i ] to be the conditional average treatment effect. Then it is easy to see the following expression of V ( π ) V(\pi) V ( π ) , which we will also refer to as the advantage of policy π \pi π .
A ( π ) = E X ∼ P 0 [ ( 2 π ( X ) − 1 ) τ ( X ) ] A(\pi)=\mathbb{E}_{X\sim P_0}[(2\pi(X)-1)\tau(X)] A ( π ) = E X ∼ P 0 [( 2 π ( X ) − 1 ) τ ( X )]
The goal of policy learning from observation data is to learn a policy π ( ⋅ ) \pi(\cdot) π ( ⋅ ) that maximizes the advantage A ( ⋅ ) A(\cdot) A ( ⋅ ) , or equivalently minimizes the following notion of regret.3
R ( π ) = max π ′ ∈ Π A ( π ′ ) − A ( π ) R(\pi)=\max_{\pi'\in\Pi} A(\pi')-A(\pi) R ( π ) = π ′ ∈ Π max A ( π ′ ) − A ( π )
It is usually assumed that the underlying distribution P 0 P_0 P 0 over the covariates is fixed and the observed covariates { X i } i = 1 n \{X_i\}_{i=1}^n { X i } i = 1 n are drawn from the same distribution. However, in many applications, the distribution during deployment of the policy differs significantly from the distribution P 0 P_0 P 0 . In order to capture this phenomenon, we consider a distributionally robust version of policy learning problem. Let us define a distributionally robust version of advantage A ( π ) A(\pi) A ( π ) where the target distribution belongs to the set U P 0 = { Q : D f ( Q ∥ P 0 ) ≤ ρ } U_{P_0}=\{Q:D_f(Q\mathbin\Vert P_0)\le\rho\} U P 0 = { Q : D f ( Q ∥ P 0 ) ≤ ρ } .
A f ( π ) = min Q : D f ( Q ∥ P 0 ) ≤ ρ E X ∼ Q [ ( 2 π ( X ) − 1 ) τ ( X ) ] A_f(\pi)=\min_{Q:D_f(Q\mathbin\Vert P_0)\le\rho}\mathbb{E}_{X\sim Q}[(2\pi(X)-1)\tau(X)] A f ( π ) = Q : D f ( Q ∥ P 0 ) ≤ ρ min E X ∼ Q [( 2 π ( X ) − 1 ) τ ( X )]
We aim to maximize A f ( π ) A_f(\pi) A f ( π ) i.e. consider the following distributionally robust policy learning problem.
max π ∈ Π min Q : D f ( Q ∥ P 0 ) ≤ ρ E X ∼ Q [ ( 2 π ( X ) − 1 ) τ ( X ) ] (1) \max_{\pi\in\Pi}\min_{Q:D_f(Q\mathbin\Vert P_0)\le\rho}\mathbb{E}_{X\sim Q}[(2\pi(X)-1)\tau(X)] \tag*{(1)} π ∈ Π max Q : D f ( Q ∥ P 0 ) ≤ ρ min E X ∼ Q [( 2 π ( X ) − 1 ) τ ( X )] (1)
Our first method i.e. Algorithm 1 solves a finite-sample version of the above distributionally robust optimization problem.
max π ∈ Π min Q : D f ( Q ∥ P ^ n ) ≤ ρ ∑ i = 1 n q i ⋅ ( 2 π ( X i ) − 1 ) Γ ^ i . (2) \max_{\pi\in\Pi}\min_{Q:D_f(Q\mathbin\Vert\widehat P_n)\le\rho}\sum_{i=1}^{n}q_i\cdot(2\pi(X_i)-1)\widehat{\Gamma}_i. \tag*{(2)} π ∈ Π max Q : D f ( Q ∥ P n ) ≤ ρ min i = 1 ∑ n q i ⋅ ( 2 π ( X i ) − 1 ) Γ i . (2)
Here Γ ^ i \widehat{\Gamma}_i Γ i is a doubly robust score for estimating the treatment effect τ ( X i ) \tau(X_i) τ ( X i ) . Algorithm 1 builds an estimator of conditional outcomes ( m ^ ( x , w ) ) (\widehat m(x,w)) ( m ( x , w )) and propensity scores ( e ^ ( x ) ) (\widehat e(x)) ( e ( x )) and uses them to construct the doubly robust scores Γ ^ i \widehat{\Gamma}_i Γ i . 4 The optimization problem defined in eq. (2) might be nonconvex in π \pi π , however the inner minimization problem is a convex optimization problem. We first analyze the statistical properties of the policy π ^ \widehat\pi π .
Algorithm 1 Distributionally Robust Policy Learning from Observational Data
Input: Observational data D = { ( X i , Y i , W i ) } i = 1 3 n \mathcal{D}=\{(X_i,Y_i,W_i)\}_{i=1}^{3n} D = {( X i , Y i , W i ) } i = 1 3 n .
Partition dataset D \mathcal{D} D into three disjoint and equal parts D 1 \mathcal{D}_1 D 1 , D 2 \mathcal{D}_2 D 2 , and D 3 \mathcal{D}_3 D 3 .
Build a distributionally robust estimator m ^ ( x , w ) \widehat m(x,w) m ( x , w ) to estimate E [ Y i ( w ) ∣ X i = x ] \mathbb{E}[Y_i(w)\mid X_i=x] E [ Y i ( w ) ∣ X i = x ] using dataset D 2 \mathcal{D}_2 D 2 ,
m ^ ∈ arg min m max Q : D f ( Q ∥ P ^ n ) ≤ ρ ∑ i = n + 1 2 n q i ⋅ ( m ( X i , W i ) − Y i ) 2 \widehat m\in\mathop{\arg\min}_{m}\max_{Q:D_f(Q\mathbin\Vert\widehat P_n)\le\rho}\sum_{i=n+1}^{2n}q_i\cdot(m(X_i,W_i)-Y_i)^2 m ∈ arg min m Q : D f ( Q ∥ P n ) ≤ ρ max i = n + 1 ∑ 2 n q i ⋅ ( m ( X i , W i ) − Y i ) 2
Build a distributionally robust estimator e ^ ( x ) \widehat e(x) e ( x ) to estimate P ( W i = 1 ∣ X i = x ) \mathbb{P}(W_i=1\mid X_i=x) P ( W i = 1 ∣ X i = x ) using dataset D 3 \mathcal{D}_3 D 3 ,
e ^ ∈ arg min e max Q : D f ( Q ∥ P ^ n ) ≤ ρ ∑ i = 2 n + 1 3 n q i ⋅ ( W i − e ( X i ) ) 2 \widehat e\in\mathop{\arg\min}_{e}\max_{Q:D_f(Q\mathbin\Vert\widehat P_n)\le\rho}\sum_{i=2n+1}^{3n}q_i\cdot(W_i-e(X_i))^2 e ∈ arg min e Q : D f ( Q ∥ P n ) ≤ ρ max i = 2 n + 1 ∑ 3 n q i ⋅ ( W i − e ( X i ) ) 2
for i = 1 , … , n i=1,\ldots,n i = 1 , … , n do
Set Γ ^ i = m ^ ( X i , 1 ) − m ^ ( X i , 0 ) + W i − e ^ ( X i ) e ^ ( X i ) ( 1 − e ^ ( X i ) ) ( Y i − m ^ ( X i , W i ) ) \widehat{\Gamma}_i=\widehat m(X_i,1)-\widehat m(X_i,0)+\frac{W_i-\widehat e(X_i)}{\widehat e(X_i)(1-\widehat e(X_i))}(Y_i-\widehat m(X_i,W_i)) Γ i = m ( X i , 1 ) − m ( X i , 0 ) + e ( X i ) ( 1 − e ( X i )) W i − e ( X i ) ( Y i − m ( X i , W i )) .
end for
Return π ^ = arg max π ∈ Π min Q : D f ( Q ∥ P ^ n ) ≤ ρ ∑ i = 1 n q i ( 2 π ( X i ) − 1 ) Γ ^ i \widehat\pi=\mathop{\arg\max}_{\pi\in\Pi}\min_{Q:D_f(Q\mathbin\Vert\widehat P_n)\le\rho}\sum_{i=1}^{n}q_i(2\pi(X_i)-1)\widehat{\Gamma}_i π = arg max π ∈ Π min Q : D f ( Q ∥ P n ) ≤ ρ ∑ i = 1 n q i ( 2 π ( X i ) − 1 ) Γ i
Analysis
Similar to Kitagawa and Tetenov [ [21 ] , we will assume that the potential outcomes are bounded.
Assumption 1 (Bounded Outcomes). There exists M > 0 M > 0 M > 0 such that the outcome variables are contained in [ − M / 2 , M / 2 ] [-M/2, M/2] [ − M /2 , M /2 ] .
Additionally, we assume access to estimators of conditional outcomes (m ( x , w ) m(x,w) m ( x , w ) ) and propensity scores (e ( x ) e(x) e ( x ) ) with decaying L 2 L_2 L 2 error.
Assumption 2. There exist constants β , γ > 0 \beta, \gamma> 0 β , γ > 0 such that
(a) max Q : D f ( Q ∥ P 0 ) ≤ ρ ( E X ∼ Q [ ( m ^ ( X , 1 ) − m ( X , 1 ) ) 2 ] ) 1 / 2 ≤ n − β \max_{Q:D_f(Q\Vert P_0)\leq\rho}\left(\mathbb{E}_{X\sim Q}\left[(\widehat{m}(X,1)-m(X,1))^2\right]\right)^{1/2}\leq n^{-\beta} max Q : D f ( Q ∥ P 0 ) ≤ ρ ( E X ∼ Q [ ( m ( X , 1 ) − m ( X , 1 ) ) 2 ] ) 1/2 ≤ n − β .
(b) max Q : D f ( Q ∥ P 0 ) ≤ ρ ( E X ∼ Q [ ( m ^ ( X , 0 ) − m ( X , 0 ) ) 2 ] ) 1 / 2 ≤ n − β \max_{Q:D_f(Q\Vert P_0)\leq\rho}\left(\mathbb{E}_{X\sim Q}\left[(\widehat{m}(X,0)-m(X,0))^2\right]\right)^{1/2}\leq n^{-\beta} max Q : D f ( Q ∥ P 0 ) ≤ ρ ( E X ∼ Q [ ( m ( X , 0 ) − m ( X , 0 ) ) 2 ] ) 1/2 ≤ n − β .
(c) max Q : D f ( Q ∥ P 0 ) ≤ ρ ( E X ∼ Q [ ( e ^ ( X ) − e ( X ) ) 2 ] ) 1 / 2 ≤ n − γ \max_{Q:D_f(Q\Vert P_0)\leq\rho}\left(\mathbb{E}_{X\sim Q}\left[(\widehat{e}(X)-e(X))^2\right]\right)^{1/2}\leq n^{-\gamma} max Q : D f ( Q ∥ P 0 ) ≤ ρ ( E X ∼ Q [ ( e ( X ) − e ( X ) ) 2 ] ) 1/2 ≤ n − γ .
Finally, we make the standard “overlap” assumption similar to the prior work on policy learning Kitagawa and Tetenov [ [21 ] , Athey and Wager [ [2 ] .
Assumption 3. There exists 1 > κ > 0 1 > \kappa> 0 1 > κ > 0 s.t. P ( W i = 1 ∣ X i = x ) ∈ [ κ , 1 − κ ] \mathbb{P}(W_i=1\mid X_i=x)\in[\kappa,1-\kappa] P ( W i = 1 ∣ X i = x ) ∈ [ κ , 1 − κ ] .
Next we introduce our main theorem. Let us first introduce a few notations. Let A f ( π ) A_f(\pi) A f ( π ) be the distributionally robust value of a policy π \pi π defined as follows.
A f ( π ) = min Q : D f ( Q ∥ P 0 ) ≤ ρ E X ∼ Q [ ( 2 π ( X ) − 1 ) τ ( X ) ] A_f(\pi)=\min_{Q:D_f(Q\Vert P_0)\leq\rho}\mathbb{E}_{X\sim Q}[(2\pi(X)-1)\tau(X)] A f ( π ) = Q : D f ( Q ∥ P 0 ) ≤ ρ min E X ∼ Q [( 2 π ( X ) − 1 ) τ ( X )]
Let A ^ n , f e s t ( π ) \widehat{A}^{\mathrm{est}}_{n,f}(\pi) A n , f est ( π ) be the empirical analogue of the distributionally robust value computed using the doubly robust scores.
A ^ n , f e s t ( π ) = min Q : D f ( Q ∥ P ^ n ) ≤ ρ ∑ i = 1 n q i ⋅ ( 2 π ( X i ) − 1 ) Γ ^ i \widehat{A}^{\mathrm{est}}_{n,f}(\pi)=\min_{Q:D_f(Q\Vert\widehat{P}_n)\leq\rho}\sum_{i=1}^{n}q_i\cdot(2\pi(X_i)-1)\widehat{\Gamma}_i A n , f est ( π ) = Q : D f ( Q ∥ P n ) ≤ ρ min i = 1 ∑ n q i ⋅ ( 2 π ( X i ) − 1 ) Γ i
Algorithm 1 obtains policy π ^ n \widehat{\pi}_n π n by optimizing the empirical distributionally robust value function i.e.
π ^ n ∈ arg max π ∈ Π A ^ n , f e s t ( π ) \widehat{\pi}_n\in\underset{\pi\in\Pi}{\arg\max}\ \widehat{A}^{\mathrm{est}}_{n,f}(\pi) π n ∈ π ∈ Π arg max A n , f est ( π )
The next Theorem bounds the regret of the policy π ^ n \widehat{\pi}_n π n .
Theorem 1. Suppose assumptions 1, 2, and 3 hold. Let N = N ( Π , t c n , ∥ ⋅ ∥ ∞ ) N=\mathcal{N}\left(\Pi,\frac{t}{c\sqrt{n}},\|\cdot\|_\infty\right) N = N ( Π , c n t , ∥ ⋅ ∥ ∞ ) be the t / ( c n ) t/(c\sqrt{n}) t / ( c n ) -covering number of the class of policies Π \Pi Π for some constant c > 0 c>0 c > 0 . Then, with probability at least 1 − 2 N ⋅ exp ( − t 2 / 9 ) 1-2N\cdot\exp(-t^2/9) 1 − 2 N ⋅ exp ( − t 2 /9 ) we have the following upper bound.
max π ∈ Π A f ( π ) − A f ( π ^ n ) ≤ 4 M κ 1 n β + γ + { 4 M κ min { 1 , 1 α n } + 10 M α κ n ⋅ t if A f = A C V a R , 8 M 1 + 2 ρ κ n + 10 M 3 + 2 ρ κ n ⋅ t if A f = A χ 2 , 4 M κ n + 10 M exp ( ρ ) κ n ⋅ t if A f = A K L . \max_{\pi\in\Pi} A_f(\pi)-A_f(\widehat{\pi}_n)\leq\frac{4M}{\kappa}\frac{1}{n^{\beta+\gamma}}+
\begin{cases}
\frac{4M}{\kappa}\min\left\{1,\frac{1}{\sqrt{\alpha n}}\right\}+\frac{10M}{\alpha\kappa\sqrt{n}}\cdot t & \text{if }A_f=A_{\mathrm{CVaR}},\\
\frac{8M\sqrt{1+2\rho}}{\kappa\sqrt{n}}+\frac{10M\sqrt{3+2\rho}}{\kappa\sqrt{n}}\cdot t & \text{if }A_f=A_{\chi^2},\\
\frac{4M}{\kappa\sqrt{n}}+\frac{10M\exp(\rho)}{\kappa\sqrt{n}}\cdot t & \text{if }A_f=A_{\mathrm{KL}}.
\end{cases} π ∈ Π max A f ( π ) − A f ( π n ) ≤ κ 4 M n β + γ 1 + ⎩ ⎨ ⎧ κ 4 M min { 1 , α n 1 } + α κ n 10 M ⋅ t κ n 8 M 1 + 2 ρ + κ n 10 M 3 + 2 ρ ⋅ t κ n 4 M + κ n 10 M e x p ( ρ ) ⋅ t if A f = A CVaR , if A f = A χ 2 , if A f = A KL .
The above theorem applies to general class of policies. However, in order to interpret the bounds we now assume that the class of policies Π \Pi Π is smoothly parametrized.
Assumption 4. Policy class Π = { π ϕ : ϕ ∈ Φ } \Pi=\{\pi_\phi:\phi\in\Phi\} Π = { π ϕ : ϕ ∈ Φ } is parameterized by ϕ \phi ϕ that belongs to a compact set Φ = { ϕ ∈ R d : ∥ ϕ ∥ 2 ≤ B } \Phi=\{\phi\in\mathbb{R}^d:\|\phi\|_2\leq B\} Φ = { ϕ ∈ R d : ∥ ϕ ∥ 2 ≤ B } . Additionally, the mapping ϕ ↦ π ϕ ( x ) \phi\mapsto\pi_\phi(x) ϕ ↦ π ϕ ( x ) is L L L -Lipschitz for every x x x .
Under assumption 4, the covering number N = N ( Π , ε , ∥ ⋅ ∥ ∞ ) ≤ ( 1 + L B ε ) d N=\mathcal{N}(\Pi,\varepsilon,\|\cdot\|_\infty)\leq\left(1+\frac{LB}{\varepsilon}\right)^d N = N ( Π , ε , ∥ ⋅ ∥ ∞ ) ≤ ( 1 + ε L B ) d . After substituting t = O ( d log ( L B n ) ) t=O(\sqrt{d\log(LBn)}) t = O ( d log ( L B n ) ) and L f = O ( 1 / n ) L_f=O(1/\sqrt{n}) L f = O ( 1/ n ) it is easy to verify that
max π ∈ Π A f ( π ) − A f ( π ^ n ) ≤ O ( M κ 1 n β + γ ) + O ( M d log ( L B n ) κ n ) \max_{\pi\in\Pi} A_f(\pi)-A_f(\widehat{\pi}_n)\leq O\left(\frac{M}{\kappa}\frac{1}{n^{\beta+\gamma}}\right)+O\left(\frac{M\sqrt{d\log(LBn)}}{\kappa\sqrt{n}}\right) π ∈ Π max A f ( π ) − A f ( π n ) ≤ O ( κ M n β + γ 1 ) + O ( κ n M d log ( L B n ) )
with probability at least 1 − 1 / n c 1-1/n^c 1 − 1/ n c for some constant c > 0 c>0 c > 0 . The bound has two parts. The first parts decays as O ( n − β − γ ) O(n^{-\beta-\gamma}) O ( n − β − γ ) and is a product of the error rates for conditional outcomes and propensity scores. This is the standard “doubly robust” error bound as it guarantees that the rate of O ( n − 1 / 2 ) O(n^{-1/2}) O ( n − 1/2 ) can be recovered as long as β + γ ≥ 1 / 2 \beta+\gamma\ge1/2 β + γ ≥ 1/2 and allows the estimators e ^ , m ^ \widehat{e},\widehat{m} e , m to have slow rate of convergence. The second part decays at a rate of O ~ ( n − 1 / 2 ) \widetilde{O}(n^{-1/2}) O ( n − 1/2 ) and is the statistical error.
Proof sketch: In order to prove Theorem 1 we introduce a notation. Let
A ~ f e s t ( π ) = min Q : D f ( Q ∥ P 0 ) ≤ ρ E X ∼ Q , W ∼ π o f f ( ⋅ ∣ X ) [ ( 2 π ( X ) − 1 ) Γ ^ ( X , W ) ] \widetilde{A}^{\mathrm{est}}_f(\pi)=\min_{Q:D_f(Q\|P_0)\le\rho}\mathbb{E}_{X\sim Q,W\sim\pi_{\mathrm{off}}(\cdot\mid X)}\left[(2\pi(X)-1)\widehat{\Gamma}(X,W)\right] A f est ( π ) = Q : D f ( Q ∥ P 0 ) ≤ ρ min E X ∼ Q , W ∼ π off ( ⋅ ∣ X ) [ ( 2 π ( X ) − 1 ) Γ ( X , W ) ]
be the (distributionally robust) population advantage of policy π \pi π when using estimates Γ ^ ( X , W ) \widehat{\Gamma}(X,W) Γ ( X , W ) instead of the actual treatment effect. Then the proof establishes the following upper bound.
sup π ∈ Π A f ( π ) − A f ( π ^ n ) ≤ 2 sup π ∈ Π ∣ A f ( π ) − A ~ f e s t ( π ) ∣ + 2 sup π ∈ Π ∣ A ~ f e s t ( π ) − A ^ n , f e s t ( π ) ∣ \sup_{\pi\in\Pi} A_f(\pi)-A_f(\widehat{\pi}_n)\le2\sup_{\pi\in\Pi}\left|A_f(\pi)-\widetilde{A}^{\mathrm{est}}_f(\pi)\right|+2\sup_{\pi\in\Pi}\left|\widetilde{A}^{\mathrm{est}}_f(\pi)-\widehat{A}^{\mathrm{est}}_{n,f}(\pi)\right| π ∈ Π sup A f ( π ) − A f ( π n ) ≤ 2 π ∈ Π sup A f ( π ) − A f est ( π ) + 2 π ∈ Π sup A f est ( π ) − A n , f est ( π )
The first term is bounded by ( 2 M / κ ) n − ( β + γ ) (2M/\kappa)n^{-(\beta+\gamma)} ( 2 M / κ ) n − ( β + γ ) and the second term is bounded by O ~ ( d / n ) \widetilde{O}(\sqrt{d/n}) O ( d / n ) .
Group-Robust Policy Learning
We now consider a specific type of distributional shift where there are k k k subgroups and they are represented in an observational study by different proportions. This group-based ambiguity set is the one used in the group-DRO literature on supervised losses [15 , 39 , 12 , 40 ] ; we adapt it here to the causal policy-learning objective. Our goal is to be robust against a distribution where the representation of different groups change, however, the distribution of treatment effects within a specific group remains the same. In particular, we assume access to observational data D = { ( X i , Y i , W i , G i ) } i = 1 n \mathcal{D}=\{(X_i,Y_i,W_i,G_i)\}_{i=1}^n D = {( X i , Y i , W i , G i ) } i = 1 n where X i X_i X i is co-variate of an individual, W i W_i W i is the treatment, Y i Y_i Y i is the outcome and G i ∈ [ k ] G_i\in[k] G i ∈ [ k ] is the group of the individual with co-variate X i X_i X i . We will also write D g \mathcal{D}_g D g to represent the data available from group g g g , and let n g = ∣ D g ∣ n_g=|\mathcal{D}_g| n g = ∣ D g ∣ .
Let us define A g ( π ) = E X i ∼ P g [ ( 2 π ( X i ) − 1 ) τ ( X i ) ] A_g(\pi)=\mathbb{E}_{X_i\sim P_g}[(2\pi(X_i)-1)\tau(X_i)] A g ( π ) = E X i ∼ P g [( 2 π ( X i ) − 1 ) τ ( X i )] as the advantage of policy π \pi π conditioned on covariates drawn from distribution P g P_g P g supported over group g g g . Here we allow for the possibility that the support of the distributions can be overlapping. We measure the performance of a policy π \pi π as its worst-case advantage over a family of distributions over the k k k groups.
A f ( π ) = min p : D f ( p ∥ d 0 ) ≤ ρ ∑ g = 1 k p g ⋅ A g ( π ) A_f(\pi)=\min_{p:D_f(p\|d_0)\le\rho}\sum_{g=1}^{k}p_g\cdot A_g(\pi) A f ( π ) = p : D f ( p ∥ d 0 ) ≤ ρ min g = 1 ∑ k p g ⋅ A g ( π )
Here, d 0 d_0 d 0 is the distribution over the groups observed in the dataset i.e. d 0 ( g ) = n g n d_0(g)=\frac{n_g}{n} d 0 ( g ) = n n g . The group distributionally robust policy learning problem aims to minimize the worst case advantage over the collection of distributions { p : D f ( p ∥ d 0 ) ≤ ρ } \{p:D_f(p\|d_0)\le\rho\} { p : D f ( p ∥ d 0 ) ≤ ρ } .
max π min p : D f ( p ∥ d 0 ) ≤ ρ ∑ g = 1 k p g ⋅ A g ( π ) (3) \max_{\pi}\ \min_{p:D_f(p\|d_0)\le\rho}\sum_{g=1}^{k}p_g\cdot A_g(\pi) \tag*{(3)} π max p : D f ( p ∥ d 0 ) ≤ ρ min g = 1 ∑ k p g ⋅ A g ( π ) (3)
We will solve a finite-sample version of the optimization problem defined in eq. (3) .
π ^ n ← arg max π min p : D f ( p ∥ d 0 ) ≤ ρ ∑ g = 1 k p g ⋅ A ^ g e s t ( π ) (4) \widehat{\pi}_n\leftarrow\arg\max_{\pi}\ \min_{p:D_f(p\|d_0)\le\rho}\sum_{g=1}^{k}p_g\cdot\widehat{A}^{\mathrm{est}}_g(\pi) \tag*{(4)} π n ← arg π max p : D f ( p ∥ d 0 ) ≤ ρ min g = 1 ∑ k p g ⋅ A g est ( π ) (4)
Here A ^ g e s t ( π ) \widehat{A}^{\mathrm{est}}_g(\pi) A g est ( π ) is an estimator of the advantage of policy π \pi π for group g g g . Similar to algorithm 1, we will build an estimator using doubly robust scores Athey and Wager [ [2 ] . In particular, A ^ g e s t ( π ) = 1 ∣ D g ∣ ∑ i ∈ D g ( 2 π ( X i ) − 1 ) ⋅ Γ ^ i \widehat{A}^{\mathrm{est}}_g(\pi)=\frac{1}{|\mathcal{D}_g|}\sum_{i\in\mathcal{D}_g}(2\pi(X_i)-1)\cdot\widehat{\Gamma}_i A g est ( π ) = ∣ D g ∣ 1 ∑ i ∈ D g ( 2 π ( X i ) − 1 ) ⋅ Γ i where
Γ ^ i = m ^ g ( X i , 1 ) − m ^ g ( X i , 0 ) + W i − e ^ g ( X i ) e ^ g ( X i ) ( 1 − e ^ g ( X i ) ) ( Y i − m ^ g ( X i , W i ) ) . (5) \widehat{\Gamma}_i=\widehat{m}_g(X_i,1)-\widehat{m}_g(X_i,0)+\frac{W_i-\widehat{e}_g(X_i)}{\widehat{e}_g(X_i)(1-\widehat{e}_g(X_i))}\left(Y_i-\widehat{m}_g(X_i,W_i)\right). \tag*{(5)} Γ i = m g ( X i , 1 ) − m g ( X i , 0 ) + e g ( X i ) ( 1 − e g ( X i )) W i − e g ( X i ) ( Y i − m g ( X i , W i ) ) . (5)
Here m ^ g ( x , w ) \widehat{m}_g(x,w) m g ( x , w ) is an estimate of conditional outcome m g ( x , w ) = E [ Y i ( w ) ∣ X i = x , X i ∈ X g ] m_g(x,w)=\mathbb{E}[Y_i(w)\mid X_i=x,X_i\in\mathcal{X}_g] m g ( x , w ) = E [ Y i ( w ) ∣ X i = x , X i ∈ X g ] , and e ^ g ( x ) \widehat{e}_g(x) e g ( x ) is an estimate of the propensity score e g ( x ) = P ( W i = 1 ∣ X i = x , X i ∈ X g ) e_g(x)=\mathbb{P}(W_i=1\mid X_i=x,X_i\in\mathcal{X}_g) e g ( x ) = P ( W i = 1 ∣ X i = x , X i ∈ X g ) . Note that, compared to algorithm 1 we don’t need to build a distributionally robust estimator for conditional outcomes, and propensity scores since the group distributions { P g } g ∈ [ k ] \{P_g\}_{g\in[k]} { P g } g ∈ [ k ] are assumed to be fixed, and only the relative proportions of the groups change. Algorithm 2 describes the full algorithm.
Algorithm 2 Group Robust Policy Learning from Observational Data
Input: Observational data D = { ( X i , Y i , W i , G i ) } i = 1 n D = \{(X_i, Y_i, W_i, G_i)\}_{i=1}^{n} D = {( X i , Y i , W i , G i ) } i = 1 n .
Partition dataset D D D into three disjoint and equal parts D 1 D_1 D 1 , D 2 D_2 D 2 , and D 3 D_3 D 3 , uniformly at random.
for g = 1 , … , k g = 1, \ldots, k g = 1 , … , k do
Let D j , g = D j ∩ { ( X i , Y i , W i , G i ) : G i = g } D_{j,g} = D_j \cap\{(X_i, Y_i, W_i, G_i) : G_i = g\} D j , g = D j ∩ {( X i , Y i , W i , G i ) : G i = g } for j = 1 , 2 , 3 j = 1, 2, 3 j = 1 , 2 , 3 , and write n g = ∣ D 1 , g ∣ n_g = |D_{1,g}| n g = ∣ D 1 , g ∣ .
Build a distributionally robust estimator m ^ g ( x , w ) \widehat{m}_g(x,w) m g ( x , w ) to estimate E [ Y i ( w ) ∣ X i = x , X i ∈ X g ] \mathbb{E}[Y_i(w) \mid X_i = x, X_i \in\mathcal{X}_g] E [ Y i ( w ) ∣ X i = x , X i ∈ X g ] using dataset D 2 , g D_{2,g} D 2 , g , i.e. solve
m ^ g ∈ arg min m max Q : D f ( Q ∥ P ^ n g ) ≤ ρ ∑ i ∈ D 2 , g q i ⋅ ( m ( X i , W i ) − Y i ) 2 \widehat{m}_g \in\mathop{\arg\min}_{m}\ \max_{Q:D_f(Q\Vert\widehat{P}_{n_g})\leq\rho}\sum_{i\in D_{2,g}}q_i\cdot\left(m(X_i,W_i)-Y_i\right)^2 m g ∈ arg min m Q : D f ( Q ∥ P n g ) ≤ ρ max i ∈ D 2 , g ∑ q i ⋅ ( m ( X i , W i ) − Y i ) 2
Build a distributionally robust estimator e ^ ( x ) \widehat{e}(x) e ( x ) to estimate P ( W i = 1 ∣ X i = x ) \mathbb{P}(W_i = 1 \mid X_i = x) P ( W i = 1 ∣ X i = x ) using dataset D 3 , g D_{3,g} D 3 , g , i.e. solve
e ^ ∈ arg min e max Q : D f ( Q ∥ P ^ n g ) ≤ ρ ∑ i ∈ D 3 , g q i ⋅ ( W i − e ( X i ) ) 2 \widehat{e} \in\mathop{\arg\min}_{e}\ \max_{Q:D_f(Q\Vert\widehat{P}_{n_g})\leq\rho}\sum_{i\in D_{3,g}}q_i\cdot\left(W_i-e(X_i)\right)^2 e ∈ arg min e Q : D f ( Q ∥ P n g ) ≤ ρ max i ∈ D 3 , g ∑ q i ⋅ ( W i − e ( X i ) ) 2
end for
for g = 1 , … , k g = 1, \ldots, k g = 1 , … , k do
for i ∈ D 1 , g i \in D_{1,g} i ∈ D 1 , g do
Set Γ ^ i = m ^ g ( X i , 1 ) − m ^ g ( X i , 0 ) + W i − e ^ g ( X i ) e ^ g ( X i ) ( 1 − e ^ g ( X i ) ) ( Y i − m ^ g ( X i , W i ) ) \widehat{\Gamma}_i = \widehat{m}_g(X_i,1)-\widehat{m}_g(X_i,0)+\frac{W_i-\widehat{e}_g(X_i)}{\widehat{e}_g(X_i)(1-\widehat{e}_g(X_i))}\left(Y_i-\widehat{m}_g(X_i,W_i)\right) Γ i = m g ( X i , 1 ) − m g ( X i , 0 ) + e g ( X i ) ( 1 − e g ( X i )) W i − e g ( X i ) ( Y i − m g ( X i , W i ) ) .
end for
end for
Solve π ^ = arg max π ∈ Π min p : D f ( p ∥ d 0 ) ≤ ρ ∑ g = 1 k p g ⋅ 1 n g ∑ i ∈ D 1 , g ( 2 π ( X i ) − 1 ) Γ ^ i \widehat{\pi}=\mathop{\arg\max}_{\pi\in\Pi}\min_{p:D_f(p\Vert d_0)\leq\rho}\sum_{g=1}^{k}p_g\cdot\frac{1}{n_g}\sum_{i\in D_{1,g}}\left(2\pi(X_i)-1\right)\widehat{\Gamma}_i π = arg max π ∈ Π min p : D f ( p ∥ d 0 ) ≤ ρ ∑ g = 1 k p g ⋅ n g 1 ∑ i ∈ D 1 , g ( 2 π ( X i ) − 1 ) Γ i .
Return π ^ \widehat{\pi} π .
Analysis
We weaken Assumption 2 and adapt it to the setting of multiple groups.
Assumption 5. For each group g ∈ [ k ] g \in[k] g ∈ [ k ] , there exists constants β g \beta_g β g , γ g \gamma_g γ g such that
( E P 0 [ ( m ^ g ( X , w ) − m ( X , w ) ) 2 ] ) 1 / 2 ≤ n g − β g ∀ w ∈ { 0 , 1 } , ( E P 0 [ ( e ^ ( X ) − e ( X ) ) 2 ] ) 1 / 2 ≤ n g − γ g . \left(\mathbb{E}_{P_0}\left[\left(\widehat{m}_g(X,w)-m(X,w)\right)^2\right]\right)^{1/2}\leq n_g^{-\beta_g}\quad\forall w\in\{0,1\},\qquad\left(\mathbb{E}_{P_0}\left[\left(\widehat{e}(X)-e(X)\right)^2\right]\right)^{1/2}\leq n_g^{-\gamma_g}. ( E P 0 [ ( m g ( X , w ) − m ( X , w ) ) 2 ] ) 1/2 ≤ n g − β g ∀ w ∈ { 0 , 1 } , ( E P 0 [ ( e ( X ) − e ( X ) ) 2 ] ) 1/2 ≤ n g − γ g .
Theorem 2. Suppose Assumptions 1, 3, 4, and 5 hold. Then for a constant c ≥ 1 c \geq1 c ≥ 1 the following holds with probability at least 1 − n − c 1-n^{-c} 1 − n − c .
max π ∈ Π A f ( π ) − A f ( π ^ n ) ≤ 2 M κ max p : D f ( p ∥ d 0 ) ≤ ρ ∑ g = 1 k p g β g + γ g + O ( M k ( log k + d log ( L B n M / κ ) ) κ n ⋅ e f ) \max_{\pi\in\Pi} A_f(\pi)-A_f(\widehat{\pi}_n)\leq\frac{2M}{\kappa}\max_{p:D_f(p\Vert d_0)\leq\rho}\sum_{g=1}^{k}\frac{p_g}{\beta_g+\gamma_g}+O\left(\frac{M\sqrt{k(\log k+d\log(LBnM/\kappa))}}{\kappa\sqrt{n}}\cdot e_f\right) π ∈ Π max A f ( π ) − A f ( π n ) ≤ κ 2 M p : D f ( p ∥ d 0 ) ≤ ρ max g = 1 ∑ k β g + γ g p g + O ( κ n M k ( log k + d log ( L B n M / κ )) ⋅ e f ) where e f = 1 / α e_f=1/\alpha e f = 1/ α for CVaR constraint at level α \alpha α , 1 + 2 ρ \sqrt{1+2\rho} 1 + 2 ρ for χ 2 \chi^2 χ 2 -divergence and exp ( ρ ) \exp(\rho) exp ( ρ ) for KL-divergence.
The bound consists of two terms. The second term decays at a rate of O ~ ( k log k / n ) \widetilde{O}(\sqrt{k\log k}/\sqrt{n}) O ( k log k / n ) . The first term resembles the doubly robust error bound from Theorem 1. If β g = β \beta_g=\beta β g = β and γ g = γ \gamma_g=\gamma γ g = γ for all g g g , then this error bound is dominated by the sample sizes of the minority groups, and the designer should collect more samples from such groups. However, if further sample collection is not possible then improving β g , γ g \beta_g,\gamma_g β g , γ g for minority groups improves the overall error bound.
Fast Rates with the Wide Margin Assumption
In the previous sections, we have derived O ~ ( 1 / n ) \widetilde{O}(1/\sqrt{n}) O ( 1/ n ) rate of convergence bound for the distributionally robust policy learning problem. In the seminal work, Kitagawa and Tetenov [ [21 ] have shown that for the standard policy learning problem it is possible to improve the rate of convergence under the wide margin assumption . In this section, we propose a new policy learning algorithm that improves the rate of convergence under the wide margin assumption. To the best of our knowledge, this is the first work to derive fast rate of convergence for distributionally robust optimization under the wide margin assumption. Let us define π b ∗ \pi_b^* π b ∗ to be the Bayes-optimal decision rule i.e.
π b ∗ ( x ) = { 1 if τ ( X ) ≥ 0 0 o.w. \pi_b^*(x)=
\begin{cases}
1 & \text{if }\tau(X)\geq0\\
0 & \text{o.w.}
\end{cases} π b ∗ ( x ) = { 1 0 if τ ( X ) ≥ 0 o.w.
We make the following Margin assumption about the distribution of the treatment effect Kitagawa and Tetenov [ [21 ] .
Assumption 6. The Bayes-optimal policy lies in the space of policies Π \Pi Π i.e. π b ∗ ∈ Π \pi_b^* \in\Pi π b ∗ ∈ Π , and there exist constants 0 < η ≤ M 0 < \eta\le M 0 < η ≤ M and 0 < α < ∞ 0 < \alpha< \infty 0 < α < ∞ s.t.
P X ∼ P 0 ( ∣ τ ( X ) ∣ ≤ t ) ≤ ( t η ) α , ∀ t ∈ [ 0 , η ] . \mathbb{P}_{X\sim P_0}(|\tau(X)| \le t) \le\left(\frac{t}{\eta}\right)^\alpha,\quad\forall t \in[0,\eta]. P X ∼ P 0 ( ∣ τ ( X ) ∣ ≤ t ) ≤ ( η t ) α , ∀ t ∈ [ 0 , η ] .
The wide margin assumption was originally proposed in the literature on binary classification Tsybakov [ [43 ] , and in the context of policy learning it implies that the treatment effects are concentrated away from zero.
We modify Algorithm 1 to obtain fast rates under the Margin assumption. In its current form, Algorithm 1 has inherent limitations to achieve fast rates. Recall that Algorithm 1 essentially optimizes an empirical objective A ^ n , f ( π ) \widehat{A}_{n,f}(\pi) A n , f ( π ) , if we ignore the errors from doubly robust scores. Under the wide margin assumption, it might be possible to show fast convergence of A ^ n , f ( π ) \widehat{A}_{n,f}(\pi) A n , f ( π ) to its mean. However, as Lemma 1 (in Section B) shows the deviation between E [ A ^ n , f ( π ) ] \mathbb{E}[\widehat{A}_{n,f}(\pi)] E [ A n , f ( π )] and A f ( π ) A_f(\pi) A f ( π ) is O ( 1 / n ) O(1/\sqrt{n}) O ( 1/ n ) ; this bias is intrinsic to the plug-in DRO value [Duchi et al. 2021], and Levy et al. [ [25 ] constructs examples for several families of divergence functions to show that this bound is also optimal. Therefore, any improvement due to wide margin is killed by the O ( 1 / n ) O(1/\sqrt{n}) O ( 1/ n ) -bias term.
Instead, the new Algorithm 3 exploits the dual characterization of the inner optimization problem in eq. (2) , due to Ben-Tal et al. [ [5 ] (see also Proposition 1 of Duchi and Namkoong [2021]). In particular,
min Q : D f ( Q ∥ P ^ n ) ≤ ρ ∑ i = 1 n q i ( 2 π ( X i ) − 1 ) Γ ^ i = max λ ≥ 0 , η ∈ R 1 n ∑ i = 1 n − λ f ∗ ( − ( 2 π ( X i ) − 1 ) Γ ^ i − η λ ) − λ ρ − η \min_{Q:D_f(Q\Vert\widehat{P}_n)\le\rho}\sum_{i=1}^{n}q_i(2\pi(X_i)-1)\widehat{\Gamma}_i
=
\max_{\lambda\ge0,\eta\in\mathbb{R}}\frac{1}{n}\sum_{i=1}^{n}-\lambda f^*\left(\frac{-(2\pi(X_i)-1)\widehat{\Gamma}_i-\eta}{\lambda}\right)-\lambda\rho-\eta Q : D f ( Q ∥ P n ) ≤ ρ min i = 1 ∑ n q i ( 2 π ( X i ) − 1 ) Γ i = λ ≥ 0 , η ∈ R max n 1 i = 1 ∑ n − λ f ∗ ( λ − ( 2 π ( X i ) − 1 ) Γ i − η ) − λ ρ − η
We partition the data into four equal parts D 1 , D 2 , D 3 \mathcal{D}_1,\mathcal{D}_2,\mathcal{D}_3 D 1 , D 2 , D 3 , and D 4 \mathcal{D}_4 D 4 uniformly at random, and as before use D 2 \mathcal{D}_2 D 2 and D 3 \mathcal{D}_3 D 3 for building estimators of conditional outcome and propensity scores respectively. Then we estimate the dual variables λ \lambda λ and η \eta η for each policy π \pi π using D 4 \mathcal{D}_4 D 4 , and solve the optimization problem using the estimated dual variables and D 1 \mathcal{D}_1 D 1 . The main reason for this modification is that the dual variables are not random variables but deterministic functions of the distribution P 0 P_0 P 0 and the policy π \pi π . Therefore, we can obtain a better concentration bound for the empirical objective function by decoupling the estimation of dual variables from the optimization over policies. In particular, first compute the dual variables λ ^ ( π ) \widehat{\lambda}(\pi) λ ( π ) and η ^ ( π ) \widehat{\eta}(\pi) η ( π ) for each policy π \pi π as
( λ ^ ( π ) , η ^ ( π ) ) = a r g m a x λ ≥ 0 , η ∈ R 1 n ∑ i ∈ D 1 − λ f ∗ ( − ( 2 π ( X i ) − 1 ) Γ ^ i − η λ ) − λ ρ − η (\widehat{\lambda}(\pi),\widehat{\eta}(\pi))=\mathop{\rm arg\,max}_{\lambda\ge0,\eta\in\mathbb{R}}\frac{1}{n}\sum_{i\in\mathcal{D}_1}-\lambda f^*\left(\frac{-(2\pi(X_i)-1)\widehat{\Gamma}_i-\eta}{\lambda}\right)-\lambda\rho-\eta ( λ ( π ) , η ( π )) = arg max λ ≥ 0 , η ∈ R n 1 i ∈ D 1 ∑ − λ f ∗ ( λ − ( 2 π ( X i ) − 1 ) Γ i − η ) − λ ρ − η
Then solve the following optimization problem to obtain the policy π ^ \widehat{\pi} π .
π ^ = a r g m a x π ∈ Π 1 n ∑ i ∈ D 4 − λ ^ ( π ) f ∗ ( − ( 2 π ( X i ) − 1 ) Γ ^ i − η ^ ( π ) λ ^ ( π ) ) − λ ^ ( π ) ρ − η ^ ( π ) (6) \widehat{\pi}=\mathop{\rm arg\,max}_{\pi\in\Pi}\frac{1}{n}\sum_{i\in\mathcal{D}_4}-\widehat{\lambda}(\pi)f^*\left(\frac{-(2\pi(X_i)-1)\widehat{\Gamma}_i-\widehat{\eta}(\pi)}{\widehat{\lambda}(\pi)}\right)-\widehat{\lambda}(\pi)\rho-\widehat{\eta}(\pi) \tag*{(6)} π = arg max π ∈ Π n 1 i ∈ D 4 ∑ − λ ( π ) f ∗ ( λ ( π ) − ( 2 π ( X i ) − 1 ) Γ i − η ( π ) ) − λ ( π ) ρ − η ( π ) (6)
Note that, the current description of the algorithm is inefficient as we need to compute λ ^ ( π ) , η ^ ( π ) \widehat{\lambda}(\pi),\widehat{\eta}(\pi) λ ( π ) , η ( π ) for each policy π ∈ Π \pi\in\Pi π ∈ Π . However, they can be computed only for the policies encountered by an algorithm solving the policy optimization problem defined in Equation (6) .
The proofs for fast rates of convergence are complex, and involves several intermediate steps for all three divergence functions. We now provide an overview of the proof for the case of KL-divergence. The full proofs for KL and χ 2 \chi^2 χ 2 -divergences are included in the Appendix. There is inherent difficulty in extending this approach to CVaR constraints, which we discuss in the conclusion.
Proof for KL-divergence
Step 1. Regret Decomposition: We start with a regret decomposition different from the proof of Theorem 1. Let us write B ^ K L ( π ) = E [ A ^ n , K L e s t ( π ) ] − A ^ K L e s t ( π ) \widehat{B}_{\mathrm{KL}}(\pi)=\mathbb{E}\left[\widehat{A}_{n,\mathrm{KL}}^{\mathrm{est}}(\pi)\right]-\widehat{A}_{\mathrm{KL}}^{\mathrm{est}}(\pi) B KL ( π ) = E [ A n , KL est ( π ) ] − A KL est ( π ) to denote the bias term because of using Algorithm 3 Distributionally Robust Policy Learning from Observational Data (Exploiting Dual Characterization)
Input: Observational data D = { ( X i , Y i , W i ) } i = 1 4 n \mathcal{D} = \{(X_i, Y_i, W_i)\}_{i=1}^{4n} D = {( X i , Y i , W i ) } i = 1 4 n .
Partition dataset D \mathcal{D} D into four disjoint and equal parts D 1 \mathcal{D}_1 D 1 , D 2 \mathcal{D}_2 D 2 , D 3 \mathcal{D}_3 D 3 , and D 4 \mathcal{D}_4 D 4 .
Similar to algorithm 1, build a distributionally robust estimator m ^ ( x , w ) \widehat{m}(x,w) m ( x , w ) using dataset D 2 \mathcal{D}_2 D 2 , and a distributionally robust estimator e ^ ( x ) \widehat{e}(x) e ( x ) using dataset D 3 \mathcal{D}_3 D 3 .
for i = 1 , … , n i=1,\ldots,n i = 1 , … , n do
Set Γ ^ i = m ^ ( X i , 1 ) − m ^ ( X i , 0 ) + W i − e ^ ( X i ) e ^ ( X i ) ( 1 − e ^ ( X i ) ) ( Y i − m ^ ( X i , W i ) ) \widehat{\Gamma}_i = \widehat{m}(X_i,1) - \widehat{m}(X_i,0) + \frac{W_i-\widehat{e}(X_i)}{\widehat{e}(X_i)(1-\widehat{e}(X_i))}(Y_i-\widehat{m}(X_i,W_i)) Γ i = m ( X i , 1 ) − m ( X i , 0 ) + e ( X i ) ( 1 − e ( X i )) W i − e ( X i ) ( Y i − m ( X i , W i )) .
end for
▹ \triangleright ▹ Estimate λ ^ ( π ) \widehat{\lambda}(\pi) λ ( π ) and η ^ ( π ) \widehat{\eta}(\pi) η ( π ) for each policy π \pi π using dataset D 4 \mathcal{D}_4 D 4 .
For any policy π \pi π , set ( λ ^ ( π ) , η ^ ( π ) ) = arg max λ ≥ 0 , η ∈ R 1 n ∑ i ∈ D 1 − λ f ⋆ ( − ( 2 π ( X i ) − 1 ) Γ ^ i − η λ ) − λ ρ − η (\widehat{\lambda}(\pi),\widehat{\eta}(\pi)) = \mathop{\arg\max}_{\lambda\geq0,\eta\in\mathbb{R}} \frac{1}{n}\sum_{i\in\mathcal{D}_1}-\lambda f^{\star}\left(\frac{-(2\pi(X_i)-1)\widehat{\Gamma}_i-\eta}{\lambda}\right)-\lambda\rho-\eta ( λ ( π ) , η ( π )) = arg max λ ≥ 0 , η ∈ R n 1 ∑ i ∈ D 1 − λ f ⋆ ( λ − ( 2 π ( X i ) − 1 ) Γ i − η ) − λ ρ − η .
Return π ^ = arg max π ∈ Π 1 n ∑ i ∈ D 4 − λ ^ ( π ) f ⋆ ( − ( 2 π ( X i ) − 1 ) Γ ^ i − η ^ ( π ) λ ^ ( π ) ) − λ ^ ( π ) ρ − η ^ ( π ) \widehat{\pi} = \mathop{\arg\max}_{\pi\in\Pi}\frac{1}{n}\sum_{i\in\mathcal{D}_4}-\widehat{\lambda}(\pi)f^{\star}\left(\frac{-(2\pi(X_i)-1)\widehat{\Gamma}_i-\widehat{\eta}(\pi)}{\widehat{\lambda}(\pi)}\right)-\widehat{\lambda}(\pi)\rho-\widehat{\eta}(\pi) π = arg max π ∈ Π n 1 ∑ i ∈ D 4 − λ ( π ) f ⋆ ( λ ( π ) − ( 2 π ( X i ) − 1 ) Γ i − η ( π ) ) − λ ( π ) ρ − η ( π ) .
n n n samples to estimate the advantage of a policy π \pi π . Then we first show
A K L ( π b ⋆ ) − A K L ( π ^ n ) ≤ 4 M κ n − ( β + γ ) + B ^ K L e s t ( π ^ n ) − B ^ K L e s t ( π b ⋆ ) + E [ A ^ n , K L e s t ( π b ⋆ ) ] − A ^ n , K L e s t ( π b ⋆ ) + A ^ n , K L e s t ( π ^ n ) − E [ A ^ n , K L e s t ( π ^ n ) ] . A_{\mathrm{KL}}(\pi_b^{\star})-A_{\mathrm{KL}}(\widehat{\pi}_n)
\leq\frac{4M}{\kappa}n^{-(\beta+\gamma)}+\widehat{B}_{\mathrm{KL}}^{\mathrm{est}}(\widehat{\pi}_n)-\widehat{B}_{\mathrm{KL}}^{\mathrm{est}}(\pi_b^{\star})+\mathbb{E}\left[\widehat{A}_{n,\mathrm{KL}}^{\mathrm{est}}(\pi_b^{\star})\right]-\widehat{A}_{n,\mathrm{KL}}^{\mathrm{est}}(\pi_b^{\star})+\widehat{A}_{n,\mathrm{KL}}^{\mathrm{est}}(\widehat{\pi}_n)-\mathbb{E}\left[\widehat{A}_{n,\mathrm{KL}}^{\mathrm{est}}(\widehat{\pi}_n)\right]. A KL ( π b ⋆ ) − A KL ( π n ) ≤ κ 4 M n − ( β + γ ) + B KL est ( π n ) − B KL est ( π b ⋆ ) + E [ A n , KL est ( π b ⋆ ) ] − A n , KL est ( π b ⋆ ) + A n , KL est ( π n ) − E [ A n , KL est ( π n ) ] .
Now let us define ϕ ( π ) = A ^ n , K L e s t ( π b ⋆ ) − A ^ n , K L e s t ( π ) \phi(\pi)=\widehat{A}_{n,\mathrm{KL}}^{\mathrm{est}}(\pi_b^{\star})-\widehat{A}_{n,\mathrm{KL}}^{\mathrm{est}}(\pi) ϕ ( π ) = A n , KL est ( π b ⋆ ) − A n , KL est ( π ) , deviation of finite-sample estimate advantage of policy π \pi π relative to the Bayes-optimal policy π b ⋆ \pi_b^{\star} π b ⋆ . Then we obtain the following bound.
A K L ( π b ⋆ ) − A K L ( π ^ n ) ≤ 4 M κ n − ( β + γ ) + B ^ K L e s t ( π ^ n ) − B ^ K L e s t ( π b ⋆ ) + sup π ∈ Π ∣ ϕ ( π ) − E [ ϕ ( π ) ] ∣ (7) A_{\mathrm{KL}}(\pi_b^{\star})-A_{\mathrm{KL}}(\widehat{\pi}_n)
\leq\frac{4M}{\kappa}n^{-(\beta+\gamma)}+\widehat{B}_{\mathrm{KL}}^{\mathrm{est}}(\widehat{\pi}_n)-\widehat{B}_{\mathrm{KL}}^{\mathrm{est}}(\pi_b^{\star})+\sup_{\pi\in\Pi}|\phi(\pi)-\mathbb{E}[\phi(\pi)]| \tag*{(7)} A KL ( π b ⋆ ) − A KL ( π n ) ≤ κ 4 M n − ( β + γ ) + B KL est ( π n ) − B KL est ( π b ⋆ ) + π ∈ Π sup ∣ ϕ ( π ) − E [ ϕ ( π )] ∣ (7)
We now aim is to bound the bias terms and the deviation of ϕ ( π ) \phi(\pi) ϕ ( π ) from its mean.
Step 2: Bounding the Bias Terms: We first observe that B ^ K L e s t ( π ^ n ) ≤ 0 \widehat{B}_{\mathrm{KL}}^{\mathrm{est}}(\widehat{\pi}_n)\leq0 B KL est ( π n ) ≤ 0 , although its absolute value can be Θ ( 1 / n ) \Theta(1/\sqrt{n}) Θ ( 1/ n ) . In order to bound ∣ B ^ K L e s t ( π b ⋆ ) ∣ \left|\widehat{B}_{\mathrm{KL}}^{\mathrm{est}}(\pi_b^{\star})\right| B KL est ( π b ⋆ ) , first note that we evaluate the advantage of policy π b ⋆ \pi_b^{\star} π b ⋆ using dual variables θ ^ ( π b ⋆ ) = ( λ ^ ( π b ⋆ ) , η ^ ( π b ⋆ ) ) \widehat{\theta}(\pi_b^{\star})=(\widehat{\lambda}(\pi_b^{\star}),\widehat{\eta}(\pi_b^{\star})) θ ( π b ⋆ ) = ( λ ( π b ⋆ ) , η ( π b ⋆ )) that are estimated using dataset D 1 \mathcal{D}_1 D 1 (line 7 of Algorithm 3). Lemma 11 bounds the distance between estimated and actual dual variables and proves that ∥ θ ^ ( π b ⋆ ) − θ ⋆ ( π b ⋆ ) ∥ 2 = O ~ ( 1 / n ) \left\|\widehat{\theta}(\pi_b^{\star})-\theta^{\star}(\pi_b^{\star})\right\|_2=\widetilde{O}(1/\sqrt{n}) θ ( π b ⋆ ) − θ ⋆ ( π b ⋆ ) 2 = O ( 1/ n ) under the following two assumptions.
Assumption 7 (Interior optimum). λ ⋆ ≥ λ 0 > 0 \lambda^{\star}\geq\lambda_0>0 λ ⋆ ≥ λ 0 > 0 for a known constant λ 0 \lambda_0 λ 0 .
Assumption 8 (Non-degeneracy). Var P 0 ( ∣ τ ( X ) ∣ ) ≥ σ 0 2 > 0 \operatorname{Var}_{P_0}(|\tau(X)|)\geq\sigma_0^2>0 Var P 0 ( ∣ τ ( X ) ∣ ) ≥ σ 0 2 > 0 .
The main idea is to show that the dual objective function at π b ⋆ \pi_b^{\star} π b ⋆ i.e. g λ , η ( π b ⋆ ) g_{\lambda,\eta}(\pi_b^{\star}) g λ , η ( π b ⋆ ) is strongly concave (Lemma 10) under assumptions 7 and 8, and using gradient and Hessian concentration, we can argue that the estimated dual variables do not shift by more than O ~ ( 1 / n ) \widetilde{O}(1/\sqrt{n}) O ( 1/ n ) . Having established this result, we show that the bias term ∣ B ^ K L e s t ( π b ⋆ ) ∣ \left|\widehat{B}_{\mathrm{KL}}^{\mathrm{est}}(\pi_b^{\star})\right| B KL est ( π b ⋆ ) is quadratic in the deviation ∥ θ ^ ( π b ⋆ ) − θ ⋆ ( π b ⋆ ) ∥ 2 \left\|\widehat{\theta}(\pi_b^{\star})-\theta^{\star}(\pi_b^{\star})\right\|_2 θ ( π b ⋆ ) − θ ⋆ ( π b ⋆ ) 2 and obtain ∣ B ^ K L e s t ( π b ⋆ ) ∣ ≤ O ( log n / n ) \left|\widehat{B}_{\mathrm{KL}}^{\mathrm{est}}(\pi_b^{\star})\right|\leq O(\log n/n) B KL est ( π b ⋆ ) ≤ O ( log n / n ) . Substituting the bound in Equation (7) we obtain
A K L ( π b ⋆ ) − A K L ( π ^ n ) ≤ O ( M κ ⋅ n − ( β + γ ) ) + C 1 log n n + sup π ∈ Π ∣ ϕ ( π ) − E [ ϕ ( π ) ] ∣ (8) A_{\mathrm{KL}}(\pi_b^{\star})-A_{\mathrm{KL}}(\widehat{\pi}_n)
\leq O\left(\frac{M}{\kappa}\cdot n^{-(\beta+\gamma)}\right)+C_1\frac{\log n}{n}+\sup_{\pi\in\Pi}|\phi(\pi)-\mathbb{E}[\phi(\pi)]| \tag*{(8)} A KL ( π b ⋆ ) − A KL ( π n ) ≤ O ( κ M ⋅ n − ( β + γ ) ) + C 1 n log n + π ∈ Π sup ∣ ϕ ( π ) − E [ ϕ ( π )] ∣ (8)
Step 3. Bounding Variance of ϕ ( π ) \phi(\pi) ϕ ( π ) : We want to apply Bernstein’s inequality to bound the deviation sup π ∈ Π ∣ ϕ ( π ) − E [ ϕ ( π ) ] ∣ \sup_{\pi\in\Pi}|\phi(\pi)-\mathbb{E}[\phi(\pi)]| sup π ∈ Π ∣ ϕ ( π ) − E [ ϕ ( π )] ∣ with high probability. The first step is to bound the variance i.e. Var ( ϕ ( π ) ) \operatorname{Var}(\phi(\pi)) Var ( ϕ ( π )) . Let us define D ( π ) = E [ ∣ π b ⋆ ( X ) − π ( X ) ∣ ] D(\pi)=\mathbb{E}[\left|\pi_b^{\star}(X)-\pi(X)\right|] D ( π ) = E [ ∣ π b ⋆ ( X ) − π ( X ) ∣ ] the soft disagreement between π \pi π and π b ⋆ \pi_b^{\star} π b ⋆ . Then Lemma 17 first shows that Var ( ϕ ( π ) ) ≲ E [ ∥ θ ^ ( π b ⋆ ) − θ ^ ( π ) ∥ 2 2 ] \operatorname{Var}(\phi(\pi))\lesssim\mathbb{E}[\left\|\widehat{\theta}(\pi_b^{\star})-\widehat{\theta}(\pi)\right\|_2^2] Var ( ϕ ( π )) ≲ E [ θ ( π b ⋆ ) − θ ( π ) 2 2 ] . In order to bound this distance, we need to show that the dual objective g θ ( π ) g_{\theta}(\pi) g θ ( π ) is strongly concave in θ \theta θ for all π \pi π . However, strong concavity holds only for policies that are sufficiently close to π b ⋆ \pi_b^{\star} π b ⋆ . Then using gradient and Hessian concentration we can establish that ∥ θ ^ ( π b ⋆ ) − θ ^ ( π ) ∥ 2 = O ~ ( 1 / n ) \left\|\widehat{\theta}(\pi_b^{\star})-\widehat{\theta}(\pi)\right\|_2=\widetilde{O}(1/\sqrt{n}) θ ( π b ⋆ ) − θ ( π ) 2 = O ( 1/ n ) for any policy π \pi π with D ( π ) ≤ s max : = σ 0 2 / ( 8 M 2 ) D(\pi)\leq s_{\max}:=\sigma_0^2/(8M^2) D ( π ) ≤ s m a x := σ 0 2 / ( 8 M 2 ) . This enables us to bound the variance of such a policy by O ~ ( D ( π ) + 1 / n ) \widetilde{O}(D(\pi)+1/n) O ( D ( π ) + 1/ n ) and then an application of Bernstein’s inequality gives us the following bound with probability at least 1 − 2 exp ( − u ) 1-2\exp(-u) 1 − 2 exp ( − u ) .
∣ ϕ ( π ) − E [ ϕ ( π ) ] ∣ ≤ O ( u n + D ( π ) + 1 / n n ⋅ u ) (9) |\phi(\pi)-\mathbb{E}[\phi(\pi)]| \le O\left(\frac{u}{n}+\sqrt{\frac{D(\pi)+1/n}{n}\cdot u}\right) \tag*{(9)} ∣ ϕ ( π ) − E [ ϕ ( π )] ∣ ≤ O ( n u + n D ( π ) + 1/ n ⋅ u ) (9)
Step 4. Establishing Fast Rates with a Peeling Argument: We now utilize Equation (9) and adapt the peeling argument for deterministic policies from Tsybakov [ [43 ] , Kitagawa and Tetenov [ [21 ] to establish fast rates. Let us define the following shells of policies.
Π j = { π ∈ Π : 2 j − 1 Δ < D ( π ) ≤ 2 j Δ } s j = 2 j Δ for j = 1 , 2 , … , J : = ⌈ log 2 ( 1 / Δ ) ⌉ \Pi_j=\left\{\pi\in\Pi:2^{j-1}\Delta<D(\pi)\le2^j\Delta\right\}\qquad s_j=2^j\Delta\quad\text{for }j=1,2,\ldots,J:=\left\lceil\log_2(1/\Delta)\right\rceil Π j = { π ∈ Π : 2 j − 1 Δ < D ( π ) ≤ 2 j Δ } s j = 2 j Δ for j = 1 , 2 , … , J := ⌈ log 2 ( 1/Δ ) ⌉
with Δ = O ( d log n / n ) \Delta=O(d\log n/n) Δ = O ( d log n / n ) . First, we argue that for n n n sufficiently large π ^ n \hat{\pi}_n π ^ n must lie in a shell Π j \Pi_j Π j with s j < s max = σ 0 2 / ( 8 M 2 ) s_j<s_{\max}=\sigma_0^2/(8M^2) s j < s m a x = σ 0 2 / ( 8 M 2 ) so that the strong-concavity of the dual objective holds. Now suppose π ^ n ∈ Π j ⋆ \hat{\pi}_n\in\Pi_{j^\star} π ^ n ∈ Π j ⋆ . When Assumption 4 holds, we can choose a cover of Π j \Pi_j Π j of appropriate precision to obtain the following bound with high probability.
sup π ∈ Π j ⋆ ∣ ϕ ( π ) − E [ ϕ ( π ) ] ∣ ≤ O ( ( s j ⋆ + 1 / n ) d log n n + d log n n ) \sup_{\pi\in\Pi_{j^\star}}|\phi(\pi)-\mathbb{E}[\phi(\pi)]|\le O\left(\sqrt{\frac{(s_{j^\star}+1/n)d\log n}{n}}+\frac{d\log n}{n}\right) π ∈ Π j ⋆ sup ∣ ϕ ( π ) − E [ ϕ ( π )] ∣ ≤ O ( n ( s j ⋆ + 1/ n ) d log n + n d log n )
We can effectively ignore the 1 / n 1/n 1/ n term since if s j ⋆ ≤ 1 / n s_{j^\star}\le1/n s j ⋆ ≤ 1/ n then we obtain a bound of d log n / n d\log n/n d log n / n on the regret A K L ( π b ⋆ ) − A K L ( π ) A_{\mathrm{KL}}(\pi_b^\star)-A_{\mathrm{KL}}(\pi) A KL ( π b ⋆ ) − A KL ( π ) . For s j ⋆ ≥ 1 / n s_{j^\star}\ge1/n s j ⋆ ≥ 1/ n , substituting the above bound in Equation (8) give us
O ( s j ⋆ 1 + α α ) ≤ A K L ( π b ⋆ ) − A K L ( π ^ n ) ≤ O ( M κ n − ( β + γ ) ) + C 1 d log n n + O ( s j ⋆ ⋅ d log n n ) . O\left(\frac{s_{j^\star}^{\frac{1+\alpha}{\alpha}}}{ }\right)\le A_{\mathrm{KL}}(\pi_b^\star)-A_{\mathrm{KL}}(\hat{\pi}_n)\le O\left(\frac{M}{\kappa}n^{-(\beta+\gamma)}\right)+C_1\frac{d\log n}{n}+O\left(\sqrt{\frac{s_{j^\star}\cdot d\log n}{n}}\right). O s j ⋆ α 1 + α ≤ A KL ( π b ⋆ ) − A KL ( π ^ n ) ≤ O ( κ M n − ( β + γ ) ) + C 1 n d log n + O ( n s j ⋆ ⋅ d log n ) .
The first lower bound uses Assumption 6 and Lemma 19. After rearranging, we obtain s j ⋆ = O ( ( d log n n ) α 2 + α ) s_{j^\star}=O\left(\left(\frac{d\log n}{n}\right)^{\frac{\alpha}{2+\alpha}}\right) s j ⋆ = O ( ( n d l o g n ) 2 + α α ) and substituting back the bound on s j ⋆ s_{j^\star} s j ⋆ gives us the main result on fast rate of convergence.
Theorem 3. Suppose Assumptions 1 1 1 -4 4 4 and 6 6 6 -8 8 8 hold and n ≥ C ( σ 0 2 , M , ρ , α , d , 1 / δ ) n\ge C(\sigma_0^2,M,\rho,\alpha,d,1/\delta) n ≥ C ( σ 0 2 , M , ρ , α , d , 1/ δ ) . Then with probability at least 1 − δ 1-\delta 1 − δ we have,
A K L ( π b ⋆ ) − A K L ( π ^ n ) ≤ C 1 M κ ⋅ 1 n β + γ + C ( σ 0 2 , M , ρ , α , d , 1 / δ ) ( d log n n ) 1 + α 2 + α . A_{\mathrm{KL}}(\pi_b^\star)-A_{\mathrm{KL}}(\hat{\pi}_n)\le\frac{C_1M}{\kappa}\cdot\frac{1}{n^{\beta+\gamma}}+C(\sigma_0^2,M,\rho,\alpha,d,1/\delta)\left(\frac{d\log n}{n}\right)^{\frac{1+\alpha}{2+\alpha}}. A KL ( π b ⋆ ) − A KL ( π ^ n ) ≤ κ C 1 M ⋅ n β + γ 1 + C ( σ 0 2 , M , ρ , α , d , 1/ δ ) ( n d log n ) 2 + α 1 + α .
A full proof of the theorem along with the precise values of the constants are provided in Section D. Note that as α → 0 \alpha\to0 α → 0 the second term approaches the previous rate of convergence of O ( d log n / n ) O(\sqrt{d\log n/n}) O ( d log n / n ) . However, as α → ∞ \alpha\to\infty α → ∞ , the rate of convergence approaches O ( d log n / n ) O(d\log n/n) O ( d log n / n ) .
Semi-Synthetic Experiment
Setup. We simulate observational data calibrated through the California GAIN welfare-to-work program [36 , 16 ] . In this study, welfare recipients across six counties were assigned to job-training services, with the program’s effect on subsequent employment varying substantially across counties. Because the original microdata is not publicly redistributable, we construct a semi-synthetic version with realistic feature distributions and county-specific treatment-effect heterogeneity.
Population. The training population consists of three counties (Riverside, San Diego, Tulare) and the test population of three held-out counties (Alameda, Butte, Los Angeles). Each county contributes between 400 and 1500 individuals, each described by five covariates: age (drawn from N ( μ a g e , 6 2 ) \mathcal{N}(\mu_{\mathrm{age}},6^2) N ( μ age , 6 2 ) , clipped to [ 18 , 60 ] [18,60] [ 18 , 60 ] ), years of education (drawn from N ( μ e d u , 2 2 ) \mathcal{N}(\mu_{\mathrm{edu}},2^2) N ( μ edu , 2 2 ) , clipped to [ 6 , 18 ] [6,18] [ 6 , 18 ] ), log prior earnings (drawn from N ( μ e a r n , 1 ) \mathcal{N}(\mu_{\mathrm{earn}},1) N ( μ earn , 1 ) , clipped to [ 0 , 6 ] [0,6] [ 0 , 6 ] ), and binary indicators for gender and married status. The county-specific means μ a g e , μ e d u , μ e a r n \mu_{\mathrm{age}},\mu_{\mathrm{edu}},\mu_{\mathrm{earn}} μ age , μ edu , μ earn are calibrated to reflect the demographic differences across the GAIN counties.
Treatment assignment. Within each county, treatment is assigned with propensity
e ( X ) = clip ( p c + 0.02 ⋅ ( e d u − μ e d u c ) , 0.15 , 0.85 ) , (10) e(X)=\operatorname{clip}(p_c+0.02\cdot(\mathrm{edu}-\mu_{\mathrm{edu}}^c),0.15,0.85), \tag*{(10)} e ( X ) = clip ( p c + 0.02 ⋅ ( edu − μ edu c ) , 0.15 , 0.85 ) , (10)
where p c p_c p c is the county-specific base rate. This induces mild confounding in education while preserving overlap in [ 0.15 , 0.85 ] [0.15, 0.85] [ 0.15 , 0.85 ] .
Outcome model. The binary employment outcome Y ∈ { 0 , 1 } Y \in\{0,1\} Y ∈ { 0 , 1 } is computed by first computing a baseline employment probability μ ( X ) = σ ( β 0 + X c T β b a s e ) \mu(X) = \sigma(\beta_0 + X_c^\mathsf{T}\beta_{\mathrm{base}}) μ ( X ) = σ ( β 0 + X c T β base ) , where X c X_c X c denotes the county-centered covariates, β b a s e \beta_{\mathrm{base}} β base is shared across counties, and σ ( ⋅ ) \sigma(\cdot) σ ( ⋅ ) is the logistic function. The conditional treatment effect τ ( X ) \tau(X) τ ( X ) is heterogeneous in age and education with a quadratic component in education:
τ ( X ) = τ 0 c + τ a g e c ⋅ ( a g e − μ a g e c ) + τ e d u c ⋅ ( e d u − μ e d u c ) + τ e d u 2 c ⋅ ( e d u − μ e d u c ) 2 . \tau(X) = \tau_0^c + \tau_{\mathrm{age}}^c \cdot(\mathrm{age} - \mu_{\mathrm{age}}^c) + \tau_{\mathrm{edu}}^c \cdot(\mathrm{edu} - \mu_{\mathrm{edu}}^c) + \tau_{\mathrm{edu2}}^c \cdot(\mathrm{edu} - \mu_{\mathrm{edu}}^c)^2. τ ( X ) = τ 0 c + τ age c ⋅ ( age − μ age c ) + τ edu c ⋅ ( edu − μ edu c ) + τ edu2 c ⋅ ( edu − μ edu c ) 2 .
The potential outcomes are then
μ w ( X ) = clip ( μ ( X ) + ( 2 w − 1 ) ⋅ τ ( X ) / 2 , 0.02 , 0.98 ) , w ∈ { 0 , 1 } , \mu_w(X) = \operatorname{clip}(\mu(X) + (2w - 1) \cdot\tau(X)/2, 0.02, 0.98), \quad w \in\{0,1\}, μ w ( X ) = clip ( μ ( X ) + ( 2 w − 1 ) ⋅ τ ( X ) /2 , 0.02 , 0.98 ) , w ∈ { 0 , 1 } ,
and Y ∼ Bernoulli ( μ W ( X ) ) Y \sim\operatorname{Bernoulli}(\mu_W(X)) Y ∼ Bernoulli ( μ W ( X )) .
Distributional shift. To stress-test extrapolation across counties, we deliberately reverse the sign of the education-related coefficients between training and test counties: τ e d u c > 0 \tau_{\mathrm{edu}}^c > 0 τ edu c > 0 and τ e d u 2 c > 0 \tau_{\mathrm{edu2}}^c > 0 τ edu2 c > 0 for c ∈ { Riverside, San Diego, Tulare } c \in\{\text{Riverside, San Diego, Tulare}\} c ∈ { Riverside, San Diego, Tulare } , and τ e d u c < 0 \tau_{\mathrm{edu}}^c < 0 τ edu c < 0 and τ e d u 2 c < 0 \tau_{\mathrm{edu2}}^c < 0 τ edu2 c < 0 for c ∈ { Alameda, Butte, Los Angeles } c \in\{\text{Alameda, Butte, Los Angeles}\} c ∈ { Alameda, Butte, Los Angeles } . Concretely, training counties exhibit a positive marginal effect of treatment for higher-educated individuals (treating them helps), while test counties exhibit a negative marginal effect (treating them hurts). This shift violates the implicit assumption underlying standard policy learning that the conditional treatment-effect function generalizes from training to test counties. We train our models on pooled data from the three training counties and evaluate on each test county.
Method. Recall that we aim to learn a policy which maximizes the worst-case advantage A f ( π ) = inf Q : D f ( Q ∥ P ) ≤ ρ E Q [ ( 2 π ( X ) − 1 ) τ ( X ) ] A_f(\pi) = \inf_{Q:D_f(Q\mathbin{\|} P)\leq\rho} E_Q[(2\pi(X) - 1)\tau(X)] A f ( π ) = inf Q : D f ( Q ∥ P ) ≤ ρ E Q [( 2 π ( X ) − 1 ) τ ( X )] over an f f f -divergence ball. There are three main design choices in our implementation.
(a) Cross-fitted DR scores: We use three-fold cross-fitting to build outcome estimator m ^ \hat{m} m ^ , propensity estimator e ^ \hat{e} e ^ , and then build doubly robust scores Γ ^ i \hat{\Gamma}_i Γ ^ i using Equation (5) .
(b) DR nuisance estimation: Both m ^ \hat{m} m ^ and e ^ \hat{e} e ^ are trained using distributionally robust logistic regression. We use minibatch gradient descent Levy et al. [ [25 ] to solve the corresponding DRO problems.
(c) Outer DRO problem: For the policy class Π \Pi Π , we choose decision trees of bounded depth. We use a fictitious play based method where we alternate between weighted classification for tree-fit and a worst-case update against the running average of policies.
Results. As Figure 1 highlights, distributionally robust policy learning methods, in particular CVaR and KL perform better than standard empirical risk minimizer (ERM) on training data. Moreover, on test counties e.g. Alameda, Butte, DR(CVaR) perform significantly better than ERM.
Figure 1. Out-of-Distribution Robustness on Semi-Synthetic Data: We observe that DR(CVaR) performs significantly better than standard empirical risk minimizer (ERM) on both training and testing counties.
Conclusion and Future Work
In this work we studied distributionally robust policy learning from observational data, and designed algorithms with provable rates of convergence. Algorithm 1 combines cross-fitted doubly robust scores with an inner DRO solver and attains a regret of O ( n − ( β + γ ) ) + O ~ ( d / n ) O(n^{-(\beta+\gamma)}) + \widetilde{O}(\sqrt{d/n}) O ( n − ( β + γ ) ) + O ( d / n ) simultaneously for the KL, χ 2 \chi^2 χ 2 , and CVaR ambiguity sets (Theorem 1). It inherits the doubly robust property of its non-robust counterpart Athey and Wager [ [2 ] : the O ~ ( n − 1 / 2 ) \widetilde{O}(n^{-1/2}) O ( n − 1/2 ) rate survives as long as the product of the nuisance error exponents satisfies β + γ ≥ 1 / 2 \beta+\gamma\ge1/2 β + γ ≥ 1/2 . Our main contribution is Algorithm 3, which exploits the Lagrangian dual of the inner problem to decouple the estimation of the dual variables ( λ , η ) (\lambda,\eta) ( λ , η ) from the optimization over policies. This sample-splitting bias correction removes the Θ ( n − 1 / 2 ) \Theta(n^{-1/2}) Θ ( n − 1/2 ) bias that is inherent to the plug-in DRO value [25 ] (Duchi et al. 2021) and allows us to utilize a Tsybakov-style wide-margin condition to achieve a fast rate of O ~ ( n − ( 1 + α ) / ( 2 + α ) ) \widetilde{O}(n^{-(1+\alpha)/(2+\alpha)}) O ( n − ( 1 + α ) / ( 2 + α ) ) (Theorem 3). To the best of our knowledge this is the first fast rate obtained for any distributionally robust learning problem under a margin condition.
Robustifying policy value. Although we have focused on robustifying the advantage function, our analysis can be naturally extended to robustifying the value i.e. E [ π ( X ) τ ( X ) ] \mathbb{E}[\pi(X)\tau(X)] E [ π ( X ) τ ( X )] under distributional shift. For the advantage the corresponding loss function is ℓ ( π , X ) = ( 2 π ( X ) − 1 ) τ ( X ) \ell(\pi, X) = (2\pi(X) - 1)\tau(X) ℓ ( π , X ) = ( 2 π ( X ) − 1 ) τ ( X ) and for value function it is ℓ ( π , X ) = π ( X ) τ ( X ) \ell(\pi, X) = \pi(X)\tau(X) ℓ ( π , X ) = π ( X ) τ ( X ) . Therefore for both settings, we have ∣ ℓ ( π , X ) ∣ ≤ ∣ τ ( X ) ∣ |\ell(\pi, X)| \leq|\tau(X)| ∣ ℓ ( π , X ) ∣ ≤ ∣ τ ( X ) ∣ (Boundedness) and ∣ ℓ ( π , X ) − ℓ ( π ′ , X ) ∣ ≤ 2 ∣ π ( X ) − π ′ ( X ) ∣ ∣ τ ( X ) ∣ |\ell(\pi, X) - \ell(\pi^\prime, X)| \leq2|\pi(X) - \pi^\prime(X)||\tau(X)| ∣ ℓ ( π , X ) − ℓ ( π ′ , X ) ∣ ≤ 2∣ π ( X ) − π ′ ( X ) ∣∣ τ ( X ) ∣ (Lipschitzness). This allows us to apply the same concentration and peeling arguments.
We now discuss several directions for future work and limitations of our current approach.
Beyond small radii for χ 2 \chi^2 χ 2 . Our fast rate for KL-divergence holds unconditionally in the radius ρ \rho ρ , whereas for χ 2 \chi^2 χ 2 -divergence we require ρ ≤ σ 0 / ( 4 2 M ) \rho\leq\sigma_0/(4\sqrt{2}M) ρ ≤ σ 0 / ( 4 2 M ) . Recovering a fast rate for large ρ \rho ρ or showing that the fast rate is impossible beyond a critical radius—is an interesting direction of future research.
Fast rates for CVaR. For the CVaR constraint the present approach fails. The main difficulty is that the corresponding dual objective is neither smooth nor strongly concave in any neighbourhood of π ∗ \pi^* π ∗ . Our analysis depends on that strong concavity, and without it the bias in the estimated optimal parameters need not be o ( n − 1 / 2 ) o(n^{-1/2}) o ( n − 1/2 ) . Whether a fast rate is achievable under a CVaR constraint remains an intriguing open question.
Fast rates for group robustness and other ambiguity sets. Our fast-rate analysis covers the divergence-ball based ambiguity sets and extending it to the group-robust objective of Eq. (3) appears feasible. However, the relevant margin condition must then be stated per group and the interaction between the k log k \sqrt{k\log k} k log k term and the margin exponent is not obvious. Similarly, whether the k log k \sqrt{k\log k} k log k dependence of the regret can be improved is open.
Computation and choice of ρ \rho ρ . Our bounds treat the radius ρ \rho ρ as given by the analyst. However, it encodes the set of distributions we are guarding against and this choice directly trades worst-case protection against average-case performance. A data-driven procedure for calibrating ρ \rho ρ —for instance from a small sample from the target population, or from the observed heterogeneity in a multi-site, would be valuable for practice.
Limitations. Finally, a limitation of our work is that we assume there is no unmeasured confounding in the dataset, so that the doubly robust scores are unbiased for τ ( X ) \tau(X) τ ( X ) . It is possible that the target distribution changes precisely because of changes in such unmeasured factors. Although distributionally robust methods do provide some resilience in such settings, they can be biased because the ambiguity set is centred on an incorrect estimate and the worst-case optimization will then hedge against the wrong neighbourhood. Combining our framework with a sensitivity model over the propensity score [17 ] —so that the ambiguity set covers both covariate shift and bounded confounding—is a natural remedy and one we leave to future work. We also assume bounded outcomes and strict overlap (Assumption 1, Assumption 3); and removing these assumptions is an interesting direction for future work. Finally, our empirical evaluation is semi-synthetic, since the GAIN microdata is not publicly redistributable; validating the practical benefit of distributional robustness on real multi-site policy data remains an important next step.
Appendix
Additional Experimental Details
Semi-synthetic DGP parameter values
This appendix lists the exact parameter values used to generate the semi-synthetic California GAIN-like dataset described in § 6 . All values are fixed across simulation seeds; randomness enters only through the sampling of covariates, treatment assignments, and binary outcomes.
County-level demographic and propensity parameters
For each county c c c , individual covariates ( X 1 , … , X 5 ) = ( age , edu , log_earn , female , married ) (X_1,\ldots,X_5) = (\text{age},\text{edu},\text{log\_earn},\text{female},\text{married}) ( X 1 , … , X 5 ) = ( age , edu , log_earn , female , married ) are drawn independently with the means listed in Table 1 ; the standard deviations are common across counties: σ a g e = 6 \sigma_{\mathrm{age}} = 6 σ age = 6 , σ e d u = 2 \sigma_{\mathrm{edu}} = 2 σ edu = 2 , σ e a r n = 1 \sigma_{\mathrm{earn}} = 1 σ earn = 1 . Truncation bounds are age ∈ [ 18 , 60 ] \text{age} \in[18, 60] age ∈ [ 18 , 60 ] , edu ∈ [ 6 , 18 ] \text{edu} \in[6, 18] edu ∈ [ 6 , 18 ] , log_earn ∈ [ 0 , 6 ] \text{log\_earn} \in[0, 6] log_earn ∈ [ 0 , 6 ] . Binary indicators female, married are Bernoulli with shared probabilities 0.55 and 0.40 respectively. Treatment is assigned with propensity e ( X ) = clip ( p c + 0.02 ⋅ ( edu − μ e d u c ) , 0.15 , 0.85 ) e(X) = \operatorname{clip}(p_c + 0.02 \cdot(\text{edu} - \mu_{\mathrm{edu}}^{c}), 0.15, 0.85) e ( X ) = clip ( p c + 0.02 ⋅ ( edu − μ edu c ) , 0.15 , 0.85 ) .
Table 1. County-level demographic and propensity parameters.
Baseline outcome model
The county-shared baseline employment probability is μ ( X ) = σ ( β 0 + X c T β b a s e ) \mu(X) = \sigma(\beta_0 + X_c^{\mathsf{T}}\beta_{\mathrm{base}}) μ ( X ) = σ ( β 0 + X c T β base ) , where X c X_c X c denotes the centered covariates ( age − μ a g e c , edu − μ e d u c , log_earn − μ e a r n c , female , married ) (\text{age} - \mu_{\mathrm{age}}^{c}, \text{edu} - \mu_{\mathrm{edu}}^{c}, \text{log\_earn} - \mu_{\mathrm{earn}}^{c}, \text{female}, \text{married}) ( age − μ age c , edu − μ edu c , log_earn − μ earn c , female , married ) and σ \sigma σ is the logistic function. The intercept and coefficients are
β 0 = − 0.10 , β b a s e = ( − 0.02 , 0.18 , 0.10 , 0.05 , 0.20 ) , \beta_0 = -0.10,\qquad\beta_{\mathrm{base}} = (-0.02,\ 0.18,\ 0.10,\ 0.05,\ 0.20), β 0 = − 0.10 , β base = ( − 0.02 , 0.18 , 0.10 , 0.05 , 0.20 ) ,
where the components of β b a s e \beta_{\mathrm{base}} β base correspond to age, education, log prior earnings, female, and married, in that order. These coefficients are shared across all counties, both training and test.
County-specific treatment-effect coefficients
The conditional treatment effect is
τ c ( X ) = τ 0 c + τ a g e c ⋅ ( a g e − μ a g e c ) + τ e d u c ⋅ ( e d u − μ e d u c ) + τ e d u 2 c ⋅ ( e d u − μ e d u c ) 2 . \tau^c(X) = \tau^c_0 + \tau^c_{\mathrm{age}} \cdot(\mathrm{age} - \mu^c_{\mathrm{age}}) + \tau^c_{\mathrm{edu}} \cdot(\mathrm{edu} - \mu^c_{\mathrm{edu}}) + \tau^c_{\mathrm{edu2}} \cdot(\mathrm{edu} - \mu^c_{\mathrm{edu}})^2. τ c ( X ) = τ 0 c + τ age c ⋅ ( age − μ age c ) + τ edu c ⋅ ( edu − μ edu c ) + τ edu2 c ⋅ ( edu − μ edu c ) 2 .
The training and test counties differ in the sign of the education-related coefficients τ e d u c \tau^c_{\mathrm{edu}} τ edu c and τ e d u 2 c \tau^c_{\mathrm{edu2}} τ edu2 c (positive in training, negative in test); the intercepts τ 0 c \tau^c_0 τ 0 c are mostly positive but smaller in test counties, with one slightly negative. This is the structural shift discussed in the main text.
Table 2. County-specific treatment-effect coefficients.
Details of Methodology
Outcome-generation step
Given baseline μ ( X ) \mu(X) μ ( X ) and treatment effect τ c ( X ) \tau^c(X) τ c ( X ) , the potential-outcome probabilities are clipped to [ 0.02 , 0.98 ] [0.02, 0.98] [ 0.02 , 0.98 ] :
μ w c ( X ) = clip ( μ ( X ) + ( 2 w − 1 ) ⋅ τ c ( X ) / 2 , 0.02 , 0.98 ) , w ∈ { 0 , 1 } , \mu^c_w(X) = \operatorname{clip}\left(\mu(X) + (2w - 1) \cdot\tau^c(X)/2,\ 0.02,\ 0.98\right), \qquad w \in\{0,1\}, μ w c ( X ) = clip ( μ ( X ) + ( 2 w − 1 ) ⋅ τ c ( X ) /2 , 0.02 , 0.98 ) , w ∈ { 0 , 1 } ,
and Y ∼ Bernoulli ( μ W c ( X ) ) Y \sim\operatorname{Bernoulli}(\mu^c_W(X)) Y ∼ Bernoulli ( μ W c ( X )) .
Algorithm and evaluation parameters
Unless otherwise stated, the experiments use ambiguity radius ρ = 0.4 \rho= 0.4 ρ = 0.4 for KL and χ 2 \chi^2 χ 2 , level α = 0.3 \alpha= 0.3 α = 0.3 for CVaR, decision-tree policies of depth 2 with 10 fictitious-play iterations, and DR-logistic nuisance estimators trained for 20 epochs at batch size 128, learning rate 0.05, and ℓ 2 \ell_2 ℓ 2 penalty 10 − 3 10^{-3} 1 0 − 3 . All results are averaged over 12 independent seeds, with 95% confidence intervals computed as ± 1.96 × S E \pm1.96 \times\mathrm{SE} ± 1.96 × SE .
We aim to learn a policy π ∈ Π \pi\in\Pi π ∈ Π maximising the worst-case policy value
A f ( π ) = inf Q : D f ( Q ∥ P ) ≤ ρ E Q [ ( 2 π ( X ) − 1 ) τ ( X ) ] A_f(\pi) = \inf_{Q : D_f(Q \mathbin\Vert P) \leq\rho} \mathbb{E}_Q[(2\pi(X) - 1)\tau(X)] A f ( π ) = Q : D f ( Q ∥ P ) ≤ ρ inf E Q [( 2 π ( X ) − 1 ) τ ( X )]
over an f f f -divergence ambiguity ball of radius ρ \rho ρ around the data-generating distribution P P P . Our estimator is built on three components: cross-fitted doubly-robust scores for the conditional treatment effect, distributionally-robust nuisance estimators trained via a primal minibatch scheme, and an outer fictitious-play loop over decision-tree policies. We describe each in turn.
Cross-fitted doubly-robust scores. The treatment effect τ ( x ) = E [ Y ( 1 ) − Y ( 0 ) ∣ X = x ] \tau(x) = \mathbb{E}[Y(1) - Y(0) \mid X = x] τ ( x ) = E [ Y ( 1 ) − Y ( 0 ) ∣ X = x ] is estimated through the doubly-robust score
Γ ^ i = m ^ ( X i , 1 ) − m ^ ( X i , 0 ) + W i − e ^ ( X i ) e ^ ( X i ) ( 1 − e ^ ( X i ) ) ( Y i − m ^ ( X i , W i ) ) , (11) \widehat{\Gamma}_i = \widehat{m}(X_i,1) - \widehat{m}(X_i,0) + \frac{W_i - \widehat{e}(X_i)}{\widehat{e}(X_i)(1 - \widehat{e}(X_i))}\left(Y_i - \widehat{m}(X_i,W_i)\right), \tag*{(11)} Γ i = m ( X i , 1 ) − m ( X i , 0 ) + e ( X i ) ( 1 − e ( X i )) W i − e ( X i ) ( Y i − m ( X i , W i ) ) , (11)
where m ^ ( x , w ) = E ^ [ Y ∣ X = x , W = w ] \widehat{m}(x,w)=\widehat{\mathbb{E}}[Y\mid X=x,W=w] m ( x , w ) = E [ Y ∣ X = x , W = w ] is the conditional outcome model and e ^ ( x ) = P ^ ( W = 1 ∣ X = x ) \widehat{e}(x)=\widehat{\mathbb{P}}(W=1\mid X=x) e ( x ) = P ( W = 1 ∣ X = x ) is the propensity score. To eliminate own-observation bias we use three-fold cross-fitting: each sample i i i receives a Γ ^ i \widehat{\Gamma}_i Γ i computed from m ^ , e ^ \widehat{m},\widehat{e} m , e trained on the two folds not containing i i i . The propensity is clipped to [ 0.05 , 0.95 ] [0.05,0.95] [ 0.05 , 0.95 ] to guarantee overlap. The score (10) is unbiased for τ ( X i ) \tau(X_i) τ ( X i ) whenever either m ^ \widehat{m} m or e ^ \widehat{e} e is consistent, and its product nuisance error governs the rate at which Γ ^ \widehat{\Gamma} Γ approaches τ \tau τ .
Distributionally-robust nuisance estimation. To make m ^ \widehat{m} m and e ^ \widehat{e} e themselves robust to local distributional perturbations, we train each as a binary logistic regression under the same DRO objective:
min θ sup Q : D f ( Q ∥ P ) ≤ ρ E Q [ ℓ ( θ ; Z ) ] , \min_{\theta}\sup_{Q:D_f(Q\mathbin\Vert P)\leq\rho}\mathbb{E}_Q[\ell(\theta;Z)], θ min Q : D f ( Q ∥ P ) ≤ ρ sup E Q [ ℓ ( θ ; Z )] ,
with ℓ \ell ℓ the binary cross-entropy loss and Z = ( X , Y ) Z=(X,Y) Z = ( X , Y ) for the outcome model and Z = ( X , W ) Z=(X,W) Z = ( X , W ) for the propensity. We instantiate this for KL, χ 2 \chi^2 χ 2 , and CVaR ambiguity sets. All three are solved using the minibatch gradient descent method described next.
Primal minibatch SGD with per-batch worst-case reweighting. Following the spirit of Levy et al. [ [25 ] but in primal form, we optimise the DRO objective by alternating exact inner maximisation and primal gradient descent. At each iteration we draw a minibatch B = { i 1 , … , i b } \mathcal{B}=\{i_1,\ldots,i_b\} B = { i 1 , … , i b } , compute per-sample losses ℓ j = ℓ ( θ ; Z i j ) \ell_j=\ell(\theta;Z_{i_j}) ℓ j = ℓ ( θ ; Z i j ) , and solve
q ⋆ = arg max q ∈ Δ ( b ) : 1 b ∑ j = 1 b f ( b q j ) ≤ ρ ∑ j = 1 b q j ℓ j , (12) q^\star=\mathop{\arg\max}_{q\in\Delta(b):\frac{1}{b}\sum_{j=1}^{b}f(bq_j)\leq\rho}\sum_{j=1}^{b}q_j\ell_j, \tag*{(12)} q ⋆ = arg max q ∈ Δ ( b ) : b 1 ∑ j = 1 b f ( b q j ) ≤ ρ j = 1 ∑ b q j ℓ j , (12)
which has a closed-form or one-dimensional dual solution for each of the three divergences (sort-based for CVaR, scalar root-finding for KL, nested 1D for χ 2 \chi^2 χ 2 ). We then take a primal step
θ ← θ − η l r ( ∑ j = 1 b q j ⋆ ∇ ℓ j + λ 2 θ ) . \theta\leftarrow\theta-\eta_{\mathrm{lr}}\left(\sum_{j=1}^{b}q_j^\star\nabla\ell_j+\lambda_2\theta\right). θ ← θ − η lr ( j = 1 ∑ b q j ⋆ ∇ ℓ j + λ 2 θ ) .
Because the inner problem (12) is convex on a small simplex and solved exactly per minibatch, this scheme sidesteps the bias of dual SGD without requiring auxiliary dual variables to be tracked across iterations.
Outer policy optimisation via fictitious play. Given the cross-fitted scores { Γ ^ i } i = 1 n \{\widehat{\Gamma}_i\}_{i=1}^{n} { Γ i } i = 1 n , the policy-learning problem is the maximin problem
max π ∈ Π inf Q : D f ( Q ∥ P ^ n ) ≤ ρ 1 n ∑ i = 1 n q i ( 2 π ( X i ) − 1 ) Γ ^ i , (13) \max_{\pi\in\Pi}\inf_{Q:D_f(Q\mathbin\Vert\widehat{P}_n)\leq\rho}\frac{1}{n}\sum_{i=1}^{n}q_i\bigl(2\pi(X_i)-1\bigr)\widehat{\Gamma}_i, \tag*{(13)} π ∈ Π max Q : D f ( Q ∥ P n ) ≤ ρ inf n 1 i = 1 ∑ n q i ( 2 π ( X i ) − 1 ) Γ i , (13)
We choose Π \Pi Π as the class of decision trees of bounded depth. Exact maximisation over Π \Pi Π is NP-hard for depth at least two and we approximate (13) by fictitious play. Starting from Q 0 = P ^ n Q^0=\widehat{P}_n Q 0 = P n , at each iteration t t t we (i) compute an approximate best-response policy π t \pi^t π t by fitting a classification tree to labels y i = 1 { Γ ^ i ≥ 0 } y_i=1\{\widehat{\Gamma}_i\geq0\} y i = 1 { Γ i ≥ 0 } with weights w i = n q i t ∣ Γ ^ i ∣ w_i=nq_i^t|\widehat{\Gamma}_i| w i = n q i t ∣ Γ i ∣ , exploiting the identity ( 2 π ( X i ) − 1 ) Γ ^ i = ∣ Γ ^ i ∣ ( 2 ⋅ 1 { π ( X i ) = y i } − 1 ) (2\pi(X_i)-1)\widehat{\Gamma}_i=|\widehat{\Gamma}_i|(2\cdot1\{\pi(X_i)=y_i\}-1) ( 2 π ( X i ) − 1 ) Γ i = ∣ Γ i ∣ ( 2 ⋅ 1 { π ( X i ) = y i } − 1 ) ; and (ii) update q t + 1 q^{t+1} q t + 1 to the worst case of (12) against the running average 1 t ∑ s = 1 t ( 2 π s ( X ) − 1 ) Γ ^ \frac{1}{t}\sum_{s=1}^{t}(2\pi^s(X)-1)\widehat{\Gamma} t 1 ∑ s = 1 t ( 2 π s ( X ) − 1 ) Γ . Among the iterates { π t } t = 1 T \{\pi^t\}_{t=1}^{T} { π t } t = 1 T we return the one with the largest empirical DRO objective. The running-average update is what makes the procedure fictitious play rather than alternating maximisation: the inner player best-responds to the empirical mixture of past policies, which lies in the convex hull of Π \Pi Π and admits a saddle-point characterisation even though Π \Pi Π itself is non-convex.
Proof of thm:main-bound-dr
Let us recall a few notations. We will write P ^ n \widehat{P}_n P n to denote the empirical distribution supported over { X 1 , … , X n } \{X_1,\ldots,X_n\} { X 1 , … , X n } .
(a) A f ( π ) = min Q : D f ( Q ∥ P 0 ) ≤ ρ E X ∼ Q [ ( 2 π ( X ) − 1 ) τ ( X ) ] A_f(\pi)=\min_{Q:D_f(Q\mathbin\Vert P_0)\leq\rho}\mathbb{E}_{X\sim Q}[(2\pi(X)-1)\tau(X)] A f ( π ) = min Q : D f ( Q ∥ P 0 ) ≤ ρ E X ∼ Q [( 2 π ( X ) − 1 ) τ ( X )] is the (distributionally robust) population advantage of a policy π \pi π . (b) A ^ n , f ( π ) = min Q : D f ( Q ∥ P ^ n ) ≤ ρ ∑ i = 1 n q i ⋅ ( 2 π ( X i ) − 1 ) τ ( X i ) \widehat{A}_{n,f}(\pi)=\min_{Q:D_f(Q\|\widehat{P}_n)\le\rho}\sum_{i=1}^{n}q_i\cdot(2\pi(X_i)-1)\tau(X_i) A n , f ( π ) = min Q : D f ( Q ∥ P n ) ≤ ρ ∑ i = 1 n q i ⋅ ( 2 π ( X i ) − 1 ) τ ( X i ) is the (distributionally robust) empirical advantage of a policy π \pi π on a finite sample { X 1 , … , X n } \{X_1,\ldots,X_n\} { X 1 , … , X n } . Observe that it requires knowledge of τ ( X i ) \tau(X_i) τ ( X i ) which is unknown and A ^ n , f ( π ) \widehat{A}_{n,f}(\pi) A n , f ( π ) cannot be computed exactly.
(c) A ^ n , f e s t ( π ) = min Q : D f ( Q ∥ P ^ n ) ≤ ρ ∑ i = 1 n q i ⋅ ( 2 π ( X i ) − 1 ) Γ ^ i \widehat{A}^{\mathrm{est}}_{n,f}(\pi)=\min_{Q:D_f(Q\|\widehat{P}_n)\le\rho}\sum_{i=1}^{n}q_i\cdot(2\pi(X_i)-1)\widehat{\Gamma}_i A n , f est ( π ) = min Q : D f ( Q ∥ P n ) ≤ ρ ∑ i = 1 n q i ⋅ ( 2 π ( X i ) − 1 ) Γ i is the (distributionally robust) empirical advantage of policy π \pi π evaluated using the estimates { Γ ^ i : i ∈ [ n ] } \{\widehat{\Gamma}_i:i\in[n]\} { Γ i : i ∈ [ n ]} of treatment effects.
(d) A ~ f e s t ( π ) = min Q : D f ( Q ∥ P 0 ) ≤ ρ E X ∼ Q , W ∼ π o f f ( ⋅ ∣ X ) [ ( 2 π ( X ) − 1 ) Γ ^ ( X , W ) ] \widetilde{A}^{\mathrm{est}}_f(\pi)=\min_{Q:D_f(Q\|P_0)\le\rho}\mathbb{E}_{X\sim Q,W\sim\pi_{\mathrm{off}}(\cdot\mid X)}\left[(2\pi(X)-1)\widehat{\Gamma}(X,W)\right] A f est ( π ) = min Q : D f ( Q ∥ P 0 ) ≤ ρ E X ∼ Q , W ∼ π off ( ⋅ ∣ X ) [ ( 2 π ( X ) − 1 ) Γ ( X , W ) ] is the (distributionally robust) population advantage of policy π \pi π when using estimates Γ ^ ( X , W ) \widehat{\Gamma}(X,W) Γ ( X , W ) instead of the actual treatment effect τ ( X ) \tau(X) τ ( X ) .
Proof of Theorem 1. Recall that π ^ n ∈ arg max π A ^ n , f e s t ( π ) \widehat{\pi}_n\in\arg\max_{\pi}\widehat{A}^{\mathrm{est}}_{n,f}(\pi) π n ∈ arg max π A n , f est ( π ) and π ⋆ ∈ arg max π A f ( π ) \pi^\star\in\arg\max_{\pi}A_f(\pi) π ⋆ ∈ arg max π A f ( π ) .
A f ( π ) − A f ( π ^ n ) = A f ( π ) − A ~ f e s t ( π ) − A f ( π ^ n ) + A ~ f e s t ( π ^ n ) + A ~ f e s t ( π ) − A ^ n , f e s t ( π ) − A ~ f e s t ( π ^ n ) + A ^ n , f e s t ( π ^ n ) + A ^ n , f e s t ( π ) − A ^ n , f e s t ( π ^ n ) (14) \begin{aligned}
A_f(\pi)-A_f(\widehat{\pi}_n)
={}&A_f(\pi)-\widetilde{A}^{\mathrm{est}}_f(\pi)-A_f(\widehat{\pi}_n)+\widetilde{A}^{\mathrm{est}}_f(\widehat{\pi}_n)\\
&+\widetilde{A}^{\mathrm{est}}_f(\pi)-\widehat{A}^{\mathrm{est}}_{n,f}(\pi)-\widetilde{A}^{\mathrm{est}}_f(\widehat{\pi}_n)+\widehat{A}^{\mathrm{est}}_{n,f}(\widehat{\pi}_n)\\
&+\widehat{A}^{\mathrm{est}}_{n,f}(\pi)-\widehat{A}^{\mathrm{est}}_{n,f}(\widehat{\pi}_n) \tag*{(14)}
\end{aligned} A f ( π ) − A f ( π n ) = A f ( π ) − A f est ( π ) − A f ( π n ) + A f est ( π n ) + A f est ( π ) − A n , f est ( π ) − A f est ( π n ) + A n , f est ( π n ) + A n , f est ( π ) − A n , f est ( π n ) (14)
The third term is non-positive since π ^ n \widehat{\pi}_n π n maximizes A ^ n , f e s t ( ⋅ ) \widehat{A}^{\mathrm{est}}_{n,f}(\cdot) A n , f est ( ⋅ ) . Taking supremum over all policies π ∈ Π \pi\in\Pi π ∈ Π we obtain the following upper bound.
sup π ∈ Π A f ( π ) − A f ( π ^ n ) ≤ 2 sup π ∈ Π ∣ A f ( π ) − A ~ f e s t ( π ) ∣ + 2 sup π ∈ Π ∣ A ~ f e s t ( π ) − A ^ n , f e s t ( π ) ∣ \sup_{\pi\in\Pi} A_f(\pi)-A_f(\widehat{\pi}_n)\le2\sup_{\pi\in\Pi}\left|A_f(\pi)-\widetilde{A}^{\mathrm{est}}_f(\pi)\right|+2\sup_{\pi\in\Pi}\left|\widetilde{A}^{\mathrm{est}}_f(\pi)-\widehat{A}^{\mathrm{est}}_{n,f}(\pi)\right| π ∈ Π sup A f ( π ) − A f ( π n ) ≤ 2 π ∈ Π sup A f ( π ) − A f est ( π ) + 2 π ∈ Π sup A f est ( π ) − A n , f est ( π )
Lemma 4 bounds the term ∣ A f ( π ) − A ~ f e s t ( π ) ∣ \left|A_f(\pi)-\widetilde{A}^{\mathrm{est}}_f(\pi)\right| A f ( π ) − A f est ( π ) by 2 M κ n − ( β + γ ) \frac{2M}{\kappa}n^{-(\beta+\gamma)} κ 2 M n − ( β + γ ) . The second term sup π ∈ Π ∣ A ~ f e s t ( π ) − A ^ n , f e s t ( π ) ∣ \sup_{\pi\in\Pi}\left|\widetilde{A}^{\mathrm{est}}_f(\pi)-\widehat{A}^{\mathrm{est}}_{n,f}(\pi)\right| sup π ∈ Π A f est ( π ) − A n , f est ( π ) by Lemma 3 with high probability. □
Bound on the Bias of the Batch Estimator
Lemma 1 (Bias of batch estimator). Suppose assumption 1 holds. Then, for any π ∈ Π \pi\in\Pi π ∈ Π and n n n we have,
E [ A ^ n , f ( π ) ] − A f ( π ) ≤ { M min { 1 , 1 α n } if A f = A C V a R 2 M 1 + 2 ρ n if A f = A χ 2 M ρ n if A f = A K L \mathbb{E}\left[\widehat{A}_{n,f}(\pi)\right]-A_f(\pi)\le
\begin{cases}
M\min\left\{1,\frac{1}{\sqrt{\alpha n}}\right\} & \text{if } A_f=A_{\mathrm{CVaR}}\\
2M\sqrt{1+\frac{2\rho}{\sqrt{n}}} & \text{if } A_f=A_{\chi^2}\\
\frac{M\rho}{\sqrt{n}} & \text{if } A_f=A_{\mathrm{KL}}
\end{cases} E [ A n , f ( π ) ] − A f ( π ) ≤ ⎩ ⎨ ⎧ M min { 1 , α n 1 } 2 M 1 + n 2 ρ n M ρ if A f = A CVaR if A f = A χ 2 if A f = A KL
Proof. For CVaR constraint at level α \alpha α , f ( t ) = 1 { t ≤ 1 / α } f(t)=\mathbf{1}_{\{t\le1/\alpha\}} f ( t ) = 1 { t ≤ 1/ α } , we can apply proposition 1 of Levy et al. [ [25 ] with ℓ ( π ; X ) = − ( 2 π ( X ) − 1 ) τ ( X ) \ell(\pi;X)=-(2\pi(X)-1)\tau(X) ℓ ( π ; X ) = − ( 2 π ( X ) − 1 ) τ ( X ) and obtain the following inequality.
− A C V a R ( π ) + E [ A ^ n , C V a R ( π ) ] ≤ b b ( α ) = M min { 1 , 1 α n } -A_{\mathrm{CVaR}}(\pi)+\mathbb{E}\left[\widehat{A}_{n,\mathrm{CVaR}}(\pi)\right]\le\mathrm{bb}(\alpha)=M\min\left\{1,\frac{1}{\sqrt{\alpha n}}\right\} − A CVaR ( π ) + E [ A n , CVaR ( π ) ] ≤ bb ( α ) = M min { 1 , α n 1 }
For χ 2 \chi^2 χ 2 -divergence, we have f ( t ) = 1 / 2 ( t − 1 ) 2 f(t)=1/2(t-1)^2 f ( t ) = 1/2 ( t − 1 ) 2 . We will modify the argument presented by Levy et al. [ [25 ] (proposition 1). First, proceeding similarly as their proof we can establish the following inequality.
− A χ 2 ( π ) + E [ A ^ n , χ 2 ( π ) ] ≤ ∫ 0 1 [ r ( α ) − r ( 1 ) ] ( α ⋅ b b ( α ) ) ′ d α -A_{\chi^2}(\pi)+\mathbb{E}\left[\widehat{A}_{n,\chi^2}(\pi)\right]\le\int_0^1[r(\alpha)-r(1)](\alpha\cdot\mathrm{bb}(\alpha))'\,d\alpha − A χ 2 ( π ) + E [ A n , χ 2 ( π ) ] ≤ ∫ 0 1 [ r ( α ) − r ( 1 )] ( α ⋅ bb ( α ) ) ′ d α
Here r : [ 0 , 1 ] → R + r:[0,1]\to\mathbb{R}^{+} r : [ 0 , 1 ] → R + is a non-increasing function satisfying ∥ r ∥ 1 = 1 \lVert r\rVert_{1}=1 ∥ r ∥ 1 = 1 and ∥ r ∥ 2 2 ≤ 1 + 2 ρ \lVert r\rVert_{2}^{2}\le1+2\rho ∥ r ∥ 2 2 ≤ 1 + 2 ρ . Now we use ( α ⋅ b b ( α ) ) ′ = b b ( 0 ) ⋅ 1 { α ≤ 1 / n } + 1 2 b b ( α ) ⋅ 1 { α > 1 / n } (\alpha\cdot\mathrm{bb}(\alpha))'=\mathrm{bb}(0)\cdot\mathbf{1}_{\{\alpha\le1/n\}}+\frac{1}{2}\mathrm{bb}(\alpha)\cdot\mathbf{1}_{\{\alpha>1/n\}} ( α ⋅ bb ( α ) ) ′ = bb ( 0 ) ⋅ 1 { α ≤ 1/ n } + 2 1 bb ( α ) ⋅ 1 { α > 1/ n } .
− A χ 2 ( π ) + E [ A ^ n , χ 2 ( π ) ] ≤ ∫ 0 1 / n ( r ( α ) − r ( 1 ) ) ⋅ b b ( 0 ) d α + 1 2 ∫ 1 / n 1 ( r ( α ) − r ( 1 ) ) b b ( α ) d α -A_{\chi^2}(\pi)+\mathbb{E}\left[\widehat{A}_{n,\chi^2}(\pi)\right]\le\int_{0}^{1/n}(r(\alpha)-r(1))\cdot\mathrm{bb}(0)\,\mathrm{d}\alpha+\frac{1}{2}\int_{1/n}^{1}(r(\alpha)-r(1))\mathrm{bb}(\alpha)\,\mathrm{d}\alpha − A χ 2 ( π ) + E [ A n , χ 2 ( π ) ] ≤ ∫ 0 1/ n ( r ( α ) − r ( 1 )) ⋅ bb ( 0 ) d α + 2 1 ∫ 1/ n 1 ( r ( α ) − r ( 1 )) bb ( α ) d α
≤ M ∫ 0 1 / n r ( α ) d α + 1 2 ∫ 1 / n 1 r ( α ) ⋅ b b ( α ) d α ≤ M ( ∫ 0 1 / n r 2 ( α ) d α ) 1 / 2 ( ∫ 0 1 / n 1 d α ) 1 / 2 + 1 2 ( ∫ 1 / n 1 r 2 ( α ) ⋅ b b ( α ) d α ) 1 / 2 ( ∫ 1 / n 1 b b ( α ) d α ) 1 / 2 ≤ M ∥ r ∥ 2 1 n + M 2 ∥ r ∥ 2 ( ∫ 1 / n 1 1 α n d α ) 1 / 2 ≤ 2 M 1 + 2 ρ n . \begin{aligned}
&\le M\int_{0}^{1/n}r(\alpha)\,\mathrm{d}\alpha+\frac{1}{2}\int_{1/n}^{1}r(\alpha)\cdot\mathrm{bb}(\alpha)\,\mathrm{d}\alpha\\
&\le M\left(\int_{0}^{1/n}r^{2}(\alpha)\,\mathrm{d}\alpha\right)^{1/2}\left(\int_{0}^{1/n}1\,\mathrm{d}\alpha\right)^{1/2}\\
&\quad+\frac{1}{2}\left(\int_{1/n}^{1}r^{2}(\alpha)\cdot\mathrm{bb}(\alpha)\,\mathrm{d}\alpha\right)^{1/2}\left(\int_{1/n}^{1}\mathrm{bb}(\alpha)\,\mathrm{d}\alpha\right)^{1/2}\\
&\le M\lVert r\rVert_{2}\frac{1}{\sqrt{n}}+\frac{\sqrt{M}}{2}\lVert r\rVert_{2}\left(\int_{1/n}^{1}\frac{1}{\sqrt{\alpha n}}\,\mathrm{d}\alpha\right)^{1/2}\\
&\le\frac{2M\sqrt{1+2\rho}}{\sqrt{n}}.
\end{aligned} ≤ M ∫ 0 1/ n r ( α ) d α + 2 1 ∫ 1/ n 1 r ( α ) ⋅ bb ( α ) d α ≤ M ( ∫ 0 1/ n r 2 ( α ) d α ) 1/2 ( ∫ 0 1/ n 1 d α ) 1/2 + 2 1 ( ∫ 1/ n 1 r 2 ( α ) ⋅ bb ( α ) d α ) 1/2 ( ∫ 1/ n 1 bb ( α ) d α ) 1/2 ≤ M ∥ r ∥ 2 n 1 + 2 M ∥ r ∥ 2 ( ∫ 1/ n 1 α n 1 d α ) 1/2 ≤ n 2 M 1 + 2 ρ .
Therefore, we have the following bound for χ 2 \chi^2 χ 2 -divergence.5
− A χ 2 ( π ) + E [ A ^ n , χ 2 ( π ) ] ≤ 2 M 1 + 2 ρ n -A_{\chi^2}(\pi)+\mathbb{E}\left[\widehat{A}_{n,\chi^2}(\pi)\right]\le\frac{2M\sqrt{1+2\rho}}{\sqrt{n}} − A χ 2 ( π ) + E [ A n , χ 2 ( π ) ] ≤ n 2 M 1 + 2 ρ
We now consider the KL-divergence, for which we have f ( t ) = t log t f(t)=t\log t f ( t ) = t log t . Proceeding similarly as χ 2 \chi^2 χ 2 -divergence, we first obtain the following inequality.
− A K L ( π ) + E [ A ^ n , K L ( π ) ] ≤ ∫ 0 1 / n ( r ( α ) − r ( 1 ) ) ⋅ b b ( 0 ) d α + 1 2 ∫ 1 / n 1 ( r ( α ) − r ( 1 ) ) b b ( α ) d α -A_{\mathrm{KL}}(\pi)+\mathbb{E}\left[\widehat{A}_{n,\mathrm{KL}}(\pi)\right]\le\int_{0}^{1/n}(r(\alpha)-r(1))\cdot\mathrm{bb}(0)\,\mathrm{d}\alpha+\frac{1}{2}\int_{1/n}^{1}(r(\alpha)-r(1))\mathrm{bb}(\alpha)\,\mathrm{d}\alpha − A KL ( π ) + E [ A n , KL ( π ) ] ≤ ∫ 0 1/ n ( r ( α ) − r ( 1 )) ⋅ bb ( 0 ) d α + 2 1 ∫ 1/ n 1 ( r ( α ) − r ( 1 )) bb ( α ) d α
We are also given ∥ r ∥ 1 = 1 \lVert r\rVert_{1}=1 ∥ r ∥ 1 = 1 and ∫ 0 1 r ( α ) log r ( α ) d α ≤ ρ \int_{0}^{1}r(\alpha)\log r(\alpha)\,\mathrm{d}\alpha\le\rho ∫ 0 1 r ( α ) log r ( α ) d α ≤ ρ .
− A K L ( π ) + E [ V ^ n , K L ( π ) ] ≤ ∫ 0 1 / n r ( α ) ⋅ b b ( 0 ) d α + 1 2 ∫ 1 / n 1 r ( α ) ⋅ b b ( α ) d α ≤ M ∫ 0 1 r ( α ) α n d α = M n ∫ 0 1 r ( α ) log ( e − 1 / α Z ) d α + log Z ⋅ M n ∫ 0 1 r ( α ) d α . \begin{aligned}
-A_{\mathrm{KL}}(\pi)+\mathbb{E}\left[\widehat{V}_{n,\mathrm{KL}}(\pi)\right]&\le\int_{0}^{1/n}r(\alpha)\cdot\mathrm{bb}(0)\,\mathrm{d}\alpha+\frac{1}{2}\int_{1/n}^{1}r(\alpha)\cdot\mathrm{bb}(\alpha)\,\mathrm{d}\alpha\\
&\le M\int_{0}^{1}\frac{r(\alpha)}{\sqrt{\alpha n}}\,\mathrm{d}\alpha\\
&=\frac{M}{\sqrt{n}}\int_{0}^{1}r(\alpha)\log\left(\frac{e^{-1/\sqrt{\alpha}}}{Z}\right)\,\mathrm{d}\alpha+\log Z\cdot\frac{M}{\sqrt{n}}\int_{0}^{1}r(\alpha)\,\mathrm{d}\alpha.
\end{aligned} − A KL ( π ) + E [ V n , KL ( π ) ] ≤ ∫ 0 1/ n r ( α ) ⋅ bb ( 0 ) d α + 2 1 ∫ 1/ n 1 r ( α ) ⋅ bb ( α ) d α ≤ M ∫ 0 1 α n r ( α ) d α = n M ∫ 0 1 r ( α ) log ( Z e − 1/ α ) d α + log Z ⋅ n M ∫ 0 1 r ( α ) d α .
Here Z = ∫ 0 1 e − 1 / α d α Z=\int_{0}^{1}e^{-1/\sqrt{\alpha}}\,\mathrm{d}\alpha Z = ∫ 0 1 e − 1/ α d α is a normalizing constant. Now using Gibb’s inequality (i.e. ∫ 0 1 r ( α ) log p ( α ) d α ≤ ∫ 0 1 r ( α ) log r ( α ) d α \int_{0}^{1}r(\alpha)\log p(\alpha)\,\mathrm{d}\alpha\le\int_{0}^{1}r(\alpha)\log r(\alpha)\,\mathrm{d}\alpha ∫ 0 1 r ( α ) log p ( α ) d α ≤ ∫ 0 1 r ( α ) log r ( α ) d α ) we obtain the following upper bound.
− A K L ( π ) + E [ A ^ n , K L ( π ) ] ≤ M n ∫ 0 1 r ( α ) log r ( α ) d α + log Z ⋅ M n ≤ M ( ρ + log Z ) n . \begin{aligned}
-A_{\mathrm{KL}}(\pi)+\mathbb{E}\left[\widehat{A}_{n,\mathrm{KL}}(\pi)\right]&\le\frac{M}{\sqrt{n}}\int_{0}^{1}r(\alpha)\log r(\alpha)\,\mathrm{d}\alpha+\log Z\cdot\frac{M}{\sqrt{n}}\\
&\le\frac{M(\rho+\log Z)}{\sqrt{n}}.
\end{aligned} − A KL ( π ) + E [ A n , KL ( π ) ] ≤ n M ∫ 0 1 r ( α ) log r ( α ) d α + log Z ⋅ n M ≤ n M ( ρ + log Z ) .
A trivial upper bound on Z Z Z is 1 1 1 and we obtain the following bound for KL-divergence.
− A K L ( π ) + E [ A ^ n , K L ( π ) ] ≤ M ρ n -A_{\mathrm{KL}}(\pi)+\mathbb{E}\left[\widehat{A}_{n,\mathrm{KL}}(\pi)\right]\le\frac{M\rho}{\sqrt{n}} − A KL ( π ) + E [ A n , KL ( π ) ] ≤ n M ρ
Lemma 2. Suppose Assumption 3 holds. Then, for any policy π \pi π we have,
0 ≤ E X n , W n [ A ^ n , f e s t ( π ) ] − A ~ f e s t ( π ) ≤ { ( 2 M / κ ) min { 1 , 1 / α n } if A f = A C V a R , ( 4 M / κ ) 1 + 2 ρ / n if A f = A χ 2 , ( 2 M / κ ) ρ / n if A f = A K L . 0 \le\mathbb{E}_{X^n,W^n}\left[\widehat{A}^{\mathrm{est}}_{n,f}(\pi)\right]-\widetilde{A}^{\mathrm{est}}_f(\pi)\le
\begin{cases}
(2M/\kappa)\min\{1,1/\sqrt{\alpha n}\} & \text{if } A_f=A_{\mathrm{CVaR}},\\
(4M/\kappa)\sqrt{1+2\rho/\sqrt{n}} & \text{if } A_f=A_{\chi^2},\\
(2M/\kappa)\rho/\sqrt{n} & \text{if } A_f=A_{\mathrm{KL}}.
\end{cases} 0 ≤ E X n , W n [ A n , f est ( π ) ] − A f est ( π ) ≤ ⎩ ⎨ ⎧ ( 2 M / κ ) min { 1 , 1/ α n } ( 4 M / κ ) 1 + 2 ρ / n ( 2 M / κ ) ρ / n if A f = A CVaR , if A f = A χ 2 , if A f = A KL . Proof. The first inequality follows from the dual characterization of distributionally robust optimization (proposition 1 of Duchi and Namkoong [2021]).
E X n , W n [ A ^ n , f e s t ( π ) ] = E X n ∼ P 0 n , W n ∼ π o f f n [ min Q : D f ( Q ∥ P ^ n ) ≤ ρ ∑ i = 1 n q i ⋅ ( 2 π ( X i ) − 1 ) Γ ^ ( X i , W i ) ] \mathbb{E}_{X^n,W^n}\left[\widehat{A}^{\mathrm{est}}_{n,f}(\pi)\right]
=
\mathbb{E}_{X^n\sim P_0^n,W^n\sim\pi_{\mathrm{off}}^n}
\left[
\min_{Q:D_f(Q\Vert\widehat{P}_n)\le\rho}
\sum_{i=1}^{n}q_i\cdot(2\pi(X_i)-1)\widehat{\Gamma}(X_i,W_i)
\right] E X n , W n [ A n , f est ( π ) ] = E X n ∼ P 0 n , W n ∼ π off n [ Q : D f ( Q ∥ P n ) ≤ ρ min i = 1 ∑ n q i ⋅ ( 2 π ( X i ) − 1 ) Γ ( X i , W i ) ] = E X n ∼ P 0 n , W n ∼ π o f f n [ 1 n ∑ i = 1 n max λ ≥ 0 , η ∈ R − λ f ⋆ ( − ( 2 π ( X i ) − 1 ) Γ ^ ( X i , W i ) − η λ ) − λ ρ − η ] =
\mathbb{E}_{X^n\sim P_0^n,W^n\sim\pi_{\mathrm{off}}^n}
\left[
\frac{1}{n}\sum_{i=1}^{n}
\max_{\lambda\ge0,\eta\in\mathbb{R}}
-\lambda f^\star\left(\frac{-(2\pi(X_i)-1)\widehat{\Gamma}(X_i,W_i)-\eta}{\lambda}\right)-\lambda\rho-\eta
\right] = E X n ∼ P 0 n , W n ∼ π off n [ n 1 i = 1 ∑ n λ ≥ 0 , η ∈ R max − λ f ⋆ ( λ − ( 2 π ( X i ) − 1 ) Γ ( X i , W i ) − η ) − λ ρ − η ] ≥ max λ ≥ 0 , η ∈ R E X n ∼ P 0 n , W n ∼ π o f f n [ 1 n ∑ i = 1 n − λ f ⋆ ( − ( 2 π ( X i ) − 1 ) Γ ^ ( X i , W i ) − η λ ) − λ ρ − η ] \ge
\max_{\lambda\ge0,\eta\in\mathbb{R}}
\mathbb{E}_{X^n\sim P_0^n,W^n\sim\pi_{\mathrm{off}}^n}
\left[
\frac{1}{n}\sum_{i=1}^{n}
-\lambda f^\star\left(\frac{-(2\pi(X_i)-1)\widehat{\Gamma}(X_i,W_i)-\eta}{\lambda}\right)-\lambda\rho-\eta
\right] ≥ λ ≥ 0 , η ∈ R max E X n ∼ P 0 n , W n ∼ π off n [ n 1 i = 1 ∑ n − λ f ⋆ ( λ − ( 2 π ( X i ) − 1 ) Γ ( X i , W i ) − η ) − λ ρ − η ] = max λ ≥ 0 , η ∈ R E X ∼ P 0 , W ∼ π o f f [ − λ f ⋆ ( − ( 2 π ( X ) − 1 ) Γ ^ ( X , W ) − η λ ) − λ ρ − η ] =
\max_{\lambda\ge0,\eta\in\mathbb{R}}
\mathbb{E}_{X\sim P_0,W\sim\pi_{\mathrm{off}}}
\left[
-\lambda f^\star\left(\frac{-(2\pi(X)-1)\widehat{\Gamma}(X,W)-\eta}{\lambda}\right)-\lambda\rho-\eta
\right] = λ ≥ 0 , η ∈ R max E X ∼ P 0 , W ∼ π off [ − λ f ⋆ ( λ − ( 2 π ( X ) − 1 ) Γ ( X , W ) − η ) − λ ρ − η ] = min Q : D f ( Q ∥ P ^ n ) ≤ ρ E X ∼ Q , W ∼ π o f f [ ( 2 π ( X ) − 1 ) Γ ^ ( X , W ) ] =
\min_{Q:D_f(Q\Vert\widehat{P}_n)\le\rho}
\mathbb{E}_{X\sim Q,W\sim\pi_{\mathrm{off}}}
\left[(2\pi(X)-1)\widehat{\Gamma}(X,W)\right] = Q : D f ( Q ∥ P n ) ≤ ρ min E X ∼ Q , W ∼ π off [ ( 2 π ( X ) − 1 ) Γ ( X , W ) ] = A ~ f e s t ( π ) =
\widetilde{A}^{\mathrm{est}}_f(\pi) = A f est ( π ) The proof of the second inequality is almost identical to the proof of lemma 1 if we use the upper bound ∣ Γ ^ i ∣ ≤ 2 M / κ \lvert\widehat{\Gamma}_i\rvert\le2M/\kappa ∣ Γ i ∣ ≤ 2 M / κ .
Bound on Estimation Error Lemma 3. Suppose assumption 1 and 3 hold and N = N ( Π , L f t / 3 , ∥ ⋅ ∥ ∞ ) N=\mathcal{N}(\Pi,L_f t/3,\|\cdot\|_\infty) N = N ( Π , L f t /3 , ∥ ⋅ ∥ ∞ ) be L f t / 3 L_f t/3 L f t /3 -covering number of the policy class Π \Pi Π where,
L f = { 1 α n if A f = A C V a R , 3 + 2 ρ n if A f = A χ 2 , exp ( ρ ) n if A f = A K L . L_f=
\begin{cases}
\frac{1}{\alpha\sqrt{n}} & \text{if } A_f=A_{\mathrm{CVaR}},\\
\frac{\sqrt{3+2\rho}}{\sqrt{n}} & \text{if } A_f=A_{\chi^2},\\
\frac{\exp(\rho)}{\sqrt{n}} & \text{if } A_f=A_{\mathrm{KL}}.
\end{cases} L f = ⎩ ⎨ ⎧ α n 1 n 3 + 2 ρ n e x p ( ρ ) if A f = A CVaR , if A f = A χ 2 , if A f = A KL . Then with probability at least 1 − 2 N exp ( − t 2 ) 1-2N\exp(-t^2) 1 − 2 N exp ( − t 2 ) the following holds for any policy π ∈ Π \pi\in\Pi π ∈ Π .
∣ A ~ f e s t ( π ) − A ^ n , f e s t ( π ) ∣ ≤ { 2 M κ min { 1 , 1 α n } + 5 M t α κ n if A f = A C V a R , 4 M 1 + 2 ρ κ n + 5 M 3 + 2 ρ t κ n if A f = A χ 2 , 2 M ρ κ n + 5 M exp ( ρ ) t κ n if A f = A K L . \left\lvert\widetilde{A}^{\mathrm{est}}_f(\pi)-\widehat{A}^{\mathrm{est}}_{n,f}(\pi)\right\rvert
\le
\begin{cases}
\frac{2M}{\kappa}\min\left\{1,\frac{1}{\sqrt{\alpha n}}\right\}+\frac{5Mt}{\alpha\kappa\sqrt{n}} & \text{if } A_f=A_{\mathrm{CVaR}},\\
\frac{4M\sqrt{1+2\rho}}{\kappa\sqrt{n}}+\frac{5M\sqrt{3+2\rho}t}{\kappa\sqrt{n}} & \text{if } A_f=A_{\chi^2},\\
\frac{2M\rho}{\kappa\sqrt{n}}+\frac{5M\exp(\rho)t}{\kappa\sqrt{n}} & \text{if } A_f=A_{\mathrm{KL}}.
\end{cases} A f est ( π ) − A n , f est ( π ) ≤ ⎩ ⎨ ⎧ κ 2 M min { 1 , α n 1 } + α κ n 5 M t κ n 4 M 1 + 2 ρ + κ n 5 M 3 + 2 ρ t κ n 2 M ρ + κ n 5 M e x p ( ρ ) t if A f = A CVaR , if A f = A χ 2 , if A f = A KL . Proof. Let us define Z i ( π ) = ( 2 π ( X i ) − 1 ) Γ ^ i Z_i(\pi)=(2\pi(X_i)-1)\widehat{\Gamma}_i Z i ( π ) = ( 2 π ( X i ) − 1 ) Γ i . Then A ^ n , f e s t ( π ) = min Q : D f ( Q ∥ P ^ n ) ≤ ρ ∑ i Q i Z i ( π ) \widehat{A}^{\mathrm{est}}_{n,f}(\pi)=\min_{Q:D_f(Q\Vert\widehat{P}_n)\le\rho}\sum_i Q_iZ_i(\pi) A n , f est ( π ) = min Q : D f ( Q ∥ P n ) ≤ ρ ∑ i Q i Z i ( π ) . Since A ^ n , f e s t ( π ) \widehat{A}^{\mathrm{est}}_{n,f}(\pi) A n , f est ( π ) is a minimum over linear functions, it is convex. Additionally, the subdifferential
∂ A ^ n , f e s t ( π ; Z n ) = conv { Q ⋆ : Q ⋆ ∈ arg min Q : D f ( Q ∥ P ^ n ) ≤ ρ Q T Z n } \partial\widehat{A}^{\mathrm{est}}_{n,f}(\pi;Z^n)
=
\operatorname{conv}\left\{Q_\star:Q_\star\in\underset{Q:D_f(Q\Vert\widehat{P}_n)\le\rho}{\arg\min}\ Q^\mathsf{T}Z^n\right\} ∂ A n , f est ( π ; Z n ) = conv { Q ⋆ : Q ⋆ ∈ Q : D f ( Q ∥ P n ) ≤ ρ arg min Q T Z n } is the convex hull of all the minimizers. Now we bound ∥ Q ∥ 2 \lVert Q\rVert_2 ∥ Q ∥ 2 for any Q ∈ ∂ A ^ n , f e s t ( π ; Z n ) Q\in\partial\widehat{A}^{\mathrm{est}}_{n,f}(\pi;Z^n) Q ∈ ∂ A n , f est ( π ; Z n ) based on the choice of f f f -divergence.
For CVaR constraint at level α \alpha α , f ( t ) = 1 { t ≤ 1 / α } f(t)=\mathbf{1}_{\{t\le1/\alpha\}} f ( t ) = 1 { t ≤ 1/ α } . For any distribution Q Q Q with finite D f ( Q ∥ P ^ n ) D_f(Q\Vert\widehat{P}_n) D f ( Q ∥ P n ) , each entry is at most 1 α n \frac{1}{\alpha n} α n 1 . Therefore, ∥ Q ∥ 2 ≤ n α 2 n 2 = 1 α n \lVert Q\rVert_2\le\sqrt{\frac{n}{\alpha^2n^2}}=\frac{1}{\alpha\sqrt{n}} ∥ Q ∥ 2 ≤ α 2 n 2 n = α n 1 .6
For χ 2 \chi^2 χ 2 -divergence, f ( t ) = 1 2 ( t − 1 ) 2 f(t)=\frac{1}{2}(t-1)^2 f ( t ) = 2 1 ( t − 1 ) 2 . Now, D f ( Q ∥ P ^ n ) = 1 2 n ∑ i ( n q i − 1 ) 2 ≤ ρ D_f(Q\|\widehat{P}_n)=\frac{1}{2n}\sum_i(nq_i-1)^2\leq\rho D f ( Q ∥ P n ) = 2 n 1 ∑ i ( n q i − 1 ) 2 ≤ ρ . After rearranging one obtains n 2 ∥ Q ∥ 2 2 − 2 n ∥ Q ∥ 1 ≤ ( 1 + 2 ρ ) n n^2\|Q\|_2^2-2n\|Q\|_1\leq(1+2\rho)n n 2 ∥ Q ∥ 2 2 − 2 n ∥ Q ∥ 1 ≤ ( 1 + 2 ρ ) n . Since ∥ Q ∥ 1 = 1 \|Q\|_1=1 ∥ Q ∥ 1 = 1 , we get ∥ Q ∥ 2 ≤ 1 + 2 ρ n \|Q\|_2\leq\sqrt{\frac{1+2\rho}{n}} ∥ Q ∥ 2 ≤ n 1 + 2 ρ .
For KL-divergence, f ( t ) = t log t f(t)=t\log t f ( t ) = t log t . Now, D f ( Q ∥ P ^ n ) = ∑ i q i log ( n q i ) ≤ ρ D_f(Q\|\widehat{P}_n)=\sum_iq_i\log(nq_i)\leq\rho D f ( Q ∥ P n ) = ∑ i q i log ( n q i ) ≤ ρ . Suppose, there are M M M entries of Q Q Q with mass at least exp ( ρ ) / n \exp(\rho)/n exp ( ρ ) / n . Then ∑ i q i log ( n q i ) ≥ M ρ n exp ( ρ ) \sum_iq_i\log(nq_i)\geq M\frac{\rho}{n}\exp(\rho) ∑ i q i log ( n q i ) ≥ M n ρ exp ( ρ ) . This implies that M ≤ n exp ( ρ ) M\leq\frac{n}{\exp(\rho)} M ≤ e x p ( ρ ) n . Since the total probability mass is 1 1 1 , each of the M M M entries have mass exactly exp ( ρ ) / n \exp(\rho)/n exp ( ρ ) / n . Therefore, ∥ Q ∥ 2 2 = n ⋅ ( exp ( ρ ) n ) 2 \|Q\|_2^2=n\cdot\left(\frac{\exp(\rho)}{n}\right)^2 ∥ Q ∥ 2 2 = n ⋅ ( n e x p ( ρ ) ) 2 and ∥ Q ∥ 2 ≤ exp ( ρ ) n \|Q\|_2\leq\frac{\exp(\rho)}{\sqrt{n}} ∥ Q ∥ 2 ≤ n e x p ( ρ ) .
Therefore, we have shown that the function A ^ n , f e s t ( π ; Z n ) \widehat{A}^{\mathrm{est}}_{n,f}(\pi;Z^n) A n , f est ( π ; Z n ) is L f L_f L f -Lipschitz where L f L_f L f is defined as follows.
L f = { 1 α n if A f = A C V A R 3 + 2 ρ n if A f = A χ 2 exp ( ρ ) n if A f = A K L L_f=\begin{cases}
\frac{1}{\alpha\sqrt{n}} & \text{if } A_f=A_{\mathrm{CVAR}}\\
\frac{\sqrt{3+2\rho}}{\sqrt{n}} & \text{if } A_f=A_{\chi^2}\\
\frac{\exp(\rho)}{\sqrt{n}} & \text{if } A_f=A_{\mathrm{KL}}
\end{cases} L f = ⎩ ⎨ ⎧ α n 1 n 3 + 2 ρ n e x p ( ρ ) if A f = A CVAR if A f = A χ 2 if A f = A KL
We can apply lemma 5 and obtain the following bound for a fixed policy π ∈ Π \pi\in\Pi π ∈ Π .
P ( ∣ A ^ n , f e s t ( π ) − E [ A ^ n , f e s t ( π ) ] ∣ ≥ ε ) ≤ 2 exp ( − ε 2 κ 2 4 L f 2 M 2 ) \mathbb{P}\left(\left|\widehat{A}^{\mathrm{est}}_{n,f}(\pi)-\mathbb{E}\left[\widehat{A}^{\mathrm{est}}_{n,f}(\pi)\right]\right|\geq\varepsilon\right)\leq2\exp\left(-\frac{\varepsilon^2\kappa^2}{4L_f^2M^2}\right) P ( A n , f est ( π ) − E [ A n , f est ( π ) ] ≥ ε ) ≤ 2 exp ( − 4 L f 2 M 2 ε 2 κ 2 )
In other words, with probability at least 1 − 2 e − t 1-2e^{-t} 1 − 2 e − t we have, ∣ A ^ n , f e s t ( π ) − E [ A ^ n , f e s t ( π ) ] ∣ ≤ 2 L f M t t κ \left|\widehat{A}^{\mathrm{est}}_{n,f}(\pi)-\mathbb{E}\left[\widehat{A}^{\mathrm{est}}_{n,f}(\pi)\right]\right|\leq\frac{2L_fMt\sqrt{t}}{\kappa} A n , f est ( π ) − E [ A n , f est ( π ) ] ≤ κ 2 L f M t t .
Now, lemma 2 proves that 0 ≤ E [ A ^ n , f e s t ( π ) ] − A ^ f e s t ( π ) ≤ B f 0\leq\mathbb{E}\left[\widehat{A}^{\mathrm{est}}_{n,f}(\pi)\right]-\widehat{A}^{\mathrm{est}}_f(\pi)\leq B_f 0 ≤ E [ A n , f est ( π ) ] − A f est ( π ) ≤ B f for a constant B f B_f B f depending on the f f f -divergence. Therefore, with probability at least 1 − 2 e − t 2 1-2e^{-t^2} 1 − 2 e − t 2 ,
∣ A ^ n , f e s t ( π ) − A ~ f e s t ( π ) ∣ ≤ B f + 2 L f M t κ . \left|\widehat{A}^{\mathrm{est}}_{n,f}(\pi)-\widetilde{A}^{\mathrm{est}}_f(\pi)\right|\leq B_f+\frac{2L_fMt}{\kappa}. A n , f est ( π ) − A f est ( π ) ≤ B f + κ 2 L f M t .
Now we extend the above bound to any policy π ∈ Π \pi\in\Pi π ∈ Π using a standard argument. Let N = N ( Π , L f t 3 , ∥ ⋅ ∥ ∞ ) N=\mathcal{N}\left(\Pi,\frac{L_ft}{3},\|\cdot\|_\infty\right) N = N ( Π , 3 L f t , ∥ ⋅ ∥ ∞ ) be a L f t 3 \frac{L_ft}{3} 3 L f t -covering number of the space of policies Π \Pi Π according to the norm ∥ π ∥ ∞ = sup x ∈ X ∣ π ( x ) ∣ \|\pi\|_\infty=\sup_{x\in\mathcal{X}}|\pi(x)| ∥ π ∥ ∞ = sup x ∈ X ∣ π ( x ) ∣ . Then there exist policies π 1 , … , π N ∈ Π \pi_1,\ldots,\pi_N\in\Pi π 1 , … , π N ∈ Π so that for any π ∈ Π \pi\in\Pi π ∈ Π there exists π ~ \widetilde{\pi} π with ∥ π − π ~ ∥ ∞ ≤ L f t / 3 \|\pi-\widetilde{\pi}\|_\infty\leq L_ft/3 ∥ π − π ∥ ∞ ≤ L f t /3 .
∣ A ^ n , f e s t ( π ) − A ~ f e s t ( π ) ∣ ≤ ∣ A ^ n , f e s t ( π ) − A ^ n , f e s t ( π ~ ) ∣ + ∣ A ^ n , f e s t ( π ~ ) − A ~ f e s t ( π ~ ) ∣ + ∣ A ~ f e s t ( π ~ ) − A ~ f e s t ( π ) ∣ (15) \left|\widehat{A}^{\mathrm{est}}_{n,f}(\pi)-\widetilde{A}^{\mathrm{est}}_f(\pi)\right|\leq\left|\widehat{A}^{\mathrm{est}}_{n,f}(\pi)-\widehat{A}^{\mathrm{est}}_{n,f}(\widetilde{\pi})\right|+\left|\widehat{A}^{\mathrm{est}}_{n,f}(\widetilde{\pi})-\widetilde{A}^{\mathrm{est}}_f(\widetilde{\pi})\right|+\left|\widetilde{A}^{\mathrm{est}}_f(\widetilde{\pi})-\widetilde{A}^{\mathrm{est}}_f(\pi)\right| \tag*{(15)} A n , f est ( π ) − A f est ( π ) ≤ A n , f est ( π ) − A n , f est ( π ) + A n , f est ( π ) − A f est ( π ) + A f est ( π ) − A f est ( π ) (15)
Now observe that,
A ^ n , f e s t ( π ) − A ^ n , f e s t ( π ~ ) = min Q : D f ( Q ∥ P ^ n ) ≤ ρ ∑ i = 1 n q i ⋅ ( 2 π ( X i ) − 1 ) Γ ^ i − min Q : D f ( Q ∥ P ^ n ) ≤ ρ ∑ i = 1 n q i ⋅ ( 2 π ~ ( X i ) − 1 ) Γ ^ i ≤ max Q : D f ( Q ∥ P ^ n ) ≤ ρ ∣ ∑ i = 1 n q i ⋅ ( 2 π ( X i ) − 1 ) Γ ^ i − ∑ i = 1 n q i ⋅ ( 2 π ~ ( X i ) − 1 ) Γ ^ i ∣ = max Q : D f ( Q ∥ P ^ n ) ≤ ρ ∣ ∑ i = 1 n 2 q i ⋅ ( π ( X i ) − π ~ ( X i ) ) Γ ^ i ∣ ≤ 4 M κ max Q : D f ( Q ∥ P ^ n ) ≤ ρ ∑ i = 1 n q i ⋅ ∣ π ( X i ) − π ~ ( X i ) ∣ ≤ 4 M L f t 3 κ . \begin{aligned}
\widehat{A}^{\mathrm{est}}_{n,f}(\pi)-\widehat{A}^{\mathrm{est}}_{n,f}(\widetilde{\pi})
&=\min_{Q:D_f(Q\|\widehat{P}_n)\leq\rho}\sum_{i=1}^{n}q_i\cdot(2\pi(X_i)-1)\widehat{\Gamma}_i-\min_{Q:D_f(Q\|\widehat{P}_n)\leq\rho}\sum_{i=1}^{n}q_i\cdot(2\widetilde{\pi}(X_i)-1)\widehat{\Gamma}_i\\
&\leq\max_{Q:D_f(Q\|\widehat{P}_n)\leq\rho}\left|\sum_{i=1}^{n}q_i\cdot(2\pi(X_i)-1)\widehat{\Gamma}_i-\sum_{i=1}^{n}q_i\cdot(2\widetilde{\pi}(X_i)-1)\widehat{\Gamma}_i\right|\\
&=\max_{Q:D_f(Q\|\widehat{P}_n)\leq\rho}\left|\sum_{i=1}^{n}2q_i\cdot(\pi(X_i)-\widetilde{\pi}(X_i))\widehat{\Gamma}_i\right|\\
&\leq\frac{4M}{\kappa}\max_{Q:D_f(Q\|\widehat{P}_n)\leq\rho}\sum_{i=1}^{n}q_i\cdot|\pi(X_i)-\widetilde{\pi}(X_i)|\\
&\leq\frac{4ML_ft}{3\kappa}.
\end{aligned} A n , f est ( π ) − A n , f est ( π ) = Q : D f ( Q ∥ P n ) ≤ ρ min i = 1 ∑ n q i ⋅ ( 2 π ( X i ) − 1 ) Γ i − Q : D f ( Q ∥ P n ) ≤ ρ min i = 1 ∑ n q i ⋅ ( 2 π ( X i ) − 1 ) Γ i ≤ Q : D f ( Q ∥ P n ) ≤ ρ max i = 1 ∑ n q i ⋅ ( 2 π ( X i ) − 1 ) Γ i − i = 1 ∑ n q i ⋅ ( 2 π ( X i ) − 1 ) Γ i = Q : D f ( Q ∥ P n ) ≤ ρ max i = 1 ∑ n 2 q i ⋅ ( π ( X i ) − π ( X i )) Γ i ≤ κ 4 M Q : D f ( Q ∥ P n ) ≤ ρ max i = 1 ∑ n q i ⋅ ∣ π ( X i ) − π ( X i ) ∣ ≤ 3 κ 4 M L f t .
Similarly, we can establish ∣ A ~ f e s t ( π ) − A ~ f e s t ( π ~ ) ∣ ≤ 4 M L f t 3 κ \left|\widetilde{A}^{\mathrm{est}}_f(\pi)-\widetilde{A}^{\mathrm{est}}_f(\widetilde{\pi})\right|\leq\frac{4ML_ft}{3\kappa} A f est ( π ) − A f est ( π ) ≤ 3 κ 4 M L f t . Substituting the above bounds in eq. (15) , we obtain the following inequality for any policy π ∈ Π \pi\in\Pi π ∈ Π .
∣ A ^ n , f e s t ( π ) − A ~ f e s t ( π ) ∣ ≤ 8 M L f t 3 κ + ∣ A ^ n , f e s t ( π ~ ) − A ~ f e s t ( π ~ ) ∣ \left|\widehat{A}^{\mathrm{est}}_{n,f}(\pi)-\widetilde{A}^{\mathrm{est}}_f(\pi)\right|\leq\frac{8ML_ft}{3\kappa}+\left|\widehat{A}^{\mathrm{est}}_{n,f}(\widetilde{\pi})-\widetilde{A}^{\mathrm{est}}_f(\widetilde{\pi})\right| A n , f est ( π ) − A f est ( π ) ≤ 3 κ 8 M L f t + A n , f est ( π ) − A f est ( π )
Therefore, by a union bound over the N N N policies π 1 , … , π N \pi_1,\ldots,\pi_N π 1 , … , π N
P ( max π ∈ Π ∣ A ^ n , f e s t ( π ) − A ~ f e s t ( π ) ∣ ≥ B f + 5 M L f t κ ) ≤ 1 − 2 N exp ( − t 2 / 9 ) . \mathbb{P}\left(\max_{\pi\in\Pi}\left|\widehat{A}^{\mathrm{est}}_{n,f}(\pi)-\widetilde{A}^{\mathrm{est}}_{f}(\pi)\right|\ge B_f+\frac{5ML_f t}{\kappa}\right)\le1-2N\exp(-t^2/9). P ( π ∈ Π max A n , f est ( π ) − A f est ( π ) ≥ B f + κ 5 M L f t ) ≤ 1 − 2 N exp ( − t 2 /9 ) .
Lemma 4. Suppose assumptions 1 1 1 , 2 2 2 , and 3 3 3 hold. Then we have,
∣ A f ( π ) − A ~ f e s t ( π ) ∣ ≤ 2 M κ n − ( β + γ ) \left|A_f(\pi)-\widetilde{A}^{\mathrm{est}}_f(\pi)\right|\le\frac{2M}{\kappa}n^{-(\beta+\gamma)} A f ( π ) − A f est ( π ) ≤ κ 2 M n − ( β + γ )
for any policy π ∈ Π \pi\in\Pi π ∈ Π .
Proof. First note that,
A f ( π ) − A ~ f e s t ( π ) = min Q : D f ( Q ∥ P 0 ) ≤ ρ E X ∼ Q [ ( 2 π ( X ) − 1 ) τ ( X ) ] − min Q : D f ( Q ∥ P 0 ) ≤ ρ E X ∼ Q , W ∼ π o f f ( ⋅ ∣ X ) [ ( 2 π ( X ) − 1 ) Γ ^ ( X , W ) ] ≤ max Q : D f ( Q ∥ P 0 ) ≤ ρ ∣ E X ∼ Q [ ( 2 π ( X ) − 1 ) τ ( X ) ] − E X ∼ Q , W ∼ π o f f ( ⋅ ∣ X ) [ ( 2 π ( X ) − 1 ) Γ ^ ( X , W ) ] ∣ = max Q : D f ( Q ∥ P 0 ) ≤ ρ ∣ E X ∼ Q [ ( 2 π ( X ) − 1 ) ( τ ( X ) − E W ∼ π o f f ( ⋅ ∣ X ) [ Γ ^ ( X , W ) ] ) ] ∣ . (16) \begin{aligned}
A_f(\pi)-\widetilde{A}^{\mathrm{est}}_f(\pi)
&=\min_{Q:D_f(Q\|P_0)\le\rho}\mathbb{E}_{X\sim Q}\left[(2\pi(X)-1)\tau(X)\right]\\
&\quad-\min_{Q:D_f(Q\|P_0)\le\rho}\mathbb{E}_{X\sim Q,W\sim\pi_{\mathrm{off}}(\cdot\mid X)}\left[(2\pi(X)-1)\widehat{\Gamma}(X,W)\right]\\
&\le\max_{Q:D_f(Q\|P_0)\le\rho}\left|\mathbb{E}_{X\sim Q}\left[(2\pi(X)-1)\tau(X)\right]-\mathbb{E}_{X\sim Q,W\sim\pi_{\mathrm{off}}(\cdot\mid X)}\left[(2\pi(X)-1)\widehat{\Gamma}(X,W)\right]\right|\\
&=\max_{Q:D_f(Q\|P_0)\le\rho}\left|\mathbb{E}_{X\sim Q}\left[(2\pi(X)-1)\left(\tau(X)-\mathbb{E}_{W\sim\pi_{\mathrm{off}}(\cdot\mid X)}[\widehat{\Gamma}(X,W)]\right)\right]\right|. \tag*{(16)}
\end{aligned} A f ( π ) − A f est ( π ) = Q : D f ( Q ∥ P 0 ) ≤ ρ min E X ∼ Q [ ( 2 π ( X ) − 1 ) τ ( X ) ] − Q : D f ( Q ∥ P 0 ) ≤ ρ min E X ∼ Q , W ∼ π off ( ⋅ ∣ X ) [ ( 2 π ( X ) − 1 ) Γ ( X , W ) ] ≤ Q : D f ( Q ∥ P 0 ) ≤ ρ max E X ∼ Q [ ( 2 π ( X ) − 1 ) τ ( X ) ] − E X ∼ Q , W ∼ π off ( ⋅ ∣ X ) [ ( 2 π ( X ) − 1 ) Γ ( X , W ) ] = Q : D f ( Q ∥ P 0 ) ≤ ρ max E X ∼ Q [ ( 2 π ( X ) − 1 ) ( τ ( X ) − E W ∼ π off ( ⋅ ∣ X ) [ Γ ( X , W )] ) ] . (16)
Now we use a standard argument to bound the estimation error term Δ ( X ) \Delta(X) Δ ( X ) .{sup}7`
τ ( X ) − E W ∼ π o f f ( ⋅ ∣ X ) [ Γ ^ ( X , W ) ] = m ( X , 1 ) − m ( X , 0 ) − E W ∼ π o f f ( ⋅ ∣ X ) [ m ^ ( X , 1 ) − m ^ ( X , 0 ) + W − e ~ ( X ) e ~ ( X ) ( 1 − e ~ ( X ) ) ( Y − m ^ ( X , W ) ) ] = m ( X , 1 ) − m ^ ( X , 1 ) − ( m ( X , 0 ) − m ^ ( X , 0 ) ) − e ( X ) e ~ ( X ) ( m ( X , 1 ) − m ^ ( X , 1 ) ) + 1 − e ( X ) 1 − e ~ ( X ) ( m ( X , 0 ) − m ^ ( X , 0 ) ) = ( m ( X , 1 ) − m ^ ( X , 1 ) ) e ~ ( X ) − e ( X ) e ~ ( X ) + ( m ( X , 0 ) − m ^ ( X , 0 ) ) e ~ ( X ) − e ( X ) 1 − e ~ ( X ) . \begin{aligned}
\tau(X)-\mathbb{E}_{W\sim\pi_{\mathrm{off}}(\cdot\mid X)}[\widehat{\Gamma}(X,W)]
&=m(X,1)-m(X,0)-\mathbb{E}_{W\sim\pi_{\mathrm{off}}(\cdot\mid X)}\left[\widehat{m}(X,1)-\widehat{m}(X,0)+\frac{W-\widetilde{e}(X)}{\widetilde{e}(X)(1-\widetilde{e}(X))}(Y-\widehat{m}(X,W))\right]\\
&=m(X,1)-\widehat{m}(X,1)-(m(X,0)-\widehat{m}(X,0))\\
&\quad-\frac{e(X)}{\widetilde{e}(X)}(m(X,1)-\widehat{m}(X,1))+\frac{1-e(X)}{1-\widetilde{e}(X)}(m(X,0)-\widehat{m}(X,0))\\
&=(m(X,1)-\widehat{m}(X,1))\frac{\widetilde{e}(X)-e(X)}{\widetilde{e}(X)}+(m(X,0)-\widehat{m}(X,0))\frac{\widetilde{e}(X)-e(X)}{1-\widetilde{e}(X)}.
\end{aligned} τ ( X ) − E W ∼ π off ( ⋅ ∣ X ) [ Γ ( X , W )] = m ( X , 1 ) − m ( X , 0 ) − E W ∼ π off ( ⋅ ∣ X ) [ m ( X , 1 ) − m ( X , 0 ) + e ( X ) ( 1 − e ( X )) W − e ( X ) ( Y − m ( X , W )) ] = m ( X , 1 ) − m ( X , 1 ) − ( m ( X , 0 ) − m ( X , 0 )) − e ( X ) e ( X ) ( m ( X , 1 ) − m ( X , 1 )) + 1 − e ( X ) 1 − e ( X ) ( m ( X , 0 ) − m ( X , 0 )) = ( m ( X , 1 ) − m ( X , 1 )) e ( X ) e ( X ) − e ( X ) + ( m ( X , 0 ) − m ( X , 0 )) 1 − e ( X ) e ( X ) − e ( X ) .
We substitute the above bound in eq. (17) and use assumption 1 1 1 .
∣ A f ( π ) − A ~ f e s t ( π ) ∣ ≤ M max Q : D f ( Q ∥ P 0 ) ≤ ρ E X ∼ Q [ ∣ ( m ( X , 1 ) − m ^ ( X , 1 ) ) e ~ ( X ) − e ( X ) e ~ ( X ) + ( m ( X , 0 ) − m ^ ( X , 0 ) ) e ~ ( X ) − e ( X ) 1 − e ~ ( X ) ∣ ] ≤ M κ max Q : D f ( Q ∥ P 0 ) ≤ ρ ( E X ∼ Q [ ∣ ( m ( X , 1 ) − m ^ ( X , 1 ) ) ( e ~ ( X ) − e ( X ) ) ∣ ] + E X ∼ Q [ ∣ ( m ( X , 0 ) − m ^ ( X , 0 ) ) ( e ~ ( X ) − e ( X ) ) ∣ ] ) ≤ M κ max Q : D f ( Q ∥ P 0 ) ≤ ρ ( E X ∼ Q [ ∣ e ~ ( X ) − e ( X ) ∣ 2 ] ) 1 / 2 ⋅ ( ( E X ∼ Q [ ∣ m ^ ( X , 1 ) − m ( X , 1 ) ∣ 2 ] ) 1 / 2 + ( E X ∼ Q [ ∣ m ^ ( X , 0 ) − m ( X , 0 ) ∣ 2 ] ) 1 / 2 ) ≤ M κ max Q : D f ( Q ∥ P 0 ) ≤ ρ ( E X ∼ Q [ ∣ e ~ ( X ) − e ( X ) ∣ 2 ] ) 1 / 2 ⋅ ( max Q : D f ( Q ∥ P 0 ) ≤ ρ ( E X ∼ Q [ ∣ m ^ ( X , 1 ) − m ( X , 1 ) ∣ 2 ] ) 1 / 2 + max Q : D f ( Q ∥ P 0 ) ≤ ρ ( E X ∼ Q [ ∣ m ^ ( X , 0 ) − m ( X , 0 ) ∣ 2 ] ) 1 / 2 ) ≤ 2 M κ n − ( β + γ ) . (17) \begin{aligned}
\left|A_f(\pi)-\widetilde{A}^{\mathrm{est}}_f(\pi)\right|
&\le M\max_{Q:D_f(Q\|P_0)\le\rho}\mathbb{E}_{X\sim Q}\left[\left|(m(X,1)-\widehat{m}(X,1))\frac{\widetilde{e}(X)-e(X)}{\widetilde{e}(X)}+(m(X,0)-\widehat{m}(X,0))\frac{\widetilde{e}(X)-e(X)}{1-\widetilde{e}(X)}\right|\right]\\
&\le\frac{M}{\kappa}\max_{Q:D_f(Q\|P_0)\le\rho}\left(\mathbb{E}_{X\sim Q}\left[|(m(X,1)-\widehat{m}(X,1))(\widetilde{e}(X)-e(X))|\right]+\mathbb{E}_{X\sim Q}\left[|(m(X,0)-\widehat{m}(X,0))(\widetilde{e}(X)-e(X))|\right]\right)\\
&\le\frac{M}{\kappa}\max_{Q:D_f(Q\|P_0)\le\rho}\left(\mathbb{E}_{X\sim Q}\left[|\widetilde{e}(X)-e(X)|^2\right]\right)^{1/2}\\
&\quad\cdot\left(\left(\mathbb{E}_{X\sim Q}\left[|\widehat{m}(X,1)-m(X,1)|^2\right]\right)^{1/2}+\left(\mathbb{E}_{X\sim Q}\left[|\widehat{m}(X,0)-m(X,0)|^2\right]\right)^{1/2}\right)\\
&\le\frac{M}{\kappa}\max_{Q:D_f(Q\|P_0)\le\rho}\left(\mathbb{E}_{X\sim Q}\left[|\widetilde{e}(X)-e(X)|^2\right]\right)^{1/2}\\
&\quad\cdot\left(\max_{Q:D_f(Q\|P_0)\le\rho}\left(\mathbb{E}_{X\sim Q}\left[|\widehat{m}(X,1)-m(X,1)|^2\right]\right)^{1/2}+\max_{Q:D_f(Q\|P_0)\le\rho}\left(\mathbb{E}_{X\sim Q}\left[|\widehat{m}(X,0)-m(X,0)|^2\right]\right)^{1/2}\right)\\
&\le\frac{2M}{\kappa}n^{-(\beta+\gamma)}. \tag*{(17)}
\end{aligned} A f ( π ) − A f est ( π ) ≤ M Q : D f ( Q ∥ P 0 ) ≤ ρ max E X ∼ Q [ ( m ( X , 1 ) − m ( X , 1 )) e ( X ) e ( X ) − e ( X ) + ( m ( X , 0 ) − m ( X , 0 )) 1 − e ( X ) e ( X ) − e ( X ) ] ≤ κ M Q : D f ( Q ∥ P 0 ) ≤ ρ max ( E X ∼ Q [ ∣ ( m ( X , 1 ) − m ( X , 1 )) ( e ( X ) − e ( X )) ∣ ] + E X ∼ Q [ ∣ ( m ( X , 0 ) − m ( X , 0 )) ( e ( X ) − e ( X )) ∣ ] ) ≤ κ M Q : D f ( Q ∥ P 0 ) ≤ ρ max ( E X ∼ Q [ ∣ e ( X ) − e ( X ) ∣ 2 ] ) 1/2 ⋅ ( ( E X ∼ Q [ ∣ m ( X , 1 ) − m ( X , 1 ) ∣ 2 ] ) 1/2 + ( E X ∼ Q [ ∣ m ( X , 0 ) − m ( X , 0 ) ∣ 2 ] ) 1/2 ) ≤ κ M Q : D f ( Q ∥ P 0 ) ≤ ρ max ( E X ∼ Q [ ∣ e ( X ) − e ( X ) ∣ 2 ] ) 1/2 ⋅ ( Q : D f ( Q ∥ P 0 ) ≤ ρ max ( E X ∼ Q [ ∣ m ( X , 1 ) − m ( X , 1 ) ∣ 2 ] ) 1/2 + Q : D f ( Q ∥ P 0 ) ≤ ρ max ( E X ∼ Q [ ∣ m ( X , 0 ) − m ( X , 0 ) ∣ 2 ] ) 1/2 ) ≤ κ 2 M n − ( β + γ ) . (17)
The last line uses assumption 2 2 2 .
Lemma 5 (Boucheron et al. [ [7 ] , Theorem 6.10). Let h : R n → R h:\mathbb{R}^{n}\to\mathbb{R} h : R n → R be a convex or concave and L L L -Lipschitz with respect to the ℓ 2 \ell_{2} ℓ 2 -norm. Let Z i Z_i Z i be independent random variables with Z i ∈ [ a , b ] Z_i\in[a,b] Z i ∈ [ a , b ] for each i ∈ [ n ] i\in[n] i ∈ [ n ] . Then,
P ( ∣ h ( Z n ) − E [ h ( Z n ) ] ∣ ≥ t ) ≤ 2 exp ( − t 2 L 2 ( b − a ) 2 ) . \mathbb{P}\left(\left|h(Z^{n})-\mathbb{E}[h(Z^{n})]\right|\ge t\right)\le2\exp\left(-\frac{t^{2}}{L^{2}(b-a)^{2}}\right). P ( ∣ h ( Z n ) − E [ h ( Z n )] ∣ ≥ t ) ≤ 2 exp ( − L 2 ( b − a ) 2 t 2 ) .
Group-Robust Policy Learning
Proof of Theorem 2. Following the proof of Theorem 1 we can establish the following bound.
sup π ∈ Π A f ( π ) − A f ( π ~ n ) ≤ 2 sup π ∈ Π ∣ A f ( π ) − A ~ f e s t ( π ) ∣ + 2 sup π ∈ Π ∣ A ~ f e s t ( π ) − A ^ n , f e s t ( π ) ∣ \sup_{\pi\in\Pi}A_f(\pi)-A_f(\widetilde{\pi}_n)\le2\sup_{\pi\in\Pi}\left|A_f(\pi)-\widetilde{A}^{\mathrm{est}}_f(\pi)\right|+2\sup_{\pi\in\Pi}\left|\widetilde{A}^{\mathrm{est}}_f(\pi)-\widehat{A}^{\mathrm{est}}_{n,f}(\pi)\right| π ∈ Π sup A f ( π ) − A f ( π n ) ≤ 2 π ∈ Π sup A f ( π ) − A f est ( π ) + 2 π ∈ Π sup A f est ( π ) − A n , f est ( π )
where
A ~ f e s t ( π ) = min α : D f ( p ∥ d 0 ) ≤ ρ ∑ g = 1 k p g ⋅ A ~ g e s t ( π ) with A ~ g e s t ( π ) = E X ∼ P g , W ∼ π o f f ( ⋅ ∣ X ) [ ( 2 π ( X ) − 1 ) Γ ^ ( X , W ) ] \widetilde{A}^{\mathrm{est}}_f(\pi)=\min_{\alpha:D_f(p\Vert d_0)\le\rho}\sum_{g=1}^{k}p_g\cdot\widetilde{A}^{\mathrm{est}}_g(\pi)
\quad\text{with}\quad
\widetilde{A}^{\mathrm{est}}_g(\pi)=\mathbb{E}_{X\sim P_g,\ W\sim\pi_{\mathrm{off}(\cdot\mid X)}}\left[(2\pi(X)-1)\widehat{\Gamma}(X,W)\right] A f est ( π ) = α : D f ( p ∥ d 0 ) ≤ ρ min g = 1 ∑ k p g ⋅ A g est ( π ) with A g est ( π ) = E X ∼ P g , W ∼ π off ( ⋅ ∣ X ) [ ( 2 π ( X ) − 1 ) Γ ( X , W ) ]
and
A ^ n , f e s t ( π ) = min α : D f ( p ∥ d 0 ) ≤ ρ ∑ g = 1 k p g ⋅ A ^ g e s t ( π ) with A ^ g e s t ( π ) = 1 ∣ D g ∣ ∑ i ∈ D g ( 2 π ( X i ) − 1 ) Γ ^ ( X i , W i ) \widehat{A}^{\mathrm{est}}_{n,f}(\pi)=\min_{\alpha:D_f(p\Vert d_0)\le\rho}\sum_{g=1}^{k}p_g\cdot\widehat{A}^{\mathrm{est}}_g(\pi)
\quad\text{with}\quad
\widehat{A}^{\mathrm{est}}_g(\pi)=\frac{1}{|\mathcal{D}_g|}\sum_{i\in\mathcal{D}_g}(2\pi(X_i)-1)\widehat{\Gamma}(X_i,W_i) A n , f est ( π ) = α : D f ( p ∥ d 0 ) ≤ ρ min g = 1 ∑ k p g ⋅ A g est ( π ) with A g est ( π ) = ∣ D g ∣ 1 i ∈ D g ∑ ( 2 π ( X i ) − 1 ) Γ ( X i , W i )
We first bound the term T 1 T_1 T 1 .
A f ( π ) − A ~ f e s t ( π ) ≤ max α : D f ( p ∥ d 0 ) ≤ ρ ∑ g = 1 k p g ⋅ ∣ A g ( π ) − A ~ g e s t ( π ) ∣ = max α : D f ( p ∥ d 0 ) ≤ ρ ∑ g = 1 k p g ⋅ ∣ E X ∼ P g [ ( 2 π ( X ) − 1 ) ( τ ( X ) − E W ∼ π o f f ( ⋅ ∣ X ) [ Γ ^ ( X , W ) ∣ X ] ) ] ∣ \begin{aligned}
A_f(\pi)-\widetilde{A}^{\mathrm{est}}_f(\pi)\le\max_{\alpha:D_f(p\Vert d_0)\le\rho}\sum_{g=1}^{k}p_g\cdot\left|A_g(\pi)-\widetilde{A}^{\mathrm{est}}_g(\pi)\right| \\
&=\max_{\alpha:D_f(p\Vert d_0)\le\rho}\sum_{g=1}^{k}p_g\cdot\left|\mathbb{E}_{X\sim P_g}\left[(2\pi(X)-1)\left(\tau(X)-\mathbb{E}_{W\sim\pi_{\mathrm{off}(\cdot\mid X)}}\left[\widehat{\Gamma}(X,W)\mid X\right]\right)\right]\right|
\end{aligned} A f ( π ) − A f est ( π ) ≤ α : D f ( p ∥ d 0 ) ≤ ρ max g = 1 ∑ k p g ⋅ A g ( π ) − A g est ( π ) = α : D f ( p ∥ d 0 ) ≤ ρ max g = 1 ∑ k p g ⋅ E X ∼ P g [ ( 2 π ( X ) − 1 ) ( τ ( X ) − E W ∼ π off ( ⋅ ∣ X ) [ Γ ( X , W ) ∣ X ] ) ]
The proof of Lemma 4 upper bounds Δ ( X ) \Delta(X) Δ ( X ) by 2 M κ n g − β g − γ g \frac{2M}{\kappa}n_g^{-\beta_g-\gamma_g} κ 2 M n g − β g − γ g . Substituting this upper bound, we obtain the following bound on T 1 T_1 T 1 .
T 1 = A f ( π ) − A ~ f e s t ( π ) ≤ max α : D f ( p ∥ d 0 ) ≤ ρ ∑ g = 1 k p g ⋅ 2 M κ 1 n g β g + γ g T_1=A_f(\pi)-\widetilde{A}^{\mathrm{est}}_f(\pi)\le\max_{\alpha:D_f(p\Vert d_0)\le\rho}\sum_{g=1}^{k}p_g\cdot\frac{2M}{\kappa}\frac{1}{n_g^{\beta_g+\gamma_g}} T 1 = A f ( π ) − A f est ( π ) ≤ α : D f ( p ∥ d 0 ) ≤ ρ max g = 1 ∑ k p g ⋅ κ 2 M n g β g + γ g 1
We now bound the term T 2 T_2 T 2 .
A ~ f e s t ( π ) − A ^ n , f e s t ( π ) ≤ max α : D f ( p ∥ d 0 ) ≤ ρ ∑ g = 1 k p g ⋅ ∣ A ~ g e s t ( π ) − A ^ g e s t ( π ) ∣ \widetilde{A}^{\mathrm{est}}_f(\pi)-\widehat{A}^{\mathrm{est}}_{n,f}(\pi)\le\max_{\alpha:D_f(p\Vert d_0)\le\rho}\sum_{g=1}^{k}p_g\cdot\left|\widetilde{A}^{\mathrm{est}}_g(\pi)-\widehat{A}^{\mathrm{est}}_g(\pi)\right| A f est ( π ) − A n , f est ( π ) ≤ α : D f ( p ∥ d 0 ) ≤ ρ max g = 1 ∑ k p g ⋅ A g est ( π ) − A g est ( π )
The term ∣ A ~ g e s t ( π ) − A ^ g e s t ( π ) ∣ \left|\widetilde{A}^{\mathrm{est}}_g(\pi)-\widehat{A}^{\mathrm{est}}_g(\pi)\right| A g est ( π ) − A g est ( π ) is the difference between population mean and sample mean where each summand is bounded by at most 2 M / κ 2M/\kappa 2 M / κ . Therefore, by Hoeffding’s inequality we have,
P ( ∣ A ~ g e s t ( π ) − A ^ g e s t ( π ) ∣ ≥ t ) ≤ 2 exp ( − t 2 n g κ 2 2 M 2 ) \mathbb{P}\left(\left|\widetilde{A}^{\mathrm{est}}_g(\pi)-\widehat{A}^{\mathrm{est}}_g(\pi)\right|\ge t\right)\le2\exp\left(-\frac{t^{2}n_g\kappa^{2}}{2M^{2}}\right) P ( A g est ( π ) − A g est ( π ) ≥ t ) ≤ 2 exp ( − 2 M 2 t 2 n g κ 2 )
Substituting t = M κ 2 u n g t = \frac{M}{\kappa}\sqrt{\frac{2u}{n_g}} t = κ M n g 2 u and a union bound over the k k k groups we obtain the following upper bound with probability at least 1 − 2 k e − u 1 - 2ke^{-u} 1 − 2 k e − u .
∣ A ~ f e s t ( π ) − A ^ n , f e s t ( π ) ∣ ≤ max p : D f ( p ∥ d 0 ) ≤ ρ ∑ g = 1 k p g M κ u n g ≤ M κ u n max p : D f ( p ∥ d 0 ) ≤ ρ ∑ g = 1 k p g d 0 ( g ) ≤ M κ u k n max p : D f ( p ∥ d 0 ) ≤ ρ ∑ g = 1 k p g 2 d 0 ( g ) \begin{aligned}
\left|\widetilde{A}^{\mathrm{est}}_f(\pi)-\widehat{A}^{\mathrm{est}}_{n,f}(\pi)\right|
\le\max_{p:D_f(p\|d_0)\le\rho}\sum_{g=1}^{k}p_g\frac{M}{\kappa}\sqrt{\frac{u}{n_g}} \\
&\le\frac{M}{\kappa}\sqrt{\frac{u}{n}}\max_{p:D_f(p\|d_0)\le\rho}\sum_{g=1}^{k}\frac{p_g}{\sqrt{d_0(g)}} \\
&\le\frac{M}{\kappa}\sqrt{\frac{uk}{n}}\max_{p:D_f(p\|d_0)\le\rho}\sqrt{\sum_{g=1}^{k}\frac{p_g^2}{d_0(g)}}
\end{aligned} A f est ( π ) − A n , f est ( π ) ≤ p : D f ( p ∥ d 0 ) ≤ ρ max g = 1 ∑ k p g κ M n g u ≤ κ M n u p : D f ( p ∥ d 0 ) ≤ ρ max g = 1 ∑ k d 0 ( g ) p g ≤ κ M n u k p : D f ( p ∥ d 0 ) ≤ ρ max g = 1 ∑ k d 0 ( g ) p g 2
Using Lemma 6 we obtain a bound of B f B_f B f on the bias term.
B f = ∣ A ~ f e s t ( π ) − A ^ n , f e s t ( π ) ∣ ≤ { M κ u k n ⋅ 1 α if f = C V a R at level α M κ u k n ⋅ 1 + 2 ρ if f = χ 2 M κ u k n ⋅ exp ( ρ ) if f = K L B_f=\left|\widetilde{A}^{\mathrm{est}}_f(\pi)-\widehat{A}^{\mathrm{est}}_{n,f}(\pi)\right|
\le
\begin{cases}
\frac{M}{\kappa}\sqrt{\frac{uk}{n}}\cdot\frac{1}{\alpha} & \text{if } f=\mathrm{CVaR}\text{ at level }\alpha\\
\frac{M}{\kappa}\sqrt{\frac{uk}{n}}\cdot\sqrt{1+2\rho} & \text{if } f=\chi^2 \\
\frac{M}{\kappa}\sqrt{\frac{uk}{n}}\cdot\exp(\rho) & \text{if } f=\mathrm{KL}
\end{cases} B f = A f est ( π ) − A n , f est ( π ) ≤ ⎩ ⎨ ⎧ κ M n u k ⋅ α 1 κ M n u k ⋅ 1 + 2 ρ κ M n u k ⋅ exp ( ρ ) if f = CVaR at level α if f = χ 2 if f = KL
In order to extend the above bound to the set of all policies we first bound the Lipschitz constant of A ^ n , f e s t ( π ) \widehat{A}^{\mathrm{est}}_{n,f}(\pi) A n , f est ( π ) with respect to π \pi π .
∣ A ^ n , f e s t ( π ) − A ^ n , f e s t ( π ′ ) ∣ ≤ max p : D f ( p ∥ d 0 ) ≤ ρ ∑ g = 1 k p g ∣ A ^ g e s t ( π ) − A ^ g e s t ( π ′ ) ∣ ≤ max p : D f ( p ∥ d 0 ) ≤ ρ ∑ g = 1 k p g 1 ∣ D g ∣ ∑ i ∈ D g 2 ∣ π ( X i ) − π ′ ( X i ) ∣ ∣ Γ ^ i ∣ ≤ 2 M κ ⋅ ∥ π − π ′ ∥ ∞ \begin{aligned}
\left|\widehat{A}^{\mathrm{est}}_{n,f}(\pi)-\widehat{A}^{\mathrm{est}}_{n,f}(\pi')\right|
\le\max_{p:D_f(p\|d_0)\le\rho}\sum_{g=1}^{k}p_g\left|\widehat{A}^{\mathrm{est}}_g(\pi)-\widehat{A}^{\mathrm{est}}_g(\pi')\right| \\
&\le\max_{p:D_f(p\|d_0)\le\rho}\sum_{g=1}^{k}p_g\frac{1}{|D_g|}\sum_{i\in D_g}2\left|\pi(X_i)-\pi'(X_i)\right|\left|\widehat{\Gamma}_i\right|
\le\frac{2M}{\kappa}\cdot\|\pi-\pi'\|_\infty
\end{aligned} A n , f est ( π ) − A n , f est ( π ′ ) ≤ p : D f ( p ∥ d 0 ) ≤ ρ max g = 1 ∑ k p g A g est ( π ) − A g est ( π ′ ) ≤ p : D f ( p ∥ d 0 ) ≤ ρ max g = 1 ∑ k p g ∣ D g ∣ 1 i ∈ D g ∑ 2 ∣ π ( X i ) − π ′ ( X i ) ∣ Γ i ≤ κ 2 M ⋅ ∥ π − π ′ ∥ ∞
Similarly, we can show that ∣ A ~ f e s t ( π ) − A ~ f e s t ( π ′ ) ∣ ≤ 2 M κ ⋅ ∥ π − π ′ ∥ ∞ \left|\widetilde{A}^{\mathrm{est}}_f(\pi)-\widetilde{A}^{\mathrm{est}}_f(\pi')\right|\le\frac{2M}{\kappa}\cdot\|\pi-\pi'\|_\infty A f est ( π ) − A f est ( π ′ ) ≤ κ 2 M ⋅ ∥ π − π ′ ∥ ∞ . This gives us
∣ ∣ A ~ f e s t ( π ) − A ^ n , f e s t ( π ) ∣ − ∣ A ~ f e s t ( π ′ ) − A ^ n , f e s t ( π ′ ) ∣ ∣ ≤ 4 M κ ⋅ ∥ π − π ′ ∥ ∞ . \left|\left|\widetilde{A}^{\mathrm{est}}_f(\pi)-\widehat{A}^{\mathrm{est}}_{n,f}(\pi)\right|-\left|\widetilde{A}^{\mathrm{est}}_f(\pi')-\widehat{A}^{\mathrm{est}}_{n,f}(\pi')\right|\right|
\le\frac{4M}{\kappa}\cdot\|\pi-\pi'\|_\infty. A f est ( π ) − A n , f est ( π ) − A f est ( π ′ ) − A n , f est ( π ′ ) ≤ κ 4 M ⋅ ∥ π − π ′ ∥ ∞ .
Now suppose { π 1 , … , π N } \{\pi_1,\ldots,\pi_N\} { π 1 , … , π N } is a ε \varepsilon ε -cover of the policy set Π \Pi Π . From Assumption 4, N ≤ ( 1 + L B / ε ) d N\le(1+LB/\varepsilon)^d N ≤ ( 1 + L B / ε ) d . By a union bound we obtain the following inequality with probability at least 1 − 2 N K e − u 1-2NK e^{-u} 1 − 2 N K e − u ,
max i ∈ [ N ] ∣ A ~ f e s t ( π i ) − A ^ n , f e s t ( π i ) ∣ ≤ B f . \max_{i\in[N]}\left|\widetilde{A}^{\mathrm{est}}_f(\pi_i)-\widehat{A}^{\mathrm{est}}_{n,f}(\pi_i)\right|\le B_f. i ∈ [ N ] max A f est ( π i ) − A n , f est ( π i ) ≤ B f .
Given any policy π \pi π , pick π ′ \pi' π ′ belonging to the cover that is closest to π \pi π . Then,
∣ A ~ f e s t ( π ) − A ^ n , f e s t ( π ) ∣ ≤ B f + 4 M κ ε . \left|\widetilde{A}^{\mathrm{est}}_f(\pi)-\widehat{A}^{\mathrm{est}}_{n,f}(\pi)\right|\le B_f+\frac{4M}{\kappa}\varepsilon. A f est ( π ) − A n , f est ( π ) ≤ B f + κ 4 M ε .
Choosing ε = B f κ 4 M \varepsilon=\frac{B_f\kappa}{4M} ε = 4 M B f κ we obtain that with probability at least 1 − 2 N K e − u 1-2NKe^{-u} 1 − 2 N K e − u ,
T 2 : = sup π ∈ Π ∣ A ~ f e s t ( π ) − A ^ n , f e s t ( π ) ∣ ≤ 2 B f . T_2:=\sup_{\pi\in\Pi}\left|\widetilde{A}^{\mathrm{est}}_f(\pi)-\widehat{A}^{\mathrm{est}}_{n,f}(\pi)\right|\le2B_f. T 2 := π ∈ Π sup A f est ( π ) − A n , f est ( π ) ≤ 2 B f .
Since ε = O ( κ M n ) \varepsilon=O\left(\frac{\kappa}{M\sqrt{n}}\right) ε = O ( M n κ ) , the failure probability is bounded by ( 1 + L B / ε ) d k e − u = O ( exp ( − u + log k + d log ( L B n M / κ ) ) ) (1+LB/\varepsilon)^dke^{-u}=O(\exp(-u+\log k+d\log(LBnM/\kappa))) ( 1 + L B / ε ) d k e − u = O ( exp ( − u + log k + d log ( L B n M / κ ))) . Now substituting u = log k + c d log ( L B n M / κ ) u=\log k+cd\log(LBnM/\kappa) u = log k + c d log ( L B n M / κ ) for some constant c > 1 c>1 c > 1 we obtain the desired upper bound on T 2 T_2 T 2 . □
Lemma 6. Suppose p ∈ Δ k p\in\Delta_k p ∈ Δ k be a distribution such that D f ( p ∥ d 0 ) ≤ ρ D_f(p\|d_0)\le\rho D f ( p ∥ d 0 ) ≤ ρ . Then we have,
( ∑ g = 1 k p g 2 d 0 ( g ) ) 1 / 2 ≤ { 1 / α if f = C V a R at level α 1 + 2 ρ if f = χ 2 exp ( ρ ) if f = K L \left(\sum_{g=1}^{k}\frac{p_g^2}{d_0(g)}\right)^{1/2}
\le
\begin{cases}
1/\alpha& \text{if } f=\mathrm{CVaR}\text{ at level }\alpha\\
\sqrt{1+2\rho} & \text{if } f=\chi^2 \\
\exp(\rho) & \text{if } f=\mathrm{KL}
\end{cases} ( g = 1 ∑ k d 0 ( g ) p g 2 ) 1/2 ≤ ⎩ ⎨ ⎧ 1/ α 1 + 2 ρ exp ( ρ ) if f = CVaR at level α if f = χ 2 if f = KL
Proof. For CVaR constraint at level α \alpha α , each entry of p g p_g p g cannot be more than d 0 ( g ) / α d_0(g)/\alpha d 0 ( g ) / α . This gives us
∑ g p g 2 d 0 ( g ) ≤ 1 α . \sqrt{\sum_g \frac{p_g^2}{d_0(g)}} \le\frac{1}{\alpha}. g ∑ d 0 ( g ) p g 2 ≤ α 1 .
For χ 2 \chi^2 χ 2 divergence, D f ( p ∥ d 0 ) = ∑ g d 0 ( g ) 2 ( p g / d 0 ( g ) − 1 ) 2 = 1 / 2 ∑ g p g 2 / d 0 ( g ) − 1 / 2 ≤ ρ D_f(p\|d_0)=\sum_g \frac{d_0(g)}{2}(p_g/d_0(g)-1)^2=1/2\sum_g p_g^2/d_0(g)-1/2\le\rho D f ( p ∥ d 0 ) = ∑ g 2 d 0 ( g ) ( p g / d 0 ( g ) − 1 ) 2 = 1/2 ∑ g p g 2 / d 0 ( g ) − 1/2 ≤ ρ . After rearranging we obtain, ∑ g p g 2 / d 0 ( g ) ≤ 1 + 2 ρ \sqrt{\sum_g p_g^2/d_0(g)}\le\sqrt{1+2\rho} ∑ g p g 2 / d 0 ( g ) ≤ 1 + 2 ρ .
For KL divergence, we are given ∑ g p g log ( p g / d 0 ( g ) ) ≤ ρ \sum_g p_g\log(p_g/d_0(g))\le\rho ∑ g p g log ( p g / d 0 ( g )) ≤ ρ . Suppose there is a subset S ⊆ [ k ] S\subseteq[k] S ⊆ [ k ] with all entries of p p p with mass at least exp ( ρ ) ⋅ d 0 ( g ) \exp(\rho)\cdot d_0(g) exp ( ρ ) ⋅ d 0 ( g ) and ∑ g ∈ S d 0 ( g ) ≥ M \sum_{g\in S}d_0(g)\ge M ∑ g ∈ S d 0 ( g ) ≥ M . Then ∑ g p g log ( p g / d 0 ( g ) ) ≥ M ρ exp ( ρ ) \sum_g p_g\log(p_g/d_0(g))\ge M\rho\exp(\rho) ∑ g p g log ( p g / d 0 ( g )) ≥ M ρ exp ( ρ ) . This implies that M M M is at most exp ( − ρ ) \exp(-\rho) exp ( − ρ ) . Additionally, since ∑ g p g ≤ 1 \sum_g p_g\le1 ∑ g p g ≤ 1 for all entries in S S S we must have p g / d g = exp ( ρ ) p_g/d_g=\exp(\rho) p g / d g = exp ( ρ ) . Therefore,
∑ g p g 2 d 0 ( g ) ≤ exp ( 2 ρ ) ∑ g d g = exp ( 2 ρ ) . \sum_g\frac{p_g^2}{d_0(g)}\le\exp(2\rho)\sum_g d_g=\exp(2\rho). g ∑ d 0 ( g ) p g 2 ≤ exp ( 2 ρ ) g ∑ d g = exp ( 2 ρ ) .
Fast Rate of Convergence with KL-Divergence
Recall that the dual optimization problem is given as
min π ∈ Π max λ ≥ 0 , η ∈ R − λ ⋅ E X ∼ P 0 [ f ⋆ ( − ( 2 π ( X ) − 1 ) Γ ( X , W ) − η λ ) ] − λ ρ − η = : g λ , η ( π ) . \min_{\pi\in\Pi}\max_{\lambda\ge0,\eta\in\mathbb{R}}-\lambda\cdot\mathbb{E}_{X\sim P_0}\left[f^\star\left(\frac{-(2\pi(X)-1)\Gamma(X,W)-\eta}{\lambda}\right)\right]-\lambda\rho-\eta=:g_{\lambda,\eta}(\pi). π ∈ Π min λ ≥ 0 , η ∈ R max − λ ⋅ E X ∼ P 0 [ f ⋆ ( λ − ( 2 π ( X ) − 1 ) Γ ( X , W ) − η ) ] − λ ρ − η =: g λ , η ( π ) .
The proof is divided into four main parts.
Bounding the Deviation of the Dual Parameters at π b ⋆ \pi_b^\star π b ⋆
We will first consider the Bayes Optimal policy π b ⋆ \pi_b^\star π b ⋆ for which it can be checked that ( 2 π b ⋆ ( X ) − 1 ) τ ( X ) = ∣ τ ( X ) ∣ (2\pi_b^\star(X)-1)\tau(X)=|\tau(X)| ( 2 π b ⋆ ( X ) − 1 ) τ ( X ) = ∣ τ ( X ) ∣ . The KL dual has the two-parameter Lagrangian form
g λ , η ( π b ⋆ ) = − λ e − 1 E [ e ( − ∣ τ ( X ) ∣ − η ) / λ ] − λ ρ − η , (18) g_{\lambda,\eta}(\pi_b^\star)=-\lambda e^{-1}\mathbb{E}\left[e^{(-|\tau(X)|-\eta)/\lambda}\right]-\lambda\rho-\eta, \tag*{(18)} g λ , η ( π b ⋆ ) = − λ e − 1 E [ e ( − ∣ τ ( X ) ∣ − η ) / λ ] − λ ρ − η , (18)
with population optimizer θ ⋆ = ( λ ⋆ , η ⋆ ) = arg max λ > 0 , η ∈ R g λ , η ( π b ⋆ ) \theta^\star=(\lambda^\star,\eta^\star)=\mathop{\arg\max}_{\lambda>0,\eta\in\mathbb{R}}g_{\lambda,\eta}(\pi_b^\star) θ ⋆ = ( λ ⋆ , η ⋆ ) = arg max λ > 0 , η ∈ R g λ , η ( π b ⋆ ) , so that A ^ K L ( π b ⋆ ) = g λ ⋆ , η ⋆ ( π b ⋆ ) \widehat{A}_{\mathrm{KL}}(\pi_b^\star)=g_{\lambda^\star,\eta^\star}(\pi_b^\star) A KL ( π b ⋆ ) = g λ ⋆ , η ⋆ ( π b ⋆ ) . The empirical dual is
g ^ λ , η ( π b ⋆ ) = − λ e − 1 n ∑ i = 1 n e ( − ∣ τ ( X i ) ∣ − η ) / λ − λ ρ − η , \widehat{g}_{\lambda,\eta}(\pi_b^\star)=-\frac{\lambda e^{-1}}{n}\sum_{i=1}^{n}e^{(-|\tau(X_i)|-\eta)/\lambda}-\lambda\rho-\eta, g λ , η ( π b ⋆ ) = − n λ e − 1 i = 1 ∑ n e ( − ∣ τ ( X i ) ∣ − η ) / λ − λ ρ − η ,
with empirical optimizer θ ^ n = ( λ ^ n , η ^ n ) \widehat{\theta}_n=(\widehat{\lambda}_n,\widehat{\eta}_n) θ n = ( λ n , η n ) . We first prove a useful lemma.
Lemma 7. Suppose Assumptions 7 and 8 hold. Let Q ⋆ Q^\star Q ⋆ be the exponentially tilted distribution defined as d Q ⋆ d P 0 = exp ( − ( ∣ τ ( X ) ∣ + η ⋆ ) / λ ⋆ − 1 ) \frac{dQ^\star}{dP_0}=\exp(-(|\tau(X)|+\eta^\star)/\lambda^\star-1) d P 0 d Q ⋆ = exp ( − ( ∣ τ ( X ) ∣ + η ⋆ ) / λ ⋆ − 1 ) . Then
λ ⋆ ≤ 2 M / ρ \lambda^\star\le2M/\rho λ ⋆ ≤ 2 M / ρ , and ∣ η ⋆ ∣ ≤ M |\eta^\star|\le M ∣ η ⋆ ∣ ≤ M .
Var Q ⋆ ( ∣ τ ( X ) ∣ ) ≥ e − ( 1 + ρ ) ⋅ σ 0 2 \operatorname{Var}_{Q^\star}(|\tau(X)|)\ge e^{-(1+\rho)}\cdot\sigma_0^2 Var Q ⋆ ( ∣ τ ( X ) ∣ ) ≥ e − ( 1 + ρ ) ⋅ σ 0 2 .
Proof. Note that the optimal dual variable η ⋆ \eta^\star η ⋆ can be expressed as
η ⋆ = λ ⋆ ⋅ log E [ exp ( − ∣ τ ( X ) ∣ λ ⋆ ) ] (19) \eta^\star=\lambda^\star\cdot\log\mathbb{E}\left[\exp\left(-\frac{|\tau(X)|}{\lambda^\star}\right)\right] \tag*{(19)} η ⋆ = λ ⋆ ⋅ log E [ exp ( − λ ⋆ ∣ τ ( X ) ∣ ) ] (19)
Substituting the optimal value of η ⋆ \eta^\star η ⋆ we obtain the following optimization problem.
max λ ≥ 0 − λ ρ − λ ⋅ log E [ exp ( − ∣ τ ( X ) ∣ λ ) ] \max_{\lambda\ge0}-\lambda\rho-\lambda\cdot\log\mathbb{E}\left[\exp\left(-\frac{|\tau(X)|}{\lambda}\right)\right] λ ≥ 0 max − λ ρ − λ ⋅ log E [ exp ( − λ ∣ τ ( X ) ∣ ) ]
By Assumption 7 the optimal value of λ \lambda λ is positive and the first derivative is zero i.e.
− ρ − log E [ exp ( − ∣ τ ( X ) ∣ λ ⋆ ) ] − 1 λ ⋆ ⋅ E [ ∣ τ ( X ) ∣ exp ( − ∣ τ ( X ) ∣ / λ ⋆ ) ] E [ exp ( − ∣ τ ( X ) ∣ / λ ⋆ ) ] = 0 -\rho-\log\mathbb{E}\left[\exp\left(-\frac{|\tau(X)|}{\lambda^\star}\right)\right]-\frac{1}{\lambda^\star}\cdot\frac{\mathbb{E}\left[|\tau(X)|\exp\left(-|\tau(X)|/\lambda^\star\right)\right]}{\mathbb{E}\left[\exp\left(-|\tau(X)|/\lambda^\star\right)\right]}=0 − ρ − log E [ exp ( − λ ⋆ ∣ τ ( X ) ∣ ) ] − λ ⋆ 1 ⋅ E [ exp ( − ∣ τ ( X ) ∣/ λ ⋆ ) ] E [ ∣ τ ( X ) ∣ exp ( − ∣ τ ( X ) ∣/ λ ⋆ ) ] = 0
Let us define Q 0 ⋆ Q_0^\star Q 0 ⋆ as the distribution with density d Q 0 ⋆ d P 0 = e − ∣ τ ( X ) ∣ / λ ⋆ − 1 \frac{dQ_0^\star}{dP_0}=e^{-|\tau(X)|/\lambda^\star-1} d P 0 d Q 0 ⋆ = e − ∣ τ ( X ) ∣/ λ ⋆ − 1 . Then
λ ⋆ ≤ E [ ∣ τ ( X ) ∣ ] − E Q 0 ⋆ [ ∣ τ ( X ) ∣ ] ρ \lambda^\star\leq\frac{\mathbb{E}[|\tau(X)|]-\mathbb{E}_{Q_0^\star}[|\tau(X)|]}{\rho} λ ⋆ ≤ ρ E [ ∣ τ ( X ) ∣ ] − E Q 0 ⋆ [ ∣ τ ( X ) ∣ ]
This implies λ ⋆ ≤ 2 M / ρ \lambda^\star\leq2M/\rho λ ⋆ ≤ 2 M / ρ . Furthermore, using eq. (19) we obtain ∣ η ⋆ ∣ < M |\eta^\star|<M ∣ η ⋆ ∣ < M .
Let Q ⋆ Q^\star Q ⋆ be the exponentially tilted distribution d Q ⋆ d P 0 = e ( − ∣ τ ( X ) ∣ − η ⋆ ) / λ ⋆ − 1 ≥ e − ( 1 + ρ ) \frac{dQ^\star}{dP_0}=e^{(-|\tau(X)|-\eta^\star)/\lambda^\star-1}\geq e^{-(1+\rho)} d P 0 d Q ⋆ = e ( − ∣ τ ( X ) ∣ − η ⋆ ) / λ ⋆ − 1 ≥ e − ( 1 + ρ ) .
Var Q ⋆ ( ∣ τ ( X ) ∣ ) = E Q ⋆ [ ( ∣ τ ( X ) ∣ − E Q ⋆ [ ∣ τ ( X ) ∣ ] ) 2 ] ≥ e − ( 1 + ρ ) ⋅ E P 0 [ ( ∣ τ ( X ) ∣ − E Q ⋆ [ ∣ τ ( X ) ∣ ] ) 2 ] ≥ e − ( 1 + ρ ) ⋅ Var P 0 ( ∣ τ ( X ) ∣ ) ≥ e − ( 1 + ρ ) ⋅ σ 0 2 \begin{aligned}
\operatorname{Var}_{Q^\star}(|\tau(X)|)=\mathbb{E}_{Q^\star}\left[\left(|\tau(X)|-\mathbb{E}_{Q^\star}[|\tau(X)|]\right)^2\right] \\
&\geq e^{-(1+\rho)}\cdot\mathbb{E}_{P_0}\left[\left(|\tau(X)|-\mathbb{E}_{Q^\star}[|\tau(X)|]\right)^2\right] \\
&\geq e^{-(1+\rho)}\cdot\operatorname{Var}_{P_0}(|\tau(X)|)\geq e^{-(1+\rho)}\cdot\sigma_0^2
\end{aligned} Var Q ⋆ ( ∣ τ ( X ) ∣ ) = E Q ⋆ [ ( ∣ τ ( X ) ∣ − E Q ⋆ [ ∣ τ ( X ) ∣ ] ) 2 ] ≥ e − ( 1 + ρ ) ⋅ E P 0 [ ( ∣ τ ( X ) ∣ − E Q ⋆ [ ∣ τ ( X ) ∣ ] ) 2 ] ≥ e − ( 1 + ρ ) ⋅ Var P 0 ( ∣ τ ( X ) ∣ ) ≥ e − ( 1 + ρ ) ⋅ σ 0 2
Because of Lemma 7 we can restrict our attention to the compact set Θ = [ 0 , 2 M / ρ ] × [ − M , M ] \Theta=[0,2M/\rho]\times[-M,M] Θ = [ 0 , 2 M / ρ ] × [ − M , M ] for the dual parameters. We now show that the dual function is strongly concave over this set.
Lemma 8 (Gradient concentration). Suppose Assumption 7 holds. Then for any t > 0 t>0 t > 0 we have,
P ( ∥ ∇ g ^ θ ⋆ ( π b ⋆ ) ∥ ≥ t ) ≤ 4 exp ( − n t 2 4 B 2 ) \mathbb{P}\left(\left\|\nabla\widehat{g}_{\theta^\star}(\pi_b^\star)\right\|\geq t\right)\leq4\exp\left(-\frac{nt^2}{4B^2}\right) P ( ∥ ∇ g θ ⋆ ( π b ⋆ ) ∥ ≥ t ) ≤ 4 exp ( − 4 B 2 n t 2 ) where B = 8 M λ 0 ⋅ exp ( 4 M / λ 0 ) B=\frac{8M}{\lambda_0}\cdot\exp(4M/\lambda_0) B = λ 0 8 M ⋅ exp ( 4 M / λ 0 ) .
Proof. By Assumption 7 we can assume that θ ⋆ \theta^\star θ ⋆ belongs to the interior of the set Θ = [ λ 0 / 4 , 2 M / ρ ] × [ − M , M ] \Theta=[\lambda_0/4,2M/\rho]\times[-M,M] Θ = [ λ 0 /4 , 2 M / ρ ] × [ − M , M ] and ∇ G ( θ ⋆ ) = 0 \nabla G(\theta^\star)=0 ∇ G ( θ ⋆ ) = 0 . Furthermore, ∇ g ^ θ ⋆ ( π b ⋆ ) = 1 n ∑ i = 1 n s ( X i ) \nabla\widehat{g}_{\theta^\star}(\pi_b^\star)=\frac{1}{n}\sum_{i=1}^n s(X_i) ∇ g θ ⋆ ( π b ⋆ ) = n 1 ∑ i = 1 n s ( X i ) , where
s ( X i ) = ∇ θ G ( θ ⋆ ; X i ) = [ − 1 ϵ ⋅ ( 1 + η ⋆ + ∣ τ ( X i ) ∣ λ ⋆ ) exp ( − ∣ τ ( X i ) ∣ − η ⋆ λ ⋆ ) − ρ 1 ϵ ⋅ exp ( − ∣ τ ( X i ) ∣ − η ⋆ λ ⋆ ) − 1 ] s(X_i)=\nabla_\theta G(\theta^\star;X_i)=
\begin{bmatrix}
-\frac{1}{\epsilon}\cdot\left(1+\frac{\eta^\star+|\tau(X_i)|}{\lambda^\star}\right)\exp\left(\frac{-|\tau(X_i)|-\eta^\star}{\lambda^\star}\right)-\rho\\
\frac{1}{\epsilon}\cdot\exp\left(\frac{-|\tau(X_i)|-\eta^\star}{\lambda^\star}\right)-1
\end{bmatrix} s ( X i ) = ∇ θ G ( θ ⋆ ; X i ) = − ϵ 1 ⋅ ( 1 + λ ⋆ η ⋆ + ∣ τ ( X i ) ∣ ) exp ( λ ⋆ − ∣ τ ( X i ) ∣ − η ⋆ ) − ρ ϵ 1 ⋅ exp ( λ ⋆ − ∣ τ ( X i ) ∣ − η ⋆ ) − 1
Since ∣ τ ( X i ) ∣ ≤ M |\tau(X_i)|\leq M ∣ τ ( X i ) ∣ ≤ M we have ∣ ∂ g λ ⋆ , η ⋆ ( π b ⋆ ) ∂ λ ∣ ≤ 8 M λ 0 ⋅ exp ( 4 M / λ 0 ) : = B λ \left|\frac{\partial g_{\lambda^\star,\eta^\star}(\pi_b^\star)}{\partial\lambda}\right|\leq\frac{8M}{\lambda_0}\cdot\exp(4M/\lambda_0):=B_\lambda ∂ λ ∂ g λ ⋆ , η ⋆ ( π b ⋆ ) ≤ λ 0 8 M ⋅ exp ( 4 M / λ 0 ) := B λ , and ∣ ∂ g λ ⋆ , η ⋆ ( π b ⋆ ) ∂ η ∣ ≤ exp ( 4 M / λ 0 ) : = B η \left|\frac{\partial g_{\lambda^\star,\eta^\star}(\pi_b^\star)}{\partial\eta}\right|\leq\exp(4M/\lambda_0):=B_\eta ∂ η ∂ g λ ⋆ , η ⋆ ( π b ⋆ ) ≤ exp ( 4 M / λ 0 ) := B η .
Therefore, ∇ g ^ θ ⋆ ( π b ⋆ ) \nabla\widehat{g}_{\theta^\star}(\pi_b^\star) ∇ g θ ⋆ ( π b ⋆ ) is an average of mean-zero bounded vectors. We can bound ∇ g ^ θ ⋆ ( π b ⋆ ) \nabla\widehat{g}_{\theta^\star}(\pi_b^\star) ∇ g θ ⋆ ( π b ⋆ ) by applying Hoeffding’s inequality to each coordinate:
P ( ∥ ∇ g ^ θ ⋆ ( π b ⋆ ) ∥ ≥ t ) ≤ P ( ∣ ∇ λ g ^ θ ⋆ ( π b ⋆ ) ∣ ≥ t 2 ) + P ( ∣ ∇ η g ^ θ ⋆ ( π b ⋆ ) ∣ ≥ t 2 ) \mathbb{P}\left(\left\|\nabla\widehat{g}_{\theta^\star}(\pi_b^\star)\right\|\geq t\right)
\leq
\mathbb{P}\left(\left|\nabla_\lambda\widehat{g}_{\theta^\star}(\pi_b^\star)\right|\geq\frac{t}{\sqrt{2}}\right)
+
\mathbb{P}\left(\left|\nabla_\eta\widehat{g}_{\theta^\star}(\pi_b^\star)\right|\geq\frac{t}{\sqrt{2}}\right) P ( ∥ ∇ g θ ⋆ ( π b ⋆ ) ∥ ≥ t ) ≤ P ( ∣ ∇ λ g θ ⋆ ( π b ⋆ ) ∣ ≥ 2 t ) + P ( ∣ ∇ η g θ ⋆ ( π b ⋆ ) ∣ ≥ 2 t )
≤ 2 exp ( − n t 2 4 B λ 2 ) + 2 exp ( − n t 2 4 B η 2 ) ≤ 4 exp ( − n t 2 4 B 2 ) \leq2\exp\left(-\frac{nt^2}{4B_\lambda^2}\right)+2\exp\left(-\frac{nt^2}{4B_\eta^2}\right)\leq4\exp\left(-\frac{nt^2}{4B^2}\right) ≤ 2 exp ( − 4 B λ 2 n t 2 ) + 2 exp ( − 4 B η 2 n t 2 ) ≤ 4 exp ( − 4 B 2 n t 2 )
Lemma 9 (Hessian concentration). Suppose Assumption 7 holds. Then for any θ = ( λ , η ) ∈ [ λ 0 / 2 , 2 M / ρ ] × [ − M , M ] \theta=(\lambda,\eta)\in[\lambda_0/2,2M/\rho]\times[-M,M] θ = ( λ , η ) ∈ [ λ 0 /2 , 2 M / ρ ] × [ − M , M ] , we have,
∥ ∇ 2 g ^ θ ( π b ⋆ ) − ∇ 2 g θ ( π b ⋆ ) ∥ o p ≤ t \left\|\nabla^2\widehat{g}_{\theta}(\pi_b^\star)-\nabla^2g_{\theta}(\pi_b^\star)\right\|_{\mathrm{op}}\leq t ∇ 2 g θ ( π b ⋆ ) − ∇ 2 g θ ( π b ⋆ ) op ≤ t with probability at least 1 − C ⋅ exp ( − n t 2 λ 0 6 M 4 log M ρ λ 0 t ) 1-C\cdot\exp\left(-\frac{nt^2\lambda_0^6}{M^4}\log\frac{M}{\rho\lambda_0t}\right) 1 − C ⋅ exp ( − M 4 n t 2 λ 0 6 log ρ λ 0 t M ) for a universal constant C > 0 C>0 C > 0 .
Proof. Let us define μ k ( λ , η ) = E [ ( ∣ τ ( X ) ∣ + η ) k e ( − ∣ τ ( X ) ∣ − η ) / λ ] \mu_k(\lambda,\eta)=\mathbb{E}\left[(|\tau(X)|+\eta)^k e^{(-|\tau(X)|-\eta)/\lambda}\right] μ k ( λ , η ) = E [ ( ∣ τ ( X ) ∣ + η ) k e ( − ∣ τ ( X ) ∣ − η ) / λ ] for k = 0 , 1 , 2 k=0,1,2 k = 0 , 1 , 2 . The Hessian of g g g at a general point ( λ , η ) (\lambda,\eta) ( λ , η ) is
∇ 2 g λ , η ( π b ⋆ ) = ( − e − 1 μ 2 / λ 3 e − 1 μ 1 / λ 2 e − 1 μ 1 / λ 2 − e − 1 μ 0 / λ ) . \nabla^2 g_{\lambda,\eta}(\pi_b^\star)=\begin{pmatrix}-e^{-1}\mu_2/\lambda^3 & e^{-1}\mu_1/\lambda^2 \\ e^{-1}\mu_1/\lambda^2 & -e^{-1}\mu_0/\lambda\end{pmatrix}. ∇ 2 g λ , η ( π b ⋆ ) = ( − e − 1 μ 2 / λ 3 e − 1 μ 1 / λ 2 e − 1 μ 1 / λ 2 − e − 1 μ 0 / λ ) .
Additionally, let us define μ k i = ( ∣ τ ( X i ) ∣ + η ) k exp ( ( − ∣ τ ( X i ) − η ∣ ) / λ ) \mu_k^i=(|\tau(X_i)|+\eta)^k\exp((-|\tau(X_i)-\eta|)/\lambda) μ k i = ( ∣ τ ( X i ) ∣ + η ) k exp (( − ∣ τ ( X i ) − η ∣ ) / λ ) for k = 0 , 1 , 2 k=0,1,2 k = 0 , 1 , 2 and i = 1 , … , n i=1,\ldots,n i = 1 , … , n . Then ∇ 2 g ^ θ ( π b ⋆ ) = 1 n ∑ i = 1 n S ( X i ) \nabla^2\hat{g}_\theta(\pi_b^\star)=\frac{1}{n}\sum_{i=1}^n S(X_i) ∇ 2 g ^ θ ( π b ⋆ ) = n 1 ∑ i = 1 n S ( X i ) where
S ( X i ) = ( − e − 1 μ 2 i / λ 3 e − 1 μ 1 i / λ 2 e − 1 μ 1 i / λ 2 − e − 1 μ 0 i / λ ) . S(X_i)=\begin{pmatrix}-e^{-1}\mu_2^i/\lambda^3 & e^{-1}\mu_1^i/\lambda^2 \\ e^{-1}\mu_1^i/\lambda^2 & -e^{-1}\mu_0^i/\lambda\end{pmatrix}. S ( X i ) = ( − e − 1 μ 2 i / λ 3 e − 1 μ 1 i / λ 2 e − 1 μ 1 i / λ 2 − e − 1 μ 0 i / λ ) .
By Assumption 7, each entry of S ( X i ) S(X_i) S ( X i ) is bounded by at most 4 M 2 / λ 0 3 = : B 4M^2/\lambda_0^3=:B 4 M 2 / λ 0 3 =: B . Therefore by Hoeffding’s inequality and union bound, for any fixed θ \theta θ we have
P ( ∥ ∇ 2 g ^ θ ( π b ⋆ ) − ∇ 2 g θ ( π b ⋆ ) ∥ o p ≥ t ) ≤ P ( ∥ ∇ 2 g ^ θ ( π b ⋆ ) − ∇ 2 g θ ( π b ⋆ ) ∥ F ≥ t ) ≤ P ( ∃ j , k ∣ ∇ 2 g ^ θ ( π b ⋆ ) j k − ∇ 2 g θ ( π b ⋆ ) j k ∣ ≥ t ) ≤ 8 exp ( − n t 2 32 B 2 ) . \begin{aligned}
P\left(\left\|\nabla^2\hat{g}_\theta(\pi_b^\star)-\nabla^2g_\theta(\pi_b^\star)\right\|_{\mathrm{op}}\ge t\right)
\le P\left(\left\|\nabla^2\hat{g}_\theta(\pi_b^\star)-\nabla^2g_\theta(\pi_b^\star)\right\|_{\mathrm{F}}\ge t\right) \\
&\le P\left(\exists j,k\ \left|\nabla^2\hat{g}_\theta(\pi_b^\star)_{jk}-\nabla^2g_\theta(\pi_b^\star)_{jk}\right|\ge t\right)
\le8\exp\left(-\frac{nt^2}{32B^2}\right).
\end{aligned} P ( ∇ 2 g ^ θ ( π b ⋆ ) − ∇ 2 g θ ( π b ⋆ ) op ≥ t ) ≤ P ( ∇ 2 g ^ θ ( π b ⋆ ) − ∇ 2 g θ ( π b ⋆ ) F ≥ t ) ≤ P ( ∃ j , k ∇ 2 g ^ θ ( π b ⋆ ) j k − ∇ 2 g θ ( π b ⋆ ) j k ≥ t ) ≤ 8 exp ( − 32 B 2 n t 2 ) .
We now extend this pointwise concentration to a uniform concentration over Θ = [ λ 0 / 2 , 2 M / ρ ] × [ − M , M ] \Theta=[\lambda_0/2,2M/\rho]\times[-M,M] Θ = [ λ 0 /2 , 2 M / ρ ] × [ − M , M ] . First we derive an upper bound on the Lipschitz constant of ∇ 2 g ^ θ ( π b ⋆ ) \nabla^2\hat{g}_\theta(\pi_b^\star) ∇ 2 g ^ θ ( π b ⋆ ) . Note that,
∂ 3 g λ , η ( π b ⋆ ) ∂ λ 3 = 3 e − 1 μ 2 λ 4 − e − 1 μ 3 λ 5 and ∂ 3 g λ , η ( π b ⋆ ) ∂ λ 2 ∂ η = − 2 e − 1 μ 1 λ 3 + e − 1 μ 2 λ 4 . \frac{\partial^3 g_{\lambda,\eta}(\pi_b^\star)}{\partial\lambda^3}=\frac{3e^{-1}\mu_2}{\lambda^4}-\frac{e^{-1}\mu_3}{\lambda^5}
\quad\text{and}\quad
\frac{\partial^3 g_{\lambda,\eta}(\pi_b^\star)}{\partial\lambda^2\partial\eta}=-\frac{2e^{-1}\mu_1}{\lambda^3}+\frac{e^{-1}\mu_2}{\lambda^4}. ∂ λ 3 ∂ 3 g λ , η ( π b ⋆ ) = λ 4 3 e − 1 μ 2 − λ 5 e − 1 μ 3 and ∂ λ 2 ∂ η ∂ 3 g λ , η ( π b ⋆ ) = − λ 3 2 e − 1 μ 1 + λ 4 e − 1 μ 2 .
Therefore, assumption 1 we obtain that max { ∣ ∂ 3 g λ , η ( π b ⋆ ) ∂ λ 3 ∣ , ∣ ∂ 3 g λ , η ( π b ⋆ ) ∂ λ 2 ∂ η ∣ } ≤ C ⋅ M 3 λ 0 5 \max\left\{\left|\frac{\partial^3g_{\lambda,\eta}(\pi_b^\star)}{\partial\lambda^3}\right|,\left|\frac{\partial^3g_{\lambda,\eta}(\pi_b^\star)}{\partial\lambda^2\partial\eta}\right|\right\}\le\frac{C\cdot M^3}{\lambda_0^5} max { ∂ λ 3 ∂ 3 g λ , η ( π b ⋆ ) , ∂ λ 2 ∂ η ∂ 3 g λ , η ( π b ⋆ ) } ≤ λ 0 5 C ⋅ M 3 for a large constant C C C . Similarly, we can bound the other third order partial derivatives of g g g by C ⋅ M 3 λ 0 5 \frac{C\cdot M^3}{\lambda_0^5} λ 0 5 C ⋅ M 3 . Therefore, ∇ 2 g ^ θ ( π b ⋆ ) \nabla^2\hat{g}_\theta(\pi_b^\star) ∇ 2 g ^ θ ( π b ⋆ ) is Lipschitz in θ \theta θ with Lipschitz constant at most C ⋅ M 3 λ 0 5 = : L C\cdot\frac{M^3}{\lambda_0^5}=:L C ⋅ λ 0 5 M 3 =: L .
Now, suppose N = N ( Θ , t 3 L , ∥ ⋅ ∥ ∞ ) N=\mathcal{N}\left(\Theta,\frac{t}{3L},\|\cdot\|_\infty\right) N = N ( Θ , 3 L t , ∥ ⋅ ∥ ∞ ) be a t 3 L \frac{t}{3L} 3 L t -covering number of Θ \Theta Θ with respect to the ℓ ∞ \ell_\infty ℓ ∞ norm. Then for any θ ∈ Θ \theta\in\Theta θ ∈ Θ , there exists θ 1 , … , θ N \theta_1,\ldots,\theta_N θ 1 , … , θ N such that for any θ ′ \theta' θ ′ there exists θ i \theta_i θ i such that ∥ θ ′ − θ i ∥ ∞ ≤ t 3 L \|\theta'-\theta_i\|_\infty\le\frac{t}{3L} ∥ θ ′ − θ i ∥ ∞ ≤ 3 L t . Furthermore, this gives us,
∥ ∇ 2 g ^ θ ′ − ∇ 2 g θ ′ ∥ o p ≤ ∥ ∇ 2 g ^ θ ′ − ∇ 2 g ^ θ i ∥ o p + ∥ ∇ 2 g ^ θ i − ∇ 2 g θ i ∥ o p + ∥ ∇ 2 g ( θ i ) − ∇ 2 g ( θ ′ ) ∥ o p ≤ 2 t 3 + ∥ ∇ 2 g ^ n ( θ i ) − ∇ 2 g ( θ i ) ∥ o p . \begin{aligned}
\left\|\nabla^2\hat{g}_{\theta'}-\nabla^2g_{\theta'}\right\|_{\mathrm{op}}
\le\left\|\nabla^2\hat{g}_{\theta'}-\nabla^2\hat{g}_{\theta_i}\right\|_{\mathrm{op}}+\left\|\nabla^2\hat{g}_{\theta_i}-\nabla^2g_{\theta_i}\right\|_{\mathrm{op}}+\left\|\nabla^2g(\theta_i)-\nabla^2g(\theta')\right\|_{\mathrm{op}} \\
&\le\frac{2t}{3}+\left\|\nabla^2\hat{g}_n(\theta_i)-\nabla^2g(\theta_i)\right\|_{\mathrm{op}}.
\end{aligned} ∇ 2 g ^ θ ′ − ∇ 2 g θ ′ op ≤ ∇ 2 g ^ θ ′ − ∇ 2 g ^ θ i op + ∇ 2 g ^ θ i − ∇ 2 g θ i op + ∇ 2 g ( θ i ) − ∇ 2 g ( θ ′ ) op ≤ 3 2 t + ∇ 2 g ^ n ( θ i ) − ∇ 2 g ( θ i ) op .
Now by a union bound over { θ 1 , … , θ N } \{\theta_1,\ldots,\theta_N\} { θ 1 , … , θ N } , we can guarantee that ∥ ∇ 2 g ^ n ( θ i ) − ∇ 2 g ( θ i ) ∥ o p ≤ t 3 \left\|\nabla^2\hat{g}_n(\theta_i)-\nabla^2g(\theta_i)\right\|_{\mathrm{op}}\le\frac{t}{3} ∇ 2 g ^ n ( θ i ) − ∇ 2 g ( θ i ) op ≤ 3 t for all i i i with probability at least 1 − 8 N ⋅ exp ( − n t 2 / ( 288 B 2 ) ) 1-8N\cdot\exp(-nt^2/(288B^2)) 1 − 8 N ⋅ exp ( − n t 2 / ( 288 B 2 )) . Since, ∣ N ∣ ≤ 36 M L 2 ρ t 2 = C ⋅ M 7 ρ t 2 λ 0 10 |N|\le\frac{36ML^2}{\rho t^2}=C\cdot\frac{M^7}{\rho t^2\lambda_0^{10}} ∣ N ∣ ≤ ρ t 2 36 M L 2 = C ⋅ ρ t 2 λ 0 10 M 7 the probability of failure is at most C ⋅ M 7 ρ t 2 λ 0 10 ⋅ exp ( − n t 2 / ( 288 B 2 ) ) = C ⋅ exp ( − n t 2 λ 0 6 M 4 log M ρ λ 0 t ) C\cdot\frac{M^7}{\rho t^2\lambda_0^{10}}\cdot\exp(-nt^2/(288B^2))=C\cdot\exp\left(-\frac{nt^2\lambda_0^6}{M^4}\log\frac{M}{\rho\lambda_0t}\right) C ⋅ ρ t 2 λ 0 10 M 7 ⋅ exp ( − n t 2 / ( 288 B 2 )) = C ⋅ exp ( − M 4 n t 2 λ 0 6 log ρ λ 0 t M ) .
Lemma 10 (Strong concavity). Under Assumption 8, for any θ ∈ Θ = [ λ 0 / 2 , 2 M / ρ ] × [ − M , M ] \theta\in\Theta=[\lambda_0/2,2M/\rho]\times[-M,M] θ ∈ Θ = [ λ 0 /2 , 2 M / ρ ] × [ − M , M ] , we have
− ∇ 2 g θ ( π b ⋆ ) ⪰ e − ( 1 + ρ ) σ 0 2 λ 0 2 ρ 3 16 M 5 Id . -\nabla^2g_\theta(\pi_b^\star)\succeq\frac{e^{-(1+\rho)\sigma_0^2}\lambda_0^2\rho^3}{16M^5}\operatorname{Id}. − ∇ 2 g θ ( π b ⋆ ) ⪰ 16 M 5 e − ( 1 + ρ ) σ 0 2 λ 0 2 ρ 3 Id .
Proof. Let us define μ k ( λ , η ) = E [ ( ∣ τ ( X ) ∣ + η ) k e ( − ∣ τ ( X ) ∣ − η ) / λ ] \mu_k(\lambda,\eta)=\mathbb{E}\left[(|\tau(X)|+\eta)^k e^{(-|\tau(X)|-\eta)/\lambda}\right] μ k ( λ , η ) = E [ ( ∣ τ ( X ) ∣ + η ) k e ( − ∣ τ ( X ) ∣ − η ) / λ ] for k = 0 , 1 , 2 k=0,1,2 k = 0 , 1 , 2 . The Hessian of G G G at a general point ( λ , η ) (\lambda,\eta) ( λ , η ) is
∇ 2 g λ , η ( π b ⋆ ) = ( − e − 1 μ 2 / λ 3 e − 1 μ 1 / λ 2 e − 1 μ 1 / λ 2 − e − 1 μ 0 / λ ) , \nabla^2g_{\lambda,\eta}(\pi_b^\star)=\begin{pmatrix}-e^{-1}\mu_2/\lambda^3 & e^{-1}\mu_1/\lambda^2 \\ e^{-1}\mu_1/\lambda^2 & -e^{-1}\mu_0/\lambda\end{pmatrix}, ∇ 2 g λ , η ( π b ⋆ ) = ( − e − 1 μ 2 / λ 3 e − 1 μ 1 / λ 2 e − 1 μ 1 / λ 2 − e − 1 μ 0 / λ ) ,
so
− ∇ 2 g = e − 1 λ ( μ 2 / λ 2 − μ 1 / λ − μ 1 / λ μ 0 ) . -\nabla^2g=\frac{e^{-1}}{\lambda}\begin{pmatrix}\mu_2/\lambda^2 & -\mu_1/\lambda\\ -\mu_1/\lambda& \mu_0\end{pmatrix}. − ∇ 2 g = λ e − 1 ( μ 2 / λ 2 − μ 1 / λ − μ 1 / λ μ 0 ) .
The determinant of − ∇ 2 g -\nabla^{2}g − ∇ 2 g is
det ( − ∇ 2 g ) = e − 2 λ 4 ( μ 2 μ 0 − μ 1 2 ) = e − 2 μ 0 2 λ 4 ( μ 2 μ 0 − ( μ 1 μ 0 ) 2 ) . \det(-\nabla^{2}g)=\frac{e^{-2}}{\lambda^{4}}\left(\mu_{2}\mu_{0}-\mu_{1}^{2}\right)=\frac{e^{-2}\mu_{0}^{2}}{\lambda^{4}}\left(\frac{\mu_{2}}{\mu_{0}}-\left(\frac{\mu_{1}}{\mu_{0}}\right)^{2}\right). det ( − ∇ 2 g ) = λ 4 e − 2 ( μ 2 μ 0 − μ 1 2 ) = λ 4 e − 2 μ 0 2 ( μ 0 μ 2 − ( μ 0 μ 1 ) 2 ) .
Now let us define the tilted distribution Q ~ \widetilde{Q} Q such that d Q ~ / d P 0 ∝ e − ( ∣ τ ( X ) ∣ − η ) / λ − 1 \mathrm{d}\widetilde{Q}/\mathrm{d}P_{0}\propto e^{-(|\tau(X)|-\eta)/\lambda-1} d Q / d P 0 ∝ e − ( ∣ τ ( X ) ∣ − η ) / λ − 1 . Then μ 2 / μ 0 = E Q ~ [ ( ∣ τ ( X ) ∣ + η ) 2 ] \mu_{2}/\mu_{0}=\mathbb{E}_{\widetilde{Q}}[(|\tau(X)|+\eta)^{2}] μ 2 / μ 0 = E Q [( ∣ τ ( X ) ∣ + η ) 2 ] and μ 1 / μ 0 = E Q ~ [ ( ∣ τ ( X ) ∣ + η ) ] \mu_{1}/\mu_{0}=\mathbb{E}_{\widetilde{Q}}[(|\tau(X)|+\eta)] μ 1 / μ 0 = E Q [( ∣ τ ( X ) ∣ + η )] , and we have.
det ( − ∇ 2 g ) = e − 2 μ 0 2 λ 4 ⋅ Var Q ~ ( ∣ τ ( X ) ∣ + η ) ≥ e − 2 μ 0 2 λ 4 ⋅ Var Q ~ ( ∣ τ ( X ) ∣ ) . \det(-\nabla^{2}g)=\frac{e^{-2}\mu_{0}^{2}}{\lambda^{4}}\cdot\operatorname{Var}_{\widetilde{Q}}(|\tau(X)|+\eta)\geq\frac{e^{-2}\mu_{0}^{2}}{\lambda^{4}}\cdot\operatorname{Var}_{\widetilde{Q}}(|\tau(X)|). det ( − ∇ 2 g ) = λ 4 e − 2 μ 0 2 ⋅ Var Q ( ∣ τ ( X ) ∣ + η ) ≥ λ 4 e − 2 μ 0 2 ⋅ Var Q ( ∣ τ ( X ) ∣ ) .
Since λ , η ∈ Θ = [ λ 0 , 2 M / ρ ] × [ − M , M ] \lambda,\eta\in\Theta=[\lambda_{0},2M/\rho]\times[-M,M] λ , η ∈ Θ = [ λ 0 , 2 M / ρ ] × [ − M , M ] and assumption 8 holds, by following a proof similar to Lemma 7 we can show that Var Q ~ ( ∣ τ ( X ) ∣ ) ≥ e − ( 1 + ρ ) ⋅ σ 0 2 \operatorname{Var}_{\widetilde{Q}}(|\tau(X)|)\geq e^{-(1+\rho)}\cdot\sigma_{0}^{2} Var Q ( ∣ τ ( X ) ∣ ) ≥ e − ( 1 + ρ ) ⋅ σ 0 2 . This gives us the following bound on the minimum eigenvalue of − ∇ 2 g -\nabla^{2}g − ∇ 2 g .
λ min ( − ∇ 2 g ) ≥ det ( − ∇ 2 g ) tr ( − ∇ 2 g ) ≥ e − 1 μ 0 2 λ 3 ⋅ e − ( 1 + ρ ) ⋅ σ 0 2 μ 0 + μ 2 λ 2 . \lambda_{\min}(-\nabla^{2}g)\geq\frac{\det(-\nabla^{2}g)}{\operatorname{tr}(-\nabla^{2}g)}\geq\frac{e^{-1}\mu_{0}^{2}}{\lambda^{3}}\cdot\frac{e^{-(1+\rho)}\cdot\sigma_{0}^{2}}{\mu_{0}+\frac{\mu_{2}}{\lambda^{2}}}. λ m i n ( − ∇ 2 g ) ≥ tr ( − ∇ 2 g ) det ( − ∇ 2 g ) ≥ λ 3 e − 1 μ 0 2 ⋅ μ 0 + λ 2 μ 2 e − ( 1 + ρ ) ⋅ σ 0 2 .
We now use μ 0 = e \mu_{0}=e μ 0 = e , μ 2 ≤ 4 M 2 \mu_{2}\leq4M^{2} μ 2 ≤ 4 M 2 , and M / ρ ≥ λ ≥ λ 0 / 2 M/\rho\geq\lambda\geq\lambda_{0}/2 M / ρ ≥ λ ≥ λ 0 /2 to obtain
λ min ( − ∇ 2 g ) ≥ e − ( 1 + ρ ) σ 0 2 λ 0 2 ρ 3 16 M 5 . \lambda_{\min}(-\nabla^{2}g)\geq\frac{e^{-(1+\rho)}\sigma_{0}^{2}\lambda_{0}^{2}\rho^{3}}{16M^{5}}. λ m i n ( − ∇ 2 g ) ≥ 16 M 5 e − ( 1 + ρ ) σ 0 2 λ 0 2 ρ 3 .
Lemma 11 (Finite-sample convergence, KL). Suppose assumptions 7 and 8 hold and n ≥ C M 14 λ 0 10 σ 0 6 ρ 6 n\geq\frac{CM^{14}}{\lambda_{0}^{10}\sigma_{0}^{6}\rho^{6}} n ≥ λ 0 10 σ 0 6 ρ 6 C M 14 . Then for any δ ∈ ( 0 , 1 ) \delta\in(0,1) δ ∈ ( 0 , 1 ) ,
∥ θ ^ n − θ ⋆ ∥ 2 ≤ C M 6 σ 0 2 λ 0 3 ρ 3 e 4 M / λ 0 + ρ log ( 4 / δ ) n \left\|\widehat{\theta}_{n}-\theta^{\star}\right\|_{2}\leq\frac{CM^{6}}{\sigma_{0}^{2}\lambda_{0}^{3}\rho^{3}}e^{4M/\lambda_{0}+\rho}\sqrt{\frac{\log(4/\delta)}{n}} θ n − θ ⋆ 2 ≤ σ 0 2 λ 0 3 ρ 3 C M 6 e 4 M / λ 0 + ρ n log ( 4/ δ ) with probability at least 1 − δ 1-\delta 1 − δ .
Proof. There are two cases to consider. First, suppose θ ^ n \widehat{\theta}_{n} θ n belongs to the interior of Θ \Theta Θ . Then by the first order optimality conditions and Taylor’s expansion we obtain,
0 = ∇ g ^ θ ^ n ( π ⋆ ) = ∇ g ^ θ ⋆ ( π ⋆ ) + ∇ 2 g ^ θ ‾ ( π ⋆ ) ( θ ^ n − θ ⋆ ) . (20) 0=\nabla\widehat{g}_{\widehat{\theta}_{n}}(\pi^{\star})=\nabla\widehat{g}_{\theta^{\star}}(\pi^{\star})+\nabla^{2}\widehat{g}_{\overline{\theta}}(\pi^{\star})(\widehat{\theta}_{n}-\theta^{\star}). \tag*{(20)} 0 = ∇ g θ n ( π ⋆ ) = ∇ g θ ⋆ ( π ⋆ ) + ∇ 2 g θ ( π ⋆ ) ( θ n − θ ⋆ ) . (20)
Here θ ‾ \overline{\theta} θ is a point on the line segment between θ ⋆ \theta^{\star} θ ⋆ and θ ^ n \widehat{\theta}_{n} θ n and belongs to Θ \Theta Θ . Lemma 10 shows that
∥ − ∇ 2 g ( θ ‾ ) ∥ o p ≥ e − ( 1 + ρ ) σ 0 2 λ 0 2 ρ 3 16 M 5 = : t . \left\|-\nabla^{2}g(\overline{\theta})\right\|_{\mathrm{op}}\geq\frac{e^{-(1+\rho)}\sigma_{0}^{2}\lambda_{0}^{2}\rho^{3}}{16M^{5}}=:t. − ∇ 2 g ( θ ) op ≥ 16 M 5 e − ( 1 + ρ ) σ 0 2 λ 0 2 ρ 3 =: t .
By Lemma 9, we have ∥ ∇ 2 g ( θ ‾ ) − ∇ 2 g ^ θ ⋆ ( π ⋆ ) ∥ o p ≤ t / 2 \left\|\nabla^{2}g(\overline{\theta})-\nabla^{2}\widehat{g}_{\theta^{\star}}(\pi^{\star})\right\|_{\mathrm{op}}\leq t/2 ∇ 2 g ( θ ) − ∇ 2 g θ ⋆ ( π ⋆ ) op ≤ t /2 , with probability at least 1 − C ⋅ exp ( − n t 2 λ 0 6 M 4 log M ρ λ 0 t ) 1-C\cdot\exp\left(-\frac{nt^{2}\lambda_{0}^{6}}{M^{4}}\log\frac{M}{\rho\lambda_{0}t}\right) 1 − C ⋅ exp ( − M 4 n t 2 λ 0 6 log ρ λ 0 t M ) . Therefore, as long as n ≥ C M 4 λ 0 6 t 2 n\geq\frac{CM^{4}}{\lambda_{0}^{6}t^{2}} n ≥ λ 0 6 t 2 C M 4 we are guaranteed that ∥ − ∇ 2 g ^ θ ⋆ ( π ⋆ ) ∥ o p ≥ t / 2 \left\|-\nabla^{2}\widehat{g}_{\theta^{\star}}(\pi^{\star})\right\|_{\mathrm{op}}\geq t/2 − ∇ 2 g θ ⋆ ( π ⋆ ) op ≥ t /2 and the matrix − ∇ 2 g ^ θ ⋆ ( π ⋆ ) -\nabla^{2}\widehat{g}_{\theta^{\star}}(\pi^{\star}) − ∇ 2 g θ ⋆ ( π ⋆ ) is invertible. Therefore, eq. (20) gives us
∥ θ ^ n − θ ⋆ ∥ 2 = ∥ ( − ∇ 2 g ^ θ ⋆ ( π ⋆ ) ) − 1 ∇ g ^ θ ⋆ ( π ⋆ ) ∥ ≤ 2 t ∥ ∇ g ^ θ ⋆ ( π ⋆ ) ∥ ≤ 32 M λ 0 t e 4 M / λ 0 log ( 4 / δ ) n (21) \left\|\widehat{\theta}_{n}-\theta^{\star}\right\|_{2}=\left\|(-\nabla^{2}\widehat{g}_{\theta^{\star}}(\pi^{\star}))^{-1}\nabla\widehat{g}_{\theta^{\star}}(\pi^{\star})\right\|\leq\frac{2}{t}\left\|\nabla\widehat{g}_{\theta^{\star}}(\pi^{\star})\right\|\leq\frac{32M}{\lambda_{0}t}e^{4M/\lambda_{0}}\sqrt{\frac{\log(4/\delta)}{n}} \tag*{(21)} θ n − θ ⋆ 2 = ( − ∇ 2 g θ ⋆ ( π ⋆ ) ) − 1 ∇ g θ ⋆ ( π ⋆ ) ≤ t 2 ∥ ∇ g θ ⋆ ( π ⋆ ) ∥ ≤ λ 0 t 32 M e 4 M / λ 0 n log ( 4/ δ ) (21)
with probability at least 1 − δ / 2 1-\delta/2 1 − δ /2 by Lemma 8. Substituting the value of t t t gives us
∥ θ ^ n − θ ⋆ ∥ 2 ≤ C M 6 σ 0 2 λ 0 3 ρ 3 e 4 M / λ 0 + ρ log ( 4 / δ ) n \left\|\widehat{\theta}_{n}-\theta^{\star}\right\|_{2}\leq\frac{CM^{6}}{\sigma_{0}^{2}\lambda_{0}^{3}\rho^{3}}e^{4M/\lambda_{0}+\rho}\sqrt{\frac{\log(4/\delta)}{n}} θ n − θ ⋆ 2 ≤ σ 0 2 λ 0 3 ρ 3 C M 6 e 4 M / λ 0 + ρ n log ( 4/ δ )
for a universal constant C C C , as long as n ≥ C M 14 λ 0 10 σ 0 6 ρ 6 n\geq\frac{CM^{14}}{\lambda_{0}^{10}\sigma_{0}^{6}\rho^{6}} n ≥ λ 0 10 σ 0 6 ρ 6 C M 14 .
We now consider the case when θ ^ n \widehat{\theta}_{n} θ n is on the boundary of Θ \Theta Θ . Since Θ \Theta Θ is convex, the first-order optimality condition for constrained optimization implies ⟨ ∇ g ^ θ ^ n ( π ⋆ ) , θ ^ n − θ ⋆ ⟩ ≥ 0 \left\langle\nabla\widehat{g}_{\widehat{\theta}_{n}}(\pi^{\star}),\widehat{\theta}_{n}-\theta^{\star}\right\rangle\geq0 ⟨ ∇ g θ n ( π ⋆ ) , θ n − θ ⋆ ⟩ ≥ 0 for any θ ∈ Θ \theta\in\Theta θ ∈ Θ . Using eq. (20) we obtain
0 ≤ ⟨ ∇ g ^ θ ^ n ( π ⋆ ) , θ ^ n − θ ⋆ ⟩ = ⟨ ∇ g ^ θ ⋆ ( π ⋆ ) + ∇ 2 g ^ θ ‾ ( π ⋆ ) ( θ ^ n − θ ⋆ ) , θ ^ n − θ ⋆ ⟩ . 0\leq\left\langle\nabla\widehat{g}_{\widehat{\theta}_{n}}(\pi^{\star}),\widehat{\theta}_{n}-\theta^{\star}\right\rangle=\left\langle\nabla\widehat{g}_{\theta^{\star}}(\pi^{\star})+\nabla^{2}\widehat{g}_{\overline{\theta}}(\pi^{\star})(\widehat{\theta}_{n}-\theta^{\star}),\widehat{\theta}_{n}-\theta^{\star}\right\rangle. 0 ≤ ⟨ ∇ g θ n ( π ⋆ ) , θ n − θ ⋆ ⟩ = ⟨ ∇ g θ ⋆ ( π ⋆ ) + ∇ 2 g θ ( π ⋆ ) ( θ n − θ ⋆ ) , θ n − θ ⋆ ⟩ .
After rearranging the terms we obtain,
( θ ^ n − θ ⋆ ) ⊤ ( − g ^ θ ⋆ ( π b ⋆ ) ) ( θ ^ n − θ ⋆ ) ≤ ⟨ ∇ g θ ⋆ ( π b ⋆ ) , θ ^ n − θ ⋆ ⟩ (\hat{\theta}_n-\theta^\star)^\top(-\hat{g}_{\theta^\star}(\pi_b^\star))(\hat{\theta}_n-\theta^\star)\leq\langle\nabla g_{\theta^\star}(\pi_b^\star),\hat{\theta}_n-\theta^\star\rangle ( θ ^ n − θ ⋆ ) ⊤ ( − g ^ θ ⋆ ( π b ⋆ )) ( θ ^ n − θ ⋆ ) ≤ ⟨ ∇ g θ ⋆ ( π b ⋆ ) , θ ^ n − θ ⋆ ⟩
Since ∥ − ∇ 2 g ^ θ ⋆ ( π b ⋆ ) ∥ o p ≥ t / 2 \lVert-\nabla^2\hat{g}_{\theta^\star}(\pi_b^\star)\rVert_{\mathrm{op}}\geq t/2 ∥ − ∇ 2 g ^ θ ⋆ ( π b ⋆ ) ∥ op ≥ t /2 with high probability, we have
t 2 ⋅ ∥ θ ^ n − θ ⋆ ∥ 2 ≤ ⟨ ∇ g ^ θ ⋆ ( π b ⋆ ) , θ ^ n − θ ⋆ ⟩ ≤ ∥ ∇ g ^ θ ⋆ ( π b ⋆ ) ∥ ⋅ ∥ θ ^ n − θ ⋆ ∥ . \frac{t}{2}\cdot\lVert\hat{\theta}_n-\theta^\star\rVert^2\leq\langle\nabla\hat{g}_{\theta^\star}(\pi_b^\star),\hat{\theta}_n-\theta^\star\rangle\leq\lVert\nabla\hat{g}_{\theta^\star}(\pi_b^\star)\rVert\cdot\lVert\hat{\theta}_n-\theta^\star\rVert. 2 t ⋅ ∥ θ ^ n − θ ⋆ ∥ 2 ≤ ⟨ ∇ g ^ θ ⋆ ( π b ⋆ ) , θ ^ n − θ ⋆ ⟩ ≤ ∥ ∇ g ^ θ ⋆ ( π b ⋆ )∥ ⋅ ∥ θ ^ n − θ ⋆ ∥ .
After rearranging, we get the following bound.
∥ θ ^ n − θ ⋆ ∥ ≤ 2 t ⋅ ∥ ∇ g ^ θ ⋆ ( π b ⋆ ) ∥ \lVert\hat{\theta}_n-\theta^\star\rVert\leq\frac{2}{t}\cdot\lVert\nabla\hat{g}_{\theta^\star}(\pi_b^\star)\rVert ∥ θ ^ n − θ ⋆ ∥ ≤ t 2 ⋅ ∥ ∇ g ^ θ ⋆ ( π b ⋆ )∥
The rest of the proof follows the same argument as in case 1.
Bounding the Bias at π b ⋆ \pi_b^\star π b ⋆
Lemma 12. Suppose assumptions 7 and 8 hold and n ≥ C M 14 λ 0 4 σ 0 6 n\geq\frac{CM^{14}}{\lambda_0^4\sigma_0^{6}} n ≥ λ 0 4 σ 0 6 C M 14 . Consider algorithm 3 with KL-divergence. Then, for the Bayes optimal policy π b ⋆ \pi_b^\star π b ⋆ , we have
∣ B ( π b ⋆ ) ∣ = ∣ E [ A ^ n , K L ( π b ⋆ ) ] − A ~ K L ( π b ⋆ ) ∣ ≤ C M 16 σ 0 4 λ 0 8 ρ 6 exp ( 8 M / λ 0 ) ⋅ log n n \left|B(\pi_b^\star)\right|=\left|\mathbb{E}\left[\hat{A}_{n,\mathrm{KL}}(\pi_b^\star)\right]-\tilde{A}_{\mathrm{KL}}(\pi_b^\star)\right|\leq\frac{CM^{16}}{\sigma_0^4\lambda_0^8\rho^6}\exp(8M/\lambda_0)\cdot\frac{\log n}{n} ∣ B ( π b ⋆ ) ∣ = E [ A ^ n , KL ( π b ⋆ ) ] − A ~ KL ( π b ⋆ ) ≤ σ 0 4 λ 0 8 ρ 6 C M 16 exp ( 8 M / λ 0 ) ⋅ n log n for a universal constant C > 0 C>0 C > 0 .
Proof. Since the policy π b ⋆ \pi_b^\star π b ⋆ is fixed throughout the proof, we will write Z ( X ) = ( 2 π b ⋆ ( X ) − 1 ) τ ( X ) = ∣ τ ( X ) ∣ Z(X)=(2\pi_b^\star(X)-1)\tau(X)=|\tau(X)| Z ( X ) = ( 2 π b ⋆ ( X ) − 1 ) τ ( X ) = ∣ τ ( X ) ∣ for brevity. For KL divergence, the dual problem is given as follows.
max λ ≥ 0 λ e ⋅ E [ exp ( − ∣ τ ( X ) ∣ − η λ ) ] − λ ρ − η = : g λ , η ( π b ⋆ ) \max_{\lambda\geq0}\frac{\lambda}{e}\cdot\mathbb{E}\left[\exp\left(\frac{-|\tau(X)|-\eta}{\lambda}\right)\right]-\lambda\rho-\eta=:g_{\lambda,\eta}(\pi_b^\star) λ ≥ 0 max e λ ⋅ E [ exp ( λ − ∣ τ ( X ) ∣ − η ) ] − λ ρ − η =: g λ , η ( π b ⋆ )
We will also write g ^ λ , η ( π b ⋆ ) \hat{g}_{\lambda,\eta}(\pi_b^\star) g ^ λ , η ( π b ⋆ ) to denote the empirical version of the dual objective function. Let us first compute the derivatives.
∂ g λ , η ( π b ⋆ ) ∂ λ = − 1 e ⋅ ( 1 + η λ ) E [ exp ( − ∣ τ ( X ) ∣ − η λ ) ] − 1 e λ ⋅ E [ ∣ τ ( X ) ∣ ⋅ exp ( − ∣ τ ( X ) ∣ − η λ ) ] − ρ \frac{\partial g_{\lambda,\eta}(\pi_b^\star)}{\partial\lambda}=-\frac{1}{e}\cdot\left(1+\frac{\eta}{\lambda}\right)\mathbb{E}\left[\exp\left(\frac{-|\tau(X)|-\eta}{\lambda}\right)\right]-\frac{1}{e\lambda}\cdot\mathbb{E}\left[|\tau(X)|\cdot\exp\left(\frac{-|\tau(X)|-\eta}{\lambda}\right)\right]-\rho ∂ λ ∂ g λ , η ( π b ⋆ ) = − e 1 ⋅ ( 1 + λ η ) E [ exp ( λ − ∣ τ ( X ) ∣ − η ) ] − e λ 1 ⋅ E [ ∣ τ ( X ) ∣ ⋅ exp ( λ − ∣ τ ( X ) ∣ − η ) ] − ρ
∂ g λ , η ( π b ⋆ ) ∂ η = 1 e ⋅ E [ exp ( − ∣ τ ( X ) ∣ − η λ ) ] − 1 \frac{\partial g_{\lambda,\eta}(\pi_b^\star)}{\partial\eta}=\frac{1}{e}\cdot\mathbb{E}\left[\exp\left(\frac{-|\tau(X)|-\eta}{\lambda}\right)\right]-1 ∂ η ∂ g λ , η ( π b ⋆ ) = e 1 ⋅ E [ exp ( λ − ∣ τ ( X ) ∣ − η ) ] − 1
At an optimal solution ( λ ⋆ , η ⋆ ) (\lambda^\star,\eta^\star) ( λ ⋆ , η ⋆ ) we have ∂ g λ ⋆ , η ⋆ ( π b ⋆ ) ∂ λ = 0 \frac{\partial g_{\lambda^\star,\eta^\star}(\pi_b^\star)}{\partial\lambda}=0 ∂ λ ∂ g λ ⋆ , η ⋆ ( π b ⋆ ) = 0 and ∂ g λ ⋆ , η ⋆ ( π b ⋆ ) ∂ η = 0 \frac{\partial g_{\lambda^\star,\eta^\star}(\pi_b^\star)}{\partial\eta}=0 ∂ η ∂ g λ ⋆ , η ⋆ ( π b ⋆ ) = 0 . The second condition implies that E [ exp ( − ∣ τ ( X ) ∣ − η ⋆ λ ⋆ ) ] = e \mathbb{E}\left[\exp\left(\frac{-|\tau(X)|-\eta^\star}{\lambda^\star}\right)\right]=e E [ exp ( λ ⋆ − ∣ τ ( X ) ∣ − η ⋆ ) ] = e . Substituting this value in the first condition we obtain
E [ ∣ τ ( X ) ∣ ⋅ exp ( − ∣ τ ( X ) ∣ − η ⋆ λ ⋆ ) ] = − e λ ⋆ ( 1 + ρ ) − e η ⋆ . \mathbb{E}\left[|\tau(X)|\cdot\exp\left(\frac{-|\tau(X)|-\eta^\star}{\lambda^\star}\right)\right]=-e\lambda^\star(1+\rho)-e\eta^\star. E [ ∣ τ ( X ) ∣ ⋅ exp ( λ ⋆ − ∣ τ ( X ) ∣ − η ⋆ ) ] = − e λ ⋆ ( 1 + ρ ) − e η ⋆ .
Let us now define the distribution Q ⋆ Q^\star Q ⋆ as
d Q ⋆ ( X ) d P 0 ( X ) = exp ( − ∣ τ ( X ) ∣ − η ⋆ λ ⋆ − 1 ) . \frac{dQ^\star(X)}{dP_0(X)}=\exp\left(\frac{-|\tau(X)|-\eta^\star}{\lambda^\star}-1\right). d P 0 ( X ) d Q ⋆ ( X ) = exp ( λ ⋆ − ∣ τ ( X ) ∣ − η ⋆ − 1 ) .
Then we have,
E X ∼ Q ⋆ [ ∣ τ ( X ) ∣ ] = − λ ⋆ ( 1 + ρ ) − η ⋆ \mathbb{E}_{X\sim Q^\star}[|\tau(X)|]=-\lambda^\star(1+\rho)-\eta^\star E X ∼ Q ⋆ [ ∣ τ ( X ) ∣ ] = − λ ⋆ ( 1 + ρ ) − η ⋆
Let us now compute the second derivatives of g λ , η ( π b ⋆ ) g_{\lambda,\eta}(\pi_b^\star) g λ , η ( π b ⋆ ) with respect to λ \lambda λ at the optimal solution ( λ ⋆ , η ⋆ ) (\lambda^\star,\eta^\star) ( λ ⋆ , η ⋆ ) .
∂ 2 g λ ⋆ , η ⋆ ( π b ⋆ ) ∂ η 2 = − 1 e λ ⋆ ⋅ E [ exp ( − ∣ τ ( X ) ∣ − η ⋆ λ ⋆ ) ] = − 1 λ ⋆ \frac{\partial^2g_{\lambda^\star,\eta^\star}(\pi_b^\star)}{\partial\eta^2}=-\frac{1}{e\lambda^\star}\cdot\mathbb{E}\left[\exp\left(\frac{-|\tau(X)|-\eta^\star}{\lambda^\star}\right)\right]=-\frac{1}{\lambda^\star} ∂ η 2 ∂ 2 g λ ⋆ , η ⋆ ( π b ⋆ ) = − e λ ⋆ 1 ⋅ E [ exp ( λ ⋆ − ∣ τ ( X ) ∣ − η ⋆ ) ] = − λ ⋆ 1
∂ 2 g λ , η ( π b ⋆ ) ∂ η ∂ λ = 1 e λ ⋆ 2 ⋅ E [ ( ∣ τ ( X ) ∣ + η ⋆ ) exp ( − ∣ τ ( X ) ∣ − η ⋆ λ ⋆ ) ] = η ⋆ λ ⋆ 2 + 1 λ ⋆ 2 E X ∼ Q ⋆ [ ∣ τ ( X ) ∣ ] = η ⋆ λ ⋆ 2 − 1 + ρ λ ⋆ − η ⋆ λ ⋆ 2 = − 1 + ρ λ ⋆ \begin{aligned}
\frac{\partial^2g_{\lambda,\eta}(\pi_b^\star)}{\partial\eta\partial\lambda}=\frac{1}{e\lambda^{\star2}}\cdot\mathbb{E}\left[(|\tau(X)|+\eta^\star)\exp\left(\frac{-|\tau(X)|-\eta^\star}{\lambda^\star}\right)\right] \\
&=\frac{\eta^\star}{\lambda^{\star2}}+\frac{1}{\lambda^{\star2}}\mathbb{E}_{X\sim Q^\star}[|\tau(X)|]=\frac{\eta^\star}{\lambda^{\star2}}-\frac{1+\rho}{\lambda^\star}-\frac{\eta^\star}{\lambda^{\star2}}=-\frac{1+\rho}{\lambda^\star}
\end{aligned} ∂ η ∂ λ ∂ 2 g λ , η ( π b ⋆ ) = e λ ⋆ 2 1 ⋅ E [ ( ∣ τ ( X ) ∣ + η ⋆ ) exp ( λ ⋆ − ∣ τ ( X ) ∣ − η ⋆ ) ] = λ ⋆ 2 η ⋆ + λ ⋆ 2 1 E X ∼ Q ⋆ [ ∣ τ ( X ) ∣ ] = λ ⋆ 2 η ⋆ − λ ⋆ 1 + ρ − λ ⋆ 2 η ⋆ = − λ ⋆ 1 + ρ
∂ 2 g λ ⋆ , η ⋆ ( π b ⋆ ) ∂ λ ⋆ 2 = η ⋆ e λ ⋆ E [ exp ( − ∣ τ ( X ) ∣ − η ⋆ λ ⋆ ) ] − 1 e λ ⋆ 2 ⋅ ( 1 + η ⋆ λ ⋆ ) E [ ( ∣ τ ( X ) ∣ + η ⋆ ) exp ( − ∣ τ ( X ) ∣ − η ⋆ λ ⋆ ) ] + 1 e λ ⋆ 2 E [ ∣ τ ( X ) ∣ exp ( − ∣ τ ( X ) ∣ − η ⋆ λ ⋆ ) ] − 1 e λ ⋆ 3 E [ ∣ τ ( X ) ∣ ( ∣ τ ( X ) ∣ + η ⋆ ) ⋅ exp ( − ∣ τ ( X ) ∣ − η ⋆ λ ⋆ ) ] = η ⋆ λ ⋆ 2 − 1 λ ⋆ 2 ⋅ ( 1 + η ⋆ λ ⋆ ) E X ∼ Q ⋆ [ ∣ τ ( X ) ∣ + η ⋆ ] + 1 λ ⋆ 2 E X ∼ Q ⋆ [ ∣ τ ( X ) ∣ ] − 1 λ ⋆ 3 E X ∼ Q ⋆ [ ∣ τ ( X ) ∣ ( ∣ τ ( X ) ∣ + η ⋆ ) ] = − E X ∼ Q ⋆ [ ∣ τ ( X ) ∣ 2 ] λ ⋆ 3 + η ⋆ 2 λ ⋆ 3 + 2 η ⋆ ( 1 + ρ ) λ ⋆ 2 = − Var Q ⋆ ( ∣ τ ( X ) ∣ ) λ ⋆ 3 − ( 1 + ρ ) 2 λ ⋆ . \begin{aligned}
\frac{\partial^2 g_{\lambda^\star,\eta^\star}(\pi_b^\star)}{\partial\lambda^{\star2}} &= \frac{\eta^\star}{e^{\lambda^\star}} \mathbb{E}\left[\exp\left(\frac{-|\tau(X)|-\eta^\star}{\lambda^\star}\right)\right]-\frac{1}{e^{\lambda^{\star2}}}\cdot\left(1+\frac{\eta^\star}{\lambda^\star}\right)\mathbb{E}\left[(|\tau(X)|+\eta^\star)\exp\left(\frac{-|\tau(X)|-\eta^\star}{\lambda^\star}\right)\right]\\
&\quad+\frac{1}{e^{\lambda^{\star2}}}\mathbb{E}\left[|\tau(X)|\exp\left(\frac{-|\tau(X)|-\eta^\star}{\lambda^\star}\right)\right]-\frac{1}{e^{\lambda^{\star3}}}\mathbb{E}\left[|\tau(X)|(|\tau(X)|+\eta^\star)\cdot\exp\left(\frac{-|\tau(X)|-\eta^\star}{\lambda^\star}\right)\right]\\
&= \frac{\eta^\star}{\lambda^{\star2}}-\frac{1}{\lambda^{\star2}}\cdot\left(1+\frac{\eta^\star}{\lambda^\star}\right)\mathbb{E}_{X\sim Q^\star}[|\tau(X)|+\eta^\star]+\frac{1}{\lambda^{\star2}}\mathbb{E}_{X\sim Q^\star}[|\tau(X)|]-\frac{1}{\lambda^{\star3}}\mathbb{E}_{X\sim Q^\star}[|\tau(X)|(|\tau(X)|+\eta^\star)]\\
&= -\frac{\mathbb{E}_{X\sim Q^\star}[|\tau(X)|^2]}{\lambda^{\star3}}+\frac{\eta^{\star2}}{\lambda^{\star3}}+\frac{2\eta^\star(1+\rho)}{\lambda^{\star2}}\\
&= -\frac{\operatorname{Var}_{Q^\star}(|\tau(X)|)}{\lambda^{\star3}}-\frac{(1+\rho)^2}{\lambda^\star}.
\end{aligned} ∂ λ ⋆ 2 ∂ 2 g λ ⋆ , η ⋆ ( π b ⋆ ) = e λ ⋆ η ⋆ E [ exp ( λ ⋆ − ∣ τ ( X ) ∣ − η ⋆ ) ] − e λ ⋆ 2 1 ⋅ ( 1 + λ ⋆ η ⋆ ) E [ ( ∣ τ ( X ) ∣ + η ⋆ ) exp ( λ ⋆ − ∣ τ ( X ) ∣ − η ⋆ ) ] + e λ ⋆ 2 1 E [ ∣ τ ( X ) ∣ exp ( λ ⋆ − ∣ τ ( X ) ∣ − η ⋆ ) ] − e λ ⋆ 3 1 E [ ∣ τ ( X ) ∣ ( ∣ τ ( X ) ∣ + η ⋆ ) ⋅ exp ( λ ⋆ − ∣ τ ( X ) ∣ − η ⋆ ) ] = λ ⋆ 2 η ⋆ − λ ⋆ 2 1 ⋅ ( 1 + λ ⋆ η ⋆ ) E X ∼ Q ⋆ [ ∣ τ ( X ) ∣ + η ⋆ ] + λ ⋆ 2 1 E X ∼ Q ⋆ [ ∣ τ ( X ) ∣ ] − λ ⋆ 3 1 E X ∼ Q ⋆ [ ∣ τ ( X ) ∣ ( ∣ τ ( X ) ∣ + η ⋆ )] = − λ ⋆ 3 E X ∼ Q ⋆ [ ∣ τ ( X ) ∣ 2 ] + λ ⋆ 3 η ⋆ 2 + λ ⋆ 2 2 η ⋆ ( 1 + ρ ) = − λ ⋆ 3 Var Q ⋆ ( ∣ τ ( X ) ∣ ) − λ ⋆ ( 1 + ρ ) 2 .
Therefore, the Hessian matrix of g λ , η ( π b ⋆ ) g_{\lambda,\eta}(\pi_b^\star) g λ , η ( π b ⋆ ) at the optimal solution ( λ ⋆ , η ⋆ ) (\lambda^\star,\eta^\star) ( λ ⋆ , η ⋆ ) is given as follows.
∇ 2 g λ ⋆ , η ⋆ ( π b ⋆ ) = [ − Var Q ⋆ ( ∣ τ ( X ) ∣ ) λ ⋆ 3 − ( 1 + ρ ) 2 λ ⋆ − 1 + ρ λ ⋆ − 1 + ρ λ ⋆ − 1 λ ⋆ ] \nabla^2 g_{\lambda^\star,\eta^\star}(\pi_b^\star)=
\begin{bmatrix}
-\frac{\operatorname{Var}_{Q^\star}(|\tau(X)|)}{\lambda^{\star3}}-\frac{(1+\rho)^2}{\lambda^\star} & -\frac{1+\rho}{\lambda^\star}\\
-\frac{1+\rho}{\lambda^\star} & -\frac{1}{\lambda^\star}
\end{bmatrix} ∇ 2 g λ ⋆ , η ⋆ ( π b ⋆ ) = [ − λ ⋆ 3 Var Q ⋆ ( ∣ τ ( X ) ∣ ) − λ ⋆ ( 1 + ρ ) 2 − λ ⋆ 1 + ρ − λ ⋆ 1 + ρ − λ ⋆ 1 ]
Now let us write θ ^ = ( λ ^ ( π b ⋆ ) , η ^ ( π b ⋆ ) ) \hat{\theta}=(\hat{\lambda}(\pi_b^\star),\hat{\eta}(\pi_b^\star)) θ ^ = ( λ ^ ( π b ⋆ ) , η ^ ( π b ⋆ )) to denote the optimal solution to the empirical dual objective g ^ λ , η ( π b ⋆ ) \hat{g}_{\lambda,\eta}(\pi_b^\star) g ^ λ , η ( π b ⋆ ) . Additionally, let θ ⋆ = ( λ ⋆ , η ⋆ ) \theta^\star=(\lambda^\star,\eta^\star) θ ⋆ = ( λ ⋆ , η ⋆ ) , and Δ θ = θ ^ − θ ⋆ \Delta\theta=\hat{\theta}-\theta^\star Δ θ = θ ^ − θ ⋆ . By Taylor’s expansion we have,
g θ ^ = g θ ⋆ + ∇ g θ ⋆ T Δ θ + 1 2 ( θ ^ − θ ⋆ ) T ⋅ ∇ 2 g ( θ ˉ ) ⋅ ( θ ^ − θ ⋆ ) g_{\hat{\theta}}=g_{\theta^\star}+\nabla g_{\theta^\star}^{\mathsf{T}}\Delta\theta+\frac{1}{2}(\hat{\theta}-\theta^\star)^{\mathsf{T}}\cdot\nabla^2g(\bar{\theta})\cdot(\hat{\theta}-\theta^\star) g θ ^ = g θ ⋆ + ∇ g θ ⋆ T Δ θ + 2 1 ( θ ^ − θ ⋆ ) T ⋅ ∇ 2 g ( θ ˉ ) ⋅ ( θ ^ − θ ⋆ )
where θ ˉ \bar{\theta} θ ˉ is a point on the line segment connecting θ ^ \hat{\theta} θ ^ and θ ⋆ \theta^\star θ ⋆ . Since ∇ g θ ⋆ = 0 \nabla g_{\theta^\star}=0 ∇ g θ ⋆ = 0 , we have
g θ ⋆ − g θ ^ = − 1 2 ⋅ Δ θ T ⋅ ∇ 2 g ( θ ˉ ) ⋅ Δ θ . g_{\theta^\star}-g_{\hat{\theta}}=-\frac{1}{2}\cdot\Delta\theta^{\mathsf{T}}\cdot\nabla^2g(\bar{\theta})\cdot\Delta\theta. g θ ⋆ − g θ ^ = − 2 1 ⋅ Δ θ T ⋅ ∇ 2 g ( θ ˉ ) ⋅ Δ θ .
Let us also define H ( θ ˉ ) = − ∇ 2 g ( θ ˉ ) H(\bar{\theta})=-\nabla^2g(\bar{\theta}) H ( θ ˉ ) = − ∇ 2 g ( θ ˉ ) . Then we have,
∣ B K L ( π b ⋆ ) ∣ = ∣ E [ g θ ^ ] − g θ ⋆ ∣ ≤ ∣ E [ g θ ⋆ − g θ ^ ] ∣ = 1 2 ⋅ E [ ∣ Δ θ T ⋅ H ( θ ˉ ) ⋅ Δ θ ∣ ] ≤ 1 2 ⋅ E [ ∥ H ( θ ˉ ) ∥ o p ⋅ ∥ Δ θ ∥ 2 2 ] . (22) \begin{aligned}
|B_{\mathrm{KL}}(\pi_b^\star)| &= |\mathbb{E}[g_{\hat{\theta}}]-g_{\theta^\star}|\leq|\mathbb{E}[g_{\theta^\star}-g_{\hat{\theta}}]|\\
&=\frac{1}{2}\cdot\mathbb{E}\left[|\Delta\theta^{\mathsf{T}}\cdot H(\bar{\theta})\cdot\Delta\theta|\right]\leq\frac{1}{2}\cdot\mathbb{E}\left[\left\|H(\bar{\theta})\right\|_{\mathrm{op}}\cdot\|\Delta\theta\|_2^2\right]. \tag*{(22)}
\end{aligned} ∣ B KL ( π b ⋆ ) ∣ = ∣ E [ g θ ^ ] − g θ ⋆ ∣ ≤ ∣ E [ g θ ⋆ − g θ ^ ] ∣ = 2 1 ⋅ E [ ∣Δ θ T ⋅ H ( θ ˉ ) ⋅ Δ θ ∣ ] ≤ 2 1 ⋅ E [ H ( θ ˉ ) op ⋅ ∥Δ θ ∥ 2 2 ] . (22)
We now upper bound the two terms in the above expression. First, Lemma 11 shows that ∥ Δ θ ∥ 2 ≤ C 1 ⋅ log ( 1 / δ ) n \|\Delta\theta\|_2\leq C_1\cdot\sqrt{\frac{\log(1/\delta)}{n}} ∥Δ θ ∥ 2 ≤ C 1 ⋅ n l o g ( 1/ δ ) with probability at least 1 − δ 1-\delta 1 − δ for a suitable constant C 1 C_1 C 1 as long as n ≥ C M 14 λ 0 10 σ 0 6 n\geq\frac{CM^{14}}{\lambda_0^{10}\sigma_0^{6}} n ≥ λ 0 10 σ 0 6 C M 14 . Furthermore,
H ( θ ˉ ) = − ∇ 2 g ( λ ˉ , η ˉ ) = [ Var Q ˉ ( Z ( X ) ) λ ˉ 3 + ( 1 + ρ ) 2 λ ˉ 1 + ρ λ ˉ 1 + ρ λ ˉ 1 λ ˉ ] H(\bar{\theta})=-\nabla^2g(\bar{\lambda},\bar{\eta})=
\begin{bmatrix}
\frac{\operatorname{Var}_{\bar{Q}}(Z(X))}{\bar{\lambda}^3}+\frac{(1+\rho)^2}{\bar{\lambda}} & \frac{1+\rho}{\bar{\lambda}}\\
\frac{1+\rho}{\bar{\lambda}} & \frac{1}{\bar{\lambda}}
\end{bmatrix} H ( θ ˉ ) = − ∇ 2 g ( λ ˉ , η ˉ ) = [ λ ˉ 3 Var Q ˉ ( Z ( X )) + λ ˉ ( 1 + ρ ) 2 λ ˉ 1 + ρ λ ˉ 1 + ρ λ ˉ 1 ]
Here Q ˉ \bar{Q} Q ˉ is the distribution defined as d Q ˉ ( X ) d P 0 ( X ) = exp ( − Z ( X ) − η ˉ λ ˉ − 1 ) \frac{d\bar{Q}(X)}{dP_0(X)}=\exp\left(-\frac{Z(X)-\bar{\eta}}{\bar{\lambda}}-1\right) d P 0 ( X ) d Q ˉ ( X ) = exp ( − λ ˉ Z ( X ) − η ˉ − 1 ) . Therefore,
∥ H ( θ ˉ ) ∥ o p 2 ≤ ∥ H ( θ ˉ ) ∥ F 2 ≤ 16 λ 0 2 + ( 2 Var Q ˉ ( Z ( X ) ) λ 0 ) 2 ≤ 16 λ 0 2 + ( 2 E Q ˉ ( Z ( X ) 2 ) λ 0 ) 2 ≤ 16 + M 4 λ 0 2 . \|H(\bar{\theta})\|_{\mathrm{op}}^2\leq\|H(\bar{\theta})\|_{\mathrm{F}}^2\leq\frac{16}{\lambda_0^2}+\left(\frac{2\operatorname{Var}_{\bar{Q}}(Z(X))}{\lambda_0}\right)^2\leq\frac{16}{\lambda_0^2}+\left(\frac{2\mathbb{E}_{\bar{Q}}(Z(X)^2)}{\lambda_0}\right)^2\leq\frac{16+M^4}{\lambda_0^2}. ∥ H ( θ ˉ ) ∥ op 2 ≤ ∥ H ( θ ˉ ) ∥ F 2 ≤ λ 0 2 16 + ( λ 0 2 Var Q ˉ ( Z ( X )) ) 2 ≤ λ 0 2 16 + ( λ 0 2 E Q ˉ ( Z ( X ) 2 ) ) 2 ≤ λ 0 2 16 + M 4 .
Therefore, ∥ H ( θ ˉ ) ∥ o p ≤ 4 + M 2 λ 0 \|H(\bar{\theta})\|_{\mathrm{op}}\leq\frac{4+M^2}{\lambda_0} ∥ H ( θ ˉ ) ∥ op ≤ λ 0 4 + M 2 . Substituting the upper bounds on ∥ Δ ∥ 2 \|\Delta\|_2 ∥Δ ∥ 2 and ∥ H ( θ ˉ ) ∥ o p \|H(\bar{\theta})\|_{\mathrm{op}} ∥ H ( θ ˉ ) ∥ op in eq. (22) we obtain the following bound.
∣ B K L ( π b ⋆ ) ∣ ≤ 16 + M 4 2 λ 0 2 ⋅ E [ ∥ Δ θ ∥ 2 2 ] ≤ 16 + M 4 2 λ 0 2 ( 1 n ⋅ ( 4 M 2 ρ ) 2 + C 1 2 ⋅ log n n ) ≤ 16 + M 4 2 λ 0 2 ( 1 n ⋅ ( 4 M 2 ρ ) 2 + C 2 M 12 σ 0 4 λ 0 6 ρ 6 exp ( 8 M / λ 0 ) ⋅ log n n ) \begin{aligned}
|B_{\mathrm{KL}}(\pi_b^\star)|\leq\frac{16+M^4}{2\lambda_0^2}\cdot\mathbb{E}\left[\|\Delta\theta\|_2^2\right] \\
&\leq\frac{16+M^4}{2\lambda_0^2}\left(\frac{1}{n}\cdot\left(\frac{4M^2}{\rho}\right)^2+C_1^2\cdot\frac{\log n}{n}\right) \\
&\leq\frac{16+M^4}{2\lambda_0^2}\left(\frac{1}{n}\cdot\left(\frac{4M^2}{\rho}\right)^2+\frac{C^2M^{12}}{\sigma_0^4\lambda_0^6\rho^6}\exp(8M/\lambda_0)\cdot\frac{\log n}{n}\right)
\end{aligned} ∣ B KL ( π b ⋆ ) ∣ ≤ 2 λ 0 2 16 + M 4 ⋅ E [ ∥Δ θ ∥ 2 2 ] ≤ 2 λ 0 2 16 + M 4 ( n 1 ⋅ ( ρ 4 M 2 ) 2 + C 1 2 ⋅ n log n ) ≤ 2 λ 0 2 16 + M 4 ( n 1 ⋅ ( ρ 4 M 2 ) 2 + σ 0 4 λ 0 6 ρ 6 C 2 M 12 exp ( 8 M / λ 0 ) ⋅ n log n )
for a universal constant C > 0 C>0 C > 0 . □
Lemma 13. Consider algorithm 3 with general f f f -divergence. Then, for any policy π \pi π , we have
B ^ f ( π ) = E [ A ~ n , f e s t ( π ) ] − A ~ f e s t ( π ) ≤ 0. \widehat{B}_f(\pi)=\mathbb{E}\left[\widetilde{A}^{\mathrm{est}}_{n,f}(\pi)\right]-\widetilde{A}^{\mathrm{est}}_f(\pi)\leq0. B f ( π ) = E [ A n , f est ( π ) ] − A f est ( π ) ≤ 0. Proof. Let us write λ ^ ( π ) \widehat{\lambda}(\pi) λ ( π ) and η ^ ( π ) \widehat{\eta}(\pi) η ( π ) to denote the estimated dual variables for policy π \pi π . Then,
E [ A ~ n , f e s t ( π ) ] = E [ − λ ^ ( π ) ⋅ 1 n ∑ i = 1 n f ⋆ ( − ( 2 π ( X i ) − 1 ) Γ ^ i − η ^ ( π ) λ ^ ( π ) ) − λ ^ ( π ) ρ − η ^ ( π ) ] = − λ ^ ( π ) ⋅ E X ∼ P 0 [ f ⋆ ( − ( 2 π ( X ) − 1 ) Γ ^ ( X , W ) − η ^ ( π ) λ ^ ( π ) ) ] − λ ^ ( π ) ρ − η ^ ( π ) ≤ max λ ≥ 0 , η ∈ R − λ ⋅ E X ∼ P 0 [ f ⋆ ( − ( 2 π ( X ) − 1 ) Γ ^ ( X , W ) − η λ ) ] − λ ρ − η = A ~ f e s t ( π ) . \begin{aligned}
\mathbb{E}\left[\widetilde{A}^{\mathrm{est}}_{n,f}(\pi)\right]
&=\mathbb{E}\left[-\widehat{\lambda}(\pi)\cdot\frac{1}{n}\sum_{i=1}^{n}f^{\star}\left(\frac{-(2\pi(X_i)-1)\widehat{\Gamma}_i-\widehat{\eta}(\pi)}{\widehat{\lambda}(\pi)}\right)-\widehat{\lambda}(\pi)\rho-\widehat{\eta}(\pi)\right]\\
&=-\widehat{\lambda}(\pi)\cdot\mathbb{E}_{X\sim P_0}\left[f^{\star}\left(\frac{-(2\pi(X)-1)\widehat{\Gamma}(X,W)-\widehat{\eta}(\pi)}{\widehat{\lambda}(\pi)}\right)\right]-\widehat{\lambda}(\pi)\rho-\widehat{\eta}(\pi)\\
&\leq\max_{\lambda\geq0,\eta\in\mathbb{R}}-\lambda\cdot\mathbb{E}_{X\sim P_0}\left[f^{\star}\left(\frac{-(2\pi(X)-1)\widehat{\Gamma}(X,W)-\eta}{\lambda}\right)\right]-\lambda\rho-\eta=\widetilde{A}^{\mathrm{est}}_f(\pi).
\end{aligned} E [ A n , f est ( π ) ] = E [ − λ ( π ) ⋅ n 1 i = 1 ∑ n f ⋆ ( λ ( π ) − ( 2 π ( X i ) − 1 ) Γ i − η ( π ) ) − λ ( π ) ρ − η ( π ) ] = − λ ( π ) ⋅ E X ∼ P 0 [ f ⋆ ( λ ( π ) − ( 2 π ( X ) − 1 ) Γ ( X , W ) − η ( π ) ) ] − λ ( π ) ρ − η ( π ) ≤ λ ≥ 0 , η ∈ R max − λ ⋅ E X ∼ P 0 [ f ⋆ ( λ − ( 2 π ( X ) − 1 ) Γ ( X , W ) − η ) ] − λ ρ − η = A f est ( π ) .
Dual Variable Stability
Lemma 14 (Variance perturbation). For any policy π \pi π :
∣ Var ( Z π ( X ) ) − Var ( Z π b ∗ ( X ) ) ∣ ≤ 8 M 2 D ( π ) . (23) \left|\operatorname{Var}(Z_\pi(X))-\operatorname{Var}(Z_{\pi_b^{\ast}}(X))\right|\leq8M^2D(\pi). \tag*{(23)} Var ( Z π ( X )) − Var ( Z π b ∗ ( X )) ≤ 8 M 2 D ( π ) . (23) In particular, if D ( π ) ≤ s max : = σ 0 2 / ( 8 M 2 ) D(\pi)\leq s_{\max}:=\sigma_0^2/(8M^2) D ( π ) ≤ s m a x := σ 0 2 / ( 8 M 2 ) , then Var ( Z π ( X ) ) ≥ σ 0 2 / 2 \operatorname{Var}(Z_\pi(X))\geq\sigma_0^2/2 Var ( Z π ( X )) ≥ σ 0 2 /2 .
Proof. Let us write Var ( Z π ( X ) ) = E [ Z π ( X ) 2 ] − ( E [ Z π ( X ) ] ) 2 \operatorname{Var}(Z_{\pi}(X))=\mathbb{E}[Z_{\pi}(X)^2]-(\mathbb{E}[Z_{\pi}(X)])^2 Var ( Z π ( X )) = E [ Z π ( X ) 2 ] − ( E [ Z π ( X )] ) 2 and bound each perturbation separately.
Second moment perturbation. Since Z π b ∗ ( X ) 2 = τ ( X ) 2 Z_{\pi_b^{\ast}}(X)^2=\tau(X)^2 Z π b ∗ ( X ) 2 = τ ( X ) 2 (because π b ∗ \pi_b^{\ast} π b ∗ is binary) and Z π ( X ) 2 = ( 2 π ( X ) − 1 ) 2 τ ( X ) 2 Z_{\pi}(X)^2=(2\pi(X)-1)^2\tau(X)^2 Z π ( X ) 2 = ( 2 π ( X ) − 1 ) 2 τ ( X ) 2 :
E [ Z π ( X ) 2 ] − E [ Z π b ∗ ( X ) 2 ] = E [ ( 1 − ( 2 π ( X ) − 1 ) 2 ) τ ( X ) 2 ] = 4 E [ π ( X ) ( 1 − π ( X ) ) τ ( X ) 2 ] . \mathbb{E}[Z_{\pi}(X)^2]-\mathbb{E}[Z_{\pi_b^{\ast}}(X)^2]=\mathbb{E}[(1-(2\pi(X)-1)^2)\tau(X)^2]=4\mathbb{E}[\pi(X)(1-\pi(X))\tau(X)^2]. E [ Z π ( X ) 2 ] − E [ Z π b ∗ ( X ) 2 ] = E [( 1 − ( 2 π ( X ) − 1 ) 2 ) τ ( X ) 2 ] = 4 E [ π ( X ) ( 1 − π ( X )) τ ( X ) 2 ] .
We claim π ( x ) ( 1 − π ( x ) ) ≤ ∣ π b ∗ ( x ) − π ( x ) ∣ \pi(x)(1-\pi(x))\leq|\pi_b^{\ast}(x)-\pi(x)| π ( x ) ( 1 − π ( x )) ≤ ∣ π b ∗ ( x ) − π ( x ) ∣ pointwise. When π b ∗ ( x ) = 1 \pi_b^{\ast}(x)=1 π b ∗ ( x ) = 1 : π ( x ) ( 1 − π ( x ) ) ≤ 1 − π ( x ) = ∣ π b ∗ ( x ) − π ( x ) ∣ \pi(x)(1-\pi(x))\leq1-\pi(x)=|\pi_b^{\ast}(x)-\pi(x)| π ( x ) ( 1 − π ( x )) ≤ 1 − π ( x ) = ∣ π b ∗ ( x ) − π ( x ) ∣ . When π b ∗ ( x ) = 0 \pi_b^{\ast}(x)=0 π b ∗ ( x ) = 0 : π ( x ) ( 1 − π ( x ) ) ≤ π ( x ) = ∣ π b ∗ ( x ) − π ( x ) ∣ \pi(x)(1-\pi(x))\leq\pi(x)=|\pi_b^{\ast}(x)-\pi(x)| π ( x ) ( 1 − π ( x )) ≤ π ( x ) = ∣ π b ∗ ( x ) − π ( x ) ∣ , since π ( x ) ( 1 − π ( x ) ) ≤ π ( x ) \pi(x)(1-\pi(x))\leq\pi(x) π ( x ) ( 1 − π ( x )) ≤ π ( x ) for π ( x ) ∈ [ 0 , 1 ] \pi(x)\in[0,1] π ( x ) ∈ [ 0 , 1 ] . Therefore:
0 ≤ E [ Z π ( X ) 2 ] − E [ Z π b ∗ ( X ) 2 ] ≤ 4 M 2 D ( π ) . (24) 0\leq\mathbb{E}[Z_{\pi}(X)^2]-\mathbb{E}[Z_{\pi_b^{\ast}}(X)^2]\leq4M^2D(\pi). \tag*{(24)} 0 ≤ E [ Z π ( X ) 2 ] − E [ Z π b ∗ ( X ) 2 ] ≤ 4 M 2 D ( π ) . (24)
Squared mean perturbation. By Lemma 20 of the main paper, E [ Z π ( X ) ] − E [ Z π b ∗ ( X ) ] = 2 ϵ ( π ) \mathbb{E}[Z_{\pi}(X)]-\mathbb{E}[Z_{\pi_b^{\ast}}(X)]=2\epsilon(\pi) E [ Z π ( X )] − E [ Z π b ∗ ( X )] = 2 ϵ ( π ) and E [ Z π b ∗ ( X ) ] + E [ Z π ( X ) ] = 2 E [ ∣ τ ( X ) ∣ ] − 2 ϵ ( π ) \mathbb{E}[Z_{\pi_b^{\ast}}(X)]+\mathbb{E}[Z_{\pi}(X)]=2\mathbb{E}[|\tau(X)|]-2\epsilon(\pi) E [ Z π b ∗ ( X )] + E [ Z π ( X )] = 2 E [ ∣ τ ( X ) ∣ ] − 2 ϵ ( π ) . Therefore:
( E [ Z π b ∗ ( X ) ] ) 2 − ( E [ Z π ( X ) ] ) 2 = ( E [ Z π b ∗ ( X ) ] + E [ Z π ( X ) ] ) ( E [ Z π b ∗ ( X ) ] − E [ Z π ( X ) ] ) = 2 ϵ ( π ) ( 2 E [ ∣ τ ( X ) ∣ ] − 2 ϵ ( π ) ) . \begin{aligned}
(\mathbb{E}[Z_{\pi_b^{\ast}}(X)])^2-(\mathbb{E}[Z_{\pi}(X)])^2
&=(\mathbb{E}[Z_{\pi_b^{\ast}}(X)]+\mathbb{E}[Z_{\pi}(X)])(\mathbb{E}[Z_{\pi_b^{\ast}}(X)]-\mathbb{E}[Z_{\pi}(X)])\\
&=2\epsilon(\pi)(2\mathbb{E}[|\tau(X)|]-2\epsilon(\pi)).
\end{aligned} ( E [ Z π b ∗ ( X )] ) 2 − ( E [ Z π ( X )] ) 2 = ( E [ Z π b ∗ ( X )] + E [ Z π ( X )]) ( E [ Z π b ∗ ( X )] − E [ Z π ( X )]) = 2 ϵ ( π ) ( 2 E [ ∣ τ ( X ) ∣ ] − 2 ϵ ( π )) .
Since ϵ ( π ) ≤ M D ( π ) \epsilon(\pi)\leq MD(\pi) ϵ ( π ) ≤ M D ( π ) (because ∣ τ ( X ) ∣ ≤ M |\tau(X)|\leq M ∣ τ ( X ) ∣ ≤ M ) and 2 E [ ∣ τ ( X ) ∣ ] − 2 ϵ ( π ) ≤ 2 M 2\mathbb{E}[|\tau(X)|]-2\epsilon(\pi)\leq2M 2 E [ ∣ τ ( X ) ∣ ] − 2 ϵ ( π ) ≤ 2 M :
0 ≤ ( E [ Z π b ∗ ( X ) ] ) 2 − ( E [ Z π ( X ) ] ) 2 ≤ 4 M 2 D ( π ) . (25) 0\leq(\mathbb{E}[Z_{\pi_b^{\ast}}(X)])^2-(\mathbb{E}[Z_{\pi}(X)])^2\leq4M^2D(\pi). \tag*{(25)} 0 ≤ ( E [ Z π b ∗ ( X )] ) 2 − ( E [ Z π ( X )] ) 2 ≤ 4 M 2 D ( π ) . (25)
From (24) and (25) :
Var ( Z π b ∗ ( X ) ) − Var ( Z π ( X ) ) = [ E [ Z π ( X ) 2 ] − E [ Z π b ∗ ( X ) 2 ] ] − [ ( E [ Z π b ∗ ( X ) ] ) 2 − ( E [ Z π ( X ) ] ) 2 ] . \operatorname{Var}(Z_{\pi_b^{\ast}}(X))-\operatorname{Var}(Z_{\pi}(X))=[\mathbb{E}[Z_{\pi}(X)^2]-\mathbb{E}[Z_{\pi_b^{\ast}}(X)^2]]-[(\mathbb{E}[Z_{\pi_b^{\ast}}(X)])^2-(\mathbb{E}[Z_{\pi}(X)])^2]. Var ( Z π b ∗ ( X )) − Var ( Z π ( X )) = [ E [ Z π ( X ) 2 ] − E [ Z π b ∗ ( X ) 2 ]] − [( E [ Z π b ∗ ( X )] ) 2 − ( E [ Z π ( X )] ) 2 ] .
Each term lies in [ 0 , 4 M 2 D ( π ) ] [0,4M^2D(\pi)] [ 0 , 4 M 2 D ( π )] , so their difference lies in [ − 4 M 2 D ( π ) , 4 M 2 D ( π ) ] [-4M^2D(\pi),4M^2D(\pi)] [ − 4 M 2 D ( π ) , 4 M 2 D ( π )] . Hence
∣ Var ( Z π ( X ) ) − Var ( Z π b ∗ ( X ) ) ∣ ≤ 4 M 2 D ( π ) . |\operatorname{Var}(Z_{\pi}(X))-\operatorname{Var}(Z_{\pi_b^{\ast}}(X))|\leq4M^2D(\pi). ∣ Var ( Z π ( X )) − Var ( Z π b ∗ ( X )) ∣ ≤ 4 M 2 D ( π ) .
For the final consequence, note that if D ( π ) ≤ s max = σ 0 2 / ( 8 M 2 ) D(\pi)\leq s_{\max}=\sigma_0^2/(8M^2) D ( π ) ≤ s m a x = σ 0 2 / ( 8 M 2 ) , then
Var ( Z π ( X ) ) ≥ Var ( Z π b ∗ ( X ) ) − 4 M 2 s max = Var ( ∣ τ ( X ) ∣ ) − σ 0 2 / 2 = σ 0 2 / 2. \operatorname{Var}(Z_{\pi}(X))\geq\operatorname{Var}(Z_{\pi_b^{\ast}}(X))-4M^2s_{\max}=\operatorname{Var}(|\tau(X)|)-\sigma_0^2/2=\sigma_0^2/2. Var ( Z π ( X )) ≥ Var ( Z π b ∗ ( X )) − 4 M 2 s m a x = Var ( ∣ τ ( X ) ∣ ) − σ 0 2 /2 = σ 0 2 /2.
Corollary 1 (Locally Strongly Concave). For any policy π \pi π with D ( π ) ≤ s max = σ 0 2 / ( 8 M 2 ) D(\pi) \le s_{\max} = \sigma_0^2/(8M^2) D ( π ) ≤ s m a x = σ 0 2 / ( 8 M 2 ) , the dual objective g λ , η ( π ) g_{\lambda,\eta}(\pi) g λ , η ( π ) is ( γ / 2 ) (\gamma/2) ( γ /2 ) -strongly concave on the domain Θ = [ λ 0 / 2 , 2 M / ρ ] × [ − M , M ] \Theta= [\lambda_0/2, 2M/\rho] \times[-M,M] Θ = [ λ 0 /2 , 2 M / ρ ] × [ − M , M ] , where γ \gamma γ is the strong concavity constant from Lemma 10.
Proof. By following the proof of Lemma 10 we can establish that for any λ ∈ [ λ 0 / 2 , 2 M / ρ ] \lambda\in[\lambda_0/2, 2M/\rho] λ ∈ [ λ 0 /2 , 2 M / ρ ] and η ∈ [ − M , M ] \eta\in[-M,M] η ∈ [ − M , M ] ,
λ min ( − ∇ 2 g λ , η ( π ) ) ≥ λ 0 2 ρ 3 16 M 5 Var Q ~ ( Z ( π ) ) . \lambda_{\min}(-\nabla^2 g_{\lambda,\eta}(\pi)) \ge\frac{\lambda_0^2\rho^3}{16M^5}\operatorname{Var}_{\widetilde{Q}}(Z(\pi)). λ m i n ( − ∇ 2 g λ , η ( π )) ≥ 16 M 5 λ 0 2 ρ 3 Var Q ( Z ( π )) .
where Q ~ \widetilde{Q} Q is a distribution defined as d Q ~ / d P 0 ∝ e ( − Z ( π ) − η ) / λ d\widetilde{Q}/dP_0 \propto e^{(-Z(\pi)-\eta)/\lambda} d Q / d P 0 ∝ e ( − Z ( π ) − η ) / λ . Now, by following a proof similar to Lemma 7 (part ii) we can establish that:
Var Q ~ ( Z ( π ) ) ≥ e − ( 1 + ρ ) Var P 0 ( Z ( π ) ) ≥ e − ( 1 + ρ ) ⋅ σ 0 2 2 . \operatorname{Var}_{\widetilde{Q}}(Z(\pi)) \ge e^{-(1+\rho)}\operatorname{Var}_{P_0}(Z(\pi)) \ge e^{-(1+\rho)}\cdot\frac{\sigma_0^2}{2}. Var Q ( Z ( π )) ≥ e − ( 1 + ρ ) Var P 0 ( Z ( π )) ≥ e − ( 1 + ρ ) ⋅ 2 σ 0 2 .
Substituting this lower bound on Var Q ~ ( Z ( π ) ) \operatorname{Var}_{\widetilde{Q}}(Z(\pi)) Var Q ( Z ( π )) in the above expression gives us the desired strong concavity constant γ / 2 \gamma/2 γ /2 .
Corollary 2. Suppose Assumption 2 and Assumption 1 hold. For any policy π \pi π with D ( π ) ≤ s max = σ 0 2 / ( 8 M 2 ) D(\pi) \le s_{\max} = \sigma_0^2/(8M^2) D ( π ) ≤ s m a x = σ 0 2 / ( 8 M 2 ) , the dual objective g ~ λ , η ( π ) \widetilde{g}_{\lambda,\eta}(\pi) g λ , η ( π ) is λ 0 2 ρ 3 κ 2 σ 0 2 64 M 5 e − ( 1 + ρ / κ ) \frac{\lambda_0^2\rho^3\kappa^2\sigma_0^2}{64M^5}e^{-(1+\rho/\kappa)} 64 M 5 λ 0 2 ρ 3 κ 2 σ 0 2 e − ( 1 + ρ / κ ) -strongly concave on the domain Θ = [ λ 0 / 2 , 2 M / ρ ] × [ − M , M ] \Theta= [\lambda_0/2, 2M/\rho] \times[-M,M] Θ = [ λ 0 /2 , 2 M / ρ ] × [ − M , M ] , as long as n ≥ ( 16 M κ σ 0 2 ) 1 / ( β + γ ) n \ge\left(\frac{16M}{\kappa\sigma_0^2}\right)^{1/(\beta+\gamma)} n ≥ ( κ σ 0 2 16 M ) 1/ ( β + γ ) .
Proof. Let us define Z ^ π ( X , W ) = ( 2 π ( X ) − 1 ) Γ ^ ( X , W ) \widehat{Z}_{\pi}(X,W)=(2\pi(X)-1)\widehat{\Gamma}(X,W) Z π ( X , W ) = ( 2 π ( X ) − 1 ) Γ ( X , W ) . Now by following the proof of Lemma 10 we can establish that for any λ ∈ [ λ 0 / 2 , 2 M / ρ ] \lambda\in[\lambda_0/2, 2M/\rho] λ ∈ [ λ 0 /2 , 2 M / ρ ] and η ∈ [ − M , M ] \eta\in[-M,M] η ∈ [ − M , M ] ,
λ min ( − ∇ 2 g ~ λ , η ( π ) ) ≥ λ 0 2 ρ 3 κ 2 16 M 5 Var Q ~ ( Z ^ π ( X , W ) ) . \lambda_{\min}(-\nabla^2\widetilde{g}_{\lambda,\eta}(\pi)) \ge\frac{\lambda_0^2\rho^3\kappa^2}{16M^5}\operatorname{Var}_{\widetilde{Q}}(\widehat{Z}_{\pi}(X,W)). λ m i n ( − ∇ 2 g λ , η ( π )) ≥ 16 M 5 λ 0 2 ρ 3 κ 2 Var Q ( Z π ( X , W )) .
where Q ~ \widetilde{Q} Q is a distribution defined as d Q ~ / d P 0 ( X , W ) ∝ e ( − Z ^ π ( X , W ) − η ) / λ d\widetilde{Q}/dP_0(X,W) \propto e^{(-\widehat{Z}_{\pi}(X,W)-\eta)/\lambda} d Q / d P 0 ( X , W ) ∝ e ( − Z π ( X , W ) − η ) / λ . Now, by following a proof similar to Lemma 7 (part ii) we can establish that:
Var Q ~ ( Z ^ π ( X , W ) ) ≥ e − ( 1 + ρ / κ ) Var P 0 ( Z ^ π ( X , W ) ) . \operatorname{Var}_{\widetilde{Q}}(\widehat{Z}_{\pi}(X,W)) \ge e^{-(1+\rho/\kappa)}\operatorname{Var}_{P_0}(\widehat{Z}_{\pi}(X,W)). Var Q ( Z π ( X , W )) ≥ e − ( 1 + ρ / κ ) Var P 0 ( Z π ( X , W )) .
We now bound the variance of Z ^ π ( X , W ) \widehat{Z}_{\pi}(X,W) Z π ( X , W ) under P 0 P_0 P 0 .
Var P 0 ( Z ^ π ( X , W ) ) = E P 0 [ Z ^ π ( X , W ) 2 ] − ( E P 0 [ Z ^ π ( X , W ) ] ) 2 = Var P 0 ( Z π ( X ) ) + E P 0 [ Z ^ π ( X , W ) 2 − Z π ( X ) 2 ] − ( E P 0 [ Z ^ π ( X , W ) ] ) 2 + ( E P 0 [ Z π ( X ) ] ) 2 \begin{aligned}
\operatorname{Var}_{P_0}(\widehat{Z}_{\pi}(X,W)) = \mathbb{E}_{P_0}[\widehat{Z}_{\pi}(X,W)^2] - (\mathbb{E}_{P_0}[\widehat{Z}_{\pi}(X,W)])^2 \\
&= \operatorname{Var}_{P_0}(Z_{\pi}(X)) + \mathbb{E}_{P_0}[\widehat{Z}_{\pi}(X,W)^2-Z_{\pi}(X)^2] - (\mathbb{E}_{P_0}[\widehat{Z}_{\pi}(X,W)])^2 + (\mathbb{E}_{P_0}[Z_{\pi}(X)])^2
\end{aligned} Var P 0 ( Z π ( X , W )) = E P 0 [ Z π ( X , W ) 2 ] − ( E P 0 [ Z π ( X , W )] ) 2 = Var P 0 ( Z π ( X )) + E P 0 [ Z π ( X , W ) 2 − Z π ( X ) 2 ] − ( E P 0 [ Z π ( X , W )] ) 2 + ( E P 0 [ Z π ( X )] ) 2
Therefore,
∣ Var P 0 ( Z ^ π ( X , W ) ) − Var P 0 ( Z π ( X ) ) ∣ ≤ ∣ E P 0 [ Z ^ π ( X , W ) 2 − Z π ( X ) 2 ] ∣ + ∣ ( E P 0 [ Z ^ π ( X , W ) ] ) 2 − ( E P 0 [ Z π ( X ) ] ) 2 ∣ = ∣ E P 0 [ ( Z ^ π ( X , W ) − Z π ( X ) ) ( Z ^ π ( X , W ) + Z π ( X ) ) ] ∣ + ∣ ( E P 0 [ Z ^ π ( X , W ) ] − E P 0 [ Z π ( X ) ] ) ( E P 0 [ Z ^ π ( X , W ) ] + E P 0 [ Z π ( X ) ] ) ∣ ≤ 4 M κ ⋅ E P 0 [ ∣ E W [ Γ ^ ( X , W ) ] − τ ( X ) ∣ ] . \begin{aligned}
\left|\operatorname{Var}_{P_0}(\widehat{Z}_{\pi}(X,W))-\operatorname{Var}_{P_0}(Z_{\pi}(X))\right|
\le\left|\mathbb{E}_{P_0}[\widehat{Z}_{\pi}(X,W)^2-Z_{\pi}(X)^2]\right|+\left|(\mathbb{E}_{P_0}[\widehat{Z}_{\pi}(X,W)])^2-(\mathbb{E}_{P_0}[Z_{\pi}(X)])^2\right| \\
&= \left|\mathbb{E}_{P_0}[(\widehat{Z}_{\pi}(X,W)-Z_{\pi}(X))(\widehat{Z}_{\pi}(X,W)+Z_{\pi}(X))]\right|
+\left|(\mathbb{E}_{P_0}[\widehat{Z}_{\pi}(X,W)]-\mathbb{E}_{P_0}[Z_{\pi}(X)])(\mathbb{E}_{P_0}[\widehat{Z}_{\pi}(X,W)]+\mathbb{E}_{P_0}[Z_{\pi}(X)])\right| \\
&\le\frac{4M}{\kappa}\cdot\mathbb{E}_{P_0}\left[\left|\mathbb{E}_{W}[\widehat{\Gamma}(X,W)]-\tau(X)\right|\right].
\end{aligned} Var P 0 ( Z π ( X , W )) − Var P 0 ( Z π ( X )) ≤ E P 0 [ Z π ( X , W ) 2 − Z π ( X ) 2 ] + ( E P 0 [ Z π ( X , W )] ) 2 − ( E P 0 [ Z π ( X )] ) 2 = E P 0 [( Z π ( X , W ) − Z π ( X )) ( Z π ( X , W ) + Z π ( X ))] + ( E P 0 [ Z π ( X , W )] − E P 0 [ Z π ( X )]) ( E P 0 [ Z π ( X , W )] + E P 0 [ Z π ( X )]) ≤ κ 4 M ⋅ E P 0 [ E W [ Γ ( X , W )] − τ ( X ) ] .
When Assumption 2 and Assumption 1 hold, the proof of Lemma 4 establishes E P 0 [ ∣ E W [ Γ ^ ( X , W ) ] − τ ( X ) ∣ ] ≤ n − ( β + γ ) \mathbb{E}_{P_0}\left[\left|\mathbb{E}_{W}[\widehat{\Gamma}(X,W)]-\tau(X)\right|\right]\le n^{-(\beta+\gamma)} E P 0 [ E W [ Γ ( X , W )] − τ ( X ) ] ≤ n − ( β + γ ) . Substituting this gives us the following lower bound on the variance of Z ^ π ( X , W ) \widehat{Z}_{\pi}(X,W) Z π ( X , W ) under P 0 P_0 P 0 .
Var P 0 ( Z ^ π ( X , W ) ) ≥ Var P 0 ( Z π ( X ) ) − 4 M κ n − ( β + γ ) ≥ σ 0 2 2 − 4 M κ n − ( β + γ ) \begin{aligned}
\operatorname{Var}_{P_0}(\widehat{Z}_{\pi}(X,W)) \ge\operatorname{Var}_{P_0}(Z_{\pi}(X))-\frac{4M}{\kappa}n^{-(\beta+\gamma)} \\
&\ge\frac{\sigma_0^2}{2}-\frac{4M}{\kappa}n^{-(\beta+\gamma)}
\end{aligned} Var P 0 ( Z π ( X , W )) ≥ Var P 0 ( Z π ( X )) − κ 4 M n − ( β + γ ) ≥ 2 σ 0 2 − κ 4 M n − ( β + γ )
by Lemma 14 and the assumption that D ( π ) ≤ s max = σ 0 2 / ( 8 M 2 ) D(\pi)\le s_{\max}=\sigma_0^2/(8M^2) D ( π ) ≤ s m a x = σ 0 2 / ( 8 M 2 ) . Therefore, if n ≥ ( 16 M κ σ 0 2 ) 1 / ( β + γ ) n \ge\left(\frac{16M}{\kappa\sigma_0^2}\right)^{1/(\beta+\gamma)} n ≥ ( κ σ 0 2 16 M ) 1/ ( β + γ ) , then Var P 0 ( Z ^ π ( X , W ) ) ≥ σ 0 2 / 4 \operatorname{Var}_{P_0}(\widehat{Z}_{\pi}(X,W))\ge\sigma_0^2/4 Var P 0 ( Z π ( X , W )) ≥ σ 0 2 /4 . Substituting this lower bound on Var Q ~ ( Z ( π ) ) \operatorname{Var}_{\widetilde{Q}}(Z(\pi)) Var Q ( Z ( π )) in the above expression gives us the desired strong concavity constant.
Lemma 15 (Hessian concentration for General Policies). Suppose Assumptions 7 7 7 and 4 4 4 hold. Then for any θ = ( λ , η ) ∈ [ λ 0 / 2 , 2 M / ρ ] × [ − M , M ] \theta=(\lambda,\eta)\in[\lambda_{0}/2,2M/\rho]\times[-M,M] θ = ( λ , η ) ∈ [ λ 0 /2 , 2 M / ρ ] × [ − M , M ] , and for any policy π \pi π we have,
∥ ∇ 2 g ^ θ ( π ) − ∇ 2 g ~ θ ( π ) ∥ o p ≤ C M 2 λ 0 3 κ 2 d log ( L B n ) + log ( 1 / δ ) n ⋅ log ( M n / ( ρ κ λ 0 ) ) \left\|\nabla^{2}\hat{g}_{\theta}(\pi)-\nabla^{2}\tilde{g}_{\theta}(\pi)\right\|_{\mathrm{op}}\leq\frac{CM^{2}}{\lambda_{0}^{3}\kappa^{2}}\sqrt{\frac{d\log(LBn)+\log(1/\delta)}{n\cdot\log(Mn/(\rho\kappa\lambda_{0}))}} ∇ 2 g ^ θ ( π ) − ∇ 2 g ~ θ ( π ) op ≤ λ 0 3 κ 2 C M 2 n ⋅ log ( M n / ( ρ κ λ 0 )) d log ( L B n ) + log ( 1/ δ ) with probability at least 1 − δ 1-\delta 1 − δ for a universal constant C > 0 C>0 C > 0 .
Proof. Fix a policy π \pi π . Then following the same proof of Lemma 9 9 9 we can establish that
∥ ∇ 2 g ^ λ , η ( π ) − ∇ 2 g ~ λ , η ( π ) ∥ o p ≤ t \left\|\nabla^{2}\hat{g}_{\lambda,\eta}(\pi)-\nabla^{2}\tilde{g}_{\lambda,\eta}(\pi)\right\|_{\mathrm{op}}\leq t ∇ 2 g ^ λ , η ( π ) − ∇ 2 g ~ λ , η ( π ) op ≤ t
with probability at least 1 − C ⋅ exp ( − n t 2 κ 4 λ 0 6 M 4 log M ρ κ λ 0 t ) 1-C\cdot\exp\left(-\frac{nt^{2}\kappa^{4}\lambda_{0}^{6}}{M^{4}}\log\frac{M}{\rho\kappa\lambda_{0}t}\right) 1 − C ⋅ exp ( − M 4 n t 2 κ 4 λ 0 6 log ρ κ λ 0 t M ) . We now extend this result to the class of all policies under Assumption 4 4 4 .
Let us define μ ~ k ( λ , η ; π ) = E [ ( Z ^ π ( X ) + η ) k exp ( − ( Z ^ π ( X ) + η ) / λ ) ] \tilde{\mu}_{k}(\lambda,\eta;\pi)=\mathbb{E}\left[(\hat{Z}_{\pi}(X)+\eta)^{k}\exp\left(-(\hat{Z}_{\pi}(X)+\eta)/\lambda\right)\right] μ ~ k ( λ , η ; π ) = E [ ( Z ^ π ( X ) + η ) k exp ( − ( Z ^ π ( X ) + η ) / λ ) ] . Then we can express the Hessian of the (estimated) dual objective as
∇ 2 g ~ λ , η ( π ) = [ − e − 1 μ ~ 2 ( π ) / λ 3 e − 1 μ ~ 1 ( π ) / λ 2 e − 1 μ ~ 1 ( π ) / λ 2 − e − 1 μ ~ 0 ( π ) / λ ] \nabla^{2}\tilde{g}_{\lambda,\eta}(\pi)=
\begin{bmatrix}
-e^{-1}\tilde{\mu}_{2}(\pi)/\lambda^{3} & e^{-1}\tilde{\mu}_{1}(\pi)/\lambda^{2}\\
e^{-1}\tilde{\mu}_{1}(\pi)/\lambda^{2} & -e^{-1}\tilde{\mu}_{0}(\pi)/\lambda
\end{bmatrix} ∇ 2 g ~ λ , η ( π ) = [ − e − 1 μ ~ 2 ( π ) / λ 3 e − 1 μ ~ 1 ( π ) / λ 2 e − 1 μ ~ 1 ( π ) / λ 2 − e − 1 μ ~ 0 ( π ) / λ ]
Consider first the term μ ~ 2 ( λ , η ; π ) \tilde{\mu}_{2}(\lambda,\eta;\pi) μ ~ 2 ( λ , η ; π ) . It can be expressed as E [ h ( Z ^ π ( X ) ) ] \mathbb{E}\left[h(\hat{Z}_{\pi}(X))\right] E [ h ( Z ^ π ( X )) ] where h ( z ) = ( z + η ) 2 exp ( − ( z + η ) / λ ) h(z)=(z+\eta)^{2}\exp(-(z+\eta)/\lambda) h ( z ) = ( z + η ) 2 exp ( − ( z + η ) / λ ) . Moreover,
∣ h ′ ( z ) ∣ = ∣ ( z + η ) exp ( − ( z + η ) / λ ) ⋅ ( 2 − z + η λ ) ∣ ≤ 4 M 2 κ 2 λ 0 exp ( 4 M κ λ 0 ) . \left|h'(z)\right|=\left|(z+\eta)\exp(-(z+\eta)/\lambda)\cdot\left(2-\frac{z+\eta}{\lambda}\right)\right|\leq\frac{4M^{2}}{\kappa^{2}\lambda_{0}}\exp\left(\frac{4M}{\kappa\lambda_{0}}\right). ∣ h ′ ( z ) ∣ = ( z + η ) exp ( − ( z + η ) / λ ) ⋅ ( 2 − λ z + η ) ≤ κ 2 λ 0 4 M 2 exp ( κ λ 0 4 M ) .
Therefore,
∣ μ ~ 2 ( λ , η ; π 1 ) − μ ~ 2 ( λ , η ; π 2 ) ∣ ≤ E [ ∣ h ( Z ^ π 1 ( X ) ) − h ( Z ^ π 2 ( X ) ) ∣ ] ≤ 4 M 2 κ 2 λ 0 exp ( 4 M κ λ 0 ) ⋅ E [ ∣ Z ^ π 1 ( X ) − Z ^ π 2 ( X ) ∣ ] ≤ 8 M 3 κ 3 λ 0 exp ( 4 M κ λ 0 ) ⋅ ∥ π 1 − π 2 ∥ ∞ . \begin{aligned}
\left|\tilde{\mu}_{2}(\lambda,\eta;\pi_{1})-\tilde{\mu}_{2}(\lambda,\eta;\pi_{2})\right|
\leq\mathbb{E}\left[\left|h(\hat{Z}_{\pi_{1}}(X))-h(\hat{Z}_{\pi_{2}}(X))\right|\right] \\
&\leq\frac{4M^{2}}{\kappa^{2}\lambda_{0}}\exp\left(\frac{4M}{\kappa\lambda_{0}}\right)\cdot\mathbb{E}\left[\left|\hat{Z}_{\pi_{1}}(X)-\hat{Z}_{\pi_{2}}(X)\right|\right] \\
&\leq\frac{8M^{3}}{\kappa^{3}\lambda_{0}}\exp\left(\frac{4M}{\kappa\lambda_{0}}\right)\cdot\|\pi_{1}-\pi_{2}\|_{\infty}.
\end{aligned} ∣ μ ~ 2 ( λ , η ; π 1 ) − μ ~ 2 ( λ , η ; π 2 ) ∣ ≤ E [ h ( Z ^ π 1 ( X )) − h ( Z ^ π 2 ( X )) ] ≤ κ 2 λ 0 4 M 2 exp ( κ λ 0 4 M ) ⋅ E [ Z ^ π 1 ( X ) − Z ^ π 2 ( X ) ] ≤ κ 3 λ 0 8 M 3 exp ( κ λ 0 4 M ) ⋅ ∥ π 1 − π 2 ∥ ∞ .
We can similarly establish that μ ~ 1 ( λ , η ; π ) \tilde{\mu}_{1}(\lambda,\eta;\pi) μ ~ 1 ( λ , η ; π ) and μ ~ 0 ( λ , η ; π ) \tilde{\mu}_{0}(\lambda,\eta;\pi) μ ~ 0 ( λ , η ; π ) are also Lipschitz in π \pi π with the same Lipschitz constant. Therefore,
∥ ∇ 2 g ~ λ , η ( π 1 ) − ∇ 2 g ~ λ , η ( π 2 ) ∥ o p ≤ 8 M 3 κ 3 λ 0 3 exp ( 4 M κ λ 0 ) ⋅ ∥ π 1 − π 2 ∥ ∞ . \left\|\nabla^{2}\tilde{g}_{\lambda,\eta}(\pi_{1})-\nabla^{2}\tilde{g}_{\lambda,\eta}(\pi_{2})\right\|_{\mathrm{op}}\leq\frac{8M^{3}}{\kappa^{3}\lambda_{0}^{3}}\exp\left(\frac{4M}{\kappa\lambda_{0}}\right)\cdot\|\pi_{1}-\pi_{2}\|_{\infty}. ∇ 2 g ~ λ , η ( π 1 ) − ∇ 2 g ~ λ , η ( π 2 ) op ≤ κ 3 λ 0 3 8 M 3 exp ( κ λ 0 4 M ) ⋅ ∥ π 1 − π 2 ∥ ∞ .
Let us write L = 8 M 3 κ 3 λ 0 3 exp ( 4 M κ λ 0 ) L=\frac{8M^{3}}{\kappa^{3}\lambda_{0}^{3}}\exp\left(\frac{4M}{\kappa\lambda_{0}}\right) L = κ 3 λ 0 3 8 M 3 exp ( κ λ 0 4 M ) .
Since ∇ 2 g ~ λ , η ( π ϕ ) \nabla^{2}\tilde{g}_{\lambda,\eta}(\pi_{\phi}) ∇ 2 g ~ λ , η ( π ϕ ) is L L L -Lipschitz in ϕ \phi ϕ (with respect to ∥ ⋅ ∥ 2 \|\cdot\|_{2} ∥ ⋅ ∥ 2 ). Let us pick an ϵ \epsilon ϵ -net Φ ϵ \Phi_{\epsilon} Φ ϵ of Φ \Phi Φ of size N ϵ ≤ ( 1 + 2 B / ϵ ) d N_{\epsilon}\leq(1+2B/\epsilon)^{d} N ϵ ≤ ( 1 + 2 B / ϵ ) d . A union bound over the net at precision ϵ = t / ( 2 L ) \epsilon=t/(2L) ϵ = t / ( 2 L ) gives:
P ( sup ϕ ∈ Φ t / 2 L ∥ ∇ 2 g ^ λ , η ( π ϕ ) − ∇ 2 g ~ λ , η ( π ϕ ) ∥ ≥ t ) ≤ 2 C ( 1 + 4 L B t ) d exp ( − n t 2 κ 4 λ 0 6 M 4 log M ρ κ λ 0 t ) . P\left(\sup_{\phi\in\Phi_{t/2L}}\left\|\nabla^{2}\hat{g}_{\lambda,\eta}(\pi_{\phi})-\nabla^{2}\tilde{g}_{\lambda,\eta}(\pi_{\phi})\right\|\geq t\right)
\leq2C\left(1+\frac{4LB}{t}\right)^{d}\exp\left(-\frac{nt^{2}\kappa^{4}\lambda_{0}^{6}}{M^{4}}\log\frac{M}{\rho\kappa\lambda_{0}t}\right). P ( ϕ ∈ Φ t /2 L sup ∇ 2 g ^ λ , η ( π ϕ ) − ∇ 2 g ~ λ , η ( π ϕ ) ≥ t ) ≤ 2 C ( 1 + t 4 L B ) d exp ( − M 4 n t 2 κ 4 λ 0 6 log ρ κ λ 0 t M ) .
Now for an arbitrary ϕ ∈ Φ \phi\in\Phi ϕ ∈ Φ , let ϕ ′ ∈ Φ t / 2 L \phi'\in\Phi_{t/2L} ϕ ′ ∈ Φ t /2 L be the closest point in the net. Then:
∥ ∇ 2 g ^ λ , η ( π ϕ ) − ∇ 2 g ~ λ , η ( π ϕ ) ∥ ≤ ∥ ∇ 2 g ^ λ , η ( π ϕ ) − ∇ 2 g ^ λ , η ( π ϕ ′ ) ∥ + ∥ ∇ 2 g ^ λ , η ( π ϕ ′ ) − ∇ 2 g ~ λ , η ( π ϕ ′ ) ∥ + ∥ ∇ 2 g ~ λ , η ( π ϕ ′ ) − ∇ 2 g ~ λ , η ( π ϕ ) ∥ \begin{aligned}
\left\|\nabla^{2}\hat{g}_{\lambda,\eta}(\pi_{\phi})-\nabla^{2}\tilde{g}_{\lambda,\eta}(\pi_{\phi})\right\| \\
&\leq\left\|\nabla^{2}\hat{g}_{\lambda,\eta}(\pi_{\phi})-\nabla^{2}\hat{g}_{\lambda,\eta}(\pi_{\phi'})\right\|+\left\|\nabla^{2}\hat{g}_{\lambda,\eta}(\pi_{\phi'})-\nabla^{2}\tilde{g}_{\lambda,\eta}(\pi_{\phi'})\right\|+\left\|\nabla^{2}\tilde{g}_{\lambda,\eta}(\pi_{\phi'})-\nabla^{2}\tilde{g}_{\lambda,\eta}(\pi_{\phi})\right\|
\end{aligned} ∇ 2 g ^ λ , η ( π ϕ ) − ∇ 2 g ~ λ , η ( π ϕ ) ≤ ∇ 2 g ^ λ , η ( π ϕ ) − ∇ 2 g ^ λ , η ( π ϕ ′ ) + ∇ 2 g ^ λ , η ( π ϕ ′ ) − ∇ 2 g ~ λ , η ( π ϕ ′ ) + ∇ 2 g ~ λ , η ( π ϕ ′ ) − ∇ 2 g ~ λ , η ( π ϕ )
≤ 2 L ∥ ϕ − ϕ ′ ∥ 2 + ∥ ∇ 2 g ^ λ , η ( π ϕ ′ ) − ∇ 2 g ~ λ , η ( π ϕ ′ ) ∥ ≤ t . \leq2L\|\phi-\phi'\|_{2}+\left\|\nabla^{2}\hat{g}_{\lambda,\eta}(\pi_{\phi'})-\nabla^{2}\tilde{g}_{\lambda,\eta}(\pi_{\phi'})\right\|\leq t. ≤ 2 L ∥ ϕ − ϕ ′ ∥ 2 + ∇ 2 g ^ λ , η ( π ϕ ′ ) − ∇ 2 g ~ λ , η ( π ϕ ′ ) ≤ t .
Therefore, we have,
P ( sup π ∈ Π ∥ ∇ 2 g ^ λ , η ( π ) − ∇ 2 g ~ λ , η ( π ) ∥ ≥ t ) ≤ 2 ( 1 + 4 L B t ) d exp ( − n t 2 κ 4 λ 0 6 M 4 log M ρ κ λ 0 t ) . P\left(\sup_{\pi\in\Pi}\left\|\nabla^{2}\hat{g}_{\lambda,\eta}(\pi)-\nabla^{2}\tilde{g}_{\lambda,\eta}(\pi)\right\|\geq t\right)
\leq2\left(1+\frac{4LB}{t}\right)^{d}\exp\left(-\frac{nt^{2}\kappa^{4}\lambda_{0}^{6}}{M^{4}}\log\frac{M}{\rho\kappa\lambda_{0}t}\right). P ( π ∈ Π sup ∇ 2 g ^ λ , η ( π ) − ∇ 2 g ~ λ , η ( π ) ≥ t ) ≤ 2 ( 1 + t 4 L B ) d exp ( − M 4 n t 2 κ 4 λ 0 6 log ρ κ λ 0 t M ) .
Let us define D 1 = κ 4 λ 0 6 2 M 4 D_{1}=\frac{\kappa^{4}\lambda_{0}^{6}}{2M^{4}} D 1 = 2 M 4 κ 4 λ 0 6 and D 2 = M ρ κ λ 0 D_{2}=\frac{M}{\rho\kappa\lambda_{0}} D 2 = ρ κ λ 0 M . Then the above failure probability can be rewritten as
exp ( 4 d L B t − 2 D 1 t 2 log ( D 2 / t ) ) . \exp\left(\frac{4dLB}{t}-2D_{1}t^{2}\log(D_{2}/t)\right). exp ( t 4 d L B − 2 D 1 t 2 log ( D 2 / t ) ) .
Setting t = O ( d log ( L B n ) + log ( 1 / δ ) D 1 n ⋅ log ( D 2 n / d ) ) t=O\left(\sqrt{\frac{d\log(LBn)+\log(1/\delta)}{D_{1}n\cdot\log(D_{2}n/d)}}\right) t = O ( D 1 n ⋅ l o g ( D 2 n / d ) d l o g ( L B n ) + l o g ( 1/ δ ) ) yields the claim. □
Lemma 16 (Dual variable stability). Suppose Assumptions 1 1 1 , 2 2 2 , 4 4 4 , 7 7 7 hold and n ≥ O ~ ( M 14 e 2 ρ / κ κ 8 ρ 6 λ 0 10 σ 0 4 ⋅ ( d + log ( 1 / δ ) ) ) n \ge\widetilde{O}\left(\frac{M^{14}e^{2\rho/\kappa}}{\kappa^{8}\rho^{6}\lambda_{0}^{10}\sigma_{0}^{4}}\cdot\left(d+\log(1/\delta)\right)\right) n ≥ O ( κ 8 ρ 6 λ 0 10 σ 0 4 M 14 e 2 ρ / κ ⋅ ( d + log ( 1/ δ ) ) ) . Let θ ^ ( π ) = ( λ ^ ( π ) , η ^ ( π ) ) \hat{\theta}(\pi)=(\hat{\lambda}(\pi),\hat{\eta}(\pi)) θ ^ ( π ) = ( λ ^ ( π ) , η ^ ( π )) denote the empirical dual optimizer on D 1 D_{1} D 1 for policy π \pi π . Then for any two policies π , π ~ \pi,\tilde{\pi} π , π ~ with 1 ∣ D 1 ∣ ∑ i ∈ D 1 ∣ π ( X i ) − π ~ ( X i ) ∣ ≤ ε \frac{1}{|D_{1}|}\sum_{i\in D_{1}}|\pi(X_i)-\tilde{\pi}(X_i)|\le\varepsilon ∣ D 1 ∣ 1 ∑ i ∈ D 1 ∣ π ( X i ) − π ~ ( X i ) ∣ ≤ ε and max ( D ( π ) , D ( π ~ ) ) ≤ s max = σ 0 2 8 M 2 \max(D(\pi),D(\tilde{\pi}))\le s_{\max}=\frac{\sigma_{0}^{2}}{8M^{2}} max ( D ( π ) , D ( π ~ )) ≤ s m a x = 8 M 2 σ 0 2 :
∥ θ ^ ( π ) − θ ^ ( π ~ ) ∥ ≤ 1024 M 7 λ 0 6 κ 4 ρ 3 σ 0 2 exp ( 1 + ρ κ + 4 M κ λ 0 ) ε \left\|\hat{\theta}(\pi)-\hat{\theta}(\tilde{\pi})\right\|\le\frac{1024M^{7}}{\lambda_{0}^{6}\kappa^{4}\rho^{3}\sigma_{0}^{2}}\exp\left(1+\frac{\rho}{\kappa}+\frac{4M}{\kappa\lambda_{0}}\right)\varepsilon θ ^ ( π ) − θ ^ ( π ~ ) ≤ λ 0 6 κ 4 ρ 3 σ 0 2 1024 M 7 exp ( 1 + κ ρ + κ λ 0 4 M ) ε with probability at least 1 − δ 1-\delta 1 − δ .
Proof. The proof has two main steps. We first bound the pointwise perturbation of the gradient, and then use strong concavity to convert gradient perturbation into optimizer perturbation.
Step 1: Perturbation of the dual objective. The empirical dual for policy π \pi π on D 1 D_{1} D 1 is
g ^ λ , η ( π ) = − λ e − 1 ∣ D 1 ∣ ∑ i ∈ D 1 e ( − Z i ( π ) − η ) / λ − λ ρ − η \hat{g}_{\lambda,\eta}(\pi)=-\frac{\lambda e^{-1}}{|D_{1}|}\sum_{i\in D_{1}}e^{(-Z_i(\pi)-\eta)/\lambda}-\lambda\rho-\eta g ^ λ , η ( π ) = − ∣ D 1 ∣ λ e − 1 i ∈ D 1 ∑ e ( − Z i ( π ) − η ) / λ − λ ρ − η
where Z i ( π ) = ( 2 π ( X i ) − 1 ) Γ ^ i Z_i(\pi)=(2\pi(X_i)-1)\hat{\Gamma}_i Z i ( π ) = ( 2 π ( X i ) − 1 ) Γ ^ i . Let us also define the perturbation in the dual objective as the policy is changed from π \pi π to π ~ \tilde{\pi} π ~ .
Δ g ^ λ , η ( π , π ~ ) = g ^ λ , η ( π ) − g ^ λ , η ( π ~ ) = − λ e − 1 ∣ D 1 ∣ ∑ i ∈ D 1 [ e ( − Z i ( π ) − η ) / λ − e ( − Z i ( π ~ ) − η ) / λ ] = − λ e − 1 − η / λ ∣ D 1 ∣ ∑ i ∈ D 1 [ e − Z i ( π ) / λ − e − Z i ( π ~ ) / λ ] \begin{aligned}
\Delta\hat{g}_{\lambda,\eta}(\pi,\tilde{\pi})=\hat{g}_{\lambda,\eta}(\pi)-\hat{g}_{\lambda,\eta}(\tilde{\pi})=-\frac{\lambda e^{-1}}{|D_{1}|}\sum_{i\in D_{1}}\left[e^{(-Z_i(\pi)-\eta)/\lambda}-e^{(-Z_i(\tilde{\pi})-\eta)/\lambda}\right] \\
&=-\frac{\lambda e^{-1-\eta/\lambda}}{|D_{1}|}\sum_{i\in D_{1}}\left[e^{-Z_i(\pi)/\lambda}-e^{-Z_i(\tilde{\pi})/\lambda}\right]
\end{aligned} Δ g ^ λ , η ( π , π ~ ) = g ^ λ , η ( π ) − g ^ λ , η ( π ~ ) = − ∣ D 1 ∣ λ e − 1 i ∈ D 1 ∑ [ e ( − Z i ( π ) − η ) / λ − e ( − Z i ( π ~ ) − η ) / λ ] = − ∣ D 1 ∣ λ e − 1 − η / λ i ∈ D 1 ∑ [ e − Z i ( π ) / λ − e − Z i ( π ~ ) / λ ]
We now bound the partial derivatives of the perturbation Δ g ^ λ , η ( π , π ~ ) \Delta\hat{g}_{\lambda,\eta}(\pi,\tilde{\pi}) Δ g ^ λ , η ( π , π ~ ) with respect to λ \lambda λ and η \eta η .
∂ Δ g ^ λ , η ( π , π ~ ) ∂ η = e − 1 − η / λ ∣ D 1 ∣ ∑ i ∈ D 1 [ e − Z i ( π ) / λ − e − Z i ( π ~ ) / λ ] . \frac{\partial\Delta\hat{g}_{\lambda,\eta}(\pi,\tilde{\pi})}{\partial\eta}=\frac{e^{-1-\eta/\lambda}}{|D_{1}|}\sum_{i\in D_{1}}\left[e^{-Z_i(\pi)/\lambda}-e^{-Z_i(\tilde{\pi})/\lambda}\right]. ∂ η ∂ Δ g ^ λ , η ( π , π ~ ) = ∣ D 1 ∣ e − 1 − η / λ i ∈ D 1 ∑ [ e − Z i ( π ) / λ − e − Z i ( π ~ ) / λ ] .
By the mean value theorem applied to z ↦ e − z / λ z\mapsto e^{-z/\lambda} z ↦ e − z / λ with derivative − 1 λ e − z / λ -\frac{1}{\lambda}e^{-z/\lambda} − λ 1 e − z / λ :
∣ e − Z i ( π ) / λ − e − Z i ( π ~ ) / λ ∣ ≤ 1 λ exp ( M κ λ ) ∣ Z i ( π ) − Z i ( π ~ ) ∣ ≤ 2 λ exp ( M κ λ ) ∣ π ( X i ) − π ~ ( X i ) ∣ ∣ Γ ^ i ∣ ≤ 2 M κ λ exp ( M κ λ ) ∣ π ( X i ) − π ~ ( X i ) ∣ . \begin{aligned}
\left|e^{-Z_i(\pi)/\lambda}-e^{-Z_i(\tilde{\pi})/\lambda}\right|\le\frac{1}{\lambda}\exp\left(\frac{M}{\kappa\lambda}\right)|Z_i(\pi)-Z_i(\tilde{\pi})| \\
&\le\frac{2}{\lambda}\exp\left(\frac{M}{\kappa\lambda}\right)|\pi(X_i)-\tilde{\pi}(X_i)|\,|\hat{\Gamma}_i| \\
&\le\frac{2M}{\kappa\lambda}\exp\left(\frac{M}{\kappa\lambda}\right)|\pi(X_i)-\tilde{\pi}(X_i)|.
\end{aligned} e − Z i ( π ) / λ − e − Z i ( π ~ ) / λ ≤ λ 1 exp ( κλ M ) ∣ Z i ( π ) − Z i ( π ~ ) ∣ ≤ λ 2 exp ( κλ M ) ∣ π ( X i ) − π ~ ( X i ) ∣ ∣ Γ ^ i ∣ ≤ κλ 2 M exp ( κλ M ) ∣ π ( X i ) − π ~ ( X i ) ∣.
For λ ≥ λ 0 / 2 \lambda\ge\lambda_{0}/2 λ ≥ λ 0 /2 and ∣ η ∣ ≤ M |\eta|\le M ∣ η ∣ ≤ M :
∣ ∂ Δ g ^ λ , η ( π , π ~ ) ∂ η ∣ ≤ 4 M κ λ 0 exp ( 4 M κ λ 0 ) ε . (26) \left|\frac{\partial\Delta\hat{g}_{\lambda,\eta}(\pi,\tilde{\pi})}{\partial\eta}\right|\le\frac{4M}{\kappa\lambda_{0}}\exp\left(\frac{4M}{\kappa\lambda_{0}}\right)\varepsilon. \tag*{(26)} ∂ η ∂ Δ g ^ λ , η ( π , π ~ ) ≤ κ λ 0 4 M exp ( κ λ 0 4 M ) ε . (26)
Let us now define h ( z ) = e ( − z − η ) / λ ( 1 + z + η λ ) h(z)=e^{(-z-\eta)/\lambda}\left(1+\frac{z+\eta}{\lambda}\right) h ( z ) = e ( − z − η ) / λ ( 1 + λ z + η ) . Then
∂ Δ g ^ λ , η ( π , π ~ ) ∂ λ = − e − 1 ∣ D 1 ∣ ∑ i ∈ D 1 [ h ( Z i ( π ) ) − h ( Z i ( π ~ ) ) ] . \frac{\partial\Delta\hat{g}_{\lambda,\eta}(\pi,\tilde{\pi})}{\partial\lambda}=-\frac{e^{-1}}{|D_{1}|}\sum_{i\in D_{1}}\left[h(Z_i(\pi))-h(Z_i(\tilde{\pi}))\right]. ∂ λ ∂ Δ g ^ λ , η ( π , π ~ ) = − ∣ D 1 ∣ e − 1 i ∈ D 1 ∑ [ h ( Z i ( π )) − h ( Z i ( π ~ )) ] .
The derivative of h h h with respect to z z z is
h ′ ( z ) = − 1 λ e ( − z − η ) / λ ( 1 + z + η λ ) + 1 λ e ( − z − η ) / λ = − z + η λ 2 e ( − z − η ) / λ . h'(z)=-\frac{1}{\lambda}e^{(-z-\eta)/\lambda}\left(1+\frac{z+\eta}{\lambda}\right)+\frac{1}{\lambda}e^{(-z-\eta)/\lambda}=-\frac{z+\eta}{\lambda^{2}}e^{(-z-\eta)/\lambda}. h ′ ( z ) = − λ 1 e ( − z − η ) / λ ( 1 + λ z + η ) + λ 1 e ( − z − η ) / λ = − λ 2 z + η e ( − z − η ) / λ .
As ∣ Z i ( π ) ∣ ≤ M / κ |Z_i(\pi)|\le M/\kappa ∣ Z i ( π ) ∣ ≤ M / κ , we have
∣ h ′ ( Z i ( π ) ) ∣ ≤ ∣ Z i ( π ) ∣ + ∣ η ∣ λ 2 e ( ∣ Z i ( π ) ∣ + ∣ η ∣ ) / λ ≤ M / κ + M λ 2 e ( M / κ + M ) / λ , |h'(Z_i(\pi))|\le\frac{|Z_i(\pi)|+|\eta|}{\lambda^{2}}e^{(|Z_i(\pi)|+|\eta|)/\lambda}\le\frac{M/\kappa+M}{\lambda^{2}}e^{(M/\kappa+M)/\lambda}, ∣ h ′ ( Z i ( π )) ∣ ≤ λ 2 ∣ Z i ( π ) ∣ + ∣ η ∣ e ( ∣ Z i ( π ) ∣ + ∣ η ∣ ) / λ ≤ λ 2 M / κ + M e ( M / κ + M ) / λ ,
and by the Mean value theorem applied to h h h :
∣ h ( Z i ( π ) ) − h ( Z i ( π ~ ) ) ∣ ≤ 2 M κ λ 2 exp ( 2 M κ λ ) ∣ Z i ( π ) − Z i ( π ~ ) ∣ ≤ 2 M 2 κ 2 λ 2 exp ( 2 M κ λ ) ∣ π ( X i ) − π ~ ( X i ) ∣ . |h(Z_i(\pi))-h(Z_i(\tilde{\pi}))|\le\frac{2M}{\kappa\lambda^{2}}\exp\left(\frac{2M}{\kappa\lambda}\right)|Z_i(\pi)-Z_i(\tilde{\pi})|\le\frac{2M^{2}}{\kappa^{2}\lambda^{2}}\exp\left(\frac{2M}{\kappa\lambda}\right)|\pi(X_i)-\tilde{\pi}(X_i)|. ∣ h ( Z i ( π )) − h ( Z i ( π ~ )) ∣ ≤ κ λ 2 2 M exp ( κλ 2 M ) ∣ Z i ( π ) − Z i ( π ~ ) ∣ ≤ κ 2 λ 2 2 M 2 exp ( κλ 2 M ) ∣ π ( X i ) − π ~ ( X i ) ∣.
For λ ≥ λ 0 / 2 \lambda\ge\lambda_0/2 λ ≥ λ 0 /2 and ∣ η ∣ ≤ M |\eta| \le M ∣ η ∣ ≤ M :
∣ ∂ Δ g ^ λ , η ( π , π ~ ) ∂ λ ∣ ≤ 4 M 2 κ 2 λ 0 2 exp ( 4 M κ λ 0 ) ε . (27) \left|\frac{\partial\Delta\hat{g}_{\lambda,\eta}(\pi,\tilde{\pi})}{\partial\lambda}\right| \le\frac{4M^2}{\kappa^2\lambda_0^2}\exp\left(\frac{4M}{\kappa\lambda_0}\right)\varepsilon. \tag*{(27)} ∂ λ ∂ Δ g ^ λ , η ( π , π ~ ) ≤ κ 2 λ 0 2 4 M 2 exp ( κ λ 0 4 M ) ε . (27)
Combining (26) and (27) :
∥ ∇ Δ g ^ λ , η ( π , π ~ ) ∥ ≤ 8 M 2 κ 2 λ 0 2 exp ( 4 M κ λ 0 ) ε for all θ ∈ Θ . (28) \left\|\nabla\Delta\hat{g}_{\lambda,\eta}(\pi,\tilde{\pi})\right\| \le\frac{8M^2}{\kappa^2\lambda_0^2}\exp\left(\frac{4M}{\kappa\lambda_0}\right)\varepsilon\quad\text{for all }\theta\in\Theta. \tag*{(28)} ∥ ∇Δ g ^ λ , η ( π , π ~ ) ∥ ≤ κ 2 λ 0 2 8 M 2 exp ( κ λ 0 4 M ) ε for all θ ∈ Θ. (28)
Step 2: From gradient perturbation to optimizer perturbation.
Let θ ^ 1 = θ ^ ( π ) \hat{\theta}_1=\hat{\theta}(\pi) θ ^ 1 = θ ^ ( π ) and θ ^ 2 = θ ^ ( π ~ ) \hat{\theta}_2=\hat{\theta}(\tilde{\pi}) θ ^ 2 = θ ^ ( π ~ ) . Suppose first that both are interior optimizers (the boundary case is handled below). The first-order conditions are
∇ g ^ θ ^ 1 ( π ) = 0 , ∇ g ^ θ ^ 2 ( π ~ ) = 0. \nabla\hat{g}_{\hat{\theta}_1}(\pi)=0,\qquad\nabla\hat{g}_{\hat{\theta}_2}(\tilde{\pi})=0. ∇ g ^ θ ^ 1 ( π ) = 0 , ∇ g ^ θ ^ 2 ( π ~ ) = 0.
Since g ^ θ ^ 1 ( π ) = g ^ θ ^ 1 ( π ~ ) + Δ g ^ θ ^ 1 ( π , π ~ ) \hat{g}_{\hat{\theta}_1}(\pi)=\hat{g}_{\hat{\theta}_1}(\tilde{\pi})+\Delta\hat{g}_{\hat{\theta}_1}(\pi,\tilde{\pi}) g ^ θ ^ 1 ( π ) = g ^ θ ^ 1 ( π ~ ) + Δ g ^ θ ^ 1 ( π , π ~ ) , the first condition becomes
∇ g ^ θ ^ 1 ( π ~ ) + ∇ Δ g ^ θ ^ 1 ( π , π ~ ) = 0. (29) \nabla\hat{g}_{\hat{\theta}_1}(\tilde{\pi})+\nabla\Delta\hat{g}_{\hat{\theta}_1}(\pi,\tilde{\pi})=0. \tag*{(29)} ∇ g ^ θ ^ 1 ( π ~ ) + ∇Δ g ^ θ ^ 1 ( π , π ~ ) = 0. (29)
By the mean value theorem applied to ∇ g ^ ( ⋅ ) ( π ~ ) \nabla\hat{g}_{(\cdot)}(\tilde{\pi}) ∇ g ^ ( ⋅ ) ( π ~ ) between θ ^ 2 \hat{\theta}_2 θ ^ 2 and θ ^ 1 \hat{\theta}_1 θ ^ 1 , there exists θ ˉ \bar{\theta} θ ˉ on the segment between θ ^ 1 \hat{\theta}_1 θ ^ 1 and θ ^ 2 \hat{\theta}_2 θ ^ 2 such that
∇ g ^ θ ^ 1 ( π ~ ) = ∇ g ^ θ ^ 2 ( π ~ ) ⏟ = 0 + ∇ 2 g ^ θ ˉ ( π ~ ) ( θ ^ 1 − θ ^ 2 ) . \nabla\hat{g}_{\hat{\theta}_1}(\tilde{\pi})=\underbrace{\nabla\hat{g}_{\hat{\theta}_2}(\tilde{\pi})}_{=0}+\nabla^2\hat{g}_{\bar{\theta}}(\tilde{\pi})(\hat{\theta}_1-\hat{\theta}_2). ∇ g ^ θ ^ 1 ( π ~ ) = = 0 ∇ g ^ θ ^ 2 ( π ~ ) + ∇ 2 g ^ θ ˉ ( π ~ ) ( θ ^ 1 − θ ^ 2 ) .
Substituting into (29) and rearranging:
∇ 2 g ^ θ ˉ ( π ~ ) ( θ ^ 1 − θ ^ 2 ) = − ∇ Δ g ^ θ ^ 1 ( π , π ~ ) . (30) \nabla^2\hat{g}_{\bar{\theta}}(\tilde{\pi})(\hat{\theta}_1-\hat{\theta}_2)=-\nabla\Delta\hat{g}_{\hat{\theta}_1}(\pi,\tilde{\pi}). \tag*{(30)} ∇ 2 g ^ θ ˉ ( π ~ ) ( θ ^ 1 − θ ^ 2 ) = − ∇Δ g ^ θ ^ 1 ( π , π ~ ) . (30)
By Corollary 2 (which applies since D ( π ~ ) ≤ s max D(\tilde{\pi})\le s_{\max} D ( π ~ ) ≤ s m a x ) and Lemma 15 (Hessian concentration for general policies),
λ min ( − ∇ 2 g ^ θ ˉ ( π ~ ) ) ≥ λ 0 2 ρ 3 κ 2 σ 0 2 64 M 5 e − ( 1 + ρ / κ ) ⏟ ≔ γ ˉ − C M 2 λ 0 3 κ 2 d log ( L B n ) + log ( 1 / δ ) n ⋅ log ( M n / ( ρ κ λ 0 ) ) ≥ γ ˉ 2 \lambda_{\min}\left(-\nabla^2\hat{g}_{\bar{\theta}}(\tilde{\pi})\right)\ge\underbrace{\frac{\lambda_0^2\rho^3\kappa^2\sigma_0^2}{64M^5}e^{-(1+\rho/\kappa)}}_{\coloneqq\bar{\gamma}}-\frac{CM^2}{\lambda_0^3\kappa^2}\sqrt{\frac{d\log(LBn)+\log(1/\delta)}{n\cdot\log(Mn/(\rho\kappa\lambda_0))}}\ge\frac{\bar{\gamma}}{2} λ m i n ( − ∇ 2 g ^ θ ˉ ( π ~ ) ) ≥ : = γ ˉ 64 M 5 λ 0 2 ρ 3 κ 2 σ 0 2 e − ( 1 + ρ / κ ) − λ 0 3 κ 2 C M 2 n ⋅ log ( M n / ( ρ κ λ 0 )) d log ( L B n ) + log ( 1/ δ ) ≥ 2 γ ˉ
as long as n ≥ O ~ ( M 14 e 2 ρ / κ κ 8 ρ 6 λ 0 10 σ 0 4 ⋅ ( d + log ( 1 / δ ) ) ) n\ge\widetilde{O}\left(\frac{M^{14}e^{2\rho/\kappa}}{\kappa^8\rho^6\lambda_0^{10}\sigma_0^4}\cdot(d+\log(1/\delta))\right) n ≥ O ( κ 8 ρ 6 λ 0 10 σ 0 4 M 14 e 2 ρ / κ ⋅ ( d + log ( 1/ δ )) ) . Therefore − ∇ 2 g ^ θ ˉ ( π ~ ) -\nabla^2\hat{g}_{\bar{\theta}}(\tilde{\pi}) − ∇ 2 g ^ θ ˉ ( π ~ ) is invertible, and
∥ [ − ∇ 2 g ^ θ ˉ ( π ~ ) ] − 1 ∥ o p ≤ 2 γ ˉ , \left\|\left[-\nabla^2\hat{g}_{\bar{\theta}}(\tilde{\pi})\right]^{-1}\right\|_{\mathrm{op}}\le\frac{2}{\bar{\gamma}}, [ − ∇ 2 g ^ θ ˉ ( π ~ ) ] − 1 op ≤ γ ˉ 2 ,
which gives us the following bound.
∥ θ ^ 1 − θ ^ 2 ∥ ≤ ∥ [ − ∇ 2 g ^ θ ˉ ( π ~ ) ] − 1 ∥ o p ∥ ∇ Δ g ^ θ ^ 1 ( π , π ~ ) ∥ ≤ 16 M 2 γ ˉ κ 2 λ 0 2 exp ( 4 M κ λ 0 ) ε . \|\hat{\theta}_1-\hat{\theta}_2\|\le\left\|\left[-\nabla^2\hat{g}_{\bar{\theta}}(\tilde{\pi})\right]^{-1}\right\|_{\mathrm{op}}\left\|\nabla\Delta\hat{g}_{\hat{\theta}_1}(\pi,\tilde{\pi})\right\|\le\frac{16M^2}{\bar{\gamma}\kappa^2\lambda_0^2}\exp\left(\frac{4M}{\kappa\lambda_0}\right)\varepsilon. ∥ θ ^ 1 − θ ^ 2 ∥ ≤ [ − ∇ 2 g ^ θ ˉ ( π ~ ) ] − 1 op ∇Δ g ^ θ ^ 1 ( π , π ~ ) ≤ γ ˉ κ 2 λ 0 2 16 M 2 exp ( κ λ 0 4 M ) ε .
Boundary case. If θ ^ 1 \hat{\theta}_1 θ ^ 1 or θ ^ 2 \hat{\theta}_2 θ ^ 2 lies on the boundary of the constraint set Θ = [ λ 0 / 2 , Λ ] × [ − R , R ] \Theta=[\lambda_0/2,\Lambda]\times[-R,R] Θ = [ λ 0 /2 , Λ ] × [ − R , R ] , the first-order condition is replaced by the variational inequality for constrained optimization. Without loss of generality suppose θ ^ 1 \hat{\theta}_1 θ ^ 1 lies at the boundary. Then, ⟨ ∇ g ^ θ ^ 1 ( π ) , θ − θ ^ 1 ⟩ ≤ 0 \langle\nabla\hat{g}_{\hat{\theta}_1}(\pi),\theta-\hat{\theta}_1\rangle\le0 ⟨ ∇ g ^ θ ^ 1 ( π ) , θ − θ ^ 1 ⟩ ≤ 0 for all feasible θ \theta θ . Taking θ = θ ^ 2 \theta=\hat{\theta}_2 θ = θ ^ 2 and using (29) :
⟨ ∇ g ^ θ ^ 1 ( π ~ ) + ∇ Δ g ^ θ ^ 1 ( π , π ~ ) , θ ^ 2 − θ ^ 1 ⟩ ≥ 0. \left\langle\nabla\hat{g}_{\hat{\theta}_1}(\tilde{\pi})+\nabla\Delta\hat{g}_{\hat{\theta}_1}(\pi,\tilde{\pi}),\hat{\theta}_2-\hat{\theta}_1\right\rangle\ge0. ⟨ ∇ g ^ θ ^ 1 ( π ~ ) + ∇Δ g ^ θ ^ 1 ( π , π ~ ) , θ ^ 2 − θ ^ 1 ⟩ ≥ 0.
Expanding ∇ g ^ θ ^ 1 ( π ~ ) \nabla\hat{g}_{\hat{\theta}_1}(\tilde{\pi}) ∇ g ^ θ ^ 1 ( π ~ ) via the mean value theorem as before:
( θ ^ 1 − θ ^ 2 ) ⊤ ( − ∇ 2 g ^ θ ˉ ( π ~ ) ) ( θ ^ 1 − θ ^ 2 ) ≤ ( θ ^ 1 − θ ^ 2 ) ⊤ ∇ Δ g ^ θ ^ 1 ( π , π ~ ) . (\hat{\theta}_1-\hat{\theta}_2)^{\top}(-\nabla^2\hat{g}_{\bar{\theta}}(\tilde{\pi}))(\hat{\theta}_1-\hat{\theta}_2)\le(\hat{\theta}_1-\hat{\theta}_2)^{\top}\nabla\Delta\hat{g}_{\hat{\theta}_1}(\pi,\tilde{\pi}). ( θ ^ 1 − θ ^ 2 ) ⊤ ( − ∇ 2 g ^ θ ˉ ( π ~ )) ( θ ^ 1 − θ ^ 2 ) ≤ ( θ ^ 1 − θ ^ 2 ) ⊤ ∇Δ g ^ θ ^ 1 ( π , π ~ ) .
By strong concavity ( γ ˉ / 2 ) (\bar{\gamma}/2) ( γ ˉ /2 ) on the left and Cauchy–Schwarz on the right:
γ ˉ 2 ∥ θ ^ 1 − θ ^ 2 ∥ 2 ≤ ∥ θ ^ 1 − θ ^ 2 ∥ ⋅ ∥ ∇ Δ g ^ θ ^ 1 ( π , π ~ ) ∥ ≤ ∥ θ ^ 1 − θ ^ 2 ∥ ⋅ 8 M 2 κ 2 λ 0 2 exp ( 4 M κ λ 0 ) ε . \frac{\bar{\gamma}}{2}\|\hat{\theta}_1-\hat{\theta}_2\|^2\le\|\hat{\theta}_1-\hat{\theta}_2\|\cdot\|\nabla\Delta\hat{g}_{\hat{\theta}_1}(\pi,\tilde{\pi})\|\le\|\hat{\theta}_1-\hat{\theta}_2\|\cdot\frac{8M^2}{\kappa^2\lambda_0^2}\exp\left(\frac{4M}{\kappa\lambda_0}\right)\varepsilon. 2 γ ˉ ∥ θ ^ 1 − θ ^ 2 ∥ 2 ≤ ∥ θ ^ 1 − θ ^ 2 ∥ ⋅ ∥∇Δ g ^ θ ^ 1 ( π , π ~ ) ∥ ≤ ∥ θ ^ 1 − θ ^ 2 ∥ ⋅ κ 2 λ 0 2 8 M 2 exp ( κ λ 0 4 M ) ε .
After rearranging, we get the same bound on ∥ θ ^ 1 − θ ^ 2 ∥ \|\hat{\theta}_1-\hat{\theta}_2\| ∥ θ ^ 1 − θ ^ 2 ∥ as in the interior case. This completes the proof.
Establishing Fast Rates with Peeling Argument
Precise Statement and Proof of Theorem 3
Let us define the following constants.
(a) Λ 1 ( δ ) : = max { O ~ ( M 14 e 2 ρ / κ κ 8 ρ 6 λ 0 10 σ 0 4 ⋅ ( d + log ( 1 / δ ) ) ) , O ( ( M κ σ 0 2 ) 1 / ( β + γ ) ) } \Lambda_1(\delta) := \max\left\{ \widetilde{O}\left(\frac{M^{14}e^{2\rho/\kappa}}{\kappa^8\rho^6\lambda_0^{10}\sigma_0^4}\cdot(d+\log(1/\delta))\right), O\left(\left(\frac{M}{\kappa\sigma_0^2}\right)^{1/(\beta+\gamma)}\right)\right\} Λ 1 ( δ ) := max { O ( κ 8 ρ 6 λ 0 10 σ 0 4 M 14 e 2 ρ / κ ⋅ ( d + log ( 1/ δ )) ) , O ( ( κ σ 0 2 M ) 1/ ( β + γ ) ) } .
(b) Λ 2 ( δ ) : = 16 2 + 1 α M 2 η 2 ( 8 M 2 σ 0 2 ) 2 ( 1 + α ) α ⋅ e ρ + 4 M λ 0 ( d log ( L B n ) + log ( 1 / δ ) ) \Lambda_2(\delta) := \frac{16^{2+\frac{1}{\alpha}}M^2}{\eta^2}\left(\frac{8M^2}{\sigma_0^2}\right)^{\frac{2(1+\alpha)}{\alpha}}\cdot e^{\rho+\frac{4M}{\lambda_0}}\left(d\log(LBn)+\log(1/\delta)\right) Λ 2 ( δ ) := η 2 1 6 2 + α 1 M 2 ( σ 0 2 8 M 2 ) α 2 ( 1 + α ) ⋅ e ρ + λ 0 4 M ( d log ( L B n ) + log ( 1/ δ ) ) .
(c) Λ 3 ( δ ) : = ( 4 2 + 1 α M κ η ⋅ ( 8 M 2 σ 0 2 ) α + 1 α ⋅ e ρ 2 + 2 M λ 0 ) 1 β + γ \Lambda_3(\delta) := \left(\frac{4^{2+\frac{1}{\alpha}}M}{\kappa\eta}\cdot\left(\frac{8M^2}{\sigma_0^2}\right)^{\frac{\alpha+1}{\alpha}}\cdot e^{\frac{\rho}{2}+\frac{2M}{\lambda_0}}\right)^{\frac{1}{\beta+\gamma}} Λ 3 ( δ ) := ( κ η 4 2 + α 1 M ⋅ ( σ 0 2 8 M 2 ) α α + 1 ⋅ e 2 ρ + λ 0 2 M ) β + γ 1 .
Theorem 4. Suppose Assumptions 1-4 and 6-8 hold and n ≥ max { Λ 1 ( δ ) , Λ 2 ( δ ) , Λ 3 ( δ ) } n \ge\max\{\Lambda_1(\delta),\Lambda_2(\delta),\Lambda_3(\delta)\} n ≥ max { Λ 1 ( δ ) , Λ 2 ( δ ) , Λ 3 ( δ )} . Then with probability at least 1 − δ 1-\delta 1 − δ we have,
A K L ( π b ∗ ) − A K L ( π ^ n ) ≤ 8 M κ ⋅ 1 n β + γ + C 1 ⋅ log n n + C v C 2 ( Q ( d , n ) n ) ( 1 + α ) / ( 2 + α ) + C v Q ( d , n ) n . A_{\mathrm{KL}}(\pi_b^*)-A_{\mathrm{KL}}(\widehat{\pi}_n)\le\frac{8M}{\kappa}\cdot\frac{1}{n^{\beta+\gamma}}+C_1\cdot\frac{\log n}{n}+C_vC_2\left(\frac{Q(d,n)}{n}\right)^{(1+\alpha)/(2+\alpha)}+\frac{C_vQ(d,n)}{n}. A KL ( π b ∗ ) − A KL ( π n ) ≤ κ 8 M ⋅ n β + γ 1 + C 1 ⋅ n log n + C v C 2 ( n Q ( d , n ) ) ( 1 + α ) / ( 2 + α ) + n C v Q ( d , n ) . where C 1 = C M 16 σ 0 4 λ 0 8 ρ 6 e 8 M / λ 0 C_1=\frac{CM^{16}}{\sigma_0^4\lambda_0^8\rho^6}e^{8M/\lambda_0} C 1 = σ 0 4 λ 0 8 ρ 6 C M 16 e 8 M / λ 0 , C v = C M 8 κ 5 λ 0 3 σ 0 2 e ρ / κ + 8 M / ( κ λ 0 ) C_v=\frac{CM^8}{\kappa^5\lambda_0^3\sigma_0^2}e^{\rho/\kappa+8M/(\kappa\lambda_0)} C v = κ 5 λ 0 3 σ 0 2 C M 8 e ρ / κ + 8 M / ( κ λ 0 ) , $C_2=\left(\frac{5}{2\eta}4^{(1+\alpha)/\alpha}e^{\rho/2+2M/\lambda_0}\right)^{\alpha/(2+\alpha)},
and Q ( d , n ) = C d log ( L B n M κ λ 0 ρ σ 0 ) + C log ( ( log n ) / δ ) Q(d,n)=Cd\log\left(\frac{LBnM}{\kappa\lambda_0\rho\sigma_0}\right)+C\log((\log n)/\delta) Q ( d , n ) = C d log ( κ λ 0 ρ σ 0 L B n M ) + C log (( log n ) / δ ) for a universal (parameter-independent) constant C C C .
Proof. Regret decomposition. Similar to the regret decomposition of Theorem 1 we can write
A K L ( π ∗ ) − A K L ( π ^ n ) = A f ( π ∗ ) − A ~ K L e s t ( π ∗ ) − A K L ( π ^ n ) + A ~ K L e s t ( π ^ n ) + A ~ K L e s t ( π ∗ ) − A ^ n , K L e s t ( π ∗ ) − A ~ K L e s t ( π ^ n ) + A ^ n , K L e s t ( π ^ n ) + A ^ n , K L e s t ( π ∗ ) − A ^ n , K L e s t ( π ^ n ) (31) \begin{aligned}
A_{\mathrm{KL}}(\pi^*)-A_{\mathrm{KL}}(\widehat{\pi}_n)
={}&A_f(\pi^*)-\widetilde{A}^{\mathrm{est}}_{\mathrm{KL}}(\pi^*)-A_{\mathrm{KL}}(\widehat{\pi}_n)+\widetilde{A}^{\mathrm{est}}_{\mathrm{KL}}(\widehat{\pi}_n)\\
&+\widetilde{A}^{\mathrm{est}}_{\mathrm{KL}}(\pi^*)-\widehat{A}^{\mathrm{est}}_{n,\mathrm{KL}}(\pi^*)-\widetilde{A}^{\mathrm{est}}_{\mathrm{KL}}(\widehat{\pi}_n)+\widehat{A}^{\mathrm{est}}_{n,\mathrm{KL}}(\widehat{\pi}_n)\\
&+\widehat{A}^{\mathrm{est}}_{n,\mathrm{KL}}(\pi^*)-\widehat{A}^{\mathrm{est}}_{n,\mathrm{KL}}(\widehat{\pi}_n) \tag*{(31)}
\end{aligned} A KL ( π ∗ ) − A KL ( π n ) = A f ( π ∗ ) − A KL est ( π ∗ ) − A KL ( π n ) + A KL est ( π n ) + A KL est ( π ∗ ) − A n , KL est ( π ∗ ) − A KL est ( π n ) + A n , KL est ( π n ) + A n , KL est ( π ∗ ) − A n , KL est ( π n ) (31) The last difference term is non-positive since π ^ n \widehat{\pi}_n π n maximizes A ^ n , K L e s t ( ⋅ ) \widehat{A}^{\mathrm{est}}_{n,\mathrm{KL}}(\cdot) A n , KL est ( ⋅ ) . By Lemma 4 we can bound the first term by 2 M κ n − ( β + γ ) \frac{2M}{\kappa}n^{-(\beta+\gamma)} κ 2 M n − ( β + γ ) . Let us also write B ^ K L e s t ( π ) = E [ A ^ n , K L e s t ( π ) ] − A ~ K L e s t ( π ) \widehat{B}^{\mathrm{est}}_{\mathrm{KL}}(\pi)=\mathbb{E}\left[\widehat{A}^{\mathrm{est}}_{n,\mathrm{KL}}(\pi)\right]-\widetilde{A}^{\mathrm{est}}_{\mathrm{KL}}(\pi) B KL est ( π ) = E [ A n , KL est ( π ) ] − A KL est ( π ) to denote the bias term. Then,
A K L ( π ∗ ) − A K L ( π ^ n ) ≤ 4 M κ n − ( β + γ ) + A ~ K L e s t ( π ∗ ) − E [ A ^ n , K L e s t ( π ∗ ) ] + E [ A ^ n , K L e s t ( π ∗ ) ] − A ^ n , K L e s t ( π ∗ ) − A ~ K L e s t ( π ^ n ) + E [ A ^ n , K L e s t ( π ^ n ) ] − E [ A ^ n , K L e s t ( π ^ n ) ] + A ^ n , K L e s t ( π ^ n ) = 4 M κ n − ( β + γ ) + B ^ K L e s t ( π ^ n ) − B ^ K L e s t ( π ∗ ) + E [ A ^ n , K L e s t ( π ∗ ) ] − A ^ n , K L e s t ( π ∗ ) + A ^ n , K L e s t ( π ^ n ) − E [ A ^ n , K L e s t ( π ^ n ) ] ≤ 6 M κ n − ( β + γ ) + B ^ K L e s t ( π ^ n ) − B K L ( π ∗ ) + E [ A ^ n , K L e s t ( π ∗ ) ] − A ^ n , K L e s t ( π ∗ ) + A ^ n , K L e s t ( π ^ n ) − E [ A ^ n , K L e s t ( π ^ n ) ] . \begin{aligned}
A_{\mathrm{KL}}(\pi^*)-A_{\mathrm{KL}}(\widehat{\pi}_n)
\le{}&\frac{4M}{\kappa}n^{-(\beta+\gamma)}+\widetilde{A}^{\mathrm{est}}_{\mathrm{KL}}(\pi^*)-\mathbb{E}\left[\widehat{A}^{\mathrm{est}}_{n,\mathrm{KL}}(\pi^*)\right]+\mathbb{E}\left[\widehat{A}^{\mathrm{est}}_{n,\mathrm{KL}}(\pi^*)\right]-\widehat{A}^{\mathrm{est}}_{n,\mathrm{KL}}(\pi^*)\\
&-\widetilde{A}^{\mathrm{est}}_{\mathrm{KL}}(\widehat{\pi}_n)+\mathbb{E}\left[\widehat{A}^{\mathrm{est}}_{n,\mathrm{KL}}(\widehat{\pi}_n)\right]-\mathbb{E}\left[\widehat{A}^{\mathrm{est}}_{n,\mathrm{KL}}(\widehat{\pi}_n)\right]+\widehat{A}^{\mathrm{est}}_{n,\mathrm{KL}}(\widehat{\pi}_n)\\
={}&\frac{4M}{\kappa}n^{-(\beta+\gamma)}+\widehat{B}^{\mathrm{est}}_{\mathrm{KL}}(\widehat{\pi}_n)-\widehat{B}^{\mathrm{est}}_{\mathrm{KL}}(\pi^*)+\mathbb{E}\left[\widehat{A}^{\mathrm{est}}_{n,\mathrm{KL}}(\pi^*)\right]-\widehat{A}^{\mathrm{est}}_{n,\mathrm{KL}}(\pi^*)+\widehat{A}^{\mathrm{est}}_{n,\mathrm{KL}}(\widehat{\pi}_n)-\mathbb{E}\left[\widehat{A}^{\mathrm{est}}_{n,\mathrm{KL}}(\widehat{\pi}_n)\right]\\
\le{}&\frac{6M}{\kappa}n^{-(\beta+\gamma)}+\widehat{B}^{\mathrm{est}}_{\mathrm{KL}}(\widehat{\pi}_n)-B_{\mathrm{KL}}(\pi^*)+\mathbb{E}\left[\widehat{A}^{\mathrm{est}}_{n,\mathrm{KL}}(\pi^*)\right]-\widehat{A}^{\mathrm{est}}_{n,\mathrm{KL}}(\pi^*)+\widehat{A}^{\mathrm{est}}_{n,\mathrm{KL}}(\widehat{\pi}_n)-\mathbb{E}\left[\widehat{A}^{\mathrm{est}}_{n,\mathrm{KL}}(\widehat{\pi}_n)\right].
\end{aligned} A KL ( π ∗ ) − A KL ( π n ) ≤ = ≤ κ 4 M n − ( β + γ ) + A KL est ( π ∗ ) − E [ A n , KL est ( π ∗ ) ] + E [ A n , KL est ( π ∗ ) ] − A n , KL est ( π ∗ ) − A KL est ( π n ) + E [ A n , KL est ( π n ) ] − E [ A n , KL est ( π n ) ] + A n , KL est ( π n ) κ 4 M n − ( β + γ ) + B KL est ( π n ) − B KL est ( π ∗ ) + E [ A n , KL est ( π ∗ ) ] − A n , KL est ( π ∗ ) + A n , KL est ( π n ) − E [ A n , KL est ( π n ) ] κ 6 M n − ( β + γ ) + B KL est ( π n ) − B KL ( π ∗ ) + E [ A n , KL est ( π ∗ ) ] − A n , KL est ( π ∗ ) + A n , KL est ( π n ) − E [ A n , KL est ( π n ) ] . The last inequality uses Lemma 21. Now, Lemma 13 shows B ^ K L e s t ( π ^ n ) ≤ 0 \widehat{B}^{\mathrm{est}}_{\mathrm{KL}}(\widehat{\pi}_n)\le0 B KL est ( π n ) ≤ 0 . Additionally, Lemma 12 establishes an upper bound ∣ B K L ( π ∗ ) ∣ ≤ C 1 ⋅ log n n \lvert B_{\mathrm{KL}}(\pi^*)\rvert\le C_1\cdot\frac{\log n}{n} ∣ B KL ( π ∗ )∣ ≤ C 1 ⋅ n l o g n for a suitable constant C 1 C_1 C 1 as long as assumptions 7 and 8 hold. This gives us the following bound.
A K L ( π ∗ ) − A K L ( π ^ n ) ≤ 6 M κ n − ( β + γ ) + C 1 ⋅ log n n + E [ A ^ n , K L e s t ( π ∗ ) − A ^ n , K L e s t ( π ^ n ) ] − ( A ^ n , K L e s t ( π ∗ ) − A ^ n , K L e s t ( π ^ n ) ) . A_{\mathrm{KL}}(\pi^*)-A_{\mathrm{KL}}(\widehat{\pi}_n)\le\frac{6M}{\kappa}n^{-(\beta+\gamma)}+C_1\cdot\frac{\log n}{n}+\mathbb{E}\left[\widehat{A}^{\mathrm{est}}_{n,\mathrm{KL}}(\pi^*)-\widehat{A}^{\mathrm{est}}_{n,\mathrm{KL}}(\widehat{\pi}_n)\right]-\left(\widehat{A}^{\mathrm{est}}_{n,\mathrm{KL}}(\pi^*)-\widehat{A}^{\mathrm{est}}_{n,\mathrm{KL}}(\widehat{\pi}_n)\right). A KL ( π ∗ ) − A KL ( π n ) ≤ κ 6 M n − ( β + γ ) + C 1 ⋅ n log n + E [ A n , KL est ( π ∗ ) − A n , KL est ( π n ) ] − ( A n , KL est ( π ∗ ) − A n , KL est ( π n ) ) . In order to bound the final term, let us define ϕ ( π ) = A ^ n , K L e s t ( π ∗ ) − A ^ n , K L e s t ( π ) \phi(\pi)=\widehat{A}^{\mathrm{est}}_{n,\mathrm{KL}}(\pi^*)-\widehat{A}^{\mathrm{est}}_{n,\mathrm{KL}}(\pi) ϕ ( π ) = A n , KL est ( π ∗ ) − A n , KL est ( π ) . Then we obtain the following regret decomposition.
A K L ( π ∗ ) − A K L ( π ^ n ) ≤ 6 M κ n − ( β + γ ) + C 1 ⋅ log n n + E [ ϕ ( π ^ n ) ] − ϕ ( π ^ n ) (32) A_{\mathrm{KL}}(\pi^*)-A_{\mathrm{KL}}(\widehat{\pi}_n)\le\frac{6M}{\kappa}n^{-(\beta+\gamma)}+C_1\cdot\frac{\log n}{n}+\mathbb{E}[\phi(\widehat{\pi}_n)]-\phi(\widehat{\pi}_n) \tag*{(32)} A KL ( π ∗ ) − A KL ( π n ) ≤ κ 6 M n − ( β + γ ) + C 1 ⋅ n log n + E [ ϕ ( π n )] − ϕ ( π n ) (32)
Let us define D ^ 1 ( π ) = 1 ∣ D 1 ∣ ∑ i ∈ D 1 ∣ π ( X i ) − π b ∗ ( X i ) ∣ \widehat{D}_1(\pi) = \frac{1}{|D_1|}\sum_{i\in D_1}|\pi(X_i)-\pi^*_b(X_i)| D 1 ( π ) = ∣ D 1 ∣ 1 ∑ i ∈ D 1 ∣ π ( X i ) − π b ∗ ( X i ) ∣ to be the empirical soft disagreement between policies π \pi π and π b ∗ \pi^*_b π b ∗ measured on the dataset D 1 D_1 D 1 . Additionally, E 1 \mathcal{E}_1 E 1 to be the following event.
E 1 = { sup π ∈ Π ∣ D ^ 1 ( π ) − D ( π ) ∣ ≤ ϵ ‾ 1 } where ϵ ‾ 1 = O ( d log ( L B n ) + log ( 1 / δ ) n ) . \mathcal{E}_1=\left\{\sup_{\pi\in\Pi}\left|\widehat{D}_1(\pi)-D(\pi)\right|\leq\overline{\epsilon}_1\right\}\qquad\text{where}\qquad\overline{\epsilon}_1=O\left(\sqrt{\frac{d\log(LBn)+\log(1/\delta)}{n}}\right). E 1 = { π ∈ Π sup D 1 ( π ) − D ( π ) ≤ ϵ 1 } where ϵ 1 = O ( n d log ( L B n ) + log ( 1/ δ ) ) .
When assumption 4 holds, Lemma 18 shows that E 1 \mathcal{E}_1 E 1 holds with probability at least 1 − δ 1-\delta 1 − δ . We will now condition on the event that E 1 \mathcal{E}_1 E 1 holds.
In order to proceed with the proof, fix Δ ∈ ( 0 , 1 ) \Delta\in(0,1) Δ ∈ ( 0 , 1 ) (to be chosen later) and define the shells of policies
Π j = { π ∈ Π : 2 j − 1 Δ < D ( π ) ≤ 2 j Δ } , s j = 2 j Δ for j = 1 , 2 , … , J : = ⌈ log 2 ( 1 / Δ ) ⌉ . \Pi_j=\left\{\pi\in\Pi:2^{j-1}\Delta<D(\pi)\leq2^j\Delta\right\},\qquad s_j=2^j\Delta\text{ for }j=1,2,\ldots,J:=\left\lceil\log_2(1/\Delta)\right\rceil. Π j = { π ∈ Π : 2 j − 1 Δ < D ( π ) ≤ 2 j Δ } , s j = 2 j Δ for j = 1 , 2 , … , J := ⌈ log 2 ( 1/Δ ) ⌉ .
Deviation in Shell j j j : We first prove an upper bound on the deviation ∣ ϕ ( π ) − E [ ϕ ( π ) ] ∣ |\phi(\pi)-\mathbb{E}[\phi(\pi)]| ∣ ϕ ( π ) − E [ ϕ ( π )] ∣ for any policy π \pi π in the shell Π j \Pi_j Π j whose policies are sufficiently close to π b ∗ \pi_b^* π b ∗ (i.e., D ( π ) ≤ s max = σ 0 2 / ( 8 M 2 ) D(\pi)\leq s_{\max}=\sigma_0^2/(8M^2) D ( π ) ≤ s m a x = σ 0 2 / ( 8 M 2 ) for any π ∈ Π j \pi\in\Pi_j π ∈ Π j ).
Consider two policies π , π ~ ∈ Π j \pi,\widetilde{\pi}\in\Pi_j π , π ∈ Π j with D ( π ) , D ( π ~ ) ≤ s max D(\pi),D(\widetilde{\pi})\leq s_{\max} D ( π ) , D ( π ) ≤ s m a x . Then ϕ ( π ) − ϕ ( π ~ ) = A ^ n , K L e s t ( π ~ ) − A ^ n , K L e s t ( π ) \phi(\pi)-\phi(\widetilde{\pi})=\widehat{A}^{\mathrm{est}}_{n,\mathrm{KL}}(\widetilde{\pi})-\widehat{A}^{\mathrm{est}}_{n,\mathrm{KL}}(\pi) ϕ ( π ) − ϕ ( π ) = A n , KL est ( π ) − A n , KL est ( π ) . By the definition of A ^ n , K L e s t \widehat{A}^{\mathrm{est}}_{n,\mathrm{KL}} A n , KL est , this can be written as an average of n n n i.i.d. terms (conditional on D 1 D_1 D 1 ):
ϕ ( π ) − ϕ ( π ~ ) = 1 n ∑ i ∈ D 4 ξ i ( π , π ~ ) \phi(\pi)-\phi(\widetilde{\pi})=\frac{1}{n}\sum_{i\in D_4}\xi_i(\pi,\widetilde{\pi}) ϕ ( π ) − ϕ ( π ) = n 1 i ∈ D 4 ∑ ξ i ( π , π )
where
ξ i ( π , π ~ ) = g λ ^ ( π ~ ) , η ^ ( π ~ ) ( Z i ( π ~ ) ) − g λ ^ ( π ) , η ^ ( π ) ( Z i ( π ) ) \xi_i(\pi,\widetilde{\pi})=g_{\widehat{\lambda}(\widetilde{\pi}),\widehat{\eta}(\widetilde{\pi})}(Z_i(\widetilde{\pi}))-g_{\widehat{\lambda}(\pi),\widehat{\eta}(\pi)}(Z_i(\pi)) ξ i ( π , π ) = g λ ( π ) , η ( π ) ( Z i ( π )) − g λ ( π ) , η ( π ) ( Z i ( π ))
where Z i ( π ) = ( 2 π ( X i ) − 1 ) Γ ^ i Z_i(\pi)=(2\pi(X_i)-1)\widehat{\Gamma}_i Z i ( π ) = ( 2 π ( X i ) − 1 ) Γ i . We can decompose ξ i ( π , π ~ ) \xi_i(\pi,\widetilde{\pi}) ξ i ( π , π ) as follows.
ξ i ( π , π ~ ) = g λ ^ ( π ~ ) , η ^ ( π ~ ) ( Z i ( π ~ ) ) − g λ ^ ( π ~ ) , η ^ ( π ~ ) ( Z i ( π ) ) g λ ^ ( π ~ ) , η ^ ( π ~ ) ( Z i ( π ~ ) ) − g λ ^ ( π ~ ) , η ^ ( π ~ ) ( Z i ( π ) ) ⏟ : = T 1 + g λ ^ ( π ~ ) , η ^ ( π ~ ) ( Z i ( π ) ) − g λ ^ ( π ) , η ^ ( π ) ( Z i ( π ) ) g λ ^ ( π ~ ) , η ^ ( π ~ ) ( Z i ( π ) ) − g λ ^ ( π ) , η ^ ( π ) ( Z i ( π ) ) ⏟ : = T 2 . \begin{aligned}
\xi_i(\pi,\widetilde{\pi})
={}&g_{\widehat{\lambda}(\widetilde{\pi}),\widehat{\eta}(\widetilde{\pi})}(Z_i(\widetilde{\pi}))
-g_{\widehat{\lambda}(\widetilde{\pi}),\widehat{\eta}(\widetilde{\pi})}(Z_i(\pi))\\
&\underbrace{\vphantom{g_{\widehat{\lambda}(\widetilde{\pi}),\widehat{\eta}(\widetilde{\pi})}(Z_i(\widetilde{\pi}))-g_{\widehat{\lambda}(\widetilde{\pi}),\widehat{\eta}(\widetilde{\pi})}(Z_i(\pi))}}_{\mathrel{:=}T_1}
+g_{\widehat{\lambda}(\widetilde{\pi}),\widehat{\eta}(\widetilde{\pi})}(Z_i(\pi))
-g_{\widehat{\lambda}(\pi),\widehat{\eta}(\pi)}(Z_i(\pi))\\
&\underbrace{\vphantom{g_{\widehat{\lambda}(\widetilde{\pi}),\widehat{\eta}(\widetilde{\pi})}(Z_i(\pi))-g_{\widehat{\lambda}(\pi),\widehat{\eta}(\pi)}(Z_i(\pi))}}_{\mathrel{:=}T_2}.
\end{aligned} ξ i ( π , π ) = g λ ( π ) , η ( π ) ( Z i ( π )) − g λ ( π ) , η ( π ) ( Z i ( π )) := T 1 g λ ( π ) , η ( π ) ( Z i ( π )) − g λ ( π ) , η ( π ) ( Z i ( π )) + g λ ( π ) , η ( π ) ( Z i ( π )) − g λ ( π ) , η ( π ) ( Z i ( π )) := T 2 g λ ( π ) , η ( π ) ( Z i ( π )) − g λ ( π ) , η ( π ) ( Z i ( π )) .
The proof of Lemma 17 shows that
∣ ∂ g λ , η ( z ) ∂ z ∣ ≤ exp ( 4 M κ λ 0 ) . \left|\frac{\partial g_{\lambda,\eta}(z)}{\partial z}\right|\leq\exp\left(\frac{4M}{\kappa\lambda_0}\right). ∂ z ∂ g λ , η ( z ) ≤ exp ( κ λ 0 4 M ) .
Therefore,
T 1 ≤ exp ( 4 M κ λ 0 ) ∣ Z i ( π ) − Z i ( π ~ ) ∣ ≤ 2 M κ exp ( 4 M κ λ 0 ) ∣ π ( X i ) − π ~ ( X i ) ∣ . T_1\leq\exp\left(\frac{4M}{\kappa\lambda_0}\right)|Z_i(\pi)-Z_i(\widetilde{\pi})|\leq\frac{2M}{\kappa}\exp\left(\frac{4M}{\kappa\lambda_0}\right)|\pi(X_i)-\widetilde{\pi}(X_i)|. T 1 ≤ exp ( κ λ 0 4 M ) ∣ Z i ( π ) − Z i ( π ) ∣ ≤ κ 2 M exp ( κ λ 0 4 M ) ∣ π ( X i ) − π ( X i ) ∣.
The proof of Lemma 17 also shows ∥ ∇ λ , η g λ , η ( z ) ∥ ≤ 4 M κ λ 0 exp ( 4 M κ λ 0 ) \|\nabla_{\lambda,\eta}g_{\lambda,\eta}(z)\|\leq\frac{4M}{\kappa\lambda_0}\exp\left(\frac{4M}{\kappa\lambda_0}\right) ∥ ∇ λ , η g λ , η ( z ) ∥ ≤ κ λ 0 4 M exp ( κ λ 0 4 M ) for all λ ≥ λ 0 / 2 \lambda\geq\lambda_0/2 λ ≥ λ 0 /2 , ∣ η ∣ ≤ M |\eta|\leq M ∣ η ∣ ≤ M and ∣ z ∣ ≤ M / κ |z|\leq M/\kappa ∣ z ∣ ≤ M / κ . Therefore,
T 2 ≤ 4 M κ λ 0 exp ( 4 M κ λ 0 ) ∥ θ ^ ( π ) − θ ^ ( π ~ ) ∥ 2 ≤ 4096 M 8 κ 5 λ 0 5 ρ 0 3 σ 0 2 exp ( 1 + ρ κ + 8 M κ λ 0 ) 1 ∣ D 1 ∣ ∑ j ∈ D 1 ∣ π ( X j ) − π ~ ( X j ) ∣ \begin{aligned}
T_2\leq\frac{4M}{\kappa\lambda_0}\exp\left(\frac{4M}{\kappa\lambda_0}\right)\|\widehat{\theta}(\pi)-\widehat{\theta}(\widetilde{\pi})\|_2 \\
&\leq\frac{4096M^8}{\kappa^5\lambda_0^5\rho_0^3\sigma_0^2}\exp\left(1+\frac{\rho}{\kappa}+\frac{8M}{\kappa\lambda_0}\right)\frac{1}{|D_1|}\sum_{j\in D_1}|\pi(X_j)-\widetilde{\pi}(X_j)|
\end{aligned} T 2 ≤ κ λ 0 4 M exp ( κ λ 0 4 M ) ∥ θ ( π ) − θ ( π ) ∥ 2 ≤ κ 5 λ 0 5 ρ 0 3 σ 0 2 4096 M 8 exp ( 1 + κ ρ + κ λ 0 8 M ) ∣ D 1 ∣ 1 j ∈ D 1 ∑ ∣ π ( X j ) − π ( X j ) ∣
by Lemma 16 as long as n ≥ Λ 1 ( δ ) : = max { O ~ ( M 14 e 2 ρ / κ κ 8 ρ 6 λ 0 4 ⋅ ( d + log ( 1 / δ ) ) ) , O ( ( M κ σ 0 2 ) 1 / ( β + γ ) ) } n\geq\Lambda_1(\delta):=\max\left\{\widetilde{O}\left(\frac{M^{14}e^{2\rho/\kappa}}{\kappa^8\rho^6\lambda_0^4}\cdot(d+\log(1/\delta))\right),O\left(\left(\frac{M}{\kappa\sigma_0^2}\right)^{1/(\beta+\gamma)}\right)\right\} n ≥ Λ 1 ( δ ) := max { O ( κ 8 ρ 6 λ 0 4 M 14 e 2 ρ / κ ⋅ ( d + log ( 1/ δ )) ) , O ( ( κ σ 0 2 M ) 1/ ( β + γ ) ) } .
Substituting the bounds on T 1 T_1 T 1 and T 2 T_2 T 2 we obtain the following bound.
∣ ϕ ( π ) − ϕ ( π ~ ) ∣ ≤ 8192 M 8 κ 5 λ 0 5 ρ 0 3 σ 0 2 exp ( 1 + ρ κ + 8 M κ λ 0 ) ⏟ : = C ℓ ∥ π − π ~ ∥ ∞ |\phi(\pi)-\phi(\widetilde{\pi})|
\leq
\underbrace{\frac{8192M^8}{\kappa^5\lambda_0^5\rho_0^3\sigma_0^2}
\exp\left(1+\frac{\rho}{\kappa}+\frac{8M}{\kappa\lambda_0}\right)}_{:=C_\ell}
\|\pi-\widetilde{\pi}\|_\infty ∣ ϕ ( π ) − ϕ ( π ) ∣ ≤ := C ℓ κ 5 λ 0 5 ρ 0 3 σ 0 2 8192 M 8 exp ( 1 + κ ρ + κ λ 0 8 M ) ∥ π − π ∥ ∞
Now, let { π 1 , … , π N j } \{\pi_1,\ldots,\pi_{N_j}\} { π 1 , … , π N j } be an ϵ j \epsilon_j ϵ j -cover of Π j \Pi_j Π j with respect to the ℓ ∞ \ell_\infty ℓ ∞ norm, and N j = N ( Π j , ϵ j , ∥ ⋅ ∥ ∞ ) N_j=N(\Pi_j,\epsilon_j,\|\cdot\|_\infty) N j = N ( Π j , ϵ j , ∥ ⋅ ∥ ∞ ) be the corresponding covering number. We will choose the parameter ϵ j \epsilon_j ϵ j later. By the union bound and Lemma 17 (applied to each π k \pi_k π k in the cover), with probability at least 1 − δ 1-\delta 1 − δ :
P ( sup k ∈ [ N j ] ∣ ϕ ( π k ) − E [ ϕ ( π k ) ] ∣ ≥ t ∣ D 1 ) ≤ 2 N j exp ( − n t 2 / 2 C v 2 ( 4 s j + ϵ ‾ 1 2 ) + C v t / 3 ) (33) \mathbb{P}\left(\left.\sup_{k\in[N_j]}|\phi(\pi_k)-\mathbb{E}[\phi(\pi_k)]|\geq t\right|D_1\right)
\leq2N_j\exp\left(-\frac{nt^2/2}{C_v^2(4s_j+\overline{\epsilon}_1^2)+C_vt/3}\right) \tag*{(33)} P ( k ∈ [ N j ] sup ∣ ϕ ( π k ) − E [ ϕ ( π k )] ∣ ≥ t D 1 ) ≤ 2 N j exp ( − C v 2 ( 4 s j + ϵ 1 2 ) + C v t /3 n t 2 /2 ) (33)
as long as n ≥ Λ ( δ / N j ) n \ge\Lambda(\delta/\mathcal{N}_j) n ≥ Λ ( δ / N j ) . We now substitute t ≤ 2 C v u j 3 n + 2 C v 4 s j + ε ˉ 1 2 n ⋅ u j t \le\frac{2C_vu_j}{3n}+2C_v\sqrt{\frac{4s_j+\bar{\varepsilon}_1^2}{n}}\cdot u_j t ≤ 3 n 2 C v u j + 2 C v n 4 s j + ε ˉ 1 2 ⋅ u j to get an upper bound of 2 N j e − u j 2\mathcal{N}_j e^{-u_j} 2 N j e − u j , and obtain the following bound.
P ( sup k ∈ [ N j ] ∣ ϕ ( π k ) − E [ ϕ ( π k ) ] ∣ ≥ 2 C v u j 3 n + 2 C v 4 s j + ε ˉ 1 2 n ⋅ u j | D 1 ) ≤ 2 N j e − u j . (34) \mathbb{P}\left(\sup_{k\in[\mathcal{N}_j]}\left|\phi(\pi_k)-\mathbb{E}[\phi(\pi_k)]\right|\ge\frac{2C_vu_j}{3n}+2C_v\sqrt{\frac{4s_j+\bar{\varepsilon}_1^2}{n}}\cdot u_j \,\middle|\, D_1\right)\le2\mathcal{N}_j e^{-u_j}. \tag*{(34)} P ( k ∈ [ N j ] sup ∣ ϕ ( π k ) − E [ ϕ ( π k )] ∣ ≥ 3 n 2 C v u j + 2 C v n 4 s j + ε ˉ 1 2 ⋅ u j D 1 ) ≤ 2 N j e − u j . (34)
In order to extend this bound to all policies in Π j \Pi_j Π j , we will use the Lipschitz property of ϕ ( ⋅ ) \phi(\cdot) ϕ ( ⋅ ) established above. For any π ∈ Π j \pi\in\Pi_j π ∈ Π j , let π k \pi_k π k be an element of the cover such that ∥ π − π k ∥ ∞ ≤ ε j \lVert\pi-\pi_k\rVert_\infty\le\varepsilon_j ∥ π − π k ∥ ∞ ≤ ε j . Then,
∣ ϕ ( π ) − E [ ϕ ( π ) ] ∣ ≤ ∣ ϕ ( π ) − ϕ ( π k ) ∣ + ∣ ϕ ( π k ) − E [ ϕ ( π k ) ] ∣ + ∣ E [ ϕ ( π k ) ] − E [ ϕ ( π ) ] ∣ |\phi(\pi)-\mathbb{E}[\phi(\pi)]|\le|\phi(\pi)-\phi(\pi_k)|+|\phi(\pi_k)-\mathbb{E}[\phi(\pi_k)]|+|\mathbb{E}[\phi(\pi_k)]-\mathbb{E}[\phi(\pi)]| ∣ ϕ ( π ) − E [ ϕ ( π )] ∣ ≤ ∣ ϕ ( π ) − ϕ ( π k ) ∣ + ∣ ϕ ( π k ) − E [ ϕ ( π k )] ∣ + ∣ E [ ϕ ( π k )] − E [ ϕ ( π )] ∣
≤ 2 C ℓ ε j + 2 C v u j 3 n + 2 C v 4 s j + ε ˉ 1 2 n ⋅ u j . \le2C\ell\varepsilon_j+\frac{2C_vu_j}{3n}+2C_v\sqrt{\frac{4s_j+\bar{\varepsilon}_1^2}{n}}\cdot u_j. ≤ 2 C ℓ ε j + 3 n 2 C v u j + 2 C v n 4 s j + ε ˉ 1 2 ⋅ u j .
Peeling Argument. Let us define ϵ ( π ) = E [ ∣ π b ∗ ( X ) − π ( X ) ∣ ∣ τ ( X ) ∣ ] \epsilon(\pi)=\mathbb{E}[|\pi_b^*(X)-\pi(X)|\mid\tau(X)|] ϵ ( π ) = E [ ∣ π b ∗ ( X ) − π ( X ) ∣ ∣ τ ( X ) ∣ ] . First, we provide a lower bound on ϵ ( π ) \epsilon(\pi) ϵ ( π ) .
D ( π ) = E [ ∣ π b ∗ ( X ) − π ( X ) ∣ ] = E [ ∣ π b ∗ ( X ) − π ( X ) ∣ ∣ ∣ τ ( X ) ∣ ≤ t ] P ( ∣ τ ( X ) ∣ ≤ t ) + E [ ∣ π b ∗ ( X ) − π ( X ) ∣ ∣ ∣ τ ( X ) ∣ > t ] P ( ∣ τ ( X ) ∣ > t ) ≤ P ( ∣ τ ( X ) ∣ ≤ t ) + 1 t E [ ∣ π b ∗ ( X ) − π ( X ) ∣ ∣ τ ( X ) ∣ ] ≤ ( t η ) α + ϵ ( π ) t . \begin{aligned}
D(\pi)&=\mathbb{E}[|\pi_b^*(X)-\pi(X)|]\\
&=\mathbb{E}[|\pi_b^*(X)-\pi(X)|\mid|\tau(X)|\le t]\mathbb{P}(|\tau(X)|\le t)\\
&\quad+\mathbb{E}[|\pi_b^*(X)-\pi(X)|\mid|\tau(X)|>t]\mathbb{P}(|\tau(X)|>t)\\
&\le\mathbb{P}(|\tau(X)|\le t)+\frac{1}{t}\mathbb{E}[|\pi_b^*(X)-\pi(X)||\tau(X)|]\\
&\le\left(\frac{t}{\eta}\right)^\alpha+\frac{\epsilon(\pi)}{t}.
\end{aligned} D ( π ) = E [ ∣ π b ∗ ( X ) − π ( X ) ∣ ] = E [ ∣ π b ∗ ( X ) − π ( X ) ∣ ∣ ∣ τ ( X ) ∣ ≤ t ] P ( ∣ τ ( X ) ∣ ≤ t ) + E [ ∣ π b ∗ ( X ) − π ( X ) ∣ ∣ ∣ τ ( X ) ∣ > t ] P ( ∣ τ ( X ) ∣ > t ) ≤ P ( ∣ τ ( X ) ∣ ≤ t ) + t 1 E [ ∣ π b ∗ ( X ) − π ( X ) ∣∣ τ ( X ) ∣ ] ≤ ( η t ) α + t ϵ ( π ) .
The upper bound is minimized at t = ( η α ϵ ( π ) ) 1 α + 1 t=(\eta^\alpha\epsilon(\pi))^{\frac{1}{\alpha+1}} t = ( η α ϵ ( π ) ) α + 1 1 . Substituting this value of t t t we obtain the following upper bound on D ( π ) D(\pi) D ( π ) .
D ( π ) ≤ 2 η − α 1 + α ⋅ ϵ ( π ) α 1 + α . D(\pi)\le2\eta^{-\frac{\alpha}{1+\alpha}}\cdot\epsilon(\pi)^{\frac{\alpha}{1+\alpha}}. D ( π ) ≤ 2 η − 1 + α α ⋅ ϵ ( π ) 1 + α α .
After rearranging the above inequality we obtain the following lower bound on ϵ ( π ) \epsilon(\pi) ϵ ( π ) .
ϵ ( π ) ≥ 1 2 α + 1 α ⋅ η ⋅ D ( π ) α + 1 α . (35) \epsilon(\pi)\ge\frac{1}{2^{\frac{\alpha+1}{\alpha}}}\cdot\eta\cdot D(\pi)^{\frac{\alpha+1}{\alpha}}. \tag*{(35)} ϵ ( π ) ≥ 2 α α + 1 1 ⋅ η ⋅ D ( π ) α α + 1 . (35)
Now consider any shell j j j with s j > s max ( : = σ 0 2 / ( 8 M 2 ) ) s_j>s_{\max}( :=\sigma_0^2/(8M^2)) s j > s m a x ( := σ 0 2 / ( 8 M 2 )) . If π ^ n ∈ Π j \hat{\pi}_n\in\Pi_j π ^ n ∈ Π j , then we must have,
ϵ ( π ^ n ) > 1 2 α + 1 α ⋅ η ⋅ ( s j 2 ) α + 1 α ≥ 1 4 α + 1 α ⋅ η ⋅ ( σ 0 2 8 M 2 ) α + 1 α . (36) \epsilon(\hat{\pi}_n)>\frac{1}{2^{\frac{\alpha+1}{\alpha}}}\cdot\eta\cdot\left(\frac{s_j}{2}\right)^{\frac{\alpha+1}{\alpha}}\ge\frac{1}{4^{\frac{\alpha+1}{\alpha}}}\cdot\eta\cdot\left(\frac{\sigma_0^2}{8M^2}\right)^{\frac{\alpha+1}{\alpha}}. \tag*{(36)} ϵ ( π ^ n ) > 2 α α + 1 1 ⋅ η ⋅ ( 2 s j ) α α + 1 ≥ 4 α α + 1 1 ⋅ η ⋅ ( 8 M 2 σ 0 2 ) α α + 1 . (36)
However, Theorem 1 proves that when Assumption 4 holds,
A K L ( π b ∗ ) − A K L ( π ^ n ) ≤ 4 M κ n − ( β + γ ) + 4 M d log ( L B n ) + log ( 1 / δ ) n . A_{\mathrm{KL}}(\pi_b^*)-A_{\mathrm{KL}}(\hat{\pi}_n)\le\frac{4M}{\kappa}n^{-(\beta+\gamma)}+4M\sqrt{\frac{d\log(LBn)+\log(1/\delta)}{n}}. A KL ( π b ∗ ) − A KL ( π ^ n ) ≤ κ 4 M n − ( β + γ ) + 4 M n d log ( L B n ) + log ( 1/ δ ) .
Furthermore, using Lemma 19 we obtain,
2 e − ρ / 2 − 2 M / λ 0 ϵ ( π ) ≤ 4 M κ n − ( β + γ ) + 4 M d log ( L B n ) + log ( 1 / δ ) n . 2e^{-\rho/2-2M/\lambda_0}\epsilon(\pi)\le\frac{4M}{\kappa}n^{-(\beta+\gamma)}+4M\sqrt{\frac{d\log(LBn)+\log(1/\delta)}{n}}. 2 e − ρ /2 − 2 M / λ 0 ϵ ( π ) ≤ κ 4 M n − ( β + γ ) + 4 M n d log ( L B n ) + log ( 1/ δ ) .
Therefore, as long as
n ≥ Λ 2 ( δ ) : = max { 16 2 + 1 α M 2 η 2 ( 8 M 2 σ 0 2 ) 2 ( 1 + α ) α ⋅ e ρ + 4 M λ 0 ( d log ( L B n ) + log ( 1 / δ ) ) , ( 4 2 + 1 α M κ η ⋅ ( 8 M 2 σ 0 2 ) α + 1 α ⋅ e 2 M λ 0 ) 1 β + γ } n\ge\Lambda_2(\delta):=\max\left\{\frac{16^{2+\frac{1}{\alpha}}M^2}{\eta^2}\left(\frac{8M^2}{\sigma_0^2}\right)^{\frac{2(1+\alpha)}{\alpha}}\cdot e^{\rho+\frac{4M}{\lambda_0}\left(d\log(LBn)+\log(1/\delta)\right)},\left(\frac{4^{2+\frac{1}{\alpha}}M}{\kappa\eta}\cdot\left(\frac{8M^2}{\sigma_0^2}\right)^{\frac{\alpha+1}{\alpha}}\cdot e^{\frac{2M}{\lambda_0}}\right)^{\frac{1}{\beta+\gamma}}\right\} n ≥ Λ 2 ( δ ) := max ⎩ ⎨ ⎧ η 2 1 6 2 + α 1 M 2 ( σ 0 2 8 M 2 ) α 2 ( 1 + α ) ⋅ e ρ + λ 0 4 M ( d l o g ( L B n ) + l o g ( 1/ δ ) ) , ( κ η 4 2 + α 1 M ⋅ ( σ 0 2 8 M 2 ) α α + 1 ⋅ e λ 0 2 M ) β + γ 1 ⎭ ⎬ ⎫
we have ϵ ( π ^ n ) < 1 4 α + 1 α ⋅ η ⋅ ( σ 0 2 8 M 2 ) α + 1 α \epsilon(\hat{\pi}_n)<\frac{1}{4^{\frac{\alpha+1}{\alpha}}}\cdot\eta\cdot\left(\frac{\sigma_0^2}{8M^2}\right)^{\frac{\alpha+1}{\alpha}} ϵ ( π ^ n ) < 4 α α + 1 1 ⋅ η ⋅ ( 8 M 2 σ 0 2 ) α α + 1 , which contradicts (36) . Therefore, π ^ n \hat{\pi}_n π ^ n cannot lie in any shell j j j with s j > s max s_j>s_{\max} s j > s m a x .
We now choose Δ = d log n n \Delta=\frac{d\log n}{n} Δ = n d l o g n and ε j = C v C ℓ 4 s j + ε ˉ 1 2 n ⋅ u j \varepsilon_j=\frac{C_v}{C_\ell}\sqrt{\frac{4s_j+\bar{\varepsilon}_1^2}{n}}\cdot u_j ε j = C ℓ C v n 4 s j + ε ˉ 1 2 ⋅ u j and u j = log log n + log ( 1 / δ ) + d log ( L B n C ℓ / C v ) u_j=\log\log n+\log(1/\delta)+d\log(LBnC_\ell/C_v) u j = log log n + log ( 1/ δ ) + d log ( L B n C ℓ / C v ) . Then it can be verified that the probability of failure within each shell j j j is at most δ / log n \delta/\log n δ / log n . By the union bound, with probability at least 1 − δ 1-\delta 1 − δ , for all shells j j j with s j ≤ s max s_j \le s_{\max} s j ≤ s m a x , we have
sup π ∈ Π j ∣ ϕ ( π ) − E [ ϕ ( π ) ] ∣ ≤ 2 C ℓ ε j + 2 C v u j 3 n + 2 C v 4 s j + ε ˉ 1 2 n ⋅ u j = 4 C v 4 s j + ε ˉ 1 2 n ⋅ ( d log ( L B n C ℓ / C v ) + log ( ( log n ) / δ ) ) + 2 C v ( d log ( L B n C ℓ / C v ) + log ( ( log n ) / δ ) ) 3 n ⏟ : = t j \begin{aligned}
\sup_{\pi\in\Pi_j} \left|\phi(\pi)-\mathbb{E}[\phi(\pi)]\right| \le2C_{\ell}\varepsilon_j+\frac{2C_vu_j}{3n}+2C_v\sqrt{\frac{4s_j+\bar{\varepsilon}_1^2}{n}\cdot u_j} \\
&=4C_v\sqrt{\frac{4s_j+\bar{\varepsilon}_1^2}{n}}\cdot\left(d\log(LBnC_{\ell}/C_v)+\log((\log n)/\delta)\right)+\frac{2C_v\left(d\log(LBnC_{\ell}/C_v)+\log((\log n)/\delta)\right)}{3n}
\qquad\underbrace{\hspace{4cm}}_{:=t_j}
\end{aligned} π ∈ Π j sup ∣ ϕ ( π ) − E [ ϕ ( π )] ∣ ≤ 2 C ℓ ε j + 3 n 2 C v u j + 2 C v n 4 s j + ε ˉ 1 2 ⋅ u j = 4 C v n 4 s j + ε ˉ 1 2 ⋅ ( d log ( L B n C ℓ / C v ) + log (( log n ) / δ ) ) + 3 n 2 C v ( d log ( L B n C ℓ / C v ) + log (( log n ) / δ ) ) := t j
Suppose π ^ n \hat{\pi}_n π ^ n lies in the shell Π j ⋆ \Pi_{j_\star} Π j ⋆ . Then substituting this bound into the regret decomposition from (32) we obtain the following bound on the regret.
A K L ( π b ⋆ ) − A K L ( π ^ n ) ≤ 6 M κ n − ( β + γ ) + C 1 ⋅ log n n + t j ⋆ . (37) A_{\mathrm{KL}}(\pi_b^\star)-A_{\mathrm{KL}}(\hat{\pi}_n)\leq\frac{6M}{\kappa}n^{-(\beta+\gamma)}+C_1\cdot\frac{\log n}{n}+t_{j_\star}. \tag*{(37)} A KL ( π b ⋆ ) − A KL ( π ^ n ) ≤ κ 6 M n − ( β + γ ) + C 1 ⋅ n log n + t j ⋆ . (37)
Now there are two cases to consider. If s j ⋆ ≤ ε ˉ 1 2 s_{j_\star}\leq\bar{\varepsilon}_1^2 s j ⋆ ≤ ε ˉ 1 2 , then t j ⋆ ≤ 9 C v ( d log ( L B n C ℓ / C v ) + log ( ( log n ) / δ ) ) n t_{j_\star}\leq\frac{9C_v\left(d\log(LBnC_{\ell}/C_v)+\log((\log n)/\delta)\right)}{n} t j ⋆ ≤ n 9 C v ( d l o g ( L B n C ℓ / C v ) + l o g (( l o g n ) / δ ) ) , and we have proved the desired bound of the theorem.
Therefore, we now consider the case s j ⋆ > ε ˉ 1 2 s_{j_\star}>\bar{\varepsilon}_1^2 s j ⋆ > ε ˉ 1 2 . Let us write Q = d log ( L B n C ℓ / C v ) + log ( ( log n ) / δ ) Q=d\log(LBnC_{\ell}/C_v)+\log((\log n)/\delta) Q = d log ( L B n C ℓ / C v ) + log (( log n ) / δ ) , then Equation (37) simplies to
A K L ( π b ⋆ ) − A K L ( π ^ n ) ≤ 6 M κ n − ( β + γ ) + C 1 ⋅ log n n + 5 C v 4 s j ⋆ n ⋅ Q + 2 C v Q 3 n . (38) A_{\mathrm{KL}}(\pi_b^\star)-A_{\mathrm{KL}}(\hat{\pi}_n)\leq\frac{6M}{\kappa}n^{-(\beta+\gamma)}+C_1\cdot\frac{\log n}{n}+5C_v\sqrt{\frac{4s_{j_\star}}{n}}\cdot Q+\frac{2C_vQ}{3n}. \tag*{(38)} A KL ( π b ⋆ ) − A KL ( π ^ n ) ≤ κ 6 M n − ( β + γ ) + C 1 ⋅ n log n + 5 C v n 4 s j ⋆ ⋅ Q + 3 n 2 C v Q . (38)
Using Lemma 19 and Equation (36) we obtain the following inequality on s j ⋆ s_{j_\star} s j ⋆ .
2 exp ( − ρ / 2 − 2 M / λ 0 ) η ⋅ s j ⋆ 1 + α 4 α 4 1 + α 4 α ≤ 6 M κ n − ( β + γ ) + C 1 ⋅ log n n + 5 C v 4 s j ⋆ n ⋅ Q + 2 C v Q 3 n . (39) 2\exp(-\rho/2-2M/\lambda_0)\frac{\eta\cdot s_{j_\star}^{\frac{1+\alpha}{4\alpha}}}{4^{\frac{1+\alpha}{4\alpha}}}\leq\frac{6M}{\kappa}n^{-(\beta+\gamma)}+C_1\cdot\frac{\log n}{n}+5C_v\sqrt{\frac{4s_{j_\star}}{n}}\cdot Q+\frac{2C_vQ}{3n}. \tag*{(39)} 2 exp ( − ρ /2 − 2 M / λ 0 ) 4 4 α 1 + α η ⋅ s j ⋆ 4 α 1 + α ≤ κ 6 M n − ( β + γ ) + C 1 ⋅ n log n + 5 C v n 4 s j ⋆ ⋅ Q + 3 n 2 C v Q . (39)
From the above inequality it is easy to check that s j ⋆ s_{j_\star} s j ⋆ satisfies the following upper bound.
s j ⋆ ≤ ( 5 2 η 4 1 + α 4 α e ρ / 2 + 2 M / λ 0 ) 2 α α + 2 ( Q n ) α α + 2 s_{j_\star}\leq\left(\frac{5}{2\eta}4^{\frac{1+\alpha}{4\alpha}}e^{\rho/2+2M/\lambda_0}\right)^{\frac{2\alpha}{\alpha+2}}\left(\frac{Q}{n}\right)^{\frac{\alpha}{\alpha+2}} s j ⋆ ≤ ( 2 η 5 4 4 α 1 + α e ρ /2 + 2 M / λ 0 ) α + 2 2 α ( n Q ) α + 2 α
Substituting the upper bound on s j ⋆ s_{j_\star} s j ⋆ in Equation (38) we obtain the following bound.
A K L ( π b ⋆ ) − A K L ( π ^ n ) ≤ 6 M κ n − ( β + γ ) + C 1 ⋅ log n n + 10 C v ( 5 2 η 4 1 + α 4 α e ρ / 2 + 2 M / λ 0 ) α α + 2 ( Q n ) 1 + α α + 2 + 2 C v Q 3 n . A_{\mathrm{KL}}(\pi_b^\star)-A_{\mathrm{KL}}(\hat{\pi}_n)\leq\frac{6M}{\kappa}n^{-(\beta+\gamma)}+C_1\cdot\frac{\log n}{n}+10C_v\left(\frac{5}{2\eta}4^{\frac{1+\alpha}{4\alpha}}e^{\rho/2+2M/\lambda_0}\right)^{\frac{\alpha}{\alpha+2}}\left(\frac{Q}{n}\right)^{\frac{1+\alpha}{\alpha+2}}+\frac{2C_vQ}{3n}. A KL ( π b ⋆ ) − A KL ( π ^ n ) ≤ κ 6 M n − ( β + γ ) + C 1 ⋅ n log n + 10 C v ( 2 η 5 4 4 α 1 + α e ρ /2 + 2 M / λ 0 ) α + 2 α ( n Q ) α + 2 1 + α + 3 n 2 C v Q .
Substituting back the values of Q Q Q , C v C_v C v , C 1 C_1 C 1 we obtain the desired statement of the theorem. □ \square □
Lemma 17. Suppose n ≥ max { O ~ ( M 14 e 2 ρ / κ κ 8 λ 0 6 σ 0 2 ⋅ ( d + log ( 1 / δ ) ) ) , O ( M κ σ 0 2 ) 1 / ( β + γ ) } n\geq\max\left\{\widetilde{O}\left(\frac{M^{14}e^{2\rho/\kappa}}{\kappa^8\lambda_0^6\sigma_0^2}\cdot(d+\log(1/\delta))\right),O\left(\frac{M}{\kappa\sigma_0^2}\right)^{1/(\beta+\gamma)}\right\} n ≥ max { O ( κ 8 λ 0 6 σ 0 2 M 14 e 2 ρ / κ ⋅ ( d + log ( 1/ δ )) ) , O ( κ σ 0 2 M ) 1/ ( β + γ ) } . On the event E 1 \mathcal{E}_1 E 1 , for any π ∈ Π \pi\in\Pi π ∈ Π with D ( π ) ≤ s max = σ 0 2 / ( 8 M 2 ) D(\pi)\leq s_{\max}=\sigma_0^2/(8M^2) D ( π ) ≤ s m a x = σ 0 2 / ( 8 M 2 ) , conditional on D 1 D_1 D 1 :
P D 4 ( ∣ ϕ ( π ) − E D 4 [ ϕ ( π ) ] ∣ ≥ t ∣ D 1 ) ≤ 2 exp ( − n t 2 / 2 C v 2 ( 4 D ( π ) + ε ˉ 1 2 ) + C v t / 3 ) , P_{D_4}\left(\left|\phi(\pi)-\mathbb{E}_{D_4}[\phi(\pi)]\right|\geq t\mid D_1\right)\leq2\exp\left(-\frac{nt^2/2}{C_v^2(4D(\pi)+\bar{\varepsilon}_1^2)+C_vt/3}\right), P D 4 ( ∣ ϕ ( π ) − E D 4 [ ϕ ( π )] ∣ ≥ t ∣ D 1 ) ≤ 2 exp ( − C v 2 ( 4 D ( π ) + ε ˉ 1 2 ) + C v t /3 n t 2 /2 ) , where C v = 1024 M 8 κ 5 λ 0 5 σ 0 2 exp ( ρ κ + 8 M κ λ 0 ) C_v=\frac{1024M^8}{\kappa^5\lambda_0^5\sigma_0^2}\exp\left(\frac{\rho}{\kappa}+\frac{8M}{\kappa\lambda_0}\right) C v = κ 5 λ 0 5 σ 0 2 1024 M 8 exp ( κ ρ + κ λ 0 8 M ) .
Proof. Let us define g λ , η ( z ) = − λ e ⋅ exp ( − z − η λ ) − λ ρ − η g_{\lambda,\eta}(z)=-\frac{\lambda}{e}\cdot\exp\left(\frac{-z-\eta}{\lambda}\right)-\lambda\rho-\eta g λ , η ( z ) = − e λ ⋅ exp ( λ − z − η ) − λ ρ − η , and Z i ( π ) = ( 2 π ( X i ) − 1 ) Γ ^ i Z_i(\pi)=(2\pi(X_i)-1)\hat{\Gamma}_i Z i ( π ) = ( 2 π ( X i ) − 1 ) Γ ^ i . Then, we can write
ϕ ( π ) = 1 n ∑ i ∈ D 4 ξ i ( π ) \phi(\pi)=\frac{1}{n}\sum_{i\in D_4}\xi_i(\pi) ϕ ( π ) = n 1 i ∈ D 4 ∑ ξ i ( π )
where ξ i ( π ) = g λ ^ ( π b ⋆ ) , η ^ ( π b ⋆ ) ( Z i ( π b ⋆ ) ) − g λ ^ ( π ) , η ^ ( π ) ( Z i ( π ) ) \xi_i(\pi)=g_{\hat{\lambda}(\pi_b^\star),\hat{\eta}(\pi_b^\star)}(Z_i(\pi_b^\star))-g_{\hat{\lambda}(\pi),\hat{\eta}(\pi)}(Z_i(\pi)) ξ i ( π ) = g λ ^ ( π b ⋆ ) , η ^ ( π b ⋆ ) ( Z i ( π b ⋆ )) − g λ ^ ( π ) , η ^ ( π ) ( Z i ( π )) .
We will apply Bernstein’s inequality to bound the deviation of ϕ ( π ) \phi(\pi) ϕ ( π ) from its mean. In that regard, we first bound the variance of each summand ξ i ( π ) \xi_i(\pi) ξ i ( π ) conditional on D 1 D_1 D 1 .
ξ i ( π ) = g λ ^ ( π b ⋆ ) , η ^ ( π b ⋆ ) ( Z i ( π b ⋆ ) ) − g λ ^ ( π b ⋆ ) , η ^ ( π b ⋆ ) ( Z i ( π ) ) ⏟ : = T 1 + g λ ^ ( π b ⋆ ) , η ^ ( π b ⋆ ) ( Z i ( π ) ) − g λ ^ ( π ) , η ^ ( π ) ( Z i ( π ) ) ⏟ : = T 2 \begin{aligned}
\xi_i(\pi)={}&g_{\hat{\lambda}(\pi_b^\star),\hat{\eta}(\pi_b^\star)}(Z_i(\pi_b^\star))-g_{\hat{\lambda}(\pi_b^\star),\hat{\eta}(\pi_b^\star)}(Z_i(\pi))\\
&\underbrace{\hspace{4cm}}_{:=T_1}
+g_{\hat{\lambda}(\pi_b^\star),\hat{\eta}(\pi_b^\star)}(Z_i(\pi))-g_{\hat{\lambda}(\pi),\hat{\eta}(\pi)}(Z_i(\pi))\\
&\underbrace{\hspace{4cm}}_{:=T_2}
\end{aligned} ξ i ( π ) = g λ ^ ( π b ⋆ ) , η ^ ( π b ⋆ ) ( Z i ( π b ⋆ )) − g λ ^ ( π b ⋆ ) , η ^ ( π b ⋆ ) ( Z i ( π )) := T 1 + g λ ^ ( π b ⋆ ) , η ^ ( π b ⋆ ) ( Z i ( π )) − g λ ^ ( π ) , η ^ ( π ) ( Z i ( π )) := T 2
Since g λ , η ( z ) = − z λ exp ( − z − η λ ) − λ ρ − η g_{\lambda,\eta}(z)=-\frac{z}{\lambda}\exp\left(\frac{-z-\eta}{\lambda}\right)-\lambda\rho-\eta g λ , η ( z ) = − λ z exp ( λ − z − η ) − λ ρ − η , it’s derivative with respect to z z z is ∂ g λ , η ( z ) ∂ z = e − 1 exp ( − z − η λ ) \frac{\partial g_{\lambda,\eta}(z)}{\partial z}=e^{-1}\exp\left(\frac{-z-\eta}{\lambda}\right) ∂ z ∂ g λ , η ( z ) = e − 1 exp ( λ − z − η ) . Therefore,
T 1 ≤ exp ( 4 M κ λ 0 ) ⋅ ∣ Z i ( π ) − Z i ( π b ⋆ ) ∣ ≤ 2 exp ( 4 M κ λ 0 ) ⋅ ∣ π ( X i ) − π b ⋆ ( X i ) ∣ ∣ ε ^ i ∣ ≤ 4 M κ exp ( 4 M κ λ 0 ) ⋅ ∣ π ( X i ) − π b ⋆ ( X i ) ∣ . T_1\leq\exp\left(\frac{4M}{\kappa\lambda_0}\right)\cdot|Z_i(\pi)-Z_i(\pi_b^\star)|\leq2\exp\left(\frac{4M}{\kappa\lambda_0}\right)\cdot|\pi(X_i)-\pi_b^\star(X_i)|\,|\hat{\varepsilon}_i|\leq\frac{4M}{\kappa}\exp\left(\frac{4M}{\kappa\lambda_0}\right)\cdot|\pi(X_i)-\pi_b^\star(X_i)|. T 1 ≤ exp ( κ λ 0 4 M ) ⋅ ∣ Z i ( π ) − Z i ( π b ⋆ ) ∣ ≤ 2 exp ( κ λ 0 4 M ) ⋅ ∣ π ( X i ) − π b ⋆ ( X i ) ∣ ∣ ε ^ i ∣ ≤ κ 4 M exp ( κ λ 0 4 M ) ⋅ ∣ π ( X i ) − π b ⋆ ( X i ) ∣.
Taking expectation with respect to X i ∈ D 4 X_i\in D_4 X i ∈ D 4 and using the fact that ∣ π ( X i ) − π b ⋆ ( X i ) ∣ ∈ [ 0 , 1 ] |\pi(X_i)-\pi_b^\star(X_i)|\in[0,1] ∣ π ( X i ) − π b ⋆ ( X i ) ∣ ∈ [ 0 , 1 ] , we obtain the following bound.
E [ T 1 2 ∣ D 1 ] ≤ 16 M 2 κ 2 exp ( 8 M κ λ 0 ) ⋅ D ( π ) . \mathbb{E}[T_1^2\mid D_1]\leq\frac{16M^2}{\kappa^2}\exp\left(\frac{8M}{\kappa\lambda_0}\right)\cdot D(\pi). E [ T 1 2 ∣ D 1 ] ≤ κ 2 16 M 2 exp ( κ λ 0 8 M ) ⋅ D ( π ) .
In order to bound T 2 T_2 T 2 , we first bound the partial derivatives of g λ , η ( z ) g_{\lambda,\eta}(z) g λ , η ( z ) with respect to λ \lambda λ and η \eta η . By direct calculation:
∂ g λ , η ( z ) ∂ λ = − 1 e exp ( − z − η λ ) ( 1 + z + η λ ) , ∂ g λ , η ( z ) ∂ η = 1 e exp ( − z − η λ ) . \frac{\partial g_{\lambda,\eta}(z)}{\partial\lambda}=-\frac{1}{e}\exp\left(\frac{-z-\eta}{\lambda}\right)\left(1+\frac{z+\eta}{\lambda}\right),\qquad\frac{\partial g_{\lambda,\eta}(z)}{\partial\eta}=\frac{1}{e}\exp\left(\frac{-z-\eta}{\lambda}\right). ∂ λ ∂ g λ , η ( z ) = − e 1 exp ( λ − z − η ) ( 1 + λ z + η ) , ∂ η ∂ g λ , η ( z ) = e 1 exp ( λ − z − η ) .
which gives us ∥ ∇ ( λ , η ) g λ , η ( z ) ∥ ≤ 4 M κ λ 0 exp ( 4 M κ λ 0 ) \left\|\nabla_{(\lambda,\eta)}g_{\lambda,\eta}(z)\right\|\leq\frac{4M}{\kappa\lambda_0}\exp\left(\frac{4M}{\kappa\lambda_0}\right) ∇ ( λ , η ) g λ , η ( z ) ≤ κ λ 0 4 M exp ( κ λ 0 4 M ) . Therefore,
T 2 ≤ 4 M κ λ 0 exp ( 4 M κ λ 0 ) ∥ θ ^ ( π b ⋆ ) − θ ^ ( π ) ∥ 2 ≤ 4096 M 8 κ 5 λ 0 5 σ 0 3 σ 0 2 exp ( ρ κ + 8 M κ λ 0 ) D ^ 1 ( π ) \begin{aligned}
T_2\leq\frac{4M}{\kappa\lambda_0}\exp\left(\frac{4M}{\kappa\lambda_0}\right)\left\|\hat{\theta}(\pi_b^\star)-\hat{\theta}(\pi)\right\|_2 \\
&\leq\frac{4096M^8}{\kappa^5\lambda_0^5\sigma_0^3\sigma_0^2}\exp\left(\frac{\rho}{\kappa}+\frac{8M}{\kappa\lambda_0}\right)\hat{D}_1(\pi)
\end{aligned} T 2 ≤ κ λ 0 4 M exp ( κ λ 0 4 M ) θ ^ ( π b ⋆ ) − θ ^ ( π ) 2 ≤ κ 5 λ 0 5 σ 0 3 σ 0 2 4096 M 8 exp ( κ ρ + κ λ 0 8 M ) D ^ 1 ( π )
by Lemma 16 as long as n ≥ max { O ~ ( M 14 e 2 ρ / κ κ 8 σ 0 6 λ 0 4 σ 0 4 ⋅ ( d + log ( 1 / δ ) ) ) , O ( M κ σ 0 2 ) 1 / ( β + γ ) } n\geq\max\left\{\widetilde{O}\left(\frac{M^{14}e^{2\rho/\kappa}}{\kappa^8\sigma_0^6\lambda_0^4\sigma_0^4}\cdot(d+\log(1/\delta))\right),O\left(\frac{M}{\kappa\sigma_0^2}\right)^{1/(\beta+\gamma)}\right\} n ≥ max { O ( κ 8 σ 0 6 λ 0 4 σ 0 4 M 14 e 2 ρ / κ ⋅ ( d + log ( 1/ δ )) ) , O ( κ σ 0 2 M ) 1/ ( β + γ ) } . Therefore,
E [ T 2 2 ∣ D 1 ] ≤ 2 24 M 16 κ 10 λ 0 10 ρ 6 σ 0 4 exp ( 2 ρ κ + 16 M κ λ 0 ) D ^ 1 ( π ) 2 \mathbb{E}[T_2^2\mid D_1]\leq\frac{2^{24}M^{16}}{\kappa^{10}\lambda_0^{10}\rho^6\sigma_0^4}\exp\left(\frac{2\rho}{\kappa}+\frac{16M}{\kappa\lambda_0}\right)\hat{D}_1(\pi)^2 E [ T 2 2 ∣ D 1 ] ≤ κ 10 λ 0 10 ρ 6 σ 0 4 2 24 M 16 exp ( κ 2 ρ + κ λ 0 16 M ) D ^ 1 ( π ) 2
When event E 1 \mathcal{E}_1 E 1 holds, D ^ 1 ( π ) ≤ D ( π ) + ε 1 \hat{D}_1(\pi)\leq D(\pi)+\varepsilon_1 D ^ 1 ( π ) ≤ D ( π ) + ε 1 , which gives us D ^ 1 ( π ) 2 ≤ D ( π ) 2 + 2 D ( π ) ε 1 + ε 1 2 ≤ 3 D ( π ) + ε 1 2 \hat{D}_1(\pi)^2\leq D(\pi)^2+2D(\pi)\varepsilon_1+\varepsilon_1^2\leq3D(\pi)+\varepsilon_1^2 D ^ 1 ( π ) 2 ≤ D ( π ) 2 + 2 D ( π ) ε 1 + ε 1 2 ≤ 3 D ( π ) + ε 1 2 . Combining the bounds for T 1 T_1 T 1 and T 2 T_2 T 2 , we have
Var ( ξ i ( π ) ∣ D 1 ) ≤ 2 E [ T 1 2 + T 2 2 ∣ D 1 ] ≤ 2 25 M 16 κ 10 λ 0 10 ρ 6 σ 0 4 exp ( 2 ρ κ + 16 M κ λ 0 ) ⏟ = : C ν 2 ( 4 D ( π ) + ε 1 2 ) \operatorname{Var}(\xi_i(\pi)\mid D_1)\leq2\mathbb{E}[T_1^2+T_2^2\mid D_1]\leq\underbrace{\frac{2^{25}M^{16}}{\kappa^{10}\lambda_0^{10}\rho^6\sigma_0^4}\exp\left(\frac{2\rho}{\kappa}+\frac{16M}{\kappa\lambda_0}\right)}_{=:C_\nu^2}(4D(\pi)+\varepsilon_1^2) Var ( ξ i ( π ) ∣ D 1 ) ≤ 2 E [ T 1 2 + T 2 2 ∣ D 1 ] ≤ =: C ν 2 κ 10 λ 0 10 ρ 6 σ 0 4 2 25 M 16 exp ( κ 2 ρ + κ λ 0 16 M ) ( 4 D ( π ) + ε 1 2 )
It can be also verified that ∣ ξ i ( π ) ∣ ≤ C ν |\xi_i(\pi)|\leq C_\nu ∣ ξ i ( π ) ∣ ≤ C ν . Therefore, an application of Bernstein’s inequality to the sample average ϕ ( π ) = 1 n ∑ i ∈ D 4 ξ i ( π ) \phi(\pi)=\frac{1}{n}\sum_{i\in D_4}\xi_i(\pi) ϕ ( π ) = n 1 ∑ i ∈ D 4 ξ i ( π ) gives us the following bound.
P D 4 ( ∣ ϕ ( π ) − E D 4 [ ϕ ( π ) ] ∣ ≥ t ∣ D 1 ) ≤ 2 exp ( − n t 2 / 2 C ν 2 ( 4 D ( π ) + ε 1 2 ) + C ν t / 3 ) . \mathbb{P}_{D_4}\left(|\phi(\pi)-\mathbb{E}_{D_4}[\phi(\pi)]|\geq t\mid D_1\right)\leq2\exp\left(-\frac{nt^2/2}{C_\nu^2(4D(\pi)+\varepsilon_1^2)+C_\nu t/3}\right). P D 4 ( ∣ ϕ ( π ) − E D 4 [ ϕ ( π )] ∣ ≥ t ∣ D 1 ) ≤ 2 exp ( − C ν 2 ( 4 D ( π ) + ε 1 2 ) + C ν t /3 n t 2 /2 ) .
Lemma 18. For a policy π \pi π , let us define D ^ 1 ( π ) = 1 ∣ D 1 ∣ ∑ i ∈ D 1 ∣ π ( X i ) − π b ⋆ ( X i ) ∣ \hat{D}_1(\pi)=\frac{1}{|D_1|}\sum_{i\in D_1}|\pi(X_i)-\pi_b^\star(X_i)| D ^ 1 ( π ) = ∣ D 1 ∣ 1 ∑ i ∈ D 1 ∣ π ( X i ) − π b ⋆ ( X i ) ∣ . If Assumption 4 holds, then with probability at least 1 − δ 1-\delta 1 − δ ,
sup π ∈ Π ∣ D ^ 1 ( π ) − D ( π ) ∣ ≤ O ( d log ( L B n ) + log ( 1 / δ ) n ) . \sup_{\pi\in\Pi}|\hat{D}_1(\pi)-D(\pi)|\leq O\left(\sqrt{\frac{d\log(LBn)+\log(1/\delta)}{n}}\right). π ∈ Π sup ∣ D ^ 1 ( π ) − D ( π ) ∣ ≤ O ( n d log ( L B n ) + log ( 1/ δ ) ) .
Proof. For a fixed policy π \pi π , D ^ 1 ( π ) = 1 ∣ D 1 ∣ ∑ i ∈ D 1 ∣ π ( X i ) − π ⋆ ( X i ) ∣ \hat{D}_1(\pi)=\frac{1}{|D_1|}\sum_{i\in D_1}|\pi(X_i)-\pi^\star(X_i)| D ^ 1 ( π ) = ∣ D 1 ∣ 1 ∑ i ∈ D 1 ∣ π ( X i ) − π ⋆ ( X i ) ∣ is an average of [ 0 , 1 ] [0,1] [ 0 , 1 ] -bounded i.i.d. random variables with mean D ( π ) D(\pi) D ( π ) . By Hoeffding’s inequality:
P ( ∣ D ^ 1 ( π ) − D ( π ) ∣ ≥ t ) ≤ 2 exp ( − 2 n t 2 ) . P\left(|\hat{D}_1(\pi)-D(\pi)|\geq t\right)\leq2\exp(-2nt^2). P ( ∣ D ^ 1 ( π ) − D ( π ) ∣ ≥ t ) ≤ 2 exp ( − 2 n t 2 ) .
To extend to all π ∈ Π \pi\in\Pi π ∈ Π uniformly, we use a covering argument. For any two parameters ϕ 1 , ϕ 2 ∈ Φ \phi_1,\phi_2 \in\Phi ϕ 1 , ϕ 2 ∈ Φ , we have
∣ D ^ 1 ( π ϕ 1 ) − D ^ 1 ( π ϕ 2 ) ∣ ≤ 1 ∣ D 1 ∣ ∑ i ∈ D 1 ∣ ∣ π ϕ 1 ( X i ) − π b ∗ ( X i ) ∣ − ∣ π ϕ 2 ( X i ) − π b ∗ ( X i ) ∣ ∣ ≤ 1 ∣ D 1 ∣ ∑ i ∈ D 1 ∣ π ϕ 1 ( X i ) − π ϕ 2 ( X i ) ∣ ≤ L ∥ ϕ 1 − ϕ 2 ∥ 2 . \begin{aligned}
\left|\widehat{D}_1(\pi_{\phi_1})-\widehat{D}_1(\pi_{\phi_2})\right| \le\frac{1}{|D_1|}\sum_{i\in D_1}\left|\left|\pi_{\phi_1}(X_i)-\pi_b^*(X_i)\right|-\left|\pi_{\phi_2}(X_i)-\pi_b^*(X_i)\right|\right| \\
&\le\frac{1}{|D_1|}\sum_{i\in D_1}\left|\pi_{\phi_1}(X_i)-\pi_{\phi_2}(X_i)\right| \le L\|\phi_1-\phi_2\|_2.
\end{aligned} D 1 ( π ϕ 1 ) − D 1 ( π ϕ 2 ) ≤ ∣ D 1 ∣ 1 i ∈ D 1 ∑ ∣ ∣ π ϕ 1 ( X i ) − π b ∗ ( X i ) ∣ − ∣ π ϕ 2 ( X i ) − π b ∗ ( X i ) ∣ ∣ ≤ ∣ D 1 ∣ 1 i ∈ D 1 ∑ ∣ π ϕ 1 ( X i ) − π ϕ 2 ( X i ) ∣ ≤ L ∥ ϕ 1 − ϕ 2 ∥ 2 .
Therefore D ^ 1 ( π ϕ ) \widehat{D}_1(\pi_\phi) D 1 ( π ϕ ) is L L L -Lipschitz in ϕ \phi ϕ (with respect to ∥ ⋅ ∥ 2 \|\cdot\|_2 ∥ ⋅ ∥ 2 ). Let us pick an ϵ \epsilon ϵ -net Φ ϵ \Phi_\epsilon Φ ϵ of Φ \Phi Φ of size N ϵ ≤ ( 1 + 2 B / ϵ ) d N_\epsilon\le(1+2B/\epsilon)^d N ϵ ≤ ( 1 + 2 B / ϵ ) d . A union bound over the net at precision ϵ = t / ( 2 L ) \epsilon=t/(2L) ϵ = t / ( 2 L ) gives:
P ( sup ϕ ∈ Φ t / 2 L ∣ D ^ 1 ( π ϕ ) − D ( π ϕ ) ∣ ≥ t ) ≤ 2 ( 1 + 4 L B t ) d exp ( − 2 n t 2 ) . P\left(\sup_{\phi\in\Phi_{t/2L}}\left|\widehat{D}_1(\pi_\phi)-D(\pi_\phi)\right|\ge t\right)\le2\left(1+\frac{4LB}{t}\right)^d\exp(-2nt^2). P ( ϕ ∈ Φ t /2 L sup D 1 ( π ϕ ) − D ( π ϕ ) ≥ t ) ≤ 2 ( 1 + t 4 L B ) d exp ( − 2 n t 2 ) .
Now for an arbitrary ϕ ∈ Φ \phi\in\Phi ϕ ∈ Φ , let ϕ ′ ∈ Φ t / 2 L \phi'\in\Phi_{t/2L} ϕ ′ ∈ Φ t /2 L be the closest point in the net. Then:
∣ D ^ 1 ( π ϕ ) − D ( π ϕ ) ∣ ≤ ∣ D ^ 1 ( π ϕ ) − D ^ 1 ( π ϕ ′ ) ∣ + ∣ D ^ 1 ( π ϕ ′ ) − D ( π ϕ ′ ) ∣ + ∣ D ( π ϕ ′ ) − D ( π ϕ ) ∣ \left|\widehat{D}_1(\pi_\phi)-D(\pi_\phi)\right|\le\left|\widehat{D}_1(\pi_\phi)-\widehat{D}_1(\pi_{\phi'})\right|+\left|\widehat{D}_1(\pi_{\phi'})-D(\pi_{\phi'})\right|+\left|D(\pi_{\phi'})-D(\pi_\phi)\right| D 1 ( π ϕ ) − D ( π ϕ ) ≤ D 1 ( π ϕ ) − D 1 ( π ϕ ′ ) + D 1 ( π ϕ ′ ) − D ( π ϕ ′ ) + ∣ D ( π ϕ ′ ) − D ( π ϕ ) ∣
≤ 2 L ∥ ϕ − ϕ ′ ∥ 2 + ∣ D ^ 1 ( π ϕ ′ ) − D ( π ϕ ′ ) ∣ ≤ t . \le2L\|\phi-\phi'\|_2+\left|\widehat{D}_1(\pi_{\phi'})-D(\pi_{\phi'})\right|\le t. ≤ 2 L ∥ ϕ − ϕ ′ ∥ 2 + D 1 ( π ϕ ′ ) − D ( π ϕ ′ ) ≤ t .
Therefore, we have,
P ( sup π ∈ Π ∣ D ^ 1 ( π ) − D ( π ) ∣ ≥ t ) ≤ 2 ( 1 + 4 L B t ) d exp ( − 2 n t 2 ) . P\left(\sup_{\pi\in\Pi}\left|\widehat{D}_1(\pi)-D(\pi)\right|\ge t\right)\le2\left(1+\frac{4LB}{t}\right)^d\exp(-2nt^2). P ( π ∈ Π sup D 1 ( π ) − D ( π ) ≥ t ) ≤ 2 ( 1 + t 4 L B ) d exp ( − 2 n t 2 ) .
Setting t = O ( ( d log ( L B n ) + log ( 1 / δ ) ) / n ) t=O\left(\sqrt{(d\log(LBn)+\log(1/\delta))/n}\right) t = O ( ( d log ( L B n ) + log ( 1/ δ )) / n ) yields the claim. □ \square □
Lemma 19. Suppose assumptions 1 1 1 and 7 7 7 hold. Then, for any policy π \pi π ,
A K L ( π b ∗ ) − A K L ( π ) ≥ 2 exp ( − ρ 2 − 2 M λ 0 ) ε ( π ) . A_{\mathrm{KL}}(\pi_b^*)-A_{\mathrm{KL}}(\pi)\ge2\exp\left(-\frac{\rho}{2}-\frac{2M}{\lambda_0}\right)\varepsilon(\pi). A KL ( π b ∗ ) − A KL ( π ) ≥ 2 exp ( − 2 ρ − λ 0 2 M ) ε ( π ) .
Proof. From the dual formulation of A K L ( π ) A_{\mathrm{KL}}(\pi) A KL ( π ) , we have
A K L ( π ) = max λ ≥ 0 , η ∈ R − λ e E [ exp ( − ( 2 π ( X ) − 1 ) τ ( X ) − η λ ) ] − λ ρ − η . A_{\mathrm{KL}}(\pi)=\max_{\lambda\ge0,\eta\in\mathbb{R}}-\frac{\lambda}{e}\mathbb{E}\left[\exp\left(\frac{-(2\pi(X)-1)\tau(X)-\eta}{\lambda}\right)\right]-\lambda\rho-\eta. A KL ( π ) = λ ≥ 0 , η ∈ R max − e λ E [ exp ( λ − ( 2 π ( X ) − 1 ) τ ( X ) − η ) ] − λ ρ − η .
For a fixed λ ≥ 0 \lambda\ge0 λ ≥ 0 , the optimal choice of η \eta η is η = λ log E [ exp ( − ( 2 π ( X ) − 1 ) τ ( X ) λ ) ] − λ \eta=\lambda\log\mathbb{E}\left[\exp\left(\frac{-(2\pi(X)-1)\tau(X)}{\lambda}\right)\right]-\lambda η = λ log E [ exp ( λ − ( 2 π ( X ) − 1 ) τ ( X ) ) ] − λ . Substituting this value of η \eta η we obtain the following lower bound on A K L ( π ) A_{\mathrm{KL}}(\pi) A KL ( π ) .
A K L ( π ) = max λ ≥ 0 − λ ρ − λ log E [ exp ( − ( 2 π ( X ) − 1 ) τ ( X ) λ ) ] A_{\mathrm{KL}}(\pi)=\max_{\lambda\ge0}-\lambda\rho-\lambda\log\mathbb{E}\left[\exp\left(\frac{-(2\pi(X)-1)\tau(X)}{\lambda}\right)\right] A KL ( π ) = λ ≥ 0 max − λ ρ − λ log E [ exp ( λ − ( 2 π ( X ) − 1 ) τ ( X ) ) ]
Suppose λ ( π ) \lambda(\pi) λ ( π ) is the optimal value of λ \lambda λ for the policy π \pi π . Then,
A K L ( π b ∗ ) − A K L ( π ) ≥ − λ ( π ) log E [ exp ( − ( 2 π b ∗ ( X ) − 1 ) τ ( X ) λ ( π ) ) ] + λ ( π ) log E [ exp ( − ( 2 π ( X ) − 1 ) τ ( X ) λ ( π ) ) ] = λ ( π ) log E [ exp ( − ( 2 π ( X ) − 1 ) τ ( X ) λ ( π ) ) ] E [ exp ( − ( 2 π b ∗ ( X ) − 1 ) τ ( X ) λ ( π ) ) ] . \begin{aligned}
A_{\mathrm{KL}}(\pi_b^*)-A_{\mathrm{KL}}(\pi)\ge-\lambda(\pi)\log\mathbb{E}\left[\exp\left(\frac{-(2\pi_b^*(X)-1)\tau(X)}{\lambda(\pi)}\right)\right]+\lambda(\pi)\log\mathbb{E}\left[\exp\left(\frac{-(2\pi(X)-1)\tau(X)}{\lambda(\pi)}\right)\right] \\
&=\lambda(\pi)\log\frac{\mathbb{E}\left[\exp\left(\frac{-(2\pi(X)-1)\tau(X)}{\lambda(\pi)}\right)\right]}{\mathbb{E}\left[\exp\left(\frac{-(2\pi_b^*(X)-1)\tau(X)}{\lambda(\pi)}\right)\right]}.
\end{aligned} A KL ( π b ∗ ) − A KL ( π ) ≥ − λ ( π ) log E [ exp ( λ ( π ) − ( 2 π b ∗ ( X ) − 1 ) τ ( X ) ) ] + λ ( π ) log E [ exp ( λ ( π ) − ( 2 π ( X ) − 1 ) τ ( X ) ) ] = λ ( π ) log E [ exp ( λ ( π ) − ( 2 π b ∗ ( X ) − 1 ) τ ( X ) ) ] E [ exp ( λ ( π ) − ( 2 π ( X ) − 1 ) τ ( X ) ) ] .
Let us define the distribution Q Q Q over X X X with d Q / d P 0 ( X ) dQ/dP_0(X) d Q / d P 0 ( X ) proportional to exp ( − ( 2 π b ∗ ( X ) − 1 ) τ ( X ) λ ( π ) ) \exp\left(\frac{-(2\pi_b^*(X)-1)\tau(X)}{\lambda(\pi)}\right) exp ( λ ( π ) − ( 2 π b ∗ ( X ) − 1 ) τ ( X ) ) . Then, the above bound can be rewritten as
A K L ( π b ∗ ) − A K L ( π ) ≥ λ ( π ) log E X ∼ Q [ exp ( 2 ( π b ∗ ( X ) − π ( X ) ) τ ( X ) λ ( π ) ) ] ≥ E X ∼ Q [ 2 ( π b ∗ ( X ) − π ( X ) ) τ ( X ) ] \begin{aligned}
A_{\mathrm{KL}}(\pi_b^*)-A_{\mathrm{KL}}(\pi)\ge\lambda(\pi)\log\mathbb{E}_{X\sim Q}\left[\exp\left(\frac{2(\pi_b^*(X)-\pi(X))\tau(X)}{\lambda(\pi)}\right)\right] \\
&\ge\mathbb{E}_{X\sim Q}\left[2(\pi_b^*(X)-\pi(X))\tau(X)\right]
\end{aligned} A KL ( π b ∗ ) − A KL ( π ) ≥ λ ( π ) log E X ∼ Q [ exp ( λ ( π ) 2 ( π b ∗ ( X ) − π ( X )) τ ( X ) ) ] ≥ E X ∼ Q [ 2 ( π b ∗ ( X ) − π ( X )) τ ( X ) ]
The last inequality follows from Jensen’s inequality. We now use a few observations. First, ( π b ∗ ( X ) − π ( X ) ) τ ( X ) = ∣ π b ∗ ( X ) − π ( X ) ∣ ∣ τ ( X ) ∣ (\pi_b^*(X)-\pi(X))\tau(X)=|\pi_b^*(X)-\pi(X)||\tau(X)| ( π b ∗ ( X ) − π ( X )) τ ( X ) = ∣ π b ∗ ( X ) − π ( X ) ∣∣ τ ( X ) ∣ from the definition of π b ∗ \pi_b^* π b ∗ . Second, for ( λ , η ) ∈ [ λ 0 / 2 , 2 M / ρ ] × [ − M , M ] (\lambda,\eta)\in[\lambda_0/2,2M/\rho]\times[-M,M] ( λ , η ) ∈ [ λ 0 /2 , 2 M / ρ ] × [ − M , M ]
d Q ( X ) d P 0 ( X ) = exp ( − ∣ τ ( X ) ∣ λ ( π ) ) E [ exp ( − ∣ τ ( X ) ∣ λ ( π ) ) ] ≥ exp ( − ρ 2 − 2 M λ 0 ) . \frac{dQ(X)}{dP_0(X)}=\frac{\exp\left(-\frac{|\tau(X)|}{\lambda(\pi)}\right)}{\mathbb{E}\left[\exp\left(-\frac{|\tau(X)|}{\lambda(\pi)}\right)\right]}\ge\exp\left(-\frac{\rho}{2}-\frac{2M}{\lambda_0}\right). d P 0 ( X ) d Q ( X ) = E [ exp ( − λ ( π ) ∣ τ ( X ) ∣ ) ] exp ( − λ ( π ) ∣ τ ( X ) ∣ ) ≥ exp ( − 2 ρ − λ 0 2 M ) .
This give us,
A K L ( π b ∗ ) − A K L ( π ) ≥ 2 exp ( − ρ 2 − 2 M λ 0 ) ε ( π ) . A_{\mathrm{KL}}(\pi_b^*)-A_{\mathrm{KL}}(\pi)\ge2\exp\left(-\frac{\rho}{2}-\frac{2M}{\lambda_0}\right)\varepsilon(\pi). A KL ( π b ∗ ) − A KL ( π ) ≥ 2 exp ( − 2 ρ − λ 0 2 M ) ε ( π ) .
Additional Useful Lemmas
Lemma 20. Let h π ( X ) = ( 2 π ( X ) − 1 ) τ ( X ) h_\pi(X)=(2\pi(X)-1)\tau(X) h π ( X ) = ( 2 π ( X ) − 1 ) τ ( X ) for any policy π \pi π . Then the following identities hold.
E [ h π b ∗ ( X ) ] − E [ h π ( X ) ] = 2 ε ( π ) \mathbb{E}[h_{\pi_b^*}(X)]-\mathbb{E}[h_\pi(X)]=2\varepsilon(\pi) E [ h π b ∗ ( X )] − E [ h π ( X )] = 2 ε ( π ) .
E [ h π b ∗ ( X ) ] + E [ h π ( X ) ] = − 2 ε ( π ) + 2 E [ ∣ τ ( X ) ∣ ] \mathbb{E}[h_{\pi_b^*}(X)]+\mathbb{E}[h_\pi(X)]=-2\varepsilon(\pi)+2\mathbb{E}[|\tau(X)|] E [ h π b ∗ ( X )] + E [ h π ( X )] = − 2 ε ( π ) + 2 E [ ∣ τ ( X ) ∣ ] .
E [ h π b ∗ 2 ( X ) ] − E [ h π 2 ( X ) ] = E [ − 4 ∣ π b ∗ ( X ) − π ( X ) ∣ 2 ∣ τ ( X ) ∣ 2 + 4 ∣ π b ∗ ( X ) − π ( X ) ∣ ∣ τ ( X ) ∣ 2 ] \mathbb{E}[h_{\pi_b^*}^2(X)]-\mathbb{E}[h_\pi^2(X)]=\mathbb{E}\left[-4|\pi_b^*(X)-\pi(X)|^2|\tau(X)|^2+4|\pi_b^*(X)-\pi(X)||\tau(X)|^2\right] E [ h π b ∗ 2 ( X )] − E [ h π 2 ( X )] = E [ − 4∣ π b ∗ ( X ) − π ( X ) ∣ 2 ∣ τ ( X ) ∣ 2 + 4∣ π b ∗ ( X ) − π ( X ) ∣∣ τ ( X ) ∣ 2 ] .
Proof. Since π b ∗ \pi_b^* π b ∗ is binary ( 2 π b ∗ ( X ) − 1 ) τ ( X ) = ∣ τ ( X ) ∣ (2\pi_b^*(X)-1)\tau(X)=|\tau(X)| ( 2 π b ∗ ( X ) − 1 ) τ ( X ) = ∣ τ ( X ) ∣ . Therefore, E [ h π b ∗ ( X ) ] − E [ h π ( X ) ] = E [ ∣ τ ( X ) ∣ − ( 2 π ( X ) − 1 ) τ ( X ) ] \mathbb{E}[h_{\pi_b^*}(X)]-\mathbb{E}[h_\pi(X)]=\mathbb{E}[|\tau(X)|-(2\pi(X)-1)\tau(X)] E [ h π b ∗ ( X )] − E [ h π ( X )] = E [ ∣ τ ( X ) ∣ − ( 2 π ( X ) − 1 ) τ ( X )] . When τ ( X ) ≥ 0 \tau(X)\ge0 τ ( X ) ≥ 0 ,
∣ τ ( X ) ∣ − ( 2 π ( X ) − 1 ) τ ( X ) = 2 ( 1 − π ( X ) ) τ ( X ) = 2 ( π b ∗ ( X ) − π ( X ) ) τ ( X ) = 2 ∣ ( π b ∗ ( X ) − π ( X ) ) ∣ ∣ τ ( X ) ∣ . |\tau(X)|-(2\pi(X)-1)\tau(X)=2(1-\pi(X))\tau(X)=2(\pi_b^*(X)-\pi(X))\tau(X)=2|(\pi_b^*(X)-\pi(X))||\tau(X)|. ∣ τ ( X ) ∣ − ( 2 π ( X ) − 1 ) τ ( X ) = 2 ( 1 − π ( X )) τ ( X ) = 2 ( π b ∗ ( X ) − π ( X )) τ ( X ) = 2∣ ( π b ∗ ( X ) − π ( X )) ∣∣ τ ( X ) ∣. On the other hand, when τ ( X ) < 0 \tau(X)<0 τ ( X ) < 0 ,
∣ τ ( X ) ∣ − ( 2 π ( X ) − 1 ) τ ( X ) = − 2 π ( X ) τ ( X ) = 2 ∣ ( π b ∗ ( X ) − π ( X ) ) ∣ ∣ τ ( X ) ∣ . |\tau(X)|-(2\pi(X)-1)\tau(X)=-2\pi(X)\tau(X)=2|(\pi_b^*(X)-\pi(X))||\tau(X)|. ∣ τ ( X ) ∣ − ( 2 π ( X ) − 1 ) τ ( X ) = − 2 π ( X ) τ ( X ) = 2∣ ( π b ∗ ( X ) − π ( X )) ∣∣ τ ( X ) ∣. After taking expectation with respect to X X X we obtain the first result.
In order to prove the second result, note that E [ h π b ∗ ( X ) ] + E [ h π ( X ) ] = E [ ∣ τ ( X ) ∣ + ( 2 π ( X ) − 1 ) τ ( X ) ] \mathbb{E}[h_{\pi_b^*}(X)]+\mathbb{E}[h_\pi(X)]=\mathbb{E}[|\tau(X)|+(2\pi(X)-1)\tau(X)] E [ h π b ∗ ( X )] + E [ h π ( X )] = E [ ∣ τ ( X ) ∣ + ( 2 π ( X ) − 1 ) τ ( X )] . When τ ( X ) ≥ 0 \tau(X)\ge0 τ ( X ) ≥ 0 ,
∣ τ ( X ) ∣ + ( 2 π ( X ) − 1 ) τ ( X ) = 2 π ( X ) τ ( X ) = − 2 ( π b ∗ ( X ) − π ( X ) ) τ ( X ) + 2 τ ( X ) = − 2 ∣ ( π b ∗ ( X ) − π ( X ) ) ∣ ∣ τ ( X ) ∣ + 2 ∣ τ ( X ) ∣ . \begin{aligned}
|\tau(X)|+(2\pi(X)-1)\tau(X)&=2\pi(X)\tau(X)=-2(\pi_b^*(X)-\pi(X))\tau(X)+2\tau(X)\\
&=-2|(\pi_b^*(X)-\pi(X))||\tau(X)|+2|\tau(X)|.
\end{aligned} ∣ τ ( X ) ∣ + ( 2 π ( X ) − 1 ) τ ( X ) = 2 π ( X ) τ ( X ) = − 2 ( π b ∗ ( X ) − π ( X )) τ ( X ) + 2 τ ( X ) = − 2∣ ( π b ∗ ( X ) − π ( X )) ∣∣ τ ( X ) ∣ + 2∣ τ ( X ) ∣. On the other hand, when τ ( X ) < 0 \tau(X)<0 τ ( X ) < 0 ,
∣ τ ( X ) ∣ + ( 2 π ( X ) − 1 ) τ ( X ) = 2 π ( X ) τ ( X ) − 2 τ ( X ) = − 2 ∣ ( π b ∗ ( X ) − π ( X ) ) ∣ ∣ τ ( X ) ∣ + 2 ∣ τ ( X ) ∣ . \begin{aligned}
|\tau(X)|+(2\pi(X)-1)\tau(X)&=2\pi(X)\tau(X)-2\tau(X)\\
&=-2|(\pi_b^*(X)-\pi(X))||\tau(X)|+2|\tau(X)|.
\end{aligned} ∣ τ ( X ) ∣ + ( 2 π ( X ) − 1 ) τ ( X ) = 2 π ( X ) τ ( X ) − 2 τ ( X ) = − 2∣ ( π b ∗ ( X ) − π ( X )) ∣∣ τ ( X ) ∣ + 2∣ τ ( X ) ∣. After taking expectation with respect to X X X we obtain the second result.
In order to prove the third result, we make the following observation.
h π b ∗ 2 ( X ) − h π 2 ( X ) = ( h π b ∗ ( X ) + h π ( X ) ) ( h π b ∗ ( X ) − h π ( X ) ) = − 4 ∣ π b ∗ ( X ) − π ( X ) ∣ 2 ∣ τ ( X ) ∣ 2 + 4 ∣ π b ∗ ( X ) − π ( X ) ∣ ∣ τ ( X ) ∣ 2 . \begin{aligned}
h_{\pi_b^*}^2(X)-h_\pi^2(X)&=(h_{\pi_b^*}(X)+h_\pi(X))(h_{\pi_b^*}(X)-h_\pi(X))\\
&=-4|\pi_b^*(X)-\pi(X)|^2|\tau(X)|^2+4|\pi_b^*(X)-\pi(X)||\tau(X)|^2.
\end{aligned} h π b ∗ 2 ( X ) − h π 2 ( X ) = ( h π b ∗ ( X ) + h π ( X )) ( h π b ∗ ( X ) − h π ( X )) = − 4∣ π b ∗ ( X ) − π ( X ) ∣ 2 ∣ τ ( X ) ∣ 2 + 4∣ π b ∗ ( X ) − π ( X ) ∣∣ τ ( X ) ∣ 2 . Taking expectation with respect to X X X we obtain the final result.
Lemma 21. For any policy π \pi π , let us define B ^ f e s t ( π ) = E [ A ^ n , f e s t ( π ) ] − V ~ f e s t ( π ) \widehat{B}^{\mathrm{est}}_f(\pi)=\mathbb{E}\left[\widehat{A}^{\mathrm{est}}_{n,f}(\pi)\right]-\widetilde{V}^{\mathrm{est}}_f(\pi) B f est ( π ) = E [ A n , f est ( π ) ] − V f est ( π ) as the empirical bias, and B f ( π ) = E [ A ^ n , f ( π ) ] − A f ( π ) B_f(\pi)=\mathbb{E}\left[\widehat{A}_{n,f}(\pi)\right]-A_f(\pi) B f ( π ) = E [ A n , f ( π ) ] − A f ( π ) to be the exact bias term. If Assumption 2 holds then,
∣ B ^ f e s t ( π ) − B f ( π ) ∣ ≤ 4 M κ n − β − γ . \left|\widehat{B}^{\mathrm{est}}_f(\pi)-B_f(\pi)\right|\le\frac{4M}{\kappa}n^{-\beta-\gamma}. B f est ( π ) − B f ( π ) ≤ κ 4 M n − β − γ .
Proof.
B ^ f ( π ) − B f ( π ) = E [ A ^ n , f e s t ( π ) − V ~ f e s t ( π ) ] − E [ A ^ n , f ( π ) ] + A f ( π ) = E [ A ^ n , f e s t ( π ) − A ^ n , f ( π ) ] + A f ( π ) − V ~ f e s t ( π ) \begin{aligned}
\widehat{B}_f(\pi)-B_f(\pi)=\mathbb{E}\left[\widehat{A}^{\mathrm{est}}_{n,f}(\pi)-\widetilde{V}^{\mathrm{est}}_f(\pi)\right]-\mathbb{E}\left[\widehat{A}_{n,f}(\pi)\right]+A_f(\pi) \\
&=\mathbb{E}\left[\widehat{A}^{\mathrm{est}}_{n,f}(\pi)-\widehat{A}_{n,f}(\pi)\right]+A_f(\pi)-\widetilde{V}^{\mathrm{est}}_f(\pi)
\end{aligned} B f ( π ) − B f ( π ) = E [ A n , f est ( π ) − V f est ( π ) ] − E [ A n , f ( π ) ] + A f ( π ) = E [ A n , f est ( π ) − A n , f ( π ) ] + A f ( π ) − V f est ( π )
By Lemma 4 we have ∣ A f ( π ) − V ~ f e s t ( π ) ∣ ≤ M κ n − ( β + γ ) \left|A_f(\pi)-\widetilde{V}^{\mathrm{est}}_f(\pi)\right|\le\frac{M}{\kappa}n^{-(\beta+\gamma)} A f ( π ) − V f est ( π ) ≤ κ M n − ( β + γ ) . Moreover, by proceeding very similarly to the proof of Lemma 4 we can also establish the following bound on the first term.
∣ E [ A ^ n , f e s t ( π ) − A ^ n , f ( π ) ] ∣ ≤ M κ ⋅ E P ^ 0 n [ max Q : D f ( Q ∥ P ^ 0 n ) ≤ ρ E X ∼ Q [ ∣ ( m ( X , 1 ) − m ^ ( X , 1 ) ) ( e ^ ( X ) − e ( X ) ) ∣ ] + E X ∼ Q [ ∣ ( m ( X , 0 ) − m ^ ( X , 0 ) ) ( e ^ ( X ) − e ( X ) ) ∣ ] ] (40) \left|\mathbb{E}\left[\widehat{A}^{\mathrm{est}}_{n,f}(\pi)-\widehat{A}_{n,f}(\pi)\right]\right|
\le\frac{M}{\kappa}\cdot\mathbb{E}_{\widehat{P}^{n}_{0}}\left[\max_{Q:D_f(Q\mathbin\Vert\widehat{P}^{n}_{0})\le\rho}\mathbb{E}_{X\sim Q}\left[\left|(m(X,1)-\widehat{m}(X,1))(\widehat{e}(X)-e(X))\right|\right]+\mathbb{E}_{X\sim Q}\left[\left|(m(X,0)-\widehat{m}(X,0))(\widehat{e}(X)-e(X))\right|\right]\right] \tag*{(40)} E [ A n , f est ( π ) − A n , f ( π ) ] ≤ κ M ⋅ E P 0 n [ Q : D f ( Q ∥ P 0 n ) ≤ ρ max E X ∼ Q [ ∣ ( m ( X , 1 ) − m ( X , 1 )) ( e ( X ) − e ( X )) ∣ ] + E X ∼ Q [ ∣ ( m ( X , 0 ) − m ( X , 0 )) ( e ( X ) − e ( X )) ∣ ] ] (40)
Now consider a general distributionally robust optimization problem of the form φ ( P ) = max Q : D f ( Q ∥ P ) ≤ ρ E X ∼ Q [ g ( X ) ] \varphi(P)=\max_{Q:D_f(Q\mathbin\Vert P)\le\rho}\mathbb{E}_{X\sim Q}[g(X)] φ ( P ) = max Q : D f ( Q ∥ P ) ≤ ρ E X ∼ Q [ g ( X )] where g ( X ) g(X) g ( X ) is a non-negative random variable. Using the duality result for distributionally robust optimization we can write the following identity.
φ ( P ) = max Q : D f ( Q ∥ P ) ≤ ρ E X ∼ Q [ g ( X ) ] = min λ ≥ 0 , η ∈ R E X ∼ P [ λ f ⋆ ( g ( X ) − η λ ) ] + λ ρ + η \varphi(P)=\max_{Q:D_f(Q\mathbin\Vert P)\le\rho}\mathbb{E}_{X\sim Q}[g(X)]=\min_{\lambda\ge0,\eta\in\mathbb{R}}\mathbb{E}_{X\sim P}\left[\lambda f^\star\left(\frac{g(X)-\eta}{\lambda}\right)\right]+\lambda\rho+\eta φ ( P ) = Q : D f ( Q ∥ P ) ≤ ρ max E X ∼ Q [ g ( X )] = λ ≥ 0 , η ∈ R min E X ∼ P [ λ f ⋆ ( λ g ( X ) − η ) ] + λ ρ + η
Therefore, for any λ ≥ 0 , ν ∈ R \lambda\ge0,\nu\in\mathbb{R} λ ≥ 0 , ν ∈ R
φ ( P ^ 0 n ) ≤ E P ^ 0 n [ λ f ⋆ ( g ( X ) − η λ ) ] + λ ρ + η \varphi(\widehat{P}^{n}_{0})\le\mathbb{E}_{\widehat{P}^{n}_{0}}\left[\lambda f^\star\left(\frac{g(X)-\eta}{\lambda}\right)\right]+\lambda\rho+\eta φ ( P 0 n ) ≤ E P 0 n [ λ f ⋆ ( λ g ( X ) − η ) ] + λ ρ + η
Taking expectation with respect to the randomness in P ^ 0 n \widehat{P}^{n}_{0} P 0 n we obtain the following upper bound.
E P ^ 0 n [ φ ( P ^ 0 n ) ] ≤ E P 0 [ λ f ⋆ ( g ( X ) − η λ ) ] + λ ρ + η \mathbb{E}_{\widehat{P}^{n}_{0}}\left[\varphi(\widehat{P}^{n}_{0})\right]\le\mathbb{E}_{P_0}\left[\lambda f^\star\left(\frac{g(X)-\eta}{\lambda}\right)\right]+\lambda\rho+\eta E P 0 n [ φ ( P 0 n ) ] ≤ E P 0 [ λ f ⋆ ( λ g ( X ) − η ) ] + λ ρ + η
Now taking infimum over λ ≥ 0 , η ∈ R \lambda\ge0,\eta\in\mathbb{R} λ ≥ 0 , η ∈ R we obtain,
E P ^ 0 n [ φ ( P ^ 0 n ) ] ≤ φ ( P 0 ) . \mathbb{E}_{\widehat{P}^{n}_{0}}\left[\varphi(\widehat{P}^{n}_{0})\right]\le\varphi(P_0). E P 0 n [ φ ( P 0 n ) ] ≤ φ ( P 0 ) .
Substituting this result in eq. (40) we obtain the following upper bound.
∣ E [ A ^ n , f e s t ( π ) − A ^ n , f ( π ) ] ∣ ≤ M κ ⋅ max Q : D f ( Q ∥ P 0 ) ≤ ρ E X ∼ Q [ ∣ ( m ( X , 1 ) − m ^ ( X , 1 ) ) ( e ^ ( X ) − e ( X ) ) ∣ ] + max Q : D f ( Q ∥ P 0 ) ≤ ρ E X ∼ Q [ ∣ ( m ( X , 0 ) − m ^ ( X , 0 ) ) ( e ^ ( X ) − e ( X ) ) ∣ ] ≤ M κ ⋅ max Q : D f ( Q ∥ P 0 ) ≤ ρ E X ∼ Q [ ( e ^ ( X ) − e ( X ) ) 2 ] ( max Q : D f ( Q ∥ P 0 ) ≤ ρ E X ∼ Q [ ( m ( X , 1 ) − m ^ ( X , 1 ) ) 2 ] + max Q : D f ( Q ∥ P 0 ) ≤ ρ E X ∼ Q [ ( m ( X , 1 ) − m ^ ( X , 1 ) ) 2 ] ) ≤ 2 M κ n − β − γ \begin{aligned}
\left|\mathbb{E}\left[\widehat{A}^{\mathrm{est}}_{n,f}(\pi)-\widehat{A}_{n,f}(\pi)\right]\right|
&\le\frac{M}{\kappa}\cdot\max_{Q:D_f(Q\mathbin\Vert P_0)\le\rho}\mathbb{E}_{X\sim Q}\left[\left|(m(X,1)-\widehat{m}(X,1))(\widehat{e}(X)-e(X))\right|\right]\\
&\quad+\max_{Q:D_f(Q\mathbin\Vert P_0)\le\rho}\mathbb{E}_{X\sim Q}\left[\left|(m(X,0)-\widehat{m}(X,0))(\widehat{e}(X)-e(X))\right|\right]\\
&\le\frac{M}{\kappa}\cdot\max_{Q:D_f(Q\mathbin\Vert P_0)\le\rho}\sqrt{\mathbb{E}_{X\sim Q}\left[(\widehat{e}(X)-e(X))^2\right]}\\
&\quad\left(\max_{Q:D_f(Q\mathbin\Vert P_0)\le\rho}\sqrt{\mathbb{E}_{X\sim Q}\left[(m(X,1)-\widehat{m}(X,1))^2\right]}+\max_{Q:D_f(Q\mathbin\Vert P_0)\le\rho}\sqrt{\mathbb{E}_{X\sim Q}\left[(m(X,1)-\widehat{m}(X,1))^2\right]}\right)\\
&\le\frac{2M}{\kappa}n^{-\beta-\gamma}
\end{aligned} E [ A n , f est ( π ) − A n , f ( π ) ] ≤ κ M ⋅ Q : D f ( Q ∥ P 0 ) ≤ ρ max E X ∼ Q [ ∣ ( m ( X , 1 ) − m ( X , 1 )) ( e ( X ) − e ( X )) ∣ ] + Q : D f ( Q ∥ P 0 ) ≤ ρ max E X ∼ Q [ ∣ ( m ( X , 0 ) − m ( X , 0 )) ( e ( X ) − e ( X )) ∣ ] ≤ κ M ⋅ Q : D f ( Q ∥ P 0 ) ≤ ρ max E X ∼ Q [ ( e ( X ) − e ( X ) ) 2 ] ( Q : D f ( Q ∥ P 0 ) ≤ ρ max E X ∼ Q [ ( m ( X , 1 ) − m ( X , 1 ) ) 2 ] + Q : D f ( Q ∥ P 0 ) ≤ ρ max E X ∼ Q [ ( m ( X , 1 ) − m ( X , 1 ) ) 2 ] ) ≤ κ 2 M n − β − γ
Fast Rate of Convergence with χ 2 \chi^2 χ 2 -Divergence
For χ 2 \chi^2 χ 2 -divergence, f ( t ) = 1 / 2 ( t − 1 ) 2 f(t)=1/2(t-1)^2 f ( t ) = 1/2 ( t − 1 ) 2 and f ⋆ ( s ) = 1 / 2 s 2 + s f^\star(s)=1/2s^2+s f ⋆ ( s ) = 1/2 s 2 + s . The dual optimization problem is given as
min π ∈ Π max λ ≥ 0 , η ∈ R − 1 2 λ E [ ( ( 2 π ( X ) − 1 ) τ ( X ) + η ) 2 ] + E [ ( 2 π ( X ) − 1 ) τ ( X ) ] − λ ρ = : g λ , η ( π ) . (41) \min_{\pi\in\Pi}\max_{\lambda\ge0,\eta\in\mathbb{R}}-\frac{1}{2\lambda}\mathbb{E}\left[((2\pi(X)-1)\tau(X)+\eta)^2\right]+\mathbb{E}\left[(2\pi(X)-1)\tau(X)\right]-\lambda\rho=:g_{\lambda,\eta}(\pi). \tag*{(41)} π ∈ Π min λ ≥ 0 , η ∈ R max − 2 λ 1 E [ (( 2 π ( X ) − 1 ) τ ( X ) + η ) 2 ] + E [ ( 2 π ( X ) − 1 ) τ ( X ) ] − λ ρ =: g λ , η ( π ) . (41)
Lemma 22. Consider the dual optimization problem defined in Equation (39). For any policy π ∈ Π \pi\in\Pi π ∈ Π , the optimal dual variables are
λ ⋆ ( π ) = Var ( ( 2 π ( X ) − 1 ) τ ( X ) ) 2 ρ and η ⋆ ( π ) = − E [ ( 2 π ( X ) − 1 ) τ ( X ) ] . \lambda^{\star}(\pi)=\sqrt{\frac{\operatorname{Var}((2\pi(X)-1)\tau(X))}{2\rho}}
\quad\text{and}\quad
\eta^{\star}(\pi)=-\mathbb{E}[(2\pi(X)-1)\tau(X)]. λ ⋆ ( π ) = 2 ρ Var (( 2 π ( X ) − 1 ) τ ( X )) and η ⋆ ( π ) = − E [( 2 π ( X ) − 1 ) τ ( X )] . Additionally,
A χ 2 ( π ) = E [ ( 2 π ( X ) − 1 ) τ ( X ) ] − 2 ρ ⋅ Var ( ( 2 π ( X ) − 1 ) τ ( X ) ) . A_{\chi^2}(\pi)=\mathbb{E}[(2\pi(X)-1)\tau(X)]-\sqrt{2\rho\cdot\operatorname{Var}((2\pi(X)-1)\tau(X))}. A χ 2 ( π ) = E [( 2 π ( X ) − 1 ) τ ( X )] − 2 ρ ⋅ Var (( 2 π ( X ) − 1 ) τ ( X )) . Proof. Fix any policy π ∈ Π \pi\in\Pi π ∈ Π and define Z π ( X ) = ( 2 π ( X ) − 1 ) τ ( X ) Z_\pi(X)=(2\pi(X)-1)\tau(X) Z π ( X ) = ( 2 π ( X ) − 1 ) τ ( X ) . The optimal solution for η \eta η is η ⋆ ( π ) = − E [ Z π ( X ) ] \eta^{\star}(\pi)=-\mathbb{E}[Z_\pi(X)] η ⋆ ( π ) = − E [ Z π ( X )] . Substituting this value in Equation (39) we obtain the following optimization problem in λ \lambda λ .
min λ ≥ 0 1 2 λ Var ( Z π ( X ) ) + λ ρ \min_{\lambda\geq0}\frac{1}{2\lambda}\operatorname{Var}(Z_\pi(X))+\lambda\rho λ ≥ 0 min 2 λ 1 Var ( Z π ( X )) + λ ρ The optimal solution for λ \lambda λ is λ ⋆ ( π ) = Var ( Z π ( X ) ) 2 ρ \lambda^{\star}(\pi)=\sqrt{\frac{\operatorname{Var}(Z_\pi(X))}{2\rho}} λ ⋆ ( π ) = 2 ρ Var ( Z π ( X )) . Substituting λ ⋆ ( π ) \lambda^{\star}(\pi) λ ⋆ ( π ) and η ⋆ ( π ) \eta^{\star}(\pi) η ⋆ ( π ) in Equation (39) we obtain the final expression on A χ 2 ( π ) A_{\chi^2}(\pi) A χ 2 ( π ) .
Note that, because of Lemma 22 we can choose ( λ , η ) ∈ [ 0 , 2 M / ρ ] × [ − M , M ] = Θ (\lambda,\eta)\in[0,2M/\sqrt{\rho}]\times[-M,M]=\Theta ( λ , η ) ∈ [ 0 , 2 M / ρ ] × [ − M , M ] = Θ .
Bounding Bias at π b ⋆ \pi_b^{\star} π b ⋆
Lemma 23 (Strong Concavity of Dual). Suppose Assumption 8 and Assumption 7 hold, then
λ min ( − ∇ 2 g λ , η ( π b ⋆ ) ) ≥ σ 0 2 λ 0 2 ρ 3 / 2 16 M 5 . \lambda_{\min}(-\nabla^2g_{\lambda,\eta}(\pi_b^{\star}))\geq\frac{\sigma_0^2\lambda_0^2\rho^{3/2}}{16M^5}. λ m i n ( − ∇ 2 g λ , η ( π b ⋆ )) ≥ 16 M 5 σ 0 2 λ 0 2 ρ 3/2 . Proof. Let us define μ k ( λ , η ) = E [ ( ∣ τ ( X ) ∣ + η ) k ] \mu_k(\lambda,\eta)=\mathbb{E}[(|\tau(X)|+\eta)^k] μ k ( λ , η ) = E [( ∣ τ ( X ) ∣ + η ) k ] for k = 0 , 1 , 2 k=0,1,2 k = 0 , 1 , 2 . The Hessian of g g g at a general point ( λ , η ) (\lambda,\eta) ( λ , η ) is
∇ 2 g λ , η ( π b ⋆ ) = ( − μ 2 / λ 3 μ 1 / λ 2 μ 1 / λ 2 − μ 0 / λ ) , \nabla^2g_{\lambda,\eta}(\pi_b^{\star})=
\begin{pmatrix}
-\mu_2/\lambda^3 & \mu_1/\lambda^2\\
\mu_1/\lambda^2 & -\mu_0/\lambda
\end{pmatrix}, ∇ 2 g λ , η ( π b ⋆ ) = ( − μ 2 / λ 3 μ 1 / λ 2 μ 1 / λ 2 − μ 0 / λ ) , so
− ∇ 2 g = 1 λ ( μ 2 / λ 2 − μ 1 / λ − μ 1 / λ μ 0 ) . -\nabla^2g=
\frac{1}{\lambda}
\begin{pmatrix}
\mu_2/\lambda^2 & -\mu_1/\lambda\\
-\mu_1/\lambda& \mu_0
\end{pmatrix}. − ∇ 2 g = λ 1 ( μ 2 / λ 2 − μ 1 / λ − μ 1 / λ μ 0 ) . The determinant of − ∇ 2 g -\nabla^2g − ∇ 2 g is
det ( − ∇ 2 g ) = 1 λ 4 ( μ 2 μ 0 − μ 1 2 ) = Var ( ∣ τ ( X ) ∣ + η ) λ 4 = Var ( ∣ τ ( X ) ∣ ) λ 4 ≥ σ 0 2 λ 4 \det(-\nabla^2g)=\frac{1}{\lambda^4}(\mu_2\mu_0-\mu_1^2)=\frac{\operatorname{Var}(|\tau(X)|+\eta)}{\lambda^4}=\frac{\operatorname{Var}(|\tau(X)|)}{\lambda^4}\geq\frac{\sigma_0^2}{\lambda^4} det ( − ∇ 2 g ) = λ 4 1 ( μ 2 μ 0 − μ 1 2 ) = λ 4 Var ( ∣ τ ( X ) ∣ + η ) = λ 4 Var ( ∣ τ ( X ) ∣ ) ≥ λ 4 σ 0 2 by Assumption 8. Furthermore, tr ( − ∇ 2 g ) ≤ 1 λ ( μ 2 λ 2 + μ 0 ) \operatorname{tr}(-\nabla^2g)\leq\frac{1}{\lambda}(\frac{\mu_2}{\lambda^2}+\mu_0) tr ( − ∇ 2 g ) ≤ λ 1 ( λ 2 μ 2 + μ 0 ) . Therefore,
λ min ( − ∇ 2 g ) ≥ det ( − ∇ 2 g ) tr ( − ∇ 2 g ) ≥ 1 λ 3 ⋅ σ 0 2 1 + μ 2 / λ 2 . \lambda_{\min}(-\nabla^2g)\geq\frac{\det(-\nabla^2g)}{\operatorname{tr}(-\nabla^2g)}\geq\frac{1}{\lambda^3}\cdot\frac{\sigma_0^2}{1+\mu_2/\lambda^2}. λ m i n ( − ∇ 2 g ) ≥ tr ( − ∇ 2 g ) det ( − ∇ 2 g ) ≥ λ 3 1 ⋅ 1 + μ 2 / λ 2 σ 0 2 . We now use μ 2 ≤ 4 M 2 \mu_2\leq4M^2 μ 2 ≤ 4 M 2 , and 2 M / ρ ≥ λ ≥ λ 0 / 2 2M/\sqrt{\rho}\geq\lambda\geq\lambda_0/2 2 M / ρ ≥ λ ≥ λ 0 /2 (by assumption 7 and lemma 22) to obtain
λ min ( − ∇ 2 g ) ≥ σ 0 2 λ 0 2 ρ 3 / 2 16 M 5 . \lambda_{\min}(-\nabla^2g)\geq\frac{\sigma_0^2\lambda_0^2\rho^{3/2}}{16M^5}. λ m i n ( − ∇ 2 g ) ≥ 16 M 5 σ 0 2 λ 0 2 ρ 3/2 .
Lemma 24 (Hessian and Gradient Concentration). Suppose Assumption 7 holds. Then
P ( ∥ ∇ g ^ θ ( π b ⋆ ) ∥ ≥ t ) ≤ 4 exp ( − n t 2 4 B 2 ) with B = 16 M 2 λ 0 2 P\left(\left\|\nabla\widehat{g}_{\theta}(\pi_b^{\star})\right\|\geq t\right)\leq4\exp\left(-\frac{nt^2}{4B^2}\right)
\quad\text{with}\quad B=\frac{16M^2}{\lambda_0^2} P ( ∥ ∇ g θ ( π b ⋆ ) ∥ ≥ t ) ≤ 4 exp ( − 4 B 2 n t 2 ) with B = λ 0 2 16 M 2 and
sup θ = ( λ , η ) ∈ [ λ 0 / 2 , 2 M / ρ ] × [ − M , M ] ∥ ∇ 2 g ^ θ ( π b ⋆ ) − ∇ 2 g θ ( π b ⋆ ) ∥ o p ≤ t \sup_{\theta=(\lambda,\eta)\in[\lambda_0/2,2M/\sqrt{\rho}]\times[-M,M]}
\left\|\nabla^2\widehat{g}_{\theta}(\pi_b^{\star})-\nabla^2g_{\theta}(\pi_b^{\star})\right\|_{\mathrm{op}}\leq t θ = ( λ , η ) ∈ [ λ 0 /2 , 2 M / ρ ] × [ − M , M ] sup ∇ 2 g θ ( π b ⋆ ) − ∇ 2 g θ ( π b ⋆ ) op ≤ t with probability at least 1 − C ⋅ exp ( − n t 2 λ 0 6 M 4 log M ρ λ 0 t ) 1-C\cdot\exp\left(-\frac{nt^2\lambda_0^6}{M^4}\log\frac{M}{\rho\lambda_0t}\right) 1 − C ⋅ exp ( − M 4 n t 2 λ 0 6 log ρ λ 0 t M ) for some universal constant C > 0 C>0 C > 0 .
The proof of the above lemma is almost identical to Lemma 8 and Lemma 9 if one uses the dual function g λ , η ( π b ∗ ) g_{\lambda,\eta}(\pi_b^*) g λ , η ( π b ∗ ) corresponding to the χ 2 \chi^2 χ 2 -divergence.
Lemma 25 (Finite-sample convergence, χ 2 \chi^2 χ 2 ). Suppose assumptions 7 and 8 hold and n ≥ C M 14 λ 0 10 σ 0 4 ρ 3 n \ge\frac{CM^{14}}{\lambda_0^{10}\sigma_0^4\rho^3} n ≥ λ 0 10 σ 0 4 ρ 3 C M 14 . Then for any δ ∈ ( 0 , 1 ) \delta\in(0,1) δ ∈ ( 0 , 1 ) ,
∥ θ ^ n − θ ∗ ∥ 2 ≤ C M 7 σ 0 2 λ 0 4 ρ 3 / 2 log ( 4 / δ ) n \left\|\hat{\theta}_n-\theta^*\right\|_2 \le\frac{CM^7}{\sigma_0^2\lambda_0^4\rho^{3/2}}\sqrt{\frac{\log(4/\delta)}{n}} θ ^ n − θ ∗ 2 ≤ σ 0 2 λ 0 4 ρ 3/2 C M 7 n log ( 4/ δ ) with probability at least 1 − δ 1-\delta 1 − δ .
Proof. We proceed very similar to the proof of Lemma 11. There are two cases to consider. First, suppose θ ^ n \hat{\theta}_n θ ^ n belongs to the interior of Θ \Theta Θ . Then by the first order optimality conditions and Taylor’s expansion we obtain,
0 = ∇ g θ ^ n ( π b ∗ ) = ∇ g θ ∗ ( π b ∗ ) + ∇ 2 g θ ˉ ( π b ∗ ) ( θ ^ n − θ ∗ ) . (42) 0=\nabla g_{\hat{\theta}_n}(\pi_b^*)=\nabla g_{\theta^*}(\pi_b^*)+\nabla^2 g_{\bar{\theta}}(\pi_b^*)(\hat{\theta}_n-\theta^*). \tag*{(42)} 0 = ∇ g θ ^ n ( π b ∗ ) = ∇ g θ ∗ ( π b ∗ ) + ∇ 2 g θ ˉ ( π b ∗ ) ( θ ^ n − θ ∗ ) . (42)
Here θ ˉ \bar{\theta} θ ˉ is a point on the line segment between θ ∗ \theta^* θ ∗ and θ ^ n \hat{\theta}_n θ ^ n and belongs to Θ \Theta Θ . Lemma 23 shows that
∥ − ∇ 2 g ( θ ˉ ) ∥ o p ≥ σ 0 2 λ 0 3 / 2 ρ 3 / 2 16 M 5 = : t . \left\|-\nabla^2g(\bar{\theta})\right\|_{\mathrm{op}}\ge\frac{\sigma_0^2\lambda_0^{3/2}\rho^{3/2}}{16M^5}=:t. − ∇ 2 g ( θ ˉ ) op ≥ 16 M 5 σ 0 2 λ 0 3/2 ρ 3/2 =: t .
By Lemma 24 , we have
∥ ∇ 2 g ( θ ˉ ) − ∇ 2 g θ ^ n ( π b ∗ ) ∥ o p ≤ t / 2 , \left\|\nabla^2g(\bar{\theta})-\nabla^2g_{\hat{\theta}_n}(\pi_b^*)\right\|_{\mathrm{op}}\le t/2, ∇ 2 g ( θ ˉ ) − ∇ 2 g θ ^ n ( π b ∗ ) op ≤ t /2 ,
with probability at least 1 − C ⋅ exp ( − n t 2 λ 0 6 M 4 Δ 0 4 log M ρ λ 0 t ) 1-C\cdot\exp\left(-\frac{nt^2\lambda_0^6}{M^4\Delta_0^4}\log\frac{M}{\rho\lambda_0t}\right) 1 − C ⋅ exp ( − M 4 Δ 0 4 n t 2 λ 0 6 log ρ λ 0 t M ) . Therefore, as long as n ≥ C M 4 λ 0 6 t 2 n\ge\frac{CM^4}{\lambda_0^6t^2} n ≥ λ 0 6 t 2 C M 4 we are guaranteed that
∥ − ∇ 2 g θ ^ n ( π b ∗ ) ∥ o p ≥ t / 2 \left\|-\nabla^2g_{\hat{\theta}_n}(\pi_b^*)\right\|_{\mathrm{op}}\ge t/2 − ∇ 2 g θ ^ n ( π b ∗ ) op ≥ t /2
and the matrix − ∇ 2 g θ ^ n ( π b ∗ ) -\nabla^2g_{\hat{\theta}_n}(\pi_b^*) − ∇ 2 g θ ^ n ( π b ∗ ) is invertible. Therefore, (42) gives us
∥ θ ^ n − θ ∗ ∥ 2 = ∥ ( − ∇ 2 g θ ^ n ( π b ∗ ) ) − 1 ∇ g θ ∗ ( π b ∗ ) ∥ ≤ 2 t ⋅ ∥ ∇ g θ ∗ ( π b ∗ ) ∥ ≤ 32 M 2 λ 0 2 t log ( 4 / δ ) n (43) \left\|\hat{\theta}_n-\theta^*\right\|_2
=\left\|\left(-\nabla^2g_{\hat{\theta}_n}(\pi_b^*)\right)^{-1}\nabla g_{\theta^*}(\pi_b^*)\right\|
\le\frac{2}{t}\cdot\left\|\nabla g_{\theta^*}(\pi_b^*)\right\|
\le\frac{32M^2}{\lambda_0^2t}\sqrt{\frac{\log(4/\delta)}{n}} \tag*{(43)} θ ^ n − θ ∗ 2 = ( − ∇ 2 g θ ^ n ( π b ∗ ) ) − 1 ∇ g θ ∗ ( π b ∗ ) ≤ t 2 ⋅ ∥ ∇ g θ ∗ ( π b ∗ ) ∥ ≤ λ 0 2 t 32 M 2 n log ( 4/ δ ) (43)
with probability at least 1 − δ / 2 1-\delta/2 1 − δ /2 by Lemma 24 . Substituting the value of t t t gives us
∥ θ ^ n − θ ∗ ∥ 2 ≤ C M 7 σ 0 2 λ 0 4 ρ 3 / 2 log ( 4 / δ ) n \left\|\hat{\theta}_n-\theta^*\right\|_2\le\frac{CM^7}{\sigma_0^2\lambda_0^4\rho^{3/2}}\sqrt{\frac{\log(4/\delta)}{n}} θ ^ n − θ ∗ 2 ≤ σ 0 2 λ 0 4 ρ 3/2 C M 7 n log ( 4/ δ )
for a universal constant C C C , as long as n ≥ C M 14 λ 0 10 σ 0 4 ρ 3 n\ge\frac{CM^{14}}{\lambda_0^{10}\sigma_0^4\rho^3} n ≥ λ 0 10 σ 0 4 ρ 3 C M 14 .
We now consider the case when θ ^ n \hat{\theta}_n θ ^ n is on the boundary of Θ \Theta Θ . Since Θ \Theta Θ is convex, the first-order optimality condition for constrained optimization implies ⟨ ∇ g θ ^ n ( π b ∗ ) , θ ^ n − θ ∗ ⟩ ≥ 0 \left\langle\nabla g_{\hat{\theta}_n}(\pi_b^*),\hat{\theta}_n-\theta^*\right\rangle\ge0 ⟨ ∇ g θ ^ n ( π b ∗ ) , θ ^ n − θ ∗ ⟩ ≥ 0 for any θ ∈ Θ \theta\in\Theta θ ∈ Θ . Using eq. (20) we obtain
0 ≤ ⟨ ∇ g θ ^ n ( π b ∗ ) , θ ^ n − θ ∗ ⟩ = ⟨ ∇ g θ ∗ ( π b ∗ ) + ∇ 2 g θ ˉ ( π b ∗ ) ( θ ^ n − θ ∗ ) , θ ^ n − θ ∗ ⟩ . 0\le\left\langle\nabla g_{\hat{\theta}_n}(\pi_b^*),\hat{\theta}_n-\theta^*\right\rangle
=\left\langle\nabla g_{\theta^*}(\pi_b^*)+\nabla^2g_{\bar{\theta}}(\pi_b^*)(\hat{\theta}_n-\theta^*),\hat{\theta}_n-\theta^*\right\rangle. 0 ≤ ⟨ ∇ g θ ^ n ( π b ∗ ) , θ ^ n − θ ∗ ⟩ = ⟨ ∇ g θ ∗ ( π b ∗ ) + ∇ 2 g θ ˉ ( π b ∗ ) ( θ ^ n − θ ∗ ) , θ ^ n − θ ∗ ⟩ .
After rearranging the terms we obtain,
( θ ^ n − θ ∗ ) ⊤ ( − g ˉ θ ∗ ( π b ∗ ) ) ( θ ^ n − θ ∗ ) ≤ ⟨ ∇ g θ ∗ ( π b ∗ ) , θ ^ n − θ ∗ ⟩ . (\hat{\theta}_n-\theta^*)^\top\left(-\bar{g}_{\theta^*}(\pi_b^*)\right)(\hat{\theta}_n-\theta^*)
\le\left\langle\nabla g_{\theta^*}(\pi_b^*),\hat{\theta}_n-\theta^*\right\rangle. ( θ ^ n − θ ∗ ) ⊤ ( − g ˉ θ ∗ ( π b ∗ ) ) ( θ ^ n − θ ∗ ) ≤ ⟨ ∇ g θ ∗ ( π b ∗ ) , θ ^ n − θ ∗ ⟩ .
Since ∥ − ∇ 2 g θ ^ n ( π b ∗ ) ∥ o p ≥ t / 2 \left\|-\nabla^2g_{\hat{\theta}_n}(\pi_b^*)\right\|_{\mathrm{op}}\ge t/2 − ∇ 2 g θ ^ n ( π b ∗ ) op ≥ t /2 with high probability, we have
t 2 ⋅ ∥ θ ^ n − θ ∗ ∥ 2 ≤ ⟨ ∇ g θ ∗ ( π b ∗ ) , θ ^ n − θ ∗ ⟩ ≤ ∥ ∇ g θ ∗ ( π b ∗ ) ∥ ⋅ ∥ θ ^ n − θ ∗ ∥ . \frac{t}{2}\cdot\left\|\hat{\theta}_n-\theta^*\right\|^2
\le\left\langle\nabla g_{\theta^*}(\pi_b^*),\hat{\theta}_n-\theta^*\right\rangle
\le\left\|\nabla g_{\theta^*}(\pi_b^*)\right\|\cdot\left\|\hat{\theta}_n-\theta^*\right\|. 2 t ⋅ θ ^ n − θ ∗ 2 ≤ ⟨ ∇ g θ ∗ ( π b ∗ ) , θ ^ n − θ ∗ ⟩ ≤ ∥ ∇ g θ ∗ ( π b ∗ ) ∥ ⋅ θ ^ n − θ ∗ .
After rearranging, we get the following bound.
∥ θ ^ n − θ ∗ ∥ ≤ 2 t ⋅ ∥ ∇ g θ ∗ ( π b ∗ ) ∥ . \left\|\hat{\theta}_n-\theta^*\right\|\le\frac{2}{t}\cdot\left\|\nabla g_{\theta^*}(\pi_b^*)\right\|. θ ^ n − θ ∗ ≤ t 2 ⋅ ∥ ∇ g θ ∗ ( π b ∗ ) ∥ .
The rest of the proof follows the same argument as in case 1. □
Lemma 26. Suppose assumptions 7 and 8 hold and n ≥ C M 14 λ 0 10 σ 0 4 ρ 3 n\ge\frac{CM^{14}}{\lambda_0^{10}\sigma_0^4\rho^3} n ≥ λ 0 10 σ 0 4 ρ 3 C M 14 . Consider algorithm 3 with χ 2 \chi^2 χ 2 -divergence. Then, for the Bayes optimal policy π b ∗ \pi_b^* π b ∗ , we have
∣ B ( π b ∗ ) ∣ = ∣ E [ A ^ n , χ 2 ( π b ∗ ) ] − A ~ χ 2 ( π b ∗ ) ∣ ≤ C M 17 σ 0 4 λ 0 9 ρ 3 log n n \left|B(\pi_b^*)\right|=\left|\mathbb{E}\left[\hat{A}_{n,\chi^2}(\pi_b^*)\right]-\tilde{A}_{\chi^2}(\pi_b^*)\right|
\le\frac{CM^{17}}{\sigma_0^4\lambda_0^9\rho^3}\frac{\log n}{n} ∣ B ( π b ∗ ) ∣ = E [ A ^ n , χ 2 ( π b ∗ ) ] − A ~ χ 2 ( π b ∗ ) ≤ σ 0 4 λ 0 9 ρ 3 C M 17 n log n for a universal constant C > 0 C>0 C > 0 .
Proof. Since the policy π b ∗ \pi_b^* π b ∗ is fixed throughout the proof, we will write Z ( X ) = ( 2 π b ∗ ( X ) − 1 ) τ ( X ) = ∣ τ ( X ) ∣ Z(X) = (2\pi_b^*(X)-1)\tau(X) = |\tau(X)| Z ( X ) = ( 2 π b ∗ ( X ) − 1 ) τ ( X ) = ∣ τ ( X ) ∣ for brevity. For χ b 2 \chi_b^2 χ b 2 divergence, the dual problem is given as follows.
max λ ≥ 0 , η ∈ R − 1 2 λ ⋅ E [ ( ∣ τ ( X ) ∣ + η ) 2 ] − λ ρ ≔ g λ , η ( π b ∗ ) \max_{\lambda\ge0,\eta\in\mathbb{R}} -\frac{1}{2\lambda}\cdot\mathbb{E}\left[\left(|\tau(X)|+\eta\right)^2\right]-\lambda\rho\coloneqq g_{\lambda,\eta}(\pi_b^*) λ ≥ 0 , η ∈ R max − 2 λ 1 ⋅ E [ ( ∣ τ ( X ) ∣ + η ) 2 ] − λ ρ : = g λ , η ( π b ∗ )
We will also write g ^ λ , η ( π b ∗ ) \hat{g}_{\lambda,\eta}(\pi_b^*) g ^ λ , η ( π b ∗ ) to denote the empirical version of the dual objective function. Let us define μ k = E [ ( ∣ τ ( X ) ∣ + η ) k ] \mu_k=\mathbb{E}\left[(|\tau(X)|+\eta)^k\right] μ k = E [ ( ∣ τ ( X ) ∣ + η ) k ] . Then the Hessian of g λ , η ( π b ∗ ) g_{\lambda,\eta}(\pi_b^*) g λ , η ( π b ∗ ) is given as
∇ 2 g λ , η ( π b ∗ ) = [ − μ 2 λ 3 μ 1 λ 2 μ 1 λ 2 − 1 λ ] \nabla^2g_{\lambda,\eta}(\pi_b^*)=
\begin{bmatrix}
-\frac{\mu_2}{\lambda^3} & \frac{\mu_1}{\lambda^2}\\
\frac{\mu_1}{\lambda^2} & -\frac{1}{\lambda}
\end{bmatrix} ∇ 2 g λ , η ( π b ∗ ) = [ − λ 3 μ 2 λ 2 μ 1 λ 2 μ 1 − λ 1 ]
By Lemma 22, λ ∗ = Var ( ∣ τ ∣ ) / 2 ρ \lambda^*=\sqrt{\operatorname{Var}(|\tau|)/2\rho} λ ∗ = Var ( ∣ τ ∣ ) /2 ρ and η ∗ = − E [ ∣ τ ∣ ] \eta^*=-\mathbb{E}[|\tau|] η ∗ = − E [ ∣ τ ∣ ] . After substitution, we obtain the Hessian matrix of g λ , η ( π b ∗ ) g_{\lambda,\eta}(\pi_b^*) g λ , η ( π b ∗ ) at the optimal solution ( λ ∗ , η ∗ ) (\lambda^*,\eta^*) ( λ ∗ , η ∗ ) .
\nabla^2g_{\lambda^*,\eta^*}(\pi_b^*)=
\begin{bmatrix}
-\sqrt{\frac{8\rho^3\operatorname{Var}(|\tau(X)|)} } & 0\\
0 & -\sqrt{\frac{2\rho}{\operatorname{Var}(|\tau(X)|)}}
\end{bmatrix}
Now let us write θ ^ = ( λ ^ ( π b ∗ ) , η ^ ( π b ∗ ) ) \hat{\theta}=(\hat{\lambda}(\pi_b^*),\hat{\eta}(\pi_b^*)) θ ^ = ( λ ^ ( π b ∗ ) , η ^ ( π b ∗ )) to denote the optimal solution to the empirical dual objective g ^ λ , η ( π b ∗ ) \hat{g}_{\lambda,\eta}(\pi_b^*) g ^ λ , η ( π b ∗ ) . Additionally, let θ ∗ = ( λ ∗ , η ∗ ) \theta^*=(\lambda^*,\eta^*) θ ∗ = ( λ ∗ , η ∗ ) , and Δ θ = θ ^ − θ ∗ \Delta\theta=\hat{\theta}-\theta^* Δ θ = θ ^ − θ ∗ . By Taylor’s expansion we have,
g θ ^ = g θ ∗ + ∇ g θ ∗ T Δ θ + 1 2 ( θ ^ − θ ∗ ) T ⋅ ∇ 2 g ( θ ˉ ) ⋅ ( θ ^ − θ ∗ ) g_{\hat{\theta}}=g_{\theta^*}+\nabla g_{\theta^*}^{\mathsf{T}}\Delta\theta+\frac{1}{2}(\hat{\theta}-\theta^*)^{\mathsf{T}}\cdot\nabla^2g(\bar{\theta})\cdot(\hat{\theta}-\theta^*) g θ ^ = g θ ∗ + ∇ g θ ∗ T Δ θ + 2 1 ( θ ^ − θ ∗ ) T ⋅ ∇ 2 g ( θ ˉ ) ⋅ ( θ ^ − θ ∗ )
where θ ˉ \bar{\theta} θ ˉ is a point on the line segment connecting θ ^ \hat{\theta} θ ^ and θ ∗ \theta^* θ ∗ . Since ∇ g θ ∗ = 0 \nabla g_{\theta^*}=0 ∇ g θ ∗ = 0 , we have
g θ ∗ − g θ ^ = − 1 2 Δ θ T ⋅ ∇ 2 g ( θ ˉ ) ⋅ Δ θ . g_{\theta^*}-g_{\hat{\theta}}=-\frac{1}{2}\Delta\theta^{\mathsf{T}}\cdot\nabla^2g(\bar{\theta})\cdot\Delta\theta. g θ ∗ − g θ ^ = − 2 1 Δ θ T ⋅ ∇ 2 g ( θ ˉ ) ⋅ Δ θ .
Let us also define H ( θ ˉ ) = − ∇ 2 g ( θ ˉ ) H(\bar{\theta})=-\nabla^2g(\bar{\theta}) H ( θ ˉ ) = − ∇ 2 g ( θ ˉ ) . Then we have,
∣ B χ b 2 ( π b ∗ ) ∣ = ∣ E [ g ^ θ ^ ] − g θ ∗ ∣ ≤ E ∣ g θ ∗ − g θ ^ ∣ = 1 2 ⋅ E [ ∣ Δ θ T ⋅ H ( θ ˉ ) ⋅ Δ θ ∣ ] ≤ 1 2 ⋅ E [ ∥ H ( θ ˉ ) ∥ o p ⋅ ∥ Δ θ ∥ 2 2 ] (44) \left|B_{\chi_b^2}(\pi_b^*)\right|=\left|\mathbb{E}[\hat{g}_{\hat{\theta}}]-g_{\theta^*}\right|\leq\mathbb{E}\left|g_{\theta^*}-g_{\hat{\theta}}\right|
=\frac{1}{2}\cdot\mathbb{E}\left[\left|\Delta\theta^{\mathsf{T}}\cdot H(\bar{\theta})\cdot\Delta\theta\right|\right]\leq\frac{1}{2}\cdot\mathbb{E}\left[\left\|H(\bar{\theta})\right\|_{\mathrm{op}}\cdot\left\|\Delta\theta\right\|_2^2\right] \tag*{(44)} B χ b 2 ( π b ∗ ) = E [ g ^ θ ^ ] − g θ ∗ ≤ E g θ ∗ − g θ ^ = 2 1 ⋅ E [ Δ θ T ⋅ H ( θ ˉ ) ⋅ Δ θ ] ≤ 2 1 ⋅ E [ H ( θ ˉ ) op ⋅ ∥ Δ θ ∥ 2 2 ] (44)
We now upper bound the two terms in the above expression. First, Lemma 25 shows that ∥ Δ θ ∥ 2 ≤ C 1 ⋅ log ( 1 / δ ) n \|\Delta\theta\|_2\leq C_1\cdot\sqrt{\frac{\log(1/\delta)}{n}} ∥Δ θ ∥ 2 ≤ C 1 ⋅ n l o g ( 1/ δ ) with probability at least 1 − δ 1-\delta 1 − δ for a suitable constant C 1 C_1 C 1 as long as n ≥ C M 14 λ 0 10 σ 0 4 ρ 3 n\geq\frac{CM^{14}}{\lambda_0^{10}\sigma_0^4\rho^3} n ≥ λ 0 10 σ 0 4 ρ 3 C M 14 . Furthermore,
H ( θ ˉ ) = − ∇ 2 g ( λ ˉ , η ˉ ) = [ E [ ( ∣ τ ( X ) ∣ + η ˉ ) 3 ] λ ˉ 3 − E [ ∣ τ ( X ) ∣ + η ˉ ] λ ˉ 2 − E [ ∣ τ ( X ) ∣ + η ˉ ] λ ˉ 2 1 λ ˉ ] H(\bar{\theta})=-\nabla^2g(\bar{\lambda},\bar{\eta})=
\begin{bmatrix}
\frac{\mathbb{E}[(|\tau(X)|+\bar{\eta})^3]}{\bar{\lambda}^3} & -\frac{\mathbb{E}[|\tau(X)|+\bar{\eta}]}{\bar{\lambda}^2}\\
-\frac{\mathbb{E}[|\tau(X)|+\bar{\eta}]}{\bar{\lambda}^2} & \frac{1}{\bar{\lambda}}
\end{bmatrix} H ( θ ˉ ) = − ∇ 2 g ( λ ˉ , η ˉ ) = [ λ ˉ 3 E [( ∣ τ ( X ) ∣ + η ˉ ) 3 ] − λ ˉ 2 E [ ∣ τ ( X ) ∣ + η ˉ ] − λ ˉ 2 E [ ∣ τ ( X ) ∣ + η ˉ ] λ ˉ 1 ]
Therefore,
∥ H ( θ ˉ ) ∥ o p 2 ≤ ∥ H ( θ ˉ ) ∥ F 2 ≤ 16 M 2 λ 0 2 + ( 8 E [ ( ∣ τ ( X ) ∣ + η ˉ ) 3 ] λ 0 3 ) 2 ≤ 16 M 2 λ 0 2 + 2 12 M 6 λ 0 6 . \|H(\bar{\theta})\|_{\mathrm{op}}^2\leq\|H(\bar{\theta})\|_{\mathrm{F}}^2\leq\frac{16M^2}{\lambda_0^2}+\left(\frac{8\mathbb{E}[(|\tau(X)|+\bar{\eta})^3]}{\lambda_0^3}\right)^2\leq\frac{16M^2}{\lambda_0^2}+\frac{2^{12}M^6}{\lambda_0^6}. ∥ H ( θ ˉ ) ∥ op 2 ≤ ∥ H ( θ ˉ ) ∥ F 2 ≤ λ 0 2 16 M 2 + ( λ 0 3 8 E [( ∣ τ ( X ) ∣ + η ˉ ) 3 ] ) 2 ≤ λ 0 2 16 M 2 + λ 0 6 2 12 M 6 .
Therefore, ∥ H ( θ ˉ ) ∥ o p ≤ 128 M 3 λ 0 3 \|H(\bar{\theta})\|_{\mathrm{op}}\leq\frac{128M^3}{\lambda_0^3} ∥ H ( θ ˉ ) ∥ op ≤ λ 0 3 128 M 3 . Substituting the upper bounds on ∥ Δ ∥ 2 \|\Delta\|_2 ∥Δ ∥ 2 and ∥ H ( θ ˉ ) ∥ o p \|H(\bar{\theta})\|_{\mathrm{op}} ∥ H ( θ ˉ ) ∥ op in eq. (44) we obtain the following bound.
∣ B χ b 2 ( π b ∗ ) ∣ ≤ 64 M 3 λ 0 3 ⋅ E [ ∥ Δ θ ∥ 2 2 ] ≤ 64 M 3 λ 0 3 ( 1 n ⋅ ( 4 M 2 ρ ) 2 + C 1 2 ⋅ log n n ) ≤ 64 M 3 λ 0 3 ( 1 n ⋅ ( 4 M 2 ρ ) 2 + C 2 M 14 σ 0 4 λ 0 6 ρ 3 log n n ) \begin{aligned}
\left|B_{\chi_b^2}(\pi_b^*)\right|\leq\frac{64M^3}{\lambda_0^3}\cdot\mathbb{E}\left[\|\Delta\theta\|_2^2\right] \\
&\leq\frac{64M^3}{\lambda_0^3}\left(\frac{1}{n}\cdot\left(\frac{4M^2}{\rho}\right)^2+C_1^2\cdot\frac{\log n}{n}\right) \\
&\leq\frac{64M^3}{\lambda_0^3}\left(\frac{1}{n}\cdot\left(\frac{4M^2}{\rho}\right)^2+\frac{C^2M^{14}}{\sigma_0^4\lambda_0^6\rho^3}\frac{\log n}{n}\right)
\end{aligned} B χ b 2 ( π b ∗ ) ≤ λ 0 3 64 M 3 ⋅ E [ ∥Δ θ ∥ 2 2 ] ≤ λ 0 3 64 M 3 ( n 1 ⋅ ( ρ 4 M 2 ) 2 + C 1 2 ⋅ n log n ) ≤ λ 0 3 64 M 3 ( n 1 ⋅ ( ρ 4 M 2 ) 2 + σ 0 4 λ 0 6 ρ 3 C 2 M 14 n log n )
for a universal constant C > 0 C>0 C > 0 .
Dual Variable Stability
Lemma 27 (Locally Strongly Concave). For any policy π \pi π with D ( π ) ≤ s max = σ 0 2 / ( 8 M 2 ) D(\pi) \le s_{\max} = \sigma_0^2/(8M^2) D ( π ) ≤ s m a x = σ 0 2 / ( 8 M 2 ) ,
The dual objective g λ , η ( π ) g_{\lambda,\eta}(\pi) g λ , η ( π ) is ( γ / 2 ) (\gamma/2) ( γ /2 ) -strongly concave on the domain Θ = [ λ 0 / 2 , 2 M / ρ ] × [ − M , M ] \Theta= [\lambda_0/2, 2M/\sqrt{\rho}] \times[-M,M] Θ = [ λ 0 /2 , 2 M / ρ ] × [ − M , M ] , where γ \gamma γ is the strong concavity constant from Lemma 23 .
Suppose assumption 2 holds. The dual objective g ~ λ , η ( π ) \widetilde{g}_{\lambda,\eta}(\pi) g λ , η ( π ) is ( κ 2 γ / 4 ) (\kappa^2\gamma/4) ( κ 2 γ /4 ) -strongly concave on the domain Θ = [ λ 0 / 2 , 2 M / ρ ] × [ − M , M ] \Theta= [\lambda_0/2, 2M/\sqrt{\rho}] \times[-M,M] Θ = [ λ 0 /2 , 2 M / ρ ] × [ − M , M ] , as long as n ≥ ( 16 M κ σ 0 2 ) 1 / ( β + γ ) n \ge\left(\frac{16M}{\kappa\sigma_0^2}\right)^{1/(\beta+\gamma)} n ≥ ( κ σ 0 2 16 M ) 1/ ( β + γ ) .
Proof. By following the proof of Lemma 23 we can establish that for any λ ∈ [ λ 0 / 2 , 2 M / ρ ] \lambda\in[\lambda_0/2,2M/\rho] λ ∈ [ λ 0 /2 , 2 M / ρ ] and η ∈ [ − M , M ] \eta\in[-M,M] η ∈ [ − M , M ] ,
λ min ( − ∇ 2 g λ , η ( π ) ) ≥ λ 0 2 ρ 3 / 2 16 M 5 Var ( Z ( π ) ) . \lambda_{\min}(-\nabla^2 g_{\lambda,\eta}(\pi)) \ge\frac{\lambda_0^2\rho^{3/2}}{16M^5}\operatorname{Var}(Z(\pi)). λ m i n ( − ∇ 2 g λ , η ( π )) ≥ 16 M 5 λ 0 2 ρ 3/2 Var ( Z ( π )) .
Now note that the variance perturbation (Lemma 14 ) holds for general divergence function f f f . Therefore, Var ( Z ( π ) ) ≥ Var ( Z ( π ∗ ) ) − σ 0 2 / 2 ≥ σ 0 2 / 2 \operatorname{Var}(Z(\pi)) \ge\operatorname{Var}(Z(\pi^*))-\sigma_0^2/2 \ge\sigma_0^2/2 Var ( Z ( π )) ≥ Var ( Z ( π ∗ )) − σ 0 2 /2 ≥ σ 0 2 /2 as D ( π ) ≤ s max D(\pi) \le s_{\max} D ( π ) ≤ s m a x . Substituting this lower bound gives us the first result.
In order to prove the second result, let us define Z ~ π ( X , W ) = ( 2 π ( X ) − 1 ) Γ ^ ( X , W ) \widetilde{Z}_{\pi}(X,W) = (2\pi(X)-1)\widehat{\Gamma}(X,W) Z π ( X , W ) = ( 2 π ( X ) − 1 ) Γ ( X , W ) . Now by following the proof of Lemma 23 we can establish that for any λ ∈ [ λ 0 / 2 , 2 M / ρ ] \lambda\in[\lambda_0/2,2M/\sqrt{\rho}] λ ∈ [ λ 0 /2 , 2 M / ρ ] and η ∈ [ − M , M ] \eta\in[-M,M] η ∈ [ − M , M ] ,
λ min ( − ∇ 2 g ~ λ , η ( π ) ) ≥ λ 0 2 ρ 3 / 2 κ 2 16 M 5 Var ( Z ~ π ( X , W ) ) . \lambda_{\min}(-\nabla^2\widetilde{g}_{\lambda,\eta}(\pi)) \ge\frac{\lambda_0^2\rho^{3/2}\kappa^2}{16M^5}\operatorname{Var}(\widetilde{Z}_{\pi}(X,W)). λ m i n ( − ∇ 2 g λ , η ( π )) ≥ 16 M 5 λ 0 2 ρ 3/2 κ 2 Var ( Z π ( X , W )) .
The proof of Corollary 2 establishes the following lower bound on the variance of Z ~ π ( X , W ) \widetilde{Z}_{\pi}(X,W) Z π ( X , W ) under P 0 P_0 P 0 .
Var P 0 ( Z ~ π ( X , W ) ) ≥ Var P 0 ( Z π ( X ) ) − 4 M κ n − ( β + γ ) ≥ σ 0 2 2 − 4 M κ n − ( β + γ ) \begin{aligned}
\operatorname{Var}_{P_0}(\widetilde{Z}_{\pi}(X,W)) \ge\operatorname{Var}_{P_0}(Z_{\pi}(X))-\frac{4M}{\kappa}n^{-(\beta+\gamma)} \\
&\ge\frac{\sigma_0^2}{2}-\frac{4M}{\kappa}n^{-(\beta+\gamma)}
\end{aligned} Var P 0 ( Z π ( X , W )) ≥ Var P 0 ( Z π ( X )) − κ 4 M n − ( β + γ ) ≥ 2 σ 0 2 − κ 4 M n − ( β + γ )
for any policy π \pi π with D ( π ) ≤ s max = σ 0 2 / ( 8 M 2 ) D(\pi) \le s_{\max} = \sigma_0^2/(8M^2) D ( π ) ≤ s m a x = σ 0 2 / ( 8 M 2 ) . Therefore, if n ≥ ( 16 M κ σ 0 2 ) 1 / ( β + γ ) n \ge\left(\frac{16M}{\kappa\sigma_0^2}\right)^{1/(\beta+\gamma)} n ≥ ( κ σ 0 2 16 M ) 1/ ( β + γ ) , then Var P 0 ( Z ~ π ( X , W ) ) ≥ σ 0 2 / 4 \operatorname{Var}_{P_0}(\widetilde{Z}_{\pi}(X,W)) \ge\sigma_0^2/4 Var P 0 ( Z π ( X , W )) ≥ σ 0 2 /4 . Substituting this lower bound on Var Q ~ ( Z ( π ) ) \operatorname{Var}_{\widetilde{Q}}(Z(\pi)) Var Q ( Z ( π )) in the above expression gives us the desired strong concavity constant.
Lemma 28 (Hessian concentration for General Policies). Suppose Assumptions 7 and 4 hold. Then for any θ = ( λ , η ) ∈ [ λ 0 / 2 , 2 M / ρ ] × [ − M , M ] \theta= (\lambda,\eta) \in[\lambda_0/2,2M/\sqrt{\rho}] \times[-M,M] θ = ( λ , η ) ∈ [ λ 0 /2 , 2 M / ρ ] × [ − M , M ] , and for any policy π \pi π we have,
∥ ∇ 2 g ^ θ ( π ) − ∇ 2 g ~ θ ( π ) ∥ o p ≤ C M 2 λ 0 3 κ 0 2 d log ( M L B n / ( κ λ 0 ) ) + log ( 1 / δ ) n ⋅ log ( M n / ( ρ κ λ 0 ) ) \left\|\nabla^2\widehat{g}_{\theta}(\pi)-\nabla^2\widetilde{g}_{\theta}(\pi)\right\|_{\mathrm{op}} \le\frac{CM^2}{\lambda_0^3\kappa_0^2}\sqrt{\frac{d\log(MLBn/(\kappa\lambda_0))+\log(1/\delta)}{n\cdot\log(Mn/(\rho\kappa\lambda_0))}} ∇ 2 g θ ( π ) − ∇ 2 g θ ( π ) op ≤ λ 0 3 κ 0 2 C M 2 n ⋅ log ( M n / ( ρ κ λ 0 )) d log ( M L B n / ( κ λ 0 )) + log ( 1/ δ ) with probability at least 1 − δ 1-\delta 1 − δ for a universal constant C > 0 C>0 C > 0 .
The proof of the above Lemma is almost identical to the proof of Lemma 15 .
Lemma 29 (Dual variable stability). Suppose Assumptions 1, 2, 4, 7 hold and n ≥ O ~ ( M 14 κ 8 ρ 3 λ 0 10 σ 0 4 ⋅ ( d + log ( 1 / δ ) ) ) n \ge\widetilde{O}\left(\frac{M^{14}}{\kappa^8\rho^3\lambda_0^{10}\sigma_0^4}\cdot(d+\log(1/\delta))\right) n ≥ O ( κ 8 ρ 3 λ 0 10 σ 0 4 M 14 ⋅ ( d + log ( 1/ δ )) ) . Let θ ^ ( π ) = ( λ ^ ( π ) , η ^ ( π ) ) \widehat{\theta}(\pi)=(\widehat{\lambda}(\pi),\widehat{\eta}(\pi)) θ ( π ) = ( λ ( π ) , η ( π )) denote the empirical dual optimizer on D 1 D_1 D 1 for policy π \pi π . Then for any two policies π , π ~ \pi,\widetilde{\pi} π , π with 1 ∣ D 1 ∣ ∑ i ∈ D 1 ∣ π ( X i ) − π ~ ( X i ) ∣ ≤ ϵ \frac{1}{|D_1|}\sum_{i\in D_1}|\pi(X_i)-\widetilde{\pi}(X_i)|\le\epsilon ∣ D 1 ∣ 1 ∑ i ∈ D 1 ∣ π ( X i ) − π ( X i ) ∣ ≤ ϵ and max ( D ( π ) , D ( π ~ ) ) ≤ s max = σ 0 2 / ( 8 M 2 ) \max(D(\pi),D(\widetilde{\pi}))\le s_{\max}=\sigma_0^2/(8M^2) max ( D ( π ) , D ( π )) ≤ s m a x = σ 0 2 / ( 8 M 2 ) :
∥ θ ^ ( π ) − θ ^ ( π ~ ) ∥ ≤ 6144 M 7 λ 0 κ 4 ρ 3 / 2 σ 0 2 ϵ \|\widehat{\theta}(\pi)-\widehat{\theta}(\widetilde{\pi})\| \le\frac{6144M^7}{\lambda_0\kappa^4\rho^{3/2}\sigma_0^2}\epsilon ∥ θ ( π ) − θ ( π ) ∥ ≤ λ 0 κ 4 ρ 3/2 σ 0 2 6144 M 7 ϵ with probability at least 1 − δ 1-\delta 1 − δ .
Proof. The proof has two main steps. We first bound the pointwise perturbation of the gradient, and then use strong concavity to convert gradient perturbation into optimizer perturbation.
Step 1: Perturbation of the dual objective. The empirical dual for policy π \pi π on D 1 D_1 D 1 is
g ^ λ , η ( π ) = − 1 2 λ ∣ D 1 ∣ ∑ i ∈ D 1 ( Z i ( π ) + η ) 2 − λ ρ \hat{g}_{\lambda,\eta}(\pi)=-\frac{1}{2\lambda\lvert D_1\rvert}\sum_{i\in D_1}\left(Z_i(\pi)+\eta\right)^2-\lambda\rho g ^ λ , η ( π ) = − 2 λ ∣ D 1 ∣ 1 i ∈ D 1 ∑ ( Z i ( π ) + η ) 2 − λ ρ
where Z i ( π ) = ( 2 π ( X i ) − 1 ) Γ ^ i Z_i(\pi)=(2\pi(X_i)-1)\hat{\Gamma}_i Z i ( π ) = ( 2 π ( X i ) − 1 ) Γ ^ i . Let us also define the perturbation in the dual objective as the policy is changed from π \pi π to π ~ \tilde{\pi} π ~ .
Δ g ^ λ , η ( π , π ~ ) = g ^ λ , η ( π ) − g ^ λ , η ( π ~ ) = − 1 2 λ ∣ D 1 ∣ ∑ i ∈ D 1 [ ( Z i ( π ) + η ) 2 − ( Z i ( π ~ ) + η ) 2 ] \Delta\hat{g}_{\lambda,\eta}(\pi,\tilde{\pi})=\hat{g}_{\lambda,\eta}(\pi)-\hat{g}_{\lambda,\eta}(\tilde{\pi})=-\frac{1}{2\lambda\lvert D_1\rvert}\sum_{i\in D_1}\left[\left(Z_i(\pi)+\eta\right)^2-\left(Z_i(\tilde{\pi})+\eta\right)^2\right] Δ g ^ λ , η ( π , π ~ ) = g ^ λ , η ( π ) − g ^ λ , η ( π ~ ) = − 2 λ ∣ D 1 ∣ 1 i ∈ D 1 ∑ [ ( Z i ( π ) + η ) 2 − ( Z i ( π ~ ) + η ) 2 ]
We now bound the partial derivatives of the perturbation Δ g ^ λ , η ( π , π ~ ) \Delta\hat{g}_{\lambda,\eta}(\pi,\tilde{\pi}) Δ g ^ λ , η ( π , π ~ ) with respect to λ \lambda λ and η \eta η .
∂ Δ g ^ λ , η ( π , π ~ ) ∂ η = 1 λ ∣ D 1 ∣ ∑ i ∈ D 1 [ Z i ( π ) − Z i ( π ~ ) ] . \frac{\partial\Delta\hat{g}_{\lambda,\eta}(\pi,\tilde{\pi})}{\partial\eta}=\frac{1}{\lambda\lvert D_1\rvert}\sum_{i\in D_1}\left[Z_i(\pi)-Z_i(\tilde{\pi})\right]. ∂ η ∂ Δ g ^ λ , η ( π , π ~ ) = λ ∣ D 1 ∣ 1 i ∈ D 1 ∑ [ Z i ( π ) − Z i ( π ~ ) ] .
For λ ≥ λ 0 / 2 \lambda\ge\lambda_0/2 λ ≥ λ 0 /2 :
∣ ∂ Δ g ^ λ , η ( π , π ~ ) ∂ η ∣ ≤ 2 λ 0 1 ∣ D 1 ∣ ∑ i ∈ D 1 ∣ Z i ( π ) − Z i ( π ~ ) ∣ ≤ 4 M κ λ 0 1 ∣ D 1 ∣ ∑ i ∈ D 1 ∣ π ( X i ) − π ~ ( X i ) ∣ ≤ 4 M κ λ 0 ε . (45) \left\lvert\frac{\partial\Delta\hat{g}_{\lambda,\eta}(\pi,\tilde{\pi})}{\partial\eta}\right\rvert\le\frac{2}{\lambda_0}\frac{1}{\lvert D_1\rvert}\sum_{i\in D_1}\left\lvert Z_i(\pi)-Z_i(\tilde{\pi})\right\rvert\le\frac{4M}{\kappa\lambda_0}\frac{1}{\lvert D_1\rvert}\sum_{i\in D_1}\left\lvert\pi(X_i)-\tilde{\pi}(X_i)\right\rvert\le\frac{4M}{\kappa\lambda_0}\varepsilon. \tag*{(45)} ∂ η ∂ Δ g ^ λ , η ( π , π ~ ) ≤ λ 0 2 ∣ D 1 ∣ 1 i ∈ D 1 ∑ ∣ Z i ( π ) − Z i ( π ~ ) ∣ ≤ κ λ 0 4 M ∣ D 1 ∣ 1 i ∈ D 1 ∑ ∣ π ( X i ) − π ~ ( X i ) ∣ ≤ κ λ 0 4 M ε . (45)
Let us now consider the derivative with respect to λ \lambda λ .
∂ Δ g ^ λ , η ( π , π ~ ) ∂ λ = 1 2 λ 2 ∣ D 1 ∣ ∑ i ∈ D 1 [ ( Z i ( π ) + η ) 2 − ( Z i ( π ~ ) + η ) 2 ] = 1 2 λ 2 ∣ D 1 ∣ ∑ i ∈ D 1 ( Z i ( π ) + Z i ( π ~ ) + 2 η ) ( Z i ( π ) − Z i ( π ~ ) ) . \begin{aligned}
\frac{\partial\Delta\hat{g}_{\lambda,\eta}(\pi,\tilde{\pi})}{\partial\lambda}=\frac{1}{2\lambda^2\lvert D_1\rvert}\sum_{i\in D_1}\left[\left(Z_i(\pi)+\eta\right)^2-\left(Z_i(\tilde{\pi})+\eta\right)^2\right] \\
&=\frac{1}{2\lambda^2\lvert D_1\rvert}\sum_{i\in D_1}\left(Z_i(\pi)+Z_i(\tilde{\pi})+2\eta\right)\left(Z_i(\pi)-Z_i(\tilde{\pi})\right).
\end{aligned} ∂ λ ∂ Δ g ^ λ , η ( π , π ~ ) = 2 λ 2 ∣ D 1 ∣ 1 i ∈ D 1 ∑ [ ( Z i ( π ) + η ) 2 − ( Z i ( π ~ ) + η ) 2 ] = 2 λ 2 ∣ D 1 ∣ 1 i ∈ D 1 ∑ ( Z i ( π ) + Z i ( π ~ ) + 2 η ) ( Z i ( π ) − Z i ( π ~ ) ) .
For λ ≥ λ 0 / 2 \lambda\ge\lambda_0/2 λ ≥ λ 0 /2 and ∣ η ∣ ≤ M \lvert\eta\rvert\le M ∣ η ∣ ≤ M :
∣ ∂ Δ g ^ λ , η ( π , π ~ ) ∂ λ ∣ ≤ 12 M κ λ 0 1 ∣ D 1 ∣ ∑ i ∈ D 1 ∣ Z i ( π ) − Z i ( π ~ ) ∣ ≤ 24 M 2 κ 2 λ 0 2 ε . (46) \left\lvert\frac{\partial\Delta\hat{g}_{\lambda,\eta}(\pi,\tilde{\pi})}{\partial\lambda}\right\rvert\le\frac{12M}{\kappa\lambda_0}\frac{1}{\lvert D_1\rvert}\sum_{i\in D_1}\left\lvert Z_i(\pi)-Z_i(\tilde{\pi})\right\rvert\le\frac{24M^2}{\kappa^2\lambda_0^2}\varepsilon. \tag*{(46)} ∂ λ ∂ Δ g ^ λ , η ( π , π ~ ) ≤ κ λ 0 12 M ∣ D 1 ∣ 1 i ∈ D 1 ∑ ∣ Z i ( π ) − Z i ( π ~ ) ∣ ≤ κ 2 λ 0 2 24 M 2 ε . (46)
Combining (45) and (46) :
∥ ∇ Δ g ^ λ , η ( π , π ~ ) ∥ ≤ 48 M 2 κ 2 λ 0 ε for all θ ∈ Θ . (47) \left\lVert\nabla\Delta\hat{g}_{\lambda,\eta}(\pi,\tilde{\pi})\right\rVert\le\frac{48M^2}{\kappa^2\lambda_0}\varepsilon\quad\text{for all }\theta\in\Theta. \tag*{(47)} ∥ ∇Δ g ^ λ , η ( π , π ~ ) ∥ ≤ κ 2 λ 0 48 M 2 ε for all θ ∈ Θ. (47)
Step 2: From gradient perturbation to optimizer perturbation.
Let θ ^ 1 = θ ^ ( π ) \hat{\theta}_1=\hat{\theta}(\pi) θ ^ 1 = θ ^ ( π ) and θ ^ 2 = θ ^ ( π ~ ) \hat{\theta}_2=\hat{\theta}(\tilde{\pi}) θ ^ 2 = θ ^ ( π ~ ) . Suppose first that both are interior optimizers (the boundary case is handled below). The first-order conditions are
∇ g ^ θ ^ 1 ( π ) = 0 , ∇ g ^ θ ^ 2 ( π ~ ) = 0. \nabla\hat{g}_{\hat{\theta}_1}(\pi)=0,\qquad\nabla\hat{g}_{\hat{\theta}_2}(\tilde{\pi})=0. ∇ g ^ θ ^ 1 ( π ) = 0 , ∇ g ^ θ ^ 2 ( π ~ ) = 0.
Since g ^ θ ^ 1 ( π ) = g ^ θ ^ 1 ( π ~ ) + Δ g ^ θ ^ 1 ( π , π ~ ) \hat{g}_{\hat{\theta}_1}(\pi)=\hat{g}_{\hat{\theta}_1}(\tilde{\pi})+\Delta\hat{g}_{\hat{\theta}_1}(\pi,\tilde{\pi}) g ^ θ ^ 1 ( π ) = g ^ θ ^ 1 ( π ~ ) + Δ g ^ θ ^ 1 ( π , π ~ ) , the first condition becomes
∇ g ^ θ ^ 1 ( π ~ ) + ∇ Δ g ^ θ ^ 1 ( π , π ~ ) = 0. (48) \nabla\hat{g}_{\hat{\theta}_1}(\tilde{\pi})+\nabla\Delta\hat{g}_{\hat{\theta}_1}(\pi,\tilde{\pi})=0. \tag*{(48)} ∇ g ^ θ ^ 1 ( π ~ ) + ∇Δ g ^ θ ^ 1 ( π , π ~ ) = 0. (48)
By the mean value theorem applied to ∇ g ^ ( ⋅ ) ( π ~ ) \nabla\hat{g}_{(\cdot)}(\tilde{\pi}) ∇ g ^ ( ⋅ ) ( π ~ ) between θ ^ 2 \hat{\theta}_2 θ ^ 2 and θ ^ 1 \hat{\theta}_1 θ ^ 1 , there exists θ ˉ \bar{\theta} θ ˉ on the segment between θ ^ 1 \hat{\theta}_1 θ ^ 1 and θ ^ 2 \hat{\theta}_2 θ ^ 2 such that
∇ g ^ θ ^ 1 ( π ~ ) = ∇ g ^ θ ^ 2 ( π ~ ) ⏟ = 0 + ∇ 2 g ^ θ ˉ ( π ) ( θ ^ 1 − θ ^ 2 ) . \nabla\hat{g}_{\hat{\theta}_1}(\tilde{\pi})=\underbrace{\nabla\hat{g}_{\hat{\theta}_2}(\tilde{\pi})}_{=0}+\nabla^2\hat{g}_{\bar{\theta}}(\pi)(\hat{\theta}_1-\hat{\theta}_2). ∇ g ^ θ ^ 1 ( π ~ ) = = 0 ∇ g ^ θ ^ 2 ( π ~ ) + ∇ 2 g ^ θ ˉ ( π ) ( θ ^ 1 − θ ^ 2 ) .
Substituting into (48) and rearranging:
∇ 2 g ^ θ ^ ( π ~ ) ( θ ^ 1 − θ ^ 2 ) = − ∇ Δ g ^ θ ^ 1 ( π , π ~ ) . (49) \nabla^2 \hat{g}_{\hat{\theta}}(\tilde{\pi})(\hat{\theta}_1-\hat{\theta}_2)=-\nabla\Delta\hat{g}_{\hat{\theta}_1}(\pi,\tilde{\pi}). \tag*{(49)} ∇ 2 g ^ θ ^ ( π ~ ) ( θ ^ 1 − θ ^ 2 ) = − ∇Δ g ^ θ ^ 1 ( π , π ~ ) . (49)
By Lemma 27 (which applies since D ( π ~ ) ≤ s max D(\tilde{\pi})\le s_{\max} D ( π ~ ) ≤ s m a x ) and Lemma 28 (Hessian concentration for general policies),
λ min ( − ∇ 2 g ^ θ ^ ( π ~ ) ) ≥ λ 0 2 σ 0 3 / 2 κ 2 σ 0 2 64 M 5 ⏟ γ − C M 2 λ 0 3 κ 2 d log ( M L B n / ( κ λ 0 ) ) + log ( 1 / δ ) n ⋅ log ( M n / ( ρ κ λ 0 ) ) ≥ γ ˉ 2 \lambda_{\min}\left(-\nabla^2\hat{g}_{\hat{\theta}}(\tilde{\pi})\right)\ge\underbrace{\frac{\lambda_0^2\sigma_0^{3/2}\kappa^2\sigma_0^2}{64M^5}}_{\gamma}-\frac{CM^2}{\lambda_0^3\kappa^2}\sqrt{\frac{d\log(MLBn/(\kappa\lambda_0))+\log(1/\delta)}{n\cdot\log(Mn/(\rho\kappa\lambda_0))}}\ge\frac{\bar{\gamma}}{2} λ m i n ( − ∇ 2 g ^ θ ^ ( π ~ ) ) ≥ γ 64 M 5 λ 0 2 σ 0 3/2 κ 2 σ 0 2 − λ 0 3 κ 2 C M 2 n ⋅ log ( M n / ( ρ κ λ 0 )) d log ( M L B n / ( κ λ 0 )) + log ( 1/ δ ) ≥ 2 γ ˉ
as long as n ≥ O ~ ( M 14 κ 8 ρ 3 λ 0 10 σ 0 4 ( d + log ( 1 / δ ) ) ) n\ge\widetilde{O}\left(\frac{M^{14}}{\kappa^8\rho^3\lambda_0^{10}\sigma_0^4}(d+\log(1/\delta))\right) n ≥ O ( κ 8 ρ 3 λ 0 10 σ 0 4 M 14 ( d + log ( 1/ δ )) ) . Therefore − ∇ 2 g ^ θ ^ ( π ~ ) -\nabla^2\hat{g}_{\hat{\theta}}(\tilde{\pi}) − ∇ 2 g ^ θ ^ ( π ~ ) is invertible, and ∥ [ − ∇ 2 g ^ θ ^ ( π ~ ) ] − 1 ∥ o p ≤ 2 / γ ˉ \left\|\left[-\nabla^2\hat{g}_{\hat{\theta}}(\tilde{\pi})\right]^{-1}\right\|_{\mathrm{op}}\le2/\bar{\gamma} [ − ∇ 2 g ^ θ ^ ( π ~ ) ] − 1 op ≤ 2/ γ ˉ , which gives us the following bound.
∥ θ ^ 1 − θ ^ 2 ∥ ≤ ∥ [ − ∇ 2 g ^ θ ^ ( π ~ ) ] − 1 ∥ o p ∥ ∇ Δ g ^ θ ^ 1 ( π , π ~ ) ∥ ≤ 96 M 2 γ ˉ κ 2 λ 0 ε . \|\hat{\theta}_1-\hat{\theta}_2\|\le\left\|\left[-\nabla^2\hat{g}_{\hat{\theta}}(\tilde{\pi})\right]^{-1}\right\|_{\mathrm{op}}\|\nabla\Delta\hat{g}_{\hat{\theta}_1}(\pi,\tilde{\pi})\|\le\frac{96M^2}{\bar{\gamma}\kappa^2\lambda_0}\varepsilon. ∥ θ ^ 1 − θ ^ 2 ∥ ≤ [ − ∇ 2 g ^ θ ^ ( π ~ ) ] − 1 op ∥∇Δ g ^ θ ^ 1 ( π , π ~ ) ∥ ≤ γ ˉ κ 2 λ 0 96 M 2 ε .
Boundary case. If θ ^ 1 \hat{\theta}_1 θ ^ 1 or θ ^ 2 \hat{\theta}_2 θ ^ 2 lies on the boundary of the constraint set Θ = [ λ 0 / 2 , 2 M / ρ ] × [ − M , M ] \Theta=[\lambda_0/2,2M/\sqrt{\rho}]\times[-M,M] Θ = [ λ 0 /2 , 2 M / ρ ] × [ − M , M ] , the first-order condition is replaced by the variational inequality for constrained optimization. The argument in this case is identical to the proof of Lemma 16 .
Establishing Fast Rate with Peeling Argument
Let us define the following constants.
(a) Λ 1 ( δ ) : = max { O ~ ( M 14 κ 8 ρ 3 λ 0 10 σ 0 4 ( d + log ( 1 / δ ) ) ) , O ( M κ σ 0 2 ) 1 / ( β + γ ) } \Lambda_1(\delta):=\max\left\{\widetilde{O}\left(\frac{M^{14}}{\kappa^8\rho^3\lambda_0^{10}\sigma_0^4}(d+\log(1/\delta))\right),O\left(\frac{M}{\kappa\sigma_0^2}\right)^{1/(\beta+\gamma)}\right\} Λ 1 ( δ ) := max { O ( κ 8 ρ 3 λ 0 10 σ 0 4 M 14 ( d + log ( 1/ δ )) ) , O ( κ σ 0 2 M ) 1/ ( β + γ ) } .
(b) Λ 2 ( δ ) : = 16 2 + 1 / α M 2 η 2 ( 8 M 2 σ 0 2 ) 2 ( 1 + α ) α ( d log ( L B n ) + log ( 1 / δ ) ) \Lambda_2(\delta):=\frac{16^{2+1/\alpha}M^2}{\eta^2}\left(\frac{8M^2}{\sigma_0^2}\right)^{\frac{2(1+\alpha)}{\alpha}}\left(d\log(LBn)+\log(1/\delta)\right) Λ 2 ( δ ) := η 2 1 6 2 + 1/ α M 2 ( σ 0 2 8 M 2 ) α 2 ( 1 + α ) ( d log ( L B n ) + log ( 1/ δ ) ) .
(c) Λ 3 ( δ ) : = ( 4 2 + 1 / α M κ η ⋅ ( 8 M 2 σ 0 2 ) α + 1 α ) 1 β + γ \Lambda_3(\delta):=\left(\frac{4^{2+1/\alpha}M}{\kappa\eta}\cdot\left(\frac{8M^2}{\sigma_0^2}\right)^{\frac{\alpha+1}{\alpha}}\right)^{\frac{1}{\beta+\gamma}} Λ 3 ( δ ) := ( κ η 4 2 + 1/ α M ⋅ ( σ 0 2 8 M 2 ) α α + 1 ) β + γ 1 .
Theorem 5. Suppose Assumptions 1 1 1 -4 and 6 6 6 -8 hold, ρ ≤ σ 0 4 2 M \rho\le\frac{\sigma_0}{4\sqrt{2M}} ρ ≤ 4 2 M σ 0 , and n ≥ max { Λ 1 ( δ ) , Λ 2 ( δ ) , Λ 3 ( δ ) } n\ge\max\{\Lambda_1(\delta),\Lambda_2(\delta),\Lambda_3(\delta)\} n ≥ max { Λ 1 ( δ ) , Λ 2 ( δ ) , Λ 3 ( δ )} . Then with probability at least 1 − δ 1-\delta 1 − δ we have,
A χ 2 ( π b ⋆ ) − A χ 2 ( π ^ n ) ≤ 8 M κ ⋅ 1 n β + γ + C 1 ⋅ log n n + C v C 2 ( Q ( d , n ) n ) ( 1 + α ) / ( 2 + α ) + C v Q ( d , n ) n A_{\chi^2}(\pi_b^\star)-A_{\chi^2}(\hat{\pi}_n)\le\frac{8M}{\kappa}\cdot\frac{1}{n^{\beta+\gamma}}+C_1\cdot\frac{\log n}{n}+C_vC_2\left(\frac{Q(d,n)}{n}\right)^{(1+\alpha)/(2+\alpha)}+\frac{C_vQ(d,n)}{n} A χ 2 ( π b ⋆ ) − A χ 2 ( π ^ n ) ≤ κ 8 M ⋅ n β + γ 1 + C 1 ⋅ n log n + C v C 2 ( n Q ( d , n ) ) ( 1 + α ) / ( 2 + α ) + n C v Q ( d , n ) where C 1 = C M 17 σ 0 4 λ 0 8 ρ 3 C_1=\frac{CM^{17}}{\sigma_0^4\lambda_0^8\rho^3} C 1 = σ 0 4 λ 0 8 ρ 3 C M 17 , C v = C M 9 κ 6 λ 0 5 ρ 3 / 2 σ 0 2 C_v=\frac{CM^9}{\kappa^6\lambda_0^5\rho^{3/2}\sigma_0^2} C v = κ 6 λ 0 5 ρ 3/2 σ 0 2 C M 9 , C 2 = ( 5 2 η 4 ( 1 + α ) / α ) α / ( α + 2 ) C_2=\left(\frac{5}{2\eta}4^{(1+\alpha)/\alpha}\right)^{\alpha/( \alpha+2)} C 2 = ( 2 η 5 4 ( 1 + α ) / α ) α / ( α + 2 ) , and Q ( d , n ) = C d log ( L B n M κ λ 0 ρ σ 0 ) + C log ( ( log n ) / δ ) Q(d,n)=Cd\log\left(\frac{LBnM}{\kappa\lambda_0\rho\sigma_0}\right)+C\log((\log n)/\delta) Q ( d , n ) = C d log ( κ λ 0 ρ σ 0 L B n M ) + C log (( log n ) / δ ) for a universal (parameter-independent) constant C C C .
Proof. Proceeding similarly to the proof of Theorem 4 we can establish the following regret decomposition.
A χ 2 ( π b ⋆ ) − A χ 2 ( π ^ n ) ≤ 6 M κ n − ( β + γ ) − B χ 2 ( π b ⋆ ) + E [ ϕ ( π ^ n ) ] − ϕ ( π ^ n ) (50) A_{\chi^2}(\pi_b^\star)-A_{\chi^2}(\hat{\pi}_n)\le\frac{6M}{\kappa}n^{-(\beta+\gamma)}-B_{\chi^2}(\pi_b^\star)+\mathbb{E}[\phi(\hat{\pi}_n)]-\phi(\hat{\pi}_n) \tag*{(50)} A χ 2 ( π b ⋆ ) − A χ 2 ( π ^ n ) ≤ κ 6 M n − ( β + γ ) − B χ 2 ( π b ⋆ ) + E [ ϕ ( π ^ n )] − ϕ ( π ^ n ) (50)
We can apply Lemma 26 to bound ∣ B χ 2 ( π b ⋆ ) ∣ ≤ C 1 ⋅ log n n \left|B_{\chi^2}(\pi_b^\star)\right|\le C_1\cdot\frac{\log n}{n} B χ 2 ( π b ⋆ ) ≤ C 1 ⋅ n l o g n for a suitable constant C 1 C_1 C 1 . Therefore, we focus on bounding the centered deviation sup π ∈ Π ∣ ϕ ( π ) − E [ ϕ ( π ) ] ∣ \sup_{\pi\in\Pi}|\phi(\pi)-\mathbb{E}[\phi(\pi)]| sup π ∈ Π ∣ ϕ ( π ) − E [ ϕ ( π )] ∣ . In that regard, let us define D ^ 1 ( π ) = 1 ∣ D 1 ∣ ∑ i ∈ D 1 ∣ π ( X i ) − π b ⋆ ( X i ) ∣ \hat{D}_1(\pi)=\frac{1}{|D_1|}\sum_{i\in D_1}|\pi(X_i)-\pi_b^\star(X_i)| D ^ 1 ( π ) = ∣ D 1 ∣ 1 ∑ i ∈ D 1 ∣ π ( X i ) − π b ⋆ ( X i ) ∣ to be the empirical soft disagreement between policies π \pi π and π b ⋆ \pi_b^\star π b ⋆ measured on the dataset D 1 D_1 D 1 . Additionally, E 1 \mathcal{E}_1 E 1 to be the following event.
E 1 = { sup π ∈ Π ∣ D ^ 1 ( π ) − D ( π ) ∣ ≤ ε ˉ 1 } where ε ˉ 1 = O ( d log ( L B n ) + log ( 1 / δ ) n ) . \mathcal{E}_1=\left\{\sup_{\pi\in\Pi}|\hat{D}_1(\pi)-D(\pi)|\le\bar{\varepsilon}_1\right\}\qquad\text{where}\qquad\bar{\varepsilon}_1=O\left(\sqrt{\frac{d\log(LBn)+\log(1/\delta)}{n}}\right). E 1 = { π ∈ Π sup ∣ D ^ 1 ( π ) − D ( π ) ∣ ≤ ε ˉ 1 } where ε ˉ 1 = O ( n d log ( L B n ) + log ( 1/ δ ) ) .
When assumption 4 holds, Lemma 18 applies to general divergence function and shows that E 1 \mathcal{E}_1 E 1 holds with probability at least 1 − δ 1-\delta 1 − δ . We will now condition on the event that E 1 \mathcal{E}_1 E 1 holds. As in the proof of Theorem 4 let us fix Δ ∈ ( 0 , 1 ) \Delta\in(0,1) Δ ∈ ( 0 , 1 ) (to be chosen later) and define the shells of policies
Π j = { π ∈ Π : 2 j − 1 Δ < D ( π ) ≤ 2 j Δ } s j = 2 j Δ for j = 1 , 2 , … , J : = ⌈ log 2 ( 1 / Δ ) ⌉ . \Pi_j=\left\{\pi\in\Pi:2^{j-1}\Delta<D(\pi)\le2^j\Delta\right\}\qquad s_j=2^j\Delta\quad\text{for }j=1,2,\ldots,J:=\left\lceil\log_2(1/\Delta)\right\rceil. Π j = { π ∈ Π : 2 j − 1 Δ < D ( π ) ≤ 2 j Δ } s j = 2 j Δ for j = 1 , 2 , … , J := ⌈ log 2 ( 1/Δ ) ⌉ .
Deviation in Shell j j j : We first prove an upper bound on the deviation ∣ ϕ ( π ) − E [ ϕ ( π ) ] ∣ |\phi(\pi)-\mathbb{E}[\phi(\pi)]| ∣ ϕ ( π ) − E [ ϕ ( π )] ∣ for any policy π \pi π in the shell Π j \Pi_j Π j whose policies are sufficiently close to π b ∗ \pi_b^* π b ∗ (i.e., D ( π ) ≤ s max : = σ 0 2 / ( 8 M 2 ) D(\pi)\le s_{\max}:=\sigma_0^2/(8M^2) D ( π ) ≤ s m a x := σ 0 2 / ( 8 M 2 ) for any π ∈ Π j \pi\in\Pi_j π ∈ Π j ).
Consider two policies π , π ~ ∈ Π j \pi,\widetilde{\pi}\in\Pi_j π , π ∈ Π j with D ( π ) , D ( π ~ ) ≤ s max D(\pi),D(\widetilde{\pi})\le s_{\max} D ( π ) , D ( π ) ≤ s m a x . Then ϕ ( π ) − ϕ ( π ~ ) = A ^ n , χ 2 e s t ( π ~ ) − A ^ n , χ 2 e s t ( π ) \phi(\pi)-\phi(\widetilde{\pi})=\widehat{A}^{\mathrm{est}}_{n,\chi^2}(\widetilde{\pi})-\widehat{A}^{\mathrm{est}}_{n,\chi^2}(\pi) ϕ ( π ) − ϕ ( π ) = A n , χ 2 est ( π ) − A n , χ 2 est ( π ) . By the definition of A ^ n , χ 2 e s t \widehat{A}^{\mathrm{est}}_{n,\chi^2} A n , χ 2 est , this can be written as an average of n n n i.i.d. terms (conditional on D 1 D_1 D 1 ):
ϕ ( π ) − ϕ ( π ~ ) = 1 n ∑ i ∈ D 4 ξ i ( π , π ~ ) where ξ i ( π , π ~ ) = g λ ^ ( π ~ ) , η ^ ( π ~ ) ( Z i ( π ~ ) ) − g λ ^ ( π ) , η ^ ( π ) ( Z i ( π ) ) \phi(\pi)-\phi(\widetilde{\pi})=\frac{1}{n}\sum_{i\in D_4}\xi_i(\pi,\widetilde{\pi})\qquad\text{where }\xi_i(\pi,\widetilde{\pi})=g_{\widehat{\lambda}(\widetilde{\pi}),\widehat{\eta}(\widetilde{\pi})}(Z_i(\widetilde{\pi}))-g_{\widehat{\lambda}(\pi),\widehat{\eta}(\pi)}(Z_i(\pi)) ϕ ( π ) − ϕ ( π ) = n 1 i ∈ D 4 ∑ ξ i ( π , π ) where ξ i ( π , π ) = g λ ( π ) , η ( π ) ( Z i ( π )) − g λ ( π ) , η ( π ) ( Z i ( π ))
where Z i ( π ) = ( 2 π ( X i ) − 1 ) Γ ^ i Z_i(\pi)=(2\pi(X_i)-1)\widehat{\Gamma}_i Z i ( π ) = ( 2 π ( X i ) − 1 ) Γ i . We can decompose ξ i ( π , π ~ ) \xi_i(\pi,\widetilde{\pi}) ξ i ( π , π ) as follows.
ξ i ( π , π ~ ) = g λ ^ ( π ~ ) , η ^ ( π ~ ) ( Z i ( π ~ ) ) − g λ ^ ( π ~ ) , η ^ ( π ~ ) ( Z i ( π ) ) + g λ ^ ( π ~ ) , η ^ ( π ~ ) ( Z i ( π ) ) − g λ ^ ( π ) , η ^ ( π ) ( Z i ( π ) ) . \begin{aligned}
\xi_i(\pi,\widetilde{\pi})={}&g_{\widehat{\lambda}(\widetilde{\pi}),\widehat{\eta}(\widetilde{\pi})}(Z_i(\widetilde{\pi}))-g_{\widehat{\lambda}(\widetilde{\pi}),\widehat{\eta}(\widetilde{\pi})}(Z_i(\pi))\\
&+g_{\widehat{\lambda}(\widetilde{\pi}),\widehat{\eta}(\widetilde{\pi})}(Z_i(\pi))-g_{\widehat{\lambda}(\pi),\widehat{\eta}(\pi)}(Z_i(\pi)).
\end{aligned} ξ i ( π , π ) = g λ ( π ) , η ( π ) ( Z i ( π )) − g λ ( π ) , η ( π ) ( Z i ( π )) + g λ ( π ) , η ( π ) ( Z i ( π )) − g λ ( π ) , η ( π ) ( Z i ( π )) .
The proof of Lemma 30 shows that ∣ ∂ g λ , η ( z ) ∂ z ∣ ≤ 4 M κ λ 0 \left|\frac{\partial g_{\lambda,\eta}(z)}{\partial z}\right|\le\frac{4M}{\kappa\lambda_0} ∂ z ∂ g λ , η ( z ) ≤ κ λ 0 4 M . Therefore,
T 1 ≤ 4 M κ λ 0 ∣ Z i ( π ) − Z i ( π ~ ) ∣ ≤ 8 M 2 κ 2 λ 0 ∣ π ( X i ) − π ~ ( X i ) ∣ . T_1\le\frac{4M}{\kappa\lambda_0}|Z_i(\pi)-Z_i(\widetilde{\pi})|\le\frac{8M^2}{\kappa^2\lambda_0}|\pi(X_i)-\widetilde{\pi}(X_i)|. T 1 ≤ κ λ 0 4 M ∣ Z i ( π ) − Z i ( π ) ∣ ≤ κ 2 λ 0 8 M 2 ∣ π ( X i ) − π ( X i ) ∣.
The proof of Lemma 30 also shows ∥ ∇ λ , η g λ , η ( z ) ∥ ≤ 16 M 2 κ 2 λ 0 2 \|\nabla_{\lambda,\eta}g_{\lambda,\eta}(z)\|\le\frac{16M^2}{\kappa^2\lambda_0^2} ∥ ∇ λ , η g λ , η ( z ) ∥ ≤ κ 2 λ 0 2 16 M 2 for all λ ≥ λ 0 / 2 \lambda\ge\lambda_0/2 λ ≥ λ 0 /2 , ∣ η ∣ ≤ M |\eta|\le M ∣ η ∣ ≤ M and ∣ z ∣ ≤ M / κ |z|\le M/\kappa ∣ z ∣ ≤ M / κ . Therefore,
T 2 ≤ 16 M 2 κ 2 λ 0 2 ∥ θ ^ ( π ) − θ ^ ( π ~ ) ∥ 2 ≤ C M 9 κ 6 λ 0 4 ρ 3 / σ 0 2 1 ∣ D 1 ∣ ∑ j ∈ D 1 ∣ π ( X j ) − π ~ ( X j ) ∣ \begin{aligned}
T_2&\le\frac{16M^2}{\kappa^2\lambda_0^2}\|\widehat{\theta}(\pi)-\widehat{\theta}(\widetilde{\pi})\|_2\\
&\le\frac{CM^9}{\kappa^6\lambda_0^4\rho^3/\sigma_0^2}\frac{1}{|D_1|}\sum_{j\in D_1}|\pi(X_j)-\widetilde{\pi}(X_j)|
\end{aligned} T 2 ≤ κ 2 λ 0 2 16 M 2 ∥ θ ( π ) − θ ( π ) ∥ 2 ≤ κ 6 λ 0 4 ρ 3 / σ 0 2 C M 9 ∣ D 1 ∣ 1 j ∈ D 1 ∑ ∣ π ( X j ) − π ( X j ) ∣
by Lemma 29 as long as n ≥ Λ 1 ( δ ) : = max { O ~ ( M 14 κ 8 σ 0 10 σ 0 4 ( d + log ( 1 / δ ) ) ) , O ( ( M κ σ 0 2 ) 1 / ( β + γ ) ) } n\ge\Lambda_1(\delta):=\max\left\{\widetilde{O}\left(\frac{M^{14}}{\kappa^8\sigma_0^{10}\sigma_0^4}(d+\log(1/\delta))\right),O\left(\left(\frac{M}{\kappa\sigma_0^2}\right)^{1/(\beta+\gamma)}\right)\right\} n ≥ Λ 1 ( δ ) := max { O ( κ 8 σ 0 10 σ 0 4 M 14 ( d + log ( 1/ δ )) ) , O ( ( κ σ 0 2 M ) 1/ ( β + γ ) ) } .
Substituting the bounds on T 1 T_1 T 1 and T 2 T_2 T 2 we obtain the following bound.
∣ ϕ ( π ) − ϕ ( π ~ ) ∣ ≤ C M 9 κ 6 λ 0 4 ρ 3 / σ 0 2 ⏟ : = C ℓ ∥ π − π ~ ∥ ∞ |\phi(\pi)-\phi(\widetilde{\pi})|\le\underbrace{\frac{CM^9}{\kappa^6\lambda_0^4\rho^3/\sigma_0^2}}_{:=C_\ell}\|\pi-\widetilde{\pi}\|_\infty ∣ ϕ ( π ) − ϕ ( π ) ∣ ≤ := C ℓ κ 6 λ 0 4 ρ 3 / σ 0 2 C M 9 ∥ π − π ∥ ∞
Now proceeding very similar to the proof of Theorem 4 we can establish
∣ ϕ ( π ) − E [ ϕ ( π ) ] ∣ ≤ 2 C ℓ ε j + 2 C v u j 3 n + 2 C v 4 s j + ε 1 2 n ⋅ u j |\phi(\pi)-\mathbb{E}[\phi(\pi)]|\le2C_\ell\varepsilon_j+\frac{2C_vu_j}{3n}+2C_v\sqrt{\frac{4s_j+\varepsilon_1^2}{n}}\cdot u_j ∣ ϕ ( π ) − E [ ϕ ( π )] ∣ ≤ 2 C ℓ ε j + 3 n 2 C v u j + 2 C v n 4 s j + ε 1 2 ⋅ u j
with probability at least 1 − 2 N j e − u j 1-2\mathcal{N}_j e^{-u_j} 1 − 2 N j e − u j . Here C v C_v C v is a constant defined in Lemma 30 and N j \mathcal{N}_j N j is ε j \varepsilon_j ε j -covering number of Π j \Pi_j Π j .
Peeling Argument: Theorem 4 shows that if π ^ n ∈ Π j \widehat{\pi}_n\in\Pi_j π n ∈ Π j with s j > s max : = σ 0 2 / ( 8 M 2 ) s_j>s_{\max}:=\sigma_0^2/(8M^2) s j > s m a x := σ 0 2 / ( 8 M 2 ) , then
ε ( π ^ n ) > 1 4 ( 1 + α ) / α ⋅ η ⋅ ( σ 0 2 8 M 2 ) 1 + α α . \varepsilon(\widehat{\pi}_n)>\frac{1}{4(1+\alpha)/\alpha}\cdot\eta\cdot\left(\frac{\sigma_0^2}{8M^2}\right)^{\frac{1+\alpha}{\alpha}}. ε ( π n ) > 4 ( 1 + α ) / α 1 ⋅ η ⋅ ( 8 M 2 σ 0 2 ) α 1 + α .
However, using Lemma 31 and Theorem 1 we have,
2 ( 1 − 2 2 ρ M σ 0 ) ε ( π ) ≤ A χ 2 ( π b ∗ ) − A χ 2 ( π ^ n ) ≤ 4 M κ n − β − γ + 4 M d log ( L B n ) + log ( 1 / δ ) n 2\left(1-\frac{2\sqrt{2}\rho M}{\sigma_0}\right)\varepsilon(\pi)\le A_{\chi^2}(\pi_b^*)-A_{\chi^2}(\widehat{\pi}_n)\le\frac{4M}{\kappa}n^{-\beta-\gamma}+4M\sqrt{\frac{d\log(LBn)+\log(1/\delta)}{n}} 2 ( 1 − σ 0 2 2 ρM ) ε ( π ) ≤ A χ 2 ( π b ∗ ) − A χ 2 ( π n ) ≤ κ 4 M n − β − γ + 4 M n d log ( L B n ) + log ( 1/ δ )
Therefore, as long as ρ ≤ σ 0 4 2 M \rho\le\frac{\sigma_0}{4\sqrt{2M}} ρ ≤ 4 2 M σ 0 and
n ≥ Λ 2 ( δ ) : = max { 16 2 + 1 α M 2 η 2 ( 8 M 2 σ 0 2 ) 2 ( 1 + α ) α ( d log ( L B n ) + log ( 1 / δ ) ) , ( 4 2 + 1 α M κ η ⋅ ( 8 M 2 σ 0 2 ) α + 1 α ) 1 β + γ } n \ge\Lambda_2(\delta) := \max\left\{ \frac{16^{2+\frac{1}{\alpha}}M^2}{\eta^2}\left(\frac{8M^2}{\sigma_0^2}\right)^{\frac{2(1+\alpha)}{\alpha}}\left(d\log(LBn)+\log(1/\delta)\right),\left(\frac{4^{2+\frac{1}{\alpha}}M}{\kappa\eta}\cdot\left(\frac{8M^2}{\sigma_0^2}\right)^{\frac{\alpha+1}{\alpha}}\right)^{\frac{1}{\beta+\gamma}}\right\} n ≥ Λ 2 ( δ ) := max ⎩ ⎨ ⎧ η 2 1 6 2 + α 1 M 2 ( σ 0 2 8 M 2 ) α 2 ( 1 + α ) ( d log ( L B n ) + log ( 1/ δ ) ) , ( κ η 4 2 + α 1 M ⋅ ( σ 0 2 8 M 2 ) α α + 1 ) β + γ 1 ⎭ ⎬ ⎫
we have ϵ ( π ^ n ) < 1 4 ( 1 + α ) / α ⋅ η ⋅ ( σ 0 2 8 M 2 ) 1 + α α \epsilon(\hat{\pi}_n) < \frac{1}{4^{(1+\alpha)/\alpha}}\cdot\eta\cdot\left(\frac{\sigma_0^2}{8M^2}\right)^{\frac{1+\alpha}{\alpha}} ϵ ( π ^ n ) < 4 ( 1 + α ) / α 1 ⋅ η ⋅ ( 8 M 2 σ 0 2 ) α 1 + α , which is a contradiction. Therefore, π ^ n \hat{\pi}_n π ^ n cannot lie in a shell j j j with s j > s max s_j > s_{\max} s j > s m a x .
The rest of the argument is almost identical to the proof of Theorem 4. We avoid the repetition and present the final bound.
A χ 2 ( π b ∗ ) − A χ 2 ( π ^ n ) ≤ 6 M κ n − ( β + γ ) + C 1 ⋅ log n n + 10 C v ( 5 2 n ) 1 + α α ( α α + 2 ) 1 + α α + 2 ( Q n ) 1 + α α + 2 + 2 C v Q 3 n , A_{\chi^2}(\pi_b^*)-A_{\chi^2}(\hat{\pi}_n)\le\frac{6M}{\kappa}n^{-(\beta+\gamma)}+C_1\cdot\frac{\log n}{n}+10C_v\left(\frac{5}{2n}\right)^{\frac{1+\alpha}{\alpha}}\left(\frac{\alpha}{\alpha+2}\right)^{\frac{1+\alpha}{\alpha+2}}\left(\frac{Q}{n}\right)^{\frac{1+\alpha}{\alpha+2}}+\frac{2C_vQ}{3n}, A χ 2 ( π b ∗ ) − A χ 2 ( π ^ n ) ≤ κ 6 M n − ( β + γ ) + C 1 ⋅ n log n + 10 C v ( 2 n 5 ) α 1 + α ( α + 2 α ) α + 2 1 + α ( n Q ) α + 2 1 + α + 3 n 2 C v Q ,
where Q = d log ( L B n C ℓ / C v ) + log ( ( log n ) / δ ) Q=d\log(LBnC_\ell/C_v)+\log((\log n)/\delta) Q = d log ( L B n C ℓ / C v ) + log (( log n ) / δ ) .
Lemma 30. Suppose n ≥ max { O ~ ( M 14 κ 8 ρ 3 λ 0 σ 0 4 ⋅ ( d + log ( 1 / δ ) ) ) , O ( M κ σ 0 2 ) 1 / ( β + γ ) } n \ge\max\left\{\widetilde{O}\left(\frac{M^{14}}{\kappa^8\rho^3\lambda_0\sigma_0^4}\cdot(d+\log(1/\delta))\right),O\left(\frac{M}{\kappa\sigma_0^2}\right)^{1/(\beta+\gamma)}\right\} n ≥ max { O ( κ 8 ρ 3 λ 0 σ 0 4 M 14 ⋅ ( d + log ( 1/ δ )) ) , O ( κ σ 0 2 M ) 1/ ( β + γ ) } . On the event E 1 \mathcal{E}_1 E 1 , for any π ∈ Π \pi\in\Pi π ∈ Π with D ( π ) ≤ s max = σ 0 2 8 M 2 D(\pi)\le s_{\max}=\frac{\sigma_0^2}{8M^2} D ( π ) ≤ s m a x = 8 M 2 σ 0 2 , conditional on D 1 D_1 D 1 :
P D 4 ( ∣ ϕ ( π ) − E D 4 [ ϕ ( π ) ] ∣ ≥ t ∣ D 1 ) ≤ 2 exp ( − n t 2 / 2 C v 2 ( 4 D ( π ) + ε 1 2 ) + C v t / 3 ) . P_{D_4}\left(\left|\phi(\pi)-\mathbb{E}_{D_4}[\phi(\pi)]\right|\ge t\mid D_1\right)\le2\exp\left(-\frac{nt^2/2}{C_v^2(4D(\pi)+\varepsilon_1^2)+C_vt/3}\right). P D 4 ( ∣ ϕ ( π ) − E D 4 [ ϕ ( π )] ∣ ≥ t ∣ D 1 ) ≤ 2 exp ( − C v 2 ( 4 D ( π ) + ε 1 2 ) + C v t /3 n t 2 /2 ) . where C v = 2 16 M 9 κ 6 λ 0 3 ρ 3 / 2 σ 0 2 C_v=\frac{2^{16}M^9}{\kappa^6\lambda_0^3\rho^{3/2}\sigma_0^2} C v = κ 6 λ 0 3 ρ 3/2 σ 0 2 2 16 M 9 .
Proof. Let us define g λ , η ( z ) = − 1 2 λ ( z + η ) 2 − λ ρ g_{\lambda,\eta}(z)=-\frac{1}{2\lambda}(z+\eta)^2-\lambda\rho g λ , η ( z ) = − 2 λ 1 ( z + η ) 2 − λ ρ , and Z i ( π ) = ( 2 π ( X i ) − 1 ) Γ ^ i Z_i(\pi)=(2\pi(X_i)-1)\hat{\Gamma}_i Z i ( π ) = ( 2 π ( X i ) − 1 ) Γ ^ i . Then, we can write
ϕ ( π ) = 1 n ∑ i ∈ D 4 ξ i ( π ) where ξ i ( π ) = g λ ^ ( π b ∗ ) , η ^ ( π b ∗ ) ( Z i ( π b ∗ ) ) − g λ ^ ( π ) , η ^ ( π ) ( Z i ( π ) ) . \phi(\pi)=\frac{1}{n}\sum_{i\in D_4}\xi_i(\pi)\ \text{ where }\ \xi_i(\pi)=g_{\hat{\lambda}(\pi_b^*),\hat{\eta}(\pi_b^*)}(Z_i(\pi_b^*))-g_{\hat{\lambda}(\pi),\hat{\eta}(\pi)}(Z_i(\pi)). ϕ ( π ) = n 1 i ∈ D 4 ∑ ξ i ( π ) where ξ i ( π ) = g λ ^ ( π b ∗ ) , η ^ ( π b ∗ ) ( Z i ( π b ∗ )) − g λ ^ ( π ) , η ^ ( π ) ( Z i ( π )) .
We will apply Bernstein’s inequality to bound the deviation of ϕ ( π ) \phi(\pi) ϕ ( π ) from its mean. In that regard, we first bound the variance of each summand ξ i ( π ) \xi_i(\pi) ξ i ( π ) conditional on D 1 D_1 D 1 .
ξ i ( π ) = g λ ^ ( π b ∗ ) , η ^ ( π b ∗ ) ( Z i ( π b ∗ ) ) − g λ ^ ( π b ∗ ) , η ^ ( π b ∗ ) ( Z i ( π ) ) + g λ ^ ( π b ∗ ) , η ^ ( π b ∗ ) ( Z i ( π ) ) − g λ ^ ( π ) , η ^ ( π ) ( Z i ( π ) ) \begin{aligned}
\xi_i(\pi)={}&g_{\hat{\lambda}(\pi_b^*),\hat{\eta}(\pi_b^*)}(Z_i(\pi_b^*))-g_{\hat{\lambda}(\pi_b^*),\hat{\eta}(\pi_b^*)}(Z_i(\pi))\\
&+g_{\hat{\lambda}(\pi_b^*),\hat{\eta}(\pi_b^*)}(Z_i(\pi))-g_{\hat{\lambda}(\pi),\hat{\eta}(\pi)}(Z_i(\pi))
\end{aligned} ξ i ( π ) = g λ ^ ( π b ∗ ) , η ^ ( π b ∗ ) ( Z i ( π b ∗ )) − g λ ^ ( π b ∗ ) , η ^ ( π b ∗ ) ( Z i ( π )) + g λ ^ ( π b ∗ ) , η ^ ( π b ∗ ) ( Z i ( π )) − g λ ^ ( π ) , η ^ ( π ) ( Z i ( π ))
Since g λ , η ( z ) = − 1 2 λ ( z + η ) 2 − λ ρ g_{\lambda,\eta}(z)=-\frac{1}{2\lambda}(z+\eta)^2-\lambda\rho g λ , η ( z ) = − 2 λ 1 ( z + η ) 2 − λ ρ , it’s derivative with respect to z z z is − 1 λ ( z + η ) -\frac{1}{\lambda}(z+\eta) − λ 1 ( z + η ) . Therefore,
T 1 ≤ 4 M κ λ 0 ⋅ ∣ Z i ( π ) − Z i ( π b ∗ ) ∣ ≤ 8 M κ λ 0 ⋅ ∣ π ( X i ) − π b ∗ ( X i ) ∣ ∣ Γ ^ i ∣ ≤ 16 M 2 κ 2 λ 0 2 ∣ π ( X i ) − π b ∗ ( X i ) ∣ . T_1\le\frac{4M}{\kappa\lambda_0}\cdot\left|Z_i(\pi)-Z_i(\pi_b^*)\right|\le\frac{8M}{\kappa\lambda_0}\cdot\left|\pi(X_i)-\pi_b^*(X_i)\right|\left|\hat{\Gamma}_i\right|\le\frac{16M^2}{\kappa^2\lambda_0^2}\left|\pi(X_i)-\pi_b^*(X_i)\right|. T 1 ≤ κ λ 0 4 M ⋅ ∣ Z i ( π ) − Z i ( π b ∗ ) ∣ ≤ κ λ 0 8 M ⋅ ∣ π ( X i ) − π b ∗ ( X i ) ∣ Γ ^ i ≤ κ 2 λ 0 2 16 M 2 ∣ π ( X i ) − π b ∗ ( X i ) ∣ .
Taking expectation with respect to X i ∈ D 4 X_i\in D_4 X i ∈ D 4 and using the fact that ∣ π ( X i ) − π b ∗ ( X i ) ∣ ∈ [ 0 , 1 ] \left|\pi(X_i)-\pi_b^*(X_i)\right|\in[0,1] ∣ π ( X i ) − π b ∗ ( X i ) ∣ ∈ [ 0 , 1 ] , we obtain the following bound.
E [ T 1 2 ∣ D 1 ] ≤ 256 M 4 κ 4 λ 0 4 ⋅ D ( π ) . \mathbb{E}[T_1^2\mid D_1]\le\frac{256M^4}{\kappa^4\lambda_0^4}\cdot D(\pi). E [ T 1 2 ∣ D 1 ] ≤ κ 4 λ 0 4 256 M 4 ⋅ D ( π ) .
In order to bound T 2 T_2 T 2 , we first bound the partial derivatives of g λ , η ( z ) g_{\lambda,\eta}(z) g λ , η ( z ) with respect to λ \lambda λ and η \eta η . By direct calculation:
∂ g λ , η ( z ) ∂ λ = − ρ + 1 2 λ 2 ( z + η ) 2 , ∂ g λ , η ( z ) ∂ η = − 1 λ ( z + η ) . \frac{\partial g_{\lambda,\eta}(z)}{\partial\lambda}=-\rho+\frac{1}{2\lambda^2}(z+\eta)^2,\qquad\frac{\partial g_{\lambda,\eta}(z)}{\partial\eta}=-\frac{1}{\lambda}(z+\eta). ∂ λ ∂ g λ , η ( z ) = − ρ + 2 λ 2 1 ( z + η ) 2 , ∂ η ∂ g λ , η ( z ) = − λ 1 ( z + η ) .
which gives us ∥ ∇ ( λ , η ) g λ , η ( z ) ∥ ≤ 16 M 2 κ 2 λ 0 2 \left\|\nabla_{(\lambda,\eta)}g_{\lambda,\eta}(z)\right\|\le\frac{16M^2}{\kappa^2\lambda_0^2} ∇ ( λ , η ) g λ , η ( z ) ≤ κ 2 λ 0 2 16 M 2 . Therefore,
T 2 ≤ 16 M 2 κ 2 λ 0 2 ∥ θ ^ ( π b ∗ ) − θ ^ ( π ) ∥ 2 ≤ 2 15 M 9 κ 6 λ 0 3 ρ 3 / 2 σ 0 2 D ^ 1 ( π ) \begin{aligned}
T_2\le\frac{16M^2}{\kappa^2\lambda_0^2}\left\|\hat{\theta}(\pi_b^*)-\hat{\theta}(\pi)\right\|_2 \\
&\le\frac{2^{15}M^9}{\kappa^6\lambda_0^3\rho^{3/2}\sigma_0^2}\widehat{D}_1(\pi)
\end{aligned} T 2 ≤ κ 2 λ 0 2 16 M 2 θ ^ ( π b ∗ ) − θ ^ ( π ) 2 ≤ κ 6 λ 0 3 ρ 3/2 σ 0 2 2 15 M 9 D 1 ( π )
by Lemma 16 as long as n ≥ max { O ~ ( M 14 κ 8 ρ 0 3 λ 0 4 σ 0 4 ⋅ ( d + log ( 1 / δ ) ) ) , O ( M κ σ 0 2 ) 1 / ( β + γ ) } n \ge\max\left\{\widetilde{O}\left(\frac{M^{14}}{\kappa^{8}\rho_{0}^{3}\lambda_{0}^{4}\sigma_{0}^{4}}\cdot(d+\log(1/\delta))\right),O\left(\frac{M}{\kappa\sigma_{0}^{2}}\right)^{1/(\beta+\gamma)}\right\} n ≥ max { O ( κ 8 ρ 0 3 λ 0 4 σ 0 4 M 14 ⋅ ( d + log ( 1/ δ )) ) , O ( κ σ 0 2 M ) 1/ ( β + γ ) } . Therefore,
E [ T 2 2 ∣ D 1 ] ≤ 2 30 M 18 κ 12 λ 0 10 ρ 0 3 σ 0 4 D ^ 1 ( π ) 2 \mathbb{E}[T_{2}^{2}\mid D_{1}]\le\frac{2^{30}M^{18}}{\kappa^{12}\lambda_{0}^{10}\rho_{0}^{3}\sigma_{0}^{4}}\widehat{D}_{1}(\pi)^{2} E [ T 2 2 ∣ D 1 ] ≤ κ 12 λ 0 10 ρ 0 3 σ 0 4 2 30 M 18 D 1 ( π ) 2
When event E 1 \mathcal{E}_{1} E 1 holds, D ^ 1 ( π ) ≤ D ( π ) + ε ‾ 1 \widehat{D}_{1}(\pi)\le D(\pi)+\overline{\varepsilon}_{1} D 1 ( π ) ≤ D ( π ) + ε 1 , which gives us D ^ 1 ( π ) 2 ≤ D ( π ) 2 + 2 D ( π ) ε ‾ 1 + ε ‾ 1 2 ≤ 3 D ( π ) + ε ‾ 1 2 \widehat{D}_{1}(\pi)^{2}\le D(\pi)^{2}+2D(\pi)\overline{\varepsilon}_{1}+\overline{\varepsilon}_{1}^{2}\le3D(\pi)+\overline{\varepsilon}_{1}^{2} D 1 ( π ) 2 ≤ D ( π ) 2 + 2 D ( π ) ε 1 + ε 1 2 ≤ 3 D ( π ) + ε 1 2 . Combining the bounds for T 1 T_{1} T 1 and T 2 T_{2} T 2 , we have
Var ( ξ i ( π ) ∣ D 1 ) ≤ 2 E [ T 1 2 + T 2 2 ∣ D 1 ] ≤ 2 31 M 18 κ 12 λ 0 10 ρ 0 3 σ 0 4 ( 4 D ( π ) + ε ‾ 1 2 ) \operatorname{Var}(\xi_{i}(\pi)\mid D_{1})\le2\mathbb{E}[T_{1}^{2}+T_{2}^{2}\mid D_{1}]\le\frac{2^{31}M^{18}}{\kappa^{12}\lambda_{0}^{10}\rho_{0}^{3}\sigma_{0}^{4}}\left(4D(\pi)+\overline{\varepsilon}_{1}^{2}\right) Var ( ξ i ( π ) ∣ D 1 ) ≤ 2 E [ T 1 2 + T 2 2 ∣ D 1 ] ≤ κ 12 λ 0 10 ρ 0 3 σ 0 4 2 31 M 18 ( 4 D ( π ) + ε 1 2 )
It can be also verified that ∣ ξ i ( π ) ∣ ≤ C v |\xi_{i}(\pi)|\le C_{v} ∣ ξ i ( π ) ∣ ≤ C v . Therefore, an application of Bernstein’s inequality to the sample average ϕ ( π ) = 1 n ∑ i ∈ D 4 ξ i ( π ) \phi(\pi)=\frac{1}{n}\sum_{i\in D_{4}}\xi_{i}(\pi) ϕ ( π ) = n 1 ∑ i ∈ D 4 ξ i ( π ) gives us the following bound.
P D 4 ( ∣ ϕ ( π ) − E D 4 [ ϕ ( π ) ] ∣ ≥ t ∣ D 1 ) ≤ 2 exp ( − n t 2 / 2 C v 2 ( 4 D ( π ) + ε ‾ 1 2 ) + C v t / 3 ) . \mathbb{P}_{D_{4}}\left(|\phi(\pi)-\mathbb{E}_{D_{4}}[\phi(\pi)]|\ge t\mid D_{1}\right)\le2\exp\left(-\frac{nt^{2}/2}{C_{v}^{2}(4D(\pi)+\overline{\varepsilon}_{1}^{2})+C_{v}t/3}\right). P D 4 ( ∣ ϕ ( π ) − E D 4 [ ϕ ( π )] ∣ ≥ t ∣ D 1 ) ≤ 2 exp ( − C v 2 ( 4 D ( π ) + ε 1 2 ) + C v t /3 n t 2 /2 ) .
Lemma 31. Suppose Assumption 8 holds. For any policy π \pi π , V χ 2 ( π b ∗ ) − V χ 2 ( π ) ≥ 2 ε ( π ) ( 1 − 2 2 ρ ⋅ M − E [ ∣ τ ( X ) ∣ ] σ 0 ) V_{\chi^{2}}(\pi_{b}^{*})-V_{\chi^{2}}(\pi)\ge2\varepsilon(\pi)\left(1-2\sqrt{2\rho}\cdot\frac{M-\mathbb{E}[|\tau(X)|]}{\sigma_{0}}\right) V χ 2 ( π b ∗ ) − V χ 2 ( π ) ≥ 2 ε ( π ) ( 1 − 2 2 ρ ⋅ σ 0 M − E [ ∣ τ ( X ) ∣ ] ) .
Proof. Using lemma 22 we can write the following identity for any policy π \pi π .
V χ 2 ( π b ∗ ) − V χ 2 ( π ) = E X ∼ P 0 [ h π b ∗ ( X ) ] − E X ∼ P 0 [ h π ( X ) ] − 2 ρ ⋅ Var X ∼ P 0 ( h π b ∗ ( X ) ) + 2 ρ ⋅ Var X ∼ P 0 ( h π ( X ) ) V_{\chi^{2}}(\pi_{b}^{*})-V_{\chi^{2}}(\pi)=\mathbb{E}_{X\sim P_{0}}[h_{\pi_{b}^{*}}(X)]-\mathbb{E}_{X\sim P_{0}}[h_{\pi}(X)]-\sqrt{2\rho}\cdot\operatorname{Var}_{X\sim P_{0}}(h_{\pi_{b}^{*}}(X))+\sqrt{2\rho}\cdot\operatorname{Var}_{X\sim P_{0}}(h_{\pi}(X)) V χ 2 ( π b ∗ ) − V χ 2 ( π ) = E X ∼ P 0 [ h π b ∗ ( X )] − E X ∼ P 0 [ h π ( X )] − 2 ρ ⋅ Var X ∼ P 0 ( h π b ∗ ( X )) + 2 ρ ⋅ Var X ∼ P 0 ( h π ( X ))
By Lemma 20 we know that 2 ε ( π ) = E [ h π b ∗ ( X ) ] − E [ h π ( X ) ] 2\varepsilon(\pi)=\mathbb{E}[h_{\pi_{b}^{*}}(X)]-\mathbb{E}[h_{\pi}(X)] 2 ε ( π ) = E [ h π b ∗ ( X )] − E [ h π ( X )] . We now establish a lower bound on the difference Var X ∼ P 0 ( h π ( X ) ) − Var X ∼ P 0 ( h π b ∗ ( X ) ) \sqrt{\operatorname{Var}_{X\sim P_{0}}(h_{\pi}(X))}-\sqrt{\operatorname{Var}_{X\sim P_{0}}(h_{\pi_{b}^{*}}(X))} Var X ∼ P 0 ( h π ( X )) − Var X ∼ P 0 ( h π b ∗ ( X )) .
Var X ∼ P 0 ( h π ( X ) ) − Var X ∼ P 0 ( h π b ∗ ( X ) ) = Var X ∼ P 0 ( h π ( X ) ) − Var X ∼ P 0 ( h π b ∗ ( X ) ) Var X ∼ P 0 ( h π ( X ) ) + Var X ∼ P 0 ( h π b ∗ ( X ) ) = 1 σ π + σ π b ∗ ( E [ h π 2 ( X ) ] − E [ h π b ∗ 2 ( X ) ] − ( E [ h π ( X ) ] ) 2 + ( E [ h π b ∗ ( X ) ] ) 2 ) \begin{aligned}
\sqrt{\operatorname{Var}_{X\sim P_{0}}(h_{\pi}(X))}-\sqrt{\operatorname{Var}_{X\sim P_{0}}(h_{\pi_{b}^{*}}(X))}=\frac{\operatorname{Var}_{X\sim P_{0}}(h_{\pi}(X))-\operatorname{Var}_{X\sim P_{0}}(h_{\pi_{b}^{*}}(X))}{\sqrt{\operatorname{Var}_{X\sim P_{0}}(h_{\pi}(X))}+\sqrt{\operatorname{Var}_{X\sim P_{0}}(h_{\pi_{b}^{*}}(X))}} \\
&=\frac{1}{\sigma_{\pi}+\sigma_{\pi_{b}^{*}}}\left(\mathbb{E}[h_{\pi}^{2}(X)]-\mathbb{E}[h_{\pi_{b}^{*}}^{2}(X)]-\left(\mathbb{E}[h_{\pi}(X)]\right)^{2}+\left(\mathbb{E}[h_{\pi_{b}^{*}}(X)]\right)^{2}\right)
\end{aligned} Var X ∼ P 0 ( h π ( X )) − Var X ∼ P 0 ( h π b ∗ ( X )) = Var X ∼ P 0 ( h π ( X )) + Var X ∼ P 0 ( h π b ∗ ( X )) Var X ∼ P 0 ( h π ( X )) − Var X ∼ P 0 ( h π b ∗ ( X )) = σ π + σ π b ∗ 1 ( E [ h π 2 ( X )] − E [ h π b ∗ 2 ( X )] − ( E [ h π ( X )] ) 2 + ( E [ h π b ∗ ( X )] ) 2 )
First note that using lemma 20
− ( E [ h π ( X ) ] ) 2 + ( E [ h π b ∗ ( X ) ] ) 2 = E [ h π ( X ) + h π b ∗ ( X ) ] E [ h π b ∗ ( X ) − h π ( X ) ] = 2 ε ( π ) ( − 2 ε ( π ) + 2 E [ ∣ τ ( X ) ∣ ] ) = − 4 ε 2 ( π ) + 4 ε ( π ) E [ ∣ τ ( X ) ∣ ] ≥ − 4 E [ ∣ π b ∗ ( X ) − π ( X ) ∣ 2 τ 2 ( X ) ] + 4 ε ( π ) E [ ∣ τ ( X ) ∣ ] . (51) \begin{aligned}
-\left(\mathbb{E}[h_{\pi}(X)]\right)^{2}+\left(\mathbb{E}[h_{\pi_{b}^{*}}(X)]\right)^{2}=\mathbb{E}[h_{\pi}(X)+h_{\pi_{b}^{*}}(X)]\mathbb{E}[h_{\pi_{b}^{*}}(X)-h_{\pi}(X)] \\
&=2\varepsilon(\pi)(-2\varepsilon(\pi)+2\mathbb{E}[|\tau(X)|]) \\
&=-4\varepsilon^{2}(\pi)+4\varepsilon(\pi)\mathbb{E}[|\tau(X)|] \\
&\ge-4\mathbb{E}\left[|\pi_{b}^{*}(X)-\pi(X)|^{2}\tau^{2}(X)\right]+4\varepsilon(\pi)\mathbb{E}[|\tau(X)|]. \tag*{(51)}
\end{aligned} − ( E [ h π ( X )] ) 2 + ( E [ h π b ∗ ( X )] ) 2 = E [ h π ( X ) + h π b ∗ ( X )] E [ h π b ∗ ( X ) − h π ( X )] = 2 ε ( π ) ( − 2 ε ( π ) + 2 E [ ∣ τ ( X ) ∣ ]) = − 4 ε 2 ( π ) + 4 ε ( π ) E [ ∣ τ ( X ) ∣ ] ≥ − 4 E [ ∣ π b ∗ ( X ) − π ( X ) ∣ 2 τ 2 ( X ) ] + 4 ε ( π ) E [ ∣ τ ( X ) ∣ ] . (51)
Moreover by applying the third result of Lemma 20
E [ h π 2 ( X ) ] − E [ h π b ∗ 2 ( X ) ] = 4 E [ ∣ π b ∗ ( X ) − π ( X ) ∣ 2 ∣ τ ( X ) ∣ 2 ] − 4 E [ ∣ π b ∗ ( X ) − π ( X ) ∣ 2 ∣ τ ( X ) ∣ ] . (52) \mathbb{E}[h_{\pi}^{2}(X)]-\mathbb{E}[h_{\pi_{b}^{*}}^{2}(X)]=4\mathbb{E}\left[|\pi_{b}^{*}(X)-\pi(X)|^{2}|\tau(X)|^{2}\right]-4\mathbb{E}\left[|\pi_{b}^{*}(X)-\pi(X)|^{2}|\tau(X)|\right]. \tag*{(52)} E [ h π 2 ( X )] − E [ h π b ∗ 2 ( X )] = 4 E [ ∣ π b ∗ ( X ) − π ( X ) ∣ 2 ∣ τ ( X ) ∣ 2 ] − 4 E [ ∣ π b ∗ ( X ) − π ( X ) ∣ 2 ∣ τ ( X ) ∣ ] . (52)
Using eq. (51) and eq. (52) we obtain the following lower bound.
Var X ∼ P 0 ( h π ( X ) ) − Var X ∼ P 0 ( h π b ∗ ( X ) ) ≥ 4 ε ( π ) E [ ∣ τ ( X ) ∣ ] − 4 E [ ∣ π b ∗ ( X ) − π ( X ) ∣ ∣ τ ( X ) ∣ 2 ] σ π + σ π b ∗ ≥ 4 ε ( π ) ( E [ ∣ τ ( X ) ∣ ] − M ) σ π + σ π b ∗ \begin{aligned}
\sqrt{\operatorname{Var}_{X\sim P_{0}}(h_{\pi}(X))}-\sqrt{\operatorname{Var}_{X\sim P_{0}}(h_{\pi_{b}^{*}}(X))}\ge\frac{4\varepsilon(\pi)\mathbb{E}[|\tau(X)|]-4\mathbb{E}\left[|\pi_{b}^{*}(X)-\pi(X)||\tau(X)|^{2}\right]}{\sigma_{\pi}+\sigma_{\pi_{b}^{*}}} \\
&\ge\frac{4\varepsilon(\pi)\left(\mathbb{E}[|\tau(X)|]-M\right)}{\sigma_{\pi}+\sigma_{\pi_{b}^{*}}}
\end{aligned} Var X ∼ P 0 ( h π ( X )) − Var X ∼ P 0 ( h π b ∗ ( X )) ≥ σ π + σ π b ∗ 4 ε ( π ) E [ ∣ τ ( X ) ∣ ] − 4 E [ ∣ π b ∗ ( X ) − π ( X ) ∣∣ τ ( X ) ∣ 2 ] ≥ σ π + σ π b ∗ 4 ε ( π ) ( E [ ∣ τ ( X ) ∣ ] − M )
::: This gives us the following lower bound on the difference V χ 2 ( π b ∗ ) − V χ 2 ( π ) V_{\chi^2}(\pi_b^*) - V_{\chi^2}(\pi) V χ 2 ( π b ∗ ) − V χ 2 ( π ) .
V χ 2 ( π b ∗ ) − V χ 2 ( π ) ≥ 2 ε ( π ) ( 1 − 2 2 ρ ⋅ M − E [ ∣ τ ( X ) ∣ ] σ π + σ π b ∗ ) . V_{\chi^2}(\pi_b^*) - V_{\chi^2}(\pi) \ge2\varepsilon(\pi)\left(1 - 2\sqrt{2\rho}\cdot\frac{M-\mathbb{E}[|\tau(X)|]}{\sigma_\pi+\sigma_{\pi_b^*}}\right). V χ 2 ( π b ∗ ) − V χ 2 ( π ) ≥ 2 ε ( π ) ( 1 − 2 2 ρ ⋅ σ π + σ π b ∗ M − E [ ∣ τ ( X ) ∣ ] ) .
Finally, under Assumption 8 we can substitute σ π b ∗ ≥ σ 0 \sigma_{\pi_b^*} \ge\sigma_0 σ π b ∗ ≥ σ 0 .