Introduction
Multi-armed bandits (MABs) provide a fundamental framework for interactive decision-making, in which a policymaker repeatedly selects actions, observes feedback, and adapts its strategy based on past observations. In many modern applications, the policymaker also has access to covariate information that can inform its decisions beyond the feedback collected from previous actions. This naturally motivates the study of contextual MABs, which have become a central framework for adaptive sequential policy designs, with successful applications in healthcare and clinical trials ([3 ] ), online recommendation ([59 ] ) and finance ([9 ] ), among others.
In many applications, designing policies solely to maximise rewards, however, may raise serious ethical or even legal concerns. Empirical evidence from online advertising demonstrates that optimisation-driven delivery systems can produce unequal exposure across demographic groups, even in the absence of explicitly discriminatory targeting ([2 ] ). These concerns are also recognised in broader AI governance frameworks. For instance, the National Institute of Standards and Technology’s AI Risk Management Framework of the U.S. emphasises the importance of identifying and mitigating bias and potential harms to individuals, organisations and society through the design:VEVENT and deployment of AI systems (National Institute of Standards and Technology, [41 ] ). Similarly, China’s Provisions on the Administration of Algorithmic Recommendation for Internet Information Services explicitly prohibit the use of algorithmic recommendations to impose unreasonable differential pricing based on consumers’ preferences and transaction histories (Cyberspace Administration of China et al., [19 ] ). In response, algorithmic fairness has emerged as a critical consideration in both research and practice, motivating substantial efforts to mitigate bias in existing machine learning methods and to embed explicit fairness guarantees into sequential decision-making policies.
Fairness in decision-making has been studied from various perspectives, with two prominent approaches focusing on group fairness and individual fairness. Group fairness concerns disparities across groups defined by sensitive attributes, such as race or gender, and typically requires certain statistical measures, such as selection rates or false positive rates, to be comparable across groups (e.g. Zeng et al., [58 ] ; Hou and Zhang, [29 ] ; Fukuchi and Sakuma, [23 ] ). Individual fairness instead requires individuals who are similar with respect to the task at hand to receive similar treatment, a property not guaranteed by group-level criteria (Dwork et al., [21 ] ).
Individual fairness is particularly relevant to contextual bandit problems, where decisions are made sequentially based on individuals’ covariates. In automated lending, for example, an unconstrained reward-maximising policy may deterministically approve one applicant and reject another with a nearly identical financial profile simply because they lie on opposite sides of a decision boundary. This discontinuity underscores the importance of treating similar individuals similarly, as advocated by the fairness through awareness principle of Dwork et al. [21 ] .
In this paper, we study contextual bandits under a global covariate-based Lipschitz fairness constraint, requiring individuals with similar contexts to receive similar action selection probabilities. Imposing this constraint fundamentally changes both the oracle problem structure and the statistical complexity of bandit learning. Without fairness constraints, the optimal policy in a linear contextual bandit is determined pointwise by the sign of a linear reward gap between the two arms, yielding a parametric learning problem whose minimax regret scales as T \sqrt{T} T up to logarithmic factors (Lattimore and Szepesvári, [33 ] ). Under Lipschitz fairness, by contrast, a policy cannot switch abruptly across the decision boundary. Instead, it must transition smoothly across a neighbourhood of the boundary, and the optimal shape of this transition depends on how covariate probability mass is distributed near the boundary. Consequently, even under a linear reward model, learning an optimal fair policy entails a nonparametric functional optimisation problem along the ( d − 1 ) (d - 1) ( d − 1 ) -dimensional decision boundary. We formalise this phenomenon by characterising the geometry and canonical structure of optimal fair policies, quantifying the population-level cost of fairness, and establishing matching upper and minimax lower regret bounds.
Related literature
To highlight the contribution of this paper, we briefly summarise the literature most closely related to fair bandits and interactive decision making in this subsection.
A prominent strand of the fair bandit literature imposes arm-level allocation constraints, requiring each arm to receive a minimum share of selections. Chen et al. [16 ] study contextual MABs with constraints that guarantee the marginal probability of each arm being pulled is uniformly lower-bounded. Patil et al. [42 ] consider stochastic Bernoulli bandits with per-arm quota constraints, requiring each arm to be pulled at least a prescribed fraction of rounds. In the combinatorial bandit setting, where users can choose a combination of arms per round, Li et al. [34 ] enforce asymptotic minimum pull-proportion constraints using virtual queues, while Xu et al. (2020) impose lower bounds on the empirical selection proportion of each arm throughout the time horizon. Furthermore, Liu et al. [36 ] develop a more general framework for combinatorial bandits with linear constraints, which includes Xu et al. (2020) as a special case. Wu and Li [55 ] consider Bernoulli bandits with minimum average reward requirements for each arm, shifting the requirement from selection frequency. to realised reward. Crucially, these formulations enforce fairness across arms by guaranteeing minimum exposure or reward per action, rather than across individuals with varying covariates.
Another related line of work studies individual fairness through meritocratic or similarity-based constraints on the policy. Joseph et al. [32 ] introduce a meritocratic notion of fairness, requiring that an arm should not be selected with higher probability than another arm unless it has higher expected reward. This meritocratic perspective is further developed by Joseph et al. [31 ] , studying fairness in contextual bandits with infinite convex action sets. In a similar spirit, Wang et al. [52 ] study fairness of exposure, under which the probability of selecting an arm is required to be proportional to an application-dependent merit function of its expected reward. Moving from merit-based exposure to similarity-based constraints, Liu et al. [37 ] study smooth and calibrated fairness in non-contextual MABs, where selection probabilities are constrained by similarities or divergences between reward distributions. Gillen et al. [24 ] consider linear contextual bandits with Lipschitz-type individual fairness constraints. Their constraints compare arm-specific contexts within each single round rather than coupling treatment probabilities globally across individuals arriving over the covariate space.
Other approaches incorporate fairness into the learning objective, rather than imposing it as an explicit constraint. For example, Blanchard and Goyal [7 ] study collaborative bandits using min-max regret as a fairness-aware objective. Similar objective based formulations also include Nash regret in Barman et al. [4 ] and Sawarni et al. [46 ] , and α \alpha α -fair utility functions in Chaudhary and Sinha [11 ] . Group fairness has also been studied in contextual bandits. Grazzi et al. [26 ] consider linear contextual bandits with a group meritocratic fairness constraint based on sensitive-group rankings. This notion is closely related to the meritocratic fairness concepts of Joseph et al. [32 ] and Wang et al. [52 ] , but applies fairness at the level of sensitive groups rather than individual arms. These group-level criteria do not generally ensure similar action probabilities for individuals with similar contexts, as required by our individual fairness constraint.
Beyond bandits, individual fairness has also been studied in other interactive decision-making problems. Gupta and Kamble [27 ] introduce a notion named individual fairness in hindsight, comparing current and past decisions through a one-sided similarity constraint, and establish sublinear regret. Salem et al. [45 ] study envy-freeness across groups and time. In contextual dynamic pricing, Chen et al. [15 ] require similar prices for customers with similar baseline utilities, and characterise optimal fair pricing policies and the regret of demand learning. Although related in spirit, their constraint acts directly on continuous prices, whereas ours acts on selection probabilities over discrete actions, inducing fundamentally different boundary geometry and policy-learning challenges that preclude a direct application of their techniques. Other pricing studies focus on disparities across customer groups and over time. Cohen et al. [18 ] consider price and demand fairness, while Chen et al. [13 ] constrain relative price differences between groups under nonparametric demand. In online resource allocation, Chen et al. [12 ] formulate the problem as a linear program and use the analytic centre of its offline optimal solution set to guide fair online allocations.
In this paper, we focus on individual fairness across contexts, formalised through a global Lipschitz constraint on action selection probabilities in Definition 1, with more discussions in Section 2.1.
List of contributions
We summarise the main contributions of the paper.
To the best of our knowledge, this is the first paper to systematically study contextual bandits under a global covariate-based Lipschitz fairness constraint (Definition 1), extending the individual fairness concepts in Dwork et al. [21 ] to bandit learning. Our framework incorporates similarity-based fairness into contextual bandit problems, requiring the smoothness of the policy with respect to the underlying context space.
domisation leads to a deviation from the unconstrained reward-maximising policy. We use this deviation to explicitly quantify the population-level cost of fairness, i.e. the expected reward loss relative to the unconstrained oracle (Theorem 2 ).
We establish a canonical representation of optimal fair policies (Theorem 3 ), showing how a policy’s value on the decision boundary determines its reward-maximising Lipschitz extension over the entire context space. Moreover, we show that the resulting infinite-dimensional empirical boundary optimisation reduces without relaxation gap to a finite-dimensional linear program. Guided by this structure, we develop a lazy adaptive greedy algorithm for continuous contexts that exploits the fairness induced exploration (Algorithms 1 and 5). For more general context distributions, we develop a lazy polyhedral upper confidence bound (UCB) algorithm with efficient vertex-based optimisation (Algorithms 3 and 6).
We show that both proposed algorithms satisfy the Lipschitz fairness constraint without slackness and attain a cumulative regret bound of order T + T ( d − 2 ) / ( d − 1 ) 1 ( d ≥ 4 ) \sqrt{T}+T^{(d-2)/(d-1)}\mathbf{1}(d\ge4) T + T ( d − 2 ) / ( d − 1 ) 1 ( d ≥ 4 ) , up to poly-logarithmic factors (Theorem 4 and 6 ). The additional dimension-dependent term reflects the complexity of learning over the class of Lipschitz functions defined on the ( d − 1 ) (d-1) ( d − 1 ) -dimensional decision boundary, arising from the Lipschitz fairness constraints. We further establish a matching minimax lower bound (Theorem 7 ), demonstrating the rate optimality of both proposed algorithms and that, whenever d ≥ 4 d\ge4 d ≥ 4 , Lipschitz fairness incurs an unavoidable nonparametric dimension-dependent cost that dominates the intrinsic T \sqrt{T} T parametric bandit complexity.
The rest of the paper is organised as follows. § 2 introduces the problem formulation and the Lipschitz fairness constraint. § 3 characterises the structural properties of optimal fair policies, including the cost of fairness and the canonical boundary representation. § 4 develops fair learning algorithms and establishes their regret upper bounds under continuous and general context distributions. § 5 derives matching minimax lower bounds. § 6 presents numerical experiments on synthetic and real datasets. § 7 concludes. Additional algorithms, technical proofs and supplementary numerical results are provided in the Appendix.
Notation. For any positive integer m m m , denote [ m ] = { 1 , … , m } [m]=\{1,\ldots,m\} [ m ] = { 1 , … , m } . For a ∈ R a\in\mathbb{R} a ∈ R , let ⌈ a ⌉ \lceil a\rceil ⌈ a ⌉ and ⌊ a ⌋ \lfloor a\rfloor ⌊ a ⌋ , and let [ a ] [ 0 , 1 ] = min { 1 , max { 0 , a } } [a]_{[0,1]}=\min\{1,\max\{0,a\}\} [ a ] [ 0 , 1 ] = min { 1 , max { 0 , a }} denote the clipping of a a a onto [ 0 , 1 ] [0,1] [ 0 , 1 ] . For a , b ∈ R a,b\in\mathbb{R} a , b ∈ R , write a ∨ b = max { a , b } a\vee b=\max\{a,b\} a ∨ b = max { a , b } and a ∧ b = min { a , b } a\wedge b=\min\{a,b\} a ∧ b = min { a , b } , and let { a } + = a ∨ 0 \{a\}_{+}=a\vee0 { a } + = a ∨ 0 . For ν ≥ 1 \nu\ge1 ν ≥ 1 and any vector v ∈ R p v\in\mathbb{R}^{p} v ∈ R p , let ∥ v ∥ ν \lVert v\rVert_{\nu} ∥ v ∥ ν denote its ℓ ν \ell_{\nu} ℓ ν -norm, and let ∥ v ∥ ν ∗ \lVert v\rVert_{\nu^{*}} ∥ v ∥ ν ∗ denote its dual norm, where 1 / ν + 1 / ν ∗ = 1 1/\nu+1/\nu^{*}=1 1/ ν + 1/ ν ∗ = 1 , with the convention 1 / ∞ = 0 1/\infty=0 1/∞ = 0 . For a point x x x and any sets A A A and B B B , write dist ν ( x , A ) = inf z ∈ A ∥ x − z ∥ ν \operatorname{dist}_{\nu}(x,A)=\inf_{z\in A}\lVert x-z\rVert_{\nu} dist ν ( x , A ) = inf z ∈ A ∥ x − z ∥ ν , dist ν ( A , B ) = inf y ∈ A , z ∈ B ∥ y − z ∥ ν \operatorname{dist}_{\nu}(A,B)=\inf_{y\in A,z\in B}\lVert y-z\rVert_{\nu} dist ν ( A , B ) = inf y ∈ A , z ∈ B ∥ y − z ∥ ν , and diam ν ( A ) = sup x , z ∈ A ∥ x − z ∥ ν \operatorname{diam}_{\nu}(A)=\sup_{x,z\in A}\lVert x-z\rVert_{\nu} diam ν ( A ) = sup x , z ∈ A ∥ x − z ∥ ν . For any two symmetric matrices A A A and B B B , write A ⪰ B A\succeq B A ⪰ B when A − B A-B A − B is positive semidefinite, and let λ min ( A ) \lambda_{\min}(A) λ m i n ( A ) be the smallest eigenvalue of A A A . For any function f : X → R f:\mathcal{X}\to\mathbb{R} f : X → R and a subset A ⊆ X A\subseteq\mathcal{X} A ⊆ X , denote by f ∣ A : A → R f|_{A}:A\to\mathbb{R} f ∣ A : A → R the restriction of f f f to A A A . For two sequences of positive numbers { a n } \{a_n\} { a n } and { b n } \{b_n\} { b n } , write a n ≲ b n a_n\lesssim b_n a n ≲ b n if there exist a universal constant C > 0 C>0 C > 0 and some n 0 ∈ N n_0\in\mathbb{N} n 0 ∈ N such that a n ≤ C b n a_n\le Cb_n a n ≤ C b n for all n ≥ n 0 n\ge n_0 n ≥ n 0 , and write a n ≍ b n a_n\asymp b_n a n ≍ b n if a n ≲ b n a_n\lesssim b_n a n ≲ b n and a n ≳ b n a_n\gtrsim b_n a n ≳ b n . We further write a n ≲ log b n a_n\lesssim_{\log} b_n a n ≲ l o g b n (or a n = O log ( b n ) a_n=O_{\log}(b_n) a n = O l o g ( b n ) ) if a n ≲ b n a_n\lesssim b_n a n ≲ b n holds up to poly-logarithmic factors. For random variables { X n } \{X_n\} { X n } and positive numbers { a n } \{a_n\} { a n } , write X n = O p ( a n ) X_n=O_p(a_n) X n = O p ( a n ) if lim M → ∞ lim sup n → ∞ P ( ∣ X n ∣ ≥ M a n ) = 0 \lim_{M\to\infty}\limsup_{n\to\infty}\mathbb{P}(|X_n|\ge Ma_n)=0 lim M → ∞ lim sup n → ∞ P ( ∣ X n ∣ ≥ M a n ) = 0 . For any probability measure P P P , let ∥ f ∥ L 2 ( P ) = { E P ( f 2 ) } 1 / 2 \lVert f\rVert_{L_2(P)}=\{\mathbb{E}_P(f^2)\}^{1/2} ∥ f ∥ L 2 ( P ) = { E P ( f 2 ) } 1/2 . We write λ d ( ⋅ ) \lambda_d(\cdot) λ d ( ⋅ ) for d d d -dimensional Lebesgue measure.
Problem formulation
In this section, we formalise the problem setup for Lipschitz fair contextual bandits in Section 2.1, followed by technical assumptions in § 2.2 .
Setup and ( L , ν ) (L,\nu) ( L , ν ) -Lipschitz fairness
Consider a two-armed contextual bandit problem over a finite time horizon T ∈ Z + T \in\mathbb{Z}_{+} T ∈ Z + . The two-arm setting is widely studied and arises naturally in applications such as clinical trials, where the decision involves whether to administer a particular treatment. We focus on this setting for simplicity of exposition and to highlight the key ideas of our framework, with discussions on the extensions to multiple arms (K ≥ 3 K \ge3 K ≥ 3 ) in § 7 . At each round t ∈ [ T ] t \in[T] t ∈ [ T ] , a context X t ∈ X = [ 0 , 1 ] d X_t \in\mathcal{X} = [0,1]^d X t ∈ X = [ 0 , 1 ] d is observed before an action is chosen, with both arms sharing the same context. After observing X t X_t X t , the learner chooses an action A t ∈ [ 2 ] A_t \in[2] A t ∈ [ 2 ] and receives the realised reward Y t = R t , A t Y_t = R_{t,A_t} Y t = R t , A t of the selected arm.
For x ∈ [ 0 , 1 ] d x \in[0,1]^d x ∈ [ 0 , 1 ] d , write x ~ = ( 1 , x ⊤ ) ⊤ ∈ R d + 1 \widetilde{x} = (1, x^\top)^\top\in\mathbb{R}^{d+1} x = ( 1 , x ⊤ ) ⊤ ∈ R d + 1 . We assume the linear reward model R t , k = X ~ t ⊤ β k ∗ + ξ t , k R_{t,k} = \widetilde{X}_t^\top\beta_k^* + \xi_{t,k} R t , k = X t ⊤ β k ∗ + ξ t , k , k ∈ [ 2 ] k \in[2] k ∈ [ 2 ] , where β k ∗ ∈ R d + 1 \beta_k^* \in\mathbb{R}^{d+1} β k ∗ ∈ R d + 1 is an unknown arm-specific parameter and ξ t , k \xi_{t,k} ξ t , k is a mean-zero noise variable. Write θ ∗ = β 1 ∗ − β 2 ∗ = ( a ∗ , γ ∗ ⊤ ) ⊤ \theta^* = \beta_1^* - \beta_2^* = (a^*, \gamma^{*\top})^\top θ ∗ = β 1 ∗ − β 2 ∗ = ( a ∗ , γ ∗ ⊤ ) ⊤ , where a ∗ ∈ R a^* \in\mathbb{R} a ∗ ∈ R and γ ∗ ∈ R d \gamma^* \in\mathbb{R}^d γ ∗ ∈ R d denote the intercept and slope differences, respectively.
Remark 1. Our framework also extends to generalised linear rewards R t , k = μ ( X ~ t ⊤ β k ∗ ) + ξ t , k R_{t,k} = \mu(\widetilde{X}_t^\top\beta_k^*) + \xi_{t,k} R t , k = μ ( X t ⊤ β k ∗ ) + ξ t , k , k ∈ [ 2 ] k \in[2] k ∈ [ 2 ] , with a known, strictly increasing inverse link function μ \mu μ , and mean-zero noise ξ t , k \xi_{t,k} ξ t , k (e.g. [22 ] (Li et al. 2017)). Monotonicity of μ \mu μ preserves the sign of the reward gap and consequently the linear decision boundary, allowing the structural characterisation of fair policies discussed in § 3 to extend under suitable regularity conditions.
A policy is modelled as a measurable function p ( ⋅ ) : [ 0 , 1 ] d → [ 0 , 1 ] p(\cdot) : [0,1]^d \to[0,1] p ( ⋅ ) : [ 0 , 1 ] d → [ 0 , 1 ] , where p ( x ) p(x) p ( x ) denotes the probability of pulling arm 1 at context x x x , i.e. p ( x ) = P ( A t = 1 ∣ X = x ) p(x) = \mathbb{P}(A_t = 1 \mid X = x) p ( x ) = P ( A t = 1 ∣ X = x ) . In this setting, at the population level, the standard reward-maximisation objective for bandits is to seek a policy p p p that maximises the expected reward
R ( p ) = E [ X ~ ⊤ β 1 ∗ p ( X ) + X ~ ⊤ β 2 ∗ { 1 − p ( X ) } ] = E [ X ~ ⊤ β 2 ∗ + Δ ∗ ( X ) p ( X ) ] , (1) R(p) = \mathbb{E}\left[\widetilde{X}^\top\beta_1^*p(X) + \widetilde{X}^\top\beta_2^*\{1-p(X)\}\right] = \mathbb{E}\left[\widetilde{X}^\top\beta_2^* + \Delta^*(X)p(X)\right], \tag*{(1)} R ( p ) = E [ X ⊤ β 1 ∗ p ( X ) + X ⊤ β 2 ∗ { 1 − p ( X )} ] = E [ X ⊤ β 2 ∗ + Δ ∗ ( X ) p ( X ) ] , (1)
where Δ ∗ ( x ) = x ~ ⊤ ( β 1 ∗ − β 2 ∗ ) = x ~ ⊤ θ ∗ \Delta^*(x) = \widetilde{x}^\top(\beta_1^* - \beta_2^*) = \widetilde{x}^\top\theta^* Δ ∗ ( x ) = x ⊤ ( β 1 ∗ − β 2 ∗ ) = x ⊤ θ ∗ denotes the conditional mean reward gap between arms 1 and 2 at context x x x , and the expectation is taken with respect to the stochastic context X X X .
By (1) , the expected reward R ( p ) R(p) R ( p ) depends on p p p only through E { Δ ∗ ( X ) p ( X ) } \mathbb{E}\{\Delta^*(X)p(X)\} E { Δ ∗ ( X ) p ( X )} . Without fairness constraints, the reward-maximising policy simply selects the arm with the larger conditional mean reward at each x x x , i.e. by comparing the sign of Δ ∗ ( x ) \Delta^*(x) Δ ∗ ( x ) as
p u n ∗ ( x ) = 1 { Δ ∗ ( x ) > 0 } , (2) p_{\mathrm{un}}^*(x) = \mathbb{1}\{\Delta^*(x) > 0\}, \tag*{(2)} p un ∗ ( x ) = 1 { Δ ∗ ( x ) > 0 } , (2)
where the policy values on the decision boundary H θ ∗ = { x ∈ [ 0 , 1 ] d : Δ ∗ ( x ) = 0 } \mathcal{H}_{\theta^*} = \{x \in[0,1]^d : \Delta^*(x) = 0\} H θ ∗ = { x ∈ [ 0 , 1 ] d : Δ ∗ ( x ) = 0 } do not affect R ( p ) R(p) R ( p ) and ties may be broken uniformly at random.
To incorporate fairness into contextual bandits, we introduce the notion of ( L , ν ) (L,\nu) ( L , ν ) -Lipschitz fairness, following the individual fairness framework of Dwork et al. [21 ] .
Definition 1 (( L , ν ) (L,\nu) ( L , ν ) -Lipschitz fairness). For L ≥ 0 L \ge0 L ≥ 0 and ν ∈ [ 1 , ∞ ] \nu\in[1,\infty] ν ∈ [ 1 , ∞ ] , a policy p ( ⋅ ) : [ 0 , 1 ] d → [ 0 , 1 ] p(\cdot) : [0,1]^d \to[0,1] p ( ⋅ ) : [ 0 , 1 ] d → [ 0 , 1 ] is ( L , ν ) (L,\nu) ( L , ν ) -Lipschitz fair if
∣ p ( x ) − p ( x ′ ) ∣ ≤ L ∥ x − x ′ ∥ ν , for all x , x ′ ∈ [ 0 , 1 ] d . |p(x) - p(x')| \le L\lVert x - x'\rVert_\nu,\qquad\text{for all } x,x' \in[0,1]^d. ∣ p ( x ) − p ( x ′ ) ∣ ≤ L ∥ x − x ′ ∥ ν , for all x , x ′ ∈ [ 0 , 1 ] d . We denote the class of all ( L , ν ) (L,\nu) ( L , ν ) -Lipschitz fair policies by
P L , ν = { p : [ 0 , 1 ] d → [ 0 , 1 ] : ∣ p ( x ) − p ( x ′ ) ∣ ≤ L ∥ x − x ′ ∥ ν , for all x , x ′ ∈ [ 0 , 1 ] d } . \mathcal{P}_{L,\nu} = \left\{p : [0,1]^d \to[0,1] : |p(x) - p(x')| \le L\lVert x - x'\rVert_\nu,\ \text{for all } x,x' \in[0,1]^d\right\}. P L , ν = { p : [ 0 , 1 ] d → [ 0 , 1 ] : ∣ p ( x ) − p ( x ′ ) ∣ ≤ L ∥ x − x ′ ∥ ν , for all x , x ′ ∈ [ 0 , 1 ] d } .
Definition 1 ensures that individuals with similar contexts receive similar treatment probabilities through the ( L , ν ) (L,\nu) ( L , ν ) -Lipschitz constraints on p p p . The parameter L L L controls the strength of this constraint, with smaller values imposing stronger smoothness constraints. When L = ∞ L = \infty L = ∞ , the fairness constraint is inactive, and recovers the unconstrained optimal policy p u n ∗ p_{\mathrm{un}}^* p un ∗ in (2) . When the constraint is active with L < ∞ L < \infty L < ∞ , fairness prevents abrupt shifts in action probabilities across nearby contexts, which may require assigning positive selection probability to the arm with the smaller conditional mean reward.
Remark 2. The fairness requirement in Definition 1 is ex ante . Individuals with similar covariates receive similar treatment allocation probabilities, although their realised assignments may differ. Wei et al. [53 ] adopt a related perspective when studying group fairness in adaptive experiments.
Remark 3 (Choice of the similarity metric). The similarity metric can be chosen according to the specific application [21 ] . While we work with the unweighted ℓ ν \ell_\nu ℓ ν -norm for concreteness, our framework can readily accommodate alternative metrics, such as the weighted distance d W ( x , x ′ ) = ∥ W ( x − x ′ ) ∥ ν d_W(x,x') = \lVert W(x-x')\rVert_\nu d W ( x , x ′ ) = ∥ W ( x − x ′ ) ∥ ν , where W W W is a suitable weight matrix reflecting task-specific similarities.
Remark 4. Under suitable regularity conditions, reward-maximising fair policies assign both arms marginal selection probabilities bounded away from zero (Lemma 29 ). This connects our Lipschitz fairness formulation (Definition 1) to arm-level minimum pull-proportion fairness notions (e.g. Chen et al. [16 ] ; Patil et al. [42 ] ).
With Definition 1, for a fixed pair ( L , ν ) (L,\nu) ( L , ν ) , we define the optimal ( L , ν ) (L,\nu) ( L , ν ) -Lipschitz fair policy by
p L , ν ∗ ∈ arg max p ∈ P L , ν R ( p ) . p^*_{L,\nu} \in\underset{p\in\mathcal{P}_{L,\nu}}{\arg\max}\ R(p). p L , ν ∗ ∈ p ∈ P L , ν arg max R ( p ) .
When there is no ambiguity, write p ∗ p^* p ∗ for p L , ν ∗ p^*_{L,\nu} p L , ν ∗ . In the online learning setting, our goal is to construct a sequence of estimated policies { p ^ t } t = 1 T \{\widehat{p}_t\}_{t=1}^{T} { p t } t = 1 T such that p ^ t ∈ P L , ν \widehat{p}_t \in\mathcal{P}_{L,\nu} p t ∈ P L , ν for all t ∈ [ T ] t \in[T] t ∈ [ T ] , while minimising the cumulative regret defined as
R T ( { p ^ t } t = 1 T ) = ∑ t = 1 T { R ( p ∗ ) − R ( p ^ t ) } . (3) \mathfrak{R}_T(\{\widehat{p}_t\}_{t=1}^{T}) = \sum_{t=1}^{T}\{R(p^*)-R(\widehat{p}_t)\}. \tag*{(3)} R T ({ p t } t = 1 T ) = t = 1 ∑ T { R ( p ∗ ) − R ( p t )} . (3)
Here, we emphasize that for a data-dependent policy p ^ t \widehat{p}_t p t measurable with respect to the history prior to round t t t , R ( p ^ t ) = E [ X ~ T β 2 ∗ + Δ ∗ ( X ) p ^ t ∣ p ^ t ] R(\widehat{p}_t) = \mathbb{E}\left[\widetilde{X}^{\mathsf{T}}\beta^*_2+\Delta^*(X)\widehat{p}_t\mid\widehat{p}_t\right] R ( p t ) = E [ X T β 2 ∗ + Δ ∗ ( X ) p t ∣ p t ] denotes its conditional population reward with respect to a new draw X ∼ P X X\sim P_X X ∼ P X .
Assumptions
In this subsection, we present the assumptions needed.
Assumption 1 (Contexts). The contexts X 1 , … , X T X_1,\ldots,X_T X 1 , … , X T are independent and identically distributed (i.i.d.) following an arbitrary unknown law P X P_X P X supported on X = [ 0 , 1 ] d \mathcal{X}=[0,1]^d X = [ 0 , 1 ] d . Assume that g ( ⋅ ) g(\cdot) g ( ⋅ ) , the density of P X P_X P X , exists and is uniformly bounded above and away from zero, i.e. c g ≤ inf x ∈ [ 0 , 1 ] d g ( x ) ≤ sup x ∈ [ 0 , 1 ] d g ( x ) ≤ C g c_g\le\inf_{x\in[0,1]^d}g(x)\le\sup_{x\in[0,1]^d}g(x)\le C_g c g ≤ inf x ∈ [ 0 , 1 ] d g ( x ) ≤ sup x ∈ [ 0 , 1 ] d g ( x ) ≤ C g , where 0 < c g ≤ C g 0<c_g\le C_g 0 < c g ≤ C g are absolute constants.
Assumption 1 is standard in the nonparametric and contextual bandit literature (e.g. Perchet and Rigollet [43 ] ). It ensures that the contexts are sufficiently spread across the covariate space, supporting our structural analysis and the theoretical guarantees of the proposed greedy algorithm in Algorithm 1. In § 4.3 , we relax it to Assumption 1 ′ 1' 1 ′ , allowing discrete and mixed context distributions.
Assumption 2 (Noise). For each t ∈ [ T ] t\in[T] t ∈ [ T ] and k ∈ [ 2 ] k\in[2] k ∈ [ 2 ] , the noise variables ξ t , k \xi_{t,k} ξ t , k 's are independent across t t t and k k k , independent of the contexts { X t } t ≥ 1 \{X_t\}_{t\ge1} { X t } t ≥ 1 and the algorithmic randomness. Assume that ξ t , k \xi_{t,k} ξ t , k 's are mean-zero and sub-Gaussian, i.e. there exists an absolute constant σ > 0 \sigma>0 σ > 0 such that E [ exp ( λ ξ t , k ) ] ≤ exp ( λ 2 σ 2 / 2 ) \mathbb{E}[\exp(\lambda\xi_{t,k})]\le\exp(\lambda^2\sigma^2/2) E [ exp ( λ ξ t , k )] ≤ exp ( λ 2 σ 2 /2 ) for all λ ∈ R \lambda\in\mathbb{R} λ ∈ R .
Assumption 3 (Bounded parameters). There exists an absolute constant 0 < C β < ∞ 0<C_\beta<\infty 0 < C β < ∞ such that, for k ∈ [ 2 ] k\in[2] k ∈ [ 2 ] , we have ∥ β k ∗ ∥ 2 ≤ C β \lVert\beta_k^*\rVert_2\le C_\beta ∥ β k ∗ ∥ 2 ≤ C β .
Assumptions 2 and 3 regulate the noise distribution and arm regression parameters, respectively. Both are commonly adopted in the linear contextual bandit literature (e.g. Goldenshluger and Zeevi [25 ] ; Wu et al. (2020); Bastani et al. [5 ] ).
Assumption 4 (Boundary parameter sets). There exist absolute constants A 0 , B 0 > 0 A_0, B_0 > 0 A 0 , B 0 > 0 , 0 < c 0 < C 0 < B 0 0 < c_0 < C_0 < B_0 0 < c 0 < C 0 < B 0 , and ρ 0 ∈ ( 0 , 1 / 2 ) \rho_0 \in(0, 1/2) ρ 0 ∈ ( 0 , 1/2 ) such that θ ∗ = β 1 ∗ − β 2 ∗ ∈ Θ = ⋃ j = 1 d Θ j \theta^* = \beta^*_1 - \beta^*_2 \in\Theta= \bigcup_{j=1}^{d} \Theta_j θ ∗ = β 1 ∗ − β 2 ∗ ∈ Θ = ⋃ j = 1 d Θ j , where, for each j ∈ [ d ] j \in[d] j ∈ [ d ] , γ − j \gamma_{-j} γ − j denotes the vector obtained by removing the j j j -th coordinate of γ \gamma γ , and
Θ j = { θ = ( a , γ ⊤ ) ⊤ ∈ R × R d : ∣ a ∣ ≤ A 0 , ∥ γ ∥ 2 ≤ B 0 , c 0 ≤ γ j ≤ C 0 , \Theta_j = \{\theta= (a, \gamma^\top)^\top\in\mathbb{R} \times\mathbb{R}^d : |a| \le A_0,\ \|\gamma\|_2 \le B_0,\ c_0 \le\gamma_j \le C_0, Θ j = { θ = ( a , γ ⊤ ) ⊤ ∈ R × R d : ∣ a ∣ ≤ A 0 , ∥ γ ∥ 2 ≤ B 0 , c 0 ≤ γ j ≤ C 0 , and there exists u θ ∈ [ 0 , 1 ] d − 1 such that ρ 0 ≤ − ( a + γ − j ⊤ u θ ) γ j ≤ 1 − ρ 0 } , \text{and there exists } u_\theta\in[0,1]^{d-1} \text{ such that } \rho_0 \le-\frac{(a+\gamma_{-j}^\top u_\theta)}{\gamma_j} \le1-\rho_0\}, and there exists u θ ∈ [ 0 , 1 ] d − 1 such that ρ 0 ≤ − γ j ( a + γ − j ⊤ u θ ) ≤ 1 − ρ 0 } , with the convention that γ − 1 ⊤ u θ = 0 \gamma_{-1}^\top u_\theta= 0 γ − 1 ⊤ u θ = 0 if d = 1 d=1 d = 1 .
Assumption 4 imposes geometric regularity on the decision boundary H θ = { x ∈ [ 0 , 1 ] d : a + γ ⊤ x = 0 } H_\theta= \{x \in[0,1]^d : a+\gamma^\top x = 0\} H θ = { x ∈ [ 0 , 1 ] d : a + γ ⊤ x = 0 } , with θ = ( a , γ ⊤ ) ⊤ \theta= (a,\gamma^\top)^\top θ = ( a , γ ⊤ ) ⊤ . Each Θ j \Theta_j Θ j , j ∈ [ d ] j \in[d] j ∈ [ d ] , requires the boundary to cross the interior interval [ ρ 0 , 1 − ρ 0 ] [\rho_0,1-\rho_0] [ ρ 0 , 1 − ρ 0 ] along the j j j -th coordinate, with γ j ∈ [ c 0 , C 0 ] \gamma_j \in[c_0,C_0] γ j ∈ [ c 0 , C 0 ] preventing the reward gap from vanishing or degenerating along that coordinate, which would otherwise reduce the effective dimension of the problem. Taking the union over j ∈ [ d ] j \in[d] j ∈ [ d ] allows this direction to vary with θ \theta θ . Together with the boundedness of ∥ γ ∥ 2 ≤ B 0 \|\gamma\|_2 \le B_0 ∥ γ ∥ 2 ≤ B 0 , these conditions exclude boundaries confined to arbitrarily small regions near the edges or corners of the covariate space, where one arm dominates almost everywhere; see Figure 1 for an illustration. Note that the constants in Assumption 4 need not be known to the learning algorithms proposed in § 4 . They enter only through the constants in the theoretical guarantees.
Figure 1. Illustration of Assumption 4 when d = 2 d=2 d = 2 . Plots (a) and (b) illustrate feasible decision boundaries satisfying the conditions for j = 1 j=1 j = 1 and j = 2 j=2 j = 2 , respectively, with the red points marking the corresponding interior crossings. Plot (c) illustrates two examples of the ruled-out boundaries.
Structural consequences of the Lipschitz fairness
In this section, we characterise how the Lipschitz fairness constraint reshapes the optimal fair policy in § 3.1 , explicitly quantify the resulting cost of fairness in § 3.2 , and further identify a canonical structure of the optimal fair policy in § 3.3 , providing the key structural insight underlying the proposed online learning algorithm in § 4 .
Boundary randomisation under Lipschitz fairness
To understand how the Lipschitz fairness constraint characterises the optimal policy, we first introduce some notation on the geometry induced by a general reward-gap parameter θ ∈ Θ \theta\in\Theta θ ∈ Θ . For any θ ∈ Θ \theta\in\Theta θ ∈ Θ , the decision boundary H θ = { x ∈ [ 0 , 1 ] d : x ~ ⊤ θ = 0 } H_\theta= \{x \in[0,1]^d : \tilde{x}^{\top}\theta= 0\} H θ = { x ∈ [ 0 , 1 ] d : x ~ ⊤ θ = 0 } consists of contexts where the two arms have equal expected rewards. It separates the covariate space into the regions
X θ + = { x ∈ [ 0 , 1 ] d : x ~ ⊤ θ > 0 } and X θ − = { x ∈ [ 0 , 1 ] d : x ~ ⊤ θ < 0 } , \mathcal{X}_{\theta}^{+} = \{x \in[0,1]^d : \tilde{x}^{\top}\theta> 0\}
\qquad\text{and}\qquad
\mathcal{X}_{\theta}^{-} = \{x \in[0,1]^d : \tilde{x}^{\top}\theta< 0\}, X θ + = { x ∈ [ 0 , 1 ] d : x ~ ⊤ θ > 0 } and X θ − = { x ∈ [ 0 , 1 ] d : x ~ ⊤ θ < 0 } ,
where arms 1 and 2, respectively, have the higher expected reward under θ \theta θ .
For θ = θ ∗ \theta= \theta^* θ = θ ∗ , the unconstrained optimal policy p u n ∗ p_{\mathrm{un}}^* p un ∗ in (2) switches abruptly from always selecting arm 1 on X θ ∗ + \mathcal{X}_{\theta^*}^{+} X θ ∗ + to always selecting arm 2 on X θ ∗ − \mathcal{X}_{\theta^*}^{-} X θ ∗ − ; see Figure 2a . Lipschitz fairness prohibits this discontinuity. The following lemma characterises the resulting randomisation near the decision boundary.
Lemma 1. Suppose Assumptions 1 and 4 hold and fix an L > 0 L > 0 L > 0 . Define the randomisation region associated with the optimal policy p ∗ p^* p ∗ by M ( p ∗ ) = { x ∈ [ 0 , 1 ] d : 0 < p ∗ ( x ) < 1 } \mathcal{M}(p^*) = \{x \in[0,1]^d : 0 < p^*(x) < 1\} M ( p ∗ ) = { x ∈ [ 0 , 1 ] d : 0 < p ∗ ( x ) < 1 } , and denote θ ∗ = ( a ∗ , γ ∗ ⊤ ) ⊤ \theta^* = (a^*, \gamma^{*\top})^\top θ ∗ = ( a ∗ , γ ∗ ⊤ ) ⊤ , where a ∗ ∈ R a^* \in\mathbb{R} a ∗ ∈ R and γ ∗ ∈ R d \gamma^* \in\mathbb{R}^d γ ∗ ∈ R d . It then holds that
M ( p ∗ ) ⊆ { x ∈ [ 0 , 1 ] d : dist ν ( x , H θ ∗ ) ≤ L − 1 } ⊆ { x ∈ [ 0 , 1 ] d : ∣ Δ ∗ ( x ) ∣ / ∥ γ ∗ ∥ ν ∗ ≤ L − 1 } . \mathcal{M}(p^*) \subseteq\{x \in[0,1]^d : \operatorname{dist}_{\nu}(x,H_{\theta^*}) \le L^{-1}\} \subseteq\{x \in[0,1]^d : |\Delta^*(x)|/\|\gamma^*\|_{\nu^*} \le L^{-1}\}. M ( p ∗ ) ⊆ { x ∈ [ 0 , 1 ] d : dist ν ( x , H θ ∗ ) ≤ L − 1 } ⊆ { x ∈ [ 0 , 1 ] d : ∣ Δ ∗ ( x ) ∣/∥ γ ∗ ∥ ν ∗ ≤ L − 1 } . Furthermore, if dist ν ( x , H θ ∗ ) > 1 / L \operatorname{dist}_{\nu}(x,H_{\theta^*}) > 1/L dist ν ( x , H θ ∗ ) > 1/ L , then p ∗ ( x ) ∈ { 0 , 1 } p^*(x) \in\{0,1\} p ∗ ( x ) ∈ { 0 , 1 } , with p ∗ ( x ) = 1 { Δ ∗ ( x ) > 0 } p^*(x) = 1\{\Delta^*(x) > 0\} p ∗ ( x ) = 1 { Δ ∗ ( x ) > 0 } .
The proof of Lemma 1 is given in Appendix § 10 .§ 1 . Lemma 1 shows that, rather than switching directly between the two arms across the decision boundary, the optimal fair policy may interpolate between assignment probabilities 0 and 1. Such interpolation can only occur inside a 1 / L 1/L 1/ L -neighbourhood of the decision boundary. The randomisation region is, therefore, contained in a slab around the decision boundary with width at most 2 / L 2/L 2/ L in ℓ ν \ell_{\nu} ℓ ν -distance. Outside this region, p ∗ p^* p ∗ agrees with the unconstrained greedy rule p u n ∗ p_{\mathrm{un}}^* p un ∗ , as illustrated in Figure 2b .
(a) Unconstrained optimal policy.
(b) Optimal fair policy under the Lipschitz constraint.
Figure 2. Comparison between the unconstrained and fair optimal policies.
Under Assumption 3, ∥ γ ∗ ∥ ν ∗ ≲ 1 \|\gamma^*\|_{\nu^*} \lesssim1 ∥ γ ∗ ∥ ν ∗ ≲ 1 , so Lemma 1 also implies that the randomisation only occurs where ∣ Δ ∗ ( x ) ∣ ≲ 1 / L |\Delta^*(x)| \lesssim1/L ∣ Δ ∗ ( x ) ∣ ≲ 1/ L . Thus, the optimal fair policy only randomises between arms when the reward gap is of order at most 1 / L 1/L 1/ L , limiting the resulting reward loss. This observation provides the key intuition behind the analysis of the cost of fairness in Theorem 2 in § 3.2 .
Cost of fairness
Let R u n ∗ = R ( p u n ∗ ) R_{\mathrm{un}}^* = R(p_{\mathrm{un}}^*) R un ∗ = R ( p un ∗ ) denote the reward for unconstrained optimal policy in (2) and R L ∗ = sup p ∈ P L , ν R ( p ) R_L^* = \sup_{p \in\mathcal{P}_{L,\nu}} R(p) R L ∗ = sup p ∈ P L , ν R ( p ) the optimal reward under ( L , ν ) (L,\nu) ( L , ν ) -Lipschitz fairness constraint. Building on the boundary randomisation result in Lemma 1, we quantify the cost of fairness Cost ( L ) = R u n ∗ − R L ∗ \operatorname{Cost}(L) = R_{\mathrm{un}}^* - R_L^* Cost ( L ) = R un ∗ − R L ∗ in the following theorem.
Theorem 2 (Cost of fairness). Suppose Assumptions 1, 3 and 4 hold. Let d ν = d 1 / ν d_{\nu}=d^{1/\nu} d ν = d 1/ ν , with the convention d ∞ = 1 d_{\infty}=1 d ∞ = 1 .
For every L > 0 L>0 L > 0 , we have that max { 0 , Cost ( 0 ) − L d ν E ∣ Δ ∗ ( X ) ∣ } ≤ Cost ( L ) ≤ min { Cost ( 0 ) , 1 / L 2 } \max\{0,\operatorname{Cost}(0)-Ld_{\nu}\mathbb{E}|\Delta^*(X)|\}\leq\operatorname{Cost}(L)\leq\min\{\operatorname{Cost}(0),1/L^2\} max { 0 , Cost ( 0 ) − L d ν E ∣ Δ ∗ ( X ) ∣ } ≤ Cost ( L ) ≤ min { Cost ( 0 ) , 1/ L 2 } .
Suppose additionally that L ≥ L 0 L\geq L_0 L ≥ L 0 for a constant L 0 < ∞ L_0<\infty L 0 < ∞ , depending only on the constants in Assumptions 1 and 4. Then, there exist constants 0 < c f a i r ≤ C f a i r < ∞ 0<c_{\mathrm{fair}}\leq C_{\mathrm{fair}}<\infty 0 < c fair ≤ C fair < ∞ depending on d d d and the absolute constants in Assumptions 3 and 4, such that
c f a i r L − 2 ≤ Cost ( L ) ≤ C f a i r L − 2 . c_{\mathrm{fair}}L^{-2}\leq\operatorname{Cost}(L)\leq C_{\mathrm{fair}}L^{-2}. c fair L − 2 ≤ Cost ( L ) ≤ C fair L − 2 .
The proof of Theorem 2 is in Appendix § 10 .§ 2 . Theorem 2 characterises the cost of fairness as the fairness parameter L L L varies.
When L = 0 L=0 L = 0 , the fairness constraint forces the policy to be constant across the entire covariate space, so it is optimal to always select the arm with the larger expected reward. The baseline cost is therefore
Cost ( 0 ) = 2 − 1 ( E ∣ Δ ∗ ( X ) ∣ − ∣ E Δ ∗ ( X ) ∣ ) ≤ 2 − 1 Var ( Δ ∗ ( X ) ) ≤ ( 4 3 ) − 1 C g B 0 = O ( 1 ) , \operatorname{Cost}(0)=2^{-1}\left(\mathbb{E}|\Delta^*(X)|-|\mathbb{E}\Delta^*(X)|\right)\leq2^{-1}\sqrt{\operatorname{Var}(\Delta^*(X))}\leq(4\sqrt{3})^{-1}\sqrt{C_g}B_0=O(1), Cost ( 0 ) = 2 − 1 ( E ∣ Δ ∗ ( X ) ∣ − ∣ E Δ ∗ ( X ) ∣ ) ≤ 2 − 1 Var ( Δ ∗ ( X )) ≤ ( 4 3 ) − 1 C g B 0 = O ( 1 ) ,
where the last inequality follows from Assumptions 1, 3 and 4. Under these conditions, Cost ( 0 ) \operatorname{Cost}(0) Cost ( 0 ) is strictly positive for each fixed d d d and bounded above independent of d d d . Since increasing L L L enlarges the admissible policy class, Cost ( L ) \operatorname{Cost}(L) Cost ( L ) is non-increasing in L L L , with Cost ( 0 ) \operatorname{Cost}(0) Cost ( 0 ) being its maximum.
For small L > 0 L>0 L > 0 , Theorem 2 shows that Cost ( 0 ) − Cost ( L ) = O ( L ) \operatorname{Cost}(0)-\operatorname{Cost}(L)=O(L) Cost ( 0 ) − Cost ( L ) = O ( L ) , so the cost remains bounded away from zero for sufficiently small L L L . As L L L increases further, the cost is controlled by the local behaviour of the policy near the decision boundary. By Lemma 1, randomisation is confined to a slab of width at most of order O ( L − 1 ) O(L^{-1}) O ( L − 1 ) , within which the reward gap ∣ Δ ∗ ( X ) ∣ = O ( L − 1 ) |\Delta^*(X)|=O(L^{-1}) ∣ Δ ∗ ( X ) ∣ = O ( L − 1 ) , yielding a total reward loss of order at most O ( L − 2 ) O(L^{-2}) O ( L − 2 ) . Thus, Cost ( L ) → 0 \operatorname{Cost}(L)\to0 Cost ( L ) → 0 as L → ∞ L\to\infty L → ∞ .
For a constant L 0 L_0 L 0 depending on the boundary separation constant ρ 0 \rho_0 ρ 0 in Assumption 4, when L ≥ L 0 L\geq L_0 L ≥ L 0 , the upper bound of order O ( L − 2 ) O(L^{-2}) O ( L − 2 ) is sharp. By Assumption 4, the decision boundary H θ ∗ \mathcal{H}_{\theta^*} H θ ∗ contains an interior region away from the edge of the covariate domain. Across this boundary, the unconstrained oracle p u n ∗ p^*_{\mathrm{un}} p un ∗ switches between arms, whereas a Lipschitz fair policy requires a distance of at least order O ( L − 1 ) O(L^{-1}) O ( L − 1 ) to transition from probability 0 to 1. Since the reward gap grows linearly with the distance from the boundary, this restriction creates an unavoidable loss of order ∫ 0 L − 1 s d s ≍ L − 2 \int_0^{L^{-1}}s\,\mathrm{d}s\asymp L^{-2} ∫ 0 L − 1 s d s ≍ L − 2 . The condition L ≥ L 0 L\geq L_0 L ≥ L 0 ensures that the boundary band of width 1 / L 1/L 1/ L remains inside the covariate domain, so that the local lower bound construction is not affected by boundary effects.
Remark 5. In general, the optimal fair policy need not be unique or admit a simple closed-form expression. In Appendix § 10 .2.1, we consider an example with explicit expressions for an optimal policy and the cost of fairness. Numerical results in § 6.1 further illustrate the theoretical findings in Theorem 2.
Canonical structures
We next establish a canonical representation of the optimal fair policy, determined by its values on the decision boundary. We formulate this representation for any θ ∈ Θ \theta\in\Theta θ ∈ Θ . Taking θ = θ ∗ \theta=\theta^* θ = θ ∗ characterises optimal policies under the true reward model, whereas allowing θ \theta θ to vary supports learning with unknown reward parameters in § 4 .
Motivated by the form of the unconstrained optimal policy in (2) and the boundary randomisation phenomenon in Lemma 1, for any θ ∈ Θ \theta\in\Theta θ ∈ Θ , reward maximisation favours increasing p p p towards 1 on X θ + \mathcal{X}_{\theta}^{+} X θ + and decreasing on X θ − \mathcal{X}_{\theta}^{-} X θ − . Although policy values on H θ \mathcal{H}_{\theta} H θ contribute no reward gap, they constrain the feasible policies elsewhere through the Lipschitz fairness constraint. This motivates a two-stage construction: first specify the values on H θ \mathcal{H}_{\theta} H θ and then extend them off the boundary to the rest of the covariate space through an extremal Lipschitz extension.
For any parameter θ ∈ Θ \theta\in\Theta θ ∈ Θ , define Q θ , L \mathcal{Q}_{\theta,L} Q θ , L the collection of ( L , ν ) (L,\nu) ( L , ν ) -Lipschitz fair policies over boundary, i.e.
Q θ , L = { q : H θ → [ 0 , 1 ] : ∣ q ( h ) − q ( h ′ ) ∣ ≤ L ∥ h − h ′ ∥ ν for all h , h ′ ∈ H θ } . (4) \mathcal{Q}_{\theta,L} = \{q:\mathcal{H}_{\theta} \to[0,1] : |q(h)-q(h')| \le L\lVert h-h'\rVert_{\nu}\text{ for all }h,h' \in\mathcal{H}_{\theta}\}. \tag*{(4)} Q θ , L = { q : H θ → [ 0 , 1 ] : ∣ q ( h ) − q ( h ′ ) ∣ ≤ L ∥ h − h ′ ∥ ν for all h , h ′ ∈ H θ } . (4)
For q ∈ Q θ , L q \in\mathcal{Q}_{\theta,L} q ∈ Q θ , L and x ∈ [ 0 , 1 ] d x \in[0,1]^d x ∈ [ 0 , 1 ] d , define the upper and lower ( L , ν ) (L,\nu) ( L , ν ) -McShane envelopes [40 ] by
U θ , q ( x ) = inf h ∈ H θ { q ( h ) + L ∥ x − h ∥ ν } and M θ , q ( x ) = sup h ∈ H θ { q ( h ) − L ∥ x − h ∥ ν } . (5) U_{\theta,q}(x)=\inf_{h\in\mathcal{H}_{\theta}}\{q(h)+L\lVert x-h\rVert_{\nu}\}\quad\text{and}\quad M_{\theta,q}(x)=\sup_{h\in\mathcal{H}_{\theta}}\{q(h)-L\lVert x-h\rVert_{\nu}\}. \tag*{(5)} U θ , q ( x ) = h ∈ H θ inf { q ( h ) + L ∥ x − h ∥ ν } and M θ , q ( x ) = h ∈ H θ sup { q ( h ) − L ∥ x − h ∥ ν } . (5)
The boundary-canonical policy associated with ( θ , q ) (\theta,q) ( θ , q ) takes the largest feasible value on X θ + \mathcal{X}_{\theta}^{+} X θ + and the smallest on X θ − \mathcal{X}_{\theta}^{-} X θ − , i.e.
p θ , q ( x ) = { min { 1 , U θ , q ( x ) } , x ∈ X θ + , max { 0 , M θ , q ( x ) } , x ∈ X θ − . (6) p_{\theta,q}(x)=
\begin{cases}
\min\{1,U_{\theta,q}(x)\}, & x\in\mathcal{X}_{\theta}^{+},\\
\max\{0,M_{\theta,q}(x)\}, & x\in\mathcal{X}_{\theta}^{-}.
\end{cases} \tag*{(6)} p θ , q ( x ) = { min { 1 , U θ , q ( x )} , max { 0 , M θ , q ( x )} , x ∈ X θ + , x ∈ X θ − . (6)
Note that U θ , q ( x ) = M θ , q ( x ) = q ( x ) U_{\theta,q}(x)=M_{\theta,q}(x)=q(x) U θ , q ( x ) = M θ , q ( x ) = q ( x ) for every x ∈ H θ x\in\mathcal{H}_{\theta} x ∈ H θ , so p θ , q p_{\theta,q} p θ , q agrees continuously along H θ \mathcal{H}_{\theta} H θ . The following theorem establishes a canonical representation of optimal Lipschitz fair policies through this construction.
Theorem 3 (Canonical representation). Let θ ∈ R d + 1 \theta\in\mathbb{R}^{d+1} θ ∈ R d + 1 satisfy H θ ≠ ∅ \mathcal{H}_{\theta}\ne\varnothing H θ = ∅ . For every q ∈ Q θ , L q\in\mathcal{Q}_{\theta,L} q ∈ Q θ , L , it holds that p θ , q p_{\theta,q} p θ , q in (6) belongs to P L , ν \mathcal{P}_{L,\nu} P L , ν . Moreover, for any p ∈ P L , ν p\in\mathcal{P}_{L,\nu} p ∈ P L , ν , let q = p ∣ H θ q=p|_{\mathcal{H}_{\theta}} q = p ∣ H θ , then the canonical extension does not decrease the pointwise reward contribution, i.e.
( x ~ ⊤ θ ) p θ , q ( x ) ≥ ( x ~ ⊤ θ ) p ( x ) , x ∈ X . (\widetilde{x}^{\top}\theta)p_{\theta,q}(x)\ge(\widetilde{x}^{\top}\theta)p(x),\qquad x\in\mathcal{X}. ( x ⊤ θ ) p θ , q ( x ) ≥ ( x ⊤ θ ) p ( x ) , x ∈ X . Consequently, we have that
sup p ∈ P L , ν E { ( X ~ ⊤ θ ) p ( X ) } = sup q ∈ Q θ , L E { ( X ~ ⊤ θ ) p θ , q ( X ) } , (7) \sup_{p\in\mathcal{P}_{L,\nu}}\mathbb{E}\{(\widetilde{X}^{\top}\theta)p(X)\}
=
\sup_{q\in\mathcal{Q}_{\theta,L}}\mathbb{E}\{(\widetilde{X}^{\top}\theta)p_{\theta,q}(X)\}, \tag*{(7)} p ∈ P L , ν sup E {( X ⊤ θ ) p ( X )} = q ∈ Q θ , L sup E {( X ⊤ θ ) p θ , q ( X )} , (7) and both optimisation problems in (7) admit at least one maximiser.
The proof of Theorem 3 is in Appendix § 10 .§ 3 . For each fixed θ \theta θ , Theorem 3 shows that replacing a feasible policy by its canonical extension p θ , p ∣ H θ p_{\theta,p|_{\mathcal{H}_{\theta}}} p θ , p ∣ H θ cannot decrease the objective associated with the reward gap x ~ ⊤ θ \widetilde{x}^{\top}\theta x ⊤ θ . For a given candidate parameter, this reduces optimisation over all Lipschitz fair policies to optimisation over their boundary values. Since θ ∗ ∈ Θ \theta^{*}\in\Theta θ ∗ ∈ Θ , the canonical class { p θ , q : θ ∈ Θ , q ∈ Q θ , L } ⊆ P L , ν \{p_{\theta,q}:\theta\in\Theta,\ q\in\mathcal{Q}_{\theta,L}\}\subseteq\mathcal{P}_{L,\nu} { p θ , q : θ ∈ Θ , q ∈ Q θ , L } ⊆ P L , ν contains at least one policy attaining the optimal population reward.
The key reason why the canonical representation cannot decrease the objective value is that, once the policy values q q q on H θ \mathcal{H}_{\theta} H θ are fixed, any ( L , ν ) (L,\nu) ( L , ν ) -Lipschitz extension p p p satisfying p ∣ H θ = q p|_{\mathcal{H}_{\theta}}=q p ∣ H θ = q must satisfy
M θ , q ( x ) ≤ p ( x ) ≤ U θ , q ( x ) , x ∈ X . M_{\theta,q}(x)\le p(x)\le U_{\theta,q}(x),\qquad x\in\mathcal{X}. M θ , q ( x ) ≤ p ( x ) ≤ U θ , q ( x ) , x ∈ X .
The canonical policy attains the upper bound on X θ + \mathcal{X}_{\theta}^{+} X θ + and the lower bound on X θ − \mathcal{X}_{\theta}^{-} X θ − , maximising the pointwise reward while preserving q q q and Lipschitz fairness. When θ = θ ∗ \theta=\theta^{*} θ = θ ∗ , this pushes the policy towards the unconstrained optimal policy p u n ∗ p_{\mathrm{un}}^{*} p un ∗ in (2) as far as the fairness permits. This canonical representation guides the construction of Algorithm 1 in § 4 .
Importantly, Theorem 3 reduces functional optimisation over d d d -dimensional Lipschitz policies P L , ν \mathcal{P}_{L,\nu} P L , ν to optimisation over ( d − 1 ) (d-1) ( d − 1 ) -dimensional boundary traces Q θ , L \mathcal{Q}_{\theta,L} Q θ , L . Since θ ∗ ∈ Θ \theta^{*}\in\Theta θ ∗ ∈ Θ ensures H θ ∗ ≠ ∅ \mathcal{H}_{\theta^{*}}\ne\varnothing H θ ∗ = ∅ , the canonical family { p θ , q : θ ∈ Θ , q ∈ Q θ , L } ⊆ P L , ν \{p_{\theta,q}:\theta\in\Theta,\ q\in\mathcal{Q}_{\theta,L}\}\subseteq\mathcal{P}_{L,\nu} { p θ , q : θ ∈ Θ , q ∈ Q θ , L } ⊆ P L , ν contains a population-optimal fair policy for any context law P X P_X P X . This dimension reduction both motivates our algorithms and explains why the nonparametric learning complexity in § 4 is governed by the intrinsic boundary dimension d − 1 d-1 d − 1 instead of the ambient dimension d d d .
Optimal fair policy with incomplete information
In this section, we study online learning of the optimal Lipschitz fair policy under incomplete information, where both the arm-specific reward parameters { β 1 ∗ , β 2 ∗ } \{\beta_1^*,\beta_2^*\} { β 1 ∗ , β 2 ∗ } and the context distribution P X P_X P X are unknown. A distinctive feature of our problem is that, whereas the unconstrained oracle p u n ∗ p_{\mathrm{un}}^* p un ∗ in (2) depends solely on θ ∗ = β 1 ∗ − β 2 ∗ \theta^*=\beta_1^*-\beta_2^* θ ∗ = β 1 ∗ − β 2 ∗ , an optimal fair policy p ∗ p^* p ∗ also depends on P X P_X P X , and the optimal boundary policy values q ∗ q^* q ∗ in (7) balances reward gains on opposite sides of H θ ∗ \mathcal{H}_{\theta^*} H θ ∗ weighted by the covariate distribution near that boundary. Consequently, even if the reward parameters ( β 1 ∗ , β 2 ∗ ) (\beta_1^*,\beta_2^*) ( β 1 ∗ , β 2 ∗ ) were known, identifying an optimal fair policy p ∗ p^* p ∗ would still require functional information about the unknown context distribution P X P_X P X . This introduces an additional nonparametric component to the learning problem, despite the linear specification of the reward functions.
In the rest of this section, we first introduce a lazy fair greedy algorithm for continuously distributed contexts in § 4.1 , then establish its fairness and regret guarantees in § 4.2 . In § 4.3 , we relax the density assumption to allow for general context distributions, including discrete and mixed contexts, and develop a UCB-based algorithm with matching theoretical guarantees.
Policy learning over continuous feature spaces
Building upon the canonical representation in Theorem 3 and the fairness-induced exploration property established in Lemma 29 , we introduce in Algorithm 1, a lazy fair greedy algorithm for learning under incomplete information. We focus on the case d ≥ 2 d \ge2 d ≥ 2 , where the policy is constructed from values on an estimated ( d − 1 ) (d-1) ( d − 1 ) -dimensional decision boundary. The analogous algorithm for d = 1 d=1 d = 1 , where the boundary is a single point, is given in Algorithm 5 in Appendix.
There are two main components in Algorithm 1: an initial exploration phase followed by an adaptive lazy policy-update phase.
Exploration. During the first 2 T 0 2T_0 2 T 0 rounds, the algorithm uses the constant policy p t ( ⋅ ) = 1 / 2 p_t(\cdot)=1/2 p t ( ⋅ ) = 1/2 and samples each arm with equal probability, independently across rounds and of the contexts and reward noise. This provides sufficient data to accurately estimate both arm-specific parameters β 1 ∗ \beta_1^* β 1 ∗ and β 2 ∗ \beta_2^* β 2 ∗ with high probability (Lemma 24), and avoids projecting subsequent estimators θ ^ s \hat{\theta}_s θ ^ s onto Θ \Theta Θ , which would require prior knowledge of the constants in Assumption 4. Note that p t ( ⋅ ) = 1 / 2 p_t(\cdot)=1/2 p t ( ⋅ ) = 1/2 trivially belongs to P L , ν \mathcal{P}_{L,\nu} P L , ν , which is fair.
Adaptive lazy updates. After the exploration phase, the algorithm updates the ridge estimators and constructs a fair plug-in policy using the canonical representation in Theorem 3 . Under continuous contexts in Assumption 1 and the boundary regularity conditions in Assumption 4, any canonical policy automatically induces a randomization region with positive Lebesgue measure and thus ensures sufficient covariate diversity for both arms (Lemma 26 ). This property guarantees that both arms can continue to receive sufficiently informative observations, enabling estimation of their reward parameters during the greedy exploitation phase without additional forced exploration (Lemma 29 ).
Policy updates are performed lazily using a dyadic schedule with the policy p ^ \hat{p} p ^ fixed between two consecutive update rounds. This ensures that the linear program in (9) is solved at most ⌈ log 2 ( T ) ⌉ \lceil\log_2(T)\rceil ⌈ log 2 ( T )⌉ times, which relieves the computational burden. Similar ideas based on lazy updates have been commonly used in related problems to reduce the number of policy updates while preserving regret guarantees, including linear stochastic bandits (e.g. [1 ] ) and fair dynamic pricing ([14 ] ).
At each update time s s s , we fix θ ^ s \hat{\theta}_s θ ^ s and chooses the boundary values q ∈ Q θ ^ s , L q\in\mathcal{Q}_{\hat{\theta}_s,L} q ∈ Q θ ^ s , L that maximise the empirical plug-in reward gap,
q ^ s ∈ arg max q ∈ Q θ ^ s , L 1 s − 1 ∑ i = 1 s − 1 X ~ i T θ ^ s p θ ^ s , q ( X i ) , (8) \hat{q}_s \in\arg\max_{q\in\mathcal{Q}_{\hat{\theta}_s,L}} \frac{1}{s-1}\sum_{i=1}^{s-1}\widetilde{X}_i^{\mathsf{T}}\hat{\theta}_s p_{\hat{\theta}_s,q}(X_i), \tag*{(8)} q ^ s ∈ arg q ∈ Q θ ^ s , L max s − 1 1 i = 1 ∑ s − 1 X i T θ ^ s p θ ^ s , q ( X i ) , (8)
Algorithm 1 Lazy fair greedy algorithm with boundary learning (d ≥ 2 d \ge2 d ≥ 2 )
INPUT: Time horizon T ∈ Z + T \in\mathbb{Z}_{+} T ∈ Z + , fairness parameter ( L , ν ) (L,\nu) ( L , ν ) , ridge parameter λ > 0 \lambda> 0 λ > 0 , exploration period T o ∈ Z + T_o \in\mathbb{Z}_{+} T o ∈ Z + .
for k ∈ { 1 , 2 } k \in\{1,2\} k ∈ { 1 , 2 } do
V k ← λ I d + 1 V_k \leftarrow\lambda I_{d+1} V k ← λ I d + 1 , b k ← ( 0 , … , 0 ) T ∈ R d + 1 b_k \leftarrow(0,\ldots,0)^{\mathsf{T}} \in\mathbb{R}^{d+1} b k ← ( 0 , … , 0 ) T ∈ R d + 1 .
end for
Set s ← T o s \leftarrow T_o s ← T o .
Exploration:
for t ∈ { 1 , 2 , … , 2 T o } t \in\{1,2,\ldots,2T_o\} t ∈ { 1 , 2 , … , 2 T o } do
Observe X t X_t X t , set X ~ t = ( 1 , X t T ) T \widetilde{X}_t = (1,X_t^{\mathsf{T}})^{\mathsf{T}} X t = ( 1 , X t T ) T .
Sample A t = 1 A_t=1 A t = 1 with probability 1 / 2 1/2 1/2 and A t = 2 A_t=2 A t = 2 otherwise. Observe Y t Y_t Y t .
Update V A t ← V A t + X ~ t X ~ t T V_{A_t} \leftarrow V_{A_t}+\widetilde{X}_t\widetilde{X}_t^{\mathsf{T}} V A t ← V A t + X t X t T and b A t ← b A t + X ~ t Y t b_{A_t} \leftarrow b_{A_t}+\widetilde{X}_tY_t b A t ← b A t + X t Y t .
end for
Adaptive lazy update:
for t ∈ { 2 T o + 1 , … , T } t \in\{2T_o+1,\ldots,T\} t ∈ { 2 T o + 1 , … , T } do
Observe X t X_t X t and set X ~ t = ( 1 , X t T ) T \widetilde{X}_t=(1,X_t^{\mathsf{T}})^{\mathsf{T}} X t = ( 1 , X t T ) T .
if t ≥ 2 s t \ge2s t ≥ 2 s then
Set s ← t s \leftarrow t s ← t .
for k ∈ { 1 , 2 } k \in\{1,2\} k ∈ { 1 , 2 } do
Set β ^ k , s − 1 ← V k − 1 b k \widehat{\beta}_{k,s-1} \leftarrow V_k^{-1}b_k β k , s − 1 ← V k − 1 b k .
end for
Set θ ^ s ← β ^ 1 , s − 1 − β ^ 2 , s − 1 \widehat{\theta}_s \leftarrow\widehat{\beta}_{1,s-1}-\widehat{\beta}_{2,s-1} θ s ← β 1 , s − 1 − β 2 , s − 1 .
Obtain p ^ s \widehat{p}_s p s from Algorithm 2 and set p ^ ← p ^ s \widehat{p} \leftarrow\widehat{p}_s p ← p s .
end if
Sample A t = 1 A_t=1 A t = 1 with probability p ^ ( X t ) \widehat{p}(X_t) p ( X t ) and A t = 2 A_t=2 A t = 2 otherwise. Observe Y t Y_t Y t .
Update V A t ← V A t + X ~ t X ~ t T V_{A_t} \leftarrow V_{A_t}+\widetilde{X}_t\widetilde{X}_t^{\mathsf{T}} V A t ← V A t + X t X t T and b A t ← b A t + X ~ t Y t b_{A_t} \leftarrow b_{A_t}+\widetilde{X}_tY_t b A t ← b A t + X t Y t .
end for
Algorithm 2 Boundary optimisation at update time s s s
INPUT: X 1 , … , X s − 1 X_1,\ldots,X_{s-1} X 1 , … , X s − 1 , θ ^ s \widehat{\theta}_s θ s , L L L , and ν \nu ν .
Solve the finite-dimensional linear program
z ^ s = ( z ^ s , 1 , … , z ^ s , s − 1 ) ∈ arg max z 1 , … , z s − 1 1 s − 1 ∑ i = 1 s − 1 X ~ i T θ ^ s z i \widehat{z}_s=(\widehat{z}_{s,1},\ldots,\widehat{z}_{s,s-1}) \in\underset{z_1,\ldots,z_{s-1}}{\arg\max}\ \frac{1}{s-1}\sum_{i=1}^{s-1}\widetilde{X}_i^{\mathsf{T}}\widehat{\theta}_s z_i z s = ( z s , 1 , … , z s , s − 1 ) ∈ z 1 , … , z s − 1 arg max s − 1 1 i = 1 ∑ s − 1 X i T θ s z i
s.t. 0 ≤ z i ≤ 1 , i = 1 , … , s − 1 , \text{s.t.}\quad0 \le z_i \le1,\qquad i=1,\ldots,s-1, s.t. 0 ≤ z i ≤ 1 , i = 1 , … , s − 1 ,
∣ z i − z j ∣ ≤ L ∥ X i − X j ∥ ν , i , j = 1 , … , s − 1. (9) \left|z_i-z_j\right| \le L\left\|X_i-X_j\right\|^\nu,\qquad i,j=1,\ldots,s-1. \tag*{(9)} ∣ z i − z j ∣ ≤ L ∥ X i − X j ∥ ν , i , j = 1 , … , s − 1. (9)
Set p ^ s M c E x t ( x ) ← min { 1 , min 1 ≤ i ≤ s − 1 { z ^ s , i + L ∥ x − X i ∥ ν } } \widehat{p}^{\mathrm{McExt}}_s(x) \leftarrow\min\{1,\min_{1\le i\le s-1}\{\widehat{z}_{s,i}+L\|x-X_i\|^\nu\}\} p s McExt ( x ) ← min { 1 , min 1 ≤ i ≤ s − 1 { z s , i + L ∥ x − X i ∥ ν }} for x ∈ [ 0 , 1 ] d x \in[0,1]^d x ∈ [ 0 , 1 ] d .
Set q ^ s ← p ^ s M c E x t ∣ H θ ^ s \widehat{q}_s \leftarrow\widehat{p}^{\mathrm{McExt}}_s|_{\mathcal{H}_{\widehat{\theta}_s}} q s ← p s McExt ∣ H θ s and p ^ s ← p θ ^ s , q ^ s \widehat{p}_s \leftarrow p_{\widehat{\theta}_s,\widehat{q}_s} p s ← p θ s , q s , with p θ ^ s , q ^ s p_{\widehat{\theta}_s,\widehat{q}_s} p θ s , q s defined in (6) .
return p ^ s \widehat{p}_s p s .
where Q θ ^ s , L Q_{\hat{\theta}_s,L} Q θ ^ s , L and p θ , q p_{\theta,q} p θ , q are defined in (4) and (6) , respectively. Although (8) is posed over an infinite-dimensional class of Lipschitz functions on H θ ^ s \mathcal{H}_{\hat{\theta}_s} H θ ^ s , Algorithm 2 computes an exact maximiser via the finite-dimensional linear program (9) without any relaxation gap (Lemma 13).
The linear program in (9) in Algorithm 2 can be interpreted as first finding an optimal discretised fair policy on the observed contexts { X i } i ∈ [ s − 1 ] \{X_i\}_{i\in[s-1]} { X i } i ∈ [ s − 1 ] , where the variables { z ^ s , i } i ∈ [ s − 1 ] \{\hat{z}_{s,i}\}_{i\in[s-1]} { z ^ s , i } i ∈ [ s − 1 ] represent the probability of choosing arm 1 at the sample point X i X_i X i , i.e. p ^ ( X i ) \hat{p}(X_i) p ^ ( X i ) . Once these values on the observed contexts have been determined, the McShane extension then interpolates { z ^ s , i } i ∈ [ s − 1 ] \{\hat{z}_{s,i}\}_{i\in[s-1]} { z ^ s , i } i ∈ [ s − 1 ] to a globally defined ( L , ν ) (L,\nu) ( L , ν ) -Lipschitz policy p ^ M c E x t \hat{p}^{\mathrm{McExt}} p ^ McExt over the entire covariate space. Restricting p ^ M c E x t \hat{p}^{\mathrm{McExt}} p ^ McExt to the decision boundary produces a feasible and optimal boundary function q ^ s ∈ Q θ ^ s , L \hat{q}_s\in Q_{\hat{\theta}_s,L} q ^ s ∈ Q θ ^ s , L whose canonical extension p ^ s = p θ ^ s , q ^ s \hat{p}_s=p_{\hat{\theta}_s,\hat{q}_s} p ^ s = p θ ^ s , q ^ s solves the infinite-dimensional problem (8) .
Remark 6 (Equivalence between (9) and (8) ). The equivalence can be derived as follows.
The optimal value of (9) provides an upper bound on that of (8) . By Theorem 3 , every feasible boundary function q ∈ Q θ ^ s , L q\in Q_{\hat{\theta}_s,L} q ∈ Q θ ^ s , L induces an ( L , ν ) (L,\nu) ( L , ν ) -Lipschitz canonical policy p θ ^ s , q p_{\hat{\theta}_s,q} p θ ^ s , q , whose sample values z i = p θ ^ s , q ( X i ) z_i=p_{\hat{\theta}_s,q}(X_i) z i = p θ ^ s , q ( X i ) , i ∈ [ s − 1 ] i\in[s-1] i ∈ [ s − 1 ] , satisfy the constraints in (9) . Consequently, (9) is a relaxation of (8) , and provides an upper bound.
The optimal value of (8) provides an upper bound on that of (9) . The empirical Lipschitz constraints in (9) allow its linear programming solution { z ^ s , i } i ∈ [ s − 1 ] \{\hat{z}_{s,i}\}_{i\in[s-1]} { z ^ s , i } i ∈ [ s − 1 ] to be extended to a globally fair McShane extension which satisfies p ^ s M c E x t ( X i ) = z ^ s , i \hat{p}^{\mathrm{McExt}}_s(X_i)=\hat{z}_{s,i} p ^ s McExt ( X i ) = z ^ s , i for all i ∈ [ s − 1 ] i\in[s-1] i ∈ [ s − 1 ] . Restricting p ^ s M c E x t \hat{p}^{\mathrm{McExt}}_s p ^ s McExt to the estimated decision boundary then yields q ^ s = p ^ s M c E x t ∣ H θ ^ s ∈ Q θ ^ s , L \hat{q}_s=\hat{p}^{\mathrm{McExt}}_s|_{\mathcal{H}_{\hat{\theta}_s}}\in Q_{\hat{\theta}_s,L} q ^ s = p ^ s McExt ∣ H θ ^ s ∈ Q θ ^ s , L . By Theorem 3 , ( X ~ i ⊤ θ ^ s ) p θ ^ s , q ^ s ( X i ) ≥ ( X ~ i ⊤ θ ^ s ) p ^ s M c E x t ( X i ) = ( X ~ i ⊤ θ ^ s ) z ^ s , i (\tilde{X}_i^\top\hat{\theta}_s)p_{\hat{\theta}_s,\hat{q}_s}(X_i)\ge(\tilde{X}_i^\top\hat{\theta}_s)\hat{p}^{\mathrm{McExt}}_s(X_i)=(\tilde{X}_i^\top\hat{\theta}_s)\hat{z}_{s,i} ( X ~ i ⊤ θ ^ s ) p θ ^ s , q ^ s ( X i ) ≥ ( X ~ i ⊤ θ ^ s ) p ^ s McExt ( X i ) = ( X ~ i ⊤ θ ^ s ) z ^ s , i , so (8) attains at least the optimal value of (9) .
Remark 7 (Computational complexity). Over the T T T horizon, the computational complexity of Algorithm 1 scales as O ( T 2 + log ( T ) Cost L P ( T , T 2 ) + T Cost c a n o n i c a l ) O(T^2+\log(T)\operatorname{Cost}_{\mathrm{LP}}(T,T^2)+T\operatorname{Cost}_{\mathrm{canonical}}) O ( T 2 + log ( T ) Cost LP ( T , T 2 ) + T Cost canonical ) , where T 2 T^2 T 2 is the cost of computing T 2 T^2 T 2 constraints in Algorithm 2, Cost L P ( T , T 2 ) \operatorname{Cost}_{\mathrm{LP}}(T,T^2) Cost LP ( T , T 2 ) is the cost of solving a linear program with T T T variables and T 2 T^2 T 2 constraints, and Cost c a n o n i c a l \operatorname{Cost}_{\mathrm{canonical}} Cost canonical is the cost of evaluating p ^ s ( X t ) \hat{p}_s(X_t) p ^ s ( X t ) via (6) in Algorithm 2, which involves the global McShane extension, restriction to the boundary, and then canonical extension. Computational details under the ℓ 1 \ell_1 ℓ 1 -metric are provided in Appendix § 15 .
Fairness and regret control
In this subsection, we present the main theorem analysing the fairness and regret of Algorithms 1 and 5.
Theorem 4 (Fairness and regret control). Suppose that Assumptions 1–4 hold and L L L is an absolute constant satisfying 0 < L < L 0 ′ 0<L<L'_0 0 < L < L 0 ′ for some L 0 ′ > 0 L'_0>0 L 0 ′ > 0 . For any η ∈ ( 0 , 1 / 2 ) \eta\in(0,1/2) η ∈ ( 0 , 1/2 ) , take λ = C λ \lambda=C_\lambda λ = C λ and T 0 = C 1 [ T log ( T / η ) ] T_0=C_1[\sqrt{T}\log(T/\eta)] T 0 = C 1 [ T log ( T / η )] , where C 1 , C λ > 0 C_1,C_\lambda>0 C 1 , C λ > 0 are absolute constants. Then the following statements hold for Algorithms 1 and 5.
(Fairness.) The collection of policies { p ^ t } t = 1 T \{\hat{p}_t\}_{t=1}^{T} { p ^ t } t = 1 T satisfies ( L , ν ) (L,\nu) ( L , ν ) -Lipschitz fairness, i.e. p ^ t ∈ P L , ν \hat{p}_t\in\mathcal{P}_{L,\nu} p ^ t ∈ P L , ν for all t ∈ [ T ] t\in[T] t ∈ [ T ] .
(Regret.) There exist constants C 2 , C 3 , C 4 > 0 C_2,C_3,C_4>0 C 2 , C 3 , C 4 > 0 , depending only on d , L , c g , C g , σ , C β , λ , ν d,L,c_g,C_g,\sigma,C_\beta,\lambda,\nu d , L , c g , C g , σ , C β , λ , ν and the constants in Assumption 4, such that whenever T ≥ C 4 log ( T / η ) T\ge C_4\log(T/\eta) T ≥ C 4 log ( T / η ) , it holds with probability at least 1 − η 1-\eta 1 − η that
R T ( { p ^ t } t = 1 T ) ≤ C 2 T log ( T / η ) + C 3 { T log ( e T 2 / η ) , d = 1 , 2 , T log 3 2 ( e T 2 / η ) , d = 3 , T d − 2 d − 1 log ( e T 2 / η ) , d ≥ 4. \mathfrak{R}_T(\{\hat{p}_t\}_{t=1}^{T})\le C_2\sqrt{T}\log(T/\eta)+C_3
\begin{cases}
\sqrt{T}\log(eT^2/\eta), & d=1,2,\\
\sqrt{T}\log^{\frac{3}{2}}(eT^2/\eta), & d=3,\\
T^{\frac{d-2}{d-1}}\log(eT^2/\eta), & d\ge4.
\end{cases} R T ({ p ^ t } t = 1 T ) ≤ C 2 T log ( T / η ) + C 3 ⎩ ⎨ ⎧ T log ( e T 2 / η ) , T log 2 3 ( e T 2 / η ) , T d − 1 d − 2 log ( e T 2 / η ) , d = 1 , 2 , d = 3 , d ≥ 4. where R T ( { p ^ t } t = 1 T ) \mathfrak{R}_T(\{\hat{p}_t\}_{t=1}^{T}) R T ({ p ^ t } t = 1 T ) is the accumulated regret defined in (3) .
We defer the detailed proof of Theorem 4 to Appendix § 11 .§ 2 , with a sketch at the end of this subsection. Theorem 4 demonstrates that both Algorithms 1 and 5 satisfy the fairness constraints and achieve a regret sublinear in T T T . Theorem 4 considers a fixed constant fairness parameter L ∈ ( 0 , L 0 ′ ) L\in(0,L'_0) L ∈ ( 0 , L 0 ′ ) , for some L 0 ′ > 0 L'_0 > 0 L 0 ′ > 0 , allowing us to study how regret scales with the time horizon under a fixed fairness requirement.
Up to poly-logarithmic factors, the upper bound on regret in Theorem 4 can be rewritten as
R T = O P ( T + T d − 2 d − 1 1 { d ≥ 4 } ) , \mathfrak{R}_T = O_{\mathsf{P}}\left(\sqrt{T} + T^{\frac{d-2}{d-1}}\mathbf{1}\{d \ge4\}\right), R T = O P ( T + T d − 1 d − 2 1 { d ≥ 4 } ) ,
which at a high level, is of the form
R T = intrinsic bandit regret + fairness-induced regret . \mathfrak{R}_T = \text{intrinsic bandit regret} + \text{fairness-induced regret}. R T = intrinsic bandit regret + fairness-induced regret .
This decomposition highlights the coexistence of a parametric bandit learning cost and a non-parametric fairness cost. The first term, of order T \sqrt{T} T , captures the classical cost of learning finite-dimensional linear reward parameters under bandit feedback, which represents the intrinsic minimax parametric bandit regret (e.g. Dani et al., [20 ] ; Chu et al., [17 ] ). The second term, of order T ( d − 2 ) / ( d − 1 ) T^{(d-2)/(d-1)} T ( d − 2 ) / ( d − 1 ) for d ≥ 4 d \ge4 d ≥ 4 , captures the additional statistical cost of enforcing Lipschitz fairness by learning the unknown ( d − 1 ) (d - 1) ( d − 1 ) -dimensional decision boundary. In particular, for n n n historical contexts, the uniform empirical-process deviation over the ( d − 1 ) (d - 1) ( d − 1 ) -dimensional boundary-canonical Lipschitz class decays at the nonparametric rate n − 1 / ( d − 1 ) n^{-1/(d-1)} n − 1/ ( d − 1 ) when d − 1 ≥ 3 d-1 \ge3 d − 1 ≥ 3 , which sums over n ∈ [ T ] n \in[T] n ∈ [ T ] to O ( T ( d − 2 ) / ( d − 1 ) ) O(T^{(d-2)/(d-1)}) O ( T ( d − 2 ) / ( d − 1 ) ) . Notably, similar dimension-dependent rates also arise in other nonparametric contextual bandit problems involving Lipschitz function classes, e.g. Cesa-Bianchi et al. [10 ] and Mao et al. [39 ] , where the regret scales with the ambient dimension d d d . By contrast, in our setting, Theorem 3 reduces the effective nonparametric dimension to d − 1 d - 1 d − 1 . Consequently, for d ≤ 3 d \le3 d ≤ 3 , the fairness-induced term is absorbed into the parametric T \sqrt{T} T rate up to logarithmic terms (see, e.g., Van Der Vaart and Wellner, [49 ] ). For d ≥ 4 d \ge4 d ≥ 4 , it strictly dominates the overall regret, making learning the fair policy on the boundary the main statistical difficulty.
Sketch of the proof. We conclude this subsection with a sketch of the proof of the regret upper bound in Theorem 4 for d ≥ 2 d \ge2 d ≥ 2 , while the fairness guarantee is a direct consequence of Theorem 3 .
Under the boundedness assumptions in Assumptions 1 and 3, the initial exploration incurs O ( T 0 ) = O log ( T ) O(T_0) = O_{\log}(\sqrt{T}) O ( T 0 ) = O l o g ( T ) regret and localises θ ^ s \hat{\theta}_s θ ^ s within an enlarged regular neighbourhood Θ ~ ⊃ Θ \widetilde{\Theta} \supset\Theta Θ ⊃ Θ across all lazy updates with high probability (Lemma 25 ). In each subsequent lazy update phase starting at s s s , the regret has two sources: (i) estimation error in θ ∗ \theta^* θ ∗ , which leads to a mismatch between the true and estimated decision boundaries H θ ∗ \mathcal{H}_{\theta^*} H θ ∗ and H θ ^ s \mathcal{H}_{\hat{\theta}_s} H θ ^ s , and (ii) empirical-process statistical error from replacing the population reward objective by its empirical plug-in counterpart in (8) . Since the lazy update batches grow geometrically, summing the per-update bounds over all batches inflates the regret only by logarithmic factors.
Specifically, at each update time s s s , we compare the learned policy p ^ s \hat{p}_s p ^ s with the optimal fair policy p ∗ p^* p ∗ through an intermediate policy defined by the canonical extension over the estimated boundary. Let q s † = p ∗ ∣ H θ ^ s q_s^\dagger= p^*|_{\mathcal{H}_{\hat{\theta}_s}} q s † = p ∗ ∣ H θ ^ s be the restriction of the oracle policy to the estimated decision boundary, and define p s † = p θ ^ s , q s † p_s^\dagger= p_{\hat{\theta}_s,q_s^\dagger} p s † = p θ ^ s , q s † . Since p ∗ p^* p ∗ is ( L , ν ) (L,\nu) ( L , ν ) -Lipschitz, it holds that q s † ∈ Q θ ^ s , L q_s^\dagger\in\mathcal{Q}_{\hat{\theta}_s,L} q s † ∈ Q θ ^ s , L , and p s † p_s^\dagger p s † is a feasible competitor for the optimisation problem in (8) . The instantaneous regret is then decomposed as
R ( p ∗ ) − R ( p ^ s ) = { R ( p ∗ ) − R ( p s † ) } + { R ( p s † ) − R ( p ^ s ) } = boundary mismatch error + empirical reward estimation error . \begin{aligned}
R(p^*) - R(\hat{p}_s) = \{R(p^*) - R(p_s^\dagger)\} + \{R(p_s^\dagger) - R(\hat{p}_s)\} \\
&= \text{boundary mismatch error} + \text{empirical reward estimation error}.
\end{aligned} R ( p ∗ ) − R ( p ^ s ) = { R ( p ∗ ) − R ( p s † )} + { R ( p s † ) − R ( p ^ s )} = boundary mismatch error + empirical reward estimation error .
Boundary mismatch error: To bound R ( p ∗ ) − R ( p s † ) R(p^*) - R(p_s^\dagger) R ( p ∗ ) − R ( p s † ) , we partition X \mathcal{X} X according to whether x x x lies on the same side or opposite sides of H θ ∗ \mathcal{H}_{\theta^*} H θ ∗ and H θ ^ s \mathcal{H}_{\hat{\theta}_s} H θ ^ s . On the same-side event, both p ∗ ( x ) p^*(x) p ∗ ( x ) and p s † ( x ) p_s^\dagger(x) p s † ( x ) use the same envelope type, hence ∣ p ∗ ( x ) − p s † ( x ) ∣ |p^*(x) - p_s^\dagger(x)| ∣ p ∗ ( x ) − p s † ( x ) ∣ can be upper bounded by O ( ∥ θ ^ s − θ ∗ ∥ 2 ) O(\|\hat{\theta}_s - \theta^*\|_2) O ( ∥ θ ^ s − θ ∗ ∥ 2 ) using the Lipschitz property (Steps 2 and 3 in Lemma 17 ) and Hausdorff stability of H θ ^ s \mathcal{H}_{\hat{\theta}_s} H θ ^ s (Lemma 18). On the opposite-side event, since ∣ p ∗ ( x ) − p s † ( x ) ∣ ≤ 1 |p^*(x) - p_s^\dagger(x)| \le1 ∣ p ∗ ( x ) − p s † ( x ) ∣ ≤ 1 , we use a crude pointwise bound of order O ( 1 ) O(1) O ( 1 ) . This
event has small probability due to the stability of the decision boundary. Lemma 18 controls the Hausdorff distance between the true and estimated decision boundaries by O ( ∥ θ ^ s − θ ∗ ∥ 2 ) O(\lVert\hat{\theta}_s-\theta^*\rVert_2) O (∥ θ ^ s − θ ∗ ∥ 2 ) , confining the sign disagreement to a thin slab of the covariate around the true boundary under Assumption 1 (Lemma 19 ). Combining the Lipschitz control on the same-side event with the slab probability bound on the sign-disagreement event shows that the loss caused by replacing H θ s ∗ H_{\theta_s^*} H θ s ∗ with H θ ^ s H_{\hat{\theta}_s} H θ ^ s is only of the same order as the parameter estimation error ∥ θ ^ s − θ ∗ ∥ 2 \lVert\hat{\theta}_s-\theta^*\rVert_2 ∥ θ ^ s − θ ∗ ∥ 2 (Lemma 17).
Empirical reward estimation error: Using the empirical optimality of q ^ s \hat{q}_s q ^ s (and hence p ^ s \hat{p}_s p ^ s ) in (8) , and that ∣ X ~ i T ( θ ^ s − θ ∗ ) ∣ ≲ ∥ θ ^ s − θ ∗ ∥ 2 \lvert\widetilde{X}_i^{\mathsf{T}}(\hat{\theta}_s-\theta^*)\rvert\lesssim\lVert\hat{\theta}_s-\theta^*\rVert_2 ∣ X i T ( θ ^ s − θ ∗ )∣ ≲ ∥ θ ^ s − θ ∗ ∥ 2 , then up to the estimation error of O ( ∥ θ ^ s − θ ∗ ∥ 2 ) O(\lVert\hat{\theta}_s-\theta^*\rVert_2) O (∥ θ ^ s − θ ∗ ∥ 2 ) , the term R ( p s ∗ ) − R ( p ^ s ) R(p_s^*)-R(\hat{p}_s) R ( p s ∗ ) − R ( p ^ s ) can be bounded by the uniform empirical deviation over the canonical policy class,
sup θ ∈ Θ , q ∈ Q θ , L ∣ 1 n ∑ i = 1 n X ~ i T θ p θ , q ( X i ) − E { X ~ T θ p θ , q ( X ) } ∣ (10) \sup_{\theta\in\Theta,\ q\in\mathcal{Q}_{\theta,L}}\left\lvert\frac{1}{n}\sum_{i=1}^{n}\widetilde{X}_i^{\mathsf{T}}\theta p_{\theta,q}(X_i)-\mathbb{E}\left\{\widetilde{X}^{\mathsf{T}}\theta p_{\theta,q}(X)\right\}\right\rvert \tag*{(10)} θ ∈ Θ , q ∈ Q θ , L sup n 1 i = 1 ∑ n X i T θ p θ , q ( X i ) − E { X T θ p θ , q ( X ) } (10)
Since Q θ ^ s , L \mathcal{Q}_{\hat{\theta}_s,L} Q θ ^ s , L is parametrised by functions living only on the decision boundary, the complexity of Q θ ^ s , L \mathcal{Q}_{\hat{\theta}_s,L} Q θ ^ s , L scales with the intrinsic boundary dimension d − 1 d-1 d − 1 , instead of the ambient dimension d d d (Lemma 22 ). This low-dimensional representation reduces the learning problem to an L L L -Lipschitz function class on the ( d − 1 ) (d-1) ( d − 1 ) -dimensional decision boundary, giving bracketing entropy of order O log ( ϵ − ( d − 1 ) ) O_{\log}(\epsilon^{-(d-1)}) O l o g ( ϵ − ( d − 1 ) ) . Letting n n n be the effective sample size, the bracketing maximal inequality ([49 Theorem 2.14.2] ) leads to a high probability bound on (10) , which is of order O log ( n − 1 / 2 ) O_{\log}(n^{-1/2}) O l o g ( n − 1/2 ) when d ∈ { 2 , 3 } d\in\{2,3\} d ∈ { 2 , 3 } , and O log ( n − 1 / ( d − 1 ) ) O_{\log}(n^{-1/(d-1)}) O l o g ( n − 1/ ( d − 1 ) ) when d ≥ 4 d\ge4 d ≥ 4 . Summing over the dyadic schedule gives the rate in Lemma 23 and explains the dimension-dependent term in Theorem 4.
Extensions beyond continuous covariates
Our analysis so far is established for continuously distributed contexts, detailed in Assumption 1. It allows us to control the probability mass near the decision boundary, crucial in quantifying the cost of fairness in § 3.2 . It also guarantees sufficient covariate diversity for both arms under the canonical policies, enabling the greedy learning approach and its regret guarantees in § 4.1 and § 4.2 . To allow discrete and mixed discrete-continuous covariates, we relax Assumption 1 as follows.
Assumption 1′ (General contexts). The contexts X 1 , … , X T X_1,\ldots,X_T X 1 , … , X T are i.i.d. from an arbitrary, unknown Borel probability law P X P_X P X supported on X = [ 0 , 1 ] d \mathcal{X}=[0,1]^d X = [ 0 , 1 ] d . Let S = supp ( P X ) \mathcal{S}=\operatorname{supp}(P_X) S = supp ( P X ) and Σ X = E ( X ~ 1 X ~ 1 T ) \Sigma_X=\mathbb{E}(\widetilde{X}_1\widetilde{X}_1^{\mathsf{T}}) Σ X = E ( X 1 X 1 T ) . We assume that there exists an absolute constant κ X > 0 \kappa_X>0 κ X > 0 such that λ min ( Σ X ) ≥ κ X \lambda_{\min}(\Sigma_X)\ge\kappa_X λ m i n ( Σ X ) ≥ κ X .
Assumption 1′ permits continuous, discrete and mixed context distributions, and the support S \mathcal{S} S can be unknown. We continue to define fair policies on the entire covariate space X \mathcal{X} X without loss of generality, since any Lipschitz fair policy defined on S \mathcal{S} S can be extended to X \mathcal{X} X while preserving its Lipschitz constant and its values on S \mathcal{S} S (Lemma 67 in Appendix § 14 ). Conversely, any globally defined fair policy remains fair when restricted to S \mathcal{S} S . Since X ∈ S X\in\mathcal{S} X ∈ S almost surely, the expected reward depends only on policy values on S \mathcal{S} S , and hence the two formulations yield the same optimal reward. Even when the decision boundary H θ H_{\theta} H θ does not intersect the support S \mathcal{S} S , the canonical representation in Theorem 3 remains valid for arbitrary context distributions, since its construction relies on the Lipschitz geometry rather than the distribution of contexts. We defer structural results under general context distributions, including the boundary randomisation property restricted to S \mathcal{S} S , to Appendix § 14 .
Although the structural characterisation remains applicable, the covariate diversity property underlying the greedy algorithm is no longer guaranteed under Assumption 1′: for instance, S \mathcal{S} S may place zero probability mass inside the randomisation region around H θ ∗ H_{\theta_*} H θ ∗ , causing a plug-in canonical policy to act deterministically on S \mathcal{S} S and under-explore one arm. To overcome this difficulty, we develop a UCB-based learning algorithm that explicitly accounts for uncertainty in the reward parameters, achieving fairness and regret guarantees for general context distributions. For d ≥ 2 d\ge2 d ≥ 2 , the algorithm
Algorithm 3 Lazy polyhedral UCB based algorithm (d ≥ 2 d \ge2 d ≥ 2 )
INPUT: T , T o ∈ N T, T_o \in\mathbb{N} T , T o ∈ N , 2 T o < T 2T_o < T 2 T o < T , λ > 0 \lambda> 0 λ > 0 , failure probability η ∈ ( 0 , 1 ) \eta\in(0,1) η ∈ ( 0 , 1 ) , σ \sigma σ , C β C_\beta C β , L L L , ν \nu ν .
Set s 0 ← 2 T o + 1 s_0 \leftarrow2T_o+1 s 0 ← 2 T o + 1 and tolerance η c ← η / 8 \eta_c \leftarrow\eta/8 η c ← η /8 . Define
α T ← 2 σ 2 { log ( 2 η c ) + d + 1 2 log ( 1 + T λ ) } + C β λ . (11) \alpha_T \leftarrow\sqrt{2\sigma^2\left\{\log\left(\frac{2}{\eta_c}\right)+\frac{d+1}{2}\log\left(1+\frac{T}{\lambda}\right)\right\}}+C_\beta\sqrt{\lambda}. \tag*{(11)} α T ← 2 σ 2 { log ( η c 2 ) + 2 d + 1 log ( 1 + λ T ) } + C β λ . (11)
Initialise V k ← λ I d + 1 V_k \leftarrow\lambda I_{d+1} V k ← λ I d + 1 , B k ← 0 B_k \leftarrow0 B k ← 0 , and ℓ k ← log det ( V k ) \ell_k \leftarrow\log\det(V_k) ℓ k ← log det ( V k ) for k ∈ { 1 , 2 } k \in\{1,2\} k ∈ { 1 , 2 } .
For t ≥ 1 t \ge1 t ≥ 1 and k ∈ { 1 , 2 } k \in\{1,2\} k ∈ { 1 , 2 } , write
V t , k = λ I d + 1 + ∑ i = 1 t − 1 X ~ i X ~ i T 1 { A i = k } , β ^ t , k = V t , k − 1 ∑ i = 1 t − 1 X ~ i Y i 1 { A i = k } . V_{t,k}=\lambda I_{d+1}+\sum_{i=1}^{t-1}\widetilde{X}_i\widetilde{X}_i^{\mathsf T}\mathbf{1}\{A_i=k\},\qquad
\widehat{\beta}_{t,k}=V_{t,k}^{-1}\sum_{i=1}^{t-1}\widetilde{X}_iY_i\mathbf{1}\{A_i=k\}. V t , k = λ I d + 1 + i = 1 ∑ t − 1 X i X i T 1 { A i = k } , β t , k = V t , k − 1 i = 1 ∑ t − 1 X i Y i 1 { A i = k } .
for t = 1 , … , 2 T o t=1,\ldots,2T_o t = 1 , … , 2 T o do
Observe X t X_t X t and set X ~ t = ( 1 , X t T ) T \widetilde{X}_t=(1,X_t^{\mathsf T})^{\mathsf T} X t = ( 1 , X t T ) T .
Set p ^ t ( ⋅ ) ← 1 / 2 \widehat{p}_t(\cdot)\leftarrow1/2 p t ( ⋅ ) ← 1/2 , sample A t = 1 A_t=1 A t = 1 with probability 1 / 2 1/2 1/2 and A t = 2 A_t=2 A t = 2 otherwise, and observe Y t Y_t Y t .
Update
V A t ← V A t + X ~ t X ~ t T , B A t ← B A t + X ~ t Y t , ℓ A t ← log det ( V A t ) . V_{A_t}\leftarrow V_{A_t}+\widetilde{X}_t\widetilde{X}_t^{\mathsf T},\qquad B_{A_t}\leftarrow B_{A_t}+\widetilde{X}_tY_t,\qquad\ell_{A_t}\leftarrow\log\det(V_{A_t}). V A t ← V A t + X t X t T , B A t ← B A t + X t Y t , ℓ A t ← log det ( V A t ) .
end for
Set s ← s 0 s\leftarrow s_0 s ← s 0 and for k ∈ { 1 , 2 } k\in\{1,2\} k ∈ { 1 , 2 } , set V s , k ← V k V_{s,k}\leftarrow V_k V s , k ← V k and ℓ s , k ← ℓ k \ell_{s,k}\leftarrow\ell_k ℓ s , k ← ℓ k .
for t = s 0 , … , T t=s_0,\ldots,T t = s 0 , … , T do
if t = s 0 t=s_0 t = s 0 or t ≥ 2 s t\ge2s t ≥ 2 s or ∃ k ∈ { 1 , 2 } : ℓ k − ℓ s , k ≥ log ( 2 ) \exists k\in\{1,2\}:\ell_k-\ell_{s,k}\ge\log(2) ∃ k ∈ { 1 , 2 } : ℓ k − ℓ s , k ≥ log ( 2 ) then
Set s ← t s\leftarrow t s ← t and and, for k ∈ { 1 , 2 } k\in\{1,2\} k ∈ { 1 , 2 } , set V s , k ← V k V_{s,k}\leftarrow V_k V s , k ← V k , β ^ s , k ← V k − 1 B k \widehat{\beta}_{s,k}\leftarrow V_k^{-1}B_k β s , k ← V k − 1 B k , and ℓ s , k ← ℓ k \ell_{s,k}\leftarrow\ell_k ℓ s , k ← ℓ k . Obtain p s p_s p s from Algorithm 4 and set p ^ ← p s \widehat{p}\leftarrow p_s p ← p s .
end if
Observe X t X_t X t and set X ~ t = ( 1 , X t T ) T \widetilde{X}_t=(1,X_t^{\mathsf T})^{\mathsf T} X t = ( 1 , X t T ) T .
Sample A t = 1 A_t=1 A t = 1 with probability p ^ ( X t ) \widehat{p}(X_t) p ( X t ) and A t = 2 A_t=2 A t = 2 otherwise, and observe Y t Y_t Y t .
Update
V A t ← V A t + X ~ t X ~ t T , B A t ← B A t + X ~ t Y t , ℓ A t ← log det ( V A t ) . V_{A_t}\leftarrow V_{A_t}+\widetilde{X}_t\widetilde{X}_t^{\mathsf T},\qquad B_{A_t}\leftarrow B_{A_t}+\widetilde{X}_tY_t,\qquad\ell_{A_t}\leftarrow\log\det(V_{A_t}). V A t ← V A t + X t X t T , B A t ← B A t + X t Y t , ℓ A t ← log det ( V A t ) .
end for
Algorithm 4 Vertex optimisation at update time s s s
INPUT: X 1 , … , X s − 1 X_1,\ldots,X_{s-1} X 1 , … , X s − 1 , V s , k V_{s,k} V s , k , β ^ s , k \hat{\beta}_{s,k} β ^ s , k for k ∈ { 1 , 2 } k \in\{1,2\} k ∈ { 1 , 2 } , α T \alpha_T α T , L L L , and ν \nu ν .
For k ∈ { 1 , 2 } k \in\{1,2\} k ∈ { 1 , 2 } , form the vertex set as
V s , k = { β ^ s , k ± d + 1 α T V s , k − 1 / 2 e r : r ∈ [ d + 1 ] } , (12) \mathcal{V}_{s,k}=\left\{\hat{\beta}_{s,k}\pm\sqrt{d+1}\alpha_TV_{s,k}^{-1/2}e_r:r\in[d+1]\right\}, \tag*{(12)} V s , k = { β ^ s , k ± d + 1 α T V s , k − 1/2 e r : r ∈ [ d + 1 ] } , (12)
where e r e_r e r is the r r r th standard basis vector and V s , k − 1 / 2 V_{s,k}^{-1/2} V s , k − 1/2 is the symmetric inverse square root.
for each pair v = ( v 1 , v 2 ) ∈ V s , 1 × V s , 2 v=(v_1,v_2)\in\mathcal{V}_{s,1}\times\mathcal{V}_{s,2} v = ( v 1 , v 2 ) ∈ V s , 1 × V s , 2 do
Set θ v = v 1 − v 2 \theta_v=v_1-v_2 θ v = v 1 − v 2 and solve (with Z s − 1 , L \mathcal{Z}_{s-1,L} Z s − 1 , L defined in (17) )
z v ∈ arg max z ∈ Z s − 1 , L 1 s − 1 ∑ i = 1 s − 1 ( X ~ i T θ v ) z i . (13) z_v\in\mathop{\arg\max}_{z\in\mathcal{Z}_{s-1,L}}\frac{1}{s-1}\sum_{i=1}^{s-1}(\widetilde{X}_i^{\mathsf{T}}\theta_v)z_i. \tag*{(13)} z v ∈ arg max z ∈ Z s − 1 , L s − 1 1 i = 1 ∑ s − 1 ( X i T θ v ) z i . (13)
Set
S v = 1 s − 1 ∑ i = 1 s − 1 X ~ i T v 2 + 1 s − 1 ∑ i = 1 s − 1 ( X ~ i T θ v ) z v , i . S_v=\frac{1}{s-1}\sum_{i=1}^{s-1}\widetilde{X}_i^{\mathsf{T}}v_2+\frac{1}{s-1}\sum_{i=1}^{s-1}(\widetilde{X}_i^{\mathsf{T}}\theta_v)z_{v,i}. S v = s − 1 1 i = 1 ∑ s − 1 X i T v 2 + s − 1 1 i = 1 ∑ s − 1 ( X i T θ v ) z v , i .
end for
Select
v ^ = ( v ^ 1 , v ^ 2 ) ∈ arg max v ∈ V s , 1 × V s , 2 S v . \hat{v}=(\hat{v}_1,\hat{v}_2)\in\mathop{\arg\max}_{v\in\mathcal{V}_{s,1}\times\mathcal{V}_{s,2}}S_v. v ^ = ( v ^ 1 , v ^ 2 ) ∈ arg max v ∈ V s , 1 × V s , 2 S v .
Set β ˉ s = ( β ˉ s , 1 , β ˉ s , 2 ) = ( v ^ 1 , v ^ 2 ) \bar{\beta}_s=(\bar{\beta}_{s,1},\bar{\beta}_{s,2})=(\hat{v}_1,\hat{v}_2) β ˉ s = ( β ˉ s , 1 , β ˉ s , 2 ) = ( v ^ 1 , v ^ 2 ) and define θ s = β ˉ s , 1 − β ˉ s , 2 \theta_s=\bar{\beta}_{s,1}-\bar{\beta}_{s,2} θ s = β ˉ s , 1 − β ˉ s , 2 . Let z s = z v ^ z_s=z_{\hat{v}} z s = z v ^ be the optimiser of (13) corresponding to the selected vertex pair v ^ \hat{v} v ^ , with z s , i = z v ^ , i z_{s,i}=z_{\hat{v},i} z s , i = z v ^ , i for i ∈ [ s − 1 ] i\in[s-1] i ∈ [ s − 1 ] .
Define
p ˉ s ( x ) = min { 1 , min 1 ≤ i ≤ s − 1 [ z s , i + L ∥ x − X i ∥ ν ] } . \bar{p}_s(x)=\min\left\{1,\min_{1\leq i\leq s-1}\left[z_{s,i}+L\lVert x-X_i\rVert^\nu\right]\right\}. p ˉ s ( x ) = min { 1 , 1 ≤ i ≤ s − 1 min [ z s , i + L ∥ x − X i ∥ ν ] } .
Construct q s = p ˉ s ∣ H θ s q_s=\bar{p}_s|_{H_{\theta_s}} q s = p ˉ s ∣ H θ s and return p s = p θ s , q s p_s=p_{\theta_s,q_s} p s = p θ s , q s .
and its optimistic update are presented in Algorithms 3 and 4. The d = 1 d = 1 d = 1 analogue is deferred to Algorithm 6 in Appendix § 13 .§ 3 .
Unlike the greedy approach in Algorithm 1, which constructs policies using point estimates of the reward parameters, the proposed algorithm adopts an optimistic learning strategy inspired by UCB techniques [e.g. 1 ] . By optimising over confidence sets of plausible reward parameters, the algorithm explicitly accounts for estimation uncertainty and encourages exploration when the reward parameters are insufficiently learned, without relying on the covariate diversity property established under Assumption 1.
Confidence set construction. At each update time s s s , instead of directly using the point estimators β ^ s , k \widehat{\beta}_{s,k} β s , k as in Algorithm 1, we construct a confidence set C s , k C_{s,k} C s , k for each arm and optimise over all candidate reward models contained in these sets, which preserve statistical coverage while making the optimistic policy optimisation tractable. With V s , k V_{s,k} V s , k and β ^ s , k \widehat{\beta}_{s,k} β s , k defined in Algorithm 3, we form
C s , k = { b : ∥ V s , k 1 / 2 ( b − β ^ s , k ) ∥ 1 ≤ d + 1 α T } , (14) C_{s,k} = \left\{b : \left\|V_{s,k}^{1/2}(b-\widehat{\beta}_{s,k})\right\|_{1} \le\sqrt{d+1}\alpha_T\right\}, \tag*{(14)} C s , k = { b : V s , k 1/2 ( b − β s , k ) 1 ≤ d + 1 α T } , (14)
which replaces the usual ℓ 2 \ell_2 ℓ 2 -ball of V s , k 1 / 2 ( b − β ^ s , k ) V_{s,k}^{1/2}(b-\widehat{\beta}_{s,k}) V s , k 1/2 ( b − β s , k ) with radius α T \alpha_T α T in the standard UCB algorithms by an ℓ 1 \ell_1 ℓ 1 -ball with enlarged radius d + 1 α T \sqrt{d+1}\alpha_T d + 1 α T . This modification preserves the confidence coverage while replacing the ellipsoidal region with a polytope of 2 ( d + 1 ) 2(d+1) 2 ( d + 1 ) explicit vertices, motivating the design of the computationally-efficient vertex optimisation in Algorithm 4.
Vertex optimisation. The model parameters and policy are estimated via the optimisation problem
( β ˉ s , 1 ∘ , β ˉ s , 2 ∘ , q s ∘ ) ∈ arg max b k ∈ C s , k , k ∈ { 1 , 2 } , H b 1 − b 2 ≠ ∅ q ∈ Q b 1 − b 2 , L 1 s − 1 ∑ i = 1 s − 1 [ X ~ i T b 1 p b 1 − b 2 , q ( X i ) + X ~ i T b 2 { 1 − p b 1 − b 2 , q ( X i ) } ] = arg max b k ∈ C s , k , k ∈ { 1 , 2 } H b 1 − b 2 ≠ ∅ { 1 s − 1 ∑ i = 1 s − 1 X ~ i T b 2 + max q ∈ Q b 1 − b 2 , L 1 s − 1 ∑ i = 1 s − 1 { X ~ i T ( b 1 − b 2 ) } p b 1 − b 2 , q ( X i ) } , (15) \begin{aligned}
(\bar{\beta}^{\circ}_{s,1},\bar{\beta}^{\circ}_{s,2},q_s^{\circ}) \in\underset{\substack{b_k\in C_{s,k},\ k\in\{1,2\},\ \mathcal{H}_{b_1-b_2}\ne\varnothing\\ q\in\mathcal{Q}_{b_1-b_2,L}}}{\operatorname{arg\,max}} \frac{1}{s-1}\sum_{i=1}^{s-1}\left[\widetilde{X}_i^{\mathsf{T}}b_1p_{b_1-b_2,q}(X_i)+\widetilde{X}_i^{\mathsf{T}}b_2\{1-p_{b_1-b_2,q}(X_i)\}\right] \\
&= \underset{\substack{b_k\in C_{s,k},\ k\in\{1,2\}\\ \mathcal{H}_{b_1-b_2}\ne\varnothing}}{\operatorname{arg\,max}}\left\{\frac{1}{s-1}\sum_{i=1}^{s-1}\widetilde{X}_i^{\mathsf{T}}b_2+\max_{q\in\mathcal{Q}_{b_1-b_2,L}}\frac{1}{s-1}\sum_{i=1}^{s-1}\{\widetilde{X}_i^{\mathsf{T}}(b_1-b_2)\}p_{b_1-b_2,q}(X_i)\right\}, \tag*{(15)}
\end{aligned} ( β ˉ s , 1 ∘ , β ˉ s , 2 ∘ , q s ∘ ) ∈ b k ∈ C s , k , k ∈ { 1 , 2 } , H b 1 − b 2 = ∅ q ∈ Q b 1 − b 2 , L arg max s − 1 1 i = 1 ∑ s − 1 [ X i T b 1 p b 1 − b 2 , q ( X i ) + X i T b 2 { 1 − p b 1 − b 2 , q ( X i )} ] = b k ∈ C s , k , k ∈ { 1 , 2 } H b 1 − b 2 = ∅ arg max { s − 1 1 i = 1 ∑ s − 1 X i T b 2 + q ∈ Q b 1 − b 2 , L max s − 1 1 i = 1 ∑ s − 1 { X i T ( b 1 − b 2 )} p b 1 − b 2 , q ( X i ) } , (15)
consisting of two optimisation problems.
The inner optimisation over Q b 1 − b 2 , L \mathcal{Q}_{b_1-b_2,L} Q b 1 − b 2 , L has an exact linear program representation. Specifically, just as the greedy boundary problem (8) in Algorithm 1 reduces without relaxation gap to the finite-dimensional linear program (9) , for any fixed pair ( b 1 , b 2 ) (b_1,b_2) ( b 1 , b 2 ) with θ = b 1 − b 2 \theta=b_1-b_2 θ = b 1 − b 2 , the inner maximisation in (15) reduces to
H s ( θ ) = max z ∈ Z s − 1 , L 1 s − 1 ∑ i = 1 s − 1 ( X ~ i T θ ) z i , (16) H_s(\theta)=\max_{z\in\mathcal{Z}_{s-1,L}}\frac{1}{s-1}\sum_{i=1}^{s-1}(\widetilde{X}_i^{\mathsf{T}}\theta)z_i, \tag*{(16)} H s ( θ ) = z ∈ Z s − 1 , L max s − 1 1 i = 1 ∑ s − 1 ( X i T θ ) z i , (16)
where
Z s − 1 , L = { z ∈ [ 0 , 1 ] s − 1 : ∣ z i − z j ∣ ≤ L ∥ X i − X j ∥ ν , i , j ∈ [ s − 1 ] } . (17) \mathcal{Z}_{s-1,L}=\left\{z\in[0,1]^{s-1}: \lvert z_i-z_j\rvert\le L\lVert X_i-X_j\rVert_{\nu},\ i,j\in[s-1]\right\}. \tag*{(17)} Z s − 1 , L = { z ∈ [ 0 , 1 ] s − 1 : ∣ z i − z j ∣ ≤ L ∥ X i − X j ∥ ν , i , j ∈ [ s − 1 ] } . (17)
An optimiser z z z of (16) can be interpolated to a global Lipschitz policy by the McShane extension and then replaced by its boundary-canonical representative without decreasing the empirical objective. A detailed formulation of the equivalence between the inner optimisation in (15) and (16) is left in Lemma 13.
The outer optimisation over C s , k C_{s,k} C s , k ’s in (15) , with the notation in (16) , can be written as
max b k ∈ C s , k , k ∈ { 1 , 2 } F s ( b 1 , b 2 ) , with F s ( b 1 , b 2 ) = 1 s − 1 ∑ i = 1 s − 1 X ~ i T b 2 + H s ( b 1 − b 2 ) , (18) \max_{b_k\in C_{s,k},\ k\in\{1,2\}}F_s(b_1,b_2),\qquad\text{with }F_s(b_1,b_2)=\frac{1}{s-1}\sum_{i=1}^{s-1}\widetilde{X}_i^{\mathsf{T}}b_2+H_s(b_1-b_2), \tag*{(18)} b k ∈ C s , k , k ∈ { 1 , 2 } max F s ( b 1 , b 2 ) , with F s ( b 1 , b 2 ) = s − 1 1 i = 1 ∑ s − 1 X i T b 2 + H s ( b 1 − b 2 ) , (18)
which, as shown in Lemma 5, can be reduced to an equivalent optimisation over the vertex pairs of confidence sets.
Lemma 5 (Vertex optimisation). For every update time s s s , the function F s F_s F s in (18) is jointly convex and continuous, and we have that
max b k ∈ C s , k , k ∈ { 1 , 2 } F s ( b 1 , b 2 ) = max v k ∈ V s , k , k ∈ { 1 , 2 } F s ( v 1 , v 2 ) . (19) \max_{b_k \in\mathcal{C}_{s,k},\, k \in\{1,2\}} F_s(b_1,b_2)
=
\max_{v_k \in\mathcal{V}_{s,k},\, k \in\{1,2\}} F_s(v_1,v_2). \tag*{(19)} b k ∈ C s , k , k ∈ { 1 , 2 } max F s ( b 1 , b 2 ) = v k ∈ V s , k , k ∈ { 1 , 2 } max F s ( v 1 , v 2 ) . (19) where C s , k \mathcal{C}_{s,k} C s , k and V s , k \mathcal{V}_{s,k} V s , k are defined in (14) and (12) respectively. Moreover, if the selected model pair ( β ˉ s , 1 , β ˉ s , 2 ) (\bar{\beta}_{s,1},\bar{\beta}_{s,2}) ( β ˉ s , 1 , β ˉ s , 2 ) satisfies H β ˉ s , 1 − β ˉ s , 2 ≠ ∅ \mathcal{H}_{\bar{\beta}_{s,1}-\bar{\beta}_{s,2}} \ne\varnothing H β ˉ s , 1 − β ˉ s , 2 = ∅ , then the output of Algorithm 4 solves the original optimistic update in (15) , i.e.
1 s − 1 ∑ i = 1 s − 1 r β ˉ s , p s ( X i ) = sup b k ∈ C s , k , k ∈ { 1 , 2 } q ∈ Q b 1 − b 2 , L 1 s − 1 ∑ i = 1 s − 1 r b , p b 1 − b 2 , q ( X i ) , (20) \frac{1}{s-1}\sum_{i=1}^{s-1}r_{\bar{\beta}_{s,p_s}}(X_i)
=
\sup_{\substack{b_k \in\mathcal{C}_{s,k},\, k \in\{1,2\}\\ q \in\mathcal{Q}_{b_1-b_2,L}}}
\frac{1}{s-1}\sum_{i=1}^{s-1}r_{b,p_{b_1-b_2,q}}(X_i), \tag*{(20)} s − 1 1 i = 1 ∑ s − 1 r β ˉ s , p s ( X i ) = b k ∈ C s , k , k ∈ { 1 , 2 } q ∈ Q b 1 − b 2 , L sup s − 1 1 i = 1 ∑ s − 1 r b , p b 1 − b 2 , q ( X i ) , (20) where r b , p ( x ) = x ~ T b 1 p ( x ) + x ~ T b 2 { 1 − p ( x ) } r_{b,p}(x)=\tilde{x}^{\mathsf{T}}b_1p(x)+\tilde{x}^{\mathsf{T}}b_2\{1-p(x)\} r b , p ( x ) = x ~ T b 1 p ( x ) + x ~ T b 2 { 1 − p ( x )} .
The proof of Lemma 5 is in Appendix § 11 .§ 3 . It follows from Lemma 34 that, a sufficiently long initial exploration phase ensures that all candidate reward-gap parameters remain within the regular parameter neighbourhood Θ \Theta Θ with high probability. Their decision boundaries are therefore nonempty, so Lemma 5 guarantees that Algorithm 4 provides an exact solution to the optimistic update in (15) through a finite number of linear programs on this event.
The equivalence of the vertex optimisation in (19) follows directly from the construction of the polyhedral confidence sets in (14) . Under the transformation u = V s , k 1 / 2 ( b − β ^ s , k ) u=V_{s,k}^{1/2}(b-\widehat{\beta}_{s,k}) u = V s , k 1/2 ( b − β s , k ) , the confidence set C s , k \mathcal{C}_{s,k} C s , k becomes an ℓ 1 \ell_1 ℓ 1 -ball with radius d + 1 α T \sqrt{d+1}\alpha_T d + 1 α T , which is the convex hull of its 2 ( d + 1 ) 2(d+1) 2 ( d + 1 ) vertices. Mapping these vertices back through b = β ^ s , k + V s , k − 1 / 2 u b=\widehat{\beta}_{s,k}+V_{s,k}^{-1/2}u b = β s , k + V s , k − 1/2 u gives the vertex set V s , k \mathcal{V}_{s,k} V s , k in the original parameter space. Hence, every b ∈ C s , k b\in\mathcal{C}_{s,k} b ∈ C s , k can be written as a convex combination of the vertices in V s , k \mathcal{V}_{s,k} V s , k and maximising the jointly convex function F s F_s F s over C s , 1 × C s , 2 \mathcal{C}_{s,1}\times\mathcal{C}_{s,2} C s , 1 × C s , 2 is therefore equivalent to maximising it over the finite set of vertex pairs V s , 1 × V s , 2 \mathcal{V}_{s,1}\times\mathcal{V}_{s,2} V s , 1 × V s , 2 .
Remark 8 (Computational complexity). The vertex formulation ensures that extending the method beyond continuously distributed covariates incurs only an additional computational cost of order O ( d 2 ) O(d^2) O ( d 2 ) , compared to Algorithm 1. In particular, each confidence polytope V s , k \mathcal{V}_{s,k} V s , k has 2 ( d + 1 ) 2(d+1) 2 ( d + 1 ) vertices, so Algorithm 4 only needs to consider 4 ( d + 1 ) 2 4(d+1)^2 4 ( d + 1 ) 2 vertex pairs and solves one linear program for each pair.
Theorem 6 (Regret control). Suppose Assumptions 1, 2–4 hold and L L L is an absolute constant satisfying 0 < L < L 0 ′ 0<L<L'_0 0 < L < L 0 ′ for some L 0 ′ > 0 L'_0>0 L 0 ′ > 0 . Implement Algorithm 3 (d ≥ 2 d\ge2 d ≥ 2 ) or Algorithm 6 (d = 1 d=1 d = 1 ), with λ = C λ \lambda=C_\lambda λ = C λ , where C λ > 0 C_\lambda>0 C λ > 0 is an absolute constant, T 0 = ⌈ T ⌉ T_0=\lceil\sqrt{T}\rceil T 0 = ⌈ T ⌉ , and α T \alpha_T α T in (11) . Then for any η ∈ ( 0 , 1 / 2 ) \eta\in(0,1/2) η ∈ ( 0 , 1/2 ) , we have with probability at least 1 − η 1-\eta 1 − η that
R T ≤ C 1 α T T { log ( 1 + T λ ) + log ( 8 / η ) } + C 2 { T log ( T / η ) , d = 1 , 2 , T log 3 / 2 ( T / η ) , d = 3 , T ( d − 2 ) / ( d − 1 ) log ( T / η ) , d ≥ 4. \mathfrak{R}_T \le C_1\alpha_T\sqrt{T\left\{\log\left(1+\frac{T}{\lambda}\right)+\log(8/\eta)\right\}}+
C_2
\begin{cases}
\sqrt{T}\log(T/\eta), & d=1,2,\\
\sqrt{T}\log^{3/2}(T/\eta), & d=3,\\
T^{(d-2)/(d-1)}\log(T/\eta), & d\ge4.
\end{cases} R T ≤ C 1 α T T { log ( 1 + λ T ) + log ( 8/ η ) } + C 2 ⎩ ⎨ ⎧ T log ( T / η ) , T log 3/2 ( T / η ) , T ( d − 2 ) / ( d − 1 ) log ( T / η ) , d = 1 , 2 , d = 3 , d ≥ 4. whenever T ≥ 16 ∨ C 3 ( 1 + d + 1 ) 4 α T 4 / κ X 2 T\ge16\vee C_3(1+\sqrt{d+1})^4\alpha_T^4/\kappa_X^2 T ≥ 16 ∨ C 3 ( 1 + d + 1 ) 4 α T 4 / κ X 2 and C 1 , C 2 , C 3 > 0 C_1,C_2,C_3>0 C 1 , C 2 , C 3 > 0 are constants depending on d , ν , L , σ , C β d,\nu,L,\sigma,C_\beta d , ν , L , σ , C β and the constants in Assumption 4.
The regret upper bound in Theorem 6 matches the continuous context distribution rate in Theorem 4, up to poly-logarithmic factors, despite the absence of guaranteed covariate diversity for the greedy approach.
The detailed proof of Theorem 6 is in Appendix § 11 .§ 4 , with a sketch below. The proof follows a similar strategy to that of Theorem 4, particularly in controlling the uniform deviation between empirical and population rewards through the complexity of the boundary-canonical policy class. The main difference lies in the treatment of parameter estimation uncertainty under the UCB framework.
Specifically, let R ^ s ( b , p ) = ( s − 1 ) − 1 ∑ i = 1 s − 1 r b , p ( X i ) \widehat{R}_s(b,p)=(s-1)^{-1}\sum_{i=1}^{s-1}r_{b,p}(X_i) R s ( b , p ) = ( s − 1 ) − 1 ∑ i = 1 s − 1 r b , p ( X i ) and R b ( p ) = E [ r b , p ( X ) ] R_b(p)=\mathbb{E}[r_{b,p}(X)] R b ( p ) = E [ r b , p ( X )] , with R ( p ) = R β ∗ ( p ) R(p)=R_{\beta^*}(p) R ( p ) = R β ∗ ( p ) . Since the true reward parameters lie within the confidence sets with high probability, the selected model pair ( β ˉ s , p s ) (\bar{\beta}_s,p_s) ( β ˉ s , p s ) in (20) satisfies R ^ s ( β ˉ s , p s ) ≥ R ^ s ( β ∗ , p ∗ ) \widehat{R}_s(\bar{\beta}_s,p_s)\geq\widehat{R}_s(\beta^*,p^*) R s ( β ˉ s , p s ) ≥ R s ( β ∗ , p ∗ ) (Lemma 48). Consequently, the instantaneous regret can be decomposed and bounded by
R ( p ∗ ) − R ( p s ) ≤ { R ( p ∗ ) − R ^ s ( β ∗ , p ∗ ) } + { R ^ s ( β ˉ s , p s ) − R β ˉ s ( p s ) } + { R β ˉ s ( p s ) − R ( p s ) } . R(p^*)-R(p_s)\leq\{R(p^*)-\widehat{R}_s(\beta^*,p^*)\}+\{\widehat{R}_s(\bar{\beta}_s,p_s)-R_{\bar{\beta}_s}(p_s)\}+\{R_{\bar{\beta}_s}(p_s)-R(p_s)\}. R ( p ∗ ) − R ( p s ) ≤ { R ( p ∗ ) − R s ( β ∗ , p ∗ )} + { R s ( β ˉ s , p s ) − R β ˉ s ( p s )} + { R β ˉ s ( p s ) − R ( p s )} .
where the first two terms can be controlled using the uniform empirical-process bounds in Lemma 42, and the last term captures the uncertainty in the estimated reward parameters. Unlike the proof of Theorem 4, which requires controlling the boundary mismatch error, the UCB analysis bounds the remaining parameter estimation error through arm-specific confidence widths. Writing s = τ ( t ) s=\tau(t) s = τ ( t ) for the most recent update time before t t t and w s , k ( x ) = min { 1 , ∥ x ~ ∥ V s , k − 1 } w_{s,k}(x)=\min\{1,\|\tilde{x}\|_{V_{s,k}^{-1}}\} w s , k ( x ) = min { 1 , ∥ x ~ ∥ V s , k − 1 } , we obtain from the proof of Lemma 49 that
R β ˉ s ( p s ) − R ( p s ) ≲ log E [ p s ( X t ) w s , 1 ( X t ) + { 1 − p s ( X t ) } w s , 2 ( X t ) ∣ F t − 1 ] = E [ w s , A t ( X t ) ∣ F t − 1 ] . R_{\bar{\beta}_s}(p_s)-R(p_s)\lesssim_{\log}\mathbb{E}\left[p_s(X_t)w_{s,1}(X_t)+\{1-p_s(X_t)\}w_{s,2}(X_t)\mid\mathcal{F}_{t-1}\right]=\mathbb{E}\left[w_{s,A_t}(X_t)\mid\mathcal{F}_{t-1}\right]. R β ˉ s ( p s ) − R ( p s ) ≲ l o g E [ p s ( X t ) w s , 1 ( X t ) + { 1 − p s ( X t )} w s , 2 ( X t ) ∣ F t − 1 ] = E [ w s , A t ( X t ) ∣ F t − 1 ] .
where the equality uses the fact that, conditional on ( F t − 1 , X t ) (\mathcal{F}_{t-1},X_t) ( F t − 1 , X t ) , arm 1 is pulled with probability p s ( X t ) p_s(X_t) p s ( X t ) . This allows us to express the uncertainty under the randomised fair policy in terms of the confidence width of the arm actually selected, whose cumulative contribution can be bounded using Lemma 47.
Optimality and minimax lower bounds
To establish the rate optimality of the algorithms developed in § 4 , and to quantify the inevitable learning regret under the fairness constraint, in this section, we derive minimax lower bounds.
While the upper bounds in § 4 hold for general ℓ ν \ell_\nu ℓ ν -Lipschitz fairness, the minimax lower bound depends on the metric, ν \nu ν in particular, through the geometry of the admissible policy classes. Here we focus on the ℓ 1 \ell_1 ℓ 1 formulation, which fixes a concrete metric geometry of the policy class and serves as a representative case to understand the unavoidable boundary-learning difficulty induced by Lipschitz fairness.
Let G c t s \mathcal{G}_{\mathrm{cts}} G cts denote the collection of data generating distributions satisfying Assumptions 1, 2, 3 and 4, and G g e n \mathcal{G}_{\mathrm{gen}} G gen the collection satisfying Assumptions 1 ′ 1' 1 ′ , 2, 3 and 4. The continuous-context class is a subclass of the general-context class, i.e. G c t s ⊆ G g e n \mathcal{G}_{\mathrm{cts}}\subseteq\mathcal{G}_{\mathrm{gen}} G cts ⊆ G gen , since under Assumption 1, the context density is uniformly bounded away from zero on [ 0 , 1 ] d [0,1]^d [ 0 , 1 ] d , which implies that λ min { E ( X ~ X ~ T ) } \lambda_{\min}\{\mathbb{E}(\widetilde{X}\widetilde{X}^{\mathsf{T}})\} λ m i n { E ( X X T )} is bounded away from zero and hence Assumption 1 ′ 1' 1 ′ is satisfied. It therefore suffices to only establish the lower bound over G c t s \mathcal{G}_{\mathrm{cts}} G cts .
Denote F t − 1 = σ { ( X s , A s , Y s ) : 1 ≤ s ≤ t − 1 } \mathcal{F}_{t-1}=\sigma\{(X_s,A_s,Y_s):1\leq s\leq t-1\} F t − 1 = σ {( X s , A s , Y s ) : 1 ≤ s ≤ t − 1 } as the observed history before round t t t . We define Π L , 1 \Pi_{L,1} Π L , 1 as the class of non-anticipating ( L , 1 ) (L,1) ( L , 1 ) -Lipschitz fair learning algorithms π = { p t } t ≥ 1 \pi=\{p_t\}_{t\geq1} π = { p t } t ≥ 1 , where at each round t t t , it selects an F t − 1 \mathcal{F}_{t-1} F t − 1 -measurable random function p t : [ 0 , 1 ] d → [ 0 , 1 ] p_t:[0,1]^d\to[0,1] p t : [ 0 , 1 ] d → [ 0 , 1 ] such that p t ∈ P L , 1 p_t\in\mathcal{P}_{L,1} p t ∈ P L , 1 almost surely, i.e.
Π L , 1 = { { p t } t = 1 ∞ : p t is F t − 1 -measurable and p t ∈ P L , 1 a.s. for every t } . \Pi_{L,1}=\left\{\{p_t\}_{t=1}^{\infty}:p_t\text{ is }\mathcal{F}_{t-1}\text{-measurable and }p_t\in\mathcal{P}_{L,1}\text{ a.s. for every }t\right\}. Π L , 1 = { { p t } t = 1 ∞ : p t is F t − 1 -measurable and p t ∈ P L , 1 a.s. for every t } .
For a policy sequence π = { p t } t ≥ 1 \pi=\{p_t\}_{t\geq1} π = { p t } t ≥ 1 , define its expected regret under G G G by
R T ( π ; G ) = E G π [ ∑ t = 1 T { R G ( p G ∗ ) − R G ( p t ) } ] . \mathfrak{R}_T(\pi;G)=\mathbb{E}^{\pi}_G\left[\sum_{t=1}^{T}\{R_G(p_G^*)-R_G(p_t)\}\right]. R T ( π ; G ) = E G π [ t = 1 ∑ T { R G ( p G ∗ ) − R G ( p t )} ] .
where R G ( p ) = E G [ p ( X ) X ~ T β 1 ∗ + { 1 − p ( X ) } X ~ T β 2 ∗ ] R_G(p)=\mathbb{E}_G[p(X)\widetilde{X}^{\mathsf{T}}\beta_1^*+\{1-p(X)\}\widetilde{X}^{\mathsf{T}}\beta_2^*] R G ( p ) = E G [ p ( X ) X T β 1 ∗ + { 1 − p ( X )} X T β 2 ∗ ] and p G ∗ ∈ arg max p ∈ P L , 1 R G ( p ) p_G^*\in\arg\max_{p\in\mathcal{P}_{L,1}}R_G(p) p G ∗ ∈ arg max p ∈ P L , 1 R G ( p ) .
Theorem 7 (Minimax lower bound). For any fixed L ∈ ( 0 , 1 ) L \in(0,1) L ∈ ( 0 , 1 ) , there is a constant c d , L > 0 c_{d,L} > 0 c d , L > 0 depending only on d d d and L L L , such that
inf π ∈ Π L , 1 sup G ∈ G c t s R T ( π ; G ) ≥ c d , L { T , d ∈ { 1 , 2 , 3 } , T d − 2 d − 1 , d ≥ 4. \inf_{\pi\in\Pi_{L,1}}\sup_{G \in\mathcal{G}_{\mathrm{cts}}}\mathfrak{R}_{T}(\pi;G) \geq c_{d,L}
\begin{cases}
\sqrt{T}, & d \in\{1,2,3\},\\
T^{\frac{d-2}{d-1}}, & d \geq4.
\end{cases} π ∈ Π L , 1 inf G ∈ G cts sup R T ( π ; G ) ≥ c d , L { T , T d − 1 d − 2 , d ∈ { 1 , 2 , 3 } , d ≥ 4.
The upper bound in Theorem 4 and the lower bound in Theorem 7 together show that Algorithm 1 for d ≥ 2 d \geq2 d ≥ 2 and Algorithm 5 for d = 1 d=1 d = 1 , given in the appendix, achieve optimal regret up to poly-logarithmic factors. In particular, the extra fairness-induced regret of the order T ( d − 2 ) / ( d − 1 ) T^{(d-2)/(d-1)} T ( d − 2 ) / ( d − 1 ) is tight and inevitable when d ≥ 4 d \geq4 d ≥ 4 .
The detailed proof of Theorem 7 is deferred to Appendix § 12 .§ 1 . The key idea is to construct covariate distributions that are statistically difficult to distinguish but induce different optimal fair policies. In the unconstrained problem, once the reward gap Δ ∗ ( x ) \Delta^*(x) Δ ∗ ( x ) is fixed, the oracle policy is determined pointwise by the sign of Δ ∗ ( x ) \Delta^*(x) Δ ∗ ( x ) and is therefore independent of the covariate density. Under Lipschitz fairness, however, the policy must transition smoothly across the decision boundary, and the optimal shape of this transition depends on how probability mass is distributed near the boundary. We exploit this feature by keeping the reward gap fixed, specifically Δ ∗ ( x ) = x 1 − 1 / 2 \Delta^*(x)=x_1-1/2 Δ ∗ ( x ) = x 1 − 1/2 , and perturbing only the covariate density locally around the boundary H θ ∗ = { x ∈ [ 0 , 1 ] d : x 1 = 1 / 2 } \mathcal{H}_{\theta^*}=\{x \in[0,1]^d:x_1=1/2\} H θ ∗ = { x ∈ [ 0 , 1 ] d : x 1 = 1/2 } . These local density perturbations shift probability mass across the boundary and hence alter the optimal fairness correction, while keeping the KL divergence between environments small.
For d ∈ { 1 , 2 } d \in\{1,2\} d ∈ { 1 , 2 } , a two-point density perturbation is sufficient to capture the parametric difficulty at rate T \sqrt{T} T , leading to a Le Cam argument (e.g. Yu, [57 ] ). The two environments favour different local shifts of the boundary value, forcing any fair policy to incur Ω ( T ) \Omega(\sqrt{T}) Ω ( T ) regret under at least one of them. For d ≥ 3 d \geq3 d ≥ 3 , the ( d − 1 ) (d-1) ( d − 1 ) -dimensional decision boundary accommodates a packing of m d − 1 m^{d-1} m d − 1 spatially separated local perturbation regions along the tangential coordinates ( x 2 , … , x d ) (x_2,\ldots,x_d) ( x 2 , … , x d ) . By placing independent signed density perturbations across these regions and applying Assouad’s lemma (e.g. Yu, [57 ] ), any fair policy must simultaneously estimate all m d − 1 m^{d-1} m d − 1 local boundaries. Hence, by balancing local statistical detectability against the aggregate reward loss induced by the Lipschitz constraint, we choose m ≍ T 1 / ( d − 1 ) m \asymp T^{1/(d-1)} m ≍ T 1/ ( d − 1 ) to obtain the sharp nonparametric lower bound of order Ω ( T ( d − 2 ) / ( d − 1 ) ) \Omega(T^{(d-2)/(d-1)}) Ω ( T ( d − 2 ) / ( d − 1 ) ) , which matches T \sqrt{T} T at d = 3 d=3 d = 3 and strictly dominates T \sqrt{T} T for d ≥ 4 d \geq4 d ≥ 4 .
Numerical experiments
In § 4 , we proposed Algorithm 1 for continuous contexts and Algorithm 3 for general contexts, with their one-dimensional counterparts in Algorithms 5 and 6, respectively. In this section, we evaluate the empirical performance of our framework, focusing on the lazy fair greedy algorithms (Algorithms 1 and 5) owing to their computational efficiency; an additional mixed-context experiment in Section G.1.1 further illustrates their empirical performance beyond purely continuous contexts. Simulation and real-data analysis are collected in § 6.1 and § 6.2 , respectively. The code and datasets used in the experiments are available online at https://github.com/chrisfanwang/fair-contextual-bandits .
Simulation studies
We consider the two-arm linear reward model in Section 2.1, with d ∈ { 1 , 2 , 3 , 4 } d \in\{1,2,3,4\} d ∈ { 1 , 2 , 3 , 4 } and independent noise random variables ξ t , k ∼ N ( 0 , 0.3 2 ) \xi_{t,k} \sim\mathcal{N}(0,0.3^2) ξ t , k ∼ N ( 0 , 0. 3 2 ) . Throughout, fairness is measured using the ℓ 1 \ell_1 ℓ 1 -distance (ν = 1 \nu=1 ν = 1 in Definition 1). Contexts are sampled independently across t t t from one of the following two distributions:
Truncated Gaussian design where each coordinate of X t X_t X t is independently drawn from N ( 0.5 , 0.18 2 ) \mathcal{N}(0.5,0.18^2) N ( 0.5 , 0.1 8 2 ) and then truncated to [ 0 , 1 ] [0,1] [ 0 , 1 ] .
For each context distribution, we consider two reward regimes:
β 1 ∗ = ( − d 4 , 1 2 d 1 d T ) T and β 2 ∗ = − β 1 ∗ . \beta_1^*=\left(-\frac{\sqrt{d}}{4},\frac{1}{2\sqrt{d}}\mathbf{1}_d^{\mathsf{T}}\right)^{\mathsf{T}}
\quad\text{and}\quad
\beta_2^*=-\beta_1^*. β 1 ∗ = ( − 4 d , 2 d 1 1 d T ) T and β 2 ∗ = − β 1 ∗ .
The resulting reward gap is Δ ∗ ( x ) = d − 1 / 2 ∑ j = 1 d ( x j − 1 / 2 ) \Delta^*(x)=d^{-1/2}\sum_{j=1}^{d}(x_j-1/2) Δ ∗ ( x ) = d − 1/2 ∑ j = 1 d ( x j − 1/2 ) , so that all covariates contribute equally to the gap.
Table 1. Coefficients in the heterogeneous reward regime. The first component denotes the intercept.
These choices yield four baseline settings in which each arm is optimal with probability 1 / 2 1/2 1/2 .
To examine learning when one arm is optimal over a smaller region, we consider two additional d = 2 d=2 d = 2 settings under the uniform design:
Arm 1 small. We set β 1 ∗ = ( − 0.41 , 0.135 , 0.5 ) T \beta_1^*=(-0.41,0.135,0.5)^{\mathsf{T}} β 1 ∗ = ( − 0.41 , 0.135 , 0.5 ) T and β 2 ∗ = − β 1 ∗ \beta_2^*=-\beta_1^* β 2 ∗ = − β 1 ∗ , giving Δ ∗ ( x ) = 0.27 x 1 + x 2 − 0.82 \Delta^*(x)=0.27x_1+x_2-0.82 Δ ∗ ( x ) = 0.27 x 1 + x 2 − 0.82 . Arm 1 is optimal with probability 0.315 0.315 0.315 .
Arm 2 small. We set β 1 ∗ = ( − 0.1 , 0.7 , 0.4 ) T \beta_1^*=(-0.1,0.7,0.4)^{\mathsf{T}} β 1 ∗ = ( − 0.1 , 0.7 , 0.4 ) T and β 2 ∗ = ( 0.1 , − 0.3 , − 0.6 ) T \beta_2^*=(0.1,-0.3,-0.6)^{\mathsf{T}} β 2 ∗ = ( 0.1 , − 0.3 , − 0.6 ) T , giving Δ ∗ ( x ) = x 1 + x 2 − 0.2 \Delta^*(x)=x_1+x_2-0.2 Δ ∗ ( x ) = x 1 + x 2 − 0.2 . Arm 2 is optimal with probability 0.02 0.02 0.02 .
Learning, oracle policies and evaluation. We implement Algorithm 1 and its one-dimensional counterpart in Algorithm 5. We consider horizons T ∈ { 1024 , 2048 , 4096 , 8192 , 16384 , 32768 } T\in\{1024,2048,4096,8192,16384,32768\} T ∈ { 1024 , 2048 , 4096 , 8192 , 16384 , 32768 } , with a separate experiment for each horizon because the exploration length depends on T T T . The initial exploration phase lasts 2 T o ( T ) 2T_o(T) 2 T o ( T ) rounds, where T o ( T ) = ⌈ T log ( 4 T ) ⌉ T_o(T)=\lceil\sqrt{T\log(4T)}\rceil T o ( T ) = ⌈ T log ( 4 T ) ⌉ , following the choice in Theorem 4 with C 1 = 1 C_1=1 C 1 = 1 . We use ridge regularisation with λ = 1 \lambda=1 λ = 1 throughout.
The baseline reward gaps can be written as Δ ∗ ( x ) = c ∑ j = 1 d s j ( x j − 1 / 2 ) \Delta^*(x)=c\sum_{j=1}^{d}s_j(x_j-1/2) Δ ∗ ( x ) = c ∑ j = 1 d s j ( x j − 1/2 ) , where c > 0 c>0 c > 0 and s j ∈ { − 1 , 0 , 1 } s_j\in\{-1,0,1\} s j ∈ { − 1 , 0 , 1 } . Under both the uniform and truncated Gaussian designs, an optimal fair policy is
p 1 , L ∗ ( x ) = [ 1 2 + L ∑ j = 1 d s j ( x j − 1 / 2 ) ] [ 0 , 1 ] , (21) p_{1,L}^*(x)=\left[\frac{1}{2}+L\sum_{j=1}^{d}s_j(x_j-1/2)\right]_{[0,1]}, \tag*{(21)} p 1 , L ∗ ( x ) = [ 2 1 + L j = 1 ∑ d s j ( x j − 1/2 ) ] [ 0 , 1 ] , (21)
where [ x ] [ a , b ] = max { a , min { x , b } } [x]_{[a,b]}=\max\{a,\min\{x,b\}\} [ x ] [ a , b ] = max { a , min { x , b }} . The Arm 1 small and Arm 2 small settings also admit explicit oracle policies; see Section G.1.
We evaluate the population reward R ( p ) R(p) R ( p ) in (1) by integrating over the covariate distribution. Because analytical integration becomes computationally demanding as the dimension or policy complexity grows, we use exact analytical integration for d ∈ { 1 , 2 } d\in\{1,2\} d ∈ { 1 , 2 } (as well as for the oracle cost-of-fairness curves under the uniform design across all d ∈ { 1 , 2 , 3 , 4 } d\in\{1,2,3,4\} d ∈ { 1 , 2 , 3 , 4 } ) and quasi-Monte Carlo integration for d ∈ { 3 , 4 } d\in\{3,4\} d ∈ { 3 , 4 } . For regret evaluation under either design, we apply the same integration rule to the oracle and learned policies. All quasi-Monte Carlo evaluations use 16384 Halton points for d = 3 d=3 d = 3 and 32768 for d = 4 d = 4 d = 4 , after discarding the first 128 burin-in points. Under the truncated-Gaussian design, we transform these points coordinate-wise using the inverse cumulative distribution function of the truncated normal distribution. We compute cumulative regret R T \mathfrak{R}_T R T , defined in (3) , by summing the population reward differences between the oracle and learned policies over all rounds, including initial exploration.
Cost of fairness. We first examine Cost ( L ) \operatorname{Cost}(L) Cost ( L ) , defined in § 3.2 as the reward difference between the unconstrained and fair oracles. Figure 3 shows the results for the four baseline settings with d ∈ { 1 , 2 , 3 , 4 } d \in\{1,2,3,4\} d ∈ { 1 , 2 , 3 , 4 } , using 81 logarithmically spaced values of L L L from 0.02 to 10. The curves show a similar pattern across distributions, reward regimes and dimensions. For small L L L (L ≤ 0.1 L \le0.1 L ≤ 0.1 ), the cost remains close to Cost ( 0 ) \operatorname{Cost}(0) Cost ( 0 ) , the cost associated with the best constant policy. As L L L increases, the fairness constraint becomes less restrictive and the cost decreases. Over 3 ≤ L ≤ 10 3 \le L \le10 3 ≤ L ≤ 10 , the curves are approximately linear on log-log axes, with slopes close to − 2 -2 − 2 , consistent with the L − 2 L^{-2} L − 2 rate in Theorem 2 .
Figure 3. Cost of fairness, Cost ( L ) \operatorname{Cost}(L) Cost ( L ) , on log–log axes. Columns show homogeneous (left) and heterogeneous (right) rewards; rows show uniform (top) and truncated-Gaussian (bottom) covariates.
For the same settings, Figure 4 reports Cost ( 0 ) − Cost ( L ) = R L ∗ − R 0 ∗ \operatorname{Cost}(0) - \operatorname{Cost}(L) = R_L^* - R_0^* Cost ( 0 ) − Cost ( L ) = R L ∗ − R 0 ∗ , the reward gain achieved by the optimal policy under the Lipschitz fairness constraint relative to the best constant policy. For small L L L , the gain increases approximately linearly with L L L (slope near 1 on log-log axes), matching the O ( L ) O(L) O ( L ) upper bound in Theorem 2 . As L L L increases further, the gain approaches Cost ( 0 ) \operatorname{Cost}(0) Cost ( 0 ) and the curves flatten. In the special case of d = 1 d = 1 d = 1 with uniform covariates and homogeneous rewards, the results in both figures also agree with the closed-form expressions derived in Example A.1.
Figure 4. Improvement over the best constant policy, Cost ( 0 ) − Cost ( L ) \operatorname{Cost}(0)-\operatorname{Cost}(L) Cost ( 0 ) − Cost ( L ) , on log–log axes. Columns show homogeneous (left) and heterogeneous (right) rewards; rows show uniform (top) and truncated-Gaussian (bottom) covariates.
Policy visualisation. We next compare the structure of three policies: the fair oracle p L , 1 ∗ p^*_{L,1} p L , 1 ∗ , our learned fair policy p ^ T \widehat{p}_T p T and unconstrained standard UCB. Figure 5 shows action-selection probabilities in the Arm 1 small setting with uniform covariates, d = 2 d = 2 d = 2 and L = 0.5 L = 0.5 L = 0.5 . For standard UCB, the probability of selecting arm 1 jumps between 0 and 1 across the estimated decision boundary. The fair oracle and our learned policy instead exhibit a smooth transition, illustrating the boundary randomisation and geometry structure described in Lemma 1 and Theorem 3 . We also observe that the learned policy is closer to the oracle at T = 16384 T=16384 T = 16384 than at T = 8192 T=8192 T = 8192 . The population reward gap relative to the oracle, R ( p L , 1 ∗ ) − R ( p ^ T ) R(p_{L,1}^{*})-R(\widehat{p}_{T}) R ( p L , 1 ∗ ) − R ( p T ) , decreases from 5.54 × 10 − 4 5.54\times10^{-4} 5.54 × 1 0 − 4 at T = 8192 T=8192 T = 8192 to 2.94 × 10 − 5 2.94\times10^{-5} 2.94 × 1 0 − 5 at T = 16384 T=16384 T = 16384 , a reduction of approximately 94.7%, indicating improved performance at the longer horizon.
Figure 5. Probability p ( x ) p(x) p ( x ) of choosing arm 1 in the Arm 1 small setting with uniform covariates, d = 2 d = 2 d = 2 and L = 0.5 L = 0.5 L = 0.5 . Columns show the fair oracle (left), our learned policy (middle) and unconstrained Standard UCB (right); rows show separate experiments at T = 8192 T = 8192 T = 8192 (top) and T = 16384 T = 16384 T = 16384 (bottom). The red plane marks the true decision boundary.
Regret. To evaluate learning performance, we first examine the relative regret, defined as
RelReg T = R T T R ( p L , 1 ∗ ) . \operatorname{RelReg}_{T}=\frac{\mathfrak{R}_{T}}{T R(p_{L,1}^{*})}. RelReg T = T R ( p L , 1 ∗ ) R T .
which is our algorithm’s cumulative expected reward loss relative to the fair oracle, normalised by the oracle’s cumulative expected reward. Figure 6 shows the results at L = 0.5 L=0.5 L = 0.5 , based on 50 Monte Carlo trials. Across all four baseline settings and dimensions d ∈ { 1 , 2 , 3 , 4 } d\in\{1,2,3,4\} d ∈ { 1 , 2 , 3 , 4 } , the relative regret decreases steadily with T T T , consistent with the sublinear regret bounds in Theorem 4. Additional experiments in Appendix § 16 .§ 1 further examines the sensitivity of regret to the fairness parameter L ∈ { 0.25 , 0.5 , 1 , 2 , 4 } L\in\{0.25,0.5,1,2,4\} L ∈ { 0.25 , 0.5 , 1 , 2 , 4 } under uniform covariates and homogeneous rewards with d = 2 d=2 d = 2 .
Figure 6. Relative regret for d = 1 , 2 , 3 , 4 d = 1, 2, 3, 4 d = 1 , 2 , 3 , 4 at L = 0.5 L = 0.5 L = 0.5 . Columns show homogeneous (left) and heterogeneous (right) rewards; rows show uniform (top) and truncated-Gaussian (bottom) covariates. Lines show means over 50 independent Monte Carlo trials; shaded bands show the mean ± \pm ± one standard deviation.
Appendix § 16 .§ 1 further plots the unnormalised cumulative regret R T \mathfrak{R}_{T} R T against the time horizon T T T on log–log axes for all four baseline settings and d ∈ { 1 , 2 , 3 , 4 } d\in\{1,2,3,4\} d ∈ { 1 , 2 , 3 , 4 } . The log-log curves are approximately linear, with empirical slopes close to 1 / 2 1/2 1/2 even for d = 4 d=4 d = 4 , where the worst-case theoretical upper bound scales as T 2 / 3 T^{2/3} T 2/3 up to logarithmic factors. We conjecture that the discrepancy in dimension dependence arises because the symmetric baseline designs yield the clipped affine policy in (21) , whose boundary decision p L , 1 ∗ ∣ H θ p_{L,1}^{*}\mid\mathcal{H}_{\theta} p L , 1 ∗ ∣ H θ equals to the constant 1 / 2 1/2 1/2 uniformly, whereas the minimax rate T ( d − 2 ) / ( d − 1 ) T^{(d-2)/(d-1)} T ( d − 2 ) / ( d − 1 ) reflects the difficulty of learning a spatially varying d − 1 d-1 d − 1 -dimensional boundary. To probe this nonparametric regime, we construct a mixed discrete-continuous design with d = 4 d=4 d = 4 in Appendix § 16 .§ 1.1 . It consists of a categorical covariate with six levels and another three continuous covariates, with conditional category probabilities differing across regions of the continuous covariate space. The fair oracle’s arm selection probability is thus nonconstant along the decision boundary. In this setting, we find the cumulative regret curve is approximately linear on log-log axes, with an empirical slope of about 0.65 with respect to T T T , numerically matching the theoretical exponent 2 / 3 2/3 2/3 in Theorem 4 and Theorem 7 .
Real data analysis
In this section, we evaluate the proposed methods on two binary classification datasets: Adult (Becker and Kohavi [6 ] ) and Jobs (Tankha [47 ] ). After data preprocessing, the Adult dataset contains 43303 census records with 12 demographic and socioeconomic covariates (such as age, education, occupation and working hours), where the task is to predict whether annual income exceeds $50000. The Jobs dataset contains 73462 records with 11 personal and professional covariates (such as age, education, programming experience, previous salary and computer skills) with the binary employment status as the response. Further details on sample selection, covariate definitions and preprocessing details are provided in Appendix § 16 .§ 2 .
For each dataset, we hold out a fixed yet randomly sampled 10% subset to fit a covariate preprocessing transformation based on principal component analysis (PCA) (Jolliffe and Cadima [30 ] ), which retains the first three principal components and rescales and clips their scores to [ 0 , 1 ] 3 [0, 1]^3 [ 0 , 1 ] 3 . The fitted fitted transformation is then applied to all remaining 90% of records, which serve as the online sampling dataset, and Lipschitz fairness is enforced with respect to the ℓ 1 \ell_1 ℓ 1 -distance on the transformed covariates X t ∈ [ 0 , 1 ] 3 X_t \in[0, 1]^3 X t ∈ [ 0 , 1 ] 3 .
Following Ma et al. (2026), we formulate each classification task as a two-arm bandit problem. Selecting arm 1 assigns the predicted label 1 to the current record, while selecting arm 2 assigns the predicted label 0. For a record with true label B t ∈ { 0 , 1 } B_t \in\{0, 1\} B t ∈ { 0 , 1 } , we define the arm reward as R t , 1 = B t R_{t,1} = B_t R t , 1 = B t and R t , 2 = 1 − B t R_{t,2} = 1 - B_t R t , 2 = 1 − B t . At round t t t , the algorithm observes the transformed covariates X t X_t X t , selects arm A t A_t A t and receives Y t = R t , A t Y_t = R_{t,A_t} Y t = R t , A t , which equals one for a correct prediction and zero otherwise. Since the population reward functions are unknown, regret relative to the fair oracle cannot be evaluated directly. We therefore report cumulative online accuracy,
A c c T = 1 T ∑ t = 1 T Y t , \mathrm{Acc}_T = \frac{1}{T}\sum_{t=1}^{T} Y_t, Acc T = T 1 t = 1 ∑ T Y t ,
the proportion of correct predictions over all T T T rounds, including initial exploration.
We consider T ∈ { 1024 , 2048 , 4096 , 8192 , 16384 , 32768 } T \in\{1024, 2048, 4096, 8192, 16384, 32768\} T ∈ { 1024 , 2048 , 4096 , 8192 , 16384 , 32768 } and L ∈ { 0.25 , 0.5 , 1 , 2 , 4 } L \in\{0.25, 0.5, 1, 2, 4\} L ∈ { 0.25 , 0.5 , 1 , 2 , 4 } with 50 independent replications for each dataset and ( T , L ) (T, L) ( T , L ) pair. Within each replication, we randomly permute the online sampling pool and feed its first T T T records to a separate learner for each ( T , L ) (T, L) ( T , L ) pair sharing the same permutation across all values of T T T and L L L , using the same exploration schedule T o ( T ) T_o(T) T o ( T ) as in § 6.1 . Figure 7 reports mean cumulative online accuracy against T T T . For both datasets, mean accuracy increases with T T T , indicating improved predictive performance at longer horizons. Accuracy is also consistently higher for larger L L L . At T = 32768 T = 32768 T = 32768 , increasing L L L from 0.25 to 4 raises mean accuracy from 74.31% to 76.76% for Adult and from 54.01% to 57.47% for Jobs. This reflects the theoretical trade-off between the strength of the individual fairness constraint and the rewards.
Figure 7. Cumulative online accuracy against T T T for L ∈ { 0.25 , 0.5 , 1 , 2 , 4 } L \in\{0.25, 0.5, 1, 2, 4\} L ∈ { 0.25 , 0.5 , 1 , 2 , 4 } . Panels show Adult (left) and Jobs (right). Lines show means over 50 independent Monte Carlo trials; shaded bands show the mean ± \pm ± one standard deviation.
Conclusions
In this paper, we study optimal learning in contextual bandits under a covariate-based Lipschitz fairness constraint. Building upon the canonical boundary representations, we develop computationally efficient lazy greedy and UCB-type learning algorithms and establish matching upper and lower minimax regret bounds up to poly-logarithmic factors. Together, our results characterise the fundamental cost of individual fairness at both the population and sequential learning levels, revealing a dimension phase transition at d = 4 d = 4 d = 4 , beyond which nonparametric boundary estimation becomes more challenging than parametric reward learning.
We envisage several potential extensions for future research. First, one may generalise the current framework to multi-armed contextual bandits with K ≥ 3 K \ge3 K ≥ 3 . Our results in both § 3.1 and § 3.3 reveal that the decision boundary is central to both the structure and learning difficulty of fair policies. In the setting when K ≥ 3 K \ge3 K ≥ 3 , however, the geometry may involve up to the order of O ( K 2 ) O(K^2) O ( K 2 ) pairwise decision boundaries, which substantially complicates both the canonical representation and theoretical analysis. A potential remedy is to assume lower-dimensional structures of the fairness constraint, for example imposing Lipschitz continuity on expected rewards or utilities (e.g. Chen et al., [15 ] ). Second, it would be interesting to move beyond fairness within each period and study temporal fairness constraints across time (e.g. Gupta and Kamble, [27 ] ; Cohen et al., [18 ] ). Under these settings, decisions are required to remain fair not only across covariates at a given time but also across nearby rounds over time. Deriving practical sequential learning algorithms and theoretical guarantees on the inevitable cost of fairness under these scenarios remains an intriguing area for further investigation.
Acknowledgements
Xue is supported by the EPSRC programme grant DASS (EP/Z531327/1). Jiang is supported in part by the NSFC (Nos. 72522009, 12331009 and 72432002), and by the National Key R&D Program of China (No. 2024YFA1015700). Yu is partially supported by the EPSRC programme grant DASS (EP/Z531327/1), UKRI-JST grant Statistical Safeguarding and Leverhulme Trust (Philip Leverhulme Prize).
Appendices
Additional algorithms and all technical details are collected in the Appendices. Proofs of the results in § 3 , § 4 and § 5 are provided in Appendices A, B and C, respectively. Appendix D collects the one-dimensional results, with a shared setup followed by separate presentations of the greedy algorithm for continuous covariates (analogue of Algorithm 1) and the UCB algorithm for general covariates (analogue of Algorithm 3), their theoretical guarantees, and their proofs. Appendix E contains additional structural results for arbitrary context laws, including discrete and mixed distributions. Appendix F describes how to evaluate the boundary-canonical policy at arbitrary covariates under the ℓ 1 \ell_1 ℓ 1 -metric. Appendix G provides additional simulation results and preprocessing details for the real-data analysis. Throughout the appendix, with a slight abuse of notation, unless specifically stated otherwise, let C 1 , C 2 , … > 0 C_1,C_2,\ldots> 0 C 1 , C 2 , … > 0 denote constants whose values may vary from place to place and depend on d d d .
Proofs of results in Section 3
Proof of Lemma 1
Proof of Lemma 1. Note that under Assumption 4, the decision boundary H θ ∗ \mathcal{H}_{\theta_*} H θ ∗ is non-empty. Hence, Lemma 1 is a direct consequence of Theorem 8 and Corollary 9.
Theorem 8. Suppose Assumption 1 holds and let p ∗ ∈ arg max p ∈ P L , ν R ( p ) p^* \in\arg\max_{p \in\mathcal{P}_{L,\nu}} R(p) p ∗ ∈ arg max p ∈ P L , ν R ( p ) . Define the randomisation region by
M ( p ∗ ) = { x ∈ X : 0 < p ∗ ( x ) < 1 } \mathcal{M}(p^*) = \{x \in\mathcal{X}: 0 < p^*(x) < 1\} M ( p ∗ ) = { x ∈ X : 0 < p ∗ ( x ) < 1 } and write dist ν ( x , A ) = inf z ∈ A ∥ x − z ∥ ν \operatorname{dist}_{\nu}(x,A) = \inf_{z \in A} \lVert x-z \rVert_{\nu} dist ν ( x , A ) = inf z ∈ A ∥ x − z ∥ ν for any A ⊆ X A \subseteq\mathcal{X} A ⊆ X .
If H θ ∗ = ∅ \mathcal{H}_{\theta_*} = \emptyset H θ ∗ = ∅ , then Δ ∗ \Delta^* Δ ∗ has constant sign on X \mathcal{X} X and the optimal fair policy is deterministic: p ∗ ( x ) = 1 p^*(x) = 1 p ∗ ( x ) = 1 if Δ ∗ ( x ) > 0 \Delta^*(x) > 0 Δ ∗ ( x ) > 0 for all x ∈ X x \in\mathcal{X} x ∈ X , and p ∗ ( x ) = 0 p^*(x) = 0 p ∗ ( x ) = 0 if Δ ∗ ( x ) < 0 \Delta^*(x) < 0 Δ ∗ ( x ) < 0 for all x ∈ X x \in\mathcal{X} x ∈ X .
If H θ ∗ ≠ ∅ \mathcal{H}_{\theta_*} \ne\emptyset H θ ∗ = ∅ , then it holds that
M ( p ∗ ) ⊆ { x ∈ X : dist ν ( x , H θ ∗ ) ≤ 1 L } . \mathcal{M}(p^*) \subseteq\left\{x \in\mathcal{X}: \operatorname{dist}_{\nu}(x,\mathcal{H}_{\theta_*}) \le\frac{1}{L}\right\}. M ( p ∗ ) ⊆ { x ∈ X : dist ν ( x , H θ ∗ ) ≤ L 1 } . Equivalently, if dist ν ( x , H θ ∗ ) > 1 / L \operatorname{dist}_{\nu}(x,\mathcal{H}_{\theta_*}) > 1/L dist ν ( x , H θ ∗ ) > 1/ L , then p ∗ ( x ) ∈ { 0 , 1 } p^*(x) \in\{0,1\} p ∗ ( x ) ∈ { 0 , 1 } , with p ∗ ( x ) = 1 p^*(x) = 1 p ∗ ( x ) = 1 when Δ ∗ ( x ) > 0 \Delta^*(x) > 0 Δ ∗ ( x ) > 0 and p ∗ ( x ) = 0 p^*(x) = 0 p ∗ ( x ) = 0 when Δ ∗ ( x ) < 0 \Delta^*(x) < 0 Δ ∗ ( x ) < 0 .
Proof. For every policy p p p , we can write
R ( p ) = E [ X ~ T β 2 ∗ ] + E [ Δ ∗ ( X ) p ( X ) ] . R(p) = \mathbb{E}[\widetilde{X}^{\mathsf{T}}\beta^*_2] + \mathbb{E}[\Delta^*(X)p(X)]. R ( p ) = E [ X T β 2 ∗ ] + E [ Δ ∗ ( X ) p ( X )] .
The first term is independent of p p p , so it is enough to consider only the second term.
Case 1: H θ ∗ = ∅ \mathcal{H}_{\theta_*} = \emptyset H θ ∗ = ∅ . In this case, by the continuity of Δ ∗ \Delta^* Δ ∗ , we have that Δ ∗ \Delta^* Δ ∗ has constant sign on X \mathcal{X} X . If Δ ∗ ( x ) > 0 \Delta^*(x) > 0 Δ ∗ ( x ) > 0 for all x ∈ X x \in\mathcal{X} x ∈ X , then the objective is pointwise increasing in p ( x ) p(x) p ( x ) , and hence p ∗ ( x ) = 1 p^*(x) = 1 p ∗ ( x ) = 1 is optimal. If Δ ∗ ( x ) < 0 \Delta^*(x) < 0 Δ ∗ ( x ) < 0 for all x ∈ X x \in\mathcal{X} x ∈ X , the same argument gives p ∗ ( x ) = 0 p^*(x) = 0 p ∗ ( x ) = 0 .
Case 2: H θ ∗ ≠ ∅ \mathcal{H}_{\theta_*} \ne\emptyset H θ ∗ = ∅ . This case can happen, for example under the construction of Assumption 4. Fix x 0 ∈ X x_0 \in\mathcal{X} x 0 ∈ X such that dist ν ( x 0 , H θ ∗ ) > 1 / L \operatorname{dist}_{\nu}(x_0,\mathcal{H}_{\theta_*}) > 1/L dist ν ( x 0 , H θ ∗ ) > 1/ L . We first consider the case Δ ∗ ( x 0 ) > 0 \Delta^*(x_0) > 0 Δ ∗ ( x 0 ) > 0 . Since Δ ∗ \Delta^* Δ ∗ is continuous, there exists r > 0 r > 0 r > 0 such that A = B ‾ ν ( x 0 , r ) ∩ X A = \overline{B}_{\nu}(x_0,r) \cap\mathcal{X} A = B ν ( x 0 , r ) ∩ X satisfies Δ ∗ ( x ) > 0 \Delta^*(x) > 0 Δ ∗ ( x ) > 0 for all x ∈ A x \in A x ∈ A and dist ν ( A , H θ ∗ ) > 1 / L \operatorname{dist}_{\nu}(A,\mathcal{H}_{\theta_*}) > 1/L dist ν ( A , H θ ∗ ) > 1/ L . Define
ϕ A ( x ) = max { 0 , 1 − L dist ν ( x , A ) } . (22) \phi_A(x) = \max\{0,1-L\operatorname{dist}_{\nu}(x,A)\}. \tag*{(22)} ϕ A ( x ) = max { 0 , 1 − L dist ν ( x , A )} . (22)
Since x ↦ dist ν ( x , A ) x \mapsto\operatorname{dist}_{\nu}(x,A) x ↦ dist ν ( x , A ) is $1$-Lipschitz with respect to ∥ ⋅ ∥ ν \lVert\cdot\rVert_{\nu} ∥ ⋅ ∥ ν , we have that ϕ A \phi_A ϕ A is L L L -Lipschitz. Moreover, by construction, ϕ A ∈ [ 0 , 1 ] \phi_A \in[0,1] ϕ A ∈ [ 0 , 1 ] and ϕ A ( x ) = 1 \phi_A(x) = 1 ϕ A ( x ) = 1 for all x ∈ A x \in A x ∈ A . We claim that Δ ∗ ( x ) > 0 \Delta^*(x) > 0 Δ ∗ ( x ) > 0 whenever ϕ A ( x ) > 0 \phi_A(x) > 0 ϕ A ( x ) > 0 . Indeed, if ϕ A ( x ) > 0 \phi_A(x) > 0 ϕ A ( x ) > 0 , then dist ν ( x , A ) < 1 / L \operatorname{dist}_{\nu}(x,A) < 1/L dist ν ( x , A ) < 1/ L . Suppose Δ ∗ ( x ) ≤ 0 \Delta^*(x) \le0 Δ ∗ ( x ) ≤ 0 , then for some $a \in A with ∥ x − a ∥ ν < 1 / L \lVert x-a\rVert_{\nu}<1/L ∥ x − a ∥ ν < 1/ L , the line segment between a a a and x x x intersects H θ ∗ \mathcal{H}_{\theta^{*}} H θ ∗ by convexity of X \mathcal{X} X and linearity of Δ ∗ \Delta^{*} Δ ∗ . This gives dist ν ( A , H θ ∗ ) < 1 / L \operatorname{dist}_{\nu}(A,\mathcal{H}_{\theta^{*}})<1/L dist ν ( A , H θ ∗ ) < 1/ L , which is a contradiction.
Define
p ~ ( x ) = max { p ∗ ( x ) , ϕ A ( x ) } . \widetilde{p}(x)=\max\{p^{*}(x),\phi_{A}(x)\}. p ( x ) = max { p ∗ ( x ) , ϕ A ( x )} .
Since both p ∗ p^{*} p ∗ and ϕ A \phi_{A} ϕ A are L L L -Lipschitz and the pointwise maximum of two L L L -Lipschitz functions is L L L -Lipschitz, we have that p ~ ∈ P L , ν \widetilde{p}\in\mathcal{P}_{L,\nu} p ∈ P L , ν . Also, by construction, we have that p ~ ≥ p ∗ \widetilde{p}\geq p^{*} p ≥ p ∗ everywhere and p ~ = 1 \widetilde{p}=1 p = 1 on A A A .
Suppose for contradiction that p ∗ ( x 0 ) < 1 p^{*}(x_{0})<1 p ∗ ( x 0 ) < 1 . Since p ∗ p^{*} p ∗ is continuous, after shrinking r r r if necessary, there exists an η > 0 \eta>0 η > 0 such that p ∗ ( x ) ≤ 1 − η p^{*}(x)\leq1-\eta p ∗ ( x ) ≤ 1 − η for all x ∈ A x\in A x ∈ A . Hence, we have p ~ ( x ) − p ∗ ( x ) ≥ η \widetilde{p}(x)-p^{*}(x)\geq\eta p ( x ) − p ∗ ( x ) ≥ η on A A A . Since p ~ − p ∗ \widetilde{p}-p^{*} p − p ∗ can be positive only when ϕ A > 0 \phi_{A}>0 ϕ A > 0 , the claim above gives Δ ∗ ( x ) { p ~ ( x ) − p ∗ ( x ) } ≥ 0 \Delta^{*}(x)\{\widetilde{p}(x)-p^{*}(x)\}\geq0 Δ ∗ ( x ) { p ( x ) − p ∗ ( x )} ≥ 0 for all x ∈ X x\in\mathcal{X} x ∈ X . Therefore,
R ( p ~ ) − R ( p ∗ ) = E [ Δ ∗ ( X ) { p ~ ( X ) − p ∗ ( X ) } ] ≥ ∫ A g ( x ) Δ ∗ ( x ) { p ~ ( x ) − p ∗ ( x ) } d x ≥ η ∫ A g ( x ) Δ ∗ ( x ) d x > 0 , \begin{aligned}
R(\widetilde{p})-R(p^{*})=\mathbb{E}[\Delta^{*}(X)\{\widetilde{p}(X)-p^{*}(X)\}] \\
&\geq\int_{A}g(x)\Delta^{*}(x)\{\widetilde{p}(x)-p^{*}(x)\}\,dx \\
&\geq\eta\int_{A}g(x)\Delta^{*}(x)\,dx>0,
\end{aligned} R ( p ) − R ( p ∗ ) = E [ Δ ∗ ( X ) { p ( X ) − p ∗ ( X )}] ≥ ∫ A g ( x ) Δ ∗ ( x ) { p ( x ) − p ∗ ( x )} d x ≥ η ∫ A g ( x ) Δ ∗ ( x ) d x > 0 ,
which contradicts the optimality of p ∗ p^{*} p ∗ . Hence p ∗ ( x 0 ) = 1 p^{*}(x_{0})=1 p ∗ ( x 0 ) = 1 .
The case Δ ∗ ( x 0 ) < 0 \Delta^{*}(x_{0})<0 Δ ∗ ( x 0 ) < 0 is analogous. Choose A = B ‾ ν ( x 0 , r ) ∩ X A=\overline{B}_{\nu}(x_{0},r)\cap\mathcal{X} A = B ν ( x 0 , r ) ∩ X such that Δ ∗ ( x ) < 0 \Delta^{*}(x)<0 Δ ∗ ( x ) < 0 for all x ∈ A x\in A x ∈ A and dist ν ( A , H θ ∗ ) > 1 / L \operatorname{dist}_{\nu}(A,\mathcal{H}_{\theta^{*}})>1/L dist ν ( A , H θ ∗ ) > 1/ L . With the same ϕ A \phi_{A} ϕ A as in (22) , it holds that Δ ∗ ( x ) < 0 \Delta^{*}(x)<0 Δ ∗ ( x ) < 0 whenever ϕ A ( x ) > 0 \phi_{A}(x)>0 ϕ A ( x ) > 0 . Set
p ~ ( x ) = min { p ∗ ( x ) , 1 − ϕ A ( x ) } . \widetilde{p}(x)=\min\{p^{*}(x),1-\phi_{A}(x)\}. p ( x ) = min { p ∗ ( x ) , 1 − ϕ A ( x )} .
If p ∗ ( x 0 ) > 0 p^{*}(x_{0})>0 p ∗ ( x 0 ) > 0 , then after shrinking r r r there exists η > 0 \eta>0 η > 0 such that p ∗ ( x ) ≥ η p^{*}(x)\geq\eta p ∗ ( x ) ≥ η on A A A , so p ~ ( x ) = 0 \widetilde{p}(x)=0 p ( x ) = 0 on A A A and p ∗ ( x ) − p ~ ( x ) ≥ η p^{*}(x)-\widetilde{p}(x)\geq\eta p ∗ ( x ) − p ( x ) ≥ η on A A A . Since p ~ − p ∗ \widetilde{p}-p^{*} p − p ∗ can be negative only when ϕ A > 0 \phi_{A}>0 ϕ A > 0 , we again get
R ( p ~ ) − R ( p ∗ ) = E [ Δ ∗ ( X ) { p ~ ( X ) − p ∗ ( X ) } ] > 0 , R(\widetilde{p})-R(p^{*})=\mathbb{E}[\Delta^{*}(X)\{\widetilde{p}(X)-p^{*}(X)\}]>0, R ( p ) − R ( p ∗ ) = E [ Δ ∗ ( X ) { p ( X ) − p ∗ ( X )}] > 0 ,
a contradiction. Therefore $p^{*}(x_{0})=0.
Thus, whenever dist ν ( x , H θ ∗ ) > 1 / L \operatorname{dist}_{\nu}(x,\mathcal{H}_{\theta^{*}})>1/L dist ν ( x , H θ ∗ ) > 1/ L , the optimal policy is deterministic with the correct sign. This proves the claim.
Corollary 9. Denote θ ∗ = ( a ∗ , γ ∗ ⊤ ) ⊤ \theta^{*}=(a^{*},\gamma^{*\top})^{\top} θ ∗ = ( a ∗ , γ ∗ ⊤ ) ⊤ , where a ∗ ∈ R a^{*}\in\mathbb{R} a ∗ ∈ R and γ ∗ ∈ R d \gamma^{*}\in\mathbb{R}^{d} γ ∗ ∈ R d . Suppose the assumptions of Theorem 8 hold and γ ∗ ≠ 0 \gamma^{*}\ne0 γ ∗ = 0 . Let ν ∗ \nu^{*} ν ∗ be the dual exponent of ν \nu ν , i.e. 1 / ν + 1 / ν ∗ = 1 1/\nu+1/\nu^{*}=1 1/ ν + 1/ ν ∗ = 1 , with the convention that ν ∗ = ∞ \nu^{*}=\infty ν ∗ = ∞ when ν = 1 \nu=1 ν = 1 and ν ∗ = 1 \nu^{*}=1 ν ∗ = 1 when ν = ∞ \nu=\infty ν = ∞ . Define the full affine hyperplane H ‾ θ ∗ = { z ∈ R d : ( 1 , z ⊤ ) ⊤ θ ∗ = 0 } \overline{\mathcal{H}}_{\theta^{*}}=\{z\in\mathbb{R}^{d}:(1,z^{\top})^{\top}\theta^{*}=0\} H θ ∗ = { z ∈ R d : ( 1 , z ⊤ ) ⊤ θ ∗ = 0 } . Then, for any x ∈ X x\in\mathcal{X} x ∈ X , we have that
dist ν ( x , H ‾ θ ∗ ) = ∣ Δ ∗ ( x ) ∣ ∥ γ ∗ ∥ ν ∗ . \operatorname{dist}_{\nu}(x,\overline{\mathcal{H}}_{\theta^{*}})=\frac{|\Delta^{*}(x)|}{\lVert\gamma^{*}\rVert_{\nu^{*}}}. dist ν ( x , H θ ∗ ) = ∥ γ ∗ ∥ ν ∗ ∣ Δ ∗ ( x ) ∣ . Consequently, the randomisation region satisfies
M ( p ∗ ) ⊆ { x ∈ X : ∣ Δ ∗ ( x ) ∣ ∥ γ ∗ ∥ ν ∗ ≤ 1 L } . \mathcal{M}(p^{*})\subseteq\left\{x\in\mathcal{X}:\frac{|\Delta^{*}(x)|}{\lVert\gamma^{*}\rVert_{\nu^{*}}}\leq\frac{1}{L}\right\}. M ( p ∗ ) ⊆ { x ∈ X : ∥ γ ∗ ∥ ν ∗ ∣ Δ ∗ ( x ) ∣ ≤ L 1 } . Equivalently, with u ν ∗ = γ ∗ / ∥ γ ∗ ∥ ν ∗ u_{\nu}^{*}=\gamma^{*}/\lVert\gamma^{*}\rVert_{\nu^{*}} u ν ∗ = γ ∗ / ∥ γ ∗ ∥ ν ∗ ,
M ( p ∗ ) ⊆ { x ∈ X : ∣ a ∗ ∥ γ ∗ ∥ ν ∗ + x ⊤ u ν ∗ ∣ ≤ 1 L } . \mathcal{M}(p^{*})\subseteq\left\{x\in\mathcal{X}:\left|\frac{a^{*}}{\lVert\gamma^{*}\rVert_{\nu^{*}}}+x^{\top}u_{\nu}^{*}\right|\leq\frac{1}{L}\right\}. M ( p ∗ ) ⊆ { x ∈ X : ∥ γ ∗ ∥ ν ∗ a ∗ + x ⊤ u ν ∗ ≤ L 1 } . In particular, the randomisation region is contained in a slab around the decision boundary whose full thickness, measured in ℓ ν \ell_{\nu} ℓ ν -distance, is at most $2/L.
Proof. For any h ∈ H ‾ θ ∗ h \in\overline{H}_{\theta^*} h ∈ H θ ∗ , Hölder’s inequality gives
∣ Δ ∗ ( x ) ∣ = ∣ γ ∗ ⊤ ( x − h ) ∣ ≤ ∥ γ ∗ ∥ ν ∗ ∥ x − h ∥ ν . |\Delta^*(x)| = |\gamma^{*\top}(x-h)| \le\|\gamma^*\|_{\nu^*}\|x-h\|_\nu. ∣ Δ ∗ ( x ) ∣ = ∣ γ ∗ ⊤ ( x − h ) ∣ ≤ ∥ γ ∗ ∥ ν ∗ ∥ x − h ∥ ν .
Taking the infimum over h ∈ H ‾ θ ∗ h \in\overline{H}_{\theta^*} h ∈ H θ ∗ gives dist ν ( x , H ‾ θ ∗ ) ≥ ∣ Δ ∗ ( x ) ∣ / ∥ γ ∗ ∥ ν ∗ \operatorname{dist}_\nu(x,\overline{H}_{\theta^*}) \ge|\Delta^*(x)|/\|\gamma^*\|_{\nu^*} dist ν ( x , H θ ∗ ) ≥ ∣ Δ ∗ ( x ) ∣/∥ γ ∗ ∥ ν ∗ . Conversely, by duality, there exists w ∈ R d w \in\mathbb{R}^d w ∈ R d such that ∥ w ∥ ν = 1 \|w\|_\nu= 1 ∥ w ∥ ν = 1 and γ ∗ ⊤ w = ∥ γ ∗ ∥ ν ∗ \gamma^{*\top}w = \|\gamma^*\|_{\nu^*} γ ∗ ⊤ w = ∥ γ ∗ ∥ ν ∗ . Setting h = x − { Δ ∗ ( x ) / ∥ γ ∗ ∥ ν ∗ } w h = x - \{\Delta^*(x)/\|\gamma^*\|_{\nu^*}\}w h = x − { Δ ∗ ( x ) /∥ γ ∗ ∥ ν ∗ } w gives h ∈ H ‾ θ ∗ h \in\overline{H}_{\theta^*} h ∈ H θ ∗ and ∥ x − h ∥ ν = ∣ Δ ∗ ( x ) ∣ / ∥ γ ∗ ∥ ν ∗ \|x-h\|_\nu= |\Delta^*(x)|/\|\gamma^*\|_{\nu^*} ∥ x − h ∥ ν = ∣ Δ ∗ ( x ) ∣/∥ γ ∗ ∥ ν ∗ . Therefore the distance identity holds. Since H ‾ θ ∗ \overline{H}_{\theta^*} H θ ∗ contains H θ ∗ H_{\theta^*} H θ ∗ , we have dist ν ( x , H ‾ θ ∗ ) ≤ dist ν ( x , H θ ∗ ) \operatorname{dist}_\nu(x,\overline{H}_{\theta^*}) \le\operatorname{dist}_\nu(x,H_{\theta^*}) dist ν ( x , H θ ∗ ) ≤ dist ν ( x , H θ ∗ ) . Combining this with Theorem 8 gives
M ( p ∗ ) ⊆ { x ∈ X : dist ν ( x , H ‾ θ ∗ ) ≤ 1 L } ⊆ { x ∈ X : ∣ Δ ∗ ( x ) ∣ ∥ γ ∗ ∥ ν ∗ ≤ 1 L } . \mathcal{M}(p^*) \subseteq\left\{x \in\mathcal{X} : \operatorname{dist}_\nu(x,\overline{H}_{\theta^*}) \le\frac{1}{L}\right\} \subseteq\left\{x \in\mathcal{X} : \frac{|\Delta^*(x)|}{\|\gamma^*\|_{\nu^*}} \le\frac{1}{L}\right\}. M ( p ∗ ) ⊆ { x ∈ X : dist ν ( x , H θ ∗ ) ≤ L 1 } ⊆ { x ∈ X : ∥ γ ∗ ∥ ν ∗ ∣ Δ ∗ ( x ) ∣ ≤ L 1 } .
The final display follows by writing Δ ∗ ( x ) = a ∗ + x ⊤ γ ∗ \Delta^*(x) = a^* + x^\top\gamma^* Δ ∗ ( x ) = a ∗ + x ⊤ γ ∗ .
Proof of Theorem 2
Proof of Theorem 2. The proposition is a direct consequence of Lemmas 10, 11 and 12 . It remains only to justify E ∣ Δ ∗ ( X ) ∣ ≲ 1 \mathbb{E}|\Delta^*(X)| \lesssim1 E ∣ Δ ∗ ( X ) ∣ ≲ 1 . Note that ∣ Δ ∗ ( X ) ∣ ≤ ∥ X ~ ∥ 2 ∥ β 1 ∗ − β 2 ∗ ∥ 2 ≤ 2 C β d + 1 |\Delta^*(X)| \le\|\widetilde{X}\|_2\|\beta_1^*-\beta_2^*\|_2 \le2C_\beta\sqrt{d+1} ∣ Δ ∗ ( X ) ∣ ≤ ∥ X ∥ 2 ∥ β 1 ∗ − β 2 ∗ ∥ 2 ≤ 2 C β d + 1 by Assumptions 1 and 3. Therefore, the Theorem holds.
Lemma 10 (Upper bound on the cost of fairness). Suppose Assumptions 1 and 4 hold. Write θ ∗ = ( a ∗ , γ ∗ ⊤ ) ⊤ \theta^*=(a^*,\gamma^{*\top})^\top θ ∗ = ( a ∗ , γ ∗ ⊤ ) ⊤ and let ν ∗ \nu^* ν ∗ be the dual exponent of ν \nu ν . Then
0 ≤ Cost ( L ) ≤ E [ ∣ Δ ∗ ( X ) ∣ 1 { dist ν ( X , H θ ∗ ) ≤ 1 L } ] . 0 \le\operatorname{Cost}(L) \le\mathbb{E}\left[|\Delta^*(X)|\mathbf{1}\left\{\operatorname{dist}_\nu(X,H_{\theta^*}) \le\frac{1}{L}\right\}\right]. 0 ≤ Cost ( L ) ≤ E [ ∣ Δ ∗ ( X ) ∣ 1 { dist ν ( X , H θ ∗ ) ≤ L 1 } ] .
Consequently, there exists a constant C f a i r > 0 C_{\mathrm{fair}}>0 C fair > 0 , depending only on C g C_g C g and the constants in Assumption 4, such that
Cost ( L ) ≤ C f a i r ∥ γ ∗ ∥ ν ∗ 2 L 2 . \operatorname{Cost}(L) \le C_{\mathrm{fair}}\frac{\|\gamma^*\|_{\nu^*}^2}{L^2}. Cost ( L ) ≤ C fair L 2 ∥ γ ∗ ∥ ν ∗ 2 .
Proof. By definition, we have that
Cost ( L ) = E [ { Δ ∗ ( X ) } + − Δ ∗ ( X ) p ∗ ( X ) ] . \operatorname{Cost}(L)=\mathbb{E}[\{\Delta^*(X)\}_+-\Delta^*(X)p^*(X)]. Cost ( L ) = E [{ Δ ∗ ( X ) } + − Δ ∗ ( X ) p ∗ ( X )] .
Note that the cost is non-negative. Indeed, if Δ ∗ ( x ) ≥ 0 \Delta^*(x) \ge0 Δ ∗ ( x ) ≥ 0 , then we have that
{ Δ ∗ ( x ) } + − Δ ∗ ( x ) p ∗ ( x ) = Δ ∗ ( x ) { 1 − p ∗ ( x ) } ≥ 0. \{\Delta^*(x)\}_+-\Delta^*(x)p^*(x)=\Delta^*(x)\{1-p^*(x)\}\ge0. { Δ ∗ ( x ) } + − Δ ∗ ( x ) p ∗ ( x ) = Δ ∗ ( x ) { 1 − p ∗ ( x )} ≥ 0.
If Δ ∗ ( x ) < 0 \Delta^*(x)<0 Δ ∗ ( x ) < 0 , then
{ Δ ∗ ( x ) } + − Δ ∗ ( x ) p ∗ ( x ) = − Δ ∗ ( x ) p ∗ ( x ) ≥ 0. \{\Delta^*(x)\}_+-\Delta^*(x)p^*(x)=-\Delta^*(x)p^*(x)\ge0. { Δ ∗ ( x ) } + − Δ ∗ ( x ) p ∗ ( x ) = − Δ ∗ ( x ) p ∗ ( x ) ≥ 0.
Thus { Δ ∗ ( x ) } + − Δ ∗ ( x ) p ∗ ( x ) ≥ 0 \{\Delta^*(x)\}_+-\Delta^*(x)p^*(x)\ge0 { Δ ∗ ( x ) } + − Δ ∗ ( x ) p ∗ ( x ) ≥ 0 for all x ∈ X x \in\mathcal{X} x ∈ X .
Moreover, by Lemma 1, if dist ν ( x , H θ ∗ ) > 1 / L \operatorname{dist}_\nu(x,H_{\theta^*})>1/L dist ν ( x , H θ ∗ ) > 1/ L , then p ∗ ( x ) = 1 p^*(x)=1 p ∗ ( x ) = 1 whenever Δ ∗ ( x ) > 0 \Delta^*(x)>0 Δ ∗ ( x ) > 0 and p ∗ ( x ) = 0 p^*(x)=0 p ∗ ( x ) = 0 when Δ ∗ ( x ) < 0 \Delta^*(x)<0 Δ ∗ ( x ) < 0 . Hence, { Δ ∗ ( x ) } + − Δ ∗ ( x ) p ∗ ( x ) = 0 \{\Delta^*(x)\}_+-\Delta^*(x)p^*(x)=0 { Δ ∗ ( x ) } + − Δ ∗ ( x ) p ∗ ( x ) = 0 outside the set { x ∈ X : dist ν ( x , H θ ∗ ) ≤ 1 / L } \{x \in\mathcal{X}:\operatorname{dist}_\nu(x,H_{\theta^*})\le1/L\} { x ∈ X : dist ν ( x , H θ ∗ ) ≤ 1/ L } . Since { Δ ∗ ( x ) } + − Δ ∗ ( x ) p ∗ ( x ) \{\Delta^*(x)\}_+-\Delta^*(x)p^*(x) { Δ ∗ ( x ) } + − Δ ∗ ( x ) p ∗ ( x ) is always bounded above by ∣ Δ ∗ ( x ) ∣ |\Delta^*(x)| ∣ Δ ∗ ( x ) ∣ , we obtain that
Cost ( L ) ≤ E [ ∣ Δ ∗ ( X ) ∣ 1 { dist ν ( X , H θ ∗ ) ≤ 1 L } ] . \operatorname{Cost}(L)\le\mathbb{E}\left[|\Delta^*(X)|\mathbf{1}\left\{\operatorname{dist}_\nu(X,H_{\theta^*})\le\frac{1}{L}\right\}\right]. Cost ( L ) ≤ E [ ∣ Δ ∗ ( X ) ∣ 1 { dist ν ( X , H θ ∗ ) ≤ L 1 } ] .
It remains to bound the expectation. Since H ‾ θ ∗ \overline{H}_{\theta^*} H θ ∗ contains H θ ∗ H_{\theta^*} H θ ∗ , we have dist ν ( x , H ‾ θ ∗ ) ≤ dist ν ( x , H θ ∗ ) \operatorname{dist}_\nu(x,\overline{H}_{\theta^*})\le\operatorname{dist}_\nu(x,H_{\theta^*}) dist ν ( x , H θ ∗ ) ≤ dist ν ( x , H θ ∗ ) . By Corollary 9 , we have that
dist ν ( x , H ‾ θ ∗ ) = ∣ Δ ∗ ( x ) ∣ ∥ γ ∗ ∥ ν ∗ . \operatorname{dist}_\nu(x,\overline{H}_{\theta^*})=\frac{|\Delta^*(x)|}{\|\gamma^*\|_{\nu^*}}. dist ν ( x , H θ ∗ ) = ∥ γ ∗ ∥ ν ∗ ∣ Δ ∗ ( x ) ∣ .
Thus, it holds that
{ x ∈ X : dist ν ( x , H θ ∗ ) ≤ 1 L } ⊆ { x ∈ X : ∣ Δ ∗ ( x ) ∣ ≤ ∥ γ ∗ ∥ ν ∗ L } . \left\{x \in\mathcal{X}: \operatorname{dist}_{\nu}(x,H_{\theta^*}) \le\frac{1}{L}\right\} \subseteq\left\{x \in\mathcal{X}: |\Delta^*(x)| \le\frac{\|\gamma^*\|_{\nu^*}}{L}\right\}. { x ∈ X : dist ν ( x , H θ ∗ ) ≤ L 1 } ⊆ { x ∈ X : ∣ Δ ∗ ( x ) ∣ ≤ L ∥ γ ∗ ∥ ν ∗ } .
By Assumption 4, there exists j ∗ ∈ { 1 , … , d } j^* \in\{1,\ldots,d\} j ∗ ∈ { 1 , … , d } such that θ ∗ ∈ Θ j ∗ \theta^* \in\Theta_{j^*} θ ∗ ∈ Θ j ∗ . We will write x = ( u , z ) ∈ [ 0 , 1 ] d − 1 × [ 0 , 1 ] x=(u,z) \in[0,1]^{d-1} \times[0,1] x = ( u , z ) ∈ [ 0 , 1 ] d − 1 × [ 0 , 1 ] , where z = x j ∗ z=x_{j^*} z = x j ∗ , and write Δ ∗ ( x ) = a ∗ + γ − j ∗ ∗ ⊤ u + γ j ∗ ∗ z \Delta^*(x)=a^*+\gamma^*_{-j^*}{}^\top u+\gamma^*_{j^*}z Δ ∗ ( x ) = a ∗ + γ − j ∗ ∗ ⊤ u + γ j ∗ ∗ z . Set r L = ∥ γ ∗ ∥ ν ∗ / L r_L=\|\gamma^*\|_{\nu^*}/L r L = ∥ γ ∗ ∥ ν ∗ / L .
Since g ( x ) ≤ C g g(x)\le C_g g ( x ) ≤ C g by Assumption 1,
E [ ∣ Δ ∗ ( X ) ∣ 1 { ∣ Δ ∗ ( X ) ∣ ≤ r L } ] ≤ C g ∫ [ 0 , 1 ] d ∣ Δ ∗ ( x ) ∣ 1 { ∣ Δ ∗ ( x ) ∣ ≤ r L } d x = C g ∫ [ 0 , 1 ] d − 1 ∫ 0 1 ∣ a ∗ + γ − j ∗ ∗ ⊤ u + γ j ∗ ∗ z ∣ 1 { ∣ a ∗ + γ − j ∗ ∗ ⊤ u + γ j ∗ ∗ z ∣ ≤ r L } d z d u . \begin{aligned}
\mathbb{E}\left[|\Delta^*(X)|\mathbf{1}\{|\Delta^*(X)|\le r_L\}\right]\le C_g\int_{[0,1]^d}|\Delta^*(x)|\mathbf{1}\{|\Delta^*(x)|\le r_L\}\,dx \\
&= C_g\int_{[0,1]^{d-1}}\int_0^1|a^*+\gamma^*_{-j^*}{}^\top u+\gamma^*_{j^*}z|\mathbf{1}\{|a^*+\gamma^*_{-j^*}{}^\top u+\gamma^*_{j^*}z|\le r_L\}\,dz\,du.
\end{aligned} E [ ∣ Δ ∗ ( X ) ∣ 1 { ∣ Δ ∗ ( X ) ∣ ≤ r L } ] ≤ C g ∫ [ 0 , 1 ] d ∣ Δ ∗ ( x ) ∣ 1 { ∣ Δ ∗ ( x ) ∣ ≤ r L } d x = C g ∫ [ 0 , 1 ] d − 1 ∫ 0 1 ∣ a ∗ + γ − j ∗ ∗ ⊤ u + γ j ∗ ∗ z ∣ 1 { ∣ a ∗ + γ − j ∗ ∗ ⊤ u + γ j ∗ ∗ z ∣ ≤ r L } d z d u .
By Assumption 4, there exists j ∗ ∈ { 1 , … , d } j^* \in\{1,\ldots,d\} j ∗ ∈ { 1 , … , d } such that ∣ γ j ∗ ∗ ∣ ≥ c 0 |\gamma^*_{j^*}|\ge c_0 ∣ γ j ∗ ∗ ∣ ≥ c 0 . Write x = ( u , z ) ∈ [ 0 , 1 ] d − 1 × [ 0 , 1 ] x=(u,z)\in[0,1]^{d-1}\times[0,1] x = ( u , z ) ∈ [ 0 , 1 ] d − 1 × [ 0 , 1 ] , where z = x j ∗ z=x_{j^*} z = x j ∗ and u = x − j ∗ u=x_{-j^*} u = x − j ∗ , and set A u = a ∗ + γ − j ∗ ∗ ⊤ u A_u=a^*+\gamma^*_{-j^*}{}^\top u A u = a ∗ + γ − j ∗ ∗ ⊤ u . Then
Cost ( L ) ≤ C g ∫ [ 0 , 1 ] d − 1 ∫ 0 1 ∣ A u + γ j ∗ ∗ z ∣ 1 { ∣ A u + γ j ∗ ∗ z ∣ ≤ r L } d z d u = C g ∫ [ 0 , 1 ] d − 1 1 ∣ γ j ∗ ∗ ∣ ∫ I u ∣ y ∣ 1 { ∣ y ∣ ≤ r L } d y d u ≤ C g ∫ [ 0 , 1 ] d − 1 1 ∣ γ j ∗ ∗ ∣ ∫ − r L r L ∣ y ∣ d y d u = C g ∫ [ 0 , 1 ] d − 1 r L 2 ∣ γ j ∗ ∗ ∣ d u ≤ C g c 0 r L 2 = C g c 0 ∥ γ ∗ ∥ ν ∗ 2 L 2 , \begin{aligned}
\operatorname{Cost}(L)\le C_g\int_{[0,1]^{d-1}}\int_0^1|A_u+\gamma^*_{j^*}z|\mathbf{1}\{|A_u+\gamma^*_{j^*}z|\le r_L\}\,dz\,du \\
&= C_g\int_{[0,1]^{d-1}}\frac{1}{|\gamma^*_{j^*}|}\int_{I_u}|y|\mathbf{1}\{|y|\le r_L\}\,dy\,du \\
&\le C_g\int_{[0,1]^{d-1}}\frac{1}{|\gamma^*_{j^*}|}\int_{-r_L}^{r_L}|y|\,dy\,du \\
&= C_g\int_{[0,1]^{d-1}}\frac{r_L^2}{|\gamma^*_{j^*}|}\,du \\
&\le\frac{C_g}{c_0}r_L^2=\frac{C_g}{c_0}\frac{\|\gamma^*\|_{\nu^*}^2}{L^2},
\end{aligned} Cost ( L ) ≤ C g ∫ [ 0 , 1 ] d − 1 ∫ 0 1 ∣ A u + γ j ∗ ∗ z ∣ 1 { ∣ A u + γ j ∗ ∗ z ∣ ≤ r L } d z d u = C g ∫ [ 0 , 1 ] d − 1 ∣ γ j ∗ ∗ ∣ 1 ∫ I u ∣ y ∣ 1 { ∣ y ∣ ≤ r L } d y d u ≤ C g ∫ [ 0 , 1 ] d − 1 ∣ γ j ∗ ∗ ∣ 1 ∫ − r L r L ∣ y ∣ d y d u = C g ∫ [ 0 , 1 ] d − 1 ∣ γ j ∗ ∗ ∣ r L 2 d u ≤ c 0 C g r L 2 = c 0 C g L 2 ∥ γ ∗ ∥ ν ∗ 2 ,
where I u I_u I u is the interval between A u A_u A u and A u + γ j ∗ ∗ A_u+\gamma^*_{j^*} A u + γ j ∗ ∗ ; the first equality follows from the change of variable y = A u + γ j ∗ ∗ z y=A_u+\gamma^*_{j^*}z y = A u + γ j ∗ ∗ z for fixed u u u , the first inequality follows because ∣ y ∣ 1 { ∣ y ∣ ≤ r L } |y|\mathbf{1}\{|y|\le r_L\} ∣ y ∣ 1 { ∣ y ∣ ≤ r L } is supported on [ − r L , r L ] [-r_L,r_L] [ − r L , r L ] , and the second inequality follows from Assumption 4.
The lemma holds with C f a i r = C g / c 0 C_{\mathrm{fair}}=C_g/c_0 C fair = C g / c 0 .
Lemma 11 (Lower bound on the cost of fairness when L L L is large). Suppose Assumptions 1 and 4 hold. Write θ ∗ = ( a ∗ , γ ∗ ⊤ ) ⊤ \theta^*=(a^*,\gamma^{*\top})^\top θ ∗ = ( a ∗ , γ ∗ ⊤ ) ⊤ . Then there exist constants c f a i r > 0 c_{\mathrm{fair}}>0 c fair > 0 and L 0 < ∞ L_0<\infty L 0 < ∞ , depending only on the constants in Assumptions 1 and 4, such that, for all L ≥ L 0 L\ge L_0 L ≥ L 0 , we have that
Cost ( L ) ≥ c f a i r L 2 . \operatorname{Cost}(L)\ge\frac{c_{\mathrm{fair}}}{L^2}. Cost ( L ) ≥ L 2 c fair .
Proof. Since θ ∗ ∈ Θ \theta^*\in\Theta θ ∗ ∈ Θ , there exists j ∈ { 1 , … , d } j\in\{1,\ldots,d\} j ∈ { 1 , … , d } such that θ ∗ ∈ Θ j \theta^*\in\Theta_j θ ∗ ∈ Θ j . By relabelling coordinates inside this chart, write x = ( u , z ) ∈ [ 0 , 1 ] d − 1 × [ 0 , 1 ] x=(u,z)\in[0,1]^{d-1}\times[0,1] x = ( u , z ) ∈ [ 0 , 1 ] d − 1 × [ 0 , 1 ] and write γ ∗ = ( b ∗ , c ∗ ) \gamma^*=(b^*,c^*) γ ∗ = ( b ∗ , c ∗ ) , where c 0 ≤ c ∗ ≤ C 0 c_0\le c^*\le C_0 c 0 ≤ c ∗ ≤ C 0 . Define
ϕ θ ∗ ( u ) = − a ∗ + b ∗ ⊤ u c ∗ . \phi_{\theta^*}(u)=-\frac{a^*+b^{*\top}u}{c^*}. ϕ θ ∗ ( u ) = − c ∗ a ∗ + b ∗ ⊤ u .
By Step 1 in the proof of Lemma 27 , it holds that ϕ θ ∗ \phi_{\theta^*} ϕ θ ∗ is K K K -Lipschitz with respect to ∥ ⋅ ∥ ν \|\cdot\|_{\nu} ∥ ⋅ ∥ ν , where K < ∞ K<\infty K < ∞ depends only on d , ν , B 0 , c 0 d,\nu,B_0,c_0 d , ν , B 0 , c 0 . Moreover, there exist constants m 0 > 0 m_0>0 m 0 > 0 and ρ > 0 \rho>0 ρ > 0 , depending only on the constants in Assumption 4, and a measurable set S ⊆ [ 0 , 1 ] d − 1 S \subseteq[0,1]^{d-1} S ⊆ [ 0 , 1 ] d − 1 with λ d − 1 ( S ) ≥ m 0 \lambda_{d-1}(S) \ge m_0 λ d − 1 ( S ) ≥ m 0 , such that
ρ ≤ ϕ θ ∗ ( u ) ≤ 1 − ρ , u ∈ S . (23) \rho\le\phi_{\theta^*}(u) \le1-\rho,\qquad u \in S. \tag*{(23)} ρ ≤ ϕ θ ∗ ( u ) ≤ 1 − ρ , u ∈ S . (23)
Thus the decision boundary contains a uniformly positive-measure interior patch.
Let p ∈ P L , ν p \in\mathcal{P}_{L,\nu} p ∈ P L , ν be an arbitrary policy. For u ∈ S u \in S u ∈ S , write h ( u ) = ( u , ϕ θ ∗ ( u ) ) h(u) = (u,\phi_{\theta^*}(u)) h ( u ) = ( u , ϕ θ ∗ ( u )) and set q ( u ) = p ( h ( u ) ) q(u) = p(h(u)) q ( u ) = p ( h ( u )) . For s ≥ 0 s \ge0 s ≥ 0 , define
x + ( u , s ) = ( u , ϕ θ ∗ ( u ) + s ) , x − ( u , s ) = ( u , ϕ θ ∗ ( u ) − s ) . x_+(u,s) = (u,\phi_{\theta^*}(u)+s),\qquad x_-(u,s) = (u,\phi_{\theta^*}(u)-s). x + ( u , s ) = ( u , ϕ θ ∗ ( u ) + s ) , x − ( u , s ) = ( u , ϕ θ ∗ ( u ) − s ) .
Note that, by (23) , for 0 ≤ s ≤ min { ρ , 1 / L } 0 \le s \le\min\{\rho,1/L\} 0 ≤ s ≤ min { ρ , 1/ L } , both points belong to X \mathcal{X} X . Moreover, we have that
Δ ∗ ( x + ( u , s ) ) = c ∗ s , Δ ∗ ( x − ( u , s ) ) = − c ∗ s . \Delta^*(x_+(u,s)) = c^*s,\qquad\Delta^*(x_-(u,s)) = -c^*s. Δ ∗ ( x + ( u , s )) = c ∗ s , Δ ∗ ( x − ( u , s )) = − c ∗ s .
Since p p p is L L L -Lipschitz, we further have that
p ( x + ( u , s ) ) ≤ p ( h ( u ) ) + L ∥ h ( u ) − x + ( u , s ) ∥ ν = q ( u ) + L s , p ( x − ( u , s ) ) ≥ q ( u ) − L s . p(x_+(u,s)) \le p(h(u)) + L\lVert h(u)-x_+(u,s)\rVert_\nu= q(u)+Ls,\qquad p(x_-(u,s)) \ge q(u)-Ls. p ( x + ( u , s )) ≤ p ( h ( u )) + L ∥ h ( u ) − x + ( u , s ) ∥ ν = q ( u ) + L s , p ( x − ( u , s )) ≥ q ( u ) − L s .
Define the pointwise loss relative to the unconstrained oracle by ℓ p ( x ) = { Δ ∗ ( x ) } + − Δ ∗ ( x ) p ( x ) \ell_p(x) = \{\Delta^*(x)\}_+ - \Delta^*(x)p(x) ℓ p ( x ) = { Δ ∗ ( x ) } + − Δ ∗ ( x ) p ( x ) . Then, we have that
ℓ p ( x + ( u , s ) ) = c ∗ s { 1 − p ( x + ( u , s ) ) } ≥ c ∗ s ( 1 − q ( u ) − L s ) + , \ell_p(x_+(u,s)) = c^*s\{1-p(x_+(u,s))\} \ge c^*s(1-q(u)-Ls)_+, ℓ p ( x + ( u , s )) = c ∗ s { 1 − p ( x + ( u , s ))} ≥ c ∗ s ( 1 − q ( u ) − L s ) + ,
and
ℓ p ( x − ( u , s ) ) = c ∗ s p ( x − ( u , s ) ) ≥ c ∗ s ( q ( u ) − L s ) + . \ell_p(x_-(u,s)) = c^*sp(x_-(u,s)) \ge c^*s(q(u)-Ls)_+. ℓ p ( x − ( u , s )) = c ∗ s p ( x − ( u , s )) ≥ c ∗ s ( q ( u ) − L s ) + .
For L ≥ 1 / ρ L \ge1/\rho L ≥ 1/ ρ , which gives ρ ≥ 1 / L \rho\ge1/L ρ ≥ 1/ L , integrating over s ∈ [ 0 , 1 / L ] s \in[0,1/L] s ∈ [ 0 , 1/ L ] gives
∫ 0 1 / L { ℓ p ( x + ( u , s ) ) + ℓ p ( x − ( u , s ) ) } d s ≥ c ∗ ∫ 0 1 / L s ( 1 − q ( u ) − L s ) + d s + c ∗ ∫ 0 1 / L s ( q ( u ) − L s ) + d s = c ∗ 6 L 2 { ( 1 − q ( u ) ) 3 + q ( u ) 3 } ≥ c 0 24 L 2 , \begin{aligned}
\int_0^{1/L}\{\ell_p(x_+(u,s))+\ell_p(x_-(u,s))\}\,ds
&\ge c^*\int_0^{1/L}s(1-q(u)-Ls)_+\,ds+c^*\int_0^{1/L}s(q(u)-Ls)_+\,ds \\
&= \frac{c^*}{6L^2}\{(1-q(u))^3+q(u)^3\} \\
&\ge\frac{c_0}{24L^2},
\end{aligned} ∫ 0 1/ L { ℓ p ( x + ( u , s )) + ℓ p ( x − ( u , s ))} d s ≥ c ∗ ∫ 0 1/ L s ( 1 − q ( u ) − L s ) + d s + c ∗ ∫ 0 1/ L s ( q ( u ) − L s ) + d s = 6 L 2 c ∗ {( 1 − q ( u ) ) 3 + q ( u ) 3 } ≥ 24 L 2 c 0 ,
where the last inequality follows from the fact that c ∗ ≥ c 0 c^* \ge c_0 c ∗ ≥ c 0 in Assumption 4 and ( 1 − q ) 3 + q 3 ≥ 1 / 4 (1-q)^3+q^3 \ge1/4 ( 1 − q ) 3 + q 3 ≥ 1/4 for q ∈ [ 0 , 1 ] q \in[0,1] q ∈ [ 0 , 1 ] .
Using the density lower bound g ( x ) ≥ c g g(x) \ge c_g g ( x ) ≥ c g from Assumption 1 and integrating over u ∈ S u \in S u ∈ S , we obtain
R u n ∗ − R ( p ) = E [ ℓ p ( X ) ] ≥ c g λ d − 1 ( S ) c 0 24 L 2 ≥ c g m 0 c 0 24 L 2 . R^*_{\mathrm{un}}-R(p)=\mathbb{E}[\ell_p(X)]\ge c_g\lambda_{d-1}(S)\frac{c_0}{24L^2}\ge\frac{c_gm_0c_0}{24L^2}. R un ∗ − R ( p ) = E [ ℓ p ( X )] ≥ c g λ d − 1 ( S ) 24 L 2 c 0 ≥ 24 L 2 c g m 0 c 0 .
Since this holds for every p ∈ P L , ν p \in\mathcal{P}_{L,\nu} p ∈ P L , ν , it also holds for the optimal fair policy. Therefore
Cost ( L ) = R u n ∗ − R L ∗ ≥ c g m 0 c 0 24 L 2 . \operatorname{Cost}(L)=R^*_{\mathrm{un}}-R^*_L\ge\frac{c_gm_0c_0}{24L^2}. Cost ( L ) = R un ∗ − R L ∗ ≥ 24 L 2 c g m 0 c 0 .
Taking c f a i r = c g m 0 c 0 / 24 c_{\mathrm{fair}}=c_gm_0c_0/24 c fair = c g m 0 c 0 /24 and L 0 = 1 / ρ L_0=1/\rho L 0 = 1/ ρ proves the result.
Lemma 12 (Small-L L L cost of fairness). Let R L ∗ = sup p ∈ P L , ν R ( p ) R^*_L=\sup_{p\in\mathcal{P}_{L,\nu}}R(p) R L ∗ = sup p ∈ P L , ν R ( p ) and define Cost ( L ) = R u n ∗ − R L ∗ \operatorname{Cost}(L)=R^*_{\mathrm{un}}-R^*_L Cost ( L ) = R un ∗ − R L ∗ . Let d ν = d 1 / ν d_\nu=d^{1/\nu} d ν = d 1/ ν , with the convention d ∞ = 1 d_\infty=1 d ∞ = 1 , and let C d , ν = d max { 1 / ν − 1 / 2 , 0 } C_{d,\nu}=d^{\max\{1/\nu-1/2,0\}} C d , ν = d m a x { 1/ ν − 1/2 , 0 } . Then the following statements hold.
For L = 0 L=0 L = 0 , we have that
Cost ( 0 ) = E [ { Δ ∗ ( X ) } + ] − { E [ Δ ∗ ( X ) ] } + . \operatorname{Cost}(0)=\mathbb{E}[\{\Delta^*(X)\}_+]-\{\mathbb{E}[\Delta^*(X)]\}_+. Cost ( 0 ) = E [{ Δ ∗ ( X ) } + ] − { E [ Δ ∗ ( X )] } + .
Under Assumption 1 , for every L ≥ 0 L \ge0 L ≥ 0 , we have that
Cost ( 0 ) − L d ν E ∣ Δ ∗ ( X ) ∣ ≤ Cost ( L ) ≤ Cost ( 0 ) . \operatorname{Cost}(0) - Ld_{\nu}\mathbb{E}|\Delta^*(X)| \le\operatorname{Cost}(L) \le\operatorname{Cost}(0). Cost ( 0 ) − L d ν E ∣ Δ ∗ ( X ) ∣ ≤ Cost ( L ) ≤ Cost ( 0 ) .
Equivalently,
0 ≤ Cost ( 0 ) − Cost ( L ) ≤ L d ν E ∣ Δ ∗ ( X ) ∣ . 0 \le\operatorname{Cost}(0) - \operatorname{Cost}(L) \le Ld_{\nu}\mathbb{E}|\Delta^*(X)|. 0 ≤ Cost ( 0 ) − Cost ( L ) ≤ L d ν E ∣ Δ ∗ ( X ) ∣.
Proof. Proof of (i). When L = 0 L = 0 L = 0 , every policy in P 0 , ν \mathcal{P}_{0,\nu} P 0 , ν is constant on X \mathcal{X} X . Hence any p ∈ P 0 , ν p \in\mathcal{P}_{0,\nu} p ∈ P 0 , ν can be written as p ( x ) = q p(x) = q p ( x ) = q for some q ∈ [ 0 , 1 ] q \in[0,1] q ∈ [ 0 , 1 ] . Since
R ( p ) = E [ X ~ T β 2 ∗ ] + E [ Δ ∗ ( X ) p ( X ) ] , R(p) = \mathbb{E}[\widetilde{X}^{\mathsf{T}}\beta_2^*] + \mathbb{E}[\Delta^*(X)p(X)], R ( p ) = E [ X T β 2 ∗ ] + E [ Δ ∗ ( X ) p ( X )] ,
it holds that
R 0 ∗ = E [ X ~ T β 2 ∗ ] + sup q ∈ [ 0 , 1 ] q E [ Δ ∗ ( X ) ] = E [ X ~ T β 2 ∗ ] + { E [ Δ ∗ ( X ) ] } + . (24) R_0^* = \mathbb{E}[\widetilde{X}^{\mathsf{T}}\beta_2^*] + \sup_{q \in[0,1]} q\mathbb{E}[\Delta^*(X)] = \mathbb{E}[\widetilde{X}^{\mathsf{T}}\beta_2^*] + \{\mathbb{E}[\Delta^*(X)]\}_+. \tag*{(24)} R 0 ∗ = E [ X T β 2 ∗ ] + q ∈ [ 0 , 1 ] sup q E [ Δ ∗ ( X )] = E [ X T β 2 ∗ ] + { E [ Δ ∗ ( X )] } + . (24)
On the other hand, by the definition, we have that R u n ∗ = E [ X ~ T β 2 ∗ ] + E [ { Δ ∗ ( X ) } + ] R_{\mathrm{un}}^* = \mathbb{E}[\widetilde{X}^{\mathsf{T}}\beta_2^*] + \mathbb{E}[\{\Delta^*(X)\}_+] R un ∗ = E [ X T β 2 ∗ ] + E [{ Δ ∗ ( X ) } + ] . Therefore, direct calculation gives
Cost ( 0 ) = R u n ∗ − R 0 ∗ = E [ { Δ ∗ ( X ) } + ] − { E [ Δ ∗ ( X ) ] } + . \operatorname{Cost}(0) = R_{\mathrm{un}}^* - R_0^* = \mathbb{E}[\{\Delta^*(X)\}_+] - \{\mathbb{E}[\Delta^*(X)]\}_+. Cost ( 0 ) = R un ∗ − R 0 ∗ = E [{ Δ ∗ ( X ) } + ] − { E [ Δ ∗ ( X )] } + .
Since P 0 , ν ⊆ P L , ν \mathcal{P}_{0,\nu} \subseteq\mathcal{P}_{L,\nu} P 0 , ν ⊆ P L , ν for every L ≥ 0 L \ge0 L ≥ 0 , we have R L ∗ ≥ R 0 ∗ R_L^* \ge R_0^* R L ∗ ≥ R 0 ∗ . Hence Cost ( L ) = R u n ∗ − R L ∗ ≤ R u n ∗ − R 0 ∗ = Cost ( 0 ) \operatorname{Cost}(L) = R_{\mathrm{un}}^* - R_L^* \le R_{\mathrm{un}}^* - R_0^* = \operatorname{Cost}(0) Cost ( L ) = R un ∗ − R L ∗ ≤ R un ∗ − R 0 ∗ = Cost ( 0 ) .
Proof of (ii). Fixing any p ∈ P L , ν p \in\mathcal{P}_{L,\nu} p ∈ P L , ν and any x 0 ∈ X x_0 \in\mathcal{X} x 0 ∈ X , we set q = p ( x 0 ) q = p(x_0) q = p ( x 0 ) . Since p p p is L L L -Lipschitz with respect to ∥ ⋅ ∥ ν \lVert\cdot\rVert_\nu ∥ ⋅ ∥ ν , and since X ⊆ [ 0 , 1 ] d \mathcal{X} \subseteq[0,1]^d X ⊆ [ 0 , 1 ] d under Assumption 1 , we have that ∣ p ( x ) − q ∣ ≤ L ∥ x − x 0 ∥ ν ≤ L d ν |p(x)-q| \le L\lVert x-x_0\rVert_\nu\le Ld_\nu ∣ p ( x ) − q ∣ ≤ L ∥ x − x 0 ∥ ν ≤ L d ν for every x ∈ X x \in\mathcal{X} x ∈ X . Therefore, we have that
R ( p ) = E [ X ~ T β 2 ∗ ] + E [ Δ ∗ ( X ) p ( X ) ] = E [ X ~ T β 2 ∗ ] + q E [ Δ ∗ ( X ) ] + E [ Δ ∗ ( X ) { p ( X ) − q } ] ≤ E [ X ~ T β 2 ∗ ] + { E [ Δ ∗ ( X ) ] } + + E [ ∣ Δ ∗ ( X ) ∣ ∣ p ( X ) − q ∣ ] ≤ E [ X ~ T β 2 ∗ ] + { E [ Δ ∗ ( X ) ] } + + L d ν E ∣ Δ ∗ ( X ) ∣ = R 0 ∗ + L d ν E ∣ Δ ∗ ( X ) ∣ . \begin{aligned}
R(p) &= \mathbb{E}[\widetilde{X}^{\mathsf{T}}\beta_2^*] + \mathbb{E}[\Delta^*(X)p(X)] \\
&= \mathbb{E}[\widetilde{X}^{\mathsf{T}}\beta_2^*] + q\mathbb{E}[\Delta^*(X)] + \mathbb{E}[\Delta^*(X)\{p(X)-q\}] \\
&\le\mathbb{E}[\widetilde{X}^{\mathsf{T}}\beta_2^*] + \{\mathbb{E}[\Delta^*(X)]\}_+ + \mathbb{E}[|\Delta^*(X)|\,|p(X)-q|] \\
&\le\mathbb{E}[\widetilde{X}^{\mathsf{T}}\beta_2^*] + \{\mathbb{E}[\Delta^*(X)]\}_+ + Ld_\nu\mathbb{E}|\Delta^*(X)| \\
&= R_0^* + Ld_\nu\mathbb{E}|\Delta^*(X)|.
\end{aligned} R ( p ) = E [ X T β 2 ∗ ] + E [ Δ ∗ ( X ) p ( X )] = E [ X T β 2 ∗ ] + q E [ Δ ∗ ( X )] + E [ Δ ∗ ( X ) { p ( X ) − q }] ≤ E [ X T β 2 ∗ ] + { E [ Δ ∗ ( X )] } + + E [ ∣ Δ ∗ ( X ) ∣ ∣ p ( X ) − q ∣ ] ≤ E [ X T β 2 ∗ ] + { E [ Δ ∗ ( X )] } + + L d ν E ∣ Δ ∗ ( X ) ∣ = R 0 ∗ + L d ν E ∣ Δ ∗ ( X ) ∣.
where the first inequality follows as q < 1 q < 1 q < 1 . Taking the supremum over p ∈ P L , ν p \in\mathcal{P}_{L,\nu} p ∈ P L , ν gives R L ∗ ≤ R 0 ∗ + L d ν E ∣ Δ ∗ ( X ) ∣ R_L^* \le R_0^* + Ld_\nu\mathbb{E}|\Delta^*(X)| R L ∗ ≤ R 0 ∗ + L d ν E ∣ Δ ∗ ( X ) ∣ . Hence, it holds that
Cost ( L ) = R u n ∗ − R L ∗ ≥ R u n ∗ − R 0 ∗ − L d ν E ∣ Δ ∗ ( X ) ∣ = Cost ( 0 ) − L d ν E ∣ Δ ∗ ( X ) ∣ . \operatorname{Cost}(L) = R_{\mathrm{un}}^* - R_L^* \ge R_{\mathrm{un}}^* - R_0^* - Ld_\nu\mathbb{E}|\Delta^*(X)| = \operatorname{Cost}(0) - Ld_\nu\mathbb{E}|\Delta^*(X)|. Cost ( L ) = R un ∗ − R L ∗ ≥ R un ∗ − R 0 ∗ − L d ν E ∣ Δ ∗ ( X ) ∣ = Cost ( 0 ) − L d ν E ∣ Δ ∗ ( X ) ∣.
Examples for Section 3.2
In general, the optimal fair policy need not be unique and may not admit a simple closed-form expression. To provide a better understanding of the cost of fairness described in § 3.2 , in this section, we consider a special example in which both the optimal policy and the cost of fairness can be computed explicitly.
Example A.1 (One-dimensional example). Let X = [ 0 , 1 ] \mathcal{X} = [0,1] X = [ 0 , 1 ] , X ∼ Unif [ 0 , 1 ] X \sim\operatorname{Unif}[0,1] X ∼ Unif [ 0 , 1 ] , and Δ ∗ ( x ) = x − 1 / 2 \Delta^*(x) = x - 1/2 Δ ∗ ( x ) = x − 1/2 . Consider the class of policies P L , 1 = { p : [ 0 , 1 ] → [ 0 , 1 ] : ∣ p ( x ) − p ( y ) ∣ ≤ L ∣ x − y ∣ , x , y ∈ [ 0 , 1 ] } \mathcal{P}_{L,1} = \{p : [0,1] \to[0,1] : |p(x)-p(y)| \le L|x-y|,\ x,y \in[0,1]\} P L , 1 = { p : [ 0 , 1 ] → [ 0 , 1 ] : ∣ p ( x ) − p ( y ) ∣ ≤ L ∣ x − y ∣ , x , y ∈ [ 0 , 1 ]} .
When L = 0 L = 0 L = 0 , every feasible policy is constant, and hence Cost ( 0 ) = 1 / 8 \operatorname{Cost}(0) = 1/8 Cost ( 0 ) = 1/8 .
When 0 < L ≤ 1 0 < L \le1 0 < L ≤ 1 , the optimal fair policy is not unique. For any c ∈ [ L / 2 , 1 − L / 2 ] c \in[L/2,1-L/2] c ∈ [ L /2 , 1 − L /2 ] , a policy of the form p c ( x ) = c + L ( x − 1 / 2 ) p_c(x) = c + L(x-1/2) p c ( x ) = c + L ( x − 1/2 ) is optimal, and we have that Cost ( L ) = 1 / 8 − L / 12 \operatorname{Cost}(L) = 1/8 - L/12 Cost ( L ) = 1/8 − L /12 .
When L ≥ 1 L \ge1 L ≥ 1 , an optimal fair policy is p ∗ ( x ) = [ 1 2 + L ( x − 1 2 ) ] [ 0 , 1 ] p^*(x) = \left[\frac{1}{2} + L\left(x-\frac{1}{2}\right)\right]_{[0,1]} p ∗ ( x ) = [ 2 1 + L ( x − 2 1 ) ] [ 0 , 1 ] . In this case, we have that Cost ( L ) = 1 / ( 24 L 2 ) \operatorname{Cost}(L) = 1/(24L^2) Cost ( L ) = 1/ ( 24 L 2 ) .
Thus, we have that
Cost ( 0 ) − Cost ( L ) = L 12 , when 0 < L ≤ 1 , and Cost ( L ) = 1 24 L 2 , when L > 1. \operatorname{Cost}(0)-\operatorname{Cost}(L)=\frac{L}{12},\quad\text{when }0<L\le1,\quad\text{and}\quad\operatorname{Cost}(L)=\frac{1}{24L^2},\quad\text{when }L>1. Cost ( 0 ) − Cost ( L ) = 12 L , when 0 < L ≤ 1 , and Cost ( L ) = 24 L 2 1 , when L > 1.
Proof of Example A.1. Let X = [ 0 , 1 ] \mathcal{X}=[0,1] X = [ 0 , 1 ] , X ∼ Unif [ 0 , 1 ] X\sim\operatorname{Unif}[0,1] X ∼ Unif [ 0 , 1 ] , and Δ ∗ ( x ) = x − 1 2 \Delta^*(x)=x-\frac{1}{2} Δ ∗ ( x ) = x − 2 1 . Consider the fair policy class
P L = { p : [ 0 , 1 ] → [ 0 , 1 ] : ∣ p ( x ) − p ( y ) ∣ ≤ L ∣ x − y ∣ , x , y ∈ [ 0 , 1 ] } . \mathcal{P}_L=\left\{p:[0,1]\to[0,1]:|p(x)-p(y)|\le L|x-y|,\ x,y\in[0,1]\right\}. P L = { p : [ 0 , 1 ] → [ 0 , 1 ] : ∣ p ( x ) − p ( y ) ∣ ≤ L ∣ x − y ∣ , x , y ∈ [ 0 , 1 ] } .
We focus on the non-constant part of the reward, where
R ( p ) = ∫ 0 1 ( x − 1 2 ) p ( x ) d x . R(p)=\int_0^1\left(x-\frac{1}{2}\right)p(x)\,\mathrm{d}x. R ( p ) = ∫ 0 1 ( x − 2 1 ) p ( x ) d x .
Note that the unconstrained oracle chooses arm 1 when x > 1 / 2 x>1/2 x > 1/2 and arm 2 when x < 1 / 2 x<1/2 x < 1/2 , and hence
R u n ∗ = ∫ 1 / 2 1 ( x − 1 2 ) d x = 1 8 . R^*_{\mathrm{un}}=\int_{1/2}^1\left(x-\frac{1}{2}\right)\,\mathrm{d}x=\frac{1}{8}. R un ∗ = ∫ 1/2 1 ( x − 2 1 ) d x = 8 1 .
When L = 0 L=0 L = 0 . In this case, every feasible policy must be constant on [ 0 , 1 ] [0,1] [ 0 , 1 ] . Thus any p ∈ P 0 p\in\mathcal{P}_0 p ∈ P 0 can be written as p ( x ) = q p(x)=q p ( x ) = q for some fixed q ∈ [ 0 , 1 ] q\in[0,1] q ∈ [ 0 , 1 ] . Hence
R ( p ) = ∫ 0 1 ( x − 1 2 ) q d x = q ∫ 0 1 ( x − 1 2 ) d x = 0. R(p)=\int_0^1\left(x-\frac{1}{2}\right)q\,\mathrm{d}x=q\int_0^1\left(x-\frac{1}{2}\right)\,\mathrm{d}x=0. R ( p ) = ∫ 0 1 ( x − 2 1 ) q d x = q ∫ 0 1 ( x − 2 1 ) d x = 0.
Therefore R 0 ∗ = 0 R^*_0=0 R 0 ∗ = 0 , and consequently, Cost ( 0 ) = 1 / 8 \operatorname{Cost}(0)=1/8 Cost ( 0 ) = 1/8 .
When 0 < L ≤ 1 0<L\le1 0 < L ≤ 1 . We now compute the optimal fair value R L ∗ = sup p ∈ P L R ( p ) R^*_L=\sup_{p\in\mathcal{P}_L}R(p) R L ∗ = sup p ∈ P L R ( p ) . For any p ∈ P L p\in\mathcal{P}_L p ∈ P L , write q = p ( 1 / 2 ) q=p(1/2) q = p ( 1/2 ) . By the Lipschitz constraint, for x ≥ 1 / 2 x\ge1/2 x ≥ 1/2 , p ( x ) ≤ q + L ( x − 1 / 2 ) p(x)\le q+L(x-1/2) p ( x ) ≤ q + L ( x − 1/2 ) , while for x ≤ 1 / 2 x\le1/2 x ≤ 1/2 , p ( x ) ≥ q + L ( x − 1 / 2 ) p(x)\ge q+L(x-1/2) p ( x ) ≥ q + L ( x − 1/2 ) . Since x − 1 / 2 x-1/2 x − 1/2 is non-negative on [ 1 / 2 , 1 ] [1/2,1] [ 1/2 , 1 ] and non-positive on [ 0 , 1 / 2 ] [0,1/2] [ 0 , 1/2 ] , both inequalities imply
( x − 1 2 ) p ( x ) ≤ ( x − 1 2 ) { q + L ( x − 1 2 ) } , x ∈ [ 0 , 1 ] . \left(x-\frac{1}{2}\right)p(x)\le\left(x-\frac{1}{2}\right)\left\{q+L\left(x-\frac{1}{2}\right)\right\},\qquad x\in[0,1]. ( x − 2 1 ) p ( x ) ≤ ( x − 2 1 ) { q + L ( x − 2 1 ) } , x ∈ [ 0 , 1 ] .
Therefore, we have that
R ( p ) ≤ q ∫ 0 1 ( x − 1 2 ) d x + L ∫ 0 1 ( x − 1 2 ) 2 d x = L 12 . R(p)\le q\int_0^1\left(x-\frac{1}{2}\right)\,\mathrm{d}x+L\int_0^1\left(x-\frac{1}{2}\right)^2\,\mathrm{d}x=\frac{L}{12}. R ( p ) ≤ q ∫ 0 1 ( x − 2 1 ) d x + L ∫ 0 1 ( x − 2 1 ) 2 d x = 12 L .
In this case, upper bound of L / 12 L/12 L /12 is attainable. Indeed, for any c ∈ [ L / 2 , 1 − L / 2 ] c\in[L/2,1-L/2] c ∈ [ L /2 , 1 − L /2 ] , the policy
p c ( x ) = c + L ( x − 1 2 ) p_c(x)=c+L\left(x-\frac{1}{2}\right) p c ( x ) = c + L ( x − 2 1 )
belongs to P L \mathcal{P}_L P L , and we have
R ( p c ) = ∫ 0 1 ( x − 1 2 ) { c + L ( x − 1 2 ) } d x = L 12 . R(p_c)=\int_0^1\left(x-\frac{1}{2}\right)\left\{c+L\left(x-\frac{1}{2}\right)\right\}\,\mathrm{d}x=\frac{L}{12}. R ( p c ) = ∫ 0 1 ( x − 2 1 ) { c + L ( x − 2 1 ) } d x = 12 L .
Hence, for 0 < L ≤ 1 0<L\le1 0 < L ≤ 1 , we have that R L ∗ = L / 12 R^*_L=L/12 R L ∗ = L /12 and Cost ( L ) = 1 / 8 − L / 12 \operatorname{Cost}(L)=1/8-L/12 Cost ( L ) = 1/8 − L /12 .
When L > 1 L > 1 L > 1 . In this case, by the canonical structure in Theorem 3 , it is enough to optimise over the boundary value q = p ( 1 / 2 ) q = p(1/2) q = p ( 1/2 ) . The corresponding canonical policy is
p q ( x ) = { max { 0 , q + L ( x − 1 / 2 ) } , x ≤ 1 / 2 , min { 1 , q + L ( x − 1 / 2 ) } , x ≥ 1 / 2. p_q(x) =
\begin{cases}
\max\{0, q + L(x - 1/2)\}, & x \le1/2,\\
\min\{1, q + L(x - 1/2)\}, & x \ge1/2.
\end{cases} p q ( x ) = { max { 0 , q + L ( x − 1/2 )} , min { 1 , q + L ( x − 1/2 )} , x ≤ 1/2 , x ≥ 1/2.
We compute the loss relative to the unconstrained oracle.
On the right side of the boundary, i.e. x ≥ 1 / 2 x \ge1/2 x ≥ 1/2 , write x = 1 / 2 + t x = 1/2 + t x = 1/2 + t with t ∈ [ 0 , 1 / 2 ] t \in[0, 1/2] t ∈ [ 0 , 1/2 ] . Then Δ ∗ ( x ) = t > 0 \Delta^*(x) = t > 0 Δ ∗ ( x ) = t > 0 , so the oracle chooses arm 1 and obtains contribution t t t , while the fair policy obtains contribution t p q ( 1 / 2 + t ) t p_q(1/2 + t) t p q ( 1/2 + t ) . Hence the pointwise loss is
t { 1 − p q ( 1 / 2 + t ) } = t { 1 − min ( 1 , q + L t ) } = t ( 1 − q − L t ) + . t\{1 - p_q(1/2 + t)\} = t\{1 - \min(1, q + Lt)\} = t(1 - q - Lt)_+. t { 1 − p q ( 1/2 + t )} = t { 1 − min ( 1 , q + L t )} = t ( 1 − q − L t ) + .
On the left side of the boundary, i.e. x ≤ 1 / 2 x \le1/2 x ≤ 1/2 , write x = 1 / 2 − t x = 1/2 - t x = 1/2 − t with t ∈ [ 0 , 1 / 2 ] t \in[0, 1/2] t ∈ [ 0 , 1/2 ] . Then Δ ∗ ( x ) = − t < 0 \Delta^*(x) = -t < 0 Δ ∗ ( x ) = − t < 0 , so the oracle chooses arm 2 and obtains contribution 0 0 0 , while the fair policy obtains contribution − t p q ( 1 / 2 − t ) -t p_q(1/2 - t) − t p q ( 1/2 − t ) . Hence the pointwise loss is
0 − { − t p q ( 1 / 2 − t ) } = t p q ( 1 / 2 − t ) = t max { 0 , q − L t } = t ( q − L t ) + . 0 - \{-t p_q(1/2 - t)\} = t p_q(1/2 - t) = t\max\{0, q - Lt\} = t(q - Lt)_+. 0 − { − t p q ( 1/2 − t )} = t p q ( 1/2 − t ) = t max { 0 , q − L t } = t ( q − L t ) + .
Consequently, combining the two sides, the loss relative to the unconstrained oracle is
R u n ∗ − R ( p q ) = ∫ 0 1 / 2 t ( 1 − q − L t ) + d t + ∫ 0 1 / 2 t ( q − L t ) + d t . R^*_{\mathrm{un}} - R(p_q) = \int_0^{1/2} t(1 - q - Lt)_+\,dt + \int_0^{1/2} t(q - Lt)_+\,dt. R un ∗ − R ( p q ) = ∫ 0 1/2 t ( 1 − q − L t ) + d t + ∫ 0 1/2 t ( q − L t ) + d t .
Note that the right-hand side is convex in q q q and symmetric around q = 1 / 2 q = 1/2 q = 1/2 , so it is minimised at q = 1 / 2 q = 1/2 q = 1/2 . Therefore, when L > 1 L > 1 L > 1 , the optimal policy is
p ∗ ( x ) = { 0 , x ≤ 1 2 − 1 2 L , 1 2 + L ( x − 1 2 ) , ∣ x − 1 2 ∣ ≤ 1 2 L , 1 , x ≥ 1 2 + 1 2 L . p^*(x) =
\begin{cases}
0, & x \le\frac{1}{2} - \frac{1}{2L},\\
\frac{1}{2} + L\left(x - \frac{1}{2}\right), & \left|x - \frac{1}{2}\right| \le\frac{1}{2L},\\
1, & x \ge\frac{1}{2} + \frac{1}{2L}.
\end{cases} p ∗ ( x ) = ⎩ ⎨ ⎧ 0 , 2 1 + L ( x − 2 1 ) , 1 , x ≤ 2 1 − 2 L 1 , x − 2 1 ≤ 2 L 1 , x ≥ 2 1 + 2 L 1 .
For this policy,
R u n ∗ − R L ∗ = 2 ∫ 0 1 / ( 2 L ) t ( 1 2 − L t ) d t = 1 24 L 2 . R^*_{\mathrm{un}} - R^*_L = 2\int_0^{1/(2L)} t\left(\frac{1}{2} - Lt\right)\,dt
= \frac{1}{24L^2}. R un ∗ − R L ∗ = 2 ∫ 0 1/ ( 2 L ) t ( 2 1 − L t ) d t = 24 L 2 1 .
Combining all three cases above, the cost of fairness is
Cost ( L ) = R u n ∗ − R L ∗ = { 1 8 , L = 0 , 1 8 − L 12 , 0 < L ≤ 1 , 1 24 L 2 , L > 1. \operatorname{Cost}(L) = R^*_{\mathrm{un}} - R^*_L =
\begin{cases}
\frac{1}{8}, & L = 0,\\
\frac{1}{8} - \frac{L}{12}, & 0 < L \le1,\\
\frac{1}{24L^2}, & L > 1.
\end{cases} Cost ( L ) = R un ∗ − R L ∗ = ⎩ ⎨ ⎧ 8 1 , 8 1 − 12 L , 24 L 2 1 , L = 0 , 0 < L ≤ 1 , L > 1.
Proof of Theorem 3
Proof of Theorem 3. Fix θ ∈ R d + 1 \theta\in\mathbb{R}^{d+1} θ ∈ R d + 1 such that H θ ≠ ∅ \mathcal{H}_\theta\ne\varnothing H θ = ∅ and q ∈ Q θ , L q \in\mathcal{Q}_{\theta,L} q ∈ Q θ , L .
Step 1: the envelopes agree with the trace on the boundary. Let x ∈ H θ x \in\mathcal{H}_\theta x ∈ H θ . Since x x x itself is admissible in the infimum defining U θ , q U_{\theta,q} U θ , q ,
U θ , q ( x ) ≤ q ( x ) + L ∥ x − x ∥ ν = q ( x ) . U_{\theta,q}(x) \le q(x) + L\lVert x - x\rVert_\nu= q(x). U θ , q ( x ) ≤ q ( x ) + L ∥ x − x ∥ ν = q ( x ) .
On the other hand, for every h ∈ H θ h \in\mathcal{H}_{\theta} h ∈ H θ , the L L L -Lipschitz property of q q q gives q ( h ) ≥ q ( x ) − L ∥ h − x ∥ ν q(h) \ge q(x) - L\lVert h-x\rVert_{\nu} q ( h ) ≥ q ( x ) − L ∥ h − x ∥ ν , and hence
q ( h ) + L ∥ x − h ∥ ν ≥ q ( x ) . q(h) + L\lVert x-h\rVert_{\nu} \ge q(x). q ( h ) + L ∥ x − h ∥ ν ≥ q ( x ) .
Taking the infimum over h h h gives U θ , q ( x ) ≥ q ( x ) U_{\theta,q}(x) \ge q(x) U θ , q ( x ) ≥ q ( x ) . Therefore U θ , q ( x ) = q ( x ) U_{\theta,q}(x) = q(x) U θ , q ( x ) = q ( x ) . Similarly, using h = x h=x h = x gives M θ , q ( x ) ≥ q ( x ) M_{\theta,q}(x) \ge q(x) M θ , q ( x ) ≥ q ( x ) , while q ( h ) ≤ q ( x ) + L ∥ h − x ∥ ν q(h) \le q(x) + L\lVert h-x\rVert_{\nu} q ( h ) ≤ q ( x ) + L ∥ h − x ∥ ν gives
q ( h ) − L ∥ x − h ∥ ν ≤ q ( x ) q(h) - L\lVert x-h\rVert_{\nu} \le q(x) q ( h ) − L ∥ x − h ∥ ν ≤ q ( x )
for every h ∈ H θ h \in\mathcal{H}_{\theta} h ∈ H θ . Taking the supremum over h h h gives M θ , q ( x ) ≤ q ( x ) M_{\theta,q}(x) \le q(x) M θ , q ( x ) ≤ q ( x ) . Thus M θ , q ( x ) = q ( x ) M_{\theta,q}(x) = q(x) M θ , q ( x ) = q ( x ) .
Step 2: the envelopes are L L L -Lipschitz. For any x , y ∈ X x,y \in\mathcal{X} x , y ∈ X and every h ∈ H θ h \in\mathcal{H}_{\theta} h ∈ H θ , the triangle inequality gives
q ( h ) + L ∥ x − h ∥ ν ≤ q ( h ) + L ∥ y − h ∥ ν + L ∥ x − y ∥ ν . q(h) + L\lVert x-h\rVert_{\nu} \le q(h) + L\lVert y-h\rVert_{\nu} + L\lVert x-y\rVert_{\nu}. q ( h ) + L ∥ x − h ∥ ν ≤ q ( h ) + L ∥ y − h ∥ ν + L ∥ x − y ∥ ν .
Taking the infimum over h h h yields U θ , q ( x ) ≤ U θ , q ( y ) + L ∥ x − y ∥ ν U_{\theta,q}(x) \le U_{\theta,q}(y) + L\lVert x-y\rVert_{\nu} U θ , q ( x ) ≤ U θ , q ( y ) + L ∥ x − y ∥ ν . Interchanging x x x and y y y gives
∣ U θ , q ( x ) − U θ , q ( y ) ∣ ≤ L ∥ x − y ∥ ν . \lvert U_{\theta,q}(x) - U_{\theta,q}(y)\rvert\le L\lVert x-y\rVert_{\nu}. ∣ U θ , q ( x ) − U θ , q ( y )∣ ≤ L ∥ x − y ∥ ν .
Similarly, for every h ∈ H θ h \in\mathcal{H}_{\theta} h ∈ H θ ,
q ( h ) − L ∥ x − h ∥ ν ≤ q ( h ) − L ∥ y − h ∥ ν + L ∥ x − y ∥ ν . q(h) - L\lVert x-h\rVert_{\nu} \le q(h) - L\lVert y-h\rVert_{\nu} + L\lVert x-y\rVert_{\nu}. q ( h ) − L ∥ x − h ∥ ν ≤ q ( h ) − L ∥ y − h ∥ ν + L ∥ x − y ∥ ν .
Taking the supremum over h h h and then interchanging x x x and y y y gives
∣ M θ , q ( x ) − M θ , q ( y ) ∣ ≤ L ∥ x − y ∥ ν . \lvert M_{\theta,q}(x) - M_{\theta,q}(y)\rvert\le L\lVert x-y\rVert_{\nu}. ∣ M θ , q ( x ) − M θ , q ( y )∣ ≤ L ∥ x − y ∥ ν .
Step 3: the unclipped piecewise envelope is globally L L L -Lipschitz. Define
F θ , q ( x ) = { U θ , q ( x ) , x ∈ X θ + , M θ , q ( x ) , x ∈ X θ − . F_{\theta,q}(x) =
\begin{cases}
U_{\theta,q}(x), & x \in\mathcal{X}_{\theta}^{+},\\
M_{\theta,q}(x), & x \in\mathcal{X}_{\theta}^{-}.
\end{cases} F θ , q ( x ) = { U θ , q ( x ) , M θ , q ( x ) , x ∈ X θ + , x ∈ X θ − .
If x x x and y y y lie on the same side of H θ \mathcal{H}_{\theta} H θ , the claim follows from Step 2. It remains to consider the cross-boundary case. Let x ∈ X θ + x \in\mathcal{X}_{\theta}^{+} x ∈ X θ + and y ∈ X θ − y \in\mathcal{X}_{\theta}^{-} y ∈ X θ − . If x ~ T θ = 0 \widetilde{x}^{\mathsf{T}}\theta= 0 x T θ = 0 , take h = x h=x h = x . Otherwise, set
a 0 = x ~ T θ x ~ T θ − y ~ T θ and h = ( 1 − a 0 ) x + a 0 y . a_{0} = \frac{\widetilde{x}^{\mathsf{T}}\theta}{\widetilde{x}^{\mathsf{T}}\theta- \widetilde{y}^{\mathsf{T}}\theta}
\qquad\text{and}\qquad
h = (1-a_{0})x + a_{0}y. a 0 = x T θ − y T θ x T θ and h = ( 1 − a 0 ) x + a 0 y .
Since x ~ T θ > 0 \widetilde{x}^{\mathsf{T}}\theta> 0 x T θ > 0 and y ~ T θ < 0 \widetilde{y}^{\mathsf{T}}\theta< 0 y T θ < 0 , we have a 0 ∈ ( 0 , 1 ) a_{0} \in(0,1) a 0 ∈ ( 0 , 1 ) . By convexity of X = [ 0 , 1 ] d \mathcal{X} = [0,1]^{d} X = [ 0 , 1 ] d , h ∈ X h \in\mathcal{X} h ∈ X . Moreover,
h ~ T θ = ( 1 − a 0 ) x ~ T θ + a 0 y ~ T θ = 0 , \widetilde{h}^{\mathsf{T}}\theta= (1-a_{0})\widetilde{x}^{\mathsf{T}}\theta+ a_{0}\widetilde{y}^{\mathsf{T}}\theta= 0, h T θ = ( 1 − a 0 ) x T θ + a 0 y T θ = 0 ,
so h ∈ H θ h \in\mathcal{H}_{\theta} h ∈ H θ . Since h h h lies on the line segment between x x x and y y y ,
∥ x − h ∥ ν + ∥ y − h ∥ ν = ∥ x − y ∥ ν . \lVert x-h\rVert_{\nu} + \lVert y-h\rVert_{\nu} = \lVert x-y\rVert_{\nu}. ∥ x − h ∥ ν + ∥ y − h ∥ ν = ∥ x − y ∥ ν .
Using this particular boundary point h h h in the definitions of U θ , q U_{\theta,q} U θ , q and M θ , q M_{\theta,q} M θ , q gives
U θ , q ( x ) − M θ , q ( y ) ≤ q ( h ) + L ∥ x − h ∥ ν − { q ( h ) − L ∥ y − h ∥ ν } = L ∥ x − y ∥ ν . U_{\theta,q}(x) - M_{\theta,q}(y) \le q(h) + L\lVert x-h\rVert_{\nu} - \{q(h) - L\lVert y-h\rVert_{\nu}\} = L\lVert x-y\rVert_{\nu}. U θ , q ( x ) − M θ , q ( y ) ≤ q ( h ) + L ∥ x − h ∥ ν − { q ( h ) − L ∥ y − h ∥ ν } = L ∥ x − y ∥ ν .
For the reverse direction, for every g ∈ H θ g \in\mathcal{H}_{\theta} g ∈ H θ ,
q ( g ) − L ∥ y − g ∥ ν ≤ q ( h ) + L ∥ g − h ∥ ν − L ∥ y − g ∥ ν ≤ q ( h ) + L ∥ y − h ∥ ν , q(g) - L\lVert y-g\rVert_{\nu} \le q(h) + L\lVert g-h\rVert_{\nu} - L\lVert y-g\rVert_{\nu} \le q(h) + L\lVert y-h\rVert_{\nu}, q ( g ) − L ∥ y − g ∥ ν ≤ q ( h ) + L ∥ g − h ∥ ν − L ∥ y − g ∥ ν ≤ q ( h ) + L ∥ y − h ∥ ν ,
where the last inequality follows from ∥ g − h ∥ ν ≤ ∥ g − y ∥ ν + ∥ y − h ∥ ν \lVert g-h\rVert_{\nu}\le\lVert g-y\rVert_{\nu}+\lVert y-h\rVert_{\nu} ∥ g − h ∥ ν ≤ ∥ g − y ∥ ν + ∥ y − h ∥ ν . Taking the supremum over g g g gives
M θ , q ( y ) ≤ q ( h ) + L ∥ y − h ∥ ν . M_{\theta,q}(y)\le q(h)+L\lVert y-h\rVert_{\nu}. M θ , q ( y ) ≤ q ( h ) + L ∥ y − h ∥ ν .
Similarly, for every g ∈ H θ g\in\mathcal{H}_{\theta} g ∈ H θ ,
q ( g ) + L ∥ x − g ∥ ν ≥ q ( h ) − L ∥ g − h ∥ ν + L ∥ x − g ∥ ν ≥ q ( h ) − L ∥ x − h ∥ ν . q(g)+L\lVert x-g\rVert_{\nu}\ge q(h)-L\lVert g-h\rVert_{\nu}+L\lVert x-g\rVert_{\nu}\ge q(h)-L\lVert x-h\rVert_{\nu}. q ( g ) + L ∥ x − g ∥ ν ≥ q ( h ) − L ∥ g − h ∥ ν + L ∥ x − g ∥ ν ≥ q ( h ) − L ∥ x − h ∥ ν .
Taking the infimum over g g g gives
U θ , q ( x ) ≥ q ( h ) − L ∥ x − h ∥ ν . U_{\theta,q}(x)\ge q(h)-L\lVert x-h\rVert_{\nu}. U θ , q ( x ) ≥ q ( h ) − L ∥ x − h ∥ ν .
Therefore
M θ , q ( y ) − U θ , q ( x ) ≤ L ∥ y − h ∥ ν + L ∥ x − h ∥ ν = L ∥ x − y ∥ ν . M_{\theta,q}(y)-U_{\theta,q}(x)\le L\lVert y-h\rVert_{\nu}+L\lVert x-h\rVert_{\nu}=L\lVert x-y\rVert_{\nu}. M θ , q ( y ) − U θ , q ( x ) ≤ L ∥ y − h ∥ ν + L ∥ x − h ∥ ν = L ∥ x − y ∥ ν .
Thus F θ , q F_{\theta,q} F θ , q is L L L -Lipschitz on X \mathcal{X} X .
Step 4: clipping preserves the Lipschitz constant. Since q q q takes values in [ 0 , 1 ] [0,1] [ 0 , 1 ] , we have U θ , q ( x ) ≥ 0 U_{\theta,q}(x)\ge0 U θ , q ( x ) ≥ 0 and M θ , q ( x ) ≤ 1 M_{\theta,q}(x)\le1 M θ , q ( x ) ≤ 1 for every x ∈ X x\in\mathcal{X} x ∈ X . Hence p θ , q ( x ) = [ F θ , q ( x ) ] [ 0 , 1 ] p_{\theta,q}(x)=[F_{\theta,q}(x)]_{[0,1]} p θ , q ( x ) = [ F θ , q ( x ) ] [ 0 , 1 ] .
The clipping map u ↦ [ u ] [ 0 , 1 ] u\mapsto[u]_{[0,1]} u ↦ [ u ] [ 0 , 1 ] is 1-Lipschitz on R \mathbb{R} R , so for all x , y ∈ X x,y\in\mathcal{X} x , y ∈ X ,
∣ p θ , q ( x ) − p θ , q ( y ) ∣ ≤ ∣ F θ , q ( x ) − F θ , q ( y ) ∣ ≤ L ∥ x − y ∥ ν . \lvert p_{\theta,q}(x)-p_{\theta,q}(y)\rvert\le\lvert F_{\theta,q}(x)-F_{\theta,q}(y)\rvert\le L\lVert x-y\rVert_{\nu}. ∣ p θ , q ( x ) − p θ , q ( y )∣ ≤ ∣ F θ , q ( x ) − F θ , q ( y )∣ ≤ L ∥ x − y ∥ ν .
Therefore p θ , q ∈ P L , ν p_{\theta,q}\in\mathcal{P}_{L,\nu} p θ , q ∈ P L , ν .
Step 5: domination by the canonical representative. Fix θ \theta θ with H θ ≠ ∅ \mathcal{H}_{\theta}\ne\varnothing H θ = ∅ and p ∈ P L , ν p\in\mathcal{P}_{L,\nu} p ∈ P L , ν , and set q = p ∣ H θ q=p|_{\mathcal{H}_{\theta}} q = p ∣ H θ . If x ∈ X θ + x\in\mathcal{X}_{\theta}^{+} x ∈ X θ + , then for every h ∈ H θ h\in\mathcal{H}_{\theta} h ∈ H θ ,
p ( x ) ≤ q ( h ) + L ∥ x − h ∥ ν . p(x)\le q(h)+L\lVert x-h\rVert_{\nu}. p ( x ) ≤ q ( h ) + L ∥ x − h ∥ ν .
Taking the infimum over h h h gives p ( x ) ≤ U θ , q ( x ) p(x)\le U_{\theta,q}(x) p ( x ) ≤ U θ , q ( x ) . Since p ( x ) ≤ 1 p(x)\le1 p ( x ) ≤ 1 , we get
p ( x ) ≤ min { 1 , U θ , q ( x ) } = p θ , q ( x ) . p(x)\le\min\{1,U_{\theta,q}(x)\}=p_{\theta,q}(x). p ( x ) ≤ min { 1 , U θ , q ( x )} = p θ , q ( x ) .
If x ∈ X θ − x\in\mathcal{X}_{\theta}^{-} x ∈ X θ − , then for every h ∈ H θ h\in\mathcal{H}_{\theta} h ∈ H θ ,
p ( x ) ≥ q ( h ) − L ∥ x − h ∥ ν . p(x)\ge q(h)-L\lVert x-h\rVert_{\nu}. p ( x ) ≥ q ( h ) − L ∥ x − h ∥ ν .
Taking the supremum over h h h gives p ( x ) ≥ M θ , q ( x ) p(x)\ge M_{\theta,q}(x) p ( x ) ≥ M θ , q ( x ) . Since p ( x ) ≥ 0 p(x)\ge0 p ( x ) ≥ 0 , we get
p ( x ) ≥ max { 0 , M θ , q ( x ) } = p θ , q ( x ) . p(x)\ge\max\{0,M_{\theta,q}(x)\}=p_{\theta,q}(x). p ( x ) ≥ max { 0 , M θ , q ( x )} = p θ , q ( x ) .
By definition, ( x ~ T θ ) ≥ 0 (\widetilde{x}^{\mathsf{T}}\theta)\ge0 ( x T θ ) ≥ 0 on X θ + \mathcal{X}_{\theta}^{+} X θ + and ( x ~ T θ ) < 0 (\widetilde{x}^{\mathsf{T}}\theta)<0 ( x T θ ) < 0 on X θ − \mathcal{X}_{\theta}^{-} X θ − . Therefore the two inequalities above imply
( x ~ T θ ) p θ , q ( x ) ≥ ( x ~ T θ ) p ( x ) for all x ∈ X . (\widetilde{x}^{\mathsf{T}}\theta)p_{\theta,q}(x)\ge(\widetilde{x}^{\mathsf{T}}\theta)p(x)\qquad\text{for all }x\in\mathcal{X}. ( x T θ ) p θ , q ( x ) ≥ ( x T θ ) p ( x ) for all x ∈ X .
Step 6: reduction of the optimiser. Fix any distribution of X X X on X \mathcal{X} X and retain the parameter θ \theta θ with H θ ≠ ∅ \mathcal{H}_{\theta}\ne\varnothing H θ = ∅ . Taking expectations in the pointwise inequality in Step 5 shows that replacing any p ∈ P L , ν p\in\mathcal{P}_{L,\nu} p ∈ P L , ν by p θ , p ∣ H θ p_{\theta,p|_{\mathcal{H}_{\theta}}} p θ , p ∣ H θ cannot decrease the objective E { ( X ~ T θ ) p ( X ) } \mathbb{E}\{(\widetilde{X}^{\mathsf{T}}\theta)p(X)\} E {( X T θ ) p ( X )} .
The class P L , ν \mathcal{P}_{L,\nu} P L , ν is uniformly bounded and equicontinuous on the compact set X \mathcal{X} X . It is also closed under uniform limits: if p n ∈ P L , ν p_n\in\mathcal{P}_{L,\nu} p n ∈ P L , ν and p n → p p_n\to p p n → p uniformly, then 0 ≤ p ≤ 1 0\le p\le1 0 ≤ p ≤ 1 and
∣ p ( x ) − p ( y ) ∣ = lim n → ∞ ∣ p n ( x ) − p n ( y ) ∣ ≤ L ∥ x − y ∥ ν , x , y ∈ X . \lvert p(x)-p(y)\rvert=\lim_{n\to\infty}\lvert p_n(x)-p_n(y)\rvert\le L\lVert x-y\rVert_{\nu},\qquad x,y\in\mathcal{X}. ∣ p ( x ) − p ( y )∣ = n → ∞ lim ∣ p n ( x ) − p n ( y )∣ ≤ L ∥ x − y ∥ ν , x , y ∈ X .
Thus P L , ν \mathcal{P}_{L,\nu} P L , ν is compact in the sup norm by Arzela–Ascoli (e.g. Theorem 7.25 in Rudin, 1976). The objective is continuous in this norm because, for p , r ∈ P L , ν p,r\in\mathcal{P}_{L,\nu} p , r ∈ P L , ν ,
∣ E [ ( X ~ ⊤ θ ) { p ( X ) − r ( X ) } ] ∣ ≤ d + 1 ∥ θ ∥ 2 ∥ p − r ∥ ∞ . \left|\mathbb{E}\left[(\widetilde{X}^{\top}\theta)\{p(X)-r(X)\}\right]\right|\le\sqrt{d+1}\|\theta\|_2\|p-r\|_{\infty}. E [ ( X ⊤ θ ) { p ( X ) − r ( X )} ] ≤ d + 1 ∥ θ ∥ 2 ∥ p − r ∥ ∞ .
An optimiser p ∘ ∈ P L , ν p^{\circ}\in\mathcal{P}_{L,\nu} p ∘ ∈ P L , ν therefore exists. Set q θ = p ∘ ∣ H θ q_{\theta}=p^{\circ}|_{\mathcal{H}_{\theta}} q θ = p ∘ ∣ H θ . By Step 5 ,
E { ( X ~ ⊤ θ ) p θ , q θ ( X ) } ≥ E { ( X ~ ⊤ θ ) p ∘ ( X ) } . \mathbb{E}\{(\widetilde{X}^{\top}\theta)p_{\theta,q_{\theta}}(X)\}\ge\mathbb{E}\{(\widetilde{X}^{\top}\theta)p^{\circ}(X)\}. E {( X ⊤ θ ) p θ , q θ ( X )} ≥ E {( X ⊤ θ ) p ∘ ( X )} .
The reverse inequality follows from optimality of p ∘ p^{\circ} p ∘ and feasibility of p θ , q θ p_{\theta,q_{\theta}} p θ , q θ . Hence q θ q_{\theta} q θ attains the same optimal value.
Proofs of results in Section 4
Proof of Lemma 13
Lemma 13. Fix θ ∈ R d + 1 \theta\in\mathbb{R}^{d+1} θ ∈ R d + 1 with H θ ≠ ∅ \mathcal{H}_{\theta}\ne\varnothing H θ = ∅ . For any given contexts X 1 , … , X n ∈ [ 0 , 1 ] d X_1,\ldots,X_n\in[0,1]^d X 1 , … , X n ∈ [ 0 , 1 ] d , consider the finite-dimensional linear program
z ^ = ( z ^ 1 , … , z ^ n ) ∈ a r g m a x z 1 , … , z n 1 n ∑ i = 1 n θ ⊤ X ~ i z i , (25) \widehat{z}=(\widehat{z}_1,\ldots,\widehat{z}_n)\in\mathop{\rm arg\,max}_{z_1,\ldots,z_n}\frac{1}{n}\sum_{i=1}^{n}\theta^{\top}\widetilde{X}_i z_i, \tag*{(25)} z = ( z 1 , … , z n ) ∈ arg max z 1 , … , z n n 1 i = 1 ∑ n θ ⊤ X i z i , (25)
s.t. 0 ≤ z i ≤ 1 , i = 1 , … , n , \text{s.t.}\quad0\le z_i\le1,\qquad i=1,\ldots,n, s.t. 0 ≤ z i ≤ 1 , i = 1 , … , n ,
∣ z i − z j ∣ ≤ L ∥ X i − X j ∥ ν , i , j = 1 , … , n . |z_i-z_j|\le L\|X_i-X_j\|_{\nu},\qquad i,j=1,\ldots,n. ∣ z i − z j ∣ ≤ L ∥ X i − X j ∥ ν , i , j = 1 , … , n .
Define the McShane extension of z ^ \widehat{z} z by p ^ ( x ) = min { 1 , min 1 ≤ j ≤ n { z ^ j + L ∥ x − X j ∥ ν } } \widehat{p}(x)=\min\{1,\min_{1\le j\le n}\{\widehat{z}_j+L\|x-X_j\|_{\nu}\}\} p ( x ) = min { 1 , min 1 ≤ j ≤ n { z j + L ∥ x − X j ∥ ν }} , for all x ∈ [ 0 , 1 ] d x\in[0,1]^d x ∈ [ 0 , 1 ] d . Let q ^ = p ^ ∣ H θ \widehat{q}=\widehat{p}|_{\mathcal{H}_{\theta}} q = p ∣ H θ . Then, we have that q ^ ∈ Q θ , L \widehat{q}\in\mathcal{Q}_{\theta,L} q ∈ Q θ , L and
q ^ ∈ a r g m a x q ∈ Q θ , L 1 n ∑ i = 1 n X ~ i ⊤ θ p θ , q ( X i ) . \widehat{q}\in\mathop{\rm arg\,max}_{q\in\mathcal{Q}_{\theta,L}}\frac{1}{n}\sum_{i=1}^{n}\widetilde{X}_i^{\top}\theta p_{\theta,q}(X_i). q ∈ arg max q ∈ Q θ , L n 1 i = 1 ∑ n X i ⊤ θ p θ , q ( X i ) .
Proof. We prove the result for every θ ∈ R d + 1 \theta\in\mathbb{R}^{d+1} θ ∈ R d + 1 such that H θ ≠ ∅ \mathcal{H}_{\theta}\ne\varnothing H θ = ∅ , which includes both Θ \Theta Θ and Θ ~ \widetilde{\Theta} Θ . Define
OPT θ , n = sup q ∈ Q θ , L 1 n ∑ i = 1 n θ ⊤ X ~ i p θ , q ( X i ) \operatorname{OPT}_{\theta,n}=\sup_{q\in\mathcal{Q}_{\theta,L}}\frac{1}{n}\sum_{i=1}^{n}\theta^{\top}\widetilde{X}_i p_{\theta,q}(X_i) OPT θ , n = q ∈ Q θ , L sup n 1 i = 1 ∑ n θ ⊤ X i p θ , q ( X i )
and let OPT L P \operatorname{OPT}_{\mathrm{LP}} OPT LP denote the optimal value of (25) . The feasible region of (25) is a non-empty compact polytope and hence an optimiser z ^ \widehat{z} z exists.
Step 1: Reduction from canonical policies to the finite LP. Fix any q ∈ Q θ , L q\in\mathcal{Q}_{\theta,L} q ∈ Q θ , L . By Theorem 3 , we have p θ , q ∈ P L , ν p_{\theta,q}\in\mathcal{P}_{L,\nu} p θ , q ∈ P L , ν . Therefore, the sample values
z i = p θ , q ( X i ) , i = 1 , … , n , z_i=p_{\theta,q}(X_i),\qquad i=1,\ldots,n, z i = p θ , q ( X i ) , i = 1 , … , n ,
satisfy 0 ≤ z i ≤ 1 0\le z_i\le1 0 ≤ z i ≤ 1 for i = 1 , … , n i=1,\ldots,n i = 1 , … , n , and
∣ z i − z j ∣ = ∣ p θ , q ( X i ) − p θ , q ( X j ) ∣ ≤ L ∥ X i − X j ∥ ν , i , j = 1 , … , n . |z_i-z_j|=|p_{\theta,q}(X_i)-p_{\theta,q}(X_j)|\le L\|X_i-X_j\|_{\nu},\qquad i,j=1,\ldots,n. ∣ z i − z j ∣ = ∣ p θ , q ( X i ) − p θ , q ( X j ) ∣ ≤ L ∥ X i − X j ∥ ν , i , j = 1 , … , n .
Thus ( z 1 , … , z n ) (z_1,\ldots,z_n) ( z 1 , … , z n ) is a feasible solution to (25) . It then follows that
1 n ∑ i = 1 n θ ⊤ X ~ i p θ , q ( X i ) ≤ OPT L P . \frac{1}{n}\sum_{i=1}^{n}\theta^{\top}\widetilde{X}_i p_{\theta,q}(X_i)\le\operatorname{OPT}_{\mathrm{LP}}. n 1 i = 1 ∑ n θ ⊤ X i p θ , q ( X i ) ≤ OPT LP .
Taking the supremum over q ∈ Q θ , L q \in\mathcal{Q}_{\theta,L} q ∈ Q θ , L gives
O P T θ , n ≤ O P T L P . (26) \mathrm{OPT}_{\theta,n} \leq\mathrm{OPT}_{\mathrm{LP}}. \tag*{(26)} OPT θ , n ≤ OPT LP . (26)
Step 2: McShane interpolation of the linear program (LP) optimiser. For each j ∈ { 1 , … , n } j \in\{1,\ldots,n\} j ∈ { 1 , … , n } , the map x ↦ z ^ j + L ∥ x − X j ∥ ν x \mapsto\widehat{z}_j + L\|x-X_j\|_\nu x ↦ z j + L ∥ x − X j ∥ ν is L L L -Lipschitz. Hence its pointwise minimum is also L L L -Lipschitz. Thus, we have p ^ ∈ P L , ν \widehat{p} \in\mathcal{P}_{L,\nu} p ∈ P L , ν .
We next verify that p ^ \widehat{p} p interpolates the LP solution. Fix i ∈ { 1 , … , n } i \in\{1,\ldots,n\} i ∈ { 1 , … , n } . Note that
p ^ ( X i ) = min 1 ≤ j ≤ n { z ^ j + L ∥ X i − X j ∥ ν } ≤ z ^ i + L ∥ X i − X i ∥ ν = z ^ i . \widehat{p}(X_i)=\min_{1\leq j\leq n}\{\widehat{z}_j+L\|X_i-X_j\|_\nu\}\leq\widehat{z}_i+L\|X_i-X_i\|_\nu=\widehat{z}_i. p ( X i ) = 1 ≤ j ≤ n min { z j + L ∥ X i − X j ∥ ν } ≤ z i + L ∥ X i − X i ∥ ν = z i .
On the other hand, by feasibility of z ^ \widehat{z} z , for every j = 1 , … , n j=1,\ldots,n j = 1 , … , n , we have that
z ^ j + L ∥ X i − X j ∥ ν ≥ z ^ i . \widehat{z}_j+L\|X_i-X_j\|_\nu\geq\widehat{z}_i. z j + L ∥ X i − X j ∥ ν ≥ z i .
Taking the minimum over j j j and using 0 ≤ z ^ i ≤ 1 0\leq\widehat{z}_i\leq1 0 ≤ z i ≤ 1 gives
p ^ ( X i ) ≥ z ^ i . \widehat{p}(X_i)\geq\widehat{z}_i. p ( X i ) ≥ z i .
Therefore, we have that p ^ ( X i ) = z ^ i \widehat{p}(X_i)=\widehat{z}_i p ( X i ) = z i for all i = 1 , … , n i=1,\ldots,n i = 1 , … , n . Consequently,
1 n ∑ i = 1 n θ T X ~ i p ^ ( X i ) = 1 n ∑ i = 1 n θ T X ~ i z ^ i = O P T L P . (27) \frac{1}{n}\sum_{i=1}^{n}\theta^{\mathsf{T}}\widetilde{X}_i\widehat{p}(X_i)=\frac{1}{n}\sum_{i=1}^{n}\theta^{\mathsf{T}}\widetilde{X}_i\widehat{z}_i=\mathrm{OPT}_{\mathrm{LP}}. \tag*{(27)} n 1 i = 1 ∑ n θ T X i p ( X i ) = n 1 i = 1 ∑ n θ T X i z i = OPT LP . (27)
Step 3: Canonicalisation of the interpolated LP optimiser. Let q ^ = p ^ ∣ H θ \widehat{q}=\widehat{p}|_{\mathcal{H}_\theta} q = p ∣ H θ . Since p ^ ∈ P L , ν \widehat{p}\in\mathcal{P}_{L,\nu} p ∈ P L , ν , we have q ^ ∈ Q θ , L \widehat{q}\in\mathcal{Q}_{\theta,L} q ∈ Q θ , L . Applying Theorem 3 gives
( x ~ T θ ) p θ , q ^ ( x ) ≥ ( x ~ T θ ) p ^ ( x ) , x ∈ [ 0 , 1 ] d . (28) (\widetilde{x}^{\mathsf{T}}\theta)p_{\theta,\widehat{q}}(x)\geq(\widetilde{x}^{\mathsf{T}}\theta)\widehat{p}(x),\qquad x\in[0,1]^d. \tag*{(28)} ( x T θ ) p θ , q ( x ) ≥ ( x T θ ) p ( x ) , x ∈ [ 0 , 1 ] d . (28)
Applying (28) at the sample points and using (27) , we obtain
1 n ∑ i = 1 n θ T X ~ i p θ , q ^ ( X i ) ≥ 1 n ∑ i = 1 n θ T X ~ i p ^ ( X i ) = O P T L P . \frac{1}{n}\sum_{i=1}^{n}\theta^{\mathsf{T}}\widetilde{X}_ip_{\theta,\widehat{q}}(X_i)\geq\frac{1}{n}\sum_{i=1}^{n}\theta^{\mathsf{T}}\widetilde{X}_i\widehat{p}(X_i)=\mathrm{OPT}_{\mathrm{LP}}. n 1 i = 1 ∑ n θ T X i p θ , q ( X i ) ≥ n 1 i = 1 ∑ n θ T X i p ( X i ) = OPT LP .
Since q ^ ∈ Q θ , L \widehat{q}\in\mathcal{Q}_{\theta,L} q ∈ Q θ , L , the left-hand side is bounded above by O P T θ , n \mathrm{OPT}_{\theta,n} OPT θ , n . Hence
O P T θ , n ≥ O P T L P . (29) \mathrm{OPT}_{\theta,n}\geq\mathrm{OPT}_{\mathrm{LP}}. \tag*{(29)} OPT θ , n ≥ OPT LP . (29)
Combining (26) and (29) gives
O P T θ , n = O P T L P . \mathrm{OPT}_{\theta,n}=\mathrm{OPT}_{\mathrm{LP}}. OPT θ , n = OPT LP .
Moreover, the preceding display shows that the feasible trace q ^ = p ^ ∣ H θ \widehat{q}=\widehat{p}|_{\mathcal{H}_\theta} q = p ∣ H θ attains this common value. Therefore, we have that
q ^ ∈ arg max q ∈ Q θ , L 1 n ∑ i = 1 n θ T X ~ i p θ , q ( X i ) . \widehat{q}\in\underset{q\in\mathcal{Q}_{\theta,L}}{\arg\max}\ \frac{1}{n}\sum_{i=1}^{n}\theta^{\mathsf{T}}\widetilde{X}_ip_{\theta,q}(X_i). q ∈ q ∈ Q θ , L arg max n 1 i = 1 ∑ n θ T X i p θ , q ( X i ) .
This completes the proof.
Proof of Theorem 4
Proof of Theorem 4. Note that the fairness guarantee follows directly from the canonical representation in Theorem 3 , while regret control is a direct consequence of Theorems 15 and 58 .
Enlarged parameter space
Lemma 14. For constants A 0 A_0 A 0 , B 0 B_0 B 0 , C 0 C_0 C 0 , c 0 c_0 c 0 , ρ 0 \rho_0 ρ 0 in Assumption 4, denote A ~ 0 = 3 A 0 / 2 \widetilde{A}_0 = 3A_0/2 A 0 = 3 A 0 /2 , B ~ 0 = 3 B 0 / 2 \widetilde{B}_0 = 3B_0/2 B 0 = 3 B 0 /2 , c ~ 0 = c 0 / 2 \widetilde{c}_0 = c_0/2 c 0 = c 0 /2 , C ~ 0 = 3 C 0 / 2 \widetilde{C}_0 = 3C_0/2 C 0 = 3 C 0 /2 , and ρ ~ 0 = ρ 0 / 2 \widetilde{\rho}_0 = \rho_0/2 ρ 0 = ρ 0 /2 . Define the enlarged parameter space Θ ~ = ⋃ j = 1 d Θ ~ j \widetilde{\Theta} = \bigcup_{j=1}^{d} \widetilde{\Theta}_j Θ = ⋃ j = 1 d Θ j , where, for each j ∈ { 1 , … , d } j \in\{1,\ldots,d\} j ∈ { 1 , … , d } ,
Θ ~ j = { ( a , γ ⊤ ) ⊤ ∈ R × R d : ∣ a ∣ ≤ A ~ 0 , ∥ γ ∥ 2 ≤ B ~ 0 , c ~ 0 ≤ γ j ≤ C ~ 0 , \widetilde{\Theta}_j = \{(a,\gamma^\top)^\top\in\mathbb{R} \times\mathbb{R}^d : |a| \le\widetilde{A}_0,\ \|\gamma\|_2 \le\widetilde{B}_0,\ \widetilde{c}_0 \le\gamma_j \le\widetilde{C}_0, Θ j = {( a , γ ⊤ ) ⊤ ∈ R × R d : ∣ a ∣ ≤ A 0 , ∥ γ ∥ 2 ≤ B 0 , c 0 ≤ γ j ≤ C 0 , and there exists u θ ∈ [ 0 , 1 ] d − 1 such that − ( 1 − ρ ~ 0 ) γ j ≤ a + γ − j ⊤ u θ ≤ − ρ ~ 0 γ j } . \text{and there exists }u_\theta\in[0,1]^{d-1}\text{ such that }-(1-\widetilde{\rho}_0)\gamma_j \le a+\gamma_{-j}^{\top}u_\theta\le-\widetilde{\rho}_0\gamma_j\}. and there exists u θ ∈ [ 0 , 1 ] d − 1 such that − ( 1 − ρ 0 ) γ j ≤ a + γ − j ⊤ u θ ≤ − ρ 0 γ j } . Then it holds that θ ∗ ∈ Θ ~ \theta^* \in\widetilde{\Theta} θ ∗ ∈ Θ . Moreover, there exists a constant r l o c > 0 r_{\mathrm{loc}}>0 r loc > 0 , depending only on θ ∗ \theta^* θ ∗ and the constants in Assumption 4, such that { θ : ∥ θ − θ ∗ ∥ 2 ≤ 2 r l o c } ⊆ Θ ~ j ∗ ⊆ Θ ~ \{\theta:\|\theta-\theta^*\|_2\le2r_{\mathrm{loc}}\}\subseteq\widetilde{\Theta}_{j^*}\subseteq\widetilde{\Theta} { θ : ∥ θ − θ ∗ ∥ 2 ≤ 2 r loc } ⊆ Θ j ∗ ⊆ Θ .
Proof. Write θ ∗ = ( a ∗ , γ ∗ ⊤ ) ⊤ \theta^*=(a^*,\gamma^{*\top})^\top θ ∗ = ( a ∗ , γ ∗ ⊤ ) ⊤ . By Assumption 4, there exist an index j ∗ ∈ { 1 , … , d } j^*\in\{1,\ldots,d\} j ∗ ∈ { 1 , … , d } and a point u ∗ ∈ [ 0 , 1 ] d − 1 u^*\in[0,1]^{d-1} u ∗ ∈ [ 0 , 1 ] d − 1 such that ∣ a ∗ ∣ ≤ A 0 |a^*|\le A_0 ∣ a ∗ ∣ ≤ A 0 , ∥ γ ∗ ∥ 2 ≤ B 0 \|\gamma^*\|_2\le B_0 ∥ γ ∗ ∥ 2 ≤ B 0 , c 0 ≤ γ j ∗ ∗ ≤ C 0 c_0\le\gamma^*_{j^*}\le C_0 c 0 ≤ γ j ∗ ∗ ≤ C 0 , and
− ( 1 − ρ 0 ) γ j ∗ ∗ ≤ a ∗ + γ − j ∗ ∗ ⊤ u ∗ ≤ − ρ 0 γ j ∗ ∗ . -(1-\rho_0)\gamma^*_{j^*}\le a^*+\gamma^{*\top}_{-j^*}u^*\le-\rho_0\gamma^*_{j^*}. − ( 1 − ρ 0 ) γ j ∗ ∗ ≤ a ∗ + γ − j ∗ ∗ ⊤ u ∗ ≤ − ρ 0 γ j ∗ ∗ . Since A ~ 0 = 3 A 0 / 2 \widetilde{A}_0=3A_0/2 A 0 = 3 A 0 /2 , B ~ 0 = 3 B 0 / 2 \widetilde{B}_0=3B_0/2 B 0 = 3 B 0 /2 , c ~ 0 = c 0 / 2 \widetilde{c}_0=c_0/2 c 0 = c 0 /2 , C ~ 0 = 3 C 0 / 2 \widetilde{C}_0=3C_0/2 C 0 = 3 C 0 /2 , and ρ ~ 0 = ρ 0 / 2 \widetilde{\rho}_0=\rho_0/2 ρ 0 = ρ 0 /2 , the bounds ∣ a ∗ ∣ ≤ A ~ 0 |a^*|\le\widetilde{A}_0 ∣ a ∗ ∣ ≤ A 0 , ∥ γ ∗ ∥ 2 ≤ B ~ 0 \|\gamma^*\|_2\le\widetilde{B}_0 ∥ γ ∗ ∥ 2 ≤ B 0 , and c ~ 0 ≤ γ j ∗ ∗ ≤ C ~ 0 \widetilde{c}_0\le\gamma^*_{j^*}\le\widetilde{C}_0 c 0 ≤ γ j ∗ ∗ ≤ C 0 trivially holds. Moreover, since γ j ∗ ∗ ≥ c 0 > 0 \gamma^*_{j^*}\ge c_0>0 γ j ∗ ∗ ≥ c 0 > 0 and ρ ~ 0 < ρ 0 \widetilde{\rho}_0<\rho_0 ρ 0 < ρ 0 , we also have
− ( 1 − ρ ~ 0 ) γ j ∗ ∗ ≤ a ∗ + γ − j ∗ ∗ ⊤ u ∗ ≤ − ρ ~ 0 γ j ∗ ∗ . -(1-\widetilde{\rho}_0)\gamma^*_{j^*}\le a^*+\gamma^{*\top}_{-j^*}u^*\le-\widetilde{\rho}_0\gamma^*_{j^*}. − ( 1 − ρ 0 ) γ j ∗ ∗ ≤ a ∗ + γ − j ∗ ∗ ⊤ u ∗ ≤ − ρ 0 γ j ∗ ∗ . Thus θ ∗ ∈ Θ ~ j ∗ ⊆ Θ ~ \theta^*\in\widetilde{\Theta}_{j^*}\subseteq\widetilde{\Theta} θ ∗ ∈ Θ j ∗ ⊆ Θ .
It remains to show that a small neighbourhood of θ ∗ \theta^* θ ∗ is contained in Θ ~ j ∗ \widetilde{\Theta}_{j^*} Θ j ∗ . Note that the functions θ ↦ ∣ a ∣ \theta\mapsto|a| θ ↦ ∣ a ∣ , θ ↦ ∥ γ ∥ 2 \theta\mapsto\|\gamma\|_2 θ ↦ ∥ γ ∥ 2 , θ ↦ γ j ∗ \theta\mapsto\gamma_{j^*} θ ↦ γ j ∗ , and θ ↦ a + γ − j ∗ ⊤ u ∗ \theta\mapsto a+\gamma_{-j^*}^{\top}u^* θ ↦ a + γ − j ∗ ⊤ u ∗ are continuous. Since all the inequalities defining Θ ~ j ∗ \widetilde{\Theta}_{j^*} Θ j ∗ hold with slack at θ ∗ \theta^* θ ∗ , there exists r l o c > 0 r_{\mathrm{loc}}>0 r loc > 0 such that, whenever ∥ θ − θ ∗ ∥ 2 ≤ 2 r l o c \|\theta-\theta^*\|_2\le2r_{\mathrm{loc}} ∥ θ − θ ∗ ∥ 2 ≤ 2 r loc , the same inequalities remain valid with j = j ∗ j=j^* j = j ∗ . Therefore, the lemma holds. □ \square □
Throughout the rest of the proof, we work over the enlarged parameter space Θ ~ \widetilde{\Theta} Θ instead of Θ \Theta Θ . This enlargement ensures that the estimator θ ^ \widehat{\theta} θ belongs to the parameter space under consideration, thereby avoiding the need to project it back onto Θ \Theta Θ .
For θ = ( a , γ ⊤ ) ⊤ ∈ Θ ~ j \theta=(a,\gamma^\top)^\top\in\widetilde{\Theta}_j θ = ( a , γ ⊤ ) ⊤ ∈ Θ j , write s θ ( x ) = x ~ ⊤ θ = a + γ ⊤ x s_\theta(x)=\widetilde{x}^\top\theta=a+\gamma^\top x s θ ( x ) = x ⊤ θ = a + γ ⊤ x and x = ( u ( j ) , z j ) x=(u^{(j)},z_j) x = ( u ( j ) , z j ) , where z j = x j z_j=x_j z j = x j and u ( j ) u^{(j)} u ( j ) is obtained from x x x by deleting its j j j -th coordinate. The corresponding chart representation is
ϕ θ , j ( u ) = − a + γ − j ⊤ u γ j . \phi_{\theta,j}(u)=-\frac{a+\gamma_{-j}^{\top}u}{\gamma_j}. ϕ θ , j ( u ) = − γ j a + γ − j ⊤ u . Let { F t } t ≥ 0 \{\mathcal{F}_t\}_{t\ge0} { F t } t ≥ 0 be the filtration generated by the history up to the end of round t t t , namely
F t = σ { X i , A i , Y i : 1 ≤ i ≤ t } , F 0 = { ∅ , Ω } . \mathcal{F}_t=\sigma\{X_i,A_i,Y_i:1\le i\le t\},\qquad\mathcal{F}_0=\{\varnothing,\Omega\}. F t = σ { X i , A i , Y i : 1 ≤ i ≤ t } , F 0 = { ∅ , Ω } . For each t > 2 T 0 t>2T_0 t > 2 T 0 , let τ ( t ) \tau(t) τ ( t ) denote the most recent lazy update time before the action at round t t t is chosen:
τ ( t ) = max { s ≤ t : Algorithm 1 recomputes ( θ ^ s , q ^ s ) at the beginning of round s } . \tau(t)=\max\left\{s\le t:\text{ Algorithm 1 recomputes }(\widehat{\theta}_s,\widehat{q}_s)\text{ at the beginning of round }s\right\}. τ ( t ) = max { s ≤ t : Algorithm 1 recomputes ( θ s , q s ) at the beginning of round s } . Upper bound on regret For any F t − 1 \mathcal{F}_{t-1} F t − 1 -measurable policy p p p , define
R t ( p ) = E { X ~ ⊤ β 1 ∗ p ( X ) + X ~ ⊤ β 2 ∗ ( 1 − p ( X ) ) ∣ F t − 1 } = E { X ~ ⊤ β 2 ∗ } + E { X ~ ⊤ θ ∗ p ( X ) ∣ F t − 1 } , R_t(p)=\mathbb{E}\{\widetilde{X}^{\top}\beta_1^*p(X)+\widetilde{X}^{\top}\beta_2^*(1-p(X))\mid\mathcal{F}_{t-1}\}
=\mathbb{E}\{\widetilde{X}^{\top}\beta_2^*\}+\mathbb{E}\{\widetilde{X}^{\top}\theta^*p(X)\mid\mathcal{F}_{t-1}\}, R t ( p ) = E { X ⊤ β 1 ∗ p ( X ) + X ⊤ β 2 ∗ ( 1 − p ( X )) ∣ F t − 1 } = E { X ⊤ β 2 ∗ } + E { X ⊤ θ ∗ p ( X ) ∣ F t − 1 } , where X X X is an independent copy of the context and X ~ = ( 1 , X ⊤ ) ⊤ \widetilde{X}=(1,X^\top)^\top X = ( 1 , X ⊤ ) ⊤ .
Let
q ∗ ∈ arg max q ∈ Q θ ∗ , L R ( p θ ∗ , q ) , p ∗ = p θ ∗ , q ∗ . q^* \in\arg\max_{q \in\mathcal{Q}_{\theta^*,L}} R(p_{\theta^*,q}), \qquad p^* = p_{\theta^*,q^*}. q ∗ ∈ arg q ∈ Q θ ∗ , L max R ( p θ ∗ , q ) , p ∗ = p θ ∗ , q ∗ .
By Theorem 3 , it holds that sup p ∈ P L , ν R ( p ) = R ( p θ ∗ , q ∗ ) \sup_{p \in\mathcal{P}_{L,\nu}} R(p) = R(p_{\theta^*,q^*}) sup p ∈ P L , ν R ( p ) = R ( p θ ∗ , q ∗ ) .
For t > 2 T o t > 2T_o t > 2 T o , let τ ( t ) \tau(t) τ ( t ) be the most recent lazy update time before the action at round t t t is chosen. Denote the cumulative regret
R T = ∑ t = 1 2 T o { R t ( p ∗ ) − R t ( p ^ t ) } + ∑ t = 2 T o + 1 T { R t ( p ∗ ) − R t ( p ^ τ ( t ) ) } . \mathfrak{R}_T = \sum_{t=1}^{2T_o} \{R_t(p^*) - R_t(\hat{p}_t)\} + \sum_{t=2T_o+1}^{T} \{R_t(p^*) - R_t(\hat{p}_{\tau(t)})\}. R T = t = 1 ∑ 2 T o { R t ( p ∗ ) − R t ( p ^ t )} + t = 2 T o + 1 ∑ T { R t ( p ∗ ) − R t ( p ^ τ ( t ) )} .
Theorem 15 (Regret upper bound for the lazy adaptive boundary greedy algorithm). Suppose Assumptions 1-4 hold. Take λ = C λ \lambda= C_\lambda λ = C λ and T o = C 1 ⌈ T log ( T / η ) ⌉ T_o = C_1\lceil\sqrt{T\log(T/\eta)}\rceil T o = C 1 ⌈ T log ( T / η ) ⌉ , for any η ∈ ( 0 , 1 / 2 ) \eta\in(0,1/2) η ∈ ( 0 , 1/2 ) . Assume that T ≥ C 0 log ( T / η ) T \ge C_0\log(T/\eta) T ≥ C 0 log ( T / η ) , where C 0 > 0 C_0 > 0 C 0 > 0 is a sufficiently large constant depending only on the constants listed below. Then, with probability at least 1 − η 1-\eta 1 − η , there exists constants C 1 , C 2 > 0 C_1,C_2 > 0 C 1 , C 2 > 0 , depending only on d d d , L L L , c g c_g c g , C g C_g C g , σ \sigma σ , C β C_\beta C β , λ \lambda λ , ν \nu ν , the constants in Assumption 1 and the constants in Assumption 4, such that
R T ≤ C 1 T log ( T / η ) + C 2 { T log ( e T 2 / η ) , d = 2 , T log 3 / 2 ( e T 2 / η ) , d = 3 , T 1 − 1 d − 1 log ( e T 2 / η ) , d ≥ 4. \mathfrak{R}_T \le C_1\sqrt{T}\log(T/\eta) + C_2
\begin{cases}
\sqrt{T}\log(eT^2/\eta), & d=2,\\
\sqrt{T}\log^{3/2}(eT^2/\eta), & d=3,\\
T^{1-\frac{1}{d-1}}\log(eT^2/\eta), & d\ge4.
\end{cases} R T ≤ C 1 T log ( T / η ) + C 2 ⎩ ⎨ ⎧ T log ( e T 2 / η ) , T log 3/2 ( e T 2 / η ) , T 1 − d − 1 1 log ( e T 2 / η ) , d = 2 , d = 3 , d ≥ 4.
Proof. Let S T S_T S T denote the set of lazy update times up to time T T T . Since the algorithm updates only when t ≥ 2 s t \ge2s t ≥ 2 s , the update times grow geometrically. Hence, for every α > 0 \alpha> 0 α > 0 , we have that ∑ s ∈ S T s α ≤ C α T α \sum_{s \in S_T} s^\alpha\le C_\alpha T^\alpha ∑ s ∈ S T s α ≤ C α T α , where C α = ( 1 − 2 − α ) − 1 C_\alpha= (1-2^{-\alpha})^{-1} C α = ( 1 − 2 − α ) − 1 .
Write ℓ T = log ( T / η ) \ell_T = \log(T/\eta) ℓ T = log ( T / η ) . Since ℓ T ≥ log 2 \ell_T \ge\log2 ℓ T ≥ log 2 , there is a constant K > 0 K > 0 K > 0 , depending only on the fixed model parameters, such that
C T log { 2 ( d + 1 ) T / η } ∨ C l o c log ( 2 T / η ) r l o c 2 ≤ K ℓ T , C_T\log\{2(d+1)T/\eta\} \vee\frac{C_{\mathrm{loc}}\log(2T/\eta)}{r_{\mathrm{loc}}^2} \le K\ell_T, C T log { 2 ( d + 1 ) T / η } ∨ r loc 2 C loc log ( 2 T / η ) ≤ K ℓ T ,
where r l o c r_{\mathrm{loc}} r loc is the constant defined in Lemma 14 . Choose C 0 ≥ max { 4 , 16 C 1 2 , ( K / C 1 ) 2 } C_0 \ge\max\{4,16C_1^2,(K/C_1)^2\} C 0 ≥ max { 4 , 16 C 1 2 , ( K / C 1 ) 2 } . The condition T ≥ C 0 ℓ T T \ge C_0\ell_T T ≥ C 0 ℓ T then gives
T o = C 1 ⌈ T ℓ T ⌉ ≥ C 1 C 0 ℓ T ≥ K ℓ T . T_o = C_1\lceil\sqrt{T\ell_T}\rceil\ge C_1\sqrt{C_0}\ell_T \ge K\ell_T. T o = C 1 ⌈ T ℓ T ⌉ ≥ C 1 C 0 ℓ T ≥ K ℓ T .
Thus the exploration lower bounds in Proposition 16 and Lemma 25 hold, including with confidence parameter η / 2 \eta/2 η /2 . Moreover, the same horizon condition ensures T ≥ 2 T \ge2 T ≥ 2 , so ⌈ T ℓ T ⌉ ≤ 2 T ℓ T \lceil\sqrt{T\ell_T}\rceil\le2\sqrt{T\ell_T} ⌈ T ℓ T ⌉ ≤ 2 T ℓ T and
2 T o ≤ 4 C 1 T ℓ T ≤ 4 C 1 C 0 T ≤ T . 2T_o \le4C_1\sqrt{T\ell_T} \le\frac{4C_1}{\sqrt{C_0}}T \le T. 2 T o ≤ 4 C 1 T ℓ T ≤ C 0 4 C 1 T ≤ T .
Hence the exploration phase fits within the time horizon. If 2 T o = T 2T_o = T 2 T o = T , the claimed regret bound follows directly from the bounded one-round regret and T o ≲ T ℓ T T_o \lesssim\sqrt{T\ell_T} T o ≲ T ℓ T . We therefore consider 2 T o < T 2T_o < T 2 T o < T below.
By Lemma 25 , with probability at least 1 − η / 2 1-\eta/2 1 − η /2 , for every s ∈ S T s \in S_T s ∈ S T , we have that θ ^ s ∈ Θ ~ j ∗ \hat{\theta}_s \in\widetilde{\Theta}_{j^*} θ ^ s ∈ Θ j ∗ and
∥ θ ^ s − θ ∗ ∥ 2 ≤ C 2 log ( T / η ) s − 1 . (30) \|\hat{\theta}_s-\theta^*\|_2 \le C_2\sqrt{\frac{\log(T/\eta)}{s-1}}. \tag*{(30)} ∥ θ ^ s − θ ∗ ∥ 2 ≤ C 2 s − 1 log ( T / η ) . (30)
By Lemma 23, together with a union bound over n = 1 , … , T n=1,\ldots,T n = 1 , … , T , with probability at least 1 − η / 2 1-\eta/2 1 − η /2 , for every n = 1 , … , T n=1,\ldots,T n = 1 , … , T , Δ n L ≤ C ρ n , d − 1 ( η / T ) \Delta_n^L \le C\rho_{n,d-1}(\eta/T) Δ n L ≤ C ρ n , d − 1 ( η / T ) , where Δ n L \Delta_n^L Δ n L and ρ n \rho_n ρ n are defined in Lemma 23. Equivalently, for every s ∈ S T s \in S_T s ∈ S T , we have that
Δ s − 1 L ≤ C 3 { ( s − 1 ) − 1 / 2 log ( e T 2 / η ) , d = 2 , ( s − 1 ) − 1 / 2 log 3 / 2 ( e T 2 / η ) , d = 3 , ( s − 1 ) − 1 / ( d − 1 ) log ( e T 2 / η ) , d ≥ 4. (31) \Delta_{s-1}^L \le C_3
\begin{cases}
(s-1)^{-1/2}\log(eT^2/\eta), & d=2,\\
(s-1)^{-1/2}\log^{3/2}(eT^2/\eta), & d=3,\\
(s-1)^{-1/(d-1)}\log(eT^2/\eta), & d\ge4.
\end{cases} \tag*{(31)} Δ s − 1 L ≤ C 3 ⎩ ⎨ ⎧ ( s − 1 ) − 1/2 log ( e T 2 / η ) , ( s − 1 ) − 1/2 log 3/2 ( e T 2 / η ) , ( s − 1 ) − 1/ ( d − 1 ) log ( e T 2 / η ) , d = 2 , d = 3 , d ≥ 4. (31)
By a union bound argument, the rest of the proof is constructed conditioning on the intersection of the high-probability events in (30) and (31) . For any post-initialisation round t > 2 T o t > 2T_o t > 2 T o , write s = τ ( t ) s = \tau(t) s = τ ( t ) for the lazy update time at which the policy used at round t t t was computed. By Proposition 16 , we have that
R s ( p ∗ ) − R s ( p ^ s ) ≤ C θ , L ∥ θ ^ s − θ ∗ ∥ 2 + 2 Δ s − 1 L . R_s(p^*) - R_s(\widehat{p}_s) \le C_{\theta,L}\lVert\widehat{\theta}_s-\theta^*\rVert_2 + 2\Delta^L_{s-1}. R s ( p ∗ ) − R s ( p s ) ≤ C θ , L ∥ θ s − θ ∗ ∥ 2 + 2 Δ s − 1 L .
Therefore,
R s ( p ∗ ) − R s ( p ^ s ) ≤ C 2 log ( T / η ) s − 1 + 2 Δ s − 1 L . (32) R_s(p^*) - R_s(\widehat{p}_s) \le C_2\sqrt{\frac{\log(T/\eta)}{s-1}} + 2\Delta^L_{s-1}. \tag*{(32)} R s ( p ∗ ) − R s ( p s ) ≤ C 2 s − 1 log ( T / η ) + 2 Δ s − 1 L . (32)
Note that the regret from the first 2 T o 2T_o 2 T o forced-exploration rounds is bounded by C 4 T o C_4T_o C 4 T o under Assumptions 1 and 3. Moreover, by the lazy update mechanism, if s ∈ S T s \in S_T s ∈ S T is an update time, then the policy p ^ s \widehat{p}_s p s is kept fixed until the next update. Since the next update occurs no later than time 2 s 2s 2 s , we have that # { t ≤ T : τ ( t ) = s } ≤ s \#\{t \le T : \tau(t) = s\} \le s # { t ≤ T : τ ( t ) = s } ≤ s . Therefore, applying the one-step bound to every round with τ ( t ) = s \tau(t) = s τ ( t ) = s in (32) gives
R T ≤ C 4 T o + ∑ s ∈ S T # { t ≤ T : τ ( t ) = s } { C 2 log ( T / η ) s − 1 + 2 Δ s − 1 L } ≤ C 4 T o + C 2 ∑ s ∈ S T s log ( T / η ) s − 1 + 2 ∑ s ∈ S T s Δ s − 1 L . \begin{aligned}
\mathfrak{R}_T \le C_4T_o+\sum_{s\in S_T}\#\{t\le T:\tau(t)=s\}\left\{C_2\sqrt{\frac{\log(T/\eta)}{s-1}}+2\Delta^L_{s-1}\right\} \\
&\le C_4T_o+C_2\sum_{s\in S_T}s\sqrt{\frac{\log(T/\eta)}{s-1}}+2\sum_{s\in S_T}s\Delta^L_{s-1}.
\end{aligned} R T ≤ C 4 T o + s ∈ S T ∑ # { t ≤ T : τ ( t ) = s } { C 2 s − 1 log ( T / η ) + 2 Δ s − 1 L } ≤ C 4 T o + C 2 s ∈ S T ∑ s s − 1 log ( T / η ) + 2 s ∈ S T ∑ s Δ s − 1 L .
Since s − 1 ≍ s s-1\asymp s s − 1 ≍ s for s ∈ S T s\in S_T s ∈ S T , the estimation term satisfies
∑ s ∈ S T s log ( T / η ) s − 1 ≤ C 5 log ( T / η ) ∑ s ∈ S T s 1 / 2 ≤ C 6 T log ( T / η ) . \sum_{s\in S_T}s\sqrt{\frac{\log(T/\eta)}{s-1}}\le C_5\sqrt{\log(T/\eta)}\sum_{s\in S_T}s^{1/2}\le C_6\sqrt{T\log(T/\eta)}. s ∈ S T ∑ s s − 1 log ( T / η ) ≤ C 5 log ( T / η ) s ∈ S T ∑ s 1/2 ≤ C 6 T log ( T / η ) .
It remains to control the empirical-process term. If d = 2 d=2 d = 2 , then m = d − 1 = 1 m=d-1=1 m = d − 1 = 1 , so
∑ s ∈ S T s Δ s − 1 L ≤ C 3 log ( e T 2 / η ) ∑ s ∈ S T s 1 / 2 ≤ C 7 T log ( e T 2 / η ) . \sum_{s\in S_T}s\Delta^L_{s-1}\le C_3\log(eT^2/\eta)\sum_{s\in S_T}s^{1/2}\le C_7\sqrt{T}\log(eT^2/\eta). s ∈ S T ∑ s Δ s − 1 L ≤ C 3 log ( e T 2 / η ) s ∈ S T ∑ s 1/2 ≤ C 7 T log ( e T 2 / η ) .
If d = 3 d=3 d = 3 , then m = d − 1 = 2 m=d-1=2 m = d − 1 = 2 , so
∑ s ∈ S T s Δ s − 1 L ≤ C 3 log 3 / 2 ( e T 2 / η ) ∑ s ∈ S T s 1 / 2 ≤ C 8 T log 3 / 2 ( e T 2 / η ) . \sum_{s\in S_T}s\Delta^L_{s-1}\le C_3\log^{3/2}(eT^2/\eta)\sum_{s\in S_T}s^{1/2}\le C_8\sqrt{T}\log^{3/2}(eT^2/\eta). s ∈ S T ∑ s Δ s − 1 L ≤ C 3 log 3/2 ( e T 2 / η ) s ∈ S T ∑ s 1/2 ≤ C 8 T log 3/2 ( e T 2 / η ) .
If d ≥ 4 d\ge4 d ≥ 4 , then m = d − 1 ≥ 3 m=d-1\ge3 m = d − 1 ≥ 3 , so
∑ s ∈ S T s Δ s − 1 L ≤ C 3 log ( e T 2 / η ) ∑ s ∈ S T s 1 − 1 d − 1 ≤ C 9 T 1 − 1 d − 1 log ( e T 2 / η ) . \sum_{s\in S_T}s\Delta^L_{s-1}\le C_3\log(eT^2/\eta)\sum_{s\in S_T}s^{1-\frac{1}{d-1}}\le C_9T^{1-\frac{1}{d-1}}\log(eT^2/\eta). s ∈ S T ∑ s Δ s − 1 L ≤ C 3 log ( e T 2 / η ) s ∈ S T ∑ s 1 − d − 1 1 ≤ C 9 T 1 − d − 1 1 log ( e T 2 / η ) .
Combining the preceding bounds yields
R T ≲ T o + T log ( T / η ) + { T log ( e T 2 / η ) , d = 2 , T log 3 / 2 ( e T 2 / η ) , d = 3 , T 1 − 1 d − 1 log ( e T 2 / η ) , d ≥ 4. \mathfrak{R}_T\lesssim T_o+\sqrt{T\log(T/\eta)}+
\begin{cases}
\sqrt{T}\log(eT^2/\eta), & d=2,\\
\sqrt{T}\log^{3/2}(eT^2/\eta), & d=3,\\
T^{1-\frac{1}{d-1}}\log(eT^2/\eta), & d\ge4.
\end{cases} R T ≲ T o + T log ( T / η ) + ⎩ ⎨ ⎧ T log ( e T 2 / η ) , T log 3/2 ( e T 2 / η ) , T 1 − d − 1 1 log ( e T 2 / η ) , d = 2 , d = 3 , d ≥ 4.
This completes the proof.
Proposition 16 (Regret decomposition). Suppose Assumptions 1 1 1 -4 hold and for any η ∈ ( 0 , 1 / 2 ) \eta\in(0,1/2) η ∈ ( 0 , 1/2 ) ,
T o ≥ C T log { ( d + 1 ) T / η } ∨ C l o c log ( T / η ) r l o c 2 . T_o \ge C_T \log\{(d+1)T/\eta\}\vee\frac{C_{\mathrm{loc}}\log(T/\eta)}{r_{\mathrm{loc}}^2}. T o ≥ C T log {( d + 1 ) T / η } ∨ r loc 2 C loc log ( T / η ) . Then, for every lazy update time s ∈ S T s\in\mathcal{S}_T s ∈ S T , we have with probability at least 1 − η 1-\eta 1 − η that
R s ( p ∗ ) − R s ( p ^ s ) ≤ C θ , L ∥ θ ^ s − θ ∗ ∥ 2 + 2 Δ s − 1 L , R_s(p^*)-R_s(\widehat{p}_s)\le C_{\theta,L}\|\widehat{\theta}_s-\theta^*\|_2+2\Delta^L_{s-1}, R s ( p ∗ ) − R s ( p s ) ≤ C θ , L ∥ θ s − θ ∗ ∥ 2 + 2 Δ s − 1 L , where Δ s − 1 L \Delta^L_{s-1} Δ s − 1 L is defined in Lemma 23, C θ , L C_{\theta,L} C θ , L is a constant depending only on L , d , ν L,d,\nu L , d , ν and the constants in Assumptions 1 1 1 and 4 4 4 .
Proof. Fix t > 2 T o t>2T_o t > 2 T o and write s = τ ( t ) s=\tau(t) s = τ ( t ) . Let δ s = ∥ θ ^ s − θ ∗ ∥ 2 \delta_s=\|\widehat{\theta}_s-\theta^*\|_2 δ s = ∥ θ s − θ ∗ ∥ 2 . Conditional on F s − 1 \mathcal{F}_{s-1} F s − 1 , the quantities θ ^ s \widehat{\theta}_s θ s , q ^ s \widehat{q}_s q s and p ^ s \widehat{p}_s p s are fixed.
Define
E = { θ ^ s ∈ Θ ~ j ∗ for every lazy update time s ≤ T } . \mathcal{E}=\left\{\widehat{\theta}_s\in\widetilde{\Theta}_{j^*}\text{ for every lazy update time }s\le T\right\}. E = { θ s ∈ Θ j ∗ for every lazy update time s ≤ T } .
Then by Lemma 25, it holds that P ( E ) ≥ 1 − η \mathbb{P}(\mathcal{E})\ge1-\eta P ( E ) ≥ 1 − η . The rest of the proof is constructed conditioning on E \mathcal{E} E happening.
On E \mathcal{E} E , by construction of the algorithm, it holds that θ ^ s ∈ Θ ~ j ∗ \widehat{\theta}_s\in\widetilde{\Theta}_{j^*} θ s ∈ Θ j ∗ , q ^ s ∈ Q θ ^ s , L \widehat{q}_s\in Q_{\widehat{\theta}_s,L} q s ∈ Q θ s , L and p ^ s = p θ ^ s , q ^ s \widehat{p}_s=p_{\widehat{\theta}_s,\widehat{q}_s} p s = p θ s , q s .
Under the same construction as Lemma 17, for q ∗ ∈ arg max q ∈ Q θ ∗ , L R ( p θ ∗ , q ) q^*\in\arg\max_{q\in Q_{\theta^*,L}}R(p_{\theta^*,q}) q ∗ ∈ arg max q ∈ Q θ ∗ , L R ( p θ ∗ , q ) and p ∗ = p θ ∗ , q ∗ p^*=p_{\theta^*,q^*} p ∗ = p θ ∗ , q ∗ , define the global McShane extension of q ∗ q^* q ∗ by
q ‾ ∗ ( x ) = [ inf h ∈ H θ ∗ { q ∗ ( h ) + L ∥ x − h ∥ ν } ] [ 0 , 1 ] , x ∈ [ 0 , 1 ] d . \overline{q}^*(x)=\left[\inf_{h\in\mathcal{H}_{\theta^*}}\{q^*(h)+L\|x-h\|_\nu\}\right]_{[0,1]},\qquad x\in[0,1]^d. q ∗ ( x ) = [ h ∈ H θ ∗ inf { q ∗ ( h ) + L ∥ x − h ∥ ν } ] [ 0 , 1 ] , x ∈ [ 0 , 1 ] d .
Define the trace on the estimated boundary by
q s † ( h ) = q ‾ ∗ ( h ) , h ∈ H θ ^ s , q_s^\dagger(h)=\overline{q}^*(h),\qquad h\in\mathcal{H}_{\widehat{\theta}_s}, q s † ( h ) = q ∗ ( h ) , h ∈ H θ s ,
and set p s † = p θ ^ s , q s † p_s^\dagger=p_{\widehat{\theta}_s,q_s^\dagger} p s † = p θ s , q s † .
Decomposition of regret. For any update time s ∈ S T s\in\mathcal{S}_T s ∈ S T , the policy played is p ^ s \widehat{p}_s p s , and we can upper bounded the regret by
R s ( p ∗ ) − R s ( p ^ s ) = E [ X ~ T θ ∗ { p ∗ ( X ) − p ^ s ( X ) } ∣ F s − 1 ] = E [ X ~ T θ ∗ { p ∗ ( X ) − p s † ( X ) } ∣ F s − 1 ] + E [ X ~ T θ ∗ { p s † ( X ) − p ^ s ( X ) } ∣ F s − 1 ] , \begin{aligned}
R_s(p^*)-R_s(\widehat{p}_s)=\mathbb{E}\left[\widetilde{X}^{\mathsf T}\theta^*\{p^*(X)-\widehat{p}_s(X)\}\mid\mathcal{F}_{s-1}\right] \\
&=\mathbb{E}\left[\widetilde{X}^{\mathsf T}\theta^*\{p^*(X)-p_s^\dagger(X)\}\mid\mathcal{F}_{s-1}\right]+\mathbb{E}\left[\widetilde{X}^{\mathsf T}\theta^*\{p_s^\dagger(X)-\widehat{p}_s(X)\}\mid\mathcal{F}_{s-1}\right],
\end{aligned} R s ( p ∗ ) − R s ( p s ) = E [ X T θ ∗ { p ∗ ( X ) − p s ( X )} ∣ F s − 1 ] = E [ X T θ ∗ { p ∗ ( X ) − p s † ( X )} ∣ F s − 1 ] + E [ X T θ ∗ { p s † ( X ) − p s ( X )} ∣ F s − 1 ] ,
where the first term is further bounded by
E [ X ~ T θ ∗ { p ∗ ( X ) − p s † ( X ) } ∣ F s − 1 ] ≤ C t r , L δ s \mathbb{E}\left[\widetilde{X}^{\mathsf T}\theta^*\{p^*(X)-p_s^\dagger(X)\}\mid\mathcal{F}_{s-1}\right]\le C_{\mathrm{tr},L}\delta_s E [ X T θ ∗ { p ∗ ( X ) − p s † ( X )} ∣ F s − 1 ] ≤ C tr , L δ s
as shown in Lemma 17.
For the second term, by adding and subtracting the empirical averages, we have that
E [ X ~ T θ ∗ { p s † ( X ) − p ^ s ( X ) } ∣ F s − 1 ] = [ E [ X ~ T θ ∗ p s † ( X ) ∣ F s − 1 ] − 1 s − 1 ∑ i = 1 s − 1 X ~ i T θ ∗ p s † ( X i ) ] + 1 s − 1 ∑ i = 1 s − 1 X ~ i T θ ∗ { p s † ( X i ) − p ^ s ( X i ) } + [ 1 s − 1 ∑ i = 1 s − 1 X ~ i T θ ∗ p ^ s ( X i ) − E { X ~ T θ ∗ p ^ s ( X ) ∣ F s − 1 } ] . (33) \begin{aligned}
\mathbb{E}\left[\widetilde{X}^{\mathsf T}\theta^*\{p_s^\dagger(X)-\widehat{p}_s(X)\}\mid\mathcal{F}_{s-1}\right] \\
&=\left[\mathbb{E}\left[\widetilde{X}^{\mathsf T}\theta^*p_s^\dagger(X)\mid\mathcal{F}_{s-1}\right]-\frac{1}{s-1}\sum_{i=1}^{s-1}\widetilde{X}_i^{\mathsf T}\theta^*p_s^\dagger(X_i)\right]
+\frac{1}{s-1}\sum_{i=1}^{s-1}\widetilde{X}_i^{\mathsf T}\theta^*\{p_s^\dagger(X_i)-\widehat{p}_s(X_i)\} \\
&+\left[\frac{1}{s-1}\sum_{i=1}^{s-1}\widetilde{X}_i^{\mathsf{T}}\theta^*\widehat{p}_s(X_i)-\mathbb{E}\left\{\widetilde{X}^{\mathsf{T}}\theta^*\widehat{p}_s(X)\mid\mathcal{F}_{s-1}\right\}\right]. \tag*{(33)}
\end{aligned} E [ X T θ ∗ { p s † ( X ) − p s ( X )} ∣ F s − 1 ] = [ E [ X T θ ∗ p s † ( X ) ∣ F s − 1 ] − s − 1 1 i = 1 ∑ s − 1 X i T θ ∗ p s † ( X i ) ] + s − 1 1 i = 1 ∑ s − 1 X i T θ ∗ { p s † ( X i ) − p s ( X i )} + [ s − 1 1 i = 1 ∑ s − 1 X i T θ ∗ p s ( X i ) − E { X T θ ∗ p s ( X ) ∣ F s − 1 } ] . (33)
Note that under E \mathcal{E} E , we have that θ ^ s ∈ Θ ~ \widehat{\theta}_s\in\widetilde{\Theta} θ s ∈ Θ . Moreover, since q s † ∈ Q θ ^ s , L q_s^\dagger\in\mathcal{Q}_{\widehat{\theta}_s,L} q s † ∈ Q θ s , L and q ^ s ∈ Q θ ^ s , L \widehat{q}_s\in\mathcal{Q}_{\widehat{\theta}_s,L} q s ∈ Q θ s , L , both p s † p_s^\dagger p s † and p ^ s \widehat{p}_s p s belong to the class F L \mathcal{F}_L F L defining Δ s − 1 L \Delta^L_{s-1} Δ s − 1 L in Lemma 23. Therefore the two bracketed terms are both bounded above by Δ s − 1 L \Delta^L_{s-1} Δ s − 1 L , and hence
E [ X ~ T θ ∗ { p s † ( X ) − p ^ s ( X ) } ∣ F s − 1 ] ≤ 2 Δ s − 1 L + 1 s − 1 ∑ i = 1 s − 1 X ~ i T θ ∗ { p s † ( X i ) − p ^ s ( X i ) } . \mathbb{E}\left[\widetilde{X}^{\mathsf{T}}\theta^*\{p_s^\dagger(X)-\widehat{p}_s(X)\}\mid\mathcal{F}_{s-1}\right]\leq2\Delta^L_{s-1}+\frac{1}{s-1}\sum_{i=1}^{s-1}\widetilde{X}_i^{\mathsf{T}}\theta^*\{p_s^\dagger(X_i)-\widehat{p}_s(X_i)\}. E [ X T θ ∗ { p s † ( X ) − p s ( X )} ∣ F s − 1 ] ≤ 2 Δ s − 1 L + s − 1 1 i = 1 ∑ s − 1 X i T θ ∗ { p s † ( X i ) − p s ( X i )} .
It remains to control the empirical term. Note that we can decompose it as
1 s − 1 ∑ i = 1 s − 1 X ~ i T θ ∗ { p s † ( X i ) − p ^ s ( X i ) } = 1 s − 1 ∑ i = 1 s − 1 X ~ i T θ ^ s { p s † ( X i ) − p ^ s ( X i ) } + 1 s − 1 ∑ i = 1 s − 1 X ~ i T ( θ ∗ − θ ^ s ) { p s † ( X i ) − p ^ s ( X i ) } . \begin{aligned}
\frac{1}{s-1}\sum_{i=1}^{s-1}\widetilde{X}_i^{\mathsf{T}}\theta^*\{p_s^\dagger(X_i)-\widehat{p}_s(X_i)\}=\frac{1}{s-1}\sum_{i=1}^{s-1}\widetilde{X}_i^{\mathsf{T}}\widehat{\theta}_s\{p_s^\dagger(X_i)-\widehat{p}_s(X_i)\} \\
&+\frac{1}{s-1}\sum_{i=1}^{s-1}\widetilde{X}_i^{\mathsf{T}}(\theta^*-\widehat{\theta}_s)\{p_s^\dagger(X_i)-\widehat{p}_s(X_i)\}.
\end{aligned} s − 1 1 i = 1 ∑ s − 1 X i T θ ∗ { p s † ( X i ) − p s ( X i )} = s − 1 1 i = 1 ∑ s − 1 X i T θ s { p s † ( X i ) − p s ( X i )} + s − 1 1 i = 1 ∑ s − 1 X i T ( θ ∗ − θ s ) { p s † ( X i ) − p s ( X i )} .
The first term is nonpositive by empirical optimality of q ^ s \widehat{q}_s q s . Indeed, q s † ∈ Q θ ^ s , L q_s^\dagger\in\mathcal{Q}_{\widehat{\theta}_s,L} q s † ∈ Q θ s , L is a feasible competitor in the empirical optimization over Q θ ^ s , L \mathcal{Q}_{\widehat{\theta}_s,L} Q θ s , L , so
1 s − 1 ∑ i = 1 s − 1 X ~ i T θ ^ s p ^ s ( X i ) ≥ 1 s − 1 ∑ i = 1 s − 1 X ~ i T θ ^ s p s † ( X i ) . \frac{1}{s-1}\sum_{i=1}^{s-1}\widetilde{X}_i^{\mathsf{T}}\widehat{\theta}_s\widehat{p}_s(X_i)\geq\frac{1}{s-1}\sum_{i=1}^{s-1}\widetilde{X}_i^{\mathsf{T}}\widehat{\theta}_sp_s^\dagger(X_i). s − 1 1 i = 1 ∑ s − 1 X i T θ s p s ( X i ) ≥ s − 1 1 i = 1 ∑ s − 1 X i T θ s p s † ( X i ) .
Thus, it holds that
1 s − 1 ∑ i = 1 s − 1 X ~ i T θ ^ s { p s † ( X i ) − p ^ s ( X i ) } ≤ 0. \frac{1}{s-1}\sum_{i=1}^{s-1}\widetilde{X}_i^{\mathsf{T}}\widehat{\theta}_s\{p_s^\dagger(X_i)-\widehat{p}_s(X_i)\}\leq0. s − 1 1 i = 1 ∑ s − 1 X i T θ s { p s † ( X i ) − p s ( X i )} ≤ 0.
For the second term, since 0 ≤ p s † , p ^ s ≤ 1 0\leq p_s^\dagger,\widehat{p}_s\leq1 0 ≤ p s † , p s ≤ 1 and X ~ i = ( 1 , X i T ) T \widetilde{X}_i=(1,X_i^{\mathsf{T}})^{\mathsf{T}} X i = ( 1 , X i T ) T with X i ∈ [ 0 , 1 ] d X_i\in[0,1]^d X i ∈ [ 0 , 1 ] d , we have that
1 s − 1 ∑ i = 1 s − 1 X ~ i T ( θ ∗ − θ ^ s ) { p s † ( X i ) − p ^ s ( X i ) } ≤ 1 s − 1 ∑ i = 1 s − 1 ∣ X ~ i T ( θ ∗ − θ ^ s ) ∣ ≤ d + 1 ∥ θ ^ s − θ ∗ ∥ 2 = d + 1 δ s . \begin{aligned}
\frac{1}{s-1}\sum_{i=1}^{s-1}\widetilde{X}_i^{\mathsf{T}}(\theta^*-\widehat{\theta}_s)\{p_s^\dagger(X_i)-\widehat{p}_s(X_i)\}\leq\frac{1}{s-1}\sum_{i=1}^{s-1}\left|\widetilde{X}_i^{\mathsf{T}}(\theta^*-\widehat{\theta}_s)\right| \\
&\leq\sqrt{d+1}\|\widehat{\theta}_s-\theta^*\|_2=\sqrt{d+1}\delta_s.
\end{aligned} s − 1 1 i = 1 ∑ s − 1 X i T ( θ ∗ − θ s ) { p s † ( X i ) − p s ( X i )} ≤ s − 1 1 i = 1 ∑ s − 1 X i T ( θ ∗ − θ s ) ≤ d + 1 ∥ θ s − θ ∗ ∥ 2 = d + 1 δ s .
Combining the preceding bounds thus gives
R s ( p ∗ ) − R s ( p ^ s ) ≤ C t r , L δ s + 2 Δ s − 1 L + d + 1 δ s . R_s(p^*)-R_s(\widehat{p}_s)\leq C_{\mathrm{tr},L}\delta_s+2\Delta^L_{s-1}+\sqrt{d+1}\delta_s. R s ( p ∗ ) − R s ( p s ) ≤ C tr , L δ s + 2 Δ s − 1 L + d + 1 δ s .
Therefore,
R s ( p ∗ ) − R s ( p ^ s ) ≤ C θ , L ∥ θ ^ s − θ ∗ ∥ 2 + 2 Δ s − 1 L , R_s(p^*)-R_s(\widehat{p}_s)\leq C_{\theta,L}\|\widehat{\theta}_s-\theta^*\|_2+2\Delta^L_{s-1}, R s ( p ∗ ) − R s ( p s ) ≤ C θ , L ∥ θ s − θ ∗ ∥ 2 + 2 Δ s − 1 L ,
with C θ , L = C t r , L + d + 1 C_{\theta,L}=C_{\mathrm{tr},L}+\sqrt{d+1} C θ , L = C tr , L + d + 1 . This completes the proof.
Lemma 17. Suppose Assumptions 1-4 hold. For any θ ′ ∈ Θ ~ j ∗ \theta'\in\widetilde{\Theta}_{j^*} θ ′ ∈ Θ j ∗ , define
q ‾ ∗ ( x ) = [ inf h ∈ H θ ∗ { q ∗ ( h ) + L ∥ x − h ∥ ν } ] [ 0 , 1 ] , x ∈ [ 0 , 1 ] d , \overline{q}^*(x)=\left[\inf_{h\in\mathcal{H}_{\theta^*}}\{q^*(h)+L\|x-h\|_\nu\}\right]_{[0,1]},\qquad x\in[0,1]^d, q ∗ ( x ) = [ h ∈ H θ ∗ inf { q ∗ ( h ) + L ∥ x − h ∥ ν } ] [ 0 , 1 ] , x ∈ [ 0 , 1 ] d , and define the restricted trace on H θ ′ \mathcal{H}_{\theta'} H θ ′ by setting q θ ′ † ( h ′ ) = q ‾ ∗ ( h ′ ) q_{\theta'}^\dagger(h')=\overline{q}^*(h') q θ ′ † ( h ′ ) = q ∗ ( h ′ ) , h ′ ∈ H θ ′ h'\in\mathcal{H}_{\theta'} h ′ ∈ H θ ′ . Set p θ ′ † = p q θ ′ † p_{\theta'}^\dagger=p_{q_{\theta'}^\dagger} p θ ′ † = p q θ ′ † . Then it holds that q θ ′ † ∈ Q θ ′ , L q_{\theta'}^\dagger\in\mathcal{Q}_{\theta',L} q θ ′ † ∈ Q θ ′ , L , and there exists a constant C t r , L < ∞ C_{\mathrm{tr},L}<\infty C tr , L < ∞ , depending only on d , ν , L d,\nu,L d , ν , L , the constants in Assumption 1 and the constants in Assumption 4, such that
E [ ∣ X ~ T θ ∗ ∣ ∣ p ∗ ( X ) − p θ ′ † ( X ) ∣ ] ≤ C t r , L ∥ θ ′ − θ ∗ ∥ 2 . \mathbb{E}\left[\left|\widetilde{X}^{\mathsf{T}}\theta^*\right||p^*(X)-p_{\theta'}^\dagger(X)|\right]\leq C_{\mathrm{tr},L}\|\theta'-\theta^*\|_2. E [ X T θ ∗ ∣ p ∗ ( X ) − p θ ′ † ( X ) ∣ ] ≤ C tr , L ∥ θ ′ − θ ∗ ∥ 2 .
Proof. Let δ = ∥ θ ′ − θ ∗ ∥ 2 \delta= \lVert\theta' - \theta^* \rVert_2 δ = ∥ θ ′ − θ ∗ ∥ 2 . Since θ ∗ , θ ′ ∈ Θ ~ j ∗ \theta^*, \theta' \in\widetilde{\Theta}_{j^*} θ ∗ , θ ′ ∈ Θ j ∗ , in the proof below, we worked with the fixed j ∗ j^* j ∗ -th chart Θ ~ j ∗ \widetilde{\Theta}_{j^*} Θ j ∗ .
Step 1: q θ ′ † q_{\theta'}^\dagger q θ ′ † is a feasible L L L -Lipschitz trace. The follow of this step follows from a similar argument as the one in the proof of Theorem 3 . Define G ( x ) = inf h ∈ H θ ∗ { q ∗ ( h ) + L ∥ x − h ∥ ν } G(x) = \inf_{h \in\mathcal{H}_{\theta^*}} \{q^*(h) + L\lVert x-h\rVert_\nu\} G ( x ) = inf h ∈ H θ ∗ { q ∗ ( h ) + L ∥ x − h ∥ ν } , where q ∗ ∈ arg max q ∈ Q θ ∗ , L R ( p θ ∗ , q ) q^* \in\operatorname*{arg\,max}_{q \in\mathcal{Q}_{\theta^*,L}} R(p_{\theta^*},q) q ∗ ∈ arg max q ∈ Q θ ∗ , L R ( p θ ∗ , q ) . For any x , y ∈ [ 0 , 1 ] d x,y \in[0,1]^d x , y ∈ [ 0 , 1 ] d and any h ∈ H θ ∗ h \in\mathcal{H}_{\theta^*} h ∈ H θ ∗ , the triangle inequality gives
q ∗ ( h ) + L ∥ x − h ∥ ν ≤ q ∗ ( h ) + L ∥ y − h ∥ ν + L ∥ x − y ∥ ν . q^*(h) + L\lVert x-h\rVert_\nu\leq q^*(h) + L\lVert y-h\rVert_\nu+ L\lVert x-y\rVert_\nu. q ∗ ( h ) + L ∥ x − h ∥ ν ≤ q ∗ ( h ) + L ∥ y − h ∥ ν + L ∥ x − y ∥ ν .
Taking the infimum over h ∈ H θ ∗ h \in\mathcal{H}_{\theta^*} h ∈ H θ ∗ gives G ( x ) ≤ G ( y ) + L ∥ x − y ∥ ν G(x) \leq G(y) + L\lVert x-y\rVert_\nu G ( x ) ≤ G ( y ) + L ∥ x − y ∥ ν . Reversing the roles of x x x and y y y gives ∣ G ( x ) − G ( y ) ∣ ≤ L ∥ x − y ∥ ν \lvert G(x)-G(y)\rvert\leq L\lVert x-y\rVert_\nu ∣ G ( x ) − G ( y )∣ ≤ L ∥ x − y ∥ ν . Since the clipping map z ↦ [ z ] [ 0 , 1 ] z \mapsto[z]_{[0,1]} z ↦ [ z ] [ 0 , 1 ] is nonexpansive, q ‾ ∗ \overline{q}^* q ∗ is also L L L -Lipschitz on [ 0 , 1 ] d [0,1]^d [ 0 , 1 ] d and takes values in [ 0 , 1 ] [0,1] [ 0 , 1 ] . Therefore, for any h 1 ′ , h 2 ′ ∈ H θ ′ h'_1,h'_2 \in\mathcal{H}_{\theta'} h 1 ′ , h 2 ′ ∈ H θ ′ , it holds that
∣ q θ ′ † ( h 1 ′ ) − q θ ′ † ( h 2 ′ ) ∣ = ∣ q ‾ ∗ ( h 1 ′ ) − q ‾ ∗ ( h 2 ′ ) ∣ ≤ L ∥ h 1 ′ − h 2 ′ ∥ ν . \lvert q_{\theta'}^\dagger(h'_1)-q_{\theta'}^\dagger(h'_2)\rvert= \lvert\overline{q}^*(h'_1)-\overline{q}^*(h'_2)\rvert\leq L\lVert h'_1-h'_2\rVert_\nu. ∣ q θ ′ † ( h 1 ′ ) − q θ ′ † ( h 2 ′ )∣ = ∣ q ∗ ( h 1 ′ ) − q ∗ ( h 2 ′ )∣ ≤ L ∥ h 1 ′ − h 2 ′ ∥ ν .
Thus q θ ′ † ∈ Q θ ′ , L q_{\theta'}^\dagger\in\mathcal{Q}_{\theta',L} q θ ′ † ∈ Q θ ′ , L .
Step 2: Upper bound on ∥ U θ ∗ , q ∗ − U θ ′ , q θ ′ † ∥ ∞ \lVert U_{\theta^*,q^*}-U_{\theta',q_{\theta'}^\dagger}\rVert_\infty ∥ U θ ∗ , q ∗ − U θ ′ , q θ ′ † ∥ ∞ . For any x ∈ [ 0 , 1 ] d x \in[0,1]^d x ∈ [ 0 , 1 ] d , define
U θ ∗ , q ∗ ( x ) = inf h ∈ H θ ∗ { q ∗ ( h ) + L ∥ x − h ∥ ν } , U θ ′ , q θ ′ † ( x ) = inf h ∈ H θ ′ { q θ ′ † ( h ) + L ∥ x − h ∥ ν } . U_{\theta^*,q^*}(x)=\inf_{h\in\mathcal{H}_{\theta^*}}\{q^*(h)+L\lVert x-h\rVert_\nu\},\qquad U_{\theta',q_{\theta'}^\dagger}(x)=\inf_{h\in\mathcal{H}_{\theta'}}\{q_{\theta'}^\dagger(h)+L\lVert x-h\rVert_\nu\}. U θ ∗ , q ∗ ( x ) = h ∈ H θ ∗ inf { q ∗ ( h ) + L ∥ x − h ∥ ν } , U θ ′ , q θ ′ † ( x ) = h ∈ H θ ′ inf { q θ ′ † ( h ) + L ∥ x − h ∥ ν } .
By Lemma 18, for every h ∈ H θ ∗ h \in\mathcal{H}_{\theta^*} h ∈ H θ ∗ , there exists h ′ ∈ H θ ′ h' \in\mathcal{H}_{\theta'} h ′ ∈ H θ ′ such that ∥ h − h ′ ∥ ν ≤ C H δ \lVert h-h'\rVert_\nu\leq C_H\delta ∥ h − h ′ ∥ ν ≤ C H δ . Note that the constant C H C_H C H already contains the norm-comparison factor C d , ν C_{d,\nu} C d , ν from Lemma 30 , as shown explicitly in the proof of Lemma 18. Hence, for such a pair, it holds that
U θ ′ , q θ ′ † ( x ) ≤ q θ ′ † ( h ′ ) + L ∥ x − h ′ ∥ ν = q ‾ ∗ ( h ′ ) + L ∥ x − h ′ ∥ ν ≤ q ∗ ( h ) + L ∥ h ′ − h ∥ ν + L ∥ x − h ′ ∥ ν ≤ q ∗ ( h ) + L ∥ h ′ − h ∥ ν + L ∥ h − h ′ ∥ ν + L ∥ x − h ∥ ν ≤ q ∗ ( h ) + L ∥ x − h ∥ ν + 2 L C H δ . \begin{aligned}
U_{\theta',q_{\theta'}^\dagger}(x)
&\leq q_{\theta'}^\dagger(h')+L\lVert x-h'\rVert_\nu\\
&=\overline{q}^*(h')+L\lVert x-h'\rVert_\nu\\
&\leq q^*(h)+L\lVert h'-h\rVert_\nu+L\lVert x-h'\rVert_\nu\\
&\leq q^*(h)+L\lVert h'-h\rVert_\nu+L\lVert h-h'\rVert_\nu+L\lVert x-h\rVert_\nu\\
&\leq q^*(h)+L\lVert x-h\rVert_\nu+2LC_H\delta.
\end{aligned} U θ ′ , q θ ′ † ( x ) ≤ q θ ′ † ( h ′ ) + L ∥ x − h ′ ∥ ν = q ∗ ( h ′ ) + L ∥ x − h ′ ∥ ν ≤ q ∗ ( h ) + L ∥ h ′ − h ∥ ν + L ∥ x − h ′ ∥ ν ≤ q ∗ ( h ) + L ∥ h ′ − h ∥ ν + L ∥ h − h ′ ∥ ν + L ∥ x − h ∥ ν ≤ q ∗ ( h ) + L ∥ x − h ∥ ν + 2 L C H δ .
Taking the infimum over h ∈ H θ ∗ h \in\mathcal{H}_{\theta^*} h ∈ H θ ∗ thus gives
U θ ′ , q θ ′ † ( x ) ≤ U θ ∗ , q ∗ ( x ) + 2 L C H δ . U_{\theta',q_{\theta'}^\dagger}(x)\leq U_{\theta^*,q^*}(x)+2LC_H\delta. U θ ′ , q θ ′ † ( x ) ≤ U θ ∗ , q ∗ ( x ) + 2 L C H δ .
Conversely, for every h ′ ∈ H θ ′ h' \in\mathcal{H}_{\theta'} h ′ ∈ H θ ′ and every h ∈ H θ ∗ h \in\mathcal{H}_{\theta^*} h ∈ H θ ∗ , the triangle inequality gives
q ∗ ( h ) + L ∥ h ′ − h ∥ ν + L ∥ x − h ′ ∥ ν ≥ q ∗ ( h ) + L ∥ x − h ∥ ν . q^*(h)+L\lVert h'-h\rVert_\nu+L\lVert x-h'\rVert_\nu\geq q^*(h)+L\lVert x-h\rVert_\nu. q ∗ ( h ) + L ∥ h ′ − h ∥ ν + L ∥ x − h ′ ∥ ν ≥ q ∗ ( h ) + L ∥ x − h ∥ ν .
Taking the infimum over h ∈ H θ ∗ h \in\mathcal{H}_{\theta^*} h ∈ H θ ∗ gives
G ( h ′ ) + L ∥ x − h ′ ∥ ν ≥ U θ ∗ , q ∗ ( x ) . (34) G(h')+L\lVert x-h'\rVert_\nu\geq U_{\theta^*,q^*}(x). \tag*{(34)} G ( h ′ ) + L ∥ x − h ′ ∥ ν ≥ U θ ∗ , q ∗ ( x ) . (34)
Since q ∗ q^* q ∗ takes values in [ 0 , 1 ] [0,1] [ 0 , 1 ] , we have G ( h ′ ) ≥ 0 G(h')\geq0 G ( h ′ ) ≥ 0 . Moreover, by Lemma 18, for every h ′ ∈ H θ ′ h' \in\mathcal{H}_{\theta'} h ′ ∈ H θ ′ there exists h ∈ H θ ∗ h \in\mathcal{H}_{\theta^*} h ∈ H θ ∗ such that ∥ h − h ′ ∥ ν ≤ C H δ \lVert h-h'\rVert_\nu\leq C_H\delta ∥ h − h ′ ∥ ν ≤ C H δ , and hence
G ( h ′ ) ≤ q ∗ ( h ) + L ∥ h ′ − h ∥ ν ≤ 1 + L C H δ . G(h')\leq q^*(h)+L\lVert h'-h\rVert_\nu\leq1+LC_H\delta. G ( h ′ ) ≤ q ∗ ( h ) + L ∥ h ′ − h ∥ ν ≤ 1 + L C H δ .
Therefore, since q ‾ ∗ ( h ′ ) \overline{q}^*(h') q ∗ ( h ′ ) is obtained by truncating G ( h ′ ) G(h') G ( h ′ ) to [ 0 , 1 ] [0,1] [ 0 , 1 ] , we have that
0 ≤ G ( h ′ ) − q ‾ ∗ ( h ′ ) ≤ L C H δ . 0\leq G(h')-\overline{q}^*(h')\leq LC_H\delta. 0 ≤ G ( h ′ ) − q ∗ ( h ′ ) ≤ L C H δ .
Combining this with (34) yields
q ˉ ∗ ( h ′ ) + L ∥ x − h ′ ∥ ν ≥ U θ ∗ , q ∗ ( x ) − L C H δ . \bar{q}^{*}(h') + L\lVert x-h'\rVert_{\nu} \ge U_{\theta^{*},q^{*}}(x)-LC_{H}\delta. q ˉ ∗ ( h ′ ) + L ∥ x − h ′ ∥ ν ≥ U θ ∗ , q ∗ ( x ) − L C H δ .
Taking the infimum over h ′ ∈ H θ ′ h' \in\mathcal{H}_{\theta'} h ′ ∈ H θ ′ gives
U θ ′ , q ˉ θ ′ ′ ( x ) ≥ U θ ∗ , q ∗ ( x ) − L C H δ . U_{\theta',\bar{q}'_{\theta'}}(x) \ge U_{\theta^{*},q^{*}}(x)-LC_{H}\delta. U θ ′ , q ˉ θ ′ ′ ( x ) ≥ U θ ∗ , q ∗ ( x ) − L C H δ .
Therefore, we have that, for any x ∈ [ 0 , 1 ] d x \in[0,1]^{d} x ∈ [ 0 , 1 ] d ,
∣ U θ ∗ , q ∗ ( x ) − U θ ′ , q ˉ θ ′ ′ ( x ) ∣ ≤ 2 L C H δ . \lvert U_{\theta^{*},q^{*}}(x)-U_{\theta',\bar{q}'_{\theta'}}(x)\rvert\le2LC_{H}\delta. ∣ U θ ∗ , q ∗ ( x ) − U θ ′ , q ˉ θ ′ ′ ( x )∣ ≤ 2 L C H δ .
Step 3: Upper bound on ∥ M θ ∗ , q ∗ − M θ ′ , q ˉ θ ′ ′ ∥ ∞ \lVert M_{\theta^{*},q^{*}}-M_{\theta',\bar{q}'_{\theta'}}\rVert_{\infty} ∥ M θ ∗ , q ∗ − M θ ′ , q ˉ θ ′ ′ ∥ ∞ . For any x ∈ [ 0 , 1 ] d x \in[0,1]^{d} x ∈ [ 0 , 1 ] d , define
M θ ∗ , q ∗ ( x ) = sup h ∈ H θ ∗ { q ∗ ( h ) − L ∥ x − h ∥ ν } , M θ ′ , q ˉ θ ′ ′ ( x ) = sup h ∈ H θ ′ { q ˉ θ ′ ′ ( h ) − L ∥ x − h ∥ ν } . M_{\theta^{*},q^{*}}(x)=\sup_{h\in\mathcal{H}_{\theta^{*}}}\{q^{*}(h)-L\lVert x-h\rVert_{\nu}\},\qquad M_{\theta',\bar{q}'_{\theta'}}(x)=\sup_{h\in\mathcal{H}_{\theta'}}\{\bar{q}'_{\theta'}(h)-L\lVert x-h\rVert_{\nu}\}. M θ ∗ , q ∗ ( x ) = h ∈ H θ ∗ sup { q ∗ ( h ) − L ∥ x − h ∥ ν } , M θ ′ , q ˉ θ ′ ′ ( x ) = h ∈ H θ ′ sup { q ˉ θ ′ ′ ( h ) − L ∥ x − h ∥ ν } .
First, by Lemma 18, for every h ′ ∈ H θ ′ h' \in\mathcal{H}_{\theta'} h ′ ∈ H θ ′ , we can choose h ∈ H θ ∗ h \in\mathcal{H}_{\theta^{*}} h ∈ H θ ∗ such that ∥ h − h ′ ∥ ν ≤ C H δ \lVert h-h'\rVert_{\nu}\le C_{H}\delta ∥ h − h ′ ∥ ν ≤ C H δ . Since q ˉ ∗ ( h ′ ) ≤ q ∗ ( h ) + L ∥ h ′ − h ∥ ν \bar{q}^{*}(h')\le q^{*}(h)+L\lVert h'-h\rVert_{\nu} q ˉ ∗ ( h ′ ) ≤ q ∗ ( h ) + L ∥ h ′ − h ∥ ν , we have
q ˉ ∗ ( h ′ ) − L ∥ x − h ′ ∥ ν ≤ q ∗ ( h ) + L ∥ h ′ − h ∥ ν − L ∥ x − h ′ ∥ ν ≤ q ∗ ( h ) − L ∥ x − h ∥ ν + 2 L C H δ ≤ M θ ∗ , q ∗ ( x ) + 2 L C H δ , \begin{aligned}
\bar{q}^{*}(h')-L\lVert x-h'\rVert_{\nu}\le q^{*}(h)+L\lVert h'-h\rVert_{\nu}-L\lVert x-h'\rVert_{\nu} \\
&\le q^{*}(h)-L\lVert x-h\rVert_{\nu}+2LC_{H}\delta\\
&\le M_{\theta^{*},q^{*}}(x)+2LC_{H}\delta,
\end{aligned} q ˉ ∗ ( h ′ ) − L ∥ x − h ′ ∥ ν ≤ q ∗ ( h ) + L ∥ h ′ − h ∥ ν − L ∥ x − h ′ ∥ ν ≤ q ∗ ( h ) − L ∥ x − h ∥ ν + 2 L C H δ ≤ M θ ∗ , q ∗ ( x ) + 2 L C H δ ,
where the second inequality follows from the triangle inequality. Taking the supremum over h ′ ∈ H θ ′ h' \in\mathcal{H}_{\theta'} h ′ ∈ H θ ′ gives
M θ ′ , q ˉ θ ′ ′ ( x ) ≤ M θ ∗ , q ∗ ( x ) + 2 L C H δ . M_{\theta',\bar{q}'_{\theta'}}(x)\le M_{\theta^{*},q^{*}}(x)+2LC_{H}\delta. M θ ′ , q ˉ θ ′ ′ ( x ) ≤ M θ ∗ , q ∗ ( x ) + 2 L C H δ .
For the reverse inequality, by Lemma 18, for any fix h ∈ H θ ∗ h\in\mathcal{H}_{\theta^{*}} h ∈ H θ ∗ , we can choose h ′ ∈ H θ ′ h'\in\mathcal{H}_{\theta'} h ′ ∈ H θ ′ such that ∥ h − h ′ ∥ ν ≤ C H δ \lVert h-h'\rVert_{\nu}\le C_{H}\delta ∥ h − h ′ ∥ ν ≤ C H δ . For any h ~ ∈ H θ ∗ \tilde{h}\in\mathcal{H}_{\theta^{*}} h ~ ∈ H θ ∗ , since q ∗ q^{*} q ∗ is L L L -Lipschitz on H θ ∗ \mathcal{H}_{\theta^{*}} H θ ∗ , we have that
q ∗ ( h ~ ) + L ∥ h ′ − h ~ ∥ ν ≥ q ∗ ( h ) − L ∥ h − h ~ ∥ ν + L ∥ h ′ − h ~ ∥ ν ≥ q ∗ ( h ) − L ∥ h − h ′ ∥ ν . q^{*}(\tilde{h})+L\lVert h'-\tilde{h}\rVert_{\nu}\ge q^{*}(h)-L\lVert h-\tilde{h}\rVert_{\nu}+L\lVert h'-\tilde{h}\rVert_{\nu}\ge q^{*}(h)-L\lVert h-h'\rVert_{\nu}. q ∗ ( h ~ ) + L ∥ h ′ − h ~ ∥ ν ≥ q ∗ ( h ) − L ∥ h − h ~ ∥ ν + L ∥ h ′ − h ~ ∥ ν ≥ q ∗ ( h ) − L ∥ h − h ′ ∥ ν .
Taking the infimum over h ~ ∈ H θ ∗ \tilde{h}\in\mathcal{H}_{\theta^{*}} h ~ ∈ H θ ∗ gives q ˉ ∗ ( h ′ ) ≥ q ∗ ( h ) − L C H δ \bar{q}^{*}(h')\ge q^{*}(h)-LC_{H}\delta q ˉ ∗ ( h ′ ) ≥ q ∗ ( h ) − L C H δ . Hence, we have that
M θ ′ , q ˉ θ ′ ′ ( x ) ≥ q ˉ ∗ ( h ′ ) − L ∥ x − h ′ ∥ ν ≥ q ∗ ( h ) − L C H δ − L ∥ x − h ′ ∥ ν ≥ q ∗ ( h ) − L ∥ x − h ∥ ν − 2 L C H δ . \begin{aligned}
M_{\theta',\bar{q}'_{\theta'}}(x)\ge\bar{q}^{*}(h')-L\lVert x-h'\rVert_{\nu} \\
&\ge q^{*}(h)-LC_{H}\delta-L\lVert x-h'\rVert_{\nu} \\
&\ge q^{*}(h)-L\lVert x-h\rVert_{\nu}-2LC_{H}\delta.
\end{aligned} M θ ′ , q ˉ θ ′ ′ ( x ) ≥ q ˉ ∗ ( h ′ ) − L ∥ x − h ′ ∥ ν ≥ q ∗ ( h ) − L C H δ − L ∥ x − h ′ ∥ ν ≥ q ∗ ( h ) − L ∥ x − h ∥ ν − 2 L C H δ .
Taking the supremum over h ∈ H θ ∗ h\in\mathcal{H}_{\theta^{*}} h ∈ H θ ∗ gives
M θ ′ , q ˉ θ ′ ′ ( x ) ≥ M θ ∗ , q ∗ ( x ) − 2 L C H δ . M_{\theta',\bar{q}'_{\theta'}}(x)\ge M_{\theta^{*},q^{*}}(x)-2LC_{H}\delta. M θ ′ , q ˉ θ ′ ′ ( x ) ≥ M θ ∗ , q ∗ ( x ) − 2 L C H δ .
Therefore,
∣ M θ ∗ , q ∗ ( x ) − M θ ′ , q ˉ θ ′ ′ ( x ) ∣ ≤ 2 L C H δ . \lvert M_{\theta^{*},q^{*}}(x)-M_{\theta',\bar{q}'_{\theta'}}(x)\rvert\le2LC_{H}\delta. ∣ M θ ∗ , q ∗ ( x ) − M θ ′ , q ˉ θ ′ ′ ( x )∣ ≤ 2 L C H δ .
Step 4: Upper bound on E [ ∣ X ~ T θ ∗ ∣ ∣ p ∗ ( X ) − p θ ′ † ( X ) ∣ ] \mathbb{E}[\lvert\tilde{X}^{\mathsf{T}}\theta^{*}\rvert\lvert p^{*}(X)-p^{\dagger}_{\theta'}(X)\rvert] E [∣ X ~ T θ ∗ ∣ ∣ p ∗ ( X ) − p θ ′ † ( X )∣] . For θ = ( a , γ T ) T ∈ Θ ~ \theta=(a,\gamma^{\mathsf{T}})^{\mathsf{T}}\in\widetilde{\Theta} θ = ( a , γ T ) T ∈ Θ , define s θ ( x ) = x ~ T θ = a + γ T x s_{\theta}(x)=\tilde{x}^{\mathsf{T}}\theta=a+\gamma^{\mathsf{T}}x s θ ( x ) = x ~ T θ = a + γ T x . Also, define the sign-disagreement set
S ( θ ∗ , θ ′ ) = { x ∈ [ 0 , 1 ] d : s θ ∗ ( x ) ≥ 0 , s θ ′ ( x ) < 0 } ∪ { x ∈ [ 0 , 1 ] d : s θ ∗ ( x ) < 0 , s θ ′ ( x ) ≥ 0 } . S(\theta^{*},\theta')=\{x\in[0,1]^{d}:s_{\theta^{*}}(x)\ge0,\ s_{\theta'}(x)<0\}\cup\{x\in[0,1]^{d}:s_{\theta^{*}}(x)<0,\ s_{\theta'}(x)\ge0\}. S ( θ ∗ , θ ′ ) = { x ∈ [ 0 , 1 ] d : s θ ∗ ( x ) ≥ 0 , s θ ′ ( x ) < 0 } ∪ { x ∈ [ 0 , 1 ] d : s θ ∗ ( x ) < 0 , s θ ′ ( x ) ≥ 0 } .
On S ( θ ∗ , θ ′ ) c S(\theta^*,\theta')^c S ( θ ∗ , θ ′ ) c , both policies use the same type of envelope, either both upper envelopes or both lower envelopes. Since y ↦ min { 1 , y } y \mapsto\min\{1,y\} y ↦ min { 1 , y } and y ↦ max { 0 , y } y \mapsto\max\{0,y\} y ↦ max { 0 , y } are nonexpansive, results in Steps 2 and 3 imply that
∣ p ∗ ( x ) − p θ ′ † ( x ) ∣ ≤ 2 L C H δ , for all x ∈ S ( θ ∗ , θ ′ ) c . |p^*(x)-p_{\theta'}^\dagger(x)| \le2LC_H\delta,\quad\text{for all }x\in S(\theta^*,\theta')^c. ∣ p ∗ ( x ) − p θ ′ † ( x ) ∣ ≤ 2 L C H δ , for all x ∈ S ( θ ∗ , θ ′ ) c .
On S ( θ ∗ , θ ′ ) S(\theta^*,\theta') S ( θ ∗ , θ ′ ) , we will use the crude bound ∣ p ∗ ( x ) − p θ ′ † ( x ) ∣ ≤ 1 |p^*(x)-p_{\theta'}^\dagger(x)|\le1 ∣ p ∗ ( x ) − p θ ′ † ( x ) ∣ ≤ 1 .
Hence we have that
E [ ∣ X ~ ⊤ θ ∗ ∣ ∣ p ∗ ( X ) − p θ ′ † ( X ) ∣ ] = E [ ∣ X ~ ⊤ θ ∗ ∣ ∣ p ∗ ( X ) − p θ ′ † ( X ) ∣ 1 { X ∈ S ( θ ∗ , θ ′ ) c } ] + E [ ∣ X ~ ⊤ θ ∗ ∣ ∣ p ∗ ( X ) − p θ ′ † ( X ) ∣ 1 { X ∈ S ( θ ∗ , θ ′ ) } ] ≤ 2 C β d + 1 { 2 L C H δ P { X ∈ S ( θ ∗ , θ ′ ) c } + P { X ∈ S ( θ ∗ , θ ′ ) } } ≤ 2 C β d + 1 { 2 L C H δ + P { X ∈ S ( θ ∗ , θ ′ ) } } ≤ 2 C β d + 1 { 2 L C H δ + C s l a b δ } = 2 C β d + 1 ( 2 L C H + C s l a b ) δ , \begin{aligned}
\mathbb{E}\left[|\widetilde{X}^{\top}\theta^*||p^*(X)-p_{\theta'}^\dagger(X)|\right]
&=\mathbb{E}\left[|\widetilde{X}^{\top}\theta^*||p^*(X)-p_{\theta'}^\dagger(X)|1\{X\in S(\theta^*,\theta')^c\}\right]\\
&\quad+\mathbb{E}\left[|\widetilde{X}^{\top}\theta^*||p^*(X)-p_{\theta'}^\dagger(X)|1\{X\in S(\theta^*,\theta')\}\right]\\
&\le2C_\beta\sqrt{d+1}\left\{2LC_H\delta\mathbb{P}\{X\in S(\theta^*,\theta')^c\}+\mathbb{P}\{X\in S(\theta^*,\theta')\}\right\}\\
&\le2C_\beta\sqrt{d+1}\left\{2LC_H\delta+\mathbb{P}\{X\in S(\theta^*,\theta')\}\right\}\\
&\le2C_\beta\sqrt{d+1}\left\{2LC_H\delta+C_{\mathrm{slab}}\delta\right\}\\
&=2C_\beta\sqrt{d+1}(2LC_H+C_{\mathrm{slab}})\delta,
\end{aligned} E [ ∣ X ⊤ θ ∗ ∣∣ p ∗ ( X ) − p θ ′ † ( X ) ∣ ] = E [ ∣ X ⊤ θ ∗ ∣∣ p ∗ ( X ) − p θ ′ † ( X ) ∣1 { X ∈ S ( θ ∗ , θ ′ ) c } ] + E [ ∣ X ⊤ θ ∗ ∣∣ p ∗ ( X ) − p θ ′ † ( X ) ∣1 { X ∈ S ( θ ∗ , θ ′ )} ] ≤ 2 C β d + 1 { 2 L C H δ P { X ∈ S ( θ ∗ , θ ′ ) c } + P { X ∈ S ( θ ∗ , θ ′ )} } ≤ 2 C β d + 1 { 2 L C H δ + P { X ∈ S ( θ ∗ , θ ′ )} } ≤ 2 C β d + 1 { 2 L C H δ + C slab δ } = 2 C β d + 1 ( 2 L C H + C slab ) δ ,
where the first inequality follows from the fact that ∣ X ~ ⊤ θ ∗ ∣ ≤ d + 1 ∥ θ ∗ ∥ 2 ≤ 2 C β d + 1 |\widetilde{X}^{\top}\theta^*|\le\sqrt{d+1}\|\theta^*\|_2\le2C_\beta\sqrt{d+1} ∣ X ⊤ θ ∗ ∣ ≤ d + 1 ∥ θ ∗ ∥ 2 ≤ 2 C β d + 1 and the third inequality follows from Lemma 19. Thus, the claim is proved.
Lemma 18 (Hausdorff stability). Suppose Assumption 4 holds. Then there exists a constant C H < ∞ C_H<\infty C H < ∞ , depending only on d , ν d,\nu d , ν and constants A ~ 0 , B ~ 0 , c ~ 0 , C ~ 0 , ρ ~ 0 \widetilde{A}_0,\widetilde{B}_0,\widetilde{c}_0,\widetilde{C}_0,\widetilde{\rho}_0 A 0 , B 0 , c 0 , C 0 , ρ 0 in Lemma 14, such that, for every j ∈ { 1 , … , d } j\in\{1,\ldots,d\} j ∈ { 1 , … , d } and all θ , θ ′ ∈ Θ ~ j \theta,\theta'\in\widetilde{\Theta}_j θ , θ ′ ∈ Θ j ,
d H , ν ( H θ , H θ ′ ) ≤ C H ∥ θ − θ ′ ∥ 2 . d_{H,\nu}(\mathcal{H}_{\theta},\mathcal{H}_{\theta'})\le C_H\|\theta-\theta'\|_2. d H , ν ( H θ , H θ ′ ) ≤ C H ∥ θ − θ ′ ∥ 2 .
Proof. Fix j ∈ { 1 , … , d } j\in\{1,\ldots,d\} j ∈ { 1 , … , d } . By relabelling coordinates, without loss of generality, we assume that j = d j=d j = d . Write θ = ( a , γ ⊤ ) ⊤ \theta=(a,\gamma^\top)^\top θ = ( a , γ ⊤ ) ⊤ and θ ′ = ( a ′ , γ ′ ⊤ ) ⊤ \theta'=(a',\gamma'^\top)^\top θ ′ = ( a ′ , γ ′⊤ ) ⊤ , where a , a ′ ∈ R a,a'\in\mathbb{R} a , a ′ ∈ R and γ , γ ′ ∈ R d \gamma,\gamma'\in\mathbb{R}^d γ , γ ′ ∈ R d . Then write γ = ( b , c ) \gamma=(b,c) γ = ( b , c ) and γ ′ = ( b ′ , c ′ ) \gamma'=(b',c') γ ′ = ( b ′ , c ′ ) in this chart, where b , b ′ ∈ R d − 1 b,b'\in\mathbb{R}^{d-1} b , b ′ ∈ R d − 1 and c , c ′ ∈ R c,c'\in\mathbb{R} c , c ′ ∈ R . Note that, for every u ∈ [ 0 , 1 ] d − 1 u\in[0,1]^{d-1} u ∈ [ 0 , 1 ] d − 1 , it holds that
∣ ϕ θ ( u ) − ϕ θ ′ ( u ) ∣ = ∣ a + b ⊤ u c − a ′ + b ′ ⊤ u c ′ ∣ ≤ ∣ a − a ′ ∣ + ∣ ( b − b ′ ) ⊤ u ∣ c ~ 0 + ∣ a ′ + b ′ ⊤ u ∣ c ~ 0 2 ∣ c − c ′ ∣ ≤ C ϕ ∥ θ − θ ′ ∥ 2 , (35) \begin{aligned}
|\phi_\theta(u)-\phi_{\theta'}(u)|
&=\left|\frac{a+b^\top u}{c}-\frac{a'+b'^\top u}{c'}\right|\\
&\le\frac{|a-a'|+|(b-b')^\top u|}{\widetilde{c}_0}
+\frac{|a'+b'^\top u|}{\widetilde{c}_0^2}|c-c'|\\
&\le C_\phi\|\theta-\theta'\|_2, \tag*{(35)}
\end{aligned} ∣ ϕ θ ( u ) − ϕ θ ′ ( u ) ∣ = c a + b ⊤ u − c ′ a ′ + b ′⊤ u ≤ c 0 ∣ a − a ′ ∣ + ∣ ( b − b ′ ) ⊤ u ∣ + c 0 2 ∣ a ′ + b ′⊤ u ∣ ∣ c − c ′ ∣ ≤ C ϕ ∥ θ − θ ′ ∥ 2 , (35)
where
C ϕ = 1 + d − 1 c ~ 0 + A ~ 0 + B ~ 0 d − 1 c ~ 0 2 . C_\phi=\frac{1+\sqrt{d-1}}{\widetilde{c}_0}
+\frac{\widetilde{A}_0+\widetilde{B}_0\sqrt{d-1}}{\widetilde{c}_0^2}. C ϕ = c 0 1 + d − 1 + c 0 2 A 0 + B 0 d − 1 .
Set η = C ϕ ∥ θ − θ ′ ∥ 2 \eta=C_\phi\|\theta-\theta'\|_2 η = C ϕ ∥ θ − θ ′ ∥ 2 and let C d , ν = d max { 1 / ν − 1 / 2 , 0 } C_{d,\nu}=d^{\max\{1/\nu-1/2,0\}} C d , ν = d m a x { 1/ ν − 1/2 , 0 } be the constant in Lemma 30. We show that every point of H θ \mathcal{H}_\theta H θ is within ℓ ν \ell_\nu ℓ ν -distance C η C\eta C η of H θ ′ \mathcal{H}_{\theta'} H θ ′ . The reverse direction then follows by symmetry.
Take any h ∈ H θ h\in\mathcal{H}_\theta h ∈ H θ and, by Assumption 4, we can always find u ∈ [ 0 , 1 ] d − 1 u\in[0,1]^{d-1} u ∈ [ 0 , 1 ] d − 1 such that it can be decomposed as h = ( u , ϕ θ ( u ) ) h=(u,\phi_\theta(u)) h = ( u , ϕ θ ( u )) . By the definition of H θ \mathcal{H}_\theta H θ , 0 ≤ ϕ θ ( u ) ≤ 1 0\le\phi_\theta(u)\le1 0 ≤ ϕ θ ( u ) ≤ 1 . Therefore, by (34) and the triangle inequality, we have that − η ≤ ϕ θ ′ ( u ) ≤ 1 + η -\eta\le\phi_{\theta'}(u)\le1+\eta − η ≤ ϕ θ ′ ( u ) ≤ 1 + η . Case 1: If 0 ≤ ϕ θ ′ ( u ) ≤ 1 0 \le\phi_{\theta'}(u) \le1 0 ≤ ϕ θ ′ ( u ) ≤ 1 , then h ′ = ( u , ϕ θ ′ ( u ) ) ∈ H θ ′ h'=(u,\phi_{\theta'}(u)) \in H_{\theta'} h ′ = ( u , ϕ θ ′ ( u )) ∈ H θ ′ , and hence
∥ h − h ′ ∥ ν = ∣ ϕ θ ( u ) − ϕ θ ′ ( u ) ∣ ≤ η . \|h-h'\|_{\nu}=|\phi_{\theta}(u)-\phi_{\theta'}(u)|\le\eta. ∥ h − h ′ ∥ ν = ∣ ϕ θ ( u ) − ϕ θ ′ ( u ) ∣ ≤ η .
Case 2: Next suppose ϕ θ ′ ( u ) < 0 \phi_{\theta'}(u)<0 ϕ θ ′ ( u ) < 0 . By the definition of Θ ~ j \widetilde{\Theta}_j Θ j , there exists u θ ′ ∈ [ 0 , 1 ] d − 1 u_{\theta'}\in[0,1]^{d-1} u θ ′ ∈ [ 0 , 1 ] d − 1 such that ρ ~ 0 ≤ ϕ θ ′ ( u θ ′ ) ≤ 1 − ρ ~ 0 \widetilde{\rho}_0\le\phi_{\theta'}(u_{\theta'})\le1-\widetilde{\rho}_0 ρ 0 ≤ ϕ θ ′ ( u θ ′ ) ≤ 1 − ρ 0 . For λ ∈ [ 0 , 1 ] \lambda\in[0,1] λ ∈ [ 0 , 1 ] , define u λ = ( 1 − λ ) u + λ u θ ′ u_{\lambda}=(1-\lambda)u+\lambda u_{\theta'} u λ = ( 1 − λ ) u + λ u θ ′ . Since ϕ θ ′ \phi_{\theta'} ϕ θ ′ is affine, the map λ ↦ ϕ θ ′ ( u λ ) \lambda\mapsto\phi_{\theta'}(u_{\lambda}) λ ↦ ϕ θ ′ ( u λ ) is affine. Since ϕ θ ′ ( u ) < 0 \phi_{\theta'}(u)<0 ϕ θ ′ ( u ) < 0 and ϕ θ ′ ( u θ ′ ) ≥ ρ ~ 0 \phi_{\theta'}(u_{\theta'})\ge\widetilde{\rho}_0 ϕ θ ′ ( u θ ′ ) ≥ ρ 0 , there exists λ 0 ∈ [ 0 , 1 ] \lambda_0\in[0,1] λ 0 ∈ [ 0 , 1 ] such that ϕ θ ′ ( u λ 0 ) = 0 \phi_{\theta'}(u_{\lambda_0})=0 ϕ θ ′ ( u λ 0 ) = 0 . Moreover,
λ 0 = − ϕ θ ′ ( u ) ϕ θ ′ ( u θ ′ ) − ϕ θ ′ ( u ) ≤ η ρ ~ 0 + η ≤ η ρ ~ 0 . \lambda_0=\frac{-\phi_{\theta'}(u)}{\phi_{\theta'}(u_{\theta'})-\phi_{\theta'}(u)}
\le\frac{\eta}{\widetilde{\rho}_0+\eta}\le\frac{\eta}{\widetilde{\rho}_0}. λ 0 = ϕ θ ′ ( u θ ′ ) − ϕ θ ′ ( u ) − ϕ θ ′ ( u ) ≤ ρ 0 + η η ≤ ρ 0 η .
Set v = u λ 0 v=u_{\lambda_0} v = u λ 0 and h ′ = ( v , 0 ) h'=(v,0) h ′ = ( v , 0 ) . Then h ′ ∈ H θ ′ h'\in H_{\theta'} h ′ ∈ H θ ′ . We first bound the distance in the Euclidean norm and then convert it to the fairness norm. Since u , u θ ′ ∈ [ 0 , 1 ] d − 1 u,u_{\theta'}\in[0,1]^{d-1} u , u θ ′ ∈ [ 0 , 1 ] d − 1 ,
∥ v − u ∥ 2 ≤ λ 0 ∥ u θ ′ − u ∥ 2 ≤ d − 1 ρ ~ 0 η . \|v-u\|_2\le\lambda_0\|u_{\theta'}-u\|_2\le\frac{\sqrt{d-1}}{\widetilde{\rho}_0}\eta. ∥ v − u ∥ 2 ≤ λ 0 ∥ u θ ′ − u ∥ 2 ≤ ρ 0 d − 1 η .
By Lemma 30 ,
∥ v − u ∥ ν ≤ C d , ν ∥ v − u ∥ 2 ≤ C d , ν d − 1 ρ ~ 0 η . \|v-u\|_{\nu}\le C_{d,\nu}\|v-u\|_2\le\frac{C_{d,\nu}\sqrt{d-1}}{\widetilde{\rho}_0}\eta. ∥ v − u ∥ ν ≤ C d , ν ∥ v − u ∥ 2 ≤ ρ 0 C d , ν d − 1 η .
Since ϕ θ ′ ( u ) < 0 \phi_{\theta'}(u)<0 ϕ θ ′ ( u ) < 0 and ∣ ϕ θ ( u ) − ϕ θ ′ ( u ) ∣ ≤ η |\phi_{\theta}(u)-\phi_{\theta'}(u)|\le\eta ∣ ϕ θ ( u ) − ϕ θ ′ ( u ) ∣ ≤ η , we also have ϕ θ ( u ) ≤ η \phi_{\theta}(u)\le\eta ϕ θ ( u ) ≤ η . Hence, using the triangle inequality for ∥ ⋅ ∥ ν \|\cdot\|_{\nu} ∥ ⋅ ∥ ν ,
∥ h − h ′ ∥ ν ≤ ∥ u − v ∥ ν + ∣ ϕ θ ( u ) ∣ ≤ ( 1 + C d , ν d − 1 ρ ~ 0 ) η . \begin{aligned}
\|h-h'\|_{\nu}\le\|u-v\|_{\nu}+|\phi_{\theta}(u)| \\
&\le\left(1+\frac{C_{d,\nu}\sqrt{d-1}}{\widetilde{\rho}_0}\right)\eta.
\end{aligned} ∥ h − h ′ ∥ ν ≤ ∥ u − v ∥ ν + ∣ ϕ θ ( u ) ∣ ≤ ( 1 + ρ 0 C d , ν d − 1 ) η .
Case 3: Finally suppose ϕ θ ′ ( u ) > 1 \phi_{\theta'}(u)>1 ϕ θ ′ ( u ) > 1 . Again, by the definition of Θ ~ j \widetilde{\Theta}_j Θ j , there exists u θ ′ ∈ [ 0 , 1 ] d − 1 u_{\theta'}\in[0,1]^{d-1} u θ ′ ∈ [ 0 , 1 ] d − 1 such that ρ ~ 0 ≤ ϕ θ ′ ( u θ ′ ) ≤ 1 − ρ ~ 0 \widetilde{\rho}_0\le\phi_{\theta'}(u_{\theta'})\le1-\widetilde{\rho}_0 ρ 0 ≤ ϕ θ ′ ( u θ ′ ) ≤ 1 − ρ 0 . Along the same segment u λ = ( 1 − λ ) u + λ u θ ′ u_{\lambda}=(1-\lambda)u+\lambda u_{\theta'} u λ = ( 1 − λ ) u + λ u θ ′ , there exists λ 0 ∈ [ 0 , 1 ] \lambda_0\in[0,1] λ 0 ∈ [ 0 , 1 ] such that ϕ θ ′ ( u λ 0 ) = 1 \phi_{\theta'}(u_{\lambda_0})=1 ϕ θ ′ ( u λ 0 ) = 1 . Moreover,
λ 0 = ϕ θ ′ ( u ) − 1 ϕ θ ′ ( u ) − ϕ θ ′ ( u θ ′ ) ≤ η ρ ~ 0 + η ≤ η ρ ~ 0 . \lambda_0=\frac{\phi_{\theta'}(u)-1}{\phi_{\theta'}(u)-\phi_{\theta'}(u_{\theta'})}
\le\frac{\eta}{\widetilde{\rho}_0+\eta}\le\frac{\eta}{\widetilde{\rho}_0}. λ 0 = ϕ θ ′ ( u ) − ϕ θ ′ ( u θ ′ ) ϕ θ ′ ( u ) − 1 ≤ ρ 0 + η η ≤ ρ 0 η .
Set v = u λ 0 v=u_{\lambda_0} v = u λ 0 and h ′ = ( v , 1 ) h'=(v,1) h ′ = ( v , 1 ) . Then h ′ ∈ H θ ′ h'\in H_{\theta'} h ′ ∈ H θ ′ . As in Case 2 ,
∥ v − u ∥ ν ≤ C d , ν ∥ v − u ∥ 2 ≤ C d , ν d − 1 ρ ~ 0 η , \|v-u\|_{\nu}\le C_{d,\nu}\|v-u\|_2\le\frac{C_{d,\nu}\sqrt{d-1}}{\widetilde{\rho}_0}\eta, ∥ v − u ∥ ν ≤ C d , ν ∥ v − u ∥ 2 ≤ ρ 0 C d , ν d − 1 η ,
where the first inequality follows from Lemma 30 . Since ϕ θ ′ ( u ) > 1 \phi_{\theta'}(u)>1 ϕ θ ′ ( u ) > 1 and ∣ ϕ θ ( u ) − ϕ θ ′ ( u ) ∣ ≤ η |\phi_{\theta}(u)-\phi_{\theta'}(u)|\le\eta ∣ ϕ θ ( u ) − ϕ θ ′ ( u ) ∣ ≤ η , we have ϕ θ ( u ) ≥ 1 − η \phi_{\theta}(u)\ge1-\eta ϕ θ ( u ) ≥ 1 − η . Hence
∥ h − h ′ ∥ ν ≤ ∥ u − v ∥ ν + ∣ ϕ θ ( u ) − 1 ∣ ≤ ( 1 + C d , ν d − 1 ρ ~ 0 ) η . \begin{aligned}
\|h-h'\|_{\nu}\le\|u-v\|_{\nu}+|\phi_{\theta}(u)-1| \\
&\le\left(1+\frac{C_{d,\nu}\sqrt{d-1}}{\widetilde{\rho}_0}\right)\eta.
\end{aligned} ∥ h − h ′ ∥ ν ≤ ∥ u − v ∥ ν + ∣ ϕ θ ( u ) − 1∣ ≤ ( 1 + ρ 0 C d , ν d − 1 ) η .
Combining the three cases gives
sup h ∈ H θ dist ν ( h , H θ ′ ) = sup h ∈ H θ inf h ~ ∈ H θ ′ ∥ h − h ~ ∥ ν ≤ C C ϕ ∥ θ − θ ′ ∥ 2 , \sup_{h\in H_{\theta}}\operatorname{dist}_{\nu}(h,H_{\theta'})
=\sup_{h\in H_{\theta}}\inf_{\widetilde{h}\in H_{\theta'}}\|h-\widetilde{h}\|_{\nu}
\le CC_{\phi}\|\theta-\theta'\|_2, h ∈ H θ sup dist ν ( h , H θ ′ ) = h ∈ H θ sup h ∈ H θ ′ inf ∥ h − h ∥ ν ≤ C C ϕ ∥ θ − θ ′ ∥ 2 ,
where C < ∞ C<\infty C < ∞ depends only on d d d , ν \nu ν and ρ ~ 0 \widetilde{\rho}_0 ρ 0 . By symmetry, we also have
sup h ′ ∈ H θ ′ dist ν ( h ′ , H θ ) ≤ C C ϕ ∥ θ − θ ′ ∥ 2 . \sup_{h'\in H_{\theta'}}\operatorname{dist}_{\nu}(h',H_{\theta})
\le CC_{\phi}\|\theta-\theta'\|_2. h ′ ∈ H θ ′ sup dist ν ( h ′ , H θ ) ≤ C C ϕ ∥ θ − θ ′ ∥ 2 .
Therefore, the lemma follows.
Control of complexity of functional class
Lemma 19 (Probability bound for disagreement). Suppose Assumptions 1-4 hold. For θ = ( a , γ ⊤ ) ⊤ ∈ Θ ~ \theta=(a,\gamma^\top)^\top\in\widetilde{\Theta} θ = ( a , γ ⊤ ) ⊤ ∈ Θ , define s θ ( x ) = x ~ ⊤ θ = a + γ ⊤ x s_\theta(x)=\widetilde{x}^{\top}\theta=a+\gamma^\top x s θ ( x ) = x ⊤ θ = a + γ ⊤ x . Then there exists a constant C s l a b < ∞ C_{\mathrm{slab}}<\infty C slab < ∞ , depending only on d d d , c ~ 0 \widetilde{c}_0 c 0 and C g C_g C g , such that, for all θ , θ ′ ∈ Θ ~ \theta,\theta'\in\widetilde{\Theta} θ , θ ′ ∈ Θ ,
P [ { s θ ( X ) ≥ 0 , s θ ′ ( X ) < 0 } ∪ { s θ ( X ) < 0 , s θ ′ ( X ) ≥ 0 } ] ≤ C s l a b ∥ θ − θ ′ ∥ 2 . \mathbb{P}\left[\{s_\theta(X)\geq0,\ s_{\theta'}(X)<0\}\cup\{s_\theta(X)<0,\ s_{\theta'}(X)\geq0\}\right]\leq C_{\mathrm{slab}}\|\theta-\theta'\|_2. P [ { s θ ( X ) ≥ 0 , s θ ′ ( X ) < 0 } ∪ { s θ ( X ) < 0 , s θ ′ ( X ) ≥ 0 } ] ≤ C slab ∥ θ − θ ′ ∥ 2 .
Proof. Fix θ = ( a , γ ⊤ ) ⊤ \theta=(a,\gamma^\top)^\top θ = ( a , γ ⊤ ) ⊤ and θ ′ = ( a ′ , γ ′ ⊤ ) ⊤ \theta'=(a',\gamma'^\top)^\top θ ′ = ( a ′ , γ ′⊤ ) ⊤ in Θ ~ \widetilde{\Theta} Θ , and write δ = ∥ θ − θ ′ ∥ 2 \delta=\|\theta-\theta'\|_2 δ = ∥ θ − θ ′ ∥ 2 . Define the sign-disagreement set
S ( θ , θ ′ ) = { x ∈ [ 0 , 1 ] d : s θ ( x ) ≥ 0 , s θ ′ ( x ) < 0 } ∪ { x ∈ [ 0 , 1 ] d : s θ ( x ) < 0 , s θ ′ ( x ) ≥ 0 } . S(\theta,\theta')=\{x\in[0,1]^d:s_\theta(x)\geq0,\ s_{\theta'}(x)<0\}\cup\{x\in[0,1]^d:s_\theta(x)<0,\ s_{\theta'}(x)\geq0\}. S ( θ , θ ′ ) = { x ∈ [ 0 , 1 ] d : s θ ( x ) ≥ 0 , s θ ′ ( x ) < 0 } ∪ { x ∈ [ 0 , 1 ] d : s θ ( x ) < 0 , s θ ′ ( x ) ≥ 0 } .
We first show that S ( θ , θ ′ ) S(\theta,\theta') S ( θ , θ ′ ) is contained in a slab around H θ \mathcal{H}_\theta H θ . Note that if x ∈ S ( θ , θ ′ ) x\in S(\theta,\theta') x ∈ S ( θ , θ ′ ) , then it is the case that s θ ( x ) s_\theta(x) s θ ( x ) and s θ ′ ( x ) s_{\theta'}(x) s θ ′ ( x ) have different signs. If s θ ( x ) ≥ 0 s_\theta(x)\geq0 s θ ( x ) ≥ 0 and s θ ′ ( x ) < 0 s_{\theta'}(x)<0 s θ ′ ( x ) < 0 , then ∣ s θ ( x ) ∣ = s θ ( x ) ≤ s θ ( x ) − s θ ′ ( x ) \lvert s_\theta(x)\rvert=s_\theta(x)\leq s_\theta(x)-s_{\theta'}(x) ∣ s θ ( x )∣ = s θ ( x ) ≤ s θ ( x ) − s θ ′ ( x ) . If s θ ( x ) < 0 s_\theta(x)<0 s θ ( x ) < 0 and s θ ′ ( x ) ≥ 0 s_{\theta'}(x)\geq0 s θ ′ ( x ) ≥ 0 , then ∣ s θ ( x ) ∣ = − s θ ( x ) ≤ s θ ′ ( x ) − s θ ( x ) \lvert s_\theta(x)\rvert=-s_\theta(x)\leq s_{\theta'}(x)-s_\theta(x) ∣ s θ ( x )∣ = − s θ ( x ) ≤ s θ ′ ( x ) − s θ ( x ) . Therefore, in both cases, we have that
∣ s θ ( x ) ∣ ≤ ∣ s θ ( x ) − s θ ′ ( x ) ∣ . \lvert s_\theta(x)\rvert\leq\lvert s_\theta(x)-s_{\theta'}(x)\rvert. ∣ s θ ( x )∣ ≤ ∣ s θ ( x ) − s θ ′ ( x )∣ .
Since ∥ x ~ ∥ 2 ≤ d + 1 \|\widetilde{x}\|_2\leq\sqrt{d+1} ∥ x ∥ 2 ≤ d + 1 on [ 0 , 1 ] d [0,1]^d [ 0 , 1 ] d , the Cauchy–Schwarz inequality gives
∣ s θ ( x ) − s θ ′ ( x ) ∣ ≤ d + 1 ∥ θ − θ ′ ∥ 2 = d + 1 δ . \lvert s_\theta(x)-s_{\theta'}(x)\rvert\leq\sqrt{d+1}\|\theta-\theta'\|_2=\sqrt{d+1}\delta. ∣ s θ ( x ) − s θ ′ ( x )∣ ≤ d + 1 ∥ θ − θ ′ ∥ 2 = d + 1 δ .
Therefore, it holds that
S ( θ , θ ′ ) ⊆ { x ∈ [ 0 , 1 ] d : ∣ s θ ( x ) ∣ ≤ d + 1 δ } . (36) S(\theta,\theta')\subseteq\{x\in[0,1]^d:\lvert s_\theta(x)\rvert\leq\sqrt{d+1}\delta\}. \tag*{(36)} S ( θ , θ ′ ) ⊆ { x ∈ [ 0 , 1 ] d : ∣ s θ ( x )∣ ≤ d + 1 δ } . (36)
Next we bound the probability of this slab. Since θ ∈ Θ ~ \theta\in\widetilde{\Theta} θ ∈ Θ , there exists at least one index j j j such that θ ∈ Θ ~ j \theta\in\widetilde{\Theta}_j θ ∈ Θ j . By relabelling coordinates, write x = ( u , z ) ∈ [ 0 , 1 ] d − 1 × [ 0 , 1 ] x=(u,z)\in[0,1]^{d-1}\times[0,1] x = ( u , z ) ∈ [ 0 , 1 ] d − 1 × [ 0 , 1 ] and s θ ( x ) = a + b ⊤ u + c z s_\theta(x)=a+b^\top u+cz s θ ( x ) = a + b ⊤ u + cz , where c = γ j ≥ c ~ 0 c=\gamma_j\geq\widetilde{c}_0 c = γ j ≥ c 0 . Let r > 0 r>0 r > 0 . For fixed u ∈ [ 0 , 1 ] d − 1 u\in[0,1]^{d-1} u ∈ [ 0 , 1 ] d − 1 , the set of z ∈ [ 0 , 1 ] z\in[0,1] z ∈ [ 0 , 1 ] such that ∣ a + b ⊤ u + c z ∣ ≤ r \lvert a+b^\top u+cz\rvert\leq r ∣ a + b ⊤ u + cz ∣ ≤ r is an interval, possibly empty, of length at most 2 r / c 2r/c 2 r / c , and hence at most 2 r / c ~ 0 2r/\widetilde{c}_0 2 r / c 0 . Therefore, by the density upper bound in Assumption 1 and the Fubini’s theorem, we have that
P { ∣ s θ ( X ) ∣ ≤ r } ≤ C g λ d { x ∈ [ 0 , 1 ] d : ∣ s θ ( x ) ∣ ≤ r } ≤ C g ∫ [ 0 , 1 ] d − 1 2 r c ~ 0 d u = 2 C g c ~ 0 r . (37) \mathbb{P}\{\lvert s_\theta(X)\rvert\leq r\}\leq C_g\lambda_d\{x\in[0,1]^d:\lvert s_\theta(x)\rvert\leq r\}
\leq C_g\int_{[0,1]^{d-1}}\frac{2r}{\widetilde{c}_0}\,du
=\frac{2C_g}{\widetilde{c}_0}r. \tag*{(37)} P {∣ s θ ( X )∣ ≤ r } ≤ C g λ d { x ∈ [ 0 , 1 ] d : ∣ s θ ( x )∣ ≤ r } ≤ C g ∫ [ 0 , 1 ] d − 1 c 0 2 r d u = c 0 2 C g r . (37)
Taking r = d + 1 δ r=\sqrt{d+1}\delta r = d + 1 δ gives
P { X ∈ S ( θ , θ ′ ) } ≤ P { ∣ s θ ( X ) ∣ ≤ d + 1 δ } ≤ 2 C g d + 1 c ~ 0 δ . \mathbb{P}\{X\in S(\theta,\theta')\}\leq\mathbb{P}\{\lvert s_\theta(X)\rvert\leq\sqrt{d+1}\delta\}\leq\frac{2C_g\sqrt{d+1}}{\widetilde{c}_0}\delta. P { X ∈ S ( θ , θ ′ )} ≤ P {∣ s θ ( X )∣ ≤ d + 1 δ } ≤ c 0 2 C g d + 1 δ .
Thus the claim holds with C s l a b = 2 C g d + 1 / c ~ 0 C_{\mathrm{slab}}=2C_g\sqrt{d+1}/\widetilde{c}_0 C slab = 2 C g d + 1 / c 0 .
Lemma 20 (Transfer in L 2 ( P ) L_2(P) L 2 ( P ) ). Suppose Assumptions 1-4 hold. Then there exists a constant C t r < ∞ C_{\mathrm{tr}}<\infty C tr < ∞ such that, for every j ∈ { 1 , … , d } j\in\{1,\ldots,d\} j ∈ { 1 , … , d } , every θ , θ ′ ∈ Θ ~ j \theta,\theta'\in\widetilde{\Theta}_j θ , θ ′ ∈ Θ j and every q ∈ Q θ , L q\in\mathcal{Q}_{\theta,L} q ∈ Q θ , L , there exists q ′ ∈ Q θ ′ , L q'\in\mathcal{Q}_{\theta',L} q ′ ∈ Q θ ′ , L satisfying
∥ p θ , q − p θ ′ , q ′ ∥ L 2 ( P ) ≤ C t r ∥ θ − θ ′ ∥ 2 1 / 2 . \|p_{\theta,q}-p_{\theta',q'}\|_{L_2(P)}\leq C_{\mathrm{tr}}\|\theta-\theta'\|_2^{1/2}. ∥ p θ , q − p θ ′ , q ′ ∥ L 2 ( P ) ≤ C tr ∥ θ − θ ′ ∥ 2 1/2 . Proof. The proof follows from a similar argument as the proof for Lemma 17. We only include the difference here.
Fix j ∈ { 1 , … , d } j\in\{1,\ldots,d\} j ∈ { 1 , … , d } , θ , θ ′ ∈ Θ ~ j \theta,\theta'\in\widetilde{\Theta}_j θ , θ ′ ∈ Θ j and q ∈ Q θ , L q\in\mathcal{Q}_{\theta,L} q ∈ Q θ , L . Write δ = ∥ θ − θ ′ ∥ 2 \delta=\|\theta-\theta'\|_2 δ = ∥ θ − θ ′ ∥ 2 . Define the global McShane extension of q q q from H θ \mathcal{H}_\theta H θ to [ 0 , 1 ] d [0,1]^d [ 0 , 1 ] d by
G ( x ) = inf h ∈ H θ { q ( h ) + L ∥ x − h ∥ ν } , q ˉ ( x ) = [ G ( x ) ] [ 0 , 1 ] , x ∈ [ 0 , 1 ] d , G(x)=\inf_{h\in\mathcal{H}_\theta}\{q(h)+L\|x-h\|_\nu\},\qquad\bar q(x)=[G(x)]_{[0,1]},\qquad x\in[0,1]^d, G ( x ) = h ∈ H θ inf { q ( h ) + L ∥ x − h ∥ ν } , q ˉ ( x ) = [ G ( x ) ] [ 0 , 1 ] , x ∈ [ 0 , 1 ] d ,
where [ y ] [ 0 , 1 ] = min { 1 , max { 0 , y } } [y]_{[0,1]}=\min\{1,\max\{0,y\}\} [ y ] [ 0 , 1 ] = min { 1 , max { 0 , y }} . Define the transferred trace on H θ ′ \mathcal{H}_{\theta'} H θ ′ by q ′ ( h ′ ) = q ˉ ( h ′ ) q'(h')=\bar{q}(h') q ′ ( h ′ ) = q ˉ ( h ′ ) for all h ′ ∈ H θ ′ h'\in\mathcal{H}_{\theta'} h ′ ∈ H θ ′ .
Step 1: Feasibility of the transferred trace. By the same argument as in Step 1 of the proof of Lemma 17, with ( θ ∗ , q ∗ ) (\theta^*,q^*) ( θ ∗ , q ∗ ) replaced by ( θ , q ) (\theta,q) ( θ , q ) , the function G G G is L L L -Lipschitz on [ 0 , 1 ] d [0,1]^d [ 0 , 1 ] d . Since the clipping map y ↦ [ y ] [ 0 , 1 ] y\mapsto[y]_{[0,1]} y ↦ [ y ] [ 0 , 1 ] is nonexpansive, q ˉ \bar{q} q ˉ is also L L L -Lipschitz and takes values in [ 0 , 1 ] [0,1] [ 0 , 1 ] . Therefore, its restriction q ′ = q ˉ ∣ H θ ′ q'=\left.\bar{q}\right|_{\mathcal{H}_{\theta'}} q ′ = q ˉ ∣ H θ ′ belongs to Q θ ′ , L \mathcal{Q}_{\theta',L} Q θ ′ , L .
Step 2: Upper-envelope comparison. Define
U θ , q ( x ) = inf h ∈ H θ { q ( h ) + L ∥ x − h ∥ V } , U θ ′ , q ′ ( x ) = inf h ′ ∈ H θ ′ { q ′ ( h ′ ) + L ∥ x − h ′ ∥ V } . U_{\theta,q}(x)=\inf_{h\in\mathcal{H}_{\theta}}\{q(h)+L\lVert x-h\rVert_{\mathcal{V}}\},\qquad U_{\theta',q'}(x)=\inf_{h'\in\mathcal{H}_{\theta'}}\{q'(h')+L\lVert x-h'\rVert_{\mathcal{V}}\}. U θ , q ( x ) = h ∈ H θ inf { q ( h ) + L ∥ x − h ∥ V } , U θ ′ , q ′ ( x ) = h ′ ∈ H θ ′ inf { q ′ ( h ′ ) + L ∥ x − h ′ ∥ V } .
By the same argument as in Step 2 of the proof of Lemma 17, with ( θ ∗ , q ∗ ) (\theta^*,q^*) ( θ ∗ , q ∗ ) replaced by ( θ , q ) (\theta,q) ( θ , q ) . Therefore, for every x ∈ [ 0 , 1 ] d x\in[0,1]^d x ∈ [ 0 , 1 ] d , we can show that
∣ U θ , q ( x ) − U θ ′ , q ′ ( x ) ∣ ≤ 2 L C H δ . \lvert U_{\theta,q}(x)-U_{\theta',q'}(x)\rvert\le2LC_H\delta. ∣ U θ , q ( x ) − U θ ′ , q ′ ( x )∣ ≤ 2 L C H δ .
Step 3: Lower-envelope comparison. Define
M θ , q ( x ) = sup h ∈ H θ { q ( h ) − L ∥ x − h ∥ V } , M θ ′ , q ′ ( x ) = sup h ′ ∈ H θ ′ { q ′ ( h ′ ) − L ∥ x − h ′ ∥ V } . M_{\theta,q}(x)=\sup_{h\in\mathcal{H}_{\theta}}\{q(h)-L\lVert x-h\rVert_{\mathcal{V}}\},\qquad M_{\theta',q'}(x)=\sup_{h'\in\mathcal{H}_{\theta'}}\{q'(h')-L\lVert x-h'\rVert_{\mathcal{V}}\}. M θ , q ( x ) = h ∈ H θ sup { q ( h ) − L ∥ x − h ∥ V } , M θ ′ , q ′ ( x ) = h ′ ∈ H θ ′ sup { q ′ ( h ′ ) − L ∥ x − h ′ ∥ V } .
By the same argument as in Step 3 of the proof of Lemma 17, again with ( θ ∗ , q ∗ ) (\theta^*,q^*) ( θ ∗ , q ∗ ) replaced by ( θ , q ) (\theta,q) ( θ , q ) , we obtain, for every x ∈ [ 0 , 1 ] d x\in[0,1]^d x ∈ [ 0 , 1 ] d ,
∣ M θ , q ( x ) − M θ ′ , q ′ ( x ) ∣ ≤ 2 L C H δ . \lvert M_{\theta,q}(x)-M_{\theta',q'}(x)\rvert\le2LC_H\delta. ∣ M θ , q ( x ) − M θ ′ , q ′ ( x )∣ ≤ 2 L C H δ .
Step 4: L 2 ( P ) L_2(P) L 2 ( P ) bound. Under the same notation as in Lemma 19 , define the sign-disagreement set
S ( θ , θ ′ ) = { x ∈ [ 0 , 1 ] d : s θ ( x ) ≥ 0 , s θ ′ ( x ) < 0 } ∪ { x ∈ [ 0 , 1 ] d : s θ ( x ) < 0 , s θ ′ ( x ) ≥ 0 } . S(\theta,\theta')=\{x\in[0,1]^d:s_\theta(x)\ge0,\ s_{\theta'}(x)<0\}\cup\{x\in[0,1]^d:s_\theta(x)<0,\ s_{\theta'}(x)\ge0\}. S ( θ , θ ′ ) = { x ∈ [ 0 , 1 ] d : s θ ( x ) ≥ 0 , s θ ′ ( x ) < 0 } ∪ { x ∈ [ 0 , 1 ] d : s θ ( x ) < 0 , s θ ′ ( x ) ≥ 0 } .
On S ( θ , θ ′ ) c S(\theta,\theta')^c S ( θ , θ ′ ) c , the two policies use the same type of clipped envelope; either both use the upper envelopes or both use the lower envelopes. Since the maps y ↦ min { 1 , y } y\mapsto\min\{1,y\} y ↦ min { 1 , y } and y ↦ max { 0 , y } y\mapsto\max\{0,y\} y ↦ max { 0 , y } are nonexpansive, Steps 2 and 3 imply that
∣ p θ , q ( x ) − p θ ′ , q ′ ( x ) ∣ ≤ 2 L C H δ , x ∈ S ( θ , θ ′ ) c . \lvert p_{\theta,q}(x)-p_{\theta',q'}(x)\rvert\le2LC_H\delta,\qquad x\in S(\theta,\theta')^c. ∣ p θ , q ( x ) − p θ ′ , q ′ ( x )∣ ≤ 2 L C H δ , x ∈ S ( θ , θ ′ ) c .
On S ( θ , θ ′ ) S(\theta,\theta') S ( θ , θ ′ ) , we use the crude bound ∣ p θ , q ( x ) − p θ ′ , q ′ ( x ) ∣ ≤ 1 \lvert p_{\theta,q}(x)-p_{\theta',q'}(x)\rvert\le1 ∣ p θ , q ( x ) − p θ ′ , q ′ ( x )∣ ≤ 1 . Hence, by Lemma 19 , we have
∥ p θ , q − p θ ′ , q ′ ∥ L 2 ( P ) 2 = E [ ∣ p θ , q ( X ) − p θ ′ , q ′ ( X ) ∣ 2 ] = E [ ∣ p θ , q ( X ) − p θ ′ , q ′ ( X ) ∣ 2 1 { X ∈ S ( θ , θ ′ ) c } ] + E [ ∣ p θ , q ( X ) − p θ ′ , q ′ ( X ) ∣ 2 1 { X ∈ S ( θ , θ ′ ) } ] ≤ ( 2 L C H ) 2 δ 2 + P { X ∈ S ( θ , θ ′ ) } ≤ ( 2 L C H ) 2 δ 2 + C s l a b δ . \begin{aligned}
\lVert p_{\theta,q}-p_{\theta',q'}\rVert_{L_2(P)}^2
&=\mathbb{E}\left[\lvert p_{\theta,q}(X)-p_{\theta',q'}(X)\rvert^2\right]\\
&=\mathbb{E}\left[\lvert p_{\theta,q}(X)-p_{\theta',q'}(X)\rvert^2\mathbf{1}\{X\in S(\theta,\theta')^c\}\right]\\
&\quad+\mathbb{E}\left[\lvert p_{\theta,q}(X)-p_{\theta',q'}(X)\rvert^2\mathbf{1}\{X\in S(\theta,\theta')\}\right]\\
&\le(2LC_H)^2\delta^2+\mathbb{P}\{X\in S(\theta,\theta')\}\\
&\le(2LC_H)^2\delta^2+C_{\mathrm{slab}}\delta.
\end{aligned} ∥ p θ , q − p θ ′ , q ′ ∥ L 2 ( P ) 2 = E [ ∣ p θ , q ( X ) − p θ ′ , q ′ ( X ) ∣ 2 ] = E [ ∣ p θ , q ( X ) − p θ ′ , q ′ ( X ) ∣ 2 1 { X ∈ S ( θ , θ ′ ) c } ] + E [ ∣ p θ , q ( X ) − p θ ′ , q ′ ( X ) ∣ 2 1 { X ∈ S ( θ , θ ′ )} ] ≤ ( 2 L C H ) 2 δ 2 + P { X ∈ S ( θ , θ ′ )} ≤ ( 2 L C H ) 2 δ 2 + C slab δ .
Moreover, by the definition of Θ ~ \widetilde{\Theta} Θ , it holds that δ ≤ diam ( Θ ~ ) < ∞ \delta\le\operatorname{diam}(\widetilde{\Theta})<\infty δ ≤ diam ( Θ ) < ∞ . Therefore, we have that
( 2 L C H ) 2 δ 2 + C s l a b δ ≤ { ( 2 L C H ) 2 diam ( Θ ~ ) + C s l a b } δ = : C t r δ . (2LC_H)^2\delta^2+C_{\mathrm{slab}}\delta\le\{(2LC_H)^2\operatorname{diam}(\widetilde{\Theta})+C_{\mathrm{slab}}\}\delta=:C_{\mathrm{tr}}\delta. ( 2 L C H ) 2 δ 2 + C slab δ ≤ {( 2 L C H ) 2 diam ( Θ ) + C slab } δ =: C tr δ .
Taking square roots gives
∥ p θ , q − p θ ′ , q ′ ∥ L 2 ( P ) ≤ C t r δ 1 / 2 = C t r ∥ θ − θ ′ ∥ 2 1 / 2 . \lVert p_{\theta,q}-p_{\theta',q'}\rVert_{L_2(P)}\le C_{\mathrm{tr}}\delta^{1/2}=C_{\mathrm{tr}}\lVert\theta-\theta'\rVert_2^{1/2}. ∥ p θ , q − p θ ′ , q ′ ∥ L 2 ( P ) ≤ C tr δ 1/2 = C tr ∥ θ − θ ′ ∥ 2 1/2 .
Lemma 21 (Uniform trace entropy). Suppose Assumption 4 holds. Then there exists a constant C Q < ∞ C_Q < \infty C Q < ∞ , depending only on d d d , ν \nu ν , L L L , A 0 A_0 A 0 , B 0 B_0 B 0 , c ~ 0 \widetilde{c}_0 c 0 , C ~ 0 \widetilde{C}_0 C 0 and ρ ~ 0 \widetilde{\rho}_0 ρ 0 , such that, for every fixed θ ∈ Θ ~ \theta\in\widetilde{\Theta} θ ∈ Θ and every ϵ ∈ ( 0 , 1 ) \epsilon\in(0,1) ϵ ∈ ( 0 , 1 ) ,
log N ( ϵ , Q θ , L , ∥ ⋅ ∥ ∞ ) ≤ C Q ϵ − ( d − 1 ) log ( C Q / ϵ ) . \log N(\epsilon,\mathcal{Q}_{\theta,L},\|\cdot\|_\infty) \le C_Q\epsilon^{-(d-1)}\log(C_Q/\epsilon). log N ( ϵ , Q θ , L , ∥ ⋅ ∥ ∞ ) ≤ C Q ϵ − ( d − 1 ) log ( C Q / ϵ ) .
Proof. Fix θ ∈ Θ ~ \theta\in\widetilde{\Theta} θ ∈ Θ . If L = 0 L=0 L = 0 , then Q θ , 0 \mathcal{Q}_{\theta,0} Q θ , 0 consists of constant functions with values in [ 0 , 1 ] [0,1] [ 0 , 1 ] , and the result follows by discretising this interval. Suppose henceforth that L > 0 L>0 L > 0 . Since H θ ⊆ [ 0 , 1 ] d \mathcal{H}_\theta\subseteq[0,1]^d H θ ⊆ [ 0 , 1 ] d is contained in the affine hyperplane { x : s θ ( x ) = 0 } \{x:s_\theta(x)=0\} { x : s θ ( x ) = 0 } , it is a bounded subset of a ( d − 1 ) (d-1) ( d − 1 ) -dimensional affine space. By the Euclidean covering bound for bounded subsets of finite-dimensional affine spaces (e.g. Proposition 4.2.12 in Vershynin, [50 ] ), for every r ∈ ( 0 , 1 ] r \in(0,1] r ∈ ( 0 , 1 ] , there exists an ( r / C d , ν ) (r/C_{d,\nu}) ( r / C d , ν ) -net N r = { h 1 , … , h N } ⊆ H θ \mathcal{N}_r=\{h_1,\ldots,h_N\}\subseteq\mathcal{H}_\theta N r = { h 1 , … , h N } ⊆ H θ with respect to ∥ ⋅ ∥ 2 \|\cdot\|_2 ∥ ⋅ ∥ 2 such that N ≤ C Q r − ( d − 1 ) N\leq C_Qr^{-(d-1)} N ≤ C Q r − ( d − 1 ) , where C Q < ∞ C_Q<\infty C Q < ∞ depends only on d d d and ν \nu ν . By Lemma 30 , we have that ∥ x − y ∥ ν ≤ C d , ν ∥ x − y ∥ 2 \|x-y\|_\nu\leq C_{d,\nu}\|x-y\|_2 ∥ x − y ∥ ν ≤ C d , ν ∥ x − y ∥ 2 for all x , y ∈ R d x,y\in\mathbb{R}^d x , y ∈ R d . Hence N r \mathcal{N}_r N r is an r r r -net of H θ \mathcal{H}_\theta H θ with respect to ∥ ⋅ ∥ ν \|\cdot\|_\nu ∥ ⋅ ∥ ν . For ϵ ∈ ( 0 , 1 ) \epsilon\in(0,1) ϵ ∈ ( 0 , 1 ) , we choose r = 1 ∧ { ϵ / ( 8 L ) } r=1\wedge\{\epsilon/(8L)\} r = 1 ∧ { ϵ / ( 8 L )} .
We next discretise [ 0 , 1 ] [0,1] [ 0 , 1 ] using a grid G ϵ ⊂ [ 0 , 1 ] \mathcal{G}_\epsilon\subset[0,1] G ϵ ⊂ [ 0 , 1 ] . Let G ϵ ⊂ [ 0 , 1 ] \mathcal{G}_\epsilon\subset[0,1] G ϵ ⊂ [ 0 , 1 ] be a finite grid with grid size at most ϵ / 4 \epsilon/4 ϵ /4 , meaning that for every y ∈ [ 0 , 1 ] y\in[0,1] y ∈ [ 0 , 1 ] , there exists g ∈ G ϵ g\in\mathcal{G}_\epsilon g ∈ G ϵ such that ∣ y − g ∣ ≤ ϵ / 4 |y-g|\leq\epsilon/4 ∣ y − g ∣ ≤ ϵ /4 . Moreover, we have that ∣ G ϵ ∣ ≤ C / ϵ |\mathcal{G}_\epsilon|\leq C/\epsilon ∣ G ϵ ∣ ≤ C / ϵ , where C < ∞ C<\infty C < ∞ is a universal constant.
For each q ∈ Q θ , L q\in\mathcal{Q}_{\theta,L} q ∈ Q θ , L , choose v j ∈ G ϵ v_j\in\mathcal{G}_\epsilon v j ∈ G ϵ such that ∣ v j − q ( h j ) ∣ ≤ ϵ / 4 |v_j-q(h_j)|\leq\epsilon/4 ∣ v j − q ( h j ) ∣ ≤ ϵ /4 for j = 1 , … , N j=1,\ldots,N j = 1 , … , N . Define
q ~ ( h ) = [ min 1 ≤ j ≤ N { v j + L ∥ h − h j ∥ ν } ] [ 0 , 1 ] , h ∈ H θ . \widetilde{q}(h)=\left[\min_{1\leq j\leq N}\{v_j+L\|h-h_j\|_\nu\}\right]_{[0,1]},\qquad h\in\mathcal{H}_\theta. q ( h ) = [ 1 ≤ j ≤ N min { v j + L ∥ h − h j ∥ ν } ] [ 0 , 1 ] , h ∈ H θ .
Note that q ~ \widetilde{q} q is L L L -Lipschitz and takes values in [ 0 , 1 ] [0,1] [ 0 , 1 ] .
We now show that q ~ \widetilde{q} q approximates q q q uniformly. For any h ∈ H θ h\in\mathcal{H}_\theta h ∈ H θ , choose h j ∈ N r h_j\in\mathcal{N}_r h j ∈ N r such that ∥ h − h j ∥ ν ≤ r \|h-h_j\|_\nu\leq r ∥ h − h j ∥ ν ≤ r . Then we have that
q ~ ( h ) ≤ v j + L ∥ h − h j ∥ ν ≤ q ( h j ) + ϵ 4 + L r ≤ q ( h ) + ϵ 2 , \widetilde{q}(h)\leq v_j+L\|h-h_j\|_\nu\leq q(h_j)+\frac{\epsilon}{4}+Lr\leq q(h)+\frac{\epsilon}{2}, q ( h ) ≤ v j + L ∥ h − h j ∥ ν ≤ q ( h j ) + 4 ϵ + L r ≤ q ( h ) + 2 ϵ ,
where the last inequality follows from 2 L r ≤ ϵ / 4 2Lr\leq\epsilon/4 2 L r ≤ ϵ /4 . For the lower bound, for every j j j ,
v j + L ∥ h − h j ∥ ν ≥ q ( h j ) − ϵ 4 + L ∥ h − h j ∥ ν ≥ q ( h ) − ϵ 4 , v_j+L\|h-h_j\|_\nu\geq q(h_j)-\frac{\epsilon}{4}+L\|h-h_j\|_\nu\geq q(h)-\frac{\epsilon}{4}, v j + L ∥ h − h j ∥ ν ≥ q ( h j ) − 4 ϵ + L ∥ h − h j ∥ ν ≥ q ( h ) − 4 ϵ ,
where the last inequality follows since q q q is L L L -Lipschitz over H θ \mathcal{H}_\theta H θ . Taking the infimum over j j j and then clipping to [ 0 , 1 ] [0,1] [ 0 , 1 ] cannot decrease the value below q ( h ) − ϵ / 4 q(h)-\epsilon/4 q ( h ) − ϵ /4 , since q ( h ) ∈ [ 0 , 1 ] q(h)\in[0,1] q ( h ) ∈ [ 0 , 1 ] . Thus we have that
q ~ ( h ) ≥ q ( h ) − ϵ 4 . \widetilde{q}(h)\geq q(h)-\frac{\epsilon}{4}. q ( h ) ≥ q ( h ) − 4 ϵ .
Combining the results, it then holds that ∥ q ~ − q ∥ ∞ ≤ ϵ \|\widetilde{q}-q\|_\infty\leq\epsilon ∥ q − q ∥ ∞ ≤ ϵ .
Note that the number of possible choices of ( v 1 , … , v N ) (v_1,\ldots,v_N) ( v 1 , … , v N ) is at most ( C / ϵ ) N (C/\epsilon)^N ( C / ϵ ) N . Since each coordinate v j v_j v j is chosen independently from the grid G ϵ \mathcal{G}_\epsilon G ϵ , and ∣ G ϵ ∣ ≤ C / ϵ |\mathcal{G}_\epsilon|\leq C/\epsilon ∣ G ϵ ∣ ≤ C / ϵ , the number of possible vectors ( v 1 , … , v N ) (v_1,\ldots,v_N) ( v 1 , … , v N ) is at most ∣ G ϵ ∣ N ≤ ( C / ϵ ) N |\mathcal{G}_\epsilon|^N\leq(C/\epsilon)^N ∣ G ϵ ∣ N ≤ ( C / ϵ ) N . Consequently, it holds that
log N ( ϵ , Q θ , L , ∥ ⋅ ∥ ∞ ) ≤ N log ( C / ϵ ) ≤ C Q ϵ − ( d − 1 ) log ( C Q / ϵ ) . \log N(\epsilon,\mathcal{Q}_{\theta,L},\|\cdot\|_\infty)\leq N\log(C/\epsilon)\leq C_Q\epsilon^{-(d-1)}\log(C_Q/\epsilon). log N ( ϵ , Q θ , L , ∥ ⋅ ∥ ∞ ) ≤ N log ( C / ϵ ) ≤ C Q ϵ − ( d − 1 ) log ( C Q / ϵ ) .
:::
Lemma 22 (L 2 ( P ) L_2(P) L 2 ( P ) bracketing entropy). Suppose Assumptions 1–4 hold and d ≥ 2 d\geq2 d ≥ 2 . Let
F L = { x ↦ s θ ∗ ( x ) p 0 , q ( x ) : θ ∈ Θ ~ , q ∈ Q θ , L } , \mathcal{F}_L=\{x\mapsto s_{\theta^*}(x)p_{0,q}(x):\theta\in\widetilde{\Theta},\ q\in\mathcal{Q}_{\theta,L}\}, F L = { x ↦ s θ ∗ ( x ) p 0 , q ( x ) : θ ∈ Θ , q ∈ Q θ , L } , where s θ ∗ ( x ) = x T θ ∗ s_{\theta^*}(x)=x^\mathsf{T}\theta^* s θ ∗ ( x ) = x T θ ∗ . Then there exists a constant C F < ∞ C_F<\infty C F < ∞ such that, for every ϵ ∈ ( 0 , 1 ) \epsilon\in(0,1) ϵ ∈ ( 0 , 1 ) ,
log N [ ] ( ϵ , F L , L 2 ( P ) ) ≤ C F ϵ − ( d − 1 ) log ( C F / ϵ ) , \log N_{[]}(\epsilon,\mathcal{F}_L,L_2(P))\leq C_F\epsilon^{-(d-1)}\log(C_F/\epsilon), log N [ ] ( ϵ , F L , L 2 ( P )) ≤ C F ϵ − ( d − 1 ) log ( C F / ϵ ) , where N [ ] ( ϵ , F L , L 2 ( P ) ) N_{[]}(\epsilon,\mathcal{F}_L,L_2(P)) N [ ] ( ϵ , F L , L 2 ( P )) denotes the L 2 ( P ) L_2(P) L 2 ( P ) bracketing number.
Proof. Write m = d − 1 m=d-1 m = d − 1 and define B ∗ = sup x ∈ [ 0 , 1 ] d ∣ s θ ∗ ( x ) ∣ B_{*}=\sup_{x\in[0,1]^{d}}|s_{\theta^{*}}(x)| B ∗ = sup x ∈ [ 0 , 1 ] d ∣ s θ ∗ ( x ) ∣ . By Assumption 4, it holds that B ∗ < ∞ B_{*}<\infty B ∗ < ∞ . If B ∗ = 0 B_{*}=0 B ∗ = 0 , then every function in F L \mathcal{F}_{L} F L is identically zero and the claim is trivial. Suppose B ∗ > 0 B_{*}>0 B ∗ > 0 and set B ˉ = 1 + B ∗ \bar{B}=1+B_{*} B ˉ = 1 + B ∗ .
For each r ∈ { 1 , … , d } r\in\{1,\ldots,d\} r ∈ { 1 , … , d } , define
F L , r = { x ↦ s θ ∗ ( x ) p θ , q ( x ) : θ ∈ Θ ~ r , q ∈ Q θ , L } . \mathcal{F}_{L,r}=\{x\mapsto s_{\theta^{*}}(x)p_{\theta,q}(x):\theta\in\widetilde{\Theta}_{r},\ q\in Q_{\theta,L}\}. F L , r = { x ↦ s θ ∗ ( x ) p θ , q ( x ) : θ ∈ Θ r , q ∈ Q θ , L } .
Then F L = ⋃ r = 1 d F L , r \mathcal{F}_{L}=\bigcup_{r=1}^{d}\mathcal{F}_{L,r} F L = ⋃ r = 1 d F L , r . We first bound the bracketing number of each fixed chart F L , r \mathcal{F}_{L,r} F L , r .
Fix r ∈ { 1 , … , d } r\in\{1,\ldots,d\} r ∈ { 1 , … , d } and ϵ ∈ ( 0 , 1 ) \epsilon\in(0,1) ϵ ∈ ( 0 , 1 ) . Let δ θ = c 1 ϵ 2 \delta_{\theta}=c_{1}\epsilon^{2} δ θ = c 1 ϵ 2 , where c 1 > 0 c_{1}>0 c 1 > 0 will be chosen sufficiently small. Let { θ 1 , … , θ M } \{\theta_{1},\ldots,\theta_{M}\} { θ 1 , … , θ M } be a δ θ \delta_{\theta} δ θ -net of Θ ~ r \widetilde{\Theta}_{r} Θ r with respect to ∥ ⋅ ∥ 2 \|\cdot\|_{2} ∥ ⋅ ∥ 2 . Since Θ ~ r ⊂ R d + 1 \widetilde{\Theta}_{r}\subset\mathbb{R}^{d+1} Θ r ⊂ R d + 1 is bounded by the definition of Θ ~ r \widetilde{\Theta}_{r} Θ r , we have that
log M ≤ ( d + 1 ) log ( C ′ / δ θ ) ≤ C log ( C / ϵ ) . \log M\leq(d+1)\log(C'/\delta_{\theta})\leq C\log(C/\epsilon). log M ≤ ( d + 1 ) log ( C ′ / δ θ ) ≤ C log ( C / ϵ ) .
where C C C is a constant depending on d d d .
For each fixed representative boundary H θ j \mathcal{H}_{\theta_{j}} H θ j , Lemma 21 gives a finite c 2 ϵ / B ˉ c_{2}\epsilon/\bar{B} c 2 ϵ / B ˉ -cover of Q θ j , L Q_{\theta_{j},L} Q θ j , L in the sup-norm, where c 2 > 0 c_{2}>0 c 2 > 0 is chosen sufficiently small. Thus there exists a finite set Q j ⊂ Q θ j , L Q_{j}\subset Q_{\theta_{j},L} Q j ⊂ Q θ j , L such that, for every q ∘ ∈ Q θ j , L q^{\circ}\in Q_{\theta_{j},L} q ∘ ∈ Q θ j , L , there exists q ~ ∈ Q j \widetilde{q}\in Q_{j} q ∈ Q j satisfying ∥ q ∘ − q ~ ∥ ∞ ≤ c 2 ϵ / B ˉ \|q^{\circ}-\widetilde{q}\|_{\infty}\leq c_{2}\epsilon/\bar{B} ∥ q ∘ − q ∥ ∞ ≤ c 2 ϵ / B ˉ . Moreover, it holds that
log ∣ Q j ∣ ≤ C ϵ − m log ( C / ϵ ) . \log|Q_{j}|\leq C\epsilon^{-m}\log(C/\epsilon). log ∣ Q j ∣ ≤ C ϵ − m log ( C / ϵ ) .
Now fix any f ∈ F L , r f\in\mathcal{F}_{L,r} f ∈ F L , r , and we can write f ( x ) = s θ ∗ ( x ) p θ , q ( x ) f(x)=s_{\theta^{*}}(x)p_{\theta,q}(x) f ( x ) = s θ ∗ ( x ) p θ , q ( x ) , for some θ ∈ Θ ~ r \theta\in\widetilde{\Theta}_{r} θ ∈ Θ r and q ∈ Q θ , L q\in Q_{\theta,L} q ∈ Q θ , L . Choose θ j \theta_{j} θ j such that ∥ θ − θ j ∥ 2 ≤ δ θ \|\theta-\theta_{j}\|_{2}\leq\delta_{\theta} ∥ θ − θ j ∥ 2 ≤ δ θ . Since θ \theta θ and θ j \theta_{j} θ j lie in the same chart Θ ~ r \widetilde{\Theta}_{r} Θ r , by the construction in Steps 2,3 and 4 of the proof of Lemma 20 , there exists q ∘ ∈ Q θ j , L q^{\circ}\in Q_{\theta_{j},L} q ∘ ∈ Q θ j , L such that, for all x ∈ [ 0 , 1 ] d x\in[0,1]^{d} x ∈ [ 0 , 1 ] d ,
∣ p θ , q ( x ) − p θ j , q ∘ ( x ) ∣ ≤ 2 L C H δ θ + 1 S ( θ , θ j ) ( x ) ≤ 2 L C H δ θ + 1 A j ( x ) , (38) |p_{\theta,q}(x)-p_{\theta_{j},q^{\circ}}(x)|\leq2LC_{H}\delta_{\theta}+\mathbb{1}_{S(\theta,\theta_{j})}(x)\leq2LC_{H}\delta_{\theta}+\mathbb{1}_{A_{j}}(x), \tag*{(38)} ∣ p θ , q ( x ) − p θ j , q ∘ ( x ) ∣ ≤ 2 L C H δ θ + 1 S ( θ , θ j ) ( x ) ≤ 2 L C H δ θ + 1 A j ( x ) , (38)
where
S ( θ , θ j ) = { x : s θ ( x ) ≥ 0 , s θ j ( x ) < 0 } ∪ { x : s θ ( x ) < 0 , s θ j ( x ) ≥ 0 } S(\theta,\theta_{j})=\{x:s_{\theta}(x)\geq0,\ s_{\theta_{j}}(x)<0\}\cup\{x:s_{\theta}(x)<0,\ s_{\theta_{j}}(x)\geq0\} S ( θ , θ j ) = { x : s θ ( x ) ≥ 0 , s θ j ( x ) < 0 } ∪ { x : s θ ( x ) < 0 , s θ j ( x ) ≥ 0 }
is the disagreement set,
A j = { x : ∣ s θ j ( x ) ∣ ≤ d + 1 δ θ } , A_{j}=\{x:|s_{\theta_{j}}(x)|\leq\sqrt{d+1}\delta_{\theta}\}, A j = { x : ∣ s θ j ( x ) ∣ ≤ d + 1 δ θ } ,
and the last inequality follows as S ( θ j , θ ) ⊆ A j S(\theta_{j},\theta)\subseteq A_{j} S ( θ j , θ ) ⊆ A j by (36) in the proof of Lemma 19 .
Choose q ~ ∈ Q j \widetilde{q}\in Q_{j} q ∈ Q j such that ∥ q ∘ − q ~ ∥ ∞ ≤ c 2 ϵ / B ˉ \|q^{\circ}-\widetilde{q}\|_{\infty}\leq c_{2}\epsilon/\bar{B} ∥ q ∘ − q ∥ ∞ ≤ c 2 ϵ / B ˉ . Since the envelope map is nonexpansive in the boundary trace, we have that
∥ p θ j , q ∘ − p θ j , q ~ ∥ ∞ ≤ c 2 ϵ / B ˉ . (39) \|p_{\theta_{j},q^{\circ}}-p_{\theta_{j},\widetilde{q}}\|_{\infty}\leq c_{2}\epsilon/\bar{B}. \tag*{(39)} ∥ p θ j , q ∘ − p θ j , q ∥ ∞ ≤ c 2 ϵ / B ˉ . (39)
Therefore, by (38) , (39) and the triangle inequality, for every x ∈ [ 0 , 1 ] d x\in[0,1]^{d} x ∈ [ 0 , 1 ] d ,
∣ p θ , q ( x ) − p θ j , q ~ ( x ) ∣ ≤ C δ θ + c 2 ϵ / B ˉ + 1 A j ( x ) . |p_{\theta,q}(x)-p_{\theta_{j},\widetilde{q}}(x)|\leq C\delta_{\theta}+c_{2}\epsilon/\bar{B}+\mathbb{1}_{A_{j}}(x). ∣ p θ , q ( x ) − p θ j , q ( x ) ∣ ≤ C δ θ + c 2 ϵ / B ˉ + 1 A j ( x ) .
Since ∣ s θ ∗ ( x ) ∣ ≤ B ∗ ≤ B ˉ |s_{\theta^{*}}(x)|\leq B_{*}\leq\bar{B} ∣ s θ ∗ ( x ) ∣ ≤ B ∗ ≤ B ˉ , this implies
∣ s θ ∗ ( x ) { p θ , q ( x ) − p θ j , q ~ ( x ) } ∣ ≤ B ˉ { C δ θ + c 2 ϵ / B ˉ + 1 A j ( x ) } . |s_{\theta^{*}}(x)\{p_{\theta,q}(x)-p_{\theta_{j},\widetilde{q}}(x)\}|\leq\bar{B}\{C\delta_{\theta}+c_{2}\epsilon/\bar{B}+\mathbb{1}_{A_{j}}(x)\}. ∣ s θ ∗ ( x ) { p θ , q ( x ) − p θ j , q ( x )} ∣ ≤ B ˉ { C δ θ + c 2 ϵ / B ˉ + 1 A j ( x )} .
For each pair ( j , q ~ ) (j,\widetilde{q}) ( j , q ) , define f j , q ~ ( x ) = s θ ∗ ( x ) p θ j , q ~ ( x ) f_{j,\widetilde{q}}(x)=s_{\theta^{*}}(x)p_{\theta_{j},\widetilde{q}}(x) f j , q ( x ) = s θ ∗ ( x ) p θ j , q ( x ) . Define the bracket
ℓ j , q ~ ( x ) = f j , q ~ ( x ) − B ˉ { C δ θ + c 2 ϵ / B ˉ + 1 A j ( x ) } , \ell_{j,\widetilde{q}}(x)=f_{j,\widetilde{q}}(x)-\bar{B}\{C\delta_{\theta}+c_{2}\epsilon/\bar{B}+\mathbb{1}_{A_{j}}(x)\}, ℓ j , q ( x ) = f j , q ( x ) − B ˉ { C δ θ + c 2 ϵ / B ˉ + 1 A j ( x )} ,
and
u j , q ~ ( x ) = f j , q ~ ( x ) + B ˉ { C δ θ + c 2 ϵ / B ˉ + 1 A j ( x ) } . u_{j,\widetilde{q}}(x)=f_{j,\widetilde{q}}(x)+\bar{B}\{C\delta_{\theta}+c_{2}\epsilon/\bar{B}+\mathbb{1}_{A_{j}}(x)\}. u j , q ( x ) = f j , q ( x ) + B ˉ { C δ θ + c 2 ϵ / B ˉ + 1 A j ( x )} .
We have that
ℓ j , q ~ ( x ) ≤ f ( x ) ≤ u j , q ~ ( x ) \ell_{j,\widetilde{q}}(x)\leq f(x)\leq u_{j,\widetilde{q}}(x) ℓ j , q ( x ) ≤ f ( x ) ≤ u j , q ( x )
for every x ∈ [ 0 , 1 ] d x \in[0,1]^d x ∈ [ 0 , 1 ] d . Moreover, the L 2 ( P ) L_2(P) L 2 ( P ) width of this bracket satisfies
∥ u j , q ~ − ℓ j , q ~ ∥ L 2 ( P ) ≤ C B ‾ { δ θ + ε / B ‾ + P ( A j ) 1 / 2 } ≤ C B ‾ { δ θ + ε / B ‾ + C 1 / 2 δ 0 1 / 2 } . \lVert u_{j,\widetilde q}-\ell_{j,\widetilde q}\rVert_{L_2(P)} \le C\overline{B}\left\{\delta_\theta+\varepsilon/\overline{B}+\mathbb{P}(A_j)^{1/2}\right\} \le C\overline{B}\left\{\delta_\theta+\varepsilon/\overline{B}+C^{1/2}\delta_0^{1/2}\right\}. ∥ u j , q − ℓ j , q ∥ L 2 ( P ) ≤ C B { δ θ + ε / B + P ( A j ) 1/2 } ≤ C B { δ θ + ε / B + C 1/2 δ 0 1/2 } .
where the last inequality follows as P ( A j ) ≤ C δ θ \mathbb{P}(A_j) \le C\delta_\theta P ( A j ) ≤ C δ θ by the same calculations as (37) in the proof of Lemma 19 .
Since δ θ = c 1 ε 2 \delta_\theta=c_1\varepsilon^2 δ θ = c 1 ε 2 , choosing c 1 , c 2 > 0 c_1,c_2>0 c 1 , c 2 > 0 sufficiently small gives ∥ u j , q ~ − ℓ j , q ~ ∥ L 2 ( P ) ≤ ε \lVert u_{j,\widetilde q}-\ell_{j,\widetilde q}\rVert_{L_2(P)} \le\varepsilon ∥ u j , q − ℓ j , q ∥ L 2 ( P ) ≤ ε .
Thus the brackets indexed by ( j , q ~ ) (j,\widetilde q) ( j , q ) form an L 2 ( P ) L_2(P) L 2 ( P ) ε \varepsilon ε -bracketing cover of F L , r \mathcal{F}_{L,r} F L , r , where the number of brackets is at most M max 1 ≤ j ≤ M ∣ Q j ∣ M\max_{1\le j\le M}|Q_j| M max 1 ≤ j ≤ M ∣ Q j ∣ . Therefore, for every fixed r ∈ { 1 , … , d } r \in\{1,\ldots,d\} r ∈ { 1 , … , d } , we have that
log N [ ] ( ε , F L , r , L 2 ( P ) ) ≤ C log ( C / ε ) + C ε − m log ( C / ε ) ≤ C F ε − m log ( C F / ε ) . \log N_{[]}(\varepsilon,\mathcal{F}_{L,r},L_2(P)) \le C\log(C/\varepsilon)+C\varepsilon^{-m}\log(C/\varepsilon) \le C_F\varepsilon^{-m}\log(C_F/\varepsilon). log N [ ] ( ε , F L , r , L 2 ( P )) ≤ C log ( C / ε ) + C ε − m log ( C / ε ) ≤ C F ε − m log ( C F / ε ) .
Since F L = ⋃ r = 1 d F L , r \mathcal{F}_L=\bigcup_{r=1}^d\mathcal{F}_{L,r} F L = ⋃ r = 1 d F L , r , the union of the chart-wise bracketing covers is an ε \varepsilon ε -bracketing cover of F L \mathcal{F}_L F L . Hence,
N [ ] ( ε , F L , L 2 ( P ) ) ≤ ∑ r = 1 d N [ ] ( ε , F L , r , L 2 ( P ) ) . N_{[]}(\varepsilon,\mathcal{F}_L,L_2(P)) \le\sum_{r=1}^d N_{[]}(\varepsilon,\mathcal{F}_{L,r},L_2(P)). N [ ] ( ε , F L , L 2 ( P )) ≤ r = 1 ∑ d N [ ] ( ε , F L , r , L 2 ( P )) .
Taking logarithms and absorbing log ( d ) \log(d) log ( d ) into the constant prove the claim.
Lemma 23 (Control of Δ n L \Delta_n^L Δ n L ). Suppose Assumptions 1-4 hold. Let
F L = { x ↦ s θ ∗ ( x ) p θ , q ( x ) : θ ∈ Θ ~ , q ∈ Q θ , L } , \mathcal{F}_L=\{x\mapsto s_{\theta^*}(x)p_{\theta,q}(x):\theta\in\widetilde{\Theta},\ q\in Q_{\theta,L}\}, F L = { x ↦ s θ ∗ ( x ) p θ , q ( x ) : θ ∈ Θ , q ∈ Q θ , L } , where s θ ∗ ( x ) = x ~ ⊤ θ ∗ s_{\theta^*}(x)=\widetilde{x}^{\top}\theta^* s θ ∗ ( x ) = x ⊤ θ ∗ . Define
Δ n L = sup θ ∈ Θ ~ , q ∈ Q θ , L ∣ 1 n ∑ i = 1 n s θ ∗ ( X i ) p θ , q ( X i ) − E { s θ ∗ ( X ) p θ , q ( X ) } ∣ . \Delta_n^L=\sup_{\theta\in\widetilde{\Theta},\ q\in Q_{\theta,L}}\left|\frac{1}{n}\sum_{i=1}^n s_{\theta^*}(X_i)p_{\theta,q}(X_i)-\mathbb{E}\{s_{\theta^*}(X)p_{\theta,q}(X)\}\right|. Δ n L = θ ∈ Θ , q ∈ Q θ , L sup n 1 i = 1 ∑ n s θ ∗ ( X i ) p θ , q ( X i ) − E { s θ ∗ ( X ) p θ , q ( X )} . Then there exists a constant C < ∞ C<\infty C < ∞ , depending only on d d d , L L L , A ~ 0 \widetilde{A}_0 A 0 , B ~ 0 \widetilde{B}_0 B 0 , c ~ 0 \widetilde{c}_0 c 0 , C ~ 0 \widetilde{C}_0 C 0 , ρ ~ 0 \widetilde{\rho}_0 ρ 0 , c g c_g c g , C g C_g C g , such that, for every n ≥ 2 n\ge2 n ≥ 2 and every η ∈ ( 0 , 1 ) \eta\in(0,1) η ∈ ( 0 , 1 ) , with probability at least 1 − η 1-\eta 1 − η ,
Δ n L ≤ C ρ n , d − 1 ( η ) , \Delta_n^L\le C\rho_{n,d-1}(\eta), Δ n L ≤ C ρ n , d − 1 ( η ) , where, for m = d − 1 m=d-1 m = d − 1 ,
ρ n , m ( η ) = { n − 1 / 2 log ( e n / η ) , m = 1 , n − 1 / 2 log 3 / 2 ( e n / η ) , m = 2 , n − 1 / m log ( e n / η ) , m ≥ 3. \rho_{n,m}(\eta)=
\begin{cases}
n^{-1/2}\log(en/\eta), & m=1,\\
n^{-1/2}\log^{3/2}(en/\eta), & m=2,\\
n^{-1/m}\log(en/\eta), & m\ge3.
\end{cases} ρ n , m ( η ) = ⎩ ⎨ ⎧ n − 1/2 log ( e n / η ) , n − 1/2 log 3/2 ( e n / η ) , n − 1/ m log ( e n / η ) , m = 1 , m = 2 , m ≥ 3. Proof. Write m = d − 1 m=d-1 m = d − 1 and θ ∗ = ( a ∗ , γ ∗ ⊤ ) ⊤ \theta^*=(a^*,\gamma^{*\top})^\top θ ∗ = ( a ∗ , γ ∗ ⊤ ) ⊤ . For every f ∈ F L f\in\mathcal{F}_L f ∈ F L , we can rewrite it as f ( x ) = s θ ∗ ( x ) p θ , q ( x ) f(x)=s_{\theta^*}(x)p_{\theta,q}(x) f ( x ) = s θ ∗ ( x ) p θ , q ( x ) for some θ ∈ Θ ~ \theta\in\widetilde{\Theta} θ ∈ Θ and q ∈ Q θ , L q\in Q_{\theta,L} q ∈ Q θ , L . Since 0 ≤ p θ , q ≤ 1 0\le p_{\theta,q}\le1 0 ≤ p θ , q ≤ 1 and θ ∗ ∈ Θ \theta^*\in\Theta θ ∗ ∈ Θ , Assumption 4 gives
∣ s θ ∗ ( x ) ∣ ≤ ∣ a ∗ ∣ + ∥ γ ∗ ∥ 2 ∥ x ∥ 2 ≤ A 0 + B 0 d = : B . |s_{\theta^*}(x)|\le|a^*|+\lVert\gamma^*\rVert_2\lVert x\rVert_2\le A_0+B_0\sqrt{d}=:B. ∣ s θ ∗ ( x ) ∣ ≤ ∣ a ∗ ∣ + ∥ γ ∗ ∥ 2 ∥ x ∥ 2 ≤ A 0 + B 0 d =: B . Therefore, we have that ∥ f ∥ ∞ ≤ ∥ s θ ∗ ∥ ∞ ≤ B \lVert f\rVert_\infty\le\lVert s_{\theta^*}\rVert_\infty\le B ∥ f ∥ ∞ ≤ ∥ s θ ∗ ∥ ∞ ≤ B for every f ∈ F L f\in\mathcal{F}_L f ∈ F L .
By the bracketing maximal inequality for empirical processes (e.g. Theorem 2.14.2 in [49 ] ), we have that
E Δ n L ≤ C 1 inf α ∈ ( 0 , 1 ) { α + 1 n ∫ α 1 1 + log N [ ] ( ε , F L , L 2 ( P ) ) d ε } . \mathbb{E}\Delta_n^L\le C_1\inf_{\alpha\in(0,1)}\left\{\alpha+\frac{1}{\sqrt{n}}\int_\alpha^1\sqrt{1+\log N_{[]}(\varepsilon,\mathcal{F}_L,L_2(P))}\,\mathrm{d}\varepsilon\right\}. E Δ n L ≤ C 1 α ∈ ( 0 , 1 ) inf { α + n 1 ∫ α 1 1 + log N [ ] ( ε , F L , L 2 ( P )) d ε } . Moreover, by Lemma 22 , we have that
log N [ ] ( ε , F L , L 2 ( P ) ) ≤ C 2 ε − m log ( C 2 / ε ) . \log N_{[]}(\varepsilon,\mathcal{F}_L,L_2(P))\le C_2\varepsilon^{-m}\log(C_2/\varepsilon). log N [ ] ( ε , F L , L 2 ( P )) ≤ C 2 ε − m log ( C 2 / ε ) .
Therefore, we have that
E Δ n L ≤ C 3 inf α ∈ ( 0 , 1 ) { α + 1 n ∫ α 1 ε − m / 2 { log ( C 2 / ε ) } 1 / 2 d ε } . \mathbb{E}\Delta_n^L \le C_3 \inf_{\alpha\in(0,1)}\left\{\alpha+\frac{1}{\sqrt{n}}\int_\alpha^1 \varepsilon^{-m/2}\{\log(C_2/\varepsilon)\}^{1/2}\,d\varepsilon\right\}. E Δ n L ≤ C 3 α ∈ ( 0 , 1 ) inf { α + n 1 ∫ α 1 ε − m /2 { log ( C 2 / ε ) } 1/2 d ε } .
Case 1: m = 1 m=1 m = 1 . In this case,
∫ α 1 ε − 1 / 2 { log ( C 2 / ε ) } 1 / 2 d ε ≤ ∫ 0 1 ε − 1 / 2 { log ( C 2 / ε ) } 1 / 2 d ε ≤ log C 2 ∫ 0 1 ε − 1 / 2 d ε + ∫ 0 1 ε − 1 / 2 { log ( 1 / ε ) } 1 / 2 d ε = 2 log C 2 + ∫ 0 ∞ u 1 / 2 e − u / 2 d u = : C 4 \begin{aligned}
\int_\alpha^1 \varepsilon^{-1/2}\{\log(C_2/\varepsilon)\}^{1/2}\,d\varepsilon\le\int_0^1 \varepsilon^{-1/2}\{\log(C_2/\varepsilon)\}^{1/2}\,d\varepsilon\\
&\le\sqrt{\log C_2}\int_0^1\varepsilon^{-1/2}\,d\varepsilon+\int_0^1\varepsilon^{-1/2}\{\log(1/\varepsilon)\}^{1/2}\,d\varepsilon\\
&=2\sqrt{\log C_2}+\int_0^\infty u^{1/2}e^{-u/2}\,du=:C_4
\end{aligned} ∫ α 1 ε − 1/2 { log ( C 2 / ε ) } 1/2 d ε ≤ ∫ 0 1 ε − 1/2 { log ( C 2 / ε ) } 1/2 d ε ≤ log C 2 ∫ 0 1 ε − 1/2 d ε + ∫ 0 1 ε − 1/2 { log ( 1/ ε ) } 1/2 d ε = 2 log C 2 + ∫ 0 ∞ u 1/2 e − u /2 d u =: C 4
Therefore, taking α = n − 1 / 2 \alpha=n^{-1/2} α = n − 1/2 gives E Δ n L ≤ C 5 n − 1 / 2 \mathbb{E}\Delta_n^L\le C_5n^{-1/2} E Δ n L ≤ C 5 n − 1/2 .
Case 2: m = 2 m=2 m = 2 . In this case, we have that
∫ α 1 ε − 1 { log ( C 2 / ε ) } 1 / 2 d ε ≤ C 6 { log ( 1 / α ) } 3 / 2 . \int_\alpha^1\varepsilon^{-1}\{\log(C_2/\varepsilon)\}^{1/2}\,d\varepsilon\le C_6\{\log(1/\alpha)\}^{3/2}. ∫ α 1 ε − 1 { log ( C 2 / ε ) } 1/2 d ε ≤ C 6 { log ( 1/ α ) } 3/2 .
Taking α = n − 1 / 2 \alpha=n^{-1/2} α = n − 1/2 gives E Δ n L ≤ C 7 n − 1 / 2 ( log n ) 3 / 2 \mathbb{E}\Delta_n^L\le C_7n^{-1/2}(\log n)^{3/2} E Δ n L ≤ C 7 n − 1/2 ( log n ) 3/2 .
Case 3: m ≥ 3 m\ge3 m ≥ 3 . In this case, we have that
∫ α 1 ε − m / 2 { log ( C 2 / ε ) } 1 / 2 d ε ≤ C 8 α 1 − m / 2 { log ( 1 / α ) } 1 / 2 . \int_\alpha^1\varepsilon^{-m/2}\{\log(C_2/\varepsilon)\}^{1/2}\,d\varepsilon\le C_8\alpha^{1-m/2}\{\log(1/\alpha)\}^{1/2}. ∫ α 1 ε − m /2 { log ( C 2 / ε ) } 1/2 d ε ≤ C 8 α 1 − m /2 { log ( 1/ α ) } 1/2 .
Taking α = n − 1 / m ( log n ) 1 / m \alpha=n^{-1/m}(\log n)^{1/m} α = n − 1/ m ( log n ) 1/ m gives E Δ n L ≤ C 9 n − 1 / m ( log n ) 1 / m \mathbb{E}\Delta_n^L\le C_9n^{-1/m}(\log n)^{1/m} E Δ n L ≤ C 9 n − 1/ m ( log n ) 1/ m .
Thus, for all m ≥ 1 m\ge1 m ≥ 1 , we have that
E Δ n L ≤ C 10 { n − 1 / 2 , m = 1 , n − 1 / 2 ( log n ) 3 / 2 , m = 2 , n − 1 / m ( log n ) 1 / m , m ≥ 3. \mathbb{E}\Delta_n^L\le C_{10}
\begin{cases}
n^{-1/2}, & m=1,\\
n^{-1/2}(\log n)^{3/2}, & m=2,\\
n^{-1/m}(\log n)^{1/m}, & m\ge3.
\end{cases} E Δ n L ≤ C 10 ⎩ ⎨ ⎧ n − 1/2 , n − 1/2 ( log n ) 3/2 , n − 1/ m ( log n ) 1/ m , m = 1 , m = 2 , m ≥ 3.
Since ∥ f ∥ ∞ ≤ B \|f\|_\infty\le B ∥ f ∥ ∞ ≤ B for all f ∈ F L f\in\mathcal{F}_L f ∈ F L , the Bennett concentration inequality for bounded empirical processes (e.g. Theorem 2.3 in Bousquet, [8 ] ) implies that, with probability at least 1 − η 1-\eta 1 − η ,
Δ n L ≤ 2 E Δ n L + C B log ( 1 / η ) n + C B log ( 1 / η ) n . \Delta_n^L\le2\mathbb{E}\Delta_n^L+CB\sqrt{\frac{\log(1/\eta)}{n}}+CB\frac{\log(1/\eta)}{n}. Δ n L ≤ 2 E Δ n L + C B n log ( 1/ η ) + C B n log ( 1/ η ) .
Since log ( e n / η ) ≥ 1 \log(en/\eta)\ge1 log ( e n / η ) ≥ 1 and taking m = d − 1 m=d-1 m = d − 1 , the lemma follows.
Lemma 24 (Initial estimation error after exploration). Suppose Assumptions 1-4 hold. During the first 2 T o 2T_o 2 T o rounds, use the constant policy p t ( ⋅ ) = 1 / 2 p_t(\cdot)=1/2 p t ( ⋅ ) = 1/2 , with independent arm assignments across rounds and independent of the contexts and reward noise. For η ∈ ( 0 , 1 / 2 ) \eta\in(0,1/2) η ∈ ( 0 , 1/2 ) , define β ^ k , 2 T o = V k , 2 T o − 1 b k , 2 T o \widehat{\beta}_{k,2T_o}=V_{k,2T_o}^{-1}b_{k,2T_o} β k , 2 T o = V k , 2 T o − 1 b k , 2 T o . Suppose there exist constants C i n i t , C T < ∞ C_{\mathrm{init}}, C_T<\infty C init , C T < ∞ , depending only on d , c g , σ , λ d,c_g,\sigma,\lambda d , c g , σ , λ and C β C_\beta C β , such that T o ≥ C T log { ( d + 1 ) / η } T_o\ge C_T\log\{(d+1)/\eta\} T o ≥ C T log {( d + 1 ) / η } . Then, taking λ = C λ \lambda=C_\lambda λ = C λ , where C λ > 0 C_\lambda>0 C λ > 0 is an absolute constant, it holds with probability at least 1 − η 1-\eta 1 − η that
∥ β ^ k , 2 T o − β k ∗ ∥ 2 ≤ C i n i t log ( T / η ) T o , k ∈ { 1 , 2 } . \|\widehat{\beta}_{k,2T_o}-\beta_k^*\|_2\le C_{\mathrm{init}}\sqrt{\frac{\log(T/\eta)}{T_o}},\qquad k\in\{1,2\}. ∥ β k , 2 T o − β k ∗ ∥ 2 ≤ C init T o log ( T / η ) , k ∈ { 1 , 2 } .
Consequently, if s 0 s_0 s 0 is the first lazy update time, then we have with probability at least 1 − η 1-\eta 1 − η that
∥ θ ^ s 0 − θ ∗ ∥ 2 ≤ 2 C i n i t log ( T / η ) T 0 . \lVert\widehat{\theta}_{s_0}-\theta^*\rVert_2 \le2C_{\mathrm{init}}\sqrt{\frac{\log(T/\eta)}{T_0}}. ∥ θ s 0 − θ ∗ ∥ 2 ≤ 2 C init T 0 log ( T / η ) .
Proof. For k ∈ { 1 , 2 } k \in\{1,2\} k ∈ { 1 , 2 } , let I k 0 = { i ≤ 2 T 0 : A i = k } I_k^0=\{i\le2T_0:A_i=k\} I k 0 = { i ≤ 2 T 0 : A i = k } and N 0 , k = ∣ I k 0 ∣ N_{0,k}=\lvert I_k^0\rvert N 0 , k = ∣ I k 0 ∣ . Let λ X = c g / ( 128 d 2 ) \lambda_X=c_g/(128d^2) λ X = c g / ( 128 d 2 ) . By Lemma 28 with A = [ 0 , 1 ] d A=[0,1]^d A = [ 0 , 1 ] d , we have λ min { E ( X ~ X ~ T ) } ≥ λ X \lambda_{\min}\{\mathbb{E}(\widetilde{X}\widetilde{X}^{\mathsf{T}})\}\ge\lambda_X λ m i n { E ( X X T )} ≥ λ X . Define
E a l l o c = { N 0 , 1 ≥ T 0 / 2 , N 0 , 2 ≥ T 0 / 2 } , \mathcal{E}_{\mathrm{alloc}}=\{N_{0,1}\ge T_0/2,\ N_{0,2}\ge T_0/2\}, E alloc = { N 0 , 1 ≥ T 0 /2 , N 0 , 2 ≥ T 0 /2 } ,
E i n i t = { λ min ( V k , 2 T 0 ) ≥ λ + λ X T 0 / 4 , for every k ∈ { 1 , 2 } } . \mathcal{E}_{\mathrm{init}}=\{\lambda_{\min}(V_{k,2T_0})\ge\lambda+\lambda_XT_0/4,\ \text{for every }k\in\{1,2\}\}. E init = { λ m i n ( V k , 2 T 0 ) ≥ λ + λ X T 0 /4 , for every k ∈ { 1 , 2 }} .
Under p t ( ⋅ ) = 1 / 2 p_t(\cdot)=1/2 p t ( ⋅ ) = 1/2 , the exploration actions are independent of the contexts and reward noise, and N 0 , k ∼ Bin ( 2 T 0 , 1 / 2 ) N_{0,k}\sim\operatorname{Bin}(2T_0,1/2) N 0 , k ∼ Bin ( 2 T 0 , 1/2 ) for each k ∈ { 1 , 2 } k\in\{1,2\} k ∈ { 1 , 2 } . By Hoeffding’s inequality (e.g. Theorem 2.2.6 in Vershynin, [50 ] ) and a union bound over k ∈ { 1 , 2 } k\in\{1,2\} k ∈ { 1 , 2 } , we have that
P ( E a l l o c c ) ≤ 2 exp ( − T 0 4 ) . \mathbb{P}(\mathcal{E}_{\mathrm{alloc}}^{c})\le2\exp\left(-\frac{T_0}{4}\right). P ( E alloc c ) ≤ 2 exp ( − 4 T 0 ) .
Fix k ∈ { 1 , 2 } k\in\{1,2\} k ∈ { 1 , 2 } and condition on the exploration actions A 1 , … , A 2 T 0 A_1,\ldots,A_{2T_0} A 1 , … , A 2 T 0 . Since these actions are sampled independently of the contexts, the matrices { X ~ i X ~ i T : i ∈ I k 0 } \{\widetilde{X}_i\widetilde{X}_i^{\mathsf{T}}:i\in I_k^0\} { X i X i T : i ∈ I k 0 } remain independent and each has the same distribution as X ~ X ~ T \widetilde{X}\widetilde{X}^{\mathsf{T}} X X T . Moreover, X ~ i X ~ i T ⪰ 0 \widetilde{X}_i\widetilde{X}_i^{\mathsf{T}}\succeq0 X i X i T ⪰ 0 and λ max ( X ~ i X ~ i T ) = ∥ X ~ i ∥ 2 2 ≤ d + 1 \lambda_{\max}(\widetilde{X}_i\widetilde{X}_i^{\mathsf{T}})=\lVert\widetilde{X}_i\rVert_2^2\le d+1 λ m a x ( X i X i T ) = ∥ X i ∥ 2 2 ≤ d + 1 . Therefore, on E a l l o c \mathcal{E}_{\mathrm{alloc}} E alloc , we have that
μ min : = λ min { ∑ i ∈ I k 0 E ( X ~ i X ~ i T ∣ A 1 , … , A 2 T 0 ) } ≥ N 0 , k λ X ≥ λ X T 0 2 . \mu_{\min}:=\lambda_{\min}\left\{\sum_{i\in I_k^0}\mathbb{E}(\widetilde{X}_i\widetilde{X}_i^{\mathsf{T}}\mid A_1,\ldots,A_{2T_0})\right\}\ge N_{0,k}\lambda_X\ge\frac{\lambda_XT_0}{2}. μ m i n := λ m i n ⎩ ⎨ ⎧ i ∈ I k 0 ∑ E ( X i X i T ∣ A 1 , … , A 2 T 0 ) ⎭ ⎬ ⎫ ≥ N 0 , k λ X ≥ 2 λ X T 0 .
By the matrix Chernoff inequality (e.g. Theorem 5.1.1 in Tropp, [48 ] ), applied with matrix dimension d + 1 d+1 d + 1 , upper eigenvalue bound d + 1 d+1 d + 1 and ε = 1 / 2 \varepsilon=1/2 ε = 1/2 , it holds that
P { λ min ( ∑ i ∈ I k 0 X ~ i X ~ i T ) ≤ μ min 2 | A 1 , … , A 2 T 0 } ≤ ( d + 1 ) ( e − 1 / 2 ( 1 / 2 ) 1 / 2 ) μ min / ( d + 1 ) ≤ ( d + 1 ) exp { − μ min 8 ( d + 1 ) } . \mathbb{P}\left\{\lambda_{\min}\left(\sum_{i\in I_k^0}\widetilde{X}_i\widetilde{X}_i^{\mathsf{T}}\right)\le\frac{\mu_{\min}}{2}\,\middle|\,A_1,\ldots,A_{2T_0}\right\}
\le(d+1)\left(\frac{e^{-1/2}}{(1/2)^{1/2}}\right)^{\mu_{\min}/(d+1)}
\le(d+1)\exp\left\{-\frac{\mu_{\min}}{8(d+1)}\right\}. P ⎩ ⎨ ⎧ λ m i n i ∈ I k 0 ∑ X i X i T ≤ 2 μ m i n A 1 , … , A 2 T 0 ⎭ ⎬ ⎫ ≤ ( d + 1 ) ( ( 1/2 ) 1/2 e − 1/2 ) μ m i n / ( d + 1 ) ≤ ( d + 1 ) exp { − 8 ( d + 1 ) μ m i n } .
Since μ min ≥ λ X T 0 / 2 \mu_{\min}\ge\lambda_XT_0/2 μ m i n ≥ λ X T 0 /2 on E a l l o c \mathcal{E}_{\mathrm{alloc}} E alloc and V k , 2 T 0 = λ I d + 1 + ∑ i ∈ I k 0 X ~ i X ~ i T V_{k,2T_0}=\lambda I_{d+1}+\sum_{i\in I_k^0}\widetilde{X}_i\widetilde{X}_i^{\mathsf{T}} V k , 2 T 0 = λ I d + 1 + ∑ i ∈ I k 0 X i X i T , a union bound over k ∈ { 1 , 2 } k\in\{1,2\} k ∈ { 1 , 2 } and the tower property give
P ( E i n i t c ∩ E a l l o c ) ≤ 2 ( d + 1 ) exp { − λ X T 0 16 ( d + 1 ) } . \mathbb{P}(\mathcal{E}_{\mathrm{init}}^c\cap\mathcal{E}_{\mathrm{alloc}})\le2(d+1)\exp\left\{-\frac{\lambda_XT_0}{16(d+1)}\right\}. P ( E init c ∩ E alloc ) ≤ 2 ( d + 1 ) exp { − 16 ( d + 1 ) λ X T 0 } .
Combining the preceding two bounds, we obtain
P ( E i n i t c ) ≤ 2 exp ( − T 0 4 ) + 2 ( d + 1 ) exp { − λ X T 0 16 ( d + 1 ) } ≤ η 2 , \mathbb{P}(\mathcal{E}_{\mathrm{init}}^c)\le2\exp\left(-\frac{T_0}{4}\right)+2(d+1)\exp\left\{-\frac{\lambda_XT_0}{16(d+1)}\right\}\le\frac{\eta}{2}, P ( E init c ) ≤ 2 exp ( − 4 T 0 ) + 2 ( d + 1 ) exp { − 16 ( d + 1 ) λ X T 0 } ≤ 2 η ,
where the last inequality follows whenever T 0 ≥ 4 log ( 8 / η ) T_0\ge4\log(8/\eta) T 0 ≥ 4 log ( 8/ η ) and T 0 ≥ { 16 ( d + 1 ) / λ X } log { 8 ( d + 1 ) / η } T_0\ge\{16(d+1)/\lambda_X\}\log\{8(d+1)/\eta\} T 0 ≥ { 16 ( d + 1 ) / λ X } log { 8 ( d + 1 ) / η } . Both inequalities hold under T 0 ≥ C T log { ( d + 1 ) / η } T_0\ge C_T\log\{(d+1)/\eta\} T 0 ≥ C T log {( d + 1 ) / η } for C T C_T C T sufficiently large. Consequently, with probability at least 1 − η / 2 1-\eta/2 1 − η /2 , we have that
λ min ( V k , 2 T 0 ) ≥ λ + λ X T 0 4 , k ∈ { 1 , 2 } . (40) \lambda_{\min}(V_{k,2T_0})\ge\lambda+\frac{\lambda_XT_0}{4},\qquad k\in\{1,2\}. \tag*{(40)} λ m i n ( V k , 2 T 0 ) ≥ λ + 4 λ X T 0 , k ∈ { 1 , 2 } . (40)
Next, for k ∈ { 1 , 2 } k \in\{1,2\} k ∈ { 1 , 2 } , write S k 0 = ∑ i ∈ I k 0 X ~ i ξ i , k S_k^0=\sum_{i\in I_k^0}\widetilde{X}_i\xi_{i,k} S k 0 = ∑ i ∈ I k 0 X i ξ i , k . For j = 1 , … , d + 1 j=1,\ldots,d+1 j = 1 , … , d + 1 , let S k , j 0 = ∑ i ∈ I k 0 X ~ i , j ξ i , k S_{k,j}^0=\sum_{i\in I_k^0}\widetilde{X}_{i,j}\xi_{i,k} S k , j 0 = ∑ i ∈ I k 0 X i , j ξ i , k denote the j j j th coordinate of S k 0 S_k^0 S k 0 . Conditional on the exploration actions and contexts, the noise variables remain independent, mean-zero and σ \sigma σ -sub-Gaussian by Assumption 2. Since ∣ X ~ i , j ∣ ≤ 1 |\widetilde{X}_{i,j}|\le1 ∣ X i , j ∣ ≤ 1 and N 0 , k ≤ 2 T o N_{0,k}\le2T_o N 0 , k ≤ 2 T o , the general Hoeffding inequality for sums of independent sub-Gaussian random variables (e.g. Theorem 2.6.3 in Vershynin, 2018) gives, for every u > 0 u>0 u > 0 ,
P { ∣ S k , j 0 ∣ ≥ u ∣ A 1 , … , A 2 T o , X 1 , … , X 2 T o } ≤ 2 exp { − c u 2 σ 2 T o } , \mathbb{P}\left\{|S_{k,j}^0|\ge u\mid A_1,\ldots,A_{2T_o},X_1,\ldots,X_{2T_o}\right\}\le2\exp\left\{-\frac{cu^2}{\sigma^2T_o}\right\}, P { ∣ S k , j 0 ∣ ≥ u ∣ A 1 , … , A 2 T o , X 1 , … , X 2 T o } ≤ 2 exp { − σ 2 T o c u 2 } ,
where c > 0 c>0 c > 0 is an absolute constant. Taking expectation over the actions and contexts and a union bound over k ∈ { 1 , 2 } k\in\{1,2\} k ∈ { 1 , 2 } and j = 1 , … , d + 1 j=1,\ldots,d+1 j = 1 , … , d + 1 , we obtain
P { max k ∈ { 1 , 2 } max 1 ≤ j ≤ d + 1 ∣ S k , j 0 ∣ ≥ u } ≤ 4 ( d + 1 ) exp { − c u 2 σ 2 T o } . \mathbb{P}\left\{\max_{k\in\{1,2\}}\max_{1\le j\le d+1}|S_{k,j}^0|\ge u\right\}\le4(d+1)\exp\left\{-\frac{cu^2}{\sigma^2T_o}\right\}. P { k ∈ { 1 , 2 } max 1 ≤ j ≤ d + 1 max ∣ S k , j 0 ∣ ≥ u } ≤ 4 ( d + 1 ) exp { − σ 2 T o c u 2 } .
Choosing u = C 1 T o log ( T / η ) u=C_1\sqrt{T_o\log(T/\eta)} u = C 1 T o log ( T / η ) with C 1 C_1 C 1 sufficiently large, we have with probability at least 1 − η / 2 1-\eta/2 1 − η /2 that
∥ S k 0 ∥ 2 ≤ d + 1 max 1 ≤ j ≤ d + 1 ∣ S k , j 0 ∣ ≤ C 2 T o log ( T / η ) , k ∈ { 1 , 2 } . \|S_k^0\|_2\le\sqrt{d+1}\max_{1\le j\le d+1}|S_{k,j}^0|\le C_2\sqrt{T_o\log(T/\eta)},\qquad k\in\{1,2\}. ∥ S k 0 ∥ 2 ≤ d + 1 1 ≤ j ≤ d + 1 max ∣ S k , j 0 ∣ ≤ C 2 T o log ( T / η ) , k ∈ { 1 , 2 } .
Combining this noise bound with (40) , with probability at least 1 − η 1-\eta 1 − η , both bounds hold. We work under these high-probability events in the rest of the proof.
Since Y i = X ~ i ⊤ β k ∗ + ξ i , k Y_i=\widetilde{X}_i^\top\beta_k^*+\xi_{i,k} Y i = X i ⊤ β k ∗ + ξ i , k for i ∈ I k 0 i\in I_k^0 i ∈ I k 0 , the ridge estimator satisfies
β ^ k , 2 T o − β k ∗ = V k , 2 T o − 1 S k 0 − λ V k , 2 T o − 1 β k ∗ . \widehat{\beta}_{k,2T_o}-\beta_k^*=V_{k,2T_o}^{-1}S_k^0-\lambda V_{k,2T_o}^{-1}\beta_k^*. β k , 2 T o − β k ∗ = V k , 2 T o − 1 S k 0 − λ V k , 2 T o − 1 β k ∗ .
Therefore, for any k ∈ { 1 , 2 } k\in\{1,2\} k ∈ { 1 , 2 } , we have that
∥ β ^ k , 2 T o − β k ∗ ∥ 2 ≤ ∥ V k , 2 T o − 1 S k 0 ∥ 2 + λ ∥ V k , 2 T o − 1 β k ∗ ∥ 2 ≤ ∥ V k , 2 T o − 1 ∥ o p ( ∥ S k 0 ∥ 2 + λ ∥ β k ∗ ∥ 2 ) ≤ 4 C 2 T o log ( T / η ) λ X T o + 4 C λ C β λ X T o ≤ C i n i t log ( T / η ) T o . \begin{aligned}
\|\widehat{\beta}_{k,2T_o}-\beta_k^*\|_2
&\le\|V_{k,2T_o}^{-1}S_k^0\|_2+\lambda\|V_{k,2T_o}^{-1}\beta_k^*\|_2\\
&\le\|V_{k,2T_o}^{-1}\|_{\mathrm{op}}\left(\|S_k^0\|_2+\lambda\|\beta_k^*\|_2\right)\\
&\le\frac{4C_2\sqrt{T_o\log(T/\eta)}}{\lambda_XT_o}+\frac{4C_\lambda C_\beta}{\lambda_XT_o}\\
&\le C_{\mathrm{init}}\sqrt{\frac{\log(T/\eta)}{T_o}}.
\end{aligned} ∥ β k , 2 T o − β k ∗ ∥ 2 ≤ ∥ V k , 2 T o − 1 S k 0 ∥ 2 + λ ∥ V k , 2 T o − 1 β k ∗ ∥ 2 ≤ ∥ V k , 2 T o − 1 ∥ op ( ∥ S k 0 ∥ 2 + λ ∥ β k ∗ ∥ 2 ) ≤ λ X T o 4 C 2 T o log ( T / η ) + λ X T o 4 C λ C β ≤ C init T o log ( T / η ) .
where the last inequality follows by taking T o ≥ C T log { ( d + 1 ) / η } T_o\ge C_T\log\{(d+1)/\eta\} T o ≥ C T log {( d + 1 ) / η } and enlarging C i n i t C_{\mathrm{init}} C init if necessary.
Finally, at the first lazy update time s 0 s_0 s 0 , the algorithm uses the estimator based on the first 2 T o 2T_o 2 T o rounds. Hence, on the same event, we have that
∥ θ ^ s 0 − θ ∗ ∥ 2 ≤ ∥ β ^ 1 , 2 T o − β 1 ∗ ∥ 2 + ∥ β ^ 2 , 2 T o − β 2 ∗ ∥ 2 ≤ 2 C i n i t log ( T / η ) T o . \|\widehat{\theta}_{s_0}-\theta^*\|_2\le\|\widehat{\beta}_{1,2T_o}-\beta_1^*\|_2+\|\widehat{\beta}_{2,2T_o}-\beta_2^*\|_2\le2C_{\mathrm{init}}\sqrt{\frac{\log(T/\eta)}{T_o}}. ∥ θ s 0 − θ ∗ ∥ 2 ≤ ∥ β 1 , 2 T o − β 1 ∗ ∥ 2 + ∥ β 2 , 2 T o − β 2 ∗ ∥ 2 ≤ 2 C init T o log ( T / η ) .
This proves the claim. □
Lemma 25 (Estimation at lazy updates). Suppose Assumptions 1-4 hold. Let S T S_T S T be the set of lazy update times up to time T T T , and let s 0 s_0 s 0 be the first lazy update time. Suppose that there exists a constant C l o c < ∞ C_{\mathrm{loc}}<\infty C loc < ∞ , depending only on d d d , L L L , c g c_g c g , σ \sigma σ , λ \lambda λ , C β C_\beta C β and the constants in Assumption 4, such that
T o ≥ C T log { ( d + 1 ) / η } ∨ C l o c log ( T / η ) r l o c 2 . T_o\ge C_T\log\{(d+1)/\eta\}\vee\sqrt{\frac{C_{\mathrm{loc}}\log(T/\eta)}{r_{\mathrm{loc}}^2}}. T o ≥ C T log {( d + 1 ) / η } ∨ r loc 2 C loc log ( T / η ) .
Then, it holds with probability at least 1 − η 1-\eta 1 − η that
θ ^ s ∈ Θ ~ j ∗ for every s ∈ S T . \widehat{\theta}_s \in\widetilde{\Theta}_{j^*} \quad\text{for every } s \in\mathcal{S}_T. θ s ∈ Θ j ∗ for every s ∈ S T .
Moreover, with probability at least 1 − η 1-\eta 1 − η , for every s ∈ S T s \in\mathcal{S}_T s ∈ S T and k ∈ { 1 , 2 } k \in\{1,2\} k ∈ { 1 , 2 } ,
∥ β ^ k , s − 1 − β k ∗ ∥ 2 ≤ C e s t log ( T / η ) s − 1 , \lVert\widehat{\beta}_{k,s-1}-\beta_k^* \rVert_2 \leq C_{\mathrm{est}}\sqrt{\frac{\log(T/\eta)}{s-1}}, ∥ β k , s − 1 − β k ∗ ∥ 2 ≤ C est s − 1 log ( T / η ) ,
where C e s t < ∞ C_{\mathrm{est}} < \infty C est < ∞ depends only on d d d , L L L , c g c_g c g , σ \sigma σ , λ \lambda λ , C β C_\beta C β and the constants in Assumption 4. Moreover, we have that
∥ θ ^ s − θ ∗ ∥ 2 ≤ 2 C e s t log ( T / η ) s − 1 . \lVert\widehat{\theta}_s-\theta^* \rVert_2 \leq2C_{\mathrm{est}}\sqrt{\frac{\log(T/\eta)}{s-1}}. ∥ θ s − θ ∗ ∥ 2 ≤ 2 C est s − 1 log ( T / η ) .
Proof. By Lemma 24, if
T o ≥ C T log { ( d + 1 ) / η } ∨ C l o c log ( T / η ) r l o c 2 , T_o \geq C_T \log\{(d+1)/\eta\} \lor\frac{C_{\mathrm{loc}}\log(T/\eta)}{r_{\mathrm{loc}}^2}, T o ≥ C T log {( d + 1 ) / η } ∨ r loc 2 C loc log ( T / η ) ,
then, with probability at least 1 − η / 4 1-\eta/4 1 − η /4 , we have that
∥ θ ^ s 0 − θ ∗ ∥ 2 ≤ r l o c . \lVert\widehat{\theta}_{s_0}-\theta^* \rVert_2 \leq r_{\mathrm{loc}}. ∥ θ s 0 − θ ∗ ∥ 2 ≤ r loc .
Moreover, by the same allocation and conditional matrix Chernoff arguments as in the proof of Lemma 24, it also holds with probability at least 1 − η / 4 1-\eta/4 1 − η /4 that
λ min ( V k , 2 T o ) ≥ λ + λ X T o 4 , k ∈ { 1 , 2 } . \lambda_{\min}(V_{k,2T_o}) \geq\lambda+\frac{\lambda_XT_o}{4}, \qquad k\in\{1,2\}. λ m i n ( V k , 2 T o ) ≥ λ + 4 λ X T o , k ∈ { 1 , 2 } .
Now consider the post-initialization rounds. For i > 2 T o i>2T_o i > 2 T o , define Z i , k = X ~ i X ~ i ⊤ 1 { A i = k } Z_{i,k}=\widetilde{X}_i\widetilde{X}_i^{\top}\mathbf{1}\{A_i=k\} Z i , k = X i X i ⊤ 1 { A i = k } . Then, we have that Z i , k ⪰ 0 Z_{i,k}\succeq0 Z i , k ⪰ 0 and λ max ( Z i , k ) ≤ d + 1 \lambda_{\max}(Z_{i,k})\leq d+1 λ m a x ( Z i , k ) ≤ d + 1 . Fix t ≥ 4 T o t\geq4T_o t ≥ 4 T o and k ∈ { 1 , 2 } k\in\{1,2\} k ∈ { 1 , 2 } . Suppose that θ ^ r ∈ Θ ~ \widehat{\theta}_r\in\widetilde{\Theta} θ r ∈ Θ for every r ∈ S T r\in\mathcal{S}_T r ∈ S T with r ≤ t r\leq t r ≤ t . For each i = 2 T o + 1 , … , t i=2T_o+1,\ldots,t i = 2 T o + 1 , … , t , let τ ( i ) \tau(i) τ ( i ) be the lazy update time at which the policy used at round i i i is computed. Then τ ( i ) ∈ S T \tau(i)\in\mathcal{S}_T τ ( i ) ∈ S T and τ ( i ) ≤ i ≤ t \tau(i)\leq i\leq t τ ( i ) ≤ i ≤ t . Hence by the assumption that θ ^ r ∈ Θ ~ \widehat{\theta}_r\in\widetilde{\Theta} θ r ∈ Θ for every r ∈ S T r\in\mathcal{S}_T r ∈ S T with r ≤ t r\leq t r ≤ t , it holds that θ ^ τ ( i ) ∈ Θ ~ \widehat{\theta}_{\tau(i)}\in\widetilde{\Theta} θ τ ( i ) ∈ Θ . Therefore, the policy used at round i i i is generated by a parameter in Θ ~ \widetilde{\Theta} Θ , and by Lemma 26 , we have that
E [ Z i , k ∣ F i − 1 ] ⪰ λ d i v I d + 1 . \mathbb{E}[Z_{i,k}\mid\mathcal{F}_{i-1}]\succeq\lambda_{\mathrm{div}}I_{d+1}. E [ Z i , k ∣ F i − 1 ] ⪰ λ div I d + 1 .
Therefore, we have
λ min ( ∑ i = 2 T o + 1 t E [ Z i , k ∣ F i − 1 ] ) ≥ λ d i v ( t − 2 T o ) . \lambda_{\min}\left(\sum_{i=2T_o+1}^{t}\mathbb{E}[Z_{i,k}\mid\mathcal{F}_{i-1}]\right)\geq\lambda_{\mathrm{div}}(t-2T_o). λ m i n ( i = 2 T o + 1 ∑ t E [ Z i , k ∣ F i − 1 ] ) ≥ λ div ( t − 2 T o ) .
Applying the matrix concentration inequality (e.g. Lemma 9 in Bastani et al., [5 ] , with R = d + 1 R=d+1 R = d + 1 , γ = 1 / 2 \gamma=1/2 γ = 1/2 and μ = λ d i v ( t − 2 T o ) \mu=\lambda_{\mathrm{div}}(t-2T_o) μ = λ div ( t − 2 T o ) ) gives
P { λ min ( ∑ i = 2 T o + 1 t Z i , k ) ≤ λ d i v 2 ( t − 2 T o ) , λ min ( ∑ i = 2 T o + 1 t E [ Z i , k ∣ F i − 1 ] ) ≥ λ d i v ( t − 2 T o ) } ≤ ( d + 1 ) ( e − 1 / 2 ( 1 / 2 ) 1 / 2 ) λ d i v ( t − 2 T o ) / ( d + 1 ) ≤ C 1 ( d + 1 ) exp { − c λ d i v ( t − 2 T o ) d + 1 } . \begin{aligned}
\mathbb{P}\left\{\lambda_{\min}\left(\sum_{i=2T_o+1}^{t}Z_{i,k}\right)\leq\frac{\lambda_{\mathrm{div}}}{2}(t-2T_o),\ \lambda_{\min}\left(\sum_{i=2T_o+1}^{t}\mathbb{E}[Z_{i,k}\mid\mathcal{F}_{i-1}]\right)\geq\lambda_{\mathrm{div}}(t-2T_o)\right\} \\
&\leq(d+1)\left(\frac{e^{-1/2}}{(1/2)^{1/2}}\right)^{\lambda_{\mathrm{div}}(t-2T_o)/(d+1)}
\leq C_1(d+1)\exp\left\{-c\frac{\lambda_{\mathrm{div}}(t-2T_o)}{d+1}\right\}.
\end{aligned} P { λ m i n ( i = 2 T o + 1 ∑ t Z i , k ) ≤ 2 λ div ( t − 2 T o ) , λ m i n ( i = 2 T o + 1 ∑ t E [ Z i , k ∣ F i − 1 ] ) ≥ λ div ( t − 2 T o ) } ≤ ( d + 1 ) ( ( 1/2 ) 1/2 e − 1/2 ) λ div ( t − 2 T o ) / ( d + 1 ) ≤ C 1 ( d + 1 ) exp { − c d + 1 λ div ( t − 2 T o ) } .
Taking a union bound over t = 4 T o , … , T t=4T_o,\ldots,T t = 4 T o , … , T and k ∈ { 1 , 2 } k\in\{1,2\} k ∈ { 1 , 2 } , then it holds with probability at least 1 − η / 4 1-\eta/4 1 − η /4 that if θ ^ r ∈ Θ ~ \widehat{\theta}_r\in\widetilde{\Theta} θ r ∈ Θ for every r ∈ S T r\in\mathcal{S}_T r ∈ S T with r ≤ t r\leq t r ≤ t ,
λ min ( ∑ i = 2 T o + 1 t Z i , k ) ≥ λ d i v 2 ( t − 2 T o ) , \lambda_{\min}\left(\sum_{i=2T_o+1}^{t}Z_{i,k}\right)\geq\frac{\lambda_{\mathrm{div}}}{2}(t-2T_o), λ m i n ( i = 2 T o + 1 ∑ t Z i , k ) ≥ 2 λ div ( t − 2 T o ) ,
whenever T o ≥ C 2 log ( ( d + 1 ) T / η ) T_o \ge C_2 \log((d+1)T/\eta) T o ≥ C 2 log (( d + 1 ) T / η ) . Combining the preceding forced-exploration and post-initialization bounds, with probability at least 1 − η / 2 1-\eta/2 1 − η /2 , it holds for every t ∈ { 2 T o , … , T } t \in\{2T_o,\ldots,T\} t ∈ { 2 T o , … , T } and k ∈ { 1 , 2 } k \in\{1,2\} k ∈ { 1 , 2 } that, if θ ^ r ∈ Θ ~ \hat{\theta}_r \in\widetilde{\Theta} θ ^ r ∈ Θ for every r ∈ S T r \in S_T r ∈ S T with r ≤ t r \le t r ≤ t , then
λ min ( V k , t ) ≥ c V t . \lambda_{\min}(V_{k,t}) \ge c_V t. λ m i n ( V k , t ) ≥ c V t .
Next we control the noise term uniformly over time. This step requires a martingale argument because, after the forced-exploration phase, the action A i A_i A i is adaptive. For k ∈ { 1 , 2 } k \in\{1,2\} k ∈ { 1 , 2 } and j = 1 , … , d + 1 j=1,\ldots,d+1 j = 1 , … , d + 1 , define
M t , k , j = ∑ i = 1 t X ~ i , j ξ i , k 1 { A i = k } . M_{t,k,j}=\sum_{i=1}^{t}\widetilde{X}_{i,j}\xi_{i,k}\mathbf{1}\{A_i=k\}. M t , k , j = i = 1 ∑ t X i , j ξ i , k 1 { A i = k } .
Let G i \mathcal{G}_i G i denote the sigma-field generated by the history before round i i i , the current covariate X i X_i X i , the randomisation used by the policy at round i i i , and the chosen action A i A_i A i . In particular, conditional on G i \mathcal{G}_i G i , the quantity X ~ i , j 1 { A i = k } \widetilde{X}_{i,j}\mathbf{1}\{A_i=k\} X i , j 1 { A i = k } is fixed and satisfies ∣ X ~ i , j 1 { A i = k } ∣ ≤ 1 |\widetilde{X}_{i,j}\mathbf{1}\{A_i=k\}| \le1 ∣ X i , j 1 { A i = k } ∣ ≤ 1 . Since the action is chosen before observing the reward noise, and since ξ i , k \xi_{i,k} ξ i , k is conditionally mean-zero and σ \sigma σ -sub-Gaussian, for every u ∈ R u \in\mathbb{R} u ∈ R , we have that
E [ exp { u X ~ i , j ξ i , k 1 { A i = k } } ∣ G i ] ≤ exp { σ 2 u 2 2 } . \mathbb{E}\left[\exp\left\{u\widetilde{X}_{i,j}\xi_{i,k}\mathbf{1}\{A_i=k\}\right\}\mid\mathcal{G}_i\right]\le\exp\left\{\frac{\sigma^2u^2}{2}\right\}. E [ exp { u X i , j ξ i , k 1 { A i = k } } ∣ G i ] ≤ exp { 2 σ 2 u 2 } .
Therefore, by the tower property, for every fixed t t t , k k k and j j j , the sum M t , k , j M_{t,k,j} M t , k , j is sub-Gaussian with variance proxy σ 2 t \sigma^2t σ 2 t . Hence, for every x > 0 x>0 x > 0 , we have that
P { ∣ M t , k , j ∣ ≥ x } ≤ 2 exp { − x 2 2 σ 2 t } . \mathbb{P}\{|M_{t,k,j}|\ge x\}\le2\exp\left\{-\frac{x^2}{2\sigma^2t}\right\}. P { ∣ M t , k , j ∣ ≥ x } ≤ 2 exp { − 2 σ 2 t x 2 } .
Taking
x t = σ 2 t log ( 16 ( d + 1 ) T η ) , x_t=\sigma\sqrt{2t\log\left(\frac{16(d+1)T}{\eta}\right)}, x t = σ 2 t log ( η 16 ( d + 1 ) T ) ,
and applying a union bound over t = 1 , … , T t=1,\ldots,T t = 1 , … , T , k ∈ { 1 , 2 } k\in\{1,2\} k ∈ { 1 , 2 } and j = 1 , … , d + 1 j=1,\ldots,d+1 j = 1 , … , d + 1 , we obtain
P { max k ∈ { 1 , 2 } max 1 ≤ j ≤ d + 1 ∣ M t , k , j ∣ ≤ σ 2 t log ( 16 ( d + 1 ) T η ) , for all t ∈ [ T ] } ≥ 1 − η 4 . \mathbb{P}\left\{\max_{k\in\{1,2\}}\max_{1\le j\le d+1}|M_{t,k,j}|\le\sigma\sqrt{2t\log\left(\frac{16(d+1)T}{\eta}\right)},\ \text{for all }t\in[T]\right\}\ge1-\frac{\eta}{4}. P { k ∈ { 1 , 2 } max 1 ≤ j ≤ d + 1 max ∣ M t , k , j ∣ ≤ σ 2 t log ( η 16 ( d + 1 ) T ) , for all t ∈ [ T ] } ≥ 1 − 4 η .
Consequently, with probability at least 1 − η / 4 1-\eta/4 1 − η /4 , for every t = 1 , … , T t=1,\ldots,T t = 1 , … , T and k ∈ { 1 , 2 } k\in\{1,2\} k ∈ { 1 , 2 } ,
∥ ∑ i = 1 t X ~ i ξ i , k 1 { A i = k } ∥ 2 ≤ d + 1 max 1 ≤ j ≤ d + 1 ∣ M t , k , j ∣ ≤ σ 2 ( d + 1 ) t log ( 16 ( d + 1 ) T η ) . \left\|\sum_{i=1}^{t}\widetilde{X}_i\xi_{i,k}\mathbf{1}\{A_i=k\}\right\|_2\le\sqrt{d+1}\max_{1\le j\le d+1}|M_{t,k,j}|\le\sigma\sqrt{2(d+1)t\log\left(\frac{16(d+1)T}{\eta}\right)}. i = 1 ∑ t X i ξ i , k 1 { A i = k } 2 ≤ d + 1 1 ≤ j ≤ d + 1 max ∣ M t , k , j ∣ ≤ σ 2 ( d + 1 ) t log ( η 16 ( d + 1 ) T ) .
This implies that, with probability at least 1 − η / 4 1-\eta/4 1 − η /4 , for every t = 1 , … , T t=1,\ldots,T t = 1 , … , T and k ∈ { 1 , 2 } k\in\{1,2\} k ∈ { 1 , 2 } , we have that
∥ ∑ i = 1 t X ~ i ξ i , k 1 { A i = k } ∥ 2 ≤ C ξ t log ( T / η ) . \left\|\sum_{i=1}^{t}\widetilde{X}_i\xi_{i,k}\mathbf{1}\{A_i=k\}\right\|_2\le C_{\xi}\sqrt{t\log(T/\eta)}. i = 1 ∑ t X i ξ i , k 1 { A i = k } 2 ≤ C ξ t log ( T / η ) .
By a union bound argument, we work under these high-probability events in the rest of the proof. We prove by induction over the lazy update times that θ ~ s ∈ Θ ~ j ∗ \widetilde{\theta}_s \in\widetilde{\Theta}_{j^*} θ s ∈ Θ j ∗ . For the first lazy update time s 0 s_0 s 0 , we have ∥ θ ~ s 0 − θ ∗ ∥ 2 ≤ r l o c \|\widetilde{\theta}_{s_0}-\theta^*\|_2\le r_{\mathrm{loc}} ∥ θ s 0 − θ ∗ ∥ 2 ≤ r loc . By Lemma 14 , we have that { θ : ∥ θ − θ ∗ ∥ 2 ≤ 2 r l o c } ⊆ Θ ~ j ∗ \{\theta:\|\theta-\theta^*\|_2\le2r_{\mathrm{loc}}\}\subseteq\widetilde{\Theta}_{j^*} { θ : ∥ θ − θ ∗ ∥ 2 ≤ 2 r loc } ⊆ Θ j ∗ . Hence θ ~ s 0 ∈ Θ ~ j ∗ \widetilde{\theta}_{s_0}\in\widetilde{\Theta}_{j^*} θ s 0 ∈ Θ j ∗ . Now fix a lazy update time s > s 0 s > s_0 s > s 0 and suppose that θ ^ r ∈ Θ ~ j ∗ \hat{\theta}_r \in\widetilde{\Theta}_{j^*} θ ^ r ∈ Θ j ∗ for every previous lazy update time r < s r < s r < s . Since Θ ~ j ∗ ⊆ Θ ~ \widetilde{\Theta}_{j^*} \subseteq\widetilde{\Theta} Θ j ∗ ⊆ Θ , the induction hypothesis gives that θ ^ r ∈ Θ ~ \hat{\theta}_r \in\widetilde{\Theta} θ ^ r ∈ Θ for every r ∈ S T r \in S_T r ∈ S T with r ≤ s − 1 r \le s - 1 r ≤ s − 1 . Therefore, applying the eigenvalue bound with t = s − 1 t = s - 1 t = s − 1 gives
λ min ( V k , s − 1 ) ≥ c V ( s − 1 ) , k ∈ { 1 , 2 } . \lambda_{\min}(V_{k,s-1}) \ge c_V(s-1), \qquad k \in\{1,2\}. λ m i n ( V k , s − 1 ) ≥ c V ( s − 1 ) , k ∈ { 1 , 2 } .
For k ∈ { 1 , 2 } k \in\{1,2\} k ∈ { 1 , 2 } , write S k , s − 1 = ∑ i = 1 s − 1 X ~ i ξ i , k 1 { A i = k } S_{k,s-1} = \sum_{i=1}^{s-1} \widetilde{X}_i \xi_{i,k} \mathbf{1}\{A_i = k\} S k , s − 1 = ∑ i = 1 s − 1 X i ξ i , k 1 { A i = k } . As in the proof of Lemma 24, the ridge estimator satisfies
β ^ k , s − 1 − β k ∗ = V k , s − 1 − 1 S k , s − 1 − λ V k , s − 1 − 1 β k ∗ . \widehat{\beta}_{k,s-1} - \beta_k^* = V_{k,s-1}^{-1}S_{k,s-1} - \lambda V_{k,s-1}^{-1}\beta_k^*. β k , s − 1 − β k ∗ = V k , s − 1 − 1 S k , s − 1 − λ V k , s − 1 − 1 β k ∗ .
Therefore,
∥ β ^ k , s − 1 − β k ∗ ∥ 2 ≤ ∥ V k , s − 1 − 1 S k , s − 1 ∥ 2 + λ ∥ V k , s − 1 − 1 β k ∗ ∥ 2 ≤ ∥ V k , s − 1 − 1 ∥ o p ∥ S k , s − 1 ∥ 2 + λ ∥ V k , s − 1 − 1 ∥ o p ∥ β k ∗ ∥ 2 ≤ C ξ ( s − 1 ) log ( T / η ) c V ( s − 1 ) + C λ C β c V ( s − 1 ) ≤ C e s t log ( T / η ) s − 1 . \begin{aligned}
\left\|\widehat{\beta}_{k,s-1} - \beta_k^*\right\|_2
&\le\left\|V_{k,s-1}^{-1}S_{k,s-1}\right\|_2 + \lambda\left\|V_{k,s-1}^{-1}\beta_k^*\right\|_2 \\
&\le\left\|V_{k,s-1}^{-1}\right\|_{\mathrm{op}}\left\|S_{k,s-1}\right\|_2 + \lambda\left\|V_{k,s-1}^{-1}\right\|_{\mathrm{op}}\left\|\beta_k^*\right\|_2 \\
&\le\frac{C_{\xi}\sqrt{(s-1)\log(T/\eta)}}{c_V(s-1)} + \frac{C_{\lambda}C_{\beta}}{c_V(s-1)} \\
&\le C_{\mathrm{est}}\sqrt{\frac{\log(T/\eta)}{s-1}}.
\end{aligned} β k , s − 1 − β k ∗ 2 ≤ V k , s − 1 − 1 S k , s − 1 2 + λ V k , s − 1 − 1 β k ∗ 2 ≤ V k , s − 1 − 1 op ∥ S k , s − 1 ∥ 2 + λ V k , s − 1 − 1 op ∥ β k ∗ ∥ 2 ≤ c V ( s − 1 ) C ξ ( s − 1 ) log ( T / η ) + c V ( s − 1 ) C λ C β ≤ C est s − 1 log ( T / η ) .
Hence,
∥ θ ^ s − θ ∗ ∥ 2 ≤ ∥ β ^ 1 , s − 1 − β 1 ∗ ∥ 2 + ∥ β ^ 2 , s − 1 − β 2 ∗ ∥ 2 ≤ 2 C e s t log ( T / η ) s − 1 . \left\|\widehat{\theta}_s - \theta^*\right\|_2 \le\left\|\widehat{\beta}_{1,s-1} - \beta_1^*\right\|_2 + \left\|\widehat{\beta}_{2,s-1} - \beta_2^*\right\|_2 \le2C_{\mathrm{est}}\sqrt{\frac{\log(T/\eta)}{s-1}}. θ s − θ ∗ 2 ≤ β 1 , s − 1 − β 1 ∗ 2 + β 2 , s − 1 − β 2 ∗ 2 ≤ 2 C est s − 1 log ( T / η ) .
Since s − 1 ≥ 2 T 0 s - 1 \ge2T_0 s − 1 ≥ 2 T 0 , we have
∥ θ ^ s − θ ∗ ∥ 2 ≤ 2 C e s t log ( T / η ) 2 T 0 . \left\|\widehat{\theta}_s - \theta^*\right\|_2 \le2C_{\mathrm{est}}\sqrt{\frac{\log(T/\eta)}{2T_0}}. θ s − θ ∗ 2 ≤ 2 C est 2 T 0 log ( T / η ) .
By the lower bound on T 0 T_0 T 0 , the final term is at most r l o c r_{\mathrm{loc}} r loc . Therefore ∥ θ ^ s − θ ∗ ∥ 2 ≤ r l o c \left\|\widehat{\theta}_s - \theta^*\right\|_2 \le r_{\mathrm{loc}} θ s − θ ∗ 2 ≤ r loc . Again using { θ : ∥ θ − θ ∗ ∥ 2 ≤ 2 r l o c } ⊆ Θ ~ j ∗ \{\theta: \|\theta- \theta^*\|_2 \le2r_{\mathrm{loc}}\} \subseteq\widetilde{\Theta}_{j^*} { θ : ∥ θ − θ ∗ ∥ 2 ≤ 2 r loc } ⊆ Θ j ∗ , this implies θ ^ s ∈ Θ ~ j ∗ \widehat{\theta}_s \in\widetilde{\Theta}_{j^*} θ s ∈ Θ j ∗ .
Therefore, by induction, θ ^ s ∈ Θ ~ j ∗ \widehat{\theta}_s \in\widetilde{\Theta}_{j^*} θ s ∈ Θ j ∗ for every s ∈ S T s \in S_T s ∈ S T . The displayed estimation error bound was also proved at each lazy update time. This completes the proof.
Auxiliary results for covariate diversity
In this section, we provide proofs of covariate diversity used in the proof of Lemma 25 . By the construction of the algorithm, the policy played at round t t t can be written as p ^ t = p θ ^ τ ( t ) , q ^ τ ( t ) \widehat{p}_t = p_{\widehat{\theta}_{\tau(t)},\widehat{q}_{\tau(t)}} p t = p θ τ ( t ) , q τ ( t ) .
We would like to remark that throughout this subsection, all proofs are constructed under the condition that θ ^ τ ( t ) ∈ Θ ~ \widehat{\theta}_{\tau(t)} \in\widetilde{\Theta} θ τ ( t ) ∈ Θ . These conditional results will then be used in the induction argument in the proof of Lemma 25 , where this event is shown to hold over all lazy update times on suitable large-probability events.
Lemma 26. Suppose Assumptions 1-4 hold. There exists a constant λ d i v = c g ε 0 v 3 2 / ( 128 d 2 ) \lambda_{\mathrm{div}} = c_g\varepsilon_0v_3^2/(128d^2) λ div = c g ε 0 v 3 2 / ( 128 d 2 ) depending only on d d d , ν \nu ν , L L L , c g c_g c g and the constants in Assumption 4, such that, for every t > 2 T 0 t > 2T_0 t > 2 T 0 satisfying θ ^ τ ( t ) ∈ Θ ~ \widehat{\theta}_{\tau(t)} \in\widetilde{\Theta} θ τ ( t ) ∈ Θ , we have that
E [ X ~ t X ~ t ⊤ 1 { A t = 1 } ∣ F t − 1 ] ⪰ λ d i v I d + 1 , \mathbb{E}\left[\widetilde{X}_t\widetilde{X}_t^\top\mathbf{1}\{A_t = 1\}\mid\mathcal{F}_{t-1}\right] \succeq\lambda_{\mathrm{div}}I_{d+1}, E [ X t X t ⊤ 1 { A t = 1 } ∣ F t − 1 ] ⪰ λ div I d + 1 , and
E [ X ~ t X ~ t ⊤ 1 { A t = 2 } ∣ F t − 1 ] ⪰ λ d i v I d + 1 . \mathbb{E}\left[\widetilde{X}_t\widetilde{X}_t^\top\mathbf{1}\{A_t = 2\}\mid\mathcal{F}_{t-1}\right] \succeq\lambda_{\mathrm{div}}I_{d+1}. E [ X t X t ⊤ 1 { A t = 2 } ∣ F t − 1 ] ⪰ λ div I d + 1 . Equivalently, it holds that
E [ X ~ t X ~ t ⊤ p ^ t ( X t ) ∣ F t − 1 ] ⪰ λ d i v I d + 1 , E [ X ~ t X ~ t ⊤ { 1 − p ^ t ( X t ) } ∣ F t − 1 ] ⪰ λ d i v I d + 1 . \mathbb{E}\left[\widetilde{X}_t\widetilde{X}_t^\top\widehat{p}_t(X_t)\mid\mathcal{F}_{t-1}\right] \succeq\lambda_{\mathrm{div}}I_{d+1}, \qquad\mathbb{E}\left[\widetilde{X}_t\widetilde{X}_t^\top\{1-\widehat{p}_t(X_t)\}\mid\mathcal{F}_{t-1}\right] \succeq\lambda_{\mathrm{div}}I_{d+1}. E [ X t X t ⊤ p t ( X t ) ∣ F t − 1 ] ⪰ λ div I d + 1 , E [ X t X t ⊤ { 1 − p t ( X t )} ∣ F t − 1 ] ⪰ λ div I d + 1 .
Proof. Fix t > 2 T 0 t > 2T_0 t > 2 T 0 and write X ~ t = ( 1 , X t ⊤ ) ⊤ \widetilde{X}_t = (1, X_t^\top)^\top X t = ( 1 , X t ⊤ ) ⊤ . By the lazy-update structure of Algorithm 1, the policy used at round t t t is computed at the most recent lazy update time τ ( t ) \tau(t) τ ( t ) . Hence p ^ t ( ⋅ ) = p θ ^ τ ( t ) , q ^ τ ( t ) ( ⋅ ) \widehat{p}_t(\cdot) = p_{\widehat{\theta}_{\tau(t)},\widehat{q}_{\tau(t)}}(\cdot) p t ( ⋅ ) = p θ τ ( t ) , q τ ( t ) ( ⋅ ) .
Since θ ^ τ ( t ) ∈ Θ ~ \widehat{\theta}_{\tau(t)} \in\widetilde{\Theta} θ τ ( t ) ∈ Θ and q ^ τ ( t ) ∈ Q θ ^ τ ( t ) , L \widehat{q}_{\tau(t)} \in Q_{\widehat{\theta}_{\tau(t)},L} q τ ( t ) ∈ Q θ τ ( t ) , L , Lemma 27 can be applied to the played policy. Define
B t = { x ∈ [ 0 , 1 ] d : ε 0 ≤ p ^ t ( x ) ≤ 1 − ε 0 } . B_t = \left\{x \in[0,1]^d : \varepsilon_0 \le\widehat{p}_t(x) \le1-\varepsilon_0\right\}. B t = { x ∈ [ 0 , 1 ] d : ε 0 ≤ p t ( x ) ≤ 1 − ε 0 } .
Then λ d ( B t ) ≥ v 0 \lambda_d(B_t) \ge v_0 λ d ( B t ) ≥ v 0 . Moreover, note that p ^ t \widehat{p}_t p t is F t − 1 \mathcal{F}_{t-1} F t − 1 -measurable, because at round t t t the policy is either inherited from an earlier lazy update or recomputed using only observations up to time t − 1 t-1 t − 1 .
Since X t X_t X t is independent of F t − 1 \mathcal{F}_{t-1} F t − 1 and has density g g g satisfying g ( x ) ≥ c g g(x) \ge c_g g ( x ) ≥ c g on [ 0 , 1 ] d [0,1]^d [ 0 , 1 ] d , Lemma 28 gives
E [ X ~ t X ~ t ⊤ 1 { X t ∈ B t } ∣ F t − 1 ] ⪰ c g v 0 3 128 d 2 I d + 1 . \mathbb{E}\left[\widetilde{X}_t\widetilde{X}_t^\top\mathbf{1}\{X_t \in B_t\} \mid\mathcal{F}_{t-1}\right] \succeq c_g \frac{v_0^3}{128d^2} I_{d+1}. E [ X t X t ⊤ 1 { X t ∈ B t } ∣ F t − 1 ] ⪰ c g 128 d 2 v 0 3 I d + 1 .
On B t B_t B t , we have p ^ t ( x ) ≥ ε 0 \widehat{p}_t(x) \ge\varepsilon_0 p t ( x ) ≥ ε 0 and 1 − p ^ t ( x ) ≥ ε 0 1-\widehat{p}_t(x) \ge\varepsilon_0 1 − p t ( x ) ≥ ε 0 . Therefore, it holds that
E [ X ~ t X ~ t ⊤ p ^ t ( X t ) ∣ F t − 1 ] ⪰ ε 0 E [ X ~ t X ~ t ⊤ 1 { X t ∈ B t } ∣ F t − 1 ] \mathbb{E}\left[\widetilde{X}_t\widetilde{X}_t^\top\widehat{p}_t(X_t) \mid\mathcal{F}_{t-1}\right] \succeq\varepsilon_0\mathbb{E}\left[\widetilde{X}_t\widetilde{X}_t^\top\mathbf{1}\{X_t \in B_t\} \mid\mathcal{F}_{t-1}\right] E [ X t X t ⊤ p t ( X t ) ∣ F t − 1 ] ⪰ ε 0 E [ X t X t ⊤ 1 { X t ∈ B t } ∣ F t − 1 ]
⪰ c g ε 0 v 0 3 128 d 2 I d + 1 . \succeq c_g\varepsilon_0 \frac{v_0^3}{128d^2} I_{d+1}. ⪰ c g ε 0 128 d 2 v 0 3 I d + 1 .
The same argument gives
E [ X ~ t X ~ t ⊤ { 1 − p ^ t ( X t ) } ∣ F t − 1 ] ⪰ c g ε 0 v 0 3 128 d 2 I d + 1 . \mathbb{E}\left[\widetilde{X}_t\widetilde{X}_t^\top\{1-\widehat{p}_t(X_t)\} \mid\mathcal{F}_{t-1}\right] \succeq c_g\varepsilon_0 \frac{v_0^3}{128d^2} I_{d+1}. E [ X t X t ⊤ { 1 − p t ( X t )} ∣ F t − 1 ] ⪰ c g ε 0 128 d 2 v 0 3 I d + 1 .
Finally, by the action-sampling rule in Algorithm 1,
P ( A t = 1 ∣ X t , F t − 1 ) = p ^ t ( X t ) , P ( A t = 2 ∣ X t , F t − 1 ) = 1 − p ^ t ( X t ) . \mathbb{P}(A_t = 1 \mid X_t,\mathcal{F}_{t-1}) = \widehat{p}_t(X_t), \qquad\mathbb{P}(A_t = 2 \mid X_t,\mathcal{F}_{t-1}) = 1-\widehat{p}_t(X_t). P ( A t = 1 ∣ X t , F t − 1 ) = p t ( X t ) , P ( A t = 2 ∣ X t , F t − 1 ) = 1 − p t ( X t ) .
Thus,
E [ X ~ t X ~ t ⊤ 1 { A t = 1 } ∣ F t − 1 ] = E [ X ~ t X ~ t ⊤ p ^ t ( X t ) ∣ F t − 1 ] , \mathbb{E}\left[\widetilde{X}_t\widetilde{X}_t^\top\mathbf{1}\{A_t = 1\} \mid\mathcal{F}_{t-1}\right] = \mathbb{E}\left[\widetilde{X}_t\widetilde{X}_t^\top\widehat{p}_t(X_t) \mid\mathcal{F}_{t-1}\right], E [ X t X t ⊤ 1 { A t = 1 } ∣ F t − 1 ] = E [ X t X t ⊤ p t ( X t ) ∣ F t − 1 ] ,
and similarly for arm 2. This proves the claim.
Lemma 27. Suppose Assumption 4 holds. Then there exist constants ε 0 > 0 \varepsilon_0 > 0 ε 0 > 0 and v 0 > 0 v_0 > 0 v 0 > 0 , depending only on d d d , ν \nu ν , L L L and the constants in Assumption 4, such that, for every θ ∈ Θ ~ \theta\in\widetilde{\Theta} θ ∈ Θ and every q ∈ Q θ , L q \in Q_{\theta,L} q ∈ Q θ , L ,
λ d ( { x ∈ [ 0 , 1 ] d : ε 0 ≤ p θ , q ( x ) ≤ 1 − ε 0 } ) ≥ v 0 , \lambda_d\left(\left\{x \in[0,1]^d : \varepsilon_0 \le p_{\theta,q}(x) \le1-\varepsilon_0\right\}\right) \ge v_0, λ d ( { x ∈ [ 0 , 1 ] d : ε 0 ≤ p θ , q ( x ) ≤ 1 − ε 0 } ) ≥ v 0 , where λ d ( ⋅ ) \lambda_d(\cdot) λ d ( ⋅ ) denotes the Lebesgue measure. In particular, the same conclusion holds for any reward-maximizing trace q θ ⋆ ∈ arg max q ∈ Q θ , L R ( p θ , q ) q^\star_\theta\in\arg\max_{q\in Q_{\theta,L}} R(p_{\theta,q}) q θ ⋆ ∈ arg max q ∈ Q θ , L R ( p θ , q ) .
Proof. Fix θ = ( a , γ ⊤ ) ⊤ ∈ Θ ~ \theta= (a,\gamma^\top)^\top\in\widetilde{\Theta} θ = ( a , γ ⊤ ) ⊤ ∈ Θ . Then θ ∈ Θ ~ j \theta\in\widetilde{\Theta}_j θ ∈ Θ j for at least one j ∈ { 1 , … , d } j \in\{1,\ldots,d\} j ∈ { 1 , … , d } . By relabelling coordinates inside this chart, write x = ( u , z ) ∈ [ 0 , 1 ] d − 1 × [ 0 , 1 ] x = (u,z) \in[0,1]^{d-1} \times[0,1] x = ( u , z ) ∈ [ 0 , 1 ] d − 1 × [ 0 , 1 ] and γ = ( b , c ) \gamma= (b,c) γ = ( b , c ) . Since c > 0 c > 0 c > 0 , define ϕ θ ( u ) = − ( a + b ⊤ u ) / c \phi_\theta(u) = -(a+b^\top u)/c ϕ θ ( u ) = − ( a + b ⊤ u ) / c . Denote
H θ = { ( u , z ) ∈ [ 0 , 1 ] d − 1 × [ 0 , 1 ] : a + b ⊤ u + c z = 0 } = { ( u , ϕ θ ( u ) ) : u ∈ [ 0 , 1 ] d − 1 , 0 ≤ ϕ θ ( u ) ≤ 1 } . H_\theta= \left\{(u,z) \in[0,1]^{d-1} \times[0,1] : a+b^\top u+cz=0\right\} = \left\{(u,\phi_\theta(u)) : u \in[0,1]^{d-1},\ 0 \le\phi_\theta(u) \le1\right\}. H θ = { ( u , z ) ∈ [ 0 , 1 ] d − 1 × [ 0 , 1 ] : a + b ⊤ u + cz = 0 } = { ( u , ϕ θ ( u )) : u ∈ [ 0 , 1 ] d − 1 , 0 ≤ ϕ θ ( u ) ≤ 1 } . Step 1: Bounded slope plus one crossing gives a positive-measure interior patch. Since ϕ θ ( u ) = − ( a + b ⊤ u ) / c \phi_\theta(u) = -(a+b^\top u)/c ϕ θ ( u ) = − ( a + b ⊤ u ) / c , norm equivalence on R d − 1 \mathbb{R}^{d-1} R d − 1 implies that there is a constant κ d , ν < ∞ \kappa_{d,\nu} < \infty κ d , ν < ∞ , depending only on d d d and ν \nu ν , such that
∣ ϕ θ ( u ) − ϕ θ ( v ) ∣ ≤ ∥ b ∥ 2 c ~ 0 ∥ u − v ∥ 2 ≤ K ∥ u − v ∥ ν . \left|\phi_\theta(u)-\phi_\theta(v)\right| \le\frac{\lVert b\rVert_2}{\widetilde{c}_0}\lVert u-v\rVert_2 \le K\lVert u-v\rVert_\nu. ∣ ϕ θ ( u ) − ϕ θ ( v ) ∣ ≤ c 0 ∥ b ∥ 2 ∥ u − v ∥ 2 ≤ K ∥ u − v ∥ ν .
Here K = B ~ 0 κ d , ν / c ~ 0 K=\widetilde{B}_0\kappa_{d,\nu}/\widetilde{c}_0 K = B 0 κ d , ν / c 0 . Thus uniformly over θ \theta θ , we have ∣ ϕ θ ( u ) − ϕ θ ( v ) ∣ ≤ K ∥ u − v ∥ ν |\phi_\theta(u)-\phi_\theta(v)|\le K\|u-v\|_\nu ∣ ϕ θ ( u ) − ϕ θ ( v ) ∣ ≤ K ∥ u − v ∥ ν .
By the definition of Θ ~ j \widetilde{\Theta}_j Θ j , there is some u θ ∈ [ 0 , 1 ] d − 1 u_\theta\in[0,1]^{d-1} u θ ∈ [ 0 , 1 ] d − 1 such that ρ ~ 0 ≤ ϕ θ ( u θ ) ≤ 1 − ρ ~ 0 \widetilde{\rho}_0\le\phi_\theta(u_\theta)\le1-\widetilde{\rho}_0 ρ 0 ≤ ϕ θ ( u θ ) ≤ 1 − ρ 0 . Let D ν = sup v ∈ [ − 1 , 1 ] d − 1 ∥ v ∥ ν D_\nu=\sup_{v\in[-1,1]^{d-1}}\|v\|_\nu D ν = sup v ∈ [ − 1 , 1 ] d − 1 ∥ v ∥ ν . Define
r 0 = { 1 , K = 0 , min { 1 , ρ ~ 0 2 K D ν } , K > 0. (41) r_0=
\begin{cases}
1, & K=0,\\
\min\left\{1,\dfrac{\widetilde{\rho}_0}{2KD_\nu}\right\}, & K>0.
\end{cases} \tag*{(41)} r 0 = ⎩ ⎨ ⎧ 1 , min { 1 , 2 K D ν ρ 0 } , K = 0 , K > 0. (41)
Let S θ = [ 0 , 1 ] d − 1 ∩ { u : ∥ u − u θ ∥ ∞ ≤ r 0 } S_\theta=[0,1]^{d-1}\cap\{u:\|u-u_\theta\|_\infty\le r_0\} S θ = [ 0 , 1 ] d − 1 ∩ { u : ∥ u − u θ ∥ ∞ ≤ r 0 } . Then standard calculation gives λ d − 1 ( S θ ) ≥ r 0 d − 1 \lambda_{d-1}(S_\theta)\ge r_0^{d-1} λ d − 1 ( S θ ) ≥ r 0 d − 1 . Moreover, for every u ∈ S θ u\in S_\theta u ∈ S θ , it holds that ∥ u − u θ ∥ ν ≤ D ν r 0 \|u-u_\theta\|_\nu\le D_\nu r_0 ∥ u − u θ ∥ ν ≤ D ν r 0 . Therefore, it holds that
∣ ϕ θ ( u ) − ϕ θ ( u θ ) ∣ ≤ K ∥ u − u θ ∥ ν ≤ K D ν r 0 ≤ ρ ~ 0 2 . |\phi_\theta(u)-\phi_\theta(u_\theta)|\le K\|u-u_\theta\|_\nu\le KD_\nu r_0\le\frac{\widetilde{\rho}_0}{2}. ∣ ϕ θ ( u ) − ϕ θ ( u θ ) ∣ ≤ K ∥ u − u θ ∥ ν ≤ K D ν r 0 ≤ 2 ρ 0 .
Hence, we have that
ρ ~ 0 2 ≤ ϕ θ ( u ) ≤ 1 − ρ ~ 0 2 , for all u ∈ S θ . \frac{\widetilde{\rho}_0}{2}\le\phi_\theta(u)\le1-\frac{\widetilde{\rho}_0}{2},
\qquad\text{for all }u\in S_\theta. 2 ρ 0 ≤ ϕ θ ( u ) ≤ 1 − 2 ρ 0 , for all u ∈ S θ .
In particular, for every u ∈ S θ u\in S_\theta u ∈ S θ , the point h ( u ) = ( u , ϕ θ ( u ) ) h(u)=(u,\phi_\theta(u)) h ( u ) = ( u , ϕ θ ( u )) belongs to H θ \mathcal{H}_\theta H θ . We define ρ = ρ ~ 0 / 2 \rho=\widetilde{\rho}_0/2 ρ = ρ 0 /2 .
Step 2: Choose a uniform interior probability level. Let C g r = 1 + K C_{\mathrm{gr}}=1+K C gr = 1 + K , and we will use this constant to control how vertical distance compares to the ℓ ν \ell_\nu ℓ ν distance from the full affine graph of ϕ θ \phi_\theta ϕ θ . Since H θ \mathcal{H}_\theta H θ is contained in this full graph, for x = ( u , ϕ θ ( u ) + s ) x=(u,\phi_\theta(u)+s) x = ( u , ϕ θ ( u ) + s ) , for any h = ( v , ϕ θ ( v ) ) ∈ H θ h=(v,\phi_\theta(v))\in\mathcal{H}_\theta h = ( v , ϕ θ ( v )) ∈ H θ , we have that
s = ϕ θ ( u ) + s − ϕ θ ( u ) ≤ ∣ ϕ θ ( u ) + s − ϕ θ ( v ) ∣ + ∣ ϕ θ ( v ) − ϕ θ ( u ) ∣ ≤ ∣ ϕ θ ( u ) + s − ϕ θ ( v ) ∣ + K ∥ u − v ∥ ν ≤ ( 1 + K ) ∥ x − h ∥ ν = C g r ∥ x − h ∥ ν , (42) \begin{aligned}
s&=\phi_\theta(u)+s-\phi_\theta(u)\\
&\le|\phi_\theta(u)+s-\phi_\theta(v)|+|\phi_\theta(v)-\phi_\theta(u)|\\
&\le|\phi_\theta(u)+s-\phi_\theta(v)|+K\|u-v\|_\nu\\
&\le(1+K)\|x-h\|_\nu\\
&=C_{\mathrm{gr}}\|x-h\|_\nu, \tag*{(42)}
\end{aligned} s = ϕ θ ( u ) + s − ϕ θ ( u ) ≤ ∣ ϕ θ ( u ) + s − ϕ θ ( v ) ∣ + ∣ ϕ θ ( v ) − ϕ θ ( u ) ∣ ≤ ∣ ϕ θ ( u ) + s − ϕ θ ( v ) ∣ + K ∥ u − v ∥ ν ≤ ( 1 + K ) ∥ x − h ∥ ν = C gr ∥ x − h ∥ ν , (42)
where the second inequality uses the K K K -Lipschitz property of ϕ θ \phi_\theta ϕ θ , and the third inequality follows because both ∣ ϕ θ ( u ) + s − ϕ θ ( v ) ∣ |\phi_\theta(u)+s-\phi_\theta(v)| ∣ ϕ θ ( u ) + s − ϕ θ ( v ) ∣ and ∥ u − v ∥ ν \|u-v\|_\nu ∥ u − v ∥ ν are bounded by ∥ x − h ∥ ν \|x-h\|_\nu ∥ x − h ∥ ν . Since this holds for every h ∈ H θ h\in\mathcal{H}_\theta h ∈ H θ , taking the infimum over h ∈ H θ h\in\mathcal{H}_\theta h ∈ H θ gives
dist ν ( x , H θ ) ≥ s C g r , (43) \operatorname{dist}_\nu(x,\mathcal{H}_\theta)\ge\frac{s}{C_{\mathrm{gr}}}, \tag*{(43)} dist ν ( x , H θ ) ≥ C gr s , (43)
The case x = ( u , ϕ θ ( u ) − s ) x=(u,\phi_\theta(u)-s) x = ( u , ϕ θ ( u ) − s ) is identical. Now define ϵ 0 = min { 1 , L ρ } / { 4 ( C g r + 4 ) } \epsilon_0=\min\{1,L\rho\}/\{4(C_{\mathrm{gr}}+4)\} ϵ 0 = min { 1 , L ρ } / { 4 ( C gr + 4 )} . This ensures
( C g r + 4 ) ϵ 0 ≤ 1 , (44) (C_{\mathrm{gr}}+4)\epsilon_0\le1, \tag*{(44)} ( C gr + 4 ) ϵ 0 ≤ 1 , (44)
( C g r + 1 ) ϵ 0 L ≤ ρ . (45) \frac{(C_{\mathrm{gr}}+1)\epsilon_0}{L}\le\rho. \tag*{(45)} L ( C gr + 1 ) ϵ 0 ≤ ρ . (45)
Note that (45) ensures that all the small vertical intervals in Step 4 , e.g. (46) , remain inside the cube.
Step 3: Partition the boundary trace. For u ∈ S θ u\in S_\theta u ∈ S θ , write h ( u ) = ( u , ϕ θ ( u ) ) ∈ H θ h(u)=(u,\phi_\theta(u))\in\mathcal{H}_\theta h ( u ) = ( u , ϕ θ ( u )) ∈ H θ , and define q θ ( u ) = q ( h ( u ) ) q_\theta(u)=q(h(u)) q θ ( u ) = q ( h ( u )) . Partition S θ S_\theta S θ into
A 0 = { u ∈ S θ : q θ ( u ) ≤ 2 ϵ 0 } A_0=\{u\in S_\theta:q_\theta(u)\le2\epsilon_0\} A 0 = { u ∈ S θ : q θ ( u ) ≤ 2 ϵ 0 } , A m = { u ∈ S θ : 2 ϵ 0 < q θ ( u ) < 1 − 2 ϵ 0 } A_m=\{u\in S_\theta:2\epsilon_0<q_\theta(u)<1-2\epsilon_0\} A m = { u ∈ S θ : 2 ϵ 0 < q θ ( u ) < 1 − 2 ϵ 0 } , A 1 = { u ∈ S θ : q θ ( u ) ≥ 1 − 2 ϵ 0 } A_1=\{u\in S_\theta:q_\theta(u)\ge1-2\epsilon_0\} A 1 = { u ∈ S θ : q θ ( u ) ≥ 1 − 2 ϵ 0 } . These three sets cover S θ S_\theta S θ . We will show that for every u ∈ S θ u \in S_\theta u ∈ S θ , there is a vertical interval of length at least ε 0 / L \varepsilon_0/L ε 0 / L on which ε 0 ≤ p θ , q ( u , z ) ≤ 1 − ε 0 \varepsilon_0 \le p_{\theta,q}(u,z) \le1-\varepsilon_0 ε 0 ≤ p θ , q ( u , z ) ≤ 1 − ε 0 .
Step 4: Low trace values give an interior region above the boundary. Take u ∈ A 0 u \in A_0 u ∈ A 0 . Then q θ ( u ) ≤ 2 ε 0 q_\theta(u) \le2\varepsilon_0 q θ ( u ) ≤ 2 ε 0 . Consider points above the boundary:
x = ( u , ϕ θ ( u ) + s ) , s ∈ [ C g r ε 0 L , ( C g r + 1 ) ε 0 L ] . (46) x = (u,\phi_\theta(u)+s), \qquad s \in\left[\frac{C_{\mathrm{gr}}\varepsilon_0}{L},\frac{(C_{\mathrm{gr}}+1)\varepsilon_0}{L}\right]. \tag*{(46)} x = ( u , ϕ θ ( u ) + s ) , s ∈ [ L C gr ε 0 , L ( C gr + 1 ) ε 0 ] . (46)
This interval has length ε 0 / L \varepsilon_0/L ε 0 / L . Since u ∈ S θ u \in S_\theta u ∈ S θ , the boundary height satisfies ϕ θ ( u ) ≤ 1 − ρ \phi_\theta(u) \le1-\rho ϕ θ ( u ) ≤ 1 − ρ . By (45) , we have that ( C g r + 1 ) ε 0 / L ≤ ρ (C_{\mathrm{gr}}+1)\varepsilon_0/L \le\rho ( C gr + 1 ) ε 0 / L ≤ ρ , therefore, these points remain inside [ 0 , 1 ] d [0,1]^d [ 0 , 1 ] d . Moreover, it holds that x ∈ X θ + x \in\mathcal{X}_\theta^+ x ∈ X θ + , as by construction, a + γ − j u + γ j { ϕ θ ( u ) + s } = γ j s > 0 a+\gamma_{-j}u+\gamma_j\{\phi_\theta(u)+s\}=\gamma_js>0 a + γ − j u + γ j { ϕ θ ( u ) + s } = γ j s > 0 .
Since x x x lies above the boundary, we have that p θ , q ( x ) = min { 1 , U θ , q ( x ) } p_{\theta,q}(x)=\min\{1,U_{\theta,q}(x)\} p θ , q ( x ) = min { 1 , U θ , q ( x )} . Since q ≥ 0 q \ge0 q ≥ 0 , it holds that
U θ , q ( x ) = inf h ∈ H θ { q ( h ) + L ∥ x − h ∥ ν } ≥ L dist ν ( x , H θ ) . U_{\theta,q}(x)=\inf_{h\in\mathcal{H}_\theta}\{q(h)+L\lVert x-h\rVert_\nu\}\ge L\operatorname{dist}_\nu(x,\mathcal{H}_\theta). U θ , q ( x ) = h ∈ H θ inf { q ( h ) + L ∥ x − h ∥ ν } ≥ L dist ν ( x , H θ ) .
Therefore, it holds that U θ , q ( x ) ≥ L s / C g r ≥ ε 0 U_{\theta,q}(x)\ge Ls/C_{\mathrm{gr}}\ge\varepsilon_0 U θ , q ( x ) ≥ L s / C gr ≥ ε 0 by (43) . On the other hand, using the specific boundary point h ( u ) h(u) h ( u ) ,
U θ , q ( x ) ≤ q ( h ( u ) ) + L ∥ x − h ( u ) ∥ ν . U_{\theta,q}(x)\le q(h(u))+L\lVert x-h(u)\rVert_\nu. U θ , q ( x ) ≤ q ( h ( u )) + L ∥ x − h ( u ) ∥ ν .
Since ∥ x − h ( u ) ∥ ν = s \lVert x-h(u)\rVert_\nu=s ∥ x − h ( u ) ∥ ν = s , we have
U θ , q ( x ) ≤ q θ ( u ) + L s ≤ 2 ε 0 + ( C g r + 1 ) ε 0 = ( C g r + 3 ) ε 0 ≤ 1 − ε 0 , U_{\theta,q}(x)\le q_\theta(u)+Ls\le2\varepsilon_0+(C_{\mathrm{gr}}+1)\varepsilon_0=(C_{\mathrm{gr}}+3)\varepsilon_0\le1-\varepsilon_0, U θ , q ( x ) ≤ q θ ( u ) + L s ≤ 2 ε 0 + ( C gr + 1 ) ε 0 = ( C gr + 3 ) ε 0 ≤ 1 − ε 0 ,
where the last inequality follows from (44) . Hence ε 0 ≤ p θ , q ( x ) ≤ 1 − ε 0 \varepsilon_0\le p_{\theta,q}(x)\le1-\varepsilon_0 ε 0 ≤ p θ , q ( x ) ≤ 1 − ε 0 . Thus, over every u ∈ A 0 u\in A_0 u ∈ A 0 , there is a vertical interval of length ε 0 / L \varepsilon_0/L ε 0 / L above the boundary where the policy is interior.
Step 5: High trace values give an interior region below the boundary. Take u ∈ A 1 u\in A_1 u ∈ A 1 . Then q θ ( u ) ≥ 1 − 2 ε 0 q_\theta(u)\ge1-2\varepsilon_0 q θ ( u ) ≥ 1 − 2 ε 0 . Consider points below the boundary:
x = ( u , ϕ θ ( u ) − s ) , s ∈ [ C g r ε 0 L , ( C g r + 1 ) ε 0 L ] . x=(u,\phi_\theta(u)-s), \qquad s\in\left[\frac{C_{\mathrm{gr}}\varepsilon_0}{L},\frac{(C_{\mathrm{gr}}+1)\varepsilon_0}{L}\right]. x = ( u , ϕ θ ( u ) − s ) , s ∈ [ L C gr ε 0 , L ( C gr + 1 ) ε 0 ] .
Again this interval has length ε 0 / L \varepsilon_0/L ε 0 / L . Since ϕ θ ( u ) ≥ ρ \phi_\theta(u)\ge\rho ϕ θ ( u ) ≥ ρ and ( C g r + 1 ) ε 0 / L ≤ ρ (C_{\mathrm{gr}}+1)\varepsilon_0/L\le\rho ( C gr + 1 ) ε 0 / L ≤ ρ by (45) , these points remain inside the cube.
Since x x x is below the boundary, p θ , q ( x ) = max { 0 , M θ , q ( x ) } p_{\theta,q}(x)=\max\{0,M_{\theta,q}(x)\} p θ , q ( x ) = max { 0 , M θ , q ( x )} . Using the boundary point h ( u ) h(u) h ( u ) , we have that
M θ , q ( x ) ≥ q ( h ( u ) ) − L ∥ x − h ( u ) ∥ ν = q θ ( u ) − L s . M_{\theta,q}(x)\ge q(h(u))-L\lVert x-h(u)\rVert_\nu=q_\theta(u)-Ls. M θ , q ( x ) ≥ q ( h ( u )) − L ∥ x − h ( u ) ∥ ν = q θ ( u ) − L s .
Thus,
M θ , q ( x ) ≥ 1 − 2 ε 0 − ( C g r + 1 ) ε 0 = 1 − ( C g r + 3 ) ε 0 ≥ ε 0 , M_{\theta,q}(x)\ge1-2\varepsilon_0-(C_{\mathrm{gr}}+1)\varepsilon_0=1-(C_{\mathrm{gr}}+3)\varepsilon_0\ge\varepsilon_0, M θ , q ( x ) ≥ 1 − 2 ε 0 − ( C gr + 1 ) ε 0 = 1 − ( C gr + 3 ) ε 0 ≥ ε 0 ,
where the last inequality follows from (44) . Also, since q ≤ 1 q\le1 q ≤ 1 , we have that
M θ , q ( x ) = sup h ∈ H θ { q ( h ) − L ∥ x − h ∥ ν } ≤ 1 − L dist ν ( x , H θ ) . M_{\theta,q}(x)=\sup_{h\in\mathcal{H}_\theta}\{q(h)-L\lVert x-h\rVert_\nu\}\le1-L\operatorname{dist}_\nu(x,\mathcal{H}_\theta). M θ , q ( x ) = h ∈ H θ sup { q ( h ) − L ∥ x − h ∥ ν } ≤ 1 − L dist ν ( x , H θ ) .
Therefore, we have that M θ , q ( x ) ≤ 1 − L s / C g r ≤ 1 − ε 0 M_{\theta,q}(x)\le1-Ls/C_{\mathrm{gr}}\le1-\varepsilon_0 M θ , q ( x ) ≤ 1 − L s / C gr ≤ 1 − ε 0 , and hence, ε 0 ≤ p θ , q ( x ) ≤ 1 − ε 0 \varepsilon_0\le p_{\theta,q}(x)\le1-\varepsilon_0 ε 0 ≤ p θ , q ( x ) ≤ 1 − ε 0 . So over every u ∈ A 1 u\in A_1 u ∈ A 1 , there is a vertical interval of length ε 0 / L \varepsilon_0/L ε 0 / L below the boundary where the policy is interior.
Step 6: Middle trace values give an interior region near the boundary. Take u ∈ A m u\in A_m u ∈ A m . Then 2 ε 0 < q θ ( u ) < 1 − 2 ε 0 2\varepsilon_0<q_\theta(u)<1-2\varepsilon_0 2 ε 0 < q θ ( u ) < 1 − 2 ε 0 . Consider points just above the boundary:
x = ( u , ϕ θ ( u ) + s ) , s ∈ [ 0 , ε 0 L ] . x=(u,\phi_\theta(u)+s), \qquad s\in\left[0,\frac{\varepsilon_0}{L}\right]. x = ( u , ϕ θ ( u ) + s ) , s ∈ [ 0 , L ε 0 ] .
These points remain inside the cube since ε 0 / L ≤ ρ \varepsilon_0/L\le\rho ε 0 / L ≤ ρ , which follows from (45) . Since x x x is above the boundary, p θ , q ( x ) = min { 1 , U θ , q ( x ) } p_{\theta,q}(x)=\min\{1,U_{\theta,q}(x)\} p θ , q ( x ) = min { 1 , U θ , q ( x )} . Using h ( u ) h(u) h ( u ) gives
U θ , q ( x ) ≤ q θ ( u ) + L s ≤ q θ ( u ) + ε 0 < 1 − ε 0 . U_{\theta,q}(x)\le q_\theta(u)+Ls\le q_\theta(u)+\varepsilon_0<1-\varepsilon_0. U θ , q ( x ) ≤ q θ ( u ) + L s ≤ q θ ( u ) + ε 0 < 1 − ε 0 .
For the lower bound, for every h ∈ H θ h\in\mathcal{H}_\theta h ∈ H θ , we have that
q ( h ) ≥ q ( h ( u ) ) − L ∥ h − h ( u ) ∥ ν . q(h)\ge q(h(u))-L\lVert h-h(u)\rVert_\nu. q ( h ) ≥ q ( h ( u )) − L ∥ h − h ( u ) ∥ ν .
Hence, it holds that
q ( h ) + L ∥ x − h ∥ ν ≥ q ( h ( u ) ) − L ∥ h − h ( u ) ∥ ν + L ∥ x − h ∥ ν . q(h)+L\lVert x-h\rVert_\nu\ge q(h(u))-L\lVert h-h(u)\rVert_\nu+L\lVert x-h\rVert_\nu. q ( h ) + L ∥ x − h ∥ ν ≥ q ( h ( u )) − L ∥ h − h ( u ) ∥ ν + L ∥ x − h ∥ ν .
Using the triangle inequality, ∥ h − h ( u ) ∥ ν ≤ ∥ h − x ∥ ν + ∥ x − h ( u ) ∥ ν \lVert h-h(u)\rVert_\nu\le\lVert h-x\rVert_\nu+\lVert x-h(u)\rVert_\nu ∥ h − h ( u ) ∥ ν ≤ ∥ h − x ∥ ν + ∥ x − h ( u ) ∥ ν , so
q ( h ) + L ∥ x − h ∥ ν ≥ q ( h ( u ) ) − L ∥ x − h ( u ) ∥ ν . q(h)+L\lVert x-h\rVert_\nu\ge q(h(u))-L\lVert x-h(u)\rVert_\nu. q ( h ) + L ∥ x − h ∥ ν ≥ q ( h ( u )) − L ∥ x − h ( u ) ∥ ν .
Taking the infimum over h ∈ H θ h\in\mathcal{H}_\theta h ∈ H θ gives
U θ , q ( x ) ≥ q θ ( u ) − L s ≥ q θ ( u ) − ε 0 > ε 0 . U_{\theta,q}(x)\ge q_\theta(u)-Ls\ge q_\theta(u)-\varepsilon_0>\varepsilon_0. U θ , q ( x ) ≥ q θ ( u ) − L s ≥ q θ ( u ) − ε 0 > ε 0 .
Therefore ε 0 ≤ p θ , q ( x ) ≤ 1 − ε 0 \varepsilon_0\le p_{\theta,q}(x)\le1-\varepsilon_0 ε 0 ≤ p θ , q ( x ) ≤ 1 − ε 0 . Thus over every u ∈ A m u\in A_m u ∈ A m , there is also a vertical interval of length ε 0 / L \varepsilon_0/L ε 0 / L on which the policy is interior.
Step 7: Integrate over the base patch. For every u ∈ S θ u\in S_\theta u ∈ S θ , one of the three cases u ∈ A 0 u\in A_0 u ∈ A 0 , u ∈ A m u\in A_m u ∈ A m , or u ∈ A 1 u\in A_1 u ∈ A 1 holds. In each case, the vertical fiber { z ∈ [ 0 , 1 ] : ( u , z ) ∈ [ 0 , 1 ] d } \{z\in[0,1]:(u,z)\in[0,1]^d\} { z ∈ [ 0 , 1 ] : ( u , z ) ∈ [ 0 , 1 ] d } contains a subinterval of length at least ε 0 / L \varepsilon_0/L ε 0 / L on which ε 0 ≤ p θ , q ( u , z ) ≤ 1 − ε 0 \varepsilon_0\le p_{\theta,q}(u,z)\le1-\varepsilon_0 ε 0 ≤ p θ , q ( u , z ) ≤ 1 − ε 0 . Therefore, by Fubini’s theorem,
λ d ( { x ∈ [ 0 , 1 ] d : ε 0 ≤ p θ , q ( x ) ≤ 1 − ε 0 } ) ≥ λ d − 1 ( S θ ) ε 0 L . \lambda_d\left(\left\{x\in[0,1]^d:\varepsilon_0\le p_{\theta,q}(x)\le1-\varepsilon_0\right\}\right)\ge\lambda_{d-1}(S_\theta)\frac{\varepsilon_0}{L}. λ d ( { x ∈ [ 0 , 1 ] d : ε 0 ≤ p θ , q ( x ) ≤ 1 − ε 0 } ) ≥ λ d − 1 ( S θ ) L ε 0 .
Since λ d − 1 ( S θ ) ≥ r 0 d − 1 \lambda_{d-1}(S_\theta)\ge r_0^{d-1} λ d − 1 ( S θ ) ≥ r 0 d − 1 , we get
λ d ( { x ∈ [ 0 , 1 ] d : ε 0 ≤ p θ , q ( x ) ≤ 1 − ε 0 } ) ≥ r 0 d − 1 ε 0 L . \lambda_d\left(\left\{x\in[0,1]^d:\varepsilon_0\le p_{\theta,q}(x)\le1-\varepsilon_0\right\}\right)\ge r_0^{d-1}\frac{\varepsilon_0}{L}. λ d ( { x ∈ [ 0 , 1 ] d : ε 0 ≤ p θ , q ( x ) ≤ 1 − ε 0 } ) ≥ r 0 d − 1 L ε 0 .
Taking v 0 = r 0 d − 1 ε 0 / L v_0=r_0^{d-1}\varepsilon_0/L v 0 = r 0 d − 1 ε 0 / L proves the claim. □
Lemma 28 (Covariate diversity on positive-measure sets). Suppose Assumption 1 holds. Let A ⊆ [ 0 , 1 ] d A\subseteq[0,1]^d A ⊆ [ 0 , 1 ] d be measurable and suppose λ d ( A ) ≥ v 0 > 0 \lambda_d(A)\ge v_0>0 λ d ( A ) ≥ v 0 > 0 . Define X ~ = ( 1 , X ⊤ ) ⊤ \widetilde{X}=(1,X^\top)^\top X = ( 1 , X ⊤ ) ⊤ . Then, we have that
λ min ( E [ X ~ X ~ ⊤ 1 { X ∈ A } ] ) ≥ c g v 0 3 128 d 2 . \lambda_{\min}\left(\mathbb{E}[\widetilde{X}\widetilde{X}^{\top}\mathbf{1}\{X\in A\}]\right)\ge\frac{c_gv_0^3}{128d^2}. λ m i n ( E [ X X ⊤ 1 { X ∈ A }] ) ≥ 128 d 2 c g v 0 3 .
Proof of Lemma 28. Let M A = E [ X ~ X ~ ⊤ 1 { X ∈ A } ] M_A=\mathbb{E}[\widetilde{X}\widetilde{X}^{\top}\mathbf{1}\{X\in A\}] M A = E [ X X ⊤ 1 { X ∈ A }] . Fix any unit vector w = ( w 0 , w ‾ ) ∈ R d + 1 w=(w_0,\overline{w})\in\mathbb{R}^{d+1} w = ( w 0 , w ) ∈ R d + 1 , where w ‾ ∈ R d \overline{w}\in\mathbb{R}^d w ∈ R d . We first show that there exists a constant C d = 4 d C_d=4d C d = 4 d such that, for every t > 0 t>0 t > 0 ,
λ d { x ∈ [ 0 , 1 ] d : ∣ w 0 + w ‾ ⊤ x ∣ ≤ t } ≤ C d t . \lambda_d\{x\in[0,1]^d:|w_0+\overline{w}^{\top}x|\le t\}\le C_dt. λ d { x ∈ [ 0 , 1 ] d : ∣ w 0 + w ⊤ x ∣ ≤ t } ≤ C d t .
If ∥ w ‾ ∥ 2 ≥ 1 / ( 2 d ) \lVert\overline{w}\rVert_2\ge1/(2\sqrt{d}) ∥ w ∥ 2 ≥ 1/ ( 2 d ) , choose j ∈ { 1 , … , d } j\in\{1,\ldots,d\} j ∈ { 1 , … , d } such that ∣ w ‾ j ∣ ≥ ∥ w ‾ ∥ 2 / d ≥ 1 / ( 2 d ) |\overline{w}_j|\ge\lVert\overline{w}\rVert_2/\sqrt{d}\ge1/(2d) ∣ w j ∣ ≥ ∥ w ∥ 2 / d ≥ 1/ ( 2 d ) . For fixed x − j x_{-j} x − j , the set of x j ∈ [ 0 , 1 ] x_j\in[0,1] x j ∈ [ 0 , 1 ] satisfying ∣ w 0 + w ‾ ⊤ x ∣ ≤ t |w_0+\overline{w}^{\top}x|\le t ∣ w 0 + w ⊤ x ∣ ≤ t is an interval of length at most 2 t / ∣ w ‾ j ∣ ≤ 4 d t 2t/|\overline{w}_j|\le4dt 2 t /∣ w j ∣ ≤ 4 d t . Hence, by Fubini’s theorem, λ d { x ∈ [ 0 , 1 ] d : ∣ w 0 + w ‾ ⊤ x ∣ ≤ t } ≤ 4 d t \lambda_d\{x\in[0,1]^d:|w_0+\overline{w}^{\top}x|\le t\}\le4dt λ d { x ∈ [ 0 , 1 ] d : ∣ w 0 + w ⊤ x ∣ ≤ t } ≤ 4 d t .
If ∥ w ‾ ∥ 2 < 1 / ( 2 d ) \lVert\overline{w}\rVert_2<1/(2\sqrt{d}) ∥ w ∥ 2 < 1/ ( 2 d ) , then ∣ w 0 ∣ = ( 1 − ∥ w ‾ ∥ 2 2 ) 1 / 2 |w_0|=(1-\lVert\overline{w}\rVert_2^2)^{1/2} ∣ w 0 ∣ = ( 1 − ∥ w ∥ 2 2 ) 1/2 and ∣ w ‾ ⊤ x ∣ ≤ ∥ w ‾ ∥ 2 ∥ x ∥ 2 < 1 / 2 |\overline{w}^{\top}x|\le\lVert\overline{w}\rVert_2\lVert x\rVert_2<1/2 ∣ w ⊤ x ∣ ≤ ∥ w ∥ 2 ∥ x ∥ 2 < 1/2 for all x ∈ [ 0 , 1 ] d x\in[0,1]^d x ∈ [ 0 , 1 ] d . Hence, it holds that ∣ w 0 + w ‾ ⊤ x ∣ > 1 / 4 |w_0+\overline{w}^{\top}x|>1/4 ∣ w 0 + w ⊤ x ∣ > 1/4 from the triangle inequality for all x ∈ [ 0 , 1 ] d x\in[0,1]^d x ∈ [ 0 , 1 ] d . Therefore the slab is empty when t ≤ 1 / 4 t\le1/4 t ≤ 1/4 , while for t > 1 / 4 t>1/4 t > 1/4 its Lebesgue measure is at most 1 ≤ 4 t ≤ 4 d t 1\le4t\le4dt 1 ≤ 4 t ≤ 4 d t . Thus, in all cases, we have that
λ d { x ∈ [ 0 , 1 ] d : ∣ w 0 + w ‾ ⊤ x ∣ ≤ t } ≤ 4 d t . \lambda_d\{x\in[0,1]^d:|w_0+\overline{w}^{\top}x|\le t\}\le4dt. λ d { x ∈ [ 0 , 1 ] d : ∣ w 0 + w ⊤ x ∣ ≤ t } ≤ 4 d t .
Take t 0 = v 0 / ( 8 d ) t_0 = v_0/(8d) t 0 = v 0 / ( 8 d ) . Then, we have that
λ d { x ∈ [ 0 , 1 ] d : ∣ w 0 + w ˉ T x ∣ ≤ t 0 } ≤ v 0 2 . \lambda_d\{x \in[0,1]^d : |w_0 + \bar{w}^{\mathsf{T}}x| \le t_0\} \le\frac{v_0}{2}. λ d { x ∈ [ 0 , 1 ] d : ∣ w 0 + w ˉ T x ∣ ≤ t 0 } ≤ 2 v 0 .
Since λ d ( A ) ≥ v 0 \lambda_d(A) \ge v_0 λ d ( A ) ≥ v 0 , we have
λ d ( A ∩ { x : ∣ w 0 + w ˉ T x ∣ > t 0 } ) ≥ v 0 2 . \lambda_d\left(A \cap\{x : |w_0 + \bar{w}^{\mathsf{T}}x| > t_0\}\right) \ge\frac{v_0}{2}. λ d ( A ∩ { x : ∣ w 0 + w ˉ T x ∣ > t 0 } ) ≥ 2 v 0 .
Therefore, using the density lower bound g ( x ) ≥ c g g(x) \ge c_g g ( x ) ≥ c g from Assumption 1,
w T M A w = E [ ( w T X ~ ) 2 1 { X ∈ A } ] = ∫ A ( w 0 + w ˉ T x ) 2 g ( x ) d x ≥ c g ∫ A ( w 0 + w ˉ T x ) 2 d x ≥ c g ∫ A ∩ { ∣ w 0 + w ˉ T x ∣ > t 0 } ( w 0 + w ˉ T x ) 2 d x ≥ c g t 0 2 v 0 2 = c g v 0 3 128 d 2 . \begin{aligned}
w^{\mathsf{T}}M_Aw
&= \mathbb{E}[(w^{\mathsf{T}}\widetilde{X})^2\mathbf{1}\{X \in A\}] \\
&= \int_A (w_0+\bar{w}^{\mathsf{T}}x)^2g(x)\,dx \\
&\ge c_g\int_A (w_0+\bar{w}^{\mathsf{T}}x)^2\,dx \\
&\ge c_g\int_{A\cap\{|w_0+\bar{w}^{\mathsf{T}}x|>t_0\}}(w_0+\bar{w}^{\mathsf{T}}x)^2\,dx \\
&\ge c_gt_0^2\frac{v_0}{2}
= c_g\frac{v_0^3}{128d^2}.
\end{aligned} w T M A w = E [( w T X ) 2 1 { X ∈ A }] = ∫ A ( w 0 + w ˉ T x ) 2 g ( x ) d x ≥ c g ∫ A ( w 0 + w ˉ T x ) 2 d x ≥ c g ∫ A ∩ { ∣ w 0 + w ˉ T x ∣ > t 0 } ( w 0 + w ˉ T x ) 2 d x ≥ c g t 0 2 2 v 0 = c g 128 d 2 v 0 3 .
Taking the infimum over all unit vectors w ∈ R d + 1 w \in\mathbb{R}^{d+1} w ∈ R d + 1 gives λ min ( M A ) ≥ c g v 0 3 / ( 128 d 2 ) \lambda_{\min}(M_A) \ge c_gv_0^3/(128d^2) λ m i n ( M A ) ≥ c g v 0 3 / ( 128 d 2 ) .
Lemma 29 (Exploration property). Suppose Assumptions 1 and 4 hold. Let θ ∈ Θ \theta\in\Theta θ ∈ Θ and q ∈ Q θ , L q \in\mathcal{Q}_{\theta,L} q ∈ Q θ , L , with Q θ , L \mathcal{Q}_{\theta,L} Q θ , L defined in (4) . Define ε L = min { 1 , L ρ 0 / 4 } \varepsilon_L = \min\{1,L\rho_0/4\} ε L = min { 1 , L ρ 0 /4 } . Then there exists a constant c p u l l > 0 c_{\mathrm{pull}} > 0 c pull > 0 , depending only on d d d , ν \nu ν , c g c_g c g and the constants in Assumption 4, such that
E { p θ , q ( X ) } ∧ E { 1 − p θ , q ( X ) } ≥ c p u l l ε L 2 L . \mathbb{E}\{p_{\theta,q}(X)\}\wedge\mathbb{E}\{1-p_{\theta,q}(X)\}\ge c_{\mathrm{pull}}\frac{\varepsilon_L^2}{L}. E { p θ , q ( X )} ∧ E { 1 − p θ , q ( X )} ≥ c pull L ε L 2 . In particular, if L ≥ 4 / ρ 0 L \ge4/\rho_0 L ≥ 4/ ρ 0 , then E { p θ , q ( X ) } ∧ E { 1 − p θ , q ( X ) } ≥ c p u l l / L \mathbb{E}\{p_{\theta,q}(X)\}\wedge\mathbb{E}\{1-p_{\theta,q}(X)\}\ge c_{\mathrm{pull}}/L E { p θ , q ( X )} ∧ E { 1 − p θ , q ( X )} ≥ c pull / L , whereas if L < 4 / ρ 0 L < 4/\rho_0 L < 4/ ρ 0 , then E { p θ , q ( X ) } ∧ E { 1 − p θ , q ( X ) } ≥ c p u l l ′ L \mathbb{E}\{p_{\theta,q}(X)\}\wedge\mathbb{E}\{1-p_{\theta,q}(X)\}\ge c'_{\mathrm{pull}}L E { p θ , q ( X )} ∧ E { 1 − p θ , q ( X )} ≥ c pull ′ L , where c p u l l ′ = c p u l l ρ 0 2 / 4 c'_{\mathrm{pull}}=c_{\mathrm{pull}}\rho_0^2/4 c pull ′ = c pull ρ 0 2 /4 .
Proof of Lemma 29. Fix θ ∈ Θ ~ \theta\in\widetilde{\Theta} θ ∈ Θ and q ∈ Q θ , L q \in\mathcal{Q}_{\theta,L} q ∈ Q θ , L . By Lemma 27 , there exists a measurable set B θ , q ⊆ [ 0 , 1 ] d B_{\theta,q}\subseteq[0,1]^d B θ , q ⊆ [ 0 , 1 ] d such that
ε L 4 ( C g r + 4 ) ≤ p θ , q ( x ) ≤ 1 − ε L 4 ( C g r + 4 ) , x ∈ B θ , q , \frac{\varepsilon_L}{4(C_{\mathrm{gr}}+4)}
\le p_{\theta,q}(x)
\le1-\frac{\varepsilon_L}{4(C_{\mathrm{gr}}+4)},
\qquad x\in B_{\theta,q}, 4 ( C gr + 4 ) ε L ≤ p θ , q ( x ) ≤ 1 − 4 ( C gr + 4 ) ε L , x ∈ B θ , q ,
where C g r > 0 C_{\mathrm{gr}}>0 C gr > 0 is the constant defined in Equation (42) . Moreover, it holds that
λ d ( B θ , q ) ≥ r 0 d − 1 ε L 4 ( C g r + 4 ) L , \lambda_d(B_{\theta,q})\ge r_0^{d-1}\frac{\varepsilon_L}{4(C_{\mathrm{gr}}+4)L}, λ d ( B θ , q ) ≥ r 0 d − 1 4 ( C gr + 4 ) L ε L ,
where r 0 > 0 r_0>0 r 0 > 0 is defined in (41) and depends only on d d d , ν \nu ν and the constants in Assumption 4. Since p θ , q ( x ) ≥ ε L / { 4 ( C g r + 4 ) } p_{\theta,q}(x)\ge\varepsilon_L/\{4(C_{\mathrm{gr}}+4)\} p θ , q ( x ) ≥ ε L / { 4 ( C gr + 4 )} on B θ , q B_{\theta,q} B θ , q , we have that
E { p θ , q ( X ) } ≥ E { p θ , q ( X ) 1 ( X ∈ B θ , q ) } ≥ ε L 4 ( C g r + 4 ) P ( X ∈ B θ , q ) . (47) \mathbb{E}\{p_{\theta,q}(X)\}
\ge\mathbb{E}\{p_{\theta,q}(X)\mathbf{1}(X\in B_{\theta,q})\}
\ge\frac{\varepsilon_L}{4(C_{\mathrm{gr}}+4)}\mathbb{P}(X\in B_{\theta,q}). \tag*{(47)} E { p θ , q ( X )} ≥ E { p θ , q ( X ) 1 ( X ∈ B θ , q )} ≥ 4 ( C gr + 4 ) ε L P ( X ∈ B θ , q ) . (47)
By Assumption 1, the density of X X X is bounded below by c g c_g c g on [ 0 , 1 ] d [0,1]^d [ 0 , 1 ] d . Hence, we have that
P ( X ∈ B θ , q ) = ∫ B θ , q g ( x ) d x ≥ c g λ d ( B θ , q ) ≥ c g r 0 d − 1 ε L 4 ( C g r + 4 ) L . (48) \mathbb{P}(X\in B_{\theta,q})
=\int_{B_{\theta,q}}g(x)\,dx
\ge c_g\lambda_d(B_{\theta,q})
\ge c_gr_0^{d-1}\frac{\varepsilon_L}{4(C_{\mathrm{gr}}+4)L}. \tag*{(48)} P ( X ∈ B θ , q ) = ∫ B θ , q g ( x ) d x ≥ c g λ d ( B θ , q ) ≥ c g r 0 d − 1 4 ( C gr + 4 ) L ε L . (48)
Substituting (48) into (47) gives
E { p θ , q ( X ) } ≥ c g r r 0 d − 1 16 ( C g r + 4 ) 2 ε L 2 L . \mathbb{E}\{p_{\theta,q}(X)\} \ge\frac{c_{\mathrm{gr}}r_0^{d-1}}{16(C_{\mathrm{gr}}+4)^2}\frac{\varepsilon_L^2}{L}. E { p θ , q ( X )} ≥ 16 ( C gr + 4 ) 2 c gr r 0 d − 1 L ε L 2 .
Similarly, since 1 − p θ , q ( x ) ≥ ε L / { 4 ( C g r + 4 ) } 1-p_{\theta,q}(x)\ge\varepsilon_L/\{4(C_{\mathrm{gr}}+4)\} 1 − p θ , q ( x ) ≥ ε L / { 4 ( C gr + 4 )} on B θ , q B_{\theta,q} B θ , q , we have that
E { 1 − p θ , q ( X ) } ≥ E [ { 1 − p θ , q ( X ) } 1 ( X ∈ B θ , q ) ] ≥ ε L 4 ( C g r + 4 ) P ( X ∈ B θ , q ) ≥ c g r r 0 d − 1 16 ( C g r + 4 ) 2 ε L 2 L . \mathbb{E}\{1-p_{\theta,q}(X)\}\ge\mathbb{E}[\{1-p_{\theta,q}(X)\}\mathbf{1}(X\in B_{\theta,q})]\ge\frac{\varepsilon_L}{4(C_{\mathrm{gr}}+4)}\mathbb{P}(X\in B_{\theta,q})\ge\frac{c_{\mathrm{gr}}r_0^{d-1}}{16(C_{\mathrm{gr}}+4)^2}\frac{\varepsilon_L^2}{L}. E { 1 − p θ , q ( X )} ≥ E [{ 1 − p θ , q ( X )} 1 ( X ∈ B θ , q )] ≥ 4 ( C gr + 4 ) ε L P ( X ∈ B θ , q ) ≥ 16 ( C gr + 4 ) 2 c gr r 0 d − 1 L ε L 2 .
Therefore, we have that
E { p θ , q ( X ) } ∧ E { 1 − p θ , q ( X ) } ≥ c g r r 0 d − 1 16 ( C g r + 4 ) 2 ε L 2 L . \mathbb{E}\{p_{\theta,q}(X)\}\wedge\mathbb{E}\{1-p_{\theta,q}(X)\}\ge\frac{c_{\mathrm{gr}}r_0^{d-1}}{16(C_{\mathrm{gr}}+4)^2}\frac{\varepsilon_L^2}{L}. E { p θ , q ( X )} ∧ E { 1 − p θ , q ( X )} ≥ 16 ( C gr + 4 ) 2 c gr r 0 d − 1 L ε L 2 .
Taking c p u l l = c g r r 0 d − 1 / { 16 ( C g r + 4 ) 2 } c_{\mathrm{pull}}=c_{\mathrm{gr}}r_0^{d-1}/\{16(C_{\mathrm{gr}}+4)^2\} c pull = c gr r 0 d − 1 / { 16 ( C gr + 4 ) 2 } proves the first claim.
In the case when L ≥ 2 / ρ ~ 0 L\ge2/\widetilde{\rho}_0 L ≥ 2/ ρ 0 , we have that ε L = 1 \varepsilon_L=1 ε L = 1 . Hence the previous display gives
E { p θ , q ( X ) } ∧ E { 1 − p θ , q ( X ) } ≥ c p u l l L . \mathbb{E}\{p_{\theta,q}(X)\}\wedge\mathbb{E}\{1-p_{\theta,q}(X)\}\ge\frac{c_{\mathrm{pull}}}{L}. E { p θ , q ( X )} ∧ E { 1 − p θ , q ( X )} ≥ L c pull .
On the other hand, when L < 2 / ρ ~ 0 L<2/\widetilde{\rho}_0 L < 2/ ρ 0 , we have that ε L = L ρ ~ 0 / 2 \varepsilon_L=L\widetilde{\rho}_0/2 ε L = L ρ 0 /2 . Hence the previous display gives
E { p θ , q ( X ) } ∧ E { 1 − p θ , q ( X ) } ≥ c p u l l ( L ρ ~ 0 / 2 ) 2 L = c p u l l ρ ~ 0 2 4 L = c p u l l ′ L , \mathbb{E}\{p_{\theta,q}(X)\}\wedge\mathbb{E}\{1-p_{\theta,q}(X)\}\ge c_{\mathrm{pull}}\frac{(L\widetilde{\rho}_0/2)^2}{L}=\frac{c_{\mathrm{pull}}\widetilde{\rho}_0^2}{4}L=c'_{\mathrm{pull}}L, E { p θ , q ( X )} ∧ E { 1 − p θ , q ( X )} ≥ c pull L ( L ρ 0 /2 ) 2 = 4 c pull ρ 0 2 L = c pull ′ L ,
where c p u l l ′ = c p u l l ρ ~ 0 2 / 4 c'_{\mathrm{pull}}=c_{\mathrm{pull}}\widetilde{\rho}_0^2/4 c pull ′ = c pull ρ 0 2 /4 . This completes the proof.
Lemma 30 (Norm comparison). Fix ν ∈ [ 1 , ∞ ] \nu\in[1,\infty] ν ∈ [ 1 , ∞ ] . Let C d , ν = d max { 1 / ν − 1 / 2 , 0 } C_{d,\nu}=d^{\max\{1/\nu-1/2,0\}} C d , ν = d m a x { 1/ ν − 1/2 , 0 } , where 1 / ∞ = 0 1/\infty=0 1/∞ = 0 . Then, for every z ∈ R d z\in\mathbb{R}^d z ∈ R d ,
∥ z ∥ ν ≤ C d , ν ∥ z ∥ 2 . \|z\|_\nu\le C_{d,\nu}\|z\|_2. ∥ z ∥ ν ≤ C d , ν ∥ z ∥ 2 . The same bound also holds for vectors in R d − 1 \mathbb{R}^{d-1} R d − 1 , after viewing them as vectors in R d \mathbb{R}^d R d by adding one zero coordinate.
Proof. If ν ≥ 2 \nu\ge2 ν ≥ 2 , then the monotonicity of ℓ p \ell_p ℓ p norms gives ∥ z ∥ ν ≤ ∥ z ∥ 2 \|z\|_\nu\le\|z\|_2 ∥ z ∥ ν ≤ ∥ z ∥ 2 , so the claim holds because C d , ν = 1 C_{d,\nu}=1 C d , ν = 1 . If 1 ≤ ν < 2 1\le\nu<2 1 ≤ ν < 2 , Hölder’s inequality gives
∥ z ∥ ν = ( ∑ j = 1 d ∣ z j ∣ ν ) 1 / ν ≤ d 1 / ν − 1 / 2 ( ∑ j = 1 d z j 2 ) 1 / 2 = C d , ν ∥ z ∥ 2 . \|z\|_\nu=\left(\sum_{j=1}^{d}|z_j|^\nu\right)^{1/\nu}\le d^{1/\nu-1/2}\left(\sum_{j=1}^{d}z_j^2\right)^{1/2}=C_{d,\nu}\|z\|_2. ∥ z ∥ ν = ( j = 1 ∑ d ∣ z j ∣ ν ) 1/ ν ≤ d 1/ ν − 1/2 ( j = 1 ∑ d z j 2 ) 1/2 = C d , ν ∥ z ∥ 2 . This proves the claim.
Proof of Lemma 5
Proof of Lemma 5. By (16) and (18) , we have that
F s ( b 1 , b 2 ) = max z ∈ Z s − 1 , L 1 s − 1 ∑ i = 1 s − 1 [ X ~ i ⊤ b 1 z i + X ~ i ⊤ b 2 ( 1 − z i ) ] . F_s(b_1,b_2)=\max_{z\in\mathcal{Z}_{s-1,L}}\frac{1}{s-1}\sum_{i=1}^{s-1}\left[\widetilde{X}_i^\top b_1z_i+\widetilde{X}_i^\top b_2(1-z_i)\right]. F s ( b 1 , b 2 ) = z ∈ Z s − 1 , L max s − 1 1 i = 1 ∑ s − 1 [ X i ⊤ b 1 z i + X i ⊤ b 2 ( 1 − z i ) ] .
For any fixed z ∈ Z s − 1 , L z\in\mathcal{Z}_{s-1,L} z ∈ Z s − 1 , L , the expression inside the maximum is affine in ( b 1 , b 2 ) (b_1,b_2) ( b 1 , b 2 ) , meaning that it depends linearly on b 1 b_1 b 1 and b 2 b_2 b 2 . In particular, for any two model pairs ( b 1 , b 2 ) (b_1,b_2) ( b 1 , b 2 ) and ( b 1 ′ , b 2 ′ ) (b'_1,b'_2) ( b 1 ′ , b 2 ′ ) and any λ ∈ [ 0 , 1 ] \lambda\in[0,1] λ ∈ [ 0 , 1 ] , its value at the convex combination λ b 1 + ( 1 − λ ) b 1 ′ , λ b 2 + ( 1 − λ ) b 2 ′ \lambda b_1+(1-\lambda)b'_1,\lambda b_2+(1-\lambda)b'_2 λ b 1 + ( 1 − λ ) b 1 ′ , λ b 2 + ( 1 − λ ) b 2 ′ is exactly
1 s − 1 ∑ i = 1 s − 1 [ X ~ i ⊤ λ b 1 + ( 1 − λ ) b 1 ′ z i + X ~ i ⊤ λ b 2 + ( 1 − λ ) b 2 ′ ( 1 − z i ) ] = λ s − 1 ∑ i = 1 s − 1 [ X ~ i ⊤ b 1 z i + X ~ i ⊤ b 2 ( 1 − z i ) ] + 1 − λ s − 1 ∑ i = 1 s − 1 [ X ~ i ⊤ b 1 ′ z i + X ~ i ⊤ b 2 ′ ( 1 − z i ) ] . \begin{aligned}
\frac{1}{s-1}\sum_{i=1}^{s-1}\left[\widetilde{X}_i^\top\lambda b_1+(1-\lambda)b'_1z_i+\widetilde{X}_i^\top\lambda b_2+(1-\lambda)b'_2(1-z_i)\right] \\
&= \frac{\lambda}{s-1}\sum_{i=1}^{s-1}\left[\tilde{X}_i^\top b_1 z_i+\tilde{X}_i^\top b_2(1-z_i)\right]+\frac{1-\lambda}{s-1}\sum_{i=1}^{s-1}\left[\tilde{X}_i^\top b'_1 z_i+\tilde{X}_i^\top b'_2(1-z_i)\right].
\end{aligned} s − 1 1 i = 1 ∑ s − 1 [ X i ⊤ λ b 1 + ( 1 − λ ) b 1 ′ z i + X i ⊤ λ b 2 + ( 1 − λ ) b 2 ′ ( 1 − z i ) ] = s − 1 λ i = 1 ∑ s − 1 [ X ~ i ⊤ b 1 z i + X ~ i ⊤ b 2 ( 1 − z i ) ] + s − 1 1 − λ i = 1 ∑ s − 1 [ X ~ i ⊤ b 1 ′ z i + X ~ i ⊤ b 2 ′ ( 1 − z i ) ] .
Since the feasible set Z s − 1 , L \mathcal{Z}_{s-1,L} Z s − 1 , L is the same for every choice of ( b 1 , b 2 ) (b_1,b_2) ( b 1 , b 2 ) , taking the maximum over z z z on the left-hand side cannot exceed taking the maximum separately in the two terms on the right-hand side. Therefore, it holds that F s F_s F s is jointly convex. Continuity follows from the compactness of Z s − 1 , L \mathcal{Z}_{s-1,L} Z s − 1 , L and the continuity of the objective.
By Lemma 31 , any feasible pair can be written as
b 1 = ∑ v 1 ∈ V s , 1 a v 1 v 1 , b 2 = ∑ v 2 ∈ V s , 2 c v 2 v 2 . b_1=\sum_{v_1\in\mathcal{V}_{s,1}}a_{v_1}v_1,\qquad b_2=\sum_{v_2\in\mathcal{V}_{s,2}}c_{v_2}v_2. b 1 = v 1 ∈ V s , 1 ∑ a v 1 v 1 , b 2 = v 2 ∈ V s , 2 ∑ c v 2 v 2 .
where the coefficients are nonnegative and each collection sums to one. Combining these two representations gives
( b 1 , b 2 ) = ∑ v 1 ∈ V s , 1 ∑ v 2 ∈ V s , 2 a v 1 c v 2 ( v 1 , v 2 ) . (b_1,b_2)=\sum_{v_1\in\mathcal{V}_{s,1}}\sum_{v_2\in\mathcal{V}_{s,2}}a_{v_1}c_{v_2}(v_1,v_2). ( b 1 , b 2 ) = v 1 ∈ V s , 1 ∑ v 2 ∈ V s , 2 ∑ a v 1 c v 2 ( v 1 , v 2 ) .
The coefficients a v 1 c v 2 a_{v_1}c_{v_2} a v 1 c v 2 are nonnegative and satisfy
∑ v 1 ∈ V s , 1 ∑ v 2 ∈ V s , 2 a v 1 c v 2 = ( ∑ v 1 ∈ V s , 1 a v 1 ) ( ∑ v 2 ∈ V s , 2 c v 2 ) = 1. \sum_{v_1\in\mathcal{V}_{s,1}}\sum_{v_2\in\mathcal{V}_{s,2}}a_{v_1}c_{v_2}=\left(\sum_{v_1\in\mathcal{V}_{s,1}}a_{v_1}\right)\left(\sum_{v_2\in\mathcal{V}_{s,2}}c_{v_2}\right)=1. v 1 ∈ V s , 1 ∑ v 2 ∈ V s , 2 ∑ a v 1 c v 2 = v 1 ∈ V s , 1 ∑ a v 1 v 2 ∈ V s , 2 ∑ c v 2 = 1.
Thus, every feasible pair ( b 1 , b 2 ) (b_1,b_2) ( b 1 , b 2 ) is a convex combination of the vertex pairs ( v 1 , v 2 ) (v_1,v_2) ( v 1 , v 2 ) . Joint convexity therefore gives
F s ( b 1 , b 2 ) ≤ ∑ v 1 ∈ V s , 1 ∑ v 2 ∈ V s , 2 a v 1 c v 2 F s ( v 1 , v 2 ) ≤ max v k ∈ V s , k , k ∈ { 1 , 2 } F s ( v 1 , v 2 ) . F_s(b_1,b_2)\leq\sum_{v_1\in\mathcal{V}_{s,1}}\sum_{v_2\in\mathcal{V}_{s,2}}a_{v_1}c_{v_2}F_s(v_1,v_2)\leq\max_{v_k\in\mathcal{V}_{s,k},\,k\in\{1,2\}}F_s(v_1,v_2). F s ( b 1 , b 2 ) ≤ v 1 ∈ V s , 1 ∑ v 2 ∈ V s , 2 ∑ a v 1 c v 2 F s ( v 1 , v 2 ) ≤ v k ∈ V s , k , k ∈ { 1 , 2 } max F s ( v 1 , v 2 ) .
where the second inequality follows because the middle expression is a convex combination of the values F s ( v 1 , v 2 ) F_s(v_1,v_2) F s ( v 1 , v 2 ) and therefore cannot exceed their largest value. Since every vertex pair is feasible, this proves (19) .
Suppose the selected model pair satisfies H β ˉ s , 1 − β ˉ s , 2 ≠ ∅ \mathcal{H}_{\bar{\beta}_{s,1-\bar{\beta}_{s,2}}}\ne\varnothing H β ˉ s , 1 − β ˉ s , 2 = ∅ . By Theorem 3 , every feasible canonical policy p b 1 − b 2 , q p_{b_1-b_2,q} p b 1 − b 2 , q belongs to P L , ν \mathcal{P}_{L,\nu} P L , ν , so its sample values belong to Z s − 1 , L \mathcal{Z}_{s-1,L} Z s − 1 , L . Hence, its empirical objective is bounded above by F s ( b 1 , b 2 ) F_s(b_1,b_2) F s ( b 1 , b 2 ) , and the supremum on the right-hand side of (20) is bounded above by the left-hand side of (19) . For every vertex pair v v v , the optimality of z v z_v z v gives
S v = 1 s − 1 ∑ i = 1 s − 1 X ~ i ⊤ v 2 + H s ( v 1 − v 2 ) = F s ( v 1 , v 2 ) . S_v=\frac{1}{s-1}\sum_{i=1}^{s-1}\tilde{X}_i^\top v_2+H_s(v_1-v_2)=F_s(v_1,v_2). S v = s − 1 1 i = 1 ∑ s − 1 X ~ i ⊤ v 2 + H s ( v 1 − v 2 ) = F s ( v 1 , v 2 ) .
If v ∘ v^\circ v ∘ maximises F s F_s F s over the vertex pairs, the selected pair satisfies S v ˉ = S v ∘ = F s ( v 1 ∘ , v 2 ∘ ) S_{\bar{v}}=S_{v^\circ}=F_s(v_1^\circ,v_2^\circ) S v ˉ = S v ∘ = F s ( v 1 ∘ , v 2 ∘ ) . The McShane extension of z s z_s z s belongs to P L , ν \mathcal{P}_{L,\nu} P L , ν and agrees with z s z_s z s at the sample points. By Theorem 3 , its boundary-canonical representative does not decrease the empirical objective. Hence, it holds that
1 s − 1 ∑ i = 1 s − 1 r β ˉ s , p s ( X i ) ≥ S v ˉ = max b k ∈ C s , k , k ∈ { 1 , 2 } F s ( b 1 , b 2 ) . \frac{1}{s-1}\sum_{i=1}^{s-1}r_{\bar{\beta}_s,p_s}(X_i)\geq S_{\bar{v}}=\max_{b_k\in\mathcal{C}_{s,k},\,k\in\{1,2\}}F_s(b_1,b_2). s − 1 1 i = 1 ∑ s − 1 r β ˉ s , p s ( X i ) ≥ S v ˉ = b k ∈ C s , k , k ∈ { 1 , 2 } max F s ( b 1 , b 2 ) .
Since the returned model-policy pair is feasible in (15) , the reverse inequality also holds. This proves (20) .
Polyhedral confidence sets
Lemma 31 (Polyhedral confidence sets). For every update time s s s and k ∈ { 1 , 2 } k \in\{1,2\} k ∈ { 1 , 2 } , we have that
{ b : ∥ b − β ^ s , k ∥ V s , k ≤ α T } ⊆ C s , k = conv ( V s , k ) , \left\{b : \|b-\widehat{\beta}_{s,k}\|_{V_{s,k}} \le\alpha_T\right\} \subseteq C_{s,k} = \operatorname{conv}(\mathcal{V}_{s,k}), { b : ∥ b − β s , k ∥ V s , k ≤ α T } ⊆ C s , k = conv ( V s , k ) , where C s , k C_{s,k} C s , k is defined in (14) , V s , k \mathcal{V}_{s,k} V s , k is defined in (12) , and conv ( ⋅ ) \operatorname{conv}(\cdot) conv ( ⋅ ) denotes the convex hull. Suppose ∥ β ^ s , k − β k ∗ ∥ V s , k ≤ α T \|\widehat{\beta}_{s,k}-\beta_k^*\|_{V_{s,k}} \le\alpha_T ∥ β s , k − β k ∗ ∥ V s , k ≤ α T , then it holds that β k ∗ ∈ C s , k \beta_k^* \in C_{s,k} β k ∗ ∈ C s , k and, for every b k ∈ C s , k b_k \in C_{s,k} b k ∈ C s , k ,
∥ b k − β k ∗ ∥ V s , k ≤ ( 1 + d + 1 ) α T . (49) \|b_k-\beta_k^*\|_{V_{s,k}} \le(1+\sqrt{d+1})\alpha_T. \tag*{(49)} ∥ b k − β k ∗ ∥ V s , k ≤ ( 1 + d + 1 ) α T . (49) Proof. For every u ∈ R d + 1 u \in\mathbb{R}^{d+1} u ∈ R d + 1 , we have ∥ u ∥ 2 ≤ ∥ u ∥ 1 ≤ d + 1 ∥ u ∥ 2 \|u\|_2 \le\|u\|_1 \le\sqrt{d+1}\|u\|_2 ∥ u ∥ 2 ≤ ∥ u ∥ 1 ≤ d + 1 ∥ u ∥ 2 . Note that C s , k = { b : ∥ V s , k 1 / 2 ( b − β ^ s , k ) ∥ 1 ≤ d + 1 α T } C_{s,k} = \{b : \|V_{s,k}^{1/2}(b-\widehat{\beta}_{s,k})\|_1 \le\sqrt{d+1}\alpha_T\} C s , k = { b : ∥ V s , k 1/2 ( b − β s , k ) ∥ 1 ≤ d + 1 α T } , applying the inequality to u = V s , k 1 / 2 ( b − β ^ s , k ) u = V_{s,k}^{1/2}(b-\widehat{\beta}_{s,k}) u = V s , k 1/2 ( b − β s , k ) proves the first claim.
We next show that C s , k = conv ( V s , k ) C_{s,k} = \operatorname{conv}(\mathcal{V}_{s,k}) C s , k = conv ( V s , k ) . Under the transformation u = V s , k 1 / 2 ( b − β ^ s , k ) u = V_{s,k}^{1/2}(b-\widehat{\beta}_{s,k}) u = V s , k 1/2 ( b − β s , k ) , the set C s , k C_{s,k} C s , k corresponds to the ℓ 1 \ell_1 ℓ 1 -ball B 1 ( d + 1 α T ) = { u ∈ R d + 1 : ∥ u ∥ 1 ≤ d + 1 α T } B_1(\sqrt{d+1}\alpha_T) = \{u \in\mathbb{R}^{d+1} : \|u\|_1 \le\sqrt{d+1}\alpha_T\} B 1 ( d + 1 α T ) = { u ∈ R d + 1 : ∥ u ∥ 1 ≤ d + 1 α T } , i.e. C s , k = β ^ s , k + V s , k − 1 / 2 B 1 ( d + 1 α T ) C_{s,k} = \widehat{\beta}_{s,k} + V_{s,k}^{-1/2}B_1(\sqrt{d+1}\alpha_T) C s , k = β s , k + V s , k − 1/2 B 1 ( d + 1 α T ) . Note that this ℓ 1 \ell_1 ℓ 1 -ball is the convex hull of its 2 ( d + 1 ) 2(d+1) 2 ( d + 1 ) vertices { ± d + 1 α T e r : r ∈ [ d + 1 ] } \{\pm\sqrt{d+1}\alpha_T e_r : r \in[d+1]\} { ± d + 1 α T e r : r ∈ [ d + 1 ]} . To see this, for any u u u in the ball, the contribution of its r r r th coordinate can be written as
∣ u r ∣ d + 1 α T { sign ( u r ) d + 1 α T e r } = u r e r . \frac{|u_r|}{\sqrt{d+1}\alpha_T}\{\operatorname{sign}(u_r)\sqrt{d+1}\alpha_T e_r\}=u_r e_r. d + 1 α T ∣ u r ∣ { sign ( u r ) d + 1 α T e r } = u r e r . Summing over r r r therefore recovers u u u , while the corresponding nonnegative coefficients have total weight ∥ u ∥ 1 / ( d + 1 α T ) ≤ 1 \|u\|_1/(\sqrt{d+1}\alpha_T) \le1 ∥ u ∥ 1 / ( d + 1 α T ) ≤ 1 . If this total weight is strictly smaller than one, the remaining weight can be split equally between any pair of opposite vertices, e.g. d + 1 α T e 1 \sqrt{d+1}\alpha_T e_1 d + 1 α T e 1 and − d + 1 α T e 1 -\sqrt{d+1}\alpha_T e_1 − d + 1 α T e 1 . Since these two vertices sum to zero, assigning equal additional weight to them leaves u u u unchanged while making the total weight equal to one. Hence every point in the ℓ 1 \ell_1 ℓ 1 -ball is a convex combination of these 2 ( d + 1 ) 2(d+1) 2 ( d + 1 ) vertices. Conversely, all these vertices belong to the ℓ 1 \ell_1 ℓ 1 -ball, which is convex, so the ℓ 1 \ell_1 ℓ 1 -ball is exactly their convex hull.
Since u = V s , k 1 / 2 ( b − β ^ s , k ) u = V_{s,k}^{1/2}(b-\widehat{\beta}_{s,k}) u = V s , k 1/2 ( b − β s , k ) by construction, we equivalently have b = β ^ s , k + V s , k − 1 / 2 u b = \widehat{\beta}_{s,k} + V_{s,k}^{-1/2}u b = β s , k + V s , k − 1/2 u . Therefore, the vertices ± d + 1 α T e r \pm\sqrt{d+1}\alpha_T e_r ± d + 1 α T e r are mapped to β ^ s , k ± d + 1 α T V s , k − 1 / 2 e r \widehat{\beta}_{s,k} \pm\sqrt{d+1}\alpha_T V_{s,k}^{-1/2}e_r β s , k ± d + 1 α T V s , k − 1/2 e r . These transformed vertices form the set
V s , k = { β ^ s , k ± d + 1 α T V s , k − 1 / 2 e r : r ∈ [ d + 1 ] } . \mathcal{V}_{s,k}=\left\{\widehat{\beta}_{s,k}\pm\sqrt{d+1}\alpha_T V_{s,k}^{-1/2}e_r:r\in[d+1]\right\}. V s , k = { β s , k ± d + 1 α T V s , k − 1/2 e r : r ∈ [ d + 1 ] } . Since affine transformations preserve convex combinations, this gives C s , k = conv ( V s , k ) C_{s,k}=\operatorname{conv}(\mathcal{V}_{s,k}) C s , k = conv ( V s , k ) .
Suppose ∥ β ^ s , k − β k ∗ ∥ V s , k ≤ α T \|\widehat{\beta}_{s,k}-\beta_k^*\|_{V_{s,k}} \le\alpha_T ∥ β s , k − β k ∗ ∥ V s , k ≤ α T . The containment proved above gives β k ∗ ∈ C s , k \beta_k^* \in C_{s,k} β k ∗ ∈ C s , k . Finally, for every b k ∈ C s , k b_k \in C_{s,k} b k ∈ C s , k , we have ∥ b k − β ^ s , k ∥ V s , k ≤ d + 1 α T \|b_k-\widehat{\beta}_{s,k}\|_{V_{s,k}} \le\sqrt{d+1}\alpha_T ∥ b k − β s , k ∥ V s , k ≤ d + 1 α T , and the triangle inequality gives
∥ b k − β k ∗ ∥ V s , k ≤ ∥ b k − β ^ s , k ∥ V s , k + ∥ β ^ s , k − β k ∗ ∥ V s , k \|b_k-\beta_k^*\|_{V_{s,k}} \le\|b_k-\widehat{\beta}_{s,k}\|_{V_{s,k}}+\|\widehat{\beta}_{s,k}-\beta_k^*\|_{V_{s,k}} ∥ b k − β k ∗ ∥ V s , k ≤ ∥ b k − β s , k ∥ V s , k + ∥ β s , k − β k ∗ ∥ V s , k ≤ d + 1 α T + α T = ( 1 + d + 1 ) α T . \le\sqrt{d+1}\alpha_T+\alpha_T=(1+\sqrt{d+1})\alpha_T. ≤ d + 1 α T + α T = ( 1 + d + 1 ) α T . This completes the proof.
Lemma 32 (Computational complexity). It holds that each update in Algorithm 4 requires solving 4 ( d + 1 ) 2 4(d+1)^2 4 ( d + 1 ) 2 LPs with s − 1 s-1 s − 1 variables and at most s ( s − 1 ) s(s-1) s ( s − 1 ) linear inequalities. With s 0 = 2 T 0 + 1 ≤ T s_0=2T_0+1\le T s 0 = 2 T 0 + 1 ≤ T , the number K T K_T K T of updates in Algorithm 3 satisfies
K T ≤ 1 + log ( T / s 0 ) log 2 + 2 ( d + 1 ) log 2 log ( 1 + T λ ) . (50) K_T \le1+\frac{\log(T/s_0)}{\log2}+\frac{2(d+1)}{\log2}\log\left(1+\frac{T}{\lambda}\right). \tag*{(50)} K T ≤ 1 + log 2 log ( T / s 0 ) + log 2 2 ( d + 1 ) log ( 1 + λ T ) . (50) Consequently, the total number of policy LPs is at most 4 ( d + 1 ) 2 K T 4(d+1)^2K_T 4 ( d + 1 ) 2 K T .
Proof. Each confidence polytope has 2 ( d + 1 ) 2(d+1) 2 ( d + 1 ) vertices, giving 4 ( d + 1 ) 2 4(d+1)^2 4 ( d + 1 ) 2 pairs. For n = s − 1 n=s-1 n = s − 1 , the constraints in (17) consist of 2 n 2n 2 n box inequalities and n ( n − 1 ) n(n-1) n ( n − 1 ) pairwise Lipschitz inequalities, for a total of n ( n + 1 ) = s ( s − 1 ) n(n+1)=s(s-1) n ( n + 1 ) = s ( s − 1 ) . The two inverse square roots and the pairwise distances are computed once per update.
To count the total number of updates, write s 1 = s 0 < s 2 < ⋯ < s K T ≤ T s_1=s_0<s_2<\cdots<s_{K_T}\le T s 1 = s 0 < s 2 < ⋯ < s K T ≤ T . We will update when index doubles or the determinant doubles. If K 0 K_0 K 0 updates are assigned to the time trigger, then
K 0 log 2 ≤ ∑ j = 1 K T − 1 log ( s j + 1 / s j ) ≤ log ( T / s 0 ) . K_0\log2\le\sum_{j=1}^{K_T-1}\log(s_{j+1}/s_j)\le\log(T/s_0). K 0 log 2 ≤ j = 1 ∑ K T − 1 log ( s j + 1 / s j ) ≤ log ( T / s 0 ) .
If K k K_k K k updates are assigned to arm k k k , monotonicity of V s , k V_{s,k} V s , k gives
K k log 2 ≤ ∑ j = 1 K T − 1 log det ( V s j + 1 , k ) det ( V s j , k ) ≤ log det ( V T , k ) λ d + 1 ≤ ( d + 1 ) log ( 1 + T λ ) , K_k\log2\le\sum_{j=1}^{K_T-1}\log\frac{\det(V_{s_{j+1},k})}{\det(V_{s_j,k})}\le\log\frac{\det(V_{T,k})}{\lambda^{d+1}}\le(d+1)\log\left(1+\frac{T}{\lambda}\right), K k log 2 ≤ j = 1 ∑ K T − 1 log det ( V s j , k ) det ( V s j + 1 , k ) ≤ log λ d + 1 det ( V T , k ) ≤ ( d + 1 ) log ( 1 + λ T ) ,
where the last inequality follows from the trace bound tr ( V T , k ) ≤ ( d + 1 ) ( λ + T ) \operatorname{tr}(V_{T,k})\le(d+1)(\lambda+T) tr ( V T , k ) ≤ ( d + 1 ) ( λ + T ) and the arithmetic–geometric mean inequality for eigenvalues, similar as in Lemma 47. Since K T = 1 + K 0 + K 1 + K 2 K_T=1+K_0+K_1+K_2 K T = 1 + K 0 + K 1 + K 2 , this proves (50) .
Proof of Theorem 6
For this subsection, let τ ( t ) = max { s ≤ t : s is an update time of Algorithm 3 } \tau(t)=\max\{s\le t:s\text{ is an update time of Algorithm 3}\} τ ( t ) = max { s ≤ t : s is an update time of Algorithm 3 } denote the most recent update time before t t t , and let F t = σ { X i , A i , Y i : 1 ≤ i ≤ t } \mathcal{F}_t=\sigma\{X_i,A_i,Y_i:1\le i\le t\} F t = σ { X i , A i , Y i : 1 ≤ i ≤ t } .
Proof of Theorem 6. Let B = d + 1 ( C β + 1 ) B=\sqrt{d+1}(C_\beta+1) B = d + 1 ( C β + 1 ) and fix r l o c ∈ ( 0 , 1 ] r_{\mathrm{loc}}\in(0,1] r loc ∈ ( 0 , 1 ] as in Lemma 14. By Theorem 3, there exists a population-optimal canonical policy p ∗ = p θ ∗ , q ∗ p^*=p_{\theta^*,q^*} p ∗ = p θ ∗ , q ∗ . During the first 2 T o 2T_o 2 T o rounds, the algorithm uses p t = 1 / 2 p_t=1/2 p t = 1/2 . Since the conditional mean reward of each arm has magnitude at most B B B , the regret in each exploration round is at most 2 B 2B 2 B . Hence, since T o = ⌈ T ⌉ T_o=\lceil\sqrt{T}\rceil T o = ⌈ T ⌉ , the total regret during exploration is at most 4 B T o ≤ C 1 T 4BT_o\le C_1\sqrt{T} 4 B T o ≤ C 1 T , where C 1 = 8 B C_1=8B C 1 = 8 B .
High-probability events. We first construct a high-probability event E = E c o n f ∩ E e m p ∩ E m a r t ∩ E i n i t \mathcal{E}=\mathcal{E}_{\mathrm{conf}}\cap\mathcal{E}_{\mathrm{emp}}\cap\mathcal{E}_{\mathrm{mart}}\cap\mathcal{E}_{\mathrm{init}} E = E conf ∩ E emp ∩ E mart ∩ E init , on which the remaining analysis is carried out.
Define
E c o n f = { ∥ β ^ t , k − β k ∗ ∥ V t , k ≤ α T , for every t ∈ [ T ] and k ∈ { 1 , 2 } } . (51) \mathcal{E}_{\mathrm{conf}}=\left\{\|\widehat{\beta}_{t,k}-\beta_k^*\|_{V_{t,k}}\le\alpha_T,\text{ for every }t\in[T]\text{ and }k\in\{1,2\}\right\}. \tag*{(51)} E conf = { ∥ β t , k − β k ∗ ∥ V t , k ≤ α T , for every t ∈ [ T ] and k ∈ { 1 , 2 } } . (51)
Applying Corollary 44 with failure probability η c = η / 8 \eta_c=\eta/8 η c = η /8 , it then holds with probability at least 1 − η / 8 1-\eta/8 1 − η /8 that for every t ∈ [ T ] t\in[T] t ∈ [ T ] and k ∈ { 1 , 2 } k\in\{1,2\} k ∈ { 1 , 2 } ,
∥ S t , k ∥ V t , k − 1 ≤ 2 σ 2 { log ( 2 η c ) + d + 1 2 log ( 1 + T λ ) } . \|S_{t,k}\|_{V_{t,k}^{-1}}\le\sqrt{2\sigma^2\left\{\log\left(\frac{2}{\eta_c}\right)+\frac{d+1}{2}\log\left(1+\frac{T}{\lambda}\right)\right\}}. ∥ S t , k ∥ V t , k − 1 ≤ 2 σ 2 { log ( η c 2 ) + 2 d + 1 log ( 1 + λ T ) } .
Combining this with (69) and ∥ β k ∗ ∥ 2 ≤ C β \|\beta_k^*\|_2\le C_\beta ∥ β k ∗ ∥ 2 ≤ C β from Assumption 3, the definition of α T \alpha_T α T in (11) gives
∥ β ^ t , k − β k ∗ ∥ V t , k ≤ α T \|\widehat{\beta}_{t,k}-\beta_k^*\|_{V_{t,k}}\le\alpha_T ∥ β t , k − β k ∗ ∥ V t , k ≤ α T
for every t ∈ [ T ] t\in[T] t ∈ [ T ] and k ∈ { 1 , 2 } k\in\{1,2\} k ∈ { 1 , 2 } . Hence P ( E c o n f ) ≥ 1 − η / 8 P(\mathcal{E}_{\mathrm{conf}})\ge1-\eta/8 P ( E conf ) ≥ 1 − η /8 . On E c o n f \mathcal{E}_{\mathrm{conf}} E conf , Lemma 31 gives β k ∗ ∈ C s , k \beta_k^*\in\mathcal{C}_{s,k} β k ∗ ∈ C s , k for every update time s s s and k ∈ { 1 , 2 } k\in\{1,2\} k ∈ { 1 , 2 } .
Next, with Δ n U C B \Delta_n^{\mathrm{UCB}} Δ n UCB as in (67) , define
E e m p = { Δ n U C B ≤ C 2 ρ n , d − 1 ( η 8 T ) , for every n ∈ { 2 , … , T } } . (52) \mathcal{E}_{\mathrm{emp}}=\left\{\Delta_n^{\mathrm{UCB}}\le C_2\rho_{n,d-1}\left(\frac{\eta}{8T}\right),\text{ for every }n\in\{2,\ldots,T\}\right\}. \tag*{(52)} E emp = { Δ n UCB ≤ C 2 ρ n , d − 1 ( 8 T η ) , for every n ∈ { 2 , … , T } } . (52)
By Lemma 42, for each n ∈ { 2 , … , T } n \in\{2,\ldots,T\} n ∈ { 2 , … , T } the displayed inequality fails with probability at most η / ( 8 T ) \eta/(8T) η / ( 8 T ) . A union bound over these T − 1 T-1 T − 1 values of n n n therefore gives P ( E e m p ) ≥ 1 − ( T − 1 ) η / ( 8 T ) ≥ 1 − η / 8 P(\mathcal{E}_{\mathrm{emp}}) \ge1-(T-1)\eta/(8T) \ge1-\eta/8 P ( E emp ) ≥ 1 − ( T − 1 ) η / ( 8 T ) ≥ 1 − η /8 .
For s 0 ≤ t ≤ T s_0 \le t \le T s 0 ≤ t ≤ T , write W t = min { 1 , ∥ X ~ t ∥ V τ ( t ) , A t − 1 } W_t=\min\{1,\|\widetilde{X}_t\|_{V_{\tau(t),A_t}^{-1}}\} W t = min { 1 , ∥ X t ∥ V τ ( t ) , A t − 1 } , and define
E m a r t = { ∑ t = s 0 T E [ W t ∣ F t − 1 ] ≤ ∑ t = s 0 T W t + 2 T log ( 8 / η ) } . (53) \mathcal{E}_{\mathrm{mart}}=\left\{\sum_{t=s_0}^{T}\mathbb{E}[W_t\mid\mathcal{F}_{t-1}]\le\sum_{t=s_0}^{T}W_t+\sqrt{2T\log(8/\eta)}\right\}. \tag*{(53)} E mart = { t = s 0 ∑ T E [ W t ∣ F t − 1 ] ≤ t = s 0 ∑ T W t + 2 T log ( 8/ η ) } . (53)
Since W t W_t W t is F t \mathcal{F}_t F t -measurable and 0 ≤ W t ≤ 1 0\le W_t\le1 0 ≤ W t ≤ 1 by construction, the variables Z t = E [ W t ∣ F t − 1 ] − W t Z_t=\mathbb{E}[W_t\mid\mathcal{F}_{t-1}]-W_t Z t = E [ W t ∣ F t − 1 ] − W t form a martingale difference sequence with ∣ Z t ∣ ≤ 1 |Z_t|\le1 ∣ Z t ∣ ≤ 1 . The Azuma–Hoeffding inequality (e.g. Corollary 2.20 in Wainwright, [51 ] ), applied to at most T T T increments, gives P ( E m a r t ) ≥ 1 − η / 8 P(\mathcal{E}_{\mathrm{mart}})\ge1-\eta/8 P ( E mart ) ≥ 1 − η /8 .
Finally, let E i n i t \mathcal{E}_{\mathrm{init}} E init be the event associated with the exploration phase in (57) . Since the first coordinate of X ~ \widetilde{X} X is one, we have that κ X ≤ 1 \kappa_X\le1 κ X ≤ 1 . By construction of α T \alpha_T α T in (11) , we have that α T 2 ≥ 2 σ 2 log ( 16 / η ) \alpha_T^2\ge2\sigma^2\log(16/\eta) α T 2 ≥ 2 σ 2 log ( 16/ η ) . Choose C 3 C_3 C 3 sufficiently large so that the condition
T ≥ C 3 ( 1 + d + 1 ) 4 α T 4 κ X 2 T\ge\frac{C_3(1+\sqrt{d+1})^4\alpha_T^4}{\kappa_X^2} T ≥ κ X 2 C 3 ( 1 + d + 1 ) 4 α T 4
implies
T ≥ 8 log ( 16 / η ) , T ≥ 16 ( d + 1 ) κ X log ( 16 ( d + 1 ) η ) . \sqrt{T}\ge8\log(16/\eta),\qquad
\sqrt{T}\ge\frac{16(d+1)}{\kappa_X}\log\left(\frac{16(d+1)}{\eta}\right). T ≥ 8 log ( 16/ η ) , T ≥ κ X 16 ( d + 1 ) log ( η 16 ( d + 1 ) ) .
Since T 0 ≥ T T_0\ge\sqrt{T} T 0 ≥ T , (58) and (59) then give P ( E i n i t c ) ≤ η / 4 P(\mathcal{E}_{\mathrm{init}}^c)\le\eta/4 P ( E init c ) ≤ η /4 . Set E = E c o n f ∩ E e m p ∩ E m a r t ∩ E i n i t \mathcal{E}=\mathcal{E}_{\mathrm{conf}}\cap\mathcal{E}_{\mathrm{emp}}\cap\mathcal{E}_{\mathrm{mart}}\cap\mathcal{E}_{\mathrm{init}} E = E conf ∩ E emp ∩ E mart ∩ E init . By a union bound,
P ( E ) ≥ 1 − η 8 − η 8 − η 8 − η 4 = 1 − 5 η 8 ≥ 1 − η . (54) P(\mathcal{E})\ge1-\frac{\eta}{8}-\frac{\eta}{8}-\frac{\eta}{8}-\frac{\eta}{4}
=1-\frac{5\eta}{8}\ge1-\eta. \tag*{(54)} P ( E ) ≥ 1 − 8 η − 8 η − 8 η − 4 η = 1 − 8 5 η ≥ 1 − η . (54)
Analysis on E \mathcal{E} E . The rest of the proof is carried out on E \mathcal{E} E . It remains to verify that, at every update time s s s , every difference θ = b 1 − b 2 \theta=b_1-b_2 θ = b 1 − b 2 with b k ∈ C s , k b_k\in C_{s,k} b k ∈ C s , k belongs to the fixed regular neighbourhood Θ ~ j ∗ \widetilde{\Theta}_{j^*} Θ j ∗ . Together with the bound ∥ β ˉ s , k ∥ 2 ≤ C β + 1 \|\bar{\beta}_{s,k}\|_2\le C_\beta+1 ∥ β ˉ s , k ∥ 2 ≤ C β + 1 from Lemma 34 and the construction of p s p_s p s , this ensures that r β ˉ s , p s ( x ) = x ~ s , 1 β ˉ s , 1 p s ( x ) + x ~ s , 2 { 1 − p s ( x ) } r_{\bar{\beta}_s,p_s}(x)=\widetilde{x}_{s,1}\bar{\beta}_{s,1}p_s(x)+\widetilde{x}_{s,2}\{1-p_s(x)\} r β ˉ s , p s ( x ) = x s , 1 β ˉ s , 1 p s ( x ) + x s , 2 { 1 − p s ( x )} belongs to the deterministic class F U C B \mathcal{F}_{\mathrm{UCB}} F UCB defined in Lemma 40 , so that the empirical-process bound in Lemma 42 applies. On E c o n f ∩ E i n i t \mathcal{E}_{\mathrm{conf}}\cap\mathcal{E}_{\mathrm{init}} E conf ∩ E init , (55) gives, for every update time s s s and every b k ∈ C s , k b_k\in C_{s,k} b k ∈ C s , k ,
∥ ( b 1 − b 2 ) − θ ∗ ∥ 2 ≤ 2 ( 1 + d + 1 ) α T λ + κ X T 0 / 4 ≤ 4 ( 1 + d + 1 ) α T κ X T 1 / 4 . \|(b_1-b_2)-\theta^*\|_2\le\frac{2(1+\sqrt{d+1})\alpha_T}{\sqrt{\lambda+\kappa_XT_0/4}}\le\frac{4(1+\sqrt{d+1})\alpha_T}{\sqrt{\kappa_XT^{1/4}}}. ∥ ( b 1 − b 2 ) − θ ∗ ∥ 2 ≤ λ + κ X T 0 /4 2 ( 1 + d + 1 ) α T ≤ κ X T 1/4 4 ( 1 + d + 1 ) α T .
The assumed lower bound on T T T yields
4 ( 1 + d + 1 ) α T κ X T 1 / 4 ≤ 4 C 3 1 / 4 ≤ 2 r l o c . \frac{4(1+\sqrt{d+1})\alpha_T}{\sqrt{\kappa_XT^{1/4}}}\le\frac{4}{C_3^{1/4}}\le2r_{\mathrm{loc}}. κ X T 1/4 4 ( 1 + d + 1 ) α T ≤ C 3 1/4 4 ≤ 2 r loc .
Thus, by Lemma 14 , b 1 − b 2 ∈ Θ ~ j ∗ b_1-b_2\in\widetilde{\Theta}_{j^*} b 1 − b 2 ∈ Θ j ∗ for every update time. Moreover, (55) and the same lower bound on T T T give ∥ b k − β k ∗ ∥ 2 ≤ r l o c ≤ 1 \|b_k-\beta^*_k\|_2\le r_{\mathrm{loc}}\le1 ∥ b k − β k ∗ ∥ 2 ≤ r loc ≤ 1 , and hence ∥ b k ∥ 2 ≤ C β + 1 \|b_k\|_2\le C_\beta+1 ∥ b k ∥ 2 ≤ C β + 1 for every candidate.
Applying Lemma 49 with s = τ ( t ) s=\tau(t) s = τ ( t ) and summing from t = s 0 t=s_0 t = s 0 to T T T gives
∑ t = s 0 T { R ( p ∗ ) − R ( p τ ( t ) ) } ≤ 2 ∑ t = s 0 T Δ τ ( t ) − 1 U C B + { ( 1 + d + 1 ) α T + 2 B } ∑ t = s 0 T E [ min { 1 , ∥ X ~ t ∥ V τ ( t ) , A t − 1 } ∣ F t − 1 ] . \sum_{t=s_0}^{T}\{R(p^*)-R(p_{\tau(t)})\}\le2\sum_{t=s_0}^{T}\Delta^{\mathrm{UCB}}_{\tau(t)-1}+\{(1+\sqrt{d+1})\alpha_T+2B\}\sum_{t=s_0}^{T}\mathbb{E}\left[\min\{1,\|\widetilde{X}_t\|_{V_{\tau(t),A_t}^{-1}}\}\mid\mathcal{F}_{t-1}\right]. t = s 0 ∑ T { R ( p ∗ ) − R ( p τ ( t ) )} ≤ 2 t = s 0 ∑ T Δ τ ( t ) − 1 UCB + {( 1 + d + 1 ) α T + 2 B } t = s 0 ∑ T E [ min { 1 , ∥ X t ∥ V τ ( t ) , A t − 1 } ∣ F t − 1 ] .
By the definition of E e m p \mathcal{E}_{\mathrm{emp}} E emp in (52) and Lemma 50 , with u = η / ( 8 T ) u=\eta/(8T) u = η / ( 8 T ) , it holds that
2 ∑ t = s 0 T Δ τ ( t ) − 1 U C B ≤ C 5 { T log ( e T / u ) , d = 2 , T log 3 / 2 ( e T / u ) , d = 3 , T ( d − 2 ) / ( d − 1 ) log ( e T / u ) , d ≥ 4. 2\sum_{t=s_0}^{T}\Delta^{\mathrm{UCB}}_{\tau(t)-1}\le C_5
\begin{cases}
\sqrt{T}\log(eT/u), & d=2,\\
\sqrt{T}\log^{3/2}(eT/u), & d=3,\\
T^{(d-2)/(d-1)}\log(eT/u), & d\ge4.
\end{cases} 2 t = s 0 ∑ T Δ τ ( t ) − 1 UCB ≤ C 5 ⎩ ⎨ ⎧ T log ( e T / u ) , T log 3/2 ( e T / u ) , T ( d − 2 ) / ( d − 1 ) log ( e T / u ) , d = 2 , d = 3 , d ≥ 4.
Since u = η / ( 8 T ) u=\eta/(8T) u = η / ( 8 T ) , we have log ( e T / u ) = log ( 8 e T 2 / η ) ≤ C 6 log ( T / η ) \log(eT/u)=\log(8eT^2/\eta)\le C_6\log(T/\eta) log ( e T / u ) = log ( 8 e T 2 / η ) ≤ C 6 log ( T / η ) for η < 1 / 2 \eta<1/2 η < 1/2 , where C 6 = 6 C_6=6 C 6 = 6 . Consequently,
2 ∑ t = s 0 T Δ τ ( t ) − 1 U C B ≤ C 7 { T log ( T / η ) , d = 2 , T log 3 / 2 ( T / η ) , d = 3 , T ( d − 2 ) / ( d − 1 ) log ( T / η ) , d ≥ 4. 2\sum_{t=s_0}^{T}\Delta^{\mathrm{UCB}}_{\tau(t)-1}\le C_7\begin{cases}\sqrt{T}\log(T/\eta),&d=2,\\ \sqrt{T}\log^{3/2}(T/\eta),&d=3,\\ T^{(d-2)/(d-1)}\log(T/\eta),&d\ge4.\end{cases} 2 t = s 0 ∑ T Δ τ ( t ) − 1 UCB ≤ C 7 ⎩ ⎨ ⎧ T log ( T / η ) , T log 3/2 ( T / η ) , T ( d − 2 ) / ( d − 1 ) log ( T / η ) , d = 2 , d = 3 , d ≥ 4.
For the remaining term, the event E m a r t \mathcal{E}_{\mathrm{mart}} E mart in (53) and Lemma 47 give
∑ t = s 0 T E [ min { 1 , ∥ X ~ t ∥ V τ ( t ) , A t − 1 } | F t − 1 ] ≤ C 8 T ( d + 1 ) log ( 1 + T λ ) + C 9 T log ( 8 / η ) . \sum_{t=s_0}^{T}\mathbb{E}\left[\min\left\{1,\left\|\widetilde{X}_t\right\|_{V^{-1}_{\tau(t),A_t}}\right\}\middle|\mathcal{F}_{t-1}\right]\le C_8\sqrt{T(d+1)\log\left(1+\frac{T}{\lambda}\right)}+C_9\sqrt{T\log(8/\eta)}. t = s 0 ∑ T E [ min { 1 , X t V τ ( t ) , A t − 1 } F t − 1 ] ≤ C 8 T ( d + 1 ) log ( 1 + λ T ) + C 9 T log ( 8/ η ) .
Since α T ≥ C β λ \alpha_T\ge C_\beta\sqrt{\lambda} α T ≥ C β λ , while λ \lambda λ is fixed, we have ( 1 + d + 1 ) α T + 2 B ≤ C 10 ( 1 + d + 1 ) α T (1+\sqrt{d+1})\alpha_T+2B\le C_{10}(1+\sqrt{d+1})\alpha_T ( 1 + d + 1 ) α T + 2 B ≤ C 10 ( 1 + d + 1 ) α T , where C 10 = 1 + 2 ( C β + 1 ) / ( C β λ ) C_{10}=1+2(C_\beta+1)/(C_\beta\sqrt{\lambda}) C 10 = 1 + 2 ( C β + 1 ) / ( C β λ ) . Therefore, using a + b ≤ 2 ( a + b ) \sqrt{a}+\sqrt{b}\le\sqrt{2(a+b)} a + b ≤ 2 ( a + b ) and setting C 11 = 2 C 10 max { C 8 , C 9 } C_{11}=\sqrt{2}C_{10}\max\{C_8,C_9\} C 11 = 2 C 10 max { C 8 , C 9 } ,
{ ( 1 + d + 1 ) α T + 2 B } ∑ t = s 0 T E [ min { 1 , ∥ X ~ t ∥ V τ ( t ) , A t − 1 } | F t − 1 ] ≤ C 11 ( 1 + d + 1 ) α T T { ( d + 1 ) log ( 1 + T λ ) + log ( 8 / η ) } . \begin{aligned}
&\{(1+\sqrt{d+1})\alpha_T+2B\}\sum_{t=s_0}^{T}\mathbb{E}\left[\min\left\{1,\left\|\widetilde{X}_t\right\|_{V^{-1}_{\tau(t),A_t}}\right\}\middle|\mathcal{F}_{t-1}\right]\\
&\qquad\le C_{11}(1+\sqrt{d+1})\alpha_T\sqrt{T\left\{(d+1)\log\left(1+\frac{T}{\lambda}\right)+\log(8/\eta)\right\}}.
\end{aligned} {( 1 + d + 1 ) α T + 2 B } t = s 0 ∑ T E [ min { 1 , X t V τ ( t ) , A t − 1 } F t − 1 ] ≤ C 11 ( 1 + d + 1 ) α T T { ( d + 1 ) log ( 1 + λ T ) + log ( 8/ η ) } .
Combining the preceding bounds with the exploration regret proves the bound in Theorem 6 .
Corollary 33. Under the conditions of Theorem 6 , for fixed model constants, it holds that
R T = { O ~ p ( T ) , d = 2 , 3 , O ~ p ( T ( d − 2 ) / ( d − 1 ) ) , d ≥ 4. \mathfrak{R}_T=\begin{cases}\widetilde{O}_{p}(\sqrt{T}),&d=2,3,\\ \widetilde{O}_{p}(T^{(d-2)/(d-1)}),&d\ge4.\end{cases} R T = { O p ( T ) , O p ( T ( d − 2 ) / ( d − 1 ) ) , d = 2 , 3 , d ≥ 4.
Proof. By (11) , α T = O ( log ( T / η ) ) \alpha_T=O(\sqrt{\log(T/\eta)}) α T = O ( log ( T / η ) ) for fixed d d d , λ \lambda λ , σ \sigma σ , C β C_\beta C β . Substituting this bound into Theorem 6 gives the stated rates.
Initial exploration and localisation
Lemma 34 (Exploration phase). Suppose Assumption 1 ′ 1' 1 ′ and assumptions 2 2 2 to 4 4 4 hold. Suppose we play p t = 1 / 2 p_t=1/2 p t = 1/2 in the exploration period when t ≤ 2 T o t\le2T_o t ≤ 2 T o and select the confidence radius α T \alpha_T α T as in (11) . For η ∈ ( 0 , 1 / 2 ) \eta\in(0,1/2) η ∈ ( 0 , 1/2 ) , if
T o ≥ 16 ( d + 1 ) κ X log ( 16 ( d + 1 ) η ) , T_o\ge\frac{16(d+1)}{\kappa_X}\log\left(\frac{16(d+1)}{\eta}\right), T o ≥ κ X 16 ( d + 1 ) log ( η 16 ( d + 1 ) ) , then, with probability at least 1 − η 1-\eta 1 − η , for every update time s s s , every k ∈ { 1 , 2 } k\in\{1,2\} k ∈ { 1 , 2 } , and every b k ∈ C s , k b_k\in C_{s,k} b k ∈ C s , k , we have that
∥ b k − β k ∗ ∥ 2 ≤ ( 1 + d + 1 ) α T λ + κ X T o / 4 . (55) \left\|b_k-\beta_k^*\right\|_2\le\frac{(1+\sqrt{d+1})\alpha_T}{\sqrt{\lambda+\kappa_XT_o/4}}. \tag*{(55)} ∥ b k − β k ∗ ∥ 2 ≤ λ + κ X T o /4 ( 1 + d + 1 ) α T . (55) If we choose T o = ⌈ T ⌉ T_o=\lceil\sqrt{T}\rceil T o = ⌈ T ⌉ , take r l o c ≤ 1 r_{\mathrm{loc}}\le1 r loc ≤ 1 , and suppose T ≥ 16 ∨ C 1 ( 1 + d + 1 ) 4 α T 4 / ( κ X 4 r l o c 4 ) T\ge16\vee C_1(1+\sqrt{d+1})^4\alpha_T^4/(\kappa_X^4r_{\mathrm{loc}}^4) T ≥ 16 ∨ C 1 ( 1 + d + 1 ) 4 α T 4 / ( κ X 4 r loc 4 ) , then, on the same event, every candidate satisfies ∥ b k ∥ 2 ≤ C β + 1 \|b_k\|_2\le C_\beta+1 ∥ b k ∥ 2 ≤ C β + 1 and every difference b 1 − b 2 b_1-b_2 b 1 − b 2 with b k ∈ C s , k b_k\in C_{s,k} b k ∈ C s , k belongs to the fixed regular neighbourhood Θ ~ ∗ \widetilde{\Theta}_{*} Θ ∗ in Lemma 14, for every update time s s s .
Proof. For k ∈ { 1 , 2 } k\in\{1,2\} k ∈ { 1 , 2 } , let I k 0 = { i ≤ 2 T o : A i = k } I_k^0=\{i\le2T_o:A_i=k\} I k 0 = { i ≤ 2 T o : A i = k } and N 0 , k = ∣ I k 0 ∣ = ∑ t = 1 2 T o 1 { A t = k } N_{0,k}=|I_k^0|=\sum_{t=1}^{2T_o}\mathbb{1}\{A_t=k\} N 0 , k = ∣ I k 0 ∣ = ∑ t = 1 2 T o 1 { A t = k } . Define
E a l l o c = { N 0 , 1 ≥ T o / 2 , N 0 , 2 ≥ T o / 2 } , (56) \mathcal{E}_{\mathrm{alloc}}=\{N_{0,1}\ge T_o/2,\ N_{0,2}\ge T_o/2\}, \tag*{(56)} E alloc = { N 0 , 1 ≥ T o /2 , N 0 , 2 ≥ T o /2 } , (56)
E i n i t = { λ min ( V s 0 , k ) ≥ λ + κ X T o / 4 , for every k ∈ { 1 , 2 } } . (57) \mathcal{E}_{\mathrm{init}}=\{\lambda_{\min}(V_{s_0,k})\ge\lambda+\kappa_XT_o/4,\text{ for every }k\in\{1,2\}\}. \tag*{(57)} E init = { λ m i n ( V s 0 , k ) ≥ λ + κ X T o /4 , for every k ∈ { 1 , 2 }} . (57)
Under p t = 1 / 2 p_t=1/2 p t = 1/2 , the exploration actions are independent of the contexts and N 0 , 1 ∼ Bin ( 2 T o , 1 / 2 ) N_{0,1}\sim\operatorname{Bin}(2T_o,1/2) N 0 , 1 ∼ Bin ( 2 T o , 1/2 ) , with N 0 , 2 = 2 T o − N 0 , 1 N_{0,2}=2T_o-N_{0,1} N 0 , 2 = 2 T o − N 0 , 1 . By Hoeffding’s inequality (e.g. Theorem 2.2.6 in Vershynin, [50 ] ), we have that
P ( N 0 , k < T o / 2 ) ≤ exp ( − T o 4 ) ≤ exp ( − T o 8 ) , k ∈ { 1 , 2 } . P(N_{0,k}<T_o/2)\le\exp\left(-\frac{T_o}{4}\right)\le\exp\left(-\frac{T_o}{8}\right),\qquad k\in\{1,2\}. P ( N 0 , k < T o /2 ) ≤ exp ( − 4 T o ) ≤ exp ( − 8 T o ) , k ∈ { 1 , 2 } .
A union bound over k ∈ { 1 , 2 } k\in\{1,2\} k ∈ { 1 , 2 } gives
P ( E a l l o c c ) ≤ 2 exp ( − T o 8 ) . (58) P(\mathcal{E}_{\mathrm{alloc}}^{c})\le2\exp\left(-\frac{T_o}{8}\right). \tag*{(58)} P ( E alloc c ) ≤ 2 exp ( − 8 T o ) . (58)
Fix k ∈ { 1 , 2 } k\in\{1,2\} k ∈ { 1 , 2 } and condition on the exploration actions A 1 , … , A 2 T o A_1,\ldots,A_{2T_o} A 1 , … , A 2 T o . Since the exploration actions are sampled independently of the contexts, the matrices { X ~ i X ~ i T : i ∈ I k 0 } \{\widetilde{X}_i\widetilde{X}_i^{\mathsf{T}}:i\in I_k^0\} { X i X i T : i ∈ I k 0 } remain independent and each has the same distribution as X ~ X ~ T \widetilde{X}\widetilde{X}^{\mathsf{T}} X X T . Moreover, X ~ i X ~ i T ⪰ 0 \widetilde{X}_i\widetilde{X}_i^{\mathsf{T}}\succeq0 X i X i T ⪰ 0 , λ max ( X ~ i X ~ i T ) = ∥ X ~ i ∥ 2 2 ≤ d + 1 \lambda_{\max}(\widetilde{X}_i\widetilde{X}_i^{\mathsf{T}})=\|\widetilde{X}_i\|_2^2\le d+1 λ m a x ( X i X i T ) = ∥ X i ∥ 2 2 ≤ d + 1 , and E ( X ~ i X ~ i T ) = Σ X ⪰ κ X I d + 1 \mathbb{E}(\widetilde{X}_i\widetilde{X}_i^{\mathsf{T}})=\Sigma_X\succeq\kappa_XI_{d+1} E ( X i X i T ) = Σ X ⪰ κ X I d + 1 by Assumption 1 1 1 . Therefore, on E a l l o c \mathcal{E}_{\mathrm{alloc}} E alloc , we have that
μ min : = λ min { ∑ i ∈ I k 0 E ( X ~ i X ~ i T ∣ A 1 , … , A 2 T o ) } ≥ N 0 , k κ X ≥ κ X T o 2 . \mu_{\min}:=\lambda_{\min}\left\{\sum_{i\in I_k^0}\mathbb{E}(\widetilde{X}_i\widetilde{X}_i^{\mathsf{T}}\mid A_1,\ldots,A_{2T_o})\right\}\ge N_{0,k}\kappa_X\ge\frac{\kappa_XT_o}{2}. μ m i n := λ m i n ⎩ ⎨ ⎧ i ∈ I k 0 ∑ E ( X i X i T ∣ A 1 , … , A 2 T o ) ⎭ ⎬ ⎫ ≥ N 0 , k κ X ≥ 2 κ X T o .
By the matrix Chernoff inequality (e.g. Theorem 5.1.1 in Tropp, [48 ] ), applied with matrix dimension d + 1 d+1 d + 1 , upper eigenvalue bound d + 1 d+1 d + 1 , and ε = 1 / 2 \varepsilon=1/2 ε = 1/2 , it holds that
P { λ min ( ∑ i ∈ I k 0 X ~ i X ~ i T ) ≤ μ min 2 | A 1 , … , A 2 T o } ≤ ( d + 1 ) ( e − 1 / 2 ( 1 / 2 ) 1 / 2 ) μ min / ( d + 1 ) ≤ ( d + 1 ) exp { − μ min 8 ( d + 1 ) } . P\left\{\lambda_{\min}\left(\sum_{i\in I_k^0}\widetilde{X}_i\widetilde{X}_i^{\mathsf{T}}\right)\le\frac{\mu_{\min}}{2}\,\middle|\,A_1,\ldots,A_{2T_o}\right\}
\le(d+1)\left(\frac{e^{-1/2}}{(1/2)^{1/2}}\right)^{\mu_{\min}/(d+1)}
\le(d+1)\exp\left\{-\frac{\mu_{\min}}{8(d+1)}\right\}. P ⎩ ⎨ ⎧ λ m i n i ∈ I k 0 ∑ X i X i T ≤ 2 μ m i n A 1 , … , A 2 T o ⎭ ⎬ ⎫ ≤ ( d + 1 ) ( ( 1/2 ) 1/2 e − 1/2 ) μ m i n / ( d + 1 ) ≤ ( d + 1 ) exp { − 8 ( d + 1 ) μ m i n } .
Since μ min ≥ κ X T o / 2 \mu_{\min}\ge\kappa_XT_o/2 μ m i n ≥ κ X T o /2 on E a l l o c \mathcal{E}_{\mathrm{alloc}} E alloc , we have
P { λ min ( ∑ i ∈ I k 0 X ~ i X ~ i T ) ≤ κ X T o 4 | A 1 , … , A 2 T o } ≤ ( d + 1 ) exp { − κ X T o 16 ( d + 1 ) } . P\left\{\lambda_{\min}\left(\sum_{i\in I_k^0}\widetilde{X}_i\widetilde{X}_i^{\mathsf{T}}\right)\le\frac{\kappa_XT_o}{4}\,\middle|\,A_1,\ldots,A_{2T_o}\right\}\le(d+1)\exp\left\{-\frac{\kappa_XT_o}{16(d+1)}\right\}. P ⎩ ⎨ ⎧ λ m i n i ∈ I k 0 ∑ X i X i T ≤ 4 κ X T o A 1 , … , A 2 T o ⎭ ⎬ ⎫ ≤ ( d + 1 ) exp { − 16 ( d + 1 ) κ X T o } .
Recall from Algorithm 3 that V s 0 , k = λ I d + 1 + ∑ i ∈ I k 0 X ~ i X ~ i T V_{s_0,k}=\lambda I_{d+1}+\sum_{i\in I_k^0}\widetilde{X}_i\widetilde{X}_i^{\mathsf{T}} V s 0 , k = λ I d + 1 + ∑ i ∈ I k 0 X i X i T . Therefore, a union bound over k ∈ { 1 , 2 } k\in\{1,2\} k ∈ { 1 , 2 } and tower property give
P ( E i n i t c ∩ E a l l o c ) ≤ 2 ( d + 1 ) exp { − κ X T o 16 ( d + 1 ) } . (59) P(\mathcal{E}_{\mathrm{init}}^{c}\cap\mathcal{E}_{\mathrm{alloc}})\le2(d+1)\exp\left\{-\frac{\kappa_XT_o}{16(d+1)}\right\}. \tag*{(59)} P ( E init c ∩ E alloc ) ≤ 2 ( d + 1 ) exp { − 16 ( d + 1 ) κ X T o } . (59)
Combining (58) and (59) , we obtain
P ( E i n i t c ) ≤ 2 exp ( − T o 8 ) + 2 ( d + 1 ) exp { − κ X T o 16 ( d + 1 ) } ≤ η 4 , P(\mathcal{E}_{\mathrm{init}}^{c})\le2\exp\left(-\frac{T_o}{8}\right)+2(d+1)\exp\left\{-\frac{\kappa_XT_o}{16(d+1)}\right\}\le\frac{\eta}{4}, P ( E init c ) ≤ 2 exp ( − 8 T o ) + 2 ( d + 1 ) exp { − 16 ( d + 1 ) κ X T o } ≤ 4 η ,
where the last inequality follows whenever
T o ≥ 8 log ( 16 η ) and T o ≥ 16 ( d + 1 ) κ X log ( 16 ( d + 1 ) η ) . T_o\ge8\log\left(\frac{16}{\eta}\right)\qquad\text{and}\qquad T_o\ge\frac{16(d+1)}{\kappa_X}\log\left(\frac{16(d+1)}{\eta}\right). T o ≥ 8 log ( η 16 ) and T o ≥ κ X 16 ( d + 1 ) log ( η 16 ( d + 1 ) ) .
Since e 1 T Σ X e 1 = 1 \mathbf{e}_1^{\mathsf{T}}\Sigma_X\mathbf{e}_1=1 e 1 T Σ X e 1 = 1 , we have κ X ≤ 1 \kappa_X\le1 κ X ≤ 1 , and therefore the assumed lower bound on T o T_o T o implies both inequalities.
Recall the confidence event E c o n f \mathcal{E}_{\mathrm{conf}} E conf in (51) . By the same concentration argument following that display, P ( E c o n f ) ≥ 1 − η / 8 P(\mathcal{E}_{\mathrm{conf}})\ge1-\eta/8 P ( E conf ) ≥ 1 − η /8 . A union bound therefore gives
P ( E i n i t ∩ E c o n f ) ≥ 1 − η 4 − η 8 = 1 − 3 η 8 ≥ 1 − η . P(\mathcal{E}_{\mathrm{init}}\cap\mathcal{E}_{\mathrm{conf}})\ge1-\frac{\eta}{4}-\frac{\eta}{8}=1-\frac{3\eta}{8}\ge1-\eta. P ( E init ∩ E conf ) ≥ 1 − 4 η − 8 η = 1 − 8 3 η ≥ 1 − η .
The rest of the proof is carried out conditioning on E i n i t ∩ E c o n f \mathcal{E}_{\mathrm{init}}\cap\mathcal{E}_{\mathrm{conf}} E init ∩ E conf happening. By Lemma 31 , β k ∗ ∈ C s , k \beta_k^*\in\mathcal{C}_{s,k} β k ∗ ∈ C s , k and ∥ b k − β k ∗ ∥ V s , k ≤ ( 1 + d + 1 ) α T \lVert b_k-\beta_k^*\rVert_{V_{s,k}}\le(1+\sqrt{d+1})\alpha_T ∥ b k − β k ∗ ∥ V s , k ≤ ( 1 + d + 1 ) α T for every update time s s s and every b k ∈ C s , k b_k\in\mathcal{C}_{s,k} b k ∈ C s , k . Since V s , k ⪰ V s 0 , k V_{s,k}\succeq V_{s_0,k} V s , k ⪰ V s 0 , k for every s ≥ s 0 s\ge s_0 s ≥ s 0 , we also have V s , k ⪰ ( λ + κ X T o / 4 ) I d + 1 V_{s,k}\succeq(\lambda+\kappa_XT_o/4)I_{d+1} V s , k ⪰ ( λ + κ X T o /4 ) I d + 1 . Therefore,
λ + κ X T o / 4 ∥ b k − β k ∗ ∥ 2 ≤ ∥ b k − β k ∗ ∥ V s , k ≤ ( 1 + d + 1 ) α T , \sqrt{\lambda+\kappa_XT_o/4}\lVert b_k-\beta_k^*\rVert_2\le\lVert b_k-\beta_k^*\rVert_{V_{s,k}}\le(1+\sqrt{d+1})\alpha_T, λ + κ X T o /4 ∥ b k − β k ∗ ∥ 2 ≤ ∥ b k − β k ∗ ∥ V s , k ≤ ( 1 + d + 1 ) α T ,
which proves (55) for every update time s s s .
Finally, for θ = b 1 − b 2 \theta=b_1-b_2 θ = b 1 − b 2 , the triangle inequality and (55) give
∥ θ − θ ∗ ∥ 2 ≤ 2 ( 1 + d + 1 ) α T λ + κ X T o / 4 . \lVert\theta-\theta^*\rVert_2\le\frac{2(1+\sqrt{d+1})\alpha_T}{\sqrt{\lambda+\kappa_XT_o/4}}. ∥ θ − θ ∗ ∥ 2 ≤ λ + κ X T o /4 2 ( 1 + d + 1 ) α T .
If T o = ⌈ T ⌉ T_o=\lceil\sqrt{T}\rceil T o = ⌈ T ⌉ , then we have that
2 ( 1 + d + 1 ) α T λ + κ X T o / 4 ≤ 4 ( 1 + d + 1 ) α T κ X T 1 / 4 . \frac{2(1+\sqrt{d+1})\alpha_T}{\sqrt{\lambda+\kappa_XT_o/4}}\le\frac{4(1+\sqrt{d+1})\alpha_T}{\sqrt{\kappa_X}T^{1/4}}. λ + κ X T o /4 2 ( 1 + d + 1 ) α T ≤ κ X T 1/4 4 ( 1 + d + 1 ) α T .
Hence, whenever
T ≥ 16 ( 1 + d + 1 ) 4 α T 4 κ X 2 r l o c 4 , T\ge\frac{16(1+\sqrt{d+1})^4\alpha_T^4}{\kappa_X^2r_{\mathrm{loc}}^4}, T ≥ κ X 2 r loc 4 16 ( 1 + d + 1 ) 4 α T 4 ,
we have ∥ θ − θ ∗ ∥ 2 ≤ 2 r l o c \lVert\theta-\theta^*\rVert_2\le2r_{\mathrm{loc}} ∥ θ − θ ∗ ∥ 2 ≤ 2 r loc . It therefore follows from Lemma 14 that θ ∈ Θ ~ j ∗ \theta\in\widetilde{\Theta}_{j^*} θ ∈ Θ j ∗ , for every update time s s s on the same event. Under the same lower bound on T T T , (55) gives ∥ b k − β k ∗ ∥ 2 ≤ r l o c ≤ 1 \lVert b_k-\beta_k^*\rVert_2\le r_{\mathrm{loc}}\le1 ∥ b k − β k ∗ ∥ 2 ≤ r loc ≤ 1 , and hence ∥ b k ∥ 2 ≤ C β + 1 \lVert b_k\rVert_2\le C_\beta+1 ∥ b k ∥ 2 ≤ C β + 1 .
Complexity under arbitrary context laws
Throughout this subsubsection, we let s θ ( x ) = x ~ T θ s_\theta(x)=\widetilde{x}^{\mathsf{T}}\theta s θ ( x ) = x T θ . The geometric results in Lemma 18 and the trace-entropy bound in Lemma 21 depend only on the parameter and Lipschitz assumptions, so they apply to arbitrary context laws. However, the slab-probability bound in Lemma 19 uses a bounded density and cannot be used here. We instead control policy differences pointwise, including on the sign-disagreement set, and obtain entropy bounds uniformly over all probability measures. This uniformity allows us to use the empirical measure P n P_n P n in the conditional Rademacher argument below.
Lemma 35 (Localisation of sign disagreement). Suppose Assumption 4 holds. Then there exists a constant C l o c < ∞ C_{\mathrm{loc}}<\infty C loc < ∞ , depending only on d d d , ν \nu ν and the constants in Lemma 14 , such that, for every j ∈ { 1 , … , d } j\in\{1,\ldots,d\} j ∈ { 1 , … , d } and all θ , θ ′ ∈ Θ ~ j \theta,\theta'\in\widetilde{\Theta}_j θ , θ ′ ∈ Θ j ,
sup x ∈ S ( θ , θ ′ ) { dist ν ( x , H θ ) ∨ dist ν ( x , H θ ′ ) } ≤ C l o c ∥ θ − θ ′ ∥ 2 , \sup_{x\in S(\theta,\theta')}\{\operatorname{dist}_{\nu}(x,\mathcal{H}_{\theta})\vee\operatorname{dist}_{\nu}(x,\mathcal{H}_{\theta'})\}\le C_{\mathrm{loc}}\lVert\theta-\theta'\rVert_2, x ∈ S ( θ , θ ′ ) sup { dist ν ( x , H θ ) ∨ dist ν ( x , H θ ′ )} ≤ C loc ∥ θ − θ ′ ∥ 2 , where
S ( θ , θ ′ ) = { x ∈ [ 0 , 1 ] d : s θ ( x ) ≥ 0 , s θ ′ ( x ) < 0 } ∪ { x ∈ [ 0 , 1 ] d : s θ ( x ) < 0 , s θ ′ ( x ) ≥ 0 } . S(\theta,\theta')=\{x\in[0,1]^d:s_\theta(x)\ge0,\ s_{\theta'}(x)<0\}\cup\{x\in[0,1]^d:s_\theta(x)<0,\ s_{\theta'}(x)\ge0\}. S ( θ , θ ′ ) = { x ∈ [ 0 , 1 ] d : s θ ( x ) ≥ 0 , s θ ′ ( x ) < 0 } ∪ { x ∈ [ 0 , 1 ] d : s θ ( x ) < 0 , s θ ′ ( x ) ≥ 0 } .
Proof. Fix j ∈ { 1 , … , d } j \in\{1,\ldots,d\} j ∈ { 1 , … , d } . By relabelling coordinates, without loss of generality, we assume that j = d j=d j = d . Write θ = ( a , γ ) \theta=(a,\gamma) θ = ( a , γ ) and θ ′ = ( a ′ , γ ′ ) \theta'=(a',\gamma') θ ′ = ( a ′ , γ ′ ) , where a , a ′ ∈ R a,a'\in\mathbb{R} a , a ′ ∈ R and γ , γ ′ ∈ R d \gamma,\gamma'\in\mathbb{R}^{d} γ , γ ′ ∈ R d . Then write γ = ( b , c ) \gamma=(b,c) γ = ( b , c ) and γ ′ = ( b ′ , c ′ ) \gamma'=(b',c') γ ′ = ( b ′ , c ′ ) , where b , b ′ ∈ R d − 1 b,b'\in\mathbb{R}^{d-1} b , b ′ ∈ R d − 1 and c , c ′ ∈ R c,c'\in\mathbb{R} c , c ′ ∈ R . For u ∈ [ 0 , 1 ] d − 1 u\in[0,1]^{d-1} u ∈ [ 0 , 1 ] d − 1 , define
ϕ θ ( u ) = − a + b T u c , ϕ θ ′ ( u ) = − a ′ + b ′ T u c ′ . \phi_\theta(u)=-\frac{a+b^{\mathsf T}u}{c},\qquad\phi_{\theta'}(u)=-\frac{a'+b'^{\mathsf T}u}{c'}. ϕ θ ( u ) = − c a + b T u , ϕ θ ′ ( u ) = − c ′ a ′ + b ′ T u .
Note that the calculation in (35) gives
∥ ϕ θ − ϕ θ ′ ∥ ∞ ≤ C ϕ ∥ θ − θ ′ ∥ 2 . (60) \|\phi_\theta-\phi_{\theta'}\|_\infty\le C_\phi\|\theta-\theta'\|_2. \tag*{(60)} ∥ ϕ θ − ϕ θ ′ ∥ ∞ ≤ C ϕ ∥ θ − θ ′ ∥ 2 . (60)
Write δ = ∥ θ − θ ′ ∥ 2 \delta=\|\theta-\theta'\|_2 δ = ∥ θ − θ ′ ∥ 2 and set η = C ϕ δ \eta=C_\phi\delta η = C ϕ δ . Let C d , ν = d max { 1 / ν − 1 / 2 , 0 } C_{d,\nu}=d^{\max\{1/\nu-1/2,0\}} C d , ν = d m a x { 1/ ν − 1/2 , 0 } be the constant in Lemma 30 . Take any x = ( u , z ) ∈ S ( θ , θ ′ ) x=(u,z)\in S(\theta,\theta') x = ( u , z ) ∈ S ( θ , θ ′ ) . Since s θ ( u , z ) = c { z − ϕ θ ( u ) } s_\theta(u,z)=c\{z-\phi_\theta(u)\} s θ ( u , z ) = c { z − ϕ θ ( u )} and s θ ′ ( u , z ) = c ′ { z − ϕ θ ′ ( u ) } s_{\theta'}(u,z)=c'\{z-\phi_{\theta'}(u)\} s θ ′ ( u , z ) = c ′ { z − ϕ θ ′ ( u )} with c , c ′ > 0 c,c'>0 c , c ′ > 0 , the value z z z lies between ϕ θ ( u ) \phi_\theta(u) ϕ θ ( u ) and ϕ θ ′ ( u ) \phi_{\theta'}(u) ϕ θ ′ ( u ) . Therefore, by (60) ,
∣ z − ϕ θ ( u ) ∣ ∨ ∣ z − ϕ θ ′ ( u ) ∣ ≤ η . (61) |z-\phi_\theta(u)|\vee|z-\phi_{\theta'}(u)|\le\eta. \tag*{(61)} ∣ z − ϕ θ ( u ) ∣ ∨ ∣ z − ϕ θ ′ ( u ) ∣ ≤ η . (61)
In the rest of the proof, we show that x x x is within ℓ ν \ell_\nu ℓ ν -distance C η C\eta C η of H θ \mathcal{H}_\theta H θ . The bound for H θ ′ \mathcal{H}_{\theta'} H θ ′ then follows by symmetry.
Case 1: If 0 ≤ ϕ θ ( u ) ≤ 1 0\le\phi_\theta(u)\le1 0 ≤ ϕ θ ( u ) ≤ 1 , then h = ( u , ϕ θ ( u ) ) ∈ H θ h=(u,\phi_\theta(u))\in\mathcal{H}_\theta h = ( u , ϕ θ ( u )) ∈ H θ , and hence
∥ x − h ∥ ν = ∣ z − ϕ θ ( u ) ∣ ≤ η . \|x-h\|_\nu=|z-\phi_\theta(u)|\le\eta. ∥ x − h ∥ ν = ∣ z − ϕ θ ( u ) ∣ ≤ η .
Case 2: Next suppose ϕ θ ( u ) < 0 \phi_\theta(u)<0 ϕ θ ( u ) < 0 . By the definition of Θ ~ j \widetilde{\Theta}_j Θ j , there exists u θ ∈ [ 0 , 1 ] d − 1 u_\theta\in[0,1]^{d-1} u θ ∈ [ 0 , 1 ] d − 1 such that ρ ~ 0 ≤ ϕ θ ( u θ ) ≤ 1 − ρ ~ 0 \widetilde{\rho}_0\le\phi_\theta(u_\theta)\le1-\widetilde{\rho}_0 ρ 0 ≤ ϕ θ ( u θ ) ≤ 1 − ρ 0 . For λ ∈ [ 0 , 1 ] \lambda\in[0,1] λ ∈ [ 0 , 1 ] , define u λ = ( 1 − λ ) u + λ u θ u_\lambda=(1-\lambda)u+\lambda u_\theta u λ = ( 1 − λ ) u + λ u θ . Since ϕ θ \phi_\theta ϕ θ is affine, the map λ ↦ ϕ θ ( u λ ) \lambda\mapsto\phi_\theta(u_\lambda) λ ↦ ϕ θ ( u λ ) is affine. Since ϕ θ ( u ) < 0 \phi_\theta(u)<0 ϕ θ ( u ) < 0 and ϕ θ ( u θ ) ≥ ρ ~ 0 \phi_\theta(u_\theta)\ge\widetilde{\rho}_0 ϕ θ ( u θ ) ≥ ρ 0 , there exists λ 0 ∈ [ 0 , 1 ] \lambda_0\in[0,1] λ 0 ∈ [ 0 , 1 ] such that ϕ θ ( u λ 0 ) = 0 \phi_\theta(u_{\lambda_0})=0 ϕ θ ( u λ 0 ) = 0 . Moreover, by (61) and z ≥ 0 z\ge0 z ≥ 0 , we have that
λ 0 = − ϕ θ ( u ) ϕ θ ( u θ ) − ϕ θ ( u ) ≤ η ρ ~ 0 + η ≤ η ρ ~ 0 , z ≤ η . \lambda_0=\frac{-\phi_\theta(u)}{\phi_\theta(u_\theta)-\phi_\theta(u)}
\le\frac{\eta}{\widetilde{\rho}_0+\eta}\le\frac{\eta}{\widetilde{\rho}_0},
\qquad z\le\eta. λ 0 = ϕ θ ( u θ ) − ϕ θ ( u ) − ϕ θ ( u ) ≤ ρ 0 + η η ≤ ρ 0 η , z ≤ η .
Set v = u λ 0 v=u_{\lambda_0} v = u λ 0 and h = ( v , 0 ) h=(v,0) h = ( v , 0 ) . Then h ∈ H θ h\in\mathcal{H}_\theta h ∈ H θ . We first bound the distance in the Euclidean norm and then convert it to the fairness norm. Since u , u θ ∈ [ 0 , 1 ] d − 1 u,u_\theta\in[0,1]^{d-1} u , u θ ∈ [ 0 , 1 ] d − 1 ,
∥ v − u ∥ 2 ≤ λ 0 ∥ u θ − u ∥ 2 ≤ d − 1 ρ ~ 0 η . \|v-u\|_2\le\lambda_0\|u_\theta-u\|_2\le\frac{\sqrt{d-1}}{\widetilde{\rho}_0}\eta. ∥ v − u ∥ 2 ≤ λ 0 ∥ u θ − u ∥ 2 ≤ ρ 0 d − 1 η .
By Lemma 30 ,
∥ v − u ∥ ν ≤ C d , ν ∥ v − u ∥ 2 ≤ C d , ν d − 1 ρ ~ 0 η . \|v-u\|_\nu\le C_{d,\nu}\|v-u\|_2\le\frac{C_{d,\nu}\sqrt{d-1}}{\widetilde{\rho}_0}\eta. ∥ v − u ∥ ν ≤ C d , ν ∥ v − u ∥ 2 ≤ ρ 0 C d , ν d − 1 η .
Hence, by the triangle inequality,
∥ x − h ∥ ν ≤ ∥ u − v ∥ ν + z ≤ ( 1 + C d , ν d − 1 ρ ~ 0 ) η . \|x-h\|_\nu\le\|u-v\|_\nu+z\le\left(1+\frac{C_{d,\nu}\sqrt{d-1}}{\widetilde{\rho}_0}\right)\eta. ∥ x − h ∥ ν ≤ ∥ u − v ∥ ν + z ≤ ( 1 + ρ 0 C d , ν d − 1 ) η .
Case 3: Finally suppose ϕ θ ( u ) > 1 \phi_\theta(u)>1 ϕ θ ( u ) > 1 . Again, by the definition of Θ ~ j \widetilde{\Theta}_j Θ j , there exists u θ ∈ [ 0 , 1 ] d − 1 u_\theta\in[0,1]^{d-1} u θ ∈ [ 0 , 1 ] d − 1 such that ρ ~ 0 ≤ ϕ θ ( u θ ) ≤ 1 − ρ ~ 0 \widetilde{\rho}_0\le\phi_\theta(u_\theta)\le1-\widetilde{\rho}_0 ρ 0 ≤ ϕ θ ( u θ ) ≤ 1 − ρ 0 . Along the same segment u λ = ( 1 − λ ) u + λ u θ u_\lambda=(1-\lambda)u+\lambda u_\theta u λ = ( 1 − λ ) u + λ u θ , there exists λ 0 ∈ [ 0 , 1 ] \lambda_0\in[0,1] λ 0 ∈ [ 0 , 1 ] such that ϕ θ ( u λ 0 ) = 1 \phi_\theta(u_{\lambda_0})=1 ϕ θ ( u λ 0 ) = 1 . Moreover, by (61) and z ≤ 1 z\le1 z ≤ 1 ,
λ 0 = ϕ θ ( u ) − 1 ϕ θ ( u ) − ϕ θ ( u θ ) ≤ η ρ ~ 0 + η ≤ η ρ ~ 0 , 1 − z ≤ η . \lambda_0=\frac{\phi_\theta(u)-1}{\phi_\theta(u)-\phi_\theta(u_\theta)}
\le\frac{\eta}{\widetilde{\rho}_0+\eta}\le\frac{\eta}{\widetilde{\rho}_0},
\qquad1-z\le\eta. λ 0 = ϕ θ ( u ) − ϕ θ ( u θ ) ϕ θ ( u ) − 1 ≤ ρ 0 + η η ≤ ρ 0 η , 1 − z ≤ η .
Set v = u λ 0 v=u_{\lambda_0} v = u λ 0 and h = ( v , 1 ) h=(v,1) h = ( v , 1 ) . Then h ∈ H θ h\in\mathcal{H}_{\theta} h ∈ H θ . As in Case 2 ,
∥ v − u ∥ ν ≤ C d , ν ∥ v − u ∥ 2 ≤ C d , ν d − 1 ρ ~ 0 η . \|v-u\|_{\nu}\leq C_{d,\nu}\|v-u\|_{2}\leq\frac{C_{d,\nu}\sqrt{d-1}}{\widetilde{\rho}_0}\eta. ∥ v − u ∥ ν ≤ C d , ν ∥ v − u ∥ 2 ≤ ρ 0 C d , ν d − 1 η .
where the first inequality follows from Lemma 30 . Hence
∥ x − h ∥ ν ≤ ∥ u − v ∥ ν + ∣ z − 1 ∣ ≤ ( 1 + C d , ν d − 1 ρ ~ 0 ) η . \|x-h\|_{\nu}\leq\|u-v\|_{\nu}+|z-1|\leq\left(1+\frac{C_{d,\nu}\sqrt{d-1}}{\widetilde{\rho}_0}\right)\eta. ∥ x − h ∥ ν ≤ ∥ u − v ∥ ν + ∣ z − 1∣ ≤ ( 1 + ρ 0 C d , ν d − 1 ) η .
Combining Cases 1–3 gives dist ν ( x , H θ ) ≤ C η \operatorname{dist}_{\nu}(x,\mathcal{H}_{\theta})\leq C\eta dist ν ( x , H θ ) ≤ C η . Applying the same argument with θ ′ \theta' θ ′ in place of θ \theta θ gives dist ν ( x , H θ ′ ) ≤ C η \operatorname{dist}_{\nu}(x,\mathcal{H}_{\theta'})\leq C\eta dist ν ( x , H θ ′ ) ≤ C η . Since η = C ϕ ∥ θ − θ ′ ∥ 2 \eta=C_{\phi}\|\theta-\theta'\|_{2} η = C ϕ ∥ θ − θ ′ ∥ 2 , the claim follows.
Lemma 36. Suppose Assumption 4 holds. Fix j ∈ { 1 , … , d } j\in\{1,\ldots,d\} j ∈ { 1 , … , d } , θ , θ ′ ∈ Θ ~ j \theta,\theta'\in\widetilde{\Theta}_{j} θ , θ ′ ∈ Θ j and q ∈ Q θ , L q\in\mathcal{Q}_{\theta,L} q ∈ Q θ , L . Define the global McShane extension of q q q from H θ \mathcal{H}_{\theta} H θ to [ 0 , 1 ] d [0,1]^d [ 0 , 1 ] d by
G ( x ) = inf h ∈ H θ { q ( h ) + L ∥ x − h ∥ ν } , q ‾ ( x ) = [ G ( x ) ] [ 0 , 1 ] , x ∈ [ 0 , 1 ] d , G(x)=\inf_{h\in\mathcal{H}_{\theta}}\{q(h)+L\|x-h\|_{\nu}\},\qquad\overline{q}(x)=[G(x)]_{[0,1]},\qquad x\in[0,1]^d, G ( x ) = h ∈ H θ inf { q ( h ) + L ∥ x − h ∥ ν } , q ( x ) = [ G ( x ) ] [ 0 , 1 ] , x ∈ [ 0 , 1 ] d ,
*where [ y ] [ 0 , 1 ] = min { 1 , max { 0 , y } } [y]_{[0,1]}=\min\{1,\max\{0,y\}\} [ y ] [ 0 , 1 ] = min { 1 , max { 0 , y }} . Define the transferred trace on H θ ′ \mathcal{H}_{\theta'} H θ ′ by q ′ ( h ′ ) = q ‾ ( h ′ ) q'(h')=\overline{q}(h') q ′ ( h ′ ) = q ( h ′ ) for all h ′ ∈ H θ ′ h'\in\mathcal{H}_{\theta'} h ′ ∈ H θ ′ . Then q ′ ∈ Q θ ′ , L q'\in\mathcal{Q}_{\theta',L} q ′ ∈ Q θ ′ , L and with U θ , q U_{\theta,q} U θ , q and M θ , q M_{\theta,q} M θ , q defined in (5) ,
∥ U θ , q − U θ ′ , q ′ ∥ ∞ ∨ ∥ M θ , q − M θ ′ , q ′ ∥ ∞ ≤ 2 L C H ∥ θ − θ ′ ∥ 2 . \|U_{\theta,q}-U_{\theta',q'}\|_{\infty}\vee\|M_{\theta,q}-M_{\theta',q'}\|_{\infty}\leq2LC_H\|\theta-\theta'\|_{2}. ∥ U θ , q − U θ ′ , q ′ ∥ ∞ ∨ ∥ M θ , q − M θ ′ , q ′ ∥ ∞ ≤ 2 L C H ∥ θ − θ ′ ∥ 2 .
Proof. The construction is the same as in Steps 1–3 of the proof of Lemma 17, with ( θ ∗ , q ∗ ) (\theta^{*},q^{*}) ( θ ∗ , q ∗ ) replaced by ( θ , q ) (\theta,q) ( θ , q ) . Those steps use only the Lipschitz property of the trace, nonexpansiveness of clipping, and the Hausdorff bound in Lemma 18, and they do not rely on the context density or optimality of the trace. Thus q ′ ∈ Q θ ′ , L q'\in\mathcal{Q}_{\theta',L} q ′ ∈ Q θ ′ , L and both envelope differences are bounded by 2 L C H ∥ θ − θ ′ ∥ 2 2LC_H\|\theta-\theta'\|_{2} 2 L C H ∥ θ − θ ′ ∥ 2 , as claimed.
Lemma 37 (Gap between the clipped canonical envelopes). For every θ ∈ Θ ~ \theta\in\widetilde{\Theta} θ ∈ Θ , q ∈ Q θ , L q\in\mathcal{Q}_{\theta,L} q ∈ Q θ , L and x ∈ [ 0 , 1 ] d x\in[0,1]^d x ∈ [ 0 , 1 ] d , define U ‾ θ , q ( x ) = 1 ∧ U θ , q ( x ) \overline{U}_{\theta,q}(x)=1\wedge U_{\theta,q}(x) U θ , q ( x ) = 1 ∧ U θ , q ( x ) and M ‾ θ , q ( x ) = 0 ∨ M θ , q ( x ) \overline{M}_{\theta,q}(x)=0\vee M_{\theta,q}(x) M θ , q ( x ) = 0 ∨ M θ , q ( x ) . Then, we have that
0 ≤ U ‾ θ , q ( x ) − M ‾ θ , q ( x ) ≤ 2 L dist ν ( x , H θ ) . 0\leq\overline{U}_{\theta,q}(x)-\overline{M}_{\theta,q}(x)\leq2L\operatorname{dist}_{\nu}(x,\mathcal{H}_{\theta}). 0 ≤ U θ , q ( x ) − M θ , q ( x ) ≤ 2 L dist ν ( x , H θ ) .
Proof. For any h , g ∈ H θ h,g\in\mathcal{H}_{\theta} h , g ∈ H θ , we have that
q ( h ) − L ∥ x − h ∥ ν ≤ q ( g ) + L ∥ h − g ∥ ν − L ∥ x − h ∥ ν ≤ q ( g ) + L ∥ x − g ∥ ν . q(h)-L\|x-h\|_{\nu}\leq q(g)+L\|h-g\|_{\nu}-L\|x-h\|_{\nu}\leq q(g)+L\|x-g\|_{\nu}. q ( h ) − L ∥ x − h ∥ ν ≤ q ( g ) + L ∥ h − g ∥ ν − L ∥ x − h ∥ ν ≤ q ( g ) + L ∥ x − g ∥ ν .
Taking the supremum over h h h followed by the infimum over g g g gives M θ , q ( x ) ≤ U θ , q ( x ) M_{\theta,q}(x)\leq U_{\theta,q}(x) M θ , q ( x ) ≤ U θ , q ( x ) . Moreover, since U θ , q ≥ 0 U_{\theta,q}\geq0 U θ , q ≥ 0 and M θ , q ≤ 1 M_{\theta,q}\leq1 M θ , q ≤ 1 , clipping preserves the ordering and we have that
0 ≤ U ‾ θ , q ( x ) − M ‾ θ , q ( x ) ≤ U θ , q ( x ) − M θ , q ( x ) . 0\leq\overline{U}_{\theta,q}(x)-\overline{M}_{\theta,q}(x)\leq U_{\theta,q}(x)-M_{\theta,q}(x). 0 ≤ U θ , q ( x ) − M θ , q ( x ) ≤ U θ , q ( x ) − M θ , q ( x ) .
Moreover, since set H θ \mathcal{H}_{\theta} H θ is nonempty and compact, let h x ∈ H θ h_x\in\mathcal{H}_{\theta} h x ∈ H θ attain dist ν ( x , H θ ) \operatorname{dist}_{\nu}(x,\mathcal{H}_{\theta}) dist ν ( x , H θ ) . Then
U θ , q ( x ) ≤ q ( h x ) + L dist ν ( x , H θ ) , M θ , q ( x ) ≥ q ( h x ) − L dist ν ( x , H θ ) . U_{\theta,q}(x)\leq q(h_x)+L\operatorname{dist}_{\nu}(x,\mathcal{H}_{\theta}),\qquad M_{\theta,q}(x)\geq q(h_x)-L\operatorname{dist}_{\nu}(x,\mathcal{H}_{\theta}). U θ , q ( x ) ≤ q ( h x ) + L dist ν ( x , H θ ) , M θ , q ( x ) ≥ q ( h x ) − L dist ν ( x , H θ ) .
Combining the preceding bounds proves the result.
Lemma 38 (Uniform transfer in the sup-norm). Suppose Assumption 4 holds. Then there exists a constant C t r < ∞ C_{\mathrm{tr}}<\infty C tr < ∞ , depending only on d , ν , L d,\nu,L d , ν , L and the constants in Lemma 14 , such that, for every j ∈ { 1 , … , d } j\in\{1,\ldots,d\} j ∈ { 1 , … , d } , every θ , θ ′ ∈ Θ ~ j \theta,\theta'\in\widetilde{\Theta}_{j} θ , θ ′ ∈ Θ j and every q ∈ Q θ , L q\in\mathcal{Q}_{\theta,L} q ∈ Q θ , L , there exists q ′ ∈ Q θ ′ , L q'\in\mathcal{Q}_{\theta',L} q ′ ∈ Q θ ′ , L satisfying
∥ p θ , q − p θ ′ , q ′ ∥ ∞ ≤ C t r ∥ θ − θ ′ ∥ 2 . \|p_{\theta,q}-p_{\theta',q'}\|_{\infty}\leq C_{\mathrm{tr}}\|\theta-\theta'\|_{2}. ∥ p θ , q − p θ ′ , q ′ ∥ ∞ ≤ C tr ∥ θ − θ ′ ∥ 2 .
Proof. Fix j ∈ { 1 , … , d } j \in\{1,\ldots,d\} j ∈ { 1 , … , d } , θ , θ ′ ∈ Θ ~ j \theta,\theta' \in\widetilde{\Theta}_j θ , θ ′ ∈ Θ j and q ∈ Q θ , L q \in\mathcal{Q}_{\theta,L} q ∈ Q θ , L . Write δ = ∥ θ − θ ′ ∥ 2 \delta= \lVert\theta-\theta'\rVert_2 δ = ∥ θ − θ ′ ∥ 2 , and let q ′ q' q ′ be the transferred trace constructed in Lemma 36. Define the sign-disagreement set
S ( θ , θ ′ ) = { x ∈ [ 0 , 1 ] d : s θ ( x ) ≥ 0 , s θ ′ ( x ) < 0 } ∪ { x ∈ [ 0 , 1 ] d : s θ ( x ) < 0 , s θ ′ ( x ) ≥ 0 } . S(\theta,\theta')=\{x\in[0,1]^d:s_\theta(x)\ge0,\ s_{\theta'}(x)<0\}\cup\{x\in[0,1]^d:s_\theta(x)<0,\ s_{\theta'}(x)\ge0\}. S ( θ , θ ′ ) = { x ∈ [ 0 , 1 ] d : s θ ( x ) ≥ 0 , s θ ′ ( x ) < 0 } ∪ { x ∈ [ 0 , 1 ] d : s θ ( x ) < 0 , s θ ′ ( x ) ≥ 0 } .
Step 1: Control on S ( θ , θ ′ ) c S(\theta,\theta')^c S ( θ , θ ′ ) c . On S ( θ , θ ′ ) c S(\theta,\theta')^c S ( θ , θ ′ ) c , the two policies use the same type of clipped envelope; either both use the upper envelopes or both use the lower envelopes. Since the maps y ↦ min { 1 , y } y\mapsto\min\{1,y\} y ↦ min { 1 , y } and y ↦ max { 0 , y } y\mapsto\max\{0,y\} y ↦ max { 0 , y } are nonexpansive, Lemma 36 implies that
∣ p θ , q ( x ) − p θ ′ , q ′ ( x ) ∣ ≤ 2 L C H δ , x ∈ S ( θ , θ ′ ) c . (62) |p_{\theta,q}(x)-p_{\theta',q'}(x)|\le2LC_H\delta,\qquad x\in S(\theta,\theta')^c. \tag*{(62)} ∣ p θ , q ( x ) − p θ ′ , q ′ ( x ) ∣ ≤ 2 L C H δ , x ∈ S ( θ , θ ′ ) c . (62)
Step 2: Control on S ( θ , θ ′ ) S(\theta,\theta') S ( θ , θ ′ ) .
Case 1: Suppose s θ ( x ) ≥ 0 s_\theta(x)\ge0 s θ ( x ) ≥ 0 and s θ ′ ( x ) < 0 s_{\theta'}(x)<0 s θ ′ ( x ) < 0 . Then p θ , q ( x ) = U ‾ θ , q ( x ) p_{\theta,q}(x)=\overline{U}_{\theta,q}(x) p θ , q ( x ) = U θ , q ( x ) and p θ ′ , q ′ ( x ) = M ‾ θ ′ , q ′ ( x ) p_{\theta',q'}(x)=\overline{M}_{\theta',q'}(x) p θ ′ , q ′ ( x ) = M θ ′ , q ′ ( x ) . By Lemmas 35 , 36, and 37, we have that
∣ p θ , q ( x ) − p θ ′ , q ′ ( x ) ∣ = ∣ U ‾ θ , q ( x ) − M ‾ θ ′ , q ′ ( x ) ∣ ≤ ∣ U ‾ θ , q ( x ) − M ‾ θ , q ( x ) ∣ + ∣ M ‾ θ , q ( x ) − M ‾ θ ′ , q ′ ( x ) ∣ ≤ 2 L dist ν ( x , H θ ) + 2 L C H δ ≤ 2 L ( C l o c + C H ) δ . \begin{aligned}
|p_{\theta,q}(x)-p_{\theta',q'}(x)|
&=|\overline{U}_{\theta,q}(x)-\overline{M}_{\theta',q'}(x)|\\
&\le|\overline{U}_{\theta,q}(x)-\overline{M}_{\theta,q}(x)|+|\overline{M}_{\theta,q}(x)-\overline{M}_{\theta',q'}(x)|\\
&\le2L\operatorname{dist}_{\nu}(x,\mathcal{H}_{\theta})+2LC_H\delta\\
&\le2L(C_{\mathrm{loc}}+C_H)\delta.
\end{aligned} ∣ p θ , q ( x ) − p θ ′ , q ′ ( x ) ∣ = ∣ U θ , q ( x ) − M θ ′ , q ′ ( x ) ∣ ≤ ∣ U θ , q ( x ) − M θ , q ( x ) ∣ + ∣ M θ , q ( x ) − M θ ′ , q ′ ( x ) ∣ ≤ 2 L dist ν ( x , H θ ) + 2 L C H δ ≤ 2 L ( C loc + C H ) δ .
Case 2: Suppose s θ ( x ) < 0 s_\theta(x)<0 s θ ( x ) < 0 and s θ ′ ( x ) ≥ 0 s_{\theta'}(x)\ge0 s θ ′ ( x ) ≥ 0 . Then p θ , q ( x ) = M ‾ θ , q ( x ) p_{\theta,q}(x)=\overline{M}_{\theta,q}(x) p θ , q ( x ) = M θ , q ( x ) and p θ ′ , q ′ ( x ) = U ‾ θ ′ , q ′ ( x ) p_{\theta',q'}(x)=\overline{U}_{\theta',q'}(x) p θ ′ , q ′ ( x ) = U θ ′ , q ′ ( x ) . By the same argument as Case 1 , we have that
∣ p θ , q ( x ) − p θ ′ , q ′ ( x ) ∣ = ∣ M ‾ θ , q ( x ) − U ‾ θ ′ , q ′ ( x ) ∣ ≤ ∣ M ‾ θ , q ( x ) − U ‾ θ , q ( x ) ∣ + ∣ U ‾ θ , q ( x ) − U ‾ θ ′ , q ′ ( x ) ∣ ≤ 2 L dist ν ( x , H θ ) + 2 L C H δ ≤ 2 L ( C l o c + C H ) δ . \begin{aligned}
|p_{\theta,q}(x)-p_{\theta',q'}(x)|
&=|\overline{M}_{\theta,q}(x)-\overline{U}_{\theta',q'}(x)|\\
&\le|\overline{M}_{\theta,q}(x)-\overline{U}_{\theta,q}(x)|+|\overline{U}_{\theta,q}(x)-\overline{U}_{\theta',q'}(x)|\\
&\le2L\operatorname{dist}_{\nu}(x,\mathcal{H}_{\theta})+2LC_H\delta\\
&\le2L(C_{\mathrm{loc}}+C_H)\delta.
\end{aligned} ∣ p θ , q ( x ) − p θ ′ , q ′ ( x ) ∣ = ∣ M θ , q ( x ) − U θ ′ , q ′ ( x ) ∣ ≤ ∣ M θ , q ( x ) − U θ , q ( x ) ∣ + ∣ U θ , q ( x ) − U θ ′ , q ′ ( x ) ∣ ≤ 2 L dist ν ( x , H θ ) + 2 L C H δ ≤ 2 L ( C loc + C H ) δ .
Combining Cases 1 and 2 with (62) proves the claim with C t r = 2 L ( C l o c + C H ) C_{\mathrm{tr}}=2L(C_{\mathrm{loc}}+C_H) C tr = 2 L ( C loc + C H ) .
Lemma 39 (Sup-norm entropy of the canonical policy class). Suppose Assumption 4 holds and d ≥ 2 d\ge2 d ≥ 2 . Let P L = { p θ , q : θ ∈ Θ ~ , q ∈ Q θ , L } \mathcal{P}_L=\{p_{\theta,q}:\theta\in\widetilde{\Theta},\ q\in\mathcal{Q}_{\theta,L}\} P L = { p θ , q : θ ∈ Θ , q ∈ Q θ , L } . Then there exists a constant C P < ∞ C_P<\infty C P < ∞ , depending only on d , ν , L d,\nu,L d , ν , L and the constants in Lemma 14 , such that, for every ϵ ∈ ( 0 , 1 ) \epsilon\in(0,1) ϵ ∈ ( 0 , 1 ) ,
log N ( ϵ , P L , ∥ ⋅ ∥ ∞ ) ≤ C P ϵ − ( d − 1 ) log ( C P / ϵ ) . \log N(\epsilon,\mathcal{P}_L,\lVert\cdot\rVert_\infty)\le C_P\epsilon^{-(d-1)}\log(C_P/\epsilon). log N ( ϵ , P L , ∥ ⋅ ∥ ∞ ) ≤ C P ϵ − ( d − 1 ) log ( C P / ϵ ) .
Proof. Write m = d − 1 m=d-1 m = d − 1 . If L = 0 L=0 L = 0 , then every q ∈ Q θ , 0 q\in\mathcal{Q}_{\theta,0} q ∈ Q θ , 0 is constant on H θ \mathcal{H}_{\theta} H θ , the upper and lower canonical envelopes equal that constant on [ 0 , 1 ] d [0,1]^d [ 0 , 1 ] d , and hence P 0 \mathcal{P}_0 P 0 is the class of constant functions with values in [ 0 , 1 ] [0,1] [ 0 , 1 ] . Discretising [ 0 , 1 ] [0,1] [ 0 , 1 ] proves the result. Suppose henceforth that L > 0 L>0 L > 0 .
For each r ∈ { 1 , … , d } r\in\{1,\ldots,d\} r ∈ { 1 , … , d } , define
P L , r = { p θ , q : θ ∈ Θ ~ r , q ∈ Q θ , L } . \mathcal{P}_{L,r}=\{p_{\theta,q}:\theta\in\widetilde{\Theta}_r,\ q\in\mathcal{Q}_{\theta,L}\}. P L , r = { p θ , q : θ ∈ Θ r , q ∈ Q θ , L } .
Then P L = ⋃ r = 1 d P L , r \mathcal{P}_L=\bigcup_{r=1}^{d}\mathcal{P}_{L,r} P L = ⋃ r = 1 d P L , r . We first bound the covering number of each fixed chart P L , r \mathcal{P}_{L,r} P L , r .
Fix r ∈ { 1 , … , d } r\in\{1,\ldots,d\} r ∈ { 1 , … , d } and ϵ ∈ ( 0 , 1 ) \epsilon\in(0,1) ϵ ∈ ( 0 , 1 ) . Let δ θ = ϵ / ( 2 C t r ) \delta_\theta=\epsilon/(2C_{\mathrm{tr}}) δ θ = ϵ / ( 2 C tr ) . Let { θ 1 , … , θ M } \{\theta_1,\ldots,\theta_M\} { θ 1 , … , θ M } be a δ θ \delta_\theta δ θ -net of Θ ~ r \widetilde{\Theta}_r Θ r with respect to ∥ ⋅ ∥ 2 \lVert\cdot\rVert_2 ∥ ⋅ ∥ 2 . Since Θ ~ r ⊂ R d + 1 \widetilde{\Theta}_r\subset\mathbb{R}^{d+1} Θ r ⊂ R d + 1 is bounded by the definition of Θ ~ r \widetilde{\Theta}_r Θ r , we have that
log M ≤ ( d + 1 ) log ( C ′ / δ θ ) ≤ C log ( C / ϵ ) , \log M\le(d+1)\log(C'/\delta_\theta)\le C\log(C/\epsilon), log M ≤ ( d + 1 ) log ( C ′ / δ θ ) ≤ C log ( C / ϵ ) ,
where C C C is a constant depending on d d d .
For each fixed representative boundary H θ j \mathcal{H}_{\theta_j} H θ j , Lemma 21 gives a finite ε / 2 \varepsilon/2 ε /2 -cover of Q θ j , L Q_{\theta_j,L} Q θ j , L in the sup-norm. Thus there exists a finite set Q ~ j ⊂ Q θ j , L \widetilde{Q}_j \subset Q_{\theta_j,L} Q j ⊂ Q θ j , L such that, for every q ∘ ∈ Q θ j , L q^\circ\in Q_{\theta_j,L} q ∘ ∈ Q θ j , L , there exists q ~ ∈ Q ~ j \widetilde{q} \in\widetilde{Q}_j q ∈ Q j satisfying ∥ q ∘ − q ~ ∥ ∞ ≤ ε / 2 \lVert q^\circ- \widetilde{q}\rVert_\infty\le\varepsilon/2 ∥ q ∘ − q ∥ ∞ ≤ ε /2 . Moreover, it holds that log ∣ Q ~ j ∣ ≤ C ε − m log ( C / ε ) \log\lvert\widetilde{Q}_j\rvert\le C\varepsilon^{-m}\log(C/\varepsilon) log ∣ Q j ∣ ≤ C ε − m log ( C / ε ) .
Now fix any p ∈ P L , r p \in\mathcal{P}_{L,r} p ∈ P L , r , and we can write p = p θ , q p = p_{\theta,q} p = p θ , q for some θ ∈ Θ ~ r \theta\in\widetilde{\Theta}_r θ ∈ Θ r and q ∈ Q θ , L q \in Q_{\theta,L} q ∈ Q θ , L . Choose θ j \theta_j θ j such that ∥ θ − θ j ∥ 2 ≤ δ θ \lVert\theta- \theta_j\rVert_2 \le\delta_\theta ∥ θ − θ j ∥ 2 ≤ δ θ . Since θ \theta θ and θ j \theta_j θ j lie in the same chart Θ ~ r \widetilde{\Theta}_r Θ r , by Lemma 38, there exists q ∘ ∈ Q θ j , L q^\circ\in Q_{\theta_j,L} q ∘ ∈ Q θ j , L such that
∥ p θ , q − p θ j , q ∘ ∥ ∞ ≤ C t r δ θ = ε / 2. (63) \lVert p_{\theta,q} - p_{\theta_j,q^\circ}\rVert_\infty\le C_{\mathrm{tr}}\delta_\theta= \varepsilon/2. \tag*{(63)} ∥ p θ , q − p θ j , q ∘ ∥ ∞ ≤ C tr δ θ = ε /2. (63)
Choose q ~ ∈ Q ~ j \widetilde{q} \in\widetilde{Q}_j q ∈ Q j such that ∥ q ∘ − q ~ ∥ ∞ ≤ ε / 2 \lVert q^\circ- \widetilde{q}\rVert_\infty\le\varepsilon/2 ∥ q ∘ − q ∥ ∞ ≤ ε /2 . Since the envelope map is nonexpansive in the boundary trace, we have that
∥ p θ j , q ∘ − p θ j , q ~ ∥ ∞ ≤ ε / 2. (64) \lVert p_{\theta_j,q^\circ} - p_{\theta_j,\widetilde{q}}\rVert_\infty\le\varepsilon/2. \tag*{(64)} ∥ p θ j , q ∘ − p θ j , q ∥ ∞ ≤ ε /2. (64)
Therefore, by (63) , (64) and the triangle inequality, the policies p θ j , q ~ p_{\theta_j,\widetilde{q}} p θ j , q form an ε \varepsilon ε -cover of P L , r \mathcal{P}_{L,r} P L , r .
Thus, for every fixed r ∈ { 1 , … , d } r \in\{1,\ldots,d\} r ∈ { 1 , … , d } , we have that
log N ( ε , P L , r , ∥ ⋅ ∥ ∞ ) ≤ C log ( C / ε ) + C ε − m log ( C / ε ) ≤ C P ε − m log ( C P / ε ) . \log N(\varepsilon,\mathcal{P}_{L,r},\lVert\cdot\rVert_\infty) \le C\log(C/\varepsilon)+C\varepsilon^{-m}\log(C/\varepsilon) \le C_{\mathcal{P}}\varepsilon^{-m}\log(C_{\mathcal{P}}/\varepsilon). log N ( ε , P L , r , ∥ ⋅ ∥ ∞ ) ≤ C log ( C / ε ) + C ε − m log ( C / ε ) ≤ C P ε − m log ( C P / ε ) .
Since P L = ⋃ r = 1 d P L , r \mathcal{P}_L=\bigcup_{r=1}^{d}\mathcal{P}_{L,r} P L = ⋃ r = 1 d P L , r , the union of the chart-wise covers is an ε \varepsilon ε -cover of P L \mathcal{P}_L P L . Hence,
N ( ε , P L , ∥ ⋅ ∥ ∞ ) ≤ ∑ r = 1 d N ( ε , P L , r , ∥ ⋅ ∥ ∞ ) . N(\varepsilon,\mathcal{P}_L,\lVert\cdot\rVert_\infty) \le\sum_{r=1}^{d}N(\varepsilon,\mathcal{P}_{L,r},\lVert\cdot\rVert_\infty). N ( ε , P L , ∥ ⋅ ∥ ∞ ) ≤ r = 1 ∑ d N ( ε , P L , r , ∥ ⋅ ∥ ∞ ) .
Taking logarithms and absorbing log ( d ) \log(d) log ( d ) into the constant prove the claim. □
Lemma 40 (L 2 ( P ) L_2(P) L 2 ( P ) bracketing entropy). Suppose Assumptions 3 and 4 hold and d ≥ 2 d \ge2 d ≥ 2 . For b = ( b 1 , b 2 ) b=(b_1,b_2) b = ( b 1 , b 2 ) and a policy p : [ 0 , 1 ] d → [ 0 , 1 ] p:[0,1]^d\to[0,1] p : [ 0 , 1 ] d → [ 0 , 1 ] , define
r b , p ( x ) = x ~ ⊤ b 1 p ( x ) + x ~ ⊤ b 2 { 1 − p ( x ) } . r_{b,p}(x)=\widetilde{x}^{\top}b_1p(x)+\widetilde{x}^{\top}b_2\{1-p(x)\}. r b , p ( x ) = x ⊤ b 1 p ( x ) + x ⊤ b 2 { 1 − p ( x )} . Let
F U C B = { x ↦ r b θ , q , p θ , q ( x ) : ∥ b k ∥ 2 ≤ C β + 1 , k ∈ { 1 , 2 } , θ = b 1 − b 2 ∈ Θ ~ , q ∈ Q θ , L } . \mathcal{F}_{\mathrm{UCB}}=\left\{x\mapsto r_{b_{\theta,q},p_{\theta,q}}(x):\lVert b_k\rVert_2\le C_\beta+1,\ k\in\{1,2\},\ \theta=b_1-b_2\in\widetilde{\Theta},\ q\in Q_{\theta,L}\right\}. F UCB = { x ↦ r b θ , q , p θ , q ( x ) : ∥ b k ∥ 2 ≤ C β + 1 , k ∈ { 1 , 2 } , θ = b 1 − b 2 ∈ Θ , q ∈ Q θ , L } . Let P P P be an arbitrary Borel probability measure supported on [ 0 , 1 ] d [0,1]^d [ 0 , 1 ] d . Then there exists a constant C F < ∞ C_F<\infty C F < ∞ , depending only on d d d , ν \nu ν , L L L , C β C_\beta C β and the constants in Lemma 14 , and independent of P P P , such that, for every ε ∈ ( 0 , 1 ) \varepsilon\in(0,1) ε ∈ ( 0 , 1 ) ,
log N [ ] ( ε , F U C B , L 2 ( P ) ) ≤ C F ε − ( d − 1 ) log ( C F / ε ) , \log N_{[]}(\varepsilon,\mathcal{F}_{\mathrm{UCB}},L_2(P))\le C_F\varepsilon^{-(d-1)}\log(C_F/\varepsilon), log N [ ] ( ε , F UCB , L 2 ( P )) ≤ C F ε − ( d − 1 ) log ( C F / ε ) , where N [ ] ( ε , F U C B , L 2 ( P ) ) N_{[]}(\varepsilon,\mathcal{F}_{\mathrm{UCB}},L_2(P)) N [ ] ( ε , F UCB , L 2 ( P )) denotes the L 2 ( P ) L_2(P) L 2 ( P ) bracketing number.
Proof. Write m = d − 1 m=d-1 m = d − 1 , R X = sup x ∈ [ 0 , 1 ] d ∥ x ~ ∥ 2 = d + 1 R_X=\sup_{x\in[0,1]^d}\lVert\widetilde{x}\rVert_2=\sqrt{d+1} R X = sup x ∈ [ 0 , 1 ] d ∥ x ∥ 2 = d + 1 and B Θ = 1 ∨ sup θ ∈ Θ ~ ∥ θ ∥ 2 B_\Theta=1\vee\sup_{\theta\in\widetilde{\Theta}}\lVert\theta\rVert_2 B Θ = 1 ∨ sup θ ∈ Θ ∥ θ ∥ 2 . By Lemma 14 , it holds that B Θ < ∞ B_\Theta<\infty B Θ < ∞ . Let
δ b = ε 6 R X , δ θ = ε 6 R X , δ p = ε 6 R X B Θ . \delta_b=\frac{\varepsilon}{6R_X},\qquad\delta_\theta=\frac{\varepsilon}{6R_X},\qquad\delta_p=\frac{\varepsilon}{6R_XB_\Theta}. δ b = 6 R X ε , δ θ = 6 R X ε , δ p = 6 R X B Θ ε .
Let B β = { b ∈ R d + 1 : ∥ b ∥ 2 ≤ C β + 1 } \mathcal{B}_\beta=\{b\in\mathbb{R}^{d+1}:\lVert b\rVert_2\le C_\beta+1\} B β = { b ∈ R d + 1 : ∥ b ∥ 2 ≤ C β + 1 } . Since B β \mathcal{B}_\beta B β is a bounded subset of R d + 1 \mathbb{R}^{d+1} R d + 1 , there exists a δ b \delta_b δ b -net { v 1 , … , v N b } \{v_1,\ldots,v_{N_b}\} { v 1 , … , v N b } of B β \mathcal{B}_\beta B β under ∥ ⋅ ∥ 2 \lVert\cdot\rVert_2 ∥ ⋅ ∥ 2 satisfying
log N b ≤ ( d + 1 ) log ( C / δ b ) ≤ C log ( C / ε ) . \log N_b\le(d+1)\log(C/\delta_b)\le C\log(C/\varepsilon). log N b ≤ ( d + 1 ) log ( C / δ b ) ≤ C log ( C / ε ) .
Similarly, since Θ ~ \widetilde{\Theta} Θ is a bounded subset of R d + 1 \mathbb{R}^{d+1} R d + 1 , there exists a δ θ \delta_\theta δ θ -net { ϑ 1 , … , ϑ N θ } \{\vartheta_1,\ldots,\vartheta_{N_\theta}\} { ϑ 1 , … , ϑ N θ } of Θ ~ \widetilde{\Theta} Θ under ∥ ⋅ ∥ 2 \lVert\cdot\rVert_2 ∥ ⋅ ∥ 2 satisfying
log N θ ≤ ( d + 1 ) log ( C / δ θ ) ≤ C log ( C / ε ) . \log N_\theta\le(d+1)\log(C/\delta_\theta)\le C\log(C/\varepsilon). log N θ ≤ ( d + 1 ) log ( C / δ θ ) ≤ C log ( C / ε ) .
By Lemma 39 , there exists a δ p \delta_p δ p -cover { π 1 , … , π N p } \{\pi_1,\ldots,\pi_{N_p}\} { π 1 , … , π N p } of P L \mathcal{P}_L P L under the sup-norm with
log N p ≤ C δ p − m log ( C / δ p ) ≤ C ε − m log ( C / ε ) . \log N_p \le C\delta_p^{-m}\log(C/\delta_p) \le C\varepsilon^{-m}\log(C/\varepsilon). log N p ≤ C δ p − m log ( C / δ p ) ≤ C ε − m log ( C / ε ) .
Now fix any f ∈ F U C B f \in\mathcal{F}_{\mathrm{UCB}} f ∈ F UCB . We can write
f ( x ) = x ~ T b 2 + x ~ T θ p θ , q ( x ) f(x) = \widetilde{x}^{\mathsf{T}}b_2+\widetilde{x}^{\mathsf{T}}\theta p_{\theta,q}(x) f ( x ) = x T b 2 + x T θ p θ , q ( x )
for some b 1 , b 2 ∈ B β b_1,b_2 \in\mathcal{B}_{\beta} b 1 , b 2 ∈ B β , θ = b 1 − b 2 ∈ Θ ~ \theta=b_1-b_2 \in\widetilde{\Theta} θ = b 1 − b 2 ∈ Θ and q ∈ Q θ , L q \in\mathcal{Q}_{\theta,L} q ∈ Q θ , L . Choose v ℓ v_\ell v ℓ , ϑ j \vartheta_j ϑ j and π k \pi_k π k such that
∥ b 2 − v ℓ ∥ 2 ≤ δ b , ∥ θ − ϑ j ∥ 2 ≤ δ θ , ∥ p θ , q − π k ∥ ∞ ≤ δ p . \lVert b_2-v_\ell\rVert_2\le\delta_b,\qquad\lVert\theta-\vartheta_j\rVert_2\le\delta_\theta,\qquad\lVert p_{\theta,q}-\pi_k\rVert_\infty\le\delta_p. ∥ b 2 − v ℓ ∥ 2 ≤ δ b , ∥ θ − ϑ j ∥ 2 ≤ δ θ , ∥ p θ , q − π k ∥ ∞ ≤ δ p .
Define
f ℓ , j , k ( x ) = x ~ T v ℓ + x ~ T ϑ j π k ( x ) . f_{\ell,j,k}(x)=\widetilde{x}^{\mathsf{T}}v_\ell+\widetilde{x}^{\mathsf{T}}\vartheta_j\pi_k(x). f ℓ , j , k ( x ) = x T v ℓ + x T ϑ j π k ( x ) .
Since 0 ≤ p θ , q ≤ 1 0\le p_{\theta,q}\le1 0 ≤ p θ , q ≤ 1 , for every x ∈ [ 0 , 1 ] d x\in[0,1]^d x ∈ [ 0 , 1 ] d we have that
∣ f ( x ) − f ℓ , j , k ( x ) ∣ ≤ ∣ x ~ T ( b 2 − v ℓ ) ∣ + ∣ x ~ T ( θ − ϑ j ) ∣ p θ , q ( x ) + ∣ x ~ T ϑ j ∣ ∣ p θ , q ( x ) − π k ( x ) ∣ ≤ R X δ b + R X δ θ + R X B Θ δ p = ε 2 . \begin{aligned}
|f(x)-f_{\ell,j,k}(x)|
&\le\left|\widetilde{x}^{\mathsf{T}}(b_2-v_\ell)\right|+\left|\widetilde{x}^{\mathsf{T}}(\theta-\vartheta_j)\right|p_{\theta,q}(x)+\left|\widetilde{x}^{\mathsf{T}}\vartheta_j\right||p_{\theta,q}(x)-\pi_k(x)|\\
&\le R_X\delta_b+R_X\delta_\theta+R_XB_\Theta\delta_p=\frac{\varepsilon}{2}.
\end{aligned} ∣ f ( x ) − f ℓ , j , k ( x ) ∣ ≤ x T ( b 2 − v ℓ ) + x T ( θ − ϑ j ) p θ , q ( x ) + x T ϑ j ∣ p θ , q ( x ) − π k ( x ) ∣ ≤ R X δ b + R X δ θ + R X B Θ δ p = 2 ε .
For each triple ( ℓ , j , k ) (\ell,j,k) ( ℓ , j , k ) , define the bracket
ℓ ℓ , j , k ( x ) = f ℓ , j , k ( x ) − ε / 2 \ell_{\ell,j,k}(x)=f_{\ell,j,k}(x)-\varepsilon/2 ℓ ℓ , j , k ( x ) = f ℓ , j , k ( x ) − ε /2
and
u ℓ , j , k ( x ) = f ℓ , j , k ( x ) + ε / 2. u_{\ell,j,k}(x)=f_{\ell,j,k}(x)+\varepsilon/2. u ℓ , j , k ( x ) = f ℓ , j , k ( x ) + ε /2.
For every f ∈ F U C B f\in\mathcal{F}_{\mathrm{UCB}} f ∈ F UCB , there exists a triple ( ℓ , j , k ) (\ell,j,k) ( ℓ , j , k ) such that
ℓ ℓ , j , k ( x ) ≤ f ( x ) ≤ u ℓ , j , k ( x ) \ell_{\ell,j,k}(x)\le f(x)\le u_{\ell,j,k}(x) ℓ ℓ , j , k ( x ) ≤ f ( x ) ≤ u ℓ , j , k ( x )
for every x ∈ [ 0 , 1 ] d x\in[0,1]^d x ∈ [ 0 , 1 ] d . Moreover, the L 2 ( P ) L_2(P) L 2 ( P ) width of this bracket satisfies
∥ u ℓ , j , k − ℓ ℓ , j , k ∥ L 2 ( P ) = ε . \lVert u_{\ell,j,k}-\ell_{\ell,j,k}\rVert_{L_2(P)}=\varepsilon. ∥ u ℓ , j , k − ℓ ℓ , j , k ∥ L 2 ( P ) = ε .
Thus the brackets indexed by ( ℓ , j , k ) (\ell,j,k) ( ℓ , j , k ) form an L 2 ( P ) L_2(P) L 2 ( P ) ε \varepsilon ε -bracketing cover of F U C B \mathcal{F}_{\mathrm{UCB}} F UCB . Since the number of brackets is at most N b N θ N p N_bN_\theta N_p N b N θ N p , we have that
log N [ ] ( ε , F U C B , L 2 ( P ) ) ≤ log N b + log N θ + log N p ≤ C log ( C / ε ) + C ε − m log ( C / ε ) ≤ C F ε − m log ( C F / ε ) . \begin{aligned}
\log N_{[]}(\varepsilon,\mathcal{F}_{\mathrm{UCB}},L_2(P))\le\log N_b+\log N_\theta+\log N_p \\
&\le C\log(C/\varepsilon)+C\varepsilon^{-m}\log(C/\varepsilon) \\
&\le C_F\varepsilon^{-m}\log(C_F/\varepsilon).
\end{aligned} log N [ ] ( ε , F UCB , L 2 ( P )) ≤ log N b + log N θ + log N p ≤ C log ( C / ε ) + C ε − m log ( C / ε ) ≤ C F ε − m log ( C F / ε ) .
Since the construction is independent of P P P , the claim follows. □
Lemma 41 (Conditional Rademacher complexity). Suppose the conditions of Lemma 40 hold. For arbitrary deterministic points x 1 , … , x n ∈ [ 0 , 1 ] d x_1,\ldots,x_n\in[0,1]^d x 1 , … , x n ∈ [ 0 , 1 ] d , let P n = n − 1 ∑ i = 1 n δ x i P_n=n^{-1}\sum_{i=1}^n\delta_{x_i} P n = n − 1 ∑ i = 1 n δ x i be the empirical measure and define
R ^ n ( F U C B ) = E σ [ sup f ∈ F U C B ∣ 1 n ∑ i = 1 n σ i f ( x i ) ∣ ] , \widehat{\mathcal{R}}_n(\mathcal{F}_{\mathrm{UCB}})=\mathbb{E}_{\sigma}\left[\sup_{f\in\mathcal{F}_{\mathrm{UCB}}\!}\left|\frac{1}{n}\sum_{i=1}^n\sigma_i f(x_i)\right|\right], R n ( F UCB ) = E σ [ f ∈ F UCB sup n 1 i = 1 ∑ n σ i f ( x i ) ] , where σ 1 , … , σ n \sigma_1,\ldots,\sigma_n σ 1 , … , σ n are independent Rademacher random variables. Then there exists a constant C < ∞ C<\infty C < ∞ , depending only on d , ν , L , C β d,\nu,L,C_\beta d , ν , L , C β and the constants in Lemma 14 , and independent of x 1 , … , x n x_1,\ldots,x_n x 1 , … , x n , such that, with m = d − 1 m=d-1 m = d − 1 ,
R ^ n ( F U C B ) ≤ C { n − 1 / 2 , m = 1 , n − 1 / 2 log 3 / 2 ( e n ) , m = 2 , { log ( e n ) / n } 1 / m , m ≥ 3. \widehat{\mathcal{R}}_n(\mathcal{F}_{\mathrm{UCB}})\le C
\begin{cases}
n^{-1/2}, & m=1,\\
n^{-1/2}\log^{3/2}(en), & m=2,\\
\{\log(en)/n\}^{1/m}, & m\ge3.
\end{cases} R n ( F UCB ) ≤ C ⎩ ⎨ ⎧ n − 1/2 , n − 1/2 log 3/2 ( e n ) , { log ( e n ) / n } 1/ m , m = 1 , m = 2 , m ≥ 3.
Proof. For n = 1 n=1 n = 1 , the bound follows from sup f ∈ F U C B ∥ f ∥ ∞ ≤ d + 1 C β \sup_{f\in\mathcal{F}_{\mathrm{UCB}}}\|f\|_{\infty}\le\sqrt{d+1}C_{\beta} sup f ∈ F UCB ∥ f ∥ ∞ ≤ d + 1 C β , so assume n ≥ 2 n\ge2 n ≥ 2 . By Lemma 22 , selecting one function from every nonempty bracket gives, for every probability measure Q Q Q supported on [ 0 , 1 ] d [0,1]^d [ 0 , 1 ] d ,
log N ( ε , F U C B , L 2 ( Q ) ) ≤ C 1 ε − m log ( C 1 / ε ) , ε ∈ ( 0 , 1 ) . (65) \log N(\varepsilon,\mathcal{F}_{\mathrm{UCB}},L_{2}(Q))\le C_{1}\varepsilon^{-m}\log(C_{1}/\varepsilon),\qquad\varepsilon\in(0,1). \tag*{(65)} log N ( ε , F UCB , L 2 ( Q )) ≤ C 1 ε − m log ( C 1 / ε ) , ε ∈ ( 0 , 1 ) . (65)
In particular, this holds for Q = P n Q=P_{n} Q = P n . Let F U C B ± = F U C B ∪ ( − F U C B ) ∪ { 0 } \mathcal{F}_{\mathrm{UCB}}^{\pm}=\mathcal{F}_{\mathrm{UCB}}\cup(-\mathcal{F}_{\mathrm{UCB}})\cup\{0\} F UCB ± = F UCB ∪ ( − F UCB ) ∪ { 0 } and consider the Rademacher process Z f = n − 1 ∑ i = 1 n σ i f ( x i ) Z_{f}=n^{-1}\sum_{i=1}^{n}\sigma_{i}f(x_{i}) Z f = n − 1 ∑ i = 1 n σ i f ( x i ) indexed by F U C B ± \mathcal{F}_{\mathrm{UCB}}^{\pm} F UCB ± . Conditionally on x 1 , … , x n x_{1},\ldots,x_{n} x 1 , … , x n , its increments satisfy
E σ exp { λ ( Z f − Z g ) } = E σ exp { λ n ∑ i = 1 n σ i { f ( x i ) − g ( x i ) } } = ∏ i = 1 n E σ i exp { λ n σ i { f ( x i ) − g ( x i ) } } ≤ ∏ i = 1 n exp { λ 2 { f ( x i ) − g ( x i ) } 2 2 n 2 } = exp { λ 2 2 n ∥ f − g ∥ L 2 ( P n ) 2 } , \begin{aligned}
\mathbb{E}_{\sigma}\exp\{\lambda(Z_{f}-Z_{g})\}
&=\mathbb{E}_{\sigma}\exp\left\{\frac{\lambda}{n}\sum_{i=1}^{n}\sigma_{i}\{f(x_{i})-g(x_{i})\}\right\}\\
&=\prod_{i=1}^{n}\mathbb{E}_{\sigma_{i}}\exp\left\{\frac{\lambda}{n}\sigma_{i}\{f(x_{i})-g(x_{i})\}\right\}\\
&\le\prod_{i=1}^{n}\exp\left\{\frac{\lambda^{2}\{f(x_{i})-g(x_{i})\}^{2}}{2n^{2}}\right\}\\
&=\exp\left\{\frac{\lambda^{2}}{2n}\|f-g\|_{L_{2}(P_{n})}^{2}\right\},
\end{aligned} E σ exp { λ ( Z f − Z g )} = E σ exp { n λ i = 1 ∑ n σ i { f ( x i ) − g ( x i )} } = i = 1 ∏ n E σ i exp { n λ σ i { f ( x i ) − g ( x i )} } ≤ i = 1 ∏ n exp { 2 n 2 λ 2 { f ( x i ) − g ( x i ) } 2 } = exp { 2 n λ 2 ∥ f − g ∥ L 2 ( P n ) 2 } ,
where the second equality follows from the independence of σ 1 , … , σ n \sigma_{1},\ldots,\sigma_{n} σ 1 , … , σ n . Thus, the process is sub-Gaussian under the pseudometric d n ( f , g ) = ∥ f − g ∥ L 2 ( P n ) n d_{n}(f,g)=\|f-g\|_{L_{2}(P_{n})}\sqrt{n} d n ( f , g ) = ∥ f − g ∥ L 2 ( P n ) n .
Since F U C B ± \mathcal{F}_{\mathrm{UCB}}^{\pm} F UCB ± is symmetric and contains the zero function, we have
E σ [ sup f , g ∈ F U C B ± ( Z f − Z g ) ] = 2 E σ [ sup f ∈ F U C B ± Z f ] ≥ 2 R ^ n ( F U C B ) . \mathbb{E}_{\sigma}\left[\sup_{f,g\in\mathcal{F}_{\mathrm{UCB}}^{\pm}}(Z_{f}-Z_{g})\right]
=2\mathbb{E}_{\sigma}\left[\sup_{f\in\mathcal{F}_{\mathrm{UCB}}^{\pm}}Z_{f}\right]
\ge2\widehat{\mathfrak{R}}_{n}(\mathcal{F}_{\mathrm{UCB}}). E σ [ f , g ∈ F UCB ± sup ( Z f − Z g ) ] = 2 E σ [ f ∈ F UCB ± sup Z f ] ≥ 2 R n ( F UCB ) .
Let D n = diam ( F U C B ± , d n ) D_{n}=\operatorname{diam}(\mathcal{F}_{\mathrm{UCB}}^{\pm},d_{n}) D n = diam ( F UCB ± , d n ) . If d n ( f , g ) ≤ δ d_{n}(f,g)\le\delta d n ( f , g ) ≤ δ , the Cauchy–Schwarz inequality gives
∣ Z f − Z g ∣ ≤ ∥ f − g ∥ L 2 ( P n ) = n d n ( f , g ) ≤ n δ . |Z_{f}-Z_{g}|\le\|f-g\|_{L_{2}(P_{n})}=\sqrt{n}\,d_{n}(f,g)\le\sqrt{n}\,\delta. ∣ Z f − Z g ∣ ≤ ∥ f − g ∥ L 2 ( P n ) = n d n ( f , g ) ≤ n δ .
Therefore, the truncated Dudley entropy bound (e.g. Theorem 5.22 in Wainwright, [51 ] ) implies
2 R ^ n ( F U C B ) ≤ 2 n δ + 32 ∫ δ / 4 D n log N ( u , F U C B ± , d n ) d u = 2 n δ + 32 n ∫ n δ / 4 n D n log N ( ε , F U C B ± , L 2 ( P n ) ) d ε , \begin{aligned}
2\widehat{\mathfrak{R}}_{n}(\mathcal{F}_{\mathrm{UCB}})
\le2\sqrt{n}\,\delta+32\int_{\delta/4}^{D_{n}}\sqrt{\log N(u,\mathcal{F}_{\mathrm{UCB}}^{\pm},d_{n})}\,du \\
&=2\sqrt{n}\,\delta+\frac{32}{\sqrt{n}}\int_{\sqrt{n}\delta/4}^{\sqrt{n}D_{n}}\sqrt{\log N(\varepsilon,\mathcal{F}_{\mathrm{UCB}}^{\pm},L_{2}(P_{n}))}\,d\varepsilon,
\end{aligned} 2 R n ( F UCB ) ≤ 2 n δ + 32 ∫ δ /4 D n log N ( u , F UCB ± , d n ) d u = 2 n δ + n 32 ∫ n δ /4 n D n log N ( ε , F UCB ± , L 2 ( P n )) d ε ,
where the equality follows from N ( u , F U C B ± , d n ) = N ( n u , F U C B ± , L 2 ( P n ) ) N(u,\mathcal{F}_{\mathrm{UCB}}^{\pm},d_{n})=N(\sqrt{n}u,\mathcal{F}_{\mathrm{UCB}}^{\pm},L_{2}(P_{n})) N ( u , F UCB ± , d n ) = N ( n u , F UCB ± , L 2 ( P n )) , and the change of variables ε = n u \varepsilon=\sqrt{n}u ε = n u . Moreover, since
N ( ε , F U C B ± , L 2 ( P n ) ) ≤ 2 N ( ε , F U C B , L 2 ( P n ) ) + 1 , N(\varepsilon,\mathcal{F}_{\mathrm{UCB}}^{\pm},L_{2}(P_{n}))\le2N(\varepsilon,\mathcal{F}_{\mathrm{UCB}},L_{2}(P_{n}))+1, N ( ε , F UCB ± , L 2 ( P n )) ≤ 2 N ( ε , F UCB , L 2 ( P n )) + 1 ,
and every f ∈ F U C B f\in\mathcal{F}_{\mathrm{UCB}} f ∈ F UCB satisfies
∥ f ∥ ∞ ≤ sup x ∈ [ 0 , 1 ] d ∥ x ~ ∥ 2 C β = d + 1 C β , \|f\|_{\infty}\le\sup_{x\in[0,1]^{d}}\|\widetilde{x}\|_{2}C_{\beta}=\sqrt{d+1}C_{\beta}, ∥ f ∥ ∞ ≤ x ∈ [ 0 , 1 ] d sup ∥ x ∥ 2 C β = d + 1 C β ,
we have that n D n = diam ( F U C B ± , L 2 ( P n ) ) \sqrt{n}D_{n}=\operatorname{diam}(\mathcal{F}_{\mathrm{UCB}}^{\pm},L_{2}(P_{n})) n D n = diam ( F UCB ± , L 2 ( P n )) is bounded by a fixed constant. Therefore, setting α = n δ / 4 \alpha=\sqrt{n}\delta/4 α = n δ /4 , rescaling the fixed upper endpoint to one, and absorbing universal constants, (64) yields
R ^ n ( F U C B ) ≤ C 2 inf α ∈ ( 0 , 1 ) { α + 1 n ∫ α 1 ε − m / 2 { log ( C 1 / ε ) } 1 / 2 d ε } . (66) \widehat{\mathfrak{R}}_{n}(\mathcal{F}_{\mathrm{UCB}})
\le C_{2}\inf_{\alpha\in(0,1)}
\left\{\alpha+\frac{1}{\sqrt{n}}\int_{\alpha}^{1}\varepsilon^{-m/2}\{\log(C_{1}/\varepsilon)\}^{1/2}\,d\varepsilon\right\}. \tag*{(66)} R n ( F UCB ) ≤ C 2 α ∈ ( 0 , 1 ) inf { α + n 1 ∫ α 1 ε − m /2 { log ( C 1 / ε ) } 1/2 d ε } . (66)
Case 1: m = 1 m=1 m = 1 . In this case,
∫ α 1 ε − 1 / 2 { log ( C 1 / ε ) } 1 / 2 d ε ≤ ∫ 0 1 ε − 1 / 2 { log ( C 1 / ε ) } 1 / 2 d ε ≤ log C 1 ∫ 0 1 ε − 1 / 2 d ε + ∫ 0 1 ε − 1 / 2 { log ( 1 / ε ) } 1 / 2 d ε = 2 log C 1 + ∫ 0 ∞ u 1 / 2 e − u / 2 d u = : C 3 . \begin{aligned}
\int_{\alpha}^{1}\varepsilon^{-1/2}\{\log(C_1/\varepsilon)\}^{1/2}\,\mathrm{d}\varepsilon\leq\int_{0}^{1}\varepsilon^{-1/2}\{\log(C_1/\varepsilon)\}^{1/2}\,\mathrm{d}\varepsilon\\
&\leq\sqrt{\log C_1}\int_{0}^{1}\varepsilon^{-1/2}\,\mathrm{d}\varepsilon+\int_{0}^{1}\varepsilon^{-1/2}\{\log(1/\varepsilon)\}^{1/2}\,\mathrm{d}\varepsilon\\
&=2\sqrt{\log C_1}+\int_{0}^{\infty}u^{1/2}e^{-u/2}\,\mathrm{d}u=:C_3.
\end{aligned} ∫ α 1 ε − 1/2 { log ( C 1 / ε ) } 1/2 d ε ≤ ∫ 0 1 ε − 1/2 { log ( C 1 / ε ) } 1/2 d ε ≤ log C 1 ∫ 0 1 ε − 1/2 d ε + ∫ 0 1 ε − 1/2 { log ( 1/ ε ) } 1/2 d ε = 2 log C 1 + ∫ 0 ∞ u 1/2 e − u /2 d u =: C 3 .
Therefore, taking α = n − 1 / 2 \alpha=n^{-1/2} α = n − 1/2 gives R ^ n ( F U C B ) ≤ C 4 n − 1 / 2 \widehat{\mathcal{R}}_n(\mathcal{F}_{\mathrm{UCB}})\leq C_4n^{-1/2} R n ( F UCB ) ≤ C 4 n − 1/2 .
Case 2: m = 2 m=2 m = 2 . In this case, we have that
∫ α 1 ε − 1 { log ( C 1 / ε ) } 1 / 2 d ε ≤ C 5 { log ( C 1 / α ) } 3 / 2 . \int_{\alpha}^{1}\varepsilon^{-1}\{\log(C_1/\varepsilon)\}^{1/2}\,\mathrm{d}\varepsilon\leq C_5\{\log(C_1/\alpha)\}^{3/2}. ∫ α 1 ε − 1 { log ( C 1 / ε ) } 1/2 d ε ≤ C 5 { log ( C 1 / α ) } 3/2 .
Taking α = n − 1 / 2 \alpha=n^{-1/2} α = n − 1/2 gives R ^ n ( F U C B ) ≤ C 6 n − 1 / 2 { log ( e n ) } 3 / 2 \widehat{\mathcal{R}}_n(\mathcal{F}_{\mathrm{UCB}})\leq C_6n^{-1/2}\{\log(en)\}^{3/2} R n ( F UCB ) ≤ C 6 n − 1/2 { log ( e n ) } 3/2 .
Case 3: m ≥ 3 m\geq3 m ≥ 3 . In this case, we have that
∫ α 1 ε − m / 2 { log ( C 1 / ε ) } 1 / 2 d ε ≤ C 7 α 1 − m / 2 { log ( 1 / α ) } 1 / 2 . \int_{\alpha}^{1}\varepsilon^{-m/2}\{\log(C_1/\varepsilon)\}^{1/2}\,\mathrm{d}\varepsilon\leq C_7\alpha^{1-m/2}\{\log(1/\alpha)\}^{1/2}. ∫ α 1 ε − m /2 { log ( C 1 / ε ) } 1/2 d ε ≤ C 7 α 1 − m /2 { log ( 1/ α ) } 1/2 .
Taking α = n − 1 / m { log ( e n ) } 1 / m \alpha=n^{-1/m}\{\log(en)\}^{1/m} α = n − 1/ m { log ( e n ) } 1/ m gives R ^ n ( F U C B ) ≤ C 8 { log ( e n ) / n } 1 / m \widehat{\mathcal{R}}_n(\mathcal{F}_{\mathrm{UCB}})\leq C_8\{\log(en)/n\}^{1/m} R n ( F UCB ) ≤ C 8 { log ( e n ) / n } 1/ m .
Thus, for all m ≥ 1 m\geq1 m ≥ 1 , we have that
R ^ n ( F U C B ) ≤ C { n − 1 / 2 , m = 1 , n − 1 / 2 log 3 / 2 ( e n ) , m = 2 , { log ( e n ) / n } 1 / m , m ≥ 3. \widehat{\mathcal{R}}_n(\mathcal{F}_{\mathrm{UCB}})\leq C
\begin{cases}
n^{-1/2}, & m=1,\\
n^{-1/2}\log^{3/2}(en), & m=2,\\
\{\log(en)/n\}^{1/m}, & m\geq3.
\end{cases} R n ( F UCB ) ≤ C ⎩ ⎨ ⎧ n − 1/2 , n − 1/2 log 3/2 ( e n ) , { log ( e n ) / n } 1/ m , m = 1 , m = 2 , m ≥ 3.
This completes the proof.
Lemma 42 (Control of Δ n U C B \Delta_n^{\mathrm{UCB}} Δ n UCB ). Suppose Assumptions 3 and 4 hold and d ≥ 2 d\geq2 d ≥ 2 . Let F U C B \mathcal{F}_{\mathrm{UCB}} F UCB be the function class defined in Lemma 40 . Let X 1 , … , X n X_1,\ldots,X_n X 1 , … , X n be i.i.d. from an arbitrary Borel probability measure P X P_X P X supported on [ 0 , 1 ] d [0,1]^d [ 0 , 1 ] d . Define
Δ n U C B = sup f ∈ F U C B ∣ 1 n ∑ i = 1 n f ( X i ) − E { f ( X ) } ∣ . (67) \Delta_n^{\mathrm{UCB}}=\sup_{f\in\mathcal{F}_{\mathrm{UCB}}}\left|\frac{1}{n}\sum_{i=1}^{n}f(X_i)-\mathbb{E}\{f(X)\}\right|. \tag*{(67)} Δ n UCB = f ∈ F UCB sup n 1 i = 1 ∑ n f ( X i ) − E { f ( X )} . (67) Then there exists a constant C < ∞ C<\infty C < ∞ , depending only on d d d , ν \nu ν , L L L , C β C_\beta C β and the constants in Lemma 14 , and independent of P X P_X P X , such that, for every n ≥ 2 n\geq2 n ≥ 2 and every η ∈ ( 0 , 1 ) \eta\in(0,1) η ∈ ( 0 , 1 ) , with probability at least 1 − η 1-\eta 1 − η ,
Δ n U C B ≤ C ρ n , d − 1 ( η ) , \Delta_n^{\mathrm{UCB}}\leq C\rho_{n,d-1}(\eta), Δ n UCB ≤ C ρ n , d − 1 ( η ) , where ρ n , m \rho_{n,m} ρ n , m is defined in Lemma 23.
Proof. Write m = d − 1 m=d-1 m = d − 1 . Since every f ∈ F U C B f\in\mathcal{F}_{\mathrm{UCB}} f ∈ F UCB is a convex combination of the two linear reward functions at each context, we have
∥ f ∥ ∞ ≤ d + 1 ( C β + 1 ) = : B . \lVert f\rVert_{\infty}\leq\sqrt{d+1}(C_\beta+1)=:B. ∥ f ∥ ∞ ≤ d + 1 ( C β + 1 ) =: B . Note that the finite brackets of pointwise width ε \varepsilon ε constructed in Lemma 40 imply that F U C B \mathcal{F}_{\mathrm{UCB}} F UCB is separable in the sup-norm. Consequently, the empirical and Rademacher suprema are measurable. By the standard symmetrisation inequality,
E X Δ n U C B ≤ 2 E X R ^ n ( F U C B ) = 2 R n ( F U C B ) . \mathbb{E}_{X}\Delta_n^{\mathrm{UCB}}\leq2\mathbb{E}_{X}\widehat{\mathcal{R}}_n(\mathcal{F}_{\mathrm{UCB}})=2\mathcal{R}_n(\mathcal{F}_{\mathrm{UCB}}). E X Δ n UCB ≤ 2 E X R n ( F UCB ) = 2 R n ( F UCB ) .
Replacing one observation changes Δ n U C B \Delta_n^{\mathrm{UCB}} Δ n UCB by at most 2 B / n 2B/n 2 B / n . The bounded-differences inequality therefore gives, with probability at least 1 − η 1-\eta 1 − η ,
Δ n U C B ≤ E X Δ n U C B + B 2 log ( 1 / η ) n . \Delta_n^{\mathrm{UCB}} \le\mathbb{E}_X\Delta_n^{\mathrm{UCB}} + B\sqrt{\frac{2\log(1/\eta)}{n}}. Δ n UCB ≤ E X Δ n UCB + B n 2 log ( 1/ η ) .
Combining this with symmetrisation gives the usual uniform convergence bound based on Rademacher complexity (e.g. [e.g. 51 ] ):
Δ n U C B ≤ 2 R n ( F U C B ) + B 2 log ( 1 / η ) n , \Delta_n^{\mathrm{UCB}} \le2\mathfrak{R}_n(\mathcal{F}_{\mathrm{UCB}}) + B\sqrt{\frac{2\log(1/\eta)}{n}}, Δ n UCB ≤ 2 R n ( F UCB ) + B n 2 log ( 1/ η ) ,
where R n ( F U C B ) = E X { R ^ n ( F U C B ) } \mathfrak{R}_n(\mathcal{F}_{\mathrm{UCB}})=\mathbb{E}_X\{\widehat{\mathfrak{R}}_n(\mathcal{F}_{\mathrm{UCB}})\} R n ( F UCB ) = E X { R n ( F UCB )} . Note that the bound in Lemma 41 holds for every deterministic sample, therefore by the tower property, it also holds after taking the expectation over X 1 , … , X n X_1,\ldots,X_n X 1 , … , X n . Combining the two bounds gives
Δ n U C B ≤ C { n − 1 / 2 + log ( 1 / η ) / n , m = 1 , n − 1 / 2 log 3 / 2 ( e n ) + log ( 1 / η ) / n , m = 2 , { log ( e n ) / n } 1 / m + log ( 1 / η ) / n , m ≥ 3. \Delta_n^{\mathrm{UCB}} \le C
\begin{cases}
n^{-1/2}+\sqrt{\log(1/\eta)/n}, & m=1,\\
n^{-1/2}\log^{3/2}(en)+\sqrt{\log(1/\eta)/n}, & m=2,\\
\{\log(en)/n\}^{1/m}+\sqrt{\log(1/\eta)/n}, & m\ge3.
\end{cases} Δ n UCB ≤ C ⎩ ⎨ ⎧ n − 1/2 + log ( 1/ η ) / n , n − 1/2 log 3/2 ( e n ) + log ( 1/ η ) / n , { log ( e n ) / n } 1/ m + log ( 1/ η ) / n , m = 1 , m = 2 , m ≥ 3.
Since log ( e n / η ) ≥ 1 \log(en/\eta)\ge1 log ( e n / η ) ≥ 1 , each line is bounded by C ρ n , m ( η ) C\rho_{n,m}(\eta) C ρ n , m ( η ) for n ≥ 2 n\ge2 n ≥ 2 and η ∈ ( 0 , 1 ) \eta\in(0,1) η ∈ ( 0 , 1 ) . This completes the proof.
Self-normalised confidence
Lemma 43 (Self-normalised concentration). Suppose Assumptions 2 and 3 hold. For any t ∈ [ T ] t\in[T] t ∈ [ T ] and k ∈ { 1 , 2 } k\in\{1,2\} k ∈ { 1 , 2 } , define
S t , k = ∑ s = 1 t − 1 X ~ s ξ s , k 1 { A s = k } , V t , k = λ I d + 1 + ∑ s = 1 t − 1 X ~ s X ~ s ⊤ 1 { A s = k } . S_{t,k}=\sum_{s=1}^{t-1}\widetilde{X}_s\xi_{s,k}\mathbf{1}\{A_s=k\}, \qquad V_{t,k}=\lambda I_{d+1}+\sum_{s=1}^{t-1}\widetilde{X}_s\widetilde{X}_s^\top\mathbf{1}\{A_s=k\}. S t , k = s = 1 ∑ t − 1 X s ξ s , k 1 { A s = k } , V t , k = λ I d + 1 + s = 1 ∑ t − 1 X s X s ⊤ 1 { A s = k } . Then, it holds that
P [ ∥ S t , k ∥ V t , k − 1 2 ≤ 2 σ 2 log { 2 det ( V t , k ) 1 / 2 det ( λ I d + 1 ) − 1 / 2 η } , for all t ∈ [ T ] , k ∈ { 1 , 2 } ] ≥ 1 − η . \mathbb{P}\left[\left\|S_{t,k}\right\|_{V_{t,k}^{-1}}^2\le2\sigma^2\log\left\{\frac{2\det(V_{t,k})^{1/2}\det(\lambda I_{d+1})^{-1/2}}{\eta}\right\},\ \text{for all }t\in[T],\ k\in\{1,2\}\right]\ge1-\eta. P [ ∥ S t , k ∥ V t , k − 1 2 ≤ 2 σ 2 log { η 2 det ( V t , k ) 1/2 det ( λ I d + 1 ) − 1/2 } , for all t ∈ [ T ] , k ∈ { 1 , 2 } ] ≥ 1 − η .
Proof. The proof is constructed based on the proof of Theorem 1 in Abbasi-Yadkori et al. [1 ] . For any t ∈ [ T ] t\in[T] t ∈ [ T ] , denote the filtration F t = σ ( { X s , A s , Y s } s ∈ [ t ] ) \mathcal{F}_t=\sigma(\{X_s,A_s,Y_s\}_{s\in[t]}) F t = σ ({ X s , A s , Y s } s ∈ [ t ] ) and G t = σ ( F t − 1 , X t , A t ) \mathcal{G}_t=\sigma(\mathcal{F}_{t-1},X_t,A_t) G t = σ ( F t − 1 , X t , A t ) .
Construction of Supermartingale. For any v ∈ R d + 1 v\in\mathbb{R}^{d+1} v ∈ R d + 1 and t ∈ [ T ] ∖ { 1 } t\in[T]\setminus\{1\} t ∈ [ T ] ∖ { 1 } , define
M t , k ( v ) = exp { σ − 1 ⟨ v , S t , k ⟩ − 1 2 v ⊤ ( ∑ s = 1 t − 1 1 { A s = k } X ~ s X ~ s ⊤ ) v } , M_{t,k}(v)=\exp\left\{\sigma^{-1}\langle v,S_{t,k}\rangle-\frac{1}{2}v^\top\left(\sum_{s=1}^{t-1}\mathbf{1}\{A_s=k\}\widetilde{X}_s\widetilde{X}_s^\top\right)v\right\}, M t , k ( v ) = exp { σ − 1 ⟨ v , S t , k ⟩ − 2 1 v ⊤ ( s = 1 ∑ t − 1 1 { A s = k } X s X s ⊤ ) v } ,
and set M 1 , k ( v ) = 1 M_{1,k}(v)=1 M 1 , k ( v ) = 1 . Since
S t + 1 , k = ∑ s = 1 t X ~ s ξ s , k 1 { A s = k } = S t , k + X ~ t ξ t , k 1 { A t = k } , S_{t+1,k}=\sum_{s=1}^{t}\widetilde{X}_s\xi_{s,k}\mathbf{1}\{A_s=k\}=S_{t,k}+\widetilde{X}_t\xi_{t,k}\mathbf{1}\{A_t=k\}, S t + 1 , k = s = 1 ∑ t X s ξ s , k 1 { A s = k } = S t , k + X t ξ t , k 1 { A t = k } ,
and
∑ s = 1 t 1 { A s = k } X ~ s X ~ s ⊤ = ∑ s = 1 t − 1 1 { A s = k } X ~ s X ~ s ⊤ + 1 { A t = k } X ~ t X ~ t ⊤ , \sum_{s=1}^{t}\mathbf{1}\{A_s=k\}\widetilde{X}_s\widetilde{X}_s^\top=\sum_{s=1}^{t-1}\mathbf{1}\{A_s=k\}\widetilde{X}_s\widetilde{X}_s^\top+\mathbf{1}\{A_t=k\}\widetilde{X}_t\widetilde{X}_t^\top, s = 1 ∑ t 1 { A s = k } X s X s ⊤ = s = 1 ∑ t − 1 1 { A s = k } X s X s ⊤ + 1 { A t = k } X t X t ⊤ ,
we have
M t + 1 , k ( v ) = exp { σ − 1 ⟨ v , S t + 1 , k ⟩ − 1 2 v ⊤ ( ∑ s = 1 t 1 { A s = k } X ~ s X ~ s ⊤ ) v } = M t , k ( v ) exp { σ − 1 ⟨ v , X ~ t ξ t , k 1 { A t = k } ⟩ − 1 { A t = k } 2 v ⊤ X ~ t X ~ t ⊤ v } . \begin{aligned}
M_{t+1,k}(v)=\exp\left\{\sigma^{-1}\langle v,S_{t+1,k}\rangle-\frac{1}{2}v^{\top}\left(\sum_{s=1}^{t}\mathbb{1}\{A_s=k\}\widetilde{X}_s\widetilde{X}_s^{\top}\right)v\right\} \\
&=M_{t,k}(v)\exp\left\{\sigma^{-1}\langle v,\widetilde{X}_t\xi_{t,k}\mathbb{1}\{A_t=k\}\rangle-\frac{\mathbb{1}\{A_t=k\}}{2}v^{\top}\widetilde{X}_t\widetilde{X}_t^{\top}v\right\}.
\end{aligned} M t + 1 , k ( v ) = exp { σ − 1 ⟨ v , S t + 1 , k ⟩ − 2 1 v ⊤ ( s = 1 ∑ t 1 { A s = k } X s X s ⊤ ) v } = M t , k ( v ) exp { σ − 1 ⟨ v , X t ξ t , k 1 { A t = k }⟩ − 2 1 { A t = k } v ⊤ X t X t ⊤ v } .
Since for t ∈ [ T ] t\in[T] t ∈ [ T ] , both M t , k ( v ) M_{t,k}(v) M t , k ( v ) and X ~ t \widetilde{X}_t X t are G t \mathcal{G}_t G t -measurable, the sub-Gaussian condition gives
E [ M t + 1 , k ( v ) ∣ G t ] ≤ M t , k ( v ) exp { − 1 { A t = k } 2 v ⊤ X ~ t X ~ t ⊤ v } exp { 1 { A t = k } 2 v ⊤ X ~ t X ~ t ⊤ v } = M t , k ( v ) . \begin{aligned}
\mathbb{E}[M_{t+1,k}(v)\mid\mathcal{G}_t]\leq M_{t,k}(v)\exp\left\{-\frac{\mathbb{1}\{A_t=k\}}{2}v^{\top}\widetilde{X}_t\widetilde{X}_t^{\top}v\right\}\exp\left\{\frac{\mathbb{1}\{A_t=k\}}{2}v^{\top}\widetilde{X}_t\widetilde{X}_t^{\top}v\right\} \\
&=M_{t,k}(v).
\end{aligned} E [ M t + 1 , k ( v ) ∣ G t ] ≤ M t , k ( v ) exp { − 2 1 { A t = k } v ⊤ X t X t ⊤ v } exp { 2 1 { A t = k } v ⊤ X t X t ⊤ v } = M t , k ( v ) .
Hence { M t , k } t ∈ [ T ] \{M_{t,k}\}_{t\in[T]} { M t , k } t ∈ [ T ] is a sequence of non-negative supermartingales.
Method of mixtures. Let V ∼ N ( 0 , λ − 1 I d + 1 ) V\sim\mathcal{N}(0,\lambda^{-1}I_{d+1}) V ∼ N ( 0 , λ − 1 I d + 1 ) be a Gaussian random variable which is independent of all other random variables and define the mixture process
M t , k V = ∫ R d + 1 M t , k ( v ) d N ( 0 , λ − 1 I d + 1 ) ( v ) . M_{t,k}^{V}=\int_{\mathbb{R}^{d+1}}M_{t,k}(v)\,\mathrm{d}\mathcal{N}(0,\lambda^{-1}I_{d+1})(v). M t , k V = ∫ R d + 1 M t , k ( v ) d N ( 0 , λ − 1 I d + 1 ) ( v ) .
By Tonelli’s theorem, we have that for any t ∈ [ T ] t\in[T] t ∈ [ T ]
E [ M t + 1 , k V ∣ G t ] = ∫ R d + 1 E { M t + 1 , k ( v ) ∣ G t } d N ( 0 , λ − 1 I d + 1 ) ( v ) ≤ ∫ R d + 1 M t , k ( v ) d N ( 0 , λ − 1 I d + 1 ) ( v ) = M t , k V . \begin{aligned}
\mathbb{E}[M_{t+1,k}^{V}\mid\mathcal{G}_t]=\int_{\mathbb{R}^{d+1}}\mathbb{E}\{M_{t+1,k}(v)\mid\mathcal{G}_t\}\,\mathrm{d}\mathcal{N}(0,\lambda^{-1}I_{d+1})(v) \\
&\leq\int_{\mathbb{R}^{d+1}}M_{t,k}(v)\,\mathrm{d}\mathcal{N}(0,\lambda^{-1}I_{d+1})(v) \\
&=M_{t,k}^{V}.
\end{aligned} E [ M t + 1 , k V ∣ G t ] = ∫ R d + 1 E { M t + 1 , k ( v ) ∣ G t } d N ( 0 , λ − 1 I d + 1 ) ( v ) ≤ ∫ R d + 1 M t , k ( v ) d N ( 0 , λ − 1 I d + 1 ) ( v ) = M t , k V .
Hence, we can conclude that { M t , k V } t ∈ [ T ] \{M_{t,k}^{V}\}_{t\in[T]} { M t , k V } t ∈ [ T ] is still a sequence of non-negative supermartingales with E [ M 1 , k V ] = 1 \mathbb{E}[M_{1,k}^{V}]=1 E [ M 1 , k V ] = 1 . Next, we compute the integral explicitly. For notational simplicity, we let
Q t , k = ∑ s = 1 t − 1 1 { A s = k } X ~ s X ~ s ⊤ . Q_{t,k}=\sum_{s=1}^{t-1}\mathbb{1}\{A_s=k\}\widetilde{X}_s\widetilde{X}_s^{\top}. Q t , k = s = 1 ∑ t − 1 1 { A s = k } X s X s ⊤ .
Then
M t , k V = 1 ( 2 π λ − 1 ) ( d + 1 ) / 2 ∫ R d + 1 exp { σ − 1 ⟨ v , S t , k ⟩ − v ⊤ Q t , k v 2 } exp ( − λ ∥ v ∥ 2 2 2 ) d v = 1 ( 2 π λ − 1 ) ( d + 1 ) / 2 ∫ R d + 1 exp { σ − 1 ⟨ v , S t , k ⟩ − 1 2 v ⊤ ( Q t , k + λ I d + 1 ) v } d v . \begin{aligned}
M_{t,k}^{V}=\frac{1}{(2\pi\lambda^{-1})^{(d+1)/2}}\int_{\mathbb{R}^{d+1}}\exp\left\{\sigma^{-1}\langle v,S_{t,k}\rangle-\frac{v^{\top}Q_{t,k}v}{2}\right\}\exp\left(-\frac{\lambda\lVert v\rVert_2^2}{2}\right)\,\mathrm{d}v \\
&=\frac{1}{(2\pi\lambda^{-1})^{(d+1)/2}}\int_{\mathbb{R}^{d+1}}\exp\left\{\sigma^{-1}\langle v,S_{t,k}\rangle-\frac{1}{2}v^{\top}(Q_{t,k}+\lambda I_{d+1})v\right\}\,\mathrm{d}v.
\end{aligned} M t , k V = ( 2 π λ − 1 ) ( d + 1 ) /2 1 ∫ R d + 1 exp { σ − 1 ⟨ v , S t , k ⟩ − 2 v ⊤ Q t , k v } exp ( − 2 λ ∥ v ∥ 2 2 ) d v = ( 2 π λ − 1 ) ( d + 1 ) /2 1 ∫ R d + 1 exp { σ − 1 ⟨ v , S t , k ⟩ − 2 1 v ⊤ ( Q t , k + λ I d + 1 ) v } d v .
Note that
σ − 1 v ⊤ S t , k − v ⊤ ( Q t , k + λ I d + 1 ) v 2 = − 1 2 { v − σ − 1 ( Q t , k + λ I d + 1 ) − 1 S t , k } ⊤ ( Q t , k + λ I d + 1 ) { v − σ − 1 ( Q t , k + λ I d + 1 ) − 1 S t , k } \begin{aligned}
\sigma^{-1}v^{\top}S_{t,k}-\frac{v^{\top}(Q_{t,k}+\lambda I_{d+1})v}{2} \\
&=-\frac{1}{2}\left\{v-\sigma^{-1}(Q_{t,k}+\lambda I_{d+1})^{-1}S_{t,k}\right\}^{\top}(Q_{t,k}+\lambda I_{d+1})\left\{v-\sigma^{-1}(Q_{t,k}+\lambda I_{d+1})^{-1}S_{t,k}\right\}
\end{aligned} σ − 1 v ⊤ S t , k − 2 v ⊤ ( Q t , k + λ I d + 1 ) v = − 2 1 { v − σ − 1 ( Q t , k + λ I d + 1 ) − 1 S t , k } ⊤ ( Q t , k + λ I d + 1 ) { v − σ − 1 ( Q t , k + λ I d + 1 ) − 1 S t , k }
+ 1 2 σ 2 S t , k ⊤ ( Q t , k + λ I d + 1 ) − 1 S t , k . \qquad+\frac{1}{2\sigma^2}S_{t,k}^{\top}(Q_{t,k}+\lambda I_{d+1})^{-1}S_{t,k}. + 2 σ 2 1 S t , k ⊤ ( Q t , k + λ I d + 1 ) − 1 S t , k .
Then, the integral becomes
M t , k V = 1 ( 2 π λ − 1 ) ( d + 1 ) / 2 exp { 1 2 σ 2 S t , k T ( Q t , k + λ I d + 1 ) − 1 S t , k } ⋅ ∫ R d + 1 exp [ − 1 2 { v − σ − 1 ( Q t , k + λ I d + 1 ) − 1 S t , k } T ( Q t , k + λ I d + 1 ) { v − σ − 1 ( Q t , k + λ I d + 1 ) − 1 S t , k } ] d v = 1 ( 2 π λ − 1 ) ( d + 1 ) / 2 exp { 1 2 σ 2 S t , k T ( Q t , k + λ I d + 1 ) − 1 S t , k } ( 2 π ) ( d + 1 ) / 2 det ( Q t , k + λ I d + 1 ) − 1 / 2 = det ( λ I d + 1 ) det ( V t , k ) exp { 1 2 σ 2 ∥ S t , k ∥ V t , k − 1 2 } . \begin{aligned}
M_{t,k}^{V}
&=\frac{1}{(2\pi\lambda^{-1})^{(d+1)/2}}\exp\left\{\frac{1}{2\sigma^2}S_{t,k}^{\mathsf T}(Q_{t,k}+\lambda I_{d+1})^{-1}S_{t,k}\right\}\\
&\quad\cdot\int_{\mathbb{R}^{d+1}}\exp\left[-\frac{1}{2}\left\{v-\sigma^{-1}(Q_{t,k}+\lambda I_{d+1})^{-1}S_{t,k}\right\}^{\mathsf T}\right.\\
&\qquad\qquad\left.(Q_{t,k}+\lambda I_{d+1})\left\{v-\sigma^{-1}(Q_{t,k}+\lambda I_{d+1})^{-1}S_{t,k}\right\}\right]\mathrm{d}v\\
&=\frac{1}{(2\pi\lambda^{-1})^{(d+1)/2}}\exp\left\{\frac{1}{2\sigma^2}S_{t,k}^{\mathsf T}(Q_{t,k}+\lambda I_{d+1})^{-1}S_{t,k}\right\}(2\pi)^{(d+1)/2}\det(Q_{t,k}+\lambda I_{d+1})^{-1/2}\\
&=\sqrt{\frac{\det(\lambda I_{d+1})}{\det(V_{t,k})}}\exp\left\{\frac{1}{2\sigma^2}\left\|S_{t,k}\right\|_{V_{t,k}^{-1}}^2\right\}.
\end{aligned} M t , k V = ( 2 π λ − 1 ) ( d + 1 ) /2 1 exp { 2 σ 2 1 S t , k T ( Q t , k + λ I d + 1 ) − 1 S t , k } ⋅ ∫ R d + 1 exp [ − 2 1 { v − σ − 1 ( Q t , k + λ I d + 1 ) − 1 S t , k } T ( Q t , k + λ I d + 1 ) { v − σ − 1 ( Q t , k + λ I d + 1 ) − 1 S t , k } ] d v = ( 2 π λ − 1 ) ( d + 1 ) /2 1 exp { 2 σ 2 1 S t , k T ( Q t , k + λ I d + 1 ) − 1 S t , k } ( 2 π ) ( d + 1 ) /2 det ( Q t , k + λ I d + 1 ) − 1/2 = det ( V t , k ) det ( λ I d + 1 ) exp { 2 σ 2 1 ∥ S t , k ∥ V t , k − 1 2 } .
Tail bound. By Ville’s inequality, for any η / 2 > 0 \eta/2>0 η /2 > 0 , we have with probability at least 1 − η / 2 1-\eta/2 1 − η /2 that
det ( λ I d + 1 ) det ( V t , k ) exp { 1 2 σ 2 ∥ S t , k ∥ V t , k − 1 2 } ≤ 2 η , for all t ∈ [ T ] . \sqrt{\frac{\det(\lambda I_{d+1})}{\det(V_{t,k})}}\exp\left\{\frac{1}{2\sigma^2}\left\|S_{t,k}\right\|_{V_{t,k}^{-1}}^2\right\}\leq\frac{2}{\eta},\quad\text{for all }t\in[T]. det ( V t , k ) det ( λ I d + 1 ) exp { 2 σ 2 1 ∥ S t , k ∥ V t , k − 1 2 } ≤ η 2 , for all t ∈ [ T ] .
Thus, by another union bound over k ∈ { 1 , 2 } k\in\{1,2\} k ∈ { 1 , 2 } , we have that
P [ ∥ S t , k ∥ V t , k − 1 2 ≤ 2 σ 2 log { 2 det ( V t , k ) 1 / 2 det ( λ I d + 1 ) − 1 / 2 η } , for all t ∈ [ T ] , k ∈ { 1 , 2 } ] ≥ 1 − η . \mathbb{P}\left[\left\|S_{t,k}\right\|_{V_{t,k}^{-1}}^2\leq2\sigma^2\log\left\{\frac{2\det(V_{t,k})^{1/2}\det(\lambda I_{d+1})^{-1/2}}{\eta}\right\},\ \text{for all }t\in[T],\ k\in\{1,2\}\right]\geq1-\eta. P [ ∥ S t , k ∥ V t , k − 1 2 ≤ 2 σ 2 log { η 2 det ( V t , k ) 1/2 det ( λ I d + 1 ) − 1/2 } , for all t ∈ [ T ] , k ∈ { 1 , 2 } ] ≥ 1 − η .
Corollary 44. Under the conditions of Lemma 43, with probability at least $1-\eta$, it holds for all t ∈ [ T ] t\in[T] t ∈ [ T ] and $k\in{1,2} that
∥ S t , k ∥ V t , k − 1 2 ≤ 2 σ 2 { log ( 2 η ) + d + 1 2 log ( 1 + T λ ) } . \left\|S_{t,k}\right\|_{V_{t,k}^{-1}}^2\leq2\sigma^2\left\{\log\left(\frac{2}{\eta}\right)+\frac{d+1}{2}\log\left(1+\frac{T}{\lambda}\right)\right\}. ∥ S t , k ∥ V t , k − 1 2 ≤ 2 σ 2 { log ( η 2 ) + 2 d + 1 log ( 1 + λ T ) } .
Proof. By Lemma 43, with probability at least 1 − η 1-\eta 1 − η , it holds for all t ∈ [ T ] t\in[T] t ∈ [ T ] and k ∈ { 1 , 2 } k\in\{1,2\} k ∈ { 1 , 2 } that
∥ S t , k ∥ V t , k − 1 2 ≤ 2 σ 2 log { 2 det ( V t , k ) 1 / 2 det ( λ I d + 1 ) − 1 / 2 η } = 2 σ 2 { log ( 2 η ) + 1 2 log ( det ( V t , k ) det ( λ I d + 1 ) ) } . \begin{aligned}
\left\|S_{t,k}\right\|_{V_{t,k}^{-1}}^2\leq2\sigma^2\log\left\{\frac{2\det(V_{t,k})^{1/2}\det(\lambda I_{d+1})^{-1/2}}{\eta}\right\} \\
&=2\sigma^2\left\{\log\left(\frac{2}{\eta}\right)+\frac{1}{2}\log\left(\frac{\det(V_{t,k})}{\det(\lambda I_{d+1})}\right)\right\}.
\end{aligned} ∥ S t , k ∥ V t , k − 1 2 ≤ 2 σ 2 log { η 2 det ( V t , k ) 1/2 det ( λ I d + 1 ) − 1/2 } = 2 σ 2 { log ( η 2 ) + 2 1 log ( det ( λ I d + 1 ) det ( V t , k ) ) } .
Since ∥ X ~ t ∥ 2 ≤ d + 1 \left\|\widetilde{X}_t\right\|_2\leq\sqrt{d+1} X t 2 ≤ d + 1 , Lemma 10 in Abbasi-Yadkori et al. [1 ] gives
log ( det ( V t , k ) det ( λ I d + 1 ) ) ≤ ( d + 1 ) log ( 1 + T λ ) . \log\left(\frac{\det(V_{t,k})}{\det(\lambda I_{d+1})}\right)\leq(d+1)\log\left(1+\frac{T}{\lambda}\right). log ( det ( λ I d + 1 ) det ( V t , k ) ) ≤ ( d + 1 ) log ( 1 + λ T ) .
The result follows.
Lemma 45 (UCB confidence bound). Suppose Assumptions 2 2 2 and 3 3 3 hold. For each t ∈ [ T ] t\in[T] t ∈ [ T ] and k ∈ { 1 , 2 } k\in\{1,2\} k ∈ { 1 , 2 } , define
V t , k = λ I d + 1 + ∑ s = 1 t − 1 X ~ s X ~ s T 1 { A s = k } , β ^ t , k = V t , k − 1 ∑ s = 1 t − 1 X ~ s Y s 1 { A s = k } . V_{t,k}=\lambda I_{d+1}+\sum_{s=1}^{t-1}\widetilde{X}_s\widetilde{X}_s^{\mathsf T}\mathbb{1}\{A_s=k\},\qquad\widehat{\beta}_{t,k}=V_{t,k}^{-1}\sum_{s=1}^{t-1}\widetilde{X}_sY_s\mathbb{1}\{A_s=k\}. V t , k = λ I d + 1 + s = 1 ∑ t − 1 X s X s T 1 { A s = k } , β t , k = V t , k − 1 s = 1 ∑ t − 1 X s Y s 1 { A s = k } .
Then, with probability at least 1 − η 1-\eta 1 − η that, for all x ~ ∈ [ 0 , 1 ] d \widetilde{x}\in[0,1]^d x ∈ [ 0 , 1 ] d , k ∈ { 1 , 2 } k\in\{1,2\} k ∈ { 1 , 2 } and t ∈ [ T ] t\in[T] t ∈ [ T ] ,
∣ x ~ ⊤ ( β ^ t , k − β k ∗ ) ∣ ≤ [ 2 σ 2 { log ( 2 η ) + d + 1 2 log ( 1 + T λ ) } + C β λ ] x ~ ⊤ V t , k − 1 x ~ . \left|\widetilde{x}^{\top}(\widehat{\beta}_{t,k}-\beta_k^*)\right|
\leq
\left[
\sqrt{2\sigma^2\left\{\log\left(\frac{2}{\eta}\right)+\frac{d+1}{2}\log\left(1+\frac{T}{\lambda}\right)\right\}}+C_{\beta}\sqrt{\lambda}
\right]\sqrt{\widetilde{x}^{\top}V_{t,k}^{-1}\widetilde{x}}. x ⊤ ( β t , k − β k ∗ ) ≤ [ 2 σ 2 { log ( η 2 ) + 2 d + 1 log ( 1 + λ T ) } + C β λ ] x ⊤ V t , k − 1 x .
Consequently, taking
α T = C [ σ 2 { log ( 1 η ) + ( d + 1 ) log ( 1 + T λ ) } + C β λ ] , \alpha_T=C\left[
\sqrt{\sigma^2\left\{\log\left(\frac{1}{\eta}\right)+(d+1)\log\left(1+\frac{T}{\lambda}\right)\right\}}+C_{\beta}\sqrt{\lambda}
\right], α T = C [ σ 2 { log ( η 1 ) + ( d + 1 ) log ( 1 + λ T ) } + C β λ ] ,
it holds with probability at least 1 − η 1-\eta 1 − η that, for all x ~ ∈ [ 0 , 1 ] d \widetilde{x}\in[0,1]^d x ∈ [ 0 , 1 ] d , k ∈ { 1 , 2 } k\in\{1,2\} k ∈ { 1 , 2 } and t ∈ [ T ] t\in[T] t ∈ [ T ] ,
x ~ ⊤ β k ∗ ≤ x ~ ⊤ β ^ t , k + α T x ~ ⊤ V t , k − 1 x ~ . \widetilde{x}^{\top}\beta_k^*
\leq\widetilde{x}^{\top}\widehat{\beta}_{t,k}+\alpha_T\sqrt{\widetilde{x}^{\top}V_{t,k}^{-1}\widetilde{x}}. x ⊤ β k ∗ ≤ x ⊤ β t , k + α T x ⊤ V t , k − 1 x .
Proof. Define the event for self-normalised norms
E s n = { ∥ S t , k ∥ V t , k − 1 2 ≤ 2 σ 2 { log ( 2 η ) + d + 1 2 log ( 1 + T λ ) } , for all t ∈ [ T ] and k ∈ { 1 , 2 } } . \mathcal{E}_{\mathrm{sn}}=\left\{\|S_{t,k}\|_{V_{t,k}^{-1}}^2\leq2\sigma^2\left\{\log\left(\frac{2}{\eta}\right)+\frac{d+1}{2}\log\left(1+\frac{T}{\lambda}\right)\right\},\ \text{for all }t\in[T]\text{ and }k\in\{1,2\}\right\}. E sn = { ∥ S t , k ∥ V t , k − 1 2 ≤ 2 σ 2 { log ( η 2 ) + 2 d + 1 log ( 1 + λ T ) } , for all t ∈ [ T ] and k ∈ { 1 , 2 } } .
By Corollary 44 , it holds that P ( E s n ) ≥ 1 − η \mathbb{P}(\mathcal{E}_{\mathrm{sn}})\geq1-\eta P ( E sn ) ≥ 1 − η . In the rest of the proof, we condition on E s n \mathcal{E}_{\mathrm{sn}} E sn happening. For any x ~ ∈ [ 0 , 1 ] d \widetilde{x}\in[0,1]^d x ∈ [ 0 , 1 ] d , by the Cauchy–Schwarz inequality in the V t , k V_{t,k} V t , k -norm, we have that
∣ x ~ ⊤ ( β ^ t , k − β k ∗ ) ∣ ≤ ∥ β ^ t , k − β k ∗ ∥ V t , k ∥ x ~ ∥ V t , k − 1 = ∥ β ^ t , k − β k ∗ ∥ V t , k x ~ ⊤ V t , k − 1 x ~ . (68) \left|\widetilde{x}^{\top}(\widehat{\beta}_{t,k}-\beta_k^*)\right|
\leq\|\widehat{\beta}_{t,k}-\beta_k^*\|_{V_{t,k}}\|\widetilde{x}\|_{V_{t,k}^{-1}}
= \|\widehat{\beta}_{t,k}-\beta_k^*\|_{V_{t,k}}\sqrt{\widetilde{x}^{\top}V_{t,k}^{-1}\widetilde{x}}. \tag*{(68)} x ⊤ ( β t , k − β k ∗ ) ≤ ∥ β t , k − β k ∗ ∥ V t , k ∥ x ∥ V t , k − 1 = ∥ β t , k − β k ∗ ∥ V t , k x ⊤ V t , k − 1 x . (68)
When A s = k A_s=k A s = k , we have Y s = X ~ s ⊤ β k ∗ + ξ s , k Y_s=\widetilde{X}_s^{\top}\beta_k^*+\xi_{s,k} Y s = X s ⊤ β k ∗ + ξ s , k . Therefore,
β ^ t , k − β k ∗ = V t , k − 1 ∑ s = 1 t − 1 X ~ s Y s 1 { A s = k } − β k ∗ = V t , k − 1 ∑ s = 1 t − 1 X ~ s X ~ s ⊤ β k ∗ 1 { A s = k } + V t , k − 1 S t , k − β k ∗ = V t , k − 1 ( V t , k − λ I d + 1 ) β k ∗ + V t , k − 1 S t , k − β k ∗ = V t , k − 1 S t , k − λ V t , k − 1 β k ∗ . \begin{aligned}
\widehat{\beta}_{t,k}-\beta_k^*
=V_{t,k}^{-1}\sum_{s=1}^{t-1}\widetilde{X}_sY_s\mathbf{1}\{A_s=k\}-\beta_k^* \\
&=V_{t,k}^{-1}\sum_{s=1}^{t-1}\widetilde{X}_s\widetilde{X}_s^{\top}\beta_k^*\mathbf{1}\{A_s=k\}+V_{t,k}^{-1}S_{t,k}-\beta_k^* \\
&=V_{t,k}^{-1}(V_{t,k}-\lambda I_{d+1})\beta_k^*+V_{t,k}^{-1}S_{t,k}-\beta_k^* \\
&=V_{t,k}^{-1}S_{t,k}-\lambda V_{t,k}^{-1}\beta_k^*.
\end{aligned} β t , k − β k ∗ = V t , k − 1 s = 1 ∑ t − 1 X s Y s 1 { A s = k } − β k ∗ = V t , k − 1 s = 1 ∑ t − 1 X s X s ⊤ β k ∗ 1 { A s = k } + V t , k − 1 S t , k − β k ∗ = V t , k − 1 ( V t , k − λ I d + 1 ) β k ∗ + V t , k − 1 S t , k − β k ∗ = V t , k − 1 S t , k − λ V t , k − 1 β k ∗ .
Therefore,
∥ β ^ t , k − β k ∗ ∥ V t , k ≤ ∥ S t , k ∥ V t , k − 1 + λ ∥ β k ∗ ∥ 2 ≤ ∥ S t , k ∥ V t , k − 1 + C β λ . (69) \|\widehat{\beta}_{t,k}-\beta_k^*\|_{V_{t,k}}
\leq\|S_{t,k}\|_{V_{t,k}^{-1}}+\sqrt{\lambda}\|\beta_k^*\|_2
\leq\|S_{t,k}\|_{V_{t,k}^{-1}}+C_{\beta}\sqrt{\lambda}. \tag*{(69)} ∥ β t , k − β k ∗ ∥ V t , k ≤ ∥ S t , k ∥ V t , k − 1 + λ ∥ β k ∗ ∥ 2 ≤ ∥ S t , k ∥ V t , k − 1 + C β λ . (69)
Combining (69) with (68) and by construction of E s n \mathcal{E}_{\mathrm{sn}} E sn , the first claim follows. The second claim follows directly by the choice of α T \alpha_T α T .
Lazy width summation and regret decomposition
Lemma 46. Let t ≥ τ 0 t\geq\tau_0 t ≥ τ 0 and s = τ ( t ) s=\tau(t) s = τ ( t ) be the lazy update time used at round t t t . Then, for k ∈ { 1 , 2 } k\in\{1,2\} k ∈ { 1 , 2 } ,
V s , k − 1 ⪯ 2 V t , k − 1 . V_{s,k}^{-1}\preceq2V_{t,k}^{-1}. V s , k − 1 ⪯ 2 V t , k − 1 .
Consequently, for any x ∈ [ 0 , 1 ] d x\in[0,1]^d x ∈ [ 0 , 1 ] d ,
∥ x ~ ∥ V s , k − 1 ≤ 2 ∥ x ~ ∥ V t , k − 1 . \|\widetilde{x}\|_{V_{s,k}^{-1}}\leq\sqrt{2}\|\widetilde{x}\|_{V_{t,k}^{-1}}. ∥ x ∥ V s , k − 1 ≤ 2 ∥ x ∥ V t , k − 1 .
Proof. At each lazy update time s s s , the algorithm sets ℓ s , k : = log det ( V s , k ) \ell_{s,k} := \log\det(V_{s,k}) ℓ s , k := log det ( V s , k ) . Fix t ≥ s 0 t \ge s_0 t ≥ s 0 and let s = τ ( t ) s = \tau(t) s = τ ( t ) . If t = s t = s t = s , then V t , k = V s , k V_{t,k} = V_{s,k} V t , k = V s , k . If t > s t > s t > s , no update occurs at round t t t , so the determinant trigger, evaluated with ℓ k = log det ( V t , k ) \ell_k = \log\det(V_{t,k}) ℓ k = log det ( V t , k ) before observing X t X_t X t , has not fired. Thus, in both cases,
log det ( V t , k ) − log det ( V s , k ) < log 2. \log\det(V_{t,k}) - \log\det(V_{s,k}) < \log2. log det ( V t , k ) − log det ( V s , k ) < log 2.
Moreover, V t , k ⪰ V s , k V_{t,k} \succeq V_{s,k} V t , k ⪰ V s , k . Let the eigenvalues of V s , k − 1 / 2 V t , k V s , k − 1 / 2 V_{s,k}^{-1/2}V_{t,k}V_{s,k}^{-1/2} V s , k − 1/2 V t , k V s , k − 1/2 be μ 1 , … , μ d + 1 \mu_1,\ldots,\mu_{d+1} μ 1 , … , μ d + 1 . Then μ j ≥ 1 \mu_j \ge1 μ j ≥ 1 for all j j j and ∏ j = 1 d + 1 μ j = det ( V t , k ) / det ( V s , k ) < 2 \prod_{j=1}^{d+1}\mu_j = \det(V_{t,k})/\det(V_{s,k}) < 2 ∏ j = 1 d + 1 μ j = det ( V t , k ) / det ( V s , k ) < 2 . Hence μ j < 2 \mu_j < 2 μ j < 2 for all j j j , which gives V t , k ⪯ 2 V s , k V_{t,k} \preceq2V_{s,k} V t , k ⪯ 2 V s , k . Taking inverses yields V s , k − 1 ⪯ 2 V t , k − 1 V_{s,k}^{-1} \preceq2V_{t,k}^{-1} V s , k − 1 ⪯ 2 V t , k − 1 .
Lemma 47 (Realised lazy width sum). For the sequence generated by Algorithm 3, we have that
∑ t = s 0 T min { 1 , ∥ X ~ t ∥ V τ ( t ) , A t − 1 } ≤ C T ( d + 1 ) log ( 1 + T λ ) , \sum_{t=s_0}^{T}\min\{1,\lVert\widetilde{X}_t\rVert_{V_{\tau(t),A_t}^{-1}}\} \le C\sqrt{T(d+1)\log\left(1+\frac{T}{\lambda}\right)}, t = s 0 ∑ T min { 1 , ∥ X t ∥ V τ ( t ) , A t − 1 } ≤ C T ( d + 1 ) log ( 1 + λ T ) , where C > 0 C > 0 C > 0 is an absolute constant.
Proof. By Lemma 46, we have that
min { 1 , ∥ X ~ t ∥ V τ ( t ) , A t − 1 } ≤ 2 min { 1 , ∥ X ~ t ∥ V t , A t − 1 } . \min\{1,\lVert\widetilde{X}_t\rVert_{V_{\tau(t),A_t}^{-1}}\} \le\sqrt{2}\min\{1,\lVert\widetilde{X}_t\rVert_{V_{t,A_t}^{-1}}\}. min { 1 , ∥ X t ∥ V τ ( t ) , A t − 1 } ≤ 2 min { 1 , ∥ X t ∥ V t , A t − 1 } .
Therefore, by the Cauchy–Schwarz inequality,
∑ t = s 0 T min { 1 , ∥ X ~ t ∥ V τ ( t ) , A t − 1 } ≤ 2 T ∑ t = s 0 T min { 1 , ∥ X ~ t ∥ V t , A t − 1 2 } . \sum_{t=s_0}^{T}\min\{1,\lVert\widetilde{X}_t\rVert_{V_{\tau(t),A_t}^{-1}}\} \le\sqrt{2T\sum_{t=s_0}^{T}\min\{1,\lVert\widetilde{X}_t\rVert_{V_{t,A_t}^{-1}}^2\}}. t = s 0 ∑ T min { 1 , ∥ X t ∥ V τ ( t ) , A t − 1 } ≤ 2 T t = s 0 ∑ T min { 1 , ∥ X t ∥ V t , A t − 1 2 } .
For each arm k k k , Lemma 10 in Abbasi-Yadkori et al. [1 ] gives
∑ t = s 0 T min { 1 , ∥ X ~ t ∥ V t , k − 1 2 } 1 { A t = k } ≤ 2 log ( det ( V T + 1 , k ) det ( λ I d + 1 ) ) . \sum_{t=s_0}^{T}\min\{1,\lVert\widetilde{X}_t\rVert_{V_{t,k}^{-1}}^2\}\mathbb{1}\{A_t=k\} \le2\log\left(\frac{\det(V_{T+1,k})}{\det(\lambda I_{d+1})}\right). t = s 0 ∑ T min { 1 , ∥ X t ∥ V t , k − 1 2 } 1 { A t = k } ≤ 2 log ( det ( λ I d + 1 ) det ( V T + 1 , k ) ) .
Note that, since V 1 , k = λ I d + 1 V_{1,k}=\lambda I_{d+1} V 1 , k = λ I d + 1 and V t + 1 , k = V t , k + 1 { A t = k } X ~ t X ~ t T V_{t+1,k}=V_{t,k}+\mathbb{1}\{A_t=k\}\widetilde{X}_t\widetilde{X}_t^{\mathsf T} V t + 1 , k = V t , k + 1 { A t = k } X t X t T , the matrix determinant lemma (e.g. Theorem 18.1.1 in Harville, [28 ] ) gives
det ( V t + 1 , k ) = det ( V t , k ) { 1 + 1 { A t = k } X ~ t T V t , k − 1 X ~ t } . \det(V_{t+1,k})=\det(V_{t,k})\left\{1+\mathbb{1}\{A_t=k\}\widetilde{X}_t^{\mathsf T}V_{t,k}^{-1}\widetilde{X}_t\right\}. det ( V t + 1 , k ) = det ( V t , k ) { 1 + 1 { A t = k } X t T V t , k − 1 X t } .
Using the fact that min { 1 , a } ≤ 2 log ( 1 + a ) \min\{1,a\}\le2\log(1+a) min { 1 , a } ≤ 2 log ( 1 + a ) for all a ≥ 0 a\ge0 a ≥ 0 , we have
1 { A t = k } min { 1 , X ~ t T V t , k − 1 X ~ t } ≤ 2 log { 1 + 1 { A t = k } X ~ t T V t , k − 1 X ~ t } = 2 log ( det ( V t + 1 , k ) det ( V t , k ) ) . \mathbb{1}\{A_t=k\}\min\{1,\widetilde{X}_t^{\mathsf T}V_{t,k}^{-1}\widetilde{X}_t\}\le2\log\left\{1+\mathbb{1}\{A_t=k\}\widetilde{X}_t^{\mathsf T}V_{t,k}^{-1}\widetilde{X}_t\right\}=2\log\left(\frac{\det(V_{t+1,k})}{\det(V_{t,k})}\right). 1 { A t = k } min { 1 , X t T V t , k − 1 X t } ≤ 2 log { 1 + 1 { A t = k } X t T V t , k − 1 X t } = 2 log ( det ( V t , k ) det ( V t + 1 , k ) ) .
Summing over t = 1 , … , T t=1,\ldots,T t = 1 , … , T gives
∑ t = 1 T 1 { A t = k } min { 1 , X ~ t T V t , k − 1 X ~ t } ≤ 2 ∑ t = 1 T log ( det ( V t + 1 , k ) det ( V t , k ) ) = 2 log ( det ( V T + 1 , k ) det ( λ I d + 1 ) ) . \sum_{t=1}^{T}\mathbb{1}\{A_t=k\}\min\{1,\widetilde{X}_t^{\mathsf T}V_{t,k}^{-1}\widetilde{X}_t\}\le2\sum_{t=1}^{T}\log\left(\frac{\det(V_{t+1,k})}{\det(V_{t,k})}\right)=2\log\left(\frac{\det(V_{T+1,k})}{\det(\lambda I_{d+1})}\right). t = 1 ∑ T 1 { A t = k } min { 1 , X t T V t , k − 1 X t } ≤ 2 t = 1 ∑ T log ( det ( V t , k ) det ( V t + 1 , k ) ) = 2 log ( det ( λ I d + 1 ) det ( V T + 1 , k ) ) .
Now let N k ( T ) = ∑ t = 1 T 1 { A t = k } N_k(T)=\sum_{t=1}^{T}\mathbb{1}\{A_t=k\} N k ( T ) = ∑ t = 1 T 1 { A t = k } . Since ∥ X ~ t ∥ 2 2 ≤ d + 1 \lVert\widetilde{X}_t\rVert_2^2\le d+1 ∥ X t ∥ 2 2 ≤ d + 1 and N k ( T ) ≤ T N_k(T)\le T N k ( T ) ≤ T , we have that
tr ( V T + 1 , k ) = λ ( d + 1 ) + ∑ t = 1 T 1 { A t = k } ∥ X ~ t ∥ 2 2 ≤ ( d + 1 ) ( λ + T ) . \operatorname{tr}(V_{T+1,k})=\lambda(d+1)+\sum_{t=1}^{T}\mathbb{1}\{A_t=k\}\lVert\widetilde{X}_t\rVert_2^2\le(d+1)(\lambda+T). tr ( V T + 1 , k ) = λ ( d + 1 ) + t = 1 ∑ T 1 { A t = k } ∥ X t ∥ 2 2 ≤ ( d + 1 ) ( λ + T ) .
Therefore, by Lemma 10 in Abbasi-Yadkori et al. [1 ] ,
det ( V T + 1 , k ) ≤ { tr ( V T + 1 , k ) d + 1 } d + 1 ≤ ( λ + T ) d + 1 . \det(V_{T+1,k}) \le\left\{\frac{\operatorname{tr}(V_{T+1,k})}{d+1}\right\}^{d+1} \le(\lambda+T)^{d+1}. det ( V T + 1 , k ) ≤ { d + 1 tr ( V T + 1 , k ) } d + 1 ≤ ( λ + T ) d + 1 .
Since det ( λ I d + 1 ) = λ d + 1 \det(\lambda I_{d+1})=\lambda^{d+1} det ( λ I d + 1 ) = λ d + 1 , this implies
log ( det ( V T + 1 , k ) det ( λ I d + 1 ) ) ≤ ( d + 1 ) log ( 1 + T λ ) . \log\left(\frac{\det(V_{T+1,k})}{\det(\lambda I_{d+1})}\right) \le(d+1)\log\left(1+\frac{T}{\lambda}\right). log ( det ( λ I d + 1 ) det ( V T + 1 , k ) ) ≤ ( d + 1 ) log ( 1 + λ T ) .
Consequently,
∑ t = 1 T 1 { A t = k } min { 1 , X ~ t ⊤ V t , k − 1 X ~ t } ≤ 2 ( d + 1 ) log ( 1 + T λ ) . \sum_{t=1}^{T}\mathbb{1}\{A_t=k\}\min\{1,\widetilde{X}_t^{\top}V_{t,k}^{-1}\widetilde{X}_t\}\le2(d+1)\log\left(1+\frac{T}{\lambda}\right). t = 1 ∑ T 1 { A t = k } min { 1 , X t ⊤ V t , k − 1 X t } ≤ 2 ( d + 1 ) log ( 1 + λ T ) .
Summing the last display over k ∈ { 1 , 2 } k\in\{1,2\} k ∈ { 1 , 2 } , we obtain
∑ t = 1 T min { 1 , ∥ X ~ t ∥ V t , A t 2 2 } = ∑ k = 1 2 ∑ t = 1 T 1 { A t = k } min { 1 , X ~ t ⊤ V t , k − 1 X ~ t } ≤ 4 ( d + 1 ) log ( 1 + T λ ) . \sum_{t=1}^{T}\min\{1,\|\widetilde{X}_t\|_{V_{t,A_t}^{2}}^2\}=\sum_{k=1}^{2}\sum_{t=1}^{T}\mathbb{1}\{A_t=k\}\min\{1,\widetilde{X}_t^{\top}V_{t,k}^{-1}\widetilde{X}_t\}\le4(d+1)\log\left(1+\frac{T}{\lambda}\right). t = 1 ∑ T min { 1 , ∥ X t ∥ V t , A t 2 2 } = k = 1 ∑ 2 t = 1 ∑ T 1 { A t = k } min { 1 , X t ⊤ V t , k − 1 X t } ≤ 4 ( d + 1 ) log ( 1 + λ T ) .
Since the left-hand side is non-negative, this also implies
∑ t = s 0 T min { 1 , ∥ X ~ t ∥ V t , A t 2 2 } ≤ 4 ( d + 1 ) log ( 1 + T λ ) . \sum_{t=s_0}^{T}\min\{1,\|\widetilde{X}_t\|_{V_{t,A_t}^{2}}^2\}\le4(d+1)\log\left(1+\frac{T}{\lambda}\right). t = s 0 ∑ T min { 1 , ∥ X t ∥ V t , A t 2 2 } ≤ 4 ( d + 1 ) log ( 1 + λ T ) .
Combining with the Cauchy–Schwarz inequality gives
∑ t = s 0 T min { 1 , ∥ X ~ t ∥ V τ ( t ) , A t − 1 } ≤ 2 T ∑ t = s 0 T min { 1 , ∥ X ~ t ∥ V τ ( t ) , A t − 1 2 } ≤ 8 T ( d + 1 ) log ( 1 + T λ ) . \sum_{t=s_0}^{T}\min\{1,\|\widetilde{X}_t\|_{V_{\tau(t),A_t}^{-1}}\}\le\sqrt{2T\sum_{t=s_0}^{T}\min\{1,\|\widetilde{X}_t\|_{V_{\tau(t),A_t}^{-1}}^2\}}\le\sqrt{8T(d+1)\log\left(1+\frac{T}{\lambda}\right)}. t = s 0 ∑ T min { 1 , ∥ X t ∥ V τ ( t ) , A t − 1 } ≤ 2 T t = s 0 ∑ T min { 1 , ∥ X t ∥ V τ ( t ) , A t − 1 2 } ≤ 8 T ( d + 1 ) log ( 1 + λ T ) .
The lemma thus follows.
Lemma 48 (Empirical optimism). On E c o n f \mathcal{E}_{\mathrm{conf}} E conf in (51) and the event in Lemma 34 on which every feasible difference b 1 − b 2 b_1-b_2 b 1 − b 2 , with b k ∈ C s , k b_k\in\mathcal{C}_{s,k} b k ∈ C s , k , belongs to Θ ~ j ∗ \widetilde{\Theta}_{j^*} Θ j ∗ for every update time s s s , the model pair β ‾ s = ( β ‾ s , 1 , β ‾ s , 2 ) \overline{\beta}_s=(\overline{\beta}_{s,1},\overline{\beta}_{s,2}) β s = ( β s , 1 , β s , 2 ) and policy p s p_s p s returned by Algorithm 4 satisfy
1 s − 1 ∑ i = 1 s − 1 r β ‾ s , p s ( X i ) ≥ 1 s − 1 ∑ i = 1 s − 1 r β ∗ , p ∗ ( X i ) , (70) \frac{1}{s-1}\sum_{i=1}^{s-1}r_{\overline{\beta}_s,p_s}(X_i)\ge\frac{1}{s-1}\sum_{i=1}^{s-1}r_{\beta^*,p^*}(X_i), \tag*{(70)} s − 1 1 i = 1 ∑ s − 1 r β s , p s ( X i ) ≥ s − 1 1 i = 1 ∑ s − 1 r β ∗ , p ∗ ( X i ) , (70) where, for b = ( b 1 , b 2 ) b=(b_1,b_2) b = ( b 1 , b 2 ) and a policy p p p , r b , p ( x ) = x ~ ⊤ b 1 p ( x ) + x ~ ⊤ b 2 { 1 − p ( x ) } r_{b,p}(x)=\widetilde{x}^{\top}b_1p(x)+\widetilde{x}^{\top}b_2\{1-p(x)\} r b , p ( x ) = x ⊤ b 1 p ( x ) + x ⊤ b 2 { 1 − p ( x )} , β ∗ = ( β 1 ∗ , β 2 ∗ ) \beta^*=(\beta_1^*,\beta_2^*) β ∗ = ( β 1 ∗ , β 2 ∗ ) , and p ∗ = p θ ∗ , q ∗ p^*=p_{\theta^*,q^*} p ∗ = p θ ∗ , q ∗ is chosen to be a canonical population-optimal policy as guaranteed by Theorem 3 .
Proof. On E c o n f \mathcal{E}_{\mathrm{conf}} E conf , we have β k ∗ ∈ C s , k \beta_k^*\in\mathcal{C}_{s,k} β k ∗ ∈ C s , k for k ∈ { 1 , 2 } k\in\{1,2\} k ∈ { 1 , 2 } . Moreover, by Lemma 34 , every feasible difference b 1 − b 2 b_1-b_2 b 1 − b 2 belongs to Θ ~ j ∗ ⊆ Θ ~ \widetilde{\Theta}_{j^*}\subseteq\widetilde{\Theta} Θ j ∗ ⊆ Θ . Hence, Lemma 5 applies and gives
1 s − 1 ∑ i = 1 s − 1 r β ‾ s , p s ( X i ) = sup b k ∈ C s , k , q ∈ Q b 1 − b 2 , L 1 s − 1 ∑ i = 1 s − 1 r b , p b 1 − b 2 , q ( X i ) . \frac{1}{s-1}\sum_{i=1}^{s-1}r_{\overline{\beta}_s,p_s}(X_i)=\sup_{b_k\in\mathcal{C}_{s,k},\ q\in\mathcal{Q}_{b_1-b_2,L}}\frac{1}{s-1}\sum_{i=1}^{s-1}r_{b,p_{b_1-b_2,q}}(X_i). s − 1 1 i = 1 ∑ s − 1 r β s , p s ( X i ) = b k ∈ C s , k , q ∈ Q b 1 − b 2 , L sup s − 1 1 i = 1 ∑ s − 1 r b , p b 1 − b 2 , q ( X i ) .
Since θ ∗ = β 1 ∗ − β 2 ∗ \theta^*=\beta_1^*-\beta_2^* θ ∗ = β 1 ∗ − β 2 ∗ , β k ∗ ∈ C s , k \beta_k^*\in\mathcal{C}_{s,k} β k ∗ ∈ C s , k , and q ∗ ∈ Q θ ∗ , L q^*\in\mathcal{Q}_{\theta^*,L} q ∗ ∈ Q θ ∗ , L with p ∗ = p θ ∗ , q ∗ p^*=p_{\theta^*,q^*} p ∗ = p θ ∗ , q ∗ , the true model-policy pair ( β ∗ , q ∗ ) (\beta^*,q^*) ( β ∗ , q ∗ ) is feasible. Therefore,
sup b k ∈ C s , k , q ∈ Q b 1 − b 2 , L 1 s − 1 ∑ i = 1 s − 1 r b , p b 1 − b 2 , q ( X i ) ≥ 1 s − 1 ∑ i = 1 s − 1 r β ∗ , p ∗ ( X i ) . \sup_{b_k\in\mathcal{C}_{s,k},\ q\in\mathcal{Q}_{b_1-b_2,L}}\frac{1}{s-1}\sum_{i=1}^{s-1}r_{b,p_{b_1-b_2,q}}(X_i)\ge\frac{1}{s-1}\sum_{i=1}^{s-1}r_{\beta^*,p^*}(X_i). b k ∈ C s , k , q ∈ Q b 1 − b 2 , L sup s − 1 1 i = 1 ∑ s − 1 r b , p b 1 − b 2 , q ( X i ) ≥ s − 1 1 i = 1 ∑ s − 1 r β ∗ , p ∗ ( X i ) .
Combining the preceding equality and inequality proves (70) .
Lemma 49 (One-step regret). On E c o n f ∩ E e m p \mathcal{E}_{\mathrm{conf}} \cap\mathcal{E}_{\mathrm{emp}} E conf ∩ E emp , defined in (51) and (52) , and the event in Lemma 34 on which every feasible difference b 1 − b 2 \mathbf{b}_1-\mathbf{b}_2 b 1 − b 2 , with b k ∈ C s , k \mathbf{b}_k \in\mathcal{C}_{s,k} b k ∈ C s , k , belongs to Θ ~ j ∗ \widetilde{\Theta}_{j^*} Θ j ∗ and every candidate satisfies ∥ b k ∥ 2 ≤ C β + 1 \lVert\mathbf{b}_k\rVert_2 \le C_\beta+1 ∥ b k ∥ 2 ≤ C β + 1 for every update time s s s , for every t ≥ s 0 t \ge s_0 t ≥ s 0 and s = τ ( t ) s=\tau(t) s = τ ( t ) ,
R ( p ∗ ) − R ( p s ) ≤ 2 Δ s − 1 U C B + { ( 1 + d + 1 ) α T + 2 B } E [ min { 1 , ∥ X ~ t ∥ V s , A t − 1 } ∣ F t − 1 ] . (71) R(p^*)-R(p_s) \le2\Delta^{\mathrm{UCB}}_{s-1}+\{(1+\sqrt{d+1})\alpha_T+2B\}\mathbb{E}\left[\min\{1,\lVert\widetilde{X}_t\rVert_{V_{s,A_t}^{-1}}\}\mid\mathcal{F}_{t-1}\right]. \tag*{(71)} R ( p ∗ ) − R ( p s ) ≤ 2 Δ s − 1 UCB + {( 1 + d + 1 ) α T + 2 B } E [ min { 1 , ∥ X t ∥ V s , A t − 1 } ∣ F t − 1 ] . (71) where B = d + 1 ( C β + 1 ) B=\sqrt{d+1}(C_\beta+1) B = d + 1 ( C β + 1 ) .
Proof. Since R ( p ) = E { r β ∗ , p ( X ) } R(p)=\mathbb{E}\{r_{\beta^*,p}(X)\} R ( p ) = E { r β ∗ , p ( X )} , where X ∼ P X X\sim P_X X ∼ P X , we decompose
R ( p ∗ ) − R ( p s ) = [ E { r β ∗ , p ∗ ( X ) } − 1 s − 1 ∑ i = 1 s − 1 r β ∗ , p ∗ ( X i ) ] + [ 1 s − 1 ∑ i = 1 s − 1 r β ∗ , p ∗ ( X i ) − 1 s − 1 ∑ i = 1 s − 1 r β ˉ s , p s ( X i ) ] + [ 1 s − 1 ∑ i = 1 s − 1 r β ˉ s , p s ( X i ) − E X { r β ˉ s , p s ( X ) } ] + E X { r β ˉ s , p s ( X ) − r β ∗ , p s ( X ) } . \begin{aligned}
R(p^*)-R(p_s)={}&\left[\mathbb{E}\{r_{\beta^*,p^*}(X)\}-\frac{1}{s-1}\sum_{i=1}^{s-1}r_{\beta^*,p^*}(X_i)\right]\\
&+\left[\frac{1}{s-1}\sum_{i=1}^{s-1}r_{\beta^*,p^*}(X_i)-\frac{1}{s-1}\sum_{i=1}^{s-1}r_{\bar{\beta}_s,p_s}(X_i)\right]\\
&+\left[\frac{1}{s-1}\sum_{i=1}^{s-1}r_{\bar{\beta}_s,p_s}(X_i)-\mathbb{E}_X\{r_{\bar{\beta}_s,p_s}(X)\}\right]\\
&+\mathbb{E}_X\{r_{\bar{\beta}_s,p_s}(X)-r_{\beta^*,p_s}(X)\}.
\end{aligned} R ( p ∗ ) − R ( p s ) = [ E { r β ∗ , p ∗ ( X )} − s − 1 1 i = 1 ∑ s − 1 r β ∗ , p ∗ ( X i ) ] + [ s − 1 1 i = 1 ∑ s − 1 r β ∗ , p ∗ ( X i ) − s − 1 1 i = 1 ∑ s − 1 r β ˉ s , p s ( X i ) ] + [ s − 1 1 i = 1 ∑ s − 1 r β ˉ s , p s ( X i ) − E X { r β ˉ s , p s ( X )} ] + E X { r β ˉ s , p s ( X ) − r β ∗ , p s ( X )} .
Since p ∗ = p θ ∗ , q ∗ p^*=p_{\theta^*,q^*} p ∗ = p θ ∗ , q ∗ , θ ∗ ∈ Θ ~ \theta^*\in\widetilde{\Theta} θ ∗ ∈ Θ , and ∥ β k ∗ ∥ 2 ≤ C β \lVert\beta_k^*\rVert_2\le C_\beta ∥ β k ∗ ∥ 2 ≤ C β , we have r β ∗ , p ∗ ∈ F U C B r_{\beta^*,p^*}\in\mathcal{F}_{\mathrm{UCB}} r β ∗ , p ∗ ∈ F UCB . Moreover, on the event of Lemma 34 , θ s = β ˉ s , 1 − β ˉ s , 2 ∈ Θ ~ j ∗ ⊆ Θ ~ \theta_s=\bar{\beta}_{s,1}-\bar{\beta}_{s,2}\in\widetilde{\Theta}_{j^*}\subseteq\widetilde{\Theta} θ s = β ˉ s , 1 − β ˉ s , 2 ∈ Θ j ∗ ⊆ Θ , and the same lemma gives ∥ β ˉ s , k ∥ 2 ≤ C β + 1 \lVert\bar{\beta}_{s,k}\rVert_2\le C_\beta+1 ∥ β ˉ s , k ∥ 2 ≤ C β + 1 , so it holds that r β ˉ s , p s ∈ F U C B r_{\bar{\beta}_s,p_s}\in\mathcal{F}_{\mathrm{UCB}} r β ˉ s , p s ∈ F UCB . Therefore, we can control the first and third terms of the order at most Δ s − 1 U C B \Delta^{\mathrm{UCB}}_{s-1} Δ s − 1 UCB each on E e m p \mathcal{E}_{\mathrm{emp}} E emp . Finally, Lemma 48 shows that the second term is nonpositive.
It remains to control the final term. For k ∈ { 1 , 2 } k\in\{1,2\} k ∈ { 1 , 2 } , Lemma 31 gives ∥ β ˉ s , k − β k ∗ ∥ V s , k ≤ ( 1 + d + 1 ) α T \lVert\bar{\beta}_{s,k}-\beta_k^*\rVert_{V_{s,k}}\le(1+\sqrt{d+1})\alpha_T ∥ β ˉ s , k − β k ∗ ∥ V s , k ≤ ( 1 + d + 1 ) α T on E c o n f \mathcal{E}_{\mathrm{conf}} E conf . By Lemma 34 and Assumption 3, both β ˉ s , k \bar{\beta}_{s,k} β ˉ s , k and β k ∗ \beta_k^* β k ∗ have Euclidean norm at most C β + 1 C_\beta+1 C β + 1 . Write B = d + 1 ( C β + 1 ) B=\sqrt{d+1}(C_\beta+1) B = d + 1 ( C β + 1 ) . Hence,
∣ x ~ T ( β ˉ s , k − β k ∗ ) ∣ ≤ min { ( 1 + d + 1 ) α T ∥ x ~ ∥ V s , k − 1 , 2 B } ≤ { ( 1 + d + 1 ) α T + 2 B } min { 1 , ∥ x ~ ∥ V s , k − 1 } . \left|\widetilde{x}^{\mathsf{T}}(\bar{\beta}_{s,k}-\beta_k^*)\right|\le\min\{(1+\sqrt{d+1})\alpha_T\lVert\widetilde{x}\rVert_{V_{s,k}^{-1}},2B\}\le\{(1+\sqrt{d+1})\alpha_T+2B\}\min\{1,\lVert\widetilde{x}\rVert_{V_{s,k}^{-1}}\}. x T ( β ˉ s , k − β k ∗ ) ≤ min {( 1 + d + 1 ) α T ∥ x ∥ V s , k − 1 , 2 B } ≤ {( 1 + d + 1 ) α T + 2 B } min { 1 , ∥ x ∥ V s , k − 1 } .
Note that the update time s = τ ( t ) s=\tau(t) s = τ ( t ) and the quantities β ˉ s \bar{\beta}_s β ˉ s , p s p_s p s , and V s , k V_{s,k} V s , k used at round t t t are F t − 1 \mathcal{F}_{t-1} F t − 1 -measurable. Since X t X_t X t is independent of F t − 1 \mathcal{F}_{t-1} F t − 1 and has law P X P_X P X , we have that
E { r β ˉ s , p s ( X ) − r β ∗ , p s ( X ) } = E [ X ~ t T ( β ˉ s , 1 − β 1 ∗ ) p s ( X t ) + X ~ t T ( β ˉ s , 2 − β 2 ∗ ) { 1 − p s ( X t ) } ∣ F t − 1 ] ≤ { ( 1 + d + 1 ) α T + 2 B } E [ p s ( X t ) min { 1 , ∥ X ~ t ∥ V s , 1 − 1 } + { 1 − p s ( X t ) } min { 1 , ∥ X ~ t ∥ V s , 2 − 1 } ∣ F t − 1 ] = { ( 1 + d + 1 ) α T + 2 B } E [ ∑ k = 1 2 min { 1 , ∥ X ~ t ∥ V s , k − 1 } P ( A t = k ∣ X t , F t − 1 ) ∣ F t − 1 ] = { ( 1 + d + 1 ) α T + 2 B } E [ E [ min { 1 , ∥ X ~ t ∥ V s , A t − 1 } ∣ X t , F t − 1 ] ∣ F t − 1 ] = { ( 1 + d + 1 ) α T + 2 B } E [ min { 1 , ∥ X ~ t ∥ V s , A t − 1 } ∣ F t − 1 ] = { ( 1 + d + 1 ) α T + 2 B } E [ min { 1 , ∥ X ~ t ∥ V τ ( t ) , A t − 1 } ∣ F t − 1 ] , \begin{aligned}
\mathbb{E}\{r_{\bar{\beta}_s,p_s}(X)-r_{\beta^*,p_s}(X)\}
={}&\mathbb{E}\left[\widetilde{X}_t^{\mathsf{T}}(\bar{\beta}_{s,1}-\beta_1^*)p_s(X_t)+\widetilde{X}_t^{\mathsf{T}}(\bar{\beta}_{s,2}-\beta_2^*)\{1-p_s(X_t)\}\mid\mathcal{F}_{t-1}\right]\\
\le{}&\{(1+\sqrt{d+1})\alpha_T+2B\}\mathbb{E}\left[p_s(X_t)\min\{1,\lVert\widetilde{X}_t\rVert_{V_{s,1}^{-1}}\}+\{1-p_s(X_t)\}\min\{1,\lVert\widetilde{X}_t\rVert_{V_{s,2}^{-1}}\}\mid\mathcal{F}_{t-1}\right]\\
={}&\{(1+\sqrt{d+1})\alpha_T+2B\}\mathbb{E}\left[\sum_{k=1}^{2}\min\{1,\lVert\widetilde{X}_t\rVert_{V_{s,k}^{-1}}\}\mathbb{P}(A_t=k\mid X_t,\mathcal{F}_{t-1})\mid\mathcal{F}_{t-1}\right]\\
={}&\{(1+\sqrt{d+1})\alpha_T+2B\}\mathbb{E}\left[\mathbb{E}\left[\min\{1,\lVert\widetilde{X}_t\rVert_{V_{s,A_t}^{-1}}\}\mid X_t,\mathcal{F}_{t-1}\right]\mid\mathcal{F}_{t-1}\right]\\
={}&\{(1+\sqrt{d+1})\alpha_T+2B\}\mathbb{E}\left[\min\{1,\lVert\widetilde{X}_t\rVert_{V_{s,A_t}^{-1}}\}\mid\mathcal{F}_{t-1}\right]\\
={}&\{(1+\sqrt{d+1})\alpha_T+2B\}\mathbb{E}\left[\min\{1,\lVert\widetilde{X}_t\rVert_{V_{\tau(t),A_t}^{-1}}\}\mid\mathcal{F}_{t-1}\right],
\end{aligned} E { r β ˉ s , p s ( X ) − r β ∗ , p s ( X )} = ≤ = = = = E [ X t T ( β ˉ s , 1 − β 1 ∗ ) p s ( X t ) + X t T ( β ˉ s , 2 − β 2 ∗ ) { 1 − p s ( X t )} ∣ F t − 1 ] {( 1 + d + 1 ) α T + 2 B } E [ p s ( X t ) min { 1 , ∥ X t ∥ V s , 1 − 1 } + { 1 − p s ( X t )} min { 1 , ∥ X t ∥ V s , 2 − 1 } ∣ F t − 1 ] {( 1 + d + 1 ) α T + 2 B } E [ k = 1 ∑ 2 min { 1 , ∥ X t ∥ V s , k − 1 } P ( A t = k ∣ X t , F t − 1 ) ∣ F t − 1 ] {( 1 + d + 1 ) α T + 2 B } E [ E [ min { 1 , ∥ X t ∥ V s , A t − 1 } ∣ X t , F t − 1 ] ∣ F t − 1 ] {( 1 + d + 1 ) α T + 2 B } E [ min { 1 , ∥ X t ∥ V s , A t − 1 } ∣ F t − 1 ] {( 1 + d + 1 ) α T + 2 B } E [ min { 1 , ∥ X t ∥ V τ ( t ) , A t − 1 } ∣ F t − 1 ] ,
where the fourth equality follows from the tower property, and the last equality uses s = τ ( t ) s=\tau(t) s = τ ( t ) . Substituting this bound into the preceding decomposition proves (71) .
Lemma 50 (Summation of empirical errors). For the lazy schedule in Algorithm 3, τ ( t ) > t / 2 \tau(t)>t/2 τ ( t ) > t /2 for every t ≥ s 0 t\ge s_0 t ≥ s 0 . Consequently, with u = η / ( 8 T ) u=\eta/(8T) u = η / ( 8 T ) ,
∑ t = s 0 T ρ τ ( t ) − 1 , d − 1 ( u ) ≤ C { T log ( e T / u ) , d = 2 , T log 3 / 2 ( e T / u ) , d = 3 , T ( d − 2 ) / ( d − 1 ) log ( e T / u ) , d ≥ 4. \sum_{t=s_0}^{T}\rho_{\tau(t)-1,d-1}(u)\le C\begin{cases}\sqrt{T}\log(eT/u), & d=2,\\ \sqrt{T}\log^{3/2}(eT/u), & d=3,\\ T^{(d-2)/(d-1)}\log(eT/u), & d\ge4.\end{cases} t = s 0 ∑ T ρ τ ( t ) − 1 , d − 1 ( u ) ≤ C ⎩ ⎨ ⎧ T log ( e T / u ) , T log 3/2 ( e T / u ) , T ( d − 2 ) / ( d − 1 ) log ( e T / u ) , d = 2 , d = 3 , d ≥ 4.
Proof. If an update occurs at t t t , then τ ( t ) = t \tau(t)=t τ ( t ) = t . Otherwise, the time trigger has not fired, so t < 2 τ ( t ) t<2\tau(t) t < 2 τ ( t ) . Thus τ ( t ) > t / 2 \tau(t)>t/2 τ ( t ) > t /2 . Monotonicity gives ρ τ ( t ) − 1 , m ( u ) ≤ C ρ ⌊ t / 2 ⌋ , m ( u ) \rho_{\tau(t)-1,m}(u)\le C\rho_{\lfloor t/2\rfloor,m}(u) ρ τ ( t ) − 1 , m ( u ) ≤ C ρ ⌊ t /2 ⌋ , m ( u ) after changing a universal constant. Summing the three displayed forms of ρ \rho ρ and using
∑ t = 1 T t − a ≤ C a T 1 − a \sum_{t=1}^{T}t^{-a}\le C_aT^{1-a} t = 1 ∑ T t − a ≤ C a T 1 − a
for 0 < a < 1 0<a<1 0 < a < 1 proves the result.
Proofs of results in Section 5
Proof of Theorem 7
Assume that X t ∈ X = [ 0 , 1 ] d X_t\in\mathcal{X}=[0,1]^d X t ∈ X = [ 0 , 1 ] d is drawn independently from a density g g g on [ 0 , 1 ] d [0,1]^d [ 0 , 1 ] d . The learner observes X t X_t X t , chooses arm A t ∈ { 1 , 2 } A_t\in\{1,2\} A t ∈ { 1 , 2 } , and then observes
Y t = r A t ( X t ) + ξ t , ξ t ∼ i . i . d . N ( 0 , 1 ) . Y_t=r_{A_t}(X_t)+\xi_t,\qquad\xi_t\stackrel{\mathrm{i.i.d.}}{\sim}N(0,1). Y t = r A t ( X t ) + ξ t , ξ t ∼ i.i.d. N ( 0 , 1 ) .
The noises are independent of the contexts and the learner’s internal randomization. We write
Δ ( x ) = r 1 ( x ) − r 2 ( x ) . \Delta(x)=r_1(x)-r_2(x). Δ ( x ) = r 1 ( x ) − r 2 ( x ) .
The policy used at time t t t is a measurable function p t : [ 0 , 1 ] d → [ 0 , 1 ] p_t:[0,1]^d\to[0,1] p t : [ 0 , 1 ] d → [ 0 , 1 ] , where p t ( x ) p_t(x) p t ( x ) is the conditional probability of pulling arm 1 at context x x x . The policy may be random and adapted to the previous history, but it is selected before seeing the current context and must satisfy the fairness constraint. Define
P L , 1 = { p : [ 0 , 1 ] d → [ 0 , 1 ] : ∣ p ( x ) − p ( y ) ∣ ≤ L ∥ x − y ∥ 1 for all x , y ∈ [ 0 , 1 ] d } . \mathcal{P}_{L,1}=\{p:[0,1]^d\to[0,1]:|p(x)-p(y)|\le L\lVert x-y\rVert_1\text{ for all }x,y\in[0,1]^d\}. P L , 1 = { p : [ 0 , 1 ] d → [ 0 , 1 ] : ∣ p ( x ) − p ( y ) ∣ ≤ L ∥ x − y ∥ 1 for all x , y ∈ [ 0 , 1 ] d } .
The population value of a policy p p p under density g g g is
J g ( p ) = ∫ [ 0 , 1 ] d g ( x ) Δ ( x ) p ( x ) d x . J_g(p)=\int_{[0,1]^d}g(x)\Delta(x)p(x)\,\mathrm{d}x. J g ( p ) = ∫ [ 0 , 1 ] d g ( x ) Δ ( x ) p ( x ) d x .
We consider arbitrary non-anticipating learning algorithms. Let F t − 1 \mathcal{F}_{t-1} F t − 1 contain the observed history σ { X s , A s , Y s : 1 ≤ s ≤ t − 1 } \sigma\{X_s,A_s,Y_s:1\le s\le t-1\} σ { X s , A s , Y s : 1 ≤ s ≤ t − 1 } together with the learner’s internal randomization available when selecting p t p_t p t . All randomization kernels of the learner are fixed across environments and independent of future contexts and reward noise. A learning algorithm π \pi π is called non-anticipating if, at each round t t t , it selects an F t − 1 \mathcal{F}_{t-1} F t − 1 -measurable random function
p t : [ 0 , 1 ] d → [ 0 , 1 ] , p_t:[0,1]^d\to[0,1], p t : [ 0 , 1 ] d → [ 0 , 1 ] ,
such that p t ∈ P L , 1 p_t\in\mathcal{P}_{L,1} p t ∈ P L , 1 a.s.
After observing the current covariate X t X_t X t , the algorithm pulls arm 1 with probability p t ( X t ) p_t(X_t) p t ( X t ) and arm 2 with probability 1 − p t ( X t ) 1-p_t(X_t) 1 − p t ( X t ) . Thus π \pi π induces a sequence of fair policies { p t } t = 1 T ⊂ P L , 1 \{p_t\}_{t=1}^{T}\subset\mathcal{P}_{L,1} { p t } t = 1 T ⊂ P L , 1 almost surely. Let p g ⋆ ∈ arg max q ∈ P L , 1 J g ( q ) p_g^\star\in\operatorname*{arg\,max}_{q\in\mathcal{P}_{L,1}} J_g(q) p g ⋆ ∈ arg max q ∈ P L , 1 J g ( q ) be the optimal policy. Then the regret of an algorithm { p t π } t = 1 T \{p_t^\pi\}_{t=1}^T { p t π } t = 1 T is
Reg g = E g [ ∑ t = 1 T { J g ( p g ⋆ ) − J g ( p t π ) } ] . \operatorname{Reg}_g = \mathbb{E}_g\left[\sum_{t=1}^T\{J_g(p_g^\star)-J_g(p_t^\pi)\}\right]. Reg g = E g [ t = 1 ∑ T { J g ( p g ⋆ ) − J g ( p t π )} ] .
Since J g J_g J g is linear in the policy, hence for p ˉ π = 1 T ∑ t = 1 T p t π \bar{p}^\pi=\frac{1}{T}\sum_{t=1}^T p_t^\pi p ˉ π = T 1 ∑ t = 1 T p t π , we can write
Reg g ( p ˉ π , T ) = T E g { J g ( p g ⋆ ) − J g ( p ˉ π ) } . \operatorname{Reg}_g(\bar{p}^\pi,T)=T\mathbb{E}_g\{J_g(p_g^\star)-J_g(\bar{p}^\pi)\}. Reg g ( p ˉ π , T ) = T E g { J g ( p g ⋆ ) − J g ( p ˉ π )} .
Theorem 51 (Lower bound). For the reward functions r 1 ( x ) = x 1 − 1 / 2 r_1(x)=x_1-1/2 r 1 ( x ) = x 1 − 1/2 and r 2 ( x ) = 0 r_2(x)=0 r 2 ( x ) = 0 , fix L ∈ ( 0 , 1 ) L\in(0,1) L ∈ ( 0 , 1 ) and let P L , 1 \mathcal{P}_{L,1} P L , 1 be the collection of policies satisfying the ( L , 1 ) (L,1) ( L , 1 ) -Lipschitz constraint. Let G \mathcal{G} G be the collection of bounded densities on X \mathcal{X} X . Then, there is a constant c d , L > 0 c_{d,L}>0 c d , L > 0 depending only on d d d and L L L , such that for every integer T ≥ 1 T\geq1 T ≥ 1 ,
inf π ∈ Π sup g ∈ G Reg g ( p ˉ π , T ) ≥ c d , L { T , d = 1 , 2 , T 1 − 1 d − 1 , d ≥ 3 , \inf_{\pi\in\Pi}\sup_{g\in\mathcal{G}}\operatorname{Reg}_g(\bar{p}^\pi,T)\geq c_{d,L}
\begin{cases}
\sqrt{T}, & d=1,2,\\
T^{1-\frac{1}{d-1}}, & d\geq3,
\end{cases} π ∈ Π inf g ∈ G sup Reg g ( p ˉ π , T ) ≥ c d , L { T , T 1 − d − 1 1 , d = 1 , 2 , d ≥ 3 , where Π \Pi Π is the class of all the non-anticipating fair policies such that
Π = { { p t } t = 1 ∞ : p t is F t − 1 -measurable and p t ∈ P L , 1 a.s. for every t } . \Pi=\left\{\{p_t\}_{t=1}^{\infty}:p_t\text{ is }\mathcal{F}_{t-1}\text{-measurable and }p_t\in\mathcal{P}_{L,1}\text{ a.s. for every }t\right\}. Π = { { p t } t = 1 ∞ : p t is F t − 1 -measurable and p t ∈ P L , 1 a.s. for every t } .
Proof. Fix an arbitrary learning algorithm π ∈ Π \pi\in\Pi π ∈ Π , common to all densities. Write p ˉ = T − 1 ∑ t = 1 T p t π \bar{p}=T^{-1}\sum_{t=1}^T p_t^\pi p ˉ = T − 1 ∑ t = 1 T p t π . By convexity, p ˉ ∈ P L , 1 \bar{p}\in\mathcal{P}_{L,1} p ˉ ∈ P L , 1 almost surely. All interaction laws below include the learner’s internal randomization, so p ˉ \bar{p} p ˉ is the same measurable function of the full interaction under every environment. Throughout the proof, the reward gap is
Δ ( x ) = x 1 − 1 2 , r 1 ( x ) = Δ ( x ) and r 2 = 0 , x ∈ [ 0 , 1 ] d . \Delta(x)=x_1-\frac{1}{2},\qquad r_1(x)=\Delta(x)\quad\text{and}\quad r_2=0,\qquad x\in[0,1]^d. Δ ( x ) = x 1 − 2 1 , r 1 ( x ) = Δ ( x ) and r 2 = 0 , x ∈ [ 0 , 1 ] d .
The following lower bound proof uses two different hard constructions. When d = 1 d=1 d = 1 , we use a two-point density perturbation around the unique decision boundary x 1 = 1 / 2 x_1=1/2 x 1 = 1/2 and apply a Le Cam strategy. The one-dimensional construction is embedded into d = 2 d=2 d = 2 below. When d ≥ 3 d\geq3 d ≥ 3 , we instead use an Assouad hypercube construction by encoding many independent bits in the ( d − 1 ) (d-1) ( d − 1 ) tangential boundary coordinates u = ( x 2 , x 3 , … , x d ) u=(x_2,x_3,\ldots,x_d) u = ( x 2 , x 3 , … , x d ) .
(1) Proof for d = 1 d=1 d = 1 and d = 2 d=2 d = 2 .
Step 1. Density construction. Note that we assume L ∈ ( 0 , 1 ) L\in(0,1) L ∈ ( 0 , 1 ) . We set c 0 = ( 1 − L ) / 4 > 0 c_0=(1-L)/4>0 c 0 = ( 1 − L ) /4 > 0 , and define
p 0 ( x ) = 1 2 + L Δ ( x ) , p 1 ( x ) = p 0 ( x ) + c 0 . p^0(x)=\frac{1}{2}+L\Delta(x),\qquad p^1(x)=p^0(x)+c_0. p 0 ( x ) = 2 1 + L Δ ( x ) , p 1 ( x ) = p 0 ( x ) + c 0 .
Clearly, both p 0 , p 1 ∈ P L , 1 p^0,p^1\in\mathcal{P}_{L,1} p 0 , p 1 ∈ P L , 1 . Moreover, since p 0 ( [ 0 , 1 ] ) ⊂ [ ( 1 − L ) / 2 , ( 1 + L ) / 2 ] p^0([0,1])\subset[(1-L)/2,(1+L)/2] p 0 ([ 0 , 1 ]) ⊂ [( 1 − L ) /2 , ( 1 + L ) /2 ] and c 0 = ( 1 − L ) / 4 c_0=(1-L)/4 c 0 = ( 1 − L ) /4 , both policies take values in [ 0 , 1 ] [0,1] [ 0 , 1 ] .
Let
s = 1 4 , h = s 16 , x + = 1 2 + s 2 , x − = 1 2 − s 2 , s=\frac{1}{4},\qquad h=\frac{s}{16},\qquad x_+=\frac{1}{2}+\frac{s}{2},\qquad x_-=\frac{1}{2}-\frac{s}{2}, s = 4 1 , h = 16 s , x + = 2 1 + 2 s , x − = 2 1 − 2 s ,
and define
I + = [ x + − h , x + + h ] , I − = [ x − − h , x − + h ] . I_+=[x_+-h,x_++h],\qquad I_-=[x_--h,x_-+h]. I + = [ x + − h , x + + h ] , I − = [ x − − h , x − + h ] .
For ε ∈ ( 0 , 1 / 8 ] \varepsilon\in(0,1/8] ε ∈ ( 0 , 1/8 ] , set
f + ( x ) = ( 1 2 + ε ) 1 I + ( x ) 2 h + ( 1 2 − ε ) 1 I − ( x ) 2 h , f_+(x)=\left(\frac{1}{2}+\varepsilon\right)\frac{\mathbf{1}_{I_+}(x)}{2h}+\left(\frac{1}{2}-\varepsilon\right)\frac{\mathbf{1}_{I_-}(x)}{2h}, f + ( x ) = ( 2 1 + ε ) 2 h 1 I + ( x ) + ( 2 1 − ε ) 2 h 1 I − ( x ) ,
and
f − ( x ) = ( 1 2 − ε ) 1 I + ( x ) 2 h + ( 1 2 + ε ) 1 I − ( x ) 2 h . f_-(x)=\left(\frac{1}{2}-\varepsilon\right)\frac{\mathbf{1}_{I_+}(x)}{2h}+\left(\frac{1}{2}+\varepsilon\right)\frac{\mathbf{1}_{I_-}(x)}{2h}. f − ( x ) = ( 2 1 − ε ) 2 h 1 I + ( x ) + ( 2 1 + ε ) 2 h 1 I − ( x ) .
To keep the densities bounded away from zero, we mix them with the uniform density:
g + = ( 1 + f + ) / 2 , g − = ( 1 + f − ) / 2. g_{+} = (1 + f_{+})/2,\qquad g_{-} = (1 + f_{-})/2. g + = ( 1 + f + ) /2 , g − = ( 1 + f − ) /2.
Then, by construction, we have g ± ≥ 1 / 2 g_{\pm} \ge1/2 g ± ≥ 1/2 and g ± ≤ 1 / 2 + 1 / ( 2 h ) g_{\pm} \le1/2 + 1/(2h) g ± ≤ 1/2 + 1/ ( 2 h ) , where h = 1 / 64 h = 1/64 h = 1/64 is fixed. Let f 0 = ( f + + f − ) / 2 f_{0} = (f_{+} + f_{-})/2 f 0 = ( f + + f − ) /2 and g 0 = ( g + + g − ) / 2 g_{0} = (g_{+} + g_{-})/2 g 0 = ( g + + g − ) /2 .
Step 2. Optimality of p 0 p^{0} p 0 for the baseline density g 0 g_{0} g 0 . We first prove that p 0 p^{0} p 0 is optimal for J f 0 J_{f_{0}} J f 0 . Let τ ( x ) = 1 − x \tau(x) = 1 - x τ ( x ) = 1 − x , which maps I + I_{+} I + onto I − I_{-} I − . Since f 0 f_{0} f 0 puts equal mass on the two intervals, it holds for any q ∈ P L , 1 q \in\mathcal{P}_{L,1} q ∈ P L , 1 that
∫ 0 1 Δ ( x ) q ( x ) f 0 ( x ) d x = 1 4 h { ∫ I + Δ ( x ) q ( x ) d x + ∫ I − Δ ( x ) q ( x ) d x } = 1 4 h { ∫ I + Δ ( x ) q ( x ) d x + ∫ I + Δ ( τ ( x ) ) q ( τ ( x ) ) d x } = 1 4 h { ∫ I + Δ ( x ) q ( x ) d x − ∫ I + Δ ( x ) q ( τ ( x ) ) d x } = 1 4 h ∫ I + Δ ( x ) { q ( x ) − q ( τ ( x ) ) } d x . \begin{aligned}
\int_{0}^{1} \Delta(x)q(x)f_{0}(x)\,\mathrm{d}x
= \frac{1}{4h}\left\{\int_{I_{+}} \Delta(x)q(x)\,\mathrm{d}x + \int_{I_{-}} \Delta(x)q(x)\,\mathrm{d}x\right\} \\
&= \frac{1}{4h}\left\{\int_{I_{+}} \Delta(x)q(x)\,\mathrm{d}x + \int_{I_{+}} \Delta(\tau(x))q(\tau(x))\,\mathrm{d}x\right\} \\
&= \frac{1}{4h}\left\{\int_{I_{+}} \Delta(x)q(x)\,\mathrm{d}x - \int_{I_{+}} \Delta(x)q(\tau(x))\,\mathrm{d}x\right\} \\
&= \frac{1}{4h}\int_{I_{+}} \Delta(x)\{q(x) - q(\tau(x))\}\,\mathrm{d}x.
\end{aligned} ∫ 0 1 Δ ( x ) q ( x ) f 0 ( x ) d x = 4 h 1 { ∫ I + Δ ( x ) q ( x ) d x + ∫ I − Δ ( x ) q ( x ) d x } = 4 h 1 { ∫ I + Δ ( x ) q ( x ) d x + ∫ I + Δ ( τ ( x )) q ( τ ( x )) d x } = 4 h 1 { ∫ I + Δ ( x ) q ( x ) d x − ∫ I + Δ ( x ) q ( τ ( x )) d x } = 4 h 1 ∫ I + Δ ( x ) { q ( x ) − q ( τ ( x ))} d x .
where the third equality uses Δ ( τ ( x ) ) = − Δ ( x ) \Delta(\tau(x)) = -\Delta(x) Δ ( τ ( x )) = − Δ ( x ) . For x ∈ I + x \in I_{+} x ∈ I + , we have that Δ ( x ) ≥ 0 \Delta(x) \ge0 Δ ( x ) ≥ 0 , and
q ( x ) − q ( τ ( x ) ) ≤ L ∣ x − τ ( x ) ∣ = 2 L Δ ( x ) , q(x) - q(\tau(x)) \le L|x - \tau(x)| = 2L\Delta(x), q ( x ) − q ( τ ( x )) ≤ L ∣ x − τ ( x ) ∣ = 2 L Δ ( x ) ,
Therefore, we have that
∫ 0 1 Δ ( x ) q ( x ) f 0 ( x ) d x ≤ L 2 h ∫ I + Δ 2 ( x ) d x . \int_{0}^{1} \Delta(x)q(x)f_{0}(x)\,\mathrm{d}x
\le\frac{L}{2h}\int_{I_{+}} \Delta^{2}(x)\,\mathrm{d}x. ∫ 0 1 Δ ( x ) q ( x ) f 0 ( x ) d x ≤ 2 h L ∫ I + Δ 2 ( x ) d x .
Note that this upper bound is attained by p 0 p^{0} p 0 , because by construction
p 0 ( x ) − p 0 ( τ ( x ) ) = 2 L Δ ( x ) , x ∈ I + . p^{0}(x) - p^{0}(\tau(x)) = 2L\Delta(x),\qquad x \in I_{+}. p 0 ( x ) − p 0 ( τ ( x )) = 2 L Δ ( x ) , x ∈ I + .
Hence p 0 p^{0} p 0 maximizes J f 0 J_{f_{0}} J f 0 over P L , 1 \mathcal{P}_{L,1} P L , 1 .
We then prove that p 0 p^{0} p 0 is optimal for J 1 J_{1} J 1 , note that for any q ∈ P L , 1 q \in\mathcal{P}_{L,1} q ∈ P L , 1 , it holds that
∫ 0 1 Δ ( x ) q ( x ) d x = ∫ 1 / 2 1 Δ ( x ) { q ( x ) − q ( 1 − x ) } d x ≤ ∫ 1 / 2 1 2 L Δ 2 ( x ) d x , \int_{0}^{1} \Delta(x)q(x)\,\mathrm{d}x
= \int_{1/2}^{1} \Delta(x)\{q(x) - q(1-x)\}\,\mathrm{d}x
\le\int_{1/2}^{1} 2L\Delta^{2}(x)\,\mathrm{d}x, ∫ 0 1 Δ ( x ) q ( x ) d x = ∫ 1/2 1 Δ ( x ) { q ( x ) − q ( 1 − x )} d x ≤ ∫ 1/2 1 2 L Δ 2 ( x ) d x ,
and equality is attained by p 0 p^{0} p 0 . Indeed, by the definition of p 0 p^{0} p 0 , for every x ∈ [ 1 / 2 , 1 ] x \in[1/2,1] x ∈ [ 1/2 , 1 ] , we have that
p 0 ( x ) − p 0 ( 1 − x ) = 2 L Δ ( x ) . p^{0}(x) - p^{0}(1-x) = 2L\Delta(x). p 0 ( x ) − p 0 ( 1 − x ) = 2 L Δ ( x ) .
Therefore,
∫ 0 1 Δ ( x ) p 0 ( x ) d x = ∫ 1 / 2 1 Δ ( x ) { p 0 ( x ) − p 0 ( 1 − x ) } d x = ∫ 1 / 2 1 2 L Δ 2 ( x ) d x . \int_{0}^{1} \Delta(x)p^{0}(x)\,\mathrm{d}x
= \int_{1/2}^{1} \Delta(x)\{p^{0}(x) - p^{0}(1-x)\}\,\mathrm{d}x
= \int_{1/2}^{1} 2L\Delta^{2}(x)\,\mathrm{d}x. ∫ 0 1 Δ ( x ) p 0 ( x ) d x = ∫ 1/2 1 Δ ( x ) { p 0 ( x ) − p 0 ( 1 − x )} d x = ∫ 1/2 1 2 L Δ 2 ( x ) d x .
Since by construction, we have that g 0 = ( 1 + f 0 ) / 2 g_{0} = (1 + f_{0})/2 g 0 = ( 1 + f 0 ) /2 , we have
J g 0 ( q ) = 1 2 ∫ 0 1 Δ ( x ) q ( x ) d x + 1 2 ∫ 0 1 Δ ( x ) q ( x ) f 0 ( x ) d x . J_{g_{0}}(q) = \frac{1}{2}\int_{0}^{1} \Delta(x)q(x)\,\mathrm{d}x
+ \frac{1}{2}\int_{0}^{1} \Delta(x)q(x)f_{0}(x)\,\mathrm{d}x. J g 0 ( q ) = 2 1 ∫ 0 1 Δ ( x ) q ( x ) d x + 2 1 ∫ 0 1 Δ ( x ) q ( x ) f 0 ( x ) d x .
Therefore p 0 p^{0} p 0 maximizes both components of g 0 g_{0} g 0 , and hence by linearity, maximizes J g 0 J_{g_{0}} J g 0 .
Step 3. Separation of p 1 p^{1} p 1 and p 0 p^{0} p 0 . We next compute the separation between p 1 p^{1} p 1 and p 0 p^{0} p 0 under g + g_{+} g + . Since p 1 − p 0 = c 0 p^{1}-p^{0}=c_{0} p 1 − p 0 = c 0 and ∫ 0 1 Δ ( x ) d x = 0 \int_{0}^{1}\Delta(x)\,dx=0 ∫ 0 1 Δ ( x ) d x = 0 , we have
J g + ( p 1 ) − J g + ( p 0 ) = c 0 ∫ 0 1 Δ ( x ) g + ( x ) d x = c 0 2 ∫ 0 1 Δ ( x ) f + ( x ) d x . \begin{aligned}
J_{g_{+}}(p^{1})-J_{g_{+}}(p^{0})=c_{0}\int_{0}^{1}\Delta(x)g_{+}(x)\,dx \\
&=\frac{c_{0}}{2}\int_{0}^{1}\Delta(x)f_{+}(x)\,dx.
\end{aligned} J g + ( p 1 ) − J g + ( p 0 ) = c 0 ∫ 0 1 Δ ( x ) g + ( x ) d x = 2 c 0 ∫ 0 1 Δ ( x ) f + ( x ) d x .
Note that
1 2 h ∫ I + Δ ( x ) d x = 1 2 h ∫ x + − h x + + h ( x − 1 2 ) d x = x + − 1 2 = s 2 , \frac{1}{2h}\int_{I_{+}}\Delta(x)\,dx=\frac{1}{2h}\int_{x_{+}-h}^{x_{+}+h}\left(x-\frac{1}{2}\right)\,dx=x_{+}-\frac{1}{2}=\frac{s}{2}, 2 h 1 ∫ I + Δ ( x ) d x = 2 h 1 ∫ x + − h x + + h ( x − 2 1 ) d x = x + − 2 1 = 2 s ,
and similarly,
1 2 h ∫ I − Δ ( x ) d x = 1 2 h ∫ x − − h x − + h ( x − 1 2 ) d x = x − − 1 2 = − s 2 . \frac{1}{2h}\int_{I_{-}}\Delta(x)\,dx=\frac{1}{2h}\int_{x_{-}-h}^{x_{-}+h}\left(x-\frac{1}{2}\right)\,dx=x_{-}-\frac{1}{2}=-\frac{s}{2}. 2 h 1 ∫ I − Δ ( x ) d x = 2 h 1 ∫ x − − h x − + h ( x − 2 1 ) d x = x − − 2 1 = − 2 s .
Therefore,
∫ 0 1 Δ ( x ) f + ( x ) d x = ( 1 2 + ε ) 1 2 h ∫ I + Δ ( x ) d x + ( 1 2 − ε ) 1 2 h ∫ I − Δ ( x ) d x = ( 1 2 + ε ) s 2 + ( 1 2 − ε ) ( − s 2 ) = s 4 + ε s 2 − s 4 + ε s 2 = ε s . \begin{aligned}
\int_{0}^{1}\Delta(x)f_{+}(x)\,dx=\left(\frac{1}{2}+\varepsilon\right)\frac{1}{2h}\int_{I_{+}}\Delta(x)\,dx+\left(\frac{1}{2}-\varepsilon\right)\frac{1}{2h}\int_{I_{-}}\Delta(x)\,dx \\
&=\left(\frac{1}{2}+\varepsilon\right)\frac{s}{2}+\left(\frac{1}{2}-\varepsilon\right)\left(-\frac{s}{2}\right) \\
&=\frac{s}{4}+\frac{\varepsilon s}{2}-\frac{s}{4}+\frac{\varepsilon s}{2} \\
&=\varepsilon s.
\end{aligned} ∫ 0 1 Δ ( x ) f + ( x ) d x = ( 2 1 + ε ) 2 h 1 ∫ I + Δ ( x ) d x + ( 2 1 − ε ) 2 h 1 ∫ I − Δ ( x ) d x = ( 2 1 + ε ) 2 s + ( 2 1 − ε ) ( − 2 s ) = 4 s + 2 ε s − 4 s + 2 ε s = ε s .
Thus, we have that
J g + ( p 1 ) − J g + ( p 0 ) = c 0 ε s / 2. (72) J_{g_{+}}(p^{1})-J_{g_{+}}(p^{0})=c_{0}\varepsilon s/2. \tag*{(72)} J g + ( p 1 ) − J g + ( p 0 ) = c 0 ε s /2. (72)
By a symmetry argument, we can also show
J g − ( p 0 ) − J g − ( p 1 ) = c 0 ε s / 2. J_{g_{-}}(p^{0})-J_{g_{-}}(p^{1})=c_{0}\varepsilon s/2. J g − ( p 0 ) − J g − ( p 1 ) = c 0 ε s /2.
Step 4. KL divergence
Conditional on the history and the selected policy, the action kernel and the conditional reward law given ( X t , A t ) (X_t,A_t) ( X t , A t ) are the same under both environments. Only the context density changes. The learner’s internal randomization also has the same conditional law. Therefore, the chain rule for KL divergence gives
KL ( P g + ∥ P g − ) = T KL ( g + ∥ g − ) . \operatorname{KL}(\mathbb{P}_{g_{+}}\Vert\mathbb{P}_{g_{-}})=T\operatorname{KL}(g_{+}\Vert g_{-}). KL ( P g + ∥ P g − ) = T KL ( g + ∥ g − ) .
Because g − ≥ 1 / 2 g_{-}\ge1/2 g − ≥ 1/2 ,
KL ( g + ∥ g − ) ≤ ∫ 0 1 ( g + g − − 1 ) g + d x ≤ ∫ 0 1 ( g + − g − ) 2 g − d x ≤ 1 2 ∫ 0 1 ( f + − f − ) 2 d x . \operatorname{KL}(g_{+}\Vert g_{-})\le\int_{0}^{1}\left(\frac{g_{+}}{g_{-}}-1\right)g_{+}\,dx\le\int_{0}^{1}\frac{(g_{+}-g_{-})^{2}}{g_{-}}\,dx\le\frac{1}{2}\int_{0}^{1}(f_{+}-f_{-})^{2}\,dx. KL ( g + ∥ g − ) ≤ ∫ 0 1 ( g − g + − 1 ) g + d x ≤ ∫ 0 1 g − ( g + − g − ) 2 d x ≤ 2 1 ∫ 0 1 ( f + − f − ) 2 d x .
On each of I + I_{+} I + and I − I_{-} I − , note that ∣ f + − f − ∣ = ε / h |f_{+}-f_{-}|=\varepsilon/h ∣ f + − f − ∣ = ε / h , and the total length of the interval that they are different is 4 h 4h 4 h . Hence, we have that
KL ( P g + ∥ P g − ) = T KL ( g + ∥ g − ) ≤ 2 T ε 2 / h . (73) \operatorname{KL}(\mathbb{P}_{g_{+}}\Vert\mathbb{P}_{g_{-}})=T\operatorname{KL}(g_{+}\Vert g_{-})\le2T\varepsilon^{2}/h. \tag*{(73)} KL ( P g + ∥ P g − ) = T KL ( g + ∥ g − ) ≤ 2 T ε 2 / h . (73)
Step 5. Reduction to regret. Set
A = { J g + ( p ˉ ) ≤ J g + ( p 0 ) + J g + ( p 1 ) 2 } . \mathcal{A}=\left\{J_{g_{+}}(\bar{p})\le\frac{J_{g_{+}}(p^{0})+J_{g_{+}}(p^{1})}{2}\right\}. A = { J g + ( p ˉ ) ≤ 2 J g + ( p 0 ) + J g + ( p 1 ) } .
On A \mathcal{A} A , by the optimality of p g + ⋆ p_{g_+}^{\star} p g + ⋆ and (72) , we have
J g + ( p g + ⋆ ) − J g + ( p ˉ ) ≥ J g + ( p 1 ) − J g + ( p ˉ ) ≥ 1 2 [ J g + ( p 1 ) − J g + ( p 0 ) ] ≥ c 0 ε s 4 . (74) J_{g_+}(p_{g_+}^{\star})-J_{g_+}(\bar{p}) \ge J_{g_+}(p^{1})-J_{g_+}(\bar{p}) \ge\frac{1}{2}\left[J_{g_+}(p^{1})-J_{g_+}(p^{0})\right] \ge\frac{c_0\varepsilon s}{4}. \tag*{(74)} J g + ( p g + ⋆ ) − J g + ( p ˉ ) ≥ J g + ( p 1 ) − J g + ( p ˉ ) ≥ 2 1 [ J g + ( p 1 ) − J g + ( p 0 ) ] ≥ 4 c 0 ε s . (74)
On A c \mathcal{A}^{c} A c , we have
J g + ( p ˉ ) > 1 2 ( J g + ( p 0 ) + J g + ( p 1 ) ) . (75) J_{g_+}(\bar{p})>\frac{1}{2}\left(J_{g_+}(p^{0})+J_{g_+}(p^{1})\right). \tag*{(75)} J g + ( p ˉ ) > 2 1 ( J g + ( p 0 ) + J g + ( p 1 ) ) . (75)
Moreover, by (72) , we have
1 2 [ J g + ( p 0 ) + J g + ( p 1 ) ] = J g + ( p 0 ) + 1 2 ( J g + ( p 1 ) − J g + ( p 0 ) ) = J g + ( p 0 ) + c 0 ε s 4 . (76) \frac{1}{2}\left[J_{g_+}(p^{0})+J_{g_+}(p^{1})\right]=J_{g_+}(p^{0})+\frac{1}{2}\left(J_{g_+}(p^{1})-J_{g_+}(p^{0})\right)=J_{g_+}(p^{0})+\frac{c_0\varepsilon s}{4}. \tag*{(76)} 2 1 [ J g + ( p 0 ) + J g + ( p 1 ) ] = J g + ( p 0 ) + 2 1 ( J g + ( p 1 ) − J g + ( p 0 ) ) = J g + ( p 0 ) + 4 c 0 ε s . (76)
Combining (75) and (76) yields that, on A c \mathcal{A}^{c} A c ,
J g + ( p ˉ ) − J g + ( p 0 ) > c 0 ε s 4 . (77) J_{g_+}(\bar{p})-J_{g_+}(p^{0})>\frac{c_0\varepsilon s}{4}. \tag*{(77)} J g + ( p ˉ ) − J g + ( p 0 ) > 4 c 0 ε s . (77)
Next, using the decomposition that for any p p p ,
J g 0 ( p ) = 1 2 ( J g + ( p ) + J g − ( p ) ) , (78) J_{g_0}(p)=\frac{1}{2}\left(J_{g_+}(p)+J_{g_-}(p)\right), \tag*{(78)} J g 0 ( p ) = 2 1 ( J g + ( p ) + J g − ( p ) ) , (78)
we have
J g + ( p ˉ ) − J g + ( p 0 ) = ( J g 0 ( p ˉ ) − J g 0 ( p 0 ) ) + 1 2 ( J g + ( p ˉ ) − J g − ( p ˉ ) − ( J g + ( p 0 ) − J g − ( p 0 ) ) ) . J_{g_+}(\bar{p})-J_{g_+}(p^{0})=\left(J_{g_0}(\bar{p})-J_{g_0}(p^{0})\right)+\frac{1}{2}\left(J_{g_+}(\bar{p})-J_{g_-}(\bar{p})-\left(J_{g_+}(p^{0})-J_{g_-}(p^{0})\right)\right). J g + ( p ˉ ) − J g + ( p 0 ) = ( J g 0 ( p ˉ ) − J g 0 ( p 0 ) ) + 2 1 ( J g + ( p ˉ ) − J g − ( p ˉ ) − ( J g + ( p 0 ) − J g − ( p 0 ) ) ) .
Since p 0 p^{0} p 0 optimizes on g 0 g_0 g 0 , we have J g 0 ( p ˉ ) − J g 0 ( p 0 ) ≤ 0 J_{g_0}(\bar{p})-J_{g_0}(p^{0})\le0 J g 0 ( p ˉ ) − J g 0 ( p 0 ) ≤ 0 , so
1 2 ( J g + ( p ˉ ) − J g − ( p ˉ ) − ( J g + ( p 0 ) − J g − ( p 0 ) ) ) ≥ J g + ( p ˉ ) − J g + ( p 0 ) , \frac{1}{2}\left(J_{g_+}(\bar{p})-J_{g_-}(\bar{p})-\left(J_{g_+}(p^{0})-J_{g_-}(p^{0})\right)\right)\ge J_{g_+}(\bar{p})-J_{g_+}(p^{0}), 2 1 ( J g + ( p ˉ ) − J g − ( p ˉ ) − ( J g + ( p 0 ) − J g − ( p 0 ) ) ) ≥ J g + ( p ˉ ) − J g + ( p 0 ) ,
i.e.
J g − ( p 0 ) − J g − ( p ˉ ) ≥ J g + ( p ˉ ) − J g + ( p 0 ) . J_{g_-}(p^{0})-J_{g_-}(\bar{p})\ge J_{g_+}(\bar{p})-J_{g_+}(p^{0}). J g − ( p 0 ) − J g − ( p ˉ ) ≥ J g + ( p ˉ ) − J g + ( p 0 ) .
Together with (77) , this implies on A c \mathcal{A}^{c} A c ,
J g − ( p 0 ) − J g − ( p ˉ ) > c 0 ε s 4 . (79) J_{g_-}(p^{0})-J_{g_-}(\bar{p})>\frac{c_0\varepsilon s}{4}. \tag*{(79)} J g − ( p 0 ) − J g − ( p ˉ ) > 4 c 0 ε s . (79)
Thus, combining (74) and (79) ,
Reg g + ( p ˉ , T ) + Reg g − ( p ˉ , T ) ≥ T c 0 ε s 4 { P g + ( A ) + P g − ( A c ) } . \operatorname{Reg}_{g_+}(\bar{p},T)+\operatorname{Reg}_{g_-}(\bar{p},T)\ge\frac{Tc_0\varepsilon s}{4}\left\{\mathbb{P}_{g_+}(\mathcal{A})+\mathbb{P}_{g_-}(\mathcal{A}^{c})\right\}. Reg g + ( p ˉ , T ) + Reg g − ( p ˉ , T ) ≥ 4 T c 0 ε s { P g + ( A ) + P g − ( A c ) } .
The Bretagnolle–Huber inequality gives
P g + ( A ) + P g − ( A c ) ≥ 1 2 exp { − KL ( P g + ∥ P g − ) } . \mathbb{P}_{g_+}(\mathcal{A})+\mathbb{P}_{g_-}(\mathcal{A}^{c})\ge\frac{1}{2}\exp\left\{-\operatorname{KL}(\mathbb{P}_{g_+}\Vert\mathbb{P}_{g_-})\right\}. P g + ( A ) + P g − ( A c ) ≥ 2 1 exp { − KL ( P g + ∥ P g − ) } .
Finally, choosing
ε = min { 1 8 , h 2 T } , \varepsilon=\min\left\{\frac{1}{8},\sqrt{\frac{h}{2T}}\right\}, ε = min { 8 1 , 2 T h } ,
since h = 1 / 64 h=1/64 h = 1/64 and T ≥ 1 T\ge1 T ≥ 1 , we have ε = h / ( 2 T ) \varepsilon=\sqrt{h/(2T)} ε = h / ( 2 T ) and, by (73) , KL ( P g + ∥ P g − ) ≤ 1 \operatorname{KL}(\mathbb{P}_{g_+}\Vert\mathbb{P}_{g_-})\le1 KL ( P g + ∥ P g − ) ≤ 1 . Consequently,
max g ∈ { g + , g − } Reg g ( p ˉ , T ) ≥ c 0 s 16 e h 2 T . \max_{g\in\{g_+,g_-\}}\operatorname{Reg}_{g}(\bar{p},T)\ge\frac{c_0s}{16e}\sqrt{\frac{h}{2}}\sqrt{T}. g ∈ { g + , g − } max Reg g ( p ˉ , T ) ≥ 16 e c 0 s 2 h T .
For d = 2 d=2 d = 2 , use the product densities g ± ( x 1 , x 2 ) = g ± ( x 1 ) g_{\pm}(x_{1},x_{2})=g_{\pm}(x_{1}) g ± ( x 1 , x 2 ) = g ± ( x 1 ) . For every q ∈ P L , 1 q\in\mathcal{P}_{L,1} q ∈ P L , 1 , the function
q ~ ( x 1 ) = ∫ 0 1 q ( x 1 , x 2 ) d x 2 \widetilde{q}(x_{1})=\int_{0}^{1}q(x_{1},x_{2})\,\mathrm{d}x_{2} q ( x 1 ) = ∫ 0 1 q ( x 1 , x 2 ) d x 2
is a one-dimensional L L L -Lipschitz policy with the same population value. Extend p 0 , p 1 p^{0},p^{1} p 0 , p 1 independently of x 2 x_{2} x 2 . Thus the baseline optimality and separation arguments remain valid, and the independent uniform second coordinate leaves the KL divergence unchanged. The same testing argument proves the bound for d = 2 d=2 d = 2 .
**(2) Proof for d ≥ 3. ∗ ∗ W e l e t d\geq3.** We let d ≥ 3. ∗ ∗ W e l e t u=(x_{2},x_{3},\ldots,x_{d})$, and
p 0 ( x 1 , u ) = 1 2 + L ( x 1 − 1 / 2 ) . p_{0}(x_{1},u)=\frac{1}{2}+L(x_{1}-1/2). p 0 ( x 1 , u ) = 2 1 + L ( x 1 − 1/2 ) .
Since L < 1 L<1 L < 1 , p 0 p_{0} p 0 takes values in [ ( 1 − L ) / 2 , ( 1 + L ) / 2 ] [(1-L)/2,(1+L)/2] [( 1 − L ) /2 , ( 1 + L ) /2 ] and belongs to P L , 1 \mathcal{P}_{L,1} P L , 1 .
Step 1. Packing in the boundary coordinates Fix h ∈ ( 0 , 1 / 64 ] h\in(0,1/64] h ∈ ( 0 , 1/64 ] and let M = ⌊ 1 / ( 32 h ) ⌋ M=\lfloor1/(32h)\rfloor M = ⌊ 1/ ( 32 h )⌋ . For each multi-index
k = ( k 1 , … , k d − 1 ) ∈ { 0 , 1 , … , M − 1 } d − 1 , k=(k_{1},\ldots,k_{d-1})\in\{0,1,\ldots,M-1\}^{d-1}, k = ( k 1 , … , k d − 1 ) ∈ { 0 , 1 , … , M − 1 } d − 1 ,
define the grid point as
a k = ( 16 h k 1 , … , 16 h k d − 1 ) + 1 / 4 , a_{k}=(16hk_{1},\ldots,16hk_{d-1})+1/4, a k = ( 16 h k 1 , … , 16 h k d − 1 ) + 1/4 ,
and the boundary cube as Q k = a k + [ 0 , h ] d − 1 Q_{k}=a_{k}+[0,h]^{d-1} Q k = a k + [ 0 , h ] d − 1 . Denote a k , r = 1 / 4 + 16 h k r a_{k,r}=1/4+16hk_{r} a k , r = 1/4 + 16 h k r , where r ∈ { 1 , … , d − 1 } r\in\{1,\ldots,d-1\} r ∈ { 1 , … , d − 1 } . Since
a k , r + h ≤ 1 4 + 16 h ( M − 1 ) + h ≤ 1 4 + 16 h M ≤ 3 4 , a_{k,r}+h\leq\frac{1}{4}+16h(M-1)+h\leq\frac{1}{4}+16hM\leq\frac{3}{4}, a k , r + h ≤ 4 1 + 16 h ( M − 1 ) + h ≤ 4 1 + 16 h M ≤ 4 3 ,
it holds that each Q k Q_{k} Q k is contained in [ 1 / 4 , 3 / 4 ] d − 1 [1/4,3/4]^{d-1} [ 1/4 , 3/4 ] d − 1 . If k ≠ k ′ k\neq k' k = k ′ , then for some coordinate r r r , we have that ∣ k r − k r ′ ∣ ≥ 1 |k_{r}-k'_{r}|\geq1 ∣ k r − k r ′ ∣ ≥ 1 . Hence the distance between the two cubes in coordinate r r r is at least 15 h 15h 15 h . Since the ℓ 1 \ell_{1} ℓ 1 distance between two points is at least the absolute difference in any one coordinate, we get dist 1 ( Q k , Q k ′ ) ≥ 15 h \operatorname{dist}_{1}(Q_{k},Q_{k'})\geq15h dist 1 ( Q k , Q k ′ ) ≥ 15 h . In particular, the cubes are disjoint, and the number of cubes is N = M d − 1 N=M^{d-1} N = M d − 1 . In particular, for h ≤ 1 / 64 h\leq1/64 h ≤ 1/64 , we have that
( 64 h ) − ( d − 1 ) ≤ N ≤ ( 32 h ) − ( d − 1 ) . (64h)^{-(d-1)}\leq N\leq(32h)^{-(d-1)}. ( 64 h ) − ( d − 1 ) ≤ N ≤ ( 32 h ) − ( d − 1 ) .
Inside each Q j Q_{j} Q j , j ∈ [ N ] j\in[N] j ∈ [ N ] , choose two subcubes A j A_{j} A j and B j B_{j} B j of side length ℓ = h / 16 \ell=h/16 ℓ = h /16 as follows. Let e 1 = ( 1 , 0 , … , 0 ) ∈ R d − 1 e_{1}=(1,0,\ldots,0)\in\mathbb{R}^{d-1} e 1 = ( 1 , 0 , … , 0 ) ∈ R d − 1 denote the first coordinate direction in the tangential variable u ∈ [ 0 , 1 ] d − 1 u\in[0,1]^{d-1} u ∈ [ 0 , 1 ] d − 1 , and set A j = a j + [ 0 , ℓ ] d − 1 A_{j}=a_{j}+[0,\ell]^{d-1} A j = a j + [ 0 , ℓ ] d − 1 and B j = A j + h e 1 / 2 B_{j}=A_{j}+he_{1}/2 B j = A j + h e 1 /2 . Thus B j B_{j} B j is obtained from A j A_{j} A j by shifting it by distance h / 2 h/2 h /2 in the first tangential coordinate. Since ℓ = h / 16 \ell=h/16 ℓ = h /16 , both subcubes are contained in Q j = a j + [ 0 , h ] d − 1 Q_{j}=a_{j}+[0,h]^{d-1} Q j = a j + [ 0 , h ] d − 1 . Moreover, ∣ A j ∣ = ∣ B j ∣ = ℓ d − 1 |A_{j}|=|B_{j}|=\ell^{d-1} ∣ A j ∣ = ∣ B j ∣ = ℓ d − 1 , and the separation between them in the first tangential coordinate is lower bounded by h / 4 h/4 h /4 , hence dist 1 ( A j , B j ) ≥ h / 4 \operatorname{dist}_{1}(A_{j},B_{j})\geq h/4 dist 1 ( A j , B j ) ≥ h /4 .
Define the signed bump
φ j ( u ) = 1 A j ( u ) − 1 B j ( u ) . \varphi_{j}(u)=\mathbf{1}_{A_{j}}(u)-\mathbf{1}_{B_{j}}(u). φ j ( u ) = 1 A j ( u ) − 1 B j ( u ) .
Note that the supports of φ j \varphi_{j} φ j are disjoint across j j j , and ∫ [ 0 , 1 ] d − 1 φ j ( u ) d u = 0 \int_{[0,1]^{d-1}}\varphi_{j}(u)\,\mathrm{d}u=0 ∫ [ 0 , 1 ] d − 1 φ j ( u ) d u = 0 .
Step 2. Density Construction. Fix α = 1 / 8 \alpha=1/8 α = 1/8 . For each θ = ( θ 1 , … , θ N ) ∈ { ± 1 } N \theta=(\theta_{1},\ldots,\theta_{N})\in\{\pm1\}^{N} θ = ( θ 1 , … , θ N ) ∈ { ± 1 } N , define
g θ ( x 1 , u ) = 1 + α ( x 1 − 1 / 2 ) ∑ j = 1 N θ j φ j ( u ) . g_{\theta}(x_{1},u)=1+\alpha(x_{1}-1/2)\sum_{j=1}^{N}\theta_{j}\varphi_{j}(u). g θ ( x 1 , u ) = 1 + α ( x 1 − 1/2 ) j = 1 ∑ N θ j φ j ( u ) .
Then, it is easy to see that for each θ \theta θ , g θ g_{\theta} g θ is a density on X \mathcal{X} X that is upper and lower bounded. Indeed, for every u u u , at most one φ j ( u ) \varphi_{j}(u) φ j ( u ) is nonzero, and its value is in { − 1 , 0 , 1 } \{-1,0,1\} { − 1 , 0 , 1 } . Since ∣ x 1 − 1 / 2 ∣ ≤ 1 / 2 |x_{1}-1/2|\leq1/2 ∣ x 1 − 1/2∣ ≤ 1/2 and α = 1 / 8 \alpha=1/8 α = 1/8 , we have
∣ g θ ( x 1 , u ) − 1 ∣ = ∣ α ( x 1 − 1 2 ) ∑ j = 1 N θ j φ j ( u ) ∣ ≤ α ∣ x 1 − 1 2 ∣ ∣ ∑ j = 1 N θ j φ j ( u ) ∣ ≤ 1 16 . \left|g_{\theta}(x_{1},u)-1\right|
=\left|\alpha\left(x_{1}-\frac{1}{2}\right)\sum_{j=1}^{N}\theta_{j}\varphi_{j}(u)\right|
\leq\alpha\left|x_{1}-\frac{1}{2}\right|\left|\sum_{j=1}^{N}\theta_{j}\varphi_{j}(u)\right|
\leq\frac{1}{16}. ∣ g θ ( x 1 , u ) − 1 ∣ = α ( x 1 − 2 1 ) j = 1 ∑ N θ j φ j ( u ) ≤ α x 1 − 2 1 j = 1 ∑ N θ j φ j ( u ) ≤ 16 1 .
Therefore, it holds that
15 16 ≤ g θ ( x 1 , u ) ≤ 17 16 . \frac{15}{16} \le g_\theta(x_1,u) \le\frac{17}{16}. 16 15 ≤ g θ ( x 1 , u ) ≤ 16 17 .
It remains to check that g θ g_\theta g θ integrates to one. By the definition of g θ g_\theta g θ , we have that
∫ [ 0 , 1 ] d g θ ( x 1 , u ) d x 1 d u = ∫ [ 0 , 1 ] d { 1 + α ( x 1 − 1 2 ) ∑ j = 1 N θ j φ j ( u ) } d x 1 d u = 1 + α ∑ j = 1 N θ j ∫ 0 1 ( x 1 − 1 2 ) d x 1 ∫ [ 0 , 1 ] d − 1 φ j ( u ) d u = 1. \begin{aligned}
\int_{[0,1]^d} g_\theta(x_1,u)\,\mathrm{d}x_1\,\mathrm{d}u
&= \int_{[0,1]^d} \left\{1+\alpha\left(x_1-\frac{1}{2}\right)\sum_{j=1}^{N}\theta_j\varphi_j(u)\right\}\,\mathrm{d}x_1\,\mathrm{d}u \\
&= 1+\alpha\sum_{j=1}^{N}\theta_j\int_0^1\left(x_1-\frac{1}{2}\right)\,\mathrm{d}x_1\int_{[0,1]^{d-1}}\varphi_j(u)\,\mathrm{d}u \\
&=1.
\end{aligned} ∫ [ 0 , 1 ] d g θ ( x 1 , u ) d x 1 d u = ∫ [ 0 , 1 ] d { 1 + α ( x 1 − 2 1 ) j = 1 ∑ N θ j φ j ( u ) } d x 1 d u = 1 + α j = 1 ∑ N θ j ∫ 0 1 ( x 1 − 2 1 ) d x 1 ∫ [ 0 , 1 ] d − 1 φ j ( u ) d u = 1.
Thus each g θ g_\theta g θ is a density on [ 0 , 1 ] d [0,1]^d [ 0 , 1 ] d .
Step 3. Optimal Policies.
Assume additionally that h ≤ ( 1 − L ) / L h \le(1-L)/L h ≤ ( 1 − L ) / L . For each j ∈ [ N ] j \in[N] j ∈ [ N ] , define u j : [ 0 , 1 ] d − 1 → R u_j:[0,1]^{d-1}\to\mathbb{R} u j : [ 0 , 1 ] d − 1 → R as follows. For each u = ( x 2 , x 3 , … , x d ) ⊤ u=(x_2,x_3,\ldots,x_d)^\top u = ( x 2 , x 3 , … , x d ) ⊤ , if dist 1 ( u , Q j ) ≥ h / 2 \operatorname{dist}_1(u,Q_j)\ge h/2 dist 1 ( u , Q j ) ≥ h /2 , set u j ( u ) = 0 u_j(u)=0 u j ( u ) = 0 ; if dist 1 ( u , Q j ) < h / 2 \operatorname{dist}_1(u,Q_j)<h/2 dist 1 ( u , Q j ) < h /2 , set
u j ( u ) = { L ( h / 2 − dist 1 ( u , Q j ) ) , x 2 < a j , 1 + dist 1 ( u , Q j ) , L ( dist 1 ( u , Q j ) − h / 2 ) , x 2 > a j , 1 + h − dist 1 ( u , Q j ) , L ( a j , 1 + h / 2 − x 2 ) , otherwise . u_j(u)=
\begin{cases}
L\left(h/2-\operatorname{dist}_1(u,Q_j)\right), & x_2<a_{j,1}+\operatorname{dist}_1(u,Q_j),\\
L\left(\operatorname{dist}_1(u,Q_j)-h/2\right), & x_2>a_{j,1}+h-\operatorname{dist}_1(u,Q_j),\\
L\left(a_{j,1}+h/2-x_2\right), & \text{otherwise}.
\end{cases} u j ( u ) = ⎩ ⎨ ⎧ L ( h /2 − dist 1 ( u , Q j ) ) , L ( dist 1 ( u , Q j ) − h /2 ) , L ( a j , 1 + h /2 − x 2 ) , x 2 < a j , 1 + dist 1 ( u , Q j ) , x 2 > a j , 1 + h − dist 1 ( u , Q j ) , otherwise .
Here we recall that a j , 1 = 1 / 4 + 16 h j 1 a_{j,1}=1/4+16hj_1 a j , 1 = 1/4 + 16 h j 1 is the first coordinate of the lower-left corner a j a_j a j of the cube Q j Q_j Q j .
Intuitively, u j u_j u j is a clipped affine function that retains slope − L -L − L in the direction of x 2 x_2 x 2 on Q j Q_j Q j and vanishes away from Q j Q_j Q j , thereby attaining the largest paired difference between A j A_j A j and B j B_j B j allowed by the Lipschitz constraint. Figure 8 visualizes the construction.
Figure 8. Localized perturbation used in the lower-bound construction. Panel (a) shows a slice through the centres of A j A_j A j and B j = A j + ( h / 2 ) e 1 B_j=A_j+(h/2)e_1 B j = A j + ( h /2 ) e 1 . Within Q j Q_j Q j , the function u j u_j u j is affine with slope − L -L − L in x 2 x_2 x 2 , so every paired difference u j ( u ) − u j ( u + ( h / 2 ) e 1 ) u_j(u)-u_j(u+(h/2)e_1) u j ( u ) − u j ( u + ( h /2 ) e 1 ) , u ∈ A j u\in A_j u ∈ A j , equals L h / 2 Lh/2 L h /2 , attaining the Lipschitz bound. Outside Q j Q_j Q j , the perturbation tapers to zero and vanishes when dist 1 ( u , Q j ) ≥ h / 2 \operatorname{dist}_1(u,Q_j)\ge h/2 dist 1 ( u , Q j ) ≥ h /2 . Panel (b) displays the perturbation for d = 3 d=3 d = 3 , with red and blue denoting positive and negative values. The solid square is Q j Q_j Q j , the outer dashed outline bounds its h / 2 h/2 h /2 -neighbourhood in the ℓ 1 \ell_1 ℓ 1 metric, and the horizontal dashed line indicates the slice in panel (a).
Now set
U θ ( u ) = ∑ j = 1 N θ j u j ( u ) , p θ ∘ ( x 1 , u ) = p 0 ( x 1 , u ) + U θ ( u ) . U_\theta(u)=\sum_{j=1}^{N}\theta_j u_j(u),\qquad p^\circ_\theta(x_1,u)=p_0(x_1,u)+U_\theta(u). U θ ( u ) = j = 1 ∑ N θ j u j ( u ) , p θ ∘ ( x 1 , u ) = p 0 ( x 1 , u ) + U θ ( u ) .
By Lemma 53, p θ ∘ p^\circ_\theta p θ ∘ is feasible and maximizes J 1 J_1 J 1 . By Lemma 55 , it also maximizes J g θ J_{g_\theta} J g θ .
Step 4. Reduction of regret to bit errors.
For q ∈ P L , 1 q\in\mathcal{P}_{L,1} q ∈ P L , 1 and j ∈ [ N ] j\in[N] j ∈ [ N ] , define
S j ( q ) = ∫ [ 0 , 1 ] d ( x 1 − 1 / 2 ) 2 φ j ( u ) q ( x 1 , u ) d x 1 d u , S_j(q)=\int_{[0,1]^d}(x_1-1/2)^2\varphi_j(u)q(x_1,u)\,dx_1\,du, S j ( q ) = ∫ [ 0 , 1 ] d ( x 1 − 1/2 ) 2 φ j ( u ) q ( x 1 , u ) d x 1 d u ,
and the plug-in bit estimator
θ ^ j ( q ) = { 1 , S j ( q ) ≥ 0 , − 1 , S j ( q ) < 0. \widehat{\theta}_j(q)=
\begin{cases}
1, & S_j(q)\ge0,\\
-1, & S_j(q)<0.
\end{cases} θ j ( q ) = { 1 , − 1 , S j ( q ) ≥ 0 , S j ( q ) < 0.
In Lemma 55 , we reduce the regret to classification error, and we have that for every θ ∈ { ± 1 } N \theta\in\{\pm1\}^N θ ∈ { ± 1 } N and every q ∈ P L , 1 q\in\mathcal{P}_{L,1} q ∈ P L , 1 ,
J g θ ( p θ ∘ ) − J g θ ( q ) ≥ c 2 α L h d ∑ j = 1 N 1 { θ ^ j ( q ) ≠ θ j } . J_{g_\theta}(p^\circ_\theta)-J_{g_\theta}(q)\ge c_2\alpha Lh^d\sum_{j=1}^{N}\mathbf{1}\{\widehat{\theta}_j(q)\ne\theta_j\}. J g θ ( p θ ∘ ) − J g θ ( q ) ≥ c 2 α L h d j = 1 ∑ N 1 { θ j ( q ) = θ j } .
Step 5. Neighbouring KL divergence.
Let θ ( j ) \theta^{(j)} θ ( j ) be the vector obtained from θ \theta θ by flipping the j j j th coordinate. Let P θ \mathbb{P}_\theta P θ be the joint law of the full interaction, including the learner’s internal randomization, under g θ g_\theta g θ . Then by Lemma 56 , there exists C K L > 0 C_{\mathrm{KL}}>0 C KL > 0 , depending only on d d d , such that for every θ ∈ { ± 1 } N \theta\in\{\pm1\}^N θ ∈ { ± 1 } N and j ∈ [ N ] j\in[N] j ∈ [ N ] ,
KL ( P θ ∥ P θ ( j ) ) ≤ C K L T α 2 h d − 1 . \operatorname{KL}(\mathbb{P}_\theta\Vert\mathbb{P}_{\theta^{(j)}})\le C_{\mathrm{KL}}T\alpha^2h^{d-1}. KL ( P θ ∥ P θ ( j ) ) ≤ C KL T α 2 h d − 1 .
Step 6. Bounding the classification error by Assouad’s method
For a fixed θ ∈ { ± 1 } N \theta\in\{\pm1\}^N θ ∈ { ± 1 } N ,
Reg g θ ( p ˉ , T ) = T E θ { J g θ ( p g θ ⋆ ) − J g θ ( p ˉ ) } ≥ T E θ { J g θ ( p θ ∘ ) − J g θ ( p ˉ ) } . \operatorname{Reg}_{g_\theta}(\bar p,T)=T\mathbb{E}_\theta\{J_{g_\theta}(p^\star_{g_\theta})-J_{g_\theta}(\bar p)\}\ge T\mathbb{E}_\theta\{J_{g_\theta}(p^\circ_\theta)-J_{g_\theta}(\bar p)\}. Reg g θ ( p ˉ , T ) = T E θ { J g θ ( p g θ ⋆ ) − J g θ ( p ˉ )} ≥ T E θ { J g θ ( p θ ∘ ) − J g θ ( p ˉ )} .
Applying Lemma 55 with q = p ˉ q=\bar p q = p ˉ , we have
Reg g θ ( p ˉ , T ) ≥ T c 2 α L h d ∑ j = 1 N P θ { θ ^ j ( p ˉ ) ≠ θ j } . \operatorname{Reg}_{g_\theta}(\bar p,T)\ge Tc_2\alpha Lh^d\sum_{j=1}^{N}\mathbb{P}_\theta\{\widehat{\theta}_j(\bar p)\ne\theta_j\}. Reg g θ ( p ˉ , T ) ≥ T c 2 α L h d j = 1 ∑ N P θ { θ j ( p ˉ ) = θ j } .
Average this inequality over θ \theta θ gives
1 2 N ∑ θ ∈ { ± 1 } N Reg g θ ( p ˉ , T ) ≥ T c 2 α L h d ∑ j = 1 N 1 2 N ∑ θ P θ { θ ^ j ( p ˉ ) ≠ θ j } . (80) \frac{1}{2^N}\sum_{\theta\in\{\pm1\}^N}\operatorname{Reg}_{g_\theta}(\bar p,T)\ge Tc_2\alpha Lh^d\sum_{j=1}^{N}\frac{1}{2^N}\sum_{\theta}\mathbb{P}_\theta\{\widehat{\theta}_j(\bar p)\ne\theta_j\}. \tag*{(80)} 2 N 1 θ ∈ { ± 1 } N ∑ Reg g θ ( p ˉ , T ) ≥ T c 2 α L h d j = 1 ∑ N 2 N 1 θ ∑ P θ { θ j ( p ˉ ) = θ j } . (80)
Now, we choose
h 0 = min { 1 64 , 1 − L L } , h = h 0 T − 1 / ( d − 1 ) . h_0=\min\left\{\frac{1}{64},\frac{1-L}{L}\right\},\qquad h=h_0T^{-1/(d-1)}. h 0 = min { 64 1 , L 1 − L } , h = h 0 T − 1/ ( d − 1 ) .
By Lemma 56 , with C K L = 32 / ( 45 ⋅ 16 d − 1 ) C_{\mathrm{KL}}=32/(45\cdot16^{d-1}) C KL = 32/ ( 45 ⋅ 1 6 d − 1 ) and α = 1 / 8 \alpha=1/8 α = 1/8 ,
KL ( P θ ∥ P θ ( j ) ) ≤ C K L α 2 h 0 d − 1 = h 0 d − 1 90 ⋅ 16 d − 1 ≤ 1. \operatorname{KL}(\mathbb{P}_\theta\Vert\mathbb{P}_{\theta^{(j)}})\le C_{\mathrm{KL}}\alpha^2h_0^{d-1}=\frac{h_0^{d-1}}{90\cdot16^{d-1}}\le1. KL ( P θ ∥ P θ ( j ) ) ≤ C KL α 2 h 0 d − 1 = 90 ⋅ 1 6 d − 1 h 0 d − 1 ≤ 1.
For each j j j , pair every θ \theta θ with θ j = 1 \theta_j=1 θ j = 1 with θ ( j ) \theta^{(j)} θ ( j ) , and let E j = { θ ^ j ( p ˉ ) = 1 } E_j=\{\hat{\theta}_j(\bar{p})=1\} E j = { θ ^ j ( p ˉ ) = 1 } . For each such pair,
P θ { θ ^ j ( p ˉ ) ≠ θ j } + P θ ( j ) { θ ^ j ( p ˉ ) ≠ θ j ( j ) } = P θ ( E j c ) + P θ ( j ) ( E j ) ≥ 1 2 exp { − KL ( P θ ∥ P θ ( j ) ) } ≥ 1 2 e , \begin{aligned}
\mathbb{P}_{\theta}\{\hat{\theta}_j(\bar{p})\ne\theta_j\}+\mathbb{P}_{\theta^{(j)}}\{\hat{\theta}_j(\bar{p})\ne\theta^{(j)}_j\} \\
&=\mathbb{P}_{\theta}(E_j^c)+\mathbb{P}_{\theta^{(j)}}(E_j)\ge\frac{1}{2}\exp\{-\operatorname{KL}(\mathbb{P}_{\theta}\Vert\mathbb{P}_{\theta^{(j)}})\}\ge\frac{1}{2e},
\end{aligned} P θ { θ ^ j ( p ˉ ) = θ j } + P θ ( j ) { θ ^ j ( p ˉ ) = θ j ( j ) } = P θ ( E j c ) + P θ ( j ) ( E j ) ≥ 2 1 exp { − KL ( P θ ∥ P θ ( j ) )} ≥ 2 e 1 ,
where the first inequality is the Bretagnolle–Huber inequality. There are 2 N − 1 2^{N-1} 2 N − 1 such pairs, so
1 2 N ∑ θ P θ { θ ^ j ( p ˉ ) ≠ θ j } ≥ 1 4 e > 1 12 . \frac{1}{2^N}\sum_{\theta}\mathbb{P}_{\theta}\{\hat{\theta}_j(\bar{p})\ne\theta_j\}\ge\frac{1}{4e}>\frac{1}{12}. 2 N 1 θ ∑ P θ { θ ^ j ( p ˉ ) = θ j } ≥ 4 e 1 > 12 1 .
Substituting this into (80) and using N ≥ ( 64 h ) − ( d − 1 ) N\ge(64h)^{-(d-1)} N ≥ ( 64 h ) − ( d − 1 ) gives
1 2 N ∑ θ Reg g θ ( p ˉ , T ) ≥ T c 2 α L h d N 12 ≥ c 2 α L 12 ⋅ 64 d − 1 T h = c 2 α L h 0 12 ⋅ 64 d − 1 T ( d − 2 ) / ( d − 1 ) . \frac{1}{2^N}\sum_{\theta}\operatorname{Reg}_{g_\theta}(\bar{p},T)\ge\frac{Tc_2\alpha Lh^dN}{12}\ge\frac{c_2\alpha L}{12\cdot64^{d-1}}Th=\frac{c_2\alpha Lh_0}{12\cdot64^{d-1}}T^{(d-2)/(d-1)}. 2 N 1 θ ∑ Reg g θ ( p ˉ , T ) ≥ 12 T c 2 α L h d N ≥ 12 ⋅ 6 4 d − 1 c 2 α L T h = 12 ⋅ 6 4 d − 1 c 2 α L h 0 T ( d − 2 ) / ( d − 1 ) .
Thus, for every learner and every integer T ≥ 1 T\ge1 T ≥ 1 , at least one density in the construction has regret of the claimed order. Taking the infimum over learners completes the proof.
Auxiliary results for lower bound construction
Lemma 52 (Uniform-density optimizer). For every q ∈ P L , 1 q\in\mathcal{P}_{L,1} q ∈ P L , 1 , we have that
J 1 ( q ) : = ∫ [ 0 , 1 ] d ( x 1 − 1 / 2 ) q ( x 1 , u ) d x 1 d u ≤ J 1 ( p 0 ) . J_1(q):=\int_{[0,1]^d}(x_1-1/2)q(x_1,u)\,dx_1\,du\le J_1(p_0). J 1 ( q ) := ∫ [ 0 , 1 ] d ( x 1 − 1/2 ) q ( x 1 , u ) d x 1 d u ≤ J 1 ( p 0 ) . Moreover, if U : [ 0 , 1 ] d − 1 ↦ R U:[0,1]^{d-1}\mapsto\mathbb{R} U : [ 0 , 1 ] d − 1 ↦ R is measurable and p 0 ( x 1 , u ) + U ( u ) ∈ [ 0 , 1 ] p_0(x_1,u)+U(u)\in[0,1] p 0 ( x 1 , u ) + U ( u ) ∈ [ 0 , 1 ] , then
J 1 ( p 0 + U ) = J 1 ( p 0 ) . J_1(p_0+U)=J_1(p_0). J 1 ( p 0 + U ) = J 1 ( p 0 ) .
Proof. For each fixed u ∈ [ 0 , 1 ] d − 1 u\in[0,1]^{d-1} u ∈ [ 0 , 1 ] d − 1 , we pair every point ( x 1 , u ) (x_1,u) ( x 1 , u ) with its reflection ( 1 − x 1 , u ) (1-x_1,u) ( 1 − x 1 , u ) across the hyperplane x 1 = 1 / 2 x_1=1/2 x 1 = 1/2 . Since Δ ( x 1 , u ) = x 1 − 1 / 2 \Delta(x_1,u)=x_1-1/2 Δ ( x 1 , u ) = x 1 − 1/2 , this reflection changes the sign of the reward gap:
Δ ( 1 − x 1 , u ) = 1 − x 1 − 1 2 = − ( x 1 − 1 2 ) = − Δ ( x 1 , u ) . \Delta(1-x_1,u)=1-x_1-\frac{1}{2}=-\left(x_1-\frac{1}{2}\right)=-\Delta(x_1,u). Δ ( 1 − x 1 , u ) = 1 − x 1 − 2 1 = − ( x 1 − 2 1 ) = − Δ ( x 1 , u ) .
Therefore, for any fixed u u u and q ∈ P L , 1 q\in\mathcal{P}_{L,1} q ∈ P L , 1 , we have that
∫ 0 1 Δ ( x ) q ( x 1 , u ) d x 1 = ∫ 0 1 / 2 ( x 1 − 1 2 ) q ( x 1 , u ) d x 1 + ∫ 1 / 2 1 ( x 1 − 1 2 ) q ( x 1 , u ) d x 1 = − ∫ 1 / 2 1 ( x 1 − 1 2 ) q ( 1 − x 1 , u ) d x 1 + ∫ 1 / 2 1 ( x 1 − 1 2 ) q ( x 1 , u ) d x 1 = ∫ 1 / 2 1 ( x 1 − 1 2 ) { q ( x 1 , u ) − q ( 1 − x 1 , u ) } d x 1 . \begin{aligned}
\int_0^1\Delta(x)q(x_1,u)\,dx_1 \\
&=\int_0^{1/2}\left(x_1-\frac{1}{2}\right)q(x_1,u)\,dx_1+\int_{1/2}^1\left(x_1-\frac{1}{2}\right)q(x_1,u)\,dx_1 \\
&=-\int_{1/2}^1\left(x_1-\frac{1}{2}\right)q(1-x_1,u)\,dx_1+\int_{1/2}^1\left(x_1-\frac{1}{2}\right)q(x_1,u)\,dx_1 \\
&=\int_{1/2}^1\left(x_1-\frac{1}{2}\right)\{q(x_1,u)-q(1-x_1,u)\}\,dx_1.
\end{aligned} ∫ 0 1 Δ ( x ) q ( x 1 , u ) d x 1 = ∫ 0 1/2 ( x 1 − 2 1 ) q ( x 1 , u ) d x 1 + ∫ 1/2 1 ( x 1 − 2 1 ) q ( x 1 , u ) d x 1 = − ∫ 1/2 1 ( x 1 − 2 1 ) q ( 1 − x 1 , u ) d x 1 + ∫ 1/2 1 ( x 1 − 2 1 ) q ( x 1 , u ) d x 1 = ∫ 1/2 1 ( x 1 − 2 1 ) { q ( x 1 , u ) − q ( 1 − x 1 , u )} d x 1 .
For x 1 ≥ 1 / 2 x_1\ge1/2 x 1 ≥ 1/2 , we have that x 1 − 1 / 2 ≥ 0 x_1-1/2\ge0 x 1 − 1/2 ≥ 0 , and
q ( x 1 , u ) − q ( 1 − x 1 , u ) ≤ ∣ q ( x 1 , u ) − q ( 1 − x 1 , u ) ∣ ≤ L ∥ ( x 1 , u ) − ( 1 − x 1 , u ) ∥ 1 = 2 L ( x 1 − 1 / 2 ) . q(x_1,u)-q(1-x_1,u)\le\lvert q(x_1,u)-q(1-x_1,u)\rvert\le L\lVert(x_1,u)-(1-x_1,u)\rVert_1=2L(x_1-1/2). q ( x 1 , u ) − q ( 1 − x 1 , u ) ≤ ∣ q ( x 1 , u ) − q ( 1 − x 1 , u )∣ ≤ L ∥( x 1 , u ) − ( 1 − x 1 , u ) ∥ 1 = 2 L ( x 1 − 1/2 ) .
Therefore we have that
∫ 0 1 Δ ( x ) q ( x 1 , u ) d x 1 ≤ ∫ 1 / 2 1 2 L ( x 1 − 1 / 2 ) 2 d x 1 = ∫ 0 1 L ( x 1 − 1 / 2 ) 2 d x 1 . \int_0^1 \Delta(x)q(x_1,u)\,\mathrm{d}x_1 \le\int_{1/2}^1 2L(x_1-1/2)^2\,\mathrm{d}x_1 = \int_0^1 L(x_1-1/2)^2\,\mathrm{d}x_1. ∫ 0 1 Δ ( x ) q ( x 1 , u ) d x 1 ≤ ∫ 1/2 1 2 L ( x 1 − 1/2 ) 2 d x 1 = ∫ 0 1 L ( x 1 − 1/2 ) 2 d x 1 .
Integrating over u u u gives
J 1 ( q ) ≤ L ∫ 0 1 ( x 1 − 1 / 2 ) 2 d x 1 = J 1 ( p 0 ) , J_1(q) \le L\int_0^1 (x_1-1/2)^2\,\mathrm{d}x_1 = J_1(p_0), J 1 ( q ) ≤ L ∫ 0 1 ( x 1 − 1/2 ) 2 d x 1 = J 1 ( p 0 ) ,
because p 0 ( x 1 , u ) = 1 / 2 + L ( x 1 − 1 / 2 ) p_0(x_1,u)=1/2+L(x_1-1/2) p 0 ( x 1 , u ) = 1/2 + L ( x 1 − 1/2 ) and ∫ 0 1 ( x 1 − 1 / 2 ) d x 1 = 0 \int_0^1(x_1-1/2)\,\mathrm{d}x_1=0 ∫ 0 1 ( x 1 − 1/2 ) d x 1 = 0 .
For the second claim, it is easy to see that
J 1 ( p 0 + U ) − J 1 ( p 0 ) = ( ∫ 0 1 ( x 1 − 1 / 2 ) d x 1 ) ( ∫ [ 0 , 1 ] d − 1 U ( u ) d u ) = 0. J_1(p_0+U)-J_1(p_0)=\left(\int_0^1(x_1-1/2)\,\mathrm{d}x_1\right)\left(\int_{[0,1]^{d-1}}U(u)\,\mathrm{d}u\right)=0. J 1 ( p 0 + U ) − J 1 ( p 0 ) = ( ∫ 0 1 ( x 1 − 1/2 ) d x 1 ) ( ∫ [ 0 , 1 ] d − 1 U ( u ) d u ) = 0.
Lemma 53 (Feasibility and Optimality). For every θ \theta θ , p θ ∘ ∈ P L , 1 p^\circ_\theta\in\mathcal{P}_{L,1} p θ ∘ ∈ P L , 1 and
J 1 ( p θ ∘ ) = J 1 ( p 0 ) = sup q ∈ P L , 1 J 1 ( q ) . J_1(p^\circ_\theta)=J_1(p_0)=\sup_{q\in\mathcal{P}_{L,1}}J_1(q). J 1 ( p θ ∘ ) = J 1 ( p 0 ) = q ∈ P L , 1 sup J 1 ( q ) . Proof. We first prove that U θ U_\theta U θ is L L L -Lipschitz. The piecewise definition in Step 3 is equivalent to clipping L ( a j , 1 + h / 2 − x 2 ) L(a_{j,1}+h/2-x_2) L ( a j , 1 + h /2 − x 2 ) to the interval
[ − ( L h / 2 − L dist 1 ( u , Q j ) ) + , ( L h / 2 − L dist 1 ( u , Q j ) ) + ] , \left[-(Lh/2-L\operatorname{dist}_1(u,Q_j))_+,\,(Lh/2-L\operatorname{dist}_1(u,Q_j))_+\right], [ − ( L h /2 − L dist 1 ( u , Q j ) ) + , ( L h /2 − L dist 1 ( u , Q j ) ) + ] , where t + = max { t , 0 } t_+=\max\{t,0\} t + = max { t , 0 } . Since u ↦ dist 1 ( u , Q j ) u\mapsto\operatorname{dist}_1(u,Q_j) u ↦ dist 1 ( u , Q j ) is 1-Lipschitz, the affine function and both clipping bounds are L L L -Lipschitz. Clipping is a composition of a pointwise minimum and maximum, each of which preserves a common Lipschitz bound. Thus each u j u_j u j is L L L -Lipschitz.
Moreover, ∥ u j ∥ ∞ ≤ L h / 2 \lVert u_j\rVert_\infty\le Lh/2 ∥ u j ∥ ∞ ≤ L h /2 , and its support is contained in the h / 2 h/2 h /2 -neighbourhood of Q j Q_j Q j . Thus, for j ≠ j ′ j\ne j' j = j ′ ,
dist 1 { supp ( u j ) , supp ( u j ′ ) } ≥ 15 h − h = 14 h . \operatorname{dist}_1\{\operatorname{supp}(u_j),\operatorname{supp}(u_{j'})\}\ge15h-h=14h. dist 1 { supp ( u j ) , supp ( u j ′ )} ≥ 15 h − h = 14 h . In particular, at most one u j u_j u j is nonzero at any point. If u u u and v v v belong to different supports, then
∣ U θ ( u ) − U θ ( v ) ∣ ≤ L h ≤ L ∥ u − v ∥ 1 . |U_\theta(u)-U_\theta(v)|\le Lh\le L\lVert u-v\rVert_1. ∣ U θ ( u ) − U θ ( v ) ∣ ≤ L h ≤ L ∥ u − v ∥ 1 . If they belong to the same support, or one lies outside all supports, the same inequality follows from the Lipschitz property of the corresponding u j u_j u j . If both lie outside all supports, the difference is zero. Therefore U θ U_\theta U θ is L L L -Lipschitz.
Recall that p 0 p_0 p 0 is L L L -Lipschitz in the x 1 x_1 x 1 coordinate. For any ( x 1 , u ) , ( x 1 ′ , v ) (x_1,u),(x'_1,v) ( x 1 , u ) , ( x 1 ′ , v ) ,
∣ p θ ∘ ( x 1 , u ) − p θ ∘ ( x 1 ′ , v ) ∣ ≤ L ∣ x 1 − x 1 ′ ∣ + L ∥ u − v ∥ 1 = L ∥ ( x 1 , u ) − ( x 1 ′ , v ) ∥ 1 . |p^\circ_\theta(x_1,u)-p^\circ_\theta(x'_1,v)|\le L|x_1-x'_1|+L\lVert u-v\rVert_1=L\lVert(x_1,u)-(x'_1,v)\rVert_1. ∣ p θ ∘ ( x 1 , u ) − p θ ∘ ( x 1 ′ , v ) ∣ ≤ L ∣ x 1 − x 1 ′ ∣ + L ∥ u − v ∥ 1 = L ∥( x 1 , u ) − ( x 1 ′ , v ) ∥ 1 . Moreover, since the supports are disjoint,
∥ U θ ∥ ∞ ≤ L h 2 ≤ 1 − L 2 . \lVert U_\theta\rVert_\infty\le\frac{Lh}{2}\le\frac{1-L}{2}. ∥ U θ ∥ ∞ ≤ 2 L h ≤ 2 1 − L . Since p 0 p_0 p 0 is at distance at least ( 1 − L ) / 2 (1-L)/2 ( 1 − L ) /2 from the endpoints 0 and 1, p θ ∘ p^\circ_\theta p θ ∘ takes values in [ 0 , 1 ] [0,1] [ 0 , 1 ] . Hence p θ ∘ ∈ P L , 1 p^\circ_\theta\in\mathcal{P}_{L,1} p θ ∘ ∈ P L , 1 .
Finally, Lemma 52 gives J 1 ( p θ ∘ ) = J 1 ( p 0 ) = sup q ∈ P L , 1 J 1 ( q ) J_1(p^\circ_\theta)=J_1(p_0)=\sup_{q\in\mathcal{P}_{L,1}}J_1(q) J 1 ( p θ ∘ ) = J 1 ( p 0 ) = sup q ∈ P L , 1 J 1 ( q ) .
Lemma 54 (One-bit signal). There is a constant c 1 > 0 c_1 > 0 c 1 > 0 , depending only on d d d , such that for every θ ∈ { ± 1 } N \theta\in\{\pm1\}^{N} θ ∈ { ± 1 } N and every j ∈ [ N ] j \in[N] j ∈ [ N ] ,
θ j S j ( p θ ∘ ) = c 1 L h d . \theta_j S_j(p_\theta^\circ) = c_1 Lh^d. θ j S j ( p θ ∘ ) = c 1 L h d . Proof. Since ∫ φ j ( u ) d u = 0 \int\varphi_j(u)\,\mathrm{d}u = 0 ∫ φ j ( u ) d u = 0 and p 0 p_0 p 0 depends only on x 1 x_1 x 1 , we have S j ( p 0 ) = 0 S_j(p_0) = 0 S j ( p 0 ) = 0 . Therefore
S j ( p θ ∘ ) = ∫ [ 0 , 1 ] d ( x 1 − 1 / 2 ) 2 φ j ( u ) U θ ( u ) d x 1 d u . S_j(p_\theta^\circ) = \int_{[0,1]^d} (x_1 - 1/2)^2 \varphi_j(u)U_\theta(u)\,\mathrm{d}x_1\,\mathrm{d}u. S j ( p θ ∘ ) = ∫ [ 0 , 1 ] d ( x 1 − 1/2 ) 2 φ j ( u ) U θ ( u ) d x 1 d u . For u ∈ Q j u \in Q_j u ∈ Q j , note that
∣ L ( a j , 1 + h / 2 − x 2 ) ∣ ≤ L h / 2. \lvert L(a_{j,1} + h/2 - x_2)\rvert\le Lh/2. ∣ L ( a j , 1 + h /2 − x 2 )∣ ≤ L h /2. Consequently, u j ( u ) = L ( a j , 1 + h / 2 − x 2 ) u_j(u) = L(a_{j,1} + h/2 - x_2) u j ( u ) = L ( a j , 1 + h /2 − x 2 ) throughout Q j Q_j Q j . Also, u k ( u ) = 0 u_k(u) = 0 u k ( u ) = 0 on Q j Q_j Q j for k ≠ j k \ne j k = j , by the separation of the cubes. Since B j = A j + ( h / 2 ) e 1 B_j = A_j + (h/2)e_1 B j = A j + ( h /2 ) e 1 , for every u ∈ A j u \in A_j u ∈ A j we have
U θ ( u ) − U θ ( u + ( h / 2 ) e 1 ) = θ j L h / 2. U_\theta(u) - U_\theta(u + (h/2)e_1) = \theta_j Lh/2. U θ ( u ) − U θ ( u + ( h /2 ) e 1 ) = θ j L h /2. Changing variables in the integral over B j B_j B j therefore gives
θ j S j ( p θ ∘ ) = ∫ 0 1 ( x 1 − 1 / 2 ) 2 d x 1 ∫ A j θ j { U θ ( u ) − U θ ( u + ( h / 2 ) e 1 ) } d u \theta_j S_j(p_\theta^\circ) = \int_0^1 (x_1 - 1/2)^2\,\mathrm{d}x_1 \int_{A_j} \theta_j\{U_\theta(u) - U_\theta(u + (h/2)e_1)\}\,\mathrm{d}u θ j S j ( p θ ∘ ) = ∫ 0 1 ( x 1 − 1/2 ) 2 d x 1 ∫ A j θ j { U θ ( u ) − U θ ( u + ( h /2 ) e 1 )} d u = 1 12 ⋅ L h 2 ⋅ ℓ d − 1 = L h d 24 ⋅ 16 d − 1 . = \frac{1}{12}\cdot\frac{Lh}{2}\cdot\ell^{d-1} = \frac{Lh^d}{24\cdot16^{d-1}}. = 12 1 ⋅ 2 L h ⋅ ℓ d − 1 = 24 ⋅ 1 6 d − 1 L h d . This proves the claim with c 1 = 1 / ( 24 ⋅ 16 d − 1 ) c_1 = 1/(24\cdot16^{d-1}) c 1 = 1/ ( 24 ⋅ 1 6 d − 1 ) .
Lemma 55 (Bridge inequality). There exists c 2 > 0 c_2 > 0 c 2 > 0 , depending only on d d d , such that for every θ ∈ { ± 1 } N \theta\in\{\pm1\}^{N} θ ∈ { ± 1 } N and every q ∈ P L , 1 q \in\mathcal{P}_{L,1} q ∈ P L , 1 ,
J g θ ( p θ ∘ ) − J g θ ( q ) ≥ c 2 α L h d ∑ j = 1 N 1 { θ ^ j ( q ) ≠ θ j } . J_{g_\theta}(p_\theta^\circ) - J_{g_\theta}(q) \ge c_2\alpha Lh^d \sum_{j=1}^{N} \boldsymbol{1}\{\widehat{\theta}_j(q) \ne\theta_j\}. J g θ ( p θ ∘ ) − J g θ ( q ) ≥ c 2 α L h d j = 1 ∑ N 1 { θ j ( q ) = θ j } . Proof. For every q ∈ P L , 1 q \in\mathcal{P}_{L,1} q ∈ P L , 1 , the definition of g θ g_\theta g θ gives
J g θ ( q ) = J 1 ( q ) + α ∑ j = 1 N θ j S j ( q ) . J_{g_\theta}(q) = J_1(q) + \alpha\sum_{j=1}^{N}\theta_j S_j(q). J g θ ( q ) = J 1 ( q ) + α j = 1 ∑ N θ j S j ( q ) . Therefore
J g θ ( p θ ∘ ) − J g θ ( q ) = { J 1 ( p θ ∘ ) − J 1 ( q ) } + α ∑ j = 1 N θ j { S j ( p θ ∘ ) − S j ( q ) } . J_{g_\theta}(p_\theta^\circ) - J_{g_\theta}(q) = \{J_1(p_\theta^\circ) - J_1(q)\} + \alpha\sum_{j=1}^{N}\theta_j\{S_j(p_\theta^\circ) - S_j(q)\}. J g θ ( p θ ∘ ) − J g θ ( q ) = { J 1 ( p θ ∘ ) − J 1 ( q )} + α j = 1 ∑ N θ j { S j ( p θ ∘ ) − S j ( q )} . The first term is non-negative by Lemma 53. Note that B j = A j + ( h / 2 ) e 1 B_j = A_j + (h/2)e_1 B j = A j + ( h /2 ) e 1 , we can write
S j ( q ) = ∫ 0 1 ( x 1 − 1 / 2 ) 2 ∫ A j { q ( x 1 , u ) − q ( x 1 , u + ( h / 2 ) e 1 ) } d u d x 1 . S_j(q) = \int_0^1 (x_1 - 1/2)^2 \int_{A_j}\{q(x_1,u) - q(x_1,u + (h/2)e_1)\}\,\mathrm{d}u\,\mathrm{d}x_1. S j ( q ) = ∫ 0 1 ( x 1 − 1/2 ) 2 ∫ A j { q ( x 1 , u ) − q ( x 1 , u + ( h /2 ) e 1 )} d u d x 1 . The Lipschitz constraint and the proof of Lemma 54 imply
∣ S j ( q ) ∣ ≤ 1 12 ⋅ L h 2 ⋅ ℓ d − 1 = θ j S j ( p θ ∘ ) . \lvert S_j(q)\rvert\le\frac{1}{12}\cdot\frac{Lh}{2}\cdot\ell^{d-1} = \theta_j S_j(p_\theta^\circ). ∣ S j ( q )∣ ≤ 12 1 ⋅ 2 L h ⋅ ℓ d − 1 = θ j S j ( p θ ∘ ) . Thus θ j { S j ( p θ ∘ ) − S j ( q ) } ≥ 0 \theta_j\{S_j(p_\theta^\circ) - S_j(q)\} \ge0 θ j { S j ( p θ ∘ ) − S j ( q )} ≥ 0 for every j j j . If θ ^ j ( q ) ≠ θ j \widehat{\theta}_j(q) \ne\theta_j θ j ( q ) = θ j , then θ j S j ( q ) ≤ 0 \theta_j S_j(q) \le0 θ j S j ( q ) ≤ 0 . Hence
θ j { S j ( p θ ∘ ) − S j ( q ) } ≥ θ j S j ( p θ ∘ ) 1 { θ ^ j ( q ) ≠ θ j } . \theta_j\{S_j(p_\theta^\circ) - S_j(q)\} \ge\theta_j S_j(p_\theta^\circ)\boldsymbol{1}\{\widehat{\theta}_j(q) \ne\theta_j\}. θ j { S j ( p θ ∘ ) − S j ( q )} ≥ θ j S j ( p θ ∘ ) 1 { θ j ( q ) = θ j } . Applying Lemma 54 and summing over j j j proves the claim, with c 2 = c 1 = 1 / ( 24 ⋅ 16 d − 1 ) c_2 = c_1 = 1/(24\cdot16^{d-1}) c 2 = c 1 = 1/ ( 24 ⋅ 1 6 d − 1 ) .
Lemma 56 (Neighbors KL). There exists C K L > 0 C_{\mathrm{KL}}>0 C KL > 0 , depending only on d d d , such that for every θ ∈ { ± 1 } N \theta\in\{\pm1\}^{N} θ ∈ { ± 1 } N and j ∈ [ N ] j \in[N] j ∈ [ N ] ,
KL ( P θ ∥ P θ ( j ) ) ≤ C K L T α 2 h d − 1 . \operatorname{KL}(\mathbb{P}_{\theta}\Vert\mathbb{P}_{\theta^{(j)}}) \le C_{\mathrm{KL}}T\alpha^{2}h^{d-1}. KL ( P θ ∥ P θ ( j ) ) ≤ C KL T α 2 h d − 1 . One may take C K L = 32 / ( 45 ⋅ 16 d − 1 ) C_{\mathrm{KL}}=32/(45\cdot16^{d-1}) C KL = 32/ ( 45 ⋅ 1 6 d − 1 ) .
Proof. The reward model is the same under all θ \theta θ ; only the context density changes. Conditional on the history before round t t t , the learner uses the same randomized decision rule under both environments, and the conditional reward law given ( X t , A t ) (X_t,A_t) ( X t , A t ) is also the same. Therefore the chain rule for KL gives
KL ( P θ ∥ P θ ( j ) ) = T KL ( g θ ∥ g θ ( j ) ) . \operatorname{KL}(\mathbb{P}_{\theta}\Vert\mathbb{P}_{\theta^{(j)}})=T\operatorname{KL}(g_{\theta}\Vert g_{\theta^{(j)}}). KL ( P θ ∥ P θ ( j ) ) = T KL ( g θ ∥ g θ ( j ) ) .
Moreover, by construction,
g θ ( x 1 , u ) − g θ ( j ) ( x 1 , u ) = 2 α ( x 1 − 1 / 2 ) θ j φ j ( u ) . g_{\theta}(x_1,u)-g_{\theta^{(j)}}(x_1,u)=2\alpha(x_1-1/2)\theta_j\varphi_j(u). g θ ( x 1 , u ) − g θ ( j ) ( x 1 , u ) = 2 α ( x 1 − 1/2 ) θ j φ j ( u ) .
Since g θ ( j ) ≥ 15 / 16 g_{\theta^{(j)}}\ge15/16 g θ ( j ) ≥ 15/16 by construction,
KL ( g θ ∥ g θ ( j ) ) ≤ ∫ [ 0 , 1 ] d ( g θ − g θ ( j ) ) 2 g θ ( j ) d x 1 d u ≤ 16 15 ∫ [ 0 , 1 ] d 4 α 2 ( x 1 − 1 / 2 ) 2 φ j ( u ) 2 d x 1 d u = 64 15 α 2 ( ∫ 0 1 ( x 1 − 1 / 2 ) 2 d x 1 ) { ∣ A j ∣ + ∣ B j ∣ } = C K L α 2 h d − 1 . \begin{aligned}
\operatorname{KL}(g_{\theta}\Vert g_{\theta^{(j)}})
&\le\int_{[0,1]^d}\frac{(g_{\theta}-g_{\theta^{(j)}})^2}{g_{\theta^{(j)}}}\,dx_1\,du\\
&\le\frac{16}{15}\int_{[0,1]^d}4\alpha^2(x_1-1/2)^2\varphi_j(u)^2\,dx_1\,du\\
&=\frac{64}{15}\alpha^2\left(\int_0^1(x_1-1/2)^2\,dx_1\right)\{|A_j|+|B_j|\}\\
&=C_{\mathrm{KL}}\alpha^2h^{d-1}.
\end{aligned} KL ( g θ ∥ g θ ( j ) ) ≤ ∫ [ 0 , 1 ] d g θ ( j ) ( g θ − g θ ( j ) ) 2 d x 1 d u ≤ 15 16 ∫ [ 0 , 1 ] d 4 α 2 ( x 1 − 1/2 ) 2 φ j ( u ) 2 d x 1 d u = 15 64 α 2 ( ∫ 0 1 ( x 1 − 1/2 ) 2 d x 1 ) { ∣ A j ∣ + ∣ B j ∣ } = C KL α 2 h d − 1 .
Multiplying by T T T proves the result.
One-dimensional algorithms and theoretical guarantees
This section collects the algorithms and theoretical guarantees for d = 1 d=1 d = 1 . We first present a one-dimensional canonical extension for the fair oracle, and then present the greedy algorithm for continuous covariates and the UCB algorithm for general covariates separately, together with their regret guarantees and proofs.
Common setup and canonical representation
When d = 1 d=1 d = 1 , the decision boundary is a single threshold point instead of a ( d − 1 ) (d-1) ( d − 1 ) -dimensional set. We take X = [ 0 , 1 ] \mathcal{X}=[0,1] X = [ 0 , 1 ] and use the following one-dimensional analogue of Assumption 4 for both algorithms.
Assumption 5 (One-dimensional boundary parameters). Suppose that there exist absolute constants A 0 > 0 A_0>0 A 0 > 0 , 0 < c 0 < C 0 < ∞ 0<c_0<C_0<\infty 0 < c 0 < C 0 < ∞ , and ρ 0 ∈ ( 0 , 1 / 2 ) \rho_0\in(0,1/2) ρ 0 ∈ ( 0 , 1/2 ) such that
Θ 1 d = { ( a , c ) ∈ R 2 : ∣ a ∣ ≤ A 0 , c 0 ≤ c ≤ C 0 , ρ 0 ≤ − a c ≤ 1 − ρ 0 } . \Theta_{1\mathrm{d}}=\left\{(a,c)\in\mathbb{R}^2: |a|\le A_0,\ c_0\le c\le C_0,\ \rho_0\le-\frac{a}{c}\le1-\rho_0\right\}. Θ 1 d = { ( a , c ) ∈ R 2 : ∣ a ∣ ≤ A 0 , c 0 ≤ c ≤ C 0 , ρ 0 ≤ − c a ≤ 1 − ρ 0 } . We assume that θ ∗ = β 1 ∗ − β 2 ∗ = ( a ∗ , c ∗ ) ∈ Θ 1 d \theta^*=\beta_1^*-\beta_2^*=(a^*,c^*)\in\Theta_{1\mathrm{d}} θ ∗ = β 1 ∗ − β 2 ∗ = ( a ∗ , c ∗ ) ∈ Θ 1 d .
Let A ~ 0 > A 0 \widetilde{A}_0>A_0 A 0 > A 0 , 0 < c ~ 0 < c 0 < C 0 < C ~ 0 < ∞ 0<\widetilde{c}_0<c_0<C_0<\widetilde{C}_0<\infty 0 < c 0 < c 0 < C 0 < C 0 < ∞ , and 0 < ρ ~ 0 < ρ 0 0<\widetilde{\rho}_0<\rho_0 0 < ρ 0 < ρ 0 . Define the enlarged parameter set
Θ ~ 1 d = { ( a , c ) ∈ R 2 : ∣ a ∣ ≤ A ~ 0 , c ~ 0 ≤ c ≤ C ~ 0 , ρ ~ 0 ≤ − a c ≤ 1 − ρ ~ 0 } . \widetilde{\Theta}_{1\mathrm{d}}=\left\{(a,c)\in\mathbb{R}^2: |a|\le\widetilde{A}_0,\ \widetilde{c}_0\le c\le\widetilde{C}_0,\ \widetilde{\rho}_0\le-\frac{a}{c}\le1-\widetilde{\rho}_0\right\}. Θ 1 d = { ( a , c ) ∈ R 2 : ∣ a ∣ ≤ A 0 , c 0 ≤ c ≤ C 0 , ρ 0 ≤ − c a ≤ 1 − ρ 0 } .
By the strict enlargement, θ ∗ ∈ Θ ~ 1 d \theta^* \in\widetilde{\Theta}_{1\mathrm{d}} θ ∗ ∈ Θ 1 d and there exists a constant r l o c ( 1 ) > 0 r_{\mathrm{loc}}^{(1)} > 0 r loc ( 1 ) > 0 , depending only on θ ∗ \theta^* θ ∗ and the constants above, such that { θ : ∥ θ − θ ∗ ∥ 2 ≤ 2 r l o c ( 1 ) } ⊆ Θ ~ 1 d \{\theta: \lVert\theta-\theta^* \rVert_2 \le2r_{\mathrm{loc}}^{(1)}\} \subseteq\widetilde{\Theta}_{1\mathrm{d}} { θ : ∥ θ − θ ∗ ∥ 2 ≤ 2 r loc ( 1 ) } ⊆ Θ 1 d .
For θ = ( a , c ) ∈ Θ ~ 1 d \theta=(a,c) \in\widetilde{\Theta}_{1\mathrm{d}} θ = ( a , c ) ∈ Θ 1 d , write x θ = − a / c x_\theta=-a/c x θ = − a / c . For q ∈ [ 0 , 1 ] q \in[0,1] q ∈ [ 0 , 1 ] , define the one-dimensional policy
p θ , q ( 1 ) ( x ) = [ q + L ( x − x θ ) ] [ 0 , 1 ] , x ∈ [ 0 , 1 ] . p_{\theta,q}^{(1)}(x) = [q+L(x-x_\theta)]_{[0,1]}, \qquad x \in[0,1]. p θ , q ( 1 ) ( x ) = [ q + L ( x − x θ ) ] [ 0 , 1 ] , x ∈ [ 0 , 1 ] .
This is the one-dimensional version of the boundary-canonical policy, with the scalar q q q playing the role of the value of the policy at the threshold x θ x_\theta x θ .
Lemma 57. (One-dimensional canonical form and empirical optimisation). Suppose Assumption 5 holds. For every θ ∈ Θ ~ 1 d \theta\in\widetilde{\Theta}_{1\mathrm{d}} θ ∈ Θ 1 d and q ∈ [ 0 , 1 ] q \in[0,1] q ∈ [ 0 , 1 ] , we have p θ , q ( 1 ) ∈ P L , ν p_{\theta,q}^{(1)} \in\mathcal{P}_{L,\nu} p θ , q ( 1 ) ∈ P L , ν . For any p ∈ P L , ν p \in\mathcal{P}_{L,\nu} p ∈ P L , ν , taking q = p ( x θ ) q=p(x_\theta) q = p ( x θ ) gives
( x ~ T θ ) p θ , q ( 1 ) ( x ) ≥ ( x ~ T θ ) p ( x ) , x ∈ [ 0 , 1 ] . (\widetilde{x}^{\mathsf T}\theta)p_{\theta,q}^{(1)}(x) \ge(\widetilde{x}^{\mathsf T}\theta)p(x), \qquad x \in[0,1]. ( x T θ ) p θ , q ( 1 ) ( x ) ≥ ( x T θ ) p ( x ) , x ∈ [ 0 , 1 ] . Consequently, there exists q ∗ ∈ [ 0 , 1 ] q^* \in[0,1] q ∗ ∈ [ 0 , 1 ] such that p ∗ = p θ ∗ , q ∗ ( 1 ) p^*=p_{\theta^*,q^*}^{(1)} p ∗ = p θ ∗ , q ∗ ( 1 ) is a population-optimal fair policy. Moreover, with H s H_s H s as in (16) and d = 1 d=1 d = 1 , for every s ≥ 2 s \ge2 s ≥ 2 and θ ∈ Θ ~ 1 d \theta\in\widetilde{\Theta}_{1\mathrm{d}} θ ∈ Θ 1 d , let z θ z^\theta z θ be an optimiser in the definition of H s ( θ ) H_s(\theta) H s ( θ ) and let p ‾ θ \overline{p}^{\theta} p θ be its McShane extension. Then q θ = p ‾ θ ( x θ ) q^\theta=\overline{p}^{\theta}(x_\theta) q θ = p θ ( x θ ) satisfies
q θ ∈ arg max q ∈ [ 0 , 1 ] 1 s − 1 ∑ i = 1 s − 1 ( X ~ i T θ ) p θ , q ( 1 ) ( X i ) (81) q^\theta\in\underset{q\in[0,1]}{\arg\max}\ \frac{1}{s-1}\sum_{i=1}^{s-1}(\widetilde{X}_i^{\mathsf T}\theta)p_{\theta,q}^{(1)}(X_i) \tag*{(81)} q θ ∈ q ∈ [ 0 , 1 ] arg max s − 1 1 i = 1 ∑ s − 1 ( X i T θ ) p θ , q ( 1 ) ( X i ) (81) and
H s ( θ ) = 1 s − 1 ∑ i = 1 s − 1 ( X ~ i T θ ) p θ , q θ ( 1 ) ( X i ) = max q ∈ [ 0 , 1 ] 1 s − 1 ∑ i = 1 s − 1 ( X ~ i T θ ) p θ , q ( 1 ) ( X i ) . H_s(\theta)=\frac{1}{s-1}\sum_{i=1}^{s-1}(\widetilde{X}_i^{\mathsf T}\theta)p_{\theta,q^\theta}^{(1)}(X_i)
=\underset{q\in[0,1]}{\max}\ \frac{1}{s-1}\sum_{i=1}^{s-1}(\widetilde{X}_i^{\mathsf T}\theta)p_{\theta,q}^{(1)}(X_i). H s ( θ ) = s − 1 1 i = 1 ∑ s − 1 ( X i T θ ) p θ , q θ ( 1 ) ( X i ) = q ∈ [ 0 , 1 ] max s − 1 1 i = 1 ∑ s − 1 ( X i T θ ) p θ , q ( 1 ) ( X i ) .
Proof. Since H θ = { x θ } \mathcal{H}_\theta=\{x_\theta\} H θ = { x θ } and c > 0 c>0 c > 0 , the canonical policy in (6) is exactly p θ , q ( 1 ) p_{\theta,q}^{(1)} p θ , q ( 1 ) . Its feasibility, pointwise domination for every θ \theta θ , and population optimality at θ ∗ \theta^* θ ∗ follow from Theorem 3 . Applying the general finite-LP argument in Appendix § 11 .§ 1 with this singleton boundary gives q θ = p ‾ θ ( x θ ) q^\theta=\overline{p}^{\theta}(x_\theta) q θ = p θ ( x θ ) and the displayed equality with H s ( θ ) H_s(\theta) H s ( θ ) .
Greedy algorithm for continuous covariates
Algorithm
Algorithm 5 is the one-dimensional version of Algorithm 1.
Theoretical guarantee
Theorem 58. (One-dimensional regret upper bound). Suppose Assumptions 1-3 and 5 hold with d = 1 d=1 d = 1 and 0 < L ≤ L 0 ′ 0<L\le L'_0 0 < L ≤ L 0 ′ . For η ∈ ( 0 , 1 / 2 ) \eta\in(0,1/2) η ∈ ( 0 , 1/2 ) , suppose that 2 T o ≤ T 2T_o\le T 2 T o ≤ T and T o T_o T o satisfies
T o ≥ C T log ( T / η ) ∨ C l o c ( 1 ) log ( T / η ) { r l o c ( 1 ) } 2 . T_o \ge C_T\log(T/\eta)\vee\frac{C_{\mathrm{loc}}^{(1)}\log(T/\eta)}{\{r_{\mathrm{loc}}^{(1)}\}^2}. T o ≥ C T log ( T / η ) ∨ { r loc ( 1 ) } 2 C loc ( 1 ) log ( T / η ) . Then the policy played at every round in Algorithm 5 is ( L , ν ) (L,\nu) ( L , ν ) -fair. Moreover, with probability at least 1 − η 1-\eta 1 − η ,
R T ≤ C 1 T o + C 2 T log ( e T 2 / η ) , \mathcal{R}_T\le C_1T_o+C_2\sqrt{T\log(eT^2/\eta)}, R T ≤ C 1 T o + C 2 T log ( e T 2 / η ) , where C 1 , C 2 < ∞ C_1,C_2<\infty C 1 , C 2 < ∞ depend only on L , c g , C g , σ , C β , λ L,c_g,C_g,\sigma,C_\beta,\lambda L , c g , C g , σ , C β , λ and the constants defining Θ ~ 1 d \widetilde{\Theta}_{1\mathrm{d}} Θ 1 d . In particular, if T ≥ C 0 log ( T / η ) T\ge C_0\log(T/\eta) T ≥ C 0 log ( T / η ) for a sufficiently large constant C 0 C_0 C 0 depending only on these parameters, r l o c ( 1 ) r_{\mathrm{loc}}^{(1)} r loc ( 1 ) and the fixed exploration multiplier, taking T o ≍ T log ( T / η ) T_o\asymp\sqrt{T\log(T/\eta)} T o ≍ T log ( T / η ) gives
R T ≤ C T log ( e T 2 / η ) . \mathcal{R}_T\le C\sqrt{T\log(eT^2/\eta)}. R T ≤ C T log ( e T 2 / η ) .
Algorithm 5 Lazy adaptive one-dimensional fair greedy algorithm ($d = 1$)
INPUT: Time horizon $T$, fairness parameter $(L, \nu)$, ridge parameter $\lambda$, exploration period $T_o$
for $k \in \{1,2\}$ do
$V_k \leftarrow \lambda I_2$, $B_k \leftarrow 0$.
end for
$s \leftarrow T_o$, $\widehat{p}(\cdot) = 1/2$.
for $t = 1,2,\ldots,2T_o$ do
Observe $X_t$, set $\widetilde{X}_t = (1,X_t)^\mathsf{T}$.
Play arm 1 with probability $1/2$ and arm 2 otherwise; observe $Y_t$.
Update $V_{A_t} \leftarrow V_{A_t} + \widetilde{X}_t\widetilde{X}_t^\mathsf{T}$ and $B_{A_t} \leftarrow B_{A_t} + \widetilde{X}_tY_t$.
end for
for $t = 2T_o + 1,\ldots,T$ do
if $t \ge 2s$ then
$s \leftarrow t$.
for $k \in \{1,2\}$ do
Set $\widehat{\beta}_{k,s-1} \leftarrow V_k^{-1}B_k$.
end for
Set $\widehat{\theta}_s \leftarrow \widehat{\beta}_{1,s-1} - \widehat{\beta}_{2,s-1}$ and write $\widehat{\theta}_s = (\widehat{a}_s,\widehat{c}_s)$.
if $\widehat{c}_s > 0$ and $-\widehat{a}_s \in [0,\widehat{c}_s]$ then
Set $\widehat{x}_s \leftarrow -\widehat{a}_s/\widehat{c}_s$ and compute
$\widehat{q}_s \in \arg\max_{q\in[0,1]} \frac{1}{s-1}\sum_{i=1}^{s-1}\widetilde{X}_i^\mathsf{T}\widehat{\theta}_s[q+L(X_i-\widehat{x}_s)]_{[0,1]}$.
Set $\widehat{p}(\cdot) \leftarrow p_{\widehat{\theta}_s,\widehat{q}_s}^{(1)}(\cdot)$.
else
Set $\widehat{p}(\cdot) \leftarrow 1/2$.
end if
end if
Observe $X_t$ and set $\widetilde{X}_t = (1,X_t)^\mathsf{T}$.
Play arm 1 with probability $\widehat{p}(X_t)$ and arm 2 otherwise; observe $Y_t$.
Update $V_{A_t} \leftarrow V_{A_t} + \widetilde{X}_t\widetilde{X}_t^\mathsf{T}$ and $B_{A_t} \leftarrow B_{A_t} + \widetilde{X}_tY_t$.
end for
Proof of Theorem 58
Lemma 59 (Threshold perturbation). For any θ = ( a , c ) , θ ′ = ( a ′ , c ′ ) ∈ Θ ~ 1 d \theta=(a,c),\theta'=(a',c')\in\widetilde{\Theta}_{1d} θ = ( a , c ) , θ ′ = ( a ′ , c ′ ) ∈ Θ 1 d ,
∣ x θ − x θ ′ ∣ ≤ C x ∥ θ − θ ′ ∥ 2 , |x_\theta-x_{\theta'}|\le C_x\|\theta-\theta'\|_2, ∣ x θ − x θ ′ ∣ ≤ C x ∥ θ − θ ′ ∥ 2 , where C x = c ~ 0 − 1 + A ~ 0 c ~ 0 − 2 C_x=\tilde{c}_0^{-1}+\tilde{A}_0\tilde{c}_0^{-2} C x = c ~ 0 − 1 + A ~ 0 c ~ 0 − 2 .
Proof. Since x θ = − a / c x_\theta=-a/c x θ = − a / c and x θ ′ = − a ′ / c ′ x_{\theta'}=-a'/c' x θ ′ = − a ′ / c ′ , we have
∣ x θ − x θ ′ ∣ ≤ ∣ a − a ′ ∣ c ~ 0 + ∣ a ′ ∣ c ~ 0 2 ∣ c − c ′ ∣ ≤ C x ∥ θ − θ ′ ∥ 2 . |x_\theta-x_{\theta'}|\le\frac{|a-a'|}{\tilde{c}_0}+\frac{|a'|}{\tilde{c}_0^2}|c-c'|\le C_x\|\theta-\theta'\|_2. ∣ x θ − x θ ′ ∣ ≤ c ~ 0 ∣ a − a ′ ∣ + c ~ 0 2 ∣ a ′ ∣ ∣ c − c ′ ∣ ≤ C x ∥ θ − θ ′ ∥ 2 .
Lemma 60 (One-dimensional policy diversity). Suppose Assumptions 1 and 5 hold with d = 1 d=1 d = 1 and L > 0 L>0 L > 0 . There exists a constant λ d i v ( 1 ) > 0 \lambda_{\mathrm{div}}^{(1)}>0 λ div ( 1 ) > 0 , depending only on L L L , c g c_g c g and ρ ~ 0 \tilde{\rho}_0 ρ ~ 0 , such that, for every θ ∈ Θ ~ 1 d \theta\in\widetilde{\Theta}_{1d} θ ∈ Θ 1 d and q ∈ [ 0 , 1 ] q\in[0,1] q ∈ [ 0 , 1 ] ,
E [ X ~ X ~ ⊤ p θ , q ( 1 ) ( X ) ] ⪰ λ d i v ( 1 ) I 2 \mathbb{E}\left[\widetilde{X}\widetilde{X}^{\top}p_{\theta,q}^{(1)}(X)\right]\succeq\lambda_{\mathrm{div}}^{(1)}I_2 E [ X X ⊤ p θ , q ( 1 ) ( X ) ] ⪰ λ div ( 1 ) I 2 and
E [ X ~ X ~ ⊤ { 1 − p θ , q ( 1 ) ( X ) } ] ⪰ λ d i v ( 1 ) I 2 . \mathbb{E}\left[\widetilde{X}\widetilde{X}^{\top}\{1-p_{\theta,q}^{(1)}(X)\}\right]\succeq\lambda_{\mathrm{div}}^{(1)}I_2. E [ X X ⊤ { 1 − p θ , q ( 1 ) ( X )} ] ⪰ λ div ( 1 ) I 2 .
Proof. Let v 1 = ρ ~ 0 / 2 v_1=\tilde{\rho}_0/2 v 1 = ρ ~ 0 /2 and ϵ 1 = min { 1 , L ρ ~ 0 / 2 } \epsilon_1=\min\{1,L\tilde{\rho}_0/2\} ϵ 1 = min { 1 , L ρ ~ 0 /2 } . Since x θ ∈ [ ρ ~ 0 , 1 − ρ ~ 0 ] x_\theta\in[\tilde{\rho}_0,1-\tilde{\rho}_0] x θ ∈ [ ρ ~ 0 , 1 − ρ ~ 0 ] , for every x ∈ [ 1 − ρ ~ 0 / 2 , 1 ] x\in[1-\tilde{\rho}_0/2,1] x ∈ [ 1 − ρ ~ 0 /2 , 1 ] , we have x − x θ ≥ ρ ~ 0 / 2 x-x_\theta\ge\tilde{\rho}_0/2 x − x θ ≥ ρ ~ 0 /2 . Therefore p θ , q ( 1 ) ( x ) ≥ ϵ 1 p_{\theta,q}^{(1)}(x)\ge\epsilon_1 p θ , q ( 1 ) ( x ) ≥ ϵ 1 on [ 1 − ρ ~ 0 / 2 , 1 ] [1-\tilde{\rho}_0/2,1] [ 1 − ρ ~ 0 /2 , 1 ] .
Similarly, for every x ∈ [ 0 , ρ ~ 0 / 2 ] x\in[0,\tilde{\rho}_0/2] x ∈ [ 0 , ρ ~ 0 /2 ] , we have x θ − x ≥ ρ ~ 0 / 2 x_\theta-x\ge\tilde{\rho}_0/2 x θ − x ≥ ρ ~ 0 /2 . If q − L ( x θ − x ) < 0 q-L(x_\theta-x)<0 q − L ( x θ − x ) < 0 , then p θ , q ( 1 ) ( x ) = 0 p_{\theta,q}^{(1)}(x)=0 p θ , q ( 1 ) ( x ) = 0 , while otherwise 1 − p θ , q ( 1 ) ( x ) = 1 − q + L ( x θ − x ) ≥ L ρ ~ 0 / 2 1-p_{\theta,q}^{(1)}(x)=1-q+L(x_\theta-x)\ge L\tilde{\rho}_0/2 1 − p θ , q ( 1 ) ( x ) = 1 − q + L ( x θ − x ) ≥ L ρ ~ 0 /2 . Hence 1 − p θ , q ( 1 ) ( x ) ≥ ϵ 1 1-p_{\theta,q}^{(1)}(x)\ge\epsilon_1 1 − p θ , q ( 1 ) ( x ) ≥ ϵ 1 on [ 0 , ρ ~ 0 / 2 ] [0,\tilde{\rho}_0/2] [ 0 , ρ ~ 0 /2 ] .
By Lemma 28 with d = 1 d=1 d = 1 , any interval of length at least v 1 v_1 v 1 satisfies
E [ X ~ X ~ ⊤ 1 { X ∈ A } ] ⪰ c g v 1 3 128 I 2 . \mathbb{E}\left[\widetilde{X}\widetilde{X}^{\top}\mathbf{1}\{X\in A\}\right]\succeq c_g\frac{v_1^3}{128}I_2. E [ X X ⊤ 1 { X ∈ A } ] ⪰ c g 128 v 1 3 I 2 .
Thus the two displayed lower bounds hold with λ d i v ( 1 ) = c g ϵ 1 v 1 3 / 128 \lambda_{\mathrm{div}}^{(1)}=c_g\epsilon_1v_1^3/128 λ div ( 1 ) = c g ϵ 1 v 1 3 /128 .
Lemma 61 (One-dimensional estimation at lazy updates). Suppose Assumptions 1–3 and 5 hold with d = 1 d=1 d = 1 . Let S T \mathcal{S}_T S T be the set of lazy update times up to time T T T . Suppose
T 0 ≥ C T log ( T / η ) ∨ C l o c ( 1 ) log ( T / η ) { η l o c ( 1 ) } 2 . T_0\ge C_T\log(T/\eta)\vee\frac{C_{\mathrm{loc}}^{(1)}\log(T/\eta)}{\{\eta_{\mathrm{loc}}^{(1)}\}^2}. T 0 ≥ C T log ( T / η ) ∨ { η loc ( 1 ) } 2 C loc ( 1 ) log ( T / η ) . Then, with probability at least 1 − η 1-\eta 1 − η , for every s ∈ S T s\in\mathcal{S}_T s ∈ S T and k ∈ { 1 , 2 } k\in\{1,2\} k ∈ { 1 , 2 } ,
∥ β ^ k , s − 1 − β k ∗ ∥ 2 ≤ C e s t ( 1 ) log ( T / η ) s − 1 . \|\widehat{\beta}_{k,s-1}-\beta^*_k\|_2\le C_{\mathrm{est}}^{(1)}\sqrt{\frac{\log(T/\eta)}{s-1}}. ∥ β k , s − 1 − β k ∗ ∥ 2 ≤ C est ( 1 ) s − 1 log ( T / η ) . Moreover, for every s ∈ S T s\in\mathcal{S}_T s ∈ S T ,
θ ^ s ∈ Θ ~ 1 d and ∥ θ ^ s − θ ∗ ∥ 2 ≤ 2 C e s t ( 1 ) log ( T / η ) s − 1 . \widehat{\theta}_s\in\widetilde{\Theta}_{1d}\quad\text{and}\quad\|\widehat{\theta}_s-\theta^*\|_2\le2C_{\mathrm{est}}^{(1)}\sqrt{\frac{\log(T/\eta)}{s-1}}. θ s ∈ Θ 1 d and ∥ θ s − θ ∗ ∥ 2 ≤ 2 C est ( 1 ) s − 1 log ( T / η ) .
Proof. The proof follows the same argument as the proof of Lemma 25 , so we only include the difference here.
First, during the first 2 T o 2T_o 2 T o rounds, the algorithm uses a constant policy p t ( ⋅ ) = 1 / 2 p_t(\cdot)=1/2 p t ( ⋅ ) = 1/2 . By the same allocation, conditional matrix Chernoff and noise concentration arguments as in the proof of Lemma 24, after enlarging the constants in the lower bound on T o T_o T o , with probability at least 1 − η / 4 1-\eta/4 1 − η /4 , we have ∥ θ ^ s 0 − θ ∗ ∥ 2 ≤ r l o c ( 1 ) \lVert\widehat{\theta}_{s_0}-\theta^*\rVert_2\le r_{\mathrm{loc}}^{(1)} ∥ θ s 0 − θ ∗ ∥ 2 ≤ r loc ( 1 ) and
λ min ( V k , 2 T o ) ≥ λ + λ X ( 1 ) T o 4 , k ∈ { 1 , 2 } . \lambda_{\min}(V_{k,2T_o})\ge\lambda+\frac{\lambda_X^{(1)}T_o}{4},\qquad k\in\{1,2\}. λ m i n ( V k , 2 T o ) ≥ λ + 4 λ X ( 1 ) T o , k ∈ { 1 , 2 } .
where s 0 s_0 s 0 is the first lazy update time and λ X ( 1 ) > 0 \lambda_X^{(1)}>0 λ X ( 1 ) > 0 depends only on the covariate distribution.
Next, consider the post-initialisation rounds. Suppose that θ ^ r ∈ Θ ~ 1 d \widehat{\theta}_r\in\widetilde{\Theta}_{1d} θ r ∈ Θ 1 d for every lazy update time r ≤ t r\le t r ≤ t . Then, for each round 2 T o < i ≤ t 2T_o<i\le t 2 T o < i ≤ t , the policy used at round i i i is generated by a parameter in Θ ~ 1 d \widetilde{\Theta}_{1d} Θ 1 d . Therefore, by Lemma 60 , we have that
E [ X ~ i X ~ i T 1 { A i = k } ∣ F i − 1 ] ⪰ λ d i v ( 1 ) I 2 , k ∈ { 1 , 2 } . \mathbb{E}\left[\widetilde{X}_i\widetilde{X}_i^{\mathsf T}1\{A_i=k\}\mid\mathcal{F}_{i-1}\right]\succeq\lambda_{\mathrm{div}}^{(1)}I_2,\qquad k\in\{1,2\}. E [ X i X i T 1 { A i = k } ∣ F i − 1 ] ⪰ λ div ( 1 ) I 2 , k ∈ { 1 , 2 } .
Moreover, for i ≤ 2 T o i\le2T_o i ≤ 2 T o , the independent arm assignments give
E [ X ~ i X ~ i T 1 { A i = k } ∣ F i − 1 ] = 1 2 E [ X ~ X ~ T ] ⪰ c g 256 I 2 ⪰ λ d i v ( 1 ) I 2 , k ∈ { 1 , 2 } , \mathbb{E}\left[\widetilde{X}_i\widetilde{X}_i^{\mathsf T}1\{A_i=k\}\mid\mathcal{F}_{i-1}\right]=\frac{1}{2}\mathbb{E}\left[\widetilde{X}\widetilde{X}^{\mathsf T}\right]\succeq\frac{c_g}{256}I_2\succeq\lambda_{\mathrm{div}}^{(1)}I_2,\qquad k\in\{1,2\}, E [ X i X i T 1 { A i = k } ∣ F i − 1 ] = 2 1 E [ X X T ] ⪰ 256 c g I 2 ⪰ λ div ( 1 ) I 2 , k ∈ { 1 , 2 } ,
where the first inequality follows from Lemma 28 with A = [ 0 , 1 ] A=[0,1] A = [ 0 , 1 ] , and the second follows from the value of λ d i v ( 1 ) \lambda_{\mathrm{div}}^{(1)} λ div ( 1 ) in the proof of Lemma 60 . Applying the same matrix concentration argument as in the proof of Lemma 25 , and taking a union bound over t ≤ T t\le T t ≤ T and k ∈ { 1 , 2 } k\in\{1,2\} k ∈ { 1 , 2 } , we obtain that, with probability at least 1 − η / 2 1-\eta/2 1 − η /2 , for every t ∈ { 2 T o , … , T } t\in\{2T_o,\ldots,T\} t ∈ { 2 T o , … , T } and k ∈ { 1 , 2 } k\in\{1,2\} k ∈ { 1 , 2 } , if θ ^ r ∈ Θ ~ 1 d \widehat{\theta}_r\in\widetilde{\Theta}_{1d} θ r ∈ Θ 1 d for every lazy update time r ≤ t r\le t r ≤ t ,
λ min ( V k , t ) ≥ c V ( 1 ) t . \lambda_{\min}(V_{k,t})\ge c_V^{(1)}t. λ m i n ( V k , t ) ≥ c V ( 1 ) t .
On the same high-probability event, the martingale noise term is controlled uniformly over time by the same coordinatewise sub-Gaussian argument as in the proof of Lemma 25 . Thus, with probability at least 1 − η / 4 1-\eta/4 1 − η /4 , for every t ≤ T t\le T t ≤ T and k ∈ { 1 , 2 } k\in\{1,2\} k ∈ { 1 , 2 } ,
∥ ∑ i = 1 t X ~ i ξ i , k 1 { A i = k } ∥ 2 ≤ C ξ ( 1 ) t log ( T / η ) . \left\lVert\sum_{i=1}^{t}\widetilde{X}_i\xi_{i,k}1\{A_i=k\}\right\rVert_2\le C_{\xi}^{(1)}\sqrt{t\log(T/\eta)}. i = 1 ∑ t X i ξ i , k 1 { A i = k } 2 ≤ C ξ ( 1 ) t log ( T / η ) .
We work on the intersection of the preceding high-probability events, which by a union bound argument, has probability at least 1 − η 1-\eta 1 − η .
It remains to prove that θ ^ s ∈ Θ ~ 1 d \widehat{\theta}_s\in\widetilde{\Theta}_{1d} θ s ∈ Θ 1 d at every lazy update time. We prove this by induction over s ∈ S T s\in S_T s ∈ S T . For the first lazy update time s 0 s_0 s 0 , we already have ∥ θ ^ s 0 − θ ∗ ∥ 2 ≤ r l o c ( 1 ) \lVert\widehat{\theta}_{s_0}-\theta^*\rVert_2\le r_{\mathrm{loc}}^{(1)} ∥ θ s 0 − θ ∗ ∥ 2 ≤ r loc ( 1 ) . Since { θ : ∥ θ − θ ∗ ∥ 2 ≤ 2 r l o c ( 1 ) } ⊆ Θ ~ 1 d \{\theta:\lVert\theta-\theta^*\rVert_2\le2r_{\mathrm{loc}}^{(1)}\}\subseteq\widetilde{\Theta}_{1d} { θ : ∥ θ − θ ∗ ∥ 2 ≤ 2 r loc ( 1 ) } ⊆ Θ 1 d , we have θ ^ s 0 ∈ Θ ~ 1 d \widehat{\theta}_{s_0}\in\widetilde{\Theta}_{1d} θ s 0 ∈ Θ 1 d .
Now fix a lazy update time s > s 0 s>s_0 s > s 0 and suppose that θ ^ r ∈ Θ ~ 1 d \widehat{\theta}_r\in\widetilde{\Theta}_{1d} θ r ∈ Θ 1 d for every previous lazy update time r < s r<s r < s . Then the eigenvalue bound above gives λ min ( V k , s − 1 ) ≥ c V ( 1 ) ( s − 1 ) \lambda_{\min}(V_{k,s-1})\ge c_V^{(1)}(s-1) λ m i n ( V k , s − 1 ) ≥ c V ( 1 ) ( s − 1 ) , k ∈ { 1 , 2 } k\in\{1,2\} k ∈ { 1 , 2 } . Writing S k , s − 1 = ∑ i = 1 s − 1 X ~ i ξ i , k 1 { A i = k } S_{k,s-1}=\sum_{i=1}^{s-1}\widetilde{X}_i\xi_{i,k}1\{A_i=k\} S k , s − 1 = ∑ i = 1 s − 1 X i ξ i , k 1 { A i = k } , the ridge estimator satisfies
β ^ k , s − 1 − β k ∗ = V k , s − 1 − 1 S k , s − 1 − λ V k , s − 1 − 1 β k ∗ . \widehat{\beta}_{k,s-1}-\beta_k^*=V_{k,s-1}^{-1}S_{k,s-1}-\lambda V_{k,s-1}^{-1}\beta_k^*. β k , s − 1 − β k ∗ = V k , s − 1 − 1 S k , s − 1 − λ V k , s − 1 − 1 β k ∗ .
Therefore, we have that
∥ β ^ k , s − 1 − β k ∗ ∥ 2 ≤ C ξ ( 1 ) ( s − 1 ) log ( T / η ) c V ( 1 ) ( s − 1 ) + λ C β c V ( 1 ) ( s − 1 ) ≤ C e s t ( 1 ) log ( T / η ) s − 1 . \lVert\widehat{\beta}_{k,s-1}-\beta_k^*\rVert_2\le\frac{C_{\xi}^{(1)}\sqrt{(s-1)\log(T/\eta)}}{c_V^{(1)}(s-1)}+\frac{\lambda C_\beta}{c_V^{(1)}(s-1)}\le C_{\mathrm{est}}^{(1)}\sqrt{\frac{\log(T/\eta)}{s-1}}. ∥ β k , s − 1 − β k ∗ ∥ 2 ≤ c V ( 1 ) ( s − 1 ) C ξ ( 1 ) ( s − 1 ) log ( T / η ) + c V ( 1 ) ( s − 1 ) λ C β ≤ C est ( 1 ) s − 1 log ( T / η ) .
Consequently, it follows that
∥ θ ^ s − θ ∗ ∥ 2 ≤ ∥ β ^ 1 , s − 1 − β 1 ∗ ∥ 2 + ∥ β ^ 2 , s − 1 − β 2 ∗ ∥ 2 ≤ 2 C e s t ( 1 ) log ( T / η ) s − 1 . \lVert\widehat{\theta}_s-\theta^*\rVert_2 \le\lVert\widehat{\beta}_{1,s-1}-\beta_1^*\rVert_2+\lVert\widehat{\beta}_{2,s-1}-\beta_2^*\rVert_2 \le2C_{\mathrm{est}}^{(1)}\sqrt{\frac{\log(T/\eta)}{s-1}}. ∥ θ s − θ ∗ ∥ 2 ≤ ∥ β 1 , s − 1 − β 1 ∗ ∥ 2 + ∥ β 2 , s − 1 − β 2 ∗ ∥ 2 ≤ 2 C est ( 1 ) s − 1 log ( T / η ) .
Since s − 1 ≥ 2 T o s-1\ge2T_o s − 1 ≥ 2 T o , the lower bound on T o T_o T o implies
2 C e s t ( 1 ) log ( T / η ) s − 1 ≤ r l o c ( 1 ) . 2C_{\mathrm{est}}^{(1)}\sqrt{\frac{\log(T/\eta)}{s-1}}\le r_{\mathrm{loc}}^{(1)}. 2 C est ( 1 ) s − 1 log ( T / η ) ≤ r loc ( 1 ) .
Hence, we conclude that ∥ θ ^ s − θ ∗ ∥ 2 ≤ r l o c ( 1 ) \lVert\widehat{\theta}_s-\theta^*\rVert_2\le r_{\mathrm{loc}}^{(1)} ∥ θ s − θ ∗ ∥ 2 ≤ r loc ( 1 ) , and so θ ^ s ∈ Θ ~ 1 d \widehat{\theta}_s\in\widetilde{\Theta}_{1\mathrm{d}} θ s ∈ Θ 1 d . This completes the induction and the lemma follows.
For n ≥ 1 n\ge1 n ≥ 1 , define
Δ n L , 1 = sup θ ∈ Θ ~ 1 d , q ∈ [ 0 , 1 ] ∣ 1 n ∑ i = 1 n X ~ i T θ p θ , q ∗ ( 1 ) ( X i ) − E { X ~ T θ p θ , q ∗ ( 1 ) ( X ) } ∣ . \Delta_n^{L,1}=\sup_{\theta\in\widetilde{\Theta}_{1\mathrm{d}},\,q\in[0,1]}\left|\frac{1}{n}\sum_{i=1}^{n}\widetilde{X}_i^{\mathsf{T}}\theta_{p_{\theta,q}^{*(1)}}(X_i)-\mathbb{E}\{\widetilde{X}^{\mathsf{T}}\theta_{p_{\theta,q}^{*(1)}}(X)\}\right|. Δ n L , 1 = θ ∈ Θ 1 d , q ∈ [ 0 , 1 ] sup n 1 i = 1 ∑ n X i T θ p θ , q ∗ ( 1 ) ( X i ) − E { X T θ p θ , q ∗ ( 1 ) ( X )} .
Lemma 62 (Control of Δ n L , 1 \Delta_n^{L,1} Δ n L , 1 ). Suppose Assumptions 1 and 5 hold with d = 1 d=1 d = 1 . There exists a constant C < ∞ C<\infty C < ∞ , depending only on L L L , c g c_g c g , C g C_g C g , A 0 A_0 A 0 , C 0 C_0 C 0 and the constants defining Θ ~ 1 d \widetilde{\Theta}_{1\mathrm{d}} Θ 1 d , such that, for every n ≥ 2 n\ge2 n ≥ 2 and every η ∈ ( 0 , 1 ) \eta\in(0,1) η ∈ ( 0 , 1 ) , with probability at least 1 − η 1-\eta 1 − η ,
Δ n L , 1 ≤ C log ( e n / η ) n . \Delta_n^{L,1}\le C\sqrt{\frac{\log(en/\eta)}{n}}. Δ n L , 1 ≤ C n log ( e n / η ) . Proof. Let
F L , 1 = { x ↦ x ~ T θ p θ , q ∗ ( 1 ) ( x ) : θ ∈ Θ ~ 1 d , q ∈ [ 0 , 1 ] } . \mathcal{F}_{L,1}=\left\{x\mapsto\widetilde{x}^{\mathsf{T}}\theta p_{\theta,q}^{*(1)}(x):\theta\in\widetilde{\Theta}_{1\mathrm{d}},\ q\in[0,1]\right\}. F L , 1 = { x ↦ x T θ p θ , q ∗ ( 1 ) ( x ) : θ ∈ Θ 1 d , q ∈ [ 0 , 1 ] } . Since ∣ x ~ T θ ∗ ∣ ≤ A 0 + C 0 = : B 1 |\widetilde{x}^{\mathsf{T}}\theta^*|\le A_0+C_0=:B_1 ∣ x T θ ∗ ∣ ≤ A 0 + C 0 =: B 1 and 0 ≤ p θ , q ∗ ( 1 ) ≤ 1 0\le p_{\theta,q}^{*(1)}\le1 0 ≤ p θ , q ∗ ( 1 ) ≤ 1 , the class is uniformly bounded by B 1 B_1 B 1 . Moreover, p θ , q ∗ ( 1 ) p_{\theta,q}^{*(1)} p θ , q ∗ ( 1 ) depends on θ \theta θ only through x θ ∈ [ ρ ~ 0 , 1 − ρ ~ 0 ] x_\theta\in[\widetilde{\rho}_0,1-\widetilde{\rho}_0] x θ ∈ [ ρ 0 , 1 − ρ 0 ] . For any x 0 , x 0 ′ ∈ [ ρ ~ 0 , 1 − ρ ~ 0 ] x_0,x'_0\in[\widetilde{\rho}_0,1-\widetilde{\rho}_0] x 0 , x 0 ′ ∈ [ ρ 0 , 1 − ρ 0 ] and q , q ′ ∈ [ 0 , 1 ] q,q'\in[0,1] q , q ′ ∈ [ 0 , 1 ] ,
sup x ∈ [ 0 , 1 ] ∣ [ q + L ( x − x 0 ) ] [ 0 , 1 ] − [ q ′ + L ( x − x 0 ′ ) ] [ 0 , 1 ] ∣ ≤ ∣ q − q ′ ∣ + L ∣ x 0 − x 0 ′ ∣ . \sup_{x\in[0,1]}\left|[q+L(x-x_0)]_{[0,1]}-[q'+L(x-x'_0)]_{[0,1]}\right|\le|q-q'|+L|x_0-x'_0|. x ∈ [ 0 , 1 ] sup [ q + L ( x − x 0 ) ] [ 0 , 1 ] − [ q ′ + L ( x − x 0 ′ ) ] [ 0 , 1 ] ≤ ∣ q − q ′ ∣ + L ∣ x 0 − x 0 ′ ∣. Therefore, the class F L , 1 \mathcal{F}_{L,1} F L , 1 admits brackets with
log N [ ] ( ε , F L , 1 , L 2 ( P ) ) ≤ C 1 log ( C 1 / ε ) . \log N_{[]}(\varepsilon,\mathcal{F}_{L,1},L_2(P))\le C_1\log(C_1/\varepsilon). log N [ ] ( ε , F L , 1 , L 2 ( P )) ≤ C 1 log ( C 1 / ε ) . Applying the same bracketing maximal inequality and bounded empirical-process concentration as in the proof of Lemma 23 gives the claim.
Proposition 63 (One-dimensional regret decomposition). Suppose Assumptions 1–3 and 5 hold with d = 1 d=1 d = 1 . Suppose that T o T_o T o satisfies the lower bound in Lemma 61 . Then, for every update time s ∈ S T s\in S_T s ∈ S T , with probability at least 1 − η 1-\eta 1 − η ,
R s ( p ∗ ) − R s ( p ^ s ) ≤ C θ , L ( 1 ) ∥ θ ^ s − θ ∗ ∥ 2 + 2 Δ s − 1 L , 1 . R_s(p^*)-R_s(\widehat{p}_s)\le C_{\theta,L}^{(1)}\lVert\widehat{\theta}_s-\theta^*\rVert_2+2\Delta_{s-1}^{L,1}. R s ( p ∗ ) − R s ( p s ) ≤ C θ , L ( 1 ) ∥ θ s − θ ∗ ∥ 2 + 2 Δ s − 1 L , 1 . where p ∗ = p θ ∗ , q ∗ ( 1 ) p^*=p_{\theta^*,q^*}^{(1)} p ∗ = p θ ∗ , q ∗ ( 1 ) , q ∗ ∈ a r g m a x q ∈ [ 0 , 1 ] R ( p θ ∗ , q ( 1 ) ) q^*\in\mathop{\rm arg\,max}_{q\in[0,1]}R(p_{\theta^*,q}^{(1)}) q ∗ ∈ arg max q ∈ [ 0 , 1 ] R ( p θ ∗ , q ( 1 ) ) , and C θ , L ( 1 ) < ∞ C_{\theta,L}^{(1)}<\infty C θ , L ( 1 ) < ∞ depends only on L L L , C β C_\beta C β and the constants defining Θ ~ 1 d \widetilde{\Theta}_{1\mathrm{d}} Θ 1 d .
Proof. Fix t > 2 T o t > 2T_o t > 2 T o and write s = τ ( t ) s = \tau(t) s = τ ( t ) . The following proof is constructed conditioning on the event described in Lemma 61 , which has probability at least 1 − η 1-\eta 1 − η . On this event, θ ^ s ∈ Θ ~ 1 d \hat{\theta}_s \in\widetilde{\Theta}_{1d} θ ^ s ∈ Θ 1 d . Writing θ ^ s = ( a ^ s , c ^ s ) \hat{\theta}_s = (\hat{a}_s,\hat{c}_s) θ ^ s = ( a ^ s , c ^ s ) , this implies c ^ s ≥ c ~ 0 > 0 \hat{c}_s \ge\tilde{c}_0 > 0 c ^ s ≥ c ~ 0 > 0 . Hence the fallback step in Algorithm 5 is not used, and the algorithm uses p ^ s = p θ ^ s , q ^ s ( 1 ) \hat{p}_s = p_{\hat{\theta}_s,\hat{q}_s}^{(1)} p ^ s = p θ ^ s , q ^ s ( 1 ) . Define p s ∗ = p θ ∗ , q ∗ ( 1 ) p_s^* = p_{\theta^*,q^*}^{(1)} p s ∗ = p θ ∗ , q ∗ ( 1 ) .
By Lemma 59 and the nonexpansiveness of the clipping map,
sup x ∈ [ 0 , 1 ] ∣ p θ ∗ , q ∗ ( 1 ) ( x ) − p θ ^ s , q ^ s ( 1 ) ( x ) ∣ ≤ L ∣ x θ ∗ − x θ ^ s ∣ ≤ L C x ∥ θ ^ s − θ ∗ ∥ 2 . \sup_{x\in[0,1]} \left|p_{\theta^*,q^*}^{(1)}(x)-p_{\hat{\theta}_s,\hat{q}_s}^{(1)}(x)\right| \le L\left|x_{\theta^*}-x_{\hat{\theta}_s}\right| \le LC_x\left\|\hat{\theta}_s-\theta^*\right\|_2. x ∈ [ 0 , 1 ] sup p θ ∗ , q ∗ ( 1 ) ( x ) − p θ ^ s , q ^ s ( 1 ) ( x ) ≤ L x θ ∗ − x θ ^ s ≤ L C x θ ^ s − θ ∗ 2 .
Since ∣ X ~ T θ ∗ ∣ ≤ 2 ( ∥ β 1 ∗ ∥ 2 + ∥ β 2 ∗ ∥ 2 ) ≤ 2 2 C β \left|\widetilde{X}^{\mathsf{T}}\theta^*\right| \le\sqrt{2}\left(\|\beta_1^*\|_2+\|\beta_2^*\|_2\right) \le2\sqrt{2}C_\beta X T θ ∗ ≤ 2 ( ∥ β 1 ∗ ∥ 2 + ∥ β 2 ∗ ∥ 2 ) ≤ 2 2 C β , it follows that
E [ X ~ T θ ∗ { p ∗ ( X ) − p s t ( X ) } ∣ F s − 1 ] ≤ C 1 ∥ θ ^ s − θ ∗ ∥ 2 . \mathbb{E}\left[\left.\widetilde{X}^{\mathsf{T}}\theta^*\{p^*(X)-p_s^t(X)\}\right|\mathcal{F}_{s-1}\right] \le C_1\left\|\hat{\theta}_s-\theta^*\right\|_2. E [ X T θ ∗ { p ∗ ( X ) − p s t ( X )} F s − 1 ] ≤ C 1 θ ^ s − θ ∗ 2 .
For the remaining term, adding and subtracting empirical averages gives the same decomposition as in the proof of Proposition 16 . The two empirical-process terms are bounded by Δ s − 1 L , 1 \Delta_{s-1}^{L,1} Δ s − 1 L , 1 . Moreover, by empirical optimality of q ^ s \hat{q}_s q ^ s over q ∈ [ 0 , 1 ] q\in[0,1] q ∈ [ 0 , 1 ] ,
1 s − 1 ∑ i = 1 s − 1 X ~ i T θ ^ s { p s ∗ ( X i ) − p ^ s ( X i ) } ≤ 0. \frac{1}{s-1}\sum_{i=1}^{s-1}\widetilde{X}_i^{\mathsf{T}}\hat{\theta}_s\{p_s^*(X_i)-\hat{p}_s(X_i)\}\le0. s − 1 1 i = 1 ∑ s − 1 X i T θ ^ s { p s ∗ ( X i ) − p ^ s ( X i )} ≤ 0.
The remaining estimation term is bounded by
1 s − 1 ∑ i = 1 s − 1 ∣ X ~ i T ( θ ∗ − θ ^ s ) ∣ ≤ 2 ∥ θ ^ s − θ ∗ ∥ 2 . \frac{1}{s-1}\sum_{i=1}^{s-1}\left|\widetilde{X}_i^{\mathsf{T}}(\theta^*-\hat{\theta}_s)\right|\le\sqrt{2}\left\|\hat{\theta}_s-\theta^*\right\|_2. s − 1 1 i = 1 ∑ s − 1 X i T ( θ ∗ − θ ^ s ) ≤ 2 θ ^ s − θ ∗ 2 .
Combining the preceding bounds gives
R s ( p ∗ ) − R s ( p ^ s ) ≤ C 1 ∥ θ ^ s − θ ∗ ∥ 2 + 2 Δ s − 1 L , 1 + 2 ∥ θ ^ s − θ ∗ ∥ 2 . R_s(p^*)-R_s(\hat{p}_s)\le C_1\left\|\hat{\theta}_s-\theta^*\right\|_2+2\Delta_{s-1}^{L,1}+\sqrt{2}\left\|\hat{\theta}_s-\theta^*\right\|_2. R s ( p ∗ ) − R s ( p ^ s ) ≤ C 1 θ ^ s − θ ∗ 2 + 2 Δ s − 1 L , 1 + 2 θ ^ s − θ ∗ 2 .
Therefore,
R s ( p ∗ ) − R s ( p ^ s ) ≤ C θ , L ( 1 ) ∥ θ ^ s − θ ∗ ∥ 2 + 2 Δ s − 1 L , 1 , R_s(p^*)-R_s(\hat{p}_s)\le C_{\theta,L}^{(1)}\left\|\hat{\theta}_s-\theta^*\right\|_2+2\Delta_{s-1}^{L,1}, R s ( p ∗ ) − R s ( p ^ s ) ≤ C θ , L ( 1 ) θ ^ s − θ ∗ 2 + 2 Δ s − 1 L , 1 ,
where C θ , L ( 1 ) = C 1 + 2 C_{\theta,L}^{(1)}=C_1+\sqrt{2} C θ , L ( 1 ) = C 1 + 2 . This completes the proof.
Proof of Theorem 58 . The fairness guarantee follows because each policy is either a clipped function with slope L L L or the constant policy 1 / 2 1/2 1/2 .
If 2 T o = T 2T_o=T 2 T o = T , the claimed bound follows from the bounded one-round regret. Hence suppose 2 T o < T 2T_o<T 2 T o < T . For the stated choice T o = ⌈ K T log ( T / η ) ⌉ T_o=\lceil K\sqrt{T}\log(T/\eta)\rceil T o = ⌈ K T log ( T / η )⌉ with fixed K > 0 K>0 K > 0 , write D = C T ∨ C l o c ( 1 ) / { r l o c ( 1 ) } 2 D=C_T\vee C_{\mathrm{loc}}^{(1)}/\{r_{\mathrm{loc}}^{(1)}\}^2 D = C T ∨ C loc ( 1 ) / { r loc ( 1 ) } 2 . If
T ≥ C 0 log ( T / η ) , C 0 ≥ 16 max { 1 , K 2 , D 2 / K 2 } , T\ge C_0\log(T/\eta),\qquad C_0\ge16\max\{1,K^2,D^2/K^2\}, T ≥ C 0 log ( T / η ) , C 0 ≥ 16 max { 1 , K 2 , D 2 / K 2 } ,
then 2 T o ≤ T 2T_o\le T 2 T o ≤ T and T o ≥ 2 D log ( T / η ) ≥ D log ( 2 T / η ) T_o\ge2D\log(T/\eta)\ge D\log(2T/\eta) T o ≥ 2 D log ( T / η ) ≥ D log ( 2 T / η ) . Thus the required lower bound on T o T_o T o also holds with confidence parameter η / 2 \eta/2 η /2 .
Let S T \mathcal{S}_T S T denote the set of lazy update times up to time T T T . By Lemma 61 , after enlarging the constants in the assumed lower bound on T o T_o T o , with probability at least 1 − η / 2 1-\eta/2 1 − η /2 , for every s ∈ S T s\in\mathcal{S}_T s ∈ S T ,
∥ θ ^ s − θ ∗ ∥ 2 ≤ C log ( T / η ) s − 1 . \left\|\hat{\theta}_s-\theta^*\right\|_2\le C\sqrt{\frac{\log(T/\eta)}{s-1}}. θ ^ s − θ ∗ 2 ≤ C s − 1 log ( T / η ) .
By Lemma 62, together with a union bound over n = 2 , … , T n=2,\ldots,T n = 2 , … , T , with probability at least 1 − η / 2 1-\eta/2 1 − η /2 , for every n = 2 , … , T n=2,\ldots,T n = 2 , … , T ,
Δ n L , 1 ≤ C log ( e T 2 / η ) n . \Delta_n^{L,1}\le C\sqrt{\frac{\log(eT^2/\eta)}{n}}. Δ n L , 1 ≤ C n log ( e T 2 / η ) .
We work on the intersection of these two events. By Proposition 63, for every post-initialization round t > 2 T o t > 2T_o t > 2 T o with s = τ ( t ) s = \tau(t) s = τ ( t ) ,
R s ( p ∗ ) − R s ( p ^ s ) ≤ C log ( T / η ) s − 1 + C log ( e T 2 / η ) s − 1 . \mathfrak{R}_s(p^*) - \mathfrak{R}_s(\widehat{p}_s) \le C\sqrt{\frac{\log(T/\eta)}{s-1}} + C\sqrt{\frac{\log(eT^2/\eta)}{s-1}}. R s ( p ∗ ) − R s ( p s ) ≤ C s − 1 log ( T / η ) + C s − 1 log ( e T 2 / η ) .
The regret from the first 2 T o 2T_o 2 T o forced-exploration rounds is bounded by C 1 T o C_1T_o C 1 T o . Since the lazy update times grow geometrically and each policy computed at time s s s is used for at most s s s rounds,
R T ≤ C 1 T o + C 2 log ( e T 2 / η ) ∑ s ∈ S T s 1 / 2 ≤ C 1 T o + C 3 T log ( e T 2 / η ) . \mathfrak{R}_T \le C_1T_o + C_2\sqrt{\log(eT^2/\eta)}\sum_{s\in\mathcal{S}_T}s^{1/2} \le C_1T_o + C_3\sqrt{T\log(eT^2/\eta)}. R T ≤ C 1 T o + C 2 log ( e T 2 / η ) s ∈ S T ∑ s 1/2 ≤ C 1 T o + C 3 T log ( e T 2 / η ) .
This proves the theorem.
UCB algorithm for general covariates
Algorithm
Algorithm 6 is the one-dimensional version of Algorithm 3. Its optimistic update uses the 16 vertex pairs of the two confidence sets through Algorithm 4.
Theoretical guarantee
Theorem 64 (One-dimensional regret upper bound). Suppose Assumptions 1, 2, 3 and 5 hold with d = 1 d = 1 d = 1 . Implement Algorithm 6, with λ > 0 \lambda> 0 λ > 0 fixed, T o = ⌈ T ⌉ T_o = \lceil\sqrt{T}\rceil T o = ⌈ T ⌉ , and α T \alpha_T α T in (11) with $d = 1. Then the policy played at every round is ( L , ν ) (L,\nu) ( L , ν ) -fair. Moreover, for any η ∈ ( 0 , 1 / 2 ) \eta\in(0,1/2) η ∈ ( 0 , 1/2 ) , with probability at least $1-\eta$, we have that
R T ≤ C 1 ( 1 + 2 ) α T T { 2 log ( 1 + T λ ) + log ( 8 / η ) } + C 2 T log ( e T 2 / η ) , (82) \mathfrak{R}_T \le C_1(1+\sqrt{2})\alpha_T\sqrt{T\left\{2\log\left(1+\frac{T}{\lambda}\right)+\log(8/\eta)\right\}}+C_2\sqrt{T\log(eT^2/\eta)}, \tag*{(82)} R T ≤ C 1 ( 1 + 2 ) α T T { 2 log ( 1 + λ T ) + log ( 8/ η ) } + C 2 T log ( e T 2 / η ) , (82)
whenever T ≥ 16 ∨ C 3 ( 1 + 2 ) 4 α T 4 / ( κ X 2 { r l o c ( 1 ) } 4 ) T \ge16 \lor C_3(1+\sqrt{2})^4\alpha_T^4/(\kappa_X^2\{r_{\mathrm{loc}}^{(1)}\}^4) T ≥ 16 ∨ C 3 ( 1 + 2 ) 4 α T 4 / ( κ X 2 { r loc ( 1 ) } 4 ) . The constants depend only on L L L , σ \sigma σ , C β C_\beta C β , λ \lambda λ and the constants in Assumption 5. In particular, R T = O ~ p ( T ) \mathfrak{R}_T = \widetilde{O}_p(\sqrt{T}) R T = O p ( T ) .
Proof of Theorem 64
We use the enlarged set Θ ~ 1 d \widetilde{\Theta}_{1d} Θ 1 d and policies p θ , q ( 1 ) p_{\theta,q}^{(1)} p θ , q ( 1 ) defined in Appendix D.1, and take r l o c ( 1 ) ≤ 1 r_{\mathrm{loc}}^{(1)} \le1 r loc ( 1 ) ≤ 1 . For d = 1 d = 1 d = 1 , the ideal optimistic update is
( β ‾ s , 1 ∘ , β ‾ s , 2 ∘ , q s ∘ ) ∈ arg max b k ∈ C s , k , k ∈ { 1 , 2 } q ∈ [ 0 , 1 ] 1 s − 1 ∑ i = 1 s − 1 r b 1 − b 2 , q p ( 1 ) ( X i ) . (83) (\overline{\beta}_{s,1}^{\circ},\overline{\beta}_{s,2}^{\circ},q_s^\circ)\in\underset{\substack{b_k\in\mathcal{C}_{s,k},\ k\in\{1,2\}\\ q\in[0,1]}}{\operatorname{arg\,max}}\ \frac{1}{s-1}\sum_{i=1}^{s-1}r_{b_1-b_2,q}^{p^{(1)}}(X_i). \tag*{(83)} ( β s , 1 ∘ , β s , 2 ∘ , q s ∘ ) ∈ b k ∈ C s , k , k ∈ { 1 , 2 } q ∈ [ 0 , 1 ] arg max s − 1 1 i = 1 ∑ s − 1 r b 1 − b 2 , q p ( 1 ) ( X i ) . (83)
Lemma 65 (One-dimensional vertex optimisation). Suppose every feasible difference belongs to Θ ~ 1 d \widetilde{\Theta}_{1d} Θ 1 d . At each update time s s s of Algorithm 6, the selected model pair β ‾ s = ( β ‾ s , 1 , β ‾ s , 2 ) \overline{\beta}_s=(\overline{\beta}_{s,1},\overline{\beta}_{s,2}) β s = ( β s , 1 , β s , 2 ) and the policy p s p_s p s satisfy
1 s − 1 ∑ i = 1 s − 1 r β ‾ s , p s ( X i ) = sup b k ∈ C s , k , k ∈ { 1 , 2 } q ∈ [ 0 , 1 ] 1 s − 1 ∑ i = 1 s − 1 r b 1 − b 2 , q p ( 1 ) ( X i ) . (84) \frac{1}{s-1}\sum_{i=1}^{s-1}r_{\overline{\beta}_s,p_s}(X_i)=\sup_{\substack{b_k\in\mathcal{C}_{s,k},\ k\in\{1,2\}\\ q\in[0,1]}}\frac{1}{s-1}\sum_{i=1}^{s-1}r_{b_1-b_2,q}^{p^{(1)}}(X_i). \tag*{(84)} s − 1 1 i = 1 ∑ s − 1 r β s , p s ( X i ) = b k ∈ C s , k , k ∈ { 1 , 2 } q ∈ [ 0 , 1 ] sup s − 1 1 i = 1 ∑ s − 1 r b 1 − b 2 , q p ( 1 ) ( X i ) . (84)
Proof. By Lemma 57 , the supremum in (84) equals max b k ∈ C s , k F s ( b 1 , b 2 ) \max_{b_k\in\mathcal{C}_{s,k}}F_s(b_1,b_2) max b k ∈ C s , k F s ( b 1 , b 2 ) . Similar as Lemma 5, joint convexity of F s F_s F s and the vertex representation in Lemma 31 show that this maximum is attained at one of the 16 vertex pairs. By the optimality of z v z_v z v , each score satisfies S v = F s ( v 1 , v 2 ) S_v=F_s(v_1,v_2) S v = F s ( v 1 , v 2 ) . Selecting the pair with the largest score and applying the canonical domination in Lemma 57 gives the lower bound in (84) . The reverse bound follows because the returned model-policy pair is feasible in (83) .
Algorithm 6 Lazy one-dimensional polyhedral UCB algorithm (d = 1 d = 1 d = 1 )
INPUT: Time horizon T T T , fairness parameter ( L , ν ) (L, \nu) ( L , ν ) , ridge parameter λ \lambda λ , exploration period T o T_o T o , confidence radius α T \alpha_T α T from (11) with d = 1 d = 1 d = 1
Set s 0 ← 2 T o + 1 s_0 \leftarrow2T_o + 1 s 0 ← 2 T o + 1 .
for k ∈ { 1 , 2 } k \in\{1,2\} k ∈ { 1 , 2 } do
V k ← λ I 2 V_k \leftarrow\lambda I_2 V k ← λ I 2 , B k ← 0 B_k \leftarrow0 B k ← 0 .
end for
for t = 1 , 2 , … , 2 T o t = 1,2,\ldots,2T_o t = 1 , 2 , … , 2 T o do
Observe X t X_t X t and set X ~ t = ( 1 , X t ) T \widetilde{X}_t = (1,X_t)^{\mathsf{T}} X t = ( 1 , X t ) T .
Play arm 1 with probability 1 / 2 1/2 1/2 and arm 2 otherwise; observe Y t Y_t Y t .
Update V A t ← V A t + X ~ t X ~ t T V_{A_t} \leftarrow V_{A_t} + \widetilde{X}_t\widetilde{X}_t^{\mathsf{T}} V A t ← V A t + X t X t T and B A t ← B A t + X ~ t Y t B_{A_t} \leftarrow B_{A_t} + \widetilde{X}_tY_t B A t ← B A t + X t Y t .
end for
Set s ← s 0 s \leftarrow s_0 s ← s 0 and, for k ∈ { 1 , 2 } k \in\{1,2\} k ∈ { 1 , 2 } , set V s , k ← V k V_{s,k} \leftarrow V_k V s , k ← V k and define ℓ s , k : = log det ( V s , k ) \ell_{s,k} := \log\det(V_{s,k}) ℓ s , k := log det ( V s , k ) .
for t = s 0 , … , T t = s_0,\ldots,T t = s 0 , … , T do
if t = s 0 t = s_0 t = s 0 or t ≥ 2 s t \ge2s t ≥ 2 s or ∃ k ∈ { 1 , 2 } : log det ( V k ) − ℓ s , k ≥ log 2 \exists k \in\{1,2\} : \log\det(V_k) - \ell_{s,k} \ge\log2 ∃ k ∈ { 1 , 2 } : log det ( V k ) − ℓ s , k ≥ log 2 then
Set s ← t s \leftarrow t s ← t .
For k ∈ { 1 , 2 } k \in\{1,2\} k ∈ { 1 , 2 } , set V s , k ← V k V_{s,k} \leftarrow V_k V s , k ← V k , β ^ s , k ← V k − 1 B k \widehat{\beta}_{s,k} \leftarrow V_k^{-1}B_k β s , k ← V k − 1 B k , ℓ s , k ← log det ( V s , k ) \ell_{s,k} \leftarrow\log\det(V_{s,k}) ℓ s , k ← log det ( V s , k ) .
Apply the vertex enumeration and LP scoring steps of Algorithm 4 with d = 1 d = 1 d = 1 to select ( β ‾ s , 1 , β ‾ s , 2 , z s ) (\overline{\beta}_{s,1},\overline{\beta}_{s,2},z_s) ( β s , 1 , β s , 2 , z s ) ; set θ s ← β ‾ s , 1 − β ‾ s , 2 = ( a s , c s ) \theta_s \leftarrow\overline{\beta}_{s,1} - \overline{\beta}_{s,2} = (a_s,c_s) θ s ← β s , 1 − β s , 2 = ( a s , c s ) .
if c s > 0 c_s > 0 c s > 0 and − a s ∈ [ 0 , c s ] -a_s \in[0,c_s] − a s ∈ [ 0 , c s ] then
Set x θ s ← − a s / c s x_{\theta_s} \leftarrow-a_s/c_s x θ s ← − a s / c s and q s ← min { 1 , min 1 ≤ i ≤ s − 1 [ z s , i + L ∣ x θ s − X i ∣ ] } q_s \leftarrow\min\{1,\min_{1\le i\le s-1}[z_{s,i} + L|x_{\theta_s} - X_i|]\} q s ← min { 1 , min 1 ≤ i ≤ s − 1 [ z s , i + L ∣ x θ s − X i ∣ ]} .
Set p s ( ⋅ ) ← p θ s , q s ( 1 ) ( ⋅ ) p_s(\cdot) \leftarrow p_{\theta_s,q_s}^{(1)}(\cdot) p s ( ⋅ ) ← p θ s , q s ( 1 ) ( ⋅ ) .
else
Set p s ( ⋅ ) ← 1 / 2 p_s(\cdot) \leftarrow1/2 p s ( ⋅ ) ← 1/2 .
end if
Set p ^ ← p s \widehat{p} \leftarrow p_s p ← p s .
end if
Observe X t X_t X t and set X ~ t = ( 1 , X t ) T \widetilde{X}_t = (1,X_t)^{\mathsf{T}} X t = ( 1 , X t ) T .
Play arm 1 with probability p ^ ( X t ) \widehat{p}(X_t) p ( X t ) and arm 2 otherwise; observe Y t Y_t Y t .
Update V A t ← V A t + X ~ t X ~ t T V_{A_t} \leftarrow V_{A_t} + \widetilde{X}_t\widetilde{X}_t^{\mathsf{T}} V A t ← V A t + X t X t T and B A t ← B A t + X ~ t Y t B_{A_t} \leftarrow B_{A_t} + \widetilde{X}_tY_t B A t ← B A t + X t Y t .
end for
Define the functional class
F U C B , 1 = { x ↦ r b θ , q ( 1 ) ( x ) : ∥ b k ∥ 2 ≤ C β + 1 , k ∈ { 1 , 2 } , θ = b 1 − b 2 ∈ Θ ~ 1 d , q ∈ [ 0 , 1 ] } \mathcal{F}_{\mathrm{UCB},1}=\left\{x\mapsto r_{b_{\theta,q}^{(1)}}(x):\lVert b_k\rVert_2\le C_\beta+1,\ k\in\{1,2\},\ \theta=b_1-b_2\in\widetilde{\Theta}_{1d},\ q\in[0,1]\right\} F UCB , 1 = { x ↦ r b θ , q ( 1 ) ( x ) : ∥ b k ∥ 2 ≤ C β + 1 , k ∈ { 1 , 2 } , θ = b 1 − b 2 ∈ Θ 1 d , q ∈ [ 0 , 1 ] }
and
Δ n U C B , 1 = sup f ∈ F U C B , 1 ∣ 1 n ∑ i = 1 n f ( X i ) − E { f ( X ) } ∣ . (85) \Delta_n^{\mathrm{UCB},1}=\sup_{f\in\mathcal{F}_{\mathrm{UCB},1}}\left|\frac{1}{n}\sum_{i=1}^{n}f(X_i)-\mathbb{E}\{f(X)\}\right|. \tag*{(85)} Δ n UCB , 1 = f ∈ F UCB , 1 sup n 1 i = 1 ∑ n f ( X i ) − E { f ( X )} . (85)
Lemma 66 (Control of Δ n U C B , 1 \Delta_n^{\mathrm{UCB},1} Δ n UCB , 1 ). Suppose Assumptions 3 and 5 hold. Let X 1 , … , X n X_1,\ldots,X_n X 1 , … , X n be i.i.d. from an arbitrary Borel probability measure P X P_X P X supported on [ 0 , 1 ] [0,1] [ 0 , 1 ] . There exists a constant C < ∞ C<\infty C < ∞ , depending only on L L L , C β C_\beta C β and the constants defining Θ ~ 1 d \widetilde{\Theta}_{1d} Θ 1 d , and independent of P X P_X P X , such that, for every n ≥ 2 n\ge2 n ≥ 2 and η ∈ ( 0 , 1 ) \eta\in(0,1) η ∈ ( 0 , 1 ) , with probability at least 1 − η 1-\eta 1 − η ,
Δ n U C B , 1 ≤ C log ( e n / η ) n . \Delta_n^{\mathrm{UCB},1}\le C\sqrt{\frac{\log(en/\eta)}{n}}. Δ n UCB , 1 ≤ C n log ( e n / η ) .
Proof. The proof follows the same argument as in Lemmas 40 to 42. We verify the entropy bound for the one-dimensional policy class and evaluate the resulting entropy integral.
For x 0 , x 0 ′ ∈ [ ρ ~ 0 , 1 − ρ ~ 0 ] x_0,x'_0\in[\widetilde{\rho}_0,1-\widetilde{\rho}_0] x 0 , x 0 ′ ∈ [ ρ 0 , 1 − ρ 0 ] and q , q ′ ∈ [ 0 , 1 ] q,q'\in[0,1] q , q ′ ∈ [ 0 , 1 ] , the nonexpansiveness of the clipping map gives
sup x ∈ [ 0 , 1 ] ∣ [ q + L ( x − x 0 ) ] [ 0 , 1 ] − [ q ′ + L ( x − x 0 ′ ) ] [ 0 , 1 ] ∣ ≤ ∣ q − q ′ ∣ + L ∣ x 0 − x 0 ′ ∣ . \sup_{x\in[0,1]}\left|[q+L(x-x_0)]_{[0,1]}-[q'+L(x-x'_0)]_{[0,1]}\right|\le|q-q'|+L|x_0-x'_0|. x ∈ [ 0 , 1 ] sup [ q + L ( x − x 0 ) ] [ 0 , 1 ] − [ q ′ + L ( x − x 0 ′ ) ] [ 0 , 1 ] ≤ ∣ q − q ′ ∣ + L ∣ x 0 − x 0 ′ ∣.
For L > 0 L>0 L > 0 , discretising q q q and the boundary location x 0 x_0 x 0 at resolutions of order ε \varepsilon ε and ε / L \varepsilon/L ε / L , respectively, gives a sup-norm cover of the one-dimensional policy class with log covering number of order log ( 1 / ε ) \log(1/\varepsilon) log ( 1/ ε ) . If L = 0 L=0 L = 0 , the policy is constant and discretising q q q alone gives the same bound. Since we have that r b θ , q ( 1 ) ( x ) = x ~ ⊤ b 2 + ( x ~ ⊤ θ ) p θ , q ( 1 ) ( x ) r_{b_{\theta,q}^{(1)}}(x)=\widetilde{x}^{\top}b_2+(\widetilde{x}^{\top}\theta)p_{\theta,q}^{(1)}(x) r b θ , q ( 1 ) ( x ) = x ⊤ b 2 + ( x ⊤ θ ) p θ , q ( 1 ) ( x ) , and b 2 b_2 b 2 and θ \theta θ range over bounded subsets of R 2 \mathbb{R}^2 R 2 , discretising these finite-dimensional parameters as in Lemma 40 preserves the logarithmic entropy order. As in that proof, the resulting uniform approximations give finite brackets of pointwise width ε \varepsilon ε . Therefore, uniformly over all Borel probability measures P P P on [ 0 , 1 ] [0,1] [ 0 , 1 ] , we have that
log N [ ] ( ε , F U C B , 1 , L 2 ( P ) ) ≤ C 2 log ( C 2 / ε ) , ε ∈ ( 0 , 1 ) . \log N_{[]}(\varepsilon,\mathcal{F}_{\mathrm{UCB},1},L_2(P))\le C_2\log(C_2/\varepsilon),\qquad\varepsilon\in(0,1). log N [ ] ( ε , F UCB , 1 , L 2 ( P )) ≤ C 2 log ( C 2 / ε ) , ε ∈ ( 0 , 1 ) .
For any deterministic sample x 1 , … , x n ∈ [ 0 , 1 ] x_1,\ldots,x_n\in[0,1] x 1 , … , x n ∈ [ 0 , 1 ] , let P n = n − 1 ∑ i = 1 n δ x i P_n=n^{-1}\sum_{i=1}^n\delta_{x_i} P n = n − 1 ∑ i = 1 n δ x i . Selecting one class member from each nonempty bracket gives the same logarithmic bound for the L 2 ( P n ) L_2(P_n) L 2 ( P n ) covering number. Moreover, every f ∈ F U C B , 1 f\in\mathcal{F}_{\mathrm{UCB},1} f ∈ F UCB , 1 satisfies ∥ f ∥ ∞ ≤ B 1 : = 2 ( C β + 1 ) \lVert f\rVert_\infty\le B_1:=\sqrt{2}(C_\beta+1) ∥ f ∥ ∞ ≤ B 1 := 2 ( C β + 1 ) . Applying the conditional sub-Gaussian increment calculation and truncated Dudley argument in the proof of Lemma 41 , with this logarithmic entropy bound, gives
R ^ n ( F U C B , 1 ) ≤ C inf 0 < α < 1 { α + 1 n [ 1 + ∫ α 1 { log ( C 2 / ε ) } 1 / 2 d ε ] } . \widehat{\mathcal{R}}_n(\mathcal{F}_{\mathrm{UCB},1})\le C\inf_{0<\alpha<1}\left\{\alpha+\frac{1}{\sqrt{n}}\left[1+\int_\alpha^1\{\log(C_2/\varepsilon)\}^{1/2}\,d\varepsilon\right]\right\}. R n ( F UCB , 1 ) ≤ C 0 < α < 1 inf { α + n 1 [ 1 + ∫ α 1 { log ( C 2 / ε ) } 1/2 d ε ] } .
The additional constant inside the brackets accounts for entropy scales above one. Since the integral remains finite as α → 0 \alpha\to0 α → 0 , we obtain R ^ n ( F U C B , 1 ) ≤ C 3 n − 1 / 2 \widehat{\mathcal{R}}_n(\mathcal{F}_{\mathrm{UCB},1})\le C_3n^{-1/2} R n ( F UCB , 1 ) ≤ C 3 n − 1/2 for every sample. The finite brackets of pointwise width ε \varepsilon ε imply separability in the sup-norm. Thus the empirical and Rademacher suprema are measurable, and the symmetrisation and bounded-differences argument in Lemma 42 applies and gives, with probability at least 1 − η 1-\eta 1 − η ,
Δ n U C B , 1 ≤ 2 C 3 n − 1 / 2 + B 1 2 log ( 1 / η ) n ≤ C log ( e n / η ) n . \Delta_n^{\mathrm{UCB},1}\le2C_3n^{-1/2}+B_1\sqrt{\frac{2\log(1/\eta)}{n}}\le C\sqrt{\frac{\log(en/\eta)}{n}}. Δ n UCB , 1 ≤ 2 C 3 n − 1/2 + B 1 n 2 log ( 1/ η ) ≤ C n log ( e n / η ) .
Proof of Theorem 64. The fairness guarantee follows because each policy is either a clipped function with slope L L L or the constant policy 1 / 2 1/2 1/2 .
Let B = 2 ( C β + 1 ) B=\sqrt{2}(C_{\beta}+1) B = 2 ( C β + 1 ) and choose p ∗ = p θ ∗ , q ∗ ( 1 ) p^{*}=p_{\theta^{*},q^{*}}^{(1)} p ∗ = p θ ∗ , q ∗ ( 1 ) as in Lemma 57 . Similar as in the proof of Theorem 6 , the first 2 T o 2T_{o} 2 T o rounds contribute at most 4 B T o ≤ C T 4BT_{o}\le C\sqrt{T} 4 B T o ≤ C T to the regret.
We use the events E c o n f \mathcal{E}_{\mathrm{conf}} E conf , E i n i t \mathcal{E}_{\mathrm{init}} E init , and E m a r t \mathcal{E}_{\mathrm{mart}} E mart in (51) , (57) , and (53) , with d = 1 d=1 d = 1 and τ ( t ) \tau(t) τ ( t ) the most recent update time of Algorithm 6. The confidence and initial exploration arguments are similar as Corollary 44 and Lemma 34 . In particular, on E c o n f ∩ E i n i t \mathcal{E}_{\mathrm{conf}}\cap\mathcal{E}_{\mathrm{init}} E conf ∩ E init , for every update time s s s and every b k ∈ C s , k b_k\in\mathcal{C}_{s,k} b k ∈ C s , k , we have that
∥ ( b 1 − b 2 ) − θ ∗ ∥ 2 ≤ 2 ( 1 + 2 ) α T λ + κ X T o / 4 ≤ 4 ( 1 + 2 ) α T κ X T 1 / 4 ≤ 2 r l o c ( 1 ) . \left\|(b_1-b_2)-\theta^{*}\right\|_2\le\frac{2(1+\sqrt{2})\alpha_T}{\sqrt{\lambda+\kappa_XT_o/4}}\le\frac{4(1+\sqrt{2})\alpha_T}{\sqrt{\kappa_XT^{1/4}}}\le2r_{\mathrm{loc}}^{(1)}. ∥ ( b 1 − b 2 ) − θ ∗ ∥ 2 ≤ λ + κ X T o /4 2 ( 1 + 2 ) α T ≤ κ X T 1/4 4 ( 1 + 2 ) α T ≤ 2 r loc ( 1 ) .
where the last inequality follows from the assumed lower bound on T T T , after choosing C 3 C_3 C 3 sufficiently large. The same argument gives ∥ b k − β k ∗ ∥ 2 ≤ r l o c ( 1 ) ≤ 1 \left\|b_k-\beta_k^{*}\right\|_2\le r_{\mathrm{loc}}^{(1)}\le1 ∥ b k − β k ∗ ∥ 2 ≤ r loc ( 1 ) ≤ 1 and hence ∥ b k ∥ 2 ≤ C β + 1 \left\|b_k\right\|_2\le C_{\beta}+1 ∥ b k ∥ 2 ≤ C β + 1 . Thus every feasible difference belongs to Θ ~ 1 d \widetilde{\Theta}_{1d} Θ 1 d , so the boundary trace is well defined.
By Lemma 66 and a union bound over n = 2 , … , T n=2,\ldots,T n = 2 , … , T , the event
E e m p ( 1 ) = { Δ n U C B , 1 ≤ C log ( 8 e n T / η ) n , for every n ∈ { 2 , … , T } } \mathcal{E}_{\mathrm{emp}}^{(1)}=\left\{\Delta_n^{\mathrm{UCB},1}\le C\sqrt{\frac{\log(8enT/\eta)}{n}},\ \text{for every }n\in\{2,\ldots,T\}\right\} E emp ( 1 ) = { Δ n UCB , 1 ≤ C n log ( 8 e n T / η ) , for every n ∈ { 2 , … , T } }
has probability at least 1 − η / 8 1-\eta/8 1 − η /8 . Similar as in the proof of Theorem 6 , the same lower bound on T T T gives P ( E i n i t c ) ≤ η / 4 P(\mathcal{E}_{\mathrm{init}}^{c})\le\eta/4 P ( E init c ) ≤ η /4 , while P ( E c o n f c ) ≤ η / 8 P(\mathcal{E}_{\mathrm{conf}}^{c})\le\eta/8 P ( E conf c ) ≤ η /8 and P ( E m a r t c ) ≤ η / 8 P(\mathcal{E}_{\mathrm{mart}}^{c})\le\eta/8 P ( E mart c ) ≤ η /8 . Hence, a union bound argument gives
P ( E c o n f ∩ E i n i t ∩ E m a r t ∩ E e m p ( 1 ) ) ≥ 1 − 5 η / 8 ≥ 1 − η . \mathbb{P}\left(\mathcal{E}_{\mathrm{conf}}\cap\mathcal{E}_{\mathrm{init}}\cap\mathcal{E}_{\mathrm{mart}}\cap\mathcal{E}_{\mathrm{emp}}^{(1)}\right)\ge1-5\eta/8\ge1-\eta. P ( E conf ∩ E init ∩ E mart ∩ E emp ( 1 ) ) ≥ 1 − 5 η /8 ≥ 1 − η .
We work on this intersection for the rest of the proof.
Since the true model-policy pair is feasible in (83) , Lemma 65 gives empirical optimism similar as Lemma 48 . Note that both r β ∗ , p ∗ r_{\beta^{*},p^{*}} r β ∗ , p ∗ and r β ˉ s , p s r_{\bar{\beta}_{s},p_s} r β ˉ s , p s belong to F U C B , 1 \mathcal{F}_{\mathrm{UCB},1} F UCB , 1 , therefore, similar as Lemma 49 , for t ≥ s 0 t\ge s_0 t ≥ s 0 and s = τ ( t ) s=\tau(t) s = τ ( t ) , we have that
R ( p ∗ ) − R ( p s ) ≤ 2 Δ s − 1 U C B , 1 + { ( 1 + 2 ) α T + 2 B } E [ min { 1 , ∥ X ~ t ∥ V s , A t − 1 } | F t − 1 ] . R(p^{*})-R(p_s)\le2\Delta_{s-1}^{\mathrm{UCB},1}+\left\{(1+\sqrt{2})\alpha_T+2B\right\}\mathbb{E}\left[\min\left\{1,\left\|\widetilde{X}_t\right\|_{V_{s,A_t}^{-1}}\right\}\middle|\mathcal{F}_{t-1}\right]. R ( p ∗ ) − R ( p s ) ≤ 2 Δ s − 1 UCB , 1 + { ( 1 + 2 ) α T + 2 B } E [ min { 1 , X t V s , A t − 1 } F t − 1 ] .
Similar as Lemma 50 , we have that
2 ∑ t = s 0 T Δ τ ( t ) − 1 U C B , 1 ≤ C 1 T log ( e T 2 / η ) . 2\sum_{t=s_0}^{T}\Delta_{\tau(t)-1}^{\mathrm{UCB},1}\le C_1\sqrt{T\log(eT^2/\eta)}. 2 t = s 0 ∑ T Δ τ ( t ) − 1 UCB , 1 ≤ C 1 T log ( e T 2 / η ) .
Finally, the determinant comparison and width summation are similar as Lemmas 46 and 47. Combining them with E m a r t \mathcal{E}_{\mathrm{mart}} E mart gives
∑ t = s 0 T E [ min { 1 , ∥ X ~ t ∥ V τ ( t ) , A t − 1 } | F t − 1 ] ≤ C 2 T log ( 1 + T λ ) + 2 T log ( 8 / η ) . \sum_{t=s_0}^{T}\mathbb{E}\left[\min\left\{1,\left\|\widetilde{X}_t\right\|_{V_{\tau(t),A_t}^{-1}}\right\}\middle|\mathcal{F}_{t-1}\right]\le C\sqrt{2T\log\left(1+\frac{T}{\lambda}\right)}+\sqrt{2T\log(8/\eta)}. t = s 0 ∑ T E [ min { 1 , X t V τ ( t ) , A t − 1 } F t − 1 ] ≤ C 2 T log ( 1 + λ T ) + 2 T log ( 8/ η ) .
Since α T ≥ C β λ \alpha_T\ge C_{\beta}\sqrt{\lambda} α T ≥ C β λ and λ \lambda λ is fixed, ( 1 + 2 ) α T + 2 B ≤ C ( 1 + 2 ) α T (1+\sqrt{2})\alpha_T+2B\le C(1+\sqrt{2})\alpha_T ( 1 + 2 ) α T + 2 B ≤ C ( 1 + 2 ) α T . Summing the one-step bounds and adding the exploration regret proves (82) . The stated rate follows from α T = O ( log ( T / η ) ) \alpha_T=O(\sqrt{\log(T/\eta)}) α T = O ( log ( T / η ) ) . □
Structural results under arbitrary context laws
Throughout this section, P X P_X P X is an arbitrary Borel probability law on X = [ 0 , 1 ] d \mathcal{X}=[0,1]^d X = [ 0 , 1 ] d and S = supp ( P X ) \mathcal{S}=\operatorname{supp}(P_X) S = supp ( P X ) .
Lemma 67 (Equivalence between fairness over S \mathcal{S} S and over X \mathcal{X} X ). Let S = supp ( P X ) \mathcal{S}=\operatorname{supp}(P_X) S = supp ( P X ) . Suppose p S : S → [ 0 , 1 ] p_{\mathcal{S}}:\mathcal{S}\to[0,1] p S : S → [ 0 , 1 ] satisfies ∣ p S ( x ) − p S ( x ′ ) ∣ ≤ L ∥ x − x ′ ∥ ν \lvert p_{\mathcal{S}}(x)-p_{\mathcal{S}}(x')\rvert\le L\lVert x-x'\rVert_{\nu} ∣ p S ( x ) − p S ( x ′ )∣ ≤ L ∥ x − x ′ ∥ ν for all x , x ′ ∈ S x,x'\in\mathcal{S} x , x ′ ∈ S , then the extended policy
p ˉ ( x ) = [ inf z ∈ S { p S ( z ) + L ∥ x − z ∥ ν } ] [ 0 , 1 ] , x ∈ [ 0 , 1 ] d , \bar{p}(x)=\left[\inf_{z\in\mathcal{S}}\{p_{\mathcal{S}}(z)+L\lVert x-z\rVert_{\nu}\}\right]_{[0,1]},\qquad x\in[0,1]^d, p ˉ ( x ) = [ z ∈ S inf { p S ( z ) + L ∥ x − z ∥ ν } ] [ 0 , 1 ] , x ∈ [ 0 , 1 ] d , belongs to P L , ν \mathcal{P}_{L,\nu} P L , ν and satisfies p ˉ = p S \bar{p}=p_{\mathcal{S}} p ˉ = p S on S \mathcal{S} S . Consequently, maximising R ( p ) R(p) R ( p ) over P L , ν \mathcal{P}_{L,\nu} P L , ν is equivalent to maximising over L L L -Lipschitz maps defined only on S \mathcal{S} S .
Proof. The standard McShane argument [40 ] shows that x ↦ inf z ∈ S { p S ( z ) + L ∥ x − z ∥ ν } x\mapsto\inf_{z\in\mathcal{S}}\{p_{\mathcal{S}}(z)+L\lVert x-z\rVert_{\nu}\} x ↦ inf z ∈ S { p S ( z ) + L ∥ x − z ∥ ν } is L L L -Lipschitz. Clipping to [ 0 , 1 ] [0,1] [ 0 , 1 ] is nonexpansive, so it holds that p ˉ ∈ P L , ν \bar{p}\in\mathcal{P}_{L,\nu} p ˉ ∈ P L , ν .
Fix x ∈ S x\in\mathcal{S} x ∈ S . Taking z = x z=x z = x , by construction, we have that p ˉ ( x ) ≤ p S ( x ) \bar{p}(x)\le p_{\mathcal{S}}(x) p ˉ ( x ) ≤ p S ( x ) . Conversely, the Lipschitz property of p S p_{\mathcal{S}} p S gives p S ( z ) + L ∥ x − z ∥ ν ≥ p S ( x ) p_{\mathcal{S}}(z)+L\lVert x-z\rVert_{\nu}\ge p_{\mathcal{S}}(x) p S ( z ) + L ∥ x − z ∥ ν ≥ p S ( x ) for every z ∈ S z\in\mathcal{S} z ∈ S , so it holds that the infimum is at least p S ( x ) p_{\mathcal{S}}(x) p S ( x ) . Hence, for every x ∈ S x\in\mathcal{S} x ∈ S , we have that inf z ∈ S { p S ( z ) + L ∥ x − z ∥ ν } = p S ( x ) \inf_{z\in\mathcal{S}}\{p_{\mathcal{S}}(z)+L\lVert x-z\rVert_{\nu}\}=p_{\mathcal{S}}(x) inf z ∈ S { p S ( z ) + L ∥ x − z ∥ ν } = p S ( x ) . Since p S ( x ) ∈ [ 0 , 1 ] p_{\mathcal{S}}(x)\in[0,1] p S ( x ) ∈ [ 0 , 1 ] , clipping does not change this value, and therefore p ˉ ( x ) = p S ( x ) \bar{p}(x)=p_{\mathcal{S}}(x) p ˉ ( x ) = p S ( x ) for every x ∈ S x\in\mathcal{S} x ∈ S .
Note that X ∈ S X\in\mathcal{S} X ∈ S almost surely and the population reward depends only on the restriction of the policy to S \mathcal{S} S , therefore, the final claim holds.
Theorem 68 (Boundary randomisation). Suppose P X P_X P X is an arbitrary Borel probability law on [ 0 , 1 ] d [0,1]^d [ 0 , 1 ] d , L > 0 L>0 L > 0 , and let p ∗ ∈ arg max p ∈ P L , ν R ( p ) p^*\in\arg\max_{p\in\mathcal{P}_{L,\nu}}R(p) p ∗ ∈ arg max p ∈ P L , ν R ( p ) . Define the effective randomisation region by M S ( p ∗ ) = { x ∈ S : 0 < p ∗ ( x ) < 1 } \mathcal{M}_{\mathcal{S}}(p^*)=\{x\in\mathcal{S}:0<p^*(x)<1\} M S ( p ∗ ) = { x ∈ S : 0 < p ∗ ( x ) < 1 } . If H θ ∗ = ∅ \mathcal{H}_{\theta^*}=\varnothing H θ ∗ = ∅ , then Δ ∗ ( ⋅ ) \Delta^*(\cdot) Δ ∗ ( ⋅ ) has constant sign on [ 0 , 1 ] d [0,1]^d [ 0 , 1 ] d and a deterministic constant policy is optimal. If H θ ∗ ≠ ∅ \mathcal{H}_{\theta^*}\ne\varnothing H θ ∗ = ∅ , then it holds that
M S ( p ∗ ) ⊆ { x ∈ S : dist ν ( x , H θ ∗ ) ≤ 1 L } . (86) \mathcal{M}_{\mathcal{S}}(p^*)\subseteq\left\{x\in\mathcal{S}:\operatorname{dist}_{\nu}(x,\mathcal{H}_{\theta^*})\le\frac{1}{L}\right\}. \tag*{(86)} M S ( p ∗ ) ⊆ { x ∈ S : dist ν ( x , H θ ∗ ) ≤ L 1 } . (86) More precisely, for every x ∈ S x\in\mathcal{S} x ∈ S with dist ν ( x , H θ ∗ ) > 1 / L \operatorname{dist}_{\nu}(x,\mathcal{H}_{\theta^*})>1/L dist ν ( x , H θ ∗ ) > 1/ L , one has p ∗ ( x ) = 1 p^*(x)=1 p ∗ ( x ) = 1 if Δ ∗ ( x ) > 0 \Delta^*(x)>0 Δ ∗ ( x ) > 0 and p ∗ ( x ) = 0 p^*(x)=0 p ∗ ( x ) = 0 if Δ ∗ ( x ) < 0 \Delta^*(x)<0 Δ ∗ ( x ) < 0 .
Proof. For every policy p p p , R ( p ) = E [ X ~ T β 2 ∗ ] + E [ Δ ∗ ( X ) p ( X ) ] R(p)=\mathbb{E}[\widetilde{X}^{\mathsf T}\beta_2^*]+\mathbb{E}[\Delta^*(X)p(X)] R ( p ) = E [ X T β 2 ∗ ] + E [ Δ ∗ ( X ) p ( X )] , so it suffices to consider the second term.
Step 1. The case H θ ∗ = ∅ \mathcal{H}_{\theta^*}=\varnothing H θ ∗ = ∅ . By continuity of the affine function Δ ∗ \Delta^* Δ ∗ on the connected set [ 0 , 1 ] d [0,1]^d [ 0 , 1 ] d , its sign is constant. The objective is then pointwise increasing in p p p when Δ ∗ > 0 \Delta^*>0 Δ ∗ > 0 and pointwise decreasing in p p p when Δ ∗ < 0 \Delta^*<0 Δ ∗ < 0 , so p = 1 p=1 p = 1 or p = 0 p=0 p = 0 , respectively, is optimal.
Step 2. Saturation on the positive side. Suppose H θ ∗ ≠ ∅ \mathcal{H}_{\theta^*}\ne\varnothing H θ ∗ = ∅ . Fix x 0 ∈ S x_0\in\mathcal{S} x 0 ∈ S such that dist ν ( x 0 , H θ ∗ ) > 1 / L \operatorname{dist}_{\nu}(x_0,\mathcal{H}_{\theta^*})>1/L dist ν ( x 0 , H θ ∗ ) > 1/ L and Δ ∗ ( x 0 ) > 0 \Delta^*(x_0)>0 Δ ∗ ( x 0 ) > 0 . Suppose for contradiction that p ∗ ( x 0 ) < 1 p^*(x_0)<1 p ∗ ( x 0 ) < 1 . By continuity of Δ ∗ \Delta^* Δ ∗ and p ∗ p^* p ∗ , and since dist ν ( x 0 , H θ ∗ ) > 1 / L \operatorname{dist}_{\nu}(x_0,\mathcal{H}_{\theta^*})>1/L dist ν ( x 0 , H θ ∗ ) > 1/ L , there exist r > 0 r>0 r > 0 and η > 0 \eta>0 η > 0 such that A = B ν ( x 0 , r ) ‾ ∩ [ 0 , 1 ] d A=\overline{B_{\nu}(x_0,r)}\cap[0,1]^d A = B ν ( x 0 , r ) ∩ [ 0 , 1 ] d satisfies Δ ∗ ( x ) > 0 \Delta^*(x)>0 Δ ∗ ( x ) > 0 and p ∗ ( x ) ≤ 1 − η p^*(x)\le1-\eta p ∗ ( x ) ≤ 1 − η for all x ∈ A x\in A x ∈ A , and dist ν ( A , H θ ∗ ) > 1 / L \operatorname{dist}_{\nu}(A,\mathcal{H}_{\theta^*})>1/L dist ν ( A , H θ ∗ ) > 1/ L . Define
ϕ A ( x ) = max { 0 , 1 − L dist ν ( x , A ) } . \phi_A(x)=\max\{0,1-L\operatorname{dist}_{\nu}(x,A)\}. ϕ A ( x ) = max { 0 , 1 − L dist ν ( x , A )} .
Then ϕ A \phi_A ϕ A is L L L -Lipschitz, takes values in [ 0 , 1 ] [0,1] [ 0 , 1 ] , and equals one on A A A . We claim that Δ ∗ ( x ) > 0 \Delta^*(x)>0 Δ ∗ ( x ) > 0 whenever ϕ A ( x ) > 0 \phi_A(x)>0 ϕ A ( x ) > 0 . Indeed, if ϕ A ( x ) > 0 \phi_A(x)>0 ϕ A ( x ) > 0 , then dist ν ( x , A ) < 1 / L \operatorname{dist}_{\nu}(x,A)<1/L dist ν ( x , A ) < 1/ L . If Δ ∗ ( x ) ≤ 0 \Delta^*(x)\le0 Δ ∗ ( x ) ≤ 0 , choose a ∈ A a\in A a ∈ A with ∥ x − a ∥ ν < 1 / L \lVert x-a\rVert_{\nu}<1/L ∥ x − a ∥ ν < 1/ L . By convexity of X \mathcal{X} X and continuity of Δ ∗ \Delta^* Δ ∗ , the line segment between a a a and x x x intersects H θ ∗ \mathcal{H}_{\theta^*} H θ ∗ at some point h h h . This gives dist ν ( A , H θ ∗ ) ≤ ∥ a − h ∥ ν ≤ ∥ a − x ∥ ν < 1 / L \operatorname{dist}_{\nu}(A,\mathcal{H}_{\theta^*})\le\lVert a-h\rVert_{\nu}\le\lVert a-x\rVert_{\nu}<1/L dist ν ( A , H θ ∗ ) ≤ ∥ a − h ∥ ν ≤ ∥ a − x ∥ ν < 1/ L , which is a contradiction.
Define p ~ ( x ) = max { p ∗ ( x ) , ϕ A ( x ) } \widetilde{p}(x)=\max\{p^*(x),\phi_A(x)\} p ( x ) = max { p ∗ ( x ) , ϕ A ( x )} . Since both p ∗ p^* p ∗ and ϕ A \phi_A ϕ A are L L L -Lipschitz and the pointwise maximum preserves this Lipschitz bound, we have p ~ ∈ P L , ν \widetilde{p}\in\mathcal{P}_{L,\nu} p ∈ P L , ν . Since ϕ A = 1 \phi_A=1 ϕ A = 1 on A A A , we have p ~ ( x ) − p ∗ ( x ) ≥ η \widetilde{p}(x)-p^*(x)\ge\eta p ( x ) − p ∗ ( x ) ≥ η for all x ∈ A x\in A x ∈ A . Moreover, p ~ − p ∗ \widetilde{p}-p^* p − p ∗ can be positive only where ϕ A > 0 \phi_A>0 ϕ A > 0 , so the claim above gives
Δ ∗ ( x ) { p ~ ( x ) − p ∗ ( x ) } ≥ 0 , x ∈ X . \Delta^*(x)\{\widetilde{p}(x)-p^*(x)\}\ge0,\qquad x\in\mathcal{X}. Δ ∗ ( x ) { p ( x ) − p ∗ ( x )} ≥ 0 , x ∈ X .
Since x 0 ∈ S = supp ( P X ) x_0\in\mathcal{S}=\operatorname{supp}(P_X) x 0 ∈ S = supp ( P X ) and A A A contains a relative open neighbourhood of x 0 x_0 x 0 in X \mathcal{X} X , we have P X ( A ) > 0 P_X(A)>0 P X ( A ) > 0 . Therefore,
R ( p ~ ) − R ( p ∗ ) = E [ Δ ∗ ( X ) { p ~ ( X ) − p ∗ ( X ) } ] ≥ E [ Δ ∗ ( X ) { p ~ ( X ) − p ∗ ( X ) } 1 { X ∈ A } ] ≥ η E [ Δ ∗ ( X ) 1 { X ∈ A } ] > 0 , \begin{aligned}
R(\widetilde{p})-R(p^*)=\mathbb{E}[\Delta^*(X)\{\widetilde{p}(X)-p^*(X)\}] \\
&\ge\mathbb{E}\left[\Delta^*(X)\{\widetilde{p}(X)-p^*(X)\}\mathbf{1}\{X\in A\}\right] \\
&\ge\eta\mathbb{E}\left[\Delta^*(X)\mathbf{1}\{X\in A\}\right]>0,
\end{aligned} R ( p ) − R ( p ∗ ) = E [ Δ ∗ ( X ) { p ( X ) − p ∗ ( X )}] ≥ E [ Δ ∗ ( X ) { p ( X ) − p ∗ ( X )} 1 { X ∈ A } ] ≥ η E [ Δ ∗ ( X ) 1 { X ∈ A } ] > 0 ,
which contradicts the optimality of p ∗ p^* p ∗ . Hence p ∗ ( x 0 ) = 1 p^*(x_0)=1 p ∗ ( x 0 ) = 1 .
Step 3. Saturation on the negative side. The case Δ ∗ ( x 0 ) < 0 \Delta^*(x_0)<0 Δ ∗ ( x 0 ) < 0 is analogous. Fix x 0 ∈ S x_0\in S x 0 ∈ S such that dist ν ( x 0 , H θ ∗ ) > 1 / L \operatorname{dist}_{\nu}(x_0,\mathcal{H}_{\theta^*})>1/L dist ν ( x 0 , H θ ∗ ) > 1/ L and Δ ∗ ( x 0 ) < 0 \Delta^*(x_0)<0 Δ ∗ ( x 0 ) < 0 . Suppose for contradiction that p ∗ ( x 0 ) > 0 p^*(x_0)>0 p ∗ ( x 0 ) > 0 . Choose r > 0 r>0 r > 0 and η > 0 \eta>0 η > 0 such that A = B ν ( x 0 , r ) ∩ X A=B_{\nu}(x_0,r)\cap\mathcal{X} A = B ν ( x 0 , r ) ∩ X satisfies Δ ∗ ( x ) < 0 \Delta^*(x)<0 Δ ∗ ( x ) < 0 and p ∗ ( x ) ≥ η p^*(x)\ge\eta p ∗ ( x ) ≥ η for all x ∈ A x\in A x ∈ A , and dist ν ( A , H θ ∗ ) > 1 / L \operatorname{dist}_{\nu}(A,\mathcal{H}_{\theta^*})>1/L dist ν ( A , H θ ∗ ) > 1/ L . Define ϕ A ( x ) = max { 0 , 1 − L dist ν ( x , A ) } \phi_A(x)=\max\{0,1-L\operatorname{dist}_{\nu}(x,A)\} ϕ A ( x ) = max { 0 , 1 − L dist ν ( x , A )} . By the same line-segment argument as above, Δ ∗ ( x ) < 0 \Delta^*(x)<0 Δ ∗ ( x ) < 0 whenever ϕ A ( x ) > 0 \phi_A(x)>0 ϕ A ( x ) > 0 . Set
p ~ ( x ) = min { p ∗ ( x ) , 1 − ϕ A ( x ) } . \widetilde{p}(x)=\min\{p^*(x),1-\phi_A(x)\}. p ( x ) = min { p ∗ ( x ) , 1 − ϕ A ( x )} .
The pointwise minimum of two L L L -Lipschitz functions is L L L -Lipschitz, so p ~ ∈ P L , ν \widetilde{p}\in\mathcal{P}_{L,\nu} p ∈ P L , ν . Since ϕ A = 1 \phi_A=1 ϕ A = 1 on A A A , we have p ~ ( x ) = 0 \widetilde{p}(x)=0 p ( x ) = 0 and p ∗ ( x ) − p ~ ( x ) ≥ η p^*(x)-\widetilde{p}(x)\ge\eta p ∗ ( x ) − p ( x ) ≥ η for all x ∈ A x\in A x ∈ A . Moreover, p ~ − p ∗ \widetilde{p}-p^* p − p ∗ can be negative only where ϕ A > 0 \phi_A>0 ϕ A > 0 , so
Δ ∗ ( x ) { p ~ ( x ) − p ∗ ( x ) } ≥ 0 , x ∈ X . \Delta^*(x)\{\widetilde{p}(x)-p^*(x)\}\ge0,\qquad x\in\mathcal{X}. Δ ∗ ( x ) { p ( x ) − p ∗ ( x )} ≥ 0 , x ∈ X .
Again, since x 0 ∈ S x_0\in S x 0 ∈ S and A A A contains a relative open neighbourhood of x 0 x_0 x 0 , we have P X ( A ) > 0 P_X(A)>0 P X ( A ) > 0 . Therefore,
R ( p ~ ) − R ( p ∗ ) = E [ Δ ∗ ( X ) { p ~ ( X ) − p ∗ ( X ) } ] ≥ E [ Δ ∗ ( X ) { p ~ ( X ) − p ∗ ( X ) } 1 { X ∈ A } ] ≥ η E [ − Δ ∗ ( X ) 1 { X ∈ A } ] > 0 , \begin{aligned}
R(\widetilde{p})-R(p^*)=\mathbb{E}\left[\Delta^*(X)\{\widetilde{p}(X)-p^*(X)\}\right] \\
&\ge\mathbb{E}\left[\Delta^*(X)\{\widetilde{p}(X)-p^*(X)\}\mathbf{1}\{X\in A\}\right] \\
&\ge\eta\mathbb{E}\left[-\Delta^*(X)\mathbf{1}\{X\in A\}\right]>0,
\end{aligned} R ( p ) − R ( p ∗ ) = E [ Δ ∗ ( X ) { p ( X ) − p ∗ ( X )} ] ≥ E [ Δ ∗ ( X ) { p ( X ) − p ∗ ( X )} 1 { X ∈ A } ] ≥ η E [ − Δ ∗ ( X ) 1 { X ∈ A } ] > 0 ,
which contradicts the optimality of p ∗ p^* p ∗ . Hence $p^*(x_0)=0.
Thus, whenever x ∈ S x\in S x ∈ S satisfies dist ν ( x , H θ ∗ ) > 1 / L \operatorname{dist}_{\nu}(x,\mathcal{H}_{\theta^*})>1/L dist ν ( x , H θ ∗ ) > 1/ L , the optimal policy is deterministic and selects the arm with the larger conditional mean reward. This proves (86) .
Computation of the boundary-canonical policy
Throughout this section, we take ν = 1 \nu=1 ν = 1 in the fairness constraint. At update time s s s in Algorithm 2, let z ^ s \widehat{z}_s z s be a solution of the linear program in (9) . We describe how to compute the resulting policy p ^ s ( x ) \widehat{p}_s(x) p s ( x ) at any x ∈ [ 0 , 1 ] d x\in[0,1]^d x ∈ [ 0 , 1 ] d . Write n = s − 1 n=s-1 n = s − 1 , θ ^ s = ( a ^ s , γ ^ s ) \widehat{\theta}_s=(\widehat{a}_s,\widehat{\gamma}_s) θ s = ( a s , γ s ) and H = H θ ^ s \mathcal{H}=\mathcal{H}_{\widehat{\theta}_s} H = H θ s . We assume L > 0 L>0 L > 0 and H ≠ ∅ \mathcal{H}\ne\varnothing H = ∅ .
For each boundary point h ∈ H h\in\mathcal{H} h ∈ H , the McShane extension gives the probability of selecting arm 1
q ^ s ( h ) = p ^ s M c E x t ( h ) = min { 1 , min i ∈ [ n ] [ z ^ s , i + L ∥ h − X i ∥ 1 ] } . (87) \widehat{q}_s(h)=\widehat{p}^{\mathrm{McExt}}_s(h)=\min\left\{1,\min_{i\in[n]}\left[\widehat{z}_{s,i}+L\lVert h-X_i\rVert_1\right]\right\}. \tag*{(87)} q s ( h ) = p s McExt ( h ) = min { 1 , i ∈ [ n ] min [ z s , i + L ∥ h − X i ∥ 1 ] } . (87)
These boundary probabilities determine the canonical policy p ^ s = p θ ^ s , q ^ s \widehat{p}_s=p_{\widehat{\theta}_s,\widehat{q}_s} p s = p θ s , q s through (6) . To compute p ^ s \widehat{p}_s p s , we evaluate the upper envelope U θ ^ s , q ^ s U_{\widehat{\theta}_s,\widehat{q}_s} U θ s , q s on X θ ^ s + \mathcal{X}^+_{\widehat{\theta}_s} X θ s + , and the lower envelope M θ ^ s , q ^ s M_{\widehat{\theta}_s,\widehat{q}_s} M θ s , q s on X θ ^ s − \mathcal{X}^-_{\widehat{\theta}_s} X θ s − , both defined in (5) .
Evaluation on X θ ^ s + \mathcal{X}^+_{\widehat{\theta}_s} X θ s + . For x ∈ X θ ^ s + x\in\mathcal{X}^+_{\widehat{\theta}_s} x ∈ X θ s + , we compute
p ^ s ( x ) = min { 1 , U θ ^ s , q ^ s ( x ) } , with U θ ^ s , q ^ s ( x ) = min h ∈ H { q ^ s ( h ) + L ∥ h − x ∥ 1 } . (88) \widehat{p}_s(x)=\min\{1,U_{\widehat{\theta}_s,\widehat{q}_s}(x)\},\qquad\text{with}\qquad U_{\widehat{\theta}_s,\widehat{q}_s}(x)=\min_{h\in\mathcal{H}}\{\widehat{q}_s(h)+L\lVert h-x\rVert_1\}. \tag*{(88)} p s ( x ) = min { 1 , U θ s , q s ( x )} , with U θ s , q s ( x ) = h ∈ H min { q s ( h ) + L ∥ h − x ∥ 1 } . (88)
Substituting (87) into (88) gives that
p ^ s ( x ) = min { 1 , min i ∈ [ n ] { z ^ s , i + L d H ( X i , x ) } } . (89) \widehat{p}_s(x)=\min\left\{1,\min_{i\in[n]}\left\{\widehat{z}_{s,i}+Ld_{\mathcal{H}}(X_i,x)\right\}\right\}. \tag*{(89)} p s ( x ) = min { 1 , i ∈ [ n ] min { z s , i + L d H ( X i , x ) } } . (89)
Define the minimum total distance from y y y to x x x via a point on the boundary by
d H ( y , x ) = min h ∈ H { ∥ y − h ∥ 1 + ∥ h − x ∥ 1 } . d_{\mathcal{H}}(y,x)=\min_{h\in\mathcal{H}}\{\lVert y-h\rVert_1+\lVert h-x\rVert_1\}. d H ( y , x ) = h ∈ H min {∥ y − h ∥ 1 + ∥ h − x ∥ 1 } .
To compute d H ( y , x ) d_{\mathcal{H}}(y,x) d H ( y , x ) , for j ∈ [ d ] j\in[d] j ∈ [ d ] , let ℓ j = min { x j , y j } \ell_j=\min\{x_j,y_j\} ℓ j = min { x j , y j } , u j = max { x j , y j } u_j=\max\{x_j,y_j\} u j = max { x j , y j } and B = ∏ j = 1 d [ ℓ j , u j ] B=\prod_{j=1}^{d}[\ell_j,u_j] B = ∏ j = 1 d [ ℓ j , u j ] . Then
d H ( y , x ) = ∥ y − x ∥ 1 + 2 δ B , with δ B = min h ∈ H , v ∈ B ∥ h − v ∥ 1 . (90) d_{\mathcal{H}}(y,x)=\lVert y-x\rVert_1+2\delta_B,\qquad\text{with }\delta_B=\min_{h\in\mathcal{H},v\in B}\lVert h-v\rVert_1. \tag*{(90)} d H ( y , x ) = ∥ y − x ∥ 1 + 2 δ B , with δ B = h ∈ H , v ∈ B min ∥ h − v ∥ 1 . (90)
It therefore remains to compute δ B \delta_B δ B , the distance between B B B and H \mathcal{H} H .
If B ∩ H ≠ ∅ B\cap\mathcal{H}\ne\emptyset B ∩ H = ∅ , then δ B = 0 \delta_B=0 δ B = 0 . Otherwise, choose a vertex of B B B whose estimated reward gap is closest to zero. Let r r r denote the magnitude of this gap, initially given by
r = { a ^ s + ∑ j = 1 d min { γ ^ s , j ℓ j , γ ^ s , j u j } , B ⊂ X θ ^ s + , − a ^ s − ∑ j = 1 d max { γ ^ s , j ℓ j , γ ^ s , j u j } , B ⊂ X θ ^ s − . r=
\begin{cases}
\widehat{a}_s+\sum_{j=1}^{d}\min\{\widehat{\gamma}_{s,j}\ell_j,\widehat{\gamma}_{s,j}u_j\}, & B\subset\mathcal{X}^{+}_{\widehat{\theta}_s},\\
-\widehat{a}_s-\sum_{j=1}^{d}\max\{\widehat{\gamma}_{s,j}\ell_j,\widehat{\gamma}_{s,j}u_j\}, & B\subset\mathcal{X}^{-}_{\widehat{\theta}_s}.
\end{cases} r = { a s + ∑ j = 1 d min { γ s , j ℓ j , γ s , j u j } , − a s − ∑ j = 1 d max { γ s , j ℓ j , γ s , j u j } , B ⊂ X θ s + , B ⊂ X θ s − .
For j ∈ [ d ] j\in[d] j ∈ [ d ] , let c j c_j c j denote the maximum available movement of B B B in coordinate j j j toward the boundary while remaining in [ 0 , 1 ] d [0,1]^d [ 0 , 1 ] d . Then we have the following
Table 2.
with c j = 0 c_j=0 c j = 0 when γ ^ s , j = 0 \widehat{\gamma}_{s,j}=0 γ s , j = 0 . With r r r fixed at its initial value, the required distance is
δ B = min { ∑ j = 1 d t j : ∑ j = 1 d ∣ γ ^ s , j ∣ t j = r , 0 ≤ t j ≤ c j } . (91) \delta_B=\min\left\{\sum_{j=1}^{d}t_j:\sum_{j=1}^{d}|\widehat{\gamma}_{s,j}|t_j=r,\ 0\le t_j\le c_j\right\}. \tag*{(91)} δ B = min { j = 1 ∑ d t j : j = 1 ∑ d ∣ γ s , j ∣ t j = r , 0 ≤ t j ≤ c j } . (91)
To solve (91) , initialise t j = 0 t_j=0 t j = 0 for all j j j and process coordinates with nonzero coefficients in nonincreasing order of ∣ γ ^ s , j ∣ |\widehat{\gamma}_{s,j}| ∣ γ s , j ∣ . For each coordinate, let
t j = min { c j , r ∣ γ ^ s , j ∣ } t_j=\min\left\{c_j,\frac{r}{|\widehat{\gamma}_{s,j}|}\right\} t j = min { c j , ∣ γ s , j ∣ r }
and update r r r with r − ∣ γ ^ s , j ∣ t j r-|\widehat{\gamma}_{s,j}|t_j r − ∣ γ s , j ∣ t j . Each coordinate is processed at most once, and we stop when r = 0 r=0 r = 0 , requiring at most d d d updates. Since H \mathcal{H} H is nonempty, the available movements suffice to reach the boundary. Substituting δ B = ∑ j t j \delta_B=\sum_j t_j δ B = ∑ j t j into (90) and then (89) gives p ^ s ( x ) \widehat{p}_s(x) p s ( x ) .
Evaluation on X θ ^ s − \mathcal{X}^{-}_{\widehat{\theta}_s} X θ s − . For x ∈ X θ ^ s − x\in\mathcal{X}^{-}_{\widehat{\theta}_s} x ∈ X θ s − , we compute
p ^ s ( x ) = max { 0 , M θ ^ s , q ^ s ( x ) } , with M θ ^ s , q ^ s ( x ) = max h ∈ H { q ^ s ( h ) − L ∥ h − x ∥ 1 } . (92) \widehat{p}_s(x)=\max\{0,M_{\widehat{\theta}_s,\widehat{q}_s}(x)\},\qquad\text{with }M_{\widehat{\theta}_s,\widehat{q}_s}(x)=\max_{h\in\mathcal{H}}\{\widehat{q}_s(h)-L\lVert h-x\rVert_1\}. \tag*{(92)} p s ( x ) = max { 0 , M θ s , q s ( x )} , with M θ s , q s ( x ) = h ∈ H max { q s ( h ) − L ∥ h − x ∥ 1 } . (92)
To evaluate M θ ^ s , q ^ s ( x ) M_{\hat{\theta}_{s},\hat{q}_{s}}(x) M θ ^ s , q ^ s ( x ) , we solve the following equivalent problem:
max h , r , w , ϵ r − L ∑ j = 1 d w j subject to h ∈ H , 0 ≤ r ≤ 1 , ϵ i j ∈ { − 1 , 1 } , ϵ i j ( h j − X i j ) ≥ 0 , i ∈ [ n ] , j ∈ [ d ] , r ≤ z ^ s , i + L ∑ j = 1 d ϵ i j ( h j − X i j ) , i ∈ [ n ] , w j ≥ h j − x j , w j ≥ x j − h j , j ∈ [ d ] . (93) \begin{aligned}
\max_{h,r,w,\epsilon}\quad&r-L\sum_{j=1}^{d}w_j\\
\text{subject to}\quad&h\in\mathcal{H},\quad0\le r\le1,\\
&\epsilon_{ij}\in\{-1,1\},\quad\epsilon_{ij}(h_j-X_{ij})\ge0,\quad i\in[n],\ j\in[d],\\
&r\le\hat{z}_{s,i}+L\sum_{j=1}^{d}\epsilon_{ij}(h_j-X_{ij}),\quad i\in[n],\\
&w_j\ge h_j-x_j,\quad w_j\ge x_j-h_j,\quad j\in[d]. \tag*{(93)}
\end{aligned} h , r , w , ϵ max subject to r − L j = 1 ∑ d w j h ∈ H , 0 ≤ r ≤ 1 , ϵ ij ∈ { − 1 , 1 } , ϵ ij ( h j − X ij ) ≥ 0 , i ∈ [ n ] , j ∈ [ d ] , r ≤ z ^ s , i + L j = 1 ∑ d ϵ ij ( h j − X ij ) , i ∈ [ n ] , w j ≥ h j − x j , w j ≥ x j − h j , j ∈ [ d ] . (93)
The sign constraints ensure that ϵ i j ( h j − X i j ) = ∣ h j − X i j ∣ \epsilon_{ij}(h_j-X_{ij})=|h_j-X_{ij}| ϵ ij ( h j − X ij ) = ∣ h j − X ij ∣ . However, the formulation is nonlinear because of the products ϵ i j h j \epsilon_{ij}h_j ϵ ij h j . Introducing binary variables b i j ∈ { 0 , 1 } b_{ij}\in\{0,1\} b ij ∈ { 0 , 1 } and auxiliary variables v i j v_{ij} v ij representing ∣ h j − X i j ∣ |h_j-X_{ij}| ∣ h j − X ij ∣ gives the following equivalent mixed-integer linear programme:
max h , r , w , v , b r − L ∑ j = 1 d w j subject to h ∈ H , 0 ≤ r ≤ 1 , b i j ∈ { 0 , 1 } , i ∈ [ n ] , j ∈ [ d ] , v i j ≥ h j − X i j , v i j ≥ X i j − h j , i ∈ [ n ] , j ∈ [ d ] , v i j ≤ h j − X i j + 2 X i j ( 1 − b i j ) , i ∈ [ n ] , j ∈ [ d ] , v i j ≤ X i j − h j + 2 ( 1 − X i j ) b i j , i ∈ [ n ] , j ∈ [ d ] , r ≤ z ^ s , i + L ∑ j = 1 d v i j , i ∈ [ n ] , w j ≥ h j − x j , w j ≥ x j − h j , j ∈ [ d ] . (94) \begin{aligned}
\max_{h,r,w,v,b}\quad&r-L\sum_{j=1}^{d}w_j\\
\text{subject to}\quad&h\in\mathcal{H},\quad0\le r\le1,\\
&b_{ij}\in\{0,1\},\quad i\in[n],\ j\in[d],\\
&v_{ij}\ge h_j-X_{ij},\quad v_{ij}\ge X_{ij}-h_j,\quad i\in[n],\ j\in[d],\\
&v_{ij}\le h_j-X_{ij}+2X_{ij}(1-b_{ij}),\quad i\in[n],\ j\in[d],\\
&v_{ij}\le X_{ij}-h_j+2(1-X_{ij})b_{ij},\quad i\in[n],\ j\in[d],\\
&r\le\hat{z}_{s,i}+L\sum_{j=1}^{d}v_{ij},\quad i\in[n],\\
&w_j\ge h_j-x_j,\quad w_j\ge x_j-h_j,\quad j\in[d]. \tag*{(94)}
\end{aligned} h , r , w , v , b max subject to r − L j = 1 ∑ d w j h ∈ H , 0 ≤ r ≤ 1 , b ij ∈ { 0 , 1 } , i ∈ [ n ] , j ∈ [ d ] , v ij ≥ h j − X ij , v ij ≥ X ij − h j , i ∈ [ n ] , j ∈ [ d ] , v ij ≤ h j − X ij + 2 X ij ( 1 − b ij ) , i ∈ [ n ] , j ∈ [ d ] , v ij ≤ X ij − h j + 2 ( 1 − X ij ) b ij , i ∈ [ n ] , j ∈ [ d ] , r ≤ z ^ s , i + L j = 1 ∑ d v ij , i ∈ [ n ] , w j ≥ h j − x j , w j ≥ x j − h j , j ∈ [ d ] . (94)
The constraints enforce v i j = ∣ h j − X i j ∣ v_{ij}=|h_j-X_{ij}| v ij = ∣ h j − X ij ∣ , and maximisation sets r = q ^ s ( h ) r=\hat{q}_s(h) r = q ^ s ( h ) and w j = ∣ h j − x j ∣ w_j=|h_j-x_j| w j = ∣ h j − x j ∣ . The optimal value therefore equals M θ ^ s , q ^ s ( x ) M_{\hat{\theta}_{s},\hat{q}_{s}}(x) M θ ^ s , q ^ s ( x ) , which is clipped at zero to obtain p ^ s ( x ) \hat{p}_s(x) p ^ s ( x ) .
Additional numerical results
In this section, we present additional simulation details and results in Section G.1 and describe the preprocessing procedures for the real-data analysis in Section G.2.
Additional results for the simulation studies
In this subsection, we first give the oracle policies for the Arm 1 small and Arm 2 small settings and then present additional simulation results to complement Section 6.1.
Explicit oracle policies for the Arm 1 small and Arm 2 small settings. Both settings have uniform covariates on [ 0 , 1 ] 2 [0,1]^2 [ 0 , 1 ] 2 . The following policies are optimal under the ℓ 1 \ell_1 ℓ 1 -Lipschitz fairness constraint for every L ≥ 0 L\ge0 L ≥ 0 .
In the Arm 1 small setting, Δ ∗ ( x ) = x 2 − 0.82 + 0.27 x 1 \Delta^*(x)=x_2-0.82+0.27x_1 Δ ∗ ( x ) = x 2 − 0.82 + 0.27 x 1 . An optimal policy is
p L , 1 ∗ ( x ) = [ min { L ( x 2 − 0.64 + 0.54 x 1 ) , 1 2 + L ( x 2 − 0.82 + 0.27 x 1 ) } ] [ 0 , 1 ] . p^*_{L,1}(x)=\left[\min\left\{L(x_2-0.64+0.54x_1),\frac{1}{2}+L(x_2-0.82+0.27x_1)\right\}\right]_{[0,1]}. p L , 1 ∗ ( x ) = [ min { L ( x 2 − 0.64 + 0.54 x 1 ) , 2 1 + L ( x 2 − 0.82 + 0.27 x 1 ) } ] [ 0 , 1 ] .
In the Arm 2 small setting, Δ ∗ ( x ) = x 1 + x 2 − 0.2 \Delta^*(x)=x_1+x_2-0.2 Δ ∗ ( x ) = x 1 + x 2 − 0.2 . An optimal policy is
p L , 1 ∗ ( x ) = { [ 1 + L ( x 1 + x 2 − 0.3 ) ] [ 0 , 1 ] , 0 ≤ L ≤ 10 / 3 , [ 1 2 + L { x 1 + x 2 − 0.2 + 0.04 − 1 / ( 3 L 2 ) 2 } ] [ 0 , 1 ] , L > 10 / 3. p^*_{L,1}(x)=
\begin{cases}
\left[1+L(x_1+x_2-0.3)\right]_{[0,1]}, & 0\le L\le10/3,\\
\left[\frac{1}{2}+L\left\{x_1+x_2-\frac{0.2+\sqrt{0.04-1/(3L^2)}}{2}\right\}\right]_{[0,1]}, & L>10/3.
\end{cases} p L , 1 ∗ ( x ) = ⎩ ⎨ ⎧ [ 1 + L ( x 1 + x 2 − 0.3 ) ] [ 0 , 1 ] , [ 2 1 + L { x 1 + x 2 − 2 0.2 + 0.04 − 1/ ( 3 L 2 ) } ] [ 0 , 1 ] , 0 ≤ L ≤ 10/3 , L > 10/3.
Varying the fairness parameter. We vary L ∈ { 0.25 , 0.5 , 1 , 2 , 4 } L\in\{0.25,0.5,1,2,4\} L ∈ { 0.25 , 0.5 , 1 , 2 , 4 } under uniform covariates and homogeneous rewards with d = 2 d=2 d = 2 , using the six horizons T ∈ { 1024 , 2048 , 4096 , 8192 , 16384 , 32768 } T\in\{1024,2048,4096,8192,16384,32768\} T ∈ { 1024 , 2048 , 4096 , 8192 , 16384 , 32768 } . Figure 9 reports relative and cumulative regret, averaged over 50 independent Monte Carlo trials. Relative regret decreases with T T T for every value of L L L , consistent with the sublinear cumulative regret bound for fixed L L L in Theorem 4. The relative regret curves remain close across the values of L L L considered. Mean cumulative regret increases with L L L , while the curves are approximately linear on log–log axes, with similar slopes of about 0.53.
Figure 9. Relative regret (left) and cumulative regret (right) against T T T for L ∈ { 0.25 , 0.5 , 1 , 2 , 4 } L\in\{0.25,0.5,1,2,4\} L ∈ { 0.25 , 0.5 , 1 , 2 , 4 } , under uniform covariates and homogeneous rewards with d = 2 d=2 d = 2 . Lines show means over 50 independent Monte Carlo trials; shaded bands show the mean ± \pm ± one standard deviation.
Cumulative regret in the baseline settings. Figure 10 plots cumulative regret against T T T on log–log axes for the four baseline settings and d = 1 , 2 , 3 , 4 d=1,2,3,4 d = 1 , 2 , 3 , 4 , with L = 0.5 L=0.5 L = 0.5 . The curves are approximately linear, with slopes generally close to 1 / 2 1/2 1/2 over the observed horizons.
Figure 10. Cumulative regret against T T T on log–log axes for d = 1 , 2 , 3 , 4 d = 1,2,3,4 d = 1 , 2 , 3 , 4 at L = 0.5 L = 0.5 L = 0.5 . Columns show homogeneous (left) and heterogeneous (right) rewards; rows show uniform (top) and truncated-Gaussian (bottom) covariates. Lines show means over 50 independent Monte Carlo trials; shaded bands show the mean ± \pm ± one standard deviation.
An additional simulation setting
We next consider an additional simulation setting with d = 4 d=4 d = 4 , modifying both the covariate distribution and the reward functions. We first define the covariate distribution. Let
X = ( Z , U ) , Z = ( G − 1 ) / 5 , G ∈ { 1 , … , 6 } , and U ∼ Unif ( [ 0 , 1 ] 3 ) . X=(Z,U),\quad Z=(G-1)/5,\quad G\in\{1,\ldots,6\},\quad\text{and}\quad U\sim\operatorname{Unif}([0,1]^3). X = ( Z , U ) , Z = ( G − 1 ) /5 , G ∈ { 1 , … , 6 } , and U ∼ Unif ([ 0 , 1 ] 3 ) .
Here, Z Z Z encodes a six-category covariate whose conditional distribution depends on U U U .
To specify how the category probabilities vary with U U U , partition [ 0 , 1 ] 3 [0,1]^3 [ 0 , 1 ] 3 into m 3 m^3 m 3 equal cubic cells with m = 4 m = 4 m = 4 . For a point u u u , define its cell indices and its rescaled coordinates within that cell by
k j ( u ) = min { ⌊ m u j ⌋ , m − 1 } , r j ( u ) = m u j − k j ( u ) , ∀ j ∈ { 1 , 2 , 3 } . k_j(u) = \min\{\lfloor m u_j \rfloor, m - 1\}, \qquad r_j(u) = m u_j - k_j(u), \qquad\forall j \in\{1,2,3\}. k j ( u ) = min {⌊ m u j ⌋ , m − 1 } , r j ( u ) = m u j − k j ( u ) , ∀ j ∈ { 1 , 2 , 3 } .
Write k ( u ) = ( k 1 ( u ) , k 2 ( u ) , k 3 ( u ) ) k(u) = (k_1(u), k_2(u), k_3(u)) k ( u ) = ( k 1 ( u ) , k 2 ( u ) , k 3 ( u )) for the cell containing u u u . Within each cell, we construct two equal-volume subregions that favour opposite groups of categories. Independently draw a sign η k ∈ { − 1 , 1 } \eta_k \in\{-1,1\} η k ∈ { − 1 , 1 } with equal probabilities for each cell k k k , and keep these signs fixed across horizons and replications. Set
H ( u ) = η k ( u ) [ 1 { r 1 ( u ) ∈ [ 1 / 4 , 1 / 2 ) } − 1 { r 1 ( u ) ∈ [ 1 / 2 , 3 / 4 ) } ] ∏ j = 2 3 1 { r j ( u ) ∈ [ 1 / 4 , 3 / 4 ) } . (95) H(u) = \eta_{k(u)}\left[\mathbb{1}\{r_1(u) \in[1/4,1/2)\} - \mathbb{1}\{r_1(u) \in[1/2,3/4)\}\right]\prod_{j=2}^{3}\mathbb{1}\{r_j(u) \in[1/4,3/4)\}. \tag*{(95)} H ( u ) = η k ( u ) [ 1 { r 1 ( u ) ∈ [ 1/4 , 1/2 )} − 1 { r 1 ( u ) ∈ [ 1/2 , 3/4 )} ] j = 2 ∏ 3 1 { r j ( u ) ∈ [ 1/4 , 3/4 )} . (95)
Thus, H ( u ) H(u) H ( u ) takes opposite signs on the two subregions and is zero elsewhere. We use H ( u ) H(u) H ( u ) to define the conditional category probabilities:
w g ( u ) = P ( G = g ∣ U = u ) = 1 + s g H ( u ) / 2 6 , where s g = { − 1 , g ≤ 3 , 1 , g ≥ 4. (96) w_g(u) = \mathbb{P}(G = g \mid U = u) = \frac{1 + s_g H(u)/2}{6}, \qquad\text{where } s_g =
\begin{cases}
-1, & g \le3,\\
1, & g \ge4.
\end{cases} \tag*{(96)} w g ( u ) = P ( G = g ∣ U = u ) = 6 1 + s g H ( u ) /2 , where s g = { − 1 , 1 , g ≤ 3 , g ≥ 4. (96)
When H ( u ) = 1 H(u) = 1 H ( u ) = 1 , categories 4, 5, 6 are favoured; when H ( u ) = − 1 H(u) = -1 H ( u ) = − 1 , categories 1, 2, 3 are favoured. When H ( u ) = 0 H(u) = 0 H ( u ) = 0 , all six categories are equally likely. These conditional probabilities change discontinuously across subregions. Since the positive and negative contributions of H H H cancel within each cell, ∫ [ 0 , 1 ] 3 H ( u ) d u = 0 \int_{[0,1]^3} H(u)\,\mathrm{d}u = 0 ∫ [ 0 , 1 ] 3 H ( u ) d u = 0 . Each category therefore has marginal probability 1 / 6 1/6 1/6 , although G G G and U U U are dependent.
We next specify the mean rewards:
μ 1 ( g , u ) = g − 1 5 − 1 2 + 0.03 ∑ j = 1 3 ( u j − 1 / 2 ) and μ 2 ( g , u ) = 0. \mu_1(g,u)=\frac{g-1}{5}-\frac{1}{2}+0.03\sum_{j=1}^{3}(u_j-1/2)
\quad\text{and}\quad
\mu_2(g,u)=0. μ 1 ( g , u ) = 5 g − 1 − 2 1 + 0.03 j = 1 ∑ 3 ( u j − 1/2 ) and μ 2 ( g , u ) = 0.
We generate rewards by adding independent N ( 0 , 0.3 2 ) \mathcal{N}(0,0.3^2) N ( 0 , 0. 3 2 ) noise to these means. We use the lazy fair greedy algorithm in Algorithm 1 with L = 0.5 L=0.5 L = 0.5 and ν = 1 \nu=1 ν = 1 , following the same exploration schedule as in the baseline experiments.
Numerical oracle. We approximate the fair oracle on a regular grid V = { 0 , h , … , 1 } 3 \mathcal{V}=\{0,h,\ldots,1\}^3 V = { 0 , h , … , 1 } 3 , with h = 1 / 64 h=1/64 h = 1/64 , for each of the six categories. Let p g , v ∈ [ 0 , 1 ] p_{g,v}\in[0,1] p g , v ∈ [ 0 , 1 ] denote the probability of selecting arm 1 for category g g g at grid point v ∈ V v\in\mathcal{V} v ∈ V . We solve a linear programme in these 6 × 65 3 = 1647750 6\times65^3=1647750 6 × 6 5 3 = 1647750 variables to maximise population reward subject to the ℓ 1 \ell_1 ℓ 1 -Lipschitz fairness constraint at the grid points. We evaluate population reward for this linear programme using two-point Gaussian quadrature in each coordinate, splitting the integration domain at grid boundaries and discontinuities of H H H .
We then extend the resulting probabilities to all u ∈ [ 0 , 1 ] 3 u\in[0,1]^3 u ∈ [ 0 , 1 ] 3 by trilinear interpolation within each category. For u u u in a grid cell v + [ 0 , h ] 3 v+[0,h]^3 v + [ 0 , h ] 3 , define
α j = u j − v j h , ∀ j = 1 , 2 , 3. \alpha_j=\frac{u_j-v_j}{h},\qquad\forall j=1,2,3. α j = h u j − v j , ∀ j = 1 , 2 , 3.
The interpolated policy is
p n u m ( g − 1 5 , u ) = ∑ b ∈ { 0 , 1 } 3 { ∏ j = 1 3 ω b j ( α j ) } p g , v + h b , where ω 0 ( a ) = 1 − a and ω 1 ( a ) = a . p^{\mathrm{num}}\left(\frac{g-1}{5},u\right)
=
\sum_{b\in\{0,1\}^3}
\left\{\prod_{j=1}^{3}\omega_{b_j}(\alpha_j)\right\}p_{g,v+hb},
\quad\text{where }\omega_0(a)=1-a\text{ and }\omega_1(a)=a. p num ( 5 g − 1 , u ) = b ∈ { 0 , 1 } 3 ∑ { j = 1 ∏ 3 ω b j ( α j ) } p g , v + hb , where ω 0 ( a ) = 1 − a and ω 1 ( a ) = a .
Thus, the probability at u u u is a weighted average of the probabilities at the cell’s eight vertices. The weights are nonnegative and sum to one, so the interpolated policy takes values in [ 0 , 1 ] [0,1] [ 0 , 1 ] . It agrees with the LP solution at grid points and preserves the ℓ 1 \ell_1 ℓ 1 -Lipschitz fairness constraint.
Reward evaluation and results. We evaluate population reward by integrating over U U U using 32768 Halton points in [ 0 , 1 ] 3 [0,1]^3 [ 0 , 1 ] 3 after skipping the first 128, and summing over the six categories with weights w g ( u ) w_g(u) w g ( u ) . We denote the resulting reward estimate for a policy p p p by R ^ ( p ) \widehat{R}(p) R ( p ) . Within each replication, the numerical oracle and learned policies use the same integration points and weights. We compute estimated cumulative and relative regret as
R ^ T = ∑ t = 1 T { R ^ ( p n u m ) − R ^ ( p ^ t ) } and R e l R e g ^ T = R ^ T T R ^ ( p n u m ) , \widehat{\mathcal{R}}_T=\sum_{t=1}^{T}\left\{\widehat{R}(p^{\mathrm{num}})-\widehat{R}(\widehat{p}_t)\right\}
\quad\text{and}\quad
\widehat{\mathrm{RelReg}}_T=
\frac{\widehat{\mathcal{R}}_T}{T\widehat{R}(p^{\mathrm{num}})}, R T = t = 1 ∑ T { R ( p num ) − R ( p t ) } and RelReg T = T R ( p num ) R T ,
respectively, including initial exploration.
Figure 11 reports mean estimated relative and cumulative regret at L = 0.5 L=0.5 L = 0.5 , based on 50 independent Monte Carlo trials, at T = 1024 , 2048 , 4096 , 8192 , 16384 , 32768 T=1024, 2048, 4096, 8192, 16384, 32768 T = 1024 , 2048 , 4096 , 8192 , 16384 , 32768 , respectively. Relative regret decreases with T T T . Mean cumulative regret plotted against T T T is approximately linear on log–log axes, with an empirical slope of 0.645, numerically close to the exponent 2 / 3 2/3 2/3 in Theorem 4.
Figure 11. Estimated relative regret (left) and cumulative regret (right) against T T T for the additional simulation setting with d = 4 d=4 d = 4 and L = 0.5 L=0.5 L = 0.5 . Lines show means over 50 independent Monte Carlo trials; shaded bands show the mean ± \pm ± one standard deviation.
Additional details for the real data analysis
In this subsection, we describe the sample selection and covariate processing used in § 6.2 .
Datasets and sample selection. For Adult ([6 ] ), we remove records with missing values and retain records from Canada, El Salvador, Germany, India, Mexico, the Philippines, Puerto Rico and the United States. We set the response to B = 1 B=1 B = 1 when annual income exceeds $\$50000 and B = 0 B=0 B = 0 otherwise. The 12 covariates are age, work class, final weight, numeric education level, marital status, occupation, relationship, race, sex, capital gain, capital loss and weekly working hours. The resulting sample contains 43303 records.
For Jobs (Tankha [47 ] ), we remove the index column and the detailed list of technologies used, then discard records with missing values. We use the binary employed field as the response. The 11 covariates are age group, accessibility indicator, education level, employment status at the time of application, gender, a mental health indicator, professional developer status, years of coding experience, years of professional coding experience, previous salary and the number of technologies used. Country is excluded from the covariates, and all observed gender categories are retained. The resulting sample contains 73462 records.
Covariate processing. For each dataset with n n n eligible records, we randomly select ⌊ 0.1 n ⌋ \lfloor0.1n\rfloor ⌊ 0.1 n ⌋ records without replacement to form a preprocessing subset. We fit the covariate transformation on this subset without using response labels and apply it unchanged to all covariates used for online learning. The preprocessing subset is excluded from the online sampling pool, and both remain fixed across horizons, fairness parameters and replications.
For Adult, the categorical covariates are work class, marital status, occupation, relationship, race and sex; the remaining covariates are numeric. For Jobs, the categorical covariates are age group, accessibility, education level, gender, mental health and professional developer status; the remaining covariates are numeric. We one-hot encode categorical covariates using one indicator for each category observed in the preprocessing subset. An unseen category is represented by zeros in all indicator columns for the corresponding covariate. After encoding, Adult and Jobs have 46 and 21 covariates, respectively.
We standardise each encoded covariate using its mean and sample standard deviation in the preprocessing subset. We then fit principal component analysis (PCA) to the standardised covariates in this subset and retain the first three components for policy learning (Jolliffe and Cadima [30 ] ).
Let S i j S_{ij} S ij denote the score of record i i i on component j j j , and let a j a_j a j and b j b_j b j be the minimum and maximum scores in the preprocessing subset. We rescale and clip the scores as
X i j = [ S i j − a j b j − a j ] [ 0 , 1 ] , j = 1 , 2 , 3. X_{ij}=\left[\frac{S_{ij}-a_j}{b_j-a_j}\right]_{[0,1]}, \qquad j=1,2,3. X ij = [ b j − a j S ij − a j ] [ 0 , 1 ] , j = 1 , 2 , 3.
Applying this fixed transformation gives X i ∈ [ 0 , 1 ] 3 X_i\in[0,1]^3 X i ∈ [ 0 , 1 ] 3 for every record.