Introduction
Adversarial linear bandits ([4 , 39 , 16 , 1 , 12 ] ) are a basic model of sequential decision making with structured actions and nonstationary rewards. In each round, the learner picks an action a ∈ A ⊂ R d \mathbf{a} \in\mathcal{A} \subset\mathbb{R}^d a ∈ A ⊂ R d , an adversary picks a reward vector θ t ∈ R d \boldsymbol{\theta}_t \in\mathbb{R}^d θ t ∈ R d , and the learner observes only the reward a T θ t \mathbf{a}^{\mathsf{T}}\boldsymbol{\theta}_t a T θ t of the action it played, possibly with noise. The regret of this problem is well understood. For a finite action set of size K K K , exponential weights with a suitable exploration distribution achieves O ( d T log K ) O(\sqrt{dT\log K}) O ( d T log K ) regret ([12 ] ), which is minimax optimal up to logarithmic factors on the hypercube ([16 ] ), and a rate of the same order holds with high probability against an adaptive adversary ([46 ] ). These guarantees all assume, however, that the observed reward equals the linear reward of the chosen action plus zero-mean noise. In many applications, this assumption fails because the feedback itself is manipulated. Click fraud in online advertising, fake ratings in recommender systems, and malicious users who poison the feedback of an online learner all inject corruptions into the observed rewards, and even a small corruption budget suffices to mislead standard bandit algorithms [37 , 25 , 35 , 20 ] . Corruption-robust bandit algorithms address this issue by allowing an adversary to perturb the observed rewards subject to a total budget C C C , and they aim for regret that degrades gracefully with C C C . For stochastic linear bandits, this goal is largely achieved: He et al. (2022) obtain O ~ ( d T + d C ) \widetilde{O}(d\sqrt{T}+dC) O ( d T + d C ) regret, which matches the Ω ( d T + d C ) \Omega(d\sqrt{T}+dC) Ω ( d T + d C ) lower bound [29 , 11 ] up to logarithmic factors. When the rewards are themselves adversarial, the corruption cannot be absorbed into the reward vectors θ t \theta_t θ t , because it may depend on the realized action and is not linear in the action [36 ] . Corrupted adversarial linear bandits are therefore a genuinely different problem, and they were studied only recently.
Liu et al. [36 ] initiated the study of corrupted adversarial linear bandits and distinguished two types of adversaries. A weak adversary fixes the corruption of every action before the learner draws its action, whereas a strong adversary chooses the corruption after observing the drawn action. Against a weak adversary, Liu et al. [36 ] obtain O ~ ( d T + d C ∞ ) \widetilde{O}(d\sqrt{T}+\sqrt{d}C_\infty) O ( d T + d C ∞ ) regret, matching the stochastic case. Against a strong adversary, their bound is O ( d 3 T + d C ) O(\sqrt{d^3T}+dC) O ( d 3 T + d C ) , whose leading term is a factor d \sqrt{d} d larger than the corruption-free rate, and they leave open whether d T + d C d\sqrt{T}+dC d T + d C regret is achievable [36 Section 7] . The difficulty lies in the reward estimator. Adversarial bandit algorithms rely on importance-weighted estimates of the rewards, and the weight placed on the observed reward can be as large as 1 / p t , I t 1/p_{t,I_t} 1/ p t , I t when the drawn action I t I_t I t has small sampling probability p t , I t p_{t,I_t} p t , I t . A weak adversary cannot exploit this, because the large weight is drawn with probability p t , I t p_{t,I_t} p t , I t and cancels in expectation. A strong adversary, by contrast, can stay silent in most rounds and spend its budget only in rounds where a rarely sampled action is drawn, so that each unit of corruption shifts the reward estimates by 1 / p t , I t 1/p_{t,I_t} 1/ p t , I t instead of O ( 1 ) O(1) O ( 1 ) . Liu et al. [36 ] avoid this problem by running continuous exponential weights over a lifted action space, whose concentrated sampling keeps the weights bounded by poly ( d ) \operatorname{poly}(d) poly ( d ) , and their analysis of this approach incurs the additional d \sqrt{d} d factor in the leading term.
In this paper, we resolve this open question in the affirmative. We study adversarial linear bandits with K K K actions, where an adaptive adversary chooses the reward vectors and a strong adversary chooses the corruptions after observing both the drawn action and its clean reward, subject to a known budget C C C . Our contributions are summarized as follows.
We propose CRW-FTRL, an FTRL algorithm that combines a radius-weighted entropy regularizer with normalized kernel reward estimates. In each round, the algorithm assigns each action a nondecreasing radius via a monotone log-determinant barrier problem and constructs a reweighted, normalized kernel reward estimator. Controlling the estimator’s sensitivity through these radii rather than inverse sampling probabilities prevents a strong adversary from amplifying its corruption on rarely sampled actions, while the radius-weighted entropy regularizer with a linear bonus offsets both the cumulative corruption bias and the estimation variance.
We prove that, with probability at least 1 − δ 1-\delta 1 − δ , CRW-FTRL achieves realized regret O ( d C log ( d T ) + d T log ( d T ) log ( K T / δ ) ) = O ~ ( d T log K + d C ) O(dC\log(dT)+\sqrt{dT\log(dT)\log(KT/\delta)})=\widetilde{O}(\sqrt{dT\log K}+dC) O ( d C log ( d T ) + d T log ( d T ) log ( K T / δ ) ) = O ( d T log K + d C ) against an adaptive adversary with strong corruption (Theorem Theorem 5.1 and Corollary Corollary 5.2). The corruption-free term matches the best known high-probability rate O ~ ( d T log K ) \widetilde{O}(\sqrt{dT\log K}) O ( d T log K ) for finite action sets [12 Zimmert and]
Lattimore, 2022), and the corruption term matches the Ω ( d C ) \Omega(dC) Ω ( d C ) lower bound of Bogunovic et al. [11 ] (which holds already for O ( d ) O(d) O ( d ) actions; adjoining the zero action to their d d d standard-basis arms gives K = d + 1 K=d+1 K = d + 1 actions with full affine span in R d \mathbb{R}^{d} R d ), both up to logarithmic factors.
Because K K K enters our bound only through log K \log K log K , running CRW-FTRL on a fine discretization of a compact action set yields O ~ ( d T + d C ) \widetilde{O}(d\sqrt{T}+dC) O ( d T + d C ) regret for general action sets with bounded reward vectors (Remark 5.3 ). This answers the question of Liu et al. [36 ] on the statistical rate and improves the leading term of their bound from d 3 T \sqrt{d^{3}T} d 3 T to d T d\sqrt{T} d T . Together with their result for weak adversaries, it shows that corrupted adversarial linear bandits attain the same minimax rates as corrupted stochastic linear bandits under both corruption models. Since the discretization has exponentially many actions in d d d , this reduction is not computationally efficient, and achieving the same rate with a computationally efficient algorithm remains open.
Notation. For nonnegative sequences { x n } \{x_n\} { x n } and { y n } \{y_n\} { y n } , we write x n = O ( y n ) x_n=O(y_n) x n = O ( y n ) if lim sup n → ∞ x n / y n < ∞ \limsup_{n\to\infty}x_n/y_n<\infty lim sup n → ∞ x n / y n < ∞ , y n = Ω ( x n ) y_n=\Omega(x_n) y n = Ω ( x n ) if x n = O ( y n ) x_n=O(y_n) x n = O ( y n ) , and y n = Θ ( x n ) y_n=\Theta(x_n) y n = Θ ( x n ) if x n = O ( y n ) x_n=O(y_n) x n = O ( y n ) and x n = Ω ( y n ) x_n=\Omega(y_n) x n = Ω ( y n ) . We further employ O ~ ( ⋅ ) \widetilde{O}(\cdot) O ( ⋅ ) , Ω ~ ( ⋅ ) \widetilde{\Omega}(\cdot) Ω ( ⋅ ) , and Θ ~ ( ⋅ ) \widetilde{\Theta}(\cdot) Θ ( ⋅ ) to hide polylogarithmic factors. We denote [ N ] : = { 1 , ⋯ , N } [N]:=\{1,\cdots,N\} [ N ] := { 1 , ⋯ , N } for any positive integer N N N , and write Δ K : = { p ∈ R K : p i ≥ 0 , ∑ i = 1 K p i = 1 } \Delta_K:=\{p\in\mathbb{R}^{K}:p_i\ge0,\ \sum_{i=1}^{K}p_i=1\} Δ K := { p ∈ R K : p i ≥ 0 , ∑ i = 1 K p i = 1 } for the probability simplex over [ K ] [K] [ K ] . Boldfaced lowercase and uppercase letters are reserved for vectors and matrices, respectively. We denote by e i \mathbf{e}_i e i the i i i -th standard basis vector, by 0 \mathbf{0} 0 and 1 \mathbf{1} 1 the all-zeros and all-ones vectors, by ∥ ⋅ ∥ \|\cdot\| ∥ ⋅ ∥ the Euclidean norm, and by ⟨ x , y ⟩ : = x ⊤ y \langle\mathbf{x},\mathbf{y}\rangle:=\mathbf{x}^{\top}\mathbf{y} ⟨ x , y ⟩ := x ⊤ y the standard inner product for vectors x , y \mathbf{x},\mathbf{y} x , y of the same dimension. For symmetric matrices A \mathbf{A} A and B \mathbf{B} B , we write A ⪰ B \mathbf{A}\succeq\mathbf{B} A ⪰ B (resp. A ≻ B \mathbf{A}\succ\mathbf{B} A ≻ B ) if A − B \mathbf{A}-\mathbf{B} A − B is positive semidefinite (resp. positive definite), and use λ max ( ⋅ ) \lambda_{\max}(\cdot) λ m a x ( ⋅ ) , tr ( ⋅ ) \operatorname{tr}(\cdot) tr ( ⋅ ) , and det ( ⋅ ) \det(\cdot) det ( ⋅ ) to denote the largest eigenvalue, trace, and determinant.
Related Work
Adversarial Linear Bandits. Adversarial linear bandits were introduced by Awerbuch and Kleinberg [4 ] and McMahan and Blum [39 ] . Dani et al. [16 ] proposed GeometricHedge, which runs exponential weights over the action set and explores with a barycentric spanner, and proved an Ω ( d T ) \Omega(d\sqrt{T}) Ω ( d T ) lower bound on the hypercube with i.i.d. losses. Abernethy et al. [1 ] gave a computationally efficient algorithm with O ( d ϑ T log T ) O(d\sqrt{\vartheta T}\log T) O ( d ϑT log T ) regret by running FTRL with a ϑ \vartheta ϑ -self-concordant barrier (yielding O ~ ( d T ) \widetilde{O}(d\sqrt{T}) O ( d T ) when ϑ = O ( 1 ) \vartheta=O(1) ϑ = O ( 1 ) and O ~ ( d 3 / 2 T ) \widetilde{O}(d^{3/2}\sqrt{T}) O ( d 3/2 T ) for a general ϑ = O ( d ) \vartheta=O(d) ϑ = O ( d ) barrier). Cesa-Bianchi and Lugosi [13 ] and Bubeck et al. [12 ] refined the exploration distribution of exponential weights, and Bubeck et al. [12 ] obtained O ( d T log K ) O(\sqrt{dT\log K}) O ( d T log K ) regret for K K K actions by exploring according to John’s theorem. High-probability bounds against adaptive adversaries are more delicate, and existing approaches control the variance of importance weighting either through biased reward estimates and bonuses ([6 , 46 ] ) or by combining unbiased estimators with an increasing learning rate schedule (Lee et al. [31 ] ). Zimmert and Lattimore [46 ] obtained almost minimax optimal high-probability bounds, using exponential weights for finite action sets and, for general action sets, FTRL with a log-determinant potential over lifted covariance matrices of the action distribution. In the presence of corruptions, Liu et al. [36 ] extended this lifted log-determinant FTRL approach to weak adversaries and used continuous exponential weights against strong adversaries, while Cheng et al. [14 ] studied adversarial bandit linear optimization over convex action sets with globally bounded perturbations against an oblivious adversary. Building on the principle of inducing a negative comparator-dependent FTRL penalty to cancel estimation or corruption errors ([19 , 46 , 36 ] and the monotone matrix envelope of Liu et al. [36 ] , CRW-FTRL runs FTRL over the K K K -dimensional simplex with a radius-weighted entropy regularizer, while using a log-determinant barrier in an inner problem that selects the per-action radii for a normalized kernel estimator.
Multi-Armed Bandits with Adversarial Corruptions. Lykouris et al. [37 ] introduced stochastic multi-armed bandits with adversarial corruptions and proposed a corruption-agnostic algorithm whose regret incurs a multiplicative O ~ ( K C ) \widetilde{O}(KC) O ( K C ) factor on top of the corruption-free regret. Gupta et al. [21 ] improved this to a bound in which the corruption enters additively as O ( K C ) O(KC) O ( K C ) . In both works, the adversary corrupts the rewards of all arms before the learner draws its arm, which corresponds to the weak adversary in our terminology. For adversarial multi-armed bandits, Yang et al. [42 ] studied oblivious clean rewards with corruptions chosen after observing the pulled arm and obtained O ( K T log K + K C log T ) O(\sqrt{KT\log K}+KC\log T) O ( K T log K + K C log T ) regret for a known corruption budget C C C , whereas Zimmert and Seldin [47 ] developed a best-of-both-worlds algorithm that is simultaneously optimal for stochastic and adversarial rewards and remains robust in corrupted stochastic environments. Corruption robustness has also been studied in episodic reinforcement learning ([38 , 41 ] ). From the attacker’s perspective, Jun et al. [25 ] and Liu and Shroff [35 ] showed that a small corruption budget suffices to steer standard multi-armed bandit algorithms toward a target arm.
Stochastic Linear Bandits with Adversarial Corruptions. In the linear setting, Garcelon et al. [20 ] showed that standard linear contextual bandit algorithms are similarly vulnerable to targeted data poisoning attacks. On the defense side, Li et al. (2019) obtained gap-dependent bounds for stochastic linear bandits against a weak adversary, measuring the corruption by its per-round maximum over actions. Bogunovic et al. [11 ] considered a strong adversary that observes the pulled arm and the noise before corrupting, proposed a robust phased elimination algorithm with O ~ ( d T log K + d 3 / 2 / C ) \widetilde{O}(\sqrt{dT}\log K+d^{3/2}/C) O ( d T log K + d 3/2 / C ) regret for known C C C , and proved that any algorithm must incur Ω ( d C ) \Omega(dC) Ω ( d C ) regret even on an instance with K = d K=d K = d orthogonal arms; see also Bogunovic et al. [10 ] for Gaussian process bandits. Best-of-both-worlds algorithms for linear and linear contextual bandits that remain robust to corrupted stochastic rewards were developed by Lee et al. [32 ] and Kuroki et al. [28 ] . For linear contextual bandits, Zhao et al. [45 ] (under weak corruption) and Ding et al. [18 ] (under strong corruption) obtained corruption terms with a polynomial dependence on T T T , and He et al. (2022) achieved the nearly optimal O ~ ( d T + d C ) \widetilde{O}(d\sqrt{T}+dC) O ( d T + d C ) regret against a strong adversary by weighting each observation according to its uncertainty. This uncertainty-weighting idea has since been extended to nonlinear function classes (Ye et al., 2023), variance-aware bounds (Yu et al., 2025), and dueling bandits ([17 ] ). A closely related line of work studies misspecified linear bandits, where the reward deviates from a linear function by a bounded amount in every round ([30 , 19 , 40 ] ), and Liu et al. [36 ] unified the weak (C ∞ C_{\infty} C ∞ ) and strong (C C C ) corruption measures across stochastic and adversarial linear bandits. Table 1 compares our result with the most related work on linear bandits with corruptions.
Table 1 : Comparison of regret bounds for linear bandits with corruptions. “Stoch.” and “Adv.” indicate whether the reward vectors θ t \boldsymbol{\theta}_t θ t are fixed or chosen by an adaptive adversary. C = ∑ t = 1 T ∣ c t ∣ C = \sum_{t=1}^{T} |c_t| C = ∑ t = 1 T ∣ c t ∣ is the strong corruption measure and C ∞ ≥ C C_\infty\ge C C ∞ ≥ C is the weak measure of Liu et al. [36 ] . K K K is the number of actions, O ~ ( ⋅ ) \widetilde{O}(\cdot) O ( ⋅ ) and Ω ~ ( ⋅ ) \widetilde{\Omega}(\cdot) Ω ( ⋅ ) hide logarithmic factors, and the lower bounds hold in the nontrivial regime where the stated bound is at most O ( T ) O(T) O ( T ) .
Table 1.
Problem Setting
Adversarial Linear Bandits with Strong Corruption. Let A = { a 1 , … , a K } ⊂ R d \mathcal{A}=\{\mathbf{a}_1,\ldots,\mathbf{a}_K\}\subset\mathbb{R}^{d} A = { a 1 , … , a K } ⊂ R d be a finite action set known to the learner. The interaction between the learner and an adaptive adversary proceeds over T T T rounds. In each round t ∈ [ T ] t\in[T] t ∈ [ T ] :
Based on the interaction history before round t t t , the adversary chooses a reward vector θ t ∈ R d \boldsymbol{\theta}_t\in\mathbb{R}^{d} θ t ∈ R d , which defines the clean reward a i T θ t \mathbf{a}_i^{\mathsf{T}}\boldsymbol{\theta}_t a i T θ t of each action i ∈ [ K ] i\in[K] i ∈ [ K ] , while the learner simultaneously chooses a sampling distribution p t ∈ Δ K \mathbf{p}_t\in\Delta_K p t ∈ Δ K and draws an action I t ∼ p t I_t\sim\mathbf{p}_t I t ∼ p t .
A zero-mean noise variable ξ t ∈ R \xi_t\in\mathbb{R} ξ t ∈ R is realized, yielding the clean observation a I t T θ t + ξ t \mathbf{a}_{I_t}^{\mathsf{T}}\boldsymbol{\theta}_t+\xi_t a I t T θ t + ξ t .
After observing both the realized action I t I_t I t and the clean observation a I t T θ t + ξ t \mathbf{a}_{I_t}^{\mathsf{T}}\boldsymbol{\theta}_t+\xi_t a I t T θ t + ξ t , the adversary chooses a corruption c t ∈ R c_t\in\mathbb{R} c t ∈ R , and the learner observes only the corrupted reward
y t = a I t T θ t + ξ t + c t . y_t=\mathbf{a}_{I_t}^{\mathsf{T}}\boldsymbol{\theta}_t+\xi_t+c_t. y t = a I t T θ t + ξ t + c t .
We impose the following regularity conditions on the action set, noise, and corruption budget.
Assumption 3.1 (Actions and Clean Rewards). The action set A = { a 1 , … , a K } ⊂ R d \mathcal{A}=\{\mathbf{a}_1,\ldots,\mathbf{a}_K\}\subset\mathbb{R}^{d} A = { a 1 , … , a K } ⊂ R d is finite and its affine hull is R d \mathbb{R}^{d} R d . Almost surely, ∣ a i T θ t ∣ ≤ 1 |\mathbf{a}_i^{\mathsf{T}}\boldsymbol{\theta}_t|\le1 ∣ a i T θ t ∣ ≤ 1 for all i ∈ [ K ] i\in[K] i ∈ [ K ] and t ∈ [ T ] t\in[T] t ∈ [ T ] .
Assumption 3.2 (Filtration, Noise, and Corruption Budget). Let F t \mathcal{F}_t F t be the σ \sigma σ -field generated by { θ s , I s , ξ s , c s } s = 1 t \{\boldsymbol{\theta}_s,I_s,\xi_s,c_s\}_{s=1}^{t} { θ s , I s , ξ s , c s } s = 1 t , together with θ t + 1 \boldsymbol{\theta}_{t+1} θ t + 1 and the internal randomness of the environment and the adversary prior to sampling I t + 1 I_{t+1} I t + 1 . For all t ∈ [ T ] t\in[T] t ∈ [ T ] :
(a) θ t \boldsymbol{\theta}_t θ t and p t \mathbf{p}_t p t are F t − 1 \mathcal{F}_{t-1} F t − 1 -measurable, and P ( I t = j ∣ F t − 1 ) = p t j \mathbb{P}(I_t=j\mid\mathcal{F}_{t-1})=p_{tj} P ( I t = j ∣ F t − 1 ) = p t j for all j ∈ [ K ] j\in[K] j ∈ [ K ] .
(b) ∣ ξ t ∣ ≤ 1 |\xi_t|\le1 ∣ ξ t ∣ ≤ 1 almost surely, and E [ ξ t ∣ F t − 1 , I t ] = 0 \mathbb{E}[\xi_t\mid\mathcal{F}_{t-1},I_t]=0 E [ ξ t ∣ F t − 1 , I t ] = 0 .
(c) ∣ c t ∣ ≤ 1 |c_t| \le1 ∣ c t ∣ ≤ 1 and ∑ t = 1 T ∣ c t ∣ ≤ C \sum_{t=1}^{T} |c_t| \le C ∑ t = 1 T ∣ c t ∣ ≤ C almost surely, where C ≥ 0 C \ge0 C ≥ 0 is the total corruption budget.
(d) The learner knows the action set A \mathcal{A} A , the horizon T T T , and the corruption budget C C C .
Under Assumptions 3.1 and 3.2 , the clean observation and the corrupted observation are bounded almost surely by ∣ a I t ⊤ θ t + ξ t ∣ ≤ 2 |\mathbf{a}_{I_t}^{\top}\theta_t + \xi_t| \le2 ∣ a I t ⊤ θ t + ξ t ∣ ≤ 2 and ∣ y t ∣ ≤ 3 |y_t| \le3 ∣ y t ∣ ≤ 3 . The learner’s goal is to minimize the realized regret evaluated on the clean rewards.
Definition 3.3 (Regret). The realized regret after T T T rounds is defined as
Reg T : = max i ∈ [ K ] ∑ t = 1 T ( a i − a I t ) ⊤ θ t . \operatorname{Reg}_T := \max_{i \in[K]} \sum_{t=1}^{T} (\mathbf{a}_i - \mathbf{a}_{I_t})^\top\theta_t. Reg T := i ∈ [ K ] max t = 1 ∑ T ( a i − a I t ) ⊤ θ t .
Algorithm
Our algorithm, CRW-FTRL (Algorithm 1), maintains a sampling distribution over the K K K actions and updates it by follow-the-regularized-leader (FTRL) on estimates of the clean rewards. Before play, it fixes a reference distribution over the actions and a change of coordinates computed from it (§ 4.1 ). During play, each round has two parts (§ 4.3 ): the learner computes the sampling distribution from the past estimates, and after observing the reward of the sampled action it forms an estimate of the clean reward of every action.
Reference Design and Whitened Actions
Before play begins, the algorithm performs a static change of coordinates that depends only on the action set A \mathcal{A} A . We first lift each action a i ∈ R d \mathbf{a}_i \in\mathbb{R}^{d} a i ∈ R d to z i : = ( a i , 1 ) T ∈ R d + 1 \mathbf{z}_i := (\mathbf{a}_i, 1)^{\mathsf{T}} \in\mathbb{R}^{d+1} z i := ( a i , 1 ) T ∈ R d + 1 by appending a constant coordinate 1. Lifting preserves linearity of the clean rewards ( a i T θ t = z i T ( θ t T , 0 ) T ) (\mathbf{a}_i^{\mathsf{T}}\boldsymbol{\theta}_t = \mathbf{z}_i^{\mathsf{T}}(\boldsymbol{\theta}_t^{\mathsf{T}}, 0)^{\mathsf{T}}) ( a i T θ t = z i T ( θ t T , 0 ) T ) while ensuring that every lifted action has a constant last coordinate z i , d + 1 = 1 z_{i,d+1} = 1 z i , d + 1 = 1 , which will directly supply the uniform norm lower bound ∥ x i ∥ ≥ 1 \lVert\mathbf{x}_i\rVert\ge1 ∥ x i ∥ ≥ 1 (and hence r t i ≥ 1 r_{t i} \ge1 r t i ≥ 1 ) needed by our normalized kernel estimator without requiring any lower bound on ∥ a i ∥ \lVert\mathbf{a}_i\rVert ∥ a i ∥ . Moreover, since A \mathcal{A} A has full affine span in R d \mathbb{R}^{d} R d (Assumption 3.1), the lifted actions z 1 , … , z K \mathbf{z}_1,\ldots,\mathbf{z}_K z 1 , … , z K linearly span R d + 1 \mathbb{R}^{d+1} R d + 1 .
To put the lifted actions into a canonical position, we use a D D D -optimal (G G G -optimal) experimental design over { z 1 , … , z K } \{\mathbf{z}_1,\ldots,\mathbf{z}_K\} { z 1 , … , z K } .
Definition 4.1 (Reference Design). Let H ( p ) : = ∑ i = 1 K p i z i z i T \mathbf{H}(\mathbf{p}) := \sum_{i=1}^{K} p_i\mathbf{z}_i\mathbf{z}_i^{\mathsf{T}} H ( p ) := ∑ i = 1 K p i z i z i T for p ∈ Δ K \mathbf{p} \in\Delta_K p ∈ Δ K , and let ρ ⋆ ∈ Δ K \boldsymbol{\rho}^{\star} \in\Delta_K ρ ⋆ ∈ Δ K be a maximizer of log det H ( p ) \log\det\mathbf{H}(\mathbf{p}) log det H ( p ) . Set
ρ : = 1 2 ( ρ ⋆ + 1 K 1 ) , x i : = H ( ρ ) − 1 / 2 z i . \boldsymbol{\rho} := \frac{1}{2}\left(\boldsymbol{\rho}^{\star}+\frac{1}{K}\mathbf{1}\right), \qquad\mathbf{x}_i := \mathbf{H}(\boldsymbol{\rho})^{-1/2}\mathbf{z}_i. ρ := 2 1 ( ρ ⋆ + K 1 1 ) , x i := H ( ρ ) − 1/2 z i . Because z 1 , … , z K \mathbf{z}_1,\ldots,\mathbf{z}_K z 1 , … , z K span R d + 1 \mathbb{R}^{d+1} R d + 1 , the matrix H ( p ) \mathbf{H}(\mathbf{p}) H ( p ) is positive definite whenever p \mathbf{p} p has full support, so the maximum of log det H ( p ) \log\det\mathbf{H}(\mathbf{p}) log det H ( p ) over the compact simplex Δ K \Delta_K Δ K is finite and attained at some design ρ ⋆ \boldsymbol{\rho}^{\star} ρ ⋆ with H ( ρ ⋆ ) ≻ 0 \mathbf{H}(\boldsymbol{\rho}^{\star}) \succ0 H ( ρ ⋆ ) ≻ 0 . By the Kiefer–Wolfowitz equivalence theorem ([26 ] ), maximizing log det H ( p ) \log\det\mathbf{H}(\mathbf{p}) log det H ( p ) (D D D -optimality) is equivalent to minimizing the worst-case leverage score max i ∈ [ K ] z i T H ( p ) − 1 z i \max_{i\in[K]} \mathbf{z}_i^{\mathsf{T}}\mathbf{H}(\mathbf{p})^{-1}\mathbf{z}_i max i ∈ [ K ] z i T H ( p ) − 1 z i (G G G -optimality), and the optimal design ρ ⋆ \boldsymbol{\rho}^{\star} ρ ⋆ satisfies
z i T H ( ρ ⋆ ) − 1 z i ≤ d + 1 for all i ∈ [ K ] . \mathbf{z}_i^{\mathsf{T}}\mathbf{H}(\boldsymbol{\rho}^{\star})^{-1}\mathbf{z}_i \le d+1 \qquad\text{for all } i\in[K]. z i T H ( ρ ⋆ ) − 1 z i ≤ d + 1 for all i ∈ [ K ] . The optimal design ρ ⋆ \boldsymbol{\rho}^{\star} ρ ⋆ cannot be used directly as a floor on the sampling probabilities because it may assign zero mass to some actions. The reference design ρ \boldsymbol{\rho} ρ in Definition 4.1 remedies this by mixing ρ ⋆ \boldsymbol{\rho}^{\star} ρ ⋆ with the uniform distribution 1 K 1 \frac{1}{K}\mathbf{1} K 1 1 . This mixture retains the benefits of both components:
Isotropy and bounded norms: Under the whitened coordinates x i = H ( ρ ) − 1 / 2 z i \mathbf{x}_i = \mathbf{H}(\boldsymbol{\rho})^{-1/2}\mathbf{z}_i x i = H ( ρ ) − 1/2 z i , the actions are isotropic under ρ \boldsymbol{\rho} ρ and have bounded norms: since H ( ρ ) ⪰ 1 2 H ( ρ ⋆ ) \mathbf{H}(\boldsymbol{\rho}) \succeq\frac{1}{2}\mathbf{H}(\boldsymbol{\rho}^{\star}) H ( ρ ) ⪰ 2 1 H ( ρ ⋆ ) implies H ( ρ ) − 1 ⪯ 2 H ( ρ ⋆ ) − 1 \mathbf{H}(\boldsymbol{\rho})^{-1} \preceq2\mathbf{H}(\boldsymbol{\rho}^{\star})^{-1} H ( ρ ) − 1 ⪯ 2 H ( ρ ⋆ ) − 1 ,
∑ i = 1 K ρ i x i x i T = I , 1 ≤ ∥ x i ∥ 2 = z i T H ( ρ ) − 1 z i ≤ 2 z i T H ( ρ ⋆ ) − 1 z i ≤ 2 ( d + 1 ) for all i ∈ [ K ] , \sum_{i=1}^{K}\rho_i\mathbf{x}_i\mathbf{x}_i^{\mathsf{T}}=\mathbf{I}, \qquad1\le\lVert\mathbf{x}_i\rVert^2=\mathbf{z}_i^{\mathsf{T}}\mathbf{H}(\boldsymbol{\rho})^{-1}\mathbf{z}_i\le2\mathbf{z}_i^{\mathsf{T}}\mathbf{H}(\boldsymbol{\rho}^{\star})^{-1}\mathbf{z}_i\le2(d+1)\quad\text{for all }i\in[K], i = 1 ∑ K ρ i x i x i T = I , 1 ≤ ∥ x i ∥ 2 = z i T H ( ρ ) − 1 z i ≤ 2 z i T H ( ρ ⋆ ) − 1 z i ≤ 2 ( d + 1 ) for all i ∈ [ K ] , where the lower bound ∥ x i ∥ ≥ 1 \lVert\mathbf{x}_i\rVert\ge1 ∥ x i ∥ ≥ 1 follows from the appended coordinate 1.
Linearity of rewards: Defining the transformed parameter ϑ t : = H ( ρ ) 1 / 2 ( θ t T , 0 ) T \boldsymbol{\vartheta}_t := \mathbf{H}(\boldsymbol{\rho})^{1/2}(\boldsymbol{\theta}_t^{\mathsf{T}},0)^{\mathsf{T}} ϑ t := H ( ρ ) 1/2 ( θ t T , 0 ) T , the clean reward of every action i i i is
a i T θ t = z i T ( θ t T , 0 ) T = x i T ϑ t . \mathbf{a}_i^{\mathsf{T}}\boldsymbol{\theta}_t=\mathbf{z}_i^{\mathsf{T}}(\boldsymbol{\theta}_t^{\mathsf{T}},0)^{\mathsf{T}}=\mathbf{x}_i^{\mathsf{T}}\boldsymbol{\vartheta}_t. a i T θ t = z i T ( θ t T , 0 ) T = x i T ϑ t .
From now on, the algorithm uses A \mathcal{A} A only through the whitened actions x 1 , … , x K \mathbf{x}_1,\ldots,\mathbf{x}_K x 1 , … , x K , and treats the reference design ρ \boldsymbol{\rho} ρ as known.
Motivation: Controlling Sensitivity via Action Radii
In each round t t t , our algorithm plays a sampling distribution p t ∈ Δ K \mathbf{p}_t \in\Delta_K p t ∈ Δ K computed by follow-the-regularized-leader (FTRL) on the cumulative reward estimates S t − 1 = ∑ s < t g ^ s \mathbf{S}_{t-1} = \sum_{s<t} \widehat{\mathbf{g}}_s S t − 1 = ∑ s < t g s , draws an action I t ∼ p t I_t \sim\mathbf{p}_t I t ∼ p t , and uses the observed reward y t y_t y t to construct a new reward estimate vector g ^ t ∈ R K \widehat{\mathbf{g}}_t \in\mathbb{R}^K g t ∈ R K .
In standard EXP3 for linear bandits, the learner estimates the reward of action i i i by
g ^ t I W ( i ) = x i T ( G t I W ) − 1 x I t y t , where G t I W = ∑ j = 1 K p t j x j x j T . \widehat{g}^{\mathrm{IW}}_t(i) = \mathbf{x}_i^{\mathsf{T}} \left(\mathbf{G}^{\mathrm{IW}}_t\right)^{-1}\mathbf{x}_{I_t}y_t,
\qquad\text{where}\quad\mathbf{G}^{\mathrm{IW}}_t = \sum_{j=1}^{K}p_{tj}\mathbf{x}_j\mathbf{x}_j^{\mathsf{T}}. g t IW ( i ) = x i T ( G t IW ) − 1 x I t y t , where G t IW = j = 1 ∑ K p t j x j x j T .
In round t t t , the corruption c t c_t c t in y t = a I t T θ t + ξ t + c t y_t = \mathbf{a}_{I_t}^{\mathsf{T}}\boldsymbol{\theta}_t + \xi_t + c_t y t = a I t T θ t + ξ t + c t shifts this estimate by
Δ t I W ( i ) = c t x i T ( G t I W ) − 1 x I t . \Delta^{\mathrm{IW}}_t(i) = c_t\mathbf{x}_i^{\mathsf{T}}\left(\mathbf{G}^{\mathrm{IW}}_t\right)^{-1}\mathbf{x}_{I_t}. Δ t IW ( i ) = c t x i T ( G t IW ) − 1 x I t .
When an action j j j has small sampling probability p t j p_{tj} p t j , its leverage score can be as large as
x j T ( G t I W ) − 1 x j ≤ 1 p t j , \mathbf{x}_j^{\mathsf{T}}\left(\mathbf{G}^{\mathrm{IW}}_t\right)^{-1}\mathbf{x}_j \le\frac{1}{p_{tj}}, x j T ( G t IW ) − 1 x j ≤ p t j 1 ,
with equality whenever x j \mathbf{x}_j x j is orthogonal to the remaining actions. Under weak corruption (c t c_t c t fixed before drawing I t ∼ p t I_t \sim\mathbf{p}_t I t ∼ p t ), the factor 1 / p t j 1/p_{tj} 1/ p t j cancels against P ( I t = j ∣ F t − 1 ) = p t j \mathbb{P}(I_t = j \mid\mathcal{F}_{t-1}) = p_{tj} P ( I t = j ∣ F t − 1 ) = p t j in expectation. Under strong corruption, however, the adversary sees I t I_t I t before choosing c t c_t c t : it can set c t = 0 c_t = 0 c t = 0 unless an action I t = j I_t = j I t = j with tiny probability p t j p_{tj} p t j is drawn, in which case ∣ c t ∣ = 1 \lvert c_t\rvert= 1 ∣ c t ∣ = 1 perturbs g ^ t I W ( j ) \widehat{g}^{\mathrm{IW}}_t(j) g t IW ( j ) by up to 1 / p t j 1/p_{tj} 1/ p t j .
The issue is that in g ^ t I W ( i ) = x i T ( G t I W ) − 1 x I t y t \widehat{g}^{\mathrm{IW}}_t(i) = \mathbf{x}_i^{\mathsf{T}}\left(\mathbf{G}^{\mathrm{IW}}_t\right)^{-1}\mathbf{x}_{I_t}y_t g t IW ( i ) = x i T ( G t IW ) − 1 x I t y t , the weight on y t y_t y t is bounded by Cauchy–Schwarz as
∣ x i T ( G t I W ) − 1 x j ∣ ≤ x i T ( G t I W ) − 1 x i x j T ( G t I W ) − 1 x j . \left\lvert\mathbf{x}_i^{\mathsf{T}}\left(\mathbf{G}^{\mathrm{IW}}_t\right)^{-1}\mathbf{x}_j\right\rvert
\le\sqrt{\mathbf{x}_i^{\mathsf{T}}\left(\mathbf{G}^{\mathrm{IW}}_t\right)^{-1}\mathbf{x}_i}\sqrt{\mathbf{x}_j^{\mathsf{T}}\left(\mathbf{G}^{\mathrm{IW}}_t\right)^{-1}\mathbf{x}_j}. x i T ( G t IW ) − 1 x j ≤ x i T ( G t IW ) − 1 x i x j T ( G t IW ) − 1 x j .
where the second factor depends on the drawn action j = I t j = I_t j = I t and can be as large as 1 / p t j 1/\sqrt{p_{tj}} 1/ p t j . To eliminate this dependence on I t I_t I t , our algorithm assigns each action i i i a radius r t i = r i ( Q t ) : = x i T Q t x i ≥ 1 r_{ti} = r_i(\mathbf{Q}_t) := \sqrt{\mathbf{x}_i^{\mathsf{T}}\mathbf{Q}_t\mathbf{x}_i} \ge1 r t i = r i ( Q t ) := x i T Q t x i ≥ 1 in round t t t via a matrix Q t ≻ 0 \mathbf{Q}_t \succ0 Q t ≻ 0 , and replaces x i T ( G t I W ) − 1 x I t y t \mathbf{x}_i^{\mathsf{T}}\left(\mathbf{G}^{\mathrm{IW}}_t\right)^{-1}\mathbf{x}_{I_t}y_t x i T ( G t IW ) − 1 x I t y t with the reweighted kernel estimate k t ( i , I t ) y t k_t(i,I_t)y_t k t ( i , I t ) y t , where
k t ( i , j ) : = x i T G t − 1 x j r t j , G t : = ∑ l = 1 K p t l r t l x l x l T . k_t(i,j) := \frac{\mathbf{x}_i^{\mathsf{T}}\mathbf{G}_t^{-1}\mathbf{x}_j}{r_{tj}},
\qquad
\mathbf{G}_t := \sum_{l=1}^{K}\frac{p_{tl}}{r_{tl}}\mathbf{x}_l\mathbf{x}_l^{\mathsf{T}}. k t ( i , j ) := r t j x i T G t − 1 x j , G t := l = 1 ∑ K r tl p tl x l x l T .
Here G t \mathbf{G}_t G t is defined with weights p t l / r t l p_{tl}/r_{tl} p tl / r tl so that k t ( i , I t ) y t k_t(i,I_t)y_t k t ( i , I t ) y t remains an unbiased estimator of the clean reward a i T θ t = x i T ϑ t \mathbf{a}_i^{\mathsf{T}}\boldsymbol{\theta}_t = \mathbf{x}_i^{\mathsf{T}}\boldsymbol{\vartheta}_t a i T θ t = x i T ϑ t when c t = 0 c_t = 0 c t = 0 : taking expectation over I t ∼ p t I_t \sim\mathbf{p}_t I t ∼ p t ,
E [ k t ( i , I t ) x I t T ϑ t ∣ F t − 1 ] = ∑ j = 1 K p t j x i T G t − 1 x j r t j x j T ϑ t = x i T G t − 1 ( ∑ j = 1 K p t j r t j x j x j T ) ϑ t = x i T ϑ t = a i T θ t . \mathbb{E}\left[k_t(i,I_t)\mathbf{x}_{I_t}^{\mathsf{T}}\boldsymbol{\vartheta}_t \mid\mathcal{F}_{t-1}\right]
= \sum_{j=1}^{K}p_{tj}\frac{\mathbf{x}_i^{\mathsf{T}}\mathbf{G}_t^{-1}\mathbf{x}_j}{r_{tj}}\mathbf{x}_j^{\mathsf{T}}\boldsymbol{\vartheta}_t
= \mathbf{x}_i^{\mathsf{T}}\mathbf{G}_t^{-1}\left(\sum_{j=1}^{K}\frac{p_{tj}}{r_{tj}}\mathbf{x}_j\mathbf{x}_j^{\mathsf{T}}\right)\boldsymbol{\vartheta}_t
= \mathbf{x}_i^{\mathsf{T}}\boldsymbol{\vartheta}_t
= \mathbf{a}_i^{\mathsf{T}}\boldsymbol{\theta}_t. E [ k t ( i , I t ) x I t T ϑ t ∣ F t − 1 ] = j = 1 ∑ K p t j r t j x i T G t − 1 x j x j T ϑ t = x i T G t − 1 ( j = 1 ∑ K r t j p t j x j x j T ) ϑ t = x i T ϑ t = a i T θ t .
At the same time, dividing by r t j r_{tj} r t j in k t ( i , j ) k_t(i,j) k t ( i , j ) cancels the factor from the drawn action j = I t j = I_t j = I t : whenever the radii satisfy the envelope condition x l T G t − 1 x l ≤ r t l 2 \mathbf{x}_l^{\mathsf{T}}\mathbf{G}_t^{-1}\mathbf{x}_l \le r_{tl}^{2} x l T G t − 1 x l ≤ r tl 2 for all l ∈ [ K ] l \in[K] l ∈ [ K ] , Cauchy–Schwarz gives
∣ k t ( i , j ) ∣ = ∣ x i T G t − 1 x j ∣ r t j ≤ x i T G t − 1 x i ⋅ x j T G t − 1 x j r t j ≤ r t i ⋅ r t j r t j = r t i for all i , j ∈ [ K ] . \left\lvert k_t(i,j)\right\rvert
= \frac{\left\lvert\mathbf{x}_i^{\mathsf{T}}\mathbf{G}_t^{-1}\mathbf{x}_j\right\rvert}{r_{tj}}
\le\sqrt{\mathbf{x}_i^{\mathsf{T}}\mathbf{G}_t^{-1}\mathbf{x}_i}\cdot\frac{\sqrt{\mathbf{x}_j^{\mathsf{T}}\mathbf{G}_t^{-1}\mathbf{x}_j}}{r_{tj}}
\le r_{ti}\cdot\frac{r_{tj}}{r_{tj}}
= r_{ti}
\qquad\text{for all } i,j \in[K]. ∣ k t ( i , j ) ∣ = r t j x i T G t − 1 x j ≤ x i T G t − 1 x i ⋅ r t j x j T G t − 1 x j ≤ r t i ⋅ r t j r t j = r t i for all i , j ∈ [ K ] .
Thus, whichever action I t I_t I t is drawn, a corruption c t c_t c t shifts action i i i ’s estimate k t ( i , I t ) y t k_t(i,I_t)y_t k t ( i , I t ) y t by at most ∣ c t ∣ r t i |c_t|r_{ti} ∣ c t ∣ r t i . Although r t i r_{ti} r t i itself can be large when p t i p_{ti} p t i is small, the bound now depends only on i i i : if the radii are nondecreasing (r 1 i ≤ ⋯ ≤ r T i r_{1i} \le\cdots\le r_{Ti} r 1 i ≤ ⋯ ≤ r T i ), the total corruption accumulated on any comparator u ∈ Δ K \mathbf{u} \in\Delta_K u ∈ Δ K over T T T rounds is bounded by
∑ t = 1 T ∑ i = 1 K u i c t k t ( i , I t ) ≤ ∑ t = 1 T ∣ c t ∣ ∑ i = 1 K u i r t i ≤ C ⟨ u , r T ⟩ . \sum_{t=1}^{T}\sum_{i=1}^{K}u_i c_t k_t(i,I_t) \le\sum_{t=1}^{T}|c_t|\sum_{i=1}^{K}u_i r_{ti} \le C\langle\mathbf{u},\mathbf{r}_T\rangle. t = 1 ∑ T i = 1 ∑ K u i c t k t ( i , I t ) ≤ t = 1 ∑ T ∣ c t ∣ i = 1 ∑ K u i r t i ≤ C ⟨ u , r T ⟩ .
which is linear in the comparator’s final radius ⟨ u , r T ⟩ \langle\mathbf{u},\mathbf{r}_T\rangle ⟨ u , r T ⟩ and can therefore be canceled directly by a negative linear term − α ⟨ p , r t ⟩ -\alpha\langle\mathbf{p},\mathbf{r}_t\rangle − α ⟨ p , r t ⟩ in the FTRL regularizer. Building on this idea, our algorithm consists of four components, which we detail in § 4.3 :
Floored simplex P : = { p ∈ Δ K : p i ≥ ρ i / ( 2 T ) } \mathcal{P} := \{\mathbf{p} \in\Delta_K : p_i \ge\rho_i/(2T)\} P := { p ∈ Δ K : p i ≥ ρ i / ( 2 T )} : Enforcing p i ≥ ρ i / ( 2 T ) p_i \ge\rho_i/(2T) p i ≥ ρ i / ( 2 T ) ensures log ( 1 / p i ) ≤ log ( 4 K T ) \log(1/p_i) \le\log(4KT) log ( 1/ p i ) ≤ log ( 4 K T ) for all i ∈ [ K ] i \in[K] i ∈ [ K ] .
Radius-weighted regularizer : Weighting coordinate i i i of the negative entropy by r t i r_{ti} r t i keeps FTRL stable for estimates of scale r t i r_{ti} r t i , while a linear term − α ∑ i r t i p i -\alpha\sum_i r_{ti}p_i − α ∑ i r t i p i contributes − α ⟨ u , r T ⟩ -\alpha\langle\mathbf{u},\mathbf{r}_T\rangle − α ⟨ u , r T ⟩ at any comparator u \mathbf{u} u to cancel its estimation and corruption errors.
Monotone log-det inner problem : Choosing Q t ⪰ Q t − 1 \mathbf{Q}_t \succeq\mathbf{Q}_{t-1} Q t ⪰ Q t − 1 jointly with p t \mathbf{p}_t p t enforces x l ⊤ G t − 1 x l ≤ r t l 2 \mathbf{x}_l^\top\mathbf{G}_t^{-1}\mathbf{x}_l \le r_{tl}^2 x l ⊤ G t − 1 x l ≤ r tl 2 via first-order optimality while bounding the learner’s regularizer drift by O ( κ d log ( d T ) ) O(\kappa d\log(dT)) O ( κ d log ( d T )) .
Normalized kernel estimator : Centering k t ( i , j ) k_t(i,j) k t ( i , j ) to k ~ t ( i , j ) \widetilde{k}_t(i,j) k t ( i , j ) keeps k ~ t \widetilde{k}_t k t unbiased with ∣ k ~ t ( i , j ) ∣ ≤ 3 r t i |\widetilde{k}_t(i,j)| \le3r_{ti} ∣ k t ( i , j ) ∣ ≤ 3 r t i , and enforces ∑ i p t i k ~ t ( i , I t ) = 1 \sum_i p_{ti}\widetilde{k}_t(i,I_t)=1 ∑ i p t i k t ( i , I t ) = 1 so that c t c_t c t enters the learner’s own reward with weight 1.
CRW-FTRL: Corruption-Robust Radius-Weighted FTRL
Floored Simplex. The learner optimizes p t \mathbf{p}_t p t over the floored simplex
P : = { p ∈ Δ K : p i ≥ ρ i / ( 2 T ) for all i ∈ [ K ] } . \mathcal{P} := \{\mathbf{p} \in\Delta_K : p_i \ge\rho_i/(2T)\text{ for all }i \in[K]\}. P := { p ∈ Δ K : p i ≥ ρ i / ( 2 T ) for all i ∈ [ K ]} .
Since ρ i ≥ 1 / ( 2 K ) \rho_i \ge1/(2K) ρ i ≥ 1/ ( 2 K ) (§ 4.1 ), every p ∈ P \mathbf{p} \in\mathcal{P} p ∈ P satisfies p i ≥ 1 / ( 4 K T ) p_i \ge1/(4KT) p i ≥ 1/ ( 4 K T ) and hence 0 ≤ log ( 1 / p i ) ≤ log ( 4 K T ) 0 \le\log(1/p_i) \le\log(4KT) 0 ≤ log ( 1/ p i ) ≤ log ( 4 K T ) . Meanwhile, mixing any comparator with 1 2 T ρ \frac{1}{2T}\boldsymbol{\rho} 2 T 1 ρ changes its clean reward by at most 1 / T 1/T 1/ T per round, costing at most 1 over T T T rounds.
Regularizer Weighted by Radii. Given cumulative reward estimates S t − 1 = ∑ s = 1 t − 1 g ^ s \mathbf{S}_{t-1} = \sum_{s=1}^{t-1}\widehat{\mathbf{g}}_s S t − 1 = ∑ s = 1 t − 1 g s and a convex regularizer ψ t : P → R \psi_t:\mathcal{P} \to\mathbb{R} ψ t : P → R , the follow-the-regularized-leader (FTRL) rule selects p t ∈ P \mathbf{p}_t \in\mathcal{P} p t ∈ P by maximizing the cumulative estimated reward minus the regularizer:
p t = arg max p ∈ P { ⟨ S t − 1 , p ⟩ − ψ t ( p ) } . \mathbf{p}_t = \mathop{\arg\max}_{\mathbf{p}\in\mathcal{P}}\{\langle\mathbf{S}_{t-1},\mathbf{p}\rangle-\psi_t(\mathbf{p})\}. p t = arg max p ∈ P {⟨ S t − 1 , p ⟩ − ψ t ( p )} .
For any comparator u ∈ P \mathbf{u} \in\mathcal{P} u ∈ P , standard FTRL analysis (§ 6.2 ) bounds the regret on the estimated rewards by three terms:
∑ t = 1 T ⟨ u − p t , g ^ t ⟩ ≲ ψ T ( u ) − ψ 0 ( p 1 ) ⏟ penalty at comparator u + ∑ t = 1 T ( ψ t − 1 ( p t ) − ψ t ( p t ) ) ⏟ regularizer drift at p t + ∑ t = 1 T ∑ i = 1 K g ^ t ( i ) 2 ∂ p i 2 ψ t ( p t ) ⏟ stability term . \sum_{t=1}^{T}\langle\mathbf{u}-\mathbf{p}_t,\widehat{\mathbf{g}}_t\rangle\lesssim\underbrace{\psi_T(\mathbf{u})-\psi_0(\mathbf{p}_1)}_{\text{penalty at comparator }\mathbf{u}}+\underbrace{\sum_{t=1}^{T}\bigl(\psi_{t-1}(\mathbf{p}_t)-\psi_t(\mathbf{p}_t)\bigr)}_{\text{regularizer drift at }\mathbf{p}_t}+\underbrace{\sum_{t=1}^{T}\sum_{i=1}^{K}\frac{\widehat{g}_t(i)^2}{\partial_{p_i}^2\psi_t(\mathbf{p}_t)}}_{\text{stability term}}. t = 1 ∑ T ⟨ u − p t , g t ⟩ ≲ penalty at comparator u ψ T ( u ) − ψ 0 ( p 1 ) + regularizer drift at p t t = 1 ∑ T ( ψ t − 1 ( p t ) − ψ t ( p t ) ) + stability term t = 1 ∑ T i = 1 ∑ K ∂ p i 2 ψ t ( p t ) g t ( i ) 2 .
In standard EXP3, the regularizer is the negative entropy ψ ( p ) = η − 1 ∑ i = 1 K p i log p i \psi(\mathbf{p}) = \eta^{-1} \sum_{i=1}^{K} p_i \log p_i ψ ( p ) = η − 1 ∑ i = 1 K p i log p i with learning rate η > 0 \eta> 0 η > 0 , which has second derivative ∂ p i 2 ψ ( p ) = 1 / ( η p i ) \partial_{p_i}^2 \psi(\mathbf{p}) = 1/(\eta p_i) ∂ p i 2 ψ ( p ) = 1/ ( η p i ) . In our setting, given the radius matrix Q t ≻ 0 \mathbf{Q}_t \succ0 Q t ≻ 0 with radii r t i = r i ( Q t ) r_{t i} = r_i(\mathbf{Q}_t) r t i = r i ( Q t ) , we use ψ t = ψ Q t \psi_t = \psi_{\mathbf{Q}_t} ψ t = ψ Q t , where for any Q ≻ 0 \mathbf{Q} \succ0 Q ≻ 0 and offset α > 0 \alpha> 0 α > 0 we define
ψ Q ( p ) : = ∑ i = 1 K r i ( Q ) ( η − 1 p i log p i − α p i ) = − ∑ i = 1 K r i ( Q ) h ( p i ) , h ( s ) : = s ( α + η − 1 log ( 1 / s ) ) . \psi_{\mathbf{Q}}(\mathbf{p}) := \sum_{i=1}^{K} r_i(\mathbf{Q})\left(\eta^{-1}p_i\log p_i-\alpha p_i\right)=-\sum_{i=1}^{K}r_i(\mathbf{Q})h(p_i), \qquad h(s):=s\left(\alpha+\eta^{-1}\log(1/s)\right). ψ Q ( p ) := i = 1 ∑ K r i ( Q ) ( η − 1 p i log p i − α p i ) = − i = 1 ∑ K r i ( Q ) h ( p i ) , h ( s ) := s ( α + η − 1 log ( 1/ s ) ) .
Each part of ψ Q t ( p ) \psi_{\mathbf{Q}_t}(\mathbf{p}) ψ Q t ( p ) is designed to control one term in the FTRL bound above:
Controlling the stability term via r t i p i log p i r_{t i}p_i\log p_i r t i p i log p i : Multiplying coordinate i i i ’s negative entropy by r t i r_{t i} r t i gives second derivative ∂ p i 2 ψ Q t ( p t ) = r t i / ( η p t i ) \partial_{p_i}^2\psi_{\mathbf{Q}_t}(\mathbf{p}_t)=r_{t i}/(\eta p_{t i}) ∂ p i 2 ψ Q t ( p t ) = r t i / ( η p t i ) . Substituting this into the stability term yields
∑ i = 1 K g ^ t ( i ) 2 ∂ p i 2 ψ Q t ( p t ) = η ∑ i = 1 K p t i r t i g ^ t ( i ) 2 , \sum_{i=1}^{K}\frac{\widehat{g}_t(i)^2}{\partial_{p_i}^2\psi_{\mathbf{Q}_t}(\mathbf{p}_t)}=\eta\sum_{i=1}^{K}\frac{p_{t i}}{r_{t i}}\widehat{g}_t(i)^2, i = 1 ∑ K ∂ p i 2 ψ Q t ( p t ) g t ( i ) 2 = η i = 1 ∑ K r t i p t i g t ( i ) 2 ,
which divides g ^ t ( i ) 2 \widehat{g}_t(i)^2 g t ( i ) 2 by r t i r_{t i} r t i to compensate for the O ( r t i ) O(r_{t i}) O ( r t i ) magnitude of g ^ t ( i ) \widehat{g}_t(i) g t ( i ) .
Making the comparator penalty negative via − α r t i p i -\alpha r_{t i}p_i − α r t i p i : Since u i log u i ≤ 0 u_i\log u_i\leq0 u i log u i ≤ 0 for every u ∈ P \mathbf{u}\in\mathcal{P} u ∈ P , the comparator penalty ψ T ( u ) = ψ Q T ( u ) \psi_T(\mathbf{u})=\psi_{\mathbf{Q}_T}(\mathbf{u}) ψ T ( u ) = ψ Q T ( u ) is negative:
ψ Q T ( u ) = ∑ i = 1 K r T i ( η − 1 u i log u i − α u i ) ≤ − α ∑ i = 1 K u i r T i = − α ⟨ u , r T ⟩ . \psi_{\mathbf{Q}_T}(\mathbf{u})=\sum_{i=1}^{K}r_{T i}\left(\eta^{-1}u_i\log u_i-\alpha u_i\right)\leq-\alpha\sum_{i=1}^{K}u_i r_{T i}=-\alpha\langle\mathbf{u},\mathbf{r}_T\rangle. ψ Q T ( u ) = i = 1 ∑ K r T i ( η − 1 u i log u i − α u i ) ≤ − α i = 1 ∑ K u i r T i = − α ⟨ u , r T ⟩ .
Notice that FTRL bounds the regret on the estimated rewards g ^ t \widehat{\mathbf{g}}_t g t , whereas true regret is measured on the clean rewards a i ⊤ θ t \mathbf{a}_i^\top\boldsymbol{\theta}_t a i ⊤ θ t . Passing from ⟨ u , g ^ t ⟩ \langle\mathbf{u},\widehat{\mathbf{g}}_t\rangle ⟨ u , g t ⟩ to u \mathbf{u} u ’s true clean reward incurs two estimation errors that both scale with the estimator’s range O ( r T i ) O(r_{T i}) O ( r T i ) : a corruption error O ( C ⟨ u , r T ⟩ ) O(C\langle\mathbf{u},\mathbf{r}_T\rangle) O ( C ⟨ u , r T ⟩) from the adversarial corruptions c t \mathbf{c}_t c t in y t \mathbf{y}_t y t , and a concentration error O ( η − 1 log ( K T / δ ) ⟨ u , r T ⟩ ) O(\eta^{-1}\log(KT/\delta)\langle\mathbf{u},\mathbf{r}_T\rangle) O ( η − 1 log ( K T / δ ) ⟨ u , r T ⟩) from the random fluctuations of the clean estimates around their mean a i ⊤ θ t \mathbf{a}_i^\top\boldsymbol{\theta}_t a i ⊤ θ t (due to sampling I t ∼ p t I_t\sim p_t I t ∼ p t and noise ξ t \xi_t ξ t ). Taking α ≥ 3 C + O ( η − 1 log ( K T / δ ) ) \alpha\geq3C+O(\eta^{-1}\log(KT/\delta)) α ≥ 3 C + O ( η − 1 log ( K T / δ )) allows − α ⟨ u , r T ⟩ -\alpha\langle\mathbf{u},\mathbf{r}_T\rangle − α ⟨ u , r T ⟩ to cancel both errors directly. This follows the general principle of inducing a negative terminal comparator penalty (or fixed-point bias) in FTRL to offset estimation or corruption errors [19 , 46 , 36 ] , adapted here to a radius-weighted entropy on the simplex so that strong corruption is controlled through the radii r t i r_{t i} r t i .
Bounding the regularizer drift via h ( p i ) h(p_i) h ( p i ) : Since 0 ≤ log ( 1 / p i ) ≤ log ( 4 K T ) 0\leq\log(1/p_i)\leq\log(4KT) 0 ≤ log ( 1/ p i ) ≤ log ( 4 K T ) for all p ∈ P \mathbf{p}\in\mathcal{P} p ∈ P , setting κ : = α + η − 1 log ( 4 K T ) \kappa:=\alpha+\eta^{-1}\log(4KT) κ := α + η − 1 log ( 4 K T ) sandwiches h ( p i ) h(p_i) h ( p i ) within a constant factor of p i p_i p i :
α p i ≤ h ( p i ) ≤ κ p i for all p ∈ P and i ∈ [ K ] . \alpha p_i\leq h(p_i)\leq\kappa p_i\qquad\text{for all }\mathbf{p}\in\mathcal{P}\text{ and }i\in[K]. α p i ≤ h ( p i ) ≤ κ p i for all p ∈ P and i ∈ [ K ] .
Consequently, when r t i ≥ r t − 1 , i r_{t i}\geq r_{t-1,i} r t i ≥ r t − 1 , i , the regularizer drift in round t t t is bounded by
ψ Q t − 1 ( p t ) − ψ Q t ( p t ) = ∑ i = 1 K h ( p t i ) ( r t i − r t − 1 , i ) ≤ κ ∑ i = 1 K p t i ( r t i − r t − 1 , i ) , \psi_{\mathbf{Q}_{t-1}}(\mathbf{p}_t)-\psi_{\mathbf{Q}_t}(\mathbf{p}_t)=\sum_{i=1}^{K}h(p_{t i})(r_{t i}-r_{t-1,i})\leq\kappa\sum_{i=1}^{K}p_{t i}(r_{t i}-r_{t-1,i}), ψ Q t − 1 ( p t ) − ψ Q t ( p t ) = i = 1 ∑ K h ( p t i ) ( r t i − r t − 1 , i ) ≤ κ i = 1 ∑ K p t i ( r t i − r t − 1 , i ) ,
where the small probability p t i p_{t i} p t i multiplies the radius increment r t i − r t − 1 , i r_{t i}-r_{t-1,i} r t i − r t − 1 , i .
Choosing the Radii via a Monotone Log-Det Problem. In round t t t , the radii r t i r_{ti} r t i must satisfy the envelope condition x i T G t − 1 x i ≤ r t i 2 \mathbf{x}_i^{\mathsf{T}}\mathbf{G}_t^{-1}\mathbf{x}_i \le r_{ti}^{2} x i T G t − 1 x i ≤ r t i 2 and monotonicity r t i ≥ r t − 1 , i r_{ti} \ge r_{t-1,i} r t i ≥ r t − 1 , i for all i ∈ [ K ] i \in[K] i ∈ [ K ] , while keeping the regularizer drift ψ Q t − 1 ( p t ) − ψ Q t ( p t ) \psi_{\mathbf{Q}_{t-1}}(\mathbf{p}_t) - \psi_{\mathbf{Q}_t}(\mathbf{p}_t) ψ Q t − 1 ( p t ) − ψ Q t ( p t ) as small as possible. This motivates the following three-step design (see also Liu et al. [36 ] for a related monotone matrix envelope and log-determinant drift bound under weak corruption):
Matrix parametrization of the radii: Observing that the left-hand side of the envelope condition x i T G t − 1 x i ≤ r t i 2 \mathbf{x}_i^{\mathsf{T}}\mathbf{G}_t^{-1}\mathbf{x}_i \le r_{ti}^{2} x i T G t − 1 x i ≤ r t i 2 is a quadratic form of the ( d + 1 ) × ( d + 1 ) (d+1) \times(d+1) ( d + 1 ) × ( d + 1 ) matrix G t − 1 \mathbf{G}_t^{-1} G t − 1 , we parametrize the upper bound r t i 2 r_{ti}^{2} r t i 2 as a quadratic form of a ( d + 1 ) × ( d + 1 ) (d+1) \times(d+1) ( d + 1 ) × ( d + 1 ) matrix Q t ≻ 0 \mathbf{Q}_t \succ0 Q t ≻ 0 :
r t i = r i ( Q t ) : = x i T Q t x i . r_{ti} = r_i(\mathbf{Q}_t) := \sqrt{\mathbf{x}_i^{\mathsf{T}}\mathbf{Q}_t\mathbf{x}_i}. r t i = r i ( Q t ) := x i T Q t x i .
This reduces the K K K arm-wise monotonicity and envelope conditions to two matrix inequalities in R ( d + 1 ) × ( d + 1 ) \mathbb{R}^{(d+1)\times(d+1)} R ( d + 1 ) × ( d + 1 ) :
Q t ⪰ Q t − 1 ( ⟹ r t i ≥ r t − 1 , i ) , G t = ∑ i = 1 K p t i r i ( Q t ) x i x i T ⪰ Q t − 1 ( ⟹ x i T G t − 1 x i ≤ r t i 2 ) . \mathbf{Q}_t \succeq\mathbf{Q}_{t-1}\ \left(\Longrightarrow r_{ti} \ge r_{t-1,i}\right), \qquad\mathbf{G}_t = \sum_{i=1}^{K}\frac{p_{ti}}{r_i(\mathbf{Q}_t)}\mathbf{x}_i\mathbf{x}_i^{\mathsf{T}} \succeq\mathbf{Q}_t^{-1}\ \left(\Longrightarrow\mathbf{x}_i^{\mathsf{T}}\mathbf{G}_t^{-1}\mathbf{x}_i \le r_{ti}^{2}\right). Q t ⪰ Q t − 1 ( ⟹ r t i ≥ r t − 1 , i ) , G t = i = 1 ∑ K r i ( Q t ) p t i x i x i T ⪰ Q t − 1 ( ⟹ x i T G t − 1 x i ≤ r t i 2 ) .
Log-determinant barrier for the envelope condition: For a given p t ∈ P \mathbf{p}_t \in\mathcal{P} p t ∈ P , choosing Q t ⪰ Q t − 1 \mathbf{Q}_t \succeq\mathbf{Q}_{t-1} Q t ⪰ Q t − 1 faces a trade-off: making Q t \mathbf{Q}_t Q t larger hurts the learner through the regularizer drift
ψ Q t − 1 ( p t ) − ψ Q t ( p t ) = ∑ i = 1 K h ( p t i ) ( r i ( Q t ) − r i ( Q t − 1 ) ) , \psi_{\mathbf{Q}_{t-1}}(\mathbf{p}_t) - \psi_{\mathbf{Q}_t}(\mathbf{p}_t) = \sum_{i=1}^{K} h(p_{ti})\left(r_i(\mathbf{Q}_t) - r_i(\mathbf{Q}_{t-1})\right), ψ Q t − 1 ( p t ) − ψ Q t ( p t ) = i = 1 ∑ K h ( p t i ) ( r i ( Q t ) − r i ( Q t − 1 ) ) ,
so we want to minimize ∑ i = 1 K h ( p t i ) r i ( Q ) = − ψ Q ( p t ) \sum_{i=1}^{K} h(p_{ti})r_i(\mathbf{Q}) = -\psi_{\mathbf{Q}}(\mathbf{p}_t) ∑ i = 1 K h ( p t i ) r i ( Q ) = − ψ Q ( p t ) ; at the same time, Q t \mathbf{Q}_t Q t cannot be too small because it must satisfy G t ⪰ Q t − 1 \mathbf{G}_t \succeq\mathbf{Q}_t^{-1} G t ⪰ Q t − 1 . To enforce this lower bound on Q t \mathbf{Q}_t Q t while minimizing − ψ Q ( p t ) -\psi_{\mathbf{Q}}(\mathbf{p}_t) − ψ Q ( p t ) , we subtract the standard log-determinant barrier κ 2 log det Q \frac{\kappa}{2}\log\det\mathbf{Q} 2 κ log det Q and define
J ( Q , p ) : = ∑ i = 1 K h ( p i ) r i ( Q ) − κ 2 log det Q = − ψ Q ( p ) − κ 2 log det Q . J(\mathbf{Q},\mathbf{p}) := \sum_{i=1}^{K}h(p_i)r_i(\mathbf{Q}) - \frac{\kappa}{2}\log\det\mathbf{Q} = -\psi_{\mathbf{Q}}(\mathbf{p}) - \frac{\kappa}{2}\log\det\mathbf{Q}. J ( Q , p ) := i = 1 ∑ K h ( p i ) r i ( Q ) − 2 κ log det Q = − ψ Q ( p ) − 2 κ log det Q .
Since ∇ Q r i ( Q ) = 1 2 r i ( Q ) x i x i T \nabla_{\mathbf{Q}}r_i(\mathbf{Q}) = \frac{1}{2r_i(\mathbf{Q})}\mathbf{x}_i\mathbf{x}_i^{\mathsf{T}} ∇ Q r i ( Q ) = 2 r i ( Q ) 1 x i x i T and ∇ Q log det Q = Q − 1 \nabla_{\mathbf{Q}}\log\det\mathbf{Q} = \mathbf{Q}^{-1} ∇ Q log det Q = Q − 1 , the gradient of J ( Q , p ) J(\mathbf{Q},\mathbf{p}) J ( Q , p ) with respect to Q \mathbf{Q} Q is
∇ Q J ( Q , p ) = 1 2 ∑ i = 1 K h ( p i ) r i ( Q ) x i x i T − κ 2 Q − 1 . \nabla_{\mathbf{Q}}J(\mathbf{Q},\mathbf{p}) = \frac{1}{2}\sum_{i=1}^{K}\frac{h(p_i)}{r_i(\mathbf{Q})}\mathbf{x}_i\mathbf{x}_i^{\mathsf{T}} - \frac{\kappa}{2}\mathbf{Q}^{-1}. ∇ Q J ( Q , p ) = 2 1 i = 1 ∑ K r i ( Q ) h ( p i ) x i x i T − 2 κ Q − 1 .
Without the monotonicity constraint Q ⪰ Q t − 1 \mathbf{Q} \succeq\mathbf{Q}_{t-1} Q ⪰ Q t − 1 , setting ∇ Q J ( Q , p ) = 0 \nabla_{\mathbf{Q}}J(\mathbf{Q},\mathbf{p}) = 0 ∇ Q J ( Q , p ) = 0 yields Q − 1 = ∑ i = 1 K h ( p i ) / κ x i T Q x i x i x i T \mathbf{Q}^{-1} = \sum_{i=1}^{K}\frac{h(p_i)/\kappa}{\sqrt{\mathbf{x}_i^{\mathsf{T}}\mathbf{Q}\mathbf{x}_i}}\mathbf{x}_i\mathbf{x}_i^{\mathsf{T}} Q − 1 = ∑ i = 1 K x i T Q x i h ( p i ) / κ x i x i T , which is the classical determinant optimization characterization of ℓ 1 \ell_1 ℓ 1 Lewis weights (Lewis, [33 ] ; Cohen and Peng, [15 ] ) for the rescaled rows h ( p i ) κ x i T \frac{h(p_i)}{\kappa}\mathbf{x}_i^{\mathsf{T}} κ h ( p i ) x i T . Minimizing J ( Q , p t ) J(\mathbf{Q},\mathbf{p}_t) J ( Q , p t ) over the constrained cone Q ⪰ Q t − 1 \mathbf{Q} \succeq\mathbf{Q}_{t-1} Q ⪰ Q t − 1 extends this geometry to maintain nondecreasing radii and achieves both goals at once (Lemma B.7): first-order optimality 2 ∇ Q J ( Q t , p t ) ⪰ 0 2\nabla_{\mathbf{Q}}J(\mathbf{Q}_t,\mathbf{p}_t) \succeq0 2 ∇ Q J ( Q t , p t ) ⪰ 0 (together with h ( p t i ) ≤ κ p t i h(p_{ti}) \le\kappa p_{ti} h ( p t i ) ≤ κ p t i ) enforces G t ⪰ Q t − 1 \mathbf{G}_t \succeq\mathbf{Q}_t^{-1} G t ⪰ Q t − 1 , while J ( Q t , p t ) ≤ J ( Q t − 1 , p t ) J(\mathbf{Q}_t,\mathbf{p}_t) \le J(\mathbf{Q}_{t-1},\mathbf{p}_t) J ( Q t , p t ) ≤ J ( Q t − 1 , p t ) bounds the per-round drift ψ Q t − 1 ( p t ) − ψ Q t ( p t ) \psi_{\mathbf{Q}_{t-1}}(\mathbf{p}_t) - \psi_{\mathbf{Q}_t}(\mathbf{p}_t) ψ Q t − 1 ( p t ) − ψ Q t ( p t ) by κ 2 ( log det Q t − log det Q t − 1 ) \frac{\kappa}{2}(\log\det\mathbf{Q}_t - \log\det\mathbf{Q}_{t-1}) 2 κ ( log det Q t − log det Q t − 1 ) .
Joint saddle-point update for ( p t , Q t ) (\mathbf{p}_t,\mathbf{Q}_t) ( p t , Q t ) : Finally, p t \mathbf{p}_t p t and Q t \mathbf{Q}_t Q t are coupled: FTRL needs ψ Q t \psi_{\mathbf{Q}_t} ψ Q t to select p t \mathbf{p}_t p t , whereas the barrier problem min Q ⪰ Q t − 1 J ( Q , p t ) \min_{\mathbf{Q}\succeq\mathbf{Q}_{t-1}} J(\mathbf{Q},\mathbf{p}_t) min Q ⪰ Q t − 1 J ( Q , p t ) needs p t \mathbf{p}_t p t to select Q t \mathbf{Q}_t Q t . Because the barrier term − κ 2 log det Q -\frac{\kappa}{2}\log\det\mathbf{Q} − 2 κ log det Q in J ( Q , p ) = − ψ Q ( p ) − κ 2 log det Q J(\mathbf{Q},\mathbf{p})=-\psi_{\mathbf{Q}}(\mathbf{p})-\frac{\kappa}{2}\log\det\mathbf{Q} J ( Q , p ) = − ψ Q ( p ) − 2 κ log det Q does not depend on p \mathbf{p} p , maximizing ⟨ S t − 1 , p ⟩ − ψ Q ( p ) \langle\mathbf{S}_{t-1},\mathbf{p}\rangle-\psi_{\mathbf{Q}}(\mathbf{p}) ⟨ S t − 1 , p ⟩ − ψ Q ( p ) over p \mathbf{p} p is equivalent to maximizing ⟨ S t − 1 , p ⟩ + J ( Q , p ) \langle\mathbf{S}_{t-1},\mathbf{p}\rangle+J(\mathbf{Q},\mathbf{p}) ⟨ S t − 1 , p ⟩ + J ( Q , p ) . Starting from Q 0 = I \mathbf{Q}_0=\mathbf{I} Q 0 = I , the learner therefore computes ( p t , Q t ) (\mathbf{p}_t,\mathbf{Q}_t) ( p t , Q t ) jointly via the max-min problem
p t = arg max p ∈ P { ⟨ S t − 1 , p ⟩ + min Q ⪰ Q t − 1 J ( Q , p ) } , Q t = arg min Q ⪰ Q t − 1 J ( Q , p t ) . \mathbf{p}_t=\arg\max_{\mathbf{p}\in\mathcal{P}}\left\{\langle\mathbf{S}_{t-1},\mathbf{p}\rangle+\min_{\mathbf{Q}\succeq\mathbf{Q}_{t-1}}J(\mathbf{Q},\mathbf{p})\right\},\qquad\mathbf{Q}_t=\arg\min_{\mathbf{Q}\succeq\mathbf{Q}_{t-1}}J(\mathbf{Q},\mathbf{p}_t). p t = arg p ∈ P max { ⟨ S t − 1 , p ⟩ + Q ⪰ Q t − 1 min J ( Q , p ) } , Q t = arg Q ⪰ Q t − 1 min J ( Q , p t ) .
where the inner objective J ( Q , p ) J(\mathbf{Q},\mathbf{p}) J ( Q , p ) is strictly geodesically convex in Q \mathbf{Q} Q along the matrix geometric mean path and the outer objective is strongly concave in p \mathbf{p} p (Lemmas C.1 and C.2(a)). By a first-order optimality argument (Lemmas B.1 and C.2(b)), at the minimizer Q t \mathbf{Q}_t Q t , the outer solution p t \mathbf{p}_t p t is simultaneously the exact FTRL iterate arg max p ∈ P { ⟨ S t − 1 , p ⟩ − ψ Q t ( p ) } \arg\max_{\mathbf{p}\in\mathcal{P}}\{\langle\mathbf{S}_{t-1},\mathbf{p}\rangle-\psi_{\mathbf{Q}_t}(\mathbf{p})\} arg max p ∈ P {⟨ S t − 1 , p ⟩ − ψ Q t ( p )} for the frozen regularizer ψ Q t \psi_{\mathbf{Q}_t} ψ Q t .
Normalized Kernel Reward Estimates. Having computed ( p t , Q t ) (\mathbf{p}_t,\mathbf{Q}_t) ( p t , Q t ) and the radii r t i = r i ( Q t ) r_{ti}=r_i(\mathbf{Q}_t) r t i = r i ( Q t ) , the learner draws I t ∼ p t I_t\sim\mathbf{p}_t I t ∼ p t , observes y t y_t y t , and constructs the reward estimate vector g ^ t ∈ R K \widehat{\mathbf{g}}_t\in\mathbb{R}^K g t ∈ R K by
g ^ t ( i ) : = k ~ t ( i , I t ) y t + 4 η v t i . \widehat{g}_t(i):=\widetilde{k}_t(i,I_t)y_t+4\eta v_{ti}. g t ( i ) := k t ( i , I t ) y t + 4 η v t i .
where, with the radius-weighted covariance matrix G t : = ∑ l = 1 K ( p t l / r t l ) x l x l T \mathbf{G}_t:=\sum_{l=1}^{K}(p_{tl}/r_{tl})\mathbf{x}_l\mathbf{x}_l^{\mathsf{T}} G t := ∑ l = 1 K ( p tl / r tl ) x l x l T and the reweighted kernel k t ( i , j ) : = x i T G t − 1 x j / r t j k_t(i,j):=\mathbf{x}_i^{\mathsf{T}}\mathbf{G}_t^{-1}\mathbf{x}_j/r_{tj} k t ( i , j ) := x i T G t − 1 x j / r t j from § 4.2 , the normalized kernel k ~ t ( i , j ) \widetilde{k}_t(i,j) k t ( i , j ) and variance bonus v t i v_{ti} v t i are defined by
k ~ t ( i , j ) : = k t ( i , j ) − r t i ∑ m = 1 K p t m r t m ( ∑ l = 1 K p t l k t ( l , j ) − 1 ) , v t i : = 1 r t i ∑ j = 1 K p t j k ~ t ( i , j ) 2 . (1) \widetilde{k}_t(i,j):=k_t(i,j)-\frac{r_{ti}}{\sum_{m=1}^{K}p_{tm}r_{tm}}\left(\sum_{l=1}^{K}p_{tl}k_t(l,j)-1\right),\qquad v_{ti}:=\frac{1}{r_{ti}}\sum_{j=1}^{K}p_{tj}\widetilde{k}_t(i,j)^2. \tag*{(1)} k t ( i , j ) := k t ( i , j ) − ∑ m = 1 K p t m r t m r t i ( l = 1 ∑ K p tl k t ( l , j ) − 1 ) , v t i := r t i 1 j = 1 ∑ K p t j k t ( i , j ) 2 . (1)
Let us see how k ~ t ( i , j ) \widetilde{k}_t(i,j) k t ( i , j ) and v t i v_{ti} v t i work (Lemma B.2 ):
Unbiasedness of the normalized kernel k ~ t \widetilde{k}_t k t : When c t = 0 c_t=0 c t = 0 , we saw in § 4.2 that k t ( l , I t ) y t k_t(l,I_t)y_t k t ( l , I t ) y t is an unbiased estimator of action l l l ’s clean reward a l T θ t \mathbf{a}_l^{\mathsf{T}}\boldsymbol{\theta}_t a l T θ t . Taking the p t \mathbf{p}_t p t -weighted average over l ∈ [ K ] l\in[K] l ∈ [ K ] shows that ( ∑ l = 1 K p t l k t ( l , I t ) ) y t \left(\sum_{l=1}^{K}p_{tl}k_t(l,I_t)\right)y_t ( ∑ l = 1 K p tl k t ( l , I t ) ) y t is an unbiased estimator of the learner’s expected reward ∑ l = 1 K p t l a l T θ t \sum_{l=1}^{K}p_{tl}\mathbf{a}_l^{\mathsf{T}}\boldsymbol{\theta}_t ∑ l = 1 K p tl a l T θ t . At the same time, since I t ∼ p t I_t\sim\mathbf{p}_t I t ∼ p t , the raw observation 1 ⋅ y t 1\cdot y_t 1 ⋅ y t has the exact same conditional mean ∑ j = 1 K p t j a j T θ t \sum_{j=1}^{K}p_{tj}\mathbf{a}_j^{\mathsf{T}}\boldsymbol{\theta}_t ∑ j = 1 K p t j a j T θ t . Therefore, their difference has zero conditional mean when c t = 0 c_t=0 c t = 0 :
E [ ( ∑ l = 1 K p t l k t ( l , I t ) − 1 ) y t | F t − 1 ] = ∑ l = 1 K p t l E [ k t ( l , I t ) y t ∣ F t − 1 ] − E [ y t ∣ F t − 1 ] = 0. \mathbb{E}\left[\left(\sum_{l=1}^{K}p_{tl}k_t(l,I_t)-1\right)y_t\;\middle|\;\mathcal{F}_{t-1}\right]=\sum_{l=1}^{K}p_{tl}\mathbb{E}\left[k_t(l,I_t)y_t\mid\mathcal{F}_{t-1}\right]-\mathbb{E}\left[y_t\mid\mathcal{F}_{t-1}\right]=0. E [ ( l = 1 ∑ K p tl k t ( l , I t ) − 1 ) y t F t − 1 ] = l = 1 ∑ K p tl E [ k t ( l , I t ) y t ∣ F t − 1 ] − E [ y t ∣ F t − 1 ] = 0.
Since the prefactor r t i / ( ∑ m p t m r t m ) r_{ti}/(\sum_m p_{tm}r_{tm}) r t i / ( ∑ m p t m r t m ) is F t − 1 \mathcal{F}_{t-1} F t − 1 -measurable, subtracting this zero-mean term keeps k ~ t ( i , I t ) y t \widetilde{k}_t(i,I_t)y_t k t ( i , I t ) y t unbiased for a i T θ t \mathbf{a}_i^{\mathsf{T}}\boldsymbol{\theta}_t a i T θ t :
E [ k ~ t ( i , I t ) y t ∣ F t − 1 ] = E [ k t ( i , I t ) y t ∣ F t − 1 ] = a i T θ t . \mathbb{E}\left[\widetilde{k}_t(i,I_t)y_t\mid\mathcal{F}_{t-1}\right]=\mathbb{E}\left[k_t(i,I_t)y_t\mid\mathcal{F}_{t-1}\right]=\mathbf{a}_i^{\mathsf{T}}\boldsymbol{\theta}_t. E [ k t ( i , I t ) y t ∣ F t − 1 ] = E [ k t ( i , I t ) y t ∣ F t − 1 ] = a i T θ t .
Unit column sum and O ( r t i ) O(r_{ti}) O ( r t i ) bound: Although ( ∑ l = 1 K p t l k t ( l , I t ) ) y t \left(\sum_{l=1}^{K}p_{tl}k_t(l,I_t)\right)y_t ( ∑ l = 1 K p tl k t ( l , I t ) ) y t and y t y_t y t have the same expectation when c t = 0 c_t=0 c t = 0 , they are not equal point-wise because dividing by r t j r_{tj} r t j in k t ( l , j ) k_t(l,j) k t ( l , j ) makes
∑ l = 1 K p t l k ~ t ( l , j ) ≠ 1 \sum_{l=1}^{K} p_{tl}\tilde{k}_{t}(l,j) \ne1 ∑ l = 1 K p tl k ~ t ( l , j ) = 1 in general. Subtracting the correction term in k ~ t ( i , j ) \tilde{k}_{t}(i,j) k ~ t ( i , j ) distributes the discrepancy ∑ l = 1 K p t l k t ( l , j ) − 1 \sum_{l=1}^{K}p_{tl}k_{t}(l,j)-1 ∑ l = 1 K p tl k t ( l , j ) − 1 across coordinates i i i in proportion to r t i r_{ti} r t i , so that the p t \mathbf{p}_{t} p t -weighted column sum of k ~ t \tilde{k}_{t} k ~ t is identically 1 for every j ∈ [ K ] j \in[K] j ∈ [ K ] :
∑ i = 1 K p t i k ~ t ( i , j ) = ∑ i = 1 K p t i k t ( i , j ) − ∑ i = 1 K p t i r t i ∑ m = 1 K p t m r t m ( ∑ l = 1 K p t l k t ( l , j ) − 1 ) = 1. \sum_{i=1}^{K}p_{ti}\tilde{k}_{t}(i,j)=\sum_{i=1}^{K}p_{ti}k_{t}(i,j)-\frac{\sum_{i=1}^{K}p_{ti}r_{ti}}{\sum_{m=1}^{K}p_{tm}r_{tm}}\left(\sum_{l=1}^{K}p_{tl}k_{t}(l,j)-1\right)=1. i = 1 ∑ K p t i k ~ t ( i , j ) = i = 1 ∑ K p t i k t ( i , j ) − ∑ m = 1 K p t m r t m ∑ i = 1 K p t i r t i ( l = 1 ∑ K p tl k t ( l , j ) − 1 ) = 1.
Consequently, ∑ i = 1 K p t i k ~ t ( i , I t ) y t = y t \sum_{i=1}^{K}p_{ti}\tilde{k}_{t}(i,I_{t})y_{t}=y_{t} ∑ i = 1 K p t i k ~ t ( i , I t ) y t = y t holds point-wise for every drawn action I t I_{t} I t , ensuring that any corruption c t c_{t} c t in y t y_{t} y t enters the learner’s own weighted estimate ⟨ p t , g ^ t ⟩ \langle\mathbf{p}_{t},\widehat{\mathbf{g}}_{t}\rangle ⟨ p t , g t ⟩ with weight 1. Moreover, since ∣ k t ( l , j ) ∣ ≤ r t l \lvert k_{t}(l,j)\rvert\le r_{tl} ∣ k t ( l , j )∣ ≤ r tl and r t m ≥ 1 r_{tm}\ge1 r t m ≥ 1 (so ∑ m p t m r t m ≥ 1 \sum_{m}p_{tm}r_{tm}\ge1 ∑ m p t m r t m ≥ 1 ), the correction changes k t ( i , j ) k_{t}(i,j) k t ( i , j ) by at most 2 r t i 2r_{ti} 2 r t i :
∣ k ~ t ( i , j ) ∣ ≤ ∣ k t ( i , j ) ∣ + r t i ∑ l = 1 K p t l ∣ k t ( l , j ) ∣ + 1 ∑ m = 1 K p t m r t m ≤ r t i + r t i ∑ l = 1 K p t l r t l + 1 ∑ m = 1 K p t m r t m ≤ 3 r t i . \lvert\tilde{k}_{t}(i,j)\rvert\le\lvert k_{t}(i,j)\rvert+r_{ti}\frac{\sum_{l=1}^{K}p_{tl}\lvert k_{t}(l,j)\rvert+1}{\sum_{m=1}^{K}p_{tm}r_{tm}}\le r_{ti}+r_{ti}\frac{\sum_{l=1}^{K}p_{tl}r_{tl}+1}{\sum_{m=1}^{K}p_{tm}r_{tm}}\le3r_{ti}. ∣ k ~ t ( i , j )∣ ≤ ∣ k t ( i , j )∣ + r t i ∑ m = 1 K p t m r t m ∑ l = 1 K p tl ∣ k t ( l , j )∣ + 1 ≤ r t i + r t i ∑ m = 1 K p t m r t m ∑ l = 1 K p tl r tl + 1 ≤ 3 r t i .
Variance bonus 4 η v t i 4\eta v_{ti} 4 η v t i : As in high-probability adversarial bandit algorithms such as EXP3.P (Auer et al., [3 ] ; Beygelzimer et al., [8 ] ; Lattimore and Szepesvári, [29 ] ), we augment the unbiased estimate with an optimistic variance bonus + 4 η v t i +4\eta v_{ti} + 4 η v t i . By a Freedman-type exponential-moment bound (§ 6.3 ), this bonus cancels the per-action cumulative variance 4 η ∑ t = 1 T v t i 4\eta\sum_{t=1}^{T}v_{ti} 4 η ∑ t = 1 T v t i (which can be large for rarely sampled actions), while shifting the learner’s own weighted estimate ⟨ p t , g ^ t ⟩ \langle\mathbf{p}_{t},\widehat{\mathbf{g}}_{t}\rangle ⟨ p t , g t ⟩ by only 4 η ⟨ p t , v t ⟩ ≤ 8 η 4\eta\langle\mathbf{p}_{t},\mathbf{v}_{t}\rangle\le8\eta 4 η ⟨ p t , v t ⟩ ≤ 8 η per round (Lemma B.2 (e)).
Algorithm 1 CRW-FTRL: Corruption-Robust Radius-Weighted FTRL
Require: learning rate $\eta \in (0,1/12]$, coefficients $\alpha>0$ and $\kappa=\alpha+\eta^{-1}\log(4KT)$; the whitened actions $\mathbf{x}_{i}$ and the design $\boldsymbol{\rho}$ of Definition 4.1
1: initialize $\mathbf{Q}_{0}=\mathbf{I}$ and $\mathbf{S}_{0}=0$
2: for $t=1,\ldots,T$ do
3: $\mathbf{p}_{t}=\arg\max_{\mathbf{p}\in\mathcal{P}}\{\langle\mathbf{S}_{t-1},\mathbf{p}\rangle+\min_{\mathbf{Q}\succeq\mathbf{Q}_{t-1}}J(\mathbf{Q},\mathbf{p})\}$ // FTRL step
4: $\mathbf{Q}_{t}=\arg\min_{\mathbf{Q}\succeq\mathbf{Q}_{t-1}}J(\mathbf{Q},\mathbf{p}_{t})$ and $r_{ti}=r_{i}(\mathbf{Q}_{t})$ // radius update
5: $\mathbf{G}_{t}=\sum_{l=1}^{K}(p_{tl}/r_{tl})\mathbf{x}_{l}\mathbf{x}_{l}^{\mathsf T}$, $k_{t}(i,j)=\mathbf{x}_{i}^{\mathsf T}\mathbf{G}_{t}^{-1}\mathbf{x}_{j}/r_{tj}$, and compute $(\tilde{\mathbf{k}}_{t},\mathbf{v}_{t})$ via (4.1) // kernel
6: draw $I_{t}\sim\mathbf{p}_{t}$ and observe $y_{t}$
7: $\widehat{g}_{t}(i)=\tilde{k}_{t}(i,I_{t})y_{t}+4\eta v_{ti}$ for all $i$, and $\mathbf{S}_{t}=\mathbf{S}_{t-1}+\widehat{\mathbf{g}}_{t}$ // reward estimates
8: end for
The complete procedure of CRW-FTRL is summarized in Algorithm 1. The optimizers in lines 3 and 4 exist and are unique (Lemmas C.1 and C.2), and the matrix G t = ∑ l = 1 K ( p t l / r t l ) x l x l T \mathbf{G}_{t}=\sum_{l=1}^{K}(p_{tl}/r_{tl})\mathbf{x}_{l}\mathbf{x}_{l}^{\mathsf T} G t = ∑ l = 1 K ( p tl / r tl ) x l x l T in line 5 is positive definite because p t l > 0 p_{tl}>0 p tl > 0 and r t l > 0 r_{tl}>0 r tl > 0 for every l l l and the vectors x 1 , … , x K \mathbf{x}_{1},\ldots,\mathbf{x}_{K} x 1 , … , x K span R d + 1 \mathbb{R}^{d+1} R d + 1 .
Main Results
Our main result bounds the realized regret of CRW-FTRL (Algorithm 1) with high probability.
Theorem 5.1. Suppose Assumptions 3.1 and 3.2 hold. Fix η ∈ ( 0 , 1 / 12 ] \eta\in(0,1/12] η ∈ ( 0 , 1/12 ] and δ ∈ ( 0 , 1 ) \delta\in(0,1) δ ∈ ( 0 , 1 ) , let L = log ( 8 K T / δ ) L=\log(8KT/\delta) L = log ( 8 K T / δ ) , and run Algorithm 1 with α = 3 C + L / η \alpha=3C+L/\eta α = 3 C + L / η and κ = α + log ( 4 K T ) / η \kappa=\alpha+\log(4KT)/\eta κ = α + log ( 4 K T ) / η . Then with probability at least 1 − δ 1-\delta 1 − δ ,
Reg T ≤ 3 A T C + 2 A T L η + 48 η T + C + 2 T log ( 2 / δ ) + 1. \operatorname{Reg}_{T} \le3A_{T}C+\frac{2A_{T}L}{\eta}+48\eta T+C+\sqrt{2T\log(2/\delta)}+1. Reg T ≤ 3 A T C + η 2 A T L + 48 η T + C + 2 T log ( 2/ δ ) + 1. where A T = 2 ( d + 1 ) + ( d + 1 ) log ( 1 + 4 2 ( d + 1 ) 3 / 2 T 2 ) = O ( d log ( d T ) ) A_{T}=\sqrt{2(d+1)}+(d+1)\log\left(1+4\sqrt{2}(d+1)^{3/2}T^{2}\right)=O(d\log(dT)) A T = 2 ( d + 1 ) + ( d + 1 ) log ( 1 + 4 2 ( d + 1 ) 3/2 T 2 ) = O ( d log ( d T )) .
Proof. We give a proof overview in § 6 and the complete proof in Appendix § 9 .
Choosing the learning rate η \eta η to balance 2 A T L / η 2A_{T}L/\eta 2 A T L / η against 48 η T 48\eta T 48 η T yields our main regret bound.
Corollary 5.2. Under the conditions of Theorem 5.1 , let η = min { 1 / 12 , A T L / ( 24 T ) } \eta=\min\{1/12,\sqrt{A_{T}L/(24T)}\} η = min { 1/12 , A T L / ( 24 T ) } . With probability at least 1 − δ 1-\delta 1 − δ ,
Reg T = O ( d C log ( d T ) + d T log ( d T ) log ( K T / δ ) ) = O ~ ( d T log K + d C ) . \operatorname{Reg}_{T}=O\left(dC\log(dT)+\sqrt{dT\log(dT)\log(KT/\delta)}\right)=\widetilde{O}\left(\sqrt{dT\log K}+dC\right). Reg T = O ( d C log ( d T ) + d T log ( d T ) log ( K T / δ ) ) = O ( d T log K + d C ) .
Proof. See Appendix § 9 .
Proof Overview
In this section, we present an overview of the proof of Theorem 5.1 by decomposing the regret into four terms and bounding each term in turn. Throughout this section and the appendices, Algorithm 1 is run with the parameters η ∈ ( 0 , 1 / 12 ] \eta\in(0,1/12] η ∈ ( 0 , 1/12 ] , L = log ( 8 K T / δ ) L=\log(8KT/\delta) L = log ( 8 K T / δ ) , α = 3 C + L / η \alpha=3C+L/\eta α = 3 C + L / η , and κ = α + log ( 4 K T ) / η \kappa=\alpha+\log(4KT)/\eta κ = α + log ( 4 K T ) / η specified in Theorem 5.1 . Complete proofs of Theorem 5.1 , Corollary 5.2, and all supporting technical lemmas are given in the appendices.
Regret Decomposition
Because the adversary is adaptive, the best action in hindsight arg max i ∈ [ K ] ∑ t = 1 T a i T θ t \mathop{\arg\max}_{i\in[K]}\sum_{t=1}^{T}\mathbf{a}_{i}^{\mathsf{T}}\theta_t arg max i ∈ [ K ] ∑ t = 1 T a i T θ t is random. We therefore fix an arbitrary comparator action i ⋆ ∈ [ K ] i^{\star}\in[K] i ⋆ ∈ [ K ] and establish our high-probability bound simultaneously for all i ⋆ ∈ [ K ] i^{\star}\in[K] i ⋆ ∈ [ K ] on a single event of probability at least 1 − δ 1-\delta 1 − δ that does not depend on i ⋆ i^{\star} i ⋆ .
Since the point mass e i ⋆ \mathbf{e}_{i^{\star}} e i ⋆ does not belong to the floored simplex P = { p ∈ Δ K : p i ≥ ρ i / ( 2 T ) } \mathcal{P}=\{\mathbf{p}\in\Delta_K:p_i\geq\rho_i/(2T)\} P = { p ∈ Δ K : p i ≥ ρ i / ( 2 T )} , we compare the learner against the smoothed comparator
u : = ( 1 − 1 2 T ) e i ⋆ + 1 2 T ρ ∈ P . \mathbf{u}:=\left(1-\frac{1}{2T}\right)\mathbf{e}_{i^{\star}}+\frac{1}{2T}\boldsymbol{\rho}\in\mathcal{P}. u := ( 1 − 2 T 1 ) e i ⋆ + 2 T 1 ρ ∈ P .
Adding and subtracting ∑ i = 1 K u i a i T θ t \sum_{i=1}^{K}u_i\mathbf{a}_i^{\mathsf{T}}\theta_t ∑ i = 1 K u i a i T θ t , ⟨ u , g ^ t ⟩ \langle\mathbf{u},\widehat{\mathbf{g}}_t\rangle ⟨ u , g t ⟩ , and ⟨ p t , g ^ t ⟩ \langle\mathbf{p}_t,\widehat{\mathbf{g}}_t\rangle ⟨ p t , g t ⟩ decomposes the cumulative regret against i ⋆ i^{\star} i ⋆ into four terms:
∑ t = 1 T ( a i ⋆ − a I t ) T θ t = ∑ t = 1 T ( a i ⋆ T θ t − ∑ i = 1 K u i a i T θ t ) ⏟ T 0 : smoothing error + ∑ t = 1 T ⟨ g ^ t , u − p t ⟩ ⏟ T 1 : FTRL regret + ∑ t = 1 T ( ∑ i = 1 K u i a i T θ t − ⟨ u , g ^ t ⟩ ) ⏟ T 2 : comparator error + ∑ t = 1 T ( ⟨ p t , g ^ t ⟩ − a I t T θ t ) ⏟ T 3 : learner error . (2) \begin{aligned}
\sum_{t=1}^{T}(\mathbf{a}_{i^{\star}}-\mathbf{a}_{I_t})^{\mathsf{T}}\theta_t
={}&\underbrace{\sum_{t=1}^{T}\left(\mathbf{a}_{i^{\star}}^{\mathsf{T}}\theta_t-\sum_{i=1}^{K}u_i\mathbf{a}_i^{\mathsf{T}}\theta_t\right)}_{T_0:\ \text{smoothing error}}+\underbrace{\sum_{t=1}^{T}\langle\widehat{\mathbf{g}}_t,\mathbf{u}-\mathbf{p}_t\rangle}_{T_1:\ \text{FTRL regret}}\\
&+\underbrace{\sum_{t=1}^{T}\left(\sum_{i=1}^{K}u_i\mathbf{a}_i^{\mathsf{T}}\theta_t-\langle\mathbf{u},\widehat{\mathbf{g}}_t\rangle\right)}_{T_2:\ \text{comparator error}}+\underbrace{\sum_{t=1}^{T}\left(\langle\mathbf{p}_t,\widehat{\mathbf{g}}_t\rangle-\mathbf{a}_{I_t}^{\mathsf{T}}\theta_t\right)}_{T_3:\ \text{learner error}}. \tag*{(2)}
\end{aligned} t = 1 ∑ T ( a i ⋆ − a I t ) T θ t = T 0 : smoothing error t = 1 ∑ T ( a i ⋆ T θ t − i = 1 ∑ K u i a i T θ t ) + T 1 : FTRL regret t = 1 ∑ T ⟨ g t , u − p t ⟩ + T 2 : comparator error t = 1 ∑ T ( i = 1 ∑ K u i a i T θ t − ⟨ u , g t ⟩ ) + T 3 : learner error t = 1 ∑ T ( ⟨ p t , g t ⟩ − a I t T θ t ) . (2)
Furthermore, substituting y t = ( a I t T θ t + ξ t ) + c t y_t=(\mathbf{a}_{I_t}^{\mathsf{T}}\theta_t+\xi_t)+c_t y t = ( a I t T θ t + ξ t ) + c t into the reward estimator g ^ t ( i ) = k ~ t ( i , I t ) y t + 4 η v t i \widehat{g}_t(i)=\widetilde{k}_t(i,I_t)y_t+4\eta v_{t i} g t ( i ) = k t ( i , I t ) y t + 4 η v t i decomposes g ^ t ( i ) \widehat{g}_t(i) g t ( i ) into its clean part g ^ t 0 ( i ) \widehat{g}_t^0(i) g t 0 ( i ) , its corruption shift, and the variance bonus:
g ^ t ( i ) = g ^ t 0 ( i ) + c t k ~ t ( i , I t ) + 4 η v t i , where g ^ t 0 ( i ) : = k ~ t ( i , I t ) ( a I t T θ t + ξ t ) . (3) \widehat{g}_t(i)=\widehat{g}_t^0(i)+c_t\widetilde{k}_t(i,I_t)+4\eta v_{t i},\qquad\text{where}\qquad\widehat{g}_t^0(i):=\widetilde{k}_t(i,I_t)(\mathbf{a}_{I_t}^{\mathsf{T}}\theta_t+\xi_t). \tag*{(3)} g t ( i ) = g t 0 ( i ) + c t k t ( i , I t ) + 4 η v t i , where g t 0 ( i ) := k t ( i , I t ) ( a I t T θ t + ξ t ) . (3)
The smoothing error T 0 T_0 T 0 is bounded deterministically by 1 1 1 : since ∣ a i T θ t ∣ ≤ 1 |\mathbf{a}_i^{\mathsf{T}}\theta_t|\leq1 ∣ a i T θ t ∣ ≤ 1 (Assumption 3.1) and ∑ i = 1 K ρ i = 1 \sum_{i=1}^{K}\rho_i=1 ∑ i = 1 K ρ i = 1 , each round contributes
a i ⋆ T θ t − ∑ i = 1 K u i a i T θ t = 1 2 T ( a i ⋆ T θ t − ∑ i = 1 K ρ i a i T θ t ) ≤ 1 2 T ( 1 + 1 ) = 1 T , so T 0 ≤ 1. (4) \mathbf{a}_{i^{\star}}^{\mathsf{T}}\theta_t-\sum_{i=1}^{K}u_i\mathbf{a}_i^{\mathsf{T}}\theta_t=\frac{1}{2T}\left(\mathbf{a}_{i^{\star}}^{\mathsf{T}}\theta_t-\sum_{i=1}^{K}\rho_i\mathbf{a}_i^{\mathsf{T}}\theta_t\right)\leq\frac{1}{2T}(1+1)=\frac{1}{T},\qquad\text{so}\qquad T_0\leq1. \tag*{(4)} a i ⋆ T θ t − i = 1 ∑ K u i a i T θ t = 2 T 1 ( a i ⋆ T θ t − i = 1 ∑ K ρ i a i T θ t ) ≤ 2 T 1 ( 1 + 1 ) = T 1 , so T 0 ≤ 1. (4)
We now bound the FTRL regret T 1 T_1 T 1 (§ 6.2 ) and the estimation errors T 2 T_2 T 2 and T 3 T_3 T 3 (§ 6.3 ), and then combine the four bounds in § 6.4 .
Bounding the FTRL Regret
To bound the FTRL term T 1 = ∑ t = 1 T ⟨ g ^ t , u − p t ⟩ T_1 = \sum_{t=1}^{T} \langle\hat{\mathbf{g}}_t, \mathbf{u} - \mathbf{p}_t \rangle T 1 = ∑ t = 1 T ⟨ g ^ t , u − p t ⟩ , we first express the update in line 3 of Algorithm 1 as a standard FTRL step with a time-varying regularizer. Once Q t \mathbf{Q}_t Q t is fixed at its inner minimizer Q t = a r g m i n Q ⪰ Q t − 1 J ( Q , p t ) \mathbf{Q}_t = \mathop{\mathrm{argmin}}_{\mathbf{Q} \succeq\mathbf{Q}_{t-1}} J(\mathbf{Q}, \mathbf{p}_t) Q t = argmin Q ⪰ Q t − 1 J ( Q , p t ) , the outer iterate p t \mathbf{p}_t p t maximizes the round-t t t objective Λ t ( p ) : = ⟨ S t − 1 , p ⟩ − ψ t ( p ) \Lambda_t(\mathbf{p}) := \langle\mathbf{S}_{t-1}, \mathbf{p} \rangle- \psi_t(\mathbf{p}) Λ t ( p ) := ⟨ S t − 1 , p ⟩ − ψ t ( p ) over P \mathcal{P} P , where
ψ t ( p ) : = − ∑ i = 1 K r t i h ( p i ) = ∑ i = 1 K r t i ( η − 1 p i log p i − α p i ) . (5) \psi_t(\mathbf{p}) := -\sum_{i=1}^{K} r_{ti} h(p_i) = \sum_{i=1}^{K} r_{ti}(\eta^{-1}p_i \log p_i - \alpha p_i). \tag*{(5)} ψ t ( p ) := − i = 1 ∑ K r t i h ( p i ) = i = 1 ∑ K r t i ( η − 1 p i log p i − α p i ) . (5)
is the radius-weighted regularizer at round t t t (Lemma B.1 in Appendix § 10 ). Setting ψ T + 1 : = ψ T \psi_{T+1} := \psi_T ψ T + 1 := ψ T and letting p T + 1 ∈ a r g m a x p ∈ P Λ T + 1 ( p ) \mathbf{p}_{T+1} \in\mathop{\mathrm{argmax}}_{\mathbf{p} \in\mathcal{P}} \Lambda_{T+1}(\mathbf{p}) p T + 1 ∈ argmax p ∈ P Λ T + 1 ( p ) for the analysis, a standard telescoping FTRL inequality for time-varying regularizers [29 Exercise 28.12] (stated as Lemma B.4 in Appendix § 10 ) decomposes T 1 T_1 T 1 into three parts:
T 1 = ∑ t = 1 T ⟨ g ^ t , u − p t ⟩ ≤ ψ T ( u ) ⏟ comparator penalty + ( − ψ 1 ( p 1 ) + ∑ t = 1 T − 1 [ ψ t ( p t + 1 ) − ψ t + 1 ( p t + 1 ) ] ) initial and regularizer drift terms + ∑ t = 1 T [ Λ t ( p t + 1 ) − Λ t ( p t ) + ⟨ g ^ t , p t + 1 − p t ⟩ ] ⏟ stability terms . (6) \begin{aligned}
T_1 = \sum_{t=1}^{T} \langle\hat{\mathbf{g}}_t, \mathbf{u} - \mathbf{p}_t \rangle
&\le\underbrace{\psi_T(\mathbf{u})}_{\text{comparator penalty}} + \left( -\psi_1(\mathbf{p}_1) + \sum_{t=1}^{T-1} \left[\psi_t(\mathbf{p}_{t+1}) - \psi_{t+1}(\mathbf{p}_{t+1})\right] \right)_{\text{initial and regularizer drift terms}} \\
&\quad+ \underbrace{\sum_{t=1}^{T} \left[\Lambda_t(\mathbf{p}_{t+1}) - \Lambda_t(\mathbf{p}_t) + \langle\hat{\mathbf{g}}_t, \mathbf{p}_{t+1} - \mathbf{p}_t \rangle\right]}_{\text{stability terms}}. \tag*{(6)}
\end{aligned} T 1 = t = 1 ∑ T ⟨ g ^ t , u − p t ⟩ ≤ comparator penalty ψ T ( u ) + ( − ψ 1 ( p 1 ) + t = 1 ∑ T − 1 [ ψ t ( p t + 1 ) − ψ t + 1 ( p t + 1 ) ] ) initial and regularizer drift terms + stability terms t = 1 ∑ T [ Λ t ( p t + 1 ) − Λ t ( p t ) + ⟨ g ^ t , p t + 1 − p t ⟩ ] . (6)
We bound the three parts on the right-hand side of (6) in turn (full proofs of the supporting lemmas are given in Appendix § 10 ).
Comparator penalty. Evaluating the final regularizer ψ T \psi_T ψ T at the comparator u ∈ P \mathbf{u} \in\mathcal{P} u ∈ P gives
ψ T ( u ) = ∑ i = 1 K r T i ( η − 1 u i log u i − α u i ) ≤ − α ∑ i = 1 K u i r T i = − α ⟨ u , r T ⟩ . \psi_T(\mathbf{u}) = \sum_{i=1}^{K} r_{Ti}(\eta^{-1}u_i \log u_i - \alpha u_i) \le-\alpha\sum_{i=1}^{K} u_i r_{Ti} = -\alpha\langle\mathbf{u}, \mathbf{r}_T \rangle. ψ T ( u ) = i = 1 ∑ K r T i ( η − 1 u i log u i − α u i ) ≤ − α i = 1 ∑ K u i r T i = − α ⟨ u , r T ⟩ .
where the inequality uses u i log u i ≤ 0 u_i \log u_i \le0 u i log u i ≤ 0 for u i ∈ ( 0 , 1 ] u_i \in(0, 1] u i ∈ ( 0 , 1 ] and r T i > 0 r_{Ti} > 0 r T i > 0 . This negative linear penalty in the final radii r T \mathbf{r}_T r T is what will later cancel the comparator’s estimation and corruption errors in T 2 T_2 T 2 .
Initial and regularizer drift terms. Because J ( Q , p ) = − ψ Q ( p ) − κ 2 log det Q J(\mathbf{Q}, \mathbf{p}) = -\psi_{\mathbf{Q}}(\mathbf{p}) - \frac{\kappa}{2} \log\det\mathbf{Q} J ( Q , p ) = − ψ Q ( p ) − 2 κ log det Q , the round-t t t regularizer satisfies − ψ t ( p ) = J ( Q t , p ) + κ 2 log det Q t -\psi_t(\mathbf{p}) = J(\mathbf{Q}_t, \mathbf{p}) + \frac{\kappa}{2} \log\det\mathbf{Q}_t − ψ t ( p ) = J ( Q t , p ) + 2 κ log det Q t . Since Q t + 1 \mathbf{Q}_{t+1} Q t + 1 minimizes J ( Q , p t + 1 ) J(\mathbf{Q}, \mathbf{p}_{t+1}) J ( Q , p t + 1 ) over Q ⪰ Q t \mathbf{Q} \succeq\mathbf{Q}_t Q ⪰ Q t and Q t \mathbf{Q}_t Q t is feasible, we have J ( Q t + 1 , p t + 1 ) ≤ J ( Q t , p t + 1 ) J(\mathbf{Q}_{t+1}, \mathbf{p}_{t+1}) \le J(\mathbf{Q}_t, \mathbf{p}_{t+1}) J ( Q t + 1 , p t + 1 ) ≤ J ( Q t , p t + 1 ) , which immediately bounds each round’s regularizer drift by the log-determinant increment:
ψ t ( p t + 1 ) − ψ t + 1 ( p t + 1 ) = J ( Q t + 1 , p t + 1 ) − J ( Q t , p t + 1 ) + κ 2 log det Q t + 1 det Q t ≤ κ 2 log det Q t + 1 det Q t . \psi_t(\mathbf{p}_{t+1}) - \psi_{t+1}(\mathbf{p}_{t+1}) = J(\mathbf{Q}_{t+1}, \mathbf{p}_{t+1}) - J(\mathbf{Q}_t, \mathbf{p}_{t+1}) + \frac{\kappa}{2} \log\frac{\det\mathbf{Q}_{t+1}}{\det\mathbf{Q}_t} \le\frac{\kappa}{2} \log\frac{\det\mathbf{Q}_{t+1}}{\det\mathbf{Q}_t}. ψ t ( p t + 1 ) − ψ t + 1 ( p t + 1 ) = J ( Q t + 1 , p t + 1 ) − J ( Q t , p t + 1 ) + 2 κ log det Q t det Q t + 1 ≤ 2 κ log det Q t det Q t + 1 .
Likewise, at t = 1 t = 1 t = 1 , J ( Q 1 , p 1 ) ≤ J ( Q 0 , p 1 ) = ∑ i = 1 K h ( p 1 i ) ∥ x i ∥ J(\mathbf{Q}_1, \mathbf{p}_1) \le J(\mathbf{Q}_0, \mathbf{p}_1) = \sum_{i=1}^{K} h(p_{1i}) \lVert\mathbf{x}_i \rVert J ( Q 1 , p 1 ) ≤ J ( Q 0 , p 1 ) = ∑ i = 1 K h ( p 1 i ) ∥ x i ∥ (since Q 0 = I \mathbf{Q}_0 = \mathbf{I} Q 0 = I ). Because every p ∈ P \mathbf{p} \in\mathcal{P} p ∈ P satisfies p i ≥ ρ i / ( 2 T ) ≥ 1 / ( 4 K T ) p_i \ge\rho_i/(2T) \ge1/(4KT) p i ≥ ρ i / ( 2 T ) ≥ 1/ ( 4 K T ) (Lemma B.6(a)), the weight h ( p i ) = p i ( α + η − 1 log ( 1 / p i ) ) h(p_i) = p_i(\alpha+ \eta^{-1} \log(1/p_i)) h ( p i ) = p i ( α + η − 1 log ( 1/ p i )) is sandwiched between linear multiples of p i p_i p i (Lemma B.5):
α p i ≤ h ( p i ) ≤ κ p i , where κ = α + η − 1 log ( 4 K T ) ≤ 2 α . \alpha p_i \le h(p_i) \le\kappa p_i, \qquad\text{where } \kappa= \alpha+ \eta^{-1}\log(4KT) \le2\alpha. α p i ≤ h ( p i ) ≤ κ p i , where κ = α + η − 1 log ( 4 K T ) ≤ 2 α .
Combined with ∥ x i ∥ ≤ 2 ( d + 1 ) \lVert\mathbf{x}_i\rVert\le\sqrt{2(d+1)} ∥ x i ∥ ≤ 2 ( d + 1 ) (Lemma B.6(c)), this bounds the initial term by − ψ 1 ( p 1 ) = J ( Q 1 , p 1 ) + κ 2 log det Q 1 ≤ κ 2 ( d + 1 ) + κ 2 log det Q 1 -\psi_1(\mathbf{p}_1)=J(\mathbf{Q}_1,\mathbf{p}_1)+\frac{\kappa}{2}\log\det\mathbf{Q}_1\le\kappa\sqrt{2(d+1)}+\frac{\kappa}{2}\log\det\mathbf{Q}_1 − ψ 1 ( p 1 ) = J ( Q 1 , p 1 ) + 2 κ log det Q 1 ≤ κ 2 ( d + 1 ) + 2 κ log det Q 1 . Telescoping the log-determinant ratios over t = 1 , … , T − 1 t=1,\ldots,T-1 t = 1 , … , T − 1 and using the eigenvalue growth bound log det Q T ≤ 2 ( d + 1 ) log ( 1 + 4 2 ( d + 1 ) 3 / 2 T 2 ) \log\det\mathbf{Q}_T\le2(d+1)\log(1+4\sqrt{2}(d+1)^{3/2}T^2) log det Q T ≤ 2 ( d + 1 ) log ( 1 + 4 2 ( d + 1 ) 3/2 T 2 ) from the first-order optimality conditions of Q t \mathbf{Q}_t Q t (Lemma B.7(iii)) shows that the initial and drift terms sum to at most
− ψ 1 ( p 1 ) + ∑ t = 1 T − 1 [ ψ t ( p t + 1 ) − ψ t + 1 ( p t + 1 ) ] ≤ κ ( 2 ( d + 1 ) + 1 2 log det Q T ) ≤ κ A T . -\psi_1(\mathbf{p}_1)+\sum_{t=1}^{T-1}\left[\psi_t(\mathbf{p}_{t+1})-\psi_{t+1}(\mathbf{p}_{t+1})\right]\le\kappa\left(\sqrt{2(d+1)}+\frac{1}{2}\log\det\mathbf{Q}_T\right)\le\kappa A_T. − ψ 1 ( p 1 ) + t = 1 ∑ T − 1 [ ψ t ( p t + 1 ) − ψ t + 1 ( p t + 1 ) ] ≤ κ ( 2 ( d + 1 ) + 2 1 log det Q T ) ≤ κ A T .
Stability terms. With the radii frozen at r t \mathbf{r}_t r t in round t t t , ψ t \psi_t ψ t is a radius-weighted negative entropy. By local strong convexity of weighted entropy together with the magnitude and second-moment bounds on the reward estimate g ^ t \widehat{\mathbf{g}}_t g t , each round’s stability term is bounded as follows.
Lemma 6.1 (Per-Round Stability). For every t ∈ [ T ] t\in[T] t ∈ [ T ] and every q ∈ P \mathbf{q}\in\mathcal{P} q ∈ P ,
Λ t ( q ) − Λ t ( p t ) + ⟨ g ^ t , q − p t ⟩ ≤ η ∑ i = 1 K p t i g ^ t ( i ) 2 r t i ≤ 40 η . \Lambda_t(\mathbf{q})-\Lambda_t(\mathbf{p}_t)+\langle\widehat{\mathbf{g}}_t,\mathbf{q}-\mathbf{p}_t\rangle\le\eta\sum_{i=1}^{K}p_{ti}\frac{\widehat{g}_t(i)^2}{r_{ti}}\le40\eta. Λ t ( q ) − Λ t ( p t ) + ⟨ g t , q − p t ⟩ ≤ η i = 1 ∑ K p t i r t i g t ( i ) 2 ≤ 40 η .
Proof. See Appendix § 10 .
Applying Lemma 6.1 with q = p t + 1 \mathbf{q}=\mathbf{p}_{t+1} q = p t + 1 and summing over t = 1 , … , T t=1,\ldots,T t = 1 , … , T bounds the stability terms in (6) by 40 η T 40\eta T 40 η T .
Combining the bounds on the comparator penalty, the regularizer drift, and the stability terms in (6) yields the following deterministic bound on T 1 T_1 T 1 .
Lemma 6.2 (FTRL with Growing Radii). On every sample path and for every u ∈ P \mathbf{u}\in\mathcal{P} u ∈ P ,
T 1 = ∑ t = 1 T ⟨ g ^ t , u − p t ⟩ ≤ − α ⟨ u , r T ⟩ + κ A T + 40 η T . T_1=\sum_{t=1}^{T}\langle\widehat{\mathbf{g}}_t,\mathbf{u}-\mathbf{p}_t\rangle\le-\alpha\langle\mathbf{u},\mathbf{r}_T\rangle+\kappa A_T+40\eta T. T 1 = t = 1 ∑ T ⟨ g t , u − p t ⟩ ≤ − α ⟨ u , r T ⟩ + κ A T + 40 η T .
Proof. See Appendix § 10 .
Bounding the Comparator and Learner Errors
We next bound the comparator error T 2 = ∑ t = 1 T ( ∑ i = 1 K u i a i ⊤ θ t − ⟨ u , g ^ t ⟩ ) T_2=\sum_{t=1}^{T}\left(\sum_{i=1}^{K}u_i a_{i}^{\top}\boldsymbol{\theta}_t-\langle\mathbf{u},\widehat{\mathbf{g}}_t\rangle\right) T 2 = ∑ t = 1 T ( ∑ i = 1 K u i a i ⊤ θ t − ⟨ u , g t ⟩ ) and the learner error T 3 = ∑ t = 1 T ( ⟨ p t , g ^ t ⟩ − a t ⊤ θ t ) T_3=\sum_{t=1}^{T}(\langle\mathbf{p}_t,\widehat{\mathbf{g}}_t\rangle-\mathbf{a}_t^{\top}\boldsymbol{\theta}_t) T 3 = ∑ t = 1 T (⟨ p t , g t ⟩ − a t ⊤ θ t ) using the estimator decomposition (3) and the algebraic and moment properties of the normalized kernel k ~ t \widetilde{k}_t k t and variance bonus v t \mathbf{v}_t v t derived in § 4.2 and § 4.3 (summarized formally in Lemma B.2 in Appendix § 10 ).
Comparator error T 2 T_2 T 2 . Substituting the estimator split g ^ t ( i ) = g ^ t 0 ( i ) + c t k ~ t ( i , I t ) + 4 η v t i \widehat{g}_t(i)=\widehat{g}_t^0(i)+c_t\widetilde{k}_t(i,I_t)+4\eta v_{ti} g t ( i ) = g t 0 ( i ) + c t k t ( i , I t ) + 4 η v t i from (3) into T 2 T_2 T 2 separates the comparator error into a clean estimation error and a corruption shift:
T 2 = ∑ i = 1 K u i ∑ t = 1 T [ a i ⊤ θ t − g ^ t 0 ( i ) − 4 η v t i ] − ∑ t = 1 T c t ∑ i = 1 K u i k ~ t ( i , I t ) . (7) T_2=\sum_{i=1}^{K}u_i\sum_{t=1}^{T}\left[a_i^{\top}\boldsymbol{\theta}_t-\widehat{g}_t^0(i)-4\eta v_{ti}\right]-\sum_{t=1}^{T}c_t\sum_{i=1}^{K}u_i\widetilde{k}_t(i,I_t). \tag*{(7)} T 2 = i = 1 ∑ K u i t = 1 ∑ T [ a i ⊤ θ t − g t 0 ( i ) − 4 η v t i ] − t = 1 ∑ T c t i = 1 ∑ K u i k t ( i , I t ) . (7)
For the clean estimation error, because k ~ t \tilde{k}_t k ~ t reproduces the whitened actions ( ∑ j = 1 K p t j k ~ t ( i , j ) x j = x i (\sum_{j=1}^{K} p_{tj}\tilde{k}_t(i,j)x_j = x_i ( ∑ j = 1 K p t j k ~ t ( i , j ) x j = x i , Lemma B.2 (a)) and the clean rewards are linear in x j x_j x j , the clean part g ^ t 0 ( i ) = k ~ t ( i , I t ) ( a I t T θ t + ξ t ) \widehat{g}_t^0(i) = \tilde{k}_t(i,I_t)(\mathbf{a}_{I_t}^{\mathsf{T}}\theta_t + \xi_t) g t 0 ( i ) = k ~ t ( i , I t ) ( a I t T θ t + ξ t ) satisfies E [ g ^ t 0 ( i ) ∣ F t − 1 ] = a i T θ t \mathbb{E}[\widehat{g}_t^0(i) \mid\mathcal{F}_{t-1}] = \mathbf{a}_i^{\mathsf{T}}\theta_t E [ g t 0 ( i ) ∣ F t − 1 ] = a i T θ t . Dividing the zero-mean error a i T θ t − g ^ t 0 ( i ) \mathbf{a}_i^{\mathsf{T}}\theta_t - \widehat{g}_t^0(i) a i T θ t − g t 0 ( i ) by the predictable radius r t i r_{ti} r t i produces a martingale difference X t X_t X t with ∣ X t ∣ ≤ 7 \lvert X_t\rvert\le7 ∣ X t ∣ ≤ 7 and conditional variance at most σ t 2 : = 4 v t i / r t i \sigma_t^2 := 4v_{ti}/r_{ti} σ t 2 := 4 v t i / r t i . Applying a Freedman-style exponential supermartingale bound uniformly over all suffix sums ∑ t = τ T ( X t − η σ t 2 ) \sum_{t=\tau}^{T}(X_t-\eta\sigma_t^2) ∑ t = τ T ( X t − η σ t 2 ) and summing by parts against the nondecreasing sequence r 1 i ≤ ⋯ ≤ r T i r_{1i} \le\cdots\le r_{Ti} r 1 i ≤ ⋯ ≤ r T i yields the following high-probability concentration bound.
Lemma 6.3 (Suffix Concentration). Let L = log ( 8 K T / δ ) L = \log(8KT/\delta) L = log ( 8 K T / δ ) . With probability at least 1 − δ / 2 1-\delta/2 1 − δ /2 , simultaneously for all i ∈ [ K ] i \in[K] i ∈ [ K ] ,
∑ t = 1 T [ a i T θ t − g ^ t 0 ( i ) − 4 η v t i ] ≤ L η r T i . \sum_{t=1}^{T}\left[\mathbf{a}_i^{\mathsf{T}}\theta_t-\widehat{g}_t^0(i)-4\eta v_{ti}\right]\le\frac{L}{\eta}r_{Ti}. t = 1 ∑ T [ a i T θ t − g t 0 ( i ) − 4 η v t i ] ≤ η L r T i .
Proof. See Appendix § 10 .
For the corruption shift in (7) , the optimality of Q t \mathbf{Q}_t Q t ensures x i T G t − 1 x i ≤ r t i 2 \mathbf{x}_i^{\mathsf{T}}\mathbf{G}_t^{-1}\mathbf{x}_i \le r_{ti}^2 x i T G t − 1 x i ≤ r t i 2 (Lemma B.7(i)), which yields the magnitude bound ∣ k ~ t ( i , I t ) ∣ ≤ 3 r t i \lvert\tilde{k}_t(i,I_t)\rvert\le3r_{ti} ∣ k ~ t ( i , I t )∣ ≤ 3 r t i (Lemma B.2 (c)). Combining Lemma 6.3 with ∣ k ~ t ( i , I t ) ∣ ≤ 3 r t i \lvert\tilde{k}_t(i,I_t)\rvert\le3r_{ti} ∣ k ~ t ( i , I t )∣ ≤ 3 r t i and the monotonicity of the radii r t i ≤ r T i r_{ti}\le r_{Ti} r t i ≤ r T i (Lemma B.7(ii)) bounds T 2 T_2 T 2 on the ( 1 − δ / 2 ) (1-\delta/2) ( 1 − δ /2 ) -probability event of Lemma 6.3:
T 2 ≤ L η ⟨ u , r T ⟩ + ∑ t = 1 T ∣ c t ∣ ∑ i = 1 K u i ∣ k ~ t ( i , I t ) ∣ ≤ L η ⟨ u , r T ⟩ + 3 ∑ t = 1 T ∣ c t ∣ ⟨ u , r t ⟩ ≤ L η ⟨ u , r T ⟩ + 3 ⟨ u , r T ⟩ ∑ t = 1 T ∣ c t ∣ ≤ ( L η + 3 C ) ⟨ u , r T ⟩ , (8) \begin{aligned}
T_2 &\le\frac{L}{\eta}\langle\mathbf{u},\mathbf{r}_T\rangle+\sum_{t=1}^{T}|c_t|\sum_{i=1}^{K}u_i\lvert\tilde{k}_t(i,I_t)\rvert\\
&\le\frac{L}{\eta}\langle\mathbf{u},\mathbf{r}_T\rangle+3\sum_{t=1}^{T}|c_t|\langle\mathbf{u},\mathbf{r}_t\rangle\\
&\le\frac{L}{\eta}\langle\mathbf{u},\mathbf{r}_T\rangle+3\langle\mathbf{u},\mathbf{r}_T\rangle\sum_{t=1}^{T}|c_t| \\
&\le\left(\frac{L}{\eta}+3C\right)\langle\mathbf{u},\mathbf{r}_T\rangle, \tag*{(8)}
\end{aligned} T 2 ≤ η L ⟨ u , r T ⟩ + t = 1 ∑ T ∣ c t ∣ i = 1 ∑ K u i ∣ k ~ t ( i , I t )∣ ≤ η L ⟨ u , r T ⟩ + 3 t = 1 ∑ T ∣ c t ∣ ⟨ u , r t ⟩ ≤ η L ⟨ u , r T ⟩ + 3 ⟨ u , r T ⟩ t = 1 ∑ T ∣ c t ∣ ≤ ( η L + 3 C ) ⟨ u , r T ⟩ , (8)
where (a) applies Lemma 6.3 and the triangle inequality, (b) uses ∣ k ~ t ( i , I t ) ∣ ≤ 3 r t i \lvert\tilde{k}_t(i,I_t)\rvert\le3r_{ti} ∣ k ~ t ( i , I t )∣ ≤ 3 r t i , (c) uses r t i ≤ r T i r_{ti}\le r_{Ti} r t i ≤ r T i and u i ≥ 0 u_i\ge0 u i ≥ 0 for all i ∈ [ K ] i\in[K] i ∈ [ K ] (so ⟨ u , r t ⟩ = ∑ i = 1 K u i r t i ≤ ∑ i = 1 K u i r T i = ⟨ u , r T ⟩ \langle\mathbf{u},\mathbf{r}_t\rangle=\sum_{i=1}^{K}u_ir_{ti}\le\sum_{i=1}^{K}u_ir_{Ti}=\langle\mathbf{u},\mathbf{r}_T\rangle ⟨ u , r t ⟩ = ∑ i = 1 K u i r t i ≤ ∑ i = 1 K u i r T i = ⟨ u , r T ⟩ ), and (d) uses ∑ t = 1 T ∣ c t ∣ ≤ C \sum_{t=1}^{T}|c_t|\le C ∑ t = 1 T ∣ c t ∣ ≤ C (Assumption 3.2(c)).
Learner error T 3 T_3 T 3 . On the learner side, the rank-one normalization in (1) guarantees the unit column-sum property ∑ i = 1 K p t i k ~ t ( i , I t ) = 1 \sum_{i=1}^{K}p_{ti}\tilde{k}_t(i,I_t)=1 ∑ i = 1 K p t i k ~ t ( i , I t ) = 1 (Lemma B.2 (b)). Taking the inner product of p t \mathbf{p}_t p t with g ^ t ( i ) = k ~ t ( i , I t ) y t + 4 η v t \widehat{\mathbf{g}}_t(i)=\tilde{k}_t(i,I_t)y_t+4\eta\mathbf{v}_t g t ( i ) = k ~ t ( i , I t ) y t + 4 η v t therefore collapses the inverse-propensity estimator back to the observed reward y t = a I t T θ t + ξ t + c t y_t=\mathbf{a}_{I_t}^{\mathsf{T}}\theta_t+\xi_t+c_t y t = a I t T θ t + ξ t + c t :
⟨ p t , g ^ t ⟩ = ( ∑ i = 1 K p t i k ~ t ( i , I t ) ) ( a I t T θ t + ξ t + c t ) + 4 η ⟨ p t , v t ⟩ = a I t T θ t + ξ t + c t + 4 η ⟨ p t , v t ⟩ . \langle\mathbf{p}_t,\widehat{\mathbf{g}}_t\rangle
=\left(\sum_{i=1}^{K}p_{ti}\tilde{k}_t(i,I_t)\right)\left(\mathbf{a}_{I_t}^{\mathsf{T}}\theta_t+\xi_t+c_t\right)+4\eta\langle\mathbf{p}_t,\mathbf{v}_t\rangle
=\mathbf{a}_{I_t}^{\mathsf{T}}\theta_t+\xi_t+c_t+4\eta\langle\mathbf{p}_t,\mathbf{v}_t\rangle. ⟨ p t , g t ⟩ = ( i = 1 ∑ K p t i k ~ t ( i , I t ) ) ( a I t T θ t + ξ t + c t ) + 4 η ⟨ p t , v t ⟩ = a I t T θ t + ξ t + c t + 4 η ⟨ p t , v t ⟩ .
The clean reward a I t T θ t \mathbf{a}_{I_t}^{\mathsf{T}}\theta_t a I t T θ t cancels inside T 3 = ∑ t = 1 T ( ⟨ p t , g ^ t ⟩ − a I t T θ t ) T_3=\sum_{t=1}^{T}(\langle\mathbf{p}_t,\widehat{\mathbf{g}}_t\rangle-\mathbf{a}_{I_t}^{\mathsf{T}}\theta_t) T 3 = ∑ t = 1 T (⟨ p t , g t ⟩ − a I t T θ t ) , leaving
T 3 = ∑ t = 1 T ( ξ t + c t + 4 η ⟨ p t , v t ⟩ ) ≤ ( a ) ∑ t = 1 T ξ t + C + 8 η T ≤ ( b ) 2 T log ( 2 / δ ) + C + 8 η T . (9) T_3=\sum_{t=1}^{T}(\xi_t+c_t+4\eta\langle\mathbf{p}_t,\mathbf{v}_t\rangle)
\mathrel{\overset{(a)}{\le}}\sum_{t=1}^{T}\xi_t+C+8\eta T
\mathrel{\overset{(b)}{\le}}\sqrt{2T\log(2/\delta)}+C+8\eta T. \tag*{(9)} T 3 = t = 1 ∑ T ( ξ t + c t + 4 η ⟨ p t , v t ⟩) ≤ ( a ) t = 1 ∑ T ξ t + C + 8 η T ≤ ( b ) 2 T log ( 2/ δ ) + C + 8 η T . (9)
where (a) uses ∑ t = 1 T c t ≤ C \sum_{t=1}^{T}c_t\le C ∑ t = 1 T c t ≤ C and ⟨ p t , v t ⟩ ≤ 2 \langle\mathbf{p}_t,\mathbf{v}_t\rangle\le2 ⟨ p t , v t ⟩ ≤ 2 (Lemma B.2 (e)), and (b) holds with probability at least $1-\delta/2$ by the Azuma–Hoeffding inequality since ( ξ t ) t = 1 T (\xi_t)_{t=1}^{T} ( ξ t ) t = 1 T is a martingale difference sequence bounded in $[-1,1] (Lemma B.8 in Appendix § 10 ).
Combining (8) and (9) by a union bound over the two ( 1 − δ / 2 ) (1-\delta/2) ( 1 − δ /2 ) -probability events yields the following high-probability guarantee for T 2 T_2 T 2 and T 3 T_3 T 3 .
Lemma 6.4 (Comparator and Learner Errors). Let L = log ( 8 K T / δ ) L = \log(8KT/\delta) L = log ( 8 K T / δ ) . With probability at least 1 − δ 1-\delta 1 − δ , simultaneously for every comparator u ∈ P \mathbf{u} \in\mathcal{P} u ∈ P ,
T 2 ≤ ( L η + 3 C ) ⟨ u , r T ⟩ T_2 \le\left(\frac{L}{\eta}+3C\right)\langle\mathbf{u}, \mathbf{r}_T\rangle T 2 ≤ ( η L + 3 C ) ⟨ u , r T ⟩ T 3 ≤ 2 T log ( 2 / δ ) + C + 8 η T . T_3 \le\sqrt{2T\log(2/\delta)}+C+8\eta T. T 3 ≤ 2 T log ( 2/ δ ) + C + 8 η T .
Proof. See Appendix § 10 .
Combining the Four Bounds
We now combine the bounds on T 0 T_0 T 0 , T 1 T_1 T 1 , T 2 T_2 T 2 , T 3 T_3 T 3 to obtain Theorem 5.1 (formalized in Appendix § 9 ). On the ( 1 − δ ) (1-\delta) ( 1 − δ ) -probability event of Lemma 6.4 (which does not depend on the choice of i ⋆ ∈ [ K ] i^\star\in[K] i ⋆ ∈ [ K ] ), summing T 0 ≤ 1 T_0 \le1 T 0 ≤ 1 from (4) , Lemma 6.2 for T 1 T_1 T 1 , and Lemma 6.4 for T 2 T_2 T 2 and T 3 T_3 T 3 gives
∑ t = 1 T ( a i ⋆ − a I t ) ⊤ θ t ≤ 1 + ( − α + L η + 3 C ) ⟨ u , r T ⟩ + κ A T + 48 η T + C + 2 T log ( 2 / δ ) \sum_{t=1}^{T}(\mathbf{a}_{i^\star}-\mathbf{a}_{I_t})^\top\boldsymbol{\theta}_t \le1+\left(-\alpha+\frac{L}{\eta}+3C\right)\langle\mathbf{u},\mathbf{r}_T\rangle+\kappa A_T+48\eta T+C+\sqrt{2T\log(2/\delta)} t = 1 ∑ T ( a i ⋆ − a I t ) ⊤ θ t ≤ 1 + ( − α + η L + 3 C ) ⟨ u , r T ⟩ + κ A T + 48 η T + C + 2 T log ( 2/ δ )
≤ 3 A T C + 2 A T L η + 48 η T + C + 2 T log ( 2 / δ ) + 1. \le3A_TC+\frac{2A_TL}{\eta}+48\eta T+C+\sqrt{2T\log(2/\delta)}+1. ≤ 3 A T C + η 2 A T L + 48 η T + C + 2 T log ( 2/ δ ) + 1.
where the choice α = 3 C + L / η \alpha= 3C+L/\eta α = 3 C + L / η cancels the comparator-dependent term ⟨ u , r T ⟩ \langle\mathbf{u},\mathbf{r}_T\rangle ⟨ u , r T ⟩ identically and κ = α + log ( 4 K T ) / η ≤ 3 C + 2 L / η \kappa=\alpha+\log(4KT)/\eta\le3C+2L/\eta κ = α + log ( 4 K T ) / η ≤ 3 C + 2 L / η . Taking the maximum over i ⋆ ∈ [ K ] i^\star\in[K] i ⋆ ∈ [ K ] yields Theorem 5.1 .
Conclusion and Future Work
We studied adversarial linear bandits with strong corruption and proposed CRW-FTRL, an FTRL algorithm with a radius-weighted entropy regularizer and normalized kernel reward estimates. Its key ingredient is a set of nondecreasing per-action radii, chosen jointly with the sampling distribution by a monotone log-determinant barrier problem, which bound the influence of a corrupted observation on each reward estimate regardless of which action was drawn. With these radii, a negative linear term in the regularizer cancels the corruption bias and the estimation error of every comparator. As a result, CRW-FTRL achieves O ~ ( d T log K + d C ) \widetilde{O}(\sqrt{dT\log K}+dC) O ( d T log K + d C ) regret with high probability against an adaptive adversary, and its application to a discretization of a general compact action set yields O ~ ( d T + d C ) \widetilde{O}(d\sqrt{T}+dC) O ( d T + d C ) regret, matching the lower bounds up to logarithmic factors and answering the open question of Liu et al. [36 ] on the statistical rate.
Several directions remain open. First, our discretization argument is not computationally efficient for continuous action sets, and achieving O ~ ( d T + d C ) \widetilde{O}(d\sqrt{T}+dC) O ( d T + d C ) regret with a polynomial-time algorithm remains open. Even for finite action sets, each round of CRW-FTRL solves a max-min problem whose inner objective is only geodesically convex, and designing efficient solvers for it with provable guarantees is an interesting practical question. Second, like most algorithms for strong corruption [11 , 36 ] (He et al. 2022), CRW-FTRL requires the corruption budget C C C as an input, and it would be interesting to understand the best achievable regret when C C C is unknown, where He et al. (2022) show that a trade-off with the corruption-free regret is unavoidable. in the stochastic setting. Finally, extending our radius-based estimators to linear contextual bandits with changing action sets and to reinforcement learning with linear function approximation is a promising direction.
AI Usage Statement
Large language models, specifically Gemini 4 Argon and Claude Opus 5.5, played a central role in this work. Beyond assisting with the writing and editing of the manuscript, the models proposed the core algorithm design and the main proof strategy, and carried out the detailed mathematical derivations and technical verification. The authors formulated the research problem, guided the iterative development of the ideas, audited the proofs, and take full responsibility for all results and claims in the paper.
Proofs of the Main Results
Proof of Theorem 5.1
Proof of Theorem 5.1. Fix an arbitrary comparator action i ⋆ ∈ [ K ] i^{\star} \in[K] i ⋆ ∈ [ K ] and let u = ( 1 − 1 2 T ) e i ⋆ + 1 2 T ρ \mathbf{u} = \left(1-\frac{1}{2T}\right)\mathbf{e}_{i^{\star}}+\frac{1}{2T}\boldsymbol{\rho} u = ( 1 − 2 T 1 ) e i ⋆ + 2 T 1 ρ . Since T ≥ 1 T \ge1 T ≥ 1 and e i ⋆ , ρ ∈ Δ K \mathbf{e}_{i^{\star}},\boldsymbol{\rho} \in\Delta_K e i ⋆ , ρ ∈ Δ K , we have u ∈ Δ K \mathbf{u} \in\Delta_K u ∈ Δ K and u i ≥ ρ i / ( 2 T ) u_i \ge\rho_i/(2T) u i ≥ ρ i / ( 2 T ) for all i ∈ [ K ] i \in[K] i ∈ [ K ] , so u ∈ P \mathbf{u} \in\mathcal{P} u ∈ P . By the decomposition (6.1) and the smoothing bound (6.3) in Section 6.1,
∑ t = 1 T ( a i ⋆ − a I t ) ⊤ θ t = T 0 + T 1 + T 2 + T 3 with T 0 ≤ 1. \sum_{t=1}^{T}(\mathbf{a}_{i^{\star}}-\mathbf{a}_{I_t})^{\top}\boldsymbol{\theta}_t=T_0+T_1+T_2+T_3 \qquad\text{with } T_0\le1. t = 1 ∑ T ( a i ⋆ − a I t ) ⊤ θ t = T 0 + T 1 + T 2 + T 3 with T 0 ≤ 1.
Let E \mathcal{E} E be the event of Lemma 6.4, which holds with probability at least 1 − δ 1-\delta 1 − δ and does not depend on i ⋆ i^{\star} i ⋆ . On E \mathcal{E} E , applying Lemma 6.2 to T 1 T_1 T 1 and Lemma 6.4 to T 2 T_2 T 2 and T 3 T_3 T 3 with u ∈ P \mathbf{u} \in\mathcal{P} u ∈ P gives
∑ t = 1 T ( a i ⋆ − a I t ) ⊤ θ t ≤ 1 + ( − α ⟨ u , r T ⟩ + κ A T + 40 η T ) + ( L η + 3 C ) ⟨ u , r T ⟩ + ( 2 T log ( 2 / δ ) + C + 8 η T ) = ( b ) ( − α + L η + 3 C ) ⟨ u , r T ⟩ + κ A T + 48 η T + C + 2 T log ( 2 / δ ) + 1 = ( c ) κ A T + 48 η T + C + 2 T log ( 2 / δ ) + 1 ≤ 3 A T C + 2 A T L η + 48 η T + C + 2 T log ( 2 / δ ) + 1 , \begin{aligned}
\sum_{t=1}^{T}(\mathbf{a}_{i^{\star}}-\mathbf{a}_{I_t})^{\top}\boldsymbol{\theta}_t
&\le1+\left(-\alpha\langle\mathbf{u},\mathbf{r}_T\rangle+\kappa A_T+40\eta T\right)+\left(\frac{L}{\eta}+3C\right)\langle\mathbf{u},\mathbf{r}_T\rangle\\
&\quad+\left(\sqrt{2T\log(2/\delta)}+C+8\eta T\right)\\
&\overset{(b)}{=}\left(-\alpha+\frac{L}{\eta}+3C\right)\langle\mathbf{u},\mathbf{r}_T\rangle+\kappa A_T+48\eta T+C+\sqrt{2T\log(2/\delta)}+1\\
&\overset{(c)}{=}\kappa A_T+48\eta T+C+\sqrt{2T\log(2/\delta)}+1\\
&\le3A_TC+\frac{2A_TL}{\eta}+48\eta T+C+\sqrt{2T\log(2/\delta)}+1,
\end{aligned} t = 1 ∑ T ( a i ⋆ − a I t ) ⊤ θ t ≤ 1 + ( − α ⟨ u , r T ⟩ + κ A T + 40 η T ) + ( η L + 3 C ) ⟨ u , r T ⟩ + ( 2 T log ( 2/ δ ) + C + 8 η T ) = ( b ) ( − α + η L + 3 C ) ⟨ u , r T ⟩ + κ A T + 48 η T + C + 2 T log ( 2/ δ ) + 1 = ( c ) κ A T + 48 η T + C + 2 T log ( 2/ δ ) + 1 ≤ 3 A T C + η 2 A T L + 48 η T + C + 2 T log ( 2/ δ ) + 1 ,
where (a) substitutes the four bounds; (b) groups the ⟨ u , r T ⟩ \langle\mathbf{u},\mathbf{r}_T\rangle ⟨ u , r T ⟩ and η T \eta T η T terms; (c) uses α = 3 C + L / η \alpha=3C+L/\eta α = 3 C + L / η ; and (d) uses κ = α + log ( 4 K T ) / η ≤ 3 C + 2 L / η \kappa=\alpha+\log(4KT)/\eta\le3C+2L/\eta κ = α + log ( 4 K T ) / η ≤ 3 C + 2 L / η (since log ( 4 K T ) < log ( 8 K T / δ ) = L \log(4KT)<\log(8KT/\delta)=L log ( 4 K T ) < log ( 8 K T / δ ) = L for δ ∈ ( 0 , 1 ) \delta\in(0,1) δ ∈ ( 0 , 1 ) ) and A T ≥ 0 A_T\ge0 A T ≥ 0 . Taking the maximum over i ⋆ ∈ [ K ] i^{\star}\in[K] i ⋆ ∈ [ K ] on E \mathcal{E} E completes the proof. ◻
Proof of Corollary 5.2
Proof of Corollary 5.2. For η = min { 1 / 12 , A T L / ( 24 T ) } \eta= \min\{1/12,\sqrt{A_TL/(24T)}\} η = min { 1/12 , A T L / ( 24 T ) } , we have η − 1 = max { 12 , 24 T / ( A T L ) } \eta^{-1} = \max\{12,\sqrt{24T/(A_TL)}\} η − 1 = max { 12 , 24 T / ( A T L ) } and η ≤ A T L / ( 24 T ) \eta\le\sqrt{A_TL/(24T)} η ≤ A T L / ( 24 T ) , so the η \eta η -dependent terms in Theorem 5.1 satisfy
2 A T L η + 48 η T = O ( A T L + A T L T ) . \frac{2A_TL}{\eta} + 48\eta T = O\left(A_TL + \sqrt{A_TLT}\right). η 2 A T L + 48 η T = O ( A T L + A T L T ) .
Since A T = Θ ( d log ( d T ) ) ≥ 1 A_T = \Theta(d\log(dT)) \ge1 A T = Θ ( d log ( d T )) ≥ 1 and L = log ( 8 K T / δ ) ≥ max { log 8 , log ( 2 / δ ) } ≥ 1 L = \log(8KT/\delta) \ge\max\{\log8,\log(2/\delta)\} \ge1 L = log ( 8 K T / δ ) ≥ max { log 8 , log ( 2/ δ )} ≥ 1 , substituting this bound into Theorem 5.1 gives, with probability at least 1 − δ 1-\delta 1 − δ ,
Reg T = O ( A T C + A T L + A T L T ) . \operatorname{Reg}_T = O\left(A_TC + A_TL + \sqrt{A_TLT}\right). Reg T = O ( A T C + A T L + A T L T ) .
Moreover, because ∣ a i ⊤ θ t ∣ ≤ 1 \lvert\mathbf{a}_i^\top\boldsymbol{\theta}_t\rvert\le1 ∣ a i ⊤ θ t ∣ ≤ 1 (Assumption 3.1 ), we always have Reg T ≤ 2 T \operatorname{Reg}_T \le2T Reg T ≤ 2 T ; thus when T < A T L T < A_TL T < A T L , Reg T ≤ 2 T < 2 A T L T \operatorname{Reg}_T \le2T < 2\sqrt{A_TLT} Reg T ≤ 2 T < 2 A T L T , so the A T L A_TL A T L term is absorbed into O ( A T L T ) O(\sqrt{A_TLT}) O ( A T L T ) in all cases. Substituting A T = O ( d log ( d T ) ) A_T = O(d\log(dT)) A T = O ( d log ( d T )) and L = O ( log ( K T / δ ) ) L = O(\log(KT/\delta)) L = O ( log ( K T / δ )) yields the stated bound.
Proofs of the Lemmas in Section 6
Throughout the appendices, Algorithm 1 is run with the parameters η ∈ ( 0 , 1 / 12 ] \eta\in(0,1/12] η ∈ ( 0 , 1/12 ] , L = log ( 8 K T / δ ) L = \log(8KT/\delta) L = log ( 8 K T / δ ) , α = 3 C + L / η \alpha= 3C + L/\eta α = 3 C + L / η , and κ = α + log ( 4 K T ) / η \kappa= \alpha+ \log(4KT)/\eta κ = α + log ( 4 K T ) / η specified in Theorem 5.1 . We begin with two foundational lemmas that are used across Sections B.1–B.3 (all supporting lemmas stated in this section are proved in Appendix § 11 ). First, by construction in Algorithm 1, p t \mathbf{p}_t p t maximizes ⟨ S t − 1 , p ⟩ + min Q ⪰ Q t − 1 J ( Q , p ) \langle\mathbf{S}_{t-1},\mathbf{p}\rangle+ \min_{\mathbf{Q}\succeq\mathbf{Q}_{t-1}} J(\mathbf{Q},\mathbf{p}) ⟨ S t − 1 , p ⟩ + min Q ⪰ Q t − 1 J ( Q , p ) jointly with Q t \mathbf{Q}_t Q t . The following lemma shows that all round-t t t quantities computed before sampling I t I_t I t are F t − 1 \mathcal{F}_{t-1} F t − 1 -measurable, and that once Q t \mathbf{Q}_t Q t is fixed at its inner minimizer, p t \mathbf{p}_t p t is a standard FTRL iterate for the round-t t t regularizer ψ t \psi_t ψ t .
Lemma B.1 (Predictability and Frozen Form). For every t ∈ [ T ] t \in[T] t ∈ [ T ] , the quantities p t \mathbf{p}_t p t , Q t \mathbf{Q}_t Q t , r t \mathbf{r}_t r t , G t \mathbf{G}_t G t , k ~ t \widetilde{k}_t k t and v t \mathbf{v}_t v t are F t − 1 \mathcal{F}_{t-1} F t − 1 -measurable, and g ^ t \widehat{\mathbf{g}}_t g t and S t \mathbf{S}_t S t are F t \mathcal{F}_t F t -measurable. Moreover, with the regularizer ψ t ( p ) : = − ∑ i r t i h ( p i ) = ∑ i r t i ( η − 1 p i log p i − α p i ) \psi_t(\mathbf{p}) := -\sum_i r_{ti}h(p_i) = \sum_i r_{ti}(\eta^{-1}p_i\log p_i-\alpha p_i) ψ t ( p ) := − ∑ i r t i h ( p i ) = ∑ i r t i ( η − 1 p i log p i − α p i ) of round t t t , the iterate has the frozen form
p t = arg max p ∈ P { ⟨ S t − 1 , p ⟩ − ψ t ( p ) } . (10) \mathbf{p}_t = \underset{\mathbf{p}\in\mathcal{P}}{\arg\max}\left\{\langle\mathbf{S}_{t-1},\mathbf{p}\rangle-\psi_t(\mathbf{p})\right\}. \tag*{(10)} p t = p ∈ P arg max { ⟨ S t − 1 , p ⟩ − ψ t ( p ) } . (10)
Second, the next lemma summarizes the algebraic and moment properties of the normalized kernel k ~ t \widetilde{k}_t k t and variance bonus v t \mathbf{v}_t v t under the envelope condition x i ⊤ G t − 1 x i ≤ r t i 2 \mathbf{x}_i^\top\mathbf{G}_t^{-1}\mathbf{x}_i \le r_{ti}^2 x i ⊤ G t − 1 x i ≤ r t i 2 .
Lemma B.2 (Kernel). Let p ∈ P \mathbf{p}\in\mathcal{P} p ∈ P and r i ≥ 1 r_i\ge1 r i ≥ 1 , let G = ∑ i = 1 K ( p i / r i ) x i x i ⊤ ≻ 0 \mathbf{G}=\sum_{i=1}^{K}(p_i/r_i)\mathbf{x}_i\mathbf{x}_i^\top\succ0 G = ∑ i = 1 K ( p i / r i ) x i x i ⊤ ≻ 0 , and suppose that x i ⊤ G − 1 x i ≤ r i 2 \mathbf{x}_i^\top\mathbf{G}^{-1}\mathbf{x}_i\le r_i^2 x i ⊤ G − 1 x i ≤ r i 2 for all i ∈ [ K ] i\in[K] i ∈ [ K ] . Then for all i , j ∈ [ K ] i,j\in[K] i , j ∈ [ K ] :
(a) ∑ j = 1 K p j k ~ ( i , j ) x j = x i \sum_{j=1}^{K}p_j\widetilde{k}(i,j)\mathbf{x}_j=\mathbf{x}_i ∑ j = 1 K p j k ( i , j ) x j = x i ;
(b) ∑ i = 1 K p i k ~ ( i , j ) = 1 \sum_{i=1}^{K}p_i\widetilde{k}(i,j)=1 ∑ i = 1 K p i k ( i , j ) = 1 ;
(c) ∣ k ~ ( i , j ) ∣ ≤ 3 r i \lvert\widetilde{k}(i,j)\rvert\le3r_i ∣ k ( i , j )∣ ≤ 3 r i ;
(d) ∑ i = 1 K ( p i / r i ) k ~ ( i , j ) 2 ≤ 2 \sum_{i=1}^{K}(p_i/r_i)\widetilde{k}(i,j)^2\le2 ∑ i = 1 K ( p i / r i ) k ( i , j ) 2 ≤ 2 ;
(e) 0 ≤ v i ≤ 9 r i 0\le v_i\le9r_i 0 ≤ v i ≤ 9 r i and ∑ i = 1 K p i v i ≤ 2 \sum_{i=1}^{K}p_iv_i\le2 ∑ i = 1 K p i v i ≤ 2 .
The FTRL Bound
To prove the per-round stability bound (Lemma 6.1 ), we use the following local strong convexity inequality for a radius-weighted negative entropy regularizer with fixed radii r \mathbf{r} r . By the frozen form ((B.1)), p t \mathbf{p}_t p t maximizes Λ t ( p ) = ⟨ S t − 1 , p ⟩ − ψ t ( p ) \Lambda_t(\mathbf{p}) = \langle\mathbf{S}_{t-1},\mathbf{p}\rangle- \psi_t(\mathbf{p}) Λ t ( p ) = ⟨ S t − 1 , p ⟩ − ψ t ( p ) over P \mathcal{P} P , so once we verify the scale condition η g ^ t ( i ) ≤ r t i \eta\widehat{g}_t(i) \le r_{ti} η g t ( i ) ≤ r t i using Lemma B.2 (c),(e), Lemma B.3 bounds the round-t t t stability term by η ∑ i p t i g ^ t ( i ) 2 / r t i \eta\sum_i p_{ti}\widehat{g}_t(i)^2/r_{ti} η ∑ i p t i g t ( i ) 2 / r t i .
Lemma B.3 (Stability of Weighted Entropy). Let r ∈ R K \mathbf{r} \in\mathbb{R}^K r ∈ R K , ψ ( p ) = ∑ i r i ( η − 1 p i log p i − α p i ) \psi(\mathbf{p}) = \sum_i r_i\left(\eta^{-1}p_i\log p_i-\alpha p_i\right) ψ ( p ) = ∑ i r i ( η − 1 p i log p i − α p i ) , Λ ( p ) = ⟨ S , p ⟩ − ψ ( p ) \Lambda(\mathbf{p}) = \langle\mathbf{S},\mathbf{p}\rangle-\psi(\mathbf{p}) Λ ( p ) = ⟨ S , p ⟩ − ψ ( p ) and p ⋆ ∈ a r g m a x P Λ \mathbf{p}^{\star} \in\mathop{\rm argmax}_{\mathcal{P}}\Lambda p ⋆ ∈ argmax P Λ . Let g ∈ R K \mathbf{g} \in\mathbb{R}^K g ∈ R K satisfy η g i ≤ r i \eta g_i \le r_i η g i ≤ r i for all i i i . Then for every q ∈ P \mathbf{q} \in\mathcal{P} q ∈ P ,
Λ ( q ) − Λ ( p ⋆ ) + ⟨ g , q − p ⋆ ⟩ ≤ η ∑ i p i ⋆ g i 2 / r i . \Lambda(\mathbf{q})-\Lambda(\mathbf{p}^{\star})+\langle\mathbf{g},\mathbf{q}-\mathbf{p}^{\star}\rangle\le\eta\sum_i p_i^{\star}g_i^2/r_i. Λ ( q ) − Λ ( p ⋆ ) + ⟨ g , q − p ⋆ ⟩ ≤ η i ∑ p i ⋆ g i 2 / r i .
Proof of Lemma 6.1. We first check that the round-t t t reward estimate satisfies η g ^ t ( i ) ≤ r t i \eta\widehat{g}_t(i) \le r_{ti} η g t ( i ) ≤ r t i for all i ∈ [ K ] i \in[K] i ∈ [ K ] :
η g ^ t ( i ) r t i ≤ ( a ) η ( ∣ k ~ t ( i , I t ) ∣ ∣ y t ∣ + 4 η v t i ) r t i ≤ ( b ) η ( 3 r t i ⋅ 3 + 4 η ⋅ 9 r t i ) r t i = 9 η + 36 η 2 ≤ ( c ) 1 , \frac{\eta\widehat{g}_t(i)}{r_{ti}} \mathrel{\overset{(a)}{\le}} \frac{\eta\left(\lvert\widetilde{k}_t(i,I_t)\rvert\lvert y_t\rvert+4\eta v_{ti}\right)}{r_{ti}} \mathrel{\overset{(b)}{\le}} \frac{\eta(3r_{ti}\cdot3+4\eta\cdot9r_{ti})}{r_{ti}}=9\eta+36\eta^2 \mathrel{\overset{(c)}{\le}}1, r t i η g t ( i ) ≤ ( a ) r t i η ( ∣ k t ( i , I t )∣ ∣ y t ∣ + 4 η v t i ) ≤ ( b ) r t i η ( 3 r t i ⋅ 3 + 4 η ⋅ 9 r t i ) = 9 η + 36 η 2 ≤ ( c ) 1 ,
where (a) uses the definition of g ^ t ( i ) \widehat{g}_t(i) g t ( i ) , the triangle inequality and v t i ≥ 0 v_{ti}\ge0 v t i ≥ 0 ; (b) uses Lemma B.2 (c)(e) and ∣ y t ∣ ≤ 3 \lvert y_t\rvert\le3 ∣ y t ∣ ≤ 3 ; and (c) uses η ≤ 1 / 12 \eta\le1/12 η ≤ 1/12 , which gives 9 η + 36 η 2 ≤ 3 / 4 + 1 / 4 = 1 9\eta+36\eta^2\le3/4+1/4=1 9 η + 36 η 2 ≤ 3/4 + 1/4 = 1 . By the frozen form ((B.1)), p t ∈ a r g m a x P Λ t \mathbf{p}_t\in\mathop{\rm argmax}_{\mathcal{P}}\Lambda_t p t ∈ argmax P Λ t , so Lemma B.3 applied with S = S t − 1 \mathbf{S}=\mathbf{S}_{t-1} S = S t − 1 , r = r t \mathbf{r}=\mathbf{r}_t r = r t , g = g ^ t \mathbf{g}=\widehat{\mathbf{g}}_t g = g t and p ⋆ = p t \mathbf{p}^{\star}=\mathbf{p}_t p ⋆ = p t yields the first inequality of Lemma 6.1 for every q ∈ P \mathbf{q}\in\mathcal{P} q ∈ P . For the second inequality,
∑ i p t i g ^ t ( i ) 2 r t i ≤ ( a ) ∑ i p t i r t i ( 2 k ~ t ( i , I t ) 2 y t 2 + 2 ⋅ 16 η 2 v t i 2 ) ≤ ( b ) 18 ∑ i p t i r t i k ~ t ( i , I t ) 2 + 32 η 2 ∑ i p t i v t i v t i r t i \sum_i\frac{p_{ti}\widehat{g}_t(i)^2}{r_{ti}}\mathrel{\overset{(a)}{\le}}\sum_i\frac{p_{ti}}{r_{ti}}\left(2\widetilde{k}_t(i,I_t)^2y_t^2+2\cdot16\eta^2v_{ti}^2\right)\mathrel{\overset{(b)}{\le}}18\sum_i\frac{p_{ti}}{r_{ti}}\widetilde{k}_t(i,I_t)^2+32\eta^2\sum_i p_{ti}v_{ti}\frac{v_{ti}}{r_{ti}} i ∑ r t i p t i g t ( i ) 2 ≤ ( a ) i ∑ r t i p t i ( 2 k t ( i , I t ) 2 y t 2 + 2 ⋅ 16 η 2 v t i 2 ) ≤ ( b ) 18 i ∑ r t i p t i k t ( i , I t ) 2 + 32 η 2 i ∑ p t i v t i r t i v t i
≤ ( c ) 18 ⋅ 2 + 32 η 2 ⋅ 9 ∑ i p t i v t i ≤ ( d ) 36 + 576 η 2 ≤ ( e ) 40 , \mathrel{\overset{(c)}{\le}}18\cdot2+32\eta^2\cdot9\sum_i p_{ti}v_{ti}\mathrel{\overset{(d)}{\le}}36+576\eta^2\mathrel{\overset{(e)}{\le}}40, ≤ ( c ) 18 ⋅ 2 + 32 η 2 ⋅ 9 i ∑ p t i v t i ≤ ( d ) 36 + 576 η 2 ≤ ( e ) 40 ,
where (a) uses g ^ t ( i ) = k ~ t ( i , I t ) y t + 4 η v t i \widehat{g}_t(i)=\widetilde{k}_t(i,I_t)y_t+4\eta v_{ti} g t ( i ) = k t ( i , I t ) y t + 4 η v t i and ( a + b ) 2 ≤ 2 a 2 + 2 b 2 (a+b)^2\le2a^2+2b^2 ( a + b ) 2 ≤ 2 a 2 + 2 b 2 ; (b) uses y t 2 ≤ 9 y_t^2\le9 y t 2 ≤ 9 ; (c) uses Lemma B.2 (d) with j = I t j=I_t j = I t and v t i / r t i < 9 v_{ti}/r_{ti}<9 v t i / r t i < 9 (Lemma B.2 (e)); (d) uses ∑ i p t i v t i ≤ 2 \sum_i p_{ti}v_{ti}\le2 ∑ i p t i v t i ≤ 2 (Lemma B.2 (e)); and (e) uses η ≤ 1 / 12 \eta\le1/12 η ≤ 1/12 . Multiplying by η \eta η gives the second inequality of Lemma 6.1 . □
We now turn to the cumulative FTRL bound (Lemma 6.2 ). First, the following standard FTRL inequality for time-varying regularizers (Lattimore and Szepesvári [29 ] , Exercise 28.12) decomposes ∑ t = 1 T ⟨ g ^ t , u − p t ⟩ \sum_{t=1}^{T}\langle\widehat{\mathbf{g}}_t,\mathbf{u}-\mathbf{p}_t\rangle ∑ t = 1 T ⟨ g t , u − p t ⟩ into a comparator penalty ψ T + 1 ( u ) \psi_{T+1}(\mathbf{u}) ψ T + 1 ( u ) , an initial term − ψ 1 ( p 1 ) -\psi_1(\mathbf{p}_1) − ψ 1 ( p 1 ) , regularizer drift terms ψ t ( p t + 1 ) − ψ t + 1 ( p t + 1 ) \psi_t(\mathbf{p}_{t+1})-\psi_{t+1}(\mathbf{p}_{t+1}) ψ t ( p t + 1 ) − ψ t + 1 ( p t + 1 ) , and the per-round stability terms bounded in Lemma 6.1 .
Lemma B.4 (General FTRL Inequality; cf. Lattimore and Szepesvári [29 ] , Exercise 28.12). Let ψ 1 , … , ψ T + 1 : P → R \psi_1,\ldots,\psi_{T+1}:\mathcal{P}\to\mathbb{R} ψ 1 , … , ψ T + 1 : P → R and g 1 , … , g T ∈ R K \mathbf{g}_1,\ldots,\mathbf{g}_T\in\mathbb{R}^K g 1 , … , g T ∈ R K . Define S t = ∑ s ≤ t g s \mathbf{S}_t=\sum_{s\le t}\mathbf{g}_s S t = ∑ s ≤ t g s with S 0 = 0 \mathbf{S}_0=0 S 0 = 0 , Λ t ( p ) = ⟨ S t − 1 , p ⟩ − ψ t ( p ) \Lambda_t(\mathbf{p})=\langle\mathbf{S}_{t-1},\mathbf{p}\rangle-\psi_t(\mathbf{p}) Λ t ( p ) = ⟨ S t − 1 , p ⟩ − ψ t ( p ) , and p t ∈ a r g m a x P Λ t \mathbf{p}_t\in\mathop{\rm argmax}_{\mathcal{P}}\Lambda_t p t ∈ argmax P Λ t for t ∈ [ T + 1 ] t\in[T+1] t ∈ [ T + 1 ] . Then for every u ∈ P \mathbf{u}\in\mathcal{P} u ∈ P ,
∑ t = 1 T ⟨ g t , u − p t ⟩ ≤ ψ T + 1 ( u ) − ψ 1 ( p 1 ) + ∑ t = 1 T [ ψ t ( p t + 1 ) − ψ t + 1 ( p t + 1 ) ] + ∑ t = 1 T [ Λ t ( p t + 1 ) − Λ t ( p t ) + ⟨ g t , p t + 1 − p t ⟩ ] . \begin{aligned}
\sum_{t=1}^{T}\langle\mathbf{g}_t,\mathbf{u}-\mathbf{p}_t\rangle\le\psi_{T+1}(\mathbf{u})-\psi_1(\mathbf{p}_1)+\sum_{t=1}^{T}\left[\psi_t(\mathbf{p}_{t+1})-\psi_{t+1}(\mathbf{p}_{t+1})\right] \\
&+\sum_{t=1}^{T}\left[\Lambda_t(\mathbf{p}_{t+1})-\Lambda_t(\mathbf{p}_t)+\langle\mathbf{g}_t,\mathbf{p}_{t+1}-\mathbf{p}_t\rangle\right].
\end{aligned} t = 1 ∑ T ⟨ g t , u − p t ⟩ ≤ ψ T + 1 ( u ) − ψ 1 ( p 1 ) + t = 1 ∑ T [ ψ t ( p t + 1 ) − ψ t + 1 ( p t + 1 ) ] + t = 1 ∑ T [ Λ t ( p t + 1 ) − Λ t ( p t ) + ⟨ g t , p t + 1 − p t ⟩ ] .
To bound the initial term − ψ 1 ( p 1 ) -\psi_1(\mathbf{p}_1) − ψ 1 ( p 1 ) and the regularizer drift terms ψ t ( p t + 1 ) − ψ t + 1 ( p t + 1 ) \psi_t(\mathbf{p}_{t+1})-\psi_{t+1}(\mathbf{p}_{t+1}) ψ t ( p t + 1 ) − ψ t + 1 ( p t + 1 ) , we use the optimality of Q t = argmin Q ⪰ Q t − 1 J ( Q , p t ) \mathbf{Q}_t=\operatorname*{argmin}_{\mathbf{Q}\succeq\mathbf{Q}_{t-1}}J(\mathbf{Q},\mathbf{p}_t) Q t = argmin Q ⪰ Q t − 1 J ( Q , p t ) together with three supporting lemmas:
Lemma B.5 sandwiches the regularizer weight h ( p i ) h(p_i) h ( p i ) between α p i \alpha p_i α p i and κ p i \kappa p_i κ p i ;
Lemma B.6 establishes the properties of the reference design ρ \boldsymbol{\rho} ρ and whitened actions x i \mathbf{x}_i x i ; in particular, item (c) bounds the initial radii r 0 i = ∥ x i ∥ ≤ 2 ( d + 1 ) r_{0i}=\lVert\mathbf{x}_i\rVert\le\sqrt{2(d+1)} r 0 i = ∥ x i ∥ ≤ 2 ( d + 1 ) , while item (d) expresses the clean rewards a i T θ t \mathbf{a}_i^{\mathsf{T}}\boldsymbol{\theta}_t a i T θ t as linear functions of x i \mathbf{x}_i x i ;
Lemma B.7 establishes the properties of the radii r t i = r i ( Q t ) r_{ti}=r_i(\mathbf{Q}_t) r t i = r i ( Q t ) from the first-order optimality of Q t \mathbf{Q}_t Q t : item (i) guarantees the envelope condition x i T G t − 1 x i ≤ r t i 2 \mathbf{x}_i^{\mathsf{T}}\mathbf{G}_t^{-1}\mathbf{x}_i\le r_{ti}^2 x i T G t − 1 x i ≤ r t i 2 needed to invoke Lemma B.2 ; item (ii) ensures that the radii are at least 1 1 1 and nondecreasing (r t − 1 , i ≤ r t i r_{t-1,i}\le r_{ti} r t − 1 , i ≤ r t i ); and item (iii) bounds the final log-determinant log det Q T \log\det\mathbf{Q}_T log det Q T to yield the κ A T \kappa A_T κ A T drift bound.
Lemma B.5 (Sandwich). For every p ∈ P \mathbf{p}\in\mathcal{P} p ∈ P and every i ∈ [ K ] i\in[K] i ∈ [ K ] , α p i ≤ h ( p i ) ≤ κ p i \alpha p_i\le h(p_i)\le\kappa p_i α p i ≤ h ( p i ) ≤ κ p i , where κ ≤ 2 α \kappa\le2\alpha κ ≤ 2 α .
κ ≤ 2 α . (11) \kappa\le2\alpha. \tag*{(11)} κ ≤ 2 α . (11)
Lemma B.6 (Reference Design). H ( ρ ) ≻ 0 \mathbf{H}(\boldsymbol{\rho})\succ0 H ( ρ ) ≻ 0 , and
(a) ρ i ≥ 1 / ( 2 K ) \rho_i\ge1/(2K) ρ i ≥ 1/ ( 2 K ) ;
(b) ∑ i ρ i x i x i T = I \sum_i\rho_i\mathbf{x}_i\mathbf{x}_i^{\mathsf{T}}=\mathbf{I} ∑ i ρ i x i x i T = I ;
(c) 1 ≤ ∥ x i ∥ 2 ≤ 2 ( d + 1 ) 1\le\lVert\mathbf{x}_i\rVert^2\le2(d+1) 1 ≤ ∥ x i ∥ 2 ≤ 2 ( d + 1 ) ;
(d) a i T θ t = x i T H ( ρ ) 1 / 2 ( θ t T , 0 ) T \mathbf{a}_i^{\mathsf{T}}\boldsymbol{\theta}_t=\mathbf{x}_i^{\mathsf{T}}\mathbf{H}(\boldsymbol{\rho})^{1/2}(\boldsymbol{\theta}_t^{\mathsf{T}},0)^{\mathsf{T}} a i T θ t = x i T H ( ρ ) 1/2 ( θ t T , 0 ) T for all i i i and t t t .
Lemma B.7 (Envelope). For every t ∈ [ T ] t\in[T] t ∈ [ T ] :
(i) G t ⪰ Q t − 1 \mathbf{G}_t\succeq\mathbf{Q}_t^{-1} G t ⪰ Q t − 1 , hence x i T G t − 1 x i ≤ r t i 2 \mathbf{x}_i^{\mathsf{T}}\mathbf{G}_t^{-1}\mathbf{x}_i\le r_{ti}^2 x i T G t − 1 x i ≤ r t i 2 for all i ∈ [ K ] i\in[K] i ∈ [ K ] ;
(ii) I = Q 0 ⪯ Q t − 1 ⪯ Q t \mathbf{I}=\mathbf{Q}_0\preceq\mathbf{Q}_{t-1}\preceq\mathbf{Q}_t I = Q 0 ⪯ Q t − 1 ⪯ Q t and 1 ≤ ∥ x i ∥ ≤ r t − 1 , i ≤ r t i 1\le\lVert\mathbf{x}_i\rVert\le r_{t-1,i}\le r_{ti} 1 ≤ ∥ x i ∥ ≤ r t − 1 , i ≤ r t i ;
(iii) log det Q T ≤ 2 ( d + 1 ) log ( 1 + 4 2 ( d + 1 ) 3 / 2 T 2 ) \log\det\mathbf{Q}_T\le2(d+1)\log\left(1+4\sqrt{2}(d+1)^{3/2}T^2\right) log det Q T ≤ 2 ( d + 1 ) log ( 1 + 4 2 ( d + 1 ) 3/2 T 2 ) , hence 2 ( d + 1 ) + 1 2 log det Q T ≤ A T \sqrt{2(d+1)}+\frac{1}{2}\log\det\mathbf{Q}_T\le A_T 2 ( d + 1 ) + 2 1 log det Q T ≤ A T .
Proof of Lemma 6.2 . By the frozen form (B.1), for t ≤ T t\le T t ≤ T the iterate p t \mathbf{p}_t p t maximizes Λ t = ⟨ S t − 1 , ⋅ ⟩ − ψ t \Lambda_t=\langle\mathbf{S}_{t-1},\cdot\rangle-\psi_t Λ t = ⟨ S t − 1 , ⋅ ⟩ − ψ t over P \mathcal{P} P . We set ψ T + 1 : = ψ T \psi_{T+1}:=\psi_T ψ T + 1 := ψ T and let p T + 1 \mathbf{p}_{T+1} p T + 1 maximize ⟨ S T , p ⟩ − ψ T ( p ) \langle\mathbf{S}_T,\mathbf{p}\rangle-\psi_T(\mathbf{p}) ⟨ S T , p ⟩ − ψ T ( p ) over P \mathcal{P} P ; this point is used only in the analysis. Lemma B.4, applied with g t = g ^ t \mathbf{g}_t=\widehat{\mathbf{g}}_t g t = g t , bounds the left side of the claim by a comparator term, an initial term, drift terms and stability terms, which we bound in this order.
The comparator term reduces to its linear part:
ψ T + 1 ( u ) = ∑ i r T i ( η − 1 u i log u i − α u i ) ≤ − α ∑ i r T i u i = − α ⟨ u , r T ⟩ , \psi_{T+1}(\mathbf{u})=\sum_i r_{Ti}\left(\eta^{-1}u_i\log u_i-\alpha u_i\right)\le-\alpha\sum_i r_{Ti}u_i=-\alpha\langle\mathbf{u},\mathbf{r}_T\rangle, ψ T + 1 ( u ) = i ∑ r T i ( η − 1 u i log u i − α u i ) ≤ − α i ∑ r T i u i = − α ⟨ u , r T ⟩ ,
where the first equality uses ψ T + 1 = ψ T \psi_{T+1}=\psi_T ψ T + 1 = ψ T and the definition of ψ T \psi_T ψ T , and the inequality uses u i log u i ≤ 0 u_i\log u_i\le0 u i log u i ≤ 0 for u i ∈ ( 0 , 1 ] u_i\in(0,1] u i ∈ ( 0 , 1 ] and r T i > 0 r_{Ti}>0 r T i > 0 .
For the initial and drift terms, recall that J ( Q , p ) = − ψ Q ( p ) − κ 2 log det Q J(\mathbf{Q},\mathbf{p})=-\psi_{\mathbf{Q}}(\mathbf{p})-\frac{\kappa}{2}\log\det\mathbf{Q} J ( Q , p ) = − ψ Q ( p ) − 2 κ log det Q , so − ψ t ( p ) = J ( Q t , p ) + κ 2 log det Q t -\psi_t(\mathbf{p})=J(\mathbf{Q}_t,\mathbf{p})+\frac{\kappa}{2}\log\det\mathbf{Q}_t − ψ t ( p ) = J ( Q t , p ) + 2 κ log det Q t for each t ∈ [ T ] t\in[T] t ∈ [ T ] . Since Q 1 \mathbf{Q}_1 Q 1 minimizes J ( Q , p 1 ) J(\mathbf{Q},\mathbf{p}_1) J ( Q , p 1 ) over Q ⪰ Q 0 = I \mathbf{Q}\succeq\mathbf{Q}_0=\mathbf{I} Q ⪰ Q 0 = I , we have J ( Q 1 , p 1 ) ≤ J ( Q 0 , p 1 ) J(\mathbf{Q}_1,\mathbf{p}_1)\le J(\mathbf{Q}_0,\mathbf{p}_1) J ( Q 1 , p 1 ) ≤ J ( Q 0 , p 1 ) , and therefore
− ψ 1 ( p 1 ) = ( a ) J ( Q 1 , p 1 ) + κ 2 log det Q 1 ≤ ( b ) J ( Q 0 , p 1 ) + κ 2 log det Q 1 -\psi_1(\mathbf{p}_1)\mathrel{\overset{(a)}{=}}J(\mathbf{Q}_1,\mathbf{p}_1)+\frac{\kappa}{2}\log\det\mathbf{Q}_1\mathrel{\overset{(b)}{\le}}J(\mathbf{Q}_0,\mathbf{p}_1)+\frac{\kappa}{2}\log\det\mathbf{Q}_1 − ψ 1 ( p 1 ) = ( a ) J ( Q 1 , p 1 ) + 2 κ log det Q 1 ≤ ( b ) J ( Q 0 , p 1 ) + 2 κ log det Q 1
∑ i h ( p 1 i ) ∥ x i ∥ + κ 2 log det Q 1 ≤ κ ( 2 ( d + 1 ) + 1 2 log det Q 1 ) \sum_i h(p_{1i})\lVert\mathbf{x}_i\rVert+\frac{\kappa}{2}\log\det\mathbf{Q}_1\leq\kappa\left(\sqrt{2(d+1)}+\frac{1}{2}\log\det\mathbf{Q}_1\right) i ∑ h ( p 1 i ) ∥ x i ∥ + 2 κ log det Q 1 ≤ κ ( 2 ( d + 1 ) + 2 1 log det Q 1 )
where (a) uses the definition of J J J ; (b) uses J ( Q 1 , p 1 ) ≤ J ( Q 0 , p 1 ) J(\mathbf{Q}_1,\mathbf{p}_1)\leq J(\mathbf{Q}_0,\mathbf{p}_1) J ( Q 1 , p 1 ) ≤ J ( Q 0 , p 1 ) ; (c) uses Q 0 = I \mathbf{Q}_0=\mathbf{I} Q 0 = I , so that r i ( Q 0 ) = ∥ x i ∥ r_i(\mathbf{Q}_0)=\lVert\mathbf{x}_i\rVert r i ( Q 0 ) = ∥ x i ∥ and log det Q 0 = 0 \log\det\mathbf{Q}_0=0 log det Q 0 = 0 ; and (d) uses h ( p 1 i ) ≤ κ p 1 i h(p_{1i})\leq\kappa p_{1i} h ( p 1 i ) ≤ κ p 1 i (Lemma B.5), ∥ x i ∥ ≤ 2 ( d + 1 ) \lVert\mathbf{x}_i\rVert\leq\sqrt{2(d+1)} ∥ x i ∥ ≤ 2 ( d + 1 ) (Lemma B.6(c)), and ∑ i p 1 i = 1 \sum_i p_{1i}=1 ∑ i p 1 i = 1 . Similarly, for each t ≤ T − 1 t\leq T-1 t ≤ T − 1 , Q t + 1 \mathbf{Q}_{t+1} Q t + 1 minimizes J ( Q , p t + 1 ) J(\mathbf{Q},\mathbf{p}_{t+1}) J ( Q , p t + 1 ) over Q ⪰ Q t \mathbf{Q}\succeq\mathbf{Q}_t Q ⪰ Q t , so J ( Q t + 1 , p t + 1 ) ≤ J ( Q t , p t + 1 ) J(\mathbf{Q}_{t+1},\mathbf{p}_{t+1})\leq J(\mathbf{Q}_t,\mathbf{p}_{t+1}) J ( Q t + 1 , p t + 1 ) ≤ J ( Q t , p t + 1 ) gives
ψ t ( p t + 1 ) − ψ t + 1 ( p t + 1 ) = J ( Q t + 1 , p t + 1 ) − J ( Q t , p t + 1 ) + κ 2 log det Q t + 1 det Q t ≤ κ 2 log det Q t + 1 det Q t . \psi_t(\mathbf{p}_{t+1})-\psi_{t+1}(\mathbf{p}_{t+1})=J(\mathbf{Q}_{t+1},\mathbf{p}_{t+1})-J(\mathbf{Q}_t,\mathbf{p}_{t+1})+\frac{\kappa}{2}\log\frac{\det\mathbf{Q}_{t+1}}{\det\mathbf{Q}_t}\leq\frac{\kappa}{2}\log\frac{\det\mathbf{Q}_{t+1}}{\det\mathbf{Q}_t}. ψ t ( p t + 1 ) − ψ t + 1 ( p t + 1 ) = J ( Q t + 1 , p t + 1 ) − J ( Q t , p t + 1 ) + 2 κ log det Q t det Q t + 1 ≤ 2 κ log det Q t det Q t + 1 .
The drift term with t = T t=T t = T vanishes because ψ T + 1 = ψ T \psi_{T+1}=\psi_T ψ T + 1 = ψ T . Adding the initial term and the drift terms,
− ψ 1 ( p 1 ) + ∑ t = 1 T [ ψ t ( p t + 1 ) − ψ t + 1 ( p t + 1 ) ] ≤ κ ( 2 ( d + 1 ) + 1 2 log det Q 1 + 1 2 ∑ t = 1 T − 1 log det Q t + 1 det Q t ) = κ ( 2 ( d + 1 ) + 1 2 log det Q T ) ≤ κ A T , \begin{aligned}
-\psi_1(\mathbf{p}_1)+\sum_{t=1}^{T}\left[\psi_t(\mathbf{p}_{t+1})-\psi_{t+1}(\mathbf{p}_{t+1})\right]\leq\kappa\left(\sqrt{2(d+1)}+\frac{1}{2}\log\det\mathbf{Q}_1+\frac{1}{2}\sum_{t=1}^{T-1}\log\frac{\det\mathbf{Q}_{t+1}}{\det\mathbf{Q}_t}\right) \\
&=\kappa\left(\sqrt{2(d+1)}+\frac{1}{2}\log\det\mathbf{Q}_T\right)\leq\kappa A_T,
\end{aligned} − ψ 1 ( p 1 ) + t = 1 ∑ T [ ψ t ( p t + 1 ) − ψ t + 1 ( p t + 1 ) ] ≤ κ ( 2 ( d + 1 ) + 2 1 log det Q 1 + 2 1 t = 1 ∑ T − 1 log det Q t det Q t + 1 ) = κ ( 2 ( d + 1 ) + 2 1 log det Q T ) ≤ κ A T ,
where (a) collects the two previous displays and the vanishing term with t = T t=T t = T , (b) telescopes the sum, and (c) is Lemma B.7(iii).
Finally, applying Lemma 6.1 with q = p t + 1 \mathbf{q}=\mathbf{p}_{t+1} q = p t + 1 and summing over t = 1 , … , T t=1,\ldots,T t = 1 , … , T bounds the stability terms by 40 η T 40\eta T 40 η T , so
∑ t = 1 T ⟨ g ^ t , u − p t ⟩ ≤ ψ T + 1 ( u ) − ψ 1 ( p 1 ) + ∑ t = 1 T [ ψ t ( p t + 1 ) − ψ t + 1 ( p t + 1 ) ] + 40 η T ≤ − α ⟨ u , r T ⟩ + κ A T + 40 η T , \begin{aligned}
\sum_{t=1}^{T}\langle\widehat{\mathbf{g}}_t,\mathbf{u}-\mathbf{p}_t\rangle\leq\psi_{T+1}(\mathbf{u})-\psi_1(\mathbf{p}_1)+\sum_{t=1}^{T}\left[\psi_t(\mathbf{p}_{t+1})-\psi_{t+1}(\mathbf{p}_{t+1})\right]+40\eta T \\
&\leq-\alpha\langle\mathbf{u},\mathbf{r}_T\rangle+\kappa A_T+40\eta T,
\end{aligned} t = 1 ∑ T ⟨ g t , u − p t ⟩ ≤ ψ T + 1 ( u ) − ψ 1 ( p 1 ) + t = 1 ∑ T [ ψ t ( p t + 1 ) − ψ t + 1 ( p t + 1 ) ] + 40 η T ≤ − α ⟨ u , r T ⟩ + κ A T + 40 η T ,
where the first inequality is Lemma B.4 together with Lemma 6.1 , and the second uses the bounds on the comparator term and on the initial and drift terms.
The Concentration Bound
We now prove Lemma 6.3 by combining the predictability of the round-t t t quantities (Lemma B.1 ), the linear representation of the clean rewards in the whitened actions (Lemma B.6(d)), the action reproduction and magnitude bounds of the normalized kernel (Lemma B.2 (a),(c)), and the monotonicity of the radii (Lemma B.7(ii)).
Proof of Lemma 6.3. Fix i ∈ [ K ] i\in[K] i ∈ [ K ] ; at the end we take a union bound over i i i . We normalize each round by its radius and set
X t = a i T θ t − g ^ t ( 0 ) ( i ) r t i , σ t 2 = 4 v t i r t i , ϑ t = H ( ρ ) 1 / 2 ( θ t T , 0 ) T . X_t=\frac{\mathbf{a}_i^{\mathsf{T}}\boldsymbol{\theta}_t-\widehat{g}_t^{(0)}(i)}{r_{ti}},\qquad\sigma_t^2=\frac{4v_{ti}}{r_{ti}},\qquad\boldsymbol{\vartheta}_t=\mathbf{H}(\rho)^{1/2}(\boldsymbol{\theta}_t^{\mathsf{T}},0)^{\mathsf{T}}. X t = r t i a i T θ t − g t ( 0 ) ( i ) , σ t 2 = r t i 4 v t i , ϑ t = H ( ρ ) 1/2 ( θ t T , 0 ) T .
The quantities computed before round t t t , namely p t \mathbf{p}_t p t , the kernel k ~ t \widetilde{k}_t k t , the radii r t i r_{ti} r t i and the variance proxies v t i v_{ti} v t i , are F t − 1 \mathcal{F}_{t-1} F t − 1 -measurable by Lemma B.1 . So is ϑ t \boldsymbol{\vartheta}_t ϑ t , because H ( ρ ) \mathbf{H}(\rho) H ( ρ ) is deterministic and θ t \boldsymbol{\theta}_t θ t is F t − 1 \mathcal{F}_{t-1} F t − 1 -measurable. We first show that X t X_t X t is a martingale difference with bounded range and controlled variance. The mean uses that the clean rewards are linear in the whitened actions, which is Lemma B.6(d). For the mean,
E [ g ^ t 0 ( i ) ∣ F t − 1 ] = ( a ) ∑ j p t j k ~ t ( i , j ) E [ a j T θ t + ξ t ∣ F t − 1 , I t = j ] = ( b ) ∑ j p t j k ~ t ( i , j ) x j T ϑ t = ( c ) x i T ϑ t = ( d ) a i T θ t , \mathbb{E}[\widehat{g}_t^0(i)\mid\mathcal{F}_{t-1}] \overset{(a)}{=} \sum_j p_{tj}\widetilde{k}_t(i,j)\mathbb{E}[\mathbf{a}_j^{\mathsf T}\theta_t+\xi_t\mid\mathcal{F}_{t-1},I_t=j] \overset{(b)}{=} \sum_j p_{tj}\widetilde{k}_t(i,j)\mathbf{x}_j^{\mathsf T}\vartheta_t \overset{(c)}{=} \mathbf{x}_i^{\mathsf T}\vartheta_t \overset{(d)}{=} \mathbf{a}_i^{\mathsf T}\theta_t, E [ g t 0 ( i ) ∣ F t − 1 ] = ( a ) j ∑ p t j k t ( i , j ) E [ a j T θ t + ξ t ∣ F t − 1 , I t = j ] = ( b ) j ∑ p t j k t ( i , j ) x j T ϑ t = ( c ) x i T ϑ t = ( d ) a i T θ t ,
where (a) conditions on I t I_t I t , using P ( I t = j ∣ F t − 1 ) = p t j \mathbb{P}(I_t=j\mid\mathcal{F}_{t-1})=p_{tj} P ( I t = j ∣ F t − 1 ) = p t j and the F t − 1 \mathcal{F}_{t-1} F t − 1 -measurability of k ~ t \widetilde{k}_t k t ; (b) uses E [ ξ t ∣ F t − 1 , I t ] = 0 \mathbb{E}[\xi_t\mid\mathcal{F}_{t-1},I_t]=0 E [ ξ t ∣ F t − 1 , I t ] = 0 (Assumption 3.2 (b)) and a j T θ t = x j T ϑ t \mathbf{a}_j^{\mathsf T}\theta_t=\mathbf{x}_j^{\mathsf T}\vartheta_t a j T θ t = x j T ϑ t (Lemma B.6(d)); (c) is Lemma B.2 (a); and (d) is Lemma B.6(d). Hence E [ X t ∣ F t − 1 ] = 0 \mathbb{E}[X_t\mid\mathcal{F}_{t-1}]=0 E [ X t ∣ F t − 1 ] = 0 . For the range,
∣ X t ∣ ≤ ∣ a i T θ t ∣ + ∣ k ~ t ( i , I t ) ∣ ∣ a I t T θ t + ξ t ∣ r t i ≤ ( a ) 1 + 3 r t i ⋅ 2 r t i ≤ ( c ) 7 , |X_t|\leq\frac{|\mathbf{a}_i^{\mathsf T}\theta_t|+|\widetilde{k}_t(i,I_t)|\,|\mathbf{a}_{I_t}^{\mathsf T}\theta_t+\xi_t|}{r_{ti}} \overset{(a)}{\leq} \frac{1+3r_{ti}\cdot2}{r_{ti}} \overset{(c)}{\leq} 7, ∣ X t ∣ ≤ r t i ∣ a i T θ t ∣ + ∣ k t ( i , I t ) ∣ ∣ a I t T θ t + ξ t ∣ ≤ ( a ) r t i 1 + 3 r t i ⋅ 2 ≤ ( c ) 7 ,
where (a) is the triangle inequality; (b) uses ∣ a i T θ t ∣ ≤ 1 |\mathbf{a}_i^{\mathsf T}\theta_t|\leq1 ∣ a i T θ t ∣ ≤ 1 , Lemma B.2 (c) and ∣ a I t T θ t + ξ t ∣ ≤ 2 |\mathbf{a}_{I_t}^{\mathsf T}\theta_t+\xi_t|\leq2 ∣ a I t T θ t + ξ t ∣ ≤ 2 ; and (c) uses r t i ≥ 1 r_{ti}\geq1 r t i ≥ 1 (Lemma B.7(ii)). For the variance,
E [ X t 2 ∣ F t − 1 ] = ( a ) Var ( g ^ t 0 ( i ) ∣ F t − 1 ) r t i 2 ≤ ( b ) E [ g ^ t 0 ( i ) 2 ∣ F t − 1 ] r t i 2 ≤ ( c ) 4 ∑ j p t j k ~ t ( i , j ) 2 r t i 2 = ( d ) 4 v t i r t i = σ t 2 , \mathbb{E}[X_t^2\mid\mathcal{F}_{t-1}]\overset{(a)}{=}\frac{\operatorname{Var}(\widehat{g}_t^0(i)\mid\mathcal{F}_{t-1})}{r_{ti}^2}\overset{(b)}{\leq}\frac{\mathbb{E}[\widehat{g}_t^0(i)^2\mid\mathcal{F}_{t-1}]}{r_{ti}^2}\overset{(c)}{\leq}\frac{4\sum_jp_{tj}\widetilde{k}_t(i,j)^2}{r_{ti}^2}\overset{(d)}{=}\frac{4v_{ti}}{r_{ti}}=\sigma_t^2, E [ X t 2 ∣ F t − 1 ] = ( a ) r t i 2 Var ( g t 0 ( i ) ∣ F t − 1 ) ≤ ( b ) r t i 2 E [ g t 0 ( i ) 2 ∣ F t − 1 ] ≤ ( c ) r t i 2 4 ∑ j p t j k t ( i , j ) 2 = ( d ) r t i 4 v t i = σ t 2 ,
where (a) uses that a i T θ t \mathbf{a}_i^{\mathsf T}\theta_t a i T θ t is the conditional mean of g ^ t 0 ( i ) \widehat{g}_t^0(i) g t 0 ( i ) and that r t i r_{ti} r t i is F t − 1 \mathcal{F}_{t-1} F t − 1 -measurable; (b) bounds the variance by the second moment; (c) conditions on I t I_t I t and uses ( a I t T θ t + ξ t ) 2 ≤ 4 (\mathbf{a}_{I_t}^{\mathsf T}\theta_t+\xi_t)^2\leq4 ( a I t T θ t + ξ t ) 2 ≤ 4 ; and (d) is the definition v t i = r t i − 1 ∑ j p t j k ~ t ( i , j ) 2 v_{ti}=r_{ti}^{-1}\sum_jp_{tj}\widetilde{k}_t(i,j)^2 v t i = r t i − 1 ∑ j p t j k t ( i , j ) 2 .
These properties yield an exponential supermartingale. Since η X t ≤ 7 η ≤ 7 / 12 ≤ 1 \eta X_t\leq7\eta\leq7/12\leq1 η X t ≤ 7 η ≤ 7/12 ≤ 1 ,
E [ e η X t ∣ F t − 1 ] ≤ ( a ) 1 + η E [ X t ∣ F t − 1 ] + η 2 E [ X t 2 ∣ F t − 1 ] = ( b ) 1 + η 2 E [ X t 2 ∣ F t − 1 ] ≤ ( c ) 1 + η 2 σ t 2 ≤ ( d ) e η 2 σ t 2 , \mathbb{E}[e^{\eta X_t}\mid\mathcal{F}_{t-1}]\overset{(a)}{\leq}1+\eta\mathbb{E}[X_t\mid\mathcal{F}_{t-1}]+\eta^2\mathbb{E}[X_t^2\mid\mathcal{F}_{t-1}]\overset{(b)}{=}1+\eta^2\mathbb{E}[X_t^2\mid\mathcal{F}_{t-1}]\overset{(c)}{\leq}1+\eta^2\sigma_t^2\overset{(d)}{\leq}e^{\eta^2\sigma_t^2}, E [ e η X t ∣ F t − 1 ] ≤ ( a ) 1 + η E [ X t ∣ F t − 1 ] + η 2 E [ X t 2 ∣ F t − 1 ] = ( b ) 1 + η 2 E [ X t 2 ∣ F t − 1 ] ≤ ( c ) 1 + η 2 σ t 2 ≤ ( d ) e η 2 σ t 2 ,
where (a) uses e x ≤ 1 + x + x 2 e^x\leq1+x+x^2 e x ≤ 1 + x + x 2 for x ≤ 1 x\leq1 x ≤ 1 ; (b) uses E [ X t ∣ F t − 1 ] = 0 \mathbb{E}[X_t\mid\mathcal{F}_{t-1}]=0 E [ X t ∣ F t − 1 ] = 0 ; (c) is the variance bound; and (d) uses 1 + x ≤ e x 1+x\leq e^x 1 + x ≤ e x . Fix a start τ ∈ [ T ] \tau\in[T] τ ∈ [ T ] and let Z s = exp ( ∑ t = τ s ( η X t − η 2 σ t 2 ) ) Z_s=\exp\left(\sum_{t=\tau}^{s}(\eta X_t-\eta^2\sigma_t^2)\right) Z s = exp ( ∑ t = τ s ( η X t − η 2 σ t 2 ) ) for s ≥ τ − 1 s\geq\tau-1 s ≥ τ − 1 , so that Z τ − 1 = 1 Z_{\tau-1}=1 Z τ − 1 = 1 . Since σ s 2 \sigma_s^2 σ s 2 is F s − 1 \mathcal{F}_{s-1} F s − 1 -measurable, the last display gives E [ Z s ∣ F s − 1 ] = Z s − 1 E [ e η X s ∣ F s − 1 ] e − η 2 σ s 2 ≤ Z s − 1 \mathbb{E}[Z_s\mid\mathcal{F}_{s-1}]=Z_{s-1}\mathbb{E}[e^{\eta X_s}\mid\mathcal{F}_{s-1}]e^{-\eta^2\sigma_s^2}\leq Z_{s-1} E [ Z s ∣ F s − 1 ] = Z s − 1 E [ e η X s ∣ F s − 1 ] e − η 2 σ s 2 ≤ Z s − 1 , so ( Z s ) (Z_s) ( Z s ) is a supermartingale and E [ Z T ] ≤ 1 \mathbb{E}[Z_T]\leq1 E [ Z T ] ≤ 1 . Therefore
Pr ( ∑ t = τ T ( X t − η σ t 2 ) ≥ 1 η log 2 K T δ ) = ( a ) Pr ( Z T ≥ 2 K T δ ) ≤ ( b ) δ 2 K T E [ Z T ] ≤ ( c ) δ 2 K T , \Pr\left(\sum_{t=\tau}^{T}(X_t-\eta\sigma_t^2)\geq\frac{1}{\eta}\log\frac{2KT}{\delta}\right)\overset{(a)}{=}\Pr\left(Z_T\geq\frac{2KT}{\delta}\right)\overset{(b)}{\leq}\frac{\delta}{2KT}\mathbb{E}[Z_T]\overset{(c)}{\leq}\frac{\delta}{2KT}, Pr ( t = τ ∑ T ( X t − η σ t 2 ) ≥ η 1 log δ 2 K T ) = ( a ) Pr ( Z T ≥ δ 2 K T ) ≤ ( b ) 2 K T δ E [ Z T ] ≤ ( c ) 2 K T δ ,
where (a) multiplies the inequality inside the probability by η > 0 \eta>0 η > 0 and exponentiates, (b) is Markov’s inequality, and (c) uses E [ Z T ] ≤ 1 \mathbb{E}[Z_T]\leq1 E [ Z T ] ≤ 1 . A union bound over ( i , τ ) ∈ [ K ] × [ T ] (i,\tau)\in[K]\times[T] ( i , τ ) ∈ [ K ] × [ T ] shows that, with probability at least 1 − δ / 2 1-\delta/2 1 − δ /2 , every suffix sum U τ = ∑ t ≥ τ ( X t − η σ t 2 ) U_\tau=\sum_{t\geq\tau}(X_t-\eta\sigma_t^2) U τ = ∑ t ≥ τ ( X t − η σ t 2 ) satisfies U τ ≤ L / η U_\tau\leq L/\eta U τ ≤ L / η , since log ( 2 K T / δ ) ≤ L \log(2KT/\delta)\leq L log ( 2 K T / δ ) ≤ L .
On this event we return to the scale of the radii by summation by parts. Let U T + 1 = 0 U_{T+1}=0 U T + 1 = 0 , so that X t − η σ t 2 = U t − U t + 1 X_t-\eta\sigma_t^2=U_t-U_{t+1} X t − η σ t 2 = U t − U t + 1 . Then
∑ t = 1 T r t i ( X t − η σ t 2 ) = ( a ) r 1 i U 1 + ∑ t = 2 T ( r t i − r t − 1 , i ) U t \sum_{t=1}^{T}r_{ti}(X_t-\eta\sigma_t^2)\overset{(a)}{=}r_{1i}U_1+\sum_{t=2}^{T}(r_{ti}-r_{t-1,i})U_t t = 1 ∑ T r t i ( X t − η σ t 2 ) = ( a ) r 1 i U 1 + t = 2 ∑ T ( r t i − r t − 1 , i ) U t
≤ ( b ) L η ( r 1 i + ∑ t = 2 T ( r t i − r t − 1 , i ) ) = ( c ) L η r T i . \overset{(b)}{\leq}\frac{L}{\eta}\left(r_{1i}+\sum_{t=2}^{T}(r_{ti}-r_{t-1,i})\right)\overset{(c)}{=}\frac{L}{\eta}r_{Ti}. ≤ ( b ) η L ( r 1 i + t = 2 ∑ T ( r t i − r t − 1 , i ) ) = ( c ) η L r T i .
where (a) substitutes X t − η σ t 2 = U t − U t + 1 X_t-\eta\sigma_t^2=U_t-U_{t+1} X t − η σ t 2 = U t − U t + 1 , regroups the terms by U t U_t U t and uses U T + 1 = 0 U_{T+1}=0 U T + 1 = 0 ; (b) uses U t ≤ L / η U_t\le L/\eta U t ≤ L / η together with r 1 i ≥ 0 r_{1i}\ge0 r 1 i ≥ 0 and r t i − r t − 1 , i ≥ 0 r_{ti}-r_{t-1,i}\ge0 r t i − r t − 1 , i ≥ 0 (Lemma B.7(ii)); and (c) telescopes the sum. Finally, the definitions of X t X_t X t and σ t 2 \sigma_t^2 σ t 2 give r t i ( X t − η σ t 2 ) = a i ⊤ θ t − g ^ t ( i ) − 4 η v t i r_{ti}(X_t-\eta\sigma_t^2)=\mathbf{a}_i^\top\boldsymbol{\theta}_t-\widehat{g}_t^{(i)}-4\eta v_{ti} r t i ( X t − η σ t 2 ) = a i ⊤ θ t − g t ( i ) − 4 η v t i , so the left side equals ∑ t [ a i ⊤ θ t − g ^ t ( i ) − 4 η v t i ] \sum_t[\mathbf{a}_i^\top\boldsymbol{\theta}_t-\widehat{g}_t^{(i)}-4\eta v_{ti}] ∑ t [ a i ⊤ θ t − g t ( i ) − 4 η v t i ] , which proves the lemma for all i ∈ [ K ] i\in[K] i ∈ [ K ] simultaneously.
The Comparator and Learner Error Bounds
To prove Lemma 6.4, we combine Lemma 6.3, the algebraic properties of the normalized kernel (Lemma B.2(b),(c),(e)), and the envelope and monotonicity properties of the radii (Lemma B.7(i),(ii)) with the following concentration bound for the cumulative observation noise ∑ t = 1 T ξ t \sum_{t=1}^{T}\xi_t ∑ t = 1 T ξ t .
Lemma B.8 (Noise). With probability at least 1 − δ / 2 1-\delta/2 1 − δ /2 , ∑ t = 1 T ξ t ≤ 2 T log ( 2 / δ ) \sum_{t=1}^{T}\xi_t\le\sqrt{2T\log(2/\delta)} ∑ t = 1 T ξ t ≤ 2 T log ( 2/ δ ) .
Proof of Lemma B.8. The noise has zero conditional mean given the past:
E [ ξ t ∣ F t − 1 ] = E [ E [ ξ t ∣ F t − 1 , I t ] ∣ F t − 1 ] = 0 , \mathbb{E}[\xi_t\mid\mathcal{F}_{t-1}]=\mathbb{E}\left[\mathbb{E}[\xi_t\mid\mathcal{F}_{t-1},I_t]\mid\mathcal{F}_{t-1}\right]=0, E [ ξ t ∣ F t − 1 ] = E [ E [ ξ t ∣ F t − 1 , I t ] ∣ F t − 1 ] = 0 ,
where the first equality is the tower property and the second uses E [ ξ t ∣ F t − 1 , I t ] = 0 \mathbb{E}[\xi_t\mid\mathcal{F}_{t-1},I_t]=0 E [ ξ t ∣ F t − 1 , I t ] = 0 (Assumption 3.2(b)). Since moreover ∣ ξ t ∣ ≤ 1 |\xi_t|\le1 ∣ ξ t ∣ ≤ 1 and ξ t \xi_t ξ t is F t \mathcal{F}_t F t -measurable, the sequence ( ξ t ) (\xi_t) ( ξ t ) is a martingale difference sequence with increments in [ − 1 , 1 ] [-1,1] [ − 1 , 1 ] , and the Azuma–Hoeffding inequality (Lemma E.1) with δ ′ = δ / 2 \delta'=\delta/2 δ ′ = δ /2 gives the claim.
Proof of Lemma 6.4. At every round t ∈ [ T ] t\in[T] t ∈ [ T ] , the iterate p t ∈ P \mathbf{p}_t\in\mathcal{P} p t ∈ P satisfies p t i ≥ ρ i / ( 2 T ) > 0 p_{ti}\ge\rho_i/(2T)>0 p t i ≥ ρ i / ( 2 T ) > 0 (since ρ i ≥ 1 / ( 2 K ) \rho_i\ge1/(2K) ρ i ≥ 1/ ( 2 K ) by Lemma B.6(a)), and the radii satisfy r t i ≥ 1 r_{ti}\ge1 r t i ≥ 1 by Lemma B.7(ii). Because { z i } i = 1 K \{\mathbf{z}_i\}_{i=1}^{K} { z i } i = 1 K spans R d + 1 \mathbb{R}^{d+1} R d + 1 (Assumption 3.1) and H ( ρ ) ≻ 0 \mathbf{H}(\boldsymbol{\rho})\succ0 H ( ρ ) ≻ 0 (Lemma B.6), the whitened actions x i = H ( ρ ) − 1 / 2 z i \mathbf{x}_i=\mathbf{H}(\boldsymbol{\rho})^{-1/2}\mathbf{z}_i x i = H ( ρ ) − 1/2 z i also span R d + 1 \mathbb{R}^{d+1} R d + 1 , so G t = ∑ i = 1 K ( p t i / r t i ) x i x i ⊤ ≻ 0 \mathbf{G}_t=\sum_{i=1}^{K}(p_{ti}/r_{ti})\mathbf{x}_i\mathbf{x}_i^\top\succ0 G t = ∑ i = 1 K ( p t i / r t i ) x i x i ⊤ ≻ 0 . Moreover, x i ⊤ G t − 1 x i ≤ r t i 2 \mathbf{x}_i^\top\mathbf{G}_t^{-1}\mathbf{x}_i\le r_{ti}^2 x i ⊤ G t − 1 x i ≤ r t i 2 holds for all i ∈ [ K ] i\in[K] i ∈ [ K ] by Lemma B.7(i). Consequently, ( p t , r t , G t ) (\mathbf{p}_t,\mathbf{r}_t,\mathbf{G}_t) ( p t , r t , G t ) satisfies all hypotheses of Lemma B.2 at every round t ∈ [ T ] t\in[T] t ∈ [ T ] .
Let E c o n c \mathcal{E}_{\mathrm{conc}} E conc denote the event of Lemma 6.3, on which
∑ t = 1 T [ a i ⊤ θ t − g ^ t ( i ) − 4 η v t i ] ≤ L η r T i for all i ∈ [ K ] , (12) \sum_{t=1}^{T}\left[\mathbf{a}_i^\top\boldsymbol{\theta}_t-\widehat{g}_t^{(i)}-4\eta v_{ti}\right]\le\frac{L}{\eta}r_{Ti}\qquad\text{for all }i\in[K], \tag*{(12)} t = 1 ∑ T [ a i ⊤ θ t − g t ( i ) − 4 η v t i ] ≤ η L r T i for all i ∈ [ K ] , (12)
and let E n o i s e \mathcal{E}_{\mathrm{noise}} E noise denote the event of Lemma B.8, on which
∑ t = 1 T ξ t ≤ 2 T log ( 2 / δ ) . (13) \sum_{t=1}^{T}\xi_t\le\sqrt{2T\log(2/\delta)}. \tag*{(13)} t = 1 ∑ T ξ t ≤ 2 T log ( 2/ δ ) . (13)
By Lemma 6.3 and Lemma B.8, P ( E c o n c ) ≥ 1 − δ / 2 \mathbb{P}(\mathcal{E}_{\mathrm{conc}})\ge1-\delta/2 P ( E conc ) ≥ 1 − δ /2 and P ( E n o i s e ) ≥ 1 − δ / 2 \mathbb{P}(\mathcal{E}_{\mathrm{noise}})\ge1-\delta/2 P ( E noise ) ≥ 1 − δ /2 . By the union bound, the intersection E : = E c o n c ∩ E n o i s e \mathcal{E}:=\mathcal{E}_{\mathrm{conc}}\cap\mathcal{E}_{\mathrm{noise}} E := E conc ∩ E noise occurs with probability at least 1 − δ 1-\delta 1 − δ . Note that E \mathcal{E} E depends only on the algorithm’s trajectory and not on the choice of comparator u ∈ P \mathbf{u}\in\mathcal{P} u ∈ P .
To bound the comparator error T 2 T_2 T 2 , note that by the reward model y t = a I t ⊤ θ t + ξ t + c t y_t=\mathbf{a}_{I_t}^{\top}\boldsymbol{\theta}_t+\xi_t+c_t y t = a I t ⊤ θ t + ξ t + c t (Assumption 3.2(c)) and the definition g ^ t ( i ) = k ~ t ( i , I t ) y t + 4 η v t i \widehat{g}_t^{(i)}=\widetilde{k}_t(i,I_t)y_t+4\eta v_{ti} g t ( i ) = k t ( i , I t ) y t + 4 η v t i , each coordinate of g ^ t \widehat{\mathbf{g}}_t g t decomposes as
g ^ t ( i ) = g ^ t 0 ( i ) + c t k ~ t ( i , I t ) + 4 η v t i , where g ^ t 0 ( i ) = k ~ t ( i , I t ) ( a I t ⊤ θ t + ξ t ) . (14) \widehat{g}_t(i)=\widehat{g}_t^0(i)+c_t\widetilde{k}_t(i,I_t)+4\eta v_{ti},\qquad\text{where}\qquad\widehat{g}_t^0(i)=\widetilde{k}_t(i,I_t)(\mathbf{a}_{I_t}^{\top}\boldsymbol{\theta}_t+\xi_t). \tag*{(14)} g t ( i ) = g t 0 ( i ) + c t k t ( i , I t ) + 4 η v t i , where g t 0 ( i ) = k t ( i , I t ) ( a I t ⊤ θ t + ξ t ) . (14)
Substituting (B.5) into the definition of T 2 T_2 T 2 gives, for any comparator u ∈ P \mathbf{u} \in\mathcal{P} u ∈ P ,
T 2 = ∑ t = 1 T ( ∑ i = 1 K u i a i T θ t − ⟨ u , g ^ t ⟩ ) = ∑ i = 1 K u i ∑ t = 1 T [ a i T θ t − g ^ t 0 ( i ) − 4 η v t i ] − ∑ t = 1 T c t ∑ i = 1 K u i k ~ t ( i , I t ) . \begin{aligned}
T_2=\sum_{t=1}^{T}\left(\sum_{i=1}^{K}u_i\mathbf{a}_{i}^{\mathsf{T}}\boldsymbol{\theta}_t-\langle\mathbf{u},\widehat{\mathbf{g}}_t\rangle\right) \\
&=\sum_{i=1}^{K}u_i\sum_{t=1}^{T}\left[\mathbf{a}_{i}^{\mathsf{T}}\boldsymbol{\theta}_t-\widehat{g}_t^0(i)-4\eta v_{ti}\right]-\sum_{t=1}^{T}c_t\sum_{i=1}^{K}u_i\widetilde{k}_t(i,I_t).
\end{aligned} T 2 = t = 1 ∑ T ( i = 1 ∑ K u i a i T θ t − ⟨ u , g t ⟩ ) = i = 1 ∑ K u i t = 1 ∑ T [ a i T θ t − g t 0 ( i ) − 4 η v t i ] − t = 1 ∑ T c t i = 1 ∑ K u i k t ( i , I t ) .
On the event E \mathcal{E} E , since u i ≥ 0 u_i\ge0 u i ≥ 0 for all i ∈ [ K ] i\in[K] i ∈ [ K ] , multiplying (B.3) by u i u_i u i and summing over i ∈ [ K ] i\in[K] i ∈ [ K ] bounds the first sum by ( L / η ) ∑ i = 1 K u i r T i = ( L / η ) ⟨ u , r T ⟩ (L/\eta)\sum_{i=1}^{K}u_ir_{Ti}=(L/\eta)\langle\mathbf{u},\mathbf{r}_T\rangle ( L / η ) ∑ i = 1 K u i r T i = ( L / η ) ⟨ u , r T ⟩ . For the second sum,
− ∑ t = 1 T c t ∑ i = 1 K u i k ~ t ( i , I t ) ≤ ( a ) ∑ t = 1 T ∣ c t ∣ ∑ i = 1 K u i ∣ k ~ t ( i , I t ) ∣ ≤ ( b ) 3 ∑ t = 1 T ∣ c t ∣ ∑ i = 1 K u i r t i -\sum_{t=1}^{T}c_t\sum_{i=1}^{K}u_i\widetilde{k}_t(i,I_t)
\mathrel{\leq}^{(a)}\sum_{t=1}^{T}|c_t|\sum_{i=1}^{K}u_i\left|\widetilde{k}_t(i,I_t)\right|
\mathrel{\leq}^{(b)}3\sum_{t=1}^{T}|c_t|\sum_{i=1}^{K}u_ir_{ti} − t = 1 ∑ T c t i = 1 ∑ K u i k t ( i , I t ) ≤ ( a ) t = 1 ∑ T ∣ c t ∣ i = 1 ∑ K u i k t ( i , I t ) ≤ ( b ) 3 t = 1 ∑ T ∣ c t ∣ i = 1 ∑ K u i r t i
≤ ( c ) 3 ∑ t = 1 T ∣ c t ∣ ∑ i = 1 K u i r T i = ( d ) 3 ⟨ u , r T ⟩ ∑ t = 1 T ∣ c t ∣ ≤ ( e ) 3 C ⟨ u , r T ⟩ , \mathrel{\leq}^{(c)}3\sum_{t=1}^{T}|c_t|\sum_{i=1}^{K}u_ir_{Ti}
\mathrel{=}^{(d)}3\langle\mathbf{u},\mathbf{r}_T\rangle\sum_{t=1}^{T}|c_t|
\mathrel{\leq}^{(e)}3C\langle\mathbf{u},\mathbf{r}_T\rangle, ≤ ( c ) 3 t = 1 ∑ T ∣ c t ∣ i = 1 ∑ K u i r T i = ( d ) 3 ⟨ u , r T ⟩ t = 1 ∑ T ∣ c t ∣ ≤ ( e ) 3 C ⟨ u , r T ⟩ ,
where (a) uses the triangle inequality and u i ≥ 0 u_i\ge0 u i ≥ 0 ; (b) uses ∣ k ~ t ( i , I t ) ∣ ≤ 3 r t i \left|\widetilde{k}_t(i,I_t)\right|\le3r_{ti} k t ( i , I t ) ≤ 3 r t i from Lemma B.2 (c) and u i ≥ 0 u_i\ge0 u i ≥ 0 ; (c) uses the monotonicity of the radii r t i ≤ r T i r_{ti}\le r_{Ti} r t i ≤ r T i from Lemma B.7(ii) together with u i ≥ 0 u_i\ge0 u i ≥ 0 and ∣ c t ∣ ≥ 0 |c_t|\ge0 ∣ c t ∣ ≥ 0 ; (d) factors out ⟨ u , r T ⟩ = ∑ i = 1 K u i r T i \langle\mathbf{u},\mathbf{r}_T\rangle=\sum_{i=1}^{K}u_ir_{Ti} ⟨ u , r T ⟩ = ∑ i = 1 K u i r T i ; and (e) uses ∑ t = 1 T ∣ c t ∣ ≤ C \sum_{t=1}^{T}|c_t|\le C ∑ t = 1 T ∣ c t ∣ ≤ C (Assumption 3.2(c)) together with ⟨ u , r T ⟩ ≥ 0 \langle\mathbf{u},\mathbf{r}_T\rangle\ge0 ⟨ u , r T ⟩ ≥ 0 . Combining the two bounds gives T 2 ≤ ( L / η + 3 C ) ⟨ u , r T ⟩ T_2\le(L/\eta+3C)\langle\mathbf{u},\mathbf{r}_T\rangle T 2 ≤ ( L / η + 3 C ) ⟨ u , r T ⟩ .
Finally, to bound the learner error T 3 T_3 T 3 , taking the inner product of p t \mathbf{p}_t p t with g ^ t ( i ) = k ~ t ( i , I t ) y t + 4 η v t i \widehat{g}_t(i)=\widetilde{k}_t(i,I_t)y_t+4\eta v_{ti} g t ( i ) = k t ( i , I t ) y t + 4 η v t i at each round t ∈ [ T ] t\in[T] t ∈ [ T ] gives
⟨ p t , g ^ t ⟩ = ( ∑ i = 1 K p t i k ~ t ( i , I t ) ) y t + 4 η ⟨ p t , v t ⟩ = ( a ) y t + 4 η ⟨ p t , v t ⟩ = ( b ) a I t T θ t + ξ t + c t + 4 η ⟨ p t , v t ⟩ , \langle\mathbf{p}_t,\widehat{\mathbf{g}}_t\rangle
=\left(\sum_{i=1}^{K}p_{ti}\widetilde{k}_t(i,I_t)\right)y_t+4\eta\langle\mathbf{p}_t,\mathbf{v}_t\rangle
\mathrel{=}^{(a)}y_t+4\eta\langle\mathbf{p}_t,\mathbf{v}_t\rangle
\mathrel{=}^{(b)}\mathbf{a}_{I_t}^{\mathsf{T}}\boldsymbol{\theta}_t+\xi_t+c_t+4\eta\langle\mathbf{p}_t,\mathbf{v}_t\rangle, ⟨ p t , g t ⟩ = ( i = 1 ∑ K p t i k t ( i , I t ) ) y t + 4 η ⟨ p t , v t ⟩ = ( a ) y t + 4 η ⟨ p t , v t ⟩ = ( b ) a I t T θ t + ξ t + c t + 4 η ⟨ p t , v t ⟩ ,
where (a) uses the unit column-sum property ∑ i = 1 K p t i k ~ t ( i , j ) = 1 \sum_{i=1}^{K}p_{ti}\widetilde{k}_t(i,j)=1 ∑ i = 1 K p t i k t ( i , j ) = 1 for j = I t j=I_t j = I t from Lemma B.2 (b), and (b) substitutes y t = a I t T θ t + ξ t + c t y_t=\mathbf{a}_{I_t}^{\mathsf{T}}\boldsymbol{\theta}_t+\xi_t+c_t y t = a I t T θ t + ξ t + c t . Subtracting a I t T θ t \mathbf{a}_{I_t}^{\mathsf{T}}\boldsymbol{\theta}_t a I t T θ t and summing over t = 1 , … , T t=1,\ldots,T t = 1 , … , T , we obtain on the event E \mathcal{E} E :
T 3 = ∑ t = 1 T ( ⟨ p t , g ^ t ⟩ − a I t T θ t ) = ∑ t = 1 T ξ t + ∑ t = 1 T c t + 4 η ∑ t = 1 T ⟨ p t , v t ⟩ T_3=\sum_{t=1}^{T}\left(\langle\mathbf{p}_t,\widehat{\mathbf{g}}_t\rangle-\mathbf{a}_{I_t}^{\mathsf{T}}\boldsymbol{\theta}_t\right)
=\sum_{t=1}^{T}\xi_t+\sum_{t=1}^{T}c_t+4\eta\sum_{t=1}^{T}\langle\mathbf{p}_t,\mathbf{v}_t\rangle T 3 = t = 1 ∑ T ( ⟨ p t , g t ⟩ − a I t T θ t ) = t = 1 ∑ T ξ t + t = 1 ∑ T c t + 4 η t = 1 ∑ T ⟨ p t , v t ⟩
≤ ( a ) ∑ t = 1 T ξ t + C + 8 η T ≤ ( b ) 2 T log ( 2 / δ ) + C + 8 η T , \mathrel{\leq}^{(a)}\sum_{t=1}^{T}\xi_t+C+8\eta T
\mathrel{\leq}^{(b)}\sqrt{2T\log(2/\delta)}+C+8\eta T, ≤ ( a ) t = 1 ∑ T ξ t + C + 8 η T ≤ ( b ) 2 T log ( 2/ δ ) + C + 8 η T ,
where (a) uses ∑ t = 1 T c t ≤ ∑ t = 1 T ∣ c t ∣ ≤ C \sum_{t=1}^{T}c_t\le\sum_{t=1}^{T}|c_t|\le C ∑ t = 1 T c t ≤ ∑ t = 1 T ∣ c t ∣ ≤ C (Assumption 3.2(c)) and ⟨ p t , v t ⟩ = ∑ i = 1 K p t i v t i ≤ 2 \langle\mathbf{p}_t,\mathbf{v}_t\rangle=\sum_{i=1}^{K}p_{ti}v_{ti}\le2 ⟨ p t , v t ⟩ = ∑ i = 1 K p t i v t i ≤ 2 (Lemma B.2 (e)), and (b) uses (B.4). This completes the proof of Lemma 6.4 .
Proofs of the Supporting Lemmas in Appendix B
FTRL and Regularizer Properties
To prove Lemma B.1 , we use the following two lemmas on the inner minimization problem in line 4 and the outer maximization problem in line 3 of Algorithm 1 (both proved in Appendix § 12 ): Lemma C.1 and Lemma C.2(a) ensure that p t \mathbf{p}_t p t and Q t \mathbf{Q}_t Q t are well-defined continuous functions of ( S t − 1 , Q t − 1 ) (\mathbf{S}_{t-1},\mathbf{Q}_{t-1}) ( S t − 1 , Q t − 1 ) (giving F t − 1 \mathcal{F}_{t-1} F t − 1 -measurability by induction), while Lemma C.2(b) shows that p t \mathbf{p}_t p t also maximizes the FTRL objective with Q \mathbf{Q} Q frozen at Q t \mathbf{Q}_t Q t .
Lemma C.1 (Inner Problem). For every D ⪰ I \mathbf{D} \succeq\mathbf{I} D ⪰ I and p ∈ P \mathbf{p} \in\mathcal{P} p ∈ P , the problem min Q ⪰ D J ( Q , p ) \min_{\mathbf{Q}\succeq\mathbf{D}} J(\mathbf{Q},\mathbf{p}) min Q ⪰ D J ( Q , p ) has a unique minimizer, denoted Q ( D , p ) \mathbf{Q}(\mathbf{D},\mathbf{p}) Q ( D , p ) . The map ( D , p ) ↦ Q ( D , p ) (\mathbf{D},\mathbf{p}) \mapsto\mathbf{Q}(\mathbf{D},\mathbf{p}) ( D , p ) ↦ Q ( D , p ) is continuous on { D ⪰ I } × P \{\mathbf{D} \succeq\mathbf{I}\} \times\mathcal{P} { D ⪰ I } × P .
Lemma C.2 (Outer Problem). Fix S ∈ R K \mathbf{S} \in\mathbb{R}^{K} S ∈ R K and D ⪰ I \mathbf{D} \succeq\mathbf{I} D ⪰ I .
(a) The function p ↦ ⟨ S , p ⟩ + min Q ⪰ D J ( Q , p ) \mathbf{p} \mapsto\langle\mathbf{S},\mathbf{p}\rangle+ \min_{\mathbf{Q}\succeq\mathbf{D}} J(\mathbf{Q},\mathbf{p}) p ↦ ⟨ S , p ⟩ + min Q ⪰ D J ( Q , p ) has a unique maximizer p ( S , D ) \mathbf{p}(\mathbf{S},\mathbf{D}) p ( S , D ) over P \mathcal{P} P , and the map ( S , D ) ↦ p ( S , D ) (\mathbf{S},\mathbf{D}) \mapsto\mathbf{p}(\mathbf{S},\mathbf{D}) ( S , D ) ↦ p ( S , D ) is continuous.
(b) Let p ⋆ = p ( S , D ) \mathbf{p}^{\star} = \mathbf{p}(\mathbf{S},\mathbf{D}) p ⋆ = p ( S , D ) and Q ⋆ = Q ( D , p ⋆ ) \mathbf{Q}^{\star} = \mathbf{Q}(\mathbf{D},\mathbf{p}^{\star}) Q ⋆ = Q ( D , p ⋆ ) . Then p ⋆ \mathbf{p}^{\star} p ⋆ is also the unique maximizer of ⟨ S , p ⟩ + ∑ i r i ( Q ⋆ ) h ( p i ) \langle\mathbf{S},\mathbf{p}\rangle+ \sum_i r_i(\mathbf{Q}^{\star})h(p_i) ⟨ S , p ⟩ + ∑ i r i ( Q ⋆ ) h ( p i ) over P \mathcal{P} P .
Proof of Lemma B.1. We argue by induction on t t t , starting from the deterministic S 0 = 0 \mathbf{S}_0 = \mathbf{0} S 0 = 0 and Q 0 = I \mathbf{Q}_0 = \mathbf{I} Q 0 = I . Suppose that S t − 1 \mathbf{S}_{t-1} S t − 1 and Q t − 1 \mathbf{Q}_{t-1} Q t − 1 are F t − 1 \mathcal{F}_{t-1} F t − 1 -measurable. By Lemma C.1 and Lemma C.2(a), p t = p ( S t − 1 , Q t − 1 ) \mathbf{p}_t = \mathbf{p}(\mathbf{S}_{t-1},\mathbf{Q}_{t-1}) p t = p ( S t − 1 , Q t − 1 ) and Q t = Q ( Q t − 1 , p t ) \mathbf{Q}_t = \mathbf{Q}(\mathbf{Q}_{t-1},\mathbf{p}_t) Q t = Q ( Q t − 1 , p t ) are continuous functions of ( S t − 1 , Q t − 1 ) (\mathbf{S}_{t-1},\mathbf{Q}_{t-1}) ( S t − 1 , Q t − 1 ) , and hence are F t − 1 \mathcal{F}_{t-1} F t − 1 -measurable. The radii r t r_t r t , the matrix G t \mathbf{G}_t G t , the kernel k t \mathbf{k}_t k t and the variance proxies v t v_t v t are continuous functions of ( p t , Q t ) (\mathbf{p}_t,\mathbf{Q}_t) ( p t , Q t ) , so they are F t − 1 \mathcal{F}_{t-1} F t − 1 -measurable as well. The estimate g ^ t \widehat{\mathbf{g}}_t g t is a function of these quantities and of ( I t , y t ) (I_t,y_t) ( I t , y t ) , so it is F t \mathcal{F}_t F t -measurable, and so is S t = S t − 1 + g ^ t \mathbf{S}_t = \mathbf{S}_{t-1} + \widehat{\mathbf{g}}_t S t = S t − 1 + g t , which completes the induction. For the frozen form, we apply Lemma C.2(b) with S = S t − 1 \mathbf{S} = \mathbf{S}_{t-1} S = S t − 1 and D = Q t − 1 \mathbf{D} = \mathbf{Q}_{t-1} D = Q t − 1 . Then p ⋆ = p t \mathbf{p}^{\star} = \mathbf{p}_t p ⋆ = p t and Q ⋆ = Q ( Q t − 1 , p t ) = Q t \mathbf{Q}^{\star} = \mathbf{Q}(\mathbf{Q}_{t-1},\mathbf{p}_t) = \mathbf{Q}_t Q ⋆ = Q ( Q t − 1 , p t ) = Q t , so p t \mathbf{p}_t p t is the unique maximizer over P \mathcal{P} P of ⟨ S t − 1 , p ⟩ + ∑ i r t i h ( p i ) \langle\mathbf{S}_{t-1},\mathbf{p}\rangle+ \sum_i r_{t i} h(p_i) ⟨ S t − 1 , p ⟩ + ∑ i r t i h ( p i ) , which equals ⟨ S t − 1 , p ⟩ − ψ t ( p ) \langle\mathbf{S}_{t-1},\mathbf{p}\rangle- \psi_t(\mathbf{p}) ⟨ S t − 1 , p ⟩ − ψ t ( p ) by the definition of ψ t \psi_t ψ t .
Proof of Lemma B.3. The proof bounds the left side by a Bregman divergence and then maximizes coordinatewise. On the positive orthant, ψ \psi ψ is differentiable and convex, with ∂ i ψ ( p ) = r i ( η − 1 ( 1 + log p i ) − α ) \partial_i\psi(\mathbf{p}) = r_i\left(\eta^{-1}(1+\log p_i)-\alpha\right) ∂ i ψ ( p ) = r i ( η − 1 ( 1 + log p i ) − α ) . Its Bregman divergence is
B ( q , p ) : = ψ ( q ) − ψ ( p ) − ⟨ ∇ ψ ( p ) , q − p ⟩ = η − 1 ∑ i r i ( q i log ( q i / p i ) − q i + p i ) , B(\mathbf{q},\mathbf{p}) := \psi(\mathbf{q})-\psi(\mathbf{p})-\langle\nabla\psi(\mathbf{p}),\mathbf{q}-\mathbf{p}\rangle= \eta^{-1}\sum_i r_i\left(q_i\log(q_i/p_i)-q_i+p_i\right), B ( q , p ) := ψ ( q ) − ψ ( p ) − ⟨ ∇ ψ ( p ) , q − p ⟩ = η − 1 i ∑ r i ( q i log ( q i / p i ) − q i + p i ) ,
where the equality follows by substituting ∂ i ψ \partial_i\psi ∂ i ψ ; the linear term − α ∑ i r i p i -\alpha\sum_i r_i p_i − α ∑ i r i p i of ψ \psi ψ cancels. Since Λ \Lambda Λ is a linear function minus ψ \psi ψ ,
Λ ( q ) − Λ ( p ⋆ ) = ⟨ ∇ Λ ( p ⋆ ) , q − p ⋆ ⟩ − B ( q , p ⋆ ) ≤ − B ( q , p ⋆ ) , \Lambda(\mathbf{q})-\Lambda(\mathbf{p}^{\star})=\langle\nabla\Lambda(\mathbf{p}^{\star}),\mathbf{q}-\mathbf{p}^{\star}\rangle-B(\mathbf{q},\mathbf{p}^{\star})\le-B(\mathbf{q},\mathbf{p}^{\star}), Λ ( q ) − Λ ( p ⋆ ) = ⟨ ∇Λ ( p ⋆ ) , q − p ⋆ ⟩ − B ( q , p ⋆ ) ≤ − B ( q , p ⋆ ) ,
where the equality uses Λ = ⟨ S , ⋅ ⟩ − ψ \Lambda=\langle\mathbf{S},\cdot\rangle-\psi Λ = ⟨ S , ⋅ ⟩ − ψ and the definition of B B B , and the inequality is the first-order optimality of p ⋆ \mathbf{p}^{\star} p ⋆ on the convex set P \mathcal{P} P , which gives ⟨ ∇ Λ ( p ⋆ ) , q − p ⋆ ⟩ ≤ 0 \langle\nabla\Lambda(\mathbf{p}^{\star}),\mathbf{q}-\mathbf{p}^{\star}\rangle\le0 ⟨ ∇Λ ( p ⋆ ) , q − p ⋆ ⟩ ≤ 0 . For s > 0 s>0 s > 0 , let
φ i ( s ) = g i ( s − p i ⋆ ) − r i η ( s log ( s / p i ⋆ ) − s + p i ⋆ ) \varphi_i(s)=g_i(s-p_i^{\star})-\frac{r_i}{\eta}\left(s\log(s/p_i^{\star})-s+p_i^{\star}\right) φ i ( s ) = g i ( s − p i ⋆ ) − η r i ( s log ( s / p i ⋆ ) − s + p i ⋆ )
and u i = η g i / r i u_i=\eta g_i/r_i u i = η g i / r i . Then
Λ ( q ) − Λ ( p ⋆ ) + ⟨ g , q − p ⋆ ⟩ ≤ ( a ) ∑ i φ i ( q i ) ≤ ( b ) ∑ i sup s > 0 φ i ( s ) = ( c ) ∑ i r i p i ⋆ η ( e u i − 1 − u i ) ≤ ( d ) ∑ i r i p i ⋆ η u i 2 = ( e ) η ∑ i p i ⋆ g i 2 r i , \begin{aligned}
\Lambda(\mathbf{q})-\Lambda(\mathbf{p}^{\star})+\langle\mathbf{g},\mathbf{q}-\mathbf{p}^{\star}\rangle
&\overset{(a)}{\le}\sum_i\varphi_i(q_i)
\overset{(b)}{\le}\sum_i\sup_{s>0}\varphi_i(s)\\
&\overset{(c)}{=}\sum_i\frac{r_i p_i^{\star}}{\eta}\left(e^{u_i}-1-u_i\right)\\
&\overset{(d)}{\le}\sum_i\frac{r_i p_i^{\star}}{\eta}u_i^2
\overset{(e)}{=}\eta\sum_i p_i^{\star}\frac{g_i^2}{r_i},
\end{aligned} Λ ( q ) − Λ ( p ⋆ ) + ⟨ g , q − p ⋆ ⟩ ≤ ( a ) i ∑ φ i ( q i ) ≤ ( b ) i ∑ s > 0 sup φ i ( s ) = ( c ) i ∑ η r i p i ⋆ ( e u i − 1 − u i ) ≤ ( d ) i ∑ η r i p i ⋆ u i 2 = ( e ) η i ∑ p i ⋆ r i g i 2 ,
where (a) is the previous display; (b) maximizes each coordinate separately; (c) uses that φ i \varphi_i φ i is concave with φ i ′ ( s ) = g i − r i η log ( s / p i ⋆ ) \varphi_i'(s)=g_i-\frac{r_i}{\eta}\log(s/p_i^{\star}) φ i ′ ( s ) = g i − η r i log ( s / p i ⋆ ) , so that its maximum is attained at s = p i ⋆ e u i s=p_i^{\star}e^{u_i} s = p i ⋆ e u i , where φ i = g i p i ⋆ ( e u i − 1 ) − r i p i ⋆ η ( u i e u i − e u i + 1 ) = r i p i ⋆ η ( e u i − 1 − u i ) \varphi_i=g_i p_i^{\star}(e^{u_i}-1)-\frac{r_i p_i^{\star}}{\eta}(u_i e^{u_i}-e^{u_i}+1)=\frac{r_i p_i^{\star}}{\eta}(e^{u_i}-1-u_i) φ i = g i p i ⋆ ( e u i − 1 ) − η r i p i ⋆ ( u i e u i − e u i + 1 ) = η r i p i ⋆ ( e u i − 1 − u i ) by g i = r i u i / η g_i=r_i u_i/\eta g i = r i u i / η ; (d) uses u i ≤ 1 u_i\le1 u i ≤ 1 , which is the hypothesis η g i ≤ r i \eta g_i\le r_i η g i ≤ r i , and e u − 1 − u ≤ u 2 e^u-1-u\le u^2 e u − 1 − u ≤ u 2 for u ≤ 1 u\le1 u ≤ 1 ; and (e) substitutes u i = η g i / r i u_i=\eta g_i/r_i u i = η g i / r i .
Proof of Lemma B.4. The proof compares the value of the FTRL objective at the comparator with its values along the iterates. First,
⟨ S T , u ⟩ = Λ T + 1 ( u ) + ψ T + 1 ( u ) ≤ Λ T + 1 ( p T + 1 ) + ψ T + 1 ( u ) , \langle S_T,u\rangle=\Lambda_{T+1}(u)+\psi_{T+1}(u)\leq\Lambda_{T+1}(p_{T+1})+\psi_{T+1}(u), ⟨ S T , u ⟩ = Λ T + 1 ( u ) + ψ T + 1 ( u ) ≤ Λ T + 1 ( p T + 1 ) + ψ T + 1 ( u ) ,
where the equality is the definition of Λ T + 1 \Lambda_{T+1} Λ T + 1 and the inequality uses that p T + 1 p_{T+1} p T + 1 maximizes Λ T + 1 \Lambda_{T+1} Λ T + 1 over P \mathcal{P} P . To express Λ T + 1 ( p T + 1 ) \Lambda_{T+1}(p_{T+1}) Λ T + 1 ( p T + 1 ) through the iterates, note that for each t t t ,
Λ t + 1 ( p t + 1 ) = ⟨ S t − 1 , p t + 1 ⟩ + ⟨ g t , p t + 1 ⟩ − ψ t + 1 ( p t + 1 ) = Λ t ( p t + 1 ) + ⟨ g t , p t + 1 ⟩ + [ ψ t ( p t + 1 ) − ψ t + 1 ( p t + 1 ) ] . \begin{aligned}
\Lambda_{t+1}(p_{t+1})&=\langle S_{t-1},p_{t+1}\rangle+\langle g_t,p_{t+1}\rangle-\psi_{t+1}(p_{t+1})\\
&=\Lambda_t(p_{t+1})+\langle g_t,p_{t+1}\rangle+\left[\psi_t(p_{t+1})-\psi_{t+1}(p_{t+1})\right].
\end{aligned} Λ t + 1 ( p t + 1 ) = ⟨ S t − 1 , p t + 1 ⟩ + ⟨ g t , p t + 1 ⟩ − ψ t + 1 ( p t + 1 ) = Λ t ( p t + 1 ) + ⟨ g t , p t + 1 ⟩ + [ ψ t ( p t + 1 ) − ψ t + 1 ( p t + 1 ) ] .
where the first equality uses the definition of Λ t + 1 \Lambda_{t+1} Λ t + 1 and S t = S t − 1 + g t S_t=S_{t-1}+g_t S t = S t − 1 + g t , and the second adds and subtracts ψ t ( p t + 1 ) \psi_t(p_{t+1}) ψ t ( p t + 1 ) and uses the definition of Λ t \Lambda_t Λ t . Therefore
Λ T + 1 ( p T + 1 ) = Λ 1 ( p 1 ) + ∑ t = 1 T [ Λ t + 1 ( p t + 1 ) − Λ t ( p t ) ] . \Lambda_{T+1}(p_{T+1})=\Lambda_1(p_1)+\sum_{t=1}^{T}\left[\Lambda_{t+1}(p_{t+1})-\Lambda_t(p_t)\right]. Λ T + 1 ( p T + 1 ) = Λ 1 ( p 1 ) + t = 1 ∑ T [ Λ t + 1 ( p t + 1 ) − Λ t ( p t ) ] .
Λ T + 1 ( p T + 1 ) = − ψ 1 ( p 1 ) + ∑ t = 1 T [ Λ t ( p t + 1 ) − Λ t ( p t ) + ⟨ g t , p t + 1 ⟩ ] + ∑ t = 1 T [ ψ t ( p t + 1 ) − ψ t + 1 ( p t + 1 ) ] , \Lambda_{T+1}(p_{T+1})=-\psi_1(p_1)+\sum_{t=1}^{T}\left[\Lambda_t(p_{t+1})-\Lambda_t(p_t)+\langle g_t,p_{t+1}\rangle\right]+\sum_{t=1}^{T}\left[\psi_t(p_{t+1})-\psi_{t+1}(p_{t+1})\right], Λ T + 1 ( p T + 1 ) = − ψ 1 ( p 1 ) + t = 1 ∑ T [ Λ t ( p t + 1 ) − Λ t ( p t ) + ⟨ g t , p t + 1 ⟩ ] + t = 1 ∑ T [ ψ t ( p t + 1 ) − ψ t + 1 ( p t + 1 ) ] ,
where the first equality telescopes the sum, and the second uses Λ 1 ( p 1 ) = − ψ 1 ( p 1 ) \Lambda_1(p_1)=-\psi_1(p_1) Λ 1 ( p 1 ) = − ψ 1 ( p 1 ) , which holds because S 0 = 0 S_0=0 S 0 = 0 , together with the previous display. Finally,
∑ t = 1 T ⟨ g t , u − p t ⟩ = ⟨ S T , u ⟩ − ∑ t = 1 T ⟨ g t , p t ⟩ ≤ Λ T + 1 ( p T + 1 ) + ψ T + 1 ( u ) − ∑ t = 1 T ⟨ g t , p t ⟩ , \sum_{t=1}^{T}\langle g_t,u-p_t\rangle=\langle S_T,u\rangle-\sum_{t=1}^{T}\langle g_t,p_t\rangle\leq\Lambda_{T+1}(p_{T+1})+\psi_{T+1}(u)-\sum_{t=1}^{T}\langle g_t,p_t\rangle, t = 1 ∑ T ⟨ g t , u − p t ⟩ = ⟨ S T , u ⟩ − t = 1 ∑ T ⟨ g t , p t ⟩ ≤ Λ T + 1 ( p T + 1 ) + ψ T + 1 ( u ) − t = 1 ∑ T ⟨ g t , p t ⟩ ,
where the equality uses S T = ∑ t = 1 T g t S_T=\sum_{t=1}^{T}g_t S T = ∑ t = 1 T g t and the inequality is the first display. Substituting the expression for Λ T + 1 ( p T + 1 ) \Lambda_{T+1}(p_{T+1}) Λ T + 1 ( p T + 1 ) and combining ⟨ g t , p t + 1 ⟩ − ⟨ g t , p t ⟩ = ⟨ g t , p t + 1 − p t ⟩ \langle g_t,p_{t+1}\rangle-\langle g_t,p_t\rangle=\langle g_t,p_{t+1}-p_t\rangle ⟨ g t , p t + 1 ⟩ − ⟨ g t , p t ⟩ = ⟨ g t , p t + 1 − p t ⟩ gives the claim. □
Proof of Lemma B.5. By the definition of h h h , the ratio h ( p i ) / p i h(p_i)/p_i h ( p i ) / p i is α \alpha α plus a logarithmic term, which we bound from both sides:
α ≤ α + 1 η log 1 p i = (b) h ( p i ) η p i ≤ (c) α + log ( 4 K T ) η = (d) κ , \alpha\leq\alpha+\frac{1}{\eta}\log\frac{1}{p_i}\overset{\text{(b)}}{=}\frac{h(p_i)}{\eta p_i}\overset{\text{(c)}}{\leq}\alpha+\frac{\log(4KT)}{\eta}\overset{\text{(d)}}{=}\kappa, α ≤ α + η 1 log p i 1 = (b) η p i h ( p i ) ≤ (c) α + η log ( 4 K T ) = (d) κ ,
where (a) uses p i ≤ 1 p_i\leq1 p i ≤ 1 , so that log ( 1 / p i ) ≥ 0 \log(1/p_i)\geq0 log ( 1/ p i ) ≥ 0 ; (b) is the definition h ( s ) = s ( α − η − 1 log s ) h(s)=s(\alpha-\eta^{-1}\log s) h ( s ) = s ( α − η − 1 log s ) ; (c) uses p i ≥ ρ i / ( 2 T ) ≥ 1 / ( 4 K T ) p_i\geq\rho_i/(2T)\geq1/(4KT) p i ≥ ρ i / ( 2 T ) ≥ 1/ ( 4 K T ) , which holds by the definition of P \mathcal{P} P and Lemma B.6(a); and (d) is the definition of κ \kappa κ . Multiplying by p i > 0 p_i>0 p i > 0 gives α p i ≤ h ( p i ) ≤ κ p i \alpha p_i\leq h(p_i)\leq\kappa p_i α p i ≤ h ( p i ) ≤ κ p i . Moreover, since 4 K T < 8 K T / δ 4KT<8KT/\delta 4 K T < 8 K T / δ implies log ( 4 K T ) < log ( 8 K T / δ ) = L \log(4KT)<\log(8KT/\delta)=L log ( 4 K T ) < log ( 8 K T / δ ) = L , and L / η ≤ 3 C + L / η = α L/\eta\leq3C+L/\eta=\alpha L / η ≤ 3 C + L / η = α , we have κ = α + log ( 4 K T ) / η ≤ α + L / η ≤ 2 α \kappa=\alpha+\log(4KT)/\eta\leq\alpha+L/\eta\leq2\alpha κ = α + log ( 4 K T ) / η ≤ α + L / η ≤ 2 α . □
Design, Radii, and Kernel Properties
Proof of Lemma B.2 . Write ℓ i : = x i T G − 1 x i ≤ r i \ell_i:=\sqrt{x_i^{\mathsf T}G^{-1}x_i}\leq r_i ℓ i := x i T G − 1 x i ≤ r i , W : = ∑ m = 1 K p m r m W:=\sum_{m=1}^{K}p_mr_m W := ∑ m = 1 K p m r m , and b ( j ) : = ∑ l = 1 K p l k ( l , j ) b(j):=\sum_{l=1}^{K}p_lk(l,j) b ( j ) := ∑ l = 1 K p l k ( l , j ) , so that the normalized kernel in (1) takes the form
k ~ ( i , j ) = k ( i , j ) − r i W ( b ( j ) − 1 ) . \widetilde{k}(i,j)=k(i,j)-\frac{r_i}{W}\bigl(b(j)-1\bigr). k ( i , j ) = k ( i , j ) − W r i ( b ( j ) − 1 ) .
Since r m ≥ 1 r_m\geq1 r m ≥ 1 for all m ∈ [ K ] m\in[K] m ∈ [ K ] and ∑ m = 1 K p m = 1 \sum_{m=1}^{K}p_m=1 ∑ m = 1 K p m = 1 , we have W = ∑ m = 1 K p m r m ≥ 1 W=\sum_{m=1}^{K}p_mr_m\geq1 W = ∑ m = 1 K p m r m ≥ 1 .
Preliminary properties of the unnormalized kernel k k k . Before proving items (a)–(e), we record four basic properties of k ( i , j ) = x i T G − 1 x j / r j k(i,j)=x_i^{\mathsf T}G^{-1}x_j/r_j k ( i , j ) = x i T G − 1 x j / r j :
Action reproduction: For every i ∈ [ K ] i \in[K] i ∈ [ K ] ,
∑ j = 1 K p j k ( i , j ) x j T = x i T G − 1 ∑ j = 1 K p j r j x j x j T = x i T G − 1 G = x i T , (15) \sum_{j=1}^{K}p_jk(i,j)\mathbf{x}_j^{\mathsf{T}}=\mathbf{x}_i^{\mathsf{T}}\mathbf{G}^{-1}\sum_{j=1}^{K}\frac{p_j}{r_j}\mathbf{x}_j\mathbf{x}_j^{\mathsf{T}}=\mathbf{x}_i^{\mathsf{T}}\mathbf{G}^{-1}\mathbf{G}=\mathbf{x}_i^{\mathsf{T}}, \tag*{(15)} j = 1 ∑ K p j k ( i , j ) x j T = x i T G − 1 j = 1 ∑ K r j p j x j x j T = x i T G − 1 G = x i T , (15)
where the first equality uses the definition of k ( i , j ) k(i,j) k ( i , j ) and the second uses G = ∑ j = 1 K ( p j / r j ) x j x j T \mathbf{G}=\sum_{j=1}^{K}(p_j/r_j)\mathbf{x}_j\mathbf{x}_j^{\mathsf{T}} G = ∑ j = 1 K ( p j / r j ) x j x j T .
Pointwise magnitude bound: For all i , j ∈ [ K ] i,j \in[K] i , j ∈ [ K ] ,
∣ k ( i , j ) ∣ ≤ ( a ) ∣ x i T G − 1 x j ∣ r j ≤ ( b ) ℓ i ℓ j r j ≤ ( c ) ℓ i ≤ ( d ) r i , (16) \lvert k(i,j)\rvert\mathrel{\overset{(a)}{\leq}} \frac{\lvert\mathbf{x}_i^{\mathsf{T}}\mathbf{G}^{-1}\mathbf{x}_j\rvert}{r_j} \mathrel{\overset{(b)}{\leq}} \frac{\ell_i\ell_j}{r_j} \mathrel{\overset{(c)}{\leq}} \ell_i \mathrel{\overset{(d)}{\leq}} r_i, \tag*{(16)} ∣ k ( i , j )∣ ≤ ( a ) r j ∣ x i T G − 1 x j ∣ ≤ ( b ) r j ℓ i ℓ j ≤ ( c ) ℓ i ≤ ( d ) r i , (16)
where (a) is the definition of k k k , (b) is the Cauchy–Schwarz inequality for the inner product ( u , v ) ↦ u T G − 1 v (\mathbf{u},\mathbf{v})\mapsto\mathbf{u}^{\mathsf{T}}\mathbf{G}^{-1}\mathbf{v} ( u , v ) ↦ u T G − 1 v , and (c) and (d) use ℓ j ≤ r j \ell_j\leq r_j ℓ j ≤ r j and ℓ i ≤ r i \ell_i\leq r_i ℓ i ≤ r i .
Radius-weighted second moment: For every j ∈ [ K ] j \in[K] j ∈ [ K ] ,
∑ i = 1 K p i r i k ( i , j ) 2 = ( a ) x j T G − 1 ( ∑ i = 1 K p i r i x i x i T ) G − 1 x j r j 2 = ( b ) x j T G − 1 x j r j 2 = ( c ) ℓ j 2 r j 2 ≤ 1 , (17) \sum_{i=1}^{K}\frac{p_i}{r_i}k(i,j)^2 \mathrel{\overset{(a)}{=}} \frac{\mathbf{x}_j^{\mathsf{T}}\mathbf{G}^{-1}\left(\sum_{i=1}^{K}\frac{p_i}{r_i}\mathbf{x}_i\mathbf{x}_i^{\mathsf{T}}\right)\mathbf{G}^{-1}\mathbf{x}_j}{r_j^2} \mathrel{\overset{(b)}{=}} \frac{\mathbf{x}_j^{\mathsf{T}}\mathbf{G}^{-1}\mathbf{x}_j}{r_j^2} \mathrel{\overset{(c)}{=}} \frac{\ell_j^2}{r_j^2}\leq1, \tag*{(17)} i = 1 ∑ K r i p i k ( i , j ) 2 = ( a ) r j 2 x j T G − 1 ( ∑ i = 1 K r i p i x i x i T ) G − 1 x j = ( b ) r j 2 x j T G − 1 x j = ( c ) r j 2 ℓ j 2 ≤ 1 , (17)
where (a) writes k ( i , j ) 2 = x j T G − 1 x i x i T G − 1 x j / r j 2 k(i,j)^2=\mathbf{x}_j^{\mathsf{T}}\mathbf{G}^{-1}\mathbf{x}_i\mathbf{x}_i^{\mathsf{T}}\mathbf{G}^{-1}\mathbf{x}_j/r_j^2 k ( i , j ) 2 = x j T G − 1 x i x i T G − 1 x j / r j 2 , (b) uses the definition of G \mathbf{G} G , and (c) uses ℓ j ≤ r j \ell_j\leq r_j ℓ j ≤ r j .
Column-sum bound: For every j ∈ [ K ] j \in[K] j ∈ [ K ] ,
∣ b ( j ) ∣ = ( a ) ∣ ∑ i = 1 K p i r i ⋅ p i r i k ( i , j ) ∣ ≤ ( b ) ( ∑ i = 1 K p i r i ) 1 / 2 ( ∑ i = 1 K p i r i k ( i , j ) 2 ) 1 / 2 ≤ ( c ) W , (18) \lvert b(j)\rvert\mathrel{\overset{(a)}{=}}\left\lvert\sum_{i=1}^{K}\sqrt{p_ir_i}\cdot\sqrt{\frac{p_i}{r_i}}k(i,j)\right\rvert\mathrel{\overset{(b)}{\leq}}\left(\sum_{i=1}^{K}p_ir_i\right)^{1/2}\left(\sum_{i=1}^{K}\frac{p_i}{r_i}k(i,j)^2\right)^{1/2}\mathrel{\overset{(c)}{\leq}}\sqrt{W}, \tag*{(18)} ∣ b ( j )∣ = ( a ) i = 1 ∑ K p i r i ⋅ r i p i k ( i , j ) ≤ ( b ) ( i = 1 ∑ K p i r i ) 1/2 ( i = 1 ∑ K r i p i k ( i , j ) 2 ) 1/2 ≤ ( c ) W , (18)
where (a) writes p i = p i r i p i / r i p_i=\sqrt{p_ir_i}\sqrt{p_i/r_i} p i = p i r i p i / r i , (b) is the Cauchy–Schwarz inequality, and (c) uses W = ∑ i = 1 K p i r i W=\sum_{i=1}^{K}p_ir_i W = ∑ i = 1 K p i r i and (C.3).
We now prove items (a)–(e) in order.
Proof of (a). Using k ~ ( i , j ) = k ( i , j ) − r i W ( b ( j ) − 1 ) \widetilde{k}(i,j)=k(i,j)-\frac{r_i}{W}(b(j)-1) k ( i , j ) = k ( i , j ) − W r i ( b ( j ) − 1 ) , we have
∑ j = 1 K p j k ~ ( i , j ) x j = ( a ) ∑ j = 1 K p j k ( i , j ) x j − r i W ∑ j = 1 K p j ( b ( j ) − 1 ) x j \sum_{j=1}^{K}p_j\widetilde{k}(i,j)\mathbf{x}_j \mathrel{\overset{(a)}{=}} \sum_{j=1}^{K}p_jk(i,j)\mathbf{x}_j-\frac{r_i}{W}\sum_{j=1}^{K}p_j(b(j)-1)\mathbf{x}_j j = 1 ∑ K p j k ( i , j ) x j = ( a ) j = 1 ∑ K p j k ( i , j ) x j − W r i j = 1 ∑ K p j ( b ( j ) − 1 ) x j
= ( b ) x i − r i W ( ∑ l = 1 K p l ∑ j = 1 K p j k ( l , j ) x j − ∑ j = 1 K p j x j ) = ( c ) x i , \mathrel{\overset{(b)}{=}}\mathbf{x}_i-\frac{r_i}{W}\left(\sum_{l=1}^{K}p_l\sum_{j=1}^{K}p_jk(l,j)\mathbf{x}_j-\sum_{j=1}^{K}p_j\mathbf{x}_j\right)\mathrel{\overset{(c)}{=}}\mathbf{x}_i, = ( b ) x i − W r i ( l = 1 ∑ K p l j = 1 ∑ K p j k ( l , j ) x j − j = 1 ∑ K p j x j ) = ( c ) x i ,
where (a) substitutes the definition of k ~ ( i , j ) \widetilde{k}(i,j) k ( i , j ) , (b) applies (C.1) to the first term and expands b ( j ) = ∑ l = 1 K p l k ( l , j ) b(j)=\sum_{l=1}^{K}p_lk(l,j) b ( j ) = ∑ l = 1 K p l k ( l , j ) in the second, and (c) applies (C.1) to each l ∈ [ K ] l\in[K] l ∈ [ K ] to obtain ∑ l = 1 K p l ∑ j = 1 K p j k ( l , j ) x j = ∑ l = 1 K p l x l \sum_{l=1}^{K}p_l\sum_{j=1}^{K}p_jk(l,j)\mathbf{x}_j=\sum_{l=1}^{K}p_l\mathbf{x}_l ∑ l = 1 K p l ∑ j = 1 K p j k ( l , j ) x j = ∑ l = 1 K p l x l .
Proof of (b). Summing k ~ ( i , j ) \widetilde{k}(i,j) k ( i , j ) against p i p_i p i gives
∑ i = 1 K p i k ~ ( i , j ) = ∑ i = 1 K p i k ( i , j ) − ∑ i = 1 K p i r i W ( b ( j ) − 1 ) = b ( j ) − ( b ( j ) − 1 ) = 1. \sum_{i=1}^{K}p_i\widetilde{k}(i,j)=\sum_{i=1}^{K}p_ik(i,j)-\frac{\sum_{i=1}^{K}p_ir_i}{W}(b(j)-1)=b(j)-(b(j)-1)=1. i = 1 ∑ K p i k ( i , j ) = i = 1 ∑ K p i k ( i , j ) − W ∑ i = 1 K p i r i ( b ( j ) − 1 ) = b ( j ) − ( b ( j ) − 1 ) = 1.
where the first equality substitutes the definition of k ~ ( i , j ) \widetilde{k}(i,j) k ( i , j ) and the second uses the definitions b ( j ) = ∑ i = 1 K p i k ( i , j ) b(j)=\sum_{i=1}^{K}p_i k(i,j) b ( j ) = ∑ i = 1 K p i k ( i , j ) and W = ∑ i = 1 K p i r i W=\sum_{i=1}^{K}p_i r_i W = ∑ i = 1 K p i r i .
Proof of (c). By the triangle inequality, (C.2), and (C.4),
∣ k ~ ( i , j ) ∣ ≤ ∣ k ( i , j ) ∣ + r i W ( ∣ b ( j ) ∣ + 1 ) ≤ ( a ) r i + r i W ( W + 1 ) = r i ( 1 + W − 1 / 2 + W − 1 ) ≤ ( c ) 3 r i , \left|\widetilde{k}(i,j)\right| \le\left|k(i,j)\right|+\frac{r_i}{W}\left(\left|b(j)\right|+1\right) \mathrel{\overset{(a)}{\le}} r_i+\frac{r_i}{W}\left(\sqrt{W}+1\right)=r_i\left(1+W^{-1/2}+W^{-1}\right) \mathrel{\overset{(c)}{\le}} 3r_i, k ( i , j ) ≤ ∣ k ( i , j ) ∣ + W r i ( ∣ b ( j ) ∣ + 1 ) ≤ ( a ) r i + W r i ( W + 1 ) = r i ( 1 + W − 1/2 + W − 1 ) ≤ ( c ) 3 r i ,
where (a) is the triangle inequality, (b) uses ∣ k ( i , j ) ∣ ≤ r i \left|k(i,j)\right|\le r_i ∣ k ( i , j ) ∣ ≤ r i from (C.2) and ∣ b ( j ) ∣ ≤ W \left|b(j)\right|\le\sqrt{W} ∣ b ( j ) ∣ ≤ W from (C.4), and (c) uses W ≥ 1 W\ge1 W ≥ 1 .
Proof of (d). Expanding the square in k ~ ( i , j ) = k ( i , j ) − r i W ( b ( j ) − 1 ) \widetilde{k}(i,j)=k(i,j)-\frac{r_i}{W}(b(j)-1) k ( i , j ) = k ( i , j ) − W r i ( b ( j ) − 1 ) gives
∑ i = 1 K p i r i k ~ ( i , j ) 2 = ∑ i = 1 K p i r i k ( i , j ) 2 − 2 ( b ( j ) − 1 ) W ∑ i = 1 K p i k ( i , j ) + ( b ( j ) − 1 ) 2 W 2 ∑ i = 1 K p i r i \sum_{i=1}^{K}\frac{p_i}{r_i}\widetilde{k}(i,j)^2=\sum_{i=1}^{K}\frac{p_i}{r_i}k(i,j)^2-\frac{2(b(j)-1)}{W}\sum_{i=1}^{K}p_i k(i,j)+\frac{(b(j)-1)^2}{W^2}\sum_{i=1}^{K}p_i r_i i = 1 ∑ K r i p i k ( i , j ) 2 = i = 1 ∑ K r i p i k ( i , j ) 2 − W 2 ( b ( j ) − 1 ) i = 1 ∑ K p i k ( i , j ) + W 2 ( b ( j ) − 1 ) 2 i = 1 ∑ K p i r i
= ( b ) ℓ j 2 r j 2 − 2 b ( j ) ( b ( j ) − 1 ) W + ( b ( j ) − 1 ) 2 W = ( c ) ℓ j 2 r j 2 − b ( j ) 2 − 1 W ≤ ( d ) 1 + 1 W ≤ ( e ) 2 , \mathrel{\overset{(b)}{=}}\frac{\ell_j^2}{r_j^2}-\frac{2b(j)(b(j)-1)}{W}+\frac{(b(j)-1)^2}{W}\mathrel{\overset{(c)}{=}}\frac{\ell_j^2}{r_j^2}-\frac{b(j)^2-1}{W}\mathrel{\overset{(d)}{\le}}1+\frac{1}{W}\mathrel{\overset{(e)}{\le}}2, = ( b ) r j 2 ℓ j 2 − W 2 b ( j ) ( b ( j ) − 1 ) + W ( b ( j ) − 1 ) 2 = ( c ) r j 2 ℓ j 2 − W b ( j ) 2 − 1 ≤ ( d ) 1 + W 1 ≤ ( e ) 2 ,
where (a) expands k ~ ( i , j ) 2 \widetilde{k}(i,j)^2 k ( i , j ) 2 ; (b) uses (C.3), ∑ i = 1 K p i k ( i , j ) = b ( j ) \sum_{i=1}^{K}p_i k(i,j)=b(j) ∑ i = 1 K p i k ( i , j ) = b ( j ) , and ∑ i = 1 K p i r i = W \sum_{i=1}^{K}p_i r_i=W ∑ i = 1 K p i r i = W ; (c) uses − 2 b ( b − 1 ) + ( b − 1 ) 2 = − ( b 2 − 1 ) -2b(b-1)+(b-1)^2=-(b^2-1) − 2 b ( b − 1 ) + ( b − 1 ) 2 = − ( b 2 − 1 ) ; (d) uses ℓ j ≤ r j \ell_j\le r_j ℓ j ≤ r j and b ( j ) 2 ≥ 0 b(j)^2\ge0 b ( j ) 2 ≥ 0 ; and (e) uses W ≥ 1 W\ge1 W ≥ 1 .
Proof of (e). By definition, v i = 1 r i ∑ j = 1 K p j k ~ ( i , j ) 2 ≥ 0 v_i=\frac{1}{r_i}\sum_{j=1}^{K}p_j\widetilde{k}(i,j)^2\ge0 v i = r i 1 ∑ j = 1 K p j k ( i , j ) 2 ≥ 0 . Applying items (c) and (d) gives
v i = 1 r i ∑ j = 1 K p j k ~ ( i , j ) 2 ≤ ( a ) 1 r i ∑ j = 1 K p j ( 3 r i ) 2 = ( b ) 9 r i , ∑ i = 1 K p i v i = ( c ) ∑ j = 1 K p j ∑ i = 1 K p i r i k ~ ( i , j ) 2 ≤ ( d ) 2 , v_i=\frac{1}{r_i}\sum_{j=1}^{K}p_j\widetilde{k}(i,j)^2\mathrel{\overset{(a)}{\le}}\frac{1}{r_i}\sum_{j=1}^{K}p_j(3r_i)^2\mathrel{\overset{(b)}{=}}9r_i,\qquad\sum_{i=1}^{K}p_i v_i\mathrel{\overset{(c)}{=}}\sum_{j=1}^{K}p_j\sum_{i=1}^{K}\frac{p_i}{r_i}\widetilde{k}(i,j)^2\mathrel{\overset{(d)}{\le}}2, v i = r i 1 j = 1 ∑ K p j k ( i , j ) 2 ≤ ( a ) r i 1 j = 1 ∑ K p j ( 3 r i ) 2 = ( b ) 9 r i , i = 1 ∑ K p i v i = ( c ) j = 1 ∑ K p j i = 1 ∑ K r i p i k ( i , j ) 2 ≤ ( d ) 2 ,
where (a) uses ∣ k ~ ( i , j ) ∣ ≤ 3 r i \left|\widetilde{k}(i,j)\right|\le3r_i k ( i , j ) ≤ 3 r i from item (c), (b) uses ∑ j = 1 K p j = 1 \sum_{j=1}^{K}p_j=1 ∑ j = 1 K p j = 1 , (c) exchanges the order of summation, and (d) uses item (d) together with ∑ j = 1 K p j = 1 \sum_{j=1}^{K}p_j=1 ∑ j = 1 K p j = 1 .
Proof of Lemma B.6. We verify H ( ρ ) ≻ 0 \mathbf{H}(\boldsymbol{\rho})\succ0 H ( ρ ) ≻ 0 and items (a)–(d) in order.
**Positive definiteness H ( ρ ) ≻ 0. ∗ ∗ S i n c e \mathbf{H}(\boldsymbol{\rho})\succ0.** Since H ( ρ ) ≻ 0. ∗ ∗ S in ce {\mathbf{z}_i}_{i=1}^{K}$ spans R d + 1 \mathbb{R}^{d+1} R d + 1 , the uniform distribution gives H ( 1 / K ) ≻ 0 \mathbf{H}(1/K)\succ0 H ( 1/ K ) ≻ 0 . The function p ↦ det H ( p ) \mathbf{p}\mapsto\det\mathbf{H}(\mathbf{p}) p ↦ det H ( p ) is continuous on the compact set Δ K \Delta_K Δ K , so its maximum is attained and is positive, and any maximizer ρ ⋆ \boldsymbol{\rho}^{\star} ρ ⋆ satisfies H ⋆ : = H ( ρ ⋆ ) ≻ 0 \mathbf{H}^{\star}:=\mathbf{H}(\boldsymbol{\rho}^{\star})\succ0 H ⋆ := H ( ρ ⋆ ) ≻ 0 . By linearity of H ( ⋅ ) \mathbf{H}(\cdot) H ( ⋅ ) and the definition ρ = 1 2 ρ ⋆ + 1 2 ( 1 / K ) \boldsymbol{\rho}=\frac{1}{2}\boldsymbol{\rho}^{\star}+\frac{1}{2}(1/K) ρ = 2 1 ρ ⋆ + 2 1 ( 1/ K ) ,
H ( ρ ) = 1 2 H ⋆ + 1 2 H ( 1 / K ) ⪰ 1 2 H ⋆ ≻ 0. (19) \mathbf{H}(\boldsymbol{\rho})=\frac{1}{2}\mathbf{H}^{\star}+\frac{1}{2}\mathbf{H}(1/K)\succeq\frac{1}{2}\mathbf{H}^{\star}\succ0. \tag*{(19)} H ( ρ ) = 2 1 H ⋆ + 2 1 H ( 1/ K ) ⪰ 2 1 H ⋆ ≻ 0. (19)
Proof of (a). From ρ = 1 2 ρ ⋆ + 1 2 ( 1 / K ) \boldsymbol{\rho}=\frac{1}{2}\boldsymbol{\rho}^{\star}+\frac{1}{2}(1/K) ρ = 2 1 ρ ⋆ + 2 1 ( 1/ K ) and ρ i ⋆ ≥ 0 \rho_i^{\star}\ge0 ρ i ⋆ ≥ 0 , we immediately have ρ i ≥ 1 / ( 2 K ) \rho_i\ge1/(2K) ρ i ≥ 1/ ( 2 K ) for all i ∈ [ K ] i\in[K] i ∈ [ K ] .
Proof of (b). Using x i = H ( ρ ) − 1 / 2 z i \mathbf{x}_i=\mathbf{H}(\boldsymbol{\rho})^{-1/2}\mathbf{z}_i x i = H ( ρ ) − 1/2 z i and H ( ρ ) = ∑ i = 1 K ρ i z i z i T \mathbf{H}(\boldsymbol{\rho})=\sum_{i=1}^{K}\rho_i\mathbf{z}_i\mathbf{z}_i^{\mathsf{T}} H ( ρ ) = ∑ i = 1 K ρ i z i z i T , we obtain
∑ i = 1 K ρ i x i x i T = H ( ρ ) − 1 / 2 ( ∑ i = 1 K ρ i z i z i T ) H ( ρ ) − 1 / 2 = H ( ρ ) − 1 / 2 H ( ρ ) H ( ρ ) − 1 / 2 = I . \sum_{i=1}^{K}\rho_i\mathbf{x}_i\mathbf{x}_i^{\mathsf{T}}=\mathbf{H}(\boldsymbol{\rho})^{-1/2}\left(\sum_{i=1}^{K}\rho_i\mathbf{z}_i\mathbf{z}_i^{\mathsf{T}}\right)\mathbf{H}(\boldsymbol{\rho})^{-1/2}=\mathbf{H}(\boldsymbol{\rho})^{-1/2}\mathbf{H}(\boldsymbol{\rho})\mathbf{H}(\boldsymbol{\rho})^{-1/2}=\mathbf{I}. i = 1 ∑ K ρ i x i x i T = H ( ρ ) − 1/2 ( i = 1 ∑ K ρ i z i z i T ) H ( ρ ) − 1/2 = H ( ρ ) − 1/2 H ( ρ ) H ( ρ ) − 1/2 = I .
Proof of (c). For the upper bound ∥ x i ∥ 2 ≤ 2 ( d + 1 ) \lVert x_i\rVert^2 \le2(d+1) ∥ x i ∥ 2 ≤ 2 ( d + 1 ) , note that for every i ∈ [ K ] i \in[K] i ∈ [ K ] and ε ∈ [ 0 , 1 ] \varepsilon\in[0,1] ε ∈ [ 0 , 1 ] , the mixture ( 1 − ε ) ρ ⋆ + ε e i (1-\varepsilon)\rho^\star+\varepsilon e_i ( 1 − ε ) ρ ⋆ + ε e i lies in Δ K \Delta_K Δ K , so first-order optimality of ρ ⋆ \rho^\star ρ ⋆ gives
0 ≥ d d ε log det H ( ( 1 − ε ) ρ ⋆ + ε e i ) ∣ ε = 0 + = ( a ) tr ( H ⋆ − 1 ( z i z i T − H ⋆ ) ) = ( b ) z i T H ⋆ − 1 z i − ( d + 1 ) 0 \ge\left.\frac{d}{d\varepsilon}\log\det\mathbf{H}\bigl((1-\varepsilon)\rho^\star+\varepsilon e_i\bigr)\right|_{\varepsilon=0^+}
\overset{(a)}{=} \operatorname{tr}\bigl(\mathbf{H}^{\star-1}(z_i z_i^{\mathsf T}-\mathbf{H}^\star)\bigr)
\overset{(b)}{=} z_i^{\mathsf T}\mathbf{H}^{\star-1}z_i-(d+1) 0 ≥ d ε d log det H ( ( 1 − ε ) ρ ⋆ + ε e i ) ε = 0 + = ( a ) tr ( H ⋆− 1 ( z i z i T − H ⋆ ) ) = ( b ) z i T H ⋆− 1 z i − ( d + 1 )
where (a) uses that ρ ⋆ \rho^\star ρ ⋆ maximizes log det H \log\det\mathbf{H} log det H over Δ K \Delta_K Δ K ; (b) uses the linearity of H ( ⋅ ) \mathbf{H}(\cdot) H ( ⋅ ) , which gives H ( ( 1 − ε ) ρ ⋆ + ε e i ) = H ⋆ + ε ( z i z i T − H ⋆ ) \mathbf{H}((1-\varepsilon)\rho^\star+\varepsilon e_i)=\mathbf{H}^\star+\varepsilon(z_i z_i^{\mathsf T}-\mathbf{H}^\star) H (( 1 − ε ) ρ ⋆ + ε e i ) = H ⋆ + ε ( z i z i T − H ⋆ ) , and d d ε log det ( H ⋆ + ε Δ ) ∣ ε = 0 = tr ( H ⋆ − 1 Δ ) \left.\frac{d}{d\varepsilon}\log\det(\mathbf{H}^\star+\varepsilon\Delta)\right|_{\varepsilon=0}=\operatorname{tr}(\mathbf{H}^{\star-1}\Delta) d ε d log det ( H ⋆ + ε Δ ) ε = 0 = tr ( H ⋆− 1 Δ ) ; and (c) uses tr ( H ⋆ − 1 z i z i T ) = z i T H ⋆ − 1 z i \operatorname{tr}(\mathbf{H}^{\star-1}z_i z_i^{\mathsf T})=z_i^{\mathsf T}\mathbf{H}^{\star-1}z_i tr ( H ⋆− 1 z i z i T ) = z i T H ⋆− 1 z i and tr ( I d + 1 ) = d + 1 \operatorname{tr}(\mathbf{I}_{d+1})=d+1 tr ( I d + 1 ) = d + 1 . Combining z i T H ⋆ − 1 z i ≤ d + 1 z_i^{\mathsf T}\mathbf{H}^{\star-1}z_i\le d+1 z i T H ⋆− 1 z i ≤ d + 1 with H ( ρ ) ⪰ 1 2 H ⋆ \mathbf{H}(\rho)\succeq\frac{1}{2}\mathbf{H}^\star H ( ρ ) ⪰ 2 1 H ⋆ from (C.5) (which implies H ( ρ ) − 1 ⪯ 2 H ⋆ − 1 \mathbf{H}(\rho)^{-1}\preceq2\mathbf{H}^{\star-1} H ( ρ ) − 1 ⪯ 2 H ⋆− 1 ) yields
∥ x i ∥ 2 = z i T H ( ρ ) − 1 z i ≤ 2 z i T H ⋆ − 1 z i ≤ 2 ( d + 1 ) . \lVert x_i\rVert^2=z_i^{\mathsf T}\mathbf{H}(\rho)^{-1}z_i\le2z_i^{\mathsf T}\mathbf{H}^{\star-1}z_i\le2(d+1). ∥ x i ∥ 2 = z i T H ( ρ ) − 1 z i ≤ 2 z i T H ⋆− 1 z i ≤ 2 ( d + 1 ) .
For the lower bound ∥ x i ∥ 2 ≥ 1 \lVert x_i\rVert^2\ge1 ∥ x i ∥ 2 ≥ 1 , let e = ( 0 , … , 0 , 1 ) T ∈ R d + 1 e=(0,\ldots,0,1)^{\mathsf T}\in\mathbb{R}^{d+1} e = ( 0 , … , 0 , 1 ) T ∈ R d + 1 be the last standard basis vector and set w : = H ( ρ ) 1 / 2 e w:=\mathbf{H}(\rho)^{1/2}e w := H ( ρ ) 1/2 e . Because every z i = ( a i T , 1 ) T z_i=(a_i^{\mathsf T},1)^{\mathsf T} z i = ( a i T , 1 ) T has last coordinate e T z i = 1 e^{\mathsf T}z_i=1 e T z i = 1 ,
∥ w ∥ 2 = ( a ) e T H ( ρ ) e = ( b ) ∑ i = 1 K ρ i ( e T z i ) 2 = ( c ) 1 , w T x i = ( d ) e T z i = 1 , \lVert w\rVert^2\overset{(a)}{=}e^{\mathsf T}\mathbf{H}(\rho)e\overset{(b)}{=}\sum_{i=1}^{K}\rho_i(e^{\mathsf T}z_i)^2\overset{(c)}{=}1,\qquad
w^{\mathsf T}x_i\overset{(d)}{=}e^{\mathsf T}z_i=1, ∥ w ∥ 2 = ( a ) e T H ( ρ ) e = ( b ) i = 1 ∑ K ρ i ( e T z i ) 2 = ( c ) 1 , w T x i = ( d ) e T z i = 1 ,
where (a) is the definition of w w w , (b) is the definition of H ( ρ ) \mathbf{H}(\rho) H ( ρ ) , (c) uses e T z i = 1 e^{\mathsf T}z_i=1 e T z i = 1 and ∑ i = 1 K ρ i = 1 \sum_{i=1}^{K}\rho_i=1 ∑ i = 1 K ρ i = 1 , and (d) uses x i = H ( ρ ) − 1 / 2 z i x_i=\mathbf{H}(\rho)^{-1/2}z_i x i = H ( ρ ) − 1/2 z i . The Cauchy–Schwarz inequality then gives 1 = w T x i ≤ ∥ w ∥ ∥ x i ∥ = ∥ x i ∥ 1=w^{\mathsf T}x_i\le\lVert w\rVert\lVert x_i\rVert=\lVert x_i\rVert 1 = w T x i ≤ ∥ w ∥ ∥ x i ∥ = ∥ x i ∥ .
Proof of (d). Using x i = H ( ρ ) − 1 / 2 z i x_i=\mathbf{H}(\rho)^{-1/2}z_i x i = H ( ρ ) − 1/2 z i , the symmetry of H ( ρ ) − 1 / 2 \mathbf{H}(\rho)^{-1/2} H ( ρ ) − 1/2 , and z i = ( a i T , 1 ) T z_i=(a_i^{\mathsf T},1)^{\mathsf T} z i = ( a i T , 1 ) T , we directly compute
x i T H ( ρ ) 1 / 2 ( θ t T , 0 ) T = z i T H ( ρ ) − 1 / 2 H ( ρ ) 1 / 2 ( θ t T , 0 ) T = z i T ( θ t T , 0 ) T = a i T θ t . x_i^{\mathsf T}\mathbf{H}(\rho)^{1/2}(\theta_t^{\mathsf T},0)^{\mathsf T}
=z_i^{\mathsf T}\mathbf{H}(\rho)^{-1/2}\mathbf{H}(\rho)^{1/2}(\theta_t^{\mathsf T},0)^{\mathsf T}
=z_i^{\mathsf T}(\theta_t^{\mathsf T},0)^{\mathsf T}
=a_i^{\mathsf T}\theta_t. x i T H ( ρ ) 1/2 ( θ t T , 0 ) T = z i T H ( ρ ) − 1/2 H ( ρ ) 1/2 ( θ t T , 0 ) T = z i T ( θ t T , 0 ) T = a i T θ t .
Proof of Lemma B.7. Proof of (ii). By Lemma C.1, Q t = Q ( Q t − 1 , p t ) ⪰ Q t − 1 \mathbf{Q}_t=\mathbf{Q}(\mathbf{Q}_{t-1},\mathbf{p}_t)\succeq\mathbf{Q}_{t-1} Q t = Q ( Q t − 1 , p t ) ⪰ Q t − 1 , and since the algorithm starts from Q 0 = I \mathbf{Q}_0=\mathbf{I} Q 0 = I , we have I = Q 0 ⪯ Q t − 1 ⪯ Q t \mathbf{I}=\mathbf{Q}_0\preceq\mathbf{Q}_{t-1}\preceq\mathbf{Q}_t I = Q 0 ⪯ Q t − 1 ⪯ Q t by induction. Because Q ↦ x i T Q x i \mathbf{Q}\mapsto x_i^{\mathsf T}\mathbf{Q}x_i Q ↦ x i T Q x i is monotone in the Loewner order,
1 ≤ ∥ x i ∥ = ( a ) r i ( Q 0 ) ≤ ( b ) r t − 1 , i ≤ ( c ) r t i , 1\le\lVert x_i\rVert\overset{(a)}{=}r_i(\mathbf{Q}_0)\overset{(b)}{\le}r_{t-1,i}\overset{(c)}{\le}r_{t i}, 1 ≤ ∥ x i ∥ = ( a ) r i ( Q 0 ) ≤ ( b ) r t − 1 , i ≤ ( c ) r t i ,
where (a) is Lemma B.6(c), (b) uses Q 0 = I \mathbf{Q}_0=\mathbf{I} Q 0 = I , and (c) uses Q 0 ⪯ Q t − 1 ⪯ Q t \mathbf{Q}_0\preceq\mathbf{Q}_{t-1}\preceq\mathbf{Q}_t Q 0 ⪯ Q t − 1 ⪯ Q t . This proves item (ii).
First-order optimality conditions for Q t \mathbf{Q}_t Q t . Items (i) and (iii) rest on the first-order optimality conditions of the inner problem min Q ⪰ Q t − 1 J ( Q , p t ) \min_{\mathbf{Q}\succeq\mathbf{Q}_{t-1}}J(\mathbf{Q},\mathbf{p}_t) min Q ⪰ Q t − 1 J ( Q , p t ) , which we derive next. Write Δ Q t = Q t − Q t − 1 \Delta\mathbf{Q}_t=\mathbf{Q}_t-\mathbf{Q}_{t-1} Δ Q t = Q t − Q t − 1 and G h , t = ∑ i ( h ( p t i ) / r t i ) x i x i T \mathbf{G}_{h,t}=\sum_i(h(p_{t i})/r_{t i})x_i x_i^{\mathsf T} G h , t = ∑ i ( h ( p t i ) / r t i ) x i x i T . The function J ( ⋅ , p t ) J(\cdot,\mathbf{p}_t) J ( ⋅ , p t ) is differentiable on { Q ≻ 0 } \{\mathbf{Q}\succ0\} { Q ≻ 0 } , with ∇ Q r i ( Q ) = x i x i T / ( 2 r i ( Q ) ) \nabla_{\mathbf{Q}}r_i(\mathbf{Q})=x_i x_i^{\mathsf T}/(2r_i(\mathbf{Q})) ∇ Q r i ( Q ) = x i x i T / ( 2 r i ( Q )) and ∇ Q log det Q = Q − 1 \nabla_{\mathbf{Q}}\log\det\mathbf{Q}=\mathbf{Q}^{-1} ∇ Q log det Q = Q − 1 , so its gradient at Q t \mathbf{Q}_t Q t is
∇ J : = ∇ Q J ( Q t , p t ) = 1 2 ( G h , t − κ Q t − 1 ) . \nabla J:=\nabla_{\mathbf{Q}}J(\mathbf{Q}_t,\mathbf{p}_t)=\frac{1}{2}\left(\mathbf{G}_{h,t}-\kappa\mathbf{Q}_t^{-1}\right). ∇ J := ∇ Q J ( Q t , p t ) = 2 1 ( G h , t − κ Q t − 1 ) .
For every U ⪰ 0 \mathbf{U}\succeq0 U ⪰ 0 and ε > 0 \varepsilon>0 ε > 0 , the matrix Q t + ε U \mathbf{Q}_t+\varepsilon\mathbf{U} Q t + ε U is feasible, so the minimality of Q t \mathbf{Q}_t Q t makes the one-sided derivative tr ( ∇ J U ) \operatorname{tr}(\nabla J\mathbf{U}) tr ( ∇ J U ) of ε ↦ J ( Q t + ε U , p t ) \varepsilon\mapsto J(\mathbf{Q}_t+\varepsilon\mathbf{U},\mathbf{p}_t) ε ↦ J ( Q t + ε U , p t ) at 0 0 0 nonnegative. Taking U = v v T \mathbf{U}=\mathbf{v}\mathbf{v}^{\mathsf T} U = v v T for an arbitrary vector v \mathbf{v} v gives v T ∇ J v ≥ 0 \mathbf{v}^{\mathsf T}\nabla J\mathbf{v}\ge0 v T ∇ J v ≥ 0 , that is,
G h , t ⪰ κ Q t − 1 . (20) \mathbf{G}_{h,t}\succeq\kappa\mathbf{Q}_t^{-1}. \tag*{(20)} G h , t ⪰ κ Q t − 1 . (20)
In addition, for ε ∈ [ 0 , 1 ] \varepsilon\in[0,1] ε ∈ [ 0 , 1 ] the matrix Q t − ε Δ Q t = Q t − 1 + ( 1 − ε ) Δ Q t \mathbf{Q}_t-\varepsilon\Delta\mathbf{Q}_t=\mathbf{Q}_{t-1}+(1-\varepsilon)\Delta\mathbf{Q}_t Q t − ε Δ Q t = Q t − 1 + ( 1 − ε ) Δ Q t is feasible, so minimality also gives tr ( ∇ J Δ Q t ) ≤ 0 \operatorname{tr}(\nabla J\Delta\mathbf{Q}_t)\leq0 tr ( ∇ J Δ Q t ) ≤ 0 . As ∇ J \nabla J ∇ J and Δ Q t \Delta\mathbf{Q}_t Δ Q t are both positive semidefinite, the trace of their product is nonnegative as well, and therefore
tr ( ( G h , t − κ Q t − 1 ) Δ Q t ) = 0. (21) \operatorname{tr}\left((\mathbf{G}_{h,t}-\kappa\mathbf{Q}_t^{-1})\Delta\mathbf{Q}_t\right)=0. \tag*{(21)} tr ( ( G h , t − κ Q t − 1 ) Δ Q t ) = 0. (21)
Proof of (i). From (C.6), we obtain
κ G t ( a ) ⪰ G h , t ( b ) ⪰ κ Q t − 1 , hence x i T G t − 1 x i ≤ ( c ) x i T Q t x i = ( d ) r t i 2 . \overset{(a)}{\kappa\mathbf{G}_t}\succeq\overset{(b)}{\mathbf{G}_{h,t}}\succeq\kappa\mathbf{Q}_t^{-1},\qquad\text{hence}\qquad\mathbf{x}_i^{\mathsf T}\mathbf{G}_t^{-1}\mathbf{x}_i\overset{(c)}{\leq}\mathbf{x}_i^{\mathsf T}\mathbf{Q}_t\mathbf{x}_i\overset{(d)}{=}r_{t i}^{2}. κ G t ( a ) ⪰ G h , t ( b ) ⪰ κ Q t − 1 , hence x i T G t − 1 x i ≤ ( c ) x i T Q t x i = ( d ) r t i 2 .
where (a) uses h ( p t i ) ≤ κ p t i h(p_{t i})\leq\kappa p_{t i} h ( p t i ) ≤ κ p t i (Lemma B.5) and x i x i T / r t i ⪰ 0 \mathbf{x}_i\mathbf{x}_i^{\mathsf T}/r_{t i}\succeq0 x i x i T / r t i ⪰ 0 ; (b) is (C.6); (c) uses that inversion reverses the Loewner order on positive definite matrices, so G t ⪰ Q t − 1 \mathbf{G}_t\succeq\mathbf{Q}_t^{-1} G t ⪰ Q t − 1 implies G t − 1 ⪯ Q t \mathbf{G}_t^{-1}\preceq\mathbf{Q}_t G t − 1 ⪯ Q t ; and (d) is the definition of r t i r_{t i} r t i .
Proof of (iii). We use both optimality conditions to limit how fast the largest eigenvalue of Q t \mathbf{Q}_t Q t can grow. Let λ 1 , … , λ d + 1 \lambda_1,\ldots,\lambda_{d+1} λ 1 , … , λ d + 1 denote the eigenvalues of Q t − 1 − 1 / 2 Q t Q t − 1 − 1 / 2 \mathbf{Q}_{t-1}^{-1/2}\mathbf{Q}_t\mathbf{Q}_{t-1}^{-1/2} Q t − 1 − 1/2 Q t Q t − 1 − 1/2 , which are at least 1 because Q t ⪰ Q t − 1 \mathbf{Q}_t\succeq\mathbf{Q}_{t-1} Q t ⪰ Q t − 1 . Let λ t = λ max ( Q t ) \lambda_t=\lambda_{\max}(\mathbf{Q}_t) λ t = λ m a x ( Q t ) and s t = λ t s_t=\sqrt{\lambda_t} s t = λ t , so that s 0 = 1 s_0=1 s 0 = 1 , and let b T = 4 2 ( d + 1 ) 3 / 2 T b_T=4\sqrt{2}(d+1)^{3/2}T b T = 4 2 ( d + 1 ) 3/2 T . On the one hand,
tr ( G h , t Δ Q t ) = ( a ) κ tr ( Q t − 1 Δ Q t ) = ( b ) κ tr ( I − Q t − 1 Q t − 1 ) = ( c ) κ ∑ k = 1 d + 1 ( 1 − λ k − 1 ) ≤ ( d ) κ ( d + 1 ) , \operatorname{tr}(\mathbf{G}_{h,t}\Delta\mathbf{Q}_t)\overset{(a)}{=}\kappa\operatorname{tr}(\mathbf{Q}_t^{-1}\Delta\mathbf{Q}_t)\overset{(b)}{=}\kappa\operatorname{tr}(\mathbf{I}-\mathbf{Q}_t^{-1}\mathbf{Q}_{t-1})\overset{(c)}{=}\kappa\sum_{k=1}^{d+1}(1-\lambda_k^{-1})\overset{(d)}{\leq}\kappa(d+1), tr ( G h , t Δ Q t ) = ( a ) κ tr ( Q t − 1 Δ Q t ) = ( b ) κ tr ( I − Q t − 1 Q t − 1 ) = ( c ) κ k = 1 ∑ d + 1 ( 1 − λ k − 1 ) ≤ ( d ) κ ( d + 1 ) ,
where (a) is (C.7); (b) uses Δ Q t = Q t − Q t − 1 \Delta\mathbf{Q}_t=\mathbf{Q}_t-\mathbf{Q}_{t-1} Δ Q t = Q t − Q t − 1 ; (c) uses that Q t − 1 Q t − 1 \mathbf{Q}_t^{-1}\mathbf{Q}_{t-1} Q t − 1 Q t − 1 is similar to ( Q t − 1 − 1 / 2 Q t Q t − 1 − 1 / 2 ) − 1 (\mathbf{Q}_{t-1}^{-1/2}\mathbf{Q}_t\mathbf{Q}_{t-1}^{-1/2})^{-1} ( Q t − 1 − 1/2 Q t Q t − 1 − 1/2 ) − 1 , whose eigenvalues are λ k − 1 \lambda_k^{-1} λ k − 1 ; and (d) uses λ k ≥ 1 \lambda_k\geq1 λ k ≥ 1 . On the other hand,
tr ( G h , t Δ Q t ) = ( a ) ∑ i h ( p t i ) x i T Δ Q t x i r t i ≥ ( b ) α 2 T 2 ( d + 1 ) λ t ∑ i ρ i x i T Δ Q t x i = ( c ) α tr ( Δ Q t ) 2 T 2 ( d + 1 ) λ t , \operatorname{tr}(\mathbf{G}_{h,t}\Delta\mathbf{Q}_t)\overset{(a)}{=}\sum_i h(p_{t i})\frac{\mathbf{x}_i^{\mathsf T}\Delta\mathbf{Q}_t\mathbf{x}_i}{r_{t i}}\overset{(b)}{\geq}\frac{\alpha}{2T\sqrt{2(d+1)\lambda_t}}\sum_i\rho_i\mathbf{x}_i^{\mathsf T}\Delta\mathbf{Q}_t\mathbf{x}_i\overset{(c)}{=}\frac{\alpha\operatorname{tr}(\Delta\mathbf{Q}_t)}{2T\sqrt{2(d+1)\lambda_t}}, tr ( G h , t Δ Q t ) = ( a ) i ∑ h ( p t i ) r t i x i T Δ Q t x i ≥ ( b ) 2 T 2 ( d + 1 ) λ t α i ∑ ρ i x i T Δ Q t x i = ( c ) 2 T 2 ( d + 1 ) λ t α tr ( Δ Q t ) ,
where (a) is the definition of G h , t \mathbf{G}_{h,t} G h , t ; (b) uses h ( p t i ) ≥ α p t i ≥ α ρ i / ( 2 T ) h(p_{t i})\geq\alpha p_{t i}\geq\alpha\rho_i/(2T) h ( p t i ) ≥ α p t i ≥ α ρ i / ( 2 T ) (Lemma B.5 and p t ∈ P \mathbf{p}_t\in\mathcal{P} p t ∈ P ), r t i ≤ ∥ x i ∥ λ t ≤ 2 ( d + 1 ) λ t r_{t i}\leq\lVert\mathbf{x}_i\rVert\sqrt{\lambda_t}\leq\sqrt{2(d+1)\lambda_t} r t i ≤ ∥ x i ∥ λ t ≤ 2 ( d + 1 ) λ t (Lemma B.6(c)) and x i T Δ Q t x i ≥ 0 \mathbf{x}_i^{\mathsf T}\Delta\mathbf{Q}_t\mathbf{x}_i\geq0 x i T Δ Q t x i ≥ 0 ; and (c) uses ∑ i ρ i x i x i T = I \sum_i\rho_i\mathbf{x}_i\mathbf{x}_i^{\mathsf T}=\mathbf{I} ∑ i ρ i x i x i T = I (Lemma B.6(b)). Combining the two bounds,
tr ( Δ Q t ) ≤ κ α 2 T ( d + 1 ) 2 ( d + 1 ) λ t ≤ 4 2 ( d + 1 ) 3 / 2 T s t = b T s t , \operatorname{tr}(\Delta\mathbf{Q}_t)\leq\frac{\kappa}{\alpha}2T(d+1)\sqrt{2(d+1)\lambda_t}\leq4\sqrt{2}(d+1)^{3/2}T s_t=b_Ts_t, tr ( Δ Q t ) ≤ α κ 2 T ( d + 1 ) 2 ( d + 1 ) λ t ≤ 4 2 ( d + 1 ) 3/2 T s t = b T s t ,
where the first inequality rearranges the two displays and the second uses κ / α ≤ 2 \kappa/\alpha\leq2 κ / α ≤ 2 and s t = λ t s_t=\sqrt{\lambda_t} s t = λ t . The largest eigenvalue therefore grows slowly:
s t 2 = λ t ≤ ( a ) λ t − 1 + λ max ( Δ Q t ) ≤ ( b ) λ t − 1 + tr ( Δ Q t ) ≤ ( c ) s t − 1 2 + b T s t , s_t^2=\lambda_t\overset{(a)}{\leq}\lambda_{t-1}+\lambda_{\max}(\Delta\mathbf{Q}_t)\overset{(b)}{\leq}\lambda_{t-1}+\operatorname{tr}(\Delta\mathbf{Q}_t)\overset{(c)}{\leq}s_{t-1}^2+b_Ts_t, s t 2 = λ t ≤ ( a ) λ t − 1 + λ m a x ( Δ Q t ) ≤ ( b ) λ t − 1 + tr ( Δ Q t ) ≤ ( c ) s t − 1 2 + b T s t ,
where (a) is Weyl’s inequality (Lemma E.2 ), (b) uses Δ Q t ⪰ 0 \Delta\mathbf{Q}_t\succeq0 Δ Q t ⪰ 0 , and (c) is the previous display. Hence
s t − s t − 1 = ( a ) s t 2 − s t − 1 2 s t + s t − 1 ≤ ( b ) b T s t s t + s t − 1 ≤ ( c ) b T , s_t-s_{t-1}\overset{(a)}{=}\frac{s_t^2-s_{t-1}^2}{s_t+s_{t-1}}\overset{(b)}{\leq}\frac{b_Ts_t}{s_t+s_{t-1}}\overset{(c)}{\leq}b_T, s t − s t − 1 = ( a ) s t + s t − 1 s t 2 − s t − 1 2 ≤ ( b ) s t + s t − 1 b T s t ≤ ( c ) b T ,
where (a) multiplies and divides by s t + s t − 1 > 0 s_t+s_{t-1}>0 s t + s t − 1 > 0 , (b) is the previous display, and (c) uses s t − 1 ≥ 0 s_{t-1}\geq0 s t − 1 ≥ 0 . Summing over t t t and using s 0 = 1 s_0=1 s 0 = 1 gives s T ≤ 1 + T b T = 1 + 4 2 ( d + 1 ) 3 / 2 T 2 s_T\leq1+Tb_T=1+4\sqrt{2}(d+1)^{3/2}T^2 s T ≤ 1 + T b T = 1 + 4 2 ( d + 1 ) 3/2 T 2 , and
log det Q T ≤ ( a ) ( d + 1 ) log λ T = ( b ) 2 ( d + 1 ) log s T ≤ ( c ) 2 ( d + 1 ) log ( 1 + 4 2 ( d + 1 ) 3 / 2 T 2 ) , \log\det\mathbf{Q}_T\overset{(a)}{\leq}(d+1)\log\lambda_T\overset{(b)}{=}2(d+1)\log s_T\overset{(c)}{\leq}2(d+1)\log\left(1+4\sqrt{2}(d+1)^{3/2}T^2\right), log det Q T ≤ ( a ) ( d + 1 ) log λ T = ( b ) 2 ( d + 1 ) log s T ≤ ( c ) 2 ( d + 1 ) log ( 1 + 4 2 ( d + 1 ) 3/2 T 2 ) ,
where (a) uses that each of the d + 1 d+1 d + 1 eigenvalues of Q T \mathbf{Q}_T Q T is at most λ T \lambda_T λ T , (b) uses λ T = s T 2 \lambda_T=s_T^2 λ T = s T 2 , and (c) uses the bound on s T s_T s T . This is the first claim of item (iii). The second follows because
2 ( d + 1 ) + 1 2 log det Q T ≤ 2 ( d + 1 ) + ( d + 1 ) log ( 1 + 4 2 ( d + 1 ) 3 / 2 T 2 ) = A T . \sqrt{2(d+1)}+\frac{1}{2}\log\det\mathbf{Q}_T\leq\sqrt{2(d+1)}+(d+1)\log\left(1+4\sqrt{2}(d+1)^{3/2}T^2\right)=A_T. 2 ( d + 1 ) + 2 1 log det Q T ≤ 2 ( d + 1 ) + ( d + 1 ) log ( 1 + 4 2 ( d + 1 ) 3/2 T 2 ) = A T .
Proofs of the Saddle-Point Lemmas in Appendix C
The Inner Problem
Proof of Lemma C.1. We first show that J J J is coercive on { Q ⪰ D } \{\mathbf{Q} \succeq\mathbf{D}\} { Q ⪰ D } , which gives existence, then that it is strictly convex along a feasible path between two minimizers, which gives uniqueness, and finally continuity.
Let Q ⪰ D ⪰ I \mathbf{Q} \succeq\mathbf{D} \succeq\mathbf{I} Q ⪰ D ⪰ I and λ = λ max ( Q ) ≥ 1 \lambda= \lambda_{\max}(\mathbf{Q}) \ge1 λ = λ m a x ( Q ) ≥ 1 . For every j j j , r j ( Q ) = x j T Q x j ≤ ∥ x j ∥ λ ≤ 2 ( d + 1 ) λ r_j(\mathbf{Q}) = \sqrt{\mathbf{x}_j^{\mathsf{T}}\mathbf{Q}\mathbf{x}_j} \le\lVert\mathbf{x}_j\rVert\sqrt{\lambda} \le\sqrt{2(d+1)\lambda} r j ( Q ) = x j T Q x j ≤ ∥ x j ∥ λ ≤ 2 ( d + 1 ) λ by Lemma B.6(c). Therefore
∑ i ρ i r i ( Q ) ≥ ∑ i ρ i r i ( Q ) 2 max j r j ( Q ) ≥ tr ( Q ∑ i ρ i x i x i T ) max j r j ( Q ) ≥ tr ( Q ) max j r j ( Q ) ≥ λ 2 ( d + 1 ) λ = λ 2 ( d + 1 ) , \sum_i \rho_i r_i(\mathbf{Q}) \ge\frac{\sum_i \rho_i r_i(\mathbf{Q})^2}{\max_j r_j(\mathbf{Q})} \ge\frac{\operatorname{tr}\left(\mathbf{Q}\sum_i \rho_i\mathbf{x}_i\mathbf{x}_i^{\mathsf{T}}\right)}{\max_j r_j(\mathbf{Q})} \ge\frac{\operatorname{tr}(\mathbf{Q})}{\max_j r_j(\mathbf{Q})} \ge\frac{\lambda}{\sqrt{2(d+1)\lambda}} = \sqrt{\frac{\lambda}{2(d+1)}}, i ∑ ρ i r i ( Q ) ≥ max j r j ( Q ) ∑ i ρ i r i ( Q ) 2 ≥ max j r j ( Q ) tr ( Q ∑ i ρ i x i x i T ) ≥ max j r j ( Q ) tr ( Q ) ≥ 2 ( d + 1 ) λ λ = 2 ( d + 1 ) λ ,
where (a) uses r i ( Q ) ≤ max j r j ( Q ) r_i(\mathbf{Q}) \le\max_j r_j(\mathbf{Q}) r i ( Q ) ≤ max j r j ( Q ) ; (b) uses r i ( Q ) 2 = tr ( Q x i x i T ) r_i(\mathbf{Q})^2 = \operatorname{tr}(\mathbf{Q}\mathbf{x}_i\mathbf{x}_i^{\mathsf{T}}) r i ( Q ) 2 = tr ( Q x i x i T ) ; (c) is Lemma B.6(b); and (d) uses tr ( Q ) ≥ λ max ( Q ) = λ \operatorname{tr}(\mathbf{Q}) \ge\lambda_{\max}(\mathbf{Q}) = \lambda tr ( Q ) ≥ λ m a x ( Q ) = λ , since Q ⪰ 0 \mathbf{Q} \succeq0 Q ⪰ 0 , and the bound on r j ( Q ) r_j(\mathbf{Q}) r j ( Q ) . Consequently,
J ( Q , p ) ≥ α 2 T ∑ i ρ i r i ( Q ) − κ ( d + 1 ) 2 log λ ≥ α 2 T λ 2 ( d + 1 ) − κ ( d + 1 ) 2 log λ . (22) J(\mathbf{Q},\mathbf{p}) \ge\frac{\alpha}{2T}\sum_i \rho_i r_i(\mathbf{Q}) - \frac{\kappa(d+1)}{2}\log\lambda\ge\frac{\alpha}{2T}\sqrt{\frac{\lambda}{2(d+1)}} - \frac{\kappa(d+1)}{2}\log\lambda. \tag*{(22)} J ( Q , p ) ≥ 2 T α i ∑ ρ i r i ( Q ) − 2 κ ( d + 1 ) log λ ≥ 2 T α 2 ( d + 1 ) λ − 2 κ ( d + 1 ) log λ . (22)
where the first inequality uses h ( p i ) ≥ α p i ≥ α ρ i / ( 2 T ) h(p_i) \ge\alpha p_i \ge\alpha\rho_i/(2T) h ( p i ) ≥ α p i ≥ α ρ i / ( 2 T ) (Lemma B.5 and p ∈ P \mathbf{p} \in\mathcal{P} p ∈ P ), r i ( Q ) ≥ 0 r_i(\mathbf{Q}) \ge0 r i ( Q ) ≥ 0 , and log det Q ≤ ( d + 1 ) log λ \log\det\mathbf{Q} \le(d+1)\log\lambda log det Q ≤ ( d + 1 ) log λ , which holds because the d + 1 d+1 d + 1 eigenvalues of Q \mathbf{Q} Q are at most λ \lambda λ ; and the second is the previous display. At Q = D \mathbf{Q} = \mathbf{D} Q = D , on the other hand,
J ( D , p ) ≤ ∑ i h ( p i ) r i ( D ) ≤ κ ∑ i p i r i ( D ) ≤ κ 2 ( d + 1 ) λ max ( D ) , J(\mathbf{D},\mathbf{p}) \le\sum_i h(p_i)r_i(\mathbf{D}) \le\kappa\sum_i p_i r_i(\mathbf{D}) \le\kappa\sqrt{2(d+1)}\lambda_{\max}(\mathbf{D}), J ( D , p ) ≤ i ∑ h ( p i ) r i ( D ) ≤ κ i ∑ p i r i ( D ) ≤ κ 2 ( d + 1 ) λ m a x ( D ) ,
where (a) uses log det D ≥ 0 \log\det\mathbf{D} \ge0 log det D ≥ 0 , since D ⪰ I \mathbf{D} \succeq\mathbf{I} D ⪰ I ; (b) uses h ( p i ) ≤ κ p i h(p_i) \le\kappa p_i h ( p i ) ≤ κ p i (Lemma B.5); and (c) uses r i ( D ) ≤ 2 ( d + 1 ) λ max ( D ) r_i(\mathbf{D}) \le\sqrt{2(d+1)}\lambda_{\max}(\mathbf{D}) r i ( D ) ≤ 2 ( d + 1 ) λ m a x ( D ) and ∑ i p i = 1 \sum_i p_i = 1 ∑ i p i = 1 . As the right side of (D.1) tends to infinity with λ \lambda λ , every Q ⪰ D \mathbf{Q} \succeq\mathbf{D} Q ⪰ D with J ( Q , p ) ≤ J ( D , p ) J(\mathbf{Q},\mathbf{p}) \le J(\mathbf{D},\mathbf{p}) J ( Q , p ) ≤ J ( D , p ) satisfies λ max ( Q ) ≤ λ ˉ \lambda_{\max}(\mathbf{Q}) \le\bar{\lambda} λ m a x ( Q ) ≤ λ ˉ for some λ ˉ \bar{\lambda} λ ˉ that depends only on λ max ( D ) \lambda_{\max}(\mathbf{D}) λ m a x ( D ) and the constants. We may therefore restrict the minimization to the compact set { Q ⪰ D : λ max ( Q ) ≤ λ ˉ } \{\mathbf{Q} \succeq\mathbf{D} : \lambda_{\max}(\mathbf{Q}) \le\bar{\lambda}\} { Q ⪰ D : λ m a x ( Q ) ≤ λ ˉ } , on which J J J is continuous, so a minimizer exists.
For uniqueness, suppose that Q ( 0 ) ≠ Q ( 1 ) \mathbf{Q}^{(0)} \ne\mathbf{Q}^{(1)} Q ( 0 ) = Q ( 1 ) are both minimizers, and consider the path
Q ( s ) = ( Q ( 0 ) ) 1 / 2 M s ( Q ( 0 ) ) 1 / 2 , s ∈ [ 0 , 1 ] , where M = ( Q ( 0 ) ) − 1 / 2 Q ( 1 ) ( Q ( 0 ) ) − 1 / 2 . \mathbf{Q}^{(s)} = (\mathbf{Q}^{(0)})^{1/2}\mathbf{M}^{s}(\mathbf{Q}^{(0)})^{1/2}, \qquad s \in[0,1], \qquad\text{where } \mathbf{M} = (\mathbf{Q}^{(0)})^{-1/2}\mathbf{Q}^{(1)}(\mathbf{Q}^{(0)})^{-1/2}. Q ( s ) = ( Q ( 0 ) ) 1/2 M s ( Q ( 0 ) ) 1/2 , s ∈ [ 0 , 1 ] , where M = ( Q ( 0 ) ) − 1/2 Q ( 1 ) ( Q ( 0 ) ) − 1/2 .
In the notation of Lemma E.4 , Q ( s ) = Q ( 0 ) # s Q ( 1 ) \mathbf{Q}^{(s)} = \mathbf{Q}^{(0)} \#_s \mathbf{Q}^{(1)} Q ( s ) = Q ( 0 ) # s Q ( 1 ) . Since Q ( 0 ) , Q ( 1 ) ⪰ D \mathbf{Q}^{(0)}, \mathbf{Q}^{(1)} \succeq\mathbf{D} Q ( 0 ) , Q ( 1 ) ⪰ D , the monotonicity of the weighted geometric mean gives Q ( s ) ⪰ D # s D = D \mathbf{Q}^{(s)} \succeq\mathbf{D} \#_s \mathbf{D} = \mathbf{D} Q ( s ) ⪰ D # s D = D , so the path is feasible. Along it, the log-det term is affine in s s s , because log det Q ( s ) = log det Q ( 0 ) + s log det M \log\det\mathbf{Q}^{(s)} = \log\det\mathbf{Q}^{(0)} + s\log\det\mathbf{M} log det Q ( s ) = log det Q ( 0 ) + s log det M and log det M = log det Q ( 1 ) − log det Q ( 0 ) \log\det\mathbf{M} = \log\det\mathbf{Q}^{(1)} - \log\det\mathbf{Q}^{(0)} log det M = log det Q ( 1 ) − log det Q ( 0 ) . The radii, in contrast, are convex along the path. Writing M = ∑ k μ k u k u k T \mathbf{M} = \sum_k \mu_k\mathbf{u}_k\mathbf{u}_k^{\mathsf{T}} M = ∑ k μ k u k u k T and setting w i = ( Q ( 0 ) ) 1 / 2 x i \mathbf{w}_i = (\mathbf{Q}^{(0)})^{1/2}\mathbf{x}_i w i = ( Q ( 0 ) ) 1/2 x i , a i k = ( u k T w i ) 2 a_{ik} = (\mathbf{u}_k^{\mathsf{T}}\mathbf{w}_i)^2 a ik = ( u k T w i ) 2 and ℓ k = log μ k \ell_k = \log\mu_k ℓ k = log μ k , we have
r i ( Q ( s ) ) 2 = w i T M s w i = ∑ k μ k s ( u k T w i ) 2 = ∑ k a i k e s ℓ k , so r i ( Q ( s ) ) = exp ( 1 2 log ∑ k a i k e s ℓ k ) , r_i(\mathbf{Q}^{(s)})^2 = \mathbf{w}_i^{\mathsf{T}}\mathbf{M}^{s}\mathbf{w}_i = \sum_k \mu_k^s(\mathbf{u}_k^{\mathsf{T}}\mathbf{w}_i)^2 = \sum_k a_{ik}e^{s\ell_k}, \qquad\text{so} \qquad r_i(\mathbf{Q}^{(s)}) = \exp\left(\frac{1}{2}\log\sum_k a_{ik}e^{s\ell_k}\right), r i ( Q ( s ) ) 2 = w i T M s w i = k ∑ μ k s ( u k T w i ) 2 = k ∑ a ik e s ℓ k , so r i ( Q ( s ) ) = exp ( 2 1 log k ∑ a ik e s ℓ k ) ,
where (a) uses the definitions of Q ( s ) \mathbf{Q}^{(s)} Q ( s ) and w i \mathbf{w}_i w i , (b) uses M s = ∑ k μ k s u k u k T \mathbf{M}^{s} = \sum_k \mu_k^{s}\mathbf{u}_k\mathbf{u}_k^{\mathsf{T}} M s = ∑ k μ k s u k u k T , and (c) uses μ k s = e s ℓ k \mu_k^s = e^{s\ell_k} μ k s = e s ℓ k . Since log-sum-exp is convex and exp is increasing and convex, s ↦ r i ( Q ( s ) ) s \mapsto r_i(\mathbf{Q}^{(s)}) s ↦ r i ( Q ( s ) ) is convex. At least one radius is strictly convex. Since M ≠ I \mathbf{M} \ne\mathbf{I} M = I , some ℓ k \ell_k ℓ k is nonzero, and since { w i } \{\mathbf{w}_i\} { w i } spans R d + 1 \mathbb{R}^{d+1} R d + 1 , some i i i has a i k > 0 a_{ik} > 0 a ik > 0 for such a k k k . For this i i i , if the values { ℓ k : a i k > 0 } \{\ell_k : a_{ik} > 0\} { ℓ k : a ik > 0 } are not all equal, log-sum-exp is strictly convex, and otherwise it equals a constant plus s ℓ s\ell s ℓ with ℓ ≠ 0 \ell\ne0 ℓ = 0 ; in both cases r i ( Q ( s ) ) r_i(\mathbf{Q}^{(s)}) r i ( Q ( s ) ) is strictly convex. As h ( p i ) > 0 h(p_i) > 0 h ( p i ) > 0 and the log-det part is affine, the map s ↦ J ( Q ( s ) , p ) s \mapsto J(\mathbf{Q}^{(s)},\mathbf{p}) s ↦ J ( Q ( s ) , p ) is strictly convex. Hence J ( Q ( 1 / 2 ) , p ) < 1 2 ( J ( Q ( 0 ) , p ) + J ( Q ( 1 ) , p ) ) = min Q ⪰ D J ( Q , p ) J(\mathbf{Q}^{(1/2)},\mathbf{p}) < \frac{1}{2}(J(\mathbf{Q}^{(0)},\mathbf{p}) + J(\mathbf{Q}^{(1)},\mathbf{p})) = \min_{\mathbf{Q} \succeq\mathbf{D}} J(\mathbf{Q},\mathbf{p}) J ( Q ( 1/2 ) , p ) < 2 1 ( J ( Q ( 0 ) , p ) + J ( Q ( 1 ) , p )) = min Q ⪰ D J ( Q , p ) , which contradicts minimality, so the minimizer Q ( D , p ) \mathbf{Q}(\mathbf{D},\mathbf{p}) Q ( D , p ) is unique.
Finally, let ( D n , p n ) → ( D , p ) (\mathbf{D}_n,\mathbf{p}_n) \to(\mathbf{D},\mathbf{p}) ( D n , p n ) → ( D , p ) and Q n = Q ( D n , p n ) \mathbf{Q}_n = \mathbf{Q}(\mathbf{D}_n,\mathbf{p}_n) Q n = Q ( D n , p n ) . The coercivity bound (D.1) and the bound at D n \mathbf{D}_n D n show that λ max ( Q n ) \lambda_{\max}(\mathbf{Q}_n) λ m a x ( Q n ) is uniformly bounded, so a subsequence converges to some Q ‾ ⪰ D \overline{\mathbf{Q}} \succeq\mathbf{D} Q ⪰ D . For any Q ′ ⪰ D \mathbf{Q}' \succeq\mathbf{D} Q ′ ⪰ D , the matrices Q n ′ = Q ′ + ∥ D n − D ∥ o p I \mathbf{Q}'_n = \mathbf{Q}' + \lVert\mathbf{D}_n - \mathbf{D} \rVert_{\mathrm{op}} \mathbf{I} Q n ′ = Q ′ + ∥ D n − D ∥ op I satisfy Q n ′ ⪰ D n \mathbf{Q}'_n \succeq\mathbf{D}_n Q n ′ ⪰ D n , Q n ′ → Q ′ \mathbf{Q}'_n \to\mathbf{Q}' Q n ′ → Q ′ and J ( Q n , p n ) ≤ J ( Q n ′ , p n ) J(\mathbf{Q}_n,\mathbf{p}_n) \le J(\mathbf{Q}'_n,\mathbf{p}_n) J ( Q n , p n ) ≤ J ( Q n ′ , p n ) . Since J J J is jointly continuous on { Q ⪰ I } × P \{\mathbf{Q} \succeq\mathbf{I}\} \times\mathcal{P} { Q ⪰ I } × P , passing to the limit gives J ( Q ‾ , p ) ≤ J ( Q ′ , p ) J(\overline{\mathbf{Q}},\mathbf{p}) \le J(\mathbf{Q}',\mathbf{p}) J ( Q , p ) ≤ J ( Q ′ , p ) , and hence Q ‾ = Q ( D , p ) \overline{\mathbf{Q}} = \mathbf{Q}(\mathbf{D},\mathbf{p}) Q = Q ( D , p ) by uniqueness. Every subsequential limit is the same, so the whole sequence converges.
The Outer Problem
Proof of Lemma C.2. We prove parts (a) and (b) in turn.
Proof of (a). We show that the outer objective is continuous and strongly concave. Let
Φ D ( p ) : = min Q ⪰ D J ( Q , p ) = J ( Q ( D , p ) , p ) , \Phi_{\mathbf{D}}(\mathbf{p}) := \min_{\mathbf{Q} \succeq\mathbf{D}} J(\mathbf{Q},\mathbf{p}) = J(\mathbf{Q}(\mathbf{D},\mathbf{p}),\mathbf{p}), Φ D ( p ) := Q ⪰ D min J ( Q , p ) = J ( Q ( D , p ) , p ) ,
which is jointly continuous in ( D , p ) (\mathbf{D},\mathbf{p}) ( D , p ) by Lemma C.1. For a fixed Q ⪰ I \mathbf{Q} \succeq\mathbf{I} Q ⪰ I , the map p ↦ J ( Q , p ) + ∥ p ∥ 2 / ( 2 η ) \mathbf{p} \mapsto J(\mathbf{Q},\mathbf{p}) + \lVert\mathbf{p} \rVert^2/(2\eta) p ↦ J ( Q , p ) + ∥ p ∥ 2 / ( 2 η ) is a sum of functions of the individual coordinates, and its second derivative in p i p_i p i is
r i ( Q ) h ′ ′ ( p i ) + 1 η = − r i ( Q ) η p i + 1 η = 1 − r i ( Q ) / p i η ≤ 0 , r_i(\mathbf{Q})h''(p_i) + \frac{1}{\eta} = -\frac{r_i(\mathbf{Q})}{\eta p_i} + \frac{1}{\eta} = \frac{1-r_i(\mathbf{Q})/p_i}{\eta} \le0, r i ( Q ) h ′′ ( p i ) + η 1 = − η p i r i ( Q ) + η 1 = η 1 − r i ( Q ) / p i ≤ 0 ,
where the first equality uses h ′ ′ ( s ) = − 1 / ( η s ) h''(s) = -1/(\eta s) h ′′ ( s ) = − 1/ ( η s ) and the inequality uses r i ( Q ) ≥ ∥ x i ∥ ≥ 1 ≥ p i r_i(\mathbf{Q}) \ge\lVert\mathbf{x}_i \rVert\ge1 \ge p_i r i ( Q ) ≥ ∥ x i ∥ ≥ 1 ≥ p i , which holds because Q ⪰ I \mathbf{Q} \succeq\mathbf{I} Q ⪰ I and by Lemma B.6(c). This map is therefore concave. A pointwise minimum of concave functions is concave, so Φ D + ∥ ⋅ ∥ 2 / ( 2 η ) \Phi_{\mathbf{D}} + \lVert\cdot\rVert^2/(2\eta) Φ D + ∥ ⋅ ∥ 2 / ( 2 η ) is concave as well, that is, Φ D \Phi_{\mathbf{D}} Φ D is 1 / η 1/\eta 1/ η -strongly concave. Hence p ↦ ⟨ S , p ⟩ + Φ D ( p ) \mathbf{p} \mapsto\langle\mathbf{S},\mathbf{p}\rangle+ \Phi_{\mathbf{D}}(\mathbf{p}) p ↦ ⟨ S , p ⟩ + Φ D ( p ) is continuous and strongly concave on the compact convex set P \mathcal{P} P and has a unique maximizer, and by Berge’s maximum theorem (Lemma E.3 ), applied on the compact set P \mathcal{P} P to the jointly continuous objective ⟨ S , p ⟩ + Φ D ( p ) \langle\mathbf{S},\mathbf{p}\rangle+ \Phi_{\mathbf{D}}(\mathbf{p}) ⟨ S , p ⟩ + Φ D ( p ) , this maximizer depends continuously on ( S , D ) (\mathbf{S},\mathbf{D}) ( S , D ) .
Proof of (b). We show that p ⋆ \mathbf{p}^{\star} p ⋆ satisfies the first-order optimality condition of the frozen problem. Write L ( p , Q ) = ⟨ S , p ⟩ + J ( Q , p ) \mathcal{L}(\mathbf{p},\mathbf{Q}) = \langle\mathbf{S},\mathbf{p}\rangle+ J(\mathbf{Q},\mathbf{p}) L ( p , Q ) = ⟨ S , p ⟩ + J ( Q , p ) and Φ ~ ( p ) = ⟨ S , p ⟩ + Φ D ( p ) \widetilde{\Phi}(\mathbf{p}) = \langle\mathbf{S},\mathbf{p}\rangle+ \Phi_{\mathbf{D}}(\mathbf{p}) Φ ( p ) = ⟨ S , p ⟩ + Φ D ( p ) , take any q ∈ P \mathbf{q} \in\mathcal{P} q ∈ P , and for ε ∈ ( 0 , 1 ] \varepsilon\in(0,1] ε ∈ ( 0 , 1 ] let p ε = p ⋆ + ε ( q − p ⋆ ) \mathbf{p}_{\varepsilon} = \mathbf{p}^{\star} + \varepsilon(\mathbf{q} - \mathbf{p}^{\star}) p ε = p ⋆ + ε ( q − p ⋆ ) , which lies in P \mathcal{P} P by convexity, and Q ε = Q ( D , p ε ) \mathbf{Q}_{\varepsilon} = \mathbf{Q}(\mathbf{D},\mathbf{p}_{\varepsilon}) Q ε = Q ( D , p ε ) . Then
L ( p ε , Q ε ) = ( a ) Φ ~ ( p ε ) ≤ ( b ) Φ ~ ( p ⋆ ) = ( c ) L ( p ⋆ , Q ⋆ ) ≤ ( d ) L ( p ⋆ , Q ε ) , \mathcal{L}(\mathbf{p}_{\varepsilon},\mathbf{Q}_{\varepsilon})
\mathrel{\overset{(a)}{=}} \widetilde{\Phi}(\mathbf{p}_{\varepsilon})
\mathrel{\overset{(b)}{\le}} \widetilde{\Phi}(\mathbf{p}^{\star})
\mathrel{\overset{(c)}{=}} \mathcal{L}(\mathbf{p}^{\star},\mathbf{Q}^{\star})
\mathrel{\overset{(d)}{\le}} \mathcal{L}(\mathbf{p}^{\star},\mathbf{Q}_{\varepsilon}), L ( p ε , Q ε ) = ( a ) Φ ( p ε ) ≤ ( b ) Φ ( p ⋆ ) = ( c ) L ( p ⋆ , Q ⋆ ) ≤ ( d ) L ( p ⋆ , Q ε ) ,
where (a) and (c) use that Q ε \mathbf{Q}_{\varepsilon} Q ε and Q ⋆ \mathbf{Q}^{\star} Q ⋆ minimize J ( ⋅ , p ε ) J(\cdot,\mathbf{p}_{\varepsilon}) J ( ⋅ , p ε ) and J ( ⋅ , p ⋆ ) J(\cdot,\mathbf{p}^{\star}) J ( ⋅ , p ⋆ ) over { Q ⪰ D } \{\mathbf{Q} \succeq\mathbf{D}\} { Q ⪰ D } ; (b) uses that p ⋆ \mathbf{p}^{\star} p ⋆ maximizes Φ ~ \widetilde{\Phi} Φ over P \mathcal{P} P (part (a)); and (d) uses that Q ⋆ \mathbf{Q}^{\star} Q ⋆ minimizes J ( ⋅ , p ⋆ ) J(\cdot,\mathbf{p}^{\star}) J ( ⋅ , p ⋆ ) and Q ε ⪰ D \mathbf{Q}_{\varepsilon} \succeq\mathbf{D} Q ε ⪰ D is feasible. Hence L ( p ε , Q ε ) − L ( p ⋆ , Q ε ) ≤ 0 \mathcal{L}(\mathbf{p}_{\varepsilon},\mathbf{Q}_{\varepsilon}) - \mathcal{L}(\mathbf{p}^{\star},\mathbf{Q}_{\varepsilon}) \le0 L ( p ε , Q ε ) − L ( p ⋆ , Q ε ) ≤ 0 . By the mean value theorem, this difference equals ε ⟨ ∇ p L ( p ε , Q ε ) , q − p ⋆ ⟩ \varepsilon\langle\nabla_{\mathbf{p}}\mathcal{L}(\mathbf{p}_{\varepsilon},\mathbf{Q}_{\varepsilon}),\mathbf{q} - \mathbf{p}^{\star}\rangle ε ⟨ ∇ p L ( p ε , Q ε ) , q − p ⋆ ⟩ for some p ε \mathbf{p}_{\varepsilon} p ε on the segment between p ⋆ \mathbf{p}^{\star} p ⋆ and p ε \mathbf{p}_{\varepsilon} p ε . We divide by ε \varepsilon ε and let ε → 0 \varepsilon\to0 ε → 0 . Since Q ε → Q ⋆ \mathbf{Q}_{\varepsilon} \to\mathbf{Q}^{\star} Q ε → Q ⋆ by Lemma C.1 and ∂ p i L = S i + h ′ ( p i ) r i ( Q ) \partial_{p_i}\mathcal{L} = S_i + h'(p_i)r_i(\mathbf{Q}) ∂ p i L = S i + h ′ ( p i ) r i ( Q ) is jointly continuous, we obtain
⟨ ∇ p L ( p ⋆ , Q ⋆ ) , q − p ⋆ ⟩ ≤ 0 for all q ∈ P . (23) \langle\nabla_{\mathbf{p}}\mathcal{L}(\mathbf{p}^{\star},\mathbf{Q}^{\star}),\mathbf{q} - \mathbf{p}^{\star}\rangle\le0 \qquad\text{for all } \mathbf{q} \in\mathcal{P}. \tag*{(23)} ⟨ ∇ p L ( p ⋆ , Q ⋆ ) , q − p ⋆ ⟩ ≤ 0 for all q ∈ P . (23)
The map p ↦ L ( p , Q ⋆ ) \mathbf{p} \mapsto\mathcal{L}(\mathbf{p},\mathbf{Q}^{\star}) p ↦ L ( p , Q ⋆ ) is strictly concave, since h h h is strictly concave and r i ( Q ⋆ ) > 0 r_i(\mathbf{Q}^{\star}) > 0 r i ( Q ⋆ ) > 0 , and the last display is its first-order optimality condition on P P P ; so p ⋆ \mathbf{p}^{\star} p ⋆ is its unique maximizer. Finally, L ( p , Q ⋆ ) = ⟨ S , p ⟩ + ∑ i r i ( Q ⋆ ) h ( p i ) − K 2 log det Q ⋆ \mathcal{L}(\mathbf{p},\mathbf{Q}^{\star}) = \langle\mathbf{S},\mathbf{p}\rangle+ \sum_i r_i(\mathbf{Q}^{\star})h(p_i) - \frac{K}{2}\log\det\mathbf{Q}^{\star} L ( p , Q ⋆ ) = ⟨ S , p ⟩ + ∑ i r i ( Q ⋆ ) h ( p i ) − 2 K log det Q ⋆ , and the last term does not depend on p \mathbf{p} p , which proves part (b).
Auxiliary Lemmas
Lemma E.1 (Azuma–Hoeffding Inequality (Hoeffding, [23 ] ; Azuma [5 ] ).) Let ( M t ) t ∈ [ T ] (M_t)_{t\in[T]} ( M t ) t ∈ [ T ] be a martingale difference sequence with respect to a filtration ( F t ) (\mathcal{F}_t) ( F t ) , with ∣ M t ∣ ≤ 1 |M_t| \le1 ∣ M t ∣ ≤ 1 for every t t t . Then for every δ ′ ∈ ( 0 , 1 ) \delta' \in(0,1) δ ′ ∈ ( 0 , 1 ) , with probability at least 1 − δ ′ 1-\delta' 1 − δ ′ ,
∑ t = 1 T M t ≤ 2 T log ( 1 / δ ′ ) . \sum_{t=1}^{T} M_t \le\sqrt{2T\log(1/\delta')}. t = 1 ∑ T M t ≤ 2 T log ( 1/ δ ′ ) .
Lemma E.2 (Weyl’s Inequality (Horn and Johnson, [24 ] , Theorem 4.3.1).) For symmetric matrices A \mathbf{A} A and B \mathbf{B} B of the same size, λ max ( A + B ) ≤ λ max ( A ) + λ max ( B ) \lambda_{\max}(\mathbf{A}+\mathbf{B}) \le\lambda_{\max}(\mathbf{A})+\lambda_{\max}(\mathbf{B}) λ m a x ( A + B ) ≤ λ m a x ( A ) + λ m a x ( B ) .
Lemma E.3 (Berge’s Maximum Theorem (see Berge, [7 ] ; Aliprantis and Border [2 ] , Theorem 17.31).) Let X X X be a compact subset of R K \mathbb{R}^{K} R K , let Θ \Theta Θ be a metric space, and let f : X × Θ → R f:X\times\Theta\to\mathbb{R} f : X × Θ → R be continuous. If, for every θ ∈ Θ \theta\in\Theta θ ∈ Θ , the function f ( ⋅ , θ ) f(\cdot,\theta) f ( ⋅ , θ ) has a unique maximizer x ⋆ ( θ ) \mathbf{x}^{\star}(\theta) x ⋆ ( θ ) over X X X , then θ ↦ x ⋆ ( θ ) \theta\mapsto\mathbf{x}^{\star}(\theta) θ ↦ x ⋆ ( θ ) is continuous.
Lemma E.4 (Monotonicity of the Weighted Geometric Mean (see Kubo and Ando, [27 ] ; Bhatia [9 ] , Chapter 4).) For positive definite matrices A \mathbf{A} A and B \mathbf{B} B and s ∈ [ 0 , 1 ] s\in[0,1] s ∈ [ 0 , 1 ] , let A # s B = A 1 / 2 ( A − 1 / 2 B A − 1 / 2 ) s A 1 / 2 \mathbf{A}\#_{s}\mathbf{B}=\mathbf{A}^{1/2}(\mathbf{A}^{-1/2}\mathbf{B}\mathbf{A}^{-1/2})^{s}\mathbf{A}^{1/2} A # s B = A 1/2 ( A − 1/2 B A − 1/2 ) s A 1/2 . Then A # s A = A \mathbf{A}\#_{s}\mathbf{A}=\mathbf{A} A # s A = A , and A ⪯ A ′ \mathbf{A}\preceq\mathbf{A}' A ⪯ A ′ , B ⪯ B ′ \mathbf{B}\preceq\mathbf{B}' B ⪯ B ′ imply A # s B ⪯ A ′ # s B ′ \mathbf{A}\#_{s}\mathbf{B}\preceq\mathbf{A}'\#_{s}\mathbf{B}' A # s B ⪯ A ′ # s B ′ .