Introduction
A learner interacting with a bandit environment chooses actions sequentially to earn rewards while gathering the information needed to improve future decisions. We consider this exploration–exploitation trade-off in the stochastic linear bandit setting. Fix an unknown parameter θ ⋆ ∈ R d \theta^\star\in\mathbb{R}^d θ ⋆ ∈ R d X \mathcal{X} X R d \mathbb{R}^d R d T ≥ 1 T \ge1 T ≥ 1 t ∈ { 1 , … , T } t \in\{1,\ldots,T\} t ∈ { 1 , … , T } x t ∈ X x_t \in\mathcal{X} x t ∈ X y t = ⟨ θ ⋆ , x t ⟩ + η t y_t = \langle\theta^\star,x_t\rangle+ \eta_t y t = ⟨ θ ⋆ , x t ⟩ + η t η t \eta_t η t T T T
R T : = ∑ t = 1 T ⟨ θ ⋆ , x ⋆ − x t ⟩ , x ⋆ ∈ arg max x ∈ X ⟨ θ ⋆ , x ⟩ . R_T := \sum_{t=1}^{T}\langle\theta^\star,x^\star-x_t\rangle,\qquad x^\star\in\arg\max_{x\in\mathcal{X}}\langle\theta^\star,x\rangle. R T := t = 1 ∑ T ⟨ θ ⋆ , x ⋆ − x t ⟩ , x ⋆ ∈ arg x ∈ X max ⟨ θ ⋆ , x ⟩ . LinUCB addresses this trade-off using upper confidence bounds on rewards [9 , 21 , 1 ] . We study α \alpha α α , λ 0 > 0 \alpha,\lambda_0 > 0 α , λ 0 > 0 t t t x t x_t x t
( x t , θ t ) ∈ arg max ( x , θ ) ∈ X × C t ( α ) ⟨ θ , x ⟩ , C t ( α ) : = { θ ∈ R d : ∥ θ − θ ^ t ∥ V t ≤ α } . (x_t,\theta_t) \in\arg\max_{(x,\theta)\in\mathcal{X}\times\mathcal{C}_t(\alpha)}\langle\theta,x\rangle,\qquad\mathcal{C}_t(\alpha):=\left\{\theta\in\mathbb{R}^d:\|\theta-\hat{\theta}_t\|_{V_t}\le\alpha\right\}. ( x t , θ t ) ∈ arg ( x , θ ) ∈ X × C t ( α ) max ⟨ θ , x ⟩ , C t ( α ) := { θ ∈ R d : ∥ θ − θ ^ t ∥ V t ≤ α } . where θ ^ t = V t − 1 ∑ s = 1 t − 1 x s y s \hat{\theta}_t=V_t^{-1}\sum_{s=1}^{t-1}x_sy_s θ ^ t = V t − 1 ∑ s = 1 t − 1 x s y s θ ⋆ \theta^\star θ ⋆ V t : = λ 0 I + ∑ s = 1 t − 1 x s x s ⊤ V_t:=\lambda_0I+\sum_{s=1}^{t-1}x_sx_s^\top V t := λ 0 I + ∑ s = 1 t − 1 x s x s ⊤
x t ∈ arg max x ∈ X { ⟨ θ ^ t , x ⟩ + α ∥ x ∥ V t − 1 } . x_t\in\arg\max_{x\in\mathcal{X}}\left\{\langle\hat{\theta}_t,x\rangle+\alpha\|x\|_{V_t^{-1}}\right\}. x t ∈ arg x ∈ X max { ⟨ θ ^ t , x ⟩ + α ∥ x ∥ V t − 1 } . Thus, actions are selected through a combination of the mean and an α \alpha α ∥ x ∥ V t − 1 \|x\|_{V_t^{-1}} ∥ x ∥ V t − 1 x x x
Standard LinUCB chooses a confidence width β ≍ d log T \beta\asymp\sqrt{d\log T} β ≍ d log T θ ⋆ ∈ ⋂ t = 1 T C t ( β ) \theta^\star\in\bigcap_{t=1}^{T}\mathcal{C}_t(\beta) θ ⋆ ∈ ⋂ t = 1 T C t ( β ) ( x t , θ t ) (x_t,\theta_t) ( x t , θ t ) ⟨ θ ⋆ , x ⋆ ⟩ \langle\theta^\star,x^\star\rangle ⟨ θ ⋆ , x ⋆ ⟩ [1 ] . For arbitrary α > 0 \alpha>0 α > 0 α \alpha α α \alpha α Does LinUCB Need Valid Confidence Sets?
Under an additional curvature condition on the action set, we answer this question in the negative:
R T = O ( M m M α 2 + β 2 α d T ) , R_T=\mathcal{O}\left(\frac{M}{m}\frac{M\alpha^2+\beta^2}{\alpha}\sqrt{dT}\right), R T = O ( m M α M α 2 + β 2 d T ) , up to iterated-logarithmic factors (Theorem 4.2). The bound is sublinear for every fixed α > 0 \alpha>0 α > 0
A geometric exploration mechanism. We identify the mechanism through which α \alpha α α \alpha α t \sqrt{t} t λ min ( V t ) \lambda_{\min}(V_t) λ m i n ( V t )
A logarithmically capped potential. To analyse this exploration mechanism, we introduce a logarithmically capped spectral potential that, unlike the standard log-determinant potential, caps contributions from well-explored directions. The resulting spectral bounds yield regret control without the usual elliptical-potential argument. At the standard width α = β \alpha=\beta α = β O ( M m d T log T ) \mathcal{O}\left(\frac{M}{m}d\sqrt{T\log T}\right) O ( m M d T log T ) O ( d T log T ) \mathcal{O}(d\sqrt{T}\log T) O ( d T log T ) log T \sqrt{\log T} log T
Moreover, we construct a counterexample confirming that assumptions on the action set are necessary: on a two-dimensional polygon, α \alpha α α > 0 \alpha>0 α > 0 t \sqrt{t} t
Notation. We write [ T ] : = { 1 , … , T } [T]:=\{1,\ldots,T\} [ T ] := { 1 , … , T } I \mathbf{I} I R d × d \mathbb{R}^{d\times d} R d × d λ min ( V ) \lambda_{\min}(V) λ m i n ( V ) V V V u , v ∈ R d u,v\in\mathbb{R}^d u , v ∈ R d ⟨ u , v ⟩ \langle u,v\rangle ⟨ u , v ⟩ ∥ u ∥ 2 : = ⟨ u , u ⟩ \|u\|_2:=\sqrt{\langle u,u\rangle} ∥ u ∥ 2 := ⟨ u , u ⟩ ∥ u ∥ V : = ⟨ u , V u ⟩ \|u\|_V:=\sqrt{\langle u,Vu\rangle} ∥ u ∥ V := ⟨ u , V u ⟩ V ≻ 0 V\succ0 V ≻ 0 B 2 d ( r ) \mathbb{B}_2^d(r) B 2 d ( r ) d d d r r r
Related work
Our work is closest to that of Abeille et al. [3 ] , who use the same geometric condition to analyse Thompson sampling without forced optimism or posterior inflation. Banerjee et al. [6 ] also connect curvature with exploration, deriving growth of the smallest eigenvalue of the expected design matrix from an assumed O ( T ) \mathcal{O}(\sqrt{T}) O ( T )
Optimism and confidence sets in linear bandits. The classical O ( d T log ( T ) ) \mathcal{O}(d\sqrt{T}\log(T)) O ( d T log ( T )) [21 , 1 ] . Each contributes a log ( T ) \sqrt{\log(T)} log ( T ) Ω ( d T ) \Omega(d\sqrt{T}) Ω ( d T ) [9 , 26 , 20 ] . Lattimore [18 ] proves that the continuous-time counterpart of the self-normalised concentration bound cannot be improved for general adaptive covariates. Flynn et al. [12 ] construct tighter confidence sequences using adaptive martingale mixtures, improving empirical performance while retaining O ( d T log ( T ) ) O(d\sqrt{T}\log(T)) O ( d T log ( T ))
Classical frequentist analyses of linear Thompson sampling also rely on optimism: posterior inflation ensures a constant probability of an optimistic draw, at the cost of an additional d \sqrt{d} d [4 , 2 ] . For finite-armed linear bandits, Lattimore & Szepesvári [19 ] show that standard optimism-based algorithms can be suboptimal. Subsequent asymptotically optimal algorithms explicitly optimise exploration allocations to account for the information provided by suboptimal actions [13 , 28 ] .
Action-set curvature and exploration. When the action set is smooth and strongly convex, the played arm x t = ∇ J X ( θ t ) x_t = \nabla J_{\mathcal{X}}(\theta_t) x t = ∇ J X ( θ t ) [14 ] . In stochastic bandits, Banerjee et al. [6 ] quantify this: for action sets with bounded curvature around the optimal arm, any algorithm with O ( T ) O(\sqrt{T}) O ( T ) Ω ( T ) \Omega(\sqrt{T}) Ω ( T ) [19 ] force eventual logarithmic growth. Abeille et al. [3 ] exploit the same geometry, under the assumption we adopt here, to analyse randomised exploration without forced optimism or posterior inflation, and show that Thompson sampling attains O ( d T log ( T ) ) O(d\sqrt{T}\log(T)) O ( d T log ( T ))
Taking α = 0 \alpha= 0 α = 0 [7 ] or induced by smoothing [15 , 24 ] . On a fixed action set no such diversity is available and, accordingly, our bound degrades as α → 0 \alpha\to0 α → 0
Inference under adaptive sampling. Conditions on the growth of the design matrix play a central role in inference from adaptively collected data, beginning with Lai & Wei [17 ] . Ordinary least squares can have non-Gaussian asymptotic distributions and inflated estimation error under adaptive designs [16 , 22 ] , motivating debiasing procedures that recover asymptotic normality under conditions on the data-collection process [10 , 16 , 30 , 22 ] . A complementary approach establishes stability of the design after deterministic normalisation; Praharaj & Khamaru [23 ] use this to obtain asymptotically valid Wald intervals for a regularised EXP4 algorithm in linear contextual bandits.
Assumptions
We use the following two assumptions, placed on the noise and the action set, respectively.
Stochasticity. Let F t \mathcal{F}_t F t σ \sigma σ t t t x t + 1 x_{t+1} x t + 1 F t \mathcal{F}_t F t F t : = σ ( x 1 , y 1 , … , x t , y t , x t + 1 ) \mathcal{F}_t := \sigma(x_1,y_1,\ldots,x_t,y_t,x_{t+1}) F t := σ ( x 1 , y 1 , … , x t , y t , x t + 1 )
(A1) (Sub-Gaussian Noise). The noise variables are conditionally μ \mu μ t ∈ [ T ] t \in[T] t ∈ [ T ]
E [ η t ∣ F t − 1 ] = 0 and E [ exp ( ν η t ) ∣ F t − 1 ] ≤ exp ( μ 2 ν 2 2 ) for all ν ∈ R . \mathbb{E}[\eta_t \mid\mathcal{F}_{t-1}] = 0 \quad\text{and} \quad\mathbb{E}[\exp(\nu\eta_t) \mid\mathcal{F}_{t-1}] \leq\exp\left(\frac{\mu^2\nu^2}{2}\right) \quad\text{for all } \nu\in\mathbb{R}. E [ η t ∣ F t − 1 ] = 0 and E [ exp ( ν η t ) ∣ F t − 1 ] ≤ exp ( 2 μ 2 ν 2 ) for all ν ∈ R . The sub-Gaussian property ensures the concentration of self-normalized martingales, which we shall leverage to control the cumulative effect of noise under adaptive design.
The geometry of the action set. We consider a generic action set X \mathcal{X} X [3 ] , which are expressed in terms of the support function of X \mathcal{X} X
J X ( θ ) : = max x ∈ X ⟨ θ , x ⟩ . J_{\mathcal{X}}(\theta) := \max_{x \in\mathcal{X}} \langle\theta, x \rangle. J X ( θ ) := x ∈ X max ⟨ θ , x ⟩ . The optimal expected reward is denoted by J ∗ : = J X ( θ ∗ ) J_* := J_{\mathcal{X}}(\theta^*) J ∗ := J X ( θ ∗ ) θ t \theta_t θ t ∇ J X ( θ t ) \nabla J_{\mathcal{X}}(\theta_t) ∇ J X ( θ t ) J X J_{\mathcal{X}} J X
r t = J X ( θ ⋆ ) − ⟨ θ ⋆ , x t ⟩ = J X ( θ ⋆ ) − J X ( θ t ) − ⟨ ∇ J X ( θ t ) , θ ⋆ − θ t ⟩ = D J X ( θ ⋆ , θ t ) , r_t = J_{\mathcal{X}}(\theta^\star) - \langle\theta^\star, x_t\rangle= J_{\mathcal{X}}(\theta^\star) - J_{\mathcal{X}}(\theta_t) - \langle\nabla J_{\mathcal{X}}(\theta_t), \theta^\star- \theta_t\rangle= D_{J_{\mathcal{X}}}(\theta^\star,\theta_t), r t = J X ( θ ⋆ ) − ⟨ θ ⋆ , x t ⟩ = J X ( θ ⋆ ) − J X ( θ t ) − ⟨ ∇ J X ( θ t ) , θ ⋆ − θ t ⟩ = D J X ( θ ⋆ , θ t ) , where D f ( u , v ) : = f ( u ) − f ( v ) − ⟨ ∇ f ( v ) , u − v ⟩ D_f(u,v) := f(u) - f(v) - \langle\nabla f(v),u-v\rangle D f ( u , v ) := f ( u ) − f ( v ) − ⟨ ∇ f ( v ) , u − v ⟩ f f f
(A2) (Smooth, Strongly Convex Action Set). There exist a norm ∥ ⋅ ∥ ∗ \|\cdot\|_\ast ∥ ⋅ ∥ ∗ R d \mathbb{R}^d R d 0 < m ≤ M 0 < m \le M 0 < m ≤ M J X 2 J_{\mathcal{X}}^2 J X 2 R d \mathbb{R}^d R d θ , θ ′ ∈ R d \theta,\theta' \in\mathbb{R}^d θ , θ ′ ∈ R d
m 2 ∥ θ − θ ′ ∥ ∗ 2 ≤ D J X 2 ( θ , θ ′ ) ≤ M 2 ∥ θ − θ ′ ∥ ∗ 2 ; (1) \frac{m}{2}\|\theta-\theta'\|_\ast^2 \le D_{J_{\mathcal{X}}^2}(\theta,\theta') \le\frac{M}{2}\|\theta-\theta'\|_\ast^2; \tag*{(1)} 2 m ∥ θ − θ ′ ∥ ∗ 2 ≤ D J X 2 ( θ , θ ′ ) ≤ 2 M ∥ θ − θ ′ ∥ ∗ 2 ; (1) moreover, X \mathcal{X} X
Assumption (A2) admits a simple geometric interpretation: the boundary of X \mathcal{X} X
Strong convexity and smoothness are imposed on the squared support function J X 2 J_{\mathcal{X}}^2 J X 2 J X J_{\mathcal{X}} J X ∇ J X 2 ( θ ) = 2 J X ( θ ) ∇ J X ( θ ) \nabla J_{\mathcal{X}}^2(\theta) = 2J_{\mathcal{X}}(\theta)\nabla J_{\mathcal{X}}(\theta) ∇ J X 2 ( θ ) = 2 J X ( θ ) ∇ J X ( θ ) θ → 0 \theta\to0 θ → 0 ∇ J X \nabla J_{\mathcal{X}} ∇ J X J X ( θ ) J_{\mathcal{X}}(\theta) J X ( θ ) θ ≠ 0 \theta\ne0 θ = 0
Unit ball. For X = B 2 d ( 1 ) \mathcal{X} = \mathbb{B}_2^d(1) X = B 2 d ( 1 ) J X ( θ ) = ∥ θ ∥ 2 J_{\mathcal{X}}(\theta) = \|\theta\|_2 J X ( θ ) = ∥ θ ∥ 2 D J X 2 ( θ , θ ′ ) = ∥ θ − θ ′ ∥ 2 2 D_{J_{\mathcal{X}}^2}(\theta,\theta') = \|\theta-\theta'\|_2^2 D J X 2 ( θ , θ ′ ) = ∥ θ − θ ′ ∥ 2 2 ∥ ⋅ ∥ ∗ = ∥ ⋅ ∥ 2 \|\cdot\|_\ast= \|\cdot\|_2 ∥ ⋅ ∥ ∗ = ∥ ⋅ ∥ 2 m = M = 2 m=M=2 m = M = 2
Ellipsoids. For X = { x : x ⊤ A − 1 x ≤ 1 } = A 1 / 2 B 2 d ( 1 ) \mathcal{X} = \{x:x^\top A^{-1}x \le1\} = A^{1/2}\mathbb{B}_2^d(1) X = { x : x ⊤ A − 1 x ≤ 1 } = A 1/2 B 2 d ( 1 ) A A A J X ( θ ) = ∥ θ ∥ A J_{\mathcal{X}}(\theta) = \|\theta\|_A J X ( θ ) = ∥ θ ∥ A ∥ u ∥ ∗ : = ∥ A 1 / 2 u ∥ 2 \|u\|_\ast:= \|A^{1/2}u\|_2 ∥ u ∥ ∗ := ∥ A 1/2 u ∥ 2 D J X 2 ( θ , θ ′ ) = ∥ θ − θ ′ ∥ A 2 = ∥ θ − θ ′ ∥ ∗ 2 D_{J_{\mathcal{X}}^2}(\theta,\theta') = \|\theta-\theta'\|_A^2 = \|\theta-\theta'\|_\ast^2 D J X 2 ( θ , θ ′ ) = ∥ θ − θ ′ ∥ A 2 = ∥ θ − θ ′ ∥ ∗ 2 m = M = 2 m=M=2 m = M = 2
C 2 C^2 C 2 X ⊆ B 2 d ( 1 ) \mathcal{X} \subseteq\mathbb{B}_2^d(1) X ⊆ B 2 d ( 1 ) C 2 C^2 C 2
See Section B for more details, including the parameters M , m M,m M , m C 2 C^2 C 2
Main Results
We now present our regret upper bound for α \alpha α β \beta β E \mathcal{E} E [1 ] , which we first reproduce.
Proposition 4.1 (Confidence event; Abbasi-Yadkori et al. [1 ] ). Let Assumption (A1) hold, let ∥ θ ⋆ ∥ 2 ≤ S ⋆ \|\theta^\star\|_2 \le S^\star ∥ θ ⋆ ∥ 2 ≤ S ⋆ X ⊆ B 2 d ( 1 ) \mathcal{X} \subseteq\mathbb{B}_2^d(1) X ⊆ B 2 d ( 1 ) θ ^ t \hat{\theta}_t θ ^ t λ 0 > 0 \lambda_0 > 0 λ 0 > 0 T T T δ ∈ ( 0 , 1 ) \delta\in(0,1) δ ∈ ( 0 , 1 )
β : = μ 2 log ( 1 δ ) + d log ( 1 + T d λ 0 ) + λ 0 S ⋆ , \beta:= \mu\sqrt{2\log\left(\frac{1}{\delta}\right)} + d\log\left(1+\frac{T}{d\lambda_0}\right) + \sqrt{\lambda_0}S^\star, β := μ 2 log ( δ 1 ) + d log ( 1 + d λ 0 T ) + λ 0 S ⋆ , E : = { ∥ θ ^ t − θ ⋆ ∥ V t ≤ β for all t ∈ [ T ] } . \mathcal{E} := \left\{\|\hat{\theta}_t-\theta^\star\|_{V_t} \le\beta\text{ for all }t\in[T]\right\}. E := { ∥ θ ^ t − θ ⋆ ∥ V t ≤ β for all t ∈ [ T ] } . Then P ( E ) ≥ 1 − δ \mathbb{P}(\mathcal{E}) \ge1-\delta P ( E ) ≥ 1 − δ
The following regret bound holds for all α > 0 \alpha> 0 α > 0 β \beta β E \mathcal{E} E
Theorem 4.2 (α \alpha α E \mathcal{E} E 1 − δ 1-\delta 1 − δ
R T = O ( M m α 2 + β 2 α d T ) . \mathcal{R}_T = O\left(\frac{M}{m}\frac{\alpha^2+\beta^2}{\alpha}\sqrt{dT}\right). R T = O ( m M α α 2 + β 2 d T ) . Theorem 4.2 shows that α \alpha α α > 0 \alpha> 0 α > 0 C t ( α ) C_t(\alpha) C t ( α ) M / m M/m M / m [3 ] for randomised exploration algorithms.
As α \alpha α C t ( α ) C_t(\alpha) C t ( α ) { θ ^ t } \{\hat{\theta}_t\} { θ ^ t } α → ∞ \alpha\to\infty α → ∞ α \alpha α α = β \alpha= \beta α = β O ( ( M / m ) d T log ( T ) ) O((M/m)d\sqrt{T}\log(T)) O (( M / m ) d T log ( T )) log ( T ) \sqrt{\log(T)} log ( T ) O ( d T log ( T ) ) O(d\sqrt{T}\log(T)) O ( d T log ( T )) [1 ] .
The proof of Theorem 4.2 does not rely on the optimism inequality ⟨ θ t , x t ⟩ ≥ ⟨ θ ⋆ , x ⋆ ⟩ \langle\theta_t, x_t\rangle\ge\langle\theta^\star, x^\star\rangle ⟨ θ t , x t ⟩ ≥ ⟨ θ ⋆ , x ⋆ ⟩ θ ⋆ ∉ C t ( α ) \theta^\star\notin C_t(\alpha) θ ⋆ ∈ / C t ( α ) α \alpha α α > 0 \alpha> 0 α > 0 ∥ ⋅ ∥ ∗ \|\cdot\|_\ast ∥ ⋅ ∥ ∗ V t V_t V t
ω t : = max u ∈ B 2 d ( 1 ) ∥ V t − 1 / 2 u ∥ ∗ 2 = max ∥ g ∥ ∘ ≤ 1 ∥ g ∥ V t − 1 2 , \omega_t := \max_{u \in\mathbb{B}_2^d(1)} \|V_t^{-1/2}u\|_\ast^2 = \max_{\|g\|_\circ\le1}\|g\|_{V_t^{-1}}^2, ω t := u ∈ B 2 d ( 1 ) max ∥ V t − 1/2 u ∥ ∗ 2 = ∥ g ∥ ∘ ≤ 1 max ∥ g ∥ V t − 1 2 , where ∥ ⋅ ∥ ∘ : = sup ∥ v ∥ ∗ ≤ 1 ⟨ ⋅ , v ⟩ \|\cdot\|_\circ:= \sup_{\|v\|_\ast\le1}\langle\cdot,v\rangle ∥ ⋅ ∥ ∘ := sup ∥ v ∥ ∗ ≤ 1 ⟨ ⋅ , v ⟩ ∥ ⋅ ∥ ∗ \|\cdot\|_\ast ∥ ⋅ ∥ ∗ X \mathcal{X} X ∥ ⋅ ∥ ∗ = ∥ ⋅ ∥ 2 \|\cdot\|_\ast= \|\cdot\|_2 ∥ ⋅ ∥ ∗ = ∥ ⋅ ∥ 2 ω t = max u ∈ B 2 d ( 1 ) ∥ V t − 1 / 2 u ∥ 2 2 = λ min ( V t ) − 1 \omega_t = \max_{u \in\mathbb{B}_2^d(1)}\|V_t^{-1/2}u\|_2^2 = \lambda_{\min}(V_t)^{-1} ω t = max u ∈ B 2 d ( 1 ) ∥ V t − 1/2 u ∥ 2 2 = λ m i n ( V t ) − 1 ω t \omega_t ω t
Theorem 4.3 (Spectral Growth). Let Assumptions (A1) and (A2) hold, let E \mathcal{E} E β \beta β T B = O ( poly ( log T ) ) T_B = O(\operatorname{poly}(\log T)) T B = O ( poly ( log T )) E \mathcal{E} E 1 − δ 1-\delta 1 − δ T B ≤ t ≤ T T_B \le t \le T T B ≤ t ≤ T
ω t = max u ∈ B 2 d ( 1 ) ∥ V t − 1 / 2 u ∥ ∗ 2 ≤ 1 c 0 t , where c 0 = Ω ( m α J ⋆ d ) . \omega_t = \max_{u \in\mathbb{B}_2^d(1)}\|V_t^{-1/2}u\|_\ast^2 \le\frac{1}{c_0\sqrt{t}}, \qquad\text{where}\qquad c_0 = \Omega\left(\frac{m\alpha}{J_\star\sqrt{d}}\right). ω t = u ∈ B 2 d ( 1 ) max ∥ V t − 1/2 u ∥ ∗ 2 ≤ c 0 t 1 , where c 0 = Ω ( J ⋆ d m α ) . The bound is deterministic on E \mathcal{E} E and holds at every round after the burn-in, not on average or asymptotically; it is stated in the geometry of the assumption, with a constant proportional to m α / ( J ⋆ d ) m\alpha/(J_\star\sqrt{d}) m α / ( J ⋆ d ) every α > 0 \alpha> 0 α > 0 θ ⋆ \theta^\star θ ⋆
On E \mathcal{E} E
∥ θ t − θ ^ t ∥ ∗ ≤ α ω t ≤ α c 0 t 1 / 4 and ∥ θ ^ t − θ ⋆ ∥ ∗ ≤ β ω t ≤ β c 0 t 1 / 4 , \|\theta_t-\hat{\theta}_t\|_\ast\le\alpha\sqrt{\omega_t} \le\frac{\alpha}{\sqrt{c_0}t^{1/4}}
\qquad\text{and}\qquad
\|\hat{\theta}_t-\theta^\star\|_\ast\le\beta\sqrt{\omega_t} \le\frac{\beta}{\sqrt{c_0}t^{1/4}}, ∥ θ t − θ ^ t ∥ ∗ ≤ α ω t ≤ c 0 t 1/4 α and ∥ θ ^ t − θ ⋆ ∥ ∗ ≤ β ω t ≤ c 0 t 1/4 β , so that the ∥ ⋅ ∥ ∗ \|\cdot\|_\ast ∥ ⋅ ∥ ∗ t − 1 / 4 t^{-1/4} t − 1/4 α > 0 \alpha> 0 α > 0 λ min ( V t ) = Ω ( t d ) \lambda_{\min}(V_t) = \Omega\left(\frac{\sqrt{t}}{\sqrt{d}}\right) λ m i n ( V t ) = Ω ( d t ) α ≍ β ≍ d log T \alpha\asymp\beta\asymp\sqrt{d\log T} α ≍ β ≍ d log T Ω ( t / ( d log t ) ) \Omega\left(\sqrt{t}/(d\log t)\right) Ω ( t / ( d log t ) ) [6 ] .
Proof sketch
The detailed proofs of Theorems 4.2 and 4.3 are given in Sections D and E. The argument has two parts: an energy argument, which shows that optimizing over C t ( α ) C_t(\alpha) C t ( α )
Spectral growth
For ease of presentation, we sketch the proof in the unit-ball case, i.e. X = B 2 d ( 1 ) \mathcal{X} = \mathbb{B}_2^d(1) X = B 2 d ( 1 ) ∥ ⋅ ∥ ∗ = ∥ ⋅ ∥ 2 \|\cdot\|_\ast= \|\cdot\|_2 ∥ ⋅ ∥ ∗ = ∥ ⋅ ∥ 2 ω t = λ t , d − 1 \omega_t = \lambda_{t,d}^{-1} ω t = λ t , d − 1 θ ⋆ ≠ 0 \theta^\star\ne0 θ ⋆ = 0 α \alpha α λ t , 1 ≥ ⋯ ≥ λ t , d \lambda_{t,1} \ge\cdots\ge\lambda_{t,d} λ t , 1 ≥ ⋯ ≥ λ t , d V t V_t V t v t , 1 , … , v t , d v_{t,1},\ldots,v_{t,d} v t , 1 , … , v t , d x t x_t x t i i i i ∈ [ d ] i \in[d] i ∈ [ d ] x t , i 2 / λ t , i x_{t,i}^2/\lambda_{t,i} x t , i 2 / λ t , i x t , i : = ⟨ x t , v t , i ⟩ x_{t,i} := \langle x_t,v_{t,i}\rangle x t , i := ⟨ x t , v t , i ⟩ Figure 1 .
Figure 1. The three steps of the proof of Theorem 4.3 in the unit-ball case.
Step 1. We first define a logarithmically capped potential , rather than the usual elliptical potential (denoted by Φ e l l \Phi_{\mathrm{ell}} Φ ell ϑ > 0 \vartheta> 0 ϑ > 0 Γ ≥ 1 \Gamma\ge1 Γ ≥ 1 Γ \Gamma Γ
Definition 5.1 (Capped potential). For the scale ϑ \vartheta ϑ L : = 8 Γ ϑ L := 8\Gamma\vartheta L := 8Γ ϑ V ⪰ 0 V \succeq0 V ⪰ 0
Φ ϑ ( V ) : = tr h ( V ) , h ( λ ) : = ϑ log ( L ( λ + ϑ ) ϑ ( λ + L ) ) , λ ≥ 0. (2) \Phi_{\vartheta}(V) := \operatorname{tr} h(V), \qquad h(\lambda) := \vartheta\log\left(\frac{L(\lambda+\vartheta)}{\vartheta(\lambda+L)}\right), \qquad\lambda\ge0. \tag*{(2)} Φ ϑ ( V ) := tr h ( V ) , h ( λ ) := ϑ log ( ϑ ( λ + L ) L ( λ + ϑ ) ) , λ ≥ 0. (2) The map Φ ϑ \Phi_{\vartheta} Φ ϑ R \mathbb{R} R h h h ϑ \vartheta ϑ L L L
Φ ϑ ( V ) = ϑ ( log det ( V + ϑ I ) − log det ( V + 8 Γ ϑ I ) + d log ( 8 Γ ) ) , \Phi_{\vartheta}(V) = \vartheta\left(\log\det(V+\vartheta I)-\log\det(V+8\Gamma\vartheta I)+d\log(8\Gamma)\right), Φ ϑ ( V ) = ϑ ( log det ( V + ϑ I ) − log det ( V + 8Γ ϑ I ) + d log ( 8Γ ) ) , where the additive constant d log ( 8 Γ ) d\log(8\Gamma) d log ( 8Γ ) Φ ϑ ( 0 ) = 0 \Phi_{\vartheta}(0)=0 Φ ϑ ( 0 ) = 0 V V V 8 Γ ϑ 8\Gamma\vartheta 8Γ ϑ
0 ≤ Φ ϑ ( V ) ≤ d ϑ log ( 8 Γ ) for every V ⪰ 0. 0 \le\Phi_{\vartheta}(V) \le d\vartheta\log(8\Gamma) \qquad\text{for every } V \succeq0. 0 ≤ Φ ϑ ( V ) ≤ d ϑ log ( 8Γ ) for every V ⪰ 0. Step 2. After a burn-in of polylogarithmic length T B T_B T B Definition D.10 ), on the event E \mathcal{E} E C t ( α ) C_t(\alpha) C t ( α ) { θ : ∥ θ − θ ⋆ ∥ ≤ ∥ θ ⋆ ∥ / 2 } \{\theta: \|\theta-\theta^\star\| \le\|\theta^\star\|/2\} { θ : ∥ θ − θ ⋆ ∥ ≤ ∥ θ ⋆ ∥/2 } Proposition D.8 ). In particular, 1 2 ∥ θ ⋆ ∥ ≤ ∥ θ ∥ ≤ 3 2 ∥ θ ⋆ ∥ \frac{1}{2}\|\theta^\star\| \le\|\theta\| \le\frac{3}{2}\|\theta^\star\| 2 1 ∥ θ ⋆ ∥ ≤ ∥ θ ∥ ≤ 2 3 ∥ θ ⋆ ∥ θ ∈ C t ( α ) \theta\in C_t(\alpha) θ ∈ C t ( α ) ⟨ x t , x ⋆ ⟩ ≥ 1 / 2 \langle x_t,x^\star\rangle\ge1/2 ⟨ x t , x ⋆ ⟩ ≥ 1/2 t ≥ T B t \ge T_B t ≥ T B x t ∈ B d ( 1 ) x_t \in\mathbb{B}^d(1) x t ∈ B d ( 1 )
∥ θ t ∥ = ⟨ θ ^ t , x t ⟩ + α ∥ x t ∥ V t − 1 − 1 ≤ ∥ θ ^ t ∥ + α ∥ x t ∥ V t − 1 − 1 ⟹ α ∥ x t ∥ V t − 1 − 1 ≥ ∥ θ t ∥ − ∥ θ ^ t ∥ , \|\theta_t\| = \langle\widehat{\theta}_t,x_t\rangle+ \alpha\|x_t\|_{V_{t-1}^{-1}} \le\|\widehat{\theta}_t\|+\alpha\|x_t\|_{V_{t-1}^{-1}} \quad\Longrightarrow\quad\alpha\|x_t\|_{V_{t-1}^{-1}} \ge\|\theta_t\|-\|\widehat{\theta}_t\|, ∥ θ t ∥ = ⟨ θ t , x t ⟩ + α ∥ x t ∥ V t − 1 − 1 ≤ ∥ θ t ∥ + α ∥ x t ∥ V t − 1 − 1 ⟹ α ∥ x t ∥ V t − 1 − 1 ≥ ∥ θ t ∥ − ∥ θ t ∥ , where the equality follows from the definition of ( x t , θ t ) (x_t,\theta_t) ( x t , θ t ) x t = θ t / ∥ θ t ∥ x_t=\theta_t/\|\theta_t\| x t = θ t /∥ θ t ∥ θ ^ t ± α v t , d / λ t , d ∈ C t ( α ) \widehat{\theta}_t \pm\alpha v_{t,d}/\sqrt{\lambda_{t,d}} \in C_t(\alpha) θ t ± α v t , d / λ t , d ∈ C t ( α ) θ t \theta_t θ t C t ( α ) C_t(\alpha) C t ( α )
∥ θ t ∥ 2 ≥ 1 2 ∥ θ ^ t + α λ t , d v t , d ∥ 2 + 1 2 ∥ θ ^ t − α λ t , d v t , d ∥ 2 = ∥ θ ^ t ∥ 2 + α 2 λ t , d . \|\theta_t\|^2 \ge\frac{1}{2}\left\|\widehat{\theta}_t+\frac{\alpha}{\sqrt{\lambda_{t,d}}}v_{t,d}\right\|^2+\frac{1}{2}\left\|\widehat{\theta}_t-\frac{\alpha}{\sqrt{\lambda_{t,d}}}v_{t,d}\right\|^2=\|\widehat{\theta}_t\|^2+\frac{\alpha^2}{\lambda_{t,d}}. ∥ θ t ∥ 2 ≥ 2 1 θ t + λ t , d α v t , d 2 + 2 1 θ t − λ t , d α v t , d 2 = ∥ θ t ∥ 2 + λ t , d α 2 . Combining the above with the previous display, we have
α ∥ x t ∥ V t − 1 − 1 ≥ ∥ θ t ∥ − ∥ θ ^ t ∥ = ∥ θ t ∥ 2 − ∥ θ ^ t ∥ 2 ∥ θ t ∥ + ∥ θ ^ t ∥ ≥ α 2 3 ∥ θ ⋆ ∥ λ t , d , \alpha\|x_t\|_{V_{t-1}^{-1}} \ge\|\theta_t\|-\|\widehat{\theta}_t\|=\frac{\|\theta_t\|^2-\|\widehat{\theta}_t\|^2}{\|\theta_t\|+\|\widehat{\theta}_t\|}\ge\frac{\alpha^2}{3\|\theta^\star\|\lambda_{t,d}}, α ∥ x t ∥ V t − 1 − 1 ≥ ∥ θ t ∥ − ∥ θ t ∥ = ∥ θ t ∥ + ∥ θ t ∥ ∥ θ t ∥ 2 − ∥ θ t ∥ 2 ≥ 3∥ θ ⋆ ∥ λ t , d α 2 , where the last inequality uses that θ t , θ ^ t ∈ C t ( α ) \theta_t,\widehat{\theta}_t \in C_t(\alpha) θ t , θ t ∈ C t ( α ) ∥ θ t ∥ + ∥ θ ^ t ∥ ≤ 3 ∥ θ ⋆ ∥ \|\theta_t\|+\|\widehat{\theta}_t\| \le3\|\theta^\star\| ∥ θ t ∥ + ∥ θ t ∥ ≤ 3∥ θ ⋆ ∥
Fix a time-step τ \tau τ ϑ : = λ τ , d \vartheta:= \lambda_{\tau,d} ϑ := λ τ , d τ \tau τ τ ′ \tau' τ ′ λ t , d ≤ 2 ϑ \lambda_{t,d} \le2\vartheta λ t , d ≤ 2 ϑ t ∈ [ τ , τ ′ ] t \in[\tau,\tau'] t ∈ [ τ , τ ′ ]
We show in Lemma D.11 that for every round t ≥ τ t \ge\tau t ≥ τ
Φ ϑ ( V t + 1 ) − Φ ϑ ( V t ) ≥ ϑ 4 ∑ i ≥ 2 λ t , i < Γ ϑ x t , i 2 λ t , i . \Phi_{\vartheta}(V_{t+1})-\Phi_{\vartheta}(V_t) \ge\frac{\vartheta}{4}\sum_{\substack{i\ge2\\ \lambda_{t,i}<\Gamma\vartheta}}\frac{x_{t,i}^{2}}{\lambda_{t,i}}. Φ ϑ ( V t + 1 ) − Φ ϑ ( V t ) ≥ 4 ϑ i ≥ 2 λ t , i < Γ ϑ ∑ λ t , i x t , i 2 . a straightforward algebraic bound. Now, using our inequality on α ∥ x t ∥ V t − 1 \alpha\lVert x_t\rVert_{V_{t-1}} α ∥ x t ∥ V t − 1
x t , 1 2 λ t , 1 + ∑ i ≥ 2 λ t , i ≥ Γ ϑ x t , i 2 λ t , i + ∑ i ≥ 2 λ t , i < Γ ϑ x t , i 2 λ t , i = ∥ x t ∥ V t − 1 2 ≥ α 2 9 ∥ θ ⋆ ∥ 2 λ t , d 2 . (3) \frac{x_{t,1}^{2}}{\lambda_{t,1}}+\sum_{\substack{i\ge2\\ \lambda_{t,i}\ge\Gamma\vartheta}}\frac{x_{t,i}^{2}}{\lambda_{t,i}}+\sum_{\substack{i\ge2\\ \lambda_{t,i}<\Gamma\vartheta}}\frac{x_{t,i}^{2}}{\lambda_{t,i}}=\lVert x_t\rVert_{V_{t}^{-1}}^{2}\ge\frac{\alpha^{2}}{9\lVert\theta^{\star}\rVert^{2}\lambda_{t,d}^{2}}. \tag*{(3)} λ t , 1 x t , 1 2 + i ≥ 2 λ t , i ≥ Γ ϑ ∑ λ t , i x t , i 2 + i ≥ 2 λ t , i < Γ ϑ ∑ λ t , i x t , i 2 = ∥ x t ∥ V t − 1 2 ≥ 9 ∥ θ ⋆ ∥ 2 λ t , d 2 α 2 . (3) If we show that the first two terms are small relative to the lower bound, we can conclude that
Φ ϑ ( V t + 1 ) − Φ ϑ ( V t ) ≳ ϑ α 2 ∥ θ ⋆ ∥ 2 λ t , d 2 , (4) \Phi_{\vartheta}(V_{t+1})-\Phi_{\vartheta}(V_t)\gtrsim\frac{\vartheta\alpha^{2}}{\lVert\theta^{\star}\rVert^{2}\lambda_{t,d}^{2}}, \tag*{(4)} Φ ϑ ( V t + 1 ) − Φ ϑ ( V t ) ≳ ∥ θ ⋆ ∥ 2 λ t , d 2 ϑ α 2 , (4) where ≳ \gtrsim ≳ T B T_B T B ⟨ x t , x ⋆ ⟩ ≥ 1 / 2 \langle x_t,x^{\star}\rangle\ge1/2 ⟨ x t , x ⋆ ⟩ ≥ 1/2 t ≥ T B t\ge T_B t ≥ T B t ≥ 2 T B t\ge2T_B t ≥ 2 T B
λ t , 1 ≥ ∥ x ⋆ ∥ V t 2 ≥ ∑ s = T B t − 1 ⟨ x s , x ⋆ ⟩ 2 ≥ t − T B 4 ≥ t 8 ⟹ x t , 1 2 λ t , 1 ≤ 1 λ t , 1 ≤ 8 t . \lambda_{t,1}\ge\lVert x^{\star}\rVert_{V_t}^{2}\ge\sum_{s=T_B}^{t-1}\langle x_s,x^{\star}\rangle^{2}\ge\frac{t-T_B}{4}\ge\frac{t}{8}\Longrightarrow\frac{x_{t,1}^{2}}{\lambda_{t,1}}\le\frac{1}{\lambda_{t,1}}\le\frac{8}{t}. λ t , 1 ≥ ∥ x ⋆ ∥ V t 2 ≥ s = T B ∑ t − 1 ⟨ x s , x ⋆ ⟩ 2 ≥ 4 t − T B ≥ 8 t ⟹ λ t , 1 x t , 1 2 ≤ λ t , 1 1 ≤ t 8 . The upper bound 8 / t 8/t 8/ t (3) as long as λ t , d = o ( t ) \lambda_{t,d}=o(\sqrt{t}) λ t , d = o ( t ) t t t ∑ i ≥ 2 x t , i 2 ≤ C T / λ t , d \sum_{i\ge2}x_{t,i}^{2}\le C_T/\lambda_{t,d} ∑ i ≥ 2 x t , i 2 ≤ C T / λ t , d C T > 0 C_T>0 C T > 0 T T T
∑ i ≥ 2 λ t , i ≥ Γ ϑ x t , i 2 λ t , i ≤ 2 C T Γ λ t , d 2 . \sum_{\substack{i\ge2\\ \lambda_{t,i}\ge\Gamma\vartheta}}\frac{x_{t,i}^{2}}{\lambda_{t,i}}\le\frac{2C_T}{\Gamma\lambda_{t,d}^{2}}. i ≥ 2 λ t , i ≥ Γ ϑ ∑ λ t , i x t , i 2 ≤ Γ λ t , d 2 2 C T . The choice of Γ \Gamma Γ
Step 3. Set N : = ⌈ K ϑ 2 ⌉ N:=\lceil K\vartheta^{2}\rceil N := ⌈ K ϑ 2 ⌉ K ≳ d log ( 8 Γ ) / α 2 K\gtrsim d\log(8\Gamma)/\alpha^{2} K ≳ d log ( 8Γ ) / α 2 τ , … , τ + N − 1 \tau,\ldots,\tau+N-1 τ , … , τ + N − 1 λ t , d ≤ 2 ϑ \lambda_{t,d}\le2\vartheta λ t , d ≤ 2 ϑ ≳ α 2 / ϑ \gtrsim\alpha^{2}/\vartheta ≳ α 2 / ϑ ∥ θ ⋆ ∥ 2 \lVert\theta^{\star}\rVert^{2} ∥ θ ⋆ ∥ 2
Φ ϑ ( V τ + N ) − Φ ϑ ( V τ ) ≳ N α 2 ϑ ≳ d log ( 8 Γ ) α 2 ϑ 2 ⋅ α 2 ϑ = d ϑ log ( 8 Γ ) , \Phi_{\vartheta}(V_{\tau+N})-\Phi_{\vartheta}(V_{\tau})\gtrsim N\frac{\alpha^{2}}{\vartheta}\gtrsim\frac{d\log(8\Gamma)}{\alpha^{2}}\vartheta^{2}\cdot\frac{\alpha^{2}}{\vartheta}=d\vartheta\log(8\Gamma), Φ ϑ ( V τ + N ) − Φ ϑ ( V τ ) ≳ N ϑ α 2 ≳ α 2 d log ( 8Γ ) ϑ 2 ⋅ ϑ α 2 = d ϑ log ( 8Γ ) , which, for a suitable choice of K K K d ϑ log ( 8 Γ ) d\vartheta\log(8\Gamma) d ϑ log ( 8Γ ) τ + N \tau+N τ + N N = O ( d ϑ 2 log ( 8 Γ ) / α 2 ) N=O(d\vartheta^{2}\log(8\Gamma)/\alpha^{2}) N = O ( d ϑ 2 log ( 8Γ ) / α 2 ) λ τ + N , d ≥ 2 λ τ , d \lambda_{\tau+N,d}\ge2\lambda_{\tau,d} λ τ + N , d ≥ 2 λ τ , d t t t λ t , d < c 0 t \lambda_{t,d}<c_0\sqrt{t} λ t , d < c 0 t λ t , d ≥ c 0 t \lambda_{t,d}\ge c_0\sqrt{t} λ t , d ≥ c 0 t
While we chose to illustrate the proof sketch in the unit-ball case, the appendix presents the proof in the general case, in the geometry of ∥ ⋅ ∥ ∗ \lVert\cdot\rVert_{\ast} ∥ ⋅ ∥ ∗ adapted to ∥ ⋅ ∥ ∗ \lVert\cdot\rVert_{\ast} ∥ ⋅ ∥ ∗ D ∗ ≤ min { d , M / m } D_{\ast}\le\min\{\sqrt{d},\sqrt{M/m}\} D ∗ ≤ min { d , M / m } ∥ ⋅ ∥ ∗ \lVert\cdot\rVert_{\ast} ∥ ⋅ ∥ ∗
From spectral growth to regret
The argument has two steps. First, after the polylogarithmic burn-in phase T B T_B T B Definition D.10 ), the parameters θ t \theta_t θ t θ ⋆ \theta^{\star} θ ⋆ J X J_{\mathcal{X}} J X J ⋆ J_{\star} J ⋆ Proposition D.8 ), and the instantaneous regret is controlled by the Bregman divergence of J X 2 J_{\mathcal{X}}^{2} J X 2
r t ≤ D J X 2 ( θ ⋆ , θ t ) J ⋆ . r_t\le\frac{D_{J_{\mathcal{X}}^{2}}(\theta^{\star},\theta_t)}{J_{\star}}. r t ≤ J ⋆ D J X 2 ( θ ⋆ , θ t ) . This upper bound does not require θ ⋆ ∈ C t ( α ) \theta^\star\in C_t(\alpha) θ ⋆ ∈ C t ( α ) ω t \omega_t ω t log ( T ) \sqrt{\log(T)} log ( T )
In detail, the horizon is split at the burn-in time T B T_B T B T B T_B T B r t r_t r t θ ⋆ \theta^\star θ ⋆ T B T_B T B ω t \omega_t ω t 2 T / c 0 2\sqrt{T}/c_0 2 T / c 0 E \mathcal{E} E
R T ≤ O ( T B ) + ∑ t = T B T D J X 2 ( θ ⋆ , θ t ) J ⋆ R_T \leq\mathcal{O}(T_B) + \sum_{t=T_B}^{T} \frac{D_{J_{\mathcal{X}}^2}(\theta^\star,\theta_t)}{J_\star} R T ≤ O ( T B ) + t = T B ∑ T J ⋆ D J X 2 ( θ ⋆ , θ t ) (Lemma E.1)
≤ O ( T B ) + M 2 J ⋆ ∑ t = T B T ∥ θ ⋆ − θ t ∥ ∗ 2 \leq\mathcal{O}(T_B) + \frac{M}{2J_\star}\sum_{t=T_B}^{T}\lVert\theta^\star-\theta_t\rVert_\ast^2 ≤ O ( T B ) + 2 J ⋆ M t = T B ∑ T ∥ θ ⋆ − θ t ∥ ∗ 2 (Assumption (A2))
≤ O ( T B ) + M J ⋆ ∑ t = T B T ( ∥ θ ⋆ − θ ^ t ∥ ∗ 2 + ∥ θ ^ t − θ t ∥ ∗ 2 ) \leq\mathcal{O}(T_B) + \frac{M}{J_\star}\sum_{t=T_B}^{T}\left(\lVert\theta^\star-\hat{\theta}_t\rVert_\ast^2+\lVert\hat{\theta}_t-\theta_t\rVert_\ast^2\right) ≤ O ( T B ) + J ⋆ M t = T B ∑ T ( ∥ θ ⋆ − θ ^ t ∥ ∗ 2 + ∥ θ ^ t − θ t ∥ ∗ 2 ) ( a + b ) 2 ≤ 2 a 2 + 2 b 2 \left(a+b\right)^2\leq2a^2+2b^2 ( a + b ) 2 ≤ 2 a 2 + 2 b 2
≤ O ( T B ) + M ( α 2 + β 2 ) J ⋆ ∑ t = T B T ω t \leq\mathcal{O}(T_B) + \frac{M(\alpha^2+\beta^2)}{J_\star}\sum_{t=T_B}^{T}\omega_t ≤ O ( T B ) + J ⋆ M ( α 2 + β 2 ) t = T B ∑ T ω t ( θ ⋆ ∈ C t ( β ) , θ t ∈ C t ( α ) ) (\theta^\star\in C_t(\beta),\ \theta_t\in C_t(\alpha)) ( θ ⋆ ∈ C t ( β ) , θ t ∈ C t ( α ))
≤ O ( T B ) + M ( α 2 + β 2 ) J ⋆ c 0 ∑ t = T B T 1 t \leq\mathcal{O}(T_B) + \frac{M(\alpha^2+\beta^2)}{J_\star c_0}\sum_{t=T_B}^{T}\frac{1}{\sqrt{t}} ≤ O ( T B ) + J ⋆ c 0 M ( α 2 + β 2 ) t = T B ∑ T t 1 (Theorem 4.3)
≤ O ( T B ) + 2 M ( α 2 + β 2 ) J ⋆ c 0 T = O ( M m α 2 + β 2 α d T ) , \leq\mathcal{O}(T_B) + \frac{2M(\alpha^2+\beta^2)}{J_\star c_0}\sqrt{T}
= \mathcal{O}\left(\frac{M}{m}\frac{\alpha^2+\beta^2}{\alpha}\sqrt{dT}\right), ≤ O ( T B ) + J ⋆ c 0 2 M ( α 2 + β 2 ) T = O ( m M α α 2 + β 2 d T ) , where the last step uses c 0 = Ω ( m α / ( J ⋆ d ) ) c_0=\Omega(m\alpha/(J_\star\sqrt{d})) c 0 = Ω ( m α / ( J ⋆ d )) T B = O ( poly ( log T ) ) T_B=\mathcal{O}(\operatorname{poly}(\log T)) T B = O ( poly ( log T ))
Curvature matters: a failure case
Theorems 4.2 and 4.3 hold for every radius α > 0 \alpha>0 α > 0 § 5 explained what the curvature imposed by Assumption (A2) does for the algorithm: a perturbation of the optimistic parameter is transmitted to the played arm. On a body with a corner this transmission stops, and the next proposition shows that the conclusion of Theorems 4.2 and 4.3 then fails for every small radius: α \alpha α Figure 2 ): in R 2 \mathbb{R}^2 R 2 ( e 1 , e 2 ) (e_1,e_2) ( e 1 , e 2 ) a ∈ ( 0 , 1 ) a\in(0,1) a ∈ ( 0 , 1 )
Figure 2. Counter-example.
X a : = conv { e 1 , − e 1 , a e 2 , − a e 2 } , θ ⋆ : = ( ε , 1 ) with 0 < ε < a , η t ∼ i . i . d . N ( 0 , μ 2 ) , \mathcal{X}_a:=\operatorname{conv}\{e_1,-e_1,ae_2,-ae_2\},\qquad\theta^\star:=(\varepsilon,1)\quad\text{with }0<\varepsilon<a,\qquad\eta_t\mathrel{\overset{\mathrm{i.i.d.}}{\sim}}\mathcal{N}(0,\mu^2), X a := conv { e 1 , − e 1 , a e 2 , − a e 2 } , θ ⋆ := ( ε , 1 ) with 0 < ε < a , η t ∼ i.i.d. N ( 0 , μ 2 ) , where conv \operatorname{conv} conv X a \mathcal{X}_a X a ± e 1 \pm e_1 ± e 1 ± a e 2 \pm ae_2 ± a e 2
The set X a \mathcal{X}_a X a B 2 ( 1 ) \mathbb{B}_2(1) B 2 ( 1 ) J X a ( θ ) = max { ∣ θ 1 ∣ , a ∣ θ 2 ∣ } J_{\mathcal{X}_a}(\theta)=\max\{|\theta_1|,a|\theta_2|\} J X a ( θ ) = max { ∣ θ 1 ∣ , a ∣ θ 2 ∣ } θ = ( θ 1 , θ 2 ) \theta=(\theta_1,\theta_2) θ = ( θ 1 , θ 2 ) Figure 4 (b)). The optimal arm is x ⋆ = a e 2 x^\star=ae_2 x ⋆ = a e 2 J ⋆ = a J_\star=a J ⋆ = a ± e 1 \pm e_1 ± e 1 Δ : = a − ε > 0 \Delta:=a-\varepsilon>0 Δ := a − ε > 0
Figure 4. Assumption (A2) illustrated (parameter and action spaces superimposed). (a) A C 2 C^2 C 2 θ \theta θ x = ∇ J X ( θ ) x=\nabla J_{\mathcal{X}}(\theta) x = ∇ J X ( θ ) ∂ X \partial\mathcal{X} ∂ X θ \theta θ θ \theta θ r ∈ [ r − , r + ] r\in[r_-,r_+] r ∈ [ r − , r + ] x x x ρ min \rho_{\min} ρ m i n r + = ∞ r_+=\infty r + = ∞ M = ∞ M=\infty M = ∞ r − = 0 r_-=0 r − = 0 m = 0 m=0 m = 0
Proposition 6.1 (Linear regret on a polytope). In the instance above, for every λ 0 > 0 \lambda_0 > 0 λ 0 > 0 every horizon T T T and every radius
0 < α ≤ α 0 : = ( 1 − a 2 ) λ 0 2 a ε , 0 < \alpha\le\alpha_0 := \frac{(1-a^2)\sqrt{\lambda_0}}{2a}\varepsilon, 0 < α ≤ α 0 := 2 a ( 1 − a 2 ) λ 0 ε , with probability at least
p 0 : = 1 2 ( 1 − exp ( − ( 1 − a 2 ) λ 0 ε 2 2 a 2 μ 2 ) ) > 0 , p_0 := \frac{1}{2}\left(1-\exp\left(-\frac{(1-a^2)\lambda_0\varepsilon^2}{2a^2\mu^2}\right)\right)>0, p 0 := 2 1 ( 1 − exp ( − 2 a 2 μ 2 ( 1 − a 2 ) λ 0 ε 2 ) ) > 0 , α \alpha α x t ∈ { e 1 , − e 1 } x_t \in\{e_1,-e_1\} x t ∈ { e 1 , − e 1 } t ∈ [ T ] t \in[T] t ∈ [ T ] λ min ( V t ) = λ 0 \lambda_{\min}(V_t)=\lambda_0 λ m i n ( V t ) = λ 0 t t t R T ≥ Δ T R_T \ge\Delta T R T ≥ Δ T E [ R T ] ≥ p 0 Δ T \mathbb{E}[R_T] \ge p_0\Delta T E [ R T ] ≥ p 0 Δ T
The proof is given in Section B.1. For instance, a = 1 / 2 a=1/2 a = 1/2 ε = 1 / 4 \varepsilon=1/4 ε = 1/4 λ 0 = 1 \lambda_0=1 λ 0 = 1 μ = 1 / 8 \mu=1/8 μ = 1/8 p 0 > 0.49 p_0>0.49 p 0 > 0.49 α ≤ 3 / 16 \alpha\le3/16 α ≤ 3/16 R T ≥ T / 4 R_T\ge T/4 R T ≥ T /4
Numerical Simulations
While the contribution of this paper is mainly theoretical, we corroborate our results with simulations. We run α \alpha α d = 10 d=10 d = 10 θ ⋆ = 1 d / d \theta^\star=\mathbf{1}_d/\sqrt{d} θ ⋆ = 1 d / d β = 20 \beta=20 β = 20 T = 16,300 T=16{,}300 T = 16 , 300 α ∈ { β / 256 , β / 64 , β / 16 , β / 4 , β , 2 β } \alpha\in\{\beta/256,\beta/64,\beta/16,\beta/4,\beta,2\beta\} α ∈ { β /256 , β /64 , β /16 , β /4 , β , 2 β } α \alpha α Figure 3 the mean of the pseudo-regret R t R_t R t λ min ( V t + 1 ) \lambda_{\min}(V_{t+1}) λ m i n ( V t + 1 ) θ ⋆ ∉ C t ( α ) \theta^\star\notin C_t(\alpha) θ ⋆ ∈ / C t ( α )
Figure 3. Effect of the confidence radius on the Euclidean unit ball for d = 10 d=10 d = 10 β = 20 \beta=20 β = 20 T = 16,300 T=16{,}300 T = 16 , 300 α ∈ { β / 256 , β / 64 , β / 16 , β / 4 , β , 2 β } \alpha\in\{\beta/256,\beta/64,\beta/16,\beta/4,\beta,2\beta\} α ∈ { β /256 , β /64 , β /16 , β /4 , β , 2 β } t \sqrt{t} t t / d \sqrt{t/d} t / d θ ⋆ ∉ C t ( α ) \theta^\star\notin C_t(\alpha) θ ⋆ ∈ / C t ( α )
In accordance with Theorem 4.2, the regret scales as t \sqrt{t} t α \alpha α α = β / 4 \alpha=\beta/4 α = β /4 β \beta β λ min ( V t + 1 ) \lambda_{\min}(V_{t+1}) λ m i n ( V t + 1 ) t \sqrt{t} t α \alpha α α \alpha α α ≤ β / 16 \alpha\le\beta/16 α ≤ β /16 α ∈ { β , 2 β } \alpha\in\{\beta,2\beta\} α ∈ { β , 2 β } α = β / 4 \alpha=\beta/4 α = β /4 ≈ 40 % \approx40\% ≈ 40% α = β \alpha=\beta α = β
Conclusion
This paper studied the α \alpha α α \alpha α α > 0 \alpha> 0 α > 0 C t ( α ) C_t(\alpha) C t ( α ) α \alpha α θ ⋆ ∉ C t ( α ) \theta^\star\notin C_t(\alpha) θ ⋆ ∈ / C t ( α ) log ( T ) \sqrt{\log(T)} log ( T )
A single factor log ( T ) \sqrt{\log(T)} log ( T ) Ω ( d T ) \Omega(d\sqrt{T}) Ω ( d T ) [18 ] rules out the removal of this factor in the general adaptive case, it may still be possible to dispense with it within the family of algorithms considered here, now that the dynamics of the algorithm, and the rate at which each direction of the design matrix grows, are better understood. This question is both interesting and far from straightforward, and we leave it for future work.
AI Use Statement
In this work, we used generative AI tools to edit the manuscript for readability and to check spelling, grammar, and the consistency of notation and citations. We have not used generative AI tools to develop theoretical models or conceptual frameworks, formulate mathematical claims, provide ingredients for or write proofs, propose or refine hypotheses, design or provide feedback on the methodology or experiments, implement methods, or interpret results; the remaining tasks with required disclosure (generating synthetic data, translation, dataset cleaning, and qualitative data analysis) are not applicable to this work. We have reviewed all AI-assisted work: every suggested edit was checked by the authors against the original text and the mathematical content. We take responsibility for the final content of this work, including text, claims, or artifacts produced with the aid of generative AI.
Reproducibility Statement
This work is primarily theoretical. All assumptions required to derive our results are precisely stated and discussed in the main text (Assumptions (A1) and (A2)). Proof sketches are provided in the main text, and complete proofs are included in the appendices. The full protocol of the simulations (instance, parameters, number of replications) is specified in Section 7.
Self-normalised bound for adaptive designs
Proposition 4.1 (Confidence event; Abbasi-Yadkori et al. [1 ] ). Let Assumption (A1) hold, let ∥ θ ⋆ ∥ 2 ≤ S ⋆ \lVert\theta^\star\rVert_2 \le S^\star ∥ θ ⋆ ∥ 2 ≤ S ⋆ X ⊆ B 2 d ( 1 ) \mathcal{X} \subseteq\mathbb{B}_2^d(1) X ⊆ B 2 d ( 1 ) θ ^ t \hat{\theta}_t θ ^ t λ 0 > 0 \lambda_0 > 0 λ 0 > 0 T T T δ ∈ ( 0 , 1 ) \delta\in(0,1) δ ∈ ( 0 , 1 )
β : = μ 2 log ( 1 δ ) + d log ( 1 + T d λ 0 ) + λ 0 S ⋆ , \beta:= \mu\sqrt{2\log\left(\frac{1}{\delta}\right)+d\log\left(1+\frac{T}{d\lambda_0}\right)}+\sqrt{\lambda_0}S^\star, β := μ 2 log ( δ 1 ) + d log ( 1 + d λ 0 T ) + λ 0 S ⋆ , E : = { ∥ θ ^ t − θ ⋆ ∥ V t ≤ β for all t ∈ [ T ] } . \mathcal{E}:=\left\{\lVert\hat{\theta}_t-\theta^\star\rVert_{V_t}\leq\beta\text{ for all }t\in[T]\right\}. E := { ∥ θ ^ t − θ ⋆ ∥ V t ≤ β for all t ∈ [ T ] } . Then P ( E ) ≥ 1 − δ \mathbb{P}(\mathcal{E})\geq1-\delta P ( E ) ≥ 1 − δ
Proposition 4.1 is Theorem 2 of Abbasi-Yadkori et al. [1 ] , applied with noise parameter μ \mu μ ∥ x s ∥ 2 ≤ 1 \lVert x_s\rVert_2\leq1 ∥ x s ∥ 2 ≤ 1 λ 0 \lambda_0 λ 0 ∥ θ ⋆ ∥ 2 ≤ S ⋆ \lVert\theta^\star\rVert_2\leq S^\star ∥ θ ⋆ ∥ 2 ≤ S ⋆ μ 2 log ( ( 1 / δ ) ) + log ( ( det ( V t ) / det ( λ 0 I ) ) ) + λ 0 S ⋆ \mu\sqrt{2\log((1/\delta))+\log((\det(V_t)/\det(\lambda_0 I)))}+\sqrt{\lambda_0}S^\star μ 2 log (( 1/ δ )) + log (( det ( V t ) / det ( λ 0 I ))) + λ 0 S ⋆ det ( V t ) ≤ ( λ 0 + t / d ) d \det(V_t)\leq(\lambda_0+t/d)^d det ( V t ) ≤ ( λ 0 + t / d ) d tr V t ≤ d λ 0 + t \operatorname{tr}V_t\leq d\lambda_0+t tr V t ≤ d λ 0 + t
Adaptive design. The covariates x 1 , x 2 , … x_1,x_2,\ldots x 1 , x 2 , … x t x_t x t ( x s ) s ≤ t (x_s)_{s\leq t} ( x s ) s ≤ t ( η s ) s ≤ t (\eta_s)_{s\leq t} ( η s ) s ≤ t ≍ μ d \asymp\mu\sqrt{d} ≍ μ d [1 ] handles the sequential design through a self-normalised martingale inequality (the method of mixtures): the quantity ∥ ∑ s < t x s η s ∥ V t − 1 \left\lVert\sum_{s<t}x_s\eta_s\right\rVert_{V_t^{-1}} ∑ s < t x s η s V t − 1 t t t x s x_s x s log det V t ≤ d log ( ( 1 + t / ( d λ 0 ) ) ) \log\det V_t\leq d\log((1+t/(d\lambda_0))) log det V t ≤ d log (( 1 + t / ( d λ 0 ))) log ( T ) \sqrt{\log(T)} log ( T ) β ≍ d log ( T ) \beta\asymp\sqrt{d\log(T)} β ≍ d log ( T ) price of adaptivity : it cannot be removed in general, since Lattimore [18 ] shows that the bound is unimprovable for adversarially adaptive designs, and the same phenomenon governs estimation and inference from adaptively collected data ([16 , 22 ] ). Sharper radii are known only under additional structure on the design, such as the stability condition of Lai & Wei [17 ] , i.e. convergence of the normalised covariance to a deterministic limit; we do not use such refinements, and the radius β \beta β
How the event is used. The event E \mathcal{E} E E \mathcal{E} E β \beta β θ ⋆ \theta^\star θ ⋆ ∥ ⋅ ∥ V t \lVert\cdot\rVert_{V_t} ∥ ⋅ ∥ V t β \beta β ω t \omega_t ω t ∥ ⋅ ∥ ∗ \lVert\cdot\rVert_\ast ∥ ⋅ ∥ ∗ β ω t 1 / 2 \beta\omega_t^{1/2} β ω t 1/2 Proposition D.2 ); the set C t ( α ) C_t(\alpha) C t ( α ) α \alpha α C t ( β ) C_t(\beta) C t ( β ) β \beta β β t \beta_t β t log ( ( det ( V t ) / det ( λ 0 I ) ) ) \log((\det(V_t)/\det(\lambda_0 I))) log (( det ( V t ) / det ( λ 0 I ))) d log ( ( 1 + T / ( d λ 0 ) ) ) d\log((1+T/(d\lambda_0))) d log (( 1 + T / ( d λ 0 ))) β t \beta_t β t
Further properties of the action set
Consequences of Assumption (A2). Setting θ ′ = 0 \theta'=0 θ ′ = 0 (1) yields m 2 ∥ v ∥ 2 ≤ J X ( v ) 2 ≤ M 2 ∥ v ∥ 2 \frac{m}{2}\lVert v\rVert^2\leq J_{\mathcal{X}}(v)^2\leq\frac{M}{2}\lVert v\rVert^2 2 m ∥ v ∥ 2 ≤ J X ( v ) 2 ≤ 2 M ∥ v ∥ 2 θ ⋆ ≠ 0 \theta^\star\neq0 θ ⋆ = 0
ℓ ⋆ : = sup x ∈ X ∥ x ∥ ∗ , J ⋆ : = J X ( θ ⋆ ) , ϱ ⋆ : = J ⋆ / ∥ θ ⋆ ∥ ∗ \ell_\star:=\sup_{x\in\mathcal{X}}\lVert x\rVert_\ast,\qquad J_\star:=J_{\mathcal{X}}(\theta^\star),\qquad\varrho_\star:=J_\star/\lVert\theta^\star\rVert_\ast ℓ ⋆ := x ∈ X sup ∥ x ∥ ∗ , J ⋆ := J X ( θ ⋆ ) , ϱ ⋆ := J ⋆ / ∥ θ ⋆ ∥ ∗ satisfy J ⋆ > 0 J_\star>0 J ⋆ > 0 m / 2 ≤ ϱ ⋆ ≤ ∥ x ⋆ ∥ ∗ ≤ ℓ ⋆ ≤ M / 2 , \sqrt{m/2}\leq\varrho_\star\leq\lVert x^\star\rVert_\ast\leq\ell_\star\leq\sqrt{M/2}, m /2 ≤ ϱ ⋆ ≤ ∥ x ⋆ ∥ ∗ ≤ ℓ ⋆ ≤ M /2 ,
{#eq:5}
where x ⋆ : = ∇ J X ( θ ⋆ ) x^\star:=\nabla J_{\mathcal{X}}(\theta^\star) x ⋆ := ∇ J X ( θ ⋆ ) X \mathcal{X} X J X J_{\mathcal{X}} J X ℓ ⋆ \ell_\star ℓ ⋆ ∥ ⋅ ∥ ∗ \lVert\cdot\rVert_\ast ∥ ⋅ ∥ ∗
Unit ball and ellipsoids. For X = B 2 d ( 1 ) \mathcal{X}=\mathbb{B}_2^d(1) X = B 2 d ( 1 ) ℓ ⋆ = ϱ ⋆ = 1 \ell_\star=\varrho_\star=1 ℓ ⋆ = ϱ ⋆ = 1 J ⋆ = ∥ θ ⋆ ∥ 2 J_\star=\lVert\theta^\star\rVert_2 J ⋆ = ∥ θ ⋆ ∥ 2 X = A 1 / 2 B 2 d ( 1 ) \mathcal{X}=\mathbb{A}^{1/2}\mathbb{B}_2^d(1) X = A 1/2 B 2 d ( 1 ) ∥ u ∥ ∗ : = ∥ A 1 / 2 u ∥ 2 \lVert u\rVert_\ast:=\lVert\mathbb{A}^{1/2}u\rVert_2 ∥ u ∥ ∗ := ∥ A 1/2 u ∥ 2 ℓ ⋆ = ϱ ⋆ = 1 \ell_\star=\varrho_\star=1 ℓ ⋆ = ϱ ⋆ = 1
C 2 C^2 C 2 X ⊆ B 2 d ( 1 ) \mathcal{X} \subseteq B_2^d(1) X ⊆ B 2 d ( 1 ) C 2 C^2 C 2 C 2 C^2 C 2 [ r − , r + ] [r_-,r_+] [ r − , r + ] 0 < r − ≤ r + < ∞ 0<r_- \le r_+<\infty 0 < r − ≤ r + < ∞ C + 2 C^2_+ C + 2 C 2 C^2 C 2 R d ∖ { 0 } \mathbb{R}^d \setminus\{0\} R d ∖ { 0 } u u u ∇ 2 J X ( u ) \nabla^2 J_{\mathcal{X}}(u) ∇ 2 J X ( u ) u u u u ⊥ u^\perp u ⊥ ∂ X \partial\mathcal{X} ∂ X x = ∇ J X ( u ) x=\nabla J_{\mathcal{X}}(u) x = ∇ J X ( u ) u u u ∂ X \partial\mathcal{X} ∂ X x x x ∇ 2 J X 2 = 2 J X ∇ 2 J X + 2 ∇ J X ∇ J X ⊤ \nabla^2 J_{\mathcal{X}}^2=2J_{\mathcal{X}}\nabla^2J_{\mathcal{X}}+2\nabla J_{\mathcal{X}}\nabla J_{\mathcal{X}}^\top ∇ 2 J X 2 = 2 J X ∇ 2 J X + 2∇ J X ∇ J X ⊤ u = θ / ∥ θ ∥ 2 u=\theta/\|\theta\|_2 u = θ /∥ θ ∥ 2 v = a u + w v=au+w v = a u + w w ⊥ u w\perp u w ⊥ u
v ⊤ ∇ 2 J X 2 ( θ ) v = 2 J X ( u ) w ⊤ ∇ 2 J X ( u ) w + 2 ⟨ x , v ⟩ 2 , r − ∥ w ∥ 2 2 ≤ w ⊤ ∇ 2 J X ( u ) w ≤ r + ∥ w ∥ 2 2 . (5) v^\top\nabla^2J_{\mathcal{X}}^2(\theta)v=2J_{\mathcal{X}}(u)w^\top\nabla^2J_{\mathcal{X}}(u)w+2\langle x,v\rangle^2,\qquad r_-\|w\|_2^2\le w^\top\nabla^2J_{\mathcal{X}}(u)w\le r_+\|w\|_2^2. \tag*{(5)} v ⊤ ∇ 2 J X 2 ( θ ) v = 2 J X ( u ) w ⊤ ∇ 2 J X ( u ) w + 2 ⟨ x , v ⟩ 2 , r − ∥ w ∥ 2 2 ≤ w ⊤ ∇ 2 J X ( u ) w ≤ r + ∥ w ∥ 2 2 . (5) The first (tangential) term captures the curvature of ∂ X \partial\mathcal{X} ∂ X ⟨ x , u ⟩ = J X ( u ) \langle x,u\rangle=J_{\mathcal{X}}(u) ⟨ x , u ⟩ = J X ( u ) u u u
ρ min : = min ∥ u ∥ 2 = 1 J X ( u ) ∈ ( 0 , 1 ] \rho_{\min}:=\min_{\|u\|_2=1}J_{\mathcal{X}}(u)\in(0,1] ρ m i n := ∥ u ∥ 2 = 1 min J X ( u ) ∈ ( 0 , 1 ] denote the distance from the origin to ∂ X \partial\mathcal{X} ∂ X X \mathcal{X} X X ⊆ B 2 d ( 1 ) \mathcal{X}\subseteq B_2^d(1) X ⊆ B 2 d ( 1 ) ∥ ⋅ ∥ ∗ = ∥ ⋅ ∥ 2 \|\cdot\|_*=\|\cdot\|_2 ∥ ⋅ ∥ ∗ = ∥ ⋅ ∥ 2 ℓ ∗ ≤ 1 \ell_*\le1 ℓ ∗ ≤ 1 ϱ ∗ ≥ ρ min \varrho_*\ge\rho_{\min} ϱ ∗ ≥ ρ m i n
Upper bound. Using J X ( u ) ≤ 1 J_{\mathcal{X}}(u)\le1 J X ( u ) ≤ 1 ⟨ x , v ⟩ 2 ≤ ∥ v ∥ 2 2 \langle x,v\rangle^2\le\|v\|_2^2 ⟨ x , v ⟩ 2 ≤ ∥ v ∥ 2 2
v ⊤ ∇ 2 J X 2 ( θ ) v ≤ 2 r + ∥ w ∥ 2 2 + 2 ∥ v ∥ 2 2 ≤ 2 ( r + + 1 ) ∥ v ∥ 2 2 = : M ∥ v ∥ 2 2 . v^\top\nabla^2J_{\mathcal{X}}^2(\theta)v\le2r_+\|w\|_2^2+2\|v\|_2^2\le2(r_++1)\|v\|_2^2=:M\|v\|_2^2. v ⊤ ∇ 2 J X 2 ( θ ) v ≤ 2 r + ∥ w ∥ 2 2 + 2∥ v ∥ 2 2 ≤ 2 ( r + + 1 ) ∥ v ∥ 2 2 =: M ∥ v ∥ 2 2 . Lower bound. Using J X ( u ) ≥ ρ min J_{\mathcal{X}}(u)\ge\rho_{\min} J X ( u ) ≥ ρ m i n ∣ ⟨ x , v ⟩ ∣ ≥ ρ min ∣ a ∣ − ∥ w ∥ 2 |\langle x,v\rangle|\ge\rho_{\min}|a|-\|w\|_2 ∣ ⟨ x , v ⟩ ∣ ≥ ρ m i n ∣ a ∣ − ∥ w ∥ 2
v ⊤ ∇ 2 J X 2 ( θ ) v ≥ 2 ρ min r − ∥ w ∥ 2 2 + 2 ( ρ min ∣ a ∣ − ∥ w ∥ 2 ) + 2 . v^\top\nabla^2J_{\mathcal{X}}^2(\theta)v\ge2\rho_{\min}r_-\|w\|_2^2+2\left(\rho_{\min}|a|-\|w\|_2\right)_+^2. v ⊤ ∇ 2 J X 2 ( θ ) v ≥ 2 ρ m i n r − ∥ w ∥ 2 2 + 2 ( ρ m i n ∣ a ∣ − ∥ w ∥ 2 ) + 2 . If ∥ w ∥ 2 ≥ ρ min ∣ a ∣ / 2 \|w\|_2\ge\rho_{\min}|a|/2 ∥ w ∥ 2 ≥ ρ m i n ∣ a ∣/2 ∥ v ∥ 2 2 ≤ 5 ∥ w ∥ 2 2 / ρ min 2 \|v\|_2^2\le5\|w\|_2^2/\rho_{\min}^2 ∥ v ∥ 2 2 ≤ 5∥ w ∥ 2 2 / ρ m i n 2 2 ρ min 3 r − ∥ v ∥ 2 2 / 5 2\rho_{\min}^3r_-\|v\|_2^2/5 2 ρ m i n 3 r − ∥ v ∥ 2 2 /5 ∥ v ∥ 2 2 ≤ 5 a 2 / 4 \|v\|_2^2\le5a^2/4 ∥ v ∥ 2 2 ≤ 5 a 2 /4 2 ρ min 2 ∥ v ∥ 2 2 / 5 2\rho_{\min}^2\|v\|_2^2/5 2 ρ m i n 2 ∥ v ∥ 2 2 /5
v ⊤ ∇ 2 J X 2 ( θ ) v ≥ 2 ρ min 2 5 min { 1 , ρ min r − } ∥ v ∥ 2 2 ≥ ρ min 2 9 min { 1 , ρ min r − 3 } ∥ v ∥ 2 2 = : m ∥ v ∥ 2 2 . v^\top\nabla^2J_{\mathcal{X}}^2(\theta)v\ge\frac{2\rho_{\min}^2}{5}\min\{1,\rho_{\min}r_-\}\|v\|_2^2\ge\frac{\rho_{\min}^2}{9}\min\left\{1,\frac{\rho_{\min}r_-}{3}\right\}\|v\|_2^2=:m\|v\|_2^2. v ⊤ ∇ 2 J X 2 ( θ ) v ≥ 5 2 ρ m i n 2 min { 1 , ρ m i n r − } ∥ v ∥ 2 2 ≥ 9 ρ m i n 2 min { 1 , 3 ρ m i n r − } ∥ v ∥ 2 2 =: m ∥ v ∥ 2 2 . The two-sided bound of Assumption (A2) follows by integrating twice along the segment from θ ′ \theta' θ ′ θ \theta θ J X 2 J_{\mathcal{X}}^2 J X 2 C 1 C^1 C 1 R d \mathbb{R}^d R d 2 J X ∇ J X 2J_{\mathcal{X}}\nabla J_{\mathcal{X}} 2 J X ∇ J X 2 ∥ θ ∥ 2 2\|\theta\|_2 2∥ θ ∥ 2 0 0 0
Role of curvature and excluded geometries. In tangential directions, the Hessian of J X 2 J_X^2 J X 2 2 J X ( u ) 2J_X(u) 2 J X ( u ) r + r_+ r + M M M r − r_- r − m m m 2 J X ( u ) 2 ≥ 2 ρ min 2 2J_X(u)^2 \ge2\rho_{\min}^2 2 J X ( u ) 2 ≥ 2 ρ m i n 2 m m m Figure 4 ). The two limits are the excluded cases. A flat face is a direction θ \theta θ J X J_X J X θ \theta θ θ \theta θ M = ∞ M = \infty M = ∞ x k x_k x k J X ( θ ) = ⟨ θ , x k ⟩ J_X(\theta) = \langle\theta, x_k\rangle J X ( θ ) = ⟨ θ , x k ⟩
∇ 2 J X 2 = 2 x k x k T \nabla^2 J_X^2 = 2x_kx_k^{\mathsf T} ∇ 2 J X 2 = 2 x k x k T has rank one, D 2 J X 2 ( θ , θ ′ ) = ⟨ θ − θ ′ , x k ⟩ 2 D^2J_X^2(\theta,\theta') = \langle\theta-\theta',x_k\rangle^2 D 2 J X 2 ( θ , θ ′ ) = ⟨ θ − θ ′ , x k ⟩ 2 θ − θ ′ ⊥ x k \theta-\theta' \perp x_k θ − θ ′ ⊥ x k m = 0 m = 0 m = 0 C 2 C^2 C 2 θ ≠ 0 \theta\ne0 θ = 0 θ \theta θ Proposition D.3 ); this allows a perturbation of the optimistic parameter to reach the arm, and is what the energy argument of the proof exploits. The constants are not sharp: for the unit ball, r ± = ρ min = 1 r_{\pm} = \rho_{\min} = 1 r ± = ρ m i n = 1 (5) equals 2 ∥ v ∥ 2 2\lVert v\rVert^2 2 ∥ v ∥ 2 m = M = 2 m = M = 2 m = M = 2 m = 2 / 5 m = 2/5 m = 2/5 M = 4 M = 4 M = 4
Failure case
Proposition 6.1 (Linear regret on a polytope). In the instance above, for every λ 0 > 0 \lambda_0 > 0 λ 0 > 0 T T T
0 < α ≤ α 0 : = ( 1 − a 2 ) λ 0 2 a ε , 0 < \alpha\le\alpha_0 := \frac{(1-a^2)\sqrt{\lambda_0}}{2a}\,\varepsilon, 0 < α ≤ α 0 := 2 a ( 1 − a 2 ) λ 0 ε , with probability at least
p 0 : = 1 2 ( 1 − exp ( − ( 1 − a 2 ) λ 0 ε 2 2 a 2 μ 2 ) ) > 0 , p_0 := \frac{1}{2}\left(1-\exp\left(-\frac{(1-a^2)\lambda_0\varepsilon^2}{2a^2\mu^2}\right)\right) > 0, p 0 := 2 1 ( 1 − exp ( − 2 a 2 μ 2 ( 1 − a 2 ) λ 0 ε 2 ) ) > 0 , α \alpha α x t ∈ { e 1 , − e 1 } x_t \in\{e_1,-e_1\} x t ∈ { e 1 , − e 1 } t ∈ [ T ] t \in[T] t ∈ [ T ] λ min ( V t ) = λ 0 \lambda_{\min}(V_t) = \lambda_0 λ m i n ( V t ) = λ 0 t t t R T ≥ Δ T R_T \ge\Delta T R T ≥ Δ T E [ R T ] ≥ p 0 Δ T \mathbb{E}[R_T] \ge p_0\Delta T E [ R T ] ≥ p 0 Δ T
Proof of Proposition 6.1. Step 1: the algorithm plays vertices. For x ∈ X a x \in\mathcal{X}_a x ∈ X a max θ ∈ C t ( α ) ⟨ x , θ ⟩ = ⟨ x , θ ^ t ⟩ + α ∥ x ∥ V t − 1 = : g t ( x ) \max_{\theta\in C_t(\alpha)} \langle x,\theta\rangle= \langle x,\hat{\theta}_t\rangle+ \alpha\lVert x\rVert_{V_t^{-1}} =: g_t(x) max θ ∈ C t ( α ) ⟨ x , θ ⟩ = ⟨ x , θ ^ t ⟩ + α ∥ x ∥ V t − 1 =: g t ( x ) x t ∈ arg max X a g t x_t \in\arg\max_{\mathcal{X}_a} g_t x t ∈ arg max X a g t g t g_t g t g t g_t g t X a \mathcal{X}_a X a
V t = diag ( λ 0 + n t , 1 , λ 0 + a 2 n t , 2 ) , V_t = \operatorname{diag}(\lambda_0+n_{t,1},\lambda_0+a^2n_{t,2}), V t = diag ( λ 0 + n t , 1 , λ 0 + a 2 n t , 2 ) , where n t , i n_{t,i} n t , i s < t s<t s < t x s x_s x s i i i
θ ^ t , 1 = ( λ 0 + n t , 1 ) − 1 ∑ s < t x s = ± e 1 ⟨ x s , e 1 ⟩ y s , \hat{\theta}_{t,1}=(\lambda_0+n_{t,1})^{-1}\sum_{\substack{s<t\\x_s=\pm e_1}}\langle x_s,e_1\rangle y_s, θ ^ t , 1 = ( λ 0 + n t , 1 ) − 1 s < t x s = ± e 1 ∑ ⟨ x s , e 1 ⟩ y s , and the four indices are
g t ( ± e 1 ) = ± θ ^ t , 1 + α λ 0 + n t , 1 , g t ( ± a e 2 ) = ± a θ ^ t , 2 + a α λ 0 + a 2 n t , 2 . g_t(\pm e_1)=\pm\hat{\theta}_{t,1}+\frac{\alpha}{\sqrt{\lambda_0+n_{t,1}}},\qquad
g_t(\pm ae_2)=\pm a\hat{\theta}_{t,2}+\frac{a\alpha}{\sqrt{\lambda_0+a^2n_{t,2}}}. g t ( ± e 1 ) = ± θ ^ t , 1 + λ 0 + n t , 1 α , g t ( ± a e 2 ) = ± a θ ^ t , 2 + λ 0 + a 2 n t , 2 a α . Step 2: a sufficient condition for locking on the first axis. Let η ~ s : = ⟨ x s , e 1 ⟩ η s \widetilde{\eta}_s := \langle x_s,e_1\rangle\eta_s η s := ⟨ x s , e 1 ⟩ η s x s ∈ { ± e 1 } x_s\in\{\pm e_1\} x s ∈ { ± e 1 } η ~ s : = η s \widetilde{\eta}_s := \eta_s η s := η s W n : = ∑ s ≤ n η ~ s W_n := \sum_{s\le n}\widetilde{\eta}_s W n := ∑ s ≤ n η s ⟨ x s , e 1 ⟩ ∈ { ± 1 } \langle x_s,e_1\rangle\in\{\pm1\} ⟨ x s , e 1 ⟩ ∈ { ± 1 } F s − 1 \mathcal{F}_{s-1} F s − 1 η s ∼ N ( 0 , μ 2 ) \eta_s\sim\mathcal{N}(0,\mu^2) η s ∼ N ( 0 , μ 2 ) η ~ s \widetilde{\eta}_s η s F s − 1 \mathcal{F}_{s-1} F s − 1 N ( 0 , μ 2 ) \mathcal{N}(0,\mu^2) N ( 0 , μ 2 ) η ~ s \widetilde{\eta}_s η s N ( 0 , μ 2 ) \mathcal{N}(0,\mu^2) N ( 0 , μ 2 ) x s ∈ { ± e 1 } x_s\in\{\pm e_1\} x s ∈ { ± e 1 } s < t s<t s < t n : = t − 1 n:=t-1 n := t − 1 n t , 1 = n n_{t,1}=n n t , 1 = n n t , 2 = 0 n_{t,2}=0 n t , 2 = 0 θ ^ t , 2 = 0 \hat{\theta}_{t,2}=0 θ ^ t , 2 = 0 ⟨ x s , e 1 ⟩ y s = ε + η ~ s \langle x_s,e_1\rangle y_s=\varepsilon+\widetilde{\eta}_s ⟨ x s , e 1 ⟩ y s = ε + η s
θ ^ t , 1 = n ε + W n λ 0 + n , \hat{\theta}_{t,1}=\frac{n\varepsilon+W_n}{\lambda_0+n}, θ ^ t , 1 = λ 0 + n n ε + W n , so that x t ∈ { ± e 1 } x_t\in\{\pm e_1\} x t ∈ { ± e 1 }
∣ θ ^ t , 1 ∣ + α λ 0 + n > a α λ 0 , (6) \left|\hat{\theta}_{t,1}\right|+\frac{\alpha}{\sqrt{\lambda_0+n}}>\frac{a\alpha}{\sqrt{\lambda_0}}, \tag*{(6)} θ ^ t , 1 + λ 0 + n α > λ 0 a α , (6) the left-hand side being the larger of g t ( ± e 1 ) g_t(\pm e_1) g t ( ± e 1 ) g t ( ± a e 2 ) g_t(\pm ae_2) g t ( ± a e 2 ) n ≤ n 0 : = λ 0 ( a − 2 − 1 ) n\le n_0:=\lambda_0(a^{-2}-1) n ≤ n 0 := λ 0 ( a − 2 − 1 ) (6) holds whatever the rewards, because α / λ 0 + n > a α / λ 0 \alpha/\sqrt{\lambda_0+n}>a\alpha/\sqrt{\lambda_0} α / λ 0 + n > a α / λ 0 n ≥ N 0 : = ⌈ n 0 ⌉ n\ge N_0:=\lceil n_0\rceil n ≥ N 0 := ⌈ n 0 ⌉ (6) holds as soon as θ ^ t , 1 > a α / λ 0 \hat{\theta}_{t,1}>a\alpha/\sqrt{\lambda_0} θ ^ t , 1 > a α / λ 0
W n + δ n > c α , δ : = ε − a α λ 0 , c α : = a α λ 0 . W_n+\delta n>c_\alpha,\qquad\delta:=\varepsilon-\frac{a\alpha}{\sqrt{\lambda_0}},\qquad c_\alpha:=a\alpha\sqrt{\lambda_0}. W n + δ n > c α , δ := ε − λ 0 a α , c α := a α λ 0 . Since x 1 ∈ { ± e 1 } x_1 \in\{\pm e_1\} x 1 ∈ { ± e 1 } t = 1 t = 1 t = 1 g 1 ( ± e 1 ) = α / λ 0 > a α / λ 0 = g 1 ( ± a e 2 ) g_1(\pm e_1) = \alpha/\sqrt{\lambda_0} > a\alpha/\sqrt{\lambda_0} = g_1(\pm ae_2) g 1 ( ± e 1 ) = α / λ 0 > a α / λ 0 = g 1 ( ± a e 2 ) t t t
{ W n + δ n > c α for all n ≥ N 0 } ⊆ { x t ∈ { ± e 1 } for all t ≥ 1 } . \{W_n + \delta n > c_\alpha\text{ for all } n \ge N_0\} \subseteq\{x_t \in\{\pm e_1\} \text{ for all } t \ge1\}. { W n + δ n > c α for all n ≥ N 0 } ⊆ { x t ∈ { ± e 1 } for all t ≥ 1 } . Step 3: the probability of the left-hand event. For α ≤ α 0 \alpha\le\alpha_0 α ≤ α 0 a α / λ 0 ≤ ( 1 − a 2 ) ε / 2 a\alpha/\sqrt{\lambda_0} \le(1-a^2)\varepsilon/2 a α / λ 0 ≤ ( 1 − a 2 ) ε /2 δ ≥ ε / 2 \delta\ge\varepsilon/2 δ ≥ ε /2 ( 1 − a 2 ) ( λ 0 + N 0 ) ≤ N 0 (1-a^2)(\lambda_0+N_0) \le N_0 ( 1 − a 2 ) ( λ 0 + N 0 ) ≤ N 0 N 0 ≥ n 0 N_0 \ge n_0 N 0 ≥ n 0
h : = δ N 0 − c α = ε N 0 − a α λ 0 ( λ 0 + N 0 ) ≥ ε N 0 − ( 1 − a 2 ) ε 2 ( λ 0 + N 0 ) ≥ ε N 0 2 . h := \delta N_0-c_\alpha= \varepsilon N_0-\frac{a\alpha}{\sqrt{\lambda_0}}(\lambda_0+N_0) \ge\varepsilon N_0-\frac{(1-a^2)\varepsilon}{2}(\lambda_0+N_0) \ge\frac{\varepsilon N_0}{2}. h := δ N 0 − c α = ε N 0 − λ 0 a α ( λ 0 + N 0 ) ≥ ε N 0 − 2 ( 1 − a 2 ) ε ( λ 0 + N 0 ) ≥ 2 ε N 0 . Now { W n + δ n > c α ∀ n ≥ N 0 } ⊇ { W N 0 ≥ 0 } ∩ { W N 0 + k − W N 0 + δ k > − h ∀ k ≥ 1 } \{W_n+\delta n>c_\alpha\ \forall n\ge N_0\}\supseteq\{W_{N_0}\ge0\}\cap\{W_{N_0+k}-W_{N_0}+\delta k>-h\ \forall k\ge1\} { W n + δ n > c α ∀ n ≥ N 0 } ⊇ { W N 0 ≥ 0 } ∩ { W N 0 + k − W N 0 + δ k > − h ∀ k ≥ 1 } W N 0 + δ N 0 ≥ c α + h > c α W_{N_0}+\delta N_0\ge c_\alpha+h>c_\alpha W N 0 + δ N 0 ≥ c α + h > c α W N 0 + k + δ ( N 0 + k ) > ( c α + h ) − h W_{N_0+k}+\delta(N_0+k)>(c_\alpha+h)-h W N 0 + k + δ ( N 0 + k ) > ( c α + h ) − h F N 0 \mathcal{F}_{N_0} F N 0 1 / 2 1/2 1/2 W N 0 ∼ N ( 0 , μ 2 N 0 ) W_{N_0}\sim\mathcal{N}(0,\mu^2N_0) W N 0 ∼ N ( 0 , μ 2 N 0 ) λ : = 2 δ / μ 2 \lambda:=2\delta/\mu^2 λ := 2 δ / μ 2 F N 0 \mathcal{F}_{N_0} F N 0 M k : = exp ( − λ ( W N 0 + k − W N 0 ) − λ 2 μ 2 k / 2 ) M_k:=\exp\left(-\lambda(W_{N_0+k}-W_{N_0})-\lambda^2\mu^2k/2\right) M k := exp ( − λ ( W N 0 + k − W N 0 ) − λ 2 μ 2 k /2 ) M 0 = 1 M_0=1 M 0 = 1 W N 0 + k − W N 0 + δ k ≤ − h W_{N_0+k}-W_{N_0}+\delta k\le-h W N 0 + k − W N 0 + δ k ≤ − h log M k ≥ λ h + k ( λ δ − λ 2 μ 2 / 2 ) = λ h \log M_k\ge\lambda h+k(\lambda\delta-\lambda^2\mu^2/2)=\lambda h log M k ≥ λh + k ( λ δ − λ 2 μ 2 /2 ) = λh [29 ] ) therefore gives
P ( ∃ k ≥ 1 : W N 0 + k − W N 0 + δ k ≤ − h ∣ F N 0 ) ≤ e − λ h = e − 2 δ h / μ 2 ≤ e − ε 2 N 0 / ( 2 μ 2 ) ≤ exp ( − ( 1 − a 2 ) λ 0 ε 2 2 a 2 μ 2 ) , \mathbb{P}\left(\exists k\ge1:W_{N_0+k}-W_{N_0}+\delta k\le-h\mid\mathcal{F}_{N_0}\right)\le e^{-\lambda h}=e^{-2\delta h/\mu^2}\le e^{-\varepsilon^2N_0/(2\mu^2)}\le\exp\left(-\frac{(1-a^2)\lambda_0\varepsilon^2}{2a^2\mu^2}\right), P ( ∃ k ≥ 1 : W N 0 + k − W N 0 + δ k ≤ − h ∣ F N 0 ) ≤ e − λh = e − 2 δ h / μ 2 ≤ e − ε 2 N 0 / ( 2 μ 2 ) ≤ exp ( − 2 a 2 μ 2 ( 1 − a 2 ) λ 0 ε 2 ) , using N 0 ≥ n 0 = ( 1 − a 2 ) λ 0 / a 2 N_0\ge n_0=(1-a^2)\lambda_0/a^2 N 0 ≥ n 0 = ( 1 − a 2 ) λ 0 / a 2 P ( W N 0 ≥ 0 ) = 1 / 2 \mathbb{P}(W_{N_0}\ge0)=1/2 P ( W N 0 ≥ 0 ) = 1/2 p 0 p_0 p 0
Step 4: consequences. On the locking event, x s x s T = e 1 e 1 T x_sx_s^\mathsf{T}=e_1e_1^\mathsf{T} x s x s T = e 1 e 1 T s s s V t = diag ( λ 0 + t − 1 , λ 0 ) V_t=\operatorname{diag}(\lambda_0+t-1,\lambda_0) V t = diag ( λ 0 + t − 1 , λ 0 ) λ min ( V t ) = λ 0 \lambda_{\min}(V_t)=\lambda_0 λ m i n ( V t ) = λ 0 ⟨ θ ⋆ , x ⋆ − x t ⟩ = a ∓ ε ≥ Δ \langle\theta^\star,x^\star-x_t\rangle=a\mp\varepsilon\ge\Delta ⟨ θ ⋆ , x ⋆ − x t ⟩ = a ∓ ε ≥ Δ R T ≥ Δ T R_T\ge\Delta T R T ≥ Δ T
Pseudo-code of alpha-LinUCB
Algorithm 1 summarises the algorithm analysed in this paper; we briefly comment on its steps, complementing the description given in Section 1.
Algorithm 1: α \alpha α
Input: Action set X \mathcal{X} X T T T λ 0 > 0 \lambda_0>0 λ 0 > 0 α > 0 \alpha>0 α > 0
Initialize: V 1 ← λ 0 I V_1\leftarrow\lambda_0I V 1 ← λ 0 I
for t = 1 , … , T t=1,\ldots,T t = 1 , … , T do
Compute the ridge estimate θ ^ t \hat{\theta}_t θ ^ t
Select an action–parameter: ( x t , θ t ) ∈ arg max ( x , θ ) ∈ X × C t ( α ) ⟨ x , θ ⟩ (x_t,\theta_t)\in\arg\max_{(x,\theta)\in\mathcal{X}\times C_t(\alpha)}\langle x,\theta\rangle ( x t , θ t ) ∈ arg max ( x , θ ) ∈ X × C t ( α ) ⟨ x , θ ⟩
Play x t x_t x t y t y_t y t
Update V t + 1 ← V t + x t x t T V_{t+1}\leftarrow V_t+x_tx_t^\mathsf{T} V t + 1 ← V t + x t x t T
Estimation (line 4). The ridge estimate is θ ^ t = V t − 1 ∑ s = 1 t − 1 x s y s \hat{\theta}_t=V_t^{-1}\sum_{s=1}^{t-1}x_sy_s θ ^ t = V t − 1 ∑ s = 1 t − 1 x s y s V t = λ 0 I + ∑ s = 1 t − 1 x s x s T V_t=\lambda_0I+\sum_{s=1}^{t-1}x_sx_s^\mathsf{T} V t = λ 0 I + ∑ s = 1 t − 1 x s x s T O ( d 2 ) O(d^2) O ( d 2 )
Decision rule (line 5). The pair ( x t , θ t ) (x_t,\theta_t) ( x t , θ t ) ⟨ x , θ ⟩ \langle x,\theta\rangle ⟨ x , θ ⟩ X × C t ( α ) \mathcal{X}\times C_t(\alpha) X × C t ( α ) C t ( α ) = { θ : ∥ θ − θ ^ t ∥ V t ≤ α } C_t(\alpha)=\{\theta:\|\theta-\hat{\theta}_t\|_{V_t}\le\alpha\} C t ( α ) = { θ : ∥ θ − θ ^ t ∥ V t ≤ α } θ \theta θ x x x
x t ∈ arg max x ∈ X { ⟨ θ ^ t , x ⟩ + α ∥ x ∥ V t − 1 } , x_t\in\arg\max_{x\in\mathcal{X}}\left\{\langle\hat{\theta}_t,x\rangle+\alpha\|x\|_{V_t^{-1}}\right\}, x t ∈ arg x ∈ X max { ⟨ θ ^ t , x ⟩ + α ∥ x ∥ V t − 1 } , the familiar upper-confidence-bound rule, with ∥ x ∥ V t − 1 \|x\|_{V_t^{-1}} ∥ x ∥ V t − 1 x t x_t x t θ t = θ ^ t + α V t − 1 x t / ∥ x t ∥ V t − 1 \theta_t=\hat{\theta}_t+\alpha V_t^{-1}x_t/\|x_t\|_{V_t^{-1}} θ t = θ ^ t + α V t − 1 x t /∥ x t ∥ V t − 1
The radius α \alpha α The only departure from standard LinUCB is that the radius α \alpha α β \beta β α = β \alpha=\beta α = β [1 ] , while for α < β \alpha<\beta α < β C t ( α ) C_t(\alpha) C t ( α ) θ ⋆ \theta^\star θ ⋆ α → 0 \alpha\to0 α → 0 β \beta β
Spectral growth
Let β + : = α + β \beta_{+}:=\alpha+\beta β + := α + β α > 0 \alpha>0 α > 0 C t ( α ) C_t(\alpha) C t ( α ) α \alpha α α \alpha α β \beta β C t ( β ) C_t(\beta) C t ( β )
E : = { θ ⋆ ∈ ⋂ t ∈ [ T ] C t ( β ) } = { ∥ θ ^ t − θ ⋆ ∥ V t ≤ β for all t ∈ [ T ] } , P ( E ) ≥ 1 − δ . \mathcal{E}:=\left\{\theta^\star\in\bigcap_{t\in[T]}C_t(\beta)\right\}=\left\{\|\hat{\theta}_t-\theta^\star\|_{V_t}\leq\beta\text{ for all }t\in[T]\right\},\qquad\mathbb{P}(\mathcal{E})\geq1-\delta. E := ⎩ ⎨ ⎧ θ ⋆ ∈ t ∈ [ T ] ⋂ C t ( β ) ⎭ ⎬ ⎫ = { ∥ θ ^ t − θ ⋆ ∥ V t ≤ β for all t ∈ [ T ] } , P ( E ) ≥ 1 − δ . Preliminary results
Localisation. We introduce the set N ⋆ \mathcal{N}_\star N ⋆ ∥ ⋅ ∥ ⋆ \|\cdot\|_\star ∥ ⋅ ∥ ⋆ θ ⋆ \theta^\star θ ⋆ J X J_{\mathcal{X}} J X ∥ ⋅ ∥ ⋆ \|\cdot\|_\star ∥ ⋅ ∥ ⋆ θ ⋆ \theta^\star θ ⋆
N ⋆ : = { θ ∈ R d : ∥ θ − θ ⋆ ∥ ⋆ ≤ J ⋆ 2 ℓ ⋆ } , \mathcal{N}_\star:=\left\{\theta\in\mathbb{R}^d:\|\theta-\theta^\star\|_\star\leq\frac{J_\star}{2\ell_\star}\right\}, N ⋆ := { θ ∈ R d : ∥ θ − θ ⋆ ∥ ⋆ ≤ 2 ℓ ⋆ J ⋆ } , 1 2 J ⋆ ≤ J X ( θ ) ≤ 3 2 J ⋆ and 1 2 ∥ θ ⋆ ∥ ⋆ ≤ ∥ θ ∥ ⋆ ≤ 3 2 ∥ θ ⋆ ∥ ⋆ ( θ ∈ N ⋆ ) . (7) \frac{1}{2}J^\star\leq J_{\mathcal{X}}(\theta)\leq\frac{3}{2}J_\star\quad\text{and}\quad\frac{1}{2}\|\theta^\star\|_\star\leq\|\theta\|_\star\leq\frac{3}{2}\|\theta^\star\|_\star\qquad(\theta\in\mathcal{N}_\star). \tag*{(7)} 2 1 J ⋆ ≤ J X ( θ ) ≤ 2 3 J ⋆ and 2 1 ∥ θ ⋆ ∥ ⋆ ≤ ∥ θ ∥ ⋆ ≤ 2 3 ∥ θ ⋆ ∥ ⋆ ( θ ∈ N ⋆ ) . (7) where the two-sided bounds follow from the ℓ ⋆ \ell_\star ℓ ⋆ J X J_{\mathcal{X}} J X J ⋆ ≤ ℓ ⋆ ∥ θ ⋆ ∥ ⋆ J_\star\leq\ell_\star\|\theta^\star\|_\star J ⋆ ≤ ℓ ⋆ ∥ θ ⋆ ∥ ⋆ N ⋆ \mathcal{N}_\star N ⋆
Elementary propositions. The proofs rest on the following elementary propositions about the support function and about the width ω t = max u ∈ B 2 d ( 1 ) ∥ V t − 1 / 2 u ∥ 2 \omega_t=\max_{u\in\mathbb{B}_2^d(1)}\|V_t^{-1/2}u\|_2 ω t = max u ∈ B 2 d ( 1 ) ∥ V t − 1/2 u ∥ 2 ω t \omega_t ω t
Proposition D.1 (Consequences of the assumption). Under Assumption (A2), with X \mathcal{X} X
(i) J X J_{\mathcal{X}} J X J X 2 J_{\mathcal{X}}^2 J X 2 R d \mathbb{R}^d R d J X ( θ ) > 0 J_{\mathcal{X}}(\theta)>0 J X ( θ ) > 0 θ ≠ 0 \theta\ne0 θ = 0 J X = ( J X 2 ) 1 / 2 J_{\mathcal{X}}=(J_{\mathcal{X}}^2)^{1/2} J X = ( J X 2 ) 1/2 R d ∖ { 0 } \mathbb{R}^d\setminus\{0\} R d ∖ { 0 }
(ii) m 2 ∥ v ∥ ⋆ 2 ≤ J X ( v ) 2 ≤ M 2 ∥ v ∥ ⋆ 2 \frac{m}{2}\|v\|_\star^2\leq J_{\mathcal{X}}(v)^2\leq\frac{M}{2}\|v\|_\star^2 2 m ∥ v ∥ ⋆ 2 ≤ J X ( v ) 2 ≤ 2 M ∥ v ∥ ⋆ 2 v ∈ R d v\in\mathbb{R}^d v ∈ R d ℓ ⋆ ≤ M / 2 \ell_\star\leq\sqrt{M/2} ℓ ⋆ ≤ M /2 X \mathcal{X} X ∣ J X ( θ ) − J X ( θ ′ ) ∣ ≤ ℓ ⋆ ∥ θ − θ ′ ∥ ⋆ |J_{\mathcal{X}}(\theta)-J_{\mathcal{X}}(\theta')|\leq\ell_\star\|\theta-\theta'\|_\star ∣ J X ( θ ) − J X ( θ ′ ) ∣ ≤ ℓ ⋆ ∥ θ − θ ′ ∥ ⋆ θ , θ ′ \theta,\theta' θ , θ ′ J X ( θ ) ≤ ℓ ⋆ ∥ θ ∥ ⋆ J_{\mathcal{X}}(\theta)\leq\ell_\star\|\theta\|_\star J X ( θ ) ≤ ℓ ⋆ ∥ θ ∥ ⋆ J ⋆ ≥ m / 2 ∥ θ ⋆ ∥ ⋆ > 0 J_\star\geq\sqrt{m/2}\|\theta^\star\|_\star>0 J ⋆ ≥ m /2 ∥ θ ⋆ ∥ ⋆ > 0
(iii) For every θ ≠ 0 \theta\ne0 θ = 0 x ↦ ⟨ θ , x ⟩ x\mapsto\langle\theta,x\rangle x ↦ ⟨ θ , x ⟩ X \mathcal{X} X ∇ J X ( θ ) \nabla J_{\mathcal{X}}(\theta) ∇ J X ( θ ) ∥ ∇ J X ( θ ) ∥ ⋆ ∘ ≤ ℓ ⋆ \|\nabla J_{\mathcal{X}}(\theta)\|_\star^\circ\leq\ell_\star ∥∇ J X ( θ ) ∥ ⋆ ∘ ≤ ℓ ⋆ ∇ J X 2 ( θ ) = 2 J X ( θ ) ∇ J X ( θ ) \nabla J_{\mathcal{X}}^2(\theta)=2J_{\mathcal{X}}(\theta)\nabla J_{\mathcal{X}}(\theta) ∇ J X 2 ( θ ) = 2 J X ( θ ) ∇ J X ( θ ) x ⋆ = ∇ J X ( θ ⋆ ) x^\star=\nabla J_{\mathcal{X}}(\theta^\star) x ⋆ = ∇ J X ( θ ⋆ )
(iv) the gradient of J X 2 J_{\mathcal{X}}^2 J X 2 M M M ( R d , ∥ ⋅ ∥ ⋆ ) (\mathbb{R}^d,\|\cdot\|_\star) ( R d , ∥ ⋅ ∥ ⋆ ) ( R d , ∥ ⋅ ∥ X ∘ ) (\mathbb{R}^d,\|\cdot\|_{\mathcal{X}}^\circ) ( R d , ∥ ⋅ ∥ X ∘ )
∥ ∇ J X 2 ( θ ) − ∇ J X 2 ( θ ′ ) ∥ X ∘ ≤ M ∥ θ − θ ′ ∥ ⋆ ( θ , θ ′ ∈ R d ) . (8) \|\nabla J_{\mathcal{X}}^2(\theta)-\nabla J_{\mathcal{X}}^2(\theta')\|_{\mathcal{X}}^\circ\leq M\|\theta-\theta'\|_\star\qquad(\theta,\theta'\in\mathbb{R}^d). \tag*{(8)} ∥∇ J X 2 ( θ ) − ∇ J X 2 ( θ ′ ) ∥ X ∘ ≤ M ∥ θ − θ ′ ∥ ⋆ ( θ , θ ′ ∈ R d ) . (8) (v) At every round t t t V t ⪰ λ 0 I ≻ 0 V_t\succeq\lambda_0I\succ0 V t ⪰ λ 0 I ≻ 0 θ t ≠ 0 \theta_t\ne0 θ t = 0 x t = ∇ J X ( θ t ) x_t=\nabla J_{\mathcal{X}}(\theta_t) x t = ∇ J X ( θ t )
Proof of Proposition D.1. (i) J X J_{\mathcal{X}} J X J X ( 0 ) = 0 J_{\mathcal{X}}(0)=0 J X ( 0 ) = 0 X \mathcal{X} X R d \mathbb{R}^d R d ε > 0 \varepsilon>0 ε > 0 { x : ∥ x ∥ X ∘ ≤ ε } ⊆ X \{x:\|x\|_{\mathcal{X}}^\circ\leq\varepsilon\}\subseteq\mathcal{X} { x : ∥ x ∥ X ∘ ≤ ε } ⊆ X J X ( θ ) ≥ sup ∥ x ∥ X ∘ ≤ ε ⟨ θ , x ⟩ = ε ∥ θ ∥ ⋆ > 0 J_{\mathcal{X}}(\theta)\geq\sup_{\|x\|_{\mathcal{X}}^\circ\leq\varepsilon}\langle\theta,x\rangle=\varepsilon\|\theta\|_\star>0 J X ( θ ) ≥ sup ∥ x ∥ X ∘ ≤ ε ⟨ θ , x ⟩ = ε ∥ θ ∥ ⋆ > 0 θ ≠ 0 \theta\ne0 θ = 0 J X ≥ 0 J_{\mathcal{X}}\geq0 J X ≥ 0 s ↦ s 2 s\mapsto s^2 s ↦ s 2 [ 0 , ∞ ) [0,\infty) [ 0 , ∞ ) J X 2 J_{\mathcal{X}}^2 J X 2 J X 2 J_{\mathcal{X}}^2 J X 2 J X = ( J X 2 ) 1 / 2 J_{\mathcal{X}}=(J_{\mathcal{X}}^2)^{1/2} J X = ( J X 2 ) 1/2
(ii) J X 2 ( 0 ) = 0 J_{\mathcal{X}}^2(0)=0 J X 2 ( 0 ) = 0 0 0 0 J X 2 ≥ 0 J_{\mathcal{X}}^2\geq0 J X 2 ≥ 0 ∇ J X 2 ( 0 ) = 0 \nabla J_{\mathcal{X}}^2(0)=0 ∇ J X 2 ( 0 ) = 0 D J X 2 ( v , 0 ) = J X ( v ) 2 D J_{\mathcal{X}}^2(v,0)=J_{\mathcal{X}}(v)^2 D J X 2 ( v , 0 ) = J X ( v ) 2 θ ′ = 0 \theta'=0 θ ′ = 0 θ ∈ X \theta\in\mathcal{X} θ ∈ X ∥ x ∥ X ∘ ≤ 1 \|x\|_{\mathcal{X}}^\circ\leq1 ∥ x ∥ X ∘ ≤ 1 ∥ x ∥ ⋆ = sup ∥ v ∥ ⋆ ≤ 1 ⟨ x , v ⟩ ≤ sup ∥ v ∥ ⋆ ≤ 1 J X ( v ) ≤ M / 2 \|x\|_\star=\sup_{\|v\|_\star\leq1}\langle x,v\rangle\leq\sup_{\|v\|_\star\leq1}J_{\mathcal{X}}(v)\leq\sqrt{M/2} ∥ x ∥ ⋆ = sup ∥ v ∥ ⋆ ≤ 1 ⟨ x , v ⟩ ≤ sup ∥ v ∥ ⋆ ≤ 1 J X ( v ) ≤ M /2 ℓ ⋆ ≤ M / 2 \ell_\star\leq\sqrt{M/2} ℓ ⋆ ≤ M /2 X \mathcal{X} X x ∈ X x\in\mathcal{X} x ∈ X ⟨ θ , ⋅ ⟩ \langle\theta,\cdot\rangle ⟨ θ , ⋅ ⟩ J X ( θ ) − J X ( θ ′ ) ≤ ⟨ θ , x ⟩ − ⟨ θ ′ , x ⟩ ≤ ∥ x ∥ ⋆ ∥ θ − θ ′ ∥ ⋆ ≤ ℓ ⋆ ∥ θ − θ ′ ∥ ⋆ J_{\mathcal{X}}(\theta)-J_{\mathcal{X}}(\theta')\leq\langle\theta,x\rangle-\langle\theta',x\rangle\leq\|x\|_\star\|\theta-\theta'\|_\star\leq\ell_\star\|\theta-\theta'\|_\star J X ( θ ) − J X ( θ ′ ) ≤ ⟨ θ , x ⟩ − ⟨ θ ′ , x ⟩ ≤ ∥ x ∥ ⋆ ∥ θ − θ ′ ∥ ⋆ ≤ ℓ ⋆ ∥ θ − θ ′ ∥ ⋆ J X ( θ ′ ) ≥ ⟨ θ ′ , x ⟩ J_{\mathcal{X}}(\theta')\geq\langle\theta',x\rangle J X ( θ ′ ) ≥ ⟨ θ ′ , x ⟩ θ , θ ′ \theta,\theta' θ , θ ′ θ ′ = 0 \theta'=0 θ ′ = 0 J X ( θ ) ≤ ℓ ⋆ ∥ θ ∥ ⋆ J_{\mathcal{X}}(\theta)\leq\ell_\star\|\theta\|_\star J X ( θ ) ≤ ℓ ⋆ ∥ θ ∥ ⋆ v = θ ⋆ ≠ 0 v=\theta^\star\ne0 v = θ ⋆ = 0 J ⋆ ≥ m / 2 ∥ θ ⋆ ∥ ⋆ > 0 J_\star\geq\sqrt{m/2}\|\theta^\star\|_\star>0 J ⋆ ≥ m /2 ∥ θ ⋆ ∥ ⋆ > 0 x ∈ X x \in\mathcal{X} x ∈ X θ ′ \theta' θ ′ J X ( θ ′ ) ≥ ⟨ θ ′ , x ⟩ = J X ( θ ) + ⟨ θ ′ − θ , x ⟩ J_{\mathcal{X}}(\theta') \ge\langle\theta',x\rangle= J_{\mathcal{X}}(\theta) + \langle\theta' - \theta,x\rangle J X ( θ ′ ) ≥ ⟨ θ ′ , x ⟩ = J X ( θ ) + ⟨ θ ′ − θ , x ⟩ x x x J X J_{\mathcal{X}} J X θ \theta θ J X J_{\mathcal{X}} J X θ ≠ 0 \theta\ne0 θ = 0 { ∇ J X ( θ ) } \{\nabla J_{\mathcal{X}}(\theta)\} { ∇ J X ( θ )} [25 Thm. 25.1] ), so every maximiser equals ∇ J X ( θ ) ∈ X \nabla J_{\mathcal{X}}(\theta) \in\mathcal{X} ∇ J X ( θ ) ∈ X ∥ ∇ J X ( θ ) ∥ ∗ ∘ ≤ ℓ ∗ \|\nabla J_{\mathcal{X}}(\theta)\|_{\ast}^{\circ} \le\ell_{\ast} ∥∇ J X ( θ ) ∥ ∗ ∘ ≤ ℓ ∗ ∇ J X 2 = 2 J X ∇ J X \nabla J_{\mathcal{X}}^2 = 2J_{\mathcal{X}}\nabla J_{\mathcal{X}} ∇ J X 2 = 2 J X ∇ J X θ = θ ∗ \theta= \theta^{\ast} θ = θ ∗ J ∗ ∗ = ⟨ θ ∗ , x ∗ ⟩ ≤ ∥ x ∗ ∥ ∗ ∘ ∥ θ ∗ ∥ ∗ ∘ J_{\ast}^{\ast} = \langle\theta^{\ast},x^{\ast}\rangle\le\|x^{\ast}\|_{\ast}^{\circ}\|\theta^{\ast}\|_{\ast}^{\circ} J ∗ ∗ = ⟨ θ ∗ , x ∗ ⟩ ≤ ∥ x ∗ ∥ ∗ ∘ ∥ θ ∗ ∥ ∗ ∘ ϱ ∗ ∗ ≤ ∥ x ∗ ∥ ∗ ∘ ≤ ℓ ∗ \varrho_{\ast}^{\ast} \le\|x^{\ast}\|_{\ast}^{\circ} \le\ell_{\ast} ϱ ∗ ∗ ≤ ∥ x ∗ ∥ ∗ ∘ ≤ ℓ ∗
(iv) For a convex function that is differentiable on R d \mathbb{R}^{d} R d M M M ( R d , ∥ ⋅ ∥ ∗ ) (\mathbb{R}^{d},\|\cdot\|_{\ast}) ( R d , ∥ ⋅ ∥ ∗ ) ( R d , ∥ ⋅ ∥ ∗ ∘ ) (\mathbb{R}^{d},\|\cdot\|_{\ast}^{\circ}) ( R d , ∥ ⋅ ∥ ∗ ∘ ) [8 Thm. 5.8] ); apply this to J X 2 J_{\mathcal{X}}^2 J X 2
(v) Since V t ≻ 0 V_t \succ0 V t ≻ 0 C t ( α ) C_t(\alpha) C t ( α ) θ ≠ 0 \theta\ne0 θ = 0 J X ( θ ) > 0 J_{\mathcal{X}}(\theta) > 0 J X ( θ ) > 0
⟨ θ t , x t ⟩ = max θ ∈ C t ( α ) max x ∈ X ⟨ θ , x ⟩ = max θ ∈ C t ( α ) J X ( θ ) > 0 = J X ( 0 ) , \langle\theta_t,x_t\rangle= \max_{\theta\in C_t(\alpha)}\max_{x \in\mathcal{X}}\langle\theta,x\rangle= \max_{\theta\in C_t(\alpha)}J_{\mathcal{X}}(\theta) > 0 = J_{\mathcal{X}}(0), ⟨ θ t , x t ⟩ = θ ∈ C t ( α ) max x ∈ X max ⟨ θ , x ⟩ = θ ∈ C t ( α ) max J X ( θ ) > 0 = J X ( 0 ) , so θ t ≠ 0 \theta_t \ne0 θ t = 0 x t x_t x t ⟨ θ t , ⋅ ⟩ \langle\theta_t,\cdot\rangle ⟨ θ t , ⋅ ⟩ X \mathcal{X} X x t = ∇ J X ( θ t ) x_t = \nabla J_{\mathcal{X}}(\theta_t) x t = ∇ J X ( θ t )
Proposition D.2 (The width in the ∥ ⋅ ∥ ∗ \|\cdot\|_{\ast} ∥ ⋅ ∥ ∗ For every t t t ω t = max u ∈ B 2 d ( 1 ) ∥ V t − 1 / 2 u ∥ ∗ \omega_t = \max_{u \in\mathbb{B}_{2}^{d}(1)}\|V_t^{-1/2}u\|_{\ast} ω t = max u ∈ B 2 d ( 1 ) ∥ V t − 1/2 u ∥ ∗
(i) ω t = max { ∥ v ∥ ∗ : ∥ v ∥ V t ≤ 1 } = max ∥ g ∥ ∗ ∘ ≤ 1 ∥ g ∥ V t − 1 \omega_t = \max\{\|v\|_{\ast} : \|v\|_{V_t} \le1\} = \max_{\|g\|_{\ast}^{\circ} \le1}\|g\|_{V_t^{-1}} ω t = max { ∥ v ∥ ∗ : ∥ v ∥ V t ≤ 1 } = max ∥ g ∥ ∗ ∘ ≤ 1 ∥ g ∥ V t − 1 all maxima being attained;
(ii) for every r > 0 r > 0 r > 0 max { ∥ v ∥ ∗ : ∥ v ∥ V t ≤ r } = r ω t 1 / 2 \max\{\|v\|_{\ast} : \|v\|_{V_t} \le r\} = r\omega_t^{1/2} max { ∥ v ∥ ∗ : ∥ v ∥ V t ≤ r } = r ω t 1/2
max θ ∈ C t ( α ) ∥ θ − θ ^ t ∥ ∗ = α ω t 1 / 2 \max_{\theta\in C_t(\alpha)}\|\theta-\hat{\theta}_t\|_{\ast} = \alpha\omega_t^{1/2} θ ∈ C t ( α ) max ∥ θ − θ ^ t ∥ ∗ = α ω t 1/2 and, on E , \mathcal{E}, E ,
∥ θ ^ t − θ ∗ ∥ ∗ ≤ β ω t 1 / 2 ; \|\hat{\theta}_t-\theta^{\ast}\|_{\ast} \le\beta\omega_t^{1/2}; ∥ θ ^ t − θ ∗ ∥ ∗ ≤ β ω t 1/2 ; (iii) ∥ g ∥ V t − 1 ≤ ∥ g ∥ ∗ ∘ ω t 1 / 2 \|g\|_{V_t^{-1}} \le\|g\|_{\ast}^{\circ}\omega_t^{1/2} ∥ g ∥ V t − 1 ≤ ∥ g ∥ ∗ ∘ ω t 1/2 for every g ∈ R d g \in\mathbb{R}^{d} g ∈ R d x T V t − 1 x ≤ ℓ ∗ 2 ω t x^{\mathsf{T}}V_t^{-1}x \le\ell_{\ast}^{2}\omega_t x T V t − 1 x ≤ ℓ ∗ 2 ω t x ∈ X x \in\mathcal{X} x ∈ X
(iv) if ∥ ⋅ ∥ E \|\cdot\|_{E} ∥ ⋅ ∥ E R d \mathbb{R}^{d} R d κ ′ ∥ u ∥ E ≤ ∥ u ∥ ∗ ≤ κ ∥ u ∥ E \kappa'\|u\|_{E} \le\|u\|_{\ast} \le\kappa\|u\|_{E} κ ′ ∥ u ∥ E ≤ ∥ u ∥ ∗ ≤ κ ∥ u ∥ E λ min ( V t ) \lambda_{\min}(V_t) λ m i n ( V t ) V t V_t V t ∥ ⋅ ∥ E \|\cdot\|_{E} ∥ ⋅ ∥ E κ ′ 2 / λ min ( V t ) ≤ ω t ≤ κ 2 / λ min ( V t ) \kappa'^{2}/\lambda_{\min}(V_t) \le\omega_t \le\kappa^{2}/\lambda_{\min}(V_t) κ ′ 2 / λ m i n ( V t ) ≤ ω t ≤ κ 2 / λ m i n ( V t )
(v) t ↦ ω t t \mapsto\omega_t t ↦ ω t nonincreasing.
Proof of Proposition D.2. (i) Substituting v = V t − 1 / 2 u v = V_t^{-1/2}u v = V t − 1/2 u ω t \omega_t ω t ω t = max { ∥ v ∥ ∗ : ∥ v ∥ V t ≤ 1 } \omega_t = \max\{\|v\|_{\ast} : \|v\|_{V_t} \le1\} ω t = max { ∥ v ∥ ∗ : ∥ v ∥ V t ≤ 1 } ∥ v ∥ ∗ = sup ∥ g ∥ ∗ ∘ ≤ 1 ⟨ g , v ⟩ \|v\|_{\ast} = \sup_{\|g\|_{\ast}^{\circ} \le1}\langle g,v\rangle ∥ v ∥ ∗ = sup ∥ g ∥ ∗ ∘ ≤ 1 ⟨ g , v ⟩ sup ∥ v ∥ V t ≤ 1 ⟨ g , v ⟩ = ∥ g ∥ V t − 1 \sup_{\|v\|_{V_t}\le1}\langle g,v\rangle= \|g\|_{V_t^{-1}} sup ∥ v ∥ V t ≤ 1 ⟨ g , v ⟩ = ∥ g ∥ V t − 1 ∥ ⋅ ∥ V t \|\cdot\|_{V_t} ∥ ⋅ ∥ V t ∥ ⋅ ∥ V t − 1 \|\cdot\|_{V_t^{-1}} ∥ ⋅ ∥ V t − 1 max { ∥ v ∥ ∗ : ∥ v ∥ V t ≤ 1 } = max ∥ g ∥ ∗ ∘ ≤ 1 ∥ g ∥ V t − 1 \max\{\|v\|_{\ast} : \|v\|_{V_t} \le1\} = \max_{\|g\|_{\ast}^{\circ}\le1}\|g\|_{V_t^{-1}} max { ∥ v ∥ ∗ : ∥ v ∥ V t ≤ 1 } = max ∥ g ∥ ∗ ∘ ≤ 1 ∥ g ∥ V t − 1 { v : ∥ v ∥ V t ≤ r } = r { v : ∥ v ∥ V t ≤ 1 } \{v : \|v\|_{V_t} \le r\} = r\{v : \|v\|_{V_t} \le1\} { v : ∥ v ∥ V t ≤ r } = r { v : ∥ v ∥ V t ≤ 1 } ∥ ⋅ ∥ ∗ \|\cdot\|_{\ast} ∥ ⋅ ∥ ∗ r ω t 1 / 2 r\omega_t^{1/2} r ω t 1/2 v = θ − θ ^ t v = \theta-\hat{\theta}_t v = θ − θ ^ t r = α r = \alpha r = α v = θ ^ t − θ ∗ v = \hat{\theta}_t-\theta^{\ast} v = θ ^ t − θ ∗ r = β r = \beta r = β E \mathcal{E} E ∥ g ∥ V t − 1 = sup ∥ v ∥ V t ≤ 1 ⟨ g , v ⟩ ≤ sup ∥ v ∥ V t ≤ 1 ∥ g ∥ ∗ ∘ ∥ v ∥ ∗ = ∥ g ∥ ∗ ∘ ω t 1 / 2 \|g\|_{V_t^{-1}} = \sup_{\|v\|_{V_t}\le1}\langle g,v\rangle\le\sup_{\|v\|_{V_t}\le1}\|g\|_{\ast}^{\circ}\|v\|_{\ast} = \|g\|_{\ast}^{\circ}\omega_t^{1/2} ∥ g ∥ V t − 1 = sup ∥ v ∥ V t ≤ 1 ⟨ g , v ⟩ ≤ sup ∥ v ∥ V t ≤ 1 ∥ g ∥ ∗ ∘ ∥ v ∥ ∗ = ∥ g ∥ ∗ ∘ ω t 1/2 ∥ x ∥ ∗ ≤ ℓ ∗ \|x\|_{\ast} \le\ell_{\ast} ∥ x ∥ ∗ ≤ ℓ ∗ x ∈ X x \in\mathcal{X} x ∈ X ∥ ⋅ ∥ E \|\cdot\|_{E} ∥ ⋅ ∥ E ω t = sup ∥ u ∥ E ≤ 1 ∥ V t − 1 / 2 u ∥ ∗ \omega_t = \sup_{\|u\|_{E}\le1}\|V_t^{-1/2}u\|_{\ast} ω t = sup ∥ u ∥ E ≤ 1 ∥ V t − 1/2 u ∥ ∗
κ ′ 2 ∥ V t − 1 / 2 u ∥ E 2 ≤ ∥ V t − 1 / 2 u ∥ ∗ 2 ≤ κ 2 ∥ V t − 1 / 2 u ∥ E 2 \kappa'^{2}\|V_t^{-1/2}u\|_{E}^{2} \le\|V_t^{-1/2}u\|_{\ast}^{2} \le\kappa^{2}\|V_t^{-1/2}u\|_{E}^{2} κ ′ 2 ∥ V t − 1/2 u ∥ E 2 ≤ ∥ V t − 1/2 u ∥ ∗ 2 ≤ κ 2 ∥ V t − 1/2 u ∥ E 2 and
sup ∥ u ∥ E ≤ 1 ∥ V t − 1 / 2 u ∥ E 2 = 1 λ min ( V t ) . \sup_{\|u\|_{E}\le1}\|V_t^{-1/2}u\|_{E}^{2} = \frac{1}{\lambda_{\min}(V_t)}. ∥ u ∥ E ≤ 1 sup ∥ V t − 1/2 u ∥ E 2 = λ m i n ( V t ) 1 . (v) For each fixed g g g g T V t − 1 g g^{\mathsf{T}}V_t^{-1}g g T V t − 1 g t t t V t + 1 − 1 ⪯ V t − 1 V_{t+1}^{-1} \preceq V_t^{-1} V t + 1 − 1 ⪯ V t − 1 ∥ g ∥ ∗ ∘ ≤ 1 \|g\|_{\ast}^{\circ} \le1 ∥ g ∥ ∗ ∘ ≤ 1
Proposition D.3 (Lipschitz argmax). Under Assumption (A2), for all θ , θ ′ ∈ N ∗ \theta,\theta' \in\mathcal{N}_{\ast} θ , θ ′ ∈ N ∗
∥ ∇ J X ( θ ) − ∇ J X ( θ ′ ) ∥ ∗ ∘ ≤ L ∗ ∥ θ − θ ′ ∥ ∗ , L ∗ = M + 2 ℓ ∗ 2 J ∗ . (9) \|\nabla J_{\mathcal{X}}(\theta)-\nabla J_{\mathcal{X}}(\theta')\|_{\ast}^{\circ} \le L_{\ast}\|\theta-\theta'\|_{\ast}, \qquad L_{\ast} = \frac{M+2\ell_{\ast}^{2}}{J_{\ast}}. \tag*{(9)} ∥∇ J X ( θ ) − ∇ J X ( θ ′ ) ∥ ∗ ∘ ≤ L ∗ ∥ θ − θ ′ ∥ ∗ , L ∗ = J ∗ M + 2 ℓ ∗ 2 . (9) Lipschitz continuity of the argmax map is the condition under which Rusmevichientong & Tsitsiklis [26 ] call an arm set strongly convex; here it is derived, in the geometry of the assumption, from the smoothness of J X 2 J_{\mathcal{X}}^{2} J X 2 J X ≥ J ∗ / 2 J_{\mathcal{X}} \ge J_{\ast}/2 J X ≥ J ∗ /2 N ∗ \mathcal{N}_{\ast} N ∗
Proof of Proposition D.3. Since 0 ∉ N ∗ 0 \notin\mathcal{N}_* 0 ∈ / N ∗ ∇ J X = ∇ J X 2 / ( 2 J X ) \nabla J_{\mathcal{X}}=\nabla J_{\mathcal{X}}^2/(2J_{\mathcal{X}}) ∇ J X = ∇ J X 2 / ( 2 J X ) N ∗ \mathcal{N}_* N ∗
∥ ∇ J X ( θ ) − ∇ J X ( θ ′ ) ∥ ∗ ≤ ∥ ∇ J X 2 ( θ ) − ∇ J X 2 ( θ ′ ) ∥ ∗ 2 J X ( θ ) + ∥ ∇ J X 2 ( θ ′ ) ∥ ∗ ∣ J X ( θ ′ ) − J X ( θ ) ∣ 2 J X ( θ ) J X ( θ ′ ) ≤ M ∥ θ − θ ′ ∥ ∗ J ⋆ + 2 ℓ ⋆ 2 ∥ θ − θ ′ ∥ ∗ J ⋆ . \begin{aligned}
\lVert\nabla J_{\mathcal{X}}(\theta)-\nabla J_{\mathcal{X}}(\theta')\rVert_* \leq\frac{\lVert\nabla J_{\mathcal{X}}^2(\theta)-\nabla J_{\mathcal{X}}^2(\theta')\rVert_*}{2J_{\mathcal{X}}(\theta)}+\frac{\lVert\nabla J_{\mathcal{X}}^2(\theta')\rVert_* \lvert J_{\mathcal{X}}(\theta')-J_{\mathcal{X}}(\theta)\rvert}{2J_{\mathcal{X}}(\theta)J_{\mathcal{X}}(\theta')} \\
&\leq\frac{M\lVert\theta-\theta'\rVert_*}{J_\star}+\frac{2\ell_\star^2\lVert\theta-\theta'\rVert_*}{J_\star}.
\end{aligned} ∥ ∇ J X ( θ ) − ∇ J X ( θ ′ ) ∥ ∗ ≤ 2 J X ( θ ) ∥ ∇ J X 2 ( θ ) − ∇ J X 2 ( θ ′ ) ∥ ∗ + 2 J X ( θ ) J X ( θ ′ ) ∥ ∇ J X 2 ( θ ′ ) ∥ ∗ ∣ J X ( θ ′ ) − J X ( θ )∣ ≤ J ⋆ M ∥ θ − θ ′ ∥ ∗ + J ⋆ 2 ℓ ⋆ 2 ∥ θ − θ ′ ∥ ∗ . where we used equation (8) , J X ( θ ) ≥ J ⋆ / 2 J_{\mathcal{X}}(\theta)\geq J_\star/2 J X ( θ ) ≥ J ⋆ /2 (7) , ∥ ∇ J X 2 ( θ ′ ) ∥ ∗ = 2 J X ( θ ′ ) ∥ ∇ J X ( θ ′ ) ∥ ∗ ≤ 2 ℓ ⋆ J X ( θ ′ ) \lVert\nabla J_{\mathcal{X}}^2(\theta')\rVert_*=2J_{\mathcal{X}}(\theta')\lVert\nabla J_{\mathcal{X}}(\theta')\rVert_*\leq2\ell_\star J_{\mathcal{X}}(\theta') ∥ ∇ J X 2 ( θ ′ ) ∥ ∗ = 2 J X ( θ ′ ) ∥ ∇ J X ( θ ′ ) ∥ ∗ ≤ 2 ℓ ⋆ J X ( θ ′ ) ℓ ⋆ \ell_\star ℓ ⋆ J X J_{\mathcal{X}} J X
Geometric distortion and adapted coordinates
Adapted Euclidean structure. The proofs perform spectral bookkeeping (eigenvalues and eigenvectors of V t V_t V t α \alpha α ω t \omega_t ω t ∥ ⋅ ∥ ∗ \lVert\cdot\rVert_* ∥ ⋅ ∥ ∗ D ∗ ≥ 1 D_* \geq1 D ∗ ≥ 1 admissible distortion if there is P ≻ 0 P \succ0 P ≻ 0
∥ u ∥ ∗ ≤ ∥ P u ∥ 2 ≤ D ∗ ∥ u ∥ ∗ ( u ∈ R d ) , (10) \lVert u\rVert_* \leq\lVert Pu\rVert_2 \leq D_*\lVert u\rVert_* \qquad(u\in\mathbb{R}^d), \tag*{(10)} ∥ u ∥ ∗ ≤ ∥ P u ∥ 2 ≤ D ∗ ∥ u ∥ ∗ ( u ∈ R d ) , (10) that is, the Euclidean unit ball E E E u ↦ ∥ P u ∥ 2 u\mapsto\lVert Pu\rVert_2 u ↦ ∥ P u ∥ 2 D ∗ − 1 B ∗ ⊆ E ⊆ B ∗ D_*^{-1}B_*\subseteq E\subseteq B_* D ∗ − 1 B ∗ ⊆ E ⊆ B ∗ B ∗ B_* B ∗ D ∗ = d D_*=\sqrt{d} D ∗ = d D ∗ = M / m D_*=\sqrt{M/m} D ∗ = M / m Lemma D.4 , from the Hessian of J X 2 J_{\mathcal{X}}^2 J X 2 B ∗ B_* B ∗ D ∗ = 1 D_*=1 D ∗ = 1 ( D ∗ , P ) (D_*,P) ( D ∗ , P )
D ∗ ≤ min { d , M / m } , (11) D_* \leq\min\{\sqrt{d},\sqrt{M/m}\}, \tag*{(11)} D ∗ ≤ min { d , M / m } , (11) and use it only inside the proofs (Lemma D.5 ). The distortion D ∗ D_* D ∗ T ⋆ T_\star T ⋆ log ( 8 Γ ) \log(8\Gamma) log ( 8Γ ) 4 log ( D ∗ ) ≤ 2 log ( d ) 4\log(D_*)\leq2\log(d) 4 log ( D ∗ ) ≤ 2 log ( d )
Figure 5. The adapted Euclidean structure of Section D.2. (a) The unit ball B ∗ B_* B ∗ ∥ ⋅ ∥ ∗ \lVert\cdot\rVert_* ∥ ⋅ ∥ ∗ E E E u ↦ ∥ P u ∥ 2 u\mapsto\lVert Pu\rVert_2 u ↦ ∥ P u ∥ 2 D ∗ − 1 B ∗ D_*^{-1}B_* D ∗ − 1 B ∗ u ↦ P u u\mapsto Pu u ↦ P u E E E B ∗ B_* B ∗ D ∗ D_* D ∗ (10) , where all spectral bookkeeping takes place.
Lemma D.4 (Ellipsoidal distortion of B ∗ B_* B ∗ Under Assumption (A2), there is a symmetric matrix H ≻ 0 H\succ0 H ≻ 0
m ∥ u ∥ ∗ 2 ≤ u ⊤ H u ≤ M ∥ u ∥ ∗ 2 ( u ∈ R d ) . m\lVert u\rVert_*^2\leq u^\top Hu\leq M\lVert u\rVert_*^2 \qquad(u\in\mathbb{R}^d). m ∥ u ∥ ∗ 2 ≤ u ⊤ H u ≤ M ∥ u ∥ ∗ 2 ( u ∈ R d ) . Consequently P : = ( H / m ) 1 / 2 P:=(H/m)^{1/2} P := ( H / m ) 1/2 (10) with D ∗ = M / m D_*=\sqrt{M/m} D ∗ = M / m B ∗ B_* B ∗ d \sqrt{d} d min { d , M / m } \min\{\sqrt{d},\sqrt{M/m}\} min { d , M / m }
Proof of Lemma D.4. J X 2 J_X^2 J X 2 R d \mathbb{R}^d R d [5 ] (see also Rockafellar & Wets, 1998, Thm. 13.51, or Evans & Gariepy, 2015, Thm. 6.9), it is twice differentiable at almost every θ 0 ∈ R d \theta_0 \in\mathbb{R}^d θ 0 ∈ R d H \mathrm{H} H
D J X 2 ( θ 0 + v , θ 0 ) = J X 2 ( θ 0 + v ) − J X 2 ( θ 0 ) − ⟨ ∇ J X 2 ( θ 0 ) , v ⟩ = 1 2 v ⊤ H v + o ( ∥ v ∥ 2 ) . D_{J_X^2}(\theta_0+v,\theta_0)=J_X^2(\theta_0+v)-J_X^2(\theta_0)-\langle\nabla J_X^2(\theta_0),v\rangle=\frac{1}{2}v^{\top}\mathrm{H}v+o(\lVert v\rVert^2). D J X 2 ( θ 0 + v , θ 0 ) = J X 2 ( θ 0 + v ) − J X 2 ( θ 0 ) − ⟨ ∇ J X 2 ( θ 0 ) , v ⟩ = 2 1 v ⊤ H v + o (∥ v ∥ 2 ) . Fix such a θ 0 \theta_0 θ 0 u ∈ R d u\in\mathbb{R}^d u ∈ R d ε > 0 \varepsilon>0 ε > 0 θ = θ 0 + ε u \theta=\theta_0+\varepsilon u θ = θ 0 + ε u θ ′ = θ 0 \theta'=\theta_0 θ ′ = θ 0
m 2 ε 2 ∥ u ∥ ∗ 2 ≤ ε 2 2 u ⊤ H u + o ( ε 2 ) ≤ M 2 ε 2 ∥ u ∥ ∗ 2 . \frac{m}{2}\varepsilon^2\lVert u\rVert_*^2\leq\frac{\varepsilon^2}{2}u^{\top}\mathrm{H}u+o(\varepsilon^2)\leq\frac{M}{2}\varepsilon^2\lVert u\rVert_*^2. 2 m ε 2 ∥ u ∥ ∗ 2 ≤ 2 ε 2 u ⊤ H u + o ( ε 2 ) ≤ 2 M ε 2 ∥ u ∥ ∗ 2 . Dividing by ε 2 / 2 \varepsilon^2/2 ε 2 /2 ε → 0 \varepsilon\to0 ε → 0 m ∥ u ∥ ∗ 2 ≤ u ⊤ H u ≤ M ∥ u ∥ ∗ 2 m\lVert u\rVert_*^2\leq u^{\top}\mathrm{H}u\leq M\lVert u\rVert_*^2 m ∥ u ∥ ∗ 2 ≤ u ⊤ H u ≤ M ∥ u ∥ ∗ 2 H ≻ 0 \mathrm{H}\succ0 H ≻ 0 P = ( H / m ) 1 / 2 \mathrm{P}=(\mathrm{H}/m)^{1/2} P = ( H / m ) 1/2 ∥ P u ∥ 2 2 = u ⊤ H u / m ∈ [ ∥ u ∥ ∗ 2 , ( M / m ) ∥ u ∥ ∗ 2 ] \lVert\mathrm{P}u\rVert_2^2=u^{\top}\mathrm{H}u/m\in[\lVert u\rVert_*^2,(M/m)\lVert u\rVert_*^2] ∥ P u ∥ 2 2 = u ⊤ H u / m ∈ [∥ u ∥ ∗ 2 , ( M / m ) ∥ u ∥ ∗ 2 ] D ∗ = M / m D_*=\sqrt{M/m} D ∗ = M / m
Lemma D.5 (Adapted coordinates). Let ( D ∗ , P ) (D_*,\mathrm{P}) ( D ∗ , P )
x ~ : = P − 1 x ℓ ∗ , θ ~ : = ℓ ∗ P θ , so that ⟨ θ ~ , x ~ ⟩ = ⟨ θ , x ⟩ , \tilde{x}:=\frac{\mathrm{P}^{-1}x}{\ell_*},\qquad\tilde{\theta}:=\ell_*\mathrm{P}\theta,\qquad\text{so that}\qquad\langle\tilde{\theta},\tilde{x}\rangle=\langle\theta,x\rangle, x ~ := ℓ ∗ P − 1 x , θ ~ := ℓ ∗ P θ , so that ⟨ θ ~ , x ~ ⟩ = ⟨ θ , x ⟩ , and let X ~ : = { x ~ : x ∈ X } \widetilde{\mathcal{X}}:=\{\tilde{x}:x\in\mathcal{X}\} X := { x ~ : x ∈ X } θ ~ ∗ : = ℓ ∗ P θ ∗ \tilde{\theta}^*:=\ell_*\mathrm{P}\theta^* θ ~ ∗ := ℓ ∗ P θ ∗ ∥ ⋅ ∥ ∗ \lVert\cdot\rVert_* ∥ ⋅ ∥ ∗ ∥ ⋅ ∥ ∗ ∘ \lVert\cdot\rVert_*^\circ ∥ ⋅ ∥ ∗ ∘ θ ~ \tilde{\theta} θ ~ ∥ θ ∥ ∗ = ∥ P − 1 θ ~ ∥ ∗ / ℓ ∗ \lVert\theta\rVert_*=\lVert\mathrm{P}^{-1}\tilde{\theta}\rVert_*/\ell_* ∥ θ ∥ ∗ = ∥ P − 1 θ ~ ∥ ∗ / ℓ ∗ x ~ \tilde{x} x ~ ∥ x ∥ ∗ ∘ = ∥ ℓ ∗ P x ~ ∥ ∗ ∘ \lVert x\rVert_*^\circ=\lVert\ell_*\mathrm{P}\tilde{x}\rVert_*^\circ ∥ x ∥ ∗ ∘ = ∥ ℓ ∗ P x ~ ∥ ∗ ∘
(i) ∥ x ∥ ∗ ∘ / ( D ∗ ℓ ∗ ) ≤ ∥ x ~ ∥ 2 ≤ ∥ x ∥ ∗ ∘ / ℓ ∗ \lVert x\rVert_*^\circ/(D_*\ell_*)\leq\lVert\tilde{x}\rVert_2\leq\lVert x\rVert_*^\circ/\ell_* ∥ x ∥ ∗ ∘ / ( D ∗ ℓ ∗ ) ≤ ∥ x ~ ∥ 2 ≤ ∥ x ∥ ∗ ∘ / ℓ ∗ x x x X ~ ⊆ B 2 d ( 1 ) \widetilde{\mathcal{X}}\subseteq\mathbb{B}_2^d(1) X ⊆ B 2 d ( 1 ) ∥ x ~ ∗ ∥ 2 ≥ ϱ ∗ / ( D ∗ ℓ ∗ ) \lVert\tilde{x}^*\rVert_2\geq\varrho_*/(D_*\ell_*) ∥ x ~ ∗ ∥ 2 ≥ ϱ ∗ / ( D ∗ ℓ ∗ )
(ii) ∥ θ ~ ∥ 2 / ( D ∗ ℓ ∗ ) ≤ ∥ θ ∥ ∗ ≤ ∥ θ ~ ∥ 2 / ℓ ∗ \lVert\tilde{\theta}\rVert_2/(D_*\ell_*)\leq\lVert\theta\rVert_*\leq\lVert\tilde{\theta}\rVert_2/\ell_* ∥ θ ~ ∥ 2 / ( D ∗ ℓ ∗ ) ≤ ∥ θ ∥ ∗ ≤ ∥ θ ~ ∥ 2 / ℓ ∗ θ \theta θ
(iii) Every intrinsic quantity is unchanged: J X ~ ( θ ~ ) = J X ( θ ) J_{\widetilde{\mathcal{X}}}(\tilde{\theta})=J_{\mathcal{X}}(\theta) J X ( θ ~ ) = J X ( θ ) ∇ J X ~ ( θ ~ ) = P − 1 ∇ J X ( θ ) / ℓ ∗ \nabla J_{\widetilde{\mathcal{X}}}(\tilde{\theta})=\mathrm{P}^{-1}\nabla J_{\mathcal{X}}(\theta)/\ell_* ∇ J X ( θ ~ ) = P − 1 ∇ J X ( θ ) / ℓ ∗ ∇ J X ( θ ) \nabla J_{\mathcal{X}}(\theta) ∇ J X ( θ ) J X 2 J_{\mathcal{X}}^2 J X 2 ∥ θ − θ ′ ∥ ∗ \lVert\theta-\theta'\rVert_* ∥ θ − θ ′ ∥ ∗ ∥ x − x ′ ∥ ∗ ∘ \lVert x-x'\rVert_*^\circ ∥ x − x ′ ∥ ∗ ∘ ℓ ∗ \ell_* ℓ ∗ ϱ ∗ \varrho_* ϱ ∗ J ∗ J_* J ∗ N ∗ \mathcal{N}_* N ∗ ℓ ∗ P N ∗ \ell_*\mathrm{P}\mathcal{N}_* ℓ ∗ P N ∗ ( X ~ , θ ~ ∗ ) (\widetilde{\mathcal{X}},\tilde{\theta}^*) ( X , θ ~ ∗ ) m , M m,M m , M α \alpha α
V ~ t : = ℓ ∗ − 2 P − 1 V t P − 1 = R ~ + ∑ s < t x ~ s x ~ s ⊤ , R ~ : = λ 0 ℓ ∗ − 2 P − 2 , \widetilde{\mathrm{V}}_t:=\ell_*^{-2}\mathrm{P}^{-1}\mathrm{V}_t\mathrm{P}^{-1}=\widetilde{\mathrm{R}}+\sum_{s<t}\tilde{x}_s\tilde{x}_s^{\top},\qquad\widetilde{\mathrm{R}}:=\lambda_0\ell_*^{-2}\mathrm{P}^{-2}, V t := ℓ ∗ − 2 P − 1 V t P − 1 = R + s < t ∑ x ~ s x ~ s ⊤ , R := λ 0 ℓ ∗ − 2 P − 2 , the transported design matrix (the ridge penalty λ 0 ∥ θ ∥ 2 2 = θ ~ ⊤ R ~ θ ~ \lambda_0\lVert\theta\rVert_2^2=\tilde{\theta}^{\top}\widetilde{\mathrm{R}}\tilde{\theta} λ 0 ∥ θ ∥ 2 2 = θ ~ ⊤ R θ ~ θ ^ t = V t − 1 ∑ s < t x s y s \hat{\theta}_t=\mathrm{V}_t^{-1}\sum_{s<t}x_sy_s θ ^ t = V t − 1 ∑ s < t x s y s
V ~ t − 1 ∑ s < t x ~ s y s = ℓ ∗ P θ ^ t , ∥ θ ~ − θ ~ ′ ∥ V ~ t = ∥ θ − θ ′ ∥ V t , ∥ g ~ ∥ V ~ t − 1 = ∥ g ∥ V t − 1 , \widetilde{\mathrm{V}}_t^{-1}\sum_{s<t}\tilde{x}_sy_s=\ell_*\mathrm{P}\hat{\theta}_t,\qquad\lVert\tilde{\theta}-\tilde{\theta}'\rVert_{\widetilde{\mathrm{V}}_t}=\lVert\theta-\theta'\rVert_{\mathrm{V}_t},\qquad\lVert\tilde{g}\rVert_{\widetilde{\mathrm{V}}_t^{-1}}=\lVert g\rVert_{\mathrm{V}_t^{-1}}, V t − 1 s < t ∑ x ~ s y s = ℓ ∗ P θ ^ t , ∥ θ ~ − θ ~ ′ ∥ V t = ∥ θ − θ ′ ∥ V t , ∥ g ~ ∥ V t − 1 = ∥ g ∥ V t − 1 , so the confidence ellipsoid, the optimistic pair and the event E \mathcal{E} E ( x ~ t , θ ~ t ) t (\tilde{x}_t,\tilde{\theta}_t)_t ( x ~ t , θ ~ t ) t α \alpha α ( X ~ , θ ~ ∗ ) (\widetilde{\mathcal{X}},\tilde{\theta}^*) ( X , θ ~ ∗ ) ω ~ t = ω t \tilde{\omega}_t=\omega_t ω ~ t = ω t
(iv) In the new coordinates the smallest eigenvalue satisfies λ ~ t , d ≥ ω t − 1 / ( D ∗ 2 ℓ ∗ 2 ) \tilde{\lambda}_{t,d}\geq\omega_t^{-1}/(D_*^2\ell_*^2) λ ~ t , d ≥ ω t − 1 / ( D ∗ 2 ℓ ∗ 2 ) x ⊤ V t − 1 x = x ~ ⊤ V ~ t − 1 x ~ ≤ min { 1 / λ ~ t , d , ℓ ∗ 2 ω t } x^{\top}\mathrm{V}_t^{-1}x=\tilde{x}^{\top}\widetilde{\mathrm{V}}_t^{-1}\tilde{x}\leq\min\{1/\tilde{\lambda}_{t,d},\ell_*^2\omega_t\} x ⊤ V t − 1 x = x ~ ⊤ V t − 1 x ~ ≤ min { 1/ λ ~ t , d , ℓ ∗ 2 ω t } x ∈ X x\in\mathcal{X} x ∈ X ω t − 1 ≥ 2 D ∗ 2 ℓ ∗ 2 \omega_t^{-1}\geq2D_*^2\ell_*^2 ω t − 1 ≥ 2 D ∗ 2 ℓ ∗ 2 λ ~ t , d ≥ 2 \tilde{\lambda}_{t,d}\geq2 λ ~ t , d ≥ 2 x ⊤ V t − 1 x ≤ 1 / 2 x^{\top}\mathrm{V}_t^{-1}x\leq1/2 x ⊤ V t − 1 x ≤ 1/2
Proof of Lemma D.5. (i) ∥ P − 1 x ∥ 2 = sup ∥ v ∥ 2 ≤ 1 ⟨ x , P − 1 v ⟩ = sup ∥ P u ∥ 2 ≤ 1 ⟨ x , u ⟩ \lVert\mathrm{P}^{-1}x\rVert_2=\sup_{\lVert v\rVert_2\leq1}\langle x,\mathrm{P}^{-1}v\rangle=\sup_{\lVert\mathrm{P}u\rVert_2\leq1}\langle x,u\rangle ∥ P − 1 x ∥ 2 = sup ∥ v ∥ 2 ≤ 1 ⟨ x , P − 1 v ⟩ = sup ∥ P u ∥ 2 ≤ 1 ⟨ x , u ⟩ { u : ∥ u ∥ ∗ ≤ 1 / D ∗ } \{u:\lVert u\rVert_*\leq1/D_*\} { u : ∥ u ∥ ∗ ≤ 1/ D ∗ } { u : ∥ u ∥ ∗ ≤ 1 } \{u:\lVert u\rVert_*\leq1\} { u : ∥ u ∥ ∗ ≤ 1 } ∥ x ∥ ∗ ∘ / D ∗ ≤ ∥ P − 1 x ∥ 2 ≤ ∥ x ∥ ∗ ∘ \lVert x\rVert_*^\circ/D_*\leq\lVert\mathrm{P}^{-1}x\rVert_2\leq\lVert x\rVert_*^\circ ∥ x ∥ ∗ ∘ / D ∗ ≤ ∥ P − 1 x ∥ 2 ≤ ∥ x ∥ ∗ ∘ ℓ ∗ \ell_* ℓ ∗ ∥ x ∗ ∥ 2 ≥ ϱ ∗ \lVert x^*\rVert_2\geq\varrho_* ∥ x ∗ ∥ 2 ≥ ϱ ∗ ℓ ∗ \ell_* ℓ ∗ ∥ θ ~ ∥ 2 = ℓ ∗ ∥ P θ ∥ 2 \lVert\tilde{\theta}\rVert_2=\ell_*\lVert\mathrm{P}\theta\rVert_2 ∥ θ ~ ∥ 2 = ℓ ∗ ∥ P θ ∥ 2 J X ~ ( θ ~ ) = sup x ~ ∈ X ~ ⟨ θ ~ , x ~ ⟩ = J X ( θ ) J_{\widetilde{\mathcal{X}}}(\tilde{\theta})=\sup_{\tilde{x}\in\widetilde{\mathcal{X}}}\langle\tilde{\theta},\tilde{x}\rangle=J_{\mathcal{X}}(\theta) J X ( θ ~ ) = sup x ~ ∈ X ⟨ θ ~ , x ~ ⟩ = J X ( θ ) ∇ θ ~ J X ~ = ℓ ∗ − 1 P − 1 ∇ θ J X \nabla_{\tilde{\theta}}J_{\widetilde{\mathcal{X}}}=\ell_*^{-1}\mathrm{P}^{-1}\nabla_{\theta}J_{\mathcal{X}} ∇ θ ~ J X = ℓ ∗ − 1 P − 1 ∇ θ J X ⟨ ∇ J X ~ 2 ( θ ~ ) , θ ~ − θ ~ ′ ⟩ = ⟨ ∇ J X 2 ( θ ′ ) , θ − θ ′ ⟩ \langle\nabla J_{\widetilde{\mathcal{X}}}^2(\tilde{\theta}),\tilde{\theta}-\tilde{\theta}'\rangle=\langle\nabla J_{\mathcal{X}}^2(\theta'),\theta-\theta'\rangle ⟨ ∇ J X 2 ( θ ~ ) , θ ~ − θ ~ ′ ⟩ = ⟨ ∇ J X 2 ( θ ′ ) , θ − θ ′ ⟩ ∥ ⋅ ∥ ∗ \lVert\cdot\rVert_* ∥ ⋅ ∥ ∗ ∥ ⋅ ∥ ∗ ∘ \lVert\cdot\rVert_*^\circ ∥ ⋅ ∥ ∗ ∘ ℓ ∗ = sup x ∥ x ∥ ∗ ∘ \ell_*=\sup_{x}\lVert x\rVert_*^\circ ℓ ∗ = sup x ∥ x ∥ ∗ ∘ ϱ ∗ = J ∗ ∥ θ ∗ ∥ ∗ \varrho_*=\frac{J_*}{\lVert\theta^*\rVert_*} ϱ ∗ = ∥ θ ∗ ∥ ∗ J ∗ J ∗ J_* J ∗ N ∗ \mathcal{N}_* N ∗ (1) and equation (8) hold verbatim for the transported problem. For the algorithm:
V ~ t = ℓ ∗ − 2 P − 1 ( λ 0 I + ∑ s < t x s x s ⊤ ) P − 1 = R ~ + ∑ s < t x ~ s x ~ s ⊤ , so V ~ t − 1 = ℓ ∗ 2 P V t − 1 P \widetilde{V}_t=\ell_*^{-2}P^{-1}\left(\lambda_0 I+\sum_{s<t}x_sx_s^\top\right)P^{-1}=\widetilde{R}+\sum_{s<t}\widetilde{x}_s\widetilde{x}_s^\top,\quad\text{so }\widetilde{V}_t^{-1}=\ell_*^2PV_t^{-1}P V t = ℓ ∗ − 2 P − 1 ( λ 0 I + s < t ∑ x s x s ⊤ ) P − 1 = R + s < t ∑ x s x s ⊤ , so V t − 1 = ℓ ∗ 2 P V t − 1 P and
V ~ t − 1 ∑ s < t x ~ s y s = ℓ ∗ P V t − 1 ∑ s < t x s y s = ℓ ∗ P θ ^ t ; \widetilde{V}_t^{-1}\sum_{s<t}\widetilde{x}_s y_s=\ell_*PV_t^{-1}\sum_{s<t}x_sy_s=\ell_*P\widehat{\theta}_t; V t − 1 s < t ∑ x s y s = ℓ ∗ P V t − 1 s < t ∑ x s y s = ℓ ∗ P θ t ; the two norm identities follow by direct substitution. Hence C ~ t : = { θ ~ : θ ∈ C t ( α ) } \widetilde{C}_t:=\{\widetilde{\theta}:\theta\in C_t(\alpha)\} C t := { θ : θ ∈ C t ( α )} X ~ × C ~ t \widetilde{\mathcal{X}}\times\widetilde{C}_t X × C t X × C t ( α ) \mathcal{X}\times C_t(\alpha) X × C t ( α ) θ ≠ 0 \theta\ne0 θ = 0 θ t \theta_t θ t E \mathcal{E} E ω ~ t = max { ∥ θ ∥ 2 2 : ∥ θ ~ ∥ V ~ t ≤ 1 } = max { ∥ θ ∥ 2 2 : ∥ θ ∥ V t ≤ 1 } = ω t \widetilde{\omega}_t=\max\{\lVert\theta\rVert_2^2:\lVert\widetilde{\theta}\rVert_{\widetilde{V}_t}\le1\}=\max\{\lVert\theta\rVert_2^2:\lVert\theta\rVert_{V_t}\le1\}=\omega_t ω t = max {∥ θ ∥ 2 2 : ∥ θ ∥ V t ≤ 1 } = max {∥ θ ∥ 2 2 : ∥ θ ∥ V t ≤ 1 } = ω t Proposition D.2 (i) and the second identity. (iv) By (ii) and Proposition D.2 (iv) with κ ′ = 1 / ( D ∗ ℓ ∗ ) \kappa'=1/(D_*\ell_*) κ ′ = 1/ ( D ∗ ℓ ∗ ) λ ~ t , d ≥ ω t − 1 / ( D ∗ 2 ℓ ∗ 2 ) \widetilde{\lambda}_{t,d}\ge\omega_t^{-1}/(D_*^2\ell_*^2) λ t , d ≥ ω t − 1 / ( D ∗ 2 ℓ ∗ 2 ) Proposition D.2 (iii) together with ∥ x ~ ∥ 2 ≤ 1 \lVert\widetilde{x}\rVert_2\le1 ∥ x ∥ 2 ≤ 1
Convention. Throughout the proofs we keep the tildes of Lemma D.5 : x ~ \widetilde{x} x θ ~ \widetilde{\theta} θ V ~ t \widetilde{V}_t V t x x x θ \theta θ V t V_t V t ∥ ⋅ ∥ 2 \lVert\cdot\rVert_2 ∥ ⋅ ∥ 2 ∥ ⋅ ∥ ∗ \lVert\cdot\rVert_* ∥ ⋅ ∥ ∗ ∥ ⋅ ∥ ∘ \lVert\cdot\rVert_\circ ∥ ⋅ ∥ ∘ ∥ x ~ ∥ 2 ≤ ∥ x ∥ ∗ / ℓ ∗ \lVert\widetilde{x}\rVert_2\le\lVert x\rVert_*/\ell_* ∥ x ∥ 2 ≤ ∥ x ∥ ∗ / ℓ ∗ Lemma D.5 (i), (ii) and (iv) read: X ~ ⊆ B ~ 2 d ( 1 ) \widetilde{\mathcal{X}}\subseteq\widetilde{B}_2^d(1) X ⊆ B 2 d ( 1 ) ∥ x ~ s ∥ 2 ≤ 1 \lVert\widetilde{x}_s\rVert_2\le1 ∥ x s ∥ 2 ≤ 1 tr V ~ t ≤ t − 1 + r λ 0 \operatorname{tr}\widetilde{V}_t\le t-1+r_{\lambda_0} tr V t ≤ t − 1 + r λ 0
r λ 0 : = tr R ~ = λ 0 ℓ ∗ − 2 tr P − 2 ≤ λ 0 d ℓ ∗ 2 κ ′ 2 , κ ′ : = min ∥ u ∥ 2 = 1 ∥ u ∥ ∗ , (12) r_{\lambda_0}:=\operatorname{tr}\widetilde{R}=\lambda_0\ell_*^{-2}\operatorname{tr}P^{-2}\le\frac{\lambda_0d}{\ell_*^2\kappa'^2},\qquad\kappa':=\min_{\lVert u\rVert_2=1}\lVert u\rVert_*, \tag*{(12)} r λ 0 := tr R = λ 0 ℓ ∗ − 2 tr P − 2 ≤ ℓ ∗ 2 κ ′2 λ 0 d , κ ′ := ∥ u ∥ 2 = 1 min ∥ u ∥ ∗ , (12) is the trace of the transported regulariser (by equation (10) , λ min ( P ) ≥ κ ′ \lambda_{\min}(P)\ge\kappa' λ m i n ( P ) ≥ κ ′ r λ 0 = λ 0 d r_{\lambda_0}=\lambda_0d r λ 0 = λ 0 d r λ 0 = λ 0 tr A − 1 r_{\lambda_0}=\lambda_0\operatorname{tr}A^{-1} r λ 0 = λ 0 tr A − 1 A 1 / 2 B 2 d ( 1 ) A^{1/2}B_2^d(1) A 1/2 B 2 d ( 1 )
∥ θ ~ ∥ 2 D ∗ ℓ ∗ ≤ ∥ θ ∥ ∗ ≤ ∥ θ ~ ∥ 2 ℓ ∗ , ∥ x ∥ ∘ D ∗ ℓ ∗ ≤ ∥ x ~ ∥ 2 ≤ ∥ x ∥ ∘ ℓ ∗ , ∥ x ~ ∗ ∥ 2 ≥ ϱ ∗ D ∗ ℓ ∗ , (13) \frac{\lVert\widetilde{\theta}\rVert_2}{D_*\ell_*}\le\lVert\theta\rVert_*\le\frac{\lVert\widetilde{\theta}\rVert_2}{\ell_*},\qquad
\frac{\lVert x\rVert_\circ}{D_*\ell_*}\le\lVert\widetilde{x}\rVert_2\le\frac{\lVert x\rVert_\circ}{\ell_*},\qquad
\lVert\widetilde{x}^*\rVert_2\ge\frac{\varrho_*}{D_*\ell_*}, \tag*{(13)} D ∗ ℓ ∗ ∥ θ ∥ 2 ≤ ∥ θ ∥ ∗ ≤ ℓ ∗ ∥ θ ∥ 2 , D ∗ ℓ ∗ ∥ x ∥ ∘ ≤ ∥ x ∥ 2 ≤ ℓ ∗ ∥ x ∥ ∘ , ∥ x ∗ ∥ 2 ≥ D ∗ ℓ ∗ ϱ ∗ , (13) λ ~ t , d ≥ ω t − 1 D 2 ℓ ∗ 2 , \widetilde{\lambda}_{t,d}\ge\frac{\omega_t^{-1}}{D^2\ell_*^2}, λ t , d ≥ D 2 ℓ ∗ 2 ω t − 1 , hence λ ~ t , d ≥ 2 \widetilde{\lambda}_{t,d}\ge2 λ t , d ≥ 2 ω t − 1 ≥ 2 D 2 ℓ ∗ 2 \omega_t^{-1}\ge2D^2\ell_*^2 ω t − 1 ≥ 2 D 2 ℓ ∗ 2
where λ ~ t , 1 ≥ ⋯ ≥ λ ~ t , d > 0 \widetilde{\lambda}_{t,1}\ge\cdots\ge\widetilde{\lambda}_{t,d}>0 λ t , 1 ≥ ⋯ ≥ λ t , d > 0 V ~ t \widetilde{V}_t V t v ~ t , 1 , … , v ~ t , d \widetilde{v}_{t,1},\ldots,\widetilde{v}_{t,d} v t , 1 , … , v t , d t t t V ~ t + 1 ⪰ V ~ t \widetilde{V}_{t+1}\succeq\widetilde{V}_t V t + 1 ⪰ V t λ ~ t + 1 , i ≤ λ ~ t , i + 1 \widetilde{\lambda}_{t+1,i}\le\widetilde{\lambda}_{t,i+1} λ t + 1 , i ≤ λ t , i + 1 u u u u i : = ⟨ u , v ~ t , i ⟩ u_i:=\langle u,\widetilde{v}_{t,i}\rangle u i := ⟨ u , v t , i ⟩ x ~ t , i : = ⟨ x ~ t , v ~ t , i ⟩ \widetilde{x}_{t,i}:=\langle\widetilde{x}_t,\widetilde{v}_{t,i}\rangle x t , i := ⟨ x t , v t , i ⟩ e ~ ∗ : = x ~ ∗ / ∥ x ~ ∗ ∥ 2 \widetilde{e}^*:=\widetilde{x}^*/\lVert\widetilde{x}^*\rVert_2 e ∗ := x ∗ / ∥ x ∗ ∥ 2 P ~ ⊥ : = I − e ~ ∗ ( e ~ ∗ ) ⊤ \widetilde{P}^{\perp}:=I-\widetilde{e}^*(\widetilde{e}^*)^\top P ⊥ := I − e ∗ ( e ∗ ) ⊤ ϕ ~ t : = ∠ ( v ~ t , 1 , e ~ ∗ ) \widetilde{\phi}_t:=\angle(\widetilde{v}_{t,1},\widetilde{e}^*) ϕ t := ∠ ( v t , 1 , e ∗ ) v ~ t , 1 \widetilde{v}_{t,1} v t , 1 cos ϕ ~ t ≥ 0 \cos\widetilde{\phi}_t\ge0 cos ϕ t ≥ 0 ω t \omega_t ω t ∥ θ − θ ′ ∥ ∗ \lVert\theta-\theta'\rVert_* ∥ θ − θ ′ ∥ ∗ ∥ x − x ′ ∥ ∘ \lVert x-x'\rVert_\circ ∥ x − x ′ ∥ ∘ ∥ θ − θ ′ ∥ V t = ∥ θ ~ − θ ~ ′ ∥ V ~ t \lVert\theta-\theta'\rVert_{V_t}=\lVert\widetilde{\theta}-\widetilde{\theta}'\rVert_{\widetilde{V}_t} ∥ θ − θ ′ ∥ V t = ∥ θ − θ ′ ∥ V t x ⊤ V t − 1 x = x ~ ⊤ V ~ t − 1 x ~ x^\top V_t^{-1}x=\widetilde{x}^\top\widetilde{V}_t^{-1}\widetilde{x} x ⊤ V t − 1 x = x ⊤ V t − 1 x J X J_{\mathcal{X}} J X m m m M M M ℓ ∗ \ell_* ℓ ∗ ϱ ∗ \varrho_* ϱ ∗ J ⋆ J_\star J ⋆ α \alpha α β \beta β Lemma D.5 (ii)), so every statement about them proved below is a statement about the original problem. Proposition 4.1 is applied in the original coordinates; the event E \mathcal{E} E Lemma D.5 (iii)), and only P ( E ) ≥ 1 − δ \mathbb{P}(\mathcal{E})\ge1-\delta P ( E ) ≥ 1 − δ
Definition of the constants
Definition D.6 (Constants). All constants of this section are fixed here once, in dependency order. Two geometric inputs come first,
c e n g : = m 6 J ⋆ and L ∗ : = M + 2 ℓ ∗ 2 J ∗ , (14) c_{\mathrm{eng}}:=\frac{m}{6J_\star}\qquad\text{and}\qquad L_*:=\frac{M+2\ell_*^2}{J_*}, \tag*{(14)} c eng := 6 J ⋆ m and L ∗ := J ∗ M + 2 ℓ ∗ 2 , (14) the exploration-engine constant and the Lipschitz constant of the argmax map θ ↦ ∇ J X ( θ ) \theta\mapsto\nabla J_{\mathcal{X}}(\theta) θ ↦ ∇ J X ( θ ) ( N ∗ , ∥ ⋅ ∥ ∗ ) (\mathcal{N}_*,\lVert\cdot\rVert_*) ( N ∗ , ∥ ⋅ ∥ ∗ ) ( X , ∥ ⋅ ∥ ∘ ) (\mathcal{X},\lVert\cdot\rVert_\circ) ( X , ∥ ⋅ ∥ ∘ ) Proposition D.3 ); then
c 1 : = c e n g α 2 d , c 2 : = 3 L ∗ 2 β 2 ℓ ∗ 2 c 1 , Γ : = 5 × 10 4 d D ∗ 4 L ∗ 4 β ∗ 4 log ( T ) c e n g 4 α 4 , (15) c_1:=\frac{c_{\mathrm{eng}}\alpha}{2\sqrt{d}},\qquad
c_2:=\frac{3L_*^2\beta^2}{\ell_*^2c_1},\qquad
\Gamma:=\frac{5\times10^4\,d\,D_*^4L_*^4\beta_*^4\log(T)}{c_{\mathrm{eng}}^4\alpha^4}, \tag*{(15)} c 1 := 2 d c eng α , c 2 := ℓ ∗ 2 c 1 3 L ∗ 2 β 2 , Γ := c eng 4 α 4 5 × 1 0 4 d D ∗ 4 L ∗ 4 β ∗ 4 log ( T ) , (15) K : = 40 d log ( 8 Γ ) c e n g 2 α 2 , c 0 : = min ( 1 4 K , c 1 log ( 9 T ∗ ) ) , K:=\frac{40d\log(8\Gamma)}{c_{\mathrm{eng}}^2\alpha^2},\qquad
c_0:=\min\left(\frac{1}{4\sqrt{K}},\frac{c_1}{\sqrt{\log(9T_*)}}\right), K := c eng 2 α 2 40 d log ( 8Γ ) , c 0 := min ( 4 K 1 , log ( 9 T ∗ ) c 1 ) , where T ∗ T_* T ∗ poly ( log T ) \operatorname{poly}(\log T) poly ( log T ) Definition D.10 below.
Two facts about Γ \Gamma Γ
L ⋆ β + c e n g α = 6 ( M + 2 ℓ ⋆ 2 ) β + m α ≥ 6 (16) \frac{L_{\star}\beta_{+}}{c_{\mathrm{eng}}\alpha}=\frac{6(M+2\ell_{\star}^{2})\beta_{+}}{m\alpha}\ge6 \tag*{(16)} c eng α L ⋆ β + = m α 6 ( M + 2 ℓ ⋆ 2 ) β + ≥ 6 (16) (as M ≥ m M\ge m M ≥ m β + ≥ α \beta_{+}\ge\alpha β + ≥ α d ≥ 2 d\ge2 d ≥ 2 D ⋆ ≥ 1 D_{\star}\ge1 D ⋆ ≥ 1 log ( T ) ≥ 1 \log(T)\ge1 log ( T ) ≥ 1
Γ ≥ 80 D ⋆ 2 L + 2 β + 2 c e n g 2 α 2 and Γ ≥ 1280 D ⋆ 4 ℓ + 4 C 2 2 log ( T ) c e n g 2 α 2 = 46080 d D ⋆ 4 L + 4 β + 4 log ( T ) c e n g 4 α 4 , (17) \Gamma\ge\frac{80D_{\star}^{2}L_{+}^{2}\beta_{+}^{2}}{c_{\mathrm{eng}}^{2}\alpha^{2}}
\qquad\text{and}\qquad
\Gamma\ge\frac{1280D_{\star}^{4}\ell_{+}^{4}C_{2}^{2}\log(T)}{c_{\mathrm{eng}}^{2}\alpha^{2}}
=\frac{46080dD_{\star}^{4}L_{+}^{4}\beta_{+}^{4}\log(T)}{c_{\mathrm{eng}}^{4}\alpha^{4}}, \tag*{(17)} Γ ≥ c eng 2 α 2 80 D ⋆ 2 L + 2 β + 2 and Γ ≥ c eng 2 α 2 1280 D ⋆ 4 ℓ + 4 C 2 2 log ( T ) = c eng 4 α 4 46080 d D ⋆ 4 L + 4 β + 4 log ( T ) , (17) where C 2 2 = 9 L + 4 β + 4 / ( ℓ ⋆ 4 c 1 2 ) = 36 d L ⋆ 4 β + 4 / ( ℓ ⋆ 4 c e n g 2 α 2 ) C_{2}^{2}=9L_{+}^{4}\beta_{+}^{4}/(\ell_{\star}^{4}c_{1}^{2})=36dL_{\star}^{4}\beta_{+}^{4}/(\ell_{\star}^{4}c_{\mathrm{eng}}^{2}\alpha^{2}) C 2 2 = 9 L + 4 β + 4 / ( ℓ ⋆ 4 c 1 2 ) = 36 d L ⋆ 4 β + 4 / ( ℓ ⋆ 4 c eng 2 α 2 ) Γ ≥ 1 \Gamma\ge1 Γ ≥ 1 J ⋆ J_{\star} J ⋆ Γ \Gamma Γ Γ \Gamma Γ (16) and D ⋆ D_{\star} D ⋆ (11) 4 log ( D ⋆ ) ≤ 2 log ( d ) 4\log(D_{\star})\le2\log(d) 4 log ( D ⋆ ) ≤ 2 log ( d ) log ( 8 Γ ) = O ( log ( e d ) + log ( e + ( M + 2 ℓ ⋆ 2 ) β + / ( m α ) ) + log ( e log ( e T ) ) ) \log(8\Gamma)=O\bigl(\log(ed)+\log(e+(M+2\ell_{\star}^{2})\beta_{+}/(m\alpha))+\log(e\log(eT))\bigr) log ( 8Γ ) = O ( log ( e d ) + log ( e + ( M + 2 ℓ ⋆ 2 ) β + / ( m α )) + log ( e log ( e T )) )
c 0 ≳ m α J ⋆ d log ( 8 Γ ) + log ( 9 T ⋆ ) . c_{0}\gtrsim\frac{m\alpha}{J_{\star}\sqrt{d}\sqrt{\log(8\Gamma)+\log(9T_{\star})}}. c 0 ≳ J ⋆ d log ( 8Γ ) + log ( 9 T ⋆ ) m α . Exploration engine, burn-in and crude bound
The first step of the analysis is a crude bound on the width, ω t − 1 ≳ t / log ( t ) \omega_t^{-1}\gtrsim\sqrt{t/\log(t)} ω t − 1 ≳ t / log ( t ) Lemma D.9 ), obtained from an exploration inequality forced by optimism (the engine , Lemma D.7) and an elliptic-potential argument. The engine holds at every round, deterministically, with a constant that involves J X ( θ ^ t ) J_{\mathcal X}(\hat{\theta}_t) J X ( θ ^ t ) E \mathcal E E Λ 0 \Lambda_0 Λ 0 t 0 t_0 t 0 Proposition D.8 (i)). From t 0 t_0 t 0 C t ( α ) C_t(\alpha) C t ( α ) N ⋆ \mathcal N_{\star} N ⋆ L ⋆ L_{\star} L ⋆ Proposition D.3 ), the played arms are well conditioned in the adapted coordinates (Proposition D.8 (ii)–(v)), and the engine takes its final form with the constant c e n g c_{\mathrm{eng}} c eng
Lemma D.7 (Exploration engine). Recall c e n g = m / ( 6 J ⋆ ) c_{\mathrm{eng}}=m/(6J_{\star}) c eng = m / ( 6 J ⋆ ) β + = α + β \beta_{+}=\alpha+\beta β + = α + β
At every round t ≥ 1 t\ge1 t ≥ 1 x t ≠ 0 x_t\ne0 x t = 0
θ t = θ ^ t + α V t − 1 x t ∥ x t ∥ V t − 1 , (18) \theta_t=\hat{\theta}_t+\alpha\frac{V_t^{-1}x_t}{\lVert x_t\rVert_{V_t^{-1}}}, \tag*{(18)} θ t = θ ^ t + α ∥ x t ∥ V t − 1 V t − 1 x t , (18) and
∥ x t ∥ V t − 1 ≥ m α ω t 4 J X ( θ ^ t ) + 2 ℓ ⋆ α ω t 1 / 2 . (19) \lVert x_t\rVert_{V_t^{-1}}\ge\frac{m\alpha\omega_t}{4J_{\mathcal X}(\hat{\theta}_t)+2\ell_{\star}\alpha\omega_t^{1/2}}. \tag*{(19)} ∥ x t ∥ V t − 1 ≥ 4 J X ( θ ^ t ) + 2 ℓ ⋆ α ω t 1/2 m α ω t . (19) On E \mathcal E E t ∈ [ T ] t\in[T] t ∈ [ T ]
x t T V t − 1 x t ≥ m 2 α 2 64 min { ω t 2 J ⋆ 2 , ω t ℓ ⋆ 2 β + 2 } . (20) x_t^{\mathsf T}V_t^{-1}x_t\ge\frac{m^{2}\alpha^{2}}{64}\min\left\{\frac{\omega_t^{2}}{J_{\star}^{2}},\frac{\omega_t}{\ell_{\star}^{2}\beta_{+}^{2}}\right\}. \tag*{(20)} x t T V t − 1 x t ≥ 64 m 2 α 2 min { J ⋆ 2 ω t 2 , ℓ ⋆ 2 β + 2 ω t } . (20) On E \mathcal E E t ∈ [ T ] t\in[T] t ∈ [ T ] ℓ ⋆ β + ω t 1 / 2 ≤ J ⋆ / 2 \ell_{\star}\beta_{+}\omega_t^{1/2}\le J_{\star}/2 ℓ ⋆ β + ω t 1/2 ≤ J ⋆ /2
∥ x t ∥ V t − 1 ≥ c e n g α ω t , hence x t T V t − 1 x t ≥ c e n g 2 α 2 ω t 2 . (21) \lVert x_t\rVert_{V_t^{-1}}\ge c_{\mathrm{eng}}\alpha\omega_t,\qquad\text{hence}\qquad x_t^{\mathsf T}V_t^{-1}x_t\ge c_{\mathrm{eng}}^{2}\alpha^{2}\omega_t^{2}. \tag*{(21)} ∥ x t ∥ V t − 1 ≥ c eng α ω t , hence x t T V t − 1 x t ≥ c eng 2 α 2 ω t 2 . (21) The statements and their proof involve neither the burn-in nor a Euclidean structure.
Proof of Lemma D.7. (i) Reduction to the confidence set. By definition of the optimistic pair and Proposition D.1(v),
⟨ θ t , x t ⟩ = max θ ∈ C t ( α ) max x ∈ X ⟨ θ , x ⟩ = max θ ∈ C t ( α ) J X ( θ ) , x t = ∇ J X ( θ t ) , ⟨ θ t , x t ⟩ = J X ( θ t ) > 0 , \langle\theta_t,x_t\rangle=\max_{\theta\in C_t(\alpha)}\max_{x\in\mathcal X}\langle\theta,x\rangle=\max_{\theta\in C_t(\alpha)}J_{\mathcal X}(\theta),\qquad x_t=\nabla J_{\mathcal X}(\theta_t),\qquad\langle\theta_t,x_t\rangle=J_{\mathcal X}(\theta_t)>0, ⟨ θ t , x t ⟩ = θ ∈ C t ( α ) max x ∈ X max ⟨ θ , x ⟩ = θ ∈ C t ( α ) max J X ( θ ) , x t = ∇ J X ( θ t ) , ⟨ θ t , x t ⟩ = J X ( θ t ) > 0 , so θ t ∈ arg max θ ∈ C t ( α ) J X ( θ ) \theta_t\in\arg\max_{\theta\in C_t(\alpha)}J_{\mathcal X}(\theta) θ t ∈ arg max θ ∈ C t ( α ) J X ( θ ) x t ≠ 0 x_t\ne0 x t = 0
Stationarity identity. For x ≠ 0 x\ne0 x = 0 θ ∈ C t ( α ) \theta\in C_t(\alpha) θ ∈ C t ( α ) V t V_t V t ⟨ θ − θ ^ t , x ⟩ ≤ ∥ x ∥ V t − 1 ∥ θ − θ ^ t ∥ V t ≤ α ∥ x ∥ V t − 1 \langle\theta-\hat{\theta}_t,x\rangle\le\lVert x\rVert_{V_t^{-1}}\lVert\theta-\hat{\theta}_t\rVert_{V_t}\le\alpha\lVert x\rVert_{V_t^{-1}} ⟨ θ − θ ^ t , x ⟩ ≤ ∥ x ∥ V t − 1 ∥ θ − θ ^ t ∥ V t ≤ α ∥ x ∥ V t − 1 θ = θ ^ t + α V t − 1 x / ∥ x ∥ V t − 1 \theta=\hat{\theta}_t+\alpha V_t^{-1}x/\lVert x\rVert_{V_t^{-1}} θ = θ ^ t + α V t − 1 x / ∥ x ∥ V t − 1 θ t \theta_t θ t ⟨ ⋅ , x t ⟩ \langle\cdot,x_t\rangle ⟨ ⋅ , x t ⟩ C t ( α ) C_t(\alpha) C t ( α )
θ t − θ ^ t = α ∥ x t ∥ V t − 1 V t − 1 x t , ∥ θ t − θ ^ t ∥ V t = α . (22) \theta_t-\hat{\theta}_t=\frac{\alpha}{\lVert x_t\rVert_{V_t^{-1}}}V_t^{-1}x_t,\qquad\lVert\theta_t-\hat{\theta}_t\rVert_{V_t}=\alpha. \tag*{(22)} θ t − θ ^ t = ∥ x t ∥ V t − 1 α V t − 1 x t , ∥ θ t − θ ^ t ∥ V t = α . (22) which is the closed form in (i). It also gives the optimism inequality
J X ( θ t ) = ⟨ θ ^ t , x t ⟩ + ⟨ θ t − θ ^ t , x t ⟩ ≤ J X ( θ ^ t ) + α ∥ x t ∥ V t − 1 . (23) J_{\mathcal X}(\theta_t)=\langle\widehat{\theta}_t,x_t\rangle+\langle\theta_t-\widehat{\theta}_t,x_t\rangle\le J_{\mathcal X}(\widehat{\theta}_t)+\alpha\|x_t\|_{V_t^{-1}}. \tag*{(23)} J X ( θ t ) = ⟨ θ t , x t ⟩ + ⟨ θ t − θ t , x t ⟩ ≤ J X ( θ t ) + α ∥ x t ∥ V t − 1 . (23) The multiplier, via strong convexity of J X 2 J_{\mathcal X}^{2} J X 2 Let v ∘ v^\circ v ∘ ω t \omega_t ω t ∥ v ∘ ∥ V t ≤ 1 \|v^\circ\|_{V_t}\le1 ∥ v ∘ ∥ V t ≤ 1 ∥ v ∘ ∥ ∗ = ω t 1 / 2 \|v^\circ\|_*=\omega_t^{1/2} ∥ v ∘ ∥ ∗ = ω t 1/2 ζ : = α v ∘ \zeta:=\alpha v^\circ ζ := α v ∘
∥ ζ ∥ V t ≤ α , ∥ ζ ∥ ∗ = α ω t 1 / 2 . \|\zeta\|_{V_t}\le\alpha,\qquad\|\zeta\|_*=\alpha\omega_t^{1/2}. ∥ ζ ∥ V t ≤ α , ∥ ζ ∥ ∗ = α ω t 1/2 . the two points θ ^ t ± ζ \widehat{\theta}_t\pm\zeta θ t ± ζ C t ( α ) C_t(\alpha) C t ( α ) ∥ ⋅ ∥ ∗ \|\cdot\|_* ∥ ⋅ ∥ ∗ θ t \theta_t θ t J X J_{\mathcal X} J X C t ( α ) C_t(\alpha) C t ( α )
α ∥ x t ∥ V t − 1 ≥ G : = max { J X ( θ ^ t + ζ ) , J X ( θ ^ t − ζ ) } − J X ( θ ^ t ) ≥ 0 , (24) \alpha\|x_t\|_{V_t^{-1}}\ge G:=\max\{J_{\mathcal X}(\widehat{\theta}_t+\zeta),J_{\mathcal X}(\widehat{\theta}_t-\zeta)\}-J_{\mathcal X}(\widehat{\theta}_t)\ge0, \tag*{(24)} α ∥ x t ∥ V t − 1 ≥ G := max { J X ( θ t + ζ ) , J X ( θ t − ζ )} − J X ( θ t ) ≥ 0 , (24) the last inequality by convexity (G ≥ 1 2 [ J X ( θ ^ t + ζ ) + J X ( θ ^ t − ζ ) ] − J X ( θ ^ t ) G\ge\frac{1}{2}[J_{\mathcal X}(\widehat{\theta}_t+\zeta)+J_{\mathcal X}(\widehat{\theta}_t-\zeta)]-J_{\mathcal X}(\widehat{\theta}_t) G ≥ 2 1 [ J X ( θ t + ζ ) + J X ( θ t − ζ )] − J X ( θ t ) θ ′ = θ ^ t \theta'=\widehat{\theta}_t θ ′ = θ t θ = θ ^ t ± ζ \theta=\widehat{\theta}_t\pm\zeta θ = θ t ± ζ ± ⟨ ζ , ∇ J X 2 ( θ ^ t ) ⟩ \pm\langle\zeta,\nabla J_{\mathcal X}^{2}(\widehat{\theta}_t)\rangle ± ⟨ ζ , ∇ J X 2 ( θ t )⟩
J X 2 ( θ ^ t + ζ ) + J X 2 ( θ ^ t − ζ ) − 2 J X 2 ( θ ^ t ) = D J X 2 ( θ ^ t + ζ , θ ^ t ) + D J X 2 ( θ ^ t − ζ , θ ^ t ) ≥ m ∥ ζ ∥ ∗ 2 = m α 2 ω t . (25) J_{\mathcal X}^{2}(\widehat{\theta}_t+\zeta)+J_{\mathcal X}^{2}(\widehat{\theta}_t-\zeta)-2J_{\mathcal X}^{2}(\widehat{\theta}_t)=D_{J_{\mathcal X}^{2}}(\widehat{\theta}_t+\zeta,\widehat{\theta}_t)+D_{J_{\mathcal X}^{2}}(\widehat{\theta}_t-\zeta,\widehat{\theta}_t)\ge m\|\zeta\|_*^2=m\alpha^2\omega_t. \tag*{(25)} J X 2 ( θ t + ζ ) + J X 2 ( θ t − ζ ) − 2 J X 2 ( θ t ) = D J X 2 ( θ t + ζ , θ t ) + D J X 2 ( θ t − ζ , θ t ) ≥ m ∥ ζ ∥ ∗ 2 = m α 2 ω t . (25) On the other hand, for each sign,
J X 2 ( θ ^ t ± ζ ) − J X 2 ( θ ^ t ) = [ J X ( θ ^ t ± ζ ) − J X ( θ ^ t ) ] [ J X ( θ ^ t ± ζ ) + J X ( θ ^ t ) ] ≤ G ( 2 J X ( θ ^ t ) + G ) , J_{\mathcal X}^{2}(\widehat{\theta}_t\pm\zeta)-J_{\mathcal X}^{2}(\widehat{\theta}_t)=\bigl[J_{\mathcal X}(\widehat{\theta}_t\pm\zeta)-J_{\mathcal X}(\widehat{\theta}_t)\bigr]\bigl[J_{\mathcal X}(\widehat{\theta}_t\pm\zeta)+J_{\mathcal X}(\widehat{\theta}_t)\bigr]\le G\bigl(2J_{\mathcal X}(\widehat{\theta}_t)+G\bigr), J X 2 ( θ t ± ζ ) − J X 2 ( θ t ) = [ J X ( θ t ± ζ ) − J X ( θ t ) ] [ J X ( θ t ± ζ ) + J X ( θ t ) ] ≤ G ( 2 J X ( θ t ) + G ) , because the product is nonpositive when the first factor is, and otherwise the first factor is at most G G G 2 J X ( θ ^ t ) + G 2J_{\mathcal X}(\widehat{\theta}_t)+G 2 J X ( θ t ) + G 2 G ( 2 J X ( θ ^ t ) + G ) ≥ m α 2 ω t 2G\bigl(2J_{\mathcal X}(\widehat{\theta}_t)+G\bigr)\ge m\alpha^2\omega_t 2 G ( 2 J X ( θ t ) + G ) ≥ m α 2 ω t G ≤ α ∥ x t ∥ V t − 1 ≤ ℓ ∗ α ω t 1 / 2 G\le\alpha\|x_t\|_{V_t^{-1}}\le\ell_*\alpha\omega_t^{1/2} G ≤ α ∥ x t ∥ V t − 1 ≤ ℓ ∗ α ω t 1/2
∥ x t ∥ V t − 1 ≥ G α ≥ m α ω t 4 J X ( θ ^ t ) + 2 G ≥ m α ω t 4 J X ( θ ^ t ) + 2 ℓ ∗ α ω t 1 / 2 . (26) \|x_t\|_{V_t^{-1}}\ge\frac{G}{\alpha}\ge\frac{m\alpha\omega_t}{4J_{\mathcal X}(\widehat{\theta}_t)+2G}\ge\frac{m\alpha\omega_t}{4J_{\mathcal X}(\widehat{\theta}_t)+2\ell_*\alpha\omega_t^{1/2}}. \tag*{(26)} ∥ x t ∥ V t − 1 ≥ α G ≥ 4 J X ( θ t ) + 2 G m α ω t ≥ 4 J X ( θ t ) + 2 ℓ ∗ α ω t 1/2 m α ω t . (26) which is equation 20. The perturbation is chosen in the ∥ ⋅ ∥ ∗ \|\cdot\|_* ∥ ⋅ ∥ ∗ J X J_{\mathcal X} J X 2 ∇ J X ∇ J X T 2\nabla J_{\mathcal X}\nabla J_{\mathcal X}^{\mathsf T} 2∇ J X ∇ J X T ∇ 2 J X 2 \nabla^2J_{\mathcal X}^{2} ∇ 2 J X 2 square buys.)
(ii) On E \mathcal E E ∥ θ ^ t − θ ∗ ∥ ∗ ≤ β ω t 1 / 2 \|\widehat{\theta}_t-\theta^*\|_*\le\beta\omega_t^{1/2} ∥ θ t − θ ∗ ∥ ∗ ≤ β ω t 1/2 ℓ ∗ \ell_* ℓ ∗ J X J_{\mathcal X} J X
J X ( θ ^ t ) ≤ J ∗ + ℓ ∗ β ω t 1 / 2 , J_{\mathcal X}(\widehat{\theta}_t)\le J_*+\ell_*\beta\omega_t^{1/2}, J X ( θ t ) ≤ J ∗ + ℓ ∗ β ω t 1/2 , and the denominator in equation 20 is at most
4 J ∗ + 4 ℓ ∗ β ω t 1 / 2 ≤ 8 max { J ∗ , ℓ ∗ β ω t 1 / 2 } . 4J_*+4\ell_*\beta\omega_t^{1/2}\le8\max\{J_*,\ell_*\beta\omega_t^{1/2}\}. 4 J ∗ + 4 ℓ ∗ β ω t 1/2 ≤ 8 max { J ∗ , ℓ ∗ β ω t 1/2 } . Hence ∥ x t ∥ V t − 1 ≥ m α 8 min { ω t / J ∗ , ω t 1 / 2 / ( ℓ ∗ β ) } \|x_t\|_{V_t^{-1}}\ge\frac{m\alpha}{8}\min\{\omega_t/J_*,\omega_t^{1/2}/(\ell_*\beta)\} ∥ x t ∥ V t − 1 ≥ 8 m α min { ω t / J ∗ , ω t 1/2 / ( ℓ ∗ β )}
(iii) If ℓ ∗ β ω t 1 / 2 ≤ J ∗ / 2 \ell_*\beta\omega_t^{1/2}\le J_*/2 ℓ ∗ β ω t 1/2 ≤ J ∗ /2 6 J ∗ 6J_* 6 J ∗ ∥ x t ∥ V t − 1 ≥ m α ω t / ( 6 J ∗ ) = c e n g α ω t \|x_t\|_{V_t^{-1}}\ge m\alpha\omega_t/(6J_*)=c_{\mathrm{eng}}\alpha\omega_t ∥ x t ∥ V t − 1 ≥ m α ω t / ( 6 J ∗ ) = c eng α ω t
Proposition D.8 (Burn-in). Recall β + = α + β \beta_+=\alpha+\beta β + = α + β L ∗ = ( M + 2 ℓ ∗ 2 ) / J ∗ L_*=(M+2\ell_*^2)/J_* L ∗ = ( M + 2 ℓ ∗ 2 ) / J ∗
Λ 0 ∗ : = max { 2 D ∗ 2 ℓ ∗ 2 , 16 ℓ ∗ 2 β + 2 J ∗ 2 , 4 D ∗ 2 L ∗ 2 β + 2 ϱ ∗ 2 } . \Lambda_0^*:=\max\left\{2D_*^2\ell_*^2,\frac{16\ell_*^2\beta_+^2}{J_*^2},\frac{4D_*^2L_*^2\beta_+^2}{\varrho_*^2}\right\}. Λ 0 ∗ := max { 2 D ∗ 2 ℓ ∗ 2 , J ∗ 2 16 ℓ ∗ 2 β + 2 , ϱ ∗ 2 4 D ∗ 2 L ∗ 2 β + 2 } . t 0 : = min { t ≥ 1 : t log ( 1 + t / ( d λ 0 ) ) > max { 2 d , 128 d J ∗ 2 Λ 0 ∗ 2 m 2 α 2 } } . (27) t_0:=\min\left\{t\ge1:\frac{t}{\log(1+t/(d\lambda_0))}>\max\left\{2d,\frac{128dJ_*^2\Lambda_0^{*2}}{m^2\alpha^2}\right\}\right\}. \tag*{(27)} t 0 := min { t ≥ 1 : log ( 1 + t / ( d λ 0 )) t > max { 2 d , m 2 α 2 128 d J ∗ 2 Λ 0 ∗ 2 } } . (27) The round t 0 t_0 t 0 t ↦ t / log ( 1 + t / ( d λ 0 ) ) t\mapsto t/\log(1+t/(d\lambda_0)) t ↦ t / log ( 1 + t / ( d λ 0 )) E \mathcal E E t 0 ≤ t ≤ T t_0\le t\le T t 0 ≤ t ≤ T
(i) ω t − 1 ≥ Λ 0 ∗ \omega_t^{-1}\ge\Lambda_0^* ω t − 1 ≥ Λ 0 ∗
(ii) ∥ θ ^ t − θ ∗ ∥ ∗ ≤ β ω t 1 / 2 ≤ J ∗ / ( 4 ℓ ∗ ) \|\widehat{\theta}_t-\theta^*\|_*\le\beta\omega_t^{1/2}\le J_*/(4\ell_*) ∥ θ t − θ ∗ ∥ ∗ ≤ β ω t 1/2 ≤ J ∗ / ( 4 ℓ ∗ ) θ ^ t ∈ N ∗ \widehat{\theta}_t\in\mathcal N_* θ t ∈ N ∗
(iii) ∥ θ t − θ ^ t ∥ ∗ ≤ α ω t 1 / 2 ≤ J ∗ / ( 4 ℓ ∗ ) and ∥ θ t − θ ∗ ∥ ∗ ≤ β + ω t 1 / 2 ≤ J ∗ / ( 4 ℓ ∗ ) ; hence θ t ∈ N ∗ , and θ ^ t + ζ ∈ N ∗ for every ζ with ∥ ζ ∥ V t ≤ α (the whole confidence ellipsoid lies in N ∗ ); (iv) x t = ∇ J X ( θ t ) , ∥ x t − x ∗ ∥ ∗ ≤ L ∗ β + ω t 1 / 2 , and 1 2 J ∗ ≤ J X ( θ t ) ≤ 3 2 J ∗ ; (v) in the adapted coordinates, ∥ x ~ t − x ~ ∗ ∥ 2 ≤ L ∗ β + ω t 1 / 2 / ℓ ∗ ≤ ∥ x ~ ∗ ∥ 2 / 2 ; therefore 1 2 ∥ x ~ ∗ ∥ 2 ≤ ∥ x ~ t ∥ 2 ≤ 3 2 ∥ x ~ ∗ ∥ 2 and ⟨ e ~ ∗ , x ~ t ⟩ ≥ 1 2 ∥ x ~ ∗ ∥ 2 . (28) \begin{aligned}
\text{(iii)}\quad&\lVert\theta_t-\hat{\theta}_t\rVert_* \le\alpha\omega_t^{1/2}\le J_*/(4\ell_*)\text{ and }\lVert\theta_t-\theta^*\rVert_* \le\beta_+\omega_t^{1/2}\le J_*/(4\ell_*); \text{ hence }\theta_t\in\mathcal{N}_*,\\
&\text{and }\hat{\theta}_t+\zeta\in\mathcal{N}_*\text{ for every }\zeta\text{ with }\lVert\zeta\rVert_{V_t}\le\alpha\text{ (the whole confidence ellipsoid lies in }\mathcal{N}_*\text{);}\\
\text{(iv)}\quad&x_t=\nabla J_{\mathcal{X}}(\theta_t),\quad\lVert x_t-x^*\rVert_* \le L_*\beta_+\omega_t^{1/2},\quad\text{and }\frac{1}{2}J_*\le J_{\mathcal{X}}(\theta_t)\le\frac{3}{2}J_*;\\
\text{(v)}\quad&\text{in the adapted coordinates, }\lVert\tilde{x}_t-\tilde{x}^*\rVert_2\le L_*\beta_+\omega_t^{1/2}/\ell_*\le\lVert\tilde{x}^*\rVert_2/2;\text{ therefore}\\
&\frac{1}{2}\lVert\tilde{x}^*\rVert_2\le\lVert\tilde{x}_t\rVert_2\le\frac{3}{2}\lVert\tilde{x}^*\rVert_2\quad\text{and}\quad\langle\tilde{e}^*,\tilde{x}_t\rangle\ge\frac{1}{2}\lVert\tilde{x}^*\rVert_2. \tag*{(28)}
\end{aligned} (iii) (iv) (v) ∥ θ t − θ ^ t ∥ ∗ ≤ α ω t 1/2 ≤ J ∗ / ( 4 ℓ ∗ ) and ∥ θ t − θ ∗ ∥ ∗ ≤ β + ω t 1/2 ≤ J ∗ / ( 4 ℓ ∗ ) ; hence θ t ∈ N ∗ , and θ ^ t + ζ ∈ N ∗ for every ζ with ∥ ζ ∥ V t ≤ α (the whole confidence ellipsoid lies in N ∗ ); x t = ∇ J X ( θ t ) , ∥ x t − x ∗ ∥ ∗ ≤ L ∗ β + ω t 1/2 , and 2 1 J ∗ ≤ J X ( θ t ) ≤ 2 3 J ∗ ; in the adapted coordinates, ∥ x ~ t − x ~ ∗ ∥ 2 ≤ L ∗ β + ω t 1/2 / ℓ ∗ ≤ ∥ x ~ ∗ ∥ 2 /2 ; therefore 2 1 ∥ x ~ ∗ ∥ 2 ≤ ∥ x ~ t ∥ 2 ≤ 2 3 ∥ x ~ ∗ ∥ 2 and ⟨ e ~ ∗ , x ~ t ⟩ ≥ 2 1 ∥ x ~ ∗ ∥ 2 . (28) Nothing is assumed about the trajectory: (i) is derived from the engine equation 21 and an elliptic-potential argument run from the first round, and (ii)–(v) follow from (i).
Proof of Proposition D.8. (i) Fix T ′ ∈ [ T ] T'\in[T] T ′ ∈ [ T ] V 1 = λ 0 I V_1=\lambda_0 I V 1 = λ 0 I ∥ x t ∥ 2 ≤ 1 \lVert x_t\rVert_2\le1 ∥ x t ∥ 2 ≤ 1 X ⊆ B d ( 1 ) \mathcal{X}\subseteq\mathbb{B}^d(1) X ⊆ B d ( 1 ) tr V T ′ + 1 ≤ d λ 0 + T ′ \operatorname{tr}V_{T'+1}\le d\lambda_0+T' tr V T ′ + 1 ≤ d λ 0 + T ′
∑ t = 1 T ′ log ( 1 + x t ⊤ V t − 1 x t ) = log ( det ( V T ′ + 1 ) det ( V 1 ) ) ≤ d log ( 1 + T ′ d λ 0 ) . \sum_{t=1}^{T'}\log(1+x_t^\top V_t^{-1}x_t)=\log\left(\frac{\det(V_{T'+1})}{\det(V_1)}\right)\le d\log\left(1+\frac{T'}{d\lambda_0}\right). t = 1 ∑ T ′ log ( 1 + x t ⊤ V t − 1 x t ) = log ( det ( V 1 ) det ( V T ′ + 1 ) ) ≤ d log ( 1 + d λ 0 T ′ ) . Since min { 1 , y } ≤ 2 log ( 1 + y ) \min\{1,y\}\le2\log(1+y) min { 1 , y } ≤ 2 log ( 1 + y ) y ≥ 0 y\ge0 y ≥ 0 1 2 ∑ t ≤ T ′ min { 1 , x t ⊤ V t − 1 x t } \frac{1}{2}\sum_{t\le T'}\min\{1,x_t^\top V_t^{-1}x_t\} 2 1 ∑ t ≤ T ′ min { 1 , x t ⊤ V t − 1 x t } E \mathcal{E} E g ( ω t ) g(\omega_t) g ( ω t ) g ( ω ) : = m 2 α 2 64 min { ω 2 / J ∗ 2 , ω / ( ℓ ∗ 2 β + 2 ) } g(\omega):=\frac{m^2\alpha^2}{64}\min\{\omega^2/J_*^2,\omega/(\ell_*^2\beta_+^2)\} g ( ω ) := 64 m 2 α 2 min { ω 2 / J ∗ 2 , ω / ( ℓ ∗ 2 β + 2 )} ω \omega ω ω t ≥ ω T ′ \omega_t\ge\omega_{T'} ω t ≥ ω T ′ t ≤ T ′ t\le T' t ≤ T ′
T ′ min { 1 , g ( ω T ′ ) } ≤ ∑ t = 1 T ′ min { 1 , x t ⊤ V t − 1 x t } ≤ 2 d log ( 1 + T ′ d λ 0 ) . T'\min\{1,g(\omega_{T'})\}\le\sum_{t=1}^{T'}\min\{1,x_t^\top V_t^{-1}x_t\}\le2d\log\left(1+\frac{T'}{d\lambda_0}\right). T ′ min { 1 , g ( ω T ′ )} ≤ t = 1 ∑ T ′ min { 1 , x t ⊤ V t − 1 x t } ≤ 2 d log ( 1 + d λ 0 T ′ ) . Let T ′ ≥ t 0 T'\ge t_0 T ′ ≥ t 0 t ↦ t / log ( 1 + t / ( d λ 0 ) ) t\mapsto t/\log(1+t/(d\lambda_0)) t ↦ t / log ( 1 + t / ( d λ 0 )) t 0 t_0 t 0 T ′ T' T ′ T ′ > 2 d log ( 1 + T ′ / ( d λ 0 ) ) T'>2d\log(1+T'/(d\lambda_0)) T ′ > 2 d log ( 1 + T ′ / ( d λ 0 )) g ( ω T ′ ) < 1 g(\omega_{T'})<1 g ( ω T ′ ) < 1
g ( ω T ′ ) ≤ 2 d log ( 1 + T ′ / ( d λ 0 ) ) T ′ < m 2 α 2 64 J ∗ 2 Λ 0 ∗ 2 . g(\omega_{T'})\le\frac{2d\log(1+T'/(d\lambda_0))}{T'}<\frac{m^2\alpha^2}{64J_*^2\Lambda_0^{*2}}. g ( ω T ′ ) ≤ T ′ 2 d log ( 1 + T ′ / ( d λ 0 )) < 64 J ∗ 2 Λ 0 ∗ 2 m 2 α 2 . If ω T ′ > 1 / Λ 0 ∗ \omega_{T'}>1/\Lambda_0^* ω T ′ > 1/ Λ 0 ∗ g g g Λ 0 ∗ \Lambda_0^* Λ 0 ∗ ℓ ∗ 2 β + 2 ≤ J ∗ 2 Λ 0 ∗ \ell_*^2\beta_+^2\le J_*^2\Lambda_0^* ℓ ∗ 2 β + 2 ≤ J ∗ 2 Λ 0 ∗ ω = 1 / Λ 0 ∗ \omega=1/\Lambda_0^* ω = 1/ Λ 0 ∗
g ( ω T ′ ) ≥ g ( 1 / Λ 0 ∗ ) = m 2 α 2 64 min { 1 J ∗ 2 Λ 0 ∗ 2 , 1 ℓ ∗ 2 β + 2 Λ 0 ∗ } = m 2 α 2 64 J ∗ 2 Λ 0 ∗ 2 , g(\omega_{T'})\ge g(1/\Lambda_0^*)=\frac{m^2\alpha^2}{64}\min\left\{\frac{1}{J_*^2\Lambda_0^{*2}},\frac{1}{\ell_*^2\beta_+^2\Lambda_0^*}\right\}=\frac{m^2\alpha^2}{64J_*^2\Lambda_0^{*2}}, g ( ω T ′ ) ≥ g ( 1/ Λ 0 ∗ ) = 64 m 2 α 2 min { J ∗ 2 Λ 0 ∗ 2 1 , ℓ ∗ 2 β + 2 Λ 0 ∗ 1 } = 64 J ∗ 2 Λ 0 ∗ 2 m 2 α 2 , a contradiction. Hence ω T ′ ≤ 1 / Λ 0 ∗ \omega_{T'}\le1/\Lambda_0^* ω T ′ ≤ 1/ Λ 0 ∗
For (ii)–(v), fix t 0 ≤ t ≤ T t_0\le t\le T t 0 ≤ t ≤ T ω t − 1 ≥ Λ 0 ∗ \omega_t^{-1}\ge\Lambda_0^* ω t − 1 ≥ Λ 0 ∗ α , β ≤ β + \alpha,\beta\le\beta_+ α , β ≤ β + E \mathcal{E} E ∥ θ ^ t − θ ∗ ∥ ∗ ≤ β ω t 1 / 2 ≤ β + / Λ 0 ∗ ≤ J ∗ / ( 4 ℓ ∗ ) \lVert\hat{\theta}_t-\theta^*\rVert_*\le\beta\omega_t^{1/2}\le\beta_+/\sqrt{\Lambda_0^*}\le J_*/(4\ell_*) ∥ θ ^ t − θ ∗ ∥ ∗ ≤ β ω t 1/2 ≤ β + / Λ 0 ∗ ≤ J ∗ / ( 4 ℓ ∗ ) Λ 0 ∗ \Lambda_0^* Λ 0 ∗ θ t ∈ C t ( α ) \theta_t\in\mathcal{C}_t(\alpha) θ t ∈ C t ( α ) ∥ θ t − θ ^ t ∥ ∗ ≤ α ω t 1 / 2 ≤ J ∗ / ( 4 ℓ ∗ ) \lVert\theta_t-\hat{\theta}_t\rVert_*\le\alpha\omega_t^{1/2}\le J_*/(4\ell_*) ∥ θ t − θ ^ t ∥ ∗ ≤ α ω t 1/2 ≤ J ∗ / ( 4 ℓ ∗ ) ∥ θ t − θ ∗ ∥ ∗ ≤ ( α + β ) ω t 1 / 2 = β + ω t 1 / 2 ≤ J ∗ / ( 4 ℓ ∗ ) \lVert\theta_t-\theta^*\rVert_*\le(\alpha+\beta)\omega_t^{1/2}=\beta_+\omega_t^{1/2}\le J_*/(4\ell_*) ∥ θ t − θ ∗ ∥ ∗ ≤ ( α + β ) ω t 1/2 = β + ω t 1/2 ≤ J ∗ / ( 4 ℓ ∗ ) ∥ ζ ∥ V t ≤ α \lVert\zeta\rVert_{V_t}\le\alpha ∥ ζ ∥ V t ≤ α ∥ ζ ∥ ∗ ≤ α ω t 1 / 2 ≤ J ∗ / ( 4 ℓ ∗ ) \lVert\zeta\rVert_*\le\alpha\omega_t^{1/2}\le J_*/(4\ell_*) ∥ ζ ∥ ∗ ≤ α ω t 1/2 ≤ J ∗ / ( 4 ℓ ∗ ) ∥ θ ^ t + ζ − θ ∗ ∥ ∗ ≤ J ∗ / ( 2 ℓ ∗ ) \lVert\hat{\theta}_t+\zeta-\theta^*\rVert_*\le J_*/(2\ell_*) ∥ θ ^ t + ζ − θ ∗ ∥ ∗ ≤ J ∗ / ( 2 ℓ ∗ ) x t = ∇ J X ( θ t ) x_t=\nabla J_{\mathcal{X}}(\theta_t) x t = ∇ J X ( θ t ) θ t , θ ∗ ∈ N ∗ \theta_t,\theta^*\in\mathcal{N}_* θ t , θ ∗ ∈ N ∗ ∥ x t − x ∗ ∥ ∗ ≤ L ∗ ∥ θ t − θ ∗ ∥ ∗ ≤ L ∗ β + ω t 1 / 2 \lVert x_t-x^*\rVert_*\le L_*\lVert\theta_t-\theta^*\rVert_*\le L_*\beta_+\omega_t^{1/2} ∥ x t − x ∗ ∥ ∗ ≤ L ∗ ∥ θ t − θ ∗ ∥ ∗ ≤ L ∗ β + ω t 1/2 J X ( θ t ) J_{\mathcal{X}}(\theta_t) J X ( θ t ) ∥ x ~ t − x ~ ∗ ∥ 2 ≤ ∥ x t − x ∗ ∥ ∘ ∗ / ℓ ∗ ≤ L ∗ β + ω t 1 / 2 / ℓ ∗ ≤ L ∗ β + / ( ℓ ∗ Λ 0 ∗ ) ≤ ϱ ∗ / ( 2 D ∗ ℓ ∗ ) ≤ ∥ x ~ ∗ ∥ 2 / 2 \lVert\tilde{x}_t-\tilde{x}^*\rVert_2\le\lVert x_t-x^*\rVert_\circ^*/\ell_*\le L_*\beta_+\omega_t^{1/2}/\ell_*\le L_*\beta_+/(\ell_*\sqrt{\Lambda_0^*})\le\varrho_*/(2D_*\ell_*)\le\lVert\tilde{x}^*\rVert_2/2 ∥ x ~ t − x ~ ∗ ∥ 2 ≤ ∥ x t − x ∗ ∥ ∘ ∗ / ℓ ∗ ≤ L ∗ β + ω t 1/2 / ℓ ∗ ≤ L ∗ β + / ( ℓ ∗ Λ 0 ∗ ) ≤ ϱ ∗ / ( 2 D ∗ ℓ ∗ ) ≤ ∥ x ~ ∗ ∥ 2 /2 Λ 0 ∗ \Lambda_0^* Λ 0 ∗ ∥ x ~ ∗ ∥ 2 ≥ ϱ ∗ / ( D ∗ ℓ ∗ ) \lVert\tilde{x}^*\rVert_2\ge\varrho_*/(D_*\ell_*) ∥ x ~ ∗ ∥ 2 ≥ ϱ ∗ / ( D ∗ ℓ ∗ ) ⟨ e ~ ∗ , x ~ t ⟩ = ∥ x ~ ∗ ∥ 2 + ⟨ e ~ ∗ , x ~ t − x ~ ∗ ⟩ ≥ ∥ x ~ ∗ ∥ 2 − ∥ x ~ t − x ~ ∗ ∥ 2 \langle\tilde{e}^*,\tilde{x}_t\rangle=\lVert\tilde{x}^*\rVert_2+\langle\tilde{e}^*,\tilde{x}_t-\tilde{x}^*\rangle\ge\lVert\tilde{x}^*\rVert_2-\lVert\tilde{x}_t-\tilde{x}^*\rVert_2 ⟨ e ~ ∗ , x ~ t ⟩ = ∥ x ~ ∗ ∥ 2 + ⟨ e ~ ∗ , x ~ t − x ~ ∗ ⟩ ≥ ∥ x ~ ∗ ∥ 2 − ∥ x ~ t − x ~ ∗ ∥ 2
Size of the burn-in. By equation 5 and equation 12 (ℓ ∗ 2 ≤ M / 2 \ell_*^2\le M/2 ℓ ∗ 2 ≤ M /2 ϱ ∗ − 2 ≤ 2 / m \varrho_*^{-2}\le2/m ϱ ∗ − 2 ≤ 2/ m D ∗ 2 ≤ M / m D_*^2\le M/m D ∗ 2 ≤ M / m L ∗ ≤ 2 M / J ∗ L_*\le2M/J_* L ∗ ≤ 2 M / J ∗
Λ 0 ∗ ≤ max { M 2 m , 8 M β + 2 J ∗ 2 , 32 M 3 β + 2 m 2 J ∗ 2 } ; \Lambda_0^*\le\max\left\{\frac{M^2}{m},\frac{8M\beta_+^2}{J_*^2},\frac{32M^3\beta_+^2}{m^2J_*^2}\right\}; Λ 0 ∗ ≤ max { m M 2 , J ∗ 2 8 M β + 2 , m 2 J ∗ 2 32 M 3 β + 2 } ; for the unit ball and for every ellipsoid with its adapted norm, Λ 0 ⋆ = max { 2 , 64 β + 2 / J ⋆ 2 } \Lambda_0^\star= \max\{2,64\beta_+^2/J_\star^2\} Λ 0 ⋆ = max { 2 , 64 β + 2 / J ⋆ 2 } t 0 = O ~ ( d + d J ⋆ 2 Λ 0 ⋆ 2 / ( m 2 α 2 ) ) t_0 = \widetilde{O}\left(d+dJ_\star^2\Lambda_0^{\star2}/(m^2\alpha^2)\right) t 0 = O ( d + d J ⋆ 2 Λ 0 ⋆ 2 / ( m 2 α 2 ) )
t 0 = O ~ ( d + d M 4 J ⋆ 2 m 4 α 2 + d M 6 β + 4 m 6 J ⋆ 2 α 2 ) . t_0=\widetilde{O}\left(d+\frac{dM^4J_\star^2}{m^4\alpha^2}+\frac{dM^6\beta_+^4}{m^6J_\star^2\alpha^2}\right). t 0 = O ( d + m 4 α 2 d M 4 J ⋆ 2 + m 6 J ⋆ 2 α 2 d M 6 β + 4 ) . with a logarithmic dependence on 1 / ( d λ 0 ) 1/(d\lambda_0) 1/ ( d λ 0 )
Lemma D.9 (Crude bound). Recall c 1 = c e n g α / ( 2 d ) = m α / ( 12 J ⋆ d ) c_1=c_{\mathrm{eng}}\alpha/(2\sqrt{d})=m\alpha/(12J_\star\sqrt{d}) c 1 = c eng α / ( 2 d ) = m α / ( 12 J ⋆ d ) and let T 1 : = max ( 2 t 0 , 3 d ) T_1:=\max(2t_0,3d) T 1 := max ( 2 t 0 , 3 d ) On E \mathcal{E} E the engine equation 22 holds at every round t 0 ≤ t ≤ T t_0\le t\le T t 0 ≤ t ≤ T and
ω t − 1 ≥ c 1 t log ( t ) for all T 1 ≤ t ≤ T . (29) \omega_t^{-1}\ge c_1\sqrt{\frac{t}{\log(t)}}\qquad\text{for all }T_1\le t\le T. \tag*{(29)} ω t − 1 ≥ c 1 log ( t ) t for all T 1 ≤ t ≤ T . (29) Proof of Lemma D.9. For t ≥ t 0 t\ge t_0 t ≥ t 0 ℓ ⋆ β + ω t 1 / 2 ≤ ℓ ⋆ β + / Λ 0 ⋆ ≤ J ⋆ / 4 \ell_\star\beta_+ \omega_t^{1/2}\le\ell_\star\beta_+/\sqrt{\Lambda_0^\star}\le J_\star/4 ℓ ⋆ β + ω t 1/2 ≤ ℓ ⋆ β + / Λ 0 ⋆ ≤ J ⋆ /4 Proposition D.8 (i) and the second entry of Λ 0 ⋆ \Lambda_0^\star Λ 0 ⋆ t ≥ t 0 t\ge t_0 t ≥ t 0 (21) into equation (29) . We use the adapted coordinates. By the matrix determinant lemma, det ( V ~ t + 1 ) = det ( V ~ t ) ( 1 + x t T V t − 1 x t ) \det(\widetilde{V}_{t+1})=\det(\widetilde{V}_t)(1+x_t^\mathsf{T}V_t^{-1}x_t) det ( V t + 1 ) = det ( V t ) ( 1 + x t T V t − 1 x t ) t = t 0 , … , T ′ t=t_0,\ldots,T' t = t 0 , … , T ′ t 0 ≤ T ′ ≤ T t_0\le T'\le T t 0 ≤ T ′ ≤ T
∑ t = t 0 T ′ log ( 1 + x t T V t − 1 x t ) = log ( det ( V ~ T ′ + 1 ) det ( V ~ t 0 ) ) ≤ d log ( T ′ ) , \sum_{t=t_0}^{T'}\log(1+x_t^\mathsf{T}V_t^{-1}x_t)=\log\left(\frac{\det(\widetilde{V}_{T'+1})}{\det(\widetilde{V}_{t_0})}\right)\le d\log(T'), t = t 0 ∑ T ′ log ( 1 + x t T V t − 1 x t ) = log ( det ( V t 0 ) det ( V T ′ + 1 ) ) ≤ d log ( T ′ ) , since V ~ T ′ + 1 = V ~ t 0 + ∑ t = t 0 T ′ x ~ t x ~ t T ⪯ V ~ t 0 + ( T ′ − t 0 + 1 ) I \widetilde{V}_{T'+1}=\widetilde{V}_{t_0}+\sum_{t=t_0}^{T'}\widetilde{x}_t\widetilde{x}_t^\mathsf{T}\preceq\widetilde{V}_{t_0}+(T'-t_0+1)I V T ′ + 1 = V t 0 + ∑ t = t 0 T ′ x t x t T ⪯ V t 0 + ( T ′ − t 0 + 1 ) I ∥ x ~ s ∥ 2 ≤ 1 \|\widetilde{x}_s\|_2\le1 ∥ x s ∥ 2 ≤ 1 λ ~ t 0 , d ≥ 2 \widetilde{\lambda}_{t_0,d}\ge2 λ t 0 , d ≥ 2 (13) and Proposition D.8 (i), so that det ( V ~ T ′ + 1 ) / det ( V ~ t 0 ) = det ( I + V ~ t 0 − 1 / 2 ( V ~ T ′ + 1 − V ~ t 0 ) V ~ t 0 − 1 / 2 ) ≤ ( 1 + T ′ / 2 ) d ≤ T ′ d \det(\widetilde{V}_{T'+1})/\det(\widetilde{V}_{t_0})=\det\left(I+\widetilde{V}_{t_0}^{-1/2}(\widetilde{V}_{T'+1}-\widetilde{V}_{t_0})\widetilde{V}_{t_0}^{-1/2}\right)\le(1+T'/2)^d\le T'^d det ( V T ′ + 1 ) / det ( V t 0 ) = det ( I + V t 0 − 1/2 ( V T ′ + 1 − V t 0 ) V t 0 − 1/2 ) ≤ ( 1 + T ′ /2 ) d ≤ T ′ d T ′ ≥ 2 T'\ge2 T ′ ≥ 2 t ≥ t 0 t\ge t_0 t ≥ t 0 x t T V t − 1 x t ≤ 1 / 2 x_t^\mathsf{T}V_t^{-1}x_t\le1/2 x t T V t − 1 x t ≤ 1/2 Lemma D.5 (iv), as ω t − 1 ≥ Λ 0 ⋆ ≥ 2 D ⋆ 2 φ ⋆ 2 \omega_t^{-1}\ge\Lambda_0^\star\ge2D_\star^2\varphi_\star^2 ω t − 1 ≥ Λ 0 ⋆ ≥ 2 D ⋆ 2 φ ⋆ 2 log ( 1 + y ) ≥ y / 2 \log(1+y)\ge y/2 log ( 1 + y ) ≥ y /2 [ 0 , 1 ] [0,1] [ 0 , 1 ] ℓ ⋆ β + ω t 1 / 2 ≤ J ⋆ / 4 \ell_\star\beta_+\omega_t^{1/2}\le J_\star/4 ℓ ⋆ β + ω t 1/2 ≤ J ⋆ /4 t ≥ t 0 t\ge t_0 t ≥ t 0 Λ 0 ⋆ \Lambda_0^\star Λ 0 ⋆ ω \omega ω
( T ′ − t 0 + 1 ) c e n g 2 α 2 ω T ′ 2 2 ≤ 1 2 ∑ t = t 0 T ′ x t T V t − 1 x t ≤ d log ( T ′ ) . (T'-t_0+1)\frac{c_{\mathrm{eng}}^2\alpha^2\omega_{T'}^2}{2}\le\frac{1}{2}\sum_{t=t_0}^{T'}x_t^\mathsf{T}V_t^{-1}x_t\le d\log(T'). ( T ′ − t 0 + 1 ) 2 c eng 2 α 2 ω T ′ 2 ≤ 2 1 t = t 0 ∑ T ′ x t T V t − 1 x t ≤ d log ( T ′ ) . For T ′ ≥ T 1 = max ( 2 t 0 , 3 d ) T'\ge T_1=\max(2t_0,3d) T ′ ≥ T 1 = max ( 2 t 0 , 3 d ) T ′ − t 0 + 1 ≥ T ′ / 2 T'-t_0+1\ge T'/2 T ′ − t 0 + 1 ≥ T ′ /2 ω T ′ − 2 ≥ c e n g 2 α 2 T ′ / ( 4 d log ( T ′ ) ) = c 1 2 T ′ / log ( T ′ ) \omega_{T'}^{-2}\ge c_{\mathrm{eng}}^2\alpha^2T'/(4d\log(T'))=c_1^2T'/\log(T') ω T ′ − 2 ≥ c eng 2 α 2 T ′ / ( 4 d log ( T ′ )) = c 1 2 T ′ / log ( T ′ ) T ′ T' T ′ t t t (29) . □ \square □
The burn-in ladder
All burn-in thresholds are collected in one explicit chain t 0 t_0 t 0 Proposition D.8 ) → T 1 : = max ( 2 t 0 , 3 d ) → T ⋆ → T B \to T_1:=\max(2t_0,3d)\to T_\star\to T_B → T 1 := max ( 2 t 0 , 3 d ) → T ⋆ → T B
Definition D.10 (Burn-in ladder). With T 1 = max ( 2 t 0 , 3 d ) T_1=\max(2t_0,3d) T 1 = max ( 2 t 0 , 3 d ) c 1 = m α / ( 12 J ⋆ d ) c_1=m\alpha/(12J_\star\sqrt{d}) c 1 = m α / ( 12 J ⋆ d ) and C 2 = 3 L ⋆ 2 β + 2 / ( ℓ ⋆ 2 c 1 ) C_2=3L_\star^2\beta_+^2/(\ell_\star^2c_1) C 2 = 3 L ⋆ 2 β + 2 / ( ℓ ⋆ 2 c 1 ) as in equation (15) , define
T ⋆ : = min { t ≥ max ( T 1 , 20 ) : ∀ s ≥ t , 4 T 1 + r λ 0 ≤ L ⋆ 2 β + 2 ℓ ⋆ 2 c 1 s log ( s ) and C 2 s log ( s ) ≤ φ ⋆ 2 16 D ⋆ 2 φ ⋆ 2 } , T_\star:=\min\left\{t\ge\max(T_1,20):\ \forall s\ge t,\quad
4T_1+r\lambda_0\le\frac{L_\star^2\beta_+^2}{\ell_\star^2c_1}\sqrt{s\log(s)}
\ \text{and}\
C_2\sqrt{s\log(s)}\le\frac{\varphi_\star^2}{16D_\star^2\varphi_\star^2}\right\}, T ⋆ := min { t ≥ max ( T 1 , 20 ) : ∀ s ≥ t , 4 T 1 + r λ 0 ≤ ℓ ⋆ 2 c 1 L ⋆ 2 β + 2 s log ( s ) and C 2 s log ( s ) ≤ 16 D ⋆ 2 φ ⋆ 2 φ ⋆ 2 } , T B : = T ⋆ . T_B:=T_\star. T B := T ⋆ . The first inequality makes the pre-T 1 T_1 T 1
A : = 256 D ⋆ 4 φ ⋆ 4 C 2 2 ℓ ⋆ 4 , B : = ( 4 T 1 + r λ 0 ) 2 c 1 2 φ ⋆ 4 L ⋆ 4 β + 4 = ( 4 T 1 + r λ 0 ) 2 c e n g 2 α 2 φ ⋆ 4 4 d L ⋆ 4 β + 4 , A:=\frac{256D_\star^4\varphi_\star^4C_2^2}{\ell_\star^4},
\qquad
B:=\frac{(4T_1+r\lambda_0)^2c_1^2\varphi_\star^4}{L_\star^4\beta_+^4}
=\frac{(4T_1+r\lambda_0)^2c_{\mathrm{eng}}^2\alpha^2\varphi_\star^4}{4dL_\star^4\beta_+^4}, A := ℓ ⋆ 4 256 D ⋆ 4 φ ⋆ 4 C 2 2 , B := L ⋆ 4 β + 4 ( 4 T 1 + r λ 0 ) 2 c 1 2 φ ⋆ 4 = 4 d L ⋆ 4 β + 4 ( 4 T 1 + r λ 0 ) 2 c eng 2 α 2 φ ⋆ 4 , @-vesm the second inequality reads s / log ( s ) ≥ A s/\log(s) \ge A s / log ( s ) ≥ A s log ( s ) ≥ B s\log(s) \ge B s log ( s ) ≥ B s ≥ max { 3 A ′ log ( A ′ ) , B , 3 } s \ge\max\{3A'\log(A'), B, 3\} s ≥ max { 3 A ′ log ( A ′ ) , B , 3 } A ′ : = max { A , 3 } A' := \max\{A,3\} A ′ := max { A , 3 }
T ⋆ ≤ max { T 1 , 20 , 3 A ′ log ( A ′ ) , B } (30) T_{\star} \le\max\{T_1, 20, 3A'\log(A'), B\} \tag*{(30)} T ⋆ ≤ max { T 1 , 20 , 3 A ′ log ( A ′ ) , B } (30) Dependence on the horizon. T ⋆ T_{\star} T ⋆ T T T β + = α + β \beta_{+} = \alpha+ \beta β + = α + β (30) scales as β + 4 / α 2 \beta_{+}^{4}/\alpha^{2} β + 4 / α 2 ( d , M / m , J − 1 ) (d, M/m, J^{-1}) ( d , M / m , J − 1 )
T B = T ⋆ = O ( poly ( log T ) ) . T_{\mathrm{B}} = T_{\star} = O(\operatorname{poly}(\log T)). T B = T ⋆ = O ( poly ( log T )) . Proof of Theorem spectral growth
The engine equation (21) is a lower bound on x t T V t − 1 x t x_t^{\mathsf T}V_t^{-1}x_t x t T V t − 1 x t (29) is an upper bound on the sum of these quantities. To go from t / log ( t ) \sqrt{t}/\log(t) t / log ( t ) t \sqrt{t} t t t t doubling lemma shows that ω t − 1 \omega_t^{-1} ω t − 1 O ( ϑ 2 ) O(\vartheta^2) O ( ϑ 2 ) ϑ \vartheta ϑ ω t − 1 ≥ c 0 t \omega_t^{-1} \ge c_0\sqrt{t} ω t − 1 ≥ c 0 t
Epochs, band, and the capped potential
Epochs are measured on the inverse width ω t − 1 \omega_t^{-1} ω t − 1 V ~ \widetilde{V} V epoch start τ ≥ T ⋆ \tau\ge T_{\star} τ ≥ T ⋆ ϑ : = ω τ − 1 \vartheta:= \omega_{\tau}^{-1} ϑ := ω τ − 1 t ≥ τ t \ge\tau t ≥ τ ω t − 1 ≤ 2 ϑ \omega_t^{-1} \le2\vartheta ω t − 1 ≤ 2 ϑ ω ⋅ \omega_{\cdot} ω ⋅ ω t − 1 ≥ ϑ \omega_t^{-1} \ge\vartheta ω t − 1 ≥ ϑ t ≥ τ t \ge\tau t ≥ τ
ϑ ˉ : = ϑ D ∗ 2 φ ∗ 2 , so that 0 < ϑ ˉ ≤ λ ~ t , d for all t ≥ τ , (31) \bar{\vartheta} := \frac{\vartheta}{D_{\ast}^{2}\varphi_{\ast}^{2}},
\qquad\text{so that}\qquad
0 < \bar{\vartheta} \le\widetilde{\lambda}_{t,d}\quad\text{for all }t \ge\tau, \tag*{(31)} ϑ ˉ := D ∗ 2 φ ∗ 2 ϑ , so that 0 < ϑ ˉ ≤ λ t , d for all t ≥ τ , (31) because λ ~ t , d ≥ λ ~ τ , d ≥ ω τ − 1 / ( D ∗ 2 φ ∗ 2 ) \widetilde{\lambda}_{t,d} \ge\widetilde{\lambda}_{\tau,d} \ge\omega_{\tau}^{-1}/(D_{\ast}^{2}\varphi_{\ast}^{2}) λ t , d ≥ λ τ , d ≥ ω τ − 1 / ( D ∗ 2 φ ∗ 2 ) (13) ; ϑ ˉ \bar{\vartheta} ϑ ˉ Γ ≥ 1 \Gamma\ge1 Γ ≥ 1 (15) , define the band , the mid-scale set , and the band energy
B t : = { i ≥ 2 : λ ~ t , i < Γ ϑ ˉ } , J t : = { i ≥ 2 : λ ~ t , i ≥ Γ ϑ ˉ } , E t B : = ∑ i ∈ B t x ^ t , i 2 λ ~ t , i , \mathcal{B}_t := \{i \ge2 : \widetilde{\lambda}_{t,i} < \Gamma\bar{\vartheta}\},
\qquad
\mathcal{J}_t := \{i \ge2 : \widetilde{\lambda}_{t,i} \ge\Gamma\bar{\vartheta}\},
\qquad
E_t^{\mathrm B} := \sum_{i\in\mathcal{B}_t}\frac{\widehat{x}_{t,i}^{\,2}}{\widetilde{\lambda}_{t,i}}, B t := { i ≥ 2 : λ t , i < Γ ϑ ˉ } , J t := { i ≥ 2 : λ t , i ≥ Γ ϑ ˉ } , E t B := i ∈ B t ∑ λ t , i x t , i 2 , and, with L : = 8 Γ ϑ ˉ L := 8\Gamma\bar{\vartheta} L := 8Γ ϑ ˉ L ∗ L_{\ast} L ∗ logarithmically capped potential
Φ ϑ ˉ ( W ) : = tr h ( W ) = ∑ i = 1 d h ( λ i ( W ) ) , h ( λ ) : = ϑ ˉ log ( L ( λ + ϑ ˉ ) ϑ ˉ ( λ + L ) ) , W ⪰ 0 , (32) \Phi_{\bar{\vartheta}}(\mathbf{W}) := \operatorname{tr}h(\mathbf{W}) = \sum_{i=1}^{d}h(\lambda_i(\mathbf{W})),
\qquad
h(\lambda) := \bar{\vartheta}\log\left(\frac{L(\lambda+\bar{\vartheta})}{\bar{\vartheta}(\lambda+L)}\right),
\qquad
\mathbf{W} \succeq0, \tag*{(32)} Φ ϑ ˉ ( W ) := tr h ( W ) = i = 1 ∑ d h ( λ i ( W )) , h ( λ ) := ϑ ˉ log ( ϑ ˉ ( λ + L ) L ( λ + ϑ ˉ ) ) , W ⪰ 0 , (32) where W \mathbf{W} W V ~ t \widetilde{V}_t V t h h h h ( 0 ) = 0 h(0)=0 h ( 0 ) = 0 h ( ∞ ) = ϑ ˉ log ( L / ϑ ˉ ) = ϑ ˉ log ( 8 Γ ) h(\infty)=\bar{\vartheta}\log(L/\bar{\vartheta})=\bar{\vartheta}\log(8\Gamma) h ( ∞ ) = ϑ ˉ log ( L / ϑ ˉ ) = ϑ ˉ log ( 8Γ ) ϑ ˉ \bar{\vartheta} ϑ ˉ ϑ ˉ \bar{\vartheta} ϑ ˉ L L L
Lemma D.11 (Capacity and per-round gain of the capped potential). Let τ ≥ T ⋆ \tau\ge T_{\star} τ ≥ T ⋆ ϑ : = ω τ − 1 \vartheta:= \omega_{\tau}^{-1} ϑ := ω τ − 1 ϑ ˉ : = ϑ / ( D ∗ 2 φ ∗ 2 ) \bar{\vartheta} := \vartheta/(D_{\ast}^{2}\varphi_{\ast}^{2}) ϑ ˉ := ϑ / ( D ∗ 2 φ ∗ 2 ) L = 8 Γ ϑ ˉ L = 8\Gamma\bar{\vartheta} L = 8Γ ϑ ˉ
(i) For every W ⪰ 0 \mathbf{W} \succeq0 W ⪰ 0 0 ≤ Φ ϑ ˉ ( W ) ≤ d ϑ ˉ log ( 8 Γ ) 0 \le\Phi_{\bar{\vartheta}}(\mathbf{W}) \le d\bar{\vartheta}\log(8\Gamma) 0 ≤ Φ ϑ ˉ ( W ) ≤ d ϑ ˉ log ( 8Γ )
(ii) For every t ≥ τ t \ge\tau t ≥ τ Φ ϑ ˉ ( V ~ t + 1 ) − Φ ϑ ˉ ( V ~ t ) ≥ ϑ ˉ 4 E t B ≥ 0 \Phi_{\bar{\vartheta}}(\widetilde{V}_{t+1})-\Phi_{\bar{\vartheta}}(\widetilde{V}_t) \ge\frac{\bar{\vartheta}}{4}E_t^{\mathrm B} \ge0 Φ ϑ ˉ ( V t + 1 ) − Φ ϑ ˉ ( V t ) ≥ 4 ϑ ˉ E t B ≥ 0
Proof of Lemma D.11. Step 0: a determinant representation. Since
h ( λ ) = ϑ ˉ [ log ( λ + ϑ ˉ ) − log ( λ + L ) ] + ϑ ˉ log ( L / ϑ ˉ ) , h(\lambda)=\bar{\vartheta}\left[\log(\lambda+\bar{\vartheta})-\log(\lambda+L)\right]+\bar{\vartheta}\log(L/\bar{\vartheta}), h ( λ ) = ϑ ˉ [ log ( λ + ϑ ˉ ) − log ( λ + L ) ] + ϑ ˉ log ( L / ϑ ˉ ) , we have, for W ⪰ 0 \mathbf{W}\succeq0 W ⪰ 0
Φ ϑ ˉ ( W ) = ϑ ˉ [ log det ( W + ϑ ˉ I ) − log det ( W + L I ) ] + d ϑ ˉ log ( L / ϑ ˉ ) . \Phi_{\bar{\vartheta}}(\mathbf{W})=\bar{\vartheta}\left[\log\det(\mathbf{W}+\bar{\vartheta}\mathbf{I})-\log\det(\mathbf{W}+L\mathbf{I})\right]+d\bar{\vartheta}\log(L/\bar{\vartheta}). Φ ϑ ˉ ( W ) = ϑ ˉ [ log det ( W + ϑ ˉ I ) − log det ( W + L I ) ] + d ϑ ˉ log ( L / ϑ ˉ ) . Proof of (i). h h h [ 0 , ∞ ) [0,\infty) [ 0 , ∞ ) h ( 0 ) = 0 h(0)=0 h ( 0 ) = 0 ϑ ˉ < L \bar{\vartheta}<L ϑ ˉ < L h ( λ ) ≤ ϑ ˉ log ( L / ϑ ˉ ) = ϑ ˉ log ( 8 Γ ) h(\lambda)\le\bar{\vartheta}\log(L/\bar{\vartheta})=\bar{\vartheta}\log(8\Gamma) h ( λ ) ≤ ϑ ˉ log ( L / ϑ ˉ ) = ϑ ˉ log ( 8Γ ) λ ≥ 0 \lambda\ge0 λ ≥ 0 d d d
Proof of (ii). Fix t ≥ τ t\ge\tau t ≥ τ V ~ : = V ~ t \widetilde{V}:=\widetilde{V}_t V := V t x ~ : = x ~ t \widetilde{x}:=\widetilde{x}_t x := x t λ ~ i : = λ ~ t , i \widetilde{\lambda}_i:=\widetilde{\lambda}_{t,i} λ i := λ t , i x ~ i : = x ~ t , i \widetilde{x}_i:=\widetilde{x}_{t,i} x i := x t , i V ~ t + 1 = V ~ + x ~ x ~ ⊤ \widetilde{V}_{t+1}=\widetilde{V}+\widetilde{x}\widetilde{x}^{\top} V t + 1 = V + x x ⊤ ϑ ˉ ≤ λ ~ d \bar{\vartheta}\le\widetilde{\lambda}_d ϑ ˉ ≤ λ d λ ~ d ≥ 2 \widetilde{\lambda}_d\ge2 λ d ≥ 2 ω t − 1 ≥ Λ 0 ∗ ≥ 2 D ∗ 2 ℓ ∗ 2 \omega_t^{-1}\ge\Lambda_0^*\ge2D_*^2\ell_*^2 ω t − 1 ≥ Λ 0 ∗ ≥ 2 D ∗ 2 ℓ ∗ 2 t ≥ t 0 t\ge t_0 t ≥ t 0
a : = x ~ ⊤ ( V ~ + ϑ ˉ I ) − 1 x ~ = ∑ i = 1 d x ~ i 2 λ ~ i + ϑ ˉ , b : = x ~ ⊤ ( V ~ + L I ) − 1 x ~ = ∑ i = 1 d x ~ i 2 λ ~ i + L . a:=\widetilde{x}^{\top}(\widetilde{V}+\bar{\vartheta}I)^{-1}\widetilde{x}=\sum_{i=1}^{d}\frac{\widetilde{x}_i^2}{\widetilde{\lambda}_i+\bar{\vartheta}},\qquad b:=\widetilde{x}^{\top}(\widetilde{V}+LI)^{-1}\widetilde{x}=\sum_{i=1}^{d}\frac{\widetilde{x}_i^2}{\widetilde{\lambda}_i+L}. a := x ⊤ ( V + ϑ ˉ I ) − 1 x = i = 1 ∑ d λ i + ϑ ˉ x i 2 , b := x ⊤ ( V + L I ) − 1 x = i = 1 ∑ d λ i + L x i 2 . (a) Determinant lemma. By the matrix determinant lemma applied to V ~ + ϑ ˉ I \widetilde{V}+\bar{\vartheta}I V + ϑ ˉ I V ~ + L I \widetilde{V}+LI V + L I Φ ϑ ˉ ( V ~ + x ~ x ~ ⊤ ) − Φ ϑ ˉ ( V ~ ) = ϑ ˉ log ( ( 1 + a ) / ( 1 + b ) ) \Phi_{\bar{\vartheta}}(\widetilde{V}+\widetilde{x}\widetilde{x}^{\top})-\Phi_{\bar{\vartheta}}(\widetilde{V})=\bar{\vartheta}\log((1+a)/(1+b)) Φ ϑ ˉ ( V + x x ⊤ ) − Φ ϑ ˉ ( V ) = ϑ ˉ log (( 1 + a ) / ( 1 + b )) 0 ≤ b ≤ a 0\le b\le a 0 ≤ b ≤ a L > ϑ ˉ L>\bar{\vartheta} L > ϑ ˉ a ≤ x ~ ⊤ V ~ − 1 x ~ ≤ 1 / λ ~ d ≤ 1 / 2 a\le\widetilde{x}^{\top}\widetilde{V}^{-1}\widetilde{x}\le1/\widetilde{\lambda}_d\le1/2 a ≤ x ⊤ V − 1 x ≤ 1/ λ d ≤ 1/2 ( V ~ + ϑ ˉ I ) − 1 ⪯ V ~ − 1 (\widetilde{V}+\bar{\vartheta}I)^{-1}\preceq\widetilde{V}^{-1} ( V + ϑ ˉ I ) − 1 ⪯ V − 1 ∥ x ~ ∥ 2 ≤ 1 \|\widetilde{x}\|_2\le1 ∥ x ∥ 2 ≤ 1 λ ~ d ≥ 2 \widetilde{\lambda}_d\ge2 λ d ≥ 2
(b) Logarithm. For 0 ≤ b ≤ a ≤ 1 / 2 0\le b\le a\le1/2 0 ≤ b ≤ a ≤ 1/2 log ( ( 1 + a ) / ( 1 + b ) ) = log ( 1 + a − b 1 + b ) ≥ a − b ( 1 + b ) ( 1 + a − b 1 + b ) = a − b 1 + a ≥ 2 3 ( a − b ) \log((1+a)/(1+b))=\log\left(1+\frac{a-b}{1+b}\right)\ge\frac{a-b}{(1+b)(1+\frac{a-b}{1+b})}=\frac{a-b}{1+a}\ge\frac{2}{3}(a-b) log (( 1 + a ) / ( 1 + b )) = log ( 1 + 1 + b a − b ) ≥ ( 1 + b ) ( 1 + 1 + b a − b ) a − b = 1 + a a − b ≥ 3 2 ( a − b ) log ( 1 + y ) ≥ y / ( 1 + y ) \log(1+y)\ge y/(1+y) log ( 1 + y ) ≥ y / ( 1 + y ) y ≥ 0 y\ge0 y ≥ 0
(c) Term-by-term difference. Each summand of a − b a-b a − b
a − b = ∑ i = 1 d x ~ i 2 ( 1 λ ~ i + ϑ ˉ − 1 λ ~ i + L ) = ∑ i = 1 d ( L − ϑ ˉ ) x ~ i 2 ( λ ~ i + ϑ ˉ ) ( λ ~ i + L ) ≥ ∑ i ∈ B t ( L − ϑ ˉ ) x ~ i 2 ( λ ~ i + ϑ ˉ ) ( λ ~ i + L ) . a-b=\sum_{i=1}^{d}\widetilde{x}_i^2\left(\frac{1}{\widetilde{\lambda}_i+\bar{\vartheta}}-\frac{1}{\widetilde{\lambda}_i+L}\right)=\sum_{i=1}^{d}\frac{(L-\bar{\vartheta})\widetilde{x}_i^2}{(\widetilde{\lambda}_i+\bar{\vartheta})(\widetilde{\lambda}_i+L)}\ge\sum_{i\in B_t}\frac{(L-\bar{\vartheta})\widetilde{x}_i^2}{(\widetilde{\lambda}_i+\bar{\vartheta})(\widetilde{\lambda}_i+L)}. a − b = i = 1 ∑ d x i 2 ( λ i + ϑ ˉ 1 − λ i + L 1 ) = i = 1 ∑ d ( λ i + ϑ ˉ ) ( λ i + L ) ( L − ϑ ˉ ) x i 2 ≥ i ∈ B t ∑ ( λ i + ϑ ˉ ) ( λ i + L ) ( L − ϑ ˉ ) x i 2 . (d) Band coordinates. For i ∈ B t i\in B_t i ∈ B t ϑ ˉ ≤ λ ~ d ≤ λ ~ i < Γ ϑ ˉ \bar{\vartheta}\le\widetilde{\lambda}_d\le\widetilde{\lambda}_i<\Gamma\bar{\vartheta} ϑ ˉ ≤ λ d ≤ λ i < Γ ϑ ˉ λ ~ i + ϑ ˉ ≤ 2 λ ~ i \widetilde{\lambda}_i+\bar{\vartheta}\le2\widetilde{\lambda}_i λ i + ϑ ˉ ≤ 2 λ i λ ~ i + L < ( Γ + 8 Γ ) ϑ ˉ = 9 Γ ϑ ˉ \widetilde{\lambda}_i+L<(\Gamma+8\Gamma)\bar{\vartheta}=9\Gamma\bar{\vartheta} λ i + L < ( Γ + 8Γ ) ϑ ˉ = 9Γ ϑ ˉ L − ϑ ˉ ≥ 7 Γ ϑ ˉ L-\bar{\vartheta}\ge7\Gamma\bar{\vartheta} L − ϑ ˉ ≥ 7Γ ϑ ˉ
( L − ϑ ˉ ) x ~ i 2 ( λ ~ i + ϑ ˉ ) ( λ ~ i + L ) ≥ 7 Γ ϑ ˉ x ~ i 2 2 λ ~ i ⋅ 9 Γ ϑ ˉ = 7 18 x ~ i 2 λ ~ i . \frac{(L-\bar{\vartheta})\widetilde{x}_i^2}{(\widetilde{\lambda}_i+\bar{\vartheta})(\widetilde{\lambda}_i+L)}
\ge\frac{7\Gamma\bar{\vartheta}\widetilde{x}_i^2}{2\widetilde{\lambda}_i\cdot9\Gamma\bar{\vartheta}}
=\frac{7}{18}\frac{\widetilde{x}_i^2}{\widetilde{\lambda}_i}. ( λ i + ϑ ˉ ) ( λ i + L ) ( L − ϑ ˉ ) x i 2 ≥ 2 λ i ⋅ 9Γ ϑ ˉ 7Γ ϑ ˉ x i 2 = 18 7 λ i x i 2 . (e) Conclusion. Combining (a)–(d), Φ ϑ ˉ ( V ~ t + 1 ) − Φ ϑ ˉ ( V ~ t ) ≥ ϑ ˉ ⋅ 2 3 ⋅ 7 18 ∑ i ∈ B t x ~ i 2 / λ ~ i = 7 27 ϑ ˉ E t B ≥ ϑ ˉ 4 E t B \Phi_{\bar{\vartheta}}(\widetilde{V}_{t+1})-\Phi_{\bar{\vartheta}}(\widetilde{V}_t)\ge\bar{\vartheta}\cdot\frac{2}{3}\cdot\frac{7}{18}\sum_{i\in B_t}\widetilde{x}_i^2/\widetilde{\lambda}_i=\frac{7}{27}\bar{\vartheta}E_t^B\ge\frac{\bar{\vartheta}}{4}E_t^B Φ ϑ ˉ ( V t + 1 ) − Φ ϑ ˉ ( V t ) ≥ ϑ ˉ ⋅ 3 2 ⋅ 18 7 ∑ i ∈ B t x i 2 / λ i = 27 7 ϑ ˉ E t B ≥ 4 ϑ ˉ E t B
Leakage decomposition
Lemma D.12 (Leakage decomposition). Let τ ≥ T ⋆ \tau\ge T_\star τ ≥ T ⋆ ϑ = ω τ − 1 \vartheta=\omega_\tau^{-1} ϑ = ω τ − 1 ϑ ˉ = ϑ / ( D ⋆ 2 ℓ ⋆ 2 ) \bar{\vartheta}=\vartheta/(D_\star^2\ell_\star^2) ϑ ˉ = ϑ / ( D ⋆ 2 ℓ ⋆ 2 ) B t B_t B t E t B E_t^B E t B max ( τ , T ⋆ ) ≤ t ≤ T \max(\tau,T_\star)\le t\le T max ( τ , T ⋆ ) ≤ t ≤ T ϑ ≤ ω t − 1 ≤ 2 ϑ \vartheta\le\omega_t^{-1}\le2\vartheta ϑ ≤ ω t − 1 ≤ 2 ϑ E \mathcal{E} E
x t ⊤ V t − 1 x t ≤ E t B + 24 t + c e n g 2 α 2 ω t 2 10 . (33) x_t^{\top}V_t^{-1}x_t\le E_t^B+\frac{24}{t}+\frac{c_{\mathrm{eng}}^2\alpha^2\omega_t^2}{10}. \tag*{(33)} x t ⊤ V t − 1 x t ≤ E t B + t 24 + 10 c eng 2 α 2 ω t 2 . (33) Proof of Lemma D.12. Step 0: spectral preliminaries. We first establish, for every T ⋆ ≤ t ≤ T T_\star\le t\le T T ⋆ ≤ t ≤ T E \mathcal{E} E
(P1) Q ~ t + r λ 0 ≤ C 2 t log ( t ) , where Q ~ t : = ∑ s = 1 t − 1 ∥ x ~ s − x ⋆ ∥ 2 2 , \text{(P1)}\quad\widetilde{Q}_t+r\lambda_0\le C_2\sqrt{t\log(t)},\qquad\text{where }\widetilde{Q}_t:=\sum_{s=1}^{t-1}\|\widetilde{x}_s-x^\star\|_2^2, (P1) Q t + r λ 0 ≤ C 2 t log ( t ) , where Q t := s = 1 ∑ t − 1 ∥ x s − x ⋆ ∥ 2 2 , (P2) λ ~ t , 1 ≥ ∥ x ~ ⋆ ∥ 2 2 t 8 , λ ~ t , 1 − λ ~ t , 2 ≥ ∥ x ~ ⋆ ∥ 2 2 t 16 , (P3) ∥ x ~ ⋆ ∥ 2 2 sin 2 ϕ ~ t ⋅ λ ~ t , d ≤ 16 C 2 2 log ( t ) . (34) \text{(P2)}\quad\widetilde{\lambda}_{t,1}\ge\frac{\|\widetilde{x}^\star\|_2^2t}{8},\qquad\widetilde{\lambda}_{t,1}-\widetilde{\lambda}_{t,2}\ge\frac{\|\widetilde{x}^\star\|_2^2t}{16},\qquad\text{(P3)}\quad\|\widetilde{x}^\star\|_2^2\sin^2\widetilde{\phi}_t\cdot\widetilde{\lambda}_{t,d}\le16C_2^2\log(t). \tag*{(34)} (P2) λ t , 1 ≥ 8 ∥ x ⋆ ∥ 2 2 t , λ t , 1 − λ t , 2 ≥ 16 ∥ x ⋆ ∥ 2 2 t , (P3) ∥ x ⋆ ∥ 2 2 sin 2 ϕ t ⋅ λ t , d ≤ 16 C 2 2 log ( t ) . (34) Note that the chord sum Q ~ t \widetilde{Q}_t Q t ∑ s < t ∥ P ~ ⊥ x ~ s ∥ 2 2 \sum_{s<t}\|\widetilde{P}^{\perp}\widetilde{x}_s\|_2^2 ∑ s < t ∥ P ⊥ x s ∥ 2 2 ∥ P ~ ⊥ x ~ s ∥ 2 2 = ∥ P ~ ⊥ ( x ~ s − x ~ ⋆ ) ∥ 2 2 ≤ ∥ x ~ s − x ~ ⋆ ∥ 2 2 \|\widetilde{P}^{\perp}\widetilde{x}_s\|_2^2=\|\widetilde{P}^{\perp}(\widetilde{x}_s-\widetilde{x}^\star)\|_2^2\le\|\widetilde{x}_s-\widetilde{x}^\star\|_2^2 ∥ P ⊥ x s ∥ 2 2 = ∥ P ⊥ ( x s − x ⋆ ) ∥ 2 2 ≤ ∥ x s − x ⋆ ∥ 2 2
(P1): a per-round chord bound, summed against the crude bound. For t 0 ≤ s ≤ T t_0\le s\le T t 0 ≤ s ≤ T E \mathcal{E} E ∥ x ~ s − x ~ ⋆ ∥ 2 2 ≤ L ∗ 2 β s 2 + ω s / ℓ ∗ 2 \|\widetilde{x}_s-\widetilde{x}^\star\|_2^2\le L_*^2\beta_s^2+\omega_s/\ell_*^2 ∥ x s − x ⋆ ∥ 2 2 ≤ L ∗ 2 β s 2 + ω s / ℓ ∗ 2 Q ~ t \widetilde{Q}_t Q t T 1 T_1 T 1 ∥ x ~ s − x ~ ⋆ ∥ 2 2 ≤ 4 \|\widetilde{x}_s-\widetilde{x}^\star\|_2^2\le4 ∥ x s − x ⋆ ∥ 2 2 ≤ 4 T 1 T_1 T 1 B 2 ( 1 ) \mathbb{B}_2(1) B 2 ( 1 ) ω s \omega_s ω s
∑ s = T 1 t − 1 log ( s ) s ≤ 2 t log ( t ) . \sum_{s=T_1}^{t-1}\sqrt{\frac{\log(s)}{s}}\le2\sqrt{t\log(t)}. s = T 1 ∑ t − 1 s log ( s ) ≤ 2 t log ( t ) . Q ~ t + r λ 0 ≤ 4 T 1 + r λ 0 + L ∗ 2 β 2 ℓ ∗ 2 c 1 ∑ s = T 1 t − 1 log ( s ) s ≤ 4 T 1 + r λ 0 + 2 L ∗ 2 β 2 ℓ ∗ 2 c 1 t log ( t ) ≤ C 2 t log ( t ) , \widetilde{Q}_t+r\lambda_0\le4T_1+r\lambda_0+\frac{L_*^2\beta^2}{\ell_*^2c_1}\sum_{s=T_1}^{t-1}\sqrt{\frac{\log(s)}{s}}\le4T_1+r\lambda_0+\frac{2L_*^2\beta^2}{\ell_*^2c_1}\sqrt{t\log(t)}\le C_2\sqrt{t\log(t)}, Q t + r λ 0 ≤ 4 T 1 + r λ 0 + ℓ ∗ 2 c 1 L ∗ 2 β 2 s = T 1 ∑ t − 1 s log ( s ) ≤ 4 T 1 + r λ 0 + ℓ ∗ 2 c 1 2 L ∗ 2 β 2 t log ( t ) ≤ C 2 t log ( t ) , the last step by the first defining inequality of T ⋆ T_\star T ⋆ t ≥ T ⋆ t\ge T_\star t ≥ T ⋆
(P2): Rayleigh quotient of e ~ ⋆ \widetilde{e}^\star e ⋆ Let B ~ t : = U ~ ⊤ V ~ t U ~ \widetilde{B}_t:=\widetilde{U}^{\top}\widetilde{V}_t\widetilde{U} B t := U ⊤ V t U V ~ t \widetilde{V}_t V t ( e ~ ⋆ ) ⊥ (\widetilde{e}^\star)^\perp ( e ⋆ ) ⊥ U ~ \widetilde{U} U ( e ~ ⋆ ) ⊥ (\widetilde{e}^\star)^\perp ( e ⋆ ) ⊥ tr B ~ t = tr ( U ~ ⊤ R ~ U ~ ) + ∑ s < t ∥ P ~ ⊥ x ~ s ∥ 2 ≤ r λ 0 + Q ~ t \operatorname{tr}\widetilde{B}_t=\operatorname{tr}(\widetilde{U}^{\top}\widetilde{R}\widetilde{U})+\sum_{s<t}\|\widetilde{P}^{\perp}\widetilde{x}_s\|^2\le r\lambda_0+\widetilde{Q}_t tr B t = tr ( U ⊤ R U ) + ∑ s < t ∥ P ⊥ x s ∥ 2 ≤ r λ 0 + Q t λ ~ t , 2 ≤ λ max ( B ~ t ) ≤ tr B ~ t ≤ Q ~ t + r λ 0 ≤ C 2 t log ( t ) \widetilde{\lambda}_{t,2}\le\lambda_{\max}(\widetilde{B}_t)\le\operatorname{tr}\widetilde{B}_t\le\widetilde{Q}_t+r\lambda_0\le C_2\sqrt{t\log(t)} λ t , 2 ≤ λ m a x ( B t ) ≤ tr B t ≤ Q t + r λ 0 ≤ C 2 t log ( t ) (28) , ⟨ e ~ ⋆ , x ~ s ⟩ ≥ ∥ x ~ ⋆ ∥ 2 / 2 \langle\widetilde{e}^\star,\widetilde{x}_s\rangle\ge\|\widetilde{x}^\star\|_2/2 ⟨ e ⋆ , x s ⟩ ≥ ∥ x ⋆ ∥ 2 /2 s ≥ t 0 s\ge t_0 s ≥ t 0 e ~ ⋆ \widetilde{e}^\star e ⋆
λ ~ t , 1 ≥ ( e ~ ⋆ ) ⊤ V ~ t e ~ ⋆ ≥ ∑ s < t ⟨ e ~ ⋆ , x ~ s ⟩ 2 ≥ ∥ x ~ ⋆ ∥ 2 2 ( t − t 0 ) 4 ≥ ∥ x ~ ⋆ ∥ 2 2 t 8 , \widetilde{\lambda}_{t,1}\ge(\widetilde{e}^\star)^\top\widetilde{V}_t\widetilde{e}^\star\ge\sum_{s<t}\langle\widetilde{e}^\star,\widetilde{x}_s\rangle^2\ge\frac{\|\widetilde{x}^\star\|_2^2(t-t_0)}{4}\ge\frac{\|\widetilde{x}^\star\|_2^2t}{8}, λ t , 1 ≥ ( e ⋆ ) ⊤ V t e ⋆ ≥ s < t ∑ ⟨ e ⋆ , x s ⟩ 2 ≥ 4 ∥ x ⋆ ∥ 2 2 ( t − t 0 ) ≥ 8 ∥ x ⋆ ∥ 2 2 t , as R ~ ⪰ 0 \widetilde{R}\succeq0 R ⪰ 0 t ≥ T 1 ≥ 2 t 0 t\ge T_1\ge2t_0 t ≥ T 1 ≥ 2 t 0 λ ~ t , 1 − λ ~ t , 2 ≥ ∥ x ~ ⋆ ∥ 2 2 t / 8 − C 2 t log ( t ) ≥ ∥ x ~ ⋆ ∥ 2 2 t / 16 \widetilde{\lambda}_{t,1}-\widetilde{\lambda}_{t,2}\ge\|\widetilde{x}^\star\|_2^2t/8-C_2\sqrt{t\log(t)}\ge\|\widetilde{x}^\star\|_2^2t/16 λ t , 1 − λ t , 2 ≥ ∥ x ⋆ ∥ 2 2 t /8 − C 2 t log ( t ) ≥ ∥ x ⋆ ∥ 2 2 t /16 T ⋆ T_\star T ⋆ ∥ x ~ ⋆ ∥ 2 2 ≥ ϱ ⋆ 2 / ( D ⋆ 2 φ ⋆ 2 ) \|\widetilde{x}^\star\|_2^2\ge\varrho_\star^2/(D_\star^2\varphi_\star^2) ∥ x ⋆ ∥ 2 2 ≥ ϱ ⋆ 2 / ( D ⋆ 2 φ ⋆ 2 ) (13) .
(P3): misalignment of the top eigenvector. For the unit vector e ~ ⋆ \widetilde{e}^\star e ⋆ ( e ~ ⋆ ) ⊤ V ~ t e ~ ⋆ = tr V ~ t − tr B ~ t (\widetilde{e}^\star)^\top\widetilde{V}_t\widetilde{e}^\star=\operatorname{tr}\widetilde{V}_t-\operatorname{tr}\widetilde{B}_t ( e ⋆ ) ⊤ V t e ⋆ = tr V t − tr B t ( e ~ ⋆ ) ⊤ R ~ e ~ ⋆ + ∑ s < t ⟨ e ~ ⋆ , x ~ s ⟩ 2 (\widetilde{e}^\star)^\top\widetilde{R}\widetilde{e}^\star+\sum_{s<t}\langle\widetilde{e}^\star,\widetilde{x}_s\rangle^2 ( e ⋆ ) ⊤ R e ⋆ + ∑ s < t ⟨ e ⋆ , x s ⟩ 2 ∥ x ~ s ∥ 2 = ⟨ e ~ ⋆ , x ~ s ⟩ 2 + ∥ P ~ ⊥ x ~ s ∥ 2 \|\widetilde{x}_s\|^2=\langle\widetilde{e}^\star,\widetilde{x}_s\rangle^2+\|\widetilde{P}^{\perp}\widetilde{x}_s\|^2 ∥ x s ∥ 2 = ⟨ e ⋆ , x s ⟩ 2 + ∥ P ⊥ x s ∥ 2 ( e ~ ⋆ ) ⊤ V ~ t e ~ ⋆ ≥ λ ~ t , 1 − Q ~ t − r λ 0 (\widetilde{e}^\star)^\top\widetilde{V}_t\widetilde{e}^\star\ge\widetilde{\lambda}_{t,1}-\widetilde{Q}_t-r\lambda_0 ( e ⋆ ) ⊤ V t e ⋆ ≥ λ t , 1 − Q t − r λ 0 e ~ ⋆ \widetilde{e}^\star e ⋆ ( e ~ ⋆ ) ⊤ V ~ t e ~ ⋆ ≤ cos 2 ϕ ~ t λ ~ t , 1 + sin 2 ϕ ~ t λ ~ t , 2 (\widetilde{e}^\star)^\top\widetilde{V}_t\widetilde{e}^\star\le\cos^2\widetilde{\phi}_t\widetilde{\lambda}_{t,1}+\sin^2\widetilde{\phi}_t\widetilde{\lambda}_{t,2} ( e ⋆ ) ⊤ V t e ⋆ ≤ cos 2 ϕ t λ t , 1 + sin 2 ϕ t λ t , 2
sin 2 ϕ ~ t ( λ ~ t , 1 − λ ~ t , 2 ) ≤ Q ~ t + r λ 0 , so sin 2 ϕ ~ t ≤ 16 ( Q ~ t + r λ 0 ) ∥ x ~ ⋆ ∥ 2 2 t ≤ 16 C 2 ∥ x ~ ⋆ ∥ 2 2 log ( t ) t \sin^2\widetilde{\phi}_t(\widetilde{\lambda}_{t,1}-\widetilde{\lambda}_{t,2})\le\widetilde{Q}_t+r\lambda_0,\qquad\text{so}\qquad\sin^2\widetilde{\phi}_t\le\frac{16(\widetilde{Q}_t+r\lambda_0)}{\|\widetilde{x}^\star\|_2^2t}\le\frac{16C_2}{\|\widetilde{x}^\star\|_2^2}\sqrt{\frac{\log(t)}{t}} sin 2 ϕ t ( λ t , 1 − λ t , 2 ) ≤ Q t + r λ 0 , so sin 2 ϕ t ≤ ∥ x ⋆ ∥ 2 2 t 16 ( Q t + r λ 0 ) ≤ ∥ x ⋆ ∥ 2 2 16 C 2 t log ( t ) by (P1)–(P2). Multiplying by λ ~ t , d ≤ λ ~ t , 2 ≤ C 2 t log ( t ) \widetilde{\lambda}_{t,d}\le\widetilde{\lambda}_{t,2}\le C_2\sqrt{t\log(t)} λ t , d ≤ λ t , 2 ≤ C 2 t log ( t ) d ≥ 2 d\ge2 d ≥ 2
Step 1: the decomposition. Split the energy according to { 1 } ∪ J t ∪ B t \{1\}\cup\mathcal{J}_t\cup\mathcal{B}_t { 1 } ∪ J t ∪ B t x t ⊤ V t − 1 x t x_t^\top V_t^{-1}x_t x t ⊤ V t − 1 x t
x t ⊤ V t − 1 x t = x ~ t ⊤ V ~ t − 1 x ~ t = x ~ t , 1 2 λ ~ t , 1 + ∑ i ∈ J t x ~ t , i 2 λ ~ t , i + E t B . x_t^\top V_t^{-1}x_t=\widetilde{x}_t^\top\widetilde{V}_t^{-1}\widetilde{x}_t=\frac{\widetilde{x}_{t,1}^2}{\widetilde{\lambda}_{t,1}}+\sum_{i\in\mathcal{J}_t}\frac{\widetilde{x}_{t,i}^2}{\widetilde{\lambda}_{t,i}}+E_t^{\mathcal{B}}. x t ⊤ V t − 1 x t = x t ⊤ V t − 1 x t = λ t , 1 x t , 1 2 + i ∈ J t ∑ λ t , i x t , i 2 + E t B . (a) Top direction. By equation (28) and (P2),
x ~ t , 1 2 λ ~ t , 1 ≤ ∥ x ~ t ∥ 2 2 λ ~ t , 1 ≤ 9 ∥ x ~ ⋆ ∥ 2 2 / 4 ∥ x ~ ⋆ ∥ 2 2 t / 8 = 18 t ≤ 24 t . \frac{\widetilde{x}_{t,1}^2}{\widetilde{\lambda}_{t,1}}\le\frac{\|\widetilde{x}_t\|_2^2}{\widetilde{\lambda}_{t,1}}\le\frac{9\|\widetilde{x}^\star\|_2^2/4}{\|\widetilde{x}^\star\|_2^2t/8}=\frac{18}{t}\le\frac{24}{t}. λ t , 1 x t , 1 2 ≤ λ t , 1 ∥ x t ∥ 2 2 ≤ ∥ x ⋆ ∥ 2 2 t /8 9∥ x ⋆ ∥ 2 2 /4 = t 18 ≤ t 24 . (b) Mid-scale coordinates. Write x ~ t , i = ⟨ x ~ t − x ~ ⋆ , v ~ t , i ⟩ + ⟨ x ~ ⋆ , v ~ t , i ⟩ \widetilde{x}_{t,i}=\langle\widetilde{x}_t-\widetilde{x}^\star,\widetilde{v}_{t,i}\rangle+\langle\widetilde{x}^\star,\widetilde{v}_{t,i}\rangle x t , i = ⟨ x t − x ⋆ , v t , i ⟩ + ⟨ x ⋆ , v t , i ⟩ i ∈ J t i\in\mathcal{J}_t i ∈ J t ( u + v ) 2 ≤ 2 u 2 + 2 v 2 (u+v)^2\le2u^2+2v^2 ( u + v ) 2 ≤ 2 u 2 + 2 v 2
q ~ t J : = ∑ i ∈ J t x ~ t , i 2 ≤ 2 ∥ x ~ t − x ~ ⋆ ∥ 2 2 + 2 ∑ i ≥ 2 ⟨ x ~ ⋆ , v ~ t , i ⟩ 2 = 2 ∥ x ~ t − x ~ ⋆ ∥ 2 2 + 2 ∥ x ~ ⋆ ∥ 2 2 sin 2 ϕ ~ t , \widetilde{q}_t^{\mathcal{J}}:=\sum_{i\in\mathcal{J}_t}\widetilde{x}_{t,i}^2\le2\|\widetilde{x}_t-\widetilde{x}^\star\|_2^2+2\sum_{i\ge2}\langle\widetilde{x}^\star,\widetilde{v}_{t,i}\rangle^2=2\|\widetilde{x}_t-\widetilde{x}^\star\|_2^2+2\|\widetilde{x}^\star\|_2^2\sin^2\widetilde{\phi}_t, q t J := i ∈ J t ∑ x t , i 2 ≤ 2∥ x t − x ⋆ ∥ 2 2 + 2 i ≥ 2 ∑ ⟨ x ⋆ , v t , i ⟩ 2 = 2∥ x t − x ⋆ ∥ 2 2 + 2∥ x ⋆ ∥ 2 2 sin 2 ϕ t , since ∑ i ≥ 2 ⟨ x ~ ⋆ , v ~ t , i ⟩ 2 = ∥ x ~ ⋆ ∥ 2 2 − ⟨ x ~ ⋆ , v ~ t , 1 ⟩ 2 = ∥ x ~ ⋆ ∥ 2 2 sin 2 ϕ ~ t \sum_{i\ge2}\langle\widetilde{x}^\star,\widetilde{v}_{t,i}\rangle^2=\|\widetilde{x}^\star\|_2^2-\langle\widetilde{x}^\star,\widetilde{v}_{t,1}\rangle^2=\|\widetilde{x}^\star\|_2^2\sin^2\widetilde{\phi}_t ∑ i ≥ 2 ⟨ x ⋆ , v t , i ⟩ 2 = ∥ x ⋆ ∥ 2 2 − ⟨ x ⋆ , v t , 1 ⟩ 2 = ∥ x ⋆ ∥ 2 2 sin 2 ϕ t min i ∈ J t λ ~ t , i ≥ Γ ‾ ϑ = Γ ϑ / ( D ⋆ 2 φ ⋆ 2 ) ≥ Γ / ( 2 D ⋆ 2 φ ⋆ 2 ω t ) \min_{i\in\mathcal{J}_t}\widetilde{\lambda}_{t,i}\ge\overline{\Gamma}\vartheta=\Gamma\vartheta/(D_\star^2\varphi_\star^2)\ge\Gamma/(2D_\star^2\varphi_\star^2\omega_t) min i ∈ J t λ t , i ≥ Γ ϑ = Γ ϑ / ( D ⋆ 2 φ ⋆ 2 ) ≥ Γ/ ( 2 D ⋆ 2 φ ⋆ 2 ω t ) ( ϑ ≥ ω t / 2 ) (\vartheta\ge\omega_t/2) ( ϑ ≥ ω t /2 )
∑ i ∈ J t x ~ t , i 2 λ ~ t , i ≤ 2 D ⋆ 2 φ ⋆ 2 ω t q ~ t J Γ ≤ 4 D ⋆ 2 φ ⋆ 2 ω t ∥ x ~ t − x ~ ⋆ ∥ 2 2 Γ + 4 D ⋆ 2 φ ⋆ 2 ω t ∥ x ~ ⋆ ∥ 2 2 sin 2 ϕ ~ t Γ . \sum_{i\in\mathcal{J}_t}\frac{\widetilde{x}_{t,i}^2}{\widetilde{\lambda}_{t,i}}\le\frac{2D_\star^2\varphi_\star^2\omega_t\widetilde{q}_t^{\mathcal{J}}}{\Gamma}\le\frac{4D_\star^2\varphi_\star^2\omega_t\|\widetilde{x}_t-\widetilde{x}^\star\|_2^2}{\Gamma}+\frac{4D_\star^2\varphi_\star^2\omega_t\|\widetilde{x}^\star\|_2^2\sin^2\widetilde{\phi}_t}{\Gamma}. i ∈ J t ∑ λ t , i x t , i 2 ≤ Γ 2 D ⋆ 2 φ ⋆ 2 ω t q t J ≤ Γ 4 D ⋆ 2 φ ⋆ 2 ω t ∥ x t − x ⋆ ∥ 2 2 + Γ 4 D ⋆ 2 φ ⋆ 2 ω t ∥ x ⋆ ∥ 2 2 sin 2 ϕ t . Estimation part. By Proposition D.8(v) and the first fact in equation (17) ,
4 D ⋆ 2 φ ⋆ 2 ω t ∥ x ~ t − x ~ ⋆ ∥ 2 2 Γ ≤ 4 D ⋆ 2 L ⋆ 2 β 2 ω t 2 Γ ≤ c e n g 2 α 2 ω t 2 20 . \frac{4D_\star^2\varphi_\star^2\omega_t\|\widetilde{x}_t-\widetilde{x}^\star\|_2^2}{\Gamma}\le\frac{4D_\star^2L_\star^2\beta^2\omega_t^2}{\Gamma}\le\frac{c_{\mathrm{eng}}^2\alpha^2\omega_t^2}{20}. Γ 4 D ⋆ 2 φ ⋆ 2 ω t ∥ x t − x ⋆ ∥ 2 2 ≤ Γ 4 D ⋆ 2 L ⋆ 2 β 2 ω t 2 ≤ 20 c eng 2 α 2 ω t 2 . Misalignment part. By (P3), log ( t ) ≤ log ( T ) \log(t)\le\log(T) log ( t ) ≤ log ( T ) λ ~ t , d ≥ ω t − 1 / ( D ⋆ 2 φ ⋆ 2 ) \widetilde{\lambda}_{t,d}\ge\omega_t^{-1}/(D_\star^2\varphi_\star^2) λ t , d ≥ ω t − 1 / ( D ⋆ 2 φ ⋆ 2 ) (17) ,
4 D ⋆ 2 φ ⋆ 2 ω t ∥ x ~ ⋆ ∥ 2 2 sin 2 ϕ ~ t Γ ≤ 64 D ⋆ 2 φ ⋆ 2 C 2 2 log ( T ) ω t Γ λ ~ t , d ≤ 64 D ⋆ 4 φ ⋆ 4 C 2 2 log ( T ) ω t 2 Γ ≤ c e n g 2 α 2 ω t 2 20 . \frac{4D_\star^2\varphi_\star^2\omega_t\|\widetilde{x}^\star\|_2^2\sin^2\widetilde{\phi}_t}{\Gamma}\le\frac{64D_\star^2\varphi_\star^2C_2^2\log(T)\omega_t}{\Gamma\widetilde{\lambda}_{t,d}}\le\frac{64D_\star^4\varphi_\star^4C_2^2\log(T)\omega_t^2}{\Gamma}\le\frac{c_{\mathrm{eng}}^2\alpha^2\omega_t^2}{20}. Γ 4 D ⋆ 2 φ ⋆ 2 ω t ∥ x ⋆ ∥ 2 2 sin 2 ϕ t ≤ Γ λ t , d 64 D ⋆ 2 φ ⋆ 2 C 2 2 log ( T ) ω t ≤ Γ 64 D ⋆ 4 φ ⋆ 4 C 2 2 log ( T ) ω t 2 ≤ 20 c eng 2 α 2 ω t 2 . Conclusion. Adding (a) and the two parts of (b) to E t B E_t^{\mathcal{B}} E t B (33) , with 1 20 + 1 20 = 1 10 \frac{1}{20}+\frac{1}{20}=\frac{1}{10} 20 1 + 20 1 = 10 1
Corollary D.13 (Band energy capture). Under the hypotheses of Lemma D.12, combining equation (33) with the engine equation 22,
E t B ≥ ( 1 − 1 10 ) c e n g 2 α 2 ω t 2 − 24 t ≥ 9 c e n g 2 α 2 40 ϑ 2 − 24 t . E_t^B \ge\left(1-\frac{1}{10}\right)c_{\mathrm{eng}}^2\alpha^2\omega_t^2-\frac{24}{t}\ge\frac{9c_{\mathrm{eng}}^2\alpha^2}{40\vartheta^2}-\frac{24}{t}. E t B ≥ ( 1 − 10 1 ) c eng 2 α 2 ω t 2 − t 24 ≥ 40 ϑ 2 9 c eng 2 α 2 − t 24 . the last step since ω t − 1 ≤ 2 ϑ \omega_t^{-1}\le2\vartheta ω t − 1 ≤ 2 ϑ
Doubling
Lemma D.14 (Doubling). Let τ ≥ T ⋆ \tau\ge T_\star τ ≥ T ⋆ ϑ : = ω τ − 1 \vartheta:=\omega_\tau^{-1} ϑ := ω τ − 1 K = 40 d log ( 8 Γ ) / ( c e n g 2 α 2 ) K=40d\log(8\Gamma)/(c_{\mathrm{eng}}^2\alpha^2) K = 40 d log ( 8Γ ) / ( c eng 2 α 2 ) (15) , and suppose τ + ⌈ K ϑ 2 ⌉ ≤ T \tau+\lceil K\vartheta^2\rceil\le T τ + ⌈ K ϑ 2 ⌉ ≤ T E \mathcal{E} E
ω τ + ⌈ K ϑ 2 ⌉ − 1 ≥ min ( 2 ϑ , c e n g α τ 32 ) . \omega_{\tau+\lceil K\vartheta^2\rceil}^{-1}\ge\min\left(2\vartheta,\frac{c_{\mathrm{eng}}\alpha\sqrt{\tau}}{32}\right). ω τ + ⌈ K ϑ 2 ⌉ − 1 ≥ min ( 2 ϑ , 32 c eng α τ ) . Proof of Lemma D.14. Set N : = ⌈ K ϑ 2 ⌉ N:=\lceil K\vartheta^2\rceil N := ⌈ K ϑ 2 ⌉ ω τ + N − 1 < min ( 2 ϑ , c e n g α τ / 32 ) \omega_{\tau+N}^{-1}<\min(2\vartheta,c_{\mathrm{eng}}\alpha\sqrt{\tau}/32) ω τ + N − 1 < min ( 2 ϑ , c eng α τ /32 ) t ∈ [ τ , τ + N ) t\in[\tau,\tau+N) t ∈ [ τ , τ + N ) ω t − 1 ≤ ω τ + N − 1 < 2 ϑ \omega_t^{-1}\le\omega_{\tau+N}^{-1}<2\vartheta ω t − 1 ≤ ω τ + N − 1 < 2 ϑ τ > 1024 ϑ 2 / ( c e n g 2 α 2 ) \tau>1024\vartheta^2/(c_{\mathrm{eng}}^2\alpha^2) τ > 1024 ϑ 2 / ( c eng 2 α 2 ) 24 / t ≤ 24 / τ < 24 c e n g 2 α 2 / ( 1024 ϑ 2 ) ≤ c e n g 2 α 2 / ( 40 ϑ 2 ) 24/t\le24/\tau<24c_{\mathrm{eng}}^2\alpha^2/(1024\vartheta^2)\le c_{\mathrm{eng}}^2\alpha^2/(40\vartheta^2) 24/ t ≤ 24/ τ < 24 c eng 2 α 2 / ( 1024 ϑ 2 ) ≤ c eng 2 α 2 / ( 40 ϑ 2 ) t ≥ τ t\ge\tau t ≥ τ t ∈ [ τ , τ + N ) t\in[\tau,\tau+N) t ∈ [ τ , τ + N )
E t B ≥ 9 c e n g 2 α 2 40 ϑ 2 − c e n g 2 α 2 40 ϑ 2 = c e n g 2 α 2 5 ϑ 2 . E_t^B\ge\frac{9c_{\mathrm{eng}}^2\alpha^2}{40\vartheta^2}-\frac{c_{\mathrm{eng}}^2\alpha^2}{40\vartheta^2}=\frac{c_{\mathrm{eng}}^2\alpha^2}{5\vartheta^2}. E t B ≥ 40 ϑ 2 9 c eng 2 α 2 − 40 ϑ 2 c eng 2 α 2 = 5 ϑ 2 c eng 2 α 2 . By Lemma D.11(ii), each such round deposits at least ϑ ˉ 4 ⋅ c e n g 2 α 2 5 ϑ 2 = ϑ ˉ c e n g 2 α 2 20 ϑ 2 \frac{\bar\vartheta}{4}\cdot\frac{c_{\mathrm{eng}}^2\alpha^2}{5\vartheta^2}=\frac{\bar\vartheta c_{\mathrm{eng}}^2\alpha^2}{20\vartheta^2} 4 ϑ ˉ ⋅ 5 ϑ 2 c eng 2 α 2 = 20 ϑ 2 ϑ ˉ c eng 2 α 2
Φ ϑ ˉ ( V ~ τ + N ) − Φ ϑ ˉ ( V ~ τ ) ≥ N ϑ ˉ c e n g 2 α 2 20 ϑ 2 ≥ K ϑ 2 ϑ ˉ c e n g 2 α 2 20 ϑ 2 = 2 d ϑ ˉ log ( 8 Γ ) . \Phi_{\bar\vartheta}(\widetilde V_{\tau+N})-\Phi_{\bar\vartheta}(\widetilde V_\tau)\ge N\frac{\bar\vartheta c_{\mathrm{eng}}^2\alpha^2}{20\vartheta^2}\ge K\vartheta^2\frac{\bar\vartheta c_{\mathrm{eng}}^2\alpha^2}{20\vartheta^2}=2d\bar\vartheta\log(8\Gamma). Φ ϑ ˉ ( V τ + N ) − Φ ϑ ˉ ( V τ ) ≥ N 20 ϑ 2 ϑ ˉ c eng 2 α 2 ≥ K ϑ 2 20 ϑ 2 ϑ ˉ c eng 2 α 2 = 2 d ϑ ˉ log ( 8Γ ) . But by Lemma D.11(i) the left side is at most d ϑ ˉ log ( 8 Γ ) d\bar\vartheta\log(8\Gamma) d ϑ ˉ log ( 8Γ ) ϑ ˉ \bar\vartheta ϑ ˉ K K K D ∗ D_\ast D ∗ ℓ ∗ \ell_\ast ℓ ∗ d d d d ϑ ˉ log ( 8 Γ ) d\bar\vartheta\log(8\Gamma) d ϑ ˉ log ( 8Γ ) α 2 \alpha^2 α 2
Proof of the Main Theorem
Lemma D.15 (Initialisation window). On E \mathcal{E} E ω t − 1 ≥ c 0 t \omega_t^{-1}\ge c_0\sqrt{t} ω t − 1 ≥ c 0 t t ∈ [ T B , min ( 9 T B , T ) ] t\in[T_B,\min(9T_B,T)] t ∈ [ T B , min ( 9 T B , T )]
Proof of Lemma D.15. For t ≥ T B = T ⋆ ≥ T 1 t\ge T_B=T_\star\ge T_1 t ≥ T B = T ⋆ ≥ T 1 ω t − 1 ≥ c 1 t / log ( t ) \omega_t^{-1}\ge c_1\sqrt{t/\log(t)} ω t − 1 ≥ c 1 t / log ( t ) t ≤ 9 T ⋆ t\le9T_\star t ≤ 9 T ⋆ t / log ( t ) ≥ t / log ( 9 T ⋆ ) \sqrt{t/\log(t)}\ge\sqrt{t}/\sqrt{\log(9T_\star)} t / log ( t ) ≥ t / log ( 9 T ⋆ ) ω t − 1 ≥ c 1 t / log ( 9 T ⋆ ) ≥ c 0 t \omega_t^{-1}\ge c_1\sqrt{t}/\sqrt{\log(9T_\star)}\ge c_0\sqrt{t} ω t − 1 ≥ c 1 t / log ( 9 T ⋆ ) ≥ c 0 t c 0 c_0 c 0
Proposition D.16 (Induction). On E \mathcal{E} E ω t − 1 ≥ c 0 t \omega_t^{-1}\ge c_0\sqrt{t} ω t − 1 ≥ c 0 t T B ≤ t ≤ T T_B\le t\le T T B ≤ t ≤ T
Proof of Proposition D.16. Suppose not, and let t † : = min { t ∈ [ T B , T ] : ω t − 1 < c 0 t } t^\dagger:=\min\{t\in[T_B,T]:\omega_t^{-1}<c_0\sqrt{t}\} t † := min { t ∈ [ T B , T ] : ω t − 1 < c 0 t } t † > 9 T B t^\dagger>9T_B t † > 9 T B ϑ : = ω t † − 1 < c 0 t † \vartheta:=\omega_{t^\dagger}^{-1}<c_0\sqrt{t^\dagger} ϑ := ω t † − 1 < c 0 t † N : = ⌈ K ϑ 2 ⌉ N:=\lceil K\vartheta^2\rceil N := ⌈ K ϑ 2 ⌉ K ϑ 2 < K c 0 2 t † ≤ t † / 16 K\vartheta^2<Kc_0^2t^\dagger\le t^\dagger/16 K ϑ 2 < K c 0 2 t † ≤ t † /16 c 0 ≤ 1 / ( 4 K ) c_0\le1/(4\sqrt K) c 0 ≤ 1/ ( 4 K ) N ≤ t † / 16 + 1 ≤ t † / 8 N\le t^\dagger/16+1\le t^\dagger/8 N ≤ t † /16 + 1 ≤ t † /8 t † ≥ 20 ≥ 16 t^\dagger\ge20\ge16 t † ≥ 20 ≥ 16 τ : = t † − N ≥ 7 T B ≥ T ⋆ \tau:=t^\dagger-N\ge7T_B\ge T_\star τ := t † − N ≥ 7 T B ≥ T ⋆ t † t^\dagger t † ω τ − 1 ≥ c 0 τ \omega_\tau^{-1}\ge c_0\sqrt{\tau} ω τ − 1 ≥ c 0 τ ω τ − 1 ≤ ω t † − 1 = ϑ \omega_\tau^{-1}\le\omega_{t^\dagger}^{-1}=\vartheta ω τ − 1 ≤ ω t † − 1 = ϑ τ \tau τ ϑ ′ : = ω τ − 1 ≤ ϑ \vartheta':=\omega_\tau^{-1}\le\vartheta ϑ ′ := ω τ − 1 ≤ ϑ ⌈ K ϑ ′ 2 ⌉ ≤ N \lceil K\vartheta'^2\rceil\le N ⌈ K ϑ ′2 ⌉ ≤ N τ + ⌈ K ϑ ′ 2 ⌉ ≤ t † ≤ T \tau+\lceil K\vartheta'^2\rceil\le t^\dagger\le T τ + ⌈ K ϑ ′2 ⌉ ≤ t † ≤ T
ϑ = ω t † − 1 ≥ ω τ + ⌈ K ϑ ′ 2 ⌉ − 1 ≥ min ( 2 ϑ ′ , c e n g α τ 32 ) . \vartheta=\omega_{t^\dagger}^{-1}\ge\omega_{\tau+\lceil K\vartheta'^2\rceil}^{-1}\ge\min\left(2\vartheta',\frac{c_{\mathrm{eng}}\alpha\sqrt{\tau}}{32}\right). ϑ = ω t † − 1 ≥ ω τ + ⌈ K ϑ ′2 ⌉ − 1 ≥ min ( 2 ϑ ′ , 32 c eng α τ ) . Two cases. If the minimum is 2 ϑ ′ 2\vartheta' 2 ϑ ′ ϑ ≥ 2 ϑ ′ ≥ 2 c 0 τ ≥ 2 c 0 t † / 8 > c 0 t † \vartheta\ge2\vartheta'\ge2c_0\sqrt{\tau}\ge2c_0\sqrt{t^\dagger/8}>c_0\sqrt{t^\dagger} ϑ ≥ 2 ϑ ′ ≥ 2 c 0 τ ≥ 2 c 0 t † /8 > c 0 t † ϑ < c 0 t † \vartheta<c_0\sqrt{t^\dagger} ϑ < c 0 t † c e n g α τ / 32 c_{\mathrm{eng}}\alpha\sqrt{\tau}/32 c eng α τ /32 ϑ ≥ ( c e n g α / 32 ) t † / 8 ≥ c 0 t † \vartheta\ge(c_{\mathrm{eng}}\alpha/32)\sqrt{t^\dagger/8}\ge c_0\sqrt{t^\dagger} ϑ ≥ ( c eng α /32 ) t † /8 ≥ c 0 t †
c 0 ≤ 1 4 K = c e n g α 4 40 d log ( 8 Γ ) ≤ c e n g α 4 160 < c e n g α 32 8 7 c_0\le\frac{1}{4\sqrt K}=\frac{c_{\mathrm{eng}}\alpha}{4\sqrt{40d\log(8\Gamma)}}\le\frac{c_{\mathrm{eng}}\alpha}{4\sqrt{160}}<\frac{c_{\mathrm{eng}}\alpha}{32}\sqrt{\frac{8}{7}} c 0 ≤ 4 K 1 = 4 40 d log ( 8Γ ) c eng α ≤ 4 160 c eng α < 32 c eng α 7 8 (using log ( 8 Γ ) ≥ log ( 8 ) ≥ 2 \log(8\Gamma)\ge\log(8)\ge2 log ( 8Γ ) ≥ log ( 8 ) ≥ 2 d ≥ 2 d\ge2 d ≥ 2 4 160 > 32 8 / 7 4\sqrt{160}>32\sqrt{8/7} 4 160 > 32 8/7 ϑ \vartheta ϑ
With this in hand, the proof of Theorem 4.3 is straightforward.
Theorem 4.3 (Spectral Growth). Let Assumptions (A1) and (A2) hold, let E \mathcal{E} E β \beta β T B = O ( poly ( log T ) ) T_B = O(\operatorname{poly}(\log T)) T B = O ( poly ( log T )) E \mathcal{E} E 1 − δ 1-\delta 1 − δ T B ≤ t ≤ T T_B \le t \le T T B ≤ t ≤ T
ω t = max u ∈ B 2 d ( 1 ) ∥ V t − 1 / 2 u ∥ ∗ 2 ≤ 1 c 0 t , where c 0 = Ω ( m α J ⋆ d ) . \omega_t = \max_{u \in\mathbb{B}_2^d(1)} \lVert V_t^{-1/2}u\rVert_*^2 \le\frac{1}{c_0\sqrt{t}}, \qquad\text{where} \qquad c_0 = \Omega\left(\frac{m\alpha}{J_\star\sqrt{d}}\right). ω t = u ∈ B 2 d ( 1 ) max ∥ V t − 1/2 u ∥ ∗ 2 ≤ c 0 t 1 , where c 0 = Ω ( J ⋆ d m α ) . Proof of Theorem 4.3. By Proposition D.16, ω t − 1 ≥ c 0 t \omega_t^{-1} \ge c_0\sqrt{t} ω t − 1 ≥ c 0 t E \mathcal{E} E T B ≤ t ≤ T T_B \le t \le T T B ≤ t ≤ T ω t \omega_t ω t Lemma D.5 , this is a statement about the original problem, and it is the first display by Proposition D.2 (ii). The three consequences are Proposition D.8 (ii)–(iv) with ω t − 1 ≥ c 0 t \omega_t^{-1} \ge c_0\sqrt{t} ω t − 1 ≥ c 0 t E \mathcal{E} E 1 − δ 1-\delta 1 − δ
1 4 K = c e n g α 640 d log ( 8 Γ ) and c 1 log ( 9 T ⋆ ) = c e n g α 4 d log ( 9 T ⋆ ) , \frac{1}{4\sqrt{K}} = \frac{c_{\mathrm{eng}}\alpha}{\sqrt{640d\log(8\Gamma)}} \qquad\text{and} \qquad\frac{c_1}{\sqrt{\log(9T_\star)}} = \frac{c_{\mathrm{eng}}\alpha}{\sqrt{4d\log(9T_\star)}}, 4 K 1 = 640 d log ( 8Γ ) c eng α and log ( 9 T ⋆ ) c 1 = 4 d log ( 9 T ⋆ ) c eng α , so
c 0 ≥ c e n g α 640 d ( log ( 8 Γ ) + log ( 9 T ⋆ ) ) ≥ m α 152 J ⋆ d log ( 8 Γ ) + log ( 9 T ⋆ ) . c_0 \ge\frac{c_{\mathrm{eng}}\alpha}{\sqrt{640d\left(\log(8\Gamma)+\log(9T_\star)\right)}} \ge\frac{m\alpha}{152J_\star\sqrt{d}\sqrt{\log(8\Gamma)+\log(9T_\star)}}. c 0 ≥ 640 d ( log ( 8Γ ) + log ( 9 T ⋆ ) ) c eng α ≥ 152 J ⋆ d log ( 8Γ ) + log ( 9 T ⋆ ) m α . where, by Definitions D.6 and D.10, the factor log ( 8 Γ ) + log ( 9 T ⋆ ) = O ( log log T ) \log(8\Gamma)+\log(9T_\star)=O(\log\log T) log ( 8Γ ) + log ( 9 T ⋆ ) = O ( log log T ) T T T □ \square □
Regret of alpha-LinUCB
Regret decomposition
Lemma E.1 (Per-round regret). On the event E \mathcal{E} E T B ≤ t ≤ T T_B \le t \le T T B ≤ t ≤ T r t : = J ⋆ − ⟨ θ ⋆ , x t ⟩ r_t := J_\star-\langle\theta^\star,x_t\rangle r t := J ⋆ − ⟨ θ ⋆ , x t ⟩
r t ≤ D X 2 ( θ ⋆ , θ t ) J ⋆ . r_t \le\frac{D_{\mathcal{X}}^2(\theta^\star,\theta_t)}{J_\star}. r t ≤ J ⋆ D X 2 ( θ ⋆ , θ t ) . Proof of Lemma E.1. Fix T B ≤ t ≤ T T_B \le t \le T T B ≤ t ≤ T ∇ J X 2 ( θ t ) = 2 J X ( θ t ) x t \nabla J_{\mathcal{X}}^2(\theta_t)=2J_{\mathcal{X}}(\theta_t)x_t ∇ J X 2 ( θ t ) = 2 J X ( θ t ) x t ⟨ θ t , x t ⟩ = J X ( θ t ) \langle\theta_t,x_t\rangle=J_{\mathcal{X}}(\theta_t) ⟨ θ t , x t ⟩ = J X ( θ t ) Proposition D.8 (iv), J X ( θ t ) ≥ 1 2 J ⋆ J_{\mathcal{X}}(\theta_t)\ge\frac{1}{2}J_\star J X ( θ t ) ≥ 2 1 J ⋆ t ≥ T B ≥ t 0 t\ge T_B\ge t_0 t ≥ T B ≥ t 0 ⟨ θ ⋆ , x t ⟩ ≤ J ⋆ \langle\theta^\star,x_t\rangle\le J_\star ⟨ θ ⋆ , x t ⟩ ≤ J ⋆ x t ∈ X x_t\in\mathcal{X} x t ∈ X
D X 2 ( θ ⋆ , θ t ) J ⋆ − r t = J ⋆ − J X 2 ( θ t ) J ⋆ − 2 J X ( θ t ) J ⋆ ⟨ x t , θ ⋆ − θ t ⟩ − J ⋆ + ⟨ θ ⋆ , x t ⟩ = J X 2 ( θ t ) J ⋆ − ( 2 J X ( θ t ) J ⋆ − 1 ) ⟨ θ ⋆ , x t ⟩ ( ⟨ θ t , x t ⟩ = J X ( θ t ) ) ≥ J X 2 ( θ t ) J ⋆ − ( 2 J X ( θ t ) J ⋆ − 1 ) J ⋆ ( J X ( θ t ) ≥ 1 2 J ⋆ , ⟨ θ ⋆ , x t ⟩ ≤ J ⋆ ) = ( J X ( θ t ) − J ⋆ ) 2 J ⋆ ≥ 0. \begin{aligned}
\frac{D_{\mathcal{X}}^2(\theta^\star,\theta_t)}{J_\star}-r_t
=J_\star-\frac{J_{\mathcal{X}}^2(\theta_t)}{J_\star}-\frac{2J_{\mathcal{X}}(\theta_t)}{J_\star}\langle x_t,\theta^\star-\theta_t\rangle-J_\star+\langle\theta^\star,x_t\rangle\\
&=\frac{J_{\mathcal{X}}^2(\theta_t)}{J_\star}-\left(\frac{2J_{\mathcal{X}}(\theta_t)}{J_\star}-1\right)\langle\theta^\star,x_t\rangle
\qquad(\langle\theta_t,x_t\rangle=J_{\mathcal{X}}(\theta_t)) \\
&\ge\frac{J_{\mathcal{X}}^2(\theta_t)}{J_\star}-\left(\frac{2J_{\mathcal{X}}(\theta_t)}{J_\star}-1\right)J_\star
\qquad(J_{\mathcal{X}}(\theta_t)\ge\tfrac{1}{2}J_\star,\ \langle\theta^\star,x_t\rangle\le J_\star) \\
&=\frac{\left(J_{\mathcal{X}}(\theta_t)-J_\star\right)^2}{J_\star}\ge0.
\end{aligned} J ⋆ D X 2 ( θ ⋆ , θ t ) − r t = J ⋆ − J ⋆ J X 2 ( θ t ) − J ⋆ 2 J X ( θ t ) ⟨ x t , θ ⋆ − θ t ⟩ − J ⋆ + ⟨ θ ⋆ , x t ⟩ = J ⋆ J X 2 ( θ t ) − ( J ⋆ 2 J X ( θ t ) − 1 ) ⟨ θ ⋆ , x t ⟩ (⟨ θ t , x t ⟩ = J X ( θ t )) ≥ J ⋆ J X 2 ( θ t ) − ( J ⋆ 2 J X ( θ t ) − 1 ) J ⋆ ( J X ( θ t ) ≥ 2 1 J ⋆ , ⟨ θ ⋆ , x t ⟩ ≤ J ⋆ ) = J ⋆ ( J X ( θ t ) − J ⋆ ) 2 ≥ 0. □ \square □
Proof of thm:alpha-UCB
Theorem 4.2 (α \alpha α E \mathcal{E} E 1 − δ 1-\delta 1 − δ
R T = O ( M m α 2 + β 2 α d T ) . \mathcal{R}_T = O\left(\frac{M}{m}\frac{\alpha^2+\beta^2}{\alpha}\sqrt{dT}\right). R T = O ( m M α α 2 + β 2 d T ) . Proof of Theorem 4.2. We have R T = ∑ t = 1 T r t \mathcal{R}_T=\sum_{t=1}^{T}r_t R T = ∑ t = 1 T r t r t = J ⋆ − ⟨ θ ⋆ , x t ⟩ ≥ 0 r_t=J_\star-\langle\theta^\star,x_t\rangle\ge0 r t = J ⋆ − ⟨ θ ⋆ , x t ⟩ ≥ 0 t < T B t<T_B t < T B
r t = ⟨ θ ⋆ , x ⋆ − x t ⟩ ≤ ∥ θ ⋆ ∥ ∗ ∥ x ⋆ − x t ∥ ∘ ∗ ≤ 2 ℓ ⋆ ∥ θ ⋆ ∥ ∗ ≤ 2 ℓ ⋆ J ⋆ ϱ ⋆ . r_t=\langle\theta^\star,x^\star-x_t\rangle\le\lVert\theta^\star\rVert_*\lVert x^\star-x_t\rVert_\circ^*\le2\ell_\star\lVert\theta^\star\rVert_*\le\frac{2\ell_\star J_\star}{\varrho_\star}. r t = ⟨ θ ⋆ , x ⋆ − x t ⟩ ≤ ∥ θ ⋆ ∥ ∗ ∥ x ⋆ − x t ∥ ∘ ∗ ≤ 2 ℓ ⋆ ∥ θ ⋆ ∥ ∗ ≤ ϱ ⋆ 2 ℓ ⋆ J ⋆ . since ∥ x ∥ ∗ ≤ ℓ ∗ \lVert x\rVert_* \le\ell_* ∥ x ∥ ∗ ≤ ℓ ∗ X \mathcal{X} X ϱ ∗ ∥ θ ∗ ∥ ∗ ≤ J X ( θ ∗ ) = J ∗ \varrho_* \lVert\theta^*\rVert_* \le J_{\mathcal{X}}(\theta^*) = J_* ϱ ∗ ∥ θ ∗ ∥ ∗ ≤ J X ( θ ∗ ) = J ∗ E \mathcal{E} E T B ≤ t ≤ T T_B \le t \le T T B ≤ t ≤ T
r t ≤ D X 2 ( θ ∗ , θ t ) J ∗ ≤ M 2 J ∗ ∥ θ ∗ − θ t ∥ ∗ 2 r_t \le\frac{D_{\mathcal{X}}^2(\theta^*,\theta_t)}{J_*} \le\frac{M}{2J_*}\lVert\theta^*-\theta_t\rVert_*^2 r t ≤ J ∗ D X 2 ( θ ∗ , θ t ) ≤ 2 J ∗ M ∥ θ ∗ − θ t ∥ ∗ 2 (Lemma E.1 and equation (1) )
≤ M J ∗ ( ∥ θ ∗ − θ ^ t ∥ ∗ 2 + ∥ θ ^ t − θ t ∥ ∗ 2 ) \le\frac{M}{J_*}\left(\lVert\theta^*-\hat{\theta}_t\rVert_*^2+\lVert\hat{\theta}_t-\theta_t\rVert_*^2\right) ≤ J ∗ M ( ∥ θ ∗ − θ ^ t ∥ ∗ 2 + ∥ θ ^ t − θ t ∥ ∗ 2 ) (( a + b ) 2 ≤ 2 a 2 + 2 b 2 (a+b)^2 \le2a^2+2b^2 ( a + b ) 2 ≤ 2 a 2 + 2 b 2
≤ M J ∗ ( α 2 + β 2 ) ω t \le\frac{M}{J_*}(\alpha^2+\beta^2)\omega_t ≤ J ∗ M ( α 2 + β 2 ) ω t (Proposition D.2 (ii)),
where the last step uses θ ∗ ∈ C t ( β ) \theta^* \in C_t(\beta) θ ∗ ∈ C t ( β ) E \mathcal{E} E θ t ∈ C t ( α ) \theta_t \in C_t(\alpha) θ t ∈ C t ( α ) Proposition D.2 (ii)). Summing,
R T ≤ 2 ℓ ∗ J ∗ ϱ ∗ T B + M ( α 2 + β 2 ) J ∗ ∑ t = T B T ω t ≤ 2 ℓ ∗ J ∗ ϱ ∗ T B + M ( α 2 + β 2 ) J ∗ c 0 ∑ t = T B T 1 t \begin{aligned}
R_T \le\frac{2\ell_*J_*}{\varrho_*}T_B+\frac{M(\alpha^2+\beta^2)}{J_*}\sum_{t=T_B}^{T}\omega_t \\
&\le\frac{2\ell_*J_*}{\varrho_*}T_B+\frac{M(\alpha^2+\beta^2)}{J_*c_0}\sum_{t=T_B}^{T}\frac{1}{\sqrt{t}}
\end{aligned} R T ≤ ϱ ∗ 2 ℓ ∗ J ∗ T B + J ∗ M ( α 2 + β 2 ) t = T B ∑ T ω t ≤ ϱ ∗ 2 ℓ ∗ J ∗ T B + J ∗ c 0 M ( α 2 + β 2 ) t = T B ∑ T t 1 (Theorem 4.3, t ≥ T B t\ge T_B t ≥ T B
≤ 2 ℓ ∗ J ∗ ϱ ∗ T B + 2 M ( α 2 + β 2 ) J ∗ c 0 T \le\frac{2\ell_*J_*}{\varrho_*}T_B+\frac{2M(\alpha^2+\beta^2)}{J_*c_0}\sqrt{T} ≤ ϱ ∗ 2 ℓ ∗ J ∗ T B + J ∗ c 0 2 M ( α 2 + β 2 ) T (∑ t ≤ T t − 1 / 2 ≤ 2 T \sum_{t\le T}t^{-1/2}\le2\sqrt{T} ∑ t ≤ T t − 1/2 ≤ 2 T
= O ( M m d α 2 + β 2 α T ) . =O\left(\frac{M}{m}\sqrt{d}\frac{\alpha^2+\beta^2}{\alpha}\sqrt{T}\right). = O ( m M d α α 2 + β 2 T ) . The last step uses c 0 ≳ m α J ∗ d c_0\gtrsim\frac{m\alpha}{J_*\sqrt{d}} c 0 ≳ J ∗ d m α O ( M m d α 2 + β 2 α T ) O\left(\frac{M}{m}\sqrt{d}\frac{\alpha^2+\beta^2}{\alpha}\sqrt{T}\right) O ( m M d α α 2 + β 2 T ) T B = O ( poly ( log T ) ) T_B=O(\operatorname{poly}(\log T)) T B = O ( poly ( log T )) Definition D.10 ). □
