Introduction
Online gradient descent is classically understood as an external-regret minimizer [1 , 2 , 3 ] . Against a sequence of linear losses, its iterates compete with every fixed comparator x ′ ∈ K x' \in K x ′ ∈ K [4 , 5 ] . From the viewpoint of swap regret and the notion of hindsight rationality, however, external regret is only the simplest notion out of a wide spectrum. External regret compares x t x_t x t x t x_t x t ϕ ( x t ) ∈ K \phi(x_t) \in K ϕ ( x t ) ∈ K [6 , 7 , 8 ] .
Recent work makes clear that gradient dynamics control more than fixed comparators, although the guarantees appear in different forms. In first-order equilibrium theory, Ahunbay [ [9 ] studies infinitesimal strategy modifications generated by continuous vector fields. In the coarse adversarial setting, the fields certified by projected gradient dynamics are conservative gradient fields with the appropriate tangency to the action set. In normal-form games, Ahunbay and Bichler [ [10 ] specialize this picture to a tractable linear–quadratic family and obtain semicoarse correlated equilibria through LP-based guarantees. In parallel, Cai, Daskalakis, Luo, Wei, and Zheng [ [11 ] showed that online gradient descent and mirror descent control deviations generated by proximal operators of weakly convex functions. As discussed by Cai et al. [ [11 ] and Ahunbay [ [9 ] , these lines of work have substantial overlap but are not comparable in general, reflecting the different deviation classes and guarantees they study.
In this paper, we study Φ \Phi Φ ϕ : K → K \phi: K \to K ϕ : K → K D ϕ ( x ) : = x − ϕ ( x ) D_{\phi}(x) := x - \phi(x) D ϕ ( x ) := x − ϕ ( x ) O ( T ) O(\sqrt{T}) O ( T ) D ϕ = ∇ Ψ D_{\phi} = \nabla\Psi D ϕ = ∇Ψ ϕ ( x t ) \phi(x_t) ϕ ( x t ) [9 ] to displacement fields generated by feasible endomorphisms. Working directly with endomorphisms, however, removes the tangent-projection machinery and allows the boundary geometry to be handled directly through feasibility, convexity, and normal cones. This yields explicit finite-time bounds for feasible endomorphisms.
A simple example reveals the geometry behind this formulation. Consider an affine deviation ϕ ( x ) = A x + b \phi(x) = Ax+b ϕ ( x ) = A x + b D ϕ ( x ) = ( I − A ) x − b D_{\phi}(x) = (I-A)x-b D ϕ ( x ) = ( I − A ) x − b I − A I-A I − A [10 ] , Cai et al. [ [11 ] are finite maps with exact displacement fields. The example suggests that the relevant property is not linearity itself, but exactness of the displacement.
The same structure appears in proximal deviations. If F = f + ι K F = f + \iota_K F = f + ι K x − prox F ( x ) = ∇ M F ( x ) x-\operatorname{prox}_F(x) = \nabla M_F(x) x − prox F ( x ) = ∇ M F ( x ) M F M_F M F F F F
As it turns out, this characterization is tight, as witnessed by circulations. Exact one-forms have zero circulation around closed loops, whereas nonzero circulation can be converted into a bounded cyclic adversary with linear regret. For feasible finite maps, this gives an exactness-versus-circulation criterion under the regularity and step-size conditions developed below. Together with the positive result, this identifies the geometric boundary of the finite deviation class controlled by online gradient descent.
We next extend this finite-map viewpoint to mirror descent. The relevant notion of exactness is determined by the mirror geometry, through the one-form D ϕ ( x ) T d ∇ R ( x ) D_{\phi}(x)^{\mathsf{T}}\mathrm{d}\nabla R(x) D ϕ ( x ) T d ∇ R ( x )
Bregman proximality leads to a separate representation problem. Identity interpolation always transfers proximal regret to the radial closure of the Bregman proximal class, and this radial class is contained in the mirror-exact class. In Euclidean geometry the two classes coincide after radial closure, but this can fail for a general mirror regularizer. In the Legendre regime, we characterize proximal interpolation through the convexity of R ∗ − α Ψ R^*-\alpha\Psi R ∗ − α Ψ ℓ p \ell_p ℓ p
We also treat FTRL separately. Mirror descent and FTRL coincide for constant-step linearized losses in the Legendre interior, but their finite deviation classes can separate once boundary effects are present. We develop the corresponding exactness-versus-circulation theory in the cumulative dual state of FTRL and show that FTRL exactness implies mirror exactness, while the converse can fail. A powered ℓ p \ell_p ℓ p O ( T ) O(\sqrt{T}) O ( T )
Finally, we study the equilibrium notions induced by these finite deviation classes. We call the equilibrium notion associated with Euclidean exact-form deviations conservative correlated equilibrium (ConCE). The name reflects the fact that these deviations are generated by conservative, or exact, displacement fields, and that players are correspondingly restricted to this conservative family of deviations rather than arbitrary measurable remappings. Although smooth exact-form maps strictly contain weakly convex proximal maps, their unrestricted equilibrium constraints, in convex games with compact action sets and jointly continuous losses, coincide when the approximation error is zero, so ConCE = PCE. We further show that this common notion coincides with full correlated equilibrium on finite support and, under continuous losses, when every player’s action space is an interval. In contrast, with two-dimensional individual action spaces we construct an uncountably supported distribution with absolutely continuous marginals for which CE ⊊ \subsetneq ⊊
Contributions relative to prior work. Relative to Ahunbay [ [9 ] , our Euclidean analysis specializes the first-order geometric framework to finite feasible endomorphisms and studies the resulting Φ \Phi Φ [10 ] , our exact-form class extends the tractable linear-quadratic family to nonlinear finite deviations while retaining the symmetric affine maps as a special case. Relative to Cai et al. [ [11 ] , we identify exactness as the broader finite-map mechanism containing weakly convex proximal deviations, prove strict map-level containment, and characterize the role of identity interpolation in relating the two classes.
Beyond the Euclidean setting, we develop the finite-map theory for standard constrained mirror descent without Legendre duality, separate mirror exactness from Bregman proximal representability, and show that the Euclidean interpolation phenomenon can fail in nonquadratic mirror geometry. We also develop a separate cumulative-dual theory for FTRL and exhibit a boundary separation between the finite deviation classes controlled by FTRL and mirror descent. These algorithmic results are complemented by the interpolation and equilibrium results above, which determine when distinctions between deviation classes persist at the level of correlated-equilibrium constraints.
Related Work
The notion of regret is one of the central concepts in the theory of online algorithms. In its simplest incarnation, regret measures competitiveness against the best fixed comparator in hindsight. In other words, letting K K K x t x_t x t t t t ⟨ g t , ⋅ ⟩ \langle g_t,\cdot\rangle ⟨ g t , ⋅ ⟩ t t t
sup x ^ ∈ K ∑ t = 1 T ⟨ g t , x t − x ^ ⟩ . \sup_{\widehat{x}\in K}\sum_{t=1}^{T}\langle g_t,x_t-\widehat{x}\rangle. x ∈ K sup t = 1 ∑ T ⟨ g t , x t − x ⟩ . As is standard, we refer to this notion of regret as external regret . Since the seminal work of Zinkevich [ [1 ] , it is known that online gradient descent keeps external regret sublinear. Generalizations of this result to online mirror descent and follow-the-regularized-leader (FTRL) are by now standard [12 , 2 ] (Oрабona 2019), in particular optimistic algorithms and their variants for the self-play setting in games [Rakhlin and Sridharan, 2013a,b, Syrgkanis et al., 2015, Chen and Peng, 2020, Daskalakis et al., 2021, Farina et al., 2022a, Piliouras et al., 2022a, Soleymani et al., 2025a,b].
Over the past few decades researchers have been interested in obtaining guarantees for stronger notions of online competitiveness than external regret. Within the constellation of alternative notions (which includes concepts such as interval regret, dynamic regret, policy regret, switching regret, among others), the notion of Φ \Phi Φ [6 ] is the most related to our work.
Φ \Phi Φ Φ \Phi Φ deviation functions ϕ : K → K \phi:K\to K ϕ : K → K x t x_t x t ϕ ( x t ) \phi(x_t) ϕ ( x t ) ϕ ∈ Φ \phi\in\Phi ϕ ∈ Φ
sup ϕ ∈ Φ ∑ t = 1 T ⟨ g t , x t − ϕ ( x t ) ⟩ . \sup_{\phi\in\Phi}\sum_{t=1}^{T}\langle g_t,x_t-\phi(x_t)\rangle. ϕ ∈ Φ sup t = 1 ∑ T ⟨ g t , x t − ϕ ( x t )⟩ . External regret is then understood as the special instantiation of Φ \Phi Φ Φ \Phi Φ constant functions . Partially motivated by game-theoretic applications and the celebrated wealth of connections between regret and equilibrium notions [22 , 4 ] Cesa-Bianchi and Lugosi, [5 ] , a long line of work has been concerned with characterizing the largest set of deviations Φ \Phi Φ Φ \Phi Φ [8 , 23 , 24 , 25 , 26 , 27 ] and references therein). Among the most general results in that space, it is now understood that Φ \Phi Φ K K K K K K [28 , 29 ] . These algorithms are quite involved, and make use of complex separation and fixed-point machinery to deal with the geometry of endomorphisms.
While the common wisdom had been for a long time that simple algorithms such as online gradient descent only minimize external regret—and the more complex algorithms mentioned above are thus necessary to target more challenging variants of Φ \Phi Φ [9 ] , Ahunbay and Bichler [ [10 ] , Cai et al. [ [30 , 11 ] demonstrated that online gradient descent keeps under control far more than just external regret: for example, it minimizes regret against all symmetric linear endomorphic transformations, and all proximal transformations of K K K Φ \Phi Φ Φ \Phi Φ Φ \Phi Φ Φ \Phi Φ
Our Euclidean results are closely related to the first-order framework of Ahunbay [ [9 ] . After specializing that framework to displacement fields generated by feasible endomorphisms, its qualitative conservativity and circulation conclusions yield the corresponding exactness picture for projected gradient descent. We focus instead on Φ \Phi Φ ϕ ( x t ) \phi(x_t) ϕ ( x t ) [10 ] , our exact-form class extends the tractable linear–quadratic family to nonlinear feasible deviations while retaining symmetric affine endomorphisms as a special case.
Our results also address an open question raised by Cai et al. [ [11 ] . After showing that gradient descent minimizes proximal regret, they ask for a tight characterization of the regret guarantees of gradient descent and note that the class controlled by the algorithm may be strictly broader than the proximal class. We characterize this larger finite-map class geometrically through exactness and circulation. For online gradient descent, smooth exact-form deviations strictly contain weakly convex proximal deviations, while the circulation converse identifies the corresponding obstruction to sublinear regret. We develop the analogous characterization for standard constrained mirror descent with differentiable strongly convex regularizers, where the relevant object is the mirror one-form D ϕ ( x ) T d ∇ R ( x ) D_{\phi}(x)^{\mathsf{T}}\mathrm{d}\nabla R(x) D ϕ ( x ) T d ∇ R ( x ) [11 ] . We also study FTRL, which is not treated in Cai et al. [ [11 ] , and show that its controlled deviation class can differ from that of mirror descent once boundary effects are present. We further show that proximal representability is a separate question. In Euclidean geometry, identity interpolation recovers the smooth exact-form class, while in general mirror geometry the radial Bregman proximal class can be strictly smaller than the mirror-exact class. Finally, we derive the equilibrium consequences of these finite deviation classes. To our knowledge, the resulting equilibrium results are new, including the equality of ConCE and PCE under the unrestricted definitions, their coincidence with CE on finite support and on interval action spaces under continuous losses, and their strict separation from CE on a continuous support in higher dimension.
The gradient-equilibrium framework of Angelopoulos et al. [ [31 ] is closely connected to our potential-based viewpoint. In the unconstrained setting, their gradient-equilibrium condition can be recovered from the linear potentials Ψ v ( x ) = ⟨ v , x ⟩ \Psi_v(x)=\langle v,x\rangle Ψ v ( x ) = ⟨ v , x ⟩ ∇ Ψ v = v \nabla\Psi_v=v ∇ Ψ v = v D ϕ ( x ) = ∇ Ψ ( x ) D_{\phi}(\mathbf{x}) = \nabla\Psi(\mathbf{x}) D ϕ ( x ) = ∇Ψ ( x ) ϕ : K → K \phi: K \to K ϕ : K → K Φ \Phi Φ
Setting and Notation
Let K ⊆ R d K \subseteq\mathbb{R}^{d} K ⊆ R d R d \mathbb{R}^{d} R d relint ( K ) \operatorname{relint}(K) relint ( K ) aff ( K ) \operatorname{aff}(K) aff ( K ) x \mathbf{x} x g \mathbf{g} g x [ j ] \mathbf{x}[j] x [ j ] [ T ] : = { 1 , … , T } [T] := \{1,\ldots,T\} [ T ] := { 1 , … , T } ∥ ⋅ ∥ \lVert\cdot\rVert ∥ ⋅ ∥ ∥ ⋅ ∥ 2 \lVert\cdot\rVert_{2} ∥ ⋅ ∥ 2 K K K Π K \Pi_{K} Π K x ∈ K \mathbf{x} \in K x ∈ K
N K ( x ) : = { n ∈ R d : ⟨ n , y − x ⟩ ≤ 0 for every y ∈ K } . N_{K}(\mathbf{x}) := \left\{\mathbf{n} \in\mathbb{R}^{d} : \langle\mathbf{n}, \mathbf{y} - \mathbf{x}\rangle\le0 \text{ for every } \mathbf{y} \in K\right\}. N K ( x ) := { n ∈ R d : ⟨ n , y − x ⟩ ≤ 0 for every y ∈ K } . A learner plays x t ∈ K \mathbf{x}_{t} \in K x t ∈ K g t ∈ R d \mathbf{g}_{t} \in\mathbb{R}^{d} g t ∈ R d ϕ : K → K \phi: K \to K ϕ : K → K
D ϕ ( x ) : = x − ϕ ( x ) , Reg T ( ϕ ) : = ∑ t = 1 T ⟨ g t , D ϕ ( x t ) ⟩ . D_{\phi}(\mathbf{x}) := \mathbf{x} - \phi(\mathbf{x}), \qquad\operatorname{Reg}_{T}(\phi) := \sum_{t=1}^{T} \langle\mathbf{g}_{t}, D_{\phi}(\mathbf{x}_{t})\rangle. D ϕ ( x ) := x − ϕ ( x ) , Reg T ( ϕ ) := t = 1 ∑ T ⟨ g t , D ϕ ( x t )⟩ . More generally, for a displacement field D : K → R d D: K \to\mathbb{R}^{d} D : K → R d Reg T ( D ) : = ∑ t = 1 T ⟨ g t , D ( x t ) ⟩ \operatorname{Reg}_{T}(D) := \sum_{t=1}^{T} \langle\mathbf{g}_{t}, D(\mathbf{x}_{t})\rangle Reg T ( D ) := ∑ t = 1 T ⟨ g t , D ( x t )⟩ ℓ t : K → R \ell_{t}: K \to\mathbb{R} ℓ t : K → R g t = ∇ ℓ t ( x t ) \mathbf{g}_{t} = \nabla\ell_{t}(\mathbf{x}_{t}) g t = ∇ ℓ t ( x t )
ℓ t ( x t ) − ℓ t ( ϕ ( x t ) ) ≤ ⟨ ∇ ℓ t ( x t ) , x t − ϕ ( x t ) ⟩ . \ell_{t}(\mathbf{x}_{t}) - \ell_{t}(\phi(\mathbf{x}_{t})) \le\langle\nabla\ell_{t}(\mathbf{x}_{t}), \mathbf{x}_{t} - \phi(\mathbf{x}_{t})\rangle. ℓ t ( x t ) − ℓ t ( ϕ ( x t )) ≤ ⟨ ∇ ℓ t ( x t ) , x t − ϕ ( x t )⟩ . Thus every linearized regret bound below immediately implies the corresponding convex-loss deviation-regret bound.
Definition 3.1 (Exact-form deviation). A deviation ϕ : K → K \phi: K \to K ϕ : K → K continuously differentiable potential Ψ \Psi Ψ K K K
x − ϕ ( x ) = ∇ Ψ ( x ) for every x ∈ K . \mathbf{x} - \phi(\mathbf{x}) = \nabla\Psi(\mathbf{x}) \qquad\text{for every } \mathbf{x} \in K. x − ϕ ( x ) = ∇Ψ ( x ) for every x ∈ K . When a result refers to a smooth exact-form deviation , the gradient of its potential is locally Lipschitz on a neighborhood of K K K C 1 C^{1} C 1
We denote the Euclidean exact-form class by
Φ e x a c t : = { ϕ : K → K : ϕ is a Euclidean exact-form deviation } . \Phi_{\mathrm{exact}} := \{\phi: K \to K : \phi\text{ is a Euclidean exact-form deviation}\}. Φ exact := { ϕ : K → K : ϕ is a Euclidean exact-form deviation } . Throughout the paper, the subscript identifies the deviation class. Thus Φ p r o x \Phi_{\mathrm{prox}} Φ prox Φ e x a c t s m \Phi_{\mathrm{exact}}^{\mathrm{sm}} Φ exact sm R R R
Finally, we use the symbol osc K \operatorname{osc}_{K} osc K K K K
osc K ( Ψ ) : = sup x ∈ K Ψ ( x ) − inf x ∈ K Ψ ( x ) . \operatorname{osc}_{K}(\Psi) := \sup_{\mathbf{x}\in K} \Psi(\mathbf{x}) - \inf_{\mathbf{x}\in K} \Psi(\mathbf{x}). osc K ( Ψ ) := x ∈ K sup Ψ ( x ) − x ∈ K inf Ψ ( x ) . Bounds for a fixed deviation may depend on the parameters of that deviation and therefore need not yield a uniform Φ \Phi Φ
Definition 3.2 (Normalized exact-form class). For constants B , L ≥ 0 B,L \ge0 B , L ≥ 0 Φ e x a c t ( B , L ) \Phi_{\mathrm{exact}}(B,L) Φ exact ( B , L ) ϕ : K → K \phi:K\to K ϕ : K → K Ψ ϕ \Psi_{\phi} Ψ ϕ K K K
x − ϕ ( x ) = ∇ Ψ ϕ ( x ) , osc K ( Ψ ϕ ) ≤ B , x-\phi(x)=\nabla\Psi_{\phi}(x),\qquad\operatorname{osc}_{K}(\Psi_{\phi})\le B, x − ϕ ( x ) = ∇ Ψ ϕ ( x ) , osc K ( Ψ ϕ ) ≤ B , and whose gradient satisfies
∥ ∇ Ψ ϕ ( y ) − ∇ Ψ ϕ ( x ) ∥ ≤ L ∥ y − x ∥ for all x , y ∈ K . \lVert\nabla\Psi_{\phi}(y)-\nabla\Psi_{\phi}(x)\rVert\le L\lVert y-x\rVert\qquad\text{for all }x,y\in K. ∥ ∇ Ψ ϕ ( y ) − ∇ Ψ ϕ ( x )∥ ≤ L ∥ y − x ∥ for all x , y ∈ K . Proofs not given in the main text are found in the appendices. Appendix A records the convex-analytic preliminaries; Appendix B proves the scaled proximal representation; Appendix C proves the finite interpolation theorem; and Appendix D contains the remaining proofs, organized in the same order as the main text.
Online gradient descent with constant step size η > 0 \eta>0 η > 0
x t + 1 : = Π K ( x t − η g t ) . x_{t+1}:=\Pi_K(x_t-\eta g_t). x t + 1 := Π K ( x t − η g t ) . (OGD)
We prove a telescoping regret bound for every smooth feasible exact-form deviation ϕ \phi ϕ Reg T ( ϕ ) = O ( T ) \operatorname{Reg}_T(\phi)=O(\sqrt{T}) Reg T ( ϕ ) = O ( T ) Ψ \Psi Ψ ϕ ( K ) ⊆ K \phi(K)\subseteq K ϕ ( K ) ⊆ K
Theorem 4.1. Let { x t } t = 1 T + 1 \{x_t\}_{t=1}^{T+1} { x t } t = 1 T + 1 ϕ : K → K \phi:K\to K ϕ : K → K D ϕ ( x ) = x − ϕ ( x ) = ∇ Ψ ( x ) D_{\phi}(x)=x-\phi(x)=\nabla\Psi(x) D ϕ ( x ) = x − ϕ ( x ) = ∇Ψ ( x ) Ψ \Psi Ψ L L L g 1 , … , g T , g_1,\ldots,g_T, g 1 , … , g T , 1
∑ t = 1 T ⟨ g t , x t − ϕ ( x t ) ⟩ ≤ Ψ ( x 1 ) − Ψ ( x T + 1 ) η + 3 L η 2 ∑ t = 1 T ∥ g t ∥ 2 . (1) \sum_{t=1}^{T}\langle g_t,x_t-\phi(x_t)\rangle\le\frac{\Psi(x_1)-\Psi(x_{T+1})}{\eta}+\frac{3L\eta}{2}\sum_{t=1}^{T}\lVert g_t\rVert^2. \tag*{(1)} t = 1 ∑ T ⟨ g t , x t − ϕ ( x t )⟩ ≤ η Ψ ( x 1 ) − Ψ ( x T + 1 ) + 2 3 L η t = 1 ∑ T ∥ g t ∥ 2 . (1) The proof of Theorem 4.1 is given in Appendix D.1.
Corollary 4.2. Let { x t } t = 1 T + 1 \{x_t\}_{t=1}^{T+1} { x t } t = 1 T + 1 K K K ∥ g t ∥ ≤ G \lVert g_t\rVert\le G ∥ g t ∥ ≤ G osc K ( Ψ ) ≤ B \operatorname{osc}_{K}(\Psi)\le B osc K ( Ψ ) ≤ B B , L , G > 0 B,L,G>0 B , L , G > 0
Reg T ( ϕ ) ≤ B η + 3 L η G 2 T 2 . \operatorname{Reg}_T(\phi)\le\frac{B}{\eta}+\frac{3L\eta G^2T}{2}. Reg T ( ϕ ) ≤ η B + 2 3 L η G 2 T . In particular, choosing η = 2 B 3 L G 2 T \eta=\sqrt{\frac{2B}{3LG^2T}} η = 3 L G 2 T 2 B Reg T ( ϕ ) ≤ G 6 L B T = O ( T ) \operatorname{Reg}_T(\phi)\le G\sqrt{6LBT}=O(\sqrt{T}) Reg T ( ϕ ) ≤ G 6 L B T = O ( T )
Corollary 4.3. Suppose ∥ g t ∥ ≤ G \lVert g_t\rVert\le G ∥ g t ∥ ≤ G B , L > 0 B,L>0 B , L > 0 η = 2 B 3 L G 2 T \eta=\sqrt{\frac{2B}{3LG^2T}} η = 3 L G 2 T 2 B
sup ϕ ∈ Φ e x a c t ( B , L ) Reg T ( ϕ ) ≤ G 6 L B T . \sup_{\phi\in\Phi_{\mathrm{exact}}(B,L)}\operatorname{Reg}_T(\phi)\le G\sqrt{6LBT}. ϕ ∈ Φ exact ( B , L ) sup Reg T ( ϕ ) ≤ G 6 L B T . Thus the algorithm controls the whole normalized class Φ e x a c t ( B , L ) \Phi_{\mathrm{exact}}(B,L) Φ exact ( B , L )
Remark 4.4. The proof of Theorem 4.1 and Corollary 4.2 uses the standard quadratic upper bound for Ψ \Psi Ψ L L L ∇ Ψ \nabla\Psi ∇Ψ
This section proves that the exact-form deviation class includes the proximal deviations of Cai et al. [ [11 ] . The key identity is the Moreau-envelope formula Moreau, [32 ] , which was also observed by Ahunbay [ [9 ] .
x − prox F ( x ) = ∇ M F ( x ) . (2) x-\operatorname{prox}_{F}(x)=\nabla M_{F}(x). \tag*{(2)} x − prox F ( x ) = ∇ M F ( x ) . (2) These proximal-regret deviations are generated by ρ \rho ρ 0 ≤ ρ < 1 0 \le\rho< 1 0 ≤ ρ < 1 F : = f + ι K F := f+\iota_{K} F := f + ι K ι K \iota_{K} ι K K K K 0 ≤ ρ < 1 0 \le\rho< 1 0 ≤ ρ < 1
prox F ( x ) : = arg min y ∈ R d { F ( y ) + 1 2 ∥ y − x ∥ 2 } , M F ( x ) : = min y ∈ R d { F ( y ) + 1 2 ∥ y − x ∥ 2 } . \operatorname{prox}_{F}(x):=\underset{y\in\mathbb{R}^{d}}{\arg\min}\left\{F(y)+\frac{1}{2}\lVert y-x\rVert^{2}\right\},\qquad M_{F}(x):=\underset{y\in\mathbb{R}^{d}}{\min}\left\{F(y)+\frac{1}{2}\lVert y-x\rVert^{2}\right\}. prox F ( x ) := y ∈ R d arg min { F ( y ) + 2 1 ∥ y − x ∥ 2 } , M F ( x ) := y ∈ R d min { F ( y ) + 2 1 ∥ y − x ∥ 2 } . Because F F F ι K \iota_{K} ι K K K K
Theorem 5.1. Let F : = f + ι K F:=f+\iota_{K} F := f + ι K ρ \rho ρ 0 ≤ ρ < 1 0 \le\rho< 1 0 ≤ ρ < 1 ϕ F ( x ) : = prox F ( x ) \phi_{F}(x):=\operatorname{prox}_{F}(x) ϕ F ( x ) := prox F ( x ) ϕ F : K → K \phi_{F}:K\to K ϕ F : K → K M F M_{F} M F x − ϕ F ( x ) = ∇ M F ( x ) x-\phi_{F}(x)=\nabla M_{F}(x) x − ϕ F ( x ) = ∇ M F ( x ) L F = ( 2 − ρ ) / ( 1 − ρ ) L_{F}=(2-\rho)/(1-\rho) L F = ( 2 − ρ ) / ( 1 − ρ ) K K K ∥ g t ∥ ≤ G \lVert g_{t}\rVert\le G ∥ g t ∥ ≤ G osc K ( M F ) ≤ B F \operatorname{osc}_{K}(M_{F})\le B_{F} osc K ( M F ) ≤ B F B F , G > 0 B_{F},G>0 B F , G > 0 η = 2 B F / ( 3 L F G 2 T ) \eta=\sqrt{2B_{F}/(3L_{F}G^{2}T)} η = 2 B F / ( 3 L F G 2 T )
∑ t = 1 T ⟨ g t , x t − prox F ( x t ) ⟩ ≤ B F η + 3 L F η G 2 T 2 = O ( T ) . \sum_{t=1}^{T}\langle g_{t},x_{t}-\operatorname{prox}_{F}(x_{t})\rangle\le\frac{B_{F}}{\eta}+\frac{3L_{F}\eta G^{2}T}{2}=O(\sqrt{T}). t = 1 ∑ T ⟨ g t , x t − prox F ( x t )⟩ ≤ η B F + 2 3 L F η G 2 T = O ( T ) . If B F = 0 B_{F}=0 B F = 0 G = 0 G=0 G = 0
The Moreau-envelope identity immediately recovers several standard deviation classes. The point of the following proposition is not to introduce new deviations, but to show that many of the deviations in Cai et al. [ [11 ] have exact displacement fields, and therefore fall under our regret bound.
Proposition 5.2. The exact-form class contains the following deviations.
External regret. For u ∈ K u\in K u ∈ K F : = ι { u } F:=\iota_{\{u\}} F := ι { u } prox F ( x ) = u \operatorname{prox}_{F}(x)=u prox F ( x ) = u x − u = ∇ 1 2 ∥ x − u ∥ 2 x-u=\nabla\frac{1}{2}\lVert x-u\rVert^{2} x − u = ∇ 2 1 ∥ x − u ∥ 2
Projection to a convex subset. For a nonempty closed convex set S ⊆ K S\subseteq K S ⊆ K F : = ι S F:=\iota_{S} F := ι S prox F ( x ) = Π S ( x ) \operatorname{prox}_{F}(x)=\Pi_{S}(x) prox F ( x ) = Π S ( x )
Projection-based/no-move regret. For a vector v v v F ( y ) : = ⟨ v , y ⟩ + ι K ( y ) F(y):=\langle v,y\rangle+\iota_{K}(y) F ( y ) := ⟨ v , y ⟩ + ι K ( y ) prox F ( x ) = Π K ( x − v ) \operatorname{prox}_{F}(x)=\Pi_{K}(x-v) prox F ( x ) = Π K ( x − v )
Interpolation deviations. For u ∈ K u\in K u ∈ K α ∈ ( 0 , 1 ) \alpha\in(0,1) α ∈ ( 0 , 1 ) ϕ ( x ) : = ( 1 − α ) x + α u \phi(x):=(1-\alpha)x+\alpha u ϕ ( x ) := ( 1 − α ) x + α u
F ( y ) : = α 2 ( 1 − α ) ∥ y − u ∥ 2 + ι K ( y ) . F(y):=\frac{\alpha}{2(1-\alpha)}\lVert y-u\rVert^{2}+\iota_{K}(y). F ( y ) := 2 ( 1 − α ) α ∥ y − u ∥ 2 + ι K ( y ) . The case α = 1 \alpha=1 α = 1
Local proximal deviations. If ∥ x − prox F ( x ) ∥ ≤ δ \lVert x-\operatorname{prox}_{F}(x)\rVert\le\delta ∥ x − prox F ( x )∥ ≤ δ K K K ∥ ∇ M F ( x ) ∥ ≤ δ \lVert\nabla M_{F}(x)\rVert\le\delta ∥ ∇ M F ( x )∥ ≤ δ
Symmetric affine swaps. Let ϕ ( x ) : = A x + b \phi(x):=Ax+b ϕ ( x ) := A x + b K K K K K K T : = span ( K − K ) T:=\operatorname{span}(K-K) T := span ( K − K ) aff ( K ) \operatorname{aff}(K) aff ( K ) a ∈ K a\in K a ∈ K A T ⊆ T AT\subseteq T A T ⊆ T I − A I-A I − A T T T ϕ \phi ϕ aff ( K ) \operatorname{aff}(K) aff ( K )
Ψ ( a + h ) : = ⟨ D ϕ ( a ) , h ⟩ + 1 2 ⟨ h , ( I − A ) h ⟩ , h ∈ T . (3) \Psi(a+h):=\langle D_{\phi}(a),h\rangle+\frac{1}{2}\langle h,(I-A)h\rangle,\qquad h\in T. \tag*{(3)} Ψ ( a + h ) := ⟨ D ϕ ( a ) , h ⟩ + 2 1 ⟨ h , ( I − A ) h ⟩ , h ∈ T . (3) An ambient extension of this potential is given in the proof.
Remark 5.3. Theorem 5.1 and Proposition 5.2 cover the Euclidean exact-form deviation classes appearing in Cai et al. [ [11 ] including point indicators for external regret, linear functions for projection-based/no-move regret, bounded-displacement local proximal deviations, interpolation deviations, and symmetric affine swaps. The symmetric affine case is even more direct in the exact-form view since the displacement of a symmetric affine map is already the gradient of a quadratic potential, without first representing the map as a proximal operator.
The Exact-form Class Strictly Contains the Proximal Class
The inclusion from proximal deviations to exact-form deviations is strict, and the separation already appears in one dimension. Our proof strategy is to use the fact that every weakly convex proximal map with parameter 0 ≤ ρ < 1 0 \le\rho< 1 0 ≤ ρ < 1
Proposition 6.1. There exists a smooth feasible exact-form deviation ϕ : [ 0 , 1 ] → [ 0 , 1 ] \phi: [0,1] \to[0,1] ϕ : [ 0 , 1 ] → [ 0 , 1 ] prox F \operatorname{prox}_{F} prox F ρ \rho ρ F F F 0 ≤ ρ < 1 0 \le\rho< 1 0 ≤ ρ < 1
Proof. Let k ≥ 2 k \ge2 k ≥ 2 a ∈ ( 0 , 1 ] a \in(0,1] a ∈ ( 0 , 1 ] a π k / 2 > 1 a\pi k/2 > 1 aπ k /2 > 1
D ϕ ( x ) : = a x ( 1 − x ) sin ( 2 π k x ) , ϕ ( x ) : = x − D ϕ ( x ) . D_{\phi}(x) := ax(1-x)\sin(2\pi kx), \qquad\phi(x) := x-D_{\phi}(x). D ϕ ( x ) := a x ( 1 − x ) sin ( 2 π k x ) , ϕ ( x ) := x − D ϕ ( x ) . Because ∣ D ϕ ( x ) ∣ ≤ a x ( 1 − x ) ≤ min { x , 1 − x } |D_{\phi}(x)| \le ax(1-x) \le\min\{x,1-x\} ∣ D ϕ ( x ) ∣ ≤ a x ( 1 − x ) ≤ min { x , 1 − x } ϕ ( x ) ∈ [ 0 , 1 ] \phi(x) \in[0,1] ϕ ( x ) ∈ [ 0 , 1 ] D ϕ = Ψ ′ D_{\phi} = \Psi' D ϕ = Ψ ′
Ψ ( x ) : = ∫ 0 x a s ( 1 − s ) sin ( 2 π k s ) d s , \Psi(x) := \int_{0}^{x} as(1-s)\sin(2\pi ks)\,\mathrm{d}s, Ψ ( x ) := ∫ 0 x a s ( 1 − s ) sin ( 2 π k s ) d s , so ϕ \phi ϕ
At x = 1 / 2 x=1/2 x = 1/2 k k k
D ϕ ′ ( 1 / 2 ) = a π k 2 > 1 , ϕ ′ ( 1 / 2 ) = 1 − D ϕ ′ ( 1 / 2 ) < 0. D_{\phi}'(1/2) = \frac{a\pi k}{2} > 1, \qquad\phi'(1/2) = 1-D_{\phi}'(1/2) < 0. D ϕ ′ ( 1/2 ) = 2 aπ k > 1 , ϕ ′ ( 1/2 ) = 1 − D ϕ ′ ( 1/2 ) < 0. Thus ϕ \phi ϕ 0 ≤ ρ < 1 0 \le\rho< 1 0 ≤ ρ < 1 ϕ \phi ϕ prox F \operatorname{prox}_{F} prox F F F F □ \square □
Thus while exact-form maps strictly contain weakly-convex proximal maps, we show that this strict inclusion does not carry over to the corresponding equilibrium sets, in particular when the approximation error is zero. The following scaling result is the key mechanism behind this equivalence. We show that scaled exact deviations are proximal.
Theorem 6.2. Let K ⊆ R d K \subseteq\mathbb{R}^{d} K ⊆ R d ϕ ( x ) : = x − ∇ Ψ ( x ) \phi(x) := x-\nabla\Psi(x) ϕ ( x ) := x − ∇Ψ ( x ) Ψ ∈ C l o c 1 , 1 \Psi\in C^{1,1}_{\mathrm{loc}} Ψ ∈ C loc 1 , 1 K K K α 0 > 0 \alpha_{0} > 0 α 0 > 0 α ∈ ( 0 , α 0 ] \alpha\in(0,\alpha_{0}] α ∈ ( 0 , α 0 ]
ϕ α ( x ) : = ( 1 − α ) x + α ϕ ( x ) = x − α ∇ Ψ ( x ) \phi_{\alpha}(x) := (1-\alpha)x+\alpha\phi(x) = x-\alpha\nabla\Psi(x) ϕ α ( x ) := ( 1 − α ) x + α ϕ ( x ) = x − α ∇Ψ ( x ) coincides on K K K prox F α ( x ) \operatorname{prox}_{F_{\alpha}}(x) prox F α ( x ) F α = f α + ι K F_{\alpha}=f_{\alpha}+\iota_{K} F α = f α + ι K ρ α \rho_{\alpha} ρ α ρ α < 1 \rho_{\alpha}<1 ρ α < 1
The construction is closely related to the characterization of proximity operators in terms of convex potentials developed by Gribonval and Nikolova [ [33 ] . The construction and its proof are given in Appendix § 14 . It reconciles the strict inclusion
Φ p r o x ⊊ Φ e x a c t s m ⊆ Φ e x a c t . \Phi_{\mathrm{prox}} \subsetneq\Phi_{\mathrm{exact}}^{\mathrm{sm}} \subseteq\Phi_{\mathrm{exact}}. Φ prox ⊊ Φ exact sm ⊆ Φ exact . Although a smooth exact-form deviation need not itself be proximal, every such deviation admits a sufficiently small interpolation with the identity that is proximal.
Write Φ p r o x \Phi_{\mathrm{prox}} Φ prox Φ e x a c t s m \Phi_{\mathrm{exact}}^{\mathrm{sm}} Φ exact sm C l o c 1 , 1 C_{\mathrm{loc}}^{1,1} C loc 1 , 1 K K K A \mathcal{A} A identity-radial closure by
Rad I ( A ) : = { ϕ feasible : for some α ∈ ( 0 , 1 ] , ( 1 − α ) I + α ϕ ∈ A } . \operatorname{Rad}_{I}(\mathcal{A}) := \{\phi\text{ feasible}: \text{ for some }\alpha\in(0,1],\ (1-\alpha)I+\alpha\phi\in\mathcal{A}\}. Rad I ( A ) := { ϕ feasible : for some α ∈ ( 0 , 1 ] , ( 1 − α ) I + α ϕ ∈ A } . Corollary 6.3. For the radial closure of Euclidean proximal deviations, we have Rad I ( Φ p r o x ) = Φ e x a c t s m \operatorname{Rad}_{I}(\Phi_{\mathrm{prox}})=\Phi_{\mathrm{exact}}^{\mathrm{sm}} Rad I ( Φ prox ) = Φ exact sm
Although Corollary 6.3 does not identify the two map classes, it shows that their difference disappears after allowing interpolation with the identity. Indeed, every smooth exact-form deviation admits a sufficiently small positive interpolation that is proximal. Thus, while Φ p r o x ⊊ Φ e x a c t \Phi_{\mathrm{prox}}\subsetneq\Phi_{\mathrm{exact}} Φ prox ⊊ Φ exact § 10 .
Remark 6.4. For D ϕ α = α D ϕ D_{\phi_{\alpha}}=\alpha D_{\phi} D ϕ α = α D ϕ Reg T ( ϕ α ) = α Reg T ( ϕ ) \operatorname{Reg}_{T}(\phi_{\alpha})=\alpha\operatorname{Reg}_{T}(\phi) Reg T ( ϕ α ) = α Reg T ( ϕ ) ℓ \ell ℓ ℓ ( x ) − ℓ ( ϕ α ( x ) ) ≥ α ( ℓ ( x ) − ℓ ( ϕ ( x ) ) ) \ell(x)-\ell(\phi_{\alpha}(x))\geq\alpha(\ell(x)-\ell(\phi(x))) ℓ ( x ) − ℓ ( ϕ α ( x )) ≥ α ( ℓ ( x ) − ℓ ( ϕ ( x ))) ConCE = PCE \operatorname{ConCE}=\operatorname{PCE} ConCE = PCE
Curl Induces Linear Regret for Online Gradient Descent
The upper bound telescopes because exact one-forms integrate to zero around closed loops. We now show that nonzero circulation gives the corresponding obstruction for finite deviation maps. The construction is the finite-map analogue of the nonconservative vector-field lower bound of Ahunbay [ [9 ] ; here we formulate it using a globally feasible endpoint map and the regret convention of our setting.
Definition 7.1 (Circulation). Let D : K → R d D:K\to\mathbb{R}^{d} D : K → R d γ : [ 0 , 1 ] → K \gamma:[0,1]\to K γ : [ 0 , 1 ] → K C 1 C^{1} C 1 D D D γ \gamma γ
∮ γ D ( x ) T d x : = ∫ 0 1 ⟨ D ( γ ( s ) ) , γ ′ ( s ) ⟩ d s . \oint_{\gamma}D(x)^{\mathsf{T}}\mathrm{d}x:=\int_{0}^{1}\langle D(\gamma(s)),\gamma'(s)\rangle\mathrm{d}s. ∮ γ D ( x ) T d x := ∫ 0 1 ⟨ D ( γ ( s )) , γ ′ ( s )⟩ d s . First, we show that exact fields have zero circulation, which is why this obstruction does not arise for exact-form regret.
Lemma 7.2. If D = ∇ Ψ D=\nabla\Psi D = ∇Ψ K K K ∮ γ D ( x ) T d x = 0 \oint_{\gamma}D(x)^{\mathsf{T}}\mathrm{d}x=0 ∮ γ D ( x ) T d x = 0 γ ⊆ K \gamma\subseteq K γ ⊆ K
Thus exactness rules out circulation. The next theorem proves the converse algorithmic statement. Nonzero circulation can be converted into a bounded cyclic adversary and hence into linear regret.
Theorem 7.3. Let D : K → R d D:K\to\mathbb{R}^{d} D : K → R d C 1 C^{1} C 1 γ : [ 0 , 1 ] → K \gamma:[0,1]\to K γ : [ 0 , 1 ] → K
− ∮ γ D ( x ) T d x = : c 0 > 0. -\oint_{\gamma}D(x)^{\mathsf{T}}\mathrm{d}x=:c_{0}>0. − ∮ γ D ( x ) T d x =: c 0 > 0. Then for every gradient bound G > 0 G>0 G > 0 η > 0 \eta>0 η > 0 ∥ g t ∥ ≤ G \lVert g_{t}\rVert\leq G ∥ g t ∥ ≤ G γ ( 0 ) \gamma(0) γ ( 0 )
∑ t = 1 T ⟨ g t , D ( x t ) ⟩ ≥ c T − O ( 1 / η ) \sum_{t=1}^{T}\langle g_{t},D(x_{t})\rangle\geq cT-O(1/\eta) t = 1 ∑ T ⟨ g t , D ( x t )⟩ ≥ c T − O ( 1/ η ) for a constant c > 0 c>0 c > 0 D D D γ \gamma γ G G G Ω ( T ) \Omega(T) Ω ( T ) K K K O ( 1 / η ) O(1/\eta) O ( 1/ η )
The same lower-bound mechanism also extends to standard anytime learning-rate schedules.
Together, the upper and lower bounds give the following Euclidean curl criterion for finite deviation maps. Curl-free fields admit a potential and, under the feasibility and smoothness conditions below, yield sublinear regret. By contrast, nonzero curl produces a local loop along which gradient descent can be forced to incur linear regret. The two directions involve slightly different learning-rate assumptions. The lower bound uses a sufficiently small constant step size that may depend on the witnessing loop, while the uniform upper bound over a normalized family is given in Corollary 4.3.
Corollary 7.5 (Euclidean curl criterion). Let E K : = span ( K − K ) E_K := \operatorname{span}(K-K) E K := span ( K − K ) U : = relint ( K ) U := \operatorname{relint}(K) U := relint ( K ) aff ( K ) \operatorname{aff}(K) aff ( K ) D : U → E K D : U \to E_K D : U → E K C 1 C^1 C 1 E K E_K E K ∂ j D i = ∂ i D j \partial_jD_i = \partial_iD_j ∂ j D i = ∂ i D j U U U D = ∇ E K Ψ D = \nabla_{E_K}\Psi D = ∇ E K Ψ U U U C 2 C^2 C 2 Ψ \Psi Ψ D D D Ψ \Psi Ψ Ψ ~ \widetilde{\Psi} Ψ K K K ϕ ~ ( x ) : = x − ∇ Ψ ~ ( x ) \widetilde{\phi}(x) := x-\nabla\widetilde{\Psi}(x) ϕ ( x ) := x − ∇ Ψ ( x ) K K K x ∈ K x \in K x ∈ K O ( T ) O(\sqrt{T}) O ( T ) ϕ ~ \widetilde{\phi} ϕ
Online Mirror Descent and Exact One-forms
Our Euclidean result in Section 4 says that online gradient descent controls deviations whose displacement is a gradient. Mirror descent changes the coordinates in which the same statement is true. Its natural variable is not the primal point x x x
θ : = ∇ R ( x ) , \theta:= \nabla R(x), θ := ∇ R ( x ) , for a distance-generating function R R R
θ t + 1 = θ t − η g t . \theta_{t+1} = \theta_t-\eta g_t. θ t + 1 = θ t − η g t . Therefore, as expected we show that the same telescoping proof works whenever the primal displacement D ϕ ( x ) D_{\phi}(x) D ϕ ( x ) θ \theta θ
To make the mirror generalization transparent, we first work in the Legendre interior. In this setting, ∇ R \nabla R ∇ R [11 ] as a special case.
We then return to standard constrained mirror descent in § 8.4 and remove the Legendre assumption. The iterates may now reach the boundary, and the mirror map need not provide a globally invertible dual coordinate system. A normal-cone residual therefore appears in the mirror-descent optimality condition. The underlying geometric condition, however, remains the same. Exactness of the mirror one-form gives the upper bound after the boundary residual is controlled, while nonzero circulation gives the corresponding lower bound. This shows that the exactness principle is not a consequence of Legendre duality, even though the Legendre setting provides the cleanest way to see it initially.
With the regret characterization in place, § 8.5 returns to the identity interpolation of Theorem 6.2. We keep the original weakly convex Bregman proximal class and use the scaling identity to transfer proximal regret to its radial closure. The Euclidean conclusion does not extend in full generality. The radial proximal class is always contained in the mirror-exact class, but the containment can be strict. In the Legendre setting, convexity of R ∗ − α Ψ R^{*} - \alpha\Psi R ∗ − α Ψ
There is a further distinction once one leaves the Legendre interior. For constant-step linearized losses, Legendre mirror descent has an additive dual recursion, and unrolling that recursion gives the corresponding follow-the-regularized-leader (FTRL) update. This relationship between mirror descent, FTRL, and closely related dual-averaging methods is well known in online optimization [35 , 36 ] (McMahan 2011). At the boundary, however, standard constrained mirror descent and FTRL need not generate the same dynamics. Mirror descent performs a one-step Bregman update and may carry a normal-cone residual, whereas FTRL retains the accumulated linear losses in a single regularized minimization. The distinction also affects which deviations the two algorithms control. In § 9 , we show that their exact deviation classes can differ and give a smooth powered ℓ p \ell_p ℓ p O ( T ) O(\sqrt{T}) O ( T ) § 9 .
Throughout the first three subsections of this section, all convex-analytic objects are taken in the affine hull of K K K E E E R : E → ( − ∞ , + ∞ ] R:E \to(-\infty,+\infty] R : E → ( − ∞ , + ∞ ]
dom R = K , D : = int E ( dom R ) = relint ( K ) . \operatorname{dom} R = K,\qquad\mathcal{D} := \operatorname{int}_{E}(\operatorname{dom} R) = \operatorname{relint}(K). dom R = K , D := int E ( dom R ) = relint ( K ) . Assume that R R R C 2 C^{2} C 2 D \mathcal{D} D ∇ 2 R ( x ) \nabla^{2}R(x) ∇ 2 R ( x )
R ∗ ( θ ) : = sup x ∈ E { ⟨ θ , x ⟩ − R ( x ) } , Θ : = int E ( dom R ∗ ) = ∇ R ( D ) . R^{*}(\theta) := \sup_{x \in E}\{\langle\theta,x\rangle- R(x)\},\qquad\Theta:= \operatorname{int}_{E}(\operatorname{dom} R^{*}) = \nabla R(\mathcal{D}). R ∗ ( θ ) := x ∈ E sup {⟨ θ , x ⟩ − R ( x )} , Θ := int E ( dom R ∗ ) = ∇ R ( D ) . Legendre duality gives mutually inverse C 1 C^{1} C 1 ∇ R : D → Θ \nabla R:\mathcal{D}\to\Theta ∇ R : D → Θ ∇ R ∗ : Θ → D \nabla R^{*}:\Theta\to\mathcal{D} ∇ R ∗ : Θ → D
D R ∗ ( θ ′ , θ ) = D R ( x , x ′ ) when θ = ∇ R ( x ) , θ ′ = ∇ R ( x ′ ) . D_{R^{*}}(\theta',\theta)=D_{R}(x,x')\quad\text{when }\theta=\nabla R(x),\quad\theta'=\nabla R(x'). D R ∗ ( θ ′ , θ ) = D R ( x , x ′ ) when θ = ∇ R ( x ) , θ ′ = ∇ R ( x ′ ) . We initialize mirror descent in D \mathcal{D} D D \mathcal{D} D θ : = ∇ R ( x ) \theta:=\nabla R(x) θ := ∇ R ( x )
We use the Bregman divergence generated by R R R
D R ( y , x ) : = R ( y ) − R ( x ) − ⟨ ∇ R ( x ) , y − x ⟩ . D_R(y,x):=R(y)-R(x)-\langle\nabla R(x),y-x\rangle. D R ( y , x ) := R ( y ) − R ( x ) − ⟨ ∇ R ( x ) , y − x ⟩ . With this notation, online mirror descent takes the form
x t + 1 : = arg min x ∈ K { η ⟨ g t , x ⟩ + D R ( x , x t ) } . (4) x_{t+1}:=\underset{x\in K}{\operatorname{arg\,min}}\left\{\eta\langle g_t,x\rangle+D_R(x,x_t)\right\}. \tag*{(4)} x t + 1 := x ∈ K arg min { η ⟨ g t , x ⟩ + D R ( x , x t ) } . (4) Definition 8.1 (Mirror-exact deviation). A deviation ϕ : D → K \phi:\mathcal{D}\to K ϕ : D → K D ϕ ( x ) : = x − ϕ ( x ) D_{\phi}(x):=x-\phi(x) D ϕ ( x ) := x − ϕ ( x ) R R R Ψ : Θ → R \Psi:\Theta\to\mathbb{R} Ψ : Θ → R
D ϕ ( x ) = ∇ θ Ψ ( ∇ R ( x ) ) ∀ x ∈ D . D_{\phi}(x)=\nabla_{\theta}\Psi(\nabla R(x))\qquad\forall x\in\mathcal{D}. D ϕ ( x ) = ∇ θ Ψ ( ∇ R ( x )) ∀ x ∈ D . Equivalently, the one-form D ϕ ( x ) T d ∇ R ( x ) D_{\phi}(x)^{\mathsf{T}}\mathrm{d}\nabla R(x) D ϕ ( x ) T d ∇ R ( x )
We write Φ e x a c t R ( D ) \Phi^{R}_{\mathrm{exact}}(\mathcal{D}) Φ exact R ( D )
Remark 8.2. If K = E K = E K = E R ( x ) = 1 2 ∥ x ∥ 2 R(x) = \frac{1}{2}\lVert x\rVert^2 R ( x ) = 2 1 ∥ x ∥ 2 ∇ R ( x ) = x \nabla R(x) = x ∇ R ( x ) = x D ϕ ( x ) = ∇ Ψ ( x ) D_{\phi}(x) = \nabla\Psi(x) D ϕ ( x ) = ∇Ψ ( x )
The next proposition gives a useful primal-coordinate test for this condition. It shows that mirror-exactness is equivalent to ordinary exactness after multiplying the displacement by the mirror metric ∇ 2 R ( x ) \nabla^2 R(x) ∇ 2 R ( x )
Proposition 8.3. A field D : D → E D : \mathcal{D} \to E D : D → E R R R V : D → R V : \mathcal{D} \to\mathbb{R} V : D → R
∇ V ( x ) = ∇ 2 R ( x ) D ( x ) . (5) \nabla V(x) = \nabla^2 R(x)D(x). \tag*{(5)} ∇ V ( x ) = ∇ 2 R ( x ) D ( x ) . (5) Equivalently, D ( x ) T d ∇ R ( x ) = d V ( x ) D(x)^{\mathsf{T}}\mathrm{d}\nabla R(x) = \mathrm{d}V(x) D ( x ) T d ∇ R ( x ) = d V ( x )
Examples of Mirror-exact Deviations
Definition 8.1 can look abstract because it is written in dual coordinates. The quickest way to build examples is simple: “choose a scalar potential Ψ ( θ ) \Psi(\theta) Ψ ( θ ) ∇ θ Ψ ( θ ) \nabla_{\theta}\Psi(\theta) ∇ θ Ψ ( θ ) θ = ∇ R ( x ) \theta= \nabla R(x) θ = ∇ R ( x ) ϕ ( x ) = x − ∇ θ Ψ ( ∇ R ( x ) ) \phi(x) = x - \nabla_{\theta}\Psi(\nabla R(x)) ϕ ( x ) = x − ∇ θ Ψ ( ∇ R ( x ))
In Appendix § 16 .5, we provide several examples of mirror-exact deviations. They show that the mirror-exact deviation family contains every fixed-comparator deviation. They also give nontrivial deviations for different instances of online mirror descent, such as, for the multiplicative weights update algorithm, multiplicative deviations with arbitrary temperature vector a ∈ R d a \in\mathbb{R}^d a ∈ R d
ϕ a ( x ) [ i ] : = x [ i ] exp ( − a [ i ] ) ∑ j = 1 d x [ j ] exp ( − a [ j ] ) . \phi_a(x)[i] := \frac{x[i]\exp(-a[i])}{\sum_{j=1}^{d}x[j]\exp(-a[j])}. ϕ a ( x ) [ i ] := ∑ j = 1 d x [ j ] exp ( − a [ j ]) x [ i ] exp ( − a [ i ]) . and geometric interpolation deviations anchored at u ∈ relint ( Δ d ) u \in\operatorname{relint}(\Delta_d) u ∈ relint ( Δ d ) λ ∈ [ 0 , 1 ) \lambda\in[0,1) λ ∈ [ 0 , 1 )
ϕ λ , u ( x ) [ i ] = x [ i ] 1 − λ u [ i ] λ ∑ j = 1 d x [ j ] 1 − λ u [ j ] λ . \phi_{\lambda,u}(x)[i] = \frac{x[i]^{1-\lambda}u[i]^{\lambda}}{\sum_{j=1}^{d}x[j]^{1-\lambda}u[j]^{\lambda}}. ϕ λ , u ( x ) [ i ] = ∑ j = 1 d x [ j ] 1 − λ u [ j ] λ x [ i ] 1 − λ u [ i ] λ . and there exist mirror-exact deviations that are not Euclidean-exact.
Online Mirror Descent Controls Mirror-exact Regret
We now prove no-regret results for mirror-exact deviations under online mirror descent in the Legendre interior. For a differentiable function A A A
D A ( θ ′ , θ ) : = A ( θ ′ ) − A ( θ ) − ⟨ ∇ A ( θ ) , θ ′ − θ ⟩ . D_A(\theta',\theta) := A(\theta') - A(\theta) - \langle\nabla A(\theta), \theta' - \theta\rangle. D A ( θ ′ , θ ) := A ( θ ′ ) − A ( θ ) − ⟨ ∇ A ( θ ) , θ ′ − θ ⟩ . and, for L ≥ 0 L \ge0 L ≥ 0 Ψ \Psi Ψ L L L R ∗ R^* R ∗
D Ψ ( θ ′ , θ ) ≤ L D R ∗ ( θ ′ , θ ) (6) D_{\Psi}(\theta',\theta) \le L D_{R^*}(\theta',\theta) \tag*{(6)} D Ψ ( θ ′ , θ ) ≤ L D R ∗ ( θ ′ , θ ) (6) for every such pair θ ′ , θ ∈ Θ \theta', \theta\in\Theta θ ′ , θ ∈ Θ K = E K = E K = E R ( x ) = 1 2 ∥ x ∥ 2 R(x) = \frac{1}{2}\lVert x\rVert^2 R ( x ) = 2 1 ∥ x ∥ 2 Ψ \Psi Ψ
Theorem 8.4. Assume the standing Legendre-domain conditions and that R R R m m m D \mathcal{D} D ϕ : D → K \phi: \mathcal{D} \to K ϕ : D → K R R R Ψ \Psi Ψ { x t } t = 1 T + 1 \{x_t\}_{t=1}^{T+1} { x t } t = 1 T + 1 (4) , initialized in D \mathcal{D} D D \mathcal{D} D
θ t : = ∇ R ( x t ) , D ϕ ( x t ) = ∇ θ Ψ ( θ t ) . \theta_t := \nabla R(x_t), \qquad D_{\phi}(x_t) = \nabla_{\theta}\Psi(\theta_t). θ t := ∇ R ( x t ) , D ϕ ( x t ) = ∇ θ Ψ ( θ t ) . Assume that (6) holds for every consecutive pair ( θ t + 1 , θ t ) (\theta_{t+1},\theta_t) ( θ t + 1 , θ t ) t ∈ [ T ] t \in[T] t ∈ [ T ] Θ × Θ \Theta\times\Theta Θ × Θ
∑ t = 1 T ⟨ g t , D ϕ ( x t ) ⟩ ≤ Ψ ( θ 1 ) − Ψ ( θ T + 1 ) η + L η m ∑ t = 1 T ∥ g t ∥ 2 . (7) \sum_{t=1}^{T} \langle g_t,D_{\phi}(x_t)\rangle\le\frac{\Psi(\theta_1)-\Psi(\theta_{T+1})}{\eta}+\frac{L\eta}{m}\sum_{t=1}^{T}\lVert g_t\rVert^2. \tag*{(7)} t = 1 ∑ T ⟨ g t , D ϕ ( x t )⟩ ≤ η Ψ ( θ 1 ) − Ψ ( θ T + 1 ) + m L η t = 1 ∑ T ∥ g t ∥ 2 . (7) Consequently, suppose ∥ g t ∥ ≤ G \lVert g_t\rVert\le G ∥ g t ∥ ≤ G B B B
max 1 ≤ t ≤ T + 1 Ψ ( θ t ) − min 1 ≤ t ≤ T + 1 Ψ ( θ t ) ≤ B , \max_{1\le t\le T+1}\Psi(\theta_t)-\min_{1\le t\le T+1}\Psi(\theta_t)\le B, 1 ≤ t ≤ T + 1 max Ψ ( θ t ) − 1 ≤ t ≤ T + 1 min Ψ ( θ t ) ≤ B , Then choosing η ≳ 1 / T \eta\gtrsim1/\sqrt{T} η ≳ 1/ T O ( T ) O(\sqrt{T}) O ( T ) ϕ \phi ϕ B , L , G > 0 B,L,G>0 B , L , G > 0 η = m B / ( L G 2 T ) \eta=\sqrt{mB/(LG^2T)} η = m B / ( L G 2 T ) 2 G L B T / m 2G\sqrt{LBT/m} 2 G L B T / m osc Θ ( Ψ ) ≤ B \operatorname{osc}_{\Theta}(\Psi)\le B osc Θ ( Ψ ) ≤ B
The coefficient L / m L/m L / m relint ( K ) \operatorname{relint}(K) relint ( K ) § 8.4 , where we remove the Legendre assumption and analyze standard constrained mirror descent directly. The circulation converse is also developed in § 8.4 . In particular, Theorem 8.11 applies directly to standard constrained mirror descent, allows boundary iterates, and gives the corresponding lower bound without relying on the interior dual-coordinate recursion.
Remark 8.5. For the regret proof, it is enough that (6) hold along the realized trajectory, namely for ( θ t + 1 , θ t ) (\theta_{t+1},\theta_t) ( θ t + 1 , θ t ) Θ × Θ \Theta\times\Theta Θ × Θ Θ \Theta Θ
Bregman Proximal Regret is Mirror-exact
We next show that the Bregman proximal deviations of Cai et al. [ [11 ] are mirror-exact. A related Hessian-weighted integrability observation and discussion appear in Ahunbay [ [9 ] , but there the argument requires a steep, smooth regularizer and does not recover the full Bregman proximal guarantee of Cai et al. [ [11 ] . Here we first work in the Legendre interior and later, in § 8.4 , remove the Legendre assumption and allow boundary iterates. The proof is the mirror version of the Moreau-envelope identity (2) .
Let F : = f + ι K F:=f+\iota_K F := f + ι K ρ \rho ρ ρ ≥ 0 \rho\ge0 ρ ≥ 0 K K K x ∈ D x\in\mathcal{D} x ∈ D
prox F R ( x ) : = arg min y ∈ E { F ( y ) + D R ( y , x ) } , M F R ( x ) : = min y ∈ E { F ( y ) + D R ( y , x ) } . \operatorname{prox}^{R}_{F}(x):=\mathop{\arg\min}_{y\in E}\{F(y)+D_R(y,x)\},\qquad M^{R}_{F}(x):=\min_{y\in E}\{F(y)+D_R(y,x)\}. prox F R ( x ) := arg min y ∈ E { F ( y ) + D R ( y , x )} , M F R ( x ) := y ∈ E min { F ( y ) + D R ( y , x )} . Because F F F ι K \iota_K ι K y ∈ K y\in K y ∈ K
Theorem 8.6. Assume the standing Legendre-domain conditions. Let F : = f + ι K F:=f+\iota_K F := f + ι K prox F R \operatorname{prox}^{R}_{F} prox F R D \mathcal{D} D ϕ F R : D → K \phi^{R}_{F}:\mathcal{D}\to K ϕ F R : D → K ϕ F R ( x ) : = prox F R ( x ) \phi^{R}_{F}(x):=\operatorname{prox}^{R}_{F}(x) ϕ F R ( x ) := prox F R ( x ) R R R
( x − prox F R ( x ) ) T ∇ R ( x ) = d M F R ( x ) . (8) \left(x-\operatorname{prox}^{R}_{F}(x)\right)^{\mathsf{T}}\nabla R(x)=\mathrm{d}M^{R}_{F}(x). \tag*{(8)} ( x − prox F R ( x ) ) T ∇ R ( x ) = d M F R ( x ) . (8) Thus Bregman proximal regret is a special case of mirror-exact regret.
We can now directly invoke Theorem 8.4 for this class of deviations in the following proposition.
Proposition 8.7. Assume the standing Legendre-domain conditions. Assume, in the extended-valued sense on E E E R R R m m m F : = f + ι K F := f + \iota_K F := f + ι K ρ \rho ρ 0 ≤ ρ < m 0 \le\rho< m 0 ≤ ρ < m H : = F + R H := F + R H := F + R D D D { x t } t = 1 T + 1 ⊂ D \{x_t\}_{t=1}^{T+1} \subset D { x t } t = 1 T + 1 ⊂ D (4) . Then these iterates control Bregman proximal regret with the bound
∑ t = 1 T ⟨ g t , x t − prox F R ( x t ) ⟩ ≤ Ψ F ( θ 1 ) − Ψ F ( θ T + 1 ) η + η m ∑ t = 1 T ∥ g t ∥ ∗ 2 , (9) \sum_{t=1}^{T} \langle g_t, x_t - \operatorname{prox}^{R}_{F}(x_t) \rangle\le\frac{\Psi_F(\theta_1)-\Psi_F(\theta_{T+1})}{\eta}+\frac{\eta}{m}\sum_{t=1}^{T}\lVert g_t\rVert_*^2, \tag*{(9)} t = 1 ∑ T ⟨ g t , x t − prox F R ( x t )⟩ ≤ η Ψ F ( θ 1 ) − Ψ F ( θ T + 1 ) + m η t = 1 ∑ T ∥ g t ∥ ∗ 2 , (9) where Ψ F ( θ t ) = M F R ( x t ) \Psi_F(\theta_t) = M_F^R(x_t) Ψ F ( θ t ) = M F R ( x t ) t ∈ [ T + 1 ] t \in[T+1] t ∈ [ T + 1 ]
The key idea behind this result is to show that weakly convex Bregman proximal potentials are ( L = 1 ) (L = 1) ( L = 1 ) R ∗ R^* R ∗ (6) , and hence Theorem 8.4 applies.
Full Mirror Characterization Beyond Legendre Regularizers
The preceding subsections use Legendre duality to regard θ : = ∇ R ( x ) \theta:= \nabla R(x) θ := ∇ R ( x )
D ϕ ( x ) T d ∇ R ( x ) . D_\phi(x)^{\mathsf T}\mathrm{d}\nabla R(x). D ϕ ( x ) T d ∇ R ( x ) . We now keep the standard constrained mirror-descent update (4) , allow boundary iterates, and prove matching upper and lower bounds without assuming that ∇ R \nabla R ∇ R R R R
Let K K K ( E , ∥ ⋅ ∥ ) (E,\lVert\cdot\rVert) ( E , ∥ ⋅ ∥) ∥ ⋅ ∥ ∗ \lVert\cdot\rVert_* ∥ ⋅ ∥ ∗ R R R K K K m m m K K K
R ( y ) ≥ R ( x ) + ⟨ ∇ R ( x ) , y − x ⟩ + m 2 ∥ y − x ∥ 2 ( x , y ∈ K ) . R(y) \ge R(x) + \langle\nabla R(x), y-x\rangle+ \frac{m}{2}\lVert y-x\rVert^2 \qquad(x,y \in K). R ( y ) ≥ R ( x ) + ⟨ ∇ R ( x ) , y − x ⟩ + 2 m ∥ y − x ∥ 2 ( x , y ∈ K ) . The gradient is uniformly continuous on K K K
ω R ( δ ) : = sup { ∥ ∇ R ( x ) − ∇ R ( y ) ∥ ∗ : x , y ∈ K , ∥ x − y ∥ ≤ δ } . \omega_R(\delta) := \sup\{\lVert\nabla R(x)-\nabla R(y)\rVert_* : x,y \in K,\ \lVert x-y\rVert\le\delta\}. ω R ( δ ) := sup {∥ ∇ R ( x ) − ∇ R ( y ) ∥ ∗ : x , y ∈ K , ∥ x − y ∥ ≤ δ } . Thus ω R ( δ ) → 0 \omega_R(\delta) \to0 ω R ( δ ) → 0 δ ↓ 0 \delta\downarrow0 δ ↓ 0
The same exactness condition extends beyond the Legendre setting by defining the potential directly on a neighborhood of ∇ R ( K ) \nabla R(K) ∇ R ( K )
Definition 8.8. A feasible deviation ϕ : K → K \phi: K \to K ϕ : K → K R R R Ω ⊇ ∇ R ( K ) \Omega\supseteq\nabla R(K) Ω ⊇ ∇ R ( K ) Ψ : Ω → R \Psi: \Omega\to\mathbb{R} Ψ : Ω → R
D ϕ ( x ) = ∇ θ Ψ ( ∇ R ( x ) ) ( x ∈ K ) . D_\phi(x) = \nabla_\theta\Psi(\nabla R(x)) \qquad(x \in K). D ϕ ( x ) = ∇ θ Ψ ( ∇ R ( x )) ( x ∈ K ) . We denote this class by Φ e x a c t R ( K ) \Phi_{\mathrm{exact}}^R(K) Φ exact R ( K ) D D D R R R V = Ψ ∘ ∇ R V = \Psi\circ\nabla R V = Ψ ∘ ∇ R
∇ V ( x ) = ∇ 2 R ( x ) D ϕ ( x ) , \nabla V(x) = \nabla^2 R(x)D_\phi(x), ∇ V ( x ) = ∇ 2 R ( x ) D ϕ ( x ) , so Definition 8.8 is equivalent to exactness of D ϕ ( x ) T d ∇ R ( x ) D_\phi(x)^{\mathsf T}\mathrm{d}\nabla R(x) D ϕ ( x ) T d ∇ R ( x )
The discrete potential argument requires a one-sided curvature bound. We assume that, for some L ≥ 0 L \ge0 L ≥ 0
D Ψ ( ∇ R ( y ) , ∇ R ( x ) ) ≤ L D R ( x , y ) ( x , y ∈ K ) . (10) D_\Psi(\nabla R(y), \nabla R(x)) \le L D_R(x,y) \qquad(x,y \in K). \tag*{(10)} D Ψ ( ∇ R ( y ) , ∇ R ( x )) ≤ L D R ( x , y ) ( x , y ∈ K ) . (10) Equivalently,
Ψ ( ∇ R ( y ) ) ≤ Ψ ( ∇ R ( x ) ) + ⟨ D ϕ ( x ) , ∇ R ( y ) − ∇ R ( x ) ⟩ + L D R ( x , y ) . \Psi(\nabla R(y)) \le\Psi(\nabla R(x)) + \langle D_\phi(x), \nabla R(y)-\nabla R(x)\rangle+ L D_R(x,y). Ψ ( ∇ R ( y )) ≤ Ψ ( ∇ R ( x )) + ⟨ D ϕ ( x ) , ∇ R ( y ) − ∇ R ( x )⟩ + L D R ( x , y ) . In the Legendre interior, D R ( x , y ) = D R ∗ ( ∇ R ( y ) , ∇ R ( x ) ) D_R(x,y) = D_{R^*}(\nabla R(y), \nabla R(x)) D R ( x , y ) = D R ∗ ( ∇ R ( y ) , ∇ R ( x )) Theorem 8.4 .
These ingredients yield the corresponding exact-form regret bound for standard constrained mirror descent.
Theorem 8.9. Assume R R R K K K X = K \mathcal{X}=K X = K { x t } t = 1 T + 1 \{x_t\}_{t=1}^{T+1} { x t } t = 1 T + 1 (4) with a constant step size η > 0 \eta>0 η > 0 ϕ : K → K \phi:K\to K ϕ : K → K R R R Ψ \Psi Ψ (10) holds and D ϕ D_\phi D ϕ L ϕ L_\phi L ϕ K K K
∑ t = 1 T ⟨ g t , D ϕ ( x t ) ⟩ ≤ Ψ ( ∇ R ( x 1 ) ) − Ψ ( ∇ R ( x T + 1 ) ) η + L η m ∑ t = 1 T ∥ g t ∥ ∗ 2 + L ϕ m ∑ t = 1 T ∥ g t ∥ ∗ [ η ∥ g t ∥ ∗ + ω R ( η m ∥ g t ∥ ∗ ) ] . \sum_{t=1}^{T}\langle g_t,D_\phi(x_t)\rangle\leq\frac{\Psi(\nabla R(x_1))-\Psi(\nabla R(x_{T+1}))}{\eta}+\frac{L\eta}{m}\sum_{t=1}^{T}\|g_t\|_*^2
+\frac{L_\phi}{m}\sum_{t=1}^{T}\|g_t\|_*\left[\eta\|g_t\|_*+\omega_R\left(\frac{\eta}{m}\|g_t\|_*\right)\right]. t = 1 ∑ T ⟨ g t , D ϕ ( x t )⟩ ≤ η Ψ ( ∇ R ( x 1 )) − Ψ ( ∇ R ( x T + 1 )) + m L η t = 1 ∑ T ∥ g t ∥ ∗ 2 + m L ϕ t = 1 ∑ T ∥ g t ∥ ∗ [ η ∥ g t ∥ ∗ + ω R ( m η ∥ g t ∥ ∗ ) ] . The first line is the same potential estimate as in the Legendre interior. The second line controls the boundary normal-cone residual. Feasibility gives the correct sign at x t + 1 x_{t+1} x t + 1 x t x_t x t
The preceding bound immediately gives sublinear regret under bounded gradients and bounded potential oscillation. When ∇ R \nabla R ∇ R K K K O ( T ) O(\sqrt{T}) O ( T )
Corollary 8.10. Assume the hypotheses of Theorem 8.9, ∥ g t ∥ ∗ ≤ G \|g_t\|_*\leq G ∥ g t ∥ ∗ ≤ G osc ∇ R ( K ) ( Ψ ) ≤ B \operatorname{osc}_{\nabla R(K)}(\Psi)\leq B osc ∇ R ( K ) ( Ψ ) ≤ B η = T − 1 / 2 \eta=T^{-1/2} η = T − 1/2
1 T ∑ t = 1 T ⟨ g t , D ϕ ( x t ) ⟩ ≤ B T + ( L + L ϕ ) G 2 m T + L ϕ G m ω R ( G m T ) . \frac{1}{T}\sum_{t=1}^{T}\langle g_t,D_\phi(x_t)\rangle\leq\frac{B}{\sqrt{T}}+\frac{(L+L_\phi)G^2}{m\sqrt{T}}+\frac{L_\phi G}{m}\omega_R\left(\frac{G}{m\sqrt{T}}\right). T 1 t = 1 ∑ T ⟨ g t , D ϕ ( x t )⟩ ≤ T B + m T ( L + L ϕ ) G 2 + m L ϕ G ω R ( m T G ) . Consequently, mirror descent has o ( T ) o(T) o ( T ) ϕ \phi ϕ ∇ R \nabla R ∇ R M M M K K K
∑ t = 1 T ⟨ g t , D ϕ ( x t ) ⟩ ≥ B η + C R , ϕ η G 2 T , C R , ϕ : = L + L ϕ m + L ϕ M m 2 . \sum_{t=1}^{T}\langle g_t,D_\phi(x_t)\rangle\geq\frac{B}{\eta}+C_{R,\phi}\eta G^2T,\qquad C_{R,\phi}:=\frac{L+L_\phi}{m}+\frac{L_\phi M}{m^2}. t = 1 ∑ T ⟨ g t , D ϕ ( x t )⟩ ≥ η B + C R , ϕ η G 2 T , C R , ϕ := m L + L ϕ + m 2 L ϕ M . The optimized bound is 2 G B C R , ϕ T 2G\sqrt{BC_{R,\phi}T} 2 G B C R , ϕ T B C R , ϕ > 0 BC_{R,\phi}>0 B C R , ϕ > 0
We next establish necessity of exactness at the geometric level and show how nonzero circulation obstructs sublinear regret. Let D : K → E D:K\to E D : K → E γ : [ 0 , 1 ] → K \gamma:[0,1]\to K γ : [ 0 , 1 ] → K C 1 C^1 C 1 ∇ R ∘ γ \nabla R\circ\gamma ∇ R ∘ γ C 1 C^1 C 1 D D D γ \gamma γ
∮ γ D ( x ) T d ∇ R ( x ) : = ∫ 0 1 ⟨ D ( γ ( s ) ) , d d s ∇ R ( γ ( s ) ) ⟩ d s . (11) \oint_\gamma D(x)^{\mathsf T}\mathrm{d}\nabla R(x):=\int_0^1\left\langle D(\gamma(s)),\frac{\mathrm{d}}{\mathrm{d}s}\nabla R(\gamma(s))\right\rangle\mathrm{d}s. \tag*{(11)} ∮ γ D ( x ) T d ∇ R ( x ) := ∫ 0 1 ⟨ D ( γ ( s )) , d s d ∇ R ( γ ( s )) ⟩ d s . (11) By the chain rule, every R R R
Theorem 8.11. Suppose that, after reversing the orientation of γ \gamma γ
− ∮ γ D ( x ) T d ∇ R ( x ) : = c 0 > 0. -\oint_\gamma D(x)^{\mathsf T}\mathrm{d}\nabla R(x):=c_0>0. − ∮ γ D ( x ) T d ∇ R ( x ) := c 0 > 0. For every G > 0 G>0 G > 0 η > 0 \eta>0 η > 0 ∥ g t ∥ ∗ ≤ G \|g_t\|_*\leq G ∥ g t ∥ ∗ ≤ G γ ( 0 ) \gamma(0) γ ( 0 )
∑ t = 1 T ⟨ g t , D ( x t ) ⟩ ≥ c T − O ( 1 / η ) \sum_{t=1}^{T}\langle g_t,D(x_t)\rangle\geq cT-O(1/\eta) t = 1 ∑ T ⟨ g t , D ( x t )⟩ ≥ c T − O ( 1/ η ) for a constant c > 0 c>0 c > 0 D D D γ \gamma γ R R R
The adversary discretizes the loop in the variables ∇ R ( x ) \nabla R(x) ∇ R ( x ) z j , z j + 1 z_j,z_{j+1} z j , z j + 1
g j + 1 = ∇ R ( z j ) − ∇ R ( z j + 1 ) η . g_{j+1}=\frac{\nabla R(z_j)-\nabla R(z_{j+1})}{\eta}. g j + 1 = η ∇ R ( z j ) − ∇ R ( z j + 1 ) . The derivative of the mirror-descent objective vanishes at z j + 1 z_{j+1} z j + 1 z j + 1 z_{j+1} z j + 1 η \eta η
Combining the exactness, cross-derivative, and circulation viewpoints gives the following characterization for standard constrained mirror descent.
Corollary 8.12. Let U : = relint ( K ) U := \operatorname{relint}(K) U := relint ( K ) U U U aff ( K ) \operatorname{aff}(K) aff ( K ) R ∈ C 2 ( U ) R \in C^2(U) R ∈ C 2 ( U ) ∇ 2 R ( x ) \nabla^2R(x) ∇ 2 R ( x ) x ∈ U x \in U x ∈ U x ↦ ∇ 2 R ( x ) D ( x ) x \mapsto\nabla^2R(x)D(x) x ↦ ∇ 2 R ( x ) D ( x ) C 1 C^1 C 1 aff ( K ) \operatorname{aff}(K) aff ( K )
(i) The one-form D ( x ) T d ∇ R ( x ) D(x)^{\mathsf T}\mathrm{d}\nabla R(x) D ( x ) T d ∇ R ( x ) U U U
(ii) For every coordinate pair i , j i,j i , j
∂ j [ ∇ 2 R ( x ) D ( x ) ] i = ∂ i [ ∇ 2 R ( x ) D ( x ) ] j ( x ∈ U ) . \partial_j\left[\nabla^2R(x)D(x)\right]_i=\partial_i\left[\nabla^2R(x)D(x)\right]_j \qquad(x \in U). ∂ j [ ∇ 2 R ( x ) D ( x ) ] i = ∂ i [ ∇ 2 R ( x ) D ( x ) ] j ( x ∈ U ) . (iii) The mirror circulation vanishes on every closed piecewise C 1 C^1 C 1 U U U
If these conditions fail, standard mirror descent can be forced to incur linear regret along a sufficiently small loop contained in U U U D D D D ϕ D_\phi D ϕ K K K O ( T ) O(\sqrt{T}) O ( T )
Exactness is therefore the sharp geometric boundary for standard constrained mirror descent under the stated regularity. Smoothness and potential oscillation determine the quantitative rate, while nonzero circulation rules out no regret. Proximality plays no role in this characterization. It is addressed next as a separate representation problem.
Identity Interpolation of Bregman Proximal Deviations
Theorem 6.2 has two consequences in Euclidean geometry. First, interpolation with the identity scales the displacement exactly. Second, every smooth exact displacement has a sufficiently small interpolation that is proximal. The first fact is purely algebraic and survives for every mirror map. The second depends on the curvature of the regularizer and can fail even for a smooth Legendre geometry. This subsection separates these two statements.
The representation and interpolation arguments do not require compactness. Let K K K R : E → ( − ∞ , + ∞ ] R:E\to(-\infty,+\infty] R : E → ( − ∞ , + ∞ ] m m m X ⊆ K ∩ dom R \mathcal{X}\subseteq K\cap\operatorname{dom}R X ⊆ K ∩ dom R R R R X = K \mathcal{X}=K X = K R R R K K K X = D \mathcal{X}=D X = D
For a proper lower-semicontinuous function F : = f + ι K F:=f+\iota_K F := f + ι K F + R F+R F + R dom F ∩ dom R ≠ ∅ \operatorname{dom}F\cap\operatorname{dom}R\ne\varnothing dom F ∩ dom R = ∅
prox F R ( x ) : = arg min y ∈ E { F ( y ) + D R ( y , x ) } , x ∈ X . \operatorname{prox}^{R}_{F}(x):=\underset{y\in E}{\arg\min}\left\{F(y)+D_R(y,x)\right\},\qquad x\in\mathcal{X}. prox F R ( x ) := y ∈ E arg min { F ( y ) + D R ( y , x ) } , x ∈ X . The indicator in F F F K K K F F F ρ \rho ρ 0 ≤ ρ < m 0\leq\rho<m 0 ≤ ρ < m ( m − ρ ) (m-\rho) ( m − ρ )
Definition 8.13. For a specified input domain X ⊆ K ∩ dom R \mathcal{X} \subseteq K \cap\operatorname{dom} R X ⊆ K ∩ dom R
Φ p r o x R ( X ) : = { prox F R ∣ X : F = f + ι K is proper, lower semicontinuous, and ρ -weakly convex for some 0 ≤ ρ < m } , \Phi_{\mathrm{prox}}^{R}(\mathcal{X}) := \left\{ \operatorname{prox}_{F}^{R}\big|_{\mathcal{X}} : \begin{array}{l} F=f+\iota_{K}\text{ is proper, lower semicontinuous,}\\ \text{and $\rho$-weakly convex for some $0\leq\rho<m$} \end{array} \right\}, Φ prox R ( X ) := { prox F R X : F = f + ι K is proper, lower semicontinuous, and ρ -weakly convex for some 0 ≤ ρ < m } , Φ r a d R ( X ) : = Rad I ( Φ p r o x R ( X ) ) . \Phi_{\mathrm{rad}}^{R}(\mathcal{X}) := \operatorname{Rad}_{I}\left(\Phi_{\mathrm{prox}}^{R}(\mathcal{X})\right). Φ rad R ( X ) := Rad I ( Φ prox R ( X ) ) . Thus ϕ ∈ Φ r a d R ( X ) \phi\in\Phi_{\mathrm{rad}}^{R}(\mathcal{X}) ϕ ∈ Φ rad R ( X ) ϕ α : = ( 1 − α ) I + α ϕ \phi_{\alpha}:=(1-\alpha)I+\alpha\phi ϕ α := ( 1 − α ) I + α ϕ Φ p r o x R ( X ) \Phi_{\mathrm{prox}}^{R}(\mathcal{X}) Φ prox R ( X ) α ∈ ( 0 , 1 ] \alpha\in(0,1] α ∈ ( 0 , 1 ]
For the constrained regularizer, we write R K : = R + ι K R_{K}:=R+\iota_{K} R K := R + ι K R R R
The Bregman proximal-regret guarantee of Cai et al. [ [11 ] applies to standard constrained mirror descent and does not require Legendre duality. We therefore take that guarantee as given and ask how the corresponding deviation class relates to mirror exactness. In short, we show that radial proximal deviations are mirror-exact.
Proposition 8.14. Assume X = K \mathcal{X}=K X = K F F F Definition 8.13 , the function Ψ F ( θ ) : = R K ∗ ( θ ) − ( F + R ) ∗ ( θ ) \Psi_{F}(\theta):=R_{K}^{*}(\theta)-(F+R)^{*}(\theta) Ψ F ( θ ) := R K ∗ ( θ ) − ( F + R ) ∗ ( θ )
x − prox F R ( x ) = ∇ θ Ψ F ( ∇ R ( x ) ) ( x ∈ K ) . x-\operatorname{prox}_{F}^{R}(x)=\nabla_{\theta}\Psi_{F}(\nabla R(x)) \qquad(x\in K). x − prox F R ( x ) = ∇ θ Ψ F ( ∇ R ( x )) ( x ∈ K ) . Consequently, Φ p r o x R ( K ) ⊆ Φ r a d R ( K ) ⊆ Φ e x a c t R ( K ) \Phi_{\mathrm{prox}}^{R}(K)\subseteq\Phi_{\mathrm{rad}}^{R}(K)\subseteq\Phi_{\mathrm{exact}}^{R}(K) Φ prox R ( K ) ⊆ Φ rad R ( K ) ⊆ Φ exact R ( K ) Φ e x a c t R ( K ) \Phi_{\mathrm{exact}}^{R}(K) Φ exact R ( K ) ϕ α = prox F R \phi_{\alpha}=\operatorname{prox}_{F}^{R} ϕ α = prox F R ϕ \phi ϕ Ψ F / α \Psi_{F}/\alpha Ψ F / α
The first inclusion follows by taking α = 1 \alpha=1 α = 1 ϕ α = prox F R \phi_{\alpha}=\operatorname{prox}_{F}^{R} ϕ α = prox F R x − ϕ α ( x ) = α ( x − ϕ ( x ) ) x-\phi_{\alpha}(x)=\alpha(x-\phi(x)) x − ϕ α ( x ) = α ( x − ϕ ( x )) ϕ α \phi_{\alpha} ϕ α ϕ \phi ϕ [11 ] . For linearized regret, Reg T ( ϕ ) = α − 1 Reg T ( ϕ α ) \operatorname{Reg}_{T}(\phi)=\alpha^{-1}\operatorname{Reg}_{T}(\phi_{\alpha}) Reg T ( ϕ ) = α − 1 Reg T ( ϕ α ) ℓ ( x ) − ℓ ( ϕ α ( x ) ) ≥ α [ ℓ ( x ) − ℓ ( ϕ ( x ) ) ] \ell(x)-\ell(\phi_{\alpha}(x))\geq\alpha[\ell(x)-\ell(\phi(x))] ℓ ( x ) − ℓ ( ϕ α ( x )) ≥ α [ ℓ ( x ) − ℓ ( ϕ ( x ))] Φ r a d R ( K ) \Phi_{\mathrm{rad}}^{R}(K) Φ rad R ( K )
Thus standard mirror descent controls the identity-radial closure of the Bregman proximal class. The remaining question is whether this radial class exhausts the mirror-exact class. Theorem 6.2 gives an affirmative answer for smooth Euclidean exact deviations. We now show that the corresponding statement can fail even for Legendre regularizers. It is therefore enough to return temporarily to the Legendre setting, where duality provides a convenient test for when a mirror-exact deviation can admit a Bregman proximal interpolation.
The Legendre test. We now return to the Legendre setting to test when a mirror-exact deviation admits a Bregman proximal interpolation. We use the assumptions of the first three subsections, except that no C 2 C^{2} C 2 R R R dom R ‾ = K \overline{\operatorname{dom}R}=K dom R = K m m m D : = int E ( dom R ) \mathcal{D}:=\operatorname{int}_{E}(\operatorname{dom}R) D := int E ( dom R ) Θ : = int E ( dom R ∗ ) \Theta:=\operatorname{int}_{E}(\operatorname{dom}R^{*}) Θ := int E ( dom R ∗ ) ∇ R \nabla R ∇ R ∇ R ∗ \nabla R^{*} ∇ R ∗
Let ϕ : D → K \phi:\mathcal{D}\to K ϕ : D → K R R R Ψ \Psi Ψ α ∈ ( 0 , 1 ] \alpha\in(0,1] α ∈ ( 0 , 1 ]
u α ( θ ) : = R ∗ ( θ ) − α Ψ ( θ ) . u_{\alpha}(\theta):=R^{*}(\theta)-\alpha\Psi(\theta). u α ( θ ) := R ∗ ( θ ) − α Ψ ( θ ) . If x = ∇ R ∗ ( θ ) x=\nabla R^{*}(\theta) x = ∇ R ∗ ( θ ) ϕ α \phi_{\alpha} ϕ α
ϕ α ( ∇ R ∗ ( θ ) ) = ∇ u α ( θ ) . \phi_{\alpha}(\nabla R^{*}(\theta))=\nabla u_{\alpha}(\theta). ϕ α ( ∇ R ∗ ( θ )) = ∇ u α ( θ ) . Thus, in dual coordinates, the interpolated map ϕ α \phi_{\alpha} ϕ α u α u_{\alpha} u α ϕ α \phi_{\alpha} ϕ α
Proposition 8.15. Suppose ϕ α = prox F α R \phi_{\alpha}=\operatorname{prox}_{F_{\alpha}}^{R} ϕ α = prox F α R D \mathcal{D} D F α F_{\alpha} F α u α u_{\alpha} u α F α + R F_{\alpha}+R F α + R
precisely, there is a constant c α ∈ R c_\alpha\in\mathbb{R} c α ∈ R u α = ( F α + R ) ∗ + c α u_\alpha= (F_\alpha+ R)^* + c_\alpha u α = ( F α + R ) ∗ + c α Θ \Theta Θ u α u_\alpha u α
D Ψ ( θ ′ , θ ) ≤ 1 α D R ∗ ( θ ′ , θ ) ( θ , θ ′ ∈ Θ ) . D_\Psi(\theta',\theta) \le\frac{1}{\alpha}D_{R^*}(\theta',\theta) \qquad(\theta,\theta' \in\Theta). D Ψ ( θ ′ , θ ) ≤ α 1 D R ∗ ( θ ′ , θ ) ( θ , θ ′ ∈ Θ ) . Under the domain assumptions of Proposition 8.16, convexity of u α u_\alpha u α (6) used in the mirror-exact regret bound, since any proximal interpolation necessarily satisfies such a condition with constant 1 / α 1/\alpha 1/ α F α F_\alpha F α
Proposition 8.16. Suppose u α u_\alpha u α Θ \Theta Θ
u ‾ α : = cl ( u α + ι Θ ) . \overline{u}_\alpha:= \operatorname{cl}(u_\alpha+ \iota_\Theta). u α := cl ( u α + ι Θ ) . Assume u ‾ α \overline{u}_\alpha u α ϕ α ( D ) ⊆ dom R \phi_\alpha(\mathcal{D}) \subseteq\operatorname{dom} R ϕ α ( D ) ⊆ dom R
F α ( y ) : = { u ‾ α ∗ ( y ) − R ( y ) , y ∈ dom R , + ∞ , y ∉ dom R . F_\alpha(y) :=
\begin{cases}
\overline{u}_\alpha^*(y) - R(y), & y \in\operatorname{dom} R,\\
+\infty, & y \notin\operatorname{dom} R.
\end{cases} F α ( y ) := { u α ∗ ( y ) − R ( y ) , + ∞ , y ∈ dom R , y ∈ / dom R . Then, for every x ∈ D x \in\mathcal{D} x ∈ D
ϕ α ( x ) ∈ arg min y { F α ( y ) + D R ( y , x ) } . \phi_\alpha(x) \in\underset{y}{\operatorname{arg\,min}}\{F_\alpha(y) + D_R(y,x)\}. ϕ α ( x ) ∈ y arg min { F α ( y ) + D R ( y , x )} . Under these domain assumptions, convexity of u α u_\alpha u α ϕ α \phi_\alpha ϕ α F α F_\alpha F α ρ α \rho_\alpha ρ α ρ α < m \rho_\alpha< m ρ α < m ϕ α = prox F α R \phi_\alpha= \operatorname{prox}^{R}_{F_\alpha} ϕ α = prox F α R ϕ ∈ Φ r a d R ( D ) \phi\in\Phi_{\mathrm{rad}}^R(\mathcal{D}) ϕ ∈ Φ rad R ( D )
Propositions 8.15 and 8.16 separate proximal representability into two requirements. First, R ∗ − α Ψ R^* - \alpha\Psi R ∗ − α Ψ
We begin with the first requirement. Define
L R ( Ψ ) : = inf { L ≥ 0 : D Ψ ( θ ′ , θ ) ≤ L D R ∗ ( θ ′ , θ ) for all θ , θ ′ ∈ Θ } . L_R(\Psi) := \inf\{L \ge0 : D_\Psi(\theta',\theta) \le L D_{R^*}(\theta',\theta) \text{ for all } \theta,\theta' \in\Theta\}. L R ( Ψ ) := inf { L ≥ 0 : D Ψ ( θ ′ , θ ) ≤ L D R ∗ ( θ ′ , θ ) for all θ , θ ′ ∈ Θ } . This quantity compares the curvature of Ψ \Psi Ψ R ∗ R^* R ∗
Corollary 8.17. For 0 < α ≤ 1 0 < \alpha\le1 0 < α ≤ 1 u α = R ∗ − α Ψ u_\alpha= R^* - \alpha\Psi u α = R ∗ − α Ψ α L R ( Ψ ) ≤ 1 \alpha L_R(\Psi) \le1 α L R ( Ψ ) ≤ 1
α c v x : = min { 1 , 1 L R ( Ψ ) } , \alpha_{\mathrm{cvx}} := \min\left\{1,\frac{1}{L_R(\Psi)}\right\}, α cvx := min { 1 , L R ( Ψ ) 1 } , with the conventions 1 / 0 : = + ∞ 1/0 := +\infty 1/0 := + ∞ 1 / ( + ∞ ) : = 0 1/(+\infty) := 0 1/ ( + ∞ ) := 0 0 < α ≤ α c v x 0 < \alpha\le\alpha_{\mathrm{cvx}} 0 < α ≤ α cvx α \alpha α u α u_\alpha u α ϕ α ( D ) ⊆ dom R \phi_\alpha(\mathcal{D}) \subseteq\operatorname{dom} R ϕ α ( D ) ⊆ dom R F α F_\alpha F α
The endpoint-domain condition is automatic for 0 < α < 1 0 < \alpha< 1 0 < α < 1 D = int E ( dom R ) \mathcal{D} = \operatorname{int}_E(\operatorname{dom} R) D = int E ( dom R ) K = dom R ‾ K = \overline{\operatorname{dom} R} K = dom R K K K D \mathcal{D} D α = 1 \alpha= 1 α = 1 ϕ ( D ) ⊆ dom R \phi(\mathcal{D}) \subseteq\operatorname{dom} R ϕ ( D ) ⊆ dom R
When R ∗ R^* R ∗ Ψ \Psi Ψ
α ∇ 2 Ψ ( θ ) ⪯ ∇ 2 R ∗ ( θ ) ( θ ∈ Θ ) . \alpha\nabla^2\Psi(\theta) \preceq\nabla^2 R^*(\theta) \qquad(\theta\in\Theta). α ∇ 2 Ψ ( θ ) ⪯ ∇ 2 R ∗ ( θ ) ( θ ∈ Θ ) . Thus the curvature contributed by α Ψ \alpha\Psi α Ψ R ∗ R^* R ∗ ∇ Ψ \nabla\Psi ∇Ψ
We next consider the second requirement. Suppose R R R M M M u ‾ α \overline{u}_\alpha u α L α L_\alpha L α u ‾ α ∗ \overline{u}_\alpha^* u α ∗ 1 / L α 1/L_\alpha 1/ L α M M M R R R F α F_\alpha F α
ρ α : = ( M − 1 L α ) + \rho_\alpha:= \left(M-\frac{1}{L_\alpha}\right)_+ ρ α := ( M − L α 1 ) + weakly convex. Hence it is admissible when ρ α < m \rho_\alpha< m ρ α < m F α F_\alpha F α
In the Euclidean case M = m = 1 M=m=1 M = m = 1 u ‾ α \overline{u}_\alpha u α F α F_\alpha F α
The next example shows that failure can already occur at the first step. Although the dual potential is smooth with globally Lipschitz gradient, R ∗ − α Ψ R^*-\alpha\Psi R ∗ − α Ψ
Proposition 8.18. Let 1 < p < 2 1<p<2 1 < p < 2 q : = p / ( p − 1 ) > 2 q:=p/(p-1)>2 q := p / ( p − 1 ) > 2
R ( x ) : = 1 2 ∥ x ∥ p 2 ( x ∈ R 2 ) . R(x):=\frac{1}{2}\lVert x\rVert_p^2 \qquad(x\in\mathbb{R}^2). R ( x ) := 2 1 ∥ x ∥ p 2 ( x ∈ R 2 ) . There is a feasible R R R C ∞ C^\infty C ∞ Φ r a d R ( R 2 ) \Phi_{\mathrm{rad}}^R(\mathbb{R}^2) Φ rad R ( R 2 )
To see why, note that R ∗ ( θ ) = 1 2 ∥ θ ∥ q 2 R^*(\theta)=\frac{1}{2}\lVert\theta\rVert_q^2 R ∗ ( θ ) = 2 1 ∥ θ ∥ q 2
Ψ ( θ ) : = 1 2 θ [ 2 ] 2 , ϕ ( x ) : = x − ∇ θ Ψ ( ∇ R ( x ) ) . \Psi(\theta):=\frac{1}{2}\theta_{[2]}^2,\qquad\phi(x):=x-\nabla_\theta\Psi(\nabla R(x)). Ψ ( θ ) := 2 1 θ [ 2 ] 2 , ϕ ( x ) := x − ∇ θ Ψ ( ∇ R ( x )) . The action space is all of R 2 \mathbb{R}^2 R 2 ∇ Ψ \nabla\Psi ∇Ψ 1 1 1 θ = e 1 + t e 2 \theta=e_1+te_2 θ = e 1 + t e 2
u α ( e 1 + t e 2 ) = 1 2 ( 1 + ∣ t ∣ q ) 2 / q − α 2 t 2 . u_\alpha(e_1+te_2)=\frac{1}{2}(1+|t|^q)^{2/q}-\frac{\alpha}{2}t^2. u α ( e 1 + t e 2 ) = 2 1 ( 1 + ∣ t ∣ q ) 2/ q − 2 α t 2 . As t → 0 t\to0 t → 0
u α ( e 1 + t e 2 ) − u α ( e 1 ) = 1 q ∣ t ∣ q − α 2 t 2 + O ( ∣ t ∣ 2 q ) . u_\alpha(e_1+te_2)-u_\alpha(e_1)=\frac{1}{q}|t|^q-\frac{\alpha}{2}t^2+O(|t|^{2q}). u α ( e 1 + t e 2 ) − u α ( e 1 ) = q 1 ∣ t ∣ q − 2 α t 2 + O ( ∣ t ∣ 2 q ) . Because q > 2 q>2 q > 2 α > 0 \alpha>0 α > 0 t t t R ∗ − α Ψ R^*-\alpha\Psi R ∗ − α Ψ α \alpha α
The obstruction comes from a mismatch in curvature. At e 1 e_1 e 1 e 2 e_2 e 2 ∣ t ∣ q |t|^q ∣ t ∣ q Ψ \Psi Ψ Ψ \Psi Ψ 1 2 ∥ θ ∥ 2 2 \frac{1}{2}\lVert\theta\rVert_2^2 2 1 ∥ θ ∥ 2 2
Φ p r o x R ( R 2 ) ⊆ Φ r a d R ( R 2 ) ⊊ Φ e x a c t R ( R 2 ) \Phi_{\mathrm{prox}}^R(\mathbb{R}^2)\subseteq\Phi_{\mathrm{rad}}^R(\mathbb{R}^2)\subsetneq\Phi_{\mathrm{exact}}^R(\mathbb{R}^2) Φ prox R ( R 2 ) ⊆ Φ rad R ( R 2 ) ⊊ Φ exact R ( R 2 ) can hold even for a Legendre regularizer with a C ∞ C^\infty C ∞ C 2 C^2 C 2
Follow-the-Regularized-Leader Beyond Legendre Regularizers
In the Legendre interior, online mirror descent and follow-the-regularized-leader (FTRL) coincide for constant-step linearized losses [35 ] (McMahan 2011). Indeed, mirror descent satisfies
∇ R ( x t + 1 ) = ∇ R ( x t ) − η g t , \nabla R(x_{t+1}) = \nabla R(x_t) - \eta g_t, ∇ R ( x t + 1 ) = ∇ R ( x t ) − η g t , which unrolls to the FTRL update
x t : = arg min x ∈ E { η ∑ s = 1 t − 1 ⟨ g s , x ⟩ + R ( x ) − ⟨ ∇ R ( x 1 ) , x ⟩ } . x_t := \mathop{\arg\min}_{x\in E}\left\{\eta\sum_{s=1}^{t-1}\langle g_s,x\rangle+ R(x) - \langle\nabla R(x_1),x\rangle\right\}. x t := arg min x ∈ E { η s = 1 ∑ t − 1 ⟨ g s , x ⟩ + R ( x ) − ⟨ ∇ R ( x 1 ) , x ⟩ } . Beyond the Legendre interior, this equivalence need not persist. Boundary constraints introduce normal-cone terms into mirror descent, while FTRL continues to use the cumulative linear losses in a single regularized minimization; related distinctions between constrained mirror descent and cumulative-dual methods are discussed by Fang et al. [ [37 ] . In § 8.4 , we characterized the full deviation class controlled by standard constrained mirror descent beyond the Legendre setting. For FTRL, the relevant geometry is different because the algorithm evolves through its cumulative dual state rather than the one-step constrained mirror update. We therefore study FTRL separately and characterize the deviation class for which it has sublinear regret, together with the corresponding dual-circulation obstruction.
Coincidence with Mirror Descent in the Legendre Interior
The equivalence between mirror descent and follow-the-regularized-leader (FTRL) for constant-step linearized losses in the Legendre interior is standard; see, for example, McMahan [ [35 ] (2011). We record the precise form used here for completeness.
Proposition 9.1. Under the standing Legendre assumptions of § 8 , let θ 1 : = ∇ R ( x 1 ) \theta_1 := \nabla R(x_1) θ 1 := ∇ R ( x 1 ) η \eta η
x t = arg min x ∈ E { η ∑ s = 1 t − 1 ⟨ g s , x ⟩ + R ( x ) − ⟨ θ 1 , x ⟩ } = arg min x ∈ K { η ∑ s = 1 t − 1 ⟨ g s , x ⟩ + D R ( x , x 1 ) } . x_t = \mathop{\arg\min}_{x\in E}\left\{\eta\sum_{s=1}^{t-1}\langle g_s,x\rangle+ R(x) - \langle\theta_1,x\rangle\right\}
= \mathop{\arg\min}_{x\in K}\left\{\eta\sum_{s=1}^{t-1}\langle g_s,x\rangle+ D_R(x,x_1)\right\}. x t = arg min x ∈ E { η s = 1 ∑ t − 1 ⟨ g s , x ⟩ + R ( x ) − ⟨ θ 1 , x ⟩ } = arg min x ∈ K { η s = 1 ∑ t − 1 ⟨ g s , x ⟩ + D R ( x , x 1 ) } . Thus they are the FTRL iterates for the linearized losses and the tilted regularizer R − ⟨ θ 1 , ⋅ ⟩ R-\langle\theta_1,\cdot\rangle R − ⟨ θ 1 , ⋅ ⟩
Since the two algorithms generate the same iterates in this setting, the Legendre-interior regret guarantees established above for mirror descent, including the mirror-exact and Bregman proximal guarantees of § 8.2 and § 8.3 , apply verbatim to FTRL. The situation changes beyond the Legendre interior. Standard constrained mirror descent can acquire a boundary normal-cone residual, whereas FTRL continues to evolve through the cumulative linear losses. We analyzed the resulting deviation class for mirror descent in § 8.4 . We now carry out the corresponding analysis for FTRL and show that, once boundary effects are present, the two algorithms can control different deviation classes.
Exactness and Circulation in the Cumulative Dual State
Unlike constrained mirror descent, FTRL continues to evolve additively through its cumulative dual state beyond the Legendre setting. This suggests imposing exactness directly in that variable.
Let R : E → ( − ∞ , + ∞ ] R : E \to(-\infty,+\infty] R : E → ( − ∞ , + ∞ ] K : = dom R K := \operatorname{dom} R K := dom R R ∗ R^* R ∗ O : = int E ( dom R ∗ ) O := \operatorname{int}_E(\operatorname{dom} R^*) O := int E ( dom R ∗ ) θ 1 ∈ O \theta_1 \in O θ 1 ∈ O
θ t : = θ 1 − η ∑ s = 1 t − 1 g s , x t : = ∇ R ∗ ( θ t ) . (12) \theta_t := \theta_1 - \eta\sum_{s=1}^{t-1}g_s,\qquad x_t := \nabla R^*(\theta_t). \tag*{(12)} θ t := θ 1 − η s = 1 ∑ t − 1 g s , x t := ∇ R ∗ ( θ t ) . (12) Equivalently,
x t = arg min x ∈ E { η ∑ s = 1 t − 1 ⟨ g s , x ⟩ + R ( x ) − ⟨ θ 1 , x ⟩ } x_t = \mathop{\arg\min}_{x\in E}\left\{\eta\sum_{s=1}^{t-1}\langle g_s,x\rangle+ R(x) - \langle\theta_1,x\rangle\right\} x t = arg min x ∈ E { η s = 1 ∑ t − 1 ⟨ g s , x ⟩ + R ( x ) − ⟨ θ 1 , x ⟩ } Because R R R K K K
The relevant displacement is therefore the displacement viewed as a function of the cumulative dual state.
Definition 9.2 (FTRL-exact deviation). A feasible map ϕ : K → K \phi: K\to K ϕ : K → K R R R Ψ : Θ → R \Psi:\Theta\to\mathbb{R} Ψ : Θ → R
∇ R ∗ ( θ ) − ϕ ( ∇ R ∗ ( θ ) ) = ∇ θ Ψ ( θ ) ( θ ∈ Θ ) . \nabla R^*(\theta)-\phi(\nabla R^*(\theta))=\nabla_\theta\Psi(\theta)\qquad(\theta\in\Theta). ∇ R ∗ ( θ ) − ϕ ( ∇ R ∗ ( θ )) = ∇ θ Ψ ( θ ) ( θ ∈ Θ ) . When R R R
We denote this class by Φ F T R L R ( K ) \Phi_{\mathrm{FTRL}}^R(K) Φ FTRL R ( K )
D ~ ϕ ( θ ) : = ∇ R ∗ ( θ ) − ϕ ( ∇ R ∗ ( θ ) ) . \widetilde{D}_{\phi}(\theta):=\nabla R^*(\theta)-\phi(\nabla R^*(\theta)). D ϕ ( θ ) := ∇ R ∗ ( θ ) − ϕ ( ∇ R ∗ ( θ )) . Thus FTRL-exactness is ordinary exactness of D ~ ϕ \widetilde{D}_{\phi} D ϕ
To obtain a quantitative regret bound, we use uniform convexity of the regularizer. Let ∥ ⋅ ∥ \|\cdot\| ∥ ⋅ ∥ E E E ∥ ⋅ ∥ ∗ \|\cdot\|_* ∥ ⋅ ∥ ∗ R R R ( σ , r ) (\sigma,r) ( σ , r ) x , y ∈ dom R x,y\in\operatorname{dom}R x , y ∈ dom R θ ∈ ∂ R ( x ) \theta\in\partial R(x) θ ∈ ∂ R ( x )
R ( y ) ≥ R ( x ) + ⟨ θ , y − x ⟩ + σ r ∥ y − x ∥ r . (13) R(y)\ge R(x)+\langle\theta,y-x\rangle+\frac{\sigma}{r}\|y-x\|^r. \tag*{(13)} R ( y ) ≥ R ( x ) + ⟨ θ , y − x ⟩ + r σ ∥ y − x ∥ r . (13) Here σ > 0 \sigma>0 σ > 0 r > 1 r>1 r > 1 q : = r / ( r − 1 ) q:=r/(r-1) q := r / ( r − 1 )
Exactness makes the potential differences telescope, while relative smoothness and uniform convexity control the Bregman error incurred by each dual update. We are now ready to state and prove our exact-form regret result for FTRL.
Theorem 9.3. Assume that R R R ( σ , r ) (\sigma,r) ( σ , r ) R ∗ R^* R ∗ E E E (12) . If ϕ \phi ϕ R R R Ψ \Psi Ψ
D Ψ ( θ t + 1 , θ t ) ≤ L D R ∗ ( θ t + 1 , θ t ) ( t ∈ [ T ] ) , D_{\Psi}(\theta_{t+1},\theta_t)\le L D_{R^*}(\theta_{t+1},\theta_t)\qquad(t\in[T]), D Ψ ( θ t + 1 , θ t ) ≤ L D R ∗ ( θ t + 1 , θ t ) ( t ∈ [ T ]) , then
∑ t = 1 T ⟨ g t , x t − ϕ ( x t ) ⟩ ≤ Ψ ( θ 1 ) − Ψ ( θ T + 1 ) η + L η q − 1 q σ q − 1 ∑ t = 1 T ∥ g t ∥ ∗ q . \sum_{t=1}^{T}\langle g_t,x_t-\phi(x_t)\rangle
\le
\frac{\Psi(\theta_1)-\Psi(\theta_{T+1})}{\eta}
+
\frac{L\eta^{q-1}}{q\sigma^{q-1}}\sum_{t=1}^{T}\|g_t\|_*^q. t = 1 ∑ T ⟨ g t , x t − ϕ ( x t )⟩ ≤ η Ψ ( θ 1 ) − Ψ ( θ T + 1 ) + q σ q − 1 L η q − 1 t = 1 ∑ T ∥ g t ∥ ∗ q . As in the mirror-descent analysis and Condition (6) in § 8.2 , the relative-smoothness condition is needed only along the realized trajectory, namely on the consecutive pairs ( θ t + 1 , θ t ) (\theta_{t+1},\theta_t) ( θ t + 1 , θ t ) Θ × Θ \Theta\times\Theta Θ × Θ
Under bounded gradients and bounded oscillation of the potential, optimizing the preceding bound over the step size gives the following rate.
Corollary 9.4. Suppose ∥ g t ∥ ∗ ≤ G \|g_t\|_*\le G ∥ g t ∥ ∗ ≤ G Ψ \Psi Ψ B B B B , L , G > 0 B,L,G>0 B , L , G > 0
η = ( r B σ q − 1 L G q T ) 1 / q \eta=\left(\frac{rB\sigma^{q-1}}{LG^qT}\right)^{1/q} η = ( L G q T r B σ q − 1 ) 1/ q gives
∑ t = 1 T ⟨ g t , x t − ϕ ( x t ) ⟩ ≤ r 1 / r G L 1 / q B 1 / r σ 1 / r T 1 / q . \sum_{t=1}^{T}\langle g_t,x_t-\phi(x_t)\rangle
\le
r^{1/r}\frac{GL^{1/q}B^{1/r}}{\sigma^{1/r}}T^{1/q}. t = 1 ∑ T ⟨ g t , x t − ϕ ( x t )⟩ ≤ r 1/ r σ 1/ r G L 1/ q B 1/ r T 1/ q . For r = 2 r=2 r = 2 G 2 L B T / σ G\sqrt{2LBT/\sigma} G 2 L B T / σ
As in the mirror-descent converse of Theorem 8.11, the obstruction is nonzero circulation. Here the relevant loop lives in the cumulative dual state of FTRL. If the dual displacement has nonzero circulation, an adversary can force the iterates to repeatedly traverse such a loop and incur linear regret.
Theorem 9.5. Suppose D ~ ϕ \widetilde{D}_{\phi} D ϕ C 1 C^1 C 1 γ ⊂ Θ \gamma\subset\Theta γ ⊂ Θ
∮ γ ⟨ D ~ ϕ ( θ ) , d θ ⟩ ≠ 0. \oint_{\gamma} \left\langle\widetilde{D}_{\phi}(\theta),\mathrm{d}\theta\right\rangle\ne0. ∮ γ ⟨ D ϕ ( θ ) , d θ ⟩ = 0. For every G > 0 G > 0 G > 0 η > 0 \eta> 0 η > 0 ∥ g t ∥ ∗ ≤ G \lVert g_t\rVert_* \le G ∥ g t ∥ ∗ ≤ G (12) to incur c T − O ( 1 / η ) cT - O(1/\eta) c T − O ( 1/ η ) ϕ \phi ϕ c > 0 c > 0 c > 0 Θ \Theta Θ
Combining the upper and lower bounds gives the corresponding exactness-versus-circulation characterization for FTRL.
Corollary 9.6. Assume that D ~ ϕ \widetilde{D}_{\phi} D ϕ C 1 C^1 C 1 Θ \Theta Θ R R R
Mirror Descent and FTRL Control Different Deviation Classes
The equivalence in Proposition 9.1 relies on the Legendre interior. Once boundary points are allowed, the two algorithms use different dual information. Standard constrained mirror descent is governed by the primal mirror coordinate ∇ R ( x ) \nabla R(x) ∇ R ( x ) R K : = R + ι K R_K := R + \iota_K R K := R + ι K
Proposition 9.7. Let K K K R R R K K K ϕ : K → K \phi: K \to K ϕ : K → K R K R_K R K R R R
Φ F T R L R K ( K ) ⊆ Φ e x a c t R ( K ) . \Phi_{\mathrm{FTRL}}^{R_K}(K) \subseteq\Phi_{\mathrm{exact}}^R(K). Φ FTRL R K ( K ) ⊆ Φ exact R ( K ) . We next show that this inclusion can be strict, even for a smooth power-type regularizer. Fix 1 < p < ∞ 1 < p < \infty 1 < p < ∞ 0 < a < b 0 < a < b 0 < a < b
K : = [ a , b ] 2 , R p ( x ) : = 1 p ∥ x ∥ p p = 1 p ( x [ 1 ] p + x [ 2 ] p ) , R p , K : = R p + ι K . K := [a,b]^2,\qquad R_p(x) := \frac{1}{p}\lVert x\rVert_p^p = \frac{1}{p}\left(x[1]^p + x[2]^p\right),\qquad R_{p,K} := R_p + \iota_K. K := [ a , b ] 2 , R p ( x ) := p 1 ∥ x ∥ p p = p 1 ( x [ 1 ] p + x [ 2 ] p ) , R p , K := R p + ι K . Let c : = ( a + b ) / 2 c := (a+b)/2 c := ( a + b ) /2 β ∈ ( 0 , 1 ) \beta\in(0,1) β ∈ ( 0 , 1 )
S β : = ( 1 β β 1 ) , m p : = ( p − 1 ) min s ∈ [ a , b ] s p − 2 . S_\beta:= \begin{pmatrix} 1 & \beta\\ \beta& 1 \end{pmatrix},\qquad m_p := (p-1)\min_{s\in[a,b]} s^{p-2}. S β := ( 1 β β 1 ) , m p := ( p − 1 ) s ∈ [ a , b ] min s p − 2 . Choose 0 < ε ≤ m p ( 1 − β ) / ( 1 + β ) 0 < \varepsilon\le m_p(1-\beta)/(1+\beta) 0 < ε ≤ m p ( 1 − β ) / ( 1 + β )
D p , ε ( x ) : = ε [ ∇ 2 R p ( x ) ] − 1 S β ( x − c 1 ) , ϕ p , ε ( x ) : = x − D p , ε ( x ) . (14) D_{p,\varepsilon}(x) := \varepsilon\left[\nabla^2 R_p(x)\right]^{-1}S_\beta(x-c\mathbf{1}),\qquad\phi_{p,\varepsilon}(x) := x-D_{p,\varepsilon}(x). \tag*{(14)} D p , ε ( x ) := ε [ ∇ 2 R p ( x ) ] − 1 S β ( x − c 1 ) , ϕ p , ε ( x ) := x − D p , ε ( x ) . (14) The same deviation has opposite regret behavior under the two algorithms.
Theorem 9.8. The map ϕ p , ε \phi_{p,\varepsilon} ϕ p , ε R p R_p R p O ( T ) O(\sqrt{T}) O ( T ) T − 1 / 2 T^{-1/2} T − 1/2 R p , K R_{p,K} R p , K c T − O ( 1 / η ) cT - O(1/\eta) c T − O ( 1/ η ) Φ F T R L R p ( K ) ⊊ Φ e x a c t R p ( K ) \Phi_{\mathrm{FTRL}}^{R_p}(K) \subsetneq\Phi_{\mathrm{exact}}^{R_p}(K) Φ FTRL R p ( K ) ⊊ Φ exact R p ( K )
For mirror descent, exactness is visible directly in primal coordinates. The one-form satisfies
[ ∇ 2 R p ( x ) D ~ p , ε ( x ) ] T d x = d [ ε 2 ( x − c 1 ) T S β ( x − c 1 ) ] . \left[\nabla^2 R_p(x)\widetilde{D}_{p,\varepsilon}(x)\right]^{\mathsf{T}}dx
= d\left[\frac{\varepsilon}{2}(x-c\mathbf{1})^{\mathsf{T}}S_\beta(x-c\mathbf{1})\right]. [ ∇ 2 R p ( x ) D p , ε ( x ) ] T d x = d [ 2 ε ( x − c 1 ) T S β ( x − c 1 ) ] . All coefficients are smooth and bounded on the positive box, so the relative-curvature and Lipschitz conditions in Corollary 8.10 hold with finite constants. The mirror-descent regret bound therefore applies.
For FTRL, the obstruction appears when the cumulative dual state reaches a region where one coordinate is saturated at the boundary. For the extended regularizer, the choice map is coordinatewise
∇ R p , K ∗ ( θ ) [ i ] = { a , θ [ i ] ≤ a p − 1 , θ [ i ] 1 / ( p − 1 ) , a p − 1 < θ [ i ] < b p − 1 , b , θ [ i ] ≥ b p − 1 . \nabla R_{p,K}^{*}(\theta)[i] =
\begin{cases}
a, & \theta[i] \le a^{p-1},\\
\theta[i]^{1/(p-1)}, & a^{p-1} < \theta[i] < b^{p-1},\\
b, & \theta[i] \ge b^{p-1}.
\end{cases} ∇ R p , K ∗ ( θ ) [ i ] = ⎩ ⎨ ⎧ a , θ [ i ] 1/ ( p − 1 ) , b , θ [ i ] ≤ a p − 1 , a p − 1 < θ [ i ] < b p − 1 , θ [ i ] ≥ b p − 1 . On the open strip, we have
θ [ 1 ] > b p − 1 , a p − 1 < θ [ 2 ] < b p − 1 . \theta[1] > b^{p-1}, \qquad a^{p-1} < \theta[2] < b^{p-1}. θ [ 1 ] > b p − 1 , a p − 1 < θ [ 2 ] < b p − 1 . The dual displacement D ~ p , ε \widetilde{D}_{p,\varepsilon} D p , ε
∂ D ~ p , ε [ 1 ] ∂ θ [ 2 ] = ε β ( p − 1 ) 2 b 2 − p θ [ 2 ] ( 2 − p ) / ( p − 1 ) > 0 , ∂ D ~ p , ε [ 2 ] ∂ θ [ 1 ] = 0. \frac{\partial\widetilde{D}_{p,\varepsilon}[1]}{\partial\theta[2]}
= \frac{\varepsilon\beta}{(p-1)^2}b^{2-p}\theta[2]^{(2-p)/(p-1)} > 0,
\qquad
\frac{\partial\widetilde{D}_{p,\varepsilon}[2]}{\partial\theta[1]} = 0. ∂ θ [ 2 ] ∂ D p , ε [ 1 ] = ( p − 1 ) 2 εβ b 2 − p θ [ 2 ] ( 2 − p ) / ( p − 1 ) > 0 , ∂ θ [ 1 ] ∂ D p , ε [ 2 ] = 0. The unequal cross derivatives give nonzero dual circulation on a small rectangle in this strip. Hence Theorem 9.5 gives linear regret for FTRL, even though the same deviation is mirror-exact and has sublinear regret under standard constrained mirror descent. At p = 2 p=2 p = 2 (14) is affine and the example reduces to a symmetric cross-coordinate deviation on a box.
Examples and Applications of the FTRL Bound
We conclude the FTRL analysis with two standard power-type regularizers that illustrate how Theorem 9.3 specializes in concrete geometries. In both examples, K ⊆ R d K \subseteq\mathbb{R}^{d} K ⊆ R d R R R ϕ \phi ϕ L L L Ψ \Psi Ψ B B B
For squared ℓ p \ell_p ℓ p 1 < p ≤ 2 1 < p \le2 1 < p ≤ 2
R ( x ) ≔ 1 2 ∥ x ∥ p 2 + ι K ( x ) . R(x) \coloneqq\frac{1}{2}\lVert x\rVert_p^2 + \iota_K(x). R ( x ) : = 2 1 ∥ x ∥ p 2 + ι K ( x ) . Since this regularizer is ( p − 1 ) (p-1) ( p − 1 ) ∥ ⋅ ∥ p \lVert\cdot\rVert_p ∥ ⋅ ∥ p
Corollary 9.9. Let 1 < p ≤ 2 1 < p \le2 1 < p ≤ 2 q ≔ p / ( p − 1 ) q \coloneqq p/(p-1) q : = p / ( p − 1 )
∑ t = 1 T ⟨ g t , x t − ϕ ( x t ) ⟩ ≤ B η + L η 2 ( p − 1 ) ∑ t = 1 T ∥ g t ∥ q 2 . \sum_{t=1}^{T}\langle g_t,x_t-\phi(x_t)\rangle
\le\frac{B}{\eta}+\frac{L\eta}{2(p-1)}\sum_{t=1}^{T}\lVert g_t\rVert_q^2. t = 1 ∑ T ⟨ g t , x t − ϕ ( x t )⟩ ≤ η B + 2 ( p − 1 ) L η t = 1 ∑ T ∥ g t ∥ q 2 . If ∥ g t ∥ q ≤ G \lVert g_t\rVert_q \le G ∥ g t ∥ q ≤ G G 2 L B T / ( p − 1 ) G\sqrt{2LBT/(p-1)} G 2 L B T / ( p − 1 )
For p ≥ 2 p \ge2 p ≥ 2
R ( x ) ≔ 1 p ∥ x ∥ p p + ι K ( x ) . R(x) \coloneqq\frac{1}{p}\lVert x\rVert_p^p + \iota_K(x). R ( x ) : = p 1 ∥ x ∥ p p + ι K ( x ) . Its uniform convexity has exponent p p p
Corollary 9.10. Let 2 ≤ p < ∞ 2 \le p < \infty 2 ≤ p < ∞ q : = p / ( p − 1 ) q := p/(p - 1) q := p / ( p − 1 )
∑ t = 1 T ⟨ g t , x t − ϕ ( x t ) ⟩ ≤ B η + L η q − 1 q ( 2 2 − p ) q − 1 ∑ t = 1 T ∥ g t ∥ q q . \sum_{t=1}^{T} \langle g_t, x_t - \phi(x_t) \rangle\le\frac{B}{\eta} + \frac{L\eta^{q-1}}{q(2^{2-p})^{q-1}} \sum_{t=1}^{T} \lVert g_t\rVert_q^q. t = 1 ∑ T ⟨ g t , x t − ϕ ( x t )⟩ ≤ η B + q ( 2 2 − p ) q − 1 L η q − 1 t = 1 ∑ T ∥ g t ∥ q q . If ∥ g t ∥ q ≤ G \lVert g_t\rVert_q \le G ∥ g t ∥ q ≤ G p 1 / p 2 ( p − 2 ) / p G L 1 / q B 1 / p T 1 / q p^{1/p}2^{(p-2)/p}GL^{1/q}B^{1/p}T^{1/q} p 1/ p 2 ( p − 2 ) / p G L 1/ q B 1/ p T 1/ q O ( T 1 − 1 / p ) O(T^{1-1/p}) O ( T 1 − 1/ p )
These examples show how the geometry of the regularizer determines the quantitative FTRL rate through its uniform-convexity exponent. The one-homogeneous norm ∥ x ∥ p \lVert x\rVert_p ∥ x ∥ p p = 1 p = 1 p = 1 p = ∞ p = \infty p = ∞ ℓ p \ell_p ℓ p 1 < p < ∞ 1 < p < \infty 1 < p < ∞
Sections 8.4 and 9 give parallel characterizations for different algorithms. For standard constrained mirror descent, exactness of D ϕ ( x ) T ∇ R ( x ) D_\phi(\mathbf{x})^\mathsf{T}\nabla R(\mathbf{x}) D ϕ ( x ) T ∇ R ( x ) ℓ p \ell_p ℓ p
Conservative Correlated Equilibria in Convex Games
The regret guarantees developed above translate directly into equilibrium constraints in convex games. It is useful to distinguish three questions. The first concerns inclusion between deviation classes, the second concerns quantitative approximate equilibria for normalized families, and the third concerns equilibrium constraints when the approximation error is zero. In particular, a strict inclusion between deviation classes, such as the one in Proposition 6.1 , does not by itself imply a strict inclusion between the corresponding equilibrium sets.
Consider an N N N i i i K i K_i K i ℓ i ( x i , x − i ) \ell_i(x_i, x_{-i}) ℓ i ( x i , x − i ) K : = ∏ i = 1 N K i K := \prod_{i=1}^{N} K_i K := ∏ i = 1 N K i ℓ i : K → R \ell_i: K \to\mathbb{R} ℓ i : K → R K K K μ \mu μ ϕ i \phi_i ϕ i
Γ i ( μ , ϕ i ) : = E x ∼ μ [ ℓ i ( x i , x − i ) − ℓ i ( ϕ i ( x i ) , x − i ) ] . \Gamma_i(\mu,\phi_i) := \mathbb{E}_{x\sim\mu}\left[\ell_i(x_i,x_{-i})-\ell_i(\phi_i(x_i),x_{-i})\right]. Γ i ( μ , ϕ i ) := E x ∼ μ [ ℓ i ( x i , x − i ) − ℓ i ( ϕ i ( x i ) , x − i ) ] . The interpolation results in Sections 6 and 8.5 have the same consequence when the approximation error is zero. Passing from a deviation class to its identity-radial closure does not change the resulting equilibrium constraints.
Proposition 10.1. Let A i \mathcal{A}_i A i i i i Γ i ( μ , ϕ i ) ≤ 0 \Gamma_i(\mu,\phi_i) \le0 Γ i ( μ , ϕ i ) ≤ 0 ϕ i ∈ A i \phi_i \in\mathcal{A}_i ϕ i ∈ A i i i i ϕ i ∈ Rad ( A i ) \phi_i \in\operatorname{Rad}(\mathcal{A}_i) ϕ i ∈ Rad ( A i )
This equivalence concerns the equilibrium constraints themselves. Quantitative bounds over a radial closure additionally require a common lower bound on the admissible interpolation scales.
For each player i i i Φ i , p r o x \Phi_{i,\mathrm{prox}} Φ i , prox prox F ∣ K i \operatorname{prox}_F \mid K_i prox F ∣ K i F : = f + ι K i F := f + \iota_{K_i} F := f + ι K i ρ \rho ρ 0 ≤ ρ < 1 0 \le\rho< 1 0 ≤ ρ < 1
A distribution μ \mu μ coarse correlated equilibrium (CCE) if Γ i ( μ , τ i , u i ) ≤ 0 \Gamma_i(\mu,\tau_{i,u_i}) \le0 Γ i ( μ , τ i , u i ) ≤ 0 i i i τ i , u i ( x i ) ≡ u i ∈ K i \tau_{i,u_i}(x_i) \equiv u_i \in K_i τ i , u i ( x i ) ≡ u i ∈ K i
It is a proximal correlated equilibrium (PCE) if Γ i ( μ , τ i ) ≤ 0 \Gamma_i(\mu,\tau_i) \le0 Γ i ( μ , τ i ) ≤ 0 i i i τ i ∈ Φ i , p r o x \tau_i \in\Phi_{i,\mathrm{prox}} τ i ∈ Φ i , prox
It is a correlated equilibrium (CE) if Γ i ( μ , τ i ) ≤ 0 \Gamma_i(\mu,\tau_i) \le0 Γ i ( μ , τ i ) ≤ 0 i i i τ i : K i → K i \tau_i: K_i \to K_i τ i : K i → K i
Thus, under the unrestricted proximal convention used here,
C E ⊆ P C E ⊆ C C E . \mathrm{CE} \subseteq\mathrm{PCE} \subseteq\mathrm{CCE}. CE ⊆ PCE ⊆ CCE . The CE notion originates with Aumann [ [38 ] ; here we use its standard measurable-deviation extension to compact convex action spaces. The original PCE work introduced PCE as a tractable refinement of CCE [11 ] . We determine below how PCE compares with measurable-deviation CE. The two notions coincide on finite support, while C E ⊊ P C E \mathrm{CE} \subsetneq\mathrm{PCE} CE ⊊ PCE
We next define the equilibrium notion associated with exact-form deviations. The quantitative formulation and the case of zero approximation error require slightly different conventions because uniform regret bounds require common bounds on the potentials.
Definition 10.2 (ConCE for a specified deviation family). Fix for every player a specified family Φ i \Phi_i Φ i μ \mu μ ε \varepsilon ε ( Φ i ) i (\Phi_i)_i ( Φ i ) i
sup ϕ i ∈ Φ i Γ i ( μ , ϕ i ) ≤ ε for every player i . \sup_{\phi_i \in\Phi_i} \Gamma_i(\mu,\phi_i) \le\varepsilon\qquad\text{for every player } i. ϕ i ∈ Φ i sup Γ i ( μ , ϕ i ) ≤ ε for every player i . For approximate, quantitative conclusions, we use a family such as Φ i : = Φ i , e x a c t ( B i , L i ) \Phi_i := \Phi_{i,\mathrm{exact}}(B_i,L_i) Φ i := Φ i , exact ( B i , L i ) Φ i , e x a c t ( B i , L i ) \Phi_{i,\mathrm{exact}}(B_i,L_i) Φ i , exact ( B i , L i ) Φ e x a c t ( B i , L i ) \Phi_{\mathrm{exact}}(B_i,L_i) Φ exact ( B i , L i ) K i K_i K i B i B_i B i L i L_i L i C l o c 1 , 1 C^{1,1}_{\mathrm{loc}} C loc 1 , 1 K i K_i K i
The standard regret-to-equilibrium argument gives the following quantitative connection.
Theorem 10.3. Let μ ˉ T : = T − 1 ∑ t = 1 T δ x t \bar{\mu}_T := T^{-1}\sum_{t=1}^{T}\delta_{x_t} μ ˉ T := T − 1 ∑ t = 1 T δ x t g i , t : = ∇ x i ℓ i ( x i , t , x − i , t ) g_{i,t} := \nabla_{x_i}\ell_i(x_{i,t},x_{-i,t}) g i , t := ∇ x i ℓ i ( x i , t , x − i , t ) i i i
sup ϕ i ∈ Φ i ∑ t = 1 T ⟨ g i , t , x i , t − ϕ i ( x i , t ) ⟩ ≤ R i , T , \sup_{\phi_i \in\Phi_i} \sum_{t=1}^{T}\langle g_{i,t},x_{i,t}-\phi_i(x_{i,t})\rangle\le R_{i,T}, ϕ i ∈ Φ i sup t = 1 ∑ T ⟨ g i , t , x i , t − ϕ i ( x i , t )⟩ ≤ R i , T , then μ ˉ T \bar{\mu}_T μ ˉ T ε T \varepsilon_T ε T ( Φ i ) i (\Phi_i)_i ( Φ i ) i ε T = max i R i , T / T \varepsilon_T = \max_i R_{i,T}/T ε T = max i R i , T / T Φ i , e x a c t ( B i , L i ) \Phi_{i,\mathrm{exact}}(B_i,L_i) Φ i , exact ( B i , L i ) ∥ g i , t ∥ ≤ G i \lVert g_{i,t}\rVert\le G_i ∥ g i , t ∥ ≤ G i ε T ≤ max i G i 6 L i B i / T \varepsilon_T \le\max_i G_i\sqrt{6L_iB_i/T} ε T ≤ max i G i 6 L i B i / T
The proof uses only convexity of the losses and the linearized regret bound. The same argument therefore applies to the regularizer-dependent mirror and FTRL deviation families. We introduce their corresponding equilibrium notions in Section 10.2. The distinction between them matters at the boundary, as shown by Theorem 9.8.
Exact and Proximal Equilibrium Constraints
The class Φ i , p r o x \Phi_{i,\mathrm{prox}} Φ i , prox [11 ] . We impose no fixed ρ \rho ρ Φ i , e x a c t s m \Phi^{\mathrm{sm}}_{i,\mathrm{exact}} Φ i , exact sm
Rad I ( Φ i , p r o x ) = Φ i , e x a c t s m . \operatorname{Rad}_{I}(\Phi_{i,\mathrm{prox}})=\Phi^{\mathrm{sm}}_{i,\mathrm{exact}}. Rad I ( Φ i , prox ) = Φ i , exact sm . Together with the radial invariance in Proposition 10.1 , this immediately gives the equality of the corresponding equilibrium constraints.
Theorem 10.4. Under the unrestricted conventions above, every convex game with compact convex action sets and jointly continuous losses satisfies
ConCE = PCE . \operatorname{ConCE} = \operatorname{PCE}. ConCE = PCE . The proof is given in Appendix § 16 .14. This equality relies on the unrestricted deviation classes and does not directly extend to classes with a fixed weak-convexity parameter, normalization, locality restriction, or computational restriction. For normalized ε \varepsilon ε
Mirror-exact, Mirror-radial, and FTRL Equilibrium Constraints
We now record the equilibrium consequences of the regularizer-dependent deviation classes studied in § 8 and § 9 . We begin with standard constrained mirror descent. Fix for every player i i i m i m_i m i R i R_i R i K i K_i K i § 8.4 . The full geometric class controlled by the exactness characterization is Φ e x a c t R i ( K i ) \Phi_{\mathrm{exact}}^{R_i}(K_i) Φ exact R i ( K i )
Φ i M D ⊆ Φ e x a c t R i ( K i ) \Phi_i^{\mathrm{MD}} \subseteq\Phi_{\mathrm{exact}}^{R_i}(K_i) Φ i MD ⊆ Φ exact R i ( K i ) be such a family. Assume that every ϕ i ∈ Φ i M D \phi_i \in\Phi_i^{\mathrm{MD}} ϕ i ∈ Φ i MD B i M D B_i^{\mathrm{MD}} B i MD L i M D L_i^{\mathrm{MD}} L i MD S i S_i S i ∇ R i \nabla R_i ∇ R i M i M_i M i K i K_i K i ∥ g i , t ∥ ∗ ≤ G i \lVert g_{i,t}\rVert_* \le G_i ∥ g i , t ∥ ∗ ≤ G i
C i M D : = L i M D + S i m i + S i M i m i 2 . C_i^{\mathrm{MD}} := \frac{L_i^{\mathrm{MD}}+S_i}{m_i}+\frac{S_iM_i}{m_i^2}. C i MD := m i L i MD + S i + m i 2 S i M i . If player i i i
η i M D : = B i M D C i M D G i 2 T , \eta_i^{\mathrm{MD}} := \sqrt{\frac{B_i^{\mathrm{MD}}}{C_i^{\mathrm{MD}}G_i^2T}}, η i MD := C i MD G i 2 T B i MD , whenever B i M D C i M D > 0 B_i^{\mathrm{MD}}C_i^{\mathrm{MD}}>0 B i MD C i MD > 0
sup ϕ i ∈ Φ i M D Γ i ( μ ˉ T , ϕ i ) ≤ 2 G i B i M D C i M D T . \sup_{\phi_i\in\Phi_i^{\mathrm{MD}}}\Gamma_i(\bar{\mu}_T,\phi_i)\le2G_i\sqrt{\frac{B_i^{\mathrm{MD}}C_i^{\mathrm{MD}}}{T}}. ϕ i ∈ Φ i MD sup Γ i ( μ ˉ T , ϕ i ) ≤ 2 G i T B i MD C i MD . The degenerate cases follow from the corresponding unoptimized bound. If ∇ R i \nabla R_i ∇ R i o ( 1 ) o(1) o ( 1 ) B i M D B_i^{\mathrm{MD}} B i MD L i M D L_i^{\mathrm{MD}} L i MD S i S_i S i
The identity-interpolation results of § 8.5 give a second equilibrium notion associated with the same regularizer. Let Φ p r o x R i ( K i ) \Phi_{\mathrm{prox}}^{R_i}(K_i) Φ prox R i ( K i ) Φ r a d R i ( K i ) \Phi_{\mathrm{rad}}^{R_i}(K_i) Φ rad R i ( K i ) Definition 8.13 . A distribution is a Bregman R R R Φ p r o x R i ( K i ) \Phi_{\mathrm{prox}}^{R_i}(K_i) Φ prox R i ( K i ) R R R Φ r a d R i ( K i ) \Phi_{\mathrm{rad}}^{R_i}(K_i) Φ rad R i ( K i ) PCE ( R i ) i B r e g \operatorname{PCE}_{(R_i)_i}^{\mathrm{Breg}} PCE ( R i ) i Breg Eq ( R i ) i r a d \operatorname{Eq}_{(R_i)_i}^{\mathrm{rad}} Eq ( R i ) i rad
By radial invariance, these two equilibrium notions coincide. We record this result in the following corollary.
Corollary 10.5. For any set of regularizers R i R_i R i i ∈ [ N ] i\in[N] i ∈ [ N ]
PCE ( R i ) i B r e g = Eq ( R i ) i r a d . \operatorname{PCE}_{(R_i)_i}^{\mathrm{Breg}}=\operatorname{Eq}_{(R_i)_i}^{\mathrm{rad}}. PCE ( R i ) i Breg = Eq ( R i ) i rad . This equality concerns the proximal class and its identity-radial closure; it does not identify the radial class with the full mirror-exact class. To make this distinction explicit, define
Eq ( R i ) i e x a c t : = { μ : Γ i ( μ , ϕ i ) ≤ 0 for every player i and every ϕ i ∈ Φ e x a c t R i ( K i ) } . \operatorname{Eq}^{\mathrm{exact}}_{(R_i)_i} := \left\{ \mu: \Gamma_i(\mu,\phi_i) \le0 \text{ for every player } i \text{ and every } \phi_i \in\Phi^{R_i}_{\mathrm{exact}}(K_i) \right\}. Eq ( R i ) i exact := { μ : Γ i ( μ , ϕ i ) ≤ 0 for every player i and every ϕ i ∈ Φ exact R i ( K i ) } . Proposition 8.14 gives
Eq ( R i ) i e x a c t ⊆ Eq ( R i ) i r a d = P C E ( R i ) i B r e g . \operatorname{Eq}^{\mathrm{exact}}_{(R_i)_i} \subseteq\operatorname{Eq}^{\mathrm{rad}}_{(R_i)_i} = \mathrm{PCE}^{\mathrm{Breg}}_{(R_i)_i}. Eq ( R i ) i exact ⊆ Eq ( R i ) i rad = PCE ( R i ) i Breg . The first inclusion can be strict at the level of deviation maps. In the Legendre regime, Proposition 8.15 shows that a mirror-exact deviation can belong to the radial class only if it passes the relative curvature test in Corollary 8.17. Proposition 8.16 gives the converse when the reconstructed generator is an admissible weakly convex Bregman proximal function, while Proposition 8.18 shows that no positive interpolation may exist even for a smooth dual potential with globally Lipschitz gradient. As in the Euclidean setting, strict inclusion of deviation classes does not by itself imply strict inclusion of the corresponding equilibrium sets.
FTRL produces a third, distinct family of equilibrium constraints. For each player, let R i : R d i → ( − ∞ , + ∞ ] \mathcal{R}_i : \mathbb{R}^{d_i} \to(-\infty,+\infty] R i : R d i → ( − ∞ , + ∞ ] ( σ i , r i ) (\sigma_i,r_i) ( σ i , r i ) K i = dom R i K_i=\operatorname{dom}\mathcal{R}_i K i = dom R i
Φ i F T R L ⊆ Φ F T R L R i ( K i ) \Phi_i^{\mathrm{FTRL}} \subseteq\Phi_{\mathrm{FTRL}}^{\mathcal{R}_i}(K_i) Φ i FTRL ⊆ Φ FTRL R i ( K i ) be a specified family. Player i i i R R R R i \mathcal{R}_i R i R i : = R i + ι K i \mathcal{R}_i := R_i+\iota_{K_i} R i := R i + ι K i q i : = r i / ( r i − 1 ) q_i:=r_i/(r_i-1) q i := r i / ( r i − 1 ) Φ i F T R L \Phi_i^{\mathrm{FTRL}} Φ i FTRL B i F T R L B_i^{\mathrm{FTRL}} B i FTRL L i F T R L L_i^{\mathrm{FTRL}} L i FTRL ∥ g i , t ∥ ∗ ≤ G i \lVert g_{i,t}\rVert_* \le G_i ∥ g i , t ∥ ∗ ≤ G i ε T F T R L \varepsilon_T^{\mathrm{FTRL}} ε T FTRL ( Φ i F T R L ) i (\Phi_i^{\mathrm{FTRL}})_i ( Φ i FTRL ) i
ε T F T R L ≤ max i { r i 1 / r i G i ( L i F T R L ) 1 / q i ( B i F T R L ) 1 / r i σ i 1 / r i T − 1 / r i } . \varepsilon_T^{\mathrm{FTRL}} \le\max_i \left\{ r_i^{1/r_i}\frac{G_i\left(L_i^{\mathrm{FTRL}}\right)^{1/q_i}\left(B_i^{\mathrm{FTRL}}\right)^{1/r_i}}{\sigma_i^{1/r_i}}T^{-1/r_i} \right\}. ε T FTRL ≤ i max { r i 1/ r i σ i 1/ r i G i ( L i FTRL ) 1/ q i ( B i FTRL ) 1/ r i T − 1/ r i } . For r i = 2 r_i=2 r i = 2 O ( T − 1 / 2 ) O(T^{-1/2}) O ( T − 1/2 )
The remaining subsections return to the Euclidean ConCE and PCE notions. Their comparison with unrestricted measurable CE does not depend on a selected mirror regularizer.
Finite Support and Boundary-compatible Interpolation
On a finite support, only finitely many recommended actions and deviation values matter. The following result shows that any such finite collection can be realized, after a sufficiently small common interpolation with the identity, by a single proximal map. Its proof appears in Appendix C.
Theorem 10.6. Let K ⊂ R d K \subset\mathbb{R}^d K ⊂ R d x 1 , … , x m ∈ K x^1,\ldots,x^m \in K x 1 , … , x m ∈ K y 1 , … , y m ∈ K y^1,\ldots,y^m \in K y 1 , … , y m ∈ K ρ ∈ ( 0 , 1 ) \rho\in(0,1) ρ ∈ ( 0 , 1 ) α 0 > 0 \alpha_0>0 α 0 > 0 α ∈ ( 0 , α 0 ) \alpha\in(0,\alpha_0) α ∈ ( 0 , α 0 ) F α = f α + ι K F_\alpha=f_\alpha+\iota_K F α = f α + ι K ρ \rho ρ
prox F α ( x j ) = ( 1 − α ) x j + α y j , j = 1 , … , m . \operatorname{prox}_{F_\alpha}(x^j)=(1-\alpha)x^j+\alpha y^j,\qquad j=1,\ldots,m. prox F α ( x j ) = ( 1 − α ) x j + α y j , j = 1 , … , m . The proximal minimizer is unique.
It follows that the three equilibrium notions cannot be separated on a finite support.
Corollary 10.7. For every finitely supported distribution in a convex game,
μ ∈ C E ⟺ μ ∈ C o n C E ⟺ μ ∈ P C E . \mu\in\mathrm{CE}\quad\Longleftrightarrow\quad\mu\in\mathrm{ConCE}\quad\Longleftrightarrow\quad\mu\in\mathrm{PCE}. μ ∈ CE ⟺ μ ∈ ConCE ⟺ μ ∈ PCE . In particular the three notions coincide in every finite game.
For the finite-game statement, we use the standard mixed extension, with each pure-action set embedded as the vertices of its mixed-strategy simplex and the distribution supported on pure-action profiles. Since the extended losses are affine in a player’s own mixed action, any profitable deviation to a mixed action has a profitable pure action in its support. The proof of Corollary 10.7 is given in Appendix § 16 .14.
Indeed, Monnot and Piliouras [ [39 ] show that C E = C C E \mathrm{CE} = \mathrm{CCE} CE = CCE
Full Correlated Equilibrium on Intervals
On an interval, every smooth scalar displacement has an antiderivative. Under the joint continuity assumption on the losses, smooth feasible deviations are also sufficiently rich to approximate all measurable CE deviations.
Theorem 10.8 (Full CE on one-dimensional action spaces). Suppose that every player has a compact interval action set K i : = [ a i , b i ] ⊆ R K_i := [a_i, b_i] \subseteq\mathbb{R} K i := [ a i , b i ] ⊆ R ℓ i : K → R \ell_i: K \to\mathbb{R} ℓ i : K → R K : = ∏ j = 1 N K j K := \prod_{j=1}^{N} K_j K := ∏ j = 1 N K j
C E = C o n C E = P C E \mathrm{CE} = \mathrm{ConCE} = \mathrm{PCE} CE = ConCE = PCE under the unrestricted definitions used in this section when the approximation error is zero.
The proof of Theorem 10.8 is given in Appendix § 16 .14. In particular, infinite support alone does not separate these equilibrium notions. Under continuous losses, such a separation requires at least one player’s individual action set to have affine dimension at least two.
Two related results help place the statement above. In finite two-action games, Monnot and Piliouras [ [39 ] show that C E = C C E \mathrm{CE} = \mathrm{CCE} CE = CCE ( q , 1 − q ) (q, 1-q) ( q , 1 − q ) [40 ] study continuous-time replicator dynamics with two actions and obtain a localized regret guarantee under a condition on how the resulting one-dimensional trajectory crosses an interval. Their main game-theoretic results likewise focus on two-player 2 × 2 2 \times2 2 × 2
Strict Separation from CE on Continuous Support
Finite interpolation shows that any separation from CE must involve infinite support. The following result establishes such a separation and gives it a simple variational interpretation.
The construction uses the diagonal law ( X , X ) (X,X) ( X , X ) X X X
Theorem 10.9. There is a two-player convex game and a distribution μ \mu μ
μ ∈ C o n C E = P C E but μ ∉ C E . \mu\in\mathrm{ConCE} = \mathrm{PCE} \qquad\text{but} \qquad\mu\notin\mathrm{CE}. μ ∈ ConCE = PCE but μ ∈ / CE . Consequently C E ⊊ C o n C E = P C E \mathrm{CE} \subsetneq\mathrm{ConCE} = \mathrm{PCE} CE ⊊ ConCE = PCE under the definitions above.
The proof of Theorem 10.9 is given in Appendix § 16 .14. It constructs a conditional loss-gradient field that is invisible to exact-form deviations but can be exploited by unrestricted deviations. The construction also illustrates why a simple localization argument cannot establish CE equality on general continuous supports. We defer the question of whether a similar strict separation can be obtained with an absolutely continuous, full-dimensional joint law to future work.
Conclusion
This paper develops a geometric theory of the action-dependent comparators controlled by projected first-order methods. Its central principle is that exactness determines the interior behavior of a deviation, while feasibility controls its interaction with the boundary. In Euclidean geometry, exact feasible displacement fields yield explicit regret bounds for projected gradient descent, whereas nonzero circulation can be exploited to force linear regret. The same exactness-versus-circulation principle extends to mirror geometry, where the regularizer determines the relevant one-form and the additional regularity needed to control boundary residuals.
This viewpoint also clarifies several distinctions among deviation classes, algorithms, and equilibrium notions. Proximal deviations and their identity interpolations form a subclass of exact-form deviations, but the Euclidean radial-closure representation does not extend to general Bregman geometry: convexity of the reconstructed dual potential provides an additional obstruction. Mirror descent and FTRL likewise agree in the Legendre interior but can control different deviation classes once constraints create nontrivial boundary effects. At the equilibrium level, strict inclusion of deviation maps need not produce strict inclusion of zero-tolerance equilibrium sets. In particular, we showed that unrestricted ConCE and PCE coincide, and they coincide with CE on finite supports and on one-dimensional action spaces with continuous losses, while a two-dimensional continuous-support example yields
C E ⊊ C o n C E = P C E . \mathrm{CE} \subsetneq\mathrm{ConCE} = \mathrm{PCE}. CE ⊊ ConCE = PCE . Two questions appear particularly natural. First, can the relative-curvature, Lipschitz, and ambient regularity assumptions in the positive mirror-descent result be weakened while retaining control of the boundary residual and hence sublinear regret? A sharp answer would bring the sufficient conditions closer to the circulation-based obstruction, which requires only continuity of the field. Second, can the strict continuous-support equilibrium separation be realized with a full-dimensional absolutely continuous joint distribution? The present construction has absolutely continuous marginals but is supported on a lower-dimensional diagonal, so resolving this question would determine whether the separation persists for genuinely diffuse outcome distributions.
More broadly, the exact-form viewpoint suggests connections with Blackwell approachability [41 , 42 ] and gradient equilibrium [31 , 43 ] . All three frameworks exploit geometric structure to identify quantities that first-order dynamics can drive toward a target set or make telescope over time. Understanding these connections may lead to alternative characterizations of exact-form regret and to broader classes of deviations controlled by simple first-order methods. A related direction is to combine exactness with lifting [18 , 21 ] . Deviations that are not exact in the original action space may admit an exact representation in a higher-dimensional feature space, potentially allowing gradient-based algorithms to control richer families such as general linear endomorphisms. Developing such exact lifts, and relating them to approachability and gradient equilibrium, could provide a bridge between the geometric simplicity of the present framework and more general Φ \Phi Φ
Acknowledgments
This research was partially supported by the Office of Naval Research (ONR) grants N00014-24-1-2470 and N00014-25-1-2296, National Science Foundation awards CCF-2443068 and IIS-2552046, and a Schmidt Sciences AI2050 Early Career Fellowship. Part of this research was performed while A.S. was visiting the Institute for Mathematical and Statistical Innovation (IMSI), which is supported by the National Science Foundation (Grant No. DMS-2425650).
Preliminaries
Lemma A.1. Let u ∈ R d u \in\mathbb{R}^{d} u ∈ R d and x + : = Π K ( u ) x^{+}:=\Pi_{K}(u) x + := Π K ( u ) Then
u − x + ∈ N K ( x + ) , u-x^{+}\in N_{K}(x^{+}), u − x + ∈ N K ( x + ) , Proof. The point x + x^{+} x + y ↦ 1 2 ∥ y − u ∥ 2 y\mapsto\frac{1}{2}\lVert y-u\rVert^{2} y ↦ 2 1 ∥ y − u ∥ 2 K K K
⟨ x + − u , y − x + ⟩ ≥ 0 ∀ y ∈ K . \langle x^{+}-u,y-x^{+}\rangle\ge0 \qquad\forall y\in K. ⟨ x + − u , y − x + ⟩ ≥ 0 ∀ y ∈ K . Equivalently, ⟨ u − x + , y − x + ⟩ ≤ 0 \langle u-x^{+},y-x^{+}\rangle\le0 ⟨ u − x + , y − x + ⟩ ≤ 0 y ∈ K y\in K y ∈ K
Definition A.2. For ρ ≥ 0 \rho\ge0 ρ ≥ 0 a proper lower-semicontinuous function F : R d → ( − ∞ , + ∞ ] F:\mathbb{R}^{d}\to(-\infty,+\infty] F : R d → ( − ∞ , + ∞ ] is ρ \rho ρ weakly convex if
x ↦ F ( x ) + ρ 2 ∥ x ∥ 2 x\mapsto F(x)+\frac{\rho}{2}\lVert x\rVert^{2} x ↦ F ( x ) + 2 ρ ∥ x ∥ 2 is convex. We use the weakly convex subdifferential convention
∂ F ( x ) : = ∂ c v x ( F + ρ 2 ∥ ⋅ ∥ 2 ) ( x ) − ρ x , \partial F(x):=\partial_{\mathrm{cvx}}\left(F+\frac{\rho}{2}\lVert\cdot\rVert^{2}\right)(x)-\rho x, ∂ F ( x ) := ∂ cvx ( F + 2 ρ ∥ ⋅ ∥ 2 ) ( x ) − ρ x , where ∂ c v x \partial_{\mathrm{cvx}} ∂ cvx is the convex subdifferential. With this convention, ∂ F \partial F ∂ F is ρ \rho ρ hypomonotone: for all u ∈ ∂ F ( p ) u\in\partial F(p) u ∈ ∂ F ( p ) and v ∈ ∂ F ( q ) v\in\partial F(q) v ∈ ∂ F ( q )
⟨ u − v , p − q ⟩ ≥ − ρ ∥ p − q ∥ 2 . (15) \langle u-v,p-q\rangle\ge-\rho\lVert p-q\rVert^{2}. \tag*{(15)} ⟨ u − v , p − q ⟩ ≥ − ρ ∥ p − q ∥ 2 . (15) Lemma A.3. Assume F : = f + ι K F:=f+\iota_{K} F := f + ι K is proper, lower semicontinuous, and ρ \rho ρ weakly convex with 0 ≤ ρ < 1 0\le\rho<1 0 ≤ ρ < 1 Then, for every x ∈ R d x\in\mathbb{R}^{d} x ∈ R d the proximal objective attains its minimum at a unique point. Moreover, M F M_{F} M F is differentiable, and
∇ M F ( x ) = x − prox F ( x ) . (16) \nabla M_{F}(x)=x-\operatorname{prox}_{F}(x). \tag*{(16)} ∇ M F ( x ) = x − prox F ( x ) . (16) Moreover,
∥ prox F ( x ) − prox F ( y ) ∥ ≤ 1 1 − ρ ∥ x − y ∥ , (17) \lVert\operatorname{prox}_{F}(x)-\operatorname{prox}_{F}(y)\rVert\le\frac{1}{1-\rho}\lVert x-y\rVert, \tag*{(17)} ∥ prox F ( x ) − prox F ( y )∥ ≤ 1 − ρ 1 ∥ x − y ∥ , (17) so ∇ M F \nabla M_{F} ∇ M F is Lipschitz with the crude bound
∥ ∇ M F ( x ) − ∇ M F ( y ) ∥ ≤ 2 − ρ 1 − ρ ∥ x − y ∥ . (18) \lVert\nabla M_{F}(x)-\nabla M_{F}(y)\rVert\le\frac{2-\rho}{1-\rho}\lVert x-y\rVert. \tag*{(18)} ∥ ∇ M F ( x ) − ∇ M F ( y )∥ ≤ 1 − ρ 2 − ρ ∥ x − y ∥ . (18) Proof. For fixed x x x y ↦ F ( y ) + 1 2 ∥ y − x ∥ 2 y\mapsto F(y)+\frac{1}{2}\lVert y-x\rVert^{2} y ↦ F ( y ) + 2 1 ∥ y − x ∥ 2 ( 1 − ρ ) (1-\rho) ( 1 − ρ )
p = prox F ( x ) , q = prox F ( y ) . p=\operatorname{prox}_{F}(x),\qquad q=\operatorname{prox}_{F}(y). p = prox F ( x ) , q = prox F ( y ) . The first-order optimality conditions are
x − p ∈ ∂ F ( p ) , y − q ∈ ∂ F ( q ) . x-p\in\partial F(p),\qquad y-q\in\partial F(q). x − p ∈ ∂ F ( p ) , y − q ∈ ∂ F ( q ) . Applying hypomonotonicity (15) gives
⟨ ( x − p ) − ( y − q ) , p − q ⟩ ≥ − ρ ∥ p − q ∥ 2 , \left\langle(x-p)-(y-q),p-q\right\rangle\ge-\rho\lVert p-q\rVert^{2}, ⟨ ( x − p ) − ( y − q ) , p − q ⟩ ≥ − ρ ∥ p − q ∥ 2 , ⟨ x − y , p − q ⟩ − ∥ p − q ∥ 2 ≥ − ρ ∥ p − q ∥ 2 . \langle x-y,p-q\rangle-\lVert p-q\rVert^{2}\ge-\rho\lVert p-q\rVert^{2}. ⟨ x − y , p − q ⟩ − ∥ p − q ∥ 2 ≥ − ρ ∥ p − q ∥ 2 . Thus
( 1 − ρ ) ∥ p − q ∥ 2 ≤ ⟨ x − y , p − q ⟩ ≤ ∥ x − y ∥ ∥ p − q ∥ (1-\rho)\lVert p-q\rVert^2 \le\langle x-y,p-q\rangle\le\lVert x-y\rVert\lVert p-q\rVert ( 1 − ρ ) ∥ p − q ∥ 2 ≤ ⟨ x − y , p − q ⟩ ≤ ∥ x − y ∥ ∥ p − q ∥ which proves (17) .
For fixed y y y x x x F ( y ) + 1 2 ∥ y − x ∥ 2 F(y)+\frac{1}{2}\lVert y-x\rVert^2 F ( y ) + 2 1 ∥ y − x ∥ 2 x − y x-y x − y ∇ M F ( x ) = x − prox F ( x ) \nabla M_F(x)=x-\operatorname{prox}_F(x) ∇ M F ( x ) = x − prox F ( x )
∥ ∇ M F ( x ) − ∇ M F ( y ) ∥ = ∥ ( x − y ) − ( prox F ( x ) − prox F ( y ) ) ∥ ≤ ∥ x − y ∥ + ∥ prox F ( x ) − prox F ( y ) ∥ ≤ ( 1 + 1 1 − ρ ) ∥ x − y ∥ . \begin{aligned}
\lVert\nabla M_F(x)-\nabla M_F(y)\rVert
&=\lVert(x-y)-(\operatorname{prox}_F(x)-\operatorname{prox}_F(y))\rVert\\
&\le\lVert x-y\rVert+\lVert\operatorname{prox}_F(x)-\operatorname{prox}_F(y)\rVert\\
&\le\left(1+\frac{1}{1-\rho}\right)\lVert x-y\rVert.
\end{aligned} ∥ ∇ M F ( x ) − ∇ M F ( y )∥ = ∥( x − y ) − ( prox F ( x ) − prox F ( y ))∥ ≤ ∥ x − y ∥ + ∥ prox F ( x ) − prox F ( y )∥ ≤ ( 1 + 1 − ρ 1 ) ∥ x − y ∥ . This is (18) .
In turn, we recall that weakly-convex proximal maps are monotone.
Lemma A.4. Let F : R d → ( − ∞ , + ∞ ] F:\mathbb{R}^d\to(-\infty,+\infty] F : R d → ( − ∞ , + ∞ ] ρ \rho ρ 0 ≤ ρ < 1 0\le\rho<1 0 ≤ ρ < 1 prox F \operatorname{prox}_F prox F x , y ∈ R d x,y\in\mathbb{R}^d x , y ∈ R d
⟨ x − y , prox F ( x ) − prox F ( y ) ⟩ ≥ 0. \langle x-y,\operatorname{prox}_F(x)-\operatorname{prox}_F(y)\rangle\ge0. ⟨ x − y , prox F ( x ) − prox F ( y )⟩ ≥ 0. Proof. Fix x , y ∈ R d x,y\in\mathbb{R}^d x , y ∈ R d
x − prox F ( x ) ∈ ∂ F ( prox F ( x ) ) , y − prox F ( y ) ∈ ∂ F ( prox F ( y ) ) . x-\operatorname{prox}_F(x)\in\partial F(\operatorname{prox}_F(x)),\qquad
y-\operatorname{prox}_F(y)\in\partial F(\operatorname{prox}_F(y)). x − prox F ( x ) ∈ ∂ F ( prox F ( x )) , y − prox F ( y ) ∈ ∂ F ( prox F ( y )) . Since F F F ρ \rho ρ ρ \rho ρ
⟨ ( x − prox F ( x ) ) − ( y − prox F ( y ) ) , prox F ( x ) − prox F ( y ) ⟩ ≥ − ρ ∥ prox F ( x ) − prox F ( y ) ∥ 2 . \left\langle(x-\operatorname{prox}_F(x))-(y-\operatorname{prox}_F(y)),\operatorname{prox}_F(x)-\operatorname{prox}_F(y)\right\rangle
\ge-\rho\lVert\operatorname{prox}_F(x)-\operatorname{prox}_F(y)\rVert^2. ⟨ ( x − prox F ( x )) − ( y − prox F ( y )) , prox F ( x ) − prox F ( y ) ⟩ ≥ − ρ ∥ prox F ( x ) − prox F ( y ) ∥ 2 . Rearranging gives
⟨ x − y , prox F ( x ) − prox F ( y ) ⟩ ≥ ( 1 − ρ ) ∥ prox F ( x ) − prox F ( y ) ∥ 2 ≥ 0. \langle x-y,\operatorname{prox}_F(x)-\operatorname{prox}_F(y)\rangle
\ge(1-\rho)\lVert\operatorname{prox}_F(x)-\operatorname{prox}_F(y)\rVert^2\ge0. ⟨ x − y , prox F ( x ) − prox F ( y )⟩ ≥ ( 1 − ρ ) ∥ prox F ( x ) − prox F ( y ) ∥ 2 ≥ 0. Thus prox F \operatorname{prox}_F prox F
Scaled Exact Deviations and Proximal Representation
Lemma B.1. Let K K K Ψ \Psi Ψ K K K Ψ \Psi Ψ K K K Ψ ~ \widetilde{\Psi} Ψ
Proof. Choose open sets K ⊂ V ⋐ W K\subset V\Subset W K ⊂ V ⋐ W W ‾ \overline{W} W Ψ \Psi Ψ χ \chi χ V V V W W W Ψ ~ : = χ Ψ \widetilde{\Psi}:=\chi\Psi Ψ := χ Ψ W ‾ \overline{W} W
Theorem 6.2. Let K ⊆ R d K\subseteq\mathbb{R}^d K ⊆ R d ϕ ( x ) : = x − ∇ Ψ ( x ) \phi(x):=x-\nabla\Psi(x) ϕ ( x ) := x − ∇Ψ ( x ) Ψ ∈ C l o c 1 , 1 \Psi\in C_{\mathrm{loc}}^{1,1} Ψ ∈ C loc 1 , 1 K K K α 0 > 0 \alpha_0>0 α 0 > 0 α ∈ ( 0 , α 0 ] \alpha\in(0,\alpha_0] α ∈ ( 0 , α 0 ]
ϕ α ( x ) : = ( 1 − α ) x + α ϕ ( x ) = x − α ∇ Ψ ( x ) \phi_\alpha(x):=(1-\alpha)x+\alpha\phi(x)=x-\alpha\nabla\Psi(x) ϕ α ( x ) := ( 1 − α ) x + α ϕ ( x ) = x − α ∇Ψ ( x ) coincides on K K K prox F α ( x ) \operatorname{prox}_{F_\alpha}(x) prox F α ( x ) F α = f α + ι K F_\alpha=f_\alpha+\iota_K F α = f α + ι K ρ α \rho_\alpha ρ α ρ α < 1 \rho_\alpha<1 ρ α < 1
Proof. By Lemma B.1, replace Ψ \Psi Ψ K K K ∇ Ψ \nabla\Psi ∇Ψ L L L L = 0 L=0 L = 0 α 0 : = 1 / 2 \alpha_0 := 1/2 α 0 := 1/2 α 0 : = 1 2 min { 1 , 1 / L } \alpha_0 := \frac{1}{2}\min\{1,1/L\} α 0 := 2 1 min { 1 , 1/ L } α ∈ ( 0 , α 0 ] \alpha\in(0,\alpha_0] α ∈ ( 0 , α 0 ]
h α ( x ) : = 1 2 ∥ x ∥ 2 − α Ψ ( x ) , f α ( y ) : = h α ∗ ( y ) − 1 2 ∥ y ∥ 2 . h_\alpha(x) := \frac{1}{2}\lVert x\rVert^2-\alpha\Psi(x), \qquad f_\alpha(y) := h_\alpha^*(y)-\frac{1}{2}\lVert y\rVert^2. h α ( x ) := 2 1 ∥ x ∥ 2 − α Ψ ( x ) , f α ( y ) := h α ∗ ( y ) − 2 1 ∥ y ∥ 2 . The function h α h_\alpha h α ( 1 − α L ) (1-\alpha L) ( 1 − α L ) ( 1 + α L ) (1+\alpha L) ( 1 + α L ) h α ∗ h_\alpha^* h α ∗ 1 / ( 1 + α L ) 1/(1+\alpha L) 1/ ( 1 + α L ) f α f_\alpha f α ρ α \rho_\alpha ρ α ρ α : = α L / ( 1 + α L ) < 1 \rho_\alpha:= \alpha L/(1+\alpha L)<1 ρ α := α L / ( 1 + α L ) < 1 F α : = f α + ι K F_\alpha:= f_\alpha+\iota_K F α := f α + ι K
For fixed x x x ∥ x ∥ 2 / 2 \lVert x\rVert^2/2 ∥ x ∥ 2 /2
h α ∗ ( y ) − ⟨ x , y ⟩ . h_\alpha^*(y)-\langle x,y\rangle. h α ∗ ( y ) − ⟨ x , y ⟩ . Its unique minimizer is y = ∇ h α ( x ) = x − α ∇ Ψ ( x ) = ϕ α ( x ) y=\nabla h_\alpha(x)=x-\alpha\nabla\Psi(x)=\phi_\alpha(x) y = ∇ h α ( x ) = x − α ∇Ψ ( x ) = ϕ α ( x ) x ∈ K x\in K x ∈ K K K K ϕ \phi ϕ ϕ α ( x ) : = ( 1 − α ) x + α ϕ ( x ) ∈ K \phi_\alpha(x):=(1-\alpha)x+\alpha\phi(x)\in K ϕ α ( x ) := ( 1 − α ) x + α ϕ ( x ) ∈ K prox F α = ϕ α \operatorname{prox}_{F_\alpha}=\phi_\alpha prox F α = ϕ α K K K
Corollary 6.3. For the radial closure of Euclidean proximal deviations, we have Rad I ( Φ p r o x ) = Φ e x a c t s m \operatorname{Rad}_{I}(\Phi_{\mathrm{prox}})=\Phi_{\mathrm{exact}}^{\mathrm{sm}} Rad I ( Φ prox ) = Φ exact sm
Proof. Let ϕ ∈ Φ e x a c t s m \phi\in\Phi_{\mathrm{exact}}^{\mathrm{sm}} ϕ ∈ Φ exact sm α > 0 \alpha>0 α > 0 ϕ α = ( 1 − α ) I + α ϕ \phi_\alpha=(1-\alpha)I+\alpha\phi ϕ α = ( 1 − α ) I + α ϕ ϕ ∈ Rad I ( Φ p r o x ) \phi\in\operatorname{Rad}_{I}(\Phi_{\mathrm{prox}}) ϕ ∈ Rad I ( Φ prox )
Conversely, let ϕ ∈ Rad I ( Φ p r o x ) \phi\in\operatorname{Rad}_{I}(\Phi_{\mathrm{prox}}) ϕ ∈ Rad I ( Φ prox ) α ∈ ( 0 , 1 ] \alpha\in(0,1] α ∈ ( 0 , 1 ] F F F ϕ α = prox F \phi_\alpha=\operatorname{prox}_F ϕ α = prox F K K K Theorem 5.1 ,
x − ϕ α ( x ) = ∇ M F ( x ) , x-\phi_\alpha(x)=\nabla M_F(x), x − ϕ α ( x ) = ∇ M F ( x ) , and ∇ M F \nabla M_F ∇ M F x − ϕ α ( x ) = α ( x − ϕ ( x ) ) x-\phi_\alpha(x)=\alpha(x-\phi(x)) x − ϕ α ( x ) = α ( x − ϕ ( x ))
x − ϕ ( x ) = ∇ ( 1 α M F ) ( x ) . x-\phi(x)=\nabla\left(\frac{1}{\alpha}M_F\right)(x). x − ϕ ( x ) = ∇ ( α 1 M F ) ( x ) . Thus ϕ \phi ϕ
Boundary-Compatible Finite Interpolation
Lemma C.1. For finite data ( z j , s j ) j = 1 m (z^j,s^j)_{j=1}^m ( z j , s j ) j = 1 m H H H s j ∈ ∂ H ( z j ) s^j\in\partial H(z^j) s j ∈ ∂ H ( z j ) j j j
∑ r = 1 q ⟨ s j r , z j r + 1 − z j r ⟩ ≤ 0 for every cycle j 1 , … , j q , j q + 1 = j 1 . \sum_{r=1}^{q}\langle s^{j_r},z^{j_{r+1}}-z^{j_r}\rangle\le0 \qquad\text{for every cycle }j_1,\ldots,j_q,j_{q+1}=j_1. r = 1 ∑ q ⟨ s j r , z j r + 1 − z j r ⟩ ≤ 0 for every cycle j 1 , … , j q , j q + 1 = j 1 . It is enough to check nontrivial simple cycles.
Proof. Necessity follows by summing the subgradient inequalities around a cycle. For sufficiency, form the complete directed graph on [ m ] [m] [ m ] w ( j , k ) = ⟨ s j , z k − z j ⟩ w(j,k)=\langle s^j,z^k-z^j\rangle w ( j , k ) = ⟨ s j , z k − z j ⟩ 0 0 0 j j j
a k = max { ∑ r = 0 q − 1 w ( j r , j r + 1 ) : j q = k , j 0 ∈ [ m ] , and the path is simple } . a_k=\max\left\{\sum_{r=0}^{q-1}w(j_r,j_{r+1}):j_q=k,\ j_0\in[m],\ \text{and the path is simple}\right\}. a k = max { r = 0 ∑ q − 1 w ( j r , j r + 1 ) : j q = k , j 0 ∈ [ m ] , and the path is simple } . The maximum is finite because there are finitely many simple paths. Any walk can be reduced to a simple path by deleting closed subwalks; cyclic monotonicity says each deleted closed subwalk has nonpositive weight. Hence a k a_k a k k k k ( j , k ) (j,k) ( j , k ) j j j
a k ≥ a j + w ( j , k ) = a j + ⟨ s j , z k − z j ⟩ ( j , k ∈ [ m ] ) . a_k\ge a_j+w(j,k)=a_j+\langle s^j,z^k-z^j\rangle\qquad(j,k\in[m]). a k ≥ a j + w ( j , k ) = a j + ⟨ s j , z k − z j ⟩ ( j , k ∈ [ m ]) . Define the finite polyhedral function
H ( z ) : = max j ∈ [ m ] { a j + ⟨ s j , z − z j ⟩ } . H(z) := \max_{j \in[m]} \left\{a_j + \langle s^j, z-z^j\rangle\right\}. H ( z ) := j ∈ [ m ] max { a j + ⟨ s j , z − z j ⟩ } . It is proper, closed, and convex. At z k z^k z k k k k a k a_k a k j j j H ( z k ) = a k H(z^k)=a_k H ( z k ) = a k s k ∈ ∂ H ( z k ) s^k \in\partial H(z^k) s k ∈ ∂ H ( z k )
Theorem 10.6. Let K ⊆ R d K \subseteq\mathbb{R}^d K ⊆ R d x 1 , … , x m ∈ K x^1,\ldots,x^m \in K x 1 , … , x m ∈ K y 1 , … , y m ∈ K y^1,\ldots,y^m \in K y 1 , … , y m ∈ K ρ ∈ ( 0 , 1 ) \rho\in(0,1) ρ ∈ ( 0 , 1 ) α 0 > 0 \alpha_0>0 α 0 > 0 α ∈ ( 0 , α 0 ] \alpha\in(0,\alpha_0] α ∈ ( 0 , α 0 ] F α = f α + ι K F_\alpha=f_\alpha+\iota_K F α = f α + ι K ρ \rho ρ
prox F α ( x j ) = ( 1 − α ) x j + α y j , j = 1 , … , m . \operatorname{prox}_{F_\alpha}(x^j)=(1-\alpha)x^j+\alpha y^j,\qquad j=1,\ldots,m. prox F α ( x j ) = ( 1 − α ) x j + α y j , j = 1 , … , m . The proximal minimizer is unique.
Proof. Write d j : = x j − y j d^j:=x^j-y^j d j := x j − y j
z α j : = x j − α d j , s α j : = x j − ( 1 − ρ ) z α j . z_\alpha^j:=x^j-\alpha d^j,\qquad s_\alpha^j:=x^j-(1-\rho)z_\alpha^j. z α j := x j − α d j , s α j := x j − ( 1 − ρ ) z α j . Convexity of K K K z α j ∈ K z_\alpha^j \in K z α j ∈ K α ∈ [ 0 , 1 ] \alpha\in[0,1] α ∈ [ 0 , 1 ] α = 0 \alpha=0 α = 0 z 0 j = x j z_0^j=x^j z 0 j = x j s 0 j = ρ x j s_0^j=\rho x^j s 0 j = ρ x j C C C
∑ ( j , k ) ∈ C ⟨ s 0 j , z 0 k − z 0 j ⟩ = − ρ 2 ∑ ( j , k ) ∈ C ∥ x k − x j ∥ 2 < 0. \sum_{(j,k)\in C}\langle s_0^j,z_0^k-z_0^j\rangle
=-\frac{\rho}{2}\sum_{(j,k)\in C}\lVert x^k-x^j\rVert^2<0. ( j , k ) ∈ C ∑ ⟨ s 0 j , z 0 k − z 0 j ⟩ = − 2 ρ ( j , k ) ∈ C ∑ ∥ x k − x j ∥ 2 < 0. There are finitely many simple cycles and their cycle sums vary continuously with α \alpha α α > 0 \alpha>0 α > 0 ( z α j , s α j ) (z_\alpha^j,s_\alpha^j) ( z α j , s α j ) Lemma C.1 , choose a proper closed convex H α H_\alpha H α s α j ∈ ∂ H α ( z α j ) s_\alpha^j \in\partial H_\alpha(z_\alpha^j) s α j ∈ ∂ H α ( z α j )
f α ( z ) : = H α ( z ) − ρ 2 ∥ z ∥ 2 , F α : = f α + ι K . f_\alpha(z):=H_\alpha(z)-\frac{\rho}{2}\lVert z\rVert^2,\qquad F_\alpha:=f_\alpha+\iota_K. f α ( z ) := H α ( z ) − 2 ρ ∥ z ∥ 2 , F α := f α + ι K . Then F α F_\alpha F α ρ \rho ρ N K ( z α j ) N_K(z_\alpha^j) N K ( z α j )
0 = s α j − ρ z α j + z α j − x j ∈ ∂ ( F α ( z ) + 1 2 ∥ z − x j ∥ 2 ) z = z α j . 0=s_\alpha^j-\rho z_\alpha^j+z_\alpha^j-x^j\in\partial\left(F_\alpha(z)+\frac{1}{2}\lVert z-x^j\rVert^2\right)_{z=z_\alpha^j}. 0 = s α j − ρ z α j + z α j − x j ∈ ∂ ( F α ( z ) + 2 1 ∥ z − x j ∥ 2 ) z = z α j . The proximal objective is ( 1 − ρ ) (1-\rho) ( 1 − ρ ) z α j = ( 1 − α ) x j + α y j z_\alpha^j=(1-\alpha)x^j+\alpha y^j z α j = ( 1 − α ) x j + α y j
Other Proofs and Details
Proofs for Section 2
First, we show that the feasibility of exact-form deviations controls the normal vectors that are introduced due to the projection step in projected online gradient descent (OGD).
Lemma D.1. Let ϕ : K → K \phi:K\to K ϕ : K → K D ϕ ( x ) : = x − ϕ ( x ) D_\phi(x):=x-\phi(x) D ϕ ( x ) := x − ϕ ( x ) n ∈ N K ( x ) n\in N_K(x) n ∈ N K ( x )
⟨ n , D ϕ ( x ) ⟩ ≥ 0. \langle n,D_\phi(x)\rangle\geq0. ⟨ n , D ϕ ( x )⟩ ≥ 0. Proof. Since ϕ ( x ) ∈ K \phi(x)\in K ϕ ( x ) ∈ K n ∈ N K ( x ) n\in N_K(x) n ∈ N K ( x )
⟨ n , ϕ ( x ) − x ⟩ ≤ 0. \langle n,\phi(x)-x\rangle\leq0. ⟨ n , ϕ ( x ) − x ⟩ ≤ 0. Multiplying by − 1 -1 − 1
Theorem 4.1. Let { x t } t = 1 T + 1 \{x_t\}_{t=1}^{T+1} { x t } t = 1 T + 1 ϕ : K → K \phi: K \to K ϕ : K → K D ϕ ( x ) = x − ϕ ( x ) = ∇ Ψ ( x ) D_{\phi}(x) = x - \phi(x) = \nabla\Psi(x) D ϕ ( x ) = x − ϕ ( x ) = ∇Ψ ( x ) Ψ \Psi Ψ L L L g 1 , … , g T g_1,\ldots,g_T g 1 , … , g T
∑ t = 1 T ⟨ g t , x t − ϕ ( x t ) ⟩ ≤ Ψ ( x 1 ) − Ψ ( x T + 1 ) η + 3 L η 2 ∑ t = 1 T ∥ g t ∥ 2 . (19) \sum_{t=1}^{T} \langle g_t, x_t-\phi(x_t)\rangle\le\frac{\Psi(x_1)-\Psi(x_{T+1})}{\eta}+\frac{3L\eta}{2}\sum_{t=1}^{T}\lVert g_t\rVert^2 . \tag*{(19)} t = 1 ∑ T ⟨ g t , x t − ϕ ( x t )⟩ ≤ η Ψ ( x 1 ) − Ψ ( x T + 1 ) + 2 3 L η t = 1 ∑ T ∥ g t ∥ 2 . (19) Proof. Fix a round t t t
s t : = x t − x t + 1 , n t : = x t − η g t − x t + 1 . s_t := x_t-x_{t+1}, \qquad n_t := x_t-\eta g_t-x_{t+1}. s t := x t − x t + 1 , n t := x t − η g t − x t + 1 . By Lemma A.1 , n t ∈ N K ( x t + 1 ) n_t \in N_K(x_{t+1}) n t ∈ N K ( x t + 1 )
η g t = s t − n t . \eta g_t=s_t-n_t. η g t = s t − n t . Thus,
η ⟨ g t , D ϕ ( x t ) ⟩ = ⟨ s t , D ϕ ( x t ) ⟩ − ⟨ n t , D ϕ ( x t ) ⟩ . (20) \eta\langle g_t,D_{\phi}(x_t)\rangle= \langle s_t,D_{\phi}(x_t)\rangle-\langle n_t,D_{\phi}(x_t)\rangle. \tag*{(20)} η ⟨ g t , D ϕ ( x t )⟩ = ⟨ s t , D ϕ ( x t )⟩ − ⟨ n t , D ϕ ( x t )⟩ . (20) We bound the two terms separately. First, L L L Ψ \Psi Ψ
Ψ ( x t + 1 ) ≤ Ψ ( x t ) + ⟨ ∇ Ψ ( x t ) , x t + 1 − x t ⟩ + L 2 ∥ x t + 1 − x t ∥ 2 = Ψ ( x t ) − ⟨ D ϕ ( x t ) , s t ⟩ + L 2 ∥ s t ∥ 2 . \begin{aligned}
\Psi(x_{t+1}) \le\Psi(x_t)+\langle\nabla\Psi(x_t),x_{t+1}-x_t\rangle+\frac{L}{2}\lVert x_{t+1}-x_t\rVert^2 \\
&=\Psi(x_t)-\langle D_{\phi}(x_t),s_t\rangle+\frac{L}{2}\lVert s_t\rVert^2 .
\end{aligned} Ψ ( x t + 1 ) ≤ Ψ ( x t ) + ⟨ ∇Ψ ( x t ) , x t + 1 − x t ⟩ + 2 L ∥ x t + 1 − x t ∥ 2 = Ψ ( x t ) − ⟨ D ϕ ( x t ) , s t ⟩ + 2 L ∥ s t ∥ 2 . Therefore
⟨ s t , D ϕ ( x t ) ⟩ ≤ Ψ ( x t ) − Ψ ( x t + 1 ) + L 2 ∥ s t ∥ 2 . (21) \langle s_t,D_{\phi}(x_t)\rangle\le\Psi(x_t)-\Psi(x_{t+1})+\frac{L}{2}\lVert s_t\rVert^2 . \tag*{(21)} ⟨ s t , D ϕ ( x t )⟩ ≤ Ψ ( x t ) − Ψ ( x t + 1 ) + 2 L ∥ s t ∥ 2 . (21) Second, by Lemma D.1 applied at x t + 1 x_{t+1} x t + 1
⟨ n t , D ϕ ( x t + 1 ) ⟩ ≥ 0. \langle n_t,D_{\phi}(x_{t+1})\rangle\ge0. ⟨ n t , D ϕ ( x t + 1 )⟩ ≥ 0. Thus
− ⟨ n t , D ϕ ( x t ) ⟩ = − ⟨ n t , D ϕ ( x t + 1 ) ⟩ − ⟨ n t , D ϕ ( x t ) − D ϕ ( x t + 1 ) ⟩ ≤ ∥ n t ∥ ∥ D ϕ ( x t ) − D ϕ ( x t + 1 ) ∥ ≤ L ∥ n t ∥ ∥ s t ∥ . \begin{aligned}
-\langle n_t,D_{\phi}(x_t)\rangle=-\langle n_t,D_{\phi}(x_{t+1})\rangle-\langle n_t,D_{\phi}(x_t)-D_{\phi}(x_{t+1})\rangle\\
&\le\lVert n_t\rVert\lVert D_{\phi}(x_t)-D_{\phi}(x_{t+1})\rVert\\
&\le L\lVert n_t\rVert\lVert s_t\rVert.
\end{aligned} − ⟨ n t , D ϕ ( x t )⟩ = − ⟨ n t , D ϕ ( x t + 1 )⟩ − ⟨ n t , D ϕ ( x t ) − D ϕ ( x t + 1 )⟩ ≤ ∥ n t ∥ ∥ D ϕ ( x t ) − D ϕ ( x t + 1 )∥ ≤ L ∥ n t ∥ ∥ s t ∥ . Projection is nonexpansive and x t = Π K ( x t ) x_t=\Pi_K(x_t) x t = Π K ( x t )
∥ s t ∥ = ∥ Π K ( x t ) − Π K ( x t − η g t ) ∥ ≤ η ∥ g t ∥ . (22) \lVert s_t\rVert=\lVert\Pi_K(x_t)-\Pi_K(x_t-\eta g_t)\rVert\le\eta\lVert g_t\rVert. \tag*{(22)} ∥ s t ∥ = ∥ Π K ( x t ) − Π K ( x t − η g t )∥ ≤ η ∥ g t ∥ . (22) Moreover,
∥ n t ∥ = dist ( x t − η g t , K ) ≤ ∥ x t − η g t − x t ∥ = η ∥ g t ∥ . \lVert n_t\rVert=\operatorname{dist}(x_t-\eta g_t,K)\le\lVert x_t-\eta g_t-x_t\rVert=\eta\lVert g_t\rVert. ∥ n t ∥ = dist ( x t − η g t , K ) ≤ ∥ x t − η g t − x t ∥ = η ∥ g t ∥ . Therefore
− ⟨ n t , D ϕ ( x t ) ⟩ ≤ L η 2 ∥ g t ∥ 2 . (23) -\langle n_t,D_{\phi}(x_t)\rangle\le L\eta^2\lVert g_t\rVert^2 . \tag*{(23)} − ⟨ n t , D ϕ ( x t )⟩ ≤ L η 2 ∥ g t ∥ 2 . (23) Combining (20) , (21) , and (23) , and using ∥ s t ∥ ≤ η ∥ g t ∥ \lVert s_t\rVert\le\eta\lVert g_t\rVert ∥ s t ∥ ≤ η ∥ g t ∥
η ⟨ g t , D ϕ ( x t ) ⟩ ≤ Ψ ( x t ) − Ψ ( x t + 1 ) + 3 L η 2 2 ∥ g t ∥ 2 . \eta\langle g_t,D_{\phi}(x_t)\rangle\le\Psi(x_t)-\Psi(x_{t+1})+\frac{3L\eta^2}{2}\lVert g_t\rVert^2 . η ⟨ g t , D ϕ ( x t )⟩ ≤ Ψ ( x t ) − Ψ ( x t + 1 ) + 2 3 L η 2 ∥ g t ∥ 2 . Dividing by η \eta η t ∈ [ T ] t\in[T] t ∈ [ T ]
Corollary 4.3. Suppose ∥ g t ∥ ≤ G \lVert g_t\rVert\le G ∥ g t ∥ ≤ G B , L > 0 B,L>0 B , L > 0 η = 2 B / ( 3 L G 2 T ) \eta=\sqrt{2B/(3LG^2T)} η = 2 B / ( 3 L G 2 T )
sup ϕ ∈ Φ e x a c t ( B , L ) Reg T ( ϕ ) ≤ G 6 L B T . \sup_{\phi\in\Phi_{\mathrm{exact}}(B,L)} \operatorname{Reg}_T(\phi)\le G\sqrt{6LBT}. ϕ ∈ Φ exact ( B , L ) sup Reg T ( ϕ ) ≤ G 6 L B T . Thus the algorithm controls the whole normalized class Φ e x a c t ( B , L ) \Phi_{\mathrm{exact}}(B,L) Φ exact ( B , L )
Proof. For every ϕ ∈ Φ e x a c t ( B , L ) \phi\in\Phi_{\mathrm{exact}}(B,L) ϕ ∈ Φ exact ( B , L ) B B B ϕ \phi ϕ ∑ t = 1 T ∥ g t ∥ 2 ≤ G 2 T \sum_{t=1}^{T}\lVert g_t\rVert^2\le G^2T ∑ t = 1 T ∥ g t ∥ 2 ≤ G 2 T
Reg T ( ϕ ) ≤ B η + 3 L η G 2 T 2 \operatorname{Reg}_T(\phi)\le\frac{B}{\eta}+\frac{3L\eta G^2T}{2} Reg T ( ϕ ) ≤ η B + 2 3 L η G 2 T for every ϕ ∈ Φ e x a c t ( B , L ) \phi\in\Phi_{\mathrm{exact}}(B,L) ϕ ∈ Φ exact ( B , L )
Taking the supremum and substituting the displayed value of η \eta η η \eta η
Proofs for Section 5
Theorem 5.1. Let F : = f + ι K F:=f+\iota_K F := f + ι K ρ \rho ρ 0 ≤ ρ < 1 0\le\rho<1 0 ≤ ρ < 1 ϕ F ( x ) : = prox F ( x ) \phi_F(x):=\operatorname{prox}_F(x) ϕ F ( x ) := prox F ( x ) ϕ F : K → K \phi_F:K\to K ϕ F : K → K M F M_F M F x − ϕ F ( x ) = ∇ M F ( x ) x-\phi_F(x)=\nabla M_F(x) x − ϕ F ( x ) = ∇ M F ( x ) L F = ( 2 − ρ ) / ( 1 − ρ ) L_F=(2-\rho)/(1-\rho) L F = ( 2 − ρ ) / ( 1 − ρ ) K K K ∥ g t ∥ ≤ G \lVert g_t\rVert\le G ∥ g t ∥ ≤ G osc K ( M F ) ≤ B F \operatorname{osc}_K(M_F)\le B_F osc K ( M F ) ≤ B F B F , G > 0 B_F,G>0 B F , G > 0 η = 2 B F / ( 3 L F G 2 T ) \eta=\sqrt{2B_F/(3L_FG^2T)} η = 2 B F / ( 3 L F G 2 T )
∑ t = 1 T ⟨ g t , x t − prox F ( x t ) ⟩ ≤ B F η + 3 L F η G 2 T 2 = O ( T ) . \sum_{t=1}^{T}\langle g_t,x_t-\operatorname{prox}_F(x_t)\rangle\le\frac{B_F}{\eta}+\frac{3L_F\eta G^2T}{2}=O(\sqrt{T}). t = 1 ∑ T ⟨ g t , x t − prox F ( x t )⟩ ≤ η B F + 2 3 L F η G 2 T = O ( T ) . If B F = 0 B_F=0 B F = 0 G = 0 G=0 G = 0
Proof. Feasibility holds because F = f + ι K F=f+\iota_K F = f + ι K prox F ( x ) ∈ K \operatorname{prox}_F(x)\in K prox F ( x ) ∈ K Ψ = M F \Psi=M_F Ψ = M F
Proposition 5.2. The exact-form class contains the following deviations.
External regret. For u ∈ K u\in K u ∈ K F : = ι { u } F:=\iota_{\{u\}} F := ι { u } prox F ( x ) = u \operatorname{prox}_F(x)=u prox F ( x ) = u x − u = ∇ 1 2 ∥ x − u ∥ 2 x-u=\nabla\frac{1}{2}\lVert x-u\rVert^2 x − u = ∇ 2 1 ∥ x − u ∥ 2
Projection to a convex subset. For a nonempty closed convex set S ⊆ K S\subseteq K S ⊆ K F : = ι S F:=\iota_S F := ι S prox F ( x ) = Π S ( x ) \operatorname{prox}_F(x)=\Pi_S(x) prox F ( x ) = Π S ( x )
Projection-based/no-move regret. For a vector v v v F ( y ) : = ⟨ v , y ⟩ + ι K ( y ) F(y):=\langle v,y\rangle+\iota_K(y) F ( y ) := ⟨ v , y ⟩ + ι K ( y ) prox F ( x ) = Π K ( x − v ) \operatorname{prox}_F(x)=\Pi_K(x-v) prox F ( x ) = Π K ( x − v )
Interpolation deviations. For u ∈ K u\in K u ∈ K α ∈ ( 0 , 1 ) \alpha\in(0,1) α ∈ ( 0 , 1 ) ϕ ( x ) : = ( 1 − α ) x + α u \phi(x):=(1-\alpha)x+\alpha u ϕ ( x ) := ( 1 − α ) x + α u
F ( y ) : = α 2 ( 1 − α ) ∥ y − u ∥ 2 + ι K ( y ) . F(y):=\frac{\alpha}{2(1-\alpha)}\lVert y-u\rVert^2+\iota_K(y). F ( y ) := 2 ( 1 − α ) α ∥ y − u ∥ 2 + ι K ( y ) . The case α = 1 \alpha=1 α = 1
Local proximal deviations. If ∥ x − prox F ( x ) ∥ ≤ δ \lVert x-\operatorname{prox}_F(x)\rVert\le\delta ∥ x − prox F ( x )∥ ≤ δ K K K ∥ ∇ M F ( x ) ∥ ≤ δ \lVert\nabla M_F(x)\rVert\le\delta ∥ ∇ M F ( x )∥ ≤ δ
Symmetric affine swaps. Let ϕ ( x ) : = A x + b \phi(x):=Ax+b ϕ ( x ) := A x + b K K K K K K T : = span ( K − K ) T:=\operatorname{span}(K-K) T := span ( K − K ) aff ( K ) \operatorname{aff}(K) aff ( K ) a ∈ K a\in K a ∈ K A T ⊆ T AT\subseteq T A T ⊆ T I − A I-A I − A T T T ϕ \phi ϕ aff ( K ) \operatorname{aff}(K) aff ( K )
Ψ ( a + h ) : = ⟨ D ϕ ( a ) , h ⟩ + 1 2 ⟨ h , ( I − A ) h ⟩ , h ∈ T . (24) \Psi(a+h):=\langle D\phi(a),h\rangle+\frac{1}{2}\langle h,(I-A)h\rangle,\qquad h\in T. \tag*{(24)} Ψ ( a + h ) := ⟨ D ϕ ( a ) , h ⟩ + 2 1 ⟨ h , ( I − A ) h ⟩ , h ∈ T . (24) An ambient extension of this potential is given in the proof.
Proof. Items 1 and 2 are immediate from Theorem 5.1 . For item 3, complete the square:
arg min y ∈ K { ⟨ v , y ⟩ + 1 2 ∥ y − x ∥ 2 } = arg min y ∈ K 1 2 ∥ y − ( x − v ) ∥ 2 = Π K ( x − v ) . \underset{y \in K}{\operatorname{arg\,min}}\left\{\langle v,y\rangle+\frac{1}{2}\lVert y-x\rVert^2\right\}
=
\underset{y \in K}{\operatorname{arg\,min}}\frac{1}{2}\lVert y-(x-v)\rVert^2
=
\Pi_K(x-v). y ∈ K arg min { ⟨ v , y ⟩ + 2 1 ∥ y − x ∥ 2 } = y ∈ K arg min 2 1 ∥ y − ( x − v ) ∥ 2 = Π K ( x − v ) . For item 4, the unconstrained first-order condition for
α 2 ( 1 − α ) ∥ y − u ∥ 2 + 1 2 ∥ y − x ∥ 2 \frac{\alpha}{2(1-\alpha)}\lVert y-u\rVert^2+\frac{1}{2}\lVert y-x\rVert^2 2 ( 1 − α ) α ∥ y − u ∥ 2 + 2 1 ∥ y − x ∥ 2 gives y = ( 1 − α ) x + α u y=(1-\alpha)x+\alpha u y = ( 1 − α ) x + α u K K K x = a + h x=a+h x = a + h h ∈ T h\in T h ∈ T a ∈ K a\in K a ∈ K ϕ ( a ) ∈ K \phi(a)\in K ϕ ( a ) ∈ K D ϕ ( a ) = a − ϕ ( a ) D_\phi(a)=a-\phi(a) D ϕ ( a ) = a − ϕ ( a ) T T T r ∈ T r\in T r ∈ T
D ϕ ( a + h + r ) − D ϕ ( a + h ) = ( I − A ) r . D_\phi(a+h+r)-D_\phi(a+h)=(I-A)r. D ϕ ( a + h + r ) − D ϕ ( a + h ) = ( I − A ) r . Differentiating (3) in direction r r r
d Ψ ( a + h ) [ r ] = ⟨ D ϕ ( a ) , r ⟩ + ⟨ h , ( I − A ) r ⟩ = ⟨ D ϕ ( a ) + ( I − A ) h , r ⟩ = ⟨ D ϕ ( a + h ) , r ⟩ . \begin{aligned}
\mathrm{d}\Psi(a+h)[r]&=\langle D_\phi(a),r\rangle+\langle h,(I-A)r\rangle\\
&=\langle D_\phi(a)+(I-A)h,r\rangle\\
&=\langle D_\phi(a+h),r\rangle.
\end{aligned} d Ψ ( a + h ) [ r ] = ⟨ D ϕ ( a ) , r ⟩ + ⟨ h , ( I − A ) r ⟩ = ⟨ D ϕ ( a ) + ( I − A ) h , r ⟩ = ⟨ D ϕ ( a + h ) , r ⟩ . The second equality uses self-adjointness on T T T Ψ \Psi Ψ D ϕ D_\phi D ϕ Definition 3.1 , let P T P_T P T T T T z ∈ R d z\in\mathbb{R}^d z ∈ R d
Ψ ~ ( z ) : = ⟨ D ϕ ( a ) , P T ( z − a ) ⟩ + 1 2 ⟨ P T ( z − a ) , ( I − A ) P T ( z − a ) ⟩ . \widetilde{\Psi}(z):=\langle D_\phi(a),P_T(z-a)\rangle+\frac{1}{2}\langle P_T(z-a),(I-A)P_T(z-a)\rangle. Ψ ( z ) := ⟨ D ϕ ( a ) , P T ( z − a )⟩ + 2 1 ⟨ P T ( z − a ) , ( I − A ) P T ( z − a )⟩ . Self-adjointness on T T T D ϕ ( a ) ∈ T D_\phi(a)\in T D ϕ ( a ) ∈ T ∇ Ψ ~ ( a + h ) = D ϕ ( a ) + ( I − A ) h = D ϕ ( a + h ) \nabla\widetilde{\Psi}(a+h)=D_\phi(a)+(I-A)h=D_\phi(a+h) ∇ Ψ ( a + h ) = D ϕ ( a ) + ( I − A ) h = D ϕ ( a + h ) h ∈ T h\in T h ∈ T Ψ ~ \widetilde{\Psi} Ψ K K K ϕ ( K ) ⊆ K \phi(K)\subseteq K ϕ ( K ) ⊆ K
Proofs for Section 4
Lemma 7.2. If D = ∇ Ψ D=\nabla\Psi D = ∇Ψ K K K ∮ γ D ( x ) T d x = 0 \oint_\gamma D(x)^{\mathsf{T}}\mathrm{d}x=0 ∮ γ D ( x ) T d x = 0 γ ⊂ K \gamma\subset K γ ⊂ K
Proof. By the chain rule,
∫ 0 1 ⟨ ∇ Ψ ( γ ( s ) ) , γ ′ ( s ) ⟩ d s = ∫ 0 1 d d s Ψ ( γ ( s ) ) d s = Ψ ( γ ( 1 ) ) − Ψ ( γ ( 0 ) ) = 0. \int_0^1\langle\nabla\Psi(\gamma(s)),\gamma'(s)\rangle\mathrm{d}s
=
\int_0^1\frac{\mathrm{d}}{\mathrm{d}s}\Psi(\gamma(s))\mathrm{d}s
=
\Psi(\gamma(1))-\Psi(\gamma(0))=0. ∫ 0 1 ⟨ ∇Ψ ( γ ( s )) , γ ′ ( s )⟩ d s = ∫ 0 1 d s d Ψ ( γ ( s )) d s = Ψ ( γ ( 1 )) − Ψ ( γ ( 0 )) = 0. Lemma D.2. Let z 0 , z 1 , … , z m = z 0 z_0,z_1,\ldots,z_m=z_0 z 0 , z 1 , … , z m = z 0 K K K
C : = ∑ i = 0 m − 1 ⟨ D ( z i ) , z i − z i + 1 ⟩ > 0. (25) C:=\sum_{i=0}^{m-1}\langle D(z_i),z_i-z_{i+1}\rangle>0. \tag*{(25)} C := i = 0 ∑ m − 1 ⟨ D ( z i ) , z i − z i + 1 ⟩ > 0. (25) Run projected GD with step size η \eta η x 1 = z 0 x_1=z_0 x 1 = z 0
g i + 1 : = z i − z i + 1 η , i = 0 , … , m − 1. (26) g_{i+1}:=\frac{z_i-z_{i+1}}{\eta},\qquad i=0,\ldots,m-1. \tag*{(26)} g i + 1 := η z i − z i + 1 , i = 0 , … , m − 1. (26) Then the iterates follow the polygon exactly and the regret accumulated in one traversal is C / η C/\eta C / η
Proof. If x i + 1 = z i x_{i+1}=z_i x i + 1 = z i
x i + 1 − η g i + 1 = z i − ( z i − z i + 1 ) = z i + 1 ∈ K . x_{i+1}-\eta g_{i+1}=z_i-(z_i-z_{i+1})=z_{i+1}\in K. x i + 1 − η g i + 1 = z i − ( z i − z i + 1 ) = z i + 1 ∈ K . Projection leaves it unchanged, so x i + 2 = z i + 1 x_{i+2}=z_{i+1} x i + 2 = z i + 1
∑ i = 0 m − 1 ⟨ g i + 1 , D ( z i ) ⟩ = 1 η ∑ i = 0 m − 1 ⟨ D ( z i ) , z i − z i + 1 ⟩ = C η . \sum_{i=0}^{m-1}\langle g_{i+1},D(z_i)\rangle=\frac{1}{\eta}\sum_{i=0}^{m-1}\langle D(z_i),z_i-z_{i+1}\rangle=\frac{C}{\eta}. i = 0 ∑ m − 1 ⟨ g i + 1 , D ( z i )⟩ = η 1 i = 0 ∑ m − 1 ⟨ D ( z i ) , z i − z i + 1 ⟩ = η C . Theorem 7.3. Let D : K → R d D:K\to\mathbb{R}^d D : K → R d C 1 C^1 C 1 γ : [ 0 , 1 ] → K \gamma:[0,1]\to K γ : [ 0 , 1 ] → K
− ∮ γ D ( x ) T d x ≔ c 0 > 0. -\oint_\gamma D(x)^{\mathsf T}\mathrm{d}x\coloneqq c_0>0. − ∮ γ D ( x ) T d x : = c 0 > 0. Then for every gradient bound G > 0 G>0 G > 0 η > 0 \eta>0 η > 0 ∥ g t ∥ ≤ G \lVert g_t\rVert\le G ∥ g t ∥ ≤ G γ ( 0 ) \gamma(0) γ ( 0 )
∑ t = 1 T ⟨ g t , D ( x t ) ⟩ ≥ c T − O ( 1 / η ) \sum_{t=1}^{T}\langle g_t,D(x_t)\rangle\ge cT-O(1/\eta) t = 1 ∑ T ⟨ g t , D ( x t )⟩ ≥ c T − O ( 1/ η ) for a constant c > 0 c>0 c > 0 D D D γ \gamma γ G G G Ω ( T ) \Omega(T) Ω ( T ) K K K O ( 1 / η ) O(1/\eta) O ( 1/ η )
Proof. Let L γ L_\gamma L γ γ \gamma γ
− ∮ γ D ( x ) T d x -\oint_\gamma D(x)^{\mathsf T}\mathrm{d}x − ∮ γ D ( x ) T d x is the limit of left-endpoint Riemann sums
∑ i = 0 m − 1 ⟨ D ( z i ) , z i − z i + 1 ⟩ , z i = γ ( s i ) , \sum_{i=0}^{m-1}\langle D(z_i),z_i-z_{i+1}\rangle,\qquad z_i=\gamma(s_i), i = 0 ∑ m − 1 ⟨ D ( z i ) , z i − z i + 1 ⟩ , z i = γ ( s i ) , as the mesh of the partition 0 = s 0 < s 1 < ⋯ < s m = 1 0=s_0<s_1<\cdots<s_m=1 0 = s 0 < s 1 < ⋯ < s m = 1 η \eta η η G \eta G η G
∑ i = 0 m − 1 ⟨ D ( z i ) , z i − z i + 1 ⟩ ≥ c 0 2 , \sum_{i=0}^{m-1}\langle D(z_i),z_i-z_{i+1}\rangle\ge\frac{c_0}{2}, i = 0 ∑ m − 1 ⟨ D ( z i ) , z i − z i + 1 ⟩ ≥ 2 c 0 , and with m ≤ 2 L γ / ( η G ) m\le2L_\gamma/(\eta G) m ≤ 2 L γ / ( η G )
g i + 1 ≔ z i − z i + 1 η , i = 0 , … , m − 1. g_{i+1}\coloneqq\frac{z_i-z_{i+1}}{\eta},\qquad i=0,\ldots,m-1. g i + 1 : = η z i − z i + 1 , i = 0 , … , m − 1. Then ∥ g i + 1 ∥ ≤ G \lVert g_{i+1}\rVert\le G ∥ g i + 1 ∥ ≤ G c 0 / ( 2 η ) c_0/(2\eta) c 0 / ( 2 η )
c 0 / ( 2 η ) 2 L γ / ( η G ) = c 0 G 4 L γ . \frac{c_0/(2\eta)}{2L_\gamma/(\eta G)}=\frac{c_0G}{4L_\gamma}. 2 L γ / ( η G ) c 0 / ( 2 η ) = 4 L γ c 0 G . Repeat the cycle for ⌊ T / m ⌋ \lfloor T/m\rfloor ⌊ T / m ⌋
∑ t = 1 T ⟨ g t , D ( x t ) ⟩ ≥ c 0 G 4 L γ T − O ( 1 / η ) , \sum_{t=1}^{T} \langle g_t, D(x_t)\rangle\ge\frac{c_0G}{4L\gamma}T - O(1/\eta), t = 1 ∑ T ⟨ g t , D ( x t )⟩ ≥ 4 L γ c 0 G T − O ( 1/ η ) , which proves the claim.
Corollary 7.5. Let E K : = span ( K − K ) E_K := \operatorname{span}(K-K) E K := span ( K − K ) U : = relint ( K ) U := \operatorname{relint}(K) U := relint ( K ) aff ( K ) \operatorname{aff}(K) aff ( K ) D : U → E K D : U \to E_K D : U → E K C 1 C^1 C 1 E K E_K E K ∂ j D i = ∂ i D j \partial_jD_i = \partial_iD_j ∂ j D i = ∂ i D j U U U D = ∇ E K Ψ D = \nabla_{E_K}\Psi D = ∇ E K Ψ U U U C 2 C^2 C 2 Ψ \Psi Ψ D D D Ψ \Psi Ψ Ψ ~ \widetilde{\Psi} Ψ K K K ϕ ~ ( x ) : = x − ∇ Ψ ~ ( x ) \widetilde{\phi}(x) := x-\nabla\widetilde{\Psi}(x) ϕ ( x ) := x − ∇ Ψ ( x ) K K K x ∈ K x\in K x ∈ K O ( T ) O(\sqrt{T}) O ( T ) ϕ ~ \widetilde{\phi} ϕ
Proof. The equivalence between vanishing skew derivatives and existence of a potential is the Poincaré lemma for C 1 C^1 C 1 U U U E K E_K E K U U U
Proofs for Section 5
Proposition 8.3. A field D : D → E D : \mathcal{D} \to E D : D → E R R R V : D → R V : \mathcal{D} \to\mathbb{R} V : D → R
∇ V ( x ) = ∇ 2 R ( x ) D ( x ) . (27) \nabla V(x) = \nabla^2 R(x)D(x). \tag*{(27)} ∇ V ( x ) = ∇ 2 R ( x ) D ( x ) . (27) Equivalently, D ( x ) T d ∇ R ( x ) = d V ( x ) D(x)^{\mathsf{T}}\mathrm{d}\nabla R(x) = \mathrm{d}V(x) D ( x ) T d ∇ R ( x ) = d V ( x )
Proof. First suppose D ( x ) = ∇ θ Ψ ( ∇ R ( x ) ) D(x) = \nabla_\theta\Psi(\nabla R(x)) D ( x ) = ∇ θ Ψ ( ∇ R ( x ))
V ( x ) : = Ψ ( ∇ R ( x ) ) . V(x) := \Psi(\nabla R(x)). V ( x ) := Ψ ( ∇ R ( x )) . By the chain rule,
∇ V ( x ) = ∇ 2 R ( x ) ∇ θ Ψ ( ∇ R ( x ) ) = ∇ 2 R ( x ) D ( x ) . \nabla V(x) = \nabla^2R(x)\nabla_\theta\Psi(\nabla R(x)) = \nabla^2R(x)D(x). ∇ V ( x ) = ∇ 2 R ( x ) ∇ θ Ψ ( ∇ R ( x )) = ∇ 2 R ( x ) D ( x ) . Conversely, suppose (5) holds. For θ ∈ Θ \theta\in\Theta θ ∈ Θ x : = ( ∇ R ) − 1 ( θ ) x := (\nabla R)^{-1}(\theta) x := ( ∇ R ) − 1 ( θ )
Ψ ( θ ) : = V ( ( ∇ R ) − 1 ( θ ) ) . \Psi(\theta) := V((\nabla R)^{-1}(\theta)). Ψ ( θ ) := V (( ∇ R ) − 1 ( θ )) . Another application of the chain rule gives
∇ θ Ψ ( θ ) = ( ∇ 2 R ( x ) ) − 1 ∇ V ( x ) = D ( x ) , \nabla_\theta\Psi(\theta) = (\nabla^2R(x))^{-1}\nabla V(x) = D(x), ∇ θ Ψ ( θ ) = ( ∇ 2 R ( x ) ) − 1 ∇ V ( x ) = D ( x ) , which is the desired dual-gradient representation.
Examples of Mirror-exact Deviations for Section 5.1
Throughout this subsection, R ∗ R^* R ∗
∇ R ∗ ( θ ) = ( ∇ R ) − 1 ( θ ) \nabla R^*(\theta) = (\nabla R)^{-1}(\theta) ∇ R ∗ ( θ ) = ( ∇ R ) − 1 ( θ ) whenever the inverse is defined. Each displayed map is a valid deviation once it is pointwise feasible, i.e. once it maps D \mathcal{D} D K K K
1. Fixed comparators: external regret is always mirror-exact. Fix u ∈ K u \in K u ∈ K
ϕ u ( x ) : = u , D u ( x ) : = x − u . \phi_u(x) := u,\qquad D_u(x) := x-u. ϕ u ( x ) := u , D u ( x ) := x − u . Take the dual potential
Ψ u ( θ ) : = R ∗ ( θ ) − ⟨ u , θ ⟩ . \Psi_u(\theta) := R^*(\theta) - \langle u,\theta\rangle. Ψ u ( θ ) := R ∗ ( θ ) − ⟨ u , θ ⟩ . Then, with θ = ∇ R ( x ) \theta= \nabla R(x) θ = ∇ R ( x )
∇ θ Ψ u ( θ ) = ∇ R ∗ ( θ ) − u = x − u = D u ( x ) . \nabla_\theta\Psi_u(\theta) = \nabla R^*(\theta) - u = x-u = D_u(x). ∇ θ Ψ u ( θ ) = ∇ R ∗ ( θ ) − u = x − u = D u ( x ) . Thus every fixed-comparator deviation is R R R ϕ ( x ) = x \phi(x)=x ϕ ( x ) = x D ϕ ≡ 0 D_\phi\equiv0 D ϕ ≡ 0 Ψ ≡ 0 \Psi\equiv0 Ψ ≡ 0
2. Dual translations: one mirror step on a linear loss. Let a ∈ E a \in E a ∈ E θ − a ∈ Θ \theta-a \in\Theta θ − a ∈ Θ θ ∈ Θ \theta\in\Theta θ ∈ Θ
ϕ a ( x ) : = ∇ R ∗ ( ∇ R ( x ) − a ) . (28) \phi_a(x) := \nabla R^*(\nabla R(x)-a). \tag*{(28)} ϕ a ( x ) := ∇ R ∗ ( ∇ R ( x ) − a ) . (28) This is the unconstrained Bregman proximal map for the linear function y ↦ ⟨ a , y ⟩ y \mapsto\langle a,y\rangle y ↦ ⟨ a , y ⟩
Ψ a ( θ ) : = R ∗ ( θ ) − R ∗ ( θ − a ) , \Psi_a(\theta) := R^*(\theta)-R^*(\theta-a), Ψ a ( θ ) := R ∗ ( θ ) − R ∗ ( θ − a ) , since
∇ θ Ψ a ( θ ) = ∇ R ∗ ( θ ) − ∇ R ∗ ( θ − a ) = x − ϕ a ( x ) . \nabla_\theta\Psi_a(\theta) = \nabla R^*(\theta)-\nabla R^*(\theta-a)=x-\phi_a(x). ∇ θ Ψ a ( θ ) = ∇ R ∗ ( θ ) − ∇ R ∗ ( θ − a ) = x − ϕ a ( x ) . So a fixed translation in dual space is an exact-form deviation for mirror descent. This is often the most concrete way to remember the definition.
3. Mirror interpolation toward a comparator. Fix u ∈ D u \in D u ∈ D θ u : = ∇ R ( u ) \theta_u := \nabla R(u) θ u := ∇ R ( u ) λ ∈ [ 0 , 1 ) \lambda\in[0,1) λ ∈ [ 0 , 1 )
( 1 − λ ) θ + λ θ u ∈ Θ ∀ θ ∈ Θ . (1-\lambda)\theta+\lambda\theta_u \in\Theta\qquad\forall\theta\in\Theta. ( 1 − λ ) θ + λ θ u ∈ Θ ∀ θ ∈ Θ. Define
ϕ λ , u R ( x ) : = ∇ R ∗ ( ( 1 − λ ) ∇ R ( x ) + λ ∇ R ( u ) ) . (29) \phi_{\lambda,u}^{R}(x) := \nabla R^*((1-\lambda)\nabla R(x)+\lambda\nabla R(u)). \tag*{(29)} ϕ λ , u R ( x ) := ∇ R ∗ (( 1 − λ ) ∇ R ( x ) + λ ∇ R ( u )) . (29) This moves x x x λ \lambda λ u u u
Ψ λ , u ( θ ) : = R ∗ ( θ ) − 1 1 − λ R ∗ ( ( 1 − λ ) θ + λ θ u ) \Psi_{\lambda,u}(\theta) := R^*(\theta)-\frac{1}{1-\lambda}R^*((1-\lambda)\theta+\lambda\theta_u) Ψ λ , u ( θ ) := R ∗ ( θ ) − 1 − λ 1 R ∗ (( 1 − λ ) θ + λ θ u ) works because
∇ θ Ψ λ , u ( θ ) = ∇ R ∗ ( θ ) − ∇ R ∗ ( ( 1 − λ ) θ + λ θ u ) = x − ϕ λ , u R ( x ) . \begin{aligned}
\nabla_\theta\Psi_{\lambda,u}(\theta)
&=\nabla R^*(\theta)-\nabla R^*((1-\lambda)\theta+\lambda\theta_u)\\
&=x-\phi_{\lambda,u}^{R}(x).
\end{aligned} ∇ θ Ψ λ , u ( θ ) = ∇ R ∗ ( θ ) − ∇ R ∗ (( 1 − λ ) θ + λ θ u ) = x − ϕ λ , u R ( x ) . When λ = 0 \lambda=0 λ = 0 λ ↑ 1 \lambda\uparrow1 λ ↑ 1 ϕ u \phi_u ϕ u
Entropy on the simplex: multiplicative deviations
Let K : = Δ d K := \Delta_d K := Δ d relint ( Δ d ) \operatorname{relint}(\Delta_d) relint ( Δ d )
R ( x ) : = ∑ i = 1 d x [ i ] log x [ i ] . R(x) := \sum_{i=1}^{d} x[i]\log x[i]. R ( x ) := i = 1 ∑ d x [ i ] log x [ i ] . All derivatives are understood on the affine hull of the simplex. Equivalently, one may identify dual vectors that differ by a multiple of 1 \mathbf{1} 1 ∑ i θ [ i ] = 0 \sum_i \theta[i] = 0 ∑ i θ [ i ] = 0 ∇ R ∗ ( θ ) = softmax ( θ ) \nabla R^*(\theta) = \operatorname{softmax}(\theta) ∇ R ∗ ( θ ) = softmax ( θ ) (28) becomes
ϕ a ( x ) [ i ] : = x [ i ] exp ( − a [ i ] ) ∑ j = 1 d x [ j ] exp ( − a [ j ] ) . \phi_a(x)[i] := \frac{x[i]\exp(-a[i])}{\sum_{j=1}^{d}x[j]\exp(-a[j])}. ϕ a ( x ) [ i ] := ∑ j = 1 d x [ j ] exp ( − a [ j ]) x [ i ] exp ( − a [ i ]) . This is multiplicative-weights reweighting by exp ( − a ) \exp(-a) exp ( − a )
Ψ a ( θ ) : = log ∑ i = 1 d exp ( θ [ i ] ) − log ∑ i = 1 d exp ( θ [ i ] − a [ i ] ) , \Psi_a(\theta) := \log\sum_{i=1}^{d}\exp(\theta[i])-\log\sum_{i=1}^{d}\exp(\theta[i]-a[i]), Ψ a ( θ ) := log i = 1 ∑ d exp ( θ [ i ]) − log i = 1 ∑ d exp ( θ [ i ] − a [ i ]) , whose gradient is
∇ θ Ψ a ( θ ) = softmax ( θ ) − softmax ( θ − a ) . \nabla_\theta\Psi_a(\theta)=\operatorname{softmax}(\theta)-\operatorname{softmax}(\theta-a). ∇ θ Ψ a ( θ ) = softmax ( θ ) − softmax ( θ − a ) . Since softmax ( ∇ R ( x ) ) = x \operatorname{softmax}(\nabla R(x))=x softmax ( ∇ R ( x )) = x x − ϕ a ( x ) x-\phi_a(x) x − ϕ a ( x )
The dual interpolation (29) gives another familiar entropy example. For u ∈ relint ( Δ d ) u \in\operatorname{relint}(\Delta_d) u ∈ relint ( Δ d )
ϕ λ , u R ( x ) [ i ] : = x [ i ] 1 − λ u [ i ] λ ∑ j = 1 d x [ j ] 1 − λ u [ j ] λ . \phi_{\lambda,u}^{R}(x)[i] := \frac{x[i]^{1-\lambda}u[i]^\lambda}{\sum_{j=1}^{d}x[j]^{1-\lambda}u[j]^\lambda}. ϕ λ , u R ( x ) [ i ] := ∑ j = 1 d x [ j ] 1 − λ u [ j ] λ x [ i ] 1 − λ u [ i ] λ . Thus mirror-exact deviations include geometric interpolation on the simplex, not only Euclidean arithmetic interpolation.
Metric-symmetric affine swaps
Let K : = E : = R d K := E := \mathbb{R}^d K := E := R d R H ( x ) : = 1 2 x T H x R_H(x) := \frac{1}{2}x^\mathsf{T}Hx R H ( x ) := 2 1 x T H x H ≻ 0 H \succ0 H ≻ 0 ∇ R H ( x ) = H x \nabla R_H(x)=Hx ∇ R H ( x ) = H x D D D R H R_H R H H D ( x ) HD(x) H D ( x )
Consider an affine deviation
ϕ ( x ) : = A x + b , D ( x ) : = x − ϕ ( x ) = ( I − A ) x − b . \phi(x):=Ax+b,\qquad D(x):=x-\phi(x)=(I-A)x-b. ϕ ( x ) := A x + b , D ( x ) := x − ϕ ( x ) = ( I − A ) x − b . If
H ( I − A ) = ( I − A ) T H , (30) H(I-A)=(I-A)^\mathsf{T}H, \tag*{(30)} H ( I − A ) = ( I − A ) T H , (30) then D D D R H R_H R H K = R d K=\mathbb{R}^d K = R d
V ( x ) : = 1 2 x T H ( I − A ) x − ⟨ H b , x ⟩ V(x):=\frac{1}{2}x^\mathsf{T}H(I-A)x-\langle Hb,x\rangle V ( x ) := 2 1 x T H ( I − A ) x − ⟨ H b , x ⟩ satisfies ∇ V ( x ) = H D ( x ) \nabla V(x)=HD(x) ∇ V ( x ) = H D ( x ) H = I H=I H = I (30) says that I − A I-A I − A H H H
6. A mirror-exact field that is not Euclidean-exact. The previous example gives a feasible separation on an unconstrained action space. Let
H : = ( 1 0 0 2 ) , K : = R 2 . H := \begin{pmatrix} 1 & 0 \\ 0 & 2 \end{pmatrix}, \qquad K := \mathbb{R}^{2}. H := ( 1 0 0 2 ) , K := R 2 . use R H ( x ) : = 1 2 x T H x R_H(x) := \frac{1}{2}x^{\mathsf{T}} Hx R H ( x ) := 2 1 x T H x ε ≥ 0 \varepsilon\ge0 ε ≥ 0
D ( x ) [ 1 ] : = ε ( x [ 1 ] + x [ 2 ] ) , D ( x ) [ 2 ] : = ε 2 ( x [ 1 ] + x [ 2 ] ) , ϕ ( x ) : = x − D ( x ) . D(x)[1] := \varepsilon(x[1] + x[2]), \qquad D(x)[2] := \frac{\varepsilon}{2}(x[1] + x[2]), \qquad\phi(x) := x - D(x). D ( x ) [ 1 ] := ε ( x [ 1 ] + x [ 2 ]) , D ( x ) [ 2 ] := 2 ε ( x [ 1 ] + x [ 2 ]) , ϕ ( x ) := x − D ( x ) . The map is feasible because it maps R 2 \mathbb{R}^{2} R 2 S : = ( 1 1 1 1 ) S := \begin{pmatrix} 1 & 1 \\ 1 & 1 \end{pmatrix} S := ( 1 1 1 1 ) R H R_H R H
H D ( x ) = ε S x = ∇ ( ε 2 ( x [ 1 ] + x [ 2 ] ) 2 ) . HD(x) = \varepsilon Sx = \nabla\left(\frac{\varepsilon}{2}(x[1] + x[2])^{2}\right). H D ( x ) = εS x = ∇ ( 2 ε ( x [ 1 ] + x [ 2 ] ) 2 ) . But D D D ε > 0 \varepsilon> 0 ε > 0
∂ 2 D ( x ) [ 1 ] = ε , ∂ 1 D ( x ) [ 2 ] = ε 2 . \partial_{2}D(x)[1] = \varepsilon, \qquad\partial_{1}D(x)[2] = \frac{\varepsilon}{2}. ∂ 2 D ( x ) [ 1 ] = ε , ∂ 1 D ( x ) [ 2 ] = 2 ε . So mirror-exact regret is not just Euclidean exact-form regret written in different notation. The mirror map changes which deviations telescope.
Proofs for Section 5.2
Theorem 8.4. Assume the standing Legendre-domain conditions and that R R R m m m D \mathcal{D} D ϕ : D → K \phi: \mathcal{D} \to K ϕ : D → K R R R Ψ \Psi Ψ { x t } t = 1 T + 1 \{x_t\}_{t=1}^{T+1} { x t } t = 1 T + 1 (4) , initialized in D \mathcal{D} D D \mathcal{D} D
θ t : = ∇ R ( x t ) , D ϕ ( x t ) = ∇ θ Ψ ( θ t ) . \theta_t := \nabla R(x_t), \qquad D_{\phi}(x_t) = \nabla_{\theta}\Psi(\theta_t). θ t := ∇ R ( x t ) , D ϕ ( x t ) = ∇ θ Ψ ( θ t ) . Assume that (6) holds for every consecutive pair ( θ t + 1 , θ t ) (\theta_{t+1}, \theta_t) ( θ t + 1 , θ t ) t ∈ [ T ] t \in[T] t ∈ [ T ] Θ × Θ \Theta\times\Theta Θ × Θ
∑ t = 1 T ⟨ g t , D ϕ ( x t ) ⟩ ≤ Ψ ( θ 1 ) − Ψ ( θ T + 1 ) η + L η m ∑ t = 1 T ∥ g t ∥ 2 . (31) \sum_{t=1}^{T} \langle g_t, D_{\phi}(x_t)\rangle\le\frac{\Psi(\theta_1) - \Psi(\theta_{T+1})}{\eta} + \frac{L\eta}{m}\sum_{t=1}^{T}\lVert g_t\rVert^{2}. \tag*{(31)} t = 1 ∑ T ⟨ g t , D ϕ ( x t )⟩ ≤ η Ψ ( θ 1 ) − Ψ ( θ T + 1 ) + m L η t = 1 ∑ T ∥ g t ∥ 2 . (31) Consequently, suppose ∥ g t ∥ ≤ G \lVert g_t\rVert\le G ∥ g t ∥ ≤ G B B B
max 1 ≤ t ≤ T + 1 Ψ ( θ t ) − min 1 ≤ t ≤ T + 1 Ψ ( θ t ) ≤ B . \max_{1 \le t \le T+1}\Psi(\theta_t) - \min_{1 \le t \le T+1}\Psi(\theta_t) \le B. 1 ≤ t ≤ T + 1 max Ψ ( θ t ) − 1 ≤ t ≤ T + 1 min Ψ ( θ t ) ≤ B . Then choosing η ≍ 1 / T \eta\asymp1/\sqrt{T} η ≍ 1/ T O ( T ) O(\sqrt{T}) O ( T ) ϕ \phi ϕ B , L , G > 0 B, L, G > 0 B , L , G > 0 η = m B / ( L G 2 T ) \eta= \sqrt{mB/(LG^{2}T)} η = m B / ( L G 2 T ) 2 G L B T / m 2G\sqrt{LBT/m} 2 G L B T / m osc Θ ( Ψ ) ≤ B \operatorname{osc}_{\Theta}(\Psi) \le B osc Θ ( Ψ ) ≤ B
Proof. Because x t + 1 ∈ D = relint ( K ) x_{t+1} \in\mathcal{D} = \operatorname{relint}(K) x t + 1 ∈ D = relint ( K ) K K K x t + 1 x_{t+1} x t + 1 { 0 } \{0\} { 0 } (4) therefore reduces to the unconstrained dual update
θ t − θ t + 1 = η g t . \theta_t - \theta_{t+1} = \eta g_t. θ t − θ t + 1 = η g t . Since D ϕ ( x t ) = ∇ θ Ψ ( θ t ) D_{\phi}(x_t) = \nabla_{\theta}\Psi(\theta_t) D ϕ ( x t ) = ∇ θ Ψ ( θ t )
η ⟨ g t , D ϕ ( x t ) ⟩ = Ψ ( θ t ) − Ψ ( θ t + 1 ) + D Ψ ( θ t + 1 , θ t ) ≤ Ψ ( θ t ) − Ψ ( θ t + 1 ) + L D R ∗ ( θ t + 1 , θ t ) . \begin{aligned}
\eta\langle g_t, D_{\phi}(x_t)\rangle= \Psi(\theta_t) - \Psi(\theta_{t+1}) + D_{\Psi}(\theta_{t+1}, \theta_t) \\
&\le\Psi(\theta_t) - \Psi(\theta_{t+1}) + L D_{R^{*}}(\theta_{t+1}, \theta_t).
\end{aligned} η ⟨ g t , D ϕ ( x t )⟩ = Ψ ( θ t ) − Ψ ( θ t + 1 ) + D Ψ ( θ t + 1 , θ t ) ≤ Ψ ( θ t ) − Ψ ( θ t + 1 ) + L D R ∗ ( θ t + 1 , θ t ) . By the Legendre conjugacy identity stated with the standing assumptions,
D R ∗ ( θ t + 1 , θ t ) = D R ( x t , x t + 1 ) . D_{R^*}(\theta_{t+1},\theta_t) = D_R(x_t,x_{t+1}). D R ∗ ( θ t + 1 , θ t ) = D R ( x t , x t + 1 ) . Therefore
η ⟨ g t , D ϕ ( x t ) ⟩ ≤ Ψ ( θ t ) − Ψ ( θ t + 1 ) + L D R ( x t , x t + 1 ) . (32) \eta\langle g_t,D_\phi(x_t)\rangle\le\Psi(\theta_t)-\Psi(\theta_{t+1})+LD_R(x_t,x_{t+1}). \tag*{(32)} η ⟨ g t , D ϕ ( x t )⟩ ≤ Ψ ( θ t ) − Ψ ( θ t + 1 ) + L D R ( x t , x t + 1 ) . (32) Let s t : = x t − x t + 1 s_t := x_t-x_{t+1} s t := x t − x t + 1
D R ( x t , x t + 1 ) + D R ( x t + 1 , x t ) = ⟨ θ t − θ t + 1 , s t ⟩ = η ⟨ g t , s t ⟩ . D_R(x_t,x_{t+1})+D_R(x_{t+1},x_t)=\langle\theta_t-\theta_{t+1},s_t\rangle=\eta\langle g_t,s_t\rangle. D R ( x t , x t + 1 ) + D R ( x t + 1 , x t ) = ⟨ θ t − θ t + 1 , s t ⟩ = η ⟨ g t , s t ⟩ . Strong convexity of R R R
m ∥ s t ∥ 2 ≤ η ⟨ g t , s t ⟩ ≤ η ∥ g t ∥ ∥ s t ∥ . m\lVert s_t\rVert^2\le\eta\langle g_t,s_t\rangle\le\eta\lVert g_t\rVert\lVert s_t\rVert. m ∥ s t ∥ 2 ≤ η ⟨ g t , s t ⟩ ≤ η ∥ g t ∥ ∥ s t ∥ . Hence ∥ s t ∥ ≤ ( η / m ) ∥ g t ∥ \lVert s_t\rVert\le(\eta/m)\lVert g_t\rVert ∥ s t ∥ ≤ ( η / m ) ∥ g t ∥ s t = 0 s_t=0 s t = 0
D R ( x t , x t + 1 ) ≤ η ⟨ g t , s t ⟩ ≤ η 2 m ∥ g t ∥ 2 . D_R(x_t,x_{t+1})\le\eta\langle g_t,s_t\rangle\le\frac{\eta^2}{m}\lVert g_t\rVert^2. D R ( x t , x t + 1 ) ≤ η ⟨ g t , s t ⟩ ≤ m η 2 ∥ g t ∥ 2 . Combining this estimate with (32) gives the slightly stronger one-step bound
η ⟨ g t , D ϕ ( x t ) ⟩ ≤ Ψ ( θ t ) − Ψ ( θ t + 1 ) + L η 2 m ∥ g t ∥ 2 . \eta\langle g_t,D_\phi(x_t)\rangle\le\Psi(\theta_t)-\Psi(\theta_{t+1})+\frac{L\eta^2}{m}\lVert g_t\rVert^2. η ⟨ g t , D ϕ ( x t )⟩ ≤ Ψ ( θ t ) − Ψ ( θ t + 1 ) + m L η 2 ∥ g t ∥ 2 . Dividing by η \eta η t t t O ( T ) O(\sqrt{T}) O ( T ) B / η B/\eta B / η η \eta η
Proofs for Section 5.3
Lemma D.3. For fixed y ∈ dom R y\in\operatorname{dom}R y ∈ dom R
∇ x D R ( y , x ) = ∇ 2 R ( x ) ( x − y ) . \nabla_xD_R(y,x)=\nabla^2R(x)(x-y). ∇ x D R ( y , x ) = ∇ 2 R ( x ) ( x − y ) . Proof. By definition,
D R ( y , x ) : = R ( y ) − R ( x ) − ⟨ ∇ R ( x ) , y − x ⟩ . D_R(y,x):=R(y)-R(x)-\langle\nabla R(x),y-x\rangle. D R ( y , x ) := R ( y ) − R ( x ) − ⟨ ∇ R ( x ) , y − x ⟩ . Only the last two terms depend on x x x
h ( x ) = ⟨ ∇ R ( x ) , y − x ⟩ . h(x)=\langle\nabla R(x),y-x\rangle. h ( x ) = ⟨ ∇ R ( x ) , y − x ⟩ . Then
∇ h ( x ) = ∇ 2 R ( x ) ( y − x ) − ∇ R ( x ) . \nabla h(x)=\nabla^2R(x)(y-x)-\nabla R(x). ∇ h ( x ) = ∇ 2 R ( x ) ( y − x ) − ∇ R ( x ) . Thus
∇ x D R ( y , x ) = − ∇ R ( x ) − ∇ h ( x ) = ∇ 2 R ( x ) ( x − y ) . \begin{aligned}
\nabla_xD_R(y,x)&=-\nabla R(x)-\nabla h(x)\\
&=\nabla^2R(x)(x-y).
\end{aligned} ∇ x D R ( y , x ) = − ∇ R ( x ) − ∇ h ( x ) = ∇ 2 R ( x ) ( x − y ) . Lemma D.4 (Bregman-Moreau identity). Assume the standing Legendre-domain conditions. Let F : = f + ι K F := f + \iota_K F := f + ι K prox F R \operatorname{prox}^{R}_{F} prox F R D D D M F R M^{R}_{F} M F R D D D
∇ M F R ( x ) = ∇ 2 R ( x ) ( x − prox F R ( x ) ) . (33) \nabla M^{R}_{F}(x) = \nabla^{2}R(x)\left(x-\operatorname{prox}^{R}_{F}(x)\right). \tag*{(33)} ∇ M F R ( x ) = ∇ 2 R ( x ) ( x − prox F R ( x ) ) . (33) Consequently, if θ = ∇ R ( x ) \theta= \nabla R(x) θ = ∇ R ( x )
Ψ F ( θ ) = M F R ( ( ∇ R ) − 1 ( θ ) ) , \Psi_{F}(\theta) = M^{R}_{F}\left((\nabla R)^{-1}(\theta)\right), Ψ F ( θ ) = M F R ( ( ∇ R ) − 1 ( θ ) ) , then
∇ θ Ψ F ( θ ) = x − prox F R ( x ) , x = ( ∇ R ) − 1 ( θ ) . (34) \nabla_{\theta}\Psi_{F}(\theta) = x-\operatorname{prox}^{R}_{F}(x), \qquad x = (\nabla R)^{-1}(\theta). \tag*{(34)} ∇ θ Ψ F ( θ ) = x − prox F R ( x ) , x = ( ∇ R ) − 1 ( θ ) . (34) Proof. The key point is that we do not differentiate the minimizer. Continuity of prox F R \operatorname{prox}^{R}_{F} prox F R x x x
∇ M F R ( x ) = ∇ x D R ( prox F R ( x ) , x ) . \nabla M^{R}_{F}(x) = \nabla_{x}D_{R}\left(\operatorname{prox}^{R}_{F}(x),x\right). ∇ M F R ( x ) = ∇ x D R ( prox F R ( x ) , x ) . Using Lemma D.3, we obtain
∇ M F R ( x ) = ∇ 2 R ( x ) ( x − prox F R ( x ) ) , \nabla M^{R}_{F}(x) = \nabla^{2}R(x)\left(x-\operatorname{prox}^{R}_{F}(x)\right), ∇ M F R ( x ) = ∇ 2 R ( x ) ( x − prox F R ( x ) ) , which proves (33) .
For the dual identity, write x : = ( ∇ R ) − 1 ( θ ) x := (\nabla R)^{-1}(\theta) x := ( ∇ R ) − 1 ( θ )
∂ x ∂ θ = ( ∇ 2 R ( x ) ) − 1 , \frac{\partial x}{\partial\theta} = \left(\nabla^{2}R(x)\right)^{-1}, ∂ θ ∂ x = ( ∇ 2 R ( x ) ) − 1 , the chain rule gives
∇ θ Ψ F ( θ ) = ( ∇ 2 R ( x ) ) − 1 ∇ M F R ( x ) = x − prox F R ( x ) . \nabla_{\theta}\Psi_{F}(\theta) = \left(\nabla^{2}R(x)\right)^{-1}\nabla M^{R}_{F}(x) = x-\operatorname{prox}^{R}_{F}(x). ∇ θ Ψ F ( θ ) = ( ∇ 2 R ( x ) ) − 1 ∇ M F R ( x ) = x − prox F R ( x ) . Theorem 8.6. Assume the standing Legendre-domain conditions. Let F : = f + ι K F := f + \iota_K F := f + ι K prox F R \operatorname{prox}^{R}_{F} prox F R D D D ϕ F R : D → K \phi^{R}_{F}:D\to K ϕ F R : D → K ϕ F R ( x ) : = prox F R ( x ) \phi^{R}_{F}(x):=\operatorname{prox}^{R}_{F}(x) ϕ F R ( x ) := prox F R ( x ) R R R
( x − prox F R ( x ) ) T d ∇ R ( x ) = d M F R ( x ) . (35) \left(x-\operatorname{prox}^{R}_{F}(x)\right)^{\mathsf T}\mathrm{d}\nabla R(x) = \mathrm{d}M^{R}_{F}(x). \tag*{(35)} ( x − prox F R ( x ) ) T d ∇ R ( x ) = d M F R ( x ) . (35) Thus Bregman proximal regret is a special case of mirror-exact regret.
Proof. Feasibility follows from the indicator ι K \iota_K ι K K K K (34) in Lemma D.4 , the displacement
x − ϕ F R ( x ) = x − prox F R ( x ) x-\phi^{R}_{F}(x)=x-\operatorname{prox}^{R}_{F}(x) x − ϕ F R ( x ) = x − prox F R ( x ) equals the dual gradient ∇ θ Ψ F ( ∇ R ( x ) ) \nabla_{\theta}\Psi_{F}(\nabla R(x)) ∇ θ Ψ F ( ∇ R ( x )) R R R (33) in Lemma D.4 .
Proposition 8.7. Assume the standing Legendre-domain conditions. Assume, in the extended-valued sense on E E E R R R m m m F : = f + ι K F := f + \iota_K F := f + ι K ρ \rho ρ 0 ≤ ρ < m 0 \le\rho< m 0 ≤ ρ < m H : = F + R H := F + R H := F + R D D D
and its solution map is continuous there. Let { x t } t = 1 T + 1 ⊂ D \{x_t\}_{t=1}^{T+1} \subset D { x t } t = 1 T + 1 ⊂ D (4) . Then these iterates control Bregman proximal regret with the bound
∑ t = 1 T ⟨ g t , x t − prox F R ( x t ) ⟩ ≤ Ψ F ( θ 1 ) − Ψ F ( θ T + 1 ) η + η m ∑ t = 1 T ∥ g t ∥ ∗ 2 , (36) \sum_{t=1}^{T} \langle g_t, x_t-\operatorname{prox}^{R}_{F}(x_t)\rangle\le\frac{\Psi_F(\theta_1)-\Psi_F(\theta_{T+1})}{\eta}+\frac{\eta}{m}\sum_{t=1}^{T}\lVert g_t\rVert_*^2, \tag*{(36)} t = 1 ∑ T ⟨ g t , x t − prox F R ( x t )⟩ ≤ η Ψ F ( θ 1 ) − Ψ F ( θ T + 1 ) + m η t = 1 ∑ T ∥ g t ∥ ∗ 2 , (36) where Ψ F ( θ t ) = M F R ( x t ) \Psi_F(\theta_t)=M_F^R(x_t) Ψ F ( θ t ) = M F R ( x t ) t ∈ [ T + 1 ] t\in[T+1] t ∈ [ T + 1 ]
Proof. Since F F F ρ \rho ρ R R R m m m H = F + R H=F+R H = F + R ( m − ρ ) (m-\rho) ( m − ρ ) θ ∈ Θ \theta\in\Theta θ ∈ Θ x x x
min y ∈ E { H ( y ) − ⟨ θ , y ⟩ } . \min_{y\in E}\{H(y)-\langle\theta,y\rangle\}. y ∈ E min { H ( y ) − ⟨ θ , y ⟩} . It therefore has the unique solution ∇ H ∗ ( θ ) \nabla H^*(\theta) ∇ H ∗ ( θ ) H ∗ H^* H ∗ 1 / ( m − ρ ) 1/(m-\rho) 1/ ( m − ρ ) D D D
Let x : = ∇ R ∗ ( θ ) x:=\nabla R^*(\theta) x := ∇ R ∗ ( θ )
Ψ F ( θ ) = M F R ( x ) = min y { F ( y ) + R ( y ) − R ( x ) − ⟨ ∇ R ( x ) , y − x ⟩ } = min y { F ( y ) + R ( y ) − ⟨ θ , y ⟩ } + ⟨ θ , x ⟩ − R ( x ) = − ( F + R ) ∗ ( θ ) + R ∗ ( θ ) . \begin{aligned}
\Psi_F(\theta)&=M_F^R(x)\\
&=\min_y\{F(y)+R(y)-R(x)-\langle\nabla R(x),y-x\rangle\}\\
&=\min_y\{F(y)+R(y)-\langle\theta,y\rangle\}+\langle\theta,x\rangle-R(x)\\
&=-(F+R)^*(\theta)+R^*(\theta).
\end{aligned} Ψ F ( θ ) = M F R ( x ) = y min { F ( y ) + R ( y ) − R ( x ) − ⟨ ∇ R ( x ) , y − x ⟩} = y min { F ( y ) + R ( y ) − ⟨ θ , y ⟩} + ⟨ θ , x ⟩ − R ( x ) = − ( F + R ) ∗ ( θ ) + R ∗ ( θ ) . This proves Ψ F ( θ ) = R ∗ ( θ ) − H ∗ ( θ ) \Psi_F(\theta)=R^*(\theta)-H^*(\theta) Ψ F ( θ ) = R ∗ ( θ ) − H ∗ ( θ ) H : = F + R H:=F+R H := F + R H ∗ H^* H ∗
D H ∗ ( θ ′ , θ ) ≥ 0. D_{H^*}(\theta',\theta)\ge0. D H ∗ ( θ ′ , θ ) ≥ 0. Taking the Bregman divergence of the identity Ψ F = R ∗ − H ∗ \Psi_F=R^*-H^* Ψ F = R ∗ − H ∗
D Ψ F ( θ ′ , θ ) = D R ∗ ( θ ′ , θ ) − D H ∗ ( θ ′ , θ ) ≤ D R ∗ ( θ ′ , θ ) , D_{\Psi_F}(\theta',\theta)=D_{R^*}(\theta',\theta)-D_{H^*}(\theta',\theta)\le D_{R^*}(\theta',\theta), D Ψ F ( θ ′ , θ ) = D R ∗ ( θ ′ , θ ) − D H ∗ ( θ ′ , θ ) ≤ D R ∗ ( θ ′ , θ ) , which proves the relative-smoothness with L = 1 L=1 L = 1 Ψ F \Psi_F Ψ F ∇ θ Ψ F ( θ t ) \nabla_\theta\Psi_F(\theta_t) ∇ θ Ψ F ( θ t ) Theorem 8.4 with L = 1 L=1 L = 1
Proofs for Section 8.4
Theorem 8.9. Assume R R R K K K X = K \mathcal{X}=K X = K { x t } t = 1 T + 1 \{x_t\}_{t=1}^{T+1} { x t } t = 1 T + 1 (4) with a constant step size η > 0 \eta>0 η > 0 ϕ : K → K \phi:K\to K ϕ : K → K R R R Ψ \Psi Ψ (10) holds and D ϕ D_\phi D ϕ L ϕ L_\phi L ϕ K K K
∑ t = 1 T ⟨ g t , D ϕ ( x t ) ⟩ ≤ Ψ ( ∇ R ( x 1 ) ) − Ψ ( ∇ R ( x T + 1 ) ) η + L ϕ m ∑ t = 1 T ∥ g t ∥ ∗ 2 + L ϕ m ∑ t = 1 T ∥ g t ∥ ∗ [ η ∥ g t ∥ ∗ + ω R ( η m ∥ g t ∥ ∗ ) ] . \begin{aligned}
\sum_{t=1}^{T}\langle g_t,D_\phi(x_t)\rangle
&\le\frac{\Psi(\nabla R(x_1))-\Psi(\nabla R(x_{T+1}))}{\eta}
+\frac{L_\phi}{m}\sum_{t=1}^{T}\lVert g_t\rVert_*^2\\
&\quad+\frac{L_\phi}{m}\sum_{t=1}^{T}\lVert g_t\rVert_*
\left[\eta\lVert g_t\rVert_*+\omega_R\left(\frac{\eta}{m}\lVert g_t\rVert_*\right)\right].
\end{aligned} t = 1 ∑ T ⟨ g t , D ϕ ( x t )⟩ ≤ η Ψ ( ∇ R ( x 1 )) − Ψ ( ∇ R ( x T + 1 )) + m L ϕ t = 1 ∑ T ∥ g t ∥ ∗ 2 + m L ϕ t = 1 ∑ T ∥ g t ∥ ∗ [ η ∥ g t ∥ ∗ + ω R ( m η ∥ g t ∥ ∗ ) ] . Proof. Set
θ t : = ∇ R ( x t ) , s t = x t − x t + 1 . \theta_t:=\nabla R(x_t),\qquad s_t=x_t-x_{t+1}. θ t := ∇ R ( x t ) , s t = x t − x t + 1 . The first-order optimality condition for the mirror-descent subproblem provides n t + 1 ∈ N K ( x t + 1 ) n_{t+1} \in N_K(x_{t+1}) n t + 1 ∈ N K ( x t + 1 )
η g t = θ t − θ t + 1 − n t + 1 . (37) \eta g_t = \theta_t - \theta_{t+1} - n_{t+1}. \tag*{(37)} η g t = θ t − θ t + 1 − n t + 1 . (37) The symmetrized Bregman identity gives
D R ( x t , x t + 1 ) + D R ( x t + 1 , x t ) = ⟨ θ t − θ t + 1 , s t ⟩ = η ⟨ g t , s t ⟩ + ⟨ n t + 1 , s t ⟩ . D_R(x_t,x_{t+1}) + D_R(x_{t+1},x_t) = \langle\theta_t - \theta_{t+1},s_t\rangle
= \eta\langle g_t,s_t\rangle+ \langle n_{t+1},s_t\rangle. D R ( x t , x t + 1 ) + D R ( x t + 1 , x t ) = ⟨ θ t − θ t + 1 , s t ⟩ = η ⟨ g t , s t ⟩ + ⟨ n t + 1 , s t ⟩ . Since x t ∈ K x_t \in K x t ∈ K n t + 1 ∈ N K ( x t + 1 ) n_{t+1} \in N_K(x_{t+1}) n t + 1 ∈ N K ( x t + 1 )
⟨ n t + 1 , s t ⟩ ≤ 0. \langle n_{t+1},s_t\rangle\le0. ⟨ n t + 1 , s t ⟩ ≤ 0. Strong convexity of R R R
m ∥ s t ∥ 2 ≤ η ∥ g t ∥ ∗ ∥ s t ∥ . m\lVert s_t\rVert^2 \le\eta\lVert g_t\rVert_* \lVert s_t\rVert. m ∥ s t ∥ 2 ≤ η ∥ g t ∥ ∗ ∥ s t ∥ . Consequently,
∥ s t ∥ ≤ η m ∥ g t ∥ ∗ , D R ( x t , x t + 1 ) ≤ η 2 m ∥ g t ∥ ∗ 2 . (38) \lVert s_t\rVert\le\frac{\eta}{m}\lVert g_t\rVert_*, \qquad D_R(x_t,x_{t+1}) \le\frac{\eta^2}{m}\lVert g_t\rVert_*^2. \tag*{(38)} ∥ s t ∥ ≤ m η ∥ g t ∥ ∗ , D R ( x t , x t + 1 ) ≤ m η 2 ∥ g t ∥ ∗ 2 . (38) The normal decomposition and the definition of ω R \omega_R ω R
∥ n t + 1 ∥ ∗ ≤ η ∥ g t ∥ ∗ + ω R ( η m ∥ g t ∥ ∗ ) . (39) \lVert n_{t+1}\rVert_* \le\eta\lVert g_t\rVert_* + \omega_R\left(\frac{\eta}{m}\lVert g_t\rVert_*\right). \tag*{(39)} ∥ n t + 1 ∥ ∗ ≤ η ∥ g t ∥ ∗ + ω R ( m η ∥ g t ∥ ∗ ) . (39) Write D t : = D ϕ ( x t ) D_t := D_\phi(x_t) D t := D ϕ ( x t ) (37) ,
η ⟨ g t , D t ⟩ = ⟨ θ t − θ t + 1 , D t ⟩ − ⟨ n t + 1 , D t ⟩ . \eta\langle g_t,D_t\rangle= \langle\theta_t-\theta_{t+1},D_t\rangle- \langle n_{t+1},D_t\rangle. η ⟨ g t , D t ⟩ = ⟨ θ t − θ t + 1 , D t ⟩ − ⟨ n t + 1 , D t ⟩ . Mirror exactness and (10) give
⟨ θ t − θ t + 1 , D t ⟩ = Ψ ( θ t ) − Ψ ( θ t + 1 ) + D Ψ ( θ t + 1 , θ t ) ≤ Ψ ( θ t ) − Ψ ( θ t + 1 ) + L D R ( x t , x t + 1 ) . \langle\theta_t-\theta_{t+1},D_t\rangle= \Psi(\theta_t)-\Psi(\theta_{t+1}) + D_\Psi(\theta_{t+1},\theta_t)
\le\Psi(\theta_t)-\Psi(\theta_{t+1}) + L D_R(x_t,x_{t+1}). ⟨ θ t − θ t + 1 , D t ⟩ = Ψ ( θ t ) − Ψ ( θ t + 1 ) + D Ψ ( θ t + 1 , θ t ) ≤ Ψ ( θ t ) − Ψ ( θ t + 1 ) + L D R ( x t , x t + 1 ) . It remains to control the normal term. Feasibility of ϕ \phi ϕ ϕ ( x t + 1 ) ∈ K \phi(x_{t+1}) \in K ϕ ( x t + 1 ) ∈ K
⟨ n t + 1 , D ϕ ( x t + 1 ) ⟩ ≥ 0. \langle n_{t+1},D_\phi(x_{t+1})\rangle\ge0. ⟨ n t + 1 , D ϕ ( x t + 1 )⟩ ≥ 0. Therefore
− ⟨ n t + 1 , D t ⟩ = − ⟨ n t + 1 , D ϕ ( x t + 1 ) ⟩ − ⟨ n t + 1 , D ϕ ( x t ) − D ϕ ( x t + 1 ) ⟩ ≤ L ϕ ∥ n t + 1 ∥ ∗ ∥ s t ∥ . -\langle n_{t+1},D_t\rangle= -\langle n_{t+1},D_\phi(x_{t+1})\rangle- \langle n_{t+1},D_\phi(x_t)-D_\phi(x_{t+1})\rangle
\le L_\phi\lVert n_{t+1}\rVert_*\lVert s_t\rVert. − ⟨ n t + 1 , D t ⟩ = − ⟨ n t + 1 , D ϕ ( x t + 1 )⟩ − ⟨ n t + 1 , D ϕ ( x t ) − D ϕ ( x t + 1 )⟩ ≤ L ϕ ∥ n t + 1 ∥ ∗ ∥ s t ∥ . Substituting (38) and (39) gives
η ⟨ g t , D t ⟩ ≤ Ψ ( θ t ) − Ψ ( θ t + 1 ) + L η 2 m ∥ g t ∥ ∗ 2 + L ϕ η m ∥ g t ∥ ∗ [ η ∥ g t ∥ ∗ + ω R ( η m ∥ g t ∥ ∗ ) ] . \eta\langle g_t,D_t\rangle\le\Psi(\theta_t)-\Psi(\theta_{t+1}) + \frac{L\eta^2}{m}\lVert g_t\rVert_*^2
+ \frac{L_\phi\eta}{m}\lVert g_t\rVert_*\left[\eta\lVert g_t\rVert_*+\omega_R\left(\frac{\eta}{m}\lVert g_t\rVert_*\right)\right]. η ⟨ g t , D t ⟩ ≤ Ψ ( θ t ) − Ψ ( θ t + 1 ) + m L η 2 ∥ g t ∥ ∗ 2 + m L ϕ η ∥ g t ∥ ∗ [ η ∥ g t ∥ ∗ + ω R ( m η ∥ g t ∥ ∗ ) ] . Divide by η \eta η t t t
Corollary 8.10. Assume the hypotheses of Theorem 8.9, ∥ g t ∥ ∗ ≤ G \lVert g_t\rVert_* \le G ∥ g t ∥ ∗ ≤ G osc ∇ R ( K ) ( Ψ ) ≤ B \operatorname{osc}_{\nabla R(K)}(\Psi) \le B osc ∇ R ( K ) ( Ψ ) ≤ B η = T − 1 / 2 \eta= T^{-1/2} η = T − 1/2
1 T ∑ t = 1 T ⟨ g t , D ϕ ( x t ) ⟩ ≤ B T + ( L + L ϕ ) G 2 m T + L ϕ G m ω R ( G m T ) . \frac{1}{T}\sum_{t=1}^{T}\langle g_t,D_\phi(x_t)\rangle\le\frac{B}{\sqrt{T}}+\frac{(L+L_\phi)G^2}{m\sqrt{T}}+\frac{L_\phi G}{m}\omega_R\left(\frac{G}{m\sqrt{T}}\right). T 1 t = 1 ∑ T ⟨ g t , D ϕ ( x t )⟩ ≤ T B + m T ( L + L ϕ ) G 2 + m L ϕ G ω R ( m T G ) . Consequently, mirror descent has o ( T ) o(T) o ( T ) ϕ \phi ϕ ∇ R \nabla R ∇ R M M M K K K
∑ t = 1 T ⟨ g t , D ϕ ( x t ) ⟩ ≤ B η + C R , ϕ η G 2 T , C R , ϕ : = L + L ϕ m + L ϕ M m 2 . \sum_{t=1}^{T}\langle g_t,D_\phi(x_t)\rangle\le\frac{B}{\eta}+C_{R,\phi}\eta G^2T,\qquad C_{R,\phi}:=\frac{L+L_\phi}{m}+\frac{L_\phi M}{m^2}. t = 1 ∑ T ⟨ g t , D ϕ ( x t )⟩ ≤ η B + C R , ϕ η G 2 T , C R , ϕ := m L + L ϕ + m 2 L ϕ M . The optimized bound is 2 G B C R , ϕ T 2G\sqrt{BC_{R,\phi}T} 2 G B C R , ϕ T B C R , ϕ > 0 BC_{R,\phi}>0 B C R , ϕ > 0
Proof. Apply Theorem 8.9 and bound the telescoping term by B B B ∇ R \nabla R ∇ R K K K
ω R ( G m T ) ⟶ 0 , \omega_R\left(\frac{G}{m\sqrt{T}}\right)\longrightarrow0, ω R ( m T G ) ⟶ 0 , so the average regret converges to zero. If ω R ( δ ) ≤ M δ \omega_R(\delta)\le M\delta ω R ( δ ) ≤ M δ C R , ϕ C_{R,\phi} C R , ϕ B / η + C R , ϕ η G 2 T B/\eta+C_{R,\phi}\eta G^2T B / η + C R , ϕ η G 2 T 2 G B C R , ϕ T 2G\sqrt{BC_{R,\phi}T} 2 G B C R , ϕ T B C R , ϕ > 0 BC_{R,\phi}>0 B C R , ϕ > 0
Theorem 8.11. Suppose that, after reversing the orientation of γ \gamma γ
− ∮ γ D ( x ) T ∇ R ( x ) : = c 0 > 0. -\oint_\gamma\mathbf{D}(x)^{\mathsf{T}}\nabla R(x):=c_0>0. − ∮ γ D ( x ) T ∇ R ( x ) := c 0 > 0. For every G > 0 G>0 G > 0 η > 0 \eta>0 η > 0 ∥ g t ∥ ∗ ≤ G \lVert g_t\rVert_*\le G ∥ g t ∥ ∗ ≤ G γ ( 0 ) \gamma(0) γ ( 0 )
∑ t = 1 T ⟨ g t , D ( x t ) ⟩ ≥ c T − O ( 1 / η ) \sum_{t=1}^{T}\langle g_t,D(x_t)\rangle\ge cT-O(1/\eta) t = 1 ∑ T ⟨ g t , D ( x t )⟩ ≥ c T − O ( 1/ η ) for a constant c > 0 c>0 c > 0 D D D γ \gamma γ R R R G G G
Proof. Let
θ ( s ) = ∇ R ( γ ( s ) ) . \theta(s)=\nabla R(\gamma(s)). θ ( s ) = ∇ R ( γ ( s )) . The curve θ \theta θ
∑ j = 0 q − 1 ⟨ D ( z j ) , θ j − θ j + 1 ⟩ , z j = γ ( s j ) , θ j = θ ( s j ) . \sum_{j=0}^{q-1}\langle D(z_j),\theta_j-\theta_{j+1}\rangle,\qquad z_j=\gamma(s_j),\quad\theta_j=\theta(s_j). j = 0 ∑ q − 1 ⟨ D ( z j ) , θ j − θ j + 1 ⟩ , z j = γ ( s j ) , θ j = θ ( s j ) . For every sufficiently small η \eta η 0 = s 0 < ⋯ < s q = 1 0=s_0<\cdots<s_q=1 0 = s 0 < ⋯ < s q = 1
∥ θ j − θ j + 1 ∥ ∗ ≤ η G , q ≤ C γ η , \lVert\theta_j-\theta_{j+1}\rVert_*\le\eta G,\qquad q\le\frac{C_\gamma}{\eta}, ∥ θ j − θ j + 1 ∥ ∗ ≤ η G , q ≤ η C γ , and
∑ j = 0 q − 1 ⟨ D ( z j ) , θ j − θ j + 1 ⟩ ≥ c 0 2 . (40) \sum_{j=0}^{q-1}\langle D(z_j),\theta_j-\theta_{j+1}\rangle\ge\frac{c_0}{2}. \tag*{(40)} j = 0 ∑ q − 1 ⟨ D ( z j ) , θ j − θ j + 1 ⟩ ≥ 2 c 0 . (40) Such partitions exist because θ \theta θ C 1 C^1 C 1 D ∘ γ D\circ\gamma D ∘ γ z 0 z_0 z 0
g j + 1 : = θ j − θ j + 1 η , j = 0 , … , q − 1. g_{j+1} := \frac{\theta_j-\theta_{j+1}}{\eta}, \qquad j=0,\ldots,q-1. g j + 1 := η θ j − θ j + 1 , j = 0 , … , q − 1. The dual norm of every gradient is at most G G G
y ⟼ η ⟨ g j + 1 , y ⟩ + D R ( y , z j ) . y \longmapsto\eta\langle g_{j+1},y\rangle+D_R(y,z_j). y ⟼ η ⟨ g j + 1 , y ⟩ + D R ( y , z j ) . Its gradient at z j + 1 z_{j+1} z j + 1
η g j + 1 + ∇ R ( z j + 1 ) − ∇ R ( z j ) = 0. \eta g_{j+1}+\nabla R(z_{j+1})-\nabla R(z_j)=0. η g j + 1 + ∇ R ( z j + 1 ) − ∇ R ( z j ) = 0. The objective is m m m K K K z j + 1 z_{j+1} z j + 1
By (40) , the regret during one traversal is at least
∑ j = 0 q − 1 ⟨ g j + 1 , D ( z j ) ⟩ ≥ c 0 2 η . \sum_{j=0}^{q-1}\langle g_{j+1},D(z_j)\rangle\geq\frac{c_0}{2\eta}. j = 0 ∑ q − 1 ⟨ g j + 1 , D ( z j )⟩ ≥ 2 η c 0 . Since q ≤ C γ / η q\leq C_\gamma/\eta q ≤ C γ / η c 0 / ( 2 C γ ) c_0/(2C_\gamma) c 0 / ( 2 C γ )
∑ t = 1 T ⟨ g t , D ( x t ) ⟩ ≥ c 0 2 C γ T − O ( 1 / η ) . \sum_{t=1}^{T}\langle g_t,D(x_t)\rangle\geq\frac{c_0}{2C_\gamma}T-O(1/\eta). t = 1 ∑ T ⟨ g t , D ( x t )⟩ ≥ 2 C γ c 0 T − O ( 1/ η ) . Corollary 8.12. Let U : = relint ( K ) U:=\operatorname{relint}(K) U := relint ( K ) U U U aff ( K ) \operatorname{aff}(K) aff ( K ) R ∈ C 2 ( U ) R\in C^2(U) R ∈ C 2 ( U ) ∇ 2 R ( x ) \nabla^2R(x) ∇ 2 R ( x ) x ∈ U x\in U x ∈ U x ⟼ ∇ 2 R ( x ) D ( x ) x\longmapsto\nabla^2R(x)D(x) x ⟼ ∇ 2 R ( x ) D ( x ) C 1 C^1 C 1 aff ( K ) \operatorname{aff}(K) aff ( K )
The one-form D ( x ) T d ∇ R ( x ) D(x)^{\mathsf T}\mathrm{d}\nabla R(x) D ( x ) T d ∇ R ( x ) U U U
For every coordinate pair i , j i,j i , j
∂ j [ ∇ 2 R ( x ) D ( x ) ] i = ∂ i [ ∇ 2 R ( x ) D ( x ) ] j , ( x ∈ U ) . \partial_j\left[\nabla^2R(x)D(x)\right]_i=\partial_i\left[\nabla^2R(x)D(x)\right]_j,\qquad(x\in U). ∂ j [ ∇ 2 R ( x ) D ( x ) ] i = ∂ i [ ∇ 2 R ( x ) D ( x ) ] j , ( x ∈ U ) . The mirror circulation vanishes on every closed piecewise C 1 C^1 C 1
If these conditions fail, standard mirror descent can be forced to incur linear regret along a sufficiently small loop contained in U U U D D D D ϕ D_\phi D ϕ K K K O ( T ) O(\sqrt{T}) O ( T )
Proof. In affine coordinates on U = relint ( K ) U=\operatorname{relint}(K) U = relint ( K )
β ( x ) = ∇ 2 R ( x ) D ( x ) . \beta(x)=\nabla^2R(x)D(x). β ( x ) = ∇ 2 R ( x ) D ( x ) . The Poincaré lemma gives the equivalence between exactness of β ( x ) T d x \beta(x)^{\mathsf T}\mathrm{d}x β ( x ) T d x β \beta β U U U
Suppose instead that the form is exact. Strong convexity makes ∇ R \nabla R ∇ R U U U ∇ 2 R \nabla^{2}R ∇ 2 R ∇ R : U → ∇ R ( U ) \nabla R: U \to\nabla R(U) ∇ R : U → ∇ R ( U ) V V V
Ψ ( θ ) : = V ( ( ∇ R ) − 1 ( θ ) ) . \Psi(\theta) := V\left((\nabla R)^{-1}(\theta)\right). Ψ ( θ ) := V ( ( ∇ R ) − 1 ( θ ) ) . The chain rule gives
∇ θ Ψ ( ∇ R ( x ) ) = D ( x ) . \nabla_{\theta}\Psi(\nabla R(x)) = D(x). ∇ θ Ψ ( ∇ R ( x )) = D ( x ) . The extension, feasibility, and regularity assumptions in the corollary then allow Theorem 8.9 to be applied on K K K
Proofs for Section 8.5
Proposition 8.14. Assume X = K \mathcal{X}=K X = K F F F Definition 8.13 , the function Ψ F ( θ ) : = R K ∗ ( θ ) − ( F + R ) ∗ ( θ ) \Psi_F(\theta):=R_K^{*}(\theta)-(F+R)^{*}(\theta) Ψ F ( θ ) := R K ∗ ( θ ) − ( F + R ) ∗ ( θ )
x − prox F R ( x ) = ∇ θ Ψ F ( ∇ R ( x ) ) ( x ∈ K ) . x-\operatorname{prox}_{F}^{R}(x)=\nabla_{\theta}\Psi_F(\nabla R(x)) \qquad(x\in K). x − prox F R ( x ) = ∇ θ Ψ F ( ∇ R ( x )) ( x ∈ K ) . Consequently, Φ p r o x R ( K ) ⊆ Φ r a d R ( K ) ⊆ Φ e x a c t R ( K ) \Phi_{\mathrm{prox}}^{R}(K)\subseteq\Phi_{\mathrm{rad}}^{R}(K)\subseteq\Phi_{\mathrm{exact}}^{R}(K) Φ prox R ( K ) ⊆ Φ rad R ( K ) ⊆ Φ exact R ( K ) Φ e x a c t R ( K ) \Phi_{\mathrm{exact}}^{R}(K) Φ exact R ( K ) ϕ α = prox F R \phi_{\alpha}=\operatorname{prox}_{F}^{R} ϕ α = prox F R ϕ \phi ϕ Ψ F / α \Psi_F/\alpha Ψ F / α
Proof. Since R K R_K R K m m m F + R F+R F + R ( m − ρ ) (m-\rho) ( m − ρ ) x ∈ K x\in K x ∈ K
∇ R ( x ) ∈ ∂ R K ( x ) , \nabla R(x)\in\partial R_K(x), ∇ R ( x ) ∈ ∂ R K ( x ) , because 0 ∈ N K ( x ) 0\in N_K(x) 0 ∈ N K ( x )
∇ R K ∗ ( ∇ R ( x ) ) = x . \nabla R_K^{*}(\nabla R(x))=x. ∇ R K ∗ ( ∇ R ( x )) = x . If p = prox F R ( x ) p=\operatorname{prox}_{F}^{R}(x) p = prox F R ( x )
∇ R ( x ) ∈ ∂ ( F + R ) ( p ) , \nabla R(x)\in\partial(F+R)(p), ∇ R ( x ) ∈ ∂ ( F + R ) ( p ) , and hence
∇ ( F + R ) ∗ ( ∇ R ( x ) ) = p . \nabla(F+R)^{*}(\nabla R(x))=p. ∇ ( F + R ) ∗ ( ∇ R ( x )) = p . Subtracting the two identities yields
∇ θ Ψ F ( ∇ R ( x ) ) = x − prox F R ( x ) , \nabla_{\theta}\Psi_F(\nabla R(x))=x-\operatorname{prox}_{F}^{R}(x), ∇ θ Ψ F ( ∇ R ( x )) = x − prox F R ( x ) , so every map in Φ p r o x R ( K ) \Phi_{\mathrm{prox}}^{R}(K) Φ prox R ( K ) Φ p r o x R ( K ) ⊆ Φ r a d R ( K ) \Phi_{\mathrm{prox}}^{R}(K)\subseteq\Phi_{\mathrm{rad}}^{R}(K) Φ prox R ( K ) ⊆ Φ rad R ( K ) α = 1 \alpha=1 α = 1 ϕ α = prox F R \phi_{\alpha}=\operatorname{prox}_{F}^{R} ϕ α = prox F R
x − ϕ ( x ) = 1 α ( x − ϕ α ( x ) ) = ∇ θ ( Ψ F α ) ( ∇ R ( x ) ) . x-\phi(x)=\frac{1}{\alpha}\left(x-\phi_{\alpha}(x)\right)=\nabla_{\theta}\left(\frac{\Psi_F}{\alpha}\right)(\nabla R(x)). x − ϕ ( x ) = α 1 ( x − ϕ α ( x ) ) = ∇ θ ( α Ψ F ) ( ∇ R ( x )) . Thus ϕ \phi ϕ Ψ F / α \Psi_F/\alpha Ψ F / α Φ r a d R ( K ) ⊆ Φ e x a c t R ( K ) \Phi_{\mathrm{rad}}^{R}(K)\subseteq\Phi_{\mathrm{exact}}^{R}(K) Φ rad R ( K ) ⊆ Φ exact R ( K )
Proposition 8.15. Suppose ϕ α = prox F α R \phi_\alpha=\operatorname{prox}^{R}_{F_\alpha} ϕ α = prox F α R D D D F α F_\alpha F α u α u_\alpha u α F α + R F_\alpha+R F α + R c α ∈ R c_\alpha\in\mathbb{R} c α ∈ R u α = ( F α + R ) ∗ + c α u_\alpha=(F_\alpha+R)^*+c_\alpha u α = ( F α + R ) ∗ + c α Θ \Theta Θ u α u_\alpha u α
D Ψ ( θ ′ , θ ) ≤ 1 α D R ∗ ( θ ′ , θ ) ( θ , θ ′ ∈ Θ ) . D_{\Psi}(\theta',\theta)\leq\frac{1}{\alpha}D_{R^*}(\theta',\theta)\qquad(\theta,\theta'\in\Theta). D Ψ ( θ ′ , θ ) ≤ α 1 D R ∗ ( θ ′ , θ ) ( θ , θ ′ ∈ Θ ) . Proof. Let H α : = F α + R H_\alpha:=F_\alpha+R H α := F α + R F α F_\alpha F α ρ α \rho_\alpha ρ α ρ α < m \rho_\alpha<m ρ α < m H α H_\alpha H α H α ∗ H_\alpha^* H α ∗ x = ∇ R ∗ ( θ ) x=\nabla R^*(\theta) x = ∇ R ∗ ( θ )
min y { H α ( y ) − ⟨ θ , y ⟩ } . \min_y\{H_\alpha(y)-\langle\theta,y\rangle\}. y min { H α ( y ) − ⟨ θ , y ⟩} . Hence
ϕ α ( ∇ R ∗ ( θ ) ) = ∇ H α ∗ ( θ ) . \phi_\alpha(\nabla R^*(\theta))=\nabla H_\alpha^*(\theta). ϕ α ( ∇ R ∗ ( θ )) = ∇ H α ∗ ( θ ) . Mirror exactness gives the independent identity
ϕ α ( ∇ R ∗ ( θ ) ) = ∇ ( R ∗ − α Ψ ) ( θ ) = ∇ u α ( θ ) . \phi_\alpha(\nabla R^*(\theta))=\nabla(R^*-\alpha\Psi)(\theta)=\nabla u_\alpha(\theta). ϕ α ( ∇ R ∗ ( θ )) = ∇ ( R ∗ − α Ψ ) ( θ ) = ∇ u α ( θ ) . The open convex set Θ \Theta Θ
u α = H α ∗ + c α u_\alpha=H_\alpha^*+c_\alpha u α = H α ∗ + c α on Θ \Theta Θ
D u α ( θ ′ , θ ) = D R ∗ ( θ ′ , θ ) − α D Ψ ( θ ′ , θ ) ≥ 0 , D_{u_\alpha}(\theta',\theta)=D_{R^*}(\theta',\theta)-\alpha D_{\Psi}(\theta',\theta)\geq0, D u α ( θ ′ , θ ) = D R ∗ ( θ ′ , θ ) − α D Ψ ( θ ′ , θ ) ≥ 0 , which is the stated relative-curvature inequality.
Proposition 8.16. Suppose u α u_\alpha u α Θ \Theta Θ
u ‾ α : = cl ( u α + ι Θ ) . \overline{u}_\alpha:=\operatorname{cl}(u_\alpha+\iota_\Theta). u α := cl ( u α + ι Θ ) . Assume u ‾ α \overline{u}_\alpha u α ϕ α ( D ) ⊆ dom R \phi_\alpha(D)\subseteq\operatorname{dom}R ϕ α ( D ) ⊆ dom R
F α ( y ) : = { u ‾ α ∗ ( y ) − R ( y ) , y ∈ dom R , + ∞ , y ∉ dom R . F_\alpha(y):=
\begin{cases}
\overline{u}_\alpha^*(y)-R(y), & y\in\operatorname{dom}R,\\
+\infty, & y\notin\operatorname{dom}R.
\end{cases} F α ( y ) := { u α ∗ ( y ) − R ( y ) , + ∞ , y ∈ dom R , y ∈ / dom R . Then, for every x ∈ D x\in D x ∈ D
ϕ α ( x ) ∈ arg min y { F α ( y ) + D R ( y , x ) } . \phi_\alpha(x)\in\underset{y}{\operatorname{arg\,min}}\{F_\alpha(y)+D_R(y,x)\}. ϕ α ( x ) ∈ y arg min { F α ( y ) + D R ( y , x )} . Under these domain assumptions, convexity of u α u_\alpha u α ϕ α \phi_\alpha ϕ α F α F_\alpha F α ρ α \rho_\alpha ρ α ρ α < m \rho_\alpha<m ρ α < m ϕ α = prox F α R \phi_\alpha=\operatorname{prox}^{R}_{F_\alpha} ϕ α = prox F α R ϕ ∈ Φ r a d R ( D ) \phi\in\Phi_{\mathrm{rad}}^{R}(D) ϕ ∈ Φ rad R ( D )
Proof. A finite differentiable convex function on the open set Θ \Theta Θ
u ‾ α = cl ( u α + ι Θ ) \overline{u}_\alpha=\operatorname{cl}(u_\alpha+\iota_\Theta) u α = cl ( u α + ι Θ ) agrees with u α u_\alpha u α Θ \Theta Θ θ ∈ Θ \theta\in\Theta θ ∈ Θ
z : = ∇ u α ( θ ) = ϕ α ( ∇ R ∗ ( θ ) ) . z:=\nabla u_\alpha(\theta)=\phi_\alpha(\nabla R^*(\theta)). z := ∇ u α ( θ ) = ϕ α ( ∇ R ∗ ( θ )) . Convexity gives z ∈ ∂ u ‾ α ( θ ) z \in\partial\overline{u}_{\alpha}(\theta) z ∈ ∂ u α ( θ )
θ ∈ ∂ u ‾ α ∗ ( z ) . \theta\in\partial\overline{u}_{\alpha}^{*}(z). θ ∈ ∂ u α ∗ ( z ) . The endpoint-domain hypothesis gives z ∈ dom R z \in\operatorname{dom} R z ∈ dom R u ‾ α ∗ ( z ) < + ∞ \overline{u}_{\alpha}^{*}(z) < +\infty u α ∗ ( z ) < + ∞ F α + R F_{\alpha}+R F α + R z z z
y ⟼ u ‾ α ∗ ( y ) − ⟨ θ , y ⟩ . y \longmapsto\overline{u}_{\alpha}^{*}(y)-\langle\theta,y\rangle. y ⟼ u α ∗ ( y ) − ⟨ θ , y ⟩ . For x = ∇ R ∗ ( θ ) x=\nabla R^{*}(\theta) x = ∇ R ∗ ( θ ) y ∈ dom R y \in\operatorname{dom} R y ∈ dom R
F α ( y ) + D R ( y , x ) = u ‾ α ∗ ( y ) − R ( y ) + R ( y ) − R ( x ) − ⟨ θ , y − x ⟩ = u ‾ α ∗ ( y ) − ⟨ θ , y ⟩ + ⟨ θ , x ⟩ − R ( x ) . \begin{aligned}
F_{\alpha}(y)+D_{R}(y,x)
&=\overline{u}_{\alpha}^{*}(y)-R(y)+R(y)-R(x)-\langle\theta,y-x\rangle\\
&=\overline{u}_{\alpha}^{*}(y)-\langle\theta,y\rangle+\langle\theta,x\rangle-R(x).
\end{aligned} F α ( y ) + D R ( y , x ) = u α ∗ ( y ) − R ( y ) + R ( y ) − R ( x ) − ⟨ θ , y − x ⟩ = u α ∗ ( y ) − ⟨ θ , y ⟩ + ⟨ θ , x ⟩ − R ( x ) . The last two terms do not depend on y y y z ∈ dom R z \in\operatorname{dom} R z ∈ dom R dom R \operatorname{dom} R dom R z = ϕ α ( x ) z=\phi_{\alpha}(x) z = ϕ α ( x ) F α F_{\alpha} F α ρ α \rho_{\alpha} ρ α ρ α < m \rho_{\alpha}<m ρ α < m F α + R F_{\alpha}+R F α + R prox F α R ( x ) \operatorname{prox}_{F_{\alpha}}^{R}(x) prox F α R ( x )
Corollary 8.17. For 0 < α ≤ 1 0<\alpha\leq1 0 < α ≤ 1 u α = R ∗ − α Ψ u_{\alpha}=R^{*}-\alpha\Psi u α = R ∗ − α Ψ α L R ( Ψ ) ≤ 1 \alpha L_{R}(\Psi)\leq1 α L R ( Ψ ) ≤ 1
α c v x : = min { 1 , 1 L R ( Ψ ) } , \alpha_{\mathrm{cvx}}:=\min\left\{1,\frac{1}{L_{R}(\Psi)}\right\}, α cvx := min { 1 , L R ( Ψ ) 1 } , with the conventions 1 / 0 : = + ∞ 1/0:=+\infty 1/0 := + ∞ 1 / ( + ∞ ) : = 0 1/(+\infty):=0 1/ ( + ∞ ) := 0 0 < α ≤ α c v x 0<\alpha\leq\alpha_{\mathrm{cvx}} 0 < α ≤ α cvx α \alpha α u α u_{\alpha} u α ϕ α ( D ) ⊆ dom R \phi_{\alpha}(\mathcal{D})\subseteq\operatorname{dom} R ϕ α ( D ) ⊆ dom R F α F_{\alpha} F α
Proof. For every ordered pair in Θ \Theta Θ
D R ∗ − α Ψ ( θ ′ , θ ) = D R ∗ ( θ ′ , θ ) − α D Ψ ( θ ′ , θ ) . D_{R^{*}-\alpha\Psi}(\theta',\theta)=D_{R^{*}}(\theta',\theta)-\alpha D_{\Psi}(\theta',\theta). D R ∗ − α Ψ ( θ ′ , θ ) = D R ∗ ( θ ′ , θ ) − α D Ψ ( θ ′ , θ ) . A differentiable function on an open convex set is convex if and only if its Bregman divergence is nonnegative on every ordered pair. Hence u α u_{\alpha} u α
D Ψ ( θ ′ , θ ) ≤ 1 α D R ∗ ( θ ′ , θ ) for all θ , θ ′ ∈ Θ . D_{\Psi}(\theta',\theta)\leq\frac{1}{\alpha}D_{R^{*}}(\theta',\theta)\qquad\text{for all }\theta,\theta'\in\Theta. D Ψ ( θ ′ , θ ) ≤ α 1 D R ∗ ( θ ′ , θ ) for all θ , θ ′ ∈ Θ. If L R ( Ψ ) < + ∞ L_{R}(\Psi)<+\infty L R ( Ψ ) < + ∞ L R ( Ψ ) = + ∞ L_{R}(\Psi)=+\infty L R ( Ψ ) = + ∞ α ∈ ( 0 , 1 ] \alpha\in(0,1] α ∈ ( 0 , 1 ] min { 1 , 1 / L R ( Ψ ) } \min\{1,1/L_{R}(\Psi)\} min { 1 , 1/ L R ( Ψ )} u α u_{\alpha} u α ϕ α ( D ) ⊆ dom R \phi_{\alpha}(\mathcal{D})\subseteq\operatorname{dom} R ϕ α ( D ) ⊆ dom R
Proposition 8.18. Let 1 < p < 2 1<p<2 1 < p < 2 q : = p / ( p − 1 ) > 2 q:=p/(p-1)>2 q := p / ( p − 1 ) > 2
R ( x ) : = 1 2 ∥ x ∥ p 2 ( x ∈ R 2 ) . R(x):=\frac{1}{2}\lVert x\rVert_{p}^{2}\qquad(x\in\mathbb{R}^{2}). R ( x ) := 2 1 ∥ x ∥ p 2 ( x ∈ R 2 ) . There is a feasible R R R C ∞ C^{\infty} C ∞ Φ r a d R ( R 2 ) \Phi_{\mathrm{rad}}^{R}(\mathbb{R}^{2}) Φ rad R ( R 2 )
Proof. Let q : = p / ( p − 1 ) > 2 q := p/(p-1) > 2 q := p / ( p − 1 ) > 2
R ∗ ( θ ) = 1 2 ∥ θ ∥ q 2 . R^*(\theta) = \frac{1}{2}\lVert\theta\rVert_q^2. R ∗ ( θ ) = 2 1 ∥ θ ∥ q 2 . Set
Ψ ( θ ) : = 1 2 θ [ 2 ] 2 , ϕ ( x ) : = x − ∇ θ Ψ ( ∇ R ( x ) ) . \Psi(\theta) := \frac{1}{2}\theta[2]^2,\qquad\phi(x) := x - \nabla_\theta\Psi(\nabla R(x)). Ψ ( θ ) := 2 1 θ [ 2 ] 2 , ϕ ( x ) := x − ∇ θ Ψ ( ∇ R ( x )) . The map is feasible because the action space is R 2 \mathbb{R}^2 R 2 ∇ Ψ \nabla\Psi ∇Ψ
Fix α > 0 \alpha> 0 α > 0 u α = R ∗ − α Ψ u_\alpha= R^* - \alpha\Psi u α = R ∗ − α Ψ θ = e 1 + t e 2 \theta= e_1 + te_2 θ = e 1 + t e 2
u α ( e 1 + t e 2 ) = 1 2 ( 1 + ∣ t ∣ q ) 2 / q − α 2 t 2 . u_\alpha(e_1 + te_2) = \frac{1}{2}(1+|t|^q)^{2/q} - \frac{\alpha}{2}t^2. u α ( e 1 + t e 2 ) = 2 1 ( 1 + ∣ t ∣ q ) 2/ q − 2 α t 2 . The expansion ( 1 + s ) 2 / q = 1 + ( 2 / q ) s + O ( s 2 ) (1+s)^{2/q} = 1 + (2/q)s + O(s^2) ( 1 + s ) 2/ q = 1 + ( 2/ q ) s + O ( s 2 )
u α ( e 1 + t e 2 ) − u α ( e 1 ) = 1 q ∣ t ∣ q − α 2 t 2 + O ( ∣ t ∣ 2 q ) . u_\alpha(e_1 + te_2) - u_\alpha(e_1) = \frac{1}{q}|t|^q - \frac{\alpha}{2}t^2 + O(|t|^{2q}). u α ( e 1 + t e 2 ) − u α ( e 1 ) = q 1 ∣ t ∣ q − 2 α t 2 + O ( ∣ t ∣ 2 q ) . Since q > 2 q > 2 q > 2 t t t
u α ( e 1 ) ≤ 1 2 u α ( e 1 + t e 2 ) + 1 2 u α ( e 1 − t e 2 ) = u α ( e 1 + t e 2 ) , u_\alpha(e_1) \le\frac{1}{2}u_\alpha(e_1 + te_2) + \frac{1}{2}u_\alpha(e_1 - te_2) = u_\alpha(e_1 + te_2), u α ( e 1 ) ≤ 2 1 u α ( e 1 + t e 2 ) + 2 1 u α ( e 1 − t e 2 ) = u α ( e 1 + t e 2 ) , a contradiction. Thus u α u_\alpha u α α > 0 \alpha> 0 α > 0
Proofs for Section 9.1
Proposition 9.1. Under the standing Legendre assumptions of § 8 , let θ 1 : = ∇ R ( x 1 ) \theta_1 := \nabla R(x_1) θ 1 := ∇ R ( x 1 ) η \eta η
x t = arg min x ∈ E { η ∑ s = 1 t − 1 ⟨ g s , x ⟩ + R ( x ) − ⟨ θ 1 , x ⟩ } = arg min x ∈ K { η ∑ s = 1 t − 1 ⟨ g s , x ⟩ + D R ( x , x 1 ) } . x_t = \underset{x\in E}{\arg\min}\left\{\eta\sum_{s=1}^{t-1}\langle g_s,x\rangle+ R(x) - \langle\theta_1,x\rangle\right\}
= \underset{x\in K}{\arg\min}\left\{\eta\sum_{s=1}^{t-1}\langle g_s,x\rangle+ D_R(x,x_1)\right\}. x t = x ∈ E arg min { η s = 1 ∑ t − 1 ⟨ g s , x ⟩ + R ( x ) − ⟨ θ 1 , x ⟩ } = x ∈ K arg min { η s = 1 ∑ t − 1 ⟨ g s , x ⟩ + D R ( x , x 1 ) } . Thus they are the FTRL iterates for the linearized losses and the tilted regularizer R − ⟨ θ 1 , ⋅ ⟩ R-\langle\theta_1,\cdot\rangle R − ⟨ θ 1 , ⋅ ⟩
Proof. The interior optimality condition for mirror descent is
∇ R ( x t + 1 ) = ∇ R ( x t ) − η g t . \nabla R(x_{t+1}) = \nabla R(x_t) - \eta g_t. ∇ R ( x t + 1 ) = ∇ R ( x t ) − η g t . Writing θ t : = ∇ R ( x t ) \theta_t := \nabla R(x_t) θ t := ∇ R ( x t ) θ t = θ 1 − η ∑ s < t g s \theta_t = \theta_1 - \eta\sum_{s<t}g_s θ t = θ 1 − η ∑ s < t g s
x t = ∇ R ∗ ( θ t ) = arg min x ∈ E { R ( x ) − ⟨ θ t , x ⟩ } . x_t = \nabla R^*(\theta_t) = \underset{x\in E}{\arg\min}\{R(x) - \langle\theta_t,x\rangle\}. x t = ∇ R ∗ ( θ t ) = x ∈ E arg min { R ( x ) − ⟨ θ t , x ⟩} . This is the first FTRL formula. The second follows because R ( x ) − ⟨ θ 1 , x ⟩ R(x) - \langle\theta_1,x\rangle R ( x ) − ⟨ θ 1 , x ⟩ D R ( x , x 1 ) D_R(x,x_1) D R ( x , x 1 ) x x x
Proofs for Section 9.2
Theorem 9.3. Assume that R R R ( σ , r ) (\sigma,r) ( σ , r ) R ∗ R^* R ∗ E E E (12) . If ϕ \phi ϕ R R R Ψ \Psi Ψ
D Ψ ( θ t + 1 , θ t ) ≤ L D R ∗ ( θ t + 1 , θ t ) ( t ∈ [ T ] ) , D_{\Psi}(\theta_{t+1},\theta_t) \le L D_{R^*}(\theta_{t+1},\theta_t) \qquad(t \in[T]), D Ψ ( θ t + 1 , θ t ) ≤ L D R ∗ ( θ t + 1 , θ t ) ( t ∈ [ T ]) , then
∑ t = 1 T ⟨ g t , x t − ϕ ( x t ) ⟩ ≤ Ψ ( θ 1 ) − Ψ ( θ T + 1 ) η + L η q − 1 q σ q − 1 ∑ t = 1 T ∥ g t ∥ ∗ q . \sum_{t=1}^{T} \langle g_t,x_t-\phi(x_t)\rangle\le\frac{\Psi(\theta_1)-\Psi(\theta_{T+1})}{\eta}+\frac{L\eta^{q-1}}{q\sigma^{q-1}}\sum_{t=1}^{T}\lVert g_t\rVert_*^q. t = 1 ∑ T ⟨ g t , x t − ϕ ( x t )⟩ ≤ η Ψ ( θ 1 ) − Ψ ( θ T + 1 ) + q σ q − 1 L η q − 1 t = 1 ∑ T ∥ g t ∥ ∗ q . Proof. Uniform convexity makes ∂ R ∗ ( θ ) \partial R^*(\theta) ∂ R ∗ ( θ ) R ∗ R^* R ∗ E E E E E E
FTRL-exactness and the additive dual update give
η ⟨ g t , x t − ϕ ( x t ) ⟩ = ⟨ θ t − θ t + 1 , ∇ θ Ψ ( θ t ) ⟩ = Ψ ( θ t ) − Ψ ( θ t + 1 ) + D Ψ ( θ t + 1 , θ t ) ≤ Ψ ( θ t ) − Ψ ( θ t + 1 ) + L D R ∗ ( θ t + 1 , θ t ) . (41) \eta\langle g_t,x_t-\phi(x_t)\rangle=\langle\theta_t-\theta_{t+1},\nabla_\theta\Psi(\theta_t)\rangle
=\Psi(\theta_t)-\Psi(\theta_{t+1})+D_{\Psi}(\theta_{t+1},\theta_t)
\le\Psi(\theta_t)-\Psi(\theta_{t+1})+L D_{R^*}(\theta_{t+1},\theta_t). \tag*{(41)} η ⟨ g t , x t − ϕ ( x t )⟩ = ⟨ θ t − θ t + 1 , ∇ θ Ψ ( θ t )⟩ = Ψ ( θ t ) − Ψ ( θ t + 1 ) + D Ψ ( θ t + 1 , θ t ) ≤ Ψ ( θ t ) − Ψ ( θ t + 1 ) + L D R ∗ ( θ t + 1 , θ t ) . (41) Fix θ ∈ E \theta\in E θ ∈ E x : = ∇ R ∗ ( θ ) x := \nabla R^*(\theta) x := ∇ R ∗ ( θ ) d ∈ E d \in E d ∈ E y ∈ dom R y \in\operatorname{dom} R y ∈ dom R
R ( y ) ≥ R ( x ) + ⟨ θ , y − x ⟩ + σ r ∥ y − x ∥ r . R(y) \ge R(x)+\langle\theta,y-x\rangle+\frac{\sigma}{r}\lVert y-x\rVert^r. R ( y ) ≥ R ( x ) + ⟨ θ , y − x ⟩ + r σ ∥ y − x ∥ r . Hence
R ∗ ( θ + d ) ≤ R ∗ ( θ ) + ⟨ d , x ⟩ + sup z ∈ E { ⟨ d , z ⟩ − σ r ∥ z ∥ r } = R ∗ ( θ ) + ⟨ d , ∇ R ∗ ( θ ) ⟩ + 1 q σ q − 1 ∥ d ∥ ∗ q . \begin{aligned}
R^*(\theta+d)\le R^*(\theta)+\langle d,x\rangle+\sup_{z\in E}\left\{\langle d,z\rangle-\frac{\sigma}{r}\lVert z\rVert^r\right\} \\
&=R^*(\theta)+\langle d,\nabla R^*(\theta)\rangle+\frac{1}{q\sigma^{q-1}}\lVert d\rVert_*^q.
\end{aligned} R ∗ ( θ + d ) ≤ R ∗ ( θ ) + ⟨ d , x ⟩ + z ∈ E sup { ⟨ d , z ⟩ − r σ ∥ z ∥ r } = R ∗ ( θ ) + ⟨ d , ∇ R ∗ ( θ )⟩ + q σ q − 1 1 ∥ d ∥ ∗ q . Thus
D R ∗ ( θ + d , θ ) ≤ 1 q σ q − 1 ∥ d ∥ ∗ q . D_{R^*}(\theta+d,\theta)\le\frac{1}{q\sigma^{q-1}}\lVert d\rVert_*^q. D R ∗ ( θ + d , θ ) ≤ q σ q − 1 1 ∥ d ∥ ∗ q . Apply this with d : = θ t + 1 − θ t = − η g t d:=\theta_{t+1}-\theta_t=-\eta g_t d := θ t + 1 − θ t = − η g t (41) , divide by η \eta η □ \square □
Corollary 9.4. Suppose ∥ g t ∥ ∗ ≤ G \lVert g_t\rVert_* \le G ∥ g t ∥ ∗ ≤ G Ψ \Psi Ψ B B B B , L , G > 0 B,L,G>0 B , L , G > 0 η = ( r B σ q − 1 L G q T ) 1 / q \eta=\left(\frac{rB\sigma^{q-1}}{LG^qT}\right)^{1/q} η = ( L G q T r B σ q − 1 ) 1/ q
∑ t = 1 T ⟨ g t , x t − ϕ ( x t ) ⟩ ≤ r 1 / r G L 1 / q B 1 / r σ 1 / r T 1 / q . \sum_{t=1}^{T}\langle g_t,x_t-\phi(x_t)\rangle\le r^{1/r}\frac{GL^{1/q}B^{1/r}}{\sigma^{1/r}}T^{1/q}. t = 1 ∑ T ⟨ g t , x t − ϕ ( x t )⟩ ≤ r 1/ r σ 1/ r G L 1/ q B 1/ r T 1/ q . For r = 2 r=2 r = 2 G 2 L B T / σ G\sqrt{2LBT/\sigma} G 2 L B T / σ
Proof. The theorem gives
B η + L G q T q σ q − 1 η q − 1 . \frac{B}{\eta}+\frac{LG^qT}{q\sigma^{q-1}}\eta^{q-1}. η B + q σ q − 1 L G q T η q − 1 . Differentiation yields the displayed optimizer and bound. □ \square □
Theorem 9.5. Suppose D ~ ϕ \widetilde{D}_{\phi} D ϕ C 1 C^{1} C 1 γ ⊂ Θ \gamma\subset\Theta γ ⊂ Θ
∮ γ ⟨ D ~ ϕ ( θ ) , d θ ⟩ ≠ 0. \oint_{\gamma} \langle\widetilde{D}_{\phi}(\theta),\mathrm{d}\theta\rangle\ne0. ∮ γ ⟨ D ϕ ( θ ) , d θ ⟩ = 0. For every G > 0 G>0 G > 0 η > 0 \eta>0 η > 0 ∥ g t ∥ ∗ ≤ G \|g_t\|_* \le G ∥ g t ∥ ∗ ≤ G (12) to incur c T − O ( 1 / η ) cT-O(1/\eta) c T − O ( 1/ η ) ϕ \phi ϕ c > 0 c>0 c > 0 Θ \Theta Θ
Proof. Reverse the orientation if necessary so that
− ∮ γ ⟨ D ~ ϕ ( θ ) , d θ ⟩ = c 0 > 0. -\oint_{\gamma} \langle\widetilde{D}_{\phi}(\theta),\mathrm{d}\theta\rangle= c_0>0. − ∮ γ ⟨ D ϕ ( θ ) , d θ ⟩ = c 0 > 0. Because Θ \Theta Θ Θ \Theta Θ η G \eta G η G O ( 1 / η ) O(1/\eta) O ( 1/ η ) O ( 1 / η ) O(1/\eta) O ( 1/ η )
Let L γ L_{\gamma} L γ η \eta η θ 0 , … , θ s = θ 0 \theta_0,\ldots,\theta_s=\theta_0 θ 0 , … , θ s = θ 0
∥ θ j − θ j + 1 ∥ ∗ ≤ η G , s ≤ 2 L γ η G , \|\theta_j-\theta_{j+1}\|_* \le\eta G,\qquad s\le\frac{2L_{\gamma}}{\eta G}, ∥ θ j − θ j + 1 ∥ ∗ ≤ η G , s ≤ η G 2 L γ , and whose left Riemann sum is at least c 0 / 2 c_0/2 c 0 /2
g j + 1 : = θ j − θ j + 1 η . g_{j+1}:=\frac{\theta_j-\theta_{j+1}}{\eta}. g j + 1 := η θ j − θ j + 1 . The FTRL dual state follows the polygon exactly, and one traversal earns at least c 0 / ( 2 η ) c_0/(2\eta) c 0 / ( 2 η ) c 0 G / ( 4 L γ ) c_0G/(4L_{\gamma}) c 0 G / ( 4 L γ )
∑ t = 1 T ⟨ g t , x t − ϕ ( x t ) ⟩ ≥ c 0 G 4 L γ T − O ( 1 / η ) . \sum_{t=1}^{T}\langle g_t,x_t-\phi(x_t)\rangle\ge\frac{c_0G}{4L_{\gamma}}T-O(1/\eta). t = 1 ∑ T ⟨ g t , x t − ϕ ( x t )⟩ ≥ 4 L γ c 0 G T − O ( 1/ η ) . :::
Corollary 9.6. Assume that D ~ ϕ \widetilde{D}_{\phi} D ϕ C 1 C^{1} C 1 Θ \Theta Θ R R R
Proof. The equivalence between a symmetric Jacobian and a potential follows from the Poincaré lemma on the open convex set Θ \Theta Θ O ( T 1 / q ) O(T^{1/q}) O ( T 1/ q ) q > 1 q>1 q > 1
Proofs for Section 9.3
Proposition 9.7. Let K K K R R R K K K ϕ : K → K \phi:K\to K ϕ : K → K R K R_K R K R R R
Φ F T R L R K ( K ) ⊆ Φ e x a c t R ( K ) . \Phi_{\mathrm{FTRL}}^{R_K}(K)\subseteq\Phi_{\mathrm{exact}}^{R}(K). Φ FTRL R K ( K ) ⊆ Φ exact R ( K ) . Proof. Strong convexity makes R K ∗ R_K^* R K ∗ x ∈ K x \in K x ∈ K
∇ R ( x ) ∈ ∇ R ( x ) + N K ( x ) = ∂ R K ( x ) , \nabla R(x) \in\nabla R(x) + N_K(x) = \partial R_K(x), ∇ R ( x ) ∈ ∇ R ( x ) + N K ( x ) = ∂ R K ( x ) , because 0 ∈ N K ( x ) 0 \in N_K(x) 0 ∈ N K ( x )
∇ R K ∗ ( ∇ R ( x ) ) = x . \nabla R_K^*(\nabla R(x)) = x. ∇ R K ∗ ( ∇ R ( x )) = x . If ϕ \phi ϕ R K R_K R K Ψ \Psi Ψ θ = ∇ R ( x ) \theta= \nabla R(x) θ = ∇ R ( x ) Definition 9.2 to obtain
x − ϕ ( x ) = ∇ θ Ψ ( ∇ R ( x ) ) . x - \phi(x) = \nabla_\theta\Psi(\nabla R(x)). x − ϕ ( x ) = ∇ θ Ψ ( ∇ R ( x )) . This is Definition 8.8.
Theorem 9.8. The map ϕ p , ε \phi_{p,\varepsilon} ϕ p , ε R p R_p R p O ( T ) O(\sqrt{T}) O ( T ) T − 1 / 2 T^{-1/2} T − 1/2 R p , K R_{p,K} R p , K c T − O ( 1 / η ) cT - O(1/\eta) c T − O ( 1/ η ) Φ F T R L R p , K ( K ) ⊊ Φ e x a c t R p ( K ) \Phi_{\mathrm{FTRL}}^{R_{p,K}}(K) \subsetneq\Phi_{\mathrm{exact}}^{R_p}(K) Φ FTRL R p , K ( K ) ⊊ Φ exact R p ( K )
Proof. Let c : = ( a + b ) / 2 c := (a+b)/2 c := ( a + b ) /2 d : = ( b − a ) / 2 d := (b-a)/2 d := ( b − a ) /2 β ∈ ( 0 , 1 ) \beta\in(0,1) β ∈ ( 0 , 1 ) (14) . The Hessian is
∇ 2 R p ( x ) = ( p − 1 ) diag ( x [ 1 ] p − 2 , x [ 2 ] p − 2 ) , \nabla^2 R_p(x) = (p-1)\operatorname{diag}\left(x[1]^{p-2},x[2]^{p-2}\right), ∇ 2 R p ( x ) = ( p − 1 ) diag ( x [ 1 ] p − 2 , x [ 2 ] p − 2 ) , and every diagonal entry is at least m p : = ( p − 1 ) min s ∈ [ a , b ] s p − 2 > 0 m_p := (p-1)\min_{s\in[a,b]}s^{p-2} > 0 m p := ( p − 1 ) min s ∈ [ a , b ] s p − 2 > 0 z : = x − c 1 z := x-c\mathbf{1} z := x − c 1 i i i j j j
D p , ε ( x ) [ i ] = ε ( p − 1 ) x [ i ] p − 2 ( z [ i ] + β z [ j ] ) , D_{p,\varepsilon}(x)[i] = \frac{\varepsilon}{(p-1)x[i]^{p-2}}\left(z[i]+\beta z[j]\right), D p , ε ( x ) [ i ] = ( p − 1 ) x [ i ] p − 2 ε ( z [ i ] + β z [ j ] ) , ∣ D p , ε ( x ) [ i ] ∣ ≤ ε ( 1 + β ) d m p . \left|D_{p,\varepsilon}(x)[i]\right| \le\frac{\varepsilon(1+\beta)d}{m_p}. ∣ D p , ε ( x ) [ i ] ∣ ≤ m p ε ( 1 + β ) d . If D p , ε ( x ) [ i ] > 0 D_{p,\varepsilon}(x)[i] > 0 D p , ε ( x ) [ i ] > 0 z [ i ] > − β z [ j ] ≥ − β d z[i] > -\beta z[j] \ge-\beta d z [ i ] > − β z [ j ] ≥ − β d x [ i ] − a > ( 1 − β ) d x[i]-a > (1-\beta)d x [ i ] − a > ( 1 − β ) d ε \varepsilon ε D p , ε ( x ) [ i ] ≤ x [ i ] − a D_{p,\varepsilon}(x)[i] \le x[i]-a D p , ε ( x ) [ i ] ≤ x [ i ] − a ϕ p , ε ( x ) [ i ] ≥ a \phi_{p,\varepsilon}(x)[i] \ge a ϕ p , ε ( x ) [ i ] ≥ a D p , ε ( x ) [ i ] < 0 D_{p,\varepsilon}(x)[i] < 0 D p , ε ( x ) [ i ] < 0 b − x [ i ] > ( 1 − β ) d b-x[i] > (1-\beta)d b − x [ i ] > ( 1 − β ) d ∣ D p , ε ( x ) [ i ] ∣ ≤ b − x [ i ] \left|D_{p,\varepsilon}(x)[i]\right| \le b-x[i] ∣ D p , ε ( x ) [ i ] ∣ ≤ b − x [ i ] ϕ p , ε ( K ) ⊆ K \phi_{p,\varepsilon}(K) \subseteq K ϕ p , ε ( K ) ⊆ K
Define
V ( x ) : = ε 2 ( x − c 1 ) T S β ( x − c 1 ) . V(x) := \frac{\varepsilon}{2}(x-c\mathbf{1})^{\mathsf T}S_\beta(x-c\mathbf{1}). V ( x ) := 2 ε ( x − c 1 ) T S β ( x − c 1 ) . Then
∇ V ( x ) = ε S β ( x − c 1 ) = ∇ 2 R p ( x ) D p , ε ( x ) . \nabla V(x) = \varepsilon S_\beta(x-c\mathbf{1}) = \nabla^2 R_p(x)D_{p,\varepsilon}(x). ∇ V ( x ) = ε S β ( x − c 1 ) = ∇ 2 R p ( x ) D p , ε ( x ) . Hence D p , ε ( x ) T d ∇ R p ( x ) = d V ( x ) D_{p,\varepsilon}(x)^{\mathsf T}\mathrm{d}\nabla R_p(x) = \mathrm{d}V(x) D p , ε ( x ) T d ∇ R p ( x ) = d V ( x )
Ψ ( θ ) : = ε 2 ( ( θ [ 1 ] 1 / ( p − 1 ) θ [ 2 ] 1 / ( p − 1 ) ) − c 1 ) T S β ( ( θ [ 1 ] 1 / ( p − 1 ) θ [ 2 ] 1 / ( p − 1 ) ) − c 1 ) \Psi(\theta) := \frac{\varepsilon}{2}\left(\begin{pmatrix}\theta[1]^{1/(p-1)}\\\theta[2]^{1/(p-1)}\end{pmatrix}-c\mathbf{1}\right)^{\mathsf T}S_\beta\left(\begin{pmatrix}\theta[1]^{1/(p-1)}\\\theta[2]^{1/(p-1)}\end{pmatrix}-c\mathbf{1}\right) Ψ ( θ ) := 2 ε ( ( θ [ 1 ] 1/ ( p − 1 ) θ [ 2 ] 1/ ( p − 1 ) ) − c 1 ) T S β ( ( θ [ 1 ] 1/ ( p − 1 ) θ [ 2 ] 1/ ( p − 1 ) ) − c 1 ) satisfies D p , ε ( x ) = ∇ θ Ψ ( ∇ R p ( x ) ) D_{p,\varepsilon}(x) = \nabla_\theta\Psi(\nabla R_p(x)) D p , ε ( x ) = ∇ θ Ψ ( ∇ R p ( x )) R p R_p R p D p , ε D_{p,\varepsilon} D p , ε Ψ \Psi Ψ R p ∗ R_p^* R p ∗ O ( T ) O(\sqrt{T}) O ( T )
For FTRL, the minimization defining ∇ R p , K ∗ \nabla R^{*}_{p,K} ∇ R p , K ∗ i i i
∇ R p , K ∗ ( θ ) [ i ] = { a , θ [ i ] ≤ a p − 1 , θ [ i ] 1 / ( p − 1 ) , a p − 1 < θ [ i ] < b p − 1 , b , θ [ i ] ≥ b p − 1 . \nabla R^{*}_{p,K}(\theta)[i] =
\begin{cases}
a, & \theta[i] \le a^{p-1},\\
\theta[i]^{1/(p-1)}, & a^{p-1} < \theta[i] < b^{p-1},\\
b, & \theta[i] \ge b^{p-1}.
\end{cases} ∇ R p , K ∗ ( θ ) [ i ] = ⎩ ⎨ ⎧ a , θ [ i ] 1/ ( p − 1 ) , b , θ [ i ] ≤ a p − 1 , a p − 1 < θ [ i ] < b p − 1 , θ [ i ] ≥ b p − 1 . Choose numbers u < v u < v u < v b p − 1 < u < v b^{p-1} < u < v b p − 1 < u < v s < t s < t s < t a p − 1 < s < t < b p − 1 a^{p-1} < s < t < b^{p-1} a p − 1 < s < t < b p − 1 [ u , v ] × [ s , t ] [u,v] \times[s,t] [ u , v ] × [ s , t ] b b b θ [ 2 ] 1 / ( p − 1 ) \theta[2]^{1/(p-1)} θ [ 2 ] 1/ ( p − 1 ) θ [ 2 ] \theta[2] θ [ 2 ]
∂ D ~ p , ε [ 1 ] ∂ θ [ 2 ] = ε β ( p − 1 ) 2 b 2 − p θ [ 2 ] ( 2 − p ) / ( p − 1 ) > 0 , \frac{\partial\widetilde{D}_{p,\varepsilon}[1]}{\partial\theta[2]} = \frac{\varepsilon\beta}{(p-1)^2}b^{2-p}\theta[2]^{(2-p)/(p-1)} > 0, ∂ θ [ 2 ] ∂ D p , ε [ 1 ] = ( p − 1 ) 2 εβ b 2 − p θ [ 2 ] ( 2 − p ) / ( p − 1 ) > 0 , whereas ∂ D ~ p , ε [ 2 ] / ∂ θ [ 1 ] = 0 \partial\widetilde{D}_{p,\varepsilon}[2]/\partial\theta[1] = 0 ∂ D p , ε [ 2 ] / ∂ θ [ 1 ] = 0
( v − u ) [ D ~ p , ε [ 1 ] ( u , s ) − D ~ p , ε [ 1 ] ( u , t ) ] < 0. (v-u)\left[\widetilde{D}_{p,\varepsilon}[1](u,s)-\widetilde{D}_{p,\varepsilon}[1](u,t)\right] < 0. ( v − u ) [ D p , ε [ 1 ] ( u , s ) − D p , ε [ 1 ] ( u , t ) ] < 0. The dual displacement is continuous, so Theorem 9.5 produces bounded gradients with c T − O ( 1 / η ) cT-O(1/\eta) c T − O ( 1/ η ) □ \square □
Proofs for Section 9.4
Corollary 9.9. Let 1 < p ≤ 2 1 < p \le2 1 < p ≤ 2 q : = p / ( p − 1 ) q := p/(p-1) q := p / ( p − 1 )
∑ t = 1 T ⟨ g t , x t − ϕ ( x t ) ⟩ ≤ B η + L η 2 ( p − 1 ) ∑ t = 1 T ∥ g t ∥ q 2 . \sum_{t=1}^{T}\langle g_t,x_t-\phi(x_t)\rangle\le\frac{B}{\eta}+\frac{L\eta}{2(p-1)}\sum_{t=1}^{T}\lVert g_t\rVert_q^2. t = 1 ∑ T ⟨ g t , x t − ϕ ( x t )⟩ ≤ η B + 2 ( p − 1 ) L η t = 1 ∑ T ∥ g t ∥ q 2 . If ∥ g t ∥ q ≤ G \lVert g_t\rVert_q \le G ∥ g t ∥ q ≤ G G 2 L B T / ( p − 1 ) G\sqrt{2LBT/(p-1)} G 2 L B T / ( p − 1 )
Proof. The function 1 2 ∥ ⋅ ∥ p 2 \frac{1}{2}\lVert\cdot\rVert_p^2 2 1 ∥ ⋅ ∥ p 2 ( p − 1 ) (p-1) ( p − 1 ) ∥ ⋅ ∥ p \lVert\cdot\rVert_p ∥ ⋅ ∥ p 1 < p ≤ 2 1 < p \le2 1 < p ≤ 2 θ = ∇ ( 1 2 ∥ x ∥ p 2 ) + n \theta=\nabla(\frac{1}{2}\lVert x\rVert_p^2)+n θ = ∇ ( 2 1 ∥ x ∥ p 2 ) + n n ∈ N K ( x ) n\in N_K(x) n ∈ N K ( x ) ⟨ n , y − x ⟩ ≤ 0 \langle n,y-x\rangle\le0 ⟨ n , y − x ⟩ ≤ 0 y ∈ K y\in K y ∈ K ι K \iota_K ι K r = 2 r=2 r = 2 2 2 2 σ = p − 1 \sigma=p-1 σ = p − 1
Corollary 9.10. Let 2 ≤ p < ∞ 2 \le p < \infty 2 ≤ p < ∞ q : = p / ( p − 1 ) q := p/(p-1) q := p / ( p − 1 )
∑ t = 1 T ⟨ g t , x t − ϕ ( x t ) ⟩ ≤ B η + L η q − 1 q ( 2 2 − p ) q − 1 ∑ t = 1 T ∥ g t ∥ q q . \sum_{t=1}^{T}\langle g_t,x_t-\phi(x_t)\rangle\le\frac{B}{\eta}+\frac{L\eta^{q-1}}{q(2^{2-p})^{q-1}}\sum_{t=1}^{T}\lVert g_t\rVert_q^q. t = 1 ∑ T ⟨ g t , x t − ϕ ( x t )⟩ ≤ η B + q ( 2 2 − p ) q − 1 L η q − 1 t = 1 ∑ T ∥ g t ∥ q q . If ∥ g t ∥ q ≤ G \lVert g_t\rVert_q \le G ∥ g t ∥ q ≤ G p 1 / p 2 ( p − 2 ) / p G L 1 / q B 1 / p T 1 / q p^{1/p}2^{(p-2)/p}GL^{1/q}B^{1/p}T^{1/q} p 1/ p 2 ( p − 2 ) / p G L 1/ q B 1/ p T 1/ q O ( T 1 − 1 / p ) O(T^{1-1/p}) O ( T 1 − 1/ p )
Proof. For A ( s ) = ∣ s ∣ p − 2 s A(s)=|s|^{p-2}s A ( s ) = ∣ s ∣ p − 2 s
( A ( a ) − A ( b ) ) ( a − b ) ≥ 2 2 − p ∣ a − b ∣ p (A(a)-A(b))(a-b)\ge2^{2-p}|a-b|^p ( A ( a ) − A ( b )) ( a − b ) ≥ 2 2 − p ∣ a − b ∣ p holds for all real a , b a,b a , b x x x y y y
D p − 1 ∥ ⋅ ∥ p p ( y , x ) ≥ 2 2 − p p ∥ y − x ∥ p p . D_{p-1\lVert\cdot\rVert_p^p}(y,x)\ge\frac{2^{2-p}}{p}\lVert y-x\rVert_p^p. D p − 1 ∥ ⋅ ∥ p p ( y , x ) ≥ p 2 2 − p ∥ y − x ∥ p p . The same normal-cone argument as above shows that adding ι K \iota_K ι K ( 2 2 − p , p ) (2^{2-p},p) ( 2 2 − p , p ) □ \square □
Proofs for Section 10
Theorem 10.3. Let μ ˉ T : = T − 1 ∑ t = 1 T δ x t \bar{\mu}_T := T^{-1}\sum_{t=1}^{T}\delta_{x_t} μ ˉ T := T − 1 ∑ t = 1 T δ x t g i , t : = ∇ x i ℓ i ( x i , t , x − i , t ) g_{i,t} := \nabla_{x_i}\ell_i(x_{i,t},x_{-i,t}) g i , t := ∇ x i ℓ i ( x i , t , x − i , t ) i i i
sup ϕ i ∈ Φ i ∑ t = 1 T ⟨ g i , t , x i , t − ϕ i ( x i , t ) ⟩ ≤ R i , T , \sup_{\phi_i\in\Phi_i}\sum_{t=1}^{T}\langle g_{i,t},x_{i,t}-\phi_i(x_{i,t})\rangle\le R_{i,T}, ϕ i ∈ Φ i sup t = 1 ∑ T ⟨ g i , t , x i , t − ϕ i ( x i , t )⟩ ≤ R i , T , then μ ˉ T \bar{\mu}_T μ ˉ T ε T \varepsilon_T ε T ( Φ i ) i (\Phi_i)_i ( Φ i ) i ε T = max i R i , T / T \varepsilon_T=\max_i R_{i,T}/T ε T = max i R i , T / T Φ i , e x a c t ( B i , L i ) \Phi_{i,\mathrm{exact}}(B_i,L_i) Φ i , exact ( B i , L i ) ∥ g i , t ∥ ≤ G i \lVert g_{i,t}\rVert\le G_i ∥ g i , t ∥ ≤ G i ε T ≤ max i G i 6 L i B i / T \varepsilon_T\le\max_i G_i\sqrt{6L_iB_i/T} ε T ≤ max i G i 6 L i B i / T
Proof. Fix player i i i ϕ i ∈ Φ i \phi_i\in\Phi_i ϕ i ∈ Φ i ℓ i \ell_i ℓ i
ℓ i ( x i , t , x − i , t ) − ℓ i ( ϕ i ( x i , t ) , x − i , i , t ) ≤ ⟨ ∇ x i ℓ i ( x i , t , x − i , t ) , x i , t − ϕ i ( x i , t ) ⟩ . \ell_i(x_{i,t},x_{-i,t})-\ell_i(\phi_i(x_{i,t}),x_{-i,i,t})
\le\langle\nabla_{x_i}\ell_i(x_{i,t},x_{-i,t}),x_{i,t}-\phi_i(x_{i,t})\rangle. ℓ i ( x i , t , x − i , t ) − ℓ i ( ϕ i ( x i , t ) , x − i , i , t ) ≤ ⟨ ∇ x i ℓ i ( x i , t , x − i , t ) , x i , t − ϕ i ( x i , t )⟩ . Summing, taking the supremum over Φ i \Phi_i Φ i T T T ε T \varepsilon_T ε T μ ˉ T \bar{\mu}_T μ ˉ T
Proposition 10.1. Let A i A_i A i i i i Γ i ( μ , ϕ i ) ≤ 0 \Gamma_i(\mu,\phi_i)\le0 Γ i ( μ , ϕ i ) ≤ 0 ϕ i ∈ A i \phi_i\in A_i ϕ i ∈ A i ϕ i ∈ Rad I ( A i ) \phi_i\in\operatorname{Rad}_{I}(A_i) ϕ i ∈ Rad I ( A i )
Proof. Since A i ⊆ Rad I ( A i ) A_i\subseteq\operatorname{Rad}_{I}(A_i) A i ⊆ Rad I ( A i ) α = 1 \alpha=1 α = 1 μ \mu μ A i A_i A i ϕ i ∈ Rad I ( A i ) \phi_i\in\operatorname{Rad}_{I}(A_i) ϕ i ∈ Rad I ( A i ) α ∈ ( 0 , 1 ] \alpha\in(0,1] α ∈ ( 0 , 1 ] ϕ i , α = ( 1 − α ) I + α ϕ i \phi_{i,\alpha}=(1-\alpha)I+\alpha\phi_i ϕ i , α = ( 1 − α ) I + α ϕ i A i A_i A i ℓ i \ell_i ℓ i
ℓ i ( x i , x − i ) − ℓ i ( ϕ i , α ( x i ) , x − i ) ≥ α ( ℓ i ( x i , x − i ) − ℓ i ( ϕ i ( x i ) , x − i ) ) . \ell_i(x_i,x_{-i})-\ell_i(\phi_{i,\alpha}(x_i),x_{-i})
\ge\alpha\bigl(\ell_i(x_i,x_{-i})-\ell_i(\phi_i(x_i),x_{-i})\bigr). ℓ i ( x i , x − i ) − ℓ i ( ϕ i , α ( x i ) , x − i ) ≥ α ( ℓ i ( x i , x − i ) − ℓ i ( ϕ i ( x i ) , x − i ) ) . The joint continuity and compactness assumptions ensure that both loss differences are bounded and measurable. Thus both gains are integrable. Taking expectations yields Γ i ( μ , ϕ i , α ) ≥ α Γ i ( μ , ϕ i ) \Gamma_i(\mu,\phi_{i,\alpha})\ge\alpha\Gamma_i(\mu,\phi_i) Γ i ( μ , ϕ i , α ) ≥ α Γ i ( μ , ϕ i ) Γ i ( μ , ϕ i ) ≤ 0 \Gamma_i(\mu,\phi_i)\le0 Γ i ( μ , ϕ i ) ≤ 0
Theorem 10.4. Under the unrestricted conventions above, every convex game with compact convex action sets and jointly continuous losses satisfies
ConCE = PCE . \operatorname{ConCE}=\operatorname{PCE}. ConCE = PCE . Proof. By Corollary 6.3, the smooth exact-form class is the identity-radial closure of the unrestricted proximal class. Apply Proposition 10.1 player by player.
Corollary 10.5. For any set of regularizers R i R_i R i i ∈ [ N ] i\in[N] i ∈ [ N ]
PCE ( R i ) i B r e g = Eq ( R i ) i r a d . \operatorname{PCE}^{\mathrm{Breg}}_{(R_i)_i}=\operatorname{Eq}^{\mathrm{rad}}_{(R_i)_i}. PCE ( R i ) i Breg = Eq ( R i ) i rad . Proof. For every player, Φ r a d R i ( K i ) = Rad I ( Φ p r o x R i ( K i ) ) \Phi_{\mathrm{rad}}^{R_i}(K_i)=\operatorname{Rad}_{I}(\Phi_{\mathrm{prox}}^{R_i}(K_i)) Φ rad R i ( K i ) = Rad I ( Φ prox R i ( K i )) Definition 8.13 . Apply Proposition 10.1 player by player.
Corollary 10.7. For every finitely supported distribution in a convex game,
μ ∈ CE ⟺ μ ∈ ConCE ⟺ μ ∈ PCE . \mu\in\operatorname{CE}\iff\mu\in\operatorname{ConCE}\iff\mu\in\operatorname{PCE}. μ ∈ CE ⟺ μ ∈ ConCE ⟺ μ ∈ PCE . In particular the three notions coincide in every finite game.
Proof. Only PCE ⊆ \subseteq ⊆ τ i \tau_i τ i x 1 , … , x m x^1,\ldots,x^m x 1 , … , x m i i i y j : = τ i ( x j ) y^j:=\tau_i(x^j) y j := τ i ( x j ) α > 0 \alpha>0 α > 0
ϕ i , α ( x j ) = ( 1 − α ) x j + α τ i ( x j ) \phi_{i,\alpha}(x^j)=(1-\alpha)x^j+\alpha\tau_i(x^j) ϕ i , α ( x j ) = ( 1 − α ) x j + α τ i ( x j ) at every supported recommendation. Convexity yields Γ i ( μ , ϕ i , α ) ≥ α Γ i ( μ , τ i ) > 0 \Gamma_i(\mu,\phi_{i,\alpha})\geq\alpha\Gamma_i(\mu,\tau_i)>0 Γ i ( μ , ϕ i , α ) ≥ α Γ i ( μ , τ i ) > 0 μ ∉ P C E \mu\notin\mathrm{PCE} μ ∈ / PCE
Theorem 10.8. Suppose that every player has a compact interval action set K i : = [ a i , b i ] ⊆ R K_i:=[a_i,b_i]\subseteq\mathbb{R} K i := [ a i , b i ] ⊆ R ℓ i : K → R \ell_i:K\to\mathbb{R} ℓ i : K → R K : = ∏ j = 1 N K j K:=\prod_{j=1}^{N}K_j K := ∏ j = 1 N K j
C E = C o n C E = P C E \mathrm{CE}=\mathrm{ConCE}=\mathrm{PCE} CE = ConCE = PCE under the unrestricted definitions used in this section when the approximation error is zero.
Proof. The inclusion C E ⊆ C o n C E \mathrm{CE}\subseteq\mathrm{ConCE} CE ⊆ ConCE C o n C E = P C E \mathrm{ConCE}=\mathrm{PCE} ConCE = PCE C o n C E ⊆ C E \mathrm{ConCE}\subseteq\mathrm{CE} ConCE ⊆ CE
Suppose μ ∉ C E \mu\notin\mathrm{CE} μ ∈ / CE i i i τ i : K i → K i \tau_i:K_i\to K_i τ i : K i → K i Γ i ( μ , τ i ) > 0 \Gamma_i(\mu,\tau_i)>0 Γ i ( μ , τ i ) > 0 ν i \nu_i ν i i i i μ \mu μ K i : = [ a i , b i ] K_i:=[a_i,b_i] K i := [ a i , b i ] a i < b i a_i<b_i a i < b i K i K_i K i
For every n n n A n ⊆ K i A_n\subseteq K_i A n ⊆ K i ν i ( K i ∖ A n ) < 1 / n \nu_i(K_i\setminus A_n)<1/n ν i ( K i ∖ A n ) < 1/ n τ i ∣ A n \tau_i|_{A_n} τ i ∣ A n h n : K i → K i h_n:K_i\to K_i h n : K i → K i τ i \tau_i τ i A n A_n A n u : = ( x − a i ) / ( b i − a i ) u:=(x-a_i)/(b_i-a_i) u := ( x − a i ) / ( b i − a i ) B m h n B_mh_n B m h n
( B m h n ) ( x ) = ∑ k = 0 m h n ( a i + k m ( b i − a i ) ) ( m k ) u k ( 1 − u ) m − k . (B_mh_n)(x)=\sum_{k=0}^{m}h_n\left(a_i+\frac{k}{m}(b_i-a_i)\right)\binom{m}{k}u^k(1-u)^{m-k}. ( B m h n ) ( x ) = k = 0 ∑ m h n ( a i + m k ( b i − a i ) ) ( k m ) u k ( 1 − u ) m − k . Bernstein approximation is uniform for continuous functions, so choose m n m_n m n ∥ B m n h n − h n ∥ ∞ < 1 / n \lVert B_{m_n}h_n-h_n\rVert_\infty<1/n ∥ B m n h n − h n ∥ ∞ < 1/ n ϕ i , n : = B m n h n \phi_{i,n}:=B_{m_n}h_n ϕ i , n := B m n h n ϕ i , n ( K i ) ⊆ K i \phi_{i,n}(K_i)\subseteq K_i ϕ i , n ( K i ) ⊆ K i ν i ( ∣ ϕ i , n − τ i ∣ > 1 / n ) ≤ ν i ( K i ∖ A n ) < 1 / n \nu_i(\lvert\phi_{i,n}-\tau_i\rvert>1/n)\leq\nu_i(K_i\setminus A_n)<1/n ν i (∣ ϕ i , n − τ i ∣ > 1/ n ) ≤ ν i ( K i ∖ A n ) < 1/ n ϕ i , n ( X i ) → τ i ( X i ) \phi_{i,n}(X_i)\to\tau_i(X_i) ϕ i , n ( X i ) → τ i ( X i ) μ \mu μ
Continuity on compact K K K ℓ i \ell_i ℓ i ℓ i ( ϕ i , n ( X i ) , X − i ) ⟶ ℓ i ( τ i ( X i ) , X − i ) \ell_i(\phi_{i,n}(X_i),X_{-i})\longrightarrow\ell_i(\tau_i(X_i),X_{-i}) ℓ i ( ϕ i , n ( X i ) , X − i ) ⟶ ℓ i ( τ i ( X i ) , X − i ) L 1 ( μ ) L^1(\mu) L 1 ( μ ) Γ i ( μ , ϕ i , n ) ⟶ Γ i ( μ , τ i ) > 0 \Gamma_i(\mu,\phi_{i,n})\longrightarrow\Gamma_i(\mu,\tau_i)>0 Γ i ( μ , ϕ i , n ) ⟶ Γ i ( μ , τ i ) > 0 ϕ i , n \phi_{i,n} ϕ i , n n n n
But every ϕ i , n \phi_{i,n} ϕ i , n
Ψ i , n ( x ) = ∫ a i x ( s − ϕ i , n ( s ) ) d s \Psi_{i,n}(x)=\int_{a_i}^{x}(s-\phi_{i,n}(s))\,\mathrm{d}s Ψ i , n ( x ) = ∫ a i x ( s − ϕ i , n ( s )) d s is defined on a neighborhood of K i K_i K i Ψ i , n ′ ( x ) = x − ϕ i , n ( x ) \Psi'_{i,n}(x)=x-\phi_{i,n}(x) Ψ i , n ′ ( x ) = x − ϕ i , n ( x )
So a measurable profitable deviation produces a smooth feasible exact-form profitable deviation, so μ ∉ C o n C E \mu\notin\mathrm{ConCE} μ ∈ / ConCE C o n C E ⊆ C E \mathrm{ConCE}\subseteq\mathrm{CE} ConCE ⊆ CE
Theorem 10.9. There is a two-player convex game and a distribution μ \mu μ
μ ∈ C o n C E = P C E but μ ∉ C E . \mu\in\mathrm{ConCE}=\mathrm{PCE}\qquad\text{but}\qquad\mu\notin\mathrm{CE}. μ ∈ ConCE = PCE but μ ∈ / CE . Consequently C E ⊆ C o n C E = P C E \mathrm{CE}\subseteq\mathrm{ConCE}=\mathrm{PCE} CE ⊆ ConCE = PCE
Proof. Let K ⊆ R 2 K\subseteq\mathbb{R}^2 K ⊆ R 2
J : = ( 0 − 1 1 0 ) J:=\begin{pmatrix}0&-1\\1&0\end{pmatrix} J := ( 0 1 − 1 0 ) be the counterclockwise 90 ∘ 90^\circ 9 0 ∘ ℓ 1 ( a , s ) = ⟨ J s , a ⟩ \ell_1(a,s)=\langle Js,a\rangle ℓ 1 ( a , s ) = ⟨ J s , a ⟩ μ \mu μ ( X , X ) (X,X) ( X , X ) X X X K × K K \times K K × K ϕ = I − ∇ Ψ \phi=I-\nabla\Psi ϕ = I − ∇Ψ
Γ 1 ( μ , ϕ ) = 1 π ∫ K ⟨ J x , ∇ Ψ ( x ) ⟩ d x = 0 , \Gamma_1(\mu,\phi)=\frac{1}{\pi}\int_K \langle Jx,\nabla\Psi(x)\rangle\,\mathrm{d}x=0, Γ 1 ( μ , ϕ ) = π 1 ∫ K ⟨ J x , ∇Ψ ( x )⟩ d x = 0 , because div ( J x ) = 0 \operatorname{div}(Jx)=0 div ( J x ) = 0 J x Jx J x μ ∈ ConCE = PCE \mu\in\operatorname{ConCE}=\operatorname{PCE} μ ∈ ConCE = PCE τ ( x ) = − J x / ∥ x ∥ \tau(x)=-Jx/\lVert x\rVert τ ( x ) = − J x / ∥ x ∥ x ≠ 0 x\ne0 x = 0 τ ( 0 ) = 0 \tau(0)=0 τ ( 0 ) = 0 E ∥ X ∥ = 2 / 3 \mathbb{E}\lVert X\rVert=2/3 E ∥ X ∥ = 2/3 μ ∉ CE \mu\notin\operatorname{CE} μ ∈ / CE □ \square □
