Introduction
No-regret learning is a central paradigm for decision making under uncertainty and for learning in games. In each of T T T rounds, a learner chooses a probability distribution x t x^t x t over n n n actions, observes a loss vector ℓ t ∈ [ 0 , 1 ] n \ell^t \in[0,1]^n ℓ t ∈ [ 0 , 1 ] n , and suffers the expected loss ⟨ x t , ℓ t ⟩ \langle x^t,\ell^t\rangle ⟨ x t , ℓ t ⟩ . The learner’s regret , Reg ( T ) = ∑ t = 1 T ⟨ x t , ℓ t ⟩ − min a ∈ [ n ] ∑ t = 1 T ℓ t ( a ) \operatorname{Reg}(T) = \sum_{t=1}^{T}\langle x^t,\ell^t\rangle- \min_{a\in[n]}\sum_{t=1}^{T}\ell^t(a) Reg ( T ) = ∑ t = 1 T ⟨ x t , ℓ t ⟩ − min a ∈ [ n ] ∑ t = 1 T ℓ t ( a ) , compares their cumulative loss with that of the best fixed action in hindsight. The canonical no-regret algorithm is multiplicative weights, or Hedge [25 , 17 ] , which plays x t ( a ) ∝ exp ( − η ∑ s < t ℓ s ( a ) ) x^t(a) \propto\exp(-\eta\sum_{s<t}\ell^s(a)) x t ( a ) ∝ exp ( − η ∑ s < t ℓ s ( a )) and, for a learning rate η ≃ log n / T \eta\simeq\sqrt{\log n/T} η ≃ log n / T , guarantees Reg ( T ) = O ( T log n ) \operatorname{Reg}(T) = O(\sqrt{T\log n}) Reg ( T ) = O ( T log n ) against every sequence of losses. Algorithms of this kind underlie boosting [17 ] , fast approximate solvers for zero-sum games, linear programs and combinatorial problems [18 , 3 ] , and counterfactual regret minimization, the engine behind superhuman poker programs [44 , 7 ] ; they also model bidder behavior in ad auctions [31 ] . In games, when every player has regret o ( T ) o(T) o ( T ) , the empirical distribution of play converges to the set of coarse correlated equilibria [22 , 30 ] , at a rate governed by the largest average regret among the players; under the stronger requirement of vanishing internal regret, it converges to correlated equilibria [19 , 23 ] .
Against an adversary, who may choose every loss vector with full knowledge of the learner’s algorithm, the rate T log n \sqrt{T\log n} T log n cannot be improved [8 ] . A player in a game, however, faces other learners rather than an adversary. If they adapt gradually, the loss vector ℓ t + 1 \ell^{t+1} ℓ t + 1 resembles ℓ t \ell^t ℓ t , and anticipating this may yield regret far below the adversarial rate. Optimistic Hedge [35 , 36 , 37 ] builds this anticipation into Hedge by counting the most recent loss vector twice, the second time as a prediction of the next one. Suitably tuned, it retains the adversarial guarantee of Hedge and requires no knowledge of the game and no coordination beyond a shared learning rate.
A long line of work has studied the regret of Optimistic Hedge when all m m m players of a general-sum game, each with n n n actions, run it with a common, fixed learning rate. Its starting point is the RVU (regret bounded by variation in utilities) property of Syrgkanis, Agarwal, Luo and Schapire [37 ] , which bounds the regret by log n / η \log n/\eta log n / η , plus a term proportional to the variation of the losses, minus one proportional to the movement of the player’s own strategy. Since strategies move by O ( η ) O(\eta) O ( η ) per round, this can be shown to yield regret O ( T 1 / 4 ) O(T^{1/4}) O ( T 1/4 ) . The exponent was lowered to 1 / 6 1/6 1/6 for two players by Chen and Peng [10 ] , and the first polylogarithmic bound, O ( m log n log 4 T ) O(m\log n\log^4 T) O ( m log n log 4 T ) , was obtained by Daskalakis, Fishelson and Golowich [13 ] . Ha [21 ] recently improved it to O ( m log n log T ) O(\sqrt{m}\log n\log T) O ( m log n log T ) , and later showed that, for a suitable learning rate, the regret is bounded by a constant C ( m , n ) C(m,n) C ( m , n ) independent of T T T [20 ] .
This development settles the dependence of the regret on the horizon, but not on the size of the game. The proof of [20 ] is non-constructive: it derives an exact finite-order linear relation among the temporal differences of the trajectory from a Noetherianity theorem of Frisch [16 ] . This shows that C ( m , n ) C(m,n) C ( m , n ) is finite, but gives no control over its dependence on m m m and n n n . Were C ( m , n ) C(m,n) C ( m , n ) exponential in n n n , the constant bound would be weaker than the logarithmic one at every horizon of practical interest. Meanwhile, more elaborate dynamics, based on higher-order or multiplicative forms of optimism, were recently shown to attain horizon-independent regret polynomial in m m m and n n n , and even polylogarithmic in n n n for external regret [24 , 4 , 39 , 26 ] . As Optimistic Hedge remains the most natural and best studied of the optimistic dynamics, we ask whether its simple one-step optimism already suffices:
Does fixed-learning-rate Optimistic Hedge guarantee every player regret poly ( m , n ) \operatorname{poly}(m,n) poly ( m , n ) , independently of the horizon?
We show that, perhaps surprisingly, the answer is affirmative for two players and negative for three or more, although the polylogarithmic bounds above depend on m m m only polynomially.
Main results
Throughout, all players run Optimistic Hedge with a common, fixed learning rate η \eta η , starting from the uniform distribution and observing their expected loss vectors. We obtain three results.
(i) Logarithmic regret for m m m players. We develop a temporal approximation framework for the analysis of Optimistic Hedge, and use it to reprove the logarithmic bound of Ha [21 ] , with the same dependence on m m m and n n n and with explicit constants: in every m m m -player game, the learning rate η = c / ( m log T ) \eta= c/(\sqrt{m}\log T) η = c / ( m log T ) , for a small enough absolute constant c > 0 c > 0 c > 0 , guarantees each player regret O ( m log n log T ) O(\sqrt{m}\log n\log T) O ( m log n log T ) (Theorem 3.3 ).
(ii) Constant regret for two players. There is an absolute constant c > 0 c > 0 c > 0 such that in every two-player game, any learning rate 0 < η ≤ c / n 0 < \eta\le c/n 0 < η ≤ c / n guarantees each player regret at most log n / η + η \log n/\eta+ \eta log n / η + η at every horizon T T T . In particular, η = c / n \eta= c/n η = c / n gives regret O ( n log n ) O(n\log n) O ( n log n ) for all T T T (Theorem 3.4 ).
(iii) An exponential barrier for three players. There are absolute constants C , η 0 > 0 C,\eta_0 > 0 C , η 0 > 0 such that for every n ≥ C n \ge C n ≥ C and every learning rate 0 < η ≤ η 0 0 < \eta\le\eta_0 0 < η ≤ η 0 , some three-player game with n n n actions per player forces regret 2 Ω ( n ) 2^{\Omega(n)} 2 Ω ( n ) on some player for all sufficiently large T T T . If moreover η ≥ C n 2 − n / 8 \eta\ge Cn^{2^{-n/8}} η ≥ C n 2 − n /8 , the game can be chosen so that this player’s regret grows linearly in T T T (Theorem 3.6 ). More generally, for m ≥ 3 m \ge3 m ≥ 3 players the lower bound becomes 2 Ω ( m n ) 2^{\Omega(mn)} 2 Ω ( mn ) (Theorem 6.24 ).
Our proof of (i) is short and significantly simplifies earlier polylogarithmic analyses of Optimistic Hedge [13 , 21 ] . The bound in (ii) is explicit and polynomial in n n n , and the learning rate η = c / n \eta= c/n η = c / n is oblivious to both the game and the horizon. It is also tight up to an additive constant, since even against constant losses the regret of Optimistic Hedge can reach log n / η − 1 \log n/\eta- 1 log n / η − 1 . The same elementary fact gives the first part of (iii) when η < C n 2 − n / 8 \eta< Cn^{2^{-n/8}} η < C n 2 − n /8 , so the substance of (iii) lies in its second part. Already for η = c / n \eta= c/n η = c / n , some three-player game forces linear regret once n n n is large, so adding a single player turns a constant regret bound into a linear one. To our knowledge, (iii) is the first lower bound for Optimistic Hedge beyond two-player zero-sum games, and it separates the algorithm from the more elaborate dynamics mentioned above.
Techniques. Through the variance form of the RVU bound [37 , 13 ] , both upper bounds reduce to bounding the variation of a player’s losses by their variance: along the trajectory of play, the variation ∑ t Var x t ( ℓ t − ℓ t − 1 ) \sum_t \operatorname{Var}_{x_t}(\ell^t-\ell^{t-1}) ∑ t Var x t ( ℓ t − ℓ t − 1 ) must be at most half the cumulative variance ∑ t Var x t ( ℓ t − 1 ) \sum_t \operatorname{Var}_{x_t}(\ell^{t-1}) ∑ t Var x t ( ℓ t − 1 ) , up to an additive term. Where previous work relied on the high-order smoothness of the trajectory [13 , 21 ] , we approximate the losses on short windows of time by simple functions for which such an inequality is classical. Inequalities of Markov and Bernstein type bound the derivative of a polynomial or of an exponential sum by the size of the function itself; in the integral form relevant here, every f f f in the approximating class F \mathcal{F} F satisfies
∫ I f ′ ( s ) 2 d s ≤ C F W 2 ∫ 0 W f ( s ) 2 d s , \int_I f'(s)^2\,ds \le\frac{C_{\mathcal{F}}}{W^2}\int_0^W f(s)^2\,ds, ∫ I f ′ ( s ) 2 d s ≤ W 2 C F ∫ 0 W f ( s ) 2 d s ,
where I I I is the middle part of the interval [ 0 , W ] [0,W] [ 0 , W ] . We use a discrete analogue of this inequality on windows of W W W rounds. For m m m players, F \mathcal{F} F consists of polynomials of degree O ( log T ) O(\log T) O ( log T ) , with C F = O ( log 2 T ) C_{\mathcal{F}} = O(\log^2 T) C F = O ( log 2 T ) , and the approximation error is paid for by the horizon. For two players, freezing the losses of player 2 on a window turns those of player 1 into ratios of sums of n n n exponentials, for which a Bernstein inequality of Borwein and Erdélyi [5 ] gives C F = O ( n 2 ) C_F = O(n^2) C F = O ( n 2 ) . Crucially, the error of freezing is paid for by the variance itself, through a feedback loop: the losses of player 2 move only when player 1 moves, and player 1 moves only when their own losses vary. A third player breaks this loop, and (iii) shows that the breakdown is not an artifact of the analysis.
The lower bound rests on the observation that optimism can also hurt. To leading order, a small periodic perturbation of frequency ω \omega ω in a player’s losses changes the slope of their regret in proportion to 1 2 − cos ω \frac{1}{2} - \cos\omega 2 1 − cos ω . Slow oscillations are anticipated by the optimistic prediction and reduce the regret, whereas fast ones throw the prediction out of phase and make the regret grow linearly. In our game, a third player observes two players driven into a cycle of period four. Since a small learning rate only produces slow oscillations, this cycle is generated by a cascade of blocks of eight actions, each driven quadratically by its predecessor and hence oscillating twice as fast; the log ( 1 / η ) \log(1/\eta) log ( 1/ η ) blocks this requires account for the exponential barrier.
Related work
Optimism. The optimistic step goes back to Popov’s modification of the saddle-point method of Arrow and Hurwicz [32 ] , now viewed as a single-call variant of the extragradient method and as an explicit approximation of the proximal point method [27 , 34 , 28 ] . Rakhlin and Sridharan [35 , 36 ] brought it to online learning, building on gradual-variation bounds [11 ] ; with the entropic regularizer, it yields Optimistic Hedge.
Optimistic Hedge in games. The analysis of [13 ] shows that the finite differences of the dynamics of every order k k k up to O ( log T ) O(\log T) O ( log T ) are of order ( m η ) k (m\eta)^k ( m η ) k , up to k k k -dependent factors, and converts them into control of the first-order differences through a Fourier-analytic interpolation inequality. The same scheme underlies the bound of [21 ] and has been extended to swap regret, and hence to correlated equilibria [1 ] . More is known in two-player zero-sum games, where near-optimal no-regret dynamics were first designed [12 ] : the sum of the two regrets is bounded by a constant [36 ] , and Optimistic Hedge’s individual regret is logarithmic in the numbers of actions, with algorithm-specific lower bounds tight for the sum of the regrets [42 ] .
Other dynamics. Logarithmic regret is also achieved by optimistic follow-the-regularized-leader with logarithmic regularizers [2 , 15 ] and by cautious optimism, which adapts the learning rate [40 ] , while clairvoyant multiplicative weights approach coarse correlated equilibria at rate O ( log T / T ) O(\log T/T) O ( log T / T ) [33 ] . The horizon-independent bounds discussed above rely on higher-order predictors built from exponential moving averages [24 ] or on a discounted ( m + 1 ) (m+1) ( m + 1 ) -th order extrapolation over a lifted strategy space [4 ] , on multiplicatively optimistic regret matching [39 ] , and, for swap regret, on two-scale higher-order optimism [26 ] .
Cycles. Multiplicative weights and related dynamics are known to cycle, or even diverge, in zero-sum games [29 , 6 ] , where optimistic methods instead converge in the last iterate when the equilibrium is unique [14 , 43 ] , though not necessarily fast [9 ] . Our construction shows that in general-sum games Optimistic Hedge itself may cycle, at a high cost in regret.
Problem setting
Games. A finite (general sum) game with m ≥ 2 m \ge2 m ≥ 2 players is specified by action sets [ n i ] = { 1 , … , n i } [n_i] = \{1,\ldots,n_i\} [ n i ] = { 1 , … , n i } and loss functions L i : ∏ j = 1 m [ n j ] → [ 0 , 1 ] L_i : \prod_{j=1}^{m}[n_j] \to[0,1] L i : ∏ j = 1 m [ n j ] → [ 0 , 1 ] , one for each player i ∈ [ m ] i \in[m] i ∈ [ m ] . A mixed strategy of player i i i is a probability vector x i ∈ Δ n i x_i \in\Delta_{n_i} x i ∈ Δ n i , and a profile x = ( x 1 , … , x m ) x = (x_1,\ldots,x_m) x = ( x 1 , … , x m ) of mixed strategies induces for each player the vector of expected losses of their actions,
ℓ i ( x ) ( a i ) = ∑ a − i L i ( a i , a − i ) ∏ j ≠ i x j ( a j ) , a i ∈ [ n i ] , (1) \ell_i(x)(a_i) = \sum_{a_{-i}} L_i(a_i,a_{-i}) \prod_{j\ne i} x_j(a_j), \qquad a_i \in[n_i], \tag*{(1)} ℓ i ( x ) ( a i ) = a − i ∑ L i ( a i , a − i ) j = i ∏ x j ( a j ) , a i ∈ [ n i ] , (1)
which is multilinear in the opponents’ strategies. We write G ( m , n ) G(m,n) G ( m , n ) for the class of games with m m m players and n n n actions per player.
Repeated play and feedback. We consider a general sum game played repeatedly for T T T rounds. In round t t t the players simultaneously choose a mixed strategy profile x t = ( x 1 t , … , x m t ) x^t = (x_1^t,\ldots,x_m^t) x t = ( x 1 t , … , x m t ) , and player i i i observes their entire expected loss vector ℓ i t = ℓ i ( x t ) ∈ [ 0 , 1 ] n i \ell_i^t = \ell_i(x^t) \in[0,1]^{n_i} ℓ i t = ℓ i ( x t ) ∈ [ 0 , 1 ] n i . This is the full-information, expected-loss feedback model of [37 , 13 ] ; the players observe exact expectations rather than the losses of sampled action profiles. The individual (external) regret of player i i i after T T T rounds is
Reg i ( T ) = ∑ t = 1 T ⟨ x i t , ℓ i t ⟩ − min a ∈ [ n i ] ∑ t = 1 T ℓ i t ( a ) . (2) \operatorname{Reg}_i(T) = \sum_{t=1}^{T} \langle x_i^t,\ell_i^t\rangle- \min_{a\in[n_i]} \sum_{t=1}^{T} \ell_i^t(a). \tag*{(2)} Reg i ( T ) = t = 1 ∑ T ⟨ x i t , ℓ i t ⟩ − a ∈ [ n i ] min t = 1 ∑ T ℓ i t ( a ) . (2)
Optimistic Hedge. All players update their strategies using Optimistic Hedge with a common, fixed learning rate η > 0 \eta> 0 η > 0 . Each player i i i starts from the uniform distribution and, having observed ℓ i t \ell_i^t ℓ i t , updates
x i t + 1 ( a ) = x i t ( a ) exp [ − η ( 2 ℓ i t ( a ) − ℓ i t − 1 ( a ) ) ] ∑ c ∈ [ n i ] x i t ( c ) exp [ − η ( 2 ℓ i t ( c ) − ℓ i t − 1 ( c ) ) ] , a ∈ [ n i ] , ℓ i 0 = 0. (3) x_i^{t+1}(a) = \frac{x_i^t(a)\exp\left[-\eta\left(2\ell_i^t(a)-\ell_i^{t-1}(a)\right)\right]}{\sum_{c\in[n_i]}x_i^t(c)\exp\left[-\eta\left(2\ell_i^t(c)-\ell_i^{t-1}(c)\right)\right]}, \qquad a\in[n_i], \qquad\ell_i^0=0. \tag*{(3)} x i t + 1 ( a ) = ∑ c ∈ [ n i ] x i t ( c ) exp [ − η ( 2 ℓ i t ( c ) − ℓ i t − 1 ( c ) ) ] x i t ( a ) exp [ − η ( 2 ℓ i t ( a ) − ℓ i t − 1 ( a ) ) ] , a ∈ [ n i ] , ℓ i 0 = 0. (3)
Notation. Given a distribution p ∈ Δ n p\in\Delta_n p ∈ Δ n and z ∈ R n z\in\mathbb{R}^n z ∈ R n we write E p [ z ] \mathbb{E}_p[z] E p [ z ] for the exponential tilt of p p p by z z z , i.e. the probability vector with entries E p [ z ] ( a ) = p ( a ) e z ( a ) / ∑ c p ( c ) e z ( c ) \mathbb{E}_p[z](a)=p(a)e^{z(a)}/\sum_c p(c)e^{z(c)} E p [ z ] ( a ) = p ( a ) e z ( a ) / ∑ c p ( c ) e z ( c ) , so that for an Optimistic Hedge update, x t + 1 = E x t [ − η ( 2 ℓ t − ℓ t − 1 ) ] x^{t+1}=\mathbb{E}_{x^t}[-\eta(2\ell^t-\ell^{t-1})] x t + 1 = E x t [ − η ( 2 ℓ t − ℓ t − 1 )] . We also denote the variance of z z z with respect to p p p by Var p ( z ) = ∑ a p ( a ) ( z ( a ) − ⟨ p , z ⟩ ) 2 \operatorname{Var}_p(z)=\sum_a p(a)(z(a)-\langle p,z\rangle)^2 Var p ( z ) = ∑ a p ( a ) ( z ( a ) − ⟨ p , z ⟩ ) 2 , the variance seminorm ∥ z ∥ p = Var p ( z ) \|z\|_p=\sqrt{\operatorname{Var}_p(z)} ∥ z ∥ p = Var p ( z ) , and osc ( z ) = max a ∈ [ n ] z ( a ) − min a ∈ [ n ] z ( a ) \operatorname{osc}(z)=\max_{a\in[n]}z(a)-\min_{a\in[n]}z(a) osc ( z ) = max a ∈ [ n ] z ( a ) − min a ∈ [ n ] z ( a ) .
Technical overview
This section states our results formally and outlines their proofs.1 We begin by introducing the temporal approximation framework we use to derive our two upper bounds.
Temporal approximation framework
Fix a player i i i with n n n actions who runs Optimistic Hedge against an arbitrary sequence ℓ i 1 , … , ℓ i T ∈ [ 0 , 1 ] n \ell_i^1,\ldots,\ell_i^T\in[0,1]^n ℓ i 1 , … , ℓ i T ∈ [ 0 , 1 ] n of loss vectors, with ℓ i 0 = 0 \ell_i^0=0 ℓ i 0 = 0 , and let x i t x_i^t x i t be their strategy in round t t t . Consider the total variation and the cumulative variance of the losses along the player’s trajectory,
D i ( T ) = ∑ t = 1 T ∥ ℓ i t − ℓ i t − 1 ∥ x i t 2 , V i ( T ) = ∑ t = 1 T ∥ ℓ i t − 1 ∥ x i t 2 . (4) D_i(T)=\sum_{t=1}^{T}\|\ell_i^t-\ell_i^{t-1}\|_{x_i^t}^2,\qquad V_i(T)=\sum_{t=1}^{T}\|\ell_i^{t-1}\|_{x_i^t}^2. \tag*{(4)} D i ( T ) = t = 1 ∑ T ∥ ℓ i t − ℓ i t − 1 ∥ x i t 2 , V i ( T ) = t = 1 ∑ T ∥ ℓ i t − 1 ∥ x i t 2 . (4)
Here, for a probability vector p ∈ Δ n p \in\Delta_n p ∈ Δ n and v ∈ R n v \in\mathbb{R}^n v ∈ R n , we write ∥ v ∥ p 2 = Var p ( v ) = ∑ a ∈ [ n ] p ( a ) ( v ( a ) − ⟨ p , v ⟩ ) 2 \lVert v\rVert_p^2 = \operatorname{Var}_p(v) = \sum_{a \in[n]} p(a)\left(v(a) - \langle p,v\rangle\right)^2 ∥ v ∥ p 2 = Var p ( v ) = ∑ a ∈ [ n ] p ( a ) ( v ( a ) − ⟨ p , v ⟩ ) 2 for the variance of v v v under p p p . Throughout, ≲ \lesssim ≲ hides absolute constants.
Our framework relies on the variance form of the RVU inequality of [37 ] (Lemma 4.1 in [13 ] ), proven for completeness in Appendix § 11 .
Lemma 3.1 (Variance RVU [13 ] ). For 0 < η ≤ 1 / 8 0 < \eta\le1/8 0 < η ≤ 1/8 ,
Reg i ( T ) ≤ log n η + 2 η 3 D i ( T ) − η 3 V i ( T ) . (5) \operatorname{Reg}_i(T) \le\frac{\log n}{\eta} + \frac{2\eta}{3}D_i(T) - \frac{\eta}{3}V_i(T). \tag*{(5)} Reg i ( T ) ≤ η log n + 3 2 η D i ( T ) − 3 η V i ( T ) . (5)
Corollary 3.2. If 0 < η ≤ 1 / 8 0 < \eta\le1/8 0 < η ≤ 1/8 and D i ( T ) ≤ 1 2 V i ( T ) + B D_i(T) \le\frac{1}{2}V_i(T) + B D i ( T ) ≤ 2 1 V i ( T ) + B , then Reg i ( T ) ≤ log n / η + 2 3 η B \operatorname{Reg}_i(T) \le\log n/\eta+ \frac{2}{3}\eta B Reg i ( T ) ≤ log n / η + 3 2 η B .
Each of our regret upper bounds therefore culminates in a variation–variance inequality D i ( T ) ≤ 1 2 V i ( T ) + B D_i(T) \le\frac{1}{2}V_i(T) + B D i ( T ) ≤ 2 1 V i ( T ) + B along the trajectory of play. To prove this inequality, we consider windows of W W W consecutive rounds, and within each window (that is, locally) we approximate the Optimistic Hedge losses, for each action separately, (as functions of the local time s = 0 , 1 , … , W s = 0,1,\ldots,W s = 0 , 1 , … , W within the window) by a class F \mathcal{F} F of simple functions for which the RVU property is straightforward to prove. Schematically, every f ∈ F f \in\mathcal{F} f ∈ F satisfies
∑ s = 0 W ( f ( s + 1 ) − f ( s ) ) 2 ≲ C W 2 ∑ s = 0 W f ( s ) 2 , (6) \sum_{s=0}^{W}\left(f(s+1)-f(s)\right)^2 \lesssim\frac{C}{W^2}\sum_{s=0}^{W}f(s)^2, \tag*{(6)} s = 0 ∑ W ( f ( s + 1 ) − f ( s ) ) 2 ≲ W 2 C s = 0 ∑ W f ( s ) 2 , (6)
where C C C is a constant which depends on the choice of F \mathcal{F} F , which would suffice for Corollary 3.2 as long as C / W 2 C/W^2 C / W 2 can be bounded by a small constant. We remark that we ultimately need a version of Equation (6) which holds in terms of the variance with respect to a distribution p ∈ Δ n p \in\Delta_n p ∈ Δ n , but this step only amounts to centering followed by taking expectation with respect to p p p , which are both straightforward. We also note that the quantity C / W 2 C/W^2 C / W 2 is not sufficiently small in general for the losses induced by Optimistic Hedge, an example being the construction of the lower bound given in Theorem 3.6 which gives a loss sequence which oscillates frequently while being small in magnitude. The function class F \mathcal{F} F approximates the true loss trajectory in the sense that for every window { t 0 , t 0 + 1 , … , t 0 + W } ⊆ [ T ] \{t_0,t_0+1,\ldots,t_0+W\} \subseteq[T] { t 0 , t 0 + 1 , … , t 0 + W } ⊆ [ T ] , local time 0 ≤ s ≤ W 0 \le s \le W 0 ≤ s ≤ W and action a ∈ [ n ] a \in[n] a ∈ [ n ] it holds that
inf f ∈ F ∣ ℓ i t 0 + s ( a ) − f ( s ) ∣ ≤ ε s , (7) \inf_{f \in\mathcal{F}}\left|\ell_i^{t_0+s}(a)-f(s)\right| \le\varepsilon_s, \tag*{(7)} f ∈ F inf ℓ i t 0 + s ( a ) − f ( s ) ≤ ε s , (7)
where ε s \varepsilon_s ε s measures the approximation error in local time s s s . By sliding all W W W -length windows over the horizon, Equations (6) and (7) ultimately result in
D i ( T ) ≲ C W 2 V i ( T ) + W ( 1 + ∑ t = 1 T ε t ) , (8) D_i(T) \lesssim\frac{C}{W^2}V_i(T) + W\left(1+\sum_{t=1}^{T}\varepsilon_t\right), \tag*{(8)} D i ( T ) ≲ W 2 C V i ( T ) + W ( 1 + t = 1 ∑ T ε t ) , (8)
which reduces the desired variation–variance inequality to choosing an approximating class F \mathcal{F} F for which we can control the ratio C / W 2 C/W^2 C / W 2 and the approximation errors. In the two subsections that follow, we outline how this framework can be instantiated in two settings in order to prove regret bounds for m m m -player and two-player games, respectively.
m m m players: a logarithmic regret bound
Theorem 3.3 gives a bound which matches the one of [21 ] , and is proved in full in § 4 . The theorem follows from Corollary 3.2 once we show that for each player i i i the loss trajectory satisfies
D i ( T ) ≤ 1 2 V i ( T ) + O ( log T ) , (9) D_i(T) \le\frac{1}{2}V_i(T) + O(\log T), \tag*{(9)} D i ( T ) ≤ 2 1 V i ( T ) + O ( log T ) , (9)
which we prove using the temporal approximation framework outlined in § 3.1 .
Theorem 3.3 (Logarithmic regret for m m m players). Let T ≥ 2 T \ge2 T ≥ 2 . If all players of an m m m -player game, with n n n actions for player i i i , run Optimistic Hedge with learning rate 0 < η ≤ 1 / ( 8704 m log T ) 0 < \eta\le1/(8704\sqrt{m}\log T) 0 < η ≤ 1/ ( 8704 m log T ) , then the regret of every player is bounded via
Reg i ( T ) ≤ log n η + 68 η n log T . \operatorname{Reg}_{i}(T) \le\frac{\log n}{\eta} + 68\eta n\log T. Reg i ( T ) ≤ η log n + 68 η n log T . In particular, for η = 1 / ( 8704 m log T ) \eta= 1/(8704\sqrt{m}\log T) η = 1/ ( 8704 m log T ) every player achieves regret at most O ( m log n log T ) O(\sqrt{m}\log n\log T) O ( m log n log T ) .
Losses are locally polynomial in time. We consider windows of size W = Θ ( log T ) W = \Theta(\log T) W = Θ ( log T ) and let
F = { q : R → R ∣ q is a polynomial of degree at most d } , \mathcal{F} = \{q : \mathbb{R} \to\mathbb{R} \mid q \text{ is a polynomial of degree at most } d\}, F = { q : R → R ∣ q is a polynomial of degree at most d } ,
where d = log 2 T d = \log_{2} T d = log 2 T . The following approximation result holds (see Lemma 4.1 ): if η ≲ 1 / ( m ( W + d ) ) \eta\lesssim1/(\sqrt{m}(W+d)) η ≲ 1/ ( m ( W + d )) , then for every 1 ≤ t 0 ≤ T − W 1 \le t_{0} \le T-W 1 ≤ t 0 ≤ T − W and a ∈ [ n ] a \in[n] a ∈ [ n ] ,
inf q ∈ F max 0 ≤ s ≤ W ∣ ℓ i t 0 + s ( a ) − q ( s ) ∣ ≤ 2 − d , (10) \inf_{q \in\mathcal{F}} \max_{0 \le s \le W} \left|\ell_{i}^{t_{0}+s}(a)-q(s)\right| \le2^{-d}, \tag*{(10)} q ∈ F inf 0 ≤ s ≤ W max ℓ i t 0 + s ( a ) − q ( s ) ≤ 2 − d , (10)
that is, the losses can be locally approximated by polynomials uniformly over the window. At a high level, the proof relies on expanding the loss trajectory over the given window as a power series in the variable z z z which replaces the learning rate η \eta η , i.e., we write ℓ i t 0 + s ( a ; z ) = ∑ k = 0 ∞ c i , a , k ( s ) z k \ell_{i}^{t_{0}+s}(a;z)=\sum_{k=0}^{\infty}c_{i,a,k}(s)z^{k} ℓ i t 0 + s ( a ; z ) = ∑ k = 0 ∞ c i , a , k ( s ) z k where the coefficients are functions of local time within the window, and the true trajectory is obtained by substituting z = η z=\eta z = η . A crucial property of Optimistic Hedge dynamics is that under this expansion, each coefficient c i , a , k ( s ) c_{i,a,k}(s) c i , a , k ( s ) coincides with a polynomial of degree at most k k k in local time s s s . Moreover, having chosen η \eta η sufficiently small, we can truncate this expansion after d d d terms resulting in an error which is exponentially small in d d d , i.e. ∣ ℓ i t 0 + s ( a ; z ) − ∑ k = 0 d c i , a , k ( s ) z k ∣ ≈ 2 − d \left|\ell_{i}^{t_{0}+s}(a;z)-\sum_{k=0}^{d}c_{i,a,k}(s)z^{k}\right|\approx2^{-d} ℓ i t 0 + s ( a ; z ) − ∑ k = 0 d c i , a , k ( s ) z k ≈ 2 − d . In a bit more detail, if we expand ℓ i t 0 + s ( a ; z ) \ell_{i}^{t_{0}+s}(a;z) ℓ i t 0 + s ( a ; z ) in powers of the real variable z z z , the truncation error of the series is exponentially small in d d d as long as ∑ k = 0 ∞ ∣ c i , a , k ( s ) ∣ z k \sum_{k=0}^{\infty}|c_{i,a,k}(s)|z^{k} ∑ k = 0 ∞ ∣ c i , a , k ( s ) ∣ z k is uniformly bounded over 0 ≤ z ≤ η 0 \le z \le\eta 0 ≤ z ≤ η , and indeed, this holds as long as η ≲ 1 / ( m ( W + d ) ) \eta\lesssim1/(m(W+d)) η ≲ 1/ ( m ( W + d )) which would result in a regret bound which is linear in m m m . The m \sqrt{m} m factor improvement comes from expanding ℓ i t 0 + s ( a ; z ) \ell_{i}^{t_{0}+s}(a;z) ℓ i t 0 + s ( a ; z ) in powers of the complex variable z z z , and using standard techniques from complex analysis, it suffices to prove that ∣ ℓ i t 0 + s ( a ; z ) ∣ \left|\ell_{i}^{t_{0}+s}(a;z)\right| ℓ i t 0 + s ( a ; z ) is uniformly bounded on the entire complex disk ∣ z ∣ ≤ η |z| \le\eta ∣ z ∣ ≤ η , which in turn holds as long as η ≲ 1 / ( m ( W + d ) ) \eta\lesssim1/(\sqrt{m}(W+d)) η ≲ 1/ ( m ( W + d )) .
Polynomials obey a variation–variance inequality. Having defined F \mathcal{F} F as the class of polynomials of degree at most d d d , it is not hard to prove the specific version of Equation (6) for this class, namely, for each q ∈ F q \in\mathcal{F} q ∈ F :
∑ s = W / 4 3 W / 4 − 1 ( q ( s + 1 ) − q ( s ) ) 2 ≤ 16 d ( d + 1 ) 3 W 2 ∑ s = 0 W q ( s ) 2 . (11) \sum_{s=W/4}^{3W/4-1}(q(s+1)-q(s))^{2} \le\frac{16d(d+1)}{3W^{2}}\sum_{s=0}^{W}q(s)^{2}. \tag*{(11)} s = W /4 ∑ 3 W /4 − 1 ( q ( s + 1 ) − q ( s ) ) 2 ≤ 3 W 2 16 d ( d + 1 ) s = 0 ∑ W q ( s ) 2 . (11)
Note that the left-hand side contains a summation over only the middle of the window, but this still suffices to obtain Equation (8) with C ≲ d 2 C \lesssim d^{2} C ≲ d 2 . The proof of Equation (11) relies on basic properties of the Legendre operator [41 ] , which in particular imply that if q q q is a polynomial of degree at most d d d , then for every 0 < γ < 1 2 0 < \gamma< \frac{1}{2} 0 < γ < 2 1 it holds that
∫ γ 1 − γ q ′ ( x ) 2 d x ≤ d ( d + 1 ) γ ( 1 − γ ) ∫ 0 1 q ( x ) 2 d x . (12) \int_{\gamma}^{1-\gamma}q'(x)^{2}\,dx \le\frac{d(d+1)}{\gamma(1-\gamma)}\int_{0}^{1}q(x)^{2}\,dx. \tag*{(12)} ∫ γ 1 − γ q ′ ( x ) 2 d x ≤ γ ( 1 − γ ) d ( d + 1 ) ∫ 0 1 q ( x ) 2 d x . (12)
which can be seen as a continuous analogue of Equation (11) .
Two players: a constant regret bound
We now outline the proof of our two-player upper bound:
Theorem 3.4 (Constant regret for two players). There is an absolute constant c > 0 c > 0 c > 0 such that in every two-player game G ∈ G ( 2 , n ) G \in\mathcal{G}(2,n) G ∈ G ( 2 , n ) , for every learning rate 0 < η ≤ c / n 0 < \eta\le c/n 0 < η ≤ c / n and every horizon T ≥ 1 T \ge1 T ≥ 1 , if both players run Optimistic Hedge with learning rate η \eta η , then the regret of each player i i i is bounded by
Reg i ( T ) ≤ log n η + η \operatorname{Reg}_{i}(T) \le\frac{\log n}{\eta} + \eta Reg i ( T ) ≤ η log n + η In particular, for η = c / n \eta= c/n η = c / n , both players have regret O ( n log n ) O(n\log n) O ( n log n ) for all T T T .
Setup. Consider a two-player game with loss matrices A , B ∈ [ 0 , 1 ] n × n A,B \in[0,1]^{n \times n} A , B ∈ [ 0 , 1 ] n × n : when player 1 plays action a a a and player 2 plays action b b b , their losses are A a b A_{ab} A ab and B a b B_{ab} B ab . The players’ strategies in round t t t are x 1 t , x 2 t x_1^t,x_2^t x 1 t , x 2 t and their loss vectors are ℓ 1 t = A x 2 t \ell_1^t = Ax_2^t ℓ 1 t = A x 2 t and ℓ 2 t = B T x 1 t \ell_2^t = B^\mathsf{T}x_1^t ℓ 2 t = B T x 1 t ; both players run Optimistic Hedge (3) . We analyze player 1 (the arguments are symmetric between the two players).
Let D 1 ( T ) D_1(T) D 1 ( T ) and V 1 ( T ) V_1(T) V 1 ( T ) be as in (4) . By Corollary 3.2 it suffices to prove the variation–variance inequality D 1 ( T ) ≤ 1 2 V 1 ( T ) + 1 2 D_1(T) \le\frac{1}{2}V_1(T) + \frac{1}{2} D 1 ( T ) ≤ 2 1 V 1 ( T ) + 2 1 , which yields the bound Reg 1 ( T ) ≤ log n / η + η / 3 \operatorname{Reg}_1(T) \le\log n/\eta+ \eta/3 Reg 1 ( T ) ≤ log n / η + η /3 ; for η ≤ c / n \eta\le c/n η ≤ c / n we in fact prove D 1 ( T ) ≤ 1 64 V 1 ( T ) + 1 2 D_1(T) \le\frac{1}{64}V_1(T) + \frac{1}{2} D 1 ( T ) ≤ 64 1 V 1 ( T ) + 2 1 (Lemma 5.4). As in the m m m -player case, we use the temporal approximation framework of § 3.1 , which relies on proving appropriate versions of Equations (6) and (7) .
Defining the class of simple functions F \mathcal{F} F : when the second player’s loss is "frozen". Keeping with the framework of § 3.1 , we define a class of simple functions F \mathcal{F} F . The functions in the class follow the losses of the first player that result assuming the second player plays Optimistic Hedge while “freezing” its own loss for a fixed window of 3 W + 2 3W+2 3 W + 2 iterates, from round t 0 − 1 t_0 - 1 t 0 − 1 to t 0 + 3 W t_0 + 3W t 0 + 3 W , for some starting round t 0 t_0 t 0 . We turn to examine the form of the losses for player 1 under this assumption. Let x 2 t 0 ( b ) x_2^{t_0}(b) x 2 t 0 ( b ) denote player 2’s probability for action b b b on round t 0 t_0 t 0 , and ℓ 2 t 0 ( b ) \ell_2^{t_0}(b) ℓ 2 t 0 ( b ) its frozen loss. We denote player 1’s losses over this window by ℓ ˉ 1 t 0 + s \bar{\ell}_1^{t_0+s} ℓ ˉ 1 t 0 + s and player 2’s probability vector by x ˉ 2 t 0 + s \bar{x}_2^{t_0+s} x ˉ 2 t 0 + s , where the local time s s s ranges from 0 to 3 W 3W 3 W (the bar notation highlights that player 2’s losses are frozen). In this case, player 2’s score 2 ℓ 2 t 0 + s − ℓ 2 t 0 + s − 1 2\ell_2^{t_0+s} - \ell_2^{t_0+s-1} 2 ℓ 2 t 0 + s − ℓ 2 t 0 + s − 1 equals ℓ 2 t 0 \ell_2^{t_0} ℓ 2 t 0 in every round of the window, so its probability distribution on iterate t 0 + s t_0+s t 0 + s is
x ˉ 2 t 0 + s ( b ) = x 2 t 0 ( b ) e − η ℓ 2 t 0 ( b ) s ∑ b ′ = 1 n x 2 t 0 ( b ′ ) e − η ℓ 2 t 0 ( b ′ ) s ( b ∈ [ n ] ) . \bar{x}_2^{t_0+s}(b) = \frac{x_2^{t_0}(b)e^{-\eta\ell_2^{t_0}(b)s}}{\sum_{b'=1}^{n}x_2^{t_0}(b')e^{-\eta\ell_2^{t_0}(b')s}} \qquad(b \in[n]). x ˉ 2 t 0 + s ( b ) = ∑ b ′ = 1 n x 2 t 0 ( b ′ ) e − η ℓ 2 t 0 ( b ′ ) s x 2 t 0 ( b ) e − η ℓ 2 t 0 ( b ) s ( b ∈ [ n ]) .
which implies that player 1’s loss on each action a ∈ [ n ] a \in[n] a ∈ [ n ] is
ℓ ˉ 1 t 0 + s ( a ) = ( A x ˉ 2 t 0 + s ) ( a ) = ∑ b = 1 n x 2 t 0 ( b ) e − η ℓ 2 t 0 ( b ) s A a b ∑ b = 1 n x 2 t 0 ( b ) e − η ℓ 2 t 0 ( b ) s . (13) \bar{\ell}_1^{t_0+s}(a) = \left(A\bar{x}_2^{t_0+s}\right)(a) = \frac{\sum_{b=1}^{n}x_2^{t_0}(b)e^{-\eta\ell_2^{t_0}(b)s}A_{ab}}{\sum_{b=1}^{n}x_2^{t_0}(b)e^{-\eta\ell_2^{t_0}(b)s}}. \tag*{(13)} ℓ ˉ 1 t 0 + s ( a ) = ( A x ˉ 2 t 0 + s ) ( a ) = ∑ b = 1 n x 2 t 0 ( b ) e − η ℓ 2 t 0 ( b ) s ∑ b = 1 n x 2 t 0 ( b ) e − η ℓ 2 t 0 ( b ) s A ab . (13)
Ignoring the denominator whose impact is negligible, for every action a a a the losses ℓ ˉ 1 t 0 + s ( a ) \bar{\ell}_1^{t_0+s}(a) ℓ ˉ 1 t 0 + s ( a ) , 0 ≤ s ≤ 3 W 0 \le s \le3W 0 ≤ s ≤ 3 W , are the values f ( s ) f(s) f ( s ) of a linear combination f f f of n n n exponentials, with coefficients β b = x 2 t 0 ( b ) A a b \beta_b = x_2^{t_0}(b)A_{ab} β b = x 2 t 0 ( b ) A ab and exponents λ b = − η ℓ 2 t 0 ( b ) \lambda_b = -\eta\ell_2^{t_0}(b) λ b = − η ℓ 2 t 0 ( b ) ; that is, f f f belongs to the class
F = { f : { 0 , 1 , … , 3 W } → R | f ( s ) = ∑ b = 1 n β b e λ b s for some β b , λ b ∈ R } . \mathcal{F} = \left\{f:\{0,1,\ldots,3W\}\to\mathbb{R}\ \middle|\ f(s)=\sum_{b=1}^{n}\beta_b e^{\lambda_b s}\text{ for some }\beta_b,\lambda_b\in\mathbb{R}\right\}. F = { f : { 0 , 1 , … , 3 W } → R f ( s ) = b = 1 ∑ n β b e λ b s for some β b , λ b ∈ R } .
A Bernstein inequality for exponential sums. The functions in F \mathcal{F} F satisfy the following discrete Bernstein-type inequality, which is a variant of Equation (6) : for every f ∈ F f \in\mathcal{F} f ∈ F ,
∑ s = W + 1 2 W ( f ( s ) − f ( s − 1 ) ) 2 ≲ n 2 W 2 ∑ s = 0 3 W f ( s ) 2 . (14) \sum_{s=W+1}^{2W} \left(f(s)-f(s-1)\right)^2 \lesssim\frac{n^2}{W^2}\sum_{s=0}^{3W} f(s)^2. \tag*{(14)} s = W + 1 ∑ 2 W ( f ( s ) − f ( s − 1 ) ) 2 ≲ W 2 n 2 s = 0 ∑ 3 W f ( s ) 2 . (14)
It follows from the continuous Bernstein inequality for exponential sums of Borwein and Erdélyi [5 ] (see Lemma 5.2). When we apply this to player 1’s loss vectors, we get, for every action a a a of player 1,
∑ t = t 0 + W + 1 t 0 + 2 W ( ℓ ˉ 1 t ( a ) − ℓ ˉ 1 t − 1 ( a ) ) 2 ≲ n 2 W 2 ∑ t = t 0 t 0 + 3 W ℓ ˉ 1 t ( a ) 2 . (15) \sum_{t=t_0+W+1}^{t_0+2W}\left(\bar{\ell}_1^t(a)-\bar{\ell}_1^{t-1}(a)\right)^2 \lesssim\frac{n^2}{W^2}\sum_{t=t_0}^{t_0+3W}\bar{\ell}_1^t(a)^2. \tag*{(15)} t = t 0 + W + 1 ∑ t 0 + 2 W ( ℓ ˉ 1 t ( a ) − ℓ ˉ 1 t − 1 ( a ) ) 2 ≲ W 2 n 2 t = t 0 ∑ t 0 + 3 W ℓ ˉ 1 t ( a ) 2 . (15)
In a similar way, while hiding some technicalities, one can get a similar bound, where instead of fixing an action a a a we take variances over the probability distribution x 1 t x_1^t x 1 t , which yields:
∑ t = t 0 + W + 1 t 0 + 2 W ∥ ℓ ˉ 1 t − ℓ ˉ 1 t − 1 ∥ x 1 t 2 ≲ n 2 W 2 ∑ t = t 0 t 0 + 3 W ∥ ℓ ˉ 1 t ∥ x 1 t 2 . (16) \sum_{t=t_0+W+1}^{t_0+2W}\left\|\bar{\ell}_1^t-\bar{\ell}_1^{t-1}\right\|_{x_1^t}^2 \lesssim\frac{n^2}{W^2}\sum_{t=t_0}^{t_0+3W}\left\|\bar{\ell}_1^t\right\|_{x_1^t}^2. \tag*{(16)} t = t 0 + W + 1 ∑ t 0 + 2 W ℓ ˉ 1 t − ℓ ˉ 1 t − 1 x 1 t 2 ≲ W 2 n 2 t = t 0 ∑ t 0 + 3 W ℓ ˉ 1 t x 1 t 2 . (16)
We build on the fact that since the step size is selected small enough compared to the window that we sum over, x 1 t x_1^t x 1 t does not change by much over this window. Therefore, Equation (16) is obtained from Equation (15) by summing over the actions a a a of player 1, with coefficients x 1 t 0 ( a ) x_1^{t_0}(a) x 1 t 0 ( a ) .
Accounting for the changing losses of Player 2. Equation (16) is taken consider player 2’s losses are frozen. When one accounts for the true losses of player 2, one obtains a slightly different inequality:
∑ t = t 0 + W + 1 t 0 + 2 W ∥ ℓ 1 t − ℓ 1 t − 1 ∥ x 1 t 2 ≲ n 2 W 2 ∑ t = t 0 t 0 + 3 W ∥ ℓ 1 t ∥ x 1 t 2 + η 2 n 2 W 2 ∑ r = t 0 t 0 + 3 W − 1 ∥ ℓ 2 r − ℓ 2 r − 1 ∥ ∞ 2 . (17) \sum_{t=t_0+W+1}^{t_0+2W}\left\|\ell_1^t-\ell_1^{t-1}\right\|_{x_1^t}^2 \lesssim\frac{n^2}{W^2}\sum_{t=t_0}^{t_0+3W}\left\|\ell_1^t\right\|_{x_1^t}^2+\eta^2n^2W^2\sum_{r=t_0}^{t_0+3W-1}\left\|\ell_2^r-\ell_2^{r-1}\right\|_\infty^2. \tag*{(17)} t = t 0 + W + 1 ∑ t 0 + 2 W ℓ 1 t − ℓ 1 t − 1 x 1 t 2 ≲ W 2 n 2 t = t 0 ∑ t 0 + 3 W ℓ 1 t x 1 t 2 + η 2 n 2 W 2 r = t 0 ∑ t 0 + 3 W − 1 ℓ 2 r − ℓ 2 r − 1 ∞ 2 . (17)
The error term is still in terms of player 2’s losses, but in a two-player game these move only if player 1 moves: ℓ 2 r − ℓ 2 r − 1 = B ⊤ ( x 1 r − x 1 r − 1 ) \ell_2^r-\ell_2^{r-1}=B^\top(x_1^r-x_1^{r-1}) ℓ 2 r − ℓ 2 r − 1 = B ⊤ ( x 1 r − x 1 r − 1 ) with B ∈ [ 0 , 1 ] n × n B \in[0,1]^{n\times n} B ∈ [ 0 , 1 ] n × n ,
∥ ℓ 2 r − ℓ 2 r − 1 ∥ ∞ ≤ ∥ x 1 r − x 1 r − 1 ∥ 1 . \left\|\ell_2^r-\ell_2^{r-1}\right\|_\infty\leq\left\|x_1^r-x_1^{r-1}\right\|_1. ℓ 2 r − ℓ 2 r − 1 ∞ ≤ x 1 r − x 1 r − 1 1 .
By definition of Optimistic Hedge the iterates of the first player move only if their (previous two) losses have variance with relation to their current mixed strategy distribution. Concretely, we may obtain the following bounds:
∥ x 1 r − x 1 r − 1 ∥ 1 ≤ 3 η ( ∥ ℓ 1 r − 1 ∥ x 1 r − 1 + ∥ ℓ 1 r − 2 ∥ x 1 r − 2 ) . \left\|x_1^r-x_1^{r-1}\right\|_1\leq3\eta\left(\left\|\ell_1^{r-1}\right\|_{x_1^{r-1}}+\left\|\ell_1^{r-2}\right\|_{x_1^{r-2}}\right). x 1 r − x 1 r − 1 1 ≤ 3 η ( ℓ 1 r − 1 x 1 r − 1 + ℓ 1 r − 2 x 1 r − 2 ) .
Squaring and substituting it into (17) expresses the error in terms of player 1’s own variances,
∑ t = t 0 + W + 1 t 0 + 2 W ∥ ℓ 1 t − ℓ 1 t − 1 ∥ x 1 t 2 ≲ n 2 W 2 ∑ t = t 0 t 0 + 3 W ∥ ℓ 1 t ∥ x 1 t 2 + η 4 n 2 W 2 ∑ r = t 0 − 1 t 0 + 3 W − 2 ( ∥ ℓ 1 r ∥ x 1 r 2 + ∥ ℓ 1 r − 1 ∥ x 1 r − 1 2 ) . (18) \sum_{t=t_0+W+1}^{t_0+2W}\left\|\ell_1^t-\ell_1^{t-1}\right\|_{x_1^t}^2\lesssim\frac{n^2}{W^2}\sum_{t=t_0}^{t_0+3W}\left\|\ell_1^t\right\|_{x_1^t}^2+\eta^4n^2W^2\sum_{r=t_0-1}^{t_0+3W-2}\left(\left\|\ell_1^r\right\|_{x_1^r}^2+\left\|\ell_1^{r-1}\right\|_{x_1^{r-1}}^2\right). \tag*{(18)} t = t 0 + W + 1 ∑ t 0 + 2 W ℓ 1 t − ℓ 1 t − 1 x 1 t 2 ≲ W 2 n 2 t = t 0 ∑ t 0 + 3 W ℓ 1 t x 1 t 2 + η 4 n 2 W 2 r = t 0 − 1 ∑ t 0 + 3 W − 2 ( ∥ ℓ 1 r ∥ x 1 r 2 + ℓ 1 r − 1 x 1 r − 1 2 ) . (18)
Setting η = Θ ( 1 / n ) \eta=\Theta(1/n) η = Θ ( 1/ n ) with a small enough constant and W = Θ ( n ) W=\Theta(n) W = Θ ( n ) with can roughly obtain
∑ t = t 0 + W + 1 t 0 + 2 W ∥ ℓ 1 t − ℓ 1 t − 1 ∥ x 1 t 2 ≤ 1 3 ∑ t = t 0 t 0 + 3 W ∥ ℓ 1 t ∥ x 1 t 2 . \sum_{t=t_0+W+1}^{t_0+2W}\left\|\ell_1^t-\ell_1^{t-1}\right\|_{x_1^t}^2\leq\frac{1}{3}\sum_{t=t_0}^{t_0+3W}\left\|\ell_1^t\right\|_{x_1^t}^2. t = t 0 + W + 1 ∑ t 0 + 2 W ℓ 1 t − ℓ 1 t − 1 x 1 t 2 ≤ 3 1 t = t 0 ∑ t 0 + 3 W ℓ 1 t x 1 t 2 .
After summing over T / W T/W T / W overlapping windows of size 3 W 3W 3 W each, we approximately get D 1 ( T ) ≤ V 1 ( T ) / 2 D_1(T)\leq V_1(T)/2 D 1 ( T ) ≤ V 1 ( T ) /2 , which is what needed in order to invoke Lemma 3.1 . This yields the regret bound R 1 ( T ) ≤ O ( log ( n ) / η ) = O ( n log n ) R_1(T)\leq O(\log(n)/\eta)=O(n\log n) R 1 ( T ) ≤ O ( log ( n ) / η ) = O ( n log n ) . By symmetry, this regret bound also holds for player 2.
Remark 3.5. For clarity, the sketch above works directly with the discrete iterates and applies the discrete Bernstein inequality (14) on each window. The formal proof differs in this respect: it works with continuous interpolations of the losses, obtained by running player 2’s exponential tilt continuously between consecutive rounds, so that the frozen losses become a ratio of exponential sums on the whole window. It then applies the continuous Bernstein inequality, and shows that the resulting integrals are close to the discrete sums that define D 1 ( T ) D_1(T) D 1 ( T ) and V 1 ( T ) V_1(T) V 1 ( T ) .
Why the bound for two players does not carry over to the > 2 >2 > 2 player case. In order to prove that D 1 ( T ) ≤ 1 64 V 1 ( T ) + 1 2 D_1(T) \le\frac{1}{64}V_1(T) + \frac{1}{2} D 1 ( T ) ≤ 64 1 V 1 ( T ) + 2 1 , we use the fact that the (original) loss vector iterates are sufficiently close to the iterates when the second player "freezes" their loss. However, as we discussed, the drift between the faithful and approximate continuous curves is dependent solely on the same player’s loss iterates (recall Equation (18) ). It is this dependency on the same player’s loss iterates , rather than any other player’s, which makes the "frozen" loss vector iterates sufficiently close to the original loss vector iterates.
The same-player dependency is due to the strictly coupled dynamics between the two players (Figure 1 (a)). The iterates ℓ 2 t \ell_2^t ℓ 2 t are dependent only on themselves and the iterates x 1 t x_1^t x 1 t , which are only dependent on themselves and the iterates ℓ 1 t \ell_1^t ℓ 1 t . As such, the change between keeping ℓ 2 t \ell_2^t ℓ 2 t constant and allowing ℓ 2 t \ell_2^t ℓ 2 t to progress is strictly dependent on how much the iterates ℓ 1 t \ell_1^t ℓ 1 t change. Using a similar syllogism, we find that the iterates ℓ 1 t \ell_1^t ℓ 1 t are also strictly dependent on themselves and ℓ 2 t \ell_2^t ℓ 2 t . Therefore, intuitively, it can be observed that the change in ℓ 1 t \ell_1^t ℓ 1 t from the original dynamics to when ℓ 2 t \ell_2^t ℓ 2 t is kept constant is strictly dependent upon the original ℓ 1 t \ell_1^t ℓ 1 t iterates themselves.
Figure 1. Dependencies between the players’ loss and mixed strategy iterates. (a) With two players the only route from ℓ 1 t \ell_1^t ℓ 1 t back to itself is the loop ℓ 1 → x 1 → ℓ 2 → x 2 → ℓ 1 \ell_1 \to x_1 \to\ell_2 \to x_2 \to\ell_1 ℓ 1 → x 1 → ℓ 2 → x 2 → ℓ 1 (orange). (b) A third player, with strategy x 3 t x_3^t x 3 t and loss ℓ 3 t \ell_3^t ℓ 3 t , also feeds ℓ 2 t \ell_2^t ℓ 2 t and is fed by x 2 t x_2^t x 2 t , so the coupled dynamics of players 2 and 3 (violet) enter player 1’s loop.
This is not the case, however, in games with more than two players (Figure 1 (b)). Any player i i i ’s loss iterates are dependent on the other player’s mixed strategies, which are dependent on their own losses. However, if we were to try and "freeze" players’ losses for a similar proof technique, we find instead that the other player’s losses are dependent on each other . This co-dependency means that the drift from the original loss player i i i iterates to the iterates when the rest of the player’s losses are "frozen" has to account for the codynamics between the other players in the original iterates; these codynamics add unavoidable terms dependent on the other players to the drift.
Three players: oscillators and frequency doubling
Theorem 3.6 (An exponential barrier for three players). There are absolute constants C 0 C_0 C 0 and η 0 > 0 \eta_0 > 0 η 0 > 0 such that for every n ≥ C 0 n \ge C_0 n ≥ C 0 and every learning rate η ∈ ( 0 , η 0 ] \eta\in(0,\eta_0] η ∈ ( 0 , η 0 ] there is a three-player game G ∈ G ( 3 , n ) G \in\mathcal{G}(3,n) G ∈ G ( 3 , n ) in which, when all players run Optimistic Hedge with learning rate η \eta η starting from the uniform distribution,
sup T Reg 3 ( T ) ≥ 2 n / 8 log n C 0 n − 1 = 2 Ω ( n ) . \sup_T \operatorname{Reg}_3(T) \ge\frac{2^{n/8}\log n}{C_0 n} - 1 = 2^{\Omega(n)}. T sup Reg 3 ( T ) ≥ C 0 n 2 n /8 log n − 1 = 2 Ω ( n ) . In particular inf 0 < η ≤ η 0 sup T ≥ 1 sup G ∈ G ( 3 , n ) max i Reg i ( T ) ≥ 2 n / 8 log n / ( C 0 n ) − 1 \inf_{0<\eta\le\eta_0}\sup_{T\ge1}\sup_{G\in\mathcal{G}(3,n)}\max_i \operatorname{Reg}_i(T) \ge2^{n/8}\log n/(C_0n) - 1 inf 0 < η ≤ η 0 sup T ≥ 1 sup G ∈ G ( 3 , n ) max i Reg i ( T ) ≥ 2 n /8 log n / ( C 0 n ) − 1 . Moreover, if η ≥ C 0 n 2 − n / 8 \eta\ge C_0 n 2^{-n/8} η ≥ C 0 n 2 − n /8 , then G G G can be chosen so that player 3’s regret grows linearly: Reg 3 ( T ) ≥ c T − C \operatorname{Reg}_3(T) \ge cT-C Reg 3 ( T ) ≥ c T − C for all T T T , for some c = c ( G , η ) > 0 c=c(G,\eta)>0 c = c ( G , η ) > 0 and C = C ( G , η ) C=C(G,\eta) C = C ( G , η ) .
The two regimes are of different nature. For η < C 0 n 2 − n / 8 \eta< C_0n2^{-n/8} η < C 0 n 2 − n /8 the learning rate is so small that even a game with losses which are independent of the other players’ actions already forces log n / η \log n/\eta log n / η regret (Proposition A.3 ). For larger η \eta η we construct a game in which player 3’s regret is unbounded; the rest of this subsection sketches this construction. We first describe the construction for three players, which gives the 2 Ω ( n ) 2^{\Omega(n)} 2 Ω ( n ) lower bound, and only at the end of this subsection explain how it extends to m m m players, with a lower bound of 2 Ω ( m n ) 2^{\Omega(mn)} 2 Ω ( mn ) .
The idea. The game has three players. Players 1 and 2 play against each other and ignore player 3, and their utilities are designed so that Optimistic Hedge never converges: their strategies settle into a cycle and follow it forever. That learning dynamics in games may cycle rather than converge is well known [29 , 6 ] , and the last-iterate behavior of optimistic dynamics has been studied in detail [14 , 43 , 9 ] ; to our knowledge, turning such a cycle into a regret lower bound is new. Player 3, the observer , has losses that depend only on player 1’s action, so the losses they see oscillate too. Optimistic Hedge bets that the next loss will resemble the last one. If the oscillation is fast enough, this bet is systematically wrong, and the observer’s regret grows linearly. The full proof is in § 7 .
Fast oscillations hurt the observer. Split the observer’s actions into two groups, + + + and − - − , and suppose that the advantage f t f_t f t of group + + + over group − - − oscillates with period four: f t = α , 0 , − α , 0 , … f_t = \alpha, 0, -\alpha, 0, \ldots f t = α , 0 , − α , 0 , … . Optimistic Hedge reacts to the last loss twice, once through the cumulative loss and once more through its prediction. As a result the observer favors + + + most one round after it was good, and still favors it in the next round, when it is bad (Figure 2 ). She loses a fixed amount in every period, while either group alone breaks even, so their regret grows linearly. For slow oscillations the prediction is accurate and helps instead, which is what makes the upper bounds possible. It therefore suffices that players 1 and 2 oscillate with period four.
Figure 2. Illustration of the induced loss sequence of the observer.
A cascade of blocks. Getting players 1 and 2 to oscillate this fast is the main difficulty. A small learning rate changes the probabilities only slightly in each round, so an oscillation of two players is necessarily slow, with a period of order at least 1 / η 1/\eta 1/ η . We therefore build the fast oscillation in stages (Figure 3 ). The actions of players 1 and 2 are split into k k k blocks of eight actions. Block 1 oscillates on its own, slowly. Each later block is driven by the block before it and oscillates with half its period, so after about log 2 ( 1 / η ) \log_2(1/\eta) log 2 ( 1/ η ) blocks the period is four, and the last block drives the observer. Each halving costs eight actions, which is the source of the exponential dependence on n n n .
Figure 3. The three-player game. Block 1 of players 1 and 2 rotates slowly, with period M = 2 k + 1 M = 2^{k+1} M = 2 k + 1 (§ 7.1 ). Each later block sees the square of its predecessor’s displacement and so rotates twice as fast (§ 9.1 ). Block k k k has period 4, fast enough to give the observer linear regret (§ 10.1 ). All blocks live in the same two players (§ 10.5 ).
One block’s rotation. Fix a block j j j . Its eight actions are identified with points w w w of the complex plane: the four corners and the four edge midpoints of a square around 0. The game is designed so that the players’ strategy on block j j j is described by a single complex number, the block’s score z j , t z_{j,t} z j , t : action w w w is played with probability proportional to e Re ( w z j , t ) e^{\operatorname{Re}(wz_{j,t})} e Re ( w z j , t ) , which is larger the more z j , t z_{j,t} z j , t points toward w w w . Along the cycle, the score rotates around a small circle (Figure 4 (a)). As it turns, it points toward each action in turn, so each action’s probability rises and falls at the same speed as
Figure 4. Frequency doubling from block j j j to block j + 1 j+1 j + 1 (§ 3.4 ), drawn schematically over one period P P P of block j j j ; the actual oscillations are much smaller. (a) The score z j , t z_{j,t} z j , t . (b) The probabilities of the actions ± 1 \pm1 ± 1 , ± i \pm i ± i ; the four corner actions behave alike. (c) The combination Q t = [ p t ( 1 ) + p t ( − 1 ) ] − [ p t ( i ) + p t ( − i ) ] + i 2 [ p t ( 1 + i ) + p t ( − 1 − i ) ] − i 2 [ p t ( 1 − i ) + p t ( − 1 + i ) ] ≈ z j , t 2 / n Q_t = [p_t(1)+p_t(-1)]-[p_t(i)+p_t(-i)]+\frac{i}{2}[p_t(1+i)+p_t(-1-i)]-\frac{i}{2}[p_t(1-i)+p_t(-1+i)] \approx z_{j,t}^{2}/n Q t = [ p t ( 1 ) + p t ( − 1 )] − [ p t ( i ) + p t ( − i )] + 2 i [ p t ( 1 + i ) + p t ( − 1 − i )] − 2 i [ p t ( 1 − i ) + p t ( − 1 + i )] ≈ z j , t 2 / n , which follows by expanding p t ( w ) ∝ e Re ( w ‾ z j , t ) p_t(w) \propto e^{\operatorname{Re}(\overline{w}z_{j,t})} p t ( w ) ∝ e Re ( w z j , t ) to second order.
the score rotates (Figure 4 (b)). In block 1 this rotation is produced by the block’s own utilities; in every later block it is driven by the block before it, as follows.
Halving the period. Opposite actions rise and fall in antiphase, so adding up their probabilities cancels the oscillation to first order, and what remains oscillates twice as fast. In terms of the score, a fixed linear combination of the probabilities of block j j j equals the square of the score, to leading order, and squaring a rotation doubles its speed (Figure 4 (c)). The utilities of block j + 1 j+1 j + 1 depend on the actions of block j j j through exactly this combination. The score of block j + 1 j+1 j + 1 is therefore pushed around a circle twice as fast as the score of block j j j , so its period is half as long.
Technical subtleties. This picture would be exact if the dynamics of the scores were linear. The softmax makes the actual dynamics nonlinear, and the proof has to deal with three issues.
An exact cycle. The first block only rotates along a neutral direction, where small nonlinear errors are not damped and would accumulate over time. The implicit function theorem gives an exactly periodic orbit, after a small adjustment of the block’s utilities.
Convergence from the start. The cycle does not attract every nearby starting point. The stable manifold theorem gives a manifold of starting points, the stable manifold of the cycle, from which the dynamics converge to the cycle. Using the implicit function theorem again, we adjust the game slightly, by shifting the center of the cycle, so that the uniform starting point lies on this stable manifold.
The later blocks. Driving a block only by its predecessor does not make it converge to a cycle. We therefore modify its losses slightly: besides the term coming from the previous block, each block has a small self-interaction term that depends on its own actions. This term makes the block’s dynamics contracting, so it settles into an exact cycle even though its input is only approximately periodic, and the remaining nonlinear errors do not accumulate.
More than three players. With m m m players the same mechanism gives a barrier 2 Ω ( m n ) 2^{\Omega(mn)} 2 Ω ( mn ) (the Theorem 6.24 ). All players except the last are grouped into ⌊ ( m − 1 ) / 2 ⌋ \lfloor(m-1)/2 \rfloor ⌊( m − 1 ) /2 ⌋ pairs, and each pair carries about n / 8 n/8 n /8 blocks of its own, so the cascade has about m n / 16 mn/16 mn /16 blocks in total. The first block of each pair is driven by the last block of the previous pair, in the same way as a block is driven by its predecessor within a pair, and player m m m observes the last block of the last pair. Since a pair is never influenced by the pairs after it, the convergence argument carries over unchanged, and every additional pair halves the period about n / 8 n/8 n /8 more times.
Proof of the O ( m log n log T ) O(\sqrt{m}\log n\log T) O ( m log n log T ) regret upper bound for m m m players
This section includes a fool proof of Theorem 3.3 . The proof consists of three main parts: a local polynomial approximation lemma (§ 4.1 ), the discrete variation–variance inequality for polynomials (§ 4.2 ), and a sliding-window lemma which gives the variation–variance inequality over the entire horizon (§ 4.3 ). § 4.4 completes the proof.
Local polynomial approximation
Lemma 4.1 (Local approximation). Let d ≥ 0 d \ge0 d ≥ 0 and W ≥ 1 W \ge1 W ≥ 1 be integers, and suppose that all players run (3) with
0 < η ≤ 1 256 m ( W + d ) . 0 < \eta\le\frac{1}{256\sqrt{m}(W+d)}. 0 < η ≤ 256 m ( W + d ) 1 . Then for every player i i i , every integer r r r with d + 1 ≤ r ≤ T − W d+1 \le r \le T-W d + 1 ≤ r ≤ T − W and every action a ∈ [ n ] a \in[n] a ∈ [ n ] there is a polynomial q i , r , a : R → R q_{i,r,a}:\mathbb{R}\to\mathbb{R} q i , r , a : R → R of degree at most d d d such that
max s ∈ { 0 , … , W } ∣ ℓ i r + s ( a ) − q i , r , a ( s ) ∣ ≤ 2 − d . (19) \max_{s\in\{0,\ldots,W\}}\left|\ell_i^{r+s}(a)-q_{i,r,a}(s)\right|\le2^{-d}. \tag*{(19)} s ∈ { 0 , … , W } max ℓ i r + s ( a ) − q i , r , a ( s ) ≤ 2 − d . (19) Proof. Fix r r r , put t 0 = r − d ≥ 1 t_0=r-d\ge1 t 0 = r − d ≥ 1 and N = W + d N=W+d N = W + d , and freeze the state of the dynamics at round t 0 t_0 t 0 : let p j = x j t 0 p_j=x_j^{t_0} p j = x j t 0 and v j = ℓ j t 0 − 1 v_j=\ell_j^{t_0-1} v j = ℓ j t 0 − 1 for every player j j j .
Step 1: defining the auxiliary trajectories. For a complex number z z z define, simultaneously for all players and for local times u = 0 , 1 , … , N u=0,1,\ldots,N u = 0 , 1 , … , N ,
Y j 0 = 0 , Λ j − 1 = v j , D j u ( z ) = ∑ a p j ( a ) e Y j u ( a ; z ) , X j u ( a ; z ) = p j ( a ) e Y j u ( a ; z ) D j u ( z ) , Λ j u ( z ) = ℓ j ( X u ( z ) ) , Y j u + 1 ( z ) = Y j u ( z ) − z ( 2 Λ j u ( z ) − Λ j u − 1 ( z ) ) ( u < N ) . (20) \begin{aligned}
Y_j^0&=0, & \Lambda_j^{-1}&=v_j, & D_j^u(z)&=\sum_a p_j(a)e^{Y_j^u(a;z)}, & X_j^u(a;z)&=\frac{p_j(a)e^{Y_j^u(a;z)}}{D_j^u(z)},\\
\Lambda_j^u(z)&=\ell_j(X^u(z)), & Y_j^{u+1}(z)&=Y_j^u(z)-z\left(2\Lambda_j^u(z)-\Lambda_j^{u-1}(z)\right) && (u<N). \tag*{(20)}
\end{aligned} Y j 0 Λ j u ( z ) = 0 , = ℓ j ( X u ( z )) , Λ j − 1 Y j u + 1 ( z ) = v j , = Y j u ( z ) − z ( 2 Λ j u ( z ) − Λ j u − 1 ( z ) ) D j u ( z ) = a ∑ p j ( a ) e Y j u ( a ; z ) , ( u < N ) . X j u ( a ; z ) = D j u ( z ) p j ( a ) e Y j u ( a ; z ) , (20) Since D j u ( 0 ) = 1 D_j^u(0)=1 D j u ( 0 ) = 1 , induction over u u u shows that all these quantities are analytic functions of z z z on some neighborhood of 0 0 0 , and Step 3 identifies a disk on which they are all defined. Substituting the score update into the formula for X j u + 1 X_j^{u+1} X j u + 1 shows that X j u + 1 = E X j u [ − z ( 2 Λ j u − Λ j u − 1 ) ] X_j^{u+1}= \mathbb{E}_{X_j^u}[-z(2\Lambda_j^u-\Lambda_j^{u-1})] X j u + 1 = E X j u [ − z ( 2 Λ j u − Λ j u − 1 )] , which for z = η z=\eta z = η is precisely the update (3) with the same losses. Since X j 0 = p j = x j t 0 X_j^0=p_j=x_j^{t_0} X j 0 = p j = x j t 0 and Λ j − 1 = v j = ℓ j t 0 − 1 \Lambda_j^{-1}=v_j=\ell_j^{t_0-1} Λ j − 1 = v j = ℓ j t 0 − 1 , induction gives
X j u ( η ) = x j t 0 + u , Λ j u ( η ) = ℓ j t 0 + u ( 0 ≤ u ≤ N ) . (21) X_j^u(\eta)=x_j^{t_0+u},\qquad\Lambda_j^u(\eta)=\ell_j^{t_0+u}\qquad(0\le u\le N). \tag*{(21)} X j u ( η ) = x j t 0 + u , Λ j u ( η ) = ℓ j t 0 + u ( 0 ≤ u ≤ N ) . (21) that is, at z = η z=\eta z = η we obtain the actual trajectory exactly. At z = 0 z=0 z = 0 nothing moves: X j u ( 0 ) = p j X_j^u(0)=p_j X j u ( 0 ) = p j and Λ j u ( 0 ) = ℓ j ( p ) \Lambda_j^u(0)=\ell_j(p) Λ j u ( 0 ) = ℓ j ( p ) for all $u\ge 0.
Step 2: the coefficient of z k z^k z k is a polynomial of degree k k k in local time. Write
Y j u ( z ) = ∑ k ≥ 1 y j , k ( u ) z k , X j u ( z ) = ∑ k ≥ 0 ξ j , k ( u ) z k , Λ j u ( z ) = ∑ k ≥ 0 c j , k ( u ) z k . Y_j^u(z)=\sum_{k\ge1} y_{j,k}(u)z^k,\qquad X_j^u(z)=\sum_{k\ge0}\xi_{j,k}(u)z^k,\qquad\Lambda_j^u(z)=\sum_{k\ge0}c_{j,k}(u)z^k. Y j u ( z ) = k ≥ 1 ∑ y j , k ( u ) z k , X j u ( z ) = k ≥ 0 ∑ ξ j , k ( u ) z k , Λ j u ( z ) = k ≥ 0 ∑ c j , k ( u ) z k .
We claim that for each k ≤ d k \le d k ≤ d there are polynomials y ^ j , k , ξ ^ j , k , c ^ j , k \hat{y}_{j,k},\hat{\xi}_{j,k},\hat{c}_{j,k} y ^ j , k , ξ ^ j , k , c ^ j , k of degree at most k k k (coordinatewise) that agree with y j , k , ξ j , k , c j , k y_{j,k},\xi_{j,k},c_{j,k} y j , k , ξ j , k , c j , k at every integer u u u with k ≤ u ≤ N k \le u \le N k ≤ u ≤ N . We prove this by induction on k k k , simultaneously for all players. For k = 0 k=0 k = 0 the coefficients are the constants p j p_j p j and ℓ j ( p ) \ell_j(p) ℓ j ( p ) . Suppose the claim holds below order k k k . Extracting the coefficient of z k z^k z k from (20) gives
y j , k ( u + 1 ) − y j , k ( u ) = − 2 c j , k − 1 ( u ) + c j , k − 1 ( u − 1 ) . (22) y_{j,k}(u+1)-y_{j,k}(u)=-2c_{j,k-1}(u)+c_{j,k-1}(u-1). \tag*{(22)} y j , k ( u + 1 ) − y j , k ( u ) = − 2 c j , k − 1 ( u ) + c j , k − 1 ( u − 1 ) . (22)
For k ≤ u < N k \le u<N k ≤ u < N both arguments on the right are at least k − 1 k-1 k − 1 , so the right-hand side coincides with the polynomial H ( u ) = − 2 c ^ j , k − 1 ( u ) + c ^ j , k − 1 ( u − 1 ) H(u)=-2\hat{c}_{j,k-1}(u)+\hat{c}_{j,k-1}(u-1) H ( u ) = − 2 c ^ j , k − 1 ( u ) + c ^ j , k − 1 ( u − 1 ) of degree at most k − 1 k-1 k − 1 . Expand H H H in the binomial basis, H ( u ) = ∑ h < k α h ( u h ) H(u)=\sum_{h<k}\alpha_h\binom{u}{h} H ( u ) = ∑ h < k α h ( h u ) , and set
y ^ j , k ( u ) = y j , k ( k ) + ∑ h < k α h [ ( u h + 1 ) − ( k h + 1 ) ] . \hat{y}_{j,k}(u)=y_{j,k}(k)+\sum_{h<k}\alpha_h\left[\binom{u}{h+1}-\binom{k}{h+1}\right]. y ^ j , k ( u ) = y j , k ( k ) + h < k ∑ α h [ ( h + 1 u ) − ( h + 1 k ) ] .
By Pascal’s rule ( u + 1 h + 1 ) − ( u h + 1 ) = ( u h ) \binom{u+1}{h+1}-\binom{u}{h+1}=\binom{u}{h} ( h + 1 u + 1 ) − ( h + 1 u ) = ( h u ) , this polynomial has degree at most k k k , increments H ( u ) H(u) H ( u ) , and the correct value at u = k u=k u = k ; telescoping (22) from u = k u=k u = k shows that it agrees with y j , k ( u ) y_{j,k}(u) y j , k ( u ) for all integers k ≤ u ≤ N k \le u \le N k ≤ u ≤ N .
The coefficient of z k z^k z k in e Y j u ( a ; z ) e^{Y_j^u(a;z)} e Y j u ( a ; z ) is
A j , k ( u , a ) = ∑ q ≥ 1 1 q ! ∑ h 1 + ⋯ + h q = k h ν ≥ 1 ∏ ν y j , h ν ( u , a ) , A_{j,k}(u,a)=\sum_{q\ge1}\frac{1}{q!}\sum_{\substack{h_1+\cdots+h_q=k\\h_\nu\ge1}}\prod_\nu y_{j,h_\nu}(u,a), A j , k ( u , a ) = q ≥ 1 ∑ q ! 1 h 1 + ⋯ + h q = k h ν ≥ 1 ∑ ν ∏ y j , h ν ( u , a ) ,
and replacing each y j , h y_{j,h} y j , h by y ^ j , h \hat{y}_{j,h} y ^ j , h gives a polynomial of degree at most k k k that agrees with it for u ≥ k u\ge k u ≥ k , since every h ν ≤ k ≤ u h_\nu\le k\le u h ν ≤ k ≤ u . Setting
B j , k ( u ) = ∑ a p j ( a ) A j , k ( u , a ) , B j , 0 = 1 , B_{j,k}(u)=\sum_a p_j(a)A_{j,k}(u,a),\qquad B_{j,0}=1, B j , k ( u ) = a ∑ p j ( a ) A j , k ( u , a ) , B j , 0 = 1 ,
then B j , k B_{j,k} B j , k has degree at most k k k and agrees with the coefficient of z k z^k z k in D j u D_j^u D j u for u ≥ k u\ge k u ≥ k . Defining recursively
E j , k ( u ) = − ∑ h = 1 k B j , h ( u ) E j , k − h ( u ) , E j , 0 = 1 , E_{j,k}(u)=-\sum_{h=1}^{k}B_{j,h}(u)E_{j,k-h}(u),\qquad E_{j,0}=1, E j , k ( u ) = − h = 1 ∑ k B j , h ( u ) E j , k − h ( u ) , E j , 0 = 1 ,
by the recursive definition of the formal coefficients of ( D j u ) − 1 (D_j^u)^{-1} ( D j u ) − 1 we deduce that E j , k E_{j,k} E j , k has degree at most k k k and agrees with the coefficient of z k z^k z k in the expansion of ( D j u ) − 1 (D_j^u)^{-1} ( D j u ) − 1 for all u ≥ k u\ge k u ≥ k . Therefore, defining
ξ ^ j , k ( u , a ) = p j ( a ) ∑ h = 0 k A j , h ( u , a ) E j , k − h ( u ) , \hat{\xi}_{j,k}(u,a)=p_j(a)\sum_{h=0}^{k}A_{j,h}(u,a)E_{j,k-h}(u), ξ ^ j , k ( u , a ) = p j ( a ) h = 0 ∑ k A j , h ( u , a ) E j , k − h ( u ) ,
gives a polynomial of degree at most k k k which agrees with ξ j , k ( u , a ) \xi_{j,k}(u,a) ξ j , k ( u , a ) for all $k\le u\le N. To obtain the coefficients for the losses, define
c ^ i , k ( u , a ) = ∑ a − i L i ( a , a − i ) ∑ h j ≥ 0 j ≠ i ∑ j ≠ i h j = k ∏ j ≠ i ξ ^ j , h j ( u , a j ) , \hat{c}_{i,k}(u,a)=\sum_{a_{-i}}L_i(a,a_{-i})\sum_{\substack{h_j\ge0\\j\ne i\\\sum_{j\ne i}h_j=k}}\prod_{j\ne i}\hat{\xi}_{j,h_j}(u,a_j), c ^ i , k ( u , a ) = a − i ∑ L i ( a , a − i ) h j ≥ 0 j = i ∑ j = i h j = k ∑ j = i ∏ ξ ^ j , h j ( u , a j ) ,
where the second summation runs over h j ≥ 0 h_j\ge0 h j ≥ 0 (j ≠ i j\ne i j = i ) with ∑ j ≠ i h j = k \sum_{j\ne i}h_j=k ∑ j = i h j = k . It is straightforward to see that this is a polynomial of degree at most k k k which agrees with the loss coefficients for all u ≥ k u\ge k u ≥ k .
Step 3: a disk of analyticity and the Taylor tail. Let R = 1 / ( 64 m N ) R = 1/(64\sqrt{m}N) R = 1/ ( 64 m N ) , b = 1 / ( 8 m ) b = 1/(8\sqrt{m}) b = 1/ ( 8 m ) and M = e 1 / 64 < 2 M = e^{1/64} < 2 M = e 1/64 < 2 . We show by induction on u ≤ N u \le N u ≤ N that all quantities in (20) up to local time u u u are analytic on a neighborhood of the closed disk { ∣ z ∣ ≤ R } \{|z| \le R\} { ∣ z ∣ ≤ R } and satisfy
max j , a ∣ Y j u ( a ; z ) ∣ ≤ 6 u R < b , max j , a ∣ Λ j u ( a ; z ) ∣ ≤ M . (23) \max_{j,a}|Y_j^u(a;z)| \le6uR < b,\qquad\max_{j,a}|\Lambda_j^u(a;z)| \le M. \tag*{(23)} j , a max ∣ Y j u ( a ; z ) ∣ ≤ 6 u R < b , j , a max ∣ Λ j u ( a ; z ) ∣ ≤ M . (23)
At u = 0 u = 0 u = 0 all quantities are constant, and the losses, including Λ j − 1 = v j \Lambda_j^{-1} = v_j Λ j − 1 = v j , have entries in [ 0 , 1 ] [0,1] [ 0 , 1 ] . Suppose ∣ Y j u ∣ ≤ 6 u R < b |Y_j^u| \le6uR < b ∣ Y j u ∣ ≤ 6 u R < b holds at time u u u . Then ∣ Im Y j u ∣ < b ≤ 1 8 |\operatorname{Im}Y_j^u| < b \le\frac{1}{8} ∣ Im Y j u ∣ < b ≤ 8 1 , so by Lemma 4.4 D j u ( z ) ≠ 0 D_j^u(z) \ne0 D j u ( z ) = 0 on the disk; more precisely Re D j u ≥ e − b cos b > 0 \operatorname{Re}D_j^u \ge e^{-b}\cos b > 0 Re D j u ≥ e − b cos b > 0 , so by continuity and compactness D j u D_j^u D j u is nonzero on a neighborhood of the disk, on which X u X^u X u and Λ j u \Lambda_j^u Λ j u are therefore analytic, and Lemma 4.4 gives ∣ Λ j u ∣ ≤ e ( m − 1 ) b 2 ≤ e 1 / 64 = M |\Lambda_j^u| \le e^{(m-1)b^2} \le e^{1/64} = M ∣ Λ j u ∣ ≤ e ( m − 1 ) b 2 ≤ e 1/64 = M . The score update then gives ∣ Y j u + 1 ∣ ≤ ∣ Y j u ∣ + ∣ z ∣ ( 2 ∣ Λ j u ∣ + ∣ Λ j u − 1 ∣ ) ≤ 6 u R + 6 R |Y_j^{u+1}| \le|Y_j^u|+|z|(2|\Lambda_j^u|+|\Lambda_j^{u-1}|) \le6uR+6R ∣ Y j u + 1 ∣ ≤ ∣ Y j u ∣ + ∣ z ∣ ( 2∣ Λ j u ∣ + ∣ Λ j u − 1 ∣ ) ≤ 6 u R + 6 R , and 6 N R = 3 32 m < b 6NR = \frac{3}{32\sqrt{m}} < b 6 N R = 32 m 3 < b closes the induction. Intersecting the finitely many neighborhoods involved, all quantities are analytic on one neighborhood of the disk. By Cauchy’s estimate, ∣ c i , k ( u , a ) ∣ ≤ 2 R − k |c_{i,k}(u,a)| \le2R^{-k} ∣ c i , k ( u , a ) ∣ ≤ 2 R − k for all k k k , and since η ≤ 1 / ( 256 m N ) = R / 4 \eta\le1/(256\sqrt{m}N) = R/4 η ≤ 1/ ( 256 m N ) = R /4 , Lemma 4.5 gives, uniformly in u ≤ N u \le N u ≤ N and a a a ,
∣ Λ i u ( a ; η ) − ∑ k = 0 d c i , k ( u , a ) η k ∣ ≤ 2 ∑ k > d 4 − k = 2 3 ⋅ 4 − d ≤ 2 − d . (24) \left|\Lambda_i^u(a;\eta)-\sum_{k=0}^{d}c_{i,k}(u,a)\eta^k\right|
\le2\sum_{k>d}4^{-k}
=\frac{2}{3}\cdot4^{-d}
\le2^{-d}. \tag*{(24)} Λ i u ( a ; η ) − k = 0 ∑ d c i , k ( u , a ) η k ≤ 2 k > d ∑ 4 − k = 3 2 ⋅ 4 − d ≤ 2 − d . (24)
Step 4: the polynomial. Define q i , r , a ( s ) = ∑ k ≤ d c ^ i , k ( d + s , a ) q_{i,r,a}(s)=\sum_{k\le d}\hat{c}_{i,k}(d+s,a) q i , r , a ( s ) = ∑ k ≤ d c ^ i , k ( d + s , a ) , which is a polynomial of degree at most d d d in s s s with real coefficients. For every integer 0 ≤ s ≤ W 0\le s\le W 0 ≤ s ≤ W the local time u = d + s u=d+s u = d + s satisfies k ≤ u ≤ N k\le u\le N k ≤ u ≤ N for all k ≤ d k\le d k ≤ d , so q i , r , a ( s ) = ∑ k ≤ d c i , k ( d + s , a ) η k q_{i,r,a}(s)=\sum_{k\le d}c_{i,k}(d+s,a)\eta^k q i , r , a ( s ) = ∑ k ≤ d c i , k ( d + s , a ) η k , while ℓ i r + s = Λ i d + s ( η ) \ell_i^{r+s}=\Lambda_i^{d+s}(\eta) ℓ i r + s = Λ i d + s ( η ) by (21) because t 0 + d + s = r + s t_0+d+s=r+s t 0 + d + s = r + s . Inequality (24) completes the proof.
Polynomial variation–variance inequality
Lemma 4.2 (Variation–variance inequality for polynomials). Let W ≥ 4 W\ge4 W ≥ 4 be divisible by four, let 0 ≤ d ≤ W 0\le d\le W 0 ≤ d ≤ W , and let I W = { W / 4 , W / 4 + 1 , … , 3 W / 4 − 1 } I_W=\{W/4,W/4+1,\ldots,3W/4-1\} I W = { W /4 , W /4 + 1 , … , 3 W /4 − 1 } . Then every real polynomial q : R → R q:\mathbb{R}\to\mathbb{R} q : R → R of degree at most d d d satisfies
∑ s ∈ I W ( q ( s + 1 ) − q ( s ) ) 2 ≤ 16 d ( d + 1 ) 3 W 2 ∑ s = 0 W q ( s ) 2 . (25) \sum_{s\in I_W}(q(s+1)-q(s))^2\le\frac{16d(d+1)}{3W^2}\sum_{s=0}^{W}q(s)^2. \tag*{(25)} s ∈ I W ∑ ( q ( s + 1 ) − q ( s ) ) 2 ≤ 3 W 2 16 d ( d + 1 ) s = 0 ∑ W q ( s ) 2 . (25)
In particular, if each entry of q : R → R n q:\mathbb{R}\to\mathbb{R}^n q : R → R n is a polynomial of degree at most d d d and p ∈ Δ n p\in\Delta_n p ∈ Δ n ,
∑ s ∈ I W ∥ q ( s + 1 ) − q ( s ) ∥ p 2 ≤ 16 d ( d + 1 ) 3 W 2 ∑ s = 0 W ∥ q ( s ) ∥ p 2 . (26) \sum_{s\in I_W}\|q(s+1)-q(s)\|_p^2\le\frac{16d(d+1)}{3W^2}\sum_{s=0}^{W}\|q(s)\|_p^2. \tag*{(26)} s ∈ I W ∑ ∥ q ( s + 1 ) − q ( s ) ∥ p 2 ≤ 3 W 2 16 d ( d + 1 ) s = 0 ∑ W ∥ q ( s ) ∥ p 2 . (26)
Proof. Let w s = ( s + 1 ) ( W − s ) w_s=(s+1)(W-s) w s = ( s + 1 ) ( W − s ) for 0 ≤ s < W 0\le s<W 0 ≤ s < W and w − 1 = w W = 0 w_{-1}=w_W=0 w − 1 = w W = 0 ; Define the symmetric operator K K K on R W + 1 \mathbb{R}^{W+1} R W + 1 by ( K v ) s = w s ( v s − v s + 1 ) + w s − 1 ( v s − v s − 1 ) (Kv)_s=w_s(v_s-v_{s+1})+w_{s-1}(v_s-v_{s-1}) ( K v ) s = w s ( v s − v s + 1 ) + w s − 1 ( v s − v s − 1 ) , where terms with indices outside { 0 , … , W } \{0,\ldots,W\} { 0 , … , W } are omitted. Summation by parts gives
⟨ v , K v ⟩ = ∑ s = 0 W − 1 w s ( v s + 1 − v s ) 2 . \langle v,Kv\rangle=\sum_{s=0}^{W-1}w_s(v_{s+1}-v_s)^2. ⟨ v , K v ⟩ = s = 0 ∑ W − 1 w s ( v s + 1 − v s ) 2 .
Let P d ⊆ R W + 1 \mathcal{P}_d\subseteq\mathbb{R}^{W+1} P d ⊆ R W + 1 be the space of vectors ( q ( 0 ) , … , q ( W ) ) (q(0),\ldots,q(W)) ( q ( 0 ) , … , q ( W )) with q q q a polynomial of degree at most d d d ; since d ≤ W d\le W d ≤ W , such a vector determines q q q . We claim that K K K maps P d \mathcal{P}_d P d into itself and acts on monomials by K ( s k ) = k ( k + 1 ) s k + K(s^k)=k(k+1)s^k+ K ( s k ) = k ( k + 1 ) s k + (a polynomial of degree ≤ k − 1 \le k-1 ≤ k − 1 ) for k ≥ 1 k\ge1 k ≥ 1 , and $K1=0. Indeed, writing ( K q ) ( s ) = − w s ( q ( s + 1 ) − q ( s ) ) + w s − 1 ( q ( s ) − q ( s − 1 ) ) (Kq)(s)=-w_s(q(s+1)-q(s))+w_{s-1}(q(s)-q(s-1)) ( K q ) ( s ) = − w s ( q ( s + 1 ) − q ( s )) + w s − 1 ( q ( s ) − q ( s − 1 )) and expanding q ( s ± 1 ) = s k ± k s k − 1 + ( k 2 ) s k − 2 ± ⋯ q(s\pm1)=s^k\pm ks^{k-1}+\binom{k}{2}s^{k-2}\pm\cdots q ( s ± 1 ) = s k ± k s k − 1 + ( 2 k ) s k − 2 ± ⋯ , the first-order terms contribute k s k − 1 ( w s − 1 − w s ) = k s k − 1 ( 2 s − W ) ks^{k-1}(w_{s-1}-w_s)=ks^{k-1}(2s-W) k s k − 1 ( w s − 1 − w s ) = k s k − 1 ( 2 s − W ) , with leading coefficient 2 k 2k 2 k , and the second-order terms contribute − ( k 2 ) s k − 2 ( w s + w s − 1 ) -\binom{k}{2}s^{k-2}(w_s+w_{s-1}) − ( 2 k ) s k − 2 ( w s + w s − 1 ) , whose leading coefficient is k ( k − 1 ) k(k-1) k ( k − 1 ) because the coefficient of s 2 s^2 s 2 in w s + w s − 1 w_s+w_{s-1} w s + w s − 1 is − 2 -2 − 2 ; the remaining terms have degree at most k − 1 k-1 k − 1 , and the vanishing of w − 1 w_{-1} w − 1 and w W w_W w W makes the computation valid at the endpoints. Hence K K K restricted to P d P_d P d is triangular in the basis of monomials with diagonal entries k ( k + 1 ) k(k+1) k ( k + 1 ) , 0 ≤ k ≤ d 0\le k\le d 0 ≤ k ≤ d , which are therefore its eigenvalues on P d P_d P d . Being symmetric, K K K restricted to the invariant subspace P d P_d P d is self-adjoint, so ⟨ v , K v ⟩ ≤ d ( d + 1 ) ⟨ v , v ⟩ \langle v,Kv\rangle\le d(d+1)\langle v,v\rangle ⟨ v , K v ⟩ ≤ d ( d + 1 ) ⟨ v , v ⟩ for v ∈ P d v\in P_d v ∈ P d , that is,
∑ s = 0 W − 1 w s ( q ( s + 1 ) − q ( s ) ) 2 ≤ d ( d + 1 ) ∑ s = 0 W q ( s ) 2 . \sum_{s=0}^{W-1}w_s(q(s+1)-q(s))^2\le d(d+1)\sum_{s=0}^{W}q(s)^2. s = 0 ∑ W − 1 w s ( q ( s + 1 ) − q ( s ) ) 2 ≤ d ( d + 1 ) s = 0 ∑ W q ( s ) 2 .
For s ∈ I W s\in\mathcal{I}_W s ∈ I W the number θ s = ( s + 1 ) / ( W + 1 ) \theta_s=(s+1)/(W+1) θ s = ( s + 1 ) / ( W + 1 ) lies in [ 1 / 4 , 3 / 4 ] [1/4,3/4] [ 1/4 , 3/4 ] , so w s = ( W + 1 ) 2 θ s ( 1 − θ s ) ≥ 3 16 ( W + 1 ) 2 ≥ 3 16 W 2 w_s=(W+1)^2\theta_s(1-\theta_s)\ge\frac{3}{16}(W+1)^2\ge\frac{3}{16}W^2 w s = ( W + 1 ) 2 θ s ( 1 − θ s ) ≥ 16 3 ( W + 1 ) 2 ≥ 16 3 W 2 ; dropping the terms with s ∉ I W s\notin\mathcal{I}_W s ∈ / I W and dividing by 3 16 W 2 \frac{3}{16}W^2 16 3 W 2 gives (25) . For (26) , apply (25) to each centered coordinate s ↦ q a ( s ) − ⟨ p , q ( s ) ⟩ s\mapsto q_a(s)-\langle p,q(s)\rangle s ↦ q a ( s ) − ⟨ p , q ( s )⟩ , which has degree at most d d d , multiply by p ( a ) p(a) p ( a ) and sum over a a a .
Sliding windows
The next lemma converts a temporal (or local) approximation by polynomials into the global variation–variance inequality.
Lemma 4.3 (Sliding-window comparison). Let T ≥ 1 T\ge1 T ≥ 1 , d ≥ 0 d\ge0 d ≥ 0 and r 0 ≥ 1 r_0\ge1 r 0 ≥ 1 be integers, let W ≥ 4 ( d + 1 ) W\ge4(d+1) W ≥ 4 ( d + 1 ) be divisible by four, and let ϵ ≥ 0 \epsilon\ge0 ϵ ≥ 0 . Let Z 1 , … , Z T ∈ [ 0 , 1 ] n Z^1,\ldots,Z^T\in[0,1]^n Z 1 , … , Z T ∈ [ 0 , 1 ] n and let P 1 , … , P T P^1,\ldots,P^T P 1 , … , P T be probability vectors with full support such that, for some κ ≥ 0 \kappa\ge0 κ ≥ 0 with e 2 κ W ≤ 2 e^{2\kappa W}\le2 e 2 κW ≤ 2 ,
e − κ ≤ P t + 1 ( a ) P t ( a ) ≤ e κ ( 1 ≤ t < T , a ∈ [ n ] ) . (27) e^{-\kappa}\le\frac{P^{t+1}(a)}{P^t(a)}\le e^\kappa\qquad(1\le t<T,\ a\in[n]). \tag*{(27)} e − κ ≤ P t ( a ) P t + 1 ( a ) ≤ e κ ( 1 ≤ t < T , a ∈ [ n ]) . (27) Suppose that for every integer r r r with r 0 ≤ r ≤ T − W r_0\le r\le T-W r 0 ≤ r ≤ T − W there exists q r : R → R n q_r:\mathbb{R}\to\mathbb{R}^n q r : R → R n in which every entry is a polynomial of degree at most d d d with max 0 ≤ s ≤ W ∥ Z r + s − q r ( s ) ∥ ∞ ≤ ϵ \max_{0\le s\le W}\|Z^{r+s}-q_r(s)\|_\infty\le\epsilon max 0 ≤ s ≤ W ∥ Z r + s − q r ( s ) ∥ ∞ ≤ ϵ . Then
∑ t = 1 T ∥ Z t − Z t − 1 ∥ P t 2 ≤ 256 d ( d + 1 ) ( W + 1 ) 3 W 3 ( ∑ t = 1 T ∥ Z t − 1 ∥ P t 2 + 1 4 ) + r 0 + 2 W + 32 T ϵ 2 . (28) \sum_{t=1}^{T}\|Z^t-Z^{t-1}\|_{P^t}^2\le\frac{256d(d+1)(W+1)}{3W^3}\left(\sum_{t=1}^{T}\|Z^{t-1}\|_{P^t}^2+\frac14\right)+r_0+2W+32T\epsilon^2. \tag*{(28)} t = 1 ∑ T ∥ Z t − Z t − 1 ∥ P t 2 ≤ 3 W 3 256 d ( d + 1 ) ( W + 1 ) ( t = 1 ∑ T ∥ Z t − 1 ∥ P t 2 + 4 1 ) + r 0 + 2 W + 32 T ϵ 2 . (28)
Proof. Write A t = ∥ Z t − Z t − 1 ∥ P t 2 A_t=\|Z^t-Z^{t-1}\|_{P_t}^2 A t = ∥ Z t − Z t − 1 ∥ P t 2 , V t = ∥ Z t − 1 ∥ P t 2 V_t=\|Z^{t-1}\|_{P_t}^2 V t = ∥ Z t − 1 ∥ P t 2 , A = ∑ t ≤ T A t A=\sum_{t\le T}A_t A = ∑ t ≤ T A t , V = ∑ t ≤ T V t V=\sum_{t\le T}V_t V = ∑ t ≤ T V t and λ = 16 d ( d + 1 ) / ( 3 W 2 ) \lambda=16d(d+1)/(3W^2) λ = 16 d ( d + 1 ) / ( 3 W 2 ) . The hypothesis W ≥ 4 ( d + 1 ) W\ge4(d+1) W ≥ 4 ( d + 1 ) gives d ( d + 1 ) ≤ W 2 / 16 d(d+1)\le W^2/16 d ( d + 1 ) ≤ W 2 /16 and hence λ ( W + 1 ) ≤ ( W + 1 ) / 3 ≤ W \lambda(W+1)\le(W+1)/3\le W λ ( W + 1 ) ≤ ( W + 1 ) /3 ≤ W . For bookkeeping set P T + 1 = P T P^{T+1}=P^T P T + 1 = P T and V T + 1 = ∥ Z T ∥ P T 2 ≤ 1 4 V_{T+1}=\|Z^T\|_{P^T}^2\le\frac14 V T + 1 = ∥ Z T ∥ P T 2 ≤ 4 1 .
One window with frozen weights. Fix b b b with r 0 + 1 ≤ b ≤ T − W + 1 r_0+1\le b\le T-W+1 r 0 + 1 ≤ b ≤ T − W + 1 and let z s = Z b − 1 + s z_s=Z^{b-1+s} z s = Z b − 1 + s , q s = q b − 1 ( s ) q_s=q_{b-1}(s) q s = q b − 1 ( s ) and e s = z s − q s e_s=z_s-q_s e s = z s − q s for 0 ≤ s ≤ W 0\le s\le W 0 ≤ s ≤ W , so that ∥ e s ∥ ∞ ≤ ϵ \|e_s\|_\infty\le\epsilon ∥ e s ∥ ∞ ≤ ϵ . For a fixed probability vector p p p , the inequality ∥ u + v ∥ p 2 ≤ 2 ∥ u ∥ p 2 + 2 ∥ v ∥ p 2 \|u+v\|_p^2\le2\|u\|_p^2+2\|v\|_p^2 ∥ u + v ∥ p 2 ≤ 2∥ u ∥ p 2 + 2∥ v ∥ p 2 , Lemma 4.2 and ∣ I W ∣ = W / 2 |\mathcal{I}_W|=W/2 ∣ I W ∣ = W /2 give
∑ s ∈ I W ∥ z s + 1 − z s ∥ p 2 ≤ 2 ∑ s ∈ I W ∥ q s + 1 − q s ∥ p 2 + 4 W ϵ 2 ≤ 2 λ ∑ s = 0 W ∥ q s ∥ p 2 + 4 W ϵ 2 \sum_{s\in\mathcal{I}_W}\|z_{s+1}-z_s\|_p^2\le2\sum_{s\in\mathcal{I}_W}\|q_{s+1}-q_s\|_p^2+4W\epsilon^2\le2\lambda\sum_{s=0}^{W}\|q_s\|_p^2+4W\epsilon^2 s ∈ I W ∑ ∥ z s + 1 − z s ∥ p 2 ≤ 2 s ∈ I W ∑ ∥ q s + 1 − q s ∥ p 2 + 4 W ϵ 2 ≤ 2 λ s = 0 ∑ W ∥ q s ∥ p 2 + 4 W ϵ 2
≤ 4 λ ∑ s = 0 W ∥ z s ∥ p 2 + ( 4 λ ( W + 1 ) + 4 W ) ϵ 2 . \le4\lambda\sum_{s=0}^{W}\|z_s\|_p^2+\bigl(4\lambda(W+1)+4W\bigr)\epsilon^2. ≤ 4 λ s = 0 ∑ W ∥ z s ∥ p 2 + ( 4 λ ( W + 1 ) + 4 W ) ϵ 2 .
Restoring the time-varying weights. Take p = P b p=P^b p = P b . By (27) , the coordinates of P b P^b P b and P b + s P^{b+s} P b + s differ by a factor at most Γ = e κ W \Gamma=e^{\kappa W} Γ = e κW for 0 ≤ s ≤ W 0\le s\le W 0 ≤ s ≤ W , and Γ 2 ≤ 2 \Gamma^2\le2 Γ 2 ≤ 2 . Since z s + 1 − z s = Z b + s − Z b + s − 1 z_{s+1}-z_s=Z^{b+s}-Z^{b+s-1} z s + 1 − z s = Z b + s − Z b + s − 1 and z s = Z b + s − 1 z_s=Z^{b+s-1} z s = Z b + s − 1 are the vectors whose variances under P b + s P^{b+s} P b + s are A b + s A_{b+s} A b + s and V b + s V_{b+s} V b + s , the fact that Var p ′ ≤ Γ Var p \operatorname{Var}_{p'}\le\Gamma\operatorname{Var}_p Var p ′ ≤ Γ Var p , applied once on each side, gives
∑ s ∈ I W A b + s ≤ 4 λ Γ 2 ∑ s = 0 W V b + s + Γ ( 4 λ ( W + 1 ) + 4 W ) ε 2 ≤ 8 λ ∑ s = 0 W V b + s + 16 W ε 2 . (29) \sum_{s\in I_W} A_{b+s}\le4\lambda\Gamma^2\sum_{s=0}^{W}V_{b+s}+\Gamma(4\lambda(W+1)+4W)\varepsilon^2\le8\lambda\sum_{s=0}^{W}V_{b+s}+16W\varepsilon^2. \tag*{(29)} s ∈ I W ∑ A b + s ≤ 4 λ Γ 2 s = 0 ∑ W V b + s + Γ ( 4 λ ( W + 1 ) + 4 W ) ε 2 ≤ 8 λ s = 0 ∑ W V b + s + 16 W ε 2 . (29)
using λ ( W + 1 ) ≤ W \lambda(W+1)\le W λ ( W + 1 ) ≤ W in the last step.
Summing over windows. Each A t A_t A t is trivially at most 1 1 1 . If T < r 0 + 2 W T<r_0+2W T < r 0 + 2 W then A ≤ T < r 0 + 2 W A\le T<r_0+2W A ≤ T < r 0 + 2 W and we are done, so assume T ≥ r 0 + 2 W T\ge r_0+2W T ≥ r 0 + 2 W and sum (29) over all admissible b b b . A time step t t t with r 0 + 3 4 W ≤ t ≤ T − 3 4 W + 1 r_0+\frac{3}{4}W\le t\le T-\frac{3}{4}W+1 r 0 + 4 3 W ≤ t ≤ T − 4 3 W + 1 occurs on the left-hand side once for every s ∈ I W s\in I_W s ∈ I W , with b = t − s b=t-s b = t − s , and these W / 2 W/2 W /2 values of b b b are all admissible; every other t t t occurs at most W / 2 W/2 W /2 times, and there are at most r 0 + 3 2 W ≤ r 0 + 2 W r_0+\frac{3}{2}W\le r_0+2W r 0 + 2 3 W ≤ r 0 + 2 W such t t t , contributing at most r 0 + 2 W r_0+2W r 0 + 2 W to A A A in total. Hence the left-hand sides add up to at least W 2 ( A − r 0 − 2 W ) \frac{W}{2}(A-r_0-2W) 2 W ( A − r 0 − 2 W ) . On the right, each V t V_t V t appears in at most W + 1 W+1 W + 1 windows and there are at most T T T windows, so
W 2 ( A − r 0 − 2 W ) ≤ 8 λ ( W + 1 ) ( V + 1 4 ) + 16 W T ε 2 . \frac{W}{2}(A-r_0-2W)\le8\lambda(W+1)\left(V+\frac{1}{4}\right)+16WT\varepsilon^2. 2 W ( A − r 0 − 2 W ) ≤ 8 λ ( W + 1 ) ( V + 4 1 ) + 16 W T ε 2 .
Dividing by W / 2 W/2 W /2 and substituting λ \lambda λ gives (28) .
Completion of the proof
Proof of Theorem 3.3 . Fix a player i i i . The vector 2 ℓ i t − ℓ i t − 1 2\ell_i^t-\ell_i^{t-1} 2 ℓ i t − ℓ i t − 1 has entries in [ − 1 , 2 ] [-1,2] [ − 1 , 2 ] , so by (3) consecutive strategies satisfy e − 3 η ≤ x t t + 1 ( a ) / x t t ( a ) ≤ e 3 η e^{-3\eta}\le x_t^{t+1}(a)/x_t^t(a)\le e^{3\eta} e − 3 η ≤ x t t + 1 ( a ) / x t t ( a ) ≤ e 3 η . We apply Lemma 4.3 with Z t = ℓ i t Z^t=\ell_i^t Z t = ℓ i t , P t = x i t P^t=x_i^t P t = x i t , κ = 3 η \kappa=3\eta κ = 3 η , r 0 = d + 1 r_0=d+1 r 0 = d + 1 , ε = 2 − d \varepsilon=2^{-d} ε = 2 − d and the polynomials of Lemma 4.1, whose hypothesis is the learning-rate condition of the theorem. The window W = 32 ( d + 1 ) W=32(d+1) W = 32 ( d + 1 ) is divisible by four and satisfies W ≥ 4 ( d + 1 ) W\ge4(d+1) W ≥ 4 ( d + 1 ) , and
2 κ W = 6 η W ≤ 6 256 m ⋅ W W + d < log 2 , 2\kappa W=6\eta W\le\frac{6}{256\sqrt{m}}\cdot\frac{W}{W+d}<\log2, 2 κW = 6 η W ≤ 256 m 6 ⋅ W + d W < log 2 ,
so (27) holds. Lemma 4.3 yields
D i ( T ) ≤ β ( V i ( T ) + 1 4 ) + ( d + 1 ) + 2 W + 32 T 4 − d , β = 256 d ( d + 1 ) ( W + 1 ) 3 W 3 , D_i(T)\le\beta\left(V_i(T)+\frac{1}{4}\right)+(d+1)+2W+32T4^{-d},\qquad\beta=\frac{256d(d+1)(W+1)}{3W^3}, D i ( T ) ≤ β ( V i ( T ) + 4 1 ) + ( d + 1 ) + 2 W + 32 T 4 − d , β = 3 W 3 256 d ( d + 1 ) ( W + 1 ) ,
and the choice W = 32 ( d + 1 ) W=32(d+1) W = 32 ( d + 1 ) gives
β = d 12 ( d + 1 ) ( 1 + 1 32 ( d + 1 ) ) < 1 12 . \beta=\frac{d}{12(d+1)}\left(1+\frac{1}{32(d+1)}\right)<\frac{1}{12}. β = 12 ( d + 1 ) d ( 1 + 32 ( d + 1 ) 1 ) < 12 1 .
Since 4 d ≥ ( T + 1 ) 2 4^d\ge(T+1)^2 4 d ≥ ( T + 1 ) 2 we have 32 T 4 − d ≤ 32 32T4^{-d}\le32 32 T 4 − d ≤ 32 , and since d ≥ 1 d\ge1 d ≥ 1 we have ( d + 1 ) + 32 + 1 48 ≤ 32 ( d + 1 ) = W (d+1)+32+\frac{1}{48}\le32(d+1)=W ( d + 1 ) + 32 + 48 1 ≤ 32 ( d + 1 ) = W ; therefore
D i ( T ) ≤ 1 4 V i ( T ) + 3 W . (30) D_i(T)\le\frac{1}{4}V_i(T)+3W. \tag*{(30)} D i ( T ) ≤ 4 1 V i ( T ) + 3 W . (30)
The learning-rate condition implies η ≤ 1 / 8 \eta\le1/8 η ≤ 1/8 , so Lemma 3.1 and (30) give
Reg i ( T ) ≤ log n i η + 2 η 3 ( 1 4 V i ( T ) + 3 W ) − η 3 V i ( T ) = log n i η − η 6 V i ( T ) + 2 η W ≤ log n i η + 2 η W . \operatorname{Reg}_i(T)\le\frac{\log n_i}{\eta}+\frac{2\eta}{3}\left(\frac{1}{4}V_i(T)+3W\right)-\frac{\eta}{3}V_i(T)=\frac{\log n_i}{\eta}-\frac{\eta}{6}V_i(T)+2\eta W\le\frac{\log n_i}{\eta}+2\eta W. Reg i ( T ) ≤ η log n i + 3 2 η ( 4 1 V i ( T ) + 3 W ) − 3 η V i ( T ) = η log n i − 6 η V i ( T ) + 2 η W ≤ η log n i + 2 η W .
For η = 1 / ( 256 m ( W + d ) ) \eta=1/(256\sqrt{m}(W+d)) η = 1/ ( 256 m ( W + d )) we have 2 η W ≤ 1 / ( 128 m ) ≤ 1 2\eta W\le1/(128\sqrt{m})\le1 2 η W ≤ 1/ ( 128 m ) ≤ 1 and W + d = 33 d + 32 = O ( log ( T + 2 ) ) W+d=33d+32=O(\log(T+2)) W + d = 33 d + 32 = O ( log ( T + 2 )) , which gives the stated rate.
The following is an auxiliary section containing useful lemmas from complex analysis which allow us to bound the truncation error of the loss expansions.
Complex exponential weights
The approximation lemma expands the dynamics in a complex learning rate. The following estimate is the reason the expansion converges on a disk whose radius degrades only as m − 1 / 2 m^{-1/2} m − 1/2 with the number of players.
Lemma 4.4 (Complex normalization). Let p ∈ Δ n p \in\Delta_n p ∈ Δ n and let y ∈ C n y \in\mathbb{C}^n y ∈ C n satisfy ∣ Im y ( a ) ∣ ≤ b ≤ 1 2 |\operatorname{Im} y(a)| \le b \le\frac{1}{2} ∣ Im y ( a ) ∣ ≤ b ≤ 2 1 for every a a a . Then Z = ∑ a p ( a ) e y ( a ) Z = \sum_a p(a)e^{y(a)} Z = ∑ a p ( a ) e y ( a ) is nonzero, and the complex vector q ( a ) = p ( a ) e y ( a ) / Z q(a) = p(a)e^{y(a)}/Z q ( a ) = p ( a ) e y ( a ) / Z satisfies
∑ a ∣ q ( a ) ∣ ≤ 1 cos b ≤ e b 2 . (31) \sum_a |q(a)| \le\frac{1}{\cos b} \le e^{b^2}. \tag*{(31)} a ∑ ∣ q ( a ) ∣ ≤ cos b 1 ≤ e b 2 . (31) If the strategy of every opponent j ≠ i j \ne i j = i is a complex vector q j q_j q j of this form, with the same b b b , then the loss map (1) extended multilinearly satisfies ∥ ℓ i ( q ) ∥ ∞ ≤ ∏ j ≠ i ∑ a ∣ q j ( a ) ∣ ≤ e ( m − 1 ) b 2 \|\ell_i(q)\|_\infty\le\prod_{j\ne i}\sum_a |q_j(a)| \le e^{(m-1)b^2} ∥ ℓ i ( q ) ∥ ∞ ≤ ∏ j = i ∑ a ∣ q j ( a ) ∣ ≤ e ( m − 1 ) b 2 .
Proof. Write y ( a ) = α a + i θ a y(a) = \alpha_a + i\theta_a y ( a ) = α a + i θ a . Then Re Z = ∑ a p ( a ) e α a cos θ a ≥ cos b ∑ a p ( a ) e α a > 0 \operatorname{Re} Z = \sum_a p(a)e^{\alpha_a}\cos\theta_a \ge\cos b\sum_a p(a)e^{\alpha_a} > 0 Re Z = ∑ a p ( a ) e α a cos θ a ≥ cos b ∑ a p ( a ) e α a > 0 , so Z ≠ 0 Z \ne0 Z = 0 and
∑ a ∣ q ( a ) ∣ = ∑ a p ( a ) e α a ∣ Z ∣ ≤ ∑ a p ( a ) e α a Re Z ≤ 1 cos b . \sum_a |q(a)| = \frac{\sum_a p(a)e^{\alpha_a}}{|Z|} \le\frac{\sum_a p(a)e^{\alpha_a}}{\operatorname{Re} Z} \le\frac{1}{\cos b}. a ∑ ∣ q ( a ) ∣ = ∣ Z ∣ ∑ a p ( a ) e α a ≤ Re Z ∑ a p ( a ) e α a ≤ cos b 1 .
For 0 ≤ t ≤ 1 2 0 \le t \le\frac{1}{2} 0 ≤ t ≤ 2 1 we have sin t ≤ t \sin t \le t sin t ≤ t and cos t ≥ 1 − t 2 / 2 ≥ 7 8 \cos t \ge1-t^2/2 \ge\frac{7}{8} cos t ≥ 1 − t 2 /2 ≥ 8 7 , hence tan t ≤ 2 t \tan t \le2t tan t ≤ 2 t and − log cos b = ∫ 0 b tan t d t ≤ b 2 -\log\cos b = \int_0^b \tan t\,\mathrm{d}t \le b^2 − log cos b = ∫ 0 b tan t d t ≤ b 2 . For the loss map, ∣ ℓ i ( q ) ( a i ) ∣ ≤ ∑ a − i L i ( a i , a − i ) ∏ j ≠ i ∣ q j ( a j ) ∣ ≤ ∏ j ≠ i ∑ a j ∣ q j ( a j ) ∣ |\ell_i(q)(a_i)| \le\sum_{a_{-i}} L_i(a_i,a_{-i})\prod_{j\ne i}|q_j(a_j)| \le\prod_{j\ne i}\sum_{a_j}|q_j(a_j)| ∣ ℓ i ( q ) ( a i ) ∣ ≤ ∑ a − i L i ( a i , a − i ) ∏ j = i ∣ q j ( a j ) ∣ ≤ ∏ j = i ∑ a j ∣ q j ( a j ) ∣ , where the last step distributes the product over the opponents’ action sets and uses L i ≤ 1 L_i \le1 L i ≤ 1 .
The following is a classical result from complex analysis which follow from the Cauchy integral estimate of analytic functions:
Lemma 4.5 (Analytic function tail bound). Let f : C → R f:\mathbb{C}\to\mathbb{R} f : C → R be analytic in a neighborhood of the closed disk { ∣ z ∣ ≤ R } \{|z|\le R\} { ∣ z ∣ ≤ R } and let M > 0 M>0 M > 0 be a constant for which ∣ f ( z ) ∣ ≤ M |f(z)|\le M ∣ f ( z ) ∣ ≤ M for all ∣ z ∣ ≤ R |z|\le R ∣ z ∣ ≤ R . Then the Taylor expansion f ( z ) = ∑ k = 0 ∞ f k z k f(z)=\sum_{k=0}^{\infty} f_k z^k f ( z ) = ∑ k = 0 ∞ f k z k satisfies, for all 0 ≤ η < R 0\le\eta<R 0 ≤ η < R :
∣ f ( η ) − ∑ k = 0 d f k η k ∣ ≤ M ( η / R ) d + 1 1 − η / R . (32) \left|f(\eta)-\sum_{k=0}^{d}f_k\eta^k\right|\le\frac{M(\eta/R)^{d+1}}{1-\eta/R}. \tag*{(32)} f ( η ) − k = 0 ∑ d f k η k ≤ 1 − η / R M ( η / R ) d + 1 . (32)
Proof of the constant regret upper bound for two players
Setting and notation. Throughout this section there are two players with n ≥ 2 n \ge2 n ≥ 2 actions each and loss matrices A , B ∈ [ 0 , 1 ] n × n A,B \in[0,1]^{n\times n} A , B ∈ [ 0 , 1 ] n × n : when player 1 plays a a a and player 2 plays b b b , their losses are A a b A_{ab} A ab and B a b B_{ab} B ab . As in § 3.3 , we write x 1 t , x 2 t ∈ Δ n x_1^t,x_2^t \in\Delta_n x 1 t , x 2 t ∈ Δ n for the strategies of players 1 and 2 in round t t t and
ℓ 1 t = A x 2 t , ℓ 2 t = B T x 1 t \ell_1^t = A x_2^t,\qquad\ell_2^t = B^\mathsf{T}x_1^t ℓ 1 t = A x 2 t , ℓ 2 t = B T x 1 t
for the loss vectors, so that the dynamics are x 1 t + 1 = E x 1 t [ − η ( 2 ℓ 1 t − ℓ 1 t − 1 ) ] x_1^{t+1} = \mathbb{E}_{x_1^t}[-\eta(2\ell_1^t-\ell_1^{t-1})] x 1 t + 1 = E x 1 t [ − η ( 2 ℓ 1 t − ℓ 1 t − 1 )] and x 2 t + 1 = E x 2 t [ − η ( 2 ℓ 2 t − ℓ 2 t − 1 ) ] x_2^{t+1} = \mathbb{E}_{x_2^t}[-\eta(2\ell_2^t-\ell_2^{t-1})] x 2 t + 1 = E x 2 t [ − η ( 2 ℓ 2 t − ℓ 2 t − 1 )] , with x 1 1 = x 2 1 x_1^1=x_2^1 x 1 1 = x 2 1 uniform and ℓ 1 0 = ℓ 2 0 = 0 \ell_1^0=\ell_2^0=0 ℓ 1 0 = ℓ 2 0 = 0 . We prove the regret bound for player 1; player 2 is treated symmetrically. For a fixed horizon T T T the argument is organised around three sums: the variance of the previous losses ∑ t = 1 T Var x 1 t ( ℓ 1 t − 1 ) \sum_{t=1}^T\operatorname{Var}_{x_1^t}(\ell_1^{t-1}) ∑ t = 1 T Var x 1 t ( ℓ 1 t − 1 ) and the variance of the increments ∑ t = 1 T Var x 1 t ( ℓ 1 t − ℓ 1 t − 1 ) \sum_{t=1}^T\operatorname{Var}_{x_1^t}(\ell_1^t-\ell_1^{t-1}) ∑ t = 1 T Var x 1 t ( ℓ 1 t − ℓ 1 t − 1 ) , which are the two quantities V 1 ( T ) V_1(T) V 1 ( T ) and D 1 ( T ) D_1(T) D 1 ( T ) appearing in Lemma 3.1 , and the variance of the current losses ∑ t = 1 T Var x 1 t ( ℓ 1 t ) \sum_{t=1}^T\operatorname{Var}_{x_1^t}(\ell_1^t) ∑ t = 1 T Var x 1 t ( ℓ 1 t ) , convenient because it is what controls the movement of player 1. We write v t = ∥ ℓ 1 t ∥ x 1 t = Var x 1 t ( ℓ 1 t ) v^t=\|\ell_1^t\|_{x_1^t}=\sqrt{\operatorname{Var}_{x_1^t}(\ell_1^t)} v t = ∥ ℓ 1 t ∥ x 1 t = Var x 1 t ( ℓ 1 t ) , with v 0 = 0 v^0=0 v 0 = 0 . Elementary facts about exponential tilting and variances are collected in Lemmas A.2 and A.1 of Appendix § 11 ; we use them freely. The letters C , C ′ , C ′ ′ C,C',C'' C , C ′ , C ′′ denote absolute constants.
Preliminary inequalities
Lemma 5.1 (Movement and variance shift). Let 0 < η ≤ 1 / 100 0 < \eta\le1/100 0 < η ≤ 1/100 . For every t ≥ 1 t \ge1 t ≥ 1 ,
e − 3 η ≤ x 1 t + 1 ( a ) x 1 t ( a ) ≤ e 3 η ( a ∈ [ n ] ) , ∥ x 1 t + 1 − x 1 t ∥ 1 ≤ 3 η ( v t + v t − 1 ) , (33) e^{-3\eta} \le\frac{x_1^{t+1}(a)}{x_1^t(a)} \le e^{3\eta} \quad(a \in[n]), \qquad\lVert x_1^{t+1}-x_1^t\rVert_1 \le3\eta(v^t+v^{t-1}), \tag*{(33)} e − 3 η ≤ x 1 t ( a ) x 1 t + 1 ( a ) ≤ e 3 η ( a ∈ [ n ]) , ∥ x 1 t + 1 − x 1 t ∥ 1 ≤ 3 η ( v t + v t − 1 ) , (33) and
∑ t = 1 T Var x 1 t ( ℓ 1 t ) ≤ e 3 η ∑ t = 1 T Var x 1 t ( ℓ 1 t − 1 ) + 1 4 . (34) \sum_{t=1}^{T} \operatorname{Var}_{x_1^t}(\ell_1^t) \le e^{3\eta}\sum_{t=1}^{T} \operatorname{Var}_{x_1^t}(\ell_1^{t-1})+\frac{1}{4}. \tag*{(34)} t = 1 ∑ T Var x 1 t ( ℓ 1 t ) ≤ e 3 η t = 1 ∑ T Var x 1 t ( ℓ 1 t − 1 ) + 4 1 . (34)
Proof. The vector z = − η ( 2 ℓ 1 t − ℓ 1 t − 1 ) z=-\eta(2\ell_1^t-\ell_1^{t-1}) z = − η ( 2 ℓ 1 t − ℓ 1 t − 1 ) satisfies max a , b ∈ [ n ] z ( a ) − z ( b ) ≤ 3 η \max_{a,b\in[n]}z(a)-z(b)\le3\eta max a , b ∈ [ n ] z ( a ) − z ( b ) ≤ 3 η , so the density bound (i) of Lemma A.2 gives the ratio bound. The movement bound (iii) of Lemma A.2 gives
∥ x 1 t + 1 − x 1 t ∥ 1 ≤ e 3 η / 2 η ∥ 2 ℓ 1 t − ℓ 1 t − 1 ∥ x 1 t ≤ η e 3 η / 2 ( 2 v t + ∥ ℓ 1 t − 1 ∥ x 1 t ) . \lVert x_1^{t+1}-x_1^t\rVert_1\le e^{3\eta/2}\eta\lVert2\ell_1^t-\ell_1^{t-1}\rVert_{x_1^t}\le\eta e^{3\eta/2}(2v^t+\lVert\ell_1^{t-1}\rVert_{x_1^t}). ∥ x 1 t + 1 − x 1 t ∥ 1 ≤ e 3 η /2 η ∥ 2 ℓ 1 t − ℓ 1 t − 1 ∥ x 1 t ≤ η e 3 η /2 ( 2 v t + ∥ ℓ 1 t − 1 ∥ x 1 t ) .
For t ≥ 2 t\ge2 t ≥ 2 the ratio bound between x 1 t x_1^t x 1 t and x 1 t − 1 x_1^{t-1} x 1 t − 1 and the comparison fact for variances (Lemma A.1) give ∥ ℓ 1 t − 1 ∥ x 1 t ≤ e 3 η / 2 ∥ ℓ 1 t − 1 ∥ x 1 t − 1 = e 3 η / 2 v t − 1 \lVert\ell_1^{t-1}\rVert_{x_1^t}\le e^{3\eta/2}\lVert\ell_1^{t-1}\rVert_{x_1^{t-1}}=e^{3\eta/2}v^{t-1} ∥ ℓ 1 t − 1 ∥ x 1 t ≤ e 3 η /2 ∥ ℓ 1 t − 1 ∥ x 1 t − 1 = e 3 η /2 v t − 1 , while for t = 1 t=1 t = 1 the term vanishes. Hence ∥ x 1 t + 1 − x 1 t ∥ 1 ≤ 2 η e 3 η / 2 v t + η e 3 η v t − 1 ≤ 3 η ( v t + v t − 1 ) \lVert x_1^{t+1}-x_1^t\rVert_1\le2\eta e^{3\eta/2}v^t+\eta e^{3\eta}v^{t-1}\le3\eta(v^t+v^{t-1}) ∥ x 1 t + 1 − x 1 t ∥ 1 ≤ 2 η e 3 η /2 v t + η e 3 η v t − 1 ≤ 3 η ( v t + v t − 1 ) for η ≤ 1 / 100 \eta\le1/100 η ≤ 1/100 . Finally, for t < T t<T t < T the ratio bound gives Var x 1 t ( ℓ 1 t ) ≤ e 3 η Var x 1 t + 1 ( ℓ 1 t ) \operatorname{Var}_{x_1^t}(\ell_1^t)\le e^{3\eta}\operatorname{Var}_{x_1^{t+1}}(\ell_1^t) Var x 1 t ( ℓ 1 t ) ≤ e 3 η Var x 1 t + 1 ( ℓ 1 t ) , and summing over t < T t<T t < T yields
∑ t < T Var x 1 t ( ℓ 1 t ) ≤ e 3 η ∑ t < T Var x 1 t + 1 ( ℓ 1 t ) ≤ e 3 η ∑ t = 1 T Var x 1 t ( ℓ 1 t − 1 ) ; \sum_{t<T}\operatorname{Var}_{x_1^t}(\ell_1^t)\le e^{3\eta}\sum_{t<T}\operatorname{Var}_{x_1^{t+1}}(\ell_1^t)\le e^{3\eta}\sum_{t=1}^{T}\operatorname{Var}_{x_1^t}(\ell_1^{t-1}); t < T ∑ Var x 1 t ( ℓ 1 t ) ≤ e 3 η t < T ∑ Var x 1 t + 1 ( ℓ 1 t ) ≤ e 3 η t = 1 ∑ T Var x 1 t ( ℓ 1 t − 1 ) ;
the remaining term Var x 1 T ( ℓ 1 T ) \operatorname{Var}_{x_1^T}(\ell_1^T) Var x 1 T ( ℓ 1 T ) is at most 1 4 \frac14 4 1 because ℓ 1 T ∈ [ 0 , 1 ] n \ell_1^T\in[0,1]^n ℓ 1 T ∈ [ 0 , 1 ] n .
Since ℓ 2 t = B ⊤ x 1 t \ell_2^t=B^\top x_1^t ℓ 2 t = B ⊤ x 1 t with entries of B B B in [ 0 , 1 ] [0,1] [ 0 , 1 ] , the lemma gives the two-player feedback estimate
∥ ℓ 2 t + 1 − ℓ 2 t ∥ ∞ ≤ ∥ x 1 t + 1 − x 1 t ∥ 1 ≤ 3 η ( v t + v t − 1 ) ; (35) \lVert\ell_2^{t+1}-\ell_2^t\rVert_\infty\le\lVert x_1^{t+1}-x_1^t\rVert_1\le3\eta(v^t+v^{t-1}); \tag*{(35)} ∥ ℓ 2 t + 1 − ℓ 2 t ∥ ∞ ≤ ∥ x 1 t + 1 − x 1 t ∥ 1 ≤ 3 η ( v t + v t − 1 ) ; (35)
the input of player 2 moves only when player 1 moves, and player 1 moves only when their current losses have variance. This is the only place where the number of players matters. With three or more players, ℓ 2 t \ell_2^t ℓ 2 t would also depend on the strategy of player 3, whose movement is governed by player 3’s variance, a quantity that does not appear in player 1’s potential inequality.
A Bernstein inequality for exponential sums.
Lemma 5.2 (Interior Bernstein inequality, L 2 L^2 L 2 form). Let K \mathcal{K} K be a real Hilbert space, let N ( s ) = ∑ k = 1 n w k e λ k s N(s)=\sum_{k=1}^{n}w_ke^{\lambda_ks} N ( s ) = ∑ k = 1 n w k e λ k s with w k ∈ K w_k\in\mathcal{K} w k ∈ K and λ k ∈ R \lambda_k\in\mathbb{R} λ k ∈ R , and let I = [ α + W , α + 2 W ] I=[\alpha+W,\alpha+2W] I = [ α + W , α + 2 W ] be the middle third of J = [ α , α + 3 W ] J=[\alpha,\alpha+3W] J = [ α , α + 3 W ] for some α ∈ R \alpha\in\mathbb{R} α ∈ R and W > 0 W>0 W > 0 . Then
( ∫ I ∥ N ′ ( s ) ∥ 2 d s ) 1 / 2 ≤ 2 n W ( ∫ J ∥ N ( s ) ∥ 2 d s ) 1 / 2 . (36) \left(\int_I\lVert N'(s)\rVert^2\,ds\right)^{1/2}\le\frac{2n}{W}\left(\int_J\lVert N(s)\rVert^2\,ds\right)^{1/2}. \tag*{(36)} ( ∫ I ∥ N ′ ( s ) ∥ 2 d s ) 1/2 ≤ W 2 n ( ∫ J ∥ N ( s ) ∥ 2 d s ) 1/2 . (36) The same inequality holds for K = R n \mathcal{K}=\mathbb{R}^n K = R n equipped with the variance seminorm ∥ ⋅ ∥ p \lVert\cdot\rVert_p ∥ ⋅ ∥ p of any fixed probability vector p p p .
Proof. Borwein and Erdélyi [5 ] proved that every real function g ( u ) = μ 0 + ∑ k = 1 n μ k e κ k u g(u)=\mu_0+\sum_{k=1}^{n}\mu_ke^{\kappa_k u} g ( u ) = μ 0 + ∑ k = 1 n μ k e κ k u satisfies ∣ g ′ ( 0 ) ∣ ≤ 2 n W max ∣ u ∣ ≤ W ∣ g ( u ) ∣ |g'(0)|\le\frac{2n}{W}\max_{|u|\le W}|g(u)| ∣ g ′ ( 0 ) ∣ ≤ W 2 n max ∣ u ∣ ≤ W ∣ g ( u ) ∣ . Let ψ ∈ L 2 ( I ; K ) \psi\in L^2(I;\mathcal{K}) ψ ∈ L 2 ( I ; K ) have norm one and set g ( u ) = ∫ I ⟨ N ( s + u ) , ψ ( s ) ⟩ d s g(u)=\int_I\langle N(s+u),\psi(s)\rangle ds g ( u ) = ∫ I ⟨ N ( s + u ) , ψ ( s )⟩ d s . Expanding N N N , we see that g g g is a real exponential sum with exponents λ 1 , … , λ n \lambda_1,\ldots,\lambda_n λ 1 , … , λ n (and no constant term, which we may regard as μ 0 = 0 \mu_0 = 0 μ 0 = 0 . For ∣ u ∣ ≤ W |u| \le W ∣ u ∣ ≤ W the translate I + u I+u I + u is contained in J J J , so the Cauchy–Schwarz inequality gives ∣ g ( u ) ∣ ≤ ∥ N ∥ L 2 ( J ; K ) |g(u)| \le\|N\|_{L^2(J;\mathcal{K})} ∣ g ( u ) ∣ ≤ ∥ N ∥ L 2 ( J ; K ) . Since g ′ ( 0 ) = ∫ I ⟨ N ′ ( s ) , ψ ( s ) ⟩ d s g'(0) = \int_I \langle N'(s),\psi(s)\rangle ds g ′ ( 0 ) = ∫ I ⟨ N ′ ( s ) , ψ ( s )⟩ d s , the scalar inequality gives ∫ I ⟨ N ′ , ψ ⟩ ≤ 2 n W ∥ N ∥ L 2 ( J ; K ) \int_I \langle N',\psi\rangle\le\frac{2n}{W}\|N\|_{L^2(J;\mathcal{K})} ∫ I ⟨ N ′ , ψ ⟩ ≤ W 2 n ∥ N ∥ L 2 ( J ; K ) , and taking the supremum over ψ \psi ψ proves (36) .
The variance seminorm. Let p ∈ Δ n p \in\Delta_n p ∈ Δ n be fixed and recall that
∥ v ∥ p 2 = Var p ( v ) = ∑ h = 1 n p ( h ) ( v ( h ) − ⟨ p , v ⟩ ) 2 ( v ∈ R n ) . \|v\|_p^2 = \operatorname{Var}_p(v) = \sum_{h=1}^{n} p(h)\left(v(h)-\langle p,v\rangle\right)^2 \qquad(v \in\mathbb{R}^n). ∥ v ∥ p 2 = Var p ( v ) = h = 1 ∑ n p ( h ) ( v ( h ) − ⟨ p , v ⟩ ) 2 ( v ∈ R n ) .
This is only a seminorm on R n \mathbb{R}^n R n : it vanishes on every vector that is constant on the support of p p p , so R n \mathbb{R}^n R n with ∥ ⋅ ∥ p \|\cdot\|_p ∥ ⋅ ∥ p is not a Hilbert space and (36) cannot be invoked for it directly. It is, however, the Euclidean norm of a linear image of v v v . Write P = diag ( p ) P=\operatorname{diag}(p) P = diag ( p ) and define the centering map
Π p : R n → R n , Π p v = P 1 / 2 ( v − ⟨ p , v ⟩ 1 ) , \Pi_p:\mathbb{R}^n \to\mathbb{R}^n,\qquad\Pi_p v=P^{1/2}(v-\langle p,v\rangle\mathbf{1}), Π p : R n → R n , Π p v = P 1/2 ( v − ⟨ p , v ⟩ 1 ) ,
that is, ( Π p v ) ( h ) = p ( h ) ( v ( h ) − ⟨ p , v ⟩ ) (\Pi_pv)(h)=\sqrt{p(h)}(v(h)-\langle p,v\rangle) ( Π p v ) ( h ) = p ( h ) ( v ( h ) − ⟨ p , v ⟩) . Then
∥ Π p v ∥ 2 = ∑ h = 1 n p ( h ) ( v ( h ) − ⟨ p , v ⟩ ) 2 = ∥ v ∥ p 2 ( v ∈ R n ) , \|\Pi_pv\|^2=\sum_{h=1}^{n}p(h)\left(v(h)-\langle p,v\rangle\right)^2=\|v\|_p^2 \qquad(v\in\mathbb{R}^n), ∥ Π p v ∥ 2 = h = 1 ∑ n p ( h ) ( v ( h ) − ⟨ p , v ⟩ ) 2 = ∥ v ∥ p 2 ( v ∈ R n ) ,
so Π p \Pi_p Π p carries the seminorm ∥ ⋅ ∥ p \|\cdot\|_p ∥ ⋅ ∥ p isometrically onto the standard norm of R n \mathbb{R}^n R n . Two properties of Π p \Pi_p Π p are all that the argument needs. First, Π p \Pi_p Π p is linear: the centering v ↦ v − ⟨ p , v ⟩ 1 v\mapsto v-\langle p,v\rangle\mathbf{1} v ↦ v − ⟨ p , v ⟩ 1 is the linear map Id − 1 p T \operatorname{Id}-\mathbf{1}p^\mathsf{T} Id − 1 p T , and P 1 / 2 P^{1/2} P 1/2 is a fixed diagonal matrix. Second, Π p \Pi_p Π p does not depend on the time variable s s s , because p p p is fixed. Together these mean that Π p \Pi_p Π p commutes with the two operations we perform on the curve N N N , namely forming it as an exponential sum and differentiating it in s s s : for
N ( s ) = ∑ k = 1 n w k e λ k s , N(s)=\sum_{k=1}^{n}w_ke^{\lambda_ks}, N ( s ) = k = 1 ∑ n w k e λ k s ,
Π p N ( s ) = ∑ k = 1 n ( Π p w k ) e λ k s , ( Π p N ) ′ ( s ) = Π p ( N ′ ( s ) ) = ∑ k = 1 n λ k ( Π p w k ) e λ k s . \Pi_pN(s)=\sum_{k=1}^{n}(\Pi_pw_k)e^{\lambda_ks},\qquad(\Pi_pN)'(s)=\Pi_p(N'(s))=\sum_{k=1}^{n}\lambda_k(\Pi_pw_k)e^{\lambda_ks}. Π p N ( s ) = k = 1 ∑ n ( Π p w k ) e λ k s , ( Π p N ) ′ ( s ) = Π p ( N ′ ( s )) = k = 1 ∑ n λ k ( Π p w k ) e λ k s .
Thus Π p N \Pi_pN Π p N is again an exponential sum with the same exponents λ 1 , … , λ n \lambda_1,\ldots,\lambda_n λ 1 , … , λ n and at most n n n terms, now with coefficients Π p w k \Pi_pw_k Π p w k in the Hilbert space R n \mathbb{R}^n R n equipped with its standard inner product, and its derivative is the image of N ′ N' N ′ under the same map. (The number of terms does not grow, which matters because the constant 2 n 2n 2 n in (36) is the number of exponentials.) Applying (36) with K = R n \mathcal{K}=\mathbb{R}^n K = R n to the exponential sum Π p N \Pi_pN Π p N and translating back through the isometry gives
( ∫ I ∥ N ′ ( s ) ∥ p 2 d s ) 1 / 2 = ( ∫ I ∥ ( Π p N ) ′ ( s ) ∥ 2 d s ) 1 / 2 ≤ 2 n W ( ∫ J ∥ Π p N ( s ) ∥ 2 d s ) 1 / 2 = 2 n W ( ∫ J ∥ N ( s ) ∥ p 2 d s ) 1 / 2 , \left(\int_I\|N'(s)\|_p^2\,ds\right)^{1/2}=\left(\int_I\|(\Pi_pN)'(s)\|^2\,ds\right)^{1/2}\leq\frac{2n}{W}\left(\int_J\|\Pi_pN(s)\|^2\,ds\right)^{1/2}=\frac{2n}{W}\left(\int_J\|N(s)\|_p^2\,ds\right)^{1/2}, ( ∫ I ∥ N ′ ( s ) ∥ p 2 d s ) 1/2 = ( ∫ I ∥ ( Π p N ) ′ ( s ) ∥ 2 d s ) 1/2 ≤ W 2 n ( ∫ J ∥ Π p N ( s ) ∥ 2 d s ) 1/2 = W 2 n ( ∫ J ∥ N ( s ) ∥ p 2 d s ) 1/2 ,
which is (36) for the variance seminorm. Equivalently, Π p \Pi_p Π p identifies the quotient of R n \mathbb{R}^n R n by the kernel of ∥ ⋅ ∥ p \|\cdot\|_p ∥ ⋅ ∥ p with a subspace of R n \mathbb{R}^n R n ; the explicit map makes visible that the exponential-sum structure survives the passage to the quotient.
Note that the argument uses in an essential way that p p p does not vary with s s s : with time-dependent weights the map Π p ( s ) \Pi_p(s) Π p ( s ) would no longer commute with differentiation, and Π p ( s ) N ( s ) \Pi_p(s)N(s) Π p ( s ) N ( s ) would not be an exponential sum. This is why the weights are frozen at p = x 1 [ s ] p=x_1^{[s]} p = x 1 [ s ] in the proof of Lemma 5.6 , and the true weights x 1 [ s ] x_1^{[s]} x 1 [ s ] are restored only afterwards, through the ratio bound of Lemma 5.1 . □
The inequality is interior : it bounds the derivative only on the middle third, and no bound of this kind holds up to the endpoints. This is why the windows used in proving the properties of the temporal approximation framework in continuous time have length 3 W 3W 3 W and only their middle thirds are used.
Proof of Theorem 3.4
As stated in the proof overview, we use the temporal approximation framework analogously on the interpolation of the iterates in continuous time . As we prove in § 5.3 , by reducing the resulting bounds on the continuous terms to the discrete terms, we obtain a sufficient variance bound.
Defining the temporal approximation framework and the resulting variance bound. If player 2’s losses stayed equal to ℓ 2 t 0 \ell^{t_0}_2 ℓ 2 t 0 from some round t 0 t_0 t 0 on, their strategy would follow E x 2 t 0 [ − η ( s − t 0 ) ℓ 2 t 0 ] \mathop{\mathrm{E}}\nolimits_{x_2^{t_0}}[-\eta(s-t_0)\ell^{t_0}_2] E x 2 t 0 [ − η ( s − t 0 ) ℓ 2 t 0 ] , and player 1’s losses A ⋅ E x 2 t 0 [ − η ( s − t 0 ) ℓ 2 t 0 ] A\cdot\mathop{\mathrm{E}}\nolimits_{x_2^{t_0}}[-\eta(s-t_0)\ell^{t_0}_2] A ⋅ E x 2 t 0 [ − η ( s − t 0 ) ℓ 2 t 0 ] would be a ratio of two sums of n n n exponentials in s s s . Lemma 5.2 would then bound the increments of player 1’s losses over the middle third of a window of length 3 W 3W 3 W by O ( n / W ) O(n/W) O ( n / W ) times their size over the window. Player 2’s losses do move, but by the feedback estimate (35) only as fast as player 1’s current losses have variance, which is what the potential inequality allows us to charge. The following lemma makes this precise. Its statement involves only the discrete trajectory; continuous time enters only its proof, given in § 5.3 .
Lemma 5.3 (Variance bound resulting from the temporal approximation framework). There are absolute constants C C C and K 0 K_0 K 0 such that for every integer W ≥ K 0 n W\ge K_0n W ≥ K 0 n and every η \eta η with 0 < η W ≤ 10 − 3 0<\eta W\le10^{-3} 0 < η W ≤ 1 0 − 3 ,
∑ t = 1 T Var x 1 t ( ℓ 1 t − ℓ 1 t − 1 ) ≤ 1 4 + C [ ( n 2 W 2 + η 4 n 2 W 2 ) ( ∑ t = 1 T Var x 1 t ( ℓ 1 t − 1 ) + 1 ) + η 2 W ] . (37) \sum_{t=1}^{T}\operatorname{Var}_{x_1^t}\left(\ell_1^t-\ell_1^{t-1}\right)\le\frac{1}{4}+C\left[\left(\frac{n^2}{W^2}+\eta^4n^2W^2\right)\left(\sum_{t=1}^{T}\operatorname{Var}_{x_1^t}\left(\ell_1^{t-1}\right)+1\right)+\eta^2W\right]. \tag*{(37)} t = 1 ∑ T Var x 1 t ( ℓ 1 t − ℓ 1 t − 1 ) ≤ 4 1 + C [ ( W 2 n 2 + η 4 n 2 W 2 ) ( t = 1 ∑ T Var x 1 t ( ℓ 1 t − 1 ) + 1 ) + η 2 W ] . (37)
In the bracket, the factor n 2 / W 2 n^2/W^2 n 2 / W 2 comes from the Bernstein inequality of Lemma 5.2 , the factor η 4 n 2 W 2 \eta^4n^2W^2 η 4 n 2 W 2 is the freezing error, paid by the variance of the current losses through the feedback estimate (35) , and the term η 2 W \eta^2W η 2 W is the cost of the two ends of the horizon.
Tuning η \eta η , W W W we may reduce the variance bound to the necessary constants:
Lemma 5.4 (Two-player variation–variance inequality). There is an absolute constant c > 0 c>0 c > 0 such that if both players run Optimistic Hedge with 0 < η ≤ c / n 0<\eta\le c/n 0 < η ≤ c / n , then
∑ t = 1 T Var x 1 t ( ℓ 1 t − ℓ 1 t − 1 ) ≤ 1 64 ∑ t = 1 T Var x 1 t ( ℓ 1 t − 1 ) + 1 2 . (38) \sum_{t=1}^{T}\operatorname{Var}_{x_1^t}\left(\ell_1^t-\ell_1^{t-1}\right)\le\frac{1}{64}\sum_{t=1}^{T}\operatorname{Var}_{x_1^t}\left(\ell_1^{t-1}\right)+\frac{1}{2}. \tag*{(38)} t = 1 ∑ T Var x 1 t ( ℓ 1 t − ℓ 1 t − 1 ) ≤ 64 1 t = 1 ∑ T Var x 1 t ( ℓ 1 t − 1 ) + 2 1 . (38)
Proof. Let C C C and K 0 K_0 K 0 be the constants of Lemma 5.3 . Let K ≥ K 0 K\ge K_0 K ≥ K 0 be an integer, to be chosen below, and put W = K n W=Kn W = K n . If η ≤ c / n \eta\le c/n η ≤ c / n with c K ≤ 10 − 3 cK\le10^{-3} cK ≤ 1 0 − 3 , Lemma 5.3 applies and gives
∑ t = 1 T Var x 1 t ( ℓ 1 t − ℓ 1 t − 1 ) ≤ 1 4 + C ( K − 2 + c 4 K 2 ) ( ∑ t = 1 T Var x 1 t ( ℓ 1 t − 1 ) + 1 ) + C c 2 K , \sum_{t=1}^{T}\operatorname{Var}_{x_1^t}\left(\ell_1^t-\ell_1^{t-1}\right)\le\frac{1}{4}+C\left(K^{-2}+c^4K^2\right)\left(\sum_{t=1}^{T}\operatorname{Var}_{x_1^t}\left(\ell_1^{t-1}\right)+1\right)+Cc^2K, t = 1 ∑ T Var x 1 t ( ℓ 1 t − ℓ 1 t − 1 ) ≤ 4 1 + C ( K − 2 + c 4 K 2 ) ( t = 1 ∑ T Var x 1 t ( ℓ 1 t − 1 ) + 1 ) + C c 2 K ,
because n 2 / W 2 = K − 2 n^2/W^2=K^{-2} n 2 / W 2 = K − 2 , η 4 n 2 W 2 ≤ c 4 K 2 \eta^4n^2W^2\le c^4K^2 η 4 n 2 W 2 ≤ c 4 K 2 and η 2 W ≤ c 2 K / n ≤ c 2 K \eta^2W\le c^2K/n\le c^2K η 2 W ≤ c 2 K / n ≤ c 2 K . Choose K K K so large that C / K 2 ≤ 1 128 C/K^2\le\frac{1}{128} C / K 2 ≤ 128 1 , and then c c c so small that C c 4 K 2 ≤ 1 128 Cc^4K^2\le\frac{1}{128} C c 4 K 2 ≤ 128 1 , C c 2 K ≤ 1 8 Cc^2K\le\frac{1}{8} C c 2 K ≤ 8 1 , c K ≤ 10 − 3 cK\le10^{-3} cK ≤ 1 0 − 3 and c ≤ 1 100 c\le\frac{1}{100} c ≤ 100 1 ; the last condition gives η ≤ 1 100 \eta\le\frac{1}{100} η ≤ 100 1 , as required by Lemmas 5.1 and 3.1 . The coefficient of ∑ t = 1 T Var x 1 t ( ℓ 1 t − 1 ) \sum_{t=1}^{T}\operatorname{Var}_{x_1^t}(\ell_1^{t-1}) ∑ t = 1 T Var x 1 t ( ℓ 1 t − 1 ) is then at most 1 64 \frac{1}{64} 64 1 and the additive part at most 1 4 + 1 64 + 1 8 < 1 2 \frac{1}{4}+\frac{1}{64}+\frac{1}{8}<\frac{1}{2} 4 1 + 64 1 + 8 1 < 2 1 .
Proof of Theorem 3.4. Let η ≤ c / n \eta\le c/n η ≤ c / n with c c c as in Lemma 5.4 ; in particular η ≤ 1 / 100 \eta\le1/100 η ≤ 1/100 . For player 1, Lemma 3.1 and (38) give
Reg 1 ( T ) ≤ log n η + 2 η 3 ( 1 64 ∑ t = 1 T Var x 1 t ( ℓ 1 t − 1 ) + 1 2 ) − η 3 ∑ t = 1 T Var x 1 t ( ℓ 1 t − 1 ) ≤ log n η + η 3 ≤ log n η + η , \operatorname{Reg}_1(T)\le\frac{\log n}{\eta}+\frac{2\eta}{3}\left(\frac{1}{64}\sum_{t=1}^{T}\operatorname{Var}_{x_1^t}\left(\ell_1^{t-1}\right)+\frac{1}{2}\right)-\frac{\eta}{3}\sum_{t=1}^{T}\operatorname{Var}_{x_1^t}\left(\ell_1^{t-1}\right)\le\frac{\log n}{\eta}+\frac{\eta}{3}\le\frac{\log n}{\eta}+\eta, Reg 1 ( T ) ≤ η log n + 3 2 η ( 64 1 t = 1 ∑ T Var x 1 t ( ℓ 1 t − 1 ) + 2 1 ) − 3 η t = 1 ∑ T Var x 1 t ( ℓ 1 t − 1 ) ≤ η log n + 3 η ≤ η log n + η ,
since ∑ t = 1 T Var x 1 t ( ℓ 1 t − 1 ) ≥ 0 \sum_{t=1}^{T}\operatorname{Var}_{x_1^t}(\ell_1^{t-1})\ge0 ∑ t = 1 T Var x 1 t ( ℓ 1 t − 1 ) ≥ 0 has a negative coefficient. The same argument with the roles of the players exchanged bounds Reg 2 ( T ) \operatorname{Reg}_2(T) Reg 2 ( T ) .
Proof of Lemma 5.3: interpolation and windows
Throughout this subsection W ≥ 1 W \ge1 W ≥ 1 is an integer and 0 < η W ≤ 10 − 3 0 < \eta W \le10^{-3} 0 < η W ≤ 1 0 − 3 ; in particular η ≤ 1 / 100 \eta\le1/100 η ≤ 1/100 , so Lemma 5.1 applies. We first build the continuous objects and prove two lemmas about them, and then sum over windows.
The interpolation. Continuous time enters only through the following interpolation of player 2’s trajectory. For every integer j ≥ 1 j \ge1 j ≥ 1 and s ∈ [ j , j + 1 ] s \in[j,j+1] s ∈ [ j , j + 1 ] let g j = 2 ℓ 2 j − ℓ 2 j − 1 g^j = 2\ell_2^j-\ell_2^{j-1} g j = 2 ℓ 2 j − ℓ 2 j − 1 be the score direction of round j j j and
Y ( s ) = E x 2 j [ − η ( s − j ) g j ] , L ( s ) = A Y ( s ) . (39) Y(s)=\mathbb{E}_{x_2^j}[-\eta(s-j)g^j],\qquad L(s)=AY(s). \tag*{(39)} Y ( s ) = E x 2 j [ − η ( s − j ) g j ] , L ( s ) = A Y ( s ) . (39)
Then Y ( j ) = x 2 j Y(j)=x_2^j Y ( j ) = x 2 j , Y ( j + 1 ) = x 2 j + 1 Y(j+1)=x_2^{j+1} Y ( j + 1 ) = x 2 j + 1 and L ( j ) = ℓ 1 j L(j)=\ell_1^j L ( j ) = ℓ 1 j ; the curves are smooth on each unit interval, which is all that the following integrals require. The proof works with two continuous energies of the interpolated losses, the position energy ∫ 1 T Var x 1 ⌊ s ⌋ ( L ( s ) ) d s \int_1^T \operatorname{Var}_{x_1^{\lfloor s\rfloor}}(L(s))\,\mathrm{d}s ∫ 1 T Var x 1 ⌊ s ⌋ ( L ( s )) d s and the derivative energy ∫ 1 T Var x 1 ⌊ s ⌋ ( L ′ ( s ) ) d s \int_1^T \operatorname{Var}_{x_1^{\lfloor s\rfloor}}(L'(s))\,\mathrm{d}s ∫ 1 T Var x 1 ⌊ s ⌋ ( L ′ ( s )) d s , in both of which the variance weights are those of player 1 in the round containing s s s . They are the continuous counterparts of the variance of the current losses and of the variance of the increments.
Lemma 5.5 (Discrete–continuous comparisons). For every horizon T T T ,
∑ t = 1 T Var x 1 t ( ℓ 1 t − ℓ 1 t − 1 ) ≤ 1 4 + e 3 η ∫ 1 T Var x 1 ⌊ s ⌋ ( L ′ ( s ) ) d s (40) \sum_{t=1}^{T}\operatorname{Var}_{x_1^t}(\ell_1^t-\ell_1^{t-1})\le\frac{1}{4}+e^{3\eta}\int_1^T\operatorname{Var}_{x_1^{\lfloor s\rfloor}}(L'(s))\,\mathrm{d}s \tag*{(40)} t = 1 ∑ T Var x 1 t ( ℓ 1 t − ℓ 1 t − 1 ) ≤ 4 1 + e 3 η ∫ 1 T Var x 1 ⌊ s ⌋ ( L ′ ( s )) d s (40)
and
∫ 1 T Var x 1 ⌊ s ⌋ ( L ( s ) ) d s ≤ 2 ∑ t = 1 T Var x 1 t ( ℓ 1 t ) + 2 ∫ 1 T Var x 1 ⌊ s ⌋ ( L ′ ( s ) ) d s . (41) \int_1^T\operatorname{Var}_{x_1^{\lfloor s\rfloor}}(L(s))\,\mathrm{d}s\le2\sum_{t=1}^{T}\operatorname{Var}_{x_1^t}(\ell_1^t)+2\int_1^T\operatorname{Var}_{x_1^{\lfloor s\rfloor}}(L'(s))\,\mathrm{d}s. \tag*{(41)} ∫ 1 T Var x 1 ⌊ s ⌋ ( L ( s )) d s ≤ 2 t = 1 ∑ T Var x 1 t ( ℓ 1 t ) + 2 ∫ 1 T Var x 1 ⌊ s ⌋ ( L ′ ( s )) d s . (41)
Proof. The t = 1 t=1 t = 1 term on the left of (40) is Var x 1 1 ( ℓ 1 1 ) ≤ 1 4 \operatorname{Var}_{x_1^1}(\ell_1^1)\le\frac14 Var x 1 1 ( ℓ 1 1 ) ≤ 4 1 . For j ≥ 1 j\ge1 j ≥ 1 we have ℓ 1 j + 1 − ℓ 1 j = ∫ j j + 1 L ′ ( s ) d s \ell_1^{j+1}-\ell_1^j=\int_j^{j+1}L'(s)\,\mathrm{d}s ℓ 1 j + 1 − ℓ 1 j = ∫ j j + 1 L ′ ( s ) d s , and Jensen’s inequality for the Hilbert seminorm ∥ ⋅ ∥ x 1 j \|\cdot\|_{x_1^j} ∥ ⋅ ∥ x 1 j gives Var x 1 j ( ℓ 1 j + 1 − ℓ 1 j ) ≤ ∫ j j + 1 Var x 1 j ( L ′ ( s ) ) d s \operatorname{Var}_{x_1^j}(\ell_1^{j+1}-\ell_1^j)\le\int_j^{j+1}\operatorname{Var}_{x_1^j}(L'(s))\,\mathrm{d}s Var x 1 j ( ℓ 1 j + 1 − ℓ 1 j ) ≤ ∫ j j + 1 Var x 1 j ( L ′ ( s )) d s . Since x 1 j + 1 ≤ e 3 η x 1 j x_1^{j+1}\le e^{3\eta}x_1^j x 1 j + 1 ≤ e 3 η x 1 j coordinatewise, Var x 1 j + 1 ( ℓ 1 j + 1 − ℓ 1 j ) ≤ e 3 η Var x 1 j ( ℓ 1 j + 1 − ℓ 1 j ) \operatorname{Var}_{x_1^{j+1}}(\ell_1^{j+1}-\ell_1^j)\le e^{3\eta}\operatorname{Var}_{x_1^j}(\ell_1^{j+1}-\ell_1^j) Var x 1 j + 1 ( ℓ 1 j + 1 − ℓ 1 j ) ≤ e 3 η Var x 1 j ( ℓ 1 j + 1 − ℓ 1 j ) , and summing over 1 ≤ j ≤ T − 1 1\le j\le T-1 1 ≤ j ≤ T − 1 proves (40) . For (41) , write L ( s ) = ℓ 1 j + ∫ j s L ′ ( u ) d u L(s)=\ell_1^j+\int_j^sL'(u)\,\mathrm{d}u L ( s ) = ℓ 1 j + ∫ j s L ′ ( u ) d u for s ∈ [ j , j + 1 ] s\in[j,j+1] s ∈ [ j , j + 1 ] ; the triangle inequality, ( α + β ) 2 ≤ 2 α 2 + 2 β 2 (\alpha+\beta)^2\le2\alpha^2+2\beta^2 ( α + β ) 2 ≤ 2 α 2 + 2 β 2 and the Cauchy–Schwarz inequality on the unit interval give ∫ j j + 1 Var x 1 j ( L ( s ) ) d s ≤ 2 Var x 1 j ( ℓ 1 j ) + 2 ∫ j j + 1 Var x 1 j ( L ′ ( s ) ) d s \int_j^{j+1}\operatorname{Var}_{x_1^j}(L(s))\,\mathrm{d}s\le2\operatorname{Var}_{x_1^j}(\ell_1^j)+2\int_j^{j+1}\operatorname{Var}_{x_1^j}(L'(s))\,\mathrm{d}s ∫ j j + 1 Var x 1 j ( L ( s )) d s ≤ 2 Var x 1 j ( ℓ 1 j ) + 2 ∫ j j + 1 Var x 1 j ( L ′ ( s )) d s , and summing over 1 ≤ j ≤ T − 1 1\le j\le T-1 1 ≤ j ≤ T − 1 proves (41) .
The second comparison (41) carries the derivative energy on its right-hand side, which is the quantity we are trying to bound. This is harmless, because after the windows are summed the derivative energy appears on the right only with a small coefficient and can be absorbed into the left-hand side (proof of Lemma 5.3).
Windows and the frozen curve. Fix an integer W ≥ 1 W\ge1 W ≥ 1 and an integer anchor t 0 ≥ 2 t_0\ge2 t 0 ≥ 2 with t 0 + 3 W ≤ T t_0+3W\le T t 0 + 3 W ≤ T (so that the sums below, which start at r = t 0 − 1 r=t_0-1 r = t 0 − 1 , involve only rounds r ≥ 1 r\ge1 r ≥ 1 ). Let J t 0 = [ t 0 , t 0 + 3 W ] J_{t_0}=[t_0,t_0+3W] J t 0 = [ t 0 , t 0 + 3 W ] be the window and I t 0 = [ t 0 + W , t 0 + 2 W ] I_{t_0}=[t_0+W,t_0+2W] I t 0 = [ t 0 + W , t 0 + 2 W ] its middle third, and measure the movement of player 2’s losses over the window by
δ t 0 = ∑ r = t 0 − 1 t 0 + 3 W − 2 ∥ ℓ 2 r + 1 − ℓ 2 r ∥ ∞ , u t 0 = ∑ r = t 0 − 1 t 0 + 3 W − 2 ∥ ℓ 2 r + 1 − ℓ 2 r ∥ ∞ 2 , so that δ t 0 2 ≤ 3 W u t 0 . \delta_{t_0}=\sum_{r=t_0-1}^{t_0+3W-2}\|\ell_2^{r+1}-\ell_2^r\|_\infty,\qquad
u_{t_0}=\sum_{r=t_0-1}^{t_0+3W-2}\|\ell_2^{r+1}-\ell_2^r\|_\infty^2,
\qquad\text{so that }\delta_{t_0}^2\le3W u_{t_0}. δ t 0 = r = t 0 − 1 ∑ t 0 + 3 W − 2 ∥ ℓ 2 r + 1 − ℓ 2 r ∥ ∞ , u t 0 = r = t 0 − 1 ∑ t 0 + 3 W − 2 ∥ ℓ 2 r + 1 − ℓ 2 r ∥ ∞ 2 , so that δ t 0 2 ≤ 3 W u t 0 .
Freezing player 2’s losses at ℓ 2 t 0 \ell_2^{t_0} ℓ 2 t 0 defines the comparison curves Y ˉ ( s ) = E x 2 t 0 [ − η ( s − t 0 ) ℓ 2 t 0 ] \bar{Y}(s)=\mathbb{E}_{x_2^{t_0}}[-\eta(s-t_0)\ell_2^{t_0}] Y ˉ ( s ) = E x 2 t 0 [ − η ( s − t 0 ) ℓ 2 t 0 ] and L ˉ ( s ) = A Y ˉ ( s ) \bar{L}(s)=A\bar{Y}(s) L ˉ ( s ) = A Y ˉ ( s ) for s ∈ J t 0 s\in J_{t_0} s ∈ J t 0 . For a curve F F F with values in R n \mathbb{R}^n R n , an interval I I I and a probability vector p p p , we write ∥ F ∥ L 2 ( I ; p ) = ( ∫ I ∥ F ( s ) ∥ p 2 d s ) 1 / 2 \lVert F\rVert_{L^2(I;p)}=\left(\int_I\lVert F(s)\rVert_p^2\,ds\right)^{1/2} ∥ F ∥ L 2 ( I ; p ) = ( ∫ I ∥ F ( s ) ∥ p 2 d s ) 1/2 .
Lemma 5.6 (Local energy estimate). There is an absolute constant C C C such that for all integers W ≥ 1 W\ge1 W ≥ 1 and all η \eta η with 0 < η W ≤ 10 − 3 0<\eta W\le10^{-3} 0 < η W ≤ 1 0 − 3 , every anchor t 0 t_0 t 0 as above satisfies
∫ I t 0 Var x 1 [ s ] ( L ′ ( s ) ) d s ≤ C n 2 W 2 ∫ J t 0 Var x 1 [ s ] ( L ( s ) ) d s + C η 2 n 2 W 2 u t 0 . (42) \int_{I_{t_0}}\operatorname{Var}_{x_1^{[s]}}(L'(s))\,ds\le C\frac{n^2}{W^2}\int_{J_{t_0}}\operatorname{Var}_{x_1^{[s]}}(L(s))\,ds+C\eta^2n^2W^2u_{t_0}. \tag*{(42)} ∫ I t 0 Var x 1 [ s ] ( L ′ ( s )) d s ≤ C W 2 n 2 ∫ J t 0 Var x 1 [ s ] ( L ( s )) d s + C η 2 n 2 W 2 u t 0 . (42)
The first term is what the frozen curve satisfies by Lemma 5.2 ; the second is the price of freezing.
Proof. The frozen curve. Write L ˉ = N / Z \bar{L}=N/Z L ˉ = N / Z with N ( s ) = ∑ b = 1 n x 2 t 0 ( b ) e − η ( s − t 0 ) ℓ 2 t 0 ( b ) A ⋅ b N(s)=\sum_{b=1}^n x_2^{t_0}(b)e^{-\eta(s-t_0)\ell_2^{t_0}(b)}A_{\cdot b} N ( s ) = ∑ b = 1 n x 2 t 0 ( b ) e − η ( s − t 0 ) ℓ 2 t 0 ( b ) A ⋅ b and Z ( s ) = ∑ b = 1 n x 2 t 0 ( b ) e − η ( s − t 0 ) ℓ 2 t 0 ( b ) Z(s)=\sum_{b=1}^n x_2^{t_0}(b)e^{-\eta(s-t_0)\ell_2^{t_0}(b)} Z ( s ) = ∑ b = 1 n x 2 t 0 ( b ) e − η ( s − t 0 ) ℓ 2 t 0 ( b ) , where A ⋅ b A_{\cdot b} A ⋅ b is the b b b -th column of A A A . Both are sums of n n n real exponentials in s s s . On J t 0 J_{t_0} J t 0 we have e − 3 η W ≤ Z ≤ 1 e^{-3\eta W}\le Z\le1 e − 3 η W ≤ Z ≤ 1 and ∣ Z ′ ( s ) ∣ ≤ η Z ( s ) \lvert Z'(s)\rvert\le\eta Z(s) ∣ Z ′ ( s )∣ ≤ η Z ( s ) , so the quotient rule L ˉ ′ = N ′ / Z − L ˉ Z ′ / Z \bar{L}'=N'/Z-\bar{L}Z'/Z L ˉ ′ = N ′ / Z − L ˉ Z ′ / Z and Lemma 5.2 (with α = t 0 \alpha=t_0 α = t 0 , in the seminorm ∥ ⋅ ∥ p \lVert\cdot\rVert_p ∥ ⋅ ∥ p of an arbitrary fixed p p p ) give
∥ L ˉ ′ ∥ L 2 ( I t 0 ; p ) ≤ e 3 η W ∥ N ′ ∥ L 2 ( I t 0 ; p ) + η ∥ L ˉ ∥ L 2 ( I t 0 ; p ) ≤ ( 2 n e 3 η W W + η ) ∥ L ˉ ∥ L 2 ( J t 0 ; p ) ≤ C n W ∥ L ˉ ∥ L 2 ( J t 0 ; p ) . (43) \lVert\bar{L}'\rVert_{L^2(I_{t_0};p)}\le e^{3\eta W}\lVert N'\rVert_{L^2(I_{t_0};p)}+\eta\lVert\bar{L}\rVert_{L^2(I_{t_0};p)}\le\left(\frac{2ne^{3\eta W}}{W}+\eta\right)\lVert\bar{L}\rVert_{L^2(J_{t_0};p)}\le C\frac{n}{W}\lVert\bar{L}\rVert_{L^2(J_{t_0};p)}. \tag*{(43)} ∥ L ˉ ′ ∥ L 2 ( I t 0 ; p ) ≤ e 3 η W ∥ N ′ ∥ L 2 ( I t 0 ; p ) + η ∥ L ˉ ∥ L 2 ( I t 0 ; p ) ≤ ( W 2 n e 3 η W + η ) ∥ L ˉ ∥ L 2 ( J t 0 ; p ) ≤ C W n ∥ L ˉ ∥ L 2 ( J t 0 ; p ) . (43)
where we used ∥ N ∥ L 2 ( J t 0 ; p ) ≤ ∥ L ˉ ∥ L 2 ( J t 0 ; p ) \lVert N\rVert_{L^2(J_{t_0};p)}\le\lVert\bar{L}\rVert_{L^2(J_{t_0};p)} ∥ N ∥ L 2 ( J t 0 ; p ) ≤ ∥ L ˉ ∥ L 2 ( J t 0 ; p ) (as Z ≤ 1 Z\le1 Z ≤ 1 ) and η W ≤ 10 − 3 \eta W\le10^{-3} η W ≤ 1 0 − 3 .
Freezing error in position. For j ∈ [ t 0 , t 0 + 3 W − 1 ] j\in[t_0,t_0+3W-1] j ∈ [ t 0 , t 0 + 3 W − 1 ] , since g j − ℓ 2 t 0 = ( ℓ 2 j − ℓ 2 t 0 ) + ( ℓ 2 j − ℓ 2 j − 1 ) g^j-\ell_2^{t_0}=(\ell_2^j-\ell_2^{t_0})+(\ell_2^j-\ell_2^{j-1}) g j − ℓ 2 t 0 = ( ℓ 2 j − ℓ 2 t 0 ) + ( ℓ 2 j − ℓ 2 j − 1 ) , we have ∥ g j − ℓ 2 t 0 ∥ ∞ ≤ 2 δ t 0 \lVert g^j-\ell_2^{t_0}\rVert_\infty\le2\delta_{t_0} ∥ g j − ℓ 2 t 0 ∥ ∞ ≤ 2 δ t 0 . Composing the tilts in (39) shows that for s ∈ [ j , j + 1 ] s\in[j,j+1] s ∈ [ j , j + 1 ] , Y ( s ) = E x 2 t 0 [ − η ( ∑ r = t 0 j − 1 g r + ( s − j ) g j ) ] Y(s)=\mathbb{E}_{x_2^{t_0}}[-\eta(\sum_{r=t_0}^{j-1}g^r+(s-j)g^j)] Y ( s ) = E x 2 t 0 [ − η ( ∑ r = t 0 j − 1 g r + ( s − j ) g j )] , so the scores defining Y ( s ) Y(s) Y ( s ) and Y ˉ ( s ) \bar{Y}(s) Y ˉ ( s ) differ by at most η ⋅ 3 W ⋅ 2 δ t 0 \eta\cdot3W\cdot2\delta_{t_0} η ⋅ 3 W ⋅ 2 δ t 0 in ℓ ∞ \ell_\infty ℓ ∞ . The Lipschitz bound of Lemma A.2 and ∥ A v ∥ ∞ ≤ ∥ v ∥ 1 \lVert Av\rVert_\infty\le\lVert v\rVert_1 ∥ A v ∥ ∞ ≤ ∥ v ∥ 1 give
∥ Y ( s ) − Y ˉ ( s ) ∥ 1 ≤ 6 η W δ t 0 , ∥ L ( s ) − L ˉ ( s ) ∥ ∞ ≤ 6 η W δ t 0 ( s ∈ J t 0 ) . (44) \lVert Y(s)-\bar{Y}(s)\rVert_1\le6\eta W\delta_{t_0},\qquad\lVert L(s)-\bar{L}(s)\rVert_\infty\le6\eta W\delta_{t_0}\qquad(s\in J_{t_0}). \tag*{(44)} ∥ Y ( s ) − Y ˉ ( s ) ∥ 1 ≤ 6 η W δ t 0 , ∥ L ( s ) − L ˉ ( s ) ∥ ∞ ≤ 6 η W δ t 0 ( s ∈ J t 0 ) . (44)
Freezing error in derivative. Let Σ ( p ) = diag ( p ) − p p ⊤ \Sigma(p)=\operatorname{diag}(p)-pp^\top Σ ( p ) = diag ( p ) − p p ⊤ , the derivative of the tilt with respect to its score. Then Y ′ ( s ) = − η Σ ( Y ( s ) ) g j Y'(s)=-\eta\Sigma(Y(s))g^j Y ′ ( s ) = − η Σ ( Y ( s )) g j and Y ˉ ′ ( s ) = − η Σ ( Y ˉ ( s ) ) ℓ 2 t 0 \bar{Y}'(s)=-\eta\Sigma(\bar{Y}(s))\ell_2^{t_0} Y ˉ ′ ( s ) = − η Σ ( Y ˉ ( s )) ℓ 2 t 0 for s ∈ ( j , j + 1 ) s\in(j,j+1) s ∈ ( j , j + 1 ) . For probability vectors p , q p,q p , q and any v v v ,
∥ Σ ( p ) v ∥ 1 = ∑ h p ( h ) ∣ v ( h ) − ⟨ p , v ⟩ ∣ ≤ ∥ v ∥ p ≤ ∥ v ∥ ∞ , ∥ ( Σ ( p ) − Σ ( q ) ) v ∥ 1 ≤ 3 ∥ p − q ∥ 1 ∥ v ∥ ∞ , \lVert\Sigma(p)v\rVert_1=\sum_h p(h)\lvert v(h)-\langle p,v\rangle\rvert\le\lVert v\rVert_p\le\lVert v\rVert_\infty,\qquad\lVert(\Sigma(p)-\Sigma(q))v\rVert_1\le3\lVert p-q\rVert_1\lVert v\rVert_\infty, ∥ Σ ( p ) v ∥ 1 = h ∑ p ( h ) ∣ v ( h ) − ⟨ p , v ⟩∣ ≤ ∥ v ∥ p ≤ ∥ v ∥ ∞ , ∥( Σ ( p ) − Σ ( q )) v ∥ 1 ≤ 3 ∥ p − q ∥ 1 ∥ v ∥ ∞ ,
the second because Σ ( p ) v − Σ ( q ) v = ( p − q ) ⊙ v − ( p − q ) ⟨ p , v ⟩ − q ⟨ p − q , v ⟩ \Sigma(p)v-\Sigma(q)v=(p-q)\odot v-(p-q)\langle p,v\rangle-q\langle p-q,v\rangle Σ ( p ) v − Σ ( q ) v = ( p − q ) ⊙ v − ( p − q ) ⟨ p , v ⟩ − q ⟨ p − q , v ⟩ . Writing Σ ( Y ) g j − Σ ( Y ˉ ) ℓ 2 t 0 = Σ ( Y ) ( g j − ℓ 2 t 0 ) + ( Σ ( Y ) − Σ ( Y ˉ ) ) ℓ 2 t 0 \Sigma(Y)g^j-\Sigma(\bar{Y})\ell_2^{t_0}=\Sigma(Y)(g^j-\ell_2^{t_0})+(\Sigma(Y)-\Sigma(\bar{Y}))\ell_2^{t_0} Σ ( Y ) g j − Σ ( Y ˉ ) ℓ 2 t 0 = Σ ( Y ) ( g j − ℓ 2 t 0 ) + ( Σ ( Y ) − Σ ( Y ˉ )) ℓ 2 t 0 and using (44) ,
∥ L ′ ( s ) − L ˉ ′ ( s ) ∥ ∞ ≤ ∥ Y ′ ( s ) − Y ˉ ′ ( s ) ∥ 1 ≤ η ( 2 δ t 0 + 18 η W δ t 0 ) ≤ 3 η δ t 0 . (45) \lVert L'(s)-\bar{L}'(s)\rVert_\infty\le\lVert Y'(s)-\bar{Y}'(s)\rVert_1\le\eta(2\delta_{t_0}+18\eta W\delta_{t_0})\le3\eta\delta_{t_0}. \tag*{(45)} ∥ L ′ ( s ) − L ˉ ′ ( s ) ∥ ∞ ≤ ∥ Y ′ ( s ) − Y ˉ ′ ( s ) ∥ 1 ≤ η ( 2 δ t 0 + 18 η W δ t 0 ) ≤ 3 η δ t 0 . (45)
The absence of a factor W W W in (45) is essential; it is why the derivative error is estimated directly rather than inferred from (44) .
Assembling the window. Freeze the variance weights at p = x 1 t 0 p=x_1^{t_0} p = x 1 t 0 . By (43) , (44) , (45) and ∥ ⋅ ∥ p ≤ ∥ ⋅ ∥ ∞ \lVert\cdot\rVert_p\le\lVert\cdot\rVert_\infty ∥ ⋅ ∥ p ≤ ∥ ⋅ ∥ ∞ ,
∥ L ′ ∥ L 2 ( I t 0 ; p ) ≤ ∥ L ˉ ′ ∥ L 2 ( I t 0 ; p ) + W ⋅ 3 η δ t 0 ≤ C n W ( ∥ L ∥ L 2 ( J t 0 ; p ) + 3 W ⋅ 6 η W δ t 0 ) + 3 η W δ t 0 ≤ C ′ n W ∥ L ∥ L 2 ( J t 0 ; p ) + C ′ η n W δ t 0 . \begin{aligned}
\lVert L'\rVert_{L^2(I_{t_0};p)}\le\lVert\bar{L}'\rVert_{L^2(I_{t_0};p)}+\sqrt{W}\cdot3\eta\delta_{t_0}\le C\frac{n}{W}\left(\lVert L\rVert_{L^2(J_{t_0};p)}+\sqrt{3W}\cdot6\eta W\delta_{t_0}\right)+3\eta\sqrt{W}\delta_{t_0} \\
&\le C'\frac{n}{W}\lVert L\rVert_{L^2(J_{t_0};p)}+C'\eta n\sqrt{W}\delta_{t_0}.
\end{aligned} ∥ L ′ ∥ L 2 ( I t 0 ; p ) ≤ ∥ L ˉ ′ ∥ L 2 ( I t 0 ; p ) + W ⋅ 3 η δ t 0 ≤ C W n ( ∥ L ∥ L 2 ( J t 0 ; p ) + 3 W ⋅ 6 η W δ t 0 ) + 3 η W δ t 0 ≤ C ′ W n ∥ L ∥ L 2 ( J t 0 ; p ) + C ′ η n W δ t 0 .
By Lemma 5.1 , along the window the true weights x 1 [ s ] x_1^{[s]} x 1 [ s ] and x 1 t 0 x_1^{t_0} x 1 t 0 differ coordinatewise by a factor at most e 9 η W ≤ e 0.009 e^{9\eta W}\le e^{0.009} e 9 η W ≤ e 0.009 , so the variances with the true weights are within a constant factor of those with p = x 1 t 0 p=x_1^{t_0} p = x 1 t 0 , on both sides of the inequality. Squaring, using ( α + β ) 2 ≤ 2 α 2 + 2 β 2 (\alpha+\beta)^2\le2\alpha^2+2\beta^2 ( α + β ) 2 ≤ 2 α 2 + 2 β 2 and δ t 0 2 ≤ 3 W u t 0 \delta_{t_0}^2\le3Wu_{t_0} δ t 0 2 ≤ 3 W u t 0 , and enlarging the constant, we obtain (42) .
Summing over windows.
Proof of Lemma 5.3. Let C C C be the constant of Lemma 5.6. Let T = { 2 + k W : k ≥ 0 an integer , 2 + ( k + 3 ) W ≤ T } \mathcal{T}=\{2+kW:k\geq0\text{ an integer},\ 2+(k+3)W\leq T\} T = { 2 + k W : k ≥ 0 an integer , 2 + ( k + 3 ) W ≤ T } be the set of anchors. The middle thirds I t 0 = [ t 0 + W , t 0 + 2 W ] I_{t_0}=[t_0+W,t_0+2W] I t 0 = [ t 0 + W , t 0 + 2 W ] , t 0 ∈ T t_0\in\mathcal{T} t 0 ∈ T , are consecutive and have disjoint interiors; the windows J t 0 J_{t_0} J t 0 overlap at most threefold, as do the index ranges { t 0 − 1 , … , t 0 + 3 W − 2 } \{t_0-1,\ldots,t_0+3W-2\} { t 0 − 1 , … , t 0 + 3 W − 2 } of the sums u t 0 u_{t_0} u t 0 , which all lie within { 1 , … , T − 1 } \{1,\ldots,T-1\} { 1 , … , T − 1 } . By (35) and ( α + β ) 2 ≤ 2 α 2 + 2 β 2 (\alpha+\beta)^2\leq2\alpha^2+2\beta^2 ( α + β ) 2 ≤ 2 α 2 + 2 β 2 ,
∑ t 0 ∈ T u t 0 ≤ 3 ∑ t = 1 T − 1 ∥ ℓ 2 t + 1 − ℓ 2 t ∥ 2 2 ≤ 3 ∑ t = 1 T − 1 9 η 2 ( v t + v t − 1 ) 2 ≤ 108 η 2 ∑ t = 1 T Var x 1 t ( ℓ 1 t ) . \sum_{t_0\in\mathcal{T}}u_{t_0}\leq3\sum_{t=1}^{T-1}\left\|\ell_2^{t+1}-\ell_2^t\right\|_2^2\leq3\sum_{t=1}^{T-1}9\eta^2(v^t+v^{t-1})^2\leq108\eta^2\sum_{t=1}^{T}\operatorname{Var}_{x_1^t}(\ell_1^t). t 0 ∈ T ∑ u t 0 ≤ 3 t = 1 ∑ T − 1 ℓ 2 t + 1 − ℓ 2 t 2 2 ≤ 3 t = 1 ∑ T − 1 9 η 2 ( v t + v t − 1 ) 2 ≤ 108 η 2 t = 1 ∑ T Var x 1 t ( ℓ 1 t ) .
Summing (42) over t 0 ∈ T t_0\in\mathcal{T} t 0 ∈ T and using the threefold overlap of the windows,
∑ t 0 ∈ T ∫ I t 0 Var x 1 [ s ] ( L ′ ( s ) ) d s ≤ 3 C n 2 W 2 ∫ 1 T Var x 1 [ s ] ( L ( s ) ) d s + 108 C η 4 n 2 W 2 ∑ t = 1 T Var x 1 t ( ℓ 1 t ) . \sum_{t_0\in\mathcal{T}}\int_{I_{t_0}}\operatorname{Var}_{x_1^{[s]}}(L'(s))\,\mathrm{d}s\leq3C\frac{n^2}{W^2}\int_1^T\operatorname{Var}_{x_1^{[s]}}(L(s))\,\mathrm{d}s+108C\eta^4n^2W^2\sum_{t=1}^{T}\operatorname{Var}_{x_1^t}(\ell_1^t). t 0 ∈ T ∑ ∫ I t 0 Var x 1 [ s ] ( L ′ ( s )) d s ≤ 3 C W 2 n 2 ∫ 1 T Var x 1 [ s ] ( L ( s )) d s + 108 C η 4 n 2 W 2 t = 1 ∑ T Var x 1 t ( ℓ 1 t ) .
The part of [ 1 , T ] [1,T] [ 1 , T ] not covered by the middle thirds consists of an initial segment of length at most W + 1 W+1 W + 1 and a final segment of length less than 2 W 2W 2 W . Since g j ∈ [ − 1 , 2 ] n g^j\in[-1,2]^n g j ∈ [ − 1 , 2 ] n , the interpolation satisfies ∥ L ′ ( s ) ∥ ∞ ≤ ∥ Y ′ ( s ) ∥ 1 ≤ η ∥ g j ∥ ∞ ≤ 2 η \|L'(s)\|_\infty\leq\|Y'(s)\|_1\leq\eta\|g^j\|_\infty\leq2\eta ∥ L ′ ( s ) ∥ ∞ ≤ ∥ Y ′ ( s ) ∥ 1 ≤ η ∥ g j ∥ ∞ ≤ 2 η almost everywhere, so the uncovered part contributes at most 4 W ⋅ 4 η 2 = 16 η 2 W 4W\cdot4\eta^2=16\eta^2W 4 W ⋅ 4 η 2 = 16 η 2 W to ∫ 1 T Var x 1 [ s ] ( L ′ ( s ) ) d s \int_1^T\operatorname{Var}_{x_1^{[s]}}(L'(s))\,\mathrm{d}s ∫ 1 T Var x 1 [ s ] ( L ′ ( s )) d s ; this also covers horizons too short to contain a single window. Altogether,
∫ 1 T Var x 1 [ s ] ( L ′ ( s ) ) d s ≤ C ′ n 2 W 2 ∫ 1 T Var x 1 [ s ] ( L ( s ) ) d s + C ′ η 4 n 2 W 2 ∑ t = 1 T Var x 1 t ( ℓ 1 t ) + 16 η 2 W . \int_1^T\operatorname{Var}_{x_1^{[s]}}(L'(s))\,\mathrm{d}s\leq C'\frac{n^2}{W^2}\int_1^T\operatorname{Var}_{x_1^{[s]}}(L(s))\,\mathrm{d}s+C'\eta^4n^2W^2\sum_{t=1}^{T}\operatorname{Var}_{x_1^t}(\ell_1^t)+16\eta^2W. ∫ 1 T Var x 1 [ s ] ( L ′ ( s )) d s ≤ C ′ W 2 n 2 ∫ 1 T Var x 1 [ s ] ( L ( s )) d s + C ′ η 4 n 2 W 2 t = 1 ∑ T Var x 1 t ( ℓ 1 t ) + 16 η 2 W .
Now insert the second comparison (41) of Lemma 5.5, which bounds the position energy ∫ 1 T Var x 1 [ s ] ( L ( s ) ) d s \int_1^T\operatorname{Var}_{x_1^{[s]}}(L(s))\,\mathrm{d}s ∫ 1 T Var x 1 [ s ] ( L ( s )) d s by 2 ∑ t = 1 T Var x 1 t ( ℓ 1 t ) + 2 ∫ 1 T Var x 1 [ s ] ( L ′ ( s ) ) d s 2\sum_{t=1}^T\operatorname{Var}_{x_1^t}(\ell_1^t)+2\int_1^T\operatorname{Var}_{x_1^{[s]}}(L'(s))\,\mathrm{d}s 2 ∑ t = 1 T Var x 1 t ( ℓ 1 t ) + 2 ∫ 1 T Var x 1 [ s ] ( L ′ ( s )) d s , and choose K 0 K_0 K 0 so large that W ≥ K 0 n W\geq K_0n W ≥ K 0 n makes 2 C ′ n 2 / W 2 ≤ 1 2 2C'n^2/W^2\leq\frac12 2 C ′ n 2 / W 2 ≤ 2 1 . Absorbing the resulting 1 2 ∫ 1 T Var x 1 [ s ] ( L ′ ( s ) ) d s \frac12\int_1^T\operatorname{Var}_{x_1^{[s]}}(L'(s))\,\mathrm{d}s 2 1 ∫ 1 T Var x 1 [ s ] ( L ′ ( s )) d s into the left-hand side gives
∫ 1 T Var x 1 [ s ] ( L ′ ( s ) ) d s ≤ C ′ ′ [ ( n 2 W 2 + η 4 n 2 W 2 ) ∑ t = 1 T Var x 1 t ( ℓ 1 t ) + η 2 W ] . \int_1^T\operatorname{Var}_{x_1^{[s]}}(L'(s))\,\mathrm{d}s\leq C''\left[\left(\frac{n^2}{W^2}+\eta^4n^2W^2\right)\sum_{t=1}^{T}\operatorname{Var}_{x_1^t}(\ell_1^t)+\eta^2W\right]. ∫ 1 T Var x 1 [ s ] ( L ′ ( s )) d s ≤ C ′′ [ ( W 2 n 2 + η 4 n 2 W 2 ) t = 1 ∑ T Var x 1 t ( ℓ 1 t ) + η 2 W ] .
Finally, by (34) ,
∑ t = 1 T Var x 1 t ( ℓ 1 t ) ≤ e 3 η ∑ t = 1 T Var x 1 t ( ℓ 1 t − 1 ) + 1 4 ≤ 2 ( ∑ t = 1 T Var x 1 t ( ℓ 1 t − 1 ) + 1 ) , \sum_{t=1}^{T}\operatorname{Var}_{x_1^t}(\ell_1^t)\leq e^{3\eta}\sum_{t=1}^{T}\operatorname{Var}_{x_1^t}(\ell_1^{t-1})+\frac14\leq2\left(\sum_{t=1}^{T}\operatorname{Var}_{x_1^t}(\ell_1^{t-1})+1\right), t = 1 ∑ T Var x 1 t ( ℓ 1 t ) ≤ e 3 η t = 1 ∑ T Var x 1 t ( ℓ 1 t − 1 ) + 4 1 ≤ 2 ( t = 1 ∑ T Var x 1 t ( ℓ 1 t − 1 ) + 1 ) ,
and by (40) ,
∑ t = 1 T Var x 1 t ( ℓ 1 t − ℓ 1 t − 1 ) ≤ 1 4 + e 3 η ∫ 1 T Var x 1 [ s ] ( L ′ ( s ) ) d s ≤ 1 4 + 2 ∫ 1 T Var x 1 [ s ] ( L ′ ( s ) ) d s ; \sum_{t=1}^{T}\operatorname{Var}_{x_1^t}(\ell_1^t-\ell_1^{t-1})\leq\frac14+e^{3\eta}\int_1^T\operatorname{Var}_{x_1^{[s]}}(L'(s))\,\mathrm{d}s\leq\frac14+2\int_1^T\operatorname{Var}_{x_1^{[s]}}(L'(s))\,\mathrm{d}s; t = 1 ∑ T Var x 1 t ( ℓ 1 t − ℓ 1 t − 1 ) ≤ 4 1 + e 3 η ∫ 1 T Var x 1 [ s ] ( L ′ ( s )) d s ≤ 4 1 + 2 ∫ 1 T Var x 1 [ s ] ( L ′ ( s )) d s ;
combining the last three displays proves (37) .
Proof of the exponential regret lower bound for three or more players
This section constructs the three-player game of Theorem 3.6 . Players 1 and 2 play a symmetric game against each other, start identically and remain identical, and their common trajectory converges to a periodic orbit. Player 3 is the observer ; her losses depend on player 1’s action only, and she does not influence players 1 and 2. We first analyse a single eight-action block of players 1 and 2 (§ 7.1 ), then the linear response of a block driven by its predecessor (§ 9.1 ), then the observer (§ 10.1 ); § 10.5 assembles all blocks into one cascade and proves its convergence, and § 10.6 proves the theorem. Finally, § 10.7 extends the construction to m ≥ 3 m \ge3 m ≥ 3 players, with a barrier 2 Ω ( m n ) 2^{\Omega(mn)} 2 Ω ( mn ) (Theorem 6.24 ). Figure 3 shows the overall architecture.
Conventions. It is convenient to describe all payoffs in this section as utilities U ∈ [ − 1 2 , 1 2 ] U \in[-\frac{1}{2},\frac{1}{2}] U ∈ [ − 2 1 , 2 1 ] , rather than as losses in [ 0 , 1 ] [0,1] [ 0 , 1 ] as in § 2 ; the corresponding losses are 1 2 − U ∈ [ 0 , 1 ] \frac{1}{2}-U \in[0,1] 2 1 − U ∈ [ 0 , 1 ] . Since Optimistic Hedge is invariant under adding a constant to all entries of a loss vector, only the differences between utilities matter, and we write the losses simply as − U -U − U . Throughout, n n n is the number of actions of each player, M ≥ 4 M \ge4 M ≥ 4 is an integer divisible by four (the period), λ = e 2 π i / M \lambda= e^{2\pi i/M} λ = e 2 π i / M , and a > 0 a > 0 a > 0 is a small amplitude. The letters μ , δ ∈ C \mu,\delta\in\mathbb{C} μ , δ ∈ C denote payoff parameters that are fixed before play begins. Since the Optimistic Hedge update (3) uses the previous loss vector ℓ t − 1 \ell^{t-1} ℓ t − 1 as well as the current one, its state consists of the current strategy and the previous round’s utilities; we call the latter the memory and denote it by ϱ t \varrho_t ϱ t . Starred quantities are exact periodic trajectories.
A single oscillating block
Overview. We construct a symmetric two-player game whose common strategy is determined by a complex score z t z_t z t . Our goal is that the score will approximately rotate in a cycle of length M M M around some center point δ \delta δ with a radius of a a a :
z t ≈ δ + a e 2 π i t / M . z_t \approx\delta+ ae^{2\pi it/M}. z t ≈ δ + a e 2 π i t / M .
Step 1: actions, utilities and the exact score recurrence
The two players have the same n n n distinct actions. The eight active actions are named by the complex numbers
V = { 1 , − 1 , i , − i , 1 + i , − 1 − i , 1 − i , − 1 + i } . V = \{1,-1,i,-i,1+i,-1-i,1-i,-1+i\}. V = { 1 , − 1 , i , − i , 1 + i , − 1 − i , 1 − i , − 1 + i } .
The remaining n − 8 n-8 n − 8 actions have distinct names b 1 , … , b n − 8 b_1,\ldots,b_{n-8} b 1 , … , b n − 8 ; we call them neutral . These neutral actions will later serve as the actions of the other blocks (§ 10.5 ); here they have utility zero and do not control the score of this block. Each action τ \tau τ has a complex coordinate w ( τ ) w(\tau) w ( τ ) : the complex number naming it if τ \tau τ is active, and w ( b j ) = 0 w(b_j)=0 w ( b j ) = 0 if it is neutral. Below we write v v v for the coordinate of a player’s own action and w w w for that of the opponent’s action. Sums over actions still run over all n n n actions, so each neutral action contributes separately even though they all have coordinate 0.
The game will have the following property. There is a complex score z t z_t z t , with z 0 = 0 z_0=0 z 0 = 0 , such that both players assign to each action τ \tau τ the real probability
p z ( τ ) = exp ( Re ( w ( τ ) ‾ z ) ) Z ( z ) , Z ( z ) = ∑ τ exp ( Re ( w ( τ ) ‾ z ) ) . (46) p_z(\tau)=\frac{\exp(\operatorname{Re}(\overline{w(\tau)}z))}{Z(z)},\qquad Z(z)=\sum_{\tau}\exp(\operatorname{Re}(\overline{w(\tau)}z)). \tag*{(46)} p z ( τ ) = Z ( z ) exp ( Re ( w ( τ ) z )) , Z ( z ) = τ ∑ exp ( Re ( w ( τ ) z )) . (46)
If z = z 1 + i z 2 z = z_1 + iz_2 z = z 1 + i z 2 and w = w 1 + i w 2 w = w_1 + iw_2 w = w 1 + i w 2 , then Re ( w ‾ z ) = w 1 z 1 + w 2 z 2 ∈ R \operatorname{Re}(\overline{w}z) = w_1z_1 + w_2z_2 \in\mathbb{R} Re ( w z ) = w 1 z 1 + w 2 z 2 ∈ R . Hence these are positive real probabilities summing to one; Z ( 0 ) = n Z(0) = n Z ( 0 ) = n and p 0 p_0 p 0 is uniform. Figure 5 shows the eight active actions. Define c ( 1 ) = 1 c(1) = 1 c ( 1 ) = 1 , c ( − 1 ) = − 1 c(-1) = -1 c ( − 1 ) = − 1 , c ( i ) = i c(i) = i c ( i ) = i , c ( − i ) = − i c(-i) = -i c ( − i ) = − i , and c = 0 c = 0 c = 0 on diagonal and neutral actions. Let μ ∈ C \mu\in\mathbb{C} μ ∈ C be a payoff coefficient and δ ∈ C \delta\in\mathbb{C} δ ∈ C a shift parameter, to be chosen later and then held fixed throughout play. Both players use the utility table
Figure 5. The eight active actions V = { ± 1 , ± i , ± 1 ± i } V = \{\pm1, \pm i, \pm1 \pm i\} V = { ± 1 , ± i , ± 1 ± i } as points of C \mathbb{C} C , labeled by their payoff coefficients; neutral actions sit at 0 0 0 and have coefficient 0 0 0 . Left: c ( w ) = w c(w) = w c ( w ) = w on the axis actions, so the expectation of c c c is linear in the score to first order and produces the rotation of § 7.1 . Right: q ( b ) = b 2 q(b) = b^2 q ( b ) = b 2 on the axis actions and b 2 / 4 b^2/4 b 2 /4 on the diagonals, so the expectation of q q q is the square of the score’s displacement to leading order (Lemma 6.13).
U ( v , w ) = Re ( v ‾ μ c ( w ) e − Re ( w ‾ δ ) ) . (47) U(v,w)=\operatorname{Re}\left(\overline{v}\mu c(w)e^{-\operatorname{Re}(\overline{w}\delta)}\right). \tag*{(47)} U ( v , w ) = Re ( v μ c ( w ) e − Re ( w δ ) ) . (47)
The losses are − U -U − U . Define the expected complex utility coefficient
g μ , δ ( z ) = ∑ w p z ( w ) μ c ( w ) e − Re ( w ‾ δ ) Z ( z ) = 2 μ Z ( z ) [ sinh ( Re ( z − δ ) ) + i sinh ( Im ( z − δ ) ) ] . (48) \begin{aligned}
g_{\mu,\delta}(z)=\sum_w \frac{p_z(w)\mu c(w)e^{-\operatorname{Re}(\overline{w}\delta)}}{Z(z)} \\
&=\frac{2\mu}{Z(z)}\left[\sinh(\operatorname{Re}(z-\delta))+i\sinh(\operatorname{Im}(z-\delta))\right]. \tag*{(48)}
\end{aligned} g μ , δ ( z ) = w ∑ Z ( z ) p z ( w ) μ c ( w ) e − Re ( w δ ) = Z ( z ) 2 μ [ sinh ( Re ( z − δ )) + i sinh ( Im ( z − δ )) ] . (48)
Here and below p z ( w ) p_z(w) p z ( w ) in a sum means the probability of the corresponding action label. In particular g μ , δ ( δ ) = 0 g_{\mu,\delta}(\delta)=0 g μ , δ ( δ ) = 0 . The memory is ϱ t = g μ , δ ( z t − 1 ) \varrho_t=g_{\mu,\delta}(z_{t-1}) ϱ t = g μ , δ ( z t − 1 ) , the utility coefficient of the previous round, with ϱ 0 = 0 \varrho_0=0 ϱ 0 = 0 ; it enters the update through the term − ℓ t − 1 -\ell^{t-1} − ℓ t − 1 in (3) . The score and memory then follow the recurrence
z t + 1 = z t + η [ 2 g μ , δ ( z t ) − ϱ t ] , z_{t+1}=z_t+\eta[2g_{\mu,\delta}(z_t)-\varrho_t], z t + 1 = z t + η [ 2 g μ , δ ( z t ) − ϱ t ] ,
ϱ t + 1 = g μ , δ ( z t ) , ( z 0 , ϱ 0 ) = ( 0 , 0 ) . (49) \varrho_{t+1}=g_{\mu,\delta}(z_t),\qquad(z_0,\varrho_0)=(0,0). \tag*{(49)} ϱ t + 1 = g μ , δ ( z t ) , ( z 0 , ϱ 0 ) = ( 0 , 0 ) . (49)
Denote its one-step map on C 2 ≅ R 4 \mathbb{C}^2\cong\mathbb{R}^4 C 2 ≅ R 4 by Φ μ , δ \Phi_{\mu,\delta} Φ μ , δ .
Lemma 6.1 (The game realizes the recurrence). For utilities (47) , two identical players using Optimistic Hedge from the uniform distribution have the same strategy p z t p_{z_t} p z t at every round, where ( z t , ϱ t ) (z_t,\varrho_t) ( z t , ϱ t ) follows (49) . The expected utility of action v v v against p z t p_{z_t} p z t is Re ( v ‾ g μ , δ ( z t ) ) \operatorname{Re}(\overline{v}g_{\mu,\delta}(z_t)) Re ( v g μ , δ ( z t )) .
Proof. The formula for expected utility follows by averaging (47) ; the four nonzero coefficients c ( w ) c(w) c ( w ) give (48) . If γ t = g μ , δ ( z t ) \gamma_t=g_{\mu,\delta}(z_t) γ t = g μ , δ ( z t ) , the utility form of Optimistic Hedge gives
p t + 1 ( v ) ∝ p z t ( v ) e η Re ( v ‾ ( 2 γ t − ϱ t ) ) ∝ e Re ( v ‾ [ z t + η ( 2 γ t − ϱ t ) ] ) . p_{t+1}(v)\propto p_{z_t}(v)e^{\eta\operatorname{Re}(\overline{v}(2\gamma_t-\varrho_t))}\propto e^{\operatorname{Re}(\overline{v}[z_t+\eta(2\gamma_t-\varrho_t)])}. p t + 1 ( v ) ∝ p z t ( v ) e η Re ( v ( 2 γ t − ϱ t )) ∝ e Re ( v [ z t + η ( 2 γ t − ϱ t )]) .
Action-independent terms cancel on normalization. Thus the next strategy again has form (46) , with precisely the updated score in (49) . Initially the strategy is p 0 p_0 p 0 and the memory is zero, since the initial prediction of Optimistic Hedge is the same for all actions. Symmetry and identical initialization keep the two players identical by induction.
Theorem 6.2 (An oscillating block). Let n ≥ 9 n\ge9 n ≥ 9 , let M ≥ 4 M\ge4 M ≥ 4 be divisible by four, and suppose 0 < η ≤ 1 / 100 0<\eta\le1/100 0 < η ≤ 1/100 and η M ≥ 16 π n \eta M\ge16\pi n η M ≥ 16 π n . For every sufficiently small a > 0 a>0 a > 0 , there are complex parameters μ , δ \mu,\delta μ , δ , with δ = O ( a ) \delta=O(a) δ = O ( a ) , for which (47) lies in [ − 1 2 , 1 2 ] \left[-\frac{1}{2},\frac{1}{2}\right] [ − 2 1 , 2 1 ] and (49) has an orbit s t ∗ = ( z t ∗ , ϱ t ∗ ) s_t^*=(z_t^*,\varrho_t^*) s t ∗ = ( z t ∗ , ϱ t ∗ ) of exact period M M M satisfying
z t ∗ = δ + a λ t + O ( a 2 ) , λ = e 2 π i / M . z_t^*=\delta+a\lambda^t+O(a^2),\qquad\lambda=e^{2\pi i/M}. z t ∗ = δ + a λ t + O ( a 2 ) , λ = e 2 π i / M . The remainder is M M M -periodic. The trajectory from ( 0 , 0 ) (0,0) ( 0 , 0 ) satisfies
∣ z t − z t ∗ ∣ + ∣ ϱ t − ϱ t ∗ ∣ ≤ C a ρ t ( t ≥ 0 ) |z_t-z_t^*|+|\varrho_t-\varrho_t^*|\le Ca\rho^t\qquad(t\ge0) ∣ z t − z t ∗ ∣ + ∣ ϱ t − ϱ t ∗ ∣ ≤ C a ρ t ( t ≥ 0 ) for some C < ∞ C<\infty C < ∞ and ρ < 1 \rho<1 ρ < 1 , independent of t t t and of sufficiently small a a a .
Steps 2 and 3 construct an approximate rotation and close it into an exact cycle by adjusting the utility coefficient μ \mu μ . Steps 4 and 5 find the manifold of states attracted to this cycle and adjust δ \delta δ so that the uniform start lies on it. Step 6 checks the utilities. The assumption η M ≥ 16 π n \eta M\ge16\pi n η M ≥ 16 π n says that the rotation is slow: rotating by 2 π / M 2\pi/M 2 π / M per round takes a utility coefficient ∣ μ ∣ ≈ π n / ( η M ) |\mu|\approx\pi n/(\eta M) ∣ μ ∣ ≈ π n / ( η M ) , and bounded utilities cap it (Step 6). Figure 6 shows the resulting cycle.
Figure 6. The oscillating block of Theorem 6.2 over one period of M M M rounds, to leading order in the amplitude a a a . Left: the common score z t z_t z t of players 1 and 2 rotates once around the center δ \delta δ , with radius a a a and λ = e 2 π i / M \lambda=e^{2\pi i/M} λ = e 2 π i / M ; the uniform start is the point 0 0 0 . Middle and right: the log-probabilities of the four axis actions w ∈ { ± 1 , ± i } w\in\{\pm1,\pm i\} w ∈ { ± 1 , ± i } and of the four diagonal actions w ∈ { ± 1 ± i } w\in\{\pm1\pm i\} w ∈ { ± 1 ± i } , in units of a a a . They are linear in the score, log p t ( w ) = Re ( w ‾ z t ) − log Z ( z t ) \log p_t(w)=\operatorname{Re}(\overline{w}z_t)-\log Z(z_t) log p t ( w ) = Re ( w z t ) − log Z ( z t ) , with log Z ( z t ) = log n + O ( a 2 ) \log Z(z_t)=\log n+O(a^2) log Z ( z t ) = log n + O ( a 2 ) common to all actions, so each is a cosine a ∣ w ∣ cos ( 2 π t / M − arg w ) a|w|\cos(2\pi t/M-\arg w) a ∣ w ∣ cos ( 2 π t / M − arg w ) , up to the constant Re ( w ‾ δ ) \operatorname{Re}(\overline{w}\delta) Re ( w δ ) , which is omitted. Each action is most likely when z t − δ z_t-\delta z t − δ points in its direction, so within each group the curves peak a quarter period apart; the diagonal actions have a 2 \sqrt{2} 2 times larger amplitude.
For the rest of the proof, n n n , η \eta η , M M M are fixed. Constants and smallness thresholds may depend on them, but are uniform in time and in sufficiently small amplitude and shift. We use the Euclidean norm on C 2 ≅ R 4 \mathbb{C}^2\cong\mathbb{R}^4 C 2 ≅ R 4 unless another norm is specified. For a periodic sequence, O ( a 2 ) O(a^2) O ( a 2 ) means a bound uniform over all M M M phases, hence over all time.
Step 2: a circular approximation from the linearization
We first choose μ \mu μ so that the linearized score rotates once in M M M rounds. Define
β = λ ( λ − 1 ) 2 λ − 1 , χ = λ − 1 2 λ − 1 , μ 0 = η β 2 η , μ ∗ ( δ ) = Z ( δ ) β 2 η . (50) \beta= \frac{\lambda(\lambda-1)}{2\lambda-1}, \qquad\chi= \frac{\lambda-1}{2\lambda-1}, \qquad\mu_0 = \frac{\eta\beta}{2\eta}, \qquad\mu_*(\delta) = \frac{Z(\delta)\beta}{2\eta}. \tag*{(50)} β = 2 λ − 1 λ ( λ − 1 ) , χ = 2 λ − 1 λ − 1 , μ 0 = 2 η η β , μ ∗ ( δ ) = 2 η Z ( δ ) β . (50)
Here β \beta β selects the rotating mode, χ \chi χ will be the second, contracting eigenvalue of the linearized update, and μ ∗ ( δ ) \mu_*(\delta) μ ∗ ( δ ) is the reference utility coefficient at center δ \delta δ , with μ ∗ ( 0 ) = μ 0 \mu_*(0)=\mu_0 μ ∗ ( 0 ) = μ 0 .
Lemma 6.3 (Linear rotation). For small δ \delta δ and μ = μ ∗ ( δ ) \mu=\mu_*(\delta) μ = μ ∗ ( δ ) , the derivative of Φ μ , δ \Phi_{\mu,\delta} Φ μ , δ at its fixed point ( δ , 0 ) (\delta,0) ( δ , 0 ) is
A = ( 1 + 2 β − η β / η 0 ) , A = \begin{pmatrix} 1+2\beta& -\eta\\ \beta/\eta& 0 \end{pmatrix}, A = ( 1 + 2 β β / η − η 0 ) , with eigenvalues λ \lambda λ and χ \chi χ , where ∣ χ ∣ < 1 |\chi|<1 ∣ χ ∣ < 1 . In particular the trial states s ~ t = ( z ~ t , ϱ ~ t ) \widetilde{s}_t=(\widetilde{z}_t,\widetilde{\varrho}_t) s t = ( z t , ϱ t ) , with z ~ t = δ + a λ t \widetilde{z}_t=\delta+a\lambda^t z t = δ + a λ t and ϱ ~ t = a β η λ t − 1 \widetilde{\varrho}_t=\frac{a\beta}{\eta}\lambda^{t-1} ϱ t = η a β λ t − 1 , satisfy Φ μ , δ ( s ~ t ) − s ~ t + 1 = O ( a 2 ) \Phi_{\mu,\delta}(\widetilde{s}_t)-\widetilde{s}_{t+1}=O(a^2) Φ μ , δ ( s t ) − s t + 1 = O ( a 2 ) .
Proof. Taylor’s theorem at z = δ z=\delta z = δ gives g μ , δ ( δ + u ) = 2 μ Z ( δ ) u + O ( ∣ u ∣ 2 ) g_{\mu,\delta}(\delta+u)=\frac{2\mu}{Z(\delta)}u+O(|u|^2) g μ , δ ( δ + u ) = Z ( δ ) 2 μ u + O ( ∣ u ∣ 2 ) , and 2 η μ ∗ ( δ ) / Z ( δ ) = β 2\eta\mu_*(\delta)/Z(\delta)=\beta 2 η μ ∗ ( δ ) / Z ( δ ) = β ; this gives A A A . Its characteristic polynomial is ( r − λ ) ( r − χ ) (r-\lambda)(r-\chi) ( r − λ ) ( r − χ ) , and its λ \lambda λ -eigenvector is ( 1 , β / ( η λ ) ) (1,\beta/(\eta\lambda)) ( 1 , β / ( η λ )) , so s ~ t − ( δ , 0 ) \widetilde{s}_t-(\delta,0) s t − ( δ , 0 ) follows the linearized update exactly and the Taylor remainder gives the residual. Finally ∣ χ ∣ < 1 |\chi|<1 ∣ χ ∣ < 1 , because ∣ 2 λ − 1 ∣ 2 − ∣ λ − 1 ∣ 2 = 3 − 2 Re λ > 0 |2\lambda-1|^2-|\lambda-1|^2=3-2\operatorname{Re}\lambda>0 ∣2 λ − 1 ∣ 2 − ∣ λ − 1 ∣ 2 = 3 − 2 Re λ > 0 .
The residual is O ( a 2 ) O(a^2) O ( a 2 ) but not zero, and along the neutral rotation such errors could accumulate. Step 3 removes them.
Step 3: an exact cycle by the implicit function theorem
We let both the orbit’s shape and μ \mu μ vary, keeping the amplitude a a a and the center δ \delta δ as prescribed parameters.
Lemma 6.4 (Exact periodic family). For every fixed n , η , M n,\eta,M n , η , M as in Theorem 6.2 , there is a neighborhood of ( a , δ ) = ( 0 , 0 ) (a,\delta)=(0,0) ( a , δ ) = ( 0 , 0 ) on which real-analytic functions h t ( a , δ ) h_t(a,\delta) h t ( a , δ ) and μ ( a , δ ) \mu(a,\delta) μ ( a , δ ) exist, with h t + M = h t h_{t+M}=h_t h t + M = h t , such that
z t ∗ = δ + a h t , ϱ t ∗ = g μ , δ ( z t − 1 ∗ ) z_t^*=\delta+ah_t,\qquad\varrho_t^*=g_{\mu,\delta}(z_{t-1}^*) z t ∗ = δ + a h t , ϱ t ∗ = g μ , δ ( z t − 1 ∗ ) satisfy the recurrence exactly. They obey h t ( 0 , δ ) = λ t h_t(0,\delta)=\lambda^t h t ( 0 , δ ) = λ t , μ ( 0 , δ ) = μ ∗ ( δ ) \mu(0,\delta)=\mu_*(\delta) μ ( 0 , δ ) = μ ∗ ( δ ) and h t ( a , δ ) = λ t + O ( a ) h_t(a,\delta)=\lambda^t+O(a) h t ( a , δ ) = λ t + O ( a ) , uniformly for small δ \delta δ . For a ≠ 0 a\ne0 a = 0 the orbit has exact period M M M .
We use the following theorem, reading complex variables as pairs of real ones.
Theorem 6.5 (Implicit function theorem). Let F ( x , p ) F(x,p) F ( x , p ) be C r C^r C r , r ≥ 1 r\ge1 r ≥ 1 , near ( x 0 , p 0 ) (x_0,p_0) ( x 0 , p 0 ) , with equal numbers of real equations and real unknowns x x x . If F ( x 0 , p 0 ) = 0 F(x_0,p_0)=0 F ( x 0 , p 0 ) = 0 and D x F ( x 0 , p 0 ) D_xF(x_0,p_0) D x F ( x 0 , p 0 ) is invertible, then for all p p p near p 0 p_0 p 0 there is a unique nearby solution x = x ( p ) x=x(p) x = x ( p ) , which is C r C^r C r . If F F F is real analytic, so is x ( p ) x(p) x ( p ) .
The unknowns are x = ( h 0 , … , h M − 1 , μ ) x=(h_0,\ldots,h_{M-1},\mu) x = ( h 0 , … , h M − 1 , μ ) and the parameters are p = ( a , δ ) p=(a,\delta) p = ( a , δ ) . Define the residuals
R t = h t + 1 − h t − η a [ 2 g μ , δ ( δ + a h t ) − g μ , δ ( δ + a h t − 1 ) ] , N = 1 M ∑ t = 0 M − 1 h t λ − t − 1 , t mod M . (51) \begin{aligned}
\mathcal{R}_t &= h_{t+1}-h_t-\frac{\eta}{a}\left[2g_{\mu,\delta}(\delta+ah_t)-g_{\mu,\delta}(\delta+ah_{t-1})\right],\\
N &= \frac{1}{M}\sum_{t=0}^{M-1}h_t\lambda^{-t}-1,\qquad t\ \operatorname{mod} M. \tag*{(51)}
\end{aligned} R t N = h t + 1 − h t − a η [ 2 g μ , δ ( δ + a h t ) − g μ , δ ( δ + a h t − 1 ) ] , = M 1 t = 0 ∑ M − 1 h t λ − t − 1 , t mod M . (51)
These are M + 1 M+1 M + 1 complex equations in M + 1 M+1 M + 1 complex unknowns. The condition N = 0 N=0 N = 0 sets the coefficient of λ t \lambda^t λ t in h h h to 1; this removes the freedom to rescale or rotate h h h , and makes a a a the radius.
Lemma 6.6 (Properties of the periodic residuals). The map ( R 0 , … , R M − 1 , N ) (\mathcal{R}_0,\ldots,\mathcal{R}_{M-1},N) ( R 0 , … , R M − 1 , N ) extends real analytically across a = 0 a=0 a = 0 , vanishes at a = δ = 0 a=\delta=0 a = δ = 0 , h t = λ t h_t=\lambda^t h t = λ t , μ = μ 0 \mu=\mu_0 μ = μ 0 , and has an invertible real Jacobian with respect to ( h , μ ) (h,\mu) ( h , μ ) there. At a = 0 a=0 a = 0 it also vanishes for every small δ \delta δ when h t = λ t h_t=\lambda^t h t = λ t and μ = μ ∗ ( δ ) \mu=\mu_*(\delta) μ = μ ∗ ( δ ) .
Proof. Smoothness and the known zero. Since g μ , δ ( δ ) = 0 g_{\mu,\delta}(\delta)=0 g μ , δ ( δ ) = 0 , the quotient has the analytic extension
g μ , δ ( δ + a h ) a = ∫ 0 1 D g μ , δ ( δ + s a h ) [ h ] d s . \frac{g_{\mu,\delta}(\delta+ah)}{a}=\int_0^1 Dg_{\mu,\delta}(\delta+sah)[h]\,ds. a g μ , δ ( δ + ah ) = ∫ 0 1 D g μ , δ ( δ + s ah ) [ h ] d s .
The base-point identities follow from λ − 1 = β ( 2 − λ − 1 ) \lambda-1=\beta(2-\lambda^{-1}) λ − 1 = β ( 2 − λ − 1 ) and 2 η μ ∗ ( δ ) / Z ( δ ) = β 2\eta\mu_*(\delta)/Z(\delta)=\beta 2 η μ ∗ ( δ ) / Z ( δ ) = β .
Invertibility. At the base point the derivative in h h h of the first M M M residuals is
( L h ˙ ) t = h ˙ t + 1 − h ˙ t − β ( 2 h ˙ t − h ˙ t − 1 ) . (\mathcal{L}\dot{h})_t=\dot{h}_{t+1}-\dot{h}_t-\beta(2\dot{h}_t-\dot{h}_{t-1}). ( L h ˙ ) t = h ˙ t + 1 − h ˙ t − β ( 2 h ˙ t − h ˙ t − 1 ) .
On the Fourier mode ζ t \zeta^t ζ t , ζ M = 1 \zeta^M=1 ζ M = 1 , its multiplier is
L ( ζ ) = ζ − 1 − β ( 2 − ζ − 1 ) = ( ζ − λ ) ( ζ − χ ) ζ . L(\zeta)=\zeta-1-\beta(2-\zeta^{-1})=\frac{(\zeta-\lambda)(\zeta-\chi)}{\zeta}. L ( ζ ) = ζ − 1 − β ( 2 − ζ − 1 ) = ζ ( ζ − λ ) ( ζ − χ ) .
The M M M sequences ζ t \zeta^t ζ t , where ζ \zeta ζ ranges over the M M M th roots of unity, form a basis of the complex M M M -periodic sequences. Since ∣ χ ∣ < 1 |\chi|<1 ∣ χ ∣ < 1 , only ζ = λ \zeta=\lambda ζ = λ has zero multiplier. Changing h h h can therefore correct every Fourier component of the residual except this one. The derivative in μ \mu μ of these residuals is
− 2 η n ( 2 − λ − 1 ) λ t d μ , -\frac{2\eta}{n}(2-\lambda^{-1})\lambda^t\,d\mu, − n 2 η ( 2 − λ − 1 ) λ t d μ ,
a nonzero multiple of the missing mode, since ∣ 2 − λ − 1 ∣ ≥ 1 |2-\lambda^{-1}|\ge1 ∣2 − λ − 1 ∣ ≥ 1 . To solve the linearized system for arbitrary target residuals, first use their λ \lambda λ -coefficient to determine d μ d\mu d μ . Next divide each other Fourier coefficient by its nonzero multiplier L ( ζ ) L(\zeta) L ( ζ ) to determine the nonresonant part of h ˙ \dot{h} h ˙ . Finally the derivative of N N N , namely M − 1 ∑ t h t λ − t M^{-1}\sum_t h_t\lambda^{-t} M − 1 ∑ t h t λ − t , specifies the remaining coefficient of h ˙ \dot{h} h ˙ . Every target thus has exactly one preimage. The Jacobian is invertible over C \mathbb{C} C at this base point, and hence over R \mathbb{R} R .
Proof of Lemma 6.4 . Apply Theorem 6.5 using Lemma 6.6 . Local uniqueness and the explicit solutions at a = 0 a=0 a = 0 identify h t ( 0 , δ ) h_t(0,\delta) h t ( 0 , δ ) and μ ( 0 , δ ) \mu(0,\delta) μ ( 0 , δ ) . Smoothness gives h t = λ t + O ( a ) h_t=\lambda^t+O(a) h t = λ t + O ( a ) . The equations R t = 0 \mathcal{R}_t=0 R t = 0 are exactly the score recurrence with memory g ( z t − 1 ∗ ) g(z^*_{t-1}) g ( z t − 1 ∗ ) , so both score and memory repeat after M M M steps, with no accumulated update error. The normalized first Fourier coefficient is 1 1 1 ; any sequence of a proper period dividing M M M has zero coefficient at the primitive mode λ t \lambda^t λ t . Thus h h h , and hence s ∗ s^* s ∗ for a ≠ 0 a\ne0 a = 0 , has exact period M M M .
Step 4: a differentiable strong stable manifold
A cycle need not attract the uniform start. We identify the states it does attract; Figure 7 sketches this step and the next.
Figure 7. Schematic of Steps 4 and 5 in the four-dimensional state space of ( z , ρ ) (z,\rho) ( z , ρ ) . The cycle of Step 3 is neutrally stable along the center directions E c E_c E c , so a generic start does not converge to a prescribed phase. The states that do converge to the phase s 0 ∗ s_0^* s 0 ∗ form the strong stable manifold W s s W^{ss} W ss , a two-dimensional graph ξ ↦ s 0 ∗ + ξ + Γ ( a , δ , ξ ) \xi\mapsto s_0^*+\xi+\Gamma(a,\delta,\xi) ξ ↦ s 0 ∗ + ξ + Γ ( a , δ , ξ ) over E s E_s E s . Changing δ \delta δ translates the cycle, and with it W s s W^{ss} W ss , in a center direction (Lemma 6.12 ); its two real coordinates are tuned so that W s s W^{ss} W ss passes through the uniform start.
Definition 6.7 (Strong stable manifold of a cycle). For a specified phase s 0 ∗ s^*_0 s 0 ∗ of an M M M -cycle, a local strong stable manifold is a smooth manifold through s 0 ∗ s^*_0 s 0 ∗ whose points stay near the cycle and satisfy
∥ Φ t ( s ) − s t ∗ ∥ ≤ C ρ t ∥ s − s 0 ∗ ∥ ( t ≥ 0 ) \|\Phi^t(s)-s^*_t\|\le C\rho^t\|s-s^*_0\|\qquad(t\ge0) ∥ Φ t ( s ) − s t ∗ ∥ ≤ C ρ t ∥ s − s 0 ∗ ∥ ( t ≥ 0 ) for a prescribed exponential rate ρ < 1 \rho<1 ρ < 1 .
Let E c , E s ⊂ C 2 E_c,E_s \subset\mathbb{C}^2 E c , E s ⊂ C 2 be the fixed eigenspaces of the base matrix A A A for λ , χ \lambda,\chi λ , χ , with spectral projections P c , P s P_c,P_s P c , P s along the other space. Explicitly,
E c = span C ( 1 , χ / η ) , E s = span C ( 1 , λ / η ) . E_c=\operatorname{span}_{\mathbb{C}}(1,\chi/\eta), \qquad E_s=\operatorname{span}_{\mathbb{C}}(1,\lambda/\eta). E c = span C ( 1 , χ / η ) , E s = span C ( 1 , λ / η ) .
Here c c c refers to the neutral rotating directions and s s s to the strictly stable ones; each has real dimension two. The splitting is fixed and does not vary with a , δ a,\delta a , δ , so that the manifolds for all parameters are graphs over the same space.
Lemma 6.8 (Stable manifolds for the periodic family). For all sufficiently small a , δ a,\delta a , δ , the cycle of Lemma 6.4 has a local two-real-dimensional strong stable manifold of the form
W a , δ s s = { s 0 ∗ + ξ + Γ ( a , δ , ξ ) : ξ ∈ E s small } , W^{ss}_{a,\delta}=\{s^*_{0}+\xi+\Gamma(a,\delta,\xi):\xi\in E_s\text{ small}\}, W a , δ ss = { s 0 ∗ + ξ + Γ ( a , δ , ξ ) : ξ ∈ E s small } , where Γ \Gamma Γ takes values in E c E_c E c and is jointly C 1 C^1 C 1 . It satisfies Γ ( a , δ , 0 ) = 0 \Gamma(a,\delta,0)=0 Γ ( a , δ , 0 ) = 0 for all parameters and D ξ Γ ( 0 , 0 , 0 ) = 0 D_\xi\Gamma(0,0,0)=0 D ξ Γ ( 0 , 0 , 0 ) = 0 . The convergence rate and constant can be chosen uniformly on a smaller parameter neighborhood.
We obtain this graph from the parameter-dependent strong stable manifold theorem (see, e.g., [38 ] ).
Theorem 6.9 (Parameter-dependent strong stable manifold theorem). Let G p G_p G p be a jointly C r C^r C r family of local diffeomorphisms, r ≥ 2 r\ge2 r ≥ 2 , with a C r C^r C r family of fixed points s ˉ p \bar{s}_p s ˉ p . Suppose, at a base parameter, the derivative at s ˉ p \bar{s}_p s ˉ p has an invariant splitting into a strongly stable space and a complementary space, with all eigenvalue moduli in the first strictly below ρ ˉ < 1 \bar{\rho}<1 ρ ˉ < 1 and all those in the second strictly above ρ ˉ \bar{\rho} ρ ˉ . On a sufficiently small parameter neighborhood there are local strong stable manifolds through s ˉ p \bar{s}_p s ˉ p , of the dimension of the strongly stable space, depending jointly C 1 C^1 C 1 on the parameter and local manifold coordinate. They are tangent to the corresponding strong stable eigenspaces. After shrinking neighborhoods, their forward iterates converge to s ˉ p \bar{s}_p s ˉ p at a uniform rate C ρ ˉ k C\bar{\rho}^k C ρ ˉ k . They can be expressed as graphs over the fixed base strongly stable space, along its fixed complement.
Lemma 6.10 (Properties needed for the stable manifold theorem). For p = ( a , δ ) p=(a,\delta) p = ( a , δ ) put G p = Φ μ ( a , δ ) , δ M G_p=\Phi^M_{\mu(a,\delta),\delta} G p = Φ μ ( a , δ ) , δ M and s ˉ p = s 0 ∗ ( a , δ ) \bar{s}_p=s^*_0(a,\delta) s ˉ p = s 0 ∗ ( a , δ ) . These are smooth real maps with G p ( s ˉ p ) = s ˉ p G_p(\bar{s}_p)=\bar{s}_p G p ( s ˉ p ) = s ˉ p . At p = 0 p=0 p = 0 their derivative is A M A^M A M , with complex eigenvalues 1 1 1 and χ M \chi^M χ M . They are local diffeomorphisms near the cycle. For any fixed ρ ˉ \bar{\rho} ρ ˉ with ∣ χ ∣ M < ρ ˉ < 1 |\chi|^M<\bar{\rho}<1 ∣ χ ∣ M < ρ ˉ < 1 , the two real strongly stable directions remain spectrally separated from the two remaining directions for small parameters.
Proof. We apply the theorem to the return map after one period. Analyticity and the fixed-point identity follow from Step 3 and the analytic formula for Φ \Phi Φ . At zero amplitude Φ \Phi Φ fixes the origin and has derivative A A A , so the chain rule gives D G 0 ( 0 ) = A M DG_0(0)=A^M D G 0 ( 0 ) = A M . Its spectrum has the stated gap because λ M = 1 \lambda^M=1 λ M = 1 and ∣ χ ∣ < 1 |\chi|<1 ∣ χ ∣ < 1 . Also det C A = β ≠ 0 \det_{\mathbb{C}}A=\beta\ne0 det C A = β = 0 , so the corresponding real derivative is invertible. Continuity of derivatives, eigenvalue multisets and the separated spectral subspaces preserves local invertibility and the strict gap for all small parameters. □ \square □
Proof of Lemma 6.8. Apply Theorem 6.9 using Lemma 6.10 . Projection onto E s E_s E s is an invertible coordinate on each nearby manifold, giving the displayed graph in fixed coordinates. Every manifold contains its reference point, so Γ ( a , δ , 0 ) = 0 \Gamma(a,\delta,0)=0 Γ ( a , δ , 0 ) = 0 . At the base parameter it is tangent to E s E_s E s , so D ξ Γ ( 0 , 0 , 0 ) = 0 D_\xi\Gamma(0,0,0)=0 D ξ Γ ( 0 , 0 , 0 ) = 0 . The return-map estimate is C ρ ˉ k C\bar{\rho}^k C ρ ˉ k after k k k periods. The finitely many intermediate maps Φ j \Phi^j Φ j , 0 ≤ j < M 0\le j<M 0 ≤ j < M , have uniformly bounded local derivatives; hence the estimate holds at every round with ρ = ρ ˉ 1 / M < 1 \rho=\bar{\rho}^{1/M}<1 ρ = ρ ˉ 1/ M < 1 and an adjusted constant. □ \square □
We tune the two real coordinates of δ \delta δ so that the uniform start ( 0 , 0 ) (0,0) ( 0 , 0 ) lies on the stable manifold.
Lemma 6.11 (Matching the prescribed initial state). For every sufficiently small real a a a there is a C 1 C^1 C 1 choice δ = δ ( a ) = O ( a ) \delta=\delta(a)=O(a) δ = δ ( a ) = O ( a ) for which 0 ∈ W a , δ ( a ) s s 0\in W^{ss}_{a,\delta(a)} 0 ∈ W a , δ ( a ) ss . Membership of the uniform state s 0 = 0 s_0=0 s 0 = 0 is exactly
P c ( − s 0 ∗ ) = Γ ( a , δ , P s ( − s 0 ∗ ) ) . (52) P_c(-s_0^*)=\Gamma(a,\delta,P_s(-s_0^*)). \tag*{(52)} P c ( − s 0 ∗ ) = Γ ( a , δ , P s ( − s 0 ∗ )) . (52) These are two real equations, which we solve for δ ∈ C ≅ R 2 \delta\in\mathbb{C}\cong\mathbb{R}^2 δ ∈ C ≅ R 2 by the implicit function theorem, with a a a as the parameter.
Lemma 6.12 (Properties of the matching equation). The difference between the two sides of (52) , denoted
Δ ( a , δ ) = P c ( − s 0 ∗ ) − Γ ( a , δ , P s ( − s 0 ∗ ) ) ∈ E c , \Delta(a,\delta)=P_c(-s_0^*)-\Gamma(a,\delta,P_s(-s_0^*))\in E_c, Δ ( a , δ ) = P c ( − s 0 ∗ ) − Γ ( a , δ , P s ( − s 0 ∗ )) ∈ E c , is C 1 C^1 C 1 , satisfies Δ ( 0 , 0 ) = 0 \Delta(0,0)=0 Δ ( 0 , 0 ) = 0 , and has an invertible real derivative in δ \delta δ at ( 0 , 0 ) (0,0) ( 0 , 0 ) .
Proof. The key point is transversality: changing δ \delta δ moves the cycle in a center direction to first order, whereas the graph correction has zero first derivative at the base point. Steps 3 and 4 give smoothness and the base-point identity. At a = 0 a=0 a = 0 , s 0 ∗ ( 0 , δ ) = ( δ , 0 ) s_0^*(0,\delta)=(\delta,0) s 0 ∗ ( 0 , δ ) = ( δ , 0 ) . We claim that
D δ Δ ( 0 , 0 ) [ d δ ] = − P c ( d δ , 0 ) . D_\delta\Delta(0,0)[d\delta]=-P_c(d\delta,0). D δ Δ ( 0 , 0 ) [ d δ ] = − P c ( d δ , 0 ) .
To verify the claim, apply the chain rule to the graph term in Δ \Delta Δ . Its direct parameter derivative vanishes because Γ ( a , δ , 0 ) \Gamma(a,\delta,0) Γ ( a , δ , 0 ) is identically zero. The derivative through the argument P s ( − s 0 ∗ ) P_s(-s_0^*) P s ( − s 0 ∗ ) also vanishes, because it is multiplied by D ξ Γ ( 0 , 0 , 0 ) = 0 D_\xi\Gamma(0,0,0)=0 D ξ Γ ( 0 , 0 , 0 ) = 0 . Only the derivative of the center displacement remains. To see invertibility explicitly, decompose any ( z , ρ ) (z,\rho) ( z , ρ ) as
( z , ρ ) = u c ( 1 , χ / η ) + u s ( 1 , λ / η ) , u c = λ z − η ρ λ − χ . (z,\rho)=u_c(1,\chi/\eta)+u_s(1,\lambda/\eta),\qquad u_c=\frac{\lambda z-\eta\rho}{\lambda-\chi}. ( z , ρ ) = u c ( 1 , χ / η ) + u s ( 1 , λ / η ) , u c = λ − χ λ z − η ρ .
In the scalar coordinate u c u_c u c on E c E_c E c , the displayed derivative is multiplication by − λ / ( λ − χ ) ≠ 0 -\lambda/(\lambda-\chi)\ne0 − λ / ( λ − χ ) = 0 . Thus it is an invertible map of two-dimensional real spaces.
Proof of Lemma 6.11. Apply Theorem 6.5 to Δ ( a , δ ) = 0 \Delta(a,\delta)=0 Δ ( a , δ ) = 0 using Lemma 6.12 . It gives δ ( 0 ) = 0 \delta(0)=0 δ ( 0 ) = 0 and a C 1 C^1 C 1 solution for all sufficiently small signed a a a , in particular for small a > 0 a>0 a > 0 . Differentiability gives δ ( a ) = O ( a ) \delta(a)=O(a) δ ( a ) = O ( a ) . Equation (52) then puts the uniform initial state on the strong stable manifold. Its displacement from the cycle is O ( a ) O(a) O ( a ) , and hence lies in the required local neighborhood for small a a a .
Step 6: conclude exponential convergence to an exact cycle
Proof of Theorem 6.2. Choose a > 0 a>0 a > 0 small, put δ = δ ( a ) \delta=\delta(a) δ = δ ( a ) from Step 5, and set μ = μ ( a , δ ( a ) ) \mu=\mu(a,\delta(a)) μ = μ ( a , δ ( a )) from Step 3. Step 1 identifies Optimistic Hedge with the full score-and-memory recurrence from 0. Step 3 supplies an exact period-M M M orbit with z t ∗ = δ + a λ t + O ( a 2 ) z_t^*=\delta+a\lambda^t+O(a^2) z t ∗ = δ + a λ t + O ( a 2 ) . Steps 4 and 5 imply that the uniform start converges to the specified phase of this orbit, with error at most C a ρ t Ca\rho^t C a ρ t because ∥ s 0 ∗ ∥ = O ( a ) \lVert s_0^*\rVert=O(a) ∥ s 0 ∗ ∥ = O ( a ) .
It remains to check bounded utilities; this is where the assumption on η M \eta M η M is used. Since ∣ β ∣ ≤ ∣ λ − 1 ∣ ≤ 2 π / M |\beta|\le|\lambda-1|\le2\pi/M ∣ β ∣ ≤ ∣ λ − 1∣ ≤ 2 π / M , the assumption on η M \eta M η M gives
∣ μ 0 ∣ ≤ π n η M ≤ 1 16 . |\mu_0|\le\frac{\pi n}{\eta M}\le\frac{1}{16}. ∣ μ 0 ∣ ≤ η M π n ≤ 16 1 .
For v ∈ V v\in V v ∈ V , ∣ v ∣ ≤ 2 |v|\le\sqrt{2} ∣ v ∣ ≤ 2 ; for every opponent action, ∣ c ( w ) ∣ ≤ 1 |c(w)|\le1 ∣ c ( w ) ∣ ≤ 1 and e − Re ( w δ ) ≤ e 2 ∣ δ ∣ e^{-\operatorname{Re}(w\delta)}\le e^{\sqrt{2}|\delta|} e − Re ( w δ ) ≤ e 2 ∣ δ ∣ . Thus
∣ U ( v , w ) ∣ ≤ 2 ∣ μ ∣ e 2 ∣ δ ∣ < 1 2 |U(v,w)|\le\sqrt{2}|\mu|e^{\sqrt{2}|\delta|}<\frac{1}{2} ∣ U ( v , w ) ∣ ≤ 2 ∣ μ ∣ e 2 ∣ δ ∣ < 2 1
for sufficiently small a a a , by continuity from ( μ 0 , 0 ) (\mu_0,0) ( μ 0 , 0 ) . Neutral actions have utility 0. All utilities therefore lie in [ − 1 2 , 1 2 ] [-\frac{1}{2},\frac{1}{2}] [ − 2 1 , 2 1 ] , completing the proof.
A block driven by a periodic predecessor
We now build a block that responds to its predecessor at twice the frequency. The source is the preceding block, with score x t x_t x t on the orbit x t ∗ = δ + a λ t + O ( a 2 ) x_t^*=\delta+a\lambda^t+O(a^2) x t ∗ = δ + a λ t + O ( a 2 ) of Theorem 6.2 ; the new block has score y t y_t y t . Block number and round number are different indices: the source’s strategy at round t t t determines an input used to update the new block for round t + 1 t+1 t + 1 . In this subsection the new block is carried by a separate pair of identically initialized players, with its own normalizer, and we work out its linear response. This is also exactly how the first block of each further pair of players is driven in the m m m -player game of § 10.7 . § 10.5 puts all blocks into the same two players and handles the nonlinear terms and the convergence from the uniform start for all blocks at once.
Step 1: extract a signal rotating at twice the frequency
We assign fixed numbers to the source’s actions whose expectation is approximately the square of its displacement from δ \delta δ .
Lemma 6.13 (Quadratic extraction). There is a complex-valued function r δ r_\delta r δ , fixed before play begins, on the source’s actions such that its expectation Q δ ( x ) Q_\delta(x) Q δ ( x ) under p x p_x p x satisfies, for δ , u = O ( a ) \delta,u=O(a) δ , u = O ( a ) ,
Q δ ( δ + u ) = u 2 n + O ( a 4 ) . (53) Q_\delta(\delta+u)=\frac{u^2}{n}+O(a^4). \tag*{(53)} Q δ ( δ + u ) = n u 2 + O ( a 4 ) . (53) In particular, on the source orbit,
Q δ ( x t ∗ ) = a 2 n λ 2 t + O ( a 3 ) , Q_\delta(x_t^*)=\frac{a^2}{n}\lambda^{2t}+O(a^3), Q δ ( x t ∗ ) = n a 2 λ 2 t + O ( a 3 ) , with an M M M -periodic remainder uniform in t t t .
Represent the source’s active actions by b ∈ V = { ± 1 , ± i , 1 + i , − 1 − i , 1 − i , − 1 + i } b \in V = \{\pm1, \pm i, 1+i, -1-i, 1-i, -1+i\} b ∈ V = { ± 1 , ± i , 1 + i , − 1 − i , 1 − i , − 1 + i } ; its other n − 8 n-8 n − 8 actions are distinct labels with coordinate zero. Define
q ( ± 1 ) = 1 , q ( ± i ) = − 1 , q ( 1 + i ) = q ( − 1 − i ) = i / 2 , q ( 1 − i ) = q ( − 1 + i ) = − i / 2. q(\pm1)=1,\qquad q(\pm i)=-1,\qquad q(1+i)=q(-1-i)=i/2,\qquad q(1-i)=q(-1+i)=-i/2. q ( ± 1 ) = 1 , q ( ± i ) = − 1 , q ( 1 + i ) = q ( − 1 − i ) = i /2 , q ( 1 − i ) = q ( − 1 + i ) = − i /2.
and q = 0 q=0 q = 0 on neutral actions. Thus q ( b ) = b 2 q(b)=b^2 q ( b ) = b 2 on the four axis actions and q ( b ) = b 2 / 4 q(b)=b^2/4 q ( b ) = b 2 /4 on the four diagonal ones (Figure 5 ); the factor 1 4 \frac{1}{4} 4 1 compensates for the larger coordinates of the diagonal actions. For a source score x x x , recall that
p x ( b ) = e Re ( b ˉ x ) Z ( x ) , Z ( x ) = ∑ b e Re ( b ˉ x ) . p_x(b)=\frac{e^{\operatorname{Re}(\bar{b}x)}}{Z(x)},\qquad Z(x)=\sum_b e^{\operatorname{Re}(\bar{b}x)}. p x ( b ) = Z ( x ) e Re ( b ˉ x ) , Z ( x ) = b ∑ e Re ( b ˉ x ) .
where the sum is over distinct action labels, including all neutral actions. Set
r δ ( b ) = q ( b ) e − Re ( b ˉ δ ) , Q δ ( x ) = E B ∼ p x [ r δ ( B ) ] = ∑ b p x ( b ) r δ ( b ) . r_\delta(b)=q(b)e^{-\operatorname{Re}(\bar{b}\delta)},\qquad Q_\delta(x)=\mathbb{E}_{B\sim p_x}[r_\delta(B)]=\sum_b p_x(b)r_\delta(b). r δ ( b ) = q ( b ) e − Re ( b ˉ δ ) , Q δ ( x ) = E B ∼ p x [ r δ ( B )] = b ∑ p x ( b ) r δ ( b ) .
The factor involving δ \delta δ shifts the numerator to the source’s displacement x − δ x-\delta x − δ ; the denominator remains Z ( x ) Z(x) Z ( x ) .
Proof. Write u = u 1 + i u 2 u=u_1+iu_2 u = u 1 + i u 2 . Pairing opposite actions in the expectation gives
Q δ ( δ + u ) = 2 cosh u 1 − 2 cosh u 2 + 2 i sinh u 1 sinh u 2 Z ( δ + u ) . (54) Q_\delta(\delta+u)=\frac{2\cosh u_1-2\cosh u_2+2i\sinh u_1\sinh u_2}{Z(\delta+u)}. \tag*{(54)} Q δ ( δ + u ) = Z ( δ + u ) 2 cosh u 1 − 2 cosh u 2 + 2 i sinh u 1 sinh u 2 . (54)
The horizontal and vertical actions produce u 1 2 − u 2 2 u_1^2-u_2^2 u 1 2 − u 2 2 to second order; the diagonal actions produce 2 i u 1 u 2 2iu_1u_2 2 i u 1 u 2 . Thus the numerator is
u 1 2 − u 2 2 + 2 i u 1 u 2 + O ( ∣ u ∣ 4 ) = u 2 + O ( ∣ u ∣ 4 ) . u_1^2-u_2^2+2iu_1u_2+O(|u|^4)=u^2+O(|u|^4). u 1 2 − u 2 2 + 2 i u 1 u 2 + O ( ∣ u ∣ 4 ) = u 2 + O ( ∣ u ∣ 4 ) .
Symmetry of the action set gives Z ( x ) = n + O ( ∣ x ∣ 2 ) Z(x)=n+O(|x|^2) Z ( x ) = n + O ( ∣ x ∣ 2 ) near zero, proving (53) . On the source orbit, u = a λ t + O ( a 2 ) u=a\lambda^t+O(a^2) u = a λ t + O ( a 2 ) , so u 2 = a 2 λ 2 t + O ( a 3 ) u^2=a^2\lambda^{2t}+O(a^3) u 2 = a 2 λ 2 t + O ( a 3 ) . This proves the stated input estimate. Exact periodicity follows because Q δ Q_\delta Q δ is a fixed function of the exactly periodic source score. □
Figure 8 shows the mechanism in terms of probabilities: opposite actions oscillate in antiphase, so their first-order oscillations cancel in Q δ Q_\delta Q δ , and only the second-order part, at twice the frequency, survives. Complex scores are essential here: a real oscillation a cos ( ω t ) a\cos(\omega t) a cos ( ω t ) squares to a 2 2 ( 1 + cos 2 ω t ) \frac{a^2}{2}(1+\cos2\omega t) 2 a 2 ( 1 + cos 2 ω t ) , whose constant part would shift the center of the next block.
Figure 8. Frequency doubling as a linear combination of probabilities, drawn for δ = 0 \delta= 0 δ = 0 and to leading order in a a a . Left: while the source score x t ≈ a λ t x_t \approx a\lambda^t x t ≈ a λ t goes around once, each of its action probabilities oscillates once, and opposite actions oscillate in antiphase. Right: summing opposite actions cancels these first-order oscillations and leaves the second-order terms a 2 n cos 2 ( 2 π t / M ) \frac{a^2}{n}\cos^2(2\pi t/M) n a 2 cos 2 ( 2 π t / M ) and a 2 n sin 2 ( 2 π t / M ) \frac{a^2}{n}\sin^2(2\pi t/M) n a 2 sin 2 ( 2 π t / M ) . Their difference, which is Re Q t \operatorname{Re} Q_t Re Q t for the weights q ( ± 1 ) = 1 q(\pm1)=1 q ( ± 1 ) = 1 and q ( ± i ) = − 1 q(\pm i)=-1 q ( ± i ) = − 1 , is a 2 n cos ( 4 π t / M ) \frac{a^2}{n}\cos(4\pi t/M) n a 2 cos ( 4 π t / M ) , a full cycle every half period. The diagonal actions, with weights ± i 2 \pm\frac{i}{2} ± 2 i , give Im Q t = a 2 n sin ( 4 π t / M ) \operatorname{Im} Q_t=\frac{a^2}{n}\sin(4\pi t/M) Im Q t = n a 2 sin ( 4 π t / M ) in the same way, so Q t ≈ a 2 n Λ t Q_t\approx\frac{a^2}{n}\Lambda^t Q t ≈ n a 2 Λ t rotates at twice the source’s speed. Note the different units: the doubled signal is a factor a a a smaller.
Generating a rotation is not the same as making it attract trajectories.
Lemma 6.14 (No attraction without feedback). If the new block’s utility coefficient is a prescribed sequence f t f_t f t , independent of its score, two trajectories with the same initial memory and the same input retain their initial score difference forever. Moreover, if f t f_t f t is M M M -periodic, then after the initial memory is consistent with the input, the score displacement over a period is
η ∑ j = 0 M − 1 f j . \eta\sum_{j=0}^{M-1}f_j. η j = 0 ∑ M − 1 f j .
Proof. Utilities Re ( u ‾ v r δ ( b ) ) \operatorname{Re}(\overline{u}vr_\delta(b)) Re ( u v r δ ( b )) give f t = ν Q δ ( x t ) f_t=\nu Q_\delta(x_t) f t = ν Q δ ( x t ) . Once the memory equals f t − 1 f_{t-1} f t − 1 , the score update is
y t + 1 − y t = η ( 2 f t − f t − 1 ) . y_{t+1}-y_t=\eta(2f_t-f_{t-1}). y t + 1 − y t = η ( 2 f t − f t − 1 ) .
Its increment does not depend on y t y_t y t , so subtracting two updates preserves the score difference, including the first update when the initial memories agree. Summing over a full period with consistent memory gives the displayed displacement, since the shifted sum of f t − 1 f_{t-1} f t − 1 equals the sum of f t f_t f t .
For a pure rotating input f t = ϕ Λ t f_t=\phi\Lambda^t f t = ϕ Λ t , where Λ = λ 2 ≠ 1 \Lambda=\lambda^2\ne1 Λ = λ 2 = 1 , the recurrence admits the exact rotating solution
y t = ψ Λ t , ψ = η ϕ ( 2 − Λ − 1 ) Λ − 1 . y_t=\psi\Lambda^t,\qquad\psi=\frac{\eta\phi(2-\Lambda^{-1})}{\Lambda-1}. y t = ψ Λ t , ψ = Λ − 1 η ϕ ( 2 − Λ − 1 ) .
Here ϕ \phi ϕ is the input amplitude, ψ \psi ψ is the response coefficient, and the initial memory is chosen consistently with the periodic input. However, every constant translate is also a solution: there is no preferred center and no attraction. Further, small corrections to the input can have nonzero mean and cause drift. We therefore add feedback that removes score offsets rather than merely integrating the source signal. Figure 9 compares the two cases for the linearized dynamics.
Figure 9. How the next block uses the doubled signal, for linearized dynamics without errors, drawn for M = 16 M = 16 M = 16 ; the response is shown in rounds t ≥ 0 t \ge0 t ≥ 0 , from light (early) to dark (late), starting from the uniform state y 0 = 0 y_0 = 0 y 0 = 0 (open circle); the numbers mark the rounds t = 1 , … , 8 t = 1,\ldots,8 t = 1 , … , 8 of the first period of the input. Left: the input Q t ≈ a 2 n Λ t Q_t \approx\frac{a^2}{n}\Lambda^t Q t ≈ n a 2 Λ t , Λ = λ 2 \Lambda= \lambda^2 Λ = λ 2 , produced by the previous block (Figure 8 ); it repeats every M / 2 M/2 M /2 rounds. Middle: if the block only integrates this input (κ = 0 \kappa= 0 κ = 0 ), its score y t y_t y t immediately goes around a circle at the input’s speed, but the center of that circle is set by the initial state rather than by the game. Nothing pulls the score back, so any error moves the circle permanently (dashed), and errors accumulate. Right: with the restoring self-interaction − κ c ( w ) -\kappa c(w) − κ c ( w ) , all state differences contract, and the score converges from the uniform start to a single circle centered at 0 0 0 , which it then follows with a fixed phase lag behind the input. The restoring strength is exaggerated (γ = η d = 0.15 \gamma= \eta d = 0.15 γ = η d = 0.15 ) to make the convergence visible within a few periods.
Step 3: add restoring utilities and compute the linear rotation
Lemma 6.15 (Stable linear response). Fix κ = ν = 1 / 64 \kappa=\nu=1/64 κ = ν = 1/64 , put d = 2 κ / n d=2\kappa/n d = 2 κ / n , γ = η d \gamma=\eta d γ = η d , and Λ = λ 2 \Lambda=\lambda^2 Λ = λ 2 . For the linear utility coefficient
g t = − d y t + ν a 2 n Λ t g_t=-d y_t+\frac{\nu a^2}{n}\Lambda^t g t = − d y t + n ν a 2 Λ t
the score-and-memory recurrence (49) has an exact rotating solution with score y t = c 1 a 2 Λ t y_t = c_1 a^2 \Lambda^t y t = c 1 a 2 Λ t , where
c 1 = ( η ν / n ) ( 2 − Λ − 1 ) Λ − 1 + γ ( 2 − Λ − 1 ) ≠ 0. c_1 = \frac{(\eta\nu/n)(2-\Lambda^{-1})}{\Lambda-1+\gamma(2-\Lambda^{-1})} \ne0. c 1 = Λ − 1 + γ ( 2 − Λ − 1 ) ( η ν / n ) ( 2 − Λ − 1 ) = 0.
Its consistent memory is ϱ t = ( ν / n − d c 1 ) a 2 Λ t − 1 \varrho_t = (\nu/n-dc_1)a^2\Lambda^{t-1} ϱ t = ( ν / n − d c 1 ) a 2 Λ t − 1 . Every other solution approaches this one exponentially.
To realize the restoring term, the new players have the same action labels and probabilities p y p_y p y as in (46) . Write v v v for the new player’s action, w w w for her paired opponent’s action, and b b b for the source player’s action, all at the same round. Recall c ( w ) = w c(w)=w c ( w ) = w for w ∈ { ± 1 , ± i } w \in\{\pm1,\pm i\} w ∈ { ± 1 , ± i } and c ( w ) = 0 c(w)=0 c ( w ) = 0 otherwise. If the source chooses b b b , define
U ( v ; w , b ) = Re ( v ‾ [ − κ c ( w ) + ν r δ ( b ) ] ) . (55) U(v;w,b)=\operatorname{Re}\bigl(\overline{v}[-\kappa c(w)+\nu r_\delta(b)]\bigr). \tag*{(55)} U ( v ; w , b ) = Re ( v [ − κ c ( w ) + ν r δ ( b )] ) . (55)
These utilities lie in [ − 1 2 , 1 2 ] [-\frac{1}{2},\frac{1}{2}] [ − 2 1 , 2 1 ] for small a a a : ∣ v ∣ ≤ 2 |v|\le\sqrt{2} ∣ v ∣ ≤ 2 , ∣ c ( w ) ∣ ≤ 1 |c(w)|\le1 ∣ c ( w ) ∣ ≤ 1 , and ∣ r δ ( b ) ∣ ≤ e 2 ∣ δ ∣ |r_\delta(b)|\le e^{\sqrt{2}|\delta|} ∣ r δ ( b ) ∣ ≤ e 2 ∣ δ ∣ . The negative sign penalizes alignment with the opponent’s current-block action. By symmetry the two new players stay identical. Averaging over their paired opponent and the source yields expected utility Re ( v ‾ g t ( y ) ) \operatorname{Re}(\overline{v}g_t(y)) Re ( v g t ( y )) , with
g t ( y ) = − κ 2 [ sinh ( Re y ) + i sinh ( Im y ) ] Z ( y ) + ν Q δ ( x t ) . (56) g_t(y)=-\kappa\frac{2[\sinh(\operatorname{Re} y)+i\sinh(\operatorname{Im} y)]}{Z(y)}+\nu Q_\delta(x_t). \tag*{(56)} g t ( y ) = − κ Z ( y ) 2 [ sinh ( Re y ) + i sinh ( Im y )] + ν Q δ ( x t ) . (56)
The exponential-family calculation in Lemma 6.1 now gives exactly
y t + 1 = y t + η ( 2 g t ( y t ) − ϱ t ) , ϱ t + 1 = g t ( y t ) , ( y 0 , ϱ 0 ) = ( 0 , 0 ) . y_{t+1}=y_t+\eta(2g_t(y_t)-\varrho_t),\qquad\varrho_{t+1}=g_t(y_t),\qquad(y_0,\varrho_0)=(0,0). y t + 1 = y t + η ( 2 g t ( y t ) − ϱ t ) , ϱ t + 1 = g t ( y t ) , ( y 0 , ϱ 0 ) = ( 0 , 0 ) .
Near zero the first term of (56) is − d y + O ( ∣ y ∣ 3 ) -dy+O(|y|^3) − d y + O ( ∣ y ∣ 3 ) . With the source on its orbit, Lemma 6.13 therefore gives
g t ( y ) = − d y + ν a 2 n Λ t + O ( ∣ y ∣ 3 + a 3 ) . g_t(y)=-dy+\frac{\nu a^2}{n}\Lambda^t+O(|y|^3+a^3). g t ( y ) = − d y + n ν a 2 Λ t + O ( ∣ y ∣ 3 + a 3 ) .
Proof of Lemma 6.15 . A rotating sequence keeps its form under time shifts: shifting one round multiplies it by Λ \Lambda Λ . This reduces the recurrence to an equation for one complex coefficient. Substitute y t = c 1 a 2 Λ t y_t=c_1a^2\Lambda^t y t = c 1 a 2 Λ t and the consistent memory into the linear recurrence. Canceling a 2 Λ t a^2\Lambda^t a 2 Λ t gives
c 1 ( Λ − 1 ) = − γ c 1 ( 2 − Λ − 1 ) + η ν n ( 2 − Λ − 1 ) , c_1(\Lambda-1)=-\gamma c_1(2-\Lambda^{-1})+\frac{\eta\nu}{n}(2-\Lambda^{-1}), c 1 ( Λ − 1 ) = − γ c 1 ( 2 − Λ − 1 ) + n η ν ( 2 − Λ − 1 ) ,
which gives the stated coefficient. To justify division and attraction, subtract two solutions. Their state difference evolves by
B = ( 1 − 2 γ − η − γ / η 0 ) . B=\begin{pmatrix}1-2\gamma&-\eta\\-\gamma/\eta&0\end{pmatrix}. B = ( 1 − 2 γ − γ / η − η 0 ) .
Its eigenvalues are r ± = ( 1 − 2 γ ± 1 + 4 γ 2 ) / 2 r_{\pm}=(1-2\gamma\pm\sqrt{1+4\gamma^2})/2 r ± = ( 1 − 2 γ ± 1 + 4 γ 2 ) /2 . For 0 < γ < 2 / 3 0<\gamma<2/3 0 < γ < 2/3 , one has 0 < r + < 1 0<r_+<1 0 < r + < 1 and − 1 < r − < 0 -1<r_-<0 − 1 < r − < 0 ; our parameter choices satisfy this restriction. Thus powers of B B B decay exponentially. A zero denominator in the formula for c 1 c_1 c 1 would make Λ \Lambda Λ , of modulus one, an eigenvalue of B B B , a contradiction. The numerator is nonzero since ∣ Λ − 1 ∣ = 1 |\Lambda^{-1}|=1 ∣ Λ − 1 ∣ = 1 . Hence the response is a nonzero rotation and all state differences decay. □
Lemma 6.15 is the linear part of the argument. The nonlinear terms, the exact periodic response and the convergence from the uniform start are handled in § 10.5 , jointly for all blocks; no stable-manifold matching is needed for these blocks, because all their linearized directions contract.
The observer
We now use the oscillation of players 1 and 2 to give player 3 linear regret. Player 3 has two groups of actions with opposite utility advantages. Each fixed action has average utility 0, but the changing strategy of Optimistic Hedge responds with harmful timing to a sufficiently fast oscillation. By Remark 6.22 at the end of this subsection, a mode of frequency ω \omega ω increases regret exactly when cos ω < 1 2 \cos\omega<\frac{1}{2} cos ω < 2 1 ; this is why the slow first block is useless here and the last block runs at ω = π / 2 \omega=\pi/2 ω = π /2 .
Throughout this subsection, assume that players 1 and 2 have an exact M M M -periodic orbit approached exponentially from initialization. Suppose its last block, block k k k , has score
y k , t ∗ = c k a K λ K t + O ( a K + 1 ) , K = 2 k − 1 , c k ≠ 0 , λ K = e i π / 2 . y^*_{k,t}=c_k a^K\lambda_K^t+O(a^{K+1}),\qquad K=2^{k-1},\qquad c_k\ne0,\qquad\lambda_K=e^{i\pi/2}. y k , t ∗ = c k a K λ K t + O ( a K + 1 ) , K = 2 k − 1 , c k = 0 , λ K = e iπ /2 .
These are hypotheses here; Theorem 6.23 in § 10.5 provides them. All constants below may depend on the fixed game parameters. We choose a a a first and then ε \varepsilon ε .
Step 1: the signal and player 3’s utilities
We turn the last block’s rotation into a mean-zero signal and use it to define player 3’s utilities.
Lemma 6.16 (The observer’s signal). There is a bounded real function F F F on player 1’s actions such that f t = E [ F ( W t ) ] f_t=\mathbb{E}[F(W_t)] f t = E [ F ( W t )] satisfies
f t + M ∗ = f t ∗ , ∑ t = 0 M − 1 f t ∗ = 0 , ∣ f t − f t ∗ ∣ ≤ C ρ t , ρ < 1. (57) f^*_{t+M}=f^*_t,\qquad\sum_{t=0}^{M-1}f^*_t=0,\qquad|f_t-f^*_t|\le C\rho^t,\qquad\rho<1. \tag*{(57)} f t + M ∗ = f t ∗ , t = 0 ∑ M − 1 f t ∗ = 0 , ∣ f t − f t ∗ ∣ ≤ C ρ t , ρ < 1. (57)
and, for some fixed phase θ \theta θ ,
f t ∗ = α cos ( π t / 2 + θ ) + O ( a K + 1 ) , α = Θ ( a K ) > 0. (58) f_t^* = \alpha\cos(\pi t/2+\theta) + O(a^{K+1}), \qquad\alpha= \Theta(a^K)>0. \tag*{(58)} f t ∗ = α cos ( π t /2 + θ ) + O ( a K + 1 ) , α = Θ ( a K ) > 0. (58)
In particular, the partial sums of f t f_t f t are bounded uniformly in time.
Proof. Let H H H equal 1 on action ( k , 1 ) (k,1) ( k , 1 ) , − 1 -1 − 1 on action ( k , − 1 ) (k,-1) ( k , − 1 ) , and zero otherwise. Thus its expectation is the difference between these two action probabilities. Define the fixed constant
b = 1 M ∑ t = 0 M − 1 E p t ∗ [ H ] , F = H − b . b = \frac{1}{M}\sum_{t=0}^{M-1}\mathbb{E}_{p_t^*}[H], \qquad F=H-b. b = M 1 t = 0 ∑ M − 1 E p t ∗ [ H ] , F = H − b .
Subtracting b b b makes the orbit average exactly zero; this is a fixed utility-table adjustment, not a time-dependent rule. On the orbit the uncentered expectation is
2 sinh ( Re y k , t ∗ ) Z t ∗ . \frac{2\sinh(\operatorname{Re} y_{k,t}^*)}{Z_t^*}. Z t ∗ 2 sinh ( Re y k , t ∗ ) .
All scores are O ( a ) O(a) O ( a ) , so the shared normalizer satisfies Z t ∗ = n + O ( a 2 ) Z_t^*=n+O(a^2) Z t ∗ = n + O ( a 2 ) . Expanding the numerator gives the leading term ( 2 / n ) Re ( c k a K e i π t / 2 ) (2/n)\operatorname{Re}(c_k a^K e^{i\pi t/2}) ( 2/ n ) Re ( c k a K e iπ t /2 ) , with error O ( a K + 1 ) O(a^{K+1}) O ( a K + 1 ) . Its leading term has mean zero because M M M is divisible by four. Centering changes only the error term. This proves (58) , with α = 2 ∣ c k ∣ a K / n \alpha=2|c_k|a^K/n α = 2∣ c k ∣ a K / n and θ = arg c k \theta=\arg c_k θ = arg c k .
The expectation of a fixed bounded function is smooth in the scores, so exponential approach of the players’ state to its orbit gives the last bound in (57) . A periodic mean-zero sequence has bounded partial sums, and adding an exponentially decaying sequence preserves this property.
Lemma 6.17 (Fixed actions have bounded excess utility). Split player 3’s n n n actions into nonempty groups + + + and − - − . Give every action in these groups, respectively, utilities
U 3 ( + , w ) = ε F ( w ) , U 3 ( − , w ) = − ε F ( w ) , U_3(+,w)=\varepsilon F(w), \qquad U_3(-,w)=-\varepsilon F(w), U 3 ( + , w ) = εF ( w ) , U 3 ( − , w ) = − εF ( w ) , where w w w is player 1’s action. For sufficiently small ε > 0 \varepsilon>0 ε > 0 , these utilities lie in [ − 1 2 , 1 2 ] [-\frac{1}{2},\frac{1}{2}] [ − 2 1 , 2 1 ] , and every fixed action has cumulative loss O ( 1 ) O(1) O ( 1 ) .
Proof. Boundedness of F F F ensures valid utilities if ε ∥ F ∥ ∞ ≤ 1 2 \varepsilon\|F\|_\infty\le\frac{1}{2} ε ∥ F ∥ ∞ ≤ 2 1 . Expected losses are ∓ ε f t \mp\varepsilon f_t ∓ ε f t . Their cumulative sums are bounded by Lemma 6.16 . These utilities do not affect players 1 and 2, whose utilities are independent of player 3.
Our main claim is the following.
Proposition 6.18 (Linear regret of the observer). Under the orbit hypotheses above, the utilities of Lemma 6.17 give, for sufficiently small a > 0 a>0 a > 0 and then sufficiently small ε > 0 \varepsilon>0 ε > 0 ,
Reg 3 ( T ) = c o b s T + O ( 1 ) , c o b s > 0. \operatorname{Reg}_3(T)=c_{\mathrm{obs}}T+O(1), \qquad c_{\mathrm{obs}}>0. Reg 3 ( T ) = c obs T + O ( 1 ) , c obs > 0.
Step 2: the observer’s response
Lemma 6.19 (Log-odds update). Let r t = log ( Pr t ( + ) / Pr t ( − ) ) r_t=\log(\Pr_t(+)/\Pr_t(-)) r t = log ( Pr t ( + ) / Pr t ( − )) , and let r 0 = log ( ∣ + ∣ / ∣ − ∣ ) r_0=\log(|+|/|-|) r 0 = log ( ∣ + ∣/∣ − ∣ ) at uniform initialization. With f − 1 = 0 f_{-1}=0 f − 1 = 0 , Optimistic Hedge gives
r t + 1 = r t + 2 η ε ( 2 f t − f t − 1 ) , r_{t+1}=r_t+2\eta\varepsilon(2f_t-f_{t-1}), r t + 1 = r t + 2 η ε ( 2 f t − f t − 1 ) , r t = r 0 + 2 η ε ( ∑ s < t f s + f t − 1 ) . (59) r_t=r_0+2\eta\varepsilon\left(\sum_{s<t}f_s+f_{t-1}\right). \tag*{(59)} r t = r 0 + 2 η ε ( s < t ∑ f s + f t − 1 ) . (59)
Proof. Actions in the same group have identical utility sequences, so their relative weights remain constant. The utility difference between groups is 2 ε f t 2\varepsilon f_t 2 ε f t ; taking the log ratio in the Optimistic Hedge update gives the first identity. Summing it gives the second, using zero initial utility memory.
Figure 2 in § 3.4 shows the resulting timing mistake for equal groups and the signal α , 0 , − α , 0 , … \alpha, 0, -\alpha, 0, \ldots α , 0 , − α , 0 , … : at round two the observer still prefers + although it is now bad. The next steps prove that this effect persists for every phase θ \theta θ , unequal groups and the error terms of the actual signal.
Step 3: regret through the signal’s timing
Lemma 6.20 (Regret and signal correlation). Write S t = ∑ s < t f s S_t = \sum_{s<t} f_s S t = ∑ s < t f s . Then
Reg 3 ( T ) = ε ∣ S T ∣ − ε ∑ t < T f t tanh ( r t / 2 ) , (60) \operatorname{Reg}_3(T) = \varepsilon|S_T| - \varepsilon\sum_{t<T} f_t \tanh(r_t/2), \tag*{(60)} Reg 3 ( T ) = ε ∣ S T ∣ − ε t < T ∑ f t tanh ( r t /2 ) , (60) and
Reg 3 ( T ) = − η ε 2 sech 2 ( r 0 / 2 ) ∑ t < T f t ( S t + f t − 1 ) + O ( 1 ) + O ( η 2 ε 3 T ) . \operatorname{Reg}_3(T) = -\eta\varepsilon^2 \operatorname{sech}^2(r_0/2) \sum_{t<T} f_t(S_t + f_{t-1}) + O(1) + O(\eta^2\varepsilon^3 T). Reg 3 ( T ) = − η ε 2 sech 2 ( r 0 /2 ) t < T ∑ f t ( S t + f t − 1 ) + O ( 1 ) + O ( η 2 ε 3 T ) .
Proof. The probability imbalance between groups is tanh ( r t / 2 ) \tanh(r_t/2) tanh ( r t /2 ) , so the learner’s expected loss is − ε f t tanh ( r t / 2 ) -\varepsilon f_t \tanh(r_t/2) − ε f t tanh ( r t /2 ) . The best fixed action has cumulative loss − ε ∣ S T ∣ -\varepsilon|S_T| − ε ∣ S T ∣ . Subtraction proves (60) .
By bounded partial sums and (59) , r t − r 0 = O ( η ε ) r_t-r_0=O(\eta\varepsilon) r t − r 0 = O ( η ε ) . Taylor expansion gives
tanh ( r t / 2 ) = tanh ( r 0 / 2 ) + 1 2 sech 2 ( r 0 / 2 ) ( r t − r 0 ) + O ( ( r t − r 0 ) 2 ) . \tanh(r_t/2) = \tanh(r_0/2) + \frac{1}{2}\operatorname{sech}^2(r_0/2)(r_t-r_0) + O((r_t-r_0)^2). tanh ( r t /2 ) = tanh ( r 0 /2 ) + 2 1 sech 2 ( r 0 /2 ) ( r t − r 0 ) + O (( r t − r 0 ) 2 ) .
The constant term sums to O ( 1 ) O(1) O ( 1 ) because S T S_T S T is bounded. Substituting (59) into the linear term gives the stated correlation, and the quadratic remainder contributes O ( η 2 ε 3 T ) O(\eta^2\varepsilon^3T) O ( η 2 ε 3 T ) . Thus the sign is controlled, to leading order, by how the current signal correlates with accumulated past signals and the latest observation.
Step 4: the leading oscillation gives positive regret
Lemma 6.21 (Positive regret rate). For the signal in (58) , the limiting regret rate exists and satisfies
lim T → ∞ Reg 3 ( T ) T = η ε 2 sech 2 ( r 0 / 2 ) ( α 2 4 + O ( a 2 K + 1 ) ) + O ( η 2 ε 3 ) . (61) \lim_{T\to\infty} \frac{\operatorname{Reg}_3(T)}{T}
= \eta\varepsilon^2 \operatorname{sech}^2(r_0/2)
\left(\frac{\alpha^2}{4} + O(a^{2K+1})\right)
+ O(\eta^2\varepsilon^3). \tag*{(61)} T → ∞ lim T Reg 3 ( T ) = η ε 2 sech 2 ( r 0 /2 ) ( 4 α 2 + O ( a 2 K + 1 ) ) + O ( η 2 ε 3 ) . (61) The rate is positive for small a a a and then small ε \varepsilon ε , and regret differs from this rate times T T T by O ( 1 ) O(1) O ( 1 ) .
Proof. We can compute the required correlation without introducing Fourier notation. Since S t + 1 = S t + f t S_{t+1}=S_t+f_t S t + 1 = S t + f t ,
2 f t S t = S t + 1 2 − S t 2 − f t 2 . 2f_tS_t=S_{t+1}^2-S_t^2-f_t^2. 2 f t S t = S t + 1 2 − S t 2 − f t 2 .
Summing telescopes. Boundedness of S T S_T S T therefore yields
− ∑ t < T f t ( S t + f t − 1 ) = 1 2 ∑ t < T f t 2 − ∑ t < T f t f t − 1 + O ( 1 ) . -\sum_{t<T}f_t(S_t+f_{t-1})=\frac{1}{2}\sum_{t<T}f_t^2-\sum_{t<T}f_tf_{t-1}+O(1). − t < T ∑ f t ( S t + f t − 1 ) = 2 1 t < T ∑ f t 2 − t < T ∑ f t f t − 1 + O ( 1 ) .
The summable transient in (57) changes the right-hand side by O ( 1 ) O(1) O ( 1 ) . Its average consequently equals the orbit average of $\frac{1}{2}(f_t^*)^2-f_t^f_{t-1}^ . For f ˉ t = α cos ( π t / 2 + θ ) \bar{f}_t=\alpha\cos(\pi t/2+\theta) f ˉ t = α cos ( π t /2 + θ ) , averaging over four rounds gives
1 4 ∑ t = 0 3 f ˉ t 2 = α 2 2 , 1 4 ∑ t = 0 3 f ˉ t f ˉ t − 1 = 0. \frac{1}{4}\sum_{t=0}^{3}\bar{f}_t^2=\frac{\alpha^2}{2},\qquad\frac{1}{4}\sum_{t=0}^{3}\bar{f}_t\bar{f}_{t-1}=0. 4 1 t = 0 ∑ 3 f ˉ t 2 = 2 α 2 , 4 1 t = 0 ∑ 3 f ˉ t f ˉ t − 1 = 0.
Thus the squared-signal term contributes α 2 / 4 \alpha^2/4 α 2 /4 , while the lag-one correlation contributes zero, regardless of θ \theta θ . Replacing f ˉ t \bar{f}_t f ˉ t by f t ∗ = f ˉ t + O ( a K + 1 ) f_t^*=\bar{f}_t+O(a^{K+1}) f t ∗ = f ˉ t + O ( a K + 1 ) changes either average by O ( a 2 K + 1 ) O(a^{2K+1}) O ( a 2 K + 1 ) , since α = Θ ( a K ) \alpha=\Theta(a^K) α = Θ ( a K ) . Lemma 6.20 gives the expansion in (61) . Choose a a a so that the parenthesis is positive, then ε \varepsilon ε so that the cubic error is smaller than its positive quadratic term. Unequal group sizes merely change the strictly positive factor sech 2 ( r 0 / 2 ) \operatorname{sech}^2(r_0/2) sech 2 ( r 0 /2 ) .
It remains to justify the exact rate and the O ( 1 ) O(1) O ( 1 ) remainder. The actual signal equals a mean-zero periodic signal plus an exponentially decaying error. Thus its partial sums, and then r t r_t r t by (59) , approach periodic sequences with exponentially decaying errors; the partial-sum limit includes a constant shift from the transient. Smoothness of tanh \tanh tanh shows that ε f t tanh ( r t / 2 ) \varepsilon f_t\tanh(r_t/2) ε f t tanh ( r t /2 ) also approaches a periodic sequence with a summable error. Its cumulative sum is its period average times T T T , plus O ( 1 ) O(1) O ( 1 ) . The fixed-action correction ε ∣ S T ∣ \varepsilon|S_T| ε ∣ S T ∣ remains bounded. This proves the exact rate and bounded remainder, and justifies taking the limit in the expansion.
Proof of Proposition 6.18. Combine Lemmas 6.17 , 6.20 and 6.21 .
Remark 6.22 (Which oscillations hurt the observer). The argument of Steps 3 and 4 applies to any mean-zero periodic signal f f f , up to a summable transient. By Lemma 6.20 , the regret rate is − η ε 2 sech 2 ( r 0 / 2 ) -\eta\varepsilon^2\operatorname{sech}^2(r_0/2) − η ε 2 sech 2 ( r 0 /2 ) times the period average of f t ( S t + f t − 1 ) f_t(S_t+f_{t-1}) f t ( S t + f t − 1 ) . The map f ↦ S t + f t − 1 f\mapsto S_t+f_{t-1} f ↦ S t + f t − 1 multiplies the Fourier mode e i ω t e^{i\omega t} e iω t by 1 / ( e i ω − 1 ) + e − i ω 1/(e^{i\omega}-1)+e^{-i\omega} 1/ ( e iω − 1 ) + e − iω , whose real part is cos ω − 1 2 \cos\omega-\frac{1}{2} cos ω − 2 1 , so by Parseval’s identity
lim T → ∞ Reg 3 ( T ) T = η ε 2 sech 2 ( r 0 / 2 ) ∑ ω ≠ 0 ( 1 2 − cos ω ) ∣ f ^ ( ω ) ∣ 2 + O ( η 2 ε 3 ) . (62) \lim_{T\to\infty}\frac{\operatorname{Reg}_3(T)}{T}
=\eta\varepsilon^2\operatorname{sech}^2(r_0/2)\sum_{\omega\ne0}\left(\frac{1}{2}-\cos\omega\right)|\hat{f}(\omega)|^2+O(\eta^2\varepsilon^3). \tag*{(62)} T → ∞ lim T Reg 3 ( T ) = η ε 2 sech 2 ( r 0 /2 ) ω = 0 ∑ ( 2 1 − cos ω ) ∣ f ^ ( ω ) ∣ 2 + O ( η 2 ε 3 ) . (62)
The integrator 1 / ( e i ω − 1 ) 1/(e^{i\omega}-1) 1/ ( e iω − 1 ) is the Hedge part of the update, and its real part − 1 2 -\frac{1}{2} − 2 1 makes every frequency contribute positively. The term e − i ω e^{-i\omega} e − iω is the optimistic correction. For slow signals cos ω ≈ 1 \cos\omega\approx1 cos ω ≈ 1 and the rate is negative: the observer anticipates the next loss and beats the best fixed action, which is the mechanism behind the upper bounds. For ω > π / 3 \omega>\pi/3 ω > π /3 the correction is out of phase and the regret grows linearly. This is why the first block, of frequency 2 π / M ≈ 0 2\pi/M\approx0 2 π / M ≈ 0 , cannot be used directly, and why the cascade ends at ω = π / 2 \omega=\pi/2 ω = π /2 , where 1 2 − cos ω = 1 2 \frac{1}{2}-\cos\omega=\frac{1}{2} 2 1 − cos ω = 2 1 .
The cascade with a shared normalizer
In the three-player game all k k k blocks are carried by players 1 and 2 themselves. Player 1 (and symmetrically player 2) has the actions ( j , w ) (j,w) ( j , w ) for 1 ≤ j ≤ k 1\le j\le k 1 ≤ j ≤ k and w ∈ V w\in V w ∈ V , together with n − 8 k ≥ 1 n-8k\ge1 n − 8 k ≥ 1 neutral actions, and her mixed strategy is parameterized by k k k complex scores through
p ( j , w ) = e Re ( w ˉ z j ) Z ( z 1 , … , z k ) , Z ( z 1 , … , z k ) = ∑ j = 1 k ∑ w ∈ V e Re ( w ˉ z j ) + ( n − 8 k ) . p(j,w)=\frac{e^{\operatorname{Re}(\bar{w}z_j)}}{Z(z_1,\ldots,z_k)},\qquad Z(z_1,\ldots,z_k)=\sum_{j=1}^{k}\sum_{w\in V}e^{\operatorname{Re}(\bar{w}z_j)}+(n-8k). p ( j , w ) = Z ( z 1 , … , z k ) e Re ( w ˉ z j ) , Z ( z 1 , … , z k ) = j = 1 ∑ k w ∈ V ∑ e Re ( w ˉ z j ) + ( n − 8 k ) .
The utility of action ( j , v ) (j,v) ( j , v ) against the opponent’s action ( j ′ , w ) (j',w) ( j ′ , w ) is given by (47) if j = j ′ = 1 j=j'=1 j = j ′ = 1 , by the self-interaction Re ( v ˉ [ − κ c ( w ) ] ) \operatorname{Re}(\bar{v}[-\kappa c(w)]) Re ( v ˉ [ − κ c ( w )]) if j = j ′ ≥ 2 j=j'\ge2 j = j ′ ≥ 2 , by the coupling Re ( v ˉ v q ( w ) e − Re ( w ˉ δ j − 1 ) ) \operatorname{Re}(\bar{v}vq(w)e^{-\operatorname{Re}(\bar{w}\delta_{j-1})}) Re ( v ˉ v q ( w ) e − Re ( w ˉ δ j − 1 ) ) if j ′ = j − 1 j'=j-1 j ′ = j − 1 , where δ 1 = δ \delta_1=\delta δ 1 = δ and δ j = 0 \delta_j=0 δ j = 0 for j ≥ 2 j\ge2 j ≥ 2 , and equals 0 otherwise; neutral actions have utility 0 (Figure 10 ).
Figure 10. Block structure of the utility table of players 1 and 2 in the cascade. Each entry gives the utility Re ( v ‾ ⋅ ) \operatorname{Re}(\overline{v}\cdot) Re ( v ⋅ ) of own action ( j , v ) (j,v) ( j , v ) against the opponent’s action ( j ′ , w ) (j',w) ( j ′ , w ) : the oscillator (47) in block ( 1 , 1 ) (1,1) ( 1 , 1 ) (blue), the restoring self-interaction − κ c ( w ) -\kappa c(w) − κ c ( w ) on the rest of the diagonal (gray), and the coupling ν q ( w ) e − Re ( w δ j − 1 ) \nu q(w)e^{-\operatorname{Re}(w\delta_{j-1})} ν q ( w ) e − Re ( w δ j − 1 ) to the preceding block on the subdiagonal (red). Blank entries, including all neutral rows and columns, have utility 0.
Here κ = ν = 1 / 64 \kappa=\nu=1/64 κ = ν = 1/64 as in Lemma 6.15 . Unlike in § 9.1 , the paired opponent of block j j j and its source are now the same player.
For u = u 1 + i u 2 ∈ C u=u_1+iu_2\in\mathbb{C} u = u 1 + i u 2 ∈ C write
S ( u ) = sinh u 1 + i sinh u 2 , Ψ ( u ) = ∑ w ∈ V q ( w ) e Re ( w ‾ u ) = 2 cosh u 1 − 2 cosh u 2 + 2 i sinh u 1 sinh u 2 , S(u)=\sinh u_1+i\sinh u_2,\qquad\Psi(u)=\sum_{w\in V}q(w)e^{\operatorname{Re}(\overline{w}u)}=2\cosh u_1-2\cosh u_2+2i\sinh u_1\sinh u_2, S ( u ) = sinh u 1 + i sinh u 2 , Ψ ( u ) = w ∈ V ∑ q ( w ) e Re ( w u ) = 2 cosh u 1 − 2 cosh u 2 + 2 i sinh u 1 sinh u 2 ,
and, for z = ( z 1 , … , z k ) z=(z_1,\ldots,z_k) z = ( z 1 , … , z k ) ,
g 1 ( z ) = 2 μ S ( z 1 − δ ) Z ( z ) , g j ( z ) = − 2 κ S ( z j ) + ν Ψ ( z j − 1 − δ j − 1 ) Z ( z ) ( 2 ≤ j ≤ k ) . (63) g_1(z)=\frac{2\mu S(z_1-\delta)}{Z(z)},\qquad g_j(z)=\frac{-2\kappa S(z_j)+\nu\Psi(z_{j-1}-\delta_{j-1})}{Z(z)}\qquad(2\leq j\leq k). \tag*{(63)} g 1 ( z ) = Z ( z ) 2 μ S ( z 1 − δ ) , g j ( z ) = Z ( z ) − 2 κ S ( z j ) + ν Ψ ( z j − 1 − δ j − 1 ) ( 2 ≤ j ≤ k ) . (63)
These are (48) and (56) with the shared normalizer in place of the block-specific ones; for instance p ( j − 1 , w ) q ( w ) e − Re ( w ‾ δ j − 1 ) = q ( w ) e Re ( w ‾ ( z j − 1 − δ j − 1 ) ) / Z ( z ) p(j-1,w)q(w)e^{-\operatorname{Re}(\overline{w}\delta_{j-1})}=q(w)e^{\operatorname{Re}(\overline{w}(z_{j-1}-\delta_{j-1}))}/Z(z) p ( j − 1 , w ) q ( w ) e − Re ( w δ j − 1 ) = q ( w ) e Re ( w ( z j − 1 − δ j − 1 )) / Z ( z ) . Since an opponent action lies in exactly one block, and expected utility is linear in the opponent’s mixed strategy, the expected utility of action ( j , v ) (j,v) ( j , v ) is Re ( v ‾ g j ( z ) ) \operatorname{Re}(\overline{v}g_j(z)) Re ( v g j ( z )) . The calculation of Lemma 6.1 therefore applies to every block at once: the strategies keep the displayed form, the two players remain identical, and their joint state s = ( z j , ϱ j ) j ≤ k ∈ C 2 k s=(z_j,\varrho_j)_{j\leq k}\in\mathbb{C}^{2k} s = ( z j , ϱ j ) j ≤ k ∈ C 2 k evolves by
z j , t + 1 = z j , t + η [ 2 g j ( z t ) − ϱ j , t ] , ϱ j , t + 1 = g j ( z t ) , s 0 = 0. (64) z_{j,t+1}=z_{j,t}+\eta[2g_j(z_t)-\varrho_{j,t}],\qquad\varrho_{j,t+1}=g_j(z_t),\qquad s_0=0. \tag*{(64)} z j , t + 1 = z j , t + η [ 2 g j ( z t ) − ϱ j , t ] , ϱ j , t + 1 = g j ( z t ) , s 0 = 0. (64)
Denote this one-step map by Φ μ , δ \Phi_{\mu,\delta} Φ μ , δ . Write K j = 2 j − 1 K_j=2^{j-1} K j = 2 j − 1 and Λ j = λ K j \Lambda_j=\lambda^{K_j} Λ j = λ K j , so that Λ 1 = λ \Lambda_1=\lambda Λ 1 = λ and Λ k = e i π / 2 \Lambda_k=e^{i\pi/2} Λ k = e iπ /2 when M = 2 k + 1 M=2^{k+1} M = 2 k + 1 .
Theorem 6.23 (The cascade). Let k ≥ 1 k\geq1 k ≥ 1 , n ≥ 8 k + 1 n\geq8k+1 n ≥ 8 k + 1 and M = 2 k + 1 M=2^{k+1} M = 2 k + 1 , and suppose 0 < η ≤ 1 / 100 0<\eta\leq1/100 0 < η ≤ 1/100 and η M ≥ 16 π n \eta M\geq16\pi n η M ≥ 16 π n . For every sufficiently small a > 0 a>0 a > 0 there are μ , δ ∈ C \mu,\delta\in\mathbb{C} μ , δ ∈ C , with δ = O ( a ) \delta=O(a) δ = O ( a ) , for which all utilities lie in [ − 1 2 , 1 2 ] [-\frac{1}{2},\frac{1}{2}] [ − 2 1 , 2 1 ] and (64) has an orbit s t ∗ = ( z j , t ∗ , ϱ j , t ∗ ) j ≤ k s_t^*=(z_{j,t}^*,\varrho_{j,t}^*)_{j\leq k} s t ∗ = ( z j , t ∗ , ϱ j , t ∗ ) j ≤ k of exact period M M M with
z 1 , t ∗ = δ + a λ t + O ( a 2 ) , z j , t ∗ = c j a K j Λ j t + O ( a K j + 1 ) ( 2 ≤ j ≤ k ) , z_{1,t}^*=\delta+a\lambda^t+O(a^2),\qquad z_{j,t}^*=c_ja^{K_j}\Lambda_j^t+O(a^{K_j+1})\qquad(2\leq j\leq k), z 1 , t ∗ = δ + a λ t + O ( a 2 ) , z j , t ∗ = c j a K j Λ j t + O ( a K j + 1 ) ( 2 ≤ j ≤ k ) , where the constants c j ≠ 0 c_j\ne0 c j = 0 do not depend on a a a and the remainders are M M M -periodic. The trajectory from the uniform start satisfies
∥ s t − s t ∗ ∥ ≤ C a ρ t ( t ≥ 0 ) \lVert s_t-s_t^*\rVert\leq Ca\rho^t\qquad(t\geq0) ∥ s t − s t ∗ ∥ ≤ C a ρ t ( t ≥ 0 ) for some C < ∞ C<\infty C < ∞ and ρ < 1 \rho<1 ρ < 1 , independent of t t t and of sufficiently small a a a .
Feedback through the normalizer. § 7.1 and § 9.1 gave each block its own normalizer, so information flowed only forward, from block j − 1 j-1 j − 1 to block j j j . The shared normalizer couples the blocks in both directions: later blocks act back on earlier ones, including block 1, through Z Z Z . This feedback is harmless for three reasons. First, it does not change the form of the strategies: block j j j ’s probabilities remain proportional to e Re ( w ‾ z j ) e^{\operatorname{Re}(\overline{w}z_j)} e Re ( w z j ) , and Z Z Z only rescales each coefficient g j g_j g j . Second, it is small: block j j j contributes ∑ w ∈ V e Re ( w ‾ z j ) − 8 = 3 ∣ z j ∣ 2 + O ( ∣ z j ∣ 4 ) \sum_{w\in V}e^{\operatorname{Re}(\overline{w}z_j)}-8=3|z_j|^2+O(|z_j|^4) ∑ w ∈ V e Re ( w z j ) − 8 = 3∣ z j ∣ 2 + O ( ∣ z j ∣ 4 ) to Z Z Z , so the blocks j ≥ 2 j\geq2 j ≥ 2 change Z Z Z only by O ( a 4 ) O(a^4) O ( a 4 ) , and this change is nearly constant in time, because its leading term depends only on ∣ z j ∣ |z_j| ∣ z j ∣ . Third, Z − n Z-n Z − n is quadratic in the scores, so the feedback disappears from the linearizations used below. The forward structure therefore survives only in these linearizations, and the proof applies the implicit function theorem and the stable manifold theorem to all blocks jointly rather than block by block.
Proof of Theorem 6.23 . For k = 1 k=1 k = 1 this is Theorem 6.2 , so let k ≥ 2 k\ge2 k ≥ 2 . We follow Steps 3–6 of § 7.1 , with all blocks at once.
Rescaled residuals. Each block is measured in units of its expected amplitude. The unknowns are M M M -periodic complex sequences h j = ( h j , t ) t m o d M h_j=(h_{j,t})_{t\ \mathrm{mod}\ M} h j = ( h j , t ) t mod M , 1 ≤ j ≤ k 1\le j\le k 1 ≤ j ≤ k , and μ \mu μ ; the parameters are ( a , δ ) (a,\delta) ( a , δ ) . Put
z 1 , t = δ + a h 1 , t , z j , t = a K j h j , t ( 2 ≤ j ≤ k ) , z_{1,t}=\delta+ah_{1,t},\qquad z_{j,t}=a^{K_j}h_{j,t}\qquad(2\le j\le k), z 1 , t = δ + a h 1 , t , z j , t = a K j h j , t ( 2 ≤ j ≤ k ) ,
and define, for 1 ≤ j ≤ k 1\le j\le k 1 ≤ j ≤ k and t m o d M t\ \mathrm{mod}\ M t mod M ,
R j , t = h j , t + 1 − h j , t − η a K j [ 2 g j ( z t ) − g j ( z t − 1 ) ] , \mathcal{R}_{j,t}=h_{j,t+1}-h_{j,t}-\frac{\eta}{a^{K_j}}\left[2g_j(z_t)-g_j(z_{t-1})\right], R j , t = h j , t + 1 − h j , t − a K j η [ 2 g j ( z t ) − g j ( z t − 1 ) ] ,
together with the normalization N N N of (51) applied to h 1 h_1 h 1 . This gives k M + 1 kM+1 k M + 1 complex equations in k M + 1 kM+1 k M + 1 complex unknowns. A zero with a ≠ 0 a\ne0 a = 0 is an exact M M M -periodic orbit of (64) , with memories ϱ j , t ∗ = g j ( z t − 1 ∗ ) \varrho^*_{j,t}=g_j(z^*_{t-1}) ϱ j , t ∗ = g j ( z t − 1 ∗ ) .
Analyticity across a = 0 a=0 a = 0 . The functions S S S and Ψ \Psi Ψ are real analytic, S ( 0 ) = 0 S(0)=0 S ( 0 ) = 0 with derivative the identity, and Ψ ( u ) = u 2 + O ( ∣ u ∣ 4 ) \Psi(u)=u^2+O(|u|^4) Ψ ( u ) = u 2 + O ( ∣ u ∣ 4 ) by the proof of Lemma 6.13. Hence S ( ϵ v ) / ϵ S(\epsilon v)/\epsilon S ( ϵ v ) / ϵ and Ψ ( ϵ v ) / ϵ 2 \Psi(\epsilon v)/\epsilon^2 Ψ ( ϵ v ) / ϵ 2 are real analytic in ( ϵ , v ) (\epsilon,v) ( ϵ , v ) , for real ϵ \epsilon ϵ , with values v v v and v 2 v^2 v 2 at ϵ = 0 \epsilon=0 ϵ = 0 . Now z 1 , t − δ = a h 1 , t z_{1,t}-\delta=ah_{1,t} z 1 , t − δ = a h 1 , t , z j − 1 , t − δ j − 1 = a K j − 1 h j − 1 , t z_{j-1,t}-\delta_{j-1}=a^{K_{j-1}}h_{j-1,t} z j − 1 , t − δ j − 1 = a K j − 1 h j − 1 , t for j ≥ 3 j\ge3 j ≥ 3 , and K j = 2 K j − 1 K_j=2K_{j-1} K j = 2 K j − 1 . So each quotient g j ( z t ) / a K j g_j(z_t)/a^{K_j} g j ( z t ) / a K j , and hence each residual, extends real analytically across a = 0 a=0 a = 0 . At a = 0 a=0 a = 0 all scores except z 1 = δ z_1=\delta z 1 = δ vanish, so Z ( z t ) = Z ( δ ) Z(z_t)=Z(\delta) Z ( z t ) = Z ( δ ) , the single-block normalizer of (46) , and
g 1 ( z t ) a ∣ a = 0 = 2 μ h 1 , t Z ( δ ) , g j ( z t ) a K j ∣ a = 0 = − 2 κ h j , t + ν h j − 1 , t 2 Z ( δ ) ( j ≥ 2 ) . \left.\frac{g_1(z_t)}{a}\right|_{a=0}=\frac{2\mu h_{1,t}}{Z(\delta)},\qquad
\left.\frac{g_j(z_t)}{a^{K_j}}\right|_{a=0}=\frac{-2\kappa h_{j,t}+\nu h_{j-1,t}^2}{Z(\delta)}\qquad(j\ge2). a g 1 ( z t ) a = 0 = Z ( δ ) 2 μ h 1 , t , a K j g j ( z t ) a = 0 = Z ( δ ) − 2 κ h j , t + ν h j − 1 , t 2 ( j ≥ 2 ) .
In particular, at a = 0 a=0 a = 0 the residuals of block 1 are those of (51) , and those of block j ≥ 2 j\ge2 j ≥ 2 involve only h j h_j h j and h j − 1 h_{j-1} h j − 1 . The normalizer feedback has disappeared: it enters g i g_i g i through the scores z j = a K j h j z_j=a^{K_j}h_j z j = a K j h j only at positive order in a a a .
Known zeros at a = 0 a = 0 a = 0 . For each small δ \delta δ put h 1 , t = λ t h_{1,t} = \lambda^t h 1 , t = λ t , μ = μ ∗ ( δ ) \mu= \mu_*(\delta) μ = μ ∗ ( δ ) , and h j , t = c j ( δ ) Λ j t h_{j,t} = c_j(\delta)\Lambda_j^t h j , t = c j ( δ ) Λ j t for j ≥ 2 j \ge2 j ≥ 2 , where c 1 ( δ ) = 1 c_1(\delta) = 1 c 1 ( δ ) = 1 and
c j ( δ ) = ( η ν / Z ( δ ) ) ( 2 − Λ j − 1 ) Λ j − 1 + γ δ ( 2 − Λ j − 1 ) c j − 1 ( δ ) 2 , γ δ = 2 η κ Z ( δ ) . c_j(\delta) = \frac{(\eta\nu/Z(\delta))(2-\Lambda_j^{-1})}{\Lambda_j-1+\gamma_\delta(2-\Lambda_j^{-1})}c_{j-1}(\delta)^2,\qquad\gamma_\delta= \frac{2\eta\kappa}{Z(\delta)}. c j ( δ ) = Λ j − 1 + γ δ ( 2 − Λ j − 1 ) ( η ν / Z ( δ )) ( 2 − Λ j − 1 ) c j − 1 ( δ ) 2 , γ δ = Z ( δ ) 2 η κ .
Block 1 solves its equations by Lemma 6.6 . For j ≥ 2 j \ge2 j ≥ 2 , h j − 1 , t 2 = c j − 1 2 Λ j − 1 2 t = c j − 1 2 Λ j t h_{j-1,t}^2 = c_{j-1}^2\Lambda_{j-1}^{2t} = c_{j-1}^2\Lambda_j^t h j − 1 , t 2 = c j − 1 2 Λ j − 1 2 t = c j − 1 2 Λ j t , so the equations of block j j j at a = 0 a = 0 a = 0 are those of Lemma 6.15 with Λ = Λ j \Lambda= \Lambda_j Λ = Λ j , n n n replaced by Z ( δ ) Z(\delta) Z ( δ ) , and input coefficient c j − 1 2 c_{j-1}^2 c j − 1 2 ; the computation there gives c j c_j c j . Its denominator equals ( Λ j − r + ) ( Λ j − r − ) / Λ j (\Lambda_j-r_+)(\Lambda_j-r_-)/\Lambda_j ( Λ j − r + ) ( Λ j − r − ) / Λ j , where r ± r_\pm r ± are the eigenvalues of B with γ = γ δ ∈ ( 0 , 2 / 3 ) \gamma= \gamma_\delta\in(0,2/3) γ = γ δ ∈ ( 0 , 2/3 ) ; it is nonzero because ∣ r ± ∣ < 1 = ∣ Λ j ∣ \lvert r_\pm\rvert< 1 = \lvert\Lambda_j\rvert ∣ r ± ∣ < 1 = ∣ Λ j ∣ . Its numerator is nonzero, so c j ( δ ) ≠ 0 c_j(\delta) \ne0 c j ( δ ) = 0 by induction. These sequences are M M M -periodic because Λ j M = 1 \Lambda_j^M = 1 Λ j M = 1 .
Invertible Jacobian. Order the unknowns as ( h 1 , μ , h 2 , … , h k ) (h_1,\mu,h_2,\ldots,h_k) ( h 1 , μ , h 2 , … , h k ) and the equations as ( R 1 , N , R 2 , … , R k ) (\mathcal{R}_1,N,\mathcal{R}_2,\ldots,\mathcal{R}_k) ( R 1 , N , R 2 , … , R k ) . By the formulas at a = 0 a = 0 a = 0 , the Jacobian at ( a , δ ) = ( 0 , 0 ) (a,\delta) = (0,0) ( a , δ ) = ( 0 , 0 ) and the known zero is block lower-triangular. Its first diagonal block is the Jacobian of Lemma 6.6 , which is invertible. For j ≥ 2 j \ge2 j ≥ 2 the diagonal block is
h ˙ ↦ ( h ˙ t + 1 − h ˙ t + γ ( 2 h ˙ t − h ˙ t − 1 ) ) t , γ = 2 η κ n , \dot{h} \mapsto\left(\dot{h}_{t+1}-\dot{h}_t+\gamma(2\dot{h}_t-\dot{h}_{t-1})\right)_t,\qquad\gamma= \frac{2\eta\kappa}{n}, h ˙ ↦ ( h ˙ t + 1 − h ˙ t + γ ( 2 h ˙ t − h ˙ t − 1 ) ) t , γ = n 2 η κ ,
which multiplies the Fourier mode ζ t \zeta^t ζ t , ζ M = 1 \zeta^M = 1 ζ M = 1 , by ( ζ − r + ) ( ζ − r − ) / ζ ≠ 0 (\zeta-r_+)(\zeta-r_-)/\zeta\ne0 ( ζ − r + ) ( ζ − r − ) / ζ = 0 . Hence the Jacobian is invertible. Theorem 6.5 gives real-analytic h j ( a , δ ) h_j(a,\delta) h j ( a , δ ) and μ ( a , δ ) \mu(a,\delta) μ ( a , δ ) near ( 0 , 0 ) (0,0) ( 0 , 0 ) , and local uniqueness identifies them at a = 0 a = 0 a = 0 with the known zeros. Thus h j , t ( a , δ ) = c j ( δ ) Λ j t + O ( a ) h_{j,t}(a,\delta) = c_j(\delta)\Lambda_j^t+O(a) h j , t ( a , δ ) = c j ( δ ) Λ j t + O ( a ) , uniformly in t t t and small δ \delta δ . For a ≠ 0 a \ne0 a = 0 the orbit has exact period M M M , because block 1 does (proof of Lemma 6.4 ).
Strong stable manifold. At a = δ = 0 a = \delta= 0 a = δ = 0 and μ = μ 0 \mu= \mu_0 μ = μ 0 the origin of C 2 k \mathbb{C}^{2k} C 2 k is fixed by Φ μ 0 , 0 \Phi_{\mu_0,0} Φ μ 0 , 0 . Since Z − n Z^{-n} Z − n and Ψ \Psi Ψ vanish to second order at the origin, neither the normalizer nor the coupling contributes to the derivative there, and
D Φ μ 0 , 0 ( 0 ) = diag ( A , B , … , B ) . D\Phi_{\mu_0,0}(0)=\operatorname{diag}(A,B,\ldots,B). D Φ μ 0 , 0 ( 0 ) = diag ( A , B , … , B ) .
This is invertible, since det A = β ≠ 0 \det A = \beta\ne0 det A = β = 0 and det B = − γ ≠ 0 \det B = -\gamma\ne0 det B = − γ = 0 . Its M M M th power has the complex eigenvalues 1 1 1 and χ M \chi^M χ M on block 1 and r ± M r_\pm^M r ± M on every other block. Fix ρ ˉ \bar{\rho} ρ ˉ with max { ∣ χ ∣ , ∣ r + ∣ , ∣ r − ∣ } M < ρ ˉ < 1 \max\{\lvert\chi\rvert,\lvert r_+\rvert,\lvert r_-\rvert\}^M < \bar{\rho} < 1 max {∣ χ ∣ , ∣ r + ∣ , ∣ r − ∣ } M < ρ ˉ < 1 . The strongly stable space is E s ⊕ ( C 2 ) k − 1 E_s \oplus(\mathbb{C}^2)^{k-1} E s ⊕ ( C 2 ) k − 1 , of real dimension 4 k − 2 4k-2 4 k − 2 , and its complement is E c E_c E c in the coordinates of block 1, of real dimension 2. The proofs of Lemmas 6.10 and 6.8 now apply verbatim to G p = Φ μ ( p ) , δ M G_p=\Phi_{\mu(p),\delta}^M G p = Φ μ ( p ) , δ M and its fixed points s 0 ∗ ( p ) s_0^*(p) s 0 ∗ ( p ) , p = ( a , δ ) p=(a,\delta) p = ( a , δ ) . They give a local strong stable manifold
W a , δ s s = { s 0 ∗ + ξ + Γ ( a , δ , ξ ) : ξ ∈ E s ⊕ ( C 2 ) k − 1 small } , W_{a,\delta}^{ss}=\left\{s_0^*+\xi+\Gamma(a,\delta,\xi):\xi\in E_s\oplus(\mathbb{C}^2)^{k-1}\text{ small}\right\}, W a , δ ss = { s 0 ∗ + ξ + Γ ( a , δ , ξ ) : ξ ∈ E s ⊕ ( C 2 ) k − 1 small } ,
with Γ \Gamma Γ valued in E c E_c E c , jointly C 1 C^1 C 1 , Γ ( a , δ , 0 ) = 0 \Gamma(a,\delta,0)=0 Γ ( a , δ , 0 ) = 0 and D ξ Γ ( 0 , 0 , 0 ) = 0 D_\xi\Gamma(0,0,0)=0 D ξ Γ ( 0 , 0 , 0 ) = 0 , whose points converge to the orbit at a uniform rate C ρ ˉ t C\bar{\rho}^t C ρ ˉ t in every round.
Matching the uniform start. Let P c P_c P c and P s P_s P s be the projections of the joint splitting. The uniform start 0 0 0 lies on W a , δ s s W_{a,\delta}^{ss} W a , δ ss exactly when Δ ( a , δ ) = P c ( − s 0 ∗ ) − Γ ( a , δ , P s ( − s 0 ∗ ) ) = 0 \Delta(a,\delta)=P_c(-s_0^*)-\Gamma(a,\delta,P_s(-s_0^*))=0 Δ ( a , δ ) = P c ( − s 0 ∗ ) − Γ ( a , δ , P s ( − s 0 ∗ )) = 0 . At a = 0 a=0 a = 0 every score except z 1 = δ z_1=\delta z 1 = δ vanishes, and every money vanishes because each h j h_j h j vanishes there; so s 0 ∗ ( 0 , δ ) = ( ( δ , 0 ) , 0 , … , 0 ) s_0^*(0,\delta)=((\delta,0),0,\ldots,0) s 0 ∗ ( 0 , δ ) = (( δ , 0 ) , 0 , … , 0 ) . As in Lemma 6.12 , D δ Δ ( 0 , 0 ) [ d δ ] = − P c ( ( d δ , 0 ) , 0 , … , 0 ) D_\delta\Delta(0,0)[d\delta]=-P_c((d\delta,0),0,\ldots,0) D δ Δ ( 0 , 0 ) [ d δ ] = − P c (( d δ , 0 ) , 0 , … , 0 ) , which is multiplication by − λ / ( λ − χ ) ≠ 0 -\lambda/(\lambda-\chi)\ne0 − λ / ( λ − χ ) = 0 in the coordinate u c u_c u c on E c E_c E c . Theorem 6.5 gives a C 1 C^1 C 1 solution δ = δ ( a ) = O ( a ) \delta=\delta(a)=O(a) δ = δ ( a ) = O ( a ) . Since ∥ s 0 ∗ ∥ = O ( a ) \lVert s_0^*\rVert=O(a) ∥ s 0 ∗ ∥ = O ( a ) , the trajectory from the uniform start satisfies ∥ s t − s t ∗ ∥ ≤ C a ρ ˉ t \lVert s_t-s_t^*\rVert\le C a\bar{\rho}^t ∥ s t − s t ∗ ∥ ≤ C a ρ ˉ t .
Leading terms. With δ = δ ( a ) = O ( a ) \delta=\delta(a)=O(a) δ = δ ( a ) = O ( a ) we have c j ( δ ( a ) ) = c j ( 0 ) + O ( a ) c_j(\delta(a))=c_j(0)+O(a) c j ( δ ( a )) = c j ( 0 ) + O ( a ) . Setting c j = c j ( 0 ) ≠ 0 c_j=c_j(0)\ne0 c j = c j ( 0 ) = 0 gives z j , t ∗ = a K j h j , t = c j a K j Λ j t + O ( a K j + 1 ) z_{j,t}^*=a^{K_j}h_{j,t}=c_ja^{K_j}\Lambda_j^t+O(a^{K_j+1}) z j , t ∗ = a K j h j , t = c j a K j Λ j t + O ( a K j + 1 ) for j ≥ 2 j\ge2 j ≥ 2 , and z 1 , t ∗ = δ + a λ t + O ( a 2 ) z_{1,t}^*=\delta+a\lambda^t+O(a^2) z 1 , t ∗ = δ + a λ t + O ( a 2 ) .
Bounded utilities. The utilities of block 1 are bounded as in the proof of Theorem 6.2 . An action ( j , v ) (j,v) ( j , v ) with j ≥ 2 j\ge2 j ≥ 2 receives a nonzero utility only against opponent actions in block j j j , where it is at most 2 κ \sqrt{2}\kappa 2 κ in absolute value, or in block j − 1 j-1 j − 1 , where it is at most 2 ν e 2 ∣ q ∣ \sqrt{2}\nu e^{\sqrt{2}\lvert q\rvert} 2 ν e 2 ∣ q ∣ , since ∣ q ∣ ≤ 1 \lvert q\rvert\le1 ∣ q ∣ ≤ 1 . Both are below 1 2 \frac{1}{2} 2 1 for small a a a . Neutral actions and all remaining pairs have utility 0 0 0 .
Proof of Theorem 3.6
We first prove the second part of the theorem, linear regret for η ≥ C 0 n 2 − n / 8 \eta\ge C_0n2^{-n/8} η ≥ C 0 n 2 − n /8 , and then deduce the first.
Proof of linear regret for η ≥ C 0 n 2 − n / 8 \eta\ge C_0n2^{-n/8} η ≥ C 0 n 2 − n /8 . Let η 0 = 1 / 100 \eta_0 = 1/100 η 0 = 1/100 and C 0 = 16 π C_0 = 16\pi C 0 = 16 π . Given n ≥ C 0 n \ge C_0 n ≥ C 0 (so that n ≥ 9 n \ge9 n ≥ 9 ) and η ∈ [ C 0 n 2 − n / 8 , η 0 ] \eta\in[C_0n2^{-n/8},\eta_0] η ∈ [ C 0 n 2 − n /8 , η 0 ] , put k = ⌊ ( n − 1 ) / 8 ⌋ ≥ 1 k = \lfloor(n-1)/8\rfloor\ge1 k = ⌊( n − 1 ) /8 ⌋ ≥ 1 and M = 2 k + 1 M = 2^{k+1} M = 2 k + 1 , which is divisible by four. Since k ≥ n / 8 − 1 k \ge n/8-1 k ≥ n /8 − 1 ,
16 π n M = 16 π n 2 − k − 1 ≤ 16 π n 2 − n / 8 ≤ η , \frac{16\pi n}{M}=16\pi n2^{-k-1}\le16\pi n2^{-n/8}\le\eta, M 16 π n = 16 π n 2 − k − 1 ≤ 16 π n 2 − n /8 ≤ η ,
so Theorem 6.23 applies. The k k k blocks use 8 k ≤ n − 1 8k \le n-1 8 k ≤ n − 1 active actions of each of players 1 and 2, leaving at least one neutral action; the last block oscillates at frequency 2 k − 1 ⋅ 2 π / M = π / 2 2^{k-1}\cdot2\pi/M=\pi/2 2 k − 1 ⋅ 2 π / M = π /2 . By Theorem 6.23 , players 1 and 2 converge exponentially to a joint periodic orbit whose last block satisfies the hypothesis of § 10.1 , and Proposition 6.18 furnishes the observer with linear regret. Her n n n actions are split into two nonempty groups, so G ∈ G ( 3 , n ) G \in\mathcal{G}(3,n) G ∈ G ( 3 , n ) .
Proof of the exponential bound. Let 0 < η ≤ η 0 0 < \eta\le\eta_0 0 < η ≤ η 0 . If η ≥ C 0 n 2 − n / 8 \eta\ge C_0n2^{-n/8} η ≥ C 0 n 2 − n /8 , the game just constructed has sup T Reg 3 ( T ) = ∞ \sup_T \operatorname{Reg}_3(T)=\infty sup T Reg 3 ( T ) = ∞ . If η < C 0 n 2 − n / 8 \eta< C_0n2^{-n/8} η < C 0 n 2 − n /8 , let G G G be the three-player game in which players 1 and 2 have constant losses and player 3’s loss is 0 on one action and 1 on the others, irrespective of the opponents’ actions. By Proposition A.3 , sup T Reg 3 ( T ) ≥ log n / η − 1 > 2 n / 8 log n / ( C 0 n ) − 1 \sup_T \operatorname{Reg}_3(T)\ge\log n/\eta-1>2^{n/8}\log n/(C_0n)-1 sup T Reg 3 ( T ) ≥ log n / η − 1 > 2 n /8 log n / ( C 0 n ) − 1 . The bound on inf 0 < η ≤ η 0 sup T ≥ 1 sup G ∈ G ( 3 , n ) max i Reg i ( T ) \inf_{0<\eta\le\eta_0}\sup_{T\ge1}\sup_{G\in\mathcal{G}(3,n)}\max_i\operatorname{Reg}_i(T) inf 0 < η ≤ η 0 sup T ≥ 1 sup G ∈ G ( 3 , n ) max i Reg i ( T ) follows by taking the infimum over η ∈ ( 0 , η 0 ] \eta\in(0,\eta_0] η ∈ ( 0 , η 0 ] .
The exponential form of the threshold is forced by the mechanism: the first block’s period is at least of order n / η n/\eta n / η , each halving of the period costs eight actions, and the period must reach four.
Remark. The bound with inf η \inf_\eta inf η in Theorem 3.6 concerns inf η sup T \inf_\eta\sup_T inf η sup T , the natural order for a horizon-independent guarantee. The stronger statement with sup T inf η \sup_T\inf_\eta sup T inf η , in which the learning rate may be tuned to the horizon, holds as well provided the constants c ( G , η ) c(G,\eta) c ( G , η ) and C ( G , η ) C(G,\eta) C ( G , η ) in Theorem 3.6 are bounded away from zero and from infinity, respectively, uniformly for η \eta η in compact subintervals of ( 0 , η 0 ] (0,\eta_0] ( 0 , η 0 ] ; this uniformity is what the implicit-function arguments of § 7.1 –§ 9.1 provide when η \eta η is carried along as a parameter, and we omit the details.
More than three players
The construction extends to m ≥ 3 m \ge3 m ≥ 3 players by splitting all players except the observer into pairs, each pair carrying its own segment of the cascade.
Theorem 6.24 (An exponential barrier for m m m players). Let C 0 C_0 C 0 and η 0 \eta_0 η 0 be as in Theorem 3.6 , let m ≥ 3 m \ge3 m ≥ 3 , and put P = ⌊ ( m − 1 ) / 2 ⌋ P=\lfloor(m-1)/2\rfloor P = ⌊( m − 1 ) /2 ⌋ . For every n ≥ C 0 n \ge C_0 n ≥ C 0 and every learning rate η ∈ ( 0 , η 0 ] \eta\in(0,\eta_0] η ∈ ( 0 , η 0 ] there is an m m m -player game G ∈ G ( m , n ) G\in\mathcal{G}(m,n) G ∈ G ( m , n ) in which, when all players run Optimistic Hedge with learning rate η \eta η starting from the uniform distribution,
sup T Reg m ( T ) ≥ 2 1 + P ( n − 8 ) / 8 log n C 0 n − 1 = 2 Ω ( m n ) . \sup_T \operatorname{Reg}_m(T)\ge\frac{2^{1+P(n-8)/8}\log n}{C_0n}-1=2^{\Omega(mn)}. T sup Reg m ( T ) ≥ C 0 n 2 1 + P ( n − 8 ) /8 log n − 1 = 2 Ω ( mn ) . Moreover, if η ≥ C 0 n 2 1 − P ( n − 8 ) / 8 \eta\ge C_0n2^{1-P(n-8)/8} η ≥ C 0 n 2 1 − P ( n − 8 ) /8 , then G G G can be chosen so that player m m m ’s regret grows linearly: Reg m ( T ) ≥ c T − C \operatorname{Reg}_m(T)\ge cT-C Reg m ( T ) ≥ c T − C for all T T T , for some c = c ( G , η ) > 0 c=c(G,\eta)>0 c = c ( G , η ) > 0 and C = C ( G , η ) C=C(G,\eta) C = C ( G , η ) .
For m ∈ { 3 , 4 } m\in\{3,4\} m ∈ { 3 , 4 } we have P = 1 P=1 P = 1 and 2 1 + ( n − 8 ) / 8 = 2 n / 8 2^{1+(n-8)/8}=2^{n/8} 2 1 + ( n − 8 ) /8 = 2 n /8 , so the bound is that of Theorem 3.6 .
The game. Write k 0 = ⌊ ( n − 1 ) / 8 ⌋ k_0=\lfloor(n-1)/8\rfloor k 0 = ⌊( n − 1 ) /8 ⌋ and k = P k 0 k=Pk_0 k = P k 0 . For 1 ≤ p ≤ P 1\le p\le P 1 ≤ p ≤ P , players 2 p − 1 2p-1 2 p − 1 and 2 p 2p 2 p form pair p p p , which carries the blocks j j j with ( p − 1 ) k 0 < j ≤ p k 0 (p-1)k_0<j\le pk_0 ( p − 1 ) k 0 < j ≤ p k 0 . Blocks are numbered consecutively across pairs, so the predecessor of block j j j is block j − 1 j-1 j − 1 also when the two lie in different pairs. Each player of pair p p p has the actions ( j , w ) (j,w) ( j , w ) , for the k 0 k_0 k 0 blocks j j j of her pair and w ∈ V w\in V w ∈ V , together with n − 8 k 0 ≥ 1 n-8k_0\ge1 n − 8 k 0 ≥ 1 neutral actions. Her mixed strategy is parametrized by the scores of these blocks as in § 10.5 , with the normalizer of her own pair,
Z p ( z ) = ∑ ( p − 1 ) k 0 < j ≤ p k 0 ∑ w ∈ V e Re ( w z j ) + ( n − 8 k 0 ) . Z_p(z)=\sum_{(p-1)k_0<j\le pk_0}\sum_{w\in V}e^{\operatorname{Re}(wz_j)}+(n-8k_0). Z p ( z ) = ( p − 1 ) k 0 < j ≤ p k 0 ∑ w ∈ V ∑ e Re ( w z j ) + ( n − 8 k 0 ) .
Within a pair, the utility of action ( j , v ) (j,v) ( j , v ) against the partner’s action ( j ′ , w ) (j',w) ( j ′ , w ) is as in § 10.5 , except that the first block of each pair p ≥ 2 p\ge2 p ≥ 2 has only the self-interaction Re ( v ‾ [ − κ c ( w ) ] ) \operatorname{Re}(\overline{v}[-\kappa c(w)]) Re ( v [ − κ c ( w )]) , against j ′ = j j'=j j ′ = j , and no coupling to the partner. Instead, this block j = ( p − 1 ) k 0 + 1 j=(p-1)k_0+1 j = ( p − 1 ) k 0 + 1 is driven by the last block of pair p − 1 p-1 p − 1 exactly as in § 9.1 : each player of pair p p p receives the additional utility Re ( v ‾ r δ δ j − 1 ( b ) ) \operatorname{Re}(\overline{v}r_\delta\delta_{j-1}(b)) Re ( v r δ δ j − 1 ( b )) from action ( j , v ) (j,v) ( j , v ) when player 2 p − 3 2p-3 2 p − 3 plays ( j − 1 , b ) (j-1,b) ( j − 1 , b ) , and no additional utility otherwise, where r δ r_\delta r δ is the function of Lemma 6.13. Thus the utility of a player of pair p ≥ 2 p\ge2 p ≥ 2 is the sum of a term that depends on her partner’s action and a term that depends on the action of player 2 p − 3 2p-3 2 p − 3 . Player m m m is the observer of § 10.1 , with her signal taken from the last block k k k of player 2 P − 1 2P-1 2 P − 1 instead of player 1. If m m m is even, player m − 1 m-1 m − 1 has constant losses and influences nobody.
The cascade across pairs. Both players of a pair start at the uniform distribution and receive the same utilities, so they remain identical. As in § 10.5 , the joint state s = ( z j , ϱ j ) j ≤ k s=(z_j,\varrho_j)_{j\le k} s = ( z j , ϱ j ) j ≤ k of all pairs follows (64) with k k k blocks, where now, for a block j j j of pair p p p ,
g j ( z ) = { 2 μ S ( z 1 − δ ) Z 1 ( z ) , j = 1 , − 2 κ S ( z j ) + ν Ψ ( z j − 1 − δ j − 1 ) Z p ( z ) , j ≥ 2 not the first block of pair p , − 2 κ S ( z j ) Z p ( z ) + ν Ψ ( z j − 1 − δ j − 1 ) Z p − 1 ( z ) , j the first block of pair p ≥ 2. (65) g_j(z)=
\begin{cases}
\dfrac{2\mu S(z_1-\delta)}{Z_1(z)}, & j=1,\\[6pt]
\dfrac{-2\kappa S(z_j)+\nu\Psi(z_{j-1}-\delta_{j-1})}{Z_p(z)}, & j\ge2\text{ not the first block of pair }p,\\[6pt]
\dfrac{-2\kappa S(z_j)}{Z_p(z)}+\dfrac{\nu\Psi(z_{j-1}-\delta_{j-1})}{Z_{p-1}(z)}, & j\text{ the first block of pair }p\ge2.
\end{cases} \tag*{(65)} g j ( z ) = ⎩ ⎨ ⎧ Z 1 ( z ) 2 μ S ( z 1 − δ ) , Z p ( z ) − 2 κ S ( z j ) + ν Ψ ( z j − 1 − δ j − 1 ) , Z p ( z ) − 2 κ S ( z j ) + Z p − 1 ( z ) ν Ψ ( z j − 1 − δ j − 1 ) , j = 1 , j ≥ 2 not the first block of pair p , j the first block of pair p ≥ 2. (65)
The last case is (56) : the source’s signal is normalized by the source’s own normalizer. A pair is influenced only by its own blocks and by the last block of the preceding pair. Hence the feedback through the normalizer described in § 10.5 stays inside each pair, and between pairs information flows strictly forward.
Theorem 6.25 (The cascade across pairs). The conclusions of Theorem 6.23 hold for the dynamics (64) with the coefficients (65) , for all P ≥ 1 P\ge1 P ≥ 1 , k 0 ≥ 1 k_0\ge1 k 0 ≥ 1 and n ≥ 8 k 0 + 1 n\ge8k_0+1 n ≥ 8 k 0 + 1 , with k = P k 0 k=Pk_0 k = P k 0 blocks and M = 2 k + 1 M=2^{k+1} M = 2 k + 1 , under the same conditions 0 < η ≤ 1 / 100 0<\eta\le1/100 0 < η ≤ 1/100 and η M ≥ 16 π n \eta M\ge16\pi n η M ≥ 16 π n .
Proof. The proof of Theorem 6.23 applies, with k k k blocks, after three changes, all due to the separate normalizers. For a block j ≥ 2 j\ge2 j ≥ 2 , let Z j o w n Z_j^{\mathrm{own}} Z j own and Z j s r c Z_j^{\mathrm{src}} Z j src be the values at a = 0 a=0 a = 0 of the normalizers of the pairs carrying blocks j j j and j − 1 j-1 j − 1 . Since at a = 0 a=0 a = 0 all scores except z 1 = δ z_1=\delta z 1 = δ vanish, each of these equals Z ( δ ) Z(\delta) Z ( δ ) for pair 1 and n n n for every other pair.
Residuals at a = 0 a=0 a = 0 . For j ≥ 2 j\ge2 j ≥ 2 ,
g j ( z t ) a κ j ∣ a = 0 = − 2 κ h j , t Z j o w n + ν h j − 1 , t 2 Z j s r c . \left.\frac{g_j(z_t)}{a^{\kappa_j}}\right|_{a=0}
=-\frac{2\kappa h_{j,t}}{Z_j^{\mathrm{own}}}+\frac{\nu h_{j-1,t}^2}{Z_j^{\mathrm{src}}}. a κ j g j ( z t ) a = 0 = − Z j own 2 κ h j , t + Z j src ν h j − 1 , t 2 .
so the residuals of block j j j still involve only h j h_j h j and h j − 1 h_{j-1} h j − 1 .
Known zeros. They are h j , t = c j ( δ ) Λ j t h_{j,t}=c_j(\delta)\Lambda_j^t h j , t = c j ( δ ) Λ j t with c 1 = 1 c_1=1 c 1 = 1 and
c j ( δ ) = ( η ν / Z j s r c ) ( 2 − Λ j − 1 ) Λ j − 1 + γ j ( 2 − Λ j − 1 ) c j − 1 ( δ ) 2 , γ j = 2 η κ Z j w ∈ ( 0 , 2 3 ) , c_j(\delta)=\frac{(\eta\nu/Z_j^{\mathrm{src}})(2-\Lambda_j^{-1})}{\Lambda_j-1+\gamma_j(2-\Lambda_j^{-1})}c_{j-1}(\delta)^2,\qquad\gamma_j=\frac{2\eta\kappa}{Z_j^{\mathrm{w}}}\in\left(0,\frac{2}{3}\right), c j ( δ ) = Λ j − 1 + γ j ( 2 − Λ j − 1 ) ( η ν / Z j src ) ( 2 − Λ j − 1 ) c j − 1 ( δ ) 2 , γ j = Z j w 2 η κ ∈ ( 0 , 3 2 ) ,
which are nonzero by the argument given there.
Bounded utilities. An action of the first block of a pair p ≥ 2 p\ge2 p ≥ 2 receives a utility of absolute value at most 2 κ \sqrt{2}\kappa 2 κ from its partner and at most 2 ν e 2 ∣ δ j − 1 ∣ \sqrt{2}\nu e^{2|\delta_{j-1}|} 2 ν e 2∣ δ j − 1 ∣ from player 2 p − 3 2p-3 2 p − 3 ; their sum is below 1 2 \frac{1}{2} 2 1 for small a a a .
Everything else is unchanged. At ( a , δ ) = ( 0 , 0 ) (a,\delta)=(0,0) ( a , δ ) = ( 0 , 0 ) all normalizers equal n n n , so the Jacobian is block lower-triangular with the same diagonal blocks as before. Moreover, Z p − n Z_p-n Z p − n and Ψ \Psi Ψ vanish to second order at the origin, so D Φ μ 0 , 0 ( 0 ) = diag ( A , B , … , B ) D\Phi_{\mu_0,0}(0)=\operatorname{diag}(A,B,\ldots,B) D Φ μ 0 , 0 ( 0 ) = diag ( A , B , … , B ) . Finally, the matching equation still involves only the shift δ \delta δ of block 1.
Proof of Theorem 6.24 . Let η 0 = 1 / 100 \eta_0=1/100 η 0 = 1/100 , C 0 = 16 π C_0=16\pi C 0 = 16 π , n ≥ C 0 n\ge C_0 n ≥ C 0 and k 0 = ⌊ ( n − 1 ) / 8 ⌋ ≥ 1 k_0=\lfloor(n-1)/8\rfloor\ge1 k 0 = ⌊( n − 1 ) /8 ⌋ ≥ 1 . Since k 0 ≥ ( n − 8 ) / 8 k_0\ge(n-8)/8 k 0 ≥ ( n − 8 ) /8 , the k = P k 0 k=Pk_0 k = P k 0 blocks give M = 2 k + 1 ≥ 2 1 + P ( n − 8 ) / 8 M=2^{k+1}\ge2^{1+P(n-8)/8} M = 2 k + 1 ≥ 2 1 + P ( n − 8 ) /8 .
If η ≥ C 0 n 2 − 1 − P ( n − 8 ) / 8 \eta\ge C_0n^{2^{-1-P(n-8)/8}} η ≥ C 0 n 2 − 1 − P ( n − 8 ) /8 , then η M ≥ 16 π n \eta M\ge16\pi n η M ≥ 16 π n , so Theorem 6.25 applies. The last block oscillates at frequency 2 k − 1 ⋅ 2 π / M = π / 2 2^{k-1}\cdot2\pi/M=\pi/2 2 k − 1 ⋅ 2 π / M = π /2 , and Proposition 6.18 gives player m m m linear regret; in particular sup T Reg m ( T ) = ∞ \sup_T\operatorname{Reg}_m(T)=\infty sup T Reg m ( T ) = ∞ .
If η < C 0 n 2 − 1 − P ( n − 8 ) / 8 \eta<C_0n^{2^{-1-P(n-8)/8}} η < C 0 n 2 − 1 − P ( n − 8 ) /8 , let players 1 , … , m − 1 1,\ldots,m-1 1 , … , m − 1 have constant losses, and let player m m m 's loss be 0 0 0 on one action and 1 1 1 on the others, irrespective of the opponents' actions. By Proposition A.3 , sup T Reg m ( T ) ≥ log n / η − 1 > 2 1 + P ( n − 8 ) / 8 log n / ( C 0 n ) − 1 \sup_T\operatorname{Reg}_m(T)\ge\log n/\eta-1>2^{1+P(n-8)/8}\log n/(C_0n)-1 sup T Reg m ( T ) ≥ log n / η − 1 > 2 1 + P ( n − 8 ) /8 log n / ( C 0 n ) − 1 .
Finally, n ≥ C 0 > 16 n\ge C_0>16 n ≥ C 0 > 16 gives n − 8 ≥ n / 2 n-8\ge n/2 n − 8 ≥ n /2 , and P ≥ ( m − 1 ) / 3 P\ge(m-1)/3 P ≥ ( m − 1 ) /3 for m ≥ 3 m\ge3 m ≥ 3 , so the exponent 1 + P ( n − 8 ) / 8 1+P(n-8)/8 1 + P ( n − 8 ) /8 is at least ( m − 1 ) n / 48 (m-1)n/48 ( m − 1 ) n /48 .
Elementary lemmas on exponential weights
This appendix collects the facts about exponential tilting used throughout, the proof of the potential inequality, and the regret of Optimistic Hedge against a constant loss vector. For a probability vector p p p and a real vector v v v of the same dimension, the variance has the variational characterization
Var p ( v ) = ∑ a p ( a ) ( v ( a ) − ⟨ p , v ⟩ ) 2 = min c ∈ R ∑ a p ( a ) ( v ( a ) − c ) 2 (66) \operatorname{Var}_p(v)=\sum_a p(a)(v(a)-\langle p,v\rangle)^2=\min_{c\in\mathbb{R}}\sum_a p(a)(v(a)-c)^2 \tag*{(66)} Var p ( v ) = a ∑ p ( a ) ( v ( a ) − ⟨ p , v ⟩ ) 2 = c ∈ R min a ∑ p ( a ) ( v ( a ) − c ) 2 (66)
so ∥ v ∥ p = Var p ( v ) \lVert v\rVert_p=\sqrt{\operatorname{Var}_p(v)} ∥ v ∥ p = Var p ( v ) is a Hilbert seminorm whose kernel is the constant vectors when p p p has full support.
Lemma A.1 (Variance facts). Let p , p ′ p,p' p , p ′ be probability vectors and u , v u,v u , v real vectors. If p ′ ( a ) ≤ Γ p ( a ) p'(a)\le\Gamma p(a) p ′ ( a ) ≤ Γ p ( a ) for all a a a , then Var p ′ ( v ) ≤ Γ Var p ( v ) \operatorname{Var}_{p'}(v)\le\Gamma\operatorname{Var}_p(v) Var p ′ ( v ) ≤ Γ Var p ( v ) . Moreover Var p ( u + v ) ≤ 2 Var p ( u ) + 2 Var p ( v ) \operatorname{Var}_p(u+v)\le2\operatorname{Var}_p(u)+2\operatorname{Var}_p(v) Var p ( u + v ) ≤ 2 Var p ( u ) + 2 Var p ( v ) , and Var p ( v ) ≤ b 2 / 4 \operatorname{Var}_p(v)\le b^2/4 Var p ( v ) ≤ b 2 /4 whenever all entries of v v v lie in an interval of length b b b ; in particular Var p ( v ) ≤ ∥ v ∥ ∞ 2 \operatorname{Var}_p(v)\le\lVert v\rVert_\infty^2 Var p ( v ) ≤ ∥ v ∥ ∞ 2 .
Proof. The first claim follows from (66) with c = ⟨ p , v ⟩ c=\langle p,v\rangle c = ⟨ p , v ⟩ , the second from the seminorm triangle inequality and ( α + β ) 2 ≤ 2 α 2 + 2 β 2 (\alpha+\beta)^2\le2\alpha^2+2\beta^2 ( α + β ) 2 ≤ 2 α 2 + 2 β 2 , and the third from (66) with c c c the midpoint of the interval.
Recall that E p [ z ] E_p[z] E p [ z ] denotes the probability vector with entries p ( a ) e z ( a ) / ∑ c p ( c ) e z ( c ) p(a)e^{z(a)}/\sum_c p(c)e^{z(c)} p ( a ) e z ( a ) / ∑ c p ( c ) e z ( c ) , that osc ( z ) = max a z ( a ) − min a z ( a ) \operatorname{osc}(z)=\max_a z(a)-\min_a z(a) osc ( z ) = max a z ( a ) − min a z ( a ) , and that ∥ z ∥ p = Var p ( z ) \lVert z\rVert_p=\sqrt{\operatorname{Var}_p(z)} ∥ z ∥ p = Var p ( z ) . For probability vectors p , q p,q p , q with q ( a ) > 0 q(a)>0 q ( a ) > 0 whenever p ( a ) > 0 p(a)>0 p ( a ) > 0 , KL ( p ∥ q ) = ∑ a : p ( a ) > 0 p ( a ) log p ( a ) q ( a ) \operatorname{KL}(p\mathbin\Vert q)=\sum_{a:p(a)>0}p(a)\log\frac{p(a)}{q(a)} KL ( p ∥ q ) = ∑ a : p ( a ) > 0 p ( a ) log q ( a ) p ( a ) .
Lemma A.2 (Exponential tilting). Let p ∈ Δ n p \in\Delta_n p ∈ Δ n , let z ∈ R n z \in\mathbb{R}^n z ∈ R n with osc ( z ) ≤ r \operatorname{osc}(z) \le r osc ( z ) ≤ r , and let q = E p [ z ] q = \mathbb{E}_p[z] q = E p [ z ] . Then:
(i) e − r ≤ q ( a ) / p ( a ) ≤ e r e^{-r} \le q(a)/p(a) \le e^r e − r ≤ q ( a ) / p ( a ) ≤ e r for every a a a in the support of p p p ;
(ii) e − r Var p ( w ) ≤ Var q ( w ) ≤ e r Var p ( w ) e^{-r}\operatorname{Var}_p(w) \le\operatorname{Var}_q(w) \le e^r\operatorname{Var}_p(w) e − r Var p ( w ) ≤ Var q ( w ) ≤ e r Var p ( w ) for every w ∈ R n w \in\mathbb{R}^n w ∈ R n ;
(iii) ∥ E p [ z ] − p ∥ 1 ≤ e r / 2 ∥ z ∥ p \lVert\mathbb{E}_p[z]-p\rVert_1 \le e^{r/2}\lVert z\rVert_p ∥ E p [ z ] − p ∥ 1 ≤ e r /2 ∥ z ∥ p ;
(iv) ∥ E p [ z ] − E p [ w ] ∥ 1 ≤ ∥ z − w ∥ ∞ \lVert\mathbb{E}_p[z]-\mathbb{E}_p[w]\rVert_1 \le\lVert z-w\rVert_\infty ∥ E p [ z ] − E p [ w ] ∥ 1 ≤ ∥ z − w ∥ ∞ for every w ∈ R n w \in\mathbb{R}^n w ∈ R n ;
(v) 1 2 e − r Var p ( z ) ≤ KL ( p ∥ E p [ z ] ) ≤ 1 2 e r Var p ( z ) \frac{1}{2}e^{-r}\operatorname{Var}_p(z) \le\operatorname{KL}(p\mathbin\Vert\mathbb{E}_p[z]) \le\frac{1}{2}e^r\operatorname{Var}_p(z) 2 1 e − r Var p ( z ) ≤ KL ( p ∥ E p [ z ]) ≤ 2 1 e r Var p ( z ) .
Proof. (i): the normalizer ∑ c p ( c ) e z ( c ) \sum_c p(c)e^{z(c)} ∑ c p ( c ) e z ( c ) lies between e min z e^{\min z} e m i n z and e max z e^{\max z} e m a x z , and q ( a ) / p ( a ) q(a)/p(a) q ( a ) / p ( a ) is e z ( a ) e^{z(a)} e z ( a ) divided by it. (ii): by the variational formula (66) , if q ≤ Γ p q \le\Gamma p q ≤ Γ p coordinatewise then Var q ( w ) ≤ ∑ a q ( a ) ( w ( a ) − ⟨ p , w ⟩ ) 2 ≤ Γ Var p ( w ) \operatorname{Var}_q(w) \le\sum_a q(a)(w(a)-\langle p,w\rangle)^2 \le\Gamma\operatorname{Var}_p(w) Var q ( w ) ≤ ∑ a q ( a ) ( w ( a ) − ⟨ p , w ⟩ ) 2 ≤ Γ Var p ( w ) ; apply this with Γ = e r \Gamma=e^r Γ = e r and symmetrically. (iii): let p s = E p [ s z ] p_s=\mathbb{E}_p[sz] p s = E p [ sz ] for 0 ≤ s ≤ 1 0\le s\le1 0 ≤ s ≤ 1 . Differentiating, d d s p s ( a ) = p s ( a ) ( z ( a ) − ⟨ p s , z ⟩ ) \frac{d}{ds}p_s(a)=p_s(a)(z(a)-\langle p_s,z\rangle) d s d p s ( a ) = p s ( a ) ( z ( a ) − ⟨ p s , z ⟩) , so by the Cauchy–Schwarz inequality ∥ d d s p s ∥ 1 ≤ ∑ a p s ( a ) ∣ z ( a ) − ⟨ p s , z ⟩ ∣ ≤ ∥ z ∥ p s ≤ e r / 2 ∥ z ∥ p \left\lVert\frac{d}{ds}p_s\right\rVert_1\le\sum_a p_s(a)|z(a)-\langle p_s,z\rangle|\le\lVert z\rVert_{p_s}\le e^{r/2}\lVert z\rVert_p d s d p s 1 ≤ ∑ a p s ( a ) ∣ z ( a ) − ⟨ p s , z ⟩ ∣ ≤ ∥ z ∥ p s ≤ e r /2 ∥ z ∥ p , using (ii) for p s = E p [ s z ] p_s=\mathbb{E}_p[sz] p s = E p [ sz ] with osc ( s z ) ≤ r \operatorname{osc}(sz)\le r osc ( sz ) ≤ r ; integrate over s ∈ [ 0 , 1 ] s\in[0,1] s ∈ [ 0 , 1 ] . (iv): apply the same computation along the segment w + s ( z − w ) w+s(z-w) w + s ( z − w ) , 0 ≤ s ≤ 1 0\le s\le1 0 ≤ s ≤ 1 : the derivative of E p [ w + s ( z − w ) ] \mathbb{E}_p[w+s(z-w)] E p [ w + s ( z − w )] has ℓ 1 \ell_1 ℓ 1 norm at most ∥ z − w ∥ E p [ w + s ( z − w ) ] ≤ ∥ z − w ∥ ∞ \lVert z-w\rVert_{\mathbb{E}_p[w+s(z-w)]}\le\lVert z-w\rVert_\infty ∥ z − w ∥ E p [ w + s ( z − w )] ≤ ∥ z − w ∥ ∞ . (v): let ϕ ( s ) = log ∑ a p ( a ) e s z \phi(s)=\log\sum_a p(a)e^{sz} ϕ ( s ) = log ∑ a p ( a ) e sz . Then ϕ ′ ( s ) = ⟨ p s , z ⟩ \phi'(s)=\langle p_s,z\rangle ϕ ′ ( s ) = ⟨ p s , z ⟩ , ϕ ′ ′ ( s ) = Var p s ( z ) \phi''(s)=\operatorname{Var}_{p_s}(z) ϕ ′′ ( s ) = Var p s ( z ) , and a direct computation gives KL ( p ∥ E p [ z ] ) = ϕ ( 1 ) − ϕ ( 0 ) − ϕ ′ ( 0 ) = ∫ 0 1 ( 1 − s ) Var p s ( z ) d s \operatorname{KL}(p\mathbin\Vert\mathbb{E}_p[z])=\phi(1)-\phi(0)-\phi'(0)=\int_0^1(1-s)\operatorname{Var}_{p_s}(z)\,ds KL ( p ∥ E p [ z ]) = ϕ ( 1 ) − ϕ ( 0 ) − ϕ ′ ( 0 ) = ∫ 0 1 ( 1 − s ) Var p s ( z ) d s by Taylor’s formula with integral remainder. Apply (ii) to p s p_s p s and use ∫ 0 1 ( 1 − s ) d s = 1 2 \int_0^1(1-s)\,ds=\frac12 ∫ 0 1 ( 1 − s ) d s = 2 1 .
:::
Proof of Lemma 3.1. Throughout the proof we drop the player index and write ℓ t = ℓ i t \ell^t=\ell_i^t ℓ t = ℓ i t and x t = x i t x^t=x_i^t x t = x i t . Let S t = ∑ s ≤ t ℓ s S^t=\sum_{s\le t}\ell^s S t = ∑ s ≤ t ℓ s , so that x t = E u [ − η ( S t − 1 + ℓ t − 1 ) ] x^t=\mathbb{E}_u[-\eta(S^{t-1}+\ell^{t-1})] x t = E u [ − η ( S t − 1 + ℓ t − 1 )] with u u u the uniform distribution, and let Φ t = log ( 1 n ∑ a e − η S t ( a ) ) \Phi_t=\log\left(\frac{1}{n}\sum_a e^{-\eta S^t(a)}\right) Φ t = log ( n 1 ∑ a e − η S t ( a ) ) , so that Φ 0 = 0 \Phi_0=0 Φ 0 = 0 . Write Z t = ∑ a e − η ( S t − 1 ( a ) + ℓ t − 1 ( a ) ) Z_t=\sum_a e^{-\eta(S^{t-1}(a)+\ell^{t-1}(a))} Z t = ∑ a e − η ( S t − 1 ( a ) + ℓ t − 1 ( a )) , so that x t ( a ) = e − η ( S t − 1 ( a ) + ℓ t − 1 ( a ) ) / Z t x^t(a)=e^{-\eta(S^{t-1}(a)+\ell^{t-1}(a))}/Z_t x t ( a ) = e − η ( S t − 1 ( a ) + ℓ t − 1 ( a )) / Z t . Since S t = ( S t − 1 + ℓ t − 1 ) + ( ℓ t − ℓ t − 1 ) S^t=(S^{t-1}+\ell^{t-1})+(\ell^t-\ell^{t-1}) S t = ( S t − 1 + ℓ t − 1 ) + ( ℓ t − ℓ t − 1 ) and S t − 1 = ( S t − 1 + ℓ t − 1 ) − ℓ t − 1 S^{t-1}=(S^{t-1}+\ell^{t-1})-\ell^{t-1} S t − 1 = ( S t − 1 + ℓ t − 1 ) − ℓ t − 1 ,
e Φ t = Z t n E x t e − η ( ℓ t − ℓ t − 1 ) , e Φ t − 1 = Z t n E x t e η ℓ t − 1 . e^{\Phi_t}=\frac{Z_t}{n}\mathbb{E}_{x^t}e^{-\eta(\ell^t-\ell^{t-1})},\qquad e^{\Phi_{t-1}}=\frac{Z_t}{n}\mathbb{E}_{x^t}e^{\eta\ell^{t-1}}. e Φ t = n Z t E x t e − η ( ℓ t − ℓ t − 1 ) , e Φ t − 1 = n Z t E x t e η ℓ t − 1 .
and therefore Φ t − Φ t − 1 = log E x t e − η ( ℓ t − ℓ t − 1 ) − log E x t e η ℓ t − 1 \Phi_t-\Phi_{t-1}=\log\mathbb{E}_{x^t}e^{-\eta(\ell^t-\ell^{t-1})}-\log\mathbb{E}_{x^t}e^{\eta\ell^{t-1}} Φ t − Φ t − 1 = log E x t e − η ( ℓ t − ℓ t − 1 ) − log E x t e η ℓ t − 1 . For a probability vector p p p and a vector v v v let K p ( v ) = log E p e v − E p v K_p(v)=\log\mathbb{E}_p e^v-\mathbb{E}_p v K p ( v ) = log E p e v − E p v be the cumulant generating function minus its linear term; a direct computation shows K p ( v ) = KL ( p ∥ E p [ v ] ) K_p(v)=\operatorname{KL}(p\mathbin\Vert\mathbb{E}_p[v]) K p ( v ) = KL ( p ∥ E p [ v ]) , so by Lemma A.2 (v), 1 2 e − osc ( v ) Var p ( v ) ≤ K p ( v ) ≤ 1 2 e osc ( v ) Var p ( v ) \frac12e^{-\operatorname{osc}(v)}\operatorname{Var}_p(v)\le K_p(v)\le\frac12e^{\operatorname{osc}(v)}\operatorname{Var}_p(v) 2 1 e − osc ( v ) Var p ( v ) ≤ K p ( v ) ≤ 2 1 e osc ( v ) Var p ( v ) . Separating the linear terms,
η ⟨ x t , ℓ t ⟩ + Φ t − Φ t − 1 = K x t ( − η ( ℓ t − ℓ t − 1 ) ) − K x t ( η ℓ t − 1 ) . \eta\langle x^t,\ell^t\rangle+\Phi_t-\Phi_{t-1}=K_{x^t}(-\eta(\ell^t-\ell^{t-1}))-K_{x^t}(\eta\ell^{t-1}). η ⟨ x t , ℓ t ⟩ + Φ t − Φ t − 1 = K x t ( − η ( ℓ t − ℓ t − 1 )) − K x t ( η ℓ t − 1 ) .
The two arguments have oscillation at most 2 η 2\eta 2 η and η \eta η , respectively. Summing over t ≤ T t\le T t ≤ T , using Φ 0 = 0 \Phi_0=0 Φ 0 = 0 and Φ T ≥ log ( 1 n e − η min a S T ( a ) ) = − η min a S T ( a ) − log n \Phi_T\ge\log\left(\frac1n e^{-\eta\min_a S^T(a)}\right)=-\eta\min_a S^T(a)-\log n Φ T ≥ log ( n 1 e − η m i n a S T ( a ) ) = − η min a S T ( a ) − log n , we obtain
η ( ∑ t ≤ T ⟨ x t , ℓ t ⟩ − min a S T ( a ) ) − log n ≤ η 2 e 2 η 2 D i ( T ) − η 2 e − η 2 V i ( T ) . \eta\left(\sum_{t\le T}\langle x^t,\ell^t\rangle-\min_a S^T(a)\right)-\log n\le\frac{\eta^2e^{2\eta}}{2}D_i(T)-\frac{\eta^2e^{-\eta}}{2}V_i(T). η ( t ≤ T ∑ ⟨ x t , ℓ t ⟩ − a min S T ( a ) ) − log n ≤ 2 η 2 e 2 η D i ( T ) − 2 η 2 e − η V i ( T ) .
and dividing by η \eta η gives, for every η > 0 \eta>0 η > 0 ,
Reg i ( T ) ≤ log n η + η 2 ( e 2 η D i ( T ) − e − η V i ( T ) ) . \operatorname{Reg}_i(T)\le\frac{\log n}{\eta}+\frac{\eta}{2}\left(e^{2\eta}D_i(T)-e^{-\eta}V_i(T)\right). Reg i ( T ) ≤ η log n + 2 η ( e 2 η D i ( T ) − e − η V i ( T ) ) .
For η ≤ 1 / 8 \eta\le1/8 η ≤ 1/8 we have 1 2 e 2 η ≤ 2 3 \frac12e^{2\eta}\le\frac23 2 1 e 2 η ≤ 3 2 and 1 2 e − η ≥ 1 3 \frac12e^{-\eta}\ge\frac13 2 1 e − η ≥ 3 1 , which gives (5) .
Proposition A.3 (Regret against a constant loss vector). Let n ≥ 2 n \ge2 n ≥ 2 and η > 0 \eta> 0 η > 0 , and let a player with n n n actions run Optimistic Hedge against the constant sequence ℓ t = ℓ \ell^t = \ell ℓ t = ℓ for t ≥ 1 t \ge1 t ≥ 1 , where ℓ ( a ∗ ) = 0 \ell(a^*) = 0 ℓ ( a ∗ ) = 0 for one action a ∗ a^* a ∗ and ℓ ( a ) = 1 \ell(a) = 1 ℓ ( a ) = 1 for all a ≠ a ∗ a \ne a^* a = a ∗ . Then Reg ( T ) \operatorname{Reg}(T) Reg ( T ) is nondecreasing in T T T and
lim T → ∞ Reg ( T ) ≥ log n η − 1. \lim_{T \to\infty} \operatorname{Reg}(T) \ge\frac{\log n}{\eta} - 1. T → ∞ lim Reg ( T ) ≥ η log n − 1. In particular, if this player is embedded as player 3 of a three-player game whose loss table does not depend on the actions of players 1 and 2, then sup T sup G ∈ G ( 3 , n ) R ( G , T , η ) ≥ log n / η − 1 \sup_T \sup_{G \in\mathcal{G}(3,n)} R(G,T,\eta) \ge\log n/\eta- 1 sup T sup G ∈ G ( 3 , n ) R ( G , T , η ) ≥ log n / η − 1 .
Proof. With ℓ 0 = 0 \ell^0 = 0 ℓ 0 = 0 , the first update multiplies the weights by e − 2 η ℓ e^{-2\eta\ell} e − 2 η ℓ and every later update by e − η ℓ e^{-\eta\ell} e − η ℓ , so x t ( a ) ∝ e − η t ℓ ( a ) x^t(a) \propto e^{-\eta t\ell(a)} x t ( a ) ∝ e − η t ℓ ( a ) for all t ≥ 2 t \ge2 t ≥ 2 , and x 1 x^1 x 1 is uniform. The total mass on the suboptimal actions in round t t t is therefore m t = ( n − 1 ) e − η t / ( 1 + ( n − 1 ) e − η t ) m_t = (n - 1)e^{-\eta t}/(1 + (n - 1)e^{-\eta t}) m t = ( n − 1 ) e − η t / ( 1 + ( n − 1 ) e − η t ) for t ≥ 2 t \ge2 t ≥ 2 and m 1 = ( n − 1 ) / n m_1 = (n - 1)/n m 1 = ( n − 1 ) / n , and Reg ( T ) = ∑ t ≤ T m t \operatorname{Reg}(T) = \sum_{t \le T} m_t Reg ( T ) = ∑ t ≤ T m t because the best action has loss zero. Each m t m_t m t is positive, so Reg ( T ) \operatorname{Reg}(T) Reg ( T ) is nondecreasing, and since t ↦ m t t \mapsto m_t t ↦ m t is decreasing,
∑ t ≥ 1 m t ≥ ∫ 1 ∞ ( n − 1 ) e − η u 1 + ( n − 1 ) e − η u d u = 1 η log ( 1 + ( n − 1 ) e − η ) ≥ 1 η log ( n e − η ) = log n η − 1 , \sum_{t \ge1} m_t \ge\int_1^\infty\frac{(n-1)e^{-\eta u}}{1+(n-1)e^{-\eta u}}\,du = \frac{1}{\eta}\log(1+(n-1)e^{-\eta}) \ge\frac{1}{\eta}\log(ne^{-\eta}) = \frac{\log n}{\eta}-1, t ≥ 1 ∑ m t ≥ ∫ 1 ∞ 1 + ( n − 1 ) e − η u ( n − 1 ) e − η u d u = η 1 log ( 1 + ( n − 1 ) e − η ) ≥ η 1 log ( n e − η ) = η log n − 1 ,
where the last inequality uses 1 + ( n − 1 ) e − η ≥ n e − η 1+(n-1)e^{-\eta} \ge ne^{-\eta} 1 + ( n − 1 ) e − η ≥ n e − η .
AI disclosure
The proofs originated from human insight and were developed through a combination of human and AI work: most of the proofs contain a combination of ideas from the authors and OpenAI’s ChatGPT. Anthropic’s Claude was also used to draft and edit parts of the text. The authors verified the correctness and originality of all content, including references, and take full responsibility for the content of the paper.