Introduction
Learning algorithms increasingly act on behalf of agents in repeated strategic interactions: they bid in ad auctions, set prices, and allocate computing resources. The party on the other side is often not another learner but a forward-looking optimizer that knows how the learner behaves and plans against it. The learner’s predictability then becomes a liability, which raises two basic questions [5 , 20 , 31 , 55 ] : How much can an optimizer extract from a learner? And which learning algorithms are non-manipulable, in the sense that their predictability gives the optimizer no advantage?
In standard (unbudgeted) repeated games, these questions have a clean answer. The natural benchmark is the optimizer’s Stackelberg Equilibrium value SE: the best payoff it can obtain by committing to a mixed strategy to which the learner best responds. Against any no-regret learner, the optimizer can secure (essentially) SE per round by committing to its Stackelberg strategy, and against common no-regret algorithms, such as multiplicative weights, it can, in some games, earn strictly more [20 , 31 ] . A learner is protected when it has low swap regret : low swap regret caps the optimizer at S E ⋅ T + o ( T ) \mathrm{SE} \cdot T + o(T) SE ⋅ T + o ( T ) [31 ] , and it is, in a precise sense, also necessary [5 , 55 ] . Since best responding to calibrated forecasts guarantees low swap regret [40 ] , a single principle—calibration —delivers both robustness to manipulation and good learning performance.
This theory treats the rounds as decoupled. Yet in many of the interactions that motivate it—such as autobidding in online advertising [2 , 10 , 27 , 28 ] , but also cloud resource allocation and repeated procurement—agents operate under long-term budgets that couple their decisions across rounds. This motivates the study of repeated budgeted games . For T T T rounds, an optimizer with n O n_{\mathrm{O}} n O actions and a learner with n L n_{\mathrm{L}} n L actions play a stage game with utility matrices U O , U L ∈ [ 0 , 1 ] n O × n L U_{\mathrm{O}}, U_{\mathrm{L}} \in[0,1]^{n_{\mathrm{O}}\times n_{\mathrm{L}}} U O , U L ∈ [ 0 , 1 ] n O × n L and cost matrices C O , C L ∈ [ 0 , 1 ] n O × n L C_{\mathrm{O}}, C_{\mathrm{L}} \in[0,1]^{n_{\mathrm{O}}\times n_{\mathrm{L}}} C O , C L ∈ [ 0 , 1 ] n O × n L : in round t t t the optimizer plays i t ∈ [ n O ] i_t \in[n_{\mathrm{O}}] i t ∈ [ n O ] , the learner plays j t ∈ [ n L ] j_t \in[n_{\mathrm{L}}] j t ∈ [ n L ] , and each player wants to maximize its total utility while keeping its total cost within its budget—ρ O T \rho_{\mathrm{O}}T ρ O T for the optimizer and ρ L T \rho_{\mathrm{L}}T ρ L T for the learner. Budgets change both sides of the interaction:
The optimizer may want to switch strategies. A budgeted optimizer can do strictly better than commit to a single mixed strategy: because its value need not be concave in its spending, it may profit from time-multiplexing —for instance, by spending aggressively for q T qT q T rounds and frugally for the remaining ( 1 − q ) T (1-q)T ( 1 − q ) T rounds. The resulting benchmark is the Budgeted Stackelberg Equilibrium (BSE) (defined in our recent work [36 ] ): the optimizer commits to (at most) two phases, each met by the learner’s budgeted best response. When the optimizer is unbudgeted, one phase suffices, and the BSE value is simply the value of the stage game in which the follower best responds subject to its budget.
The learner can no longer guarantee low regret. Against an adversarial sequence of actions, no budgeted learner can guarantee sublinear regret [49 , 54 ] , let alone low swap regret, due to facing the spend or save dilemma. So the swap-regret characterization above has no direct budgeted analog.
What, then, should a budgeted learner aim for? We ask for two properties:
Non-manipulability : no adaptive optimizer earns more than B S E ⋅ T + o ( T ) \mathrm{BSE} \cdot T + o(T) BSE ⋅ T + o ( T ) in expectation.3
Low interval stochastic regret 4 : if the optimizer’s play is piecewise stationary—i.i.d. from a fixed mixed strategy within each of a few intervals, unknown to the learner—then on every interval, the learner’s utility is within o ( T ) o(T) o ( T ) of the utility from the best budget-feasible response to the corresponding mixed strategy.
Since adversarial no-regret is out of reach, the second property is the natural learning goal: it is the standard benchmark for learning under budgets in stochastic environments [3 , 7 ] , applied to every stationary stretch of play and it also provides a sanity check, that the learner is not just trying to harm the optimizer. Without budgets, a single property—low swap regret—yields both non-manipulability and low regret (Table 1 ). With budgets, the calibration that prevents manipulation no longer yields low regret (Challenge 3 in § 1.2 ), so each property needs its own tool. The central question of this paper is:
Table 1. What a learner can guarantee, without and with budgets.
Is there an efficient learning algorithm that, in every repeated budgeted game, is non-manipulable and has low interval stochastic regret?
Our results. We answer this question in the affirmative. We assume full-information feedback about the learner’s own payoffs and costs: after round, the learner observes the utility and cost of each of its actions, i.e., the vector ( U L [ i t , : ] , C L [ i t , : ] ) ∈ [ 0 , 1 ] 2 n L (U_L[i_t,:], C_L[i_t,:]) \in[0,1]^{2n_L} ( U L [ i t , : ] , C L [ i t , : ]) ∈ [ 0 , 1 ] 2 n L . It observes neither i t i_t i t nor the optimizer’s payoffs, and it does not know in advance which utility-cost vectors can arise. Each player has a null action that incurs no cost.
Theorem 1.1 (Informal version of Theorems 6.7 and 7.2 ). There is a learning algorithm that never exceeds the learner’s budget, runs in poly ( T , n L ) \operatorname{poly}(T,n_L) poly ( T , n L ) time per round, and in every repeated budgeted game simultaneously guarantees:
Non-manipulability: against every adaptive optimizer,
E [ ∑ t = 1 T U O [ i t , j t ] ] ≤ B S E ⋅ T + O ( T ) , \mathbb{E}\left[\sum_{t=1}^{T} U_O[i_t,j_t]\right] \le\mathrm{BSE}\cdot T + O\left(\sqrt{T}\right), E [ t = 1 ∑ T U O [ i t , j t ] ] ≤ BSE ⋅ T + O ( T ) , *where O ( ⋅ ) O(\cdot) O ( ⋅ ) hides factors polynomial in n O n_O n O , n L n_L n L , and 1 / ρ L 1/\rho_L 1/ ρ L .5 If the optimizer is also budgeted, the same holds relative to its two-phase BSE, with an additional factor of 1 / ρ O 1/\rho_O 1/ ρ O in the error term.*
Low interval stochastic regret: if [ T ] [T] [ T ] is partitioned into S S S intervals, unknown to the learner, such that within each interval I \mathcal{I} I the optimizer draws its actions i.i.d. from a mixed strategy x I x_{\mathcal{I}} x I , then on every interval,
E [ ∑ t ∈ I U L [ i t , j t ] ] ≥ ∣ I ∣ ⋅ x I T U L y I ⋆ − O ( S n L T ρ L ) , \mathbb{E}\left[\sum_{t\in\mathcal{I}} U_L[i_t,j_t]\right] \ge|\mathcal{I}|\cdot x_{\mathcal{I}}^{\mathsf{T}}U_Ly_{\mathcal{I}}^\star- O\left(\frac{\sqrt{S n_L T}}{\rho_L}\right), E [ t ∈ I ∑ U L [ i t , j t ] ] ≥ ∣ I ∣ ⋅ x I T U L y I ⋆ − O ( ρ L S n L T ) , where y I ⋆ y_{\mathcal{I}}^\star y I ⋆ is a best response to x I x_{\mathcal{I}} x I subject to the learner’s per-round budget.
Three features of Theorem 1.1 are worth highlighting. First, the regret bound is independent of the number of the optimizer’s actions: it holds even if no utility-cost vector is ever observed twice. Second, the first guarantee is essentially tight: against our learner, an optimizer that plays a fixed mixed strategy in each of (at most) two phases secures the weak BSE value (Theorem 7.4 ). Thus, up to the usual gap between weak and strong Stackelberg values, the BSE is the value of the interaction. Third, the algorithm needs no prior knowledge of the game.
Comparison With Budgeted Auctions
Budgeted auctions: simplicity from structure. In earlier work [36 ] , we studied these questions for repeated budgeted auctions in a Bayesian setting, where in each round the two bidders’ values are drawn from a joint distribution. Here, the algorithm used in practice is remarkably simple. In a second-price auction, the learner’s best response to any fixed competition is to bid its value scaled by a single pacing multiplier , so the learner only needs to track one number, and large advertising platforms adjust it with simple feedback (PID-style) controllers: raise the multiplier after underspending, lower it after overspending [27 , 43 , 60 ] . [36 ] shows that the simplest such controller—a deterministic, one-line proportional update—is non-manipulable: under a mild assumption on the value distribution (some such assumption is necessary), the optimizer earns at most BSE · T T T + O ( T 2 / 3 ) O(T^{2/3}) O ( T 2/3 ) .
General games: universality without structure. Given the previous results, a natural question arises. Is robustness to manipulation an artifact of the special structure of auctions, or can it be achieved in every budgeted game? In a general game, no scalar parameterizes the learner’s best responses: a budget-constrained best response is the solution of a linear program—often a strict mixture of two actions—and it jumps discontinuously as the environment changes. The learner does not know the optimizer’s payoffs. And there is no Bayesian structure to lean on: everything the learner faces is chosen adaptively by the optimizer. In fact, no deterministic learner is non-manipulable in every game: in matching pennies (with zero costs), an optimizer that simulates the learner always mismatches it, earning 1 per round, twice its Stackelberg value of 1 / 2 1/2 1/2 . In auctions, structure makes determinism harmless; general games have no such structure.
Why both settings. Our work on budgeted auctions [36 ] and this paper answer on general games complementary questions—is the algorithm used in practice robust? and is robustness achievable in every budgeted game? —and neither result implies the other (Table 2 in Appendix § 9 compares them in detail). The controller of [36 ] has no counterpart in general games, where no deterministic learner is robust. The same pattern is familiar without budgets: against a revenue-maximizing seller, simple no-regret bidding algorithms (in the right parameterization) are already robust [Cai-26; KSS24; Zha26], whereas in general games, robustness requires low swap regret [55 ] . As often in algorithm design, the special case admits a strikingly simple algorithm, while the general case requires more complex algorithms.
Table 2. This paper and [36 ] at a glance.
Why the techniques differ. Beyond the BSE benchmark and Lagrangian duality, the two analyses share very little. In [36 ] , the learner is a fixed, deterministic dynamical system with a one-dimensional state, and the proof bounds the value of the optimizer’s control problem over this state with a potential function built from Lagrangian duality. Here, there is no small state to control, and the learner must randomize: our learner best responds to forecasts that are calibrated in two new senses—conditionally on its own mixed response (which prevents manipulation), and weighted by its dual variable (which yields low regret)—and computes them by simulating a no-regret dynamic (see challenges and techniques in § 1.2 and § 1.3 ).
Why General Budgeted Games Are Hard: Three Challenges
Three challenges rule out natural approaches and shape our solution.
1. Interval no-regret does not prevent manipulation. Since adversarial regret is impossible, one might combine steady spending of budget with an online learning algorithm that has low regret on every interval. This fails even without budgets: projected online gradient ascent has O ( T ) O(\sqrt{T}) O ( T ) regret on every interval against every sequence, yet in a simple 3 × 3 3 \times3 3 × 3 game, an optimizer that switches actions whenever the learner’s mixed strategy reaches a corner of the simplex earns T / 2 − o ( T ) T/2 - o(T) T /2 − o ( T ) , compared with S E ⋅ T = T / 3 \mathrm{SE} \cdot T = T/3 SE ⋅ T = T /3 (see [36 ] , also included for completeness in Appendix § 11 ). Robustness in general games requires guarantees conditioned on the learner’s own responses, as in swap regret and calibration [55 ] .
2. Budgeted best responses are mixed and discontinuous. Following Foster and Vohra [40 ] , the learner could forecast the utility-cost vector that the optimizer will induce, and best respond to the forecast. Merely unbiased forecasts are not enough: since best responses are discontinuous, forecasts with zero cumulative error can lie on opposite sides of a discontinuity and induce opposite actions, which the optimizer can exploit (see Appendix § 12 for an example). Full calibration, which conditions on the forecast itself, avoids this but needs exponentially (in the dimension of forecasts) many buckets. Without budgets, the efficient remedy is to condition only on the learner’s response, and since best responses can be taken to be pure, n L n_L n L “buckets” suffice. With a budget, however, the best response solves a linear program with two constraints (the simplex and the budget), so when the budget binds, it is generically a strict mixture of two actions ( j 1 , j 2 ) (j_1, j_2) ( j 1 , j 2 ) , with a mixing weight q ∈ [ 0 , 1 ] q \in[0, 1] q ∈ [ 0 , 1 ] that varies continuously . Conditioning on the exact response would require infinitely many buckets, while coarser conditioning—ignoring the weight, or tracking the two actions separately—can be exploited; the latter because several optimizer actions may induce the same feedback for the learner but have different value for the optimizer (see Appendix § 13 for an example).
3. Non-manipulability no longer implies low regret. Unlike in unbudgeted games, response calibration does not yield low regret: there are forecasters with zero response-conditioned calibration error—so the learner cannot be manipulated—under which the learner suffers Ω ( T ) \Omega(T) Ω ( T ) regret against an i.i.d. optimizer (Appendix § 14 ). The reason is that two different forecasts can induce the same conservative action, so their errors cancel, and the learner never discovers an action that is affordable only on average.
Our Techniques
Forecast, then best respond—safely. Our learner keeps a small reserve: in any round t t t in which its past spending exceeds t P − 1 tP - 1 tP − 1 , it plays its null action, so it never exceeds its budget. In every other (active ) round, it draws a forecast f f f —a mixture of the utility-cost vectors observed so far—from a distribution F t F_t F t of such forecasts, and plays a budget-constrained best response ψ ( f ) \psi(f) ψ ( f ) to the forecast, supported on (at most) two actions J ⃗ ( f ) = ( j 1 , j 2 ) \vec{J}(f) = (j_1, j_2) J ( f ) = ( j 1 , j 2 ) with probability Q ( f ) Q(f) Q ( f ) on j 1 j_1 j 1 . Everything then hinges on the quality of the forecasts, which we measure with two new calibration errors.
Response calibration in a function space. To bound the optimizer’s gain over the BSE, forecasts must be calibrated conditionally on the learner’s response ( J ⃗ ( f ) , Q ( f ) ) (\vec{J}(f), Q(f)) ( J ( f ) , Q ( f )) . Instead of discretizing the weight Q ( f ) Q(f) Q ( f ) , we represent it by the step function q ↦ 1 q ≤ Q ( f ) q \mapsto\mathbf{1}_{q \le Q(f)} q ↦ 1 q ≤ Q ( f ) in the Hilbert space L 2 ( [ − 1 , 1 ] ) L^2([-1, 1]) L 2 ([ − 1 , 1 ]) . The response-calibration error R t R_t R t has one coordinate for each outcome k k k (a possible utility-cost vector) and each pair ( j 1 , j 2 ) (j_1, j_2) ( j 1 , j 2 ) ; in each round, coordinate ( k , J ⃗ ( f ) ) (k, \vec{J}(f)) ( k , J ( f )) accumulates the forecast error 1 k t = k − f ( k ) \mathbf{1}_{k_t=k} - f(k) 1 k t = k − f ( k ) times this step function. Two facts make this representation work. First, step functions have the closed-form inner product ⟨ 1 ≤ q , 1 ≤ q ′ ⟩ = 1 + min { q , q ′ } \langle\mathbf{1}_{\le q}, \mathbf{1}_{\le q'} \rangle= 1 + \min\{q, q'\} ⟨ 1 ≤ q , 1 ≤ q ′ ⟩ = 1 + min { q , q ′ } , so R t R_t R t can be stored and manipulated efficiently. Second, the optimizer’s best payoff against the mixed response ( j 1 , j 2 , q ) (j_1, j_2, q) ( j 1 , j 2 , q ) , among its actions that induce outcome k k k , is a 1-Lipschitz piecewise-linear function g ( q ) g(q) g ( q ) , which can be written as the inner product ⟨ g ′ , 1 ≤ q ⟩ \langle g',1_{\leq q}\rangle ⟨ g ′ , 1 ≤ q ⟩ with its derivative (working on [ − 1 , 1 ] [-1,1] [ − 1 , 1 ] rather than [ 0 , 1 ] [0,1] [ 0 , 1 ] to accommodate affine functions). By linearity and the Cauchy–Schwarz inequality, the optimizer’s total gain over the BSE in active rounds is at most ⟨ g ′ , R T ⟩ ≤ ∥ g ′ ∥ ∥ R T ∥ \langle g',R_T\rangle\leq\lVert g'\rVert\lVert R_T\rVert ⟨ g ′ , R T ⟩ ≤ ∥ g ′ ∥ ∥ R T ∥ , simultaneously for all optimizer payoff matrices (which the learner does not know).
Lagrangian calibration. To obtain low stochastic regret and keep spending on track, we maintain a second error L t L_t L t . The best response y ( f ) y(f) y ( f ) maximizes the Lagrangian u ( f ) ⊤ y + λ ( f ) ( ρ L − c ( f ) ⊤ y ) u(f)^\top y+\lambda(f)(\rho_L-c(f)^\top y) u ( f ) ⊤ y + λ ( f ) ( ρ L − c ( f ) ⊤ y ) , where u ( f ) u(f) u ( f ) and c ( f ) c(f) c ( f ) are the forecast utility and cost vectors and λ ( f ) ∈ [ 0 , 1 / ρ L ] \lambda(f)\in[0,1/\rho_L] λ ( f ) ∈ [ 0 , 1/ ρ L ] is the optimal dual multiplier of the learner’s budget constraint. When comparing the learner with a budget-feasible comparator y ⋆ y^\star y ⋆ , the troublesome term is the cost misprediction weighted by the dual variable , λ ( f ) ( c t − c ( f ) ) ⊤ y ⋆ \lambda(f)(c_t-c(f))^\top y^\star λ ( f ) ( c t − c ( f ) ) ⊤ y ⋆ , where c t c_t c t is the realized cost vector: a forecast that overestimates the cost of y ⋆ y^\star y ⋆ makes it look unaffordable exactly when the budget is tight, as in Challenge 3 in § 1.2 (see example in Appendix § 14 ). The Lagrangian-calibration error L t L_t L t tracks this dual-weighted cost error, together with the utility error and the gaps between the learner’s realized and forecast utility and cost. A small ∥ L t ∥ \lVert L_t\rVert ∥ L t ∥ bounds both the number of non-active rounds and the learner’s regret on every stationary interval.
Efficient forecasting by simulating the adversary. We combine the two errors into Z t = ( R t , L t ) Z_t=(R_t,L_t) Z t = ( R t , L t ) and keep the potential ∥ Z t ∥ 2 \lVert Z_t\rVert^2 ∥ Z t ∥ 2 growing by O ( 1 ) O(1) O ( 1 ) per round, in the spirit of Blackwell approachability and forecast hedging [17 , 39 , 40 ] . This reduces each round to a zero-sum game: the forecaster picks a forecast f f f , the environment picks an outcome k k k , and the payoff is the increase in the potential. If the environment commits to a mixed strategy w w w , the forecaster can simply predict that exact environment f = w f=w f = w and pay exactly zero. However, the payoff is discontinuous in f f f (both y ( f ) y(f) y ( f ) and λ ( f ) \lambda(f) λ ( f ) jump), so minimax theorems do not apply directly; an equilibrium forecast distribution, even when it exists, need not have a succinct representation; and discretizing the simplex requires exponentially many points. We sidestep these issues constructively: we simulate projected online gradient ascent for the environment player, and the forecaster answers each iterate f ( m ) f^{(m)} f ( m ) with the forecast f ( m ) f^{(m)} f ( m ) itself. The uniform distribution over the iterates is our forecast distribution F t F_t F t , and the regret bound of gradient ascent shows that it loses at most O ( 1 ) O(1) O ( 1 ) against every previously observed outcome. This yields ∥ Z T ∥ = O ( T + n 0 ) \lVert Z_T\rVert=O(\sqrt{T}+n_0) ∥ Z T ∥ = O ( T + n 0 ) against any optimizer—each of the at most n 0 n_0 n 0 distinct utility-cost vectors may cost O ( 1 ) O(1) O ( 1 ) when it first appears—in time poly( T , n L ) (T,n_L) ( T , n L ) . Against a piecewise-stationary optimizer, a finer argument that compares the forecasts with the empirical distribution of outcomes within each interval gives E [ max t ∥ Z t ∥ ] = O ( S T ) \mathbb{E}[\max_t\lVert Z_t\rVert]=O(\sqrt{ST}) E [ max t ∥ Z t ∥] = O ( S T ) , independently of n 0 n_0 n 0 .
Related Work
Strategizing against learners. How a utility-maximizing optimizer should play against a learning algorithm, and which algorithms resist such manipulation, was first studied by [20 ] for selling to a no-regret buyer. The question has since been studied in normal-form games [6 , 21 , 31 ] , Bayesian games [55 ] , polytope games [5 ] , mechanism design and contracting with learning agents [23 , 44 ] , Stackelberg games against calibrated agents [45 ] , and repeated auctions [20 , 22 , 30 , 52 , 59 , 62 ] . The swap-regret characterization recalled above is due to [31 , 55 ] for normal-form games, and to [5 ] for polytope games (via profile swap regret). In auctions, recent work designs bidding algorithms that are both no-regret and robust to a manipulative seller [22 , 52 , 62 ] . All of these works consider rounds that are not linked by budgets.
Calibration, approachability, and swap regret. Foster and Vohra [40 ] showed that best responding to calibrated forecasts yields no internal regret, so that such dynamics converge to correlated equilibria; see also [17 , 18 , 46 ] . Since calibrating high-dimensional forecasts by discretization is exponentially costly, a recent line of work calibrates only what downstream decisions depend on: decision calibration tion [63 ] , response-based approachability [4 , 16 ] , pseudonorm approachability [29 ] , and forecasters that guarantee low regret or swap regret to downstream agents [48 , 51 , 56 , 57 , 58 ] . Our response-calibration error is in the same spirit, as it conditions on the learner’s own response, and in unbudgeted games it reduces to errors that guarantee low swap regret and look like the ones in previous works (Appendix § 15 ). The budget, however, forces three departures (see issues and techniques in § 1.2 and § 1.3 ): (i) responses are mixtures with a continuous weight, calibrated through step functions in L 2 ( [ − 1 , 1 ] ) L^2([-1,1]) L 2 ([ − 1 , 1 ]) ; (ii) response calibration no longer implies low regret, necessitating the dual-weighted Lagrangian calibration; and (iii) since the set of outcomes is not known in advance and randomized forecasts cannot be represented explicitly, we construct them by simulating online gradient ascent.
Online learning under budget constraints. Learning under budget constraints has been studied extensively in the frameworks of bandits with knapsacks and online allocation [3 , 7 , 13 , 15 , 19 , 24 , 25 , 37 , 49 , 50 ] . In stochastic environments, O ~ ( T ) \widetilde{O}(\sqrt{T}) O ( T ) regret against the budgeted best response is achievable [3 , 7 , 13 ] . In adversarial environments, the best reward in hindsight is not attainable [49 , 54 ] : [54 ] used calibrated forecasts to attain its convex hull, and later work obtained competitive-ratio guarantees [24 , 25 , 49 , 50 ] , best-of-both-worlds algorithms [13 , 15 , 24 ] , guarantees for approximately stationary environments [37 ] , and guarantees against weaker benchmarks [19 Luc+24] . In auctions, a large literature studies budget pacing: learning to bid under budget (and return-on-investment) constraints [1 , 9 , 10 , 12 , 14 , 33 , 34 , 61 ] , pacing equilibria [8 , 11 , 26 , 27 , 28 ] , and the welfare of pacing dynamics [38 , 41 Luc+24] ; see [2 ] for a survey. Our interval stochastic regret guarantee is the full-information analog of stochastic guarantees for bandits with knapsacks, in piecewise-stationary environments; what is new is achieving it together with non-manipulability against an arbitrary adaptive opponent.
Strategizing against budgeted learners. [35 ] study a proportional controller similar to [36 ] in repeated auctions where all values equal 1, and show that each bidder wins a budget-proportional share of the rounds, even against adversarial competitors. In [36 ] , we introduced the BSE and proved that the proportional controller is non-manipulable in repeated auctions with Bayesian values; see § 1.1 and Appendix § 9 for a comparison with this paper. [42 ] study an unconstrained seller learning a static posted price via binary search against a budget- and ROI-constrained buyer. Their focus is how the seller can guarantee low regret w.r.t. the Stackelberg value and not whether more revenue than that can be achieved.
Model and Definitions
Two players, called optimizer and learner, are playing a repeated budgeted game over T T T rounds. The optimizer has n O n_O n O actions and the learner has n L n_L n L actions. In each round t t t , the optimizer picks an action i t ∈ [ n O ] i_t \in[n_O] i t ∈ [ n O ] and the learner picks j t ∈ [ n L ] j_t \in[n_L] j t ∈ [ n L ] ; the outcome of the round is described by the matrices U O , U L , C O , C L ∈ [ 0 , 1 ] n O × n L U_O, U_L, C_O, C_L \in[0,1]^{n_O \times n_L} U O , U L , C O , C L ∈ [ 0 , 1 ] n O × n L . U O [ i t , j t ] U_O[i_t,j_t] U O [ i t , j t ] and C O [ i t , j t ] C_O[i_t,j_t] C O [ i t , j t ] are the optimizer’s utility and cost, respectively, and similarly for U L , C L U_L,C_L U L , C L for the learner. The goal of each player is to maximize their utility, while they also have a budget constraint: the total cost has to be at most ρ L T \rho_L T ρ L T and ρ O T \rho_O T ρ O T respectively, for some ρ L , ρ O ∈ ( 0 , 1 ] \rho_L,\rho_O \in(0,1] ρ L , ρ O ∈ ( 0 , 1 ] ; we assume ρ L \rho_L ρ L and ρ O \rho_O ρ O are both Ω ( 1 ) \Omega(1) Ω ( 1 ) w.r.t. T T T . We also assume that each player has an action (called the null action) that guarantees 0 cost no matter what the other player is doing. We assume throughout the paper that the learner receives full information feedback: for a chosen action i t i_t i t of the optimizer in round t t t , the learner picks action j t j_t j t and learns the utilities and costs U L [ i t , : ] U_L[i_t,:] U L [ i t , : ] and C L [ i t , : ] C_L[i_t,:] C L [ i t , : ] for each of their actions, not only the cost and utility of their chosen action j t j_t j t .
For most of the paper we will focus on the version of the problem where only the learner is budget constrained. Later we show in § 7 that our non-manipulability result for an unbudgeted optimizer translates to a similar result for a budgeted optimizer.
Budgeted Stackelberg Equilibrium. In [36 ] we introduced the Budgeted Stackelberg Equilibrium (BSE), that generalizes the concept of Stackelberg Equilibrium to budgeted games. We first define the best-response function of the learner, given the expected utility and cost vectors u = x ⊤ U L u = x^{\top}U_{\mathrm{L}} u = x ⊤ U L and c = x ⊤ C L c = x^{\top}C_{\mathrm{L}} c = x ⊤ C L induced by an optimizer’s mixed strategy x ∈ Δ n O x \in\Delta_{n_{\mathrm{O}}} x ∈ Δ n O :
BR L ( u , c ) = argmax y ∈ Δ n L u ⊤ y such that c ⊤ y ≤ ρ L . (1) \operatorname{BR}_{\mathrm{L}}(u,c) = \underset{y \in\Delta_{n_{\mathrm{L}}}}{\operatorname{argmax}}\ u^{\top}y \quad\text{such that} \quad c^{\top}y \le\rho_{\mathrm{L}}. \tag*{(1)} BR L ( u , c ) = y ∈ Δ n L argmax u ⊤ y such that c ⊤ y ≤ ρ L . (1)
For unbudgeted optimizer, this makes the BSE very similar to a Stackelberg equilibrium here equal to
BSE = max x ∈ Δ n O , y ∈ Δ n L x ⊤ U O y such that y ∈ BR L ( x ⊤ U L , x ⊤ C L ) . (2) \operatorname{BSE} = \underset{x \in\Delta_{n_{\mathrm{O}}},\,y \in\Delta_{n_{\mathrm{L}}}}{\max}\ x^{\top}U_{\mathrm{O}}y \quad\text{such that} \quad y \in\operatorname{BR}_{\mathrm{L}}(x^{\top}U_{\mathrm{L}},x^{\top}C_{\mathrm{L}}). \tag*{(2)} BSE = x ∈ Δ n O , y ∈ Δ n L max x ⊤ U O y such that y ∈ BR L ( x ⊤ U L , x ⊤ C L ) . (2)
Note that there is always a best response y y y that can be represented with support at most two actions, because it is a solution to a Linear Program with two constraints: the budget and the distribution constraints.
Learner’s Desiderata. Our goal in this paper is to design a learning process to achieve the following.
Non-manipulability. No matter what strategy the optimizer follows, the expected utility should be bounded by BSE ⋅ T + o ( T ) \operatorname{BSE} \cdot T + o(T) BSE ⋅ T + o ( T ) , i.e., there is sublinear in T T T gain to deviate to any strategy other than the one suggested by the BSE. Concretely, we want this to happen in expectation: for optimizer actions i t i_t i t and learner actions j t j_t j t we want ∑ t E [ U O [ i t , j t ] ] ≤ BSE ⋅ T + o ( T ) \sum_t \mathbb{E}[U_{\mathrm{O}}[i_t,j_t]] \le\operatorname{BSE} \cdot T + o(T) ∑ t E [ U O [ i t , j t ]] ≤ BSE ⋅ T + o ( T ) .
Our definition of BSE assumes that the learner’s goal is to best-respond in every round trying to keep her spending at most ρ L \rho_{\mathrm{L}} ρ L . Alternately, we can consider learners who may be willing to overspend following historical underspending. We explore these different definitions in Appendix § 10 , where we show that they can only benefit the optimizer.
Low stochastic regret. We want that when the learner is faced with a stationary environment (e.g., the optimizer uses the same mixed strategy in every round), then the learner has no regret. In fact, we achieve a stronger guarantee: in any interval of rounds I \mathcal{I} I the optimizer samples i t i_t i t from a fixed distribution of actions x x x , then the learner should have low regret in that interval with respect to the best-response of that interval y ⋆ ∈ BR L ( x ⊤ U L , x ⊤ C L ) y^\star\in\operatorname{BR}_{\mathrm{L}}(x^{\top}U_{\mathrm{L}},x^{\top}C_{\mathrm{L}}) y ⋆ ∈ BR L ( x ⊤ U L , x ⊤ C L ) :
∑ t ∈ I E [ U L [ i t , j t ] ] ≥ x ⊤ U L y ⋆ ⋅ ∣ I ∣ − o ( T ) . \sum_{t \in\mathcal{I}} \mathbb{E}[U_{\mathrm{L}}[i_t,j_t]] \ge x^{\top}U_{\mathrm{L}}y^\star\cdot|\mathcal{I}| - o(T). t ∈ I ∑ E [ U L [ i t , j t ]] ≥ x ⊤ U L y ⋆ ⋅ ∣ I ∣ − o ( T ) .
Aside from being a strong learning guarantee for the learner, this also offers a lower bound for the optimizer’s utility, the weak Budgeted Stackelberg Equilibrium . Specifically, the BSE value we defined in Optimization Program (2) is analogous to the strong Stackelberg Equilibrium , where ties between the learner’s best-responses are broken in favor of the optimizer. In contrast, in a weak Stackelberg Equilibrium such ties are broken to punish the optimizer, replacing the maximization over y y y in Optimization Program (2) with minimization. In § 7.1 and § 7.3 , we define the strong and weak Budgeted Stackelberg Equilibrium for both budgeted and unbudgeted optimizers, and show why the optimizer can achieve the weak BSE due to the guarantee above. In § 7.4 we show that in games with generic payoffs the weak and strong version of BSE are the same.
Non-manipulable Algorithm with Low Stochastic Regret
In this section, we define the first component of our algorithm that achieves the two objectives of our paper: non-manipulability and low stochastic regret. This component is based on receiving a forecast predicting the environment induced by the optimizer and then best-responding to this forecast. Specifically, when the optimizer picks randomized actions x x x , the learner would like to best respond by knowing x ⊤ U L x^{\top}U_L x ⊤ U L and x ⊤ C L x^{\top}C_L x ⊤ C L . Ideally, this means that the learning algorithm receives a prediction for these two n L n_L n L -dimensional vectors—which we denote u c ∈ [ 0 , 1 ] 2 n L uc \in[0,1]^{2n_L} u c ∈ [ 0 , 1 ] 2 n L —and best responds to them. Our guarantees in § 4 and § 5 , then become that as long as these forecasts are calibrated according to the right calibration error, then the optimizer’s gain over its BSE and the learner’s stochastic regret are proportional to that error. In § 6 we show how the learner can generate such calibrated forecasts.
The forecast received is a distribution over utility-cost vectors that the optimizer can generate, that is, a convex combination of such utility-cost vectors. Let
U C = { u c ∈ [ 0 , 1 ] 2 n L : u c = ( U L [ i , : ] , C L [ i , : ] ) , i ∈ [ n O ] } = { u c ( 1 ) , u c ( 2 ) , u c ( 3 ) , … } \mathcal{UC} = \{uc \in[0,1]^{2n_L} : uc = (U_L[i,:], C_L[i,:]), i \in[n_O]\} = \{uc^{(1)}, uc^{(2)}, uc^{(3)}, \ldots\} U C = { u c ∈ [ 0 , 1 ] 2 n L : u c = ( U L [ i , : ] , C L [ i , : ]) , i ∈ [ n O ]} = { u c ( 1 ) , u c ( 2 ) , u c ( 3 ) , … }
and n ^ O = ∣ U C ∣ ≤ n O \hat{n}_O = |\mathcal{UC}| \le n_O n ^ O = ∣ U C ∣ ≤ n O . Note that n ^ O < n O \hat{n}_O < n_O n ^ O < n O when there are multiple optimizer actions that create the same utility and cost vectors for the learner. While such actions are identical for the learner, their utility to the optimizer may be different and may depend on which action the learner takes. We use i i i to index actions by the optimizer and k k k to index elements in [ n ^ O ] [\hat{n}_O] [ n ^ O ] that can correspond to multiple different actions by the optimizer. Therefore, the forecast f f f is a convex combination of the elements of U C \mathcal{UC} U C ; the induced expected utility and cost vector for the learner is u c ( f ) = ∑ k = 1 n ^ O f k u c ( k ) uc(f) = \sum_{k=1}^{\hat{n}_O} f_k uc^{(k)} u c ( f ) = ∑ k = 1 n ^ O f k u c ( k ) . We similarly define u ( f ) u(f) u ( f ) and c ( f ) c(f) c ( f ) . Then, given this forecast, the learner best responds with mixed strategy y ∈ B R L ( u c ( f ) ) y \in\mathrm{BR}_L(uc(f)) y ∈ BR L ( u c ( f )) . This happens in active rounds , i.e., when the historical spending is at most t ρ L − 1 t\rho_L - 1 t ρ L − 1 . The formal description of this algorithm is in Algorithm 1.
ALGORITHM 1: Forecast-Based Best-Response Algorithm
Input: Forecaster over utility-cost vectors
for t = 1, . . . , T do
if historical spending > tρ_L − 1 then // Non-active round
Play the null action
else // Active round
Receive forecast f^(t) from the forecaster
Pick best response y_t ∈ BR_L(uc(f^(t))) with support at most 2 (breaking ties consistently)
Sample and play a pure action from j_t ∼ y_t
Observe realized utility-cost vector uc_t ∈ [0, 1]^(2n_L) and report it to the forecaster
In § 4 , we will describe, response-calibration error , which, if low, will guarantee that the optimizer’s utility is BSE · T + o ( T ) T + o(T) T + o ( T ) . Unfortunately, this non-manipulability property is insufficient to achieve low stochastic regret (see Appendix § 14 ). For this reason, we describe Lagrangian-calibration error in § 5 , which, if low, guarantees low stochastic regret. Finally, in § 6 we show how to minimize both calibration errors simultaneously.
Response-calibration Error (Low Manipulability)
In this section we present the calibration error we minimize that, if small, guarantees that the learner of Algorithm 1 is not manipulated by the adversary. The obvious thing to track is the error between the realized vector u c t uc_t u c t and the one implied by the forecast. However that doesn’t work: a prediction might be very close to its realization, but because of the non-continuous nature of the best-response function, this might not lead to approximately the same action by the learner. We showcase this with a simple example in Appendix § 12 . Instead, we introduce a new error, which we call response-calibration error . This tracks the error of the response that the learner takes with respect to the forecast. Specifically, for a forecast f f f , let J ⃗ ( f ) ∈ [ n L ] 2 \vec{J}(f) \in[n_L]^2 J ( f ) ∈ [ n L ] 2 be the two pure actions in the learner’s best-response (recall the best-response can always be supported on at most two actions) and Q ( f ) ∈ [ 0 , 1 ] Q(f) \in[0,1] Q ( f ) ∈ [ 0 , 1 ] be the probability of playing the first of those actions. If the best response to f f f is a pure action j j j we assume J ⃗ ( f ) = ( j , j ) \vec{J}(f)=(j,j) J ( f ) = ( j , j ) and Q ( f ) = 0 Q(f)=0 Q ( f ) = 0 , while for multiple best responses we assume any consistent tie-breaking rule. Ideally, for any triplet k ∈ [ n ^ O ] k \in[\hat{n}_O] k ∈ [ n ^ O ] and j 1 , j 2 ∈ [ n L ] j_1,j_2 \in[n_L] j 1 , j 2 ∈ [ n L ] , if we consider rounds when the optimizer chooses an action i i i that induces a cost-utility vector k ∈ [ n ^ O ] k \in[\hat{n}_O] k ∈ [ n ^ O ] and the learner best responded to their forecast by the pair ( j 1 , j 2 ) (j_1,j_2) ( j 1 , j 2 ) with some weight q q q , the empirical frequency with which the optimizer induces u c ( k ) uc(k) u c ( k ) is close to the average probability that the forecast assigned to u c ( k ) uc(k) u c ( k ) , and the q q q values are similar. As is standard in the calibration literature, low error cannot be achieved unless the forecast is randomized. Therefore, the forecasting algorithm produces randomized forecasts F F F over the convex combinations6 f f f of the possible utility-cost vectors U C \mathcal{UC} U C .
The response-calibration error. We now show how we calculate the response-calibration error of a single round. For a chosen forecast f f f we will separately collect the error for each learner response pair ( j 1 , j 2 ) (j_1,j_2) ( j 1 , j 2 ) and for each possible k ∈ [ n ^ O ] k \in[\hat{n}_O] k ∈ [ n ^ O ] that indexes U C \mathcal{UC} U C . The error is proportional to f k f_k f k when k ≠ k t k \ne k_t k = k t and ( 1 − f k ) (1-f_k) ( 1 − f k ) when k = k t k=k_t k = k t for the k t k_t k t that the optimizer chooses. To also account for the weight of the two actions in the best response, we define this for each ( k , j 1 , j 2 ) (k,j_1,j_2) ( k , j 1 , j 2 ) as a single parameter step function that is 0 above Q ( f ) Q(f) Q ( f ) . Formally, we define the error as a vector indexed by k ∈ [ n ^ O ] k \in[\hat{n}_O] k ∈ [ n ^ O ] and two learner actions j 1 , j 2 ∈ [ n L ] j_1,j_2 \in[n_L] j 1 , j 2 ∈ [ n L ] , where each element of the vector is a one dimensional step function (parametrized by q q q ):
δ k , j 1 , j 2 R ( f , k t ) ( q ) = { 0 if ( j 1 , j 2 ) ≠ J ⃗ ( f ) or q > Q ( f ) , 1 − f k if ( j 1 , j 2 ) = J ⃗ ( f ) , q ≤ Q ( f ) , k = k t , − f k if ( j 1 , j 2 ) = J ⃗ ( f ) , q ≤ Q ( f ) , k ≠ k t = 1 ( j 1 , j 2 ) = J ⃗ ( f ) ( 1 k t = k − f k ) 1 q ≤ Q ( f ) (3) \delta^R_{k,j_1,j_2}(f,k_t)(q)=
\begin{cases}
0 & \text{if } (j_1,j_2)\ne\vec{J}(f)\text{ or }q>Q(f),\\
1-f_k & \text{if } (j_1,j_2)=\vec{J}(f),\ q\le Q(f),\ k=k_t,\\
-f_k & \text{if } (j_1,j_2)=\vec{J}(f),\ q\le Q(f),\ k\ne k_t
\end{cases}
=\mathbb{1}_{(j_1,j_2)=\vec{J}(f)}\left(\mathbb{1}_{k_t=k}-f_k\right)\mathbb{1}_{q\le Q(f)} \tag*{(3)} δ k , j 1 , j 2 R ( f , k t ) ( q ) = ⎩ ⎨ ⎧ 0 1 − f k − f k if ( j 1 , j 2 ) = J ( f ) or q > Q ( f ) , if ( j 1 , j 2 ) = J ( f ) , q ≤ Q ( f ) , k = k t , if ( j 1 , j 2 ) = J ( f ) , q ≤ Q ( f ) , k = k t = 1 ( j 1 , j 2 ) = J ( f ) ( 1 k t = k − f k ) 1 q ≤ Q ( f ) (3)
Then, the error in round t t t when the randomized forecast is F t F_t F t is E f ∼ F t [ δ R ( f , k t ) ] \mathbb{E}_{f\sim F_t}[\delta^R(f,k_t)] E f ∼ F t [ δ R ( f , k t )] , an n ^ O n L 2 \hat{n}_O n_L^2 n ^ O n L 2 dimensional vector of functions. We then define the total error up to round t t t for active rounds to be
R t = ∑ t ′ ≤ t : t ′ ∈ T E f ∼ F t ′ [ δ R ( f , k t ′ ) ] ∈ ( R → R ) n ^ O n L 2 . (4) R_t=\sum_{t'\le t:t'\in\mathcal{T}}\mathbb{E}_{f\sim F_{t'}}[\delta^R(f,k_{t'})]\in(\mathbb{R}\to\mathbb{R})^{\hat{n}_O n_L^2}. \tag*{(4)} R t = t ′ ≤ t : t ′ ∈ T ∑ E f ∼ F t ′ [ δ R ( f , k t ′ )] ∈ ( R → R ) n ^ O n L 2 . (4)
Intuitively, Eq. (3) measures whether the forecasts are calibrated conditioned on the learner’s induced best response. The middle factor 1 k t = k − f k \mathbb{1}_{k_t=k}-f_k 1 k t = k − f k is the basic forecast error for utility-cost vector u c ( k ) uc(k) u c ( k ) , comparing the indicator that u c ( k t ) uc(k_t) u c ( k t ) was realized in round t t t against its predicted probability f k f_k f k . If that was always 0 (the forecast is always correct), then the error would be 0. Because the optimizer’s utility in round t t t depends on which response the forecast induces—and conditioning on the full n ^ O \hat{n}_O n ^ O -dimensional vector f f f would require exponentially many buckets—we group these forecast errors by the learner’s best response, which is determined by the action pair J ⃗ ( f ) = ( j 1 , j 2 ) \vec{J}(f)=(j_1,j_2) J ( f ) = ( j 1 , j 2 ) and the mixing probability Q ( f ) Q(f) Q ( f ) on j 1 j_1 j 1 . Since J ⃗ ( f ) \vec{J}(f) J ( f ) ranges over the finite set [ n L ] 2 [n_L]^2 [ n L ] 2 , the first factor 1 ( j 1 , j 2 ) = J ⃗ ( f ) \mathbb{1}_{(j_1,j_2)=\vec{J}(f)} 1 ( j 1 , j 2 ) = J ( f ) simply dedicates a separate coordinate to each pair ( j 1 , j 2 ) (j_1,j_2) ( j 1 , j 2 ) . In contrast, the mixing weight Q ( f ) ∈ [ 0 , 1 ] Q(f)\in[0,1] Q ( f ) ∈ [ 0 , 1 ] is continuous; to avoid discretizing [ 0 , 1 ] [0,1] [ 0 , 1 ] , the third factor represents Q ( f ) Q(f) Q ( f ) via the step function q ↦ 1 q ≤ Q ( f ) q\mapsto\mathbb{1}_{q\le Q(f)} q ↦ 1 q ≤ Q ( f ) .
The usage of the step function q ↦ 1 q ≤ q ′ q\mapsto\mathbb{1}_{q\le q'} q ↦ 1 q ≤ q ′ is more complicated. Consider a certain ( k , j 1 , j 2 ) (k,j_1,j_2) ( k , j 1 , j 2 ) triplet and the mixing weight q ′ q' q ′ of a step function. For the realization of this triplet (the optimizer plays action i i i that corresponds to k k k and the learner uses pair ( j 1 , j 2 ) (j_1,j_2) ( j 1 , j 2 ) ) the optimizer’s expected utility is q ′ U 0 [ i , j 1 ] + ( 1 − q ′ ) U 0 [ i , j 2 ] q'U_0[i,j_1]+(1-q')U_0[i,j_2] q ′ U 0 [ i , j 1 ] + ( 1 − q ′ ) U 0 [ i , j 2 ] , which is an affine function in q ′ q' q ′ . In fact, we can represent this utility via the right inner product between the step function q ↦ 1 q ≤ q ′ q\mapsto\mathbb{1}_{q\le q'} q ↦ 1 q ≤ q ′ and the derivative of this utility function. Therefore, if the norm of our calibration error (which is a weighted sum of step functions) is small, this inner product is also going to be small by the Cauchy-Schwarz inequality. This lets us bound the optimizer’s utility for any optimizer, without knowing the optimizer’s utility. We next define this inner product.
Remark 4.1. It would be tempting to collect the calibration error for each ( k , j ) (k,j) ( k , j ) pair rather than ( k , j 1 , j 2 ) (k,j_1,j_2) ( k , j 1 , j 2 ) , tracking calibration error when the learner chooses best response ( j 1 , j 2 ) (j_1,j_2) ( j 1 , j 2 ) separately per action. Unfortunately, when multiple actions of the optimizer, say i 1 , i 2 i_1,i_2 i 1 , i 2 , create the same environment k k k for the learner, the optimizer can manipulate the learner even if such a calibration error is close to 0. In this case, it is possible that the optimizer would benefit if they can choose i 1 i_1 i 1 when the learner chooses j 1 j_1 j 1 and i 2 i_2 i 2 when the learner chooses j 2 j_2 j 2 . Specifically, we present an example in Appendix § 13 , where the calibration error can be zero for both pairs ( k , j 1 ) (k,j_1) ( k , j 1 ) and ( k , j 2 ) (k,j_2) ( k , j 2 ) but by correlating i 1 i_1 i 1 with j 1 j_1 j 1 and i 2 i_2 i 2 with j 2 j_2 j 2 the optimizer achieves Ω ( T ) \Omega(T) Ω ( T ) higher utility than B S E ⋅ T \mathrm{BSE}\cdot T BSE ⋅ T . As we show, this is not possible when the error is tracked per pair of learner actions.
Inner product and norm. The basis of our response-calibration error is a linear combination of step functions (see (3) ), so we define a norm on them to define when this error is close to 0. We define this for more general functions, namely the Hilbert space L 2 ( [ − 1 , 1 ] ) L^2([-1,1]) L 2 ([ − 1 , 1 ]) .
The space L 2 ( [ − 1 , 1 ] ) L^2([-1,1]) L 2 ([ − 1 , 1 ]) consists of functions g , g ~ : [ − 1 , 1 ] → R g,\tilde{g} : [-1,1] \to\mathbb{R} g , g ~ : [ − 1 , 1 ] → R equipped with the inner product and norm
⟨ g , g ~ ⟩ = ∫ − 1 1 g ( x ) g ~ ( x ) d x and ∥ g ∥ L 2 2 = ⟨ g , g ⟩ = ∫ − 1 1 g ( x ) 2 d x < ∞ . (5) \langle g,\tilde{g}\rangle= \int_{-1}^{1} g(x)\tilde{g}(x)\,\mathrm{d}x
\qquad\text{and}\qquad
\lVert g\rVert_{L^2}^2 = \langle g,g\rangle= \int_{-1}^{1} g(x)^2\,\mathrm{d}x < \infty. \tag*{(5)} ⟨ g , g ~ ⟩ = ∫ − 1 1 g ( x ) g ~ ( x ) d x and ∥ g ∥ L 2 2 = ⟨ g , g ⟩ = ∫ − 1 1 g ( x ) 2 d x < ∞. (5)
Now consider the case of linear combinations of step functions like a coordinate of δ R \delta^R δ R or R t R_t R t : let r ( ⋅ ) = ∑ k w k 1 ⋅ ≤ q k r(\cdot)=\sum_k w_k\mathbf{1}_{\cdot\le q_k} r ( ⋅ ) = ∑ k w k 1 ⋅ ≤ q k where w k ∈ R w_k\in\mathbb{R} w k ∈ R , q k ∈ [ 0 , 1 ] q_k\in[0,1] q k ∈ [ 0 , 1 ] , and 1 ⋅ ≤ q k \mathbf{1}_{\cdot\le q_k} 1 ⋅ ≤ q k is the step function. Similarly let r ~ ( ⋅ ) = ∑ ℓ w ~ ℓ 1 ⋅ ≤ q ~ ℓ \tilde{r}(\cdot)=\sum_\ell\tilde{w}_\ell\mathbf{1}_{\cdot\le\tilde{q}_\ell} r ~ ( ⋅ ) = ∑ ℓ w ~ ℓ 1 ⋅ ≤ q ~ ℓ , then (5) becomes
⟨ r , r ~ ⟩ = ∑ k , ℓ w k w ~ ℓ ( 1 + min { q k , q ~ ℓ } ) and ∥ r ∥ L 2 2 = ∑ k , k ′ w k w k ′ ( 1 + min { q k , q k ′ } ) . (6) \langle r,\tilde{r}\rangle= \sum_{k,\ell} w_k\tilde{w}_\ell(1+\min\{q_k,\tilde{q}_\ell\})
\qquad\text{and}\qquad
\lVert r\rVert_{L^2}^2 = \sum_{k,k'} w_kw_{k'}(1+\min\{q_k,q_{k'}\}). \tag*{(6)} ⟨ r , r ~ ⟩ = k , ℓ ∑ w k w ~ ℓ ( 1 + min { q k , q ~ ℓ }) and ∥ r ∥ L 2 2 = k , k ′ ∑ w k w k ′ ( 1 + min { q k , q k ′ }) . (6)
We also extend all the above definitions to vectors of such functions, ( L 2 ( [ − 1 , 1 ] ) ) n (L^2([-1,1]))^n ( L 2 ([ − 1 , 1 ]) ) n for some n ∈ N n\in\mathbb{N} n ∈ N . Specifically, if R = ( r 1 , … , r n ) R=(r_1,\ldots,r_n) R = ( r 1 , … , r n ) and R ~ \tilde{R} R ~ are vectors of L 2 ( [ − 1 , 1 ] ) L^2([-1,1]) L 2 ([ − 1 , 1 ]) functions then
⟨ R , R ~ ⟩ = ∑ i = 1 n ⟨ r i , r ~ i ⟩ and ∥ R ∥ L 2 2 = ∑ i = 1 n ∥ r i ∥ L 2 2 . \langle R,\tilde{R}\rangle= \sum_{i=1}^{n}\langle r_i,\tilde{r}_i\rangle
\qquad\text{and}\qquad
\lVert R\rVert_{L^2}^2 = \sum_{i=1}^{n}\lVert r_i\rVert_{L^2}^2. ⟨ R , R ~ ⟩ = i = 1 ∑ n ⟨ r i , r ~ i ⟩ and ∥ R ∥ L 2 2 = i = 1 ∑ n ∥ r i ∥ L 2 2 .
We next use this norm for the n = n ^ 0 n L 2 n=\hat{n}_0n_L^2 n = n ^ 0 n L 2 dimensional function R t R_t R t defined in (4) . The simplifications of Eq. (6) of the general form in Eq. (5) imply efficient use of these functions and computation of their norms. Specifically, we show how to efficiently keep ∥ R ∥ L 2 \lVert R\rVert_{L^2} ∥ R ∥ L 2 low in Section 6.
In Section 4.1, we will upper bound the optimizer’s utility beyond their BSE value as a product ⟨ R T , g ⟩ \langle R_T,g\rangle ⟨ R T , g ⟩ . In every coordinate ( k , j 1 , j 2 ) (k,j_1,j_2) ( k , j 1 , j 2 ) , g g g has a function that takes as input q q q and upper bounds the utility of the optimizer with the best response of ( j 1 , j 2 ) (j_1,j_2) ( j 1 , j 2 ) with mixing weight q q q , assuming the optimizer’s action needs to induce k k k . Then we can bound this additional utility using the Cauchy-Schwarz inequality. This inner product and norm allows us to do this more efficiently. In Eq. (6) we see how perturbing all the q k q_k q k ’s by a little bit would barely perturb the norm, similar to how the optimizer’s utility barely changes. Instead, an approach that treats all q k q_k q k ’s as distinct (for example via discretization) would yield huge norm increase under such perturbations. In addition, the 1 term in Eq. (6) is due to making this definition over L 2 ( [ − 1 , 1 ] ) L^2([-1,1]) L 2 ([ − 1 , 1 ]) instead of L 2 ( [ 0 , 1 ] ) L^2([0,1]) L 2 ([ 0 , 1 ]) . Without this term, always using q k = 0 q_k=0 q k = 0 would imply a 0 norm, which is not meaningful. In fact, this 1 term is key to re-writing the optimizer’s utility as an inner product between the step functions and the affine function g g g —without the 1 term we could do this only for linear functions (functions whose value is 0 at 0).
Response-calibration Error Implies Non-manipulability
In this section, we prove that low response-calibration error implies that the optimizer cannot get utility substantially more than the BSE value during active rounds. Along with the fact the number of non-active rounds is small (see Lemma 5.1 ), we get our non-manipulability result.
Theorem 4.2. Let T \mathcal{T} T be the set of active rounds of Algorithm 1. Then the optimizer’s utility is
∑ t ∈ T E f ∼ F t [ e i t T U O y ( f ) ] ≤ B S E ⋅ ∣ T ∣ + 2 n O n L ∥ R T ∥ L 2 . \sum_{t\in\mathcal{T}} \mathbb{E}_{f\sim F_t}\left[e_{i_t}^{\mathsf{T}}U_{O}y(f)\right] \leq\mathrm{BSE}\cdot|\mathcal{T}|+\sqrt{2n_{O}n_{L}}\lVert R_{\mathcal{T}}\rVert_{L^2}. t ∈ T ∑ E f ∼ F t [ e i t T U O y ( f ) ] ≤ BSE ⋅ ∣ T ∣ + 2 n O n L ∥ R T ∥ L 2 .
The proof’s first step is to observe that BSE upper-bounds the optimizer’s value if the forecast is correct, for any forecast. In other words, for a forecast f f f , if the optimizer plays a mixed strategy x ⋆ ( f ) x^\star(f) x ⋆ ( f ) that induces f f f and the learner uses y ( f ) y(f) y ( f ) , then ( x ⋆ ( f ) ) T U O y ( f ) ≤ B S E (x^\star(f))^{\mathsf{T}}U_{O}y(f)\leq\mathrm{BSE} ( x ⋆ ( f ) ) T U O y ( f ) ≤ BSE . This implies that the optimizer’s benefit in round t t t over BSE can be expressed via ( e i t − x ⋆ ( f ) ) T U O y ( f ) (e_{i_t}-x^\star(f))^{\mathsf{T}}U_{O}y(f) ( e i t − x ⋆ ( f ) ) T U O y ( f ) , which measures the difference between the optimizer’s true utility and the one that would maximize her utility while consistent with forecast f f f .
The next step is to relate this to our response-calibration error. We do this via a function g k , j 1 , j 2 ( q ) g_{k,j_1,j_2}(q) g k , j 1 , j 2 ( q ) , defined for every triplet ( k , j 1 , j 2 ) (k,j_1,j_2) ( k , j 1 , j 2 ) as the maximum reward the optimizer can get when the learner uses ( j 1 , j 2 ) (j_1,j_2) ( j 1 , j 2 ) with mixing probability q q q and the optimizer uses an action consistent with u C ( k ) u_C^{(k)} u C ( k ) . This makes the quantity we cared about above to be upper-bounded by a linear combination of terms ⟨ g k , j 1 , j 2 ′ , 1 ≤ Q ( f ) ⟩ \langle g'_{k,j_1,j_2},\mathbf{1}_{\leq Q(f)}\rangle ⟨ g k , j 1 , j 2 ′ , 1 ≤ Q ( f ) ⟩ . Summing over active rounds, yields the calibration error via ⟨ g ′ , R T ⟩ \langle g',R_{\mathcal{T}}\rangle ⟨ g ′ , R T ⟩ . Cauchy-Schwarz yields the bound.
Proof. We first show that if the learner’s forecasts are correct, then the optimizer can get at most BSE utility per round. For a forecast f f f , let x ⋆ ( f ) x^\star(f) x ⋆ ( f ) be a mixed strategy of the optimizer that aligns with the forecast, while maximizing utility under that forecast:
x ⋆ ( f ) ∈ arg max x ∈ Δ n O x T U O y ( f ) such that u ( f ) = x T U L and c ( f ) = x T C L . x^\star(f)\in\underset{x\in\Delta_{n_O}}{\arg\max}\quad x^{\mathsf{T}}U_{O}y(f)\quad\text{such that}\quad u(f)=x^{\mathsf{T}}U_L\quad\text{and}\quad c(f)=x^{\mathsf{T}}C_L. x ⋆ ( f ) ∈ x ∈ Δ n O arg max x T U O y ( f ) such that u ( f ) = x T U L and c ( f ) = x T C L .
By definition of the Budgeted Stackelberg Equilibrium, the optimizer’s utility above is at most BSE for any mixed strategy of the optimizer x ⋆ ( f ) x^\star(f) x ⋆ ( f ) , i.e., ( x ⋆ ( f ) ) T U O y ( f ) ≤ B S E (x^\star(f))^{\mathsf{T}}U_{O}y(f)\leq\mathrm{BSE} ( x ⋆ ( f ) ) T U O y ( f ) ≤ BSE .
So if the learner always responds to a correct forecast, the optimizer can get utility at most BSE.
Instead, the expected utility of the optimizer when they take action i i i and the forecast is f f f is e i T U O y ( f ) e_i^{\mathsf{T}}U_{O}y(f) e i T U O y ( f ) . Combining these two expressions, we can transform the quantity that we bound. Specifically, because
e i t T U O y ( f ) − B S E ≤ ( e i t − x ⋆ ( f ) ) T U O y ( f ) , (7) e_{i_t}^{\mathsf{T}}U_{O}y(f)-\mathrm{BSE}\leq(e_{i_t}-x^\star(f))^{\mathsf{T}}U_{O}y(f), \tag*{(7)} e i t T U O y ( f ) − BSE ≤ ( e i t − x ⋆ ( f ) ) T U O y ( f ) , (7)
if we take expectation f ∼ F t f\sim F_t f ∼ F t , sum over t ∈ T t\in\mathcal{T} t ∈ T , and upper bound this by 2 n O n L ∥ R T ∥ L 2 \sqrt{2n_On_L}\lVert R_{\mathcal{T}}\rVert_{L^2} 2 n O n L ∥ R T ∥ L 2 we get the theorem.
Recall that the response-calibration error is a vector indexed by the learner’s action pairs ( j 1 , j 2 ) (j_1,j_2) ( j 1 , j 2 ) and the environment k k k induced by the optimizer (while Q ( f ) Q(f) Q ( f ) appears inside the terms of each index). For each such ( k , j 1 , j 2 ) (k,j_1,j_2) ( k , j 1 , j 2 ) , the optimizer may choose different actions (among those inducing the environment u C ( k ) u_C^{(k)} u C ( k ) ) depending on the Q Q Q value of the learner’s response. Let g g g be a n ^ O n L 2 \hat n_O n_L^2 n ^ O n L 2 dimensional vector of functions [ 0 , 1 ] → R [0,1]\to\mathbb{R} [ 0 , 1 ] → R , defined as the maximum optimizer utility over all pure actions i ∈ [ n O ] i\in[n_O] i ∈ [ n O ] that induce u C ( k ) ∈ U C u_C^{(k)}\in\mathcal{UC} u C ( k ) ∈ U C :
g k , j 1 , j 2 ( q ) = max i ∈ [ n O ] : ( U L [ i , : ] , C L [ i , : ] ) = u C ( k ) ( q U O [ i , j 1 ] + ( 1 − q ) U O [ i , j 2 ] ) . g_{k,j_1,j_2}(q)=\max_{i\in[n_O]:(U_L[i,:],C_L[i,:])=u_C^{(k)}}\left(qU_O[i,j_1]+(1-q)U_O[i,j_2]\right). g k , j 1 , j 2 ( q ) = i ∈ [ n O ] : ( U L [ i , : ] , C L [ i , : ]) = u C ( k ) max ( q U O [ i , j 1 ] + ( 1 − q ) U O [ i , j 2 ] ) .
Using this function we bound the difference in Eq. (7) above as
( e i t − x ⋆ ( f ) ) T U O y ( f ) ≤ ∑ k ∈ [ n ^ O ] ( 1 k = k t − f k ) ⋅ g k , J 1 ( f ) , J 2 ( f ) ( Q ( f ) ) = ∑ k ∈ [ n ^ O ] ∑ j 1 , j 2 1 ( j 1 , j 2 ) = J ˉ ( f ) ( 1 k = k t − f k ) ⋅ g k , j 1 , j 2 ( Q ( f ) ) . (8) \begin{aligned}
(e_{i_t}-x^\star(f))^{\mathsf{T}}U_{O}y(f)\leq\sum_{k\in[\hat n_O]}(\mathbb{1}_{k=k_t}-f_k)\cdot g_{k,J_1(f),J_2(f)}(Q(f)) \\
&=\sum_{k\in[\hat n_O]}\sum_{j_1,j_2}\mathbb{1}_{(j_1,j_2)=\bar J(f)}(\mathbb{1}_{k=k_t}-f_k)\cdot g_{k,j_1,j_2}(Q(f)). \tag*{(8)}
\end{aligned} ( e i t − x ⋆ ( f ) ) T U O y ( f ) ≤ k ∈ [ n ^ O ] ∑ ( 1 k = k t − f k ) ⋅ g k , J 1 ( f ) , J 2 ( f ) ( Q ( f )) = k ∈ [ n ^ O ] ∑ j 1 , j 2 ∑ 1 ( j 1 , j 2 ) = J ˉ ( f ) ( 1 k = k t − f k ) ⋅ g k , j 1 , j 2 ( Q ( f )) . (8)
The inequality above holds because i t i_t i t induces u c ( k t ) u c^{(k_t)} u c ( k t ) , so e i t T U O y ( f ) ≤ g k t , J 1 ( f ) , J 2 ( f ) ( Q ( f ) ) e_{i_t}^{\mathsf{T}}U_Oy(f) \leq g_{k_t,J_1(f),J_2(f)}(Q(f)) e i t T U O y ( f ) ≤ g k t , J 1 ( f ) , J 2 ( f ) ( Q ( f )) , and because
x ⋆ ( f ) T U O y ( f ) ≥ ∑ k ∈ [ n ^ O ] f k g k , J 1 ( f ) , J 2 ( f ) ( Q ( f ) ) x^\star(f)^{\mathsf{T}}U_Oy(f) \geq\sum_{k\in[\hat n_O]} f_k g_{k,J_1(f),J_2(f)}(Q(f)) x ⋆ ( f ) T U O y ( f ) ≥ k ∈ [ n ^ O ] ∑ f k g k , J 1 ( f ) , J 2 ( f ) ( Q ( f ))
since if we consider the i i i that is the maximizer in the definition of g k , J 1 ( f ) , J 2 ( f ) ( Q ( f ) ) g_{k,J_1(f),J_2(f)}(Q(f)) g k , J 1 ( f ) , J 2 ( f ) ( Q ( f )) for the Q ( f ) Q(f) Q ( f ) value and for each k k k and take expectation over k ∼ f k\sim f k ∼ f , we would get expected utility-cost vectors that match u c ( f ) uc(f) u c ( f ) (which is what the definition of x ⋆ ( f ) x^\star(f) x ⋆ ( f ) requires).
We proceed to express g k , j 1 , j 2 g_{k,j_1,j_2} g k , j 1 , j 2 as the integral of its derivative. To make this a clean, we also define for q < 0 q<0 q < 0 , g k , j 1 , j 2 ( q ) = ( 1 + q ) g k , j 1 , j 2 ( 0 ) g_{k,j_1,j_2}(q)=(1+q)g_{k,j_1,j_2}(0) g k , j 1 , j 2 ( q ) = ( 1 + q ) g k , j 1 , j 2 ( 0 ) , which makes g k , j 1 , j 2 g_{k,j_1,j_2} g k , j 1 , j 2 have domain [ − 1 , 1 ] [-1,1] [ − 1 , 1 ] . Overall g k , j 1 , j 2 g_{k,j_1,j_2} g k , j 1 , j 2 is piecewise linear, with slope at most ∣ g k , j 1 , j 2 ′ ( q ) ∣ ≤ 1 \lvert g'_{k,j_1,j_2}(q)\rvert\leq1 ∣ g k , j 1 , j 2 ′ ( q )∣ ≤ 1 , and g k , j 1 , j 2 ( − 1 ) = 0 g_{k,j_1,j_2}(-1)=0 g k , j 1 , j 2 ( − 1 ) = 0 ; these ensure that it is an element of L 2 ( [ − 1 , 1 ] ) L^2([-1,1]) L 2 ([ − 1 , 1 ]) and
g k , j 1 , j 2 ( q ) = ∫ − 1 q g k , j 1 , j 2 ′ ( x ) d x = ∫ − 1 1 g k , j 1 , j 2 ′ ( x ) 1 x ≤ q d x = ⟨ g k , j 1 , j 2 ′ , 1 ⋅ ≤ q ⟩ . g_{k,j_1,j_2}(q)=\int_{-1}^{q}g'_{k,j_1,j_2}(x)\,dx=\int_{-1}^{1}g'_{k,j_1,j_2}(x)\mathbf{1}_{x\leq q}\,dx=\left\langle g'_{k,j_1,j_2},\mathbf{1}_{\cdot\leq q}\right\rangle. g k , j 1 , j 2 ( q ) = ∫ − 1 q g k , j 1 , j 2 ′ ( x ) d x = ∫ − 1 1 g k , j 1 , j 2 ′ ( x ) 1 x ≤ q d x = ⟨ g k , j 1 , j 2 ′ , 1 ⋅ ≤ q ⟩ .
Applying this equality to Eq. (8) we get
( e i t − x ⋆ ( f ) ) T U O y ( f ) ≤ ∑ k ∈ [ n ^ O ] ∑ j 1 , j 2 1 ( j 1 , j 2 ) = J ⃗ ( f ) ( 1 k = k t − f k ) ⟨ g k , j 1 , j 2 ′ , 1 ⋅ ≤ Q ( f ) ⟩ = ⟨ g ′ , δ R ( f , k t ) ⟩ \left(e_{i_t}-x^\star(f)\right)^{\mathsf{T}}U_Oy(f)\leq\sum_{k\in[\hat n_O]}\sum_{j_1,j_2}\mathbf{1}_{(j_1,j_2)=\vec J(f)}\left(\mathbf{1}_{k=k_t}-f_k\right)\left\langle g'_{k,j_1,j_2},\mathbf{1}_{\cdot\leq Q(f)}\right\rangle=\left\langle g',\delta^R(f,k_t)\right\rangle ( e i t − x ⋆ ( f ) ) T U O y ( f ) ≤ k ∈ [ n ^ O ] ∑ j 1 , j 2 ∑ 1 ( j 1 , j 2 ) = J ( f ) ( 1 k = k t − f k ) ⟨ g k , j 1 , j 2 ′ , 1 ⋅ ≤ Q ( f ) ⟩ = ⟨ g ′ , δ R ( f , k t ) ⟩
Using Eq. (7) , taking expectation f ∼ F t f\sim F_t f ∼ F t , and summing over t ∈ T t\in\mathcal{T} t ∈ T we get
∑ t ∈ T E f ∼ F t [ e i t T U O y ( f ) ] − B S E ⋅ ∣ T ∣ ≤ ⟨ g ′ , R T ⟩ ≤ ∥ g ′ ∥ L 2 ∥ R T ∥ L 2 ≤ 2 n ^ O n L ∥ R T ∥ L 2 \sum_{t\in\mathcal{T}}\mathbb{E}_{f\sim F_t}\left[e_{i_t}^{\mathsf{T}}U_Oy(f)\right]-\mathrm{BSE}\cdot\lvert\mathcal{T}\rvert\leq\left\langle g',R_{\mathcal{T}}\right\rangle\leq\lVert g'\rVert_{L^2}\lVert R_{\mathcal{T}}\rVert_{L^2}\leq\sqrt{2\hat n_O n_L}\lVert R_{\mathcal{T}}\rVert_{L^2} t ∈ T ∑ E f ∼ F t [ e i t T U O y ( f ) ] − BSE ⋅ ∣ T ∣ ≤ ⟨ g ′ , R T ⟩ ≤ ∥ g ′ ∥ L 2 ∥ R T ∥ L 2 ≤ 2 n ^ O n L ∥ R T ∥ L 2
where in the second to last inequality we used Cauchy-Schwarz and in the last inequality we used
∥ g ′ ∥ L 2 2 = ∑ k ∈ [ n ^ O ] ∑ j 1 , j 2 ∥ g k , j 1 , j 2 ′ ∥ L 2 2 = ∑ k ∈ [ n ^ O ] ∑ j 1 , j 2 ∫ − 1 1 g k , j 1 , j 2 ′ ( x ) 2 d x ≤ ∑ k ∈ [ n ^ O ] ∑ j 1 , j 2 2 = 2 n ^ O n L 2 \lVert g'\rVert_{L^2}^2=\sum_{k\in[\hat n_O]}\sum_{j_1,j_2}\lVert g'_{k,j_1,j_2}\rVert_{L^2}^2=\sum_{k\in[\hat n_O]}\sum_{j_1,j_2}\int_{-1}^{1}g'_{k,j_1,j_2}(x)^2\,dx\leq\sum_{k\in[\hat n_O]}\sum_{j_1,j_2}2=2\hat n_O n_L^2 ∥ g ′ ∥ L 2 2 = k ∈ [ n ^ O ] ∑ j 1 , j 2 ∑ ∥ g k , j 1 , j 2 ′ ∥ L 2 2 = k ∈ [ n ^ O ] ∑ j 1 , j 2 ∑ ∫ − 1 1 g k , j 1 , j 2 ′ ( x ) 2 d x ≤ k ∈ [ n ^ O ] ∑ j 1 , j 2 ∑ 2 = 2 n ^ O n L 2
which uses that the slope of g g g is between − 1 -1 − 1 and 1 1 1 . ■
Why this norm is not enough for stochastic regret. Unfortunately, even if R T = 0 R_{\mathcal{T}}=0 R T = 0 , the learner may suffer Ω ( T ) \Omega(T) Ω ( T ) regret in a stochastic environment by best-responding to each of the optimizer’s pure actions instead of the mixed ones. Best-responding to a mixed action allows the learner to overspend for some optimizer actions and underspend on others, which can lead to more learner utility. If the forecast is correlated with the realization of the optimizer’s action, then the learner is forced to best-respond every round, which forces not overspending in every optimizer action. The example in Appendix § 14 shows that this can happen, even if the forecast is not at all correlated with the optimizer’s random choice. If distinct forecasts induce the same best response, the errors in forecasts can cancel each other in R T R_{\mathcal{T}} R T , even though the true average environment admits a strictly better feasible response.
Lagrangian-calibration Error (Low Stochastic Regret)
As the example of Appendix § 14 illustrates, keeping ∥ R t ∥ L 2 \lVert R_t\rVert_{L^2} ∥ R t ∥ L 2 small is not enough to guarantee low regret in stochastic environments. Moreover, bounding utility and cost errors through R t R_t R t alone would introduce a dependence on n ^ O \hat n_O n ^ O , whereas the stochastic regret bound we show will be independent of n ^ O \hat n_O n ^ O , similar to classic results in online learning. To resolve both issues, we introduce a second calibration error, which we call Lagrangian-calibration error and denote by L t L_t L t . This error allows us to get low stochastic interval regret, i.e., the learner’s regret in every interval where the optimizer uses a fixed distribution of actions is o ( T ) o(T) o ( T ) . As we mentioned before, this also implies a lower bound for a budgeted optimizer’s utility, see § 7.3 for details.
Lagrangian dual of the learner’s best response. Recall that a single forecast f ∈ Δ n ^ O f \in\Delta_{\hat n_O} f ∈ Δ n ^ O induces the predicted utility and cost vectors u c ( f ) = ( u ( f ) , c ( f ) ) = ∑ k = 1 n ^ O f k u c ( k ) uc(f) = (u(f),c(f)) = \sum_{k=1}^{\hat n_O} f_k uc^{(k)} u c ( f ) = ( u ( f ) , c ( f )) = ∑ k = 1 n ^ O f k u c ( k ) , and that the learner plays y ( f ) ∈ B R L ( u c ( f ) ) y(f) \in\mathrm{BR}_{L}(uc(f)) y ( f ) ∈ BR L ( u c ( f )) . Because y ( f ) y(f) y ( f ) is the solution to the linear program max y ∈ Δ n L : c ( f ) ⊤ y ≤ ρ L u ( f ) ⊤ y \max_{y\in\Delta_{n_L}:c(f)^\top y\le\rho_L}u(f)^\top y max y ∈ Δ n L : c ( f ) ⊤ y ≤ ρ L u ( f ) ⊤ y , we can associate with each forecast f f f an optimal dual (Lagrange) multiplier λ ( f ) ≥ 0 \lambda(f) \ge0 λ ( f ) ≥ 0 for the budget constraint c ( f ) ⊤ y ≤ ρ L c(f)^\top y \le\rho_L c ( f ) ⊤ y ≤ ρ L :
λ ( f ) ∈ a r g m i n λ ≥ 0 { λ ρ L + max j ∈ [ n L ] ( u ( f ) [ j ] − λ c ( f ) [ j ] ) } . (9) \lambda(f) \in\mathop{\rm argmin}_{\lambda\ge0}\left\{\lambda\rho_L+\max_{j\in[n_L]}\left(u(f)[j]-\lambda c(f)[j]\right)\right\}. \tag*{(9)} λ ( f ) ∈ argmin λ ≥ 0 { λ ρ L + j ∈ [ n L ] max ( u ( f ) [ j ] − λ c ( f ) [ j ] ) } . (9)
Intuitively, λ ( f ) \lambda(f) λ ( f ) measures the marginal value of budget under forecast f f f (how tight the budget constraint is). At λ = 0 \lambda=0 λ = 0 the dual objective in (9) is at most 1, and because the null action has cost 0 and utility at least 0, the objective at any λ ≥ 0 \lambda\ge0 λ ≥ 0 is at least λ ρ L \lambda\rho_L λ ρ L ; hence we can always pick λ ( f ) ∈ [ 0 , 1 / ρ L ] \lambda(f)\in[0,1/\rho_L] λ ( f ) ∈ [ 0 , 1/ ρ L ] .
The Lagrangian-calibration error. Fix a round t t t with forecast distribution F t F_t F t and realized utility-cost vector u c ( k t ) = ( u ( k t ) , c ( k t ) ) uc^{(k_t)}=(u^{(k_t)},c^{(k_t)}) u c ( k t ) = ( u ( k t ) , c ( k t ) ) . For a single forecast f f f , we define the single-round Lagrangian-calibration error δ L ( f , k t ) = ( δ L , u ( f , k t ) , δ L , c ( f , k t ) , δ L , u y ( f , k t ) , δ L , c y ( f , k t ) ) ∈ R 2 n L + 2 \delta^L(f,k_t)=(\delta^{L,u}(f,k_t),\delta^{L,c}(f,k_t),\delta^{L,uy}(f,k_t),\delta^{L,cy}(f,k_t))\in\mathbb{R}^{2n_L+2} δ L ( f , k t ) = ( δ L , u ( f , k t ) , δ L , c ( f , k t ) , δ L , u y ( f , k t ) , δ L , cy ( f , k t )) ∈ R 2 n L + 2 via the four components
δ L , u ( f , k t ) : = 1 2 n L ( u ( k t ) − u ( f ) ) ∈ R n L , δ L , c ( f , k t ) : = ρ L λ ( f ) 2 n L ( c ( k t ) − c ( f ) ) ∈ R n L , \delta^{L,u}(f,k_t):=\frac{1}{2\sqrt{n_L}}\left(u^{(k_t)}-u(f)\right)\in\mathbb{R}^{n_L},\qquad
\delta^{L,c}(f,k_t):=\frac{\rho_L\lambda(f)}{2\sqrt{n_L}}\left(c^{(k_t)}-c(f)\right)\in\mathbb{R}^{n_L}, δ L , u ( f , k t ) := 2 n L 1 ( u ( k t ) − u ( f ) ) ∈ R n L , δ L , c ( f , k t ) := 2 n L ρ L λ ( f ) ( c ( k t ) − c ( f ) ) ∈ R n L ,
δ L , u y ( f , k t ) : = 1 2 n L ( u ( k t ) − u ( f ) ) ⊤ y ( f ) ∈ R , δ L , c y ( f , k t ) : = 1 2 n L ( c ( k t ) − c ( f ) ) ⊤ y ( f ) ∈ R . (10) \delta^{L,uy}(f,k_t):=\frac{1}{2\sqrt{n_L}}\left(u^{(k_t)}-u(f)\right)^\top y(f)\in\mathbb{R},\qquad
\delta^{L,cy}(f,k_t):=\frac{1}{2\sqrt{n_L}}\left(c^{(k_t)}-c(f)\right)^\top y(f)\in\mathbb{R}. \tag*{(10)} δ L , u y ( f , k t ) := 2 n L 1 ( u ( k t ) − u ( f ) ) ⊤ y ( f ) ∈ R , δ L , cy ( f , k t ) := 2 n L 1 ( c ( k t ) − c ( f ) ) ⊤ y ( f ) ∈ R . (10)
The total Lagrangian-calibration error up to round t t t is then the sum of their expectations over f ∼ F t f\sim F_t f ∼ F t :
L t = ( L t u , L t c , L t u y , L t c y ) = ∑ t ′ ≤ t : t ′ ∈ T E f ∼ F t ′ [ δ L ( f , k t ′ ) ] ∈ R 2 n L + 2 . L_t=(L_t^u,L_t^c,L_t^{uy},L_t^{cy})=\sum_{t'\le t:t'\in\mathcal{T}}\mathbb{E}_{f\sim F_{t'}}\left[\delta^L(f,k_{t'})\right]\in\mathbb{R}^{2n_L+2}. L t = ( L t u , L t c , L t u y , L t cy ) = t ′ ≤ t : t ′ ∈ T ∑ E f ∼ F t ′ [ δ L ( f , k t ′ ) ] ∈ R 2 n L + 2 .
Because L t L_t L t is a standard vector in R 2 n L + 2 \mathbb{R}^{2n_L+2} R 2 n L + 2 , we measure its magnitude using the ordinary Euclidean norm ∥ L t ∥ 2 2 = ∥ L t u ∥ 2 2 + ∥ L t c ∥ 2 2 + ( L t u y ) 2 + ( L t c y ) 2 \|L_t\|_2^2=\|L_t^u\|_2^2+\|L_t^c\|_2^2+(L_t^{uy})^2+(L_t^{cy})^2 ∥ L t ∥ 2 2 = ∥ L t u ∥ 2 2 + ∥ L t c ∥ 2 2 + ( L t u y ) 2 + ( L t cy ) 2 . Since all entries of u ( k t ) − u ( f ) u^{(k_t)}-u(f) u ( k t ) − u ( f ) and ρ L λ ( f ) ( c ( k t ) − c ( f ) ) \rho_L\lambda(f)(c^{(k_t)}-c(f)) ρ L λ ( f ) ( c ( k t ) − c ( f )) lie in [ − 1 , 1 ] [-1,1] [ − 1 , 1 ] , the normalization factor 1 2 n L \frac{1}{2\sqrt{n_L}} 2 n L 1 in (10) guarantees that every single-round error vector satisfies
∥ δ L ( f , k t ) ∥ 2 2 ≤ n L + n L + 1 + 1 4 n L ≤ 1. \|\delta^L(f,k_t)\|_2^2\le\frac{n_L+n_L+1+1}{4n_L}\le1. ∥ δ L ( f , k t ) ∥ 2 2 ≤ 4 n L n L + n L + 1 + 1 ≤ 1.
In § 5.1 , and ignoring some technical details and terms not depending on T T T , we show how each of the four components of L t L_t L t controls a distinct step in bounding the learner’s regret against an optimal budget-feasible strategy y ⋆ ∈ Δ n L y^\star\in\Delta_{n_L} y ⋆ ∈ Δ n L :
∥ L T c ∥ 2 \|L_T^c\|_2 ∥ L T c ∥ 2 bounds the gap between the learner’s predicted utility and any y ⋆ y^\star y ⋆ ’s predicted utility by O ( ∥ L T c ∥ 2 ) O(\|L_T^c\|_2) O ( ∥ L T c ∥ 2 ) ; although y ⋆ y^\star y ⋆ is budget-feasible under the true environment, a forecast f f f may overestimate y ⋆ y^\star y ⋆ ’s cost when λ ( f ) > 0 \lambda(f)>0 λ ( f ) > 0 (making y ⋆ y^\star y ⋆ appear infeasible under f f f , as in the example of Appendix § 14 ), and the resulting Lagrangian cost penalty is controlled by ∥ L T c ∥ 2 \|L_T^c\|_2 ∥ L T c ∥ 2 .
Combined, a small norm ∥ L T ∥ 2 \lVert L_{T}\rVert_{2} ∥ L T ∥ 2 guarantees that the learner’s stochastic regret is at most O ( ∥ L T ∥ 2 + T ) O\left(\lVert L_{T}\rVert_{2}+\sqrt{T}\right) O ( ∥ L T ∥ 2 + T ) .
Lagrangian-calibration Error Implies Low Stochastic Regret
We now formalize how a small Lagrangian-calibration error L t L_{t} L t bounds both the number of non-active rounds and the learner’s regret. The first part also completes the proof that the optimizer’s benefit over her BSE value is small by showing there are only a few non-active rounds and then using Theorem 4.2 . For the learner’s regret, we will focus on a piecewise-stationary stochastic optimizer that switches mixed strategies across S S S intervals (with S = 1 S=1 S = 1 recovering the standard i.i.d. stochastic setting).
Bounding the number of non-active rounds. We first bound the expected number of non-active rounds, using L t c y L_{t}^{cy} L t cy . Our bound holds against any (even adversarial) optimizer.
Lemma 5.1. Let T ⊆ [ T ] \mathcal{T}\subseteq[T] T ⊆ [ T ] be the rounds in which Algorithm 1 follows the forecast. Against any optimizer, the expected number of non-active rounds satisfies
E [ T − ∣ T ∣ ] ≤ 2 n E [ max t ≤ T L t c y ] + 2 T + 1 ρ L . \mathbb{E}\left[T-\lvert\mathcal{T}\rvert\right]\leq\frac{2\sqrt{n}\,\mathbb{E}\left[\max_{t\leq T}L_{t}^{cy}\right]+2\sqrt{T}+1}{\rho_{L}}. E [ T − ∣ T ∣ ] ≤ ρ L 2 n E [ max t ≤ T L t cy ] + 2 T + 1 .
Due to the definition of L T c y L_{T}^{cy} L T cy , the proof is quite simple. The expected overspending of the learner above ρ L \rho_{L} ρ L in round t t t is bounded by E [ δ L , c y ( f , k t ) ] \mathbb{E}\left[\delta^{L,cy}(f,k_{t})\right] E [ δ L , cy ( f , k t ) ] since there is no overspending w.r.t. the forecast, c ( f ) T y ( f ) ≤ ρ L c(f)^{\mathsf{T}}y(f)\leq\rho_{L} c ( f ) T y ( f ) ≤ ρ L . This makes the total expected overspending roughly at most L T c y L_{T}^{cy} L T cy . Relating the expected spending with the actual spending via concentration inequalities implies the lemma.
Proof of Lemma 5.1. First let T t = T ∩ [ t ] \mathcal{T}_{t}=\mathcal{T}\cap[t] T t = T ∩ [ t ] be the set of active rounds up to t t t . Also, let τ ∈ [ T ] ∖ T \tau\in[T]\setminus\mathcal{T} τ ∈ [ T ] ∖ T be the last non-active round of Algorithm 1 (if T = [ T ] \mathcal{T}=[T] T = [ T ] , then T − ∣ T ∣ = 0 T-\lvert\mathcal{T}\rvert=0 T − ∣ T ∣ = 0 ). Because round τ \tau τ triggered the spending threshold in Algorithm 1 and the null action has cost 0 0 0 , the cumulative realized cost up to step τ \tau τ , C τ − 1 = C τ = ∑ r = 1 τ C L [ i r , j r ] C_{\tau-1}=C_{\tau}=\sum_{r=1}^{\tau}C_{L}[i_{r},j_{r}] C τ − 1 = C τ = ∑ r = 1 τ C L [ i r , j r ] satisfies C τ > τ ρ L − 1 C_{\tau}>\tau\rho_{L}-1 C τ > τ ρ L − 1 . Since all rounds after τ \tau τ belong to T \mathcal{T} T , the total number of non-active rounds is T − ∣ T ∣ = τ − ∣ T τ ∣ T-\lvert\mathcal{T}\rvert=\tau-\lvert\mathcal{T}_{\tau}\rvert T − ∣ T ∣ = τ − ∣ T τ ∣ . Rearranging C τ > τ ρ L − 1 C_{\tau}>\tau\rho_{L}-1 C τ > τ ρ L − 1 therefore gives
T − ∣ T ∣ = τ − ∣ T τ ∣ < C τ − ρ L ∣ T τ ∣ + 1 ρ L ≤ max t ∈ [ T ] ( C t − ρ L ∣ T t ∣ ) + 1 ρ L . (11) T-\lvert\mathcal{T}\rvert=\tau-\lvert\mathcal{T}_{\tau}\rvert<\frac{C_{\tau}-\rho_{L}\lvert\mathcal{T}_{\tau}\rvert+1}{\rho_{L}}\leq\frac{\max_{t\in[T]}\left(C_{t}-\rho_{L}\lvert\mathcal{T}_{t}\rvert\right)+1}{\rho_{L}}. \tag*{(11)} T − ∣ T ∣ = τ − ∣ T τ ∣ < ρ L C τ − ρ L ∣ T τ ∣ + 1 ≤ ρ L max t ∈ [ T ] ( C t − ρ L ∣ T t ∣ ) + 1 . (11)
Since the forecast’s mixed strategy always has spending at most ρ L \rho_{L} ρ L w.r.t. the forecast, we can write
C t − ρ L ∣ T t ∣ ≤ ∑ t ′ ∈ T t ( e i t ′ T C L e j t ′ − E f ∼ F t ′ [ c ( f ) T y ( f ) ] ) = ∑ t ′ ∈ T t ( e i t ′ T C L e j t ′ − E f ∼ F t ′ [ e i t ′ T C L y ( f ) ] ) − ∑ t ′ ∈ T t E f ∼ F t ′ [ c ( f ) T y ( f ) − e i t ′ T C L y ( f ) ] = ∑ t ′ ∈ T t ( e i t ′ T C L e j t ′ − E f ∼ F t ′ [ e i t ′ T C L y ( f ) ] ) + 2 n L t c y . \begin{aligned}
C_{t}-\rho_{L}\lvert\mathcal{T}_{t}\rvert
&\leq\sum_{t'\in\mathcal{T}_{t}}\left(e_{i_{t'}}^{\mathsf{T}}C_{L}e_{j_{t'}}-\mathbb{E}_{f\sim F_{t'}}\left[c(f)^{\mathsf{T}}y(f)\right]\right)\\
&=\sum_{t'\in\mathcal{T}_{t}}\left(e_{i_{t'}}^{\mathsf{T}}C_{L}e_{j_{t'}}-\mathbb{E}_{f\sim F_{t'}}\left[e_{i_{t'}}^{\mathsf{T}}C_{L}y(f)\right]\right)-\sum_{t'\in\mathcal{T}_{t}}\mathbb{E}_{f\sim F_{t'}}\left[c(f)^{\mathsf{T}}y(f)-e_{i_{t'}}^{\mathsf{T}}C_{L}y(f)\right]\\
&=\sum_{t'\in\mathcal{T}_{t}}\left(e_{i_{t'}}^{\mathsf{T}}C_{L}e_{j_{t'}}-\mathbb{E}_{f\sim F_{t'}}\left[e_{i_{t'}}^{\mathsf{T}}C_{L}y(f)\right]\right)+2\sqrt{n}L_{t}^{cy}.
\end{aligned} C t − ρ L ∣ T t ∣ ≤ t ′ ∈ T t ∑ ( e i t ′ T C L e j t ′ − E f ∼ F t ′ [ c ( f ) T y ( f ) ] ) = t ′ ∈ T t ∑ ( e i t ′ T C L e j t ′ − E f ∼ F t ′ [ e i t ′ T C L y ( f ) ] ) − t ′ ∈ T t ∑ E f ∼ F t ′ [ c ( f ) T y ( f ) − e i t ′ T C L y ( f ) ] = t ′ ∈ T t ∑ ( e i t ′ T C L e j t ′ − E f ∼ F t ′ [ e i t ′ T C L y ( f ) ] ) + 2 n L t cy .
Given the above and (11), to bound the lemma we have to bound the expected maximum of the above sum. We notice that it is a martingale M t M_{t} M t : let
m t = 1 t ∈ T ( e i t T C L e j t − E f ∼ F t [ e i t T C L y ( f ) ] ) and M t = ∑ t ′ ≤ t m t ′ . m_{t}=\mathbf{1}_{t\in\mathcal{T}}\left(e_{i_{t}}^{\mathsf{T}}C_{L}e_{j_{t}}-\mathbb{E}_{f\sim F_{t}}\left[e_{i_{t}}^{\mathsf{T}}C_{L}y(f)\right]\right)\qquad\text{and}\qquad M_{t}=\sum_{t'\leq t}m_{t'}. m t = 1 t ∈ T ( e i t T C L e j t − E f ∼ F t [ e i t T C L y ( f ) ] ) and M t = t ′ ≤ t ∑ m t ′ .
The expectation of m t m_t m t conditioned on the history up to round t − 1 t-1 t − 1 is 0 0 0 , making M t M_t M t a martingale. Given that ∣ m t ∣ ≤ 1 |m_t| \le1 ∣ m t ∣ ≤ 1 , Doob’s L 2 L^2 L 2 maximal inequality [32 Theorem 4.4.4] proves that
E [ ( max t ∣ M t ∣ ) 2 ] ≤ 4 E [ M T 2 ] ⟹ Jensen’s E [ max t ∣ M t ∣ ] ≤ 2 E [ M T 2 ] \mathbb{E}\left[\left(\max_t |M_t|\right)^2\right] \le4\mathbb{E}\left[M_T^2\right] \quad\overset{\text{Jensen’s}}{\Longrightarrow} \quad\mathbb{E}\left[\max_t |M_t|\right] \le2\sqrt{\mathbb{E}\left[M_T^2\right]} E [ ( t max ∣ M t ∣ ) 2 ] ≤ 4 E [ M T 2 ] ⟹ Jensen’s E [ t max ∣ M t ∣ ] ≤ 2 E [ M T 2 ]
For E [ M T 2 ] \mathbb{E}\left[M_T^2\right] E [ M T 2 ] we notice that by expanding the sum, cross terms have expectation 0 0 0 , i.e., E [ m t m t ′ ] = 0 \mathbb{E}\left[m_t m_{t'}\right] = 0 E [ m t m t ′ ] = 0 for t ≠ t ′ t \ne t' t = t ′ . This implies
E [ M T 2 ] = E [ ( ∑ t ′ ≤ T m t ′ ) 2 ] = E [ ∑ t ′ ≤ T m t ′ 2 ] ≤ T \mathbb{E}\left[M_T^2\right] = \mathbb{E}\left[\left(\sum_{t' \le T}m_{t'}\right)^2\right] = \mathbb{E}\left[\sum_{t' \le T}m_{t'}^2\right] \le T E [ M T 2 ] = E ( t ′ ≤ T ∑ m t ′ ) 2 = E [ t ′ ≤ T ∑ m t ′ 2 ] ≤ T
This completes the proof of the lemma. ■
We now prove that the learner enjoys low regret. Specifically, we show that if the T T T rounds are split into S ≥ 1 S \ge1 S ≥ 1 phases, then the learner will approximately best respond in each phase, assuming it has low Lagrangian-calibration error.
Theorem 5.2. Let I = { t 1 + 1 , … , t 2 } ⊆ [ T ] \mathcal{I} = \{t_1 + 1, \ldots, t_2\} \subseteq[T] I = { t 1 + 1 , … , t 2 } ⊆ [ T ] be any contiguous interval of rounds in which the optimizer draws i t ∼ x i_t \sim x i t ∼ x independently in each round t ∈ I t \in\mathcal{I} t ∈ I for some mixed strategy x ∈ Δ n 0 x \in\Delta_{n_0} x ∈ Δ n 0 , and let y ⋆ ∈ B R L ( x ) y^\star\in\mathrm{BR}_{L}(x) y ⋆ ∈ BR L ( x ) be the learner’s optimal budget-feasible response to x x x . Then, regardless of the optimizer’s play outside I \mathcal{I} I , the learner’s expected utility in interval I \mathcal{I} I satisfies
E [ ∑ t ∈ I U L [ i t , j t ] ] ≥ ∣ I ∣ ⋅ x T U L y ⋆ − 14 n L ρ L E [ max t ≤ T ∥ L t ∥ 2 ] − 2 T + 1 ρ L . \mathbb{E}\left[\sum_{t \in\mathcal{I}} U_L[i_t,j_t]\right] \ge|\mathcal{I}|\cdot x^\mathsf{T} U_L y^\star- \frac{14\sqrt{n_L}}{\rho_L}\mathbb{E}\left[\max_{t \le T}\|L_t\|_2\right] - \frac{2\sqrt{T}+1}{\rho_L}. E [ t ∈ I ∑ U L [ i t , j t ] ] ≥ ∣ I ∣ ⋅ x T U L y ⋆ − ρ L 14 n L E [ t ≤ T max ∥ L t ∥ 2 ] − ρ L 2 T + 1 .
Proof. By strong duality for the learner’s best-response linear program max y ∈ Δ n L c ( f ) T y \max_{y \in\Delta_{n_L}} c(f)^\mathsf{T}y max y ∈ Δ n L c ( f ) T y s.t. y T ρ L ≤ 0 y^\mathsf{T}\rho_L \le0 y T ρ L ≤ 0 , u ( f ) T y u(f)^\mathsf{T}y u ( f ) T y and its dual (9), for any forecast f f f , the best response y ( f ) y(f) y ( f ) maximizes the Lagrangian y ↦ ( u ( f ) − λ ( f ) c ( f ) ) T y y \mapsto(u(f)-\lambda(f)c(f))^\mathsf{T}y y ↦ ( u ( f ) − λ ( f ) c ( f ) ) T y and satisfies complementary slackness λ ( f ) ( ρ L − c ( f ) T y ( f ) ) = 0 \lambda(f)(\rho_L-c(f)^\mathsf{T}y(f))=0 λ ( f ) ( ρ L − c ( f ) T y ( f )) = 0 . Consequently, for every y ∈ Δ n L y \in\Delta_{n_L} y ∈ Δ n L ,
u ( f ) T y ( f ) = ( u ( f ) − λ ( f ) c ( f ) ) T y ( f ) + λ ( f ) ρ L ≥ ( u ( f ) T y + λ ( f ) ( ρ L − c ( f ) T y ) ) . (12) u(f)^\mathsf{T}y(f) = (u(f)-\lambda(f)c(f))^\mathsf{T}y(f)+\lambda(f)\rho_L \ge(u(f)^\mathsf{T}y+\lambda(f)(\rho_L-c(f)^\mathsf{T}y)). \tag*{(12)} u ( f ) T y ( f ) = ( u ( f ) − λ ( f ) c ( f ) ) T y ( f ) + λ ( f ) ρ L ≥ ( u ( f ) T y + λ ( f ) ( ρ L − c ( f ) T y )) . (12)
Now consider the suboptimality gap at round t ∈ I ∩ T t \in\mathcal{I} \cap\mathcal{T} t ∈ I ∩ T with respect to the best response of the learner y = y ⋆ ∈ B R L ( x ) y = y^\star\in\mathrm{BR}_L(x) y = y ⋆ ∈ BR L ( x ) , some forecast f f f , and the k t k_t k t induced by the optimizer’s action
( u ( k t ) ) T ( y ( f ) − y ⋆ ) = ( u ( k t ) − u ( f ) ) T y ( f ) + u ( f ) T y ( f ) − ( u ( k t ) ) T y ⋆ ≥ ( u ( k t ) − u ( f ) ) T y ( f ) + ( u ( f ) − u ( k t ) ) T y ⋆ + λ ( f ) ( ρ L − c ( f ) T y ⋆ ) = ( u ( k t ) − u ( f ) ) T y ( f ) + ( u ( f ) − u ( k t ) ) T y ⋆ + λ ( f ) ( c ( k t ) − c ( f ) ) T y ⋆ + λ ( f ) ( ρ L − ( c ( k t ) ) T y ⋆ ) \begin{aligned}
\left(u^{(k_t)}\right)^\mathsf{T}(y(f)-y^\star)
&= \left(u^{(k_t)}-u(f)\right)^\mathsf{T}y(f)+u(f)^\mathsf{T}y(f)-\left(u^{(k_t)}\right)^\mathsf{T}y^\star\\
&\ge\left(u^{(k_t)}-u(f)\right)^\mathsf{T}y(f)+\left(u(f)-u^{(k_t)}\right)^\mathsf{T}y^\star+\lambda(f)(\rho_L-c(f)^\mathsf{T}y^\star) \\
&= \left(u^{(k_t)}-u(f)\right)^\mathsf{T}y(f)+\left(u(f)-u^{(k_t)}\right)^\mathsf{T}y^\star+\lambda(f)\left(c^{(k_t)}-c(f)\right)^\mathsf{T}y^\star+\lambda(f)\left(\rho_L-\left(c^{(k_t)}\right)^\mathsf{T}y^\star\right)
\end{aligned} ( u ( k t ) ) T ( y ( f ) − y ⋆ ) = ( u ( k t ) − u ( f ) ) T y ( f ) + u ( f ) T y ( f ) − ( u ( k t ) ) T y ⋆ ≥ ( u ( k t ) − u ( f ) ) T y ( f ) + ( u ( f ) − u ( k t ) ) T y ⋆ + λ ( f ) ( ρ L − c ( f ) T y ⋆ ) = ( u ( k t ) − u ( f ) ) T y ( f ) + ( u ( f ) − u ( k t ) ) T y ⋆ + λ ( f ) ( c ( k t ) − c ( f ) ) T y ⋆ + λ ( f ) ( ρ L − ( c ( k t ) ) T y ⋆ )
By taking expectation f ∼ F t f \sim F_t f ∼ F t and summing over t ∈ I ∩ T t \in\mathcal{I} \cap\mathcal{T} t ∈ I ∩ T we bound each of the terms. For the first term,
∑ t ∈ I ∩ T E f ∼ F t [ ( u ( k t ) − u ( f ) ) T y ( f ) ] = 2 n L ( L t 2 u y − L t 1 u y ) ≥ − 4 n L max t ∈ [ T ] ∣ L t u y ∣ . \sum_{t \in\mathcal{I}\cap\mathcal{T}}\mathbb{E}_{f\sim F_t}\left[\left(u^{(k_t)}-u(f)\right)^\mathsf{T}y(f)\right] = 2\sqrt{n_L}\left(L_{t_2}^{uy}-L_{t_1}^{uy}\right) \ge-4\sqrt{n_L}\max_{t\in[T]}|L_t^{uy}|. t ∈ I ∩ T ∑ E f ∼ F t [ ( u ( k t ) − u ( f ) ) T y ( f ) ] = 2 n L ( L t 2 u y − L t 1 u y ) ≥ − 4 n L t ∈ [ T ] max ∣ L t u y ∣.
For the second term,
∑ t ∈ I ∩ T E f ∼ F t [ ( u ( f ) − u ( k t ) ) ⊤ y ⋆ ] = − 2 n L [ ( L t 2 u − L t 1 u ) ⊤ y ⋆ ] ≥ − 4 n L max t ∈ [ T ] ∥ L t u ∥ 2 . \sum_{t\in\mathcal I\cap\mathcal T}\mathbb{E}_{f\sim F_t}\left[\left(u(f)-u^{(k_t)}\right)^\top y^\star\right]=-2\sqrt{n_L}\left[(L^u_{t_2}-L^u_{t_1})^\top y^\star\right]\geq-4\sqrt{n_L}\max_{t\in[T]}\lVert L^u_t\rVert_2. t ∈ I ∩ T ∑ E f ∼ F t [ ( u ( f ) − u ( k t ) ) ⊤ y ⋆ ] = − 2 n L [ ( L t 2 u − L t 1 u ) ⊤ y ⋆ ] ≥ − 4 n L t ∈ [ T ] max ∥ L t u ∥ 2 .
For the third term,
∑ t ∈ I ∩ T E f ∼ F t [ λ ( f ) ( c ( k t ) − c ( f ) ) ⊤ y ⋆ ] = 2 n L ρ L [ ( L t 2 c − L t 1 c ) ⊤ y ⋆ ] ≥ − 4 n L ρ L max t ∈ [ T ] ∥ L t c ∥ 2 . \sum_{t\in\mathcal I\cap\mathcal T}\mathbb{E}_{f\sim F_t}\left[\lambda(f)\left(c^{(k_t)}-c(f)\right)^\top y^\star\right]=\frac{2\sqrt{n_L}}{\rho_L}\left[(L^c_{t_2}-L^c_{t_1})^\top y^\star\right]\geq-\frac{4\sqrt{n_L}}{\rho_L}\max_{t\in[T]}\lVert L^c_t\rVert_2. t ∈ I ∩ T ∑ E f ∼ F t [ λ ( f ) ( c ( k t ) − c ( f ) ) ⊤ y ⋆ ] = ρ L 2 n L [ ( L t 2 c − L t 1 c ) ⊤ y ⋆ ] ≥ − ρ L 4 n L t ∈ [ T ] max ∥ L t c ∥ 2 .
Adding up the first three terms makes them lower-bounded by − 12 n L ρ L max t ∈ [ T ] ∥ L t ∥ 2 -\frac{12\sqrt{n_L}}{\rho_L}\max_{t\in[T]}\lVert L_t\rVert_2 − ρ L 12 n L max t ∈ [ T ] ∥ L t ∥ 2 .
For the fourth term we instead take expectation over everything (and especially over the draw of k t k_t k t ),
∑ t ∈ I E [ 1 t ∈ T λ ( f ) ( ρ L − ( c ( k t ) ) ⊤ y ⋆ ) ] = ∑ t ∈ I E [ 1 t ∈ T λ ( f ) ( ρ L − x ⊤ C L y ⋆ ) ] ≥ 0. \sum_{t\in\mathcal I}\mathbb{E}\left[\mathbf{1}_{t\in\mathcal T}\lambda(f)\left(\rho_L-\left(c^{(k_t)}\right)^\top y^\star\right)\right]=\sum_{t\in\mathcal I}\mathbb{E}\left[\mathbf{1}_{t\in\mathcal T}\lambda(f)\left(\rho_L-x^\top C_Ly^\star\right)\right]\geq0. t ∈ I ∑ E [ 1 t ∈ T λ ( f ) ( ρ L − ( c ( k t ) ) ⊤ y ⋆ ) ] = t ∈ I ∑ E [ 1 t ∈ T λ ( f ) ( ρ L − x ⊤ C L y ⋆ ) ] ≥ 0.
where in the inequality we used that y ⋆ y^\star y ⋆ is a best-response to ( x ⊤ U L , x ⊤ C L ) (x^\top U_L,x^\top C_L) ( x ⊤ U L , x ⊤ C L ) and that λ ( f ) ≥ 0 \lambda(f)\geq0 λ ( f ) ≥ 0 .
This bounds the error accumulated over active rounds. In each non-active round the loss is at most 1, and the total expected number of non-active rounds is bounded by Lemma 5.1 . Adding that error completes the proof. ■
Calibration Error Minimization
In Sections 4 and 5, we showed that both the optimizer’s utility gain over BSE ⋅ T \cdot T ⋅ T and the learner’s stochastic regret are bounded by the norms of the response-calibration error R t R_t R t and the Lagrangian-calibration error L t L_t L t . In this section, we show how to generate randomized forecasts F 1 , … , F T F_1,\ldots,F_T F 1 , … , F T that simultaneously keep ∥ R t ∥ L 2 \lVert R_t\rVert_{L_2} ∥ R t ∥ L 2 and ∥ L t ∥ 2 \lVert L_t\rVert_2 ∥ L t ∥ 2 bounded by O ( T ) O(\sqrt{T}) O ( T ) across all rounds t ∈ [ T ] t\in[T] t ∈ [ T ] , completing the analysis of our algorithm. We achieve this without prior knowledge of U C \mathcal{UC} U C or n ^ O \hat n_O n ^ O .
Combining the two calibration errors. To minimize ∥ R t ∥ L 2 \lVert R_t\rVert_{L_2} ∥ R t ∥ L 2 and ∥ L t ∥ 2 \lVert L_t\rVert_2 ∥ L t ∥ 2 simultaneously, we combine them into a single joint error vector. For a forecast f ∈ Δ n ^ O f\in\Delta_{\hat n_O} f ∈ Δ n ^ O and a realized index k ∈ [ n ^ O ] k\in[\hat n_O] k ∈ [ n ^ O ] corresponding to u c ( k ) ∈ U C u_c(k)\in\mathcal{UC} u c ( k ) ∈ U C , we define the single-round joint calibration error δ ( f , k ) : = ( δ R ( f , k ) , δ L ( f , k ) ) \delta(f,k):=(\delta^R(f,k),\delta^L(f,k)) δ ( f , k ) := ( δ R ( f , k ) , δ L ( f , k )) and the cumulative joint error up to round t t t :
Z t : = ( R t , L t ) = ∑ t ′ ≤ t : t ′ ∈ T E f ∼ F t ′ [ δ ( f , k t ′ ) ] . Z_t:=(R_t,L_t)=\sum_{\substack{t'\leq t:\\t'\in\mathcal T}}\mathbb{E}_{f\sim F_{t'}}\left[\delta(f,k_{t'})\right]. Z t := ( R t , L t ) = t ′ ≤ t : t ′ ∈ T ∑ E f ∼ F t ′ [ δ ( f , k t ′ ) ] .
We equip the joint space with the inner product and norm combining (5) –(6) on R t R_t R t and the standard Euclidean inner product on L t L_t L t :
⟨ Z , Z ′ ⟩ = ⟨ R , R ′ ⟩ + ⟨ L , L ′ ⟩ and ∥ Z ∥ = ∥ R ∥ L 2 2 + ∥ L ∥ 2 2 . (13) \langle Z,Z'\rangle=\langle R,R'\rangle+\langle L,L'\rangle
\qquad\text{and}\qquad
\lVert Z\rVert=\sqrt{\lVert R\rVert_{L_2}^2+\lVert L\rVert_2^2}. \tag*{(13)} ⟨ Z , Z ′ ⟩ = ⟨ R , R ′ ⟩ + ⟨ L , L ′ ⟩ and ∥ Z ∥ = ∥ R ∥ L 2 2 + ∥ L ∥ 2 2 . (13)
For any forecast f ∈ Δ n ^ O f\in\Delta_{\hat n_O} f ∈ Δ n ^ O and any realized index k ∈ [ n ^ O ] k\in[\hat n_O] k ∈ [ n ^ O ] , the single-round error δ ( f , k ) \delta(f,k) δ ( f , k ) satisfies two key properties:
Bounded norm: In δ R ( f , k ) \delta^R(f,k) δ R ( f , k ) , only the coordinates corresponding to the action pair J ⃗ ( f ) \vec J(f) J ( f ) are nonzero, and ∥ 1 ≤ Q ( f ) ∥ L 2 2 = 1 + Q ( f ) ≤ 2 \lVert\mathbf{1}_{\leq Q(f)}\rVert_{L_2}^2=1+Q(f)\leq2 ∥ 1 ≤ Q ( f ) ∥ L 2 2 = 1 + Q ( f ) ≤ 2 by (6) . Since ∑ k ′ ∈ [ n ^ O ] ( 1 k = k ′ − f k ′ ) 2 ≤ 2 \sum_{k'\in[\hat n_O]}(\mathbf{1}_{k=k'}-f_{k'})^2\leq2 ∑ k ′ ∈ [ n ^ O ] ( 1 k = k ′ − f k ′ ) 2 ≤ 2 for any f ∈ Δ n ^ O f\in\Delta_{\hat n_O} f ∈ Δ n ^ O , we have ∥ δ R ( f , k ) ∥ L 2 2 ≤ 4 \lVert\delta^R(f,k)\rVert_{L_2}^2\leq4 ∥ δ R ( f , k ) ∥ L 2 2 ≤ 4 . Combined with ∥ δ L ( f , k ) ∥ 2 2 ≤ 1 \lVert\delta^L(f,k)\rVert_2^2\leq1 ∥ δ L ( f , k ) ∥ 2 2 ≤ 1 from § 5 , every single-round joint error satisfies
∥ δ ( f , k ) ∥ 2 = ∥ δ R ( f , k ) ∥ L 2 2 + ∥ δ L ( f , k ) ∥ 2 2 ≤ 5. (14) \lVert\delta(f,k)\rVert^2=\lVert\delta^R(f,k)\rVert_{L_2}^2+\lVert\delta^L(f,k)\rVert_2^2\leq5. \tag*{(14)} ∥ δ ( f , k ) ∥ 2 = ∥ δ R ( f , k ) ∥ L 2 2 + ∥ δ L ( f , k ) ∥ 2 2 ≤ 5. (14)
Unbiasedness under its own forecast: If the realized index k k k is drawn from the forecast f f f itself, then the error is 0:
E k ∼ f [ δ ( f , k ) ] = 0. (15) \mathbb{E}_{k\sim f}\left[\delta(f,k)\right]=0. \tag*{(15)} E k ∼ f [ δ ( f , k ) ] = 0. (15)
This follows from E k ∼ f [ 1 k = k ′ ] = f k ′ \mathbb{E}_{k\sim f}\left[\mathbf{1}_{k=k'}\right]=f_{k'} E k ∼ f [ 1 k = k ′ ] = f k ′ for all k ′ ∈ [ n ^ 0 ] k' \in[\hat n_0] k ′ ∈ [ n ^ 0 ] (which makes E k ∼ f [ δ R ( f , k ) ] = 0 \mathbb{E}_{k\sim f}\left[\delta^{\mathsf{R}}(f,k)\right]=0 E k ∼ f [ δ R ( f , k ) ] = 0 ), along with E k ∼ f [ u ( k ) ] = u ( f ) \mathbb{E}_{k\sim f}\left[u^{(k)}\right]=u(f) E k ∼ f [ u ( k ) ] = u ( f ) , and E k ∼ f [ c ( k ) ] = c ( f ) \mathbb{E}_{k\sim f}\left[c^{(k)}\right]=c(f) E k ∼ f [ c ( k ) ] = c ( f ) (which makes E k ∼ f [ δ L ( f , k ) ] = 0 \mathbb{E}_{k\sim f}\left[\delta^{\mathsf{L}}(f,k)\right]=0 E k ∼ f [ δ L ( f , k ) ] = 0 ).
Potential function and the zero-sum game
Following the standard Blackwell approachability framework [17 , 39 , 40 ] , we control the magnitude of Z t Z_t Z t by tracking the potential ∥ Z t ∥ 2 \lVert Z_t\rVert^2 ∥ Z t ∥ 2 . In any active round t ∈ T t \in\mathcal{T} t ∈ T , expanding ∥ Z t ∥ 2 = ∥ Z t − 1 + E f ∼ F t [ δ ( f , k t ) ] ∥ 2 \lVert Z_t\rVert^2=\lVert Z_{t-1}+\mathbb{E}_{f\sim F_t}[\delta(f,k_t)]\rVert^2 ∥ Z t ∥ 2 = ∥ Z t − 1 + E f ∼ F t [ δ ( f , k t )] ∥ 2 and using (14) yields
∥ Z t ∥ 2 − ∥ Z t − 1 ∥ 2 = 2 ⟨ Z t − 1 , E f ∼ F t [ δ ( f , k t ) ] ⟩ + ∥ E f ∼ F t [ δ ( f , k t ) ] ∥ 2 ≤ 2 E f ∼ F t [ ⟨ Z t − 1 , δ ( f , k t ) ⟩ ] + 5. (16) \lVert Z_t\rVert^2-\lVert Z_{t-1}\rVert^2
=2\left\langle Z_{t-1},\mathbb{E}_{f\sim F_t}[\delta(f,k_t)]\right\rangle+\left\lVert\mathbb{E}_{f\sim F_t}[\delta(f,k_t)]\right\rVert^2
\le2\mathbb{E}_{f\sim F_t}\left[\left\langle Z_{t-1},\delta(f,k_t)\right\rangle\right]+5. \tag*{(16)} ∥ Z t ∥ 2 − ∥ Z t − 1 ∥ 2 = 2 ⟨ Z t − 1 , E f ∼ F t [ δ ( f , k t )] ⟩ + ∥ E f ∼ F t [ δ ( f , k t )] ∥ 2 ≤ 2 E f ∼ F t [ ⟨ Z t − 1 , δ ( f , k t ) ⟩ ] + 5. (16)
Consequently, to ensure that ∥ Z t ∥ 2 \lVert Z_t\rVert^2 ∥ Z t ∥ 2 increases by at most a constant regardless of which k t ∈ [ n ^ 0 ] k_t \in[\hat n_0] k t ∈ [ n ^ 0 ] the optimizer induces, the forecaster wants to pick F t F_t F t so that E f ∼ F t [ ⟨ Z t − 1 , δ ( f , k ) ⟩ ] \mathbb{E}_{f\sim F_t}[\langle Z_{t-1},\delta(f,k)\rangle] E f ∼ F t [⟨ Z t − 1 , δ ( f , k )⟩] is small for every k ∈ [ n ^ 0 ] k \in[\hat n_0] k ∈ [ n ^ 0 ] . For example, if E f ∼ F t [ ⟨ Z t − 1 , δ ( f , k ) ⟩ ] ≤ 1 \mathbb{E}_{f\sim F_t}[\langle Z_{t-1},\delta(f,k)\rangle]\le1 E f ∼ F t [⟨ Z t − 1 , δ ( f , k )⟩] ≤ 1 for every k ∈ [ n ^ 0 ] k \in[\hat n_0] k ∈ [ n ^ 0 ] , then (16) implies ∥ Z t ∥ 2 − ∥ Z t − 1 ∥ 2 ≤ 7 \lVert Z_t\rVert^2-\lVert Z_{t-1}\rVert^2\le7 ∥ Z t ∥ 2 − ∥ Z t − 1 ∥ 2 ≤ 7 in every active round, which by telescoping over at most T T T rounds guarantees ∥ Z t ∥ 2 ≤ 7 T \lVert Z_t\rVert^2\le7T ∥ Z t ∥ 2 ≤ 7 T (and hence ∥ R t ∥ L 2 , ∥ L t ∥ 2 ≤ 7 T \lVert R_t\rVert_{L^2},\lVert L_t\rVert^2\le\sqrt{7T} ∥ R t ∥ L 2 , ∥ L t ∥ 2 ≤ 7 T ) for all t ∈ [ T ] t\in[T] t ∈ [ T ] .
To get intuition why such a distribution F t F_t F t exists, fix the cumulative error Z t − 1 Z_{t-1} Z t − 1 prior to round t t t and define for each forecast f ∈ Δ n ^ 0 f\in\Delta_{\hat n_0} f ∈ Δ n ^ 0 the potential drift vector (or drift vector for short) z ( f ) ∈ R n ^ 0 z(f)\in\mathbb{R}^{\hat n_0} z ( f ) ∈ R n ^ 0 with coordinates
z k ( f ) : = ⟨ Z t − 1 , δ ( f , k ) ⟩ ∀ k ∈ [ n ^ 0 ] . (17) z_k(f):=\left\langle Z_{t-1},\delta(f,k)\right\rangle\qquad\forall k\in[\hat n_0]. \tag*{(17)} z k ( f ) := ⟨ Z t − 1 , δ ( f , k ) ⟩ ∀ k ∈ [ n ^ 0 ] . (17)
We can interpret (17) as the payoff of a two-player zero-sum game in round t t t : the forecaster (the minimization player) chooses a forecast f ∈ Δ n ^ 0 f\in\Delta_{\hat n_0} f ∈ Δ n ^ 0 , and the environment player (the maximization player) chooses an outcome k ∈ [ n ^ 0 ] k\in[\hat n_0] k ∈ [ n ^ 0 ] . If the environment player moves first and commits to any mixed strategy w ∈ Δ n ^ 0 w\in\Delta_{\hat n_0} w ∈ Δ n ^ 0 , its expected payoff against forecast f f f is ∑ k ∈ [ n ^ 0 ] w k z k ( f ) = w ⊤ z ( f ) \sum_{k\in[\hat n_0]}w_kz_k(f)=w^\top z(f) ∑ k ∈ [ n ^ 0 ] w k z k ( f ) = w ⊤ z ( f ) . The forecaster can then simply respond with the identical forecast f = w f=w f = w as the pure action, which satisfies
w ⊤ z ( w ) = ∑ k w k ⟨ Z t − 1 , δ ( w , k ) ⟩ = ⟨ Z t − 1 , ∑ k w k δ ( w , k ) ⟩ = ⟨ Z t − 1 , E k ∼ w [ δ ( w , k ) ] ⟩ = 0 (18) w^\top z(w)=\sum_k w_k\left\langle Z_{t-1},\delta(w,k)\right\rangle
=\left\langle Z_{t-1},\sum_k w_k\delta(w,k)\right\rangle
=\left\langle Z_{t-1},\mathbb{E}_{k\sim w}[\delta(w,k)]\right\rangle=0 \tag*{(18)} w ⊤ z ( w ) = k ∑ w k ⟨ Z t − 1 , δ ( w , k ) ⟩ = ⟨ Z t − 1 , k ∑ w k δ ( w , k ) ⟩ = ⟨ Z t − 1 , E k ∼ w [ δ ( w , k )] ⟩ = 0 (18)
by Eq. (15) . Thus, the value of the game when the environment player moves first is at most 0:
max w ∈ Δ n ^ 0 min f ∈ Δ n ^ 0 w ⊤ z ( f ) ≤ 0. \max_{w\in\Delta_{\hat n_0}}\min_{f\in\Delta_{\hat n_0}}w^\top z(f)\le0. w ∈ Δ n ^ 0 max f ∈ Δ n ^ 0 min w ⊤ z ( f ) ≤ 0.
If a minimax theorem applied directly to this game, the Nash equilibrium would provide the forecaster with a randomized forecast F t F_t F t over Δ n ^ 0 \Delta_{\hat n_0} Δ n ^ 0 guaranteeing max k ∈ [ n ^ 0 ] E f ∼ F t [ z k ( f ) ] ≤ 0 \max_{k\in[\hat n_0]}\mathbb{E}_{f\sim F_t}[z_k(f)]\le0 max k ∈ [ n ^ 0 ] E f ∼ F t [ z k ( f )] ≤ 0 . However, because the learner’s best response y ( f ) y(f) y ( f ) and dual multiplier λ ( f ) \lambda(f) λ ( f ) can change discontinuously with f f f , the payoff f ↦ z k ( f ) f\mapsto z_k(f) f ↦ z k ( f ) is discontinuous, standard minimax theorems for infinite games do not apply directly. Moreover, even if we use an extension like Sion’s minimax theorem to prove the existence of such a randomized forecast F t F_t F t , there is no guarantee that this F t F_t F t can be represented efficiently. The obvious solution is to discretize the forecast space to restore a finite zero-sum game, but discretizing Δ n ^ 0 \Delta_{\hat n_0} Δ n ^ 0 into a fine grid would require exponential in n ^ 0 \hat n_0 n ^ 0 number of points.
Forecasting without knowing U C \mathcal{U}\mathcal{C} U C in advance. In addition to the continuity and discretization issues above, the learner does not know the matrices ( U L , C L ) (U_L,C_L) ( U L , C L ) or the number of distinct utility-cost vectors n ^ O = ∣ U C ∣ \hat{n}_{O} = |\mathcal{UC}| n ^ O = ∣ U C ∣ in advance, aside from the all-zeros vector u c ( 1 ) = 0 ∈ U C u_{c}^{(1)} = 0 \in\mathcal{UC} u c ( 1 ) = 0 ∈ U C corresponding to the null action. Therefore, the forecaster can only use forecasts that predict utility-cost vectors that have already been observed: let U C t ⊆ U C \mathcal{UC}_{t} \subseteq\mathcal{UC} U C t ⊆ U C be the set of distinct utility-cost vectors observed in rounds 1 , … , t 1,\ldots,t 1 , … , t (initialized at U C 0 = { 0 } \mathcal{UC}_{0} = \{0\} U C 0 = { 0 } ). Without loss of generality, we index the elements of U C = { u c ( 1 ) , … , u c ( n ^ O ) } \mathcal{UC} = \{u_{c}^{(1)},\ldots,u_{c}^{(\hat{n}_{O})}\} U C = { u c ( 1 ) , … , u c ( n ^ O ) } in the order they are first observed, so that prior to round t t t the observed dictionary is U C t − 1 = { u c ( 1 ) , … , u c ( n ^ t − 1 ) } \mathcal{UC}_{t-1} = \{u_{c}^{(1)},\ldots,u_{c}^{(\hat{n}_{t-1})}\} U C t − 1 = { u c ( 1 ) , … , u c ( n ^ t − 1 ) } of size n ^ t − 1 ≔ ∣ U C t − 1 ∣ ∈ [ 1 , min { n ^ O , t } ] \hat{n}_{t-1} \coloneqq|\mathcal{UC}_{t-1}| \in[1,\min\{\hat{n}_{O},t\}] n ^ t − 1 : = ∣ U C t − 1 ∣ ∈ [ 1 , min { n ^ O , t }] . In every active round t ∈ T t \in\mathcal{T} t ∈ T , the forecaster is restricted to the known indices [ n ^ t − 1 ] [\hat{n}_{t-1}] [ n ^ t − 1 ] : it outputs a distribution F t F_t F t over forecasts f ∈ Δ n ^ t − 1 f \in\Delta_{\hat{n}_{t-1}} f ∈ Δ n ^ t − 1 (viewed as elements of Δ n ^ O \Delta_{\hat{n}_{O}} Δ n ^ O by setting f k = 0 f_k = 0 f k = 0 for all unobserved k > n ^ t − 1 k > \hat{n}_{t-1} k > n ^ t − 1 ) against outcomes k ∈ [ n ^ t − 1 ] k \in[\hat{n}_{t-1}] k ∈ [ n ^ t − 1 ] , with drift vector z ( f ) = ( z 1 ( f ) , … , z n ^ t − 1 ( f ) ) ∈ R n ^ t − 1 z(f) = (z_1(f),\ldots,z_{\hat{n}_{t-1}}(f)) \in\mathbb{R}^{\hat{n}_{t-1}} z ( f ) = ( z 1 ( f ) , … , z n ^ t − 1 ( f )) ∈ R n ^ t − 1 . Because supp ( w ) ⊆ [ n ^ t − 1 ] \operatorname{supp}(w) \subseteq[\hat{n}_{t-1}] supp ( w ) ⊆ [ n ^ t − 1 ] for any w ∈ Δ n ^ t − 1 w \in\Delta_{\hat{n}_{t-1}} w ∈ Δ n ^ t − 1 , the identity w T z ( w ) = 0 w^{\mathsf{T}}z(w) = 0 w T z ( w ) = 0 in (18) continues to hold over Δ n ^ t − 1 \Delta_{\hat{n}_{t-1}} Δ n ^ t − 1 . For the rounds where the optimizer induces a vector that has not been seen before, the potential Z Z Z can increase by more than the above bound. However, the number of rounds this can happen is bounded by n ^ O \hat{n}_{O} n ^ O , which introduces an additional error of that size.
Approximating the Nash equilibrium via projected gradient ascent. Here we show how to get an approximate Nash equilibrium strategy F t F_t F t over Δ n ^ t − 1 \Delta_{\hat{n}_{t-1}} Δ n ^ t − 1 that guarantees expected payoff at most 1 (instead of 0) against every known k ∈ [ n ^ t − 1 ] k \in[\hat{n}_{t-1}] k ∈ [ n ^ t − 1 ] . We compute F t F_t F t constructively by simulating a learning process for the above zero-sum game. Specifically, the environment player runs Online Gradient Ascent [47 ] over M t = 50 n ^ t − 1 T 2 M_t = 50\hat{n}_{t-1}T^2 M t = 50 n ^ t − 1 T 2 iterations, to which the forecaster responds every time with the strategy we used in Eq. (18) . Thus, in iteration m ∈ [ M t ] m \in[M_t] m ∈ [ M t ] the maximization player maintains a distribution over its n ^ t − 1 \hat{n}_{t-1} n ^ t − 1 actions f ( m ) ∈ Δ n ^ t − 1 f^{(m)} \in\Delta_{\hat{n}_{t-1}} f ( m ) ∈ Δ n ^ t − 1 (recall the environment player’s pure actions are the outcomes k ∈ [ n ^ t − 1 ] k \in[\hat{n}_{t-1}] k ∈ [ n ^ t − 1 ] ) and the minimization player responds with the forecast f ( m ) f^{(m)} f ( m ) (recall the forecaster’s pure actions are forecasts) that guarantees at most 0 payoff by Eq. (18) . The final randomized forecast F t F_t F t is the uniform distribution over the forecasts f ( m ) f^{(m)} f ( m ) , which as we show next guarantees expected payoff at most 1 against any k ∈ [ n ^ t − 1 ] k \in[\hat{n}_{t-1}] k ∈ [ n ^ t − 1 ] .
ALGORITHM 2: Projected Gradient Ascent Forecaster for Active Round t ∈ T t \in\mathcal{T} t ∈ T
Input: Observed dictionary U C t − 1 \mathcal{UC}_{t-1} U C t − 1 of size n ^ t − 1 ≥ 1 \hat{n}_{t-1} \ge1 n ^ t − 1 ≥ 1 and cumulative calibration error Z t − 1 = ( R t − 1 , L t − 1 ) Z_{t-1} = (R_{t-1},L_{t-1}) Z t − 1 = ( R t − 1 , L t − 1 )
Set M t = 50 n ^ t − 1 T 2 M_t = 50\hat{n}_{t-1}T^2 M t = 50 n ^ t − 1 T 2 , step size η t = 1 M t \eta_t = \frac{1}{M_t} η t = M t 1 , and initial forecast f ( 1 ) = ( 1 / n ^ t − 1 , … , 1 / n ^ t − 1 ) ∈ Δ n ^ t − 1 f^{(1)} = (1/\hat{n}_{t-1},\ldots,1/\hat{n}_{t-1}) \in\Delta_{\hat{n}_{t-1}} f ( 1 ) = ( 1/ n ^ t − 1 , … , 1/ n ^ t − 1 ) ∈ Δ n ^ t − 1
for m = 1 , … , M t m = 1,\ldots,M_t m = 1 , … , M t do
Compute drift vector z ( f ( m ) ) ∈ R n ^ t − 1 z(f^{(m)}) \in\mathbb{R}^{\hat{n}_{t-1}} z ( f ( m ) ) ∈ R n ^ t − 1 with z k ( f ( m ) ) = ⟨ Z t − 1 , δ ( f ( m ) , k ) ⟩ z_k(f^{(m)}) = \langle Z_{t-1},\delta(f^{(m)},k)\rangle z k ( f ( m ) ) = ⟨ Z t − 1 , δ ( f ( m ) , k )⟩ for each k ∈ [ n ^ t − 1 ] k \in[\hat{n}_{t-1}] k ∈ [ n ^ t − 1 ]
Update f ( m + 1 ) = Π Δ n ^ t − 1 ( f ( m ) + η t z ( f ( m ) ) ) f^{(m+1)} = \Pi_{\Delta_{\hat{n}_{t-1}}}(f^{(m)} + \eta_t z(f^{(m)})) f ( m + 1 ) = Π Δ n ^ t − 1 ( f ( m ) + η t z ( f ( m ) )) , where Π Δ n ^ t − 1 ( ⋅ ) \Pi_{\Delta_{\hat{n}_{t-1}}}(\cdot) Π Δ n ^ t − 1 ( ⋅ ) is the Euclidean projection
Output: Uniform distribution F t = Unif ( { f ( 1 ) , … , f ( M t ) } ) F_t = \operatorname{Unif}(\{f^{(1)},\ldots,f^{(M_t)}\}) F t = Unif ({ f ( 1 ) , … , f ( M t ) })
Lemma 6.1. In every active round t ∈ T t \in\mathcal{T} t ∈ T , the forecast distribution F t F_t F t returned by Algorithm 2 satisfies
E f ∼ F t [ ⟨ Z t − 1 , δ ( f , k ) ⟩ ] ≤ 1 ∀ k ∈ [ n ^ t − 1 ] . (19) \mathop{\mathbb{E}}_{f \sim F_t}\left[\langle Z_{t-1},\delta(f,k)\rangle\right] \le1 \qquad\forall k \in[\hat{n}_{t-1}]. \tag*{(19)} E f ∼ F t [ ⟨ Z t − 1 , δ ( f , k )⟩ ] ≤ 1 ∀ k ∈ [ n ^ t − 1 ] . (19)
Proof. Fix a round t t t . We start with a crude bound on Z t − 1 Z_{t-1} Z t − 1 : since Z t − 1 Z_{t-1} Z t − 1 is the sum of at most t − 1 < T t-1 < T t − 1 < T single-round errors of norm at most 5 \sqrt{5} 5 by (14) , the triangle inequality gives ∥ Z t − 1 ∥ ≤ 5 T \lVert Z_{t-1}\rVert\le\sqrt{5T} ∥ Z t − 1 ∥ ≤ 5 T . By Cauchy–Schwarz and (14) , each coordinate of the drift vector is bounded by ∣ z k ( f ) ∣ ≤ ∥ Z t − 1 ∥ ∥ δ ( f , k ) ∥ ≤ 5 T |z_k(f)| \le\lVert Z_{t-1}\rVert\lVert\delta(f,k)\rVert\le5T ∣ z k ( f ) ∣ ≤ ∥ Z t − 1 ∥ ∥ δ ( f , k )∥ ≤ 5 T for all k ∈ [ n ^ t − 1 ] k \in[\hat{n}_{t-1}] k ∈ [ n ^ t − 1 ] , and thus ∥ z ( f ) ∥ 2 2 ≤ 25 n ^ t − 1 T 2 \lVert z(f)\rVert_2^2 \le25\hat{n}_{t-1}T^2 ∥ z ( f ) ∥ 2 2 ≤ 25 n ^ t − 1 T 2 for every f ∈ Δ n ^ t − 1 f \in\Delta_{\hat{n}_{t-1}} f ∈ Δ n ^ t − 1 . This bound explains the number of iterations M t = 50 n ^ t − 1 T 2 ≥ 2 ∥ z ( f ) ∥ 2 2 M_t = 50\hat{n}_{t-1}T^2 \ge2\lVert z(f)\rVert_2^2 M t = 50 n ^ t − 1 T 2 ≥ 2 ∥ z ( f ) ∥ 2 2 . Note that a larger M t M_t M t would have led to less error (specifically we will get linear in M t M_t M t regret) but that is sufficient for the result we are looking for.
We now follow the standard analysis of Projected Gradient Descent, which we include here for completeness. Fix any pure action of the environment player k ∈ [ n ^ t − 1 ] k \in[\hat{n}_{t-1}] k ∈ [ n ^ t − 1 ] and its forecast e k ∈ Δ n ^ t − 1 e_k \in\Delta_{\hat{n}_{t-1}} e k ∈ Δ n ^ t − 1 . By definition of the Euclidean projection we have for each iteration m ∈ [ M t ] m \in[M_t] m ∈ [ M t ] ,
∥ f ( m + 1 ) − e k ∥ 2 2 ≤ ∥ f ( m ) + η t z ( f ( m ) ) − e k ∥ 2 2 = ∥ f ( m ) − e k ∥ 2 2 + 2 η t ( f ( m ) − e k ) T z ( f ( m ) ) + η t 2 ∥ z ( f ( m ) ) ∥ 2 2 \left\lVert f^{(m+1)} - e_k \right\rVert_2^2 \le\left\lVert f^{(m)} + \eta_t z(f^{(m)}) - e_k \right\rVert_2^2 = \left\lVert f^{(m)} - e_k \right\rVert_2^2 + 2\eta_t\left(f^{(m)} - e_k\right)^{\mathsf{T}}z(f^{(m)}) + \eta_t^2\left\lVert z(f^{(m)})\right\rVert_2^2 f ( m + 1 ) − e k 2 2 ≤ f ( m ) + η t z ( f ( m ) ) − e k 2 2 = f ( m ) − e k 2 2 + 2 η t ( f ( m ) − e k ) T z ( f ( m ) ) + η t 2 z ( f ( m ) ) 2 2
Rearranging, summing over m = 1 , … , M t m = 1, \ldots, M_t m = 1 , … , M t , telescoping, and using ∥ f ( M t + 1 ) − e k ∥ 2 2 ≥ 0 \lVert f^{(M_t+1)} - e_k\rVert_2^2 \ge0 ∥ f ( M t + 1 ) − e k ∥ 2 2 ≥ 0 yields the regret of this process:
∑ m = 1 M t ( e k − f ( m ) ) ⊤ z ( f ( m ) ) ≤ ∥ f ( 1 ) − e k ∥ 2 2 2 η t + η t 2 ∑ m = 1 M t ∥ z ( f ( m ) ) ∥ 2 2 ≤ 1 η t + η t 2 M t M t 2 = M t \sum_{m=1}^{M_t} \left(e_k - f^{(m)}\right)^\top z\left(f^{(m)}\right) \le\frac{\lVert f^{(1)} - e_k\rVert_2^2}{2\eta_t} + \frac{\eta_t}{2} \sum_{m=1}^{M_t} \lVert z\left(f^{(m)}\right)\rVert_2^2 \le\frac{1}{\eta_t} + \frac{\eta_t}{2}M_t\frac{M_t}{2} = M_t m = 1 ∑ M t ( e k − f ( m ) ) ⊤ z ( f ( m ) ) ≤ 2 η t ∥ f ( 1 ) − e k ∥ 2 2 + 2 η t m = 1 ∑ M t ∥ z ( f ( m ) ) ∥ 2 2 ≤ η t 1 + 2 η t M t 2 M t = M t
where we used ∥ f ( 1 ) − e k ∥ 2 2 ≤ 2 \lVert f^{(1)} - e_k\rVert_2^2 \le2 ∥ f ( 1 ) − e k ∥ 2 2 ≤ 2 , ∥ z ( f ( m ) ) ∥ 2 2 ≤ M t / 2 = 1 / η t \lVert z(f^{(m)})\rVert_2^2 \le M_t/2 = 1/\eta_t ∥ z ( f ( m ) ) ∥ 2 2 ≤ M t /2 = 1/ η t .
By (18) , the forecaster’s choice f ( m ) f^{(m)} f ( m ) satisfies ( f ( m ) ) ⊤ z ( f ( m ) ) = 0 \left(f^{(m)}\right)^\top z(f^{(m)}) = 0 ( f ( m ) ) ⊤ z ( f ( m ) ) = 0 for every m ∈ [ M t ] m \in[M_t] m ∈ [ M t ] . Dividing by M t M_t M t makes the above inequality
E f ∼ F t [ ⟨ Z t − 1 , δ ( f , k ) ⟩ ] = 1 M t ∑ m = 1 M t z k ( f ( m ) ) ≤ 1 , \mathbb{E}_{f \sim F_t}\left[\langle Z_{t-1},\delta(f,k)\rangle\right] = \frac{1}{M_t}\sum_{m=1}^{M_t} z_k\left(f^{(m)}\right) \le1, E f ∼ F t [ ⟨ Z t − 1 , δ ( f , k )⟩ ] = M t 1 m = 1 ∑ M t z k ( f ( m ) ) ≤ 1 ,
proving the lemma.
Bounding the Cumulative Calibration Error
We now use Lemma 6.1 to bound the cumulative error ∥ Z t ∥ \lVert Z_t\rVert ∥ Z t ∥ in both the adversarial and stochastic settings. By examining the stochastic setting more carefully, we get a bound that does not depend on the number of actions that the optimizer has, which matches the bounds in Stochastic BwK.
Bound against any optimizer. The bound in this case follows from Lemma 6.1. For rounds where the realized vector u c ( k t ) u^{c(k_t)} u c ( k t ) has already been observed, the strategy produced by Algorithm 2 guarantees low norm increase. On the other hand, there are at most n ^ 0 \hat n_0 n ^ 0 rounds where a new vector can be observed. This makes the total error O ( T + n ^ 0 ) O(\sqrt{T}+\hat n_0) O ( T + n ^ 0 ) .
Lemma 6.2. Let T t \mathcal{T}_t T t be the active rounds up to round t t t . Against any optimizer and in every round t ∈ [ T ] t \in[T] t ∈ [ T ] , it holds
∥ R t ∥ 2 2 + ∥ L t ∥ 2 2 = ∥ Z t ∥ ≤ 7 ∣ T t ∣ + 5 n ^ 0 ≤ 7 T + 5 n 0 . (20) \sqrt{\lVert R_t\rVert_2^2 + \lVert L_t\rVert_2^2} = \lVert Z_t\rVert\le\sqrt{7|\mathcal{T}_t|} + \sqrt{5}\hat n_0 \le\sqrt{7T} + \sqrt{5}n_0. \tag*{(20)} ∥ R t ∥ 2 2 + ∥ L t ∥ 2 2 = ∥ Z t ∥ ≤ 7∣ T t ∣ + 5 n ^ 0 ≤ 7 T + 5 n 0 . (20)
Proof. We show by induction on t t t that ∥ Z t ∥ ≤ 7 ∣ T t ∣ + 5 n ^ t \lVert Z_t\rVert\le\sqrt{7|\mathcal{T}_t|} + \sqrt{5}\hat n_t ∥ Z t ∥ ≤ 7∣ T t ∣ + 5 n ^ t . The claim trivially holds for t = 0 t = 0 t = 0 , since ∥ Z 0 ∥ = 0 \lVert Z_0\rVert= 0 ∥ Z 0 ∥ = 0 .
We now prove the lemma assuming it is true for t − 1 t - 1 t − 1 . For non-active rounds t ∉ T t \notin\mathcal{T} t ∈ / T , Z t = Z t − 1 Z_t = Z_{t-1} Z t = Z t − 1 . For active rounds t ∈ T t \in\mathcal{T} t ∈ T , we distinguish two cases. First, if k t ∈ [ n ^ t − 1 ] k_t \in[\hat n_{t-1}] k t ∈ [ n ^ t − 1 ] has already been seen, then n ^ t = n ^ t − 1 \hat n_t = \hat n_{t-1} n ^ t = n ^ t − 1 and it holds
∥ Z t ∥ 2 = ∥ Z t − 1 + E f ∼ F t [ δ ( f , k t ) ] ∥ 2 = ∥ Z t − 1 ∥ 2 + 2 ⟨ Z t − 1 , E f ∼ F t [ δ ( f , k t ) ] ⟩ + ∥ E f ∼ F t [ δ ( f , k t ) ] ∥ 2 ≤ ∥ Z t − 1 ∥ 2 + 2 + 5 \lVert Z_t\rVert^2 = \left\lVert Z_{t-1} + \mathbb{E}_{f\sim F_t}\left[\delta(f,k_t)\right]\right\rVert^2 = \lVert Z_{t-1}\rVert^2 + 2\left\langle Z_{t-1},\mathbb{E}_{f\sim F_t}\left[\delta(f,k_t)\right]\right\rangle+ \left\lVert\mathbb{E}_{f\sim F_t}\left[\delta(f,k_t)\right]\right\rVert^2 \le\lVert Z_{t-1}\rVert^2 + 2 + 5 ∥ Z t ∥ 2 = ∥ Z t − 1 + E f ∼ F t [ δ ( f , k t ) ] ∥ 2 = ∥ Z t − 1 ∥ 2 + 2 ⟨ Z t − 1 , E f ∼ F t [ δ ( f , k t ) ] ⟩ + ∥ E f ∼ F t [ δ ( f , k t ) ] ∥ 2 ≤ ∥ Z t − 1 ∥ 2 + 2 + 5
where in the inequality we used Lemma 6.1 and Eq. (14) . Using the inductive hypothesis and some algebra proves the desired bound for ∥ Z t ∥ \lVert Z_t\rVert ∥ Z t ∥ :
∥ Z t ∥ 2 ≤ ( 7 ∣ T t − 1 ∣ + 5 n ^ t − 1 ) 2 + 7 = ( 7 ( ∣ T t ∣ − 1 ) + 5 n ^ t ) 2 + 7 ≤ ( 7 ∣ T t ∣ + 5 n ^ t ) 2 \lVert Z_t\rVert^2 \le\left(\sqrt{7|\mathcal{T}_{t-1}|} + \sqrt{5}\hat n_{t-1}\right)^2 + 7 = \left(\sqrt{7(|\mathcal{T}_t|-1)} + \sqrt{5}\hat n_t\right)^2 + 7 \le\left(\sqrt{7|\mathcal{T}_t|} + \sqrt{5}\hat n_t\right)^2 ∥ Z t ∥ 2 ≤ ( 7∣ T t − 1 ∣ + 5 n ^ t − 1 ) 2 + 7 = ( 7 ( ∣ T t ∣ − 1 ) + 5 n ^ t ) 2 + 7 ≤ ( 7∣ T t ∣ + 5 n ^ t ) 2
where the last inequality holds because ∣ T t ∣ ≥ 1 |\mathcal{T}_t| \ge1 ∣ T t ∣ ≥ 1 and n ^ t ≥ 0 \hat n_t \ge0 n ^ t ≥ 0 .
The second case is when k t ∉ [ n ^ t − 1 ] k_t \notin[\hat n_{t-1}] k t ∈ / [ n ^ t − 1 ] , i.e., the utility-cost vector has not been observed before. In that case, n ^ t = n ^ t − 1 + 1 \hat n_t = \hat n_{t-1} + 1 n ^ t = n ^ t − 1 + 1 , which makes ∥ Z t ∥ ≤ ∥ Z t − 1 ∥ + 5 ≤ 7 ∣ T t ∣ + 5 n ^ t \lVert Z_t\rVert\le\lVert Z_{t-1}\rVert+ \sqrt{5} \le\sqrt{7|\mathcal{T}_t|} + \sqrt{5}\hat n_t ∥ Z t ∥ ≤ ∥ Z t − 1 ∥ + 5 ≤ 7∣ T t ∣ + 5 n ^ t by Eq. (14) . ■
Dimension-free bound in stochastic environments. While Lemma 6.2 suffices to bound the optimizer’s utility we prove in our main Theorem 6.7 using Theorem 4.2 , plugging it into Theorem 5.2 would add an O ( n ^ 0 ) O(\hat{n}_0) O ( n ^ 0 ) term to the learner’s stochastic regret. When the optimizer plays i.i.d. strategies across S ≥ 1 S \ge1 S ≥ 1 intervals, we can remove this dependence on n ^ 0 \hat{n}_0 n ^ 0 completely. This dependence is also more natural given existing bounds for Stochastic BwK, where the learner’s regret does not depend on the number of different environments it will face.
The key intuition is simple: although the forecaster does not know the true distribution of k t k_t k t in interval I s \mathcal{I}_s I s , the empirical distribution of the vectors observed in the earlier rounds of I s \mathcal{I}_s I s is supported entirely on the known vectors U C t − 1 \mathcal{UC}_{t-1} U C t − 1 . Hence in our analysis we can use Lemma 6.1 assuming the optimizer uses the empirical distribution instead of the true distribution. However, this introduces an error proportional to the error between the true and empirical distributions. We show this error is small by standard concentration bounds between the two quantities, i.e., the variance after τ \tau τ rounds of the interval has passed is O ( 1 / τ ) O(1/\sqrt{\tau}) O ( 1/ τ ) in expectation, which is independent of the support size n ^ 0 \hat{n}_0 n ^ 0 .
Lemma 6.3. Suppose the horizon [ T ] [T] [ T ] is partitioned into S ≥ 1 S \ge1 S ≥ 1 contiguous intervals I 1 , … , I S \mathcal{I}_1,\ldots,\mathcal{I}_S I 1 , … , I S , where in each interval I s \mathcal{I}_s I s the optimizer draws i t ∼ x ( s ) i_t \sim x^{(s)} i t ∼ x ( s ) independently for some x ( s ) ∈ Δ n 0 x^{(s)} \in\Delta_{n_0} x ( s ) ∈ Δ n 0 . Then the expected maximum calibration error satisfies
E [ max t ≤ T ∥ Z t ∥ ] ≤ 18 S T . (21) \mathbb{E}\left[\max_{t \le T}\|Z_t\|\right] \le18\sqrt{ST}. \tag*{(21)} E [ t ≤ T max ∥ Z t ∥ ] ≤ 18 S T . (21)
Proof. For each interval I s = { t s − 1 + 1 , … , t s } \mathcal{I}_s = \{t_{s-1}+1,\ldots,t_s\} I s = { t s − 1 + 1 , … , t s } , let f ⋆ ( s ) ∈ Δ n ^ 0 f^{\star(s)} \in\Delta_{\hat{n}_0} f ⋆ ( s ) ∈ Δ n ^ 0 be the distribution of indices k t k_t k t of U C \mathcal{UC} U C induced by i t ∼ x ( s ) i_t \sim x^{(s)} i t ∼ x ( s ) , i.e., what would be the “correct” forecast. Also let f ^ t − 1 ∈ Δ n ^ 0 \hat{f}_{t-1} \in\Delta_{\hat{n}_0} f ^ t − 1 ∈ Δ n ^ 0 be the empirical distribution of environments of rounds I s ∩ [ t − 1 ] \mathcal{I}_s \cap[t-1] I s ∩ [ t − 1 ] (for the first round in an interval define empirical distribution as e 1 e_1 e 1 ); even though f ^ t − 1 \hat{f}_{t-1} f ^ t − 1 is not a forecast, it is convenient to see it as one, since it is the optimal prediction with the samples available, assuming knowledge of when interval started.
Define δ ˉ t ( k ) : = E f ∼ F t [ δ ( f , k ) ] \bar{\delta}_t(k) := \mathbb{E}_{f \sim F_t}[\delta(f,k)] δ ˉ t ( k ) := E f ∼ F t [ δ ( f , k )] to be the expected calibration error of outcome k k k in round t t t . We decompose the potential drift of an active round t t t :
⟨ Z t − 1 , δ ˉ t ( k t ) ⟩ = ⟨ Z t − 1 , δ ˉ t ( k t ) − ∑ k ∈ [ n ^ 0 ] f k ⋆ ( s ) δ ˉ t ( k ) ⟩ + ⟨ Z t − 1 , ∑ k ∈ [ n ^ 0 ] ( f k ⋆ ( s ) − f ^ t − 1 , k ) δ ˉ t ( k ) ⟩ + ∑ k ∈ [ n ^ 0 ] f ^ t − 1 , k ⟨ Z t − 1 , δ ˉ t ( k ) ⟩ . (22) \begin{aligned}
\langle Z_{t-1},\bar{\delta}_t(k_t)\rangle
={}&\left\langle Z_{t-1},\bar{\delta}_t(k_t)-\sum_{k\in[\hat{n}_0]}f_k^{\star(s)}\bar{\delta}_t(k)\right\rangle\\
&+\left\langle Z_{t-1},\sum_{k\in[\hat{n}_0]}\left(f_k^{\star(s)}-\hat{f}_{t-1,k}\right)\bar{\delta}_t(k)\right\rangle\\
&+\sum_{k\in[\hat{n}_0]}\hat{f}_{t-1,k}\langle Z_{t-1},\bar{\delta}_t(k)\rangle. \tag*{(22)}
\end{aligned} ⟨ Z t − 1 , δ ˉ t ( k t )⟩ = ⟨ Z t − 1 , δ ˉ t ( k t ) − k ∈ [ n ^ 0 ] ∑ f k ⋆ ( s ) δ ˉ t ( k ) ⟩ + ⟨ Z t − 1 , k ∈ [ n ^ 0 ] ∑ ( f k ⋆ ( s ) − f ^ t − 1 , k ) δ ˉ t ( k ) ⟩ + k ∈ [ n ^ 0 ] ∑ f ^ t − 1 , k ⟨ Z t − 1 , δ ˉ t ( k )⟩ . (22)
The error in the first line corresponds to the difference in the optimizer’s realized behavior and expected behavior, the second to the difference in the optimizer’s expected behavior and empirical one, observed so far from rounds in this interval, and the third line to the calibration error with respect to the empirical behavior of the optimizer of each interval. We proceed to bound each term separately.
Claim 6.4. For the first term in Eq. (22) it holds
E [ max t ≤ T ∑ τ ∈ I t ∣ ⟨ Z τ − 1 , δ ˉ τ ( k τ ) − ∑ k ∈ [ n ^ 0 ] f k ⋆ ( s ) δ ˉ τ ( k ) ⟩ ∣ ] ≤ 2 5 T E [ max t ≤ T ∥ Z t ∥ 2 ] . \mathbb{E}\left[\max_{t\le T}\sum_{\tau\in\mathcal{I}_t}\left|\left\langle Z_{\tau-1},\bar{\delta}_\tau(k_\tau)-\sum_{k\in[\hat{n}_0]}f_k^{\star(s)}\bar{\delta}_\tau(k)\right\rangle\right|\right]\le2\sqrt{5T}\sqrt{\mathbb{E}\left[\max_{t\le T}\|Z_t\|^2\right]}. E t ≤ T max τ ∈ I t ∑ ⟨ Z τ − 1 , δ ˉ τ ( k τ ) − k ∈ [ n ^ 0 ] ∑ f k ⋆ ( s ) δ ˉ τ ( k ) ⟩ ≤ 2 5 T E [ t ≤ T max ∥ Z t ∥ 2 ] .
The proof of the claim is pretty straightforward: the right part of the inner product has expectation 0 so the sum is a martingale and we can bound it using Doob’s L 2 L^2 L 2 maximal inequality.
Proof of Claim 6.4. Define the sequences
ξ t : = 1 t ∈ T ⟨ Z t − 1 , δ ˉ t ( k t ) − ∑ k ∈ [ n ^ O ] f k ⋆ ( s ) δ ˉ t ( k ) ⟩ and Ξ t : = ∑ τ ≤ t ξ τ . \xi_t := \mathbb{1}_{t\in T}\left\langle Z_{t-1},\bar{\delta}_t(k_t)-\sum_{k\in[\hat n_O]}f_k^{\star(s)}\bar{\delta}_t(k)\right\rangle
\qquad\text{and}\qquad
\Xi_t:=\sum_{\tau\le t}\xi_\tau. ξ t := 1 t ∈ T ⟨ Z t − 1 , δ ˉ t ( k t ) − k ∈ [ n ^ O ] ∑ f k ⋆ ( s ) δ ˉ t ( k ) ⟩ and Ξ t := τ ≤ t ∑ ξ τ .
Conditioned on the history H t − 1 \mathcal{H}_{t-1} H t − 1 up to round t − 1 t-1 t − 1 , the quantities Z t − 1 Z_{t-1} Z t − 1 , 1 t ∈ T \mathbb{1}_{t\in T} 1 t ∈ T , and F t F_t F t (hence δ ˉ t \bar{\delta}_t δ ˉ t ) are fixed, while k t ∼ f ⋆ ( s ) k_t\sim f^{\star(s)} k t ∼ f ⋆ ( s ) is drawn independently. Thus E [ ξ t ∣ H t − 1 ] = 0 \mathbb{E}[\xi_t\mid\mathcal{H}_{t-1}]=0 E [ ξ t ∣ H t − 1 ] = 0 , making Ξ t \Xi_t Ξ t a martingale. Moreover, since the variance of a random variable is bounded by its second moment, Cauchy–Schwarz and (14) give
E [ ξ t 2 ∣ H t − 1 ] ≤ E k ∼ f ⋆ ( s ) [ ⟨ Z t − 1 , δ ˉ t ( k ) ⟩ 2 ] ≤ ∥ Z t − 1 ∥ 2 E k ∼ f ⋆ ( s ) [ ∥ δ ˉ t ( k ) ∥ 2 ] ≤ 5 ∥ Z t − 1 ∥ 2 . \mathbb{E}\left[\xi_t^2\mid\mathcal{H}_{t-1}\right]\le\mathbb{E}_{k\sim f^{\star(s)}}\left[\left\langle Z_{t-1},\bar{\delta}_t(k)\right\rangle^2\right]\le\lVert Z_{t-1}\rVert^2\mathbb{E}_{k\sim f^{\star(s)}}\left[\lVert\bar{\delta}_t(k)\rVert^2\right]\le5\lVert Z_{t-1}\rVert^2. E [ ξ t 2 ∣ H t − 1 ] ≤ E k ∼ f ⋆ ( s ) [ ⟨ Z t − 1 , δ ˉ t ( k ) ⟩ 2 ] ≤ ∥ Z t − 1 ∥ 2 E k ∼ f ⋆ ( s ) [ ∥ δ ˉ t ( k ) ∥ 2 ] ≤ 5 ∥ Z t − 1 ∥ 2 .
By Doob’s L 2 L^2 L 2 maximal inequality [32 Theorem 4.4.4] , Jensen’s inequality, and the orthogonality of martingale differences (E [ ξ t ξ t ′ ] = 0 \mathbb{E}[\xi_t\xi_{t'}]=0 E [ ξ t ξ t ′ ] = 0 for t ≠ t ′ t\ne t' t = t ′ ), we obtain
E [ max t ≤ T Ξ t ] ≤ 2 E [ Ξ T 2 ] = 2 ∑ t = 1 T E [ ξ t 2 ] ≤ 2 5 ∑ t = 1 T E [ ∥ Z t − 1 ∥ 2 ] ≤ 2 5 T E [ max t ≤ T ∥ Z t ∥ 2 ] . \mathbb{E}\left[\max_{t\le T}\Xi_t\right]\le2\sqrt{\mathbb{E}\left[\Xi_T^2\right]}=2\sqrt{\sum_{t=1}^{T}\mathbb{E}\left[\xi_t^2\right]}\le2\sqrt{5\sum_{t=1}^{T}\mathbb{E}\left[\lVert Z_{t-1}\rVert^2\right]}\le2\sqrt{5T}\sqrt{\mathbb{E}\left[\max_{t\le T}\lVert Z_t\rVert^2\right]}. E [ t ≤ T max Ξ t ] ≤ 2 E [ Ξ T 2 ] = 2 t = 1 ∑ T E [ ξ t 2 ] ≤ 2 5 t = 1 ∑ T E [ ∥ Z t − 1 ∥ 2 ] ≤ 2 5 T E [ t ≤ T max ∥ Z t ∥ 2 ] .
Claim 6.5. For the second term in Eq. (22) it holds
E [ max t ≤ T ∑ τ ∈ T t ⟨ Z τ − 1 , ∑ k ∈ [ n ^ O ] ( f k ⋆ ( s ) − f ^ τ − 1 , k ) δ ˉ τ ( k ) ⟩ ] ≤ 4 T E [ max t ≤ T ∥ Z t ∥ 2 ] . \mathbb{E}\left[\max_{t\le T}\sum_{\tau\in T_t}\left\langle Z_{\tau-1},\sum_{k\in[\hat n_O]}\left(f_k^{\star(s)}-\hat f_{\tau-1,k}\right)\bar{\delta}_\tau(k)\right\rangle\right]\le4\sqrt{T}\sqrt{\mathbb{E}\left[\max_{t\le T}\lVert Z_t\rVert^2\right]}. E t ≤ T max τ ∈ T t ∑ ⟨ Z τ − 1 , k ∈ [ n ^ O ] ∑ ( f k ⋆ ( s ) − f ^ τ − 1 , k ) δ ˉ τ ( k ) ⟩ ≤ 4 T E [ t ≤ T max ∥ Z t ∥ 2 ] .
This is the most complicated quantity to bound. On a high level, it is bounded because in expectation, f ^ τ − 1 \hat f_{\tau-1} f ^ τ − 1 is f ⋆ ( s ) f^{\star(s)} f ⋆ ( s ) for all but the first round of an interval. However, a naive bound would depend on the dimension of δ ˉ τ ( k ) \bar{\delta}_\tau(k) δ ˉ τ ( k ) , which can have n L 2 min { τ , n ^ O } n_L^2\min\{\tau,\hat n_O\} n L 2 min { τ , n ^ O } non-zero coordinates. To achieve the improved bound, we decompose the right part of the inner product into three separate errors: the variance between f ^ τ − 1 \hat f_{\tau-1} f ^ τ − 1 and f ⋆ ( s ) f^{\star(s)} f ⋆ ( s ) , along with the variance of the induced utilities and costs. Since f ^ τ − 1 \hat f_{\tau-1} f ^ τ − 1 and f ⋆ ( s ) f^{\star(s)} f ⋆ ( s ) are distributions, while the utilities and costs have dimension n L n_L n L , we can bound their variance via standard concentration of averages.
Proof of Claim 6.5. For each round t ∈ I s t\in I_s t ∈ I s , let Γ t : = ∑ k ∈ [ n ^ O ] ( f k ⋆ ( s ) − f ^ t − 1 , k ) δ ˉ t ( k ) \Gamma_t:=\sum_{k\in[\hat n_O]}(f_k^{\star(s)}-\hat f_{t-1,k})\bar{\delta}_t(k) Γ t := ∑ k ∈ [ n ^ O ] ( f k ⋆ ( s ) − f ^ t − 1 , k ) δ ˉ t ( k ) . Applying Cauchy–Schwarz first to the inner product ⟨ Z τ − 1 , Γ τ ⟩ ≤ ∥ Z τ − 1 ∥ ∥ Γ τ ∥ \langle Z_{\tau-1},\Gamma_\tau\rangle\le\lVert Z_{\tau-1}\rVert\lVert\Gamma_\tau\rVert ⟨ Z τ − 1 , Γ τ ⟩ ≤ ∥ Z τ − 1 ∥ ∥ Γ τ ∥ and then to the expectation E [ ∥ Z τ − 1 ∥ ∥ Γ τ ∥ ] ≤ E [ ∥ Z τ − 1 ∥ 2 ] E [ ∥ Γ τ ∥ 2 ] \mathbb{E}[\lVert Z_{\tau-1}\rVert\lVert\Gamma_\tau\rVert]\le\sqrt{\mathbb{E}[\lVert Z_{\tau-1}\rVert^2]}\sqrt{\mathbb{E}[\lVert\Gamma_\tau\rVert^2]} E [∥ Z τ − 1 ∥ ∥ Γ τ ∥] ≤ E [∥ Z τ − 1 ∥ 2 ] E [∥ Γ τ ∥ 2 ] yields
E [ max t ≤ T ∑ τ ∈ T t ⟨ Z τ − 1 , Γ τ ⟩ ] ≤ max t ≤ T ∑ τ = 1 t E [ ∥ Z τ − 1 ∥ ∥ Γ τ ∥ ] = ∑ t = 1 T E [ ∥ Z t − 1 ∥ ∥ Γ t ∥ ] ≤ ∑ t = 1 T E [ ∥ Z t − 1 ∥ 2 ] E [ ∥ Γ t ∥ 2 ] ≤ E [ max t ≤ T ∥ Z t ∥ 2 ] ∑ t = 1 T E [ ∥ Γ t ∥ 2 ] . (23) \begin{aligned}
\mathbb{E}\left[\max_{t\le T}\sum_{\tau\in T_t}\langle Z_{\tau-1},\Gamma_\tau\rangle\right]\le\max_{t\le T}\sum_{\tau=1}^{t}\mathbb{E}[\lVert Z_{\tau-1}\rVert\lVert\Gamma_\tau\rVert]=\sum_{t=1}^{T}\mathbb{E}[\lVert Z_{t-1}\rVert\lVert\Gamma_t\rVert] \\
&\le\sum_{t=1}^{T}\sqrt{\mathbb{E}\left[\lVert Z_{t-1}\rVert^2\right]}\sqrt{\mathbb{E}\left[\lVert\Gamma_t\rVert^2\right]}\le\sqrt{\mathbb{E}\left[\max_{t\le T}\lVert Z_t\rVert^2\right]}\sum_{t=1}^{T}\sqrt{\mathbb{E}\left[\lVert\Gamma_t\rVert^2\right]}. \tag*{(23)}
\end{aligned} E [ t ≤ T max τ ∈ T t ∑ ⟨ Z τ − 1 , Γ τ ⟩ ] ≤ t ≤ T max τ = 1 ∑ t E [∥ Z τ − 1 ∥ ∥ Γ τ ∥] = t = 1 ∑ T E [∥ Z t − 1 ∥ ∥ Γ t ∥] ≤ t = 1 ∑ T E [ ∥ Z t − 1 ∥ 2 ] E [ ∥ Γ t ∥ 2 ] ≤ E [ t ≤ T max ∥ Z t ∥ 2 ] t = 1 ∑ T E [ ∥ Γ t ∥ 2 ] . (23)
We now bound E [ ∥ Γ t ∥ 2 ] \mathbb{E}[\lVert\Gamma_t\rVert^2] E [∥ Γ t ∥ 2 ] for a fixed t ∈ I s t\in I_s t ∈ I s . Recall the definition of Γ t \Gamma_t Γ t and δ ˉ t ( k ) \bar{\delta}_t(k) δ ˉ t ( k ) inside it:
∥ Γ t ∥ 2 = ∥ ∑ k ∈ [ n ^ O ] ( f k ⋆ ( s ) − f ^ t − 1 , k ) E f ∼ F t [ δ ( f , k ) ] ∥ 2 ≤ E f ∼ F t [ ∥ ∑ k ∈ [ n ^ O ] ( f k ⋆ ( s ) − f ^ t − 1 , k ) δ ( f , k ) ∥ 2 ] . (24) \lVert\Gamma_t\rVert^2=\left\lVert\sum_{k\in[\hat n_O]}\left(f_k^{\star(s)}-\hat f_{t-1,k}\right)\mathbb{E}_{f\sim F_t}[\delta(f,k)]\right\rVert^2\le\mathbb{E}_{f\sim F_t}\left[\left\lVert\sum_{k\in[\hat n_O]}\left(f_k^{\star(s)}-\hat f_{t-1,k}\right)\delta(f,k)\right\rVert^2\right]. \tag*{(24)} ∥ Γ t ∥ 2 = k ∈ [ n ^ O ] ∑ ( f k ⋆ ( s ) − f ^ t − 1 , k ) E f ∼ F t [ δ ( f , k )] 2 ≤ E f ∼ F t k ∈ [ n ^ O ] ∑ ( f k ⋆ ( s ) − f ^ t − 1 , k ) δ ( f , k ) 2 . (24)
where the inequality uses triangle inequality of the norm ∥ ⋅ ∥ \lVert\cdot\rVert ∥ ⋅ ∥ and then Jensen’s inequality on the square function. Since f ⋆ ( s ) , f ^ t − 1 ∈ Δ n ^ O f^{\star(s)},\hat{f}_{t-1}\in\Delta_{\hat{n}_O} f ⋆ ( s ) , f ^ t − 1 ∈ Δ n ^ O are distributions, we have ∑ k ∈ [ n ^ O ] ( f k ⋆ ( s ) − f ^ t − 1 , k ) = 0 \sum_{k\in[\hat{n}_O]}(f_k^{\star(s)}-\hat{f}_{t-1,k})=0 ∑ k ∈ [ n ^ O ] ( f k ⋆ ( s ) − f ^ t − 1 , k ) = 0 ; this makes many terms of δ R ( f , k ) \delta^{R}(f,k) δ R ( f , k ) to cancel out. Let ε k = f k ⋆ ( s ) − f ^ t − 1 , k \varepsilon_k=f_k^{\star(s)}-\hat{f}_{t-1,k} ε k = f k ⋆ ( s ) − f ^ t − 1 , k ; for δ R ( f , k ) \delta^{R}(f,k) δ R ( f , k ) we have
∥ ∑ k ∈ [ n ^ O ] ε k δ R ( f , k ) ∥ L 2 2 = ∑ k ′ ∈ [ n ^ O ] ∥ ∑ k ∈ [ n ^ O ] ε k ( 1 k ′ = k − f k ′ ) 1 ≤ Q ( f ) ∥ L 2 2 = ∑ k ′ ∈ [ n ^ O ] ∥ ε k ′ 1 ≤ Q ( f ) ∥ L 2 2 = ( 1 + Q ( f ) ) ∑ k ′ ∈ [ n ^ O ] ε k ′ 2 ≤ 2 ∑ k ′ ∈ [ n ^ O ] ε k ′ 2 = 2 ∥ f ⋆ ( s ) − f ^ t − 1 ∥ 2 2 \begin{aligned}
\left\lVert\sum_{k\in[\hat{n}_O]}\varepsilon_k\delta^{R}(f,k)\right\rVert_{L^2}^{2}
&=\sum_{k'\in[\hat{n}_O]}\left\lVert\sum_{k\in[\hat{n}_O]}\varepsilon_k(\mathbf{1}_{k'=k}-f_{k'})\mathbf{1}_{\le Q(f)}\right\rVert_{L^2}^{2}\\
&=\sum_{k'\in[\hat{n}_O]}\left\lVert\varepsilon_{k'}\mathbf{1}_{\le Q(f)}\right\rVert_{L^2}^{2}\\
&=\left(1+Q(f)\right)\sum_{k'\in[\hat{n}_O]}\varepsilon_{k'}^{2}\\
&\le2\sum_{k'\in[\hat{n}_O]}\varepsilon_{k'}^{2}=2\left\lVert f^{\star(s)}-\hat{f}_{t-1}\right\rVert_{2}^{2}
\end{aligned} k ∈ [ n ^ O ] ∑ ε k δ R ( f , k ) L 2 2 = k ′ ∈ [ n ^ O ] ∑ k ∈ [ n ^ O ] ∑ ε k ( 1 k ′ = k − f k ′ ) 1 ≤ Q ( f ) L 2 2 = k ′ ∈ [ n ^ O ] ∑ ε k ′ 1 ≤ Q ( f ) L 2 2 = ( 1 + Q ( f ) ) k ′ ∈ [ n ^ O ] ∑ ε k ′ 2 ≤ 2 k ′ ∈ [ n ^ O ] ∑ ε k ′ 2 = 2 f ⋆ ( s ) − f ^ t − 1 2 2
Similarly, using the definition of δ L \delta^{L} δ L we get
∥ ∑ k ∈ [ n ^ O ] ε k δ L ( f , k ) ∥ 2 2 = 1 4 n L ( ∥ ∑ k ∈ [ n ^ O ] ε k ( u ( k ) − u ( f ) ) ∥ 2 2 + ∥ ∑ k ∈ [ n ^ O ] ε k ρ L λ ( f ) ( c ( k ) − c ( f ) ) ∥ 2 2 + ( ∑ k ∈ [ n ^ O ] ε k ( u ( k ) − u ( f ) ) T y ( f ) ) 2 + ( ∑ k ∈ [ n ^ O ] ε k ( c ( k ) − c ( f ) ) T y ( f ) ) 2 ) = 1 4 n L ( ∥ ∑ k ∈ [ n ^ O ] ε k u ( k ) ∥ 2 2 + ( ρ L λ ( f ) ) 2 ∥ ∑ k ∈ [ n ^ O ] ε k c ( k ) ∥ 2 2 + ( ∑ k ∈ [ n ^ O ] ε k u ( k ) T y ( f ) ) 2 + ( ∑ k ∈ [ n ^ O ] ε k c ( k ) T y ( f ) ) 2 ) ≤ 1 4 n L ( 2 ∥ ∑ k ∈ [ n ^ O ] ε k u ( k ) ∥ 2 2 + ( ( ρ L λ ( f ) ) 2 + 1 ) ∥ ∑ k ∈ [ n ^ O ] ε k c ( k ) ∥ 2 2 ) = 1 4 n L ( 2 ∥ u ( f ⋆ ( s ) ) − u ( f ^ t − 1 ) ∥ 2 2 + ( ( ρ L λ ( f ) ) 2 + 1 ) ∥ c ( f ⋆ ( s ) ) − c ( f ^ t − 1 ) ∥ 2 2 ) . \begin{aligned}
\left\lVert\sum_{k\in[\hat{n}_O]}\varepsilon_k\delta^{L}(f,k)\right\rVert_{2}^{2}
&=\frac{1}{4n_L}\left(\left\lVert\sum_{k\in[\hat{n}_O]}\varepsilon_k\left(u^{(k)}-u(f)\right)\right\rVert_{2}^{2}+\left\lVert\sum_{k\in[\hat{n}_O]}\varepsilon_k\rho_L\lambda(f)\left(c^{(k)}-c(f)\right)\right\rVert_{2}^{2}\right.\\
&\qquad\left.+\left(\sum_{k\in[\hat{n}_O]}\varepsilon_k\left(u^{(k)}-u(f)\right)^{\mathsf T}y(f)\right)^{2}+\left(\sum_{k\in[\hat{n}_O]}\varepsilon_k\left(c^{(k)}-c(f)\right)^{\mathsf T}y(f)\right)^{2}\right)\\
&=\frac{1}{4n_L}\left(\left\lVert\sum_{k\in[\hat{n}_O]}\varepsilon_ku^{(k)}\right\rVert_{2}^{2}+\left(\rho_L\lambda(f)\right)^{2}\left\lVert\sum_{k\in[\hat{n}_O]}\varepsilon_kc^{(k)}\right\rVert_{2}^{2}\right.\\
&\qquad\left.+\left(\sum_{k\in[\hat{n}_O]}\varepsilon_ku^{(k)\mathsf T}y(f)\right)^{2}+\left(\sum_{k\in[\hat{n}_O]}\varepsilon_kc^{(k)\mathsf T}y(f)\right)^{2}\right)\\
&\le\frac{1}{4n_L}\left(2\left\lVert\sum_{k\in[\hat{n}_O]}\varepsilon_ku^{(k)}\right\rVert_{2}^{2}+\left(\left(\rho_L\lambda(f)\right)^{2}+1\right)\left\lVert\sum_{k\in[\hat{n}_O]}\varepsilon_kc^{(k)}\right\rVert_{2}^{2}\right)\\
&=\frac{1}{4n_L}\left(2\left\lVert u(f^{\star(s)})-u(\hat{f}_{t-1})\right\rVert_{2}^{2}+\left(\left(\rho_L\lambda(f)\right)^{2}+1\right)\left\lVert c(f^{\star(s)})-c(\hat{f}_{t-1})\right\rVert_{2}^{2}\right).
\end{aligned} k ∈ [ n ^ O ] ∑ ε k δ L ( f , k ) 2 2 = 4 n L 1 k ∈ [ n ^ O ] ∑ ε k ( u ( k ) − u ( f ) ) 2 2 + k ∈ [ n ^ O ] ∑ ε k ρ L λ ( f ) ( c ( k ) − c ( f ) ) 2 2 + k ∈ [ n ^ O ] ∑ ε k ( u ( k ) − u ( f ) ) T y ( f ) 2 + k ∈ [ n ^ O ] ∑ ε k ( c ( k ) − c ( f ) ) T y ( f ) 2 = 4 n L 1 k ∈ [ n ^ O ] ∑ ε k u ( k ) 2 2 + ( ρ L λ ( f ) ) 2 k ∈ [ n ^ O ] ∑ ε k c ( k ) 2 2 + k ∈ [ n ^ O ] ∑ ε k u ( k ) T y ( f ) 2 + k ∈ [ n ^ O ] ∑ ε k c ( k ) T y ( f ) 2 ≤ 4 n L 1 2 k ∈ [ n ^ O ] ∑ ε k u ( k ) 2 2 + ( ( ρ L λ ( f ) ) 2 + 1 ) k ∈ [ n ^ O ] ∑ ε k c ( k ) 2 2 = 4 n L 1 ( 2 u ( f ⋆ ( s ) ) − u ( f ^ t − 1 ) 2 2 + ( ( ρ L λ ( f ) ) 2 + 1 ) c ( f ⋆ ( s ) ) − c ( f ^ t − 1 ) 2 2 ) .
where in the inequality we used Cauchy-Schwarz to bound the last two terms and then used ∥ y ( f ) ∥ 2 2 ≤ 1 \lVert y(f)\rVert_{2}^{2}\le1 ∥ y ( f ) ∥ 2 2 ≤ 1 . Also using λ ( f ) ≤ 1 / ρ L \lambda(f)\le1/\rho_L λ ( f ) ≤ 1/ ρ L and plugging in the above two bounds into Eq. (24) we get
∥ Γ t ∥ 2 2 ≤ 2 ∥ f ⋆ ( s ) − f ^ t − 1 ∥ 2 2 + 1 2 n L ∥ u ( f ⋆ ( s ) ) − u ( f ^ t − 1 ) ∥ 2 2 + 1 2 n L ∥ c ( f ⋆ ( s ) ) − c ( f ^ t − 1 ) ∥ 2 2 (25) \lVert\Gamma_t\rVert_{2}^{2}\le2\left\lVert f^{\star(s)}-\hat{f}_{t-1}\right\rVert_{2}^{2}+\frac{1}{2n_L}\left\lVert u(f^{\star(s)})-u(\hat{f}_{t-1})\right\rVert_{2}^{2}+\frac{1}{2n_L}\left\lVert c(f^{\star(s)})-c(\hat{f}_{t-1})\right\rVert_{2}^{2} \tag*{(25)} ∥ Γ t ∥ 2 2 ≤ 2 f ⋆ ( s ) − f ^ t − 1 2 2 + 2 n L 1 u ( f ⋆ ( s ) ) − u ( f ^ t − 1 ) 2 2 + 2 n L 1 c ( f ⋆ ( s ) ) − c ( f ^ t − 1 ) 2 2 (25)
We now prove that the above quantities are small in expectation via standard concentration of the empirical average of a quantity, i.e., E [ f ^ t − 1 ] = f ⋆ ( s ) \mathbb{E}\left[\hat{f}_{t-1}\right]=f^{\star(s)} E [ f ^ t − 1 ] = f ⋆ ( s ) , E [ u ( f ^ t − 1 ) ] = u ( f ⋆ ( s ) ) \mathbb{E}\left[u(\hat{f}_{t-1})\right]=u(f^{\star(s)}) E [ u ( f ^ t − 1 ) ] = u ( f ⋆ ( s ) ) , and E [ c ( f ^ t − 1 ) ] = c ( f ⋆ ( s ) ) \mathbb{E}\left[c(\hat{f}_{t-1})\right]=c(f^{\star(s)}) E [ c ( f ^ t − 1 ) ] = c ( f ⋆ ( s ) ) (recall u ( f ) u(f) u ( f ) and c ( f ) c(f) c ( f ) are linear in the mass of f f f ). Consider an interval I s I_s I s . In the first round t t t of I s I_s I s , ∥ Γ t ∥ 2 2 ≤ 5 \lVert\Gamma_t\rVert_2^2\le5 ∥ Γ t ∥ 2 2 ≤ 5 by the boundedness of each of the norms. For every subsequent round t t t where L L L rounds of interval I s I_s I s have elapsed we have
E [ ∥ f ⋆ ( s ) − f ^ t − 1 ∥ 2 2 ] = E [ ∥ 1 L ∑ τ ∈ I s ∩ [ t − 1 ] ( f ⋆ ( s ) − e k τ ) ∥ 2 2 ] ≤ 1 L . \mathbb{E}\left[\left\|f^\star(s)-\hat{f}_{t-1}\right\|_2^2\right]=\mathbb{E}\left[\left\|\frac{1}{L}\sum_{\tau\in\mathcal{I}_s\cap[t-1]}\left(f^\star(s)-e_{k_\tau}\right)\right\|_2^2\right]\leq\frac{1}{L}. E [ f ⋆ ( s ) − f ^ t − 1 2 2 ] = E L 1 τ ∈ I s ∩ [ t − 1 ] ∑ ( f ⋆ ( s ) − e k τ ) 2 2 ≤ L 1 .
E [ ∥ u ( f ⋆ ( s ) ) − u ( f ^ t − 1 ) ∥ 2 2 ] = E [ ∥ 1 L ∑ τ ∈ I s ∩ [ t − 1 ] ( u ( f ⋆ ( s ) ) − u ( e k τ ) ) ∥ 2 2 ] ≤ n L L . \mathbb{E}\left[\left\|u\left(f^\star(s)\right)-u\left(\hat{f}_{t-1}\right)\right\|_2^2\right]=\mathbb{E}\left[\left\|\frac{1}{L}\sum_{\tau\in\mathcal{I}_s\cap[t-1]}\left(u\left(f^\star(s)\right)-u\left(e_{k_\tau}\right)\right)\right\|_2^2\right]\leq\frac{n_L}{L}. E [ u ( f ⋆ ( s ) ) − u ( f ^ t − 1 ) 2 2 ] = E L 1 τ ∈ I s ∩ [ t − 1 ] ∑ ( u ( f ⋆ ( s ) ) − u ( e k τ ) ) 2 2 ≤ L n L .
and similarly for the costs. Plugging this in Eq. (25) implies that E [ ∥ Γ t ∥ 2 ] ≤ 3 L \mathbb{E}\left[\left\|\Gamma_t\right\|^2\right]\leq\frac{3}{L} E [ ∥ Γ t ∥ 2 ] ≤ L 3 , which in turn yields
∑ t = 1 T E [ ∥ Γ t ∥ 2 ] ≤ ∑ s = 1 S ( 5 + ∑ L = 1 ∣ I s ∣ − 1 3 L ) ≤ ∑ s = 1 S 2 3 ∣ I s ∣ ≤ 4 S T . \sum_{t=1}^{T}\sqrt{\mathbb{E}\left[\left\|\Gamma_t\right\|^2\right]}\leq\sum_{s=1}^{S}\left(\sqrt{5}+\sum_{L=1}^{\lvert\mathcal{I}_s\rvert-1}\sqrt{\frac{3}{L}}\right)\leq\sum_{s=1}^{S}2\sqrt{3\lvert\mathcal{I}_s\rvert}\leq4\sqrt{ST}. t = 1 ∑ T E [ ∥ Γ t ∥ 2 ] ≤ s = 1 ∑ S 5 + L = 1 ∑ ∣ I s ∣ − 1 L 3 ≤ s = 1 ∑ S 2 3 ∣ I s ∣ ≤ 4 S T .
Plugging the above into Eq. (23) completes the proof of the claim. ■
Claim 6.6. For the third term in Eq. (22) it holds, in every active round t ∈ T t\in\mathcal{T} t ∈ T ,
∑ k ∈ [ n ^ 0 ] f ^ t − 1 , k ⟨ Z t − 1 , δ ^ t ( k ) ⟩ ≤ 1. \sum_{k\in[\hat{n}_0]}\hat{f}_{t-1,k}\left\langle Z_{t-1},\hat{\delta}_t(k)\right\rangle\leq1. k ∈ [ n ^ 0 ] ∑ f ^ t − 1 , k ⟨ Z t − 1 , δ ^ t ( k ) ⟩ ≤ 1.
The proof follows by Lemma 6.1 since f ^ t − 1 \hat{f}_{t-1} f ^ t − 1 is supported on environments seen prior to round t t t .
Proof of Claim 6.6. In any round t ∈ I s t\in\mathcal{I}_s t ∈ I s , every outcome k t ′ k_{t'} k t ′ from the preceding rounds t ′ ∈ I s ∩ [ t − 1 ] t'\in\mathcal{I}_s\cap[t-1] t ′ ∈ I s ∩ [ t − 1 ] has already been observed prior to round t t t (and for the first round of I s \mathcal{I}_s I s , f ^ t − 1 = e 1 \hat{f}_{t-1}=e_1 f ^ t − 1 = e 1 puts all mass on u C ( 1 ) = 0 ∈ U C 0 u\mathsf{C}^{(1)}=0\in\mathsf{UC}_0 u C ( 1 ) = 0 ∈ UC 0 ). Hence supp ( f ^ t − 1 ) ⊆ [ n ^ t − 1 ] \operatorname{supp}(\hat{f}_{t-1})\subseteq[\hat{n}_{t-1}] supp ( f ^ t − 1 ) ⊆ [ n ^ t − 1 ] . By Lemma 6.1, ⟨ Z t − 1 , δ ^ t ( k ) ⟩ = E f ∼ F t [ ⟨ Z t − 1 , δ ( f , k ) ⟩ ] ≤ 1 \left\langle Z_{t-1},\hat{\delta}_t(k)\right\rangle=\mathbb{E}_{f\sim\mathcal{F}_t}\left[\left\langle Z_{t-1},\delta(f,k)\right\rangle\right]\leq1 ⟨ Z t − 1 , δ ^ t ( k ) ⟩ = E f ∼ F t [ ⟨ Z t − 1 , δ ( f , k ) ⟩ ] ≤ 1 for every k ∈ [ n ^ t − 1 ] k\in[\hat{n}_{t-1}] k ∈ [ n ^ t − 1 ] , and taking the convex combination over k ∼ f ^ t − 1 k\sim\hat{f}_{t-1} k ∼ f ^ t − 1 proves the claim. ■
This concludes the proof of Lemma 6.3 via self-bounding inequality. Let B : = E [ max t ≤ T ∥ Z t ∥ 2 ] B:=\sqrt{\mathbb{E}\left[\max_{t\leq T}\left\|Z_t\right\|^2\right]} B := E [ max t ≤ T ∥ Z t ∥ 2 ] . Summing Eq. (16) over active rounds t ′ ∈ T t t'\in\mathcal{T}_t t ′ ∈ T t , then taking a maximum over t ∈ [ T ] t\in[T] t ∈ [ T ] , then expectation and then using Claim 6.4, Claim 6.5, and Claim 6.6 into Eq. (22) we get
B 2 = E [ max t ≤ T ∥ Z t ∥ 2 ] ≤ 4 T B + 8 S T B + 2 T + 5 T ≤ 17 S T B + 7 S T B^2=\mathbb{E}\left[\max_{t\leq T}\left\|Z_t\right\|^2\right]\leq4\sqrt{T}B+8\sqrt{ST}B+2T+5T\leq17\sqrt{ST}B+7ST B 2 = E [ t ≤ T max ∥ Z t ∥ 2 ] ≤ 4 T B + 8 S T B + 2 T + 5 T ≤ 17 S T B + 7 S T
Solving this quadratic inequality gives B ≤ 17 + 317 2 S T ≤ 18 S T B\leq\frac{17+\sqrt{317}}{2}\sqrt{ST}\leq18\sqrt{ST} B ≤ 2 17 + 317 S T ≤ 18 S T . The proof is concluded by Jensen’s inequality: E [ max t ≤ T ∥ Z t ∥ ] ≤ B ≤ 18 S T \mathbb{E}\left[\max_{t\leq T}\left\|Z_t\right\|\right]\leq B\leq18\sqrt{ST} E [ max t ≤ T ∥ Z t ∥ ] ≤ B ≤ 18 S T . ■
Putting Everything Together: Main Guarantees
Combining Lemmas 6.2 and 6.3 with Theorem 4.2, Lemma 5.1, and Theorem 5.2 yields our main theorem.
Theorem 6.7. Running Algorithm 1 with the forecaster of Algorithm 2 runs in time poly ( T , n L ) \operatorname{poly}(T,n_L) poly ( T , n L ) , never runs out of budget, and it guarantees:
Non-manipulability: Against any adaptive optimizer,
E [ ∑ t = 1 T U o [ i t , j t ] ] ≤ BSE ⋅ T + o ( ( n 0 n L + n L ρ L ) ( T + n ^ 0 ) ) . \mathbb{E}\left[\sum_{t=1}^{T}U_o[i_t,j_t]\right]\leq\operatorname{BSE}\cdot T+o\left(\left(\sqrt{n_0n_L}+\frac{\sqrt{n_L}}{\rho_L}\right)\left(\sqrt{T}+\hat{n}_0\right)\right). E [ t = 1 ∑ T U o [ i t , j t ] ] ≤ BSE ⋅ T + o ( ( n 0 n L + ρ L n L ) ( T + n ^ 0 ) ) .
Low interval stochastic regret: If the horizon [ T ] [T] [ T ] is partitioned into S ≥ 1 S \ge1 S ≥ 1 contiguous intervals I 1 , … , I S \mathcal{I}_1,\ldots,\mathcal{I}_S I 1 , … , I S in which the optimizer draws i t ∼ x ( s ) i_t \sim x^{(s)} i t ∼ x ( s ) independently in each round t ∈ I s t \in\mathcal{I}_s t ∈ I s for some x ( s ) ∈ Δ n O x^{(s)} \in\Delta_{n_O} x ( s ) ∈ Δ n O , then in every interval I s \mathcal{I}_s I s and for y s ∗ ∈ BR L ( x ( s ) ) y_s^* \in\operatorname{BR}_L(x^{(s)}) y s ∗ ∈ BR L ( x ( s ) ) ,
E [ ∑ t ∈ I s U L [ i t , j t ] ] ≥ ∣ I s ∣ ⋅ ( x ( s ) ) T U L y s ∗ − O ( S n L T ρ L ) . \mathbb{E}\left[\sum_{t\in\mathcal{I}_s} U_L[i_t,j_t]\right] \geq|\mathcal{I}_s| \cdot\left(x^{(s)}\right)^\mathsf{T} U_L y_s^* - O\left(\frac{\sqrt{S n_L T}}{\rho_L}\right). E [ t ∈ I s ∑ U L [ i t , j t ] ] ≥ ∣ I s ∣ ⋅ ( x ( s ) ) T U L y s ∗ − O ( ρ L S n L T ) .
Proof. In any active round t ∈ T t \in\mathcal{T} t ∈ T , the historical spending up to round t − 1 t-1 t − 1 is at most t ρ L − 1 t\rho_L-1 t ρ L − 1 and the round-t t t cost is at most 1, so total spending up to round t t t is at most t ρ L t\rho_L t ρ L ; in non-active rounds t ∉ T t \notin\mathcal{T} t ∈ / T , the null action has cost 0, preserving the budget bound by induction.
Against an arbitrary adaptive optimizer, Lemma 6.2 guarantees ∥ R T ∥ L 2 ≤ 7 T + 5 n ^ O \lVert R_T\rVert_{L^2} \leq\sqrt{7T}+\sqrt{5}\hat n_O ∥ R T ∥ L 2 ≤ 7 T + 5 n ^ O , along with max t ≤ T L t c y ≤ max t ≤ T ∥ L t ∥ 2 ≤ 7 T + 5 n ^ O \max_{t\leq T} L_t^{cy} \leq\max_{t\leq T}\lVert L_t\rVert_2 \leq\sqrt{7T}+\sqrt{5}\hat n_O max t ≤ T L t cy ≤ max t ≤ T ∥ L t ∥ 2 ≤ 7 T + 5 n ^ O on every path. Plugging this into Lemma 5.1 bounds the expected number of non-active rounds by
E [ T − ∣ T ∣ ] ≤ 2 n L ( 7 T + 5 n ^ O ) + 2 T + 1 ρ L = O ( 1 ρ L n L ( T + n ^ O ) ) . \mathbb{E}\left[T-|\mathcal{T}|\right]\leq\frac{2\sqrt{n_L}\left(\sqrt{7T}+\sqrt{5}\hat n_O\right)+2\sqrt{T}+1}{\rho_L}
=O\left(\frac{1}{\rho_L}\sqrt{n_L}\left(\sqrt{T}+\hat n_O\right)\right). E [ T − ∣ T ∣ ] ≤ ρ L 2 n L ( 7 T + 5 n ^ O ) + 2 T + 1 = O ( ρ L 1 n L ( T + n ^ O ) ) .
Since the optimizer’s utility in each non-active round is at most 1 and B S E ⋅ ∣ T ∣ ≤ B S E ⋅ T \mathrm{BSE}\cdot|\mathcal{T}|\leq\mathrm{BSE}\cdot T BSE ⋅ ∣ T ∣ ≤ BSE ⋅ T , combining Theorem 4.2 with ∥ R T ∥ L 2 ≤ 7 T + 5 n ^ O \lVert R_T\rVert_{L^2}\leq\sqrt{7T}+\sqrt{5}\hat n_O ∥ R T ∥ L 2 ≤ 7 T + 5 n ^ O and the bound on E [ T − ∣ T ∣ ] \mathbb{E}[T-|\mathcal{T}|] E [ T − ∣ T ∣ ] establishes the non-manipulability guarantee.
In the piecewise-stationary stochastic setting, Lemma 6.3 gives E [ max t ≤ T ∥ L t ∥ 2 ] ≤ E [ max t ≤ T ∥ Z t ∥ ] ≤ 18 S T \mathbb{E}[\max_{t\leq T}\lVert L_t\rVert_2]\leq\mathbb{E}[\max_{t\leq T}\lVert Z_t\rVert]\leq18\sqrt{ST} E [ max t ≤ T ∥ L t ∥ 2 ] ≤ E [ max t ≤ T ∥ Z t ∥] ≤ 18 S T , independent of n ^ O \hat n_O n ^ O . Substituting this bound directly into Theorem 5.2 for each interval I s \mathcal{I}_s I s gives the interval stochastic regret guarantee.
Finally, for the running time, since n ^ t − 1 ≤ t ≤ T \hat n_{t-1}\leq t\leq T n ^ t − 1 ≤ t ≤ T , Algorithm 2 performs M t = 50 n ^ t − 1 T 2 ≤ 50 T 3 M_t=50\hat n_{t-1}T^2\leq50T^3 M t = 50 n ^ t − 1 T 2 ≤ 50 T 3 iterations per active round. In each iteration m m m , finding y ( f ( m ) ) y(f^{(m)}) y ( f ( m ) ) and λ ( f ( m ) ) \lambda(f^{(m)}) λ ( f ( m ) ) solves a linear program in n L n_L n L variables, and computing z ( f ( m ) ) z(f^{(m)}) z ( f ( m ) ) evaluates finite step-function inner products via (6) together with a Euclidean inner product in R 2 n L + 2 \mathbb{R}^{2n_L+2} R 2 n L + 2 , taking poly ( T , n L ) \operatorname{poly}(T,n_L) poly ( T , n L ) time in total (independent of n O n_O n O and n ^ O \hat n_O n ^ O ). ■
Budgeted Optimizer and the Value the Optimizer Can Get
In this section we first show how our non-manipulability results extend to the case when the optimizer is also budgeted. In Sections 7.1 and 7.3 we will define the strong and weak versions of BSE, and show that the optimizer can achieve the value of the weak BSE against any learner with low interval stochastic regret. In Section 7.4 we show that when utility and cost values are generic the different definitions of BSE are equal.
Recall from § 2 that the goal of the optimizer is to maximize their utility subject to keeping their cost bounded, ρ O T \rho_O T ρ O T for some ρ O ∈ ( 0 , 1 ] \rho_O \in(0,1] ρ O ∈ ( 0 , 1 ] . We assume that the optimizer also has a null action that guarantees 0 cost no matter what the learner is doing.
Budgeted Stackelberg Equilibrium for a Budgeted Optimizer
In Optimization Program (2) we defined Budgeted Stackelberg Equilibrium (BSE) for the case when the optimizer is unbudgeted. In [36 ] we introduced the Budgeted Stackelberg Equilibrium (BSE) more generally. When the optimizer is also budgeted, it is important to note that the utility of the optimizer from the best response of the learner is not a concave function of their cost. As a result, a strategic optimizer may want to use multiple strategies, maybe overspending for a while, and then underspending, effectively ironing their utility function. To define this formally, we use the same best response function as was used in Optimization Program (1) .
As shown in [36 ] the optimizer mixing two distributions is enough to achieve their maximum value, i.e., three or more distributions does not help. Allowing the optimizer to use two different action distributions in two distinct phases, makes the BSE here equal to
BSE = max x 1 , x 2 ∈ Δ n O , y 1 , y 2 ∈ Δ n L , q ∈ [ 0 , 1 ] q ⋅ x 1 T U O y 1 + ( 1 − q ) ⋅ x 2 T U O y 2 \operatorname{BSE} = \max_{\substack{x_1,x_2\in\Delta_{n_O},\ y_1,y_2\in\Delta_{n_L},\ q\in[0,1]}} q \cdot x_1^{\mathsf{T}}U_Oy_1 + (1-q)\cdot x_2^{\mathsf{T}}U_Oy_2 BSE = x 1 , x 2 ∈ Δ n O , y 1 , y 2 ∈ Δ n L , q ∈ [ 0 , 1 ] max q ⋅ x 1 T U O y 1 + ( 1 − q ) ⋅ x 2 T U O y 2
such that q ⋅ x 1 T C O y 1 + ( 1 − q ) ⋅ x 2 T C O y 2 ≤ ρ O \text{such that}\quad q\cdot x_1^{\mathsf{T}}C_Oy_1 + (1-q)\cdot x_2^{\mathsf{T}}C_Oy_2 \le\rho_O such that q ⋅ x 1 T C O y 1 + ( 1 − q ) ⋅ x 2 T C O y 2 ≤ ρ O
and y 1 ∈ BR L ( x 1 ) , y 2 ∈ BR L ( x 2 ) (26) \text{and}\quad y_1\in\operatorname{BR}_L(x_1),\ y_2\in\operatorname{BR}_L(x_2) \tag*{(26)} and y 1 ∈ BR L ( x 1 ) , y 2 ∈ BR L ( x 2 ) (26)
Relaxing the Optimizer to an Unbudgeted Optimizer
Here we see how we can replace the budget-constraint optimizer with an optimizer who does not have a budget constraint, and extend our results from Section 6.2 that upper bound the optimizer’s utility to budgeted optimizers also.
Consider the Lagrangian relaxation of the optimizer who is a pure utility maximizer and who receives utility U O [ i , j ] − μ C O [ i , j ] + μ ρ O U_O[i,j]-\mu C_O[i,j]+\mu\rho_O U O [ i , j ] − μ C O [ i , j ] + μ ρ O when the pure actions are i , j i,j i , j , for a Lagrange multiplier μ ≥ 0 \mu\ge0 μ ≥ 0 . The BSE of this optimizer is
BSE μ = max x ∈ Δ n O , y ∈ Δ n L x T U O y − μ x T C O y + μ ρ O \operatorname{BSE}_{\mu}=\max_{\substack{x\in\Delta_{n_O},\ y\in\Delta_{n_L}}} x^{\mathsf{T}}U_Oy-\mu x^{\mathsf{T}}C_Oy+\mu\rho_O BSE μ = x ∈ Δ n O , y ∈ Δ n L max x T U O y − μ x T C O y + μ ρ O
such that y ∈ BR L ( x ) (27) \text{such that}\quad y\in\operatorname{BR}_L(x) \tag*{(27)} such that y ∈ BR L ( x ) (27)
We notice that for the right μ \mu μ , the two BSEs are equal.
Lemma 7.1. It holds BSE = min μ ≥ 0 BSE μ \operatorname{BSE}=\min_{\mu\ge0}\operatorname{BSE}_{\mu} BSE = min μ ≥ 0 BSE μ . In addition, for the optimal value it holds μ ⋆ ≤ 1 / ρ O \mu^\star\le1/\rho_O μ ⋆ ≤ 1/ ρ O .
Proof. Let S = { ( x T U O y , x T C O y ) : x ∈ Δ n O , y ∈ BR L ( x ) } ⊆ [ 0 , 1 ] 2 S=\{(x^{\mathsf{T}}U_Oy,x^{\mathsf{T}}C_Oy):x\in\Delta_{n_O},\ y\in\operatorname{BR}_L(x)\}\subseteq[0,1]^2 S = {( x T U O y , x T C O y ) : x ∈ Δ n O , y ∈ BR L ( x )} ⊆ [ 0 , 1 ] 2 be the set of achievable cost-utility pairs for the optimizer under a single mixed strategy, and let S ˉ = conv ( S ) \bar{S}=\operatorname{conv}(S) S ˉ = conv ( S ) denote its convex hull. We can re-write Optimization Program (26) as
BSE = max ( u , c ) ∈ S ˉ c ≤ ρ O u \operatorname{BSE}=\max_{\substack{(u,c)\in\bar{S}\\c\le\rho_O}}u BSE = ( u , c ) ∈ S ˉ c ≤ ρ O max u
since any extreme point of S ˉ \bar{S} S ˉ can be written as a convex combination of two points in S S S (see [36 ] for a more detailed proof).
The optimizer’s null action yields a point ( u 0 , 0 ) ∈ S (u_0,0)\in S ( u 0 , 0 ) ∈ S with cost c = 0 < ρ O c=0<\rho_O c = 0 < ρ O and utility u 0 ≥ 0 u_0\ge0 u 0 ≥ 0 . This makes Slater’s condition true; by strong Lagrangian duality for the constraint c ≤ ρ O c\le\rho_O c ≤ ρ O with multiplier μ ≥ 0 \mu\ge0 μ ≥ 0 , the dual minimum is attained and
BSE = max ( u , c ) ∈ S ˉ min μ ≥ 0 ( u − μ c + μ ρ O ) = min μ ≥ 0 max ( u , c ) ∈ S ˉ ( u − μ c + μ ρ O ) = min μ ≥ 0 max ( u , c ) ∈ S ( u − μ c + μ ρ O ) = min μ ≥ 0 BSE μ , \operatorname{BSE}=\max_{(u,c)\in\bar{S}}\min_{\mu\ge0}(u-\mu c+\mu\rho_O)=\min_{\mu\ge0}\max_{(u,c)\in\bar{S}}(u-\mu c+\mu\rho_O)=\min_{\mu\ge0}\max_{(u,c)\in S}(u-\mu c+\mu\rho_O)=\min_{\mu\ge0}\operatorname{BSE}_{\mu}, BSE = ( u , c ) ∈ S ˉ max μ ≥ 0 min ( u − μ c + μ ρ O ) = μ ≥ 0 min ( u , c ) ∈ S ˉ max ( u − μ c + μ ρ O ) = μ ≥ 0 min ( u , c ) ∈ S max ( u − μ c + μ ρ O ) = μ ≥ 0 min BSE μ ,
where the third equality holds because the linear in ( u , c ) (u,c) ( u , c ) objective u − μ c + μ ρ O u-\mu c+\mu\rho_O u − μ c + μ ρ O achieves its supremum over S ˉ \bar{S} S ˉ on S S S itself. Finally, evaluating BSE μ \operatorname{BSE}_{\mu} BSE μ at ( u 0 , 0 ) ∈ S (u_0,0)\in S ( u 0 , 0 ) ∈ S gives BSE μ ≥ u 0 + μ ρ O ≥ μ ρ O \operatorname{BSE}_{\mu}\ge u_0+\mu\rho_O\ge\mu\rho_O BSE μ ≥ u 0 + μ ρ O ≥ μ ρ O ; since BSE ≤ 1 \operatorname{BSE}\le1 BSE ≤ 1 , any minimizer μ ⋆ \mu^\star μ ⋆ satisfies μ ⋆ ≤ BSE / ρ O ≤ 1 / ρ O \mu^\star\le\operatorname{BSE}/\rho_O\le1/\rho_O μ ⋆ ≤ BSE / ρ O ≤ 1/ ρ O . ■
This allows us to focus on the Lagrangian optimizer instead of the budgeted one for utility upper bounds. Specifically, the budgeted optimizer’s benefit over its BSE is bounded by the benefit of the Lagrangian optimizer’s over its BSE. Consider any sequence of pairs of actions ( i 1 , j 1 ) , … , ( i T , j T ) (i_1,j_1),\ldots,(i_T,j_T) ( i 1 , j 1 ) , … , ( i T , j T ) such that the optimizer’s budget is observed. Then,
∑ t = 1 T U O [ i t , j t ] − T ⋅ B S E ≤ ∑ t = 1 T ( U O [ i t , j t ] − μ ⋆ C O [ i t , j t ] + μ ⋆ ρ O ) − T ⋅ B S E μ ⋆ . \sum_{t=1}^{T} U_{\mathrm{O}}[i_t,j_t]-T\cdot\mathrm{BSE}\leq\sum_{t=1}^{T}\left(U_{\mathrm{O}}[i_t,j_t]-\mu^\star C_{\mathrm{O}}[i_t,j_t]+\mu^\star\rho_{\mathrm{O}}\right)-T\cdot\mathrm{BSE}_{\mu^\star}. t = 1 ∑ T U O [ i t , j t ] − T ⋅ BSE ≤ t = 1 ∑ T ( U O [ i t , j t ] − μ ⋆ C O [ i t , j t ] + μ ⋆ ρ O ) − T ⋅ BSE μ ⋆ .
Note that the weaker version B S E ≤ min μ ≥ 0 B S E μ \mathrm{BSE}\leq\min_{\mu\geq0}\mathrm{BSE}_{\mu} BSE ≤ min μ ≥ 0 BSE μ that can be trivially proved via weak duality is not enough to prove the above inequality. This allows us to extend our main result to the case when the optimizer is also budgeted, with an increase in the error bound since the Lagrangian optimizer’s rewards have range O ( 1 / ρ O ) O(1/\rho_{\mathrm{O}}) O ( 1/ ρ O ) instead of 1:
Theorem 7.2. Running Algorithm 1 with the forecaster of Algorithm 2 ensures the guarantees of Theorem 6.7 also against a budgeted optimizer with an additional factor of 1 / ρ O 1/\rho_{\mathrm{O}} 1/ ρ O in the error term of the non-manipulability guarantee.
Achieving the Weak Budgeted Stackelberg Equilibrium
We now show that against any learning algorithm with low per-interval stochastic regret and low per-interval overspending (which Algorithms 1 and 2 satisfy via Lemma 5.1 and Theorems 5.2 and 6.7), these two generic properties are sufficient to guarantee that the optimizer can always achieve the value of the Weak Budgeted Stackelberg Equilibrium (WBSE), defined by replacing optimistic learner tie-breaking in (26) with pessimistic tie-breaking over B R L ( x ) \mathrm{BR}_{\mathrm{L}}(x) BR L ( x ) :
W B S E = sup x 1 , x 2 ∈ Δ n O , q ∈ [ 0 , 1 ] q min y 1 ∈ B R L ( x 1 ) x 1 T U O y 1 + ( 1 − q ) min y 2 ∈ B R L ( x 2 ) x 2 T U O y 2 \mathrm{WBSE}=\sup_{\substack{x_1,x_2\in\Delta_{n_{\mathrm{O}}},\ q\in[0,1]}}q\min_{y_1\in\mathrm{BR}_{\mathrm{L}}(x_1)}x_1^{\mathsf{T}}U_{\mathrm{O}}y_1+(1-q)\min_{y_2\in\mathrm{BR}_{\mathrm{L}}(x_2)}x_2^{\mathsf{T}}U_{\mathrm{O}}y_2 WBSE = x 1 , x 2 ∈ Δ n O , q ∈ [ 0 , 1 ] sup q y 1 ∈ BR L ( x 1 ) min x 1 T U O y 1 + ( 1 − q ) y 2 ∈ BR L ( x 2 ) min x 2 T U O y 2
such that q max y 1 ∈ B R L ( x 1 ) x 1 T C O y 1 + ( 1 − q ) max y 2 ∈ B R L ( x 2 ) x 2 T C O y 2 ≤ ρ O . (28) \text{such that}\qquad q\max_{y_1\in\mathrm{BR}_{\mathrm{L}}(x_1)}x_1^{\mathsf{T}}C_{\mathrm{O}}y_1+(1-q)\max_{y_2\in\mathrm{BR}_{\mathrm{L}}(x_2)}x_2^{\mathsf{T}}C_{\mathrm{O}}y_2\leq\rho_{\mathrm{O}}. \tag*{(28)} such that q y 1 ∈ BR L ( x 1 ) max x 1 T C O y 1 + ( 1 − q ) y 2 ∈ BR L ( x 2 ) max x 2 T C O y 2 ≤ ρ O . (28)
When the optimizer is unbudgeted (ρ O = 1 \rho_{\mathrm{O}}=1 ρ O = 1 ), the cost constraint is vacuous and it is not useful for the optimizer to have two phases, so (28) reduces to W B S E = sup x ∈ Δ n O min y ∈ B R L ( x ) x T U O y \mathrm{WBSE}=\sup_{x\in\Delta_{n_{\mathrm{O}}}}\min_{y\in\mathrm{BR}_{\mathrm{L}}(x)}x^{\mathsf{T}}U_{\mathrm{O}}y WBSE = sup x ∈ Δ n O min y ∈ BR L ( x ) x T U O y . Whenever the learner’s best response B R L ( x ) \mathrm{BR}_{\mathrm{L}}(x) BR L ( x ) is unique at the optimal strategies, WBSE coincides with BSE.
We first show that in any contiguous interval I ⊆ [ T ] \mathcal{I}\subseteq[T] I ⊆ [ T ] where the optimizer commits to a fixed mixed strategy x ∈ Δ n O x\in\Delta_{n_{\mathrm{O}}} x ∈ Δ n O , low stochastic regret and low overspending on I \mathcal{I} I force the learner’s expected empirical play to lie close to the best-response polytope B R L ( x ) \mathrm{BR}_{\mathrm{L}}(x) BR L ( x ) , simultaneously lower-bounding the optimizer’s utility and upper-bounding the optimizer’s cost on I \mathcal{I} I . For a fixed x ∈ Δ n O x\in\Delta_{n_{\mathrm{O}}} x ∈ Δ n O , let F ( x ) ≔ { y ∈ Δ n L : x T C L y ≤ ρ L } \mathcal{F}(x)\coloneqq\{y\in\Delta_{n_{\mathrm{L}}}:x^{\mathsf{T}}C_{\mathrm{L}}y\leq\rho_{\mathrm{L}}\} F ( x ) : = { y ∈ Δ n L : x T C L y ≤ ρ L } denote the learner’s budget-feasible polytope, V L ( x ) ≔ max y ∈ F ( x ) x T U L y V_{\mathrm{L}}(x)\coloneqq\max_{y\in\mathcal{F}(x)}x^{\mathsf{T}}U_{\mathrm{L}}y V L ( x ) : = max y ∈ F ( x ) x T U L y the learner’s optimal utility, V ( x ) \mathcal{V}(x) V ( x ) the finite set of vertices of F ( x ) \mathcal{F}(x) F ( x ) , and V s u b ( x ) ≔ V ( x ) ∖ B R L ( x ) \mathcal{V}^{\mathrm{sub}}(x)\coloneqq\mathcal{V}(x)\setminus\mathrm{BR}_{\mathrm{L}}(x) V sub ( x ) : = V ( x ) ∖ BR L ( x ) the suboptimal vertices of F ( x ) \mathcal{F}(x) F ( x ) . We denote the vertex suboptimality gap of F ( x ) \mathcal{F}(x) F ( x ) by
Δ ( x ) ≔ { min v ∈ V s u b ( x ) ( V L ( x ) − x T U L v ) > 0 if V s u b ( x ) ≠ ∅ , 1 if V s u b ( x ) = ∅ . (29) \Delta(x)\coloneqq\begin{cases}
\min_{v\in\mathcal{V}^{\mathrm{sub}}(x)}\left(V_{\mathrm{L}}(x)-x^{\mathsf{T}}U_{\mathrm{L}}v\right)>0 & \text{if }\mathcal{V}^{\mathrm{sub}}(x)\neq\varnothing,\\
1 & \text{if }\mathcal{V}^{\mathrm{sub}}(x)=\varnothing.
\end{cases} \tag*{(29)} Δ ( x ) : = { min v ∈ V sub ( x ) ( V L ( x ) − x T U L v ) > 0 1 if V sub ( x ) = ∅ , if V sub ( x ) = ∅ . (29)
Lemma 7.3. Let I = { t 1 + 1 , … , t 2 } ⊆ [ T ] \mathcal{I}=\{t_1+1,\ldots,t_2\}\subseteq[T] I = { t 1 + 1 , … , t 2 } ⊆ [ T ] be any contiguous interval in which the optimizer draws i t ∼ x i_t\sim x i t ∼ x independently in each round t ∈ I t\in\mathcal{I} t ∈ I for some x ∈ Δ n O x\in\Delta_{n_{\mathrm{O}}} x ∈ Δ n O . Then against any learning algorithm that on I \mathcal{I} I has expected stochastic regret ∣ I ∣ V L ( x ) − E [ ∑ t ∈ I U L [ i t , j t ] ] ≤ Reg I |\mathcal{I}|V_{\mathrm{L}}(x)-\mathbb{E}\left[\sum_{t\in\mathcal{I}}U_{\mathrm{L}}[i_t,j_t]\right]\leq\operatorname{Reg}_{\mathcal{I}} ∣ I ∣ V L ( x ) − E [ ∑ t ∈ I U L [ i t , j t ] ] ≤ Reg I and expected overspending E [ ∑ t ∈ I C L [ i t , j t ] ] − \mathbb{E}\left[\sum_{t\in\mathcal{I}}C_{\mathrm{L}}[i_t,j_t]\right]- E [ ∑ t ∈ I C L [ i t , j t ] ] −
ρ L ∣ I ∣ ≤ Over I \rho_L|\mathcal{I}| \le\operatorname{Over}_{\mathcal{I}} ρ L ∣ I ∣ ≤ Over I for some Reg I \operatorname{Reg}_{\mathcal{I}} Reg I , Over I ≥ 0 \operatorname{Over}_{\mathcal{I}} \ge0 Over I ≥ 0 , the optimizer’s expected utility and cost in I \mathcal{I} I satisfy
E [ ∑ t ∈ I U O [ i t , j t ] ] ≥ ∣ I ∣ min y ∈ BR L ( x ) x ⊤ U O y − Reg I + 2 Over I / ρ L Δ ( x ) \mathbb{E}\left[\sum_{t\in\mathcal{I}} U_O[i_t,j_t]\right] \ge|\mathcal{I}| \min_{y\in\operatorname{BR}_L(x)} x^\top U_Oy-\frac{\operatorname{Reg}_{\mathcal{I}}+2\operatorname{Over}_{\mathcal{I}}/\rho_L}{\Delta(x)} E [ t ∈ I ∑ U O [ i t , j t ] ] ≥ ∣ I ∣ y ∈ BR L ( x ) min x ⊤ U O y − Δ ( x ) Reg I + 2 Over I / ρ L
E [ ∑ t ∈ I C O [ i t , j t ] ] ≤ ∣ I ∣ max y ∈ BR L ( x ) x ⊤ C O y + Reg I + 2 Over I / ρ L Δ ( x ) \mathbb{E}\left[\sum_{t\in\mathcal{I}} C_O[i_t,j_t]\right] \le|\mathcal{I}| \max_{y\in\operatorname{BR}_L(x)} x^\top C_Oy+\frac{\operatorname{Reg}_{\mathcal{I}}+2\operatorname{Over}_{\mathcal{I}}/\rho_L}{\Delta(x)} E [ t ∈ I ∑ C O [ i t , j t ] ] ≤ ∣ I ∣ y ∈ BR L ( x ) max x ⊤ C O y + Δ ( x ) Reg I + 2 Over I / ρ L
Proof. Let y ˉ : = 1 ∣ I ∣ ∑ t ∈ I E [ e j t ] ∈ Δ n L \bar{y} := \frac{1}{|\mathcal{I}|}\sum_{t\in\mathcal{I}}\mathbb{E}[e_{j_t}] \in\Delta_{n_L} y ˉ := ∣ I ∣ 1 ∑ t ∈ I E [ e j t ] ∈ Δ n L be the learner’s expected empirical strategy on I \mathcal{I} I . Since i t ∼ x i_t\sim x i t ∼ x is drawn independently of j t j_t j t conditioned on the history prior to round t t t , linearity of expectation gives E [ ∑ t ∈ I M [ i t , j t ] ] = ∣ I ∣ x ⊤ M y ˉ \mathbb{E}\left[\sum_{t\in\mathcal{I}}M[i_t,j_t]\right]=|\mathcal{I}|x^\top M\bar{y} E [ ∑ t ∈ I M [ i t , j t ] ] = ∣ I ∣ x ⊤ M y ˉ for every matrix M ∈ { U O , C O , U L , C L } M\in\{U_O,C_O,U_L,C_L\} M ∈ { U O , C O , U L , C L } .
We now project y ˉ \bar{y} y ˉ onto the feasible polytope F ( x ) \mathcal{F}(x) F ( x ) . Let δ : = max { 0 , x ⊤ C L y ˉ − ρ L } \delta:=\max\{0,x^\top C_L\bar{y}-\rho_L\} δ := max { 0 , x ⊤ C L y ˉ − ρ L } and γ : = δ ρ L + δ ≤ δ ρ L \gamma:=\frac{\delta}{\rho_L+\delta}\le\frac{\delta}{\rho_L} γ := ρ L + δ δ ≤ ρ L δ . Since the learner’s null action j 0 j_0 j 0 satisfies x ⊤ C L e j 0 = 0 x^\top C_Le_{j_0}=0 x ⊤ C L e j 0 = 0 , the convex combination y ~ : = ( 1 − γ ) y ˉ + γ e j 0 ∈ Δ n L \tilde{y}:=(1-\gamma)\bar{y}+\gamma e_{j_0}\in\Delta_{n_L} y ~ := ( 1 − γ ) y ˉ + γ e j 0 ∈ Δ n L satisfies x ⊤ C L y ~ = ( 1 − γ ) x ⊤ C L y ˉ ≤ ( 1 − γ ) ( ρ L + δ ) = ρ L x^\top C_L\tilde{y}=(1-\gamma)x^\top C_L\bar{y}\le(1-\gamma)(\rho_L+\delta)=\rho_L x ⊤ C L y ~ = ( 1 − γ ) x ⊤ C L y ˉ ≤ ( 1 − γ ) ( ρ L + δ ) = ρ L , so y ~ ∈ F ( x ) \tilde{y}\in\mathcal{F}(x) y ~ ∈ F ( x ) . Moreover, since U L ∈ [ 0 , 1 ] n O × n L U_L\in[0,1]^{n_O\times n_L} U L ∈ [ 0 , 1 ] n O × n L , we have x ⊤ U L y ~ ≥ ( 1 − γ ) x ⊤ U L y ˉ ≥ x ⊤ U L y ˉ − γ x^\top U_L\tilde{y}\ge(1-\gamma)x^\top U_L\bar{y}\ge x^\top U_L\bar{y}-\gamma x ⊤ U L y ~ ≥ ( 1 − γ ) x ⊤ U L y ˉ ≥ x ⊤ U L y ˉ − γ .
Because F ( x ) = conv ( V ( x ) ) \mathcal{F}(x)=\operatorname{conv}(\mathcal{V}(x)) F ( x ) = conv ( V ( x )) and BR L ( x ) = conv ( V ( x ) ∖ V s u b ( x ) ) \operatorname{BR}_L(x)=\operatorname{conv}(\mathcal{V}(x)\setminus\mathcal{V}^{\mathrm{sub}}(x)) BR L ( x ) = conv ( V ( x ) ∖ V sub ( x )) , we can decompose y ~ = ( 1 − β ) y ⋆ + β y s u b \tilde{y}=(1-\beta)y^\star+\beta y^{\mathrm{sub}} y ~ = ( 1 − β ) y ⋆ + β y sub for some β ∈ [ 0 , 1 ] \beta\in[0,1] β ∈ [ 0 , 1 ] , y ⋆ ∈ BR L ( x ) y^\star\in\operatorname{BR}_L(x) y ⋆ ∈ BR L ( x ) , and y s u b ∈ conv ( V s u b ( x ) ) y^{\mathrm{sub}}\in\operatorname{conv}(\mathcal{V}^{\mathrm{sub}}(x)) y sub ∈ conv ( V sub ( x )) (with β = 0 \beta=0 β = 0 if V s u b ( x ) = ∅ \mathcal{V}^{\mathrm{sub}}(x)=\varnothing V sub ( x ) = ∅ ). By definition of Δ ( x ) \Delta(x) Δ ( x ) in (29) , every y s u b ∈ conv ( V s u b ( x ) ) y^{\mathrm{sub}}\in\operatorname{conv}(\mathcal{V}^{\mathrm{sub}}(x)) y sub ∈ conv ( V sub ( x )) satisfies x ⊤ U L y s u b ≤ V L ( x ) − Δ ( x ) x^\top U_Ly^{\mathrm{sub}}\le V_L(x)-\Delta(x) x ⊤ U L y sub ≤ V L ( x ) − Δ ( x ) , and hence
V L ( x ) − β Δ ( x ) ≥ ( 1 − β ) x ⊤ U L y ⋆ + β x ⊤ U L y s u b = x ⊤ U L y ~ ≥ x ⊤ U L y ˉ − γ . V_L(x)-\beta\Delta(x)\ge(1-\beta)x^\top U_Ly^\star+\beta x^\top U_Ly^{\mathrm{sub}}=x^\top U_L\tilde{y}\ge x^\top U_L\bar{y}-\gamma. V L ( x ) − β Δ ( x ) ≥ ( 1 − β ) x ⊤ U L y ⋆ + β x ⊤ U L y sub = x ⊤ U L y ~ ≥ x ⊤ U L y ˉ − γ .
Rearranging gives β ≤ 1 Δ ( x ) ( V L ( x ) − x ⊤ U L y ˉ + γ ) \beta\le\frac{1}{\Delta(x)}(V_L(x)-x^\top U_L\bar{y}+\gamma) β ≤ Δ ( x ) 1 ( V L ( x ) − x ⊤ U L y ˉ + γ ) . For any matrix M ∈ [ 0 , 1 ] n O × n L M\in[0,1]^{n_O\times n_L} M ∈ [ 0 , 1 ] n O × n L , since x ⊤ M ( y ˉ − y ~ ) = γ x ⊤ M ( y ˉ − e j 0 ) ∈ [ − γ , γ ] x^\top M(\bar{y}-\tilde{y})=\gamma x^\top M(\bar{y}-e_{j_0})\in[-\gamma,\gamma] x ⊤ M ( y ˉ − y ~ ) = γ x ⊤ M ( y ˉ − e j 0 ) ∈ [ − γ , γ ] and x ⊤ M ( y ~ − y ⋆ ) = β x ⊤ M ( y s u b − y ⋆ ) ∈ [ − β , β ] x^\top M(\tilde{y}-y^\star)=\beta x^\top M(y^{\mathrm{sub}}-y^\star)\in[-\beta,\beta] x ⊤ M ( y ~ − y ⋆ ) = β x ⊤ M ( y sub − y ⋆ ) ∈ [ − β , β ] , we obtain
∣ I ∣ ∣ x ⊤ M y ˉ − x ⊤ M y ⋆ ∣ ≤ ∣ I ∣ ( γ + β ) ≤ ∣ I ∣ Δ ( x ) ( V L ( x ) − x ⊤ U L y ˉ + 2 γ ) ≤ Reg I + 2 Over I / ρ L Δ ( x ) . |\mathcal{I}|\left|x^\top M\bar{y}-x^\top My^\star\right|\le|\mathcal{I}|(\gamma+\beta)\le\frac{|\mathcal{I}|}{\Delta(x)}(V_L(x)-x^\top U_L\bar{y}+2\gamma)\le\frac{\operatorname{Reg}_{\mathcal{I}}+2\operatorname{Over}_{\mathcal{I}}/\rho_L}{\Delta(x)}. ∣ I ∣ x ⊤ M y ˉ − x ⊤ M y ⋆ ≤ ∣ I ∣ ( γ + β ) ≤ Δ ( x ) ∣ I ∣ ( V L ( x ) − x ⊤ U L y ˉ + 2 γ ) ≤ Δ ( x ) Reg I + 2 Over I / ρ L .
∣ I ∣ ( V L ( x ) − x ⊤ U L y ˉ ) ≤ Reg I |\mathcal{I}|(V_L(x)-x^\top U_L\bar{y})\le\operatorname{Reg}_{\mathcal{I}} ∣ I ∣ ( V L ( x ) − x ⊤ U L y ˉ ) ≤ Reg I and ∣ I ∣ γ ≤ ∣ I ∣ δ / ρ L ≤ Over I / ρ L |\mathcal{I}|\gamma\le|\mathcal{I}|\delta/\rho_L\le\operatorname{Over}_{\mathcal{I}}/\rho_L ∣ I ∣ γ ≤ ∣ I ∣ δ / ρ L ≤ Over I / ρ L . Taking M = U O M=U_O M = U O and M = C O M=C_O M = C O and using y ⋆ ∈ BR L ( x ) y^\star\in\operatorname{BR}_L(x) y ⋆ ∈ BR L ( x ) completes the proof. ■
Applying Lemma 7.3 across S ≤ 2 S\le2 S ≤ 2 intervals shows that a budgeted optimizer can achieve W B S E ⋅ T − o ( T ) \mathrm{WBSE}\cdot T-o(T) WBSE ⋅ T − o ( T ) against any such learner (and in particular against Algorithms 1 and 2).
Theorem 7.4. Fix any feasible solution ( x 1 , x 2 , q ) (x_1,x_2,q) ( x 1 , x 2 , q ) to (28) and let Δ : = min { Δ ( x 1 ) , Δ ( x 2 ) } > 0 \Delta:=\min\{\Delta(x_1),\Delta(x_2)\}>0 Δ := min { Δ ( x 1 ) , Δ ( x 2 )} > 0 . If the optimizer plays i t ∼ x 1 i_t\sim x_1 i t ∼ x 1 for the first T 1 = ⌊ q T ⌋ T_1=\lfloor qT\rfloor T 1 = ⌊ q T ⌋ rounds ( I 1 = { 1 , … , T 1 } ) (\mathcal{I}_1=\{1,\ldots,T_1\}) ( I 1 = { 1 , … , T 1 }) and i t ∼ x 2 i_t\sim x_2 i t ∼ x 2 for the remaining T 2 = T − T 1 T_2=T-T_1 T 2 = T − T 1 rounds ( I 2 = { T 1 + 1 , … , T } ) (\mathcal{I}_2=\{T_1+1,\ldots,T\}) ( I 2 = { T 1 + 1 , … , T }) against any learning algorithm with stochastic regret Reg I s \operatorname{Reg}_{\mathcal{I}_s} Reg I s and overspending Over I s \operatorname{Over}_{\mathcal{I}_s} Over I s on each phase s ∈ { 1 , 2 } s\in\{1,2\} s ∈ { 1 , 2 } (letting Reg : = Reg I 1 + Reg I 2 \operatorname{Reg}:=\operatorname{Reg}_{\mathcal{I}_1}+\operatorname{Reg}_{\mathcal{I}_2} Reg := Reg I 1 + Reg I 2 and Over : = Over I 1 + Over I 2 \operatorname{Over}:=\operatorname{Over}_{\mathcal{I}_1}+\operatorname{Over}_{\mathcal{I}_2} Over := Over I 1 + Over I 2 ), then
E [ ∑ t = 1 T U O [ i t , j t ] ] ≥ ( q min y 1 ∈ BR L ( x 1 ) x 1 ⊤ U O y 1 + ( 1 − q ) min y 2 ∈ BR L ( x 2 ) x 2 ⊤ U O y 2 ) T − Reg + 2 Over / ρ L Δ − 1 , \mathbb{E}\left[\sum_{t=1}^{T}U_O[i_t,j_t]\right]\ge\left(q\min_{y_1\in\operatorname{BR}_L(x_1)}x_1^\top U_Oy_1+(1-q)\min_{y_2\in\operatorname{BR}_L(x_2)}x_2^\top U_Oy_2\right)T-\frac{\operatorname{Reg}+2\operatorname{Over}/\rho_L}{\Delta}-1, E [ t = 1 ∑ T U O [ i t , j t ] ] ≥ ( q y 1 ∈ BR L ( x 1 ) min x 1 ⊤ U O y 1 + ( 1 − q ) y 2 ∈ BR L ( x 2 ) min x 2 ⊤ U O y 2 ) T − Δ Reg + 2 Over / ρ L − 1 , E [ ∑ t = 1 T C O [ i t , j t ] ] ≤ ρ O T + Reg + 2 Over / ρ L Δ + 1. \mathbb{E}\left[\sum_{t=1}^{T}C_O[i_t,j_t]\right]\le\rho_OT+\frac{\operatorname{Reg}+2\operatorname{Over}/\rho_L}{\Delta}+1. E [ t = 1 ∑ T C O [ i t , j t ] ] ≤ ρ O T + Δ Reg + 2 Over / ρ L + 1. Consequently, whenever Reg , Over = o ( T ) \operatorname{Reg},\operatorname{Over}=o(T) Reg , Over = o ( T ) , by taking ( x 1 , x 2 , q ) (x_1,x_2,q) ( x 1 , x 2 , q ) approaching the supremum in (28) (and shortening the active phases by O ( Reg + Over / ρ L + 1 ρ O Δ ) O\left(\frac{\operatorname{Reg}+\operatorname{Over}/\rho_L+1}{\rho_O\Delta}\right) O ( ρ O Δ Reg + Over / ρ L + 1 ) rounds of the optimizer’s null action), the optimizer achieves expected utility at least W B S E ⋅ T − o ( T ) \mathrm{WBSE}\cdot T-o(T) WBSE ⋅ T − o ( T ) .
Proof. Applying Lemma 7.3 to I 1 \mathcal{I}_1 I 1 and I 2 \mathcal{I}_2 I 2 , summing the two bounds, and noting that ∣ T 1 − q T ∣ ≤ 1 |T_1-qT|\le1 ∣ T 1 − q T ∣ ≤ 1 and ∣ T 2 − ( 1 − q ) T ∣ ≤ 1 |T_2-(1-q)T|\le1 ∣ T 2 − ( 1 − q ) T ∣ ≤ 1 immediately yields the stated utility and cost guarantees. ■
Equivalence of WBSE and BSE for Generic Games
Finally, we show that when the learner’s utilities and costs are generic, i.e., for all but a measure 0 subset of possible cost and utility values, the learner’s best response B R L ( x ) \mathrm{BR}_{L}(x) BR L ( x ) is unique for any fixed (or generic) optimizer strategy x ∈ Δ n O x \in\Delta_{n_{O}} x ∈ Δ n O even when the budget constraint forces the learner to randomize. Moreover, for any x ∈ Δ n O x \in\Delta_{n_{O}} x ∈ Δ n O , every optimal best-response vertex of the learner can be uniquely induced by an arbitrarily small perturbation of x x x . As a consequence, the weak and strong Budgeted Stackelberg Equilibrium values coincide (WBSE = BSE).
Theorem 7.5. For generic learner utilities and costs ( U L , C L ) (U_{L}, C_{L}) ( U L , C L ) , it holds that WBSE = BSE.
Here we outline the idea of the proof; full details are deferred to Appendix § 16 . For a given x ∈ Δ n O x \in\Delta_{n_{O}} x ∈ Δ n O , associate each pure action j ∈ [ n L ] j \in[n_{L}] j ∈ [ n L ] of the learner with its cost-utility pair p j ( x ) ≔ ( ( x T C L ) j , ( x T U L ) j ) ∈ [ 0 , 1 ] 2 p_{j}(x) \coloneqq((x^{\mathsf{T}}C_{L})_{j}, (x^{\mathsf{T}}U_{L})_{j}) \in[0,1]^{2} p j ( x ) : = (( x T C L ) j , ( x T U L ) j ) ∈ [ 0 , 1 ] 2 . By linear programming duality (which holds strongly since p j 0 ( x ) = ( 0 , 0 ) p_{j_{0}}(x) = (0,0) p j 0 ( x ) = ( 0 , 0 ) and ρ L > 0 \rho_{L} > 0 ρ L > 0 ), we have
V L ( x ) = min λ ≥ 0 max j ∈ [ n L ] ( ( x T U L ) j − λ ( ( x T C L ) j − ρ L ) ) . V_{L}(x) = \min_{\lambda\ge0} \max_{j \in[n_{L}]} \left((x^{\mathsf{T}}U_{L})_{j} - \lambda\left((x^{\mathsf{T}}C_{L})_{j} - \rho_{L}\right)\right). V L ( x ) = λ ≥ 0 min j ∈ [ n L ] max ( ( x T U L ) j − λ ( ( x T C L ) j − ρ L ) ) .
and any y ∈ B R L ( x ) y \in\mathrm{BR}_{L}(x) y ∈ BR L ( x ) is supported on the set of tight pure actions S ( x , λ ) ≔ arg max j ∈ [ n L ] ( ( x T U L ) j − λ ( x T C L ) j ) S(x,\lambda) \coloneqq\arg\max_{j \in[n_{L}]} \left((x^{\mathsf{T}}U_{L})_{j} - \lambda(x^{\mathsf{T}}C_{L})_{j}\right) S ( x , λ ) : = arg max j ∈ [ n L ] ( ( x T U L ) j − λ ( x T C L ) j ) for an optimal dual multiplier λ ≥ 0 \lambda\ge0 λ ≥ 0 , with x T C L y = ρ L x^{\mathsf{T}}C_{L}y = \rho_{L} x T C L y = ρ L whenever λ > 0 \lambda> 0 λ > 0 . When the best response is a pure action ( λ = 0 ) (\lambda= 0) ( λ = 0 ) , two different best responses j 1 j_{1} j 1 and j 2 j_{2} j 2 must satisfy ( x T U L ) j 1 = ( x T U L ) j 2 (x^{\mathsf{T}}U_{L})_{j_{1}} = (x^{\mathsf{T}}U_{L})_{j_{2}} ( x T U L ) j 1 = ( x T U L ) j 2 , which occurs with 0 probability for a fixed or generic x x x . When the budget constraint is binding ( λ > 0 ) (\lambda> 0) ( λ > 0 ) , any two actions j 1 , j 2 ∈ S ( x , λ ) j_{1}, j_{2} \in S(x,\lambda) j 1 , j 2 ∈ S ( x , λ ) with ( x T C L ) j 1 < ρ L < ( x T C L ) j 2 (x^{\mathsf{T}}C_{L})_{j_{1}} < \rho_{L} < (x^{\mathsf{T}}C_{L})_{j_{2}} ( x T C L ) j 1 < ρ L < ( x T C L ) j 2 determine a unique budget-feasible mixture q e j 1 + ( 1 − q ) e j 2 q e_{j_{1}} + (1-q)e_{j_{2}} q e j 1 + ( 1 − q ) e j 2 with q ( x T C L ) j 1 + ( 1 − q ) ( x T C L ) j 2 = ρ L q(x^{\mathsf{T}}C_{L})_{j_{1}} + (1-q)(x^{\mathsf{T}}C_{L})_{j_{2}} = \rho_{L} q ( x T C L ) j 1 + ( 1 − q ) ( x T C L ) j 2 = ρ L ; thus, unless an action has cost ( x T C L ) j = ρ L (x^{\mathsf{T}}C_{L})_{j} = \rho_{L} ( x T C L ) j = ρ L , non-uniqueness implies that there are at least three actions j 1 , j 2 , j 3 ∈ S ( x , λ ) j_{1}, j_{2}, j_{3} \in S(x,\lambda) j 1 , j 2 , j 3 ∈ S ( x , λ ) lying on the same supporting line of slope λ \lambda λ in the cost-utility plane, meaning that p j 1 ( x ) , p j 2 ( x ) , p j 3 ( x ) p_{j_{1}}(x), p_{j_{2}}(x), p_{j_{3}}(x) p j 1 ( x ) , p j 2 ( x ) , p j 3 ( x ) are collinear and the following determinant is 0:
det ( 1 ( x T C L ) j 1 ( x T U L ) j 1 1 ( x T C L ) j 2 ( x T U L ) j 2 1 ( x T C L ) j 3 ( x T U L ) j 3 ) = 0. \det\begin{pmatrix}
1 & (x^{\mathsf{T}}C_{L})_{j_{1}} & (x^{\mathsf{T}}U_{L})_{j_{1}} \\
1 & (x^{\mathsf{T}}C_{L})_{j_{2}} & (x^{\mathsf{T}}U_{L})_{j_{2}} \\
1 & (x^{\mathsf{T}}C_{L})_{j_{3}} & (x^{\mathsf{T}}U_{L})_{j_{3}}
\end{pmatrix} = 0. det 1 1 1 ( x T C L ) j 1 ( x T C L ) j 2 ( x T C L ) j 3 ( x T U L ) j 1 ( x T U L ) j 2 ( x T U L ) j 3 = 0.
This defines a non-trivial polynomial equation in the entries of ( U L , C L ) (U_{L}, C_{L}) ( U L , C L ) , and hence a hypersurface of Lebesgue measure zero.
To show the equivalence of WBSE and BSE, we also need to consider the case where the optimizer chooses x ∈ Δ n O x \in\Delta_{n_{O}} x ∈ Δ n O on a boundary between best-response regions where the learner is indifferent between multiple vertices of the budget-feasible polytope F ( x ) \mathcal{F}(x) F ( x ) . We show that for generic ( U L , C L ) (U_{L}, C_{L}) ( U L , C L ) , any optimal vertex v v v of the learner’s best-response set can be selected as the unique best response by an arbitrarily small perturbation of x x x . For a given x ∈ Δ n O x \in\Delta_{n_{O}} x ∈ Δ n O , let its support be S x = supp ( x ) ⊆ [ n O ] S_{x} = \operatorname{supp}(x) \subseteq[n_{O}] S x = supp ( x ) ⊆ [ n O ] of size s = ∣ S x ∣ ≥ 1 s = |S_{x}| \ge1 s = ∣ S x ∣ ≥ 1 , so that x x x lies in the ( s − 1 ) (s-1) ( s − 1 ) -dimensional open simplex Δ S x ∘ ≔ { x ′ ∈ Δ n O : supp ( x ′ ) = S x } \Delta_{S_{x}}^{\circ} \coloneqq\{x' \in\Delta_{n_{O}} : \operatorname{supp}(x') = S_{x}\} Δ S x ∘ : = { x ′ ∈ Δ n O : supp ( x ′ ) = S x } with tangent space T S x ≔ { d ∈ R n O : supp ( d ) ⊆ S x , 1 T d = 0 } T_{S_{x}} \coloneqq\{d \in\mathbb{R}^{n_{O}} : \operatorname{supp}(d) \subseteq S_{x},\ \mathbf{1}^{\mathsf{T}}d = 0\} T S x : = { d ∈ R n O : supp ( d ) ⊆ S x , 1 T d = 0 } . Because x ∈ Δ S x ∘ x \in\Delta_{S_{x}}^{\circ} x ∈ Δ S x ∘ has s − 1 s-1 s − 1 degrees of freedom, for generic ( U L , C L ) (U_{L}, C_{L}) ( U L , C L ) at most s + 1 s+1 s + 1 actions can simultaneously lie on the supporting line (or at most s s s can tie in utility), and in Appendix § 16 we show that the resulting tie-breaking conditions are linearly independent, allowing any optimal vertex v v v to be made the unique best response by a small perturbation in a direction d ∈ T S x d \in T_{S_{x}} d ∈ T S x . The theorem then follows by taking the limit as the perturbation goes to 0.
AI Disclosure
The problem formulation, equilibrium concepts, and desiderata for the learning algorithm were developed entirely by the authors. ChatGPT Sol 5.6 and Astra 6 were helpful in suggesting the initial approach behind the forecasting algorithm and calibration errors. Both models and Gemini 4 Argon were used interactively as technical sounding boards while refining these ideas and checking auxiliary questions.
Reaching the paper’s algorithm, guarantees, and proofs required substantial redesign and technical work by the authors. Earlier AI-generated versions contained errors or relied on assumptions that would violate standard online-learning models. For example, they assumed that the learner knows the game matrices or n 0 n_0 n 0 in advance, sees the optimizer’s action before choosing its own action, or observes the optimizer’s realized action or mixed strategy after each round. While correct, initial proofs were also heavily over-engineered, relying on optimal transport, Sobolev norm spaces, resetting calibration errors when observing a new environment, and blocks of forced null actions with suboptimal 1 / ρ L 2 1/\rho_L^2 1/ ρ L 2 dependence. Identifying the core mechanisms, stripping away this machinery, and developing the full scope of results (including interval stochastic regret) required extensive iteration and analysis by the authors.
Furthermore, even when technical details were present, AI-generated write-ups bundled all components and were inaccessible even to an expert, requiring significant effort to decipher and verify. The authors therefore designed the paper’s modular structure, decoupling the calibration notions, and wrote almost all of the text and proofs from scratch. Gemini 4 Argon was also used to check related-work references and to audit the human-written proofs for typos and minor inaccuracies.
Auction Games vs. General Games
Table 2 summarizes the main differences between our work on repeated budgeted auctions [36 ] and the general budgeted games studied in this paper. While both papers evaluate non-manipulability against the Budgeted Stackelberg Equilibrium (BSE) and prove low interval stochastic regret, neither result subsumes the other. In [36 ] , the stage game is a Bayesian auction with i.i.d. values, where the structure of auctions implies that best responses are parameterized by a single scalar pacing multiplier (bid = λ ⋅ \lambda\cdot λ ⋅ value); this allows a simple, deterministic proportional controller to be non-manipulable. In general games, no deterministic learner can prevent manipulation, and best responses are discontinuous mixtures of up to two actions with no scalar parameterization. Conversely, our normal-form model has no private values and assumes full-information feedback: casting a Bayesian auction as a normal-form game requires treating entire bidding functions as pure strategies, which makes n L n_L n L —and with it our running time and regret bounds—exponential even after discretizing values and bids (so the O ( T 2 / 3 ) O(T^{2/3}) O ( T 2/3 ) rate in [36 ] and the O ( T ) O(\sqrt{T}) O ( T ) rate here are not directly comparable).
Alternative Definitions of Budgeted Stackelberg Equilibrium
In Sections 2 and 7, we defined the BSE under the condition that the learner always tries to best-respond subject to keeping the expected per-round cost at most ρ L \rho_L ρ L . This means that in each phase where the optimizer uses a fixed mixed strategy, the learner is meant to best-respond while keeping per-round spending at most ρ L \rho_L ρ L , similar to what Algorithm 1 does. However, this is not the only behavior that makes sense. For example, if historical average spending has been substantially less than ρ L \rho_L ρ L , it is natural to aim for spending higher than ρ L \rho_L ρ L per round. In this section, we formalize two alternative definitions of BSE focusing on an unbudgeted optimizer, corresponding to how a learner might use historical underspending, and show that both alternatives can only increase the BSE value—meaning that they can only benefit the optimizer. Under these definitions it might be natural to define the manipulability of a learning algorithm differently, but such definitions would allow for more utility for the optimizer.
For any optimizer strategy x ∈ Δ n O x \in\Delta_{n_O} x ∈ Δ n O and any per-round learner budget cap ρ ~ L > 0 \tilde{\rho}_L > 0 ρ ~ L > 0 , denote the learner’s ρ ~ L \tilde{\rho}_L ρ ~ L -budgeted best-response set by
BR L ( x ; ρ ~ L ) ≔ arg max y ∈ Δ n L : x T C L y ≤ ρ ~ L x T U L y . (30) \operatorname{BR}_L(x;\tilde{\rho}_L) \coloneqq\underset{y \in\Delta_{n_L}: x^{\mathsf{T}}C_Ly \le\tilde{\rho}_L}{\arg\max}\; x^{\mathsf{T}}U_Ly. \tag*{(30)} BR L ( x ; ρ ~ L ) : = y ∈ Δ n L : x T C L y ≤ ρ ~ L arg max x T U L y . (30)
Since C L ∈ [ 0 , 1 ] n O × n L C_L \in[0,1]^{n_O \times n_L} C L ∈ [ 0 , 1 ] n O × n L , setting ρ ~ L = ρ L \tilde{\rho}_L = \rho_L ρ ~ L = ρ L recovers BR L ( x ) = BR L ( x ; ρ L ) \operatorname{BR}_L(x) = \operatorname{BR}_L(x;\rho_L) BR L ( x ) = BR L ( x ; ρ L ) , while setting ρ ~ L ≥ 1 \tilde{\rho}_L \ge1 ρ ~ L ≥ 1 makes the cost constraint vacuous and recovers the unconstrained best-response set BR L ( x ; 1 ) = arg max y ∈ Δ n L x T U L y \operatorname{BR}_L(x;1) = \underset{y \in\Delta_{n_L}}{\arg\max}\; x^{\mathsf{T}}U_Ly BR L ( x ; 1 ) = y ∈ Δ n L arg max x T U L y .
Three models of learner behavior. Depending on how the learner adjusts her per-round spending cap after historical underspending, we obtain three equilibrium benchmarks:
Fixed per-round target ρ L \rho_L ρ L (our baseline, V 0 = B S E V_0 = \mathrm{BSE} V 0 = BSE ): The learner always enforces her target rate ρ L \rho_L ρ L in every phase (y ∈ BR L ( x ; ρ L ) y \in\operatorname{BR}_L(x;\rho_L) y ∈ BR L ( x ; ρ L ) ), regardless of past underspending. When the optimizer is unbudgeted, time-multiplexing across phases cannot improve upon a single mixed strategy, yielding (2) :
V 0 ≔ B S E = max x ∈ Δ n O , y ∈ BR L ( x ; ρ L ) x T U O y . (31) V_0 \coloneqq\mathrm{BSE} = \max_{\substack{x \in\Delta_{n_O},\; y \in\operatorname{BR}_L(x;\rho_L)}} x^{\mathsf{T}}U_Oy. \tag*{(31)} V 0 : = BSE = x ∈ Δ n O , y ∈ BR L ( x ; ρ L ) max x T U O y . (31)
Unconstrained best response under budget surplus (V 1 V_1 V 1 ): Consider a learner who is willing to overspend when her historical average spending is less than ρ L \rho_L ρ L . This can happen when using a primal/dual approach (e.g., LagrangeBwK [49 ] ), where the primal algorithm is learning to maximize the Lagrangian function, controlled by the dual algorithm that is picking the Lagrange multiplier to minimize the Lagrangian function. While historical spending is low, the Lagrange multiplier tends to be low, which makes the primal algorithm use an unrestricted best-response.
We model such a learner by assuming two different behaviors:
restricted phases (historical spending equal to ρ L \rho_L ρ L ), where a constrained best response y 1 ∈ BR L ( x 1 ; ρ L ) y_1 \in\operatorname{BR}_L(x_1;\rho_L) y 1 ∈ BR L ( x 1 ; ρ L ) is used,
unrestricted phases (historical spending strictly less than ρ L \rho_L ρ L ), where the budget constraint is ignored and an unconstrained best response y 2 ∈ BR L ( x 2 ; 1 ) y_2 \in\operatorname{BR}_L(x_2;1) y 2 ∈ BR L ( x 2 ; 1 ) is used.
Assuming that the optimizer uses only one restricted and one unrestricted phase, we make the following observations. The restricted phase must precede the unrestricted phase and the learner will continue using the unrestricted phase’s y 2 y_2 y 2 while the historical average spending is at most ρ L \rho_L ρ L . This introduces our first alternate BSE value V 1 V_1 V 1 , where
V 1 ≔ max x 1 , x 2 ∈ Δ n O , y 1 , y 2 ∈ Δ n L q 1 , q 2 ≥ 0 , q 1 + q 2 = 1 ∑ i = 1 2 q i x i T U O y i such that y 1 ∈ BR L ( x 1 ; ρ L ) , y 2 ∈ BR L ( x 2 ; 1 ) , and ∑ i = 1 2 q i x i T C L y i ≤ ρ L . (32) \begin{aligned}
V_1 \coloneqq\;& \max_{\substack{x_1,x_2 \in\Delta_{n_O},\; y_1,y_2 \in\Delta_{n_L}\\ q_1,q_2 \ge0,\;q_1+q_2=1}} \sum_{i=1}^{2} q_i x_i^{\mathsf{T}}U_Oy_i \\
&\text{such that}\qquad y_1 \in\operatorname{BR}_L(x_1;\rho_L),\; y_2 \in\operatorname{BR}_L(x_2;1),\\
&\text{and}\qquad\sum_{i=1}^{2} q_i x_i^{\mathsf{T}}C_Ly_i \le\rho_L. \tag*{(32)}
\end{aligned} V 1 : = x 1 , x 2 ∈ Δ n O , y 1 , y 2 ∈ Δ n L q 1 , q 2 ≥ 0 , q 1 + q 2 = 1 max i = 1 ∑ 2 q i x i T U O y i such that y 1 ∈ BR L ( x 1 ; ρ L ) , y 2 ∈ BR L ( x 2 ; 1 ) , and i = 1 ∑ 2 q i x i T C L y i ≤ ρ L . (32)
We now explain how an optimizer can implement a feasible solution. The first q 1 T q_1T q 1 T rounds the optimizer uses x 1 x_1 x 1 and the learner best responds with y 1 y_1 y 1 . Since y 1 ∈ BR L ( x 1 ; ρ L ) y_1 \in\operatorname{BR}_L(x_1;\rho_L) y 1 ∈ BR L ( x 1 ; ρ L ) the learner’s average spending is at most her budget: x 1 T C L y 1 ≤ ρ L x_1^{\mathsf{T}}C_Ly_1 \le\rho_L x 1 T C L y 1 ≤ ρ L . We now distinguish two cases based on the spending x 1 T C L y 1 x_1^{\mathsf{T}}C_Ly_1 x 1 T C L y 1
If x 1 T C L y 1 < ρ L x_1^{\mathsf{T}}C_{\mathrm{L}}y_1 < \rho_{\mathrm{L}} x 1 T C L y 1 < ρ L , i.e., the learner has been underspending after the first phase, then when the optimizer switches to x 2 x_2 x 2 after round q 1 T q_1T q 1 T , the learner has an excess of budget. This allows them to overspend and select the unbudgeted best-response y 2 y_2 y 2 . In fact, the inequality of Optimization Program (32) guarantees that this excess of budget will continue up to the final round T T T , since for any q ′ ∈ ( 0 , q 2 ) q' \in(0,q_2) q ′ ∈ ( 0 , q 2 ) ,
q 1 x 1 T C L y 1 + q ′ x 2 T C L y 2 < ( q 1 + q ′ ) ρ L . q_1x_1^{\mathsf{T}}C_{\mathrm{L}}y_1+q'x_2^{\mathsf{T}}C_{\mathrm{L}}y_2 < (q_1+q')\rho_{\mathrm{L}}. q 1 x 1 T C L y 1 + q ′ x 2 T C L y 2 < ( q 1 + q ′ ) ρ L .
This guarantees that y 2 y_2 y 2 will continue being a best-response to x 2 x_2 x 2 up to the final round.
If x 1 T C L y 1 = ρ L x_1^{\mathsf{T}}C_{\mathrm{L}}y_1 = \rho_{\mathrm{L}} x 1 T C L y 1 = ρ L , i.e., the spending matches the budget, then by the inequality of Optimization Program (32) it must hold x 2 T C L y 2 ≤ ρ L x_2^{\mathsf{T}}C_{\mathrm{L}}y_2 \le\rho_{\mathrm{L}} x 2 T C L y 2 ≤ ρ L , i.e., the learner does not overspend in the second phase either. This implies that y 2 ∈ B R L ( x 2 ; ρ L ) y_2 \in\mathrm{BR}_{\mathrm{L}}(x_2;\rho_{\mathrm{L}}) y 2 ∈ BR L ( x 2 ; ρ L ) , and the learner best-responds with y 2 y_2 y 2 when the optimizer starts using x 2 x_2 x 2 in rounds q 1 T , … , T q_1T,\ldots,T q 1 T , … , T , and never overspends her budget. This also implies that the value of such a solution cannot be more than V 0 V_0 V 0 : Both ( x 1 , y 1 ) (x_1,y_1) ( x 1 , y 1 ) and ( x 2 , y 2 ) (x_2,y_2) ( x 2 , y 2 ) are feasible solutions of Eq. (31) , so any convex combination of x 1 T U O y 1 x_1^{\mathsf{T}}U_{\mathrm{O}}y_1 x 1 T U O y 1 and x 2 T U O y 2 x_2^{\mathsf{T}}U_{\mathrm{O}}y_2 x 2 T U O y 2 cannot exceed V 0 V_0 V 0 .
We also note that because restricted and unrestricted phases decouple linearly adding more phases of either kind cannot make the value of Optimization Program (32) larger: given a solution with more phases ( x i , y i , q i ) i = 1 , 2 , … (x_i,y_i,q_i)_{i=1,2,\ldots} ( x i , y i , q i ) i = 1 , 2 , … we can fix ( x i , y i ) (x_i,y_i) ( x i , y i ) and write an LP with only the q i q_i q i as variables; since this has only two constraints ( ∑ i q i = 1 (\sum_i q_i=1 ( ∑ i q i = 1 and the inequality of Optimization Program (32) ) the optimal solution can be supported on at most two non-zero variables.
3. Best response to the remaining per-round budget (V 2 V_2 V 2 ): Here we consider a learner who adjusts the target budget to the remaining average budget. When there are τ \tau τ rounds remaining and the remaining budget is ρ ~ L τ \widetilde{\rho}_{\mathrm{L}}\tau ρ L τ , she best-responds subject to her remaining per-round budget rate ρ ~ L \widetilde{\rho}_{\mathrm{L}} ρ L , choosing y ∈ B R L ( x ; ρ ~ L ) y \in\mathrm{BR}_{\mathrm{L}}(x;\widetilde{\rho}_{\mathrm{L}}) y ∈ BR L ( x ; ρ L ) . In a multi-phase plan with phase lengths q i T q_iT q i T and strategies ( x i , y i ) (x_i,y_i) ( x i , y i ) , the effective budget rate entering phase i i i is ρ ~ L ( i ) = ( ρ L − ∑ s < i q s x s T C L y s ) / ( 1 − ∑ s < i q s ) \widetilde{\rho}_{\mathrm{L}}^{(i)}=(\rho_{\mathrm{L}}-\sum_{s<i}q_sx_s^{\mathsf{T}}C_{\mathrm{L}}y_s)/(1-\sum_{s<i}q_s) ρ L ( i ) = ( ρ L − ∑ s < i q s x s T C L y s ) / ( 1 − ∑ s < i q s ) , and the learner plays y i ∈ B R L ( x i ; ρ ~ L ( i ) ) y_i \in\mathrm{BR}_{\mathrm{L}}(x_i;\widetilde{\rho}_{\mathrm{L}}^{(i)}) y i ∈ BR L ( x i ; ρ L ( i ) ) . Without loss of generality, the learner strictly underspends ( x i T C L y i < ρ ~ L ( i ) ) (x_i^{\mathsf{T}}C_{\mathrm{L}}y_i<\widetilde{\rho}_{\mathrm{L}}^{(i)}) ( x i T C L y i < ρ L ( i ) ) in every phase except the last: If x i T C L y i = ρ ~ L ( i ) x_i^{\mathsf{T}}C_{\mathrm{L}}y_i=\widetilde{\rho}_{\mathrm{L}}^{(i)} x i T C L y i = ρ L ( i ) in an intermediate phase i i i , the remaining budget rate is unchanged after phase i i i , so the optimizer can do at least as well by either extending phase i i i to the end of the game or removing phase i i i and proportionally rescaling the remaining phases. Since the budget constraint is non-binding in all phases prior to the final phase, each earlier y i y_i y i is an unconstrained best response and hence also lies in B R L ( x i ; ρ ~ L ) \mathrm{BR}_{\mathrm{L}}(x_i;\widetilde{\rho}_{\mathrm{L}}) BR L ( x i ; ρ L ) , where ρ ~ L ≥ ρ L \widetilde{\rho}_{\mathrm{L}}\geq\rho_{\mathrm{L}} ρ L ≥ ρ L denotes the final phase’s budget rate (and ordering the phases so that x 1 T C L y 1 ≤ x 2 T C L y 2 ≤ ⋯ g u a r a n t e e s x_1^{\mathsf{T}}C_{\mathrm{L}}y_1\leq x_2^{\mathsf{T}}C_{\mathrm{L}}y_2\leq\cdots guarantees x 1 T C L y 1 ≤ x 2 T C L y 2 ≤ ⋯ g u a r an t ees x_i^{\mathsf{T}}C_{\mathrm{L}}y_i\leq\widetilde{\rho}_{\mathrm{L}}^{(i)}\leq\widetilde{\rho}_{\mathrm{L}}$ throughout). By Carathéodory’s theorem, at most two phases again suffice, yielding
V 2 : = max x 1 , x 2 ∈ Δ O , y 1 , y 2 ∈ Δ L ρ ~ L ≥ ρ L , q 1 , q 2 ≥ 0 , q 1 + q 2 = 1 ∑ i = 1 2 q i x i T U O y i V_2 := \max_{\substack{x_1,x_2\in\Delta_{\mathrm{O}},\ y_1,y_2\in\Delta_{\mathrm{L}}\\ \widetilde{\rho}_{\mathrm{L}}\geq\rho_{\mathrm{L}},\ q_1,q_2\geq0,\ q_1+q_2=1}} \sum_{i=1}^{2}q_i x_i^{\mathsf{T}}U_{\mathrm{O}}y_i V 2 := x 1 , x 2 ∈ Δ O , y 1 , y 2 ∈ Δ L ρ L ≥ ρ L , q 1 , q 2 ≥ 0 , q 1 + q 2 = 1 max i = 1 ∑ 2 q i x i T U O y i
such that y i ∈ B R L ( x i ; ρ ~ L ) ∀ i ∈ { 1 , 2 } , \text{such that}\qquad y_i\in\mathrm{BR}_{\mathrm{L}}(x_i;\widetilde{\rho}_{\mathrm{L}})\qquad\forall i\in\{1,2\}, such that y i ∈ BR L ( x i ; ρ L ) ∀ i ∈ { 1 , 2 } ,
and ∑ i = 1 2 q i x i T C L y i ≤ ρ L . (33) \text{and}\qquad\sum_{i=1}^{2}q_i x_i^{\mathsf{T}}C_{\mathrm{L}}y_i\leq\rho_{\mathrm{L}}. \tag*{(33)} and i = 1 ∑ 2 q i x i T C L y i ≤ ρ L . (33)
Alternative definitions can only benefit the optimizer. We next show that allowing the learner to spend accumulated budget surpluses never hurts the optimizer and can strictly increase the optimizer’s equilibrium payoff in some games under both alternatives.
Proposition B.1. For every budgeted game, BSE = V 0 ≤ V 1 ≤ V 2 V_0 \leq V_1 \leq V_2 V 0 ≤ V 1 ≤ V 2 . Moreover, both inequalities can be strict: there exist games with V 0 < V 1 V_0<V_1 V 0 < V 1 and games with V 1 < V 2 V_1<V_2 V 1 < V 2 .
Proof. Setting q 1 = 1 q_1=1 q 1 = 1 and q 2 = 0 q_2=0 q 2 = 0 in Optimization Program (32) recovers Eq. (31) , so V 0 ≤ V 1 V_0 \le V_1 V 0 ≤ V 1 . Setting ρ ~ L = 1 \tilde{\rho}_L=1 ρ ~ L = 1 in Optimization Program (33) recovers Optimization Program (32) , so V 1 ≤ V 2 V_1 \le V_2 V 1 ≤ V 2 . We establish the two strict separations via the following examples:
Strict separation V 0 < V 1 V_0<V_1 V 0 < V 1 : Consider a 2 × 2 2\times2 2 × 2 game with budget ρ L = 1 / 2 \rho_L=1/2 ρ L = 1/2 and matrices
U O = [ 1 0 0 0 ] , U L = [ 1 0 0 1 / 2 ] , C L = [ 1 1 / 2 1 / 2 0 ] . U_O=\begin{bmatrix}1&0\\0&0\end{bmatrix},\qquad U_L=\begin{bmatrix}1&0\\0&1/2\end{bmatrix},\qquad C_L=\begin{bmatrix}1&1/2\\1/2&0\end{bmatrix}. U O = [ 1 0 0 0 ] , U L = [ 1 0 0 1/2 ] , C L = [ 1 1/2 1/2 0 ] .
Under V 0 V_0 V 0 , any feasible pair ( x , y ) (x,y) ( x , y ) satisfies x T C L y = x 1 + y 1 2 ≤ ρ L = 1 2 x^\mathsf{T}C_Ly=\frac{x_1+y_1}{2}\le\rho_L=\frac{1}{2} x T C L y = 2 x 1 + y 1 ≤ ρ L = 2 1 , so by the AM–GM inequality the optimizer’s utility is at most x T U O y = x 1 y 1 ≤ ( x 1 + y 1 2 ) 2 ≤ 1 4 x^\mathsf{T}U_Oy=x_1y_1\le\left(\frac{x_1+y_1}{2}\right)^2\le\frac{1}{4} x T U O y = x 1 y 1 ≤ ( 2 x 1 + y 1 ) 2 ≤ 4 1 . This is attained at x ⋆ = y ⋆ = ( 1 / 2 , 1 / 2 ) x^\star=y^\star=(1/2,1/2) x ⋆ = y ⋆ = ( 1/2 , 1/2 ) , giving BSE = V 0 = 1 / 4 =V_0=1/4 = V 0 = 1/4 .
Under V 1 V_1 V 1 , however, the optimizer achieves V 1 = 1 2 V_1=\frac{1}{2} V 1 = 2 1 . First, the optimizer uses x 1 = ( 0 , 1 ) x_1=(0,1) x 1 = ( 0 , 1 ) for the first T / 2 T/2 T /2 rounds (q 1 = 1 / 2 q_1=1/2 q 1 = 1/2 ), inducing the best response y 1 = ( 0 , 1 ) y_1=(0,1) y 1 = ( 0 , 1 ) at cost 0. Then, they switch to x 2 = ( 1 , 0 ) x_2=(1,0) x 2 = ( 1 , 0 ) for the remaining T / 2 T/2 T /2 rounds (q 2 = 1 / 2 q_2=1/2 q 2 = 1/2 ), inducing the unconstrained best response y 2 = ( 1 , 0 ) y_2=(1,0) y 2 = ( 1 , 0 ) at cost 1. The average cost across the horizon is 1 2 ⋅ 0 + 1 2 ⋅ 1 = 1 2 = ρ L \frac{1}{2}\cdot0+\frac{1}{2}\cdot1=\frac{1}{2}=\rho_L 2 1 ⋅ 0 + 2 1 ⋅ 1 = 2 1 = ρ L , and the optimizer earns V 1 = 1 2 ⋅ 0 + 1 2 ⋅ 1 = 1 2 > 1 4 = V 0 V_1=\frac{1}{2}\cdot0+\frac{1}{2}\cdot1=\frac{1}{2}>\frac{1}{4}=V_0 V 1 = 2 1 ⋅ 0 + 2 1 ⋅ 1 = 2 1 > 4 1 = V 0 .
Strict separation V 1 < V 2 V_1<V_2 V 1 < V 2 : Consider a 2 × 3 2\times3 2 × 3 game with budget ρ L = 1 / 4 \rho_L=1/4 ρ L = 1/4 and matrices
U O = [ 1 1 / 2 0 1 1 / 2 0 ] , U L = [ 3 / 4 1 0 3 / 4 1 0 ] , C L = [ 1 / 2 0 0 1 / 2 1 0 ] . U_O=\begin{bmatrix}1&1/2&0\\1&1/2&0\end{bmatrix},\qquad U_L=\begin{bmatrix}3/4&1&0\\3/4&1&0\end{bmatrix},\qquad C_L=\begin{bmatrix}1/2&0&0\\1/2&1&0\end{bmatrix}. U O = [ 1 1 1/2 1/2 0 0 ] , U L = [ 3/4 3/4 1 1 0 0 ] , C L = [ 1/2 1/2 0 1 0 0 ] .
We start by showing that V 0 = 1 / 2 V_0=1/2 V 0 = 1/2 and then that also V 1 = V 0 = 1 / 2 V_1=V_0=1/2 V 1 = V 0 = 1/2 . If x x x is such that x 2 ≤ 1 / 4 x_2\le1/4 x 2 ≤ 1/4 , then the learner’s 2nd action is budget-feasible and the unique best-response, yielding optimizer utility 1 / 2 1/2 1/2 . If x 2 > 1 / 4 x_2>1/4 x 2 > 1/4 , then the optimizer’s utility is
x T U O y = y 1 + 1 2 y 2 ≤ y 1 + 2 x 2 y 2 = 2 x T C L y ≤ 1 2 x^\mathsf{T}U_Oy=y_1+\frac{1}{2}y_2\le y_1+2x_2y_2=2x^\mathsf{T}C_Ly\le\frac{1}{2} x T U O y = y 1 + 2 1 y 2 ≤ y 1 + 2 x 2 y 2 = 2 x T C L y ≤ 2 1
where in the last inequality we assume that the learner’s response is budget-feasible. This proves that V 0 = 1 / 2 V_0=1/2 V 0 = 1/2 .
We now prove that under V 1 V_1 V 1 the optimizer cannot get additional utility. This is because for every unbudgeted best-response y ∈ BR L ( x ; 1 ) y\in\operatorname{BR}_L(x;1) y ∈ BR L ( x ; 1 ) it holds y 2 = 1 y_2=1 y 2 = 1 , which means that x T U O y = 1 / 2 x^\mathsf{T}U_Oy=1/2 x T U O y = 1/2 . Since the objective of V 1 V_1 V 1 in Optimization Program (32) is a convex combination of ( x , y ) (x,y) ( x , y ) that is feasible in V 0 V_0 V 0 and ( x , y ) (x,y) ( x , y ) with y ∈ BR L ( x ; 1 ) y\in\operatorname{BR}_L(x;1) y ∈ BR L ( x ; 1 ) , its value cannot exceed 1 / 2 1/2 1/2 .
Under V 2 V_2 V 2 , however, the optimizer achieves V 2 = 3 / 4 V_2=3/4 V 2 = 3/4 . First, the optimizer uses x = ( 1 , 0 ) x=(1,0) x = ( 1 , 0 ) for the first T / 2 T/2 T /2 rounds (q 1 = 1 / 2 q_1=1/2 q 1 = 1/2 ), inducing the best response y = ( 0 , 1 , 0 ) y=(0,1,0) y = ( 0 , 1 , 0 ) at cost 0. This earns the optimizer utility 1 / 2 1/2 1/2 and raises the learner’s effective per-round budget for the remaining T / 2 T/2 T /2 rounds (q 2 = 1 / 2 q_2=1/2 q 2 = 1/2 ) from ρ L = 1 / 4 \rho_L=1/4 ρ L = 1/4 to ρ ~ L = 1 / 2 \tilde{\rho}_L=1/2 ρ ~ L = 1/2 . Then, the optimizer switches to x = ( 0 , 1 ) x=(0,1) x = ( 0 , 1 ) , under which the unique budget-adjusted best response is ( 1 , 0 , 0 ) ∈ BR L ( ( 0 , 1 ) ; ρ ~ L ) (1,0,0)\in\operatorname{BR}_L((0,1);\tilde{\rho}_L) ( 1 , 0 , 0 ) ∈ BR L (( 0 , 1 ) ; ρ ~ L ) : the utility to cost ratios of the first two actions are ( 3 / 2 , 1 ) (3/2,1) ( 3/2 , 1 ) , so since the first action has cost ρ ~ L \tilde{\rho}_L ρ ~ L it is the unique best response. Thus the optimizer earns V 2 = 1 2 ⋅ 1 2 + 1 2 ⋅ 1 = 3 4 > 1 2 = V 1 V_2=\frac{1}{2}\cdot\frac{1}{2}+\frac{1}{2}\cdot1=\frac{3}{4}>\frac{1}{2}=V_1 V 2 = 2 1 ⋅ 2 1 + 2 1 ⋅ 1 = 4 3 > 2 1 = V 1 .
Manipulability of Online Gradient Ascent
The example in this section is just for completeness: this example is directly from [36 ] .
In this section, we present a simple 3 × 3 3\times3 3 × 3 unbudgeted game (ρ L = 1 \rho_L=1 ρ L = 1 and C L = 0 C_L=0 C L = 0 ) in which Projected Online Gradient Ascent (OGA) is manipulable, even though OGA satisfies low interval regret (and dynamic regret) on every contiguous interval I ⊆ [ T ] I \subseteq[T] I ⊆ [ T ] —a property similar to, and technically stronger than, the stochastic per-interval regret guarantee we prove in Theorem 5.2 . Specifically, by switching actions each time the learner’s mixed strategy reaches a corner of the simplex, the optimizer forces Ω ( T ) \Omega(T) Ω ( T ) more utility than the Stackelberg equilibrium value SE.
Game definition and Stackelberg value. Consider an unbudgeted game with n O = n L = 3 n_O=n_L=3 n O = n L = 3 actions and payoff matrices
U O = [ 1 0 0 0 1 0 0 0 1 ] , U L = [ 0 1 1 2 1 2 0 1 1 1 2 0 ] . (34) U_O = \begin{bmatrix} 1 & 0 & 0 \\ 0 & 1 & 0 \\ 0 & 0 & 1 \end{bmatrix}, \qquad U_L = \begin{bmatrix} 0 & 1 & \frac{1}{2} \\ \frac{1}{2} & 0 & 1 \\ 1 & \frac{1}{2} & 0 \end{bmatrix}. \tag*{(34)} U O = 1 0 0 0 1 0 0 0 1 , U L = 0 2 1 1 1 0 2 1 2 1 1 0 . (34)
When the optimizer commits to a mixed strategy x = ( x 1 , x 2 , x 3 ) ∈ Δ 3 x=(x_1,x_2,x_3)\in\Delta_3 x = ( x 1 , x 2 , x 3 ) ∈ Δ 3 , the learner’s expected utilities for her three actions are x ⊤ U L = ( 1 2 x 2 + x 3 , x 1 + 1 2 x 3 , 1 2 x 1 + x 2 ) x^\top U_L=\left(\frac{1}{2}x_2+x_3,\ x_1+\frac{1}{2}x_3,\ \frac{1}{2}x_1+x_2\right) x ⊤ U L = ( 2 1 x 2 + x 3 , x 1 + 2 1 x 3 , 2 1 x 1 + x 2 ) . For pure action 1 to be a best response for the learner, it must beat action 2:
1 2 x 2 + x 3 ≥ x 1 + 1 2 x 3 ⟺ x 1 ≤ 1 2 ( x 2 + x 3 ) = 1 2 ( 1 − x 1 ) ⟺ x 1 ≤ 1 3 . \frac{1}{2}x_2+x_3 \ge x_1+\frac{1}{2}x_3 \quad\Longleftrightarrow\quad x_1 \le\frac{1}{2}(x_2+x_3)=\frac{1}{2}(1-x_1) \quad\Longleftrightarrow\quad x_1\le\frac{1}{3}. 2 1 x 2 + x 3 ≥ x 1 + 2 1 x 3 ⟺ x 1 ≤ 2 1 ( x 2 + x 3 ) = 2 1 ( 1 − x 1 ) ⟺ x 1 ≤ 3 1 .
By cyclic symmetry, any pure action j ∈ supp ( y ) j\in\operatorname{supp}(y) j ∈ supp ( y ) in a learner best response y ∈ BR L ( x ) y\in\operatorname{BR}_L(x) y ∈ BR L ( x ) satisfies x j ≤ 1 / 3 x_j\le1/3 x j ≤ 1/3 , so x ⊤ U O y = ∑ j y j x j ≤ 1 / 3 x^\top U_Oy=\sum_j y_jx_j\le1/3 x ⊤ U O y = ∑ j y j x j ≤ 1/3 . Since the uniform strategy x ⋆ = ( 1 / 3 , 1 / 3 , 1 / 3 ) x^\star=(1/3,1/3,1/3) x ⋆ = ( 1/3 , 1/3 , 1/3 ) achieves 1 / 3 1/3 1/3 , the Stackelberg value is S E = 1 / 3 \mathrm{SE}=1/3 SE = 1/3 .
How OGA is manipulated by corner switching. Suppose the learner updates her mixed strategy y t ∈ Δ 3 y_t\in\Delta_3 y t ∈ Δ 3 via Projected OGA with step size η > 0 \eta>0 η > 0 starting from y 1 = ( 1 / 3 , 1 / 3 , 1 / 3 ) y_1=(1/3,1/3,1/3) y 1 = ( 1/3 , 1/3 , 1/3 ) .
y t + 1 = Π Δ 3 ( y t + η U L [ i t , : ] ⊤ ) , where Π Δ 3 ( z ) = arg min y ∈ Δ 3 ∥ y − z ∥ 2 2 . (35) y_{t+1}=\Pi_{\Delta_3}\left(y_t+\eta U_L[i_t,:]^\top\right), \qquad\text{where}\qquad\Pi_{\Delta_3}(z)=\underset{y\in\Delta_3}{\arg\min}\ \lVert y-z\rVert_2^2. \tag*{(35)} y t + 1 = Π Δ 3 ( y t + η U L [ i t , : ] ⊤ ) , where Π Δ 3 ( z ) = y ∈ Δ 3 arg min ∥ y − z ∥ 2 2 . (35)
By standard online convex optimization analysis [47 ] , for η = Θ ( 1 / T ) \eta=\Theta(1/\sqrt{T}) η = Θ ( 1/ T ) , Projected OGA guarantees low dynamic regret: ∑ t ∈ I U L [ i t , : ] ⊤ ( y ⋆ − y t ) ≤ 1 η + 3 η 2 ∣ I ∣ = O ( T ) \sum_{t\in I}U_L[i_t,:]^\top(y^\star-y_t)\le\frac{1}{\eta}+\frac{3\eta}{2}|I|=O(\sqrt{T}) ∑ t ∈ I U L [ i t , : ] ⊤ ( y ⋆ − y t ) ≤ η 1 + 2 3 η ∣ I ∣ = O ( T ) regret on every contiguous interval I ⊆ [ T ] I\subseteq[T] I ⊆ [ T ] against any comparator y ⋆ ∈ Δ 3 y^\star\in\Delta_3 y ⋆ ∈ Δ 3 and any adversarial sequence of actions (which directly implies the stochastic interval regret guarantee of Theorem 5.2 ). Nevertheless, the optimizer can manipulate OGA using a simple state-based switching rule (note this is known by [55 ] since OGA can have linear swap regret).
The optimizer’s strategy goes as follows. Start by playing action 1 until y t y_t y t reaches the corner e 2 = ( 0 , 1 , 0 ) e_2=(0,1,0) e 2 = ( 0 , 1 , 0 ) (the learner’s best response to action 1); as soon as y t = e 2 y_t=e_2 y t = e 2 , switch to action 2 until y t y_t y t reaches e 3 = ( 0 , 0 , 1 ) e_3=(0,0,1) e 3 = ( 0 , 0 , 1 ) ; as soon as y t = e 3 y_t=e_3 y t = e 3 , switch to action 3 until y t y_t y t reaches e 1 = ( 1 , 0 , 0 ) e_1=(1,0,0) e 1 = ( 1 , 0 , 0 ) ; and repeat cyclically ( e i → e ( i m o d 3 ) + 1 ) (e_i\to e_{(i\bmod3)+1}) ( e i → e ( i mod 3 ) + 1 ) whenever y t y_t y t reaches a corner.
To see why this yields 1 / 2 1/2 1/2 average utility per round for the optimizer, consider the phase starting at y t = e 1 = ( 1 , 0 , 0 ) y_t=e_1=(1,0,0) y t = e 1 = ( 1 , 0 , 0 ) , where the optimizer switches to action 1 and induces learner utility vector u ( 1 ) = ( 0 , 1 , 1 / 2 ) ⊤ u^{(1)}=(0,1,1/2)^\top u ( 1 ) = ( 0 , 1 , 1/2 ) ⊤ . For any point y t = ( y t , 1 , 1 − y t , 1 , 0 ) y_t=(y_{t,1},1-y_{t,1},0) y t = ( y t , 1 , 1 − y t , 1 , 0 ) along the edge e 1 e 2 e_1e_2 e 1 e 2 , the Euclidean projection Π Δ 3 ( y t + η u ( 1 ) ) \Pi_{\Delta_3}(y_t+\eta u^{(1)}) Π Δ 3 ( y t + η u ( 1 ) ) subtracts the threshold θ = 0 + 1 2 η = η 2 \theta=\frac{0+1}{2}\eta=\frac{\eta}{2} θ = 2 0 + 1 η = 2 η from each coordinate and clips at zero. Because action 3’s unprojected increment is η 2 \frac{\eta}{2} 2 η , the third coordinate remains pinned at zero ( y t + 1 , 3 = 0 ) (y_{t+1,3}=0) ( y t + 1 , 3 = 0 ) , while y t , 1 y_{t,1} y t , 1 decreases linearly from 1 to 0 in steps of η 2 \frac{\eta}{2} 2 η over ⌈ 2 / η ⌉ \lceil2/\eta\rceil ⌈ 2/ η ⌉ rounds until y t y_t y t reaches the next corner e 2 = ( 0 , 1 , 0 ) e_2=(0,1,0) e 2 = ( 0 , 1 , 0 ) . Throughout this traversal of e 1 → e 2 e_1\to e_2 e 1 → e 2 , the optimizer plays action 1 and earns utility y t , 1 y_{t,1} y t , 1 in round t t t , which averages 1 / 2 1/2 1/2 per round as y t , 1 y_{t,1} y t , 1 drops linearly from 1 to 0. By cyclic symmetry, every subsequent edge traversal e 2 → e 3 e_2\to e_3 e 2 → e 3 , e 3 → e 1 e_3\to e_1 e 3 → e 1 , and e 1 → e 2 e_1\to e_2 e 1 → e 2 likewise keeps the third coordinate pinned at 0 and yields average optimizer utility 1 / 2 1/2 1/2 , giving total optimizer utility 1 2 T − O ( 1 / η ) = S E ⋅ T + 1 6 T − O ( T ) \frac{1}{2}T-O(1/\eta)=\mathrm{SE}\cdot T+\frac{1}{6}T-O(\sqrt{T}) 2 1 T − O ( 1/ η ) = SE ⋅ T + 6 1 T − O ( T ) for η = 1 / T \eta=1/\sqrt{T} η = 1/ T . Notice that this does not contradict OGA’s O ( T ) O(\sqrt{T}) O ( T ) interval regret: each single-edge phase has length ⌈ 2 / η ⌉ = Θ ( T ) \lceil2/\eta\rceil=\Theta(\sqrt{T}) ⌈ 2/ η ⌉ = Θ ( T ) (within the O ( 1 / η ) = O ( T ) O(1/\eta)=O(\sqrt{T}) O ( 1/ η ) = O ( T ) interval-regret slack), while over any longer interval spanning multiple cycles the optimizer’s empirical play is uniform, against which every fixed action earns 1 / 2 1/2 1/2 per round—matching OGA’s own average utility of 1 / 2 1/2 1/2 .
Experimental results. We simulate Projected OGA (Eq. (35) ) over T = 100,000 T = 100{,}000 T = 100 , 000 rounds with η = 1 / T \eta= 1/\sqrt{T} η = 1/ T starting from y 1 = ( 1 / 3 , 1 / 3 , 1 / 3 ) y_1 = (1/3,1/3,1/3) y 1 = ( 1/3 , 1/3 , 1/3 ) . As shown in Figure 1 (left), during the initial phase (i t = 1 i_t = 1 i t = 1 ), y t y_t y t moves parallel to e 1 e 2 e_1e_2 e 1 e 2 until hitting the boundary face y t , 1 = 0 y_{t,1} = 0 y t , 1 = 0 at t = 211 t = 211 t = 211 and then slides along e 2 e 3 e_2e_3 e 2 e 3 into the corner e 2 = ( 0 , 1 , 0 ) e_2 = (0,1,0) e 2 = ( 0 , 1 , 0 ) at t = 633 t = 633 t = 633 ; thereafter, y t y_t y t cycles counterclockwise along the boundary edges e 2 → e 3 → e 1 → e 2 e_2 \to e_3 \to e_1 \to e_2 e 2 → e 3 → e 1 → e 2 , taking ⌈ 2 / η ⌉ = 633 \lceil2/\eta\rceil= 633 ⌈ 2/ η ⌉ = 633 rounds per edge (157 corner switches in total). As shown in Figure 1 (right), the optimizer’s time-averaged utility converges to 0.498 ≈ 1 / 2 0.498 \approx1/2 0.498 ≈ 1/2 , exceeding S E = 1 / 3 \mathrm{SE} = 1/3 SE = 1/3 by Ω ( 1 ) \Omega(1) Ω ( 1 ) per round, while the learner still achieves 0.502 0.502 0.502 average utility per round (matching the best fixed action’s 0.503 0.503 0.503 in hindsight).
Figure 1. Simulation of Projected OGA in the 3 × 3 game of Eq. (34) over T = 100,000 T = 100{,}000 T = 100 , 000 rounds with step size η = 1 / T \eta= 1/\sqrt{T} η = 1/ T . Left: Trajectory of the learner’s mixed strategy y t ∈ Δ 3 y_t \in\Delta_3 y t ∈ Δ 3 from y 1 = ( 1 / 3 , 1 / 3 , 1 / 3 ) y_1 = (1/3,1/3,1/3) y 1 = ( 1/3 , 1/3 , 1/3 ) to the boundary (dashed) and around the boundary cycle e 2 → e 3 → e 1 → e 2 e_2 \to e_3 \to e_1 \to e_2 e 2 → e 3 → e 1 → e 2 (solid). Right: Optimizer’s running average utility 1 t ∑ τ = 1 t U O [ i τ , : ] y τ \frac{1}{t}\sum_{\tau=1}^{t} U_O[i_\tau,:]y_\tau t 1 ∑ τ = 1 t U O [ i τ , : ] y τ converging to 1 / 2 > S E = 1 / 3 1/2 > \mathrm{SE} = 1/3 1/2 > SE = 1/3 .
Why Naive Forecast Error Fails for Non-Manipulability
In this section, we give a simple 2 × 2 2 \times2 2 × 2 unbudgeted example (ρ L = 1 \rho_L = 1 ρ L = 1 and C L = 0 C_L = 0 C L = 0 ) illustrating why tracking only the error between the realized vector u C t u_{C_t} u C t and the predicted vector u C ( f ( t ) ) u_C(f^{(t)}) u C ( f ( t ) ) —without conditioning on the learner’s induced best response—fails to prevent manipulation.
Consider the unbudgeted matching pennies game with n O = n L = 2 n_O = n_L = 2 n O = n L = 2 actions and utility matrices
U O = [ 0 1 1 0 ] and U L = [ 1 0 0 1 ] , U_O =
\begin{bmatrix}
0 & 1 \\
1 & 0
\end{bmatrix}
\qquad\text{and}\qquad
U_L =
\begin{bmatrix}
1 & 0 \\
0 & 1
\end{bmatrix}, U O = [ 0 1 1 0 ] and U L = [ 1 0 0 1 ] ,
where the learner wants to match the optimizer’s action (j = i j = i j = i ) and the optimizer wants the learner to mismatch (j ≠ i j \ne i j = i ). Here the two rows of U L U_L U L are u ( 1 ) = ( 1 , 0 ) u^{(1)} = (1,0) u ( 1 ) = ( 1 , 0 ) and u ( 2 ) = ( 0 , 1 ) u^{(2)} = (0,1) u ( 2 ) = ( 0 , 1 ) , so a forecast f = ( f 1 , 1 − f 1 ) ∈ Δ 2 f = (f_1,1-f_1) \in\Delta_2 f = ( f 1 , 1 − f 1 ) ∈ Δ 2 predicts u ( f ) = ( f 1 , 1 − f 1 ) u(f) = (f_1,1-f_1) u ( f ) = ( f 1 , 1 − f 1 ) . The learner’s best response B R L ( u ( f ) ) \mathrm{BR}_L(u(f)) BR L ( u ( f )) is discontinuous at f 1 = 1 / 2 f_1 = 1/2 f 1 = 1/2 : it uniquely selects pure action 1 when f 1 > 1 / 2 f_1 > 1/2 f 1 > 1/2 and pure action 2 when f 1 < 1 / 2 f_1 < 1/2 f 1 < 1/2 . Consequently, against any committed mixed strategy x = ( x 1 , 1 − x 1 ) ∈ Δ 2 x = (x_1,1-x_1) \in\Delta_2 x = ( x 1 , 1 − x 1 ) ∈ Δ 2 , the optimizer obtains utility 1 − x 1 < 1 / 2 1-x_1 < 1/2 1 − x 1 < 1/2 if x 1 > 1 / 2 x_1 > 1/2 x 1 > 1/2 , utility x 1 < 1 / 2 x_1 < 1/2 x 1 < 1/2 if x 1 < 1 / 2 x_1 < 1/2 x 1 < 1/2 , and utility 1 / 2 1/2 1/2 if x 1 = 1 / 2 x_1 = 1/2 x 1 = 1/2 ; thus B S E = 1 / 2 \mathrm{BSE} = 1/2 BSE = 1/2 .
Now fix any small ϵ ∈ ( 0 , 1 / 2 ) \epsilon\in(0,1/2) ϵ ∈ ( 0 , 1/2 ) and consider how the naive forecast error fails around the discontinuity at f 1 = 1 / 2 f_1 = 1/2 f 1 = 1/2 . Suppose that in odd rounds the forecaster predicts f ( t ) = ( 1 / 2 + ϵ , 1 / 2 − ϵ ) f^{(t)} = (1/2+\epsilon,1/2-\epsilon) f ( t ) = ( 1/2 + ϵ , 1/2 − ϵ ) while the optimizer plays i t = 2 i_t = 2 i t = 2 (realizing u ( 2 ) = ( 0 , 1 ) u^{(2)} = (0,1) u ( 2 ) = ( 0 , 1 ) ), and in even rounds the forecaster predicts f ( t ) = ( 1 / 2 − ϵ , 1 / 2 + ϵ ) f^{(t)} = (1/2-\epsilon,1/2+\epsilon) f ( t ) = ( 1/2 − ϵ , 1/2 + ϵ ) while the optimizer plays i t = 1 i_t = 1 i t = 1 (realizing u ( 1 ) = ( 1 , 0 ) u^{(1)} = (1,0) u ( 1 ) = ( 1 , 0 ) ). Over any even horizon T T T , the cumulative error between the realized and predicted utility vectors is identically zero:
∑ t = 1 T ( u t − u ( f ( t ) ) ) = T 2 [ ( 0 , 1 ) − ( 1 2 + ε , 1 2 − ε ) ] + T 2 [ ( 1 , 0 ) − ( 1 2 − ε , 1 2 + ε ) ] = ( 0 , 0 ) . \sum_{t=1}^{T}\left(u_t-u\left(f^{(t)}\right)\right)=\frac{T}{2}\left[(0,1)-\left(\frac{1}{2}+\varepsilon,\frac{1}{2}-\varepsilon\right)\right]+\frac{T}{2}\left[(1,0)-\left(\frac{1}{2}-\varepsilon,\frac{1}{2}+\varepsilon\right)\right]=(0,0). t = 1 ∑ T ( u t − u ( f ( t ) ) ) = 2 T [ ( 0 , 1 ) − ( 2 1 + ε , 2 1 − ε ) ] + 2 T [ ( 1 , 0 ) − ( 2 1 − ε , 2 1 + ε ) ] = ( 0 , 0 ) .
Yet because the two forecasts lie on opposite sides of the discontinuity at 1 / 2 1/2 1/2 , the learner plays j t = 1 j_t=1 j t = 1 in every odd round (when i t = 2 i_t=2 i t = 2 ) and j t = 2 j_t=2 j t = 2 in every even round (when i t = 1 i_t=1 i t = 1 ). The optimizer therefore earns utility 1 1 1 in every round, achieving T = BSE ⋅ T + 1 2 T T=\operatorname{BSE}\cdot T+\frac{1}{2}T T = BSE ⋅ T + 2 1 T . The response-calibration error R T R_T R T in (4) prevents this by conditioning the forecast error on the learner’s induced best response J ⃗ ( f ( t ) ) \vec{J}(f^{(t)}) J ( f ( t ) ) , separating odd rounds (J ⃗ = ( 1 , 1 ) \vec{J}=(1,1) J = ( 1 , 1 ) ) from even rounds (J ⃗ = ( 2 , 2 ) \vec{J}=(2,2) J = ( 2 , 2 ) ) so their errors cannot cancel.
Why Indexing Response-Calibration Error by ( k , j ) (k,j) ( k , j ) Pairs Fails
Instead of indexing δ R \delta^R δ R and R T R_T R T in Section 4 by triplets ( k , j 1 , j 2 ) ∈ [ n ^ O ] × [ n L ] 2 (k,j_1,j_2)\in[\hat n_O]\times[n_L]^2 ( k , j 1 , j 2 ) ∈ [ n ^ O ] × [ n L ] 2 , a natural alternative is to index by pairs ( k , j ) ∈ [ n ^ O ] × [ n L ] (k,j)\in[\hat n_O]\times[n_L] ( k , j ) ∈ [ n ^ O ] × [ n L ] , recording the step function 1 q ≤ Q ( f ) \mathbb{1}_{q\le Q(f)} 1 q ≤ Q ( f ) on coordinate ( k , J 1 ( f ) ) (k,J_1(f)) ( k , J 1 ( f )) and 1 q ≤ 1 − Q ( f ) \mathbb{1}_{q\le1-Q(f)} 1 q ≤ 1 − Q ( f ) on coordinate ( k , J 2 ( f ) ) (k,J_2(f)) ( k , J 2 ( f )) :
R ~ T , ( k , j ) ( q ) : = ∑ t ∈ T E f ∼ F t [ ( 1 k t = k − f k ) ( 1 j = J 1 ( f ) 1 q ≤ Q ( f ) + 1 j = J 2 ( f ) 1 q ≤ 1 − Q ( f ) ) ] . \widetilde{R}_{T,(k,j)}(q):=\sum_{t\in\mathcal{T}}\mathbb{E}_{f\sim F_t}\left[\left(\mathbb{1}_{k_t=k}-f_k\right)\left(\mathbb{1}_{j=J_1(f)}\mathbb{1}_{q\le Q(f)}+\mathbb{1}_{j=J_2(f)}\mathbb{1}_{q\le1-Q(f)}\right)\right]. R T , ( k , j ) ( q ) := t ∈ T ∑ E f ∼ F t [ ( 1 k t = k − f k ) ( 1 j = J 1 ( f ) 1 q ≤ Q ( f ) + 1 j = J 2 ( f ) 1 q ≤ 1 − Q ( f ) ) ] .
When each environment u C ( k ) u_C^{(k)} u C ( k ) corresponds to a unique optimizer payoff row (that is, if n ^ O = n O \hat n_O=n_O n ^ O = n O ), g k , j 1 , j 2 ( q ) = q U O [ k , j 1 ] + ( 1 − q ) U O [ k , j 2 ] g_{k,j_1,j_2}(q)=qU_O[k,j_1]+(1-q)U_O[k,j_2] g k , j 1 , j 2 ( q ) = q U O [ k , j 1 ] + ( 1 − q ) U O [ k , j 2 ] decomposes additively across j 1 j_1 j 1 and j 2 j_2 j 2 , so bounding ∥ R ~ T ∥ L 2 \|\widetilde{R}_T\|_{L_2} ∥ R T ∥ L 2 suffices and reduces the dimension from n ^ O n L 2 \hat n_O n_L^2 n ^ O n L 2 to n ^ O n L \hat n_O n_L n ^ O n L (saving a n L \sqrt{n_L} n L factor in Theorem 4.2). The same argument also works if i t i_t i t is observed, and we can collect calibration error R ~ T , ( i , j ) ( q ) \widetilde{R}_{T,(i,j)}(q) R T , ( i , j ) ( q ) for ( i , j ) (i,j) ( i , j ) pairs instead of ( k , j ) (k,j) ( k , j ) pairs.
However, when multiple optimizer actions i 1 , i 2 ∈ [ n O ] i_1,i_2\in[n_O] i 1 , i 2 ∈ [ n O ] induce the same learner environment u C ( k ) u_C^{(k)} u C ( k ) (n ^ O < n O \hat n_O<n_O n ^ O < n O ), R ~ T \widetilde{R}_T R T fails. To see why, consider a learner’s mixed best response y ( f ) = Q ( f ) e j 1 + ( 1 − Q ( f ) ) e j 2 y(f)=Q(f)e_{j_1}+(1-Q(f))e_{j_2} y ( f ) = Q ( f ) e j 1 + ( 1 − Q ( f )) e j 2 to a forecast f f f . We estimated the optimizer’s payoff under u C ( k ) u_C^{(k)} u C ( k ) as
g k , j 1 , j 2 ( q ) = max i : ( U L [ i , : ] , C L [ i , : ] ) = u C ( k ) ( q U O [ i , j 1 ] + ( 1 − q ) U O [ i , j 2 ] ) . g_{k,j_1,j_2}(q)=\max_{i:(U_L[i,:],C_L[i,:])=u_C^{(k)}}\left(qU_O[i,j_1]+(1-q)U_O[i,j_2]\right). g k , j 1 , j 2 ( q ) = i : ( U L [ i , : ] , C L [ i , : ]) = u C ( k ) max ( q U O [ i , j 1 ] + ( 1 − q ) U O [ i , j 2 ] ) .
which depends on the pair ( j 1 , j 2 ) (j_1,j_2) ( j 1 , j 2 ) jointly. Unbundling into ( k , j ) (k,j) ( k , j ) pairs forgets which other action j j j was mixed with; bounding ( k , j 1 ) (k,j_1) ( k , j 1 ) and ( k , j 2 ) (k,j_2) ( k , j 2 ) separately would replace the maximum of the sum by the sum of the maxima over i i i , allowing the optimizer to play i 1 i_1 i 1 when j t = j 1 j_t=j_1 j t = j 1 and i 2 i_2 i 2 when j t = j 2 j_t=j_2 j t = j 2 within the same coin flip. The example below shows how this can cause higher value for the optimizer despite R ~ T , ( k , j ) ( q ) = 0 \widetilde{R}_{T,(k,j)}(q)=0 R T , ( k , j ) ( q ) = 0 .
Counterexample. Consider a game with n O = n L = 4 n_O=n_L=4 n O = n L = 4 , budget ρ L = 1 / 2 \rho_L=1/2 ρ L = 1/2 , and matrices
U O = ( 0 0 0 1 1 0 0 0 1 0 1 0 0 1 0 1 ) , U L = 1 3 ( 1 0 2 2 0 0 2 3 0 1 3 2 0 1 3 2 ) , C L = ( 0 0 1 1 0 0 1 1 0 0 1 1 0 0 1 1 ) . U_O=
\begin{pmatrix}
0&0&0&1\\
1&0&0&0\\
1&0&1&0\\
0&1&0&1
\end{pmatrix},
\qquad
U_L=\frac{1}{3}
\begin{pmatrix}
1&0&2&2\\
0&0&2&3\\
0&1&3&2\\
0&1&3&2
\end{pmatrix},
\qquad
C_L=
\begin{pmatrix}
0&0&1&1\\
0&0&1&1\\
0&0&1&1\\
0&0&1&1
\end{pmatrix}. U O = 0 1 1 0 0 0 0 1 0 0 1 0 1 0 0 1 , U L = 3 1 1 0 0 0 0 0 1 1 2 2 3 3 2 3 2 2 , C L = 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 .
Here optimizer actions i = 1 i=1 i = 1 and i = 2 i=2 i = 2 induce u C ( 1 ) u_C^{(1)} u C ( 1 ) and u C ( 2 ) u_C^{(2)} u C ( 2 ) , respectively, while i = 3 , 4 i=3,4 i = 3 , 4 induce the same environment u C ( 3 ) u_C^{(3)} u C ( 3 ) (n ^ O = 3 \hat n_O=3 n ^ O = 3 ). In every row of U L U_L U L , the free actions { 1 , 2 } \{1,2\} { 1 , 2 } (cost 0 0 0 ) have utility at most 1 / 3 1/3 1/3 and the expensive actions { 3 , 4 } \{3,4\} { 3 , 4 } (cost 1 1 1 ) have utility at least 2 / 3 2/3 2/3 . Thus, for any forecast f ∈ Δ 3 f\in\Delta_3 f ∈ Δ 3 , the learner’s best response mixes one free action j 1 ∈ arg max j ∈ { 1 , 2 } u ( f ) [ j ] j_1\in\operatorname*{arg\,max}_{j\in\{1,2\}}u(f)[j] j 1 ∈ arg max j ∈ { 1 , 2 } u ( f ) [ j ] and one expensive action j 2 ∈ arg max j ∈ { 3 , 4 } u ( f ) [ j ] j_2\in\operatorname*{arg\,max}_{j\in\{3,4\}}u(f)[j] j 2 ∈ arg max j ∈ { 3 , 4 } u ( f ) [ j ] with weight Q ( f ) = 1 / 2 Q(f)=1/2 Q ( f ) = 1/2 each. Moreover, since u ( f ) [ 1 ] − u ( f ) [ 2 ] = 1 3 ( f 1 − f 3 ) u(f)[1]-u(f)[2]=\frac{1}{3}(f_1-f_3) u ( f ) [ 1 ] − u ( f ) [ 2 ] = 3 1 ( f 1 − f 3 ) and u ( f ) [ 3 ] − u ( f ) [ 4 ] = 1 3 ( f 3 − f 2 ) u(f)[3]-u(f)[4]=\frac{1}{3}(f_3-f_2) u ( f ) [ 3 ] − u ( f ) [ 4 ] = 3 1 ( f 3 − f 2 ) , the learner’s best response is determined simply by comparing f 3 f_3 f 3 to f 1 f_1 f 1 and f 2 f_2 f 2 :
Aligned pairs: ( 1 , 3 ) (1,3) ( 1 , 3 ) when f 1 > f 3 > f 2 f_1 > f_3 > f_2 f 1 > f 3 > f 2 , and ( 2 , 4 ) (2,4) ( 2 , 4 ) when f 2 > f 3 > f 1 f_2 > f_3 > f_1 f 2 > f 3 > f 1 . Under u c ( 3 ) uc^{(3)} u c ( 3 ) and when ( j 1 , j 2 ) (j_1,j_2) ( j 1 , j 2 ) is an aligned pair, the optimizer gets g 3 , j 1 , j 2 ( 1 / 2 ) = 1 g_{3,j_1,j_2}(1/2)=1 g 3 , j 1 , j 2 ( 1/2 ) = 1 by playing i = 3 i=3 i = 3 on ( 1 , 3 ) (1,3) ( 1 , 3 ) and i = 4 i=4 i = 4 on ( 2 , 4 ) (2,4) ( 2 , 4 ) .
Crossed pairs: ( 2 , 3 ) (2,3) ( 2 , 3 ) when f 3 > f 1 , f 2 f_3 > f_1,f_2 f 3 > f 1 , f 2 , and ( 1 , 4 ) (1,4) ( 1 , 4 ) when f 3 < f 1 , f 2 f_3 < f_1,f_2 f 3 < f 1 , f 2 . Under u c ( 3 ) uc^{(3)} u c ( 3 ) and when ( j 1 , j 2 ) (j_1,j_2) ( j 1 , j 2 ) is a crossed pair, the optimizer gets g 3 , j 1 , j 2 ( 1 / 2 ) = 1 / 2 g_{3,j_1,j_2}(1/2)=1/2 g 3 , j 1 , j 2 ( 1/2 ) = 1/2 for either i = 3 , 4 i=3,4 i = 3 , 4 .
Under u c ( 1 ) uc^{(1)} u c ( 1 ) (i = 1 i=1 i = 1 ) and u c ( 2 ) uc^{(2)} u c ( 2 ) (i = 2 i=2 i = 2 ), each row of U O U_O U O contains a single 1, so the optimizer gets at most 1 / 2 1/2 1/2 on every pair, and gets 0 under u c ( 1 ) uc^{(1)} u c ( 1 ) on ( 1 , 3 ) (1,3) ( 1 , 3 ) and under u c ( 2 ) uc^{(2)} u c ( 2 ) on ( 2 , 4 ) (2,4) ( 2 , 4 ) . Thus, for any forecast f ∈ Δ 3 f\in\Delta_3 f ∈ Δ 3 , the optimizer’s expected utility is at most 1 / 2 1/2 1/2 on the crossed pairs; on the aligned pair ( 1 , 3 ) (1,3) ( 1 , 3 ) (where f 1 ≥ f 3 ≥ f 2 f_1\ge f_3\ge f_2 f 1 ≥ f 3 ≥ f 2 ), it is 0 ⋅ f 1 + 1 2 f 2 + 1 ⋅ f 3 ≤ 1 2 ( f 1 + f 2 + f 3 ) = 1 / 2 0\cdot f_1+\frac{1}{2}f_2+1\cdot f_3\le\frac{1}{2}(f_1+f_2+f_3)=1/2 0 ⋅ f 1 + 2 1 f 2 + 1 ⋅ f 3 ≤ 2 1 ( f 1 + f 2 + f 3 ) = 1/2 (and symmetrically on ( 2 , 4 ) (2,4) ( 2 , 4 ) ). Hence BSE = 1 / 2 =1/2 = 1/2 (achieved, e.g., at f = ( 0 , 0 , 1 ) f=(0,0,1) f = ( 0 , 0 , 1 ) ).
Now divide T T T into four equal phases of T / 4 T/4 T /4 rounds. In each phase a deterministic forecast f f f is used that uniquely induces one of the four pairs ( j 1 , j 2 ) ∈ { 1 , 2 } × { 3 , 4 } (j_1,j_2)\in\{1,2\}\times\{3,4\} ( j 1 , j 2 ) ∈ { 1 , 2 } × { 3 , 4 } with Q ( f ) = 1 / 2 Q(f)=1/2 Q ( f ) = 1/2 . Let d : = 1 8 ( − 1 , − 1 , 2 ) d:=\frac{1}{8}(-1,-1,2) d := 8 1 ( − 1 , − 1 , 2 ) , which will be the forecasting error. Specifically, for phases with aligned pairs, the optimizer’s mixed strategy induces f ^ \hat f f ^ such that f ^ − f = d \hat f-f=d f ^ − f = d and for crossed pairs f ^ − f = − d \hat f-f=-d f ^ − f = − d , making the average forecast correct. In particular, even if we condition on a certain learner action, the average forecast is correct, making R ~ T = 0 \widetilde R_T=0 R T = 0 . This cannot happen when conditioning on pairs of learner actions (since for each pair f ^ − f \hat f-f f ^ − f is d d d or − d -d − d ) which makes ∥ R T ∥ = Ω ( T ) \lVert R_T\rVert=\Omega(T) ∥ R T ∥ = Ω ( T ) .
The phases follow.
Aligned phase ( 1 , 3 ) (1,3) ( 1 , 3 ) : The forecast is f = 1 8 ( 4 , 1 , 3 ) f=\frac{1}{8}(4,1,3) f = 8 1 ( 4 , 1 , 3 ) (f 1 > f 3 > f 2 f_1>f_3>f_2 f 1 > f 3 > f 2 ), which does induce ( 1 , 3 ) (1,3) ( 1 , 3 ) . The optimizer plays i = 1 i=1 i = 1 (u c ( 1 ) uc^{(1)} u c ( 1 ) ) on 3 / 8 3/8 3/8 of the rounds and i = 3 i=3 i = 3 (u c ( 3 ) uc^{(3)} u c ( 3 ) ) on 5 / 8 5/8 5/8 , giving empirical environment distribution f ^ = f + d = 1 8 ( 3 , 0 , 5 ) \hat f=f+d=\frac{1}{8}(3,0,5) f ^ = f + d = 8 1 ( 3 , 0 , 5 ) . The optimizer’s expected utility per round is 5 / 8 5/8 5/8 .
Aligned phase ( 2 , 4 ) (2,4) ( 2 , 4 ) : The forecast is f = 1 8 ( 1 , 4 , 3 ) f=\frac{1}{8}(1,4,3) f = 8 1 ( 1 , 4 , 3 ) (f 2 > f 3 > f 1 f_2>f_3>f_1 f 2 > f 3 > f 1 ), inducing ( 2 , 4 ) (2,4) ( 2 , 4 ) . The optimizer plays i = 2 i=2 i = 2 (u c ( 2 ) uc^{(2)} u c ( 2 ) ) on 3 / 8 3/8 3/8 of the rounds and i = 4 i=4 i = 4 (u c ( 3 ) uc^{(3)} u c ( 3 ) ) on 5 / 8 5/8 5/8 , giving f ^ = f + d = 1 8 ( 0 , 3 , 5 ) \hat f=f+d=\frac{1}{8}(0,3,5) f ^ = f + d = 8 1 ( 0 , 3 , 5 ) and optimizer utility 5 / 8 5/8 5/8 per round.
Crossed phase ( 2 , 3 ) (2,3) ( 2 , 3 ) : The forecast is f = ( 0 , 0 , 1 ) f=(0,0,1) f = ( 0 , 0 , 1 ) (f 3 > f 1 , f 2 f_3>f_1,f_2 f 3 > f 1 , f 2 ), inducing ( 2 , 3 ) (2,3) ( 2 , 3 ) . The optimizer plays i = 1 , 2 i=1,2 i = 1 , 2 on 1 / 8 1/8 1/8 of the rounds each and i = 3 i=3 i = 3 (u c ( 3 ) uc^{(3)} u c ( 3 ) ) on 6 / 8 6/8 6/8 , giving f ^ = f − d = 1 8 ( 1 , 1 , 6 ) \hat f=f-d=\frac{1}{8}(1,1,6) f ^ = f − d = 8 1 ( 1 , 1 , 6 ) and optimizer utility 6 8 ⋅ 1 2 = 3 / 8 \frac{6}{8}\cdot\frac{1}{2}=3/8 8 6 ⋅ 2 1 = 3/8 per round.
Crossed phase ( 1 , 4 ) (1,4) ( 1 , 4 ) : The forecast is f = 1 8 ( 3 , 3 , 2 ) f=\frac{1}{8}(3,3,2) f = 8 1 ( 3 , 3 , 2 ) (f 3 < f 1 , f 2 f_3<f_1,f_2 f 3 < f 1 , f 2 ), inducing ( 1 , 4 ) (1,4) ( 1 , 4 ) . The optimizer plays i = 1 , 2 i=1,2 i = 1 , 2 on 1 / 2 1/2 1/2 of the rounds each (never u c ( 3 ) uc^{(3)} u c ( 3 ) ), giving f ^ = f − d = ( 1 2 , 1 2 , 0 ) \hat f=f-d=(\frac{1}{2},\frac{1}{2},0) f ^ = f − d = ( 2 1 , 2 1 , 0 ) and optimizer utility 1 / 2 1/2 1/2 per round.
Because every pure learner action j ∈ { 1 , 2 , 3 , 4 } j\in\{1,2,3,4\} j ∈ { 1 , 2 , 3 , 4 } is played with weight 1 / 2 1/2 1/2 in exactly one aligned phase (error + d +d + d ) and one crossed phase (error − d -d − d ), the pairwise calibration error vanishes identically for every ( k , j ) ∈ [ 3 ] × [ 4 ] (k,j)\in[3]\times[4] ( k , j ) ∈ [ 3 ] × [ 4 ] :
R ~ T , ( k , j ) ( q ) = T 4 ( d k − d k ) 1 q ≤ 1 / 2 = 0. \widetilde R_{T,(k,j)}(q)=\frac{T}{4}(d_k-d_k)\mathbf{1}_{q\le1/2}=0. R T , ( k , j ) ( q ) = 4 T ( d k − d k ) 1 q ≤ 1/2 = 0.
Yet by inducing u c ( 3 ) uc^{(3)} u c ( 3 ) more often than forecast on aligned pairs (where u c ( 3 ) uc^{(3)} u c ( 3 ) pays 1) and less often on crossed pairs (where u c ( 3 ) uc^{(3)} u c ( 3 ) pays 1 / 2 1/2 1/2 ), the optimizer achieves total utility T 4 ( 5 8 + 5 8 + 3 8 + 4 8 ) = 17 32 T = B S E ⋅ T + 1 32 T \frac{T}{4}\left(\frac{5}{8}+\frac{5}{8}+\frac{3}{8}+\frac{4}{8}\right)=\frac{17}{32}T=\mathrm{BSE}\cdot T+\frac{1}{32}T 4 T ( 8 5 + 8 5 + 8 3 + 8 4 ) = 32 17 T = BSE ⋅ T + 32 1 T .
Tracking triplets ( k , j 1 , j 2 ) (k,j_1,j_2) ( k , j 1 , j 2 ) in R T R_T R T prevents this cancellation by keeping the four pairs in separate coordinates. Specifically, for each k ∈ [ 3 ] k\in[3] k ∈ [ 3 ] , the aligned triplets ( k , 1 , 3 ) (k,1,3) ( k , 1 , 3 ) and ( k , 2 , 4 ) (k,2,4) ( k , 2 , 4 ) each accumulate T 4 d k 1 q ≤ 1 / 2 \frac{T}{4}d_k\mathbf{1}_{q\le1/2} 4 T d k 1 q ≤ 1/2 in R T R_T R T , whereas the crossed triplets ( k , 2 , 3 ) (k,2,3) ( k , 2 , 3 ) and ( k , 1 , 4 ) (k,1,4) ( k , 1 , 4 ) each accumulate − T 4 d k 1 q ≤ 1 / 2 -\frac{T}{4}d_k\mathbf{1}_{q\le1/2} − 4 T d k 1 q ≤ 1/2 . Because the + d k +d_k + d k error from ( 1 , 3 ) (1,3) ( 1 , 3 ) and the − d k -d_k − d k error from ( 1 , 4 ) (1,4) ( 1 , 4 ) are no longer summed into a single ( k , 1 ) (k,1) ( k , 1 ) coordinate (and similarly for j = 2 , 3 , 4 j=2,3,4 j = 2 , 3 , 4 ), they cannot cancel across distinct pairs, yielding ∥ R T ∥ L 2 = Ω ( T ) \lVert R_T\rVert_{L_2}=\Omega(T) ∥ R T ∥ L 2 = Ω ( T ) .
Why Response-calibration is Insufficient for Stochastic Regret
In this section we show that even if R T = 0 R_T = 0 R T = 0 , the learner may suffer Ω ( T ) \Omega(T) Ω ( T ) regret in a stochastic environment by best-responding to each of the optimizer’s pure actions instead of the mixed ones. Our example is as follows. Let ρ L = 1 / 2 \rho_L = 1/2 ρ L = 1/2 , n O = n ^ O = 2 n_O = \hat{n}_O = 2 n O = n ^ O = 2 , n L = 3 n_L = 3 n L = 3 (where the third action is the null action), and
U L = [ 1 / 2 4 / 5 0 1 / 2 2 / 5 0 ] and C L = [ 1 / 2 1 0 1 / 2 0 0 ] , U_L = \begin{bmatrix} 1/2 & 4/5 & 0 \\ 1/2 & 2/5 & 0 \end{bmatrix}
\qquad\text{and}\qquad
C_L = \begin{bmatrix} 1/2 & 1 & 0 \\ 1/2 & 0 & 0 \end{bmatrix}, U L = [ 1/2 1/2 4/5 2/5 0 0 ] and C L = [ 1/2 1/2 1 0 0 0 ] ,
where the optimizer draws u c ( 1 ) u c^{(1)} u c ( 1 ) (row 1) and u c ( 2 ) u c^{(2)} u c ( 2 ) (row 2) with probability 1 / 2 1/2 1/2 each. This makes the “correct” forecast f ⋆ = ( 1 / 2 , 1 / 2 ) f^\star= (1/2, 1/2) f ⋆ = ( 1/2 , 1/2 ) whose best response is action 2, achieving expected utility 3 / 5 3/5 3/5 and cost 1 / 2 1/2 1/2 .
Under forecast f = ( 1 , 0 ) f = (1, 0) f = ( 1 , 0 ) (predicting u c ( 1 ) u c^{(1)} u c ( 1 ) ), pure action 1 is the unique best response, achieving utility 1 / 2 1/2 1/2 at cost 1 / 2 1/2 1/2 . Under f = ( 0 , 1 ) f = (0, 1) f = ( 0 , 1 ) (predicting u c ( 2 ) u c^{(2)} u c ( 2 ) ), pure action 1 is again the unique best response, achieving utility 1 / 2 1/2 1/2 at cost $1/2. Since both induce pure action 1 (so J ˉ = ( 1 , 1 ) \bar{J} = (1, 1) J ˉ = ( 1 , 1 ) and Q = 0 Q = 0 Q = 0 ), a forecaster that predicts each half the time has calibration error for the triplet ( k , 1 , 1 ) (k,1,1) ( k , 1 , 1 ) for each of k ∈ { 1 , 2 } k \in\{1,2\} k ∈ { 1 , 2 }
1 2 δ k , 1 , 1 R ( ( 1 , 0 ) , k t ) + 1 2 δ k , 1 , 1 R ( ( 0 , 1 ) , k t ) = ( 1 k t = k − 1 2 ) 1 ⋅ ≤ 0 . \frac{1}{2}\delta^R_{k,1,1}((1,0),k_t)+\frac{1}{2}\delta^R_{k,1,1}((0,1),k_t)
=\left(\mathbf{1}_{k_t=k}-\frac{1}{2}\right)\mathbf{1}_{\cdot\leq0}. 2 1 δ k , 1 , 1 R (( 1 , 0 ) , k t ) + 2 1 δ k , 1 , 1 R (( 0 , 1 ) , k t ) = ( 1 k t = k − 2 1 ) 1 ⋅ ≤ 0 .
which makes it 0 in expectation if k t k_t k t is sampled uniformly from { 1 , 2 } \{1,2\} { 1 , 2 } . Thus, the learner always plays action 1 and earns 1 / 2 1/2 1/2 per round which is suboptimal compared to the true optimal of 3 / 5 3/5 3/5 . Notice that this issue does not benefit the optimizer: each learner behavior is best-responding, so the definition of the BSE (Optimization Program (2) ) already considers this behavior as feasible actions.
For the learner’s stochastic regret, however, the problem is that because both forecasts induce the same action, R T R_T R T only tracks their average ( 1 / 2 , 1 / 2 ) (1/2,1/2) ( 1/2 , 1/2 ) , achieving R T = 0 R_T = 0 R T = 0 without ever forcing the forecaster to predict the actual mixture f = ( 1 / 2 , 1 / 2 ) f = (1/2,1/2) f = ( 1/2 , 1/2 ) . In Section 5, we introduce a calibration error that prevents such errors from canceling by adding conditions that distinguish forecasts where the budget constraint is binding from those where it is not.
We note that this cannot happen if the learner was unbudgeted. In that case, if some action is best-responding to both f = ( 1 , 0 ) f = (1,0) f = ( 1 , 0 ) and f = ( 0 , 1 ) f = (0,1) f = ( 0 , 1 ) , then it is also best-responding to f = ( 1 / 2 , 1 / 2 ) f = (1/2,1/2) f = ( 1/2 , 1/2 ) . In fact, for an unbudgeted learner our response-calibration error reduces to a calibration error that minimizes swap regret; see Appendix § 15 for details.
Response-calibration Error in Unbudgeted Games
In this section, we show how the response-calibration error R T R_T R T in (4) simplifies in unbudgeted games (ρ L = 1 \rho_L = 1 ρ L = 1 or C L = 0 C_L = 0 C L = 0 ), why bounding ∥ R T ∥ L 2 \lVert R_T\rVert_{L^2} ∥ R T ∥ L 2 directly implies low swap regret for the learner in addition to non-manipulability, and how R T R_T R T relates to Anagnostides et al. [4 ] , Bernstein and Shimkin [16 ] , and Rivkin et al. [57 ] .
Simplification in the unbudgeted case. When ρ L = 1 \rho_L = 1 ρ L = 1 or C L = 0 C_L = 0 C L = 0 , every round is active (T = [ T ] \mathcal{T} = [T] T = [ T ] ) and every forecast f ∈ Δ n ^ O f \in\Delta_{\hat{n}_O} f ∈ Δ n ^ O induces a pure best response j ( f ) ∈ arg max j ∈ [ n L ] u ( f ) [ j ] j(f) \in\arg\max_{j \in[n_L]} u(f)[j] j ( f ) ∈ arg max j ∈ [ n L ] u ( f ) [ j ] , represented by J ˉ ( f ) = ( j ( f ) , j ( f ) ) \bar{J}(f) = (j(f),j(f)) J ˉ ( f ) = ( j ( f ) , j ( f )) and Q ( f ) = 0 Q(f) = 0 Q ( f ) = 0 . In (3) , all off-diagonal coordinates ( k , j 1 , j 2 ) (k,j_1,j_2) ( k , j 1 , j 2 ) with j 1 ≠ j 2 j_1 \ne j_2 j 1 = j 2 vanish identically, and each diagonal coordinate ( k , j , j ) (k,j,j) ( k , j , j ) is a scalar multiple of the step function 1 ⋅ ≤ 0 \mathbf{1}_{\cdot\leq0} 1 ⋅ ≤ 0 :
R T , ( k , j , j ) ( q ) = r k , j 1 q ≤ 0 , where r k , j : = ∑ t = 1 T E f ∼ F t [ 1 j ( f ) = j ( 1 k t = k − f k ) ] ∈ R n ^ O × n L . R_{T,(k,j,j)}(q) = r_{k,j}\mathbf{1}_{q\leq0},
\qquad\text{where}\qquad
r_{k,j} := \sum_{t=1}^{T}\mathbb{E}_{f\sim F_t}\left[\mathbf{1}_{j(f)=j}\left(\mathbf{1}_{k_t=k}-f_k\right)\right] \in\mathbb{R}^{\hat{n}_O\times n_L}. R T , ( k , j , j ) ( q ) = r k , j 1 q ≤ 0 , where r k , j := t = 1 ∑ T E f ∼ F t [ 1 j ( f ) = j ( 1 k t = k − f k ) ] ∈ R n ^ O × n L .
Since ∥ 1 ⋅ ≤ 0 ∥ L 2 2 ( [ − 1 , 1 ] ) = ∫ 0 1 1 2 d x = 1 \|1_{\cdot\leq0}\|_{L_2^2([-1,1])}=\int_0^1 1^2\,dx=1 ∥ 1 ⋅ ≤ 0 ∥ L 2 2 ([ − 1 , 1 ]) = ∫ 0 1 1 2 d x = 1 , the function-space norm collapses to the Frobenius norm of the n ^ O × n L \hat n_O \times n_L n ^ O × n L response-conditioned frequency table r r r :
∥ R T ∥ L 2 = ∥ r ∥ F = ∑ k = 1 n ^ O ∑ j = 1 n L r k , j 2 . \|R_T\|_{L_2}=\|r\|_F=\sqrt{\sum_{k=1}^{\hat n_O}\sum_{j=1}^{n_L}r_{k,j}^2}. ∥ R T ∥ L 2 = ∥ r ∥ F = k = 1 ∑ n ^ O j = 1 ∑ n L r k , j 2 .
Why ∥ R T ∥ L 2 \|R_T\|_{L_2} ∥ R T ∥ L 2 bounds both swap regret and strategic manipulation. For any action-modification rule π : [ n L ] → [ n L ] \pi:[n_L]\to[n_L] π : [ n L ] → [ n L ] , optimality of j ( f ) = j j(f)=j j ( f ) = j under the predicted utility vector u ( f ) = ∑ k = 1 n ^ O f k u ( k ) u(f)=\sum_{k=1}^{\hat n_O}f_k u^{(k)} u ( f ) = ∑ k = 1 n ^ O f k u ( k ) implies u ( f ) [ π ( j ) ] ≤ u ( f ) [ j ] u(f)[\pi(j)]\leq u(f)[j] u ( f ) [ π ( j )] ≤ u ( f ) [ j ] . Adding and subtracting u ( f ) u(f) u ( f ) therefore bounds the learner’s swap regret directly in terms of the n L × n L n_L\times n_L n L × n L projected utility-error matrix r ~ j ′ , j : = ∑ k = 1 n ^ O u ( k ) [ j ′ ] r k , j \tilde r_{j',j}:=\sum_{k=1}^{\hat n_O}u^{(k)}[j']r_{k,j} r ~ j ′ , j := ∑ k = 1 n ^ O u ( k ) [ j ′ ] r k , j :
∑ t = 1 T E f ∼ F t [ u ( k t ) [ π ( j ( f ) ) ] − u ( k t ) [ j ( f ) ] ] ≤ ∑ j = 1 n L ∑ t = 1 T E f ∼ F t [ 1 j ( f ) = j ( ( u ( k t ) [ π ( j ) ] − u ( f ) [ π ( j ) ] ) − ( u ( k t ) [ j ] − u ( f ) [ j ] ) ) ] = ∑ j = 1 n L ( r ~ π ( j ) , j − r ~ j , j ) ≤ n ^ O n L ∥ r ∥ F = n ^ O n L ∥ R T ∥ L 2 . \begin{aligned}
\sum_{t=1}^{T}\mathbb{E}_{f\sim F_t}\left[u^{(k_t)}[\pi(j(f))]-u^{(k_t)}[j(f)]\right]
&\leq\sum_{j=1}^{n_L}\sum_{t=1}^{T}\mathbb{E}_{f\sim F_t}\left[1_{j(f)=j}\left(\left(u^{(k_t)}[\pi(j)]-u(f)[\pi(j)]\right)-\left(u^{(k_t)}[j]-u(f)[j]\right)\right)\right]\\
&=\sum_{j=1}^{n_L}\left(\tilde r_{\pi(j),j}-\tilde r_{j,j}\right)\leq\sqrt{\hat n_O n_L}\,\|r\|_F=\sqrt{\hat n_O n_L}\,\|R_T\|_{L_2}.
\end{aligned} t = 1 ∑ T E f ∼ F t [ u ( k t ) [ π ( j ( f ))] − u ( k t ) [ j ( f )] ] ≤ j = 1 ∑ n L t = 1 ∑ T E f ∼ F t [ 1 j ( f ) = j ( ( u ( k t ) [ π ( j )] − u ( f ) [ π ( j )] ) − ( u ( k t ) [ j ] − u ( f ) [ j ] ) ) ] = j = 1 ∑ n L ( r ~ π ( j ) , j − r ~ j , j ) ≤ n ^ O n L ∥ r ∥ F = n ^ O n L ∥ R T ∥ L 2 .
Notice the distinction between learner swap regret and optimizer non-manipulability: To control the learner’s swap regret, it suffices to bound the n L × n L n_L\times n_L n L × n L matrix r ~ j ′ , j = ∑ k = 1 n ^ O u ( k ) [ j ′ ] r k , j \tilde r_{j',j}=\sum_{k=1}^{\hat n_O}u^{(k)}[j']r_{k,j} r ~ j ′ , j = ∑ k = 1 n ^ O u ( k ) [ j ′ ] r k , j , which projects the type errors r k , j r_{k,j} r k , j onto the learner’s known utility vectors u ( k ) ∈ [ 0 , 1 ] n L u^{(k)}\in[0,1]^{n_L} u ( k ) ∈ [ 0 , 1 ] n L . This is the space in which Rivkin et al. [57 ] and Anagnostides et al. [4 ] operate: both works study arbitrary outcome vectors u t ∈ [ 0 , 1 ] n L u_t\in[0,1]^{n_L} u t ∈ [ 0 , 1 ] n L rather than discrete optimizer types k t ∈ [ n ^ O ] k_t\in[\hat n_O] k t ∈ [ n ^ O ] , and neither model includes an unknown optimizer utility matrix U O U_O U O . Specifically, for a fixed best-response map j ( ⋅ ) j(\cdot) j ( ⋅ ) , the partition bias of Rivkin et al. [57 ] is ∑ j = 1 n L ∥ r ~ ⋅ , j ∥ ∞ \sum_{j=1}^{n_L}\|\tilde r_{\cdot,j}\|_\infty ∑ j = 1 n L ∥ r ~ ⋅ , j ∥ ∞ , while Anagnostides et al. [4 ] apply the response-based approachability framework of Bernstein and Shimkin [16 ] directly to outer products u t ⊗ p t u_t\otimes p_t u t ⊗ p t in R n L × n L \mathbb{R}^{n_L\times n_L} R n L × n L to approach conv { u ^ ⊗ e j ( u ^ ) : u ^ ∈ [ 0 , 1 ] n L } \operatorname{conv}\{\hat u\otimes e_{j(\hat u)}:\hat u\in[0,1]^{n_L}\} conv { u ^ ⊗ e j ( u ^ ) : u ^ ∈ [ 0 , 1 ] n L } . Note that this is a special case of the setting of [4 ] , since they examine linear swap regret minimization, i.e., when the action space of the learner is not the simplex.
Equivalence of WBSE and BSE for Generic Games
Here we show that when the learner’s utilities and costs are generic, the learner’s best response B R L ( x ) \mathrm{BR}_L(x) BR L ( x ) is unique for any fixed (or generic) optimizer strategy x ∈ Δ n O x\in\Delta_{n_O} x ∈ Δ n O , even when the budget constraint forces the learner to randomize, and moreover every optimal best-response vertex at any x ∈ Δ n O x\in\Delta_{n_O} x ∈ Δ n O can be uniquely induced by an arbitrarily small perturbation of x x x . As a consequence, the weak and strong Budgeted Stackelberg Equilibrium values coincide (WBSE = BSE).
Throughout this section, because the learner’s null action j 0 ∈ [ n L ] j_0\in[n_L] j 0 ∈ [ n L ] has zero utility and cost (U L [ i , j 0 ] = C L [ i , j 0 ] = 0 U_L[i,j_0]=C_L[i,j_0]=0 U L [ i , j 0 ] = C L [ i , j 0 ] = 0 for all i ∈ [ n O ] i\in[n_O] i ∈ [ n O ] ), we say that a property holds for generic learner utilities and costs ( U L , C L ) (U_L,C_L) ( U L , C L ) if it holds for all ( U L , C L ) ∈ [ 0 , 1 ] n O × n L × [ 0 , 1 ] n O × n L (U_L,C_L)\in[0,1]^{n_O\times n_L}\times[0,1]^{n_O\times n_L} ( U L , C L ) ∈ [ 0 , 1 ] n O × n L × [ 0 , 1 ] n O × n L (subject to U L [ ⋅ , j 0 ] = C L [ ⋅ , j 0 ] = 0 U_L[\cdot,j_0]=C_L[\cdot,j_0]=0 U L [ ⋅ , j 0 ] = C L [ ⋅ , j 0 ] = 0 ) outside a set of Lebesgue measure zero on the unrestricted entries [ n O ] × ( [ n L ] ∖ { j 0 } ) [n_O]\times([n_L]\setminus\{j_0\}) [ n O ] × ([ n L ] ∖ { j 0 }) .
We first characterize the vertices V ( x ) \mathcal{V}(x) V ( x ) of the learner’s budget-feasible polytope F ( x ) = { y ∈ Δ n L : x T C L y ≤ ρ L } \mathcal{F}(x)=\{y\in\Delta_{n_L}:x^\mathsf{T}C_Ly\leq\rho_L\} F ( x ) = { y ∈ Δ n L : x T C L y ≤ ρ L } and explain why randomization due to a binding budget does not create ties in B R L ( x ) \mathrm{BR}_L(x) BR L ( x ) . Recall that F ( x ) \mathcal{F}(x) F ( x ) is the intersection of the simplex Δ n L \Delta_{n_L} Δ n L with a single linear inequality x T C L y ≤ ρ L x^\mathsf{T}C_Ly\leq\rho_L x T C L y ≤ ρ L , so every vertex v ∈ V ( x ) v\in\mathcal{V}(x) v ∈ V ( x ) has support of size at most two:
Pure vertices: v = e j v=e_j v = e j for some j ∈ [ n L ] j\in[n_L] j ∈ [ n L ] satisfying ( x ⊤ C L ) j ≤ ρ L (x^\top C_L)_j\le\rho_L ( x ⊤ C L ) j ≤ ρ L .
Two-action budget-binding vertices: v = v j , j ′ ( x ) ≔ q j , j ′ ( x ) e j + ( 1 − q j , j ′ ( x ) ) e j ′ v=v_{j,j'}(x)\coloneqq q_{j,j'}(x)e_j+(1-q_{j,j'}(x))e_{j'} v = v j , j ′ ( x ) : = q j , j ′ ( x ) e j + ( 1 − q j , j ′ ( x )) e j ′ for a pair of pure actions j , j ′ ∈ [ n L ] j,j'\in[n_L] j , j ′ ∈ [ n L ] with ( x ⊤ C L ) j < ρ L < ( x ⊤ C L ) j ′ (x^\top C_L)_j<\rho_L<(x^\top C_L)_{j'} ( x ⊤ C L ) j < ρ L < ( x ⊤ C L ) j ′ , where the mixing weight
q j , j ′ ( x ) ≔ ( x ⊤ C L ) j ′ − ρ L ( x ⊤ C L ) j ′ − ( x ⊤ C L ) j ∈ ( 0 , 1 ) (36) q_{j,j'}(x)\coloneqq\frac{(x^\top C_L)_{j'}-\rho_L}{(x^\top C_L)_{j'}-(x^\top C_L)_j}\in(0,1) \tag*{(36)} q j , j ′ ( x ) : = ( x ⊤ C L ) j ′ − ( x ⊤ C L ) j ( x ⊤ C L ) j ′ − ρ L ∈ ( 0 , 1 ) (36)
is uniquely determined by the binding budget constraint x ⊤ C L v j , j ′ ( x ) = ρ L x^\top C_Lv_{j,j'}(x)=\rho_L x ⊤ C L v j , j ′ ( x ) = ρ L .
When a two-action vertex v j , j ′ ( x ) v_{j,j'}(x) v j , j ′ ( x ) is optimal for the learner (with ( x ⊤ U L ) j < V L ( x ) < ( x ⊤ U L ) j ′ (x^\top U_L)_j<V_L(x)<(x^\top U_L)_{j'} ( x ⊤ U L ) j < V L ( x ) < ( x ⊤ U L ) j ′ ), both j j j and j ′ j' j ′ lie in the support of the optimal response and share the same Lagrangian payoff ( x ⊤ U L ) k − λ ( x ⊤ C L ) k (x^\top U_L)_k-\lambda(x^\top C_L)_k ( x ⊤ U L ) k − λ ( x ⊤ C L ) k for the optimal dual multiplier λ = ( x ⊤ U L ) j ′ − ( x ⊤ U L ) j ( x ⊤ C L ) j ′ − ( x ⊤ C L ) j > 0 \lambda=\frac{(x^\top U_L)_{j'}-(x^\top U_L)_j}{(x^\top C_L)_{j'}-(x^\top C_L)_j}>0 λ = ( x ⊤ C L ) j ′ − ( x ⊤ C L ) j ( x ⊤ U L ) j ′ − ( x ⊤ U L ) j > 0 . Thus v j , j ′ ( x ) v_{j,j'}(x) v j , j ′ ( x ) is a single point in Δ n L \Delta_{n_L} Δ n L , and BR L ( x ) = conv ( V ( x ) ∩ BR L ( x ) ) \operatorname{BR}_L(x)=\operatorname{conv}(\mathcal{V}(x)\cap\operatorname{BR}_L(x)) BR L ( x ) = conv ( V ( x ) ∩ BR L ( x )) fails to be a singleton if and only if two distinct vertices in V ( x ) \mathcal{V}(x) V ( x ) simultaneously achieve V L ( x ) V_L(x) V L ( x ) .
Lemma H.1. Fix any optimizer strategy x ∈ Δ n O x\in\Delta_{n_O} x ∈ Δ n O . For generic learner utilities and costs ( U L , C L ) (U_L,C_L) ( U L , C L ) , the learner’s best-response set BR L ( x ) \operatorname{BR}_L(x) BR L ( x ) is a singleton. Consequently, for generic ( U L , C L ) (U_L,C_L) ( U L , C L ) , BR L ( x ) \operatorname{BR}_L(x) BR L ( x ) is a singleton for almost every x ∈ Δ n O x\in\Delta_{n_O} x ∈ Δ n O (and for every pure optimizer action x = e i x=e_i x = e i , i ∈ [ n O ] i\in[n_O] i ∈ [ n O ] ).
Proof. For a given x ∈ Δ n O x\in\Delta_{n_O} x ∈ Δ n O , associate each pure action j ∈ [ n L ] j\in[n_L] j ∈ [ n L ] of the learner with its cost-utility pair p j ( x ) ≔ ( ( x ⊤ C L ) j , ( x ⊤ U L ) j ) ∈ [ 0 , 1 ] 2 p_j(x)\coloneqq((x^\top C_L)_j,(x^\top U_L)_j)\in[0,1]^2 p j ( x ) : = (( x ⊤ C L ) j , ( x ⊤ U L ) j ) ∈ [ 0 , 1 ] 2 . By linear programming duality (which holds strongly since p j 0 ( x ) = ( 0 , 0 ) p_{j_0}(x)=(0,0) p j 0 ( x ) = ( 0 , 0 ) and ρ L > 0 \rho_L>0 ρ L > 0 ), we have
V L ( x ) = min λ ≥ 0 max j ∈ [ n L ] ( ( x ⊤ U L ) j − λ ( ( x ⊤ C L ) j − ρ L ) ) . V_L(x)=\min_{\lambda\ge0}\max_{j\in[n_L]}\left((x^\top U_L)_j-\lambda\left((x^\top C_L)_j-\rho_L\right)\right). V L ( x ) = λ ≥ 0 min j ∈ [ n L ] max ( ( x ⊤ U L ) j − λ ( ( x ⊤ C L ) j − ρ L ) ) .
and any y ∈ BR L ( x ) y\in\operatorname{BR}_L(x) y ∈ BR L ( x ) is supported on the set of tight pure actions S ( x , λ ) ≔ arg max j ∈ [ n L ] ( ( x ⊤ U L ) j − λ ( x ⊤ C L ) j ) S(x,\lambda)\coloneqq\operatorname*{arg\,max}_{j\in[n_L]}((x^\top U_L)_j-\lambda(x^\top C_L)_j) S ( x , λ ) : = arg max j ∈ [ n L ] (( x ⊤ U L ) j − λ ( x ⊤ C L ) j ) for an optimal dual multiplier λ ≥ 0 \lambda\ge0 λ ≥ 0 , with x ⊤ C L y = ρ L x^\top C_Ly=\rho_L x ⊤ C L y = ρ L whenever λ > 0 \lambda>0 λ > 0 .
Suppose BR L ( x ) \operatorname{BR}_L(x) BR L ( x ) is not a singleton, so that at least two distinct vertices in V ( x ) \mathcal{V}(x) V ( x ) belong to BR L ( x ) \operatorname{BR}_L(x) BR L ( x ) . If λ = 0 \lambda=0 λ = 0 is an optimal dual multiplier, then at least two distinct pure actions j 1 ≠ j 2 j_1\ne j_2 j 1 = j 2 must tie for the maximum utility:
( x ⊤ U L ) j 1 − ( x ⊤ U L ) j 2 = 0. (37) (x^\top U_L)_{j_1}-(x^\top U_L)_{j_2}=0. \tag*{(37)} ( x ⊤ U L ) j 1 − ( x ⊤ U L ) j 2 = 0. (37)
If every optimal dual multiplier satisfies λ > 0 \lambda>0 λ > 0 , then every y ∈ BR L ( x ) y\in\operatorname{BR}_L(x) y ∈ BR L ( x ) satisfies x ⊤ C L y = ρ L x^\top C_Ly=\rho_L x ⊤ C L y = ρ L and is supported on S ( x , λ ) S(x,\lambda) S ( x , λ ) . If ∣ S ( x , λ ) ∣ = 1 |S(x,\lambda)|=1 ∣ S ( x , λ ) ∣ = 1 , say S ( x , λ ) = { j } S(x,\lambda)=\{j\} S ( x , λ ) = { j } , then BR L ( x ) = { e j } \operatorname{BR}_L(x)=\{e_j\} BR L ( x ) = { e j } is a singleton. If ∣ S ( x , λ ) ∣ = 2 |S(x,\lambda)|=2 ∣ S ( x , λ ) ∣ = 2 , say S ( x , λ ) = { j , j ′ } S(x,\lambda)=\{j,j'\} S ( x , λ ) = { j , j ′ } , then the equation q ( x ⊤ C L ) j + ( 1 − q ) ( x ⊤ C L ) j ′ = ρ L q(x^\top C_L)_j+(1-q)(x^\top C_L)_{j'}=\rho_L q ( x ⊤ C L ) j + ( 1 − q ) ( x ⊤ C L ) j ′ = ρ L has a unique solution q = q j , j ′ ( x ) ∈ [ 0 , 1 ] q=q_{j,j'}(x)\in[0,1] q = q j , j ′ ( x ) ∈ [ 0 , 1 ] unless ( x ⊤ C L ) j = ( x ⊤ C L ) j ′ = ρ L (x^\top C_L)_j=(x^\top C_L)_{j'}=\rho_L ( x ⊤ C L ) j = ( x ⊤ C L ) j ′ = ρ L , which requires
( x ⊤ C L ) j − ρ L = 0. (38) (x^\top C_L)_j-\rho_L=0. \tag*{(38)} ( x ⊤ C L ) j − ρ L = 0. (38)
Finally, if ∣ S ( x , λ ) ∣ ≥ 3 |S(x,\lambda)|\ge3 ∣ S ( x , λ ) ∣ ≥ 3 , then there exist three distinct pure actions j 1 , j 2 , j 3 ∈ [ n L ] j_1,j_2,j_3\in[n_L] j 1 , j 2 , j 3 ∈ [ n L ] lying on the same supporting line of slope λ \lambda λ in the cost-utility plane, which means p j 1 ( x ) , p j 2 ( x ) , p j 3 ( x ) p_{j_1}(x),p_{j_2}(x),p_{j_3}(x) p j 1 ( x ) , p j 2 ( x ) , p j 3 ( x ) are collinear:
det ( 1 ( x ⊤ C L ) j 1 ( x ⊤ U L ) j 1 1 ( x ⊤ C L ) j 2 ( x ⊤ U L ) j 2 1 ( x ⊤ C L ) j 3 ( x ⊤ U L ) j 3 ) = 0. (39) \det\begin{pmatrix}
1 & (x^\top C_L)_{j_1} & (x^\top U_L)_{j_1}\\
1 & (x^\top C_L)_{j_2} & (x^\top U_L)_{j_2}\\
1 & (x^\top C_L)_{j_3} & (x^\top U_L)_{j_3}
\end{pmatrix}=0. \tag*{(39)} det 1 1 1 ( x ⊤ C L ) j 1 ( x ⊤ C L ) j 2 ( x ⊤ C L ) j 3 ( x ⊤ U L ) j 1 ( x ⊤ U L ) j 2 ( x ⊤ U L ) j 3 = 0. (39)
For any fixed x ∈ Δ n O x\in\Delta_{n_O} x ∈ Δ n O (having at least one positive coordinate x i > 0 x_i>0 x i > 0 ), each of the finitely many conditions (37)–(39) over distinct tuples of actions in [ n L ] [n_L] [ n L ] is a non-trivial polynomial equation in the unrestricted entries of ( U L , C L ) (U_L,C_L) ( U L , C L ) , and hence defines a hypersurface of Lebesgue measure zero. Outside the finite union of these measure-zero hypersurfaces, BR L ( x ) \operatorname{BR}_L(x) BR L ( x ) is a singleton. The second claim follows immediately by Fubini’s theorem (and by taking a union over the n O n_O n O pure strategies x = e i x=e_i x = e i ).
When the optimizer chooses x ∈ Δ n O x \in\Delta_{n_O} x ∈ Δ n O strategically, as in the definition of the BSE in (2) or (26) , the optimizer’s best choice may be an x x x on a boundary between best-response regions where the learner is indifferent between multiple vertices of F ( x ) \mathcal{F}(x) F ( x ) . We next show that for generic ( U L , C L ) (U_L,C_L) ( U L , C L ) , no optimal vertex of F ( x ) \mathcal{F}(x) F ( x ) is isolated or degenerate: any optimal vertex v ∈ V ( x ) ∩ B R L ( x ) v \in\mathcal{V}(x) \cap\mathrm{BR}_L(x) v ∈ V ( x ) ∩ BR L ( x ) can be selected as the unique best response by an arbitrarily small perturbation of x x x .
Lemma H.2. For generic learner utilities and costs ( U L , C L ) (U_L,C_L) ( U L , C L ) , the following holds for every x ∈ Δ n O x \in\Delta_{n_O} x ∈ Δ n O and every optimal vertex v ∈ V ( x ) ∩ B R L ( x ) v \in\mathcal{V}(x) \cap\mathrm{BR}_L(x) v ∈ V ( x ) ∩ BR L ( x ) : there exists a sequence of optimizer strategies { x ε } ε > 0 ⊆ Δ n O \{x_\varepsilon\}_{\varepsilon>0} \subseteq\Delta_{n_O} { x ε } ε > 0 ⊆ Δ n O with lim ε → 0 x ε = x \lim_{\varepsilon\to0} x_\varepsilon= x lim ε → 0 x ε = x such that B R L ( x ε ) = { v ε } \mathrm{BR}_L(x_\varepsilon) = \{v_\varepsilon\} BR L ( x ε ) = { v ε } is a singleton for every ε > 0 \varepsilon> 0 ε > 0 and lim ε → 0 v ε = v \lim_{\varepsilon\to0} v_\varepsilon= v lim ε → 0 v ε = v .
Proof. Fix any x ∈ Δ n O x \in\Delta_{n_O} x ∈ Δ n O , and let support S x = supp ( x ) ⊆ [ n O ] S_x = \operatorname{supp}(x) \subseteq[n_O] S x = supp ( x ) ⊆ [ n O ] of size s = ∣ S x ∣ ≥ 1 s = |S_x| \ge1 s = ∣ S x ∣ ≥ 1 , so that x x x lies in the ( s − 1 ) (s-1) ( s − 1 ) -dimensional open simplex Δ S x ∘ : = { x ′ ∈ Δ n O : supp ( x ′ ) = S x } \Delta^\circ_{S_x} := \{x' \in\Delta_{n_O} : \operatorname{supp}(x') = S_x\} Δ S x ∘ := { x ′ ∈ Δ n O : supp ( x ′ ) = S x } with tangent space T S x : = { d ∈ R n O : supp ( d ) ⊆ S x , 1 ⊤ d = 0 } T_{S_x} := \{d \in\mathbb{R}^{n_O} : \operatorname{supp}(d) \subseteq S_x,\ \mathbf{1}^{\top}d = 0\} T S x := { d ∈ R n O : supp ( d ) ⊆ S x , 1 ⊤ d = 0 } . Let v ∈ V ( x ) ∩ B R L ( x ) v \in\mathcal{V}(x) \cap\mathrm{BR}_L(x) v ∈ V ( x ) ∩ BR L ( x ) be any optimal vertex of V ( x ) \mathcal{V}(x) V ( x ) that is a best response to x x x , and let λ ≥ 0 \lambda\ge0 λ ≥ 0 be a corresponding optimal dual multiplier with intercept τ : = V L ( x ) − λ ρ L \tau:= V_L(x) - \lambda\rho_L τ := V L ( x ) − λ ρ L and tight action set S : = { j ∈ [ n L ] : ( x ⊤ U L ) j − λ ( x ⊤ C L ) j = τ } S := \{j \in[n_L] : (x^{\top}U_L)_j - \lambda(x^{\top}C_L)_j = \tau\} S := { j ∈ [ n L ] : ( x ⊤ U L ) j − λ ( x ⊤ C L ) j = τ } .
Case 1: Unconstrained utility maximum (λ = 0 \lambda= 0 λ = 0 and v = e j ⋆ v = e_{j^\star} v = e j ⋆ with ( x ⊤ C L ) j ⋆ ≤ ρ L (x^{\top}C_L)_{j^\star} \le\rho_L ( x ⊤ C L ) j ⋆ ≤ ρ L ). Here S = { j ∈ [ n L ] : x S x ⊤ u j = V L ( x ) } S = \{j \in[n_L] : x_{S_x}^{\top}u_j = V_L(x)\} S = { j ∈ [ n L ] : x S x ⊤ u j = V L ( x )} . Note that for generic U L U_L U L , every non-null action has positive utility entries, so V L ( x ) > 0 V_L(x) > 0 V L ( x ) > 0 and j 0 ∉ S j_0 \notin S j 0 ∈ / S . Every action j ∈ S ∖ { j ⋆ } j \in S \setminus\{j^\star\} j ∈ S ∖ { j ⋆ } is tied with j ⋆ j^\star j ⋆ at x x x , meaning
x S x ⊤ ( u j − u j ⋆ ) = 0 for all j ∈ S ∖ { j ⋆ } . x_{S_x}^{\top}(u_j-u_{j^\star})=0 \qquad\text{for all }j\in S\setminus\{j^\star\}. x S x ⊤ ( u j − u j ⋆ ) = 0 for all j ∈ S ∖ { j ⋆ } .
For generic U L U_L U L , any at most s s s difference vectors of the form u j − u j ⋆ u_j-u_{j^\star} u j − u j ⋆ in R s \mathbb{R}^s R s are linearly independent. Since the nonzero vector x S x ∈ R s x_{S_x} \in\mathbb{R}^s x S x ∈ R s is orthogonal to all ∣ S ∣ − 1 |S|-1 ∣ S ∣ − 1 vectors { u j − u j ⋆ } j ∈ S ∖ { j ⋆ } \{u_j-u_{j^\star}\}_{j\in S\setminus\{j^\star\}} { u j − u j ⋆ } j ∈ S ∖ { j ⋆ } , there can be at most ∣ S ∣ − 1 ≤ s − 1 |S|-1 \le s-1 ∣ S ∣ − 1 ≤ s − 1 such vectors (so ∣ S ∣ ≤ s |S| \le s ∣ S ∣ ≤ s ), and because 1 ⊤ x S x = 1 ≠ 0 \mathbf{1}^{\top}x_{S_x}=1 \ne0 1 ⊤ x S x = 1 = 0 , the all-ones vector 1 ∈ R s \mathbf{1} \in\mathbb{R}^s 1 ∈ R s does not lie in their span. Consequently, the ∣ S ∣ |S| ∣ S ∣ vectors { 1 } ∪ { u j − u j ⋆ } j ∈ S ∖ { j ⋆ } \{\mathbf{1}\} \cup\{u_j-u_{j^\star}\}_{j\in S\setminus\{j^\star\}} { 1 } ∪ { u j − u j ⋆ } j ∈ S ∖ { j ⋆ } in R s \mathbb{R}^s R s are linearly independent (and if ( x ⊤ C L ) j ⋆ = ρ L (x^{\top}C_L)_{j^\star}=\rho_L ( x ⊤ C L ) j ⋆ = ρ L , the vector c j ⋆ − ρ L 1 c_{j^\star}-\rho_L\mathbf{1} c j ⋆ − ρ L 1 , which is also orthogonal to x S x x_{S_x} x S x and has generic entries, is likewise linearly independent of them, with ∣ S ∣ ≤ s − 1 |S| \le s-1 ∣ S ∣ ≤ s − 1 ). Therefore, the linear system
1 ⊤ d = 0 , ( u j − u j ⋆ ) ⊤ d = − 1 ( ∀ j ∈ S ∖ { j ⋆ } ) \mathbf{1}^{\top}d=0,\qquad(u_j-u_{j^\star})^{\top}d=-1\quad(\forall j\in S\setminus\{j^\star\}) 1 ⊤ d = 0 , ( u j − u j ⋆ ) ⊤ d = − 1 ( ∀ j ∈ S ∖ { j ⋆ })
(together with c j ⋆ ⊤ d = − 1 c_{j^\star}^{\top}d=-1 c j ⋆ ⊤ d = − 1 if ( x ⊤ C L ) j ⋆ = ρ L (x^{\top}C_L)_{j^\star}=\rho_L ( x ⊤ C L ) j ⋆ = ρ L ) has a solution d ∈ R s d \in\mathbb{R}^s d ∈ R s . For all sufficiently small ε > 0 \varepsilon>0 ε > 0 , x ε = x + ε d ∈ Δ n O x_\varepsilon=x+\varepsilon d\in\Delta_{n_O} x ε = x + ε d ∈ Δ n O satisfies ( x ε ⊤ C L ) j ⋆ < ρ L (x_\varepsilon^{\top}C_L)_{j^\star}<\rho_L ( x ε ⊤ C L ) j ⋆ < ρ L and ( x ε ⊤ U L ) j ⋆ > ( x ε ⊤ U L ) j (x_\varepsilon^{\top}U_L)_{j^\star}>(x_\varepsilon^{\top}U_L)_j ( x ε ⊤ U L ) j ⋆ > ( x ε ⊤ U L ) j for all j ≠ j ⋆ j\ne j^\star j = j ⋆ , so B R L ( x ε ) = { e j ⋆ } \mathrm{BR}_L(x_\varepsilon)=\{e_{j^\star}\} BR L ( x ε ) = { e j ⋆ } .
Case 2: Budget-binding vertex ( x ⊤ C L v = ρ L ) (x^{\top}C_Lv=\rho_L) ( x ⊤ C L v = ρ L ) . First suppose v = v j 1 , j 2 ( x ) v=v_{j_1,j_2}(x) v = v j 1 , j 2 ( x ) is a two-action vertex with j 1 , j 2 ∈ S j_1,j_2\in S j 1 , j 2 ∈ S and x S x ⊤ c j 1 < ρ L < x S x ⊤ c j 2 x_{S_x}^{\top}c_{j_1}<\rho_L<x_{S_x}^{\top}c_{j_2} x S x ⊤ c j 1 < ρ L < x S x ⊤ c j 2 , where the supporting line through p j 1 ( x ) p_{j_1}(x) p j 1 ( x ) and p j 2 ( x ) p_{j_2}(x) p j 2 ( x ) has slope
λ = x S x ⊤ ( u j 2 − u j 1 ) x S x ⊤ ( c j 2 − c j 1 ) . \lambda=\frac{x_{S_x}^{\top}(u_{j_2}-u_{j_1})}{x_{S_x}^{\top}(c_{j_2}-c_{j_1})}. λ = x S x ⊤ ( c j 2 − c j 1 ) x S x ⊤ ( u j 2 − u j 1 ) .
For each additional tight action j ∈ S ∖ { j 1 , j 2 } j\in S\setminus\{j_1,j_2\} j ∈ S ∖ { j 1 , j 2 } , let
β j ( x ) : = x S x ⊤ ( c j − c j 1 ) x S x ⊤ ( c j 2 − c j 1 ) \beta_j(x):=\frac{x_{S_x}^{\top}(c_j-c_{j_1})}{x_{S_x}^{\top}(c_{j_2}-c_{j_1})} β j ( x ) := x S x ⊤ ( c j 2 − c j 1 ) x S x ⊤ ( c j − c j 1 )
denote the cost-interpolation weight of j j j relative to j 1 j_1 j 1 and j 2 j_2 j 2 , and define the vector
w j ( x ) : = ( ( u j − u j 1 ) − λ ( c j − c j 1 ) ) − β j ( x ) ( ( u j 2 − u j 1 ) − λ ( c j 2 − c j 1 ) ) ∈ R s . w_j(x):=((u_j-u_{j_1})-\lambda(c_j-c_{j_1}))-\beta_j(x)((u_{j_2}-u_{j_1})-\lambda(c_{j_2}-c_{j_1}))\in\mathbb{R}^s. w j ( x ) := (( u j − u j 1 ) − λ ( c j − c j 1 )) − β j ( x ) (( u j 2 − u j 1 ) − λ ( c j 2 − c j 1 )) ∈ R s .
Notice that for any x ′ ∈ Δ n O x'\in\Delta_{n_O} x ′ ∈ Δ n O supported on S x S_x S x , the vertical distance of p j ( x ′ ) p_j(x') p j ( x ′ ) above the line passing through p j 1 ( x ′ ) p_{j_1}(x') p j 1 ( x ′ ) and p j 2 ( x ′ ) p_{j_2}(x') p j 2 ( x ′ ) is ( x S x ′ ) ⊤ w j ( x ′ ) (x'_{S_x})^{\top}w_j(x') ( x S x ′ ) ⊤ w j ( x ′ ) . At x ′ = x x'=x x ′ = x , since j , j 1 , j 2 ∈ S j,j_1,j_2\in S j , j 1 , j 2 ∈ S all lie on the supporting line of slope λ \lambda λ , both terms in the definition of w j ( x ) w_j(x) w j ( x ) are orthogonal to x S x x_{S_x} x S x , so
x S x ⊤ w j ( x ) = 0 for all j ∈ S ∖ { j 1 , j 2 } . x_{S_x}^{\top}w_j(x)=0\qquad\text{for all }j\in S\setminus\{j_1,j_2\}. x S x ⊤ w j ( x ) = 0 for all j ∈ S ∖ { j 1 , j 2 } .
and differentiating ( x S x ⊤ w j ( x ′ ) ) (x_{S_x}^{\top}w_j(x')) ( x S x ⊤ w j ( x ′ )) at x ′ = x x'=x x ′ = x along a perturbation x ε = x + ε d x_\varepsilon=x+\varepsilon d x ε = x + ε d (using x S x ⊤ ( ( u j 2 − u j 1 ) − λ ( c j 2 − c j 1 ) ) = 0 x_{S_x}^{\top}((u_{j_2}-u_{j_1})-\lambda(c_{j_2}-c_{j_1}))=0 x S x ⊤ (( u j 2 − u j 1 ) − λ ( c j 2 − c j 1 )) = 0 ) shows that the first-order change in the vertical distance of p j ( x ε ) p_j(x_\varepsilon) p j ( x ε ) above the line through p j 1 ( x ε ) p_{j_1}(x_\varepsilon) p j 1 ( x ε ) and p j 2 ( x ε ) p_{j_2}(x_\varepsilon) p j 2 ( x ε ) is simply d ⊤ w j ( x ) d^{\top}w_j(x) d ⊤ w j ( x ) .
Crucially, for each j ∈ S ∖ { j 1 , j 2 } j\in S\setminus\{j_1,j_2\} j ∈ S ∖ { j 1 , j 2 } , the vector w j ( x ) w_j(x) w j ( x ) equals u j u_j u j minus a vector that does not depend on u j u_j u j (since neither λ \lambda λ nor β j ( x ) \beta_j(x) β j ( x ) depends on u j u_j u j , and at most one action in S S S can be the null action j 0 j_0 j 0 , which can be taken as j 1 j_1 j 1 if j 0 ∈ S j_0\in S j 0 ∈ S ). Consequently, for generic ( U L , C L ) (U_L,C_L) ( U L , C L ) , the vectors { w j ( x ) } j ∈ S ∖ { j 1 , j 2 } \{w_j(x)\}_{j\in S\setminus\{j_1,j_2\}} { w j ( x ) } j ∈ S ∖ { j 1 , j 2 } in the ( s − 1 ) (s-1) ( s − 1 ) -dimensional subspace { w ∈ R s : w ⊤ x S x = 0 } \{w\in\mathbb{R}^s:w^{\top}x_{S_x}=0\} { w ∈ R s : w ⊤ x S x = 0 } are linearly independent; in particular, there can be at most ∣ S ∣ − 2 ≤ s − 1 |S|-2\le s-1 ∣ S ∣ − 2 ≤ s − 1 such vectors (so ∣ S ∣ ≤ s + 1 |S|\le s+1 ∣ S ∣ ≤ s + 1 ), and because 1 ⊤ x S x = 1 ≠ 0 \mathbf{1}^{\top}x_{S_x}=1\ne0 1 ⊤ x S x = 1 = 0 , the vectors { 1 } ∪ { w j ( x ) } j ∈ S ∖ { j 1 , j 2 } \{\mathbf{1}\}\cup\{w_j(x)\}_{j\in S\setminus\{j_1,j_2\}} { 1 } ∪ { w j ( x ) } j ∈ S ∖ { j 1 , j 2 } are linearly independent in R s \mathbb{R}^s R s . Therefore, the linear system
1 ⊤ d = 0 , w j ( x ) ⊤ d = − 1 ( ∀ j ∈ S ∖ { j 1 , j 2 } ) \mathbf{1}^{\top}d=0,\qquad w_j(x)^{\top}d=-1\quad(\forall j\in S\setminus\{j_1,j_2\}) 1 ⊤ d = 0 , w j ( x ) ⊤ d = − 1 ( ∀ j ∈ S ∖ { j 1 , j 2 })
has a solution d ∈ R s d\in\mathbb{R}^s d ∈ R s (and if λ = 0 \lambda=0 λ = 0 , i.e., x S x ⊤ ( u j 2 − u j 1 ) = 0 x_{S_x}^{\top}(u_{j_2}-u_{j_1})=0 x S x ⊤ ( u j 2 − u j 1 ) = 0 , the vector u j 2 − u j 1 u_{j_2}-u_{j_1} u j 2 − u j 1 is also orthogonal to x S x x_{S_x} x S x and linearly independent of them, so we can additionally require ( u j 2 − u j 1 ) ⊤ d = 1 (u_{j_2}-u_{j_1})^{\top}d=1 ( u j 2 − u j 1 ) ⊤ d = 1 to ensure the perturbed slope λ ε \lambda_\varepsilon λ ε is strictly positive). For ε > 0 \varepsilon>0 ε > 0 sufficiently small, let λ ε > 0 \lambda_\varepsilon>0 λ ε > 0 and τ ε ∈ R \tau_\varepsilon\in\mathbb{R} τ ε ∈ R be the slope and intercept of the line passing through p j 1 ( x ε ) p_{j_1}(x_\varepsilon) p j 1 ( x ε ) and p j 2 ( x ε ) p_{j_2}(x_\varepsilon) p j 2 ( x ε ) . By construction, for every j ∈ S ∖ { j 1 , j 2 } j\in S\setminus\{j_1,j_2\} j ∈ S ∖ { j 1 , j 2 } , the vertical distance of p j ( x ε ) p_j(x_\varepsilon) p j ( x ε ) above this line is ε d ⊤ w j ( x ) + O ( ε 2 ) = − ε + O ( ε 2 ) < 0 \varepsilon d^{\top}w_j(x)+O(\varepsilon^2)=-\varepsilon+O(\varepsilon^2)<0 ε d ⊤ w j ( x ) + O ( ε 2 ) = − ε + O ( ε 2 ) < 0 for all small ε > 0 , w h i l e a l l \varepsilon>0, while all ε > 0 , w hi l e a l l j\notin S$ remain strictly below the line and ( x ε ⊤ C L ) j 1 < ρ L < ( x ε ⊤ C L ) j 2 (x_\varepsilon^{\top}C_L)_{j_1}<\rho_L<(x_\varepsilon^{\top}C_L)_{j_2} ( x ε ⊤ C L ) j 1 < ρ L < ( x ε ⊤ C L ) j 2 holds by continuity. Hence S ( x ε , λ ε ) = { j 1 , j 2 } S(x_\varepsilon,\lambda_\varepsilon)=\{j_1,j_2\} S ( x ε , λ ε ) = { j 1 , j 2 } , so BR L ( x ε ) = { v j 1 , j 2 ( x ε ) } \operatorname{BR}_L(x_\varepsilon)=\{v_{j_1,j_2}(x_\varepsilon)\} BR L ( x ε ) = { v j 1 , j 2 ( x ε )} is a singleton, and by continuity of q j 1 , j 2 q_{j_1,j_2} q j 1 , j 2 in (36) we have lim ε → 0 v j 1 , j 2 ( x ε ) = v j 1 , j 2 ( x ) = v \lim_{\varepsilon\to0}v_{j_1,j_2}(x_\varepsilon)=v_{j_1,j_2}(x)=v lim ε → 0 v j 1 , j 2 ( x ε ) = v j 1 , j 2 ( x ) = v .
Finally, if v = e j 1 v=e_{j_1} v = e j 1 is a boundary pure vertex with j 1 ∈ S j_1\in S j 1 ∈ S and x S x ⊤ c j 1 = ρ L x_{S_x}^{\top}c_{j_1}=\rho_L x S x ⊤ c j 1 = ρ L , then if S = { j 1 } S=\{j_1\} S = { j 1 } we already have BR L ( x ) = { e j 1 } \operatorname{BR}_L(x)=\{e_{j_1}\} BR L ( x ) = { e j 1 } ; otherwise, picking any j 2 ∈ S j_2\in S j 2 ∈ S with x S x ⊤ c j 2 > ρ L x_{S_x}^{\top}c_{j_2}>\rho_L x S x ⊤ c j 2 > ρ L (or reducing to Case 1 if none exists), the additional vector c j 1 − ρ L 1 c_{j_1}-\rho_L\mathbf{1} c j 1 − ρ L 1 is also orthogonal to x S x x_{S_x} x S x and linearly independent of { 1 } ∪ { w j ( x ) } j ∈ S ∖ { j 1 , j 2 } \{\mathbf{1}\}\cup\{w_j(x)\}_{j\in S\setminus\{j_1,j_2\}} { 1 } ∪ { w j ( x ) } j ∈ S ∖ { j 1 , j 2 } , so we can additionally require c j 1 ⊤ d = − 1 c_{j_1}^{\top}d=-1 c j 1 ⊤ d = − 1 . Then ( x ε ⊤ C L ) j 1 = ρ L − ε < ρ L (x_\varepsilon^{\top}C_L)_{j_1}=\rho_L-\varepsilon<\rho_L ( x ε ⊤ C L ) j 1 = ρ L − ε < ρ L and S ( x ε , λ ε ) = { j 1 , j 2 } S(x_\varepsilon,\lambda_\varepsilon)=\{j_1,j_2\} S ( x ε , λ ε ) = { j 1 , j 2 } , so BR L ( x ε ) = { v j 1 , j 2 ( x ε ) } \operatorname{BR}_L(x_\varepsilon)=\{v_{j_1,j_2}(x_\varepsilon)\} BR L ( x ε ) = { v j 1 , j 2 ( x ε )} is a singleton with q j 1 , j 2 ( x ε ) → 1 q_{j_1,j_2}(x_\varepsilon)\to1 q j 1 , j 2 ( x ε ) → 1 and v j 1 , j 2 ( x ε ) → e j 1 = v v_{j_1,j_2}(x_\varepsilon)\to e_{j_1}=v v j 1 , j 2 ( x ε ) → e j 1 = v as ε → 0 \varepsilon\to0 ε → 0 . ■
Combining Lemma H.2 with the two-phase structure of (26) and (28) immediately establishes that WBSE and BSE coincide for generic games.
Proof of Theorem 7.5. Clearly, WBSE ≤ \le ≤ BSE. To prove the reverse inequality WBSE ≥ \ge ≥ BSE, recall from the proof of Lemma 7.1 that BSE = max ( u , c ) ∈ conv ( S ) : c ≤ ρ O u \operatorname{BSE}=\max_{(u,c)\in\operatorname{conv}(S):c\le\rho_O}u BSE = max ( u , c ) ∈ conv ( S ) : c ≤ ρ O u , where S = { ( x ⊤ U O y , x ⊤ C O y ) : x ∈ Δ n O , y ∈ BR L ( x ) } S=\{(x^{\top}U_Oy,x^{\top}C_Oy):x\in\Delta_{n_O},y\in\operatorname{BR}_L(x)\} S = {( x ⊤ U O y , x ⊤ C O y ) : x ∈ Δ n O , y ∈ BR L ( x )} . For any fixed x ∈ Δ n O x\in\Delta_{n_O} x ∈ Δ n O , the map y ↦ ( x ⊤ U O y , x ⊤ C O y ) y\mapsto(x^{\top}U_Oy,x^{\top}C_Oy) y ↦ ( x ⊤ U O y , x ⊤ C O y ) is linear and BR L ( x ) = conv ( V ( x ) ∩ BR L ( x ) ) \operatorname{BR}_L(x)=\operatorname{conv}(\mathcal{V}(x)\cap\operatorname{BR}_L(x)) BR L ( x ) = conv ( V ( x ) ∩ BR L ( x )) , so conv ( S ) = conv ( S v e r t ) \operatorname{conv}(S)=\operatorname{conv}(S^{\mathrm{vert}}) conv ( S ) = conv ( S vert ) where
S v e r t : = { ( x ⊤ U O v , x ⊤ C O v ) : x ∈ Δ n O , v ∈ V ( x ) ∩ BR L ( x ) } . S^{\mathrm{vert}}:=\{(x^{\top}U_Ov,x^{\top}C_Ov):x\in\Delta_{n_O},v\in\mathcal{V}(x)\cap\operatorname{BR}_L(x)\}. S vert := {( x ⊤ U O v , x ⊤ C O v ) : x ∈ Δ n O , v ∈ V ( x ) ∩ BR L ( x )} .
Therefore, there exist x 1 ∗ , x 2 ∗ ∈ Δ n O x_1^*,x_2^*\in\Delta_{n_O} x 1 ∗ , x 2 ∗ ∈ Δ n O , optimal vertices v 1 ∗ ∈ V ( x 1 ∗ ) ∩ BR L ( x 1 ∗ ) v_1^*\in\mathcal{V}(x_1^*)\cap\operatorname{BR}_L(x_1^*) v 1 ∗ ∈ V ( x 1 ∗ ) ∩ BR L ( x 1 ∗ ) and v 2 ∗ ∈ V ( x 2 ∗ ) ∩ BR L ( x 2 ∗ ) v_2^*\in\mathcal{V}(x_2^*)\cap\operatorname{BR}_L(x_2^*) v 2 ∗ ∈ V ( x 2 ∗ ) ∩ BR L ( x 2 ∗ ) , and q ∗ ∈ [ 0 , 1 ] q^*\in[0,1] q ∗ ∈ [ 0 , 1 ] that attain BSE in (26) , with u k ∗ : = ( x k ∗ ) ⊤ U O v k ∗ u_k^*:=(x_k^*)^{\top}U_Ov_k^* u k ∗ := ( x k ∗ ) ⊤ U O v k ∗ and c k ∗ : = ( x k ∗ ) ⊤ C O v k ∗ c_k^*:=(x_k^*)^{\top}C_Ov_k^* c k ∗ := ( x k ∗ ) ⊤ C O v k ∗ for k ∈ { 1 , 2 } k\in\{1,2\} k ∈ { 1 , 2 } . By Lemma H.2, for each k ∈ { 1 , 2 } k\in\{1,2\} k ∈ { 1 , 2 } there exists a sequence x k , ε ∈ Δ n O x_{k,\varepsilon}\in\Delta_{n_O} x k , ε ∈ Δ n O with x k , ε → x k ∗ x_{k,\varepsilon}\to x_k^* x k , ε → x k ∗ such that BR L ( x k , ε ) = { v k , ε } \operatorname{BR}_L(x_{k,\varepsilon})=\{v_{k,\varepsilon}\} BR L ( x k , ε ) = { v k , ε } is a singleton and v k , ε → v k ∗ v_{k,\varepsilon}\to v_k^* v k , ε → v k ∗ as ε → 0 \varepsilon\to0 ε → 0 . Because BR L ( x k , ε ) \operatorname{BR}_L(x_{k,\varepsilon}) BR L ( x k , ε ) is a singleton, pessimistic and optimistic tie-breaking coincide at x k , ε x_{k,\varepsilon} x k , ε :
min y ∈ BR L ( x k , ε ) x k , ε ⊤ U O y = x k , ε ⊤ U O v k , ε = : u k , ε → ε → 0 u k ∗ , max y ∈ BR L ( x k , ε ) x k , ε ⊤ C O y = x k , ε ⊤ C O v k , ε = : c k , ε → ε → 0 c k ∗ . \min_{y\in\operatorname{BR}_L(x_{k,\varepsilon})}x_{k,\varepsilon}^{\top}U_Oy=x_{k,\varepsilon}^{\top}U_Ov_{k,\varepsilon}=:u_{k,\varepsilon}\xrightarrow{\varepsilon\to0}u_k^*,\qquad
\max_{y\in\operatorname{BR}_L(x_{k,\varepsilon})}x_{k,\varepsilon}^{\top}C_Oy=x_{k,\varepsilon}^{\top}C_Ov_{k,\varepsilon}=:c_{k,\varepsilon}\xrightarrow{\varepsilon\to0}c_k^*. y ∈ BR L ( x k , ε ) min x k , ε ⊤ U O y = x k , ε ⊤ U O v k , ε =: u k , ε ε → 0 u k ∗ , y ∈ BR L ( x k , ε ) max x k , ε ⊤ C O y = x k , ε ⊤ C O v k , ε =: c k , ε ε → 0 c k ∗ .
If min { c 1 ∗ , c 2 ∗ } < ρ O < max { c 1 ∗ , c 2 ∗ } \min\{c_1^*,c_2^*\}<\rho_O<\max\{c_1^*,c_2^*\} min { c 1 ∗ , c 2 ∗ } < ρ O < max { c 1 ∗ , c 2 ∗ } (say c 1 ∗ < ρ O < c 2 ∗ c_1^*<\rho_O<c_2^* c 1 ∗ < ρ O < c 2 ∗ ), then c 1 , ε < ρ O < c 2 , ε c_{1,\varepsilon}<\rho_O<c_{2,\varepsilon} c 1 , ε < ρ O < c 2 , ε for all sufficiently small ε > 0 \varepsilon>0 ε > 0 ; setting q ε : = c 2 , ε − ρ O c 2 , ε − c 1 , ε → q ∗ q_\varepsilon:=\frac{c_{2,\varepsilon}-\rho_O}{c_{2,\varepsilon}-c_{1,\varepsilon}}\to q^* q ε := c 2 , ε − c 1 , ε c 2 , ε − ρ O → q ∗ gives a feasible solution ( x 1 , ε , x 2 , ε , q ε ) (x_{1,\varepsilon},x_{2,\varepsilon},q_\varepsilon) ( x 1 , ε , x 2 , ε , q ε ) to (28) whose objective value q ε u 1 , ε + ( 1 − q ε ) u 2 , ε q_\varepsilon u_{1,\varepsilon}+(1-q_\varepsilon)u_{2,\varepsilon} q ε u 1 , ε + ( 1 − q ε ) u 2 , ε converges to q ∗ u 1 ∗ + ( 1 − q ∗ ) u 2 ∗ = BSE q^*u_1^*+(1-q^*)u_2^*=\operatorname{BSE} q ∗ u 1 ∗ + ( 1 − q ∗ ) u 2 ∗ = BSE . Otherwise, at least one of the two points (say ( x 1 ∗ , v 1 ∗ ) (x_1^*,v_1^*) ( x 1 ∗ , v 1 ∗ ) ) satisfies both c 1 ∗ ≤ ρ O c_1^*\le\rho_O c 1 ∗ ≤ ρ O and u 1 ∗ = BSE u_1^*=\operatorname{BSE} u 1 ∗ = BSE . Pairing x 1 , ε x_{1,\varepsilon} x 1 , ε with the optimizer’s null action x 2 = e i 0 x_2=e_{i_0} x 2 = e i 0 (which has cost 0 regardless of the learner’s response) and weight q ε : = min { 1 , ρ O / c 1 , ε } → 1 q_{\varepsilon} := \min\{1, \rho_O/c_{1,\varepsilon}\} \to1 q ε := min { 1 , ρ O / c 1 , ε } → 1 yields a feasible solution ( x 1 , ε , e i 0 , q ε ) (x_{1,\varepsilon}, e_{i_0}, q_{\varepsilon}) ( x 1 , ε , e i 0 , q ε ) to (28) with objective value at least q ε u 1 , ε → u 1 ⋆ = B S E q_{\varepsilon}u_{1,\varepsilon} \to u_1^\star= \mathrm{BSE} q ε u 1 , ε → u 1 ⋆ = BSE . In either case, taking ε → 0 \varepsilon\to0 ε → 0 shows W B S E ≥ B S E \mathrm{WBSE} \ge\mathrm{BSE} WBSE ≥ BSE , completing the proof. ■