Introduction
Contextual bandits are widely used in applications where an action must be chosen after observing information about the current user or environment, including recommendation systems, online content selection, and personalized interventions [39 ] (Li et al. 2010). In such applications a new policy usually cannot be deployed simply to learn how well it performs, since online experimentation is costly, risky, and slow. Off-policy evaluation avoids this by estimating the value of a target policy from data collected under a different behavior policy [11 , 41 ] .
In practice, the available data need not come from a single behavior policy. Deployed systems are updated over time, several experiments may run in parallel, and different loggers may use different decision rules. Methods for combining observations from several loggers generally take the logger sample sizes as given [1 , 28 ] (He et al. 2019). In many applications, however, more data can still be collected, but only from loggers that already exist, and these loggers differ both in the actions they tend to take and in their per-observation prices. An inexpensive logger may concentrate on a few actions, while a more costly logger may provide the coverage that a target policy needs. The practical question is where the remaining observations should be purchased when the reward distribution is not yet known. We study a fixed set of logging policies μ 1 , … , μ K \mu_1,\ldots,\mu_K μ 1 , … , μ K with known costs c 1 , … , c K c_1,\ldots,c_K c 1 , … , c K . The policies cannot be redesigned, and acquisition chooses only which logger to sample from next, subject to a finite budget.
This problem differs from behavior-policy design, in which the logging or augmentation policy is itself chosen to reduce the variance of a downstream estimator [23 , 40 , 35 ] . When the loggers are fixed, an allocation is a count vector N = ( N 1 , … , N K ) ∈ Z + K N=(N_1,\ldots,N_K)\in\mathbb{Z}_{+}^{K} N = ( N 1 , … , N K ) ∈ Z + K with c T N ≤ B c^{\mathsf{T}}N\le B c T N ≤ B , where N k N_k N k is the number of observations purchased from logger k k k and B B B is the total budget. Each purchase increments one coordinate of N N N and bears the corresponding logger cost.
A second feature is that several target policies may have to be evaluated from the same sample. We consider a finite target class Π = { π 1 , … , π M } \Pi=\{\pi_1,\ldots,\pi_M\} Π = { π 1 , … , π M } and seek a design that controls the largest target-specific uncertainty over that class. The criterion retains each policy’s target-specific efficient variance and minimizes its maximum, so the objective can change as the target attaining that maximum changes. Prior work on evaluating multiple policies jointly provides related motivation, although it studies different sampling settings and objectives [9 , 5 ] .
The criterion depends on the conditional reward mean and variance, which are unknown at the outset and must be estimated from the observations acquired along the way. We propose a sequential rule, which we call Adaptive Greedy , that re-estimates the criterion as data accrue and at each step purchases from the affordable logger with the largest estimated reduction in the worst-target criterion per unit of cost. An additional feature is that the rule does not plan toward a declared final budget. The budget enters only through which loggers remain affordable and when acquisition stops. This is useful if the available budget is unexpectedly reduced or the process must stop early, since the data already collected were chosen to improve the current design rather than to follow a plan optimized for a larger final budget. Planners that repeatedly solve for count vectors at a declared final budget behave differently, since their early purchases are chosen with that budget in mind.
For a fixed logger composition, weighting each observation by the overall mixture of the logging policies has no larger asymptotic variance than weighting it only by the logger that generated it. We therefore combine all acquired observations using this pooled estimator. Under adaptive data collection, however, the final logger composition depends on the rewards observed along the way, so the pooled denominator is itself random. We show that as the design converges, this random denominator can be replaced at first order by its deterministic limiting value. As a result, the pooled estimator remains asymptotically normal and its large-sample variance is determined by the allocation reached by the algorithm.
Our contributions are threefold. First, we formulate budgeted off-policy evaluation as a sequential allocation problem among loggers with heterogeneous costs and propose the Adaptive Greedy algorithm. Second, assuming overlap only for the pooled behavior distribution at the optimal design, rather than for every logger, we show that the resulting design converges almost surely to the continuous minimax design. Under stronger conditions on the plug-in estimates, we also derive an explicit polynomial rate. Third, we establish joint asymptotic normality and consistent covariance estimation for the pooled estimator, even though the logger choices depend on previously observed rewards. The variance advantage of pooled over logger-specific weighting is known for fixed designs [1 , 28 ] . Our result shows that pooled inference remains valid after adaptive data collection under the stated conditions.
Related Work
Off-policy evaluation from several loggers has so far been studied with the logger sample sizes given in advance. Agarwal et al. [1 ] observe that weighting each observation by the pooled behavior distribution never increases variance relative to weighting it by the logger that generated it. He et al. (2019) study off-policy learning from data collected by multiple loggers, and Lai et al. [29 ] choose variance-minimizing weights for combining estimators computed from several behavior policies. Kallus et al. [28 ] characterize the semiparametric efficiency bound for the fixed-stratified multi-logger model and construct an estimator that attains it. This pooled bound, viewed as a function of the logger composition, is the population objective of our design problem.
Related mixture-design problems have also been studied in importance sampling. He and Owen (2014) establish convexity of variance objectives for mixtures of fixed proposal distributions and discuss minimizing the maximum variance when estimating multiple integrals from shared samples. Our setting instead concerns sequential purchases from fixed contextual-bandit loggers with heterogeneous costs, where allocation decisions depend on the observations acquired so far.
A second line of work designs or learns the data-collection policy itself for downstream evaluation. Examples include behavior-policy search ([23 , 31 ] ), experimental design in linear heteroscedastic bandits ([35 ] ), safe or robust data-collection designs ([46 , 34 , 32 , 4 ] ), and the optimization of logging probabilities ([10 ] ). In contextual bandits, Tucker and Joachims [40 ] design an augmentation logging policy for an existing dataset and evaluate with a balanced estimator that divides by the pooled behavior distribution of the original and augmentation data. In all of these works the behavior or augmentation policy is a design variable, whereas in our setting the loggers are fixed and only the number of purchases from each can be chosen.
Evaluating several target policies from shared data is a closely related problem. Dann et al. [9 ] study high-confidence evaluation of multiple policies in episodic reinforcement learning, Jain et al. [27 ] adaptively learn a behavior policy for evaluating several general value functions, and Chen et al. [5 ] optimize an offline sampling distribution for several policies in an episodic MDP. Liu et al. [32 ] and Russo and Pacchiano [38 ] also study data collection when there are several evaluation targets. These works motivate designing for a finite target class jointly, which in our setting takes the form of a minimax criterion over the target-specific pooled efficiency bounds.
Inference from data collected by outcome-adaptive rules has been developed for batched bandits, adaptive experiments, and off-policy evaluation from contextual bandits ([45 , 19 , 44 , 3 , 42 ] ). In these works the weights and normalizations entering the estimator are predictable, meaning that they are known when each observation is collected, and martingale arguments apply to the resulting estimating equations. The pooled estimator studied here divides by the final logger composition, which depends on rewards observed along the acquisition path and is known only after acquisition ends. Design convergence allows us to replace this random final mixture by its deterministic limit.
The cost-normalized feasible set and the minimax variance criterion connect the problem to classical optimal design ([14 , 37 ] ). The sequential update has a conditional-gradient interpretation related to the Fedorov–Wynn design algorithms ([43 , 6 ] ), with vertices fixed by the available loggers and a step size set by the cost of the logger purchased. § 4.1 develops this connection.
Preliminaries
Problem Setup
Let X \mathcal{X} X be a measurable context space and let A \mathcal{A} A be a finite action space. There are K K K existing logging policies (or loggers) μ 1 ( a ∣ x ) , … , μ K ( a ∣ x ) \mu_1(a \mid x), \ldots, \mu_K(a \mid x) μ 1 ( a ∣ x ) , … , μ K ( a ∣ x ) , and acquiring one observation from logger k k k costs c k > 0 c_k > 0 c k > 0 . If an observation is purchased from logger k k k , it is generated as
X ∼ P X , A ∼ μ k ( ⋅ ∣ X ) , R ∼ P R ( ⋅ ∣ X , A ) . (1) X \sim P_X,\qquad A \sim\mu_k(\cdot\mid X),\qquad R \sim P_R(\cdot\mid X,A). \tag*{(1)} X ∼ P X , A ∼ μ k ( ⋅ ∣ X ) , R ∼ P R ( ⋅ ∣ X , A ) . (1)
The context distribution P X P_X P X and conditional reward distribution P R P_R P R are common across loggers. Write q ( x , a ) = E ( R ∣ X = x , A = a ) q(x,a) = \mathbb{E}(R \mid X = x, A = a) q ( x , a ) = E ( R ∣ X = x , A = a ) and σ 2 ( x , a ) = Var ( R ∣ X = x , A = a ) \sigma^2(x,a) = \operatorname{Var}(R \mid X = x, A = a) σ 2 ( x , a ) = Var ( R ∣ X = x , A = a ) for the conditional reward mean and variance. Let Π = { π 1 , … , π M } \Pi= \{\pi_1,\ldots,\pi_M\} Π = { π 1 , … , π M } be the finite target policy class. Our estimand is the vector of policy values J = ( J 1 , … , J M ) ⊤ J = (J_1,\ldots,J_M)^\top J = ( J 1 , … , J M ) ⊤ , which we estimate after acquisition is complete, where J j = E X { q j ( X ) } J_j = \mathbb{E}_X\{q_j(X)\} J j = E X { q j ( X )} and q j ( x ) = ∑ a ∈ A π j ( a ∣ x ) q ( x , a ) q_j(x) = \sum_{a \in\mathcal{A}} \pi_j(a \mid x)q(x,a) q j ( x ) = ∑ a ∈ A π j ( a ∣ x ) q ( x , a ) . Only context-action pairs used by some target policy affect the policy values. Let S = { ( x , a ) : π j ( a ∣ x ) > 0 for some j } \mathcal{S} = \{(x,a) : \pi_j(a \mid x) > 0 \text{ for some } j\} S = {( x , a ) : π j ( a ∣ x ) > 0 for some j } be the union of the target-policy supports. For a function f f f of ( x , a ) (x,a) ( x , a ) , we write ess inf S f \operatorname{ess\,inf}_{\mathcal{S}} f ess inf S f for the infimum of f ( x , a ) f(x,a) f ( x , a ) over ( x , a ) ∈ S (x,a) \in\mathcal{S} ( x , a ) ∈ S , after removing a P X P_X P X -null set of contexts.
Data are acquired sequentially, and each logger is chosen using only the data observed before that choice. For observation i i i , let K i ∈ { 1 , … , K } K_i \in\{1,\ldots,K\} K i ∈ { 1 , … , K } denote the selected logger, and A i A_i A i for the contextual-bandit action generated by that logger. Let
F i = σ { K 1 , X 1 , A 1 , R 1 , … , K i , X i , A i , R i } , \mathcal{F}_i = \sigma\{K_1, X_1, A_1, R_1,\ldots,K_i, X_i, A_i, R_i\}, F i = σ { K 1 , X 1 , A 1 , R 1 , … , K i , X i , A i , R i } ,
with F 0 \mathcal{F}_0 F 0 trivial, be the history till time i i i . If external randomization is used to select a logger (e.g., to break a tie between two or more potential loggers), the filtration is assumed to include that randomization before the corresponding logger choice. Thus K i K_i K i is F i − 1 \mathcal{F}_{i-1} F i − 1 -measurable. Conditional on F i − 1 \mathcal{F}_{i-1} F i − 1 and K i = k K_i = k K i = k , the observation ( X i , A i , R i ) (X_i,A_i,R_i) ( X i , A i , R i ) is generated according to (1) . We impose the following structural assumption, which is standard in off-policy evaluation with multiple loggers; see Dudík et al. [11 ] and Kallus et al. [28 ] .
Assumption 1. The logging probabilities μ 1 , … , μ K \mu_1,\ldots,\mu_K μ 1 , … , μ K and the costs c 1 , … , c K c_1,\ldots,c_K c 1 , … , c K are known, 0 < c min = min k c k ≤ max k c k = c max < ∞ 0 < c_{\min} = \min_k c_k \le\max_k c_k = c_{\max} < \infty 0 < c m i n = min k c k ≤ max k c k = c m a x < ∞ , and ∣ R ∣ ≤ R max < ∞ |R| \le R_{\max} < \infty ∣ R ∣ ≤ R m a x < ∞ almost surely.
With total budget B B B , acquisition continues until no logger remains affordable. Let N t , k = ∑ i = 1 t 1 { K i = k } N_{t,k} = \sum_{i=1}^{t} 1\{K_i = k\} N t , k = ∑ i = 1 t 1 { K i = k } count the number of observations acquired from logger k k k . Let C t = ∑ k = 1 K c k N t , k C_t = \sum_{k=1}^{K} c_kN_{t,k} C t = ∑ k = 1 K c k N t , k denote the expenditure after t t t observations. Write N t = ( N t , 1 , … , N t , K ) ⊤ N_t = (N_{t,1},\ldots,N_{t,K})^\top N t = ( N t , 1 , … , N t , K ) ⊤ , so that ∑ k N t , k = t \sum_k N_{t,k} = t ∑ k N t , k = t . Acquisition stops at
T B = min { t ≥ 0 : c min > B − C t } , T_B = \min\{t \ge0 : c_{\min} > B - C_t\}, T B = min { t ≥ 0 : c m i n > B − C t } ,
which is a stopping time with respect to ( F t ) (\mathcal{F}_t) ( F t ) . It is easy to see that T B T_B T B is bounded between ⌊ ( B − c min ) / c max ⌋ \lfloor(B-c_{\min})/c_{\max}\rfloor ⌊( B − c m i n ) / c m a x ⌋ and ⌈ B / c min ⌉ \lceil B/c_{\min}\rceil ⌈ B / c m i n ⌉ , and in particular T B → ∞ T_B \to\infty T B → ∞ as B → ∞ B \to\infty B → ∞ .
Fixed Logger Sizes
The purpose of this subsection is to obtain the variance criterion that we will later use for acquisition. Here we fix the sample size t t t and the sequence of loggers k 1 , … , k t k_1,\ldots,k_t k 1 , … , k t with k i ∈ { 1 , … , K } k_i \in\{1,\ldots,K\} k i ∈ { 1 , … , K } . Let N k = N t , k N_k = N_{t,k} N k = N t , k be the resulting number of observations from logger k k k , and N = ( N 1 , … , N K ) ⊤ N = (N_1,\ldots,N_K)^\top N = ( N 1 , … , N K ) ⊤ . Thus ∑ k N k = t \sum_k N_k = t ∑ k N k = t , the counts are deterministic, and no acquisition decision depends on the observed data. This is the fixed-stratified setting studied by Kallus et al. [28 ] .
Since all loggers share P X P_X P X and P R P_R P R , we can weight every observation by the mixture of the logging policies, as if the combined sample came from that mixture. This weighting never gives a larger variance than weighting each observation by the logger that generated it (Agarwal et al., 2017; [28 ] ). The mixture is the pooled behavior distribution
μ ˉ N ( a ∣ x ) = 1 t ∑ k = 1 K N k μ k ( a ∣ x ) . (2) \bar{\mu}_{N}(a \mid x) = \frac{1}{t}\sum_{k=1}^{K} N_k\mu_k(a \mid x). \tag*{(2)} μ ˉ N ( a ∣ x ) = t 1 k = 1 ∑ K N k μ k ( a ∣ x ) . (2)
We define the pooled augmented inverse propensity weighted (AIPW) estimator of J j J_j J j as
J ~ t , j = 1 t ∑ i = 1 t [ q j ( X i ) + π j ( A i ∣ X i ) μ ˉ N ( A i ∣ X i ) { R i − q ( X i , A i ) } ] . (3) \widetilde{J}_{t,j} = \frac{1}{t}\sum_{i=1}^{t}\left[q_j(X_i) + \frac{\pi_j(A_i \mid X_i)}{\bar{\mu}_{N}(A_i \mid X_i)}\{R_i-q(X_i,A_i)\}\right]. \tag*{(3)} J t , j = t 1 i = 1 ∑ t [ q j ( X i ) + μ ˉ N ( A i ∣ X i ) π j ( A i ∣ X i ) { R i − q ( X i , A i )} ] . (3)
Kallus et al. [28 ] show that this weighting, together with the true q q q , minimizes variance within the estimator class in the fixed-stratified setting.
For a fixed logger sequence k 1 , … , k t k_1,\ldots,k_t k 1 , … , k t with ess inf S μ ˉ N > 0 \operatorname{ess\,inf}_S\bar{\mu}_N>0 ess inf S μ ˉ N > 0 , the pooled estimator is unbiased, E ( J ~ t , j ) = J j \mathbb{E}(\widetilde{J}_{t,j})=J_j E ( J t , j ) = J j , and its fixed-design variance is
Var ( J ~ t , j ) = Var { q j ( X ) } t + E X ∑ a ∈ A π j ( a ∣ X ) 2 σ 2 ( X , a ) ∑ k = 1 K N k μ k ( a ∣ X ) . (4) \operatorname{Var}(\widetilde{J}_{t,j}) = \frac{\operatorname{Var}\{q_j(X)\}}{t} + \mathbb{E}_X\sum_{a\in\mathcal{A}}\frac{\pi_j(a\mid X)^2\sigma^2(X,a)}{\sum_{k=1}^{K}N_k\mu_k(a\mid X)}. \tag*{(4)} Var ( J t , j ) = t Var { q j ( X )} + E X a ∈ A ∑ ∑ k = 1 K N k μ k ( a ∣ X ) π j ( a ∣ X ) 2 σ 2 ( X , a ) . (4)
This variance formula follows from Kallus et al. [28 ] . For notational convenience, let V j = Var { q j ( X ) } V_j=\operatorname{Var}\{q_j(X)\} V j = Var { q j ( X )} and W j ( x , a ) = π j ( a ∣ x ) 2 σ 2 ( x , a ) W_j(x,a)=\pi_j(a\mid x)^2\sigma^2(x,a) W j ( x , a ) = π j ( a ∣ x ) 2 σ 2 ( x , a ) . Note that the variance does not depend on the order in which the loggers appear but only through the count vector N N N . Write θ k = N k / t \theta_k=N_k/t θ k = N k / t for the sample fractions, which lie in the simplex S K = { θ ∈ R + K : 1 T θ = 1 } S_K=\{\theta\in\mathbb{R}_+^K:\mathbf{1}^{\mathsf{T}}\theta=1\} S K = { θ ∈ R + K : 1 T θ = 1 } , so that μ ˉ N = m θ \bar{\mu}_N=m_\theta μ ˉ N = m θ with m θ ( a ∣ x ) = ∑ k = 1 K θ k μ k ( a ∣ x ) m_\theta(a\mid x)=\sum_{k=1}^{K}\theta_k\mu_k(a\mid x) m θ ( a ∣ x ) = ∑ k = 1 K θ k μ k ( a ∣ x ) . Then (4) becomes Var ( J ~ t , j ) = A j ( θ ) / t \operatorname{Var}(\widetilde{J}_{t,j})=A_j(\theta)/t Var ( J t , j ) = A j ( θ ) / t with
A j ( θ ) = V j + E X ∑ a ∈ A W j ( X , a ) m θ ( a ∣ X ) . (5) A_j(\theta)=V_j+\mathbb{E}_X\sum_{a\in\mathcal{A}}\frac{W_j(X,a)}{m_\theta(a\mid X)}. \tag*{(5)} A j ( θ ) = V j + E X a ∈ A ∑ m θ ( a ∣ X ) W j ( X , a ) . (5)
Thus, for a fixed logger composition θ \theta θ , a smaller A j ( θ ) A_j(\theta) A j ( θ ) means a smaller variance for estimating J j J_j J j . Kallus et al. [28 ] derive the efficient influence function and the semiparametric efficiency bound for the fixed-stratified multi-logger model. Their bound for π j \pi_j π j is exactly A j ( θ ) A_j(\theta) A j ( θ ) . Hence A j ( θ ) A_j(\theta) A j ( θ ) is the smallest regular asymptotic variance attainable for estimating J j J_j J j under a fixed logger composition θ \theta θ with ess inf S m θ > 0 \operatorname{ess\,inf}_S m_\theta>0 ess inf S m θ > 0 .2 This efficiency result is for the fixed-stratified setting and does not automatically extend to our adaptive procedure, where the final logger composition depends on the observed data. We nevertheless use A j ( θ ) A_j(\theta) A j ( θ ) to construct the acquisition objective, and we establish valid post-acquisition inference in § 4.2 for the pooled estimator computed from the acquired sample.
Methodology
In this section, we develop our sequential method, Adaptive Greedy . We break it into three parts: § 3.1 defines the design criterion that we eventually minimize, § 3.2 describes our greedy approach to selecting the loggers adaptively and how we estimate the unknown quantities, and finally, we state the complete algorithm in § 3.3 . All proofs and derivations are provided in the appendices.
Design Criterion
To compare allocations and decide where the next observation is most useful, we use the variance formula (4) as our criterion, with constants added so that it is finite at N = 0 N=0 N = 0 . Let N = N= N = ( N 1 , … , N K ) ⊤ (N_1,\ldots,N_K)^\top ( N 1 , … , N K ) ⊤ be a count vector. For target policy π j \pi_j π j , we define
G j ( N ) = V j 1 ⊤ N + η 0 + E X ∑ a ∈ A W j ( X , a ) D N ( X , a ) , (6) G_j(N)=\frac{V_j}{\mathbf{1}^\top N+\eta_0}+\mathbb{E}_{X}\sum_{a\in\mathcal{A}}\frac{W_j(X,a)}{D_N(X,a)}, \tag*{(6)} G j ( N ) = 1 ⊤ N + η 0 V j + E X a ∈ A ∑ D N ( X , a ) W j ( X , a ) , (6)
where D N ( x , a ) = η + ∑ k = 1 K N k μ k ( a ∣ x ) D_N(x,a)=\eta+\sum_{k=1}^{K}N_k\mu_k(a\mid x) D N ( x , a ) = η + ∑ k = 1 K N k μ k ( a ∣ x ) and η , η 0 > 0 \eta,\eta_0>0 η , η 0 > 0 . In our experiments we use the simple choice η = η 0 = 1 \eta=\eta_0=1 η = η 0 = 1 throughout. Since every target in Π \Pi Π should be estimated accurately, we control the worst target-specific variance G ( N ) = max 1 ≤ j ≤ M G j ( N ) G(N)=\max_{1\le j\le M}G_j(N) G ( N ) = max 1 ≤ j ≤ M G j ( N ) . Other choices can also be considered, for example M − 1 ∑ j = 1 M G j ( N ) M^{-1}\sum_{j=1}^{M}G_j(N) M − 1 ∑ j = 1 M G j ( N ) , but the average can be small while one target is estimated poorly. Note that a new observation reduces the variance if it adds coverage for regions where D N ( x , a ) D_N(x,a) D N ( x , a ) is small and W j ( X , a ) W_j(X,a) W j ( X , a ) is large.
To compare allocations independently of how much has been spent, we normalize the counts by their cost. Define z N = N / C N z_N=N/C_N z N = N / C N , where C N = c ⊤ N C_N=c^\top N C N = c ⊤ N is the total expenditure. Thus z N , k z_{N,k} z N , k is the number of observations purchased from logger k k k per unit of spent budget. Since c ⊤ z N = 1 c^\top z_N=1 c ⊤ z N = 1 , let C = { z ∈ R + K : c ⊤ z = 1 } \mathcal{C}=\{z\in\mathbb{R}_+^K:c^\top z=1\} C = { z ∈ R + K : c ⊤ z = 1 } . Multiplying G j ( N ) G_j(N) G j ( N ) by the spent budget C N C_N C N removes the decrease in variance that comes simply from spending more money. For a large spent budget, C N G j ( N ) ≈ G j ( z N ) C_NG_j(N)\approx\mathcal{G}_j(z_N) C N G j ( N ) ≈ G j ( z N ) , where G j , G : C → R ∪ { ∞ } \mathcal{G}_j,\mathcal{G}:\mathcal{C}\to\mathbb{R}\cup\{\infty\} G j , G : C → R ∪ { ∞ } are
G j ( z ) = V j 1 ⊤ z + E X ∑ a ∈ A W j ( X , a ) μ ( X , a ) ⊤ z , G ( z ) = max 1 ≤ j ≤ M G j ( z ) , z ∈ C , (7) \mathcal{G}_j(z)=\frac{V_j}{\mathbf{1}^\top z}+\mathbb{E}_{X}\sum_{a\in\mathcal{A}}\frac{W_j(X,a)}{\mu(X,a)^\top z},\qquad
\mathcal{G}(z)=\max_{1\le j\le M}\mathcal{G}_j(z),\qquad z\in\mathcal{C}, \tag*{(7)} G j ( z ) = 1 ⊤ z V j + E X a ∈ A ∑ μ ( X , a ) ⊤ z W j ( X , a ) , G ( z ) = 1 ≤ j ≤ M max G j ( z ) , z ∈ C , (7)
with μ ( x , a ) = ( μ 1 ( a ∣ x ) , … , μ K ( a ∣ x ) ) ⊤ \mu(x,a)=(\mu_1(a\mid x),\ldots,\mu_K(a\mid x))^\top μ ( x , a ) = ( μ 1 ( a ∣ x ) , … , μ K ( a ∣ x ) ) ⊤ . For any fixed positive lower bound on the pooled coverage, Lemma A3.8 shows that there is a finite constant K G K_G K G such that
∣ C N G j ( N ) − G j ( z N ) ∣ ≤ K G C N . \left|C_NG_j(N)-\mathcal{G}_j(z_N)\right|\leq\frac{K_G}{C_N}. ∣ C N G j ( N ) − G j ( z N ) ∣ ≤ C N K G .
for every nonzero count vector with such coverage. We will work with G j ( N ) G_j(N) G j ( N ) for our algorithm as it is more convenient and we will use G j \mathcal{G}_j G j when we talk about convergence in § 4 .
We now define the optimal allocation we would like our algorithm to converge to. Since each G j \mathcal{G}_j G j is convex in C \mathcal{C} C whenever the denominator is positive (as V j ≥ 0 V_j\geq0 V j ≥ 0 , W j ( X , a ) ≥ 0 W_j(X,a)\geq0 W j ( X , a ) ≥ 0 , and x ↦ 1 / x x\mapsto1/x x ↦ 1/ x is convex for x > 0 x>0 x > 0 ), G = max j G j \mathcal{G}=\max_j\mathcal{G}_j G = max j G j is also convex. If V j V_j V j and W j W_j W j are known, define
z ⋆ ∈ arg min v ∈ C G ( v ) , θ ⋆ = z ⋆ 1 ⊤ z ⋆ . (8) z^\star\in\underset{v\in\mathcal{C}}{\arg\min}\mathcal{G}(v),\qquad
\theta^\star=\frac{z^\star}{\mathbf{1}^\top z^\star}. \tag*{(8)} z ⋆ ∈ v ∈ C arg min G ( v ) , θ ⋆ = 1 ⊤ z ⋆ z ⋆ . (8)
Thus z ⋆ z^\star z ⋆ gives the allocation per unit of budget that minimizes the largest target-specific variance, and θ ⋆ \theta^\star θ ⋆ gives the corresponding fractions. For a fixed logger composition, A j ( θ ) A_j(\theta) A j ( θ ) is also the semiparametric efficiency bound discussed in § 2.2 , so this choice has the same fixed-design interpretation after accounting for the logger costs. We discuss the uniqueness of z ⋆ z^\star z ⋆ in § 4.1 .
The fixed-design variance (4) is finite under our bounded-reward assumption when the pooled probabilities are bounded away from zero on the target-policy supports, even if individual loggers are not. We measure this coverage by
κ ( θ ) = ess inf ( x , a ) ∈ S m θ ( a ∣ x ) . (9) \kappa(\theta)=\underset{(x,a)\in\mathcal{S}}{\operatorname{ess\,inf}}\,m_\theta(a\mid x). \tag*{(9)} κ ( θ ) = ( x , a ) ∈ S ess inf m θ ( a ∣ x ) . (9)
When the contexts take finitely many values, for example finitely many subgroups, κ ( θ ) \kappa(\theta) κ ( θ ) is a minimum of finitely many linear functions of θ \theta θ and can be computed exactly. For general context spaces, forced sampling can instead use a known lower bound on κ ( θ ) \kappa(\theta) κ ( θ ) , for example a finite minimum of nonnegative weighted sums of the θ k \theta_k θ k (Appendix § 9.2 ). We impose this condition only at an optimal design.
Assumption 2. There is a minimizer z ⋆ z^\star z ⋆ in (8) whose composition θ ⋆ \theta^\star θ ⋆ satisfies κ ( θ ⋆ ) ≥ κ ⋆ \kappa(\theta^\star)\geq\kappa_\star κ ( θ ⋆ ) ≥ κ ⋆ for some $\kappa_\star>0.
This assumption allows individual loggers to assign zero probability to target-relevant actions, provided their mixture at the optimal design has sufficient coverage. Agarwal et al. [1 ] note that their balanced estimator needs only that the average logging policy covers the target, and Kallus et al. [28 ] distinguish this pooled overlap from the stronger requirement that every logger covers the target, which logger-specific weights need (see also He et al. (2019)). Per-logger lower bounds remain common in multi-logger evaluation [29 , 17 ] . If each logger is deployed with μ k ( a ∣ x ) ≥ μ 0 \mu_k(a \mid x) \ge\mu_0 μ k ( a ∣ x ) ≥ μ 0 on S S S for a constant μ 0 > 0 \mu_0 > 0 μ 0 > 0 , as Agarwal et al. [1 ] suggest for ensuring support, then m θ ( a ∣ x ) = ∑ k θ k μ k ( a ∣ x ) ≥ μ 0 ∑ k θ k = μ 0 m_\theta(a \mid x) = \sum_k \theta_k\mu_k(a \mid x) \ge\mu_0\sum_k \theta_k = \mu_0 m θ ( a ∣ x ) = ∑ k θ k μ k ( a ∣ x ) ≥ μ 0 ∑ k θ k = μ 0 for every composition θ \theta θ , so κ ( θ ) ≥ μ 0 \kappa(\theta) \ge\mu_0 κ ( θ ) ≥ μ 0 and Assumption 2 holds with κ ⋆ = μ 0 \kappa_\star= \mu_0 κ ⋆ = μ 0 . Assumption 2 is a quantitative form of pooled overlap, imposed only at θ ⋆ \theta^\star θ ⋆ , and individual loggers may give probability zero to actions that the targets use. Since θ k ⋆ ≤ 1 \theta_k^\star\le1 θ k ⋆ ≤ 1 , it implies ∑ k μ k ( a ∣ x ) ≥ κ ⋆ \sum_k \mu_k(a \mid x) \ge\kappa_\star ∑ k μ k ( a ∣ x ) ≥ κ ⋆ on S S S , so the uniform mixture of the loggers also covers the targets.
Sequential Selection Rule
To choose the next logger, we compare the reduction in the worst-target variance G G G that one more observation from each logger would bring, per unit of cost. Initially, we start with budget B B B and no data. At time t t t , the remaining budget is B − C t B-C_t B − C t and we choose the next logger from the set K t = { k : c k ≤ B − C t } \mathcal{K}_t = \{k : c_k \le B-C_t\} K t = { k : c k ≤ B − C t } . Now, given the current allocation N t N_t N t , we want to purchase the next logger from K t \mathcal{K}_t K t that provides the largest reduction in G G G . Therefore, for any count vector N N N , define the reduction from purchasing one observation from logger k k k by
Δ k ( N ) = G ( N ) − G ( N + e k ) . (10) \Delta_k(N) = G(N) - G(N + e_k). \tag*{(10)} Δ k ( N ) = G ( N ) − G ( N + e k ) . (10)
If the reward moments were known, the optimal allocation could be obtained by solving
min N ∈ Z + K c ⊤ N ≤ B G ( N ) . (11) \min_{\substack{N \in\mathbb{Z}_+^K \\ c^\top N \le B}} G(N). \tag*{(11)} N ∈ Z + K c ⊤ N ≤ B min G ( N ) . (11)
However, solving this integer optimization problem is computationally expensive for large budgets. Moreover, V j V_j V j and W j W_j W j are unknown in our setting. We therefore consider the simpler greedy rule
K t + 1 g r e e d y ∈ arg max k ∈ K t Δ k ( N t ) c k , (12) K_{t+1}^{\mathrm{greedy}} \in\arg\max_{k \in\mathcal{K}_t} \frac{\Delta_k(N_t)}{c_k}, \tag*{(12)} K t + 1 greedy ∈ arg k ∈ K t max c k Δ k ( N t ) , (12)
which chooses the affordable logger giving the largest immediate reduction in G G G per unit cost.
The rule (12) cannot be used directly, for two reasons. First, the reductions are small and unknown. When the pooled coverage is bounded below, G j ( N t ) G_j(N_t) G j ( N t ) is of order t − 1 t^{-1} t − 1 while the target-specific reduction Δ j , k ( N t ) = G j ( N t ) − G j ( N t + e k ) \Delta_{j,k}(N_t) = G_j(N_t) - G_j(N_t + e_k) Δ j , k ( N t ) = G j ( N t ) − G j ( N t + e k ) is of order t − 2 t^{-2} t − 2 (Appendix § 8.2 ), so the reductions behave like derivatives of the criterion. They depend on the unknown V j V_j V j and W j W_j W j , and the rule must still rank them correctly across loggers. Second, G G G is a maximum over targets. Its one-step change Δ k ( N ) \Delta_k(N) Δ k ( N ) is not a combination of the target-specific changes, and G G G is not smooth where the maximizing target switches.
We handle the second difficulty by replacing the maximum with a log-sum-exp approximation [36 ] . This puts exponential weights on the target-specific reductions and introduces a bias of at most ρ log M \rho\log M ρ log M , which vanishes as the smoothing parameter ρ t \rho_t ρ t decreases. We handle the first by plugging estimates of the reward mean and variance into the closed form of the reductions. Our theory needs these estimates to be accurate on the t − 2 t^{-2} t − 2 scale of the reductions, which is Assumption 4 in § 4.1 . The two fixes interact: as ρ t \rho_t ρ t decreases, the weights become more sensitive to estimation error, so ρ t \rho_t ρ t must decrease slowly relative to the estimation error (see Theorem 4.1).
Estimation. Writing T = 1 T N T = 1^{\mathsf{T}}N T = 1 T N , the target-specific reduction has the closed form (see Appendix § 8.1 for derivation)
Δ j , k ( N ) = G j ( N ) − G j ( N + e k ) = E X ∑ a ∈ A W j ( X , a ) μ k ( a ∣ X ) D N ( X , a ) { D N ( X , a ) + μ k ( a ∣ X ) } + V j ( T + η 0 ) ( T + 1 + η 0 ) . (13) \Delta_{j,k}(N) = G_j(N)-G_j(N+e_k) = \mathbb{E}_{X}\sum_{a\in\mathcal{A}}\frac{W_j(X,a)\mu_k(a\mid X)}{D_N(X,a)\{D_N(X,a)+\mu_k(a\mid X)\}}+\frac{V_j}{(T+\eta_0)(T+1+\eta_0)}. \tag*{(13)} Δ j , k ( N ) = G j ( N ) − G j ( N + e k ) = E X a ∈ A ∑ D N ( X , a ) { D N ( X , a ) + μ k ( a ∣ X )} W j ( X , a ) μ k ( a ∣ X ) + ( T + η 0 ) ( T + 1 + η 0 ) V j . (13)
Let G ^ j , t \widehat{G}_{j,t} G j , t and Δ ^ j , k , t \widehat{\Delta}_{j,k,t} Δ j , k , t denote G j ( N t ) G_j(N_t) G j ( N t ) and Δ j , k ( N t ) \Delta_{j,k}(N_t) Δ j , k ( N t ) with V j V_j V j and W j W_j W j computed from estimates q ^ t \widehat{q}_t q t and σ ^ t 2 \widehat{\sigma}_t^2 σ t 2 of the reward mean and variance, and with P X P_X P X replaced by the empirical distribution of the observed contexts (Appendix A3.4). Any regression estimators of the reward mean and variance can be used, and our experiments use ridge regression (Appendix § 13.1 ). Lemma A3.11 gives conditions on these estimators under which Assumption 4 holds, and Corollary A3.1 verifies them for finite context spaces.
Smoothing. To address the nonsmoothness of the worst-target criterion G ( v ) = max 1 ≤ j ≤ M G j ( v ) \mathcal{G}(v)=\max_{1\le j\le M}G_j(v) G ( v ) = max 1 ≤ j ≤ M G j ( v ) , we use a log-sum-exp approximation ([36 ] ). For ρ > 0 \rho>0 ρ > 0 , define
Φ ρ ( v ) = ρ log { ∑ j = 1 M exp ( G j ( v ) / ρ ) } , G ( v ) ≤ Φ ρ ( v ) ≤ G ( v ) + ρ log M . (14) \Phi_\rho(v)=\rho\log\left\{\sum_{j=1}^{M}\exp(G_j(v)/\rho)\right\},\qquad G(v)\le\Phi_\rho(v)\le G(v)+\rho\log M. \tag*{(14)} Φ ρ ( v ) = ρ log { j = 1 ∑ M exp ( G j ( v ) / ρ ) } , G ( v ) ≤ Φ ρ ( v ) ≤ G ( v ) + ρ log M . (14)
Hence Φ ρ ( v ) → G ( v ) \Phi_\rho(v)\to G(v) Φ ρ ( v ) → G ( v ) as ρ ↓ 0 \rho\downarrow0 ρ ↓ 0 . Its gradient is ∇ Φ ρ ( v ) = ∑ j = 1 M ω j ( v ; ρ ) ∇ G j ( v ) \nabla\Phi_\rho(v)=\sum_{j=1}^{M}\omega_j(v;\rho)\nabla G_j(v) ∇ Φ ρ ( v ) = ∑ j = 1 M ω j ( v ; ρ ) ∇ G j ( v ) , with exponential weights ω j ( v ; ρ ) = exp { G j ( v ) / ρ } / ∑ ℓ = 1 M exp { G ℓ ( v ) / ρ } \omega_j(v;\rho)=\exp\{G_j(v)/\rho\}/\sum_{\ell=1}^{M}\exp\{G_\ell(v)/\rho\} ω j ( v ; ρ ) = exp { G j ( v ) / ρ } / ∑ ℓ = 1 M exp { G ℓ ( v ) / ρ } . Therefore, policies with larger values of G j ( v ) G_j(v) G j ( v ) receive larger weights. We now relate the gradient to the one-step gains. Recall that C N 2 G j ( N ) ≈ G j ( z N ) C_N^2G_j(N)\approx G_j(z_N) C N 2 G j ( N ) ≈ G j ( z N ) for large budgets. Hence, we have C N 2 Δ j , k ( N ) ≈ − ∂ k G j ( z N ) C_N^2\Delta_{j,k}(N)\approx-\partial_kG_j(z_N) C N 2 Δ j , k ( N ) ≈ − ∂ k G j ( z N ) . Using the expression for ∇ Φ ρ \nabla\Phi_\rho ∇ Φ ρ , this gives
− ⟨ ∇ Φ ρ ( z N ) , e k c k ⟩ ≈ C N 2 c k ∑ j = 1 M ω j ( z N ; ρ ) Δ j , k ( N ) . -\left\langle\nabla\Phi_\rho(z_N),\frac{e_k}{c_k}\right\rangle\approx\frac{C_N^2}{c_k}\sum_{j=1}^{M}\omega_j(z_N;\rho)\Delta_{j,k}(N). − ⟨ ∇ Φ ρ ( z N ) , c k e k ⟩ ≈ c k C N 2 j = 1 ∑ M ω j ( z N ; ρ ) Δ j , k ( N ) .
Since C N 2 C_N^2 C N 2 is common to all loggers, this motivates using the weighted gains for selecting the next logger. Lemmas A3.8 and A3.9 make this idea precise. So that the bias ρ t log M \rho_t\log M ρ t log M vanishes, we let the smoothing parameter decrease with expenditure, ρ t = ρ 0 C t − α \rho_t=\rho_0C_t^{-\alpha} ρ t = ρ 0 C t − α with ρ 0 > 0 \rho_0>0 ρ 0 > 0 and 0 < α < 1 0<\alpha<1 0 < α < 1 , and estimate the weights by ω ^ j , t = exp ( C t G ^ j , t / ρ t ) / ∑ ℓ = 1 M exp ( C t G ^ ℓ , t / ρ t ) \widehat{\omega}_{j,t}=\exp(C_t\widehat{G}_{j,t}/\rho_t)/\sum_{\ell=1}^{M}\exp(C_t\widehat{G}_{\ell,t}/\rho_t) ω j , t = exp ( C t G j , t / ρ t ) / ∑ ℓ = 1 M exp ( C t G ℓ , t / ρ t ) . The weights therefore become increasingly concentrated on the policies with the largest estimated criterion values. We define the acquisition index for logger k k k as
S ^ t , k = 1 c k ∑ j = 1 M ω ^ j , t Δ ^ j , k , t . (15) \widehat{S}_{t,k}=\frac{1}{c_k}\sum_{j=1}^{M}\widehat{\omega}_{j,t}\widehat{\Delta}_{j,k,t}. \tag*{(15)} S t , k = c k 1 j = 1 ∑ M ω j , t Δ j , k , t . (15)
The rule selects the affordable logger with the largest index. In our experiments we use ρ 0 = 0.5 \rho_0=0.5 ρ 0 = 0.5 and α = 0.4 \alpha=0.4 α = 0.4 . Sensitivity to these choices is reported in Appendix § 13.5 . The additional condition on α \alpha α needed for the convergence result is given in § 4.1 .
Algorithm
In this section we state our final algorithm. Combining the previous subsections, the main rule is to select the affordable logger k k k at step t t t with the largest index S ^ t , k \widehat{S}_{t,k} S t , k except certain times as stated below. The full procedure is stated in Algorithm 1. The initialization gives each logger a small number of observations before selection begins. The refit interval m m m avoids refitting the nuisance models after every acquisition.
Algorithm 1 Adaptive Greedy for budgeted data acquisition from multiple loggers
Input: loggers μ 1 , … , μ K \mu_1,\ldots,\mu_K μ 1 , … , μ K , costs c 1 , … , c K c_1,\ldots,c_K c 1 , … , c K , target policies Π = { π 1 , … , π M } \Pi=\{\pi_1,\ldots,\pi_M\} Π = { π 1 , … , π M } , budget B B B , constants ρ 0 > 0 \rho_0>0 ρ 0 > 0 and α ∈ ( 0 , 1 ) \alpha\in(0,1) α ∈ ( 0 , 1 ) , forced-sampling constants χ 0 ∈ ( 0 , κ ( 1 / K ) / 8 ] \chi_0\in(0,\kappa(1/K)/8] χ 0 ∈ ( 0 , κ ( 1/ K ) /8 ] and υ ∈ ( 0 , 1 ) \upsilon\in(0,1) υ ∈ ( 0 , 1 ) , refit interval m m m , total initialization size n 0 n_0 n 0 , assumed divisible by K K K with n 0 / K ≥ 4 n_0/K\ge4 n 0 / K ≥ 4 .
Acquire n 0 / K n_0/K n 0 / K initial observations from each logger, and set t = n 0 t=n_0 t = n 0 .
while K t \mathcal{K}_t K t is nonempty do
\quad if t − n 0 t-n_0 t − n 0 is a multiple of m m m then
\qquad Update q ^ t \widehat{q}_t q t and σ ^ t 2 \widehat{\sigma}_t^2 σ t 2 using the first t t t observations, and form V j , t V_{j,t} V j , t and W j , t W_{j,t} W j , t as in (A3.13).
\quad end if
\quad if κ ( N t / t ) < χ 0 t − υ \kappa(N_t/t)<\chi_0t^{-\upsilon} κ ( N t / t ) < χ 0 t − υ then
\qquad Select K t + 1 ∈ arg min k ∈ K t N t , k K_{t+1}\in\arg\min_{k\in\mathcal{K}_t}N_{t,k} K t + 1 ∈ arg min k ∈ K t N t , k (forced sampling).
\quad else
\qquad Compute G ^ j , t \widehat{G}_{j,t} G j , t for j = 1 , … , M j=1,\ldots,M j = 1 , … , M .
\qquad Set ρ t = ρ 0 C t − α \rho_t=\rho_0C_t^{-\alpha} ρ t = ρ 0 C t − α and compute ω ^ j , t \widehat{\omega}_{j,t} ω j , t .
\qquad Compute Δ ^ j , k , t \widehat{\Delta}_{j,k,t} Δ j , k , t from (A3.15) and S ^ t , k \widehat{S}_{t,k} S t , k from (15) for all k ∈ K t k\in\mathcal{K}_t k ∈ K t .
\qquad Select K t + 1 ∈ arg max k ∈ K t S ^ t , k K_{t+1}\in\arg\max_{k\in\mathcal{K}_t}\widehat{S}_{t,k} K t + 1 ∈ arg max k ∈ K t S t , k .
\quad end if
\quad Acquire one observation from logger K t + 1 K_{t+1} K t + 1 , and set N t + 1 = N t + e K t + 1 N_{t+1}=N_t+e_{K_{t+1}} N t + 1 = N t + e K t + 1 , C t + 1 = C t + c K t + 1 C_{t+1}=C_t+c_{K_{t+1}} C t + 1 = C t + c K t + 1 , and t ← t + 1 t\leftarrow t+1 t ← t + 1 .
end while
Set T B = t T_B=t T B = t , fit the final reward regression q ^ B \widehat{q}_B q B using the completed sample, and set θ ^ B = N T B / T B \widehat{\theta}_B=N_{T_B}/T_B θ B = N T B / T B .
Compute the final estimator J ^ B \widehat{J}_B J B from (17) .
Output: the acquired sample and J ^ B \widehat{J}_B J B .
Forced sampling. Although Assumption 2 controls coverage once the algorithm is near the optimal allocation, it does not control coverage early in the acquisition. If the early estimates undervalue the loggers that cover some target actions, the rule may rarely purchase from them, and the estimates for those actions are then slow to improve. Forced sampling of under-sampled arms or strata is a standard remedy in adaptive allocation [2 , 12 , 18 ] , and adaptive designs often keep assignment probabilities above a floor that vanishes over time [19 , 3 , 8 ] . We use forced sampling of this kind, acting on pooled coverage.
Recall that κ ( θ ) \kappa(\theta) κ ( θ ) is the essential infimum of m θ ( a ∣ x ) m_\theta(a\mid x) m θ ( a ∣ x ) over the context-action pairs used by at least one target policy. If κ ( θ t ) \kappa(\theta_t) κ ( θ t ) falls below a threshold χ t \chi_t χ t that vanishes over time, the algorithm purchases an observation from the logger with the smallest count. Otherwise, it selects the logger using the usual greedy rule. We take
χ t = χ 0 t − υ , 0 < υ < 1 , \chi_t=\chi_0t^{-\upsilon},\qquad0<\upsilon<1, χ t = χ 0 t − υ , 0 < υ < 1 ,
where 0 < χ 0 ≤ κ ( 1 / K ) 8 0<\chi_0\le\frac{\kappa(1/K)}{8} 0 < χ 0 ≤ 8 κ ( 1/ K ) . Our convergence results require
0 < α < 1 , 0 < υ < 1 − α 4 , 0 < χ 0 ≤ κ ( 1 / K ) 8 , n 0 K ≥ 4. (16) 0<\alpha<1,\qquad0<\upsilon<\frac{1-\alpha}{4},\qquad0<\chi_0\le\frac{\kappa(1/K)}{8},\qquad\frac{n_0}{K}\ge4. \tag*{(16)} 0 < α < 1 , 0 < υ < 4 1 − α , 0 < χ 0 ≤ 8 κ ( 1/ K ) , K n 0 ≥ 4. (16)
Lemma A3.4 shows that forced sampling maintains κ ( θ t ) ≥ χ t / 2 \kappa(\theta_t)\ge\chi_t/2 κ ( θ t ) ≥ χ t /2 while all loggers are affordable. It also shows that the number of forced purchases up to time t t t is O ( t 1 − υ ) O(t^{1-\upsilon}) O ( t 1 − υ ) . Under the conditions of Theorem 4.1, the allocation converges to θ ⋆ \theta^\star θ ⋆ and forced sampling eventually stops.
If forced sampling uses a lower bound κ ~ ≤ κ \widetilde{\kappa}\le\kappa κ ≤ κ in place of κ \kappa κ , the same guarantees hold under the conditions given in Appendix § 9.2 .
Final estimation. When acquisition ends, we estimate J J J using the entire acquired sample and the realized logger composition θ ^ B = N T B / T B \hat{\theta}_{B}=N_{T_B}/T_B θ ^ B = N T B / T B . The final pooled estimator replaces q q q in (3) by a reward regression q ^ B \hat{q}_{B} q ^ B fitted on that sample, truncated to [ − R max , R max ] [-R_{\max},R_{\max}] [ − R m a x , R m a x ] :
J ^ B , j = 1 T B ∑ i = 1 T B [ q ^ B , j ( X i ) + π j ( A i ∣ X i ) m θ ^ B ( A i ∣ X i ) { R i − q ^ B ( X i , A i ) } ] , j = 1 , … , M , (17) \hat{J}_{B,j}=\frac{1}{T_B}\sum_{i=1}^{T_B}\left[\hat{q}_{B,j}(X_i)+\frac{\pi_j(A_i\mid X_i)}{m_{\hat{\theta}_{B}}(A_i\mid X_i)}\{R_i-\hat{q}_{B}(X_i,A_i)\}\right],\qquad j=1,\ldots,M, \tag*{(17)} J ^ B , j = T B 1 i = 1 ∑ T B [ q ^ B , j ( X i ) + m θ ^ B ( A i ∣ X i ) π j ( A i ∣ X i ) { R i − q ^ B ( X i , A i )} ] , j = 1 , … , M , (17)
where q ^ B , j ( x ) = ∑ a ∈ A π j ( a ∣ x ) q ^ B ( x , a ) \hat{q}_{B,j}(x)=\sum_{a\in\mathcal{A}}\pi_j(a\mid x)\hat{q}_{B}(x,a) q ^ B , j ( x ) = ∑ a ∈ A π j ( a ∣ x ) q ^ B ( x , a ) . Unlike the fixed-design setting, the final logger composition θ ^ B \hat{\theta}_{B} θ ^ B is random and depends on the adaptive selection process. We study its effect on inference in § 4.2 .
Theory
We divide this section into two subsections. We first prove that Adaptive Greedy actually converges to the optimal minimax design under mild assumptions in § 4.1 . We then establish the asymptotic normality of our estimator in § 4.2 . All proofs are in Appendices A3, A4, and A6.
Convergence and Asymptotics
In § 3 , we defined z ⋆ z^\star z ⋆ (equation (8) ) which minimizes the criterion G \mathcal{G} G . In this subsection, we are interested in two questions. First is when z ⋆ z^\star z ⋆ is unique and if it is, whether z T B z_{T_B} z T B converges to z ⋆ z^\star z ⋆ as B B B increases. The second is the rate. Theorem 4.1 answers both.
The main idea is that choosing the logger with the largest gain per unit cost amounts to choosing the vertex e k / c k e_k/c_k e k / c k of C C C that minimizes the linear approximation of the criterion at z t z_t z t . When the algorithm purchases one observation from logger k k k , z t = N t / C t z_t=N_t/C_t z t = N t / C t changes according to
z t + 1 = ( 1 − γ t ) z t + γ t e k c k , γ t = c k C t + c k , (18) z_{t+1}=(1-\gamma_t)z_t+\gamma_t\frac{e_k}{c_k},\qquad\gamma_t=\frac{c_k}{C_t+c_k}, \tag*{(18)} z t + 1 = ( 1 − γ t ) z t + γ t c k e k , γ t = C t + c k c k , (18)
so the purchase moves z t z_t z t toward that vertex by the step γ t \gamma_t γ t . This is similar to the Frank–Wolfe, or conditional-gradient, method [15 , 26 ] , used in optimal design by [14 ] and [43 ] . Such steps converge to the minimizer of a smooth convex function when their lengths tend to zero and their sum diverges, as γ t \gamma_t γ t , being of order C t − 1 C_t^{-1} C t − 1 , does.
Strong convexity of G \mathcal{G} G serves two purposes. If G \mathcal{G} G is m m m -strongly convex, its minimizer z ⋆ z^\star z ⋆ is unique and G ( z ) − G ( z ⋆ ) ≥ m 2 ∥ z − z ⋆ ∥ 2 2 \mathcal{G}(z)-\mathcal{G}(z^\star)\geq\frac{m}{2}\|z-z^\star\|_2^2 G ( z ) − G ( z ⋆ ) ≥ 2 m ∥ z − z ⋆ ∥ 2 2 for every z ∈ C z\in C z ∈ C , so convergence of G ( z t ) \mathcal{G}(z_t) G ( z t ) to G ( z ⋆ ) \mathcal{G}(z^\star) G ( z ⋆ ) forces z t z_t z t to converge to z ⋆ z^\star z ⋆ . Recall that a function f f f on C C C is m m m -strongly convex if f ( z ) − m 2 ∥ z ∥ 2 2 f(z)-\frac{m}{2}\|z\|_2^2 f ( z ) − 2 m ∥ z ∥ 2 2 is convex; for twice differentiable f f f this holds exactly when, along every segment in C C C with direction h h h , the second derivative of f f f is at least m ∥ h ∥ 2 2 m\|h\|_2^2 m ∥ h ∥ 2 2 . Any feasible direction h h h satisfies c ⊤ h = 0 c^\top h=0 c ⊤ h = 0 , and on such directions h ⊤ ∇ 2 G j ( z ) h ≥ 2 c min 3 h ⊤ Q j h h^\top\nabla^2G_j(z)h\geq2c_{\min}^3h^\top Q_jh h ⊤ ∇ 2 G j ( z ) h ≥ 2 c m i n 3 h ⊤ Q j h (see the proof of Proposition A3.2 in Appendix A6), where
Q j = V j 11 ⊤ + E X ∑ a ∈ A W j ( X , a ) μ ( X , a ) μ ( X , a ) ⊤ . (19) Q_j=V_j\mathbf{1}\mathbf{1}^{\top}+\mathbb{E}_X\sum_{a\in\mathcal{A}}W_j(X,a)\mu(X,a)\mu(X,a)^{\top}. \tag*{(19)} Q j = V j 1 1 ⊤ + E X a ∈ A ∑ W j ( X , a ) μ ( X , a ) μ ( X , a ) ⊤ . (19)
We therefore impose the following condition.
Assumption 3. For every target policy π j \pi_j π j , λ j = inf h ≠ 0 , c ⊤ h = 0 h ⊤ Q j h ∥ h ∥ 2 2 > 0 \lambda_j=\inf_{h\ne0,\ c^\top h=0}\frac{h^\top Q_jh}{\|h\|_2^2}>0 λ j = inf h = 0 , c ⊤ h = 0 ∥ h ∥ 2 2 h ⊤ Q j h > 0 .
This assumption ensures sufficient curvature of G \mathcal{G} G on C C C , which guarantees uniqueness of z ⋆ z^\star z ⋆ . The condition plays the role of the full-rank conditions of optimal design [37 ] . Under Assumptions 1 1 1 , 2 2 2 , and 3 3 3 , G \mathcal{G} G is strongly convex with m = 2 c min 3 min j λ j m=2c_{\min}^3\min_j\lambda_j m = 2 c m i n 3 min j λ j , and thus, z ⋆ z^\star z ⋆ is unique. These results are Proposition A3.2 and Lemma A3.2 in Appendix A3, and their proofs are in Appendix A6.
Now, we focus on the rate of convergence. Recall that our selection rule uses G ^ j , t \widehat{G}_{j,t} G j , t and Δ ^ j , k , t \widehat{\Delta}_{j,k,t} Δ j , k , t from § 3.2 . Since G j ( N t ) G_j(N_t) G j ( N t ) is of order C t − 1 C_t^{-1} C t − 1 and Δ j , k ( N t ) \Delta_{j,k}(N_t) Δ j , k ( N t ) is of order C t − 2 C_t^{-2} C t − 2 when pooled coverage is bounded below, we need their estimation errors to be small on these scales. We impose the following assumption.
Assumption 4. There is a deterministic sequence r t → 0 r_t \to0 r t → 0 such that, almost surely for all sufficiently large t t t ,
C t max j ∣ G ^ j , t − G j ( N t ) ∣ ≤ r t , C t 2 max j , k ∣ Δ ^ j , k , t − Δ j , k ( N t ) ∣ ≤ r t . (20) C_t\max_j\left|\widehat{G}_{j,t}-G_j(N_t)\right|\le r_t,\qquad C_t^2\max_{j,k}\left|\widehat{\Delta}_{j,k,t}-\Delta_{j,k}(N_t)\right|\le r_t. \tag*{(20)} C t j max G j , t − G j ( N t ) ≤ r t , C t 2 j , k max Δ j , k , t − Δ j , k ( N t ) ≤ r t . (20)
The accuracy of these estimates depends on the estimation of V j V_j V j , W j W_j W j , and P X P_X P X . Lemma A3.11 in Appendix A3 gives sufficient conditions for Assumption 4. We also verify these conditions for finite context spaces in Corollary A3.1.
We now make the Frank–Wolfe interpretation more precise. Write z t = z N t z_t=z_{N_t} z t = z N t , and recall from § 3.2 that the rescaled index C t 2 S ^ t , k C_t^2\widehat{S}_{t,k} C t 2 S t , k is approximately − ⟨ ∇ Φ ρ t ( z t ) , e k / c k ⟩ -\langle\nabla\Phi_{\rho_t}(z_t),e_k/c_k\rangle − ⟨ ∇ Φ ρ t ( z t ) , e k / c k ⟩ .
Proposition 4.1. Under Assumptions 1 and 4, there is a finite constant D D D , depending only on c min c_{\min} c m i n , c max c_{\max} c m a x , R max R_{\max} R m a x , η \eta η , η 0 \eta_0 η 0 , and ρ 0 \rho_0 ρ 0 , such that, almost surely for all sufficiently large t t t ,
max 1 ≤ k ≤ K ∣ C t 2 S ^ t , k + ⟨ ∇ Φ ρ t ( z t ) , e k c k ⟩ ∣ ≤ ε t , ε t = D { κ ( θ t ) − 2 C t α r t + κ ( θ t ) − 4 C t α − 1 } . (21) \max_{1\le k\le K}\left|C_t^2\widehat{S}_{t,k}+\left\langle\nabla\Phi_{\rho_t}(z_t),\frac{e_k}{c_k}\right\rangle\right|\le\varepsilon_t,\qquad\varepsilon_t=D\left\{\kappa(\theta_t)^{-2}C_t^\alpha r_t+\kappa(\theta_t)^{-4}C_t^{\alpha-1}\right\}. \tag*{(21)} 1 ≤ k ≤ K max C t 2 S t , k + ⟨ ∇ Φ ρ t ( z t ) , c k e k ⟩ ≤ ε t , ε t = D { κ ( θ t ) − 2 C t α r t + κ ( θ t ) − 4 C t α − 1 } . (21) Consequently, whenever all loggers are affordable and the step is not forced, the vertex v t = e K t + 1 / c K t + 1 v_t=e_{K_{t+1}}/c_{K_{t+1}} v t = e K t + 1 / c K t + 1 purchased by Algorithm 1 satisfies
⟨ ∇ Φ ρ t ( z t ) , v t ⟩ ≤ min v ∈ C ⟨ ∇ Φ ρ t ( z t ) , v ⟩ + δ t , δ t = 2 ε t . (22) \left\langle\nabla\Phi_{\rho_t}(z_t),v_t\right\rangle\le\min_{v\in\mathcal{C}}\left\langle\nabla\Phi_{\rho_t}(z_t),v\right\rangle+\delta_t,\qquad\delta_t=2\varepsilon_t. \tag*{(22)} ⟨ ∇ Φ ρ t ( z t ) , v t ⟩ ≤ v ∈ C min ⟨ ∇ Φ ρ t ( z t ) , v ⟩ + δ t , δ t = 2 ε t . (22)
This proposition formalizes the connection between our selection rule and the Frank–Wolfe method. Our algorithm chooses the logger with the largest estimated index S ^ t , k \widehat{S}_{t,k} S t , k while a Frank–Wolfe step would minimize the linear approximation of Φ ρ t \Phi_{\rho_t} Φ ρ t over C \mathcal{C} C . The first bound shows that these two are close and the second shows that our selection is a δ t \delta_t δ t -approximate conditional-gradient step. Note that the approximation error depends on both r t r_t r t and pooled coverage κ ( θ t ) \kappa(\theta_t) κ ( θ t ) . Therefore, if coverage is small, the errors can become fairly large. As discussed in § 3.3 , forced sampling is introduced to prevent exactly this. Lemma A3.4 in Appendix A3 shows that κ ( θ t ) ≥ χ t / 2 \kappa(\theta_t)\ge\chi_t/2 κ ( θ t ) ≥ χ t /2 and that the step sizes of the forced purchases are summable. Together with the accuracy assumption, these bounds allow us to establish convergence of the acquired allocation.
We now state our main result. Because the last purchases depend on the budget, we state the estimation condition along a single sequence that does not depend on B B B : the infinite-budget sequence, in which every logger remains affordable, with counts N ‾ t \overline{N}_t N t and expenditure C ‾ t \overline{C}_t C t . The relation between the budget-B B B and infinite-budget sequences is provided in Appendix A3. The conditions in the theorem are needed to control the errors in Proposition 4.1. Recall that ν \nu ν determines how quickly the forced-sampling threshold decreases. The condition ν < ( 1 − α ) / 4 \nu<(1-\alpha)/4 ν < ( 1 − α ) /4 in (16) ensures that the error terms involving the pooled coverage decrease as more observations are collected. We also require t α + 2 ν r t → 0 t^{\alpha+2\nu}r_t\to0 t α + 2 ν r t → 0 to control the error from estimating the criterion and gains.
Theorem 4.1. Suppose Assumptions 1, 2, and 3 hold, Assumption 4 holds along the infinite-budget sequence, with N t N_t N t and C t C_t C t replaced by N ‾ t \overline{N}_t N t and C ‾ t \overline{C}_t C t , the tuning constants satisfy (16) , and t α + 2 ν r t → 0 t^{\alpha+2\nu}r_t\to0 t α + 2 ν r t → 0 . Then
(i) z T B → z ∗ z_{T_B}\to z^* z T B → z ∗ and θ ^ B → θ ∗ \widehat{\theta}_B\to\theta^* θ B → θ ∗ almost surely as B → ∞ B\to\infty B → ∞ , and forced sampling occurs only finitely often along the infinite-budget sequence.
(ii) If, moreover, r t = O ( t − β r ) r_t=O(t^{-\beta_r}) r t = O ( t − β r ) for some β r > α \beta_r>\alpha β r > α , then almost surely
G ( z T B ) − G ( z ∗ ) = O ( B − β ) , ∥ θ ^ B − θ ∗ ∥ 2 = O ( B − β / 2 ) . G(z_{T_B})-G(z^*)=O(B^{-\beta}),\qquad\left\|\widehat{\theta}_B-\theta^*\right\|_2=O(B^{-\beta/2}). G ( z T B ) − G ( z ∗ ) = O ( B − β ) , θ B − θ ∗ 2 = O ( B − β /2 ) .
where β = min { α , 1 − α , β r − α } \beta= \min\{\alpha, 1-\alpha, \beta_r-\alpha\} β = min { α , 1 − α , β r − α } .
Part (i) shows that Adaptive Greedy indeed converges to the optimal design z ∗ z^* z ∗ . It also shows that forced sampling eventually stops once the allocation is sufficiently close to the optimum as the coverage is bounded away from zero and the usual greedy rule takes over. In part (ii) each term of β \beta β comes from one source of error: smoothing gives α \alpha α , purchasing one observation at a time gives 1 − α 1-\alpha 1 − α , and estimation gives β r − α \beta_r-\alpha β r − α . The first term increases in α \alpha α and the other two decrease, so taking α = min { β r , 1 } / 2 \alpha= \min\{\beta_r,1\}/2 α = min { β r , 1 } /2 gives the largest exponent, β = min { β r , 1 } / 2 \beta= \min\{\beta_r,1\}/2 β = min { β r , 1 } /2 .
Inference
In this subsection we derive the asymptotic normality of J ^ B \hat{J}_B J ^ B . The main difficulty is that, unlike in the fixed-design setting, where the estimator is simply an average of independent terms with variance A j ( θ ) / t A_j(\theta)/t A j ( θ ) / t , each observation in (17) is divided by the pooled behavior probability m θ ^ B m_{\hat{\theta}_B} m θ ^ B . θ ^ B \hat{\theta}_B θ ^ B depends on the rewards observed during purchasing. The denominator is therefore unknown when each observation is collected. Nevertheless, we show that the estimator is asymptotically normal and that its asymptotic variance for target π j \pi_j π j is A j ( θ ∗ ) A_j(\theta^*) A j ( θ ∗ ) , the fixed-design bound at the composition to which the rule converges.
We compare J ^ B \hat{J}_B J ^ B with averages in which q q q replaces q ^ B \hat{q}_B q ^ B and a fixed composition θ \theta θ with κ ( θ ) > 0 \kappa(\theta)>0 κ ( θ ) > 0 replaces θ ^ B \hat{\theta}_B θ ^ B ; their summand is the score
ψ i ( θ ) = q Π ( X i ) + π ( A i ∣ X i ) m θ ( A i ∣ X i ) { R i − q ( X i , A i ) } , (23) \psi_i(\theta) = q_{\Pi}(X_i) + \frac{\pi(A_i\mid X_i)}{m_{\theta}(A_i\mid X_i)}\{R_i-q(X_i,A_i)\}, \tag*{(23)} ψ i ( θ ) = q Π ( X i ) + m θ ( A i ∣ X i ) π ( A i ∣ X i ) { R i − q ( X i , A i )} , (23)
where π ( a ∣ x ) = { π 1 ( a ∣ x ) , … , π M ( a ∣ x ) } T \pi(a\mid x)=\{\pi_1(a\mid x),\ldots,\pi_M(a\mid x)\}^{\mathsf{T}} π ( a ∣ x ) = { π 1 ( a ∣ x ) , … , π M ( a ∣ x ) } T and q Π ( x ) = { q 1 ( x ) , … , q M ( x ) } T q_{\Pi}(x)=\{q_1(x),\ldots,q_M(x)\}^{\mathsf{T}} q Π ( x ) = { q 1 ( x ) , … , q M ( x ) } T .
The main idea involves three steps. First, consider J ^ B ∗ = 1 T B ∑ i = 1 T B ψ i ( θ ∗ ) \hat{J}_B^*=\frac{1}{T_B}\sum_{i=1}^{T_B}\psi_i(\theta^*) J ^ B ∗ = T B 1 ∑ i = 1 T B ψ i ( θ ∗ ) which uses the limiting composition θ ∗ \theta^* θ ∗ , which is unknown. Although the loggers are selected adaptively, K i K_i K i is chosen before observation i i i is collected and since R i − q ( X i , A i ) R_i-q(X_i,A_i) R i − q ( X i , A i ) has conditional mean zero, ψ i ( θ ∗ ) − J \psi_i(\theta^*)-J ψ i ( θ ∗ ) − J also has conditional mean zero given the past. Thus, these scores form a martingale difference sequence. Their average conditional variance depends on θ ^ B \hat{\theta}_B θ ^ B , which converges to θ ∗ \theta^* θ ∗ by Theorem 4.1. The limiting variance is therefore Σ A ( θ ∗ ) \Sigma_A(\theta^*) Σ A ( θ ∗ ) , where
Σ A ( θ ) = Cov { q Π ( X ) } + E X ∑ a ∈ A σ 2 ( X , a ) π ( a ∣ X ) π ( a ∣ X ) T m θ ( a ∣ X ) , (24) \Sigma_A(\theta)=\operatorname{Cov}\{q_{\Pi}(X)\}+\mathbb{E}_X\sum_{a\in\mathcal{A}}\frac{\sigma^2(X,a)\pi(a\mid X)\pi(a\mid X)^{\mathsf{T}}}{m_{\theta}(a\mid X)}, \tag*{(24)} Σ A ( θ ) = Cov { q Π ( X )} + E X a ∈ A ∑ m θ ( a ∣ X ) σ 2 ( X , a ) π ( a ∣ X ) π ( a ∣ X ) T , (24)
and the j j j th diagonal entry of Σ A ( θ ) \Sigma_A(\theta) Σ A ( θ ) is A j ( θ ) A_j(\theta) A j ( θ ) in (5) . We can then apply a martingale central limit theorem (Hall and Heyde [20 ] ). The second step is to replace θ ∗ \theta^* θ ∗ by θ ^ B \hat{\theta}_B θ ^ B . By Assumption 2, the pooled probabilities remain bounded away from zero when the composition is sufficiently close to θ ∗ \theta^* θ ∗ . Since θ ^ B → θ ∗ \hat{\theta}_B\to\theta^* θ ^ B → θ ∗ , we need to show that using θ ^ B \hat{\theta}_B θ ^ B instead of θ ∗ \theta^* θ ∗ does not change the limiting distribution. We show in Appendix § 10 that
1 T B ∑ i = 1 T B { ψ i ( θ ^ B ) − ψ i ( θ ∗ ) } = o p ( T B − 1 / 2 ) . \frac{1}{T_B}\sum_{i=1}^{T_B}\{\psi_i(\hat{\theta}_B)-\psi_i(\theta^*)\}=o_p(T_B^{-1/2}). T B 1 i = 1 ∑ T B { ψ i ( θ ^ B ) − ψ i ( θ ∗ )} = o p ( T B − 1/2 ) .
Finally, we need to account for the estimated reward regression q ^ B \hat{q}_B q ^ B . Replacing q q q by q ^ B \hat{q}_B q ^ B introduces the remainder
R B , j ( θ ) = 1 T B ∑ i = 1 T B [ { q ^ B , j − q j } ( X i ) − π j ( A i ∣ X i ) m θ ( A i ∣ X i ) { q ^ − q } ( X i , A i ) ] . (25) \mathcal{R}_{B,j}(\theta)=\frac{1}{T_B}\sum_{i=1}^{T_B}\left[\{\hat{q}_{B,j}-q_j\}(X_i)-\frac{\pi_j(A_i\mid X_i)}{m_{\theta}(A_i\mid X_i)}\{\hat{q}-q\}(X_i,A_i)\right]. \tag*{(25)} R B , j ( θ ) = T B 1 i = 1 ∑ T B [ { q ^ B , j − q j } ( X i ) − m θ ( A i ∣ X i ) π j ( A i ∣ X i ) { q ^ − q } ( X i , A i ) ] . (25)
We therefore impose the following assumption.
Assumption 5. There is a deterministic sequence r B ↓ 0 r_B \downarrow0 r B ↓ 0 such that
P ( ∥ θ ^ B − θ ⋆ ∥ 2 ≤ r B ) → 1 , max 1 ≤ j ≤ M sup ∥ θ − θ ⋆ ∥ 2 ≤ r B ∣ R B , j ( θ ) ∣ = o P ( T B − 1 / 2 ) . (26) \mathbb{P}\left(\lVert\hat{\theta}_B-\theta^\star\rVert_2\le r_B\right)\to1,\qquad\max_{1\le j\le M}\sup_{\lVert\theta-\theta^\star\rVert_2\le r_B}\lvert R_{B,j}(\theta)\rvert=o_{\mathbb{P}}\left(T_B^{-1/2}\right). \tag*{(26)} P ( ∥ θ ^ B − θ ⋆ ∥ 2 ≤ r B ) → 1 , 1 ≤ j ≤ M max ∥ θ − θ ⋆ ∥ 2 ≤ r B sup ∣ R B , j ( θ )∣ = o P ( T B − 1/2 ) . (26)
This is the usual condition under which the estimated regression does not affect an augmented estimator at first order, and in semiparametric inference it is commonly obtained by sample splitting or cross-fitting ([7 ] ). In Appendix A5 we describe a block-split estimator, motivated by cross-fitting, that avoids reusing the same observations for regression. Related conditions for adaptively collected data appear in Hadad et al. [19 ] , Zhan et al. [44 ] , and Bibaut et al. [3 ] . Because θ ^ B \hat{\theta}_B θ ^ B is random, the remainder is required to be small over a neighborhood of θ ⋆ \theta^\star θ ⋆ that contains θ ^ B \hat{\theta}_B θ ^ B with probability tending to one. Since r B → 0 r_B\to0 r B → 0 , this neighborhood eventually lies in { θ : ∥ θ − θ ⋆ ∥ 1 ≤ κ ⋆ / 2 } \{\theta:\lVert\theta-\theta^\star\rVert_1\le\kappa_\star/2\} { θ : ∥ θ − θ ⋆ ∥ 1 ≤ κ ⋆ /2 } , where m θ ≥ κ ⋆ / 2 m_\theta\ge\kappa_\star/2 m θ ≥ κ ⋆ /2 on S S S by Assumption 2, so the remainder is well defined there. The first condition in (4.9) only limits how fast this neighborhood may shrink, and by Theorem 4.1 it holds for any r B r_B r B that decreases slowly enough. The assumption concerns only the final regression in (17) and is separate from Assumption 4, which governs the estimates used while purchasing.
Theorem 4.2. Under the conditions of part (i) of Theorem 4.1 and Assumption 5,
T B ( J ^ B − J ) → N { 0 , Σ A ( θ ⋆ ) } in distribution. (27) \sqrt{T_B}\left(\hat{J}_B-J\right)\to N\{0,\Sigma_A(\theta^\star)\}\quad\text{in distribution.} \tag*{(27)} T B ( J ^ B − J ) → N { 0 , Σ A ( θ ⋆ )} in distribution. (27)
For each target, the asymptotic variance of the estimate computed from the purchased sample is therefore A j ( θ ⋆ ) A_j(\theta^\star) A j ( θ ⋆ ) , the fixed-design bound at the composition to which the rule converges, which is the quantity whose largest value over Π \Pi Π the rule was built to reduce.
We need to estimate the covariance as well. With ψ ^ i \hat{\psi}_i ψ ^ i the M M M -vector summand of (17) , we use
Σ ^ B = 1 T B ∑ i = 1 T B ( ψ ^ i − J ^ B ) ( ψ ^ i − J ^ B ) ⊤ . (28) \hat{\Sigma}_B=\frac{1}{T_B}\sum_{i=1}^{T_B}(\hat{\psi}_i-\hat{J}_B)(\hat{\psi}_i-\hat{J}_B)^\top. \tag*{(28)} Σ ^ B = T B 1 i = 1 ∑ T B ( ψ ^ i − J ^ B ) ( ψ ^ i − J ^ B ) ⊤ . (28)
Corollary 4.1. Suppose the conditions of Theorem 4.2 hold and the terminal regression is consistent in mean square over the sample, T B − 1 ∑ i ≤ T B ∑ a ∈ A { q ^ B ( X i , a ) − q ( X i , a ) } 2 → 0 T_B^{-1}\sum_{i\le T_B}\sum_{a\in\mathcal{A}}\{\hat{q}_B(X_i,a)-q(X_i,a)\}^2\to0 T B − 1 ∑ i ≤ T B ∑ a ∈ A { q ^ B ( X i , a ) − q ( X i , a ) } 2 → 0 in probability. Then Σ ^ B → Σ A ( θ ⋆ ) \hat{\Sigma}_B\to\Sigma_A(\theta^\star) Σ ^ B → Σ A ( θ ⋆ ) in probability. Consequently, for every target π j \pi_j π j with A j ( θ ⋆ ) > 0 A_j(\theta^\star)>0 A j ( θ ⋆ ) > 0 , the intervals
J ^ B , j ± z 1 − γ / 2 ( Σ ^ B ) j j / T B , j = 1 , … , M , (29) \hat{J}_{B,j}\pm z_{1-\gamma/2}\sqrt{(\hat{\Sigma}_B)_{jj}/T_B},\qquad j=1,\ldots,M, \tag*{(29)} J ^ B , j ± z 1 − γ /2 ( Σ ^ B ) j j / T B , j = 1 , … , M , (29) have asymptotic coverage 1 − γ 1-\gamma 1 − γ for J j J_j J j .
Because Theorem 4.2 is a joint statement over the target class, simultaneous regions also follow. A Bonferroni band replaces z 1 − γ / 2 z_{1-\gamma/2} z 1 − γ /2 in (29) by z 1 − γ / ( 2 M ) z_{1-\gamma/(2M)} z 1 − γ / ( 2 M ) , and a less conservative band uses the 1 − γ 1-\gamma 1 − γ quantile of max j ∣ Z j ∣ \max_j\lvert Z_j\rvert max j ∣ Z j ∣ , where Z ∼ N ( 0 , R ^ B ) Z\sim N(0,\hat{R}_B) Z ∼ N ( 0 , R ^ B ) and R ^ B \hat{R}_B R ^ B is the correlation matrix associated with Σ ^ B \hat{\Sigma}_B Σ ^ B .
The pooled weighting also matters for efficiency. A logger-specific estimator, which divides each observation by the propensity of the logger that generated it, has fixed-design covariance Σ L ( θ ) \Sigma_L(\theta) Σ L ( θ ) , defined in (A4.9) of Appendix § 10.4 , and
Σ A ( θ ) ⪯ Σ L ( θ ) for every θ ∈ S K , (30) \Sigma_A(\theta)\preceq\Sigma_L(\theta)\qquad\text{for every }\theta\in\mathcal{S}_K, \tag*{(30)} Σ A ( θ ) ⪯ Σ L ( θ ) for every θ ∈ S K , (30)
an ordering known for fixed designs ([1 , 28 ] ), where the right side is infinite when a logger with θ k > 0 \theta_k>0 θ k > 0 gives probability zero to an action that a target policy uses. Theorem 4.2 shows that the pooled estimator variance A j ( θ ⋆ ) A_j(\theta^\star) A j ( θ ⋆ ) is no larger than the fixed-design variance of the logger-specific estimator at the same composition.
Numerical Studies
In this section, we evaluate Adaptive Greedy (Algorithm 1) in three synthetic environments with known optimal designs and in a semi-synthetic application based on the AIDS Clinical Trials Group Study 175 (Hammer et al., [21 ] ). Implementation details, the complete data-generating process, and additional experiments are given in Appendix § 13 .
Simulation Setup
All three environments have K = 5 K = 5 K = 5 loggers, five actions, M = 4 M = 4 M = 4 target policies, and a five-dimensional context drawn from a three-component Gaussian mixture. The reward is a clipped normal variable,
R = min { 4 , max ( − 4 , R ∘ ) } , R ∘ ∣ X = x , A = a ∼ N { q 0 ( x , a ) , s 2 ( x , a ) } , R = \min\{4, \max(-4, R^{\circ})\}, \qquad R^{\circ} \mid X = x, A = a \sim N\{q_0(x,a), s^2(x,a)\}, R = min { 4 , max ( − 4 , R ∘ )} , R ∘ ∣ X = x , A = a ∼ N { q 0 ( x , a ) , s 2 ( x , a )} ,
where q 0 ( x , a ) q_0(x,a) q 0 ( x , a ) is linear in x x x , with an additional interaction x 1 x 2 x_1x_2 x 1 x 2 for actions 3 to 5, and the variance s 2 ( x , a ) s^2(x,a) s 2 ( x , a ) is larger for actions 4 and 5 and increases with x 3 2 x_3^2 x 3 2 . Loggers 1 to 4 concentrate on the action pairs (1, 2), (2, 3), (3, 4), and (4, 5), so each covers a different part of the action space, while Logger 5 is close to uniform. The target policies are softmax policies mixed with a uniform exploration floor. The environments differ as follows.
Environment A has costs c = ( 1 , 2 , 4 , 8 , 16 ) c = (1, 2, 4, 8, 16) c = ( 1 , 2 , 4 , 8 , 16 ) and optimal design θ ⋆ ≈ ( 0.784 , 0 , 0.085 , 0.132 , 0 ) \theta^\star\approx(0.784, 0, 0.085, 0.132, 0) θ ⋆ ≈ ( 0.784 , 0 , 0.085 , 0.132 , 0 ) .
Environment B keeps the loggers and costs of Environment A but uses more concentrated target policies. Its optimal design θ ⋆ ≈ ( 0.715 , 0.074 , 0.138 , 0.073 , 0 ) \theta^\star\approx(0.715, 0.074, 0.138, 0.073, 0) θ ⋆ ≈ ( 0.715 , 0.074 , 0.138 , 0.073 , 0 ) includes Logger 2, so a change in the targets alone changes which loggers are worth purchasing from.
Environment C keeps the policies of Environment A and changes the costs to c = ( 1 , 4 , 16 , 8 , 2 ) c = (1, 4, 16, 8, 2) c = ( 1 , 4 , 16 , 8 , 2 ) . Its optimal design θ ⋆ ≈ ( 0.257 , 0 , 0 , 0 , 0.743 ) \theta^\star\approx(0.257, 0, 0, 0, 0.743) θ ⋆ ≈ ( 0.257 , 0 , 0 , 0 , 0.743 ) puts most of the sample on Logger 5 even though Logger 1 is cheaper, so the best allocation depends on statistical value per unit cost rather than on cost alone.
We compare Adaptive Greedy with five procedures. All of them use the same initialization and, where they need them, the same estimators of the reward mean and variance.
Reoptimized plug-in re-estimates V j V_j V j and W j W_j W j every ten observations, solves a continuous relaxation of the static allocation problem (11) with these estimates, rounds the resulting allocation to integer counts, and purchases toward that allocation until the next update.
Estimated static solves the same relaxed problem once, after the initialization, and follows the resulting integer allocation to the end.
Oracle static uses the same procedure with the true V j V_j V j and W j W_j W j . It is not feasible in practice and serves as a reference.
Equal expenditure spends the same amount on each logger, so its sample fractions are proportional to 1 / c k 1/c_k 1/ c k .
Cheapest only always purchases from the cheapest affordable logger.
Every procedure starts with four observations from each logger. For each action, we estimate the reward mean by ridge regression and the reward variance from the squared residuals of that fit (Appendix § 13.1 ). Both are refitted every ten observations. Adaptive Greedy uses α = 0.4 \alpha= 0.4 α = 0.4 and ρ 0 = 0.5 \rho_0 = 0.5 ρ 0 = 0.5 in all settings, and forced sampling uses χ 0 = 10 − 4 \chi_0 = 10^{-4} χ 0 = 1 0 − 4 and ν = 0.01 \nu= 0.01 ν = 0.01 . Every logger in our experiments gives every action a probability bounded below by more than χ 0 \chi_0 χ 0 , so forced sampling never occurs (Appendix § 13.1 ). After the budget is spent, the policy values are estimated by the final estimator (17) . Since the design criterion is a worst case over the target class, our main accuracy measure is the worst-target root mean squared error,
WorstRMSE = max 1 ≤ j ≤ M { 1 R ∑ r = 1 R ( J ^ r , j − J j ) 2 } 1 / 2 , \operatorname{WorstRMSE} = \max_{1 \le j \le M} \left\{ \frac{1}{R} \sum_{r=1}^{R} \left(\widehat{J}_{r,j} - J_j\right)^2 \right\}^{1/2}, WorstRMSE = 1 ≤ j ≤ M max { R 1 r = 1 ∑ R ( J r , j − J j ) 2 } 1/2 ,
where J ^ r , j \widehat{J}_{r,j} J r , j is the estimate of J j J_j J j in replication r r r . To measure how close the purchased design z ^ = z T B \widehat{z} = z_{T_B} z = z T B is to the optimal design, we also report ∥ θ ^ − θ ⋆ ∥ 2 \lVert\widehat{\theta}-\theta^\star\rVert_2 ∥ θ − θ ⋆ ∥ 2 and G ( z ^ ) / G ( z ⋆ ) \mathcal{G}(\widehat{z})/\mathcal{G}(z^\star) G ( z ) / G ( z ⋆ ) , which equals one at the optimal design. The budgets are B ∈ { 500 , 1000 , 1500 , 3000 , 6000 , 12000 , 24000 } B \in\{500, 1000, 1500, 3000, 6000, 12000, 24000\} B ∈ { 500 , 1000 , 1500 , 3000 , 6000 , 12000 , 24000 } , with R = 200 R = 200 R = 200 replications in Environments A and B and R = 100 R = 100 R = 100 in Environment C. We use common random numbers across procedures. For RMSE quantities, one Monte Carlo standard error is computed from 5000 bootstrap resamples of the replications. For quantities averaged directly over replications, such as G ( z ^ ) / G ( z ⋆ ) \mathcal{G}(\widehat{z})/\mathcal{G}(z^\star) G ( z ) / G ( z ⋆ ) and ∥ θ ^ − θ ⋆ ∥ 2 \lVert\widehat{\theta}-\theta^\star\rVert_2 ∥ θ − θ ⋆ ∥ 2 , it is the sample standard deviation divided by R \sqrt{R} R .
Simulation Results
Figure 1 and Table 1 show that Adaptive Greedy is among the most accurate feasible procedures in all three environments. Its accuracy also approaches that of Oracle static as the budget grows. At B = 24000 B = 24000 B = 24000 , its worst-target RMSE is within about 2% of Oracle static in every environment. It is 17% and 22% lower than that of Estimated static in Environments A and B, 15% to 21% lower than that of Equal expenditure, and 42% to 51% lower than that of Cheapest only. Reoptimized plug-in reaches almost the same accuracy as Adaptive Greedy. In Environment C, Estimated static also performs well, while Cheapest only has about twice the worst-target RMSE of Adaptive Greedy, which confirms that cost alone does not identify a good design. At small budgets, the differences between procedures are smaller and their ordering varies. Paired bootstrap comparisons separate Adaptive Greedy from Cheapest only at every budget, and from Estimated static and Equal expenditure at most budgets in Environments A and B (Appendix A7.4).
Figure 1. Worst-target RMSE in the three synthetic environments. Error bars show ± 1 \pm1 ± 1 Monte Carlo standard error, and the vertical axis is logarithmic.
Figure 2 shows that the design purchased by Adaptive Greedy converges to the optimal design in all three environments, as parts (i) and (ii) of Theorem 4.1 predict. Both G ( z ^ ) / G ( z ⋆ ) \mathcal{G}(\widehat{z})/\mathcal{G}(z^\star) G ( z ) / G ( z ⋆ ) and ∥ θ ^ − θ ⋆ ∥ 2 \lVert\widehat{\theta}-\theta^\star\rVert_2 ∥ θ − θ ⋆ ∥ 2 decrease steadily with the budget.
Figure 2. Convergence of the Adaptive Greedy design. The top row shows G ( z ^ ) / G ( z ⋆ ) \mathcal{G}(\hat{z})/\mathcal{G}(z^\star) G ( z ^ ) / G ( z ⋆ ) and the bottom row shows ∥ θ ^ − θ ⋆ ∥ 2 \lVert\hat{\theta}-\theta^\star\rVert_2 ∥ θ ^ − θ ⋆ ∥ 2 . Error bars show ± 1 \pm1 ± 1 Monte Carlo standard error, and the budget axis is logarithmic.
The Wald intervals of Corollary 4.1 are well calibrated. We use R = 500 R = 500 R = 500 replications in Environments A and B at B = 6000 B = 6000 B = 6000 and B = 24000 B = 24000 B = 24000 . The ratio of the mean estimated standard error to the empirical standard deviation lies between 0.989 and 1.060. Target-specific 95% coverage lies between 0.944 and 0.966, and simultaneous coverage lies between 0.950 and 0.960 (Appendix A7.6). The appendix reports three further findings. First, Adaptive Greedy is robust to a misleading initialization, which degrades Estimated static much more (Appendix § 13.5 ). Second, its accuracy changes little over a range of α \alpha α and ρ 0 \rho_0 ρ 0 (Appendix § 13.5 ). Finally, the design criterion closely tracks estimation accuracy, with a correlation of about 0.97 at budgets of at least 3000 (Appendix A7.4).
Table 1. Worst-target RMSE (× 10 − 3 \times10^{-3} × 1 0 − 3 ) at B = 24000 B = 24000 B = 24000 in the synthetic environments, with Monte Carlo standard errors (× 10 − 3 \times10^{-3} × 1 0 − 3 ) in parentheses. Oracle static uses the true reward moments and is not feasible in practice.
Comparison with Reoptimized Plug-in
Adaptive Greedy and Reoptimized plug-in reach similar accuracy when the budget is spent, but they get there differently. Reoptimized plug-in plans for the declared budget B B B . However, Adaptive Greedy uses B B B only to decide when to stop, so its purchases up to any point do not depend on B B B . This matters when the final budget is uncertain or the run may end early. In Environment A, we ran both procedures with budget 5000 and recorded their designs at intermediate budgets. We also ran separate runs whose budget equals each intermediate budget. Figure 3 shows that the Adaptive Greedy design at every intermediate budget is almost the same as that of a run planned for that budget, so stopping early costs little. The Reoptimized plug-in design instead moves away from the optimum early on, with G ( z ˉ ) / G ( z ⋆ ) \mathcal{G}(\bar{z})/\mathcal{G}(z^\star) G ( z ˉ ) / G ( z ⋆ ) rising from 1.41 at B = 500 B = 500 B = 500 to 1.72 at B = 1000 B = 1000 B = 1000 , because its early purchases follow a plan made for B = 5000 B = 5000 B = 5000 (Appendix A7.7).
Figure 3. Designs at intermediate budgets in Environment A. Solid curves follow runs with budget 5000, and open markers show separate runs whose budget equals that intermediate budget. Shaded bands and error bars show Monte Carlo uncertainty.
Adaptive Greedy is also much cheaper to run, since it compares K K K indices at each step instead of solving an allocation problem every ten observations. In Environment A, Reoptimized plug-in took 8.7 times as long as Adaptive Greedy at B = 12000 B = 12000 B = 12000 (6423 versus 742 seconds) and 9.4 times as long at B = 24000 B = 24000 B = 24000 (16442 versus 1750 seconds). Thus Adaptive Greedy matches the accuracy of repeated reoptimization at about a tenth of its computing cost, and it provides a good design at any stopping point.
Application to ACTG 175 Data
The AIDS Clinical Trials Group Study 175 ([21 ] ) was a randomized trial in HIV-infected adults that compared four treatments, zidovudine (ZDV), ZDV with didanosine (ddI), ZDV with zalcitabine, and ddI alone. The public data ([21 ] ) contain 2139 subjects with complete records and have been used to study individualized treatment regimes ([13 ] ). We take the four treatments as the actions, seven baseline covariates as the context (age, weight, CD4 count, CD8 count, Karnofsky score, symptomatic status, and prior antiretroviral therapy), and the change in CD4 count from baseline to around week 20, divided by 100, as the reward. To make the policy values and the optimal design known, we fit an arm-specific regression of the reward on the covariates and generate rewards from it, with contexts resampled from the 2139 subjects (Appendix A7.8).
We construct five loggers with costs c = ( 1 , 2 , 4 , 3 , 8 ) \mathbf{c} = (1, 2, 4, 3, 8) c = ( 1 , 2 , 4 , 3 , 8 ) . Loggers 1 to 4 each favor a different pair of treatments, and Logger 5 is uniform. The loggers and costs are part of the experimental design and do not describe the trial itself. Of the four target policies, one favors the treatments with higher fitted rewards and three are softmax policies. The optimal design θ ⋆ ≈ ( 0.811 , 0 , 0.028 , 0.161 , 0 ) \boldsymbol{\theta}^\star\approx(0.811, 0, 0.028, 0.161, 0) θ ⋆ ≈ ( 0.811 , 0 , 0.028 , 0.161 , 0 ) uses only Loggers 1, 3, and 4. Logger 1 covers the first two treatments cheaply, Loggers 3 and 4 cover the other two, and the broad coverage of the uniform logger does not justify its cost. We use budgets 500, 1500, 6000, and 12000, with R = 200 R = 200 R = 200 replications at each.
Table 2 shows the same pattern as the simulations. At B = 12000 B = 12000 B = 12000 , Adaptive Greedy matches
Table 2. Worst-target RMSE (× 10 − 3 \times10^{-3} × 1 0 − 3 ) in the ACTG 175 application, with Monte Carlo standard errors (× 10 − 3 \times10^{-3} × 1 0 − 3 ) in parentheses. As in Table 1, Oracle static uses the true reward moments and is not feasible in practice.
Reoptimized plug-in and Oracle static, and its worst-target RMSE is 11% to 18% lower than those of Estimated static, Equal expenditure, and Cheapest only. At the smallest budget, B = 500 B = 500 B = 500 , it has the smallest reported worst-target RMSE among the procedures, including Oracle static, although the Monte Carlo uncertainty is substantial at this budget. Figure 4 shows the purchased allocation approaching the optimal design, with the mass on Loggers 2 and 5 shrinking toward zero. Note that simple cost-based rules are far from optimal here. The criterion ratio G ( z θ ) / G ⋆ G(z_\theta)/G^\star G ( z θ ) / G ⋆ , where z θ = θ / c ⊤ θ z_\theta= \theta/c^\top\theta z θ = θ / c ⊤ θ is the design with sample fractions θ \theta θ , is 2.50 for equal sample counts, 1.40 for Equal expenditure, and 1.21 for purchasing only from the cheapest logger.
Figure 4. ACTG 175 application. Left, worst-target RMSE with ± 1 \pm1 ± 1 Monte Carlo standard error. Right, the mean allocation of Adaptive Greedy across the five loggers, compared with the optimal allocation.
Discussion
In this paper, we studied budgeted data acquisition in off-policy evaluation, in which observations are purchased sequentially from a fixed set of loggers with heterogeneous costs. We derived the Adaptive Greedy rule from the pooled efficiency bound of Kallus et al. [28 ] , and showed that each purchase is an approximate Frank–Wolfe step (Frank and Wolfe, [15 ] ) on the worst-target criterion. We also showed that the purchased design converges to the minimax design. The pooled estimator computed from the purchased sample remains asymptotically normal, with variance equal to the fixed-design bound at the selected composition. Numerical studies show that the procedure matches the accuracy of repeated reoptimization at a fraction of its computational cost.
Three extensions remain open. First, we assumed that the logging probabilities are known. When they must be estimated, the estimation error enters both the purchasing rule and the final estimator, although the efficiency bound itself is unchanged [28 Theorem 5] . Second, inference for the final estimator relies on Assumption 5, and establishing it for common regression methods under adaptive sampling, in the spirit of Hadad et al. [19 ] and Bibaut et al. [3 ] , would be useful. Finally, we assumed fixed prices per observation. When prices depend on the counts already purchased (variable pricing), the feasible set and its vertices move with the design, and extending the Frank–Wolfe argument to this setting remains an open and interesting problem.
Notation and Overview
Table 1 (PDF p. 22).
Table 3.
The appendices are organized as follows. Appendix A2 contains the derivations used in § 3 , and Appendices A3 and A4 prove the results of § 4.1 and § 4.2 . Auxiliary results are stated where they are first needed; those with longer proofs are proved in Appendix A6, in the order of logical dependence. Appendix A5 treats the block-split estimator, and Appendix A7 gives the details of the numerical studies.
Derivations for Section 3
Derivation of (3.8)
Fix a logger k k k , and write T = 1 T N T = \mathbf{1}^{\mathsf{T}}N T = 1 T N . Then
Δ j , k ( N ) = G j ( N ) − G j ( N + e k ) = V j T + η 0 + E X ∑ a ∈ A W j ( X , a ) D N ( X , a ) − V j T + 1 + η 0 − E X ∑ a ∈ A W j ( X , a ) D N + e k ( X , a ) = V j ( T + η 0 ) ( T + 1 + η 0 ) + E X ∑ a ∈ A W j ( X , a ) μ k ( a ∣ X ) D N ( X , a ) { D N ( X , a ) + μ k ( a ∣ X ) } . \begin{aligned}
\Delta_{j,k}(N) &= G_j(N) - G_j(N + e_k) \\
&= \frac{V_j}{T+\eta_0} + \mathbb{E}_X \sum_{a\in\mathcal{A}} \frac{W_j(X,a)}{D_N(X,a)} - \frac{V_j}{T+1+\eta_0} - \mathbb{E}_X \sum_{a\in\mathcal{A}} \frac{W_j(X,a)}{D_{N+e_k}(X,a)} \\
&= \frac{V_j}{(T+\eta_0)(T+1+\eta_0)} + \mathbb{E}_X \sum_{a\in\mathcal{A}} \frac{W_j(X,a)\mu_k(a\mid X)}{D_N(X,a)\{D_N(X,a)+\mu_k(a\mid X)\}}.
\end{aligned} Δ j , k ( N ) = G j ( N ) − G j ( N + e k ) = T + η 0 V j + E X a ∈ A ∑ D N ( X , a ) W j ( X , a ) − T + 1 + η 0 V j − E X a ∈ A ∑ D N + e k ( X , a ) W j ( X , a ) = ( T + η 0 ) ( T + 1 + η 0 ) V j + E X a ∈ A ∑ D N ( X , a ) { D N ( X , a ) + μ k ( a ∣ X )} W j ( X , a ) μ k ( a ∣ X ) .
where the last line uses D N + e k = D N + μ k D_{N+e_k}=D_N+\mu_k D N + e k = D N + μ k and the identity u − 1 − ( u + w ) − 1 = w { u ( u + w ) } − 1 u^{-1}-(u+w)^{-1}=w\{u(u+w)\}^{-1} u − 1 − ( u + w ) − 1 = w { u ( u + w ) } − 1 , once with u = T + η 0 u=T+\eta_0 u = T + η 0 and w = 1 w=1 w = 1 and once with u = D N ( x , a ) u=D_N(x,a) u = D N ( x , a ) and w = μ k ( a ∣ x ) w=\mu_k(a\mid x) w = μ k ( a ∣ x ) . Taking the difference inside E X \mathbb{E}_X E X is justified because W j ≥ 0 W_j\geq0 W j ≥ 0 and both integrands are bounded by W j ( X , a ) / η W_j(X,a)/\eta W j ( X , a ) / η , which is integrable.
Orders of G j G_j G j and Δ j , k \Delta_{j,k} Δ j , k
Let N ≠ 0 N\neq0 N = 0 , T = ∑ k N k T=\sum_k N_k T = ∑ k N k , and θ = N / T \theta=N/T θ = N / T . Since μ k ≤ 1 \mu_k\leq1 μ k ≤ 1 and m θ ≥ κ ( θ ) m_\theta\geq\kappa(\theta) m θ ≥ κ ( θ ) on S \mathcal{S} S ,
η + T κ ( θ ) ≤ D N ( x , a ) = η + T m θ ( a ∣ x ) ≤ η + T for ( x , a ) ∈ S . \eta+T\kappa(\theta)\leq D_N(x,a)=\eta+Tm_\theta(a\mid x)\leq\eta+T \qquad\text{for }(x,a)\in\mathcal{S}. η + T κ ( θ ) ≤ D N ( x , a ) = η + T m θ ( a ∣ x ) ≤ η + T for ( x , a ) ∈ S .
Only pairs in S \mathcal{S} S contribute to (6) and (13) , because W j ( x , a ) = 0 W_j(x,a)=0 W j ( x , a ) = 0 whenever π j ( a ∣ x ) = 0 \pi_j(a\mid x)=0 π j ( a ∣ x ) = 0 , and W ‾ j = E X ∑ a W j ( X , a ) \overline{W}_j=\mathbb{E}_X\sum_a W_j(X,a) W j = E X ∑ a W j ( X , a ) is finite since σ 2 ≤ R max 2 \sigma^2\leq R_{\max}^2 σ 2 ≤ R m a x 2 . Thus
V j T + η 0 + W ‾ j η + T ≤ G j ( N ) ≤ V j T + η 0 + W ‾ j η + T κ ( θ ) . \frac{V_j}{T+\eta_0}+\frac{\overline{W}_j}{\eta+T}\leq G_j(N)\leq\frac{V_j}{T+\eta_0}+\frac{\overline{W}_j}{\eta+T\kappa(\theta)}. T + η 0 V j + η + T W j ≤ G j ( N ) ≤ T + η 0 V j + η + T κ ( θ ) W j .
and, since D N { D N + μ k } ≥ { η + T κ ( θ ) } 2 D_N\{D_N+\mu_k\}\geq\{\eta+T\kappa(\theta)\}^2 D N { D N + μ k } ≥ { η + T κ ( θ ) } 2 on S \mathcal{S} S ,
Δ j , k ( N ) ≤ W ‾ j { η + T κ ( θ ) } 2 + V j ( T + η 0 ) ( T + 1 + η 0 ) . \Delta_{j,k}(N)\leq\frac{\overline{W}_j}{\{\eta+T\kappa(\theta)\}^2}+\frac{V_j}{(T+\eta_0)(T+1+\eta_0)}. Δ j , k ( N ) ≤ { η + T κ ( θ ) } 2 W j + ( T + η 0 ) ( T + 1 + η 0 ) V j .
Along any sequence of count vectors for which κ ( θ ) \kappa(\theta) κ ( θ ) stays bounded away from zero, it follows that G j ( N ) = O ( T − 1 ) G_j(N)=O(T^{-1}) G j ( N ) = O ( T − 1 ) and Δ j , k ( N ) = O ( T − 2 ) \Delta_{j,k}(N)=O(T^{-2}) Δ j , k ( N ) = O ( T − 2 ) as T → ∞ T\to\infty T → ∞ ; the lower bound shows that the first order is exact when V j + W ‾ j > 0 V_j+\overline{W}_j>0 V j + W j > 0 .
Proofs for Section 4.1
This appendix proves Theorem 4.1 and Proposition 4.1. We first establish parts (i) and (ii) of the theorem along the infinite-budget sequence of § 4.1 , in which every logger is always available, and then transfer them to the budgeted algorithm. Auxiliary results are stated where they are first needed, and those not proved here are proved in Appendix A6.
Proof of Theorem 4.1
Our argument goes through the infinite-budget sequence ( K ˉ t , N ˉ t , C ˉ t ) t ≥ n 0 (\bar{K}_t,\bar{N}_t,\bar{C}_t)_{t\ge n_0} ( K ˉ t , N ˉ t , C ˉ t ) t ≥ n 0 of § 4.1 . It follows Algorithm 1, but keeps every logger available at every step. Write T T T for its set of forced steps (those steps t t t at which κ ( θ ˉ t ) < χ t \kappa(\bar{\theta}_t)<\chi_t κ ( θ ˉ t ) < χ t ), z ˉ t = z N ˉ t \bar{z}_t=z_{\bar{N}_t} z ˉ t = z N ˉ t , and θ ˉ t = N ˉ t / t \bar{\theta}_t=\bar{N}_t/t θ ˉ t = N ˉ t / t ; by (4.1),
z ˉ t + 1 = ( 1 − γ t ) z ˉ t + γ t v t , v t = e K ˉ t + 1 c K ˉ t + 1 , γ t = c K ˉ t + 1 C ˉ t + 1 . \bar{z}_{t+1}=(1-\gamma_t)\bar{z}_t+\gamma_t v_t,\qquad v_t=\frac{e_{\bar{K}_{t+1}}}{c_{\bar{K}_{t+1}}},\qquad\gamma_t=\frac{c_{\bar{K}_{t+1}}}{\bar{C}_{t+1}}. z ˉ t + 1 = ( 1 − γ t ) z ˉ t + γ t v t , v t = c K ˉ t + 1 e K ˉ t + 1 , γ t = C ˉ t + 1 c K ˉ t + 1 .
Along this sequence, we call a purchase satisfying (4.5), with z t z_t z t replaced by z ˉ t \bar{z}_t z ˉ t , a δ t \delta_t δ t -approximate conditional-gradient step. The proof rests on two results. The first is an infinite-budget version of the theorem, proved in Appendix § 9.2 ; the second couples the two sequences, and is proved in Appendix A6. Unlike Theorem 4.1, which concerns the actual budgeted algorithm stopped at T B T_B T B , the following proposition establishes convergence and rates for the infinite-budget sequence, in which every logger remains affordable.
Proposition A3.1. Suppose the conditions of Theorem 4.1 hold.
(i) Almost surely,
z ˉ t → z ⋆ and θ ˉ t → θ ⋆ , (31) \bar{z}_t\to z^\star\qquad\text{and}\qquad\bar{\theta}_t\to\theta^\star, \tag*{(31)} z ˉ t → z ⋆ and θ ˉ t → θ ⋆ , (31) and only finitely many steps of the infinite-budget sequence are forced.
(ii) Under the additional conditions of part (ii) of Theorem 4.1, there are, on an event of probability one, finite random constants U ˉ G \bar{U}_G U ˉ G , U ˉ z \bar{U}_z U ˉ z , and U ˉ θ \bar{U}_\theta U ˉ θ such that, for all sufficiently large t t t ,
0 ≤ G ( z ˉ t ) − G ( z ⋆ ) ≤ U ˉ G C ˉ t − β , ∥ z ˉ t − z ⋆ ∥ 2 ≤ U ˉ z C ˉ t − β / 2 , ∥ θ ˉ t − θ ⋆ ∥ 2 ≤ U ˉ θ C ˉ t − β / 2 . (32) 0\le G(\bar{z}_t)-G(z^\star)\le\bar{U}_G\bar{C}_t^{-\beta},\qquad
\|\bar{z}_t-z^\star\|_2\le\bar{U}_z\bar{C}_t^{-\beta/2},\qquad
\|\bar{\theta}_t-\theta^\star\|_2\le\bar{U}_\theta\bar{C}_t^{-\beta/2}. \tag*{(32)} 0 ≤ G ( z ˉ t ) − G ( z ⋆ ) ≤ U ˉ G C ˉ t − β , ∥ z ˉ t − z ⋆ ∥ 2 ≤ U ˉ z C ˉ t − β /2 , ∥ θ ˉ t − θ ⋆ ∥ 2 ≤ U ˉ θ C ˉ t − β /2 . (32)
Lemma A3.1. Let C n 0 C_{n_0} C n 0 be the cost of the initialization, suppose B > C n 0 + c max B>C_{n_0}+c_{\max} B > C n 0 + c m a x , and apply Algorithm 1 with budget B B B to the same observations as the infinite-budget sequence. Define τ B = min { t ≥ n 0 : B − C ˉ t < c max } \tau_B=\min\{t\ge n_0:B-\bar{C}_t<c_{\max}\} τ B = min { t ≥ n 0 : B − C ˉ t < c m a x } and L 0 = ⌈ c max / c min ⌉ L_0=\lceil c_{\max}/c_{\min}\rceil L 0 = ⌈ c m a x / c m i n ⌉ . The two sequences agree through time τ B \tau_B τ B , and
0 ≤ T B − τ B ≤ L 0 , ∥ N T B − N ˉ T B ∥ 1 ≤ 2 L 0 , ∣ C T B − C ˉ T B ∣ ≤ L 0 c max . 0\le T_B-\tau_B\le L_0,\qquad
\|N_{T_B}-\bar{N}_{T_B}\|_1\le2L_0,\qquad
|C_{T_B}-\bar{C}_{T_B}|\le L_0c_{\max}. 0 ≤ T B − τ B ≤ L 0 , ∥ N T B − N ˉ T B ∥ 1 ≤ 2 L 0 , ∣ C T B − C ˉ T B ∣ ≤ L 0 c m a x . Consequently there are finite constants K z K_z K z and K θ K_\theta K θ such that, for all sufficiently large B B B ,
∥ z T B − z ˉ T B ∥ 2 ≤ K z B − 1 , ∥ θ ^ B − N ˉ T B T B ∥ 2 ≤ K θ B − 1 . \|z_{T_B}-\bar{z}_{T_B}\|_2\le K_zB^{-1},\qquad
\left\|\hat{\theta}_B-\frac{\bar{N}_{T_B}}{T_B}\right\|_2\le K_\theta B^{-1}. ∥ z T B − z ˉ T B ∥ 2 ≤ K z B − 1 , θ ^ B − T B N ˉ T B 2 ≤ K θ B − 1 .
Given these two results, part (i) is immediate. Since T B ≥ ⌊ ( B − c max ) / c max ⌋ T_B\ge\lfloor(B-c_{\max})/c_{\max}\rfloor T B ≥ ⌊( B − c m a x ) / c m a x ⌋ tends to infinity, (A3.1) gives z ˉ T B → z ⋆ \bar{z}_{T_B}\to z^\star z ˉ T B → z ⋆ and N ˉ T B / T B → θ ⋆ \bar{N}_{T_B}/T_B\to\theta^\star N ˉ T B / T B → θ ⋆ almost surely; by Lemma A3.1, z T B z_{T_B} z T B and θ ^ B \hat{\theta}_B θ ^ B differ from these quantities by O ( B − 1 ) O(B^{-1}) O ( B − 1 ) , so they converge to the same limits.
For part (ii), we need to express the rates (A3.2) in terms of B B B . By Lemma A3.1,
C ˉ T B ≥ C T B − L 0 c max > B − ( L 0 + 1 ) c max , \bar{C}_{T_B}\ge C_{T_B}-L_0c_{\max}>B-(L_0+1)c_{\max}, C ˉ T B ≥ C T B − L 0 c m a x > B − ( L 0 + 1 ) c m a x ,
hence C ˉ T B ≥ B / 2 \bar{C}_{T_B}\ge B/2 C ˉ T B ≥ B /2 and C ˉ T B − β ≤ 2 β B − β \bar{C}_{T_B}^{-\beta}\le2^\beta B^{-\beta} C ˉ T B − β ≤ 2 β B − β for all large B B B ; since T B → ∞ T_B\to\infty T B → ∞ , the bounds (A3.2) also hold at t = T B t = T_B t = T B . Now split
G ( z T B ) − G ( z ∗ ) = [ G ( z T B ) − G ( z ˉ T B ) ] + [ G ( z ˉ T B ) − G ( z ∗ ) ] . \mathcal{G}(z_{T_B})-\mathcal{G}(z^*)=\left[\mathcal{G}(z_{T_B})-\mathcal{G}(\bar{z}_{T_B})\right]+\left[\mathcal{G}(\bar{z}_{T_B})-\mathcal{G}(z^*)\right]. G ( z T B ) − G ( z ∗ ) = [ G ( z T B ) − G ( z ˉ T B ) ] + [ G ( z ˉ T B ) − G ( z ∗ ) ] .
For large B B B , Lemma A3.5 gives κ ( N ˉ T B / T B ) ≥ κ ∗ / 2 \kappa(\bar{N}_{T_B}/T_B)\geq\kappa_*/2 κ ( N ˉ T B / T B ) ≥ κ ∗ /2 . With e ∗ = min { κ ∗ / ( 4 c max ) , 1 / 2 } e_*=\min\{\kappa_*/(4c_{\max}),1/2\} e ∗ = min { κ ∗ / ( 4 c m a x ) , 1/2 } , it follows that z ˉ T B ∈ C ( 2 e ∗ ) \bar{z}_{T_B}\in C(2e_*) z ˉ T B ∈ C ( 2 e ∗ ) . Because ∣ μ ( x , a ) ⊤ ( v − w ) ∣ ≤ K ∥ v − w ∥ 2 |\mu(x,a)^\top(v-w)|\leq\sqrt{K}\|v-w\|_2 ∣ μ ( x , a ) ⊤ ( v − w ) ∣ ≤ K ∥ v − w ∥ 2 , Lemma A3.1 then puts z T B z_{T_B} z T B in C ( e ∗ ) C(e_*) C ( e ∗ ) as well. Each G j G_j G j has a bounded gradient on the convex set C ( e ∗ ) C(e_*) C ( e ∗ ) , so G \mathcal{G} G is Lipschitz there with constant L G = max j sup C ( e ∗ ) ∥ ∇ G j ∥ 2 L_{\mathcal{G}}=\max_j\sup_{C(e_*)}\|\nabla G_j\|_2 L G = max j sup C ( e ∗ ) ∥∇ G j ∥ 2 . With Lemma A3.1, this bounds the first bracket, and Proposition A3.1 bounds the second:
G ( z T B ) − G ( z ˉ T B ) ≤ L G K z B , G ( z ˉ T B ) − G ( z ∗ ) ≤ U ˉ G C ˉ T B − β ≤ 2 β U ˉ G B − β . \mathcal{G}(z_{T_B})-\mathcal{G}(\bar{z}_{T_B})\leq\frac{L_{\mathcal{G}}K_z}{B},\qquad\mathcal{G}(\bar{z}_{T_B})-\mathcal{G}(z^*)\leq\bar{U}_{\mathcal{G}}\bar{C}_{T_B}^{-\beta}\leq2^\beta\bar{U}_{\mathcal{G}}B^{-\beta}. G ( z T B ) − G ( z ˉ T B ) ≤ B L G K z , G ( z ˉ T B ) − G ( z ∗ ) ≤ U ˉ G C ˉ T B − β ≤ 2 β U ˉ G B − β .
As β ≤ 1 / 2 \beta\leq1/2 β ≤ 1/2 ,
0 ≤ G ( z T B ) − G ( z ∗ ) ≤ U G B − β , U G = L G K z + 2 β U ˉ G . 0\leq\mathcal{G}(z_{T_B})-\mathcal{G}(z^*)\leq U_{\mathcal{G}}B^{-\beta},\qquad U_{\mathcal{G}}=L_{\mathcal{G}}K_z+2^\beta\bar{U}_{\mathcal{G}}. 0 ≤ G ( z T B ) − G ( z ∗ ) ≤ U G B − β , U G = L G K z + 2 β U ˉ G .
For the sample fractions, Lemma A3.1 and Proposition A3.1 give
∥ θ ^ B − θ ∗ ∥ 2 ≤ ∥ θ ^ B − N ˉ T B T B ∥ 2 + ∥ θ ˉ T B − θ ∗ ∥ 2 ≤ K θ B + 2 β / 2 U ˉ θ B − β / 2 ≤ U θ B − β / 2 , \|\hat{\theta}_B-\theta^*\|_2\leq\left\|\hat{\theta}_B-\frac{\bar{N}_{T_B}}{T_B}\right\|_2+\|\bar{\theta}_{T_B}-\theta^*\|_2\leq\frac{K_\theta}{B}+2^{\beta/2}\bar{U}_\theta B^{-\beta/2}\leq U_\theta B^{-\beta/2}, ∥ θ ^ B − θ ∗ ∥ 2 ≤ θ ^ B − T B N ˉ T B 2 + ∥ θ ˉ T B − θ ∗ ∥ 2 ≤ B K θ + 2 β /2 U ˉ θ B − β /2 ≤ U θ B − β /2 ,
with U θ = K θ + 2 β / 2 U ˉ θ U_\theta=K_\theta+2^{\beta/2}\bar{U}_\theta U θ = K θ + 2 β /2 U ˉ θ .
Proof of Proposition A3.1
We first collect the results that the proof relies on: strong convexity of G \mathcal{G} G , five properties of the smoothed criterion, control of the pooled coverage under forced sampling, and an elementary fact about recursions. Their proofs are given in Appendix A6. The proof also uses Lemma A3.10 of Appendix A3.3, by which unforced purchases are approximate conditional-gradient steps.
Recall that G = max j G j \mathcal{G}=\max_j G_j G = max j G j , where
G j ( v ) = V j 1 ⊤ v + E X ∑ a ∈ A W j ( X , a ) μ ( X , a ) ⊤ v , μ ( x , a ) = ( μ 1 ( a ∣ x ) , … , μ K ( a ∣ x ) ) ⊤ , v ∈ C . G_j(v)=\frac{V_j}{\mathbf{1}^\top v}+\mathbb{E}_X\sum_{a\in\mathcal{A}}\frac{W_j(X,a)}{\mu(X,a)^\top v},\qquad\mu(x,a)=(\mu_1(a\mid x),\ldots,\mu_K(a\mid x))^\top,\qquad v\in C. G j ( v ) = 1 ⊤ v V j + E X a ∈ A ∑ μ ( X , a ) ⊤ v W j ( X , a ) , μ ( x , a ) = ( μ 1 ( a ∣ x ) , … , μ K ( a ∣ x ) ) ⊤ , v ∈ C .
To bound the denominators, we restrict v v v to designs with positive coverage on S S S ,
C + = { v ∈ C : ess inf ( x , a ) ∈ S μ ( x , a ) ⊤ v > 0 } = ⋃ e > 0 C ( e ) , C ( e ) = { v ∈ C : ess inf ( x , a ) ∈ S μ ( x , a ) ⊤ v ≥ e } . C_+=\left\{v\in C:\operatorname*{ess\,inf}_{(x,a)\in S}\mu(x,a)^\top v>0\right\}=\bigcup_{e>0}C(e),\qquad C(e)=\left\{v\in C:\operatorname*{ess\,inf}_{(x,a)\in S}\mu(x,a)^\top v\geq e\right\}. C + = { v ∈ C : ( x , a ) ∈ S ess inf μ ( x , a ) ⊤ v > 0 } = e > 0 ⋃ C ( e ) , C ( e ) = { v ∈ C : ( x , a ) ∈ S ess inf μ ( x , a ) ⊤ v ≥ e } .
These sets are convex because μ ( x , a ) ⊤ v \mu(x,a)^\top v μ ( x , a ) ⊤ v is linear in v v v . Moreover, μ ( x , a ) ⊤ v = ( 1 ⊤ v ) m v / 1 ⊤ v ( a ∣ x ) \mu(x,a)^\top v=(\mathbf{1}^\top v)m_{v/\mathbf{1}^\top v}(a\mid x) μ ( x , a ) ⊤ v = ( 1 ⊤ v ) m v / 1 ⊤ v ( a ∣ x ) and 1 ⊤ v ≥ c max − 1 \mathbf{1}^\top v\geq c_{\max}^{-1} 1 ⊤ v ≥ c m a x − 1 on C C C , and so every v ∈ C v\in C v ∈ C with κ ( v / 1 ⊤ v ) > 0 \kappa(v/\mathbf{1}^\top v)>0 κ ( v / 1 ⊤ v ) > 0 lies in C ( e ) C(e) C ( e ) with e = κ ( v / 1 ⊤ v ) / c max e=\kappa(v/\mathbf{1}^\top v)/c_{\max} e = κ ( v / 1 ⊤ v ) / c m a x .
Proposition A3.2. Under Assumptions 1 and 3 , for every v ∈ C + v\in C_+ v ∈ C + and every h h h with c ⊤ h = 0 c^\top h=0 c ⊤ h = 0 ,
h ⊤ ∇ 2 G j ( v ) h ≥ 2 c min 3 λ j ∥ h ∥ 2 2 . (33) h^\top\nabla^2G_j(v)h\geq2c_{\min}^3\lambda_j\|h\|_2^2. \tag*{(33)} h ⊤ ∇ 2 G j ( v ) h ≥ 2 c m i n 3 λ j ∥ h ∥ 2 2 . (33)
Consequently each G j G_j G j , and therefore G \mathcal{G} G , is m m m -strongly convex along every segment contained in C + C_+ C + , with m = 2 c min 3 min j λ j > 0 m=2c_{\min}^3\min_j\lambda_j>0 m = 2 c m i n 3 min j λ j > 0 .
Strong convexity enters the argument only through the following quadratic growth around z ∗ z^* z ∗ , which converts convergence of the criterion into convergence of the design.
Lemma A3.2. Under Assumptions 1, 2, and 3 , G ( v ) − G ( z ∗ ) ≥ m 2 ∥ v − z ∗ ∥ 2 2 \mathcal{G}(v)-\mathcal{G}(z^*)\geq\frac{m}{2}\|v-z^*\|_2^2 G ( v ) − G ( z ∗ ) ≥ 2 m ∥ v − z ∗ ∥ 2 2 for every v ∈ C v\in C v ∈ C , so z ∗ z^* z ∗ is the unique minimizer of G \mathcal{G} G over C C C .
The recursion below uses five properties of Φ ρ \Phi_\rho Φ ρ in (3.9), where
ω j ( v ; ρ ) = exp { G j ( v ) / ρ } ∑ ℓ exp { G ℓ ( v ) / ρ } . (34) \omega_j(v;\rho)=\frac{\exp\{G_j(v)/\rho\}}{\sum_\ell\exp\{G_\ell(v)/\rho\}}. \tag*{(34)} ω j ( v ; ρ ) = ∑ ℓ exp { G ℓ ( v ) / ρ } exp { G j ( v ) / ρ } . (34)
Lemma A3.3. Under Assumption 1, parts (a), (b), (c), and (e) hold on C + C_+ C + , and part (d) holds on C ( e ) C(e) C ( e ) for every e ∈ ( 0 , 1 ] e \in(0,1] e ∈ ( 0 , 1 ] .
(a) G ( v ) ≤ Φ ρ ( v ) ≤ G ( v ) + ρ log M G(v) \le\Phi_\rho(v) \le G(v) + \rho\log M G ( v ) ≤ Φ ρ ( v ) ≤ G ( v ) + ρ log M .
(b) ∇ Φ ρ ( v ) = ∑ j ω j ( v ; ρ ) ∇ G j ( v ) \nabla\Phi_\rho(v) = \sum_j \omega_j(v;\rho)\nabla G_j(v) ∇ Φ ρ ( v ) = ∑ j ω j ( v ; ρ ) ∇ G j ( v ) .
(c) Φ ρ \Phi_\rho Φ ρ is convex.
(d) ∇ Φ ρ \nabla\Phi_\rho ∇ Φ ρ is Lipschitz on C ( e ) C(e) C ( e ) with constant at most L 0 ( e ) + L 1 ( e ) / ρ L_0(e)+L_1(e)/\rho L 0 ( e ) + L 1 ( e ) / ρ , where L 0 ( e ) = 2 K R max 2 ( c max 3 + e − 3 ) L_0(e) = 2KR_{\max}^2(c_{\max}^3+e^{-3}) L 0 ( e ) = 2 K R m a x 2 ( c m a x 3 + e − 3 ) and L 1 ( e ) = K R max 4 ( c max 2 + e − 2 ) 2 L_1(e) = KR_{\max}^4(c_{\max}^2+e^{-2})^2 L 1 ( e ) = K R m a x 4 ( c m a x 2 + e − 2 ) 2 .
(e) For fixed v v v , ρ ↦ Φ ρ ( v ) \rho\mapsto\Phi_\rho(v) ρ ↦ Φ ρ ( v ) is nondecreasing, so ρ ′ ≤ ρ \rho' \le\rho ρ ′ ≤ ρ implies Φ ρ ′ ≤ Φ ρ \Phi_{\rho'} \le\Phi_\rho Φ ρ ′ ≤ Φ ρ .
Next, we control the coverage under forced sampling. The two lemmas below concern any sequence generated by Algorithm 1 during the steps at which every logger is affordable, in particular the infinite-budget sequence. Write θ t = N t / t \theta_t = N_t/t θ t = N t / t , call step t t t forced if κ ( θ t ) < χ t \kappa(\theta_t) < \chi_t κ ( θ t ) < χ t , and let T \mathcal{T} T be the set of forced steps. For the analysis, let
κ a l l = K κ ( 1 / K ) = ess inf ( x , a ) ∈ S ∑ k = 1 K μ k ( a ∣ x ) . \kappa_{\mathrm{all}} = K\kappa(1/K) = \operatorname*{ess\,inf}_{(x,a)\in S}\sum_{k=1}^{K}\mu_k(a\mid x). κ all = K κ ( 1/ K ) = ( x , a ) ∈ S ess inf k = 1 ∑ K μ k ( a ∣ x ) .
so that the condition 0 < χ 0 ≤ κ ( 1 / K ) / 8 0 < \chi_0 \le\kappa(1/K)/8 0 < χ 0 ≤ κ ( 1/ K ) /8 of § 3.3 reads 0 < χ 0 ≤ κ a l l / ( 8 K ) 0 < \chi_0 \le\kappa_{\mathrm{all}}/(8K) 0 < χ 0 ≤ κ all / ( 8 K ) . By definition of κ a l l \kappa_{\mathrm{all}} κ all , for every θ ∈ S K \theta\in S_K θ ∈ S K and every ( x , a ) ∈ S (x,a) \in S ( x , a ) ∈ S ,
m θ ( a ∣ x ) ≥ ( min k θ k ) ∑ k = 1 K μ k ( a ∣ x ) , κ ( θ ) ≥ κ a l l min k θ k . (35) m_\theta(a\mid x) \ge\left(\min_k \theta_k\right)\sum_{k=1}^{K}\mu_k(a\mid x), \qquad\kappa(\theta) \ge\kappa_{\mathrm{all}}\min_k \theta_k. \tag*{(35)} m θ ( a ∣ x ) ≥ ( k min θ k ) k = 1 ∑ K μ k ( a ∣ x ) , κ ( θ ) ≥ κ all k min θ k . (35)
Lemma A3.4. Suppose 0 < χ 0 ≤ κ a l l / ( 8 K ) 0 < \chi_0 \le\kappa_{\mathrm{all}}/(8K) 0 < χ 0 ≤ κ all / ( 8 K ) , 0 < υ < 1 0 < \upsilon< 1 0 < υ < 1 , and the initialization purchases n 0 / K ≥ 4 n_0/K \ge4 n 0 / K ≥ 4 observations from each logger. Then, as long as every logger is affordable,
(a) κ ( θ t ) ≥ χ t / 2 \kappa(\theta_t) \ge\chi_t/2 κ ( θ t ) ≥ χ t /2 for every t ≥ n 0 t \ge n_0 t ≥ n 0 ;
(b) each logger receives at most χ t t / κ a l l + 1 \chi_t t/\kappa_{\mathrm{all}} + 1 χ t t / κ all + 1 forced purchases at steps s ≤ t s \le t s ≤ t , and ∑ t ∈ T γ t < ∞ \sum_{t\in\mathcal{T}}\gamma_t < \infty ∑ t ∈ T γ t < ∞ , where γ t = c K t + 1 / C t + 1 \gamma_t = cK_{t+1}/C_{t+1} γ t = c K t + 1 / C t + 1 .
Summing part (b) over the K K K loggers, the number of forced purchases up to time t t t is at most K ( χ t t / κ a l l + 1 ) = O ( t 1 − υ ) K(\chi_t t/\kappa_{\mathrm{all}} + 1) = O(t^{1-\upsilon}) K ( χ t t / κ all + 1 ) = O ( t 1 − υ ) , as stated in § 3.3 .
Lemma A3.5. Under Assumption 2 and the conditions of Lemma A3.4, if θ t → θ ⋆ \theta_t \to\theta^\star θ t → θ ⋆ along a sequence in which every logger is always affordable, then only finitely many steps are forced, and κ ( θ t ) ≥ κ ⋆ / 2 \kappa(\theta_t) \ge\kappa_\star/2 κ ( θ t ) ≥ κ ⋆ /2 for all sufficiently large t t t .
Computing pooled coverage. When the context support is finite and known, κ ( θ ) \kappa(\theta) κ ( θ ) can be computed by taking the minimum of m θ ( a ∣ x ) m_\theta(a\mid x) m θ ( a ∣ x ) over all target-relevant context-action pairs. For general context spaces, forced sampling may instead use
κ ~ ( θ ) = ess inf ( x , a ) ∈ S ∑ k = 1 K θ k μ ~ k ( a ∣ x ) . \widetilde{\kappa}(\theta) = \operatorname*{ess\,inf}_{(x,a)\in S}\sum_{k=1}^{K}\theta_k\widetilde{\mu}_k(a\mid x). κ ( θ ) = ( x , a ) ∈ S ess inf k = 1 ∑ K θ k μ k ( a ∣ x ) .
where the known functions μ ~ k \widetilde{\mu}_k μ k satisfy 0 ≤ μ ~ k ≤ μ k 0 \le\widetilde{\mu}_k \le\mu_k 0 ≤ μ k ≤ μ k . For example, we can partition S S S into finitely many disjoint measurable regions S 1 , … , S R S_1,\ldots,S_R S 1 , … , S R of positive measure. For each logger k k k and region r r r , choose a known constant
0 ≤ μ k , r ≤ ess inf ( x , a ) ∈ S r μ k ( a ∣ x ) , 0 \le\mu_{k,r} \le\operatorname*{ess\,inf}_{(x,a)\in S_r}\mu_k(a\mid x), 0 ≤ μ k , r ≤ ( x , a ) ∈ S r ess inf μ k ( a ∣ x ) ,
and set μ ~ k ( a ∣ x ) = μ k , r \widetilde{\mu}_k(a\mid x) = \mu_{k,r} μ k ( a ∣ x ) = μ k , r for ( x , a ) ∈ S r (x,a) \in S_r ( x , a ) ∈ S r . Then κ ~ ( θ ) = min 1 ≤ r ≤ R ∑ k = 1 K θ k μ k , r ≤ κ ( θ ) \widetilde{\kappa}(\theta) = \min_{1\le r\le R}\sum_{k=1}^{K}\theta_k\mu_{k,r} \le\kappa(\theta) κ ( θ ) = min 1 ≤ r ≤ R ∑ k = 1 K θ k μ k , r ≤ κ ( θ ) . Thus, once the regional bounds are computed, evaluating κ ~ ( θ ) \widetilde{\kappa}(\theta) κ ( θ ) requires only a finite minimum of weighted sums, with O ( K R ) O(KR) O ( K R ) operations. For structured logging policies on bounded regions, such bounds can be obtained using interval methods [16 ] . Individual regional bounds may be zero, so this construction does not require every logger to cover every target-relevant action.
This alternative requires κ ~ ( θ ⋆ ) > 0 \widetilde{\kappa}(\theta^\star)>0 κ ( θ ⋆ ) > 0 and 0 < χ 0 ≤ κ ~ ( 1 / K ) / 8 0<\chi_0\le\widetilde{\kappa}(1/K)/8 0 < χ 0 ≤ κ ( 1/ K ) /8 . Because κ ~ \widetilde{\kappa} κ is an infimum of the nonnegative linear functions θ ↦ ∑ k θ k μ ~ k ( a ∣ x ) \theta\mapsto\sum_k\theta_k\widetilde{\mu}_k(a\mid x) θ ↦ ∑ k θ k μ k ( a ∣ x ) , it has the three properties of κ \kappa κ that the coverage proofs use:
κ ~ ( t θ + e k t + 1 ) ≥ t t + 1 κ ~ ( θ ) , κ ~ ( θ ) ≥ K κ ~ ( 1 / K ) min k θ k , ∣ κ ~ ( θ ) − κ ~ ( θ ′ ) ∣ ≤ ∥ θ − θ ′ ∥ 1 \widetilde{\kappa}\left(\frac{t\theta+e_k}{t+1}\right)\ge\frac{t}{t+1}\widetilde{\kappa}(\theta),\qquad\widetilde{\kappa}(\theta)\ge K\widetilde{\kappa}(1/K)\min_k\theta_k,\qquad\left|\widetilde{\kappa}(\theta)-\widetilde{\kappa}(\theta')\right|\le\lVert\theta-\theta'\rVert_1 κ ( t + 1 tθ + e k ) ≥ t + 1 t κ ( θ ) , κ ( θ ) ≥ K κ ( 1/ K ) k min θ k , ∣ κ ( θ ) − κ ( θ ′ ) ∣ ≤ ∥ θ − θ ′ ∥ 1
the last because μ ~ k ≤ 1 \widetilde{\mu}_k\le1 μ k ≤ 1 . The proofs of Lemmas A3.4 and A3.5 therefore go through with three substitutions. A step is forced if κ ~ ( θ ~ t ) < χ t \widetilde{\kappa}(\widetilde{\theta}_t)<\chi_t κ ( θ t ) < χ t ; K κ ~ ( 1 / K ) K\widetilde{\kappa}(1/K) K κ ( 1/ K ) replaces κ a l l \kappa_{\mathrm{all}} κ all , and (A3.4) holds with μ ~ k \widetilde{\mu}_k μ k and κ ~ \widetilde{\kappa} κ in place of μ k \mu_k μ k and κ \kappa κ . Finally, κ ~ ( θ ⋆ ) \widetilde{\kappa}(\theta^\star) κ ( θ ⋆ ) replaces κ ⋆ \kappa_\star κ ⋆ in the proof of Lemma A3.5. Since κ ≥ κ ~ \kappa\ge\widetilde{\kappa} κ ≥ κ , the remaining arguments are unchanged.
Finally, we need an elementary fact about recursions.
Lemma A3.6. Let h t ≥ 0 h_t\ge0 h t ≥ 0 and γ t ∈ ( 0 , 1 ] \gamma_t\in(0,1] γ t ∈ ( 0 , 1 ] with ∑ t γ t = ∞ \sum_t\gamma_t=\infty ∑ t γ t = ∞ , and let T \mathcal{T} T be a set of indices with ∑ t ∈ T γ t < ∞ \sum_{t\in\mathcal{T}}\gamma_t<\infty ∑ t ∈ T γ t < ∞ . Suppose that, for all sufficiently large t t t , h t + 1 ≤ ( 1 − γ t ) h t + γ t q t h_{t+1}\le(1-\gamma_t)h_t+\gamma_tq_t h t + 1 ≤ ( 1 − γ t ) h t + γ t q t for t ∉ T t\notin\mathcal{T} t ∈ / T , where q t → 0 q_t\to0 q t → 0 , and that h t + 1 ≤ ( 1 + γ t ) h t + γ t b h_{t+1}\le(1+\gamma_t)h_t+\gamma_tb h t + 1 ≤ ( 1 + γ t ) h t + γ t b for t ∈ T t\in\mathcal{T} t ∈ T , where b < ∞ b<\infty b < ∞ . Then h t → 0 h_t\to0 h t → 0 .
We start with part (i). Our main task is to show that the smoothed excess h ˉ t = Φ ρ t ( z ˉ t ) − G ( z ⋆ ) \bar{h}_t=\Phi_{\rho_t}(\bar{z}_t)-G(z^\star) h ˉ t = Φ ρ t ( z ˉ t ) − G ( z ⋆ ) converges to zero, since the remaining claims then follow quickly. By Lemma A3.3(a), 0 ≤ G ( z ˉ t ) − G ( z ⋆ ) ≤ h ˉ t 0\le G(\bar{z}_t)-G(z^\star)\le\bar{h}_t 0 ≤ G ( z ˉ t ) − G ( z ⋆ ) ≤ h ˉ t , hence Lemma A3.2 gives z ˉ t → z ⋆ \bar{z}_t\to z^\star z ˉ t → z ⋆ . Because v ↦ v / 1 ⊤ v v\mapsto v/\mathbf{1}^{\top}v v ↦ v / 1 ⊤ v is Lipschitz on C C C , where 1 ⊤ v ≥ c max − 1 \mathbf{1}^{\top}v\ge c_{\max}^{-1} 1 ⊤ v ≥ c m a x − 1 , it follows that θ ˉ t → θ ⋆ \bar{\theta}_t\to\theta^\star θ ˉ t → θ ⋆ ; Lemma A3.5 then shows that only finitely many steps are forced. All statements below hold pathwise on the event of probability one on which Assumption 4 holds. Since C ˉ t ≤ C ˉ n 0 + c max t \bar{C}_t\le\bar{C}_{n_0}+c_{\max}t C ˉ t ≤ C ˉ n 0 + c m a x t , the condition t α + 2 υ r t → 0 t^{\alpha+2\upsilon}r_t\to0 t α + 2 υ r t → 0 gives C ˉ t α + 2 υ r t → 0 \bar{C}_t^{\alpha+2\upsilon}r_t\to0 C ˉ t α + 2 υ r t → 0 .
We show that h ˉ t → 0 \bar{h}_t\to0 h ˉ t → 0 by verifying the conditions of Lemma A3.6. Let e t = χ t / ( 4 c max ) e_t=\chi_t/(4c_{\max}) e t = χ t / ( 4 c m a x ) , which is at most 1 / c max 1/c_{\max} 1/ c m a x because χ 0 ≤ κ a l l / ( 8 K ) ≤ 1 / 8 \chi_0\le\kappa_{\mathrm{all}}/(8K)\le1/8 χ 0 ≤ κ all / ( 8 K ) ≤ 1/8 . Lemma A3.4(a) gives κ ( θ ˉ t ) ≥ χ t / 2 \kappa(\bar{\theta}_t)\ge\chi_t/2 κ ( θ ˉ t ) ≥ χ t /2 , so that z ˉ t ∈ C ( 2 e t ) \bar{z}_t\in C(2e_t) z ˉ t ∈ C ( 2 e t ) ; since z ˉ t + 1 ≥ ( 1 − γ t ) z ˉ t \bar{z}_{t+1}\ge(1-\gamma_t)\bar{z}_t z ˉ t + 1 ≥ ( 1 − γ t ) z ˉ t and γ t ≤ 1 / 2 \gamma_t\le1/2 γ t ≤ 1/2 for large t t t , the segment from z ˉ t \bar{z}_t z ˉ t to z ˉ t + 1 \bar{z}_{t+1} z ˉ t + 1 lies in C ( e t ) C(e_t) C ( e t ) . Write L t = L 0 ( e t ) + L 1 ( e t ) / ρ t L_t=L_0(e_t)+L_1(e_t)/\rho_t L t = L 0 ( e t ) + L 1 ( e t ) / ρ t for the Lipschitz constant of Lemma A3.3(d) on C ( e t ) C(e_t) C ( e t ) , and d C d_C d C for the diameter of C C C , so that ∥ v t − z ˉ t ∥ 2 ≤ d C \lVert v_t-\bar{z}_t\rVert_2\le d_C ∥ v t − z ˉ t ∥ 2 ≤ d C . As z ˉ t + 1 − z ˉ t = γ t ( v t − z ˉ t ) \bar{z}_{t+1}-\bar{z}_t=\gamma_t(v_t-\bar{z}_t) z ˉ t + 1 − z ˉ t = γ t ( v t − z ˉ t ) , the usual descent inequality for functions with a Lipschitz gradient gives
Φ ρ t ( z ˉ t + 1 ) ≤ Φ ρ t ( z ˉ t ) + γ t ⟨ ∇ Φ ρ t ( z ˉ t ) , v t − z ˉ t ⟩ + 1 2 d C 2 L t γ t 2 . (36) \Phi_{\rho_t}(\bar{z}_{t+1})\le\Phi_{\rho_t}(\bar{z}_t)+\gamma_t\langle\nabla\Phi_{\rho_t}(\bar{z}_t),v_t-\bar{z}_t\rangle+\frac{1}{2}d_C^2L_t\gamma_t^2. \tag*{(36)} Φ ρ t ( z ˉ t + 1 ) ≤ Φ ρ t ( z ˉ t ) + γ t ⟨ ∇ Φ ρ t ( z ˉ t ) , v t − z ˉ t ⟩ + 2 1 d C 2 L t γ t 2 . (36)
The choice of logger enters only through the middle term. At an unforced step, Lemma A3.10 and z ⋆ ∈ C z^\star\in C z ⋆ ∈ C bound the first part of that term, while convexity of Φ ρ t \Phi_{\rho_t} Φ ρ t on C + C_+ C + , which contains z ˉ t , z ⋆ \bar{z}_t,z^\star z ˉ t , z ⋆ , and the segment between them, bounds the second:
⟨ ∇ Φ ρ t ( z ˉ t ) , v t ⟩ ≤ ⟨ ∇ Φ ρ t ( z ˉ t ) , z ⋆ ⟩ + δ t , ⟨ ∇ Φ ρ t ( z ˉ t ) , z ⋆ − z ˉ t ⟩ ≤ Φ ρ t ( z ⋆ ) − Φ ρ t ( z ˉ t ) . \langle\nabla\Phi_{\rho_t}(\bar{z}_t),v_t\rangle\le\langle\nabla\Phi_{\rho_t}(\bar{z}_t),z^\star\rangle+\delta_t,\qquad\langle\nabla\Phi_{\rho_t}(\bar{z}_t),z^\star-\bar{z}_t\rangle\le\Phi_{\rho_t}(z^\star)-\Phi_{\rho_t}(\bar{z}_t). ⟨ ∇ Φ ρ t ( z ˉ t ) , v t ⟩ ≤ ⟨ ∇ Φ ρ t ( z ˉ t ) , z ⋆ ⟩ + δ t , ⟨ ∇ Φ ρ t ( z ˉ t ) , z ⋆ − z ˉ t ⟩ ≤ Φ ρ t ( z ⋆ ) − Φ ρ t ( z ˉ t ) .
Plugging both bounds into (A3.5) and using Φ ρ t + 1 ≤ Φ ρ t \Phi_{\rho_{t+1}}\le\Phi_{\rho_t} Φ ρ t + 1 ≤ Φ ρ t (as ρ t + 1 ≤ ρ t \rho_{t+1}\le\rho_t ρ t + 1 ≤ ρ t ) and Φ ρ t ( z ⋆ ) ≤ G ( z ⋆ ) + ρ t log M \Phi_{\rho_t}(z^\star)\le G(z^\star)+\rho_t\log M Φ ρ t ( z ⋆ ) ≤ G ( z ⋆ ) + ρ t log M from Lemma A3.3(e) and (a), we obtain
h ˉ t + 1 ≤ ( 1 − γ t ) h ˉ t + γ t q t , q t = ρ t log M + δ t + 1 2 d C 2 L t γ t , t ∉ T . (37) \bar{h}_{t+1}\le(1-\gamma_t)\bar{h}_t+\gamma_tq_t,\qquad q_t=\rho_t\log M+\delta_t+\frac{1}{2}d_C^2L_t\gamma_t,\qquad t\notin\mathcal{T}. \tag*{(37)} h ˉ t + 1 ≤ ( 1 − γ t ) h ˉ t + γ t q t , q t = ρ t log M + δ t + 2 1 d C 2 L t γ t , t ∈ / T . (37)
At a forced step, we only use that purchasing never increases the criterion to first order. The partial derivatives of each G j \mathcal{G}_j G j are nonpositive and v t ≥ 0 v_t\ge0 v t ≥ 0 , so that ⟨ ∇ Φ ρ t ( z ˉ t ) , v t ⟩ ≤ 0 \langle\nabla\Phi_{\rho_t}(\bar{z}_t),v_t\rangle\le0 ⟨ ∇ Φ ρ t ( z ˉ t ) , v t ⟩ ≤ 0 ; meanwhile, homogeneity of degree − 1 -1 − 1 gives
− ⟨ ∇ Φ ρ t ( z ˉ t ) , z ˉ t ⟩ = ∑ j ω j ( z ˉ t ; ρ t ) G j ( z ˉ t ) ≤ Φ ρ t ( z ˉ t ) . -\langle\nabla\Phi_{\rho_t}(\bar{z}_t),\bar{z}_t\rangle=\sum_j\omega_j(\bar{z}_t;\rho_t)\mathcal{G}_j(\bar{z}_t)\le\Phi_{\rho_t}(\bar{z}_t). − ⟨ ∇ Φ ρ t ( z ˉ t ) , z ˉ t ⟩ = j ∑ ω j ( z ˉ t ; ρ t ) G j ( z ˉ t ) ≤ Φ ρ t ( z ˉ t ) .
Thus (A3.5) and Lemma A3.3(e) give
h ˉ t + 1 ≤ ( 1 + γ t ) h ˉ t + γ t { G ( z ⋆ ) + 1 2 d C 2 L t γ t } , t ∈ T . (38) \bar{h}_{t+1}\le(1+\gamma_t)\bar{h}_t+\gamma_t\left\{G(z^\star)+\frac{1}{2}d_C^2L_t\gamma_t\right\},\qquad t\in\mathcal{T}. \tag*{(38)} h ˉ t + 1 ≤ ( 1 + γ t ) h ˉ t + γ t { G ( z ⋆ ) + 2 1 d C 2 L t γ t } , t ∈ T . (38)
It remains to check the hypotheses of Lemma A3.6. Lemma A3.4(b) gives ∑ t ∈ T γ t < ∞ \sum_{t\in\mathcal{T}}\gamma_t<\infty ∑ t ∈ T γ t < ∞ , and, using C ˉ t ≥ c min t \bar{C}_t\ge c_{\min}t C ˉ t ≥ c m i n t for the second bound,
γ t ≥ c min C ˉ n 0 + c max ( t + 1 ) , χ t − 1 ≤ χ 0 − 1 c min − ν C ˉ t ν ; \gamma_t\ge\frac{c_{\min}}{\bar{C}_{n_0}+c_{\max}(t+1)},\qquad\chi_t^{-1}\le\chi_0^{-1}c_{\min}^{-\nu}\bar{C}_t^\nu; γ t ≥ C ˉ n 0 + c m a x ( t + 1 ) c m i n , χ t − 1 ≤ χ 0 − 1 c m i n − ν C ˉ t ν ;
in particular, ∑ t γ t = ∞ \sum_t\gamma_t=\infty ∑ t γ t = ∞ . Since e t ≤ 1 / c max e_t\le1/c_{\max} e t ≤ 1/ c m a x , Lemma A3.3(d) also gives
L 0 ( e t ) ≤ 4 K R max 2 e t − 3 , L 1 ( e t ) ≤ 4 K R max 4 e t − 4 . L_0(e_t)\le4KR_{\max}^2e_t^{-3},\qquad L_1(e_t)\le4KR_{\max}^4e_t^{-4}. L 0 ( e t ) ≤ 4 K R m a x 2 e t − 3 , L 1 ( e t ) ≤ 4 K R m a x 4 e t − 4 .
Combined with γ t ≤ c max / C ˉ t \gamma_t\le c_{\max}/\bar{C}_t γ t ≤ c m a x / C ˉ t and ρ t = ρ 0 C ˉ t − α \rho_t=\rho_0\bar{C}_t^{-\alpha} ρ t = ρ 0 C ˉ t − α , these bounds yield finite constants D 4 D_4 D 4 and D 5 D_5 D 5 such that
L t γ t ≤ D 4 ( C ˉ t 3 ν − 1 + C ˉ t 4 ν + α − 1 ) , δ t ≤ D 5 ( C ˉ t α + 2 ν r t + C ˉ t α + 4 ν − 1 ) ; L_t\gamma_t\le D_4\left(\bar{C}_t^{3\nu-1}+\bar{C}_t^{4\nu+\alpha-1}\right),\qquad\delta_t\le D_5\left(\bar{C}_t^{\alpha+2\nu}r_t+\bar{C}_t^{\alpha+4\nu-1}\right); L t γ t ≤ D 4 ( C ˉ t 3 ν − 1 + C ˉ t 4 ν + α − 1 ) , δ t ≤ D 5 ( C ˉ t α + 2 ν r t + C ˉ t α + 4 ν − 1 ) ;
for the second bound, Lemma A3.10 and Proposition A3.3 show that δ t \delta_t δ t is at most a constant multiple of κ ( θ ˉ t ) − 2 C ˉ t α r t + κ ( θ ˉ t ) − 4 C ˉ t α − 1 \kappa(\bar{\theta}_t)^{-2}\bar{C}_t^\alpha r_t+\kappa(\bar{\theta}_t)^{-4}\bar{C}_t^{\alpha-1} κ ( θ ˉ t ) − 2 C ˉ t α r t + κ ( θ ˉ t ) − 4 C ˉ t α − 1 . Both bounds tend to zero because ν < ( 1 − α ) / 4 \nu<(1-\alpha)/4 ν < ( 1 − α ) /4 and C ˉ t α + 2 ν r t → 0 \bar{C}_t^{\alpha+2\nu}r_t\to0 C ˉ t α + 2 ν r t → 0 , thus q t → 0 q_t\to0 q t → 0 . Finally, the additive term in (A3.7) is at most γ t b \gamma_tb γ t b with b = G ( z ⋆ ) + sup t 1 2 d C 2 L t γ t < ∞ b=G(z^\star)+\sup_t\frac{1}{2}d_{\mathcal{C}}^2L_t\gamma_t<\infty b = G ( z ⋆ ) + sup t 2 1 d C 2 L t γ t < ∞ , and Lemma A3.6 gives h ˉ t → 0 \bar{h}_t\to0 h ˉ t → 0 .
For part (ii), part (i) and Lemma A3.5 provide a finite random T 0 T_0 T 0 after which no step is forced and κ ( θ ˉ t ) ≥ κ ⋆ / 2 \kappa(\bar{\theta}_t)\ge\kappa_\star/2 κ ( θ ˉ t ) ≥ κ ⋆ /2 . For t ≥ T 0 t\ge T_0 t ≥ T 0 , then, z ˉ t ∈ C ( 2 e ⋆ ) \bar{z}_t\in C(2e_\star) z ˉ t ∈ C ( 2 e ⋆ ) with e ⋆ = min { κ ⋆ / ( 4 c max ) , 1 / 2 } e_\star=\min\{\kappa_\star/(4c_{\max}),1/2\} e ⋆ = min { κ ⋆ / ( 4 c m a x ) , 1/2 } , and the segment to z ˉ t + 1 \bar{z}_{t+1} z ˉ t + 1 lies in C ( e ⋆ ) C(e_\star) C ( e ⋆ ) for large t t t . Thus (A3.6) holds with constants L 0 ( e ⋆ ) L_0(e_\star) L 0 ( e ⋆ ) and L 1 ( e ⋆ ) L_1(e_\star) L 1 ( e ⋆ ) that do not depend on t t t , and we bound the three terms of q t q_t q t in turn. The smoothing bias is ρ 0 ( log M ) C ˉ t − α \rho_0(\log M)\bar{C}_t^{-\alpha} ρ 0 ( log M ) C ˉ t − α . For the other two terms, Lemma A3.10, applied with m = κ ⋆ / 2 m=\kappa_\star/2 m = κ ⋆ /2 and the condition r t = O ( t − β r ) r_t=O(t^{-\beta_r}) r t = O ( t − β r ) of part (ii) of Theorem 4.1, and the bound γ t ≤ c max / C ˉ t \gamma_t\le c_{\max}/\bar{C}_t γ t ≤ c m a x / C ˉ t give
δ t ≤ D 2 ( C ˉ t α − 1 + C ˉ t α − β r ) , \delta_t\le D_2\left(\bar{C}_t^{\alpha-1}+\bar{C}_t^{\alpha-\beta_r}\right), δ t ≤ D 2 ( C ˉ t α − 1 + C ˉ t α − β r ) ,
1 2 d C 2 L t γ t ≤ 1 2 d C 2 { L 0 ( e ⋆ ) + L 1 ( e ⋆ ) C ˉ t α ρ 0 } c max C ˉ t ≤ D 3 ( C ˉ t − 1 + C ˉ t α − 1 ) . \frac{1}{2}d_{\mathcal{C}}^2L_t\gamma_t\le\frac{1}{2}d_{\mathcal{C}}^2\left\{L_0(e_\star)+\frac{L_1(e_\star)\bar{C}_t^\alpha}{\rho_0}\right\}\frac{c_{\max}}{\bar{C}_t}\le D_3\left(\bar{C}_t^{-1}+\bar{C}_t^{\alpha-1}\right). 2 1 d C 2 L t γ t ≤ 2 1 d C 2 { L 0 ( e ⋆ ) + ρ 0 L 1 ( e ⋆ ) C ˉ t α } C ˉ t c m a x ≤ D 3 ( C ˉ t − 1 + C ˉ t α − 1 ) .
Since β \beta β is no larger than α \alpha α , 1 − α 1-\alpha 1 − α , β r − α \beta_r-\alpha β r − α , or 1 1 1 , each of these powers is at most C ˉ t − β \bar{C}_t^{-\beta} C ˉ t − β once C ˉ t ≥ 1 \bar{C}_t\ge1 C ˉ t ≥ 1 ; thus there is a finite Q Q Q with q t ≤ Q C ˉ t − β q_t\le Q\bar{C}_t^{-\beta} q t ≤ Q C ˉ t − β for all large t t t .
To conclude, we solve the recursion with this bound. Because γ t = ( C ˉ t + 1 − C ˉ t ) / C ˉ t + 1 \gamma_t=(\bar{C}_{t+1}-\bar{C}_t)/\bar{C}_{t+1} γ t = ( C ˉ t + 1 − C ˉ t ) / C ˉ t + 1 , multiplying (A3.6) by C ˉ t + 1 \bar{C}_{t+1} C ˉ t + 1 makes it telescope:
C ˉ t + 1 h ˉ t + 1 ≤ C ˉ t h ˉ t + ( C ˉ t + 1 − C ˉ t ) q t . \bar{C}_{t+1}\bar{h}_{t+1}\le\bar{C}_t\bar{h}_t+(\bar{C}_{t+1}-\bar{C}_t)q_t. C ˉ t + 1 h ˉ t + 1 ≤ C ˉ t h ˉ t + ( C ˉ t + 1 − C ˉ t ) q t .
Choose t 0 ≥ T 0 t_0\ge T_0 t 0 ≥ T 0 beyond which the bound on q t q_t q t holds and with C ˉ t 0 ≥ max { 1 , c max } \bar{C}_{t_0}\ge\max\{1,c_{\max}\} C ˉ t 0 ≥ max { 1 , c m a x } . Summing from t 0 t_0 t 0 to t − 1 t-1 t − 1 ,
C ˉ t h ˉ t ≤ C ˉ t 0 h ˉ t 0 + Q ∑ s = t 0 t − 1 ( C ˉ s + 1 − C ˉ s ) C ˉ s − β . (39) \bar{C}_t\bar{h}_t\le\bar{C}_{t_0}\bar{h}_{t_0}+Q\sum_{s=t_0}^{t-1}(\bar{C}_{s+1}-\bar{C}_s)\bar{C}_s^{-\beta}. \tag*{(39)} C ˉ t h ˉ t ≤ C ˉ t 0 h ˉ t 0 + Q s = t 0 ∑ t − 1 ( C ˉ s + 1 − C ˉ s ) C ˉ s − β . (39)
The sum is a Riemann sum for ∫ u − β d u \int u^{-\beta}\,du ∫ u − β d u : for s ≥ t 0 s\ge t_0 s ≥ t 0 , C ˉ s + 1 ≤ C ˉ s + c max ≤ 2 C ˉ s \bar{C}_{s+1}\le\bar{C}_s+c_{\max}\le2\bar{C}_s C ˉ s + 1 ≤ C ˉ s + c m a x ≤ 2 C ˉ s , and so
( C ˉ s + 1 − C ˉ s ) C ˉ s − β ≤ 2 ( C ˉ s + 1 − C ˉ s ) C ˉ s + 1 − β ≤ 2 ∫ C ˉ s C ˉ s + 1 u − β d u . (\bar{C}_{s+1}-\bar{C}_s)\bar{C}_s^{-\beta}\le2(\bar{C}_{s+1}-\bar{C}_s)\bar{C}_{s+1}^{-\beta}\le2\int_{\bar{C}_s}^{\bar{C}_{s+1}}u^{-\beta}\,du. ( C ˉ s + 1 − C ˉ s ) C ˉ s − β ≤ 2 ( C ˉ s + 1 − C ˉ s ) C ˉ s + 1 − β ≤ 2 ∫ C ˉ s C ˉ s + 1 u − β d u .
The sum in (A3.8) is therefore at most 2 C ˉ t 1 − β / ( 1 − β ) 2\bar{C}_t^{1-\beta}/(1-\beta) 2 C ˉ t 1 − β / ( 1 − β ) , and dividing by C ˉ t \bar{C}_t C ˉ t gives
h ˉ t ≤ C ˉ t 0 h ˉ t 0 C ˉ t + 2 Q 1 − β C ˉ t − β ≤ U ˉ G C ˉ t − β , \bar{h}_t\le\frac{\bar{C}_{t_0}\bar{h}_{t_0}}{\bar{C}_t}+\frac{2Q}{1-\beta}\bar{C}_t^{-\beta}\le\bar{U}_G\bar{C}_t^{-\beta}, h ˉ t ≤ C ˉ t C ˉ t 0 h ˉ t 0 + 1 − β 2 Q C ˉ t − β ≤ U ˉ G C ˉ t − β ,
because β ≤ 1 / 2 < 1 \beta\le1/2<1 β ≤ 1/2 < 1 . This is the first bound in (A3.2), since G ( z ˉ t ) − G ( z ⋆ ) ≤ h ˉ t G(\bar{z}_t)-G(z^\star)\le\bar{h}_t G ( z ˉ t ) − G ( z ⋆ ) ≤ h ˉ t . Lemma A3.2 then gives ∥ z ˉ t − z ⋆ ∥ 2 2 ≤ ( 2 U ˉ G / m ) C ˉ t − β \lVert\bar{z}_t-z^\star\rVert_2^2\le(2\bar{U}_G/m)\bar{C}_t^{-\beta} ∥ z ˉ t − z ⋆ ∥ 2 2 ≤ ( 2 U ˉ G / m ) C ˉ t − β , which is the second bound, and the third follows because v ↦ v / 1 ⊤ v v\mapsto v/\mathbf{1}^{\top}v v ↦ v / 1 ⊤ v is Lipschitz on C C C . The constants depend on the realized path through t 0 t_0 t 0 and h ˉ t 0 \bar{h}_{t_0} h ˉ t 0 , but not on t t t .
Proof of Proposition 4.1
We need to show that each purchase of Algorithm 1 is a δ t \delta_t δ t -approximate conditional-gradient step with δ t → 0 \delta_t \to0 δ t → 0 . Throughout, N t N_t N t and C t C_t C t denote the counts and cost of a generic sequence generated by the algorithm, and z t = z N t z_t = z_{N_t} z t = z N t . Writing g k = ⟨ ∇ Φ ρ t ( z t ) , e k / c k ⟩ g_k = \langle\nabla\Phi_{\rho_t}(z_t), e_k/c_k\rangle g k = ⟨ ∇ Φ ρ t ( z t ) , e k / c k ⟩ for the value of the linear approximation at the vertex of logger k k k , the claim is that the purchased logger k ^ \hat{k} k ^ satisfies
g k ^ ≤ min k g k + δ t , δ t → 0. g_{\hat{k}} \le\min_k g_k + \delta_t,\qquad\delta_t \to0. g k ^ ≤ k min g k + δ t , δ t → 0.
The algorithm does not observe g k g_k g k ; it purchases from the logger with the largest estimated index S ^ t , k \widehat{S}_{t,k} S t , k of (15) , so the claim holds if C t 2 S t , k C_t^2 S_{t,k} C t 2 S t , k is close to − g k -g_k − g k for every k k k . The index differs from − g k -g_k − g k in two ways: it uses estimated rather than true quantities, and the finite-count criterion rather than the continuous one. We handle the two separately by passing through the index built from the true finite-count criterion,
ω j , t = exp { C t G j ( N t ) / ρ t } ∑ ℓ = 1 M exp { C t G ℓ ( N t ) / ρ t } , S t , k = 1 c k ∑ j = 1 M ω j , t Δ j , k ( N t ) . (40) \omega_{j,t} = \frac{\exp\{C_tG_j(N_t)/\rho_t\}}{\sum_{\ell=1}^{M}\exp\{C_tG_\ell(N_t)/\rho_t\}},\qquad S_{t,k} = \frac{1}{c_k}\sum_{j=1}^{M}\omega_{j,t}\Delta_{j,k}(N_t). \tag*{(40)} ω j , t = ∑ ℓ = 1 M exp { C t G ℓ ( N t ) / ρ t } exp { C t G j ( N t ) / ρ t } , S t , k = c k 1 j = 1 ∑ M ω j , t Δ j , k ( N t ) . (40)
Every index weights the gains by exponential weights, so both comparisons rely on the following sensitivity bound.
Lemma A3.7. For ρ > 0 \rho> 0 ρ > 0 and y ∈ R M y \in\mathbb{R}^{M} y ∈ R M , let ω ( y ) j = exp ( y j / ρ ) ∑ ℓ exp ( y ℓ / ρ ) \omega(y)_j = \frac{\exp(y_j/\rho)}{\sum_{\ell}\exp(y_\ell/\rho)} ω ( y ) j = ∑ ℓ e x p ( y ℓ / ρ ) e x p ( y j / ρ ) . Then ∥ ω ( y ) − ω ( y ′ ) ∥ 1 ≤ 2 ρ − 1 ∥ y − y ′ ∥ ∞ \|\omega(y)-\omega(y')\|_1 \le2\rho^{-1}\|y-y'\|_\infty ∥ ω ( y ) − ω ( y ′ ) ∥ 1 ≤ 2 ρ − 1 ∥ y − y ′ ∥ ∞ for all y , y ′ ∈ R M y,y' \in\mathbb{R}^{M} y , y ′ ∈ R M .
For the finite-count error, S t , k S_{t,k} S t , k uses G j G_j G j and Δ j , k \Delta_{j,k} Δ j , k where the gradient of Φ ρ t \Phi_{\rho_t} Φ ρ t uses G j G_j G j and ∂ k G j \partial_kG_j ∂ k G j . After rescaling by C C C and C 2 C^2 C 2 , these quantities differ by O ( C − 1 ) O(C^{-1}) O ( C − 1 ) ; combined with Lemma A3.7, this bounds the distance between C t 2 S t , k C_t^2S_{t,k} C t 2 S t , k and − g k -g_k − g k .
Lemma A3.8. Under Assumption 1 , for every 0 < e ≤ 1 / c max 0 < e \le1/c_{\max} 0 < e ≤ 1/ c m a x , uniformly over nonzero N N N with z N ∈ C ( e ) z_N \in\mathcal{C}(e) z N ∈ C ( e ) , targets j j j , and loggers k k k , with C = C N C = C_N C = C N and z = z N z = z_N z = z N ,
∣ C G j ( N ) − G j ( z ) ∣ ≤ K G ( e ) C − 1 , ∣ C 2 Δ j , k ( N ) + ∂ k G j ( z ) ∣ ≤ K Δ ( e ) C − 1 , (41) \left|CG_j(N)-G_j(z)\right| \le K_G(e)C^{-1},\qquad\left|C^2\Delta_{j,k}(N)+\partial_kG_j(z)\right| \le K_\Delta(e)C^{-1}, \tag*{(41)} ∣ C G j ( N ) − G j ( z ) ∣ ≤ K G ( e ) C − 1 , C 2 Δ j , k ( N ) + ∂ k G j ( z ) ≤ K Δ ( e ) C − 1 , (41) where K G ( e ) = R max 2 ( η 0 c max 2 + η e − 2 ) K_G(e)=R_{\max}^2(\eta_0c_{\max}^2+\eta e^{-2}) K G ( e ) = R m a x 2 ( η 0 c m a x 2 + η e − 2 ) and K Δ ( e ) K_\Delta(e) K Δ ( e ) is given at the end of the proof.
Lemma A3.9. Under Assumption 1 , there is a finite constant D 1 D_1 D 1 , depending only on c min c_{\min} c m i n , c max c_{\max} c m a x , R max R_{\max} R m a x , η \eta η , η 0 \eta_0 η 0 , and ρ 0 \rho_0 ρ 0 , such that at every step with z t ∈ C ( e ) z_t \in\mathcal{C}(e) z t ∈ C ( e ) for some 0 < e ≤ 1 / c max 0 < e \le1/c_{\max} 0 < e ≤ 1/ c m a x , uniformly over loggers,
∣ C t 2 S t , k + g k ∣ = ∣ C t 2 S t , k + ⟨ ∇ Φ ρ t ( z t ) , e k c k ⟩ ∣ ≤ D 1 e − 4 ( C t α − 1 + C t − 1 ) . (42) \left|C_t^2S_{t,k}+g_k\right|=\left|C_t^2S_{t,k}+\left\langle\nabla\Phi_{\rho_t}(z_t),\frac{e_k}{c_k}\right\rangle\right|\le D_1e^{-4}\left(C_t^{\alpha-1}+C_t^{-1}\right). \tag*{(42)} C t 2 S t , k + g k = C t 2 S t , k + ⟨ ∇ Φ ρ t ( z t ) , c k e k ⟩ ≤ D 1 e − 4 ( C t α − 1 + C t − 1 ) . (42)
For the estimation error, recall that the estimated index uses G ^ j , t \widehat{G}_{j,t} G j , t and Δ ^ j , k , t \widehat{\Delta}_{j,k,t} Δ j , k , t where S t , k S_{t,k} S t , k uses the true quantities; Assumption 4 controls exactly these differences.
Proposition A3.3. Under Assumptions 1 and 4 , there is a finite constant D 0 D_0 D 0 , depending only on c min c_{\min} c m i n , c max c_{\max} c m a x , and R max R_{\max} R m a x , such that, almost surely for all sufficiently large t t t ,
C t 2 max k ∣ S ^ t , k − S t , k ∣ ≤ D 0 ( r t + r t κ ( θ t ) 2 ρ t ) . (43) C_t^2\max_k\left|\widehat{S}_{t,k}-S_{t,k}\right|\le D_0\left(r_t+\frac{r_t}{\kappa(\theta_t)^2\rho_t}\right). \tag*{(43)} C t 2 k max S t , k − S t , k ≤ D 0 ( r t + κ ( θ t ) 2 ρ t r t ) . (43)
The last lemma combines the two errors.
Lemma A3.10. Suppose Assumption 1 holds, Assumption 4 holds along the sequence, every logger is available at step t t t , and step t t t is not forced. Then the purchase at step t t t is a δ t \delta_t δ t -approximate conditional-gradient step with
δ t ≤ 2 C t 2 max k ∣ S ^ t , k − S t , k ∣ + 2 D 1 c max 4 κ ( θ t ) − 4 ( C t α − 1 + C t − 1 ) , \delta_t \le2C_t^2\max_k|\widehat{S}_{t,k}-S_{t,k}|+2D_1c_{\max}^4\kappa(\theta_t)^{-4}(C_t^{\alpha-1}+C_t^{-1}), δ t ≤ 2 C t 2 k max ∣ S t , k − S t , k ∣ + 2 D 1 c m a x 4 κ ( θ t ) − 4 ( C t α − 1 + C t − 1 ) , where D 1 D_1 D 1 is the constant of Lemma A3.9. If, in addition, κ ( θ t ) ≥ m > 0 \kappa(\theta_t)\ge m>0 κ ( θ t ) ≥ m > 0 and r t ≤ R t − β r r_t\le Rt^{-\beta_r} r t ≤ R t − β r for all large t t t , then δ t ≤ D 2 ( C t α − 1 + C t α − β r ) \delta_t\le D_2(C_t^{\alpha-1}+C_t^{\alpha-\beta_r}) δ t ≤ D 2 ( C t α − 1 + C t α − β r ) for a finite D 2 D_2 D 2 .
We now prove the proposition. For the first claim, we insert the index S t , k S_{t,k} S t , k built from the true criterion. Since κ ( θ t ) ≤ 1 \kappa(\theta_t)\le1 κ ( θ t ) ≤ 1 , Proposition A3.3 gives, once C t ≥ 1 C_t\ge1 C t ≥ 1 ,
C t 2 max k ∣ S ^ t , k − S t , k ∣ ≤ D 0 ( r t + ρ 0 − 1 κ ( θ t ) − 2 C t α r t ) ≤ D 0 ( 1 + ρ 0 − 1 ) κ ( θ t ) − 2 C t α r t . C_t^2\max_k|\widehat{S}_{t,k}-S_{t,k}|\le D_0(r_t+\rho_0^{-1}\kappa(\theta_t)^{-2}C_t^\alpha r_t)\le D_0(1+\rho_0^{-1})\kappa(\theta_t)^{-2}C_t^\alpha r_t. C t 2 k max ∣ S t , k − S t , k ∣ ≤ D 0 ( r t + ρ 0 − 1 κ ( θ t ) − 2 C t α r t ) ≤ D 0 ( 1 + ρ 0 − 1 ) κ ( θ t ) − 2 C t α r t .
while Lemma A3.9, applied with e = κ ( θ t ) / c max e=\kappa(\theta_t)/c_{\max} e = κ ( θ t ) / c m a x , gives
max k ∣ C t 2 S t , k + g k ∣ ≤ 2 D 1 c max 4 κ ( θ t ) − 4 C t α − 1 . \max_k|C_t^2S_{t,k}+g_k|\le2D_1c_{\max}^4\kappa(\theta_t)^{-4}C_t^{\alpha-1}. k max ∣ C t 2 S t , k + g k ∣ ≤ 2 D 1 c m a x 4 κ ( θ t ) − 4 C t α − 1 .
The triangle inequality yields {(4.4)} with D = max { D 0 ( 1 + ρ 0 − 1 ) , 2 D 1 c max 4 } D=\max\{D_0(1+\rho_0^{-1}),2D_1c_{\max}^4\} D = max { D 0 ( 1 + ρ 0 − 1 ) , 2 D 1 c m a x 4 } . The second claim then follows by the argument in the proof of Lemma A3.10, applied with this bound on the index error.
Sufficient Conditions for Assumption 4
We now give sufficient conditions for Assumption 4. They concern the estimated quantities of Section 3.2, which are defined as follows. Let q ^ t \widehat{q}_t q t and σ ^ t 2 \widehat{\sigma}_t^2 σ t 2 be estimates of the conditional reward mean and variance from the first t t t observations, truncated to [ − R max , R max ] [-R_{\max},R_{\max}] [ − R m a x , R m a x ] and [ 0 , R max 2 ] [0,R_{\max}^2] [ 0 , R m a x 2 ] , respectively, and let P ^ t f = t − 1 ∑ i = 1 t f ( X i ) \widehat{P}_t f=t^{-1}\sum_{i=1}^t f(X_i) P t f = t − 1 ∑ i = 1 t f ( X i ) be the empirical distribution of the observed contexts. For each target policy π j \pi_j π j , define
q ^ j , t ( x ) = ∑ a ∈ A π j ( a ∣ x ) q ^ t ( x , a ) , \widehat{q}_{j,t}(x)=\sum_{a\in\mathcal{A}}\pi_j(a\mid x)\widehat{q}_t(x,a), q j , t ( x ) = a ∈ A ∑ π j ( a ∣ x ) q t ( x , a ) ,
W ^ j , t ( x , a ) = π j ( a ∣ x ) 2 σ ^ t 2 ( x , a ) , \widehat{W}_{j,t}(x,a)=\pi_j(a\mid x)^2\widehat{\sigma}_t^2(x,a), W j , t ( x , a ) = π j ( a ∣ x ) 2 σ t 2 ( x , a ) ,
V ^ j , t = P ^ t { q ^ j , t 2 } − { P ^ t q ^ j , t } 2 . (44) \widehat{V}_{j,t}=\widehat{P}_t\{\widehat{q}_{j,t}^2\}-\{\widehat{P}_t\widehat{q}_{j,t}\}^2. \tag*{(44)} V j , t = P t { q j , t 2 } − { P t q j , t } 2 . (44)
Substituting these estimates into {(3.1)} and {(3.8)} gives
G ^ j , t = V ^ j , t t + η 0 + P ^ t ∑ a ∈ A W ^ j , t ( ⋅ , a ) D N t ( ⋅ , a ) . (45) \widehat{G}_{j,t}=\frac{\widehat{V}_{j,t}}{t+\eta_0}+\widehat{P}_t\sum_{a\in\mathcal{A}}\frac{\widehat{W}_{j,t}(\cdot,a)}{D_{N_t}(\cdot,a)}. \tag*{(45)} G j , t = t + η 0 V j , t + P t a ∈ A ∑ D N t ( ⋅ , a ) W j , t ( ⋅ , a ) . (45)
and
Δ ^ j , k , t = P ^ t ∑ a ∈ A W ^ j , t ( ⋅ , a ) μ k ( a ∣ ⋅ ) D N t ( ⋅ , a ) { D N t ( ⋅ , a ) + μ k ( a ∣ ⋅ ) } + V ^ j , t ( t + η 0 ) ( t + 1 + η 0 ) . (46) \widehat{\Delta}_{j,k,t}=\widehat{P}_t\sum_{a\in\mathcal{A}}\frac{\widehat{W}_{j,t}(\cdot,a)\mu_k(a\mid\cdot)}{D_{N_t}(\cdot,a)\{D_{N_t}(\cdot,a)+\mu_k(a\mid\cdot)\}}+\frac{\widehat{V}_{j,t}}{(t+\eta_0)(t+1+\eta_0)}. \tag*{(46)} Δ j , k , t = P t a ∈ A ∑ D N t ( ⋅ , a ) { D N t ( ⋅ , a ) + μ k ( a ∣ ⋅ )} W j , t ( ⋅ , a ) μ k ( a ∣ ⋅ ) + ( t + η 0 ) ( t + 1 + η 0 ) V j , t . (46)
Since G j G_j G j and Δ j , k \Delta_{j,k} Δ j , k are linear in V j V_j V j and W j W_j W j , the rate r t r_t r t is inherited from the estimates of these moments and from the replacement of P X P_X P X by P ^ t \widehat{P}_t P t . In the next lemma, the first two conditions are rates for the estimated reward moments. The last two control the replacement of P X P_X P X by the empirical distribution of the purchased contexts, which are independent draws from P X P_X P X because each logger is chosen before its context is observed. Conditions of this type are standard in semiparametric estimation (Chernozhukov et al., [7 ] ). Both results below are proved in Appendix A6.
Lemma A3.11. *Under Assumption 1, suppose there are deterministic sequences r V , t , r W , t , r P , t → r_{V,t},r_{W,t},r_{P,t}\to r V , t , r W , t , r P , t →
0 0 0 and m t ∈ ( 0 , 1 ] m_t \in(0,1] m t ∈ ( 0 , 1 ] such that, almost surely for all sufficiently large t t t , κ ( θ t ) ≥ m t \kappa(\theta_t) \ge m_t κ ( θ t ) ≥ m t ,
max j ∣ V ^ j , t − V j ∣ ≤ r V , t , max j ∑ a ∈ A ∥ W ^ j , t ( ⋅ , a ) − W j ( ⋅ , a ) ∥ L 1 ( P X ) ≤ r W , t , \max_j |\widehat{V}_{j,t}-V_j| \le r_{V,t}, \qquad\max_j \sum_{a\in\mathcal{A}} \|\widehat{W}_{j,t}(\cdot,a)-W_j(\cdot,a)\|_{L_1(P_X)} \le r_{W,t}, j max ∣ V j , t − V j ∣ ≤ r V , t , j max a ∈ A ∑ ∥ W j , t ( ⋅ , a ) − W j ( ⋅ , a ) ∥ L 1 ( P X ) ≤ r W , t ,
and, uniformly over j j j and k k k ,
∣ t ( P ^ t − P X ) ∑ a W ^ j , t ( ⋅ , a ) D N t ( ⋅ , a ) ∣ ≤ r P , t , \left|t(\widehat{P}_t-P_X)\sum_a\frac{\widehat{W}_{j,t}(\cdot,a)}{D_{N_t}(\cdot,a)}\right|\le r_{P,t}, t ( P t − P X ) a ∑ D N t ( ⋅ , a ) W j , t ( ⋅ , a ) ≤ r P , t ,
∣ t 2 ( P ^ t − P X ) ∑ a W ^ j , t ( ⋅ , a ) μ k ( a ∣ ⋅ ) D N t ( ⋅ , a ) { D N t ( ⋅ , a ) + μ k ( a ∣ ⋅ ) } ∣ ≤ r P , t . \left|t^2(\widehat{P}_t-P_X)\sum_a\frac{\widehat{W}_{j,t}(\cdot,a)\mu_k(a\mid\cdot)}{D_{N_t}(\cdot,a)\{D_{N_t}(\cdot,a)+\mu_k(a\mid\cdot)\}}\right|\le r_{P,t}. t 2 ( P t − P X ) a ∑ D N t ( ⋅ , a ) { D N t ( ⋅ , a ) + μ k ( a ∣ ⋅ )} W j , t ( ⋅ , a ) μ k ( a ∣ ⋅ ) ≤ r P , t .
Let
a t = r V , t + r P , t + m t − 2 r W , t . a_t=r_{V,t}+r_{P,t}+m_t^{-2}r_{W,t}. a t = r V , t + r P , t + m t − 2 r W , t .
If a t → 0 a_t\to0 a t → 0 , then Assumption 4 holds with r t = K ′ sup s ≥ t a s r_t=K'\sup_{s\ge t}a_s r t = K ′ sup s ≥ t a s , where K ′ = max { c max , c max 2 } K'=\max\{c_{\max},c_{\max}^2\} K ′ = max { c m a x , c m a x 2 } .
When the context space is finite, these conditions can be checked directly.
Corollary A3.1. Suppose X \mathcal{X} X is finite, q ^ t ( x , a ) \widehat{q}_t(x,a) q t ( x , a ) and σ ^ t 2 ( x , a ) \widehat{\sigma}_t^2(x,a) σ t 2 ( x , a ) are the sample mean and sample variance of the rewards observed at ( x , a ) (x,a) ( x , a ) , and P ^ t \widehat{P}_t P t is the empirical distribution of X 1 , … , X t X_1,\ldots,X_t X 1 , … , X t . Under Assumptions 1 and 2, with the forced-sampling constants of Section § 3.3 and ν < 1 / 6 \nu<1/6 ν < 1/6 , Assumption 4 holds along the sequence generated by Algorithm 1, while every logger is affordable, with r t = K ′ t 3 ν log t / t r_t=K't^{3\nu}\sqrt{\log t/t} r t = K ′ t 3 ν log t / t , and, if θ t → θ ⋆ \theta_t\to\theta^\star θ t → θ ⋆ , also with r t = K ′ log t / t r_t=K'\sqrt{\log t/t} r t = K ′ log t / t , for a finite constant K ′ K' K ′ .
With the first of these rates, the condition t α + 2 ν r t → 0 t^{\alpha+2\nu}r_t\to0 t α + 2 ν r t → 0 of Theorem Theorem 4.1 holds when α + 5 ν < 1 / 2 \alpha+5\nu<1/2 α + 5 ν < 1/2 , which is satisfied by the values α = 0.4 \alpha=0.4 α = 0.4 and ν = 0.01 \nu=0.01 ν = 0.01 used in Section § 5 .
Proofs for Section 4.2
This appendix proves Theorem Theorem 4.2 and Corollary Corollary 4.1. We compare J ^ B \widehat{J}_B J B with two averages that cannot be computed. Replacing q ^ B \widehat{q}_B q B by q q q in (17) gives J ~ B = T B − 1 ∑ i ≤ T B ψ i ( θ ^ B ) \widetilde{J}_B=T_B^{-1}\sum_{i\le T_B}\psi_i(\widehat{\theta}_B) J B = T B − 1 ∑ i ≤ T B ψ i ( θ B ) , and also replacing θ ^ B \widehat{\theta}_B θ B by θ ⋆ \theta^\star θ ⋆ gives J ~ B ⋆ = T B − 1 ∑ i ≤ T B ψ i ( θ ⋆ ) \widetilde{J}_B^\star=T_B^{-1}\sum_{i\le T_B}\psi_i(\theta^\star) J B ⋆ = T B − 1 ∑ i ≤ T B ψ i ( θ ⋆ ) . The error then splits as
J ^ B − J = ( J ^ B − J ~ B ) + ( J ~ B − J ~ B ⋆ ) + ( J ~ B ⋆ − J ) . \widehat{J}_B-J=(\widehat{J}_B-\widetilde{J}_B)+(\widetilde{J}_B-\widetilde{J}_B^\star)+(\widetilde{J}_B^\star-J). J B − J = ( J B − J B ) + ( J B − J B ⋆ ) + ( J B ⋆ − J ) .
The last term is an average of martingale differences, and the middle term reflects the random denominator. Theorem Theorem A4.1 below shows that the former is asymptotically normal and the latter is o p ( T B − 1 / 2 ) o_p(T_B^{-1/2}) o p ( T B − 1/2 ) . The first term is the regression remainder R B ( θ ^ B ) R_B(\widehat{\theta}_B) R B ( θ B ) , which Assumption 5 controls. Beyond Assumptions 1 and 2, Theorem Theorem A4.1 uses only predictable logger selection and convergence of the final composition, so it also applies to other purchasing rules with these properties.
Throughout, for each budget B B B , ( F i ) (\mathcal{F}_i) ( F i ) is the filtration of Section § 2.1 for the sequence with that budget, extended by F i = F T B \mathcal{F}_i=\mathcal{F}_{T_B} F i = F T B for i > T B i>T_B i > T B . Since each purchase costs at least c min c_{\min} c m i n and the total cost is at most B B B , we have T B ≤ n B T_B\le n_B T B ≤ n B , where n B = ⌈ B / c min ⌉ n_B=\lceil B/c_{\min}\rceil n B = ⌈ B / c m i n ⌉ . The event { i ≤ T B } \{i\le T_B\} { i ≤ T B } equals { B − C i − 1 ≥ c min } \{B-C_{i-1}\ge c_{\min}\} { B − C i − 1 ≥ c m i n } , so it is F i − 1 \mathcal{F}_{i-1} F i − 1 -measurable.
Proof of Theorem 4.2
The key step is the following result, which we prove in Appendix Appendix § 10.2 .
Theorem A4.1. Under Assumptions 1 and 2, suppose θ ^ B → θ ⋆ \widehat{\theta}_B\to\theta^\star θ B → θ ⋆ in probability. Then
(i) T B ( J ~ B ⋆ − J ) → N { 0 , Σ A ( θ ⋆ ) } \sqrt{T_B}(\widetilde{J}_B^\star-J)\to N\{0,\Sigma_A(\theta^\star)\} T B ( J B ⋆ − J ) → N { 0 , Σ A ( θ ⋆ )} in distribution;
(ii) writing J ~ B = T B − 1 ∑ i ≤ T B ψ i ( θ ^ B ) \widetilde{J}_B=T_B^{-1}\sum_{i\leq T_B}\psi_i(\widehat{\theta}_B) J B = T B − 1 ∑ i ≤ T B ψ i ( θ B ) ,
T B ( J ~ B − J ~ B ) → 0 (47) \sqrt{T_B}\left(\widetilde{J}_B-\widetilde{J}_B\right)\to0 \tag*{(47)} T B ( J B − J B ) → 0 (47)
in probability, and consequently T B ( J ~ B − J ) → N { 0 , Σ A ( θ ∗ ) } \sqrt{T_B}(\widetilde{J}_B-J)\to N\{0,\Sigma_A(\theta^*)\} T B ( J B − J ) → N { 0 , Σ A ( θ ∗ )} in distribution.
Under the conditions of part (i) of Theorem 4.1, θ ^ B → θ ∗ \widehat{\theta}_B\to\theta^* θ B → θ ∗ almost surely. Theorem A4.1 therefore applies and gives T B ( J ~ B − J ) → N { 0 , Σ A ( θ ∗ ) } \sqrt{T_B}(\widetilde{J}_B-J)\to N\{0,\Sigma_A(\theta^*)\} T B ( J B − J ) → N { 0 , Σ A ( θ ∗ )} , and it remains to show that the estimated regression does not change this limit. By definition, the j j j th coordinate of J ^ B − J ~ B \widehat{J}_B-\widetilde{J}_B J B − J B is R B , j ( θ ^ B ) \mathcal{R}_{B,j}(\widehat{\theta}_B) R B , j ( θ B ) . On the event { ∥ θ ^ B − θ ∗ ∥ 2 ≤ r B } \{\lVert\widehat{\theta}_B-\theta^*\rVert_2\leq r_B\} {∥ θ B − θ ∗ ∥ 2 ≤ r B } , whose probability tends to one by Assumption 5,
max j ∣ R B , j ( θ ^ B ) ∣ ≤ max j sup ∥ θ − θ ∗ ∥ 2 ≤ r B ∣ R B , j ( θ ) ∣ = o p ( T B − 1 / 2 ) . \max_j|\mathcal{R}_{B,j}(\widehat{\theta}_B)|\leq\max_j\sup_{\lVert\theta-\theta^*\rVert_2\leq r_B}|\mathcal{R}_{B,j}(\theta)|=o_p(T_B^{-1/2}). j max ∣ R B , j ( θ B ) ∣ ≤ j max ∥ θ − θ ∗ ∥ 2 ≤ r B sup ∣ R B , j ( θ ) ∣ = o p ( T B − 1/2 ) .
Hence T B ( J ^ B − J ~ B ) → 0 \sqrt{T_B}(\widehat{J}_B-\widetilde{J}_B)\to0 T B ( J B − J B ) → 0 in probability, and Slutsky’s theorem gives (27) .
Proof of Theorem A4.1
We need two lemmas. The first describes the scores at a fixed composition.
Lemma A4.1. Let θ ∈ S K \theta\in S_K θ ∈ S K be deterministic with κ ( θ ) > 0 \kappa(\theta)>0 κ ( θ ) > 0 . Under Assumption 1 , { ψ i ( θ ) − J } i ≤ T B \{\psi_i(\theta)-J\}_{i\leq T_B} { ψ i ( θ ) − J } i ≤ T B is a bounded martingale difference sequence with respect to ( F i ) (\mathcal{F}_i) ( F i ) , and
E [ { ψ i ( θ ) − J } { ψ i ( θ ) − J } T ∣ F i − 1 ] = Cov { q Π ( X ) } + E X ∑ a μ K i ( a ∣ X ) σ 2 ( X , a ) π ( a ∣ X ) π ( a ∣ X ) T m θ ( a ∣ X ) 2 . (48) \mathbb{E}\left[\{\psi_i(\theta)-J\}\{\psi_i(\theta)-J\}^{\mathsf{T}}\mid\mathcal{F}_{i-1}\right]=\operatorname{Cov}\{q_\Pi(X)\}+\mathbb{E}_X\sum_a\mu_{K_i}(a\mid X)\frac{\sigma^2(X,a)\pi(a\mid X)\pi(a\mid X)^{\mathsf{T}}}{m_\theta(a\mid X)^2}. \tag*{(48)} E [ { ψ i ( θ ) − J } { ψ i ( θ ) − J } T ∣ F i − 1 ] = Cov { q Π ( X )} + E X a ∑ μ K i ( a ∣ X ) m θ ( a ∣ X ) 2 σ 2 ( X , a ) π ( a ∣ X ) π ( a ∣ X ) T . (48)
To prove Lemma A4.1, write ϵ i = R i − q ( X i , A i ) \epsilon_i=R_i-q(X_i,A_i) ϵ i = R i − q ( X i , A i ) and w i = π ( A i ∣ X i ) / m θ ( A i ∣ X i ) w_i=\pi(A_i\mid X_i)/m_\theta(A_i\mid X_i) w i = π ( A i ∣ X i ) / m θ ( A i ∣ X i ) , with w i = 0 w_i=0 w i = 0 when π ( A i ∣ X i ) = 0 \pi(A_i\mid X_i)=0 π ( A i ∣ X i ) = 0 , so that
ψ i ( θ ) − J = { q Π ( X i ) − J } + w i ϵ i . \psi_i(\theta)-J=\{q_\Pi(X_i)-J\}+w_i\epsilon_i. ψ i ( θ ) − J = { q Π ( X i ) − J } + w i ϵ i .
Given F i − 1 \mathcal{F}_{i-1} F i − 1 , the logger K i K_i K i is fixed, X i ∼ P X X_i\sim P_X X i ∼ P X , A i ∣ X i ∼ μ K i ( ⋅ ∣ X i ) A_i\mid X_i\sim\mu_{K_i}(\cdot\mid X_i) A i ∣ X i ∼ μ K i ( ⋅ ∣ X i ) , and R i ∣ X i , A i ∼ P R ( ⋅ ∣ X i , A i ) R_i\mid X_i,A_i\sim P_R(\cdot\mid X_i,A_i) R i ∣ X i , A i ∼ P R ( ⋅ ∣ X i , A i ) . Thus
E ( ϵ i ∣ F i − 1 , X i , A i ) = 0 , E ( ϵ i 2 ∣ F i − 1 , X i , A i ) = σ 2 ( X i , A i ) , \mathbb{E}(\epsilon_i\mid\mathcal{F}_{i-1},X_i,A_i)=0,\qquad\mathbb{E}(\epsilon_i^2\mid\mathcal{F}_{i-1},X_i,A_i)=\sigma^2(X_i,A_i), E ( ϵ i ∣ F i − 1 , X i , A i ) = 0 , E ( ϵ i 2 ∣ F i − 1 , X i , A i ) = σ 2 ( X i , A i ) ,
and, because θ \theta θ is deterministic, w i w_i w i depends on ( X i , A i ) (X_i,A_i) ( X i , A i ) alone. Conditioning first on ( F i − 1 , X i , A i ) (\mathcal{F}_{i-1},X_i,A_i) ( F i − 1 , X i , A i ) therefore gives E ( w i ϵ i ∣ F i − 1 ) = 0 \mathbb{E}(w_i\epsilon_i\mid\mathcal{F}_{i-1})=0 E ( w i ϵ i ∣ F i − 1 ) = 0 , while E { q Π ( X i ) ∣ F i − 1 } = E X { q Π ( X ) } = J \mathbb{E}\{q_\Pi(X_i)\mid\mathcal{F}_{i-1}\}=\mathbb{E}_X\{q_\Pi(X)\}=J E { q Π ( X i ) ∣ F i − 1 } = E X { q Π ( X )} = J ; so the scores have conditional mean J J J . They are also bounded: since m θ ≥ κ ( θ ) m_\theta\geq\kappa(\theta) m θ ≥ κ ( θ ) whenever π ( A i ∣ X i ) ≠ 0 \pi(A_i\mid X_i)\neq0 π ( A i ∣ X i ) = 0 ,
∥ q Π ( X i ) − J ∥ 2 ≤ 2 M R max , ∣ ϵ i ∣ ≤ 2 R max , ∥ w i ∥ 2 ≤ M κ ( θ ) . \lVert q_\Pi(X_i)-J\rVert_2\leq2\sqrt{M}R_{\max},\qquad|\epsilon_i|\leq2R_{\max},\qquad\lVert w_i\rVert_2\leq\frac{\sqrt{M}}{\kappa(\theta)}. ∥ q Π ( X i ) − J ∥ 2 ≤ 2 M R m a x , ∣ ϵ i ∣ ≤ 2 R m a x , ∥ w i ∥ 2 ≤ κ ( θ ) M .
For the conditional covariance, the same conditioning shows that the cross term between q Π ( X i ) − J q_\Pi(X_i)-J q Π ( X i ) − J and w i ϵ i w_i\epsilon_i w i ϵ i has conditional mean zero. The first term contributes Cov { q Π ( X ) } \operatorname{Cov}\{q_\Pi(X)\} Cov { q Π ( X )} , and the second contributes
E ( w i w i T ϵ i 2 ∣ F i − 1 ) = E { w i w i T σ 2 ( X i , A i ) ∣ F i − 1 } = E X ∑ a μ K i ( a ∣ X ) σ 2 ( X , a ) π ( a ∣ X ) π ( a ∣ X ) T m θ ( a ∣ X ) 2 , \mathbb{E}(w_iw_i^{\mathsf{T}}\epsilon_i^2\mid\mathcal{F}_{i-1})=\mathbb{E}\{w_iw_i^{\mathsf{T}}\sigma^2(X_i,A_i)\mid\mathcal{F}_{i-1}\}=\mathbb{E}_X\sum_a\mu_{K_i}(a\mid X)\frac{\sigma^2(X,a)\pi(a\mid X)\pi(a\mid X)^{\mathsf{T}}}{m_\theta(a\mid X)^2}, E ( w i w i T ϵ i 2 ∣ F i − 1 ) = E { w i w i T σ 2 ( X i , A i ) ∣ F i − 1 } = E X a ∑ μ K i ( a ∣ X ) m θ ( a ∣ X ) 2 σ 2 ( X , a ) π ( a ∣ X ) π ( a ∣ X ) T ,
where the last step averages over A i ∼ μ K i ( ⋅ ∣ X i ) A_i\sim\mu_{K_i}(\cdot\mid X_i) A i ∼ μ K i ( ⋅ ∣ X i ) and then over X i ∼ P X X_i\sim P_X X i ∼ P X . This gives (A4.2).
The second lemma relates the number of purchases to the budget.
Lemma A4.2. If θ ^ B → θ ∗ \widehat{\theta}_B\to\theta^* θ B → θ ∗ in probability, then T B / B → 1 / c T θ ∗ T_B/B\to1/c^{\mathsf{T}}\theta^* T B / B → 1/ c T θ ∗ in probability.
Indeed, at termination,
0 ≤ B − C T B < c min , C T B T B = c ⊤ θ ^ B → c ⊤ θ ⋆ in probability , 0 \le B-C_{T_B}<c_{\min}, \qquad\frac{C_{T_B}}{T_B}=c^\top\widehat{\theta}_B\to c^\top\theta^\star\text{ in probability}, 0 ≤ B − C T B < c m i n , T B C T B = c ⊤ θ B → c ⊤ θ ⋆ in probability ,
so that C T B / B → 1 C_{T_B}/B\to1 C T B / B → 1 , and the ratio of the two is T B / B T_B/B T B / B .
We now prove part (i) of Theorem A4.1. Since J ~ B ⋆ \widetilde{J}_{B}^{\star} J B ⋆ averages a data-dependent number T B T_B T B of terms, we work with a sum over the fixed range i ≤ n B i\le n_B i ≤ n B , normalized by B 1 / 2 B^{1/2} B 1/2 , and convert to the normalization by T B T_B T B at the end. For i ≤ n B i\le n_B i ≤ n B let
ξ B , i = 1 { i ≤ T B } { ψ i ( θ ⋆ ) − J } , \xi_{B,i}=\mathbf{1}\{i\le T_B\}\{\psi_i(\theta^\star)-J\}, ξ B , i = 1 { i ≤ T B } { ψ i ( θ ⋆ ) − J } ,
so that ∑ i ≤ n B ξ B , i = T B ( J ~ B ⋆ − J ) \sum_{i\le n_B}\xi_{B,i}=T_B(\widetilde{J}_{B}^{\star}-J) ∑ i ≤ n B ξ B , i = T B ( J B ⋆ − J ) . Because { i ≤ T B } \{i\le T_B\} { i ≤ T B } is F i − 1 \mathcal{F}_{i-1} F i − 1 -measurable, Lemma A4.1 shows that ( ξ B , i ) i ≤ n B (\xi_{B,i})_{i\le n_B} ( ξ B , i ) i ≤ n B is a bounded martingale difference array. By the Cramér–Wold device, it suffices to show that, for each fixed u ∈ R M u\in\mathbb{R}^M u ∈ R M ,
S B ( u ) = ∑ i = 1 n B Z B , i , Z B , i = B − 1 / 2 u ⊤ ξ B , i , S_B(u)=\sum_{i=1}^{n_B}Z_{B,i},\qquad Z_{B,i}=B^{-1/2}u^\top\xi_{B,i}, S B ( u ) = i = 1 ∑ n B Z B , i , Z B , i = B − 1/2 u ⊤ ξ B , i ,
converges in distribution to N ( 0 , s u 2 ) N(0,s_u^2) N ( 0 , s u 2 ) , where s u 2 = u ⊤ Σ A ( θ ⋆ ) u / c ⊤ θ ⋆ s_u^2=u^\top\Sigma_A(\theta^\star)u/c^\top\theta^\star s u 2 = u ⊤ Σ A ( θ ⋆ ) u / c ⊤ θ ⋆ .
For convenience, we recall the martingale central limit theorem in the form we use ([20 Corollary 3.1] ). For each n n n , let ( X n , i , H n , i ) i ≤ k n (X_{n,i},\mathcal{H}_{n,i})_{i\le k_n} ( X n , i , H n , i ) i ≤ k n be a square-integrable martingale difference sequence, and let η 2 > 0 \eta^2>0 η 2 > 0 be a constant. If, for every ϵ > 0 \epsilon>0 ϵ > 0 ,
∑ i = 1 k n E ( X n , i 2 ∣ H n , i − 1 ) → η 2 and ∑ i = 1 k n E ( X n , i 2 1 { ∣ X n , i ∣ > ϵ } ∣ H n , i − 1 ) → 0 \sum_{i=1}^{k_n}\mathbb{E}(X_{n,i}^2\mid\mathcal{H}_{n,i-1})\to\eta^2
\qquad\text{and}\qquad
\sum_{i=1}^{k_n}\mathbb{E}(X_{n,i}^2\mathbf{1}\{|X_{n,i}|>\epsilon\}\mid\mathcal{H}_{n,i-1})\to0 i = 1 ∑ k n E ( X n , i 2 ∣ H n , i − 1 ) → η 2 and i = 1 ∑ k n E ( X n , i 2 1 { ∣ X n , i ∣ > ϵ } ∣ H n , i − 1 ) → 0
in probability, then ∑ i ≤ k n X n , i → N ( 0 , η 2 ) \sum_{i\le k_n}X_{n,i}\to N(0,\eta^2) ∑ i ≤ k n X n , i → N ( 0 , η 2 ) in distribution. Because η 2 \eta^2 η 2 is constant, no nesting condition on the σ \sigma σ -fields H n , i \mathcal{H}_{n,i} H n , i is required.
Suppose first that s u 2 > 0 s_u^2>0 s u 2 > 0 , and take X B , i = Z B , i X_{B,i}=Z_{B,i} X B , i = Z B , i , k B = n B k_B=n_B k B = n B , and η 2 = s u 2 \eta^2=s_u^2 η 2 = s u 2 . The scores are bounded, so ∣ Z B , i ∣ ≤ c Z ∥ u ∥ 2 B − 1 / 2 |Z_{B,i}|\le c_Z\|u\|_2B^{-1/2} ∣ Z B , i ∣ ≤ c Z ∥ u ∥ 2 B − 1/2 for a finite c Z c_Z c Z , and the indicators in the second condition vanish once B B B is large. For the first condition, (A4.2) at θ = θ ⋆ \theta=\theta^\star θ = θ ⋆ , together with the fact that the average of μ K i \mu_{K_i} μ K i over i ≤ T B i\le T_B i ≤ T B is m θ ^ B m_{\widehat{\theta}_B} m θ B , gives
V B = ∑ i = 1 n B E ( Z B , i 2 ∣ F i − 1 ) = T B B u ⊤ Γ ( θ ^ B ) u , V_B=\sum_{i=1}^{n_B}\mathbb{E}(Z_{B,i}^2\mid\mathcal{F}_{i-1})=\frac{T_B}{B}u^\top\Gamma(\widehat{\theta}_B)u, V B = i = 1 ∑ n B E ( Z B , i 2 ∣ F i − 1 ) = B T B u ⊤ Γ ( θ B ) u ,
Γ ( θ ) = Cov { q Π ( X ) } + E X ∑ a m θ ( a ∣ X ) σ 2 ( X , a ) π ( a ∣ X ) π ( a ∣ X ) ⊤ m θ ⋆ ( a ∣ X ) 2 ( θ ∈ S K ) . \Gamma(\theta)=\operatorname{Cov}\{q_{\Pi}(X)\}+\mathbb{E}_X\sum_a m_\theta(a\mid X)\frac{\sigma^2(X,a)\pi(a\mid X)\pi(a\mid X)^\top}{m_{\theta^\star}(a\mid X)^2}\qquad(\theta\in S_K). Γ ( θ ) = Cov { q Π ( X )} + E X a ∑ m θ ( a ∣ X ) m θ ⋆ ( a ∣ X ) 2 σ 2 ( X , a ) π ( a ∣ X ) π ( a ∣ X ) ⊤ ( θ ∈ S K ) .
Here Γ ( θ ⋆ ) = Σ A ( θ ⋆ ) \Gamma(\theta^\star)=\Sigma_A(\theta^\star) Γ ( θ ⋆ ) = Σ A ( θ ⋆ ) by (24) . Moreover, m θ ⋆ ≥ κ ⋆ m_{\theta^\star}\ge\kappa_\star m θ ⋆ ≥ κ ⋆ on contributing actions, σ 2 ≤ R max 2 \sigma^2\le R_{\max}^2 σ 2 ≤ R m a x 2 , and ∣ π j π ℓ ∣ ≤ 1 |\pi_j\pi_\ell|\le1 ∣ π j π ℓ ∣ ≤ 1 , while
∑ a ∣ m θ ( a ∣ x ) − m θ ′ ( a ∣ x ) ∣ ≤ ∥ θ − θ ′ ∥ 1 \sum_a|m_\theta(a\mid x)-m_{\theta'}(a\mid x)|\le\|\theta-\theta'\|_1 a ∑ ∣ m θ ( a ∣ x ) − m θ ′ ( a ∣ x ) ∣ ≤ ∥ θ − θ ′ ∥ 1
because each μ k ( ⋅ ∣ x ) \mu_k(\cdot\mid x) μ k ( ⋅ ∣ x ) sums to one. Every entry of Γ \Gamma Γ therefore satisfies
∣ Γ ( θ ) j ℓ − Γ ( θ ′ ) j ℓ ∣ ≤ R max 2 κ ⋆ − 2 ∥ θ − θ ′ ∥ 1 . |\Gamma(\theta)_{j\ell}-\Gamma(\theta')_{j\ell}|\le R_{\max}^2\kappa_\star^{-2}\|\theta-\theta'\|_1. ∣Γ ( θ ) j ℓ − Γ ( θ ′ ) j ℓ ∣ ≤ R m a x 2 κ ⋆ − 2 ∥ θ − θ ′ ∥ 1 .
Since θ ^ B → θ ⋆ \widehat{\theta}_B\to\theta^\star θ B → θ ⋆ in probability, u ⊤ Γ ( θ ^ B ) u → u ⊤ Σ A ( θ ⋆ ) u u^\top\Gamma(\widehat{\theta}_B)u\to u^\top\Sigma_A(\theta^\star)u u ⊤ Γ ( θ B ) u → u ⊤ Σ A ( θ ⋆ ) u , and Lemma A4.2 gives V B → s u 2 V_B\to s_u^2 V B → s u 2 , both in probability. The martingale central limit theorem now gives S B ( u ) → N ( 0 , s u 2 ) S_B(u)\to N(0,s_u^2) S B ( u ) → N ( 0 , s u 2 ) .
If s u 2 = 0 s_u^2=0 s u 2 = 0 , then V B → 0 V_B\to0 V B → 0 in probability. Moreover, T B ≤ B / c min T_B\le B/c_{\min} T B ≤ B / c m i n , and every entry of Γ ( θ ) \Gamma(\theta) Γ ( θ ) is at most R max 2 ( 1 + κ ⋆ − 2 ) R_{\max}^2(1+\kappa_\star^{-2}) R m a x 2 ( 1 + κ ⋆ − 2 ) in absolute value. Hence
V B ≤ c min − 1 R max 2 ( 1 + κ ⋆ − 2 ) ∥ u ∥ 1 2 , V_B\le c_{\min}^{-1}R_{\max}^2(1+\kappa_\star^{-2})\|u\|_1^2, V B ≤ c m i n − 1 R m a x 2 ( 1 + κ ⋆ − 2 ) ∥ u ∥ 1 2 ,
and bounded convergence gives E ( V B ) → 0 \mathbb{E}(V_B) \to0 E ( V B ) → 0 . The cross products of the martingale differences Z B , i Z_{B,i} Z B , i have mean zero, so
E { S B ( u ) 2 } = ∑ i = 1 n B E ( Z B , i 2 ) = E ( V B ) → 0 , \mathbb{E}\{S_B(u)^2\} = \sum_{i=1}^{n_B}\mathbb{E}(Z_{B,i}^2) = \mathbb{E}(V_B) \to0, E { S B ( u ) 2 } = i = 1 ∑ n B E ( Z B , i 2 ) = E ( V B ) → 0 ,
and Chebyshev’s inequality gives S B ( u ) → 0 = N ( 0 , 0 ) S_B(u) \to0 = N(0,0) S B ( u ) → 0 = N ( 0 , 0 ) in probability. Combining over u u u , B − 1 / 2 T B ( J ~ B − J ) → N { 0 , Σ A ( θ ⋆ ) / c θ ⋆ ⊤ θ ⋆ } B^{-1/2}T_B(\widetilde{J}_B-J) \to N\{0,\Sigma_A(\theta^\star)/c_{\theta^\star}^{\top}\theta^\star\} B − 1/2 T B ( J B − J ) → N { 0 , Σ A ( θ ⋆ ) / c θ ⋆ ⊤ θ ⋆ } in distribution. Multiplying by B / T B \sqrt{B/T_B} B / T B , which converges to c θ ⋆ ⊤ θ ⋆ \sqrt{c_{\theta^\star}^{\top}\theta^\star} c θ ⋆ ⊤ θ ⋆ by Lemma A4.2, and applying Slutsky’s theorem gives part (i).
For part (ii), the score differences ψ i ( θ ) − ψ i ( θ ⋆ ) \psi_i(\theta)-\psi_i(\theta^\star) ψ i ( θ ) − ψ i ( θ ⋆ ) must be small uniformly over θ \theta θ near θ ⋆ \theta^\star θ ⋆ , because θ ^ B \widehat{\theta}_B θ B is random. We first record that the scores are Lipschitz in θ \theta θ ; only the residual term of (23) depends on the composition. Let B ⋆ = { θ ∈ S K : ∥ θ − θ ⋆ ∥ 1 ≤ κ ⋆ / 2 } \mathcal{B}^\star=\{\theta\in S_K:\|\theta-\theta^\star\|_1\le\kappa_\star/2\} B ⋆ = { θ ∈ S K : ∥ θ − θ ⋆ ∥ 1 ≤ κ ⋆ /2 } . For θ , θ ′ ∈ B ⋆ \theta,\theta'\in\mathcal{B}^\star θ , θ ′ ∈ B ⋆ , Assumption 2 and ∣ m θ − m θ ⋆ ∣ ≤ ∥ θ − θ ⋆ ∥ 1 |m_\theta-m_{\theta^\star}|\le\|\theta-\theta^\star\|_1 ∣ m θ − m θ ⋆ ∣ ≤ ∥ θ − θ ⋆ ∥ 1 give m θ ∧ m θ ′ ≥ κ ⋆ / 2 m_\theta\wedge m_{\theta'}\ge\kappa_\star/2 m θ ∧ m θ ′ ≥ κ ⋆ /2 on contributing actions. Since also ∣ m θ − m θ ′ ∣ ≤ ∥ θ − θ ′ ∥ 1 |m_\theta-m_{\theta'}|\le\|\theta-\theta'\|_1 ∣ m θ − m θ ′ ∣ ≤ ∥ θ − θ ′ ∥ 1 ,
∣ 1 m θ ( a ∣ x ) − 1 m θ ′ ( a ∣ x ) ∣ ≤ 4 ∣ m θ ( a ∣ x ) − m θ ′ ( a ∣ x ) ∣ κ ⋆ 2 ≤ 4 K κ ⋆ 2 ∥ θ − θ ′ ∥ 2 . (49) \left|\frac{1}{m_\theta(a\mid x)}-\frac{1}{m_{\theta'}(a\mid x)}\right|
\le\frac{4|m_\theta(a\mid x)-m_{\theta'}(a\mid x)|}{\kappa_\star^2}
\le\frac{4\sqrt{K}}{\kappa_\star^2}\|\theta-\theta'\|_2. \tag*{(49)} m θ ( a ∣ x ) 1 − m θ ′ ( a ∣ x ) 1 ≤ κ ⋆ 2 4∣ m θ ( a ∣ x ) − m θ ′ ( a ∣ x ) ∣ ≤ κ ⋆ 2 4 K ∥ θ − θ ′ ∥ 2 . (49)
The uniform bound is stated for the stopped process
Ξ ‾ B ( θ ) = 1 B ∑ i = 1 n B 1 { i ≤ T B } { ψ i ( θ ) − ψ i ( θ ⋆ ) } , (50) \overline{\Xi}_B(\theta)=\frac{1}{\sqrt{B}}\sum_{i=1}^{n_B}1\{i\le T_B\}\{\psi_i(\theta)-\psi_i(\theta^\star)\}, \tag*{(50)} Ξ B ( θ ) = B 1 i = 1 ∑ n B 1 { i ≤ T B } { ψ i ( θ ) − ψ i ( θ ⋆ )} , (50)
and is proved in Appendix A6.
Lemma A4.3. Under Assumptions 1 and 2, there is a finite constant c 6 c_6 c 6 , depending only on K K K , M M M , R max R_{\max} R m a x , κ ⋆ \kappa_\star κ ⋆ , and c min c_{\min} c m i n , such that for every 0 < r ≤ κ ⋆ / ( 2 K ) 0<r\le\kappa_\star/(2K) 0 < r ≤ κ ⋆ / ( 2 K ) and B ≥ 1 B\ge1 B ≥ 1 ,
E sup ∥ θ − θ ⋆ ∥ 2 ≤ r ∥ Ξ ‾ B ( θ ) ∥ 2 ≤ c 6 r . (51) \mathbb{E}\sup_{\|\theta-\theta^\star\|_2\le r}\|\overline{\Xi}_B(\theta)\|_2\le c_6r. \tag*{(51)} E ∥ θ − θ ⋆ ∥ 2 ≤ r sup ∥ Ξ B ( θ ) ∥ 2 ≤ c 6 r . (51)
Both averages in (A4.1) use the same observations, and so
T B ( J ~ B − J ~ B ⋆ ) = 1 T B ∑ i = 1 T B { ψ i ( θ ^ B ) − ψ i ( θ ⋆ ) } = B T B Ξ ‾ B ( θ ^ B ) . \sqrt{T_B}(\widetilde{J}_B-\widetilde{J}_B^\star)
=\frac{1}{\sqrt{T_B}}\sum_{i=1}^{T_B}\{\psi_i(\widehat{\theta}_B)-\psi_i(\theta^\star)\}
=\sqrt{\frac{B}{T_B}}\overline{\Xi}_B(\widehat{\theta}_B). T B ( J B − J B ⋆ ) = T B 1 i = 1 ∑ T B { ψ i ( θ B ) − ψ i ( θ ⋆ )} = T B B Ξ B ( θ B ) .
Since C T B ≤ c max T B C_{T_B}\le c_{\max}T_B C T B ≤ c m a x T B and C T B > B − c max C_{T_B}>B-c_{\max} C T B > B − c m a x , for B ≥ 2 c max B\ge2c_{\max} B ≥ 2 c m a x we have
T B > B c max − 1 ≥ B 2 c max , and hence B / T B ≤ L = 2 c max . T_B>\frac{B}{c_{\max}}-1\ge\frac{B}{2c_{\max}},
\qquad\text{and hence}\qquad
\sqrt{B/T_B}\le L=\sqrt{2c_{\max}}. T B > c m a x B − 1 ≥ 2 c m a x B , and hence B / T B ≤ L = 2 c m a x .
Fix ϵ > 0 \epsilon>0 ϵ > 0 and ζ > 0 \zeta>0 ζ > 0 , choose 0 < r ≤ κ ⋆ / ( 2 K ) 0<r\le\kappa_\star/(2K) 0 < r ≤ κ ⋆ / ( 2 K ) with L c 6 r / ζ < ϵ / 2 Lc_6r/\zeta<\epsilon/2 L c 6 r / ζ < ϵ /2 , and let E r = { ∥ θ ^ B − θ ⋆ ∥ 2 ≤ r } E_r=\{\|\widehat{\theta}_B-\theta^\star\|_2\le r\} E r = { ∥ θ B − θ ⋆ ∥ 2 ≤ r } . On E r E_r E r , ∥ Ξ ‾ B ( θ ^ B ) ∥ 2 \|\overline{\Xi}_B(\widehat{\theta}_B)\|_2 ∥ Ξ B ( θ B ) ∥ 2 is at most the supremum in (A4.5). Splitting according to E r E_r E r , Markov’s inequality and Lemma A4.3 give
P { T B ∥ J ~ B − J ~ B ⋆ ∥ 2 > ζ } ≤ P ( ∥ θ ^ B − θ ⋆ ∥ 2 > r ) + L ζ c 6 r ≤ P ( ∥ θ ^ B − θ ⋆ ∥ 2 > r ) + ϵ 2 . \mathbb{P}\left\{\sqrt{T_B}\|\widetilde{J}_B-\widetilde{J}_B^\star\|_2>\zeta\right\}
\le\mathbb{P}(\|\widehat{\theta}_B-\theta^\star\|_2>r)+\frac{L}{\zeta}c_6r
\le\mathbb{P}(\|\widehat{\theta}_B-\theta^\star\|_2>r)+\frac{\epsilon}{2}. P { T B ∥ J B − J B ⋆ ∥ 2 > ζ } ≤ P ( ∥ θ B − θ ⋆ ∥ 2 > r ) + ζ L c 6 r ≤ P ( ∥ θ B − θ ⋆ ∥ 2 > r ) + 2 ϵ .
The first term tends to zero because θ ^ B → θ ⋆ \widehat{\theta}_B\to\theta^\star θ B → θ ⋆ in probability, which proves (A4.1). The final claim follows from part (i) and Slutsky’s theorem.
Proof of Corollary 4.1
Recall that the summand of (17) is
ψ ^ i = q ^ Π , B ( X i ) + π ( A i ∣ X i ) { R i − q ^ B ( X i , A i ) } m θ ^ B ( A i ∣ X i ) . \widehat{\psi}_i=\widehat{q}_{\Pi,B}(X_i)+\frac{\pi(A_i\mid X_i)\{R_i-\widehat{q}_B(X_i,A_i)\}}{m_{\widehat{\theta}_B}(A_i\mid X_i)}. ψ i = q Π , B ( X i ) + m θ B ( A i ∣ X i ) π ( A i ∣ X i ) { R i − q B ( X i , A i )} .
where q ^ Π , B ( x ) = { q ^ B , 1 ( x ) , … , q ^ B , M ( x ) } T \widehat{q}_{\Pi,B}(x)=\{\widehat{q}_{B,1}(x),\ldots,\widehat{q}_{B,M}(x)\}^{\mathsf{T}} q Π , B ( x ) = { q B , 1 ( x ) , … , q B , M ( x ) } T . Let E B = { ∥ θ ^ B − θ ⋆ ∥ 1 ≤ κ ⋆ / 2 } \mathcal{E}_B=\{\|\widehat{\theta}_B-\theta^\star\|_1\le\kappa_\star/2\} E B = { ∥ θ B − θ ⋆ ∥ 1 ≤ κ ⋆ /2 } , whose probability tends to one. Restricting to such events does not affect convergence in probability, so we work on E B \mathcal{E}_B E B , where m θ ^ B ≥ κ ⋆ / 2 m_{\widehat{\theta}_B}\ge\kappa_\star/2 m θ B ≥ κ ⋆ /2 on contributing actions. Since q ^ B \widehat{q}_B q B is truncated to [ − R max , R max ] [-R_{\max},R_{\max}] [ − R m a x , R m a x ] , ∣ R i ∣ ≤ R max |R_i|\le R_{\max} ∣ R i ∣ ≤ R m a x , and π j ≤ 1 \pi_j\le1 π j ≤ 1 ,
∥ ψ ^ i ∥ 2 ≤ ∥ q ^ Π , B ( X i ) ∥ 2 + ∥ π ( A i ∣ X i ) m θ ^ B ( A i ∣ X i ) ∥ 2 ∣ R i − q ^ B ( X i , A i ) ∣ ≤ M R max + 4 M R max κ ⋆ = K ψ , \|\widehat{\psi}_i\|_2\le\|\widehat{q}_{\Pi,B}(X_i)\|_2+\left\|\frac{\pi(A_i\mid X_i)}{m_{\widehat{\theta}_B}(A_i\mid X_i)}\right\|_2|R_i-\widehat{q}_B(X_i,A_i)|\le\sqrt{M}R_{\max}+\frac{4\sqrt{M}R_{\max}}{\kappa_\star}=K_\psi, ∥ ψ i ∥ 2 ≤ ∥ q Π , B ( X i ) ∥ 2 + m θ B ( A i ∣ X i ) π ( A i ∣ X i ) 2 ∣ R i − q B ( X i , A i ) ∣ ≤ M R m a x + κ ⋆ 4 M R m a x = K ψ ,
and the same bound holds for ψ i ( θ ⋆ ) \psi_i(\theta^\star) ψ i ( θ ⋆ ) .
We first show that Σ ^ B → Σ A ( θ ⋆ ) \widehat{\Sigma}_B\to\Sigma_A(\theta^\star) Σ B → Σ A ( θ ⋆ ) in probability. We compare Σ ^ B \widehat{\Sigma}_B Σ B with the empirical second moment of the scores at the limiting composition,
Σ B ⋆ = 1 T B ∑ i ≤ T B { ψ i ( θ ⋆ ) − J } { ψ i ( θ ⋆ ) − J } T , \Sigma_B^\star=\frac{1}{T_B}\sum_{i\le T_B}\{\psi_i(\theta^\star)-J\}\{\psi_i(\theta^\star)-J\}^{\mathsf{T}}, Σ B ⋆ = T B 1 i ≤ T B ∑ { ψ i ( θ ⋆ ) − J } { ψ i ( θ ⋆ ) − J } T ,
and write
Σ ^ B − Σ A ( θ ⋆ ) = ( Σ ^ B − Σ B ⋆ ) ⏟ ( I ) + { Σ B ⋆ − Σ A ( θ ⋆ ) } ⏟ ( I I ) . (52) \widehat{\Sigma}_B-\Sigma_A(\theta^\star)=\underbrace{(\widehat{\Sigma}_B-\Sigma_B^\star)}_{\mathrm{(I)}}+\underbrace{\{\Sigma_B^\star-\Sigma_A(\theta^\star)\}}_{\mathrm{(II)}}. \tag*{(52)} Σ B − Σ A ( θ ⋆ ) = ( I ) ( Σ B − Σ B ⋆ ) + ( II ) { Σ B ⋆ − Σ A ( θ ⋆ )} . (52)
Term (II) is the average of a martingale part and a predictable part. Let D i D_i D i be the outer product { ψ i ( θ ⋆ ) − J } { ψ i ( θ ⋆ ) − J } T \{\psi_i(\theta^\star)-J\}\{\psi_i(\theta^\star)-J\}^{\mathsf{T}} { ψ i ( θ ⋆ ) − J } { ψ i ( θ ⋆ ) − J } T minus its conditional expectation given F i − 1 \mathcal{F}_{i-1} F i − 1 . Because ∥ ψ i ( θ ⋆ ) − J ∥ 2 ≤ 2 K ψ \|\psi_i(\theta^\star)-J\|_2\le2K_\psi ∥ ψ i ( θ ⋆ ) − J ∥ 2 ≤ 2 K ψ , the matrices 1 { i ≤ T B } D i \mathbb{1}\{i\le T_B\}D_i 1 { i ≤ T B } D i , i ≤ n B i\le n_B i ≤ n B , form a martingale difference array bounded entrywise by 8 K ψ 2 8K_\psi^2 8 K ψ 2 , hence their sum has second moment O ( n B ) = O ( B ) O(n_B)=O(B) O ( n B ) = O ( B ) . Dividing by T B ≥ B / ( 2 c max ) T_B\ge B/(2c_{\max}) T B ≥ B / ( 2 c m a x ) shows that T B − 1 ∑ i ≤ T B D i → 0 T_B^{-1}\sum_{i\le T_B}D_i\to0 T B − 1 ∑ i ≤ T B D i → 0 in probability. The average of the conditional expectations is Γ ( θ ^ B ) \Gamma(\widehat{\theta}_B) Γ ( θ B ) from the proof of part (i) of Theorem A4.1, which converges to Σ A ( θ ⋆ ) \Sigma_A(\theta^\star) Σ A ( θ ⋆ ) in probability. Hence (II) tends to zero.
For term (I), the identity u u T − v v T = u ( u − v ) T + ( u − v ) v T uu^{\mathsf{T}}-vv^{\mathsf{T}}=u(u-v)^{\mathsf{T}}+(u-v)v^{\mathsf{T}} u u T − v v T = u ( u − v ) T + ( u − v ) v T gives
∥ u u T − v v T ∥ ≤ ( ∥ u ∥ + ∥ v ∥ ) ∥ u − v ∥ . \|uu^{\mathsf{T}}-vv^{\mathsf{T}}\|\le(\|u\|+\|v\|)\|u-v\|. ∥ u u T − v v T ∥ ≤ ( ∥ u ∥ + ∥ v ∥ ) ∥ u − v ∥.
Both ψ ^ i − J ^ B \widehat{\psi}_i-\widehat{J}_B ψ i − J B and ψ i ( θ ⋆ ) − J \psi_i(\theta^\star)-J ψ i ( θ ⋆ ) − J have norm at most 2 K ψ 2K_\psi 2 K ψ , and so
∥ Σ ^ B − Σ B ⋆ ∥ ≤ 4 K ψ { 1 T B ∑ i = 1 T B ∥ ψ ^ i − ψ i ( θ ⋆ ) ∥ + ∥ J ^ B − J ∥ } . (53) \left\|\widehat{\Sigma}_B-\Sigma_B^\star\right\|\le4K_\psi\left\{\frac{1}{T_B}\sum_{i=1}^{T_B}\|\widehat{\psi}_i-\psi_i(\theta^\star)\|+\|\widehat{J}_B-J\|\right\}. \tag*{(53)} Σ B − Σ B ⋆ ≤ 4 K ψ { T B 1 i = 1 ∑ T B ∥ ψ i − ψ i ( θ ⋆ ) ∥ + ∥ J B − J ∥ } . (53)
The second term in braces is o p ( 1 ) o_p(1) o p ( 1 ) by Theorem 4.2 . For the first, we split the score difference into a part from the denominator and a part from the regression,
ψ ^ i − ψ i ( θ ⋆ ) = π ( A i ∣ X i ) { 1 m θ ^ B ( A i ∣ X i ) − 1 m θ ⋆ ( A i ∣ X i ) } { R i − q ^ B ( X i , A i ) } + ζ i . (54) \widehat{\psi}_i-\psi_i(\theta^\star)=\pi(A_i\mid X_i)\left\{\frac{1}{m_{\widehat{\theta}_B}(A_i\mid X_i)}-\frac{1}{m_{\theta^\star}(A_i\mid X_i)}\right\}\{R_i-\widehat{q}_B(X_i,A_i)\}+\zeta_i. \tag*{(54)} ψ i − ψ i ( θ ⋆ ) = π ( A i ∣ X i ) { m θ B ( A i ∣ X i ) 1 − m θ ⋆ ( A i ∣ X i ) 1 } { R i − q B ( X i , A i )} + ζ i . (54)
where
ζ i = { q ^ Π , B − q Π } ( X i ) − π ( A i ∣ X i ) { q ^ B − q } ( X i , A i ) m θ ⋆ ( A i ∣ X i ) . \zeta_i=\{\widehat{q}_{\Pi,B}-q_\Pi\}(X_i)-\frac{\pi(A_i\mid X_i)\{\widehat{q}_B-q\}(X_i,A_i)}{m_{\theta^\star}(A_i\mid X_i)}. ζ i = { q Π , B − q Π } ( X i ) − m θ ⋆ ( A i ∣ X i ) π ( A i ∣ X i ) { q B − q } ( X i , A i ) .
For (a), (A4.3) gives ∣ 1 / m θ ^ B − 1 / m θ ⋆ ∣ ≤ 4 K κ ⋆ − 2 ∥ θ ^ B − θ ⋆ ∥ 2 |1/m_{\widehat{\theta}_B}-1/m_{\theta^\star}| \leq4\sqrt{K}\kappa_\star^{-2}\|\widehat{\theta}_B-\theta^\star\|_2 ∣1/ m θ B − 1/ m θ ⋆ ∣ ≤ 4 K κ ⋆ − 2 ∥ θ B − θ ⋆ ∥ 2 on contributing actions. With ∣ R i − q ^ B ( X i , A i ) ∣ ≤ 2 R max |R_i-\widehat{q}_B(X_i,A_i)|\leq2R_{\max} ∣ R i − q B ( X i , A i ) ∣ ≤ 2 R m a x and ∥ π ( A i ∣ X i ) ∥ 2 ≤ M \|\pi(A_i\mid X_i)\|_2\leq\sqrt{M} ∥ π ( A i ∣ X i ) ∥ 2 ≤ M , every term of (a) has norm at most
8 M K R max κ ⋆ − 2 ∥ θ ^ B − θ ⋆ ∥ 2 , 8\sqrt{M}KR_{\max}\kappa_\star^{-2}\|\widehat{\theta}_B-\theta^\star\|_2, 8 M K R m a x κ ⋆ − 2 ∥ θ B − θ ⋆ ∥ 2 ,
and so its average over i ≤ T B i\leq T_B i ≤ T B tends to zero in probability. For (b), since π j ≤ 1 \pi_j\leq1 π j ≤ 1 and m θ ⋆ ≥ κ ⋆ m_{\theta^\star}\geq\kappa_\star m θ ⋆ ≥ κ ⋆ on contributing actions, Jensen’s inequality gives
ζ i , j 2 ≤ 2 ( 1 + κ ⋆ 2 ) ∑ a { q ^ B − q } 2 ( X i , a ) . \zeta_{i,j}^2\leq2(1+\kappa_\star^2)\sum_a\{\widehat{q}_B-q\}^2(X_i,a). ζ i , j 2 ≤ 2 ( 1 + κ ⋆ 2 ) a ∑ { q B − q } 2 ( X i , a ) .
The mean-square consistency condition then yields max j T B − 1 ∑ i ≤ T B ζ i , j 2 = o p ( 1 ) \max_j T_B^{-1}\sum_{i\leq T_B}\zeta_{i,j}^2=o_p(1) max j T B − 1 ∑ i ≤ T B ζ i , j 2 = o p ( 1 ) , and the Cauchy–Schwarz inequality gives
1 T B ∑ i = 1 T B ∥ ζ i ∥ ≤ M max j { 1 T B ∑ i = 1 T B ζ i , j 2 } 1 / 2 = o p ( 1 ) . \frac{1}{T_B}\sum_{i=1}^{T_B}\|\zeta_i\|\leq\sqrt{M}\max_j\left\{\frac{1}{T_B}\sum_{i=1}^{T_B}\zeta_{i,j}^2\right\}^{1/2}=o_p(1). T B 1 i = 1 ∑ T B ∥ ζ i ∥ ≤ M j max { T B 1 i = 1 ∑ T B ζ i , j 2 } 1/2 = o p ( 1 ) .
The first term in (A4.7) is therefore o p ( 1 ) o_p(1) o p ( 1 ) , so (I) tends to zero, and Σ ^ B → Σ A ( θ ⋆ ) \widehat{\Sigma}_B\to\Sigma_A(\theta^\star) Σ B → Σ A ( θ ⋆ ) in probability.
For the intervals, fix j j j with A j ( θ ⋆ ) > 0 A_j(\theta^\star)>0 A j ( θ ⋆ ) > 0 . By Theorem 4.2 , T B ( J ^ B , j − J j ) → N { 0 , A j ( θ ⋆ ) } \sqrt{T_B}(\widehat{J}_{B,j}-J_j)\to N\{0,A_j(\theta^\star)\} T B ( J B , j − J j ) → N { 0 , A j ( θ ⋆ )} , and ( Σ ^ B ) j j → A j ( θ ⋆ ) (\widehat{\Sigma}_B)_{jj}\to A_j(\theta^\star) ( Σ B ) j j → A j ( θ ⋆ ) in probability, since A j ( θ ⋆ ) A_j(\theta^\star) A j ( θ ⋆ ) is the j j j th diagonal entry of Σ A ( θ ⋆ ) \Sigma_A(\theta^\star) Σ A ( θ ⋆ ) . Slutsky’s theorem gives
T B ( J ^ B , j − J j ) ( Σ ^ B ) j j 1 / 2 → N ( 0 , 1 ) , \frac{\sqrt{T_B}(\widehat{J}_{B,j}-J_j)}{(\widehat{\Sigma}_B)_{jj}^{1/2}}\to N(0,1), ( Σ B ) j j 1/2 T B ( J B , j − J j ) → N ( 0 , 1 ) ,
which is the coverage statement for (29) . Finally, A j ( θ ⋆ ) > 0 A_j(\theta^\star)>0 A j ( θ ⋆ ) > 0 holds whenever V j > 0 V_j>0 V j > 0 or E X ∑ a ∈ A W j ( X , a ) > 0 \mathbb{E}_X\sum_{a\in\mathcal{A}}W_j(X,a)>0 E X ∑ a ∈ A W j ( X , a ) > 0 .
Pooled Efficiency
The logger-specific estimator divides each observation by the propensity of the logger that generated it. At a fixed logger composition θ \theta θ , its covariance is
Σ L ( θ ) = Cov { q Π ( X ) } + E X ∑ a ∈ A σ 2 ( X , a ) π ( a ∣ X ) π ( a ∣ X ) ⊤ ∑ k = 1 K θ k μ k ( a ∣ X ) . (55) \Sigma_L(\theta)=\operatorname{Cov}\{q_\Pi(X)\}+\mathbb{E}_X\sum_{a\in\mathcal{A}}\sigma^2(X,a)\pi(a\mid X)\pi(a\mid X)^\top\sum_{k=1}^{K}\frac{\theta_k}{\mu_k(a\mid X)}. \tag*{(55)} Σ L ( θ ) = Cov { q Π ( X )} + E X a ∈ A ∑ σ 2 ( X , a ) π ( a ∣ X ) π ( a ∣ X ) ⊤ k = 1 ∑ K μ k ( a ∣ X ) θ k . (55)
whose j j j th diagonal entry is
L j ( θ ) = V j + E X ∑ a ∈ A W j ( X , a ) ∑ k = 1 K θ k μ k ( a ∣ X ) . (56) L_j(\theta)=V_j+\mathbb{E}_X\sum_{a\in\mathcal{A}}W_j(X,a)\sum_{k=1}^{K}\frac{\theta_k}{\mu_k(a\mid X)}. \tag*{(56)} L j ( θ ) = V j + E X a ∈ A ∑ W j ( X , a ) k = 1 ∑ K μ k ( a ∣ X ) θ k . (56)
To derive (30) , fix u ∈ R M u\in\mathbb{R}^M u ∈ R M . The context terms of Σ A ( θ ) \Sigma_A(\theta) Σ A ( θ ) and Σ L ( θ ) \Sigma_L(\theta) Σ L ( θ ) coincide, so it suffices to compare the residual terms. For each ( x , a ) (x,a) ( x , a ) , Jensen’s inequality for the convex function y ↦ 1 / y y\mapsto1/y y ↦ 1/ y , with 1 / 0 = ∞ 1/0=\infty 1/0 = ∞ , gives
1 m θ ( a ∣ x ) = 1 ∑ k θ k μ k ( a ∣ x ) ≤ ∑ k = 1 K θ k μ k ( a ∣ x ) . \frac{1}{m_\theta(a\mid x)}=\frac{1}{\sum_k\theta_k\mu_k(a\mid x)}\leq\sum_{k=1}^{K}\frac{\theta_k}{\mu_k(a\mid x)}. m θ ( a ∣ x ) 1 = ∑ k θ k μ k ( a ∣ x ) 1 ≤ k = 1 ∑ K μ k ( a ∣ x ) θ k .
Multiplying by σ 2 ( x , a ) { u ⊤ π ( a ∣ x ) } 2 ≥ 0 \sigma^2(x,a)\{u^\top\pi(a\mid x)\}^2\geq0 σ 2 ( x , a ) { u ⊤ π ( a ∣ x ) } 2 ≥ 0 , summing over a a a , and integrating over P X P_X P X gives u ⊤ Σ A ( θ ) u ≤ u ⊤ Σ L ( θ ) u u^\top\Sigma_A(\theta)u\leq u^\top\Sigma_L(\theta)u u ⊤ Σ A ( θ ) u ≤ u ⊤ Σ L ( θ ) u . Taking u = e j u=e_j u = e j gives A j ( θ ) ≤ L j ( θ ) A_j(\theta)\leq L_j(\theta) A j ( θ ) ≤ L j ( θ ) . For a fixed contrast u ⊤ J u^\top J u ⊤ J , equality holds if and only if the logger propensities with θ k > 0 \theta_k>0 θ k > 0 agree on the context-action regions where σ 2 ( x , a ) { u ⊤ π ( a ∣ x ) } 2 > 0 \sigma^2(x,a)\{u^\top\pi(a\mid x)\}^2>0 σ 2 ( x , a ) { u ⊤ π ( a ∣ x ) } 2 > 0 , which is the equality condition of Jensen’s inequality. In particular, A j ( θ ⋆ ) ≤ L j ( θ ⋆ ) A_j(\theta^\star)\leq L_j(\theta^\star) A j ( θ ⋆ ) ≤ L j ( θ ⋆ ) for each target.
A Block-Split Estimator
In the final estimator (17) , q ^ B \widehat q_B q B is fitted on the same observations over which it is averaged, and the denominator m θ ^ B m_{\widehat\theta_B} m θ B is random. The block-split estimator divides the sample into blocks of geometrically growing size, and evaluates each block with a regression fitted on earlier blocks and with its own realized logger frequencies in the denominator.
Fix n ˉ ≥ 1 \bar n \ge1 n ˉ ≥ 1 and ϰ > 1 \varkappa> 1 ϰ > 1 with ( ϰ − 1 ) n ˉ ≥ 1 (\varkappa-1)\bar n \ge1 ( ϰ − 1 ) n ˉ ≥ 1 , let τ ˉ 0 = 0 \bar\tau_0=0 τ ˉ 0 = 0 and τ ˉ ℓ = ⌈ ϰ ℓ − 1 n ˉ ⌉ \bar\tau_\ell=\lceil\varkappa^{\ell-1}\bar n\rceil τ ˉ ℓ = ⌈ ϰ ℓ − 1 n ˉ ⌉ for ℓ ≥ 1 \ell\ge1 ℓ ≥ 1 , and let
L B = max { ℓ ≥ 1 : τ ˉ ℓ ≤ T B } , τ 0 = 0 , τ ℓ = τ ˉ ℓ ( 1 ≤ ℓ < L B ) , τ L B = T B . (57) L_B=\max\{\ell\ge1:\bar\tau_\ell\le T_B\},\qquad\tau_0=0,\qquad\tau_\ell=\bar\tau_\ell\ (1\le\ell<L_B),\qquad\tau_{L_B}=T_B. \tag*{(57)} L B = max { ℓ ≥ 1 : τ ˉ ℓ ≤ T B } , τ 0 = 0 , τ ℓ = τ ˉ ℓ ( 1 ≤ ℓ < L B ) , τ L B = T B . (57)
The condition ( ϰ − 1 ) n ˉ ≥ 1 (\varkappa-1)\bar n\ge1 ( ϰ − 1 ) n ˉ ≥ 1 gives ϰ ℓ n ˉ − ϰ ℓ − 1 n ˉ > 1 \varkappa^\ell\bar n-\varkappa^{\ell-1}\bar n>1 ϰ ℓ n ˉ − ϰ ℓ − 1 n ˉ > 1 , so the τ ˉ ℓ \bar\tau_\ell τ ˉ ℓ strictly increase and every block is nonempty. Set B ℓ = { i : τ ℓ − 1 < i ≤ τ ℓ } B_\ell=\{i:\tau_{\ell-1}<i\le\tau_\ell\} B ℓ = { i : τ ℓ − 1 < i ≤ τ ℓ } and n ℓ = ∣ B ℓ ∣ n_\ell=|B_\ell| n ℓ = ∣ B ℓ ∣ . For ℓ ≥ 2 \ell\ge2 ℓ ≥ 2 , let q ^ j ( ℓ ) \widehat q_j^{(\ell)} q j ( ℓ ) be fitted using only observations before block ℓ \ell ℓ , and let θ ^ k ( ℓ ) = n ℓ − 1 ∑ i ∈ B ℓ 1 { K i = k } \widehat\theta_k^{(\ell)}=n_\ell^{-1}\sum_{i\in B_\ell}\mathbf{1}\{K_i=k\} θ k ( ℓ ) = n ℓ − 1 ∑ i ∈ B ℓ 1 { K i = k } be the logger proportions within block ℓ \ell ℓ . The block-split estimator is
J ^ B , j B S = 1 T B − τ 1 ∑ ℓ = 2 L B ∑ i ∈ B ℓ [ q ^ j ( ℓ ) ( X i ) + π j ( A i ∣ X i ) m θ ^ ( ℓ ) ( A i ∣ X i ) { R i − q ^ ( ℓ ) ( X i , A i ) } ] . (58) \widehat J^{\mathrm{BS}}_{B,j}
=\frac{1}{T_B-\tau_1}\sum_{\ell=2}^{L_B}\sum_{i\in B_\ell}
\left[
\widehat q_j^{(\ell)}(X_i)
+\frac{\pi_j(A_i\mid X_i)}{m_{\widehat\theta^{(\ell)}}(A_i\mid X_i)}
\{R_i-\widehat q^{(\ell)}(X_i,A_i)\}
\right]. \tag*{(58)} J B , j BS = T B − τ 1 1 ℓ = 2 ∑ L B i ∈ B ℓ ∑ [ q j ( ℓ ) ( X i ) + m θ ( ℓ ) ( A i ∣ X i ) π j ( A i ∣ X i ) { R i − q ( ℓ ) ( X i , A i )} ] . (58)
The regression for block ℓ \ell ℓ is fixed before the block begins, and the denominator uses the block’s own logger frequencies. The construction is designed so that the regression error enters only through a martingale term.
Proofs of Technical Results
The proofs below use the notation of Appendices A3 and § 10 . We present them in the order of logical dependence rather than the order in which the results are stated.
Proof of Proposition A3.2
It suffices to show that the Hessian of G j \mathcal G_j G j dominates 2 c min 3 Q j 2c_{\min}^3Q_j 2 c m i n 3 Q j along feasible directions, since Assumption 3 bounds the quadratic form of Q j Q_j Q j below by λ j ∥ h ∥ 2 2 \lambda_j\lVert h\rVert_2^2 λ j ∥ h ∥ 2 2 . On each C ( e ) C(e) C ( e ) , both denominators in (7) are bounded away from zero, by c max − 1 c_{\max}^{-1} c m a x − 1 and e e e , and the integrands are bounded; thus the Hessian exists and is continuous there, and differentiating (7) twice gives
h ⊤ ∇ 2 G j ( v ) h = 2 V j ( 1 ⊤ h ) 2 ( 1 ⊤ v ) 3 + 2 E X ∑ a ∈ A W j ( X , a ) { μ ( X , a ) ⊤ h } 2 { μ ( X , a ) ⊤ v } 3 . h^\top\nabla^2\mathcal G_j(v)h=
\frac{2V_j(\mathbf{1}^\top h)^2}{(\mathbf{1}^\top v)^3}
+2\mathbb E_X\sum_{a\in\mathcal A}
\frac{W_j(X,a)\{\mu(X,a)^\top h\}^2}{\{\mu(X,a)^\top v\}^3}. h ⊤ ∇ 2 G j ( v ) h = ( 1 ⊤ v ) 3 2 V j ( 1 ⊤ h ) 2 + 2 E X a ∈ A ∑ { μ ( X , a ) ⊤ v } 3 W j ( X , a ) { μ ( X , a ) ⊤ h } 2 .
The numerators are the two terms of h ⊤ Q j h h^\top Q_jh h ⊤ Q j h . Meanwhile, 1 = c ⊤ v ≥ c min 1 ⊤ v 1=c^\top v\ge c_{\min}\mathbf{1}^\top v 1 = c ⊤ v ≥ c m i n 1 ⊤ v and μ k ≤ 1 \mu_k\le1 μ k ≤ 1 give
μ ( x , a ) ⊤ v ≤ 1 ⊤ v ≤ c min − 1 , \mu(x,a)^\top v\le\mathbf{1}^\top v\le c_{\min}^{-1}, μ ( x , a ) ⊤ v ≤ 1 ⊤ v ≤ c m i n − 1 ,
and so each reciprocal cube is at least c min 3 c_{\min}^3 c m i n 3 . By Assumption 3,
h ⊤ ∇ 2 G j ( v ) h ≥ 2 c min 3 h ⊤ Q j h ≥ 2 c min 3 λ j ∥ h ∥ 2 2 , h^\top\nabla^2\mathcal G_j(v)h\ge2c_{\min}^3h^\top Q_jh\ge2c_{\min}^3\lambda_j\lVert h\rVert_2^2, h ⊤ ∇ 2 G j ( v ) h ≥ 2 c m i n 3 h ⊤ Q j h ≥ 2 c m i n 3 λ j ∥ h ∥ 2 2 ,
which is (A3.3).
To pass to strong convexity, take y , y ′ ∈ C y,y'\in C y , y ′ ∈ C such that the segment between them lies in C + C_+ C + , and put h = y ′ − y h=y'-y h = y ′ − y , so that c ⊤ h = 0 c^\top h=0 c ⊤ h = 0 . If y ∈ C ( e ) y\in C(e) y ∈ C ( e ) and y ′ ∈ C ( e ′ ) y'\in C(e') y ′ ∈ C ( e ′ ) , linearity of the coverage puts the whole segment in C ( min { e , e ′ } ) C(\min\{e,e'\}) C ( min { e , e ′ }) ; hence s ↦ G j ( y + s h ) s\mapsto\mathcal G_j(y+sh) s ↦ G j ( y + s h ) is twice differentiable on [ 0 , 1 ] [0,1] [ 0 , 1 ] , with second derivative at least m j ∥ h ∥ 2 2 m_j\lVert h\rVert_2^2 m j ∥ h ∥ 2 2 , where m j = 2 c min 3 λ j m_j=2c_{\min}^3\lambda_j m j = 2 c m i n 3 λ j . With m = min j m j m=\min_jm_j m = min j m j , each G j − m 2 ∥ ⋅ ∥ 2 2 \mathcal G_j-\frac{m}{2}\lVert\cdot\rVert_2^2 G j − 2 m ∥ ⋅ ∥ 2 2 is therefore convex along the segment, and so is their maximum G − m 2 ∥ ⋅ ∥ 2 2 \mathcal G-\frac{m}{2}\lVert\cdot\rVert_2^2 G − 2 m ∥ ⋅ ∥ 2 2 .
Proof of Lemma A3.2
If G ( v ) = ∞ \mathcal{G}(v)=\infty G ( v ) = ∞ there is nothing to prove; otherwise, write h = v − z ⋆ h=v-z^\star h = v − z ⋆ and ϕ ( s ) = G ( z ⋆ + s h ) \phi(s)=\mathcal{G}(z^\star+sh) ϕ ( s ) = G ( z ⋆ + s h ) for s ∈ [ 0 , 1 ] s\in[0,1] s ∈ [ 0 , 1 ] . By Assumption 2, z ⋆ ∈ C ( κ ⋆ / c max ) z^\star\in\mathcal{C}(\kappa_\star/c_{\max}) z ⋆ ∈ C ( κ ⋆ / c m a x ) , and because μ T ( z ⋆ + s h ) ≥ ( 1 − s ) μ T z ⋆ \boldsymbol{\mu}^{\mathsf{T}}(z^\star+sh)\ge(1-s)\boldsymbol{\mu}^{\mathsf{T}}z^\star μ T ( z ⋆ + s h ) ≥ ( 1 − s ) μ T z ⋆ , the segment from z ⋆ z^\star z ⋆ to z ⋆ + s ′ h z^\star+s'h z ⋆ + s ′ h lies in C { ( 1 − s ′ ) κ ⋆ / c max } \mathcal{C}\{(1-s')\kappa_\star/c_{\max}\} C {( 1 − s ′ ) κ ⋆ / c m a x } for each s ′ < 1 s'<1 s ′ < 1 . Proposition A3.2 therefore gives, for 0 < s < s ′ < 1 0<s<s'<1 0 < s < s ′ < 1 ,
ϕ ( s ) ≤ ( 1 − s s ′ ) ϕ ( 0 ) + s s ′ ϕ ( s ′ ) − m 2 s s ′ ( 1 − s s ′ ) s 2 ∥ h ∥ 2 . \phi(s)\le\left(1-\frac{s}{s'}\right)\phi(0)+\frac{s}{s'}\phi(s')-\frac{m}{2}\frac{s}{s'}\left(1-\frac{s}{s'}\right)s^2\lVert h\rVert^2. ϕ ( s ) ≤ ( 1 − s ′ s ) ϕ ( 0 ) + s ′ s ϕ ( s ′ ) − 2 m s ′ s ( 1 − s ′ s ) s 2 ∥ h ∥ 2 .
Since G \mathcal{G} G is convex on C \mathcal{C} C , ϕ ( s ′ ) ≤ ( 1 − s ′ ) ϕ ( 0 ) + s ′ ϕ ( 1 ) \phi(s')\le(1-s')\phi(0)+s'\phi(1) ϕ ( s ′ ) ≤ ( 1 − s ′ ) ϕ ( 0 ) + s ′ ϕ ( 1 ) , and letting s ′ ↑ 1 s'\uparrow1 s ′ ↑ 1 yields
ϕ ( s ) ≤ ( 1 − s ) ϕ ( 0 ) + s ϕ ( 1 ) − m 2 s ( 1 − s ) ∥ h ∥ 2 . \phi(s)\le(1-s)\phi(0)+s\phi(1)-\frac{m}{2}s(1-s)\lVert h\rVert^2. ϕ ( s ) ≤ ( 1 − s ) ϕ ( 0 ) + s ϕ ( 1 ) − 2 m s ( 1 − s ) ∥ h ∥ 2 .
The left side is at least ϕ ( 0 ) = G ( z ⋆ ) \phi(0)=\mathcal{G}(z^\star) ϕ ( 0 ) = G ( z ⋆ ) ; rearranging and dividing by s s s gives
G ( v ) − G ( z ⋆ ) ≥ m 2 ( 1 − s ) ∥ h ∥ 2 , \mathcal{G}(v)-\mathcal{G}(z^\star)\ge\frac{m}{2}(1-s)\lVert h\rVert^2, G ( v ) − G ( z ⋆ ) ≥ 2 m ( 1 − s ) ∥ h ∥ 2 ,
and letting s ↓ 0 s\downarrow0 s ↓ 0 proves the claim.
Proof of Lemma A3.3
Parts (a) to (c) are standard. Part (a) follows from
exp { G ( v ) / ρ } ≤ ∑ j exp { G j ( v ) / ρ } ≤ M exp { G ( v ) / ρ } , \exp\{\mathcal{G}(v)/\rho\}\le\sum_j\exp\{G_j(v)/\rho\}\le M\exp\{\mathcal{G}(v)/\rho\}, exp { G ( v ) / ρ } ≤ j ∑ exp { G j ( v ) / ρ } ≤ M exp { G ( v ) / ρ } ,
part (b) is the chain rule, and (c) holds because the log-sum-exp function is convex and increasing in each argument, while each G j G_j G j is convex. For (d), differentiating (b) once more gives
∇ 2 Φ ρ = ∑ j ω j ∇ 2 G j + 1 ρ { ∑ j ω j ∇ G j ∇ G j T − ( ∑ j ω j ∇ G j ) ( ∑ j ω j ∇ G j ) T } . \nabla^2\Phi_\rho=\sum_j\omega_j\nabla^2G_j+\frac{1}{\rho}\left\{\sum_j\omega_j\nabla G_j\nabla G_j^{\mathsf{T}}-\left(\sum_j\omega_j\nabla G_j\right)\left(\sum_j\omega_j\nabla G_j\right)^{\mathsf{T}}\right\}. ∇ 2 Φ ρ = j ∑ ω j ∇ 2 G j + ρ 1 ⎩ ⎨ ⎧ j ∑ ω j ∇ G j ∇ G j T − ( j ∑ ω j ∇ G j ) ( j ∑ ω j ∇ G j ) T ⎭ ⎬ ⎫ .
a weighted average of the Hessians of the G j G_j G j plus ρ − 1 \rho^{-1} ρ − 1 times the covariance matrix of the target gradients under the weights ω j \omega_j ω j . For v ∈ C ( e ) v\in\mathcal{C}(e) v ∈ C ( e ) , the bounds ∥ 11 T ∥ o p = K \lVert\mathbf{1}\mathbf{1}^{\mathsf{T}}\rVert_{\mathrm{op}}=K ∥ 1 1 T ∥ op = K , ∥ μ μ T ∥ o p ≤ K \lVert\boldsymbol{\mu}\boldsymbol{\mu}^{\mathsf{T}}\rVert_{\mathrm{op}}\le K ∥ μ μ T ∥ op ≤ K , V j ≤ R max 2 V_j\le R_{\max}^2 V j ≤ R m a x 2 , and E X ∑ a W j ( X , a ) ≤ R max 2 \mathbb{E}_X\sum_a W_j(X,a)\le R_{\max}^2 E X ∑ a W j ( X , a ) ≤ R m a x 2 give
∥ ∇ 2 G j ( v ) ∥ o p ≤ 2 K R max 2 ( c max 3 + e − 3 ) , ∣ ∂ k G j ( v ) ∣ ≤ R max 2 ( c max 2 + e − 2 ) . \lVert\nabla^2G_j(v)\rVert_{\mathrm{op}}\le2KR_{\max}^2(c_{\max}^3+e^{-3}),\qquad\lvert\partial_kG_j(v)\rvert\le R_{\max}^2(c_{\max}^2+e^{-2}). ∥ ∇ 2 G j ( v ) ∥ op ≤ 2 K R m a x 2 ( c m a x 3 + e − 3 ) , ∣ ∂ k G j ( v )∣ ≤ R m a x 2 ( c m a x 2 + e − 2 ) .
The covariance term therefore has operator norm at most max j ∥ ∇ G j ( v ) ∥ 2 2 ≤ K R max 4 ( c max 2 + e − 2 ) 2 \max_j\lVert\nabla G_j(v)\rVert_2^2\le KR_{\max}^4(c_{\max}^2+e^{-2})^2 max j ∥ ∇ G j ( v ) ∥ 2 2 ≤ K R m a x 4 ( c m a x 2 + e − 2 ) 2 , and since C ( e ) \mathcal{C}(e) C ( e ) is convex, these bounds give the Lipschitz constant. Finally, for (e), a direct calculation gives
∂ ρ Φ ρ = log ∑ j exp ( G j / ρ ) − ∑ j ω j G j / ρ = − ∑ j ω j log ω j , \partial_\rho\Phi_\rho=\log\sum_j\exp(G_j/\rho)-\sum_j\omega_jG_j/\rho=-\sum_j\omega_j\log\omega_j, ∂ ρ Φ ρ = log j ∑ exp ( G j / ρ ) − j ∑ ω j G j / ρ = − j ∑ ω j log ω j ,
the entropy of the weights, which is nonnegative.
Proof of Lemma A3.4
The argument rests on two facts. First, one purchase can lower the pooled coverage by at most a factor t / ( t + 1 ) t/(t+1) t / ( t + 1 ) , whichever logger is bought: since μ K t + 1 ≥ 0 \mu_{K_{t+1}}\ge0 μ K t + 1 ≥ 0 ,
m θ t + 1 = t m θ t + μ K t + 1 t + 1 ≥ t t + 1 m θ t . (59) m_{\theta_{t+1}}=\frac{t\,m_{\theta_t}+\mu_{K_{t+1}}}{t+1}\ge\frac{t}{t+1}m_{\theta_t}. \tag*{(59)} m θ t + 1 = t + 1 t m θ t + μ K t + 1 ≥ t + 1 t m θ t . (59)
Second, a forced purchase raises a smallest count, and at most K K K loggers share it; along consecutive forced purchases, the smallest count therefore grows by at least one every K K K purchases.
For (a), the initialization gives θ n 0 , k = 1 / K \theta_{n_0,k}=1/K θ n 0 , k = 1/ K , and so (A3.4) yields
κ ( θ n 0 ) ≥ κ a l l K ≥ 8 χ 0 > χ n 0 . \kappa(\theta_{n_0}) \ge\frac{\kappa_{\mathrm{all}}}{K} \ge8\chi_0 > \chi_{n_0}. κ ( θ n 0 ) ≥ K κ all ≥ 8 χ 0 > χ n 0 .
Unforced steps have κ ( θ t ) ≥ χ t \kappa(\theta_t) \ge\chi_t κ ( θ t ) ≥ χ t by definition, so it only remains to consider a maximal block of forced steps t 0 , t 0 + 1 , … t_0,t_0+1,\ldots t 0 , t 0 + 1 , … . Here t 0 > n 0 ≥ 4 K t_0>n_0\ge4K t 0 > n 0 ≥ 4 K , and the unforced step t 0 − 1 t_0-1 t 0 − 1 has κ ( θ t 0 − 1 ) ≥ χ t 0 \kappa(\theta_{t_0-1})\ge\chi_{t_0} κ ( θ t 0 − 1 ) ≥ χ t 0 . Let n = ⌈ 2 χ t 0 t 0 / κ a l l ⌉ n=\lceil2\chi_{t_0}t_0/\kappa_{\mathrm{all}}\rceil n = ⌈ 2 χ t 0 t 0 / κ all ⌉ and L = K n L=Kn L = K n . If the block lasted L L L steps, the smallest count would reach n n n , and (A3.4) would give
κ ( θ t 0 + L ) ≥ κ a l l n t 0 + K n ≥ χ t 0 ≥ χ t 0 + L . \kappa(\theta_{t_0+L}) \ge\frac{\kappa_{\mathrm{all}}n}{t_0+Kn} \ge\chi_{t_0} \ge\chi_{t_0+L}. κ ( θ t 0 + L ) ≥ t 0 + K n κ all n ≥ χ t 0 ≥ χ t 0 + L .
where the middle inequality is equivalent to n ( κ a l l − K χ t 0 ) ≥ χ t 0 t 0 n(\kappa_{\mathrm{all}}-K\chi_{t_0})\ge\chi_{t_0}t_0 n ( κ all − K χ t 0 ) ≥ χ t 0 t 0 and so holds because K χ t 0 ≤ κ a l l / 8 K\chi_{t_0}\le\kappa_{\mathrm{all}}/8 K χ t 0 ≤ κ all /8 and n ≥ 2 χ t 0 t 0 / κ a l l n\ge2\chi_{t_0}t_0/\kappa_{\mathrm{all}} n ≥ 2 χ t 0 t 0 / κ all . The block therefore lasts at most
L ≤ 2 K χ t 0 t 0 κ a l l + K ≤ t 0 4 + t 0 4 = t 0 2 L \le\frac{2K\chi_{t_0}t_0}{\kappa_{\mathrm{all}}}+K \le\frac{t_0}{4}+\frac{t_0}{4}=\frac{t_0}{2} L ≤ κ all 2 K χ t 0 t 0 + K ≤ 4 t 0 + 4 t 0 = 2 t 0
steps, and repeated use of (A6.1) gives, within the block,
κ ( θ t ) ≥ κ ( θ t 0 − 1 ) t 0 − 1 t 0 + L ≥ χ t 0 t 0 − 1 3 t 0 / 2 ≥ χ t 0 2 ≥ χ t 2 , \kappa(\theta_t)\ge\kappa(\theta_{t_0-1})\frac{t_0-1}{t_0+L}\ge\chi_{t_0}\frac{t_0-1}{3t_0/2}\ge\frac{\chi_{t_0}}{2}\ge\frac{\chi_t}{2}, κ ( θ t ) ≥ κ ( θ t 0 − 1 ) t 0 + L t 0 − 1 ≥ χ t 0 3 t 0 /2 t 0 − 1 ≥ 2 χ t 0 ≥ 2 χ t ,
using t 0 ≥ 4 t_0\ge4 t 0 ≥ 4 and that χ t \chi_t χ t is nonincreasing.
For (b), at a forced step s s s , (A3.4) gives
κ a l l min k N s , k s ≤ κ ( θ s ) < χ s , \frac{\kappa_{\mathrm{all}}\min_k N_{s,k}}{s}\le\kappa(\theta_s)<\chi_s, s κ all min k N s , k ≤ κ ( θ s ) < χ s ,
and so the logger bought, which has the smallest count, has fewer than χ s s / κ a l l \chi_s s/\kappa_{\mathrm{all}} χ s s / κ all observations. Because χ s s = χ 0 s 1 − υ \chi_s s=\chi_0s^{1-\upsilon} χ s s = χ 0 s 1 − υ is nondecreasing, a logger is forced at steps s ≤ t s\le t s ≤ t only while its count is below χ t t / κ a l l \chi_t t/\kappa_{\mathrm{all}} χ t t / κ all , hence at most χ t t / κ a l l + 1 \chi_t t/\kappa_{\mathrm{all}}+1 χ t t / κ all + 1 times. Summing over loggers, the p p p th forced step s p s_p s p satisfies p ≤ K ( χ 0 s p 1 − υ / κ a l l + 1 ) p\le K(\chi_0s_p^{1-\upsilon}/\kappa_{\mathrm{all}}+1) p ≤ K ( χ 0 s p 1 − υ / κ all + 1 ) , and so
s p ≥ { p κ a l l 2 K χ 0 } 1 / ( 1 − υ ) s_p\ge\left\{\frac{p\kappa_{\mathrm{all}}}{2K\chi_0}\right\}^{1/(1-\upsilon)} s p ≥ { 2 K χ 0 p κ all } 1/ ( 1 − υ )
for p ≥ 2 K p\ge2K p ≥ 2 K .
Since γ s ≤ 1 \gamma_s\le1 γ s ≤ 1 and γ s ≤ c max / C s + 1 ≤ c max / ( c min s ) \gamma_s\le c_{\max}/C_{s+1}\le c_{\max}/(c_{\min}s) γ s ≤ c m a x / C s + 1 ≤ c m a x / ( c m i n s ) ,
∑ t ∈ T γ t ≤ 2 K + c max c min ∑ p ≥ 2 K { 2 K χ 0 p κ a l l } 1 / ( 1 − υ ) < ∞ , \sum_{t\in\mathcal{T}}\gamma_t\le2K+\frac{c_{\max}}{c_{\min}}\sum_{p\ge2K}\left\{\frac{2K\chi_0}{p\kappa_{\mathrm{all}}}\right\}^{1/(1-\upsilon)}<\infty, t ∈ T ∑ γ t ≤ 2 K + c m i n c m a x p ≥ 2 K ∑ { p κ all 2 K χ 0 } 1/ ( 1 − υ ) < ∞ ,
where the series converges because 1 / ( 1 − υ ) > 1 1/(1-\upsilon)>1 1/ ( 1 − υ ) > 1 .
Proof of Lemma A3.5
Because 0 ≤ μ k ≤ 1 0\le\mu_k\le1 0 ≤ μ k ≤ 1 , we have ∣ m θ ( a ∣ x ) − m θ ⋆ ( a ∣ x ) ∣ ≤ ∥ θ − θ ⋆ ∥ 1 |m_\theta(a\mid x)-m_{\theta^\star}(a\mid x)|\le\|\theta-\theta^\star\|_1 ∣ m θ ( a ∣ x ) − m θ ⋆ ( a ∣ x ) ∣ ≤ ∥ θ − θ ⋆ ∥ 1 for all ( x , a ) (x,a) ( x , a ) , and so
κ ( θ t ) ≥ κ ( θ ⋆ ) − ∥ θ t − θ ⋆ ∥ 1 ≥ κ ⋆ 2 \kappa(\theta_t)\ge\kappa(\theta^\star)-\|\theta_t-\theta^\star\|_1\ge\frac{\kappa_\star}{2} κ ( θ t ) ≥ κ ( θ ⋆ ) − ∥ θ t − θ ⋆ ∥ 1 ≥ 2 κ ⋆
once ∥ θ t − θ ⋆ ∥ 1 ≤ κ ⋆ / 2 \|\theta_t-\theta^\star\|_1\le\kappa_\star/2 ∥ θ t − θ ⋆ ∥ 1 ≤ κ ⋆ /2 . Since χ t → 0 \chi_t\to0 χ t → 0 , eventually χ t < κ ⋆ / 2 ≤ κ ( θ t ) \chi_t<\kappa_\star/2\le\kappa(\theta_t) χ t < κ ⋆ /2 ≤ κ ( θ t ) , after which no step is forced.
Proof of Lemma A3.6
We first show that h t h_t h t is bounded. Choose t 1 t_1 t 1 beyond which both recursions hold and q t ≤ 1 q_t\le1 q t ≤ 1 for t ∉ T t\notin\mathcal{T} t ∈ / T , and let g t = max { h t , b , 1 } g_t=\max\{h_t,b,1\} g t = max { h t , b , 1 } . For t ≥ t 1 t\ge t_1 t ≥ t 1 outside T \mathcal{T} T , h t + 1 h_{t+1} h t + 1 is at most a weighted average of $h_t and q t q_t q t , and so
g t + 1 ≤ g t ( t ∉ T ) , g t + 1 ≤ ( 1 + 2 γ t ) g t ( t ∈ T ) . g_{t+1} \leq g_t \quad(t \notin\mathcal{T}), \qquad g_{t+1} \leq(1+2\gamma_t)g_t \quad(t \in\mathcal{T}). g t + 1 ≤ g t ( t ∈ / T ) , g t + 1 ≤ ( 1 + 2 γ t ) g t ( t ∈ T ) .
Thus, for t ≥ t 1 t \geq t_1 t ≥ t 1 ,
h t ≤ g t ≤ g t 1 exp ( 2 ∑ s ∈ T γ s ) = H < ∞ . h_t \leq g_t \leq g_{t_1}\exp\left(2\sum_{s\in\mathcal{T}}\gamma_s\right)=H<\infty. h t ≤ g t ≤ g t 1 exp ( 2 s ∈ T ∑ γ s ) = H < ∞.
Given this bound, every step satisfies the first recursion up to a summable error. Writing ( 1 + γ t ) h t = ( 1 − γ t ) h t + 2 γ t h t (1+\gamma_t)h_t=(1-\gamma_t)h_t+2\gamma_t h_t ( 1 + γ t ) h t = ( 1 − γ t ) h t + 2 γ t h t for t ∈ T t\in\mathcal{T} t ∈ T , we have, for all t ≥ t 1 t\geq t_1 t ≥ t 1 ,
h t + 1 ≤ ( 1 − γ t ) h t + γ t q ˉ t + a t , ( q ˉ t , a t ) = { ( q t , 0 ) , t ∉ T , ( 0 , γ t ( 2 H + b ) ) , t ∈ T . h_{t+1}\leq(1-\gamma_t)h_t+\gamma_t\bar q_t+a_t,\qquad
(\bar q_t,a_t)=
\begin{cases}
(q_t,0), & t\notin\mathcal{T},\\
(0,\gamma_t(2H+b)), & t\in\mathcal{T}.
\end{cases} h t + 1 ≤ ( 1 − γ t ) h t + γ t q ˉ t + a t , ( q ˉ t , a t ) = { ( q t , 0 ) , ( 0 , γ t ( 2 H + b )) , t ∈ / T , t ∈ T .
so that q ˉ t → 0 \bar q_t\to0 q ˉ t → 0 and ∑ t a t < ∞ \sum_t a_t<\infty ∑ t a t < ∞ . Fix ϵ > 0 \epsilon>0 ϵ > 0 , and choose t 2 ≥ t 1 t_2\geq t_1 t 2 ≥ t 1 such that q ˉ t ≤ ϵ \bar q_t\leq\epsilon q ˉ t ≤ ϵ for t ≥ t 2 t\geq t_2 t ≥ t 2 and ∑ t ≥ t 2 a t ≤ ϵ \sum_{t\geq t_2}a_t\leq\epsilon ∑ t ≥ t 2 a t ≤ ϵ . Unrolling the recursion from t t t down to t 2 t_2 t 2 , with P i = ∏ s = i t ( 1 − γ s ) P_i=\prod_{s=i}^{t}(1-\gamma_s) P i = ∏ s = i t ( 1 − γ s ) and P t + 1 = 1 P_{t+1}=1 P t + 1 = 1 ,
h t + 1 ≤ P t 2 h t 2 + ∑ i = t 2 t γ i P i + 1 q ˉ i + ∑ i = t 2 t P i + 1 a i . h_{t+1}\leq P_{t_2}h_{t_2}+\sum_{i=t_2}^{t}\gamma_iP_{i+1}\bar q_i+\sum_{i=t_2}^{t}P_{i+1}a_i. h t + 1 ≤ P t 2 h t 2 + i = t 2 ∑ t γ i P i + 1 q ˉ i + i = t 2 ∑ t P i + 1 a i .
The weights γ i P i + 1 = P i + 1 − P i \gamma_iP_{i+1}=P_{i+1}-P_i γ i P i + 1 = P i + 1 − P i sum to 1 − P t 2 ≤ 1 1-P_{t_2}\leq1 1 − P t 2 ≤ 1 , and P i + 1 ≤ 1 P_{i+1}\leq1 P i + 1 ≤ 1 ; thus h t + 1 ≤ P t 2 h t 2 + 2 ϵ h_{t+1}\leq P_{t_2}h_{t_2}+2\epsilon h t + 1 ≤ P t 2 h t 2 + 2 ϵ . Finally, P t 2 ≤ exp ( − ∑ s = t 2 t γ s ) → 0 P_{t_2}\leq\exp(-\sum_{s=t_2}^{t}\gamma_s)\to0 P t 2 ≤ exp ( − ∑ s = t 2 t γ s ) → 0 because ∑ s γ s \sum_s\gamma_s ∑ s γ s diverges, hence lim sup t h t ≤ 2 ϵ \limsup_t h_t\leq2\epsilon lim sup t h t ≤ 2 ϵ for every ϵ > 0 \epsilon>0 ϵ > 0 , that is, h t → 0 h_t\to0 h t → 0 .
Proof of Lemma A3.7
Fix y y y , write p = ω ( y ) p=\omega(y) p = ω ( y ) , and let J ( y ) \mathcal{J}(y) J ( y ) denote the Jacobian of y ↦ ω ( y ) y\mapsto\omega(y) y ↦ ω ( y ) . Differentiating log ω i ( y ) = y i / ρ − log ∑ ℓ exp ( y ℓ / ρ ) \log\omega_i(y)=y_i/\rho-\log\sum_\ell\exp(y_\ell/\rho) log ω i ( y ) = y i / ρ − log ∑ ℓ exp ( y ℓ / ρ ) gives ∂ log ω i / ∂ y j = ( 1 { i = j } − p j ) / ρ \partial\log\omega_i/\partial y_j=(\mathbb{1}\{i=j\}-p_j)/\rho ∂ log ω i / ∂ y j = ( 1 { i = j } − p j ) / ρ , and so
J ( y ) = 1 ρ { diag ( p ) − p p ⊤ } . \mathcal{J}(y)=\frac{1}{\rho}\{\operatorname{diag}(p)-pp^\top\}. J ( y ) = ρ 1 { diag ( p ) − p p ⊤ } .
For any u ∈ R M u\in\mathbb{R}^M u ∈ R M , the i i i th entry of { diag ( p ) − p p ⊤ } u \{\operatorname{diag}(p)-pp^\top\}u { diag ( p ) − p p ⊤ } u is p i ( u i − p ⊤ u ) p_i(u_i-p^\top u) p i ( u i − p ⊤ u ) , and ∣ u i − p ⊤ u ∣ ≤ 2 ∥ u ∥ ∞ |u_i-p^\top u|\leq2\lVert u\rVert_\infty ∣ u i − p ⊤ u ∣ ≤ 2 ∥ u ∥ ∞ because p ⊤ u p^\top u p ⊤ u is a weighted average of the u i u_i u i . Since the p i p_i p i sum to one,
∥ J ( y ) u ∥ 1 = 1 ρ ∑ i = 1 M p i ∣ u i − p ⊤ u ∣ ≤ 2 ρ ∥ u ∥ ∞ , \lVert\mathcal{J}(y)u\rVert_1=\frac{1}{\rho}\sum_{i=1}^{M}p_i|u_i-p^\top u|\leq\frac{2}{\rho}\lVert u\rVert_\infty, ∥ J ( y ) u ∥ 1 = ρ 1 i = 1 ∑ M p i ∣ u i − p ⊤ u ∣ ≤ ρ 2 ∥ u ∥ ∞ ,
with a constant that does not depend on y y y . The map s ↦ ω { y ′ + s ( y − y ′ ) } s\mapsto\omega\{y'+s(y-y')\} s ↦ ω { y ′ + s ( y − y ′ )} is continuously differentiable on [ 0 , 1 ] [0,1] [ 0 , 1 ] , and integrating along the segment from y ′ y' y ′ to y y y gives
∥ ω ( y ) − ω ( y ′ ) ∥ 1 = ∥ ∫ 0 1 J { y ′ + s ( y − y ′ ) } ( y − y ′ ) d s ∥ 1 ≤ 2 ρ ∥ y − y ′ ∥ ∞ . \lVert\omega(y)-\omega(y')\rVert_1
=\left\lVert\int_0^1\mathcal{J}\{y'+s(y-y')\}(y-y')\,ds\right\rVert_1
\leq\frac{2}{\rho}\lVert y-y'\rVert_\infty. ∥ ω ( y ) − ω ( y ′ ) ∥ 1 = ∫ 0 1 J { y ′ + s ( y − y ′ )} ( y − y ′ ) d s 1 ≤ ρ 2 ∥ y − y ′ ∥ ∞ .
Proof of Lemma A3.8
Without the regularization constants η \eta η and η 0 \eta_0 η 0 , G j ( N ) G_j(N) G j ( N ) would equal G j ( N ) G_j(N) G j ( N ) , which is G j ( z ) / C G_j(z)/C G j ( z ) / C by homogeneity of degree − 1 -1 − 1 ; we show that the shifts by η \eta η and η 0 \eta_0 η 0 change the rescaled quantities by O ( C − 1 ) O(C^{-1}) O ( C − 1 ) . Write N = C z N=Cz N = C z , s = 1 ⊤ z s=\mathbf{1}^\top z s = 1 ⊤ z , and u = μ ( X , a ) ⊤ z u=\mu(X,a)^\top z u = μ ( X , a ) ⊤ z , so that T = C s T=Cs T = C s and D N ( X , a ) = η + C u D_N(X,a)=\eta+Cu D N ( X , a ) = η + C u . Throughout, s ≥ c max − 1 s\geq c_{\max}^{-1} s ≥ c m a x − 1 because c ⊤ z = 1 c^\top z=1 c ⊤ z = 1 ; only actions in S S S contribute, and on them u ≥ e u\geq e u ≥ e because z ∈ C ( e ) z\in C(e) z ∈ C ( e ) ; moreover, C ≥ c min C\geq c_{\min} C ≥ c m i n because N ≠ 0 N\neq0 N = 0 , and V j ≤ R max 2 V_j\leq R_{\max}^2 V j ≤ R m a x 2 and $\overline W_j=\mathbb{E}_X\sum_a W_j(X,a)\leq R_{\max}^2 because ∣ R ∣ ≤ R max |R| \le R_{\max} ∣ R ∣ ≤ R m a x . We use two elementary inequalities, valid for s > 0 s > 0 s > 0 and 0 ≤ a ≤ b 0 \le a \le b 0 ≤ a ≤ b :
∣ 1 s + a − 1 s ∣ = a s ( s + a ) ≤ a s 2 , (60) \left|\frac{1}{s+a}-\frac{1}{s}\right|=\frac{a}{s(s+a)}\le\frac{a}{s^2}, \tag*{(60)} s + a 1 − s 1 = s ( s + a ) a ≤ s 2 a , (60)
∣ 1 ( s + a ) ( s + b ) − 1 s 2 ∣ = s ( a + b ) + a b s 2 ( s + a ) ( s + b ) ≤ 2 s b + b 2 s 4 . (61) \left|\frac{1}{(s+a)(s+b)}-\frac{1}{s^2}\right|=\frac{s(a+b)+ab}{s^2(s+a)(s+b)}\le\frac{2sb+b^2}{s^4}. \tag*{(61)} ( s + a ) ( s + b ) 1 − s 2 1 = s 2 ( s + a ) ( s + b ) s ( a + b ) + ab ≤ s 4 2 s b + b 2 . (61)
For the first bound, dividing numerator and denominator by C C C gives
C G j ( N ) − G j ( z ) = V j ( 1 s + η 0 / C − 1 s ) + E X ∑ a W j ( X , a ) ( 1 u + η / C − 1 u ) . CG_j(N)-G_j(z)=V_j\left(\frac{1}{s+\eta_0/C}-\frac{1}{s}\right)+\mathbb{E}_X\sum_a W_j(X,a)\left(\frac{1}{u+\eta/C}-\frac{1}{u}\right). C G j ( N ) − G j ( z ) = V j ( s + η 0 / C 1 − s 1 ) + E X a ∑ W j ( X , a ) ( u + η / C 1 − u 1 ) .
Applying (A6.2) with a = η 0 / C a=\eta_0/C a = η 0 / C to the first term, and with s s s replaced by u u u and a = η / C a=\eta/C a = η / C to the second, the absolute value is at most
V j η 0 C s 2 + E X ∑ a W j ( X , a ) η C u 2 ≤ R max 2 ( η 0 c max 2 + η e − 2 ) C = K G ( e ) C . \frac{V_j\eta_0}{Cs^2}+\mathbb{E}_X\sum_a\frac{W_j(X,a)\eta}{Cu^2}\le\frac{R_{\max}^2(\eta_0c_{\max}^2+\eta e^{-2})}{C}=\frac{K_G(e)}{C}. C s 2 V j η 0 + E X a ∑ C u 2 W j ( X , a ) η ≤ C R m a x 2 ( η 0 c m a x 2 + η e − 2 ) = C K G ( e ) .
For the second bound, differentiating (7) and rescaling (13) , with the arguments ( X , a ) (X,a) ( X , a ) suppressed,
− ∂ k G j ( z ) = V j s 2 + E X ∑ a W j μ k u 2 , -\partial_kG_j(z)=\frac{V_j}{s^2}+\mathbb{E}_X\sum_a\frac{W_j\mu_k}{u^2}, − ∂ k G j ( z ) = s 2 V j + E X a ∑ u 2 W j μ k ,
C 2 Δ j , k ( N ) = V j ( s + η 0 / C ) { s + ( 1 + η 0 ) / C } + E X ∑ a W j μ k ( u + η / C ) { u + ( η + μ k ) / C } . C^2\Delta_{j,k}(N)=\frac{V_j}{(s+\eta_0/C)\{s+(1+\eta_0)/C\}}+\mathbb{E}_X\sum_a\frac{W_j\mu_k}{(u+\eta/C)\{u+(\eta+\mu_k)/C\}}. C 2 Δ j , k ( N ) = ( s + η 0 / C ) { s + ( 1 + η 0 ) / C } V j + E X a ∑ ( u + η / C ) { u + ( η + μ k ) / C } W j μ k .
Applying (A6.3) with a = η 0 / C a=\eta_0/C a = η 0 / C and b = ( 1 + η 0 ) / C b=(1+\eta_0)/C b = ( 1 + η 0 ) / C to the first terms, and using b 2 ≤ ( 1 + η 0 ) 2 / ( c min C ) b^2\le(1+\eta_0)^2/(c_{\min}C) b 2 ≤ ( 1 + η 0 ) 2 / ( c m i n C ) ,
∣ V j ( s + η 0 / C ) { s + ( 1 + η 0 ) / C } − V j s 2 ∣ ≤ V j { 2 ( 1 + η 0 ) s 3 + ( 1 + η 0 ) 2 c min s 4 } 1 C ≤ K 1 C , \left|\frac{V_j}{(s+\eta_0/C)\{s+(1+\eta_0)/C\}}-\frac{V_j}{s^2}\right|\le V_j\left\{\frac{2(1+\eta_0)}{s^3}+\frac{(1+\eta_0)^2}{c_{\min}s^4}\right\}\frac{1}{C}\le\frac{K_1}{C}, ( s + η 0 / C ) { s + ( 1 + η 0 ) / C } V j − s 2 V j ≤ V j { s 3 2 ( 1 + η 0 ) + c m i n s 4 ( 1 + η 0 ) 2 } C 1 ≤ C K 1 ,
with K 1 = R max 2 { 2 ( 1 + η 0 ) c max 3 + ( 1 + η 0 ) 2 c max 4 / c min } K_1=R_{\max}^2\{2(1+\eta_0)c_{\max}^3+(1+\eta_0)^2c_{\max}^4/c_{\min}\} K 1 = R m a x 2 { 2 ( 1 + η 0 ) c m a x 3 + ( 1 + η 0 ) 2 c m a x 4 / c m i n } . Likewise, applying (A6.3) with s s s replaced by u u u , a = η / C a=\eta/C a = η / C , and b = ( η + μ k ) / C ≤ ( 1 + η ) / C b=(\eta+\mu_k)/C\le(1+\eta)/C b = ( η + μ k ) / C ≤ ( 1 + η ) / C to the second terms, and using μ k ≤ 1 \mu_k\le1 μ k ≤ 1 ,
∣ E X ∑ a W j μ k [ 1 ( u + η / C ) { u + ( η + μ k ) / C } − 1 u 2 ] ∣ ≤ E X ∑ a W j { 2 ( 1 + η ) u 3 + ( 1 + η ) 2 c min u 4 } 1 C ≤ K 2 ( e ) C , \begin{aligned}
\left|\mathbb{E}_X\sum_a W_j\mu_k\left[\frac{1}{(u+\eta/C)\{u+(\eta+\mu_k)/C\}}-\frac{1}{u^2}\right]\right|\le\mathbb{E}_X\sum_a W_j\left\{\frac{2(1+\eta)}{u^3}+\frac{(1+\eta)^2}{c_{\min}u^4}\right\}\frac{1}{C} \\
&\le\frac{K_2(e)}{C},
\end{aligned} E X a ∑ W j μ k [ ( u + η / C ) { u + ( η + μ k ) / C } 1 − u 2 1 ] ≤ E X a ∑ W j { u 3 2 ( 1 + η ) + c m i n u 4 ( 1 + η ) 2 } C 1 ≤ C K 2 ( e ) ,
with K 2 ( e ) = R max 2 { 2 ( 1 + η ) e − 3 + ( 1 + η ) 2 e − 4 / c min } K_2(e)=R_{\max}^2\{2(1+\eta)e^{-3}+(1+\eta)^2e^{-4}/c_{\min}\} K 2 ( e ) = R m a x 2 { 2 ( 1 + η ) e − 3 + ( 1 + η ) 2 e − 4 / c m i n } . Adding the two gives the second bound with K Δ ( e ) = K 1 + K 2 ( e ) K_\Delta(e)=K_1+K_2(e) K Δ ( e ) = K 1 + K 2 ( e ) .
Proof of Lemma A3.9
By Lemma A3.3(b) and the definition of the index,
g k = 1 c k ∑ j ω j ( z t ; ρ t ) ∂ k G j ( z t ) , C t 2 S t , k = 1 c k ∑ j ω j , t C t 2 Δ j , k ( N t ) . g_k=\frac{1}{c_k}\sum_j\omega_j(z_t;\rho_t)\partial_kG_j(z_t),\qquad C_t^2S_{t,k}=\frac{1}{c_k}\sum_j\omega_{j,t}C_t^2\Delta_{j,k}(N_t). g k = c k 1 j ∑ ω j ( z t ; ρ t ) ∂ k G j ( z t ) , C t 2 S t , k = c k 1 j ∑ ω j , t C t 2 Δ j , k ( N t ) .
The two have the same form, with the gains in place of the derivatives and with weights computed from C t G j ( N t ) C_tG_j(N_t) C t G j ( N t ) instead of G j ( z t ) G_j(z_t) G j ( z t ) . Thus
∣ C t 2 S t , k + g k ∣ ≤ 1 c min ∑ j ω j , t ∣ C t 2 Δ j , k ( N t ) + ∂ k G j ( z t ) ∣ + 1 c min ∥ ω t − ω ( z t ; ρ t ) ∥ 1 max j ∣ ∂ k G j ( z t ) ∣ . \left|C_t^2S_{t,k}+g_k\right|\leq\frac{1}{c_{\min}}\sum_j\omega_{j,t}\left|C_t^2\Delta_{j,k}(N_t)+\partial_kG_j(z_t)\right|+\frac{1}{c_{\min}}\left\|\omega_t-\omega(z_t;\rho_t)\right\|_1\max_j\left|\partial_kG_j(z_t)\right|. C t 2 S t , k + g k ≤ c m i n 1 j ∑ ω j , t C t 2 Δ j , k ( N t ) + ∂ k G j ( z t ) + c m i n 1 ∥ ω t − ω ( z t ; ρ t ) ∥ 1 j max ∣ ∂ k G j ( z t ) ∣ .
Since the weights sum to one, Lemma A3.8 bounds the first term by K Δ ( e ) / ( c min C t ) K_\Delta(e)/(c_{\min}C_t) K Δ ( e ) / ( c m i n C t ) . For the second, the arguments of the exponential weights differ by at most K G ( e ) / C t K_G(e)/C_t K G ( e ) / C t , again by Lemma A3.8, so Lemma A3.7 gives
∥ ω t − ω ( z t ; ρ t ) ∥ 1 ≤ 2 K G ( e ) ρ t C t = 2 K G ( e ) ρ 0 C t α − 1 . \left\|\omega_t-\omega(z_t;\rho_t)\right\|_1\leq\frac{2K_G(e)}{\rho_tC_t}=\frac{2K_G(e)}{\rho_0}C_t^{\alpha-1}. ∥ ω t − ω ( z t ; ρ t ) ∥ 1 ≤ ρ t C t 2 K G ( e ) = ρ 0 2 K G ( e ) C t α − 1 .
together with ∣ ∂ k G j ( z t ) ∣ ≤ R max 2 ( c max 2 + e − 2 ) \left|\partial_kG_j(z_t)\right|\leq R_{\max}^2(c_{\max}^2+e^{-2}) ∣ ∂ k G j ( z t ) ∣ ≤ R m a x 2 ( c m a x 2 + e − 2 ) , the second term is at most
2 R max 2 ( c max 2 + e − 2 ) K G ( e ) c min ρ 0 C t α − 1 . \frac{2R_{\max}^2(c_{\max}^2+e^{-2})K_G(e)}{c_{\min}\rho_0}C_t^{\alpha-1}. c m i n ρ 0 2 R m a x 2 ( c m a x 2 + e − 2 ) K G ( e ) C t α − 1 .
Finally, because e ≤ 1 / c max e\leq1/c_{\max} e ≤ 1/ c m a x , both K Δ ( e ) K_\Delta(e) K Δ ( e ) and ( c max 2 + e − 2 ) K G ( e ) (c_{\max}^2+e^{-2})K_G(e) ( c m a x 2 + e − 2 ) K G ( e ) are at most constant multiples of e − 4 e^{-4} e − 4 , which gives (A3.11).
Proof of Proposition A3.3
Adding and subtracting c k − 1 ∑ j = 1 M ω ^ j , t Δ j , k ( N t ) c_k^{-1}\sum_{j=1}^M\widehat{\omega}_{j,t}\Delta_{j,k}(N_t) c k − 1 ∑ j = 1 M ω j , t Δ j , k ( N t ) separates the effect of the gains from that of the weights:
S ^ t , k − S t , k = 1 c k ∑ j = 1 M ω ^ j , t { Δ ^ j , k , t − Δ j , k ( N t ) } + 1 c k ∑ j = 1 M ( ω ^ j , t − ω j , t ) Δ j , k ( N t ) . \widehat{S}_{t,k}-S_{t,k}=\frac{1}{c_k}\sum_{j=1}^M\widehat{\omega}_{j,t}\{\widehat{\Delta}_{j,k,t}-\Delta_{j,k}(N_t)\}+\frac{1}{c_k}\sum_{j=1}^M(\widehat{\omega}_{j,t}-\omega_{j,t})\Delta_{j,k}(N_t). S t , k − S t , k = c k 1 j = 1 ∑ M ω j , t { Δ j , k , t − Δ j , k ( N t )} + c k 1 j = 1 ∑ M ( ω j , t − ω j , t ) Δ j , k ( N t ) .
The first term averages the gain errors with weights that sum to one, so it is at most r t / ( c min C t 2 ) r_t/(c_{\min}C_t^2) r t / ( c m i n C t 2 ) by Assumption 4. In the second, the arguments of the exponential weights C t G ^ j , t C_t\widehat{G}_{j,t} C t G j , t and C t G j ( N t ) C_tG_j(N_t) C t G j ( N t ) differ by at most r t r_t r t , again by Assumption 4, and Lemma A3.7 gives ∥ ω ^ t − ω t ∥ 1 ≤ 2 r t / ρ t \|\widehat{\omega}_t-\omega_t\|_1\leq2r_t/\rho_t ∥ ω t − ω t ∥ 1 ≤ 2 r t / ρ t . Meanwhile, (3.8), the bound D N t ≥ t κ ( θ t ) D_{N_t}\geq t\kappa(\theta_t) D N t ≥ t κ ( θ t ) on contributing actions, and C t ≤ c max t C_t\leq c_{\max}t C t ≤ c m a x t give
C t 2 Δ j , k ( N t ) ≤ 2 c max 2 R max 2 κ ( θ t ) − 2 . C_t^2\Delta_{j,k}(N_t)\leq2c_{\max}^2R_{\max}^2\kappa(\theta_t)^{-2}. C t 2 Δ j , k ( N t ) ≤ 2 c m a x 2 R m a x 2 κ ( θ t ) − 2 .
Multiplying by C t 2 C_t^2 C t 2 and adding the two bounds yields (A3.12); note also that r t / ρ t = ρ 0 − 1 C t α r t r_t/\rho_t=\rho_0^{-1}C_t^\alpha r_t r t / ρ t = ρ 0 − 1 C t α r t .
Proof of Lemma A3.10
Let k ^ \widehat{k} k be the purchased logger, which maximizes S ^ t , k \widehat{S}_{t,k} S t , k over all loggers. A linear function on C \mathcal{C} C attains its minimum at a vertex, so the minimum in (4.5) equals min k g k \min_k g_k min k g k , and we need to show that g k ^ ≤ min k g k + δ t g_{\widehat{k}}\leq\min_k g_k+\delta_t g k ≤ min k g k + δ t . Let ε t = max k ∣ C t 2 S ^ t , k + g k ∣ \varepsilon_t=\max_k|C_t^2\widehat{S}_{t,k}+g_k| ε t = max k ∣ C t 2 S t , k + g k ∣ . Inserting S t , k S_{t,k} S t , k and applying Lemma A3.9 with e = κ ( θ t ) / c max e=\kappa(\theta_t)/c_{\max} e = κ ( θ t ) / c m a x ,
ε t ≤ C t 2 max k ∣ S ^ t , k − S t , k ∣ + max k ∣ C t 2 S t , k + g k ∣ ≤ C t 2 max k ∣ S ^ t , k − S t , k ∣ + D 1 c max 4 κ ( θ t ) − 4 ( C t α − 1 + C t − 1 ) . \varepsilon_t\leq C_t^2\max_k|\widehat{S}_{t,k}-S_{t,k}|+\max_k|C_t^2S_{t,k}+g_k|\leq C_t^2\max_k|\widehat{S}_{t,k}-S_{t,k}|+D_1c_{\max}^4\kappa(\theta_t)^{-4}(C_t^{\alpha-1}+C_t^{-1}). ε t ≤ C t 2 k max ∣ S t , k − S t , k ∣ + k max ∣ C t 2 S t , k + g k ∣ ≤ C t 2 k max ∣ S t , k − S t , k ∣ + D 1 c m a x 4 κ ( θ t ) − 4 ( C t α − 1 + C t − 1 ) .
By definition of ε t \varepsilon_t ε t , g ℓ − ε t ≤ − C t 2 S ^ t , ℓ ≤ g ℓ + ε t g_\ell-\varepsilon_t\leq-C_t^2\widehat{S}_{t,\ell}\leq g_\ell+\varepsilon_t g ℓ − ε t ≤ − C t 2 S t , ℓ ≤ g ℓ + ε t for every logger ℓ \ell ℓ . Since k ^ \widehat{k} k maximizes the estimated index, it follows that, for any logger k k k ,
g k ^ ≤ − C t 2 S ^ t , k ^ + ε t ≤ − C t 2 S ^ t , k + ε t ≤ g k + 2 ε t . g_{\widehat{k}}\leq-C_t^2\widehat{S}_{t,\widehat{k}}+\varepsilon_t\leq-C_t^2\widehat{S}_{t,k}+\varepsilon_t\leq g_k+2\varepsilon_t. g k ≤ − C t 2 S t , k + ε t ≤ − C t 2 S t , k + ε t ≤ g k + 2 ε t .
The purchase is therefore a δ t \delta_t δ t -approximate conditional-gradient step with δ t = 2 ε t \delta_t=2\varepsilon_t δ t = 2 ε t , and the bound on ε t \varepsilon_t ε t gives the first claim. Under the additional conditions, C t ≤ c max t C_t\leq c_{\max}t C t ≤ c m a x t gives r t ≤ R c max β r C t − β r r_t\leq Rc_{\max}^{\beta_r}C_t^{-\beta_r} r t ≤ R c m a x β r C t − β r , and Proposition A3.3 bounds the estimation term by D 0 ( r t + ρ 0 − 1 m − 2 C t α r t ) ≤ D 0 R c max β r ( 1 + D_0(r_t+\rho_0^{-1}m^{-2}C_t^\alpha r_t)\leq D_0Rc_{\max}^{\beta_r}(1+ D 0 ( r t + ρ 0 − 1 m − 2 C t α r t ) ≤ D 0 R c m a x β r ( 1 + ρ 0 − 1 m − 2 C t α − β r \rho_0^{-1}m^{-2}C_t^{\alpha-\beta_r} ρ 0 − 1 m − 2 C t α − β r once C t ≥ 1 C_t \ge1 C t ≥ 1 ; since also C t − 1 ≤ C t α − 1 C_t^{-1} \le C_t^{\alpha-1} C t − 1 ≤ C t α − 1 , the last claim follows.
Proof of Lemma A3.1
Before τ B \tau_B τ B , at least c max c_{\max} c m a x of the budget remains, so every logger is affordable in both sequences. The sequences then share counts, estimates, indices, and tie-breaking, so they make the same choices; by induction, they agree through τ B \tau_B τ B . Afterwards, less than c max c_{\max} c m a x remains and each purchase costs at least c min c_{\min} c m i n , so the budget-B B B sequence makes at most L 0 L_0 L 0 further purchases. At the common time T B T_B T B both count vectors total T B T_B T B , hence each sequence makes T B − τ B ≤ L 0 T_B-\tau_B\le L_0 T B − τ B ≤ L 0 purchases after τ B \tau_B τ B . Write a a a and b b b for the counts that the budget-B B B and the infinite-budget sequences add, so that ∥ a ∥ 1 = ∥ b ∥ 1 ≤ L 0 \lVert a\rVert_1=\lVert b\rVert_1\le L_0 ∥ a ∥ 1 = ∥ b ∥ 1 ≤ L 0 and both costs c ⊤ a c^\top a c ⊤ a and c ⊤ b c^\top b c ⊤ b lie in [ 0 , L 0 c max ] [0,L_0c_{\max}] [ 0 , L 0 c m a x ] . Then
∥ N T B − N ˉ T B ∥ 1 = ∥ a − b ∥ 1 ≤ 2 L 0 , ∣ C T B − C ˉ T B ∣ = ∣ c ⊤ a − c ⊤ b ∣ ≤ L 0 c max . \lVert N_{T_B}-\bar N_{T_B}\rVert_1=\lVert a-b\rVert_1\le2L_0,\qquad|C_{T_B}-\bar C_{T_B}|=|c^\top a-c^\top b|\le L_0c_{\max}. ∥ N T B − N ˉ T B ∥ 1 = ∥ a − b ∥ 1 ≤ 2 L 0 , ∣ C T B − C ˉ T B ∣ = ∣ c ⊤ a − c ⊤ b ∣ ≤ L 0 c m a x .
For the normalized designs, note that the budget-B B B sequence stops only when no logger is affordable, so that C T B > B − c max ≥ B / 2 C_{T_B}>B-c_{\max}\ge B/2 C T B > B − c m a x ≥ B /2 for B ≥ 2 c max B\ge2c_{\max} B ≥ 2 c m a x . Adding and subtracting N ˉ T B / C T B \bar N_{T_B}/C_{T_B} N ˉ T B / C T B ,
N T B C T B − N ˉ T B C ˉ T B = N T B − N ˉ T B C T B + N ˉ T B C ˉ T B C ˉ T B − C T B C T B . \frac{N_{T_B}}{C_{T_B}}-\frac{\bar N_{T_B}}{\bar C_{T_B}}=\frac{N_{T_B}-\bar N_{T_B}}{C_{T_B}}+\frac{\bar N_{T_B}}{\bar C_{T_B}}\frac{\bar C_{T_B}-C_{T_B}}{C_{T_B}}. C T B N T B − C ˉ T B N ˉ T B = C T B N T B − N ˉ T B + C ˉ T B N ˉ T B C T B C ˉ T B − C T B .
The first term has ℓ 1 \ell_1 ℓ 1 norm at most 2 L 0 / C T B ≤ 4 L 0 / B 2L_0/C_{T_B}\le4L_0/B 2 L 0 / C T B ≤ 4 L 0 / B . In the second, ∥ N ˉ T B ∥ 1 / C ˉ T B ≤ c min − 1 \lVert\bar N_{T_B}\rVert_1/\bar C_{T_B}\le c_{\min}^{-1} ∥ N ˉ T B ∥ 1 / C ˉ T B ≤ c m i n − 1 , because each purchase costs at least c min c_{\min} c m i n , and ∣ C ˉ T B − C T B ∣ / C T B ≤ 2 L 0 c max / B |\bar C_{T_B}-C_{T_B}|/C_{T_B}\le2L_0c_{\max}/B ∣ C ˉ T B − C T B ∣/ C T B ≤ 2 L 0 c m a x / B , so its norm is at most 2 L 0 c max / ( c min B ) 2L_0c_{\max}/(c_{\min}B) 2 L 0 c m a x / ( c m i n B ) ; since the ℓ 2 \ell_2 ℓ 2 norm is at most the ℓ 1 \ell_1 ℓ 1 norm, the first claim holds with K z = 2 L 0 ( 2 + c max / c min ) K_z=2L_0(2+c_{\max}/c_{\min}) K z = 2 L 0 ( 2 + c m a x / c m i n ) . As for the sample fractions, θ ^ B − N ˉ T B / T B = ( N T B − N ˉ T B ) / T B \widehat\theta_B-\bar N_{T_B}/T_B=(N_{T_B}-\bar N_{T_B})/T_B θ B − N ˉ T B / T B = ( N T B − N ˉ T B ) / T B has ℓ 1 \ell_1 ℓ 1 norm at most 2 L 0 / T B 2L_0/T_B 2 L 0 / T B , and T B ≥ C T B / c max ≥ B / ( 2 c max ) T_B\ge C_{T_B}/c_{\max}\ge B/(2c_{\max}) T B ≥ C T B / c m a x ≥ B / ( 2 c m a x ) because C T B ≤ c max T B C_{T_B}\le c_{\max}T_B C T B ≤ c m a x T B ; this gives the second claim with K θ = 4 L 0 c max K_\theta=4L_0c_{\max} K θ = 4 L 0 c m a x .
Proof of Lemma A3.11
By (A3.14) and (6) ,
G ^ j , t − G j ( N t ) = V ^ j , t − V j t + η 0 + ( P ^ t − P X ) ∑ a W ^ j , t ( ⋅ , a ) D N t ( ⋅ , a ) + P X ∑ a W ^ j , t ( ⋅ , a ) − W j ( ⋅ , a ) D N t ( ⋅ , a ) . \widehat G_{j,t}-G_j(N_t)=\frac{\widehat V_{j,t}-V_j}{t+\eta_0}+(\widehat P_t-P_X)\sum_a\frac{\widehat W_{j,t}(\cdot,a)}{D_{N_t}(\cdot,a)}+P_X\sum_a\frac{\widehat W_{j,t}(\cdot,a)-W_j(\cdot,a)}{D_{N_t}(\cdot,a)}. G j , t − G j ( N t ) = t + η 0 V j , t − V j + ( P t − P X ) a ∑ D N t ( ⋅ , a ) W j , t ( ⋅ , a ) + P X a ∑ D N t ( ⋅ , a ) W j , t ( ⋅ , a ) − W j ( ⋅ , a ) .
and (A3.15) and (13) give the same decomposition for Δ ^ j , k , t − Δ j , k ( N t ) \widehat\Delta_{j,k,t}-\Delta_{j,k}(N_t) Δ j , k , t − Δ j , k ( N t ) , with 1 / D N t 1/D_{N_t} 1/ D N t replaced by μ k / { D N t ( D N t + μ k ) } \mu_k/\{D_{N_t}(D_{N_t}+\mu_k)\} μ k / { D N t ( D N t + μ k )} and 1 / ( t + η 0 ) 1/(t+\eta_0) 1/ ( t + η 0 ) by 1 / { ( t + η 0 ) ( t + 1 + η 0 ) } 1/\{(t+\eta_0)(t+1+\eta_0)\} 1/ {( t + η 0 ) ( t + 1 + η 0 )} . On contributing actions, D N t ≥ t κ ( θ t ) ≥ t m t D_{N_t}\ge t\kappa(\theta_t)\ge tm_t D N t ≥ t κ ( θ t ) ≥ t m t , and so
t D N t ≤ m t − 1 , t 2 μ k D N t ( D N t + μ k ) ≤ m t − 2 . \frac{t}{D_{N_t}}\le m_t^{-1},\qquad\frac{t^2\mu_k}{D_{N_t}(D_{N_t}+\mu_k)}\le m_t^{-2}. D N t t ≤ m t − 1 , D N t ( D N t + μ k ) t 2 μ k ≤ m t − 2 .
while the factors multiplying V ^ j , t − V j \widehat V_{j,t}-V_j V j , t − V j , after rescaling by t t t and t 2 t^2 t 2 , are at most one. Hence
t max j ∣ G ^ j , t − G j ( N t ) ∣ ≤ r V , t + r P , t + m t − 1 r W , t , t 2 max j , k ∣ Δ ^ j , k , t − Δ j , k ( N t ) ∣ ≤ r V , t + r P , t + m t − 2 r W , t . t\max_j|\widehat G_{j,t}-G_j(N_t)|\le r_{V,t}+r_{P,t}+m_t^{-1}r_{W,t},\qquad t^2\max_{j,k}|\widehat\Delta_{j,k,t}-\Delta_{j,k}(N_t)|\le r_{V,t}+r_{P,t}+m_t^{-2}r_{W,t}. t j max ∣ G j , t − G j ( N t ) ∣ ≤ r V , t + r P , t + m t − 1 r W , t , t 2 j , k max ∣ Δ j , k , t − Δ j , k ( N t ) ∣ ≤ r V , t + r P , t + m t − 2 r W , t .
Since C t ≤ c max t C_t\le c_{\max}t C t ≤ c m a x t and m t ≤ 1 m_t\le1 m t ≤ 1 , the two bounds of Assumption 4 hold with K ′ a t K'a_t K ′ a t in place of r t r_t r t , where K ′ = max { c max , c max 2 } K'=\max\{c_{\max},c_{\max}^2\} K ′ = max { c m a x , c m a x 2 } . Finally, a t ≤ sup s ≥ t a s a_t\le\sup_{s\ge t}a_s a t ≤ sup s ≥ t a s , and the right side is nonincreasing and tends to zero when a t → 0 a_t\to0 a t → 0 ; hence Assumption 4 holds with r t = K ′ sup s ≥ t a s r_t=K'\sup_{s\ge t}a_s r t = K ′ sup s ≥ t a s .
Proof of Corollary A3.1
We verify the conditions of Lemma A3.11, with rates that depend on a lower bound m t m_t m t for the pooled coverage, chosen below. Every bound rests on one inequality: if ( ξ i ) (\xi_i) ( ξ i ) are martingale difference sequence with ∣ ξ i ∣ ≤ b |\xi_i|\le b ∣ ξ i ∣ ≤ b , the Azuma–Hoeffding inequality gives $\mathbb{P}(|\sum_{i\le t}\xi_i|>b\sqrt{6t\log t})\le 2 t − 3 2t^{-3} 2 t − 3 , which is summable in t t t , thus, by the Borel–Cantelli lemma,
∣ ∑ i ≤ t ξ i ∣ ≤ b 6 t log t (62) \left|\sum_{i\le t}\xi_i\right|\le b\sqrt{6t\log t} \tag*{(62)} i ≤ t ∑ ξ i ≤ b 6 t log t (62)
almost surely for all sufficiently large t t t .
Fix a target-relevant pair ( x , a ) (x,a) ( x , a ) , meaning P X ( x ) > 0 P_X(x)>0 P X ( x ) > 0 and ( x , a ) ∈ S (x,a)\in\mathcal{S} ( x , a ) ∈ S , and let ν i = 1 { X i = x , A i = a } \nu_i=\mathbf{1}\{X_i=x,A_i=a\} ν i = 1 { X i = x , A i = a } and n t = ∑ i ≤ t ν i n_t=\sum_{i\le t}\nu_i n t = ∑ i ≤ t ν i . Given F i − 1 \mathcal{F}_{i-1} F i − 1 , the logger K i K_i K i is fixed, X i ∼ P X X_i\sim P_X X i ∼ P X , and A i ∼ μ K i ( ⋅ ∣ X i ) A_i\sim\mu_{K_i}(\cdot\mid X_i) A i ∼ μ K i ( ⋅ ∣ X i ) ; thus E ( ν i ∣ F i − 1 ) = P X ( x ) μ K i ( a ∣ x ) \mathbb{E}(\nu_i\mid\mathcal{F}_{i-1})=P_X(x)\mu_{K_i}(a\mid x) E ( ν i ∣ F i − 1 ) = P X ( x ) μ K i ( a ∣ x ) and
∑ i ≤ t E ( ν i ∣ F i − 1 ) = P X ( x ) t m θ t ( a ∣ x ) ≥ P X ( x ) t m t . \sum_{i\le t}\mathbb{E}(\nu_i\mid\mathcal{F}_{i-1})=P_X(x)t m_{\theta_t}(a\mid x)\ge P_X(x)t m_t. i ≤ t ∑ E ( ν i ∣ F i − 1 ) = P X ( x ) t m θ t ( a ∣ x ) ≥ P X ( x ) t m t .
Here m t = χ t / 2 m_t=\chi_t/2 m t = χ t /2 is a lower bound for κ ( θ t ) \kappa(\theta_t) κ ( θ t ) by Lemma A3.4(a); if θ t → θ ⋆ \theta_t\to\theta^\star θ t → θ ⋆ , we may also take m t = κ ⋆ / 2 m_t=\kappa_\star/2 m t = κ ⋆ /2 for all large t t t , by Lemma A3.5. Writing p t = P X ( x ) m t p_t=P_X(x)m_t p t = P X ( x ) m t , (A6.4) applied to ν i − E ( ν i ∣ F i − 1 ) \nu_i-\mathbb{E}(\nu_i\mid\mathcal{F}_{i-1}) ν i − E ( ν i ∣ F i − 1 ) gives, for all large t t t ,
n t ≥ p t t − 6 t log t ≥ p t t 2 . n_t\ge p_t t-\sqrt{6t\log t}\ge\frac{p_t t}{2}. n t ≥ p t t − 6 t log t ≥ 2 p t t .
because p t t / log t → ∞ p_t\sqrt{t/\log t}\to\infty p t t / log t → ∞ when υ < 1 / 2 \upsilon<1/2 υ < 1/2 .
Next, with ε i = R i − q ( x , a ) \varepsilon_i=R_i-q(x,a) ε i = R i − q ( x , a ) , the variables ν i ε i \nu_i\varepsilon_i ν i ε i and ν i { ε i 2 − σ 2 ( x , a ) } \nu_i\{\varepsilon_i^2-\sigma^2(x,a)\} ν i { ε i 2 − σ 2 ( x , a )} are martingale differences bounded by 2 R max 2R_{\max} 2 R m a x and 4 R max 2 4R_{\max}^2 4 R m a x 2 , since E ( ε i ∣ F i − 1 , X i , A i ) = 0 \mathbb{E}(\varepsilon_i\mid\mathcal{F}_{i-1},X_i,A_i)=0 E ( ε i ∣ F i − 1 , X i , A i ) = 0 and E ( ε i 2 ∣ F i − 1 , X i , A i ) = σ 2 ( X i , A i ) \mathbb{E}(\varepsilon_i^2\mid\mathcal{F}_{i-1},X_i,A_i)=\sigma^2(X_i,A_i) E ( ε i 2 ∣ F i − 1 , X i , A i ) = σ 2 ( X i , A i ) on { ν i = 1 } \{\nu_i=1\} { ν i = 1 } . By (A6.4) and the bound on n t n_t n t ,
∣ q ^ t ( x , a ) − q ( x , a ) ∣ = ∣ ∑ i ≤ t ν i ε i ∣ n t ≤ 4 R max p t 6 log t t . |\widehat{q}_t(x,a)-q(x,a)|=\frac{\left|\sum_{i\le t}\nu_i\varepsilon_i\right|}{n_t}\le\frac{4R_{\max}}{p_t}\sqrt{\frac{6\log t}{t}}. ∣ q t ( x , a ) − q ( x , a ) ∣ = n t ∑ i ≤ t ν i ε i ≤ p t 4 R m a x t 6 log t .
For the variance, write
σ ^ t 2 ( x , a ) = n t n t − 1 { 1 n t ∑ i ≤ t ν i ε i 2 − ( q ^ t − q ) 2 ( x , a ) } . \widehat{\sigma}_t^2(x,a)=\frac{n_t}{n_t-1}\left\{\frac{1}{n_t}\sum_{i\le t}\nu_i\varepsilon_i^2-(\widehat{q}_t-q)^2(x,a)\right\}. σ t 2 ( x , a ) = n t − 1 n t { n t 1 i ≤ t ∑ ν i ε i 2 − ( q t − q ) 2 ( x , a ) } .
By (A6.4), the average differs from σ 2 ( x , a ) \sigma^2(x,a) σ 2 ( x , a ) by O ( p t − 1 log t / t ) O(p_t^{-1}\sqrt{\log t/t}) O ( p t − 1 log t / t ) . The squared term is O ( p t − 2 log t / t ) O(p_t^{-2}\log t/t) O ( p t − 2 log t / t ) and n t / ( n t − 1 ) = 1 + O { ( p t t ) − 1 } n_t/(n_t-1)=1+O\{(p_t t)^{-1}\} n t / ( n t − 1 ) = 1 + O {( p t t ) − 1 } , both of smaller order because p t t / log t → ∞ p_t\sqrt{t/\log t}\to\infty p t t / log t → ∞ ; hence ∣ σ ^ t 2 − σ 2 ∣ ( x , a ) = O ( p t − 1 log t / t ) |\widehat{\sigma}_t^2-\sigma^2|(x,a)=O(p_t^{-1}\sqrt{\log t/t}) ∣ σ t 2 − σ 2 ∣ ( x , a ) = O ( p t − 1 log t / t ) . Truncation to [ 0 , R max 2 ] [0,R_{\max}^2] [ 0 , R m a x 2 ] cannot increase this error, since σ 2 ( x , a ) ∈ [ 0 , R max 2 ] \sigma^2(x,a)\in[0,R_{\max}^2] σ 2 ( x , a ) ∈ [ 0 , R m a x 2 ] , and because X \mathcal{X} X and A \mathcal{A} A are finite, these bounds hold simultaneously over all target-relevant pairs. Applying (A6.4) also to 1 { X i = x } − P X ( x ) \mathbf{1}\{X_i=x\}-P_X(x) 1 { X i = x } − P X ( x ) , a bounded martingale difference since X i X_i X i is independent of F i − 1 \mathcal{F}_{i-1} F i − 1 , gives max x ∣ P ^ t ( { x } ) − P X ( x ) ∣ ≤ 6 log t / t \max_x|\widehat{P}_t(\{x\})-P_X(x)|\le\sqrt{6\log t/t} max x ∣ P t ({ x }) − P X ( x ) ∣ ≤ 6 log t / t , and so
∣ ( P ^ t − P X ) f ∣ ≤ ∣ X ∣ sup ∣ f ∣ 6 log t t |(\widehat{P}_t-P_X)f|\le|\mathcal{X}|\sup|f|\sqrt{\frac{6\log t}{t}} ∣ ( P t − P X ) f ∣ ≤ ∣ X ∣ sup ∣ f ∣ t 6 log t
for every bounded f f f .
We can now check the three conditions; only target-relevant pairs enter W j W_j W j and q j q_j q j , because π j ( a ∣ x ) = 0 \pi_j(a\mid x)=0 π j ( a ∣ x ) = 0 elsewhere. First,
∑ a ∥ W ^ j , t ( ⋅ , a ) − W j ( ⋅ , a ) ∥ L 1 ( P X ) = ∑ x , a P X ( x ) π j ( a ∣ x ) ∣ σ ^ t 2 − σ 2 ∣ ( x , a ) = O ( m t − 1 log t t ) . \sum_a\|\widehat{W}_{j,t}(\cdot,a)-W_j(\cdot,a)\|_{L_1(P_X)}
=\sum_{x,a}P_X(x)\pi_j(a\mid x)|\widehat{\sigma}_t^2-\sigma^2|(x,a)
=O\left(m_t^{-1}\sqrt{\frac{\log t}{t}}\right). a ∑ ∥ W j , t ( ⋅ , a ) − W j ( ⋅ , a ) ∥ L 1 ( P X ) = x , a ∑ P X ( x ) π j ( a ∣ x ) ∣ σ t 2 − σ 2 ∣ ( x , a ) = O ( m t − 1 t log t ) .
Second, ∣ q ^ j , t ( x ) − q j ( x ) ∣ ≤ max a ∣ q ^ t − q ∣ ( x , a ) |\widehat{q}_{j,t}(x)-q_j(x)|\le\max_a|\widehat{q}_t-q|(x,a) ∣ q j , t ( x ) − q j ( x ) ∣ ≤ max a ∣ q t − q ∣ ( x , a ) over target-relevant a a a , and q ^ j , t \widehat{q}_{j,t} q j , t and q j q_j q j are bounded by R max R_{\max} R m a x , so that
∣ V ^ j , t − V j ∣ ≤ ∣ P ^ t ( q ^ j , t 2 − q j 2 ) ∣ + ∣ ( P ^ t − P X ) q j 2 ∣ + 2 R max ∣ P ^ t ( q ^ j , t − q j ) ∣ + ∣ ( P ^ t − P X ) q j ∣ = O ( m t − 1 log t t ) . |\widehat{V}_{j,t}-V_j|\le|\,\widehat{P}_t(\widehat{q}_{j,t}^2-q_j^2)\,|+|(\widehat{P}_t-P_X)q_j^2|+2R_{\max}|\,\widehat{P}_t(\widehat{q}_{j,t}-q_j)\,|+|(\widehat{P}_t-P_X)q_j|
=O\left(m_t^{-1}\sqrt{\frac{\log t}{t}}\right). ∣ V j , t − V j ∣ ≤ ∣ P t ( q j , t 2 − q j 2 ) ∣ + ∣ ( P t − P X ) q j 2 ∣ + 2 R m a x ∣ P t ( q j , t − q j ) ∣ + ∣ ( P t − P X ) q j ∣ = O ( m t − 1 t log t ) .
Third, t / D N t ≤ m t − 1 t/D_{N_t}\le m_t^{-1} t / D N t ≤ m t − 1 on target-relevant actions, and $0\le\widehat{W}_{j,t}\le R_{\max}^2$ vanishes elsewhere. The two functions in the last condition of Lemma A3.11 are therefore bounded by ∣ A ∣ R max 2 m t − 1 |\mathcal{A}|R_{\max}^2m_t^{-1} ∣ A ∣ R m a x 2 m t − 1 and ∣ A ∣ R max 2 m t − 2 |\mathcal{A}|R_{\max}^2m_t^{-2} ∣ A ∣ R m a x 2 m t − 2 , and the bound on P ^ t − P X \widehat{P}_t-P_X P t − P X gives r P , t = O ( m t − 2 log t / t ) r_{P,t}=O(m_t^{-2}\sqrt{\log t/t}) r P , t = O ( m t − 2 log t / t ) . Lemma A3.11 now gives Assumption 4 with r t = O ( m t − 3 log t / t ) r_t=O(m_t^{-3}\sqrt{\log t/t}) r t = O ( m t − 3 log t / t ) , which is O ( t 3 υ log t / t ) O(t^{3\upsilon}\sqrt{\log t/t}) O ( t 3 υ log t / t ) when m t = χ t / 2 m_t=\chi_t/2 m t = χ t /2 and $O(\sqrt{\log t/t}) when m t = κ ⋆ / 2 m_t = \kappa_\star/2 m t = κ ⋆ /2 .
Proof of Lemma A4.3
Because m θ m_\theta m θ is linear in θ \theta θ and the score depends on θ \theta θ only through 1 / m θ 1/m_\theta 1/ m θ , we expand 1 / m θ 1/m_\theta 1/ m θ around θ ⋆ \theta^\star θ ⋆ . Write δ = θ ⋆ − θ \delta= \theta^\star- \theta δ = θ ⋆ − θ and ϵ i = R i − q ( X i , A i ) \epsilon_i = R_i - q(X_i, A_i) ϵ i = R i − q ( X i , A i ) , and suppose ∥ δ ∥ 2 ≤ r \|\delta\|_2 \le r ∥ δ ∥ 2 ≤ r , so that ∣ δ k ∣ ≤ r |\delta_k| \le r ∣ δ k ∣ ≤ r for every k k k . On contributing actions, m θ ⋆ ≥ κ ⋆ m_{\theta^\star} \ge\kappa_\star m θ ⋆ ≥ κ ⋆ and
∣ m θ ⋆ − m θ ∣ = ∣ ∑ k δ k μ k ∣ ≤ K r ≤ κ ⋆ 2 . \left|m_{\theta^\star} - m_\theta\right| = \left|\sum_k \delta_k\mu_k\right| \le Kr \le\frac{\kappa_\star}{2}. ∣ m θ ⋆ − m θ ∣ = k ∑ δ k μ k ≤ K r ≤ 2 κ ⋆ .
so the geometric series converges and
1 m θ − 1 m θ ⋆ = ∑ p ≥ 1 ( ∑ k δ k μ k ) p m θ ⋆ p + 1 = ∑ p ≥ 1 ∑ ∣ α ∣ = p ( p α ) δ α μ α m θ ⋆ p + 1 , \frac{1}{m_\theta} - \frac{1}{m_{\theta^\star}} = \sum_{p\ge1}\frac{\left(\sum_k\delta_k\mu_k\right)^p}{m_{\theta^\star}^{p+1}} = \sum_{p\ge1}\sum_{|\alpha|=p}\binom{p}{\alpha}\frac{\delta^\alpha\mu^\alpha}{m_{\theta^\star}^{p+1}}, m θ 1 − m θ ⋆ 1 = p ≥ 1 ∑ m θ ⋆ p + 1 ( ∑ k δ k μ k ) p = p ≥ 1 ∑ ∣ α ∣ = p ∑ ( α p ) m θ ⋆ p + 1 δ α μ α ,
where α \alpha α ranges over multi-indices in Z + K \mathbb{Z}_+^K Z + K , δ α = ∏ k δ k α k \delta^\alpha= \prod_k\delta_k^{\alpha_k} δ α = ∏ k δ k α k , and μ α = ∏ k μ k α k \mu^\alpha= \prod_k\mu_k^{\alpha_k} μ α = ∏ k μ k α k . Substituting into (A4.4), and exchanging the finite sum over i i i with the absolutely convergent series,
Ξ ‾ B , j ( θ ) = ∑ p ≥ 1 ∑ ∣ α ∣ = p ( p α ) δ α G B , j , α , G B , j , α = 1 B ∑ i = 1 n B 1 { i ≤ T B } π j ( A i ∣ X i ) μ ( A i ∣ X i ) α m θ ⋆ ( A i ∣ X i ) p + 1 ϵ i . \overline{\Xi}_{B,j}(\theta) = \sum_{p\ge1}\sum_{|\alpha|=p}\binom{p}{\alpha}\delta^\alpha G_{B,j,\alpha}, \qquad G_{B,j,\alpha} = \frac{1}{\sqrt{B}}\sum_{i=1}^{n_B}\mathbf{1}\{i\le T_B\}\frac{\pi_j(A_i\mid X_i)\mu(A_i\mid X_i)^\alpha}{m_{\theta^\star}(A_i\mid X_i)^{p+1}}\epsilon_i. Ξ B , j ( θ ) = p ≥ 1 ∑ ∣ α ∣ = p ∑ ( α p ) δ α G B , j , α , G B , j , α = B 1 i = 1 ∑ n B 1 { i ≤ T B } m θ ⋆ ( A i ∣ X i ) p + 1 π j ( A i ∣ X i ) μ ( A i ∣ X i ) α ϵ i .
The variables G B , j , α G_{B,j,\alpha} G B , j , α do not depend on θ \theta θ , and each is a sum of martingale differences: 1 { i ≤ T B } \mathbf{1}\{i\le T_B\} 1 { i ≤ T B } is F i − 1 \mathcal{F}_{i-1} F i − 1 -measurable, the factors other than ϵ i \epsilon_i ϵ i are functions of ( X i , A i ) (X_i,A_i) ( X i , A i ) , and E ( ϵ i ∣ F i − 1 , X i , A i ) = 0 \mathbb{E}(\epsilon_i\mid\mathcal{F}_{i-1},X_i,A_i)=0 E ( ϵ i ∣ F i − 1 , X i , A i ) = 0 . Since π j ≤ 1 \pi_j\le1 π j ≤ 1 , μ α ≤ 1 \mu^\alpha\le1 μ α ≤ 1 , ∣ ϵ i ∣ ≤ 2 R max |\epsilon_i|\le2R_{\max} ∣ ϵ i ∣ ≤ 2 R m a x , and m θ ⋆ ≥ κ ⋆ m_{\theta^\star}\ge\kappa_\star m θ ⋆ ≥ κ ⋆ , each summand, without the factor B − 1 / 2 B^{-1/2} B − 1/2 , is at most 2 R max κ ⋆ − ( p + 1 ) 2R_{\max}\kappa_\star^{-(p+1)} 2 R m a x κ ⋆ − ( p + 1 ) . Martingale differences are uncorrelated, so with n B ≤ B / c min + 1 n_B\le B/c_{\min}+1 n B ≤ B / c m i n + 1 ,
E ∣ G B , j , α ∣ ≤ { E ( G B , j , α 2 ) } 1 / 2 ≤ ( n B B ) 1 / 2 2 R max κ ⋆ − ( p + 1 ) ≤ c κ ⋆ − ( p + 1 ) , c = 2 R max ( 1 + c min − 1 ) 1 / 2 . \mathbb{E}|G_{B,j,\alpha}| \le\{\mathbb{E}(G_{B,j,\alpha}^2)\}^{1/2} \le\left(\frac{n_B}{B}\right)^{1/2}2R_{\max}\kappa_\star^{-(p+1)} \le c\kappa_\star^{-(p+1)}, \qquad c=2R_{\max}(1+c_{\min}^{-1})^{1/2}. E ∣ G B , j , α ∣ ≤ { E ( G B , j , α 2 ) } 1/2 ≤ ( B n B ) 1/2 2 R m a x κ ⋆ − ( p + 1 ) ≤ c κ ⋆ − ( p + 1 ) , c = 2 R m a x ( 1 + c m i n − 1 ) 1/2 .
Since ∣ δ α ∣ ≤ r p |\delta^\alpha|\le r^p ∣ δ α ∣ ≤ r p and ∑ ∣ α ∣ = p ( p α ) = K p \sum_{|\alpha|=p}\binom{p}{\alpha}=K^p ∑ ∣ α ∣ = p ( α p ) = K p , it follows that
E sup ∥ θ − θ ⋆ ∥ 2 ≤ r ∣ Ξ ‾ B , j ( θ ) ∣ ≤ ∑ p ≥ 1 r p ∑ ∣ α ∣ = p ( p α ) E ∣ G B , j , α ∣ ≤ c κ ⋆ ∑ p ≥ 1 ( K r κ ⋆ ) p ≤ 2 c K κ ⋆ 2 r , \mathbb{E}\sup_{\|\theta-\theta^\star\|_2\le r}|\overline{\Xi}_{B,j}(\theta)| \le\sum_{p\ge1}r^p\sum_{|\alpha|=p}\binom{p}{\alpha}\mathbb{E}|G_{B,j,\alpha}| \le\frac{c}{\kappa_\star}\sum_{p\ge1}\left(\frac{Kr}{\kappa_\star}\right)^p \le\frac{2cK}{\kappa_\star^2}r, E ∥ θ − θ ⋆ ∥ 2 ≤ r sup ∣ Ξ B , j ( θ ) ∣ ≤ p ≥ 1 ∑ r p ∣ α ∣ = p ∑ ( α p ) E ∣ G B , j , α ∣ ≤ κ ⋆ c p ≥ 1 ∑ ( κ ⋆ K r ) p ≤ κ ⋆ 2 2 cK r ,
where the last step uses K r / κ ⋆ < 1 / 2 Kr/\kappa_\star<1/2 K r / κ ⋆ < 1/2 . Summing over the M M M coordinates, with ∥ ⋅ ∥ 2 ≤ ∑ j ∣ ⋅ j ∣ \|\cdot\|_2\le\sum_j|\cdot_j| ∥ ⋅ ∥ 2 ≤ ∑ j ∣ ⋅ j ∣ , gives (A4.5) with c 6 = 2 c K M κ ⋆ − 2 c_6=2cKM\kappa_\star^{-2} c 6 = 2 cK M κ ⋆ − 2 .
Additional Experimental Details and Results
In this appendix, we give the implementation details, the complete data-generating mechanisms, and the additional experiments referred to in Section 5. Unless stated otherwise, all policy values are estimated by the final estimator (17) .
Implementation
All experiments were implemented in Python 3.11.11 with NumPy 2.4.6, SciPy 1.17.1, pandas 3.0.5, scikit-learn 1.9.0, matplotlib 3.11.1, and PyYAML 6.0.3. The continuous optimal design is computed with SciPy’s SLSQP solver, with maxiter = 500 and ftol = 10^{-12}. The static planners solve an epigraph form of the continuous relaxation of (11) with SLSQP, using maxiter = 250 and ftol = 10^{-12}, and retry with maxiter = 1000 and ftol = 10^{-9} if the first attempt fails. The continuous solution is then rounded to integer counts. The rounding takes the integer part of each logger’s planned count, examines the neighboring counts that differ by one in each coordinate, and spends the remaining budget on the loggers with the largest reduction in the worst-target criterion per unit cost.
Adaptive Greedy and Oracle greedy break ties deterministically. Indices within a relative tolerance of 10 − 12 10^{-12} 1 0 − 12 and an absolute tolerance of 10 − 14 10^{-14} 1 0 − 14 of the largest index are treated as tied, and the logger with the smallest index is chosen. The rounding step of the static planners breaks ties in the same way.
In the synthetic environments, the reward mean is estimated separately for each action by ridge regression with penalty one on the context and the interaction x 1 x 2 x_1x_2 x 1 x 2 , once the action has at least six observations. With fewer, we use the action’s sample mean, or the pooled mean if the action has no observations. The prediction before any data is zero. The reward variance is estimated separately for each action from the squared in-sample residuals of the same mean fit, each multiplied by the degrees-of-freedom correction min { 3 , n a / max ( n a − p , 1 ) } \min\{3,n_a/\max(n_a-p,1)\} min { 3 , n a / max ( n a − p , 1 )} , where n a n_a n a is the number of observations of action a a a and p = 7 p=7 p = 7 is the number of parameters of the mean model. With at least 25 observations, these corrected squared residuals are regressed by ridge regression with penalty one on all monomials of the context of degree at most two. With 6 to 24 observations their mean is used, with 2 to 5 the sample variance of the action’s rewards, and with fewer the pooled sample variance of all rewards, or 0.1 if fewer than two rewards are available. Predicted variances are truncated to [ 0.001 , 16 ] [0.001,16] [ 0.001 , 16 ] . In the ACTG application, the mean regression uses the basis of Appendix A7.8 with penalty 10 − 8 10^{-8} 1 0 − 8 , and the variance is constant within each treatment arm, equal to the mean of the corrected squared residuals with p = 10 p=10 p = 10 , with the same fallbacks for small samples. During purchasing, each fit uses only the observations collected up to that point, and we do not use cross-fitting. Theorems 4.1 and 4.2 rely on Assumptions 4 and 5, which we do not verify for this ridge implementation. Instead, the calibration study in Appendix A7.6 examines its behavior empirically.
Every procedure begins with four observations from each logger, collected in turn, for an initialization of n 0 = 20 n_0=20 n 0 = 20 observations. The reward estimates are refitted every ten observations, counting the initialization, and once more on all data for the final estimator. Reoptimized plug-in reoptimizes at the same times. Adaptive Greedy uses α = 0.4 \alpha=0.4 α = 0.4 , ρ 0 = 0.5 \rho_0=0.5 ρ 0 = 0.5 , and η = η 0 = 1 \eta=\eta_0=1 η = η 0 = 1 , and the reward prediction before the first fit is zero. Forced sampling in Algorithm 1 uses χ 0 = 10 − 4 \chi_0=10^{-4} χ 0 = 1 0 − 4 and ν = 0.01 \nu=0.01 ν = 0.01 . Each logger mixes its policy with the uniform distribution, so every logger gives every action probability at least 0.004 in Environments A to C, 0.002 in the weak-overlap experiment of Appendix § 13.5 , and 0.0125 in the ACTG application. Hence κ ( θ ) ≥ 0.002 > χ 0 ≥ χ t \kappa(\theta)\geq0.002>\chi_0\geq\chi_t κ ( θ ) ≥ 0.002 > χ 0 ≥ χ t for every composition θ \theta θ . Thus forced sampling never occurs, and is run after every update of Algorithm 1. These floors also give κ a l l ≥ 0.01 \kappa_{\mathrm{all}}\geq0.01 κ all ≥ 0.01 , so that χ 0 ≤ κ a l l / ( 8 K ) = κ ( 1 / K ) / 8 \chi_0\leq\kappa_{\mathrm{all}}/(8K)=\kappa(1/K)/8 χ 0 ≤ κ all / ( 8 K ) = κ ( 1/ K ) /8 with K = 5 K=5 K = 5 . The initialization of four observations per logger gives n 0 / K = 4 n_0/K=4 n 0 / K = 4 .
For target π j \pi_j π j , R M S E j = { R − 1 ∑ r ( J ^ r , j − J j ) 2 } 1 / 2 \mathrm{RMSE}_j=\{R^{-1}\sum_r(\widehat{J}_{r,j}-J_j)^2\}^{1/2} RMSE j = { R − 1 ∑ r ( J r , j − J j ) 2 } 1/2 , the worst-target RMSE is max j R M S E j \max_j\mathrm{RMSE}_j max j RMSE j , and the average RMSE is M − 1 ∑ j R M S E j M^{-1}\sum_j\mathrm{RMSE}_j M − 1 ∑ j RMSE j . The worst-target RMSE is the largest of the target-specific RMSEs, not the square root of the average of the largest squared errors. Its Monte Carlo standard error is the standard deviation over 5000 bootstrap resamples of the replications. In each resample, the target-specific RMSEs are recomputed before the maximum is taken, and the same resampled replications are used for all procedures, which preserves the common random numbers. For quantities that are averages over replications, such as G ( z ^ ) / G ( z ⋆ ) G(\widehat{z})/G(z^\star) G ( z ) / G ( z ⋆ ) and ∥ θ ^ − θ ⋆ ∥ 2 \lVert\widehat{\theta}-\theta^\star\rVert_2 ∥ θ − θ ⋆ ∥ 2 , the Monte Carlo standard error is the sample standard deviation divided by R \sqrt{R} R . The main simulations, the trajectory experiment, and the ACTG application use the seeds 20260804, 20260829, and 20260911, respectively. Table 4 lists the number of replications in each experiment.
Table 4. Number of replications in each experiment.
Comparison Methods
Let P B ( V ~ , W ~ ) \mathcal{P}_B(\widetilde{V},\widetilde{W}) P B ( V , W ) denote the count plan for budget B B B produced by the static planner of Appendix § 13.1 when it is given the reward moments V ~ \widetilde{V} V and W ~ \widetilde{W} W .
Adaptive Greedy is Algorithm 1. At step t t t , it purchases from the affordable logger with the largest index S ^ t , k = c k − 1 ∑ j ω ^ j , t Δ ^ j , k , t \widehat{S}_{t,k}=c_k^{-1}\sum_j\widehat{\omega}_{j,t}\widehat{\Delta}_{j,k,t} S t , k = c k − 1 ∑ j ω j , t Δ j , k , t , where Δ ^ j , k , t \widehat{\Delta}_{j,k,t} Δ j , k , t are the estimated one-step gains and ω ^ j , t \widehat{\omega}_{j,t} ω j , t are the weights of § 3.2 .
Reoptimized plug-in computes, at each update time t t t , the plan N ~ B ( t ) = P B ( V ^ t , W ^ t ) \widetilde{N}^{(t)}_B=P_B(\widehat{V}_t,\widehat{W}_t) N B ( t ) = P B ( V t , W t ) for the declared budget B B B . Until the next update, it purchases from the loggers whose current counts fall furthest short of this plan. Each plan solves a static problem, but the procedure is adaptive because the plan is recomputed as data accumulate.
Estimated static follows the single plan P B ( V ^ n 0 , W ^ n 0 ) P_B(\widehat{V}_{n_0},\widehat{W}_{n_0}) P B ( V n 0 , W n 0 ) computed after the initialization and never updates it. Oracle static follows P B ( V , W ) P_B(V,W) P B ( V , W ) , which uses the true reward moments, and serves only as a reference. Because both plans come from the rounded relaxation, neither is guaranteed to be the exact integer optimum.
Equal expenditure spends the same amount on each logger, so that in the limit its sample fractions are θ k e q = c k − 1 / ∑ ℓ c ℓ − 1 \theta_k^{\mathrm{eq}}=c_k^{-1}/\sum_{\ell}c_\ell^{-1} θ k eq = c k − 1 / ∑ ℓ c ℓ − 1 . These differ from equal sample counts whenever the costs differ. Cheapest only purchases from the cheapest affordable logger after the initialization, which shows the effect of using cost while ignoring the logging policies.
Oracle greedy purchases from the affordable logger that maximizes the true gain per unit cost, Δ k ( N t ) / c k \Delta_k(N_t)/c_k Δ k ( N t ) / c k . It is not feasible in practice and is reported as a reference in the robustness experiments of Appendix § 13.5 and in the ACTG application.
Synthetic Environments
All three environments have a five-dimensional context, five actions, five loggers, and four target policies. Environments A and B share the context distribution, the reward model, and the loggers, and Environment C differs from Environment A only in its costs.
Context and reward
The context follows the Gaussian mixture
X ∼ 0.5 N ( m 1 , I 5 ) + 0.3 N ( m 2 , 0.7 I 5 ) + 0.2 N ( m 3 , 0.5 I 5 ) , X \sim0.5\,N(m_1,I_5)+0.3\,N(m_2,0.7I_5)+0.2\,N(m_3,0.5I_5), X ∼ 0.5 N ( m 1 , I 5 ) + 0.3 N ( m 2 , 0.7 I 5 ) + 0.2 N ( m 3 , 0.5 I 5 ) ,
with m 1 = ( 0 , 0 , 0 , 0 , 0 ) T m_1=(0,0,0,0,0)^\mathsf{T} m 1 = ( 0 , 0 , 0 , 0 , 0 ) T , m 2 = ( 1.5 , − 1 , 0 , 0 , 0 ) T m_2=(1.5,-1,0,0,0)^\mathsf{T} m 2 = ( 1.5 , − 1 , 0 , 0 , 0 ) T , and m 3 = ( − 1 , 1.5 , 1 , 0 , 0 ) T m_3=(-1,1.5,1,0,0)^\mathsf{T} m 3 = ( − 1 , 1.5 , 1 , 0 , 0 ) T . Let
B q = ( 0.25 − 0.10 0.05 0 0 0.10 0.25 − 0.05 0.08 0 − 0.10 0.15 0.20 0 0.05 − 0.15 − 0.05 0.25 0.10 0 0.05 − 0.15 0.10 − 0.05 0.20 ) . B_q=\begin{pmatrix}
0.25 & -0.10 & 0.05 & 0 & 0\\
0.10 & 0.25 & -0.05 & 0.08 & 0\\
-0.10 & 0.15 & 0.20 & 0 & 0.05\\
-0.15 & -0.05 & 0.25 & 0.10 & 0\\
0.05 & -0.15 & 0.10 & -0.05 & 0.20
\end{pmatrix}. B q = 0.25 0.10 − 0.10 − 0.15 0.05 − 0.10 0.25 0.15 − 0.05 − 0.15 0.05 − 0.05 0.20 0.25 0.10 0 0.08 0 0.10 − 0.05 0 0 0.05 0 0.20 .
For action a a a , the reward before clipping is normal with mean q 0 ( x , a ) = ( B q x ) a + 0.3 x 1 x 2 1 { a ∈ { 3 , 4 , 5 } } q_0(x,a)=(B_qx)_a+0.3x_1x_2\mathbf{1}\{a\in\{3,4,5\}\} q 0 ( x , a ) = ( B q x ) a + 0.3 x 1 x 2 1 { a ∈ { 3 , 4 , 5 }} and variance s 2 ( x , a ) = 0.05 s^2(x,a)=0.05 s 2 ( x , a ) = 0.05 for a ∈ { 1 , 2 , 3 } a\in\{1,2,3\} a ∈ { 1 , 2 , 3 } and s 2 ( x , a ) = 0.05 + 0.02 ( 1 + 0.5 x 3 2 ) s^2(x,a)=0.05+0.02(1+0.5x_3^2) s 2 ( x , a ) = 0.05 + 0.02 ( 1 + 0.5 x 3 2 ) for a ∈ { 4 , 5 } a\in\{4,5\} a ∈ { 4 , 5 } . The observed reward is clipped to [ − 4 , 4 ] [-4,4] [ − 4 , 4 ] , and the design calculations use the exact mean and variance of the clipped normal distribution.
Loggers
Loggers 1 to 4 favor the action pairs ( 1 , 2 ) (1,2) ( 1 , 2 ) , ( 2 , 3 ) (2,3) ( 2 , 3 ) , ( 3 , 4 ) (3,4) ( 3 , 4 ) , and ( 4 , 5 ) (4,5) ( 4 , 5 ) , with logit gaps 4.5, 3.5, 3.5, and 4.5 and exploration floors 0.02, 0.04, 0.04, and 0.04. Their logits do not depend on the context, so they differ only in which actions they cover. Logger 5 is close to uniform, with a diagonal context coefficient of 0.01 and exploration floor 0.10. A softmax policy p ( a ∣ x ) p(a\mid x) p ( a ∣ x ) with exploration floor ϵ \epsilon ϵ becomes ( 1 − ϵ ) p ( a ∣ x ) + ϵ / 5 (1-\epsilon)p(a\mid x)+\epsilon/5 ( 1 − ϵ ) p ( a ∣ x ) + ϵ /5 . The costs in Environments A and B are c = ( 1 , 2 , 4 , 8 , 16 ) c=(1,2,4,8,16) c = ( 1 , 2 , 4 , 8 , 16 ) .
Target policies
All four target policies share the context coefficient matrix
Γ = 0.025 ( 1 − 1 0 0 0 − 1 1 0 0 0 0 1 − 1 0 0 0 0 1 − 1 0 0 0 0 1 − 1 ) . \Gamma=0.025\begin{pmatrix}
1 & -1 & 0 & 0 & 0\\
-1 & 1 & 0 & 0 & 0\\
0 & 1 & -1 & 0 & 0\\
0 & 0 & 1 & -1 & 0\\
0 & 0 & 0 & 1 & -1
\end{pmatrix}. Γ = 0.025 1 − 1 0 0 0 − 1 1 1 0 0 0 0 − 1 1 0 0 0 0 − 1 1 0 0 0 0 − 1 .
In Environment A, their logit intercepts are b 1 = ( 0 , 0 , 0 , 0 , 0 ) b_1=(0,0,0,0,0) b 1 = ( 0 , 0 , 0 , 0 , 0 ) , b 2 = ( − 0.4 , 0.4 , 0.4 , − 0.4 , − 0.4 ) b_2=(-0.4,0.4,0.4,-0.4,-0.4) b 2 = ( − 0.4 , 0.4 , 0.4 , − 0.4 , − 0.4 ) , b 3 = ( − 0.55 , − 0.55 , − 0.55 , 0.55 , 0.55 ) b_3=(-0.55,-0.55,-0.55,0.55,0.55) b 3 = ( − 0.55 , − 0.55 , − 0.55 , 0.55 , 0.55 ) , and b 4 = ( − 0.475 , − 0.475 , 0.475 , 0.475 , − 0.475 ) b_4=(-0.475,-0.475,0.475,0.475,-0.475) b 4 = ( − 0.475 , − 0.475 , 0.475 , 0.475 , − 0.475 ) , and their exploration floors are ( 0.12 , 0.06 , 0.05 , 0.05 ) (0.12,0.06,0.05,0.05) ( 0.12 , 0.06 , 0.05 , 0.05 ) . Environment B uses the floors ( 0.09 , 0.045 , 0.0375 , 0.0375 ) (0.09,0.045,0.0375,0.0375) ( 0.09 , 0.045 , 0.0375 , 0.0375 ) and the fourth intercept b 4 = ( − 0.9 , − 0.9 , 0.9 , 0.9 , − 0.9 ) b_4=(-0.9,-0.9,0.9,0.9,-0.9) b 4 = ( − 0.9 , − 0.9 , 0.9 , 0.9 , − 0.9 ) , which has the same sign pattern as in Environment A but a larger magnitude. Environment C uses the policies of Environment A and the costs c = ( 1 , 4 , 16 , 8 , 2 ) c=(1,4,16,8,2) c = ( 1 , 4 , 16 , 8 , 2 ) . Table 5 lists the optimal designs.
Table 5. Optimal designs in the synthetic environments.
Additional Synthetic Results
Average RMSE
Figure 5 reports the average RMSE over the four targets in the three synthetic environments and the ACTG application. The comparisons are similar to those for the worst-target RMSE in § 5 .
Figure 5. Average RMSE over targets in the synthetic environments and the ACTG application. Error bars show ± 1 \pm1 ± 1 Monte Carlo standard error.
Paired comparisons
Because common random numbers are used, we also compare procedures pairwise, resampling the same replications for both procedures in each pair. The paired intervals separate Adaptive Greedy from Cheapest only at every budget. In Environments A and B, they also separate Adaptive Greedy from Estimated static and Equal expenditure at most budgets. In Environment C, the intervals comparing Adaptive Greedy with Estimated static contain zero at every budget, in line with their similar accuracy in Figure 1 . Adaptive Greedy and Reoptimized plug-in are much closer, and only two of the twenty-one comparisons between them exclude zero.
Design criterion and estimation error
Figure 6 plots, for each procedure, environment, and budget, the worst-target RMSE relative to that of Adaptive Greedy against the square root of the criterion G \mathcal{G} G relative to that of Adaptive Greedy. The square root puts G \mathcal{G} G , a variance, on the scale of the RMSE. For budgets of at least 3000, the correlation between the two is about 0.974, excluding the reference points of Adaptive Greedy itself. Thus the design criterion closely tracks estimation accuracy. The points are not independent, so the figure is descriptive.
Figure 6. Relative worst-target RMSE against the square root of the relative criterion G G G , with Adaptive Greedy as the reference. The diagonal line is the identity.
Robustness Checks and Sensitivity
We examine three departures from the standard setting in Environment A, each with budgets 1500 and 6000 and R = 100 R = 100 R = 100 replications. The results are shown in Figure 7 .
Figure 7. Robustness experiments with a misleading initialization, a misspecified reward model, and weak overlap. Error bars show ± 1 \pm1 ± 1 Monte Carlo standard error. Only the procedures run in each experiment are shown.
The first experiment uses a misleading initialization, in which the third mixture component of the context distribution is excluded from the initial observations only. At B = 1500 B = 1500 B = 1500 , the worst-target RMSE is 0.025 for Adaptive Greedy, 0.024 for Reoptimized plug-in, and 0.034 for Estimated static, so the plan fixed after the initialization suffers most. Reoptimized plug-in is slightly more accurate than Adaptive Greedy in this setting. At B = 6000 B = 6000 B = 6000 , the worst-target RMSE of Adaptive Greedy is 0.0116 under the misleading initialization and 0.0115 under the standard initialization. Its criterion ratio is about 1.011, so the effect of the misleading start has disappeared.
The second experiment misspecifies the reward model by adding 0.15 x 3 2 0.15x_3^2 0.15 x 3 2 to the mean reward of actions 3, 4, and 5, while the fitted regression omits this term. At B = 6000 B = 6000 B = 6000 , the worst-target RMSE is 0.0135 for Adaptive Greedy, 0.0130 for Oracle greedy, 0.0166 for Estimated static, and 0.0162 for Equal expenditure. Misspecification affects Adaptive Greedy only slightly, and it remains more accurate than the static and cost-based procedures.
The third experiment weakens overlap by lowering the logger exploration floors to (0.01, 0.01, 0.01, 0.025). At B = 6000 B = 6000 B = 6000 , the worst-target RMSE is 0.0113 for Adaptive Greedy, 0.0114 for Oracle greedy, 0.0126 for Estimated static, and 0.0144 for Equal expenditure.
Sensitivity to α \alpha α and ρ 0 \rho_0 ρ 0
We run Adaptive Greedy in Environment A with α ∈ { 0.25 , 0.4 } \alpha\in\{0.25, 0.4\} α ∈ { 0.25 , 0.4 } and ρ 0 ∈ { 0.25 , 0.5 , 1 } \rho_0 \in\{0.25, 0.5, 1\} ρ 0 ∈ { 0.25 , 0.5 , 1 } , at budgets 1500 and 6000 and with R = 50 R = 50 R = 50 replications per setting. Over the six settings, the worst-target
RMSE at B = 1500 B = 1500 B = 1500 lies between 0.0221 and 0.0245, and the criterion ratio at B = 6000 B = 6000 B = 6000 lies between 1.010 and 1.025 (Figure 8 ). The procedure is therefore insensitive to these choices, and we use α = 0.4 \alpha= 0.4 α = 0.4 and ρ 0 = 0.5 \rho_0 = 0.5 ρ 0 = 0.5 in all other experiments without tuning.
Figure 8. Sensitivity of Adaptive Greedy to α \alpha α and ρ 0 \rho_0 ρ 0 , with R = 50 R = 50 R = 50 replications per setting. Error bars show ± 1 \pm1 ± 1 Monte Carlo standard error.
Inference
Calibration
For Environments A and B at budgets 6000 and 24000, the 200 main replications are combined with 300 further replications drawn with different seeds, giving R = 500 R = 500 R = 500 independent replications for each environment and budget. Pointwise intervals use the normal 0.975 quantile, and simultaneous intervals use the 0.95 quantile of the largest absolute coordinate of a normal vector with the estimated correlation matrix, computed from 20000 draws. Over the four settings, the empirical standard deviations range from 0.0038 to 0.0113 and the mean estimated standard errors from 0.0039 to 0.0116. Their ratio lies between 0.989 and 1.060, pointwise coverage lies between 0.944 and 0.966, and simultaneous coverage lies between 0.950 and 0.960 (Table 6 and Table 7 ).
Table 6. Target-specific calibration of the Wald intervals, with 500 replications per environment and budget. The last column is the Monte Carlo standard error of the coverage.
Table 7. Simultaneous 95% coverage of the Wald intervals.
Pooled and logger-specific estimators
By ((30) ), the pooled estimator has a fixed-design covariance no larger than that of the logger-specific estimator. To measure this difference, we apply both estimators to the same purchased samples. Figure 9 shows that the logger-specific estimator has a larger RMSE in every reported setting.
Figure 9. Ratio of the RMSE of the logger-specific estimator to that of the pooled estimator in Environments A and B. Values above one favor the pooled estimator. Error bars show ± 1 \pm1 ± 1 Monte Carlo standard error.
Block-split estimator
We also evaluate the block-split estimator of Appendix A5 in Environment A, with budgets 1500 and 6000 and R = 200 R = 200 R = 200 replications. The first n ˉ = n 0 = 20 \bar{n} = n_0 = 20 n ˉ = n 0 = 20 observations form the first block and are not scored, and the following blocks have lengths 20, 40, 80, … \ldots … , so that κ = 2 \kappa= 2 κ = 2 . The regression for each block is fitted on all earlier observations, and each block uses its own logger frequencies in the denominator. Table 8 shows that the block-split estimator is less accurate at B = 1500 B = 1500 B = 1500 and that the gap narrows at B = 6000 B = 6000 B = 6000 , which reflects the observations it discards and the less accurate regressions it uses in early blocks.
Table 8. Worst-target RMSE of the final and block-split estimators in Environment A.
The trajectory experiment uses Environment A with the standard initialization, budget 5000, and R = 50 R = 50 R = 50 replications, and it records the design at the intermediate budgets 200, 500, 700, 1000, 2000, 3000, 4000, and 5000. Adaptive Greedy and Reoptimized plug-in use the same random numbers in each replication. The index of Adaptive Greedy does not involve the budget, since its smoothing parameter depends only on the money spent so far. The budget affects only which loggers are affordable and when the run stops. However, Reoptimized plug-in plans its total counts for the budget, which is 5000 throughout this experiment.
Table 9 reports the design diagnostics of Adaptive Greedy along these runs. The difference in G / G ⋆ \mathcal{G}/\mathcal{G}^{\star} G / G ⋆ between Adaptive Greedy and Reoptimized plug-in is negative at every intermediate budget up to 4000. At 5000 it is 0.000847, with interval [0.000126, 0.001567], so the two procedures end at nearly the same criterion, with Reoptimized plug-in slightly closer.
Table 9. Mean design diagnostics of Adaptive Greedy along runs with budget 5000 in Environment A, over 50 replications.
Table 10 compares these runs with the separate runs of § 5 , which use R = 200 R = 200 R = 200 replications and a budget equal to the intermediate budget. The Adaptive Greedy values in the two experiments agree closely, while the Reoptimized plug-in values depend strongly on whether it plans for 5000 or for the intermediate budget. When each procedure is given the intermediate budget as its total budget, both reach similar criterion ratios, so the difference between the two trajectories comes from the budget toward which each procedure plans.
Table 10. Criterion ratio G / G ⋆ \mathcal{G}/\mathcal{G}^{\star} G / G ⋆ at three intermediate budgets. The trajectory columns follow runs with budget 5000, and the separate-run columns report separate runs whose budget equals the intermediate budget.
The rise in the criterion ratio of Reoptimized plug-in early in the run occurs in almost every replication. It increases between intermediate budgets 500 and 700 in 96% of replications, and again between 700 and 1000 in 96%, with mean increases of about 0.142 and 0.166. At these budgets the procedure is still following plans made for budget 5000. Its plans also change substantially between early updates, which suggests that unstable early estimates contribute as well.
Balanced costs
Under the costs c = ( 1 , 2 , 4 , 8 , 16 ) c = (1, 2, 4, 8, 16) c = ( 1 , 2 , 4 , 8 , 16 ) , the plan of Reoptimized plug-in for budget 5000 is heavily concentrated on Logger 1, and a partial run that follows it can be far from the plan for a smaller budget. To check this explanation, we keep the statistical model and target policies of Environment A and change only the costs to c = ( 1 , 1 , 1.4 , 2.85 , 1.8 ) c = (1, 1, 1.4, 2.85, 1.8) c = ( 1 , 1 , 1.4 , 2.85 , 1.8 ) , chosen so that the optimal fractions are roughly equal over the four useful loggers, θ ⋆ = ( 0.28243 , 0.22456 , 0.24673 , 0.24628 , 0 ) \theta^\star= (0.28243, 0.22456, 0.24673, 0.24628, 0) θ ⋆ = ( 0.28243 , 0.22456 , 0.24673 , 0.24628 , 0 ) .
Two findings support the explanation. First, consider a path computed with the true reward moments that plans for budget 5000. The ratio of its criterion at intermediate budget B B B to that of the plan made for budget B B B is 1.018, 1.009, and 1.004 at B = 500 B = 500 B = 500 , 700, and 1000 under the balanced costs, compared with 1.569, 1.778, and 1.983 under the original costs. Second, Table 11 shows that under the original costs the mean G / G ⋆ G/G^\star G / G ⋆ of Reoptimized plug-in rises from 1.414 at B = 500 B = 500 B = 500 to 1.723 at $B = 1000, whereas under the balanced costs it falls from 1.323 to 1.213, and its distance to θ ⋆ \theta^\star θ ⋆ also decreases. At B = 5000 B = 5000 B = 5000 , both runs are close to their optimal designs.
Table 11. Plans for each intermediate budget and mean counts purchased by Reoptimized plug-in along runs with budget 5000, under the original and the balanced costs. The plans are deterministic, and the purchased counts are means over 50 replications. G / G ⋆ \mathcal{G}/\mathcal{G}^{\star} G / G ⋆ and ∥ θ − θ ⋆ ∥ 2 \lVert\theta-\theta^{\star}\rVert_2 ∥ θ − θ ⋆ ∥ 2 use the optimal design for the same costs.
Runtime
The runtimes refer to our implementation and are meant for comparison between procedures. In Environment A, the mean runtime at B = 12000 B = 12000 B = 12000 is about 742 seconds for Adaptive Greedy and 6423 seconds for Reoptimized plug-in, and at B = 24000 B = 24000 B = 24000 about 1750 and 16442 seconds. The difference comes mainly from the repeated reoptimization of Reoptimized plug-in.
ACTG 175 Construction
The ACTG 175 study ([21 ] ) was a randomized trial comparing nucleoside treatment regimens in HIV-infected adults. We use the public data ([21 ] ), which contain 2139 subjects with complete records. The actions are the four treatments, coded 0 for zidovudine (ZDV), 1 for ZDV with didanosine (ddI), 2 for ZDV with zalcitabine, and 3 for ddI alone. The context is
X = ( age , wtkg , cd40 , cd80 , karnof , symptom , str2 ) T , X = (\text{age},\text{wtkg},\text{cd40},\text{cd80},\text{karnof},\text{symptom},\text{str2})^{\mathsf{T}}, X = ( age , wtkg , cd40 , cd80 , karnof , symptom , str2 ) T ,
which records age, weight, baseline CD4 and CD8 counts, Karnofsky score, symptomatic status, and prior antiretroviral therapy. Continuous covariates are standardized and binary covariates are centered, using the means and standard deviations of the 2139 subjects.
The response used to build the reward model is Y = ( cd420 − cd40 ) / 100 Y=(\text{cd420}-\text{cd40})/100 Y = ( cd420 − cd40 ) /100 , the change in CD4 count from baseline to around week 20. For each treatment a a a we fit a least-squares regression of Y Y Y on the basis ϕ ( x ) = ( 1 , x 1 , … , x 7 , x 3 2 , x 1 x 3 ) T \phi(x)=(1,x_1,\ldots,x_7,x_3^2,x_1x_3)^{\mathsf{T}} ϕ ( x ) = ( 1 , x 1 , … , x 7 , x 3 2 , x 1 x 3 ) T , giving a fitted mean m ^ a ( x ) \widehat{m}_a(x) m a ( x ) and a residual variance σ ^ a 2 \widehat{\sigma}_a^2 σ a 2 . In the experiment, contexts are drawn with replacement from the 2139 subjects, and the reward is
R = min { 4 , max ( − 4 , R ∘ ) } , R ∘ = m ^ A ( X ) + ϵ A , ϵ a ∼ N ( 0 , σ ^ a 2 ) . R=\min\{4,\max(-4,R^\circ)\},\qquad R^\circ=\widehat{m}_A(X)+\epsilon_A,\qquad\epsilon_a\sim N(0,\widehat{\sigma}_a^2). R = min { 4 , max ( − 4 , R ∘ )} , R ∘ = m A ( X ) + ϵ A , ϵ a ∼ N ( 0 , σ a 2 ) .
The design calculations use the exact moments of the clipped normal distribution. This construction keeps the observed covariate distribution and the fitted treatment effects while making the policy values and the optimal design known.
Loggers
Let s = ( x 3 , x 7 , x 6 ) T s=(x_3,x_7,x_6)^{\mathsf{T}} s = ( x 3 , x 7 , x 6 ) T and
D = ( 0.35 − 0.45 − 0.30 − 0.25 0.20 0.25 − 0.25 0.15 0.25 0.15 0.10 − 0.20 ) , B = ( 1 1 − 1 − 1 − 1 1 1 − 1 − 1 − 1 1 1 1 − 1 − 1 1 ) . D=\begin{pmatrix}0.35&-0.45&-0.30\\-0.25&0.20&0.25\\-0.25&0.15&0.25\\0.15&0.10&-0.20\end{pmatrix},\qquad B=\begin{pmatrix}1&1&-1&-1\\-1&1&1&-1\\-1&-1&1&1\\1&-1&-1&1\end{pmatrix}. D = 0.35 − 0.25 − 0.25 0.15 − 0.45 0.20 0.15 0.10 − 0.30 0.25 0.25 − 0.20 , B = 1 − 1 − 1 1 1 1 − 1 − 1 − 1 1 1 − 1 − 1 − 1 1 1 .
For k = 1 , … , 4 k=1,\ldots,4 k = 1 , … , 4 , Logger k k k is the softmax policy with logits B k , a + ( D s ) a B_{k,a}+(Ds)_a B k , a + ( D s ) a , mixed with the uniform distribution with weight ϵ k \epsilon_k ϵ k , where ( ϵ 1 , ϵ 2 , ϵ 3 , ϵ 4 ) = ( 0.05 , 0.05 , 0.08 , 0.08 ) (\epsilon_1,\epsilon_2,\epsilon_3,\epsilon_4)=(0.05,0.05,0.08,0.08) ( ϵ 1 , ϵ 2 , ϵ 3 , ϵ 4 ) = ( 0.05 , 0.05 , 0.08 , 0.08 ) . Logger 5 is uniform, and the costs are c = ( 1 , 2 , 4 , 3 , 8 ) c=(1,2,4,3,8) c = ( 1 , 2 , 4 , 3 , 8 ) . Averaged over the 2139 subjects, the loggers choose the treatments with the following probabilities.
Loggers 1 to 4 therefore cover different pairs of treatments. The costs are units chosen for the experiment and are not estimates of clinical recruitment costs.
Target policies
The first target policy favors treatments with higher fitted rewards. With qclip ( x , a ) \operatorname{qclip}(x,a) qclip ( x , a ) the mean of the clipped reward and τ 1 \tau_1 τ 1 equal to half the standard deviation of qclip ( X , a ) \operatorname{qclip}(X,a) qclip ( X , a ) over subjects and treatments, it is
π 1 ( a ∣ x ) = 0.9 exp { qclip ( x , a ) / τ 1 } ∑ b exp { qclip ( x , b ) / τ 1 } + 0.1 4 . \pi_1(a\mid x)=0.9\frac{\exp\{\operatorname{qclip}(x,a)/\tau_1\}}{\sum_b\exp\{\operatorname{qclip}(x,b)/\tau_1\}}+\frac{0.1}{4}. π 1 ( a ∣ x ) = 0.9 ∑ b exp { qclip ( x , b ) / τ 1 } exp { qclip ( x , a ) / τ 1 } + 4 0.1 .
For j = 2 , 3 , 4 j=2,3,4 j = 2 , 3 , 4 , target j j j is the softmax policy with logits b j , a + ν j ( D s ) a b_{j,a}+\nu_j(Ds)_a b j , a + ν j ( D s ) a , where b 2 = ( − 0.6 , 0.6 , 0.6 , − 0.6 ) b_2=(-0.6,0.6,0.6,-0.6) b 2 = ( − 0.6 , 0.6 , 0.6 , − 0.6 ) , b 3 = ( − 0.9 , 0.3 , − 0.3 , 0.9 ) b_3=(-0.9,0.3,-0.3,0.9) b 3 = ( − 0.9 , 0.3 , − 0.3 , 0.9 ) , b 4 = ( 0.7 , 0.3 , − 0.5 , − 0.5 ) b_4=(0.7,0.3,-0.5,-0.5) b 4 = ( 0.7 , 0.3 , − 0.5 , − 0.5 ) , and ν = ( 1 , 0.7 , 1.2 ) \nu=(1,0.7,1.2) ν = ( 1 , 0.7 , 1.2 ) , mixed with the uniform distribution with floors ( 0.06 , 0.05 , 0.05 ) (0.06,0.05,0.05) ( 0.06 , 0.05 , 0.05 ) . The optimal design is
z ⋆ = ( 0.576469 , 0 , 0.020001 , 0.114509 , 0 ) , θ ⋆ = ( 0.810810 , 0 , 0.028132 , 0.161058 , 0 ) , z^{\star}=(0.576469,0,0.020001,0.114509,0),\qquad\theta^{\star}=(0.810810,0,0.028132,0.161058,0), z ⋆ = ( 0.576469 , 0 , 0.020001 , 0.114509 , 0 ) , θ ⋆ = ( 0.810810 , 0 , 0.028132 , 0.161058 , 0 ) ,
with G ⋆ = 4.237904 \mathcal{G}^{\star}=4.237904 G ⋆ = 4.237904 , and targets 1 and 3 attain the maximum at this design.
Cost-based designs
With c = ( 1 , 2 , 4 , 3 , 8 ) c=(1,2,4,3,8) c = ( 1 , 2 , 4 , 3 , 8 ) , Equal expenditure has limiting fractions θ e q ∝ ( 1 , 1 / 2 , 1 / 4 , 1 / 3 , 1 / 8 ) \theta_{\mathrm{eq}}\propto(1,1/2,1/4,1/3,1/8) θ eq ∝ ( 1 , 1/2 , 1/4 , 1/3 , 1/8 ) , that is, θ e q = ( 0.452830 , 0.226415 , 0.113208 , 0.150943 , 0.056604 ) \theta_{\mathrm{eq}}=(0.452830,0.226415,0.113208,0.150943,0.056604) θ eq = ( 0.452830 , 0.226415 , 0.113208 , 0.150943 , 0.056604 ) , which differ from the equal counts ( 0.2 , 0.2 , 0.2 , 0.2 , 0.2 ) (0.2,0.2,0.2,0.2,0.2) ( 0.2 , 0.2 , 0.2 , 0.2 , 0.2 ) . With z θ = θ / c T θ z_{\theta}=\theta/c^{\mathsf{T}}\theta z θ = θ / c T θ the design with sample fractions θ \theta θ , their criterion ratios are G ( z θ c o u n t ) / G ⋆ = 2.499102 \mathcal{G}(z_{\theta_{\mathrm{count}}})/\mathcal{G}^{\star}=2.499102 G ( z θ count ) / G ⋆ = 2.499102 and G ( z θ e q ) / G ⋆ = 1.404870 \mathcal{G}(z_{\theta_{\mathrm{eq}}})/\mathcal{G}^{\star}=1.404870 G ( z θ eq ) / G ⋆ = 1.404870 , and purchasing only from the cheapest logger gives G ( z θ c h e a p ) / G ⋆ = 1.213666 \mathcal{G}(z_{\theta_{\mathrm{cheap}}})/\mathcal{G}^{\star}=1.213666 G ( z θ cheap ) / G ⋆ = 1.213666 .
Additional ACTG 175 Results
Oracle greedy, which uses the true gains, has worst-target RMSE (× 10 − 3 \times10^{-3} × 1 0 − 3 ) of 120, 55.9, 28.2, and 18.6 at budgets 500, 1500, 6000, and 12000, with Monte Carlo standard errors 8.7, 2.5, 1.4, and 0.83. Table 12 reports the mean fractions of Adaptive Greedy. As the budget grows, the mass on Loggers 2 and 5 decreases toward zero and the fractions of Loggers 1, 3, and 4 approach their optimal values. At B = 12000 B=12000 B = 12000 , the mean runtime is about 93 seconds for Adaptive Greedy and 705 seconds for Reoptimized plug-in. Because the rewards are simulated from a fitted model, these results describe the purchasing problem and should not be read as a comparison of the clinical efficacy of the treatments.
Table 12. Mean fractions and design diagnostics of Adaptive Greedy in the ACTG application.