Introduction
Monte-Carlo Tree Search (MCTS) drives some of the most capable planning systems ever built, from AlphaGo [28 ] and AlphaZero [29 ] to MuZero [26 ] . Classical variants such as UCT [19 ] , Fixed-Depth-MCTS [27 ] , and Power-UCT [8 ] store a scalar value estimate at each edge, usually an empirical mean. This is sufficient when the objective is expected return, but it is inadequate for risk-sensitive planning. In safety-critical domains such as autonomous driving, medical treatment planning, and financial decision making, two actions can have similar means but very different lower-tail behavior.
A simple example illustrates the problem. Suppose a root action a 1 a_1 a 1 returns 5 deterministically, while another action a 2 a_2 a 2 returns 10 with probability 1 / 2 1/2 1/2 and 0 with probability 1 / 2 1/2 1/2 . Both actions have the same expected value, but their lower-tail conditional value-at-risk (CVaR) is different: for small τ \tau τ , a 1 a_1 a 1 has CVaR equal to 5, whereas a 2 a_2 a 2 has CVaR equal to 0. A scalar mean estimate cannot distinguish these two return laws once their means are known. A risk-sensitive planner, however, should prefer a 1 a_1 a 1 when the objective is to protect against poor outcomes. This requires preserving distributional information at the Q-edges of the search tree.
This paper develops distributional MCTS algorithms for finite-depth risk-sensitive planning. We introduce Categorical Thompson Sampling with Optimism (CATSO) and Particle Thompson Sampling with Optimism (PATSO). Both algorithms maintain finite-support empirical backup laws at Q-edges. CATSO represents each Q-edge law using categorical atoms with Dirichlet Thompson sampling, while PATSO represents it using adaptive weighted particles. At action selection time, both algorithms draw a Thompson-sampled Q-edge law, compute its lower-tail CVaR, and add the polynomial optimism bonus of Shah et al. [27 ] to ensure sufficient exploration.
The algorithms keep V-nodes scalar by backing up a visit-weighted average of child Q-edge CVaR scores, so Q-edges store distributions while V-nodes propagate scalar risk-sensitive summaries. This avoids distributional V-node convolutions while still letting lower-tail information drive the search. Our main result proves an O ( n − 1 / 2 ) O(n^{-1/2}) O ( n − 1/2 ) root value-error bound against the finite-depth nested CVaR target (Section 5), which yields an O ( n − 1 / 2 ) O(n^{-1/2}) O ( n − 1/2 ) risk-sensitive simple regret. CATSO incurs an additional finite-grid discretization bias, and capped PATSO adds a tunable Wasserstein compression bias.
Our contributions are:
We introduce CATSO and PATSO, two non-parametric distributional MCTS algorithms that maintain empirical backup laws at Q-edges.
We propose a CVaR-sensitive scalar V-backup that aggregates visit-weighted child Q-edge CVaR scores rather than child Q-edge means.
We prove an O ( n − 1 / 2 ) O(n^{-1/2}) O ( n − 1/2 ) root value-error bound for the finite-depth nested CVaR target after n n n rollouts, which directly upper-bounds the risk-sensitive simple regret.
We characterize the additional approximation terms introduced by finite CATSO grids and capped PATSO particle compression.
We evaluate the algorithms on stochastic planning benchmarks where mean-optimal and CVaR-optimal behavior differ.
Existing distributional MCTS methods such as DNG-MCTS [1 ] and D2NG-POMCP [2 ] rely on parametric return models, such as Gaussian or mixture distributions. In contrast, CATSO and PATSO are non-parametric: we estimate Q-edge backup laws directly from backed-up samples. This makes our methods suitable for planning problems with multimodal, skewed, or heavy-tailed return distributions, where parametric approximations can obscure the lower-tail behavior that CVaR is designed to measure.
MCTS with non-asymptotic guarantees and distributional backups. While UCT [19 ] established asymptotic convergence for MCTS, the breakthrough finite-sample analysis is due to Shah et al. [27 ] , who modeled each tree node as a non-stationary bandit and proved O ( n − 1 / 2 ) O(n^{-1/2}) O ( n − 1/2 ) root value convergence using a polynomial exploration bonus; Dam et al. [8 ] extended this framework to power-mean backups. However, these analyses rely on scalar point estimates and do not capture tail risk. Existing distributional MCTS methods place parametric posteriors over returns—such as Normal-Gamma models in DNG-MCTS [1 , 2 ] or Gaussian Wasserstein barycenters in W-MCTS [9 ] —or address heuristic multi-objective search [16 ] . Parametric assumptions can obscure lower-tail behavior in multimodal or heavy-tailed domains. Most recently, Zhang et al. [32 ] proposed CVaR-MCTS and a distributionally robust variant (also termed Wasserstein-MCTS, distinct from the Gaussian barycenter method of Dam et al. [9 ] ) that optimize expected reward subject to a static cumulative-cost CVaR constraint with PAC tail-safety and regret guarantees. In contrast, CATSO and PATSO maintain non-parametric empirical backup laws at Q-edges while keeping V-nodes scalar, directly optimizing the unconstrained nested finite-depth CVaR of reward with an O ( n − 1 / 2 ) O(n^{-1/2}) O ( n − 1/2 ) finite-budget root value-error guarantee.
Thompson sampling in stationary vs. non-stationary CVaR bandits. Thompson sampling has been combined with MCTS over scalar or Gaussian posteriors [1 , 15 ] . In the stationary multi-armed bandit literature, Riou and Honda [25 ] established logarithmic regret for Dirichlet Thompson sampling on bounded-support rewards, which Baudry et al. [3 ] extended to support-aware stationary CVaR bandits via multinomial grids (M-CVTS) and adaptive particles (B-CVTS). Our local bandit analysis builds on this Dirichlet convex-polytope machinery, but stationary CVaR bandit guarantees do not transfer directly to MCTS: the backed-up samples Q ^ t ( s h , a ) = r t ( s h , a ) + γ V ^ τ ( s h + 1 ) \widehat{Q}_t(s_h,a)=r_t(s_h,a)+\gamma\widehat{V}^{\tau}(s_{h+1}) Q t ( s h , a ) = r t ( s h , a ) + γ V τ ( s h + 1 ) have a non-stationary law that drifts as the searched subtree evolves. Controlling this coupled evolution requires pairing Dirichlet CVaR Thompson sampling with a polynomial optimism bonus and establishing our three-stage proof roadmap (local non-stationary CVaR bandit concentration, one-step W 1 W_1 W 1 -Lipschitz CVaR lifting across Q-edges, and backward induction).
Risk-sensitive RL. Beyond tree search, risk-sensitive RL has been studied through policy gradients [30 , 6 ] , distributional RL [7 , 18 ] , robust MDPs [17 , 23 ] , and Wasserstein DRO [12 ] . Recent theoretical work has established provably efficient episodic regret and PAC sample-complexity bounds across two primary objectives: static trajectory CVaR , including near-minimax-optimal learning [31 ] and reward-free CVaR exploration [22 ] , and nested (iterated) CVaR , including tabular episodic learning [11 ] , function approximation with human feedback [5 ] , and generative-model sample complexity [10 ] . Whereas these works bound global episodic regret or offline sample complexity across full MDP state spaces, our work addresses online finite-budget Monte-Carlo tree search from a given root state s 0 s_0 s 0 , proving non-asymptotic O ( n − 1 / 2 ) O(n^{-1/2}) O ( n − 1/2 ) root value-error and simple-regret guarantees for nested CVaR planning.
Setting
MDP and finite-depth planning. We consider a discounted MDP M = ⟨ S , A , R , P , γ ⟩ \mathcal{M}=\langle\mathcal{S},\mathcal{A},\mathcal{R},\mathcal{P},\gamma\rangle M = ⟨ S , A , R , P , γ ⟩ , where S \mathcal{S} S is the state space, A \mathcal{A} A is a finite action set, R ( ⋅ ∣ s , a ) \mathcal{R}(\cdot\mid s,a) R ( ⋅ ∣ s , a ) is a stochastic reward kernel with support in [ 0 , R max ] [0,R_{\max}] [ 0 , R m a x ] ,1 P ( ⋅ ∣ s , a ) \mathcal{P}(\cdot\mid s,a) P ( ⋅ ∣ s , a ) is a stochastic transition kernel over S \mathcal{S} S , and γ ∈ [ 0 , 1 ) \gamma\in[0,1) γ ∈ [ 0 , 1 ) is the discount factor. MCTS plans from a root state s 0 s_0 s 0 by simulating depth-H H H trajectories, using a fixed rollout policy π r o \pi_{\mathrm{ro}} π ro at depth H H H ; we write G H π r o ( s ) = ∑ k ≥ 0 γ k r k G_H^{\pi_{\mathrm{ro}}}(s)=\sum_{k\geq0}\gamma^k r_k G H π ro ( s ) = ∑ k ≥ 0 γ k r k for the resulting discounted return. For the analysis, every internal node has finitely many legal actions and every encountered transition kernel has finite support.
Lower-tail CVaR. For a real-valued X X X and τ ∈ ( 0 , 1 ] \tau\in(0,1] τ ∈ ( 0 , 1 ] , the lower-tail conditional value-at-risk is
CVaR τ ( X ) : = 1 τ ∫ 0 τ F X − 1 ( u ) d u , (1) \operatorname{CVaR}_{\tau}(X):=\frac{1}{\tau}\int_{0}^{\tau}F_X^{-1}(u)\,du, \tag*{(1)} CVaR τ ( X ) := τ 1 ∫ 0 τ F X − 1 ( u ) d u , (1)
the average of the worst τ \tau τ -fraction of outcomes, with CVaR 1 ( X ) = E [ X ] \operatorname{CVaR}_{1}(X)=\mathbb{E}[X] CVaR 1 ( X ) = E [ X ] . We also write CVaR τ ( P ) \operatorname{CVaR}_{\tau}(P) CVaR τ ( P ) for laws P P P ; for finitely supported P P P , this is the discrete formula (10) . The same τ \tau τ is used for selection, backup, and evaluation throughout. Although we state the theory for lower-tail CVaR, this is without loss of generality: upper-tail risk can be handled by applying the same framework to − X -X − X , equivalently using CVaR ‾ τ ( X ) : = − CVaR τ ( − X ) \overline{\operatorname{CVaR}}_{\tau}(X):=-\operatorname{CVaR}_{\tau}(-X) CVaR τ ( X ) := − CVaR τ ( − X ) .
Finite-depth nested CVaR target. Set V ~ τ ( s H ) : = CVaR τ ( G H π r o ( s H ) ) \widetilde V_{\tau}(s_H):=\operatorname{CVaR}_{\tau}(G_H^{\pi_{\mathrm{ro}}}(s_H)) V τ ( s H ) := CVaR τ ( G H π ro ( s H )) and, for h = H − 1 , … , 0 h=H-1,\ldots,0 h = H − 1 , … , 0 , define the Bellman-backup law
D τ ( s h , a ) : = L ( R ( s h , a , S ′ ) + γ V ~ τ ( S ′ ) ) , S ′ ∼ P ( ⋅ ∣ s h , a ) , (2) \mathcal{D}^{\tau}(s_h,a):=\mathcal{L}\left(R(s_h,a,S')+\gamma\widetilde V^{\tau}(S')\right),\qquad S'\sim\mathcal{P}(\cdot\mid s_h,a), \tag*{(2)} D τ ( s h , a ) := L ( R ( s h , a , S ′ ) + γ V τ ( S ′ ) ) , S ′ ∼ P ( ⋅ ∣ s h , a ) , (2)
with reward sampled from R ( s h , a , S ′ ) \mathcal{R}(s_h,a,S') R ( s h , a , S ′ ) , and
Q ~ τ ( s h , a ) : = CVaR τ ( D τ ( s h , a ) ) , V ~ τ ( s h ) : = max a ∈ A ( s h ) Q ~ τ ( s h , a ) . (3) \widetilde Q^{\tau}(s_h,a):=\operatorname{CVaR}_{\tau}\left(\mathcal{D}^{\tau}(s_h,a)\right),\qquad\widetilde V^{\tau}(s_h):=\max_{a\in\mathcal{A}(s_h)}\widetilde Q^{\tau}(s_h,a). \tag*{(3)} Q τ ( s h , a ) := CVaR τ ( D τ ( s h , a ) ) , V τ ( s h ) := a ∈ A ( s h ) max Q τ ( s h , a ) . (3)
Write V ~ τ ( s 0 ) : = V ~ 0 τ ( s 0 ) \widetilde V^{\tau}(s_0):=\widetilde V^{\tau}_0(s_0) V τ ( s 0 ) := V 0 τ ( s 0 ) . At τ = 1 \tau=1 τ = 1 , (3) reduces to the standard finite-depth risk-neutral Bellman recursion; for τ < 1 \tau<1 τ < 1 , it is a nested time-consistent CVaR target, distinct from the static CVaR τ ( ∑ t ≥ 0 γ t r t ) \operatorname{CVaR}_{\tau}\left(\sum_{t\geq0}\gamma^t r_t\right) CVaR τ ( ∑ t ≥ 0 γ t r t ) . All guarantees are stated relative to (3) .
Algorithmic root value estimate. After n n n rollouts, each visited Q-edge ( s , a ) (s,a) ( s , a ) maintains an empirical backup law D ^ T s , a ( n ) ( s , a ) \widehat{\mathcal{D}}_{T_{s,a}(n)}(s,a) D T s , a ( n ) ( s , a ) , where T s , a ( n ) T_{s,a}(n) T s , a ( n ) is its visit count, with empirical CVaR score
c ^ τ , n ( s , a ) : = CVaR τ ( D ^ T s , a ( n ) ( s , a ) ) . \widehat c_{\tau,n}(s,a):=\operatorname{CVaR}_{\tau}\left(\widehat{\mathcal{D}}_{T_{s,a}(n)}(s,a)\right). c τ , n ( s , a ) := CVaR τ ( D T s , a ( n ) ( s , a ) ) .
The scalar V-node estimate is the visit-weighted average
V ^ T s ( n ) τ ( s ) : = ∑ a ∈ A ( s ) T s , a ( n ) T s ( n ) c ^ τ , n ( s , a ) , (4) \widehat V^{\tau}_{T_s(n)}(s):=\sum_{a\in\mathcal{A}(s)}\frac{T_{s,a}(n)}{T_s(n)}\widehat c_{\tau,n}(s,a), \tag*{(4)} V T s ( n ) τ ( s ) := a ∈ A ( s ) ∑ T s ( n ) T s , a ( n ) c τ , n ( s , a ) , (4)
with each action initialized before backup; V ^ n τ ( s 0 ) \widehat V^{\tau}_n(s_0) V n τ ( s 0 ) is the root estimate returned after n n n rollouts.
Risk-sensitive simple regret. The risk-sensitive simple regret relative to (3) is
SR τ ( n ) : = E [ V ~ τ ( s 0 ) − V ^ n τ ( s 0 ) ] , (5) \operatorname{SR}_{\tau}(n):=\mathbb{E}\left[\widetilde V^{\tau}(s_0)-\widehat V^{\tau}_n(s_0)\right], \tag*{(5)} SR τ ( n ) := E [ V τ ( s 0 ) − V n τ ( s 0 ) ] , (5)
and our main theorem controls the stronger absolute error
E τ ( n ) : = E [ ∣ V ^ n τ ( s 0 ) − V ~ τ ( s 0 ) ∣ ] ≥ SR τ ( n ) , (6) \mathcal{E}_{\tau}(n):=\mathbb{E}\left[\left|\widehat V^{\tau}_n(s_0)-\widetilde V^{\tau}(s_0)\right|\right]\geq\operatorname{SR}_{\tau}(n), \tag*{(6)} E τ ( n ) := E [ V n τ ( s 0 ) − V τ ( s 0 ) ] ≥ SR τ ( n ) , (6)
so any O ( n − 1 / 2 ) O(n^{-1/2}) O ( n − 1/2 ) bound on E τ ( n ) \mathcal{E}_{\tau}(n) E τ ( n ) transfers to SR τ ( n ) \operatorname{SR}_{\tau}(n) SR τ ( n ) .
Algorithms
We replace the scalar mean at each Q-edge with an empirical backup law, used both for risk-sensitive exploration (Thompson-sampled CVaR scores) and risk-sensitive backup (empirical Q-edge CVaR scores). V V V -nodes remain scalar, propagating a visit-weighted average of child Q-edge CVaR values.
Distributional Q-Edges
On a rollout, the scalar sample stored at Q-edge ( s h , a ) (s_h,a) ( s h , a ) is
Q ‾ t ( s h , a ) : = r t ( s h , a ) + γ V ^ τ ( s h + 1 ) , (7) \overline{Q}_t(s_h,a) := r_t(s_h,a) + \gamma\widehat{V}^{\tau}(s_{h+1}), \tag*{(7)} Q t ( s h , a ) := r t ( s h , a ) + γ V τ ( s h + 1 ) , (7)
where V ^ τ ( s h + 1 ) \widehat{V}^{\tau}(s_{h+1}) V τ ( s h + 1 ) is the current scalar CVaR-sensitive continuation value. CATSO and PATSO differ only in how they represent the empirical law of these samples.
Categorical Thompson Sampling with Optimism (CATSO) represents the Q-edge law as a categorical distribution on a fixed ordered grid Z N ( s h , a ) = { z 0 ( s h , a ) < ⋯ < z N ( s h , a ) } Z_N(s_h,a)=\{z_0(s_h,a)<\cdots<z_N(s_h,a)\} Z N ( s h , a ) = { z 0 ( s h , a ) < ⋯ < z N ( s h , a )} . It maintains Dirichlet parameters α 0 ( s h , a ) , … , α N ( s h , a ) \alpha^0(s_h,a),\ldots,\alpha^N(s_h,a) α 0 ( s h , a ) , … , α N ( s h , a ) , giving the posterior-center law
D ^ t ( s h , a ) = ∑ i = 0 N p i ( s h , a ) δ z i ( s h , a ) , p i ( s h , a ) = α i ( s h , a ) ∑ ℓ = 0 N α ℓ ( s h , a ) . (8) \widehat{D}_t(s_h,a)=\sum_{i=0}^{N}p_i(s_h,a)\delta_{z_i(s_h,a)},\qquad p_i(s_h,a)=\frac{\alpha^i(s_h,a)}{\sum_{\ell=0}^{N}\alpha^\ell(s_h,a)}. \tag*{(8)} D t ( s h , a ) = i = 0 ∑ N p i ( s h , a ) δ z i ( s h , a ) , p i ( s h , a ) = ∑ ℓ = 0 N α ℓ ( s h , a ) α i ( s h , a ) . (8)
When a backed-up sample Q ‾ t ( s h , a ) \overline{Q}_t(s_h,a) Q t ( s h , a ) is observed, CATSO assigns it to the nearest grid atom and increments the corresponding Dirichlet parameter.
The theoretical analysis uses a fixed grid chosen before search. In experiments, we also consider an adaptive-grid implementation, described in Appendix § 14 (Figure 1 ), which expands the support range when new backed-up samples fall outside the current interval. This adaptive variant is treated as an implementation heuristic, introducing the projection bias of Remark 5.7, and is not the object of the fixed-grid Dirichlet posterior-tail proof.
Figure 1. Comparison of CATSO (left) and PATSO (right). Both methods share rollout, expansion, and SimulateV (which backs up scalar V-nodes using a visit-weighted average of child CVaR). They differ in their Q-edge representation (SimulateQ): CATSO uses categorical Dirichlet atoms, whereas PATSO uses adaptive weighted particles. SelectAction relies on the CVaR of a Thompson-sampled Q-edge plus a polynomial bonus B ( n , s , a ) B(n,s,a) B ( n , s , a ) (Eq. (11) ). For expected-value planning, τ = 1 \tau=1 τ = 1 .
Particle Thompson Sampling with Optimism (PATSO) uses adaptive weighted particles. With current support S ( s h , a ) = { x 1 , … , x M } S(s_h,a)=\{x_1,\ldots,x_M\} S ( s h , a ) = { x 1 , … , x M } and weights α j \alpha^j α j ,
D ^ t ( s h , a ) = ∑ j = 1 M p ‾ j ( s h , a ) δ x j ( s h , a ) , p ‾ j ( s h , a ) = α j ( s h , a ) ∑ ℓ = 1 M α ℓ ( s h , a ) . (9) \widehat{D}_t(s_h,a)=\sum_{j=1}^{M}\overline{p}_j(s_h,a)\delta_{x_j(s_h,a)},\qquad\overline{p}_j(s_h,a)=\frac{\alpha^j(s_h,a)}{\sum_{\ell=1}^{M}\alpha^\ell(s_h,a)}. \tag*{(9)} D t ( s h , a ) = j = 1 ∑ M p j ( s h , a ) δ x j ( s h , a ) , p j ( s h , a ) = ∑ ℓ = 1 M α ℓ ( s h , a ) α j ( s h , a ) . (9)
Uncapped PATSO represents the empirical law exactly; capped PATSO keeps at most U U U particles per edge, adding the compression bias of Remark 5.8.
Discrete CVaR Computation
Both algorithms only evaluate CVaR of finitely supported laws. For P = ∑ i = 1 m w i δ x i P=\sum_{i=1}^{m}w_i\delta_{x_i} P = ∑ i = 1 m w i δ x i with x 1 ≤ ⋯ ≤ x m x_1\le\cdots\le x_m x 1 ≤ ⋯ ≤ x m and ∑ i w i = 1 \sum_i w_i=1 ∑ i w i = 1 , set c j : = ∑ i = 1 j w i c_j:=\sum_{i=1}^{j}w_i c j := ∑ i = 1 j w i , c 0 : = 0 c_0:=0 c 0 := 0 , k : = min { j : c j ≥ τ } k:=\min\{j:c_j\ge\tau\} k := min { j : c j ≥ τ } . Then
CVaR τ ( P ) = 1 τ ( ∑ i = 1 k − 1 w i x i + ( τ − c k − 1 ) x k ) . (10) \operatorname{CVaR}_{\tau}(P)=\frac{1}{\tau}\left(\sum_{i=1}^{k-1}w_i x_i+(\tau-c_{k-1})x_k\right). \tag*{(10)} CVaR τ ( P ) = τ 1 ( i = 1 ∑ k − 1 w i x i + ( τ − c k − 1 ) x k ) . (10)
Since CATSO atoms and PATSO particles are stored sorted, this costs linear time in the support size.
CVaR Thompson Selection
At node s h s_h s h , each action draws a Dirichlet sample over its current support — atoms { z i } \{z_i\} { z i } for CATSO, particles { x j } \{x_j\} { x j } for PATSO — yielding the Thompson-sampled CVaR score c ~ τ ( s h , a ) : = CVaR τ ( ∑ i L i ( s h , a ) δ ⋅ ) \widetilde{c}_{\tau}(s_h,a):=\operatorname{CVaR}_{\tau}(\sum_i L_i(s_h,a)\delta_{\cdot}) c τ ( s h , a ) := CVaR τ ( ∑ i L i ( s h , a ) δ ⋅ ) . The selected action is
a t ∈ arg max a ∈ A ( s h ) { c ~ τ ( s h , a ) + B ( n , s h , a ) } , B ( n , s h , a ) : = C T s h ( n ) 1 / 4 ( T s h , a ( n ) ∨ 1 ) 1 / 2 . (11) a_t\in\arg\max_{a\in A(s_h)}\left\{\widetilde{c}_{\tau}(s_h,a)+B(n,s_h,a)\right\},\qquad B(n,s_h,a):=C\frac{T_{s_h}(n)^{1/4}}{(T_{s_h,a}(n)\vee1)^{1/2}}. \tag*{(11)} a t ∈ arg a ∈ A ( s h ) max { c τ ( s h , a ) + B ( n , s h , a ) } , B ( n , s h , a ) := C ( T s h , a ( n ) ∨ 1 ) 1/2 T s h ( n ) 1/4 . (11)
The polynomial bonus follows the finite-sample MCTS analysis of Shah et al. [27 ] : Thompson sampling drives distributional exploration, while the bonus prevents premature underexploration as descendant values still evolve.
Pseudocode
Figure 1 gives complete pseudocode. The two algorithms share tree traversal, rollout, and scalar V-backup; they differ only in the Q-edge representation and update. CATSO initializes every Q-edge with the unit Dirichlet prior ( 1 , … , 1 ) (1,\ldots,1) ( 1 , … , 1 ) ; PATSO starts with an empty particle set, with at least one forced visit per action before any Dirichlet draw so that α ( s , a ) \alpha(s,a) α ( s , a ) is nonempty.
CVaR-Average V-Backup
Define the empirical CVaR score at each Q-edge:
c ^ τ , t ( s h , a ) : = CVaR τ ( D ^ t ( s h , a ) ) , (12) \widehat{c}_{\tau,t}(s_h,a) := \operatorname{CVaR}_{\tau}\left(\widehat{\mathcal{D}}_t(s_h,a)\right), \tag*{(12)} c τ , t ( s h , a ) := CVaR τ ( D t ( s h , a ) ) , (12)
and back up at V-nodes a visit-weighted average of child Q-edge CVaR scores, so that the parent Q-edge sample is
V ^ T s h ( t ) τ ( s h ) : = ∑ a ∈ A T s h , a ( t ) T s h ( t ) c ^ τ , t ( s h , a ) , Q ‾ t ( s h , a ) = r t ( s h , a ) + γ V ^ T s h + 1 ( t ) τ ( s h + 1 ) . (13) \widehat{V}^{\tau}_{T_{s_h}(t)}(s_h) := \sum_{a\in\mathcal{A}} \frac{T_{s_h,a}(t)}{T_{s_h}(t)}\widehat{c}_{\tau,t}(s_h,a), \qquad\overline{Q}_t(s_h,a) = r_t(s_h,a) + \gamma\widehat{V}^{\tau}_{T_{s_{h+1}}(t)}(s_{h+1}). \tag*{(13)} V T s h ( t ) τ ( s h ) := a ∈ A ∑ T s h ( t ) T s h , a ( t ) c τ , t ( s h , a ) , Q t ( s h , a ) = r t ( s h , a ) + γ V T s h + 1 ( t ) τ ( s h + 1 ) . (13)
Distributions thus live only at Q-edges; V-nodes store scalar risk-sensitive summaries, avoiding distributional convolutions while still propagating lower-tail information up the tree. Note that (13) is an arithmetic average of CVaRs, not the CVaR of the mixture ∑ a w a D ^ ( s , a ) \sum_a w_a\widehat{\mathcal{D}}(s,a) ∑ a w a D ( s , a ) , which differ in general; we use the former because V-nodes are scalar by design. At τ = 1 \tau=1 τ = 1 , CVaR 1 \operatorname{CVaR}_1 CVaR 1 equals the mean, so (13) reduces to the standard average-mean V-backup V ^ T s h ( t ) 1 ( s h ) = ∑ a T s h , a ( t ) T s h ( t ) Q ^ T s h , a ( t ) ( s h , a ) \widehat{V}^{1}_{T_{s_h}(t)}(s_h)=\sum_a \frac{T_{s_h,a}(t)}{T_{s_h}(t)}\widehat{Q}_{T_{s_h,a}(t)}(s_h,a) V T s h ( t ) 1 ( s h ) = ∑ a T s h ( t ) T s h , a ( t ) Q T s h , a ( t ) ( s h , a ) with Q ^ ( s h , a ) = E D ^ t ( s h , a ) [ X ] \widehat{Q}(s_h,a)=\mathbb{E}_{\widehat{\mathcal{D}}_t(s_h,a)}[X] Q ( s h , a ) = E D t ( s h , a ) [ X ] , recovering expected-value planning.
Root Recommendation
After n n n rollouts, the planner returns the scalar root value V ^ n τ ( s 0 ) \widehat{V}^{\tau}_{n}(s_{0}) V n τ ( s 0 ) from (13) and the empirical root Q-edge CVaR action
a ^ n C V a R ∈ arg max a ∈ A ( s 0 ) c ^ τ , n ( s 0 , a ) . (14) \widehat{a}^{\mathrm{CVaR}}_{n} \in\arg\max_{a\in\mathcal{A}(s_{0})} \widehat{c}_{\tau,n}(s_{0},a). \tag*{(14)} a n CVaR ∈ arg a ∈ A ( s 0 ) max c τ , n ( s 0 , a ) . (14)
Remark 4.1. V ^ n τ ( s 0 ) \widehat{V}^{\tau}_{n}(s_{0}) V n τ ( s 0 ) is the object controlled by the main convergence theorem, while (14) is the risk-sensitive root action used in experiments.
Theoretical Analysis
We prove that CATSO and PATSO converge to the finite-depth nested CVaR target V ~ τ ( s 0 ) \widetilde{V}^{\tau}(s_{0}) V τ ( s 0 ) in three steps: (i) a single-node CVaR bandit analysis; (ii) a one-step CVaR lifting lemma showing that child V-concentration implies plug-in Q-edge CVaR concentration; (iii) backward induction over the finite-depth tree. The main quantity controlled is the scalar root estimate
V ^ T s ( n ) τ ( s ) = ∑ a ∈ A ( s ) T s , a ( n ) T s ( n ) c ^ τ , n ( s , a ) , c ^ τ , n ( s , a ) = CVaR τ ( D ^ T s , a ( n ) ( s , a ) ) , \widehat{V}^{\tau}_{T_{s}(n)}(s)=\sum_{a\in\mathcal{A}(s)}\frac{T_{s,a}(n)}{T_{s}(n)}\widehat{c}_{\tau,n}(s,a),\qquad\widehat{c}_{\tau,n}(s,a)=\operatorname{CVaR}_{\tau}\left(\widehat{\mathcal{D}}_{T_{s,a}(n)}(s,a)\right), V T s ( n ) τ ( s ) = a ∈ A ( s ) ∑ T s ( n ) T s , a ( n ) c τ , n ( s , a ) , c τ , n ( s , a ) = CVaR τ ( D T s , a ( n ) ( s , a ) ) ,
whose absolute error upper-bounds the simple regret (5) . The proof of step (i) is given in Appendix § 10 , where we analyze the local CVaR bandit problem and derive the CATSO/PATSO suboptimal-arm pull bounds. Step (ii) is proved in Appendix § 11 .§ 1 through the one-step CVaR lifting lemma. Step (iii) is completed in the rest of Appendix § 11 , which apply backward induction separately to CATSO and PATSO and then translate the root value-error bound into the risk-sensitive simple-regret guarantee.
CVaR-Objective Non-Stationary Bandits
Consider K K K arms with bounded non-stationary payoffs { R a , t } ⊂ [ 0 , R ] \{R_{a,t}\}\subset[0,R] { R a , t } ⊂ [ 0 , R ] and empirical law F ^ a , m : = 1 m ∑ t = 1 m δ R a , t \widehat{F}_{a,m}:=\frac{1}{m}\sum_{t=1}^{m}\delta_{R_{a,t}} F a , m := m 1 ∑ t = 1 m δ R a , t . Assume each F ^ a , m \widehat{F}_{a,m} F a , m stabilizes to a limiting law F a F_{a} F a , and write c a τ : = CVaR τ ( F a ) c^{\tau}_{a}:=\operatorname{CVaR}_{\tau}(F_{a}) c a τ := CVaR τ ( F a ) , c ⋆ τ : = max a c a τ c^{\tau}_{\star}:=\max_{a}c^{\tau}_{a} c ⋆ τ := max a c a τ , a τ ⋆ ∈ arg max a c a τ a^{\star}_{\tau}\in\arg\max_{a}c^{\tau}_{a} a τ ⋆ ∈ arg max a c a τ , c ^ a , m τ : = CVaR τ ( F ^ a , m ) \widehat{c}^{\tau}_{a,m}:=\operatorname{CVaR}_{\tau}(\widehat{F}_{a,m}) c a , m τ := CVaR τ ( F a , m ) . The node-level scalar backup is the visit-weighted average
c ^ n τ : = ∑ a = 1 K T a ( n ) n c ^ a , T a ( n ) τ , (15) \widehat{c}^{\tau}_{n}:=\sum_{a=1}^{K}\frac{T_{a}(n)}{n}\widehat{c}^{\tau}_{a,T_{a}(n)}, \tag*{(15)} c n τ := a = 1 ∑ K n T a ( n ) c a , T a ( n ) τ , (15)
which reduces to the standard average-mean backup at τ = 1 \tau=1 τ = 1 .
Definition 5.1 (Convergence with concentration). ( V ^ n ) n ≥ 1 (\widehat{V}_{n})_{n\geq1} ( V n ) n ≥ 1 converges with concentration to V V V , written plim n → ∞ V ^ n = V \operatorname{plim}_{n\to\infty}\widehat{V}_{n}=V plim n → ∞ V n = V , if lim n E [ V ^ n ] = V \lim_{n}\mathbb{E}[\widehat{V}_{n}]=V lim n E [ V n ] = V and there exists c > 0 c>0 c > 0 with P ( ∣ V ^ n − V ∣ > ϵ ) ≤ c / ( n ϵ 2 ) \mathbb{P}(\lvert\widehat{V}_{n}-V\rvert>\epsilon)\leq c/(n\epsilon^{2}) P (∣ V n − V ∣ > ϵ ) ≤ c / ( n ϵ 2 ) for all n ≥ 1 n\geq1 n ≥ 1 , ϵ > 0 \epsilon>0 ϵ > 0 .
Assumption 5.2 (Asymptotic CVaR stationarity with concentration). For every arm a a a , plim m → ∞ c ^ a , m τ = c a τ \operatorname{plim}_{m\to\infty}\widehat{c}^{\tau}_{a,m}=c^{\tau}_{a} plim m → ∞ c a , m τ = c a τ , and the CVaR maximizer a τ ⋆ ∈ arg max a c a τ a^{\star}_{\tau}\in\arg\max_{a}c^{\tau}_{a} a τ ⋆ ∈ arg max a c a τ is unique.
Remark 5.3 (Role of Assumption 5.2 in MCTS). Assumption 5.2 describes the local bandit problem at each V-node of the MCTS tree: fixing s h s_{h} s h , each a ∈ A ( s h ) a\in\mathcal{A}(s_{h}) a ∈ A ( s h ) is an arm with empirical score c ^ a , m τ : = Q ^ m τ ( s h , a ) \widehat{c}^{\tau}_{a,m}:=\widehat{Q}^{\tau}_{m}(s_{h},a) c a , m τ := Q m τ ( s h , a ) and limiting value c a τ : = Q ~ τ ( s h , a ) c^{\tau}_{a}:=\widetilde{Q}^{\tau}(s_{h},a) c a τ := Q τ ( s h , a ) . The armwise convergence is therefore not an extra modeling assumption: it is verified by backward induction via Lemma 5.5. If plim n → ∞ V ^ n τ ( s h + 1 ) = V ~ τ ( s h + 1 ) \operatorname{plim}_{n\to\infty}\widehat{V}^{\tau}_{n}(s_{h+1})=\widetilde{V}^{\tau}(s_{h+1}) plim n → ∞ V n τ ( s h + 1 ) = V τ ( s h + 1 ) for every child s h + 1 s_{h+1} s h + 1 , the lemma yields plim n → ∞ Q ^ n τ ( s h , a ) = Q ~ τ ( s h , a ) \operatorname{plim}_{n\to\infty}\widehat{Q}^{\tau}_{n}(s_{h},a)=\widetilde{Q}^{\tau}(s_{h},a) plim n → ∞ Q n τ ( s h , a ) = Q τ ( s h , a ) . The terminal rollout estimator starts the induction, and the one-step lifting propagates it upward through the finite-depth tree. Uniqueness of a τ ⋆ a^{\star}_{\tau} a τ ⋆ is a standard nondegeneracy condition simplifying the suboptimal-pull bounds; ties can be handled by replacing a τ ⋆ a^{\star}_{\tau} a τ ⋆ with the optimal set A τ ⋆ : = arg max a c a τ \mathcal{A}^{\star}_{\tau}:=\arg\max_{a}c^{\tau}_{a} A τ ⋆ := arg max a c a τ and treating only actions outside it as suboptimal.
Both algorithms select using the Thompson-sampled CVaR score c ~ a τ ( t ) : = CVaR τ ( D ~ a , T a ( t ) ) \widetilde{c}^{\tau}_{a}(t):=\operatorname{CVaR}_{\tau}(\widetilde{\mathcal{D}}_{a,T_{a}(t)}) c a τ ( t ) := CVaR τ ( D a , T a ( t ) ) plus the polynomial bonus,
I ( t ) ∈ arg max a ∈ [ K ] { c ~ a τ ( t ) + C t 1 / 4 ( T a ( t ) ∨ 1 ) 1 / 2 } . (16) I(t)\in\arg\max_{a\in[K]}\left\{\widetilde{c}^{\tau}_{a}(t)+C\frac{t^{1/4}}{(T_{a}(t)\vee1)^{1/2}}\right\}. \tag*{(16)} I ( t ) ∈ arg a ∈ [ K ] max { c a τ ( t ) + C ( T a ( t ) ∨ 1 ) 1/2 t 1/4 } . (16)
For a probability law F F F on [ 0 , R max ] [0,R_{\max}] [ 0 , R m a x ] and threshold c ∈ R c\in\mathbb{R} c ∈ R , define the upper-tail risk information number K i n f τ , + ( F , c ) : = inf { KL ( F ∥ G ) : G ∈ P ( [ 0 , R max ] ) , CVaR τ ( G ) ≥ c } \mathcal{K}^{\tau,+}_{\mathrm{inf}}(F,c):=\inf\{\operatorname{KL}(F\|G):G\in\mathcal{P}([0,R_{\max}]),\ \operatorname{CVaR}_{\tau}(G)\geq c\} K inf τ , + ( F , c ) := inf { KL ( F ∥ G ) : G ∈ P ([ 0 , R m a x ]) , CVaR τ ( G ) ≥ c } , and its grid-restricted version on Z N = { z 0 < ⋯ < z N } ⊂ [ 0 , R max ] \mathcal{Z}_{N}=\{z_{0}<\cdots<z_{N}\}\subset[0,R_{\max}] Z N = { z 0 < ⋯ < z N } ⊂ [ 0 , R m a x ] , K i n f ( N ) , τ , + ( F , c ) : = inf { KL ( F ∥ G ) : G ∈ P ( Z N ) , CVaR τ ( G ) ≥ c } \mathcal{K}^{(N),\tau,+}_{\mathrm{inf}}(F,c):=\inf\{\operatorname{KL}(F\|G):G\in\mathcal{P}(\mathcal{Z}_{N}),\ \operatorname{CVaR}_{\tau}(G)\geq c\} K inf ( N ) , τ , + ( F , c ) := inf { KL ( F ∥ G ) : G ∈ P ( Z N ) , CVaR τ ( G ) ≥ c } .
Theorem 5.4 (CATSO/PATSO in CVaR non-stationary bandits). Suppose Assumption 5.2 holds, each arm is sampled once initially, and selection follows (16) . Under the fixed-grid CATSO conditions of Appendix C.3 (resp. uncapped PATSO conditions of Appendix C.5), there exists C B < ∞ C_B < \infty C B < ∞ such that
E [ ∣ c ^ n τ − c ∗ τ ∣ ] ≤ C B n − 1 / 2 , E [ T a ( n ) ] ≤ ( 1 + ϵ 0 ) log n K inf ∙ , τ , + ( F a , c ∗ τ ) + O ( n 1 / 2 ) , \mathbb{E}\left[\left|\widehat{c}_n^\tau-c_*^\tau\right|\right]\le C_B n^{-1/2},\qquad\mathbb{E}[T_a(n)]\le\frac{(1+\epsilon_0)\log n}{\mathcal{K}_{\inf}^{\bullet,\tau,+}(F_a,c_*^\tau)}+O(n^{1/2}), E [ ∣ c n τ − c ∗ τ ∣ ] ≤ C B n − 1/2 , E [ T a ( n )] ≤ K i n f ∙ , τ , + ( F a , c ∗ τ ) ( 1 + ϵ 0 ) log n + O ( n 1/2 ) , for every CVaR-suboptimal a ≠ a ∗ τ a\ne a_*^\tau a = a ∗ τ and ϵ 0 > 0 \epsilon_0>0 ϵ 0 > 0 , where ∙ = ( N ) \bullet=(N) ∙ = ( N ) for CATSO and is omitted for PATSO. The proof rests on the decomposition
∣ c ^ n τ − c ∗ τ ∣ ≤ ∑ a = 1 K T a ( n ) n ∣ c ^ a , T a ( n ) τ − c a τ ∣ + R ∑ a ≠ a ∗ τ T a ( n ) n . \left|\widehat{c}_n^\tau-c_*^\tau\right|\le\sum_{a=1}^{K}\frac{T_a(n)}{n}\left|\widehat{c}_{a,T_a(n)}^\tau-c_a^\tau\right|+R\sum_{a\ne a_*^\tau}\frac{T_a(n)}{n}. ∣ c n τ − c ∗ τ ∣ ≤ a = 1 ∑ K n T a ( n ) c a , T a ( n ) τ − c a τ + R a = a ∗ τ ∑ n T a ( n ) . The first term is controlled by Assumption 5.2; the second by the CVaR suboptimal-arm pull bound—the CVaR-average analogue of the standard scalar MCTS argument.
One-Step CVaR Lifting
Why does MCTS require a non-stationary bandit formulation together with an asymptotic CVaR stationarity condition (Assumption 5.2)? Unlike variation-budget or change-point models for exogenous environment shifts, the non-stationarity in MCTS is endogenous to the search : at an internal node s h s_h s h , pulling action a a a yields a backed-up sample Q ‾ i = R i + γ V ^ N i τ ( S i ′ ) \overline{Q}_i=R_i+\gamma\widehat{V}_{N_i}^\tau(S'_i) Q i = R i + γ V N i τ ( S i ′ ) , whose distribution drifts as the descendant value estimate V ^ N i τ ( S i ′ ) \widehat{V}_{N_i}^\tau(S'_i) V N i τ ( S i ′ ) evolves. Because edge samples are not i.i.d. draws from a fixed law, stationary CVaR bandit guarantees [3 , 25 ] cannot be applied recursively as-is. Instead, following the non-stationary MCTS paradigm established for expected-value and power-mean planning [27 , 8 ] , each node requires that its drifting arm payoffs eventually stabilize with polynomial concentration—ruling out persistent or adversarial drift. Assumption 5.2 is the CVaR analogue of this local stabilization condition, replacing empirical-mean convergence with empirical-CVaR convergence. Crucially, its armwise concentration requirement is an inductive property verified across the tree rather than an external environmental assumption: Lemma 5.5 below provides the induction step, showing that scalar CVaR concentration at child nodes (s h + 1 s_{h+1} s h + 1 ) implies W 1 W_1 W 1 concentration of the plug-in backup law at ( s h , a ) (s_h,a) ( s h , a ) , which by CVaR Lipschitzness verifies Assumption 5.2 at depth h h h .
Lemma 5.5 (One-step CVaR lifting for plug-in Q-edge laws). Fix ( s h , a ) (s_h,a) ( s h , a ) and let ( X i , S i ) i ≥ 1 (X_i,S_i)_{i\ge1} ( X i , S i ) i ≥ 1 be i.i.d. one-step samples from the reward-transition kernel at ( s h , a ) (s_h,a) ( s h , a ) , with X i ∈ [ 0 , R max ] X_i\in[0,R_{\max}] X i ∈ [ 0 , R m a x ] and S i ∼ P ( ⋅ ∣ s h , a ) S_i\sim P(\cdot\mid s_h,a) S i ∼ P ( ⋅ ∣ s h , a ) . Assume P ( ⋅ ∣ s h , a ) P(\cdot\mid s_h,a) P ( ⋅ ∣ s h , a ) has finite support { s h + 1 ( 1 ) , … , s h + 1 ( M ) } \{s_{h+1}^{(1)},\ldots,s_{h+1}^{(M)}\} { s h + 1 ( 1 ) , … , s h + 1 ( M ) } with p m : = P ( s h + 1 ( m ) ∣ s h , a ) > 0 p_m:=P(s_{h+1}^{(m)}\mid s_h,a)>0 p m := P ( s h + 1 ( m ) ∣ s h , a ) > 0 , p min : = min m p m p_{\min}:=\min_m p_m p m i n := min m p m , and let N n m : = ∑ i = 1 n 1 { S i = s h + 1 ( m ) } N_n^m:=\sum_{i=1}^n\mathbf{1}\{S_i=s_{h+1}^{(m)}\} N n m := ∑ i = 1 n 1 { S i = s h + 1 ( m ) } . For each s h + 1 ( m ) s_{h+1}^{(m)} s h + 1 ( m ) , let V τ ( s h + 1 ( m ) ) V^\tau(s_{h+1}^{(m)}) V τ ( s h + 1 ( m ) ) be the limiting scalar CVaR value and V ^ k τ ( s h + 1 ( m ) ) \widehat{V}_k^\tau(s_{h+1}^{(m)}) V k τ ( s h + 1 ( m ) ) its estimator after k k k local visits, with
0 ≤ V ^ k τ ( s h + 1 ( m ) ) , V τ ( s h + 1 ( m ) ) ≤ R max 1 − γ , lim k → ∞ V ^ k τ ( s h + 1 ( m ) ) = V τ ( s h + 1 ( m ) ) . 0\le\widehat{V}_k^\tau(s_{h+1}^{(m)}),V^\tau(s_{h+1}^{(m)})\le\frac{R_{\max}}{1-\gamma},\qquad\lim_{k\to\infty}\widehat{V}_k^\tau(s_{h+1}^{(m)})=V^\tau(s_{h+1}^{(m)}). 0 ≤ V k τ ( s h + 1 ( m ) ) , V τ ( s h + 1 ( m ) ) ≤ 1 − γ R m a x , k → ∞ lim V k τ ( s h + 1 ( m ) ) = V τ ( s h + 1 ( m ) ) . Define plug-in and oracle backed-up samples q ‾ i ( n ) : = X i + γ V ^ N S i n τ ( S i ) \overline{q}_i^{(n)}:=X_i+\gamma\widehat{V}_{N_{S_i}^n}^\tau(S_i) q i ( n ) := X i + γ V N S i n τ ( S i ) , G i : = X i + γ V τ ( S i ) G_i:=X_i+\gamma V^\tau(S_i) G i := X i + γ V τ ( S i ) , where N S i n = N n m N_{S_i}^n=N_n^m N S i n = N n m when S i = s h + 1 ( m ) S_i=s_{h+1}^{(m)} S i = s h + 1 ( m ) , with empirical and population laws D ^ n ( s h , a ) : = 1 n ∑ i = 1 n δ q ‾ i ( n ) \widehat{D}_n(s_h,a):=\frac{1}{n}\sum_{i=1}^n\delta_{\overline{q}_i^{(n)}} D n ( s h , a ) := n 1 ∑ i = 1 n δ q i ( n ) , and D τ ( s h , a ) D^\tau(s_h,a) D τ ( s h , a ) is defined in (2) . The corresponding CVaR Q-values Q ^ n τ ( s h , a ) : = CVaR τ ( D ^ n ( s h , a ) ) \widehat{Q}_n^\tau(s_h,a):=\operatorname{CVaR}_\tau(\widehat{D}_n(s_h,a)) Q n τ ( s h , a ) := CVaR τ ( D n ( s h , a )) , Q τ ( s h , a ) : = CVaR τ ( D τ ( s h , a ) ) Q^\tau(s_h,a):=\operatorname{CVaR}_\tau(D^\tau(s_h,a)) Q τ ( s h , a ) := CVaR τ ( D τ ( s h , a )) . Then
plim n → ∞ Q ^ n τ ( s h , a ) = Q τ ( s h , a ) . \operatorname*{plim}_{n\to\infty}\widehat{Q}_n^\tau(s_h,a)=Q^\tau(s_h,a). n → ∞ plim Q n τ ( s h , a ) = Q τ ( s h , a ) .
From Bandits to MCTS
For each depth h h h , set Q ^ n τ ( s h , a ) : = CVaR τ ( D ^ n ( s h , a ) ) \widehat{Q}_n^\tau(s_h,a):=\operatorname{CVaR}_\tau(\widehat{D}_n(s_h,a)) Q n τ ( s h , a ) := CVaR τ ( D n ( s h , a )) and
V ^ n τ ( s h ) : = ∑ a T s h , a ( n ) T s h ( n ) Q ^ n , T s h , a ( n ) τ ( s h , a ) . \widehat{V}_n^\tau(s_h):=\sum_a\frac{T_{s_h,a}(n)}{T_{s_h}(n)}\widehat{Q}_{n,T_{s_h,a}(n)}^\tau(s_h,a). V n τ ( s h ) := a ∑ T s h ( n ) T s h , a ( n ) Q n , T s h , a ( n ) τ ( s h , a ) .
Theorem 5.6 (Convergence of CATSO and uncapped PATSO). Run CATSO (resp. uncapped PATSO) on a finite-depth tree of depth H H H with bounded rewards and finite transition support at every Q-edge. Assume the terminal rollout estimator has an O ( n − 1 / 2 ) L 1 O(n^{-1/2})L^1 O ( n − 1/2 ) L 1 rate, the local CVaR-optimal action at each internal node is unique, and Theorem 5.4 applies at every internal node. Then there exists C ′ > 0 C'>0 C ′ > 0 such that
E [ ∣ V ^ n τ ( s 0 ) − V ~ τ ( s 0 ) ∣ ] ≤ C ′ n − 1 / 2 , \mathbb{E}\left[\left|\widehat{V}_n^\tau(s_0)-\widetilde{V}^\tau(s_0)\right|\right]\le C'n^{-1/2}, E [ V n τ ( s 0 ) − V τ ( s 0 ) ] ≤ C ′ n − 1/2 ,
for the fixed-grid CATSO target and the uncapped PATSO target. For CATSO against the unprojected target with grid mesh Δ N \Delta_N Δ N , the bound becomes C ′ n − 1 / 2 + O ( Δ N / τ ) C'n^{-1/2} + O(\Delta_N/\tau) C ′ n − 1/2 + O ( Δ N / τ ) .
Since S R τ ( n ) ≤ E [ ∣ V ~ n τ ( s 0 ) − V ~ τ ( s 0 ) ∣ ] \mathrm{SR}_{\tau}(n) \leq\mathbb{E}[|\widetilde{V}^{\tau}_{n}(s_0) - \widetilde{V}^{\tau}(s_0)|] SR τ ( n ) ≤ E [ ∣ V n τ ( s 0 ) − V τ ( s 0 ) ∣ ] , Theorem 5.6 yields S R τ ( n ) = O ( n − 1 / 2 ) \mathrm{SR}_{\tau}(n) = O(n^{-1/2}) SR τ ( n ) = O ( n − 1/2 ) up to the stated approximation terms.
Remark 5.7 (CATSO discretization bias). The exact O ( n − 1 / 2 ) O(n^{-1/2}) O ( n − 1/2 ) CATSO statement is for the fixed-grid target; against the unprojected backup law with mesh Δ N \Delta_N Δ N , CVaR Lipschitzness adds an O ( Δ N / τ ) O(\Delta_N/\tau) O ( Δ N / τ ) bias, with depth-dependent constants absorbed.
Remark 5.8 (Capped PATSO). Theorem 5.6 concerns uncapped PATSO. If capped PATSO satisfies W 1 ( D ^ m , U c a p , D ^ m u n c a p ) ≤ ε U ( s , a ) W_1(\widehat{\mathcal{D}}^{\mathrm{cap}}_{m,U}, \widehat{\mathcal{D}}^{\mathrm{uncap}}_m) \leq\varepsilon_U(s,a) W 1 ( D m , U cap , D m uncap ) ≤ ε U ( s , a ) , then every CVaR estimate gains an additional ε U ( s , a ) / τ \varepsilon_U(s,a)/\tau ε U ( s , a ) / τ bias, and the root bound becomes C ′ n − 1 / 2 + O ( ε U / τ ) C'n^{-1/2} + O(\varepsilon_U/\tau) C ′ n − 1/2 + O ( ε U / τ ) .
Memory and Computation
CATSO. Each Q-edge stores N + 1 N + 1 N + 1 atoms with Dirichlet parameters, costing O ( N ) O(N) O ( N ) per edge. Selection draws one Dirichlet vector and computes the discrete CVaR of the sorted categorical law in O ( N ) O(N) O ( N ) . Total memory after n n n rollouts is O ( N ∣ E n ∣ ) O(N|\mathcal{E}_n|) O ( N ∣ E n ∣ ) over the explored Q-edges E n \mathcal{E}_n E n .
PATSO. Uncapped PATSO stores one weighted particle per distinct backed-up return (O ( m ) O(m) O ( m ) after m m m visits), representing the empirical Q-edge law exactly. For bounded memory, we cap at U U U particles per edge: when insertion would exceed U U U , the closest adjacent pair ( x i , α i ) , ( x i + 1 , α i + 1 ) (x_i,\alpha_i),(x_{i+1},\alpha_{i+1}) ( x i , α i ) , ( x i + 1 , α i + 1 ) is merged into
( α i x i + α i + 1 x i + 1 α i + α i + 1 , α i + α i + 1 ) \left(\frac{\alpha_i x_i+\alpha_{i+1}x_{i+1}}{\alpha_i+\alpha_{i+1}},\ \alpha_i+\alpha_{i+1}\right) ( α i + α i + 1 α i x i + α i + 1 x i + 1 , α i + α i + 1 )
preserving the first moment. Capped PATSO uses O ( U ) O(U) O ( U ) memory and O ( U ) O(U) O ( U ) CVaR-selection cost per edge, with the compression bias quantified by Remark 5.8; increasing U U U trades memory for a smaller bias.
Experiments
We evaluate whether distributional Q-edge information improves planning when expected return and lower-tail return favor different actions. Each benchmark contains a safe option with modest reward and a risky option with attractive mean but poor lower-tail outcomes — directly testing the paper’s central claim that scalar MCTS does not preserve enough information for CVaR. All methods plan from the same initial state with the same rollout policy, search depth, and budget; the resulting root recommendation or value estimate is evaluated by independent Monte-Carlo rollouts. Detailed information about the environments, algorithms, and setup is provided in Appendix F.
Experimental Questions
Q1 (lower-tail return). Does Monte-Carlo CVaR return of CATSO/PATSO exceed that of scalar baselines? Q2 (catastrophic outcomes). How often do recommended policies encounter environment-specific catastrophic events (bankruptcy, trap cells, severe-cost transitions, cliff falls)? Q3 (root estimate quality). For oracle-evaluable environments, does the root CVaR value gap — the empirical analogue of the simple regret (5) — shrink with planning budget? Another interesting question that one can raise is how sensitive the algorithm performance according to the representation of the empirical distribution, because of the space limit, we postpone the answer and leave it in Appendix F.
Environments
We evaluate on five stochastic benchmarks designed such that the mean-optimal and CVaR-optimal actions differ.
Autonomous Vehicle [24 ] : A road network gridworld where faster routes (like highways) typically incur low costs but carry a risk of rare, extreme penalties (e.g., crashes). Two-Level Risky Treasure: A two-step environment forcing a choice between a guaranteed safe reward or a risky branch leading to either an extreme payout or a severe penalty. Risky Shortcut Gridworld: A navigation task offering a long, safe route versus a short, hazardous route where stochastic wind can push the agent off a cliff. Betting Game [4 ] : A repeated fractional betting scenario where aggressive wagers maximize expected growth but risk bankruptcy. Guarded Maze [14 , 20 , 21 ] : A maze featuring a short path with a high risk of triggering a severe penalty from a guard, contrasted with a longer, entirely safe alternative.
Algorithms and Baselines
We compare CATSO and PATSO with scalar MCTS baselines (UCT-style planners and max/power-backup variants), which store scalar Q-edge statistics and select on scalar value estimates. CATSO and PATSO instead store empirical Q-edge laws and select via the Thompson-sampled CVaR score plus the polynomial bonus (11) . For scalar baselines, CVaR is applied only at evaluation time — the planner recommends a root action and we estimate its CVaR with independent rollouts — so all methods are compared on the same risk-sensitive objective.
Metrics
CVaR value gap. For oracle-evaluable environments, we compute the quantity Gap ^ τ ( n ) = V ^ τ ( s 0 ) − V ^ n τ ( s 0 ) \widehat{\operatorname{Gap}}_{\tau}(n) = \widehat{V}^{\tau}(s_0) - \widehat{V}^{\tau}_{n}(s_0) Gap τ ( n ) = V τ ( s 0 ) − V n τ ( s 0 ) (lower is better) at the root node.
Monte-Carlo CVaR return. Lower-tail CVaR of the return distribution induced by the recommended root action follows (14) (higher is better). Note that we use static CVaR for evaluation, that is we take the average among τ \tau τ -worst trajectories in the Monte-Carlo evaluation.
Catastrophic count. Number of evaluation rollouts that encounter the environment-specific catastrophic event (lower is better; an interpretable safety metric, not a replacement for CVaR).
Results
Figures 2a–3 summarize results across planning budgets for three environments, since the paper page limit is restricted, we refer to table 4 in appendix F for full results. CATSO and PATSO consistently improve the metrics that depend on lower-tail behavior. Scalar baselines, compressing each Q-edge to scalar statistics, are repeatedly drawn to high-mean branches with poor tails — favoring higher-cost path in Autonomous Vehicle and shorter risky routes in the Guarded Maze and Risky Shortcut Gridworld. By contrast, the distributional methods preserve enough information to identify actions with similar or even worse means but safer lower tails.
The two distributional methods are complementary: CATSO is stable when the categorical grid is fine but incurs discretization bias on sharp or widely separated modes, whereas PATSO adapts its support to observed backed-up returns and handles sparse or multimodal laws naturally, with capping trading memory for a small compression bias (Remark 5.8).
Takeaways
Three conclusions emerge. (i) Maintaining empirical Q-edge laws is useful in practice, not only in theory. (ii) The gains of CATSO and PATSO show up precisely on lower-tail metrics — CVaR value gap, Monte-Carlo CVaR return, catastrophic count. (iii) Scalar baselines remain competitive on expected return, indicating that the improvement reflects preserved tail information, not better exploration.
Discussion
CATSO and PATSO show that MCTS can reason about lower-tail risk without storing full return distributions at every node. The key design choice is to keep V-nodes scalar while maintaining empirical backup laws at Q-edges. This allows the planner to use CVaR-sensitive Thompson sampling for exploration and a CVaR-average scalar backup for value propagation. As a result, the methods preserve tail information that scalar mean-based MCTS discards, while retaining the standard O ( n − 1 / 2 ) O(n^{-1/2}) O ( n − 1/2 ) finite-depth convergence rate up to representation bias.
The theory is stated for a finite-depth nested CVaR target. This target is appropriate for risk-sensitive tree search, but it is not the same as static CVaR of the full infinite-horizon trajectory return. CATSO also depends on a finite grid and therefore incurs discretization bias, while capped PATSO trades memory for a tunable W 1 W_1 W 1 compression bias. Future work should study distributional V-nodes, adaptive risk levels, learned model dynamics, and extensions to partially observed or continuous-control settings.
Figure 3. Catastrophic-event count across evaluation rollouts. Lower is better.
Notations
Table 1 provides an overview of the standard notation adopted throughout this work. Context-specific notation, when required, will be explicitly defined within the corresponding definition.
Table 1. General notation.
CVaR and Wasserstein Preliminaries
Discrete CVaR Formula
Derivation for (1) . Recall the quantile function F X − 1 ( u ) = inf { x : F X ( x ) ≥ u } F_X^{-1}(u) = \inf\{x : F_X(x) \ge u\} F X − 1 ( u ) = inf { x : F X ( x ) ≥ u } for u ∈ ( 0 , 1 ] u \in(0,1] u ∈ ( 0 , 1 ] . By the inverse-transform identity, if U ∼ Unif ( 0 , 1 ) U \sim\operatorname{Unif}(0,1) U ∼ Unif ( 0 , 1 ) then F X − 1 ( U ) = d X F_X^{-1}(U) \mathrel{\mathop{=}\limits^{d}} X F X − 1 ( U ) = d X , so conditioning on U ≤ τ U \le\tau U ≤ τ samples the lower τ \tau τ -tail of X X X . Averaging yields
CVaR τ ( X ) = E [ F X − 1 ( U ) ∣ U ≤ τ ] = 1 τ ∫ 0 τ F X − 1 ( u ) d u , (17) \operatorname{CVaR}_{\tau}(X) = \mathbb{E}\left[F_X^{-1}(U) \mid U \le\tau\right] = \frac{1}{\tau}\int_{0}^{\tau} F_X^{-1}(u)\,du, \tag*{(17)} CVaR τ ( X ) = E [ F X − 1 ( U ) ∣ U ≤ τ ] = τ 1 ∫ 0 τ F X − 1 ( u ) d u , (17)
which, when F X F_X F X is continuous and strictly increasing, reduces to the familiar E [ X ∣ X ≤ VaR τ ( X ) ] \mathbb{E}[X \mid X \le\operatorname{VaR}_{\tau}(X)] E [ X ∣ X ≤ VaR τ ( X )] .
An equivalent Rockafellar–Uryasev variational form,
CVaR τ ( X ) = sup η ∈ R { η − 1 τ E [ ( η − X ) + ] } , (18) \operatorname{CVaR}_{\tau}(X) = \sup_{\eta\in\mathbb{R}}\left\{\eta- \frac{1}{\tau}\mathbb{E}\left[(\eta-X)_{+}\right]\right\}, \tag*{(18)} CVaR τ ( X ) = η ∈ R sup { η − τ 1 E [ ( η − X ) + ] } , (18)
handles atoms cleanly and is the form typically used for optimization.
From (1) to (10) . For P = ∑ i = 1 m w i δ x i P = \sum_{i=1}^{m} w_i\delta_{x_i} P = ∑ i = 1 m w i δ x i with x 1 ≤ ⋯ ≤ x m x_1 \le\cdots\le x_m x 1 ≤ ⋯ ≤ x m , the CDF F ( x ) = ∑ i : x i ≤ x w i F(x) = \sum_{i:x_i\le x} w_i F ( x ) = ∑ i : x i ≤ x w i is a right-continuous step function, so the quantile function is piecewise constant:
F X − 1 ( u ) = x j for u ∈ ( c j − 1 , c j ] , j = 1 , … , m , (19) F_X^{-1}(u) = x_j \qquad\text{for } u \in(c_{j-1},c_j],\quad j=1,\ldots,m, \tag*{(19)} F X − 1 ( u ) = x j for u ∈ ( c j − 1 , c j ] , j = 1 , … , m , (19)
where c j = ∑ i = 1 j w i c_j = \sum_{i=1}^{j} w_i c j = ∑ i = 1 j w i and c 0 = 0 c_0 = 0 c 0 = 0 . Letting k = min { j : c j ≥ τ } k = \min\{j : c_j \ge\tau\} k = min { j : c j ≥ τ } and splitting the integral at c k − 1 c_{k-1} c k − 1 ,
∫ 0 τ F X − 1 ( u ) d u = ∑ j = 1 k − 1 ∫ c j − 1 c j x j d u + ∫ c k − 1 τ x k d u = ∑ j = 1 k − 1 w j x j + ( τ − c k − 1 ) x k . \begin{aligned}
\int_{0}^{\tau} F_X^{-1}(u)\,du = \sum_{j=1}^{k-1}\int_{c_{j-1}}^{c_j} x_j\,du + \int_{c_{k-1}}^{\tau} x_k\,du \\
&= \sum_{j=1}^{k-1} w_jx_j + (\tau-c_{k-1})x_k.
\end{aligned} ∫ 0 τ F X − 1 ( u ) d u = j = 1 ∑ k − 1 ∫ c j − 1 c j x j d u + ∫ c k − 1 τ x k d u = j = 1 ∑ k − 1 w j x j + ( τ − c k − 1 ) x k .
The first sum collects the atoms fully contained in the lower tail (each contributing its full mass w j w_j w j ), while the second term takes the fractional remainder τ − c k − 1 \tau-c_{k-1} τ − c k − 1 of the boundary atom x k x_k x k so that the total mass integrated equals exactly τ \tau τ . Dividing by τ \tau τ gives (10) .
Remark B.1. When τ = c k \tau= c_k τ = c k for some index k k k , the boundary term vanishes and (10) reduces to the conditional mean 1 c k ∑ i = 1 k w i x i \frac{1}{c_k}\sum_{i=1}^{k} w_ix_i c k 1 ∑ i = 1 k w i x i over the smallest k k k atoms, as expected.
CVaR–Wasserstein Lipschitz Property
We first record the well-known Lipschitz property of CVaR with respect to the Wasserstein-1 distance.
Lemma B.2 (CVaR is 1 / τ 1/\tau 1/ τ -Lipschitz in W 1 W_1 W 1 ). Let P P P and Q Q Q be probability distributions supported on [ 0 , M ] [0,M] [ 0 , M ] . Then for any τ ∈ ( 0 , 1 ] \tau\in(0,1] τ ∈ ( 0 , 1 ] :
∣ CVaR τ ( P ) − CVaR τ ( Q ) ∣ ≤ 1 τ W 1 ( P , Q ) . \left|\operatorname{CVaR}_{\tau}(P)-\operatorname{CVaR}_{\tau}(Q)\right| \le\frac{1}{\tau}W_1(P,Q). ∣ CVaR τ ( P ) − CVaR τ ( Q ) ∣ ≤ τ 1 W 1 ( P , Q ) .
Proof. Recall CVaR τ ( P ) = 1 τ ∫ 0 τ F P − 1 ( u ) d u \operatorname{CVaR}_{\tau}(P)=\frac{1}{\tau}\int_0^\tau F_P^{-1}(u)\,\mathrm{d}u CVaR τ ( P ) = τ 1 ∫ 0 τ F P − 1 ( u ) d u where F P − 1 F_P^{-1} F P − 1 is the quantile function. By the quantile representation of W 1 W_1 W 1 :
W 1 ( P , Q ) = ∫ 0 1 ∣ F P − 1 ( u ) − F Q − 1 ( u ) ∣ d u ≥ ∫ 0 τ ∣ F P − 1 ( u ) − F Q − 1 ( u ) ∣ d u . W_1(P,Q)=\int_0^1\left|F_P^{-1}(u)-F_Q^{-1}(u)\right|\,\mathrm{d}u \ge\int_0^\tau\left|F_P^{-1}(u)-F_Q^{-1}(u)\right|\,\mathrm{d}u. W 1 ( P , Q ) = ∫ 0 1 F P − 1 ( u ) − F Q − 1 ( u ) d u ≥ ∫ 0 τ F P − 1 ( u ) − F Q − 1 ( u ) d u .
Therefore:
∣ CVaR τ ( P ) − CVaR τ ( Q ) ∣ = 1 τ ∣ ∫ 0 τ ( F P − 1 ( u ) − F Q − 1 ( u ) ) d u ∣ ≤ 1 τ ∫ 0 τ ∣ F P − 1 ( u ) − F Q − 1 ( u ) ∣ d u ≤ 1 τ W 1 ( P , Q ) . \begin{aligned}
\left|\operatorname{CVaR}_{\tau}(P)-\operatorname{CVaR}_{\tau}(Q)\right| = \frac{1}{\tau}\left|\int_0^\tau\left(F_P^{-1}(u)-F_Q^{-1}(u)\right)\,\mathrm{d}u\right| \\
&\le\frac{1}{\tau}\int_0^\tau\left|F_P^{-1}(u)-F_Q^{-1}(u)\right|\,\mathrm{d}u \\
&\le\frac{1}{\tau}W_1(P,Q).
\end{aligned} ∣ CVaR τ ( P ) − CVaR τ ( Q ) ∣ = τ 1 ∫ 0 τ ( F P − 1 ( u ) − F Q − 1 ( u ) ) d u ≤ τ 1 ∫ 0 τ F P − 1 ( u ) − F Q − 1 ( u ) d u ≤ τ 1 W 1 ( P , Q ) .
Empirical W1 Concentration
This subsection proves Theorem B.3 in a way that keeps the random visit counts explicit. The object controlled below is the deterministic-prefix empirical law of the first m m m backups recorded at the root edge. The stopped-time child-value term is left visible, rather than implicitly replacing a random visit count by a deterministic sample size inside the proof.
Theorem B.3 (Root-backup W 1 W_1 W 1 reduction). Fix a root action a ∈ A s 0 a \in\mathcal{A}_{s_0} a ∈ A s 0 and let τ t \tau_t τ t be the global simulation index of the t t t -th visit to edge ( s 0 , a ) (s_0,a) ( s 0 , a ) . On that visit, write ( r t , S t ′ ) (r_t,S'_t) ( r t , S t ′ ) for the sampled reward and next state, and define
Q ‾ t = r t + γ V ^ τ t ( S t ′ ) , G t = r t + γ V ~ ( S t ′ ) . \overline{Q}_t=r_t+\gamma\widehat{V}_{\tau_t}(S'_t),\qquad G_t=r_t+\gamma\widetilde{V}(S'_t). Q t = r t + γ V τ t ( S t ′ ) , G t = r t + γ V ( S t ′ ) . Let
D ^ m ( s 0 , a ) : = 1 m ∑ t = 1 m δ Q ‾ t , D ( s 0 , a ) : = L ( G 1 ) . \widehat{D}_m(s_0,a):=\frac{1}{m}\sum_{t=1}^{m}\delta_{\overline{Q}_t},\qquad D(s_0,a):=\mathcal{L}(G_1). D m ( s 0 , a ) := m 1 t = 1 ∑ m δ Q t , D ( s 0 , a ) := L ( G 1 ) . Then P A T S 0 PATS0 P A T S 0 satisfies
E [ W 1 ( D ^ m ( s 0 , a ) , D ( s 0 , a ) ) ] ≤ γ m ∑ t = 1 m Δ t ( a ) + C F G R max 1 − γ m − 1 / 2 , \mathbb{E}\left[W_1\left(\widehat{D}_m(s_0,a),D(s_0,a)\right)\right]\le\frac{\gamma}{m}\sum_{t=1}^{m}\Delta_t(a)+C_{\mathrm{FG}}\frac{R_{\max}}{1-\gamma}m^{-1/2}, E [ W 1 ( D m ( s 0 , a ) , D ( s 0 , a ) ) ] ≤ m γ t = 1 ∑ m Δ t ( a ) + C FG 1 − γ R m a x m − 1/2 , where
Δ t ( a ) : = E [ ∣ V ^ τ t ( S t ′ ) − V ~ ( S t ′ ) ∣ ] . \Delta_t(a):=\mathbb{E}\left[\left|\widehat{V}_{\tau_t}(S'_t)-\widetilde{V}(S'_t)\right|\right]. Δ t ( a ) := E [ V τ t ( S t ′ ) − V ( S t ′ ) ] . In particular, if the stopped-time child-value errors satisfy Δ t ( a ) ≤ C V t − 1 / 2 \Delta_t(a)\le C_Vt^{-1/2} Δ t ( a ) ≤ C V t − 1/2 for all t ≥ 1 t\ge1 t ≥ 1 , then
E [ W 1 ( D ^ m ( s 0 , a ) , D ( s 0 , a ) ) ] ≤ ( 2 γ C V + C F G R max 1 − γ ) m − 1 / 2 . \mathbb{E}\left[W_1\left(\widehat{D}_m(s_0,a),D(s_0,a)\right)\right]\le\left(2\gamma C_V+C_{\mathrm{FG}}\frac{R_{\max}}{1-\gamma}\right)m^{-1/2}. E [ W 1 ( D m ( s 0 , a ) , D ( s 0 , a ) ) ] ≤ ( 2 γ C V + C FG 1 − γ R m a x ) m − 1/2 . For C A T S O CATSO C A T S O , add O ( R max / ( N ( 1 − γ ) ) ) O(R_{\max}/(N(1-\gamma))) O ( R m a x / ( N ( 1 − γ ))) discretization error.
Proof. Fix a ∈ A s 0 a\in\mathcal{A}_{s_0} a ∈ A s 0 and an integer m ≥ 1 m\ge1 m ≥ 1 . Define the oracle empirical law
D ^ m o r a c l e ( s 0 , a ) : = 1 m ∑ t = 1 m δ G t . \widehat{D}^{\mathrm{oracle}}_m(s_0,a):=\frac{1}{m}\sum_{t=1}^{m}\delta_{G_t}. D m oracle ( s 0 , a ) := m 1 t = 1 ∑ m δ G t .
Because each visit to the root edge draws a fresh reward/next-state sample from ( R , P ) ( ⋅ ∣ s 0 , a ) (\mathcal{R},\mathcal{P})(\cdot\mid s_0,a) ( R , P ) ( ⋅ ∣ s 0 , a ) , the oracle samples { G t } t = 1 m \{G_t\}_{t=1}^{m} { G t } t = 1 m are i.i.d. with law D ( s 0 , a ) D(s_0,a) D ( s 0 , a ) .
By the triangle inequality for W 1 W_1 W 1 ,
W 1 ( D ^ m , D ) ≤ W 1 ( D ^ m , D ^ m o r a c l e ) ⏟ ( I ) + W 1 ( D ^ m o r a c l e , D ) ⏟ ( I I ) . (20) W_1(\widehat{D}_m,D)\le\underbrace{W_1(\widehat{D}_m,\widehat{D}^{\mathrm{oracle}}_m)}_{\mathrm{(I)}}+\underbrace{W_1(\widehat{D}^{\mathrm{oracle}}_m,D)}_{\mathrm{(II)}}. \tag*{(20)} W 1 ( D m , D ) ≤ ( I ) W 1 ( D m , D m oracle ) + ( II ) W 1 ( D m oracle , D ) . (20)
I. coupling term. Consider the natural coupling
π m : = 1 m ∑ t = 1 m δ ( Q ^ t , G t ) . \pi_m := \frac{1}{m}\sum_{t=1}^{m}\delta_{(\widehat{Q}_t,G_t)}. π m := m 1 t = 1 ∑ m δ ( Q t , G t ) .
Its marginals are D ^ m \widehat{D}_m D m and D ^ m o r a c l e \widehat{D}^{\mathrm{oracle}}_m D m oracle , so the coupling characterization of W 1 W_1 W 1 gives
W 1 ( D ^ m , D ^ m o r a c l e ) ≤ ∫ ∣ x − y ∣ d π m ( x , y ) = 1 m ∑ t = 1 m ∣ Q ^ t − G t ∣ = γ m ∑ t = 1 m ∣ V ^ t ( S t ′ ) − V ~ ( S t ′ ) ∣ . (21) \begin{aligned}
W_1\left(\widehat{D}_m,\widehat{D}^{\mathrm{oracle}}_m\right)
&\le\int|x-y|\,\mathrm{d}\pi_m(x,y) \\
&= \frac{1}{m}\sum_{t=1}^{m}|\widehat{Q}_t-G_t| \\
&= \frac{\gamma}{m}\sum_{t=1}^{m}\left|\widehat{V}_t(S'_t)-\widetilde{V}(S'_t)\right|. \tag*{(21)}
\end{aligned} W 1 ( D m , D m oracle ) ≤ ∫ ∣ x − y ∣ d π m ( x , y ) = m 1 t = 1 ∑ m ∣ Q t − G t ∣ = m γ t = 1 ∑ m V t ( S t ′ ) − V ( S t ′ ) . (21)
Taking expectations in (21) yields
E [ ( I ) ] ≤ γ m ∑ t = 1 m E [ ∣ V ^ t ( S t ′ ) − V ~ ( S t ′ ) ∣ ] = γ m ∑ t = 1 m Δ t ( a ) . \mathbb{E}[(\mathrm{I})]\le\frac{\gamma}{m}\sum_{t=1}^{m}\mathbb{E}\left[\left|\widehat{V}_t(S'_t)-\widetilde{V}(S'_t)\right|\right]=\frac{\gamma}{m}\sum_{t=1}^{m}\Delta_t(a). E [( I )] ≤ m γ t = 1 ∑ m E [ V t ( S t ′ ) − V ( S t ′ ) ] = m γ t = 1 ∑ m Δ t ( a ) .
II. i.i.d. empirical-measure term. The oracle law D ( s 0 , a ) \mathcal{D}(s_0,a) D ( s 0 , a ) is supported on [ 0 , R max / ( 1 − γ ) ] [0,R_{\max}/(1-\gamma)] [ 0 , R m a x / ( 1 − γ )] . By the one-dimensional bounded-support case of Fournier and Guillin [13 ] , there exists a universal constant C F G > 0 C_{\mathrm{FG}}>0 C FG > 0 such that
E [ W 1 ( D ^ m o r a c l e , D ) ] ≤ C F G R max 1 − γ m − 1 / 2 . \mathbb{E}\left[W_1\left(\widehat{D}^{\mathrm{oracle}}_m,\mathcal{D}\right)\right]\le C_{\mathrm{FG}}\frac{R_{\max}}{1-\gamma}m^{-1/2}. E [ W 1 ( D m oracle , D ) ] ≤ C FG 1 − γ R m a x m − 1/2 .
III. combine. Substituting the bounds from Steps 1–2 into (20) proves the first claim:
E [ W 1 ( D ^ m ( s 0 , a ) , D ( s 0 , a ) ) ] ≤ γ m ∑ t = 1 m Δ t ( a ) + C F G R max 1 − γ m − 1 / 2 . \mathbb{E}\left[W_1\left(\widehat{D}_m(s_0,a),\mathcal{D}(s_0,a)\right)\right]\le\frac{\gamma}{m}\sum_{t=1}^{m}\Delta_t(a)+C_{\mathrm{FG}}\frac{R_{\max}}{1-\gamma}m^{-1/2}. E [ W 1 ( D m ( s 0 , a ) , D ( s 0 , a ) ) ] ≤ m γ t = 1 ∑ m Δ t ( a ) + C FG 1 − γ R m a x m − 1/2 .
If moreover Δ t ( a ) ≤ C V t − 1 / 2 \Delta_t(a)\le C_Vt^{-1/2} Δ t ( a ) ≤ C V t − 1/2 for all t ≥ 1 t\ge1 t ≥ 1 , then
γ m ∑ t = 1 m Δ t ( a ) ≤ γ C V m ∑ t = 1 m t − 1 / 2 ≤ 2 γ C V m − 1 / 2 , \frac{\gamma}{m}\sum_{t=1}^{m}\Delta_t(a)\le\frac{\gamma C_V}{m}\sum_{t=1}^{m}t^{-1/2}\le2\gamma C_Vm^{-1/2}, m γ t = 1 ∑ m Δ t ( a ) ≤ m γ C V t = 1 ∑ m t − 1/2 ≤ 2 γ C V m − 1/2 ,
which gives the displayed O ( m − 1 / 2 ) O(m^{-1/2}) O ( m − 1/2 ) bound. For CATSO, nearest-atom projection contributes an additional O ( R max / ( N ( 1 − γ ) ) ) O(R_{\max}/(N(1-\gamma))) O ( R m a x / ( N ( 1 − γ ))) term in W 1 W_1 W 1 .
Remark B.4 (Why Theorem 5.3 is stated with a fixed prefix). The counts T a ( n ) T_a(n) T a ( n ) and T s , a ( n ) T_{s,a}(n) T s , a ( n ) are random variables. Theorem 5.3 therefore controls the empirical law D ^ m ( s 0 , a ) \widehat{D}_m(s_0,a) D m ( s 0 , a ) for a deterministic prefix length m m m . Passing from a fixed m m m to the adaptive choice m = T s 0 , a ( n ) m=T_{s_0,a}(n) m = T s 0 , a ( n ) requires a separate stopped-time argument; that extra step is not hidden inside the proof above.
Local CVaR Bandit Analysis
Non-Stationary CVaR Bandit Setup
Throughout Sections C.2–C.5 we fix a CVaR level τ ∈ ( 0 , 1 ] \tau\in(0,1] τ ∈ ( 0 , 1 ] and polynomial-bonus exponents
bonus ( t , n ) : = C t 1 / 4 n 1 / 2 . \operatorname{bonus}(t,n):=C\frac{t^{1/4}}{n^{1/2}}. bonus ( t , n ) := C n 1/2 t 1/4 .
For each arm a ∈ [ K ] a\in[K] a ∈ [ K ] , let F a F_a F a denote its return law and define
c a τ : = CVaR τ ( F a ) , c ⋆ τ : = max a ∈ [ K ] c a τ , a ⋆ τ ∈ arg max a ∈ [ K ] c a τ . c_a^\tau:=\operatorname{CVaR}_{\tau}(F_a),\qquad c_\star^\tau:=\max_{a\in[K]}c_a^\tau,\qquad a_\star^\tau\in\arg\max_{a\in[K]}c_a^\tau. c a τ := CVaR τ ( F a ) , c ⋆ τ := a ∈ [ K ] max c a τ , a ⋆ τ ∈ arg a ∈ [ K ] max c a τ .
We assume the maximizer is unique whenever a suboptimal-pull statement is made.
For a probability law F F F on [ 0 , R ] [0,R] [ 0 , R ] and a threshold c ∈ R c \in\mathbb{R} c ∈ R , define the upper-tail risk information number
K i n f τ , + ( F , c ) : = inf { KL ( F ∥ G ) : G ∈ P ( [ 0 , R ] ) , CVaR τ ( G ) ≥ c } . \mathcal{K}_{\mathrm{inf}}^{\tau,+}(F,c) := \inf\left\{ \operatorname{KL}(F\|G) : G \in\mathcal{P}([0,R]),\ \operatorname{CVaR}_{\tau}(G) \ge c \right\}. K inf τ , + ( F , c ) := inf { KL ( F ∥ G ) : G ∈ P ([ 0 , R ]) , CVaR τ ( G ) ≥ c } .
On a fixed grid Z N = { z 0 < ⋯ < z N } ⊂ [ 0 , R ] \mathcal{Z}_{N} = \{z_{0} < \cdots< z_{N}\} \subset[0,R] Z N = { z 0 < ⋯ < z N } ⊂ [ 0 , R ] , define the grid-restricted version
K i n f ( N ) , τ , + ( F , c ) : = inf { KL ( F ∥ G ) : G ∈ P ( Z N ) , CVaR τ ( G ) ≥ c } . \mathcal{K}_{\mathrm{inf}}^{(N),\tau,+}(F,c) := \inf\left\{ \operatorname{KL}(F\|G) : G \in\mathcal{P}(\mathcal{Z}_{N}),\ \operatorname{CVaR}_{\tau}(G) \ge c \right\}. K inf ( N ) , τ , + ( F , c ) := inf { KL ( F ∥ G ) : G ∈ P ( Z N ) , CVaR τ ( G ) ≥ c } .
When the empirical law of arm a a a after n n n pulls is F ^ a , n \widehat{F}_{a,n} F a , n and the maintained posterior is sampled as a random law P ~ a , n \widetilde{P}_{a,n} P a , n , the posterior-sampled CVaR score is
c a , n τ : = CVaR τ ( P ~ a , n ) . c_{a,n}^{\tau} := \operatorname{CVaR}_{\tau}(\widetilde{P}_{a,n}). c a , n τ := CVaR τ ( P a , n ) .
The risk-sensitive Thompson-with-bonus index is then
U t ( a ) : = c a , T a ( t ) τ + C t 1 / 4 ( T a ( t ) ∨ 1 ) 1 / 2 , U_t(a) := c_{a,T_a(t)}^{\tau} + C \frac{t^{1/4}}{(T_a(t) \vee1)^{1/2}}, U t ( a ) := c a , T a ( t ) τ + C ( T a ( t ) ∨ 1 ) 1/2 t 1/4 ,
where T a ( t ) T_a(t) T a ( t ) is the number of pulls of arm a a a before round t t t .
For the detailed setup and assumptions, refer to section 5.1.
CATSO posterior CVaR tails on a fixed grid
CATSO is the only place where we currently prove a posterior large-deviation statement for a CVaR-based Thompson score. The proof uses the fixed-simplex Dirichlet geometry of Riou and Honda [25 ] .
Lemma C.1 (Dirichlet convex-set tail; Riou and Honda [25 ] ). Let Δ N : = { q ∈ [ 0 , 1 ] N + 1 : ∑ i = 0 N q i = 1 } \Delta_N := \left\{q \in[0,1]^{N+1} : \sum_{i=0}^{N} q_i = 1\right\} Δ N := { q ∈ [ 0 , 1 ] N + 1 : ∑ i = 0 N q i = 1 } . For α ∈ N N + 1 \alpha\in\mathbb{N}^{N+1} α ∈ N N + 1 with α i ≥ 1 \alpha_i \ge1 α i ≥ 1 , set A : = ∑ i α i A := \sum_i \alpha_i A := ∑ i α i and p : = α / A ∈ Δ N p := \alpha/A \in\Delta_N p := α / A ∈ Δ N , and let L ∼ Dir ( α ) L \sim\operatorname{Dir}(\alpha) L ∼ Dir ( α ) . If C ⊆ Δ N C \subseteq\Delta_N C ⊆ Δ N is closed and convex and p ∉ C p \notin C p ∈ / C , then
P ( L ∈ C ) ≤ C N R H A N / 2 exp { − A inf q ∈ C KL ( p ∥ q ) } , C N R H : = e 1 / 12 Γ ( N + 1 ) ( 1 2 π ) N , \mathbb{P}(L \in C) \le C_N^{\mathrm{RH}} A^{N/2} \exp\left\{-A \inf_{q \in C} \operatorname{KL}(p\|q)\right\}, \qquad C_N^{\mathrm{RH}} := \frac{e^{1/12}}{\Gamma(N+1)}\left(\frac{1}{\sqrt{2\pi}}\right)^N, P ( L ∈ C ) ≤ C N RH A N /2 exp { − A q ∈ C inf KL ( p ∥ q ) } , C N RH := Γ ( N + 1 ) e 1/12 ( 2 π 1 ) N , where KL ( p ∥ q ) : = ∑ i p i log ( p i / q i ) \operatorname{KL}(p\|q) := \sum_i p_i \log(p_i/q_i) KL ( p ∥ q ) := ∑ i p i log ( p i / q i ) , with KL ( p ∥ q ) = + ∞ \operatorname{KL}(p\|q) = +\infty KL ( p ∥ q ) = + ∞ if p i > 0 p_i > 0 p i > 0 and q i = 0 q_i = 0 q i = 0 for some i i i .
Proposition C.2 (CATSO posterior CVaR upper tail on a fixed grid). Fix a risk level τ ∈ ( 0 , 1 ] \tau\in(0,1] τ ∈ ( 0 , 1 ] and an ordered grid Z N = { z 0 < z 1 < ⋯ < z N } ⊂ [ 0 , R ] \mathcal{Z}_{N} = \{z_0 < z_1 < \cdots< z_N\} \subset[0,R] Z N = { z 0 < z 1 < ⋯ < z N } ⊂ [ 0 , R ] . Let α = α a ( m ) ∈ N N + 1 \alpha= \alpha_a(m) \in\mathbb{N}^{N+1} α = α a ( m ) ∈ N N + 1 with α a , i ( m ) ≥ 1 \alpha_{a,i}(m) \ge1 α a , i ( m ) ≥ 1 , set A = A a , m : = ∑ i α a , i ( m ) A = A_{a,m} := \sum_i \alpha_{a,i}(m) A = A a , m := ∑ i α a , i ( m ) and p = P α a ( m ) : = α / A ∈ Δ N p = P_{\alpha_a(m)} := \alpha/A \in\Delta_N p = P α a ( m ) := α / A ∈ Δ N , and draw L a ( m ) ∼ Dir ( α a ( m ) ) L_a(m) \sim\operatorname{Dir}(\alpha_a(m)) L a ( m ) ∼ Dir ( α a ( m )) . Define the Thompson-sampled categorical law and its lower-tail CVaR
D ^ a , m T S : = ∑ i L a , i ( m ) δ z i , c ^ a , m τ : = CVaR τ ( D ^ a , m T S ) , \widehat{D}_{a,m}^{\mathrm{TS}} := \sum_i L_{a,i}(m)\delta_{z_i}, \qquad\widehat{c}_{a,m}^{\tau} := \operatorname{CVaR}_{\tau}\left(\widehat{D}_{a,m}^{\mathrm{TS}}\right), D a , m TS := i ∑ L a , i ( m ) δ z i , c a , m τ := CVaR τ ( D a , m TS ) , and for q ∈ Δ N q \in\Delta_N q ∈ Δ N write CVaR τ ( q ; Z N ) : = CVaR τ ( ∑ i q i δ z i ) = 1 τ ∫ 0 τ F q − 1 ( u ) d u \operatorname{CVaR}_{\tau}(q;\mathcal{Z}_N) := \operatorname{CVaR}_{\tau}\left(\sum_i q_i\delta_{z_i}\right) = \frac{1}{\tau}\int_0^\tau F_q^{-1}(u)\,du CVaR τ ( q ; Z N ) := CVaR τ ( ∑ i q i δ z i ) = τ 1 ∫ 0 τ F q − 1 ( u ) d u , with the grid-restricted upper-tail information number K i n f ( N ) , τ , + ( p , c ) : = inf { KL ( p ∥ q ) : q ∈ Δ N , CVaR τ ( q ; Z N ) ≥ c } \mathcal{K}_{\mathrm{inf}}^{(N),\tau,+}(p,c) := \inf\left\{\operatorname{KL}(p\|q) : q \in\Delta_N,\ \operatorname{CVaR}_{\tau}(q;\mathcal{Z}_N) \ge c\right\} K inf ( N ) , τ , + ( p , c ) := inf { KL ( p ∥ q ) : q ∈ Δ N , CVaR τ ( q ; Z N ) ≥ c } . Then, for every c ∈ R c \in\mathbb{R} c ∈ R ,
P ( c ^ a , m τ ≥ c ∣ α a ( m ) ) ≤ { ( N + 1 ) C N R H A a , m N / 2 exp { − A a , m K i n f ( N ) , τ , + ( P α a ( m ) , c ) } , c > CVaR τ ( P α a ( m ) ; Z N ) , 1 , c ≤ CVaR τ ( P α a ( m ) ; Z N ) . \mathbb{P}\left(\widehat{c}_{a,m}^{\tau} \ge c \mid\alpha_a(m)\right) \le
\begin{cases}
(N+1)C_N^{\mathrm{RH}} A_{a,m}^{N/2}\exp\left\{-A_{a,m}\mathcal{K}_{\mathrm{inf}}^{(N),\tau,+}\left(P_{\alpha_a(m)},c\right)\right\}, & c > \operatorname{CVaR}_{\tau}\left(P_{\alpha_a(m)};\mathcal{Z}_N\right),\\
1, & c \le\operatorname{CVaR}_{\tau}\left(P_{\alpha_a(m)};\mathcal{Z}_N\right).
\end{cases} P ( c a , m τ ≥ c ∣ α a ( m ) ) ≤ { ( N + 1 ) C N RH A a , m N /2 exp { − A a , m K inf ( N ) , τ , + ( P α a ( m ) , c ) } , 1 , c > CVaR τ ( P α a ( m ) ; Z N ) , c ≤ CVaR τ ( P α a ( m ) ; Z N ) . Equivalently, the Riou and Honda [25 ] exponential bound is valid in the genuine upper-tail regime K i n f ( N ) , τ , + ( P α a ( m ) , c ) > 0 \mathcal{K}_{\mathrm{inf}}^{(N),\tau,+}\left(P_{\alpha_a(m)},c\right) > 0 K inf ( N ) , τ , + ( P α a ( m ) , c ) > 0 .
Proof. Condition on the vector α = α a ( m ) \alpha= \alpha_a(m) α = α a ( m ) . Throughout the proof, write
A = A a , m , p = P α a ( m ) = α A . A = A_{a,m}, \qquad p = P_{\alpha_a(m)} = \frac{\alpha}{A}. A = A a , m , p = P α a ( m ) = A α . If
c ≤ CVaR τ ( p ; Z N ) , c \le\operatorname{CVaR}_{\tau}(p;\mathcal{Z}_N), c ≤ CVaR τ ( p ; Z N ) ,
then the claimed bound is simply the trivial inequality
P ( c ~ a , m ≥ c ∣ α a ( m ) ) ≤ 1. \mathbb{P}\left(\widetilde{c}_{a,m} \ge c \mid\alpha_a(m)\right) \le1. P ( c a , m ≥ c ∣ α a ( m ) ) ≤ 1.
Thus it remains to prove the exponential bound in the case
c > CVaR τ ( p ; Z N ) . c > \operatorname{CVaR}_{\tau}(p; Z_N). c > CVaR τ ( p ; Z N ) .
For q = ( q 0 , … , q N ) ∈ Δ N q=(q_0,\ldots,q_N)\in\Delta_N q = ( q 0 , … , q N ) ∈ Δ N , define cumulative sums
S − 1 ( q ) : = 0 , S k ( q ) : = ∑ i = 0 k q i , 0 ≤ k ≤ N . S_{-1}(q):=0,\qquad S_k(q):=\sum_{i=0}^{k}q_i,\qquad0\le k\le N. S − 1 ( q ) := 0 , S k ( q ) := i = 0 ∑ k q i , 0 ≤ k ≤ N .
For each k ∈ { 0 , … , N } k\in\{0,\ldots,N\} k ∈ { 0 , … , N } , define
C k + ( c ) : = { q ∈ Δ N : S k − 1 ( q ) ≤ τ ≤ S k ( q ) , τ z k + ∑ i = 0 k − 1 q i ( z i − z k ) ≥ τ c } , C_k^+(c):=\left\{q\in\Delta_N:S_{k-1}(q)\le\tau\le S_k(q),\quad\tau z_k+\sum_{i=0}^{k-1}q_i(z_i-z_k)\ge\tau c\right\}, C k + ( c ) := { q ∈ Δ N : S k − 1 ( q ) ≤ τ ≤ S k ( q ) , τ z k + i = 0 ∑ k − 1 q i ( z i − z k ) ≥ τ c } ,
where an empty sum is interpreted as zero.
We first identify the CVaR superlevel set as a finite union of these sets. Fix q ∈ Δ N q\in\Delta_N q ∈ Δ N . If
S k − 1 ( q ) ≤ τ ≤ S k ( q ) , S_{k-1}(q)\le\tau\le S_k(q), S k − 1 ( q ) ≤ τ ≤ S k ( q ) ,
then the lower τ \tau τ -tail of the discrete law ∑ i = 0 N q i δ z i \sum_{i=0}^{N}q_i\delta_{z_i} ∑ i = 0 N q i δ z i consists of all atoms z 0 , … , z k − 1 z_0,\ldots,z_{k-1} z 0 , … , z k − 1 and a fractional amount τ − S k − 1 ( q ) \tau-S_{k-1}(q) τ − S k − 1 ( q ) of the atom z k z_k z k . Hence
τ CVaR τ ( q ; Z N ) = ∫ 0 τ F q − 1 ( u ) d u = ∑ i = 0 k − 1 q i z i + ( τ − S k − 1 ( q ) ) z k = τ z k + ∑ i = 0 k − 1 q i ( z i − z k ) . \begin{aligned}
\tau\operatorname{CVaR}_{\tau}(q;Z_N)=\int_0^\tau F_q^{-1}(u)\,\mathrm{d}u \\
&=\sum_{i=0}^{k-1}q_i z_i+\left(\tau-S_{k-1}(q)\right)z_k \\
&=\tau z_k+\sum_{i=0}^{k-1}q_i(z_i-z_k).
\end{aligned} τ CVaR τ ( q ; Z N ) = ∫ 0 τ F q − 1 ( u ) d u = i = 0 ∑ k − 1 q i z i + ( τ − S k − 1 ( q ) ) z k = τ z k + i = 0 ∑ k − 1 q i ( z i − z k ) .
Therefore, on the region S k − 1 ( q ) ≤ τ ≤ S k ( q ) S_{k-1}(q)\le\tau\le S_k(q) S k − 1 ( q ) ≤ τ ≤ S k ( q ) , the condition
CVaR τ ( q ; Z N ) ≥ c \operatorname{CVaR}_{\tau}(q;Z_N)\ge c CVaR τ ( q ; Z N ) ≥ c
is exactly the affine inequality
τ z k + ∑ i = 0 k − 1 q i ( z i − z k ) ≥ τ c . \tau z_k+\sum_{i=0}^{k-1}q_i(z_i-z_k)\ge\tau c. τ z k + i = 0 ∑ k − 1 q i ( z i − z k ) ≥ τ c .
Since for every q ∈ Δ N q\in\Delta_N q ∈ Δ N there exists at least one k ∈ { 0 , … , N } k\in\{0,\ldots,N\} k ∈ { 0 , … , N } such that S k − 1 ( q ) ≤ τ ≤ S k ( q ) S_{k-1}(q)\le\tau\le S_k(q) S k − 1 ( q ) ≤ τ ≤ S k ( q ) , we obtain
{ q ∈ Δ N : CVaR τ ( q ; Z N ) ≥ c } = ⋃ k = 0 N C k + ( c ) . \left\{q\in\Delta_N:\operatorname{CVaR}_{\tau}(q;Z_N)\ge c\right\}=\bigcup_{k=0}^{N}C_k^+(c). { q ∈ Δ N : CVaR τ ( q ; Z N ) ≥ c } = k = 0 ⋃ N C k + ( c ) .
Each set C k + ( c ) C_k^+(c) C k + ( c ) is a closed convex polytope: it is the intersection of the simplex with finitely many affine half-space constraints, namely
S k − 1 ( q ) ≤ τ , S k ( q ) ≥ τ , τ z k + ∑ i = 0 k − 1 q i ( z i − z k ) ≥ τ c . S_{k-1}(q)\le\tau,\qquad S_k(q)\ge\tau,\qquad\tau z_k+\sum_{i=0}^{k-1}q_i(z_i-z_k)\ge\tau c. S k − 1 ( q ) ≤ τ , S k ( q ) ≥ τ , τ z k + i = 0 ∑ k − 1 q i ( z i − z k ) ≥ τ c .
Because we are in the case
c > CVaR τ ( p ; Z N ) , c>\operatorname{CVaR}_{\tau}(p;Z_N), c > CVaR τ ( p ; Z N ) ,
we have
p ∉ { q ∈ Δ N : CVaR τ ( q ; Z N ) ≥ c } . p\notin\left\{q\in\Delta_N:\operatorname{CVaR}_{\tau}(q;Z_N)\ge c\right\}. p ∈ / { q ∈ Δ N : CVaR τ ( q ; Z N ) ≥ c } .
Consequently,
p ∉ C k + ( c ) for every k = 0 , … , N . p \notin C_k^{+}(c) \qquad\text{for every } k = 0,\ldots,N. p ∈ / C k + ( c ) for every k = 0 , … , N .
Thus Lemma C.1 applies to each C k + ( c ) C_k^{+}(c) C k + ( c ) . With the convention that inf ∅ KL ( p ∥ ⋅ ) = + ∞ \inf_{\varnothing}\operatorname{KL}(p\mathbin\Vert\cdot)=+\infty inf ∅ KL ( p ∥ ⋅ ) = + ∞ , we get
P ( L a ( m ) ∈ C k + ( c ) ∣ α a ( m ) ) ≤ C N R H A N / 2 exp { − A inf q ∈ C k + ( c ) KL ( p ∥ q ) } . \mathbb{P}\left(L_a(m)\in C_k^{+}(c)\mid\alpha_a(m)\right)\leq C_N^{\mathrm{RH}}A^{N/2}\exp\left\{-A\inf_{q\in C_k^{+}(c)}\operatorname{KL}(p\mathbin\Vert q)\right\}. P ( L a ( m ) ∈ C k + ( c ) ∣ α a ( m ) ) ≤ C N RH A N /2 exp { − A q ∈ C k + ( c ) inf KL ( p ∥ q ) } .
Using the union representation of the superlevel event and a union bound,
P ( c ^ a , m ≥ c ∣ α a ( m ) ) = P ( L a ( m ) ∈ ⋃ k = 0 N C k + ( c ) ∣ α a ( m ) ) ≤ ∑ k = 0 N P ( L a ( m ) ∈ C k + ( c ) ∣ α a ( m ) ) ≤ C N R H A N / 2 ∑ k = 0 N exp { − A inf q ∈ C k + ( c ) KL ( p ∥ q ) } ≤ ( N + 1 ) C N R H A N / 2 exp { − A min 0 ≤ k ≤ N inf q ∈ C k + ( c ) KL ( p ∥ q ) } . \begin{aligned}
\mathbb{P}\left(\widehat{c}_{a,m}\geq c\mid\alpha_a(m)\right)
&=\mathbb{P}\left(L_a(m)\in\bigcup_{k=0}^{N}C_k^{+}(c)\mathrel{\Big|}\alpha_a(m)\right)\\
&\leq\sum_{k=0}^{N}\mathbb{P}\left(L_a(m)\in C_k^{+}(c)\mid\alpha_a(m)\right)\\
&\leq C_N^{\mathrm{RH}}A^{N/2}\sum_{k=0}^{N}\exp\left\{-A\inf_{q\in C_k^{+}(c)}\operatorname{KL}(p\mathbin\Vert q)\right\}\\
&\leq(N+1)C_N^{\mathrm{RH}}A^{N/2}\exp\left\{-A\min_{0\leq k\leq N}\inf_{q\in C_k^{+}(c)}\operatorname{KL}(p\mathbin\Vert q)\right\}.
\end{aligned} P ( c a , m ≥ c ∣ α a ( m ) ) = P ( L a ( m ) ∈ k = 0 ⋃ N C k + ( c ) α a ( m ) ) ≤ k = 0 ∑ N P ( L a ( m ) ∈ C k + ( c ) ∣ α a ( m ) ) ≤ C N RH A N /2 k = 0 ∑ N exp { − A q ∈ C k + ( c ) inf KL ( p ∥ q ) } ≤ ( N + 1 ) C N RH A N /2 exp { − A 0 ≤ k ≤ N min q ∈ C k + ( c ) inf KL ( p ∥ q ) } .
Finally, since
{ q ∈ Δ N : CVaR τ ( q ; Z N ) ≥ c } = ⋃ k = 0 N C k + ( c ) , \{q\in\Delta_N:\operatorname{CVaR}_{\tau}(q;Z_N)\geq c\}=\bigcup_{k=0}^{N}C_k^{+}(c), { q ∈ Δ N : CVaR τ ( q ; Z N ) ≥ c } = k = 0 ⋃ N C k + ( c ) ,
we have
min 0 ≤ k ≤ N inf q ∈ C k + ( c ) KL ( p ∥ q ) = inf { KL ( p ∥ q ) : q ∈ Δ N , CVaR τ ( q ; Z N ) ≥ c } = K inf ( N ) , τ , + ( p , c ) . \min_{0\leq k\leq N}\inf_{q\in C_k^{+}(c)}\operatorname{KL}(p\mathbin\Vert q)=\inf\{\operatorname{KL}(p\mathbin\Vert q):q\in\Delta_N,\operatorname{CVaR}_{\tau}(q;Z_N)\geq c\}=K_{\inf}^{(N),\tau,+}(p,c). 0 ≤ k ≤ N min q ∈ C k + ( c ) inf KL ( p ∥ q ) = inf { KL ( p ∥ q ) : q ∈ Δ N , CVaR τ ( q ; Z N ) ≥ c } = K i n f ( N ) , τ , + ( p , c ) .
Substituting p = P α a ( m ) p=P_{\alpha_a(m)} p = P α a ( m ) and A = A a , m A=A_{a,m} A = A a , m proves the claimed upper-tail bound. □
Remark C.3 (Why the statement is not valid with the same exponential prefactor for all c c c ). If
c ≤ CVaR τ ( P α a ( m ) ; Z N ) , c\leq\operatorname{CVaR}_{\tau}(P_{\alpha_a(m)};Z_N), c ≤ CVaR τ ( P α a ( m ) ; Z N ) ,
then P α a ( m ) P_{\alpha_a(m)} P α a ( m ) itself belongs to the CVaR superlevel set, and therefore
K inf ( N ) , τ , + ( P α a ( m ) , c ) = 0. K_{\inf}^{(N),\tau,+}(P_{\alpha_a(m)},c)=0. K i n f ( N ) , τ , + ( P α a ( m ) , c ) = 0.
In this non-tail regime, the Riou and Honda [25 ] convex-set tail lemma is not applicable because its end-tail condition p ∉ C p\notin C p ∈ / C fails. Moreover, the prefactor
( N + 1 ) C N R H A a , m N / 2 (N+1)C_N^{\mathrm{RH}}A_{a,m}^{N/2} ( N + 1 ) C N RH A a , m N /2
can be smaller than 1 for small A a , m A_{a,m} A a , m , so the original “for every $c” exponential display is not a valid replacement for the trivial probability bound 1.
Remark C.4 (Fractional CATSO projection weights). The proof above is fully justified by the integer-parameter form of the Riou and Honda [25 ] lemma. If CATSO uses linear categorical projection and updates
α i ← α i + ω i , ω i ∈ [ 0 , 1 ] , ∑ i ω i = 1 , \alpha_i\leftarrow\alpha_i+\omega_i,\qquad\omega_i\in[0,1],\qquad\sum_i\omega_i=1, α i ← α i + ω i , ω i ∈ [ 0 , 1 ] , i ∑ ω i = 1 ,
then the posterior parameters need not remain integer-valued. In that case one should either use hard binning, so that α i ∈ N \alpha_i\in\mathbb{N} α i ∈ N , or explicitly assume/prove the following real-parameter analogue: for α ∈ [ 1 , ∞ ) N + 1 \alpha\in[1,\infty)^{N+1} α ∈ [ 1 , ∞ ) N + 1 , A = ∑ i α i A=\sum_i\alpha_i A = ∑ i α i , p = α / A p=\alpha/A p = α / A , and every closed convex C ⊆ Δ N C\subseteq\Delta_N C ⊆ Δ N with p ∉ C p\notin C p ∈ / C ,
P ( L ∈ C ) ≤ C N D i r A N / 2 exp { − A inf q ∈ C KL ( p ∥ q ) } , \mathbb{P}(L\in C)\leq C_N^{\mathrm{Dir}}A^{N/2}\exp\left\{-A\inf_{q\in C}\operatorname{KL}(p\mathbin\Vert q)\right\}, P ( L ∈ C ) ≤ C N Dir A N /2 exp { − A q ∈ C inf KL ( p ∥ q ) } ,
for a dimension-only constant C N D i r C_N^{\mathrm{Dir}} C N Dir . Under this real-parameter lemma, the proof of Proposition C.2 is unchanged, with C N R H C_N^{\mathrm{RH}} C N RH replaced by C N D i r C_N^{\mathrm{Dir}} C N Dir .
CATSO CVaR suboptimal-arm pulls
This section provides CVaR suboptimal-arm pull bounds. The result is stated in reduction form : the post-convergence part is proved, and the optimal-arm lower-tail contribution is isolated as an explicit remainder.
Lemma C.5 (CATSO posterior-center concentration on a fixed grid). *Fix an arm a a a and grid Z N = { z 0 < z 1 < ⋯ < z N } Z_N = \{z_0 < z_1 < \cdots< z_N\} Z N = { z 0 < z 1 < ⋯ < z N } . On the r r r -th local pull of arm a a a , CATSO adds a random weight vector W a , r = ( W a , r , 0 , … , W a , r , N ) ∈ Δ N W_{a,r} = (W_{a,r,0}, \ldots, W_{a,r,N}) \in\Delta_N W a , r = ( W a , r , 0 , … , W a , r , N ) ∈ Δ N to the Dirichlet parameter, with E [ W a , r ∣ H a , r − 1 ] = p a \mathbb{E}[W_{a,r} \mid\mathcal{H}_{a,r-1}] = p_a E [ W a , r ∣ H a , r − 1 ] = p a a.s. for all r ≥ 1 r \ge1 r ≥ 1 (some p a ∈ Δ N p_a \in\Delta_N p a ∈ Δ N ) and 0 ≤ W a , r , i ≤ 1 0 \le W_{a,r,i} \le1 0 ≤ W a , r , i ≤ 1 . With initial prior α a , i ( 0 ) = 1 \alpha_{a,i}(0) = 1 α a , i ( 0 ) = 1 for all i i i , define α a , i ( m ) = 1 + ∑ r = 1 m W a , r , i \alpha_{a,i}(m) = 1 + \sum_{r=1}^{m} W_{a,r,i} α a , i ( m ) = 1 + ∑ r = 1 m W a , r , i and P α a ( m ) : = α a ( m ) / ( m + N + 1 ) P_{\alpha_a}(m) := \alpha_a(m)/(m + N + 1) P α a ( m ) := α a ( m ) / ( m + N + 1 ) . Then for every ε > 0 \varepsilon> 0 ε > 0 ,
∑ m = 1 ∞ P ( ∥ P α a ( m ) − p a ∥ ∞ > ε ) < ∞ . \sum_{m=1}^{\infty} \mathbb{P}\left(\left\|P_{\alpha_a}(m)-p_a\right\|_{\infty}>\varepsilon\right)<\infty. m = 1 ∑ ∞ P ( ∥ P α a ( m ) − p a ∥ ∞ > ε ) < ∞.
Proof. Write
W ‾ a , m : = 1 m ∑ r = 1 m W a , r . \overline{W}_{a,m}:=\frac{1}{m}\sum_{r=1}^{m}W_{a,r}. W a , m := m 1 r = 1 ∑ m W a , r .
Then
P α a ( m ) = 1 + m W ‾ a , m m + N + 1 , P_{\alpha_a}(m)=\frac{\mathbf{1}+m\overline{W}_{a,m}}{m+N+1}, P α a ( m ) = m + N + 1 1 + m W a , m ,
where 1 = ( 1 , … , 1 ) \mathbf{1}=(1,\ldots,1) 1 = ( 1 , … , 1 ) . Hence
P α a ( m ) − p a = m m + N + 1 ( W ‾ a , m − p a ) + 1 − ( N + 1 ) p a m + N + 1 . P_{\alpha_a}(m)-p_a=\frac{m}{m+N+1}\left(\overline{W}_{a,m}-p_a\right)+\frac{\mathbf{1}-(N+1)p_a}{m+N+1}. P α a ( m ) − p a = m + N + 1 m ( W a , m − p a ) + m + N + 1 1 − ( N + 1 ) p a .
Let
B a : = ∥ 1 − ( N + 1 ) p a ∥ ∞ . B_a:=\left\|\mathbf{1}-(N+1)p_a\right\|_{\infty}. B a := ∥ 1 − ( N + 1 ) p a ∥ ∞ .
Choose
m ε : = ⌈ 2 B a ε ⌉ ∨ 1. m_{\varepsilon}:=\left\lceil\frac{2B_a}{\varepsilon}\right\rceil\vee1. m ε := ⌈ ε 2 B a ⌉ ∨ 1.
For all m ≥ m ε m\ge m_{\varepsilon} m ≥ m ε ,
B a m + N + 1 ≤ B a m ≤ ε 2 . \frac{B_a}{m+N+1}\le\frac{B_a}{m}\le\frac{\varepsilon}{2}. m + N + 1 B a ≤ m B a ≤ 2 ε .
Therefore, for m ≥ m ε m\ge m_{\varepsilon} m ≥ m ε ,
{ ∥ P α a ( m ) − p a ∥ ∞ > ε } ⊆ { ∥ W ‾ a , m − p a ∥ ∞ > ε 2 } . \left\{\left\|P_{\alpha_a}(m)-p_a\right\|_{\infty}>\varepsilon\right\}\subseteq\left\{\left\|\overline{W}_{a,m}-p_a\right\|_{\infty}>\frac{\varepsilon}{2}\right\}. { ∥ P α a ( m ) − p a ∥ ∞ > ε } ⊆ { W a , m − p a ∞ > 2 ε } .
For each coordinate i i i , the sequence W a , r , i − p a , i W_{a,r,i}-p_{a,i} W a , r , i − p a , i is a bounded martingale-difference sequence with range length at most 1. By Hoeffding–Azuma,
P ( ∣ 1 m ∑ r = 1 m ( W a , r , i − p a , i ) ∣ > ε 2 ) ≤ 2 exp ( − m ε 2 2 ) . \mathbb{P}\left(\left|\frac{1}{m}\sum_{r=1}^{m}(W_{a,r,i}-p_{a,i})\right|>\frac{\varepsilon}{2}\right)\le2\exp\left(-\frac{m\varepsilon^2}{2}\right). P ( m 1 r = 1 ∑ m ( W a , r , i − p a , i ) > 2 ε ) ≤ 2 exp ( − 2 m ε 2 ) .
Taking a union bound over i = 0 , … , N i=0,\ldots,N i = 0 , … , N , for all m ≥ m ε m\ge m_{\varepsilon} m ≥ m ε ,
P ( ∥ P α a ( m ) − p a ∥ ∞ > ε ) ≤ 2 ( N + 1 ) exp ( − m ε 2 2 ) . \mathbb{P}\left(\left\|P_{\alpha_a}(m)-p_a\right\|_{\infty}>\varepsilon\right)\le2(N+1)\exp\left(-\frac{m\varepsilon^2}{2}\right). P ( ∥ P α a ( m ) − p a ∥ ∞ > ε ) ≤ 2 ( N + 1 ) exp ( − 2 m ε 2 ) .
Thus
∑ m = 1 ∞ P ( ∥ P α a ( m ) − p a ∥ ∞ > ε ) ≤ m ε + 2 ( N + 1 ) ∑ m = m ε ∞ exp ( − m ε 2 2 ) < ∞ . \sum_{m=1}^{\infty}\mathbb{P}\left(\left\|P_{\alpha_a}(m)-p_a\right\|_{\infty}>\varepsilon\right)\le m_{\varepsilon}+2(N+1)\sum_{m=m_{\varepsilon}}^{\infty}\exp\left(-\frac{m\varepsilon^2}{2}\right)<\infty. m = 1 ∑ ∞ P ( ∥ P α a ( m ) − p a ∥ ∞ > ε ) ≤ m ε + 2 ( N + 1 ) m = m ε ∑ ∞ exp ( − 2 m ε 2 ) < ∞.
Define the optimal-arm lower-tail term
L n , C A T S O : = E [ ∑ t = 1 n 1 { c ~ a ⋆ ( t ) ( t ) < c ⋆ − C t 1 / 4 ( N a ⋆ ( t ) ∨ 1 ) 1 / 2 } ] . \mathcal{L}_{n,\mathrm{CATSO}}:=\mathbb{E}\left[\sum_{t=1}^{n}\mathbf{1}\left\{\widetilde{c}_{a_{\star}(t)}(t)<c^{\star}-C\frac{t^{1/4}}{(N_{a_{\star}}(t)\vee1)^{1/2}}\right\}\right]. L n , CATSO := E [ t = 1 ∑ n 1 { c a ⋆ ( t ) ( t ) < c ⋆ − C ( N a ⋆ ( t ) ∨ 1 ) 1/2 t 1/4 } ] .
Lemma C.6 (Optimal-arm CATSO lower-tail term). Assume the fixed-grid CATSO setting of Lemma C.7 with each arm pulled once initially, and write α ⋆ τ : = α ⋆ τ \alpha^\tau_\star:= \alpha^\tau_\star α ⋆ τ := α ⋆ τ , p ⋆ : = p a ⋆ τ p_\star:= p_{a^\tau_\star} p ⋆ := p a ⋆ τ , c ⋆ τ : = CVaR τ ( p ⋆ ; Z N ) c^\tau_\star:= \operatorname{CVaR}_\tau(p_\star; Z_N) c ⋆ τ := CVaR τ ( p ⋆ ; Z N ) . Assume the fixed-grid posterior-center concentration: there exist B N , b N > 0 B_N,b_N>0 B N , b N > 0 such that for every m ≥ 1 m\ge1 m ≥ 1 and s > 0 s>0 s > 0 ,
P ( ∥ P α ⋆ τ ( m ) − p ⋆ ∥ 1 > s ) ≤ B N exp ( − b N m s 2 ) . \mathbb{P}\left(\left\|P^{(m)}_{\alpha^\tau_\star}-p_\star\right\|_1>s\right)\le B_N\exp\left(-b_Nms^2\right). P ( P α ⋆ τ ( m ) − p ⋆ 1 > s ) ≤ B N exp ( − b N m s 2 ) . Assume also the fixed-grid Dirichlet lower-tail bound: with K inf ( N ) , τ , − ( p , c ) : = inf { KL ( p ∥ q ) : q ∈ Δ N , CVaR τ ( q ; Z N ) ≤ c } K_{\inf}^{(N),\tau,-}(p,c):=\inf\{\operatorname{KL}(p\|q):q\in\Delta_N,\operatorname{CVaR}_\tau(q;Z_N)\le c\} K i n f ( N ) , τ , − ( p , c ) := inf { KL ( p ∥ q ) : q ∈ Δ N , CVaR τ ( q ; Z N ) ≤ c } , for L ∼ Dir ( α ) L\sim\operatorname{Dir}(\alpha) L ∼ Dir ( α ) , A = ∑ i α i A=\sum_i\alpha_i A = ∑ i α i , p = α / A p=\alpha/A p = α / A , and any c < CVaR τ ( p ; Z N ) c<\operatorname{CVaR}_\tau(p;Z_N) c < CVaR τ ( p ; Z N ) ,
P ( CVaR τ ( ∑ i L i δ z i ) ≤ c ∣ α ) ≤ C N low A N / 2 exp { − A K inf ( N ) , τ , − ( p , c ) } . \mathbb{P}\left(\operatorname{CVaR}_\tau\left(\sum_i L_i\delta_{z_i}\right)\le c\mid\alpha\right)\le C_N^{\operatorname{low}}A^{N/2}\exp\left\{-A K_{\inf}^{(N),\tau,-}(p,c)\right\}. P ( CVaR τ ( i ∑ L i δ z i ) ≤ c ∣ α ) ≤ C N low A N /2 exp { − A K i n f ( N ) , τ , − ( p , c ) } . Then L n , CATSO τ = O ( 1 ) L^\tau_{n,\operatorname{CATSO}}=O(1) L n , CATSO τ = O ( 1 ) , and in particular L n , CATSO τ = O ( n 1 / 2 ) L^\tau_{n,\operatorname{CATSO}}=O(n^{1/2}) L n , CATSO τ = O ( n 1/2 ) .
Proof. Write
m t : = N a ⋆ τ ( t ) ∨ 1 , Δ t : = C t 1 / 4 m t 1 / 2 . m_t:=N a^\tau_\star(t)\vee1,\qquad\Delta_t:=C\frac{t^{1/4}}{m_t^{1/2}}. m t := N a ⋆ τ ( t ) ∨ 1 , Δ t := C m t 1/2 t 1/4 .
Let
E t : = { c ~ a ⋆ τ τ ( t ) < c ⋆ τ − Δ t } . E_t:=\left\{\widetilde c^\tau_{a^\tau_\star}(t)<c^\tau_\star-\Delta_t\right\}. E t := { c a ⋆ τ τ ( t ) < c ⋆ τ − Δ t } .
It is enough to show that
∑ t = 1 ∞ P ( E t ) < ∞ . \sum_{t=1}^{\infty}\mathbb{P}(E_t)<\infty. t = 1 ∑ ∞ P ( E t ) < ∞.
For m ≥ 1 m\ge1 m ≥ 1 , define
Δ t , m : = C t 1 / 4 m 1 / 2 . \Delta_{t,m}:=C\frac{t^{1/4}}{m^{1/2}}. Δ t , m := C m 1/2 t 1/4 .
On the event { m t = m } \{m_t=m\} { m t = m } , we have Δ t = Δ t , m \Delta_t=\Delta_{t,m} Δ t = Δ t , m , and
m Δ t , m 2 = C 2 t 1 / 2 . m\Delta_{t,m}^2=C^2t^{1/2}. m Δ t , m 2 = C 2 t 1/2 .
We first record a quadratic lower bound for the lower-tail information number. The CVaR functional is 1 / τ 1/\tau 1/ τ -Lipschitz in W 1 W_1 W 1 , and on [ 0 , R ] [0,R] [ 0 , R ] ,
W 1 ( p , q ) ≤ R 2 ∥ p − q ∥ 1 . W_1(p,q)\le\frac{R}{2}\|p-q\|_1. W 1 ( p , q ) ≤ 2 R ∥ p − q ∥ 1 .
Hence
∣ CVaR τ ( p ; Z N ) − CVaR τ ( q ; Z N ) ∣ ≤ R 2 τ ∥ p − q ∥ 1 . \left|\operatorname{CVaR}_\tau(p;Z_N)-\operatorname{CVaR}_\tau(q;Z_N)\right|\le\frac{R}{2\tau}\|p-q\|_1. ∣ CVaR τ ( p ; Z N ) − CVaR τ ( q ; Z N ) ∣ ≤ 2 τ R ∥ p − q ∥ 1 .
Let
G t , m : = { ∥ P α ⋆ τ ( m ) − p ⋆ ∥ 1 ≤ τ R Δ t , m } . G_{t,m}:=\left\{\left\|P^{(m)}_{\alpha^\tau_\star}-p_\star\right\|_1\le\frac{\tau}{R}\Delta_{t,m}\right\}. G t , m := { P α ⋆ τ ( m ) − p ⋆ 1 ≤ R τ Δ t , m } .
On G t , m G_{t,m} G t , m ,
CVaR τ ( P α ⋆ τ ( m ) ; Z N ) ≥ c ⋆ τ − Δ t , m 2 . \operatorname{CVaR}_\tau(P^{(m)}_{\alpha^\tau_\star};Z_N)\ge c^\tau_\star-\frac{\Delta_{t,m}}{2}. CVaR τ ( P α ⋆ τ ( m ) ; Z N ) ≥ c ⋆ τ − 2 Δ t , m .
Therefore, for every q ∈ Δ N q\in\Delta_N q ∈ Δ N satisfying
CVaR τ ( q ; Z N ) ≤ c ⋆ τ − Δ t , m , \operatorname{CVaR}_\tau(q;Z_N)\le c^\tau_\star-\Delta_{t,m}, CVaR τ ( q ; Z N ) ≤ c ⋆ τ − Δ t , m ,
we have
CVaR τ ( P α ⋆ τ ( m ) ; Z N ) − CVaR τ ( q ; Z N ) ≥ Δ t , m 2 . \operatorname{CVaR}_\tau(P^{(m)}_{\alpha^\tau_\star};Z_N)-\operatorname{CVaR}_\tau(q;Z_N)\ge\frac{\Delta_{t,m}}{2}. CVaR τ ( P α ⋆ τ ( m ) ; Z N ) − CVaR τ ( q ; Z N ) ≥ 2 Δ t , m .
Using the same Lipschitz bound again,
∥ P α ⋆ τ ( m ) − q ∥ 1 ≥ τ R Δ t , m . \left\|P^{(m)}_{\alpha^\tau_\star}-q\right\|_1\ge\frac{\tau}{R}\Delta_{t,m}. P α ⋆ τ ( m ) − q 1 ≥ R τ Δ t , m .
By Pinsker’s inequality,
KL ( P α ⋆ τ ( m ) ∥ q ) ≥ 1 2 ∥ P α ⋆ τ ( m ) − q ∥ 1 2 ≥ τ 2 2 R 2 Δ t , m 2 . \operatorname{KL}(P^{(m)}_{\alpha^\tau_\star}\|q)\ge\frac{1}{2}\left\|P^{(m)}_{\alpha^\tau_\star}-q\right\|_1^2\ge\frac{\tau^2}{2R^2}\Delta_{t,m}^2. KL ( P α ⋆ τ ( m ) ∥ q ) ≥ 2 1 P α ⋆ τ ( m ) − q 1 2 ≥ 2 R 2 τ 2 Δ t , m 2 .
Thus, on G t , m G_{t,m} G t , m ,
K i n f ( N ) , τ , − ( P α ⋆ ( m ) , c ⋆ τ − Δ t , m ) ≥ λ τ Δ t , m 2 , λ τ : = τ 2 2 R 2 . K_{\mathrm{inf}}^{(N),\tau,-}\left(P_{\alpha^\star}^{(m)},c_\star^\tau-\Delta_{t,m}\right)\ge\lambda_\tau\Delta_{t,m}^2,\qquad\lambda_\tau:=\frac{\tau^2}{2R^2}. K inf ( N ) , τ , − ( P α ⋆ ( m ) , c ⋆ τ − Δ t , m ) ≥ λ τ Δ t , m 2 , λ τ := 2 R 2 τ 2 .
Now decompose
P ( E t ) ≤ ∑ m = 1 t P ( E t , m t = m , G t , m ) + ∑ m = 1 t P ( m t = m , G t , m c ) . \mathbb{P}(E_t)\le\sum_{m=1}^{t}\mathbb{P}(E_t,m_t=m,G_{t,m})+\sum_{m=1}^{t}\mathbb{P}(m_t=m,G_{t,m}^{c}). P ( E t ) ≤ m = 1 ∑ t P ( E t , m t = m , G t , m ) + m = 1 ∑ t P ( m t = m , G t , m c ) .
For the bad-center part, by the assumed posterior-center concentration,
P ( G t , m c ) ≤ B N exp ( − b N m τ 2 R 2 Δ t , m 2 ) = B N exp ( − b N τ 2 C 2 R 2 t 1 / 2 ) . \mathbb{P}(G_{t,m}^{c})\le B_N\exp\left(-b_Nm\frac{\tau^2}{R^2}\Delta_{t,m}^2\right)=B_N\exp\left(-b_N\frac{\tau^2 C^2}{R^2}t^{1/2}\right). P ( G t , m c ) ≤ B N exp ( − b N m R 2 τ 2 Δ t , m 2 ) = B N exp ( − b N R 2 τ 2 C 2 t 1/2 ) .
Therefore
∑ m = 1 t P ( m t = m , G t , m c ) ≤ B N t exp ( − b N τ 2 C 2 R 2 t 1 / 2 ) . \sum_{m=1}^{t}\mathbb{P}(m_t=m,G_{t,m}^{c})\le B_Nt\exp\left(-b_N\frac{\tau^2 C^2}{R^2}t^{1/2}\right). m = 1 ∑ t P ( m t = m , G t , m c ) ≤ B N t exp ( − b N R 2 τ 2 C 2 t 1/2 ) .
For the good-center part, condition on the pre-round history. On { m t = m } ∩ G t , m \{m_t=m\}\cap G_{t,m} { m t = m } ∩ G t , m , the fixed-grid Dirichlet lower-tail bound gives
P ( E t ∣ F t − 1 ) ≤ C N l o w ( m + N + 1 ) N / 2 exp ( − ( m + N + 1 ) λ τ Δ t , m 2 ) . \mathbb{P}(E_t\mid\mathcal{F}_{t-1})\le C_N^{\mathrm{low}}(m+N+1)^{N/2}\exp\left(-(m+N+1)\lambda_\tau\Delta_{t,m}^2\right). P ( E t ∣ F t − 1 ) ≤ C N low ( m + N + 1 ) N /2 exp ( − ( m + N + 1 ) λ τ Δ t , m 2 ) .
Since m ≤ t m\le t m ≤ t and m Δ t , m 2 = C 2 t 1 / 2 m\Delta_{t,m}^2=C^2t^{1/2} m Δ t , m 2 = C 2 t 1/2 ,
P ( E t ∣ F t − 1 ) ≤ C N ′ t N / 2 exp ( − λ τ C 2 t 1 / 2 ) , \mathbb{P}(E_t\mid\mathcal{F}_{t-1})\le C_N't^{N/2}\exp\left(-\lambda_\tau C^2t^{1/2}\right), P ( E t ∣ F t − 1 ) ≤ C N ′ t N /2 exp ( − λ τ C 2 t 1/2 ) ,
for a constant C N ′ C_N' C N ′ depending only on N N N . Hence
∑ m = 1 t P ( E t , m t = m , G t , m ) ≤ C N ′ t N / 2 exp ( − λ τ C 2 t 1 / 2 ) . \sum_{m=1}^{t}\mathbb{P}(E_t,m_t=m,G_{t,m})\le C_N't^{N/2}\exp\left(-\lambda_\tau C^2t^{1/2}\right). m = 1 ∑ t P ( E t , m t = m , G t , m ) ≤ C N ′ t N /2 exp ( − λ τ C 2 t 1/2 ) .
Combining the two bounds,
P ( E t ) ≤ B N t exp ( − c 1 t 1 / 2 ) + C N ′ t N / 2 exp ( − c 2 t 1 / 2 ) \mathbb{P}(E_t)\le B_Nt\exp(-c_1t^{1/2})+C_N't^{N/2}\exp(-c_2t^{1/2}) P ( E t ) ≤ B N t exp ( − c 1 t 1/2 ) + C N ′ t N /2 exp ( − c 2 t 1/2 )
for positive constants c 1 , c 2 c_1,c_2 c 1 , c 2 . Since
∑ t = 1 ∞ t r e − c t < ∞ for every fixed r < ∞ , c > 0 , \sum_{t=1}^{\infty}t^r e^{-c\sqrt{t}}<\infty\qquad\text{for every fixed }r<\infty,\ c>0, t = 1 ∑ ∞ t r e − c t < ∞ for every fixed r < ∞ , c > 0 ,
we get
sup n ≥ 1 L n , C A T S O τ ≤ ∑ t = 1 ∞ P ( E t ) < ∞ . \sup_{n\ge1}\mathcal{L}_{n,\mathrm{CATSO}}^\tau\le\sum_{t=1}^{\infty}\mathbb{P}(E_t)<\infty. n ≥ 1 sup L n , CATSO τ ≤ t = 1 ∑ ∞ P ( E t ) < ∞.
This proves the claim.
Lemma C.7 (CATSO CVaR suboptimal-arm pulls). Adopt the notation of Section 5.1, with each arm having fixed grid law F a = ∑ i p a i δ z i F_a=\sum_i p_a^i\delta_{z_i} F a = ∑ i p a i δ z i , p a ∈ Δ N p_a\in\Delta_N p a ∈ Δ N , and unique CVaR-optimal arm a ⋆ τ a_\star^\tau a ⋆ τ . Suppose CATSO selects
I ( t ) ∈ arg max a ∈ [ K ] { C ^ a τ ( t ) + C t 1 / 4 ( T a ( t ) ∨ 1 ) 1 / 2 } , I(t)\in\arg\max_{a\in[K]}\left\{\widehat{C}_a^\tau(t)+C\frac{t^{1/4}}{(T_a(t)\vee1)^{1/2}}\right\}, I ( t ) ∈ arg a ∈ [ K ] max { C a τ ( t ) + C ( T a ( t ) ∨ 1 ) 1/2 t 1/4 } , *where T a ( t ) T_a(t) T a ( t ) is the number of pulls of arm a a a before round t t t . Assume Lemma C.5, that p ↦ K i n f ( N ) , τ , + ( p , u ) p\mapsto K_{\mathrm{inf}}^{(N),\tau,+}(p,u) p ↦ K inf ( N ) , τ , + ( p , u ) is ∥ ⋅ ∥ ∞ \|\cdot\|_\infty ∥ ⋅ ∥ ∞ -continuous on Δ N \Delta_N Δ N for every fixed u < c ⋆ τ u<c_\star^\tau u < c ⋆ τ , and that c ↦ K i n f ( N ) , τ , + ( F a , c ) c\mapsto K_{\mathrm{inf}}^{(N),\tau,+}(F_a,c) c ↦ K inf ( N ) , τ , + ( F a , c ) is left-continuous at c = c ⋆ τ c=c_\star^\tau c = c ⋆ τ . Then for every suboptimal a ≠ a ⋆ τ a\ne a_\star^\tau a = a ⋆ τ and every ϵ 0 > 0 \epsilon_0>0 ϵ 0 > 0 ,
E [ T a ( n ) ] ≤ ( 1 + ϵ 0 ) log n K i n f ( N ) , τ , + ( F a , c ⋆ τ ) + O ( n 1 / 2 ) . \mathbb{E}[T_a(n)]\le\frac{(1+\epsilon_0)\log n}{K_{\mathrm{inf}}^{(N),\tau,+}(F_a,c_\star^\tau)}+O(n^{1/2}). E [ T a ( n )] ≤ K inf ( N ) , τ , + ( F a , c ⋆ τ ) ( 1 + ϵ 0 ) log n + O ( n 1/2 ) .
Proof. Fix a CVaR-suboptimal arm a ≠ a τ ⋆ a \ne a_\tau^\star a = a τ ⋆ . Write
K a ( c ) : = K inf ( N ) , τ , + ( F a , c ) , K a ⋆ : = K a ( c ⋆ τ ) . K_a(c) := \mathcal{K}_{\inf}^{(N),\tau,+}(F_a,c), \qquad K^\star_a := K_a(c_\star^\tau). K a ( c ) := K i n f ( N ) , τ , + ( F a , c ) , K a ⋆ := K a ( c ⋆ τ ) .
Since a a a is CVaR-suboptimal and the CVaR-optimal arm is unique,
c a τ < c ⋆ τ . c_a^\tau< c_\star^\tau. c a τ < c ⋆ τ .
Thus, for ϵ 1 > 0 \epsilon_1 > 0 ϵ 1 > 0 small enough,
u : = c ⋆ τ − ϵ 1 u := c_\star^\tau- \epsilon_1 u := c ⋆ τ − ϵ 1
satisfies
c a τ < u < c ⋆ τ . c_a^\tau< u < c_\star^\tau. c a τ < u < c ⋆ τ .
For such ϵ 1 \epsilon_1 ϵ 1 , the fixed-grid CVaR superlevel set
{ q ∈ Δ N : CVaR τ ( q ; Z N ) ≥ u } \left\{q \in\Delta_N : \operatorname{CVaR}_\tau(q; Z_N) \ge u\right\} { q ∈ Δ N : CVaR τ ( q ; Z N ) ≥ u }
is closed and does not contain p a p_a p a . Hence
K a ( u ) > 0. K_a(u) > 0. K a ( u ) > 0.
Choose ϵ 3 > 0 \epsilon_3 > 0 ϵ 3 > 0 such that
κ : = κ a τ ( ϵ 1 , ϵ 3 ) : = K a ( c ⋆ τ − ϵ 1 ) − ϵ 3 > 0. \kappa:= \kappa_a^\tau(\epsilon_1,\epsilon_3) := K_a(c_\star^\tau-\epsilon_1)-\epsilon_3 > 0. κ := κ a τ ( ϵ 1 , ϵ 3 ) := K a ( c ⋆ τ − ϵ 1 ) − ϵ 3 > 0.
By the assumed continuity of
p ⟼ K inf ( N ) , τ , + ( p , c ⋆ τ − ϵ 1 ) , p \longmapsto\mathcal{K}_{\inf}^{(N),\tau,+}(p,c_\star^\tau-\epsilon_1), p ⟼ K i n f ( N ) , τ , + ( p , c ⋆ τ − ϵ 1 ) ,
there exists ϵ 2 > 0 \epsilon_2 > 0 ϵ 2 > 0 such that, whenever
∥ p − p a ∥ ∞ ≤ ϵ 2 , \lVert p-p_a\rVert_\infty\le\epsilon_2, ∥ p − p a ∥ ∞ ≤ ϵ 2 ,
we have
K inf ( N ) , τ , + ( p , c ⋆ τ − ϵ 1 ) ≥ K a ( c ⋆ τ − ϵ 1 ) − ϵ 3 = κ . \mathcal{K}_{\inf}^{(N),\tau,+}(p,c_\star^\tau-\epsilon_1) \ge K_a(c_\star^\tau-\epsilon_1)-\epsilon_3=\kappa. K i n f ( N ) , τ , + ( p , c ⋆ τ − ϵ 1 ) ≥ K a ( c ⋆ τ − ϵ 1 ) − ϵ 3 = κ .
In particular, on this event the information number is strictly positive. Therefore
c ⋆ τ − ϵ 1 > CVaR τ ( p ; Z N ) , c_\star^\tau-\epsilon_1 > \operatorname{CVaR}_\tau(p;Z_N), c ⋆ τ − ϵ 1 > CVaR τ ( p ; Z N ) ,
because otherwise p p p itself would be feasible in the definition of K inf ( N ) , τ , + ( p , c ⋆ τ − ϵ 1 ) \mathcal{K}_{\inf}^{(N),\tau,+}(p,c_\star^\tau-\epsilon_1) K i n f ( N ) , τ , + ( p , c ⋆ τ − ϵ 1 ) , forcing the information number to be zero. Hence the corrected upper-tail Proposition C.2 is applicable on this good-center event.
For clarity, write
P α a ( m ) P_{\alpha_a}^{(m)} P α a ( m )
for the posterior center of arm a a a after exactly m m m local updates, so that, on the event { T a ( t ) = m } \{T_a(t)=m\} { T a ( t ) = m } ,
P α a ( t ) = P α a ( m ) , A a , t = m + N + 1. P_{\alpha_a(t)}=P_{\alpha_a}^{(m)}, \qquad A_{a,t}=m+N+1. P α a ( t ) = P α a ( m ) , A a , t = m + N + 1.
The extra N + 1 N+1 N + 1 comes from the unit Dirichlet prior on the N + 1 N+1 N + 1 grid atoms.
Define the bonus
β t ( a ) : = C t 1 / 4 ( T a ( t ) ∨ 1 ) 1 / 2 . \beta_t(a):=C\frac{t^{1/4}}{(T_a(t)\vee1)^{1/2}}. β t ( a ) := C ( T a ( t ) ∨ 1 ) 1/2 t 1/4 .
Set
m β : = ⌈ ( C ϵ 1 ) 1 / 2 n 1 / 2 ⌉ , m_\beta:=\left\lceil\left(\frac{C}{\epsilon_1}\right)^{1/2}n^{1/2}\right\rceil, m β := ⌈ ( ϵ 1 C ) 1/2 n 1/2 ⌉ ,
and
m K L : = ⌈ log n + ( N / 2 + 1 ) log log ( e n ) κ ⌉ , m_{\mathrm{KL}}:=\left\lceil\frac{\log n+(N/2+1)\log\log(en)}{\kappa}\right\rceil, m KL := ⌈ κ log n + ( N /2 + 1 ) log log ( e n ) ⌉ ,
and
m 0 : = max { 1 , m β , m K L } . m_0:=\max\{1,m_\beta,m_{\mathrm{KL}}\}. m 0 := max { 1 , m β , m KL } .
We first decompose the number of pulls of arm a a a . The event { I ( t ) = a , T a ( t ) < m 0 } \{I(t)=a,T_a(t)<m_0\} { I ( t ) = a , T a ( t ) < m 0 } can occur at most m 0 m_0 m 0 times. Therefore
E [ T a ( n ) ] = E [ ∑ t = 1 n 1 { I ( t ) = a } ] ≤ m 0 + A 1 + A 2 , (22) \mathbb{E}[T_a(n)]=\mathbb{E}\left[\sum_{t=1}^{n}\mathbb{1}\{I(t)=a\}\right]\le m_0+A_1+A_2, \tag*{(22)} E [ T a ( n )] = E [ t = 1 ∑ n 1 { I ( t ) = a } ] ≤ m 0 + A 1 + A 2 , (22)
where
A 1 : = E [ ∑ t = 1 n 1 { I ( t ) = a , T a ( t ) ≥ m 0 , c ~ a T ( t ) ≥ c ⋆ T − β t ( a ) , ∥ P α a ( t ) − p a ∥ ∞ ≤ ϵ 2 } ] , A_1 := \mathbb{E}\left[\sum_{t=1}^{n} \mathbf{1}\left\{I(t)=a,\ T_a(t)\ge m_0,\ \widetilde{c}_a^T(t)\ge c_\star^T-\beta_t(a),\ \left\|P_{\alpha_a(t)}-p_a\right\|_\infty\le\epsilon_2\right\}\right], A 1 := E [ t = 1 ∑ n 1 { I ( t ) = a , T a ( t ) ≥ m 0 , c a T ( t ) ≥ c ⋆ T − β t ( a ) , P α a ( t ) − p a ∞ ≤ ϵ 2 } ] ,
and
A 2 : = E [ ∑ t = 1 n 1 { I ( t ) = a , T a ( t ) ≥ m 0 , [ c ~ a T ( t ) < c ⋆ T − β t ( a ) ] ∪ [ ∥ P α a ( t ) − p a ∥ ∞ > ϵ 2 ] } ] . A_2 := \mathbb{E}\left[\sum_{t=1}^{n} \mathbf{1}\left\{I(t)=a,\ T_a(t)\ge m_0,\ \left[\widetilde{c}_a^T(t)<c_\star^T-\beta_t(a)\right]\cup\left[\left\|P_{\alpha_a(t)}-p_a\right\|_\infty>\epsilon_2\right]\right\}\right]. A 2 := E [ t = 1 ∑ n 1 { I ( t ) = a , T a ( t ) ≥ m 0 , [ c a T ( t ) < c ⋆ T − β t ( a ) ] ∪ [ P α a ( t ) − p a ∞ > ϵ 2 ] } ] .
Bound on A 1 A_1 A 1 . Let F t − 1 \mathcal{F}_{t-1} F t − 1 denote the history immediately before the Thompson draw at round t t t . On the event T a ( t ) ≥ m 0 T_a(t)\ge m_0 T a ( t ) ≥ m 0 , we have
β t ( a ) = C t 1 / 4 ( T a ( t ) ∨ 1 ) 1 / 2 ≤ C n 1 / 4 m 0 1 / 2 ≤ C n 1 / 4 m β 1 / 2 ≤ ϵ 1 . \beta_t(a)=C\frac{t^{1/4}}{(T_a(t)\vee1)^{1/2}}\le C\frac{n^{1/4}}{m_0^{1/2}}\le C\frac{n^{1/4}}{m_\beta^{1/2}}\le\epsilon_1. β t ( a ) = C ( T a ( t ) ∨ 1 ) 1/2 t 1/4 ≤ C m 0 1/2 n 1/4 ≤ C m β 1/2 n 1/4 ≤ ϵ 1 .
Hence
c ~ a T ( t ) ≥ c ⋆ T − β t ( a ) ⟹ c ~ a T ( t ) ≥ c ⋆ T − ϵ 1 . \widetilde{c}_a^T(t)\ge c_\star^T-\beta_t(a)\quad\Longrightarrow\quad\widetilde{c}_a^T(t)\ge c_\star^T-\epsilon_1. c a T ( t ) ≥ c ⋆ T − β t ( a ) ⟹ c a T ( t ) ≥ c ⋆ T − ϵ 1 .
Dropping the event { I ( t ) = a } \{I(t)=a\} { I ( t ) = a } , which can only increase the probability, we obtain
A 1 ≤ ∑ t = 1 n E [ 1 { T a ( t ) ≥ m 0 , ∥ P α a ( t ) − p a ∥ ∞ ≤ ϵ 2 } P ( c ~ a T ( t ) ≥ c ⋆ T − ϵ 1 ∣ F t − 1 ) ] . A_1\le\sum_{t=1}^{n}\mathbb{E}\left[\mathbf{1}\left\{T_a(t)\ge m_0,\ \left\|P_{\alpha_a(t)}-p_a\right\|_\infty\le\epsilon_2\right\}\mathbb{P}\left(\widetilde{c}_a^T(t)\ge c_\star^T-\epsilon_1\mid\mathcal{F}_{t-1}\right)\right]. A 1 ≤ t = 1 ∑ n E [ 1 { T a ( t ) ≥ m 0 , P α a ( t ) − p a ∞ ≤ ϵ 2 } P ( c a T ( t ) ≥ c ⋆ T − ϵ 1 ∣ F t − 1 ) ] .
Now fix a history with
T a ( t ) = m ≥ m 0 and ∥ P α a ( t ) − p a ∥ ∞ ≤ ϵ 2 . T_a(t)=m\ge m_0\quad\text{and}\quad\left\|P_{\alpha_a(t)}-p_a\right\|_\infty\le\epsilon_2. T a ( t ) = m ≥ m 0 and P α a ( t ) − p a ∞ ≤ ϵ 2 .
By the choice of ϵ 2 \epsilon_2 ϵ 2 ,
κ inf ( N ) , τ , + ( P α a ( t ) , c ⋆ T − ϵ 1 ) ≥ κ > 0. \kappa_{\inf}^{(N),\tau,+}\left(P_{\alpha_a(t)},c_\star^T-\epsilon_1\right)\ge\kappa>0. κ i n f ( N ) , τ , + ( P α a ( t ) , c ⋆ T − ϵ 1 ) ≥ κ > 0.
Thus c ⋆ T − ϵ 1 c_\star^T-\epsilon_1 c ⋆ T − ϵ 1 is a genuine upper-tail threshold for the posterior center, and the corrected Proposition C.2 gives
P ( c ~ a T ( t ) ≥ c ⋆ T − ϵ 1 ∣ F t − 1 ) ≤ ( N + 1 ) C N R H ( m + N + 1 ) N / 2 exp { − ( m + N + 1 ) κ } . \mathbb{P}\left(\widetilde{c}_a^T(t)\ge c_\star^T-\epsilon_1\mid\mathcal{F}_{t-1}\right)
\le(N+1)C_N^{\mathrm{RH}}(m+N+1)^{N/2}\exp\{-(m+N+1)\kappa\}. P ( c a T ( t ) ≥ c ⋆ T − ϵ 1 ∣ F t − 1 ) ≤ ( N + 1 ) C N RH ( m + N + 1 ) N /2 exp { − ( m + N + 1 ) κ } .
If one uses the real-parameter Dirichlet tail lemma instead of the integer-count Riou and Honda [25 ] lemma, replace C N R H C_N^{\mathrm{RH}} C N RH by the corresponding dimension-only constant.
Define
f κ ( m ) : = ( m + N + 1 ) N / 2 exp { − ( m + N + 1 ) κ } . f_\kappa(m):=(m+N+1)^{N/2}\exp\{-(m+N+1)\kappa\}. f κ ( m ) := ( m + N + 1 ) N /2 exp { − ( m + N + 1 ) κ } .
The function f κ ( m ) f_\kappa(m) f κ ( m ) is decreasing for all sufficiently large m m m . Since m 0 → ∞ m_0\to\infty m 0 → ∞ with n n n , for all sufficiently large n n n ,
sup m ≥ m 0 f κ ( m ) = f κ ( m 0 ) . \sup_{m\ge m_0}f_\kappa(m)=f_\kappa(m_0). m ≥ m 0 sup f κ ( m ) = f κ ( m 0 ) .
The finitely many smaller n n n can be absorbed into the final O ( n 1 / 2 ) O(n^{1/2}) O ( n 1/2 ) term. Therefore
A 1 ≤ ( N + 1 ) C N R H n ( m 0 + N + 1 ) N / 2 exp { − ( m 0 + N + 1 ) κ } . A_1\le(N+1)C_N^{\mathrm{RH}}n(m_0+N+1)^{N/2}\exp\{-(m_0+N+1)\kappa\}. A 1 ≤ ( N + 1 ) C N RH n ( m 0 + N + 1 ) N /2 exp { − ( m 0 + N + 1 ) κ } .
Since m 0 ≥ m K L m_0\ge m_{\mathrm{KL}} m 0 ≥ m KL and f κ f_\kappa f κ is eventually decreasing,
A 1 ≤ ( N + 1 ) C N R H n ( m K L + N + 1 ) N / 2 exp { − m K L κ } ⋅ O ( 1 ) . A_1\le(N+1)C_N^{\mathrm{RH}}n(m_{\mathrm{KL}}+N+1)^{N/2}\exp\{-m_{\mathrm{KL}}\kappa\}\cdot O(1). A 1 ≤ ( N + 1 ) C N RH n ( m KL + N + 1 ) N /2 exp { − m KL κ } ⋅ O ( 1 ) .
By definition of m K L m_{\mathrm{KL}} m KL ,
exp { − m K L κ } ≤ 1 n [ log ( e n ) ] N / 2 + 1 . \exp\{-m_{\mathrm{KL}}\kappa\}\le\frac{1}{n[\log(en)]^{N/2+1}}. exp { − m KL κ } ≤ n [ log ( e n ) ] N /2 + 1 1 .
Also
m K L + N + 1 = O ( log n ) . m_{\mathrm{KL}}+N+1=O(\log n). m KL + N + 1 = O ( log n ) .
Hence
A 1 ≤ O ( 1 ) ( log n ) N / 2 [ log ( e n ) ] N / 2 + 1 = O ( 1 ) . A_1\le O(1)\frac{(\log n)^{N/2}}{[\log(en)]^{N/2+1}}=O(1). A 1 ≤ O ( 1 ) [ log ( e n ) ] N /2 + 1 ( log n ) N /2 = O ( 1 ) .
Bound on A 2 A_2 A 2 . Write
A 2 ≤ R 1 + R 2 , A_2 \le R_1 + R_2, A 2 ≤ R 1 + R 2 ,
where
R 1 : = E [ ∑ t = 1 n 1 { I ( t ) = a , c ^ a τ ( t ) < c ⋆ τ − β t ( a ) } ] , R_1 := \mathbb{E}\left[\sum_{t=1}^{n} \mathbf{1}\left\{I(t)=a,\ \widehat{c}^{\tau}_{a}(t)<c^{\tau}_{\star}-\beta_t(a)\right\}\right], R 1 := E [ t = 1 ∑ n 1 { I ( t ) = a , c a τ ( t ) < c ⋆ τ − β t ( a ) } ] ,
and
R 2 : = E [ ∑ t = 1 n 1 { I ( t ) = a , ∥ P α a ( t ) − p a ∥ ∞ > ϵ 2 } ] . R_2 := \mathbb{E}\left[\sum_{t=1}^{n} \mathbf{1}\left\{I(t)=a,\ \left\|P_{\alpha_a(t)}-p_a\right\|_{\infty}>\epsilon_2\right\}\right]. R 2 := E [ t = 1 ∑ n 1 { I ( t ) = a , P α a ( t ) − p a ∞ > ϵ 2 } ] .
For R 2 R_2 R 2 , the event
{ I ( t ) = a , T a ( t ) = m } \{I(t)=a,\ T_a(t)=m\} { I ( t ) = a , T a ( t ) = m }
can occur at most once for each m ∈ { 0 , … , n − 1 } m\in\{0,\ldots,n-1\} m ∈ { 0 , … , n − 1 } , namely at the round of the ( m + 1 ) (m+1) ( m + 1 ) -st pull of arm a a a . Hence
R 2 ≤ ∑ m = 0 n − 1 P ( ∥ P α a ( m ) − p a ∥ ∞ > ϵ 2 ) = O ( 1 ) , R_2\le\sum_{m=0}^{n-1}\mathbb{P}\left(\left\|P_{\alpha_a}^{(m)}-p_a\right\|_{\infty}>\epsilon_2\right)=O(1), R 2 ≤ m = 0 ∑ n − 1 P ( P α a ( m ) − p a ∞ > ϵ 2 ) = O ( 1 ) ,
by Lemma C.5. The m = 0 m=0 m = 0 term is just a finite constant.
For R 1 R_1 R 1 , suppose that
I ( t ) = a and c ^ a τ ( t ) < c ⋆ τ − β t ( a ) . I(t)=a\quad\text{and}\quad\widehat{c}^{\tau}_{a}(t)<c^{\tau}_{\star}-\beta_t(a). I ( t ) = a and c a τ ( t ) < c ⋆ τ − β t ( a ) .
Then
U t ( a ) = c ^ a τ ( t ) + β t ( a ) < c ⋆ τ . U_t(a)=\widehat{c}^{\tau}_{a}(t)+\beta_t(a)<c^{\tau}_{\star}. U t ( a ) = c a τ ( t ) + β t ( a ) < c ⋆ τ .
Since I ( t ) = a I(t)=a I ( t ) = a maximizes the index,
U t ( a τ ⋆ ) ≤ U t ( a ) < c ⋆ τ . U_t(a_{\tau}^{\star})\le U_t(a)<c^{\tau}_{\star}. U t ( a τ ⋆ ) ≤ U t ( a ) < c ⋆ τ .
Therefore
c ^ a τ ⋆ τ ( t ) < c ⋆ τ − β t ( a τ ⋆ ) . \widehat{c}^{\tau}_{a_{\tau}^{\star}}(t)<c^{\tau}_{\star}-\beta_t(a_{\tau}^{\star}). c a τ ⋆ τ ( t ) < c ⋆ τ − β t ( a τ ⋆ ) .
Consequently,
R 1 ≤ E [ ∑ t = 1 n 1 { c ^ a τ ⋆ τ ( t ) < c ⋆ τ − β t ( a τ ⋆ ) } ] = L n , C A T S O ≤ O ( n 1 / 2 ) . R_1\le\mathbb{E}\left[\sum_{t=1}^{n}\mathbf{1}\left\{\widehat{c}^{\tau}_{a_{\tau}^{\star}}(t)<c^{\tau}_{\star}-\beta_t(a_{\tau}^{\star})\right\}\right]=\mathcal{L}_{n,\mathrm{CATSO}}\le O(n^{1/2}). R 1 ≤ E [ t = 1 ∑ n 1 { c a τ ⋆ τ ( t ) < c ⋆ τ − β t ( a τ ⋆ ) } ] = L n , CATSO ≤ O ( n 1/2 ) .
Combining the bounds on A 1 A_1 A 1 , R 1 R_1 R 1 , and R 2 R_2 R 2 , we obtain
E [ T a ( n ) ] ≤ m 0 + O ( n 1 / 2 ) + O ( 1 ) . \mathbb{E}[T_a(n)]\le m_0+O(n^{1/2})+O(1). E [ T a ( n )] ≤ m 0 + O ( n 1/2 ) + O ( 1 ) .
Since
m 0 ≤ 1 + m β + m K L , m_0\le1+m_{\beta}+m_{\mathrm{KL}}, m 0 ≤ 1 + m β + m KL ,
we have
m 0 ≤ log n + ( N / 2 + 1 ) log log ( e n ) κ + O ( n 1 / 2 ) . m_0\le\frac{\log n+(N/2+1)\log\log(en)}{\kappa}+O(n^{1/2}). m 0 ≤ κ log n + ( N /2 + 1 ) log log ( e n ) + O ( n 1/2 ) .
Hence
E [ T a ( n ) ] ≤ log n + ( N / 2 + 1 ) log log ( e n ) κ a τ ( ϵ 1 , ϵ 3 ) + O ( n 1 / 2 ) + O ( n 1 / 2 ) . \mathbb{E}[T_a(n)]\le\frac{\log n+(N/2+1)\log\log(en)}{\kappa^{\tau}_{a}(\epsilon_1,\epsilon_3)}+O(n^{1/2})+O(n^{1/2}). E [ T a ( n )] ≤ κ a τ ( ϵ 1 , ϵ 3 ) log n + ( N /2 + 1 ) log log ( e n ) + O ( n 1/2 ) + O ( n 1/2 ) .
It remains to choose ϵ 1 \epsilon_1 ϵ 1 and ϵ 3 \epsilon_3 ϵ 3 . By the assumed left-continuity of
c ⟼ K inf ( N ) , τ , + ( F a , c ) c\longmapsto\mathcal{K}^{(N),\tau,+}_{\inf}(F_a,c) c ⟼ K i n f ( N ) , τ , + ( F a , c )
at c = c ⋆ τ c=c^{\tau}_{\star} c = c ⋆ τ , for every ϵ 0 > 0 \epsilon_0>0 ϵ 0 > 0 we can choose ϵ 1 > 0 \epsilon_1>0 ϵ 1 > 0 small enough, and then ϵ 3 > 0 \epsilon_3>0 ϵ 3 > 0 small enough, so that
1 κ a τ ( ϵ 1 , ϵ 3 ) ≤ 1 + ϵ 0 / 2 K inf ( N ) , τ , + ( F a , c ⋆ τ ) . \frac{1}{\kappa^{\tau}_{a}(\epsilon_1,\epsilon_3)}
\le\frac{1+\epsilon_0/2}{\mathcal{K}^{(N),\tau,+}_{\inf}(F_a,c^{\tau}_{\star})}. κ a τ ( ϵ 1 , ϵ 3 ) 1 ≤ K i n f ( N ) , τ , + ( F a , c ⋆ τ ) 1 + ϵ 0 /2 .
Since
log log ( e n ) = o ( log n ) , \log\log(en)=o(\log n), log log ( e n ) = o ( log n ) ,
for all sufficiently large n n n ,
log n + ( N / 2 + 1 ) log log ( e n ) K i n f τ ( ϵ 1 , ϵ 3 ) ≤ ( 1 + ϵ 0 ) log n K i n f ( N ) , τ , + ( F a , c ⋆ τ ) . \frac{\log n + (N/2 + 1)\log\log(en)}{\mathcal{K}^{\tau}_{\mathrm{inf}}(\epsilon_1,\epsilon_3)} \le\frac{(1+\epsilon_0)\log n}{\mathcal{K}^{(N),\tau,+}_{\mathrm{inf}}(F_a,c_\star^\tau)}. K inf τ ( ϵ 1 , ϵ 3 ) log n + ( N /2 + 1 ) log log ( e n ) ≤ K inf ( N ) , τ , + ( F a , c ⋆ τ ) ( 1 + ϵ 0 ) log n .
Absorbing the finitely many small-n n n cases into the O ( n 1 / 2 ) O(n^{1/2}) O ( n 1/2 ) term gives
E [ T a ( n ) ] ≤ ( 1 + ϵ 0 ) log n K i n f ( N ) , τ , + ( F a , c ⋆ τ ) + O ( n 1 / 2 ) , \mathbb{E}[T_a(n)] \le\frac{(1+\epsilon_0)\log n}{\mathcal{K}^{(N),\tau,+}_{\mathrm{inf}}(F_a,c_\star^\tau)} + O(n^{1/2}), E [ T a ( n )] ≤ K inf ( N ) , τ , + ( F a , c ⋆ τ ) ( 1 + ϵ 0 ) log n + O ( n 1/2 ) ,
which proves the lemma.
PATSO Posterior CVaR Tails
Assumption C.8 (PATSO posterior CVaR upper tail, history-conditional). Fix τ ∈ ( 0 , 1 ] \tau\in(0,1] τ ∈ ( 0 , 1 ] . For each arm a a a , local sample size m ≥ 1 m \ge1 m ≥ 1 , and threshold c ∈ ( c a τ , R ] c \in(c^\tau_a,R] c ∈ ( c a τ , R ] , assume that for every η > 0 \eta> 0 η > 0 there exists C τ , η , c > 0 C_{\tau,\eta,c} > 0 C τ , η , c > 0 such that, whenever the pre-round history has T a ( t ) = m T_a(t)=m T a ( t ) = m ,
P ( c ^ a τ ( t ) ≥ c ∣ F t − 1 ) ≤ C τ , η , c exp { − m ( K i n f τ , + ( F ^ a , m , c ) − η ) } a.s. \mathbb{P}\left(\widehat{c}_a^\tau(t) \ge c \mid\mathcal{F}_{t-1}\right) \le C_{\tau,\eta,c}\exp\left\{-m\left(\mathcal{K}^{\tau,+}_{\mathrm{inf}}(\widehat{F}_{a,m},c)-\eta\right)\right\}\quad\text{a.s.} P ( c a τ ( t ) ≥ c ∣ F t − 1 ) ≤ C τ , η , c exp { − m ( K inf τ , + ( F a , m , c ) − η ) } a.s.
Proposition C.9 (Uncapped PATSO posterior CVaR upper tail). Fix τ ∈ ( 0 , 1 ] \tau\in(0,1] τ ∈ ( 0 , 1 ] and consider uncapped PATSO, so that after m ≥ 1 m \ge1 m ≥ 1 pulls of arm a a a the particle support equals the support of the empirical law of the first m m m returns. Write F ^ a , m = ∑ j = 1 M m p j δ s j \widehat{F}_{a,m}=\sum_{j=1}^{M_m}p_j\delta_{s_j} F a , m = ∑ j = 1 M m p j δ s j with s 1 < ⋯ < s M m s_1<\cdots<s_{M_m} s 1 < ⋯ < s M m , p j = n j / m p_j=n_j/m p j = n j / m , n j ∈ N n_j\in\mathbb{N} n j ∈ N , ∑ j n j = m \sum_j n_j=m ∑ j n j = m , draw L a ( m ) ∼ Dir ( n 1 , … , n M m ) L_a(m)\sim\operatorname{Dir}(n_1,\ldots,n_{M_m}) L a ( m ) ∼ Dir ( n 1 , … , n M m ) , and define
D ^ a , m T S : = ∑ j L a , j ( m ) δ s j , c ^ a , m τ : = CVaR τ ( D ^ a , m T S ) . \widehat{D}^{\mathrm{TS}}_{a,m}:=\sum_j L_{a,j}(m)\delta_{s_j},\qquad\widehat{c}^{\tau}_{a,m}:=\operatorname{CVaR}_{\tau}\left(\widehat{D}^{\mathrm{TS}}_{a,m}\right). D a , m TS := j ∑ L a , j ( m ) δ s j , c a , m τ := CVaR τ ( D a , m TS ) . For a probability law F F F on [ 0 , R ] [0,R] [ 0 , R ] , set K i n f τ , + ( F , c ) : = inf { KL ( F ∥ G ) : G ∈ P ( [ 0 , R ] ) , CVaR τ ( G ) ≥ c } \mathcal{K}^{\tau,+}_{\mathrm{inf}}(F,c):=\inf\{\operatorname{KL}(F\mathbin{\|}G):G\in\mathcal{P}([0,R]),\operatorname{CVaR}_{\tau}(G)\ge c\} K inf τ , + ( F , c ) := inf { KL ( F ∥ G ) : G ∈ P ([ 0 , R ]) , CVaR τ ( G ) ≥ c } . Then for every c ∈ R c\in\mathbb{R} c ∈ R and η > 0 \eta>0 η > 0 there exists C η < ∞ C_\eta<\infty C η < ∞ , independent of m m m , such that
P ( c ^ a , m τ ≥ c ∣ F ^ a , m ) ≤ C η exp { − m ( K i n f τ , + ( F ^ a , m , c ) − η ) } a.s. ; \mathbb{P}\left(\widehat{c}^{\tau}_{a,m}\ge c\mid\widehat{F}_{a,m}\right)\le C_\eta\exp\left\{-m\left(\mathcal{K}^{\tau,+}_{\mathrm{inf}}(\widehat{F}_{a,m},c)-\eta\right)\right\}\quad\text{a.s.}; P ( c a , m τ ≥ c ∣ F a , m ) ≤ C η exp { − m ( K inf τ , + ( F a , m , c ) − η ) } a.s. ; one may take C η = exp { 1 12 + 1 2 π η } C_\eta=\exp\left\{\frac{1}{12}+\frac{1}{2\pi\eta}\right\} C η = exp { 12 1 + 2 π η 1 } . In particular, Assumption C.8 holds for uncapped exact-particle PATSO.
Proof. Fix m ≥ 1 m\ge1 m ≥ 1 and condition on the empirical law
F ^ a , m = ∑ j = 1 M m p j δ s j , s 1 < ⋯ < s M m , p j = n j m . \widehat{F}_{a,m}=\sum_{j=1}^{M_m}p_j\delta_{s_j},\qquad s_1<\cdots<s_{M_m},\qquad p_j=\frac{n_j}{m}. F a , m = j = 1 ∑ M m p j δ s j , s 1 < ⋯ < s M m , p j = m n j .
Under uncapped PATSO, the particle support and the Dirichlet parameters are therefore fixed by F ^ a , m \widehat{F}_{a,m} F a , m :
S a ( m ) = { s 1 , … , s M m } , α a ( m ) = ( n 1 , … , n M m ) . S_a(m)=\{s_1,\ldots,s_{M_m}\},\qquad\alpha_a(m)=(n_1,\ldots,n_{M_m}). S a ( m ) = { s 1 , … , s M m } , α a ( m ) = ( n 1 , … , n M m ) .
For readability, write M : = M m M:=M_m M := M m , p = ( p 1 , … , p M ) p=(p_1,\ldots,p_M) p = ( p 1 , … , p M ) , and
Δ M − 1 : = { q = ( q 1 , … , q M ) ∈ [ 0 , 1 ] M : ∑ j = 1 M q j = 1 } . \Delta_{M-1}:=\left\{q=(q_1,\ldots,q_M)\in[0,1]^M:\sum_{j=1}^{M}q_j=1\right\}. Δ M − 1 := { q = ( q 1 , … , q M ) ∈ [ 0 , 1 ] M : j = 1 ∑ M q j = 1 } .
For q ∈ Δ M − 1 q\in\Delta_{M-1} q ∈ Δ M − 1 , define cumulative sums
S 0 ( q ) : = 0 , S k ( q ) : = ∑ j = 1 k q j , 1 ≤ k ≤ M . S_0(q):=0,\qquad S_k(q):=\sum_{j=1}^{k}q_j,\qquad1\le k\le M. S 0 ( q ) := 0 , S k ( q ) := j = 1 ∑ k q j , 1 ≤ k ≤ M .
For each k ∈ { 1 , … , M } k\in\{1,\ldots,M\} k ∈ { 1 , … , M } , define
C k + ( c ) : = { q ∈ Δ M − 1 : S k − 1 ( q ) ≤ τ ≤ S k ( q ) , τ s k + ∑ j = 1 k − 1 q j ( s j − s k ) ≥ τ c } . C_k^+(c):=\left\{q\in\Delta_{M-1}:S_{k-1}(q)\le\tau\le S_k(q),\ \tau s_k+\sum_{j=1}^{k-1}q_j(s_j-s_k)\ge\tau c\right\}. C k + ( c ) := { q ∈ Δ M − 1 : S k − 1 ( q ) ≤ τ ≤ S k ( q ) , τ s k + j = 1 ∑ k − 1 q j ( s j − s k ) ≥ τ c } .
As in the fixed-grid CATS0 proof, on the region
S k − 1 ( q ) ≤ τ ≤ S k ( q ) , S_{k-1}(q) \le\tau\le S_k(q), S k − 1 ( q ) ≤ τ ≤ S k ( q ) ,
the lower-tail CVaR of the discrete law ∑ j = 1 M q j δ s j \sum_{j=1}^{M} q_j\delta_{s_j} ∑ j = 1 M q j δ s j is given by
τ CVaR τ ( ∑ j = 1 M q j δ s j ) = ∑ j = 1 k − 1 q j s j + ( τ − S k − 1 ( q ) ) s k = τ s k + ∑ j = 1 k − 1 q j ( s j − s k ) . \tau\operatorname{CVaR}_{\tau}\left(\sum_{j=1}^{M}q_j\delta_{s_j}\right)=\sum_{j=1}^{k-1}q_js_j+\left(\tau-S_{k-1}(q)\right)s_k=\tau s_k+\sum_{j=1}^{k-1}q_j(s_j-s_k). τ CVaR τ ( j = 1 ∑ M q j δ s j ) = j = 1 ∑ k − 1 q j s j + ( τ − S k − 1 ( q ) ) s k = τ s k + j = 1 ∑ k − 1 q j ( s j − s k ) .
Therefore,
{ q ∈ Δ M − 1 : CVaR τ ( ∑ j = 1 M q j δ s j ) ≥ c } = ⋃ k = 1 M C k + ( c ) . \left\{q\in\Delta_{M-1}:\operatorname{CVaR}_{\tau}\left(\sum_{j=1}^{M}q_j\delta_{s_j}\right)\ge c\right\}=\bigcup_{k=1}^{M}C_k^+(c). { q ∈ Δ M − 1 : CVaR τ ( j = 1 ∑ M q j δ s j ) ≥ c } = k = 1 ⋃ M C k + ( c ) .
Each C k + ( c ) C_k^+(c) C k + ( c ) is a closed convex polytope, since it is the intersection of the simplex with finitely many affine half-space constraints.
Define the support-restricted information number
K S τ , + ( p , c ) : = inf { KL ( p ∥ q ) : q ∈ Δ M − 1 , CVaR τ ( ∑ j = 1 M q j δ s j ) ≥ c } . K_S^{\tau,+}(p,c):=\inf\left\{\operatorname{KL}(p\|q):q\in\Delta_{M-1},\ \operatorname{CVaR}_{\tau}\left(\sum_{j=1}^{M}q_j\delta_{s_j}\right)\ge c\right\}. K S τ , + ( p , c ) := inf { KL ( p ∥ q ) : q ∈ Δ M − 1 , CVaR τ ( j = 1 ∑ M q j δ s j ) ≥ c } .
Here
KL ( p ∥ q ) : = ∑ j = 1 M p j log p j q j . \operatorname{KL}(p\|q):=\sum_{j=1}^{M}p_j\log\frac{p_j}{q_j}. KL ( p ∥ q ) := j = 1 ∑ M p j log q j p j .
If
c ≤ CVaR τ ( F ^ a , m ) , c\le\operatorname{CVaR}_{\tau}(\widehat{F}_{a,m}), c ≤ CVaR τ ( F a , m ) ,
then p p p itself is feasible in the definition of K i n f τ , + ( F ^ a , m , c ) K_{\mathrm{inf}}^{\tau,+}(\widehat{F}_{a,m},c) K inf τ , + ( F a , m , c ) , so
K i n f τ , + ( F ^ a , m , c ) = 0. K_{\mathrm{inf}}^{\tau,+}(\widehat{F}_{a,m},c)=0. K inf τ , + ( F a , m , c ) = 0.
Thus
P ( C ^ a , m ≥ c ∣ F ^ a , m ) ≤ 1 ≤ C η e m η = C η exp { − m ( K i n f τ , + ( F ^ a , m , c ) − η ) } , \mathbb{P}\left(\widehat{C}_{a,m}\ge c\mid\widehat{F}_{a,m}\right)\le1\le C_\eta e^{m\eta}=C_\eta\exp\left\{-m\left(K_{\mathrm{inf}}^{\tau,+}(\widehat{F}_{a,m},c)-\eta\right)\right\}, P ( C a , m ≥ c ∣ F a , m ) ≤ 1 ≤ C η e m η = C η exp { − m ( K inf τ , + ( F a , m , c ) − η ) } ,
provided C η ≥ 1 C_\eta\ge1 C η ≥ 1 . Hence the desired inequality is trivial in this non-tail case.
It remains to consider the genuine upper-tail case
c > CVaR τ ( F ^ a , m ) . c>\operatorname{CVaR}_{\tau}(\widehat{F}_{a,m}). c > CVaR τ ( F a , m ) .
Then p ∉ C k + ( c ) p\notin C_k^+(c) p ∈ / C k + ( c ) for every k k k . Applying the Riou and Honda [25 ] Dirichlet convex-set tail bound to each closed convex set C k + ( c ) C_k^+(c) C k + ( c ) , with total Dirichlet mass m = ∑ j = 1 M n j m=\sum_{j=1}^{M}n_j m = ∑ j = 1 M n j , gives
P ( L a ( m ) ∈ C k + ( c ) ∣ F ^ a , m ) ≤ c M − 1 R H m ( M − 1 ) / 2 exp { − m inf q ∈ C k + ( c ) KL ( p ∥ q ) } , \mathbb{P}\left(L_a(m)\in C_k^+(c)\mid\widehat{F}_{a,m}\right)\le c_{M-1}^{\mathrm{RH}}m^{(M-1)/2}\exp\left\{-m\inf_{q\in C_k^+(c)}\operatorname{KL}(p\|q)\right\}, P ( L a ( m ) ∈ C k + ( c ) ∣ F a , m ) ≤ c M − 1 RH m ( M − 1 ) /2 exp { − m q ∈ C k + ( c ) inf KL ( p ∥ q ) } ,
where
c M − 1 R H = e 1 / 12 Γ ( M ) ( 1 2 π ) M − 1 . c_{M-1}^{\mathrm{RH}}=\frac{e^{1/12}}{\Gamma(M)}\left(\frac{1}{\sqrt{2\pi}}\right)^{M-1}. c M − 1 RH = Γ ( M ) e 1/12 ( 2 π 1 ) M − 1 .
Using the union representation of the CVaR superlevel event and a union bound,
P ( C ^ a , m ≥ c ∣ F ^ a , m ) ≤ M c M − 1 R H m ( M − 1 ) / 2 × exp { − m K S τ , + ( p , c ) } . \mathbb{P}\left(\widehat{C}_{a,m}\ge c\mid\widehat{F}_{a,m}\right)\le Mc_{M-1}^{\mathrm{RH}}m^{(M-1)/2}\times\exp\left\{-mK_S^{\tau,+}(p,c)\right\}. P ( C a , m ≥ c ∣ F a , m ) ≤ M c M − 1 RH m ( M − 1 ) /2 × exp { − m K S τ , + ( p , c ) } .
Now compare the support-restricted and unrestricted information numbers. Every probability law of the form
∑ j = 1 M q j δ s j , q ∈ Δ M − 1 , \sum_{j=1}^{M}q_j\delta_{s_j},\qquad q\in\Delta_{M-1}, j = 1 ∑ M q j δ s j , q ∈ Δ M − 1 ,
is also an element of P ( [ 0 , R ] ) \mathcal{P}([0,R]) P ([ 0 , R ]) . Therefore the unrestricted feasible set is larger, and hence
K inf τ , + ( F ^ a , m , c ) ≤ K S τ , + ( p , c ) . \mathcal{K}_{\inf}^{\tau,+}(\widehat{F}_{a,m},c) \leq\mathcal{K}_{S}^{\tau,+}(p,c). K i n f τ , + ( F a , m , c ) ≤ K S τ , + ( p , c ) .
Consequently,
exp { − m K S τ , + ( p , c ) } ≤ exp { − m K inf τ , + ( F ^ a , m , c ) } . \exp\{-m\mathcal{K}_{S}^{\tau,+}(p,c)\} \leq\exp\left\{-m\mathcal{K}_{\inf}^{\tau,+}(\widehat{F}_{a,m},c)\right\}. exp { − m K S τ , + ( p , c )} ≤ exp { − m K i n f τ , + ( F a , m , c ) } .
It remains only to control the dimension-dependent prefactor uniformly in m m m . Set
d : = M − 1 , x : = m 2 π . d:=M-1,\qquad x:=\sqrt{\frac{m}{2\pi}}. d := M − 1 , x := 2 π m .
Since M = d + 1 M=d+1 M = d + 1 ,
M C M − 1 R H m ( M − 1 ) / 2 = ( d + 1 ) e 1 / 12 Γ ( d + 1 ) ( 1 2 π ) d m d / 2 = e 1 / 12 d + 1 d ! x d ≤ e 1 / 12 ∑ r = 0 ∞ r + 1 r ! x r = e 1 / 12 ( 1 + x ) e x ≤ e 1 / 12 e 2 x . \begin{aligned}
MC_{M-1}^{\mathrm{RH}}m^{(M-1)/2}=(d+1)\frac{e^{1/12}}{\Gamma(d+1)}\left(\frac{1}{\sqrt{2\pi}}\right)^d m^{d/2} \\
&=e^{1/12}\frac{d+1}{d!}x^d \\
&\leq e^{1/12}\sum_{r=0}^{\infty}\frac{r+1}{r!}x^r \\
&=e^{1/12}(1+x)e^x \\
&\leq e^{1/12}e^{2x}.
\end{aligned} M C M − 1 RH m ( M − 1 ) /2 = ( d + 1 ) Γ ( d + 1 ) e 1/12 ( 2 π 1 ) d m d /2 = e 1/12 d ! d + 1 x d ≤ e 1/12 r = 0 ∑ ∞ r ! r + 1 x r = e 1/12 ( 1 + x ) e x ≤ e 1/12 e 2 x .
For every η > 0 \eta>0 η > 0 ,
2 x = 2 m 2 π ≤ η m + 1 2 π η . 2x=2\sqrt{\frac{m}{2\pi}}\leq\eta m+\frac{1}{2\pi\eta}. 2 x = 2 2 π m ≤ η m + 2 π η 1 .
Hence
M C M − 1 R H m ( M − 1 ) / 2 ≤ exp { 1 12 + 1 2 π η } e η m . MC_{M-1}^{\mathrm{RH}}m^{(M-1)/2}\leq\exp\left\{\frac{1}{12}+\frac{1}{2\pi\eta}\right\}e^{\eta m}. M C M − 1 RH m ( M − 1 ) /2 ≤ exp { 12 1 + 2 π η 1 } e η m .
Define
C η : = exp { 1 12 + 1 2 π η } . C_\eta:=\exp\left\{\frac{1}{12}+\frac{1}{2\pi\eta}\right\}. C η := exp { 12 1 + 2 π η 1 } .
Combining the previous displays gives
P ( c ^ a , m τ ≥ c ∣ F ^ a , m ) ≤ C η exp { − m ( K inf τ , + ( F ^ a , m , c ) − η ) } . \mathbb{P}\left(\widehat{c}_{a,m}^{\tau}\geq c\mid\widehat{F}_{a,m}\right)\leq C_\eta\exp\left\{-m\left(\mathcal{K}_{\inf}^{\tau,+}(\widehat{F}_{a,m},c)-\eta\right)\right\}. P ( c a , m τ ≥ c ∣ F a , m ) ≤ C η exp { − m ( K i n f τ , + ( F a , m , c ) − η ) } .
This proves the claim.
Lemma C.10 (PATSO empirical-law concentration). Fix an arm a a a . Suppose the local observations from arm a a a , X a , 1 , X a , 2 , … X_{a,1},X_{a,2},\ldots X a , 1 , X a , 2 , … , are i.i.d. with law F a F_a F a . Define the empirical law
F ^ a , m : = 1 m ∑ r = 1 m δ X a , r . \widehat{F}_{a,m}:=\frac{1}{m}\sum_{r=1}^{m}\delta_{X_{a,r}}. F a , m := m 1 r = 1 ∑ m δ X a , r . Let D L D_L D L denote the Lévy distance. Then, for every ϵ > 0 \epsilon>0 ϵ > 0 ,
∑ m = 1 ∞ P ( D L ( F ^ a , m , F a ) > ϵ ) < ∞ . \sum_{m=1}^{\infty}\mathbb{P}\left(D_L(\widehat{F}_{a,m},F_a)>\epsilon\right)<\infty. m = 1 ∑ ∞ P ( D L ( F a , m , F a ) > ϵ ) < ∞.
Proof. Let
∥ F ^ a , m − F a ∥ ∞ : = sup x ∈ R ∣ F ^ a , m ( x ) − F a ( x ) ∣ \left\|\widehat{F}_{a,m}-F_a\right\|_{\infty}:=\sup_{x\in\mathbb{R}}\left|\widehat{F}_{a,m}(x)-F_a(x)\right| F a , m − F a ∞ := x ∈ R sup F a , m ( x ) − F a ( x )
be the Kolmogorov distance between the two distribution functions. The Lévy distance is bounded by the Kolmogorov distance:
D L ( F ^ a , m , F a ) ≤ ∥ F ^ a , m − F a ∥ ∞ . D_L(\widehat{F}_{a,m},F_a)\leq\left\|\widehat{F}_{a,m}-F_a\right\|_{\infty}. D L ( F a , m , F a ) ≤ F a , m − F a ∞ .
Indeed, if
∥ F ^ a , m − F a ∥ ∞ ≤ ϵ , \left\|\widehat{F}_{a,m}-F_a\right\|_\infty\le\epsilon, F a , m − F a ∞ ≤ ϵ ,
then for every x x x ,
F a ( x − ϵ ) − ϵ ≤ F a ( x ) − ϵ ≤ F ^ a , m ( x ) ≤ F a ( x ) + ϵ ≤ F a ( x + ϵ ) + ϵ , F_a(x-\epsilon)-\epsilon\le F_a(x)-\epsilon\le\widehat{F}_{a,m}(x) \le F_a(x)+\epsilon\le F_a(x+\epsilon)+\epsilon, F a ( x − ϵ ) − ϵ ≤ F a ( x ) − ϵ ≤ F a , m ( x ) ≤ F a ( x ) + ϵ ≤ F a ( x + ϵ ) + ϵ ,
which is exactly the defining Lévy-distance condition with radius ϵ \epsilon ϵ .
By the Dvoretzky–Kiefer–Wolfowitz inequality,
P ( ∥ F ^ a , m − F a ∥ ∞ > ϵ ) ≤ 2 exp ( − 2 m ϵ 2 ) . \mathbb{P}\left(\left\|\widehat{F}_{a,m}-F_a\right\|_\infty>\epsilon\right)\le2\exp(-2m\epsilon^2). P ( F a , m − F a ∞ > ϵ ) ≤ 2 exp ( − 2 m ϵ 2 ) .
Therefore
P ( D L ( F ^ a , m , F a ) > ϵ ) ≤ 2 exp ( − 2 m ϵ 2 ) . \mathbb{P}\left(D_L(\widehat{F}_{a,m},F_a)>\epsilon\right)\le2\exp(-2m\epsilon^2). P ( D L ( F a , m , F a ) > ϵ ) ≤ 2 exp ( − 2 m ϵ 2 ) .
Summing over m m m ,
∑ m = 1 ∞ P ( D L ( F ^ a , m , F a ) > ϵ ) ≤ 2 ∑ m = 1 ∞ e − 2 m ϵ 2 < ∞ . \sum_{m=1}^{\infty}\mathbb{P}\left(D_L(\widehat{F}_{a,m},F_a)>\epsilon\right)\le2\sum_{m=1}^{\infty}e^{-2m\epsilon^2}<\infty. m = 1 ∑ ∞ P ( D L ( F a , m , F a ) > ϵ ) ≤ 2 m = 1 ∑ ∞ e − 2 m ϵ 2 < ∞.
Lemma C.11 (Optimal-arm lower-tail term for uncapped PATSO ). Assume the stationary local-bandit model for the optimal arm a ⋆ a_{\star} a ⋆ , with rewards supported on [ 0 , R ] [0,R] [ 0 , R ] , and write a ⋆ : = a ⋆ τ a_{\star}:=a_{\star}^{\tau} a ⋆ := a ⋆ τ , F ⋆ : = F a ⋆ τ F_{\star}:=F_{a_{\star}^{\tau}} F ⋆ := F a ⋆ τ , c ⋆ τ : = CVaR τ ( F ⋆ ) c_{\star}^{\tau}:=\operatorname{CVaR}_{\tau}(F_{\star}) c ⋆ τ := CVaR τ ( F ⋆ ) . For uncapped PATSO, after m ≥ 1 m\ge1 m ≥ 1 pulls of a ⋆ τ a_{\star}^{\tau} a ⋆ τ the empirical and Thompson-sampled particle laws are
F ^ ⋆ , m = 1 m ∑ r = 1 m δ X ⋆ , r = ∑ j = 1 M m n j m δ s j , F ~ ⋆ , m = ∑ j = 1 M m L j δ s j , L ∼ Dir ( n 1 , … , n M m ) . \widehat{F}_{\star,m}=\frac{1}{m}\sum_{r=1}^{m}\delta_{X_{\star,r}}=\sum_{j=1}^{M_m}\frac{n_j}{m}\delta_{s_j},\qquad\widetilde{F}_{\star,m}=\sum_{j=1}^{M_m}L_j\delta_{s_j},\qquad L\sim\operatorname{Dir}(n_1,\ldots,n_{M_m}). F ⋆ , m = m 1 r = 1 ∑ m δ X ⋆ , r = j = 1 ∑ M m m n j δ s j , F ⋆ , m = j = 1 ∑ M m L j δ s j , L ∼ Dir ( n 1 , … , n M m ) . Assume each arm is initialized once, and let c ~ a ⋆ τ τ ( t ) : = CVaR τ ( F ~ ⋆ , N a ⋆ τ ( t ) ) . \widetilde{c}_{a_{\star}^{\tau}}^{\tau}(t):=\operatorname{CVaR}_{\tau}(\widetilde{F}_{\star,N_{a_{\star}^{\tau}}(t)}). c a ⋆ τ τ ( t ) := CVaR τ ( F ⋆ , N a ⋆ τ ( t ) ) . Then
L n , P A T S O τ : = E [ ∑ t = 1 n 1 { c ~ a ⋆ τ τ ( t ) < c ⋆ τ − C t 1 / 4 ( N a ⋆ τ ( t ) ∨ 1 ) 1 / 2 } ] = O ( 1 ) , \mathcal{L}_{n,\mathrm{PATSO}}^{\tau}:=\mathbb{E}\left[\sum_{t=1}^{n}\mathbf{1}\left\{\widetilde{c}_{a_{\star}^{\tau}}^{\tau}(t)<c_{\star}^{\tau}-C\frac{t^{1/4}}{(N_{a_{\star}^{\tau}}(t)\vee1)^{1/2}}\right\}\right]=O(1), L n , PATSO τ := E [ t = 1 ∑ n 1 { c a ⋆ τ τ ( t ) < c ⋆ τ − C ( N a ⋆ τ ( t ) ∨ 1 ) 1/2 t 1/4 } ] = O ( 1 ) , and in particular L n , P A T S O τ = O ( n 1 / 2 ) \mathcal{L}_{n,\mathrm{PATSO}}^{\tau}=O(n^{1/2}) L n , PATSO τ = O ( n 1/2 ) .
Proof. Let
m t : = N a ⋆ τ ( t ) ∨ 1 , Δ t : = C t 1 / 4 m t 1 / 2 . m_t:=N_{a_{\star}^{\tau}}(t)\vee1,\qquad\Delta_t:=C\frac{t^{1/4}}{m_t^{1/2}}. m t := N a ⋆ τ ( t ) ∨ 1 , Δ t := C m t 1/2 t 1/4 .
Define the event
E t : = { c ~ a ⋆ τ τ ( t ) < c ⋆ τ − Δ t } . E_t:=\left\{\widetilde{c}_{a_{\star}^{\tau}}^{\tau}(t)<c_{\star}^{\tau}-\Delta_t\right\}. E t := { c a ⋆ τ τ ( t ) < c ⋆ τ − Δ t } .
It is enough to show that
∑ t = 1 ∞ P ( E t ) < ∞ . \sum_{t=1}^{\infty}\mathbb{P}(E_t)<\infty. t = 1 ∑ ∞ P ( E t ) < ∞.
We use two standard concentration facts.
First, by the Dvoretzky–Kiefer–Wolfowitz inequality, for every m ≥ 1 m\ge1 m ≥ 1 and u > 0 u>0 u > 0 ,
P ( ∥ F ^ ⋆ , m − F ⋆ ∥ ∞ > u ) ≤ 2 e − 2 m u 2 . \mathbb{P}\left(\left\|\widehat{F}_{\star,m}-F_{\star}\right\|_\infty>u\right)\le2e^{-2mu^2}. P ( F ⋆ , m − F ⋆ ∞ > u ) ≤ 2 e − 2 m u 2 .
Second, conditionally on F ^ ⋆ , m \widehat{F}_{\star,m} F ⋆ , m , the Dirichlet-weighted CDF F ~ ⋆ , m ( x ) \widetilde{F}_{\star,m}(x) F ⋆ , m ( x ) satisfies the Bayesian-bootstrap DKW bound
P ( ∥ F ~ ⋆ , m − F ^ ⋆ , m ∥ ∞ > u ∣ F ^ ⋆ , m ) ≤ 2 m e − c B m u 2 \mathbb{P}\left(\left\|\widetilde{F}_{\star,m}-\widehat{F}_{\star,m}\right\|_\infty>u\mathrel{\big|}\widehat{F}_{\star,m}\right)\le2me^{-c_Bmu^2} P ( F ⋆ , m − F ⋆ , m ∞ > u F ⋆ , m ) ≤ 2 m e − c B m u 2
for a universal constant c B > 0 c_B > 0 c B > 0 . Indeed, at each empirical support point s k s_k s k ,
F ~ ⋆ , m ( s k ) = ∑ j ≤ k L j \widetilde{F}_{\star,m}(s_k) = \sum_{j \le k} L_j F ⋆ , m ( s k ) = j ≤ k ∑ L j
is beta-distributed with mean F ^ ⋆ , m ( s k ) \widehat{F}_{\star,m}(s_k) F ⋆ , m ( s k ) , by aggregation of Dirichlet coordinates. Applying a beta Chernoff bound and union-bounding over at most m m m support points gives the display.
Because the laws are supported on [ 0 , R ] [0,R] [ 0 , R ] ,
W 1 ( P , Q ) ≤ R ∥ F P − F Q ∥ ∞ . W_1(P,Q) \le R\lVert F_P-F_Q\rVert_\infty. W 1 ( P , Q ) ≤ R ∥ F P − F Q ∥ ∞ .
Also, CVaR is 1 / τ 1/\tau 1/ τ -Lipschitz in W 1 W_1 W 1 . Hence
∣ CVaR τ ( P ) − CVaR τ ( Q ) ∣ ≤ R τ ∥ F P − F Q ∥ ∞ . \left|\operatorname{CVaR}_\tau(P)-\operatorname{CVaR}_\tau(Q)\right| \le\frac{R}{\tau}\lVert F_P-F_Q\rVert_\infty. ∣ CVaR τ ( P ) − CVaR τ ( Q ) ∣ ≤ τ R ∥ F P − F Q ∥ ∞ .
Now fix t t t and condition on the event m t = m m_t=m m t = m . Set
Δ t , m : = C t 1 / 4 m 1 / 2 . \Delta_{t,m} := C\frac{t^{1/4}}{m^{1/2}}. Δ t , m := C m 1/2 t 1/4 .
Then
m Δ t , m 2 = C 2 t 1 / 2 . m\Delta_{t,m}^2=C^2t^{1/2}. m Δ t , m 2 = C 2 t 1/2 .
If
CVaR τ ( F ~ ⋆ , m ) < c ⋆ τ − Δ t , m , \operatorname{CVaR}_\tau(\widetilde{F}_{\star,m})<c_\star^\tau-\Delta_{t,m}, CVaR τ ( F ⋆ , m ) < c ⋆ τ − Δ t , m ,
then at least one of the following two events occurs:
G t , m : = { CVaR τ ( F ^ ⋆ , m ) < c ⋆ τ − Δ t , m 2 } , G_{t,m}:=\left\{\operatorname{CVaR}_\tau(\widehat{F}_{\star,m})<c_\star^\tau-\frac{\Delta_{t,m}}{2}\right\}, G t , m := { CVaR τ ( F ⋆ , m ) < c ⋆ τ − 2 Δ t , m } ,
or
H t , m : = { CVaR τ ( F ~ ⋆ , m ) < CVaR τ ( F ^ ⋆ , m ) − Δ t , m 2 } . H_{t,m}:=\left\{\operatorname{CVaR}_\tau(\widetilde{F}_{\star,m})<\operatorname{CVaR}_\tau(\widehat{F}_{\star,m})-\frac{\Delta_{t,m}}{2}\right\}. H t , m := { CVaR τ ( F ⋆ , m ) < CVaR τ ( F ⋆ , m ) − 2 Δ t , m } .
For G t , m G_{t,m} G t , m , the CVaR–Kolmogorov Lipschitz bound and DKW give
P ( G t , m ) ≤ P ( ∥ F ^ ⋆ , m − F ⋆ ∥ ∞ > τ Δ t , m 2 R ) \mathbb{P}(G_{t,m})\le\mathbb{P}\left(\lVert\widehat{F}_{\star,m}-F_\star\rVert_\infty>\frac{\tau\Delta_{t,m}}{2R}\right) P ( G t , m ) ≤ P ( ∥ F ⋆ , m − F ⋆ ∥ ∞ > 2 R τ Δ t , m )
≤ 2 exp ( − 2 m τ 2 Δ t , m 2 4 R 2 ) = 2 exp ( − τ 2 C 2 2 R 2 t 1 / 2 ) . \begin{aligned}
\le2\exp\left(-2m\frac{\tau^2\Delta_{t,m}^2}{4R^2}\right) \\
&=2\exp\left(-\frac{\tau^2C^2}{2R^2}t^{1/2}\right).
\end{aligned} ≤ 2 exp ( − 2 m 4 R 2 τ 2 Δ t , m 2 ) = 2 exp ( − 2 R 2 τ 2 C 2 t 1/2 ) .
For H t , m H_{t,m} H t , m , conditionally on F ^ ⋆ , m \widehat{F}_{\star,m} F ⋆ , m ,
P ( H t , m ∣ F ^ ⋆ , m ) ≤ P ( ∥ F ~ ⋆ , m − F ^ ⋆ , m ∥ ∞ > τ Δ t , m 2 R ∣ F ^ ⋆ , m ) \mathbb{P}(H_{t,m}\mid\widehat{F}_{\star,m})\le\mathbb{P}\left(\lVert\widetilde{F}_{\star,m}-\widehat{F}_{\star,m}\rVert_\infty>\frac{\tau\Delta_{t,m}}{2R}\mathrel{\Big|}\widehat{F}_{\star,m}\right) P ( H t , m ∣ F ⋆ , m ) ≤ P ( ∥ F ⋆ , m − F ⋆ , m ∥ ∞ > 2 R τ Δ t , m F ⋆ , m )
≤ 2 m exp ( − c B m τ 2 Δ t , m 2 4 R 2 ) \le2m\exp\left(-c_Bm\frac{\tau^2\Delta_{t,m}^2}{4R^2}\right) ≤ 2 m exp ( − c B m 4 R 2 τ 2 Δ t , m 2 )
≤ 2 t exp ( − c B τ 2 C 2 4 R 2 t 1 / 2 ) , \le2t\exp\left(-\frac{c_B\tau^2C^2}{4R^2}t^{1/2}\right), ≤ 2 t exp ( − 4 R 2 c B τ 2 C 2 t 1/2 ) ,
because m ≤ t m\le t m ≤ t .
Therefore, using the union bound over possible values m ∈ { 1 , … , t } m\in\{1,\ldots,t\} m ∈ { 1 , … , t } ,
P ( E t ) ≤ C 1 t exp ( − C 2 t ) \mathbb{P}(E_t)\le C_1t\exp(-C_2\sqrt{t}) P ( E t ) ≤ C 1 t exp ( − C 2 t )
for constants C 1 , C 2 > 0 C_1,C_2>0 C 1 , C 2 > 0 depending only on ( C , τ , R ) (C,\tau,R) ( C , τ , R ) . Since
∑ t = 1 ∞ t e − C 2 t < ∞ , \sum_{t=1}^{\infty}te^{-C_2\sqrt{t}}<\infty, t = 1 ∑ ∞ t e − C 2 t < ∞ ,
we get
sup n ≥ 1 L n , P A T S O τ ≤ ∑ t = 1 ∞ P ( E t ) < ∞ . \sup_{n\ge1}\mathcal{L}^{\tau}_{n,\mathrm{PATSO}}
\le\sum_{t=1}^{\infty}\mathbb{P}(E_t)<\infty. n ≥ 1 sup L n , PATSO τ ≤ t = 1 ∑ ∞ P ( E t ) < ∞.
This proves the claim.
PATSO CVaR suboptimal-arm pulls
Lemma C.12 (PATSO CVaR suboptimal-arm pulls). Adopt the notation of § 5.1 with a unique CVaR-optimal arm, and write T a ( t ) : = T a ( t − 1 ) T_a(t) := T_a(t-1) T a ( t ) := T a ( t − 1 ) , c ~ a ( t ) : = c ~ a , T a ( t ) \widetilde{c}_a(t) := \widetilde{c}_{a,T_a(t)} c a ( t ) := c a , T a ( t ) . Assume PATSO selects
I ( t ) ∈ arg max a ∈ [ K ] { c ~ a ( t ) + C ( T a ( t ) ∨ 1 ) 1 / 2 } . I(t) \in\arg\max_{a \in[K]} \left\{ \widetilde{c}_a(t) + \frac{C}{(T_a(t) \vee1)^{1/2}} \right\}. I ( t ) ∈ arg a ∈ [ K ] max { c a ( t ) + ( T a ( t ) ∨ 1 ) 1/2 C } . and that F ↦ K inf τ , + ( F , u ) F \mapsto\mathcal{K}_{\inf}^{\tau,+}(F,u) F ↦ K i n f τ , + ( F , u ) is D L D_L D L -continuous for every fixed u < c ⋆ τ u < c_\star^\tau u < c ⋆ τ and c ↦ K inf τ , + ( F a , c ) c \mapsto\mathcal{K}_{\inf}^{\tau,+}(F_a,c) c ↦ K i n f τ , + ( F a , c ) is left-continuous at c = c ⋆ τ c = c_\star^\tau c = c ⋆ τ . Then for every suboptimal a ≠ a ⋆ τ a \ne a_\star^\tau a = a ⋆ τ and every ϵ 0 > 0 \epsilon_0 > 0 ϵ 0 > 0 ,
E [ T a ( n ) ] ≤ ( 1 + ϵ 0 ) log n K inf τ , + ( F a , c ⋆ τ ) + O ( n 1 / 2 ) . \mathbb{E}[T_a(n)] \le\frac{(1+\epsilon_0)\log n}{\mathcal{K}_{\inf}^{\tau,+}(F_a,c_\star^\tau)} + O(n^{1/2}). E [ T a ( n )] ≤ K i n f τ , + ( F a , c ⋆ τ ) ( 1 + ϵ 0 ) log n + O ( n 1/2 ) .
Proof. Fix a CVaR-suboptimal arm a ≠ a ⋆ τ a \ne a_\star^\tau a = a ⋆ τ , and write
β t ( a ) : = C ( T a ( t ) ∨ 1 ) 1 / 2 . \beta_t(a) := \frac{C}{(T_a(t) \vee1)^{1/2}}. β t ( a ) := ( T a ( t ) ∨ 1 ) 1/2 C .
Assume each arm is initialized once, or equivalently absorb the finitely many rounds with T a ( t ) = 0 T_a(t) = 0 T a ( t ) = 0 into the final O ( 1 ) O(1) O ( 1 ) term. Hence below F ^ a , m \widehat{F}_{a,m} F a , m is only used for m ≥ 1 m \ge1 m ≥ 1 .
Since a a a is CVaR-suboptimal and the CVaR-optimal arm is unique,
c a τ < c ⋆ τ . c_a^\tau< c_\star^\tau. c a τ < c ⋆ τ .
Choose
0 < ϵ 1 < c ⋆ τ − c a τ 0 < \epsilon_1 < c_\star^\tau- c_a^\tau 0 < ϵ 1 < c ⋆ τ − c a τ
so that
u : = c ⋆ τ − ϵ 1 u := c_\star^\tau- \epsilon_1 u := c ⋆ τ − ϵ 1
satisfies
c a τ < u < c ⋆ τ . c_a^\tau< u < c_\star^\tau. c a τ < u < c ⋆ τ .
Let η , ϵ 3 > 0 \eta,\epsilon_3 > 0 η , ϵ 3 > 0 , and define
κ a τ ( ϵ 1 , η , ϵ 3 ) : = K inf τ , + ( F a , c ⋆ τ − ϵ 1 ) − η − ϵ 3 . \kappa_a^\tau(\epsilon_1,\eta,\epsilon_3) := \mathcal{K}_{\inf}^{\tau,+}(F_a,c_\star^\tau-\epsilon_1) - \eta- \epsilon_3. κ a τ ( ϵ 1 , η , ϵ 3 ) := K i n f τ , + ( F a , c ⋆ τ − ϵ 1 ) − η − ϵ 3 .
For ϵ 1 , η , ϵ 3 \epsilon_1,\eta,\epsilon_3 ϵ 1 , η , ϵ 3 sufficiently small,
κ a τ ( ϵ 1 , η , ϵ 3 ) > 0. \kappa_a^\tau(\epsilon_1,\eta,\epsilon_3) > 0. κ a τ ( ϵ 1 , η , ϵ 3 ) > 0.
Indeed, by left-continuity of c ↦ K inf τ , + ( F a , c ) c \mapsto\mathcal{K}_{\inf}^{\tau,+}(F_a,c) c ↦ K i n f τ , + ( F a , c ) at c = c ⋆ τ c = c_\star^\tau c = c ⋆ τ , and because c ⋆ τ > c a τ c_\star^\tau> c_a^\tau c ⋆ τ > c a τ , the information number at c ⋆ τ − ϵ 1 c_\star^\tau-\epsilon_1 c ⋆ τ − ϵ 1 is positive for ϵ 1 \epsilon_1 ϵ 1 small enough.
Set
C ⋆ : = C τ , η , c ⋆ τ − ϵ 1 , C_\star:= C_{\tau,\eta,c_\star^\tau-\epsilon_1}, C ⋆ := C τ , η , c ⋆ τ − ϵ 1 ,
m β : = ⌈ ( C ϵ 1 ) 1 / 2 n 1 / 2 ⌉ , m_\beta:= \left\lceil\left(\frac{C}{\epsilon_1}\right)^{1/2} n^{1/2} \right\rceil, m β := ⌈ ( ϵ 1 C ) 1/2 n 1/2 ⌉ ,
m K L : = ⌈ log ( C ⋆ n ) κ a τ ( ϵ 1 , η , ϵ 3 ) ⌉ , m_{\mathrm{KL}} := \left\lceil\frac{\log(C_\star n)}{\kappa_a^\tau(\epsilon_1,\eta,\epsilon_3)} \right\rceil, m KL := ⌈ κ a τ ( ϵ 1 , η , ϵ 3 ) log ( C ⋆ n ) ⌉ ,
and
m 0 : = max { 1 , m β , m K L } . m_0 := \max\{1,m_\beta,m_{\mathrm{KL}}\}. m 0 := max { 1 , m β , m KL } .
For small n n n , one may replace log ( C ⋆ n ) \log(C_\star n) log ( C ⋆ n ) by log ( e C ⋆ n ) \log(eC_\star n) log ( e C ⋆ n ) ; this only changes constants and avoids irrelevant negative-log edge cases.
We decompose
E [ T a ( n ) ] = E [ ∑ t = 1 n 1 { I ( t ) = a } ] ≤ m 0 + B 1 + B 2 , \mathbb{E}[T_a(n)] = \mathbb{E}\left[\sum_{t=1}^{n} \mathbb{1}\{I(t)=a\}\right] \le m_0+B_1+B_2, E [ T a ( n )] = E [ t = 1 ∑ n 1 { I ( t ) = a } ] ≤ m 0 + B 1 + B 2 ,
where
B 1 : = E [ ∑ t = 1 n 1 { I ( t ) = a , T a ( t ) ≥ m 0 , c ~ a ( t ) ≥ c ⋆ τ − β t ( a ) , D L ( F ^ a , T a ( t ) , F a ) ≤ ϵ 2 } ] , B_1 := \mathbb{E}\left[\sum_{t=1}^{n} \mathbb{1}\left\{I(t)=a,\ T_a(t)\ge m_0,\ \widetilde{c}_a(t)\ge c_\star^\tau-\beta_t(a),\ D_L(\widehat{F}_{a,T_a(t)},F_a)\le\epsilon_2\right\}\right], B 1 := E [ t = 1 ∑ n 1 { I ( t ) = a , T a ( t ) ≥ m 0 , c a ( t ) ≥ c ⋆ τ − β t ( a ) , D L ( F a , T a ( t ) , F a ) ≤ ϵ 2 } ] ,
and
B 2 : = E [ ∑ t = 1 n 1 { I ( t ) = a , T a ( t ) ≥ m 0 , [ c ~ a ( t ) < c ⋆ τ − β t ( a ) ] ∪ [ D L ( F ^ a , T a ( t ) , F a ) > ϵ 2 ] } ] . B_2 := \mathbb{E}\left[\sum_{t=1}^{n} \mathbb{1}\left\{I(t)=a,\ T_a(t)\ge m_0,\ \left[\widetilde{c}_a(t)<c_\star^\tau-\beta_t(a)\right]\cup\left[D_L(\widehat{F}_{a,T_a(t)},F_a)>\epsilon_2\right]\right\}\right]. B 2 := E [ t = 1 ∑ n 1 { I ( t ) = a , T a ( t ) ≥ m 0 , [ c a ( t ) < c ⋆ τ − β t ( a ) ] ∪ [ D L ( F a , T a ( t ) , F a ) > ϵ 2 ] } ] .
Bound on B 1 B_1 B 1 . Dropping the event { I ( t ) = a } \{I(t)=a\} { I ( t ) = a } and conditioning on the local sample size,
B 1 ≤ ∑ t = 1 n ∑ m = m 0 n − 1 P ( c ^ a τ ( t ) ≥ c ⋆ τ − β t ( a ) , D L ( F ^ a , m , F a ) ≤ ϵ 2 , T a ( t ) = m ) . B_1 \le\sum_{t=1}^{n}\sum_{m=m_0}^{n-1}\mathbb{P}\left(\widehat{c}_a^\tau(t) \ge c_\star^\tau-\beta_t(a),\ D_L(\widehat{F}_{a,m},F_a)\le\epsilon_2,\ T_a(t)=m\right). B 1 ≤ t = 1 ∑ n m = m 0 ∑ n − 1 P ( c a τ ( t ) ≥ c ⋆ τ − β t ( a ) , D L ( F a , m , F a ) ≤ ϵ 2 , T a ( t ) = m ) .
If m ≥ m 0 ≥ m β m\ge m_0\ge m_\beta m ≥ m 0 ≥ m β and t ≤ n t\le n t ≤ n , then
β t ( a ) = C t 1 / 4 ( m ∨ 1 ) 1 / 2 ≤ C n 1 / 4 m 0 1 / 2 ≤ ϵ 1 . \beta_t(a)=C\frac{t^{1/4}}{(m\vee1)^{1/2}}\le C\frac{n^{1/4}}{m_0^{1/2}}\le\epsilon_1. β t ( a ) = C ( m ∨ 1 ) 1/2 t 1/4 ≤ C m 0 1/2 n 1/4 ≤ ϵ 1 .
Therefore
c ^ a τ ( t ) ≥ c ⋆ τ − β t ( a ) ⟹ c ^ a τ ( t ) ≥ c ⋆ τ − ϵ 1 . \widehat{c}_a^\tau(t)\ge c_\star^\tau-\beta_t(a)\quad\Longrightarrow\quad\widehat{c}_a^\tau(t)\ge c_\star^\tau-\epsilon_1. c a τ ( t ) ≥ c ⋆ τ − β t ( a ) ⟹ c a τ ( t ) ≥ c ⋆ τ − ϵ 1 .
Hence
B 1 ≤ ∑ t = 1 n ∑ m = m 0 n − 1 P ( c ^ a τ ( t ) ≥ c ⋆ τ − ϵ 1 , D L ( F ^ a , m , F a ) ≤ ϵ 2 , T a ( t ) = m ) . B_1\le\sum_{t=1}^{n}\sum_{m=m_0}^{n-1}\mathbb{P}\left(\widehat{c}_a^\tau(t)\ge c_\star^\tau-\epsilon_1,\ D_L(\widehat{F}_{a,m},F_a)\le\epsilon_2,\ T_a(t)=m\right). B 1 ≤ t = 1 ∑ n m = m 0 ∑ n − 1 P ( c a τ ( t ) ≥ c ⋆ τ − ϵ 1 , D L ( F a , m , F a ) ≤ ϵ 2 , T a ( t ) = m ) .
By the assumed continuity of
F ↦ K inf τ , + ( F , c ⋆ τ − ϵ 1 ) F\mapsto K_{\inf}^{\tau,+}(F,c_\star^\tau-\epsilon_1) F ↦ K i n f τ , + ( F , c ⋆ τ − ϵ 1 )
at F a F_a F a , for every ϵ 3 > 0 \epsilon_3>0 ϵ 3 > 0 there exists ϵ 2 > 0 \epsilon_2>0 ϵ 2 > 0 such that
D L ( F ^ a , m , F a ) ≤ ϵ 2 ⟹ K inf τ , + ( F ^ a , m , c ⋆ τ − ϵ 1 ) ≥ K inf τ , + ( F a , c ⋆ τ − ϵ 1 ) − ϵ 3 . D_L(\widehat{F}_{a,m},F_a)\le\epsilon_2\quad\Longrightarrow\quad K_{\inf}^{\tau,+}(\widehat{F}_{a,m},c_\star^\tau-\epsilon_1)\ge K_{\inf}^{\tau,+}(F_a,c_\star^\tau-\epsilon_1)-\epsilon_3. D L ( F a , m , F a ) ≤ ϵ 2 ⟹ K i n f τ , + ( F a , m , c ⋆ τ − ϵ 1 ) ≥ K i n f τ , + ( F a , c ⋆ τ − ϵ 1 ) − ϵ 3 .
Using the history-conditional form of Assumption C.8, on the event
{ T a ( t ) = m , D L ( F ^ a , m , F a ) ≤ ϵ 2 } , \{T_a(t)=m,\ D_L(\widehat{F}_{a,m},F_a)\le\epsilon_2\}, { T a ( t ) = m , D L ( F a , m , F a ) ≤ ϵ 2 } ,
we have
P ( c ^ a τ ( t ) ≥ c ⋆ τ − ϵ 1 ∣ F t − 1 ) ≤ C ⋆ exp { − m κ a τ ( ϵ 1 , η , ϵ 3 ) } . \mathbb{P}\left(\widehat{c}_a^\tau(t)\ge c_\star^\tau-\epsilon_1\mid\mathcal{F}_{t-1}\right)\le C_\star\exp\left\{-m\kappa_a^\tau(\epsilon_1,\eta,\epsilon_3)\right\}. P ( c a τ ( t ) ≥ c ⋆ τ − ϵ 1 ∣ F t − 1 ) ≤ C ⋆ exp { − m κ a τ ( ϵ 1 , η , ϵ 3 ) } .
Thus
B 1 ≤ C ⋆ ∑ t = 1 n ∑ m = m 0 n − 1 exp { − m κ a τ ( ϵ 1 , η , ϵ 3 ) } × P ( D L ( F ^ a , m , F a ) ≤ ϵ 2 , T a ( t ) = m ) . B_1\le C_\star\sum_{t=1}^{n}\sum_{m=m_0}^{n-1}\exp\left\{-m\kappa_a^\tau(\epsilon_1,\eta,\epsilon_3)\right\}\times\mathbb{P}\left(D_L(\widehat{F}_{a,m},F_a)\le\epsilon_2,\ T_a(t)=m\right). B 1 ≤ C ⋆ t = 1 ∑ n m = m 0 ∑ n − 1 exp { − m κ a τ ( ϵ 1 , η , ϵ 3 ) } × P ( D L ( F a , m , F a ) ≤ ϵ 2 , T a ( t ) = m ) .
Since m ≥ m 0 m\ge m_0 m ≥ m 0 ,
exp { − m κ a τ } ≤ exp { − m 0 κ a τ } . \exp\{-m\kappa_a^\tau\}\le\exp\{-m_0\kappa_a^\tau\}. exp { − m κ a τ } ≤ exp { − m 0 κ a τ } .
Also, for each fixed t t t ,
∑ m = m 0 n − 1 P ( D L ( F ^ a , m , F a ) ≤ ϵ 2 , T a ( t ) = m ) ≤ 1. \sum_{m=m_0}^{n-1}\mathbb{P}\left(D_L(\widehat{F}_{a,m},F_a)\le\epsilon_2,\ T_a(t)=m\right)\le1. m = m 0 ∑ n − 1 P ( D L ( F a , m , F a ) ≤ ϵ 2 , T a ( t ) = m ) ≤ 1.
Therefore
B 1 ≤ C ⋆ n exp { − m 0 κ a τ ( ϵ 1 , η , ϵ 3 ) } ≤ 1 B_1\le C_\star n\exp\{-m_0\kappa_a^\tau(\epsilon_1,\eta,\epsilon_3)\}\le1 B 1 ≤ C ⋆ n exp { − m 0 κ a τ ( ϵ 1 , η , ϵ 3 )} ≤ 1
by the definition of m K L m_{\mathrm{KL}} m KL .
Bound on B 2 B_2 B 2 . Write
B 2 ≤ R 1 + R 2 , B_2\le R_1+R_2, B 2 ≤ R 1 + R 2 ,
where
R 1 : = E [ ∑ t = 1 n 1 { I ( t ) = a , c ^ a τ ( t ) < c ⋆ τ − β t ( a ) } ] = O ( n 1 / 2 ) , R_1:=\mathbb{E}\left[\sum_{t=1}^{n}\mathbf{1}\{I(t)=a,\ \widehat{c}_a^\tau(t)<c_\star^\tau-\beta_t(a)\}\right]=O(n^{1/2}), R 1 := E [ t = 1 ∑ n 1 { I ( t ) = a , c a τ ( t ) < c ⋆ τ − β t ( a )} ] = O ( n 1/2 ) ,
and
R 2 : = E [ ∑ t = 1 n 1 { I ( t ) = a , T a ( t ) ≥ m 0 , D L ( F ^ a , T a ( t ) , F a ) > ϵ 2 } ] . R_2:=\mathbb{E}\left[\sum_{t=1}^{n}\mathbf{1}\{I(t)=a,\ T_a(t)\ge m_0,\ D_L(\widehat{F}_{a,T_a(t)},F_a)>\epsilon_2\}\right]. R 2 := E [ t = 1 ∑ n 1 { I ( t ) = a , T a ( t ) ≥ m 0 , D L ( F a , T a ( t ) , F a ) > ϵ 2 } ] .
For R 2 R_2 R 2 , the event
{ I ( t ) = a , T a ( t ) = m } \{I(t)=a,\ T_a(t)=m\} { I ( t ) = a , T a ( t ) = m }
can occur at most once for each m m m . Hence
R 2 ≤ ∑ m = m 0 n − 1 P ( D L ( F ^ a , m , F a ) > ϵ 2 ) = O ( 1 ) R_2 \le\sum_{m=m_0}^{n-1} \mathbb{P}\left(D_L(\widehat{F}_{a,m}, F_a) > \epsilon_2\right) = O(1) R 2 ≤ m = m 0 ∑ n − 1 P ( D L ( F a , m , F a ) > ϵ 2 ) = O ( 1 )
by Lemma C.10. This is why it is cleaner to sum from m = m 0 m=m_0 m = m 0 , rather than from m = 0 m=0 m = 0 .
Combining the bounds on B 1 B_1 B 1 , R 1 R_1 R 1 , and R 2 R_2 R 2 ,
E [ T a ( n ) ] ≤ m 0 + O ( n 1 / 2 ) + O ( 1 ) . \mathbb{E}[T_a(n)] \le m_0 + O(n^{1/2}) + O(1). E [ T a ( n )] ≤ m 0 + O ( n 1/2 ) + O ( 1 ) .
By the definition of m 0 m_0 m 0 ,
m 0 ≤ log ( C ∗ n ) κ a τ ( ϵ 1 , η , ϵ 3 ) + O ( n 1 / 2 ) . m_0 \le\frac{\log(C_* n)}{\kappa_a^\tau(\epsilon_1,\eta,\epsilon_3)} + O(n^{1/2}). m 0 ≤ κ a τ ( ϵ 1 , η , ϵ 3 ) log ( C ∗ n ) + O ( n 1/2 ) .
Finally, by left-continuity of
c ↦ K i n f τ , + ( F a , c ) c \mapsto\mathcal{K}_{\mathrm{inf}}^{\tau,+}(F_a,c) c ↦ K inf τ , + ( F a , c )
at c = c ∗ τ c=c_*^\tau c = c ∗ τ , for every ϵ 0 > 0 \epsilon_0>0 ϵ 0 > 0 we can choose ϵ 1 , η , ϵ 3 > 0 \epsilon_1,\eta,\epsilon_3>0 ϵ 1 , η , ϵ 3 > 0 sufficiently small so that
1 κ a τ ( ϵ 1 , η , ϵ 3 ) ≤ 1 + ϵ 0 / 2 K i n f τ , + ( F a , c ∗ τ ) . \frac{1}{\kappa_a^\tau(\epsilon_1,\eta,\epsilon_3)} \le\frac{1+\epsilon_0/2}{\mathcal{K}_{\mathrm{inf}}^{\tau,+}(F_a,c_*^\tau)}. κ a τ ( ϵ 1 , η , ϵ 3 ) 1 ≤ K inf τ , + ( F a , c ∗ τ ) 1 + ϵ 0 /2 .
Since C ∗ C_* C ∗ is fixed after ϵ 1 , η \epsilon_1,\eta ϵ 1 , η are fixed,
log ( C ∗ n ) = log n + O ( 1 ) . \log(C_*n)=\log n+O(1). log ( C ∗ n ) = log n + O ( 1 ) .
Therefore, for all sufficiently large n n n ,
log ( C ∗ n ) κ a τ ( ϵ 1 , η , ϵ 3 ) ≤ ( 1 + ϵ 0 ) log n K i n f τ , + ( F a , c ∗ τ ) . \frac{\log(C_*n)}{\kappa_a^\tau(\epsilon_1,\eta,\epsilon_3)} \le\frac{(1+\epsilon_0)\log n}{\mathcal{K}_{\mathrm{inf}}^{\tau,+}(F_a,c_*^\tau)}. κ a τ ( ϵ 1 , η , ϵ 3 ) log ( C ∗ n ) ≤ K inf τ , + ( F a , c ∗ τ ) ( 1 + ϵ 0 ) log n .
Absorbing finitely many small-n n n cases into the O ( n 1 / 2 ) O(n^{1/2}) O ( n 1/2 ) term gives
E [ T a ( n ) ] ≤ ( 1 + ϵ 0 ) log n K i n f τ , + ( F a , c ∗ τ ) + O ( n 1 / 2 ) , \mathbb{E}[T_a(n)] \le\frac{(1+\epsilon_0)\log n}{\mathcal{K}_{\mathrm{inf}}^{\tau,+}(F_a,c_*^\tau)} + O(n^{1/2}), E [ T a ( n )] ≤ K inf τ , + ( F a , c ∗ τ ) ( 1 + ϵ 0 ) log n + O ( n 1/2 ) ,
as claimed.
Remark C.13 (Uncapped PATSO). The statement above is for the uncapped particle representation. A cap-and-merge implementation adds a separate approximation bias, which should be tracked explicitly rather than folded into the risk-KL term.
Lemma C.14. Let CATSO be applied to a non-stationary bandit problem whose payoff sequence satisfies Assumption C.8. Define the estimator
c ^ n τ = ∑ a = 1 K T a ( n ) n c ^ a , T a ( n ) , \widehat{c}_n^\tau= \sum_{a=1}^{K} \frac{T_a(n)}{n}\widehat{c}_{a,T_a(n)}, c n τ = a = 1 ∑ K n T a ( n ) c a , T a ( n ) , and let δ ⋆ , n = c ⋆ τ − c ⋆ , n τ \delta_{\star,n}=c_\star^\tau-c_{\star,n}^\tau δ ⋆ , n = c ⋆ τ − c ⋆ , n τ . For any ϵ 0 > 0 \epsilon_0>0 ϵ 0 > 0 , the following upper bound holds:
E [ ∣ c ^ n − c ⋆ , n τ ∣ ] ≤ ∣ δ ⋆ , n ∣ + R n ∑ a ≠ a ⋆ K { ( 1 + ϵ 0 ) log n K i n f ( N ) , τ , + ( F a , c ⋆ τ ) + O ( n ) } . \mathbb{E}\left[\left|\widehat{c}_n-c_{\star,n}^\tau\right|\right] \le|\delta_{\star,n}|+\frac{R}{n}\sum_{a\ne a_\star}^{K}\left\{\frac{(1+\epsilon_0)\log n}{\mathcal{K}_{\mathrm{inf}}^{(N),\tau,+}(F_a,c_\star^\tau)}+O(\sqrt{n})\right\}. E [ c n − c ⋆ , n τ ] ≤ ∣ δ ⋆ , n ∣ + n R a = a ⋆ ∑ K { K inf ( N ) , τ , + ( F a , c ⋆ τ ) ( 1 + ϵ 0 ) log n + O ( n ) } .
Proof. We observe that
∣ c ^ n − c ⋆ τ ∣ ≤ ∣ c ^ n − c ⋆ , n τ ∣ + ∣ c ⋆ τ − c ⋆ , n τ ∣ = ∣ c ^ n − c ⋆ , n τ ∣ + ∣ δ ⋆ , n ∣ (23) |\widehat{c}_n-c_\star^\tau| \le|\widehat{c}_n-c_{\star,n}^\tau|+|c_\star^\tau-c_{\star,n}^\tau|=|\widehat{c}_n-c_{\star,n}^\tau|+|\delta_{\star,n}| \tag*{(23)} ∣ c n − c ⋆ τ ∣ ≤ ∣ c n − c ⋆ , n τ ∣ + ∣ c ⋆ τ − c ⋆ , n τ ∣ = ∣ c n − c ⋆ , n τ ∣ + ∣ δ ⋆ , n ∣ (23)
We have
c ^ n − c ⋆ , n τ = ∑ a ≠ a ⋆ K T a ( n ) ( c ^ a , T a ( n ) − μ ∗ , n τ ) n (24) \widehat{c}_n-c_{\star,n}^\tau=\frac{\sum_{a\ne a_\star}^{K}T_a(n)(\widehat{c}_{a,T_a(n)}-\mu_{*,n}^\tau)}{n} \tag*{(24)} c n − c ⋆ , n τ = n ∑ a = a ⋆ K T a ( n ) ( c a , T a ( n ) − μ ∗ , n τ ) (24)
Therefore
E [ ∣ c ^ n − c ⋆ , n τ ∣ ] ≤ R ∑ a ≠ a ⋆ K E [ T a ( n ) ] n (25) \mathbb{E}[|\widehat{c}_n-c_{\star,n}^\tau|]\le R\sum_{a\ne a_\star}^{K}\frac{\mathbb{E}[T_a(n)]}{n} \tag*{(25)} E [ ∣ c n − c ⋆ , n τ ∣ ] ≤ R a = a ⋆ ∑ K n E [ T a ( n )] (25)
According to Lemma C.7 , we have
E [ ∣ c ^ n τ − c ⋆ , n τ ∣ ] ≤ R ∑ a ≠ a ⋆ K E [ T a ( n ) ] n ≤ R n ∑ a ≠ a ⋆ K { ( 1 + ϵ 0 ) log n K inf ( N ) , τ , + ( F a , c ⋆ τ ) + O ( n ) } , (26) \mathbb{E}[|\widehat{c}^{\tau}_{n}-c^{\tau}_{\star,n}|] \le R \sum_{a\ne a_{\star}}^{K} \frac{\mathbb{E}[T_a(n)]}{n} \le\frac{R}{n} \sum_{a\ne a_{\star}}^{K} \left\{ \frac{(1+\epsilon_0)\log n}{\mathcal{K}^{(N),\tau,+}_{\inf}(F_a,c^{\tau}_{\star})} + O(\sqrt{n}) \right\}, \tag*{(26)} E [ ∣ c n τ − c ⋆ , n τ ∣ ] ≤ R a = a ⋆ ∑ K n E [ T a ( n )] ≤ n R a = a ⋆ ∑ K { K i n f ( N ) , τ , + ( F a , c ⋆ τ ) ( 1 + ϵ 0 ) log n + O ( n ) } , (26)
which concludes the proof. □
Lemma C.15. Let PATSO (Particle Thompson Sampling with Optimistic Bonus) be applied to a non-stationary bandit problem whose payoff process satisfies Assumption 5.2. Define the estimator
c ^ n τ = ∑ a = 1 K T a ( n ) n c ^ a , T a ( n ) τ , \widehat{c}^{\tau}_{n}=\sum_{a=1}^{K}\frac{T_a(n)}{n}\widehat{c}^{\tau}_{a,T_a(n)}, c n τ = a = 1 ∑ K n T a ( n ) c a , T a ( n ) τ , and let δ ⋆ , n = c ⋆ τ − c ⋆ , n τ \delta_{\star,n}=c^{\tau}_{\star}-c^{\tau}_{\star,n} δ ⋆ , n = c ⋆ τ − c ⋆ , n τ . For any p ≥ 1 p\ge1 p ≥ 1 and ϵ 0 > 0 \epsilon_0>0 ϵ 0 > 0 , the following holds:
E [ ∣ c ^ n τ − c ⋆ τ ∣ ] ≤ ∣ δ ⋆ , n ∣ + R n ∑ a ≠ a ⋆ K { ( 1 + ϵ 0 ) log n K inf τ , + ( F a , c ⋆ τ ) + O ( n ) } . \mathbb{E}\left[|\widehat{c}^{\tau}_{n}-c^{\tau}_{\star}|\right] \le|\delta_{\star,n}|+\frac{R}{n}\sum_{a\ne a_{\star}}^{K}\left\{\frac{(1+\epsilon_0)\log n}{\mathcal{K}^{\tau,+}_{\inf}(F_a,c^{\tau}_{\star})}+O(\sqrt{n})\right\}. E [ ∣ c n τ − c ⋆ τ ∣ ] ≤ ∣ δ ⋆ , n ∣ + n R a = a ⋆ ∑ K { K i n f τ , + ( F a , c ⋆ τ ) ( 1 + ϵ 0 ) log n + O ( n ) } .
Proof. Similar to Lemma C.14 , we can derive
E [ ∣ c ^ n τ − c ⋆ , n τ ∣ ] ≤ ∣ δ ⋆ , n ∣ + R ∑ a ≠ a ⋆ K E [ T a ( n ) ] n . (27) \mathbb{E}[|\widehat{c}^{\tau}_{n}-c^{\tau}_{\star,n}|]\le|\delta_{\star,n}|+R\sum_{a\ne a_{\star}}^{K}\frac{\mathbb{E}[T_a(n)]}{n}. \tag*{(27)} E [ ∣ c n τ − c ⋆ , n τ ∣ ] ≤ ∣ δ ⋆ , n ∣ + R a = a ⋆ ∑ K n E [ T a ( n )] . (27)
and the bound is obtained as a direct corollary of Lemma C.12 . □
Proof Of Theorem 5.4 for CATSO case
Proof. We first prove that lim n → ∞ E [ c ^ n τ ] = c ⋆ τ \lim_{n\to\infty}\mathbb{E}[\widehat{c}^{\tau}_{n}]=c^{\tau}_{\star} lim n → ∞ E [ c n τ ] = c ⋆ τ . According to the result of Lemma C.14 , we have
E [ ∣ c ^ n τ − c ⋆ τ ∣ ] ≤ ∣ δ ⋆ , n ∣ + R ∑ a = 1 , a ≠ a ⋆ K E [ T a ( n ) ] n (28) \mathbb{E}[|\widehat{c}^{\tau}_{n}-c^{\tau}_{\star}|]\le|\delta_{\star,n}|+R\sum_{a=1,a\ne a_{\star}}^{K}\frac{\mathbb{E}[T_a(n)]}{n} \tag*{(28)} E [ ∣ c n τ − c ⋆ τ ∣ ] ≤ ∣ δ ⋆ , n ∣ + R a = 1 , a = a ⋆ ∑ K n E [ T a ( n )] (28)
≤ ∣ δ ⋆ , n ∣ + R n ∑ a = 1 , a ≠ a ⋆ K { ( 1 + ϵ 0 ) log n K inf ( N ) , τ , + ( F a , c ⋆ τ ) + O ( n ) } (29) \le|\delta_{\star,n}|+\frac{R}{n}\sum_{a=1,a\ne a_{\star}}^{K}\left\{\frac{(1+\epsilon_0)\log n}{\mathcal{K}^{(N),\tau,+}_{\inf}(F_a,c^{\tau}_{\star})}+O(\sqrt{n})\right\} \tag*{(29)} ≤ ∣ δ ⋆ , n ∣ + n R a = 1 , a = a ⋆ ∑ K { K i n f ( N ) , τ , + ( F a , c ⋆ τ ) ( 1 + ϵ 0 ) log n + O ( n ) } (29)
with δ ⋆ , n = c ⋆ τ − c ⋆ , n τ \delta_{\star,n}=c^{\tau}_{\star}-c^{\tau}_{\star,n} δ ⋆ , n = c ⋆ τ − c ⋆ , n τ , and because lim n → ∞ c ⋆ , n τ = c ⋆ τ \lim_{n\to\infty}c^{\tau}_{\star,n}=c^{\tau}_{\star} lim n → ∞ c ⋆ , n τ = c ⋆ τ , we can conclude that
lim n → ∞ E [ c ^ n τ ] = c ⋆ τ . \lim_{n\to\infty}\mathbb{E}[\widehat{c}^{\tau}_{n}]=c^{\tau}_{\star}. n → ∞ lim E [ c n τ ] = c ⋆ τ .
Second, we prove that
∀ n ≥ 1 , ∀ 0 < ϵ ≤ n − 1 / 2 ( log n ) − 1 / 2 , ∃ c > 0 that P ( ∣ c ^ n τ − c ⋆ τ ∣ > ϵ ) ≤ c n − 1 ϵ − 2 . \forall n\ge1,\ \forall0<\epsilon\le n^{-1/2}(\log n)^{-1/2},\ \exists c>0\ \text{that}\ \mathbb{P}(|\widehat{c}^{\tau}_{n}-c^{\tau}_{\star}|>\epsilon)\le cn^{-1}\epsilon^{-2}. ∀ n ≥ 1 , ∀0 < ϵ ≤ n − 1/2 ( log n ) − 1/2 , ∃ c > 0 that P ( ∣ c n τ − c ⋆ τ ∣ > ϵ ) ≤ c n − 1 ϵ − 2 .
We observe that
∣ c ^ n τ − c ⋆ τ ∣ ≤ ∣ c ^ n τ − c ⋆ , n τ ∣ + ∣ c ⋆ τ − c ⋆ , n τ ∣ = ∣ c ^ n τ − c ⋆ , n τ ∣ + ∣ δ ⋆ , n ∣ (30) |\widehat{c}^{\tau}_{n}-c^{\tau}_{\star}|\le|\widehat{c}^{\tau}_{n}-c^{\tau}_{\star,n}|+|c^{\tau}_{\star}-c^{\tau}_{\star,n}|=|\widehat{c}^{\tau}_{n}-c^{\tau}_{\star,n}|+|\delta_{\star,n}| \tag*{(30)} ∣ c n τ − c ⋆ τ ∣ ≤ ∣ c n τ − c ⋆ , n τ ∣ + ∣ c ⋆ τ − c ⋆ , n τ ∣ = ∣ c n τ − c ⋆ , n τ ∣ + ∣ δ ⋆ , n ∣ (30)
⟹ P ( ∣ c ^ n τ − c ⋆ τ ∣ ≥ ϵ ) ≤ P ( ∣ c ^ n τ − c ⋆ , n τ ∣ ≥ ϵ / 2 ) + P ( ∣ δ ⋆ , n ∣ ≥ ϵ / 2 ) . (31) \Longrightarrow\mathbb{P}(|\widehat{c}^{\tau}_{n}-c^{\tau}_{\star}|\ge\epsilon)\le\mathbb{P}(|\widehat{c}^{\tau}_{n}-c^{\tau}_{\star,n}|\ge\epsilon/2)+\mathbb{P}(|\delta_{\star,n}|\ge\epsilon/2). \tag*{(31)} ⟹ P ( ∣ c n τ − c ⋆ τ ∣ ≥ ϵ ) ≤ P ( ∣ c n τ − c ⋆ , n τ ∣ ≥ ϵ /2 ) + P ( ∣ δ ⋆ , n ∣ ≥ ϵ /2 ) . (31)
Because lim n → n ∣ δ ⋆ , n ∣ = 0 \lim_{n\to n}|\delta_{\star,n}|=0 lim n → n ∣ δ ⋆ , n ∣ = 0 , therefore, ∃ N 0 > 0 \exists N_0>0 ∃ N 0 > 0 such that ∀ n ≥ N 0 \forall n\ge N_0 ∀ n ≥ N 0 , we have ∣ δ ⋆ , n ∣ < ϵ / 2 |\delta_{\star,n}|<\epsilon/2 ∣ δ ⋆ , n ∣ < ϵ /2 that means
∀ n > N 0 , P ( ∣ δ ⋆ , n ∣ ≥ ϵ / 2 ) = 0. \forall n>N_0,\ \mathbb{P}(|\delta_{\star,n}|\ge\epsilon/2)=0. ∀ n > N 0 , P ( ∣ δ ⋆ , n ∣ ≥ ϵ /2 ) = 0.
Next, according to Lemma C.14 ,
E [ ∣ c ^ n τ − c ⋆ , n τ ∣ ] ≤ R n ∑ a = 1 , a ≠ a ⋆ K { ( 1 + ϵ 0 ) log n K inf ( N ) , τ , + ( F a , c ⋆ τ ) + O ( n ) } = O ( n − 1 / 2 ) , (32) \mathbb{E}[|\widehat{c}^{\tau}_{n}-c^{\tau}_{\star,n}|]\le\frac{R}{n}\sum_{a=1,a\ne a_{\star}}^{K}\left\{\frac{(1+\epsilon_0)\log n}{\mathcal{K}^{(N),\tau,+}_{\inf}(F_a,c^{\tau}_{\star})}+O(\sqrt{n})\right\}=O(n^{-1/2}), \tag*{(32)} E [ ∣ c n τ − c ⋆ , n τ ∣ ] ≤ n R a = 1 , a = a ⋆ ∑ K { K i n f ( N ) , τ , + ( F a , c ⋆ τ ) ( 1 + ϵ 0 ) log n + O ( n ) } = O ( n − 1/2 ) , (32)
that leads to
P ( ∣ c ^ n τ − c ⋆ , n τ ∣ ≥ ϵ / 2 ) ≤ E [ ∣ c ^ n τ − c ⋆ , n τ ∣ ] ϵ / 2 = O ( n − 1 / 2 ) ϵ / 2 . (33) \mathbb{P}(|\widehat{c}^{\tau}_{n}-c^{\tau}_{\star,n}| \ge\epsilon/2) \le\frac{\mathbb{E}[|\widehat{c}^{\tau}_{n}-c^{\tau}_{\star,n}|]}{\epsilon/2} = \frac{O(n^{-1/2})}{\epsilon/2}. \tag*{(33)} P ( ∣ c n τ − c ⋆ , n τ ∣ ≥ ϵ /2 ) ≤ ϵ /2 E [ ∣ c n τ − c ⋆ , n τ ∣ ] = ϵ /2 O ( n − 1/2 ) . (33)
Therefore, ∃ c > 0 \exists c > 0 ∃ c > 0 , ∀ 0 < ϵ ≤ n − 1 / 2 \forall0 < \epsilon\le n^{-1/2} ∀0 < ϵ ≤ n − 1/2 such that
P ( ∣ c ^ n τ − c ⋆ , n τ ∣ ≥ ϵ / 2 ) ≤ c n − 1 / 2 ϵ ( ϵ − 2 ) ≤ c n − 1 / 2 ( n − 1 / 2 ) ϵ − 2 = c n − 1 ϵ − 2 , (34) \mathbb{P}(|\widehat{c}^{\tau}_{n}-c^{\tau}_{\star,n}| \ge\epsilon/2) \le cn^{-1/2}\epsilon(\epsilon^{-2}) \le cn^{-1/2}\left(n^{-1/2}\right)\epsilon^{-2} = cn^{-1}\epsilon^{-2}, \tag*{(34)} P ( ∣ c n τ − c ⋆ , n τ ∣ ≥ ϵ /2 ) ≤ c n − 1/2 ϵ ( ϵ − 2 ) ≤ c n − 1/2 ( n − 1/2 ) ϵ − 2 = c n − 1 ϵ − 2 , (34)
which means
∀ n ≥ N 0 , ∀ 0 < ϵ < n − 1 / 2 , ∃ c > 0 that P ( ∣ c ^ n τ − c ⋆ τ ∣ > ϵ ) ≤ c n − 1 ϵ − 2 . \forall n \ge N_0,\ \forall0 < \epsilon< n^{-1/2},\ \exists c > 0\ \text{that}\ \mathbb{P}(|\widehat{c}^{\tau}_{n}-c^{\tau}_{\star}| > \epsilon) \le cn^{-1}\epsilon^{-2}. ∀ n ≥ N 0 , ∀0 < ϵ < n − 1/2 , ∃ c > 0 that P ( ∣ c n τ − c ⋆ τ ∣ > ϵ ) ≤ c n − 1 ϵ − 2 .
With 0 < ϵ ≤ n − 1 / 2 ⇒ ϵ − 2 ≤ n − 1 0 < \epsilon\le n^{-1/2} \Rightarrow\epsilon^{-2} \le n^{-1} 0 < ϵ ≤ n − 1/2 ⇒ ϵ − 2 ≤ n − 1 , so that n − 1 ϵ − 2 ≤ n − 2 n^{-1}\epsilon^{-2} \le n^{-2} n − 1 ϵ − 2 ≤ n − 2 . With 1 ≤ n < N 0 ⇒ n − 1 ϵ − 2 ≤ N 0 − 2 1 \le n < N_0 \Rightarrow n^{-1}\epsilon^{-2} \le N_0^{-2} 1 ≤ n < N 0 ⇒ n − 1 ϵ − 2 ≤ N 0 − 2 . Therefore
∀ C > N 0 − 2 ⇒ P ( ∣ c ^ n τ − c ⋆ τ ∣ > ϵ ) ≤ 1 < C n − 1 ϵ − 2 , \forall C > N_0^{-2} \Rightarrow\mathbb{P}(|\widehat{c}^{\tau}_{n}-c^{\tau}_{\star}| > \epsilon) \le1 < Cn^{-1}\epsilon^{-2}, ∀ C > N 0 − 2 ⇒ P ( ∣ c n τ − c ⋆ τ ∣ > ϵ ) ≤ 1 < C n − 1 ϵ − 2 ,
which means
∀ n ≥ 1 , ∀ 0 < ϵ ≤ n − 1 / 2 , ∃ C > 0 that P ( ∣ c ^ n τ − c ⋆ τ ∣ > ϵ ) ≤ C n − 1 ϵ − 2 . \forall n \ge1,\ \forall0 < \epsilon\le n^{-1/2},\ \exists C > 0\ \text{that}\ \mathbb{P}(|\widehat{c}^{\tau}_{n}-c^{\tau}_{\star}| > \epsilon) \le Cn^{-1}\epsilon^{-2}. ∀ n ≥ 1 , ∀0 < ϵ ≤ n − 1/2 , ∃ C > 0 that P ( ∣ c n τ − c ⋆ τ ∣ > ϵ ) ≤ C n − 1 ϵ − 2 .
Next, we prove the following claim
Claim.
P ( ∣ c ^ n τ − c ⋆ τ ∣ > ϵ ) ≤ C n − 1 ϵ − 2 , ∀ ϵ > n − 1 / 2 , \mathbb{P}(|\widehat{c}^{\tau}_{n}-c^{\tau}_{\star}| > \epsilon) \le Cn^{-1}\epsilon^{-2}, \qquad\forall\epsilon> n^{-1/2}, P ( ∣ c n τ − c ⋆ τ ∣ > ϵ ) ≤ C n − 1 ϵ − 2 , ∀ ϵ > n − 1/2 , for a universal constant C > 0 C > 0 C > 0 depending only on problem parameters (K K K , R R R , etc.).
By Lemma C.14 we have
∣ c ^ n τ − c ⋆ τ ∣ ≤ R ∑ a ≠ a ⋆ T a ( n ) n = : A n . |\widehat{c}^{\tau}_{n}-c^{\tau}_{\star}| \le R\sum_{a\ne a_{\star}}\frac{T_a(n)}{n}=:A_n. ∣ c n τ − c ⋆ τ ∣ ≤ R a = a ⋆ ∑ n T a ( n ) =: A n .
Let S n = ∑ a ≠ a ⋆ T a ( n ) S_n=\sum_{a\ne a_{\star}}T_a(n) S n = ∑ a = a ⋆ T a ( n ) . Lemma C.7 implies that ∃ c 1 > 0 \exists c_1 > 0 ∃ c 1 > 0 such that E [ S n ] ≤ c 1 ( K − 1 ) n \mathbb{E}[S_n] \le c_1(K-1)\sqrt{n} E [ S n ] ≤ c 1 ( K − 1 ) n . By Azuma–Hoeffding,
P ( S n − E [ S n ] ≥ t ) ≤ exp ( − t 2 2 n ) . \mathbb{P}(S_n-\mathbb{E}[S_n]\ge t)\le\exp\left(-\frac{t^2}{2n}\right). P ( S n − E [ S n ] ≥ t ) ≤ exp ( − 2 n t 2 ) .
Choose t = ϵ n 2 R − c 1 ( K − 1 ) n t=\frac{\epsilon n}{2R}-c_1(K-1)\sqrt{n} t = 2 R ϵ n − c 1 ( K − 1 ) n (valid since ϵ > n − 1 / 2 \epsilon>n^{-1/2} ϵ > n − 1/2 ), then
P ( S n > ϵ n 2 R ) ≤ exp ( − c 3 n ϵ 2 ) ≤ 1 2 n − 1 ϵ − 2 . \mathbb{P}\left(S_n>\frac{\epsilon n}{2R}\right)\le\exp(-c_3n\epsilon^2)\le\frac{1}{2}n^{-1}\epsilon^{-2}. P ( S n > 2 R ϵ n ) ≤ exp ( − c 3 n ϵ 2 ) ≤ 2 1 n − 1 ϵ − 2 .
Therefore,
P ( ∣ c ^ n τ − c ⋆ τ ∣ > ϵ ) ≤ P ( A n > ϵ ) ≤ C n − 1 ϵ − 2 , \mathbb{P}(|\widehat{c}^{\tau}_{n}-c^{\tau}_{\star}|>\epsilon)\le\mathbb{P}(A_n>\epsilon)\le Cn^{-1}\epsilon^{-2}, P ( ∣ c n τ − c ⋆ τ ∣ > ϵ ) ≤ P ( A n > ϵ ) ≤ C n − 1 ϵ − 2 ,
which completes the proof. □
Proof Of Theorem 5.4 for PATSO case
Proof. The proof follows the same steps as Theorem 5.4 for CATSO case. We first prove that lim n → ∞ E [ c ^ n τ ] = c ⋆ τ \lim_{n\to\infty}\mathbb{E}[\widehat{c}^{\tau}_{n}]=c^{\tau}_{\star} lim n → ∞ E [ c n τ ] = c ⋆ τ . According to the result of Lemma C.15 , we have
E [ ∣ c ^ n τ − c ⋆ τ ∣ ] ≤ ∣ δ ⋆ , n ∣ + R ∑ a = 1 , a ≠ a ⋆ K E [ T a ( n ) ] n (35) \mathbb{E}[|\widehat{c}^{\tau}_{n}-c^{\tau}_{\star}|]\le|\delta_{\star,n}|+R\sum_{a=1,a\ne a_{\star}}^{K}\frac{\mathbb{E}[T_a(n)]}{n} \tag*{(35)} E [ ∣ c n τ − c ⋆ τ ∣ ] ≤ ∣ δ ⋆ , n ∣ + R a = 1 , a = a ⋆ ∑ K n E [ T a ( n )] (35)
≤ ∣ δ ⋆ , n ∣ + R n ∑ a = 1 , a ≠ a ⋆ K { ( 1 + ϵ 0 ) log n K inf τ , + ( F a , c ⋆ τ ) + o ( n ) + O ( 1 ) } (36) \le|\delta_{\star,n}|+\frac{R}{n}\sum_{a=1,a\ne a_{\star}}^{K}\left\{\frac{(1+\epsilon_0)\log n}{K^{\tau,+}_{\inf}(F_a,c^{\tau}_{\star})}+o(\sqrt{n})+O(1)\right\} \tag*{(36)} ≤ ∣ δ ⋆ , n ∣ + n R a = 1 , a = a ⋆ ∑ K { K i n f τ , + ( F a , c ⋆ τ ) ( 1 + ϵ 0 ) log n + o ( n ) + O ( 1 ) } (36)
with δ ⋆ , n = c ⋆ τ − c ⋆ , n τ \delta_{\star,n}=c^{\tau}_{\star}-c^{\tau}_{\star,n} δ ⋆ , n = c ⋆ τ − c ⋆ , n τ , and because lim n → ∞ c ⋆ , n τ = c ⋆ τ \lim_{n\to\infty}c^{\tau}_{\star,n}=c^{\tau}_{\star} lim n → ∞ c ⋆ , n τ = c ⋆ τ , we can conclude that
lim n → ∞ E [ c ^ n τ ] = c ⋆ τ . \lim_{n\to\infty}\mathbb{E}[\widehat{c}^{\tau}_{n}]=c^{\tau}_{\star}. n → ∞ lim E [ c n τ ] = c ⋆ τ .
Second, we prove that
∀ n ≥ 1 , ∀ 0 < ϵ < n − 1 / 2 , ∃ c > 0 that P ( ∣ C ~ n τ − C ⋆ τ ∣ > ϵ ) ≤ c n − 1 ϵ − 2 . \forall n \ge1,\ \forall0 < \epsilon< n^{-1/2},\ \exists c > 0\ \text{that}\ \mathbb{P}\left(\left|\tilde{C}_n^\tau- C_\star^\tau\right| > \epsilon\right) \le cn^{-1}\epsilon^{-2}. ∀ n ≥ 1 , ∀0 < ϵ < n − 1/2 , ∃ c > 0 that P ( C ~ n τ − C ⋆ τ > ϵ ) ≤ c n − 1 ϵ − 2 .
We observe that
∣ C ~ n τ − C ⋆ τ ∣ ≤ ∣ C ~ n τ − C ⋆ , n τ ∣ + ∣ C ⋆ τ − C ⋆ , n τ ∣ = ∣ C ~ n τ − C ⋆ , n τ ∣ + ∣ δ ⋆ , n ∣ (37) \left|\tilde{C}_n^\tau- C_\star^\tau\right| \le\left|\tilde{C}_n^\tau- C_{\star,n}^\tau\right| + \left|C_{\star}^\tau- C_{\star,n}^\tau\right| = \left|\tilde{C}_n^\tau- C_{\star,n}^\tau\right| + |\delta_{\star,n}| \tag*{(37)} C ~ n τ − C ⋆ τ ≤ C ~ n τ − C ⋆ , n τ + C ⋆ τ − C ⋆ , n τ = C ~ n τ − C ⋆ , n τ + ∣ δ ⋆ , n ∣ (37)
⟹ P ( ∣ C ~ n τ − C ⋆ τ ∣ ≥ ϵ ) ≤ P ( ∣ C ~ n τ − C ⋆ , n τ ∣ ≥ ϵ / 2 ) + P ( ∣ δ ⋆ , n ∣ ≥ ϵ / 2 ) . (38) \Longrightarrow\mathbb{P}\left(\left|\tilde{C}_n^\tau- C_\star^\tau\right| \ge\epsilon\right) \le\mathbb{P}\left(\left|\tilde{C}_n^\tau- C_{\star,n}^\tau\right| \ge\epsilon/2\right) + \mathbb{P}\left(|\delta_{\star,n}| \ge\epsilon/2\right). \tag*{(38)} ⟹ P ( C ~ n τ − C ⋆ τ ≥ ϵ ) ≤ P ( C ~ n τ − C ⋆ , n τ ≥ ϵ /2 ) + P ( ∣ δ ⋆ , n ∣ ≥ ϵ /2 ) . (38)
Because lim n → ∞ ∣ δ ⋆ , n ∣ = 0 \lim_{n \to\infty} |\delta_{\star,n}| = 0 lim n → ∞ ∣ δ ⋆ , n ∣ = 0 , therefore, ∃ N 0 > 0 \exists N_0 > 0 ∃ N 0 > 0 such that ∀ n ≥ N 0 \forall n \ge N_0 ∀ n ≥ N 0 , we have ∣ δ ⋆ , n ∣ < ϵ / 2 |\delta_{\star,n}| < \epsilon/2 ∣ δ ⋆ , n ∣ < ϵ /2 that means
∀ n > N 0 , P ( ∣ δ ⋆ , n ∣ ≥ ϵ / 2 ) = 0. \forall n > N_0,\ \mathbb{P}\left(|\delta_{\star,n}| \ge\epsilon/2\right) = 0. ∀ n > N 0 , P ( ∣ δ ⋆ , n ∣ ≥ ϵ /2 ) = 0.
Next, according to Lemma C.15,
E [ ∣ C ~ n τ − C ⋆ , n τ ∣ ] ≤ R n ∑ a = 1 , a ≠ a ⋆ K { ( 1 + ϵ 0 ) log n K inf τ , + ( F a , C ⋆ τ ) + o ( n ) + O ( 1 ) } = O ( n − 1 / 2 ) , (39) \mathbb{E}\left[|\tilde{C}_n^\tau- C_{\star,n}^\tau|\right] \le\frac{R}{n}\sum_{a=1,a\ne a_\star}^{K}\left\{\frac{(1+\epsilon_0)\log n}{\mathcal{K}_{\inf}^{\tau,+}(F_a,C_\star^\tau)} + o(\sqrt{n}) + O(1)\right\} = O(n^{-1/2}), \tag*{(39)} E [ ∣ C ~ n τ − C ⋆ , n τ ∣ ] ≤ n R a = 1 , a = a ⋆ ∑ K { K i n f τ , + ( F a , C ⋆ τ ) ( 1 + ϵ 0 ) log n + o ( n ) + O ( 1 ) } = O ( n − 1/2 ) , (39)
that leads to
P ( ∣ C ~ n τ − C ⋆ , n τ ∣ ≥ ϵ / 2 ) ≤ E [ ∣ C ~ n τ − C ⋆ , n τ ∣ ] ϵ / 2 = O ( n − 1 / 2 ) ϵ / 2 . (40) \mathbb{P}\left(|\tilde{C}_n^\tau- C_{\star,n}^\tau| \ge\epsilon/2\right) \le\frac{\mathbb{E}\left[|\tilde{C}_n^\tau- C_{\star,n}^\tau|\right]}{\epsilon/2} = \frac{O(n^{-1/2})}{\epsilon/2}. \tag*{(40)} P ( ∣ C ~ n τ − C ⋆ , n τ ∣ ≥ ϵ /2 ) ≤ ϵ /2 E [ ∣ C ~ n τ − C ⋆ , n τ ∣ ] = ϵ /2 O ( n − 1/2 ) . (40)
Therefore, ∃ c > 0 \exists c > 0 ∃ c > 0 , ∀ 0 < ϵ < n − 1 / 2 \forall0 < \epsilon< n^{-1/2} ∀0 < ϵ < n − 1/2 such that
P ( ∣ C ~ n τ − C ⋆ , n τ ∣ ≥ ϵ / 2 ) ≤ c n − 1 / 2 ( ϵ ) ( ϵ − 2 ) ≤ c n − 1 / 2 ( n − 1 / 2 ) ϵ − 2 = c n − 1 ϵ − 2 , (41) \mathbb{P}\left(|\tilde{C}_n^\tau- C_{\star,n}^\tau| \ge\epsilon/2\right) \le cn^{-1/2}(\epsilon)(\epsilon^{-2}) \le cn^{-1/2}\left(n^{-1/2}\right)\epsilon^{-2} = cn^{-1}\epsilon^{-2}, \tag*{(41)} P ( ∣ C ~ n τ − C ⋆ , n τ ∣ ≥ ϵ /2 ) ≤ c n − 1/2 ( ϵ ) ( ϵ − 2 ) ≤ c n − 1/2 ( n − 1/2 ) ϵ − 2 = c n − 1 ϵ − 2 , (41)
which means
∀ n ≥ N 0 , ∀ 0 < ϵ < n − 1 / 2 , ∃ c > 0 that P ( ∣ C ~ n τ − C ⋆ τ ∣ > ϵ ) ≤ c n − 1 ϵ − 2 . \forall n \ge N_0,\ \forall0 < \epsilon< n^{-1/2},\ \exists c > 0\ \text{that}\ \mathbb{P}\left(|\tilde{C}_n^\tau- C_\star^\tau| > \epsilon\right) \le cn^{-1}\epsilon^{-2}. ∀ n ≥ N 0 , ∀0 < ϵ < n − 1/2 , ∃ c > 0 that P ( ∣ C ~ n τ − C ⋆ τ ∣ > ϵ ) ≤ c n − 1 ϵ − 2 .
With 0 < ϵ < n − 1 / 2 ⇒ ϵ − 2 > n − 1 0 < \epsilon< n^{-1/2} \Rightarrow\epsilon^{-2} > n^{-1} 0 < ϵ < n − 1/2 ⇒ ϵ − 2 > n − 1 , so that n − 1 ϵ − 2 > n − 2 n^{-1}\epsilon^{-2} > n^{-2} n − 1 ϵ − 2 > n − 2 . With 1 ≤ n < N 0 ⇒ n − 1 ϵ − 2 > N 0 − 2 1 \le n < N_0 \Rightarrow n^{-1}\epsilon^{-2} > N_0^{-2} 1 ≤ n < N 0 ⇒ n − 1 ϵ − 2 > N 0 − 2 . Therefore
∀ C > N 0 − 2 ⇒ P ( ∣ C ~ n τ − C ⋆ τ ∣ > ϵ ) ≤ 1 < C n − 1 ϵ − 2 , \forall C > N_0^{-2} \Rightarrow\mathbb{P}\left(|\tilde{C}_n^\tau- C_\star^\tau| > \epsilon\right) \le1 < Cn^{-1}\epsilon^{-2}, ∀ C > N 0 − 2 ⇒ P ( ∣ C ~ n τ − C ⋆ τ ∣ > ϵ ) ≤ 1 < C n − 1 ϵ − 2 ,
which means
∀ n ≥ 1 , ∀ 0 < ϵ < n − 1 / 2 , ∃ C > 0 that P ( ∣ C ~ n τ − C ⋆ τ ∣ > ϵ ) ≤ C n − 1 ϵ − 2 . \forall n \ge1,\ \forall0 < \epsilon< n^{-1/2},\ \exists C > 0\ \text{that}\ \mathbb{P}\left(|\tilde{C}_n^\tau- C_\star^\tau| > \epsilon\right) \le Cn^{-1}\epsilon^{-2}. ∀ n ≥ 1 , ∀0 < ϵ < n − 1/2 , ∃ C > 0 that P ( ∣ C ~ n τ − C ⋆ τ ∣ > ϵ ) ≤ C n − 1 ϵ − 2 .
Similarly to CATS0, we prove the following claim
Claim. P ( ∣ C ~ n τ − C ⋆ τ ∣ > ϵ ) ≤ C n − 1 ϵ − 2 , ∀ ϵ > n − 1 / 2 , \mathbb{P}\left(|\tilde{C}_n^\tau- C_\star^\tau| > \epsilon\right) \le Cn^{-1}\epsilon^{-2},\qquad\forall\epsilon> n^{-1/2}, P ( ∣ C ~ n τ − C ⋆ τ ∣ > ϵ ) ≤ C n − 1 ϵ − 2 , ∀ ϵ > n − 1/2 ,
for a universal constant C > 0 C > 0 C > 0 depending only on problem parameters ( K , R , etc. ) (K,R,\text{etc.}) ( K , R , etc. ) .
Setup & decomposition. By Lemma C.15 we have
∣ C ~ n τ − C ⋆ τ ∣ ≤ R ∑ a ≠ a ⋆ K T a ( n ) n ≔ A n . |\tilde{C}_n^\tau- C_\star^\tau| \le R\sum_{a\ne a_\star}^{K}\frac{T_a(n)}{n} \coloneqq A_n. ∣ C ~ n τ − C ⋆ τ ∣ ≤ R a = a ⋆ ∑ K n T a ( n ) : = A n .
Let S n = ∑ a ≠ a ⋆ K T a ( n ) S_n = \sum_{a\ne a_\star}^{K} T_a(n) S n = ∑ a = a ⋆ K T a ( n ) . Lemma C.12 implies ∃ c 1 > 0 \exists c_1 > 0 ∃ c 1 > 0 such that E [ S n ] ≤ c 1 ( K − 1 ) n \mathbb{E}[S_n] \le c_1(K-1)\sqrt{n} E [ S n ] ≤ c 1 ( K − 1 ) n . By Azuma–Hoeffding,
P ( S n − E [ S n ] ≥ t ) ≤ exp ( − t 2 2 n ) . \mathbb{P}(S_n-\mathbb{E}[S_n]\ge t)\le\exp\left(-\frac{t^2}{2n}\right). P ( S n − E [ S n ] ≥ t ) ≤ exp ( − 2 n t 2 ) .
Choose t = ϵ n 2 R − c 1 ( K − 1 ) n t = \frac{\epsilon n}{2R}-c_1(K-1)\sqrt{n} t = 2 R ϵ n − c 1 ( K − 1 ) n (valid since ϵ > n − 1 / 2 \epsilon> n^{-1/2} ϵ > n − 1/2 ), then
= P ( S n > ϵ n 2 R ) ≤ exp ( − c 3 n ϵ 2 ) ≤ 1 2 n − 1 ϵ − 2 . =\mathbb{P}\left(S_n > \frac{\epsilon n}{2R}\right) \le\exp(-c_3n\epsilon^2) \le\frac{1}{2}n^{-1}\epsilon^{-2}. = P ( S n > 2 R ϵ n ) ≤ exp ( − c 3 n ϵ 2 ) ≤ 2 1 n − 1 ϵ − 2 .
Therefore,
P ( ∣ C ~ n τ − C ⋆ τ ∣ > ϵ ) ≤ P ( A n > ϵ ) ≤ C n − 1 ϵ − 2 , \mathbb{P}\left(|\tilde{C}_n^\tau-C_\star^\tau|>\epsilon\right)\le\mathbb{P}(A_n>\epsilon)\le Cn^{-1}\epsilon^{-2}, P ( ∣ C ~ n τ − C ⋆ τ ∣ > ϵ ) ≤ P ( A n > ϵ ) ≤ C n − 1 ϵ − 2 ,
which completes the proof. ∎
MCTS Convergence Proofs
One-Step CVaR Lifting for Q-Edges
Proof of Lemma 5.5
Proof. Let
R h : = R max 1 − γ . R_h := \frac{R_{\max}}{1-\gamma}. R h := 1 − γ R m a x .
All plug-in and oracle backed-up samples lie in [ 0 , R h ] [0, R_h] [ 0 , R h ] .
Define the oracle empirical law
D ^ n o r ( s h , a ) : = 1 n ∑ i = 1 n δ G i . \widehat{\mathcal{D}}^{\mathrm{or}}_n(s_h,a) := \frac{1}{n}\sum_{i=1}^{n}\delta_{G_i}. D n or ( s h , a ) := n 1 i = 1 ∑ n δ G i .
The samples G i = X i + γ V τ ( S i ) G_i = X_i + \gamma V^\tau(S_i) G i = X i + γ V τ ( S i ) are i.i.d. with law D τ ( s h , a ) \mathcal{D}^\tau(s_h,a) D τ ( s h , a ) .
By the W 1 W_1 W 1 -Lipschitz property of CVaR,
∣ Q ^ n τ ( s h , a ) − Q τ ( s h , a ) ∣ ≤ 1 τ W 1 ( D ^ n ( s h , a ) , D τ ( s h , a ) ) . \left|\widehat{Q}^{\tau}_n(s_h,a)-Q^\tau(s_h,a)\right| \le\frac{1}{\tau}W_1\left(\widehat{\mathcal{D}}_n(s_h,a),\mathcal{D}^\tau(s_h,a)\right). Q n τ ( s h , a ) − Q τ ( s h , a ) ≤ τ 1 W 1 ( D n ( s h , a ) , D τ ( s h , a ) ) .
Using the triangle inequality,
W 1 ( D ^ n , D τ ) ≤ W 1 ( D ^ n , D ^ n o r ) + W 1 ( D ^ n o r , D τ ) . W_1\left(\widehat{\mathcal{D}}_n,\mathcal{D}^\tau\right) \le W_1\left(\widehat{\mathcal{D}}_n,\widehat{\mathcal{D}}^{\mathrm{or}}_n\right) + W_1\left(\widehat{\mathcal{D}}^{\mathrm{or}}_n,\mathcal{D}^\tau\right). W 1 ( D n , D τ ) ≤ W 1 ( D n , D n or ) + W 1 ( D n or , D τ ) .
We first control the oracle empirical-measure term. Since all laws are supported on [ 0 , R h ] [0,R_h] [ 0 , R h ] ,
W 1 ( P , Q ) ≤ R h ∥ F P − F Q ∥ ∞ . W_1(P,Q) \le R_h\lVert F_P-F_Q\rVert_\infty. W 1 ( P , Q ) ≤ R h ∥ F P − F Q ∥ ∞ .
Therefore, by the Dvoretzky–Kiefer–Wolfowitz inequality,
P ( W 1 ( D ^ n o r , D τ ) > τ ε 2 ) ≤ P ( ∥ F D ^ n o r − F D τ ∥ ∞ > τ ε 2 R h ) ≤ 2 exp ( − n τ 2 ε 2 2 R h 2 ) . \mathbb{P}\left(W_1\left(\widehat{\mathcal{D}}^{\mathrm{or}}_n,\mathcal{D}^\tau\right)>\frac{\tau\varepsilon}{2}\right)
\le\mathbb{P}\left(\lVert F_{\widehat{\mathcal{D}}^{\mathrm{or}}_n}-F_{\mathcal{D}^\tau}\rVert_\infty>\frac{\tau\varepsilon}{2R_h}\right)
\le2\exp\left(-\frac{n\tau^2\varepsilon^2}{2R_h^2}\right). P ( W 1 ( D n or , D τ ) > 2 τ ε ) ≤ P ( ∥ F D n or − F D τ ∥ ∞ > 2 R h τ ε ) ≤ 2 exp ( − 2 R h 2 n τ 2 ε 2 ) .
Since 2 e − x ≤ 2 / ( e x ) 2e^{-x}\le2/(ex) 2 e − x ≤ 2/ ( e x ) for x > 0 x>0 x > 0 , this term is bounded by
4 R h 2 e τ 2 1 n ε 2 . \frac{4R_h^2}{e\tau^2}\frac{1}{n\varepsilon^2}. e τ 2 4 R h 2 n ε 2 1 .
Next control the plug-in term. Couple each plug-in sample q ˉ i ( n ) \bar{q}^{(n)}_i q ˉ i ( n ) with the corresponding oracle sample G i G_i G i . Then
W 1 ( D ^ n , D ^ n o r ) ≤ 1 n ∑ i = 1 n ∣ q ˉ i ( n ) − G i ∣ = γ n ∑ i = 1 n ∣ V ^ N S i τ ( S i ) − V τ ( S i ) ∣ = γ ∑ m = 1 M N m n n ∣ V ^ N m n τ ( s h + 1 ( m ) ) − V τ ( s h + 1 ( m ) ) ∣ ≤ γ max 1 ≤ m ≤ M ∣ V ^ N m n τ ( s h + 1 ( m ) ) − V τ ( s h + 1 ( m ) ) ∣ . \begin{aligned}
W_1\left(\widehat{\mathcal{D}}_n,\widehat{\mathcal{D}}^{\mathrm{or}}_n\right)
\le\frac{1}{n}\sum_{i=1}^{n}\left|\bar{q}^{(n)}_i-G_i\right| \\
&= \frac{\gamma}{n}\sum_{i=1}^{n}\left|\widehat{V}^{\tau}_{N_{S_i}}(S_i)-V^\tau(S_i)\right| \\
&= \gamma\sum_{m=1}^{M}\frac{N^n_m}{n}\left|\widehat{V}^{\tau}_{N^n_m}(s^{(m)}_{h+1})-V^\tau(s^{(m)}_{h+1})\right| \\
&\le\gamma\max_{1\le m\le M}\left|\widehat{V}^{\tau}_{N^n_m}(s^{(m)}_{h+1})-V^\tau(s^{(m)}_{h+1})\right|.
\end{aligned} W 1 ( D n , D n or ) ≤ n 1 i = 1 ∑ n q ˉ i ( n ) − G i = n γ i = 1 ∑ n V N S i τ ( S i ) − V τ ( S i ) = γ m = 1 ∑ M n N m n V N m n τ ( s h + 1 ( m ) ) − V τ ( s h + 1 ( m ) ) ≤ γ 1 ≤ m ≤ M max V N m n τ ( s h + 1 ( m ) ) − V τ ( s h + 1 ( m ) ) .
If γ = 0 \gamma=0 γ = 0 , this term is identically zero. Assume below that γ > 0 \gamma>0 γ > 0 .
Let
E n : = { N m n ≥ n p m 2 for all m = 1 , … , M } . \mathcal{E}_n := \left\{N^n_m \ge\frac{np_m}{2}\text{ for all }m=1,\ldots,M\right\}. E n := { N m n ≥ 2 n p m for all m = 1 , … , M } .
By a multiplicative Chernoff bound,
P ( E n c ) ≤ ∑ m = 1 M P ( N m n < n p m 2 ) ≤ M exp ( − n p min 8 ) . \mathbb{P}(\mathcal{E}_n^c) \le\sum_{m=1}^{M} \mathbb{P}\left(N_m^n < \frac{np_m}{2}\right) \le M\exp\left(-\frac{np_{\min}}{8}\right). P ( E n c ) ≤ m = 1 ∑ M P ( N m n < 2 n p m ) ≤ M exp ( − 8 n p m i n ) .
For 0 < ε ≤ R h 0 < \varepsilon\le R_h 0 < ε ≤ R h , this is bounded by
8 M R h 2 ε p min 1 n ε 2 . \frac{8MR_h^2}{\varepsilon p_{\min}}\frac{1}{n\varepsilon^2}. ε p m i n 8 M R h 2 n ε 2 1 .
For ε > R h \varepsilon> R_h ε > R h , the target probability is zero because both CVaR values lie in [ 0 , R h ] [0,R_h] [ 0 , R h ] .
On E n \mathcal{E}_n E n , every child count satisfies
N m n ≥ n p m 2 . N_m^n \ge\frac{np_m}{2}. N m n ≥ 2 n p m .
Thus, using the stopped-count concentration assumption and a union bound,
P ( W 1 ( D ^ n , D ^ n o r ) > τ ε 2 , E n ) ≤ ∑ m = 1 M P ( ∣ V ^ N m n ( s h + 1 ( m ) ) − V τ ( s h + 1 ( m ) ) ∣ > τ ε 2 γ , E n ) ≤ ∑ m = 1 M 2 c m n p m ( 2 γ τ ε ) 2 = 8 γ 2 τ 2 ( ∑ m = 1 M c m p m ) 1 n ε 2 . \begin{aligned}
\mathbb{P}\left(W_1\left(\widehat{\mathcal{D}}_n,\widehat{\mathcal{D}}_n^{\mathrm{or}}\right)>\frac{\tau\varepsilon}{2},\mathcal{E}_n\right) \\
&\le\sum_{m=1}^{M}\mathbb{P}\left(\left|\widehat{V}_{N_m^n}(s_{h+1}^{(m)})-V^\tau(s_{h+1}^{(m)})\right|>\frac{\tau\varepsilon}{2\gamma},\mathcal{E}_n\right) \\
&\le\sum_{m=1}^{M}\frac{2c_m}{np_m}\left(\frac{2\gamma}{\tau\varepsilon}\right)^2 \\
&= \frac{8\gamma^2}{\tau^2}\left(\sum_{m=1}^{M}\frac{c_m}{p_m}\right)\frac{1}{n\varepsilon^2}.
\end{aligned} P ( W 1 ( D n , D n or ) > 2 τ ε , E n ) ≤ m = 1 ∑ M P ( V N m n ( s h + 1 ( m ) ) − V τ ( s h + 1 ( m ) ) > 2 γ τ ε , E n ) ≤ m = 1 ∑ M n p m 2 c m ( τ ε 2 γ ) 2 = τ 2 8 γ 2 ( m = 1 ∑ M p m c m ) n ε 2 1 .
Combining the oracle empirical-measure term, the plug-in term, and the count event gives a constant c Q > 0 c_Q>0 c Q > 0 such that
P ( ∣ Q ^ n τ ( s h , a ) − Q τ ( s h , a ) ∣ > ε ) ≤ c Q n ε 2 \mathbb{P}\left(\left|\widehat{Q}_n^\tau(s_h,a)-Q^\tau(s_h,a)\right|>\varepsilon\right)\le\frac{c_Q}{n\varepsilon^2} P ( Q n τ ( s h , a ) − Q τ ( s h , a ) > ε ) ≤ n ε 2 c Q
for every n ≥ 1 n\ge1 n ≥ 1 and every ε > 0 \varepsilon>0 ε > 0 .
It remains to prove convergence of expectations. Since both Q ^ n τ ( s h , a ) \widehat{Q}_n^\tau(s_h,a) Q n τ ( s h , a ) and Q τ ( s h , a ) Q^\tau(s_h,a) Q τ ( s h , a ) lie in [ 0 , R h ] [0,R_h] [ 0 , R h ] , the displayed tail bound implies convergence in probability, and boundedness then implies convergence in L 1 L^1 L 1 . Hence
lim n → ∞ E [ ∣ Q ^ n τ ( s h , a ) − Q τ ( s h , a ) ∣ ] = 0. \lim_{n\to\infty}\mathbb{E}\left[\left|\widehat{Q}_n^\tau(s_h,a)-Q^\tau(s_h,a)\right|\right]=0. n → ∞ lim E [ Q n τ ( s h , a ) − Q τ ( s h , a ) ] = 0.
Therefore,
lim n → ∞ E [ Q ^ n τ ( s h , a ) ] = Q τ ( s h , a ) . \lim_{n\to\infty}\mathbb{E}\left[\widehat{Q}_n^\tau(s_h,a)\right]=Q^\tau(s_h,a). n → ∞ lim E [ Q n τ ( s h , a ) ] = Q τ ( s h , a ) .
Together with the concentration inequality, this is exactly
plim n → ∞ Q ^ n τ ( s h , a ) = Q τ ( s h , a ) , \underset{n\to\infty}{\operatorname{plim}}\widehat{Q}_n^\tau(s_h,a)=Q^\tau(s_h,a), n → ∞ plim Q n τ ( s h , a ) = Q τ ( s h , a ) ,
this concludes the proof.
Convergence of CATSO in MCTS
Theorem D.1 (Finite-depth CVaR convergence of CATSO). Let CATSO run on a finite tree of depth H H H with bounded rewards and finite transition support at every Q Q Q -edge. Let
Q ^ n τ ( s h , a ) : = CVaR τ ( D ^ n ( s h , a ) ) \widehat{Q}_n^\tau(s_h,a):=\operatorname{CVaR}_{\tau}\left(\widehat{\mathcal{D}}_n(s_h,a)\right) Q n τ ( s h , a ) := CVaR τ ( D n ( s h , a ) ) be the empirical CVaR Q Q Q -edge estimator after n n n local visits to edge ( s h , a ) (s_h,a) ( s h , a ) . Then, for every node s h s_h s h and every action a ∈ A ( s h ) a\in\mathcal{A}(s_h) a ∈ A ( s h ) ,
plim n → ∞ Q ^ n τ ( s h , a ) = Q ~ τ ( s h , a ) , \underset{n\to\infty}{\operatorname{plim}}\widehat{Q}_n^\tau(s_h,a)=\widetilde{Q}^\tau(s_h,a), n → ∞ plim Q n τ ( s h , a ) = Q τ ( s h , a ) , and
plim n → ∞ V ^ n τ ( s h ) = V ~ τ ( s h ) . \underset{n\to\infty}{\operatorname{plim}}\widehat{V}_n^\tau(s_h)=\widetilde{V}^\tau(s_h). n → ∞ plim V n τ ( s h ) = V τ ( s h ) .
Proof. We prove the result by backward induction on the remaining depth d : = H − h d:=H-h d := H − h . Base case. At terminal depth H H H , the rollout or leaf estimator is assumed to satisfy
plim n → ∞ V ^ n τ ( s h ) = V ~ τ ( s h ) . \underset{n\to\infty}{\operatorname{plim}}\widehat{V}^{\tau}_{n}(s_h)=\widetilde{V}^{\tau}(s_h). n → ∞ plim V n τ ( s h ) = V τ ( s h ) .
Thus the claim holds at depth H H H .
Induction hypothesis. Assume that for every child state s h + 1 s_{h+1} s h + 1 at depth h + 1 h+1 h + 1 ,
plim n → ∞ V ^ n τ ( s h + 1 ) = V ~ τ ( s h + 1 ) . \underset{n\to\infty}{\operatorname{plim}}\widehat{V}^{\tau}_{n}(s_{h+1})=\widetilde{V}^{\tau}(s_{h+1}). n → ∞ plim V n τ ( s h + 1 ) = V τ ( s h + 1 ) .
We prove the two claims for a node s h s_h s h at depth h h h .
I: Q-edge CVaR convergence. Fix an action a ∈ A ( s h ) a \in\mathcal{A}(s_h) a ∈ A ( s h ) . On the i i i -th local visit to edge ( s h , a ) (s_h,a) ( s h , a ) , write the sampled reward and next state as
X i ∼ R ( s h , a , S i ) , S i ∼ P ( ⋅ ∣ s h , a ) . X_i \sim\mathcal{R}(s_h,a,S_i), \qquad S_i \sim P(\cdot\mid s_h,a). X i ∼ R ( s h , a , S i ) , S i ∼ P ( ⋅ ∣ s h , a ) .
The plug-in backed-up sample stored at the Q-edge is
Q ˉ i ( n ) = X i + γ V ^ N S i n τ ( S i ) , \bar{Q}^{(n)}_i=X_i+\gamma\widehat{V}^{\tau}_{N_{S_i}^{n}}(S_i), Q ˉ i ( n ) = X i + γ V N S i n τ ( S i ) ,
where N S i n N_{S_i}^{n} N S i n is the number of times child state S i S_i S i appears among the first n n n local visits to edge ( s h , a ) (s_h,a) ( s h , a ) . The corresponding oracle backed-up sample is
G i = X i + γ V ~ τ ( S i ) . G_i=X_i+\gamma\widetilde{V}^{\tau}(S_i). G i = X i + γ V τ ( S i ) .
Therefore the population Q-edge law is
D τ ( s h , a ) : = L ( G i ) , \mathcal{D}^{\tau}(s_h,a):=\mathcal{L}(G_i), D τ ( s h , a ) := L ( G i ) ,
and the population CVaR Q-value is
Q ~ τ ( s h , a ) = CVaR τ ( D τ ( s h , a ) ) . \widetilde{Q}^{\tau}(s_h,a)=\operatorname{CVaR}_{\tau}\left(\mathcal{D}^{\tau}(s_h,a)\right). Q τ ( s h , a ) = CVaR τ ( D τ ( s h , a ) ) .
By the induction hypothesis, every child estimator V ^ n τ ( s h + 1 ) \widehat{V}^{\tau}_{n}(s_{h+1}) V n τ ( s h + 1 ) converges with concentration to V ~ τ ( s h + 1 ) \widetilde{V}^{\tau}(s_{h+1}) V τ ( s h + 1 ) . Therefore Lemma 5.5 applies and gives
plim n → ∞ Q ^ n τ ( s h , a ) = Q ~ τ ( s h , a ) . \underset{n\to\infty}{\operatorname{plim}}\widehat{Q}^{\tau}_{n}(s_h,a)=\widetilde{Q}^{\tau}(s_h,a). n → ∞ plim Q n τ ( s h , a ) = Q τ ( s h , a ) .
Since a a a was arbitrary, this proves Q-edge convergence for all actions at s h s_h s h .
II: V-node convergence from the local bandit theorem. At node s h s_h s h , view each action a ∈ A ( s h ) a \in\mathcal{A}(s_h) a ∈ A ( s h ) as an arm whose empirical CVaR estimator is
c ^ a , n τ : = Q ^ n τ ( s h , a ) , \widehat{c}^{\tau}_{a,n}:=\widehat{Q}^{\tau}_{n}(s_h,a), c a , n τ := Q n τ ( s h , a ) ,
and whose limiting CVaR value is
c a τ : = Q ~ τ ( s h , a ) . c^{\tau}_{a}:=\widetilde{Q}^{\tau}(s_h,a). c a τ := Q τ ( s h , a ) .
Step I verifies the armwise assumption
plim n → ∞ c ^ a , n τ = c a τ \underset{n\to\infty}{\operatorname{plim}}\widehat{c}^{\tau}_{a,n}=c^{\tau}_{a} n → ∞ plim c a , n τ = c a τ
for every action a a a . Hence Assumption 5.2 holds for the local bandit at node s h s_h s h . By Theorem 5.4, the local CATSO V-backup
V ^ n τ ( s h ) = ∑ a ∈ A ( s h ) T s h , a ( n ) T s h ( n ) Q ^ T s h , a ( n ) τ ( s h , a ) \widehat{V}^{\tau}_{n}(s_h)=\sum_{a\in\mathcal{A}(s_h)}\frac{T_{s_h,a}(n)}{T_{s_h}(n)}\widehat{Q}^{\tau}_{T_{s_h,a}(n)}(s_h,a) V n τ ( s h ) = a ∈ A ( s h ) ∑ T s h ( n ) T s h , a ( n ) Q T s h , a ( n ) τ ( s h , a )
satisfies
plim n → ∞ V ^ n τ ( s h ) = max a ∈ A ( s h ) Q ~ τ ( s h , a ) . \underset{n\to\infty}{\operatorname{plim}}\widehat{V}^{\tau}_{n}(s_h)=\max_{a\in\mathcal{A}(s_h)}\widetilde{Q}^{\tau}(s_h,a). n → ∞ plim V n τ ( s h ) = a ∈ A ( s h ) max Q τ ( s h , a ) .
By definition of the target recursion,
max a ∈ A ( s h ) Q ~ τ ( s h , a ) = V ~ τ ( s h ) . \max_{a\in\mathcal{A}(s_h)}\widetilde{Q}^{\tau}(s_h,a)=\widetilde{V}^{\tau}(s_h). a ∈ A ( s h ) max Q τ ( s h , a ) = V τ ( s h ) .
Therefore
plim n → ∞ V ^ n τ ( s h ) = V ~ τ ( s h ) . \underset{n\to\infty}{\operatorname{plim}}\widehat{V}^{\tau}_{n}(s_h)=\widetilde{V}^{\tau}(s_h). n → ∞ plim V n τ ( s h ) = V τ ( s h ) .
Conclusion. The induction step proves both Q-edge and V-node convergence at depth h h h whenever V-node convergence holds at depth h + 1 h+1 h + 1 . Since the terminal case holds, the result follows for every node in the finite-depth tree.
Proof of Theorem 5.6 for CATSO case
Proof. We prove the stronger statement that, for every depth h h h and every node s h s_h s h at that depth, there exist constants C h , V , C h , Q < ∞ C_{h,V}, C_{h,Q} < \infty C h , V , C h , Q < ∞ such that
E [ ∣ V ^ n τ ( s h ) − V ~ τ ( s h ) ∣ ] ≤ C h , V n − 1 / 2 \mathbb{E}\left[\left|\widehat{V}_n^\tau(s_h)-\widetilde{V}^\tau(s_h)\right|\right]\le C_{h,V}n^{-1/2} E [ V n τ ( s h ) − V τ ( s h ) ] ≤ C h , V n − 1/2
and, for every action a ∈ A ( s h ) a\in\mathcal{A}(s_h) a ∈ A ( s h ) ,
E [ ∣ Q ^ n τ ( s h , a ) − Q ~ τ ( s h , a ) ∣ ] ≤ C h , Q n − 1 / 2 . \mathbb{E}\left[\left|\widehat{Q}_n^\tau(s_h,a)-\widetilde{Q}^\tau(s_h,a)\right|\right]\le C_{h,Q}n^{-1/2}. E [ Q n τ ( s h , a ) − Q τ ( s h , a ) ] ≤ C h , Q n − 1/2 .
The desired theorem follows by taking h = 0 h=0 h = 0 and s h = s 0 s_h=s_0 s h = s 0 .
We use backward induction on the remaining depth H − h H-h H − h .
Base case. At depth H H H , the claimed V-rate holds by the terminal assumption:
E [ ∣ V ^ n τ ( s h ) − V ~ τ ( s h ) ∣ ] ≤ C H n − 1 / 2 . \mathbb{E}\left[\left|\widehat{V}_n^\tau(s_h)-\widetilde{V}^\tau(s_h)\right|\right]\le C_Hn^{-1/2}. E [ V n τ ( s h ) − V τ ( s h ) ] ≤ C H n − 1/2 .
Thus the induction starts with C H , V : = C H C_{H,V}:=C_H C H , V := C H .
Induction hypothesis. Assume that, for all nodes s h + 1 s_{h+1} s h + 1 at depth h + 1 h+1 h + 1 ,
E [ ∣ V ^ n τ ( s h + 1 ) − V ~ τ ( s h + 1 ) ∣ ] ≤ C h + 1 , V n − 1 / 2 . \mathbb{E}\left[\left|\widehat{V}_n^\tau(s_{h+1})-\widetilde{V}^\tau(s_{h+1})\right|\right]\le C_{h+1,V}n^{-1/2}. E [ V n τ ( s h + 1 ) − V τ ( s h + 1 ) ] ≤ C h + 1 , V n − 1/2 .
We prove the corresponding Q-rate and V-rate at depth h h h .
I: Q-edge CVaR rate. Fix a node s h s_h s h and an action a ∈ A ( s h ) a\in\mathcal{A}(s_h) a ∈ A ( s h ) . Let
{ s h + 1 ( 1 ) , … , s h + 1 ( M ) } \left\{s_{h+1}^{(1)},\ldots,s_{h+1}^{(M)}\right\} { s h + 1 ( 1 ) , … , s h + 1 ( M ) }
be the finite support of P ( ⋅ ∣ s h , a ) P(\cdot\mid s_h,a) P ( ⋅ ∣ s h , a ) , and write
p m : = P ( s h + 1 ( m ) ∣ s h , a ) . p_m:=P(s_{h+1}^{(m)}\mid s_h,a). p m := P ( s h + 1 ( m ) ∣ s h , a ) .
On the i i i -th local visit to edge ( s h , a ) (s_h,a) ( s h , a ) , let
X i ∼ R ( s h , a , S i ) , S i ∼ P ( ⋅ ∣ s h , a ) . X_i\sim\mathcal{R}(s_h,a,S_i),\qquad S_i\sim P(\cdot\mid s_h,a). X i ∼ R ( s h , a , S i ) , S i ∼ P ( ⋅ ∣ s h , a ) .
Define the plug-in backed-up sample
q ˉ i ( n ) : = X i + γ V ^ N S i τ ( S i ) , \bar{q}_i^{(n)}:=X_i+\gamma\widehat{V}_{N_{S_i}}^\tau(S_i), q ˉ i ( n ) := X i + γ V N S i τ ( S i ) ,
where N S i n N_{S_i}^n N S i n is the number of appearances of child state S i S_i S i among the first n n n local visits to edge ( s h , a ) (s_h,a) ( s h , a ) . Define also the oracle sample
G i : = X i + γ V ~ τ ( S i ) . G_i:=X_i+\gamma\widetilde{V}^\tau(S_i). G i := X i + γ V τ ( S i ) .
Let
D ^ n ( s h , a ) : = 1 n ∑ i = 1 n δ q ˉ i ( n ) , D h τ ( s h , a ) : = L ( G 1 ) . \widehat{\mathcal{D}}_n(s_h,a):=\frac{1}{n}\sum_{i=1}^{n}\delta_{\bar{q}_i^{(n)}},\qquad\mathcal{D}_h^\tau(s_h,a):=\mathcal{L}(G_1). D n ( s h , a ) := n 1 i = 1 ∑ n δ q ˉ i ( n ) , D h τ ( s h , a ) := L ( G 1 ) .
Then
Q ^ n τ ( s h , a ) = CVaR τ ( D ^ n ( s h , a ) ) , Q ~ τ ( s h , a ) = CVaR τ ( D h τ ( s h , a ) ) . \widehat{Q}_n^\tau(s_h,a)=\operatorname{CVaR}_\tau\left(\widehat{\mathcal{D}}_n(s_h,a)\right),\qquad\widetilde{Q}^\tau(s_h,a)=\operatorname{CVaR}_\tau\left(\mathcal{D}_h^\tau(s_h,a)\right). Q n τ ( s h , a ) = CVaR τ ( D n ( s h , a ) ) , Q τ ( s h , a ) = CVaR τ ( D h τ ( s h , a ) ) .
By the 1 / τ 1/\tau 1/ τ -Lipschitz property of CVaR in W 1 W_1 W 1 ,
∣ Q ^ n τ ( s h , a ) − Q ~ τ ( s h , a ) ∣ ≤ 1 τ W 1 ( D ^ n ( s h , a ) , D h τ ( s h , a ) ) . \left|\widehat{Q}_n^\tau(s_h,a)-\widetilde{Q}^\tau(s_h,a)\right|\le\frac{1}{\tau}W_1\left(\widehat{\mathcal{D}}_n(s_h,a),\mathcal{D}_h^\tau(s_h,a)\right). Q n τ ( s h , a ) − Q τ ( s h , a ) ≤ τ 1 W 1 ( D n ( s h , a ) , D h τ ( s h , a ) ) .
Introduce the oracle empirical law
D ^ h , n o r ( s h , a ) : = 1 n ∑ i = 1 n δ G i . \widehat{\mathcal{D}}^{\mathrm{or}}_{h,n}(s_h,a) := \frac{1}{n}\sum_{i=1}^{n}\delta_{G_i}. D h , n or ( s h , a ) := n 1 i = 1 ∑ n δ G i .
The triangle inequality gives
W 1 ( D ^ h , n , D ^ h , n τ ) ≤ W 1 ( D ^ h , n , D ^ h , n o r ) + W 1 ( D ^ h , n o r , D τ ) . W_1\left(\widehat{\mathcal{D}}_{h,n},\widehat{\mathcal{D}}^\tau_{h,n}\right) \le W_1\left(\widehat{\mathcal{D}}_{h,n},\widehat{\mathcal{D}}^{\mathrm{or}}_{h,n}\right) + W_1\left(\widehat{\mathcal{D}}^{\mathrm{or}}_{h,n},\mathcal{D}^\tau\right). W 1 ( D h , n , D h , n τ ) ≤ W 1 ( D h , n , D h , n or ) + W 1 ( D h , n or , D τ ) .
For the first term, couple each plug-in sample with its oracle counterpart. Then
W 1 ( D ^ n , D ^ n o r ) ≤ 1 n ∑ i = 1 n ∣ q ˉ i ( n ) − G i ∣ = γ n ∑ i = 1 n ∣ V ^ N s i τ ( S i ) − V ~ τ ( S i ) ∣ = γ ∑ m = 1 M N m n n ∣ V ^ N m n τ ( s h + 1 ( m ) ) − V ~ τ ( s h + 1 ( m ) ) ∣ . \begin{aligned}
W_1\left(\widehat{\mathcal{D}}_n,\widehat{\mathcal{D}}^{\mathrm{or}}_n\right) \le\frac{1}{n}\sum_{i=1}^{n}\left|\bar{q}^{(n)}_i-G_i\right| \\
&= \frac{\gamma}{n}\sum_{i=1}^{n}\left|\widehat{V}^{\tau}_{N_{s_i}}(S_i)-\widetilde{V}^{\tau}(S_i)\right| \\
&= \gamma\sum_{m=1}^{M}\frac{N^n_m}{n}\left|\widehat{V}^{\tau}_{N^n_m}\left(s^{(m)}_{h+1}\right)-\widetilde{V}^{\tau}\left(s^{(m)}_{h+1}\right)\right|.
\end{aligned} W 1 ( D n , D n or ) ≤ n 1 i = 1 ∑ n q ˉ i ( n ) − G i = n γ i = 1 ∑ n V N s i τ ( S i ) − V τ ( S i ) = γ m = 1 ∑ M n N m n V N m n τ ( s h + 1 ( m ) ) − V τ ( s h + 1 ( m ) ) .
Taking expectations and using the induction hypothesis at the stopped count N m n N^n_m N m n ,
E [ W 1 ( D ^ n , D ^ n o r ) ] ≤ γ ∑ m = 1 M E [ N m n n C h + 1 , V ( N m n ∨ 1 ) − 1 / 2 ] ≤ γ C h + 1 , V n ∑ m = 1 M E [ N m n ] ≤ γ C h + 1 , V n ∑ m = 1 M E [ N m n ] = γ C h + 1 , V ( ∑ m = 1 M p m ) n − 1 / 2 . \begin{aligned}
\mathbb{E}\left[W_1\left(\widehat{\mathcal{D}}_n,\widehat{\mathcal{D}}^{\mathrm{or}}_n\right)\right] \\
&\le\gamma\sum_{m=1}^{M}\mathbb{E}\left[\frac{N^n_m}{n}C_{h+1,V}\left(N^n_m\vee1\right)^{-1/2}\right] \\
&\le\frac{\gamma C_{h+1,V}}{n}\sum_{m=1}^{M}\mathbb{E}\left[\sqrt{N^n_m}\right] \\
&\le\frac{\gamma C_{h+1,V}}{n}\sum_{m=1}^{M}\sqrt{\mathbb{E}[N^n_m]} \\
&= \gamma C_{h+1,V}\left(\sum_{m=1}^{M}\sqrt{p_m}\right)n^{-1/2}.
\end{aligned} E [ W 1 ( D n , D n or ) ] ≤ γ m = 1 ∑ M E [ n N m n C h + 1 , V ( N m n ∨ 1 ) − 1/2 ] ≤ n γ C h + 1 , V m = 1 ∑ M E [ N m n ] ≤ n γ C h + 1 , V m = 1 ∑ M E [ N m n ] = γ C h + 1 , V ( m = 1 ∑ M p m ) n − 1/2 .
The inequality
E [ N m n ] ≤ E [ N m n ] \mathbb{E}\left[\sqrt{N^n_m}\right]\le\sqrt{\mathbb{E}[N^n_m]} E [ N m n ] ≤ E [ N m n ]
uses Jensen’s inequality.
For the second term, the oracle samples G 1 , … , G n G_1,\ldots,G_n G 1 , … , G n are i.i.d. and bounded. Let
R h : = R max ∑ ℓ = 0 H − h − 1 γ ℓ = R max 1 − γ H − h 1 − γ . R_h := R_{\max}\sum_{\ell=0}^{H-h-1}\gamma^\ell= R_{\max}\frac{1-\gamma^{H-h}}{1-\gamma}. R h := R m a x ℓ = 0 ∑ H − h − 1 γ ℓ = R m a x 1 − γ 1 − γ H − h .
Then D τ ( s h , a ) \mathcal{D}^\tau(s_h,a) D τ ( s h , a ) is supported on [ 0 , R h ] [0,R_h] [ 0 , R h ] . By the one-dimensional empirical W 1 W_1 W 1 bound,
E [ W 1 ( D ^ n o r , D τ ) ] ≤ C F G R h n − 1 / 2 . \mathbb{E}\left[W_1\left(\widehat{\mathcal{D}}^{\mathrm{or}}_n,\mathcal{D}^\tau\right)\right]\le C_{\mathrm{FG}}R_hn^{-1/2}. E [ W 1 ( D n or , D τ ) ] ≤ C FG R h n − 1/2 .
Combining the two bounds yields
E [ ∣ Q ^ n τ ( s h , a ) − Q ~ τ ( s h , a ) ∣ ] ≤ 1 τ [ γ C h + 1 , V ∑ m = 1 M p m + C F G R h ] n − 1 / 2 . \mathbb{E}\left[\left|\widehat{Q}^\tau_n(s_h,a)-\widetilde{Q}^\tau(s_h,a)\right|\right]\le\frac{1}{\tau}\left[\gamma C_{h+1,V}\sum_{m=1}^{M}\sqrt{p_m}+C_{\mathrm{FG}}R_h\right]n^{-1/2}. E [ Q n τ ( s h , a ) − Q τ ( s h , a ) ] ≤ τ 1 [ γ C h + 1 , V m = 1 ∑ M p m + C FG R h ] n − 1/2 .
Thus the Q-rate holds with
C h , Q : = 1 τ [ γ C h + 1 , V max s h , a ∑ s ′ ∈ supp ( P ( ⋅ ∣ s h , a ) ) P ( s ′ ∣ s h , a ) + C F G R h ] . C_{h,Q}:=\frac{1}{\tau}\left[\gamma C_{h+1,V}\max_{s_h,a}\sum_{s'\in\operatorname{supp}(P(\cdot\mid s_h,a))}\sqrt{P(s'\mid s_h,a)}+C_{\mathrm{FG}}R_h\right]. C h , Q := τ 1 γ C h + 1 , V s h , a max s ′ ∈ supp ( P ( ⋅ ∣ s h , a )) ∑ P ( s ′ ∣ s h , a ) + C FG R h .
II: V-node CVaR-average rate. Fix a node s h s_h s h . Write
q a τ : = Q ~ τ ( s h , a ) , q ∗ τ : = max a ∈ A ( s h ) q a τ = V ~ τ ( s h ) . q_a^\tau:= \widetilde{Q}^\tau(s_h,a), \qquad q_*^\tau:= \max_{a\in\mathcal{A}(s_h)} q_a^\tau= \widetilde{V}^\tau(s_h). q a τ := Q τ ( s h , a ) , q ∗ τ := a ∈ A ( s h ) max q a τ = V τ ( s h ) .
and let
a ∗ τ ( s h ) ∈ arg max a ∈ A ( s h ) q a τ . a_*^\tau(s_h) \in\arg\max_{a\in\mathcal{A}(s_h)} q_a^\tau. a ∗ τ ( s h ) ∈ arg a ∈ A ( s h ) max q a τ .
be the unique CVaR-optimal action. For readability, write
T a ( n ) : = T s h , a ( n ) , w a ( n ) : = T a ( n ) n . T_a(n) := T_{s_h,a}(n), \qquad w_a(n) := \frac{T_a(n)}{n}. T a ( n ) := T s h , a ( n ) , w a ( n ) := n T a ( n ) .
The algorithm uses the CVaR-average V-backup
V ^ n τ ( s h ) = ∑ a ∈ A ( s h ) w a ( n ) Q ^ h , T a ( n ) τ ( s h , a ) . \widehat{V}_n^\tau(s_h) = \sum_{a\in\mathcal{A}(s_h)} w_a(n)\widehat{Q}_{h,T_a(n)}^\tau(s_h,a). V n τ ( s h ) = a ∈ A ( s h ) ∑ w a ( n ) Q h , T a ( n ) τ ( s h , a ) .
Subtracting q ∗ τ q_*^\tau q ∗ τ and adding/subtracting q a τ q_a^\tau q a τ gives
V ^ n τ ( s h ) − q ∗ τ = ∑ a w a ( n ) ( Q ^ h , T a ( n ) τ ( s h , a ) − q a τ ) + ∑ a w a ( n ) ( q a τ − q ∗ τ ) . \widehat{V}_n^\tau(s_h)-q_*^\tau= \sum_a w_a(n)\left(\widehat{Q}_{h,T_a(n)}^\tau(s_h,a)-q_a^\tau\right)+\sum_a w_a(n)\left(q_a^\tau-q_*^\tau\right). V n τ ( s h ) − q ∗ τ = a ∑ w a ( n ) ( Q h , T a ( n ) τ ( s h , a ) − q a τ ) + a ∑ w a ( n ) ( q a τ − q ∗ τ ) .
Taking absolute values and using that all CVaR values lie in [ 0 , R h ] [0,R_h] [ 0 , R h ] ,
∣ V ^ n τ ( s h ) − q ∗ τ ∣ ≤ ∑ a w a ( n ) ∣ Q ^ h , T a ( n ) τ ( s h , a ) − q a τ ∣ + R h ∑ a ≠ a ∗ τ ( s h ) w a ( n ) . \left|\widehat{V}_n^\tau(s_h)-q_*^\tau\right| \le\sum_a w_a(n)\left|\widehat{Q}_{h,T_a(n)}^\tau(s_h,a)-q_a^\tau\right|+R_h\sum_{a\ne a_*^\tau(s_h)}w_a(n). V n τ ( s h ) − q ∗ τ ≤ a ∑ w a ( n ) Q h , T a ( n ) τ ( s h , a ) − q a τ + R h a = a ∗ τ ( s h ) ∑ w a ( n ) .
For suboptimal actions, the Q-error is also bounded by R h R_h R h . Hence
∣ V ^ n τ ( s h ) − q ∗ τ ∣ ≤ ∣ Q ^ h , T a ∗ τ ( n ) τ ( s h , a ∗ τ ) − q ∗ τ ∣ + 2 R h ∑ a ≠ a ∗ τ ( s h ) w a ( n ) . \left|\widehat{V}_n^\tau(s_h)-q_*^\tau\right| \le\left|\widehat{Q}_{h,T_{a_*^\tau}(n)}^\tau(s_h,a_*^\tau)-q_*^\tau\right|+2R_h\sum_{a\ne a_*^\tau(s_h)}w_a(n). V n τ ( s h ) − q ∗ τ ≤ Q h , T a ∗ τ ( n ) τ ( s h , a ∗ τ ) − q ∗ τ + 2 R h a = a ∗ τ ( s h ) ∑ w a ( n ) .
Taking expectations gives
E [ ∣ V ^ n τ ( s h ) − V ~ τ ( s h ) ∣ ] ≤ E [ ∣ Q ^ h , T a ∗ τ ( n ) τ ( s h , a ∗ τ ) − Q ~ τ ( s h , a ∗ τ ) ∣ ] + 2 R h n ∑ a ≠ a ∗ τ ( s h ) E [ T a ( n ) ] . (42) \mathbb{E}\left[\left|\widehat{V}_n^\tau(s_h)-\widetilde{V}^\tau(s_h)\right|\right]\le\mathbb{E}\left[\left|\widehat{Q}_{h,T_{a_*^\tau}(n)}^\tau(s_h,a_*^\tau)-\widetilde{Q}^\tau(s_h,a_*^\tau)\right|\right]+\frac{2R_h}{n}\sum_{a\ne a_*^\tau(s_h)}\mathbb{E}[T_a(n)]. \tag*{(42)} E [ V n τ ( s h ) − V τ ( s h ) ] ≤ E [ Q h , T a ∗ τ ( n ) τ ( s h , a ∗ τ ) − Q τ ( s h , a ∗ τ ) ] + n 2 R h a = a ∗ τ ( s h ) ∑ E [ T a ( n )] . (42)
We first bound the suboptimal-weight term. By the local CATSO suboptimal-arm pull bound,
E [ T a ( n ) ] ≤ L a log n + D a n 1 / 2 \mathbb{E}[T_a(n)]\le L_a\log n+D_an^{1/2} E [ T a ( n )] ≤ L a log n + D a n 1/2
for constants L a , D a < ∞ L_a,D_a<\infty L a , D a < ∞ . Therefore,
1 n ∑ a ≠ a ∗ τ ( s h ) E [ T a ( n ) ] = O ( n − 1 / 2 ) , \frac{1}{n}\sum_{a\ne a_*^\tau(s_h)}\mathbb{E}[T_a(n)]=O(n^{-1/2}), n 1 a = a ∗ τ ( s h ) ∑ E [ T a ( n )] = O ( n − 1/2 ) ,
since log n / n = O ( n − 1 / 2 ) \log n/n=O(n^{-1/2}) log n / n = O ( n − 1/2 ) .
It remains to control the optimal-action Q-error at the random count T a ∗ τ ( n ) T_{a_*^\tau}(n) T a ∗ τ ( n ) . Let
S n : = ∑ a ≠ a ∗ τ ( s h ) T a ( n ) . S_n := \sum_{a\ne a_*^\tau(s_h)}T_a(n). S n := a = a ∗ τ ( s h ) ∑ T a ( n ) .
Then
T a ∗ τ ( n ) = n − S n . T_{a_*^\tau}(n)=n-S_n. T a ∗ τ ( n ) = n − S n .
By Markov’s inequality and the suboptimal-pull bound,
P ( S n > n 2 ) ≤ 2 E [ S n ] n = O ( n − 1 / 2 ) . \mathbb{P}\left(S_n>\frac{n}{2}\right)\le\frac{2\mathbb{E}[S_n]}{n}=O(n^{-1/2}). P ( S n > 2 n ) ≤ n 2 E [ S n ] = O ( n − 1/2 ) .
On the event { S n ≤ n / 2 } \{S_n\le n/2\} { S n ≤ n /2 } , we have
T a ∗ τ ( n ) ≥ n 2 . T_{a_*^\tau}(n)\ge\frac{n}{2}. T a ∗ τ ( n ) ≥ 2 n .
Using the stopped-count form of the Q-rate from I,
E [ ∣ Q ~ ^ T a ⋆ ( n ) ( s h , a h ⋆ ) − Q ~ ( s h , a h ⋆ ) ∣ 1 { S n ≤ n 2 } ] ≤ C h , Q ( n 2 ) − 1 / 2 . \mathbb{E}\left[\left|\widehat{\widetilde{Q}}_{T_{a^\star}(n)}(s_h,a_h^\star)-\widetilde{Q}(s_h,a_h^\star)\right|\mathbb{1}\left\{S_n\leq\frac{n}{2}\right\}\right]
\leq C_{h,Q}\left(\frac{n}{2}\right)^{-1/2}. E [ Q T a ⋆ ( n ) ( s h , a h ⋆ ) − Q ( s h , a h ⋆ ) 1 { S n ≤ 2 n } ] ≤ C h , Q ( 2 n ) − 1/2 .
On the complement { S n > n / 2 } \{S_n>n/2\} { S n > n /2 } , the absolute error is bounded by R h R_h R h . Thus
E [ ∣ Q ~ ^ T a h ⋆ ( n ) ( s h , a h ⋆ ) − Q ~ ( s h , a h ⋆ ) ∣ ] ≤ 2 C h , Q n − 1 / 2 + R h P ( S n > n 2 ) = O ( n − 1 / 2 ) . \mathbb{E}\left[\left|\widehat{\widetilde{Q}}_{T_{a_h^\star}(n)}(s_h,a_h^\star)-\widetilde{Q}(s_h,a_h^\star)\right|\right]
\leq\sqrt{2}C_{h,Q}n^{-1/2}+R_h\mathbb{P}\left(S_n>\frac{n}{2}\right)
=O(n^{-1/2}). E [ Q T a h ⋆ ( n ) ( s h , a h ⋆ ) − Q ( s h , a h ⋆ ) ] ≤ 2 C h , Q n − 1/2 + R h P ( S n > 2 n ) = O ( n − 1/2 ) .
Substituting this and the suboptimal-weight bound into (42) gives
E [ ∣ V ~ ^ n ( s h ) − V ~ ( s h ) ∣ ] ≤ C h , V n − 1 / 2 \mathbb{E}\left[\left|\widehat{\widetilde{V}}_n(s_h)-\widetilde{V}(s_h)\right|\right]\leq C_{h,V}n^{-1/2} E [ V n ( s h ) − V ( s h ) ] ≤ C h , V n − 1/2
for some finite constant C h , V C_{h,V} C h , V .
In sum. The induction proves the V-rate at depth h h h whenever it holds at depth h + 1 h+1 h + 1 . Since the terminal case holds, the claim holds for every node in the finite-depth tree. In particular, at the root s 0 s_0 s 0 ,
E [ ∣ V ~ ^ n ( s 0 ) − V ~ ( s 0 ) ∣ ] ≤ C ′ n − 1 / 2 \mathbb{E}\left[\left|\widehat{\widetilde{V}}_n(s_0)-\widetilde{V}(s_0)\right|\right]\leq C'n^{-1/2} E [ V n ( s 0 ) − V ( s 0 ) ] ≤ C ′ n − 1/2
for a finite constant C ′ = C 0 , V C'=C_{0,V} C ′ = C 0 , V .
Convergence of PATSO in MCTS
Similarly, we can derive the following Theorem
Theorem D.2 (Finite-depth CVaR convergence of PATSO). Let P A T S O PATSO P A T S O run on a finite tree of depth H H H with bounded rewards and finite transition support at every Q-edge.
Let
Q ~ ^ n ( s h , a ) : = CVaR τ ( D ^ n ( s h , a ) ) \widehat{\widetilde{Q}}_n(s_h,a):=\operatorname{CVaR}_{\tau}\left(\widehat{D}_n(s_h,a)\right) Q n ( s h , a ) := CVaR τ ( D n ( s h , a ) ) be the empirical CVaR Q-edge estimator after n n n local visits to edge ( s h , a ) (s_h,a) ( s h , a ) . Then, for every node s h s_h s h and every action a ∈ A ( s h ) a\in\mathcal{A}(s_h) a ∈ A ( s h ) ,
lim n → ∞ Q ~ ^ n ( s h , a ) = Q ~ ( s h , a ) , \lim_{n\to\infty}\widehat{\widetilde{Q}}_n(s_h,a)=\widetilde{Q}(s_h,a), n → ∞ lim Q n ( s h , a ) = Q ( s h , a ) , and
lim n → ∞ V ~ ^ n ( s h ) = V ~ ( s h ) . \lim_{n\to\infty}\widehat{\widetilde{V}}_n(s_h)=\widetilde{V}(s_h). n → ∞ lim V n ( s h ) = V ( s h ) .
Proof. The proof follows the same steps as Theorem D.1 by applying the results of Lemma 5.5 and Theorem 5.4 .
Proof of Theorem 5.6 for PATSO case
Proof. We prove the stronger statement that, for every depth h h h and every node s h s_h s h at that depth, there exists a constant C h , V < ∞ C_{h,V}<\infty C h , V < ∞ such that
E [ ∣ V ~ ^ n ( s h ) − V ~ ( s h ) ∣ ] ≤ C h , V n − 1 / 2 . \mathbb{E}\left[\left|\widehat{\widetilde{V}}_n(s_h)-\widetilde{V}(s_h)\right|\right]\leq C_{h,V}n^{-1/2}. E [ V n ( s h ) − V ( s h ) ] ≤ C h , V n − 1/2 .
The desired result follows by taking h = 0 h=0 h = 0 and s h = s 0 s_h=s_0 s h = s 0 .
We proceed by backward induction on the remaining depth H − h H-h H − h .
Base case. At terminal depth H H H , the claim is exactly the assumed rollout/leaf rate:
E [ ∣ V ~ ^ n ( s H ) − V ~ ( s H ) ∣ ] ≤ C H n − 1 / 2 . \mathbb{E}\left[\left|\widehat{\widetilde{V}}_n(s_H)-\widetilde{V}(s_H)\right|\right]\leq C_Hn^{-1/2}. E [ V n ( s H ) − V ( s H ) ] ≤ C H n − 1/2 .
Thus the induction starts with C H , V : = C H C_{H,V}:=C_H C H , V := C H . Induction hypothesis. Assume that for every child node s h + 1 s_{h+1} s h + 1 at depth h + 1 h+1 h + 1 ,
E [ ∣ V ^ n τ ( s h + 1 ) − V ~ τ ( s h + 1 ) ∣ ] ≤ C h + 1 , V n − 1 / 2 . \mathbb{E}\left[\left|\widehat{V}^{\tau}_{n}(s_{h+1})-\widetilde{V}^{\tau}(s_{h+1})\right|\right] \le C_{h+1,V}n^{-1/2}. E [ V n τ ( s h + 1 ) − V τ ( s h + 1 ) ] ≤ C h + 1 , V n − 1/2 .
We prove the corresponding statement at an arbitrary node s h s_h s h at depth h h h .
I: Q-edge CVaR convergence. Fix an action a ∈ A ( s h ) a \in\mathcal{A}(s_h) a ∈ A ( s h ) . Let
{ s h + 1 ( 1 ) , … , s h + 1 ( M ) } \left\{s_{h+1}^{(1)},\ldots,s_{h+1}^{(M)}\right\} { s h + 1 ( 1 ) , … , s h + 1 ( M ) }
be the finite support of P ( ⋅ ∣ s h , a ) P(\cdot\mid s_h,a) P ( ⋅ ∣ s h , a ) , and write
p m : = P ( s h + 1 ( m ) ∣ s h , a ) . p_m := P(s_{h+1}^{(m)} \mid s_h,a). p m := P ( s h + 1 ( m ) ∣ s h , a ) .
On the i i i -th local visit to edge ( s h , a ) (s_h,a) ( s h , a ) , let
X i ∼ R ( s h , a , S i ) , S i ∼ P ( ⋅ ∣ s h , a ) . X_i \sim\mathcal{R}(s_h,a,S_i), \qquad S_i \sim P(\cdot\mid s_h,a). X i ∼ R ( s h , a , S i ) , S i ∼ P ( ⋅ ∣ s h , a ) .
The backed-up sample stored by PATSO is
q ˉ i = X i + γ V ^ ℓ i τ ( S i ) , \bar{q}_i = X_i + \gamma\widehat{V}^{\tau}_{\ell_i}(S_i), q ˉ i = X i + γ V ℓ i τ ( S i ) ,
where ℓ i \ell_i ℓ i is the local visit count of the child state S i S_i S i at the moment this backup is formed. The corresponding oracle backed-up sample is
G i = X i + γ V ~ τ ( S i ) . G_i = X_i + \gamma\widetilde{V}^{\tau}(S_i). G i = X i + γ V τ ( S i ) .
Define the empirical PATSO Q-edge law after n n n local visits by
D ^ n ( s h , a ) : = 1 n ∑ i = 1 n δ q ˉ i , \widehat{D}_n(s_h,a) := \frac{1}{n}\sum_{i=1}^{n}\delta_{\bar{q}_i}, D n ( s h , a ) := n 1 i = 1 ∑ n δ q ˉ i ,
and define the oracle law by
D τ ( s h , a ) : = L ( G 1 ) . D^{\tau}(s_h,a) := \mathcal{L}(G_1). D τ ( s h , a ) := L ( G 1 ) .
Since uncapped PATSO stores one particle per distinct observed backed-up return, its maintained particle law is exactly D ^ n ( s h , a ) \widehat{D}_n(s_h,a) D n ( s h , a ) .
The empirical and population Q-edge CVaR values are
Q ^ n τ ( s h , a ) : = CVaR τ ( D ^ n ( s h , a ) ) , \widehat{Q}^{\tau}_n(s_h,a) := \operatorname{CVaR}_{\tau}\left(\widehat{D}_n(s_h,a)\right), Q n τ ( s h , a ) := CVaR τ ( D n ( s h , a ) ) ,
and
Q ~ τ ( s h , a ) : = CVaR τ ( D τ ( s h , a ) ) . \widetilde{Q}^{\tau}(s_h,a) := \operatorname{CVaR}_{\tau}\left(D^{\tau}(s_h,a)\right). Q τ ( s h , a ) := CVaR τ ( D τ ( s h , a ) ) .
We show that the armwise condition needed by the local PATSO bandit theorem holds. Introduce the oracle empirical law
D ^ n o r ( s h , a ) : = 1 n ∑ i = 1 n δ G i . \widehat{D}^{\mathrm{or}}_n(s_h,a) := \frac{1}{n}\sum_{i=1}^{n}\delta_{G_i}. D n or ( s h , a ) := n 1 i = 1 ∑ n δ G i .
By the 1 / τ 1/\tau 1/ τ -Lipschitz property of CVaR in W 1 W_1 W 1 ,
∣ Q ^ n τ ( s h , a ) − Q ~ τ ( s h , a ) ∣ ≤ 1 τ W 1 ( D ^ n ( s h , a ) , D τ ( s h , a ) ) . \left|\widehat{Q}^{\tau}_n(s_h,a)-\widetilde{Q}^{\tau}(s_h,a)\right| \le\frac{1}{\tau}W_1\left(\widehat{D}_n(s_h,a),D^{\tau}(s_h,a)\right). Q n τ ( s h , a ) − Q τ ( s h , a ) ≤ τ 1 W 1 ( D n ( s h , a ) , D τ ( s h , a ) ) .
Using the triangle inequality,
W 1 ( D ^ n , D τ ) ≤ W 1 ( D ^ n , D ^ n o r ) + W 1 ( D ^ n o r , D τ ) . W_1\left(\widehat{D}_n,D^{\tau}\right) \le W_1\left(\widehat{D}_n,\widehat{D}^{\mathrm{or}}_n\right)+W_1\left(\widehat{D}^{\mathrm{or}}_n,D^{\tau}\right). W 1 ( D n , D τ ) ≤ W 1 ( D n , D n or ) + W 1 ( D n or , D τ ) .
For the first term, use the natural index-wise coupling
π n : = 1 n ∑ i = 1 n δ ( q ˉ i , G i ) . \pi_n := \frac{1}{n}\sum_{i=1}^{n}\delta_{(\bar{q}_i,G_i)}. π n := n 1 i = 1 ∑ n δ ( q ˉ i , G i ) .
This coupling has marginals D ^ n \widehat{\mathcal{D}}_n D n and D ^ n o r \widehat{\mathcal{D}}_n^{\mathrm{or}} D n or , so
W 1 ( D ^ n , D ^ n o r ) ≤ 1 n ∑ i = 1 n ∣ q ^ i − G i ∣ = γ n ∑ i = 1 n ∣ V ^ i τ ( S i ) − V ~ τ ( S i ) ∣ . W_1\left(\widehat{\mathcal{D}}_n,\widehat{\mathcal{D}}_n^{\mathrm{or}}\right) \le\frac{1}{n}\sum_{i=1}^{n}\left|\widehat{q}_i-G_i\right|
= \frac{\gamma}{n}\sum_{i=1}^{n}\left|\widehat{V}_i^{\tau}(S_i)-\widetilde{V}^{\tau}(S_i)\right|. W 1 ( D n , D n or ) ≤ n 1 i = 1 ∑ n ∣ q i − G i ∣ = n γ i = 1 ∑ n V i τ ( S i ) − V τ ( S i ) .
Group the sum according to the sampled child state. If child state s h + 1 ( m ) s_{h+1}^{(m)} s h + 1 ( m ) is observed N m n N_m^n N m n times among the first n n n visits to edge ( s h , a ) (s_h,a) ( s h , a ) , then its local child estimator appears with visit indices r = 1 , … , N m n r=1,\ldots,N_m^n r = 1 , … , N m n . Hence
W 1 ( D ^ n , D ^ n o r ) ≤ γ n ∑ m = 1 M ∑ r = 1 N m n ∣ V ^ r τ ( s h + 1 ( m ) ) − V ~ τ ( s h + 1 ( m ) ) ∣ . W_1\left(\widehat{\mathcal{D}}_n,\widehat{\mathcal{D}}_n^{\mathrm{or}}\right) \le\frac{\gamma}{n}\sum_{m=1}^{M}\sum_{r=1}^{N_m^n}\left|\widehat{V}_r^{\tau}\left(s_{h+1}^{(m)}\right)-\widetilde{V}^{\tau}\left(s_{h+1}^{(m)}\right)\right|. W 1 ( D n , D n or ) ≤ n γ m = 1 ∑ M r = 1 ∑ N m n V r τ ( s h + 1 ( m ) ) − V τ ( s h + 1 ( m ) ) .
Taking expectations and applying the induction hypothesis,
E [ W 1 ( D ^ n , D ^ n o r ) ] ≤ γ C h + 1 , V n ∑ m = 1 M E [ ∑ r = 1 N m n r − 1 / 2 ] ≤ 2 γ C h + 1 , V n ∑ m = 1 M E [ N m n ] ≤ 2 γ C h + 1 , V n ∑ m = 1 M E [ N m n ] = 2 γ C h + 1 , V ( ∑ m = 1 M p m ) n − 1 / 2 . \begin{aligned}
\mathbb{E}\left[W_1\left(\widehat{\mathcal{D}}_n,\widehat{\mathcal{D}}_n^{\mathrm{or}}\right)\right]
\le\frac{\gamma C_{h+1,V}}{n}\sum_{m=1}^{M}\mathbb{E}\left[\sum_{r=1}^{N_m^n}r^{-1/2}\right] \\
&\le\frac{2\gamma C_{h+1,V}}{n}\sum_{m=1}^{M}\mathbb{E}\left[\sqrt{N_m^n}\right] \\
&\le\frac{2\gamma C_{h+1,V}}{n}\sum_{m=1}^{M}\sqrt{\mathbb{E}\left[N_m^n\right]} \\
&=2\gamma C_{h+1,V}\left(\sum_{m=1}^{M}\sqrt{p_m}\right)n^{-1/2}.
\end{aligned} E [ W 1 ( D n , D n or ) ] ≤ n γ C h + 1 , V m = 1 ∑ M E r = 1 ∑ N m n r − 1/2 ≤ n 2 γ C h + 1 , V m = 1 ∑ M E [ N m n ] ≤ n 2 γ C h + 1 , V m = 1 ∑ M E [ N m n ] = 2 γ C h + 1 , V ( m = 1 ∑ M p m ) n − 1/2 .
The second inequality uses
∑ r = 1 N r − 1 / 2 ≤ 2 N , \sum_{r=1}^{N}r^{-1/2}\le2\sqrt{N}, r = 1 ∑ N r − 1/2 ≤ 2 N ,
and the third uses Jensen’s inequality.
For the second term, the oracle samples
G i = X i + γ V ~ τ ( S i ) G_i=X_i+\gamma\widetilde{V}^{\tau}(S_i) G i = X i + γ V τ ( S i )
are i.i.d. and bounded. Let
R h : = R max ∑ ℓ = 0 H − h − 1 γ ℓ = R max 1 − γ H − h 1 − γ . R_h:=R_{\max}\sum_{\ell=0}^{H-h-1}\gamma^{\ell}
=R_{\max}\frac{1-\gamma^{H-h}}{1-\gamma}. R h := R m a x ℓ = 0 ∑ H − h − 1 γ ℓ = R m a x 1 − γ 1 − γ H − h .
Then D τ ( s h , a ) \mathcal{D}^{\tau}(s_h,a) D τ ( s h , a ) is supported on [ 0 , R h ] [0,R_h] [ 0 , R h ] . By the one-dimensional empirical W 1 W_1 W 1 bound,
E [ W 1 ( D ^ n o r , D τ ) ] ≤ C F G R h n − 1 / 2 . \mathbb{E}\left[W_1\left(\widehat{\mathcal{D}}_n^{\mathrm{or}},\mathcal{D}^{\tau}\right)\right]\le C_{\mathrm{FG}}R_hn^{-1/2}. E [ W 1 ( D n or , D τ ) ] ≤ C FG R h n − 1/2 .
Combining the two W 1 W_1 W 1 bounds gives
E [ ∣ Q ^ n τ ( s h , a ) − Q ~ τ ( s h , a ) ∣ ] ≤ 1 τ [ 2 γ C h + 1 , V ∑ m = 1 M p m + C F G R h ] n − 1 / 2 . \mathbb{E}\left[\left|\widehat{Q}_n^{\tau}(s_h,a)-\widetilde{Q}^{\tau}(s_h,a)\right|\right]\le\frac{1}{\tau}\left[2\gamma C_{h+1,V}\sum_{m=1}^{M}\sqrt{p_m}+C_{\mathrm{FG}}R_h\right]n^{-1/2}. E [ Q n τ ( s h , a ) − Q τ ( s h , a ) ] ≤ τ 1 [ 2 γ C h + 1 , V m = 1 ∑ M p m + C FG R h ] n − 1/2 .
Thus, for every action a a a , the empirical Q-edge CVaR estimator converges to Q ~ τ ( s h , a ) \widetilde{Q}^{\tau}(s_h,a) Q τ ( s h , a ) at rate O ( n − 1 / 2 ) O(n^{-1/2}) O ( n − 1/2 ) in L 1 L^1 L 1 . Since all quantities are bounded, the same argument also verifies the armwise convergence-with-concentration premise used by the local PATSO bandit theorem.
II: Apply the local PATSO CVaR bandit theorem at s h s_h s h . At node s h s_h s h , view every action a ∈ A ( s h ) a \in\mathcal{A}(s_h) a ∈ A ( s h ) as a local bandit arm with limiting CVaR value
c a τ : = Q ~ τ ( s h , a ) . c_a^\tau:= \widetilde{Q}^{\tau}(s_h,a). c a τ := Q τ ( s h , a ) .
The local optimal CVaR value is
c ⋆ τ : = max a ∈ A ( s h ) c a τ = V ~ τ ( s h ) . c_\star^\tau:= \max_{a \in\mathcal{A}(s_h)} c_a^\tau= \widetilde{V}^{\tau}(s_h). c ⋆ τ := a ∈ A ( s h ) max c a τ = V τ ( s h ) .
The empirical arm score after m m m local pulls is
c ^ a , m τ : = Q ^ m τ ( s h , a ) . \widehat{c}_{a,m}^\tau:= \widehat{Q}_m^\tau(s_h,a). c a , m τ := Q m τ ( s h , a ) .
By I, the armwise empirical CVaR scores satisfy the required asymptotic stationarity/concentration condition for the local bandit at s h s_h s h .
The V-node backup used by the algorithm is exactly the CVaR-average local bandit backup:
V ^ n τ ( s h ) = ∑ a ∈ A ( s h ) T s h , a ( n ) T s h ( n ) Q ^ T s h , a ( n ) τ ( s h , a ) . \widehat{V}_n^\tau(s_h) = \sum_{a \in\mathcal{A}(s_h)} \frac{T_{s_h,a}(n)}{T_{s_h}(n)} \widehat{Q}_{T_{s_h,a}(n)}^\tau(s_h,a). V n τ ( s h ) = a ∈ A ( s h ) ∑ T s h ( n ) T s h , a ( n ) Q T s h , a ( n ) τ ( s h , a ) .
By the local PATSO CVaR bandit theorem with the O ( n − 1 / 2 ) O(n^{-1/2}) O ( n − 1/2 ) L 1 L^1 L 1 rate,
E [ ∣ V ^ n τ ( s h ) − c ⋆ τ ∣ ] ≤ C h , B n − 1 / 2 , \mathbb{E}\left[\left|\widehat{V}_n^\tau(s_h)-c_\star^\tau\right|\right] \le C_{h,B}n^{-1/2}, E [ V n τ ( s h ) − c ⋆ τ ] ≤ C h , B n − 1/2 ,
for some constant C h , B < ∞ C_{h,B} < \infty C h , B < ∞ . Since
c ⋆ τ = V ~ τ ( s h ) , c_\star^\tau= \widetilde{V}^\tau(s_h), c ⋆ τ = V τ ( s h ) ,
we obtain
E [ ∣ V ^ n τ ( s h ) − V ~ τ ( s h ) ∣ ] ≤ C h , V n − 1 / 2 , \mathbb{E}\left[\left|\widehat{V}_n^\tau(s_h)-\widetilde{V}^\tau(s_h)\right|\right] \le C_{h,V}n^{-1/2}, E [ V n τ ( s h ) − V τ ( s h ) ] ≤ C h , V n − 1/2 ,
with C h , V : = C h , B C_{h,V} := C_{h,B} C h , V := C h , B .
Conclusion. The induction step shows that if all child V-estimators at depth h + 1 h+1 h + 1 have the O ( n − 1 / 2 ) O(n^{-1/2}) O ( n − 1/2 ) L 1 L^1 L 1 rate, then the V-estimator at every parent node s h s_h s h also has the O ( n − 1 / 2 ) O(n^{-1/2}) O ( n − 1/2 ) L 1 L^1 L 1 rate. Since the terminal case holds, the claim follows for every node in the finite-depth tree. In particular, at the root node s 0 s_0 s 0 there exists a finite constant C ′ = C 0 , V C' = C_{0,V} C ′ = C 0 , V such that
E [ ∣ V ^ n τ ( s 0 ) − V ~ τ ( s 0 ) ∣ ] ≤ C ′ n − 1 / 2 . \mathbb{E}\left[\left|\widehat{V}_n^\tau(s_0)-\widetilde{V}^\tau(s_0)\right|\right] \le C'n^{-1/2}. E [ V n τ ( s 0 ) − V τ ( s 0 ) ] ≤ C ′ n − 1/2 .
□ \square □
Risk-Sensitive Simple-Regret Bound
The convergence theorems in Section 5 control the absolute root value error
ε τ ( n ) : = E [ ∣ V ^ n τ ( s 0 ) − V ~ τ ( s 0 ) ∣ ] . \varepsilon_\tau(n) := \mathbb{E}\left[\left|\widehat{V}_n^\tau(s_0)-\widetilde{V}^\tau(s_0)\right|\right]. ε τ ( n ) := E [ V n τ ( s 0 ) − V τ ( s 0 ) ] .
The risk-sensitive simple regret used in the main text is the signed root value gap
S R τ ( n ) : = E [ V ~ τ ( s 0 ) − V ^ n τ ( s 0 ) ] . \mathrm{SR}_\tau(n) := \mathbb{E}\left[\widetilde{V}^\tau(s_0)-\widehat{V}_n^\tau(s_0)\right]. SR τ ( n ) := E [ V τ ( s 0 ) − V n τ ( s 0 ) ] .
Since x ≤ ∣ x ∣ x \le|x| x ≤ ∣ x ∣ for every real number x x x , we have
S R τ ( n ) = E [ V ~ τ ( s 0 ) − V ^ n τ ( s 0 ) ] ≤ E [ ∣ V ~ τ ( s 0 ) − V ^ n τ ( s 0 ) ∣ ] = ε τ ( n ) . \mathrm{SR}_\tau(n) = \mathbb{E}\left[\widetilde{V}^\tau(s_0)-\widehat{V}_n^\tau(s_0)\right]
\le\mathbb{E}\left[\left|\widetilde{V}^\tau(s_0)-\widehat{V}_n^\tau(s_0)\right|\right] = \varepsilon_\tau(n). SR τ ( n ) = E [ V τ ( s 0 ) − V n τ ( s 0 ) ] ≤ E [ V τ ( s 0 ) − V n τ ( s 0 ) ] = ε τ ( n ) .
Thus any upper bound on the absolute root value error immediately gives the same-order upper bound on S R τ ( n ) \mathrm{SR}_\tau(n) SR τ ( n ) .
Corollary D.3 (Risk-sensitive simple regret of CATSO). Under the assumptions of Theorem 5.6, fixed-grid CATSO satisfies
S R τ ( n ) ≤ C C A T S O n − 1 / 2 . \mathrm{SR}_{\tau}(n) \le C_{\mathrm{CATSO}}n^{-1/2}. SR τ ( n ) ≤ C CATSO n − 1/2 . For the unprojected target with grid mesh Δ N \Delta_N Δ N ,
S R τ ( n ) ≤ C C A T S O n − 1 / 2 + O ( Δ N τ ) . \mathrm{SR}_{\tau}(n) \le C_{\mathrm{CATSO}}n^{-1/2} + O\left(\frac{\Delta_N}{\tau}\right). SR τ ( n ) ≤ C CATSO n − 1/2 + O ( τ Δ N ) .
Proof. Theorem 5.6 gives
E τ ( n ) ≤ C C A T S O n − 1 / 2 \mathcal{E}_{\tau}(n) \le C_{\mathrm{CATSO}}n^{-1/2} E τ ( n ) ≤ C CATSO n − 1/2
for the fixed-grid target, and adds the discretization term O ( Δ N / τ ) O(\Delta_N/\tau) O ( Δ N / τ ) for the unprojected target. Since S R τ ( n ) ≤ E τ ( n ) \mathrm{SR}_{\tau}(n) \le\mathcal{E}_{\tau}(n) SR τ ( n ) ≤ E τ ( n ) , the result follows.
Corollary D.4 (Risk-sensitive simple regret of uncapped PATSO). Under the assumptions of Theorem 5.6, uncapped PATSO satisfies
S R τ ( n ) ≤ C P A T S O n − 1 / 2 . \mathrm{SR}_{\tau}(n) \le C_{\mathrm{PATSO}}n^{-1/2}. SR τ ( n ) ≤ C PATSO n − 1/2 .
Proof. Theorem 5.6 gives
E τ ( n ) ≤ C P A T S O n − 1 / 2 . \mathcal{E}_{\tau}(n) \le C_{\mathrm{PATSO}}n^{-1/2}. E τ ( n ) ≤ C PATSO n − 1/2 .
The claim follows from
S R τ ( n ) ≤ E τ ( n ) . \mathrm{SR}_{\tau}(n) \le\mathcal{E}_{\tau}(n). SR τ ( n ) ≤ E τ ( n ) .
Corollary D.5 (Risk-sensitive simple regret of capped PATSO). Suppose capped PATSO satisfies the compression condition
W 1 ( D ^ m , K c a p ( s , a ) , D ^ m u n c a p ( s , a ) ) ≤ ε K ( s , a ) W_1\left(\widehat{\mathcal{D}}^{\mathrm{cap}}_{m,K}(s,a),\widehat{\mathcal{D}}^{\mathrm{uncap}}_m(s,a)\right) \le\varepsilon_K(s,a) W 1 ( D m , K cap ( s , a ) , D m uncap ( s , a ) ) ≤ ε K ( s , a ) at every visited Q Q Q -edge. Then
S R τ ( n ) ≤ C P A T S O n − 1 / 2 + O ( ε K τ ) , \mathrm{SR}_{\tau}(n) \le C_{\mathrm{PATSO}}n^{-1/2} + O\left(\frac{\varepsilon_K}{\tau}\right), SR τ ( n ) ≤ C PATSO n − 1/2 + O ( τ ε K ) , where depth-dependent constants are absorbed into the O ( ⋅ ) O(\cdot) O ( ⋅ ) term.
Proof. By the 1 / τ 1/\tau 1/ τ -Lipschitz property of CVaR in W 1 W_1 W 1 , replacing the uncapped empirical Q-edge law by the capped law changes each empirical CVaR score by at most ε K / τ \varepsilon_K/\tau ε K / τ . Propagating this additive bias through the finite depth tree gives the stated additional term. Combining with the uncapped PATSO bound and using S R τ ( n ) ≤ E τ ( n ) \mathrm{SR}_{\tau}(n) \le\mathcal{E}_{\tau}(n) SR τ ( n ) ≤ E τ ( n ) proves the result.
Memory Capping and W 1 W_1 W 1 Compression
Merge-on-Insert (Precise Algorithm)
We keep a sorted container of pairs { ( Q i , α i ) } i = 1 M \{(Q_i,\alpha_i)\}_{i=1}^{M} {( Q i , α i ) } i = 1 M with M ≤ K M \le K M ≤ K . Upon receiving a new sample z z z :
Match: If min i ∣ z − Q i ∣ ≤ ε t o l \min_i |z-Q_i| \le\varepsilon_{\mathrm{tol}} min i ∣ z − Q i ∣ ≤ ε tol , let i ⋆ = arg min i ∣ z − Q i ∣ i^\star= \arg\min_i |z-Q_i| i ⋆ = arg min i ∣ z − Q i ∣ and set α i ⋆ ← α i ⋆ + 1 \alpha_{i^\star} \leftarrow\alpha_{i^\star}+1 α i ⋆ ← α i ⋆ + 1 .
Insert: Else if M < K M<K M < K , insert ( z , 1 ) (z,1) ( z , 1 ) and keep the list sorted.
Merge: Else (overflow): find j j j with smallest gap ∣ Q j + 1 − Q j ∣ |Q_{j+1}-Q_j| ∣ Q j + 1 − Q j ∣ , replace the pair by
( Q n e w , α n e w ) = ( α j Q j + α j + 1 Q j + 1 α j + α j + 1 , α j + α j + 1 ) , (Q^{\mathrm{new}},\alpha^{\mathrm{new}})=\left(\frac{\alpha_jQ_j+\alpha_{j+1}Q_{j+1}}{\alpha_j+\alpha_{j+1}},\alpha_j+\alpha_{j+1}\right), ( Q new , α new ) = ( α j + α j + 1 α j Q j + α j + 1 Q j + 1 , α j + α j + 1 ) ,
then insert ( z , 1 ) (z,1) ( z , 1 ) in sorted order.
Bound on W 1 W_1 W 1 Error Under Capping
Define range : = max i Q i − min i Q i \operatorname{range}:=\max_i Q_i-\min_i Q_i range := max i Q i − min i Q i and let P ^ \widehat{P} P be the (uncapped) empirical distribution and P ~ \widetilde{P} P the capped distribution after applying the procedure above. With nearest-neighbor merges:
W 1 ( P ^ , P ~ ) ≤ ∑ m e r g e s ( α j + α j + 1 ) ⋅ ∣ Q j + 1 − Q j ∣ = O ( range K ) . W_1(\widehat{P},\widetilde{P})\leq\sum_{\mathrm{merges}}(\alpha_j+\alpha_{j+1})\cdot\lvert Q_{j+1}-Q_j\rvert=O\left(\frac{\operatorname{range}}{K}\right). W 1 ( P , P ) ≤ merges ∑ ( α j + α j + 1 ) ⋅ ∣ Q j + 1 − Q j ∣ = O ( K range ) .
Since the Bellman operator is 1 / ( 1 − γ ) 1/(1-\gamma) 1/ ( 1 − γ ) -Lipschitz in W 1 W_1 W 1 , the root-value error contributed by capping is O ( range / ( K ( 1 − γ ) ) ) O(\operatorname{range}/(K(1-\gamma))) O ( range / ( K ( 1 − γ ))) and the simple-regret additive term is O ( range / ( K ( 1 − γ ) 2 ) ) O(\operatorname{range}/(K(1-\gamma)^2)) O ( range / ( K ( 1 − γ ) 2 )) , which is dominated by the proven O ( n − 1 / 2 ) O(n^{-1/2}) O ( n − 1/2 ) term for moderate K K K .
Experimental Details
All experiments were conducted on the I9 14900K processor, 6.00 GHz, with 24 cores and 32 threads in each core, 62 GB RAM. Whenever feasible, we opted for open-source implementations of algorithms and environments. The tuning process takes around 3 hours and the evaluation takes around 2 hours with the best configuration for each environment.
Parameter Selection
We conduct the hyperparameter-tuning experiments with 10 seeds for each environments. Specifically, we search over the number of atoms N N N in the grid { 10 , 11 , … , 200 } \{10,11,\ldots,200\} { 10 , 11 , … , 200 } , polynomial bonus constant C C C in the set { 1 , 2 , 4 , 8 } \{1,2,4,8\} { 1 , 2 , 4 , 8 } and the CVaR τ \tau τ value in { 0.05 , 0.1 , 0.2 , 0.25 } \{0.05,0.1,0.2,0.25\} { 0.05 , 0.1 , 0.2 , 0.25 } for CATSO and PATSO. For the baselines, we also tune to find the best config in terms of bias c c c , exploration bonus ϵ \epsilon ϵ and the power p p p .
Environments and Algorithms Hyperparameters
For each environment, we run CATSO and PATSO with the baselines UCT, PowerUCT, MaxUCT with 50 seeds, each seed uses 10 4 10^4 1 0 4 to 10 5 10^5 1 0 5 simulations depending on the complexity of the environment and the tree depth is set to equal the environment horizon. At each simulation checkpoint, we compute the regret at the root node and perform 200 Monte Carlo evaluation rollouts at each simulation to compute the average of catastrophic count and CVaR return (worst τ \tau τ -portion of the trajectories in term of return). Below are the detailed environment configurations. For the hyperparameter used in each environment, refer to Table 2 . One important note for CATSO is that, since it is difficult to determine a reasonable heuristic grid for the return distribution in the Q-Edge, we choose to build the distribution adaptively. Specifically, we initialize q min = 0 q_{\min}=0 q m i n = 0 and q max = 0.01 q_{\max}=0.01 q m a x = 0.01 and build the empirical distribution by discretizing the interval [ q min , q max ] [q_{\min},q_{\max}] [ q m i n , q m a x ] into N N N atoms, then the planner will adaptively expand the distribution as it receives more samples. Details of the adaptive distribution construction is in line 9 to 13 of the pseudo-code in figure 5. And for PATSO, as describe in appendix § 13 , we use capped version of the particle distribution for memory capability in practical.
Table 2. Algorithm hyperparameters used for each environment. The value auto means that the code uses USE_AUTO_BIAS and computes the effective bias adaptively at runtime.
Autonomous Vehicle
Task: A route-planning problem where the agent must navigate through a road network to reach a destination. See Figure 4 a for the map.
Goal: Reach the goal while avoiding routes with rare but severe travel-cost outcomes.
Action Space: Move { Up , Down , Left , Right } \{\text{Up}, \text{Down}, \text{Left}, \text{Right}\} { Up , Down , Left , Right } .
State Space: Discrete ( x , y ) (x, y) ( x , y ) position of the agent.
Reward: Traversing the road pays a stochastic cost, based on the road types. Details are provided table 3.
Transition Dynamics: The transition across the road is deterministic, while the cost type is sampled with probability { 0.4 , 0.2 , 0.4 } \{0.4, 0.2, 0.4\} { 0.4 , 0.2 , 0.4 } . The horizon is 7.
Catastrophic Feature: Traversing a high-risk road and receiving the largest cost outcome.
Figure 4. Example risk-sensitive planning environments. Each contains a short risky route and a longer safer route; the risky route can have higher expected return but exposes the agent to catastrophic lower-tail outcomes. (2a) the map has black: highways, red: main roads, blue: streets, green: lanes (2b) S: start cell, G: goal cell, X: cliff cell, wind cell is indicated as the blue cell with an arrow points down. (2c) Red cell: guard cell, green cell: goal cell, gray cell: obstacle.
Two-level Risky Treasure
Task: A small sequential decision problem in which the agent chooses between a safe option and a risky treasure branch, and later decides whether to continue or abort after observing a hint.
Goal: Maximize return by balancing the high upside of the treasure against the risk of failure.
Action Space: At the root, the actions are { safe , risky } \{\text{safe}, \text{risky}\} { safe , risky } ; after entering the risky branch, the actions are { abort , proceed } \{\text{abort}, \text{proceed}\} { abort , proceed } .
State Space: Current decision stage, observed hint, and terminal status.
Reward: The safe branch yields a reward of 2, while the risky branch yields either a high positive reward or a negative failure penalty, namely 500 or -10, respectively. The abort reward is -1.
Transition Dynamics: Choosing the risky branch reveals a noisy hint about the hidden treasure state. The agent then chooses whether to abort safely or proceed to a stochastic success/failure outcome with success probability of 0.40. The horizon is 2.
Catastrophic Feature: Treasure failure after choosing to proceed, i.e., receiving the -10 reward.
Table 3. Cost distribution used in the Autonomous Vehicle environment.
Risky Shortcut Gridworld
Task: A cliff-style gridworld where the agent may choose between a short but dangerous shortcut and a longer safer path. See Figure 4 b for the map.
Goal: Reach the goal while avoiding the cliff.
Action Space: Four movement actions: { Up , Right , Down , Left } \{\text{Up}, \text{Right}, \text{Down}, \text{Left}\} { Up , Right , Down , Left } .
State Space: Grid position of the agent together with the current time step.
Reward: An ordinary step gives reward -1, reaching the goal gives reward 5, and falling into the cliff gives reward -20.
Transition Dynamics: The movement is deterministic, except when the agent is in a windy cell, there is a 2% chance of being dragged down the hill. The horizon is 32.
Catastrophic Feature: Entering a cliff cell.
Betting Game
Task: A finite-horizon gambling problem where the agent repeatedly chooses how much of its current bankroll to bet.
Goal: Maximize return while avoiding ruin.
Action Space: Nine discrete betting actions, where action a ∈ { 0 , … , 8 } a \in\{0,\ldots,8\} a ∈ { 0 , … , 8 } corresponds to betting a fraction a / 8 a/8 a /8 of the current bankroll.
State Space: Current bankroll and current time step, i.e. ( b , t ) (b,t) ( b , t ) .
Reward: Each action yields a stochastic gain or loss determined by the chosen bet fraction and the outcome of the bet.
Transition Dynamics: After each action, the bankroll is updated according to the stochastic bet outcome. The episode terminates when the bankroll reaches zero, reaches the maximum bankroll, or the horizon is exhausted. The horizon is 6.
Catastrophic Feature: Bankruptcy, i.e. the bankroll reaches zero.
Guarded Maze
Task: A grid-navigation problem where the agent must move from a start cell to a goal cell while deciding whether to pass through a risky guarded region.
Goal: Reach the goal with high return while avoiding dangerous guard outcomes.
Action Space: Four movement actions: { Up , Down , Left , Right } \{\text{Up}, \text{Down}, \text{Left}, \text{Right}\} { Up , Down , Left , Right } .
State Space: Agent position on the grid together with the current time step.
Reward: Each step pays -1 reward, when in the guard cell, the reward is sampled in the set { − 60 , − 30 , 0 , 30 , 60 } \{-60,-30,0,30,60\} { − 60 , − 30 , 0 , 30 , 60 } in normal distribution.
Transition Dynamics: Movement is constrained by the maze layout and walls. Entering the guarded region triggers a stochastic outcome, which may produce a large negative reward as described above. The horizon is 20.
Catastrophic Feature: A severe negative guard outcome.
Full Results
For completeness, we show the full the detail experimental results of the experiment, which are already depicted in section 6. Recall that the standard deviation of Two-Level Risky Treasure is zero because the algorithms converge in earlier simulation checkpoint due to its short in horizon and simple transition dynamics.
Sensitivity of CATSO/PATSO Grid Representation
We study the sensitivity of CATSO and PATSO to the size of their distributional representations. Specifically, we vary the number of categorical atoms N N N for CATSO and the maximum number of particles U U U for PATSO. We evaluate the resulting computational cost, memory usage, and planning performance. For comparison, UCT is shown as a scalar baseline. All experiments use the adaptive-grid implementation of CATSO and the capped version of PATSO used in our practical experiments.
Figure 6 shows peak memory and search time as a function of representation size. As expected, increasing N N N for CATSO leads to a higher computational and memory cost, since each Q-edge maintains more categorical atoms and the CVaR computation must process a larger support. Similarly, increasing the particle cap U U U increases the computational cost of PATSO. The effect is particularly pronounced for uncapped PATSO, whose support can grow with the number of distinct backed-up returns. In contrast, capped PATSO provides a bounded representation size and therefore offers a direct trade-off between computational cost and distributional fidelity.
Figure 6. Peak memory and search time of CATSO, PATSO, and UCT as a function of the representation size N N N and U U U .
Figure 7 reports the resulting planning performance. Overall, the performance of CATSO is relatively stable across the tested values of N N N , although very small grids can provide insufficient resolution for the underlying return distribution. PATSO exhibits a similar trend with respect to U U U : increasing the particle cap provides additional capacity to represent the empirical distribution, but the performance gains become smaller once a sufficiently large support is available. These results suggest that moderate representation sizes can retain most of the risk-sensitive planning benefit without incurring the full computational cost of a large representation.
Figure 7. Planning performance of CATSO, PATSO, and UCT as a function of the representation size N N N and U U U .
Finally, we measures the compression error of capped PATSO relative to uncapped PATSO. Figure 8 shows the compression error decreases as the particle cap U U U increases, showing that a larger cap allows the capped representation to more closely match the empirical distribution maintained by uncapped PATSO. This agrees with the theoretical characterization in Remark 5.8, where the compression error contributes an additional approximation term to the CVaR estimate.
Figure 8. Compression quality of capped PATSO relative to uncapped PATSO as a function of the particle cap U U U . 100k trials. Bands / bars: ± 1 \pm1 ± 1 standard error over 50 runs. Uncapped PATSO has zero compression error by definition.
Algorithm 3: CATSO Adaptive SimulateQ
SimulateQ($s_h$, $a$, $t$):
s h + 1 ∼ P ( ⋅ ∣ s h , a ) s_{h+1} \sim P(\cdot\mid s_h, a) s h + 1 ∼ P ( ⋅ ∣ s h , a ) , r t ( s h , a ) ∼ R ( s h , a , s h + 1 ) r_t(s_h, a) \sim R(s_h, a, s_{h+1}) r t ( s h , a ) ∼ R ( s h , a , s h + 1 )
if node s h + 1 s_{h+1} s h + 1 not expanded then
Rollout($s_{h+1}$)
else
SimulateV($s_{h+1}, t)$
T s h , a ( t ) = T s h , a ( t ) + 1 T_{s_h,a}(t) = T_{s_h,a}(t) + 1 T s h , a ( t ) = T s h , a ( t ) + 1
Q ‾ t ( s h , a ) = r t ( s h , a ) + γ V ^ T s h + 1 ( t ) ( s h + 1 ) \overline{Q}_t(s_h, a) = r_t(s_h, a) + \gamma\widehat{V}_{T_{s_{h+1}}(t)}(s_{h+1}) Q t ( s h , a ) = r t ( s h , a ) + γ V T s h + 1 ( t ) ( s h + 1 )
if Q ‾ t ( s h , a ) ∉ [ Q min ( s h , a ) , Q max ( s h , a ) ] \overline{Q}_t(s_h, a) \notin[Q_{\min}(s_h, a), Q_{\max}(s_h, a)] Q t ( s h , a ) ∈ / [ Q m i n ( s h , a ) , Q m a x ( s h , a )] then
z i o l d ( s h , a ) ← z i ( s h , a ) , 0 ≤ i ≤ N z_i^{\mathrm{old}}(s_h, a) \leftarrow z_i(s_h, a), \quad0 \le i \le N z i old ( s h , a ) ← z i ( s h , a ) , 0 ≤ i ≤ N
α i o l d ( s h , a ) ← α i ( s h , a ) , 0 ≤ i ≤ N \alpha_i^{\mathrm{old}}(s_h, a) \leftarrow\alpha_i(s_h, a), \quad0 \le i \le N α i old ( s h , a ) ← α i ( s h , a ) , 0 ≤ i ≤ N
Q max ( s h , a ) = max { Q ‾ t ( s h , a ) , Q max ( s h , a ) } Q_{\max}(s_h, a) = \max\{\overline{Q}_t(s_h, a), Q_{\max}(s_h, a)\} Q m a x ( s h , a ) = max { Q t ( s h , a ) , Q m a x ( s h , a )}
Q min ( s h , a ) = min { Q ‾ t ( s h , a ) , Q min ( s h , a ) } Q_{\min}(s_h, a) = \min\{\overline{Q}_t(s_h, a), Q_{\min}(s_h, a)\} Q m i n ( s h , a ) = min { Q t ( s h , a ) , Q m i n ( s h , a )}
Δ z = ( Q max ( s h , a ) − Q min ( s h , a ) ) / N \Delta z = (Q_{\max}(s_h, a) - Q_{\min}(s_h, a))/N Δ z = ( Q m a x ( s h , a ) − Q m i n ( s h , a )) / N
z i ( s h , a ) = Q min ( s h , a ) + i Δ z , 0 ≤ i ≤ N z_i(s_h, a) = Q_{\min}(s_h, a) + i\Delta z, \quad0 \le i \le N z i ( s h , a ) = Q m i n ( s h , a ) + i Δ z , 0 ≤ i ≤ N
α ~ i ( s h , a ) ← 1 , 0 ≤ i ≤ N \widetilde{\alpha}_i(s_h, a) \leftarrow1, \quad0 \le i \le N α i ( s h , a ) ← 1 , 0 ≤ i ≤ N
for j = 0 , … , N j = 0, \ldots, N j = 0 , … , N do
i ⋆ ← arg min i ∣ z j o l d ( s h , a ) − z i ( s h , a ) ∣ i^\star\leftarrow\arg\min_i |z_j^{\mathrm{old}}(s_h, a) - z_i(s_h, a)| i ⋆ ← arg min i ∣ z j old ( s h , a ) − z i ( s h , a ) ∣
α ~ i ⋆ ( s h , a ) ← α ~ i ⋆ ( s h , a ) + α j o l d ( s h , a ) − 1 \widetilde{\alpha}_{i^\star}(s_h, a) \leftarrow\widetilde{\alpha}_{i^\star}(s_h, a) + \alpha_j^{\mathrm{old}}(s_h, a) - 1 α i ⋆ ( s h , a ) ← α i ⋆ ( s h , a ) + α j old ( s h , a ) − 1
α i ( s h , a ) ← α ~ i ( s h , a ) , 0 ≤ i ≤ N \alpha_i(s_h, a) \leftarrow\widetilde{\alpha}_i(s_h, a), \quad0 \le i \le N α i ( s h , a ) ← α i ( s h , a ) , 0 ≤ i ≤ N
i ← arg min i ∣ Q ‾ t ( s h , a ) − z i ( s h , a ) ∣ i \leftarrow\arg\min_i |\overline{Q}_t(s_h, a) - z_i(s_h, a)| i ← arg min i ∣ Q t ( s h , a ) − z i ( s h , a ) ∣
α i ( s h , a ) ← α i ( s h , a ) + 1 \alpha_i(s_h, a) \leftarrow\alpha_i(s_h, a) + 1 α i ( s h , a ) ← α i ( s h , a ) + 1
Update p ( s h , a ) = [ p 0 ( s h , a ) , … , p N ( s h , a ) ] p(s_h, a) = [p_0(s_h, a), \ldots, p_N(s_h, a)] p ( s h , a ) = [ p 0 ( s h , a ) , … , p N ( s h , a )]
caption: Pseudocode for the adaptive distribution construction in SimulateQ of CATSO.
Overall, the sensitivity results demonstrate the practical trade-off between representation capacity and computational efficiency. CATSO trades a larger categorical grid for finer distributional resolution, while PATSO trades a larger particle cap for lower compression error. In practice, moderate values of N N N and U U U provide a reasonable balance between planning performance, memory usage, and search time.
Limitations
CATSO and PATSO make MCTS risk-sensitive by storing empirical backup laws at Q-edges while keeping V-nodes scalar. This design is lightweight, but it also has several limitations.
CATSO discretization bias. CATSO represents each Q-edge law on a fixed categorical grid. This gives bounded memory and simple Dirichlet updates, but it introduces a grid approximation error. If the grid is too coarse, sharp tail behavior or multimodal return distributions may be poorly represented. Adaptive grids can improve empirical performance, but changing the grid requires reprojection.
PATSO memory growth and compression bias. Uncapped PATSO represents the empirical Q-edge law exactly, but its support can grow linearly with the number of distinct backed-up returns. Capped PATSO controls memory by merging particles, but this introduces a separate W 1 W_1 W 1 compression error. Since CVaR is sensitive to tail mass, careless merging can distort the lower tail; the cap must therefore be chosen large enough that the compression bias is below the statistical error.
Scalability beyond tabular finite-depth search. The present analysis is for tabular finite-depth MCTS. Extending the approach to large or continuous state spaces, learned dynamics, partial observability, or neural value approximators requires additional approximation analysis. In particular, model error and function-approximation error would interact with the Q-edge distributional estimates and the CVaR backup.
NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
Answer: [Yes]
Justification: All the claims made in the abstract and introduction are shown properly in sections 5 and 6.
Guidelines:
The answer [N/A] means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A [No] or [N/A] answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: The limitations are discussed in appendix G.
Guidelines:
The answer [N/A] means that the paper has no limitation while the answer [No] means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate “Limitations” section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren’t acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: All assumptions are stated in section 3, 5 and appendix B. While the complete proofs are in section 5, appendix C, D and E.
Guidelines:
The answer [N/A] means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: Section 6 and Appendix F provide all the information about the experiment procedures, environments, algorithms and their configuration in the experiment.
Guidelines:
The answer [N/A] means that the paper does not include experiments.
If the paper includes experiments, a [No] answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general, releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example (a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully. (c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: An anonymized .zip file containing the code, dependency specifications, and instructions necessary to faithfully reproduce the main experimental results is included in the supplementary material, in accordance with the NeurIPS Code and Data Submission Guidelines.
Guidelines:
The answer [N/A] means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://neurips.cc/public/guides/CodeSubmissionPolicy) for more details.
While we encourage the release of code and data, we understand that this might not be possible, so [No] is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https://neurips.cc/public/guides/CodeSubmissionPolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer) necessary to understand the results?
Answer: [Yes]
Justification: Appendix F clearly defines the environment setups, algorithm hyperparameters for both the main process and tuning process.
Guidelines:
The answer [N/A] means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: The results are averaged over 50 seeds × \times × 200 independent Monte Carlo runs, with standard deviation plotted in Figure 2 and Table 4 and empirical safety verified as detailed in Section 6 and Appendix F.
(a) (a) CVaR value gap across environments vs. simulation budget. Lower is better.
(b) (b) Monte-Carlo CVaR return of the policies recommended by each planner. Higher is better.
Figure 2. CVaR planning performance across environments: value gap during planning (top) and return of recommended policies (bottom) with ± \pm ± standard deviation.
Table 4. Performance across all environments and algorithms at final simulation budget. Best values per environment and metric are shown in bold.
Guidelines:
The answer [N/A] means that the paper does not include experiments.
The authors should answer [Yes] if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g., negative error rates).
If error bars are reported in tables or plots, the authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
Experiments compute resources
Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Answer: [Yes]
Justification: Appendix F explicitly details the compute resources (CPU Intel Core i9 14900K, 62 GB RAM) and the execution times for both the tuning experiment (2 to 4 hours per environment) and the main experiments (1 to 3 hours per environment).
Guidelines:
The answer [N/A] means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn’t make it into the paper).
Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines?
Answer: [Yes]
Justification: The research focuses on foundational theoretical and algorithmic machine learning methodology (Sections 5), which fully conforms to the NeurIPS Code of Ethics.
Guidelines:
The answer [N/A] means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer [No], they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).
Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [N/A]
Justification: This is foundational theoretical research on risk-sensitive planning (Sections 4 and 5) evaluated on synthetic distributions (Section 6) presenting no immediate foreseeable negative societal impacts or direct paths to malicious applications.
Guidelines:
The answer [N/A] means that there is no societal impact of the work performed.
If the authors answer [N/A] or [No], they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate Deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mecha-
nisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pre-trained language models, image generators, or scraped datasets)?
Answer: [N/A]
Justification: The paper does not introduce or release high-risk models, language models, or scraped datasets (all data are simulated as per Appendix § 14 ).
Guidelines:
The answer [N/A] means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.
Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: All the environments represented in the paper, if adapted from existing works, are cited properly in Section 6.
Guidelines:
The answer [N/A] means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset’s creators.
New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: All new environments and algorithms are documented in detail in supplementary and Appendix § 14 .
Guidelines:
The answer [N/A] means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [N/A]
Justification: The paper studies risk awareness in MCTS planning algorithm and does not involve human subjects.
Guidelines:
The answer [N/A] means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [N/A]
Justification: The research utilizes synthetic environments (Appendix § 14 ) and does not involve human subjects.
Guidelines:
The answer [N/A] means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigor, or originality of the research, declaration is not required.
Answer: [N/A]
Justification: LLMs are not used as a component of the core theoretical methodology or algorithm design (sections 4 and 5)
Guidelines:
The answer [N/A] means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy in the NeurIPS handbook for what should or should not be described.