Introduction
Designing statistically and computationally efficient algorithms for reinforcement learning (RL) with function approximation is a central problem in modern statistical learning theory. To handle large or continuous state spaces, a major line of research studies Markov decision processes (MDPs) equipped with linear structure, most notably linear mixture MDPs ([26 , 17 , 3 , 41 , 40 , 39 ] ), where the transition kernel is a linear combination of basis transition models P h ( s ′ ∣ s , a ) = ⟨ ϕ ( s ′ ∣ s , a ) , θ h ∗ ⟩ P_h(s' \mid s,a)=\langle\phi(s' \mid s,a),\theta_h^*\rangle P h ( s ′ ∣ s , a ) = ⟨ ϕ ( s ′ ∣ s , a ) , θ h ∗ ⟩ , and linear MDPs ([30 , 20 , 2 ] (Hu et al. 2022; He et al. 2023)), where both the transition kernel P h ( s ′ ∣ s , a ) = ⟨ ϕ ( s , a ) , d θ h ( s ′ ) ⟩ P_h(s' \mid s,a)=\langle\phi(s,a),d\theta_h(s')\rangle P h ( s ′ ∣ s , a ) = ⟨ ϕ ( s , a ) , d θ h ( s ′ )⟩ and the reward function are linear in a known d d d -dimensional state-action feature map ϕ ( s , a ) \phi(s,a) ϕ ( s , a ) with unknown signed measures θ h ( ⋅ ) \theta_h(\cdot) θ h ( ⋅ ) over the state space S \mathcal{S} S .
While worst-case minimax regret bounds quantify algorithm performance in the hardest environment instances, practical RL environments often exhibit low or even zero transition stochasticity. This observation has motivated a surge of interest in variance-dependent (or second-order) regret bounds ([32 , 41 , 37 , 21 , 39 , 16 , 43 , 10 ] (Zhou and Gu 2022; Li and Sun 2023)), which scale with the transition variance rather than the worst-case horizon bound and ideally interpolate between the O ~ ( K ) \widetilde{O}(\sqrt{K}) O ( K ) minimax rate in stochastic regimes and O ~ ( 1 ) \widetilde{O}(1) O ( 1 ) (polylogarithmic in K K K ) regret in deterministic regimes. For time-homogeneous linear mixture MDPs under ∑ h = 1 H r h ≤ 1 \sum_{h=1}^{H}r_h\leq1 ∑ h = 1 H r h ≤ 1 , Zhao et al. [39 ] proposed UCRL-AVE, achieving a variance-dependent regret of O ~ ( d Var K ∗ + d 2 ) \widetilde{O}(d\sqrt{\operatorname{Var}^{*}_{K}}+d^2) O ( d Var K ∗ + d 2 ) without prior knowledge of the transition variances.
From a statistical perspective, however, linear MDPs are substantially more challenging to analyze than linear mixture MDPs. In linear mixture MDPs, for any next-stage value function V k , h + 1 V_{k,h+1} V k , h + 1 , the integrated feature vector ϕ V k , h + 1 ( s , a ) = ∑ s ′ ∈ S ϕ ( s ′ ∣ s , a ) V k , h + 1 ( s ′ ) \phi_{V_{k,h+1}}(s,a)=\sum_{s'\in\mathcal{S}}\phi(s'\mid s,a)V_{k,h+1}(s') ϕ V k , h + 1 ( s , a ) = ∑ s ′ ∈ S ϕ ( s ′ ∣ s , a ) V k , h + 1 ( s ′ ) is explicitly computable by the learner, and the regression target parameter θ h ∗ ∈ R d \theta_h^*\in\mathbb{R}^d θ h ∗ ∈ R d is fixed and independent of V k , h + 1 V_{k,h+1} V k , h + 1 . Exploiting this fixed-target structure, Zhao et al. [39 ] directly applied self-normalized martingale concentration to the single target vector θ h ∗ \theta_h^* θ h ∗ without requiring an ϵ \epsilon ϵ -net covering over the value function class. In contrast, in linear MDPs, the feature vector ϕ ( s h i , a h i ) \phi(s_h^i,a_h^i) ϕ ( s h i , a h i ) is fixed while the true Bellman backup parameter w h ( V k , h + 1 ) = ∫ S V k , h + 1 ( s ′ ) d θ h ( s ′ ) w_h(V_{k,h+1})=\int_{\mathcal{S}}V_{k,h+1}(s')d\theta_h(s') w h ( V k , h + 1 ) = ∫ S V k , h + 1 ( s ′ ) d θ h ( s ′ ) depends on the random, data-dependent value function V k , h + 1 V_{k,h+1} V k , h + 1 constructed from later stages. This statistical dependency breaks martingale adaptedness and necessitates a uniform convergence (ϵ \epsilon ϵ -net covering) argument over the value function class ([20 ] ). Because the optimistic value function class has log-covering number O ~ ( d 2 ) \widetilde{O}(d^2) O ( d 2 ) (or O ~ ( d 3 H ) \widetilde{O}(d^3H) O ( d 3 H ) under rare-switching updates), naive uniform convergence inflates the confidence radius by d \sqrt{d} d , leading to the suboptimal O ~ ( d 3 H 4 K ) \widetilde{O}(\sqrt{d^3H^4K}) O ( d 3 H 4 K ) regret of LSVI-UCB ([20 ] ).
Recently, He et al. (2023) (concurrently with Agarwal et al. [2 ] ) resolved the worst-case minimax optimality for linear MDPs by proposing LSVI-UCB++, which attains O ~ ( d H 3 K + d 7 H 8 ) \widetilde{O}(d\sqrt{H^3K}+d^7H^8) O ( d H 3 K + d 7 H 8 ) regret via variance-weighted ridge regression, reference-advantage decomposition V k , h + 1 = V h + 1 ∗ + ( V k , h + 1 − V h + 1 ∗ ) V_{k,h+1}=V^*_{h+1}+(V_{k,h+1}-V^*_{h+1}) V k , h + 1 = V h + 1 ∗ + ( V k , h + 1 − V h + 1 ∗ ) , pointwise monotonic value updates, and rare-switching policy updates. Yet, despite employing variance-weighted regression, the guarantee of LSVI-UCB++ is purely worst-case and does not exploit low or zero transition variance: even in a deterministic linear MDP where all transition variances vanish, the regret bound of He et al. (2023) remains O ~ ( d H 3 K + d 7 H 8 ) \widetilde{O}(d\sqrt{H^3K}+d^7H^8) O ( d H 3 K + d 7 H 8 ) . Specifically, the analysis of He et al. (2023) relies on (i) a coarse, variance-oblivious concentration bound over the full function classes with initial confidence radius scaling inversely with the variance floor σ min \sigma_{\min} σ m i n (with σ min = H \sigma_{\min}=\sqrt{H} σ m i n = H used in the coarse analysis, yielding large radii β ˉ = O ~ ( d 3 H 2 ) \bar{\beta}=\widetilde{O}(\sqrt{d^3H^2}) β ˉ = O ( d 3 H 2 ) and β ~ = O ~ ( d 3 H 2 ) \widetilde{\beta}=\widetilde{O}(\sqrt{d^3H^2}) β = O ( d 3 H 2 ) for the pessimistic and second-moment estimators), and (ii) Azuma–Hoeffding inequalities on the transition martingale differences of V k , h + 1 − V h + 1 π k V_{k,h+1}-V^{\pi_k}_{h+1} V k , h + 1 − V h + 1 π k and V k , h + 1 − V ~ k , h + 1 V_{k,h+1}-\widetilde{V}_{k,h+1} V k , h + 1 − V k , h + 1 that inject unconditional H 3 K \sqrt{H^3K} H 3 K and H 5 K \sqrt{H^5K} H 5 K terms. Subsequent works by Li and Sun (2023) (VARA) and Huang et al. [16 ] (HEAVY-LSVI-UCB) replaced the Azuma–Hoeffding step with Freedman-type arguments to obtain variance-aware leading terms O ~ ( d H K σ ∗ ) \widetilde{O}(d\sqrt{HK\sigma^*}) O ( d H K σ ∗ ) and O ~ ( d H K γ ∗ ) \widetilde{O}(d\sqrt{HK\gamma^*}) O ( d H K γ ∗ ) (where G ∗ , V ∗ G^{*},\mathcal{V}^{*} G ∗ , V ∗ denote expected occupancy-measure variances), but both still retained the coarse first-stage confidence radius β 0 = O ~ ( H d 3 H / σ min ) \beta_{0}=\widetilde{O}(H\sqrt{d^{3}H}/\sigma_{\min}) β 0 = O ( H d 3 H / σ m i n ) (and H β 0 = O ~ ( H 2 d 3 H / σ min ) H\beta_{0}=\widetilde{O}(H^{2}\sqrt{d^{3}H}/\sigma_{\min}) H β 0 = O ( H 2 d 3 H / σ m i n ) for the second-moment target); trading off σ min ∝ K − 1 / 4 \sigma_{\min}\propto K^{-1/4} σ m i n ∝ K − 1/4 in their regret upper bounds leaves an unconditional O ~ ( K 1 / 4 ) \widetilde{O}(K^{1/4}) O ( K 1/4 ) polynomial additive term even in deterministic MDPs. Thus, a natural question remains open:
Can we design a computationally efficient algorithm for linear MDPs that achieves \textit{Can we design a computationally efficient algorithm for linear MDPs that achieves} Can we design a computationally efficient algorithm for linear MDPs that achieves
variance-dependent regret adapting to the realized trajectory variance min { Var K ∗ , Var K π } with a \textit{variance-dependent regret adapting to the realized trajectory variance }\min\{\operatorname{Var}_{K}^{*},\operatorname{Var}_{K}^{\pi}\}\textit{ with a} variance-dependent regret adapting to the realized trajectory variance min { Var K ∗ , Var K π } with a
K -independent (up to polylogarithmic factors) lower-order remainder? \textit{\(K\)-independent (up to polylogarithmic factors) lower-order remainder?} K -independent (up to polylogarithmic factors) lower-order remainder?
Main contributions. In this paper, we answer the open question above affirmatively by proposing VA-LSVI-UCB++ (Algorithm 1), a computationally efficient algorithm for episodic time-inhomogeneous linear MDPs. A detailed comparison with prior results for time-inhomogeneous MDPs is provided in Table 1 (with a companion comparison for time-homogeneous horizon-free RL in Appendix § 12 and Table 2 ). Our main contributions are summarized as follows:
Table 1. Comparison of regret bounds for episodic time-inhomogeneous RL with linear function approximation under per-step reward r h ∈ [ 0 , 1 ] r_{h}\in[0,1] r h ∈ [ 0 , 1 ] . Regret upper bounds are shown up to universal constants and polylogarithmic factors. Here d d d is the feature dimension, H H H is the episode horizon, K K K is the number of episodes, V t o t a l ∗ = ∑ k = 1 K V 1 ∗ ( s 1 k ) V_{\mathrm{total}}^{*}=\sum_{k=1}^{K}V_{1}^{*}(s_{1}^{k}) V total ∗ = ∑ k = 1 K V 1 ∗ ( s 1 k ) is the cumulative optimal value, G ∗ \mathcal{G}^{*} G ∗ and V ∗ \mathcal{V}^{*} V ∗ are expected occupancy-measure variances [16 ] (Li and Sun 2023), and Var K ∗ , Var K π \operatorname{Var}_{K}^{*},\operatorname{Var}_{K}^{\pi} Var K ∗ , Var K π are the realized trajectory-dependent cumulative variances defined in (1) . For VARA (Li and Sun, 2023, Theorem 4.1) and HEAVY-LSVI-UCB [16 Theorem F.13] , bounds are shown adapted to known deterministic rewards r h ∈ [ 0 , 1 ] r_{h}\in[0,1] r h ∈ [ 0 , 1 ] (H = H \mathcal{H}=H H = H , zero reward noise) and optimized over σ min \sigma_{\min} σ m i n . For time-homogeneous horizon-free results under ∑ h = 1 H r h ≤ 1 \sum_{h=1}^{H}r_{h}\leq1 ∑ h = 1 H r h ≤ 1 , see Table 2 in Appendix § 12 .
Table 2. Comparison of regret bounds for episodic time-homogeneous RL with linear function approximation under bounded total reward ∑ h = 1 H r h ≤ 1 \sum_{h=1}^{H}r_h\le1 ∑ h = 1 H r h ≤ 1 . Regret upper bounds are shown up to universal constants and polylogarithmic factors in d , H , K , 1 / δ d,H,K,1/\delta d , H , K , 1/ δ . Here Var K ∗ = ∑ k = 1 K ∑ h = 1 H [ V V h + 1 ∗ ] ( s h k , a h k ) \operatorname{Var}^{*}_{K}=\sum_{k=1}^{K}\sum_{h=1}^{H}[\mathbb{V}V^{*}_{h+1}](s_h^k,a_h^k) Var K ∗ = ∑ k = 1 K ∑ h = 1 H [ V V h + 1 ∗ ] ( s h k , a h k ) denotes the realized cumulative trajectory variance of the optimal value function.
Variance-dependent regret with worst-case minimax optimal leading term. We prove that for any episodic linear MDP with r h ∈ [ 0 , 1 ] r_{h}\in[0,1] r h ∈ [ 0 , 1 ] , VA-LSVI-UCB++ achieves high-
probability regret
Regret ( K ) = O ~ ( d H ⋅ min { Var K ∗ , Var K π } + d 5 H 4 ) \operatorname{Regret}(K)=\widetilde{O}\left(d\sqrt{H\cdot\min\left\{\operatorname{Var}^{*}_{K},\operatorname{Var}^{\pi}_{K}\right\}}+d^{5}H^{4}\right) Regret ( K ) = O ( d H ⋅ min { Var K ∗ , Var K π } + d 5 H 4 )
while performing only O ( d H log K ) O(dH\log K) O ( d H log K ) policy updates and running in O ~ ( d 4 H 3 ∣ A ∣ K ) \widetilde{O}(d^{4}H^{3}|A|K) O ( d 4 H 3 ∣ A ∣ K ) total time (Theorem 5.1 ). Here Var K ∗ = ∑ k , h [ V h V h + 1 ∗ ] ( s h k , a h k ) \operatorname{Var}^{*}_{K}=\sum_{k,h}\left[\mathbb{V}_{h}V^{*}_{h+1}\right](s^{k}_{h},a^{k}_{h}) Var K ∗ = ∑ k , h [ V h V h + 1 ∗ ] ( s h k , a h k ) and Var K π = ∑ k , h [ V h V h + 1 π k ] ( s h k , a h k ) \operatorname{Var}^{\pi}_{K}=\sum_{k,h}\left[\mathbb{V}_{h}V^{\pi^{k}}_{h+1}\right](s^{k}_{h},a^{k}_{h}) Var K π = ∑ k , h [ V h V h + 1 π k ] ( s h k , a h k ) denote the realized trajectory-dependent cumulative variances along the learner’s state-action trajectory (defined in (1) ). Our bound adapts to min { Var K ∗ , Var K π } \min\{\operatorname{Var}^{*}_{K},\operatorname{Var}^{\pi}_{K}\} min { Var K ∗ , Var K π } , recovering the worst-case minimax optimal leading term O ~ ( d H 3 K ) \widetilde{O}(d\sqrt{H^{3}K}) O ( d H 3 K ) that matches the Ω ( d H 3 K ) \Omega(d\sqrt{H^{3}K}) Ω ( d H 3 K ) lower bound of Zhou et al. [41 ] and He et al. (2023) up to logarithmic factors. In deterministic MDPs where Var K ∗ = Var K π = 0 \operatorname{Var}^{*}_{K}=\operatorname{Var}^{\pi}_{K}=0 Var K ∗ = Var K π = 0 , our regret bound collapses to O ~ ( d 5 H 4 ) \widetilde{O}(d^{5}H^{4}) O ( d 5 H 4 ) , eliminating the O ~ ( K 1 / 4 ) \widetilde{O}(K^{1/4}) O ( K 1/4 ) polynomial additive term of Li and Sun (2023) and Huang et al. [16 ] while improving the burn-in overhead of LSVI-UCB++ (He et al. (2023)) from O ~ ( d 7 H 8 ) \widetilde{O}(d^{7}H^{8}) O ( d 7 H 8 ) to O ~ ( d 5 H 4 ) \widetilde{O}(d^{5}H^{4}) O ( d 5 H 4 ) .
Three-way reference-advantage decomposition and double induction. To eliminate the coarse Hoeffding concentration step in He et al. (2023) (which required a constant floor σ k , h 2 ≥ H \sigma^{2}_{k,h}\ge H σ k , h 2 ≥ H and inflated the pessimistic and second-moment confidence radii to β ˉ = O ~ ( d 3 H 2 ) \bar{\beta}=\widetilde{O}(\sqrt{d^{3}H^{2}}) β ˉ = O ( d 3 H 2 ) and β ~ = O ~ ( d 3 H 4 ) \widetilde{\beta}=\widetilde{O}(\sqrt{d^{3}H^{4}}) β = O ( d 3 H 4 ) ), we apply reference-advantage decompositions to all three regression targets simultaneously:
V k , h + 1 = V h + 1 ∗ + ( V k , h + 1 − V h + 1 ∗ ) , V ~ k , h + 1 = V h + 1 ∗ − ( V h + 1 ∗ − V ~ k , h + 1 ) , V_{k,h+1}=V^{*}_{h+1}+\left(V_{k,h+1}-V^{*}_{h+1}\right),\qquad\widetilde{V}_{k,h+1}=V^{*}_{h+1}-\left(V^{*}_{h+1}-\widetilde{V}_{k,h+1}\right), V k , h + 1 = V h + 1 ∗ + ( V k , h + 1 − V h + 1 ∗ ) , V k , h + 1 = V h + 1 ∗ − ( V h + 1 ∗ − V k , h + 1 ) ,
V k , h + 1 2 = ( V h + 1 ∗ ) 2 + ( ( V k , h + 1 2 − ( V h + 1 ∗ ) 2 ) ) . V^{2}_{k,h+1}=\left(V^{*}_{h+1}\right)^{2}+\left(\left(V^{2}_{k,h+1}-\left(V^{*}_{h+1}\right)^{2}\right)\right). V k , h + 1 2 = ( V h + 1 ∗ ) 2 + ( ( V k , h + 1 2 − ( V h + 1 ∗ ) 2 ) ) .
Combining the conditional variance bound [ V h ( V h + 1 ∗ ) 2 ] ≤ 4 H 2 [ V h V h + 1 ∗ ] \left[\mathbb{V}_{h}(V^{*}_{h+1})^{2}\right]\le4H^{2}\left[\mathbb{V}_{h}V^{*}_{h+1}\right] [ V h ( V h + 1 ∗ ) 2 ] ≤ 4 H 2 [ V h V h + 1 ∗ ] (Chen et al. [7 , 8 ] ; Foster et al. [10 ] ) with the pointwise monotonicity of V k , h + 1 V_{k,h+1} V k , h + 1 and V ~ k , h + 1 \widetilde{V}_{k,h+1} V k , h + 1 (Azar et al. [5 ] ; He et al. (2023)), we establish variance-normalized Bernstein concentration for all three estimators on a single unconditional high-probability event E c o n c \mathcal{E}_{\mathrm{conc}} E conc via a deterministic double induction over h = H → 1 h=H\to1 h = H → 1 and k = 1 → K k=1\to K k = 1 → K (Lemma A.5 ). This shaves a factor of O ~ ( d H ) \widetilde{O}(dH) O ( d H ) off both β ˉ \bar{\beta} β ˉ and β ~ \widetilde{\beta} β , reducing them to O ~ ( d ) \widetilde{O}(\sqrt{d}) O ( d ) and O ~ ( H d ) \widetilde{O}(H\sqrt{d}) O ( H d ) , and allows setting the variance floor to α 2 = H 2 / K \alpha^{2}=H^{2}/K α 2 = H 2 / K .
Freedman-type analysis for value differences and trajectory variance. Because the optimistic-to-played gap V k , h + 1 − V h + 1 π k ∈ [ 0 , H ] V_{k,h+1}-V^{\pi^{k}}_{h+1}\in[0,H] V k , h + 1 − V h + 1 π k ∈ [ 0 , H ] and the optimistic-to-pessimistic gap V k , h + 1 − V ~ k , h + 1 ∈ [ 0 , H ] V_{k,h+1}-\widetilde{V}_{k,h+1}\in[0,H] V k , h + 1 − V k , h + 1 ∈ [ 0 , H ] are non-negative under optimism and pessimism, their conditional variances are self-bounded by H [ P h ( V k , h + 1 − V h + 1 π k ) ] H[\mathbb{P}_{h}(V_{k,h+1}-V^{\pi^{k}}_{h+1})] H [ P h ( V k , h + 1 − V h + 1 π k )] and H [ P h ( V k , h + 1 − V ~ k , h + 1 ) ] H[\mathbb{P}_{h}(V_{k,h+1}-\widetilde{V}_{k,h+1})] H [ P h ( V k , h + 1 − V k , h + 1 )] . By applying a time-uniform Freedman inequality and absorbing the conditional variance sum into the left-hand side (Lemma B.2), we completely eliminate the unconditional H 3 K \sqrt{H^{3}K} H 3 K and H 5 K \sqrt{H^{5}K} H 5 K Azuma–Hoeffding errors in He et al. (2023). Using a related variance self-bounding argument in the spirit of Chen et al. [7 , 8 ] and Foster et al. [10 ] , we also establish an immediate first-order regret bound of O ~ ( d H 2 V t o t a l ∗ + d 5 H 4 ) \widetilde{O}(d\sqrt{H^{2}V^{*}_{\mathrm{total}}+d^{5}H^{4}}) O ( d H 2 V total ∗ + d 5 H 4 ) (Corollary 5.5).
Notation. We use lowercase letters to denote scalars, and boldface lowercase and uppercase letters to denote vectors and matrices, respectively. For a positive integer n ∈ N + n\in\mathbb{N}_{+} n ∈ N + , we denote by [ n ] [n] [ n ] the set { 1 , … , n } \{1,\ldots,n\} { 1 , … , n } . For a vector x ∈ R d \mathbf{x}\in\mathbb{R}^{d} x ∈ R d and a positive semi-definite matrix Σ ∈ R d × d \boldsymbol{\Sigma}\in\mathbb{R}^{d\times d} Σ ∈ R d × d , we denote by ∥ x ∥ 2 \|\mathbf{x}\|_{2} ∥ x ∥ 2 the Euclidean norm and define ∥ x ∥ Σ : = x T Σ x \|\mathbf{x}\|_{\boldsymbol{\Sigma}}:=\sqrt{\mathbf{x}^{\mathsf{T}}\boldsymbol{\Sigma}\mathbf{x}} ∥ x ∥ Σ := x T Σ x . For a matrix Γ ∈ R d × d \boldsymbol{\Gamma}\in\mathbb{R}^{d\times d} Γ ∈ R d × d , ∥ Γ ∥ F \|\boldsymbol{\Gamma}\|_{F} ∥ Γ ∥ F denotes its Frobenius norm, and A ⪰ B \mathbf{A}\succeq\mathbf{B} A ⪰ B (resp. A ⪯ B \mathbf{A}\preceq\mathbf{B} A ⪯ B ) means that A − B \mathbf{A}-\mathbf{B} A − B is positive (resp. negative) semi-definite. For any real function f : S → R f:S\to\mathbb{R} f : S → R , we define ∥ f ∥ ∞ : = sup s ∈ S ∣ f ( s ) ∣ \|f\|_{\infty}:=\sup_{s\in S}|f(s)| ∥ f ∥ ∞ := sup s ∈ S ∣ f ( s ) ∣ . We use { 1 } \{1\} { 1 } to denote the indicator function. For x ∈ R x \in\mathbb{R} x ∈ R and a ≤ b a \le b a ≤ b , we define the positive-part operator ( x ) + : = max { x , 0 } (x)_{+} := \max\{x,0\} ( x ) + := max { x , 0 } and the truncation operator [ x ] [ a , b ] : = min { max { x , a } , b } [x]_{[a,b]} := \min\{\max\{x,a\},b\} [ x ] [ a , b ] := min { max { x , a } , b } . For two positive sequences { a n } \{a_n\} { a n } and { b n } \{b_n\} { b n } , we write a n = O ( b n ) a_n = O(b_n) a n = O ( b n ) (resp. a n = Ω ( b n ) a_n = \Omega(b_n) a n = Ω ( b n ) ) if there exists an absolute constant C > 0 C > 0 C > 0 such that a n ≤ C b n a_n \le Cb_n a n ≤ C b n (resp. a n ≥ C b n a_n \ge Cb_n a n ≥ C b n ), and use O ~ ( ⋅ ) \widetilde{O}(\cdot) O ( ⋅ ) and Ω ~ ( ⋅ ) \widetilde{\Omega}(\cdot) Ω ( ⋅ ) to hide polylogarithmic factors in d d d , H H H , K K K , 1 / δ 1/\delta 1/ δ .
Related Work
RL with linear function approximation. A rich body of literature has studied provably efficient RL with linear function approximation under various structural assumptions, including low Bellman rank [18 , 9 ] , low inherent Bellman error [33 ] , linear mixture MDPs (Modi et al., 2020; Jia et al., 2020; Ayoub et al., 2020; Zhou et al., 2021b,a; Zhou and Gu, 2022; Zhao et al., 2023), and linear MDPs [30 , 20 , 31 , 29 , 27 , 2 , 16 ] (He et al. 2021; Hu et al. 2022; He et al. 2023; Li and Sun 2023). For episodic linear MDPs, Jin et al. [20 ] proposed LSVI-UCB and established an O ~ ( d 3 H 4 K ) \widetilde{O}(\sqrt{d^3H^4K}) O ( d 3 H 4 K ) regret bound. To close the d H \sqrt{dH} d H gap to the Ω ( d H 3 K ) \Omega(d\sqrt{H^3K}) Ω ( d H 3 K ) lower bound [41 ] , Hu et al. (2022) attempted to combine reference-advantage decomposition with an over-optimistic value function sequence, which was subsequently found to contain a technical flaw in its covering argument (He et al., 2023, Appendix A). He et al. (2023) and Agarwal et al. [2 ] independently achieved the worst-case minimax optimal O ~ ( d H 3 K ) \widetilde{O}(d\sqrt{H^3K}) O ( d H 3 K ) leading rate via LSVI-UCB++ and VOQL, respectively. Building on LSVI-UCB++, Li and Sun (2023) (VARA) and Huang et al. [16 ] (HEAVY-LSVI-UCB) studied variance-and central-moment-aware regret in linear MDPs with heavy-tailed rewards; however, because their transition analyses still rely on a coarse concentration stage whose radius scales inversely with the variance floor σ min \sigma_{\min} σ m i n , trading off σ min ∝ K − 1 / 4 \sigma_{\min} \propto K^{-1/4} σ m i n ∝ K − 1/4 introduces an unconditional O ~ ( K 1 / 4 ) \widetilde{O}(K^{1/4}) O ( K 1/4 ) polynomial regret overhead even in deterministic MDPs.
Variance-aware bandits and problem-dependent RL. Exploiting empirical variance via Bernstein- or Freedman-type concentration inequalities has been pivotal for shaving horizon factors in tabular RL. Pioneered by Lattimore and Hutter [23 ] and Azar et al. [4 , 5 ] , empirical Bernstein bonuses combined with the Law of Total Variance enabled UCBVI [5 ] and UCB-Advantage [38 ] to achieve minimax optimal O ~ ( H 2 S A K ) \widetilde{O}(\sqrt{H^2SAK}) O ( H 2 S A K ) and O ~ ( H 3 S A K ) \widetilde{O}(\sqrt{H^3SAK}) O ( H 3 S A K ) regret in time-homogeneous and time-inhomogeneous tabular MDPs, respectively, and subsequently enabled horizon-free regret bounds under ∑ h = 1 H r h ≤ 1 \sum_{h=1}^{H} r^h \le1 ∑ h = 1 H r h ≤ 1 [28 , 34 , 35 ] (Li et al. 2022). In parallel, self-bounding Freedman-type arguments coupled with squared-value variance bounds (Var ( X 2 ) ≤ 4 ∥ X ∥ ∞ 2 Var ( X ) \operatorname{Var}(X^2) \le4\lVert X\rVert_{\infty}^2 \operatorname{Var}(X) Var ( X 2 ) ≤ 4 ∥ X ∥ ∞ 2 Var ( X ) ) have played a central role in controlling trajectory variances in stochastic shortest path (SSP) problems [7 , 8 ] , problem-dependent tabular RL [32 , 43 ] , and horizon-independent imitation learning [10 ] .
In linear bandits with heteroscedastic noise of variance σ k 2 \sigma_k^2 σ k 2 , Kirschner and Krause [22 ] and Zhou et al. [41 ] studied known-variance weighted ridge regression, and Zhou and Gu (2022) introduced variance- and uncertainty-aware weights to achieve O ~ ( d ∑ k = 1 K σ k 2 + d ) \widetilde{O}(d\sqrt{\sum_{k=1}^{K}\sigma_k^2+d}) O ( d ∑ k = 1 K σ k 2 + d ) regret (a weighting mechanism also leveraged in corruption-robust linear bandits by He et al. (2022)). For unknown noise variances, Zhang et al. [37 ] and Kim et al. [21 ] developed statistically near-optimal but computationally inefficient algorithms (VOFUL/VOFUL2) over non-convex second-order confidence sets, which they extended to horizon-free linear mixture MDPs (VARLin/VARLin2). Zhou and Gu [40 ] achieved the first computationally efficient horizon-free O ~ ( d K + d 2 ) \widetilde{O}(d\sqrt{K}+d^2) O ( d K + d 2 ) regret in time-homogeneous linear mixture MDPs via higher-order moment estimators (HF-UCRL-VTr+), and Zhao et al. [39 ] further achieved variance-dependent O ~ ( d Var K ∗ + d 2 ) \widetilde{O}(d\sqrt{\operatorname{Var}^{*}_{K}}+d^{2}) O ( d Var K ∗ + d 2 ) regret via multi-layer stratification (UCRL-AVE). For linear MDPs, Wagenmaker et al. [27 ] obtained a computationally inefficient first-order bound O ~ ( d 3 H 3 V t o t a l ∗ + d 3.5 H 3 ) \widetilde{O}(\sqrt{d^{3}H^{3}V_{\mathrm{total}}^{*}}+d^{3.5}H^{3}) O ( d 3 H 3 V total ∗ + d 3.5 H 3 ) , Chen et al. [8 ] designed an inefficient global-optimization algorithm for linear SSP, Zhang et al. [36 ] established a computationally inefficient worst-case horizon-free bound O ~ ( d 5.5 K + d 6.5 ) \widetilde{O}(d^{5.5}\sqrt{K}+d^{6.5}) O ( d 5.5 K + d 6.5 ) for time-homogeneous linear MDPs under ∑ h = 1 H r h ≤ 1 \sum_{h=1}^{H}r_{h}\le1 ∑ h = 1 H r h ≤ 1 (see Appendix § 12 ), and Li and Sun (2023) and Huang et al. [16 ] obtained variance- and first-order bounds for time-inhomogeneous linear MDPs with O ~ ( K 1 / 4 ) \widetilde{O}(K^{1/4}) O ( K 1/4 ) lower-order overhead. Our VA-LSVI-UCB++ algorithm simultaneously achieves computational efficiency, adaptation to the realized trajectory variance min { Var K ∗ , Var K π } \min\{\operatorname{Var}^{*}_{K},\operatorname{Var}^{\pi}_{K}\} min { Var K ∗ , Var K π } , and a lower-order remainder O ~ ( d 5 H 4 ) \widetilde{O}(d^{5}H^{4}) O ( d 5 H 4 ) free of polynomial-in-K K K dependency.
Preliminaries
Episodic time-inhomogeneous MDPs. We consider an episodic time-inhomogeneous MDP denoted by a tuple M = ( S , A , H , { P h } h = 1 H , { r h } h = 1 H ) \mathcal{M}=(\mathcal{S},\mathcal{A},H,\{\mathbb{P}_{h}\}_{h=1}^{H},\{r_{h}\}_{h=1}^{H}) M = ( S , A , H , { P h } h = 1 H , { r h } h = 1 H ) , where S \mathcal{S} S is a measurable state space (possibly infinite), A \mathcal{A} A is a finite action space, H ∈ N + H\in\mathbb{N}_{+} H ∈ N + is the episode horizon, P h ( ⋅ ∣ s , a ) \mathbb{P}_{h}(\cdot\mid s,a) P h ( ⋅ ∣ s , a ) is the transition probability kernel at stage h ∈ [ H ] h\in[H] h ∈ [ H ] , and r h : S × A → [ 0 , 1 ] r_{h}:\mathcal{S}\times\mathcal{A}\to[0,1] r h : S × A → [ 0 , 1 ] is the deterministic reward function at stage h ∈ [ H ] h\in[H] h ∈ [ H ] .1 A deterministic Markov policy π = { π h } h = 1 H \pi=\{\pi_{h}\}_{h=1}^{H} π = { π h } h = 1 H is a sequence of H H H mappings π h : S → A \pi_{h}:\mathcal{S}\to\mathcal{A} π h : S → A .
For any policy π \pi π and stage h ∈ [ H ] h\in[H] h ∈ [ H ] , the state value function V h π : S → [ 0 , H ] V_{h}^{\pi}:\mathcal{S}\to[0,H] V h π : S → [ 0 , H ] and action-value function Q h π : S × A → [ 0 , H ] Q_{h}^{\pi}:\mathcal{S}\times\mathcal{A}\to[0,H] Q h π : S × A → [ 0 , H ] are defined as the expected cumulative rewards starting from stage h h h :
Q h π ( s , a ) : = r h ( s , a ) + E [ ∑ h ′ = h + 1 H r h ′ ( s h ′ , π h ′ ( s h ′ ) ) ∣ s h = s , a h = a ] , V h π ( s ) : = Q h π ( s , π h ( s ) ) , Q_{h}^{\pi}(s,a):=r_{h}(s,a)+\mathbb{E}\left[\left.\sum_{h'=h+1}^{H}r_{h'}(s_{h'},\pi_{h'}(s_{h'}))\right|s_{h}=s,a_{h}=a\right],\qquad V_{h}^{\pi}(s):=Q_{h}^{\pi}(s,\pi_{h}(s)), Q h π ( s , a ) := r h ( s , a ) + E [ h ′ = h + 1 ∑ H r h ′ ( s h ′ , π h ′ ( s h ′ )) s h = s , a h = a ] , V h π ( s ) := Q h π ( s , π h ( s )) ,
with boundary condition V H + 1 π ( s ) : = 0 V_{H+1}^{\pi}(s):=0 V H + 1 π ( s ) := 0 for all s ∈ S s\in\mathcal{S} s ∈ S . For any measurable function V : S → R V:\mathcal{S}\to\mathbb{R} V : S → R , we define the one-step conditional expectation operator P h \mathbb{P}_{h} P h and conditional variance operator V h \mathbb{V}_{h} V h at stage h ∈ [ H ] h\in[H] h ∈ [ H ] by
[ P h V ] ( s , a ) : = E s ′ ∼ P h ( ⋅ ∣ s , a ) [ V ( s ′ ) ] , [ V h V ] ( s , a ) : = [ P h V 2 ] ( s , a ) − ( [ P h V ] ( s , a ) ) 2 , [\mathbb{P}_{h}V](s,a):=\mathbb{E}_{s'\sim\mathbb{P}_{h}(\cdot\mid s,a)}[V(s')],\qquad[\mathbb{V}_{h}V](s,a):=[\mathbb{P}_{h}V^{2}](s,a)-([\mathbb{P}_{h}V](s,a))^{2}, [ P h V ] ( s , a ) := E s ′ ∼ P h ( ⋅ ∣ s , a ) [ V ( s ′ )] , [ V h V ] ( s , a ) := [ P h V 2 ] ( s , a ) − ([ P h V ] ( s , a ) ) 2 ,
where V 2 V^{2} V 2 denotes the pointwise squared function s ↦ ( V ( s ) ) 2 s\mapsto(V(s))^{2} s ↦ ( V ( s ) ) 2 . The optimal action-value function Q h ∗ ( s , a ) : = sup π Q h π ( s , a ) Q_{h}^{*}(s,a):=\sup_{\pi}Q_{h}^{\pi}(s,a) Q h ∗ ( s , a ) := sup π Q h π ( s , a ) and optimal value function V h ∗ ( s ) : = sup π V h π ( s ) V_{h}^{*}(s):=\sup_{\pi}V_{h}^{\pi}(s) V h ∗ ( s ) := sup π V h π ( s ) satisfy the Bellman equations:
Q h π ( s , a ) = r h ( s , a ) + [ P h V h + 1 π ] ( s , a ) , Q h ∗ ( s , a ) = r h ( s , a ) + [ P h V h + 1 ∗ ] ( s , a ) , V h ∗ ( s ) = max a ∈ A Q h ∗ ( s , a ) , Q_{h}^{\pi}(s,a)=r_{h}(s,a)+[\mathbb{P}_{h}V_{h+1}^{\pi}](s,a),\qquad Q_{h}^{*}(s,a)=r_{h}(s,a)+[\mathbb{P}_{h}V_{h+1}^{*}](s,a),\qquad V_{h}^{*}(s)=\max_{a\in\mathcal{A}}Q_{h}^{*}(s,a), Q h π ( s , a ) = r h ( s , a ) + [ P h V h + 1 π ] ( s , a ) , Q h ∗ ( s , a ) = r h ( s , a ) + [ P h V h + 1 ∗ ] ( s , a ) , V h ∗ ( s ) = a ∈ A max Q h ∗ ( s , a ) ,
with V H + 1 ∗ ( s ) = 0 V_{H+1}^{*}(s)=0 V H + 1 ∗ ( s ) = 0 for all s ∈ S s\in\mathcal{S} s ∈ S .
Linear MDPs. We focus on linear MDPs (Yang and Wang, [30 ] ; Jin et al., [20 ] ), defined as follows.
Definition 3.1 (Linear MDP, Jin et al. [20 ] ). An MDP M = ( S , A , H , { P h } h = 1 H , { r h } h = 1 H ) \mathcal{M}=(\mathcal{S},\mathcal{A},H,\{\mathbb{P}_{h}\}_{h=1}^{H},\{r_{h}\}_{h=1}^{H}) M = ( S , A , H , { P h } h = 1 H , { r h } h = 1 H ) is a linear MDP with a known feature map ϕ : S × A → R d \phi:\mathcal{S}\times\mathcal{A}\to\mathbb{R}^{d} ϕ : S × A → R d if for every stage h ∈ [ H ] h\in[H] h ∈ [ H ] , there exist $d
unknown signed measures θ h = ( θ h , 1 , … , θ h , d ) ⊤ \theta_h = (\theta_{h,1}, \ldots, \theta_{h,d})^\top θ h = ( θ h , 1 , … , θ h , d ) ⊤ over S S S and an unknown vector μ h ∈ R d \mu_h \in\mathbb{R}^d μ h ∈ R d such that for all ( s , a , s ′ ) ∈ S × A × S (s,a,s') \in S \times A \times S ( s , a , s ′ ) ∈ S × A × S ,
P h ( s ′ ∣ s , a ) = ⟨ ϕ ( s , a ) , d θ h ( s ′ ) ⟩ , r h ( s , a ) = ⟨ ϕ ( s , a ) , μ h ⟩ . \mathbb{P}_h(s' \mid s,a) = \langle\phi(s,a), \mathrm{d}\theta_h(s')\rangle,\qquad r_h(s,a) = \langle\phi(s,a),\mu_h\rangle. P h ( s ′ ∣ s , a ) = ⟨ ϕ ( s , a ) , d θ h ( s ′ )⟩ , r h ( s , a ) = ⟨ ϕ ( s , a ) , μ h ⟩ .
We assume that ∥ ϕ ( s , a ) ∥ 2 ≤ 1 \|\phi(s,a)\|_2 \le1 ∥ ϕ ( s , a ) ∥ 2 ≤ 1 for all ( s , a ) ∈ S × A (s,a) \in S \times A ( s , a ) ∈ S × A , ∥ μ h ∥ 2 ≤ d \|\mu_h\|_2 \le\sqrt{d} ∥ μ h ∥ 2 ≤ d , and use the following bounded-integration normalization for the signed transition measures (cf. Jin et al., [20 ] ; He et al. (2023)):
∥ θ h ( S ) ∥ 2 : = sup f : S → [ − 1 , 1 ] ∥ ∫ S f ( s ′ ) d θ h ( s ′ ) ∥ 2 ≤ d . \|\theta_h(S)\|_2 := \sup_{f:S\to[-1,1]} \left\|\int_S f(s')\mathrm{d}\theta_h(s')\right\|_2 \le\sqrt{d}. ∥ θ h ( S ) ∥ 2 := f : S → [ − 1 , 1 ] sup ∫ S f ( s ′ ) d θ h ( s ′ ) 2 ≤ d .
which holds in particular whenever the coordinate total variations satisfy ∑ j = 1 d ( ∣ θ h , j ∣ ( S ) ) 2 ≤ d \sum_{j=1}^d(|\theta_{h,j}|(S))^2 \le d ∑ j = 1 d ( ∣ θ h , j ∣ ( S ) ) 2 ≤ d (encompassing tabular MDPs with d = ∣ S ∣ ∣ A ∣ d = |S||A| d = ∣ S ∣∣ A ∣ and simplex feature spaces; see Examples 2.1–2.2 of Jin et al. [20 ] ).
A foundational property of Definition 3.1 is that for any bounded function V : S → R V:S\to\mathbb{R} V : S → R , both the first moment [ P h V ] ( s , a ) [\mathbb{P}_hV](s,a) [ P h V ] ( s , a ) and the second moment [ P h V 2 ] ( s , a ) [\mathbb{P}_hV^2](s,a) [ P h V 2 ] ( s , a ) are linear functions of the feature map ϕ ( s , a ) \phi(s,a) ϕ ( s , a ) .
Proposition 3.2 (Linear Representation of Moments, Jin et al. [20 ] ). Under Definition 3.1 , for any stage h ∈ [ H ] h \in[H] h ∈ [ H ] and any measurable function f : S → R f:S\to\mathbb{R} f : S → R satisfying ∥ f ∥ ∞ ≤ B \|f\|_\infty\le B ∥ f ∥ ∞ ≤ B , the integrated parameter vector w h ( f ) : = ∫ S f ( s ′ ) d θ h ( s ′ ) ∈ R d \mathbf{w}_h(f) := \int_S f(s')\mathrm{d}\theta_h(s') \in\mathbb{R}^d w h ( f ) := ∫ S f ( s ′ ) d θ h ( s ′ ) ∈ R d satisfies ∥ w h ( f ) ∥ 2 ≤ B d \|\mathbf{w}_h(f)\|_2 \le B\sqrt{d} ∥ w h ( f ) ∥ 2 ≤ B d and
[ P h f ] ( s , a ) = ⟨ w h ( f ) , ϕ ( s , a ) ⟩ ∀ ( s , a ) ∈ S × A . [\mathbb{P}_hf](s,a) = \langle\mathbf{w}_h(f),\phi(s,a)\rangle\qquad\forall(s,a) \in S \times A. [ P h f ] ( s , a ) = ⟨ w h ( f ) , ϕ ( s , a )⟩ ∀ ( s , a ) ∈ S × A . *In particular, for any value function V : S → [ 0 , H ] V:S\to[0,H] V : S → [ 0 , H ] , we have [ P h V ] ( s , a ) = ⟨ w h ( V ) , ϕ ( s , a ) ⟩ [\mathbb{P}_hV](s,a) = \langle\mathbf{w}_h(V),\phi(s,a)\rangle [ P h V ] ( s , a ) = ⟨ w h ( V ) , ϕ ( s , a )⟩ with ∥ w h ( V ) ∥ 2 ≤ H d \|\mathbf{w}_h(V)\|_2 \le H\sqrt{d} ∥ w h ( V ) ∥ 2 ≤ H d , and [ P h V 2 ] ( s , a ) = ⟨ w h ( V 2 ) , ϕ ( s , a ) ⟩ [\mathbb{P}_hV^2](s,a) = \langle\mathbf{w}_h(V^2),\phi(s,a)\rangle [ P h V 2 ] ( s , a ) = ⟨ w h ( V 2 ) , ϕ ( s , a )⟩ with ∥ w h ( V 2 ) ∥ 2 ≤ H 2 d \|\mathbf{w}_h(V^2)\|_2 \le H^2\sqrt{d} ∥ w h ( V 2 ) ∥ 2 ≤ H 2 d .
Learning protocol, regret, and trajectory variances. The learner interacts with the environment for K K K episodes. At the beginning of each episode k ∈ [ K ] k \in[K] k ∈ [ K ] , the environment selects an arbitrary (possibly history-dependent) initial state s 1 k ∈ S s_1^k \in S s 1 k ∈ S , and the learner chooses a policy π k = { π h k } h = 1 H \pi^k = \{\pi_h^k\}_{h=1}^H π k = { π h k } h = 1 H based on historical observations. Executing π k \pi^k π k generates a trajectory ( s 1 k , a 1 k , s 2 k , a 2 k , … , s H k , a H k , s H + 1 k ) (s_1^k,a_1^k,s_2^k,a_2^k,\ldots,s_H^k,a_H^k,s_{H+1}^k) ( s 1 k , a 1 k , s 2 k , a 2 k , … , s H k , a H k , s H + 1 k ) where a h k = π h k ( s h k ) a_h^k = \pi_h^k(s_h^k) a h k = π h k ( s h k ) and s h + 1 k ∼ P h ( ⋅ ∣ s h k , a h k ) s_{h+1}^k \sim\mathbb{P}_h(\cdot\mid s_h^k,a_h^k) s h + 1 k ∼ P h ( ⋅ ∣ s h k , a h k ) . We denote by F k , h \mathcal{F}_{k,h} F k , h the σ \sigma σ -algebra generated by all random variables up to and including ( s h k , a h k ) (s_h^k,a_h^k) ( s h k , a h k ) at stage h h h of episode k k k prior to observing s h + 1 k s_{h+1}^k s h + 1 k .
The performance of the learner over K K K episodes is measured by the cumulative regret:
Regret ( K ) : = ∑ k = 1 K ( V 1 ∗ ( s 1 k ) − V 1 π k ( s 1 k ) ) . \operatorname{Regret}(K) := \sum_{k=1}^{K}\left(V_1^*(s_1^k)-V_1^{\pi^k}(s_1^k)\right). Regret ( K ) := k = 1 ∑ K ( V 1 ∗ ( s 1 k ) − V 1 π k ( s 1 k ) ) .
To characterize the transition stochasticity encountered during learning, we define the cumulative trajectory variances of the optimal value function and of the played policies’ value functions as
Var K ∗ : = ∑ k = 1 K ∑ h = 1 H [ P h ( V h + 1 ∗ ) 2 ] ( s h k , a h k ) , Var K π : = ∑ k = 1 K ∑ h = 1 H [ P h ( V h + 1 π k ) 2 ] ( s h k , a h k ) . (1) \operatorname{Var}_K^* := \sum_{k=1}^{K}\sum_{h=1}^{H}[\mathbb{P}_h(V_{h+1}^*)^2](s_h^k,a_h^k),\qquad\operatorname{Var}_K^\pi:= \sum_{k=1}^{K}\sum_{h=1}^{H}[\mathbb{P}_h(V_{h+1}^{\pi^k})^2](s_h^k,a_h^k). \tag*{(1)} Var K ∗ := k = 1 ∑ K h = 1 ∑ H [ P h ( V h + 1 ∗ ) 2 ] ( s h k , a h k ) , Var K π := k = 1 ∑ K h = 1 ∑ H [ P h ( V h + 1 π k ) 2 ] ( s h k , a h k ) . (1)
Unlike population variance notions defined by taking expectations under a fixed policy’s rollout distribution (Zanette and Brunskill, [32 ] ; Li and Sun (2023); Huang et al. [16 ] ; Foster et al. [10 ] ), Var K ∗ \operatorname{Var}_K^* Var K ∗ (Zhao et al., [39 ] ; Zhou et al. [43 ] ) and Var K π \operatorname{Var}_K^\pi Var K π in (1) are realized, trajectory-dependent random variables evaluated along the learner’s adaptive trajectory { ( s h k , a h k ) } k ∈ [ K ] , h ∈ [ H ] \{(s_h^k,a_h^k)\}_{k\in[K],h\in[H]} {( s h k , a h k ) } k ∈ [ K ] , h ∈ [ H ] . In particular, Var K ∗ = Var K π = 0 \operatorname{Var}_K^* = \operatorname{Var}_K^\pi= 0 Var K ∗ = Var K π = 0 almost surely in deterministic MDPs, while in general stochastic MDPs Var K π \operatorname{Var}_K^\pi Var K π is bounded with high probability by both O ( H 2 ( K + ln ( 2 / δ ) ) ) O(H^2(K+\ln(2/\delta))) O ( H 2 ( K + ln ( 2/ δ ))) and O ( H V t o t a l ∗ + H 2 ln ( 4 / δ ) ) O(HV_{\mathrm{total}}^*+H^2\ln(4/\delta)) O ( H V total ∗ + H 2 ln ( 4/ δ )) (Lemma C.1 ).
Algorithm 1 VA-LSVI-UCB++: Variance-Aware LSVI-UCB++ for Linear MDPs
Require: Failure probability δ ∈ ( 0 , 1 ) \delta\in(0,1) δ ∈ ( 0 , 1 ) , regularization λ = 1 / H 2 \lambda= 1/H^2 λ = 1/ H 2 , variance floor α = H / K \alpha= H/\sqrt{K} α = H / K , uncertainty weight γ = 2 d 1.5 H 1.5 \gamma= 2d^{1.5}H^{1.5} γ = 2 d 1.5 H 1.5 , confidence radii β , β ˉ , β ~ \beta,\bar{\beta},\widetilde{\beta} β , β ˉ , β defined in (10) .
Initialize Σ 1 , h ← λ I \Sigma_{1,h} \gets\lambda I Σ 1 , h ← λ I , Q 0 , h ( s , a ) ← H Q_{0,h}(s,a) \gets H Q 0 , h ( s , a ) ← H , Q ~ 0 , h ( s , a ) ← 0 \widetilde{Q}_{0,h}(s,a) \gets0 Q 0 , h ( s , a ) ← 0 , V k , H + 1 ( s ) ← 0 V_{k,H+1}(s) \gets0 V k , H + 1 ( s ) ← 0 , V ~ k , H + 1 ( s ) ← 0 \widetilde{V}_{k,H+1}(s) \gets0 V k , H + 1 ( s ) ← 0 for all ( s , a , h , k ) ∈ S × A × [ H ] × [ K ] (s,a,h,k) \in\mathcal{S} \times\mathcal{A} \times[H] \times[K] ( s , a , h , k ) ∈ S × A × [ H ] × [ K ] , and k l a s t ← 1 k_{\mathrm{last}} \gets1 k last ← 1 .
for episode k = 1 , 2 , … , K k = 1,2,\ldots,K k = 1 , 2 , … , K do
Observe initial state s 1 k s_1^k s 1 k .
if there exists h ′ ∈ [ H ] h' \in[H] h ′ ∈ [ H ] such that det ( Σ k , h ′ ) ≥ 2 det ( Σ k l a s t , h ′ ) \det(\Sigma_{k,h'}) \ge2\det(\Sigma_{k_{\mathrm{last}},h'}) det ( Σ k , h ′ ) ≥ 2 det ( Σ k last , h ′ ) or k = 1 k=1 k = 1 then
Set u p d a t e k ← t r u e \mathrm{update}_k \gets\mathrm{true} update k ← true and k l a s t ← k k_{\mathrm{last}} \gets k k last ← k .
else
Set u p d a t e k ← f a l s e \mathrm{update}_k \gets\mathrm{false} update k ← false .
end if
for stage h = H , H − 1 , … , 1 h = H,H-1,\ldots,1 h = H , H − 1 , … , 1 do
w ^ k , h ← Σ k , h − 1 ∑ i = 1 k − 1 σ ˉ i , h − 2 ϕ ( s h i , a h i ) V k , h + 1 ( s h + 1 i ) \widehat{\mathbf{w}}_{k,h} \gets\Sigma_{k,h}^{-1}\sum_{i=1}^{k-1}\bar{\sigma}_{i,h}^{-2}\phi(s_h^i,a_h^i)V_{k,h+1}(s_{h+1}^i) w k , h ← Σ k , h − 1 ∑ i = 1 k − 1 σ ˉ i , h − 2 ϕ ( s h i , a h i ) V k , h + 1 ( s h + 1 i ) .
w ~ k , h ← Σ k , h − 1 ∑ i = 1 k − 1 σ ˉ i , h − 2 ϕ ( s h i , a h i ) V ~ k , h + 1 ( s h + 1 i ) \widetilde{\mathbf{w}}_{k,h} \gets\Sigma_{k,h}^{-1}\sum_{i=1}^{k-1}\bar{\sigma}_{i,h}^{-2}\phi(s_h^i,a_h^i)\widetilde{V}_{k,h+1}(s_{h+1}^i) w k , h ← Σ k , h − 1 ∑ i = 1 k − 1 σ ˉ i , h − 2 ϕ ( s h i , a h i ) V k , h + 1 ( s h + 1 i ) .
w ‾ k , h ← Σ k , h − 1 ∑ i = 1 k − 1 σ ˉ i , h − 2 ϕ ( s h i , a h i ) V k , h + 1 2 ( s h + 1 i ) \overline{\mathbf{w}}_{k,h} \gets\Sigma_{k,h}^{-1}\sum_{i=1}^{k-1}\bar{\sigma}_{i,h}^{-2}\phi(s_h^i,a_h^i)V_{k,h+1}^2(s_{h+1}^i) w k , h ← Σ k , h − 1 ∑ i = 1 k − 1 σ ˉ i , h − 2 ϕ ( s h i , a h i ) V k , h + 1 2 ( s h + 1 i ) .
if u p d a t e k \mathrm{update}_k update k is true then
Q k , h ( s , a ) ← [ min { r h ( s , a ) + w ^ k , h T ϕ ( s , a ) + β ∥ ϕ ( s , a ) ∥ Σ k , h − 1 , Q k − 1 , h ( s , a ) } ] [ 0 , H ] Q_{k,h}(s,a) \gets\left[\min\left\{r_h(s,a)+\widehat{\mathbf{w}}_{k,h}^{\mathsf{T}}\phi(s,a)+\beta\lVert\phi(s,a)\rVert_{\Sigma_{k,h}^{-1}},Q_{k-1,h}(s,a)\right\}\right]_{[0,H]} Q k , h ( s , a ) ← [ min { r h ( s , a ) + w k , h T ϕ ( s , a ) + β ∥ ϕ ( s , a ) ∥ Σ k , h − 1 , Q k − 1 , h ( s , a ) } ] [ 0 , H ] .
Q ~ k , h ( s , a ) ← [ max { r h ( s , a ) + w ~ k , h T ϕ ( s , a ) − β ~ ∥ ϕ ( s , a ) ∥ Σ k , h − 1 , Q ~ k − 1 , h ( s , a ) } ] [ 0 , H ] \widetilde{Q}_{k,h}(s,a) \gets\left[\max\left\{r_h(s,a)+\widetilde{\mathbf{w}}_{k,h}^{\mathsf{T}}\phi(s,a)-\widetilde{\beta}\lVert\phi(s,a)\rVert_{\Sigma_{k,h}^{-1}},\widetilde{Q}_{k-1,h}(s,a)\right\}\right]_{[0,H]} Q k , h ( s , a ) ← [ max { r h ( s , a ) + w k , h T ϕ ( s , a ) − β ∥ ϕ ( s , a ) ∥ Σ k , h − 1 , Q k − 1 , h ( s , a ) } ] [ 0 , H ] .
V k , h ( s ) ← max a ∈ A Q k , h ( s , a ) V_{k,h}(s) \gets\max_{a\in\mathcal{A}}Q_{k,h}(s,a) V k , h ( s ) ← max a ∈ A Q k , h ( s , a ) , V ~ k , h ( s ) ← max a ∈ A Q ~ k , h ( s , a ) \widetilde{V}_{k,h}(s) \gets\max_{a\in\mathcal{A}}\widetilde{Q}_{k,h}(s,a) V k , h ( s ) ← max a ∈ A Q k , h ( s , a ) .
else
Set Q k , h ← Q k − 1 , h Q_{k,h} \gets Q_{k-1,h} Q k , h ← Q k − 1 , h , Q ~ k , h ← Q ~ k − 1 , h \widetilde{Q}_{k,h} \gets\widetilde{Q}_{k-1,h} Q k , h ← Q k − 1 , h , V k , h ← V k − 1 , h V_{k,h} \gets V_{k-1,h} V k , h ← V k − 1 , h , V ~ k , h ← V ~ k − 1 , h \widetilde{V}_{k,h} \gets\widetilde{V}_{k-1,h} V k , h ← V k − 1 , h .
end if
end for
for stage h = 1 , 2 , … , H h=1,2,\ldots,H h = 1 , 2 , … , H do
Take action a h k ← a r g m a x a ∈ A Q k , h ( s h k , a ) a_h^k \gets\mathop{\mathrm{argmax}}_{a\in\mathcal{A}}Q_{k,h}(s_h^k,a) a h k ← argmax a ∈ A Q k , h ( s h k , a ) and observe next state s h + 1 k ∼ P h ( ⋅ ∣ s h k , a h k ) s_{h+1}^k \sim\mathbb{P}_h(\cdot\mid s_h^k,a_h^k) s h + 1 k ∼ P h ( ⋅ ∣ s h k , a h k ) .
Compute empirical variance σ k , h \sigma_{k,h} σ k , h via (4.3)–(4.6).
Set σ ˉ k , h ← max { σ k , h , γ ∥ ϕ ( s h k , a h k ) ∥ Σ k , h − 1 1 / 2 } \bar{\sigma}_{k,h} \gets\max\left\{\sigma_{k,h},\gamma\lVert\phi(s_h^k,a_h^k)\rVert_{\Sigma_{k,h}^{-1}}^{1/2}\right\} σ ˉ k , h ← max { σ k , h , γ ∥ ϕ ( s h k , a h k ) ∥ Σ k , h − 1 1/2 } .
Update Σ k + 1 , h ← Σ k , h + σ ˉ k , h − 2 ϕ ( s h k , a h k ) ϕ ( s h k , a h k ) T \Sigma_{k+1,h} \gets\Sigma_{k,h}+\bar{\sigma}_{k,h}^{-2}\phi(s_h^k,a_h^k)\phi(s_h^k,a_h^k)^{\mathsf{T}} Σ k + 1 , h ← Σ k , h + σ ˉ k , h − 2 ϕ ( s h k , a h k ) ϕ ( s h k , a h k ) T .
end for
end for
The Proposed Algorithm
In this section, we present our variance-aware algorithm VA-LSVI-UCB++ in Algorithm 1 and describe its core algorithmic components.
Weighted Ridge Regression and Monotonic Value Estimation
By Proposition 3.2 , for any optimistic value function estimate V k , h + 1 V_{k,h+1} V k , h + 1 , pessimistic value function estimate V ~ k , h + 1 \widetilde{V}_{k,h+1} V k , h + 1 , and squared value function V k , h + 1 2 V_{k,h+1}^2 V k , h + 1 2 , the Bellman backups [ P h V k , h + 1 ] ( s , a ) [\mathbb{P}_hV_{k,h+1}](s,a) [ P h V k , h + 1 ] ( s , a ) , [ P h V ~ k , h + 1 ] ( s , a ) [\mathbb{P}_h\widetilde{V}_{k,h+1}](s,a) [ P h V k , h + 1 ] ( s , a ) , and [ P h V k , h + 1 2 ] ( s , a ) [\mathbb{P}_hV_{k,h+1}^2](s,a) [ P h V k , h + 1 2 ] ( s , a ) are linear in ϕ ( s , a ) \phi(s,a) ϕ ( s , a ) with true parameter vectors w h ( V k , h + 1 ) \mathbf{w}_h(V_{k,h+1}) w h ( V k , h + 1 ) , w h ( V ~ k , h + 1 ) \mathbf{w}_h(\widetilde{V}_{k,h+1}) w h ( V k , h + 1 ) , and w h ( V k , h + 1 2 ) \mathbf{w}_h(V_{k,h+1}^2) w h ( V k , h + 1 2 ) . To estimate these vectors accurately in the presence of heteroscedastic transition noise, VA-LSVI-UCB++ solves three weighted ridge regression problems at each episode k ∈ [ K ] k \in[K] k ∈ [ K ] and stage h ∈ [ H ] h \in[H] h ∈ [ H ] :
w ˉ k , h ← argmin w ∈ R d λ ∥ w ∥ 2 2 + ∑ i = 1 k − 1 σ ˉ i , h − 2 ( w T ϕ ( s h i , a h i ) − V k , h + 1 ( s h + 1 i ) ) 2 \bar{w}_{k,h} \leftarrow\underset{w\in\mathbb{R}^{d}}{\operatorname{argmin}}\ \lambda\lVert w\rVert_{2}^{2}+\sum_{i=1}^{k-1}\bar{\sigma}_{i,h}^{-2}\left(w^{\mathsf{T}}\phi(s_{h}^{i},a_{h}^{i})-V_{k,h+1}(s_{h+1}^{i})\right)^{2} w ˉ k , h ← w ∈ R d argmin λ ∥ w ∥ 2 2 + i = 1 ∑ k − 1 σ ˉ i , h − 2 ( w T ϕ ( s h i , a h i ) − V k , h + 1 ( s h + 1 i ) ) 2
w ~ k , h ← argmin w ∈ R d λ ∥ w ∥ 2 2 + ∑ i = 1 k − 1 σ ˉ i , h − 2 ( w T ϕ ( s h i , a h i ) − V ~ k , h + 1 ( s h + 1 i ) ) 2 \tilde{w}_{k,h} \leftarrow\underset{w\in\mathbb{R}^{d}}{\operatorname{argmin}}\ \lambda\lVert w\rVert_{2}^{2}+\sum_{i=1}^{k-1}\bar{\sigma}_{i,h}^{-2}\left(w^{\mathsf{T}}\phi(s_{h}^{i},a_{h}^{i})-\widetilde{V}_{k,h+1}(s_{h+1}^{i})\right)^{2} w ~ k , h ← w ∈ R d argmin λ ∥ w ∥ 2 2 + i = 1 ∑ k − 1 σ ˉ i , h − 2 ( w T ϕ ( s h i , a h i ) − V k , h + 1 ( s h + 1 i ) ) 2
w ^ k , h ← argmin w ∈ R d λ ∥ w ∥ 2 2 + ∑ i = 1 k − 1 σ ˉ i , h − 2 ( w T ϕ ( s h i , a h i ) − V k , h + 1 2 ( s h + 1 i ) ) 2 \widehat{w}_{k,h} \leftarrow\underset{w\in\mathbb{R}^{d}}{\operatorname{argmin}}\ \lambda\lVert w\rVert_{2}^{2}+\sum_{i=1}^{k-1}\bar{\sigma}_{i,h}^{-2}\left(w^{\mathsf{T}}\phi(s_{h}^{i},a_{h}^{i})-V_{k,h+1}^{2}(s_{h+1}^{i})\right)^{2} w k , h ← w ∈ R d argmin λ ∥ w ∥ 2 2 + i = 1 ∑ k − 1 σ ˉ i , h − 2 ( w T ϕ ( s h i , a h i ) − V k , h + 1 2 ( s h + 1 i ) ) 2
whose closed-form solutions are given in Lines 10–12 of Algorithm 1 with weighted covariance matrix Σ k , h = λ I + ∑ i = 1 k − 1 σ ˉ i , h − 2 ϕ ( s h i , a h i ) ϕ ( s h i , a h i ) T \Sigma_{k,h}=\lambda I+\sum_{i=1}^{k-1}\bar{\sigma}_{i,h}^{-2}\phi(s_{h}^{i},a_{h}^{i})\phi(s_{h}^{i},a_{h}^{i})^{\mathsf{T}} Σ k , h = λ I + ∑ i = 1 k − 1 σ ˉ i , h − 2 ϕ ( s h i , a h i ) ϕ ( s h i , a h i ) T .
Monotonic value updates with rare switching. When the determinant-doubling criterion (Line 4) is triggered, VA-LSVI-UCB++ updates the optimistic action-value estimator Q k , h Q_{k,h} Q k , h and pessimistic action-value estimator Q ~ k , h \widetilde{Q}_{k,h} Q k , h (Lines 14–15) via
Q k , h ( s , a ) = [ min { r h ( s , a ) + w ˉ k , h T ϕ ( s , a ) + β ∥ ϕ ( s , a ) ∥ Σ k , h − 1 , Q k − 1 , h ( s , a ) } ] [ 0 , H ] , (2) Q_{k,h}(s,a)=\left[\min\left\{r_{h}(s,a)+\bar{w}_{k,h}^{\mathsf{T}}\phi(s,a)+\beta\lVert\phi(s,a)\rVert_{\Sigma_{k,h}^{-1}},Q_{k-1,h}(s,a)\right\}\right]_{[0,H]} , \tag*{(2)} Q k , h ( s , a ) = [ min { r h ( s , a ) + w ˉ k , h T ϕ ( s , a ) + β ∥ ϕ ( s , a ) ∥ Σ k , h − 1 , Q k − 1 , h ( s , a ) } ] [ 0 , H ] , (2)
Q ~ k , h ( s , a ) = [ max { r h ( s , a ) + w ~ k , h T ϕ ( s , a ) − β ˉ ∥ ϕ ( s , a ) ∥ Σ k , h − 1 , Q ~ k − 1 , h ( s , a ) } ] [ 0 , H ] . (3) \widetilde{Q}_{k,h}(s,a)=\left[\max\left\{r_{h}(s,a)+\widetilde{w}_{k,h}^{\mathsf{T}}\phi(s,a)-\bar{\beta}\lVert\phi(s,a)\rVert_{\Sigma_{k,h}^{-1}},\widetilde{Q}_{k-1,h}(s,a)\right\}\right]_{[0,H]} . \tag*{(3)} Q k , h ( s , a ) = [ max { r h ( s , a ) + w k , h T ϕ ( s , a ) − β ˉ ∥ ϕ ( s , a ) ∥ Σ k , h − 1 , Q k − 1 , h ( s , a ) } ] [ 0 , H ] . (3)
Taking the pointwise minimum with Q k − 1 , h Q_{k-1,h} Q k − 1 , h in (2) , an idea originating in Algorithm 2 (UCB-Q-values) of Azar et al. [5 ] for tabular MDPs (see also Zhang et al. [38 ] ) and extended to linear MDPs together with the symmetric pessimistic maximum (3) by He et al. (2023), ensures that for every s ∈ S s\in\mathcal{S} s ∈ S and h ∈ [ H ] h\in[H] h ∈ [ H ] , V k , h ( s ) V_{k,h}(s) V k , h ( s ) is monotonically non-increasing in k k k while V ~ k , h ( s ) \widetilde{V}_{k,h}(s) V k , h ( s ) is monotonically non-decreasing in k k k . On E c o n c \mathcal{E}_{\mathrm{conc}} E conc , where the sandwich holds at every stage and episode (Lemma A.5), this dual monotonicity guarantees that for all future episodes j ≥ k j\geq k j ≥ k ,
0 ≤ V j , h + 1 ( s ) − V h + 1 ∗ ( s ) ≤ V k , h + 1 ( s ) − V ~ k , h + 1 ( s ) , 0\leq V_{j,h+1}(s)-V_{h+1}^{*}(s)\leq V_{k,h+1}(s)-\widetilde{V}_{k,h+1}(s), 0 ≤ V j , h + 1 ( s ) − V h + 1 ∗ ( s ) ≤ V k , h + 1 ( s ) − V k , h + 1 ( s ) ,
0 ≤ V h + 1 ∗ ( s ) − V ~ j , h + 1 ( s ) ≤ V k , h + 1 ( s ) − V ~ k , h + 1 ( s ) . 0\leq V_{h+1}^{*}(s)-\widetilde{V}_{j,h+1}(s)\leq V_{k,h+1}(s)-\widetilde{V}_{k,h+1}(s). 0 ≤ V h + 1 ∗ ( s ) − V j , h + 1 ( s ) ≤ V k , h + 1 ( s ) − V k , h + 1 ( s ) .
Consequently, a variance upper bound constructed at episode k k k remains valid for the residual value functions V j , h + 1 − V h + 1 ∗ V_{j,h+1}-V_{h+1}^{*} V j , h + 1 − V h + 1 ∗ , V h + 1 ∗ − V ~ j , h + 1 V_{h+1}^{*}-\widetilde{V}_{j,h+1} V h + 1 ∗ − V j , h + 1 , and V j , h + 1 2 − ( V h + 1 ∗ ) 2 V_{j,h+1}^{2}-(V_{h+1}^{*})^{2} V j , h + 1 2 − ( V h + 1 ∗ ) 2 at all future episodes j ≥ k j\geq k j ≥ k . Moreover, because updates only occur when det ( Σ k , h ′ ) \det(\Sigma_{k,h'}) det ( Σ k , h ′ ) doubles at some stage h ′ ∈ [ H ] h'\in[H] h ′ ∈ [ H ] (Line 4), the value functions are updated at most O ( d H log K ) O(dH\log K) O ( d H log K ) times, preventing the covering entropy of the function class from growing linearly with K K K .
Variance Estimator and Confidence Radii
At stage h h h of episode k k k , after taking action a h k a_{h}^{k} a h k , VA-LSVI-UCB++ constructs the variance estimator σ k , h \sigma_{k,h} σ k , h from w ˉ k , h , w ~ k , h , w ^ k , h \bar{w}_{k,h},\widetilde{w}_{k,h},\widehat{w}_{k,h} w ˉ k , h , w k , h , w k , h via
[ V ~ k , h V k , h + 1 ] ( s h k , a h k ) : = [ w ~ k , h T ϕ ( s h k , a h k ) ] [ 0 , H 2 ] − [ w ^ k , h T ϕ ( s h k , a h k ) ] [ 0 , H ] 2 ∣ [ 0 , H 2 ] , (4) [\widetilde{V}_{k,h}V_{k,h+1}](s_{h}^{k},a_{h}^{k}):=\left[\widetilde{w}_{k,h}^{\mathsf{T}}\phi(s_{h}^{k},a_{h}^{k})\right]_{[0,H^{2}]}-\left[\widehat{w}_{k,h}^{\mathsf{T}}\phi(s_{h}^{k},a_{h}^{k})\right]_{[0,H]}^{2}\bigg|_{[0,H^{2}]} , \tag*{(4)} [ V k , h V k , h + 1 ] ( s h k , a h k ) := [ w k , h T ϕ ( s h k , a h k ) ] [ 0 , H 2 ] − [ w k , h T ϕ ( s h k , a h k ) ] [ 0 , H ] 2 [ 0 , H 2 ] , (4)
E k , h : = min { β ~ ∥ ϕ ( s h k , a h k ) ∥ Σ k , h − 1 , H 2 } + min { 2 H β ˉ ∥ ϕ ( s h k , a h k ) ∥ Σ k , h − 1 , H 2 } , (5) E_{k,h}:=\min\left\{\widetilde{\beta}\lVert\phi(s_{h}^{k},a_{h}^{k})\rVert_{\Sigma_{k,h}^{-1}},H^{2}\right\}+\min\left\{2H\bar{\beta}\lVert\phi(s_{h}^{k},a_{h}^{k})\rVert_{\Sigma_{k,h}^{-1}},H^{2}\right\}, \tag*{(5)} E k , h := min { β ∥ ϕ ( s h k , a h k ) ∥ Σ k , h − 1 , H 2 } + min { 2 H β ˉ ∥ ϕ ( s h k , a h k ) ∥ Σ k , h − 1 , H 2 } , (5)
D k , h : = min { 4 d 3 H 2 ( ( w ˉ k , h − w ~ k , h ) T ϕ ( s h k , a h k ) + 2 β ˉ ∥ ϕ ( s h k , a h k ) ∥ Σ k , h − 1 ) , d 3 H 3 } , (6) D_{k,h}:=\min\left\{4d^{3}H^{2}\left((\bar{w}_{k,h}-\widetilde{w}_{k,h})^{\mathsf{T}}\phi(s_{h}^{k},a_{h}^{k})+2\bar{\beta}\lVert\phi(s_{h}^{k},a_{h}^{k})\rVert_{\Sigma_{k,h}^{-1}}\right),d^{3}H^{3}\right\}, \tag*{(6)} D k , h := min { 4 d 3 H 2 ( ( w ˉ k , h − w k , h ) T ϕ ( s h k , a h k ) + 2 β ˉ ∥ ϕ ( s h k , a h k ) ∥ Σ k , h − 1 ) , d 3 H 3 } , (6)
σ k , h : = [ V ˉ k , h V k , h + 1 ] ( s h k , a h k ) + E k , h + D k , h + α 2 . (7) \sigma_{k,h} := \sqrt{[\bar{V}_{k,h}V_{k,h+1}](s_h^k,a_h^k)+E_{k,h}+D_{k,h}+\alpha^2}. \tag*{(7)} σ k , h := [ V ˉ k , h V k , h + 1 ] ( s h k , a h k ) + E k , h + D k , h + α 2 . (7)
Here [ V ˉ k , h V k , h + 1 ] ( s h k , a h k ) [\bar{V}_{k,h}V_{k,h+1}](s_h^k,a_h^k) [ V ˉ k , h V k , h + 1 ] ( s h k , a h k ) is the plug-in empirical variance of V k , h + 1 V_{k,h+1} V k , h + 1 , E k , h E_{k,h} E k , h compensates for the estimation error between [ V ˉ k , h V k , h + 1 ] [\bar{V}_{k,h}V_{k,h+1}] [ V ˉ k , h V k , h + 1 ] and the true variance [ V h V k , h + 1 ] [V_hV_{k,h+1}] [ V h V k , h + 1 ] , and D k , h D_{k,h} D k , h bounds both the variance gap ∣ [ V h V k , h + 1 ] − [ V h V h + 1 ∗ ] ∣ |[V_hV_{k,h+1}]-[V_hV_{h+1}^*]| ∣ [ V h V k , h + 1 ] − [ V h V h + 1 ∗ ] ∣ and 2 d 3 H 2d^3H 2 d 3 H times the residual variances [ V h ( V ~ j , h + 1 − V h + 1 ∗ ) ] [V_h(\tilde{V}_{j,h+1}-V_{h+1}^*)] [ V h ( V ~ j , h + 1 − V h + 1 ∗ )] and [ V h ( V h + 1 ∗ − V ~ j , h + 1 ) ] [V_h(V_{h+1}^*-\tilde{V}_{j,h+1})] [ V h ( V h + 1 ∗ − V ~ j , h + 1 )] for all j ≥ k j\ge k j ≥ k .
Variance floor and confidence radii. In Line 24 of Algorithm 1, the regression weight σ ˉ k , h \bar{\sigma}_{k,h} σ ˉ k , h is set as
σ ˉ k , h ← max { σ k , h , γ ∥ ϕ ( s h k , a h k ) ∥ Σ ˉ k , h − 1 1 / 2 } . (8) \bar{\sigma}_{k,h}\leftarrow\max\left\{\sigma_{k,h},\gamma\|\phi(s_h^k,a_h^k)\|_{\bar{\Sigma}_{k,h}^{-1}}^{1/2}\right\}. \tag*{(8)} σ ˉ k , h ← max { σ k , h , γ ∥ ϕ ( s h k , a h k ) ∥ Σ ˉ k , h − 1 1/2 } . (8)
combining the variance estimator σ k , h \sigma_{k,h} σ k , h (which already satisfies σ k , h ≥ α = H / K \sigma_{k,h}\ge\alpha=H/\sqrt{K} σ k , h ≥ α = H / K by (7) ) with the uncertainty-aware weight γ ∥ ϕ ( s h k , a h k ) ∥ Σ ˉ k , h − 1 1 / 2 \gamma\|\phi(s_h^k,a_h^k)\|_{\bar{\Sigma}_{k,h}^{-1}}^{1/2} γ ∥ ϕ ( s h k , a h k ) ∥ Σ ˉ k , h − 1 1/2 ([40 ] (He et al. 2022; He et al. 2023)). Define the logarithmic factors
ι : = max { ln ( 1 + K d λ α 2 ) , 1 } = max { ln ( 1 + K 2 d ) , 1 } , ι 0 : = ln ( 32 ( d H K / δ ) 4 δ ) , (9) \iota:=\max\left\{\ln\left(1+\frac{K}{d\lambda\alpha^2}\right),1\right\}=\max\left\{\ln\left(1+\frac{K^2}{d}\right),1\right\},\qquad\iota_0:=\ln\left(\frac{32(dHK/\delta)^4}{\delta}\right), \tag*{(9)} ι := max { ln ( 1 + d λ α 2 K ) , 1 } = max { ln ( 1 + d K 2 ) , 1 } , ι 0 := ln ( δ 32 ( d H K / δ ) 4 ) , (9)
and set the confidence radii β , β ˉ , β ~ \beta,\bar{\beta},\tilde{\beta} β , β ˉ , β ~ in Algorithm 1 to
β = β ˉ : = H d λ + 2 β 0 + 1 = O ~ ( d ) , β ~ : = 2 H β = O ~ ( H d ) , (10) \beta=\bar{\beta}:=H\sqrt{d\lambda}+2\beta_0+1=\widetilde{O}(\sqrt{d}),\qquad\tilde{\beta}:=2H\beta=\widetilde{O}(H\sqrt{d}), \tag*{(10)} β = β ˉ := H d λ + 2 β 0 + 1 = O ( d ) , β ~ := 2 H β = O ( H d ) , (10)
where β 0 : = C 0 ( d ι 2 ι 0 + ι ι 0 ) \beta_0:=C_0(\sqrt{d}\iota^2\iota_0+\iota\iota_0) β 0 := C 0 ( d ι 2 ι 0 + ι ι 0 ) with an absolute constant C 0 > 0 C_0>0 C 0 > 0 specified in Lemma A.4. Comparing (7) –(10) with LSVI-UCB++ ((He et al., 2023)) (as well as [16 ] (Li and Sun 2023)) highlights two closely coupled algorithmic innovations:
Vanishing variance floor α 2 = H 2 / K \alpha^2=H^2/K α 2 = H 2 / K versus constant or K − 1 / 2 K^{-1/2} K − 1/2 floors. In weighted ridge regression, a strictly positive floor σ k , h 2 ≥ α 2 > 0 \sigma_{k,h}^2\ge\alpha^2>0 σ k , h 2 ≥ α 2 > 0 can ensure that the inverse-variance weights σ ˉ k , h − 2 \bar{\sigma}_{k,h}^{-2} σ ˉ k , h − 2 remain well-defined when empirical variances vanish and the normalized features x k , h = σ ˉ k , h − 1 ϕ ( s h k , a h k ) x_{k,h}=\bar{\sigma}_{k,h}^{-1}\phi(s_h^k,a_h^k) x k , h = σ ˉ k , h − 1 ϕ ( s h k , a h k ) satisfy ∥ x k , h ∥ 2 ≤ α − 1 \|x_{k,h}\|_2\le\alpha^{-1} ∥ x k , h ∥ 2 ≤ α − 1 , keeping the determinant-doubling factor ι \iota ι logarithmic in K K K . However, because the cumulative exploration bonus scales with β d H ∑ k = 1 K ∑ h = 1 H σ k , h 2 \beta\sqrt{dH\sum_{k=1}^K\sum_{h=1}^H\sigma_{k,h}^2} β d H ∑ k = 1 K ∑ h = 1 H σ k , h 2 (Lemma B.1 ), any floor on σ k , h 2 \sigma_{k,h}^2 σ k , h 2 directly enters the total estimated variance V e s t = ∑ k = 1 K ∑ h = 1 H σ k , h 2 V_{\mathrm{est}}=\sum_{k=1}^K\sum_{h=1}^H\sigma_{k,h}^2 V est = ∑ k = 1 K ∑ h = 1 H σ k , h 2 . In LSVI-UCB++ ((He et al., 2023)), the coarse Hoeffding concentration analysis forced a constant floor σ k , h 2 ≥ H \sigma_{k,h}^2\ge H σ k , h 2 ≥ H (and σ ˉ k , h ≥ H \bar{\sigma}_{k,h}\ge H σ ˉ k , h ≥ H ), making V e s t ≥ H 2 K V_{\mathrm{est}}\ge H^2K V est ≥ H 2 K and injecting an O ~ ( d H 3 K ) \widetilde{O}(d\sqrt{H^3K}) O ( d H 3 K ) regret term even in deterministic MDPs; similarly, setting the floor to σ min 2 ∝ K − 1 / 2 \sigma_{\min}^2\propto K^{-1/2} σ m i n 2 ∝ K − 1/2 in [16 ] (Li and Sun 2023)) still leaves an O ~ ( K 1 / 4 ) \widetilde{O}(K^{1/4}) O ( K 1/4 ) regret contribution. By decoupling the confidence radii from α − 1 \alpha^{-1} α − 1 (which appears only inside the logarithm ι \iota ι ), we can set α 2 = H 2 / K \alpha^2=H^2/K α 2 = H 2 / K in (7) so that the floor contributes only ∑ k = 1 K ∑ h = 1 H α 2 = H 3 \sum_{k=1}^K\sum_{h=1}^H\alpha^2=H^3 ∑ k = 1 K ∑ h = 1 H α 2 = H 3 to V e s t V_{\mathrm{est}} V est over all K K K episodes.
Variance-normalized radii β ˉ = O ~ ( d ) \bar{\beta}=\widetilde{O}(\sqrt{d}) β ˉ = O ( d ) and β ~ = O ~ ( H d ) \tilde{\beta}=\widetilde{O}(H\sqrt{d}) β ~ = O ( H d ) . Because He et al. (2023) analyzed w ~ k , h \tilde{w}_{k,h} w ~ k , h and w ^ k , h \hat{w}_{k,h} w ^ k , h via a variance-oblivious Hoeffding inequality over the full function classes V ~ h + 1 \tilde{\mathcal{V}}_{h+1} V ~ h + 1 and V h + 1 \mathcal{V}_{h+1} V h + 1 , they required large radii β ˉ = O ~ ( d 3 H 2 ) \bar{\beta}=\widetilde{O}(\sqrt{d^3H^2}) β ˉ = O ( d 3 H 2 ) and β ~ = O ~ ( d 3 H 4 ) \tilde{\beta}=\widetilde{O}(\sqrt{d^3H^4}) β ~ = O ( d 3 H 4 ) as well as γ = 2 d 3 H 2 \gamma=2d^3H^2 γ = 2 d 3 H 2 . By establishing variance-normalized Bernstein concentration for all three estimators simultaneously (§ 6 ), we shave a factor of O ~ ( d H ) \widetilde{O}(dH) O ( d H ) off both β ˉ \bar{\beta} β ˉ and β ~ \tilde{\beta} β ~ and tighten the uncertainty weight to γ = 2 d 1.5 H 1.5 ι \gamma=2d^{1.5}H^{1.5}\iota γ = 2 d 1.5 H 1.5 ι .
Main Results
We now state our main theoretical guarantees for VA-LSVI-UCB++. The complete proof of Theorem 5.1 is provided in Appendices § 9 and § 10 .
Theorem 5.1 (Variance-Dependent Regret Bound for Linear MDPs). Under Definition 3.1 , for any δ ∈ ( 0 , 1 ) \delta\in(0,1) δ ∈ ( 0 , 1 ) , with probability at least 1 − 3 δ 1-3\delta 1 − 3 δ , the regret of VA-LSVI-UCB++ (Algorithm 1 ) is bounded by
Regret ( K ) ≤ O ( β d H ι ⋅ min { Var K ∗ , Var K π } + d 4 H 4 β 2 ι + d 4 H 4 β ι 3 ) \operatorname{Regret}(K) \le\mathcal{O}\left(\beta\sqrt{dH\iota\cdot\min\{\operatorname{Var}^{*}_{K},\operatorname{Var}^{\pi}_{K}\}}+d^{4}H^{4}\beta^{2}\iota+d^{4}H^{4}\beta\iota^{3}\right) Regret ( K ) ≤ O ( β d H ι ⋅ min { Var K ∗ , Var K π } + d 4 H 4 β 2 ι + d 4 H 4 β ι 3 ) = O ~ ( d H ⋅ min { Var K ∗ , Var K π } + d 5 H 4 ) . (11) =\widetilde{\mathcal{O}}\left(d\sqrt{H\cdot\min\{\operatorname{Var}^{*}_{K},\operatorname{Var}^{\pi}_{K}\}}+d^{5}H^{4}\right). \tag*{(11)} = O ( d H ⋅ min { Var K ∗ , Var K π } + d 5 H 4 ) . (11) Moreover, the total number of value function updates (Line 4 ) over K K K episodes is at most O ( d H ι ) \mathcal{O}(dH\iota) O ( d H ι ) , and the total computational complexity of VA-LSVI-UCB++ is O ~ ( d 4 H 3 ∣ A ∣ K ) \widetilde{\mathcal{O}}(d^{4}H^{3}|A|K) O ( d 4 H 3 ∣ A ∣ K ) .
Remark 5.2 (Worst-Case Minimax Optimality and Dual Variance Adaptivity). Since Var K π ≤ 4 H 2 K + 16 H 2 ln ( 2 / δ ) \operatorname{Var}^{\pi}_{K}\le4H^{2}K+16H^{2}\ln(2/\delta) Var K π ≤ 4 H 2 K + 16 H 2 ln ( 2/ δ ) with probability at least 1 − δ 1-\delta 1 − δ (Lemma C.1 ), combining (11) with a union bound immediately yields the worst-case regret O ~ ( d H 3 K + d 5 H 4 ) \widetilde{\mathcal{O}}(d\sqrt{H^{3}K}+d^{5}H^{4}) O ( d H 3 K + d 5 H 4 ) with probability at least 1 − 4 δ 1-4\delta 1 − 4 δ , whose leading term matches the Ω ( d H 3 K ) \Omega(d\sqrt{H^{3}K}) Ω ( d H 3 K ) minimax lower bound for linear MDPs (Zhou et al., [41 ] , Remark 23) (see also He et al. 2023, Remark 5.2) up to logarithmic factors. Moreover, whereas the transition-variance terms in prior bounds (Zhao et al., [39 ] ; Li and Sun, 2023; Huang et al., [16 ] ) are defined through the optimal continuation value V h + 1 ∗ V^{*}_{h+1} V h + 1 ∗ (with additional return-variance caps in Li and Sun 2023; Huang et al. [16 ] ), (11) adapts to min { Var K ∗ , Var K π } \min\{\operatorname{Var}^{*}_{K},\operatorname{Var}^{\pi}_{K}\} min { Var K ∗ , Var K π } , where the dependence on Var K π \operatorname{Var}^{\pi}_{K} Var K π directly yields the first-order regret bound in Corollary 5.5 via Lemma C.1 .
Remark 5.3 (Deterministic and Low-Variance Regimes). In deterministic MDPs where transitions are deterministic (Var K ∗ = Var K π = 0 \operatorname{Var}^{*}_{K}=\operatorname{Var}^{\pi}_{K}=0 Var K ∗ = Var K π = 0 ), or more generally when min { Var K ∗ , Var K π } = O ( 1 ) \min\{\operatorname{Var}^{*}_{K},\operatorname{Var}^{\pi}_{K}\}=\mathcal{O}(1) min { Var K ∗ , Var K π } = O ( 1 ) , Theorem 5.1 yields a regret bound of O ~ ( d 5 H 4 ) \widetilde{\mathcal{O}}(d^{5}H^{4}) O ( d 5 H 4 ) free of polynomial-in-K K K dependency, removing the O ~ ( K 1 / 4 ) \widetilde{\mathcal{O}}(K^{1/4}) O ( K 1/4 ) polynomial overhead of VARA (Li and Sun, 2023) and HEAVY-LSVI-UCB (Huang et al., [16 ] ).
Remark 5.4 (Burn-In Overhead and Computational Complexity). Compared with the O ~ ( d H 3 K + d 7 H 8 ) \widetilde{\mathcal{O}}(d\sqrt{H^{3}K}+d^{7}H^{8}) O ( d H 3 K + d 7 H 8 ) bound of LSVI-UCB++ (He et al., 2023), Theorem 5.1 reduces the lower-order burn-in term by a factor of d 2 H 4 d^{2}H^{4} d 2 H 4 . Computationally, because V k , h + 1 V_{k,h+1} V k , h + 1 and V ~ k , h + 1 \widetilde{V}_{k,h+1} V k , h + 1 change only at O ( d H ι ) \mathcal{O}(dH\iota) O ( d H ι ) rare-switching episodes (Line 4 ), between switching episodes the regression right-hand side vectors ∑ i = 1 k − 1 σ ˉ i , h − 2 ϕ ( s h i , a h i ) f ( s h + 1 i ) \sum_{i=1}^{k-1}\bar{\sigma}_{i,h}^{-2}\phi(s_{h}^{i},a_{h}^{i})f(s_{h+1}^{i}) ∑ i = 1 k − 1 σ ˉ i , h − 2 ϕ ( s h i , a h i ) f ( s h + 1 i ) (f ∈ { V k , h + 1 , V ~ k , h + 1 , V k , h + 1 2 } f\in\{V_{k,h+1},\widetilde{V}_{k,h+1},V_{k,h+1}^{2}\} f ∈ { V k , h + 1 , V k , h + 1 , V k , h + 1 2 } ) and Σ k , h − 1 \Sigma_{k,h}^{-1} Σ k , h − 1 are maintained incrementally in O ( d 2 ) \mathcal{O}(d^{2}) O ( d 2 ) time per step; evaluating Q k , h ( s h , a ) Q_{k,h}(s_{h},a) Q k , h ( s h , a ) and Q ~ k , h ( s h , a ) \widetilde{Q}_{k,h}(s_{h},a) Q k , h ( s h , a ) over the O ( d H ι ) \mathcal{O}(dH\iota) O ( d H ι ) stored functions takes O ~ ( d 3 H ∣ A ∣ ) \widetilde{\mathcal{O}}(d^{3}H|A|) O ( d 3 H ∣ A ∣ ) time per step, and recomputing the historical regression sums at the O ( d H ι ) \mathcal{O}(dH\iota) O ( d H ι ) switching episodes takes O ~ ( d 3 H 2 ∣ A ∣ K ⋅ d H ) = O ~ ( d 4 H 3 ∣ A ∣ K ) \widetilde{\mathcal{O}}(d^{3}H^{2}|A|K\cdot dH)=\widetilde{\mathcal{O}}(d^{4}H^{3}|A|K) O ( d 3 H 2 ∣ A ∣ K ⋅ d H ) = O ( d 4 H 3 ∣ A ∣ K ) total time over K K K episodes (He et al., 2023).
Combining Theorem 5.1 with a high-probability first-order bound on Var K π \operatorname{Var}^{\pi}_{K} Var K π also yields an immediate small-loss regret guarantee (Zanette and Brunskill, [32 ] ; Wagenmaker et al., [27 ] ), proved in Appendix § 11 .
Corollary 5.5 (First-Order Regret). In the setting of Theorem 5.1 , let V t o t a l ∗ : = ∑ k = 1 K V 1 ∗ ( s 1 k ) V^{*}_{\mathrm{total}}:=\sum_{k=1}^{K}V^{*}_{1}(s_{1}^{k}) V total ∗ := ∑ k = 1 K V 1 ∗ ( s 1 k ) denote the cumulative optimal expected return over K K K episodes. With probability at least 1 − 4 δ 1-4\delta 1 − 4 δ , VA-LSVI-UCB++ satisfies
Regret ( K ) ≤ O ~ ( d H 2 V t o t a l ∗ + d 5 H 4 ) . \operatorname{Regret}(K)\le\widetilde{\mathcal{O}}\left(d\sqrt{H^{2}V^{*}_{\mathrm{total}}+d^{5}H^{4}}\right). Regret ( K ) ≤ O ( d H 2 V total ∗ + d 5 H 4 ) .
Remark 5.6 (Comparison with First-Order Regret Bounds). Compared with the computationally inefficient O ~ ( d 3 H 3 V t o t a l ∗ + d 3.5 H 3 ) \widetilde{O}(\sqrt{d^{3}H^{3}V_{\mathrm{total}}^{*}}+d^{3.5}H^{3}) O ( d 3 H 3 V total ∗ + d 3.5 H 3 ) first-order bound of Force Wagenmaker et al., [27 ] , Corollary 5.5 is computationally efficient and improves the leading term by a factor of d H \sqrt{d}H d H . While Huang et al. [16 ] , Corollary 5.6 also obtained the leading first-order dependence O ~ ( d H 2 K V 1 ∗ ) \widetilde{O}(d\sqrt{H^{2}KV_{1}^{*}}) O ( d H 2 K V 1 ∗ ) via HEAVY-LSVI-UCB, their complete bound (see Huang et al. [16 ] , Theorem F.13) retains a variance- and value-independent O ~ ( d 3.5 H 2.75 K 1 / 4 ) \widetilde{O}(d^{3.5}H^{2.75}K^{1/4}) O ( d 3.5 H 2.75 K 1/4 ) polynomial additive term, whereas Corollary 5.5 achieves a lower-order remainder O ~ ( d 5 H 4 ) \widetilde{O}(d^{5}H^{4}) O ( d 5 H 4 ) free of polynomial-in-K K K dependency.
Overview of Key Techniques
In this section, we provide a technical overview of our analysis, explaining the structural bottlenecks that prevented LSVI-UCB++ (He et al., 2023) from achieving variance-dependent regret and how our techniques overcome them.
Three-Way Reference-Advantage Decomposition
The central challenge in linear MDPs is bounding the Bellman estimation errors ∣ ( w ^ k , h − w h ( V k , h + 1 ) ) ⊤ ϕ ( s , a ) ∣ \lvert(\widehat{\mathbf{w}}_{k,h}-\mathbf{w}_{h}(V_{k,h+1}))^{\top}\phi(s,a)\rvert ∣( w k , h − w h ( V k , h + 1 ) ) ⊤ ϕ ( s , a )∣ , ∣ ( w ~ k , h − w h ( V ~ k , h + 1 ) ) ⊤ ϕ ( s , a ) ∣ \lvert(\widetilde{\mathbf{w}}_{k,h}-\mathbf{w}_{h}(\widetilde{V}_{k,h+1}))^{\top}\phi(s,a)\rvert ∣( w k , h − w h ( V k , h + 1 ) ) ⊤ ϕ ( s , a )∣ , and ∣ ( w ~ k , h − w h ( V k , h + 1 2 ) ) ⊤ ϕ ( s , a ) ∣ \lvert(\widetilde{\mathbf{w}}_{k,h}-\mathbf{w}_{h}(V_{k,h+1}^{2}))^{\top}\phi(s,a)\rvert ∣( w k , h − w h ( V k , h + 1 2 ) ) ⊤ ϕ ( s , a )∣ . For any target function f ∈ { V k , h + 1 , V ~ k , h + 1 , V k , h + 1 2 } f\in\{V_{k,h+1},\widetilde{V}_{k,h+1},V_{k,h+1}^{2}\} f ∈ { V k , h + 1 , V k , h + 1 , V k , h + 1 2 } , standard ridge regression algebra reduces the estimation error to the self-normalized martingale sum
∥ ∑ i = 1 k − 1 x i , h σ ˉ i , h − 1 ( f ( s h + 1 i ) − [ P h f ] ( s h i , a h i ) ) ∥ Σ k , h − 1 , where x i , h : = σ ˉ i , h − 1 ϕ ( s h i , a h i ) . \left\|\sum_{i=1}^{k-1}\mathbf{x}_{i,h}\bar{\sigma}_{i,h}^{-1}\left(f(s_{h+1}^{i})-[\mathbb{P}_{h}f](s_{h}^{i},a_{h}^{i})\right)\right\|_{\Sigma_{k,h}^{-1}},\qquad\text{where }\mathbf{x}_{i,h}:=\bar{\sigma}_{i,h}^{-1}\phi(s_{h}^{i},a_{h}^{i}). i = 1 ∑ k − 1 x i , h σ ˉ i , h − 1 ( f ( s h + 1 i ) − [ P h f ] ( s h i , a h i ) ) Σ k , h − 1 , where x i , h := σ ˉ i , h − 1 ϕ ( s h i , a h i ) .
Because V k , h + 1 V_{k,h+1} V k , h + 1 and V ~ k , h + 1 \widetilde{V}_{k,h+1} V k , h + 1 depend on all trajectories up to episode k − 1 k-1 k − 1 , they are not F i , h \mathcal{F}_{i,h} F i , h -measurable for i < k i<k i < k , necessitating a uniform convergence argument over the rare-switching function classes V h + 1 \mathcal{V}_{h+1} V h + 1 , V ~ h + 1 \widetilde{\mathcal{V}}_{h+1} V h + 1 , and V h + 1 2 \mathcal{V}_{h+1}^{2} V h + 1 2 . As shown in Lemma A.3, the ϵ \epsilon ϵ -covering numbers of these classes satisfy ln ∣ N ϵ ∣ = O ~ ( d 3 H ) \ln|\mathcal{N}_{\epsilon}|=\widetilde{O}(d^{3}H) ln ∣ N ϵ ∣ = O ( d 3 H ) .
Bottleneck of coarse-to-fine concentration. In He et al. (2023), the authors employed a two-stage coarse-to-fine concentration strategy:
In Stage 1 (He et al., 2023, Lemma B.1), they applied a variance-oblivious concentration bound over the ϵ \epsilon ϵ -nets of V h + 1 \mathcal{V}_{h+1} V h + 1 , V ~ h + 1 \widetilde{\mathcal{V}}_{h+1} V h + 1 , and V h + 1 2 \mathcal{V}_{h+1}^{2} V h + 1 2 to obtain a “coarse” high-probability event E \mathcal{E} E . Because this coarse bound scales with the worst-case magnitude of the normalized noise ∣ σ ˉ i , h − 1 ( f ( s h + 1 i ) − [ P h f ] ( s h i , a h i ) ) ∣ ≤ ∥ f ∥ ∞ σ min − 1 \lvert\bar{\sigma}_{i,h}^{-1}(f(s_{h+1}^{i})-[\mathbb{P}_{h}f](s_{h}^{i},a_{h}^{i}))\rvert\leq\|f\|_{\infty}\sigma_{\min}^{-1} ∣ σ ˉ i , h − 1 ( f ( s h + 1 i ) − [ P h f ] ( s h i , a h i ))∣ ≤ ∥ f ∥ ∞ σ m i n − 1 multiplied by ln ∣ N ϵ ∣ = O ~ ( d 3 H ) \sqrt{\ln|\mathcal{N}_{\epsilon}|}=\widetilde{O}(\sqrt{d^{3}H}) ln ∣ N ϵ ∣ = O ( d 3 H ) , He et al. (2023) used the lower bound σ ˉ i , h ≥ σ min = H \bar{\sigma}_{i,h}\geq\sigma_{\min}=\sqrt{H} σ ˉ i , h ≥ σ m i n = H in this analysis, inflating the pessimistic and second-moment radii to β ˉ = O ~ ( d 3 H 2 ) \bar{\beta}=\widetilde{O}(\sqrt{d^{3}H^{2}}) β ˉ = O ( d 3 H 2 ) and β ~ = O ~ ( d 3 H 4 ) \widetilde{\beta}=\widetilde{O}(\sqrt{d^{3}H^{4}}) β = O ( d 3 H 4 ) . In the subsequent variance-aware analyses of Li and Sun (2023) and Huang et al. [16 ] , retaining this coarse concentration stage yielded initial confidence radii scaling as O ~ ( H d 3 H / σ min ) \widetilde{O}(H\sqrt{d^{3}H}/\sigma_{\min}) O ( H d 3 H / σ m i n ) (and O ~ ( H 2 d 3 H / σ min ) \widetilde{O}(H^{2}\sqrt{d^{3}H}/\sigma_{\min}) O ( H 2 d 3 H / σ m i n ) for the second-moment estimator); optimizing their resulting regret bounds over σ min ∝ K − 1 / 4 \sigma_{\min}\propto K^{-1/4} σ m i n ∝ K − 1/4 to balance the σ min K \sigma_{\min}\sqrt{K} σ m i n K bonus floor against the σ min − 1 \sigma_{\min}^{-1} σ m i n − 1 radius, retaining a O ~ ( d 3.5 H 2.75 K 1 / 4 ) \widetilde{O}(d^{3.5}H^{2.75}K^{1/4}) O ( d 3.5 H 2.75 K 1/4 ) additive term in their guarantees.
In Stage 2 (He et al., 2023, Lemma B.5), conditioned on the coarse event E \mathcal{E} E , they applied a reference-advantage decomposition V k , h + 1 = V h + 1 ∗ + ( V k , h + 1 − V h + 1 ∗ ) V_{k,h+1}=V_{h+1}^{*}+(V_{k,h+1}-V_{h+1}^{*}) V k , h + 1 = V h + 1 ∗ + ( V k , h + 1 − V h + 1 ∗ ) only to the optimistic estimator w ^ k , h \widehat{\mathbf{w}}_{k,h} w k , h to refine its radius to β = O ~ ( d ) \beta=\widetilde{O}(\sqrt{d}) β = O ( d ) .
Simultaneous three-way decomposition. We bypass the coarse Hoeffding stage altogether by applying reference-advantage decompositions [5 , 38 ] (He et al. 2023) to all three regression targets simultaneously:
V k , h + 1 = V h + 1 ∗ + Δ V k , h + 1 , where Δ V k , h + 1 : = V k , h + 1 − V h + 1 ∗ , V ~ k , h + 1 = V h + 1 ∗ − Δ V ~ k , h + 1 , where Δ V ~ k , h + 1 : = V h + 1 ∗ − V ~ k , h + 1 , V k , h + 1 2 = ( V h + 1 ∗ ) 2 + Δ ( V k , h + 1 2 ) , where Δ ( V k , h + 1 2 ) : = V k , h + 1 2 − ( V h + 1 ∗ ) 2 . (12) \begin{aligned}
V_{k,h+1} &= V^*_{h+1} + \Delta V_{k,h+1}, && \text{where } \Delta V_{k,h+1} := V_{k,h+1} - V^*_{h+1},\\
\widetilde{V}_{k,h+1} &= V^*_{h+1} - \Delta\widetilde{V}_{k,h+1}, && \text{where } \Delta\widetilde{V}_{k,h+1} := V^*_{h+1} - \widetilde{V}_{k,h+1},\\
V^2_{k,h+1} &= (V^*_{h+1})^2 + \Delta(V^2_{k,h+1}), && \text{where } \Delta(V^2_{k,h+1}) := V^2_{k,h+1} - (V^*_{h+1})^2. \tag*{(12)}
\end{aligned} V k , h + 1 V k , h + 1 V k , h + 1 2 = V h + 1 ∗ + Δ V k , h + 1 , = V h + 1 ∗ − Δ V k , h + 1 , = ( V h + 1 ∗ ) 2 + Δ ( V k , h + 1 2 ) , where Δ V k , h + 1 := V k , h + 1 − V h + 1 ∗ , where Δ V k , h + 1 := V h + 1 ∗ − V k , h + 1 , where Δ ( V k , h + 1 2 ) := V k , h + 1 2 − ( V h + 1 ∗ ) 2 . (12)
Notice how (12) separates each target into a fixed reference component ( V h + 1 ∗ (V^*_{h+1} ( V h + 1 ∗ or ( V h + 1 ∗ ) 2 ) (V^*_{h+1})^2) ( V h + 1 ∗ ) 2 ) that is deterministic and requires no ϵ \epsilon ϵ -net union bound, plus a random residual component ( Δ V k , h + 1 (\Delta V_{k,h+1} ( Δ V k , h + 1 , Δ V ~ k , h + 1 \Delta\widetilde{V}_{k,h+1} Δ V k , h + 1 , or Δ ( V k , h + 1 2 ) ) \Delta(V^2_{k,h+1})) Δ ( V k , h + 1 2 )) that requires an ϵ \epsilon ϵ -net union bound with log-covering number ln ∣ N ϵ ∣ = O ~ ( d 3 H ) \ln|\mathcal{N}_{\epsilon}| = \widetilde{O}(d^3H) ln ∣ N ϵ ∣ = O ( d 3 H ) . To apply the variance-aware self-normalized Bernstein inequality of Zhou and Gu (2022, Theorem 4.3) (Lemma E.3) to all six components without any constant floor on σ ˉ i , h \bar{\sigma}_{i,h} σ ˉ i , h , we must control the conditional variance σ ˉ i , h − 2 [ V h f ] ( s h i , a h i ) \bar{\sigma}_{i,h}^{-2}[\mathbb{V}_h f](s_h^i,a_h^i) σ ˉ i , h − 2 [ V h f ] ( s h i , a h i ) of the normalized noise σ ˉ i , h − 1 ( f ( s h + 1 i ) − [ P h f ] ( s h i , a h i ) ) \bar{\sigma}_{i,h}^{-1}(f(s_{h+1}^i)-[\mathbb{P}_h f](s_h^i,a_h^i)) σ ˉ i , h − 1 ( f ( s h + 1 i ) − [ P h f ] ( s h i , a h i )) for every historical episode i < k i < k i < k :
Controlling the fixed squared reference ( V h + 1 ∗ ) 2 (V^*_{h+1})^2 ( V h + 1 ∗ ) 2 . While σ ˉ i , h 2 ≥ [ V h V h + 1 ∗ ] ( s h i , a h i ) \bar{\sigma}_{i,h}^2 \geq[\mathbb{V}_h V^*_{h+1}](s_h^i,a_h^i) σ ˉ i , h 2 ≥ [ V h V h + 1 ∗ ] ( s h i , a h i ) directly normalizes the variance of V h + 1 ∗ V^*_{h+1} V h + 1 ∗ to at most 1 1 1 , bounding the variance of ( V h + 1 ∗ ) 2 (V^*_{h+1})^2 ( V h + 1 ∗ ) 2 without introducing a fourth-moment estimator requires linking [ V h ( V h + 1 ∗ ) 2 ] [\mathbb{V}_h(V^*_{h+1})^2] [ V h ( V h + 1 ∗ ) 2 ] back to [ V h V h + 1 ∗ ] [\mathbb{V}_h V^*_{h+1}] [ V h V h + 1 ∗ ] . We apply the conditional variance bound for squared functions (Corollary E.5 , following Chen et al. [7 ] , Chen et al. [8 ] , and Foster et al. [10 ] ): since V h + 1 ∗ ( s ′ ) ∈ [ 0 , H ] V^*_{h+1}(s') \in[0,H] V h + 1 ∗ ( s ′ ) ∈ [ 0 , H ] ,
[ V h ( V h + 1 ∗ ) 2 ] ( s h i , a h i ) ≤ 4 H 2 [ V h V h + 1 ∗ ] ( s h i , a h i ) ≤ 4 H 2 σ ˉ i , h 2 . [\mathbb{V}_h(V^*_{h+1})^2](s_h^i,a_h^i) \leq4H^2[\mathbb{V}_hV^*_{h+1}](s_h^i,a_h^i) \leq4H^2\bar{\sigma}_{i,h}^2. [ V h ( V h + 1 ∗ ) 2 ] ( s h i , a h i ) ≤ 4 H 2 [ V h V h + 1 ∗ ] ( s h i , a h i ) ≤ 4 H 2 σ ˉ i , h 2 .
Thus, the normalized noise of ( V h + 1 ∗ ) 2 (V^*_{h+1})^2 ( V h + 1 ∗ ) 2 has conditional variance bounded by 4 H 2 4H^2 4 H 2 , yielding a tight concentration radius of O ~ ( H d ) \widetilde{O}(H\sqrt{d}) O ( H d ) .
Controlling the three residual components across future episodes k > i k > i k > i . For the residual components Δ V k , h + 1 \Delta V_{k,h+1} Δ V k , h + 1 , Δ V ~ k , h + 1 \Delta\widetilde{V}_{k,h+1} Δ V k , h + 1 , and Δ ( V k , h + 1 2 ) \Delta(V^2_{k,h+1}) Δ ( V k , h + 1 2 ) , the union bound over their ϵ \epsilon ϵ -nets introduces a factor of ln ∣ N ϵ ∣ = O ~ ( d 3 H ) \sqrt{\ln|\mathcal{N}_{\epsilon}|} = \widetilde{O}(\sqrt{d^3H}) ln ∣ N ϵ ∣ = O ( d 3 H ) in the first term of Lemma E.3 . To cancel this d 3 H \sqrt{d^3H} d 3 H factor, we use pointwise monotonicity (V k , h + 1 ≤ V i , h + 1 V_{k,h+1} \leq V_{i,h+1} V k , h + 1 ≤ V i , h + 1 and V ~ k , h + 1 ≥ V ~ i , h + 1 \widetilde{V}_{k,h+1} \geq\widetilde{V}_{i,h+1} V k , h + 1 ≥ V i , h + 1 for k > i k > i k > i ) and the optimism/pessimism sandwich at stage h + 1 h+1 h + 1 to bound the residual variances uniformly for all i < k i < k i < k :
max { [ V h Δ V k , h + 1 ] ( s h i , a h i ) , [ V h Δ V ~ k , h + 1 ] ( s h i , a h i ) } ≤ D i , h 2 d 3 H ≤ σ ˉ i , h 2 2 d 3 H , \max\left\{[\mathbb{V}_h\Delta V_{k,h+1}](s_h^i,a_h^i),[\mathbb{V}_h\Delta\widetilde{V}_{k,h+1}](s_h^i,a_h^i)\right\} \leq\frac{D_{i,h}}{2d^3H} \leq\frac{\bar{\sigma}_{i,h}^2}{2d^3H}, max { [ V h Δ V k , h + 1 ] ( s h i , a h i ) , [ V h Δ V k , h + 1 ] ( s h i , a h i ) } ≤ 2 d 3 H D i , h ≤ 2 d 3 H σ ˉ i , h 2 ,
[ V h Δ ( V k , h + 1 2 ) ] ( s h i , a h i ) ≤ H 2 D i , h 2 d 3 H ≤ H 2 σ ˉ i , h 2 2 d 3 H . [\mathbb{V}_h\Delta(V^2_{k,h+1})](s_h^i,a_h^i) \leq\frac{H^2D_{i,h}}{2d^3H} \leq\frac{H^2\bar{\sigma}_{i,h}^2}{2d^3H}. [ V h Δ ( V k , h + 1 2 )] ( s h i , a h i ) ≤ 2 d 3 H H 2 D i , h ≤ 2 d 3 H H 2 σ ˉ i , h 2 .
Multiplying the normalized variance bound 1 / ( d 3 H ) 1/(d^3H) 1/ ( d 3 H ) (resp. H 2 / ( d 3 H ) H^2/(d^3H) H 2 / ( d 3 H ) ) by the covering entropy ln ∣ N ϵ ∣ = O ~ ( d 3 H ) \ln|\mathcal{N}_{\epsilon}| = \widetilde{O}(d^3H) ln ∣ N ϵ ∣ = O ( d 3 H ) inside the square root of Lemma E.3 cancels the d 3 H d^3H d 3 H factor. Meanwhile, the uncertainty lower bound σ ˉ i , h ≥ γ ∥ ϕ ( s h i , a h i ) ∥ Σ i , h − 1 1 / 2 \bar{\sigma}_{i,h} \geq\gamma\|\phi(s_h^i,a_h^i)\|_{\Sigma_{i,h}^{-1}}^{1/2} σ ˉ i , h ≥ γ ∥ ϕ ( s h i , a h i ) ∥ Σ i , h − 1 1/2 with γ = 2 d 1.5 H 1.5 ι \gamma= 2d^{1.5}H^{1.5}\iota γ = 2 d 1.5 H 1.5 ι ensures that the second-order term 2 ∥ f ∥ ∞ σ ˉ i , h − 2 ∥ ϕ ( s h i , a h i ) ∥ Σ i , h − 1 ≤ 2 ∥ f ∥ ∞ / γ 2 2\|f\|_{\infty}\bar{\sigma}_{i,h}^{-2}\|\phi(s_h^i,a_h^i)\|_{\Sigma_{i,h}^{-1}} \leq2\|f\|_{\infty}/\gamma^2 2∥ f ∥ ∞ σ ˉ i , h − 2 ∥ ϕ ( s h i , a h i ) ∥ Σ i , h − 1 ≤ 2∥ f ∥ ∞ / γ 2 in Lemma E.3 also cancels the d 3 H d^3H d 3 H covering entropy, independent of the variance floor α \alpha α .
Indicator-truncated noise and double induction. A subtle measurability issue arises when formalizing the above bounds: at historical episode i < k i < k i < k , the variance inequalities [ V h V h + 1 ∗ ] ( s h i , a h i ) ≤ σ ˉ i , h 2 [\mathbb{V}_h V^*_{h+1}](s_h^i, a_h^i) \le\bar{\sigma}_{i,h}^2 [ V h V h + 1 ∗ ] ( s h i , a h i ) ≤ σ ˉ i , h 2 and [ V h Δ V k , h + 1 ] ( s h i , a h i ) ≤ σ ˉ i , h 2 / ( d 3 H ) [\mathbb{V}_h \Delta V_{k,h+1}](s_h^i, a_h^i) \le\bar{\sigma}_{i,h}^2/(d^3H) [ V h Δ V k , h + 1 ] ( s h i , a h i ) ≤ σ ˉ i , h 2 / ( d 3 H ) only hold when stage h + 1 h+1 h + 1 satisfies the optimism/pessimism sandwich ( V ˘ m , h + 1 ≤ V h + 1 ∗ ≤ V m , h + 1 ) (\breve{V}_{m,h+1} \le V^*_{h+1} \le V_{m,h+1}) ( V ˘ m , h + 1 ≤ V h + 1 ∗ ≤ V m , h + 1 ) and episode i i i satisfies the stage-h h h Bellman concentration bounds. Conditioning on events from later stages h + 1 h+1 h + 1 would destroy the martingale property at stage h h h . To break this circularity cleanly, for any fixed deterministic function f : S → [ − H 2 , H 2 ] f : \mathcal{S} \to[-H^2,H^2] f : S → [ − H 2 , H 2 ] and threshold ν > 0 \nu> 0 ν > 0 , we introduce the indicator-truncated normalized noise (see (A.9) in Appendix § 9 .§ 2 ):
η i , h ( f , ν ) : = 1 { [ V h f ] ( s h i , a h i ) ≤ ν σ ˉ i , h 2 } ⋅ σ ˉ i , h − 1 ( f ( s h + 1 i ) − [ P h f ] ( s h i , a h i ) ) . \eta_{i,h}(f,\nu) := \mathbb{1}\left\{[\mathbb{V}_h f](s_h^i,a_h^i) \le\nu\bar{\sigma}_{i,h}^2\right\}\cdot\bar{\sigma}_{i,h}^{-1}\left(f(s_{h+1}^i)-[\mathbb{P}_h f](s_h^i,a_h^i)\right). η i , h ( f , ν ) := 1 { [ V h f ] ( s h i , a h i ) ≤ ν σ ˉ i , h 2 } ⋅ σ ˉ i , h − 1 ( f ( s h + 1 i ) − [ P h f ] ( s h i , a h i ) ) .
For any fixed f f f , both [ V h f ] ( s h i , a h i ) [\mathbb{V}_h f](s_h^i,a_h^i) [ V h f ] ( s h i , a h i ) and σ ˉ i , h \bar{\sigma}_{i,h} σ ˉ i , h are F i , h \mathcal{F}_{i,h} F i , h -measurable (determined before s h + 1 i s_{h+1}^i s h + 1 i is drawn), so the indicator prefactor is predictable and the truncated noise η i , h ( f , ν ) \eta_{i,h}(f,\nu) η i , h ( f , ν ) has conditional mean zero (E [ η i , h ( f , ν ) ∣ F i , h ] = 0 \mathbb{E}[\eta_{i,h}(f,\nu)\mid\mathcal{F}_{i,h}]=0 E [ η i , h ( f , ν ) ∣ F i , h ] = 0 ) and conditional variance at most ν \nu ν (E [ η i , h ( f , ν ) 2 ∣ F i , h ] ≤ ν \mathbb{E}[\eta_{i,h}(f,\nu)^2\mid\mathcal{F}_{i,h}]\le\nu E [ η i , h ( f , ν ) 2 ∣ F i , h ] ≤ ν ). Applying Lemma E.3 to η i , h ( f , ν ) \eta_{i,h}(f,\nu) η i , h ( f , ν ) for V h + 1 ∗ V^*_{h+1} V h + 1 ∗ , ( V h + 1 ∗ ) 2 (V^*_{h+1})^2 ( V h + 1 ∗ ) 2 , and the deterministic ϵ \epsilon ϵ -nets defines a single unconditional high-probability event E c o n c \mathcal{E}_{\mathrm{conc}} E conc (Lemma A.4 ). On any sample path in E c o n c \mathcal{E}_{\mathrm{conc}} E conc , we then run a purely deterministic backward induction over h = H → 1 h=H\to1 h = H → 1 and forward induction over k = 1 → K k=1\to K k = 1 → K (Lemma A.5 ): at step ( h , k ) (h,k) ( h , k ) , the induction hypotheses for stage h + 1 h+1 h + 1 and prior episodes i < k i<k i < k imply that all indicators in η i , h ( f , ν ) \eta_{i,h}(f,\nu) η i , h ( f , ν ) for i < k i<k i < k equal 1, establishing three-way Bellman concentration at episode k k k without circular conditioning.
Freedman-Type Analysis for Value Differences
Even with tight confidence radii, a second major obstacle in He et al. (2023) arises when unrolling the value differences across stages. On E c o n c \mathcal{E}_{\mathrm{conc}} E conc , the one-step Bellman estimation error at step ( k , h ) (k,h) ( k , h ) is bounded by the truncated exploration bonus B k , h : = 4 min { β ∥ ϕ ( s h k , a h k ) ∥ Σ k , h − 1 , H } B_{k,h}:=4\min\{\beta\|\phi(s_h^k,a_h^k)\|_{\Sigma_{k,h}^{-1}},H\} B k , h := 4 min { β ∥ ϕ ( s h k , a h k ) ∥ Σ k , h − 1 , H } (and 2 B k , h 2B_{k,h} 2 B k , h for V k , h − V ˘ k , h V_{k,h}-\breve{V}_{k,h} V k , h − V ˘ k , h ). Telescoping the recursive value differences from stage h h h to H H H bounds both the cumulative regret Regret ( K ) ≤ ∑ k = 1 K ( V k , 1 ( s 1 k ) − V 1 π k ( s 1 k ) ) \operatorname{Regret}(K)\le\sum_{k=1}^{K}(V_{k,1}(s_1^k)-V_1^{\pi^k}(s_1^k)) Regret ( K ) ≤ ∑ k = 1 K ( V k , 1 ( s 1 k ) − V 1 π k ( s 1 k )) (at h = 1 h=1 h = 1 ) and the cumulative Bellman differences S π : = ∑ k = 1 K ∑ h = 1 H [ P h ( V k , h + 1 − V h + 1 π k ) ] ( s h k , a h k ) S_\pi:=\sum_{k=1}^{K}\sum_{h=1}^{H}[\mathbb{P}_h(V_{k,h+1}-V_{h+1}^{\pi^k})](s_h^k,a_h^k) S π := ∑ k = 1 K ∑ h = 1 H [ P h ( V k , h + 1 − V h + 1 π k )] ( s h k , a h k ) and S u l : = ∑ k = 1 K ∑ h = 1 H [ P h ( V k , h + 1 − V ˘ k , h + 1 ) ] ( s h k , a h k ) S_{\mathrm{ul}}:=\sum_{k=1}^{K}\sum_{h=1}^{H}[\mathbb{P}_h(V_{k,h+1}-\breve{V}_{k,h+1})](s_h^k,a_h^k) S ul := ∑ k = 1 K ∑ h = 1 H [ P h ( V k , h + 1 − V ˘ k , h + 1 )] ( s h k , a h k ) in terms of ∑ k , h B k , h \sum_{k,h}B_{k,h} ∑ k , h B k , h and the transition martingale difference sums
∑ k = 1 K ∑ h ′ = h H M k , h ′ π = ∑ k = 1 K ∑ h ′ = h H ( [ P h ′ ( V k , h ′ + 1 − V h ′ + 1 π k ) ] ( s h ′ k , a h ′ k ) − ( V k , h ′ + 1 ( s h ′ + 1 k ) − V h ′ + 1 π k ( s h ′ + 1 k ) ) ) , \sum_{k=1}^{K}\sum_{h'=h}^{H}M_{k,h'}^\pi
=
\sum_{k=1}^{K}\sum_{h'=h}^{H}\left([\mathbb{P}_{h'}(V_{k,h'+1}-V_{h'+1}^{\pi^k})](s_{h'}^k,a_{h'}^k)-\left(V_{k,h'+1}(s_{h'+1}^k)-V_{h'+1}^{\pi^k}(s_{h'+1}^k)\right)\right), k = 1 ∑ K h ′ = h ∑ H M k , h ′ π = k = 1 ∑ K h ′ = h ∑ H ( [ P h ′ ( V k , h ′ + 1 − V h ′ + 1 π k )] ( s h ′ k , a h ′ k ) − ( V k , h ′ + 1 ( s h ′ + 1 k ) − V h ′ + 1 π k ( s h ′ + 1 k ) ) ) ,
and analogously for M k , h ′ u l M_{k,h'}^{\mathrm{ul}} M k , h ′ ul . In Lemmas C.1 and C.2 of He et al. (2023), bounding these martingale sums via the Azuma–Hoeffding inequality first incurs an unconditional O ~ ( H 3 K ) \widetilde{O}(\sqrt{H^3K}) O ( H 3 K ) deviation term at h = 1 h=1 h = 1 and an unconditional O ~ ( H 5 K ) \widetilde{O}(\sqrt{H^5K}) O ( H 5 K ) deviation term in S π S_\pi S π and S u l S_{\mathrm{ul}} S ul ; because S π S_\pi S π and S u l S_{\mathrm{ul}} S ul are subsequently used to bound the total estimated variance V e s t = ∑ k , h σ ˉ k , h 2 V_{\mathrm{est}}=\sum_{k,h}\bar{\sigma}_{k,h}^2 V est = ∑ k , h σ ˉ k , h 2 (which in turn controls ∑ k , h B k , h ∝ d H V e s t \sum_{k,h}B_{k,h}\propto\sqrt{dHV_{\mathrm{est}}} ∑ k , h B k , h ∝ d H V est ), these variance-oblivious K \sqrt{K} K terms propagate directly into the final regret upper bound.
To remove these K \sqrt{K} K terms, we combine the time-uniform Freedman inequality (Lemma E.1 ) with a variance self-bounding argument (Lemma B.2), building on techniques developed in tabular and linear RL (Zhang et al. [34 ] ; Chen et al. [7 , 8 ] ; Foster et al. [10 ] ). On E c o n c \mathcal{E}_{\mathrm{conc}} E conc , optimism and pessimism ensure that Z k , h ′ + 1 π : = V k , h ′ + 1 − V h ′ + 1 π k Z_{k,h'+1}^{\pi}:=V_{k,h'+1}-V_{h'+1}^{\pi^k} Z k , h ′ + 1 π := V k , h ′ + 1 − V h ′ + 1 π k takes values in [ 0 , H ] [0,H] [ 0 , H ] . Because any random variable Z ∈ [ 0 , H ] Z\in[0,H] Z ∈ [ 0 , H ] satisfies Var ( Z ) ≤ E [ Z 2 ] ≤ H E [ Z ] \operatorname{Var}(Z)\le\mathbb{E}[Z^2]\le H\mathbb{E}[Z] Var ( Z ) ≤ E [ Z 2 ] ≤ H E [ Z ] , the conditional variance of M k , h ′ π M_{k,h'}^\pi M k , h ′ π is self-bounded by the conditional expectation of Z k , h ′ + 1 π Z_{k,h'+1}^{\pi} Z k , h ′ + 1 π :
E [ ( M k , h ′ π ) 2 ∣ F k , h ′ ] = [ V h ′ Z k , h ′ + 1 π ] ( s h ′ k , a h ′ k ) ≤ H [ P h ′ Z k , h ′ + 1 π ] ( s h ′ k , a h ′ k ) . \mathbb{E}\left[(M_{k,h'}^\pi)^2\mid\mathcal{F}_{k,h'}\right]=[\mathbb{V}_{h'}Z_{k,h'+1}^{\pi}](s_{h'}^k,a_{h'}^k)\le H[\mathbb{P}_{h'}Z_{k,h'+1}^{\pi}](s_{h'}^k,a_{h'}^k). E [ ( M k , h ′ π ) 2 ∣ F k , h ′ ] = [ V h ′ Z k , h ′ + 1 π ] ( s h ′ k , a h ′ k ) ≤ H [ P h ′ Z k , h ′ + 1 π ] ( s h ′ k , a h ′ k ) .
Applying Lemma E.1 with step size η = 1 / ( 2 H 2 ) ≤ 1 / H \eta= 1/(2H^2) \le1/H η = 1/ ( 2 H 2 ) ≤ 1/ H bounds the martingale sum over h ′ ≥ h h' \ge h h ′ ≥ h by S π / ( 2 H ) + 2 H 2 ln ( H / δ ) S_\pi/(2H) + 2H^2 \ln(H/\delta) S π / ( 2 H ) + 2 H 2 ln ( H / δ ) . Summing across the H H H starting stages h ∈ [ H ] h \in[H] h ∈ [ H ] yields S π / 2 S_\pi/2 S π /2 , which is absorbed into the left-hand side S π S_\pi S π to give S π ≤ 2 H ∑ k , h B k , h + 4 H 3 ln ( H / δ ) S_\pi\le2H \sum_{k,h} B_{k,h} + 4H^3 \ln(H/\delta) S π ≤ 2 H ∑ k , h B k , h + 4 H 3 ln ( H / δ ) (and similarly S u l ≤ 4 H ∑ k , h B k , h + 4 H 3 ln ( H / δ ) S_{\mathrm{ul}} \le4H \sum_{k,h} B_{k,h} + 4H^3 \ln(H/\delta) S ul ≤ 4 H ∑ k , h B k , h + 4 H 3 ln ( H / δ ) and Regret ( K ) ≤ 2 ∑ k , h B k , h + 4 H 2 ln ( H / δ ) \operatorname{Regret}(K) \le2 \sum_{k,h} B_{k,h} + 4H^2 \ln(H/\delta) Regret ( K ) ≤ 2 ∑ k , h B k , h + 4 H 2 ln ( H / δ ) ), without any K \sqrt{K} K concentration overhead.
Three-Set Elliptical Bonus Summation
Finally, to bound the exploration bonus sum ∑ k = 1 K min { β ′ ∥ ϕ ( s h k , a h k ) ∥ Σ k , h − 1 , C } \sum_{k=1}^{K} \min\{\beta' \lVert\phi(s_h^k,a_h^k)\rVert_{\Sigma_{k,h}^{-1}}, C\} ∑ k = 1 K min { β ′ ∥ ϕ ( s h k , a h k ) ∥ Σ k , h − 1 , C } without relying on any worst-case upper bound on σ ˉ k , h \bar{\sigma}_{k,h} σ ˉ k , h (which introduced an extra d 4 H 6 d^4H^6 d 4 H 6 factor in He et al. 2023, Lemma E.1), we partition the episodes k ∈ [ K ] k \in[K] k ∈ [ K ] at each stage h h h into three disjoint sets according to the normalized elliptical norm ∥ x k , h ∥ Σ k , h − 1 \lVert x_{k,h}\rVert_{\Sigma_{k,h}^{-1}} ∥ x k , h ∥ Σ k , h − 1 and the active branch in (8) (Lemma B.1 , adapting Zhou and Gu [40 ] , Lemma 4.4):
Episodes with large normalized uncertainty ∥ x k , h ∥ Σ k , h − 1 ≥ 1 \lVert x_{k,h}\rVert_{\Sigma_{k,h}^{-1}} \ge1 ∥ x k , h ∥ Σ k , h − 1 ≥ 1 , whose count is at most 2 d ι 2d\iota 2 d ι by the elliptical potential lemma;
Remaining episodes where the uncertainty floor dominates (σ ˉ k , h = γ ∥ ϕ ( s h k , a h k ) ∥ Σ k , h − 1 1 / 2 \bar{\sigma}_{k,h} = \gamma\lVert\phi(s_h^k,a_h^k)\rVert_{\Sigma_{k,h}^{-1}}^{1/2} σ ˉ k , h = γ ∥ ϕ ( s h k , a h k ) ∥ Σ k , h − 1 1/2 ), where ∥ ϕ ( s h k , a h k ) ∥ Σ k , h − 1 = γ 2 ∥ x k , h ∥ Σ k , h − 1 2 \lVert\phi(s_h^k,a_h^k)\rVert_{\Sigma_{k,h}^{-1}} = \gamma^2 \lVert x_{k,h}\rVert_{\Sigma_{k,h}^{-1}}^2 ∥ ϕ ( s h k , a h k ) ∥ Σ k , h − 1 = γ 2 ∥ x k , h ∥ Σ k , h − 1 2 sums to at most 2 γ 2 d ι 2\gamma^2d\iota 2 γ 2 d ι ;
Remaining episodes where σ ˉ k , h = σ k , h \bar{\sigma}_{k,h} = \sigma_{k,h} σ ˉ k , h = σ k , h , where Cauchy–Schwarz directly bounds the sum by β ′ 2 d ι ∑ k = 1 K σ k , h 2 \beta' \sqrt{2d\iota\sum_{k=1}^{K} \sigma_{k,h}^2} β ′ 2 d ι ∑ k = 1 K σ k , h 2 .
Combined with our tighter uncertainty weight γ = 2 d 1.5 H 1.5 \gamma= 2d^{1.5}H^{1.5} γ = 2 d 1.5 H 1.5 and O ~ ( d ) \widetilde{O}(\sqrt{d}) O ( d ) confidence radii, this reduces the total estimated variance burn-in term from O ~ ( d 10.5 H 16 ) \widetilde{O}(d^{10.5}H^{16}) O ( d 10.5 H 16 ) in He et al. 2023 to O ~ ( d 8 H 7 ) \widetilde{O}(d^8H^7) O ( d 8 H 7 ) (Lemma B.3), and the final regret burn-in term from O ~ ( d 7 H 8 ) \widetilde{O}(d^7H^8) O ( d 7 H 8 ) to O ~ ( d 5 H 4 ) \widetilde{O}(d^5H^4) O ( d 5 H 4 ) .
Conclusion and Future Work
In this paper, we proposed VA-LSVI-UCB++, a computationally efficient algorithm for episodic time-inhomogeneous linear MDPs that achieves variance-dependent regret O ~ ( d H ⋅ min { Var K ∗ , Var K π } + d 5 H 4 ) \widetilde{O}\left(d\sqrt{H}\cdot\min\{\operatorname{Var}^{*}_{K},\operatorname{Var}^{\pi}_{K}\}+d^5H^4\right) O ( d H ⋅ min { Var K ∗ , Var K π } + d 5 H 4 ) adapting to the smaller of the optimal and played policies’ realized trajectory variances with a lower-order remainder free of polynomial-in-K K K dependency. By introducing a simultaneous three-way reference-advantage decomposition coupled with a variance self-bounding technique, deterministic double induction, and a self-bounding Freedman-type analysis for value differences, our approach removes both the O ~ ( K 1 / 4 ) \widetilde{O}(K^{1/4}) O ( K 1/4 ) polynomial overhead of Li and Sun (2023) and Huang et al. [16 ] and the O ~ ( H 3 K ) \widetilde{O}(\sqrt{H^3K}) O ( H 3 K ) variance-oblivious overhead of He et al. (2023). An important direction for future work is to investigate whether horizon-free, variance-dependent regret can be achieved for time-homogeneous linear MDPs under ∑ h = 1 H r h ≤ 1 \sum_{h=1}^{H} r_h \le1 ∑ h = 1 H r h ≤ 1 , and whether such guarantees can be obtained in a computationally efficient manner; while Zhao et al. [39 ] established computationally efficient horizon-free variance-dependent regret for time-homogeneous linear mixture MDPs and Zhang et al. [36 ] recently obtained a computationally inefficient worst-case horizon-free O ~ ( d 5.5 K + d 6.5 ) \widetilde{O}(d^{5.5}\sqrt{K}+d^{6.5}) O ( d 5.5 K + d 6.5 ) bound for time-homogeneous linear MDPs (see Appendix § 12 ), designing a variance-dependent or computationally efficient horizon-free algorithm for time-homogeneous linear MDPs remains open.
Acknowledgement
The proof ideas are developed through an interactive brainstorm between Gemini 4 Argon and the authors. Gemini 4 Argon and Gemini 4 Carbon are employed to further streamline the canonical pillars underlying the proof roadmap. The authors simplify the algorithms and proof techniques, re-organize and verify the proofs, and prepare the draft with further assistance from Gemini 4 Carbon and GPT-6 Astra. The authors take full responsibility for its contents.
Concentration Analysis and Master Double Induction
In this appendix, we establish the rare-switching update bound, the ϵ \epsilon ϵ -covering numbers of the shifted residual function classes, the unconditional high-probability concentration event E c o n c \mathcal{E}_{\mathrm{conc}} E conc , and the master deterministic double induction lemma (Lemma A.5 ).
Function Classes and Covering Numbers
We begin by bounding the number of episodes in which the determinant-doubling condition (Line 4 of Algorithm 1) is triggered.
Lemma A.1 (Number of Value Function Updates). In Algorithm 1, the total number of episodes k ∈ [ K ] k \in[K] k ∈ [ K ] in which update k = t r u e \operatorname{update}_k = \mathbf{true} update k = true (Line 4) is bounded by
1 + d H ln 2 ln ( 1 + K d λ α 2 ) ≤ 1 + d H ι ln 2 = : l max . 1+\frac{dH}{\ln2}\ln\left(1+\frac{K}{d\lambda\alpha^2}\right)\le1+\frac{dH\iota}{\ln2}=:l_{\max}. 1 + ln 2 d H ln ( 1 + d λ α 2 K ) ≤ 1 + ln 2 d H ι =: l m a x .
Proof. Let 1 = k 1 < k 2 < ⋯ < k m ≤ K 1=k_1<k_2<\cdots<k_m\le K 1 = k 1 < k 2 < ⋯ < k m ≤ K denote the episodes in which Line 4 is triggered. By the update rule in Line 4, for each j ∈ { 2 , … , m } j\in\{2,\ldots,m\} j ∈ { 2 , … , m } , there exists at least one stage h ′ ∈ [ H ] h'\in[H] h ′ ∈ [ H ] such that det ( Σ k j , h ′ ) ≥ 2 det ( Σ k j − 1 , h ′ ) \det(\boldsymbol{\Sigma}_{k_j,h'})\ge2\det(\boldsymbol{\Sigma}_{k_{j-1},h'}) det ( Σ k j , h ′ ) ≥ 2 det ( Σ k j − 1 , h ′ ) . Since Σ k , h \boldsymbol{\Sigma}_{k,h} Σ k , h is non-decreasing in k k k in the Loewner order for every stage h ∈ [ H ] h\in[H] h ∈ [ H ] , we have
∏ h = 1 H det ( Σ k j , h ) ≥ 2 ∏ h = 1 H det ( Σ k j − 1 , h ) \prod_{h=1}^{H}\det(\boldsymbol{\Sigma}_{k_j,h})\ge2\prod_{h=1}^{H}\det(\boldsymbol{\Sigma}_{k_{j-1},h}) h = 1 ∏ H det ( Σ k j , h ) ≥ 2 h = 1 ∏ H det ( Σ k j − 1 , h )
for all j ∈ { 2 , … , m } j\in\{2,\ldots,m\} j ∈ { 2 , … , m } . Unrolling this inequality over j = m , m − 1 , … , 2 j=m,m-1,\ldots,2 j = m , m − 1 , … , 2 gives
∏ h = 1 H det ( Σ K , h ) ≥ ∏ h = 1 H det ( Σ k m , h ) ≥ 2 m − 1 ∏ h = 1 H det ( Σ 1 , h ) = 2 m − 1 λ d H , (13) \prod_{h=1}^{H}\det(\boldsymbol{\Sigma}_{K,h})\ge\prod_{h=1}^{H}\det(\boldsymbol{\Sigma}_{k_m,h})\ge2^{m-1}\prod_{h=1}^{H}\det(\boldsymbol{\Sigma}_{1,h})=2^{m-1}\lambda^{dH}, \tag*{(13)} h = 1 ∏ H det ( Σ K , h ) ≥ h = 1 ∏ H det ( Σ k m , h ) ≥ 2 m − 1 h = 1 ∏ H det ( Σ 1 , h ) = 2 m − 1 λ d H , (13)
where the first inequality holds because Σ K , h ⪰ Σ k m , h \boldsymbol{\Sigma}_{K,h}\succeq\boldsymbol{\Sigma}_{k_m,h} Σ K , h ⪰ Σ k m , h and the equality uses Σ 1 , h = λ I \boldsymbol{\Sigma}_{1,h}=\lambda I Σ 1 , h = λ I . On the other hand, for every stage h ∈ [ H ] h\in[H] h ∈ [ H ] , we have
det ( Σ K , h ) ≤ ( tr ( Σ K , h ) d ) d ≤ ( d λ + ∑ k = 1 K − 1 σ ˉ k , h − 2 ∥ ϕ ( s h k , a h k ) ∥ 2 2 d ) d ≤ λ d ( 1 + K d λ α 2 ) d ≤ λ d e d ι . (14) \det(\boldsymbol{\Sigma}_{K,h})\le\left(\frac{\operatorname{tr}(\boldsymbol{\Sigma}_{K,h})}{d}\right)^d\le\left(\frac{d\lambda+\sum_{k=1}^{K-1}\bar{\sigma}_{k,h}^{-2}\lVert\phi(s_h^k,a_h^k)\rVert_2^2}{d}\right)^d\le\lambda^d\left(1+\frac{K}{d\lambda\alpha^2}\right)^d\le\lambda^d e^{d\iota}. \tag*{(14)} det ( Σ K , h ) ≤ ( d tr ( Σ K , h ) ) d ≤ ( d d λ + ∑ k = 1 K − 1 σ ˉ k , h − 2 ∥ ϕ ( s h k , a h k ) ∥ 2 2 ) d ≤ λ d ( 1 + d λ α 2 K ) d ≤ λ d e d ι . (14)
where the first inequality follows from the arithmetic-geometric mean inequality, the third inequality uses ∥ ϕ ( s h k , a h k ) ∥ 2 ≤ 1 \lVert\phi(s_h^k,a_h^k)\rVert_2\le1 ∥ ϕ ( s h k , a h k ) ∥ 2 ≤ 1 and σ ˉ k , h ≥ σ k , h ≥ α \bar{\sigma}_{k,h}\ge\sigma_{k,h}\ge\alpha σ ˉ k , h ≥ σ k , h ≥ α from (7) –(8) , and the last inequality uses (9) . Taking the product of (A.2) over h ∈ [ H ] h\in[H] h ∈ [ H ] and combining with (A.1) gives 2 m − 1 λ d H ≤ λ d H e d H ι 2^{m-1}\lambda^{dH}\le\lambda^{dH}e^{dH\iota} 2 m − 1 λ d H ≤ λ d H e d H ι , which implies m ≤ 1 + d H ι / ln 2 = l max m\le1+dH\iota/\ln2=l_{\max} m ≤ 1 + d H ι / ln 2 = l m a x . □
Next, we establish unconditional Euclidean norm bounds for the regression weight vectors w ^ k , h \widehat{w}_{k,h} w k , h , w ~ k , h \widetilde{w}_{k,h} w k , h , and w ~ k , h \widetilde{w}_{k,h} w k , h .
Lemma A.2 (Norm Bounds for Ridge Regression Weights). For all k ∈ [ K ] k\in[K] k ∈ [ K ] and h ∈ [ H ] h\in[H] h ∈ [ H ] , the regression weight vectors in Lines 10–12 of Algorithm 1 unconditionally satisfy
∥ w ^ k , h ∥ 2 ≤ L , ∥ w ~ k , h ∥ 2 ≤ L , ∥ w ~ k , h ∥ 2 ≤ H L , where L : = H d K α λ . \lVert\widehat{w}_{k,h}\rVert_2\le L,\qquad\lVert\widetilde{w}_{k,h}\rVert_2\le L,\qquad\lVert\widetilde{w}_{k,h}\rVert_2\le HL,\qquad\text{where }L:=\frac{H\sqrt{dK}}{\alpha\sqrt{\lambda}}. ∥ w k , h ∥ 2 ≤ L , ∥ w k , h ∥ 2 ≤ L , ∥ w k , h ∥ 2 ≤ H L , where L := α λ H d K .
Proof. Consider any vector of the form w = Σ k , h − 1 ∑ i = 1 k − 1 σ ˉ i , h − 2 ϕ ( s h i , a h i ) y i \mathbf{w}=\boldsymbol{\Sigma}_{k,h}^{-1}\sum_{i=1}^{k-1}\bar{\sigma}_{i,h}^{-2}\phi(s_h^i,a_h^i)y_i w = Σ k , h − 1 ∑ i = 1 k − 1 σ ˉ i , h − 2 ϕ ( s h i , a h i ) y i where y i ∈ [ 0 , B ] y_i\in[0,B] y i ∈ [ 0 , B ] for all i < k i<k i < k (with B = H B=H B = H for w ^ k , h \widehat{w}_{k,h} w k , h , w ~ k , h \widetilde{w}_{k,h} w k , h and B = H 2 B=H^2 B = H 2 for w ~ k , h \widetilde{w}_{k,h} w k , h ). Let ϕ i : = ϕ ( s h i , a h i ) \phi_i:=\phi(s_h^i,a_h^i) ϕ i := ϕ ( s h i , a h i ) . Then we have
∥ w ∥ 2 2 = ∥ ∑ i = 1 k − 1 σ ˉ i , h − 2 y i Σ k , h − 1 ϕ i ∥ 2 2 ≤ ( k − 1 ) ∑ i = 1 k − 1 σ ˉ i , h − 4 y i 2 ϕ i ⊤ Σ k , h − 2 ϕ i ≤ K B 2 λ α 2 ∑ i = 1 k − 1 σ ˉ i , h − 2 ϕ i ⊤ Σ k , h − 1 ϕ i = K B 2 λ α 2 tr ( Σ k , h − 1 ( Σ k , h − λ I ) ) ≤ d K B 2 λ α 2 , \begin{aligned}
\lVert\mathbf{w}\rVert_2^2=\left\lVert\sum_{i=1}^{k-1}\bar{\sigma}_{i,h}^{-2}y_i\boldsymbol{\Sigma}_{k,h}^{-1}\phi_i\right\rVert_2^2\le(k-1)\sum_{i=1}^{k-1}\bar{\sigma}_{i,h}^{-4}y_i^2\phi_i^\top\boldsymbol{\Sigma}_{k,h}^{-2}\phi_i \\
&\le\frac{KB^{2}}{\lambda\alpha^{2}}\sum_{i=1}^{k-1}\bar{\sigma}_{i,h}^{-2}\phi_i^{\top}\Sigma_{k,h}^{-1}\phi_i = \frac{KB^{2}}{\lambda\alpha^{2}}\operatorname{tr}\left(\Sigma_{k,h}^{-1}\left(\Sigma_{k,h}-\lambda I\right)\right) \le\frac{dKB^{2}}{\lambda\alpha^{2}},
\end{aligned} ∥ w ∥ 2 2 = i = 1 ∑ k − 1 σ ˉ i , h − 2 y i Σ k , h − 1 ϕ i 2 2 ≤ ( k − 1 ) i = 1 ∑ k − 1 σ ˉ i , h − 4 y i 2 ϕ i ⊤ Σ k , h − 2 ϕ i ≤ λ α 2 K B 2 i = 1 ∑ k − 1 σ ˉ i , h − 2 ϕ i ⊤ Σ k , h − 1 ϕ i = λ α 2 K B 2 tr ( Σ k , h − 1 ( Σ k , h − λ I ) ) ≤ λ α 2 d K B 2 ,
where the first inequality follows from the Cauchy–Schwarz inequality, the second inequality holds since y i ∈ [ 0 , B ] y_i \in[0,B] y i ∈ [ 0 , B ] , σ ˉ i , h ≥ α \bar{\sigma}_{i,h} \ge\alpha σ ˉ i , h ≥ α , and Σ k , h − 1 ⪯ λ − 1 I \Sigma_{k,h}^{-1} \preceq\lambda^{-1}I Σ k , h − 1 ⪯ λ − 1 I , and the last inequality uses Σ k , h − λ I ⪯ Σ k , h \Sigma_{k,h}-\lambda I \preceq\Sigma_{k,h} Σ k , h − λ I ⪯ Σ k , h in R d × d \mathbb{R}^{d \times d} R d × d . Taking the square root with B = H B=H B = H and B = H 2 B=H^2 B = H 2 completes the proof.
By Lemmas A.1 and A.2 and the truncation [ ⋅ ] [ 0 , H ] [\cdot]_{[0,H]} [ ⋅ ] [ 0 , H ] in Lines 14–15 of Algorithm 1, for every k ∈ [ K ] k \in[K] k ∈ [ K ] and h ∈ [ H ] h \in[H] h ∈ [ H ] , the optimistic and pessimistic value functions V k , h V_{k,h} V k , h and V ~ k , h \widetilde{V}_{k,h} V k , h unconditionally take values in [ 0 , H ] [0,H] [ 0 , H ] and belong to the function classes
V h : = { max a ∈ A min 1 ≤ i ≤ l max [ r h ( ⋅ , a ) + w i ⊤ ϕ ( ⋅ , a ) + β ∥ ϕ ( ⋅ , a ) ∥ Γ i ] [ 0 , H ] : ∥ w i ∥ 2 ≤ L , 0 ⪯ Γ i ⪯ 1 λ I } , (15) \mathcal{V}_h := \left\{\max_{a \in\mathcal{A}}\min_{1 \le i \le l_{\max}}\left[r_h(\cdot,a)+\mathbf{w}_i^{\top}\phi(\cdot,a)+\beta\|\phi(\cdot,a)\|_{\Gamma_i}\right]_{[0,H]}:\|\mathbf{w}_i\|_2\le L,\ 0\preceq\Gamma_i\preceq\frac{1}{\lambda}I\right\}, \tag*{(15)} V h := { a ∈ A max 1 ≤ i ≤ l m a x min [ r h ( ⋅ , a ) + w i ⊤ ϕ ( ⋅ , a ) + β ∥ ϕ ( ⋅ , a ) ∥ Γ i ] [ 0 , H ] : ∥ w i ∥ 2 ≤ L , 0 ⪯ Γ i ⪯ λ 1 I } , (15)
V ~ h : = { max a ∈ A max 1 ≤ i ≤ l max [ r h ( ⋅ , a ) + w i ⊤ ϕ ( ⋅ , a ) − β ˉ ∥ ϕ ( ⋅ , a ) ∥ Γ i ] [ 0 , H ] : ∥ w i ∥ 2 ≤ L , 0 ⪯ Γ i ⪯ 1 λ I } . (16) \widetilde{\mathcal{V}}_h := \left\{\max_{a \in\mathcal{A}}\max_{1 \le i \le l_{\max}}\left[r_h(\cdot,a)+\mathbf{w}_i^{\top}\phi(\cdot,a)-\bar{\beta}\|\phi(\cdot,a)\|_{\Gamma_i}\right]_{[0,H]}:\|\mathbf{w}_i\|_2\le L,\ 0\preceq\Gamma_i\preceq\frac{1}{\lambda}I\right\}. \tag*{(16)} V h := { a ∈ A max 1 ≤ i ≤ l m a x max [ r h ( ⋅ , a ) + w i ⊤ ϕ ( ⋅ , a ) − β ˉ ∥ ϕ ( ⋅ , a ) ∥ Γ i ] [ 0 , H ] : ∥ w i ∥ 2 ≤ L , 0 ⪯ Γ i ⪯ λ 1 I } . (16)
To analyze the three-way reference-advantage decomposition in (12) , we define the three shifted residual function classes at stage h + 1 h+1 h + 1 from (A.3)–(A.4) as
Δ V h + 1 : = { V − V h + 1 ∗ : V ∈ V h + 1 } , Δ V ~ h + 1 : = { V h + 1 ∗ − V ~ : V ~ ∈ V ~ h + 1 } , \Delta\mathcal{V}_{h+1}:=\left\{V-V^*_{h+1}:V\in\mathcal{V}_{h+1}\right\},\qquad\Delta\widetilde{\mathcal{V}}_{h+1}:=\left\{V^*_{h+1}-\widetilde{V}:\widetilde{V}\in\widetilde{\mathcal{V}}_{h+1}\right\}, Δ V h + 1 := { V − V h + 1 ∗ : V ∈ V h + 1 } , Δ V h + 1 := { V h + 1 ∗ − V : V ∈ V h + 1 } ,
Δ ( V h + 1 2 ) : = { V 2 − ( V h + 1 ∗ ) 2 : V ∈ V h + 1 } . (17) \Delta(\mathcal{V}_{h+1}^{2}):=\left\{V^2-(V^*_{h+1})^2:V\in\mathcal{V}_{h+1}\right\}. \tag*{(17)} Δ ( V h + 1 2 ) := { V 2 − ( V h + 1 ∗ ) 2 : V ∈ V h + 1 } . (17)
Because every V ∈ V h + 1 V \in\mathcal{V}_{h+1} V ∈ V h + 1 and V ~ ∈ V ~ h + 1 \widetilde{V} \in\widetilde{\mathcal{V}}_{h+1} V ∈ V h + 1 takes values in [ 0 , H ] [0,H] [ 0 , H ] , every f ∈ Δ V h + 1 ∪ Δ V ~ h + 1 f \in\Delta\mathcal{V}_{h+1}\cup\Delta\widetilde{\mathcal{V}}_{h+1} f ∈ Δ V h + 1 ∪ Δ V h + 1 satisfies ∥ f ∥ ∞ ≤ H \|f\|_{\infty}\le H ∥ f ∥ ∞ ≤ H , and every f ∈ Δ ( V h + 1 2 ) f \in\Delta(\mathcal{V}_{h+1}^{2}) f ∈ Δ ( V h + 1 2 ) satisfies ∥ f ∥ ∞ ≤ H 2 \|f\|_{\infty}\le H^2 ∥ f ∥ ∞ ≤ H 2 .
Lemma A.3 (ϵ \epsilon ϵ -Covering Numbers of Residual Function Classes). Let dist ( f 1 , f 2 ) : = ∥ f 1 − f 2 ∥ ∞ \operatorname{dist}(f_1,f_2):=\|f_1-f_2\|_{\infty} dist ( f 1 , f 2 ) := ∥ f 1 − f 2 ∥ ∞ . Set the covering radii
ϵ 1 : = α 2 λ 16 d 3 H 2 K = 1 16 d 3 H K 2 , (18) \epsilon_1:=\frac{\alpha^2\sqrt{\lambda}}{16d^3H^2K}=\frac{1}{16d^3HK^2}, \tag*{(18)} ϵ 1 := 16 d 3 H 2 K α 2 λ = 16 d 3 H K 2 1 , (18)
ϵ 2 : = 2 H ϵ 1 = α 2 λ 8 d 3 H K = 1 8 d 3 K 2 . (19) \epsilon_2:=2H\epsilon_1=\frac{\alpha^2\sqrt{\lambda}}{8d^3HK}=\frac{1}{8d^3K^2}. \tag*{(19)} ϵ 2 := 2 H ϵ 1 = 8 d 3 H K α 2 λ = 8 d 3 K 2 1 . (19)
Then for every stage h ∈ [ H ] h\in[H] h ∈ [ H ] , the residual classes in (A.5) admit deterministic nets N ϵ 1 ( Δ V h + 1 ) ⊂ Δ V h + 1 \mathcal{N}_{\epsilon_1}(\Delta\mathcal{V}_{h+1})\subset\Delta\mathcal{V}_{h+1} N ϵ 1 ( Δ V h + 1 ) ⊂ Δ V h + 1 , N ϵ 1 ( Δ V ~ h + 1 ) ⊂ Δ V ~ h + 1 \mathcal{N}_{\epsilon_1}(\Delta\widetilde{\mathcal{V}}_{h+1})\subset\Delta\widetilde{\mathcal{V}}_{h+1} N ϵ 1 ( Δ V h + 1 ) ⊂ Δ V h + 1 , and N ϵ 2 ( Δ ( V h + 1 2 ) ) ⊂ Δ ( V h + 1 2 ) \mathcal{N}_{\epsilon_2}(\Delta(\mathcal{V}_{h+1}^{2}))\subset\Delta(\mathcal{V}_{h+1}^{2}) N ϵ 2 ( Δ ( V h + 1 2 )) ⊂ Δ ( V h + 1 2 ) (with radii ϵ 1 , ϵ 1 , ϵ 2 \epsilon_1,\epsilon_1,\epsilon_2 ϵ 1 , ϵ 1 , ϵ 2 ) satisfying
max { ln ∣ N ϵ 1 ( Δ V h + 1 ) ∣ , ln ∣ N ϵ 1 ( Δ V ~ h + 1 ) ∣ , ln ∣ N ϵ 2 ( Δ ( V h + 1 2 ) ) ∣ } ≤ C N d 3 H ι 0 , \max\left\{\ln|\mathcal{N}_{\epsilon_1}(\Delta\mathcal{V}_{h+1})|,\ \ln|\mathcal{N}_{\epsilon_1}(\Delta\widetilde{\mathcal{V}}_{h+1})|,\ \ln|\mathcal{N}_{\epsilon_2}(\Delta(\mathcal{V}_{h+1}^{2}))|\right\}\le C_{\mathcal{N}}d^3H\iota_0, max { ln ∣ N ϵ 1 ( Δ V h + 1 ) ∣ , ln ∣ N ϵ 1 ( Δ V h + 1 ) ∣ , ln ∣ N ϵ 2 ( Δ ( V h + 1 2 )) ∣ } ≤ C N d 3 H ι 0 ,
where C N > 0 C_{\mathcal{N}}>0 C N > 0 is an absolute constant.
Proof. Since V h + 1 ∗ V^*_{h+1} V h + 1 ∗ and ( V h + 1 ∗ ) 2 (V^*_{h+1})^2 ( V h + 1 ∗ ) 2 are fixed deterministic functions, shifting by V h + 1 ∗ V^*_{h+1} V h + 1 ∗ or ( V h + 1 ∗ ) 2 (V^*_{h+1})^2 ( V h + 1 ∗ ) 2 is an isometry under dist ( ⋅ , ⋅ ) \operatorname{dist}(\cdot,\cdot) dist ( ⋅ , ⋅ ) . Moreover, for any V 1 , V 2 ∈ V h + 1 V_1,V_2\in\mathcal{V}_{h+1} V 1 , V 2 ∈ V h + 1 , since V 1 ( s ) , V 2 ( s ) ∈ [ 0 , H ] V_1(s),V_2(s)\in[0,H] V 1 ( s ) , V 2 ( s ) ∈ [ 0 , H ] , we have
∣ V 1 2 ( s ) − V 2 2 ( s ) ∣ = ∣ V 1 ( s ) + V 2 ( s ) ∣ ⋅ ∣ V 1 ( s ) − V 2 ( s ) ∣ ≤ 2 H dist ( V 1 , V 2 ) . |V_1^2(s)-V_2^2(s)|=|V_1(s)+V_2(s)|\cdot|V_1(s)-V_2(s)|\le2H\operatorname{dist}(V_1,V_2). ∣ V 1 2 ( s ) − V 2 2 ( s ) ∣ = ∣ V 1 ( s ) + V 2 ( s ) ∣ ⋅ ∣ V 1 ( s ) − V 2 ( s ) ∣ ≤ 2 H dist ( V 1 , V 2 ) .
Hence any ϵ 1 \epsilon_1 ϵ 1 -net of V h + 1 \mathcal{V}_{h+1} V h + 1 directly induces an ϵ 2 \epsilon_2 ϵ 2 -net of Δ ( V h + 1 2 ) \Delta(\mathcal{V}_{h+1}^{2}) Δ ( V h + 1 2 ) with ϵ 2 = 2 H ϵ 1 \epsilon_2=2H\epsilon_1 ϵ 2 = 2 H ϵ 1 of identical cardinality.
To bound the ϵ 1 \epsilon_1 ϵ 1 -covering number of V h + 1 \mathcal{V}_{h+1} V h + 1 (and identically of V ~ h + 1 \widetilde{\mathcal{V}}_{h+1} V h + 1 since β ˉ = β \bar{\beta}=\beta β ˉ = β ), observe that max a ∈ A \max_{a\in\mathcal{A}} max a ∈ A , min 1 ≤ i ≤ l max \min_{1\le i\le l_{\max}} min 1 ≤ i ≤ l m a x (or max 1 ≤ i ≤ l max \max_{1\le i\le l_{\max}} max 1 ≤ i ≤ l m a x ), and [ ⋅ ] [ 0 , H ] [\cdot]_{[0,H]} [ ⋅ ] [ 0 , H ] are all 1-Lipschitz under ∥ ⋅ ∥ ∞ \|\cdot\|_{\infty} ∥ ⋅ ∥ ∞ . Thus, for any V 1 , V 2 ∈ V h + 1 V_1,V_2\in\mathcal{V}_{h+1} V 1 , V 2 ∈ V h + 1 parameterized by { ( w 1 , i , Γ 1 , i ) } i = 1 l max \{(\mathbf{w}_{1,i},\Gamma_{1,i})\}_{i=1}^{l_{\max}} {( w 1 , i , Γ 1 , i ) } i = 1 l m a x and { ( w 2 , i , Γ 2 , i ) } i = 1 l max \{(\mathbf{w}_{2,i},\Gamma_{2,i})\}_{i=1}^{l_{\max}} {( w 2 , i , Γ 2 , i ) } i = 1 l m a x , we have
dist ( V 1 , V 2 ) ≤ max 1 ≤ i ≤ l max sup s ∈ S , a ∈ A ∣ ( w 1 , i − w 2 , i ) ⊤ ϕ ( s , a ) + β ( ∥ ϕ ( s , a ) ∥ Γ 1 , i − ∥ ϕ ( s , a ) ∥ Γ 2 , i ) ∣ ≤ max 1 ≤ i ≤ l max ∥ w 1 , i − w 2 , i ∥ 2 + β max 1 ≤ i ≤ l max ∥ Γ 1 , i − Γ 2 , i ∥ F (20) \begin{aligned}
\operatorname{dist}(V_1,V_2)\le\max_{1\le i\le l_{\max}}\sup_{s\in\mathcal{S},a\in\mathcal{A}}\left|(\mathbf{w}_{1,i}-\mathbf{w}_{2,i})^{\top}\phi(s,a)+\beta\left(\|\phi(s,a)\|_{\Gamma_{1,i}}-\|\phi(s,a)\|_{\Gamma_{2,i}}\right)\right| \\
&\leq\max_{1\leq i\leq l_{\max}}\lVert\mathbf{w}_{1,i}-\mathbf{w}_{2,i}\rVert_{2}+\beta\max_{1\leq i\leq l_{\max}}\sqrt{\lVert\boldsymbol{\Gamma}_{1,i}-\boldsymbol{\Gamma}_{2,i}\rVert_{F}} \tag*{(20)}
\end{aligned} dist ( V 1 , V 2 ) ≤ 1 ≤ i ≤ l m a x max s ∈ S , a ∈ A sup ( w 1 , i − w 2 , i ) ⊤ ϕ ( s , a ) + β ( ∥ ϕ ( s , a ) ∥ Γ 1 , i − ∥ ϕ ( s , a ) ∥ Γ 2 , i ) ≤ 1 ≤ i ≤ l m a x max ∥ w 1 , i − w 2 , i ∥ 2 + β 1 ≤ i ≤ l m a x max ∥ Γ 1 , i − Γ 2 , i ∥ F (20)
where the second inequality follows from ∥ ϕ ( s , a ) ∥ 2 ≤ 1 \lVert\phi(s,a)\rVert_{2}\leq1 ∥ ϕ ( s , a ) ∥ 2 ≤ 1 , ∣ u − v ∣ ≤ ∣ u − v ∣ \lvert\sqrt{u}-\sqrt{v}\rvert\leq\sqrt{\lvert u-v\rvert} ∣ u − v ∣ ≤ ∣ u − v ∣ for u , v ≥ 0 u,v\geq0 u , v ≥ 0 , and the Cauchy–Schwarz inequality. Notice that B 2 d ( L ) : = { w ∈ R d : ∥ w ∥ 2 ≤ L } \mathbb{B}_{2}^{d}(L):=\{\mathbf{w}\in\mathbb{R}^{d}:\lVert\mathbf{w}\rVert_{2}\leq L\} B 2 d ( L ) := { w ∈ R d : ∥ w ∥ 2 ≤ L } is a Euclidean ball of radius L L L , and S + d ( λ − 1 ) : = { Γ ∈ R d × d : 0 ⪯ Γ ⪯ λ − 1 I } \mathbb{S}_{+}^{d}(\lambda^{-1}):=\{\boldsymbol{\Gamma}\in\mathbb{R}^{d\times d}:0\preceq\boldsymbol{\Gamma}\preceq\lambda^{-1}\mathbf{I}\} S + d ( λ − 1 ) := { Γ ∈ R d × d : 0 ⪯ Γ ⪯ λ − 1 I } is contained in the Frobenius ball of radius d / λ \sqrt{d}/\lambda d / λ in R d × d \mathbb{R}^{d\times d} R d × d . Since the ϵ \epsilon ϵ -covering number of any subset of a normed ball of radius R R R (by elements within the subset) is bounded by the ( ϵ / 2 ) (\epsilon/2) ( ϵ /2 ) -covering number of the ambient ball, Lemma E.8 implies that B 2 d ( L ) \mathbb{B}_{2}^{d}(L) B 2 d ( L ) admits an ( ϵ 1 / 2 ) (\epsilon_{1}/2) ( ϵ 1 /2 ) -net of size at most ( 1 + 4 L / ϵ 1 ) d (1+4L/\epsilon_{1})^{d} ( 1 + 4 L / ϵ 1 ) d and S + d ( λ − 1 ) \mathbb{S}_{+}^{d}(\lambda^{-1}) S + d ( λ − 1 ) admits an ( ϵ 1 2 / ( 4 β 2 ) ) (\epsilon_{1}^{2}/(4\beta^{2})) ( ϵ 1 2 / ( 4 β 2 )) -net under ∥ ⋅ ∥ F \lVert\cdot\rVert_{F} ∥ ⋅ ∥ F of size at most ( 1 + 16 d β 2 / ( λ ϵ 1 2 ) ) d 2 (1+16\sqrt{d}\beta^{2}/(\lambda\epsilon_{1}^{2}))^{d^{2}} ( 1 + 16 d β 2 / ( λ ϵ 1 2 ) ) d 2 . Taking the product over 1 ≤ i ≤ l max 1\leq i\leq l_{\max} 1 ≤ i ≤ l m a x , it follows from (A.8) that
ln ∣ N ϵ 1 ( Δ V h + 1 ) ∣ ≤ d l max ln ( 1 + 4 L ϵ 1 ) + d 2 l max ln ( 1 + 16 d β 2 λ ϵ 1 2 ) ≤ C N d 3 H ι ι 0 , \ln\lvert\mathcal{N}_{\epsilon_{1}}(\Delta\mathcal{V}_{h+1})\rvert\leq dl_{\max}\ln\left(1+\frac{4L}{\epsilon_{1}}\right)+d^{2}l_{\max}\ln\left(1+\frac{16\sqrt{d}\beta^{2}}{\lambda\epsilon_{1}^{2}}\right)\leq C_{N}d^{3}H\iota\iota_{0}, ln ∣ N ϵ 1 ( Δ V h + 1 )∣ ≤ d l m a x ln ( 1 + ϵ 1 4 L ) + d 2 l m a x ln ( 1 + λ ϵ 1 2 16 d β 2 ) ≤ C N d 3 H ι ι 0 ,
where the second inequality uses l max = O ( d H ι ) l_{\max}=O(dH\iota) l m a x = O ( d H ι ) (Lemma A.1 ) and ln ( 1 + 4 L / ϵ 1 ) + ln ( 1 + 16 d β 2 / ( λ ϵ 1 2 ) ) = O ( ι 0 ) \ln(1+4L/\epsilon_{1})+\ln(1+16\sqrt{d}\beta^{2}/(\lambda\epsilon_{1}^{2}))=O(\iota_{0}) ln ( 1 + 4 L / ϵ 1 ) + ln ( 1 + 16 d β 2 / ( λ ϵ 1 2 )) = O ( ι 0 ) .
High-Probability Concentration Event
Recall that F i , h \mathcal{F}_{i,h} F i , h denotes the σ \sigma σ -algebra generated by all random variables up to ( s h i , a h i ) (s_{h}^{i},a_{h}^{i}) ( s h i , a h i ) prior to observing s h + 1 i s_{h+1}^{i} s h + 1 i . By construction of Algorithm 1, σ ˉ i , h \bar{\sigma}_{i,h} σ ˉ i , h , x i , h : = σ ˉ i , h − 1 ϕ ( s h i , a h i ) \mathbf{x}_{i,h}:=\bar{\sigma}_{i,h}^{-1}\phi(s_{h}^{i},a_{h}^{i}) x i , h := σ ˉ i , h − 1 ϕ ( s h i , a h i ) , and Σ i , h \boldsymbol{\Sigma}_{i,h} Σ i , h are all F i , h \mathcal{F}_{i,h} F i , h -measurable. For any fixed deterministic function f : S → [ − H 2 , H 2 ] f:\mathcal{S}\to[-H^{2},H^{2}] f : S → [ − H 2 , H 2 ] and any deterministic variance threshold ν > 0 \nu>0 ν > 0 , we define the indicator-truncated normalized noise:
η i , h ( f , ν ) : = 1 { [ V h f ] ( s h i , a h i ) ≤ ν σ ˉ i , h 2 } σ ˉ i , h − 1 ( f ( s h + 1 i ) − [ P h f ] ( s h i , a h i ) ) . (21) \eta_{i,h}(f,\nu):=\mathbf{1}\left\{[\mathbb{V}_{h}f](s_{h}^{i},a_{h}^{i})\leq\nu\bar{\sigma}_{i,h}^{2}\right\}\bar{\sigma}_{i,h}^{-1}\left(f(s_{h+1}^{i})-[\mathbb{P}_{h}f](s_{h}^{i},a_{h}^{i})\right). \tag*{(21)} η i , h ( f , ν ) := 1 { [ V h f ] ( s h i , a h i ) ≤ ν σ ˉ i , h 2 } σ ˉ i , h − 1 ( f ( s h + 1 i ) − [ P h f ] ( s h i , a h i ) ) . (21)
Because the indicator 1 { [ V h f ] ( s h i , a h i ) ≤ ν σ ˉ i , h 2 } \mathbf{1}\{[\mathbb{V}_{h}f](s_{h}^{i},a_{h}^{i})\leq\nu\bar{\sigma}_{i,h}^{2}\} 1 {[ V h f ] ( s h i , a h i ) ≤ ν σ ˉ i , h 2 } is F i , h \mathcal{F}_{i,h} F i , h -measurable and η i , h ( f , ν ) \eta_{i,h}(f,\nu) η i , h ( f , ν ) is observed once s h + 1 i s_{h+1}^{i} s h + 1 i is drawn (F i + 1 , h \mathcal{F}_{i+1,h} F i + 1 , h -measurable), η i , h ( f , ν ) \eta_{i,h}(f,\nu) η i , h ( f , ν ) has conditional mean zero (E [ η i , h ( f , ν ) ∣ F i , h ] = 0 \mathbb{E}[\eta_{i,h}(f,\nu)\mid\mathcal{F}_{i,h}]=0 E [ η i , h ( f , ν ) ∣ F i , h ] = 0 ) and conditional variance at most ν \nu ν (E [ η i , h ( f , ν ) 2 ∣ F i , h ] ≤ ν \mathbb{E}[\eta_{i,h}(f,\nu)^{2}\mid\mathcal{F}_{i,h}]\leq\nu E [ η i , h ( f , ν ) 2 ∣ F i , h ] ≤ ν ). This allows us to apply the variance-aware self-normalized concentration inequality (Lemma E.3 ) with filtration { F i , h } i ≥ 1 \{\mathcal{F}_{i,h}\}_{i\geq1} { F i , h } i ≥ 1 to establish the following high-probability event E c o n c \mathcal{E}_{\mathrm{conc}} E conc .
Lemma A.4 (Unconditional High-Probability Concentration Event E c o n c \mathcal{E}_{\mathrm{conc}} E conc ). Define E c o n c \mathcal{E}_{\mathrm{conc}} E conc as the event that for all stages h ∈ [ H ] h\in[H] h ∈ [ H ] and all episodes k ∈ [ K ] k\in[K] k ∈ [ K ] , the following five inequalities hold simultaneously:
∥ ∑ i = 1 k − 1 x i , h η i , h ( V h + 1 ∗ , 1 ) ∥ Σ k , h − 1 ≤ β 0 , (22) \left\lVert\sum_{i=1}^{k-1}\mathbf{x}_{i,h}\eta_{i,h}(V_{h+1}^{*},1)\right\rVert_{\boldsymbol{\Sigma}_{k,h}^{-1}}\leq\beta_{0}, \tag*{(22)} i = 1 ∑ k − 1 x i , h η i , h ( V h + 1 ∗ , 1 ) Σ k , h − 1 ≤ β 0 , (22) ∥ ∑ i = 1 k − 1 x i , h η i , h ( ( V h + 1 ∗ ) 2 , 4 H 2 ) ∥ Σ k , h − 1 ≤ 2 H β 0 , (23) \left\lVert\sum_{i=1}^{k-1}\mathbf{x}_{i,h}\eta_{i,h}\left((V_{h+1}^{*})^{2},4H^{2}\right)\right\rVert_{\boldsymbol{\Sigma}_{k,h}^{-1}}\leq2H\beta_{0}, \tag*{(23)} i = 1 ∑ k − 1 x i , h η i , h ( ( V h + 1 ∗ ) 2 , 4 H 2 ) Σ k , h − 1 ≤ 2 H β 0 , (23) sup V ~ ∈ N ϵ 1 ( Δ V h + 1 ) ∥ ∑ i = 1 k − 1 x i , h η i , h ( V ~ , 1 d 3 H ) ∥ Σ k , h − 1 ≤ β 0 , (24) \sup_{\widetilde{V}\in\mathcal{N}_{\epsilon_{1}}(\Delta\mathcal{V}_{h+1})}\left\lVert\sum_{i=1}^{k-1}\mathbf{x}_{i,h}\eta_{i,h}\left(\widetilde{V},\frac{1}{d^{3}H}\right)\right\rVert_{\boldsymbol{\Sigma}_{k,h}^{-1}}\leq\beta_{0}, \tag*{(24)} V ∈ N ϵ 1 ( Δ V h + 1 ) sup i = 1 ∑ k − 1 x i , h η i , h ( V , d 3 H 1 ) Σ k , h − 1 ≤ β 0 , (24) sup U ~ ∈ N ϵ 1 ( Δ V ^ h + 1 ) ∥ ∑ i = 1 k − 1 x i , h η i , h ( U ~ , 1 d 3 H ) ∥ Σ k , h − 1 ≤ β 0 , (25) \sup_{\widetilde{U}\in\mathcal{N}_{\epsilon_{1}}(\Delta\widehat{\mathcal{V}}_{h+1})}\left\lVert\sum_{i=1}^{k-1}\mathbf{x}_{i,h}\eta_{i,h}\left(\widetilde{U},\frac{1}{d^{3}H}\right)\right\rVert_{\boldsymbol{\Sigma}_{k,h}^{-1}}\leq\beta_{0}, \tag*{(25)} U ∈ N ϵ 1 ( Δ V h + 1 ) sup i = 1 ∑ k − 1 x i , h η i , h ( U , d 3 H 1 ) Σ k , h − 1 ≤ β 0 , (25)
sup W ~ ∈ N ϵ 2 ( Δ ( V ~ h + 1 2 ) ) ∥ ∑ i = 1 k − 1 x i , h η i , h ( W ~ , H 2 d 3 H ) ∥ Σ k , h − 1 ≤ H β 0 , (26) \sup_{\widetilde{W}\in\mathcal{N}_{\epsilon_2}(\Delta(\widetilde{V}_{h+1}^{2}))}\left\|\sum_{i=1}^{k-1}x_{i,h}\eta_{i,h}\left(\widetilde{W},\frac{H^2}{d^3H}\right)\right\|_{\Sigma_{k,h}^{-1}}\le H\beta_0, \tag*{(26)} W ∈ N ϵ 2 ( Δ ( V h + 1 2 )) sup i = 1 ∑ k − 1 x i , h η i , h ( W , d 3 H H 2 ) Σ k , h − 1 ≤ H β 0 , (26)
where β 0 : = C 0 ( d ι 2 ι 0 + ι ι 0 ) \beta_0:=C_0(\sqrt{d}\iota^2\iota_0+\iota\iota_0) β 0 := C 0 ( d ι 2 ι 0 + ι ι 0 ) with the explicit absolute constant C 0 : = 12 C N + 2 + 60 ( C N + 2 ) C_0:=12\sqrt{C_{\mathcal{N}}+2}+60(C_{\mathcal{N}}+2) C 0 := 12 C N + 2 + 60 ( C N + 2 ) . Then P ( E c o n c ) ≥ 1 − δ \mathbb{P}(\mathcal{E}_{\mathrm{conc}})\ge1-\delta P ( E conc ) ≥ 1 − δ .
Proof. Fix any stage h ∈ [ H ] h\in[H] h ∈ [ H ] , any deterministic function f : S → [ − H 2 , H 2 ] f:\mathcal{S}\to[-H^2,H^2] f : S → [ − H 2 , H 2 ] , and any threshold ν > 0 \nu>0 ν > 0 . Since 1 { [ ∇ h f ] ( s h i , a h i ) ≤ ν σ ˉ i , h } \mathbf{1}\{[\nabla_h f](s_h^i,a_h^i)\le\nu\bar{\sigma}_{i,h}\} 1 {[ ∇ h f ] ( s h i , a h i ) ≤ ν σ ˉ i , h } is F i , h \mathcal{F}_{i,h} F i , h -measurable, the sequence { x i , h , η i , h ( f , ν ) } i = 1 K \{x_{i,h},\eta_{i,h}(f,\nu)\}_{i=1}^{K} { x i , h , η i , h ( f , ν ) } i = 1 K satisfies the conditions of Lemma E.3 with E [ η i , h ( f , ν ) ∣ F i , h ] = 0 \mathbb{E}[\eta_{i,h}(f,\nu)\mid\mathcal{F}_{i,h}]=0 E [ η i , h ( f , ν ) ∣ F i , h ] = 0 and
E [ η i , h ( f , ν ) 2 ∣ F i , h ] = 1 { [ ∇ h f ] ( s h i , a h i ) ≤ ν σ ˉ i , h } σ ˉ i , h − 2 [ ∇ h f ] ( s h i , a h i ) ≤ ν , \mathbb{E}[\eta_{i,h}(f,\nu)^2\mid\mathcal{F}_{i,h}]
=\mathbf{1}\{[\nabla_h f](s_h^i,a_h^i)\le\nu\bar{\sigma}_{i,h}\}\bar{\sigma}_{i,h}^{-2}[\nabla_h f](s_h^i,a_h^i)\le\nu, E [ η i , h ( f , ν ) 2 ∣ F i , h ] = 1 {[ ∇ h f ] ( s h i , a h i ) ≤ ν σ ˉ i , h } σ ˉ i , h − 2 [ ∇ h f ] ( s h i , a h i ) ≤ ν ,
∥ x i , h ∥ 2 ≤ α − 1 , ∣ η i , h ( f , ν ) ∣ ≤ 2 ∥ f ∥ ∞ α − 1 ≤ 2 H 2 α − 1 = : R . \|x_{i,h}\|_2\le\alpha^{-1},\qquad
|\eta_{i,h}(f,\nu)|\le2\|f\|_\infty\alpha^{-1}\le2H^2\alpha^{-1}=:R. ∥ x i , h ∥ 2 ≤ α − 1 , ∣ η i , h ( f , ν ) ∣ ≤ 2∥ f ∥ ∞ α − 1 ≤ 2 H 2 α − 1 =: R .
Furthermore, for all k ∈ [ K ] k\in[K] k ∈ [ K ] , the second-order term in Lemma E.3 satisfies
max 1 ≤ i ≤ k − 1 ∣ η i , h ( f , ν ) ∣ min { 1 , ∥ x i , h ∥ Σ i , h − 1 } ≤ max 1 ≤ i ≤ k − 1 2 ∥ f ∥ ∞ σ ˉ i , h − 2 ∥ ϕ ( s h i , a h i ) ∥ Σ i , h − 1 ≤ 2 ∥ f ∥ ∞ γ 2 , (27) \max_{1\le i\le k-1}|\eta_{i,h}(f,\nu)|\min\left\{1,\|x_{i,h}\|_{\Sigma_{i,h}^{-1}}\right\}
\le\max_{1\le i\le k-1}2\|f\|_\infty\bar{\sigma}_{i,h}^{-2}\|\phi(s_h^i,a_h^i)\|_{\Sigma_{i,h}^{-1}}
\le\frac{2\|f\|_\infty}{\gamma^2}, \tag*{(27)} 1 ≤ i ≤ k − 1 max ∣ η i , h ( f , ν ) ∣ min { 1 , ∥ x i , h ∥ Σ i , h − 1 } ≤ 1 ≤ i ≤ k − 1 max 2∥ f ∥ ∞ σ ˉ i , h − 2 ∥ ϕ ( s h i , a h i ) ∥ Σ i , h − 1 ≤ γ 2 2∥ f ∥ ∞ , (27)
where the second inequality uses σ ˉ i , h ≥ γ ∥ ϕ ( s h i , a h i ) ∥ Σ i , h − 1 1 / 2 \bar{\sigma}_{i,h}\ge\gamma\|\phi(s_h^i,a_h^i)\|_{\Sigma_{i,h}^{-1}}^{1/2} σ ˉ i , h ≥ γ ∥ ϕ ( s h i , a h i ) ∥ Σ i , h − 1 1/2 from (8) . We set the discretization parameter in Lemma E.3 to ϵ c : = 1 / ( d 3 H 2 ) \epsilon_c:=1/(d^3H^2) ϵ c := 1/ ( d 3 H 2 ) and note that ln ( 1 + K α − 2 / ( d λ ) ) ≤ ι \ln(1+K\alpha^{-2}/(d\lambda))\le\iota ln ( 1 + K α − 2 / ( d λ )) ≤ ι and ι ≥ 1 \iota\ge1 ι ≥ 1 by (9) .
To prove (A.10) and (A.11), we apply Lemma E.3 to f = V h + 1 ∗ f=V_{h+1}^* f = V h + 1 ∗ (with ν = 1 \nu=1 ν = 1 , ∥ f ∥ ∞ ≤ H \|f\|_\infty\le H ∥ f ∥ ∞ ≤ H ) and to f = ( V h + 1 ∗ ) 2 f=(V_{h+1}^*)^2 f = ( V h + 1 ∗ ) 2 (with ν = 4 H 2 \nu=4H^2 ν = 4 H 2 , ∥ f ∥ ∞ ≤ H 2 \|f\|_\infty\le H^2 ∥ f ∥ ∞ ≤ H 2 ), each with failure probability δ ′ = δ / ( 5 H ) \delta'=\delta/(5H) δ ′ = δ / ( 5 H ) . Then ι c ( δ ′ ) = ln ( 32 ( ln ( R / ϵ c ) + 1 ) K 2 / δ ′ ) ≤ 2 ι 0 \iota_c(\delta')=\ln(32(\ln(R/\epsilon_c)+1)K^2/\delta')\le2\iota_0 ι c ( δ ′ ) = ln ( 32 ( ln ( R / ϵ c ) + 1 ) K 2 / δ ′ ) ≤ 2 ι 0 . For f = V h + 1 ∗ f=V_{h+1}^* f = V h + 1 ∗ , with probability at least 1 − δ / ( 5 H ) 1-\delta/(5H) 1 − δ / ( 5 H ) , simultaneously for all k ∈ [ K ] k\in[K] k ∈ [ K ] :
∥ ∑ i = 1 k − 1 x i , h η i , h ( V h + 1 ∗ , 1 ) ∥ Σ k , h − 1 ≤ 12 2 d ι ι 0 + 48 ι 0 ⋅ 2 H γ 2 + 12 ι 0 ϵ c ≤ ( 12 2 + 60 ) ( d ι 2 ι 0 + ι ι 0 ) ≤ β 0 , \left\|\sum_{i=1}^{k-1}x_{i,h}\eta_{i,h}(V_{h+1}^*,1)\right\|_{\Sigma_{k,h}^{-1}}
\le12\sqrt{2d\iota\iota_0}+48\iota_0\cdot\frac{2H}{\gamma^2}+12\iota_0\epsilon_c
\le(12\sqrt{2}+60)(\sqrt{d}\iota^2\iota_0+\iota\iota_0)
\le\beta_0, i = 1 ∑ k − 1 x i , h η i , h ( V h + 1 ∗ , 1 ) Σ k , h − 1 ≤ 12 2 d ι ι 0 + 48 ι 0 ⋅ γ 2 2 H + 12 ι 0 ϵ c ≤ ( 12 2 + 60 ) ( d ι 2 ι 0 + ι ι 0 ) ≤ β 0 ,
where the first inequality follows from Lemma E.3 and (A.15), and the second uses 2 H / γ 2 ≤ 1 2H/\gamma^2\le1 2 H / γ 2 ≤ 1 , ϵ c ≤ 1 \epsilon_c\le1 ϵ c ≤ 1 , and ι ≥ 1 \iota\ge1 ι ≥ 1 . Similarly, for f = ( V h + 1 ∗ ) 2 f=(V_{h+1}^*)^2 f = ( V h + 1 ∗ ) 2 with ν = 4 H 2 \nu=4H^2 ν = 4 H 2 and ∥ f ∥ ∞ ≤ H 2 \|f\|_\infty\le H^2 ∥ f ∥ ∞ ≤ H 2 , with probability at least 1 − δ / ( 5 H ) 1-\delta/(5H) 1 − δ / ( 5 H ) , simultaneously for all k ∈ [ K ] k\in[K] k ∈ [ K ] :
∥ ∑ i = 1 k − 1 x i , h η i , h ( ( V h + 1 ∗ ) 2 , 4 H 2 ) ∥ Σ k , h − 1 ≤ 24 H 2 d ι ι 0 + 48 ι 0 ⋅ 2 H 2 γ 2 + 12 ι 0 ϵ c ≤ 2 H β 0 , \left\|\sum_{i=1}^{k-1}x_{i,h}\eta_{i,h}((V_{h+1}^*)^2,4H^2)\right\|_{\Sigma_{k,h}^{-1}}
\le24H\sqrt{2d\iota\iota_0}+48\iota_0\cdot\frac{2H^2}{\gamma^2}+12\iota_0\epsilon_c
\le2H\beta_0, i = 1 ∑ k − 1 x i , h η i , h (( V h + 1 ∗ ) 2 , 4 H 2 ) Σ k , h − 1 ≤ 24 H 2 d ι ι 0 + 48 ι 0 ⋅ γ 2 2 H 2 + 12 ι 0 ϵ c ≤ 2 H β 0 ,
where the first inequality applies Lemma E.3 and (A.15), and the second uses 2 H 2 / γ 2 ≤ H 2H^2/\gamma^2\le H 2 H 2 / γ 2 ≤ H and ϵ c ≤ 1 ≤ H \epsilon_c\le1\le H ϵ c ≤ 1 ≤ H .
To prove (A.12), (A.13), and (A.14), we apply Lemma E.3 to each fixed function in the corresponding deterministic ϵ \epsilon ϵ -net N \mathcal{N} N with failure probability δ ′ = δ / ( 5 H ∣ N ∣ ) \delta'=\delta/(5H|\mathcal{N}|) δ ′ = δ / ( 5 H ∣ N ∣ ) and take a union bound over N \mathcal{N} N . By Lemma A.3, we have ι c ( δ ′ ) ≤ ln ∣ N ∣ + 2 ι 0 ≤ ( C N + 2 ) d 3 H ι ι 0 \iota_c(\delta')\le\ln|\mathcal{N}|+2\iota_0\le(C_{\mathcal{N}}+2)d^3H\iota\iota_0 ι c ( δ ′ ) ≤ ln ∣ N ∣ + 2 ι 0 ≤ ( C N + 2 ) d 3 H ι ι 0 . For N ϵ 1 ( Δ V h + 1 ) \mathcal{N}_{\epsilon_1}(\Delta V_{h+1}) N ϵ 1 ( Δ V h + 1 ) (and identically for N ϵ 1 ( Δ V ~ h + 1 ) \mathcal{N}_{\epsilon_1}(\Delta\widetilde{V}_{h+1}) N ϵ 1 ( Δ V h + 1 ) ), with probability at least 1 − δ / ( 5 H ) 1-\delta/(5H) 1 − δ / ( 5 H ) , simultaneously for all k ∈ [ K ] k\in[K] k ∈ [ K ] :
sup V ~ ∈ N ϵ 1 ( Δ V ~ h + 1 ) ∥ ∑ i = 1 k − 1 x i , h η i , h ( V ~ , 1 d 3 H ) ∥ Σ k , h − 1 ≤ 12 1 d 3 H ⋅ d ι ⋅ ( C N + 2 ) d 3 H ι ι 0 + 24 ( C N + 2 ) d 3 H ι ι 0 ⋅ 2 H γ 2 + 6 ( C N + 2 ) d 3 H ι ι 0 ⋅ ϵ c = 12 C N + 2 d ℓ 2 ℓ 0 + 12 ( C N + 2 ) ℓ 0 H ℓ + 6 ( C N + 2 ) ℓ 0 H ≤ C 0 ( d ℓ 2 ℓ 0 + ℓ 0 ) = β 0 , \begin{aligned}
\sup_{\widetilde{V}\in\mathcal{N}_{\epsilon_1}(\Delta\widetilde{V}_{h+1})}
\left\|\sum_{i=1}^{k-1}x_{i,h}\eta_{i,h}\left(\widetilde{V},\frac{1}{d^3H}\right)\right\|_{\Sigma_{k,h}^{-1}}
\le12\sqrt{\frac{1}{d^3H}\cdot d\iota\cdot(C_{\mathcal{N}}+2)d^3H\iota\iota_0}
+24(C_{\mathcal{N}}+2)d^3H\iota\iota_0\cdot\frac{2H}{\gamma^2}
+6(C_{\mathcal{N}}+2)d^3H\iota\iota_0\cdot\epsilon_c \\
&= 12\sqrt{C_N}+2\sqrt{d\ell^2\ell_0}+12(C_N+2)\frac{\ell_0}{H\ell}+6(C_N+2)\frac{\ell_0}{H}
\le C_0\left(\sqrt{d\ell^2\ell_0}+\ell_0\right)=\beta_0,
\end{aligned} V ∈ N ϵ 1 ( Δ V h + 1 ) sup i = 1 ∑ k − 1 x i , h η i , h ( V , d 3 H 1 ) Σ k , h − 1 ≤ 12 d 3 H 1 ⋅ d ι ⋅ ( C N + 2 ) d 3 H ι ι 0 + 24 ( C N + 2 ) d 3 H ι ι 0 ⋅ γ 2 2 H + 6 ( C N + 2 ) d 3 H ι ι 0 ⋅ ϵ c = 12 C N + 2 d ℓ 2 ℓ 0 + 12 ( C N + 2 ) H ℓ ℓ 0 + 6 ( C N + 2 ) H ℓ 0 ≤ C 0 ( d ℓ 2 ℓ 0 + ℓ 0 ) = β 0 ,
where the first inequality applies Lemma E.3 and (A.15) with ν = 1 / ( d 3 H ) \nu=1/(d^3H) ν = 1/ ( d 3 H ) and ∥ V ‾ ∥ ∞ ≤ H \lVert\overline{V}\rVert_\infty\le H ∥ V ∥ ∞ ≤ H , the equality substitutes γ 2 = 4 d 3 H 3 ℓ 2 \gamma^2=4d^3H^3\ell^2 γ 2 = 4 d 3 H 3 ℓ 2 and ϵ c = 1 / ( d 3 H 2 ) \epsilon_c=1/(d^3H^2) ϵ c = 1/ ( d 3 H 2 ) , and the last inequality uses H ≥ 1 H\ge1 H ≥ 1 and ℓ ≥ 1 \ell\ge1 ℓ ≥ 1 . Finally, for N ϵ 2 ( Δ ( V ‾ h + 1 2 ) ) N_{\epsilon^2}(\Delta(\overline{V}_{h+1}^2)) N ϵ 2 ( Δ ( V h + 1 2 )) in (A.14), replacing ( ν , ∥ V ‾ ∥ ∞ ) = ( 1 / ( d 3 H ) , H ) (\nu,\lVert\overline{V}\rVert_\infty)=(1/(d^3H),H) ( ν , ∥ V ∥ ∞ ) = ( 1/ ( d 3 H ) , H ) by ( H 2 / ( d 3 H ) , H 2 ) (H^2/(d^3H),H^2) ( H 2 / ( d 3 H ) , H 2 ) multiplies each of the three terms above by at most H H H , yielding an upper bound of H β 0 H\beta_0 H β 0 with probability at least 1 − δ / ( 5 H ) 1-\delta/(5H) 1 − δ / ( 5 H ) . Taking a union bound over all 5 H 5H 5 H events completes the proof. □
Master Double Induction
We now prove that on the single event E c o n c \mathcal{E}_{\mathrm{conc}} E conc , all three regression estimators enjoy Bernstein concentration, the variance estimator σ k , h 2 \sigma^2_{k,h} σ k , h 2 upper-bounds [ V h V h + 1 ∗ ] ( s h k , a h k ) [\mathbb{V}_hV^*_{h+1}](s_h^k,a_h^k) [ V h V h + 1 ∗ ] ( s h k , a h k ) up to known error terms, the residual variances remain uniformly bounded for all future episodes j ≥ k j\ge k j ≥ k , and the value function estimates sandwich Q h ∗ Q_h^* Q h ∗ , V h ∗ V_h^* V h ∗ .
Lemma A.5 (Master Double Induction Lemma). On the event E c o n c \mathcal{E}_{\mathrm{conc}} E conc , for every stage h ∈ [ H ] h\in[H] h ∈ [ H ] and every episode k ∈ [ K ] k\in[K] k ∈ [ K ] , the following four properties hold simultaneously:
(P1)k , h _{k,h} k , h Three-way Bellman concentration. For all ( s , a ) ∈ S × A (s,a)\in\mathcal{S}\times\mathcal{A} ( s , a ) ∈ S × A ,
∣ w ^ k , h ⊤ ϕ ( s , a ) − [ P h V k , h + 1 ] ( s , a ) ∣ ≤ β ∥ ϕ ( s , a ) ∥ Σ k , h − 1 , (28) \left|\widehat{w}_{k,h}^{\top}\phi(s,a)-[\mathbb{P}_hV_{k,h+1}](s,a)\right|\le\beta\lVert\phi(s,a)\rVert_{\Sigma_{k,h}^{-1}}, \tag*{(28)} w k , h ⊤ ϕ ( s , a ) − [ P h V k , h + 1 ] ( s , a ) ≤ β ∥ ϕ ( s , a ) ∥ Σ k , h − 1 , (28) ∣ w ~ k , h ⊤ ϕ ( s , a ) − [ P h V ~ k , h + 1 ] ( s , a ) ∣ ≤ β ‾ ∥ ϕ ( s , a ) ∥ Σ k , h − 1 , (29) \left|\widetilde{w}_{k,h}^{\top}\phi(s,a)-[\mathbb{P}_h\widetilde{V}_{k,h+1}](s,a)\right|\le\overline{\beta}\lVert\phi(s,a)\rVert_{\Sigma_{k,h}^{-1}}, \tag*{(29)} w k , h ⊤ ϕ ( s , a ) − [ P h V k , h + 1 ] ( s , a ) ≤ β ∥ ϕ ( s , a ) ∥ Σ k , h − 1 , (29) ∣ w ~ k , h ⊤ ϕ ( s , a ) − [ P h V k , h + 1 2 ] ( s , a ) ∣ ≤ β ~ ∥ ϕ ( s , a ) ∥ Σ k , h − 1 . (30) \left|\widetilde{w}_{k,h}^{\top}\phi(s,a)-[\mathbb{P}_hV_{k,h+1}^2](s,a)\right|\le\widetilde{\beta}\lVert\phi(s,a)\rVert_{\Sigma_{k,h}^{-1}}. \tag*{(30)} w k , h ⊤ ϕ ( s , a ) − [ P h V k , h + 1 2 ] ( s , a ) ≤ β ∥ ϕ ( s , a ) ∥ Σ k , h − 1 . (30) (P2)k , h _{k,h} k , h Variance estimation accuracy.
∣ [ V ‾ k , h V k , h + 1 ] ( s h k , a h k ) − [ V h V k , h + 1 ] ( s h k , a h k ) ∣ ≤ E k , h , (31) \left|[\overline{\mathbb{V}}_{k,h}V_{k,h+1}](s_h^k,a_h^k)-[\mathbb{V}_hV_{k,h+1}](s_h^k,a_h^k)\right|\le E_{k,h}, \tag*{(31)} [ V k , h V k , h + 1 ] ( s h k , a h k ) − [ V h V k , h + 1 ] ( s h k , a h k ) ≤ E k , h , (31) ∣ [ V ‾ k , h V k , h + 1 ] ( s h k , a h k ) − [ V h V h + 1 ∗ ] ( s h k , a h k ) ∣ ≤ E k , h + D k , h , (32) \left|[\overline{\mathbb{V}}_{k,h}V_{k,h+1}](s_h^k,a_h^k)-[\mathbb{V}_hV^*_{h+1}](s_h^k,a_h^k)\right|\le E_{k,h}+D_{k,h}, \tag*{(32)} [ V k , h V k , h + 1 ] ( s h k , a h k ) − [ V h V h + 1 ∗ ] ( s h k , a h k ) ≤ E k , h + D k , h , (32) and consequently σ ‾ k , h 2 ≥ σ k , h 2 ≥ [ V h V h + 1 ∗ ] ( s h k , a h k ) + α 2 \overline{\sigma}_{k,h}^2\ge\sigma_{k,h}^2\ge[\mathbb{V}_hV^*_{h+1}](s_h^k,a_h^k)+\alpha^2 σ k , h 2 ≥ σ k , h 2 ≥ [ V h V h + 1 ∗ ] ( s h k , a h k ) + α 2 .
(P3)k , h _{k,h} k , h Uniform residual variance bounds. For every j ∈ { k , k + 1 , … , K } j\in\{k,k+1,\ldots,K\} j ∈ { k , k + 1 , … , K } ,
[ V h ( V j , h + 1 − V h + 1 ∗ ) 2 ] ( s h k , a h k ) ≤ D k , h 2 d 3 H , (33) [\mathbb{V}_h(V_{j,h+1}-V^*_{h+1})^2](s_h^k,a_h^k)\le\frac{D_{k,h}}{2d^3H}, \tag*{(33)} [ V h ( V j , h + 1 − V h + 1 ∗ ) 2 ] ( s h k , a h k ) ≤ 2 d 3 H D k , h , (33) [ V h ( V h + 1 ∗ − V ~ j , h + 1 ) 2 ] ( s h k , a h k ) ≤ D k , h 2 d 3 H , (34) [\mathbb{V}_h(V^*_{h+1}-\widetilde{V}_{j,h+1})^2](s_h^k,a_h^k)\le\frac{D_{k,h}}{2d^3H}, \tag*{(34)} [ V h ( V h + 1 ∗ − V j , h + 1 ) 2 ] ( s h k , a h k ) ≤ 2 d 3 H D k , h , (34) [ V h ( V j , h + 1 2 − ( V h + 1 ∗ ) 2 ) ] ( s h k , a h k ) ≤ H 2 D k , h 2 d 3 H . (35) [\mathbb{V}_h(V_{j,h+1}^2-(V^*_{h+1})^2)](s_h^k,a_h^k)\le\frac{H^2D_{k,h}}{2d^3H}. \tag*{(35)} [ V h ( V j , h + 1 2 − ( V h + 1 ∗ ) 2 )] ( s h k , a h k ) ≤ 2 d 3 H H 2 D k , h . (35) (P4)k , h _{k,h} k , h Optimism and pessimism sandwich. For all ( s , a ) ∈ S × A (s,a)\in\mathcal{S}\times\mathcal{A} ( s , a ) ∈ S × A ,
Q ~ k , h ( s , a ) ≤ Q h ∗ ( s , a ) ≤ Q k , h ( s , a ) , V ~ k , h ( s ) ≤ V h ∗ ( s ) ≤ V k , h ( s ) . \widetilde{Q}_{k,h}(s,a)\le Q_h^*(s,a)\le Q_{k,h}(s,a),\qquad\widetilde{V}_{k,h}(s)\le V_h^*(s)\le V_{k,h}(s). Q k , h ( s , a ) ≤ Q h ∗ ( s , a ) ≤ Q k , h ( s , a ) , V k , h ( s ) ≤ V h ∗ ( s ) ≤ V k , h ( s ) .
Proof. Fix any sample path in E c o n c \mathcal{E}_{\mathrm{conc}} E conc . We proceed by outer backward induction on h = H , H − 1 , … , 1 h=H,H-1,\ldots,1 h = H , H − 1 , … , 1 and inner forward induction on k = 1 , 2 , … , K k=1,2,\ldots,K k = 1 , 2 , … , K . Outer induction hypothesis. Fix stage h ∈ [ H ] h \in[H] h ∈ [ H ] and assume that at stage h + 1 h+1 h + 1 , property (P4) m , h + 1 _{m,h+1} m , h + 1 holds for all episodes m ∈ [ K ] m \in[K] m ∈ [ K ] :
0 ≤ V ~ m , h + 1 ( s ) ≤ V h + 1 ∗ ( s ) ≤ V m , h + 1 ( s ) ≤ H ∀ s ∈ S , m ∈ [ K ] . (36) 0 \le\widetilde{V}_{m,h+1}(s) \le V^{*}_{h+1}(s) \le V_{m,h+1}(s) \le H \qquad\forall s \in\mathcal{S},\ m \in[K]. \tag*{(36)} 0 ≤ V m , h + 1 ( s ) ≤ V h + 1 ∗ ( s ) ≤ V m , h + 1 ( s ) ≤ H ∀ s ∈ S , m ∈ [ K ] . (36)
For the base case h = H h=H h = H , V m , H + 1 ( s ) = V H + 1 ∗ ( s ) = V ~ m , H + 1 ( s ) = 0 V_{m,H+1}(s)=V^{*}_{H+1}(s)=\widetilde{V}_{m,H+1}(s)=0 V m , H + 1 ( s ) = V H + 1 ∗ ( s ) = V m , H + 1 ( s ) = 0 , so (A.24) holds trivially.
Inner induction on k = 1 , … , K k=1,\ldots,K k = 1 , … , K for (P1)k , h _{k,h} k , h –(P3)k , h _{k,h} k , h . Fix k ∈ [ K ] k \in[K] k ∈ [ K ] and assume that for all i ∈ { 1 , … , k − 1 } i \in\{1,\ldots,k-1\} i ∈ { 1 , … , k − 1 } (vacuously true when k = 1 k=1 k = 1 ), properties (P1) i , h _{i,h} i , h , (P2) i , h _{i,h} i , h , and (P3) i , h _{i,h} i , h hold.
Step 1: Proof of (P1)k , h _{k,h} k , h . Consider any target function f ∈ { V k , h + 1 , V ~ k , h + 1 , V k , h + 1 2 } f \in\{V_{k,h+1},\widetilde{V}_{k,h+1},V^{2}_{k,h+1}\} f ∈ { V k , h + 1 , V k , h + 1 , V k , h + 1 2 } with true Bellman backup parameter w h ( f ) w_h(f) w h ( f ) (Proposition 3.2 ) and weighted ridge estimator w ^ ( f ) : = Σ k , h − 1 ∑ i = 1 k − 1 σ ‾ i , h − 2 ϕ ( s h i , a h i ) f ( s h + 1 i ) \widehat{w}(f):=\Sigma_{k,h}^{-1}\sum_{i=1}^{k-1}\overline{\sigma}_{i,h}^{-2}\phi(s_h^i,a_h^i)f(s_{h+1}^i) w ( f ) := Σ k , h − 1 ∑ i = 1 k − 1 σ i , h − 2 ϕ ( s h i , a h i ) f ( s h + 1 i ) . For all ( s , a ) ∈ S × A (s,a)\in\mathcal{S}\times\mathcal{A} ( s , a ) ∈ S × A , we have
∣ w ^ ( f ) ⊤ ϕ ( s , a ) − [ P h f ] ( s , a ) ∣ = ∣ ⟨ w ^ ( f ) − w h ( f ) , ϕ ( s , a ) ⟩ ∣ ≤ ∥ w ^ ( f ) − w h ( f ) ∥ Σ k , h ∥ ϕ ( s , a ) ∥ Σ k , h − 1 , (37) \left|\widehat{w}(f)^{\top}\phi(s,a)-[\mathbb{P}_h f](s,a)\right|
=\left|\left\langle\widehat{w}(f)-w_h(f),\phi(s,a)\right\rangle\right|
\le\left\|\widehat{w}(f)-w_h(f)\right\|_{\Sigma_{k,h}}\left\|\phi(s,a)\right\|_{\Sigma_{k,h}^{-1}}, \tag*{(37)} w ( f ) ⊤ ϕ ( s , a ) − [ P h f ] ( s , a ) = ∣ ⟨ w ( f ) − w h ( f ) , ϕ ( s , a ) ⟩ ∣ ≤ ∥ w ( f ) − w h ( f ) ∥ Σ k , h ∥ ϕ ( s , a ) ∥ Σ k , h − 1 , (37)
where the equality uses [ P h f ] ( s , a ) = w h ( f ) ⊤ ϕ ( s , a ) [\mathbb{P}_h f](s,a)=w_h(f)^{\top}\phi(s,a) [ P h f ] ( s , a ) = w h ( f ) ⊤ ϕ ( s , a ) and the inequality follows from the Cauchy–Schwarz inequality. Expanding w h ( f ) = Σ k , h − 1 ( λ I + ∑ i = 1 k − 1 σ ‾ i , h − 2 ϕ ( s h i , a h i ) ϕ ( s h i , a h i ) ⊤ ) w h ( f ) w_h(f)=\Sigma_{k,h}^{-1}\left(\lambda I+\sum_{i=1}^{k-1}\overline{\sigma}_{i,h}^{-2}\phi(s_h^i,a_h^i)\phi(s_h^i,a_h^i)^{\top}\right)w_h(f) w h ( f ) = Σ k , h − 1 ( λ I + ∑ i = 1 k − 1 σ i , h − 2 ϕ ( s h i , a h i ) ϕ ( s h i , a h i ) ⊤ ) w h ( f ) yields
∥ w ^ ( f ) − w h ( f ) ∥ Σ k , h = ∥ − λ w h ( f ) + ∑ i = 1 k − 1 σ ‾ i , h − 2 ϕ ( s h i , a h i ) ( f ( s h + 1 i ) − [ P h f ] ( s h i , a h i ) ) ∥ Σ k , h − 1 ≤ λ ∥ w h ( f ) ∥ 2 + ∥ ∑ i = 1 k − 1 σ ‾ i , h − 2 ϕ ( s h i , a h i ) ( f ( s h + 1 i ) − [ P h f ] ( s h i , a h i ) ) ∥ Σ k , h − 1 , (38) \begin{aligned}
\left\|\widehat{w}(f)-w_h(f)\right\|_{\Sigma_{k,h}}
&=\left\|-\lambda w_h(f)+\sum_{i=1}^{k-1}\overline{\sigma}_{i,h}^{-2}\phi(s_h^i,a_h^i)\left(f(s_{h+1}^i)-[\mathbb{P}_h f](s_h^i,a_h^i)\right)\right\|_{\Sigma_{k,h}^{-1}} \\
&\le\sqrt{\lambda}\left\|w_h(f)\right\|_2+
\left\|\sum_{i=1}^{k-1}\overline{\sigma}_{i,h}^{-2}\phi(s_h^i,a_h^i)\left(f(s_{h+1}^i)-[\mathbb{P}_h f](s_h^i,a_h^i)\right)\right\|_{\Sigma_{k,h}^{-1}}, \tag*{(38)}
\end{aligned} ∥ w ( f ) − w h ( f ) ∥ Σ k , h = − λ w h ( f ) + i = 1 ∑ k − 1 σ i , h − 2 ϕ ( s h i , a h i ) ( f ( s h + 1 i ) − [ P h f ] ( s h i , a h i ) ) Σ k , h − 1 ≤ λ ∥ w h ( f ) ∥ 2 + i = 1 ∑ k − 1 σ i , h − 2 ϕ ( s h i , a h i ) ( f ( s h + 1 i ) − [ P h f ] ( s h i , a h i ) ) Σ k , h − 1 , (38)
where the inequality applies the triangle inequality and ∥ λ w h ( f ) ∥ Σ k , h − 1 ≤ λ ∥ w h ( f ) ∥ 2 \left\|\lambda w_h(f)\right\|_{\Sigma_{k,h}^{-1}}\le\sqrt{\lambda}\left\|w_h(f)\right\|_2 ∥ λ w h ( f ) ∥ Σ k , h − 1 ≤ λ ∥ w h ( f ) ∥ 2 (since Σ k , h ⪰ λ I \Sigma_{k,h}\succeq\lambda I Σ k , h ⪰ λ I ). By Proposition 3.2 , the true parameters w k , h ∗ : = w h ( V k , h + 1 ) w^{*}_{k,h}:=w_h(V_{k,h+1}) w k , h ∗ := w h ( V k , h + 1 ) , w ~ k , h ∗ : = w h ( V ~ k , h + 1 ) \widetilde{w}^{*}_{k,h}:=w_h(\widetilde{V}_{k,h+1}) w k , h ∗ := w h ( V k , h + 1 ) , and w ~ k , h ∗ : = w h ( V k , h + 1 2 ) \widetilde{w}^{*}_{k,h}:=w_h(V^{2}_{k,h+1}) w k , h ∗ := w h ( V k , h + 1 2 ) satisfy ∥ w k , h ∗ ∥ 2 ≤ H d \left\|w^{*}_{k,h}\right\|_2\le H\sqrt{d} w k , h ∗ 2 ≤ H d , ∥ w ~ k , h ∗ ∥ 2 ≤ H d \left\|\widetilde{w}^{*}_{k,h}\right\|_2\le H\sqrt{d} w k , h ∗ 2 ≤ H d , and ∥ w ~ k , h ∗ ∥ 2 ≤ H 2 d \left\|\widetilde{w}^{*}_{k,h}\right\|_2\le H^2\sqrt{d} w k , h ∗ 2 ≤ H 2 d . When k = 1 k=1 k = 1 , the sum over i i i in (A.26) is empty, so (A.25)–(A.26) immediately yield (P1) 1 , h _{1,h} 1 , h since H d λ ≤ β = β ‾ H\sqrt{d\lambda}\le\beta=\overline{\beta} H d λ ≤ β = β and H 2 d λ ≤ β ~ H^2\sqrt{d\lambda}\le\widetilde{\beta} H 2 d λ ≤ β by (10) . Now suppose k ≥ 2 k\ge2 k ≥ 2 . We bound (A.26) for each of the three estimators in turn.
Case (a): Concentration of w ^ k , h \widehat{w}_{k,h} w k , h . Decompose V k , h + 1 = V h + 1 ∗ + Δ V k , h + 1 V_{k,h+1}=V^{*}_{h+1}+\Delta V_{k,h+1} V k , h + 1 = V h + 1 ∗ + Δ V k , h + 1 as in (12) , with Δ V k , h + 1 ∈ Δ V h + 1 \Delta V_{k,h+1}\in\Delta\mathcal{V}_{h+1} Δ V k , h + 1 ∈ Δ V h + 1 . By (A.26) and the triangle inequality, we have
∥ w ^ k , h − w k , h ∗ ∥ Σ k , h ≤ H d λ + ∥ ∑ i = 1 k − 1 σ ‾ i , h − 2 ϕ ( s h i , a h i ) ( V h + 1 ∗ ( s h + 1 i ) − [ P h V h + 1 ∗ ] ( s h i , a h i ) ) ∥ Σ k , h − 1 ⏟ = : J 1 + ∥ ∑ i = 1 k − 1 σ ‾ i , h − 2 ϕ ( s h i , a h i ) ( Δ V k , h + 1 ( s h + 1 i ) − [ P h Δ V k , h + 1 ] ( s h i , a h i ) ) ∥ Σ k , h − 1 ⏟ = : J 2 . (39) \begin{aligned}
\left\|\widehat{w}_{k,h}-w^{*}_{k,h}\right\|_{\Sigma_{k,h}}
&\le H\sqrt{d\lambda}
+\underbrace{\left\|\sum_{i=1}^{k-1}\overline{\sigma}_{i,h}^{-2}\phi(s_h^i,a_h^i)\left(V^{*}_{h+1}(s_{h+1}^i)-[\mathbb{P}_hV^{*}_{h+1}](s_h^i,a_h^i)\right)\right\|_{\Sigma_{k,h}^{-1}}}_{=:J_1}\\
&\quad+\underbrace{\left\|\sum_{i=1}^{k-1}\overline{\sigma}_{i,h}^{-2}\phi(s_h^i,a_h^i)\left(\Delta V_{k,h+1}(s_{h+1}^i)-[\mathbb{P}_h\Delta V_{k,h+1}](s_h^i,a_h^i)\right)\right\|_{\Sigma_{k,h}^{-1}}}_{=:J_2}. \tag*{(39)}
\end{aligned} w k , h − w k , h ∗ Σ k , h ≤ H d λ + =: J 1 i = 1 ∑ k − 1 σ i , h − 2 ϕ ( s h i , a h i ) ( V h + 1 ∗ ( s h + 1 i ) − [ P h V h + 1 ∗ ] ( s h i , a h i ) ) Σ k , h − 1 + =: J 2 i = 1 ∑ k − 1 σ i , h − 2 ϕ ( s h i , a h i ) ( Δ V k , h + 1 ( s h + 1 i ) − [ P h Δ V k , h + 1 ] ( s h i , a h i ) ) Σ k , h − 1 . (39)
For J 1 J_1 J 1 , by (P2) i , h _{i,h} i , h at each i ∈ { 1 , … , k − 1 } i\in\{1,\ldots,k-1\} i ∈ { 1 , … , k − 1 } , we have [ V h V h + 1 ∗ ] ( s h i , a h i ) ≤ σ ‾ i , h 2 [\mathbb{V}_hV^{*}_{h+1}](s_h^i,a_h^i)\le\overline{\sigma}_{i,h}^{2} [ V h V h + 1 ∗ ] ( s h i , a h i ) ≤ σ i , h 2 , so the indicator in η i , h ( V h + 1 ∗ , 1 ) \eta_{i,h}(V^{*}_{h+1},1) η i , h ( V h + 1 ∗ , 1 ) equals 1 1 1 for all i < k i<k i < k , and (A.10) gives J 1 ≤ β 0 J_1\le\beta_0 J 1 ≤ β 0 . For J 2 J_2 J 2 , let V ~ ∈ N ϵ 1 ( Δ V h + 1 ) \widetilde V \in\mathcal{N}_{\epsilon_1}(\Delta V_{h+1}) V ∈ N ϵ 1 ( Δ V h + 1 ) satisfy ∥ Δ V k , h + 1 − V ~ ∥ ∞ ≤ ϵ 1 \|\Delta V_{k,h+1}-\widetilde V\|_\infty\le\epsilon_1 ∥Δ V k , h + 1 − V ∥ ∞ ≤ ϵ 1 . Since ∥ Δ V k , h + 1 ∥ ∞ ≤ H \|\Delta V_{k,h+1}\|_\infty\le H ∥Δ V k , h + 1 ∥ ∞ ≤ H and ∥ V ~ ∥ ∞ ≤ H \|\widetilde V\|_\infty\le H ∥ V ∥ ∞ ≤ H , for every i ∈ { 1 , … , k − 1 } i\in\{1,\ldots,k-1\} i ∈ { 1 , … , k − 1 } we have
[ V h V ~ ] ( s h i , a h i ) − [ V h Δ V k , h + 1 ] ( s h i , a h i ) = [ P h ( V ~ 2 − ( Δ V k , h + 1 ) 2 ) ] ( s h i , a h i ) + ( [ P h Δ V k , h + 1 ] ( s h i , a h i ) ) 2 − ( [ P h V ~ ] ( s h i , a h i ) ) 2 ≤ 4 H ∥ Δ V k , h + 1 − V ~ ∥ ∞ ≤ 4 H ϵ 1 ≤ α 2 2 d 3 H , (40) \begin{aligned}
[V_h\widetilde V](s_h^i,a_h^i)-[V_h\Delta V_{k,h+1}](s_h^i,a_h^i)
&=[P_h(\widetilde V^2-(\Delta V_{k,h+1})^2)](s_h^i,a_h^i)\\
&\quad+([P_h\Delta V_{k,h+1}](s_h^i,a_h^i))^2-([P_h\widetilde V](s_h^i,a_h^i))^2\\
&\le4H\|\Delta V_{k,h+1}-\widetilde V\|_\infty
\le4H\epsilon_1\le\frac{\alpha^2}{2d^3H}, \tag*{(40)}
\end{aligned} [ V h V ] ( s h i , a h i ) − [ V h Δ V k , h + 1 ] ( s h i , a h i ) = [ P h ( V 2 − ( Δ V k , h + 1 ) 2 )] ( s h i , a h i ) + ([ P h Δ V k , h + 1 ] ( s h i , a h i ) ) 2 − ([ P h V ] ( s h i , a h i ) ) 2 ≤ 4 H ∥Δ V k , h + 1 − V ∥ ∞ ≤ 4 H ϵ 1 ≤ 2 d 3 H α 2 , (40)
where the first inequality uses ∣ a 2 − b 2 ∣ ≤ 2 H ∣ a − b ∣ |a^2-b^2|\le2H|a-b| ∣ a 2 − b 2 ∣ ≤ 2 H ∣ a − b ∣ for a , b ∈ [ − H , H ] a,b\in[-H,H] a , b ∈ [ − H , H ] and the last inequality uses (A.6). Consequently, for every i ∈ { 1 , … , k − 1 } i\in\{1,\ldots,k-1\} i ∈ { 1 , … , k − 1 } ,
[ V h V ~ ] ( s h i , a h i ) ≤ [ V h Δ V k , h + 1 ] ( s h i , a h i ) + α 2 2 d 3 H ≤ D i , h + α 2 2 d 3 H ≤ σ ‾ i , h 2 d 3 H , [V_h\widetilde V](s_h^i,a_h^i)\le[V_h\Delta V_{k,h+1}](s_h^i,a_h^i)+\frac{\alpha^2}{2d^3H}\le\frac{D_{i,h}+\alpha^2}{2d^3H}\le\frac{\overline{\sigma}_{i,h}^2}{d^3H}, [ V h V ] ( s h i , a h i ) ≤ [ V h Δ V k , h + 1 ] ( s h i , a h i ) + 2 d 3 H α 2 ≤ 2 d 3 H D i , h + α 2 ≤ d 3 H σ i , h 2 ,
where (i) follows from (A.28), (ii) applies (P3)i , h _{i,h} i , h ((A.21) at j = k j=k j = k ), and (iii) uses σ ‾ i , h 2 ≥ D i , h + α 2 \overline{\sigma}_{i,h}^2\ge D_{i,h}+\alpha^2 σ i , h 2 ≥ D i , h + α 2 from (7) –(8) . Hence the indicator in η i , h ( V ~ , 1 / ( d 3 H ) ) \eta_{i,h}(\widetilde V,1/(d^3H)) η i , h ( V , 1/ ( d 3 H )) equals 1 for all i < k i<k i < k , and the triangle inequality yields
J 2 ≤ ∥ ∑ i = 1 k − 1 x i , h η i , h ( V ~ , 1 d 3 H ) ∥ Σ k , h − 1 + ∥ ∑ i = 1 k − 1 σ ‾ i , h − 2 ϕ ( s h i , a h i ) ( ( Δ V k , h + 1 − V ~ ) ( s h + 1 i ) − [ P h ( Δ V k , h + 1 − V ~ ) ] ( s h i , a h i ) ) ∥ Σ k , h − 1 ≤ β 0 + 2 K ϵ 1 α 2 λ = β 0 + 1 8 d 3 H 2 ≤ β 0 + 1 , (41) \begin{aligned}
J_2
&\le\left\|\sum_{i=1}^{k-1}x_{i,h}\eta_{i,h}\left(\widetilde V,\frac{1}{d^3H}\right)\right\|_{\Sigma_{k,h}^{-1}}\\
&\quad+\left\|\sum_{i=1}^{k-1}\overline{\sigma}_{i,h}^{-2}\phi(s_h^i,a_h^i)\left((\Delta V_{k,h+1}-\widetilde V)(s_{h+1}^i)-[P_h(\Delta V_{k,h+1}-\widetilde V)](s_h^i,a_h^i)\right)\right\|_{\Sigma_{k,h}^{-1}}\\
&\le\beta_0+\frac{2K\epsilon_1}{\alpha^2\sqrt{\lambda}}
=\beta_0+\frac{1}{8d^3H^2}
\le\beta_0+1, \tag*{(41)}
\end{aligned} J 2 ≤ i = 1 ∑ k − 1 x i , h η i , h ( V , d 3 H 1 ) Σ k , h − 1 + i = 1 ∑ k − 1 σ i , h − 2 ϕ ( s h i , a h i ) ( ( Δ V k , h + 1 − V ) ( s h + 1 i ) − [ P h ( Δ V k , h + 1 − V )] ( s h i , a h i ) ) Σ k , h − 1 ≤ β 0 + α 2 λ 2 K ϵ 1 = β 0 + 8 d 3 H 2 1 ≤ β 0 + 1 , (41)
where the second inequality uses (A.12), σ ‾ i , h ≥ α \overline{\sigma}_{i,h}\ge\alpha σ i , h ≥ α , ∥ Σ k , h − 1 / 2 ϕ ( s h i , a h i ) ∥ 2 ≤ λ − 1 / 2 \|\boldsymbol{\Sigma}_{k,h}^{-1/2}\phi(s_h^i,a_h^i)\|_2\le\lambda^{-1/2} ∥ Σ k , h − 1/2 ϕ ( s h i , a h i ) ∥ 2 ≤ λ − 1/2 , and ∥ Δ V k , h + 1 − V ~ ∥ ∞ ≤ ϵ 1 \|\Delta V_{k,h+1}-\widetilde V\|_\infty\le\epsilon_1 ∥Δ V k , h + 1 − V ∥ ∞ ≤ ϵ 1 , and the equality substitutes (A.6). Combining J 1 ≤ β 0 J_1\le\beta_0 J 1 ≤ β 0 and (A.29) with (A.27) and (A.25) proves (A.16).
Case (b): Concentration of w ~ k , h \widetilde w_{k,h} w k , h . Symmetrically, decompose V ~ k , h + 1 = V h + 1 ∗ − Δ V ~ k , h + 1 \widetilde V_{k,h+1}=V^*_{h+1}-\Delta\widetilde V_{k,h+1} V k , h + 1 = V h + 1 ∗ − Δ V k , h + 1 as in (12) , with Δ V ~ k , h + 1 ∈ Δ V ~ h + 1 \Delta\widetilde V_{k,h+1}\in\Delta\widetilde V_{h+1} Δ V k , h + 1 ∈ Δ V h + 1 , and choose U ~ ∈ N ϵ 1 ( Δ V ~ h + 1 ) \widetilde U\in\mathcal{N}_{\epsilon_1}(\Delta\widetilde V_{h+1}) U ∈ N ϵ 1 ( Δ V h + 1 ) satisfying ∥ Δ V ~ k , h + 1 − U ~ ∥ ∞ ≤ ϵ 1 \|\Delta\widetilde V_{k,h+1}-\widetilde U\|_\infty\le\epsilon_1 ∥Δ V k , h + 1 − U ∥ ∞ ≤ ϵ 1 . For every i < k i<k i < k , the same Lipschitz bound as in (A.28) combined with (A.22) of (P3)i , h _{i,h} i , h (at j = k j=k j = k ) gives [ V h U ~ ] ( s h i , a h i ) ≤ ( D i , h + α 2 ) / ( 2 d 3 H ) ≤ σ ‾ i , h 2 / ( d 3 H ) [V_h\widetilde U](s_h^i,a_h^i)\le(D_{i,h}+\alpha^2)/(2d^3H)\le\overline{\sigma}_{i,h}^2/(d^3H) [ V h U ] ( s h i , a h i ) ≤ ( D i , h + α 2 ) / ( 2 d 3 H ) ≤ σ i , h 2 / ( d 3 H ) . Repeating (A.27)–(A.29) via (A.10) and (A.13) yields ∥ w ~ k , h − w ~ k , h ∗ ∥ Σ k , h ≤ H d λ + 2 β 0 + 1 = β ‾ \|\widetilde w_{k,h}-\widetilde w^*_{k,h}\|_{\Sigma_{k,h}}\le H\sqrt{d\lambda}+2\beta_0+1=\overline{\beta} ∥ w k , h − w k , h ∗ ∥ Σ k , h ≤ H d λ + 2 β 0 + 1 = β , proving (A.17).
Case (c): Concentration of w ~ k , h \widetilde w_{k,h} w k , h . Decompose V k , h + 1 2 = ( V h + 1 ∗ ) 2 + Δ ( V k , h + 1 2 ) V^2_{k,h+1}=(V^*_{h+1})^2+\Delta(V^2_{k,h+1}) V k , h + 1 2 = ( V h + 1 ∗ ) 2 + Δ ( V k , h + 1 2 ) as in (12) , with Δ ( V k , h + 1 2 ) ∈ Δ ( V h + 1 2 ) \Delta(V^2_{k,h+1})\in\Delta(V^2_{h+1}) Δ ( V k , h + 1 2 ) ∈ Δ ( V h + 1 2 ) . By (A.26) and the triangle inequality, we have
∥ w ~ k , h − w ~ k , h ∗ ∥ Σ k , h ≤ H 2 d λ + ∥ ∑ i = 1 k − 1 σ ‾ i , h − 2 ϕ ( s h i , a h i ) ( ( V h + 1 ∗ ) 2 ( s h + 1 i ) − [ P h ( V h + 1 ∗ ) 2 ] ( s h i , a h i ) ) ∥ Σ k , h − 1 ⏟ = : J ~ 1 + ∥ ∑ i = 1 k − 1 σ ‾ i , h − 2 ϕ ( s h i , a h i ) ( Δ ( V k , h + 1 2 ) ( s h + 1 i ) − [ P h Δ ( V k , h + 1 2 ) ] ( s h i , a h i ) ) ∥ Σ k , h − 1 ⏟ = : J ~ 2 . (42) \begin{aligned}
\|\widetilde w_{k,h}-\widetilde w^*_{k,h}\|_{\Sigma_{k,h}}
&\le H^2\sqrt{d\lambda}\\
&\quad+\underbrace{\left\|\sum_{i=1}^{k-1}\overline{\sigma}_{i,h}^{-2}\phi(s_h^i,a_h^i)\left((V^*_{h+1})^2(s_{h+1}^i)-[P_h(V^*_{h+1})^2](s_h^i,a_h^i)\right)\right\|_{\Sigma_{k,h}^{-1}}}_{=:\widetilde J_1}\\
&\quad+\underbrace{\left\|\sum_{i=1}^{k-1}\overline{\sigma}_{i,h}^{-2}\phi(s_h^i,a_h^i)\left(\Delta(V^2_{k,h+1})(s_{h+1}^i)-[P_h\Delta(V^2_{k,h+1})](s_h^i,a_h^i)\right)\right\|_{\Sigma_{k,h}^{-1}}}_{=:\widetilde J_2}. \tag*{(42)}
\end{aligned} ∥ w k , h − w k , h ∗ ∥ Σ k , h ≤ H 2 d λ + =: J 1 i = 1 ∑ k − 1 σ i , h − 2 ϕ ( s h i , a h i ) ( ( V h + 1 ∗ ) 2 ( s h + 1 i ) − [ P h ( V h + 1 ∗ ) 2 ] ( s h i , a h i ) ) Σ k , h − 1 + =: J 2 i = 1 ∑ k − 1 σ i , h − 2 ϕ ( s h i , a h i ) ( Δ ( V k , h + 1 2 ) ( s h + 1 i ) − [ P h Δ ( V k , h + 1 2 )] ( s h i , a h i ) ) Σ k , h − 1 . (42)
For J ~ 1 \widetilde{J}_1 J 1 , Corollary E.5 and (P2)i , h _{i,h} i , h imply [ V h ( V h + 1 ∗ ) 2 ] ( s h i , a h i ) ≤ 4 H 2 [ V h V h + 1 ∗ ] ( s h i , a h i ) ≤ 4 H 2 σ ˉ i , h 2 [\mathbb{V}_h(V^*_{h+1})^2](s^i_h,a^i_h) \le4H^2[\mathbb{V}_hV^*_{h+1}](s^i_h,a^i_h) \le4H^2\bar{\sigma}^2_{i,h} [ V h ( V h + 1 ∗ ) 2 ] ( s h i , a h i ) ≤ 4 H 2 [ V h V h + 1 ∗ ] ( s h i , a h i ) ≤ 4 H 2 σ ˉ i , h 2 for all i < k i<k i < k , so the indicator in η i , h ( ( V h + 1 ∗ ) 2 , 4 H 2 ) \eta_{i,h}((V^*_{h+1})^2,4H^2) η i , h (( V h + 1 ∗ ) 2 , 4 H 2 ) equals 1 and (A.11) gives J ~ 1 ≤ 2 H β 0 \widetilde{J}_1\le2H\beta_0 J 1 ≤ 2 H β 0 . For J ~ 2 \widetilde{J}_2 J 2 , let W ~ ∈ N ϵ 2 ( Δ ( V h + 1 2 ) ) \widetilde{W}\in\mathcal{N}_{\epsilon_2}(\Delta(V^2_{h+1})) W ∈ N ϵ 2 ( Δ ( V h + 1 2 )) satisfy ∥ Δ ( V k , h + 1 2 ) − W ~ ∥ ∞ ≤ ϵ 2 \|\Delta(V^2_{k,h+1})-\widetilde{W}\|_\infty\le\epsilon_2 ∥Δ ( V k , h + 1 2 ) − W ∥ ∞ ≤ ϵ 2 . Since ∥ Δ ( V k , h + 1 2 ) ∥ ∞ ≤ H 2 \|\Delta(V^2_{k,h+1})\|_\infty\le H^2 ∥Δ ( V k , h + 1 2 ) ∥ ∞ ≤ H 2 and ∥ W ~ ∥ ∞ ≤ H 2 \|\widetilde{W}\|_\infty\le H^2 ∥ W ∥ ∞ ≤ H 2 , for every i ∈ { 1 , … , k − 1 } i\in\{1,\ldots,k-1\} i ∈ { 1 , … , k − 1 } we have
[ V h W ~ ] ( s h i , a h i ) ≤ [ V h Δ ( V k , h + 1 2 ) ] ( s h i , a h i ) + 4 H 2 ϵ 2 ≤ ( i ) H 2 D i , h 2 d 3 H + α 2 2 d 3 K ≤ ( i i i ) H 2 σ ˉ i , h 2 d 3 H , [\mathbb{V}_h\widetilde{W}](s^i_h,a^i_h)\le[\mathbb{V}_h\Delta(V^2_{k,h+1})](s^i_h,a^i_h)+4H^2\epsilon_2\overset{\mathrm{(i)}}{\le}\frac{H^2D_{i,h}}{2d^3H}+\frac{\alpha^2}{2d^3K}\overset{\mathrm{(iii)}}{\le}\frac{H^2\bar{\sigma}^2_{i,h}}{d^3H}, [ V h W ] ( s h i , a h i ) ≤ [ V h Δ ( V k , h + 1 2 )] ( s h i , a h i ) + 4 H 2 ϵ 2 ≤ ( i ) 2 d 3 H H 2 D i , h + 2 d 3 K α 2 ≤ ( iii ) d 3 H H 2 σ ˉ i , h 2 ,
where (i) uses ∣ a 2 − b 2 ∣ ≤ 2 H 2 ∣ a − b ∣ |a^2-b^2|\le2H^2|a-b| ∣ a 2 − b 2 ∣ ≤ 2 H 2 ∣ a − b ∣ for a , b ∈ [ − H 2 , H 2 ] a,b\in[-H^2,H^2] a , b ∈ [ − H 2 , H 2 ] , (ii) applies (A.23) of (P3)i , h _{i,h} i , h (at j = k j=k j = k ) together with 4 H 2 ϵ 2 = α 2 / ( 2 d 3 K ) 4H^2\epsilon_2=\alpha^2/(2d^3K) 4 H 2 ϵ 2 = α 2 / ( 2 d 3 K ) from (A.7), and (iii) holds since H , K ≥ 1 H,K\ge1 H , K ≥ 1 and σ ˉ i , h 2 ≥ D i , h + α 2 \bar{\sigma}^2_{i,h}\ge D_{i,h}+\alpha^2 σ ˉ i , h 2 ≥ D i , h + α 2 . Thus the indicator in η i , h ( W ~ , H 2 / ( d 3 H ) ) \eta_{i,h}(\widetilde{W},H^2/(d^3H)) η i , h ( W , H 2 / ( d 3 H )) equals 1 for all i < k i<k i < k , and the triangle inequality with (A.14) gives
J ~ 2 ≤ H β 0 + 2 K ϵ 2 α 2 λ = H β 0 + 1 4 d 3 H ≤ H β 0 + H , \widetilde{J}_2\le H\beta_0+\frac{2K\epsilon_2}{\alpha^2\sqrt{\lambda}}=H\beta_0+\frac{1}{4d^3H}\le H\beta_0+H, J 2 ≤ H β 0 + α 2 λ 2 K ϵ 2 = H β 0 + 4 d 3 H 1 ≤ H β 0 + H ,
where the equality uses (A.7). Substituting J ~ 1 ≤ 2 H β 0 \widetilde{J}_1\le2H\beta_0 J 1 ≤ 2 H β 0 and J ~ 2 ≤ H β 0 + H \widetilde{J}_2\le H\beta_0+H J 2 ≤ H β 0 + H into (A.30) yields ∥ w ~ k , h − w ~ k , h ∗ ∥ Σ k , h ≤ H 2 d λ + 3 H β 0 + H ≤ 2 H β = β ~ \|\widetilde{w}_{k,h}-\widetilde{w}^*_{k,h}\|_{\Sigma_{k,h}}\le H^2\sqrt{d\lambda}+3H\beta_0+H\le2H\beta=\widetilde{\beta} ∥ w k , h − w k , h ∗ ∥ Σ k , h ≤ H 2 d λ + 3 H β 0 + H ≤ 2 H β = β , which together with (A.25) establishes (A.18).
Step 2: Proof of (P2)k , h _{k,h} k , h . From (4) , we have
∣ [ V ~ k , h V k , h + 1 ] ( s h k , a h k ) − [ V h V k , h + 1 ] ( s h k , a h k ) ∣ ≤ ∣ [ w ~ k , h T ϕ ( s h k , a h k ) ] [ 0 , H 2 ] − [ P h V k , h + 1 2 ] ( s h k , a h k ) ∣ + ∣ [ w ~ k , h T ϕ ( s h k , a h k ) ] [ 0 , H ] 2 − ( [ P h V k , h + 1 ] ( s h k , a h k ) ) 2 ∣ ≤ min { β ~ ∥ ϕ ( s h k , a h k ) ∥ Σ k , h − 1 , H 2 } + min { 2 H β ~ ∥ ϕ ( s h k , a h k ) ∥ Σ k , h − 1 , H 2 } = E k , h . (43) \begin{aligned}
\left|[\widetilde{\mathbb{V}}_{k,h}V_{k,h+1}](s^k_h,a^k_h)-[\mathbb{V}_hV_{k,h+1}](s^k_h,a^k_h)\right|
&\le\left|\left[\widetilde{w}^{\mathsf{T}}_{k,h}\phi(s^k_h,a^k_h)\right]_{[0,H^2]}-[\mathbb{P}_hV^2_{k,h+1}](s^k_h,a^k_h)\right|\\
&\quad+\left|\left[\widetilde{w}^{\mathsf{T}}_{k,h}\phi(s^k_h,a^k_h)\right]^2_{[0,H]}-\left([\mathbb{P}_hV_{k,h+1}](s^k_h,a^k_h)\right)^2\right|\\
&\le\min\left\{\widetilde{\beta}\|\phi(s^k_h,a^k_h)\|_{\Sigma^{-1}_{k,h}},H^2\right\}\\
&\quad+\min\left\{2H\widetilde{\beta}\|\phi(s^k_h,a^k_h)\|_{\Sigma^{-1}_{k,h}},H^2\right\}=E_{k,h}. \tag*{(43)}
\end{aligned} [ V k , h V k , h + 1 ] ( s h k , a h k ) − [ V h V k , h + 1 ] ( s h k , a h k ) ≤ [ w k , h T ϕ ( s h k , a h k ) ] [ 0 , H 2 ] − [ P h V k , h + 1 2 ] ( s h k , a h k ) + [ w k , h T ϕ ( s h k , a h k ) ] [ 0 , H ] 2 − ( [ P h V k , h + 1 ] ( s h k , a h k ) ) 2 ≤ min { β ∥ ϕ ( s h k , a h k ) ∥ Σ k , h − 1 , H 2 } + min { 2 H β ∥ ϕ ( s h k , a h k ) ∥ Σ k , h − 1 , H 2 } = E k , h . (43)
where the first inequality uses the 1-Lipschitz property of [ ⋅ ] [ 0 , H 2 ] [\cdot]_{[0,H^2]} [ ⋅ ] [ 0 , H 2 ] together with the triangle inequality, the second inequality uses [ P h V k , h + 1 2 ] ( s h k , a h k ) ∈ [ 0 , H 2 ] [\mathbb{P}_hV^2_{k,h+1}](s^k_h,a^k_h)\in[0,H^2] [ P h V k , h + 1 2 ] ( s h k , a h k ) ∈ [ 0 , H 2 ] , [ P h V k , h + 1 ] ( s h k , a h k ) ∈ [ 0 , H ] [\mathbb{P}_hV_{k,h+1}](s^k_h,a^k_h)\in[0,H] [ P h V k , h + 1 ] ( s h k , a h k ) ∈ [ 0 , H ] , ∣ a 2 − b 2 ∣ ≤ 2 H ∣ a − b ∣ |a^2-b^2|\le2H|a-b| ∣ a 2 − b 2 ∣ ≤ 2 H ∣ a − b ∣ for a , b ∈ [ 0 , H ] a,b\in[0,H] a , b ∈ [ 0 , H ] , (A.16), (A.18), and β ˉ = β \bar{\beta}=\beta β ˉ = β , and the equality uses (5) . This proves (A.19). Next, we bound the variance gap between V k , h + 1 V_{k,h+1} V k , h + 1 and V h + 1 ∗ V^*_{h+1} V h + 1 ∗ :
∣ [ V h V k , h + 1 ] ( s h k , a h k ) − [ V h V h + 1 ∗ ] ( s h k , a h k ) ∣ ≤ ∣ [ P h ( V k , h + 1 2 − ( V h + 1 ∗ ) 2 ) ] ( s h k , a h k ) ∣ + ∣ ( [ P h V k , h + 1 ] ( s h k , a h k ) ) 2 − ( [ P h V h + 1 ∗ ] ( s h k , a h k ) ) 2 ∣ ≤ 4 H [ P h ( V k , h + 1 − V h + 1 ∗ ) ] ( s h k , a h k ) ≤ 4 H [ P h ( V k , h + 1 − V ~ k , h + 1 ) ] ( s h k , a h k ) ≤ 4 H ( ( w ^ k , h − w ~ k , h ) T ϕ ( s h k , a h k ) + 2 β ˉ ∥ ϕ ( s h k , a h k ) ∥ Σ k , h − 1 ) , (44) \begin{aligned}
\left|[\mathbb{V}_hV_{k,h+1}](s^k_h,a^k_h)-[\mathbb{V}_hV^*_{h+1}](s^k_h,a^k_h)\right|
&\le\left|[\mathbb{P}_h(V^2_{k,h+1}-(V^*_{h+1})^2)](s^k_h,a^k_h)\right|\\
&\quad+\left|([\mathbb{P}_hV_{k,h+1}](s^k_h,a^k_h))^2-([\mathbb{P}_hV^*_{h+1}](s^k_h,a^k_h))^2\right|\\
&\le4H[\mathbb{P}_h(V_{k,h+1}-V^*_{h+1})](s^k_h,a^k_h)\\
&\le4H[\mathbb{P}_h(V_{k,h+1}-\widetilde{V}_{k,h+1})](s^k_h,a^k_h)\\
&\le4H\left((\widehat{w}_{k,h}-\widetilde{w}_{k,h})^{\mathsf{T}}\phi(s^k_h,a^k_h)+2\bar{\beta}\|\phi(s^k_h,a^k_h)\|_{\Sigma^{-1}_{k,h}}\right), \tag*{(44)}
\end{aligned} [ V h V k , h + 1 ] ( s h k , a h k ) − [ V h V h + 1 ∗ ] ( s h k , a h k ) ≤ [ P h ( V k , h + 1 2 − ( V h + 1 ∗ ) 2 )] ( s h k , a h k ) + ([ P h V k , h + 1 ] ( s h k , a h k ) ) 2 − ([ P h V h + 1 ∗ ] ( s h k , a h k ) ) 2 ≤ 4 H [ P h ( V k , h + 1 − V h + 1 ∗ )] ( s h k , a h k ) ≤ 4 H [ P h ( V k , h + 1 − V k , h + 1 )] ( s h k , a h k ) ≤ 4 H ( ( w k , h − w k , h ) T ϕ ( s h k , a h k ) + 2 β ˉ ∥ ϕ ( s h k , a h k ) ∥ Σ k , h − 1 ) , (44)
where the second and third inequalities follow from (A.24) at m = k m=k m = k (0 ≤ V ~ k , h + 1 ≤ V h + 1 ∗ ≤ V k , h + 1 ≤ H 0\le\widetilde{V}_{k,h+1}\le V^*_{h+1}\le V_{k,h+1}\le H 0 ≤ V k , h + 1 ≤ V h + 1 ∗ ≤ V k , h + 1 ≤ H ), and the last inequality uses (A.16), (A.17), and β ˉ = β \bar{\beta}=\beta β ˉ = β . Since both [ V h V k , h + 1 ] ( s h k , a h k ) [\mathbb{V}_hV_{k,h+1}](s^k_h,a^k_h) [ V h V k , h + 1 ] ( s h k , a h k ) and [ V h V h + 1 ∗ ] ( s h k , a h k ) [\mathbb{V}_hV^*_{h+1}](s^k_h,a^k_h) [ V h V h + 1 ∗ ] ( s h k , a h k ) lie in [ 0 , H 2 ] [0,H^2] [ 0 , H 2 ] , their absolute difference is also at most H 2 ≤ d 3 H 3 H^2\le d^3H^3 H 2 ≤ d 3 H 3 , so (A.32) and (6) imply ∣ [ V h V k , h + 1 ] ( s h k , a h k ) − [ V h V h + 1 ∗ ] ( s h k , a h k ) ∣ ≤ D k , h / ( d 3 H ) ≤ D k , h |[\mathbb{V}_hV_{k,h+1}](s^k_h,a^k_h)-[\mathbb{V}_hV^*_{h+1}](s^k_h,a^k_h)|\le D_{k,h}/(d^3H)\le D_{k,h} ∣ [ V h V k , h + 1 ] ( s h k , a h k ) − [ V h V h + 1 ∗ ] ( s h k , a h k ) ∣ ≤ D k , h / ( d 3 H ) ≤ D k , h . Combining this bound with (A.31) via the triangle inequality proves (A.20).
Step 3: Proof of (P3)k , h _{k,h} k , h . Fix any j ∈ { k , … , K } j\in\{k,\ldots,K\} j ∈ { k , … , K } . For all s ′ ∈ S s'\in\mathcal{S} s ′ ∈ S , we have
0 ≤ V j , h + 1 ( s ′ ) − V h + 1 ∗ ( s ′ ) ≤ V k , h + 1 ( s ′ ) − V ~ k , h + 1 ( s ′ ) ≤ H , (45) 0\le V_{j,h+1}(s')-V^*_{h+1}(s')\le V_{k,h+1}(s')-\widetilde{V}_{k,h+1}(s')\le H, \tag*{(45)} 0 ≤ V j , h + 1 ( s ′ ) − V h + 1 ∗ ( s ′ ) ≤ V k , h + 1 ( s ′ ) − V k , h + 1 ( s ′ ) ≤ H , (45)
0 ≤ V h + 1 ∗ ( s ′ ) − V ~ j , h + 1 ( s ′ ) ≤ V k , h + 1 ( s ′ ) − V ~ k , h + 1 ( s ′ ) ≤ H (46) 0 \le V^{*}_{h+1}(s')-\widetilde{V}_{j,h+1}(s') \le V_{k,h+1}(s')-\widetilde{V}_{k,h+1}(s') \le H \tag*{(46)} 0 ≤ V h + 1 ∗ ( s ′ ) − V j , h + 1 ( s ′ ) ≤ V k , h + 1 ( s ′ ) − V k , h + 1 ( s ′ ) ≤ H (46)
0 ≤ V j , h + 1 2 ( s ′ ) − ( V h + 1 ∗ ) 2 ( s ′ ) ≤ min { 2 H ( V k , h + 1 ( s ′ ) − V ~ k , h + 1 ( s ′ ) ) , H 2 } (47) 0 \le V^{2}_{j,h+1}(s')-(V^{*}_{h+1})^{2}(s') \le\min\left\{2H\left(V_{k,h+1}(s')-\widetilde{V}_{k,h+1}(s')\right),H^{2}\right\} \tag*{(47)} 0 ≤ V j , h + 1 2 ( s ′ ) − ( V h + 1 ∗ ) 2 ( s ′ ) ≤ min { 2 H ( V k , h + 1 ( s ′ ) − V k , h + 1 ( s ′ ) ) , H 2 } (47)
where (A.33) and (A.34) follow from (A.24) at m = j m=j m = j together with the pointwise monotonicity V j , h + 1 ≤ V k , h + 1 V_{j,h+1}\le V_{k,h+1} V j , h + 1 ≤ V k , h + 1 and V ~ j , h + 1 ≥ V ~ k , h + 1 \widetilde{V}_{j,h+1}\ge\widetilde{V}_{k,h+1} V j , h + 1 ≥ V k , h + 1 from Lines 14–18 of Algorithm 1, and (A.35) uses V j , h + 1 2 − ( V h + 1 ∗ ) 2 = ( V j , h + 1 + V h + 1 ∗ ) ( V j , h + 1 − V h + 1 ∗ ) ≤ min { 2 H ( V k , h + 1 − V ~ k , h + 1 ) , H 2 } V^{2}_{j,h+1}-(V^{*}_{h+1})^{2}=(V_{j,h+1}+V^{*}_{h+1})(V_{j,h+1}-V^{*}_{h+1})\le\min\{2H(V_{k,h+1}-\widetilde{V}_{k,h+1}),H^{2}\} V j , h + 1 2 − ( V h + 1 ∗ ) 2 = ( V j , h + 1 + V h + 1 ∗ ) ( V j , h + 1 − V h + 1 ∗ ) ≤ min { 2 H ( V k , h + 1 − V k , h + 1 ) , H 2 } . For the first-moment residuals, we obtain
max { [ V h ( V j , h + 1 − V h + 1 ∗ ) ] ( s h k , a h k ) , [ V h ( V h + 1 ∗ − V ~ j , h + 1 ) ] ( s h k , a h k ) } ≤ min { H [ P h ( V k , h + 1 − V ~ k , h + 1 ) ] ( s h k , a h k ) , H 2 2 } ≤ D k , h 2 d 3 H , \max\left\{\left[\mathbb{V}_{h}(V_{j,h+1}-V^{*}_{h+1})\right](s^{k}_{h},a^{k}_{h}),\left[\mathbb{V}_{h}(V^{*}_{h+1}-\widetilde{V}_{j,h+1})\right](s^{k}_{h},a^{k}_{h})\right\}
\le\min\left\{H\left[\mathbb{P}_{h}(V_{k,h+1}-\widetilde{V}_{k,h+1})\right](s^{k}_{h},a^{k}_{h}),\frac{H^{2}}{2}\right\}
\le\frac{D_{k,h}}{2d^{3}H}, max { [ V h ( V j , h + 1 − V h + 1 ∗ ) ] ( s h k , a h k ) , [ V h ( V h + 1 ∗ − V j , h + 1 ) ] ( s h k , a h k ) } ≤ min { H [ P h ( V k , h + 1 − V k , h + 1 ) ] ( s h k , a h k ) , 2 H 2 } ≤ 2 d 3 H D k , h ,
where the first inequality uses Var ( Z ) ≤ min { E [ Z 2 ] , B 2 / 4 } ≤ min { B E [ Z ] , B 2 / 2 } \operatorname{Var}(Z)\le\min\{\mathbb{E}[Z^{2}],B^{2}/4\}\le\min\{B\mathbb{E}[Z],B^{2}/2\} Var ( Z ) ≤ min { E [ Z 2 ] , B 2 /4 } ≤ min { B E [ Z ] , B 2 /2 } for any random variable Z ∈ [ 0 , B ] Z\in[0,B] Z ∈ [ 0 , B ] together with (A.33)–(A.34) (B = H B=H B = H ), and the second inequality follows from (A.16)–(A.17), β ˉ = β \bar{\beta}=\beta β ˉ = β , and the definition of D k , h D_{k,h} D k , h in (4.5). Similarly, for the squared residual, we have
[ V h ( V j , h + 1 2 − ( V h + 1 ∗ ) 2 ) ] ( s h k , a h k ) ≤ min { 2 H 3 [ P h ( V k , h + 1 − V ~ k , h + 1 ) ] ( s h k , a h k ) , H 4 2 } ≤ H 2 D k , h 2 d 3 H , \left[\mathbb{V}_{h}\left(V^{2}_{j,h+1}-(V^{*}_{h+1})^{2}\right)\right](s^{k}_{h},a^{k}_{h})\le\min\left\{2H^{3}\left[\mathbb{P}_{h}(V_{k,h+1}-\widetilde{V}_{k,h+1})\right](s^{k}_{h},a^{k}_{h}),\frac{H^{4}}{2}\right\}\le\frac{H^{2}D_{k,h}}{2d^{3}H}, [ V h ( V j , h + 1 2 − ( V h + 1 ∗ ) 2 ) ] ( s h k , a h k ) ≤ min { 2 H 3 [ P h ( V k , h + 1 − V k , h + 1 ) ] ( s h k , a h k ) , 2 H 4 } ≤ 2 d 3 H H 2 D k , h ,
where the first inequality applies Var ( Z ) ≤ min { B E [ Z ] , B 2 / 2 } \operatorname{Var}(Z)\le\min\{B\mathbb{E}[Z],B^{2}/2\} Var ( Z ) ≤ min { B E [ Z ] , B 2 /2 } with B = H 2 B=H^{2} B = H 2 to (A.35), and the second inequality again uses (A.16)–(A.17), β ˉ = β \bar{\beta}=\beta β ˉ = β , and (4.5). This proves (A.21)–(A.23) and completes the inner induction over k = 1 , … , K k=1,\ldots,K k = 1 , … , K .
Step 4: Proof of (P4)k , h _{k,h} k , h for k ∈ [ K ] k\in[K] k ∈ [ K ] . For all k ∈ [ K ] k\in[K] k ∈ [ K ] and ( s , a ) ∈ S × A (s,a)\in\mathcal{S}\times\mathcal{A} ( s , a ) ∈ S × A , we have
r h ( s , a ) + w ^ k , h T ϕ ( s , a ) + β ∥ ϕ ( s , a ) ∥ Σ k , h − 1 ≥ r h ( s , a ) + [ P h V k , h + 1 ] ( s , a ) ≥ Q h ∗ ( s , a ) , r_{h}(s,a)+\widehat{\mathbf{w}}_{k,h}^{\mathsf{T}}\phi(s,a)+\beta\|\phi(s,a)\|_{\Sigma^{-1}_{k,h}}\ge r_{h}(s,a)+[\mathbb{P}_{h}V_{k,h+1}](s,a)\ge Q^{*}_{h}(s,a), r h ( s , a ) + w k , h T ϕ ( s , a ) + β ∥ ϕ ( s , a ) ∥ Σ k , h − 1 ≥ r h ( s , a ) + [ P h V k , h + 1 ] ( s , a ) ≥ Q h ∗ ( s , a ) ,
r h ( s , a ) + w ~ k , h T ϕ ( s , a ) − β ˉ ∥ ϕ ( s , a ) ∥ Σ k , h − 1 ≤ r h ( s , a ) + [ P h V ~ k , h + 1 ] ( s , a ) ≤ Q h ∗ ( s , a ) , r_{h}(s,a)+\widetilde{\mathbf{w}}_{k,h}^{\mathsf{T}}\phi(s,a)-\bar{\beta}\|\phi(s,a)\|_{\Sigma^{-1}_{k,h}}\le r_{h}(s,a)+[\mathbb{P}_{h}\widetilde{V}_{k,h+1}](s,a)\le Q^{*}_{h}(s,a), r h ( s , a ) + w k , h T ϕ ( s , a ) − β ˉ ∥ ϕ ( s , a ) ∥ Σ k , h − 1 ≤ r h ( s , a ) + [ P h V k , h + 1 ] ( s , a ) ≤ Q h ∗ ( s , a ) ,
where the first inequalities follow from (A.16) and (A.17), and the second inequalities use (A.24) and the Bellman optimality equation Q h ∗ ( s , a ) = r h ( s , a ) + [ P h V h + 1 ∗ ] ( s , a ) Q^{*}_{h}(s,a)=r_{h}(s,a)+[\mathbb{P}_{h}V^{*}_{h+1}](s,a) Q h ∗ ( s , a ) = r h ( s , a ) + [ P h V h + 1 ∗ ] ( s , a ) . Because 0 = Q ~ 0 , h ( s , a ) ≤ Q h ∗ ( s , a ) ≤ Q 0 , h ( s , a ) = H 0=\widetilde{Q}_{0,h}(s,a)\le Q^{*}_{h}(s,a)\le Q_{0,h}(s,a)=H 0 = Q 0 , h ( s , a ) ≤ Q h ∗ ( s , a ) ≤ Q 0 , h ( s , a ) = H , a direct induction on k = 1 , … , K k=1,\ldots,K k = 1 , … , K via (4.1)–(4.2) and Lines 16–18 of Algorithm 1 establishes Q ~ k , h ( s , a ) ≤ Q h ∗ ( s , a ) ≤ Q k , h ( s , a ) \widetilde{Q}_{k,h}(s,a)\le Q^{*}_{h}(s,a)\le Q_{k,h}(s,a) Q k , h ( s , a ) ≤ Q h ∗ ( s , a ) ≤ Q k , h ( s , a ) and V ~ k , h ( s ) ≤ V h ∗ ( s ) ≤ V k , h ( s ) \widetilde{V}_{k,h}(s)\le V^{*}_{h}(s)\le V_{k,h}(s) V k , h ( s ) ≤ V h ∗ ( s ) ≤ V k , h ( s ) for all k ∈ [ K ] k\in[K] k ∈ [ K ] . This completes the outer induction over h = H → 1 h=H\to1 h = H → 1 .
Regret Analysis and Proof of Theorem 5.1
Throughout this appendix, we work with the following shorthand quantities along the learner’s trajectory:
B k , h : = 4 min { β ∥ ϕ ( s h k , a h k ) ∥ Σ k , h − 1 , H } , (48) B_{k,h}:=4\min\left\{\beta\|\phi(s^{k}_{h},a^{k}_{h})\|_{\Sigma^{-1}_{k,h}},H\right\}, \tag*{(48)} B k , h := 4 min { β ∥ ϕ ( s h k , a h k ) ∥ Σ k , h − 1 , H } , (48)
V e s t : = ∑ k = 1 K ∑ h = 1 H σ k , h 2 , (49) V_{\mathrm{est}}:=\sum_{k=1}^{K}\sum_{h=1}^{H}\sigma^{2}_{k,h}, \tag*{(49)} V est := k = 1 ∑ K h = 1 ∑ H σ k , h 2 , (49)
S π : = ∑ k = 1 K ∑ h = 1 H [ P h ( V k , h + 1 − V h + 1 π ∗ ) ] ( s h k , a h k ) , (50) S_{\pi}:=\sum_{k=1}^{K}\sum_{h=1}^{H}[\mathbb{P}_{h}(V_{k,h+1}-V^{\pi^{*}}_{h+1})](s^{k}_{h},a^{k}_{h}), \tag*{(50)} S π := k = 1 ∑ K h = 1 ∑ H [ P h ( V k , h + 1 − V h + 1 π ∗ )] ( s h k , a h k ) , (50)
S u l : = ∑ k = 1 K ∑ h = 1 H [ P h ( V k , h + 1 − V ~ k , h + 1 ) ] ( s h k , a h k ) . (51) S_{\mathrm{ul}}:=\sum_{k=1}^{K}\sum_{h=1}^{H}\left[\mathbb{P}_{h}(V_{k,h+1}-\widetilde{V}_{k,h+1})\right](s_h^k,a_h^k). \tag*{(51)} S ul := k = 1 ∑ K h = 1 ∑ H [ P h ( V k , h + 1 − V k , h + 1 ) ] ( s h k , a h k ) . (51)
In the following subsections, we sequentially establish the elliptical bonus summation bound (Lemma B.1 ), the Freedman-type value difference bounds (Lemma B.2), the total estimated variance bound (Lemma B.3), and the proof of Theorem 5.1 .
Elliptical Bonus Summation
Lemma B.1 (Refined Elliptical Bonus Summation). For every stage h ∈ [ H ] h \in[H] h ∈ [ H ] and any parameters β ′ > 0 \beta' > 0 β ′ > 0 , C > 0 C > 0 C > 0 , the sequence of covariance matrices { Σ k , h } k = 1 K \{\Sigma_{k,h}\}_{k=1}^{K} { Σ k , h } k = 1 K generated by Algorithm 1 satisfies
∑ k = 1 K min { β ′ ∥ ϕ ( s h k , a h k ) ∥ Σ k , h − 1 , C } ≤ 2 C d ι + 2 β ′ γ 2 d ι + β ′ 2 d ι ∑ k = 1 K σ k , h 2 . (52) \sum_{k=1}^{K}\min\left\{\beta'\left\|\phi(s_h^k,a_h^k)\right\|_{\Sigma_{k,h}^{-1}},C\right\}\le2Cd\iota+2\beta'\gamma^2d\iota+\beta'\sqrt{2d\iota\sum_{k=1}^{K}\sigma_{k,h}^2}. \tag*{(52)} k = 1 ∑ K min { β ′ ϕ ( s h k , a h k ) Σ k , h − 1 , C } ≤ 2 C d ι + 2 β ′ γ 2 d ι + β ′ 2 d ι k = 1 ∑ K σ k , h 2 . (52) In particular, the cumulative truncated bonus B k , h B_{k,h} B k , h in (B.1) satisfies
∑ k = 1 K ∑ h = 1 H B k , h ≤ 8 d H 2 ι + 8 β γ 2 d H ι + 4 β 2 d H ι V e s t . (53) \sum_{k=1}^{K}\sum_{h=1}^{H}B_{k,h}\le8dH^2\iota+8\beta\gamma^2dH\iota+4\beta\sqrt{2dH\iota V_{\mathrm{est}}}. \tag*{(53)} k = 1 ∑ K h = 1 ∑ H B k , h ≤ 8 d H 2 ι + 8 β γ 2 d H ι + 4 β 2 d H ι V est . (53)
Proof. Recall that x k , h = σ ˉ k , h ϕ ( s h k , a h k ) x_{k,h}=\bar{\sigma}_{k,h}\phi(s_h^k,a_h^k) x k , h = σ ˉ k , h ϕ ( s h k , a h k ) , so that Σ k + 1 , h = Σ k , h + x k , h x k , h T \Sigma_{k+1,h}=\Sigma_{k,h}+x_{k,h}x_{k,h}^{\mathsf{T}} Σ k + 1 , h = Σ k , h + x k , h x k , h T and ∥ x k , h ∥ 2 ≤ α − 1 \|x_{k,h}\|_2\le\alpha^{-1} ∥ x k , h ∥ 2 ≤ α − 1 . By Lemma E.6 and (9) , we have
∑ k = 1 K min { 1 , ∥ x k , h ∥ Σ k , h − 1 2 } ≤ 2 d ln ( 1 + K d λ α 2 ) ≤ 2 d ι . (54) \sum_{k=1}^{K}\min\left\{1,\|x_{k,h}\|_{\Sigma_{k,h}^{-1}}^2\right\}\le2d\ln\left(1+\frac{K}{d\lambda\alpha^2}\right)\le2d\iota. \tag*{(54)} k = 1 ∑ K min { 1 , ∥ x k , h ∥ Σ k , h − 1 2 } ≤ 2 d ln ( 1 + d λ α 2 K ) ≤ 2 d ι . (54)
Partition [ K ] [K] [ K ] into three disjoint index sets:
K 1 : = { k ∈ [ K ] : ∥ x k , h ∥ Σ k , h − 1 ≥ 1 } , K 2 : = { k ∉ K 1 : σ ˉ k , h = γ ∥ ϕ ( s h k , a h k ) ∥ Σ k , h − 1 1 / 2 } , \mathcal{K}_1:=\left\{k\in[K]:\|x_{k,h}\|_{\Sigma_{k,h}^{-1}}\ge1\right\},\qquad
\mathcal{K}_2:=\left\{k\notin\mathcal{K}_1:\bar{\sigma}_{k,h}=\gamma\|\phi(s_h^k,a_h^k)\|_{\Sigma_{k,h}^{-1}}^{1/2}\right\}, K 1 := { k ∈ [ K ] : ∥ x k , h ∥ Σ k , h − 1 ≥ 1 } , K 2 := { k ∈ / K 1 : σ ˉ k , h = γ ∥ ϕ ( s h k , a h k ) ∥ Σ k , h − 1 1/2 } ,
K 3 : = [ K ] ∖ ( K 1 ∪ K 2 ) . \mathcal{K}_3:=[K]\setminus(\mathcal{K}_1\cup\mathcal{K}_2). K 3 := [ K ] ∖ ( K 1 ∪ K 2 ) .
Splitting the sum in (B.5) across K 1 \mathcal{K}_1 K 1 , K 2 \mathcal{K}_2 K 2 , and K 3 \mathcal{K}_3 K 3 yields
∑ k = 1 K min { β ′ ∥ ϕ ( s h k , a h k ) ∥ Σ k , h − 1 , C } ≤ ∑ k ∈ K 1 C + β ′ γ 2 ∑ k ∈ K 2 min { 1 , ∥ x k , h ∥ Σ k , h − 1 2 } + β ′ ∑ k ∈ K 3 σ k , h min { 1 , ∥ x k , h ∥ Σ k , h − 1 } ≤ 2 C d ι + 2 β ′ γ 2 d ι + β ′ ∑ k = 1 K σ k , h 2 ∑ k = 1 K min { 1 , ∥ x k , h ∥ Σ k , h − 1 2 } ≤ 2 C d ι + 2 β ′ γ 2 d ι + β ′ 2 d ι ∑ k = 1 K σ k , h 2 , \begin{aligned}
\sum_{k=1}^{K}\min\left\{\beta'\|\phi(s_h^k,a_h^k)\|_{\Sigma_{k,h}^{-1}},C\right\}
&\le\sum_{k\in\mathcal{K}_1}C+\beta'\gamma^2\sum_{k\in\mathcal{K}_2}\min\left\{1,\|x_{k,h}\|_{\Sigma_{k,h}^{-1}}^2\right\}\\
&\quad+\beta'\sum_{k\in\mathcal{K}_3}\sigma_{k,h}\min\left\{1,\|x_{k,h}\|_{\Sigma_{k,h}^{-1}}\right\}\\
&\le2Cd\iota+2\beta'\gamma^2d\iota+\beta'\sqrt{\sum_{k=1}^{K}\sigma_{k,h}^2}\sqrt{\sum_{k=1}^{K}\min\left\{1,\|x_{k,h}\|_{\Sigma_{k,h}^{-1}}^2\right\}}\\
&\le2Cd\iota+2\beta'\gamma^2d\iota+\beta'\sqrt{2d\iota\sum_{k=1}^{K}\sigma_{k,h}^2},
\end{aligned} k = 1 ∑ K min { β ′ ∥ ϕ ( s h k , a h k ) ∥ Σ k , h − 1 , C } ≤ k ∈ K 1 ∑ C + β ′ γ 2 k ∈ K 2 ∑ min { 1 , ∥ x k , h ∥ Σ k , h − 1 2 } + β ′ k ∈ K 3 ∑ σ k , h min { 1 , ∥ x k , h ∥ Σ k , h − 1 } ≤ 2 C d ι + 2 β ′ γ 2 d ι + β ′ k = 1 ∑ K σ k , h 2 k = 1 ∑ K min { 1 , ∥ x k , h ∥ Σ k , h − 1 2 } ≤ 2 C d ι + 2 β ′ γ 2 d ι + β ′ 2 d ι k = 1 ∑ K σ k , h 2 ,
where the first inequality uses ∥ ϕ ( s h k , a h k ) ∥ Σ k , h − 1 = γ 2 ∥ x k , h ∥ Σ k , h − 1 2 = γ 2 min { 1 , ∥ x k , h ∥ Σ k , h − 1 2 } \|\phi(s_h^k,a_h^k)\|_{\Sigma_{k,h}^{-1}}=\gamma^2\|x_{k,h}\|_{\Sigma_{k,h}^{-1}}^2=\gamma^2\min\{1,\|x_{k,h}\|_{\Sigma_{k,h}^{-1}}^2\} ∥ ϕ ( s h k , a h k ) ∥ Σ k , h − 1 = γ 2 ∥ x k , h ∥ Σ k , h − 1 2 = γ 2 min { 1 , ∥ x k , h ∥ Σ k , h − 1 2 } for k ∈ K 2 k\in\mathcal{K}_2 k ∈ K 2 and ∥ ϕ ( s h k , a h k ) ∥ Σ k , h − 1 = σ k , h ∥ x k , h ∥ Σ k , h − 1 = σ k , h min { 1 , ∥ x k , h ∥ Σ k , h − 1 } \|\phi(s_h^k,a_h^k)\|_{\Sigma_{k,h}^{-1}}=\sigma_{k,h}\|x_{k,h}\|_{\Sigma_{k,h}^{-1}}=\sigma_{k,h}\min\{1,\|x_{k,h}\|_{\Sigma_{k,h}^{-1}}\} ∥ ϕ ( s h k , a h k ) ∥ Σ k , h − 1 = σ k , h ∥ x k , h ∥ Σ k , h − 1 = σ k , h min { 1 , ∥ x k , h ∥ Σ k , h − 1 } for k ∈ K 3 k\in\mathcal{K}_3 k ∈ K 3 (since σ ˉ k , h = σ k , h \bar{\sigma}_{k,h}=\sigma_{k,h} σ ˉ k , h = σ k , h on K 3 \mathcal{K}_3 K 3 ). by (4.7)), the second inequality follows from ∣ K 1 ∣ ≤ ∑ k ∈ K 1 min { 1 , ∥ x k , h ∥ Σ k , h − 1 2 } ≤ 2 d υ |\mathcal{K}_1| \le\sum_{k\in\mathcal{K}_1} \min\{1,\|\mathbf{x}_{k,h}\|_{\Sigma_{k,h}^{-1}}^2\} \le2d\upsilon ∣ K 1 ∣ ≤ ∑ k ∈ K 1 min { 1 , ∥ x k , h ∥ Σ k , h − 1 2 } ≤ 2 d υ by (B.7) and the Cauchy–Schwarz inequality, and the last inequality applies (B.7) once more. Finally, summing (B.5) with ( β ′ , C ) = ( 4 β , 4 H ) (\beta', C) = (4\beta, 4H) ( β ′ , C ) = ( 4 β , 4 H ) over h ∈ [ H ] h \in[H] h ∈ [ H ] and applying ∑ h = 1 H ∑ k = 1 K σ k , h 2 ≤ H V e s t \sum_{h=1}^{H}\sqrt{\sum_{k=1}^{K}\sigma_{k,h}^2} \le\sqrt{H V_{\mathrm{est}}} ∑ h = 1 H ∑ k = 1 K σ k , h 2 ≤ H V est from (B.2) proves (B.6).
Freedman-Type Value Difference Bounds
Lemma B.2 (Value Difference Bounds). There exists an event E F r e e d \mathcal{E}_{\mathrm{Freed}} E Freed with P ( E F r e e d ) ≥ 1 − 2 δ \mathbb{P}(\mathcal{E}_{\mathrm{Freed}}) \ge1 - 2\delta P ( E Freed ) ≥ 1 − 2 δ such that on E c o n c ∩ E F r e e d \mathcal{E}_{\mathrm{conc}} \cap\mathcal{E}_{\mathrm{Freed}} E conc ∩ E Freed , the quantities B k , h B_{k,h} B k , h , S π S_{\pi} S π , and S u l S_{\mathrm{ul}} S ul defined in (B.1), (B.3), and (B.4) satisfy
Regret ( K ) ≤ 2 ∑ k = 1 K ∑ h = 1 H B k , h + 4 H 2 ln ( H / δ ) , (55) \operatorname{Regret}(K) \le2\sum_{k=1}^{K}\sum_{h=1}^{H} B_{k,h} + 4H^2\ln(H/\delta), \tag*{(55)} Regret ( K ) ≤ 2 k = 1 ∑ K h = 1 ∑ H B k , h + 4 H 2 ln ( H / δ ) , (55)
S π ≤ 2 H ∑ k = 1 K ∑ h = 1 H B k , h + 4 H 3 ln ( H / δ ) , (56) S_{\pi} \le2H\sum_{k=1}^{K}\sum_{h=1}^{H} B_{k,h} + 4H^3\ln(H/\delta), \tag*{(56)} S π ≤ 2 H k = 1 ∑ K h = 1 ∑ H B k , h + 4 H 3 ln ( H / δ ) , (56)
S u l ≤ 4 H ∑ k = 1 K ∑ h = 1 H B k , h + 4 H 3 ln ( H / δ ) . (57) S_{\mathrm{ul}} \le4H\sum_{k=1}^{K}\sum_{h=1}^{H} B_{k,h} + 4H^3\ln(H/\delta). \tag*{(57)} S ul ≤ 4 H k = 1 ∑ K h = 1 ∑ H B k , h + 4 H 3 ln ( H / δ ) . (57)
Proof. For each episode k ∈ [ K ] k \in[K] k ∈ [ K ] and stage h ∈ [ H ] h \in[H] h ∈ [ H ] , define the truncated value differences
Δ ~ k , h + 1 π ( s ) : = [ V k , h + 1 ( s ) − V h + 1 π k ( s ) ] [ 0 , H ] , Δ ~ k , h + 1 u l ( s ) : = [ V k , h + 1 ( s ) − V ~ k , h + 1 ( s ) ] [ 0 , H ] , \widetilde{\Delta}^{\pi}_{k,h+1}(s) := \left[V_{k,h+1}(s) - V^{\pi^k}_{h+1}(s)\right]_{[0,H]}, \qquad\widetilde{\Delta}^{\mathrm{ul}}_{k,h+1}(s) := \left[V_{k,h+1}(s) - \widetilde{V}_{k,h+1}(s)\right]_{[0,H]}, Δ k , h + 1 π ( s ) := [ V k , h + 1 ( s ) − V h + 1 π k ( s ) ] [ 0 , H ] , Δ k , h + 1 ul ( s ) := [ V k , h + 1 ( s ) − V k , h + 1 ( s ) ] [ 0 , H ] ,
and the corresponding martingale differences:
M k , h π : = [ P h Δ ~ k , h + 1 π ] ( s h k , a h k ) − Δ ~ k , h + 1 π ( s h + 1 k ) , M k , h u l : = [ P h Δ ~ k , h + 1 u l ] ( s h k , a h k ) − Δ ~ k , h + 1 u l ( s h + 1 k ) . M^{\pi}_{k,h} := [P_h\widetilde{\Delta}^{\pi}_{k,h+1}](s_h^k,a_h^k) - \widetilde{\Delta}^{\pi}_{k,h+1}(s_{h+1}^k), \qquad M^{\mathrm{ul}}_{k,h} := [P_h\widetilde{\Delta}^{\mathrm{ul}}_{k,h+1}](s_h^k,a_h^k) - \widetilde{\Delta}^{\mathrm{ul}}_{k,h+1}(s_{h+1}^k). M k , h π := [ P h Δ k , h + 1 π ] ( s h k , a h k ) − Δ k , h + 1 π ( s h + 1 k ) , M k , h ul := [ P h Δ k , h + 1 ul ] ( s h k , a h k ) − Δ k , h + 1 ul ( s h + 1 k ) .
Because V k , h + 1 V_{k,h+1} V k , h + 1 , V ~ k , h + 1 \widetilde{V}_{k,h+1} V k , h + 1 , and π k \pi^k π k are fixed prior to episode k k k and Δ ~ k , h + 1 π ( s ′ ) \widetilde{\Delta}^{\pi}_{k,h+1}(s') Δ k , h + 1 π ( s ′ ) , Δ ~ k , h + 1 u l ( s ′ ) ∈ [ 0 , H ] \widetilde{\Delta}^{\mathrm{ul}}_{k,h+1}(s') \in[0,H] Δ k , h + 1 ul ( s ′ ) ∈ [ 0 , H ] unconditionally, we have ∣ M k , h π ∣ ≤ H |M^{\pi}_{k,h}| \le H ∣ M k , h π ∣ ≤ H , E [ M k , h π ∣ F k , h ] = 0 \mathbb{E}[M^{\pi}_{k,h} \mid\mathcal{F}_{k,h}] = 0 E [ M k , h π ∣ F k , h ] = 0 , and the conditional variance is self-bounded by
E [ ( M k , h π ) 2 ∣ F k , h ] = [ P h Δ ~ k , h + 1 π ] ( s h k , a h k ) ≤ [ P h ( Δ ~ k , h + 1 π ) 2 ] ( s h k , a h k ) ≤ H [ P h Δ ~ k , h + 1 π ] ( s h k , a h k ) , \mathbb{E}[(M^{\pi}_{k,h})^2 \mid\mathcal{F}_{k,h}] = [P_h\widetilde{\Delta}^{\pi}_{k,h+1}](s_h^k,a_h^k) \le[P_h(\widetilde{\Delta}^{\pi}_{k,h+1})^2](s_h^k,a_h^k) \le H[P_h\widetilde{\Delta}^{\pi}_{k,h+1}](s_h^k,a_h^k), E [( M k , h π ) 2 ∣ F k , h ] = [ P h Δ k , h + 1 π ] ( s h k , a h k ) ≤ [ P h ( Δ k , h + 1 π ) 2 ] ( s h k , a h k ) ≤ H [ P h Δ k , h + 1 π ] ( s h k , a h k ) ,
with the identical bound holding for M k , h u l M^{\mathrm{ul}}_{k,h} M k , h ul . For each starting stage h ∈ [ H ] h \in[H] h ∈ [ H ] , applying Lemma E.1 to { M k , h ′ π } k ∈ [ K ] , h ′ ≥ h \{M^{\pi}_{k,h'}\}_{k\in[K],h'\ge h} { M k , h ′ π } k ∈ [ K ] , h ′ ≥ h and { M k , h ′ u l } k ∈ [ K ] , h ′ ≥ h \{M^{\mathrm{ul}}_{k,h'}\}_{k\in[K],h'\ge h} { M k , h ′ ul } k ∈ [ K ] , h ′ ≥ h with range B = H B = H B = H , step size η = 1 / ( 2 H 2 ) ∈ ( 0 , 1 / H ] \eta= 1/(2H^2) \in(0,1/H] η = 1/ ( 2 H 2 ) ∈ ( 0 , 1/ H ] , and failure probability δ / H \delta/H δ / H , and taking a union bound over h ∈ [ H ] h \in[H] h ∈ [ H ] , we obtain an event E F r e e d \mathcal{E}_{\mathrm{Freed}} E Freed with P ( E F r e e d ) ≥ 1 − 2 δ \mathbb{P}(\mathcal{E}_{\mathrm{Freed}}) \ge1 - 2\delta P ( E Freed ) ≥ 1 − 2 δ on which for all h ∈ [ H ] h \in[H] h ∈ [ H ] :
∑ k = 1 K ∑ h ′ = h H M k , h ′ π ≤ 1 2 H ∑ k = 1 K ∑ h ′ = h H [ P h ′ Δ ~ k , h ′ + 1 π ] ( s h ′ k , a h ′ k ) + 2 H 2 ln ( H / δ ) , (58) \sum_{k=1}^{K}\sum_{h'=h}^{H} M^{\pi}_{k,h'} \le\frac{1}{2H}\sum_{k=1}^{K}\sum_{h'=h}^{H}[P_{h'}\widetilde{\Delta}^{\pi}_{k,h'+1}](s_{h'}^k,a_{h'}^k) + 2H^2\ln(H/\delta), \tag*{(58)} k = 1 ∑ K h ′ = h ∑ H M k , h ′ π ≤ 2 H 1 k = 1 ∑ K h ′ = h ∑ H [ P h ′ Δ k , h ′ + 1 π ] ( s h ′ k , a h ′ k ) + 2 H 2 ln ( H / δ ) , (58)
∑ k = 1 K ∑ h ′ = h H M k , h ′ u l ≤ 1 2 H ∑ k = 1 K ∑ h ′ = h H [ P h ′ Δ ~ k , h ′ + 1 u l ] ( s h ′ k , a h ′ k ) + 2 H 2 ln ( H / δ ) . (59) \sum_{k=1}^{K}\sum_{h'=h}^{H} M^{\mathrm{ul}}_{k,h'} \le\frac{1}{2H}\sum_{k=1}^{K}\sum_{h'=h}^{H}[P_{h'}\widetilde{\Delta}^{\mathrm{ul}}_{k,h'+1}](s_{h'}^k,a_{h'}^k) + 2H^2\ln(H/\delta). \tag*{(59)} k = 1 ∑ K h ′ = h ∑ H M k , h ′ ul ≤ 2 H 1 k = 1 ∑ K h ′ = h ∑ H [ P h ′ Δ k , h ′ + 1 ul ] ( s h ′ k , a h ′ k ) + 2 H 2 ln ( H / δ ) . (59)
Now condition on E c o n c ∩ E F r e e d \mathcal{E}_{\mathrm{conc}} \cap\mathcal{E}_{\mathrm{Freed}} E conc ∩ E Freed . By property (P4)k , h _{k,h} k , h of Lemma A.5, we have 0 ≤ V ~ k , h ≤ V h ∗ ≤ V k , h ≤ H 0 \le\widetilde{V}_{k,h} \le V_h^* \le V_{k,h} \le H 0 ≤ V k , h ≤ V h ∗ ≤ V k , h ≤ H and 0 ≤ V h π k ≤ V h ∗ ≤ V k , h ≤ H 0 \le V_h^{\pi^k} \le V_h^* \le V_{k,h} \le H 0 ≤ V h π k ≤ V h ∗ ≤ V k , h ≤ H , so Δ ~ k , h π = V k , h − V h π k \widetilde{\Delta}^{\pi}_{k,h} = V_{k,h} - V_h^{\pi^k} Δ k , h π = V k , h − V h π k and Δ ~ k , h u l = V k , h − V ~ k , h \widetilde{\Delta}^{\mathrm{ul}}_{k,h} = V_{k,h} - \widetilde{V}_{k,h} Δ k , h ul = V k , h − V k , h for all ( k , h ) ∈ [ K ] × [ H ] (k,h) \in[K] \times[H] ( k , h ) ∈ [ K ] × [ H ] . Let k l a s t ≤ k k_{\mathrm{last}} \le k k last ≤ k denote the most recent episode at which Line 4 of
Algorithm 1 was triggered prior to or at episode k k k . For every k ∈ [ K ] k \in[K] k ∈ [ K ] and h ∈ [ H ] h \in[H] h ∈ [ H ] , we first bound Q k l a s t , h ( s h k , a h k ) − Q h π k ( s h k , a h k ) Q_{k_{\mathrm{last},h}}(s_h^k,a_h^k)-Q_h^{\pi^k}(s_h^k,a_h^k) Q k last , h ( s h k , a h k ) − Q h π k ( s h k , a h k ) :
Q k l a s t , h ( s h k , a h k ) − Q h π k ( s h k , a h k ) ≤ r h ( s h k , a h k ) + w ^ k l a s t , h T ϕ ( s h k , a h k ) + β ∥ ϕ ( s h k , a h k ) ∥ Σ k l a s t , h − 1 − Q h π k ( s h k , a h k ) = ( w ^ k l a s t , h T ϕ ( s h k , a h k ) − [ P h V k l a s t , h + 1 ] ( s h k , a h k ) ) + [ P h Δ ~ k , h + 1 π ] ( s h k , a h k ) + β ∥ ϕ ( s h k , a h k ) ∥ Σ k l a s t , h − 1 ≤ [ P h Δ ~ k , h + 1 π ] ( s h k , a h k ) + 2 β ∥ ϕ ( s h k , a h k ) ∥ Σ k l a s t , h − 1 , \begin{aligned}
Q_{k_{\mathrm{last},h}}(s_h^k,a_h^k)-Q_h^{\pi^k}(s_h^k,a_h^k)
&\le r_h(s_h^k,a_h^k)+\widehat{w}_{k_{\mathrm{last},h}}^{\mathsf T}\phi(s_h^k,a_h^k)+\beta\|\phi(s_h^k,a_h^k)\|_{\Sigma_{k_{\mathrm{last},h}}^{-1}}-Q_h^{\pi^k}(s_h^k,a_h^k)\\
&=\left(\widehat{w}_{k_{\mathrm{last},h}}^{\mathsf T}\phi(s_h^k,a_h^k)-[\mathbb{P}_hV_{k_{\mathrm{last},h+1} }](s_h^k,a_h^k)\right)\\
&\quad+[\mathbb{P}_h\widetilde{\Delta}_{k,h+1}^{\pi}](s_h^k,a_h^k)+\beta\|\phi(s_h^k,a_h^k)\|_{\Sigma_{k_{\mathrm{last},h}}^{-1}}\\
&\le[\mathbb{P}_h\widetilde{\Delta}_{k,h+1}^{\pi}](s_h^k,a_h^k)+2\beta\|\phi(s_h^k,a_h^k)\|_{\Sigma_{k_{\mathrm{last},h}}^{-1}},
\end{aligned} Q k last , h ( s h k , a h k ) − Q h π k ( s h k , a h k ) ≤ r h ( s h k , a h k ) + w k last , h T ϕ ( s h k , a h k ) + β ∥ ϕ ( s h k , a h k ) ∥ Σ k last , h − 1 − Q h π k ( s h k , a h k ) = ( w k last , h T ϕ ( s h k , a h k ) − [ P h V k last , h + 1 ] ( s h k , a h k ) ) + [ P h Δ k , h + 1 π ] ( s h k , a h k ) + β ∥ ϕ ( s h k , a h k ) ∥ Σ k last , h − 1 ≤ [ P h Δ k , h + 1 π ] ( s h k , a h k ) + 2 β ∥ ϕ ( s h k , a h k ) ∥ Σ k last , h − 1 ,
where the first inequality follows from (2) , the equality uses Q h π k ( s h k , a h k ) = r h ( s h k , a h k ) + [ P h V h + 1 π k ] ( s h k , a h k ) Q_h^{\pi^k}(s_h^k,a_h^k)=r_h(s_h^k,a_h^k)+[\mathbb{P}_hV_{h+1}^{\pi^k}](s_h^k,a_h^k) Q h π k ( s h k , a h k ) = r h ( s h k , a h k ) + [ P h V h + 1 π k ] ( s h k , a h k ) and V k l a s t , h + 1 = V k , h + 1 V_{k_{\mathrm{last},h+1}}=V_{k,h+1} V k last , h + 1 = V k , h + 1 , and the second inequality applies (A.16) of ( P 1 ) k l a s t , h (\mathrm{P1})_{k_{\mathrm{last},h}} ( P1 ) k last , h }$. Since Δ ~ k , h π ( s h k ) = Q k l a s t , h ( s h k , a h k ) − Q h π k ( s h k , a h k ) ≤ H \widetilde{\Delta}_{k,h}^{\pi}(s_h^k)=Q_{k_{\mathrm{last},h}}(s_h^k,a_h^k)-Q_h^{\pi^k}(s_h^k,a_h^k)\le H Δ k , h π ( s h k ) = Q k last , h ( s h k , a h k ) − Q h π k ( s h k , a h k ) ≤ H , we further have
Δ ~ k , h π ( s h k ) ≤ min { H , [ P h Δ ~ k , h + 1 π ] ( s h k , a h k ) + 2 β ∥ ϕ ( s h k , a h k ) ∥ Σ k l a s t , h − 1 } ≤ [ P h Δ ~ k , h + 1 π ] ( s h k , a h k ) + 2 min { β ∥ ϕ ( s h k , a h k ) ∥ Σ k l a s t , h − 1 , H } ≤ [ P h Δ ~ k , h + 1 π ] ( s h k , a h k ) + B k , h = Δ ~ k , h + 1 π ( s h + 1 k ) + M k , h π + B k , h . (60) \begin{aligned}
\widetilde{\Delta}_{k,h}^{\pi}(s_h^k)\le\min\left\{H,[\mathbb{P}_h\widetilde{\Delta}_{k,h+1}^{\pi}](s_h^k,a_h^k)+2\beta\|\phi(s_h^k,a_h^k)\|_{\Sigma_{k_{\mathrm{last},h}}^{-1}}\right\} \\
&\le[\mathbb{P}_h\widetilde{\Delta}_{k,h+1}^{\pi}](s_h^k,a_h^k)+2\min\left\{\beta\|\phi(s_h^k,a_h^k)\|_{\Sigma_{k_{\mathrm{last},h}}^{-1}},H\right\} \\
&\le[\mathbb{P}_h\widetilde{\Delta}_{k,h+1}^{\pi}](s_h^k,a_h^k)+B_{k,h}=\widetilde{\Delta}_{k,h+1}^{\pi}(s_{h+1}^k)+M_{k,h}^{\pi}+B_{k,h}. \tag*{(60)}
\end{aligned} Δ k , h π ( s h k ) ≤ min { H , [ P h Δ k , h + 1 π ] ( s h k , a h k ) + 2 β ∥ ϕ ( s h k , a h k ) ∥ Σ k last , h − 1 } ≤ [ P h Δ k , h + 1 π ] ( s h k , a h k ) + 2 min { β ∥ ϕ ( s h k , a h k ) ∥ Σ k last , h − 1 , H } ≤ [ P h Δ k , h + 1 π ] ( s h k , a h k ) + B k , h = Δ k , h + 1 π ( s h + 1 k ) + M k , h π + B k , h . (60)
where the second inequality uses [ P h Δ ~ k , h + 1 π ] ( s h k , a h k ) ≥ 0 [\mathbb{P}_h\widetilde{\Delta}_{k,h+1}^{\pi}](s_h^k,a_h^k)\ge0 [ P h Δ k , h + 1 π ] ( s h k , a h k ) ≥ 0 , and the third inequality uses (B.1) and Lemma E.7, which gives
∥ ϕ ( s h k , a h k ) ∥ Σ k l a s t , h − 1 ≤ 2 ∥ ϕ ( s h k , a h k ) ∥ Σ k , h − 1 ≤ 2 ∥ ϕ ( s h k , a h k ) ∥ Σ k , h − 1 . \|\phi(s_h^k,a_h^k)\|_{\Sigma_{k_{\mathrm{last},h}}^{-1}}\le\sqrt{2}\|\phi(s_h^k,a_h^k)\|_{\Sigma_{k,h}^{-1}}\le2\|\phi(s_h^k,a_h^k)\|_{\Sigma_{k,h}^{-1}}. ∥ ϕ ( s h k , a h k ) ∥ Σ k last , h − 1 ≤ 2 ∥ ϕ ( s h k , a h k ) ∥ Σ k , h − 1 ≤ 2∥ ϕ ( s h k , a h k ) ∥ Σ k , h − 1 .
since Σ k , h ⪰ Σ k l a s t , h \Sigma_{k,h}\succeq\Sigma_{k_{\mathrm{last},h}} Σ k , h ⪰ Σ k last , h and det ( Σ k , h ) < 2 det ( Σ k l a s t , h ) \det(\Sigma_{k,h})<2\det(\Sigma_{k_{\mathrm{last},h}}) det ( Σ k , h ) < 2 det ( Σ k last , h ) . Telescoping (B.13) over h ′ = h , … , H h'=h,\ldots,H h ′ = h , … , H (with Δ ~ k , H + 1 π = 0 \widetilde{\Delta}_{k,H+1}^{\pi}=0 Δ k , H + 1 π = 0 ) and summing over k ∈ [ K ] k\in[K] k ∈ [ K ] yields, for every h ∈ [ H ] h\in[H] h ∈ [ H ] :
∑ k = 1 K Δ ~ k , h π ( s h k ) ≤ ∑ k = 1 K ∑ h ′ = h H B k , h ′ + ∑ k = 1 K ∑ h ′ = h H M k , h ′ π ≤ ∑ k = 1 K ∑ h = 1 H B k , h + 1 2 H S π + 2 H 2 ln ( H / δ ) . (61) \sum_{k=1}^{K}\widetilde{\Delta}_{k,h}^{\pi}(s_h^k)\le\sum_{k=1}^{K}\sum_{h'=h}^{H}B_{k,h'}+\sum_{k=1}^{K}\sum_{h'=h}^{H}M_{k,h'}^{\pi}\le\sum_{k=1}^{K}\sum_{h=1}^{H}B_{k,h}+\frac{1}{2H}S_{\pi}+2H^2\ln(H/\delta). \tag*{(61)} k = 1 ∑ K Δ k , h π ( s h k ) ≤ k = 1 ∑ K h ′ = h ∑ H B k , h ′ + k = 1 ∑ K h ′ = h ∑ H M k , h ′ π ≤ k = 1 ∑ K h = 1 ∑ H B k , h + 2 H 1 S π + 2 H 2 ln ( H / δ ) . (61)
where the second inequality applies (B.11) and ∑ k = 1 K ∑ h ′ = h H [ P h ′ Δ ~ k , h ′ + 1 π ] ( s h ′ k , a h ′ k ) ≤ S π \sum_{k=1}^{K}\sum_{h'=h}^{H}[\mathbb{P}_{h'}\widetilde{\Delta}_{k,h'+1}^{\pi}](s_{h'}^k,a_{h'}^k)\le S_{\pi} ∑ k = 1 K ∑ h ′ = h H [ P h ′ Δ k , h ′ + 1 π ] ( s h ′ k , a h ′ k ) ≤ S π from (B.3). Consequently, expanding S π S_{\pi} S π from (B.3) gives
S π = ∑ k = 1 K ∑ h = 1 H ( Δ ~ k , h + 1 π ( s h + 1 k ) + M k , h π ) = ∑ h = 2 H ∑ k = 1 K Δ ~ k , h π ( s h k ) + ∑ k = 1 K ∑ h = 1 H M k , h π ≤ ( H − 1 ) ( ∑ k = 1 K ∑ h = 1 H B k , h + 1 2 H S π + 2 H 2 ln ( H / δ ) ) + 1 2 H S π + 2 H 2 ln ( H / δ ) ≤ H ∑ k = 1 K ∑ h = 1 H B k , h + 1 2 S π + 2 H 3 ln ( H / δ ) , \begin{aligned}
S_{\pi}
&=\sum_{k=1}^{K}\sum_{h=1}^{H}\left(\widetilde{\Delta}_{k,h+1}^{\pi}(s_{h+1}^k)+M_{k,h}^{\pi}\right)\\
&=\sum_{h=2}^{H}\sum_{k=1}^{K}\widetilde{\Delta}_{k,h}^{\pi}(s_h^k)+\sum_{k=1}^{K}\sum_{h=1}^{H}M_{k,h}^{\pi}\\
&\le(H-1)\left(\sum_{k=1}^{K}\sum_{h=1}^{H}B_{k,h}+\frac{1}{2H}S_{\pi}+2H^2\ln(H/\delta)\right)+\frac{1}{2H}S_{\pi}+2H^2\ln(H/\delta)\\
&\le H\sum_{k=1}^{K}\sum_{h=1}^{H}B_{k,h}+\frac{1}{2}S_{\pi}+2H^3\ln(H/\delta),
\end{aligned} S π = k = 1 ∑ K h = 1 ∑ H ( Δ k , h + 1 π ( s h + 1 k ) + M k , h π ) = h = 2 ∑ H k = 1 ∑ K Δ k , h π ( s h k ) + k = 1 ∑ K h = 1 ∑ H M k , h π ≤ ( H − 1 ) ( k = 1 ∑ K h = 1 ∑ H B k , h + 2 H 1 S π + 2 H 2 ln ( H / δ ) ) + 2 H 1 S π + 2 H 2 ln ( H / δ ) ≤ H k = 1 ∑ K h = 1 ∑ H B k , h + 2 1 S π + 2 H 3 ln ( H / δ ) ,
where the second equality shifts the stage index h + 1 ↦ h h+1\mapsto h h + 1 ↦ h using Δ ~ k , H + 1 π = 0 \widetilde{\Delta}_{k,H+1}^{\pi}=0 Δ k , H + 1 π = 0 , and the first inequality applies (B.14) for each h ∈ { 2 , … , H } h\in\{2,\ldots,H\} h ∈ { 2 , … , H } and (B.11) at h = 1 h=1 h = 1 . Rearranging proves (B.9). Using (B.9), we bound the cumulative regret as
Regret ( K ) ≤ ∑ k = 1 K ( V k , 1 ( s 1 k ) − V 1 π k ( s 1 k ) ) = ∑ k = 1 K Δ ~ k , 1 π ( s 1 k ) \operatorname{Regret}(K)\le\sum_{k=1}^{K}\left(V_{k,1}(s_1^k)-V_1^{\pi^k}(s_1^k)\right)=\sum_{k=1}^{K}\widetilde{\Delta}_{k,1}^{\pi}(s_1^k) Regret ( K ) ≤ k = 1 ∑ K ( V k , 1 ( s 1 k ) − V 1 π k ( s 1 k ) ) = k = 1 ∑ K Δ k , 1 π ( s 1 k )
e n d s m i d = 1 ≤ ∑ k = 1 K ∑ h = 1 H B k , h + 1 2 H S π + 2 H 2 ln ( H / δ ) ≤ 2 ∑ k = 1 K ∑ h = 1 H B k , h + 4 H 2 ln ( H / δ ) \begin{aligned}
{ends_mid=1} \\
&\le\sum_{k=1}^{K}\sum_{h=1}^{H} B_{k,h}+\frac{1}{2H}S_{\pi}+2H^{2}\ln(H/\delta)\le2\sum_{k=1}^{K}\sum_{h=1}^{H}B_{k,h}+4H^{2}\ln(H/\delta)
\end{aligned} e n d s m i d = 1 ≤ k = 1 ∑ K h = 1 ∑ H B k , h + 2 H 1 S π + 2 H 2 ln ( H / δ ) ≤ 2 k = 1 ∑ K h = 1 ∑ H B k , h + 4 H 2 ln ( H / δ )
where the first inequality holds by optimism ((P4)k , 1 _{k,1} k , 1 of Lemma A.5 ), the second inequality applies (B.14) at h = 1 h=1 h = 1 , and the third substitutes (B.9). This establishes (B.8). Finally, since V ˘ k , h ( s h k ) = max a ∈ A Q ˘ k , l a s t , h ( s h k , a ) ≥ Q ˘ k , l a s t , h ( s h k , a h k ) \breve{V}_{k,h}(s_{h}^{k})=\max_{a\in\mathcal{A}}\breve{Q}_{k,\mathrm{last},h}(s_{h}^{k},a)\geq\breve{Q}_{k,\mathrm{last},h}(s_{h}^{k},a_{h}^{k}) V ˘ k , h ( s h k ) = max a ∈ A Q ˘ k , last , h ( s h k , a ) ≥ Q ˘ k , last , h ( s h k , a h k ) and β ˉ = β \bar{\beta}=\beta β ˉ = β , (4.2) and (A.17) imply that $\widetilde{\Delta}_{k,h}^{\mathrm{ul}}(s_{h}^{k})$ satisfies {xref}(B.13) with $B_{k,h}$ replaced by $2B_{k,h}$ and $M_{k,h}^{\pi}$ replaced by $M_{k,h}^{\mathrm{ul}}$. Repeating the same telescoping derivation with {xref}(B.12) and {xref}(B.4) yields {xref}(B.10)`.
Total Estimated Variance Bound and Proof of Theorem 5.1
Lemma B.3 (Total Estimated Variance Bound). On the event E c o n c ∩ E F r e e d \mathcal{E}_{\mathrm{conc}}\cap\mathcal{E}_{\mathrm{Freed}} E conc ∩ E Freed , the total estimated variance V e s t V_{\mathrm{est}} V est defined in (B.2) satisfies
V e s t ≤ 2 min { V a r K ∗ , V a r K π } + O ( d 7 H 7 β ι 2 + d 7 H 7 β ι 3 ) V_{\mathrm{est}}\leq2\min\{\mathrm{Var}_{K}^{*},\mathrm{Var}_{K}^{\pi}\}+\mathcal{O}\left(d^{7}H^{7}\beta_{\iota}^{2}+d^{7}H^{7}\beta_{\iota}^{3}\right) V est ≤ 2 min { Var K ∗ , Var K π } + O ( d 7 H 7 β ι 2 + d 7 H 7 β ι 3 )
≤ 2 min { V a r K ∗ , V a r K π } + O ( d 8 H 7 ι 0 4 + d 7 H 7 ι 0 4 , 2 ) . (62) \leq2\min\{\mathrm{Var}_{K}^{*},\mathrm{Var}_{K}^{\pi}\}+\mathcal{O}\left(d^{8}H^{7}\iota_{0}^{4}+d^{7}H^{7}\iota_{0}^{4,2}\right). \tag*{(62)} ≤ 2 min { Var K ∗ , Var K π } + O ( d 8 H 7 ι 0 4 + d 7 H 7 ι 0 4 , 2 ) . (62)
Proof. Condition on E c o n c ∩ E F r e e d \mathcal{E}_{\mathrm{conc}}\cap\mathcal{E}_{\mathrm{Freed}} E conc ∩ E Freed . Expanding V e s t V_{\mathrm{est}} V est from (7) and (B.2) gives
V e s t = ∑ k = 1 K ∑ h = 1 H [ V ˉ k , h V k , h + 1 ] ( s h k , a h k ) ⏟ = : ( ⋄ ) + ∑ k = 1 K ∑ h = 1 H E k , h + ∑ k = 1 K ∑ h = 1 H D k , h + H 3 . (63) V_{\mathrm{est}}=\underbrace{\sum_{k=1}^{K}\sum_{h=1}^{H}[\bar{V}_{k,h}V_{k,h+1}](s_{h}^{k},a_{h}^{k})}_{=:(\diamond)}+\sum_{k=1}^{K}\sum_{h=1}^{H}E_{k,h}+\sum_{k=1}^{K}\sum_{h=1}^{H}D_{k,h}+H^{3}. \tag*{(63)} V est = =: ( ⋄ ) k = 1 ∑ K h = 1 ∑ H [ V ˉ k , h V k , h + 1 ] ( s h k , a h k ) + k = 1 ∑ K h = 1 ∑ H E k , h + k = 1 ∑ K h = 1 ∑ H D k , h + H 3 . (63)
where we used ∑ k = 1 K ∑ h = 1 H α 2 = K H ( H 2 / K ) = H 3 \sum_{k=1}^{K}\sum_{h=1}^{H}\alpha^{2}=KH(H^{2}/K)=H^{3} ∑ k = 1 K ∑ h = 1 H α 2 = K H ( H 2 / K ) = H 3 . We bound ( ⋄ ) (\diamond) ( ⋄ ) in terms of V a r K ∗ \mathrm{Var}_{K}^{*} Var K ∗ and V a r K π \mathrm{Var}_{K}^{\pi} Var K π separately. On the one hand, summing (A.20) of Lemma A.5 over ( k , h ) ∈ [ K ] × [ H ] (k,h)\in[K]\times[H] ( k , h ) ∈ [ K ] × [ H ] gives
( ⋄ ) ≤ V a r K ∗ + ∑ k = 1 K ∑ h = 1 H E k , h + ∑ k = 1 K ∑ h = 1 H D k , h . (64) (\diamond)\leq\mathrm{Var}_{K}^{*}+\sum_{k=1}^{K}\sum_{h=1}^{H}E_{k,h}+\sum_{k=1}^{K}\sum_{h=1}^{H}D_{k,h}. \tag*{(64)} ( ⋄ ) ≤ Var K ∗ + k = 1 ∑ K h = 1 ∑ H E k , h + k = 1 ∑ K h = 1 ∑ H D k , h . (64)
On the other hand, relating [ V ˉ k , h V k , h + 1 ] ( s h k , a h k ) [\bar{V}_{k,h}V_{k,h+1}](s_{h}^{k},a_{h}^{k}) [ V ˉ k , h V k , h + 1 ] ( s h k , a h k ) to [ V h V h + 1 π ] ( s h k , a h k ) [V_{h}V_{h+1}^{\pi}](s_{h}^{k},a_{h}^{k}) [ V h V h + 1 π ] ( s h k , a h k ) yields
( ⋄ ) ≤ ∑ k = 1 K ∑ h = 1 H [ V h V k , h + 1 ] ( s h k , a h k ) + ∑ k = 1 K ∑ h = 1 H E k , h = V a r K π + ∑ k = 1 K ∑ h = 1 H ( [ V h V k , h + 1 ] ( s h k , a h k ) − [ V h V h + 1 π ] ( s h k , a h k ) ) + ∑ k = 1 K ∑ h = 1 H E k , h ≤ V a r K π + ∑ k = 1 K ∑ h = 1 H ( [ P h ( V k , h + 1 2 − ( V h + 1 π ) 2 ) ] ( s h k , a h k ) ) + ∑ k = 1 K ∑ h = 1 H E k , h ≤ V a r K π + 2 H S π + ∑ k = 1 K ∑ h = 1 H E k , h . (65) \begin{aligned}
(\diamond)&\leq\sum_{k=1}^{K}\sum_{h=1}^{H}[V_{h}V_{k,h+1}](s_{h}^{k},a_{h}^{k})+\sum_{k=1}^{K}\sum_{h=1}^{H}E_{k,h}\\
&=\mathrm{Var}_{K}^{\pi}+\sum_{k=1}^{K}\sum_{h=1}^{H}\left([V_{h}V_{k,h+1}](s_{h}^{k},a_{h}^{k})-[V_{h}V_{h+1}^{\pi}](s_{h}^{k},a_{h}^{k})\right)+\sum_{k=1}^{K}\sum_{h=1}^{H}E_{k,h}\\
&\leq\mathrm{Var}_{K}^{\pi}+\sum_{k=1}^{K}\sum_{h=1}^{H}\left([\mathbb{P}_{h}(V_{k,h+1}^{2}-(V_{h+1}^{\pi})^{2})](s_{h}^{k},a_{h}^{k})\right)+\sum_{k=1}^{K}\sum_{h=1}^{H}E_{k,h}\\
&\leq\mathrm{Var}_{K}^{\pi}+2HS_{\pi}+\sum_{k=1}^{K}\sum_{h=1}^{H}E_{k,h}. \tag*{(65)}
\end{aligned} ( ⋄ ) ≤ k = 1 ∑ K h = 1 ∑ H [ V h V k , h + 1 ] ( s h k , a h k ) + k = 1 ∑ K h = 1 ∑ H E k , h = Var K π + k = 1 ∑ K h = 1 ∑ H ( [ V h V k , h + 1 ] ( s h k , a h k ) − [ V h V h + 1 π ] ( s h k , a h k ) ) + k = 1 ∑ K h = 1 ∑ H E k , h ≤ Var K π + k = 1 ∑ K h = 1 ∑ H ( [ P h ( V k , h + 1 2 − ( V h + 1 π ) 2 )] ( s h k , a h k ) ) + k = 1 ∑ K h = 1 ∑ H E k , h ≤ Var K π + 2 H S π + k = 1 ∑ K h = 1 ∑ H E k , h . (65)
where the first inequality follows from (A.19) of Lemma A.5 , the second inequality holds since [ P h V k , h + 1 ] ( s h k , a h k ) ≥ [ P h V h + 1 π ] ( s h k , a h k ) ≥ 0 [\mathbb{P}_{h}V_{k,h+1}](s_{h}^{k},a_{h}^{k})\geq[\mathbb{P}_{h}V_{h+1}^{\pi}](s_{h}^{k},a_{h}^{k})\geq0 [ P h V k , h + 1 ] ( s h k , a h k ) ≥ [ P h V h + 1 π ] ( s h k , a h k ) ≥ 0 by (P4)k , h + 1 _{k,h+1} k , h + 1 , and the last inequality uses V k , h + 1 2 − V_{k,h+1}^{2}- V k , h + 1 2 − ( V h + 1 k ) 2 ≤ 2 H ( V k , h + 1 − V h + 1 k ) (V_{h+1}^{k})^2 \le2H(V_{k,h+1}-V_{h+1}^{k}) ( V h + 1 k ) 2 ≤ 2 H ( V k , h + 1 − V h + 1 k ) (since 0 ≤ V h + 1 k ≤ V k , h + 1 ≤ H 0 \le V_{h+1}^{k} \le V_{k,h+1} \le H 0 ≤ V h + 1 k ≤ V k , h + 1 ≤ H ) together with the definition of S π S_{\pi} S π in (B.3). Substituting the minimum of (B.17) and (B.18) into (B.16) gives
V e s t ≤ min { V K a r ∗ , V K a r π } + 2 ∑ k = 1 K ∑ h = 1 H E k , h ⏟ = : I 1 + 2 ∑ k = 1 K ∑ h = 1 H D k , h ⏟ = : I 2 + 2 H S π + H 3 . (66) V_{\mathrm{est}} \le\min\{V_K^{\mathrm{ar}*}, V_K^{\mathrm{ar}\pi}\} + 2\underbrace{\sum_{k=1}^{K}\sum_{h=1}^{H} E_{k,h}}_{=:I_1} + 2\underbrace{\sum_{k=1}^{K}\sum_{h=1}^{H} D_{k,h}}_{=:I_2} + 2HS_{\pi} + H^3. \tag*{(66)} V est ≤ min { V K ar ∗ , V K ar π } + 2 =: I 1 k = 1 ∑ K h = 1 ∑ H E k , h + 2 =: I 2 k = 1 ∑ K h = 1 ∑ H D k , h + 2 H S π + H 3 . (66)
We now bound I 1 I_1 I 1 and I 2 I_2 I 2 in turn. For I 1 I_1 I 1 , expanding E k , h E_{k,h} E k , h from (5) gives
I 1 = 2 ∑ h = 1 H ∑ k = 1 K min { 2 H β ∥ ϕ ( s h k , a h k ) ∥ Σ k , h − 1 , H 2 } + 2 ∑ h = 1 H ∑ k = 1 K min { 2 H β ∥ ϕ ( s h k , a h k ) ∥ Σ k , h − 1 , H 2 } I_1 = 2\sum_{h=1}^{H}\sum_{k=1}^{K}\min\left\{2H\beta\|\phi(s_h^k,a_h^k)\|_{\Sigma_{k,h}^{-1}},H^2\right\}+2\sum_{h=1}^{H}\sum_{k=1}^{K}\min\left\{2H\beta\|\phi(s_h^k,a_h^k)\|_{\Sigma_{k,h}^{-1}},H^2\right\} I 1 = 2 h = 1 ∑ H k = 1 ∑ K min { 2 H β ∥ ϕ ( s h k , a h k ) ∥ Σ k , h − 1 , H 2 } + 2 h = 1 ∑ H k = 1 ∑ K min { 2 H β ∥ ϕ ( s h k , a h k ) ∥ Σ k , h − 1 , H 2 }
≤ 8 d H 3 ι + 16 β γ 2 d H 2 ι + 8 H β 2 d ι ∑ h = 1 H ∑ k = 1 K σ k , h 2 \le8dH^3\iota+16\beta\gamma^2dH^2\iota+8H\beta\sqrt{2d\iota}\sum_{h=1}^{H}\sqrt{\sum_{k=1}^{K}\sigma_{k,h}^2} ≤ 8 d H 3 ι + 16 β γ 2 d H 2 ι + 8 H β 2 d ι h = 1 ∑ H k = 1 ∑ K σ k , h 2
≤ 8 d H 3 ι + 16 β γ 2 d H 2 ι + 8 H β 2 d H ι V e s t . (67) \le8dH^3\iota+16\beta\gamma^2dH^2\iota+8H\beta\sqrt{2dH\iota V_{\mathrm{est}}}. \tag*{(67)} ≤ 8 d H 3 ι + 16 β γ 2 d H 2 ι + 8 H β 2 d H ι V est . (67)
where the equality uses β ~ = 2 H β \widetilde{\beta}=2H\beta β = 2 H β and β ‾ = β \overline{\beta}=\beta β = β from (10) , the first inequality applies (B.5) of Lemma B.1 at each stage h ∈ [ H ] h\in[H] h ∈ [ H ] , and the second inequality follows from the Cauchy–Schwarz inequality and (B.2). For I 2 I_2 I 2 , each summand D k , h D_{k,h} D k , h in (6) satisfies
D k , h ≤ min { 4 d 3 H 2 [ P h ( V k , h + 1 − V ~ k , h + 1 ) ] ( s h k , a h k ) + 16 d 3 H 2 β ∥ ϕ ( s h k , a h k ) ∥ Σ k , h − 1 , d 3 H 3 } D_{k,h}\le\min\left\{4d^3H^2[\mathbb{P}_h(V_{k,h+1}-\widetilde{V}_{k,h+1})](s_h^k,a_h^k)+16d^3H^2\beta\|\phi(s_h^k,a_h^k)\|_{\Sigma_{k,h}^{-1}},d^3H^3\right\} D k , h ≤ min { 4 d 3 H 2 [ P h ( V k , h + 1 − V k , h + 1 )] ( s h k , a h k ) + 16 d 3 H 2 β ∥ ϕ ( s h k , a h k ) ∥ Σ k , h − 1 , d 3 H 3 }
≤ 4 d 3 H 2 [ P h ( V k , h + 1 − V ~ k , h + 1 ) ] ( s h k , a h k ) + min { 16 d 3 H 2 β ∥ ϕ ( s h k , a h k ) ∥ Σ k , h − 1 , d 3 H 3 } , \le4d^3H^2[\mathbb{P}_h(V_{k,h+1}-\widetilde{V}_{k,h+1})](s_h^k,a_h^k)+\min\left\{16d^3H^2\beta\|\phi(s_h^k,a_h^k)\|_{\Sigma_{k,h}^{-1}},d^3H^3\right\}, ≤ 4 d 3 H 2 [ P h ( V k , h + 1 − V k , h + 1 )] ( s h k , a h k ) + min { 16 d 3 H 2 β ∥ ϕ ( s h k , a h k ) ∥ Σ k , h − 1 , d 3 H 3 } ,
where the first inequality follows from (A.16) and (A.17) of Lemma A.5 together with β ‾ = β \overline{\beta}=\beta β = β , and the second inequality uses min { a + b , c } ≤ a + min { b , c } \min\{a+b,c\}\le a+\min\{b,c\} min { a + b , c } ≤ a + min { b , c } for a : = 4 d 3 H 2 [ P h ( V k , h + 1 − V ~ k , h + 1 ) ] ( s h k , a h k ) ≥ 0 a:=4d^3H^2[\mathbb{P}_h(V_{k,h+1}-\widetilde{V}_{k,h+1})](s_h^k,a_h^k)\ge0 a := 4 d 3 H 2 [ P h ( V k , h + 1 − V k , h + 1 )] ( s h k , a h k ) ≥ 0 . Summing over ( k , h ) ∈ [ K ] × [ H ] (k,h)\in[K]\times[H] ( k , h ) ∈ [ K ] × [ H ] yields
I 2 ≤ 8 d 3 H 2 S u l + 2 ∑ h = 1 H ∑ k = 1 K min { 16 d 3 H 2 β ∥ ϕ ( s h k , a h k ) ∥ Σ k , h − 1 , d 3 H 3 } I_2\le8d^3H^2S_{\mathrm{ul}}+2\sum_{h=1}^{H}\sum_{k=1}^{K}\min\left\{16d^3H^2\beta\|\phi(s_h^k,a_h^k)\|_{\Sigma_{k,h}^{-1}},d^3H^3\right\} I 2 ≤ 8 d 3 H 2 S ul + 2 h = 1 ∑ H k = 1 ∑ K min { 16 d 3 H 2 β ∥ ϕ ( s h k , a h k ) ∥ Σ k , h − 1 , d 3 H 3 }
≤ 8 d 3 H 2 S u l + 4 d 4 H 4 ι + 64 d 4 H 3 β γ 2 ι + 32 d 3 H 2 β 2 d H ι V e s t \le8d^3H^2S_{\mathrm{ul}}+4d^4H^4\iota+64d^4H^3\beta\gamma^2\iota+32d^3H^2\beta\sqrt{2dH\iota V_{\mathrm{est}}} ≤ 8 d 3 H 2 S ul + 4 d 4 H 4 ι + 64 d 4 H 3 β γ 2 ι + 32 d 3 H 2 β 2 d H ι V est
≤ 32 d 3 H 3 ∑ k = 1 K ∑ h = 1 H B k , h + 32 d 3 H 5 ln ( H / δ ) + 4 d 4 H 4 ι + 64 d 4 H 3 β γ 2 ι + 32 d 3 H 2 β 2 d H ι V e s t ≤ O ( d 4 H 4 β γ 2 ι + d 3 H 5 ln ( H / δ ) + d 3.5 H 3.5 β ι 0.5 V e s t ) . (68) \begin{aligned}
\le32d^3H^3\sum_{k=1}^{K}\sum_{h=1}^{H}B_{k,h}+32d^3H^5\ln(H/\delta)+4d^4H^4\iota+64d^4H^3\beta\gamma^2\iota+32d^3H^2\beta\sqrt{2dH\iota V_{\mathrm{est}}} \\
&\le O\left(d^4H^4\beta\gamma^2\iota+d^3H^5\ln(H/\delta)+d^{3.5}H^{3.5}\beta\iota^{0.5}\sqrt{V_{\mathrm{est}}}\right). \tag*{(68)}
\end{aligned} ≤ 32 d 3 H 3 k = 1 ∑ K h = 1 ∑ H B k , h + 32 d 3 H 5 ln ( H / δ ) + 4 d 4 H 4 ι + 64 d 4 H 3 β γ 2 ι + 32 d 3 H 2 β 2 d H ι V est ≤ O ( d 4 H 4 β γ 2 ι + d 3 H 5 ln ( H / δ ) + d 3.5 H 3.5 β ι 0.5 V est ) . (68)
where the first inequality uses the definition of S u l S_{\mathrm{ul}} S ul in (B.4), the second inequality applies (B.5) of Lemma B.1 and the Cauchy–Schwarz inequality across h ∈ [ H ] h\in[H] h ∈ [ H ] , the third inequality substitutes (B.10) of Lemma B.2, and the last inequality substitutes (B.6) of Lemma B.1. Combining (B.19), (B.20), (B.21), and (B.9) (together with (B.6)), we obtain
V e s t ≤ min { V K a r ∗ , V K a r π } + O ( d 3.5 H 3.5 β ι 0.5 V e s t + d 7 H 7 β ι 3 ) . V_{\mathrm{est}}\le\min\{V_K^{\mathrm{ar}*},V_K^{\mathrm{ar}\pi}\}+O\left(d^{3.5}H^{3.5}\beta\iota^{0.5}\sqrt{V_{\mathrm{est}}}+d^7H^7\beta\iota^3\right). V est ≤ min { V K ar ∗ , V K ar π } + O ( d 3.5 H 3.5 β ι 0.5 V est + d 7 H 7 β ι 3 ) .
where we used γ 2 = 4 d 3 H 3 ι 2 \gamma^2=4d^3H^3\iota^2 γ 2 = 4 d 3 H 3 ι 2 and ln ( H / δ ) ≤ ι 0 ≤ β \ln(H/\delta)\le\iota_0\le\beta ln ( H / δ ) ≤ ι 0 ≤ β . Applying the implication x ≤ A + B x ⟹ x ≤ 2 A + B 2 x\le A+B\sqrt{x}\Longrightarrow x\le2A+B^2 x ≤ A + B x ⟹ x ≤ 2 A + B 2 (for x , A , B ≥ 0 x,A,B\ge0 x , A , B ≥ 0 ) and substituting β = O ( d ι 2 ι 0 + ι ι 0 ) \beta=O(\sqrt{d\iota^2\iota_0}+\iota\iota_0) β = O ( d ι 2 ι 0 + ι ι 0 ) establishes (B.15).
Proof of Theorem 5.1. Condition on the joint event E c o n c ∩ E F r e e d \mathcal{E}_{\mathrm{conc}} \cap\mathcal{E}_{\mathrm{Freed}} E conc ∩ E Freed , which holds with probability at least 1 − 3 δ 1-3\delta 1 − 3 δ by Lemmas A.4 and B.2 and a union bound. We have
Regret ( K ) ≤ 2 ∑ k = 1 K ∑ h = 1 H B k , h + 4 H 2 ln ( H / δ ) \operatorname{Regret}(K) \le2\sum_{k=1}^{K}\sum_{h=1}^{H} B_{k,h}+4H^{2}\ln(H/\delta) Regret ( K ) ≤ 2 k = 1 ∑ K h = 1 ∑ H B k , h + 4 H 2 ln ( H / δ )
≤ 8 β 2 d H ι V e s t + 16 β γ 2 d H ι + 16 d H 2 ι + 4 H 2 ln ( H / δ ) ≤ O ( β d H ι ⋅ min { Var K ∗ , Var K π } + d 4 H 4 β 2 ι + d 4 H 4 β ι 3 ) ≤ O ( ( d ι 1.5 ι 0 + d ι 1.5 ι 0 ) H ⋅ min { Var K ∗ , Var K π } + d 5 H 4 ι 0 4 + d 4 H 4 ι 0 2 ) , \begin{aligned}
\le8\beta\sqrt{2dH\iota V_{\mathrm{est}}}+16\beta\gamma^{2}dH\iota+16dH^{2}\iota+4H^{2}\ln(H/\delta) \\
&\le O\left(\beta\sqrt{dH\iota}\cdot\min\left\{\operatorname{Var}^{*}_{K},\operatorname{Var}^{\pi}_{K}\right\}+d^{4}H^{4}\beta^{2}\iota+d^{4}H^{4}\beta\iota^{3}\right) \\
&\le O\left(\left(d\iota^{1.5}\sqrt{\iota_{0}}+\sqrt{d}\iota^{1.5}\iota_{0}\right)\sqrt{H\cdot\min\left\{\operatorname{Var}^{*}_{K},\operatorname{Var}^{\pi}_{K}\right\}}+d^{5}H^{4}\iota_{0}^{4}+d^{4}H^{4}\iota_{0}^{2}\right),
\end{aligned} ≤ 8 β 2 d H ι V est + 16 β γ 2 d H ι + 16 d H 2 ι + 4 H 2 ln ( H / δ ) ≤ O ( β d H ι ⋅ min { Var K ∗ , Var K π } + d 4 H 4 β 2 ι + d 4 H 4 β ι 3 ) ≤ O ( ( d ι 1.5 ι 0 + d ι 1.5 ι 0 ) H ⋅ min { Var K ∗ , Var K π } + d 5 H 4 ι 0 4 + d 4 H 4 ι 0 2 ) ,
where the first inequality is (B.8) of Lemma B.2, the second inequality substitutes (B.6) of Lemma B.1, the third inequality uses V e s t ≤ 2 min { Var K ∗ , Var K π } + O ( d 3.5 H 3.5 β ι 0.5 + d 3.5 H 3.5 β 0.5 ι 1.5 ) \sqrt{V_{\mathrm{est}}}\le\sqrt{2\min\{\operatorname{Var}^{*}_{K},\operatorname{Var}^{\pi}_{K}\}}+O(d^{3.5}H^{3.5}\beta\iota_{0.5}+d^{3.5}H^{3.5}\beta^{0.5}\iota^{1.5}) V est ≤ 2 min { Var K ∗ , Var K π } + O ( d 3.5 H 3.5 β ι 0.5 + d 3.5 H 3.5 β 0.5 ι 1.5 ) from (B.15) of Lemma B.3 and β γ 2 d H ι = 4 d 4 H 4 β ι 3 \beta\gamma^{2}dH\iota=4d^{4}H^{4}\beta\iota^{3} β γ 2 d H ι = 4 d 4 H 4 β ι 3 , and the last inequality substitutes β = O ( d ι 2 ι 0 + ι ι 0 ) \beta=O(\sqrt{d}\iota^{2}\iota_{0}+\iota\iota_{0}) β = O ( d ι 2 ι 0 + ι ι 0 ) (so that β 2 = O ( d ι 2 ι 0 2 + ι 2 ι 0 2 ) \beta^{2}=O(d\iota^{2}\iota_{0}^{2}+\iota^{2}\iota_{0}^{2}) β 2 = O ( d ι 2 ι 0 2 + ι 2 ι 0 2 ) and d 4.5 ι 0.5 0.5 ≤ d 5 ι 0 + d 4 d^{4.5}\iota_{0.5}^{0.5}\le d^{5}\iota_{0}+d^{4} d 4.5 ι 0.5 0.5 ≤ d 5 ι 0 + d 4 ). Finally, the bound O ( d H ι ) O(dH\iota) O ( d H ι ) on the number of value updates follows from Lemma A.1, and the computational complexity O ~ ( d 4 H 3 ∣ A ∣ K ) \widetilde{O}(d^{4}H^{3}|\mathcal{A}|K) O ( d 4 H 3 ∣ A ∣ K ) follows from Remark 5.4.
Trajectory Variance Bounds and Proof of Corollary 5.5
To prove the worst-case and first-order regret consequences in Remark 5.2 and Corollary 5.5, we establish high-probability bounds on the realized trajectory variance Var K π \operatorname{Var}^{\pi}_{K} Var K π using a telescoping variance self-bounding argument (Chen et al., [7 , 8 ] ; Foster et al., [10 ] ).
Lemma C.1 (Trajectory Variance Bounds). Suppose that r h ( s , a ) ≥ 0 r_{h}(s,a)\ge0 r h ( s , a ) ≥ 0 for all ( s , a , h ) ∈ S × A × [ H ] (s,a,h)\in\mathcal{S}\times\mathcal{A}\times[H] ( s , a , h ) ∈ S × A × [ H ] and ∑ h = 1 H r h ( s h , a h ) ≤ B V \sum_{h=1}^{H}r_{h}(s_{h},a_{h})\le B_{V} ∑ h = 1 H r h ( s h , a h ) ≤ B V almost surely for any trajectory. Then for any δ ∈ ( 0 , 1 ) \delta\in(0,1) δ ∈ ( 0 , 1 ) , the following two bounds hold:
Worst-case trajectory variance bound. With probability at least 1 − δ 1-\delta 1 − δ ,2
Var K π ≤ 4 B V 2 K + 16 B V 2 ln ( 2 / δ ) . (69) \operatorname{Var}^{\pi}_{K}\le4B_{V}^{2}K+16B_{V}^{2}\ln(2/\delta). \tag*{(69)} Var K π ≤ 4 B V 2 K + 16 B V 2 ln ( 2/ δ ) . (69) First-order trajectory variance bound. With probability at least 1 − δ 1-\delta 1 − δ ,
Var K π ≤ 6 B V ∑ k = 1 K V 1 π k ( s 1 k ) + 32 B V 2 ln ( 4 / δ ) ≤ 6 B V V t o t a l ∗ + 32 B V 2 ln ( 4 / δ ) . (70) \operatorname{Var}^{\pi}_{K}\le6B_{V}\sum_{k=1}^{K}V_{1}^{\pi^{k}}(s_{1}^{k})+32B_{V}^{2}\ln(4/\delta)\le6B_{V}V_{\mathrm{total}}^{*}+32B_{V}^{2}\ln(4/\delta). \tag*{(70)} Var K π ≤ 6 B V k = 1 ∑ K V 1 π k ( s 1 k ) + 32 B V 2 ln ( 4/ δ ) ≤ 6 B V V total ∗ + 32 B V 2 ln ( 4/ δ ) . (70)
Proof. Under the trajectory return assumption, along any realized trajectory we have 0 ≤ V h π k ( s h k ) ≤ B V 0\le V_{h}^{\pi^{k}}(s_{h}^{k})\le B_{V} 0 ≤ V h π k ( s h k ) ≤ B V and 0 ≤ V h + 1 π k ( s h + 1 k ) ≤ B V 0\le V_{h+1}^{\pi^{k}}(s_{h+1}^{k})\le B_{V} 0 ≤ V h + 1 π k ( s h + 1 k ) ≤ B V almost surely for all ( k , h ) ∈ [ K ] × [ H ] (k,h)\in[K]\times[H] ( k , h ) ∈ [ K ] × [ H ] . Using V H + 1 π k ( s H + 1 k ) = 0 V_{H+1}^{\pi^{k}}(s_{H+1}^{k})=0 V H + 1 π k ( s H + 1 k ) = 0 , we expand Var K π \operatorname{Var}^{\pi}_{K} Var K π via a telescoping sum over h = 1 , … , H h=1,\ldots,H h = 1 , … , H :
Var K π = ∑ k = 1 K ∑ h = 1 H ( [ P h ( V h + 1 π k ) 2 ] ( s h k , a h k ) − ( [ P h V h + 1 π k ] ( s h k , a h k ) ) 2 ) \operatorname{Var}^{\pi}_{K}=\sum_{k=1}^{K}\sum_{h=1}^{H}\left(\left[\mathbb{P}_{h} \left(V_{h+1}^{\pi^{k}}\right)^{2}\right](s_{h}^{k},a_{h}^{k})-\left(\left[\mathbb{P}_{h}V_{h+1}^{\pi^{k}}\right](s_{h}^{k},a_{h}^{k})\right)^{2}\right) Var K π = k = 1 ∑ K h = 1 ∑ H ( [ P h ( V h + 1 π k ) 2 ] ( s h k , a h k ) − ( [ P h V h + 1 π k ] ( s h k , a h k ) ) 2 )
= ∑ k = 1 K ∑ h = 1 H ( [ P h ( V h + 1 π k ) 2 ] ( s h k , a h k ) − ( V h + 1 π k ( s h + 1 k ) ) 2 ) ⏟ = : T 1 + ∑ k = 1 K ∑ h = 1 H ( ( V h π k ( s h k ) ) 2 − ( [ P h V h + 1 π k ] ( s h k , a h k ) ) 2 ) ⏟ = : T 2 − ∑ k = 1 K ( V 1 π k ( s 1 k ) ) 2 . (71) \begin{aligned}
&= \underbrace{\sum_{k=1}^{K}\sum_{h=1}^{H}\left(\left[\mathbb{P}_{h}\left(V_{h+1}^{\pi^{k}}\right)^{2}\right]\left(s_{h}^{k},a_{h}^{k}\right)-\left(V_{h+1}^{\pi^{k}}\left(s_{h+1}^{k}\right)\right)^{2}\right)}_{=:T_{1}}\\
&\quad+\underbrace{\sum_{k=1}^{K}\sum_{h=1}^{H}\left(\left(V_{h}^{\pi^{k}}\left(s_{h}^{k}\right)\right)^{2}-\left(\left[\mathbb{P}_{h}V_{h+1}^{\pi^{k}}\right]\left(s_{h}^{k},a_{h}^{k}\right)\right)^{2}\right)}_{=:T_{2}}-\sum_{k=1}^{K}\left(V_{1}^{\pi^{k}}\left(s_{1}^{k}\right)\right)^{2}. \tag*{(71)}
\end{aligned} = =: T 1 k = 1 ∑ K h = 1 ∑ H ( [ P h ( V h + 1 π k ) 2 ] ( s h k , a h k ) − ( V h + 1 π k ( s h + 1 k ) ) 2 ) + =: T 2 k = 1 ∑ K h = 1 ∑ H ( ( V h π k ( s h k ) ) 2 − ( [ P h V h + 1 π k ] ( s h k , a h k ) ) 2 ) − k = 1 ∑ K ( V 1 π k ( s 1 k ) ) 2 . (71)
For the martingale difference sum T 1 T_{1} T 1 , let X k , h : = [ P h ( V h + 1 π k ) 2 ] ( s h k , a h k ) − ( V h + 1 π k ( s h + 1 k ) ) 2 X_{k,h}:=\left[\mathbb{P}_{h}\left(V_{h+1}^{\pi^{k}}\right)^{2}\right]\left(s_{h}^{k},a_{h}^{k}\right)-\left(V_{h+1}^{\pi^{k}}\left(s_{h+1}^{k}\right)\right)^{2} X k , h := [ P h ( V h + 1 π k ) 2 ] ( s h k , a h k ) − ( V h + 1 π k ( s h + 1 k ) ) 2 . Since V h + 1 π k ( s h + 1 k ) ∈ [ 0 , B V ] V_{h+1}^{\pi^{k}}\left(s_{h+1}^{k}\right)\in[0,B_{V}] V h + 1 π k ( s h + 1 k ) ∈ [ 0 , B V ] almost surely given F k , h \mathcal{F}_{k,h} F k , h , we have ∣ X k , h ∣ ≤ B V 2 \lvert X_{k,h}\rvert\le B_{V}^{2} ∣ X k , h ∣ ≤ B V 2 almost surely, E [ X k , h ∣ F k , h ] = 0 \mathbb{E}[X_{k,h}\mid\mathcal{F}_{k,h}]=0 E [ X k , h ∣ F k , h ] = 0 , and
E [ X k , h 2 ∣ F k , h ] = [ V h ( V h + 1 π k ) 2 ] ( s h k , a h k ) ≤ 4 B V 2 [ V h V h + 1 π k ] ( s h k , a h k ) . \mathbb{E}\left[X_{k,h}^{2}\mid\mathcal{F}_{k,h}\right]=\left[\mathbb{V}_{h}\left(V_{h+1}^{\pi^{k}}\right)^{2}\right]\left(s_{h}^{k},a_{h}^{k}\right)\le4B_{V}^{2}\left[\mathbb{V}_{h}V_{h+1}^{\pi^{k}}\right]\left(s_{h}^{k},a_{h}^{k}\right). E [ X k , h 2 ∣ F k , h ] = [ V h ( V h + 1 π k ) 2 ] ( s h k , a h k ) ≤ 4 B V 2 [ V h V h + 1 π k ] ( s h k , a h k ) .
where the inequality follows from Lemma E.4 applied to V h + 1 π k ( s h + 1 k ) ∈ [ 0 , B V ] V_{h+1}^{\pi^{k}}\left(s_{h+1}^{k}\right)\in[0,B_{V}] V h + 1 π k ( s h + 1 k ) ∈ [ 0 , B V ] conditioned on F k , h \mathcal{F}_{k,h} F k , h . Applying Lemma E.1 with η = 1 / ( 8 B V 2 ) ∈ ( 0 , 1 / B V 2 ] \eta=1/(8B_{V}^{2})\in(0,1/B_{V}^{2}] η = 1/ ( 8 B V 2 ) ∈ ( 0 , 1/ B V 2 ] and failure probability δ / 2 \delta/2 δ /2 yields that with probability at least 1 − δ / 2 1-\delta/2 1 − δ /2 ,
T 1 ≤ η ∑ k = 1 K ∑ h = 1 H 4 B V 2 [ V h V h + 1 π k ] ( s h k , a h k ) + ln ( 2 / δ ) η = 1 2 Var K π + 8 B V 2 ln ( 2 / δ ) . (72) T_{1}\le\eta\sum_{k=1}^{K}\sum_{h=1}^{H}4B_{V}^{2}\left[\mathbb{V}_{h}V_{h+1}^{\pi^{k}}\right]\left(s_{h}^{k},a_{h}^{k}\right)+\frac{\ln(2/\delta)}{\eta}=\frac{1}{2}\operatorname{Var}_{K}^{\pi}+8B_{V}^{2}\ln(2/\delta). \tag*{(72)} T 1 ≤ η k = 1 ∑ K h = 1 ∑ H 4 B V 2 [ V h V h + 1 π k ] ( s h k , a h k ) + η ln ( 2/ δ ) = 2 1 Var K π + 8 B V 2 ln ( 2/ δ ) . (72)
For the Bellman-difference sum T 2 T_{2} T 2 , we have
T 2 = ∑ k = 1 K ∑ h = 1 H ( V h π k ( s h k ) + [ P h V h + 1 π k ] ( s h k , a h k ) ) r h ( s h k , a h k ) ≤ 2 B V ∑ k = 1 K ∑ h = 1 H r h ( s h k , a h k ) . (73) T_{2}=\sum_{k=1}^{K}\sum_{h=1}^{H}\left(V_{h}^{\pi^{k}}\left(s_{h}^{k}\right)+\left[\mathbb{P}_{h}V_{h+1}^{\pi^{k}}\right]\left(s_{h}^{k},a_{h}^{k}\right)\right)r_{h}\left(s_{h}^{k},a_{h}^{k}\right)\le2B_{V}\sum_{k=1}^{K}\sum_{h=1}^{H}r_{h}\left(s_{h}^{k},a_{h}^{k}\right). \tag*{(73)} T 2 = k = 1 ∑ K h = 1 ∑ H ( V h π k ( s h k ) + [ P h V h + 1 π k ] ( s h k , a h k ) ) r h ( s h k , a h k ) ≤ 2 B V k = 1 ∑ K h = 1 ∑ H r h ( s h k , a h k ) . (73)
where the equality uses the Bellman equation V h π k ( s h k ) = r h ( s h k , a h k ) + [ P h V h + 1 π k ] ( s h k , a h k ) V_{h}^{\pi^{k}}\left(s_{h}^{k}\right)=r_{h}\left(s_{h}^{k},a_{h}^{k}\right)+\left[\mathbb{P}_{h}V_{h+1}^{\pi^{k}}\right]\left(s_{h}^{k},a_{h}^{k}\right) V h π k ( s h k ) = r h ( s h k , a h k ) + [ P h V h + 1 π k ] ( s h k , a h k ) , and the inequality holds since V h π k ( s h k ) ≤ B V V_{h}^{\pi^{k}}\left(s_{h}^{k}\right)\le B_{V} V h π k ( s h k ) ≤ B V and [ P h V h + 1 π k ] ( s h k , a h k ) ≤ B V \left[\mathbb{P}_{h}V_{h+1}^{\pi^{k}}\right]\left(s_{h}^{k},a_{h}^{k}\right)\le B_{V} [ P h V h + 1 π k ] ( s h k , a h k ) ≤ B V almost surely. Because ∑ h = 1 H r h ( s h k , a h k ) ≤ B V \sum_{h=1}^{H}r_{h}\left(s_{h}^{k},a_{h}^{k}\right)\le B_{V} ∑ h = 1 H r h ( s h k , a h k ) ≤ B V almost surely for every k ∈ [ K ] k\in[K] k ∈ [ K ] , {(citep)}(C.5) deterministically gives T 2 ≤ 2 B V 2 K T_{2}\le2B_{V}^{2}K T 2 ≤ 2 B V 2 K . Substituting {(citep)}(C.4) and T 2 ≤ 2 B V 2 K T_{2}\le2B_{V}^{2}K T 2 ≤ 2 B V 2 K into {(citep)}(C.3) and rearranging proves {(citep)}(C.1).
To prove {(citep)}(C.2), let R k : = ∑ h = 1 H r h ( s h k , a h k ) ∈ [ 0 , B V ] R_{k}:=\sum_{h=1}^{H}r_{h}\left(s_{h}^{k},a_{h}^{k}\right)\in[0,B_{V}] R k := ∑ h = 1 H r h ( s h k , a h k ) ∈ [ 0 , B V ] denote the realized return in episode k k k . Since R k R_{k} R k is F k + 1 , 1 \mathcal{F}_{k+1,1} F k + 1 , 1 -measurable with E [ R k ∣ F k , 1 ] = V 1 π k ( s 1 k ) ≤ V 1 ∗ ( s 1 k ) \mathbb{E}[R_{k}\mid\mathcal{F}_{k,1}]=V_{1}^{\pi^{k}}\left(s_{1}^{k}\right)\le V_{1}^{*}\left(s_{1}^{k}\right) E [ R k ∣ F k , 1 ] = V 1 π k ( s 1 k ) ≤ V 1 ∗ ( s 1 k ) , applying Corollary E.2 to { R k } k = 1 K \{R_{k}\}_{k=1}^{K} { R k } k = 1 K with range B V B_{V} B V and failure probability δ / 2 \delta/2 δ /2 gives, with probability at least 1 − δ / 2 1-\delta/2 1 − δ /2 ,
∑ k = 1 K ∑ h = 1 H r h ( s h k , a h k ) = ∑ k = 1 K R k ≤ 3 2 ∑ k = 1 K V 1 π k ( s 1 k ) + 4 B V ln ( 4 / δ ) . \sum_{k=1}^{K}\sum_{h=1}^{H}r_{h}\left(s_{h}^{k},a_{h}^{k}\right)=\sum_{k=1}^{K}R_{k}\le\frac{3}{2}\sum_{k=1}^{K}V_{1}^{\pi^{k}}\left(s_{1}^{k}\right)+4B_{V}\ln(4/\delta). k = 1 ∑ K h = 1 ∑ H r h ( s h k , a h k ) = k = 1 ∑ K R k ≤ 2 3 k = 1 ∑ K V 1 π k ( s 1 k ) + 4 B V ln ( 4/ δ ) .
Substituting this bound into {(citep)}(C.5), combining with {(citep)}(C.4) via a union bound, and rearranging {(citep)}(C.3) establishes {(citep)}(C.2).
Proof of Corollary 5.5. In the setting of Theorem 5.1, we have B V = H B_{V}=H B V = H , so {(citep)}(C.2) of Lemma C.1 holds with probability at least 1 − δ 1-\delta 1 − δ :
min { Var K ∗ , Var K π } ≤ Var K π ≤ 6 H V t o t a l ∗ + 32 H 2 ln ( 4 / δ ) . \min\{\operatorname{Var}_{K}^{*},\operatorname{Var}_{K}^{\pi}\}\le\operatorname{Var}_{K}^{\pi}\le6H V_{\mathrm{total}}^{*}+32H^{2}\ln(4/\delta). min { Var K ∗ , Var K π } ≤ Var K π ≤ 6 H V total ∗ + 32 H 2 ln ( 4/ δ ) .
Substituting this upper bound into {(citep)}(5.1) of Theorem 5.1 and taking a union bound completes the proof.
Comparison with Time-Homogeneous Horizon-Free RL
While the main text focuses on episodic time-inhomogeneous MDPs with per-step reward r h ∈ [ 0 , 1 ] r_h \in[0,1] r h ∈ [ 0 , 1 ] (see Table 1 ), a parallel line of research studies time-homogeneous episodic MDPs (P 1 = ⋯ = P H = P P_1=\cdots=P_H=P P 1 = ⋯ = P H = P and r 1 = ⋯ = r H = r r_1=\cdots=r_H=r r 1 = ⋯ = r H = r ) under the bounded total reward normalization ∑ h = 1 H r ( s h , a h ) ≤ 1 \sum_{h=1}^{H}r(s_h,a_h)\le1 ∑ h = 1 H r ( s h , a h ) ≤ 1 , aiming to achieve horizon-free regret bounds with no polynomial dependence on H H H . Table 2 summarizes existing results for time-homogeneous linear mixture MDPs and linear MDPs in this regime.
As shown in Table 2 , in time-homogeneous linear mixture MDPs, the transition kernel P ( s ′ ∣ s , a ) = ⟨ ϕ ( s ′ ∣ s , a ) , θ ∗ ⟩ \mathbb{P}(s' \mid s,a)=\langle\phi(s'\mid s,a),\theta^{*}\rangle P ( s ′ ∣ s , a ) = ⟨ ϕ ( s ′ ∣ s , a ) , θ ∗ ⟩ is governed by a single d d d -dimensional parameter vector θ ∗ \theta^{*} θ ∗ shared across all stages h ∈ [ H ] h\in[H] h ∈ [ H ] , allowing samples from all H H H stages to be pooled into a single regression problem; exploiting this property, Zhou and Gu [40 ] (HF-UCRL-VTR+) and Zhao et al. [39 ] (UCRL-AVE) achieved computationally efficient horizon-free worst-case and variance-dependent regret bounds, respectively. In time-homogeneous linear MDPs, by contrast, even though the signed transition measures θ ( ⋅ ) \theta(\cdot) θ ( ⋅ ) are stage-independent, the optimal value functions { V h + 1 ∗ } h = 1 H \{V^{*}_{h+1}\}_{h=1}^{H} { V h + 1 ∗ } h = 1 H remain time-inhomogeneous across stages, inducing H H H distinct Bellman backup parameter vectors w ( V h + 1 ∗ ) = ∫ S V h + 1 ∗ ( s ′ ) d θ ( s ′ ) w(V^{*}_{h+1})=\int_{\mathcal{S}}V^{*}_{h+1}(s')\mathrm{d}\theta(s') w ( V h + 1 ∗ ) = ∫ S V h + 1 ∗ ( s ′ ) d θ ( s ′ ) in general. Consequently, pooling state-action features across stages faces a covariance mismatch across different stage variances [ V V h + 1 ∗ ] [\mathbb{V}V^{*}_{h+1}] [ V V h + 1 ∗ ] . Recently, Zhang et al. [36 ] overcame this statistical barrier via a computationally inefficient model-elimination algorithm that maintains function-indexed covariance matrices over an ϵ \epsilon ϵ -net of value functions and groups stages by bounding the total variation of { V h ∗ } h = 1 H \{V^{*}_{h}\}_{h=1}^{H} { V h ∗ } h = 1 H , establishing a worst-case horizon-free regret bound of O ~ ( d 5.5 K + d 6.5 ) \widetilde{O}(d^{5.5}\sqrt{K}+d^{6.5}) O ( d 5.5 K + d 6.5 ) . Whether one can achieve horizon-free variance-dependent regret in time-homogeneous linear MDPs, and whether horizon-free regret can be attained by a computationally efficient algorithm, remain open problems.
Auxiliary Lemmas
In this appendix, we collect the concentration inequalities and technical facts from prior work used throughout our proofs.
Martingale Concentration Inequalities
We first recall the time-uniform Freedman inequality for bounded martingale difference sequences [11 , 6 ] , stated below in the form of Foster et al. [10 ] .
Lemma E.1 (Time-Uniform Freedman Inequality, Foster et al. [10 ] , Lemma D.2). Let { M t } t = 1 T \{M_t\}_{t=1}^{T} { M t } t = 1 T be a real-valued martingale difference sequence adapted to a filtration { G t } t = 0 T \{\mathcal{G}_t\}_{t=0}^{T} { G t } t = 0 T such that ∣ M t ∣ ≤ B |M_t| \le B ∣ M t ∣ ≤ B almost surely for all t ∈ [ T ] t \in[T] t ∈ [ T ] . Then for any fixed η ∈ ( 0 , 1 / B ] \eta\in(0,1/B] η ∈ ( 0 , 1/ B ] and any δ ′ ∈ ( 0 , 1 ) \delta' \in(0,1) δ ′ ∈ ( 0 , 1 ) , with probability at least 1 − δ ′ 1-\delta' 1 − δ ′ , for all T ′ ∈ [ T ] T' \in[T] T ′ ∈ [ T ] :
∑ t = 1 T ′ M t ≤ η ∑ t = 1 T ′ E [ M t 2 ∣ G t − 1 ] + ln ( 1 / δ ′ ) η . \sum_{t=1}^{T'} M_t \le\eta\sum_{t=1}^{T'} \mathbb{E}[M_t^2 \mid\mathcal{G}_{t-1}] + \frac{\ln(1/\delta')}{\eta}. t = 1 ∑ T ′ M t ≤ η t = 1 ∑ T ′ E [ M t 2 ∣ G t − 1 ] + η ln ( 1/ δ ′ ) .
Corollary E.2 (Multiplicative Concentration, Foster et al. [10 ] , Lemma D.3). Let { X t } t = 1 T \{X_t\}_{t=1}^{T} { X t } t = 1 T be a sequence of random variables adapted to a filtration { G t } t = 0 T \{\mathcal{G}_t\}_{t=0}^{T} { G t } t = 0 T such that 0 ≤ X t ≤ B 0 \le X_t \le B 0 ≤ X t ≤ B almost surely for all t ∈ [ T ] t \in[T] t ∈ [ T ] . Then for any δ ′ ∈ ( 0 , 1 ) \delta' \in(0,1) δ ′ ∈ ( 0 , 1 ) , with probability at least 1 − δ ′ 1-\delta' 1 − δ ′ , for all T ′ ∈ [ T ] T' \in[T] T ′ ∈ [ T ] :
∑ t = 1 T ′ X t ≤ 3 2 ∑ t = 1 T ′ E [ X t ∣ G t − 1 ] + 4 B ln ( 2 / δ ′ ) . \sum_{t=1}^{T'} X_t \le\frac{3}{2}\sum_{t=1}^{T'} \mathbb{E}[X_t \mid\mathcal{G}_{t-1}] + 4B\ln(2/\delta'). t = 1 ∑ T ′ X t ≤ 2 3 t = 1 ∑ T ′ E [ X t ∣ G t − 1 ] + 4 B ln ( 2/ δ ′ ) .
Lemma E.3 (Variance-Aware Self-Normalized Concentration Inequality, Zhou and Gu 2022, Theorem 4.3). Let { G i } i = 1 ∞ \{\mathcal{G}_i\}_{i=1}^{\infty} { G i } i = 1 ∞ be a filtration, and let { x i , η i } i ≥ 1 \{x_i,\eta_i\}_{i\ge1} { x i , η i } i ≥ 1 be a stochastic process such that x i ∈ R d x_i \in\mathbb{R}^d x i ∈ R d is G i \mathcal{G}_i G i -measurable and η i ∈ R \eta_i \in\mathbb{R} η i ∈ R is G i + 1 \mathcal{G}_{i+1} G i + 1 -measurable. Suppose that for constants σ , R , L x > 0 \sigma, R, L_x > 0 σ , R , L x > 0 ,
E [ η i ∣ G i ] = 0 , E [ η i 2 ∣ G i ] ≤ σ 2 , ∣ η i ∣ ≤ R , ∥ x i ∥ 2 ≤ L x . \mathbb{E}[\eta_i \mid\mathcal{G}_i] = 0,\qquad\mathbb{E}[\eta_i^2 \mid\mathcal{G}_i] \le\sigma^2,\qquad|\eta_i| \le R,\qquad\|x_i\|_2 \le L_x. E [ η i ∣ G i ] = 0 , E [ η i 2 ∣ G i ] ≤ σ 2 , ∣ η i ∣ ≤ R , ∥ x i ∥ 2 ≤ L x . Let Z k = λ I + ∑ i = 1 k − 1 x i x i T Z_k = \lambda I + \sum_{i=1}^{k-1} x_i x_i^\mathsf{T} Z k = λ I + ∑ i = 1 k − 1 x i x i T for λ > 0 \lambda> 0 λ > 0 . Then for any δ ′ ∈ ( 0 , 1 ) \delta' \in(0,1) δ ′ ∈ ( 0 , 1 ) and any discretization parameter ϵ c ∈ ( 0 , R ] \epsilon_c \in(0,R] ϵ c ∈ ( 0 , R ] , with probability at least 1 − δ ′ 1-\delta' 1 − δ ′ , for all k ∈ [ K ] k \in[K] k ∈ [ K ] :
∥ ∑ i = 1 k − 1 x i η i ∥ Z k − 1 ≤ 12 σ 2 d ln ( 1 + K L x 2 d λ ) ι c ( δ ′ ) + 24 ι c ( δ ′ ) max 1 ≤ i ≤ k − 1 ∣ η i ∣ min { 1 , ∥ x i ∥ Z i − 1 − 1 } + 6 ι c ( δ ′ ) ϵ c , \left\|\sum_{i=1}^{k-1} x_i\eta_i\right\|_{Z_k^{-1}} \le12\sqrt{\sigma^2 d\ln\left(1+\frac{KL_x^2}{d\lambda}\right)\iota_c(\delta')} + 24\iota_c(\delta')\max_{1\le i\le k-1}|\eta_i|\min\left\{1,\|x_i\|_{Z_{i-1}^{-1}}\right\} + 6\iota_c(\delta')\epsilon_c, i = 1 ∑ k − 1 x i η i Z k − 1 ≤ 12 σ 2 d ln ( 1 + d λ K L x 2 ) ι c ( δ ′ ) + 24 ι c ( δ ′ ) 1 ≤ i ≤ k − 1 max ∣ η i ∣ min { 1 , ∥ x i ∥ Z i − 1 − 1 } + 6 ι c ( δ ′ ) ϵ c , where ι c ( δ ′ ) : = ln ( 32 ( ln ( R / ϵ c ) + 1 ) K 2 / δ ′ ) \iota_c(\delta') := \ln(32(\ln(R/\epsilon_c)+1)K^2/\delta') ι c ( δ ′ ) := ln ( 32 ( ln ( R / ϵ c ) + 1 ) K 2 / δ ′ ) .
Miscellaneous Bounds
We next record a variance bound for products of bounded random variables from Chen et al. [7 ] (see also Chen et al. [8 ] ).
Lemma E.4 (Chen et al. [7 ] , Lemma 30). For any two bounded real-valued random variables X X X and Y Y Y , we have
Var ( X Y ) ≤ 2 Var ( X ) ∥ Y ∥ ∞ 2 + 2 ( E [ X ] ) 2 Var ( Y ) . \operatorname{Var}(XY) \le2\operatorname{Var}(X)\|Y\|_{\infty}^{2} + 2(\mathbb{E}[X])^{2}\operatorname{Var}(Y). Var ( X Y ) ≤ 2 Var ( X ) ∥ Y ∥ ∞ 2 + 2 ( E [ X ] ) 2 Var ( Y ) . In particular, taking Y = X Y=X Y = X with ∥ X ∥ ∞ ≤ B \|X\|_{\infty} \le B ∥ X ∥ ∞ ≤ B gives Var ( X 2 ) ≤ 4 B 2 Var ( X ) \operatorname{Var}(X^2) \le4B^2\operatorname{Var}(X) Var ( X 2 ) ≤ 4 B 2 Var ( X ) .
Applying Lemma E.4 [see also 10 Lemma G.8] to the conditional distribution P h ( ⋅ ∣ s , a ) \mathbb{P}_h(\cdot\mid s,a) P h ( ⋅ ∣ s , a ) immediately yields the following inequality for squared value functions.
Corollary E.5 (Conditional Variance Bound for Squared Value Functions). For any measurable function V : S → [ 0 , B V ] V:\mathcal{S}\to[0,B_V] V : S → [ 0 , B V ] and any state-action-stage triplet ( s , a , h ) ∈ S × A × [ H ] (s,a,h)\in\mathcal{S}\times\mathcal{A}\times[H] ( s , a , h ) ∈ S × A × [ H ] , we have
[ V h V 2 ] ( s , a ) ≤ 4 B V 2 [ V h V ] ( s , a ) . [\mathbb{V}_hV^2](s,a) \le4B_V^2[\mathbb{V}_hV](s,a). [ V h V 2 ] ( s , a ) ≤ 4 B V 2 [ V h V ] ( s , a ) .
Proof. Fix ( s , a , h ) (s,a,h) ( s , a , h ) and let X = V ( s ′ ) ∈ [ 0 , B V ] X=V(s')\in[0,B_V] X = V ( s ′ ) ∈ [ 0 , B V ] where s ′ ∼ P h ( ⋅ ∣ s , a ) s'\sim\mathbb{P}_h(\cdot\mid s,a) s ′ ∼ P h ( ⋅ ∣ s , a ) . Since ∥ X ∥ ∞ ≤ B V \lVert X\rVert_\infty\le B_V ∥ X ∥ ∞ ≤ B V , Lemma E.4 with Y = X Y=X Y = X yields [ V h V 2 ] ( s , a ) = Var ( X 2 ) ≤ 4 B V 2 Var ( X ) = 4 B V 2 [ V h V ] ( s , a ) [\mathbb{V}_hV^2](s,a)=\operatorname{Var}(X^2)\le4B_V^2\operatorname{Var}(X)=4B_V^2[\mathbb{V}_hV](s,a) [ V h V 2 ] ( s , a ) = Var ( X 2 ) ≤ 4 B V 2 Var ( X ) = 4 B V 2 [ V h V ] ( s , a ) .
Lemma E.6 (Elliptical Potential Lemma, [1 Lemma 11] ). Let { x k } k = 1 K ⊂ R d \{x_k\}_{k=1}^{K}\subset\mathbb{R}^d { x k } k = 1 K ⊂ R d be a sequence of vectors satisfying ∥ x k ∥ 2 ≤ L x \lVert x_k\rVert_2\le L_x ∥ x k ∥ 2 ≤ L x for all k ∈ [ K ] k\in[K] k ∈ [ K ] , and define Z k = λ I + ∑ i = 1 k − 1 x i x i ⊤ Z_k=\lambda I+\sum_{i=1}^{k-1}x_ix_i^\top Z k = λ I + ∑ i = 1 k − 1 x i x i ⊤ for λ > 0 \lambda>0 λ > 0 . Then
∑ k = 1 K min { 1 , ∥ x k ∥ Z k − 1 2 } ≤ 2 d ln ( 1 + K L x 2 d λ ) . \sum_{k=1}^{K}\min\left\{1,\lVert x_k\rVert_{Z_k^{-1}}^2\right\}\le2d\ln\left(1+\frac{KL_x^2}{d\lambda}\right). k = 1 ∑ K min { 1 , ∥ x k ∥ Z k − 1 2 } ≤ 2 d ln ( 1 + d λ K L x 2 ) .
Lemma E.7 (Determinant Ratio Inequality, [1 Lemma 12] ). Suppose A , B ∈ R d × d \mathbf{A},\mathbf{B}\in\mathbb{R}^{d\times d} A , B ∈ R d × d are two positive definite matrices satisfying A ⪰ B \mathbf{A}\succeq\mathbf{B} A ⪰ B . Then for any x ∈ R d \mathbf{x}\in\mathbb{R}^d x ∈ R d , we have
∥ x ∥ A ≤ ∥ x ∥ B ⋅ det ( A ) / det ( B ) \lVert\mathbf{x}\rVert_{\mathbf{A}}\le\lVert\mathbf{x}\rVert_{\mathbf{B}}\cdot\sqrt{\det(\mathbf{A})/\det(\mathbf{B})} ∥ x ∥ A ≤ ∥ x ∥ B ⋅ det ( A ) / det ( B ) .
Lemma E.8 (Covering Number of Euclidean Balls, [20 Lemma D.5] ). For any ϵ > 0 \epsilon>0 ϵ > 0 and R > 0 R>0 R > 0 , the ϵ \epsilon ϵ -covering number of the Euclidean ball { x ∈ R d : ∥ x ∥ 2 ≤ R } \{\mathbf{x}\in\mathbb{R}^d:\lVert\mathbf{x}\rVert_2\le R\} { x ∈ R d : ∥ x ∥ 2 ≤ R } under the Euclidean norm ∥ ⋅ ∥ 2 \lVert\cdot\rVert_2 ∥ ⋅ ∥ 2 is upper-bounded by ( 1 + 2 R / ϵ ) d (1+2R/\epsilon)^d ( 1 + 2 R / ϵ ) d .