Introduction
Reinforcement learning (RL) (Sutton and Barto, 2018; Bertsekas, 2019), commonly formulated as sequential decision-making and learning in a Markov decision process (MDP), has garnered immense attention in recent years due to its striking empirical success across a wide range of practical domains. By enabling agents to collect data and learn through repeated, trial-and-error interaction with their environments, RL has led to major breakthroughs in applications such as competitive gaming, robotics and control, autonomous driving, e-commerce, power systems, and healthcare, often achieving state-of-the-art performance in complex decision-making tasks (Silver et al., 2017; Mnih et al., 2015; Tang et al., 2025; Kiran et al., 2021; Hu et al., 2018; Bastani et al., 2021; Chen et al., 2022c; Yu et al., 2021; Bai et al., 2025). Beyond these traditional application areas, RL has recently emerged as a cornerstone for fine-tuning and aligning large language models (LLMs). In particular, post-training techniques such as Reinforcement Learning from Human Feedback (RLHF) and Reinforcement Learning with Verifiable Rewards (RLVR) (Guo et al., 2025; Jaech et al., 2024) have proven highly effective in enhancing instruction following, safety, and sophisticated reasoning capabilities, among other things, thereby positioning RL as a key enabler in the rapid advancement of modern generative artificial intelligence (AI).
A central objective of RL is to learn, from a collection of noisy data samples, a decision-making strategy (commonly referred to as a policy in RL) that approximately maximizes long-term rewards in an environment (often quantified through a value function ), without direct access to an accurate description of the environment itself. In many modern applications, the state space (i.e., the collection of possible situations an agent may encounter in the environment), the action space (i.e., the set of actions available to the agent at each decision point), and the planning horizon can all be extraordinarily large. As a result, data collection must often be carried out at a substantial scale, which may be prohibitively expensive, time-consuming, or even high-stakes, as encountered in domains such as clinical trials, online advertising, and autonomous systems. These challenges are particularly pronounced in the “sample-starved” regime, where the intrinsic complexity of the environment vastly exceeds the amount of available data, making accurate model estimation infeasible.1 Consequently, a fundamental goal of modern RL theory is to understand how reliable learning can be achieved in such data-limited scenarios. Accomplishing this goal requires developing a precise quantitative understanding of the trade-off between statistical accuracy and sample complexity, as well as designing computationally efficient algorithms that provably attain optimal or near-optimal performance.
Statistical framework and scope
To systematically address the data-efficiency challenges outlined above, this monograph develops a coherent and rigorous statistical framework that accommodates a diverse range of settings. To help orient the reader, we briefly describe three key pillars underlying the topics studied throughout this monograph: the sampling mechanisms, the core algorithmic paradigms, and the performance metrics used to evaluate learning performance.
Data collection mechanisms. The statistical challenges of RL depend fundamentally on how data are collected and made available to the agent. We emphasize three canonical settings that are widely studied in the literature.
RL with a generative model. When equipped with access to a generative model or simulator, the agent can query the environment in a fully controlled manner. More specifically, the agent may sample transitions from arbitrary state-action pairs at will, effectively eliminating the need for exploration and isolating the statistical challenges inherent to policy optimization and value estimation. See Chapters 3 and 4.
Online RL. In online RL, the agent interacts with the environment sequentially in real time and must adaptively determine its actions based on past observations. Since data can only be gathered through the agent’s own interactions with the environment, one must balance exploration (i.e., acquiring new information about the unknown environment) and exploitation (i.e., leveraging current knowledge of the environment to improve rewards). This fundamental tension leads to the classical exploration-exploitation trade-off, which lies at the heart of online RL. See Chapter 5.
Offline RL. In offline RL, the agent is given access only to a static dataset of historical samples — typically logged by one or more (possibly unknown) behavior policies — and needs to learn without any further interaction with the environment. As a result, the agent faces fundamental challenges arising from issues such as limited data coverage and distributional mismatch between the available data and the target policy to be learned. See Chapter 6.
Algorithmic paradigms. Broadly speaking, there are at least two principal algorithmic paradigms in RL: model-based approaches and model-free approaches .
Model-based algorithms typically decouple model estimation from policy learning. Namely, one first leverages the available data samples to estimate the unknown environment model and subsequently performs policy optimization by treating the learned model as if it were accurate and applying classical dynamic programming techniques dating back to Richard Bellman (Bellman, (Bellman, 1952; Bertsekas, 2017)). A notable advantage of model-based methods is
their flexibility: once a model has been learned, it can often be reused to solve new downstream tasks without requiring additional interaction with the environment or revisiting the original data samples.
In contrast, model-free algorithms seek to learn the optimal value function or policy directly from data, bypassing explicit estimation of the environment’s dynamics and rewards. These algorithms are particularly appealing when constructing or storing an environment model is difficult. Model-free methods are also often more memory-efficient because they avoid storing an explicit, potentially large-dimensional representation of the full model.
Characterizing the sample efficiency of model-based and model-free approaches is a primary theme of this monograph. As a preview, despite their higher memory and computational costs, model-based algorithms have been shown to attain statistically optimal performance across all three canonical sampling mechanisms outlined above, even in the most sample-starved regimes. By contrast, many classical model-free methods, including standard variants of Q-learning, fail to attain optimal sample complexity in some regimes. Whether model-free methods can achieve statistical optimality across the full spectrum of problem regimes without sacrificing their memory advantages remains an important open question.
Evaluation metrics. In this monograph, we evaluate the performance of RL algorithms primarily through the lens of sample complexity , defined within the classical Probably Approximately Correct (PAC) learning framework as the minimum number of samples required to guarantee, with high probability, that the statistical error falls below a prescribed threshold. Depending on the problem goal, the notion of statistical error can be tailored to the specific object of interest. For value estimation, we assess the accuracy of the estimated value function, often measured under the ℓ ∞ \ell_\infty ℓ ∞ norm to ensure uniform precision across all initial states. For policy learning, we instead measure the performance gap between the value function induced by the learned policy and that induced by the target policy. More broadly, such sample complexity guarantees provide a precise, quantitative characterization of how statistical accuracy improves as additional data become available.
Another performance metric widely used in online settings is (cumulative) regret , which measures the gap between the cumulative reward that could have been achieved by following an optimal policy and the cumulative reward actually obtained by the learning algorithm over time. Under suitable conditions, regret guarantees can be converted into PAC-style sample complexity guarantees, a connection that we will elaborate on in Chapter § 8.1 .
Finally, we also briefly discuss the task of uncertainty quantification (see Chapter 7.4), with the goal of assessing the uncertainty associated with estimated quantities. A common objective is to construct confidence intervals that are both statistically valid and as tight as possible.
Towards non-asymptotic statistical foundations
Despite decades of extensive research, the statistical foundations of RL in finite-sample and finite-time regimes — which carry high operational value in modern practice — remained significantly underdeveloped until the past one to two decades. Early theoretical analyses of core RL algorithms — such as temporal difference learning and Q-learning — were conducted primarily in the asymptotic regime, where the number of samples or iterations tends to infinity while all other problem parameters are held fixed. Although this line of work provided important insights into the convergence behavior of RL algorithms, it offers limited guidance on their data efficiency in modern large-scale applications. As discussed previously, many contemporary RL problems have to deal with enormous state-action spaces and long planning horizons. Treating these problem parameters as fixed constants can therefore fail to capture the statistical bottlenecks in high dimensions.
To overcome these limitations, the past several years have witnessed a flurry of activity aimed at strengthening finite-sample, finite-time theory for RL. These advances draw heavily on ideas from modern development in high-dimensional statistics, high-dimensional probability, information theory, and statistical learning theory. Yet, despite this shift toward non-asymptotic analysis, a complete understanding of the fundamental sample complexity limits remained largely elusive, even for the three canonical data collection mechanisms introduced above. As we will discuss in subsequent chapters, much of the prior literature was, at best, only able to establish sharp or optimal statistical guarantees when the available sample size exceeded an enormous threshold, leaving the sample-starved regime poorly understood until fairly recently. Indeed, even today, sharp and complete statistical characterizations of the performance of RL algorithms are available only in relatively basic settings, while many important problems and settings remain far from fully resolved.
Notably, RL in high-dimensional, sample-limited settings poses significant technical hurdles that call for both refined analysis techniques and new algorithmic ideas. In some sense, the paradigm shift from asymptotic to non-asymptotic RL theory is reminiscent of the rise of high-dimensional statistics in recent decades, where classical large-sample theory proved insufficient and ultimately motivated the development of fundamentally new mathematical techniques and algorithmic paradigms throughout modern statistics.
Roadmap of this monograph
In this monograph, we aim to present a coherent, non-asymptotic framework that covers core algorithmic principles as well as cutting-edge theoretical developments. In particular, we seek to provide sharp sample complexity characterizations for several RL algorithms, benchmarked against minimax lower bounds to assess their statistical optimality or suboptimality.
The bulk of this monograph is organized around the three sampling mechanisms introduced in § 1.1 . We devote Chapters 3 and 4 to the generative-model setting, Chapter 5 to online RL, and Chapter 6 to offline RL. In each setting, we present optimal or widely used algorithms alongside sharp statistical guarantees and minimax lower bounds. For the major results, we provide largely self-contained proofs supplemented by intermediate arguments that build intuition. Chapter 7 moves beyond these canonical single-agent formulations and briefly introduces several extensions of growing importance, including distributionally robust RL, multi-agent RL, federated RL, statistical inference, and uncertainty quantification.
To develop these statistical foundations in the cleanest possible settings, we focus predominantly on tabular MDPs, where the state and action spaces are finite, and no low-dimensional function approximation is assumed. Notably, despite decades of study, the fundamental minimax sample complexity limits for tabular MDPs under the three canonical sampling mechanisms remained significantly unresolved for a long time and have only been settled—in several major, though not all, MDP models—very recently. We believe that obtaining a comprehensive understanding of these tabular fundamental limits constitutes an indispensable first step toward elucidating and ultimately achieving optimal statistical efficiency in RL at large, particularly in settings involving function approximation.
Relation to existing RL texts
Several existing books and monographs are particularly relevant to, or complementary with, the present monograph, and we briefly discuss a few of them here.
For introductory material, the foundational textbooks Sutton and Barto (2018), Bertsekas (2019), and Szepesvári (2010) provide excellent and comprehensive introductions to RL for readers with little or no prior background, albeit with much less emphasis on modern statistical foundations. The special cases of multi-armed bandits and contextual bandits have been covered in several books (e.g., Bubeck and Cesa-Bianchi (2012), Slivkins (2019), and Lattimore and Szepesvári (2020)), with Lattimore and Szepesvári (2020) offering perhaps the most comprehensive and up-to-date treatment of modern bandit theory. The monograph Agarwal et al. (2019a) has become a widely used resource for researchers entering RL theory, owing to its broad coverage of contemporary topics. A substantial portion of the present monograph instead focuses on advances in the statistical theory of tabular RL that appeared after Agarwal et al. (2019a) first became available. These include substantial progress in characterizing and attaining sharp statistical limits for RL with a generative model, online RL, and offline RL, as well as in deriving sharp sample complexity guarantees for Q-learning.
We also note the monograph by Foster and Rakhlin (2023), which develops a statistical perspective on interactive decision-making, with a primary focus on the general framework of function approximation. In addition, Zhang (2023) contains chapters on bandits and RL that briefly discuss several modern topics, including linear mixture MDPs and the Bellman–Eluder dimension. The recent book Guo et al. (2026) offers a comprehensive introduction to modern development of continuous RL from the perspective of stochastic control. The emphasis of these three texts, however, differs drastically from that of the present monograph, as none of them centers on the state-of-the-art statistical development in tabular RL.
Moreover, the present monograph grew out of several tutorials and courses taught by some of the authors in recent years, which also led to lecture notes such as Chi et al. (2025b). In comparison, Chi et al. (2025b) offers a bird’s-eye view of the recent statistical and algorithmic advances in RL, including developments in policy optimization, whereas the present monograph focuses more squarely on the statistical foundations of RL, offering a thorough and in-depth treatment accompanied by largely self-contained mathematical proofs.
Notation
For clarity and consistency, we introduce below a set of notation and conventions to be used throughout this monograph. Additional notation specific to individual chapters will be introduced later as needed.
Asymptotic notation. Let X : = ( S , A , 1 1 − γ , H , 1 ε , 1 δ ) \mathcal{X} := (S, A, \frac{1}{1-\gamma}, H, \frac{1}{\varepsilon}, \frac{1}{\delta}) X := ( S , A , 1 − γ 1 , H , ε 1 , δ 1 ) , where the meanings of these parameters will become clear shortly. The notation f ( X ) = O ( g ( X ) ) f(\mathcal{X}) = O(g(\mathcal{X})) f ( X ) = O ( g ( X )) or f ( X ) ≲ g ( X ) f(\mathcal{X}) \lesssim g(\mathcal{X}) f ( X ) ≲ g ( X ) means there exists a universal constant C 1 > 0 C_1 > 0 C 1 > 0 such that f ( X ) ≤ C 1 g ( X ) f(\mathcal{X}) \le C_1 g(\mathcal{X}) f ( X ) ≤ C 1 g ( X ) . Similarly, f ( X ) = Ω ( g ( X ) ) f(\mathcal{X}) = \Omega(g(\mathcal{X})) f ( X ) = Ω ( g ( X )) or f ( X ) ≳ g ( X ) f(\mathcal{X}) \gtrsim g(\mathcal{X}) f ( X ) ≳ g ( X ) means that g ( X ) = O ( f ( X ) ) g(\mathcal{X}) = O(f(\mathcal{X})) g ( X ) = O ( f ( X )) , while f ( X ) = Θ ( g ( X ) ) f(\mathcal{X}) = \Theta(g(\mathcal{X})) f ( X ) = Θ ( g ( X )) or f ( X ) ≍ g ( X ) f(\mathcal{X}) \asymp g(\mathcal{X}) f ( X ) ≍ g ( X ) indicates that f ( X ) = O ( g ( X ) ) f(\mathcal{X}) = O(g(\mathcal{X})) f ( X ) = O ( g ( X )) and g ( X ) = O ( f ( X ) ) g(\mathcal{X}) = O(f(\mathcal{X})) g ( X ) = O ( f ( X )) hold simultaneously. Additionally, the notation O ~ ( ⋅ ) \widetilde{O}(\cdot) O ( ⋅ ) (resp. Ω ~ ( ⋅ ) \widetilde{\Omega}(\cdot) Ω ( ⋅ ) and Θ ~ ( ⋅ ) \widetilde{\Theta}(\cdot) Θ ( ⋅ ) ) is defined analogously to O ( ⋅ ) O(\cdot) O ( ⋅ ) (resp. Ω ( ⋅ ) \Omega(\cdot) Ω ( ⋅ ) and Θ ( ⋅ ) \Theta(\cdot) Θ ( ⋅ ) ) but with logarithmic factors suppressed.
Notation
Vectors and matrices. For any vector a = [ a i ] 1 ≤ i ≤ n ∈ R n a = [a_i]_{1\le i\le n} \in\mathbb{R}^n a = [ a i ] 1 ≤ i ≤ n ∈ R n , we overload several scalar operators in an entrywise manner. For instance, we write
a : = [ a i ] 1 ≤ i ≤ n , ∣ a ∣ : = [ ∣ a i ∣ ] 1 ≤ i ≤ n , and a 2 : = [ a i 2 ] 1 ≤ i ≤ n . \sqrt{a} := [\sqrt{a_i}]_{1\le i\le n}, \qquad|a| := [|a_i|]_{1\le i\le n}, \qquad\text{and} \qquad a^2 := [a_i^2]_{1\le i\le n}. a := [ a i ] 1 ≤ i ≤ n , ∣ a ∣ := [ ∣ a i ∣ ] 1 ≤ i ≤ n , and a 2 := [ a i 2 ] 1 ≤ i ≤ n .
For vectors a = [ a i ] 1 ≤ i ≤ n a = [a_i]_{1\le i\le n} a = [ a i ] 1 ≤ i ≤ n and b = [ b i ] 1 ≤ i ≤ n b = [b_i]_{1\le i\le n} b = [ b i ] 1 ≤ i ≤ n , the notation a ≥ b a \ge b a ≥ b (resp. a ≤ b a \le b a ≤ b ) means that a i ≥ b i a_i \ge b_i a i ≥ b i (resp. a i ≤ b i a_i \le b_i a i ≤ b i ) for all 1 ≤ i ≤ n 1 \le i \le n 1 ≤ i ≤ n . We use
a ∘ b : = [ a i b i ] 1 ≤ i ≤ n a \circ b := [a_i b_i]_{1\le i\le n} a ∘ b := [ a i b i ] 1 ≤ i ≤ n
to represent the Hadamard product of a a a and b b b . We also let ∥ A ∥ \lVert A\rVert ∥ A ∥ and Tr ( A ) \operatorname{Tr}(A) Tr ( A ) denote the spectral norm and the trace of a matrix A A A , respectively.
For any finite set S = { 1 , … , S } \mathcal{S} = \{1,\ldots,S\} S = { 1 , … , S } , we let Δ ( S ) = { p ∈ R S ∣ p ≥ 0 , 1 T p = 1 } \Delta(\mathcal{S}) = \{p \in\mathbb{R}^{S} \mid p \ge0, \mathbf{1}^{\mathsf{T}}p = 1\} Δ ( S ) = { p ∈ R S ∣ p ≥ 0 , 1 T p = 1 } represent the set of probability vectors (i.e., the probability simplex) over S \mathcal{S} S . For any matrix P P P , we define its ℓ 1 \ell_1 ℓ 1 norm (i.e., the maximum row-wise ℓ 1 \ell_1 ℓ 1 norm) as
∥ P ∥ 1 : = max i ∑ j ∣ P i , j ∣ . (1) \lVert P\rVert_1 := \max_i \sum_j |P_{i,j}|. \tag*{(1)} ∥ P ∥ 1 := i max j ∑ ∣ P i , j ∣. (1)
Probability divergences. For any two probability measures P P P and Q Q Q defined on the same measurable space ( Ω , F ) (\Omega,\mathcal{F}) ( Ω , F ) , the total-variation (TV) distance between them is defined as
TV ( P , Q ) : = sup A ∈ F ∣ P ( A ) − Q ( A ) ∣ . (2) \operatorname{TV}(P,Q) := \sup_{A\in\mathcal{F}} |P(A)-Q(A)|. \tag*{(2)} TV ( P , Q ) := A ∈ F sup ∣ P ( A ) − Q ( A ) ∣. (2)
When P P P is absolutely continuous w.r.t. Q Q Q , the Kullback-Leibler (KL) divergence and χ 2 \chi^2 χ 2 divergence between them are defined respectively by
KL ( P ∥ Q ) : = ∫ ( log d P d Q ) d P , (3) \operatorname{KL}(P\mathbin\Vert Q) := \int\left(\log\frac{\mathrm{d}P}{\mathrm{d}Q}\right)\mathrm{d}P, \tag*{(3)} KL ( P ∥ Q ) := ∫ ( log d Q d P ) d P , (3)
χ 2 ( P ∥ Q ) : = ∫ ( d P d Q − 1 ) 2 d Q . (4) \chi^2(P\mathbin\Vert Q) := \int\left(\frac{\mathrm{d}P}{\mathrm{d}Q}-1\right)^2\mathrm{d}Q. \tag*{(4)} χ 2 ( P ∥ Q ) := ∫ ( d Q d P − 1 ) 2 d Q . (4)
These divergences play a central role in mathematical statistics, information theory, and probability theory as measures of the “statistical distance” between probability distributions; see, e.g., Tsybakov (2009) and Polyanskiy and Wu (2025), for in-depth discussions. These divergence metrics are also related through standard inequalities; for instance, it is well known that (Tsybakov, 2009)
2 TV 2 ( P , Q ) ≤ KL ( P ∥ Q ) ≤ χ 2 ( P ∥ Q ) . (5) 2\operatorname{TV}^2(P,Q) \le\operatorname{KL}(P\mathbin\Vert Q) \le\chi^2(P\mathbin\Vert Q). \tag*{(5)} 2 TV 2 ( P , Q ) ≤ KL ( P ∥ Q ) ≤ χ 2 ( P ∥ Q ) . (5)
Miscellaneous. For any positive integer n n n , we denote [ n ] : = { 1 , … , n } [n] := \{1,\ldots,n\} [ n ] := { 1 , … , n } . We use e i e_i e i to represent the i i i -th standard basis vector. The notation 1 { ⋅ } \mathbf{1}\{\cdot\} 1 { ⋅ } stands for the indicator function.
Preliminaries
We begin by reviewing the fundamentals of Markov decision processes (MDPs), the canonical framework for developing and analyzing reinforcement learning algorithms. This framework sets the stage for the discussions in the chapters that follow. We focus on two widely studied formulations: discounted infinite-horizon MDPs and finite-horizon episodic MDPs. Additionally, we present core concentration inequalities and techniques for deriving minimax lower bounds for multi-hypothesis testing, which serve as the primary analytical tools for this work.
Discounted infinite-horizon MDPs
We first introduce the basics and notation for discounted infinite-horizon MDPs. More comprehensive introductions can be found in classical references such as Puterman (2014) and Bertsekas (2017).
Basic components
A discounted infinite-horizon MDP can be represented by a tuple M = { S , A , P , r , γ } \mathcal{M} = \{\mathcal{S}, \mathcal{A}, P, r, \gamma\} M = { S , A , P , r , γ } , which models an agent’s sequential interaction with its environment over an unbounded time horizon (see Figure 2.1 for an illustration). Its key components are:
Figure 2.1. An illustration of an MDP, where an agent interacts sequentially with the environment. At time step t t t , the agent observes state s t s_t s t of the environment, selects action a t a_t a t , and receives an immediate reward r t r_t r t . The environment then transitions to state s t + 1 s_{t+1} s t + 1 according to the transition kernel P P P .
S = { 1 , … , S } \mathcal{S} = \{1, \ldots, S\} S = { 1 , … , S } : a finite state space consisting of S S S distinct states. A state can be used to describe, for example, the agent’s current situation (e.g., the position of a self-driving vehicle, the health condition of a patient in a treatment model).
A = { 1 , … , A } \mathcal{A} = \{1, \ldots, A\} A = { 1 , … , A } : an action space consisting of A A A distinct actions available to the agent (e.g., the direction of movement in a navigation task, the choice of medication to prescribe to a patient).
P : S × A → Δ ( S ) P : \mathcal{S} \times\mathcal{A} \to\Delta(\mathcal{S}) P : S × A → Δ ( S ) : the transition kernel of the MDP, where P ( ⋅ ∣ s , a ) P(\cdot\mid s, a) P ( ⋅ ∣ s , a ) specifies the probability distribution over the next state when action a a a is executed in state s s s .
r : S × A → [ 0 , 1 ] r : \mathcal{S} \times\mathcal{A} \to[0, 1] r : S × A → [ 0 , 1 ] : a deterministic reward function, where r ( s , a ) r(s, a) r ( s , a ) indicates the immediate reward received by the agent for taking action a a a in state s s s . Rewards are normalized to lie
within [ 0 , 1 ] [0,1] [ 0 , 1 ] without loss of generality. To streamline presentation, we assume throughout this monograph that the reward function is fully known to the learner a priori . Note, however, that this assumption is not essential and can often be substantially relaxed.
(v) γ ∈ [ 0 , 1 ) \gamma\in[0,1) γ ∈ [ 0 , 1 ) : the discount factor, which guarantees that the infinite-horizon cumulative discounted reward is always finite (as we shall see shortly). The quantity 1 1 − γ \frac{1}{1-\gamma} 1 − γ 1 is commonly referred to as the effective horizon of M \mathcal{M} M : it is proportional to the number of steps needed for a reward to be discounted to half its value, and hence reflects the time scale over which future rewards remain significant.
Here and throughout, we assume the discounted infinite-horizon MDP to be stationary (or time-homogeneous ), meaning that both the transition kernel P P P and the reward function r r r are time-invariant. In addition, we introduce the following notation for convenience:
P s , a : = P ( ⋅ ∣ s , a ) ∈ R 1 × S . (6) P_{s,a} := P(\cdot\mid s,a) \in\mathbb{R}^{1 \times S}. \tag*{(6)} P s , a := P ( ⋅ ∣ s , a ) ∈ R 1 × S . (6)
Throughout this monograph, we focus on regimes in which the number of states S S S , the number of actions A A A , and the effective horizon 1 1 − γ \frac{1}{1-\gamma} 1 − γ 1 are all large. This emphasis reflects a common challenge arising in modern RL practice, where algorithms must contend with vast state–action spaces and long horizons.
Policy, value function, Q-function, and occupancy distribution
At each time step, the agent selects an action based on the current state of the environment. Such an action-selection rule is called a policy. Formally, a stationary policy π : S → Δ ( A ) \pi: \mathcal{S} \to\Delta(\mathcal{A}) π : S → Δ ( A ) is a mapping that assigns to each state s ∈ S s \in\mathcal{S} s ∈ S a probability distribution over actions; namely, π ( a ∣ s ) \pi(a \mid s) π ( a ∣ s ) indicates the probability of selecting action a a a in state s s s . Clearly, this definition allows policies to be randomized. When π \pi π is a deterministic policy, we often abuse the notation by letting π ( s ) \pi(s) π ( s ) represent the action selected by policy π \pi π in state s s s . We further denote P π : S → Δ ( S ) P_\pi: \mathcal{S} \to\Delta(\mathcal{S}) P π : S → Δ ( S ) the equivalent transition probability when action a a a is drawn from π ( ⋅ ∣ s ) \pi(\cdot\mid s) π ( ⋅ ∣ s ) , namely,
∀ s ∈ S : P π ( ⋅ ∣ s ) : = E a ∼ π ( ⋅ ∣ s ) [ P ( ⋅ ∣ s , a ) ] . (7) \forall s \in\mathcal{S} : \quad P_\pi(\cdot\mid s) := \mathbb{E}_{a \sim\pi(\cdot\mid s)}[P(\cdot\mid s,a)]. \tag*{(7)} ∀ s ∈ S : P π ( ⋅ ∣ s ) := E a ∼ π ( ⋅ ∣ s ) [ P ( ⋅ ∣ s , a )] . (7)
As a remark, for the stationary infinite-horizon MDPs considered here, it is typically sufficient to restrict attention to time-invariant policies.
To evaluate a policy, we introduce two central performance measures (or “utility functions”) in RL: the value function and the Q Q Q -function. Consider a sample trajectory { ( s t , a t ) } t ≥ 0 \{(s_t,a_t)\}_{t \ge0} {( s t , a t ) } t ≥ 0 generated by the MDP under policy π \pi π , which selects action a t a_t a t after observing the state s t s_t s t at time step t t t (see Figure 2.2 for an illustration). Note that the trajectory depends on the policy π \pi π as the choice of action a t a_t a t affects the next state s t + 1 s_{t+1} s t + 1 . We now define the value function and the Q-function associated with this policy.
Figure 2.2. An illustration of a trajectory rolled out by an MDP, where r t = r ( s t , a t ) r_t = r(s_t, a_t) r t = r ( s t , a t ) is the immediate reward gained at time step t t t .
∀ s ∈ S : V π ( s ) : = E [ ∑ t = 0 ∞ γ t r ( s t , a t ) | s 0 = s ] , (8) \forall s \in\mathcal{S} : \quad V^\pi(s) := \mathbb{E}\left[\sum_{t=0}^{\infty}\gamma^t r(s_t, a_t) \,\middle|\, s_0 = s\right], \tag*{(8)} ∀ s ∈ S : V π ( s ) := E [ t = 0 ∑ ∞ γ t r ( s t , a t ) s 0 = s ] , (8)
where the expectation is over the sample trajectory { ( s t , a t ) } t ≥ 0 \{(s_t, a_t)\}_{t\geq0} {( s t , a t ) } t ≥ 0 generated under policy π \pi π , with a t ∼ π ( ⋅ ∣ s t ) a_t \sim\pi(\cdot\mid s_t) a t ∼ π ( ⋅ ∣ s t ) and s t + 1 ∼ P ( ⋅ ∣ s t , a t ) s_{t+1} \sim P(\cdot\mid s_t, a_t) s t + 1 ∼ P ( ⋅ ∣ s t , a t ) for all t ≥ 0 t \geq0 t ≥ 0 , starting from an initial state s 0 = s s_0 = s s 0 = s . Intuitively, V π ( s ) V^\pi(s) V π ( s ) represents the long-term reward an agent can expect when starting from state s s s and following policy π \pi π , with future rewards geometrically discounted over time. Since immediate rewards are assumed to lie in [ 0 , 1 ] [0,1] [ 0 , 1 ] , it follows that
0 ≤ V π ( s ) ≤ 1 1 − γ 0 \leq V^\pi(s) \leq\frac{1}{1-\gamma} 0 ≤ V π ( s ) ≤ 1 − γ 1
for every policy π \pi π . Moreover, if the initial state is randomly drawn from some state distribution ρ ∈ Δ ( S ) \rho\in\Delta(\mathcal{S}) ρ ∈ Δ ( S ) , then we can also define the value function of policy π \pi π under initial state distribution ρ \rho ρ as:
V π ( ρ ) : = E s ∼ ρ [ V π ( s ) ] . (9) V^\pi(\rho) := \mathbb{E}_{s\sim\rho}\left[V^\pi(s)\right]. \tag*{(9)} V π ( ρ ) := E s ∼ ρ [ V π ( s ) ] . (9)
Q-function. Analogously, the Q-function (or action-value function) Q π : S × A → R Q^\pi: \mathcal{S} \times\mathcal{A} \to\mathbb{R} Q π : S × A → R of policy π \pi π can be defined such that: for every ( s , a ) ∈ S × A (s,a) \in\mathcal{S} \times\mathcal{A} ( s , a ) ∈ S × A ,
Q π ( s , a ) : = E [ ∑ t = 0 ∞ γ t r ( s t , a t ) | s 0 = s , a 0 = a ] . (10) Q^\pi(s,a) := \mathbb{E}\left[\sum_{t=0}^{\infty}\gamma^t r(s_t, a_t) \,\middle|\, s_0 = s, a_0 = a\right]. \tag*{(10)} Q π ( s , a ) := E [ t = 0 ∑ ∞ γ t r ( s t , a t ) s 0 = s , a 0 = a ] . (10)
In contrast to the value function, the Q-function conditions on not only the initial state s 0 = s s_0 = s s 0 = s , but also on the initial action a 0 = a a_0 = a a 0 = a , with subsequent actions drawn from π \pi π . It therefore quantifies the expected long-term return by first taking action a a a in state s s s and then following policy π \pi π thereafter. This function provides a natural way to compare the relative benefits of different actions in a given state. The intimate connection between the Q-function and the value function is reflected in the following identity:
V π ( s ) = E a ∼ π ( ⋅ ∣ s ) [ Q π ( s , a ) ] . (11) V^\pi(s) = \mathbb{E}_{a\sim\pi(\cdot\mid s)}\left[Q^\pi(s,a)\right]. \tag*{(11)} V π ( s ) = E a ∼ π ( ⋅ ∣ s ) [ Q π ( s , a ) ] . (11)
As is well known, there exists at least one deterministic policy, denoted by π ⋆ \pi^\star π ⋆ throughout, that is optimal in the sense that it simultaneously maximizes V π ⋆ ( s ) V^{\pi^\star}(s) V π ⋆ ( s ) and Q π ⋆ ( s , a ) Q^{\pi^\star}(s,a) Q π ⋆ ( s , a ) for all state-action pairs ( s , a ) ∈ S × A (s,a) \in\mathcal{S} \times\mathcal{A} ( s , a ) ∈ S × A (see, e.g., Bertsekas (2017)). We adopt the following shorthand notation for the corresponding optimal value function and optimal Q Q Q -function:
∀ ( s , a ) ∈ S × A : V ⋆ ( s ) : = V π ⋆ ( s ) and Q ⋆ ( s , a ) : = Q π ⋆ ( s , a ) . (12) \forall(s,a) \in\mathcal{S} \times\mathcal{A}: \quad V^\star(s) := V^{\pi^\star}(s) \quad\text{and} \quad Q^\star(s,a) := Q^{\pi^\star}(s,a). \tag*{(12)} ∀ ( s , a ) ∈ S × A : V ⋆ ( s ) := V π ⋆ ( s ) and Q ⋆ ( s , a ) := Q π ⋆ ( s , a ) . (12)
Next, we find it helpful to also introduce the discounted occupancy distributions associated with policy π \pi π , quantifying the frequencies with which each state (or state-action pair) is visited under policy π \pi π . Given an initial state distribution ρ ∈ Δ ( S ) \rho\in\Delta(\mathcal{S}) ρ ∈ Δ ( S ) , define the state occupancy distribution d π ( s ; ρ ) d^\pi(s;\rho) d π ( s ; ρ ) and the state-action occupancy distribution d π ( s , a ; ρ ) d^\pi(s,a;\rho) d π ( s , a ; ρ ) such that: for every ( s , a ) ∈ S × A (s,a) \in\mathcal{S} \times\mathcal{A} ( s , a ) ∈ S × A ,
d π ( s ; ρ ) : = ( 1 − γ ) ∑ t = 0 ∞ γ t P ( s t = s ∣ s 0 ∼ ρ ) , (13) d^\pi(s;\rho) := (1-\gamma) \sum_{t=0}^{\infty} \gamma^t \mathbb{P}(s_t=s \mid s_0 \sim\rho), \tag*{(13)} d π ( s ; ρ ) := ( 1 − γ ) t = 0 ∑ ∞ γ t P ( s t = s ∣ s 0 ∼ ρ ) , (13)
d π ( s , a ; ρ ) : = ( 1 − γ ) ∑ t = 0 ∞ γ t P ( s t = s , a t = a ∣ s 0 ∼ ρ ) . (14) d^\pi(s,a;\rho) := (1-\gamma) \sum_{t=0}^{\infty} \gamma^t \mathbb{P}(s_t=s,a_t=a \mid s_0 \sim\rho). \tag*{(14)} d π ( s , a ; ρ ) := ( 1 − γ ) t = 0 ∑ ∞ γ t P ( s t = s , a t = a ∣ s 0 ∼ ρ ) . (14)
where the sample trajectory starts from s 0 ∼ ρ s_0 \sim\rho s 0 ∼ ρ and follows policy π \pi π (i.e., a t ∼ π ( ⋅ ∣ s t ) a_t \sim\pi(\cdot\mid s_t) a t ∼ π ( ⋅ ∣ s t ) and s t + 1 ∼ P ( ⋅ ∣ s t , a t ) s_{t+1} \sim P(\cdot\mid s_t,a_t) s t + 1 ∼ P ( ⋅ ∣ s t , a t ) for all t ≥ 0 t \ge0 t ≥ 0 ). In words, these functions describe the visitation frequencies of states and state-action pairs — with each visit discounted geometrically over time — under a policy over an infinite horizon. The prefactor 1 − γ 1-\gamma 1 − γ is the reciprocal of the total weight, ensuring that these distributions are normalized.
Correspondingly, we adopt the notation of the discounted occupancy distributions associated with the optimal deterministic policy π ⋆ \pi^\star π ⋆ , denoted by d ⋆ d^\star d ⋆ , as follows:
d ⋆ ( s ; ρ ) : = d π ⋆ ( s ; ρ ) and (15) d^\star(s;\rho) := d^{\pi^\star}(s;\rho) \quad\text{and} \tag*{(15)} d ⋆ ( s ; ρ ) := d π ⋆ ( s ; ρ ) and (15)
d ⋆ ( s , a ; ρ ) : = d π ⋆ ( s , a ; ρ ) = d ⋆ ( s ; ρ ) 1 ( a = π ⋆ ( s ) ) (16) d^\star(s,a;\rho) := d^{\pi^\star}(s,a;\rho) = d^\star(s;\rho)\,\mathbb{1}(a=\pi^\star(s)) \tag*{(16)} d ⋆ ( s , a ; ρ ) := d π ⋆ ( s , a ; ρ ) = d ⋆ ( s ; ρ ) 1 ( a = π ⋆ ( s )) (16)
for every ( s , a ) ∈ S × A (s,a) \in\mathcal{S} \times\mathcal{A} ( s , a ) ∈ S × A , where the last equality in (16) is valid since π ⋆ \pi^\star π ⋆ is assumed to be deterministic. Here, we dropped the dependency on ρ \rho ρ for notational simplicity.
Bellman operators and Bellman equations
Central to the design and analysis of reinforcement learning algorithms are the Bellman operators and the Bellman equations, named after Richard Bellman (Bellman (1952)), which we now introduce.
Bellman consistency operator. Given a policy π \pi π , the Bellman consistency operator T π : R S × A → R S × A \mathcal{T}^{\pi}: \mathbb{R}^{\mathcal{S} \times\mathcal{A}} \to\mathbb{R}^{\mathcal{S} \times\mathcal{A}} T π : R S × A → R S × A is defined such that, for any input Q ∈ R S × A Q \in\mathbb{R}^{\mathcal{S} \times\mathcal{A}} Q ∈ R S × A , the ( s , a ) (s,a) ( s , a ) -th entry of the output T π ( Q ) \mathcal{T}^{\pi}(Q) T π ( Q ) is given by
T π ( Q ) ( s , a ) : = r ( s , a ) + γ E s ′ ∼ P ( ⋅ ∣ s , a ) [ E a ′ ∼ π ( ⋅ ∣ s ′ ) [ Q ( s ′ , a ′ ) ] ] . (17) \mathcal{T}^{\pi}(Q)(s,a) := r(s,a) + \gamma\underset{s' \sim P(\cdot\mid s,a)}{\mathbb{E}}\left[\underset{a' \sim\pi(\cdot\mid s')}{\mathbb{E}}\left[Q(s',a')\right]\right]. \tag*{(17)} T π ( Q ) ( s , a ) := r ( s , a ) + γ s ′ ∼ P ( ⋅ ∣ s , a ) E [ a ′ ∼ π ( ⋅ ∣ s ′ ) E [ Q ( s ′ , a ′ ) ] ] . (17)
Intuitively, the operator T π \mathcal{T}^{\pi} T π performs a one-step lookahead: it evaluates a state-action pair ( s , a ) (s,a) ( s , a ) by adding the immediate reward r ( s , a ) r(s,a) r ( s , a ) to the discounted value of the next state-action pair, assuming that future actions are drawn from policy π \pi π .
It follows directly that the Q Q Q -function of policy π \pi π (cf. (10) ) is a fixed point of this Bellman consistency operator, satisfying
Q π = T π ( Q π ) . (18) Q^\pi= \mathcal{T}^{\pi}(Q^\pi). \tag*{(18)} Q π = T π ( Q π ) . (18)
The solution is also unique (see Bertsekas (2017) and also (2.15)). Combined with the relation (11) between V π V^\pi V π and Q π Q^\pi Q π , this further implies that the value function satisfies
V π = T π ( V π ) , (19) V^\pi= \mathcal{T}_\pi(V^\pi), \tag*{(19)} V π = T π ( V π ) , (19)
where the operator T π : R S → R S \mathcal{T}_\pi: \mathbb{R}^{S} \to\mathbb{R}^{S} T π : R S → R S is defined such that for any V ∈ R S V \in\mathbb{R}^{S} V ∈ R S , the s s s -th entry of T π ( V ) \mathcal{T}_\pi(V) T π ( V ) is given by
T π ( V ) ( s ) = E a ∼ π ( ⋅ ∣ s ) [ r ( s , a ) ] + γ E s ′ ∼ P ( ⋅ ∣ s , π ) [ V ( s ′ ) ] . (20) \mathcal{T}_\pi(V)(s) = \mathbb{E}_{a\sim\pi(\cdot\mid s)}[r(s,a)] + \gamma\mathbb{E}_{s'\sim P(\cdot\mid s,\pi)}[V(s')]. \tag*{(20)} T π ( V ) ( s ) = E a ∼ π ( ⋅ ∣ s ) [ r ( s , a )] + γ E s ′ ∼ P ( ⋅ ∣ s , π ) [ V ( s ′ )] . (20)
Since the Bellman equations in (18) and (19) form systems of linear equations in Q π Q^\pi Q π and V π V^\pi V π , respectively, they can be solved to evaluate a given policy, provided that the transition kernel P P P and reward function r r r are known a priori .
Bellman optimality operator. We now introduce another fundamental operator, the Bellman optimality operator T \mathcal{T} T , which is also a mapping from R S × A \mathbb{R}^{S\times A} R S × A to itself. Concretely, given any input Q ∈ R S × A Q \in\mathbb{R}^{S\times A} Q ∈ R S × A , the ( s , a ) (s,a) ( s , a ) -th entry of the output T ( Q ) \mathcal{T}(Q) T ( Q ) is defined as
T ( Q ) ( s , a ) : = r ( s , a ) + γ E s ′ ∼ P ( ⋅ ∣ s , a ) [ max a ′ ∈ A Q ( s ′ , a ′ ) ] . (21) \mathcal{T}(Q)(s,a) := r(s,a) + \gamma\mathbb{E}_{s'\sim P(\cdot\mid s,a)}\left[\max_{a'\in A} Q(s',a')\right]. \tag*{(21)} T ( Q ) ( s , a ) := r ( s , a ) + γ E s ′ ∼ P ( ⋅ ∣ s , a ) [ a ′ ∈ A max Q ( s ′ , a ′ ) ] . (21)
In words, T ( Q ) ( s , a ) \mathcal{T}(Q)(s,a) T ( Q ) ( s , a ) combines the immediate reward with the discounted expected return of the best possible action in the next state s ′ s' s ′ . A fundamental property of the Bellman optimality operator is that the optimal Q-function Q ⋆ Q^\star Q ⋆ (cf. (12) ) is its unique fixed point (Bertsekas, 2017), satisfying
Q ⋆ = T ( Q ⋆ ) . (22) Q^\star= \mathcal{T}(Q^\star). \tag*{(22)} Q ⋆ = T ( Q ⋆ ) . (22)
As we shall see momentarily, this celebrated result underpins the design of various algorithms like Q-learning.
Contraction of Bellman operators. A remarkable property of the above Bellman operators is γ \gamma γ -contraction: specifically, for any Q 1 , Q 2 ∈ R S × A Q_1,Q_2 \in\mathbb{R}^{S\times A} Q 1 , Q 2 ∈ R S × A , it holds that (Bertsekas, 2017)
∥ T π ( Q 1 ) − T π ( Q 2 ) ∥ ∞ ≤ γ ∥ Q 1 − Q 2 ∥ ∞ , (23) \left\|\mathcal{T}^\pi(Q_1)-\mathcal{T}^\pi(Q_2)\right\|_\infty\le\gamma\left\|Q_1-Q_2\right\|_\infty, \tag*{(23)} ∥ T π ( Q 1 ) − T π ( Q 2 ) ∥ ∞ ≤ γ ∥ Q 1 − Q 2 ∥ ∞ , (23)
∥ T ( Q 1 ) − T ( Q 2 ) ∥ ∞ ≤ γ ∥ Q 1 − Q 2 ∥ ∞ . (24) \left\|\mathcal{T}(Q_1)-\mathcal{T}(Q_2)\right\|_\infty\le\gamma\left\|Q_1-Q_2\right\|_\infty. \tag*{(24)} ∥ T ( Q 1 ) − T ( Q 2 ) ∥ ∞ ≤ γ ∥ Q 1 − Q 2 ∥ ∞ . (24)
In other words, applying either the Bellman consistency operator or the Bellman optimality operator brings two Q-functions strictly closer under the ℓ ∞ \ell_\infty ℓ ∞ norm. This contraction property is fundamental: it ensures the uniqueness of their fixed points and enables rapid convergence of many iterative algorithms in dynamic programming and RL.
Proof of γ \gamma γ -contraction (2.15). Regarding T π \mathcal{T}^\pi T π , recall the basic fact
∣ E [ Q 1 ( s , a ) − Q 2 ( s , a ) ] ∣ ≤ max s ′ , a ′ ∣ Q 1 ( s ′ , a ′ ) − Q 2 ( s ′ , a ′ ) ∣ \left|\mathbb{E}[Q_1(s,a)-Q_2(s,a)]\right| \le\max_{s',a'}\left|Q_1(s',a')-Q_2(s',a')\right| ∣ E [ Q 1 ( s , a ) − Q 2 ( s , a )] ∣ ≤ s ′ , a ′ max ∣ Q 1 ( s ′ , a ′ ) − Q 2 ( s ′ , a ′ ) ∣
for any ( s , a ) (s,a) ( s , a ) . This combined with the definition (17) yields
∣ T π ( Q 1 ) ( s , a ) − T π ( Q 2 ) ( s , a ) ∣ = γ ∣ E [ Q 1 ( s ′ , a ′ ) − Q 2 ( s ′ , a ′ ) ] ∣ ≤ γ ∥ Q 1 − Q 2 ∥ ∞ \left|\mathcal{T}^\pi(Q_1)(s,a)-\mathcal{T}^\pi(Q_2)(s,a)\right| = \gamma\left|\mathbb{E}[Q_1(s',a')-Q_2(s',a')]\right|
\le\gamma\left\|Q_1-Q_2\right\|_\infty ∣ T π ( Q 1 ) ( s , a ) − T π ( Q 2 ) ( s , a ) ∣ = γ ∣ E [ Q 1 ( s ′ , a ′ ) − Q 2 ( s ′ , a ′ )] ∣ ≤ γ ∥ Q 1 − Q 2 ∥ ∞
for any ( s , a ) (s,a) ( s , a ) pair, where the expectation is over s ′ ∼ P ( ⋅ ∣ s , a ) s' \sim P(\cdot\mid s,a) s ′ ∼ P ( ⋅ ∣ s , a ) and a ′ ∼ π ( ⋅ ∣ s ′ ) a' \sim\pi(\cdot\mid s') a ′ ∼ π ( ⋅ ∣ s ′ ) . This establishes the γ \gamma γ -contraction property of T π \mathcal{T}^{\pi} T π .
The contraction for T \mathcal{T} T can be proven by repeating the above arguments along with the basic fact that
∣ max a Q 1 ( s , a ) − max a Q 2 ( s , a ) ∣ ≤ max a ∣ Q 1 ( s , a ) − Q 2 ( s , a ) ∣ . \left|\max_{a} Q_1(s,a)-\max_{a} Q_2(s,a)\right| \leq\max_{a}\left|Q_1(s,a)-Q_2(s,a)\right|. a max Q 1 ( s , a ) − a max Q 2 ( s , a ) ≤ a max ∣ Q 1 ( s , a ) − Q 2 ( s , a ) ∣ .
Details are omitted here for brevity.
Dynamic programming under full model specification
With the Bellman operators in place, we are now ready to introduce the classical dynamic programming algorithms for solving discounted infinite-horizon MDPs, assuming a fully known model (i.e., both the transition kernel P P P and the rewards r r r are available to the learner). The interested reader is referred to Bertsekas (2017) for in-depth discussions.
Planning. Assuming access to exact model specification (i.e., P P P and r r r ), we describe two classical algorithms for computing the optimal Q-function Q ⋆ Q^\star Q ⋆ and the optimal policy π ⋆ \pi^\star π ⋆ , a task commonly referred to as planning (see Figure 2.3 ).
Figure 2.3. Planning in an MDP with a fully specified model. Classical dynamic algorithms, such as policy iteration and value iteration, can efficiently compute the optimal policy π ⋆ \pi^\star π ⋆ and the associated optimal value given complete knowledge of P P P and r r r .
Value iteration (VI). Initialize the Q-function estimate Q 0 Q_0 Q 0 . For each iteration k = 1 , 2 , … k = 1,2,\ldots k = 1 , 2 , … , update the Q-function as
Q k = T ( Q k − 1 ) , (25) Q_k = \mathcal{T}(Q_{k-1}), \tag*{(25)} Q k = T ( Q k − 1 ) , (25)
with T \mathcal{T} T the Bellman optimality operator (cf. (21) ). In words, the Q-function estimate is refined iteratively by repeatedly applying the Bellman optimality operator to the latest Q-function estimate.
Policy iteration (PI). Initialize the policy estimate π 0 \pi_0 π 0 . For each iteration k = 1 , 2 , … k = 1,2,\ldots k = 1 , 2 , … , update the policy as
∀ s ∈ S : π k ( s ) = arg max a ∈ A Q π k − 1 ( s , a ) . (26) \forall s \in\mathcal{S}: \quad\pi_k(s) = \underset{a \in\mathcal{A}}{\arg\max}\ Q^{\pi_{k-1}}(s,a). \tag*{(26)} ∀ s ∈ S : π k ( s ) = a ∈ A arg max Q π k − 1 ( s , a ) . (26)
Namely, the policy is updated by acting greedily w.r.t. the Q-function of the current policy estimate.
Encouragingly, both algorithms achieve fast linear convergence to the optimal Q-function, as asserted by the following lemma.
Lemma 2.1 (Linear convergence of VI and PI). Both value iteration (cf. (25) ) and policy iteration (cf. (26) ) achieve
∥ Q k − Q ⋆ ∥ ∞ ≤ γ k ∥ Q 0 − Q ⋆ ∥ ∞ \lVert Q_k-Q^\star\rVert_\infty\leq\gamma^k\lVert Q_0-Q^\star\rVert_\infty ∥ Q k − Q ⋆ ∥ ∞ ≤ γ k ∥ Q 0 − Q ⋆ ∥ ∞ for every k ≥ 1 k \geq1 k ≥ 1 , whereas in policy iteration we take Q k = Q π k Q_k=Q^{\pi_k} Q k = Q π k .
Proof. In light of the fixed-point property (22) and γ \gamma γ -contraction (2.15) of T \mathcal{T} T , the VI iterates obey
∥ Q k − Q ⋆ ∥ ∞ = ∥ T ( Q k − 1 ) − T ( Q ⋆ ) ∥ ∞ ≤ γ ∥ Q k − 1 − Q ⋆ ∥ ∞ . \lVert Q_k-Q^\star\rVert_\infty=\lVert\mathcal{T}(Q_{k-1})-\mathcal{T}(Q^\star)\rVert_\infty\leq\gamma\lVert Q_{k-1}-Q^\star\rVert_\infty. ∥ Q k − Q ⋆ ∥ ∞ = ∥ T ( Q k − 1 ) − T ( Q ⋆ ) ∥ ∞ ≤ γ ∥ Q k − 1 − Q ⋆ ∥ ∞ . Applying this inequality iteratively completes the proof for VI. The proof for PI is omitted here and can be found in Agarwal et al., 2019a.
This lemma reveals that, to achieve ε \varepsilon ε -accuracy in the sense that ∥ Q k − Q ⋆ ∥ ∞ ≤ ε \lVert Q_k-Q^\star\rVert_\infty\leq\varepsilon ∥ Q k − Q ⋆ ∥ ∞ ≤ ε , it takes no more than
1 1 − γ log ( ∥ Q 0 − Q ⋆ ∥ ∞ ε ) \frac{1}{1-\gamma}\log\left(\frac{\lVert Q_0-Q^\star\rVert_\infty}{\varepsilon}\right) 1 − γ 1 log ( ε ∥ Q 0 − Q ⋆ ∥ ∞ )
iterations. Importantly, this iteration complexity is on the order of the effective horizon 1 1 − γ \frac{1}{1-\gamma} 1 − γ 1 (up to some logarithmic factor), irrespective of the dimension of the state-action space.
Vector and matrix notation
It is also convenient to introduce several pieces of vector and matrix notation, which allow our results and analysis to be presented more compactly. Here and throughout, we allow a slight abuse of notation by letting the same symbols represent both vectors/matrices and functions, whenever it is clear from context (e.g., the notation r r r may denote either the reward function or the corresponding reward vector).
r ∈ R S A r \in\mathbb{R}^{SA} r ∈ R S A : a vector representing the reward function, so that its ( s , a ) (s,a) ( s , a ) -th entry is the corresponding immediate reward.
V π , V ⋆ ∈ R S V^\pi,V^\star\in\mathbb{R}^{S} V π , V ⋆ ∈ R S : vectors representing respectively the value function of policy π \pi π (cf. (8) ) and the optimal value function (cf. (12) ).
Q π , Q ⋆ ∈ R S A Q^\pi,Q^\star\in\mathbb{R}^{SA} Q π , Q ⋆ ∈ R S A : vectors representing respectively the Q-function Q π Q^\pi Q π of policy π \pi π (cf. (10) ) and the optimal Q-function (cf. (12) ).
P ∈ R S A × S P \in\mathbb{R}^{SA\times S} P ∈ R S A × S : a matrix representing the transition kernel, where its ( s , a ) (s,a) ( s , a ) -th row is a probability vector representing P ( ⋅ ∣ s , a ) P(\cdot\mid s,a) P ( ⋅ ∣ s , a ) .
Π π = ( π ( ⋅ ∣ 1 ) π ( ⋅ ∣ 2 ) ⋮ π ( ⋅ ∣ S ) ) (27) \Pi^\pi=
\begin{pmatrix}
\pi(\cdot\mid1)\\
\pi(\cdot\mid2)\\
\vdots\\
\pi(\cdot\mid S)
\end{pmatrix} \tag*{(27)} Π π = π ( ⋅ ∣ 1 ) π ( ⋅ ∣ 2 ) ⋮ π ( ⋅ ∣ S ) (27)
P π ∈ R S A × S A P^{\pi} \in\mathbb{R}^{SA \times SA} P π ∈ R S A × S A and P π ∈ R S × S P_{\pi} \in\mathbb{R}^{S \times S} P π ∈ R S × S : two square probability transition matrices induced by policy π \pi π over the state-action pairs and the states, respectively, defined by
P π ≔ P Π π and P π ≔ Π π P . (28) P^{\pi} \coloneqq P\Pi^{\pi} \qquad\text{and} \qquad P_{\pi} \coloneqq\Pi^{\pi}P. \tag*{(28)} P π : = P Π π and P π : = Π π P . (28)
r π ( s ) = E a ∼ π ( ⋅ ∣ s ) [ r ( s , a ) ] for all s ∈ S , (29) r_{\pi}(s) = \mathbb{E}_{a \sim\pi(\cdot\mid s)}[r(s,a)] \qquad\text{for all }s \in\mathcal{S}, \tag*{(29)} r π ( s ) = E a ∼ π ( ⋅ ∣ s ) [ r ( s , a )] for all s ∈ S , (29)
or simply, r π = Π π r r_{\pi} = \Pi^{\pi}r r π = Π π r .
We further introduce several variance-related objects, which shall be used frequently in our statistical analysis. For any vector V = [ V i ] 1 ≤ i ≤ S ∈ R S V = [V_i]_{1 \leq i \leq S} \in\mathbb{R}^{S} V = [ V i ] 1 ≤ i ≤ S ∈ R S , we define the vector Var P ( V ) ∈ R S A \operatorname{Var}_{P}(V) \in\mathbb{R}^{SA} Var P ( V ) ∈ R S A , whose ( s , a ) (s,a) ( s , a ) -th entry is given by
[ Var P ( V ) ] ( s , a ) = Var P s , a ( V ) = ∑ s ′ ∈ S P ( s ′ ∣ s , a ) ( V s ′ ) 2 − ( ∑ s ′ ∈ S P ( s ′ ∣ s , a ) V s ′ ) 2 ; (30) [\operatorname{Var}_{P}(V)]_{(s,a)} = \operatorname{Var}_{P_{s,a}}(V)
= \sum_{s' \in\mathcal{S}} P(s' \mid s,a)(V_{s'})^2 - \left(\sum_{s' \in\mathcal{S}} P(s' \mid s,a)V_{s'}\right)^2; \tag*{(30)} [ Var P ( V ) ] ( s , a ) = Var P s , a ( V ) = s ′ ∈ S ∑ P ( s ′ ∣ s , a ) ( V s ′ ) 2 − ( s ′ ∈ S ∑ P ( s ′ ∣ s , a ) V s ′ ) 2 ; (30)
that is, the variance of V s ′ V_{s'} V s ′ when s ′ s' s ′ is sampled from the probability vector P ( ⋅ ∣ s , a ) P(\cdot\mid s,a) P ( ⋅ ∣ s , a ) . This can be expressed compactly via matrix notation:
Var P ( V ) = P ( V ∘ V ) − ( P V ) ∘ ( P V ) , (31) \operatorname{Var}_{P}(V) = P(V \circ V) - (PV) \circ(PV), \tag*{(31)} Var P ( V ) = P ( V ∘ V ) − ( P V ) ∘ ( P V ) , (31)
where we recall that a ∘ b a \circ b a ∘ b denotes the Hadamard product [ a i b i ] 1 ≤ i ≤ S [a_i b_i]_{1 \leq i \leq S} [ a i b i ] 1 ≤ i ≤ S (see § 1.5 ). Similarly, we define, for any given policy π \pi π ,
Var P π ( V ) = P π ( V ∘ V ) − ( P π V ) ∘ ( P π V ) ∈ R S . (32) \operatorname{Var}_{P_{\pi}}(V) = P_{\pi}(V \circ V) - (P_{\pi}V) \circ(P_{\pi}V) \in\mathbb{R}^{S}. \tag*{(32)} Var P π ( V ) = P π ( V ∘ V ) − ( P π V ) ∘ ( P π V ) ∈ R S . (32)
The following lemma provides a perturbation bound for Var P ( V ) \operatorname{Var}_{P}(V) Var P ( V ) .
Lemma 2.2. For any vectors V , V ^ ∈ R S V,\widehat{V} \in\mathbb{R}^{S} V , V ∈ R S obeying 0 ≤ V ≤ 1 1 − γ 1 S 0 \leq V \leq\frac{1}{1-\gamma}\mathbf{1}_{S} 0 ≤ V ≤ 1 − γ 1 1 S and 0 ≤ V ^ ≤ 1 1 − γ 1 S 0 \leq\widehat{V} \leq\frac{1}{1-\gamma}\mathbf{1}_{S} 0 ≤ V ≤ 1 − γ 1 1 S , one has
∥ Var P ( V ^ ) − Var P ( V ) ∥ ∞ ≤ 4 1 − γ ∥ V ^ − V ∥ ∞ . \left\|\operatorname{Var}_{P}(\widehat{V})-\operatorname{Var}_{P}(V)\right\|_{\infty} \leq\frac{4}{1-\gamma}\|\widehat{V}-V\|_{\infty}. Var P ( V ) − Var P ( V ) ∞ ≤ 1 − γ 4 ∥ V − V ∥ ∞ . Proof. From (31) , we have the decomposition
Var P ( V ^ ) − Var P ( V ) = P ( V ^ ∘ V ^ − V ∘ V ) − ( P V ^ ) ∘ ( P V ^ ) + ( P V ) ∘ ( P V ) \operatorname{Var}_{P}(\widehat{V})-\operatorname{Var}_{P}(V)
= P(\widehat{V}\circ\widehat{V}-V\circ V) - (P\widehat{V})\circ(P\widehat{V}) + (PV)\circ(PV) Var P ( V ) − Var P ( V ) = P ( V ∘ V − V ∘ V ) − ( P V ) ∘ ( P V ) + ( P V ) ∘ ( P V ) = P ( ( V ^ − V ) ∘ ( V ^ + V ) ) − ( P ( V ^ − V ) ) ∘ ( P ( V ^ + V ) ) . = P((\widehat{V}-V)\circ(\widehat{V}+V)) - (P(\widehat{V}-V))\circ(P(\widehat{V}+V)). = P (( V − V ) ∘ ( V + V )) − ( P ( V − V )) ∘ ( P ( V + V )) . Hence, it follows that
∥ Var P ( V ^ ) − Var P ( V ) ∥ ∞ ≤ ∥ P ∥ 1 ∥ V ^ − V ∥ ∞ ∥ V ^ + V ∥ ∞ + ∥ P ∥ 1 2 ∥ V ^ − V ∥ ∞ ∥ V ^ + V ∥ ∞ ≤ 4 1 − γ ∥ V ^ − V ∥ ∞ , \begin{aligned}
\left\|\operatorname{Var}_{P}(\widehat{V})-\operatorname{Var}_{P}(V)\right\|_{\infty}
&\leq\|P\|_{1}\|\widehat{V}-V\|_{\infty}\|\widehat{V}+V\|_{\infty}
+\|P\|_{1}^{2}\|\widehat{V}-V\|_{\infty}\|\widehat{V}+V\|_{\infty}\\
&\leq\frac{4}{1-\gamma}\|\widehat{V}-V\|_{\infty},
\end{aligned} Var P ( V ) − Var P ( V ) ∞ ≤ ∥ P ∥ 1 ∥ V − V ∥ ∞ ∥ V + V ∥ ∞ + ∥ P ∥ 1 2 ∥ V − V ∥ ∞ ∥ V + V ∥ ∞ ≤ 1 − γ 4 ∥ V − V ∥ ∞ , where the last line is valid since ∥ V ^ ∥ ∞ , ∥ V ∥ ∞ ≤ 1 1 − γ \|\widehat{V}\|_{\infty},\|V\|_{\infty} \leq\frac{1}{1-\gamma} ∥ V ∥ ∞ , ∥ V ∥ ∞ ≤ 1 − γ 1 and ∥ P ∥ 1 = 1 \|P\|_{1}=1 ∥ P ∥ 1 = 1 .
Additional basic properties
Equipped with the above notation, we present a couple of basic properties that will be useful throughout.
To begin, the Bellman consistency equation (18) can be expressed in vector/matrix form as
Q π = r + γ P V π , (33) Q^\pi= r + \gamma P V^\pi, \tag*{(33)} Q π = r + γ P V π , (33)
which, when combined with (11) , yields
V π = Π π Q π = r π + γ P π V π . (34) V^\pi= \Pi^\pi Q^\pi= r_\pi+ \gamma P_\pi V^\pi. \tag*{(34)} V π = Π π Q π = r π + γ P π V π . (34)
Solving (34) for V π V^\pi V π immediately gives
V π = ( I − γ P π ) − 1 r π . (35) V^\pi= (I-\gamma P_\pi)^{-1}r_\pi. \tag*{(35)} V π = ( I − γ P π ) − 1 r π . (35)
Substituting (34) into (33) and solving for Q π Q^\pi Q π yield
Q π = r + γ P Π π Q π = r + γ P π Q π ⟹ Q π = ( I − γ P π ) − 1 r . (36) Q^\pi= r + \gamma P\Pi^\pi Q^\pi= r + \gamma P^\pi Q^\pi\Longrightarrow Q^\pi= (I-\gamma P^\pi)^{-1}r. \tag*{(36)} Q π = r + γ P Π π Q π = r + γ P π Q π ⟹ Q π = ( I − γ P π ) − 1 r . (36)
In addition, the Bellman optimality condition (22) admits a vector/matrix form as
Q ⋆ = r + γ P V ⋆ . (37) Q^\star= r + \gamma P V^\star. \tag*{(37)} Q ⋆ = r + γ P V ⋆ . (37)
The careful reader would immediately remark that, for (35) to hold, the matrix I − γ P π I-\gamma P_\pi I − γ P π must be invertible. Fortunately, this requirement is automatically satisfied whenever 0 ≤ γ < 1 0 \le\gamma< 1 0 ≤ γ < 1 . A few basic properties of this inversion are provided in the lemma below.
Lemma 2.3. For any 0 ≤ γ < 1 0 \le\gamma< 1 0 ≤ γ < 1 and any policy π \pi π , it holds that
( I − γ P π ) − 1 = I + ∑ i = 1 ∞ ( γ P π ) i , (38) (I-\gamma P_\pi)^{-1} = I + \sum_{i=1}^{\infty}(\gamma P_\pi)^i, \tag*{(38)} ( I − γ P π ) − 1 = I + i = 1 ∑ ∞ ( γ P π ) i , (38) ( I − γ P π ) − 1 1 S = 1 1 − γ 1 S , (39) (I-\gamma P_\pi)^{-1}\mathbf{1}_S = \frac{1}{1-\gamma}\mathbf{1}_S, \tag*{(39)} ( I − γ P π ) − 1 1 S = 1 − γ 1 1 S , (39) ∥ ( I − γ P π ) − 1 ∥ 1 = 1 1 − γ , (40) \left\|(I-\gamma P_\pi)^{-1}\right\|_1 = \frac{1}{1-\gamma}, \tag*{(40)} ( I − γ P π ) − 1 1 = 1 − γ 1 , (40) where 1 n \mathbf{1}_n 1 n denotes the n n n -dimensional all-one vector. It also follows directly from (38) that ( I − γ P π ) − 1 (I-\gamma P_\pi)^{-1} ( I − γ P π ) − 1 is a nonnegative matrix.
Additionally, the above results continue to hold if P π P_\pi P π is replaced with P π P^\pi P π and 1 S \mathbf{1}_S 1 S is replaced with 1 S A \mathbf{1}_{S_A} 1 S A .
Proof of Lemma 2.3. With regard to (38) , it suffices to note that
lim i → ∞ ∥ γ i ( P π ) i ∥ 1 = lim i → ∞ γ i = 0 , \lim_{i\to\infty}\left\|\gamma^i(P_\pi)^i\right\|_1 = \lim_{i\to\infty}\gamma^i = 0, i → ∞ lim γ i ( P π ) i 1 = i → ∞ lim γ i = 0 ,
where we have used the fact that ( P π ) i (P_\pi)^i ( P π ) i is also a probability transition matrix (so that ∥ ( P π ) i ∥ 1 = 1 \left\|(P_\pi)^i\right\|_1 = 1 ( P π ) i 1 = 1 ). This implies that the matrix geometric series in (38) is convergent, thus justifying the validity of the Neumann series expansion in (38) . Regarding (39) , we can see from (38) that
( I − γ P π ) − 1 1 S = ( I + ∑ i = 1 ∞ ( γ P π ) i ) 1 S = ( 1 + ∑ i = 1 ∞ γ i ) 1 S = 1 1 − γ 1 S , (I-\gamma P_\pi)^{-1}\mathbf{1}_S = \left(I+\sum_{i=1}^{\infty}(\gamma P_\pi)^i\right)\mathbf{1}_S = \left(1+\sum_{i=1}^{\infty}\gamma^i\right)\mathbf{1}_S = \frac{1}{1-\gamma}\mathbf{1}_S, ( I − γ P π ) − 1 1 S = ( I + i = 1 ∑ ∞ ( γ P π ) i ) 1 S = ( 1 + i = 1 ∑ ∞ γ i ) 1 S = 1 − γ 1 1 S ,
where the penultimate relation follows since ( P π ) i (P_\pi)^i ( P π ) i is a probability transition matrix. This combined with the nonnegativity of ( I − γ P π ) − 1 (I-\gamma P_\pi)^{-1} ( I − γ P π ) − 1 and the definition of ∥ ⋅ ∥ 1 \|\cdot\|_1 ∥ ⋅ ∥ 1 (cf. (1) ) immediately justifies (40) .
Moreover, combining Lemma 2.3 with the Bellman equations in (2.24) reveals basic perturbation bounds. Consider two discounted infinite-horizon MDPs M = ( S , A , P , r , γ ) \mathcal{M}=(\mathcal{S},\mathcal{A},P,r,\gamma) M = ( S , A , P , r , γ ) and M ~ = ( S , A , P , r ~ , γ ~ ) \widetilde{\mathcal{M}}=(\mathcal{S},\mathcal{A},P,\widetilde{r},\widetilde{\gamma}) M = ( S , A , P , r , γ ) , which differ only in their reward functions. Denote by Q ~ π \widetilde{Q}^{\pi} Q π (resp. V ~ π \widetilde{V}^{\pi} V π ) the Q-function (resp. value function) of policy π \pi π in M ~ \widetilde{\mathcal{M}} M , and similarly, let Q ~ ⋆ \widetilde{Q}^{\star} Q ⋆ (resp. V ~ ⋆ \widetilde{V}^{\star} V ⋆ ) represent the optimal Q-function (resp. value function) in M ~ \widetilde{\mathcal{M}} M . For any policy π \pi π , it holds that
∥ Q π − Q ~ π ∥ ∞ = ∥ ( I − γ P π ) − 1 r − ( I − γ P π ) − 1 r ~ ∥ ∞ ≤ ∥ ( I − γ P π ) − 1 ∥ 1 ∥ r − r ~ ∥ ∞ = 1 1 − γ ∥ r − r ~ ∥ ∞ . (41) \begin{aligned}
\left\|Q^{\pi}-\widetilde{Q}^{\pi}\right\|_{\infty}
=
\left\|(I-\gamma P^{\pi})^{-1}r-(I-\gamma P^{\pi})^{-1}\widetilde{r}\right\|_{\infty} \\
&\leq\left\|(I-\gamma P^{\pi})^{-1}\right\|_{1}\left\|r-\widetilde{r}\right\|_{\infty}
=
\frac{1}{1-\gamma}\left\|r-\widetilde{r}\right\|_{\infty}. \tag*{(41)}
\end{aligned} Q π − Q π ∞ = ( I − γ P π ) − 1 r − ( I − γ P π ) − 1 r ∞ ≤ ( I − γ P π ) − 1 1 ∥ r − r ∥ ∞ = 1 − γ 1 ∥ r − r ∥ ∞ . (41)
and similarly,
∥ V π − V ~ π ∥ ∞ ≤ 1 1 − γ ∥ r − r ~ ∥ ∞ . (42) \left\|V^{\pi}-\widetilde{V}^{\pi}\right\|_{\infty}
\leq\frac{1}{1-\gamma}\left\|r-\widetilde{r}\right\|_{\infty}. \tag*{(42)} V π − V π ∞ ≤ 1 − γ 1 ∥ r − r ∥ ∞ . (42)
Given the validity of (2.27) for every policy π \pi π , we immediately reach
∥ Q ⋆ − Q ~ ⋆ ∥ ∞ = ∥ max π Q π − max π Q ~ π ∥ ∞ ≤ max π ∥ Q π − Q ~ π ∥ ∞ ≤ 1 1 − γ ∥ r − r ~ ∥ ∞ , (43) \begin{aligned}
\left\|Q^{\star}-\widetilde{Q}^{\star}\right\|_{\infty}
=
\left\|\max_{\pi}Q^{\pi}-\max_{\pi}\widetilde{Q}^{\pi}\right\|_{\infty}
\leq\max_{\pi}\left\|Q^{\pi}-\widetilde{Q}^{\pi}\right\|_{\infty} \\
&\leq\frac{1}{1-\gamma}\left\|r-\widetilde{r}\right\|_{\infty}, \tag*{(43)}
\end{aligned} Q ⋆ − Q ⋆ ∞ = π max Q π − π max Q π ∞ ≤ π max Q π − Q π ∞ ≤ 1 − γ 1 ∥ r − r ∥ ∞ , (43)
and similarly,
∥ V ⋆ − V ~ ⋆ ∥ ∞ ≤ 1 1 − γ ∥ r − r ~ ∥ ∞ . (44) \left\|V^{\star}-\widetilde{V}^{\star}\right\|_{\infty}
\leq\frac{1}{1-\gamma}\left\|r-\widetilde{r}\right\|_{\infty}. \tag*{(44)} V ⋆ − V ⋆ ∞ ≤ 1 − γ 1 ∥ r − r ∥ ∞ . (44)
Finite-horizon episodic MDPs
The second class of MDPs studied extensively in this monograph is that of finite-horizon episodic MDPs, which we now introduce. In contrast to infinite-horizon MDPs, where the time step can continue indefinitely, finite-horizon episodic MDPs model problems with a predetermined number of steps in each episode.
Basic components
A finite-horizon Markov decision process, denoted by M = { S , A , H , P , r } \mathcal{M}=\{\mathcal{S},\mathcal{A},H,P,r\} M = { S , A , H , P , r } , consists of the following key components.
S = { 1 , … , S } \mathcal{S}=\{1,\ldots,S\} S = { 1 , … , S } : a state space containing S S S different states, akin to the infinite-horizon counterpart.
A = { 1 , … , A } \mathcal{A}=\{1,\ldots,A\} A = { 1 , … , A } : an action space comprised of A A A distinct actions, similar to the infinite-horizon counterpart.
H H H : the horizon length, which specifies the number of time steps in each episode of the MDP. This parameter dictates the planning horizon for decision-making.
P = { P h } 1 ≤ h ≤ H P=\{P_h\}_{1\leq h\leq H} P = { P h } 1 ≤ h ≤ H , with P h : S × A → Δ ( S ) P_h:\mathcal{S}\times\mathcal{A}\to\Delta(\mathcal{S}) P h : S × A → Δ ( S ) denoting the transition kernel at step h h h (namely, P h ( ⋅ ∣ s , a ) P_h(\cdot\mid s,a) P h ( ⋅ ∣ s , a ) stands for the transition probability of the MDP at step h h h when the current state-action pair is ( s , a ) (s,a) ( s , a ) ). Conveniently, we introduce the following S S S -dimensional row vector
P s , a , h : = P h ( ⋅ ∣ s , a ) ∈ R 1 × S (45) P_{s,a,h}:=P_h(\cdot\mid s,a)\in\mathbb{R}^{1\times S} \tag*{(45)} P s , a , h := P h ( ⋅ ∣ s , a ) ∈ R 1 × S (45)
for any ( s , a , h ) ∈ S × A × [ H ] (s,a,h)\in\mathcal{S}\times\mathcal{A}\times[H] ( s , a , h ) ∈ S × A × [ H ] to indicate the corresponding transition probability.
(v) r = { r h } 1 ≤ h ≤ H r = \{r_h\}_{1\le h\le H} r = { r h } 1 ≤ h ≤ H , with r h : S × A → [ 0 , 1 ] r_h : \mathcal{S} \times\mathcal{A} \to[0,1] r h : S × A → [ 0 , 1 ] denoting the reward function at step h h h (namely, r h ( s , a ) r_h(s,a) r h ( s , a ) indicates the immediate reward gained at step h h h when taking action a a a in state s s s ). Akin to the infinite-horizon setting, we assume throughout that all immediate rewards are deterministic, fully known to the learner, and lie within the interval [ 0 , 1 ] [0,1] [ 0 , 1 ] .
Hereafter, we focus on nonstationary (i.e., time-inhomogeneous ) MDPs within the finite-horizon framework, where both the transition probabilities and reward functions are allowed to vary across time steps. We are particularly interested in regimes in which the size S S S of the state space, the size A A A of the action space, and the horizon length H H H are all large, reflecting the challenges inherent in high-dimensional, long-horizon RL problems.
Policy, value function, Q-function and occupancy distribution
We next introduce the counterparts of the preceding concepts for finite-horizon episodic MDPs. Let us begin with the notion of a policy. A (possibly randomized) policy, denoted by π = { π h } 1 ≤ h ≤ H \pi= \{\pi_h\}_{1\le h\le H} π = { π h } 1 ≤ h ≤ H with π h : S → Δ ( A ) \pi_h : \mathcal{S} \to\Delta(\mathcal{A}) π h : S → Δ ( A ) , is an action selection rule, such that π h ( a ∣ s ) \pi_h(a \mid s) π h ( a ∣ s ) specifies the probability of choosing action a a a at step h h h when in state s s s . When π \pi π is a deterministic policy, we again overload the notation and write π h ( s ) \pi_h(s) π h ( s ) to denote the action selected by π \pi π in state s s s at step h h h .
Next, we define the value function and Q-function for finite-horizon episodic MDPs. To this end, consider an episode { ( s h , a h ) } 1 ≤ h ≤ H \{(s_h,a_h)\}_{1\le h\le H} {( s h , a h ) } 1 ≤ h ≤ H with length H H H , with s h s_h s h and a h a_h a h denoting, respectively, the state and the action at step h h h , which serve as the basis for the definitions that follow.
Value function. The value function of policy π \pi π is denoted by V π = { V h π } 1 ≤ h ≤ H V^\pi= \{V_h^\pi\}_{1\le h\le H} V π = { V h π } 1 ≤ h ≤ H , where for each step 1 ≤ h ≤ H 1 \le h \le H 1 ≤ h ≤ H , V h π : S → R V_h^\pi: \mathcal{S} \to\mathbb{R} V h π : S → R quantifies the expected cumulative reward from a given state when starting at step h h h and following π \pi π until the end of the episode. More precisely, we define
∀ s ∈ S : V h π ( s ) : = E [ ∑ t = h H r t ( s t , a t ) ∣ s h = s ] . (46) \forall s \in\mathcal{S}:\quad V_h^\pi(s) := \mathbb{E}\left[\left.\sum_{t=h}^{H} r_t(s_t,a_t)\right|s_h=s\right]. \tag*{(46)} ∀ s ∈ S : V h π ( s ) := E [ t = h ∑ H r t ( s t , a t ) s h = s ] . (46)
where the expectation is over the randomness of the trajectory { ( s t , a t ) } t = h H \{(s_t,a_t)\}_{t=h}^{H} {( s t , a t ) } t = h H when policy π \pi π is implemented (i.e., a t ∼ π t ( ⋅ ∣ s t ) a_t \sim\pi_t(\cdot\mid s_t) a t ∼ π t ( ⋅ ∣ s t ) and s t + 1 ∼ P t ( ⋅ ∣ s t , a t ) s_{t+1} \sim P_t(\cdot\mid s_t,a_t) s t + 1 ∼ P t ( ⋅ ∣ s t , a t ) for all t ≥ h t \ge h t ≥ h ). Moreover, if the initial state s 1 s_1 s 1 is drawn from some state distribution ρ ∈ Δ ( S ) \rho\in\Delta(\mathcal{S}) ρ ∈ Δ ( S ) , we find it convenient to define the value function of policy π \pi π under ρ \rho ρ as
V 1 π ( ρ ) : = E s ∼ ρ [ V 1 π ( s ) ] . (47) V_1^\pi(\rho) := \mathbb{E}_{s\sim\rho}\left[V_1^\pi(s)\right]. \tag*{(47)} V 1 π ( ρ ) := E s ∼ ρ [ V 1 π ( s ) ] . (47)
Q-function. Analogously, the Q-function of policy π \pi π is denoted by Q π = { Q h π } 1 ≤ h ≤ H Q^\pi= \{Q_h^\pi\}_{1\le h\le H} Q π = { Q h π } 1 ≤ h ≤ H , which extends the definition of the value function by also conditioning on the first action to be taken. To be precise, for each 1 ≤ h ≤ H 1 \le h \le H 1 ≤ h ≤ H and every ( s , a ) ∈ S × A (s,a) \in\mathcal{S} \times\mathcal{A} ( s , a ) ∈ S × A , we define
Q h π ( s , a ) : = E [ ∑ t = h H r t ( s t , a t ) ∣ s h = s , a h = a ] . (48) Q_h^\pi(s,a) := \mathbb{E}\left[\left.\sum_{t=h}^{H} r_t(s_t,a_t)\right|s_h=s,a_h=a\right]. \tag*{(48)} Q h π ( s , a ) := E [ t = h ∑ H r t ( s t , a t ) s h = s , a h = a ] . (48)
where the trajectory { ( s t , a t ) } t = h H \{(s_t,a_t)\}_{t=h}^{H} {( s t , a t ) } t = h H is sampled in a way such that s t + 1 ∼ P t ( ⋅ ∣ s t , a t ) s_{t+1} \sim P_t(\cdot\mid s_t,a_t) s t + 1 ∼ P t ( ⋅ ∣ s t , a t ) and a t + 1 ∼ π t + 1 ( ⋅ ∣ s t + 1 ) a_{t+1} \sim\pi_{t+1}(\cdot\mid s_{t+1}) a t + 1 ∼ π t + 1 ( ⋅ ∣ s t + 1 ) for all t ≥ h t \ge h t ≥ h . In words, Q h π ( s , a ) Q_h^\pi(s,a) Q h π ( s , a ) quantifies the remaining expected reward of taking action a a a in state s s s at step h h h and then following policy π \pi π thereafter.
No discount factor is needed because the value function and Q-function are always finite over a finite horizon. As in the infinite-horizon setting, there always exists at least one deterministic policy that simultaneously maximizes the value function and the Q-function for all ( s , a , h ) ∈ S × A × [ H ] (s,a,h) \in\mathcal{S} \times\mathcal{A} \times[H] ( s , a , h ) ∈ S × A × [ H ] (Bertsekas, 2017). In light of this, we shall denote by π ⋆ = { π h ⋆ } 1 ≤ h ≤ H \pi^\star= \{\pi_h^\star\}_{1\le h\le H} π ⋆ = { π h ⋆ } 1 ≤ h ≤ H an optimal deterministic policy throughout this monograph, which allows us to use π h ⋆ ( s ) \pi_h^\star(s) π h ⋆ ( s ) to indicate the corresponding optimal action chosen in state s s s at step h h h . The resulting optimal value function and optimal Q-function are denoted respectively by V ⋆ = { V h ⋆ } 1 ≤ h ≤ H V^\star= \{V_h^\star\}_{1\le h\le H} V ⋆ = { V h ⋆ } 1 ≤ h ≤ H and Q ⋆ = { Q h ⋆ } 1 ≤ h ≤ H Q^\star= \{Q_h^\star\}_{1\le h\le H} Q ⋆ = { Q h ⋆ } 1 ≤ h ≤ H :
∀ h ∈ [ H ] : V h ⋆ : = V h π ⋆ and Q h ⋆ : = Q h π ⋆ . (49) \forall h \in[H]:\qquad V_h^\star:= V_h^{\pi^\star}\quad\text{and}\quad Q_h^\star:= Q_h^{\pi^\star}. \tag*{(49)} ∀ h ∈ [ H ] : V h ⋆ := V h π ⋆ and Q h ⋆ := Q h π ⋆ . (49)
Furthermore, we introduce the occupancy distributions associated with policy π = { π h } 1 ≤ h ≤ H \pi= \{\pi_h\}_{1\le h\le H} π = { π h } 1 ≤ h ≤ H , describing how often states or state-action pairs are visited at a given step. Specifically, for any step 1 ≤ h ≤ H 1 \le h \le H 1 ≤ h ≤ H , define
d h π ( s ; ρ ) : = P ( s h = s ∣ s 1 ∼ ρ ; π ) , (50) d_h^\pi(s;\rho) := \mathbb{P}(s_h=s\mid s_1\sim\rho;\pi), \tag*{(50)} d h π ( s ; ρ ) := P ( s h = s ∣ s 1 ∼ ρ ; π ) , (50)
d h π ( s , a ; ρ ) : = P ( s h = s , a h = a ∣ s 1 ∼ ρ ; π ) = d h π ( s ; ρ ) π h ( a ∣ s ) , (51) d_h^\pi(s,a;\rho) := \mathbb{P}(s_h=s,a_h=a\mid s_1\sim\rho;\pi)=d_h^\pi(s;\rho)\pi_h(a\mid s), \tag*{(51)} d h π ( s , a ; ρ ) := P ( s h = s , a h = a ∣ s 1 ∼ ρ ; π ) = d h π ( s ; ρ ) π h ( a ∣ s ) , (51)
which are conditioned on the initial state distribution s 1 ∼ ρ s_1\sim\rho s 1 ∼ ρ and the event that all actions are sampled according to π \pi π . As a sanity check, note that at the beginning of the episode, the occupancy state distribution simply coincides with the initial state distribution, i.e.,
d 1 π ( s ; ρ ) = ρ ( s ) for any policy π and any state s ∈ S . (52) d_1^\pi(s;\rho)=\rho(s)\quad\text{for any policy }\pi\text{ and any state }s\in\mathcal{S}. \tag*{(52)} d 1 π ( s ; ρ ) = ρ ( s ) for any policy π and any state s ∈ S . (52)
For convenience, we introduce shorthand notation for the occupancy distributions induced by the optimal deterministic policy π ⋆ = { π h ⋆ } 1 ≤ h ≤ H \pi^\star= \{\pi_h^\star\}_{1\le h\le H} π ⋆ = { π h ⋆ } 1 ≤ h ≤ H :
d h ⋆ ( s ) : = d h π ⋆ ( s ; ρ ) , (53) d_h^\star(s):=d_h^{\pi^\star}(s;\rho), \tag*{(53)} d h ⋆ ( s ) := d h π ⋆ ( s ; ρ ) , (53)
d h ⋆ ( s , a ) : = d h π ⋆ ( s , a ; ρ ) = d h ⋆ ( s ) 1 { a = π h ⋆ ( s ) } (54) d_h^\star(s,a):=d_h^{\pi^\star}(s,a;\rho)=d_h^\star(s)\mathbb{1}\{a=\pi_h^\star(s)\} \tag*{(54)} d h ⋆ ( s , a ) := d h π ⋆ ( s , a ; ρ ) = d h ⋆ ( s ) 1 { a = π h ⋆ ( s )} (54)
for all ( s , a , h ) ∈ S × A × [ H ] (s,a,h)\in\mathcal{S}\times\mathcal{A}\times[H] ( s , a , h ) ∈ S × A × [ H ] , where the last identity holds given that policy π ⋆ \pi^\star π ⋆ is assumed to be deterministic.
Bellman equations and dynamic programming
Next, we present the Bellman equations for finite-horizon MDPs, which underlie the design of classical dynamic programming algorithms.
Denote the terminal value functions as V H + 1 π = 0 V_{H+1}^\pi=0 V H + 1 π = 0 and Q H + 1 π = 0 Q_{H+1}^\pi=0 Q H + 1 π = 0 for any policy π = { π h } 1 ≤ h ≤ H \pi=\{\pi_h\}_{1\le h\le H} π = { π h } 1 ≤ h ≤ H . For a given policy π \pi π , its associated value function and Q-function can be computed through backward recursion starting from the terminal states as follows: for each h = H , … , 1 h=H,\ldots,1 h = H , … , 1 ,
Q h π ( s , a ) = r h ( s , a ) + E s ′ ∼ P h ( ⋅ ∣ s , a ) [ V h + 1 π ( s ′ ) ] , (55) Q_h^\pi(s,a)=r_h(s,a)+\mathbb{E}_{s'\sim P_h(\cdot\mid s,a)}\left[V_{h+1}^\pi(s')\right], \tag*{(55)} Q h π ( s , a ) = r h ( s , a ) + E s ′ ∼ P h ( ⋅ ∣ s , a ) [ V h + 1 π ( s ′ ) ] , (55)
V h π ( s ) = E a ∼ π h ( ⋅ ∣ s ) [ Q h π ( s , a ) ] (56) V_h^\pi(s)=\mathbb{E}_{a\sim\pi_h(\cdot\mid s)}\left[Q_h^\pi(s,a)\right] \tag*{(56)} V h π ( s ) = E a ∼ π h ( ⋅ ∣ s ) [ Q h π ( s , a ) ] (56)
for every ( s , a ) ∈ S × A (s,a)\in\mathcal{S}\times\mathcal{A} ( s , a ) ∈ S × A . Intuitively, at each step h h h , the Q-value consists of two components: the immediate reward received at that step, and the expected remaining value of proceeding according to π \pi π from the next state at step h + 1 h+1 h + 1 . Analogously, given that V H + 1 ⋆ = 0 V_{H+1}^\star=0 V H + 1 ⋆ = 0 and Q H + 1 ⋆ = 0 Q_{H+1}^\star=0 Q H + 1 ⋆ = 0 , the Bellman optimality equations assert that: for every h = H , … , 1 h=H,\ldots,1 h = H , … , 1 ,
Q h ⋆ ( s , a ) = r h ( s , a ) + E s ′ ∼ P h ( ⋅ ∣ s , a ) [ V h + 1 ⋆ ( s ′ ) ] , (57) Q_h^\star(s,a)=r_h(s,a)+\mathbb{E}_{s'\sim P_h(\cdot\mid s,a)}\left[V_{h+1}^\star(s')\right], \tag*{(57)} Q h ⋆ ( s , a ) = r h ( s , a ) + E s ′ ∼ P h ( ⋅ ∣ s , a ) [ V h + 1 ⋆ ( s ′ ) ] , (57)
V h ⋆ ( s ) = max a ∈ A Q h ⋆ ( s , a ) (58) V_h^\star(s)=\max_{a\in A}Q_h^\star(s,a) \tag*{(58)} V h ⋆ ( s ) = a ∈ A max Q h ⋆ ( s , a ) (58)
for every ( s , a ) ∈ S × A (s,a)\in S\times A ( s , a ) ∈ S × A .
In summary, given full knowledge of the model, the above recursions — often referred to as value iteration (VI) — allow the finite-horizon value functions and Q-functions to be computed step by step via backward induction, starting from the terminal step and proceeding backward to h = 1 h=1 h = 1 . This procedure embodies the dynamic programming principle: optimal decisions at each step can be determined by reasoning backward from the end of the episode.
Vector and matrix notation
As in the discounted infinite-horizon setting, the following vector and matrix notation will streamline the presentation (with a slight abuse of notation).
r h ∈ R S A r_h\in\mathbb{R}^{SA} r h ∈ R S A : a vector representing the reward function at step h h h , so that its ( s , a ) (s,a) ( s , a ) -th entry is the corresponding immediate reward at step h h h .
V h π , V h ⋆ ∈ R S V_h^\pi,V_h^\star\in\mathbb{R}^{S} V h π , V h ⋆ ∈ R S : vectors representing, respectively, the value function of policy π \pi π at step h h h (cf. (46) ) and the optimal value function at step h h h (cf. (49) ).
Q h π , Q h ⋆ ∈ R S A Q_h^\pi,Q_h^\star\in\mathbb{R}^{SA} Q h π , Q h ⋆ ∈ R S A : vectors representing, respectively, the Q-function of policy π \pi π at step h h h (cf. (48) ) and the optimal Q-function at step h h h (cf. (49) ).
P h ∈ R S A × S P_h\in\mathbb{R}^{SA\times S} P h ∈ R S A × S : a matrix representing the probability transition kernel P h P_h P h at step h h h , whose ( s , a ) (s,a) ( s , a ) -th row is the probability vector corresponding to P h ( ⋅ ∣ s , a ) P_h(\cdot\mid s,a) P h ( ⋅ ∣ s , a ) .
The Bellman equations (cf. (2.38) and (2.39)) can therefore be expressed compactly using the above vector and matrix notation as
Q h π = r h + P h V h + 1 π , 1 ≤ h ≤ H , (59) Q_h^\pi=r_h+P_hV_{h+1}^\pi,\qquad1\le h\le H, \tag*{(59)} Q h π = r h + P h V h + 1 π , 1 ≤ h ≤ H , (59)
Q h ⋆ = r h + P h V h + 1 ⋆ , 1 ≤ h ≤ H . (60) Q_h^\star=r_h+P_hV_{h+1}^\star,\qquad1\le h\le H. \tag*{(60)} Q h ⋆ = r h + P h V h + 1 ⋆ , 1 ≤ h ≤ H . (60)
In addition, we introduce the notation for variances. For any vector V = [ V i ] 1 ≤ i ≤ S ∈ R S V=[V_i]_{1\le i\le S}\in\mathbb{R}^{S} V = [ V i ] 1 ≤ i ≤ S ∈ R S , we define the scalar quantity Var P s , a , h ( V ) ∈ R \operatorname{Var}_{P_{s,a,h}}(V)\in\mathbb{R} Var P s , a , h ( V ) ∈ R and the vector Var P h ( V ) ∈ R S A \operatorname{Var}_{P_h}(V)\in\mathbb{R}^{SA} Var P h ( V ) ∈ R S A as
Var P s , a , h ( V ) = ∑ s ′ ∈ S P h ( s ′ ∣ s , a ) ( V s ′ ) 2 − ( ∑ s ′ ∈ S P h ( s ′ ∣ s , a ) V s ′ ) 2 , (61) \operatorname{Var}_{P_{s,a,h}}(V)=\sum_{s'\in S}P_h(s'\mid s,a)(V_{s'})^2-\left(\sum_{s'\in S}P_h(s'\mid s,a)V_{s'}\right)^2, \tag*{(61)} Var P s , a , h ( V ) = s ′ ∈ S ∑ P h ( s ′ ∣ s , a ) ( V s ′ ) 2 − ( s ′ ∈ S ∑ P h ( s ′ ∣ s , a ) V s ′ ) 2 , (61)
Var P h ( V ) = [ Var P s , a , h ( V ) ] ( s , a ) ∈ S × A = P h ( V ∘ V ) − ( P h V ) ∘ ( P h V ) , (62) \begin{aligned}
\operatorname{Var}_{P_h}(V)&=\left[\operatorname{Var}_{P_{s,a,h}}(V)\right]_{(s,a)\in S\times A}\\
&=P_h(V\circ V)-(P_hV)\circ(P_hV), \tag*{(62)}
\end{aligned} Var P h ( V ) = [ Var P s , a , h ( V ) ] ( s , a ) ∈ S × A = P h ( V ∘ V ) − ( P h V ) ∘ ( P h V ) , (62)
representing the variance of V ( s ′ ) V(s') V ( s ′ ) when s ′ s' s ′ is drawn according to either a single distribution P s , a , h P_{s,a,h} P s , a , h or, collectively, all rows of the transition matrix P h P_h P h .
Model-based vs. model-free approaches
Thus far, our discussion has centered on the fundamentals of MDPs and on dynamic programming algorithms given complete model knowledge. While classical dynamic programming provides powerful and efficient methods for identifying optimal policies when the model is fully specified, reinforcement learning often has to contend with more challenging scenarios in which the model is unknown a priori and can only be learned from data — either pre-collected or gathered in real time. The agent must therefore make decisions under uncertainty.
At a high level, there are at least two principal approaches in RL, which we describe below and illustrate in Figure 2.4 .
Figure 2.4. The model-based approach vs. the model-free approach. A key distinction is that model-free methods dispense with explicit model estimation and instead learn the value and policy of interest directly without the aid of an estimated model.
Model-based RL , which decouples the learning process into two stages: a model estimation stage that estimates the transition kernel and rewards using available data, and a subsequent planning stage that applies planning algorithms on the learned model to compute near-optimal policies and values. Given the use of the meta-task of model estimation, it requires maintaining and updating the learned model from collected data.
Model-free RL , which circumvents the model estimation stage and seeks to learn the optimal values and policies directly. Prominent examples include Q-learning (Watkins and Dayan, 1992) and its many variants. Compared to the model-based counterpart, it holds the promise of lower space complexity, as it avoids the need to store and update a full description of the model. Following the convention in RL theory (e.g., Strehl et al. (2006) and Jin et al. (2018)), we classify an algorithm as model-free if its space complexity is substantially smaller than that required to represent the full model, irrespective of the number of samples collected.
In this work, we investigate both model-based and model-free approaches, with particular emphasis on their statistical efficiency. As the subsequent chapters will illustrate, the model-based approach is often able to achieve the best possible sample complexity, albeit at the expense of higher memory usage and computational overhead.
Concentration inequalities
Given that RL relies on learning from data, non-asymptotic concentration inequalities developed in the recent literature on high-dimensional probability and statistics ((Boucheron et al., 2013; Vershynin, 2018; Wainwright, 2019a)) play a central role in establishing tight sample complexity bounds. For this reason, we briefly review a few commonly used concentration results.
Consider a probability transition kernel P : S × A → Δ ( S ) P:\mathcal{S}\times\mathcal{A}\to\Delta(\mathcal{S}) P : S × A → Δ ( S ) . Suppose that for each state-action pair ( s , a ) (s,a) ( s , a ) , we have access to N N N independent samples drawn from P ( ⋅ ∣ s , a ) P(\cdot\mid s,a) P ( ⋅ ∣ s , a ) , namely,
s ( i ) ( s , a ) ∼ i . i . d . P ( ⋅ ∣ s , a ) , i = 1 , … , N . s^{(i)}(s,a)\mathrel{\overset{\mathrm{i.i.d.}}{\sim}}P(\cdot\mid s,a),\qquad i=1,\ldots,N. s ( i ) ( s , a ) ∼ i.i.d. P ( ⋅ ∣ s , a ) , i = 1 , … , N .
From these samples, we can naturally form an empirical estimate P ^ ( ⋅ ∣ s , a ) \widehat{P}(\cdot\mid s,a) P ( ⋅ ∣ s , a ) of the transition probability P ( ⋅ ∣ s , a ) P(\cdot\mid s,a) P ( ⋅ ∣ s , a ) as follows:
P ^ ( s ′ ∣ s , a ) = 1 N ∑ i = 1 N 1 { s ( i ) ( s , a ) = s ′ } for all s ′ ∈ S . (63) \widehat{P}(s'\mid s,a)=\frac{1}{N}\sum_{i=1}^{N}\mathbf{1}\{s^{(i)}(s,a)=s'\}\qquad\text{for all }s'\in\mathcal{S}. \tag*{(63)} P ( s ′ ∣ s , a ) = N 1 i = 1 ∑ N 1 { s ( i ) ( s , a ) = s ′ } for all s ′ ∈ S . (63)
Taking these estimates together yields an estimate P ^ ∈ R S A × S \widehat{P}\in\mathbb{R}^{SA\times S} P ∈ R S A × S of the probability transition matrix P P P , namely, the ( s , a ) (s,a) ( s , a ) -th row of P ^ \widehat{P} P is given by
P ^ ( s , a ) , s ′ = P ^ ( s ′ ∣ s , a ) for all ( s , a , s ′ ) ∈ S × A × S . (64) \widehat{P}_{(s,a),s'}=\widehat{P}(s'\mid s,a)\qquad\text{for all }(s,a,s')\in\mathcal{S}\times\mathcal{A}\times\mathcal{S}. \tag*{(64)} P ( s , a ) , s ′ = P ( s ′ ∣ s , a ) for all ( s , a , s ′ ) ∈ S × A × S . (64)
The following concentration bound is a direct consequence of the classical Hoeffding inequality (Vershynin, 2018) combined with the union bound.
Lemma 2.4 (Hoeffding inequality). Consider a given state-action pair ( s , a ) (s,a) ( s , a ) and a finite set of vectors V ⊂ R + S \mathcal{V}\subset\mathbb{R}_{+}^{S} V ⊂ R + S . Fix any 0 < δ < 1 0<\delta<1 0 < δ < 1 . If V \mathcal{V} V is independent of P ^ s , a \widehat{P}_{s,a} P s , a , then with probability greater than 1 − δ 1-\delta 1 − δ ,
∣ ( P ^ s , a − P s , a ) V ∣ ≤ ∥ V ∥ ∞ 2 log ( 2 ∣ V ∣ δ ) 2 N (65) \left|(\widehat{P}_{s,a}-P_{s,a})V\right|\leq\sqrt{\frac{\|V\|_{\infty}^{2}\log\left(\frac{2|\mathcal{V}|}{\delta}\right)}{2N}} \tag*{(65)} ( P s , a − P s , a ) V ≤ 2 N ∥ V ∥ ∞ 2 log ( δ 2∣ V ∣ ) (65) holds simultaneously for all V ∈ V V\in\mathcal{V} V ∈ V .
In particular, the Hoeffding bound (65) depends on V V V only through the maximum magnitude of its entries, without relying on any finer structural properties of V V V , such as its variance.
Despite its simplicity and broad applicability, Hoeffding’s inequality is often conservative, especially when the underlying standard deviation is substantially smaller than the worst-case range. In such situations, Bernstein-type inequalities can provide significantly sharper concentration bounds by explicitly incorporating variance information. To this end, we record below concentration inequalities obtained directly from the Bernstein inequality (Vershynin, 2018) in conjunction with a union bound; see also Agarwal et al. (2020a, Lemma 6).
Lemma 2.5 (Bernstein inequality). Consider a given state-action pair ( s , a ) (s,a) ( s , a ) and a finite set of vectors V ⊂ R + S \mathcal{V}\subset\mathbb{R}_{+}^{S} V ⊂ R + S . Fix any 0 < δ < 1 0<\delta<1 0 < δ < 1 . If V \mathcal{V} V is independent of P ^ s , a \widehat{P}_{s,a} P s , a , then with probability greater than 1 − δ 1-\delta 1 − δ ,
∣ ( P ^ s , a − P s , a ) V ∣ ≤ 2 log ( 4 ∣ V ∣ δ ) N Var P s , a ( V ) + 2 ∥ V ∥ ∞ log ( 4 ∣ V ∣ δ ) 3 N (66) \left|(\widehat{P}_{s,a}-P_{s,a})V\right|\leq\sqrt{\frac{2\log\left(\frac{4|\mathcal{V}|}{\delta}\right)}{N}}\sqrt{\operatorname{Var}_{P_{s,a}}(V)}+\frac{2\|V\|_{\infty}\log\left(\frac{4|\mathcal{V}|}{\delta}\right)}{3N} \tag*{(66)} ( P s , a − P s , a ) V ≤ N 2 log ( δ 4∣ V ∣ ) Var P s , a ( V ) + 3 N 2∥ V ∥ ∞ log ( δ 4∣ V ∣ ) (66) holds simultaneously for all V ∈ V V\in\mathcal{V} V ∈ V . Moreover, if V \mathcal{V} V is statistically independent of P ^ \widehat{P} P , then with probability greater than 1 − δ 1-\delta 1 − δ , one has
∣ ( P ^ − P ) V ∣ ≤ 2 log ( 4 S A ∣ V ∣ δ ) N Var P ( V ) + 2 ∥ V ∥ ∞ log ( 4 S A ∣ V ∣ δ ) 3 N 1 S A . (67) \left|(\widehat{P}-P)V\right|\leq\sqrt{\frac{2\log\left(\frac{4SA|\mathcal{V}|}{\delta}\right)}{N}}\sqrt{\operatorname{Var}_{P}(V)}+\frac{2\|V\|_{\infty}\log\left(\frac{4SA|\mathcal{V}|}{\delta}\right)}{3N}\mathbf{1}_{SA}. \tag*{(67)} ( P − P ) V ≤ N 2 log ( δ 4 S A ∣ V ∣ ) Var P ( V ) + 3 N 2∥ V ∥ ∞ log ( δ 4 S A ∣ V ∣ ) 1 S A . (67)
Here, we recall the definitions of Var P s , a ( V ) \operatorname{Var}_{P_{s,a}}(V) Var P s , a ( V ) and Var P ( V ) \operatorname{Var}_{P}(V) Var P ( V ) in (30) . We use 1 S A \mathbf{1}_{SA} 1 S A to denote the all-one vector, and for any nonnegative vector v v v , v \sqrt{v} v denotes the entrywise square root.
Concentration inequalities
In words, the Bernstein-style inequalities in Lemma 2.5 reveal that the uncertainty of P ^ s , a V \widehat{P}_{s,a}V P s , a V is governed by both its variance and the largest entrywise magnitude of V V V . When the sample size is large enough, the variance term typically dominates, a phenomenon that will become evident in subsequent chapters.
A technical challenge arises when Bernstein’s inequality is used to construct confidence intervals for quantities such as P s , a V P_{s,a}V P s , a V based on the empirical estimate P ^ s , a V \widehat{P}_{s,a}V P s , a V : the variance term Var P s , a ( V ) \operatorname{Var}_{P_{s,a}}(V) Var P s , a ( V ) depends on the true transition kernel P s , a P_{s,a} P s , a , which is unavailable in practice. To circumvent this, it is useful to resort to the following empirical Bernstein inequality — quoted from Maurer and Pontil (2009, Theorem 4) — which replaces the population variance with its empirical estimate, making the confidence interval computable from collected data.
Lemma 2.6 (Empirical Bernstein inequality). Consider a given state-action pair ( s , a ) (s,a) ( s , a ) and a finite set of vectors V ⊂ R + S \mathcal{V} \subset\mathbb{R}_{+}^{S} V ⊂ R + S . Fix any 0 < δ < 1 0 < \delta< 1 0 < δ < 1 , and suppose that N ≥ 2 N \ge2 N ≥ 2 . If V \mathcal{V} V is independent of P ^ s , a \widehat{P}_{s,a} P s , a , then with probability greater than 1 − δ 1-\delta 1 − δ ,
∣ ( P ^ s , a − P s , a ) V ∣ ≤ 2 log ( 4 ∣ V ∣ δ ) N − 1 Var P ^ s , a ( V ) + 7 ∥ V ∥ ∞ log ( 4 ∣ V ∣ δ ) 3 ( N − 1 ) \left|(\widehat{P}_{s,a}-P_{s,a})V\right| \le\sqrt{\frac{2\log\left(\frac{4|\mathcal{V}|}{\delta}\right)}{N-1}}\sqrt{\operatorname{Var}_{\widehat{P}_{s,a}}(V)}+\frac{7\|V\|_{\infty}\log\left(\frac{4|\mathcal{V}|}{\delta}\right)}{3(N-1)} ( P s , a − P s , a ) V ≤ N − 1 2 log ( δ 4∣ V ∣ ) Var P s , a ( V ) + 3 ( N − 1 ) 7∥ V ∥ ∞ log ( δ 4∣ V ∣ ) holds simultaneously for all V ∈ V V \in\mathcal{V} V ∈ V .
In addition, when coping with sums of i.i.d. Bernoulli variables, the following exponential tail bound becomes useful, which follows from standard Chernoff-type arguments.
Lemma 2.7. Let Z i ∼ i . i . d . Bern ( p ) Z_i \mathrel{\overset{\mathrm{i.i.d.}}{\sim}} \operatorname{Bern}(p) Z i ∼ i.i.d. Bern ( p ) ( 1 ≤ i ≤ n ) (1 \le i \le n) ( 1 ≤ i ≤ n ) be a set of independent Bernoulli random variables with mean 0 < p ≤ 1 0 < p \le1 0 < p ≤ 1 . Then for any a > 0 a > 0 a > 0 , one has
P ( ∑ i = 1 n ( Z i − E [ Z i ] ) < − a n ) ≤ exp ( − a 2 n 2 p ) . \mathbb{P}\left(\sum_{i=1}^{n}(Z_i-\mathbb{E}[Z_i])<-an\right)\le\exp\left(-\frac{a^2n}{2p}\right). P ( i = 1 ∑ n ( Z i − E [ Z i ]) < − an ) ≤ exp ( − 2 p a 2 n ) . In addition, for any τ ≥ 2 p n \tau\ge2pn τ ≥ 2 p n , one has
P ( ∑ i = 1 n Z i > τ ) ≤ 2 exp ( − 0.19 τ ) . \mathbb{P}\left(\sum_{i=1}^{n}Z_i>\tau\right)\le2\exp(-0.19\tau). P ( i = 1 ∑ n Z i > τ ) ≤ 2 exp ( − 0.19 τ ) .
Proof. The first inequality can be found in Alon and Spencer (2008, Theorem A.1.13), while the second inequality is an immediate consequence of Alon and Spencer (2008, Corollary A.1.14) with ϵ ≥ 1 \epsilon\ge1 ϵ ≥ 1 therein.
Moving beyond independent data, concentration inequalities for martingales are often needed to handle sequentially collected observations. Among the most useful are the Azuma-Hoeffding inequality (Azuma, 1967; Dubhashi and Panconesi, 2009), a martingale counterpart of Hoeffding’s inequality, and Freedman’s inequality (Freedman, 1975; Tropp, 2011), which serves as a martingale analogue of Bernstein’s inequality. Below, we record the standard Azuma-Hoeffding inequality and a user-friendly version of Freedman’s inequality (see Li et al. (2024a, Theorem EC.1)).
Lemma 2.8 (Azuma-Hoeffding inequality). Consider any 0 < δ < 1 0 < \delta< 1 0 < δ < 1 . Suppose that Y n = ∑ k = 1 n X k ∈ R Y_n=\sum_{k=1}^{n}X_k\in\mathbb{R} Y n = ∑ k = 1 n X k ∈ R , where { X k } \{X_k\} { X k } is a real-valued scalar sequence obeying
∣ X k ∣ ≤ B k and E [ X k ∣ { X j } j : k < k ] = 0 for all k ≥ 1 |X_k|\le B_k \qquad\text{and} \qquad\mathbb{E}\left[X_k\mid\{X_j\}_{j:k<k}\right]=0 \qquad\text{for all } k\ge1 ∣ X k ∣ ≤ B k and E [ X k ∣ { X j } j : k < k ] = 0 for all k ≥ 1
for some deterministic quantities { B k } \{B_k\} { B k } . Then with probability exceeding 1 − δ 1-\delta 1 − δ , one has
∣ Y n ∣ ≤ 2 ∑ k = 1 n B k 2 log 2 δ . (68) |Y_n| \le\sqrt{2\sum_{k=1}^{n} B_k^2 \log\frac{2}{\delta}}. \tag*{(68)} ∣ Y n ∣ ≤ 2 k = 1 ∑ n B k 2 log δ 2 . (68)
In brief, the Azuma–Hoeffding inequality shows that the fluctuations of a martingale sum can be controlled by the sum of the squared magnitudes of its martingale difference terms. This can be viewed as a direct extension of the Hoeffding inequality (Boucheron et al., 2013) from independent data to weakly dependent martingale sequences.
Lemma 2.9 (Freedman inequality). Consider any 0 < δ < 1 0<\delta<1 0 < δ < 1 . Suppose that Y n = ∑ k = 1 n X k ∈ R Y_n=\sum_{k=1}^{n}X_k\in\mathbb{R} Y n = ∑ k = 1 n X k ∈ R , where { X k } \{X_k\} { X k } is a real-valued scalar sequence obeying
∣ X k ∣ ≤ B and E [ X k ∣ { X j } j : j < k ] = 0 for all k ≥ 1. |X_k|\le B \qquad\text{and} \qquad\mathbb{E}\left[X_k\mid\{X_j\}_{j:j<k}\right]=0 \qquad\text{for all } k\ge1. ∣ X k ∣ ≤ B and E [ X k ∣ { X j } j : j < k ] = 0 for all k ≥ 1. Define
W n : = ∑ k = 1 n E [ X k 2 ∣ { X j } j : j < k ] , W_n:=\sum_{k=1}^{n}\mathbb{E}\left[X_k^2\mid\{X_j\}_{j:j<k}\right], W n := k = 1 ∑ n E [ X k 2 ∣ { X j } j : j < k ] , and suppose that W n ≤ σ 2 W_n\le\sigma^2 W n ≤ σ 2 holds deterministically. Then for any positive integer K 0 ≥ 1 K_0\ge1 K 0 ≥ 1 , with probability at least 1 − δ 1-\delta 1 − δ one has
∣ Y n ∣ ≤ 8 max { W n , σ 2 2 K 0 } log 2 K 0 δ + 4 3 B log 2 K 0 δ . (69) |Y_n|\le\sqrt{8\max\left\{W_n,\frac{\sigma^2}{2K_0}\right\}\log\frac{2K_0}{\delta}}+\frac{4}{3}B\log\frac{2K_0}{\delta}. \tag*{(69)} ∣ Y n ∣ ≤ 8 max { W n , 2 K 0 σ 2 } log δ 2 K 0 + 3 4 B log δ 2 K 0 . (69)
Informally, the Freedman inequality in Lemma 2.9 asserts that the deviation of a martingale sum is controlled by the cumulative conditional variance (i.e., W n W_n W n ) and a uniform upper bound on the magnitude of each summand (i.e., B B B ). The resulting concentration bound closely resembles the Bernstein inequality, with a key distinction that the total variance is now replaced by the cumulative conditional variance of the martingale sequence.
Minimax lower bounds for multi-hypothesis testing
To assess the statistical optimality of an RL algorithm, one typically derives a statistical lower bound and then determines whether the algorithm attains it. Arguments for statistical or information-theoretic lower bounds often differ substantially from those used for upper bounds. While lower-bound techniques constitute a central topic in statistics and information theory (see, e.g., Tsybakov (2009) and Polyanskiy and Wu (2025)), a comprehensive treatment lies beyond the scope of this monograph. Instead, we single out a simple yet remarkably powerful and versatile tool that appears frequently in lower-bound analyses: Fano’s lemma. Rooted in classical information theory and originally introduced by Robert Fano, this lemma provides a principled way to quantify the intrinsic difficulty of statistical problems through information-theoretic measures.
More specifically, suppose that there are M + 1 M+1 M + 1 probability distributions P 0 , P 1 , … , P M P_0,P_1,\ldots,P_M P 0 , P 1 , … , P M on a common measurable space ( X , A ) (\mathcal{X},\mathcal{A}) ( X , A ) . Consider any test ψ \psi ψ that maps from X \mathcal{X} X to { 0 , 1 , … , M } \{0,1,\ldots,M\} { 0 , 1 , … , M } , with the goal of identifying the index of the distribution generating the observed samples. The performance of such a test can be measured by its error probability under each hypothesis, namely, P i ( ψ ≠ i ) P_i(\psi\ne i) P i ( ψ = i ) for i = 0 , … , M i=0,\ldots,M i = 0 , … , M . The minimax probability of error is then defined as
p e : = inf ψ max 0 ≤ i ≤ M P i ( ψ ≠ i ) . (70) p_e:=\inf_{\psi}\max_{0\le i\le M}P_i(\psi\ne i). \tag*{(70)} p e := ψ inf 0 ≤ i ≤ M max P i ( ψ = i ) . (70)
with the infimum taken over all possible tests. In words, p e p_e p e captures the smallest achievable worst-case worst-case error probability across all hypotheses. It is also useful to define the average probability of error as follows:
p ˉ e : = inf ψ 1 M + 1 ∑ i = 0 M P i ( ψ ≠ i ) , (71) \bar{p}_e := \inf_{\psi} \frac{1}{M+1}\sum_{i=0}^{M} P_i(\psi\ne i), \tag*{(71)} p ˉ e := ψ inf M + 1 1 i = 0 ∑ M P i ( ψ = i ) , (71)
which measures the error probability averaged over all M + 1 M+1 M + 1 hypotheses. The Fano lemma provides a fundamental lower bound on the probability of error for testing among M + 1 M+1 M + 1 hypotheses; the version below follows from a straightforward adaptation of the results in Tsybakov (2009, Section 2.7.1).
Lemma 2.10 (Fano’s lemma). Let P 0 , P 1 , … , P M P_0, P_1, \ldots, P_M P 0 , P 1 , … , P M be probability distributions on a common measurable space. Recall that p e p_e p e (resp. p ˉ e \bar{p}_e p ˉ e ) is the minimax (resp. average) probability of error defined in (70) (resp. (71) ).
(i) Suppose that M = 1 M=1 M = 1 . If KL ( P 1 ∥ P 0 ) ≤ β < ∞ \operatorname{KL}(P_1 \mathbin\Vert P_0) \le\beta< \infty KL ( P 1 ∥ P 0 ) ≤ β < ∞ , then
p e ≥ p ˉ e ≥ max { 1 4 exp ( − β ) , 1 − β / 2 2 } . (72) p_e \ge\bar{p}_e \ge\max\left\{\frac{1}{4}\exp(-\beta), \frac{1-\sqrt{\beta/2}}{2}\right\}. \tag*{(72)} p e ≥ p ˉ e ≥ max { 4 1 exp ( − β ) , 2 1 − β /2 } . (72) (ii) Suppose that M ≥ 2 M \ge2 M ≥ 2 . If
1 M + 1 ∑ j = 1 M KL ( P j ∥ P 0 ) ≤ β log M (73) \frac{1}{M+1}\sum_{j=1}^{M}\operatorname{KL}(P_j \mathbin\Vert P_0) \le\beta\log M \tag*{(73)} M + 1 1 j = 1 ∑ M KL ( P j ∥ P 0 ) ≤ β log M (73) for some 0 < β < 1 0<\beta<1 0 < β < 1 , then
p e ≥ p ˉ e ≥ log ( M + 1 ) − log 2 log M − β . (74) p_e \ge\bar{p}_e \ge\frac{\log(M+1)-\log2}{\log M}-\beta. \tag*{(74)} p e ≥ p ˉ e ≥ log M log ( M + 1 ) − log 2 − β . (74)
In some applications, it might be more convenient to control KL ( P 0 ∥ P j ) \operatorname{KL}(P_0 \mathbin\Vert P_j) KL ( P 0 ∥ P j ) , with the two distributions appearing in the opposite order from that in Part (ii) of Lemma 2.10 . The following bound, which follows from Pinsker’s inequality, handles this situation directly.
Lemma 2.11. Let P 0 , P 1 , … , P M P_0, P_1, \ldots, P_M P 0 , P 1 , … , P M be probability distributions on a common measurable space. Then
p e ≥ p ˉ e ≥ 1 − 1 M + 1 M 2 ( M + 1 ) 2 ∑ j = 1 M KL ( P 0 ∥ P j ) . (75) p_e \ge\bar{p}_e \ge1-\frac{1}{M+1}\sqrt{\frac{M}{2(M+1)^2}\sum_{j=1}^{M}\operatorname{KL}(P_0 \mathbin\Vert P_j)}. \tag*{(75)} p e ≥ p ˉ e ≥ 1 − M + 1 1 2 ( M + 1 ) 2 M j = 1 ∑ M KL ( P 0 ∥ P j ) . (75)
Proof of Lemma 2.11 . Consider any statistical test ψ \psi ψ taking values in { 0 , 1 , … , M } \{0,1,\ldots,M\} { 0 , 1 , … , M } , and define the event A j : = { ψ = j } A_j:=\{\psi=j\} A j := { ψ = j } . By the definition of the TV distance,
P j ( A j ) − P 0 ( A j ) ≤ ∣ P j ( A j ) − P 0 ( A j ) ∣ ≤ sup A ∣ P j ( A ) − P 0 ( A ) ∣ = TV ( P 0 , P j ) . P_j(A_j)-P_0(A_j)\le\lvert P_j(A_j)-P_0(A_j)\rvert\le\sup_A\lvert P_j(A)-P_0(A)\rvert=\operatorname{TV}(P_0,P_j). P j ( A j ) − P 0 ( A j ) ≤ ∣ P j ( A j ) − P 0 ( A j )∣ ≤ A sup ∣ P j ( A ) − P 0 ( A )∣ = TV ( P 0 , P j ) .
Consequently, for every 1 ≤ j ≤ M 1\le j\le M 1 ≤ j ≤ M ,
P j ( ψ = j ) ≤ P 0 ( ψ = j ) + TV ( P 0 , P j ) . P_j(\psi=j)\le P_0(\psi=j)+\operatorname{TV}(P_0,P_j). P j ( ψ = j ) ≤ P 0 ( ψ = j ) + TV ( P 0 , P j ) .
Summing over j j j leads to
∑ j = 0 M P j ( ψ = j ) ≤ P 0 ( ψ = 0 ) + ∑ j = 1 M P 0 ( ψ = j ) + ∑ j = 1 M TV ( P 0 , P j ) = 1 + ∑ j = 1 M TV ( P 0 , P j ) \begin{aligned}
\sum_{j=0}^{M} P_j(\psi= j) \le P_0(\psi= 0) + \sum_{j=1}^{M} P_0(\psi= j) + \sum_{j=1}^{M} \operatorname{TV}(P_0, P_j) \\
&= 1 + \sum_{j=1}^{M} \operatorname{TV}(P_0, P_j)
\end{aligned} j = 0 ∑ M P j ( ψ = j ) ≤ P 0 ( ψ = 0 ) + j = 1 ∑ M P 0 ( ψ = j ) + j = 1 ∑ M TV ( P 0 , P j ) = 1 + j = 1 ∑ M TV ( P 0 , P j )
≤ 1 + ∑ j = 1 M 1 2 KL ( P 0 ∥ P j ) \le1 + \sum_{j=1}^{M} \sqrt{\frac{1}{2}\operatorname{KL}(P_0 \mathbin\Vert P_j)} ≤ 1 + j = 1 ∑ M 2 1 KL ( P 0 ∥ P j )
≤ 1 + M 2 ∑ j = 1 M KL ( P 0 ∥ P j ) . \le1 + \sqrt{\frac{M}{2}\sum_{j=1}^{M} \operatorname{KL}(P_0 \mathbin\Vert P_j)}. ≤ 1 + 2 M j = 1 ∑ M KL ( P 0 ∥ P j ) .
where the penultimate inequality follows from Pinsker’s inequality (see (5) , or Tsybakov (2009, Lemma 2.5)) and the last from Cauchy–Schwarz. Consequently,
1 M + 1 ∑ j = 0 M P j ( ψ ≠ j ) = 1 − 1 M + 1 ∑ j = 0 M P j ( ψ = j ) \frac{1}{M+1}\sum_{j=0}^{M} P_j(\psi\ne j) = 1 - \frac{1}{M+1}\sum_{j=0}^{M} P_j(\psi= j) M + 1 1 j = 0 ∑ M P j ( ψ = j ) = 1 − M + 1 1 j = 0 ∑ M P j ( ψ = j )
≥ 1 − 1 M + 1 − M 2 ( M + 1 ) 2 ∑ j = 1 M KL ( P 0 ∥ P j ) . \ge1 - \frac{1}{M+1} - \sqrt{\frac{M}{2(M+1)^2}\sum_{j=1}^{M} \operatorname{KL}(P_0 \mathbin\Vert P_j)}. ≥ 1 − M + 1 1 − 2 ( M + 1 ) 2 M j = 1 ∑ M KL ( P 0 ∥ P j ) .
Taking the infimum over ψ \psi ψ proves the lower bound on p ‾ e \overline{p}_e p e , which in turn completes the proof. □
At a high level, Fano’s lemma and Lemma 2.11 assert that if the candidate distributions are close to one another in terms of KL divergence, then no test can reliably distinguish among them, and consequently, both the minimax error probability and the average error probability must remain bounded away from zero. In this way, Fano’s lemma unearths an intimate connection between the performance limit of any statistical test and the information-theoretic separation between distributions (as measured by KL divergence). A standard strategy for proving minimax lower bounds is therefore to construct a carefully chosen collection of nearby distributions and then control their KL divergences.
Given the central role of KL divergence in Fano’s lemma, we find it useful to record several of its key properties. We begin with the tensorization (i.e., additivity) property, which characterizes how KL divergence decomposes under independent data; see, e.g., Tsybakov (2009, Section 2.4). This simple property highlights a key advantage of KL divergence: when samples are independent, analysis of the overall KL divergence reduces to computing the KL divergence for individual samples.
Lemma 2.12. For two product distributions P = P 1 ⊗ ⋯ ⊗ P n P = P_1 \otimes\cdots\otimes P_n P = P 1 ⊗ ⋯ ⊗ P n and Q = Q 1 ⊗ ⋯ ⊗ Q n Q = Q_1 \otimes\cdots\otimes Q_n Q = Q 1 ⊗ ⋯ ⊗ Q n , the KL divergence satisfies
KL ( P ∥ Q ) = ∑ i = 1 n KL ( P i ∥ Q i ) . \operatorname{KL}(P \mathbin\Vert Q) = \sum_{i=1}^{n} \operatorname{KL}(P_i \mathbin\Vert Q_i). KL ( P ∥ Q ) = i = 1 ∑ n KL ( P i ∥ Q i ) .
Additionally, when P P P and Q Q Q are both Bernoulli distributions, we adopt a slight abuse of notation and write
KL ( p ∥ q ) : = p log p q + ( 1 − p ) log 1 − p 1 − q , (76) \operatorname{KL}(p \mathbin\Vert q) := p \log\frac{p}{q} + (1-p)\log\frac{1-p}{1-q}, \tag*{(76)} KL ( p ∥ q ) := p log q p + ( 1 − p ) log 1 − q 1 − p , (76)
to denote the KL divergence between Bern ( p ) \operatorname{Bern}(p) Bern ( p ) and Bern ( q ) \operatorname{Bern}(q) Bern ( q ) , where Bern ( p ) \operatorname{Bern}(p) Bern ( p ) represents the Bernoulli distribution with mean p p p . The following lemma establishes a quadratic upper bound on this divergence, which is sometimes more convenient to work with than the original expression.
Lemma 2.13. For any 0 < p , q < 1 0 < p,q < 1 0 < p , q < 1 ,
K L ( p ∥ q ) ≤ ( p − q ) 2 q ( 1 − q ) . \mathrm{KL}(p \mathbin{\|} q) \le\frac{(p-q)^2}{q(1-q)}. KL ( p ∥ q ) ≤ q ( 1 − q ) ( p − q ) 2 . Moreover, if 1 / 2 ≤ q < p < 1 1/2 \le q < p < 1 1/2 ≤ q < p < 1 , then
K L ( p ∥ q ) ≤ K L ( q ∥ p ) ≤ ( p − q ) 2 p ( 1 − p ) . \mathrm{KL}(p \mathbin{\|} q) \le\mathrm{KL}(q \mathbin{\|} p) \le\frac{(p-q)^2}{p(1-p)}. KL ( p ∥ q ) ≤ KL ( q ∥ p ) ≤ p ( 1 − p ) ( p − q ) 2 .
Proof. We begin with the first relation. It is known that the KL divergence can be upper bounded by the χ 2 \chi^2 χ 2 divergence (see Tsybakov (2009, Lemma 2.7)), namely,
K L ( p ∥ q ) ≤ χ 2 ( p ∥ q ) : = ( p − q ) 2 q + ( 1 − p − ( 1 − q ) ) 2 1 − q = ( p − q ) 2 q ( 1 − q ) . \mathrm{KL}(p \mathbin{\|} q) \le\chi^2(p \mathbin{\|} q) := \frac{(p-q)^2}{q} + \frac{(1-p-(1-q))^2}{1-q} = \frac{(p-q)^2}{q(1-q)}. KL ( p ∥ q ) ≤ χ 2 ( p ∥ q ) := q ( p − q ) 2 + 1 − q ( 1 − p − ( 1 − q ) ) 2 = q ( 1 − q ) ( p − q ) 2 .
We now move to the second relation. Let
a = p + q 2 ∈ [ 1 2 , 1 ] and b = p − q 2 ∈ [ 0 , 1 4 ] , a = \frac{p+q}{2} \in\left[\frac{1}{2},1\right] \quad\text{and}\quad b = \frac{p-q}{2} \in\left[0,\frac{1}{4}\right], a = 2 p + q ∈ [ 2 1 , 1 ] and b = 2 p − q ∈ [ 0 , 4 1 ] ,
so that p = a + b p=a+b p = a + b and q = a − b q=a-b q = a − b . By the definition in (76) and some elementary algebra, we obtain
K L ( p ∥ q ) − K L ( q ∥ p ) = ( p + q ) log p q + ( 2 − p − q ) log 1 − p 1 − q = 2 a log ( a + b a − b ) + 2 ( 1 − a ) log ( 1 − a − b 1 − a + b ) = : g ( a , b ) . \begin{aligned}
\mathrm{KL}(p \mathbin{\|} q) - \mathrm{KL}(q \mathbin{\|} p)
&= (p+q)\log\frac{p}{q} + (2-p-q)\log\frac{1-p}{1-q}\\
&= 2a\log\left(\frac{a+b}{a-b}\right) + 2(1-a)\log\left(\frac{1-a-b}{1-a+b}\right)\\
&=: g(a,b).
\end{aligned} KL ( p ∥ q ) − KL ( q ∥ p ) = ( p + q ) log q p + ( 2 − p − q ) log 1 − q 1 − p = 2 a log ( a − b a + b ) + 2 ( 1 − a ) log ( 1 − a + b 1 − a − b ) =: g ( a , b ) .
Taking the derivative of g ( a , b ) g(a,b) g ( a , b ) with respect to b b b gives
∂ g ( a , b ) ∂ b = 2 a { 1 a + b + 1 a − b } − 2 ( 1 − a ) { 1 1 − a + b + 1 1 − a − b } = f ( a ) − f ( 1 − a ) ≤ 0 , where f ( x ) : = 2 x x + b + 2 x x − b . \begin{aligned}
\frac{\partial g(a,b)}{\partial b}
&= 2a\left\{\frac{1}{a+b}+\frac{1}{a-b}\right\} - 2(1-a)\left\{\frac{1}{1-a+b}+\frac{1}{1-a-b}\right\}\\
&= f(a)-f(1-a)\le0,\quad\text{where}\quad f(x):=\frac{2x}{x+b}+\frac{2x}{x-b}.
\end{aligned} ∂ b ∂ g ( a , b ) = 2 a { a + b 1 + a − b 1 } − 2 ( 1 − a ) { 1 − a + b 1 + 1 − a − b 1 } = f ( a ) − f ( 1 − a ) ≤ 0 , where f ( x ) := x + b 2 x + x − b 2 x .
The last relation holds since f f f is decreasing on ( b , ∞ ) (b,\infty) ( b , ∞ ) and a ≥ 1 − a a \ge1-a a ≥ 1 − a . Thus, for each fixed a a a , the function g ( a , b ) g(a,b) g ( a , b ) is non-increasing in b ≥ 0 b \ge0 b ≥ 0 . Consequently,
K L ( p ∥ q ) − K L ( q ∥ p ) = g ( a , b ) ≤ g ( a , 0 ) = 0 , \mathrm{KL}(p \mathbin{\|} q) - \mathrm{KL}(q \mathbin{\|} p) = g(a,b) \le g(a,0) = 0, KL ( p ∥ q ) − KL ( q ∥ p ) = g ( a , b ) ≤ g ( a , 0 ) = 0 ,
which proves the desired claim.
Notes
While discounted infinite-horizon stationary MDPs and finite-horizon nonstationary MDPs are arguably the two most extensively studied models, several other MDP formulations have also received considerable attention in the RL literature. Examples include (i) infinite-horizon average-reward MDPs , where discounted cumulative rewards are replaced by long-run average rewards ((Puterman, 2014; Wan et al., 2021; Jin and Sidford, 2021)), and (ii) finite-horizon stationary MDPs , in which the transition kernels and reward functions are both time-homogeneous ((Ren et al., 2021; Li et al., 2022c; Zhang et al., 2022)). Each of these models bears similarities with the ones under exposition here, yet also brings its own distinctive challenges. For further details, we refer the interested reader to the aforementioned papers and the references therein.
For broader background, we refer the interested reader to Wainwright (2019a), Fan et al. (2020a), and Samworth and Shah (2026) for comprehensive treatments of high-dimensional statistics and to Vershynin (2018) and Boucheron et al. (2013) for concentration inequalities. The texts by Tsybakov (2009), Polyanskiy and Wu (2025), and El Gamal and Raginsky (2026) provide excellent introductions to minimax lower bounds based on information-theoretic techniques.
RL with a generative model: the model-based approach
In this chapter, we begin our study of sample complexity in RL by first examining an idealized sampling mechanism, commonly referred to as the generative model or simulator. While this sampling mechanism is a substantial simplification of practical scenarios, it suggests a clean and tractable framework that serves as a natural first step towards understanding the fundamental statistical limits of RL. Our focus in this chapter is on model-based RL with the generative model, with the discussion of model-free algorithms deferred to Chapter 4.
Generative model or simulator
Consider first the discounted infinite-horizon stationary MDP introduced in § 2.1 . Initially studied by Kearns and Singh (1999), a generative model—also referred to as a simulator—allows one to query any state-action pair and sample an independent state transition from the ground-truth transition kernel. More precisely, each query to the generative model with a pair ( s , a ) ∈ S × A (s,a) \in\mathcal{S} \times\mathcal{A} ( s , a ) ∈ S × A returns an independent sample:
s ( s , a ) ∼ i n d P ( ⋅ ∣ s , a ) . (77) s(s,a) \mathrel{\overset{\mathrm{ind}}{\sim}} P(\cdot\mid s,a). \tag*{(77)} s ( s , a ) ∼ ind P ( ⋅ ∣ s , a ) . (77)
Similarly, in the case of a finite-horizon nonstationary MDP, each call to the generative model with an arbitrary state-action-step tuple ( s , a , h ) ∈ S × A × [ H ] (s,a,h) \in\mathcal{S} \times\mathcal{A} \times[H] ( s , a , h ) ∈ S × A × [ H ] returns an independent sample:
s ( s , a , h ) ∼ i n d P h ( ⋅ ∣ s , a ) . (78) s(s,a,h) \mathrel{\overset{\mathrm{ind}}{\sim}} P_h(\cdot\mid s,a). \tag*{(78)} s ( s , a , h ) ∼ ind P h ( ⋅ ∣ s , a ) . (78)
Unless otherwise noted, we assume throughout that the reward function is fully known to the agent. This assumption can, in fact, be substantially relaxed without affecting the core insights underlying algorithm design and analysis.
Studying RL with a generative model offers several key advantages. To begin with, simulators have been developed in several applications to simulate complex, real-world environments, such as for autonomous driving (Shah et al., 2018), robotic systems (Kumar et al., 2019; Yang et al., 2024a), and game playing (Bellemare et al., 2013). Beyond their practical utility, generative models provide a clean abstraction: by allowing direct sampling from arbitrary state–action pairs, they effectively decouple learning from the need for exploration. This separation simplifies the problem and makes it easier to isolate the fundamental information bottlenecks inherent to the learning part. Overall, investigating RL under this idealistic sampling mechanism helps us distill core design principles, establish theoretical benchmarks, and understand the fundamental statistical limits of RL algorithms, which in turn guide us towards more practical and robust solutions.
Model-based policy evaluation
We begin with the fundamental task of policy evaluation , which seeks to estimate, based on the collected samples, the value function V π V^\pi V π of a given policy π \pi π . Our focus throughout this section is on ℓ ∞ \ell_\infty ℓ ∞ -accurate estimation: given a target accuracy level ε > 0 \varepsilon> 0 ε > 0 , the aim is to find an estimate V ^ π \widehat{V}^\pi V π that is ε \varepsilon ε -accurate uniformly over every state s s s , namely,
∀ s ∈ S : ∣ V ^ π ( s ) − V π ( s ) ∣ ≤ ε . \forall s \in\mathcal{S} : \qquad\left|\widehat{V}^\pi(s) - V^\pi(s)\right| \leq\varepsilon. ∀ s ∈ S : V π ( s ) − V π ( s ) ≤ ε .
Recall our assumption that all immediate rewards reside within [ 0 , 1 ] [0,1] [ 0 , 1 ] , and hence the value function falls within the range ( 0 , 1 1 − γ ] \left(0,\frac{1}{1-\gamma}\right] ( 0 , 1 − γ 1 ] . This implies that the range of the target accuracy level ε \varepsilon ε should be set to ε ∈ [ 0 , 1 1 − γ ] \varepsilon\in\left[0,\frac{1}{1-\gamma}\right] ε ∈ [ 0 , 1 − γ 1 ] .
A plug-in approach
In view of classical Bellman equations (cf. (2.12)), the value function of policy π \pi π satisfies (see (35) )
V π = ( I − γ P π ) − 1 r π , V^\pi= (I-\gamma P_\pi)^{-1}r_\pi, V π = ( I − γ P π ) − 1 r π ,
which depends upon the transition kernel P π P_\pi P π induced by π \pi π over the state space S \mathcal{S} S (see (28) ), the reward function r π r_\pi r π induced by π \pi π (see (29) ), and the discount factor γ \gamma γ . Since the only unknown component of the model in our setting is the transition kernel P π ∈ R S × S P_\pi\in\mathbb{R}^{S \times S} P π ∈ R S × S , a natural strategy for policy evaluation is the “plug-in” approach, which starts by estimating P π P_\pi P π , followed by computation of the value function associated with the learned transition kernel P ^ π \widehat{P}_\pi P π . This two-stage paradigm, which separates model estimation from value estimation, is dubbed as the model-based approach. More formally, the model-based policy evaluation algorithm in the presence of a generative model can be described as:
Model estimation. For each s ∈ S s \in\mathcal{S} s ∈ S , draw N N N independent samples from the generative model:
∀ s ∈ S : s ( i ) ( s ) ∼ i n d . P π ( ⋅ ∣ s ) , i = 1 , … , N \forall s \in\mathcal{S}: \qquad s^{(i)}(s) \mathrel{\mathop{\sim}^{\mathrm{ind.}}} P_\pi(\cdot\mid s), \qquad i=1,\ldots,N ∀ s ∈ S : s ( i ) ( s ) ∼ ind. P π ( ⋅ ∣ s ) , i = 1 , … , N
with P π ( ⋅ ∣ s ) P_\pi(\cdot\mid s) P π ( ⋅ ∣ s ) defined in (7) , and compute the empirical transition kernel from state s s s as:
P ^ π ( s ′ ∣ s ) = 1 N ∑ i = 1 N I { s ′ = s ( i ) ( s ) } for all s ′ ∈ S . (79) \widehat{P}_\pi(s' \mid s) = \frac{1}{N}\sum_{i=1}^{N}\mathbb{I}\left\{s'=s^{(i)}(s)\right\} \quad\text{for all }s'\in\mathcal{S}. \tag*{(79)} P π ( s ′ ∣ s ) = N 1 i = 1 ∑ N I { s ′ = s ( i ) ( s ) } for all s ′ ∈ S . (79)
In words, P ^ π ( s ′ ∣ s ) \widehat{P}_\pi(s' \mid s) P π ( s ′ ∣ s ) counts the empirical frequency of transitions from state s s s to state s ′ s' s ′ .
Value estimation. Compute the value function estimate as
V ^ π = ( I − γ P ^ π ) − 1 r π . (80) \widehat{V}^\pi= (I-\gamma\widehat{P}_\pi)^{-1}r_\pi. \tag*{(80)} V π = ( I − γ P π ) − 1 r π . (80)
It is self-evident that the total number of samples drawn in the above model-based algorithm is N t o t a l : = N S N^{\mathrm{total}} := NS N total := N S .
Algorithm 3.1: Model-based policy evaluation with a simulator
input: sample size per state N N N , discount factor γ \gamma γ , reward function r π r_\pi r π induced by policy π \pi π .
for s ∈ S s \in\mathcal{S} s ∈ S do /* model estimation based on samples. */
draw N N N independent samples s ( i ) ( s ) ∼ P π ( ⋅ ∣ s ) s^{(i)}(s) \sim P_\pi(\cdot\mid s) s ( i ) ( s ) ∼ P π ( ⋅ ∣ s ) (cf. (7) ).
compute P ^ π ( ⋅ ∣ s ) \widehat{P}_\pi(\cdot\mid s) P π ( ⋅ ∣ s ) based on (79) .
/* value estimation using the empirical model. */
output: V ^ π = ( I − γ P ^ π ) − 1 r π \widehat{V}^{\pi} = (I-\gamma\widehat{P}_{\pi})^{-1}r_{\pi} V π = ( I − γ P π ) − 1 r π .
Theoretical guarantees
Despite its conceptual and algorithmic simplicity, the model-based policy evaluation algorithm described in Algorithm 3.1 turns out to achieve a form of statistical optimality in the generative model setting; namely, no other algorithms can outperform its sample complexity in a minimax sense, except possibly up to logarithmic factors. To formalize this, we first present an upper bound on the sample complexity of Algorithm 3.1 in the theorem below, and defer the discussion of a matching lower bound to § 4.9 .
Theorem 3.1 (Li et al., 2020; Li et al., 2024c). For any 0 < ε ≤ 1 1 − γ 0 < \varepsilon\le\frac{1}{1-\gamma} 0 < ε ≤ 1 − γ 1 and any 0 < δ < 1 0 < \delta< 1 0 < δ < 1 , Algorithm 3.1 achieves
∀ s ∈ S : ∣ V ^ π ( s ) − V π ( s ) ∣ ≤ ε \forall s \in\mathcal{S}: \quad\left|\widehat{V}^{\pi}(s)-V^{\pi}(s)\right| \le\varepsilon ∀ s ∈ S : V π ( s ) − V π ( s ) ≤ ε with probability at least 1 − δ 1-\delta 1 − δ , provided that the sample size per state exceeds
N ≥ c 0 log ( S δ ( 1 − γ ) ) ( 1 − γ ) 3 ε 2 (81) N \ge\frac{c_0\log\left(\frac{S}{\delta(1-\gamma)}\right)}{(1-\gamma)^3\varepsilon^2} \tag*{(81)} N ≥ ( 1 − γ ) 3 ε 2 c 0 log ( δ ( 1 − γ ) S ) (81) for some sufficiently large constant c 0 > 0 c_0 > 0 c 0 > 0 .
In brief, Theorem 3.1 demonstrates that the sample complexity of the model-based policy evaluation algorithm in Algorithm 3.1 is no greater than
O ~ ( S ( 1 − γ ) 3 ε 2 ) \widetilde{O}\left(\frac{S}{(1-\gamma)^3\varepsilon^2}\right) O ( ( 1 − γ ) 3 ε 2 S )
for any meaningful target precision level ε \varepsilon ε . Notably, this requirement is often significantly lower than the sample size needed to learn the transition kernel P π P_\pi P π . For instance, when both 1 1 − γ \frac{1}{1-\gamma} 1 − γ 1 and 1 ε \frac{1}{\varepsilon} ε 1 are on the constant order, this sample complexity simplifies to O ~ ( S ) \widetilde{O}(S) O ( S ) , which can be S S S times smaller than what is required to learn P π ∈ R S × S P_\pi\in\mathbb{R}^{S \times S} P π ∈ R S × S —an object with O ( S 2 ) O(S^2) O ( S 2 ) degrees of freedom. Put differently, reliable value estimation becomes statistically feasible even in regimes where faithful model estimation is not. This unveils a crucial insight: while model-based algorithms conceptually involve model estimation as a meta-task, accomplishing this meta-task with high precision is not essential for policy evaluation to succeed.
Analysis: a first attempt
We begin with an initial attempt at bounding the value estimation error for Algorithm 3.1. This analysis strategy is straightforward yet yields a suboptimal bound.
Concretely, let us begin by deriving a simple characterization of the value estimation error. The Bellman equation (34) tells us that
r π = V π − γ P π V π = ( I − γ P π ) V π r_\pi= V^\pi- \gamma P_\pi V^\pi= (I-\gamma P_\pi)V^\pi r π = V π − γ P π V π = ( I − γ P π ) V π
which taken together with (80) implies that
V ^ π = ( I − γ P ^ π ) − 1 r π = ( I − γ P ^ π ) − 1 ( I − γ P π ) V π . (82) \widehat{V}^\pi= (I-\gamma\widehat{P}_\pi)^{-1}r_\pi= (I-\gamma\widehat{P}_\pi)^{-1}(I-\gamma P_\pi)V^\pi. \tag*{(82)} V π = ( I − γ P π ) − 1 r π = ( I − γ P π ) − 1 ( I − γ P π ) V π . (82)
Invoke (82) to decompose the value estimation error as
V ^ π − V π = ( I − γ P ^ π ) − 1 ( I − γ P π ) V π − ( I − γ P ^ π ) − 1 ( I − γ P ^ π ) V π = γ ( I − γ P ^ π ) − 1 ( P ^ π − P π ) V π . (83) \begin{aligned}
\widehat{V}^\pi- V^\pi
&= (I-\gamma\widehat{P}_\pi)^{-1}(I-\gamma P_\pi)V^\pi- (I-\gamma\widehat{P}_\pi)^{-1}(I-\gamma\widehat{P}_\pi)V^\pi\\
&= \gamma(I-\gamma\widehat{P}_\pi)^{-1}(\widehat{P}_\pi-P_\pi)V^\pi. \tag*{(83)}
\end{aligned} V π − V π = ( I − γ P π ) − 1 ( I − γ P π ) V π − ( I − γ P π ) − 1 ( I − γ P π ) V π = γ ( I − γ P π ) − 1 ( P π − P π ) V π . (83)
While controlling ( I − γ P ^ π ) − 1 (I-\gamma\widehat{P}_\pi)^{-1} ( I − γ P π ) − 1 can be somewhat challenging, the other term ( P ^ π − P π ) V π (\widehat{P}_\pi-P_\pi)V^\pi ( P π − P π ) V π can be readily handled by standard concentration inequalities. Specifically, applying the Bernstein-style inequality in Lemma 2.5 with A = 1 A=1 A = 1 and a single vector fixed V π V^\pi V π shows that, with probability at least 1 − δ 1-\delta 1 − δ ,
∣ ( P ^ π − P π ) V π ∣ ≤ C 0 ( Var P π ( V π ) log ( S δ ) N + ∥ V π ∥ ∞ log ( S δ ) N 1 S ) (84) \left|(\widehat{P}_\pi-P_\pi)V^\pi\right| \le C_0\left(\sqrt{\frac{\operatorname{Var}_{P_\pi}(V^\pi)\log\left(\frac{S}{\delta}\right)}{N}}+\frac{\lVert V^\pi\rVert_\infty\log\left(\frac{S}{\delta}\right)}{N}1_S\right) \tag*{(84)} ( P π − P π ) V π ≤ C 0 N Var P π ( V π ) log ( δ S ) + N ∥ V π ∥ ∞ log ( δ S ) 1 S (84)
for some universal constant C 0 > 0 C_0>0 C 0 > 0 , where 1 S ∈ R S 1_S\in\mathbb{R}^S 1 S ∈ R S is the all-one vector. Regarding the term ( I − γ P ^ π ) − 1 (I-\gamma\widehat{P}_\pi)^{-1} ( I − γ P π ) − 1 , it follows from Lemma 2.3 that ( I − γ P ^ π ) − 1 (I-\gamma\widehat{P}_\pi)^{-1} ( I − γ P π ) − 1 is a nonnegative matrix and obeys
( I − γ P ^ π ) − 1 1 S = 1 1 − γ 1 S . (85) (I-\gamma\widehat{P}_\pi)^{-1}1_S=\frac{1}{1-\gamma}1_S. \tag*{(85)} ( I − γ P π ) − 1 1 S = 1 − γ 1 1 S . (85)
Combining these properties with (83) and (84) shows that
∣ ( I − γ P ^ π ) − 1 ( P ^ π − P π ) V π ∣ ≤ ( I − γ P ^ π ) − 1 ∣ ( P ^ π − P π ) V π ∣ ≤ C 0 { ( I − γ P ^ π ) − 1 Var P π ( V π ) log ( S δ ) N + ∥ V π ∥ ∞ log ( S δ ) N ( I − γ P ^ π ) − 1 1 S } = C 0 { ( I − γ P ^ π ) − 1 Var P π ( V π ) log ( S δ ) N + ∥ V π ∥ ∞ log ( S δ ) ( 1 − γ ) N 1 S } . (86) \begin{aligned}
\left|(I-\gamma\widehat{P}_\pi)^{-1}(\widehat{P}_\pi-P_\pi)V^\pi\right|
&\le(I-\gamma\widehat{P}_\pi)^{-1}\left|(\widehat{P}_\pi-P_\pi)V^\pi\right| \\
&\le C_0\left\{\frac{(I-\gamma\widehat{P}_\pi)^{-1}\sqrt{\operatorname{Var}_{P_\pi}(V^\pi)\log\left(\frac{S}{\delta}\right)}}{\sqrt{N}}+\frac{\lVert V^\pi\rVert_\infty\log\left(\frac{S}{\delta}\right)}{N}(I-\gamma\widehat{P}_\pi)^{-1}1_S\right\} \\
&= C_0\left\{\frac{(I-\gamma\widehat{P}_\pi)^{-1}\sqrt{\operatorname{Var}_{P_\pi}(V^\pi)\log\left(\frac{S}{\delta}\right)}}{\sqrt{N}}+\frac{\lVert V^\pi\rVert_\infty\log\left(\frac{S}{\delta}\right)}{(1-\gamma)N}1_S\right\}. \tag*{(86)}
\end{aligned} ( I − γ P π ) − 1 ( P π − P π ) V π ≤ ( I − γ P π ) − 1 ( P π − P π ) V π ≤ C 0 ⎩ ⎨ ⎧ N ( I − γ P π ) − 1 Var P π ( V π ) log ( δ S ) + N ∥ V π ∥ ∞ log ( δ S ) ( I − γ P π ) − 1 1 S ⎭ ⎬ ⎫ = C 0 ⎩ ⎨ ⎧ N ( I − γ P π ) − 1 Var P π ( V π ) log ( δ S ) + ( 1 − γ ) N ∥ V π ∥ ∞ log ( δ S ) 1 S ⎭ ⎬ ⎫ . (86)
occurs with probability exceeding 1 − δ 1-\delta 1 − δ .
We then attempt to bound ( I − γ P ^ π ) − 1 (I-\gamma\widehat{P}_\pi)^{-1} ( I − γ P π ) − 1 and Var P π ( V π ) \sqrt{\operatorname{Var}_{P_\pi}(V^\pi)} Var P π ( V π ) separately:
∥ ( I − γ P ^ π ) − 1 Var P π ( V π ) ∥ ∞ ≤ ∥ ( I − γ P ^ π ) − 1 1 S ∥ ∞ ∥ Var P π ( V π ) ∥ ∞ = 1 1 − γ ∥ Var P π ( V π ) ∥ ∞ ≤ 1 ( 1 − γ ) 2 . (87) \left\|(I-\gamma\widehat{P}_\pi)^{-1}\sqrt{\operatorname{Var}_{P_\pi}(V^\pi)}\right\|_\infty
\le\left\|(I-\gamma\widehat{P}_\pi)^{-1}1_S\right\|_\infty\left\|\sqrt{\operatorname{Var}_{P_\pi}(V^\pi)}\right\|_\infty
= \frac{1}{1-\gamma}\left\|\sqrt{\operatorname{Var}_{P_\pi}(V^\pi)}\right\|_\infty
\le\frac{1}{(1-\gamma)^2}. \tag*{(87)} ( I − γ P π ) − 1 Var P π ( V π ) ∞ ≤ ( I − γ P π ) − 1 1 S ∞ Var P π ( V π ) ∞ = 1 − γ 1 Var P π ( V π ) ∞ ≤ ( 1 − γ ) 2 1 . (87)
where the penultimate relation arises from Lemma 2.3, and the last line is valid since Var q ( V π ) ≤ E q [ V π ∘ V π ] ≤ ( 1 − γ ) − 2 \operatorname{Var}_q(V^\pi)\le\mathbb{E}_q[V^\pi\circ V^\pi]\le(1-\gamma)^{-2} Var q ( V π ) ≤ E q [ V π ∘ V π ] ≤ ( 1 − γ ) − 2 for any probability distribution q q q . Substituting the above result into (83) and (86) and applying the basic inequality ∥ V π ∥ ∞ ≤ ( 1 − γ ) − 1 \lVert V^\pi\rVert_\infty\le(1-\gamma)^{-1} ∥ V π ∥ ∞ ≤ ( 1 − γ ) − 1 give
∥ V ^ π − V π ∥ ∞ = γ ∥ ( I − γ P ^ π ) − 1 ( P ^ π − P π ) V π ∥ ∞ ≤ C 0 { log ( S δ ) ( 1 − γ ) 4 N + log ( S δ ) ( 1 − γ ) 2 N } (88) \begin{aligned}
\lVert\widehat{V}^\pi-V^\pi\rVert_\infty=\gamma\left\|(I-\gamma\widehat{P}_\pi)^{-1}(\widehat{P}_\pi-P_\pi)V^\pi\right\|_\infty\\
&\le C_0\left\{\sqrt{\frac{\log\left(\frac{S}{\delta}\right)}{(1-\gamma)^4N}}+\frac{\log\left(\frac{S}{\delta}\right)}{(1-\gamma)^2N}\right\} \tag*{(88)}
\end{aligned} ∥ V π − V π ∥ ∞ = γ ( I − γ P π ) − 1 ( P π − P π ) V π ∞ ≤ C 0 ⎩ ⎨ ⎧ ( 1 − γ ) 4 N log ( δ S ) + ( 1 − γ ) 2 N log ( δ S ) ⎭ ⎬ ⎫ (88)
with probability exceeding 1 − δ 1-\delta 1 − δ .
The high-probability bound (88) reveals that: in order to attain ε \varepsilon ε -accurate value estimation (in an ℓ ∞ \ell_\infty ℓ ∞ sense), it suffices to draw
N = O ~ ( 1 ( 1 − γ ) 4 ε 2 ) N=\widetilde{O}\left(\frac{1}{(1-\gamma)^4\varepsilon^2}\right) N = O ( ( 1 − γ ) 4 ε 2 1 )
samples per state from the generative model. Nevertheless, this simple analysis incurs an extra multiplicative factor of 1 1 − γ \frac{1}{1-\gamma} 1 − γ 1 compared to the result stated in Theorem 3.1, thus falling short of tightness for problems with long effective horizons. Upon closer inspection, the separate treatment of ( I − γ P ^ π ) − 1 (I-\gamma\widehat{P}_{\pi})^{-1} ( I − γ P π ) − 1 and Var P π ( V π ) \sqrt{\operatorname{Var}_{P_{\pi}}(V^\pi)} Var P π ( V π ) overlooks their intimate connection that could allow for tighter control. Furthermore, we have not fully exploited the variance structure in (84) ; only crude upper bounds are applied. These factors together account for the suboptimality of the above analysis.
Analysis: a second attempt
We now refine our analysis for the value estimation error of interest, leveraging the interaction between ( I − γ P ^ π ) − 1 (I-\gamma\widehat{P}_{\pi})^{-1} ( I − γ P π ) − 1 and Var P π ( V π ) \sqrt{\operatorname{Var}_{P_{\pi}}(V^\pi)} Var P π ( V π ) , as well as the underlying variance structure. To simplify the presentation and concentrate our discussion on the main term, we first note — without a formal proof — that, for N N N large enough,
( I − γ P ^ π ) − 1 Var P π ( V π ) ≈ ( I − γ P π ) − 1 Var P π ( V π ) . (I-\gamma\widehat{P}_{\pi})^{-1}\sqrt{\operatorname{Var}_{P_{\pi}}(V^\pi)}
\approx
(I-\gamma P_{\pi})^{-1}\sqrt{\operatorname{Var}_{P_{\pi}}(V^\pi)}. ( I − γ P π ) − 1 Var P π ( V π ) ≈ ( I − γ P π ) − 1 Var P π ( V π ) .
This motivates us to control ( I − γ P π ) − 1 Var P π ( V π ) (I-\gamma P_{\pi})^{-1}\sqrt{\operatorname{Var}_{P_{\pi}}(V^\pi)} ( I − γ P π ) − 1 Var P π ( V π ) .
Recall from Lemma 2.3 that ( I − γ P π ) − 1 (I-\gamma P_{\pi})^{-1} ( I − γ P π ) − 1 is a nonnegative matrix with each row summing to 1 1 − γ \frac{1}{1-\gamma} 1 − γ 1 , thus indicating that ( 1 − γ ) ( I − γ P π ) − 1 (1-\gamma)(I-\gamma P_{\pi})^{-1} ( 1 − γ ) ( I − γ P π ) − 1 is a valid probability transition matrix. This inspires one to push ( I − γ P π ) − 1 (I-\gamma P_{\pi})^{-1} ( I − γ P π ) − 1 into the square root operator, yielding
( I − γ P π ) − 1 Var P π ( V π ) = 1 1 − γ { ( 1 − γ ) ( I − γ P π ) − 1 Var P π ( V π ) } ≤ 1 1 − γ ( 1 − γ ) ( I − γ P π ) − 1 Var P π ( V π ) = 1 1 − γ ( I − γ P π ) − 1 Var P π ( V π ) . (89) \begin{aligned}
(I-\gamma P_{\pi})^{-1}\sqrt{\operatorname{Var}_{P_{\pi}}(V^\pi)}
&=\frac{1}{1-\gamma}\left\{(1-\gamma)(I-\gamma P_{\pi})^{-1}\sqrt{\operatorname{Var}_{P_{\pi}}(V^\pi)}\right\}\\
&\le\frac{1}{1-\gamma}\sqrt{(1-\gamma)(I-\gamma P_{\pi})^{-1}\operatorname{Var}_{P_{\pi}}(V^\pi)}\\
&=\sqrt{\frac{1}{1-\gamma}(I-\gamma P_{\pi})^{-1}\operatorname{Var}_{P_{\pi}}(V^\pi)}. \tag*{(89)}
\end{aligned} ( I − γ P π ) − 1 Var P π ( V π ) = 1 − γ 1 { ( 1 − γ ) ( I − γ P π ) − 1 Var P π ( V π ) } ≤ 1 − γ 1 ( 1 − γ ) ( I − γ P π ) − 1 Var P π ( V π ) = 1 − γ 1 ( I − γ P π ) − 1 Var P π ( V π ) . (89)
Here, we apply Jensen’s inequality: for any probability vector q q q and any nonnegative random variable v v v , it holds that E q [ v ] ≤ E q [ v ] \mathbb{E}_q[\sqrt{v}]\le\sqrt{\mathbb{E}_q[v]} E q [ v ] ≤ E q [ v ] .
To better exploit the underlying variance structure, rewrite and upper-bound Var P π ( V π ) \operatorname{Var}_{P_{\pi}}(V^\pi) Var P π ( V π ) (cf. (32) ) as follows:
Var P π ( V π ) = P π ( V π ∘ V π ) − ( P π V π ) ∘ ( P π V π ) = P π ( V π ∘ V π ) − 1 γ 2 ( V π − r π ) ∘ ( V π − r π ) = 1 γ 2 ( γ 2 P π − I ) ( V π ∘ V π ) + 2 γ 2 V π ∘ r π − 1 γ 2 r π ∘ r π ≤ 1 γ 2 ( γ P π − I ) ( V π ∘ V π ) + 2 γ 2 V π ∘ r π , \begin{aligned}
\operatorname{Var}_{P_{\pi}}(V^\pi)
&=P_{\pi}(V^\pi\circ V^\pi)-(P_{\pi}V^\pi)\circ(P_{\pi}V^\pi)\\
&=P_{\pi}(V^\pi\circ V^\pi)-\frac{1}{\gamma^2}(V^\pi-r_{\pi})\circ(V^\pi-r_{\pi})\\
&=\frac{1}{\gamma^2}(\gamma^2P_{\pi}-I)(V^\pi\circ V^\pi)+\frac{2}{\gamma^2}V^\pi\circ r_{\pi}-\frac{1}{\gamma^2}r_{\pi}\circ r_{\pi} \\
&\le\frac{1}{\gamma^2}(\gamma P_\pi-I)(V^\pi\circ V^\pi)+\frac{2}{\gamma^2}V^\pi\circ r_\pi,
\end{aligned} Var P π ( V π ) = P π ( V π ∘ V π ) − ( P π V π ) ∘ ( P π V π ) = P π ( V π ∘ V π ) − γ 2 1 ( V π − r π ) ∘ ( V π − r π ) = γ 2 1 ( γ 2 P π − I ) ( V π ∘ V π ) + γ 2 2 V π ∘ r π − γ 2 1 r π ∘ r π ≤ γ 2 1 ( γ P π − I ) ( V π ∘ V π ) + γ 2 2 V π ∘ r π ,
where the second line applies the Bellman equation V π = r π + γ P π V π V^\pi=r_\pi+\gamma P_\pi V^\pi V π = r π + γ P π V π (cf. (34) ). This yields a simple upper bound on ( I − γ P π ) − 1 Var P π ( V π ) (I-\gamma P_\pi)^{-1}\operatorname{Var}_{P_\pi}(V^\pi) ( I − γ P π ) − 1 Var P π ( V π ) :
0 ≤ ( I − γ P π ) − 1 Var P π ( V π ) ≤ 1 γ 2 ( I − γ P π ) − 1 ( γ P π − I ) ( V π ∘ V π ) + 2 γ 2 ( I − γ P π ) − 1 ( V π ∘ r π ) = − 1 γ 2 V π ∘ V π + 2 γ 2 ( I − γ P π ) − 1 ( V π ∘ r π ) ≤ 2 γ 2 ( I − γ P π ) − 1 ( V π ∘ r π ) ≤ 2 ∥ V π ∥ ∞ γ 2 ( I − γ P π ) − 1 r π = 2 ∥ V π ∥ ∞ γ 2 V π ≤ 2 ∥ V π ∥ ∞ 2 γ 2 1 S , (90) \begin{aligned}
0 &\le(I-\gamma P_\pi)^{-1}\operatorname{Var}_{P_\pi}(V^\pi)\\
&\le\frac{1}{\gamma^2}(I-\gamma P_\pi)^{-1}(\gamma P_\pi-I)(V^\pi\circ V^\pi)+\frac{2}{\gamma^2}(I-\gamma P_\pi)^{-1}(V^\pi\circ r_\pi)\\
&= -\frac{1}{\gamma^2}V^\pi\circ V^\pi+\frac{2}{\gamma^2}(I-\gamma P_\pi)^{-1}(V^\pi\circ r_\pi)\\
&\le\frac{2}{\gamma^2}(I-\gamma P_\pi)^{-1}(V^\pi\circ r_\pi)\\
&\le\frac{2\lVert V^\pi\rVert_\infty}{\gamma^2}(I-\gamma P_\pi)^{-1}r_\pi\\
&= \frac{2\lVert V^\pi\rVert_\infty}{\gamma^2}V^\pi\le\frac{2\lVert V^\pi\rVert_\infty^2}{\gamma^2}\mathbf{1}_S, \tag*{(90)}
\end{aligned} 0 ≤ ( I − γ P π ) − 1 Var P π ( V π ) ≤ γ 2 1 ( I − γ P π ) − 1 ( γ P π − I ) ( V π ∘ V π ) + γ 2 2 ( I − γ P π ) − 1 ( V π ∘ r π ) = − γ 2 1 V π ∘ V π + γ 2 2 ( I − γ P π ) − 1 ( V π ∘ r π ) ≤ γ 2 2 ( I − γ P π ) − 1 ( V π ∘ r π ) ≤ γ 2 2 ∥ V π ∥ ∞ ( I − γ P π ) − 1 r π = γ 2 2 ∥ V π ∥ ∞ V π ≤ γ 2 2 ∥ V π ∥ ∞ 2 1 S , (90)
where the last line uses (35) . Taking this together with (89) yields
0 ≤ ∥ ( I − γ P π ) − 1 Var P π ( V π ) ∥ ∞ ≤ 2 1 − γ ∥ V π ∥ ∞ γ . (91) 0\le\left\|(I-\gamma P_\pi)^{-1}\sqrt{\operatorname{Var}_{P_\pi}(V^\pi)}\right\|_\infty\le\sqrt{\frac{2}{1-\gamma}}\frac{\lVert V^\pi\rVert_\infty}{\gamma}. \tag*{(91)} 0 ≤ ( I − γ P π ) − 1 Var P π ( V π ) ∞ ≤ 1 − γ 2 γ ∥ V π ∥ ∞ . (91)
Substituting this into (83) and (86) , and using the basic fact that ∥ V π ∥ ∞ ≤ 1 1 − γ \lVert V^\pi\rVert_\infty\le\frac{1}{1-\gamma} ∥ V π ∥ ∞ ≤ 1 − γ 1 , we arrive at
∥ V ^ π − V π ∥ ∞ ≤ C 0 { γ N ∥ ( I − γ P ^ π ) − 1 Var P π ( V π ) ∥ ∞ + γ ( 1 − γ ) 2 N } ≈ C 0 { γ N ∥ ( I − γ P π ) − 1 Var P π ( V π ) ∥ ∞ + γ ( 1 − γ ) 2 N } ≤ C 0 { 2 N ( 1 − γ ) 3 + 1 ( 1 − γ ) 2 N } ≤ 2 C 0 2 N ( 1 − γ ) 3 , \begin{aligned}
\lVert\widehat V^\pi-V^\pi\rVert_\infty
&\le C_0\left\{\frac{\gamma}{\sqrt{N}}\left\|(I-\gamma\widehat P_\pi)^{-1}\sqrt{\operatorname{Var}_{P_\pi}(V^\pi)}\right\|_\infty+\frac{\gamma}{(1-\gamma)^2N}\right\}\\
&\approx C_0\left\{\frac{\gamma}{\sqrt{N}}\left\|(I-\gamma P_\pi)^{-1}\sqrt{\operatorname{Var}_{P_\pi}(V^\pi)}\right\|_\infty+\frac{\gamma}{(1-\gamma)^2N}\right\}\\
&\le C_0\left\{\sqrt{\frac{2}{N(1-\gamma)^3}}+\frac{1}{(1-\gamma)^2N}\right\}\\
&\le2C_0\sqrt{\frac{2}{N(1-\gamma)^3}},
\end{aligned} ∥ V π − V π ∥ ∞ ≤ C 0 { N γ ( I − γ P π ) − 1 Var P π ( V π ) ∞ + ( 1 − γ ) 2 N γ } ≈ C 0 { N γ ( I − γ P π ) − 1 Var P π ( V π ) ∞ + ( 1 − γ ) 2 N γ } ≤ C 0 { N ( 1 − γ ) 3 2 + ( 1 − γ ) 2 N 1 } ≤ 2 C 0 N ( 1 − γ ) 3 2 ,
provided that N N N is sufficiently large. This reveals that ε \varepsilon ε -accurate value estimation is attainable — at least in the asymptotic regime with N N N (resp. ε \varepsilon ε ) approaching infinity (resp. 0 0 0 ) — when the sample size per state exceeds the following order
N = O ~ ( 1 ( 1 − γ ) 3 ε 2 ) , N=\widetilde O\left(\frac{1}{(1-\gamma)^3\varepsilon^2}\right), N = O ( ( 1 − γ ) 3 ε 2 1 ) ,
thereby matching the advertised result in Theorem 3.1.
However, an obvious limitation of the above analysis stems from the stringent requirement in N N N : the sharpness of the above analysis is only guaranteed for very large N N N . Additional efforts are needed to extend the claimed results to the data-limited regime, i.e., the situation where ε \varepsilon ε is not exceedingly small.
Sharp analysis: proof of Theorem 3.1
In order to further improve the analysis strategy outlined in § 4.5 , a key step lies in controlling the following term:
( ( I − γ P ^ π ) − 1 − ( I − γ P π ) − 1 ) Var P π ( V π ) , (92) \left((I-\gamma\widehat{P}_{\pi})^{-1}-(I-\gamma P_{\pi})^{-1}\right)\sqrt{\operatorname{Var}_{P_{\pi}}(V^{\pi})}, \tag*{(92)} ( ( I − γ P π ) − 1 − ( I − γ P π ) − 1 ) Var P π ( V π ) , (92)
as it determines the range of N N N within which optimality can be achieved. Interestingly, the analyses presented in § 4.4 and § 4.5 already shed light on how to handle this term. More specifically, if we express the value estimation error as
V ^ π − V π = ( ( I − γ P ^ π ) − 1 − ( I − γ P π ) − 1 ) r π \widehat{V}^{\pi}-V^{\pi}=\left((I-\gamma\widehat{P}_{\pi})^{-1}-(I-\gamma P_{\pi})^{-1}\right)r_{\pi} V π − V π = ( ( I − γ P π ) − 1 − ( I − γ P π ) − 1 ) r π
according to the Bellman equation (35) , then the term (92) can be interpreted as a sort of value estimation error with the reward function taken to be r ( 1 ) = Var P π ( V π ) r^{(1)}=\sqrt{\operatorname{Var}_{P_{\pi}}(V^{\pi})} r ( 1 ) = Var P π ( V π ) , which can be tackled using arguments in § 4.5 . However, applying this approach results in a similar issue: the need to bound an even “higher-order” term
( ( I − γ P ^ π ) − 1 − ( I − γ P π ) − 1 ) Var P π ( V ( 1 ) ) \left((I-\gamma\widehat{P}_{\pi})^{-1}-(I-\gamma P_{\pi})^{-1}\right)\sqrt{\operatorname{Var}_{P_{\pi}}(V^{(1)})} ( ( I − γ P π ) − 1 − ( I − γ P π ) − 1 ) Var P π ( V ( 1 ) )
with V ( 1 ) ≔ ( I − γ P π ) − 1 r ( 1 ) V^{(1)}\coloneqq(I-\gamma P_{\pi})^{-1}r^{(1)} V ( 1 ) : = ( I − γ P π ) − 1 r ( 1 ) . Motivated by this line of thought, we propose a recursive analysis approach, with the aim of controlling the residual terms in an increasingly more refined fashion.
Step 1: recursive construction of auxiliary rewards and values. To apply the above recursive arguments, construct a sequence of auxiliary “reward” and “value” functions recursively:
r ( 0 ) ≔ r π , V ( 0 ) ≔ ( I − γ P π ) − 1 r ( 0 ) , r ( l ) ≔ Var P π [ V ( l − 1 ) ] , V ( l ) ≔ ( I − γ P π ) − 1 r ( l ) , l ≥ 1. (93) \begin{aligned}
r^{(0)}&\coloneqq r_{\pi}, & V^{(0)}&\coloneqq(I-\gamma P_{\pi})^{-1}r^{(0)},\\
r^{(l)}&\coloneqq\sqrt{\operatorname{Var}_{P_{\pi}}[V^{(l-1)}]}, & V^{(l)}&\coloneqq(I-\gamma P_{\pi})^{-1}r^{(l)}, \qquad l\ge1. \tag*{(93)}
\end{aligned} r ( 0 ) r ( l ) : = r π , : = Var P π [ V ( l − 1 ) ] , V ( 0 ) V ( l ) : = ( I − γ P π ) − 1 r ( 0 ) , : = ( I − γ P π ) − 1 r ( l ) , l ≥ 1. (93)
Similarly, we find it convenient to introduce the auxiliary “value estimates” as follows:
V ^ ( l ) ≔ ( I − γ P ^ π ) − 1 r ( l ) , l ≥ 0. (94) \widehat{V}^{(l)}\coloneqq(I-\gamma\widehat{P}_{\pi})^{-1}r^{(l)}, \qquad l\ge0. \tag*{(94)} V ( l ) : = ( I − γ P π ) − 1 r ( l ) , l ≥ 0. (94)
As an immediate consequence of inequality (91) , we can readily establish an entrywise bound on V ( l ) V^{(l)} V ( l ) for any l > 1 l>1 l > 1 :
∥ V ( l ) ∥ ∞ = ∥ ( I − γ P π ) − 1 r ( l ) ∥ ∞ = ∥ ( I − γ P π ) − 1 Var P π [ V ( l − 1 ) ] ∥ ∞ ≤ 2 γ 1 − γ ∥ V ( l − 1 ) ∥ ∞ ≤ ⋯ ≤ ( 2 γ 1 − γ ) l ∥ V ( 0 ) ∥ ∞ (95) \begin{aligned}
\lVert V^{(l)}\rVert_{\infty}
&=\lVert(I-\gamma P_{\pi})^{-1}r^{(l)}\rVert_{\infty}
=\left\lVert(I-\gamma P_{\pi})^{-1}\sqrt{\operatorname{Var}_{P_{\pi}}[V^{(l-1)}]}\right\rVert_{\infty}\\
&\le\frac{\sqrt{2}}{\gamma\sqrt{1-\gamma}}\lVert V^{(l-1)}\rVert_{\infty}
\le\cdots\le\left(\frac{\sqrt{2}}{\gamma\sqrt{1-\gamma}}\right)^{l}\lVert V^{(0)}\rVert_{\infty} \tag*{(95)}
\end{aligned} ∥ V ( l ) ∥ ∞ = ∥( I − γ P π ) − 1 r ( l ) ∥ ∞ = ( I − γ P π ) − 1 Var P π [ V ( l − 1 ) ] ∞ ≤ γ 1 − γ 2 ∥ V ( l − 1 ) ∥ ∞ ≤ ⋯ ≤ ( γ 1 − γ 2 ) l ∥ V ( 0 ) ∥ ∞ (95)
= ( 2 γ 1 − γ ) l ∥ V π ∥ ∞ ≤ 1 1 − γ ( 2 γ 1 − γ ) l , (96) =\left(\frac{\sqrt{2}}{\gamma\sqrt{1-\gamma}}\right)^{l}\lVert V^{\pi}\rVert_{\infty}
\le\frac{1}{1-\gamma}\left(\frac{\sqrt{2}}{\gamma\sqrt{1-\gamma}}\right)^{l}, \tag*{(96)} = ( γ 1 − γ 2 ) l ∥ V π ∥ ∞ ≤ 1 − γ 1 ( γ 1 − γ 2 ) l , (96)
where the last line holds since ∥ V π ∥ ∞ ≤ 1 1 − γ \lVert V^{\pi}\rVert_{\infty}\le\frac{1}{1-\gamma} ∥ V π ∥ ∞ ≤ 1 − γ 1 .
Step 2: establishing a recursion for V ^ ( l ) − V ( l ) \widehat{V}^{(l)}-V^{(l)} V ( l ) − V ( l ) . Equipped with the above notation, we observe that, for any l ≥ 0 l\ge0 l ≥ 0 ,
V ^ ( l ) − V ( l ) = ( I − γ P ^ π ) − 1 r ( l ) − V ( l ) = ( I − γ P ^ π ) − 1 ( I − γ P π ) V ( l ) − ( I − γ P ^ π ) − 1 ( I − γ P ^ π ) V ( l ) = γ ( I − γ P ^ π ) − 1 ( P ^ π − P π ) V ( l ) , (97) \begin{aligned}
\widehat{V}^{(l)}-V^{(l)}
&=(I-\gamma\widehat{P}_{\pi})^{-1}r^{(l)}-V^{(l)}\\
&=(I-\gamma\widehat{P}_{\pi})^{-1}(I-\gamma P_{\pi})V^{(l)}
-(I-\gamma\widehat{P}_{\pi})^{-1}(I-\gamma\widehat{P}_{\pi})V^{(l)} \\
&= \gamma(I-\gamma\widehat{P}_{\pi})^{-1}(\widehat{P}_{\pi}-P_{\pi})V^{(l)}, \tag*{(97)}
\end{aligned} V ( l ) − V ( l ) = ( I − γ P π ) − 1 r ( l ) − V ( l ) = ( I − γ P π ) − 1 ( I − γ P π ) V ( l ) − ( I − γ P π ) − 1 ( I − γ P π ) V ( l ) = γ ( I − γ P π ) − 1 ( P π − P π ) V ( l ) , (97)
where the second line follows from the identity ( I − γ P π ) V ( l ) = r ( l ) (I-\gamma P_{\pi})V^{(l)}=r^{(l)} ( I − γ P π ) V ( l ) = r ( l ) (see the construction (93) ). The Bernstein inequality in Lemma 2.5 together with the union bound asserts that, with probability exceeding 1 − δ 1-\delta 1 − δ , one has
∣ ( P ^ π − P π ) V ( l ) ∣ ≤ β 1 N Var P π [ V ( l ) ] + ∥ V ( l ) ∥ ∞ β 1 N 1 S (98) \left|(\widehat{P}_{\pi}-P_{\pi})V^{(l)}\right|
\leq\sqrt{\frac{\beta_1}{N}\operatorname{Var}_{P_{\pi}}[V^{(l)}]}+\frac{\|V^{(l)}\|_{\infty}\beta_1}{N}\mathbf{1}_{S} \tag*{(98)} ( P π − P π ) V ( l ) ≤ N β 1 Var P π [ V ( l ) ] + N ∥ V ( l ) ∥ ∞ β 1 1 S (98)
uniformly for all 0 ≤ l ≤ m 0\leq l\leq m 0 ≤ l ≤ m , where m m m is some large enough integer to be specified shortly, and β 1 = C 0 log S m δ \beta_1=C_0\log\frac{Sm}{\delta} β 1 = C 0 log δ S m for some large enough constant C 0 > 0 C_0>0 C 0 > 0 . Combining this result with (97) reveals that
∥ V ^ ( l ) − V ( l ) ∥ ∞ = γ ∥ ( I − γ P ^ π ) − 1 ( P ^ π − P π ) V ( l ) ∥ ∞ \left\|\widehat{V}^{(l)}-V^{(l)}\right\|_{\infty}
=\gamma\left\|(I-\gamma\widehat{P}_{\pi})^{-1}(\widehat{P}_{\pi}-P_{\pi})V^{(l)}\right\|_{\infty} V ( l ) − V ( l ) ∞ = γ ( I − γ P π ) − 1 ( P π − P π ) V ( l ) ∞
≤ (i) γ ∥ ( I − γ P ^ π ) − 1 ∣ ( P ^ π − P π ) V ( l ) ∣ ∥ ∞ \underset{\text{(i)}}{\leq}\gamma\left\|(I-\gamma\widehat{P}_{\pi})^{-1}\left|(\widehat{P}_{\pi}-P_{\pi})V^{(l)}\right|\right\|_{\infty} (i) ≤ γ ( I − γ P π ) − 1 ( P π − P π ) V ( l ) ∞
≤ (ii) γ β 1 N ∥ ( I − γ P ^ π ) − 1 Var P π [ V ( l ) ] ∥ ∞ + γ ∥ V ( l ) ∥ ∞ β 1 ( 1 − γ ) N \underset{\text{(ii)}}{\leq}\gamma\sqrt{\frac{\beta_1}{N}}\left\|(I-\gamma\widehat{P}_{\pi})^{-1}\sqrt{\operatorname{Var}_{P_{\pi}}[V^{(l)}]}\right\|_{\infty}
+\frac{\gamma\|V^{(l)}\|_{\infty}\beta_1}{(1-\gamma)N} (ii) ≤ γ N β 1 ( I − γ P π ) − 1 Var P π [ V ( l ) ] ∞ + ( 1 − γ ) N γ ∥ V ( l ) ∥ ∞ β 1
≤ (iii) γ β 1 N ∥ V ^ ( l + 1 ) − V ( l + 1 ) ∥ ∞ + γ β 1 N ∥ V ( l + 1 ) ∥ ∞ + γ ∥ V ( l ) ∥ ∞ β 1 ( 1 − γ ) N \underset{\text{(iii)}}{\leq}\gamma\sqrt{\frac{\beta_1}{N}}\left\|\widehat{V}^{(l+1)}-V^{(l+1)}\right\|_{\infty}
+\gamma\sqrt{\frac{\beta_1}{N}}\|V^{(l+1)}\|_{\infty}
+\frac{\gamma\|V^{(l)}\|_{\infty}\beta_1}{(1-\gamma)N} (iii) ≤ γ N β 1 V ( l + 1 ) − V ( l + 1 ) ∞ + γ N β 1 ∥ V ( l + 1 ) ∥ ∞ + ( 1 − γ ) N γ ∥ V ( l ) ∥ ∞ β 1
≤ (iv) γ β 1 N ∥ V ^ ( l + 1 ) − V ( l + 1 ) ∥ ∞ + ( 2 β 1 ( 1 − γ ) N + β 1 ( 1 − γ ) N ) ∥ V ( l ) ∥ ∞ . (99) \underset{\text{(iv)}}{\leq}\gamma\sqrt{\frac{\beta_1}{N}}\left\|\widehat{V}^{(l+1)}-V^{(l+1)}\right\|_{\infty}
+\left(\sqrt{\frac{2\beta_1}{(1-\gamma)N}}+\frac{\beta_1}{(1-\gamma)N}\right)\|V^{(l)}\|_{\infty}. \tag*{(99)} (iv) ≤ γ N β 1 V ( l + 1 ) − V ( l + 1 ) ∞ + ( ( 1 − γ ) N 2 β 1 + ( 1 − γ ) N β 1 ) ∥ V ( l ) ∥ ∞ . (99)
with probability exceeding 1 − δ 1-\delta 1 − δ . Here, (i) follows since ( I − γ P ^ π ) − 1 (I-\gamma\widehat{P}_{\pi})^{-1} ( I − γ P π ) − 1 is a nonnegative matrix, (ii) comes from (98) and the triangle inequality, (iii) arises from the definitions of r ( l ) r^{(l)} r ( l ) , V ( l ) V^{(l)} V ( l ) and V ^ ( l ) \widehat{V}^{(l)} V ( l ) , while (iv) is a consequence of (95) . Crucially, inequality (3.23) establishes a recursive relation that enables a sharp upper bound analysis.
Step 3: upper bounding via the above recursion. Define
b 1 : = γ β 1 N , b 2 : = 2 β 1 ( 1 − γ ) N + β 1 ( 1 − γ ) N , b 3 : = 2 γ 1 − γ (100) b_1:=\gamma\sqrt{\frac{\beta_1}{N}},\qquad
b_2:=\sqrt{\frac{2\beta_1}{(1-\gamma)N}}+\frac{\beta_1}{(1-\gamma)N},\qquad
b_3:=\frac{\sqrt{2}}{\gamma\sqrt{1-\gamma}} \tag*{(100)} b 1 := γ N β 1 , b 2 := ( 1 − γ ) N 2 β 1 + ( 1 − γ ) N β 1 , b 3 := γ 1 − γ 2 (100)
for notational simplicity. Combining inequalities (96) and (3.23) shows that with probability at least 1 − δ 1-\delta 1 − δ ,
∥ V ^ ( l ) − V ( l ) ∥ ∞ ≤ b 1 ∥ V ^ ( l + 1 ) − V ( l + 1 ) ∥ ∞ + b 2 ∥ V ( l ) ∥ ∞ ≤ b 1 ∥ V ^ ( l + 1 ) − V ( l + 1 ) ∥ ∞ + b 2 b 3 l ∥ V ( 0 ) ∥ ∞ \begin{aligned}
\left\|\widehat{V}^{(l)}-V^{(l)}\right\|_{\infty}
\leq b_1\left\|\widehat{V}^{(l+1)}-V^{(l+1)}\right\|_{\infty}
+b_2\|V^{(l)}\|_{\infty} \\
&\leq b_1\left\|\widehat{V}^{(l+1)}-V^{(l+1)}\right\|_{\infty}
+b_2b_3^{l}\|V^{(0)}\|_{\infty}
\end{aligned} V ( l ) − V ( l ) ∞ ≤ b 1 V ( l + 1 ) − V ( l + 1 ) ∞ + b 2 ∥ V ( l ) ∥ ∞ ≤ b 1 V ( l + 1 ) − V ( l + 1 ) ∞ + b 2 b 3 l ∥ V ( 0 ) ∥ ∞
holds for all 0 ≤ l < m 0\leq l<m 0 ≤ l < m . Apply this relation recursively and combine terms to arrive at
∥ V ^ ( 0 ) − V ( 0 ) ∥ ∞ ≤ b 1 ∥ V ^ ( 1 ) − V ( 1 ) ∥ ∞ + b 2 ∥ V ( 0 ) ∥ ∞ ≤ b 1 { b 1 ∥ V ^ ( 2 ) − V ( 2 ) ∥ ∞ + b 2 b 3 ∥ V ( 0 ) ∥ ∞ } + b 2 ∥ V ( 0 ) ∥ ∞ ≤ ⋯ ≤ b 1 m ∥ V ^ ( m ) − V ( m ) ∥ ∞ + b 2 ∑ l = 0 m − 1 ( b 1 b 3 ) l ∥ V ( 0 ) ∥ ∞ , b 1 m ∥ V ^ ( m ) − V ( m ) ∥ ∞ ⏟ = : α 1 + b 2 ∑ l = 0 m − 1 ( b 1 b 3 ) l ∥ V ( 0 ) ∥ ∞ ⏟ = : α 2 . (101) \begin{aligned}
\left\|\widehat{V}^{(0)}-V^{(0)}\right\|_{\infty}
\leq b_1\left\|\widehat{V}^{(1)}-V^{(1)}\right\|_{\infty}
+b_2\|V^{(0)}\|_{\infty} \\
&\leq b_1\left\{b_1\left\|\widehat{V}^{(2)}-V^{(2)}\right\|_{\infty}
+b_2b_3\|V^{(0)}\|_{\infty}\right\}
+b_2\|V^{(0)}\|_{\infty} \\
&\leq\cdots\\
&\leq b_1^m\left\|\widehat{V}^{(m)}-V^{(m)}\right\|_{\infty}
+b_2\sum_{l=0}^{m-1}(b_1b_3)^l\|V^{(0)}\|_{\infty},
\qquad
\underbrace{b_1^m\left\|\widehat{V}^{(m)}-V^{(m)}\right\|_{\infty}}_{=:\alpha_1}
+\underbrace{b_2\sum_{l=0}^{m-1}(b_1b_3)^l\|V^{(0)}\|_{\infty}}_{=:\alpha_2}. \tag*{(101)}
\end{aligned} V ( 0 ) − V ( 0 ) ∞ ≤ b 1 V ( 1 ) − V ( 1 ) ∞ + b 2 ∥ V ( 0 ) ∥ ∞ ≤ b 1 { b 1 V ( 2 ) − V ( 2 ) ∞ + b 2 b 3 ∥ V ( 0 ) ∥ ∞ } + b 2 ∥ V ( 0 ) ∥ ∞ ≤ ⋯ ≤ b 1 m V ( m ) − V ( m ) ∞ + b 2 l = 0 ∑ m − 1 ( b 1 b 3 ) l ∥ V ( 0 ) ∥ ∞ , =: α 1 b 1 m V ( m ) − V ( m ) ∞ + =: α 2 b 2 l = 0 ∑ m − 1 ( b 1 b 3 ) l ∥ V ( 0 ) ∥ ∞ . (101)
leaving us with two terms to control.
Step 4: controlling quantity α 1 \alpha_1 α 1 . To bound quantity α 1 \alpha_1 α 1 , we invoke identity (99) to obtain
∥ V ^ ( m ) − V ( m ) ∥ ∞ ≤ γ β 1 N ∥ ( I − γ P ^ π ) − 1 ∥ ∞ ∥ Var P π [ V ( m ) ] ∥ ∞ + γ β 1 ∥ V ( m ) ∥ ∞ ( 1 − γ ) N ≤ γ β 1 N ∥ ( I − γ P ^ π ) − 1 ∥ 1 ∥ Var P π [ V ( m ) ] ∥ ∞ + γ β 1 ∥ V ( m ) ∥ ∞ ( 1 − γ ) N ≤ γ 1 − γ β 1 N ∥ V ( m ) ∥ ∞ + γ β 1 ∥ V ( m ) ∥ ∞ ( 1 − γ ) N , \begin{aligned}
\left\|\widehat{V}^{(m)}-V^{(m)}\right\|_{\infty}\leq\gamma\sqrt{\frac{\beta_1}{N}}\left\|(I-\gamma\widehat{P}_{\pi})^{-1}\right\|_{\infty}\left\|\sqrt{\operatorname{Var}_{P_{\pi}}\left[V^{(m)}\right]}\right\|_{\infty}+\frac{\gamma\beta_1\left\|V^{(m)}\right\|_{\infty}}{(1-\gamma)N} \\
&\leq\gamma\sqrt{\frac{\beta_1}{N}}\left\|(I-\gamma\widehat{P}_{\pi})^{-1}\right\|_{1}\left\|\sqrt{\operatorname{Var}_{P_{\pi}}\left[V^{(m)}\right]}\right\|_{\infty}+\frac{\gamma\beta_1\left\|V^{(m)}\right\|_{\infty}}{(1-\gamma)N} \\
&\leq\frac{\gamma}{1-\gamma}\sqrt{\frac{\beta_1}{N}}\left\|V^{(m)}\right\|_{\infty}+\frac{\gamma\beta_1\left\|V^{(m)}\right\|_{\infty}}{(1-\gamma)N},
\end{aligned} V ( m ) − V ( m ) ∞ ≤ γ N β 1 ( I − γ P π ) − 1 ∞ Var P π [ V ( m ) ] ∞ + ( 1 − γ ) N γ β 1 V ( m ) ∞ ≤ γ N β 1 ( I − γ P π ) − 1 1 Var P π [ V ( m ) ] ∞ + ( 1 − γ ) N γ β 1 V ( m ) ∞ ≤ 1 − γ γ N β 1 V ( m ) ∞ + ( 1 − γ ) N γ β 1 V ( m ) ∞ ,
where the last inequality uses the basic fact ∥ ( I − γ P ^ π ) − 1 ∥ 1 = 1 1 − γ \left\|(I-\gamma\widehat{P}_{\pi})^{-1}\right\|_{1}=\frac{1}{1-\gamma} ( I − γ P π ) − 1 1 = 1 − γ 1 (cf. (40) ) as well as the elementary relation Var P π [ V ( m ) ] ≤ ∥ V ( m ) ∥ ∞ 2 \operatorname{Var}_{P_{\pi}}\left[V^{(m)}\right]\leq\left\|V^{(m)}\right\|_{\infty}^{2} Var P π [ V ( m ) ] ≤ V ( m ) ∞ 2 . This combined with inequality (96) yields
∥ V ^ ( m ) − V ( m ) ∥ ∞ ≤ γ ( 1 − γ ) 2 ( β 1 N + β 1 N ) ( 2 γ 1 − γ ) m . \left\|\widehat{V}^{(m)}-V^{(m)}\right\|_{\infty}\leq\frac{\gamma}{(1-\gamma)^2}\left(\sqrt{\frac{\beta_1}{N}}+\frac{\beta_1}{N}\right)\left(\frac{\sqrt{2}}{\gamma\sqrt{1-\gamma}}\right)^m. V ( m ) − V ( m ) ∞ ≤ ( 1 − γ ) 2 γ ( N β 1 + N β 1 ) ( γ 1 − γ 2 ) m .
As a consequence, we arrive at
α 1 = b 1 m ∥ V ^ ( m ) − V ( m ) ∥ ∞ ≤ ( γ β 1 N ) m γ ( 1 − γ ) 2 ( β 1 N + β 1 N ) ( 2 γ 1 − γ ) m ≤ ( 2 β 1 ( 1 − γ ) N ) m γ ( 1 − γ ) 2 ( β 1 N + β 1 N ) ≤ ( 1 e ) m 2 ( 1 − γ ) 2 β 1 N , (102) \begin{aligned}
\alpha_1&=b_1^m\left\|\widehat{V}^{(m)}-V^{(m)}\right\|_{\infty}\\
&\leq\left(\gamma\sqrt{\frac{\beta_1}{N}}\right)^m\frac{\gamma}{(1-\gamma)^2}\left(\sqrt{\frac{\beta_1}{N}}+\frac{\beta_1}{N}\right)\left(\frac{\sqrt{2}}{\gamma\sqrt{1-\gamma}}\right)^m\\
&\leq\left(\sqrt{\frac{2\beta_1}{(1-\gamma)N}}\right)^m\frac{\gamma}{(1-\gamma)^2}\left(\sqrt{\frac{\beta_1}{N}}+\frac{\beta_1}{N}\right)\\
&\leq\left(\frac{1}{e}\right)^m\frac{2}{(1-\gamma)^2}\sqrt{\frac{\beta_1}{N}}, \tag*{(102)}
\end{aligned} α 1 = b 1 m V ( m ) − V ( m ) ∞ ≤ ( γ N β 1 ) m ( 1 − γ ) 2 γ ( N β 1 + N β 1 ) ( γ 1 − γ 2 ) m ≤ ( ( 1 − γ ) N 2 β 1 ) m ( 1 − γ ) 2 γ ( N β 1 + N β 1 ) ≤ ( e 1 ) m ( 1 − γ ) 2 2 N β 1 , (102)
where the last line holds as long as N > 2 e 2 1 − γ β 1 N>\frac{2e^2}{1-\gamma}\beta_1 N > 1 − γ 2 e 2 β 1 .
Step 5: controlling quantity α 2 \alpha_2 α 2 . Regarding quantity α 2 \alpha_2 α 2 , we note that b 1 b 3 ≤ 1 / 2 b_1b_3\leq1/2 b 1 b 3 ≤ 1/2 holds as long as N ≥ 64 β 1 / ( 1 − γ ) N\geq64\beta_1/(1-\gamma) N ≥ 64 β 1 / ( 1 − γ ) . Summing the terms of the geometric sequence leads to
b 2 ∑ l = 0 m − 1 ( b 1 b 3 ) l ≤ b 2 1 − b 1 b 3 ≤ 8 β 1 ( 1 − γ ) N + 2 β 1 ( 1 − γ ) N , b_2\sum_{l=0}^{m-1}(b_1b_3)^l\leq\frac{b_2}{1-b_1b_3}\leq\sqrt{\frac{8\beta_1}{(1-\gamma)N}}+\frac{2\beta_1}{(1-\gamma)N}, b 2 l = 0 ∑ m − 1 ( b 1 b 3 ) l ≤ 1 − b 1 b 3 b 2 ≤ ( 1 − γ ) N 8 β 1 + ( 1 − γ ) N 2 β 1 ,
which taken collectively with the bound ∥ V ( 0 ) ∥ ∞ ≤ 1 1 − γ \left\|V^{(0)}\right\|_{\infty}\leq\frac{1}{1-\gamma} V ( 0 ) ∞ ≤ 1 − γ 1 gives
α 2 = { b 2 ∑ l = 0 m − 1 ( b 1 b 3 ) l } ∥ V ( 0 ) ∥ ∞ ≤ 8 β 1 ( 1 − γ ) 3 N + 2 β 1 ( 1 − γ ) 2 N , (103) \alpha_2=\left\{b_2\sum_{l=0}^{m-1}(b_1b_3)^l\right\}\left\|V^{(0)}\right\|_{\infty}\leq\sqrt{\frac{8\beta_1}{(1-\gamma)^3N}}+\frac{2\beta_1}{(1-\gamma)^2N}, \tag*{(103)} α 2 = { b 2 l = 0 ∑ m − 1 ( b 1 b 3 ) l } V ( 0 ) ∞ ≤ ( 1 − γ ) 3 N 8 β 1 + ( 1 − γ ) 2 N 2 β 1 , (103)
provided that N ≥ 64 β 1 1 − γ N\geq\frac{64\beta_1}{1-\gamma} N ≥ 1 − γ 64 β 1 .
Step 6: putting all this together. Taking inequalities (101) , (102) and (103) together shows that, with probability at least 1 − δ 1-\delta 1 − δ ,
∥ V ^ ( 0 ) − V ( 0 ) ∥ ∞ ≤ ( 1 e ) m 2 ( 1 − γ ) 2 β 1 N + 8 β 1 ( 1 − γ ) 3 N + 2 β 1 ( 1 − γ ) 2 N \left\|\widehat{V}^{(0)}-V^{(0)}\right\|_{\infty}\leq\left(\frac{1}{e}\right)^m\frac{2}{(1-\gamma)^2}\sqrt{\frac{\beta_1}{N}}+\sqrt{\frac{8\beta_1}{(1-\gamma)^3N}}+\frac{2\beta_1}{(1-\gamma)^2N} V ( 0 ) − V ( 0 ) ∞ ≤ ( e 1 ) m ( 1 − γ ) 2 2 N β 1 + ( 1 − γ ) 3 N 8 β 1 + ( 1 − γ ) 2 N 2 β 1
holds as long as N ≥ 64 β 1 1 − γ N \ge\frac{64\beta_1}{1-\gamma} N ≥ 1 − γ 64 β 1 . To finish up, set m = ⌈ log ( e 1 − γ ) ⌉ m = \left\lceil\log\left(\frac{e}{1-\gamma}\right)\right\rceil m = ⌈ log ( 1 − γ e ) ⌉ to derive
∥ V ^ π ^ − V π ∥ ∞ = ∥ V ^ ( 0 ) − V ( 0 ) ∥ ∞ ≤ 2 8 β 1 ( 1 − γ ) 3 N + 2 β 1 ( 1 − γ ) 2 N \left\|\widehat{V}^{\hat{\pi}}-V^{\pi}\right\|_{\infty}=\left\|\widehat{V}^{(0)}-V^{(0)}\right\|_{\infty}\leq2\sqrt{\frac{8\beta_1}{(1-\gamma)^3N}}+\frac{2\beta_1}{(1-\gamma)^2N} V π ^ − V π ∞ = V ( 0 ) − V ( 0 ) ∞ ≤ 2 ( 1 − γ ) 3 N 8 β 1 + ( 1 − γ ) 2 N 2 β 1
≤ 3 8 β 1 ( 1 − γ ) 3 N (104) \leq3\sqrt{\frac{8\beta_1}{(1-\gamma)^3N}} \tag*{(104)} ≤ 3 ( 1 − γ ) 3 N 8 β 1 (104)
with probability exceeding 1 − δ 1-\delta 1 − δ , provided that N ≥ β 1 1 − γ N\geq\frac{\beta_1}{1-\gamma} N ≥ 1 − γ β 1 . This concludes the proof by recalling that β 1 = C 0 log ( S m δ ) \beta_1=C_0\log\left(\frac{Sm}{\delta}\right) β 1 = C 0 log ( δ S m ) and m = ⌈ log ( e 1 − γ ) ⌉ m=\left\lceil\log\left(\frac{e}{1-\gamma}\right)\right\rceil m = ⌈ log ( 1 − γ e ) ⌉ .
Model-based policy learning
Having established the effectiveness of the model-based approach for policy evaluation, we now turn attention to model-based policy learning, which seeks to identify a policy that approximately maximizes the value function based on the data samples at hand. To be precise, suppose that the action space A \mathcal{A} A contains A ≥ 2 A\geq2 A ≥ 2 actions, then for any target accuracy level ε > 0 \varepsilon>0 ε > 0 , the aim of policy learning is to compute an ε \varepsilon ε -accurate policy π e s t \pi_{\mathrm{est}} π est obeying
∀ s ∈ S : V π e s t ( s ) ≥ V ∗ ( s ) − ε . \forall s\in\mathcal{S}: \quad V^{\pi_{\mathrm{est}}}(s)\geq V^{\ast}(s)-\varepsilon. ∀ s ∈ S : V π est ( s ) ≥ V ∗ ( s ) − ε .
Akin to the policy evaluation case, the meaningful range of target precision level ε \varepsilon ε is ε ∈ ( 0 , 1 1 − γ ] \varepsilon\in\left(0,\frac{1}{1-\gamma}\right] ε ∈ ( 0 , 1 − γ 1 ] . Naturally, one would hope to achieve policy learning using as few samples as possible.
Algorithms
A model-based policy learning algorithm typically starts by constructing an empirical MDP M ^ \widehat{\mathcal{M}} M based on all collected data samples, mirroring what is used in model-based policy evaluation. The empirical model is then fed into the Bellman recursion — via methods such as policy iteration and value iteration introduced in § 2.4 — to derive the optimal policy for the empirical model.
Algorithm 3.2: Model-based policy learning with a simulator
input: discount factor γ \gamma γ , reward function r r r , sample size per state-action pair N N N .
for ( s , a ) ∈ S × A (s,a) \in\mathcal{S} \times\mathcal{A} ( s , a ) ∈ S × A do /* model estimation based on samples. */
draw N N N independent samples s ( i ) ( s , a ) ∼ i . i . d . P ( ⋅ ∣ s , a ) s^{(i)}(s,a) \overset{\mathrm{i.i.d.}}{\sim} P(\cdot\mid s,a) s ( i ) ( s , a ) ∼ i.i.d. P ( ⋅ ∣ s , a ) .
compute P ^ ( ⋅ ∣ s , a ) \widehat{P}(\cdot\mid s,a) P ( ⋅ ∣ s , a ) based on (105) . /* planning w.r.t. the empirical model. */
output π ^ ⋆ \widehat{\pi}^{\star} π ⋆ as an optimal (deterministic) policy of the MDP M ^ = ( S , A , P ^ , r , γ ) \widehat{\mathcal{M}} = (\mathcal{S}, \mathcal{A}, \widehat{P}, r, \gamma) M = ( S , A , P , r , γ ) .
A model-based approach. For concreteness, let us describe below the most natural version of the model-based policy learning algorithm.
Model estimation. For each state-action pair ( s , a ) ∈ S × A (s,a) \in\mathcal{S} \times\mathcal{A} ( s , a ) ∈ S × A , draw N N N independent samples from the generative model:
s ( i ) ( s , a ) ∼ i . i . d . P ( ⋅ ∣ s , a ) , i = 1 , … , N , s^{(i)}(s,a) \overset{\mathrm{i.i.d.}}{\sim} P(\cdot\mid s,a), \qquad i = 1,\ldots,N, s ( i ) ( s , a ) ∼ i.i.d. P ( ⋅ ∣ s , a ) , i = 1 , … , N ,
and construct an empirical transition kernel P ^ \widehat{P} P as follows
∀ s ′ ∈ S , P ^ ( s ′ ∣ s , a ) = 1 N ∑ i = 1 N 1 { s ′ = s ( i ) ( s , a ) } . (105) \forall s' \in\mathcal{S}, \qquad\widehat{P}(s' \mid s,a) = \frac{1}{N}\sum_{i=1}^{N} \mathbb{1}\{s' = s^{(i)}(s,a)\}. \tag*{(105)} ∀ s ′ ∈ S , P ( s ′ ∣ s , a ) = N 1 i = 1 ∑ N 1 { s ′ = s ( i ) ( s , a )} . (105)
In words, P ^ ( s ′ ∣ s , a ) \widehat{P}(s' \mid s,a) P ( s ′ ∣ s , a ) records the empirical frequency of transitions from ( s , a ) (s,a) ( s , a ) to state s ′ s' s ′ . This leads to an empirical MDP M ^ = ( S , A , P ^ , r , γ ) \widehat{\mathcal{M}} = (\mathcal{S}, \mathcal{A}, \widehat{P}, r, \gamma) M = ( S , A , P , r , γ ) constructed from the data samples.
Planning. Compute an optimal (deterministic) policy
π ^ ⋆ ∈ arg max π V ^ π . (106) \widehat{\pi}^{\star} \in\mathop{\arg\max}_{\pi} \widehat{V}^{\pi}. \tag*{(106)} π ⋆ ∈ arg max π V π . (106)
of the empirical MDP M ^ \widehat{\mathcal{M}} M , using, e.g., policy iteration (see § 2.4 ). Here and below, V ^ π \widehat{V}^{\pi} V π denotes the value function of policy π \pi π in the empirical MDP M ^ \widehat{\mathcal{M}} M .
See Algorithm 3.2 for the full procedure, and Figure 3.1 for an illustration. Clearly, the total number of samples collected in the above algorithm is given by N t o t a l : = N S A N^{\mathrm{total}} := NSA N total := N S A .
Figure 3.1. An illustration of the model-based policy learning algorithm, which first computes an empirical model and then plugs it into a planning algorithm.
Theoretical guarantees
We now state the sample complexity guarantee for the model-based method in Algorithm 3.2. Sharp analyses for model-based policy learning under the generative model that cover the full range of ε \varepsilon ε were first developed by Li et al. (2020) and Li et al. (2024c); their analyses apply to a perturbed version of Algorithm 3.2, in which the reward function is randomly perturbed. More recently, Zurek and Chen (2025) showed that reward perturbation is unnecessary. The guarantee below is established through a refinement of the analysis in Li et al. (2020) and Li et al. (2024c).
Theorem 3.2 (Li et al. (2020), Li et al. (2024c), and Zurek and Chen (2025)). There exists some universal constant c 0 > 0 c_0 > 0 c 0 > 0 such that: for any 0 < δ < 1 0 < \delta< 1 0 < δ < 1 and any 0 < ε ≤ 1 1 − γ 0 < \varepsilon\le\frac{1}{1-\gamma} 0 < ε ≤ 1 − γ 1 , any optimal deterministic policy π ^ ⋆ \widehat{\pi}^{\star} π ⋆ returned by Algorithm 3.2 achieves
∀ s ∈ S , V ^ π ^ ⋆ ( s ) ≥ V ⋆ ( s ) − ε (107) \forall s \in\mathcal{S}, \qquad\widehat{V}^{\widehat{\pi}^{\star}}(s) \ge V^{\star}(s) - \varepsilon \tag*{(107)} ∀ s ∈ S , V π ⋆ ( s ) ≥ V ⋆ ( s ) − ε (107)
∀ ( s , a ) ∈ S × A , Q ^ π ^ ∗ ( s , a ) ≥ Q ∗ ( s , a ) − γ ε (108) \forall(s,a)\in\mathcal{S}\times\mathcal{A},\qquad\widehat{Q}^{\widehat{\pi}^{\ast}}(s,a)\geq Q^{\ast}(s,a)-\gamma\varepsilon \tag*{(108)} ∀ ( s , a ) ∈ S × A , Q π ∗ ( s , a ) ≥ Q ∗ ( s , a ) − γ ε (108)
with probability at least 1 − δ 1-\delta 1 − δ , provided that the sample size per state-action pair exceeds
N ≥ c 0 log ( S A N ( 1 − γ ) 2 ε δ ) ( 1 − γ ) 3 ε 2 . (109) N\geq\frac{c_{0}\log\left(\frac{SAN}{(1-\gamma)^{2}\varepsilon\delta}\right)}{(1-\gamma)^{3}\varepsilon^{2}}. \tag*{(109)} N ≥ ( 1 − γ ) 3 ε 2 c 0 log ( ( 1 − γ ) 2 ε δ S A N ) . (109)
In a nutshell, the above theorem demonstrates that the model-based approach succeeds in finding an ε \varepsilon ε -optimal policy as soon as the total sample complexity exceeds the order of S A ( 1 − γ ) 3 ε 2 \frac{SA}{(1-\gamma)^{3}\varepsilon^{2}} ( 1 − γ ) 3 ε 2 S A (modulo some log factor). This sample complexity is provably minimax optimal, as we will elaborate in § 4.9 . Importantly, this result imposes no restriction on the range of ε \varepsilon ε and, in particular, the accuracy level ε \varepsilon ε is permitted to be as large as 1 1 − γ \frac{1}{1-\gamma} 1 − γ 1 . As a consequence, this theorem is applicable even in regimes with small-to-moderate sample sizes, since its validity is guaranteed as long as
N ≥ Ω ~ ( 1 1 − γ ) . (110) N\geq\widetilde{\Omega}\left(\frac{1}{1-\gamma}\right). \tag*{(110)} N ≥ Ω ( 1 − γ 1 ) . (110)
Noteworthily, Condition (110) is necessary for learning a policy that performs strictly better than a random guess, and hence characterizes the full range of “meaningful” sample sizes. From a computational standpoint, it suffices to compute a deterministic policy π ^ \widehat{\pi} π satisfying ∥ V ^ ∗ − V ^ π ^ ∥ ∞ ≤ 1 N ( 1 − γ ) \lVert\widehat{V}^{\ast}-\widehat{V}^{\widehat{\pi}}\rVert_{\infty}\leq\frac{1}{N(1-\gamma)} ∥ V ∗ − V π ∥ ∞ ≤ N ( 1 − γ ) 1 : the same sample complexity guarantee continues to hold after increasing the universal constant c 0 c_{0} c 0 . Empirical VI and PI achieves this accuracy within O ( 1 1 − γ log ( N 1 − γ ) ) O\left(\frac{1}{1-\gamma}\log\left(\frac{N}{1-\gamma}\right)\right) O ( 1 − γ 1 log ( 1 − γ N ) ) iterations, thereby yielding a computationally lightweight solution.
Analysis: a first attempt
We now proceed to establish performance guarantees for policy learning. Note that it suffices to analyze the suboptimality (107) of π ^ ∗ \widehat{\pi}^{\ast} π ∗ w.r.t. value functions. Clearly, if (107) can be established, then the Bellman equation gives
Q ^ π ^ ∗ ( s , a ) − Q ∗ ( s , a ) = r ( s , a ) + γ E s ′ ∼ P ( ⋅ ∣ s , a ) [ V ^ π ^ ∗ ( s ′ ) ] − ( r ( s , a ) + γ E s ′ ∼ P ( ⋅ ∣ s , a ) [ V ∗ ( s ′ ) ] ) = γ E s ′ ∼ P ( ⋅ ∣ s , a ) [ V ^ π ^ ∗ ( s ′ ) − V ∗ ( s ′ ) ] ≥ − γ ε , \begin{aligned}
\widehat{Q}^{\widehat{\pi}^{\ast}}(s,a)-Q^{\ast}(s,a)=r(s,a)+\gamma\mathbb{E}_{s'\sim P(\cdot\mid s,a)}[\widehat{V}^{\widehat{\pi}^{\ast}}(s')]-\left(r(s,a)+\gamma\mathbb{E}_{s'\sim P(\cdot\mid s,a)}[V^{\ast}(s')]\right) \\
&=\gamma\mathbb{E}_{s'\sim P(\cdot\mid s,a)}[\widehat{V}^{\widehat{\pi}^{\ast}}(s')-V^{\ast}(s')]\geq-\gamma\varepsilon,
\end{aligned} Q π ∗ ( s , a ) − Q ∗ ( s , a ) = r ( s , a ) + γ E s ′ ∼ P ( ⋅ ∣ s , a ) [ V π ∗ ( s ′ )] − ( r ( s , a ) + γ E s ′ ∼ P ( ⋅ ∣ s , a ) [ V ∗ ( s ′ )] ) = γ E s ′ ∼ P ( ⋅ ∣ s , a ) [ V π ∗ ( s ′ ) − V ∗ ( s ′ )] ≥ − γ ε ,
as claimed in (108) .
In this subsection, we begin by examining the performance of the model-based policy learning algorithm in Algorithm 3.2. We first present a simple approach to establishing asymptotically optimal statistical guarantees (for large enough N N N ) for Algorithm 3.2, albeit at the cost of a significant burn-in requirement. For simplicity, in this subsection we assume (without providing a formal proof) that
P ^ ≈ P , (111) \widehat{P}\approx P, \tag*{(111)} P ≈ P , (111)
which holds in the asymptotic regime as N → ∞ N\to\infty N → ∞ , but still leaves substantial room for improvement from a non-asymptotic perspective. In § 4.8.1 , we will relax this assumption (and the large-N N N requirement) and extend the analysis all the way to the most sample-limited regime. Before proceeding, let us introduce some useful notation:
To bound the suboptimality of π ^ ⋆ \widehat{\pi}^{\star} π ⋆ in terms of the value function, we start with the following elementary decomposition:
0 ≤ V ⋆ − V π ^ ⋆ = V π ⋆ − V π ^ ⋆ = ( V π ⋆ − V ^ π ⋆ ) + ( V ^ π ⋆ − V ^ π ^ ⋆ ) + ( V ^ π ^ ⋆ − V π ^ ⋆ ) ≤ ( V π ⋆ − V ^ π ⋆ ) + ( V ^ π ^ ⋆ − V π ^ ⋆ ) , (112) \begin{aligned}
0 &\le V^{\star}-V^{\widehat{\pi}^{\star}}=V^{\pi^{\star}}-V^{\widehat{\pi}^{\star}}\\
&=(V^{\pi^{\star}}-\widehat{V}^{\pi^{\star}})+(\widehat{V}^{\pi^{\star}}-\widehat{V}^{\widehat{\pi}^{\star}})+(\widehat{V}^{\widehat{\pi}^{\star}}-V^{\widehat{\pi}^{\star}})\\
&\le(V^{\pi^{\star}}-\widehat{V}^{\pi^{\star}})+(\widehat{V}^{\widehat{\pi}^{\star}}-V^{\widehat{\pi}^{\star}}), \tag*{(112)}
\end{aligned} 0 ≤ V ⋆ − V π ⋆ = V π ⋆ − V π ⋆ = ( V π ⋆ − V π ⋆ ) + ( V π ⋆ − V π ⋆ ) + ( V π ⋆ − V π ⋆ ) ≤ ( V π ⋆ − V π ⋆ ) + ( V π ⋆ − V π ⋆ ) , (112)
where the last line follows since π ^ ⋆ = arg max π V ^ π \widehat{\pi}^{\star}=\arg\max_{\pi}\widehat{V}^{\pi} π ⋆ = arg max π V π and hence V ^ π ⋆ ≤ V ^ π ^ ⋆ \widehat{V}^{\pi^{\star}}\le\widehat{V}^{\widehat{\pi}^{\star}} V π ⋆ ≤ V π ⋆ . We are thus left with two terms to control (see 3.35), both of which are concerned with certain policy evaluation errors (i.e., the policy evaluation errors w.r.t. π ⋆ \pi^{\star} π ⋆ and π ^ ⋆ \widehat{\pi}^{\star} π ⋆ ).
Step 1: controlling V π ⋆ − V ^ π ⋆ V^{\pi^{\star}}-\widehat{V}^{\pi^{\star}} V π ⋆ − V π ⋆ . Given that π ⋆ \pi^{\star} π ⋆ is the optimal policy of the true MDP and hence independent of the data, we can readily accomplish this step by invoking the policy evaluation result presented in Theorem 3.1. Specifically, taking π = π ⋆ \pi=\pi^{\star} π = π ⋆ in the analysis of Theorem 3.1 (see 3.28) shows that, with probability at least 1 − δ 1-\delta 1 − δ ,
∥ V ^ π ⋆ − V π ⋆ ∥ ∞ ≤ c 1 log ( S δ ( 1 − γ ) ) N ( 1 − γ ) 3 (113) \left\|\widehat{V}^{\pi^{\star}}-V^{\pi^{\star}}\right\|_{\infty}\le c_{1}\sqrt{\frac{\log\left(\frac{S}{\delta(1-\gamma)}\right)}{N(1-\gamma)^{3}}} \tag*{(113)} V π ⋆ − V π ⋆ ∞ ≤ c 1 N ( 1 − γ ) 3 log ( δ ( 1 − γ ) S ) (113)
as long as N ≥ c 2 log ( S δ ( 1 − γ ) ) 1 − γ N\ge\frac{c_{2}\log\left(\frac{S}{\delta(1-\gamma)}\right)}{1-\gamma} N ≥ 1 − γ c 2 l o g ( δ ( 1 − γ ) S ) , where c 1 , c 2 > 0 c_{1},c_{2}>0 c 1 , c 2 > 0 are universal constants.
Step 2: controlling V ^ π ^ ⋆ − V π ^ ⋆ \widehat{V}^{\widehat{\pi}^{\star}}-V^{\widehat{\pi}^{\star}} V π ⋆ − V π ⋆ . Bounding the policy evaluation error V ^ π ^ ⋆ − V π ^ ⋆ \widehat{V}^{\widehat{\pi}^{\star}}-V^{\widehat{\pi}^{\star}} V π ⋆ − V π ⋆ is, however, substantially more complex than bounding V π ⋆ − V ^ π ⋆ V^{\pi^{\star}}-\widehat{V}^{\pi^{\star}} V π ⋆ − V π ⋆ . The complexity stems from the intricate statistical dependency between π ^ ⋆ \widehat{\pi}^{\star} π ⋆ and the data samples, rendering Theorem 3.1 inapplicable. We present here an argument valid in the asymptotic regime as N → ∞ N\to\infty N → ∞ . Since this approach is relatively loose in the non-asymptotic regime — and will be sharpened in Section 3.3.4 — we omit the control of second-order (or higher-order) terms here in order to streamline the presentation and better highlight the main ideas.
Note that the Bellman equation (35) implies that
V ^ π ^ ⋆ − V π ⋆ = ( I − γ P ^ π ^ ⋆ ) − 1 r π ^ ⋆ − V ^ π ⋆ = ( I − γ P ^ π ^ ⋆ ) − 1 { ( I − γ P π ⋆ ) V π ⋆ − ( I − γ P ^ π ^ ⋆ ) V π ⋆ } = γ ( I − γ P ^ π ^ ⋆ ) − 1 ( P ^ π ^ ⋆ − P π ^ ⋆ ) V π ⋆ , (114) \begin{aligned}
\widehat{V}^{\widehat{\pi}^{\star}}-V^{\pi^{\star}}
&=(I-\gamma\widehat{P}_{\widehat{\pi}^{\star}})^{-1}r_{\widehat{\pi}^{\star}}-\widehat{V}^{\pi^{\star}}\\
&=(I-\gamma\widehat{P}_{\widehat{\pi}^{\star}})^{-1}\left\{(I-\gamma P_{\pi^{\star}})V^{\pi^{\star}}-(I-\gamma\widehat{P}_{\widehat{\pi}^{\star}})V^{\pi^{\star}}\right\}\\
&=\gamma(I-\gamma\widehat{P}_{\widehat{\pi}^{\star}})^{-1}(\widehat{P}_{\widehat{\pi}^{\star}}-P_{\widehat{\pi}^{\star}})V^{\pi^{\star}}, \tag*{(114)}
\end{aligned} V π ⋆ − V π ⋆ = ( I − γ P π ⋆ ) − 1 r π ⋆ − V π ⋆ = ( I − γ P π ⋆ ) − 1 { ( I − γ P π ⋆ ) V π ⋆ − ( I − γ P π ⋆ ) V π ⋆ } = γ ( I − γ P π ⋆ ) − 1 ( P π ⋆ − P π ⋆ ) V π ⋆ , (114)
= γ ( I − γ P ^ π ^ ⋆ ) − 1 ( P ^ π ^ ⋆ − P π ^ ⋆ ) V π ⋆ + γ ( I − γ P ^ π ^ ⋆ ) − 1 ( P ^ π ^ ⋆ − P π ^ ⋆ ) ( V ^ ⋆ − V π ⋆ ) (115) \begin{aligned}
&=\gamma(I-\gamma\widehat{P}_{\widehat{\pi}^{\star}})^{-1}(\widehat{P}_{\widehat{\pi}^{\star}}-P_{\widehat{\pi}^{\star}})V^{\pi^{\star}}\\
&\quad+\gamma(I-\gamma\widehat{P}_{\widehat{\pi}^{\star}})^{-1}(\widehat{P}_{\widehat{\pi}^{\star}}-P_{\widehat{\pi}^{\star}})(\widehat{V}^{\star}-V^{\pi^{\star}}) \tag*{(115)}
\end{aligned} = γ ( I − γ P π ⋆ ) − 1 ( P π ⋆ − P π ⋆ ) V π ⋆ + γ ( I − γ P π ⋆ ) − 1 ( P π ⋆ − P π ⋆ ) ( V ⋆ − V π ⋆ ) (115)
≈ γ ( I − γ P ^ π ^ ⋆ ) − 1 ( P ^ π ^ ⋆ − P π ^ ⋆ ) V π ⋆ , (116) \approx\gamma(I-\gamma\widehat{P}_{\widehat{\pi}^{\star}})^{-1}(\widehat{P}_{\widehat{\pi}^{\star}}-P_{\widehat{\pi}^{\star}})V^{\pi^{\star}}, \tag*{(116)} ≈ γ ( I − γ P π ⋆ ) − 1 ( P π ⋆ − P π ⋆ ) V π ⋆ , (116)
where we remind the readers of the matrix notation P π P_{\pi} P π and r π r_{\pi} r π in (28) and (29) , respectively. Informally, the first term in (115) is easier to cope than its counterpart in (114) , since V π ⋆ V^{\pi^{\star}} V π ⋆ is a fixed vector independent from P ^ \widehat{P} P ; meanwhile, the last term in (115) can be regarded as a second-order term (as it involves both P ^ − P \widehat{P}-P P − P and V ^ π ⋆ − V π ⋆ \widehat{V}^{\pi^{\star}}-V^{\pi^{\star}} V π ⋆ − V π ⋆ ), and is therefore expected to be asymptotically negligible as N → ∞ N\to\infty N → ∞ , a key observation that yields (116) .
To bound (116) , it is first seen from (28) and the Bernstein inequality in Lemma 2.5 that, with probability greater than 1 − δ 1-\delta 1 − δ ,
∣ ( P ^ π ⋆ − P π ⋆ ) V π ⋆ ∣ = ∣ Π π ⋆ ( P ^ − P ) V π ⋆ ∣ ≤ Π π ⋆ ( 2 log ( 4 S A δ ) N Var P ( V π ⋆ ) + 2 ∥ V π ⋆ ∥ ∞ log ( 4 S A δ ) 3 N 1 S A ) ≤ 2 log ( 4 S A δ ) N Var P π ⋆ ( V π ⋆ ) + 2 log ( 4 S A δ ) 3 N ( 1 − γ ) 1 S ≈ 2 log ( 4 S A δ ) N Var P π ⋆ ( V π ⋆ ) , (117) \begin{aligned}
\left|(\widehat{P}_{\pi^\star}-P_{\pi^\star})V^{\pi^\star}\right|
&=\left|\Pi_{\pi^\star}(\widehat{P}-P)V^{\pi^\star}\right|\\
&\leq\Pi_{\pi^\star}\left(\sqrt{\frac{2\log\left(\frac{4SA}{\delta}\right)}{N}}\sqrt{\operatorname{Var}_{P}(V^{\pi^\star})}+\frac{2\|V^{\pi^\star}\|_{\infty}\log\left(\frac{4SA}{\delta}\right)}{3N}\mathbf{1}_{SA}\right)\\
&\leq\sqrt{\frac{2\log\left(\frac{4SA}{\delta}\right)}{N}}\sqrt{\operatorname{Var}_{P_{\pi^\star}}(V^{\pi^\star})}+\frac{2\log\left(\frac{4SA}{\delta}\right)}{3N(1-\gamma)}\mathbf{1}_{S}\\
&\approx\sqrt{\frac{2\log\left(\frac{4SA}{\delta}\right)}{N}}\sqrt{\operatorname{Var}_{P_{\pi^\star}}(V^{\pi^\star})}, \tag*{(117)}
\end{aligned} ( P π ⋆ − P π ⋆ ) V π ⋆ = Π π ⋆ ( P − P ) V π ⋆ ≤ Π π ⋆ N 2 log ( δ 4 S A ) Var P ( V π ⋆ ) + 3 N 2∥ V π ⋆ ∥ ∞ log ( δ 4 S A ) 1 S A ≤ N 2 log ( δ 4 S A ) Var P π ⋆ ( V π ⋆ ) + 3 N ( 1 − γ ) 2 log ( δ 4 S A ) 1 S ≈ N 2 log ( δ 4 S A ) Var P π ⋆ ( V π ⋆ ) , (117)
where the penultimate line holds since ∥ V π ⋆ ∥ ∞ ≤ 1 1 − γ \|V^{\pi^\star}\|_{\infty}\leq\frac{1}{1-\gamma} ∥ V π ⋆ ∥ ∞ ≤ 1 − γ 1 , and in the last line we omit the higher-order term proportional to 1 / N 1/N 1/ N (as the other term scales with 1 / N 1/\sqrt{N} 1/ N ). In view of the non-negativity of ( I − γ P ^ π ⋆ ) − 1 (I-\gamma\widehat{P}_{\pi^\star})^{-1} ( I − γ P π ⋆ ) − 1 (cf. Lemma 2.3), it follows from (117) that
γ ∣ ( I − γ P ^ π ⋆ ) − 1 ( P ^ π ⋆ − P π ⋆ ) V π ⋆ ∣ ≤ γ ( I − γ P ^ π ⋆ ) − 1 ∣ ( P ^ π ⋆ − P π ⋆ ) V π ⋆ ∣ ≈ γ ( I − γ P π ⋆ ) − 1 ∣ ( P ^ π ⋆ − P π ⋆ ) V π ⋆ ∣ ≲ γ 2 log ( 4 S A δ ) N ( I − γ P π ⋆ ) − 1 Var P π ⋆ ( V π ⋆ ) ≈ γ 2 log ( 4 S A δ ) N ( I − γ P π ⋆ ) − 1 Var P π ⋆ ( V π ^ ⋆ ) (118) \begin{aligned}
\gamma\left|(I-\gamma\widehat{P}_{\pi^\star})^{-1}(\widehat{P}_{\pi^\star}-P_{\pi^\star})V^{\pi^\star}\right|
&\leq\gamma(I-\gamma\widehat{P}_{\pi^\star})^{-1}\left|(\widehat{P}_{\pi^\star}-P_{\pi^\star})V^{\pi^\star}\right|\\
&\approx\gamma(I-\gamma P_{\pi^\star})^{-1}\left|(\widehat{P}_{\pi^\star}-P_{\pi^\star})V^{\pi^\star}\right|\\
&\lesssim\gamma\sqrt{\frac{2\log\left(\frac{4SA}{\delta}\right)}{N}}(I-\gamma P_{\pi^\star})^{-1}\sqrt{\operatorname{Var}_{P_{\pi^\star}}(V^{\pi^\star})}\\
&\approx\gamma\sqrt{\frac{2\log\left(\frac{4SA}{\delta}\right)}{N}}(I-\gamma P_{\pi^\star})^{-1}\sqrt{\operatorname{Var}_{P_{\pi^\star}}(V^{\widehat{\pi}^\star})} \tag*{(118)}
\end{aligned} γ ( I − γ P π ⋆ ) − 1 ( P π ⋆ − P π ⋆ ) V π ⋆ ≤ γ ( I − γ P π ⋆ ) − 1 ( P π ⋆ − P π ⋆ ) V π ⋆ ≈ γ ( I − γ P π ⋆ ) − 1 ( P π ⋆ − P π ⋆ ) V π ⋆ ≲ γ N 2 log ( δ 4 S A ) ( I − γ P π ⋆ ) − 1 Var P π ⋆ ( V π ⋆ ) ≈ γ N 2 log ( δ 4 S A ) ( I − γ P π ⋆ ) − 1 Var P π ⋆ ( V π ⋆ ) (118)
≤ 4 log ( 4 S A δ ) ( 1 − γ ) N ∥ V π ^ ⋆ ∥ ∞ 1 S ≤ 4 log ( 4 S A δ ) ( 1 − γ ) 3 N 1 S (119) \leq\sqrt{\frac{4\log\left(\frac{4SA}{\delta}\right)}{(1-\gamma)N}}\|V^{\widehat{\pi}^\star}\|_{\infty}\mathbf{1}_{S}
\leq\sqrt{\frac{4\log\left(\frac{4SA}{\delta}\right)}{(1-\gamma)^3N}}\mathbf{1}_{S} \tag*{(119)} ≤ ( 1 − γ ) N 4 log ( δ 4 S A ) ∥ V π ⋆ ∥ ∞ 1 S ≤ ( 1 − γ ) 3 N 4 log ( δ 4 S A ) 1 S (119)
with probability exceeding 1 − δ 1-\delta 1 − δ , where the approximation in the second and the fourth lines relies on the assumption (111) (and hence V π ^ ⋆ ≈ V π ⋆ V^{\widehat{\pi}^\star}\approx V^{\pi^\star} V π ⋆ ≈ V π ⋆ ), and the last line invokes (91) and the basic property ∥ V π ^ ⋆ ∥ ∞ ≤ 1 1 − γ \|V^{\widehat{\pi}^\star}\|_{\infty}\leq\frac{1}{1-\gamma} ∥ V π ⋆ ∥ ∞ ≤ 1 − γ 1 . We remark that the approximation in (118) serves to align the policy associated with the value function and the one associated with the transition kernel, thereby enabling the application of (91) . It is worth emphasizing that the use of property (91) here is crucial for obtaining the asymptotically sharp bound, as it carefully leverages the underlying variance structure to sharpen the horizon dependency.
Step 3: putting all this together. To finish up, substituting (113) , (116) and (119) into (112) , we can conclude that
∥ V ⋆ − V π ^ ⋆ ∥ ∞ ≲ log ( S A δ ( 1 − γ ) ) ( 1 − γ ) 3 N \|V^{\star}-V^{\widehat{\pi}^\star}\|_{\infty}\lesssim\sqrt{\frac{\log\left(\frac{SA}{\delta(1-\gamma)}\right)}{(1-\gamma)^3N}} ∥ V ⋆ − V π ⋆ ∥ ∞ ≲ ( 1 − γ ) 3 N log ( δ ( 1 − γ ) S A )
holds with high probability when N → ∞ N\to\infty N → ∞ .
Nevertheless, while the arguments above — similar in spirit to those used in Azar et al. (2013) — suffice to establish the desirable statistical bound for the model-based algorithm asymptotically, they rely on an excessively large sample size requirement on N N N ; for instance, working out the full details may require N N N to be at least polynomial in S S S . To overcome this limitation, a substantially more refined analysis strategy is needed, which we introduce next.
Sharp analysis for Theorem 3.2: leave-one-out analysis
We now present a refined analysis for establishing sharp statistical bounds for ∥ V ∗ − V ^ π ^ ∗ ∥ ∞ \lVert V^*-\widehat{V}^{\widehat{\pi}^*}\rVert_\infty ∥ V ∗ − V π ∗ ∥ ∞ . In light of the decomposition (112) and the bound (113) , it suffices to derive a tight bound on ∥ V ^ π ^ ∗ − V π ^ ∗ ∥ ∞ \lVert\widehat{V}^{\widehat{\pi}^*}-V^{\widehat{\pi}^*}\rVert_\infty ∥ V π ∗ − V π ∗ ∥ ∞ that remains valid even when N N N is small or moderately sized.
Leave-one-out analysis: key ideas. As alluded to previously, the most challenging part for bounding ∥ V ^ π ^ ∗ − V π ^ ∗ ∥ ∞ \lVert\widehat{V}^{\widehat{\pi}^*}-V^{\widehat{\pi}^*}\rVert_\infty ∥ V π ∗ − V π ∗ ∥ ∞ lies in coping with the intricate statistical dependency between π ^ ∗ \widehat{\pi}^* π ∗ and P ^ \widehat{P} P . To better illustrate this issue, we use the following alternative form of the value-estimation identity in (114) :
V ^ π ^ ∗ − V π ^ ∗ = γ ( I − γ P π ^ ∗ ) − 1 ( P ^ π ^ ∗ − P π ^ ∗ ) V ^ ∗ . (120) \widehat{V}^{\widehat{\pi}^*}-V^{\widehat{\pi}^*}=\gamma(I-\gamma P_{\widehat{\pi}^*})^{-1}(\widehat{P}_{\widehat{\pi}^*}-P_{\widehat{\pi}^*})\widehat{V}^*. \tag*{(120)} V π ∗ − V π ∗ = γ ( I − γ P π ∗ ) − 1 ( P π ∗ − P π ∗ ) V ∗ . (120)
Proof of (120) . We begin with the Bellman equations:
V ^ π ^ ∗ = r π ^ ∗ + γ P ^ π ^ ∗ V ^ π ^ ∗ , V π ^ ∗ = r π ^ ∗ + γ P π ^ ∗ V π ^ ∗ . \widehat{V}^{\widehat{\pi}^*}=r_{\widehat{\pi}^*}+\gamma\widehat{P}_{\widehat{\pi}^*}\widehat{V}^{\widehat{\pi}^*}, \qquad V^{\widehat{\pi}^*}=r_{\widehat{\pi}^*}+\gamma P_{\widehat{\pi}^*}V^{\widehat{\pi}^*}. V π ∗ = r π ∗ + γ P π ∗ V π ∗ , V π ∗ = r π ∗ + γ P π ∗ V π ∗ .
Subtracting the two Bellman equations gives
V ^ π ^ ∗ − V π ^ ∗ = γ P ^ π ^ ∗ V ^ π ^ ∗ − γ P π ^ ∗ V π ^ ∗ = γ ( P ^ π ^ ∗ − P π ^ ∗ ) V ^ π ^ ∗ + γ P π ^ ∗ ( V ^ π ^ ∗ − V π ^ ∗ ) . \widehat{V}^{\widehat{\pi}^*}-V^{\widehat{\pi}^*}=\gamma\widehat{P}_{\widehat{\pi}^*}\widehat{V}^{\widehat{\pi}^*}-\gamma P_{\widehat{\pi}^*}V^{\widehat{\pi}^*}=\gamma(\widehat{P}_{\widehat{\pi}^*}-P_{\widehat{\pi}^*})\widehat{V}^{\widehat{\pi}^*}+\gamma P_{\widehat{\pi}^*}(\widehat{V}^{\widehat{\pi}^*}-V^{\widehat{\pi}^*}). V π ∗ − V π ∗ = γ P π ∗ V π ∗ − γ P π ∗ V π ∗ = γ ( P π ∗ − P π ∗ ) V π ∗ + γ P π ∗ ( V π ∗ − V π ∗ ) .
Rearranging terms, we are left with
( I − γ P π ^ ∗ ) ( V ^ π ^ ∗ − V π ^ ∗ ) = γ ( P ^ π ^ ∗ − P π ^ ∗ ) V ^ π ^ ∗ . (I-\gamma P_{\widehat{\pi}^*})(\widehat{V}^{\widehat{\pi}^*}-V^{\widehat{\pi}^*})=\gamma(\widehat{P}_{\widehat{\pi}^*}-P_{\widehat{\pi}^*})\widehat{V}^{\widehat{\pi}^*}. ( I − γ P π ∗ ) ( V π ∗ − V π ∗ ) = γ ( P π ∗ − P π ∗ ) V π ∗ .
This together with V ^ π ^ ∗ = V ^ ∗ \widehat{V}^{\widehat{\pi}^*}=\widehat{V}^* V π ∗ = V ∗ concludes the proof.
Remark 3.1. The decomposition (120) differs from the one used in Li et al. (2024c), namely, (114) . The alternative form adopted here is particularly well suited to the leave-one-out argument developed below and, together with a self-bounding variance analysis, avoids the separation and reward-perturbation arguments used in Li et al. (2024c).
Notably, given that both π ^ ∗ \widehat{\pi}^* π ∗ and V ^ ∗ \widehat{V}^* V ∗ depend on the empirical transition kernel P ^ \widehat{P} P , one cannot directly invoke the Bernstein inequality (or other standard concentration inequalities) to control the relevant quantity
( P ^ π ^ ∗ − P π ^ ∗ ) V ^ ∗ . (\widehat{P}_{\widehat{\pi}^*}-P_{\widehat{\pi}^*})\widehat{V}^*. ( P π ∗ − P π ∗ ) V ∗ .
To remedy this issue, we resort to a powerful decoupling technique known as the “leave-one-out analysis”. Originally developed in the high-dimensional probability and statistics literature (e.g., El Karoui (2018), Ma et al. (2020), and Chen et al. (2021c)), the leave-one-out analysis was first introduced to RL theory by Agarwal et al. (2019b), and later on refined by Li et al. (2020) and Li et al. (2024c) to sharpen the sample complexity analysis. At a high level, this approach consists of the following key ingredients:
Construction of leave-one-out auxiliary MDPs. For each state s s s , construct a finite collection of auxiliary MDPs whose transition kernels coincide with the original empirical kernel P ^ \widehat{P} P , except that all transition rows associated with state s s s are replaced by absorbing transitions. The reward at state s s s is adjusted through a scalar parameter u u u . We refer to these MDPs as “leave-one-out MDPs”, as each of them effectively drops one block of data samples associated with state s s s when estimating the transition kernel.
Stability of optimal value functions. Since leave-one-out auxiliary MDPs represent a slight perturbation of the original empirical MDP, at least one of the resulting optimal value functions is often expected to be exceedingly close to the optimal value function of the original empirical MDP. This reflects a form of “stability” in leave-one-out perturbations, a property typically required in leave-one-out decoupling arguments.
Harnessing local statistical independence in leave-one-out MDPs. Since a leave-one-out MDP is built without samples collected for, say, state s s s , it is independent of the empirical transition vector P ^ s , a \widehat{P}_{s,a} P s , a for every a ∈ A a \in\mathcal{A} a ∈ A . As a result, the ( s , a ) (s,a) ( s , a ) -th component of the term ( P ^ − P ) V ^ ⋆ (\widehat{P}-P)\widehat{V}^{\star} ( P − P ) V ⋆ can be bounded using the following observation
( P ^ s , a − P s , a ) V ^ ⋆ ≈ ( P ^ s , a − P s , a ) V ^ l o o ⋆ , (121) (\widehat{P}_{s,a}-P_{s,a})\widehat{V}^{\star} \approx(\widehat{P}_{s,a}-P_{s,a})\widehat{V}^{\star}_{\mathrm{loo}}, \tag*{(121)} ( P s , a − P s , a ) V ⋆ ≈ ( P s , a − P s , a ) V loo ⋆ , (121)
Such independence enables the application of standard concentration inequalities to control (121) , which is crucial in analyzing (120) despite complicated statistical dependency.
In what follows, we present a proof that formalizes the above high-level ideas.
Preliminaries: absorbing MDPs with identical optimal value functions. Before diving into the main analysis, we first explain how, given any MDP M = ( S , A , P , r , γ ) \mathcal{M}=(\mathcal{S},\mathcal{A},P,r,\gamma) M = ( S , A , P , r , γ ) , one can construct an absorbing MDP while preserving the same optimal value function.
Concretely, consider any state s s s and any scalar u ∈ [ 0 , 1 / ( 1 − γ ) ] u \in[0,1/(1-\gamma)] u ∈ [ 0 , 1/ ( 1 − γ )] . We construct an auxiliary MDP with the same state space S \mathcal{S} S , action space A \mathcal{A} A and discount factor γ \gamma γ — denoted by M s , u \mathcal{M}_{s,u} M s , u — that resembles the original MDP M \mathcal{M} M except that state s s s becomes absorbing under every action. More precisely, its transition kernel is taken to be
P M s , u ( s ∣ s , a ) = 1 , for all a ∈ A , P M s , u ( s ′ ∣ s , a ) = 0 , for all s ′ ≠ s , a ∈ A , P M s , u ( ⋅ ∣ s ′ , a ) = P ( ⋅ ∣ s ′ , a ) , for all s ′ ≠ s , a ∈ A , (122) \begin{aligned}
P_{\mathcal{M}_{s,u}}(s\mid s,a)&=1, &&\text{for all }a\in\mathcal{A},\\
P_{\mathcal{M}_{s,u}}(s'\mid s,a)&=0, &&\text{for all }s'\ne s,\ a\in\mathcal{A},\\
P_{\mathcal{M}_{s,u}}(\cdot\mid s',a)&=P(\cdot\mid s',a), &&\text{for all }s'\ne s,\ a\in\mathcal{A}, \tag*{(122)}
\end{aligned} P M s , u ( s ∣ s , a ) P M s , u ( s ′ ∣ s , a ) P M s , u ( ⋅ ∣ s ′ , a ) = 1 , = 0 , = P ( ⋅ ∣ s ′ , a ) , for all a ∈ A , for all s ′ = s , a ∈ A , for all s ′ = s , a ∈ A , (122)
where the first two lines in (122) ensure that state s s s is absorbing under every action a a a . Meanwhile, the reward function of M s , u \mathcal{M}_{s,u} M s , u — denoted by r M s , u r_{\mathcal{M}_{s,u}} r M s , u — is given by
r M s , u ( s , a ) = ( 1 − γ ) u , for all a ∈ A , r M s , u ( s ′ , a ) = r ( s ′ , a ) , for all s ′ ≠ s , a ∈ A , (123) \begin{aligned}
r_{\mathcal{M}_{s,u}}(s,a)&=(1-\gamma)u, &&\text{for all }a\in\mathcal{A},\\
r_{\mathcal{M}_{s,u}}(s',a)&=r(s',a), &&\text{for all }s'\ne s,\ a\in\mathcal{A}, \tag*{(123)}
\end{aligned} r M s , u ( s , a ) r M s , u ( s ′ , a ) = ( 1 − γ ) u , = r ( s ′ , a ) , for all a ∈ A , for all s ′ = s , a ∈ A , (123)
so that the immediate rewards coincide with those of M \mathcal{M} M everywhere except at state s s s . For notational convenience, let V s , u ⋆ V^\star_{s,u} V s , u ⋆ represent the optimal value function in M s , u \mathcal{M}_{s,u} M s , u . Since the immediate rewards remain in [ 0 , 1 ] [0,1] [ 0 , 1 ] , one has
0 ≤ V s , u ⋆ ≤ 1 1 − γ 1 S . 0\le V^\star_{s,u}\le\frac{1}{1-\gamma}\mathbf{1}_\mathcal{S}. 0 ≤ V s , u ⋆ ≤ 1 − γ 1 1 S .
By properly choosing u u u , we can ensure that M s , u \mathcal{M}_{s,u} M s , u and M \mathcal{M} M share the same optimal value function, as formalized in the following lemma.
Lemma 3.3. Setting u ⋆ = V ⋆ ( s ) u^\star=V^\star(s) u ⋆ = V ⋆ ( s ) , one has
V s , u ⋆ ⋆ = V ⋆ . (124) V^\star_{s,u^\star}=V^\star. \tag*{(124)} V s , u ⋆ ⋆ = V ⋆ . (124)
The proof of Lemma 3.3 consists of comparing the Bellman optimality equations of M \mathcal{M} M and M s , u ⋆ \mathcal{M}_{s,u^\star} M s , u ⋆ , and is deferred to § 5.2.1 . In summary, for each state s s s , we can form an s s s -absorbing MDP M s , u ⋆ \mathcal{M}_{s,u^\star} M s , u ⋆ that shares the same optimal value function as the original MDP M \mathcal{M} M . Moreover, it decouples the randomness arising from state s s s , as we will soon demonstrate.
Step 1: constructing leave-one-out auxiliary MDPs. As alluded to previously, the leave-one-out analysis employed here requires constructing a collection of leave-one-out auxiliary MDPs. Concretely, for each state s s s and each scalar u ∈ [ 0 , 1 1 − γ ] u \in[0,\frac{1}{1-\gamma}] u ∈ [ 0 , 1 − γ 1 ] , we can apply the method described above (see (122) and (123) ) to construct an s s s -absorbing MDP — denoted by M ^ s , u \widehat{\mathcal{M}}_{s,u} M s , u — associated with the empirical MDP M ^ \widehat{\mathcal{M}} M . In other words, the transition kernel of M ^ s , u \widehat{\mathcal{M}}_{s,u} M s , u coincides with P ^ \widehat{P} P except that all rows associated with state s s s are made absorbing, while the rewards at state s s s are replaced by ( 1 − γ ) u (1-\gamma)u ( 1 − γ ) u .
Recall that each row P ^ s , a \widehat{P}_{s,a} P s , a is derived from samples associated with ( s , a ) (s,a) ( s , a ) , whereas in M ^ s , u \widehat{\mathcal{M}}_{s,u} M s , u , this row is replaced by the s s s -th standard basis vector. Consequently, for any fixed u u u , the MDP M ^ s , u \widehat{\mathcal{M}}_{s,u} M s , u and its optimal value function V ^ s , u ⋆ \widehat{V}^{\star}_{s,u} V s , u ⋆ are independent of P ^ s , a \widehat{P}_{s,a} P s , a for every a ∈ A a \in\mathcal{A} a ∈ A . Moreover, it follows from Lemma 3.3 that V ^ s , V ^ ⋆ ( s ) ⋆ = V ^ ⋆ \widehat{V}^{\star}_{s,\widehat{V}^{\star}(s)} = \widehat{V}^{\star} V s , V ⋆ ( s ) ⋆ = V ⋆ .
Step 2: linking V ^ ⋆ \widehat{V}^{\star} V ⋆ with a small set of value functions independent of P ^ s , a \widehat{P}_{s,a} P s , a . With the above construction in place, one might be tempted to use M ^ s , V ^ ⋆ ( s ) \widehat{\mathcal{M}}_{s,\widehat{V}^{\star}(s)} M s , V ⋆ ( s ) as a surrogate for M ^ \widehat{\mathcal{M}} M , since the two share identical optimal value functions. However, a subtle issue arises: M ^ s , V ^ ⋆ ( s ) \widehat{\mathcal{M}}_{s,\widehat{V}^{\star}(s)} M s , V ⋆ ( s ) is no longer statistically independent of P ^ s , a \widehat{P}_{s,a} P s , a , because the choice of u ^ ⋆ = V ^ ⋆ ( s ) \widehat{u}^{\star} = \widehat{V}^{\star}(s) u ⋆ = V ⋆ ( s ) explicitly depends on P ^ s , a \widehat{P}_{s,a} P s , a . Unfortunately, this dependence undermines the key advantage of the leave-one-out construction described earlier.
Fortunately, this statistical dependency is encapsulated in a single scalar parameter u ^ ⋆ \widehat{u}^{\star} u ⋆ . Inspired by techniques from high-dimensional probability (Vershynin, 2018), a plausible strategy is to invoke a standard covering argument: construct a finite, yet sufficiently dense, cover of the feasible range of u ^ ⋆ \widehat{u}^{\star} u ⋆ (i.e., [ 0 , 1 1 − γ ] [0,\frac{1}{1-\gamma}] [ 0 , 1 − γ 1 ] ), and then approximate M ^ s , u ^ ⋆ \widehat{\mathcal{M}}_{s,\widehat{u}^{\star}} M s , u ⋆ using a group of auxiliary MDPs { M ^ s , u } \{\widehat{\mathcal{M}}_{s,u}\} { M s , u } with u u u selected from this cover.
To formalize this idea, form an epsilon-net of [ 0 , 1 1 − γ ] [0,\frac{1}{1-\gamma}] [ 0 , 1 − γ 1 ] with precision ϵ = 1 N ( 1 − γ ) \epsilon= \frac{1}{N(1-\gamma)} ϵ = N ( 1 − γ ) 1 :
N ϵ : = { 0 , ϵ , 2 ϵ , … , N ϵ } , ϵ = 1 N ( 1 − γ ) . (125) \mathcal{N}_{\epsilon} := \{0,\epsilon,2\epsilon,\ldots,N\epsilon\}, \qquad\epsilon= \frac{1}{N(1-\gamma)}. \tag*{(125)} N ϵ := { 0 , ϵ , 2 ϵ , … , N ϵ } , ϵ = N ( 1 − γ ) 1 . (125)
This epsilon-net has cardinality N + 1 N+1 N + 1 and is independent of the samples. For each u ∈ N ϵ u \in\mathcal{N}_{\epsilon} u ∈ N ϵ , we construct an s s s -absorbing MDP M ^ s , u \widehat{\mathcal{M}}_{s,u} M s , u as described previously. Since N ϵ \mathcal{N}_{\epsilon} N ϵ is statistically independent of P ^ s , a \widehat{P}_{s,a} P s , a for any fixed ϵ \epsilon ϵ , each M ^ s , u \widehat{\mathcal{M}}_{s,u} M s , u with u ∈ N ϵ u \in\mathcal{N}_{\epsilon} u ∈ N ϵ remains statistically independent of P ^ s , a \widehat{P}_{s,a} P s , a . Notably, one can find a point u ∈ N ϵ u \in\mathcal{N}_{\epsilon} u ∈ N ϵ such that the optimal value function of its associated absorbing MDP is ϵ \epsilon ϵ -close to that of the original empirical MDP, as asserted by the lemma below.
Lemma 3.4. For each state s ∈ S s \in\mathcal{S} s ∈ S , there exists a point u 0 ∈ N ϵ u_0 \in\mathcal{N}_{\epsilon} u 0 ∈ N ϵ such that
∥ V ^ ⋆ − V ^ s , u 0 ⋆ ∥ ∞ ≤ ϵ . (126) \left\|\widehat{V}^{\star}-\widehat{V}^{\star}_{s,u_0}\right\|_{\infty}\leq\epsilon. \tag*{(126)} V ⋆ − V s , u 0 ⋆ ∞ ≤ ϵ . (126)
The proof of this lemma is postponed to Section 3.6.2. Importantly, this lemma unveils that, analyzing V ^ ⋆ \widehat{V}^{\star} V ⋆ reduces to studying a small collection of leave-one-out MDPs that are statistically independent of P ^ s , a \widehat{P}_{s,a} P s , a . The entire statistical dependence of V ^ ⋆ \widehat{V}^{\star} V ⋆ on P ^ s , a \widehat{P}_{s,a} P s , a is thus encapsulated in the choice of an auxiliary MDP from this small collection. This insight paves the way for effective decoupling of statistical dependencies.
Step 3: establishing optimal value estimation accuracy. Armed with these leave-one-out auxiliary MDPs, we are ready to derive the desired statistical bound. More concretely, we intend to establish the following lemma.
Lemma 3.5. Given 0 < δ < 1 0 < \delta< 1 0 < δ < 1 , let
β 1 = 2 log ( 16 S A N ( 1 − γ ) δ ) . (127) \beta_1 = 2 \log\left(\frac{16SAN}{(1-\gamma)\delta}\right). \tag*{(127)} β 1 = 2 log ( ( 1 − γ ) δ 16 S A N ) . (127) With probability at least 1 − δ 1-\delta 1 − δ , it holds that
∥ V ^ π ^ ⋆ − V π ^ ⋆ ∥ ∞ ≤ c 1 2 β 1 N ( 1 − γ ) 3 , (128) \lVert\widehat{V}^{\widehat{\pi}^\star}-V^{\widehat{\pi}^\star}\rVert_\infty\le c_1\sqrt{\frac{2\beta_1}{N(1-\gamma)^3}}, \tag*{(128)} ∥ V π ⋆ − V π ⋆ ∥ ∞ ≤ c 1 N ( 1 − γ ) 3 2 β 1 , (128) 0 ≤ V ⋆ − V π ^ ⋆ ≤ 2 c 1 β 1 N ( 1 − γ ) 3 1 S , (129) 0 \le V^\star- V^{\widehat{\pi}^\star} \le2c_1\sqrt{\frac{\beta_1}{N(1-\gamma)^3}}\mathbf{1}_S, \tag*{(129)} 0 ≤ V ⋆ − V π ⋆ ≤ 2 c 1 N ( 1 − γ ) 3 β 1 1 S , (129) provided that N ≥ c 0 β 1 1 − γ N \ge\frac{c_0\beta_1}{1-\gamma} N ≥ 1 − γ c 0 β 1 , where c 0 , c 1 > 0 c_0,c_1 > 0 c 0 , c 1 > 0 are sufficiently large universal constants.
Proof of Lemma 3.5. Once the first bound (128) is proven, then the second bound follows by combining it with (113) and the decomposition (112) , where we apply (113) with failure probability δ / 2 \delta/2 δ /2 and note that β 1 ≥ log ( 2 S δ ( 1 − γ ) ) \beta_1 \ge\log\left(\frac{2S}{\delta(1-\gamma)}\right) β 1 ≥ log ( δ ( 1 − γ ) 2 S ) . Thus, it suffices to establish that the first bound (128) holds with probability at least 1 − δ / 2 1-\delta/2 1 − δ /2 .
To begin, for each state-action pair ( s , a ) (s,a) ( s , a ) , the set { V ^ s , u ⋆ } u ∈ N ϵ \{\widehat{V}^{\star}_{s,u}\}_{u\in\mathcal{N}_{\epsilon}} { V s , u ⋆ } u ∈ N ϵ is independent of P ^ s , a \widehat{P}_{s,a} P s , a . The Bernstein inequality (see Lemma 2.5) taken together with a union bound implies that, with probability at least 1 − δ / 2 1-\delta/2 1 − δ /2 ,
∣ ( P ^ s , a − P s , a ) V ^ s , u ⋆ ∣ ≤ β 1 N Var P s , a [ V ^ s , u ⋆ ] + β 1 3 N ( 1 − γ ) (130) \left|(\widehat{P}_{s,a}-P_{s,a})\widehat{V}^{\star}_{s,u}\right| \le\sqrt{\frac{\beta_1}{N}}\sqrt{\operatorname{Var}_{P_{s,a}}[\widehat{V}^{\star}_{s,u}]}+\frac{\beta_1}{3N(1-\gamma)} \tag*{(130)} ( P s , a − P s , a ) V s , u ⋆ ≤ N β 1 Var P s , a [ V s , u ⋆ ] + 3 N ( 1 − γ ) β 1 (130)
holds simultaneously for all ( s , a ) ∈ S × A (s,a)\in\mathcal{S}\times\mathcal{A} ( s , a ) ∈ S × A and all u ∈ N ϵ u\in\mathcal{N}_{\epsilon} u ∈ N ϵ . Here, we have used ∥ V ^ s , u ⋆ ∥ ∞ ≤ 1 1 − γ \lVert\widehat{V}^{\star}_{s,u}\rVert_\infty\le\frac{1}{1-\gamma} ∥ V s , u ⋆ ∥ ∞ ≤ 1 − γ 1 and ∣ N ϵ ∣ = N + 1 |\mathcal{N}_{\epsilon}|=N+1 ∣ N ϵ ∣ = N + 1 .
For each s s s , Lemma 3.4 guarantees that there exists a u 0 ∈ N ϵ u_0\in\mathcal{N}_{\epsilon} u 0 ∈ N ϵ such that ∥ V ^ ⋆ − V ^ s , u 0 ⋆ ∥ ∞ ≤ ϵ \lVert\widehat{V}^{\star}-\widehat{V}^{\star}_{s,u_0}\rVert_\infty\le\epsilon ∥ V ⋆ − V s , u 0 ⋆ ∥ ∞ ≤ ϵ . Invoking this important fact and the triangle inequality, we obtain
∣ ( P ^ s , a − P s , a ) V ^ ⋆ ∣ ≤ ∣ ( P ^ s , a − P s , a ) V ^ s , u 0 ⋆ ∣ + ∣ ( P ^ s , a − P s , a ) ( V ^ ⋆ − V ^ s , u 0 ⋆ ) ∣ ≤ ∣ ( P ^ s , a − P s , a ) V ^ s , u 0 ⋆ ∣ + 2 ϵ ≤ β 1 N Var P s , a [ V ^ s , u 0 ⋆ ] + β 1 3 N ( 1 − γ ) + 2 ϵ ≤ β 1 N Var P s , a [ V ^ ⋆ ] + ( 2 + β 1 N ) ϵ + β 1 3 N ( 1 − γ ) ≤ β 1 N Var P s , a [ V ^ ⋆ ] + 2 β 1 N ( 1 − γ ) . (131) \begin{aligned}
\left|(\widehat{P}_{s,a}-P_{s,a})\widehat{V}^{\star}\right|
&\le\left|(\widehat{P}_{s,a}-P_{s,a})\widehat{V}^{\star}_{s,u_0}\right|+\left|(\widehat{P}_{s,a}-P_{s,a})(\widehat{V}^{\star}-\widehat{V}^{\star}_{s,u_0})\right|\\
&\le\left|(\widehat{P}_{s,a}-P_{s,a})\widehat{V}^{\star}_{s,u_0}\right|+2\epsilon\\
&\le\sqrt{\frac{\beta_1}{N}}\sqrt{\operatorname{Var}_{P_{s,a}}[\widehat{V}^{\star}_{s,u_0}]}+\frac{\beta_1}{3N(1-\gamma)}+2\epsilon\\
&\le\sqrt{\frac{\beta_1}{N}}\sqrt{\operatorname{Var}_{P_{s,a}}[\widehat{V}^{\star}]}+\left(2+\sqrt{\frac{\beta_1}{N}}\right)\epsilon+\frac{\beta_1}{3N(1-\gamma)}\\
&\le\sqrt{\frac{\beta_1}{N}}\sqrt{\operatorname{Var}_{P_{s,a}}[\widehat{V}^{\star}]}+\frac{2\beta_1}{N(1-\gamma)}. \tag*{(131)}
\end{aligned} ( P s , a − P s , a ) V ⋆ ≤ ( P s , a − P s , a ) V s , u 0 ⋆ + ( P s , a − P s , a ) ( V ⋆ − V s , u 0 ⋆ ) ≤ ( P s , a − P s , a ) V s , u 0 ⋆ + 2 ϵ ≤ N β 1 Var P s , a [ V s , u 0 ⋆ ] + 3 N ( 1 − γ ) β 1 + 2 ϵ ≤ N β 1 Var P s , a [ V ⋆ ] + ( 2 + N β 1 ) ϵ + 3 N ( 1 − γ ) β 1 ≤ N β 1 Var P s , a [ V ⋆ ] + N ( 1 − γ ) 2 β 1 . (131)
Here, the second line follows from ∣ ( P ^ s , a − P s , a ) ( V ^ ⋆ − V ^ s , u 0 ⋆ ) ∣ ≤ 2 ∥ V ^ ⋆ − V ^ s , u 0 ⋆ ∥ ∞ ≤ 2 ϵ \left|(\widehat{P}_{s,a}-P_{s,a})(\widehat{V}^{\star}-\widehat{V}^{\star}_{s,u_0})\right|\le2\lVert\widehat{V}^{\star}-\widehat{V}^{\star}_{s,u_0}\rVert_\infty\le2\epsilon ( P s , a − P s , a ) ( V ⋆ − V s , u 0 ⋆ ) ≤ 2 ∥ V ⋆ − V s , u 0 ⋆ ∥ ∞ ≤ 2 ϵ , the third line follows from (130) , the penultimate line follows from the triangle inequality
Var P s , a [ V ^ s , u 0 ⋆ ] ≤ Var P s , a [ V ^ ⋆ ] + Var P s , a [ V ^ s , u 0 ⋆ − V ^ ⋆ ] ≤ Var P s , a [ V ^ ⋆ ] + ϵ , \sqrt{\operatorname{Var}_{P_{s,a}}[\widehat{V}^{\star}_{s,u_0}]} \le\sqrt{\operatorname{Var}_{P_{s,a}}[\widehat{V}^{\star}]}+\sqrt{\operatorname{Var}_{P_{s,a}}[\widehat{V}^{\star}_{s,u_0}-\widehat{V}^{\star}]} \le\sqrt{\operatorname{Var}_{P_{s,a}}[\widehat{V}^{\star}]}+\epsilon, Var P s , a [ V s , u 0 ⋆ ] ≤ Var P s , a [ V ⋆ ] + Var P s , a [ V s , u 0 ⋆ − V ⋆ ] ≤ Var P s , a [ V ⋆ ] + ϵ ,
while the last step uses ϵ = 1 N ( 1 − γ ) \epsilon=\frac{1}{N(1-\gamma)} ϵ = N ( 1 − γ ) 1 , β 1 ≥ 4 \beta_1\ge4 β 1 ≥ 4 , and N ≥ 1 N\ge1 N ≥ 1 . Since this result (131) holds for every action, we may select a = π ^ ⋆ ( s ) a=\widehat{\pi}^\star(s) a = π ⋆ ( s ) for every ( s , a ) ∈ S × A (s,a)\in\mathcal{S}\times\mathcal{A} ( s , a ) ∈ S × A to demonstrates that
∣ ( P ^ π ^ ⋆ − P π ^ ⋆ ) V ^ ⋆ ∣ ≤ β 1 N Var P π ^ ⋆ [ V ^ ⋆ ] + 2 β 1 N ( 1 − γ ) 1 S (132) \left|(\widehat{P}_{\widehat{\pi}^\star}-P_{\widehat{\pi}^\star})\widehat{V}^{\star}\right| \le\sqrt{\frac{\beta_1}{N}}\sqrt{\operatorname{Var}_{P_{\widehat{\pi}^\star}}[\widehat{V}^{\star}]}+\frac{2\beta_1}{N(1-\gamma)}\mathbf{1}_S \tag*{(132)} ( P π ⋆ − P π ⋆ ) V ⋆ ≤ N β 1 Var P π ⋆ [ V ⋆ ] + N ( 1 − γ ) 2 β 1 1 S (132)
with probability exceeding 1 − δ / 2 1-\delta/2 1 − δ /2 .
It remains to convert (132) into the advertised value estimation bound. Akin to § 4.5 , the key is to exploit the interaction between the inverse matrix ( I − γ P π ⋆ ) − 1 (I-\gamma P_{\pi^\star})^{-1} ( I − γ P π ⋆ ) − 1 and the variance term Var P s , a [ V ^ ⋆ ] \sqrt{\operatorname{Var}_{P_{s,a}}[\widehat{V}^\star]} Var P s , a [ V ⋆ ] . To begin, recall that ( I − γ P π ⋆ ) − 1 (I-\gamma P_{\pi^\star})^{-1} ( I − γ P π ⋆ ) − 1 is nonnegative and has each row summing to 1 1 − γ \frac{1}{1-\gamma} 1 − γ 1 (see Lemma 2.3 ). These properties, taken together with (120) and (132) , yield
∥ V ^ π ^ ⋆ − V π ^ ⋆ ∥ ∞ ≤ γ ∥ ( I − γ P π ⋆ ) − 1 ( P ^ π ⋆ − P π ⋆ ) V ^ ⋆ ∥ ∞ ≤ γ β 1 N ∥ ( I − γ P π ⋆ ) − 1 Var P π ⋆ [ V ^ ⋆ ] ∥ ∞ + 2 β 1 N ( 1 − γ ) 2 . (133) \begin{aligned}
\left\|\widehat{V}^{\widehat{\pi}^\star}-V^{\widehat{\pi}^\star}\right\|_\infty\leq\gamma\left\|(I-\gamma P_{\pi^\star})^{-1}(\widehat{P}_{\pi^\star}-P_{\pi^\star})\widehat{V}^\star\right\|_\infty\\
&\leq\gamma\sqrt{\frac{\beta_1}{N}}\left\|(I-\gamma P_{\pi^\star})^{-1}\sqrt{\operatorname{Var}_{P_{\pi^\star}}[\widehat{V}^\star]}\right\|_\infty+\frac{2\beta_1}{N(1-\gamma)^2}. \tag*{(133)}
\end{aligned} V π ⋆ − V π ⋆ ∞ ≤ γ ( I − γ P π ⋆ ) − 1 ( P π ⋆ − P π ⋆ ) V ⋆ ∞ ≤ γ N β 1 ( I − γ P π ⋆ ) − 1 Var P π ⋆ [ V ⋆ ] ∞ + N ( 1 − γ ) 2 2 β 1 . (133)
We are thus left with the task of controlling the variance-dependent term in (133) . Towards this end, we develop the following key lemma, derived through a self-bounding argument; the proof can be found in § 5.2.3 .
Lemma 3.6. On the event where (132) holds, one has
γ ∥ ( I − γ P π ⋆ ) − 1 ( P ^ π ⋆ − P π ⋆ ) V ^ ⋆ ∥ ∞ ≤ 2 2 β 1 N ( 1 − γ ) 3 + 6 β 1 N ( 1 − γ ) 2 . (134) \gamma\left\|(I-\gamma P_{\pi^\star})^{-1}(\widehat{P}_{\pi^\star}-P_{\pi^\star})\widehat{V}^\star\right\|_\infty\leq2\sqrt{\frac{2\beta_1}{N(1-\gamma)^3}}+\frac{6\beta_1}{N(1-\gamma)^2}. \tag*{(134)} γ ( I − γ P π ⋆ ) − 1 ( P π ⋆ − P π ⋆ ) V ⋆ ∞ ≤ 2 N ( 1 − γ ) 3 2 β 1 + N ( 1 − γ ) 2 6 β 1 . (134)
Remark 3.2. Lemma 3.6 is another key ingredient in the present analysis that helps bypass the separation and reward-perturbation arguments used in Li et al. (2024c). It is closely related to the earlier bound (91) , but the latter applies to the true value function rather than the data-dependent empirical value function.
Combining Lemma 3.6 with inequality (133) readily establishes (128) , thus completing the proof of Lemma 3.5 .
Step 4: putting everything together. Lemma 3.5 implies the desired result as soon as
N ≥ c 0 β 1 ( 1 − γ ) 3 ε 2 N \geq\frac{c_0\beta_1}{(1-\gamma)^3\varepsilon^2} N ≥ ( 1 − γ ) 3 ε 2 c 0 β 1
for a sufficiently large universal constant c 0 > 0 c_0>0 c 0 > 0 . Since ε ≤ 1 1 − γ \varepsilon\leq\frac{1}{1-\gamma} ε ≤ 1 − γ 1 , this condition also guarantees N ≥ c 0 β 1 / ( 1 − γ ) N\geq c_0\beta_1/(1-\gamma) N ≥ c 0 β 1 / ( 1 − γ ) . This completes the proof.
Minimax lower bounds
To confirm the optimality of model-based algorithms, we present a matching minimax lower bound under the generative model, originally established in the seminal work of Azar et al., 2013. This result demonstrates that the performance guarantees achieved by the model-based approach are, in fact, statistically optimal (up to logarithmic factors).
Theorem 3.7. Consider any 0 < ε < 1 32 ( 1 − γ ) 0<\varepsilon<\frac{1}{32(1-\gamma)} 0 < ε < 32 ( 1 − γ ) 1 and γ ∈ [ 2 / 3 , 1 ) \gamma\in[2/3,1) γ ∈ [ 2/3 , 1 ) , and suppose S A ≥ 16 SA\geq16 S A ≥ 16 . Assume that N N N independent samples are drawn for each state-action pair from the generative model, where
N ≤ c N log ( S A ) ( 1 − γ ) 3 ε 2 N\leq\frac{c_N\log(SA)}{(1-\gamma)^3\varepsilon^2} N ≤ ( 1 − γ ) 3 ε 2 c N log ( S A ) for some small universal constant c N > 0 c_N>0 c N > 0 .
(i) For any (deterministic) policy estimator π ^ \widehat{\pi} π , there is a γ \gamma γ -discounted infinite-horizon MDP M \mathcal{M} M with S S S states and A A A actions such that
P M ( ∥ Q ^ π − Q ⋆ ∥ ∞ > ε ) ≥ 1 / 4. \mathbb{P}_{\mathcal{M}}\left(\lVert\widehat{Q}^{\pi}-Q^{\star}\rVert_{\infty}>\varepsilon\right)\geq1/4. P M ( ∥ Q π − Q ⋆ ∥ ∞ > ε ) ≥ 1/4.
(ii) For any Q-function estimator Q ^ \widehat{Q} Q , there exists a γ \gamma γ -discounted infinite-horizon MDP M \mathcal{M} M with S S S states and A A A actions such that
P M ( ∥ Q ^ − Q ⋆ ∥ ∞ > ε ) ≥ 1 / 4. \mathbb{P}_{\mathcal{M}}\left(\lVert\widehat{Q}-Q^{\star}\rVert_{\infty}>\varepsilon\right)\geq1/4. P M ( ∥ Q − Q ⋆ ∥ ∞ > ε ) ≥ 1/4.
Here, P M ( ⋅ ) \mathbb{P}_{\mathcal{M}}(\cdot) P M ( ⋅ ) denotes the probability distribution under M \mathcal{M} M .
As asserted by Theorem 3.7 , no algorithm can reliably learn an ε \varepsilon ε -optimal policy, or an ε \varepsilon ε -accurate optimal Q-function, for every γ \gamma γ -discounted infinite-horizon MDP, unless the total sample size exceeds Ω ~ ( S A ( 1 − γ ) 3 ε 2 ) \widetilde{\Omega}\left(\frac{SA}{(1-\gamma)^3\varepsilon^2}\right) Ω ( ( 1 − γ ) 3 ε 2 S A ) . By comparing this minimax lower bound with the performance guarantees of Algorithm 3.1 (see Theorem 3.1, corresponding the policy evaluation setting with A = 1 A=1 A = 1 ), as well as Algorithm 3.2 (see Theorem 3.2 ), we conclude that the model-based approach is information-theoretically optimal (up to logarithmic factors) in sample complexity for both policy evaluation and policy learning, across the entire range of precision levels ε \varepsilon ε .
Finally, we remark that Azar et al. (2013) established the first tight minimax lower bound on the sample complexity, applicable to both generative model and online sampling settings. Although the statement of Azar et al. (2013, Theorem 3) focuses on estimating optimal Q-functions and restricts the precision level to ε ≤ O ( 1 ) \varepsilon\leq O(1) ε ≤ O ( 1 ) , the lower bound developed therein actually extends to policy learning and holds for the broader range ε ≤ O ( 1 1 − γ ) \varepsilon\leq O\left(\frac{1}{1-\gamma}\right) ε ≤ O ( 1 − γ 1 ) . The original proof in Azar et al. (2013) relies on a change-of-measure argument by analyzing the likelihood ratio between two carefully constructed MDPs. In contrast, the proof to be presented shortly in this monograph adopts an information-theoretic approach based on Fano’s inequality (see Lemma 2.10 ).
Analysis: lower bound for a simpler small-scale case
Before presenting the proof of Theorem 3.7 , we first consider a related problem of Q-function estimation in a family of small-scale MDPs containing two states and one action. As it turns out, such a simplified setting suffices to illustrate the key ideas underlying the application of Fano’s lemma for lower bound analyses, while also revealing the fundamental bottleneck associated with the effective horizon that arises in more general policy learning and value estimation tasks.
Minimax lower bounds
Consider any 0 < ε < 1 32 ( 1 − γ ) 0 < \varepsilon< \frac{1}{32(1-\gamma)} 0 < ε < 32 ( 1 − γ ) 1 and any γ ∈ [ 2 / 3 , 1 ) \gamma\in[2/3,1) γ ∈ [ 2/3 , 1 ) . Our goal in this subsection is to show that one can construct two MDPs M 0 \mathcal{M}_0 M 0 and M 1 \mathcal{M}_1 M 1 — each with two states and a single action — such that if N N N independent samples are drawn from the generative model for each state-action pair and
N ≤ 0.003 ( 1 − γ ) 3 ε 2 , (135) N \le\frac{0.003}{(1-\gamma)^3\varepsilon^2}, \tag*{(135)} N ≤ ( 1 − γ ) 3 ε 2 0.003 , (135)
then any estimator Q ^ \widehat{Q} Q must satisfy
P M i ( ∣ Q ^ ( s 1 , a ) − Q M i ∗ ( s 1 , a ) ∣ > ε ) > 0.1 (136) \mathbb{P}_{\mathcal{M}_i}\left(\left|\widehat{Q}(s_1,a)-Q^*_{\mathcal{M}_i}(s_1,a)\right|>\varepsilon\right)>0.1 \tag*{(136)} P M i ( Q ( s 1 , a ) − Q M i ∗ ( s 1 , a ) > ε ) > 0.1 (136)
for some i ∈ { 0 , 1 } i \in\{0,1\} i ∈ { 0 , 1 } .
Here, we denote by P M 0 ( ⋅ ) \mathbb{P}_{\mathcal{M}_0}(\cdot) P M 0 ( ⋅ ) (resp. P M 1 ( ⋅ ) \mathbb{P}_{\mathcal{M}_1}(\cdot) P M 1 ( ⋅ ) ) the probability measure under M 0 \mathcal{M}_0 M 0 (resp. M 1 \mathcal{M}_1 M 1 ). In other words, no algorithm can simultaneously estimate the optimal Q-function for both M 0 \mathcal{M}_0 M 0 and M 1 \mathcal{M}_1 M 1 to the desired level of precision ε \varepsilon ε , unless the sample size per state-action pair exceeds the order of 1 ( 1 − γ ) 3 ε 2 \frac{1}{(1-\gamma)^3\varepsilon^2} ( 1 − γ ) 3 ε 2 1 . The main idea of the proof is as follows: we identify two simple MDPs that are difficult to distinguish, formulate a binary hypothesis testing problem to determine which of these two MDPs generates the observed samples, and then relate the resulting minimax testing error (obtained via Fano’s lemma) to the estimation risk.
Step 1: constructing hard MDP instances. The first step is to construct two MDP instances, deliberately designed to be nearly indistinguishable unless sufficiently many samples are available. We begin by specifying the common structure and properties shared by the two MDPs under consideration. Specifically, each MDP M \mathcal{M} M of interest is described as follows; see Figure 3.2 (a) for an illustration.
Figure 3.2. Illustration of MDP construction for the minimax lower bounds.
S = { s 1 , s 2 } . (137) \mathcal{S}=\{s_1,s_2\}. \tag*{(137)} S = { s 1 , s 2 } . (137)
Probability transition kernel. We now specify the transition kernel of the MDP M \mathcal{M} M , denoted by P M \mathbb{P}_{\mathcal{M}} P M .
– For state s 1 s_1 s 1 , set
P M ( s ∣ s 1 , a ) = { p M , if s = s 1 , 1 − p M , if s = s 2 (138) \mathbb{P}_{\mathcal{M}}(s\mid s_1,a)=
\begin{cases}
p_{\mathcal{M}}, & \text{if }s=s_1,\\
1-p_{\mathcal{M}}, & \text{if }s=s_2
\end{cases} \tag*{(138)} P M ( s ∣ s 1 , a ) = { p M , 1 − p M , if s = s 1 , if s = s 2 (138)
for some parameter 0 < p M < 1 0<p_{\mathcal{M}}<1 0 < p M < 1 . That is, when the current state is s 1 s_1 s 1 , the process either remains in the same state or transitions to s 2 s_2 s 2 .
– State s 2 s_2 s 2 is set to be an absorbing state, i.e.,
P M ( s 2 ∣ s 2 , a ) = 1. (139) \mathbb{P}_{\mathcal{M}}(s_2\mid s_2,a)=1. \tag*{(139)} P M ( s 2 ∣ s 2 , a ) = 1. (139)
r ( s , a ) = { 1 , if s = s 1 , 0 , else. (140) r(s,a)=
\begin{cases}
1, & \text{if }s=s_1,\\
0, & \text{else.}
\end{cases} \tag*{(140)} r ( s , a ) = { 1 , 0 , if s = s 1 , else. (140)
Clearly, a key parameter governing the optimal Q-function in the above construction of M \mathcal{M} M is p M p_{\mathcal{M}} p M . One can readily calculate the optimal Q-function in M \mathcal{M} M , denoted by Q M ⋆ Q^{\star}_{\mathcal{M}} Q M ⋆ , through the Bellman equation:
Q M ⋆ ( s 1 , a ) = 1 1 − γ p M . (141) Q^{\star}_{\mathcal{M}}(s_{1},a)=\frac{1}{1-\gamma p_{\mathcal{M}}}. \tag*{(141)} Q M ⋆ ( s 1 , a ) = 1 − γ p M 1 . (141)
Proof of (141) . First, since s 2 s_{2} s 2 is an absorbing state with 0 immediate reward, it follows that V M ⋆ ( s 2 ) = 0 V^{\star}_{\mathcal{M}}(s_{2})=0 V M ⋆ ( s 2 ) = 0 . Second, the Bellman equation gives
V M ⋆ ( s 1 ) = Q M ⋆ ( s 1 , a ) = r ( s 1 , a ) + γ { p M V M ⋆ ( s 1 ) + ( 1 − p M ) V M ⋆ ( s 2 ) } = 1 + γ p M V M ⋆ ( s 1 ) , \begin{aligned}
V^{\star}_{\mathcal{M}}(s_{1})&=Q^{\star}_{\mathcal{M}}(s_{1},a)=r(s_{1},a)+\gamma\{p_{\mathcal{M}}V^{\star}_{\mathcal{M}}(s_{1})+(1-p_{\mathcal{M}})V^{\star}_{\mathcal{M}}(s_{2})\}\\
&=1+\gamma p_{\mathcal{M}}V^{\star}_{\mathcal{M}}(s_{1}),
\end{aligned} V M ⋆ ( s 1 ) = Q M ⋆ ( s 1 , a ) = r ( s 1 , a ) + γ { p M V M ⋆ ( s 1 ) + ( 1 − p M ) V M ⋆ ( s 2 )} = 1 + γ p M V M ⋆ ( s 1 ) ,
and hence Q M ⋆ ( s 1 , a ) = V M ⋆ ( s 1 ) = 1 1 − γ p M Q^{\star}_{\mathcal{M}}(s_{1},a)=V^{\star}_{\mathcal{M}}(s_{1})=\frac{1}{1-\gamma p_{\mathcal{M}}} Q M ⋆ ( s 1 , a ) = V M ⋆ ( s 1 ) = 1 − γ p M 1 .
In the sequel, we focus on two MDPs constructed as above, denoted by M 0 \mathcal{M}_{0} M 0 and M 1 \mathcal{M}_{1} M 1 , which differ only in their choice of p M p_{\mathcal{M}} p M (cf. (138) ):
p M = { p , if M = M 0 , p + α , if M = M 1 , (142) p_{\mathcal{M}}=
\begin{cases}
p, & \text{if }\mathcal{M}=\mathcal{M}_{0},\\
p+\alpha, & \text{if }\mathcal{M}=\mathcal{M}_{1},
\end{cases} \tag*{(142)} p M = { p , p + α , if M = M 0 , if M = M 1 , (142)
where p p p and α \alpha α are parameters to be specified shortly, satisfying 1 / 2 ≤ p < p + α < 1 1/2\leq p<p+\alpha<1 1/2 ≤ p < p + α < 1 . In what follows, we denote by E M 0 [ ⋅ ] \mathbb{E}_{\mathcal{M}_{0}}[\cdot] E M 0 [ ⋅ ] (resp. E M 1 [ ⋅ ] \mathbb{E}_{\mathcal{M}_{1}}[\cdot] E M 1 [ ⋅ ] ) the expectation under M 0 \mathcal{M}_{0} M 0 (resp. M 1 \mathcal{M}_{1} M 1 ). The main step of the proof is to analyze the (in)distinguishability of these two MDPs given samples drawn from the generative model.
Step 2: applying Fano’s lemma. Suppose that for each state-pair, N N N independent samples are drawn from the generative model. In this setting, all useful information is contained in whether the process transitions to s 2 s_{2} s 2 after executing action a a a in state s 1 s_{1} s 1 .
Let ( s 1 , a , s n e x t ( i ) ) (s_{1},a,s_{\mathrm{next}}^{(i)}) ( s 1 , a , s next ( i ) ) (1 ≤ i ≤ N 1\leq i\leq N 1 ≤ i ≤ N ) denote the N N N independent samples drawn w.r.t. the state-action pair ( s 1 , a ) (s_{1},a) ( s 1 , a ) . We introduce a set of random variables ι 1 , … , ι N \iota_{1},\ldots,\iota_{N} ι 1 , … , ι N , where ι i = 1 { s n e x t ( i ) = s 1 } \iota_{i}=\mathbb{1}\{s_{\mathrm{next}}^{(i)}=s_{1}\} ι i = 1 { s next ( i ) = s 1 } is an indicator of staying in state s 1 s_{1} s 1 in the i i i -th sample, and is hence an independent Bernoulli random variable. Let P 0 P_{0} P 0 and P 1 P_{1} P 1 denote the distributions of all observed samples under M 0 \mathcal{M}_{0} M 0 and M 1 \mathcal{M}_{1} M 1 , respectively, and denote by P 0 , ι i P_{0,\iota_{i}} P 0 , ι i and P 1 , ι i P_{1,\iota_{i}} P 1 , ι i the distributions of ι i \iota_{i} ι i under M 0 \mathcal{M}_{0} M 0 and M 1 \mathcal{M}_{1} M 1 , respectively. In view of the tensorization property of KL divergence (cf. Lemma 2.12), we have
KL ( P 1 ∥ P 0 ) = ∑ i = 1 N KL ( P 1 , ι i ∥ P 0 , ι i ) = N KL ( p + α ∥ p ) ≤ N α 2 p ( 1 − p ) ≤ 2 N α 2 1 − p = : β , (143) \begin{aligned}
\operatorname{KL}(P_{1}\parallel P_{0})=\sum_{i=1}^{N}\operatorname{KL}(P_{1,\iota_{i}}\parallel P_{0,\iota_{i}})=N\operatorname{KL}(p+\alpha\parallel p) \\
&\leq\frac{N\alpha^{2}}{p(1-p)}\leq\frac{2N\alpha^{2}}{1-p}=:\beta, \tag*{(143)}
\end{aligned} KL ( P 1 ∥ P 0 ) = i = 1 ∑ N KL ( P 1 , ι i ∥ P 0 , ι i ) = N KL ( p + α ∥ p ) ≤ p ( 1 − p ) N α 2 ≤ 1 − p 2 N α 2 =: β , (143)
where KL ( p + α ∥ p ) \operatorname{KL}(p+\alpha\parallel p) KL ( p + α ∥ p ) denotes the KL divergence between Bern ( p + α ) \operatorname{Bern}(p+\alpha) Bern ( p + α ) and Bern ( p ) \operatorname{Bern}(p) Bern ( p ) , and the last line arises from Lemma 2.13 and the property p ≥ 1 / 2 p\geq1/2 p ≥ 1/2 . As a consequence, Fano’s lemma (see Lemma 2.10) implies that the minimax probability of error necessarily exceeds
p e : = inf ψ max 0 ≤ i ≤ 1 P M i ( ψ ≠ i ) ≥ max { 1 4 exp ( − β ) , 1 − β / 2 2 } . (144) p_{\mathrm{e}}:=\inf_{\psi}\max_{0\leq i\leq1}P_{\mathcal{M}_{i}}(\psi\neq i)\geq\max\left\{\frac{1}{4}\exp(-\beta),\frac{1-\sqrt{\beta/2}}{2}\right\}. \tag*{(144)} p e := ψ inf 0 ≤ i ≤ 1 max P M i ( ψ = i ) ≥ max { 4 1 exp ( − β ) , 2 1 − β /2 } . (144)
where the infimum is taken over all possible tests ψ \psi ψ . In particular,
p e ≥ max { 1 4 exp ( − β ) , 1 − β / 2 2 } ≥ 0.1 if β ≤ 0.9. (145) p_{\mathrm{e}}\geq\max\left\{\frac{1}{4}\exp(-\beta),\frac{1-\sqrt{\beta/2}}{2}\right\}\geq0.1\quad\text{if }\beta\leq0.9. \tag*{(145)} p e ≥ max { 4 1 exp ( − β ) , 2 1 − β /2 } ≥ 0.1 if β ≤ 0.9. (145)
Step 3: choosing parameters p p p and α \alpha α . In light of (145) , to guarantee that p e p_e p e is lower bounded by 0.1 0.1 0.1 , it suffices to choose the parameters p p p and α \alpha α to ensure that β ≤ 0.9 \beta\le0.9 β ≤ 0.9 . Recall that N N N needs to be no larger than the order of 1 ( 1 − γ ) 3 ε 2 \frac{1}{(1-\gamma)^3\varepsilon^2} ( 1 − γ ) 3 ε 2 1 (see (135) ). Motivated by this, we select p , α p,\alpha p , α as
p = 4 γ − 1 3 γ and α = 2 ( 1 − γ ) 2 ε γ 2 = 32 ( 1 − γ ) 2 ε 9 γ 2 . (146) p=\frac{4\gamma-1}{3\gamma}\qquad\text{and}\qquad\alpha=\frac{2(1-\gamma)^2\varepsilon}{\gamma^2}=\frac{32(1-\gamma)^2\varepsilon}{9\gamma^2}. \tag*{(146)} p = 3 γ 4 γ − 1 and α = γ 2 2 ( 1 − γ ) 2 ε = 9 γ 2 32 ( 1 − γ ) 2 ε . (146)
It is readily seen that as γ → 1 \gamma\to1 γ → 1 , we have p → 1 p\to1 p → 1 and α → 0 \alpha\to0 α → 0 , so that p M 0 p_{M_0} p M 0 and p M 1 p_{M_1} p M 1 become increasingly similar and harder to differentiate. Moreover, one can verify that 1 − p ≍ 1 − γ 1-p\asymp1-\gamma 1 − p ≍ 1 − γ , α ≍ ( 1 − γ ) 2 ε \alpha\asymp(1-\gamma)^2\varepsilon α ≍ ( 1 − γ ) 2 ε , and hence α 2 / ( 1 − p ) ≍ ( 1 − γ ) 3 ε 2 \alpha^2/(1-p)\asymp(1-\gamma)^3\varepsilon^2 α 2 / ( 1 − p ) ≍ ( 1 − γ ) 3 ε 2 , which matches the inverse of the upper bound (135) on N N N .
A few simple yet useful properties are summarized below.
Lemma 3.8. Consider the above MDPs with parameters (146) . Suppose that 0 < ε < 1 32 ( 1 − γ ) 0<\varepsilon<\frac{1}{32(1-\gamma)} 0 < ε < 32 ( 1 − γ ) 1 and γ ∈ [ 2 / 3 , 1 ) \gamma\in[2/3,1) γ ∈ [ 2/3 , 1 ) . Then one has
(i) 1 / 2 ≤ p < p + α < 1 1/2\le p<p+\alpha<1 1/2 ≤ p < p + α < 1 and 1 − p ≥ 2 α 1-p\ge2\alpha 1 − p ≥ 2 α ;
(ii) Q M 1 ⋆ ( s 1 , a ) − Q M 0 ⋆ ( s 1 , a ) > 2 ε Q^\star_{M_1}(s_1,a)-Q^\star_{M_0}(s_1,a)>2\varepsilon Q M 1 ⋆ ( s 1 , a ) − Q M 0 ⋆ ( s 1 , a ) > 2 ε ;
(iii) α 2 1 − p = 32 2 ( 1 − γ ) 3 ε 2 27 γ 3 ≤ 128 ( 1 − γ ) 3 ε 2 \frac{\alpha^2}{1-p}=\frac{32^2(1-\gamma)^3\varepsilon^2}{27\gamma^3}\le128(1-\gamma)^3\varepsilon^2 1 − p α 2 = 27 γ 3 3 2 2 ( 1 − γ ) 3 ε 2 ≤ 128 ( 1 − γ ) 3 ε 2 .
Proof of Lemma 3.8. Regarding (i), it suffices to see that
1 − p = 1 − γ 3 γ > 64 ( 1 − γ ) 2 ε 9 γ 2 = 2 α and p = 4 γ − 1 3 γ > 1 2 1-p=\frac{1-\gamma}{3\gamma}>\frac{64(1-\gamma)^2\varepsilon}{9\gamma^2}=2\alpha\qquad\text{and}\qquad p=\frac{4\gamma-1}{3\gamma}>\frac{1}{2} 1 − p = 3 γ 1 − γ > 9 γ 2 64 ( 1 − γ ) 2 ε = 2 α and p = 3 γ 4 γ − 1 > 2 1
under our conditions on ε \varepsilon ε and γ \gamma γ . Regarding (ii), (141) tells us that
Q M 1 ⋆ ( s 1 , a ) − Q M 0 ⋆ ( s 1 , a ) = 1 1 − γ ( p + α ) − 1 1 − γ p = γ α ( 1 − γ p ) ( 1 − γ ( p + α ) ) > 2 ε , (147) \begin{aligned}
Q^\star_{M_1}(s_1,a)-Q^\star_{M_0}(s_1,a)=\frac{1}{1-\gamma(p+\alpha)}-\frac{1}{1-\gamma p} \\
&=\frac{\gamma\alpha}{(1-\gamma p)(1-\gamma(p+\alpha))}>2\varepsilon, \tag*{(147)}
\end{aligned} Q M 1 ⋆ ( s 1 , a ) − Q M 0 ⋆ ( s 1 , a ) = 1 − γ ( p + α ) 1 − 1 − γ p 1 = ( 1 − γ p ) ( 1 − γ ( p + α )) γ α > 2 ε , (147)
where the last inequality uses the choice (146) of α \alpha α . Property (iii) also follows from direct calculation under our conditions on γ \gamma γ .
Combine Lemma 3.8 with (145) and the definition (143) of β \beta β to demonstrate that the minimax probability of testing error obeys
p e ≥ 0.1 as long as N ≤ 0.003 ( 1 − γ ) 3 ε 2 . (148) p_e\ge0.1\qquad\text{as long as}\qquad N\le\frac{0.003}{(1-\gamma)^3\varepsilon^2}. \tag*{(148)} p e ≥ 0.1 as long as N ≤ ( 1 − γ ) 3 ε 2 0.003 . (148)
Step 4: connecting testing error with estimation risk. Finally, we show how the minimax probability of testing error can be leveraged to bound the minimax estimation risk. Recall from Lemma 3.8(ii) that Q M 1 ⋆ ( s 1 , a ) − Q M 0 ⋆ ( s 1 , a ) > 2 ε Q^\star_{M_1}(s_1,a)-Q^\star_{M_0}(s_1,a)>2\varepsilon Q M 1 ⋆ ( s 1 , a ) − Q M 0 ⋆ ( s 1 , a ) > 2 ε . Suppose instead that there exists an estimator Q ^ \widehat{Q} Q such that
P M i ( ∣ Q ^ ( s 1 , a ) − Q M i ⋆ ( s 1 , a ) ∣ ≤ ε ) > 0.9 , ∀ i ∈ { 0 , 1 } . (149) \mathbb{P}_{\mathcal{M}_i}\left(\left|\widehat{Q}(s_1,a)-Q^\star_{M_i}(s_1,a)\right|\le\varepsilon\right)>0.9,\qquad\forall i\in\{0,1\}. \tag*{(149)} P M i ( Q ( s 1 , a ) − Q M i ⋆ ( s 1 , a ) ≤ ε ) > 0.9 , ∀ i ∈ { 0 , 1 } . (149)
Based on this estimator, consider the test
ψ = { 0 , if Q ^ ( s 1 , a ) ≤ 1 1 − γ p + ε , 1 , else. \psi=
\begin{cases}
0, & \text{if }\widehat{Q}(s_1,a)\le\frac{1}{1-\gamma p}+\varepsilon,\\
1, & \text{else.}
\end{cases} ψ = { 0 , 1 , if Q ( s 1 , a ) ≤ 1 − γ p 1 + ε , else.
By construction, under M 0 \mathcal{M}_0 M 0 we have
P M 0 ( ψ = 0 ) = P M 0 ( Q ^ ( s 1 , a ) ≤ 1 1 − γ p + ε ) = P M 0 ( Q ^ ( s 1 , a ) ≤ Q M 0 ∗ ( s 1 , a ) + ε ) > 0.9 , \mathbb{P}_{\mathcal{M}_0}(\psi= 0) = \mathbb{P}_{\mathcal{M}_0}\left(\widehat{Q}(s_1, a) \leq\frac{1}{1-\gamma p} + \varepsilon\right)
= \mathbb{P}_{\mathcal{M}_0}\left(\widehat{Q}(s_1, a) \leq Q^*_{\mathcal{M}_0}(s_1, a) + \varepsilon\right) > 0.9, P M 0 ( ψ = 0 ) = P M 0 ( Q ( s 1 , a ) ≤ 1 − γ p 1 + ε ) = P M 0 ( Q ( s 1 , a ) ≤ Q M 0 ∗ ( s 1 , a ) + ε ) > 0.9 ,
where the last inequality arises from the assumption (149) . An analogous argument shows that P M 1 ( ψ = 1 ) > 0.9 \mathbb{P}_{\mathcal{M}_1}(\psi= 1) > 0.9 P M 1 ( ψ = 1 ) > 0.9 . As a result, the testing error of ψ \psi ψ satisfies
p e ( ψ ) : = max { P M 0 ( ψ ≠ 0 ) , P M 1 ( ψ ≠ 1 ) } < 0.1. p_e(\psi) := \max\left\{\mathbb{P}_{\mathcal{M}_0}(\psi\ne0), \mathbb{P}_{\mathcal{M}_1}(\psi\ne1)\right\} < 0.1. p e ( ψ ) := max { P M 0 ( ψ = 0 ) , P M 1 ( ψ = 1 ) } < 0.1.
This contradicts the lower bound (148) on the minimax error probability, thereby establishing the claimed result (136) .
Proof of Theorem 3.7
We now turn to establishing the lower bound in Theorem 3.7, covering both policy learning and estimation of optimal Q Q Q -functions. Our construction and analysis largely build upon those developed in Section 3.4.1, with modifications to expand the state-action space to include S S S states and A A A actions. At a high level, we again construct a family of similar MDPs and formulate a multi-hypothesis testing problem aimed at identifying the underlying data-generating MDP; we then connect the minimax error probability of this testing problem with the performance of any RL algorithm.
Step 1: constructing hard MDP instances. Let us construct a family of γ \gamma γ -discounted infinite-horizon MDPs with S S S states and A A A actions. We begin by presenting their common structure before describing their differences in the transition kernels.
S = { s 1 , ⋯ , s S } . \mathcal{S} = \{s_1, \cdots, s_S\}. S = { s 1 , ⋯ , s S } .
The state space S \mathcal{S} S is partitioned into the following subsets:
S 1 = { s 1 , … , s S − 1 } , S 2 = { s S } , \mathcal{S}_1 = \{s_1, \ldots, s_{S-1}\}, \qquad\mathcal{S}_2 = \{s_S\}, S 1 = { s 1 , … , s S − 1 } , S 2 = { s S } ,
where each subset is associated with distinct transition probabilities, to be specified shortly.
Action space. For ease of exposition, we allow the action space to vary across states. For any state in S 1 \mathcal{S}_1 S 1 , the action space contains A A A actions, i.e.,
A : = { a 0 , a 1 , … , a A − 1 } . \mathcal{A} := \{a_0, a_1, \ldots, a_{A-1}\}. A := { a 0 , a 1 , … , a A − 1 } .
In contrast, the action space associated with s S s_S s S is the singleton set { a 0 } \{a_0\} { a 0 } .
r ( s , a ) = { 1 , if s ∈ S 1 , 0 , else. r(s, a) =
\begin{cases}
1, & \text{if } s \in\mathcal{S}_1,\\
0, & \text{else.}
\end{cases} r ( s , a ) = { 1 , 0 , if s ∈ S 1 , else.
Since only states in S 1 \mathcal{S}_1 S 1 yield positive rewards, this structure favors policies that induce a higher probability of remaining within S 1 \mathcal{S}_1 S 1 .
Minimax lower bounds
Having described the common structure shared by this family of MDPs, we now specify the optimal policy and transition kernel for each instance. In particular, we construct a collection of
M = 1 + ( S − 1 ) ( A − 1 ) (150) M = 1 + (S-1)(A-1) \tag*{(150)} M = 1 + ( S − 1 ) ( A − 1 ) (150)
MDP instances:
C m d p = { M 0 , 0 } ∪ { M j , k ∣ 1 ≤ j < S , 1 ≤ k < A } . (151) \mathcal{C}_{\mathrm{mdp}} = \{\mathcal{M}_{0,0}\} \cup\{\mathcal{M}_{j,k} \mid1 \le j < S, 1 \le k < A\}. \tag*{(151)} C mdp = { M 0 , 0 } ∪ { M j , k ∣ 1 ≤ j < S , 1 ≤ k < A } . (151)
As in § 4.9.1 , the construction is parameterized by two scalars p p p and α \alpha α satisfying 1 / 2 ≤ p < p + α < 1 1/2 \le p < p+\alpha< 1 1/2 ≤ p < p + α < 1 and 1 − p ≥ 2 α 1-p \ge2\alpha 1 − p ≥ 2 α . See Figure 3.2 (b) for an illustration of M 0 , 0 \mathcal{M}_{0,0} M 0 , 0 with A = 2 A=2 A = 2 .
Optimal policy. The optimal policy of M j , k \mathcal{M}_{j,k} M j , k , denoted by π j , k ⋆ \pi^\star_{j,k} π j , k ⋆ , is chosen such that
π 0 , 0 ⋆ ( s ) = a 0 ∀ s ∈ S 1 (152) \pi^\star_{0,0}(s) = a_0 \qquad\forall s \in\mathcal{S}_1 \tag*{(152)} π 0 , 0 ⋆ ( s ) = a 0 ∀ s ∈ S 1 (152)
and for any 1 ≤ j < S 1 \le j < S 1 ≤ j < S and 1 ≤ k < A 1 \le k < A 1 ≤ k < A ,
π j , k ⋆ ( s ) = { a k , if s = s j , a 0 , if s ∈ S 1 and s ≠ s j . (153) \pi^\star_{j,k}(s) =
\begin{cases}
a_k, & \text{if } s=s_j,\\
a_0, & \text{if } s\in\mathcal{S}_1 \text{ and } s\ne s_j.
\end{cases} \tag*{(153)} π j , k ⋆ ( s ) = { a k , a 0 , if s = s j , if s ∈ S 1 and s = s j . (153)
The transition probability kernels will be constructed to ensure consistency with these optimal policies.
Probability transition kernel. The probability transition kernel of M j , k \mathcal{M}_{j,k} M j , k , denoted by P M j , k P_{\mathcal{M}_{j,k}} P M j , k , is defined below.
P M j , k ( s ∣ s i , a ) = { p + α , if a = π j , k ⋆ ( s i ) and s = s i , 1 − p − α , if a = π j , k ⋆ ( s i ) and s = s S , p , if a ≠ π j , k ⋆ ( s i ) and s = s i , 1 − p , if a ≠ π j , k ⋆ ( s i ) and s = s S , 0 else . P_{\mathcal{M}_{j,k}}(s \mid s_i,a) =
\begin{cases}
p+\alpha, & \text{if } a=\pi^\star_{j,k}(s_i) \text{ and } s=s_i,\\
1-p-\alpha, & \text{if } a=\pi^\star_{j,k}(s_i) \text{ and } s=s_S,\\
p, & \text{if } a\ne\pi^\star_{j,k}(s_i) \text{ and } s=s_i,\\
1-p, & \text{if } a\ne\pi^\star_{j,k}(s_i) \text{ and } s=s_S,\\
0 & \text{else}.
\end{cases} P M j , k ( s ∣ s i , a ) = ⎩ ⎨ ⎧ p + α , 1 − p − α , p , 1 − p , 0 if a = π j , k ⋆ ( s i ) and s = s i , if a = π j , k ⋆ ( s i ) and s = s S , if a = π j , k ⋆ ( s i ) and s = s i , if a = π j , k ⋆ ( s i ) and s = s S , else .
In words, executing the action π j , k ⋆ ( s i ) \pi^\star_{j,k}(s_i) π j , k ⋆ ( s i ) in state s i ∈ S 1 s_i \in\mathcal{S}_1 s i ∈ S 1 has a higher probability of remaining in the same state. Given that states in S 1 \mathcal{S}_1 S 1 are associated with higher immediate rewards, this design ensures that taking action π j , k ⋆ ( s i ) \pi^\star_{j,k}(s_i) π j , k ⋆ ( s i ) in state s i s_i s i yields a larger Q-value.
P M j , k ( s S ∣ s S , a 0 ) = 1. P_{\mathcal{M}_{j,k}}(s_S \mid s_S,a_0)=1. P M j , k ( s S ∣ s S , a 0 ) = 1.
Evidently, the transition probabilities of these MDP instances differ only in those corresponding to the optimal actions; all non-optimal actions share the same transition probabilities. Repeating the same arguments as for (141) immediately reveals that: for each s ∈ S 1 s \in\mathcal{S}_1 s ∈ S 1 ,
Q M j , k ⋆ ( s , a ) = { 1 1 − γ ( p + α ) , if a = π j , k ⋆ ( s ) , 1 1 − γ p , if a ≠ π j , k ⋆ ( s ) . (154) Q^\star_{\mathcal{M}_{j,k}}(s,a) =
\begin{cases}
\frac{1}{1-\gamma(p+\alpha)}, & \text{if } a=\pi^\star_{j,k}(s),\\
\frac{1}{1-\gamma p}, & \text{if } a\ne\pi^\star_{j,k}(s).
\end{cases} \tag*{(154)} Q M j , k ⋆ ( s , a ) = { 1 − γ ( p + α ) 1 , 1 − γ p 1 , if a = π j , k ⋆ ( s ) , if a = π j , k ⋆ ( s ) . (154)
This characterization is clearly consistent with the optimal policy defined in (3.73).
Step 2: applying Fano’s lemma. To invoke Fano’s lemma stated in Lemma 2.10 , we need to examine the pairwise KL divergences among the constructed MDP instances. Recall that N N N independent samples are drawn from the generative model for each state-action pair. Let P M P_{\mathcal{M}} P M and π M ⋆ \pi_{\mathcal{M}}^\star π M ⋆ denote the distribution of the observed samples and the optimal policy under M \mathcal{M} M , respectively. By construction, the optimal policies of M j , k \mathcal{M}_{j,k} M j , k (for j ≠ 0 j \ne0 j = 0 ) and M 0 , 0 \mathcal{M}_{0,0} M 0 , 0 differ in exactly one state, leading to two state-action pairs with different transition probabilities in P M j , k P_{\mathcal{M}_{j,k}} P M j , k and P M 0 , 0 P_{\mathcal{M}_{0,0}} P M 0 , 0 . Consequently, the corresponding sample distributions differ only in the 2 N 2N 2 N samples associated with these two state-action pairs.
Among these 2 N 2N 2 N samples, N N N of them contribute a KL divergence of KL ( p + α ∥ p ) \operatorname{KL}(p+\alpha\mathbin\Vert p) KL ( p + α ∥ p ) each, while the remaining N N N contribute KL ( p ∥ p + α ) \operatorname{KL}(p \mathbin\Vert p+\alpha) KL ( p ∥ p + α ) each. Therefore, the overall KL divergence between P M j , k P_{\mathcal{M}_{j,k}} P M j , k and P M 0 , 0 P_{\mathcal{M}_{0,0}} P M 0 , 0 can be expressed as
KL ( P M j , k ∥ P M 0 , 0 ) = N KL ( p + α ∥ p ) + N KL ( p ∥ p + α ) \operatorname{KL}(P_{\mathcal{M}_{j,k}} \mathbin\Vert P_{\mathcal{M}_{0,0}}) = N\operatorname{KL}(p+\alpha\mathbin\Vert p) + N\operatorname{KL}(p \mathbin\Vert p+\alpha) KL ( P M j , k ∥ P M 0 , 0 ) = N KL ( p + α ∥ p ) + N KL ( p ∥ p + α )
for j ≠ 0 j \ne0 j = 0 . Repeating the arguments for (143) , we further arrive at
KL ( P M j , k ∥ P M 0 , 0 ) ≤ N α 2 p ( 1 − p ) + N α 2 ( p + α ) ( 1 − p − α ) ≤ 2 N α 2 ( p + α ) ( 1 − p − α ) ≤ 8 N α 2 1 − p , (155) \begin{aligned}
\operatorname{KL}(P_{\mathcal{M}_{j,k}} \mathbin\Vert P_{\mathcal{M}_{0,0}}) \le\frac{N\alpha^2}{p(1-p)}+\frac{N\alpha^2}{(p+\alpha)(1-p-\alpha)} \\
&\le\frac{2N\alpha^2}{(p+\alpha)(1-p-\alpha)}\le\frac{8N\alpha^2}{1-p}, \tag*{(155)}
\end{aligned} KL ( P M j , k ∥ P M 0 , 0 ) ≤ p ( 1 − p ) N α 2 + ( p + α ) ( 1 − p − α ) N α 2 ≤ ( p + α ) ( 1 − p − α ) 2 N α 2 ≤ 1 − p 8 N α 2 , (155)
with the proviso that 1 / 2 ≤ p < p + α < 1 1/2 \le p < p+\alpha< 1 1/2 ≤ p < p + α < 1 and 1 − p − α ≥ ( 1 − p ) / 2 1-p-\alpha\ge(1-p)/2 1 − p − α ≥ ( 1 − p ) /2 (a consequence of 1 − p ≥ 2 α 1-p \ge2\alpha 1 − p ≥ 2 α ). Define the minimax probability of testing error as
p e : = inf ψ max M ∈ C m d p P M ( ψ ≠ M ) , (156) p_{\mathrm{e}} := \inf_{\psi}\max_{\mathcal{M}\in\mathcal{C}_{\mathrm{mdp}}}\mathbb{P}_{\mathcal{M}}(\psi\ne\mathcal{M}), \tag*{(156)} p e := ψ inf M ∈ C mdp max P M ( ψ = M ) , (156)
where the infimum is taken over all tests that map the observed data to an MDP in C m d p \mathcal{C}_{\mathrm{mdp}} C mdp (cf. (151) ). Fano’s lemma (cf. Lemma 2.10 ) gives
p e ≥ log ( M + 1 ) − log 2 log M − 1 4 ≥ 1 4 , (157) p_{\mathrm{e}}\ge\frac{\log(M+1)-\log2}{\log M}-\frac{1}{4}\ge\frac{1}{4}, \tag*{(157)} p e ≥ log M log ( M + 1 ) − log 2 − 4 1 ≥ 4 1 , (157)
provided that M ≥ 4 M\ge4 M ≥ 4 and
1 M + 1 ∑ M ∈ C m d p KL ( P M ∥ P M 0 , 0 ) ≤ 1 4 log M . (158) \frac{1}{M+1}\sum_{\mathcal{M}\in\mathcal{C}_{\mathrm{mdp}}}\operatorname{KL}(P_{\mathcal{M}}\mathbin\Vert P_{\mathcal{M}_{0,0}})\le\frac{1}{4}\log M. \tag*{(158)} M + 1 1 M ∈ C mdp ∑ KL ( P M ∥ P M 0 , 0 ) ≤ 4 1 log M . (158)
With (155) in place, this KL condition (158) — and hence (157) — can be guaranteed if
8 N α 2 1 − p ≤ 1 4 log S A 4 ≤ 1 4 log M . (159) \frac{8N\alpha^2}{1-p}\le\frac{1}{4}\log\frac{SA}{4}\le\frac{1}{4}\log M. \tag*{(159)} 1 − p 8 N α 2 ≤ 4 1 log 4 S A ≤ 4 1 log M . (159)
Step 3: choosing parameters p p p and α \alpha α . It then comes down to choosing the parameters p p p and α \alpha α so as to satisfy (159) . Akin to § 4.9.1 , we set
p = 4 γ − 1 3 γ and α = 2 ( 1 − γ p ) 2 ε γ 2 = 32 ( 1 − γ ) 2 ε 9 γ 2 , p=\frac{4\gamma-1}{3\gamma}\qquad\text{and}\qquad\alpha=\frac{2(1-\gamma p)^2\varepsilon}{\gamma^2}=\frac{32(1-\gamma)^2\varepsilon}{9\gamma^2}, p = 3 γ 4 γ − 1 and α = γ 2 2 ( 1 − γ p ) 2 ε = 9 γ 2 32 ( 1 − γ ) 2 ε ,
From Lemma 3.8 (iii), it follows that α 2 1 − p ≤ 128 ( 1 − γ ) 3 ε 2 \frac{\alpha^2}{1-p}\le128(1-\gamma)^3\varepsilon^2 1 − p α 2 ≤ 128 ( 1 − γ ) 3 ε 2 , and as a result, Condition (159) is satisfied provided that
N ≤ 1 4096 ( 1 − γ ) 3 ε 2 log S A 4 . N\le\frac{1}{4096(1-\gamma)^3\varepsilon^2}\log\frac{SA}{4}. N ≤ 4096 ( 1 − γ ) 3 ε 2 1 log 4 S A .
Step 4: connecting testing error with performance of policy learning. We now relate the minimax probability of testing error p e p_e p e to the statistical risk of policy learning. As shown in (154) and Lemma 3.8 (ii), for any M ∈ C m d p \mathcal{M} \in\mathcal{C}_{\mathrm{mdp}} M ∈ C mdp and any state s ∈ S 1 s \in\mathcal{S}_1 s ∈ S 1 , the optimal action is separated from all suboptimal actions by the following gap:
Q M ⋆ ( s , π M ⋆ ( s ) ) − Q M ⋆ ( s , a ) > 2 ε , ∀ a ≠ π M ⋆ ( s ) . (160) Q^\star_{\mathcal{M}}(s,\pi^\star_{\mathcal{M}}(s))-Q^\star_{\mathcal{M}}(s,a)>2\varepsilon,\qquad\forall a\ne\pi^\star_{\mathcal{M}}(s). \tag*{(160)} Q M ⋆ ( s , π M ⋆ ( s )) − Q M ⋆ ( s , a ) > 2 ε , ∀ a = π M ⋆ ( s ) . (160)
Now suppose that there exists a (deterministic) policy estimate π ^ \widehat{\pi} π achieving
P M ( ∥ Q ^ π ^ − Q M ⋆ ∥ ∞ > ε ) < 1 / 4 , ∀ M ∈ C m d p . (161) \mathbb{P}_{\mathcal{M}}\left(\lVert\widehat{Q}^{\widehat{\pi}}-Q^\star_{\mathcal{M}}\rVert_\infty>\varepsilon\right)<1/4,\qquad\forall\mathcal{M}\in\mathcal{C}_{\mathrm{mdp}}. \tag*{(161)} P M ( ∥ Q π − Q M ⋆ ∥ ∞ > ε ) < 1/4 , ∀ M ∈ C mdp . (161)
In view of (160) , the only policy that can satisfy ∥ Q ^ π ^ − Q M ⋆ ∥ ∞ ≤ ε \lVert\widehat{Q}^{\widehat{\pi}}-Q^\star_{\mathcal{M}}\rVert_\infty\le\varepsilon ∥ Q π − Q M ⋆ ∥ ∞ ≤ ε is π M ⋆ \pi^\star_{\mathcal{M}} π M ⋆ . Hence, (161) is equivalent to
P M ( π ^ ≠ π M ⋆ ) < 1 / 4 , ∀ M ∈ C m d p . (162) \mathbb{P}_{\mathcal{M}}\left(\widehat{\pi}\ne\pi^\star_{\mathcal{M}}\right)<1/4,\qquad\forall\mathcal{M}\in\mathcal{C}_{\mathrm{mdp}}. \tag*{(162)} P M ( π = π M ⋆ ) < 1/4 , ∀ M ∈ C mdp . (162)
By construction, the optimal policy π M ⋆ \pi^\star_{\mathcal{M}} π M ⋆ uniquely identifies M \mathcal{M} M within the collection C m d p \mathcal{C}_{\mathrm{mdp}} C mdp . Define the test
ψ p o l i c y = M π ^ , \psi_{\mathrm{policy}}=\mathcal{M}_{\widehat{\pi}}, ψ policy = M π ,
with M π ^ \mathcal{M}_{\widehat{\pi}} M π indicating the MDP in C m d p \mathcal{C}_{\mathrm{mdp}} C mdp whose optimal policy coincides with π ^ \widehat{\pi} π . Then it follows that
P M ( ψ p o l i c y ≠ M ) = P M ( π ^ ≠ π M ⋆ ) < 1 / 4 , ∀ M ∈ C m d p , \mathbb{P}_{\mathcal{M}}\left(\psi_{\mathrm{policy}}\ne\mathcal{M}\right)=\mathbb{P}_{\mathcal{M}}\left(\widehat{\pi}\ne\pi^\star_{\mathcal{M}}\right)<1/4,\qquad\forall\mathcal{M}\in\mathcal{C}_{\mathrm{mdp}}, P M ( ψ policy = M ) = P M ( π = π M ⋆ ) < 1/4 , ∀ M ∈ C mdp ,
which contradicts the lower bound (157) . We have thus established that for any policy estimator π ^ \widehat{\pi} π , there exists some M ∈ C m d p \mathcal{M}\in\mathcal{C}_{\mathrm{mdp}} M ∈ C mdp such that
P M ( ∥ Q ^ π ^ − Q M ⋆ ∥ ∞ > ε ) ≥ 1 / 4. \mathbb{P}_{\mathcal{M}}\left(\lVert\widehat{Q}^{\widehat{\pi}}-Q^\star_{\mathcal{M}}\rVert_\infty>\varepsilon\right)\ge1/4. P M ( ∥ Q π − Q M ⋆ ∥ ∞ > ε ) ≥ 1/4.
Step 5: connecting testing error with Q-function estimation risk. It remains to link the minimax probability of testing error p e p_e p e with the estimation risk for learning optimal Q-functions. Suppose for the moment that there exists an estimator Q ^ \widehat{Q} Q such that
P M ( ∥ Q ^ − Q M ⋆ ∥ ∞ > ε ) < 1 / 4 , ∀ M ∈ C m d p . (163) \mathbb{P}_{\mathcal{M}}\left(\lVert\widehat{Q}-Q^\star_{\mathcal{M}}\rVert_\infty>\varepsilon\right)<1/4,\qquad\forall\mathcal{M}\in\mathcal{C}_{\mathrm{mdp}}. \tag*{(163)} P M ( ∥ Q − Q M ⋆ ∥ ∞ > ε ) < 1/4 , ∀ M ∈ C mdp . (163)
Combined with (160) , this reveals that: under every M ∈ C m d p \mathcal{M}\in\mathcal{C}_{\mathrm{mdp}} M ∈ C mdp , the following event occurs with probability at least 3 / 4 3/4 3/4 :
Q ^ ( s , π M ⋆ ( s ) ) − Q ^ ( s , a ) ≥ Q M ⋆ ( s , π M ⋆ ( s ) ) − Q M ⋆ ( s , a ) − 2 ∥ Q ^ − Q M ⋆ ∥ ∞ > 2 ε − 2 ε = 0 (164) \begin{aligned}
\widehat{Q}(s,\pi^\star_{\mathcal{M}}(s))-\widehat{Q}(s,a)
&\ge Q^\star_{\mathcal{M}}(s,\pi^\star_{\mathcal{M}}(s))-Q^\star_{\mathcal{M}}(s,a)-2\lVert\widehat{Q}-Q^\star_{\mathcal{M}}\rVert_\infty\\
&>2\varepsilon-2\varepsilon=0 \tag*{(164)}
\end{aligned} Q ( s , π M ⋆ ( s )) − Q ( s , a ) ≥ Q M ⋆ ( s , π M ⋆ ( s )) − Q M ⋆ ( s , a ) − 2 ∥ Q − Q M ⋆ ∥ ∞ > 2 ε − 2 ε = 0 (164)
simultaneously for all s ∈ S s\in\mathcal{S} s ∈ S and all a ≠ π M ⋆ ( s ) a\ne\pi^\star_{\mathcal{M}}(s) a = π M ⋆ ( s ) . As an immediate consequence, if we define π e s t \pi_{\mathrm{est}} π est to be the greedy policy w.r.t. Q ^ \widehat{Q} Q (namely, π e s t ( s ) = arg max a Q ^ ( s , a ) \pi_{\mathrm{est}}(s)=\operatorname*{arg\,max}_{a}\widehat{Q}(s,a) π est ( s ) = arg max a Q ( s , a ) for all s ∈ S s\in\mathcal{S} s ∈ S ), then under M \mathcal{M} M one must have π e s t = π M ⋆ \pi_{\mathrm{est}}=\pi^\star_{\mathcal{M}} π est = π M ⋆ .
In light of this observation, consider the following test:
ψ e s t = M π e s t , \psi_{\mathrm{est}}=\mathcal{M}_{\pi_{\mathrm{est}}}, ψ est = M π est ,
where M π e s t \mathcal{M}_{\pi_{\mathrm{est}}} M π est denotes the MDP in C m d p \mathcal{C}_{\mathrm{mdp}} C mdp whose optimal policy coincides with π e s t \pi_{\mathrm{est}} π est . It is then readily seen that
P M ( ψ e s t ≠ M ) = P M ( π e s t ≠ π M ⋆ ) < 1 / 4 , ∀ M ∈ C m d p , \mathbb{P}_{\mathcal{M}}\left(\psi_{\mathrm{est}}\ne\mathcal{M}\right)=\mathbb{P}_{\mathcal{M}}\left(\pi_{\mathrm{est}}\ne\pi^\star_{\mathcal{M}}\right)<1/4,\qquad\forall\mathcal{M}\in\mathcal{C}_{\mathrm{mdp}}, P M ( ψ est = M ) = P M ( π est = π M ⋆ ) < 1/4 , ∀ M ∈ C mdp ,
resulting in a contradiction with the minimax lower bound (157) . Therefore, we conclude that for any estimator Q ^ \widehat{Q} Q , there exists some M ∈ C m d p \mathcal{M}\in\mathcal{C}_{\mathrm{mdp}} M ∈ C mdp such that
P M ( ∥ Q ^ − Q M ⋆ ∥ ∞ > ε ) ≥ 1 / 4. \mathbb{P}_{\mathcal{M}}\left(\lVert\widehat{Q}-Q^\star_{\mathcal{M}}\rVert_\infty>\varepsilon\right)\ge1/4. P M ( ∥ Q − Q M ⋆ ∥ ∞ > ε ) ≥ 1/4.
Extension: finite-horizon nonstationary MDPs
Moving beyond discounted infinite-horizon stationary MDPs, we now briefly discuss model-based policy learning for finite-horizon nonstationary MDPs. The proof techniques are similar in spirit to — and in fact simpler than — those for the discounted infinite-horizon setting, owing in part to the availability of independent samples across different steps h h h over the entire horizon. For brevity, we refer interested readers to Li et al. (2024c) for the full proof details.
A model-based approach. We begin by describing what is arguably the most natural version of the model-based policy learning algorithm under the generative model. In essence, the algorithm performs model-based planning using the empirical model, just as its discounted infinite-horizon counterpart in Algorithm 3.2.
Model estimation. For each state-action-step tuple ( s , a , h ) ∈ S × A × [ H ] (s, a, h) \in\mathcal{S} \times\mathcal{A} \times[H] ( s , a , h ) ∈ S × A × [ H ] , draw N N N independent samples from the generative model:
s ( i ) ( s , a , h ) ∼ i . i . d . P h ( ⋅ ∣ s , a ) , i = 1 , … , N , s^{(i)}(s, a, h) \overset{\mathrm{i.i.d.}}{\sim} P_h(\cdot\mid s, a), \qquad i = 1, \ldots, N, s ( i ) ( s , a , h ) ∼ i.i.d. P h ( ⋅ ∣ s , a ) , i = 1 , … , N ,
and construct the empirical transition kernels { P ^ h } h = 1 H \{\widehat{P}_h\}_{h=1}^{H} { P h } h = 1 H via
∀ s ′ ∈ S , P ^ h ( s ′ ∣ s , a ) = 1 N ∑ i = 1 N 1 { s ′ = s ( i ) ( s , a , h ) } . (165) \forall s' \in\mathcal{S}, \qquad\widehat{P}_h(s' \mid s, a) = \frac{1}{N} \sum_{i=1}^{N} \mathbb{1}\{s' = s^{(i)}(s, a, h)\}. \tag*{(165)} ∀ s ′ ∈ S , P h ( s ′ ∣ s , a ) = N 1 i = 1 ∑ N 1 { s ′ = s ( i ) ( s , a , h )} . (165)
Namely, P ^ h ( s ′ ∣ s , a ) \widehat{P}_h(s' \mid s, a) P h ( s ′ ∣ s , a ) stands for the empirical frequency of transitions from ( s , a ) (s, a) ( s , a ) to state s ′ s' s ′ at step h h h .
Planning. Compute an optimal policy π ^ ⋆ = { π ^ h ⋆ } h = 1 H \widehat{\pi}^{\star} = \{\widehat{\pi}^{\star}_h\}_{h=1}^{H} π ⋆ = { π h ⋆ } h = 1 H for the empirical MDP M ^ = ( S , A , P ^ , r , H ) \widehat{\mathcal{M}} = (\mathcal{S}, \mathcal{A}, \widehat{P}, r, H) M = ( S , A , P , r , H ) .
Note that the optimal policy π ^ h ⋆ \widehat{\pi}^{\star}_h π h ⋆ of the empirical MDP M ^ \widehat{\mathcal{M}} M at step h h h is computed via backward induction starting from h = H h = H h = H . This backward recursion ensures that π ^ h ⋆ \widehat{\pi}^{\star}_h π h ⋆ depends solely on the empirical transition kernels at or after step h h h , and as a result, π ^ h ⋆ \widehat{\pi}^{\star}_h π h ⋆ is statistically independent of the empirical transitions { P ^ j } 1 ≤ j < h \{\widehat{P}_j\}_{1 \le j < h} { P j } 1 ≤ j < h . Consequently, the statistical dependencies across steps are naturally decoupled. In contrast to the discounted infinite-horizon setting, this obviates the need for the leave-one-out construction used there to decouple the statistical dependencies. As the following theorem shows, this simple model-based approach already achieves the desirable sample complexity.
Theorem 3.9 (Li et al., 2024c). There exists some universal constant c 0 > 0 c_0 > 0 c 0 > 0 such that: for any δ > 0 \delta> 0 δ > 0 and 0 < ε ≤ H 0 < \varepsilon\le H 0 < ε ≤ H , the policy π ^ ⋆ \widehat{\pi}^{\star} π ⋆ returned by the aforementioned model-based algorithm obeys
∀ ( s , h ) ∈ S × [ H ] , V h π ^ ⋆ ( s ) ≥ V h ⋆ ( s ) − ε \forall(s, h) \in\mathcal{S} \times[H], \qquad V_h^{\widehat{\pi}^{\star}}(s) \ge V_h^{\star}(s) - \varepsilon ∀ ( s , h ) ∈ S × [ H ] , V h π ⋆ ( s ) ≥ V h ⋆ ( s ) − ε ∀ ( s , a , h ) ∈ S × A × [ H ] , Q h π ^ ⋆ ( s , a ) ≥ Q h ⋆ ( s , a ) − ε \forall(s, a, h) \in\mathcal{S} \times\mathcal{A} \times[H], \qquad Q_h^{\widehat{\pi}^{\star}}(s, a) \ge Q_h^{\star}(s, a) - \varepsilon ∀ ( s , a , h ) ∈ S × A × [ H ] , Q h π ⋆ ( s , a ) ≥ Q h ⋆ ( s , a ) − ε with probability at least 1 − δ 1 - \delta 1 − δ , provided that the sample size for every tuple ( s , a , h ) (s, a, h) ( s , a , h ) exceeds
N ≥ c 0 H 3 log ( H S A δ ) ε 2 . (166) N \ge\frac{c_0 H^3 \log\left(\frac{HSA}{\delta}\right)}{\varepsilon^2}. \tag*{(166)} N ≥ ε 2 c 0 H 3 log ( δ H S A ) . (166)
Akin to the discounted infinite-horizon setting, the model-based approach achieves ε \varepsilon ε -accuracy as long as the sample size per state-action-step tuple ( s , a , h ) (s, a, h) ( s , a , h ) exceeds the order
H 3 ε 2 \frac{H^3}{\varepsilon^2} ε 2 H 3
up to logarithmic factors. This guarantee, which holds for the full range ε ∈ ( 0 , H ] \varepsilon\in(0,H] ε ∈ ( 0 , H ] , mirrors the bound (109) for the discounted infinite-horizon setting, with the effective horizon 1 1 − γ \frac{1}{1-\gamma} 1 − γ 1 now replaced by the finite horizon length H H H . Since there are S A H SAH S A H distinct ( s , a , h ) (s,a,h) ( s , a , h ) -tuples in total, the overall sample complexity scales as
O ~ ( S A H 4 ε 2 ) . (167) \widetilde{O}\left(\frac{SAH^{4}}{\varepsilon^{2}}\right). \tag*{(167)} O ( ε 2 S A H 4 ) . (167)
The quadruple horizon dependence H 4 H^{4} H 4 in this bound — as opposed to the cubic scaling 1 / ( 1 − γ ) 3 1/(1-\gamma)^{3} 1/ ( 1 − γ ) 3 seen in the discounted infinite-horizon case — stems from time inhomogeneity. Because the transition kernels P h P_h P h may vary across steps, one incurs an additional factor of H H H in the sample complexity. Encouragingly, the sample complexity (167) is provably minimax optimal in view of the matching lower bound (Domingues et al., 2021).
Appendix: proof of auxiliary lemmas in Section 3.3.4
Proof of Lemma 3.3
To prove this lemma, it suffices to verify that V ⋆ V^{\star} V ⋆ satisfies the Bellman optimality equation w.r.t. M s , u ⋆ \mathcal{M}_{s,u^{\star}} M s , u ⋆ . For notational simplicity, let P a b s P^{\mathrm{abs}} P abs and r a b s r^{\mathrm{abs}} r abs denote its probability transition kernel and reward function, respectively.
At the absorbing state s s s , every action a ∈ A a \in\mathcal{A} a ∈ A satisfies
r a b s ( s , a ) + γ ( P a b s V ⋆ ) ( s , a ) = ( 1 − γ ) u ⋆ + γ V ⋆ ( s ) = V ⋆ ( s ) , r^{\mathrm{abs}}(s,a)+\gamma(P^{\mathrm{abs}}V^{\star})(s,a)=(1-\gamma)u^{\star}+\gamma V^{\star}(s)=V^{\star}(s), r abs ( s , a ) + γ ( P abs V ⋆ ) ( s , a ) = ( 1 − γ ) u ⋆ + γ V ⋆ ( s ) = V ⋆ ( s ) ,
where the last identity follows from u ⋆ = V ⋆ ( s ) u^{\star}=V^{\star}(s) u ⋆ = V ⋆ ( s ) . Next, for any state s ′ ≠ s s' \ne s s ′ = s , the transition kernel and reward function remain unchanged, so that
max a ∈ A { r a b s ( s ′ , a ) + γ ( P a b s V ⋆ ) ( s ′ , a ) } = max a ∈ A { r ( s ′ , a ) + γ ( P V ⋆ ) ( s ′ , a ) } = V ⋆ ( s ′ ) . \max_{a\in\mathcal{A}}\left\{r^{\mathrm{abs}}(s',a)+\gamma(P^{\mathrm{abs}}V^{\star})_{(s',a)}\right\}
=\max_{a\in\mathcal{A}}\left\{r(s',a)+\gamma(PV^{\star})_{(s',a)}\right\}
=V^{\star}(s'). a ∈ A max { r abs ( s ′ , a ) + γ ( P abs V ⋆ ) ( s ′ , a ) } = a ∈ A max { r ( s ′ , a ) + γ ( P V ⋆ ) ( s ′ , a ) } = V ⋆ ( s ′ ) .
Combining the above two cases shows that V ⋆ V^{\star} V ⋆ satisfies the Bellman optimality equation for M s , u ⋆ \mathcal{M}_{s,u^{\star}} M s , u ⋆ . This taken together with the uniqueness of the value-function fixed point completes the proof.
Proof of Lemma 3.4
In view of Lemma 3.3 , setting u ^ ⋆ = V ^ ⋆ ( s ) \widehat{u}^{\star}=\widehat{V}^{\star}(s) u ⋆ = V ⋆ ( s ) gives V ^ s , u ^ ⋆ ⋆ = V ^ ⋆ \widehat{V}^{\star}_{s,\widehat{u}^{\star}}=\widehat{V}^{\star} V s , u ⋆ ⋆ = V ⋆ . By construction of the net (125) , there exists u 0 ∈ N ε u_{0}\in\mathcal{N}_{\varepsilon} u 0 ∈ N ε such that ∣ u ^ ⋆ − u 0 ∣ ≤ ε |\widehat{u}^{\star}-u_{0}|\le\varepsilon ∣ u ⋆ − u 0 ∣ ≤ ε . The two auxiliary MDPs have identical transition kernels, and their rewards differ by at most ( 1 − γ ) ∣ u ^ ⋆ − u 0 ∣ (1-\gamma)|\widehat{u}^{\star}-u_{0}| ( 1 − γ ) ∣ u ⋆ − u 0 ∣ . The basic property (44) therefore gives
∥ V ^ ⋆ − V ^ s , u 0 ⋆ ∥ ∞ = ∥ V ^ s , u ^ ⋆ ⋆ − V ^ s , u 0 ⋆ ∥ ∞ ≤ ∣ u ^ ⋆ − u 0 ∣ ≤ ε . \left\|\widehat{V}^{\star}-\widehat{V}^{\star}_{s,u_{0}}\right\|_{\infty}
=\left\|\widehat{V}^{\star}_{s,\widehat{u}^{\star}}-\widehat{V}^{\star}_{s,u_{0}}\right\|_{\infty}
\leq|\widehat{u}^{\star}-u_{0}|\leq\varepsilon. V ⋆ − V s , u 0 ⋆ ∞ = V s , u ⋆ ⋆ − V s , u 0 ⋆ ∞ ≤ ∣ u ⋆ − u 0 ∣ ≤ ε .
Proof of Lemma 3.6
It is first seen from the Bellman equation V ^ ⋆ = r ^ π ⋆ + γ P ^ π ⋆ V ^ ⋆ \widehat{V}^{\star}=\widehat{r}_{\pi^{\star}}+\gamma\widehat{P}_{\pi^{\star}}\widehat{V}^{\star} V ⋆ = r π ⋆ + γ P π ⋆ V ⋆ that
γ P π ⋆ V ^ ⋆ = V ^ ⋆ − r ^ π ⋆ − γ ( P ^ π ⋆ − P π ⋆ ) V ^ ⋆ , \gamma P_{\pi^{\star}}\widehat{V}^{\star}
=\widehat{V}^{\star}-\widehat{r}_{\pi^{\star}}-\gamma(\widehat{P}_{\pi^{\star}}-P_{\pi^{\star}})\widehat{V}^{\star}, γ P π ⋆ V ⋆ = V ⋆ − r π ⋆ − γ ( P π ⋆ − P π ⋆ ) V ⋆ ,
0 ≤ V ^ ⋆ − r ^ π ⋆ = γ P ^ π ⋆ V ^ ⋆ ≤ γ 1 − γ 1 S . 0\leq\widehat{V}^{\star}-\widehat{r}_{\pi^{\star}}
=\gamma\widehat{P}_{\pi^{\star}}\widehat{V}^{\star}
\leq\frac{\gamma}{1-\gamma}\mathbf{1}_{S}. 0 ≤ V ⋆ − r π ⋆ = γ P π ⋆ V ⋆ ≤ 1 − γ γ 1 S .
Using the first identity to expand the variance (cf. (32) ), we obtain
γ 2 Var π ⋆ [ V ^ ⋆ ] = ( γ 2 P ^ π ⋆ − I ) ( V ^ ⋆ ∘ V ^ ⋆ ) + 2 V ^ ⋆ ∘ r ^ π ⋆ − r ^ π ⋆ ∘ r ^ π ⋆ + 2 γ ( V ^ ⋆ − r ^ π ⋆ ) ∘ [ ( P ^ π ⋆ − P π ⋆ ) V ^ ⋆ ] − γ 2 [ ( P ^ π ⋆ − P π ⋆ ) V ^ ⋆ ] ∘ [ ( P ^ π ⋆ − P π ⋆ ) V ^ ⋆ ] ≤ ( γ P ^ π ⋆ − I ) ( V ^ ⋆ ∘ V ^ ⋆ ) + 2 1 − γ 1 S + 2 γ 2 1 − γ ∣ ( P ^ π ⋆ − P π ⋆ ) V ^ ⋆ ∣ . (168) \gamma^{2}\operatorname{Var}_{\pi_{\star}}\left[\widehat{V}^{\star}\right] = \left(\gamma^{2}\widehat{P}_{\pi_{\star}}-I\right)\left(\widehat{V}^{\star}\circ\widehat{V}^{\star}\right)+2\widehat{V}^{\star}\circ\widehat{r}_{\pi_{\star}}-\widehat{r}_{\pi_{\star}}\circ\widehat{r}_{\pi_{\star}}
+2\gamma\left(\widehat{V}^{\star}-\widehat{r}_{\pi_{\star}}\right)\circ\left[\left(\widehat{P}_{\pi_{\star}}-P_{\pi_{\star}}\right)\widehat{V}^{\star}\right]-\gamma^{2}\left[\left(\widehat{P}_{\pi_{\star}}-P_{\pi_{\star}}\right)\widehat{V}^{\star}\right]\circ\left[\left(\widehat{P}_{\pi_{\star}}-P_{\pi_{\star}}\right)\widehat{V}^{\star}\right]
\leq\left(\gamma\widehat{P}_{\pi_{\star}}-I\right)\left(\widehat{V}^{\star}\circ\widehat{V}^{\star}\right)+\frac{2}{1-\gamma}\mathbf{1}_{S}+\frac{2\gamma^{2}}{1-\gamma}\left|\left(\widehat{P}_{\pi_{\star}}-P_{\pi_{\star}}\right)\widehat{V}^{\star}\right|. \tag*{(168)} γ 2 Var π ⋆ [ V ⋆ ] = ( γ 2 P π ⋆ − I ) ( V ⋆ ∘ V ⋆ ) + 2 V ⋆ ∘ r π ⋆ − r π ⋆ ∘ r π ⋆ + 2 γ ( V ⋆ − r π ⋆ ) ∘ [ ( P π ⋆ − P π ⋆ ) V ⋆ ] − γ 2 [ ( P π ⋆ − P π ⋆ ) V ⋆ ] ∘ [ ( P π ⋆ − P π ⋆ ) V ⋆ ] ≤ ( γ P π ⋆ − I ) ( V ⋆ ∘ V ⋆ ) + 1 − γ 2 1 S + 1 − γ 2 γ 2 ( P π ⋆ − P π ⋆ ) V ⋆ . (168)
Here, the last inequality discards the two entrywise square terms and makes use of 0 ≤ r ^ π ⋆ ≤ 1 S 0\leq\widehat{r}_{\pi_{\star}}\leq\mathbf{1}_{S} 0 ≤ r π ⋆ ≤ 1 S , 0 ≤ V ^ ⋆ ≤ 1 1 − γ 1 S 0\leq\widehat{V}^{\star}\leq\frac{1}{1-\gamma}\mathbf{1}_{S} 0 ≤ V ⋆ ≤ 1 − γ 1 1 S , the preceding bound on V ^ ⋆ − r ^ π ⋆ \widehat{V}^{\star}-\widehat{r}_{\pi_{\star}} V ⋆ − r π ⋆ , and γ 2 ≤ γ \gamma^{2}\leq\gamma γ 2 ≤ γ .
The usefulness of this expansion becomes apparent after multiplying by ( I − γ P ^ π ⋆ ) − 1 \left(I-\gamma\widehat{P}_{\pi_{\star}}\right)^{-1} ( I − γ P π ⋆ ) − 1 , which together with ( I − γ P ^ π ⋆ ) − 1 1 S = 1 1 − γ 1 S \left(I-\gamma\widehat{P}_{\pi_{\star}}\right)^{-1}\mathbf{1}_{S}=\frac{1}{1-\gamma}\mathbf{1}_{S} ( I − γ P π ⋆ ) − 1 1 S = 1 − γ 1 1 S gives
γ 2 ( I − γ P ^ π ⋆ ) − 1 Var π ⋆ [ V ^ ⋆ ] ≤ − V ^ ⋆ ∘ V ^ ⋆ + 2 ( 1 − γ ) 2 1 S + 2 γ 2 1 − γ ( I − γ P ^ π ⋆ ) − 1 ∣ ( P ^ π ⋆ − P π ⋆ ) V ^ ⋆ ∣ ≤ 2 ( 1 − γ ) 2 1 S + 2 γ 2 1 − γ ( I − γ P ^ π ⋆ ) − 1 ∣ ( P ^ π ⋆ − P π ⋆ ) V ^ ⋆ ∣ . \gamma^{2}\left(I-\gamma\widehat{P}_{\pi_{\star}}\right)^{-1}\operatorname{Var}_{\pi_{\star}}\left[\widehat{V}^{\star}\right]\leq-\widehat{V}^{\star}\circ\widehat{V}^{\star}+\frac{2}{(1-\gamma)^{2}}\mathbf{1}_{S}+\frac{2\gamma^{2}}{1-\gamma}\left(I-\gamma\widehat{P}_{\pi_{\star}}\right)^{-1}\left|\left(\widehat{P}_{\pi_{\star}}-P_{\pi_{\star}}\right)\widehat{V}^{\star}\right|
\leq\frac{2}{(1-\gamma)^{2}}\mathbf{1}_{S}+\frac{2\gamma^{2}}{1-\gamma}\left(I-\gamma\widehat{P}_{\pi_{\star}}\right)^{-1}\left|\left(\widehat{P}_{\pi_{\star}}-P_{\pi_{\star}}\right)\widehat{V}^{\star}\right|. γ 2 ( I − γ P π ⋆ ) − 1 Var π ⋆ [ V ⋆ ] ≤ − V ⋆ ∘ V ⋆ + ( 1 − γ ) 2 2 1 S + 1 − γ 2 γ 2 ( I − γ P π ⋆ ) − 1 ( P π ⋆ − P π ⋆ ) V ⋆ ≤ ( 1 − γ ) 2 2 1 S + 1 − γ 2 γ 2 ( I − γ P π ⋆ ) − 1 ( P π ⋆ − P π ⋆ ) V ⋆ .
We can now apply the same argument as in (89) to move the inverse matrix inside the square root. Applying Jensen’s inequality, and taking the ℓ ∞ \ell_{\infty} ℓ ∞ norm and squaring, we arrive at
γ 2 ∥ ( I − γ P ^ π ⋆ ) − 1 Var π ⋆ [ V ^ ⋆ ] ∥ ∞ 2 ≤ γ 2 1 − γ ∥ ( I − γ P ^ π ⋆ ) − 1 Var π ⋆ [ V ^ ⋆ ] ∥ ∞ ≤ 2 ( 1 − γ ) 3 + 2 γ ( 1 − γ ) 2 ∥ ( I − γ P ^ π ⋆ ) − 1 ∣ ( P ^ π ⋆ − P π ⋆ ) V ^ ⋆ ∣ ∥ ∞ . (169) \gamma^{2}\left\|\left(I-\gamma\widehat{P}_{\pi_{\star}}\right)^{-1}\sqrt{\operatorname{Var}_{\pi_{\star}}\left[\widehat{V}^{\star}\right]}\right\|_{\infty}^{2}\leq\frac{\gamma^{2}}{1-\gamma}\left\|\left(I-\gamma\widehat{P}_{\pi_{\star}}\right)^{-1}\operatorname{Var}_{\pi_{\star}}\left[\widehat{V}^{\star}\right]\right\|_{\infty}
\leq\frac{2}{(1-\gamma)^{3}}+\frac{2\gamma}{(1-\gamma)^{2}}\left\|\left(I-\gamma\widehat{P}_{\pi_{\star}}\right)^{-1}\left|\left(\widehat{P}_{\pi_{\star}}-P_{\pi_{\star}}\right)\widehat{V}^{\star}\right|\right\|_{\infty}. \tag*{(169)} γ 2 ( I − γ P π ⋆ ) − 1 Var π ⋆ [ V ⋆ ] ∞ 2 ≤ 1 − γ γ 2 ( I − γ P π ⋆ ) − 1 Var π ⋆ [ V ⋆ ] ∞ ≤ ( 1 − γ ) 3 2 + ( 1 − γ ) 2 2 γ ( I − γ P π ⋆ ) − 1 ( P π ⋆ − P π ⋆ ) V ⋆ ∞ . (169)
where the last line also uses γ ≤ 1 \gamma\leq1 γ ≤ 1 .
Notice that the final term in (169) is proportional to the same nonnegative quantity that upper bounds the value estimation error in (133) . Substituting (169) into (133) and using x + y ≤ x + y \sqrt{x+y}\leq\sqrt{x}+\sqrt{y} x + y ≤ x + y for x , y ≥ 0 x,y\geq0 x , y ≥ 0 , we reach
γ ∥ ( I − γ P ^ π ⋆ ) − 1 ∣ ( P ^ π ⋆ − P π ⋆ ) V ^ ⋆ ∣ ∥ ∞ ≤ 2 β 1 N ( 1 − γ ) 3 + 2 β 1 N ( 1 − γ ) 2 + 2 γ β 1 N ( 1 − γ ) 2 ∥ ( I − γ P ^ π ⋆ ) − 1 ∣ ( P ^ π ⋆ − P π ⋆ ) V ^ ⋆ ∣ ∥ ∞ 1 / 2 ≤ 2 β 1 N ( 1 − γ ) 3 + 3 β 1 N ( 1 − γ ) 2 + γ 2 ∥ ( I − γ P ^ π ⋆ ) − 1 ∣ ( P ^ π ⋆ − P π ⋆ ) V ^ ⋆ ∣ ∥ ∞ , \gamma\left\|\left(I-\gamma\widehat{P}_{\pi_{\star}}\right)^{-1}\left|\left(\widehat{P}_{\pi_{\star}}-P_{\pi_{\star}}\right)\widehat{V}^{\star}\right|\right\|_{\infty}
\leq\sqrt{\frac{2\beta_{1}}{N(1-\gamma)^{3}}}+\sqrt{\frac{2\beta_{1}}{N(1-\gamma)^{2}}}+\sqrt{\frac{2\gamma\beta_{1}}{N(1-\gamma)^{2}}}\left\|\left(I-\gamma\widehat{P}_{\pi_{\star}}\right)^{-1}\left|\left(\widehat{P}_{\pi_{\star}}-P_{\pi_{\star}}\right)\widehat{V}^{\star}\right|\right\|_{\infty}^{1/2}
\leq\sqrt{\frac{2\beta_{1}}{N(1-\gamma)^{3}}}+\frac{3\beta_{1}}{N(1-\gamma)^{2}}+\frac{\gamma}{2}\left\|\left(I-\gamma\widehat{P}_{\pi_{\star}}\right)^{-1}\left|\left(\widehat{P}_{\pi_{\star}}-P_{\pi_{\star}}\right)\widehat{V}^{\star}\right|\right\|_{\infty}, γ ( I − γ P π ⋆ ) − 1 ( P π ⋆ − P π ⋆ ) V ⋆ ∞ ≤ N ( 1 − γ ) 3 2 β 1 + N ( 1 − γ ) 2 2 β 1 + N ( 1 − γ ) 2 2 γ β 1 ( I − γ P π ⋆ ) − 1 ( P π ⋆ − P π ⋆ ) V ⋆ ∞ 1/2 ≤ N ( 1 − γ ) 3 2 β 1 + N ( 1 − γ ) 2 3 β 1 + 2 γ ( I − γ P π ⋆ ) − 1 ( P π ⋆ − P π ⋆ ) V ⋆ ∞ ,
where the last step follows from the AM-GM inequality. Solving the above inequality yields
γ ∥ ( I − γ P ^ π ⋆ ) − 1 ∣ ( P ^ π ⋆ − P π ⋆ ) V ^ ⋆ ∣ ∥ ∞ ≤ 2 2 β 1 N ( 1 − γ ) 3 + 6 β 1 N ( 1 − γ ) 2 . \gamma\left\|\left(I-\gamma\widehat{P}_{\pi_{\star}}\right)^{-1}\left|\left(\widehat{P}_{\pi_{\star}}-P_{\pi_{\star}}\right)\widehat{V}^{\star}\right|\right\|_{\infty}\leq2\sqrt{\frac{2\beta_{1}}{N(1-\gamma)^{3}}}+\frac{6\beta_{1}}{N(1-\gamma)^{2}}. γ ( I − γ P π ⋆ ) − 1 ( P π ⋆ − P π ⋆ ) V ⋆ ∞ ≤ 2 N ( 1 − γ ) 3 2 β 1 + N ( 1 − γ ) 2 6 β 1 .
Notes
Sample complexity under a generative model. The generative model (or simulator) adopted in this chapter was first introduced by Kearns and Singh (1999). Since then, it has become a standard framework in the theoretical studies of RL and has been extensively investigated in subsequent work aimed at understanding and comparing the data efficiency of RL algorithms (e.g.,
Notes
Kearns et al. (2002), Kakade (2003), Lattimore and Hutter (2012), Azar et al. (2013), Sidford et al. (2018a), Sidford et al. (2018b), Wang (2020), Wainwright (2019c), Pananjady and Wainwright (2020), Agarwal et al. (2019b), Yang and Wang (2019), Mou et al. (2020), Wang et al. (2021a), Khamaru et al. (2021), Sidford et al. (2020), Li et al. (2022a), Vaswani et al. (2022), Shi et al. (2023), and Zhang et al. (2025c).
Several prior works have investigated the effectiveness of model-based RL for discounted infinite-horizon MDPs under a generative model. The seminal work Azar et al. (2013) was the first to establish the asymptotic minimax optimality of model-based RL, albeit under the requirement that the total sample size be at least on the order of S 2 A ( 1 − γ ) 2 \frac{S^2 A}{(1-\gamma)^2} ( 1 − γ ) 2 S 2 A . Subsequently, Agarwal et al. (2019b) significantly broadened the regime in which model-based methods are provably optimal. However, the results therein restrict the accuracy level to the range ε ∈ ( 0 , 1 1 − γ ] \varepsilon\in\left(0,\frac{1}{\sqrt{1-\gamma}}\right] ε ∈ ( 0 , 1 − γ 1 ] , and therefore do not cover the full ε \varepsilon ε -regime of interest; in other words, the optimality guarantees in Agarwal et al. (2019b) hold only when the total sample size surpasses the order of S A ( 1 − γ ) 2 \frac{SA}{(1-\gamma)^2} ( 1 − γ ) 2 S A . Similar technical barriers had already appeared in the simpler setting of model-based policy evaluation (Pananjady and Wainwright (2020); Agarwal et al. (2019b)), and had also arisen in the analysis of model-free algorithms (Sidford et al. (2018b); Wainwright (2019c)) and in the studies of finite-horizon nonstationary MDPs (Yin et al. (2021b)). In stark contrast, however, existing lower bounds (e.g., Azar et al. (2013)) do not rule out the possibility of achieving nontrivial statistical accuracy below such sample size thresholds, giving rise to a gap between upper and lower bounds in the most sample-starved regime.
The works of Li et al. (2020) and Li et al. (2024c) established the first full-range minimax-optimality guarantees for model-based policy-learning algorithms under a generative model, covering both discounted infinite-horizon and finite-horizon nonstationary MDPs. Their analyses consider a randomly perturbed variant of the model-based algorithm, in which the reward function is modified by an additional perturbation step. These results close the gap between achievable upper bounds and minimax lower bounds, thereby pinning down the fundamental sample complexity limit across the entire ε \varepsilon ε -range. More recently, Zurek and Chen (2025) showed, through a higher-order recursive analysis, that the reward-perturbation step is unnecessary; their result retains the original, unperturbed model-based algorithm. The analysis presented in this chapter builds upon the main proof ideas of Li et al. (2020) and Li et al. (2024c), while employing a different error decomposition and a new self-bounding argument to eliminate the need for reward perturbation.
Leave-one-out analyses. Finally, we remark that the construction of state-absorbing MDPs or state-action-absorbing MDPs falls under the category of “leave-one-out” type analysis, which is particularly effective in decoupling complicated statistical dependency in various high-dimensional statistical problems (Chen et al. (2021c)), such as high-dimensional M-estimation (El Karoui et al. (2013); El Karoui (2015); Javanmard and Montanari (2018)), phase retrieval (Chen et al. (2019a); Ma et al. (2020)), synchronization (Zhong and Boumal (2018); Abbe et al. (2020); Ling (2022)), ranking from pairwise comparisons (Chen et al. (2019b); Chen et al. (2022b); Fan et al. (2024); Fan et al. (2025a); Fan et al. (2026)), matrix completion and inference (Ma et al. (2020); Chen et al. (2019c); Chen et al. (2020b)), tensor completion (Cai et al. (2022)), network and clustering analysis (Abbe et al. (2020); Bhattacharya et al. (2026); Zhang and Zhou (2024); Agterberg and Zhang (2025); Ke and Wang (2025)). The first application of this powerful analysis framework to MDPs should be attributed to Agarwal et al. (2019b), which has since been applied and refined by Pananjady and Wainwright (2020), Li et al. (2020), Li et al. (2024c), Li et al. (2024b), Yan et al. (2024), and Shi et al. (2023) to establish the optimality of model-based RL in various settings.
RL with a generative model: the model-free approach
Thus far, we have established the statistical optimality of model-based RL algorithms under a generative model. Another widely studied, yet fundamentally different, paradigm is the model-free approach (see Figure 2.4 ), which avoids explicit model estimation and instead seeks to learn the desired value function and/or policy directly from data. In this chapter, we discuss some of the canonical model-free algorithms, once again assuming access to a generative model.
Prelude: stochastic approximation
The algorithmic and theoretical foundations of many model-free algorithms — such as temporal difference learning and Q-learning — are deeply rooted in the stochastic approximation framework, originally developed by Robbins and Monro (1951). In this section, we briefly review the key ideas underlying stochastic approximation.
Consider the problem of finding a fixed point of a given mapping F : R n → R n F:\mathbb{R}^{n} \to\mathbb{R}^{n} F : R n → R n , that is,
find x ∈ R n such that F ( x ) = x . (170) \text{find } x \in\mathbb{R}^{n} \qquad\text{such that } F(x)=x. \tag*{(170)} find x ∈ R n such that F ( x ) = x . (170)
In many practical settings, however, the mapping F ( ⋅ ) F(\cdot) F ( ⋅ ) is not directly accessible: one cannot evaluate F ( z ) F(z) F ( z ) exactly at an arbitrary point z ∈ R n z \in\mathbb{R}^{n} z ∈ R n . Instead, for any given query point z ∈ R n z \in\mathbb{R}^{n} z ∈ R n , one is permitted to draw a random sample f ( z ; ζ ) f(z;\zeta) f ( z ; ζ ) — with ζ \zeta ζ capturing the underlying randomness — that provides an unbiased, noisy estimate of F ( z ) F(z) F ( z ) .
To address the challenge arising from the lack of direct access to F ( ⋅ ) F(\cdot) F ( ⋅ ) , Robbins and Monro (1951) proposed the following iterative scheme for solving (170) based on successive draws of random samples:
x t = x t − 1 + η t ( f ( x t − 1 ; ζ t ) − x t − 1 ) , t = 1 , 2 , … (171) x_t = x_{t-1} + \eta_t\bigl(f(x_{t-1};\zeta_t)-x_{t-1}\bigr), \qquad t=1,2,\ldots \tag*{(171)} x t = x t − 1 + η t ( f ( x t − 1 ; ζ t ) − x t − 1 ) , t = 1 , 2 , … (171)
where x 0 ∈ R n x_0 \in\mathbb{R}^{n} x 0 ∈ R n indicates an initial guess, 0 < η t < 1 0 < \eta_t < 1 0 < η t < 1 denotes the learning rate at iteration t t t , and f ( x t − 1 ; ζ t ) f(x_{t-1};\zeta_t) f ( x t − 1 ; ζ t ) is a fresh sample satisfying
E [ f ( x t − 1 ; ζ t ) ∣ x t − 1 ] = F ( x t − 1 ) . \mathbb{E}\bigl[f(x_{t-1};\zeta_t)\mid x_{t-1}\bigr] = F(x_{t-1}). E [ f ( x t − 1 ; ζ t ) ∣ x t − 1 ] = F ( x t − 1 ) .
Under suitable conditions on the mapping F ( ⋅ ) F(\cdot) F ( ⋅ ) (e.g., monotonicity and quasi-contractivity w.r.t. appropriate cones), the data generating process (e.g., bounded noise variance), and the learning rate sequence { η t } t ≥ 0 \{\eta_t\}_{t\geq0} { η t } t ≥ 0 (e.g., linearly or polynomial decaying stepsizes), this remarkably simple procedure (171) is guaranteed to converge to a solution of (4.1 ((Lai, 2003; Borkar and Borkar, 2008)). See, for example, Wainwright (2019b, Section 2.2) for a non-asymptotic treatment.
The stochastic approximation paradigm has had far-reaching impacts across statistics, optimization, machine learning, and many other fields. A particularly influential descendant is stochastic gradient descent , now a cornerstone of modern machine learning ((Bottou, 2010; Bottou et al., 2018)).
Temporal difference learning
Equipped with the stochastic approximation framework, we are now ready to introduce several canonical model-free RL algorithms. As in Chapter 3, we begin with the simpler task of policy evaluation, previously introduced in § 4.2 . Given a policy π \pi π and a target precision level ε > 0 \varepsilon> 0 ε > 0 , the objective is to compute an estimate of the corresponding value function V π V^\pi V π , denoted by V ^ π ∈ R S \widehat{V}^\pi\in\mathbb{R}^{S} V π ∈ R S , such that
∥ V ^ π − V π ∥ ∞ ≤ ε . \left\|\widehat{V}^\pi- V^\pi\right\|_\infty\leq\varepsilon. V π − V π ∞ ≤ ε .
In sharp contrast to the model-based policy evaluation procedure (80) , which solves the Bellman equation using an explicit estimate of the probability transition kernel P ^ π \widehat{P}_\pi P π , we now consider an alternative approach that entirely avoids estimating P ^ π \widehat{P}_\pi P π .
Algorithm
Arguably, the most well-known model-free policy evaluation algorithm is temporal-difference (TD) learning, originally formalized by Sutton (1988) and extensively analyzed in the literature (e.g., Tesauro et al. (1995), Tsitsiklis and Van Roy (1997), Bhandari et al. (2021), Srikant and Ying (2019), and Li et al. (2024d)). The core idea of TD learning is to apply stochastic approximation to solve the Bellman equation (or equivalently, to find a fixed point of the Bellman operator).
More concretely, recall that the value function of a policy π \pi π is characterized by the Bellman consistency equation (2.12). Consequently, computing V π V^\pi V π amounts to finding a fixed point of the mapping
F ( V ) = T π ( V ) : = r π + γ P π V . F(V) = T_\pi(V) := r_\pi+ \gamma P_\pi V. F ( V ) = T π ( V ) := r π + γ P π V .
This reformulation situates policy evaluation within the stochastic approximation framework. In particular, it naturally suggests an iterative scheme: maintain an estimate V t : S → R V_t:\mathcal{S} \to\mathbb{R} V t : S → R of the value function in each iteration t t t , and update it according to
V t = V t − 1 + η t ( T π , t ( V t − 1 ) − V t − 1 ) , t = 1 , 2 , … (172) V_t = V_{t-1} + \eta_t\left(T_{\pi,t}(V_{t-1}) - V_{t-1}\right), \qquad t = 1,2,\ldots \tag*{(172)} V t = V t − 1 + η t ( T π , t ( V t − 1 ) − V t − 1 ) , t = 1 , 2 , … (172)
provided that, at each iteration, one can construct an unbiased estimate T π , t ( V t − 1 ) T_{\pi,t}(V_{t-1}) T π , t ( V t − 1 ) of the Bellman operator applied to the current iterate, satisfying
E [ T π , t ( V t − 1 ) ∣ V t − 1 ] = T π ( V t − 1 ) . (173) \mathbb{E}\left[T_{\pi,t}(V_{t-1}) \mid V_{t-1}\right] = T_\pi(V_{t-1}). \tag*{(173)} E [ T π , t ( V t − 1 ) ∣ V t − 1 ] = T π ( V t − 1 ) . (173)
When a generative model is available, this scheme (4.3) admits a particularly simple implementation: at each iteration t t t ,
Algorithm 4.1: TD learning with a generative model
input: reward function r π r_\pi r π induced by policy π \pi π , discount factor γ \gamma γ , learning rates { η t } \{\eta_t\} { η t } , number of iterations T T T , initialization V 0 V_0 V 0 .
for t = 1 , 2 , ⋯ , T t = 1, 2, \cdots, T t = 1 , 2 , ⋯ , T do
draw a sample s t ( s ) ∼ i n d . P π ( ⋅ ∣ s ) s_t(s) \mathrel{\overset{\mathrm{ind.}}{\sim}} P_\pi(\cdot\mid s) s t ( s ) ∼ ind. P π ( ⋅ ∣ s ) (cf. (7) ) for every s ∈ S s \in\mathcal{S} s ∈ S . /* stochastic approximation. */
for every s ∈ S s \in\mathcal{S} s ∈ S , update V t ( s ) V_t(s) V t ( s ) as
V t ( s ) = ( 1 − η t ) V t − 1 ( s ) + η t { r π ( s ) + γ V t − 1 ( s t ( s ) ) } . V_t(s) = (1-\eta_t)V_{t-1}(s) + \eta_t\{r_\pi(s) + \gamma V_{t-1}(s_t(s))\}. V t ( s ) = ( 1 − η t ) V t − 1 ( s ) + η t { r π ( s ) + γ V t − 1 ( s t ( s ))} .
V t ( s ) = V t − 1 ( s ) + η t { r π ( s ) + γ V t − 1 ( s t ) − V t − 1 ( s ) } (174) V_t(s) = V_{t-1}(s) + \eta_t\{r_\pi(s) + \gamma V_{t-1}(s_t) - V_{t-1}(s)\} \tag*{(174)} V t ( s ) = V t − 1 ( s ) + η t { r π ( s ) + γ V t − 1 ( s t ) − V t − 1 ( s )} (174)
= ( 1 − η t ) V t − 1 ( s ) + η t ( r π ( s ) + γ V t − 1 ( s t ) ) , (175) = (1-\eta_t)V_{t-1}(s) + \eta_t\left(r_\pi(s) + \gamma V_{t-1}(s_t)\right), \tag*{(175)} = ( 1 − η t ) V t − 1 ( s ) + η t ( r π ( s ) + γ V t − 1 ( s t ) ) , (175)
with s t s_t s t taken to be s t ( s ) s_t(s) s t ( s ) .
A straightforward calculation shows that this update is unbiased, i.e.,
E [ V t ∣ V t − 1 ] = ( 1 − η t ) V t − 1 + η t ( r π + γ P π V t − 1 ) , \mathbb{E}[V_t \mid V_{t-1}] = (1-\eta_t)V_{t-1} + \eta_t(r_\pi+ \gamma P_\pi V_{t-1}), E [ V t ∣ V t − 1 ] = ( 1 − η t ) V t − 1 + η t ( r π + γ P π V t − 1 ) ,
so that (175) indeed implements a stochastic approximation procedure for solving the Bellman equation. Notably, the term
r π ( s ) + γ V t − 1 ( s t ) − V t − 1 ( s ) r_\pi(s) + \gamma V_{t-1}(s_t) - V_{t-1}(s) r π ( s ) + γ V t − 1 ( s t ) − V t − 1 ( s )
in (174) is often referred to as the “temporal difference”, as it reflects the discrepancy between the current value estimate and a one-step look-ahead estimate obtained by bootstrapping one step forward in the MDP. The complete procedure is summarized in Algorithm 4.1. This variant of TD learning is commonly termed synchronous TD learning, since each iteration updates all coordinates of the value function estimate simultaneously.
In contrast to the model-based approach in Algorithm 3.1, TD learning maintains only S \mathcal{S} S -dimensional value iterates and does not require storing an explicit S 2 \mathcal{S}^2 S 2 -dimensional estimate of the transition kernel induced by π \pi π . This substantially reduces memory requirements, highlighting a central appeal of model-free algorithms.
Theoretical guarantees
We now analyze the efficacy of TD learning through the lens of sample complexity. To this end, we need to specify the learning rate schedule { η t } t ≥ 1 \{\eta_t\}_{t \ge1} { η t } t ≥ 1 , as it plays a crucial role in determining the algorithm’s performance. In this section, we choose the learning rates to satisfy
1 1 + c 1 ( 1 − γ ) T log 2 T ≤ η t ≤ 1 1 + c 2 ( 1 − γ ) log 2 T t , 1 ≤ t ≤ T , (176) \frac{1}{1+\frac{c_1(1-\gamma)T}{\log^2 T}} \le\eta_t \le\frac{1}{1+\frac{c_2(1-\gamma)}{\log^2 T}t}, \qquad1 \le t \le T, \tag*{(176)} 1 + l o g 2 T c 1 ( 1 − γ ) T 1 ≤ η t ≤ 1 + l o g 2 T c 2 ( 1 − γ ) t 1 , 1 ≤ t ≤ T , (176)
where c 1 ≥ c 2 > 0 c_1 \ge c_2 > 0 c 1 ≥ c 2 > 0 are some universal constants. Clearly, Condition (176) encompasses two commonly used learning-rate choices as special cases:
constant learning rates, where η t ≡ 1 1 + c 1 ( 1 − γ ) T log 2 T \eta_t \equiv\frac{1}{1+\frac{c_1(1-\gamma)T}{\log^2 T}} η t ≡ 1 + l o g 2 T c 1 ( 1 − γ ) T 1 for all t ≤ T t \le T t ≤ T ;
rescaled linear learning rates, where η t = 1 1 + c 2 ( 1 − γ ) log 2 T t \eta_t = \frac{1}{1+\frac{c_2(1-\gamma)}{\log^2 T}t} η t = 1 + l o g 2 T c 2 ( 1 − γ ) t 1 for all t ≤ T t \le T t ≤ T .
The following theorem characterizes the ℓ ∞ \ell_\infty ℓ ∞ -based sample complexity of TD learning under a generative model, which sharpens the sample complexity rate first established by Li et al. (2024a). It is worth noting that the guarantees in Li et al. (2024a) were established only for ε ∈ ( 0 , 1 ] \varepsilon\in(0,1] ε ∈ ( 0 , 1 ] . In this chapter, we develop a refined analysis that extends the range of ε \varepsilon ε to the full range ε ∈ ( 0 , 1 1 − γ ] \varepsilon\in\left(0,\frac{1}{1-\gamma}\right] ε ∈ ( 0 , 1 − γ 1 ] .
Theorem 4.1 (Extension of Li {italic et al.} (2024a)). Consider any δ ∈ ( 0 , 1 ) \delta\in(0,1) δ ∈ ( 0 , 1 ) , ε ∈ ( 0 , 1 1 − γ ] \varepsilon\in\left(0,\frac{1}{1-\gamma}\right] ε ∈ ( 0 , 1 − γ 1 ] , and γ ∈ [ 1 / 2 , 1 ) \gamma\in[1/2,1) γ ∈ [ 1/2 , 1 ) . Suppose that the learning rates satisfy (176) for some small enough universal constants c 1 ≥ c 2 > 0 c_1 \ge c_2 > 0 c 1 ≥ c 2 > 0 , and that the total number of iterations T ≥ 2 T \ge2 T ≥ 2 obeys
T ≥ c 3 ( log 3 T ) ( log S T δ ) ( 1 − γ ) 3 ε 2 . (177) T \ge\frac{c_3(\log^3 T)\left(\log\frac{ST}{\delta}\right)}{(1-\gamma)^3\varepsilon^2}. \tag*{(177)} T ≥ ( 1 − γ ) 3 ε 2 c 3 ( log 3 T ) ( log δ S T ) . (177) for some sufficiently large universal constant c 3 > 0 c_3 > 0 c 3 > 0 . If 0 ≤ V 0 ( s ) ≤ 1 1 − γ 0 \le V_0(s) \le\frac{1}{1-\gamma} 0 ≤ V 0 ( s ) ≤ 1 − γ 1 for all s ∈ S s \in\mathcal{S} s ∈ S , then with probability at least 1 − δ 1-\delta 1 − δ , TD learning (see Algorithm 4.1) achieves
∀ s ∈ S : ∣ V T ( s ) − V π ( s ) ∣ ≤ ε . \forall s \in\mathcal{S}: \qquad\left|V_T(s)-V^\pi(s)\right| \le\varepsilon. ∀ s ∈ S : ∣ V T ( s ) − V π ( s ) ∣ ≤ ε .
Remark 4.1 (Mean estimation guarantee). In view of the trivial upper bound ∣ V T ( s ) − V π ( s ) ∣ ≤ 1 1 − γ \left|V_T(s)-V^\pi(s)\right| \le\frac{1}{1-\gamma} ∣ V T ( s ) − V π ( s ) ∣ ≤ 1 − γ 1 , the high-probability result in Theorem 4.1 immediately implies an error bound in expectation. Specifically, choosing δ = ε ( 1 − γ ) / 2 ∈ ( 0 , 1 ) \delta=\varepsilon(1-\gamma)/2 \in(0,1) δ = ε ( 1 − γ ) /2 ∈ ( 0 , 1 ) in Theorem 4.1 and decomposing according to the high-probability event, we obtain
E [ max s ∈ S ∣ V T ( s ) − V π ( s ) ∣ ] ≤ ε ( 1 − δ ) + 1 1 − γ δ ≤ 2 ε , \mathbb{E}\left[\max_{s\in\mathcal{S}}\left|V_T(s)-V^\pi(s)\right|\right] \le\varepsilon(1-\delta)+\frac{1}{1-\gamma}\delta\le2\varepsilon, E [ s ∈ S max ∣ V T ( s ) − V π ( s ) ∣ ] ≤ ε ( 1 − δ ) + 1 − γ 1 δ ≤ 2 ε , as long as the total number of iterations exceeds
T ≥ c 3 ( log 3 T ) ( log 2 S T ε ( 1 − γ ) ) ( 1 − γ ) 3 ε 2 . T \ge\frac{c_3(\log^3 T)\left(\log\frac{2ST}{\varepsilon(1-\gamma)}\right)}{(1-\gamma)^3\varepsilon^2}. T ≥ ( 1 − γ ) 3 ε 2 c 3 ( log 3 T ) ( log ε ( 1 − γ ) 2 S T ) .
Given that each iteration of TD learning draws S S S fresh samples (one per state), the total number of samples used by Algorithm 4.1 is S T ST S T . Consequently, Theorem 4.1 immediately implies that, to yield an accuracy level ε ∈ ( 0 , 1 1 − γ ] \varepsilon\in\left(0,\frac{1}{1-\gamma}\right] ε ∈ ( 0 , 1 − γ 1 ] , TD learning needs no more than
O ~ ( S ( 1 − γ ) 3 ε 2 ) \widetilde{O}\left(\frac{S}{(1-\gamma)^3\varepsilon^2}\right) O ( ( 1 − γ ) 3 ε 2 S )
samples, and moreover, the last iterate of this iterative procedure already attains the desired accuracy. This non-asymptotic result is valid provided the learning rates are properly chosen — either as a suitable constant or according to a rescaled linear schedule (see (176) ). In comparison with the minimax lower bound presented in Theorem 3.7 (by setting A = 1 A=1 A = 1 therein), Theorem 4.1 demonstrates that plain TD learning — without any additional algorithmic trick like iterate averaging or variance reduction — attains minimax-optimal sample complexity (up to logarithmic factors) when one draws samples from the generative model, for the full accuracy range ε ∈ ( 0 , 1 1 − γ ] \varepsilon\in\left(0,\frac{1}{1-\gamma}\right] ε ∈ ( 0 , 1 − γ 1 ] .
Analysis: a first attempt
As a starting point, we present an initial — albeit suboptimal — proof strategy that captures some of the core ideas for analyzing the performance of TD learning; this approach will be refined shortly in § 7.4 .
Let us adopt the vector and matrix notation introduced in § 2.5 , viewing V t V_t V t , V π V^\pi V π and r π r_\pi r π as vectors in R S \mathbb{R}^S R S , among others. By defining an empirical transition matrix P t ∈ { 0 , 1 } S × S P_t \in\{0,1\}^{S \times S} P t ∈ { 0 , 1 } S × S such that
∀ ( s , s ′ ) ∈ S , P t ( s , s ′ ) : = { 1 , if s ′ = s t ( s ) , 0 , otherwise , (178) \forall(s,s') \in\mathcal{S}, \qquad P_t(s,s') :=
\begin{cases}
1, & \text{if } s' = s_t(s),\\
0, & \text{otherwise},
\end{cases} \tag*{(178)} ∀ ( s , s ′ ) ∈ S , P t ( s , s ′ ) := { 1 , 0 , if s ′ = s t ( s ) , otherwise , (178)
where we recall that s t ( s ) ∼ i n d . P π ( ⋅ ∣ s ) s_t(s) \mathrel{\overset{\mathrm{ind.}}{\sim}} P_\pi(\cdot\mid s) s t ( s ) ∼ ind. P π ( ⋅ ∣ s ) , we can express the TD learning update rule (175) compactly in vector-matrix form as follows:
V t = ( 1 − η t ) V t − 1 + η t ( r π + γ P t V t − 1 ) , t ≥ 1. (179) V_t = (1-\eta_t)V_{t-1}+\eta_t(r_\pi+\gamma P_tV_{t-1}), \qquad t \ge1. \tag*{(179)} V t = ( 1 − η t ) V t − 1 + η t ( r π + γ P t V t − 1 ) , t ≥ 1. (179)
Preliminary facts. Before proceeding, we gather a few elementary facts that will be useful throughout the analysis. The following lemma asserts that the TD iterates always fall within the admissible range of value functions. The proof, based on induction, is deferred to § 7.15.1 .
Lemma 4.2. If 0 ≤ V 0 ( s ) ≤ 1 1 − γ 0 \le V_0(s) \le\frac{1}{1-\gamma} 0 ≤ V 0 ( s ) ≤ 1 − γ 1 for all s ∈ S s \in\mathcal{S} s ∈ S , then the TD learning iterates (179) obey
0 ≤ V t ≤ 1 1 − γ 1 S and ∥ V t − V π ∥ ∞ ≤ 1 1 − γ 0 \le V_t \le\frac{1}{1-\gamma}\mathbf{1}_S \quad\text{and} \quad\lVert V_t-V^\pi\rVert_\infty\le\frac{1}{1-\gamma} 0 ≤ V t ≤ 1 − γ 1 1 S and ∥ V t − V π ∥ ∞ ≤ 1 − γ 1 for all 0 ≤ t ≤ T 0 \le t \le T 0 ≤ t ≤ T , provided that 0 ≤ η t ≤ 1 0 \le\eta_t \le1 0 ≤ η t ≤ 1 for all 0 ≤ t ≤ T 0 \le t \le T 0 ≤ t ≤ T .
Next, given the iterative nature of TD learning, it is convenient to introduce the following quantities that, as we shall see momentarily, capture the cumulative effect of the learning rates { η t } \{\eta_t\} { η t } :
η k ( t ) : = { ∏ i = 1 t ( 1 − η i ( 1 − γ ) ) , if k = 0 , η k ∏ i = k + 1 t ( 1 − η i ( 1 − γ ) ) , if 0 < k < t , η t , if k = t . (180) \eta_k^{(t)} :=
\begin{cases}
\prod_{i=1}^{t}(1-\eta_i(1-\gamma)), & \text{if } k=0,\\
\eta_k\prod_{i=k+1}^{t}(1-\eta_i(1-\gamma)), & \text{if } 0<k<t,\\
\eta_t, & \text{if } k=t.
\end{cases} \tag*{(180)} η k ( t ) := ⎩ ⎨ ⎧ ∏ i = 1 t ( 1 − η i ( 1 − γ )) , η k ∏ i = k + 1 t ( 1 − η i ( 1 − γ )) , η t , if k = 0 , if 0 < k < t , if k = t . (180)
Some basic properties of η k ( t ) \eta_k^{(t)} η k ( t ) are stated below. In particular, η k ( t ) \eta_k^{(t)} η k ( t ) can be very small when k k k is small. The proof is elementary and deferred to § 7.16 .
Lemma 4.3. Recall that the learning rates are chosen to satisfy (176) , and suppose c 1 c 2 ≤ 1 / 8 c_1c_2 \le1/8 c 1 c 2 ≤ 1/8 . Then for any t ≥ T / ( c 2 log T ) t \ge T/(c_2 \log T) t ≥ T / ( c 2 log T ) , one has
η k ( t ) ≤ { 1 T 2 , if 0 ≤ k ≤ t / 2 , 2 log 3 T ( 1 − γ ) T , if t / 2 < k ≤ t . \eta_k^{(t)} \le
\begin{cases}
\frac{1}{T^2}, & \text{if } 0 \le k \le t/2,\\
\frac{2\log^3 T}{(1-\gamma)T}, & \text{if } t/2<k\le t.
\end{cases} η k ( t ) ≤ { T 2 1 , ( 1 − γ ) T 2 l o g 3 T , if 0 ≤ k ≤ t /2 , if t /2 < k ≤ t .
Step 1: decomposing the error V t − V π V_t-V^\pi V t − V π . At a high level, this step unrolls the TD recursion and decomposes the estimation error into an initialization term and a martingale term. For notational convenience, define the error sequence
Δ t : = V t − V π . (181) \Delta_t := V_t-V^\pi. \tag*{(181)} Δ t := V t − V π . (181)
In view of the TD learning update rule (179) , we have the following decomposition:
Δ t = ( 1 − η t ) ( V t − 1 − V π ) + η t ( r π + γ P t V t − 1 − V π ) = ( 1 − η t ) Δ t − 1 + η t γ ( P t V t − 1 − P π V π ) = ( 1 − η t ) Δ t − 1 + η t γ ( P π Δ t − 1 + ( P t − P π ) V t − 1 ) (182) \begin{aligned}
\Delta_t &= (1-\eta_t)(V_{t-1}-V^\pi)+\eta_t(r_\pi+\gamma P_tV_{t-1}-V^\pi)\\
&=(1-\eta_t)\Delta_{t-1}+\eta_t\gamma(P_tV_{t-1}-P_\pi V^\pi)\\
&=(1-\eta_t)\Delta_{t-1}+\eta_t\gamma(P_\pi\Delta_{t-1}+(P_t-P_\pi)V_{t-1}) \tag*{(182)}
\end{aligned} Δ t = ( 1 − η t ) ( V t − 1 − V π ) + η t ( r π + γ P t V t − 1 − V π ) = ( 1 − η t ) Δ t − 1 + η t γ ( P t V t − 1 − P π V π ) = ( 1 − η t ) Δ t − 1 + η t γ ( P π Δ t − 1 + ( P t − P π ) V t − 1 ) (182)
= ( I − η t ( I − γ P π ) ) Δ t − 1 + η t γ ( P t − P π ) V t − 1 , (183) =(I-\eta_t(I-\gamma P_\pi))\Delta_{t-1}+\eta_t\gamma(P_t-P_\pi)V_{t-1}, \tag*{(183)} = ( I − η t ( I − γ P π )) Δ t − 1 + η t γ ( P t − P π ) V t − 1 , (183)
where P π P_\pi P π is defined in (28) , and the second line arises from the Bellman equation V π = r π + γ P π V π V^\pi=r_\pi+\gamma P_\pi V^\pi V π = r π + γ P π V π (cf. (34) ). Iterating the recursion (183) yields
Δ t = ∏ i = 1 t ( I − η i ( I − γ P π ) ) Δ 0 + ∑ k = 1 t Z k , (184) \Delta_t=\prod_{i=1}^{t}(I-\eta_i(I-\gamma P_\pi))\Delta_0+\sum_{k=1}^{t}Z_k, \tag*{(184)} Δ t = i = 1 ∏ t ( I − η i ( I − γ P π )) Δ 0 + k = 1 ∑ t Z k , (184)
where the process { Z k : 1 ≤ k ≤ t } \{Z_k:1\le k\le t\} { Z k : 1 ≤ k ≤ t } is defined as
Z k : = γ η k ( ∏ i = k + 1 t ( I − η i ( I − γ P π ) ) ) ( P k − P π ) V k − 1 . (185) Z_k:=\gamma\eta_k\left(\prod_{i=k+1}^{t}(I-\eta_i(I-\gamma P_\pi))\right)(P_k-P_\pi)V_{k-1}. \tag*{(185)} Z k := γ η k ( i = k + 1 ∏ t ( I − η i ( I − γ P π )) ) ( P k − P π ) V k − 1 . (185)
In words, the error at iteration t t t decomposes into a suitably contracted version of the initial error and a martingale sum, since each Z k Z_k Z k has zero conditional mean given the history up to iteration k − 1 k-1 k − 1 and hence { Z k } \{Z_k\} { Z k } forms a martingale difference sequence (this is an immediate consequence of the definition (178) of P k P_k P k ). In what follows, we shall look at the two terms on the right-hand side of (184) separately.
Step 2: bounding the first term in (184) . We first show that the first term on the right-hand side of (184) is negligible. To this end, it is observed that
∥ ∏ i = 1 t ( I − η i ( I − γ P π ) ) Δ 0 ∥ ∞ ≤ ∥ ∏ i = 1 t ( I − η i ( I − γ P π ) ) ∥ 1 ∥ Δ 0 ∥ ∞ = { ∏ i = 1 t ( 1 − η i ( 1 − γ ) ) } ∥ Δ 0 ∥ ∞ = η 0 ( t ) ∥ Δ 0 ∥ ∞ ≤ 1 ( 1 − γ ) T 2 . (186) \begin{aligned}
\left\|\prod_{i=1}^{t}(I-\eta_i(I-\gamma P_\pi))\Delta_0\right\|_\infty
&\le\left\|\prod_{i=1}^{t}(I-\eta_i(I-\gamma P_\pi))\right\|_1\|\Delta_0\|_\infty\\
&=\left\{\prod_{i=1}^{t}(1-\eta_i(1-\gamma))\right\}\|\Delta_0\|_\infty\\
&=\eta_0^{(t)}\|\Delta_0\|_\infty\le\frac{1}{(1-\gamma)T^2}. \tag*{(186)}
\end{aligned} i = 1 ∏ t ( I − η i ( I − γ P π )) Δ 0 ∞ ≤ i = 1 ∏ t ( I − η i ( I − γ P π )) 1 ∥ Δ 0 ∥ ∞ = { i = 1 ∏ t ( 1 − η i ( 1 − γ )) } ∥ Δ 0 ∥ ∞ = η 0 ( t ) ∥ Δ 0 ∥ ∞ ≤ ( 1 − γ ) T 2 1 . (186)
Here, (i) invokes the basic fact that, for any 0 ≤ k < t 0\le k<t 0 ≤ k < t ,
∥ ∏ i = k + 1 t ( I − η i ( I − γ P π ) ) ∥ 1 = ∏ i = k + 1 t ( 1 − η i ( 1 − γ ) ) , (187) \left\|\prod_{i=k+1}^{t}(I-\eta_i(I-\gamma P_\pi))\right\|_1=\prod_{i=k+1}^{t}(1-\eta_i(1-\gamma)), \tag*{(187)} i = k + 1 ∏ t ( I − η i ( I − γ P π )) 1 = i = k + 1 ∏ t ( 1 − η i ( 1 − γ )) , (187)
given that P π P_\pi P π is a probability transition matrix; and (ii) results from Lemma 4.3 and the assumption that ∥ Δ 0 ∥ ∞ ≤ 1 1 − γ \|\Delta_0\|_\infty\le\frac{1}{1-\gamma} ∥ Δ 0 ∥ ∞ ≤ 1 − γ 1 , provided that t ≥ T / ( c 2 log T ) t\ge T/(c_2\log T) t ≥ T / ( c 2 log T ) . As will be shown shortly, this upper bound (186) is negligible relative to the other term.
Step 3: bounding the martingale sum in (184) . We now turn to the second term on the right-hand side of (184) . Recall that { Z k } \{Z_k\} { Z k } forms a martingale difference sequence. In order to control the martingale sum ∑ k Z k \sum_k Z_k ∑ k Z k , a natural approach is to apply standard martingale concentration inequalities, such as the Azuma–Hoeffding inequality (cf. Lemma 2.8 ). To this end, we first note that
∥ Z k ∥ ∞ = γ ∥ η k ∏ i = k + 1 t ( I − η i ( I − γ P π ) ) ( P k − P π ) V k − 1 ∥ ∞ ≤ γ ∥ η k ∏ i = k + 1 t ( I − η i ( I − γ P π ) ) ∥ 1 ∥ ( P k − P π ) V k − 1 ∥ ∞ = (i) γ { η k ∏ i = k + 1 t ( 1 − η i ( 1 − γ ) ) } ∥ ( P k − P π ) V k − 1 ∥ ∞ ≤ γ η k ( t ) ( ∥ P k ∥ 1 + ∥ P π ∥ 1 ) ∥ V k − 1 ∥ ∞ ≤ (ii) { 4 γ log 3 T ( 1 − γ ) 2 T , if k > t / 2 , 2 γ ( 1 − γ ) T 2 , if 0 ≤ k ≤ t / 2 = : B Z k , (188) \begin{aligned}
\|Z_k\|_\infty
&= \gamma\left\|\eta_k\prod_{i=k+1}^{t}\left(I-\eta_i(I-\gamma P_\pi)\right)(P_k-P_\pi)V_{k-1}\right\|_\infty\\
&\leq\gamma\left\|\eta_k\prod_{i=k+1}^{t}\left(I-\eta_i(I-\gamma P_\pi)\right)\right\|_1\left\|(P_k-P_\pi)V_{k-1}\right\|_\infty\\
&\mathrel{\overset{\text{(i)}}{=}}\gamma\left\{\eta_k\prod_{i=k+1}^{t}\left(1-\eta_i(1-\gamma)\right)\right\}\left\|(P_k-P_\pi)V_{k-1}\right\|_\infty\\
&\leq\gamma\eta_k^{(t)}\left(\|P_k\|_1+\|P_\pi\|_1\right)\|V_{k-1}\|_\infty\\
&\mathrel{\overset{\text{(ii)}}{\leq}}
\begin{cases}
\frac{4\gamma\log^3 T}{(1-\gamma)^2T}, & \text{if } k>t/2,\\[6pt]
\frac{2\gamma}{(1-\gamma)T^2}, & \text{if } 0\leq k\leq t/2
\end{cases}
=: BZ_k, \tag*{(188)}
\end{aligned} ∥ Z k ∥ ∞ = γ η k i = k + 1 ∏ t ( I − η i ( I − γ P π ) ) ( P k − P π ) V k − 1 ∞ ≤ γ η k i = k + 1 ∏ t ( I − η i ( I − γ P π ) ) 1 ∥ ( P k − P π ) V k − 1 ∥ ∞ = (i) γ { η k i = k + 1 ∏ t ( 1 − η i ( 1 − γ ) ) } ∥ ( P k − P π ) V k − 1 ∥ ∞ ≤ γ η k ( t ) ( ∥ P k ∥ 1 + ∥ P π ∥ 1 ) ∥ V k − 1 ∥ ∞ ≤ (ii) ⎩ ⎨ ⎧ ( 1 − γ ) 2 T 4 γ l o g 3 T , ( 1 − γ ) T 2 2 γ , if k > t /2 , if 0 ≤ k ≤ t /2 =: B Z k , (188)
provided that t ≥ T / ( c 2 log T ) t\geq T/(c_2\log T) t ≥ T / ( c 2 log T ) . Here, (i) follows from (187) , whereas (ii) arises from the facts ∥ P k ∥ 1 = ∥ P π ∥ 1 = 1 \|P_k\|_1=\|P_\pi\|_1=1 ∥ P k ∥ 1 = ∥ P π ∥ 1 = 1 (recall the definition (4.7 of P k P_k P k ), as well as Lemmas 4.2 and 4.3 .
Take t = T t=T t = T . Armed with the bound (188) , invoking the Azuma–Hoeffding inequality in Lemma 2.8 and applying the union bound over all coordinates of ∑ k Z k \sum_k Z_k ∑ k Z k , we arrive at
∥ ∑ k = 1 T Z k ∥ ∞ ≤ ( 2 ∑ k = 1 T B Z k 2 log ( 2 S δ ) ) 1 / 2 ≤ γ ( 16 log 6 T log ( 2 S δ ) ( 1 − γ ) 4 T + 4 log ( 2 S δ ) ( 1 − γ ) 2 T 3 ) 1 / 2 ≤ O ~ ( 1 ( 1 − γ ) 2 T ) (189) \begin{aligned}
\left\|\sum_{k=1}^{T}Z_k\right\|_\infty
\leq\left(2\sum_{k=1}^{T}B_{Z_k}^{2}\log\left(\frac{2S}{\delta}\right)\right)^{1/2} \\
&\leq\gamma\left(\frac{16\log^6T\log\left(\frac{2S}{\delta}\right)}{(1-\gamma)^4T}+\frac{4\log\left(\frac{2S}{\delta}\right)}{(1-\gamma)^2T^3}\right)^{1/2} \\
&\leq\widetilde{O}\left(\frac{1}{(1-\gamma)^2\sqrt{T}}\right) \tag*{(189)}
\end{aligned} k = 1 ∑ T Z k ∞ ≤ ( 2 k = 1 ∑ T B Z k 2 log ( δ 2 S ) ) 1/2 ≤ γ ( ( 1 − γ ) 4 T 16 log 6 T log ( δ 2 S ) + ( 1 − γ ) 2 T 3 4 log ( δ 2 S ) ) 1/2 ≤ O ( ( 1 − γ ) 2 T 1 ) (189)
with probability greater than 1 − δ 1-\delta 1 − δ .
Step 4: putting all pieces together. Substituting the above results (186) and (189) into (184) , we arrive at
∥ Δ T ∥ ∞ ≤ O ~ ( 1 ( 1 − γ ) 2 T ) . \|\Delta_T\|_\infty\leq\widetilde{O}\left(\frac{1}{(1-\gamma)^2\sqrt{T}}\right). ∥ Δ T ∥ ∞ ≤ O ( ( 1 − γ ) 2 T 1 ) .
with high probability. In other words, this result tells us that, in order to yield ε \varepsilon ε accuracy (i.e., ∥ Δ T ∥ ∞ ≤ ε \|\Delta_T\|_\infty\leq\varepsilon ∥ Δ T ∥ ∞ ≤ ε ), it suffices to take T T T on the order of
O ~ ( 1 ( 1 − γ ) 4 ε 2 ) \widetilde{O}\left(\frac{1}{(1-\gamma)^4\varepsilon^2}\right) O ( ( 1 − γ ) 4 ε 2 1 )
iterations.
Discussion. While the above proof is simple and fully non-asymptotic, it yields an iteration complexity that exceeds the bound in Theorem 4.1 by a multiplicative factor of 1 1 − γ \frac{1}{1-\gamma} 1 − γ 1 , and therefore falls short of establishing the minimax optimality of synchronous TD learning. A closer inspection reveals that this suboptimality stems primarily from the use of the Azuma–Hoeffding inequality (cf. Lemma 2.8 ), which fails to exploit the underlying variance structure — an ingredient that is crucial for shaving the extra 1 1 − γ \frac{1}{1-\gamma} 1 − γ 1 factor. To sharpen the analysis, we would need to invoke the Freedman inequality (cf. Lemma 2.9 ) and leverage the intimate connection between the cumulative conditional covariance and the intrinsic variance structure of the discounted infinite-horizon MDP, as developed in the next subsection.
Sharp analysis: proof of Theorem 4.1
We now present a refined analysis of TD learning that, as noted above, leverages the Freedman inequality for martingales in conjunction with the intrinsic variance structure of the discounted infinite-horizon MDP.
Step 1: decomposing the error as in Section 4.2.3. As already shown in § 7.3.1 (see (184) and (186) ), we can decompose and bound the estimation error Δ t = V t − V π \Delta_t = V_t - V^\pi Δ t = V t − V π at iteration t t t as
∥ Δ t ∥ ∞ ≤ 1 ( 1 − γ ) T 2 + ∥ ∑ k = 1 t Z k ∥ ∞ , (190) \|\Delta_t\|_\infty\le\frac{1}{(1-\gamma)T^2} + \left\|\sum_{k=1}^{t} Z_k\right\|_\infty, \tag*{(190)} ∥ Δ t ∥ ∞ ≤ ( 1 − γ ) T 2 1 + k = 1 ∑ t Z k ∞ , (190)
where { Z k : 1 ≤ k ≤ t } \{Z_k : 1 \le k \le t\} { Z k : 1 ≤ k ≤ t } is defined in (185) . It thus boils down to bounding the entrywise magnitude of the martingale sum ∑ k Z k \sum_k Z_k ∑ k Z k .
Step 2: bounding the martingale sum via Freedman’s inequality. We intend to apply the Freedman inequality in Lemma 2.9 to control each coordinate of the sum ∑ k = 1 t Z k \sum_{k=1}^{t} Z_k ∑ k = 1 t Z k . This requires examining the following two quantities appearing in the Freedman inequality:
max 1 ≤ k ≤ t ∥ Z k ∥ ∞ and W t : = ∑ k = 1 t Var k − 1 ( Z k ) , (191) \max_{1\le k\le t}\|Z_k\|_\infty\quad\text{and}\quad W_t := \sum_{k=1}^{t} \operatorname{Var}_{k-1}(Z_k), \tag*{(191)} 1 ≤ k ≤ t max ∥ Z k ∥ ∞ and W t := k = 1 ∑ t Var k − 1 ( Z k ) , (191)
where Var k − 1 ( Z k ) \operatorname{Var}_{k-1}(Z_k) Var k − 1 ( Z k ) represents an S S S -dimensional vector whose j j j -th entry is the variance of the j j j -th entry of Z k Z_k Z k conditioned on what happens up to iteration k − 1 k-1 k − 1 . Throughout this step, we assume t ≥ T / ( c 2 log T ) t \ge T/(c_2 \log T) t ≥ T / ( c 2 log T ) . Note that a crude bound on the variance proxy is sufficient for now; a sharper bound exploiting the variance structure of the TD iterates is developed in the next two steps.
Bounding the maximum magnitude. Recalling inequality (188) , one immediately sees that
max k : 1 ≤ k ≤ t ∥ Z k ∥ ∞ ≤ 4 γ log 3 T ( 1 − γ ) 2 T = : B . (192) \max_{k:1\le k\le t}\|Z_k\|_\infty\le\frac{4\gamma\log^3 T}{(1-\gamma)^2T} =: B. \tag*{(192)} k : 1 ≤ k ≤ t max ∥ Z k ∥ ∞ ≤ ( 1 − γ ) 2 T 4 γ log 3 T =: B . (192)
Bounding the conditional entrywise variance of Z k Z_k Z k . Next, we turn to W t W_t W t by first examining each conditional variance term Var k − 1 ( Z k ) \operatorname{Var}_{k-1}(Z_k) Var k − 1 ( Z k ) . As can be seen from the definition (185) of Z k Z_k Z k , each coordinate of Z k Z_k Z k is a bilinear form of P k − P π P_k-P_\pi P k − P π (recall the definition of P k P_k P k in (178) ). To analyze such a bilinear form, we first make note of a useful elementary fact: for any fixed vectors u , v ∈ R S u,v\in\mathbb{R}^S u , v ∈ R S obeying u ≥ 0 u\ge0 u ≥ 0 , one has
Var ( u ⊤ ( P k − P π ) v ) = ∑ i = 1 S u i 2 Var ( ( P k − P π ) i , ⋅ v ) ≤ max i ∣ u i ∣ ⋅ u ⊤ Var P π ( v ) ≤ ∥ u ∥ 1 u ⊤ Var P π ( v ) , (193) \operatorname{Var}\bigl(u^\top(P_k-P_\pi)v\bigr)
= \sum_{i=1}^{S}u_i^2\operatorname{Var}\bigl((P_k-P_\pi)_{i,\cdot}v\bigr)
\le\max_i |u_i|\cdot u^\top\operatorname{Var}_{P_\pi}(v)
\le\|u\|_1u^\top\operatorname{Var}_{P_\pi}(v), \tag*{(193)} Var ( u ⊤ ( P k − P π ) v ) = i = 1 ∑ S u i 2 Var ( ( P k − P π ) i , ⋅ v ) ≤ i max ∣ u i ∣ ⋅ u ⊤ Var P π ( v ) ≤ ∥ u ∥ 1 u ⊤ Var P π ( v ) , (193)
where we remind the reader of the notation Var P ( V ) \operatorname{Var}_{P}(V) Var P ( V ) in (32) . Applying this relation to each entry of Z k Z_k Z k defined in (185) gives
Var k − 1 ( Z k ) ≤ γ 2 ∥ η k ∏ i = k + 1 t ( I − η i ( I − γ P π ) ) ∥ 1 2 { η k ∏ i = k + 1 t ( I − η i ( I − γ P π ) ) } Var P π ( V k − 1 ) = γ 2 η k ( t ) η k ∏ i = k + 1 t ( I − η i ( I − γ P π ) ) Var P π ( V k − 1 ) , (194) \begin{aligned}
\operatorname{Var}_{k-1}(Z_k)
\le\gamma^2\left\|\eta_k\prod_{i=k+1}^{t}\left(I-\eta_i(I-\gamma P_\pi)\right)\right\|_1^2
\left\{\eta_k\prod_{i=k+1}^{t}\left(I-\eta_i(I-\gamma P_\pi)\right)\right\}
\operatorname{Var}_{P_\pi}(V_{k-1}) \\
&= \gamma^{2}\eta_{k}^{(t)}\eta_k \prod_{i=k+1}^{t}\left(I-\eta_i(I-\gamma P_\pi)\right)\operatorname{Var}_{P_\pi}(V_{k-1}), \tag*{(194)}
\end{aligned} Var k − 1 ( Z k ) ≤ γ 2 η k i = k + 1 ∏ t ( I − η i ( I − γ P π ) ) 1 2 { η k i = k + 1 ∏ t ( I − η i ( I − γ P π ) ) } Var P π ( V k − 1 ) = γ 2 η k ( t ) η k i = k + 1 ∏ t ( I − η i ( I − γ P π ) ) Var P π ( V k − 1 ) , (194)
where the first inequality is a consequence of (193) , and the second relation results from (187) and the definition of η k ( t ) \eta_k^{(t)} η k ( t ) (cf. (180) ). Although the upper bound in (194) appears somewhat complicated, it admits a more tractable upper bound once summed as in the next step.
To control W t W_t W t , we bound two partial sums of Var k − 1 ( Z k ) \operatorname{Var}_{k-1}(Z_k) Var k − 1 ( Z k ) separately, where the first is negligible and the second constitutes the dominant contribution.
∑ k = 1 ⌊ t / 2 ⌋ Var k − 1 ( Z k ) ≤ γ 2 ∑ k = 1 ⌊ t / 2 ⌋ η k ( t ) η k ∏ i = k + 1 t ( I − η i ( I − γ P π ) ) ∥ V k − 1 ∥ ∞ 2 1 S = γ 2 ∑ k = 1 ⌊ t / 2 ⌋ ( η k ( t ) ) 2 ∥ V k − 1 ∥ ∞ 2 1 S ≤ γ 2 2 ( 1 − γ ) 2 T 3 1 S , (195) \begin{aligned}
\sum_{k=1}^{\lfloor t/2\rfloor}\operatorname{Var}_{k-1}(Z_k)
\le\gamma^2\sum_{k=1}^{\lfloor t/2\rfloor}\eta_k^{(t)}\eta_k\prod_{i=k+1}^{t}\left(I-\eta_i(I-\gamma P_\pi)\right)\left\|V_{k-1}\right\|_\infty^2\mathbf{1}_S \\
&= \gamma^2\sum_{k=1}^{\lfloor t/2\rfloor}\left(\eta_k^{(t)}\right)^2\left\|V_{k-1}\right\|_\infty^2\mathbf{1}_S
\le\frac{\gamma^2}{2(1-\gamma)^2T^3}\mathbf{1}_S, \tag*{(195)}
\end{aligned} k = 1 ∑ ⌊ t /2 ⌋ Var k − 1 ( Z k ) ≤ γ 2 k = 1 ∑ ⌊ t /2 ⌋ η k ( t ) η k i = k + 1 ∏ t ( I − η i ( I − γ P π ) ) ∥ V k − 1 ∥ ∞ 2 1 S = γ 2 k = 1 ∑ ⌊ t /2 ⌋ ( η k ( t ) ) 2 ∥ V k − 1 ∥ ∞ 2 1 S ≤ 2 ( 1 − γ ) 2 T 3 γ 2 1 S , (195)
where the penultimate step uses (187) and (180) , and the last relation follows from Lemmas 4.2–4.3 and t ≤ T t \le T t ≤ T . As will be made clear shortly, this partial sum (195) becomes negligible compared to the remaining part.
Var k − 1 ( Z k ) ≤ 2 γ 2 log 3 T ( 1 − γ ) T η k ∏ i = k + 1 t ( I − η i ( I − γ P π ) ) Var P π ( V k − 1 ) . (196) \operatorname{Var}_{k-1}(Z_k)\le\frac{2\gamma^2\log^3 T}{(1-\gamma)T}\eta_k\prod_{i=k+1}^{t}\left(I-\eta_i(I-\gamma P_\pi)\right)\operatorname{Var}_{P_\pi}(V_{k-1}). \tag*{(196)} Var k − 1 ( Z k ) ≤ ( 1 − γ ) T 2 γ 2 log 3 T η k i = k + 1 ∏ t ( I − η i ( I − γ P π ) ) Var P π ( V k − 1 ) . (196)
To obtain a deterministic bound on W t W_t W t , Lemma 4.2 gives
Var P π ( V k − 1 ) ≤ 1 ( 1 − γ ) 2 1 S . \operatorname{Var}_{P_\pi}(V_{k-1})\le\frac{1}{(1-\gamma)^2}\mathbf{1}_S. Var P π ( V k − 1 ) ≤ ( 1 − γ ) 2 1 1 S .
In addition, one observes the following telescoping property for the matrix sum of interest:
∑ k = ⌊ t / 2 ⌋ + 1 t η k ∏ i = k + 1 t ( I − η i ( I − γ P π ) ) = ( I − γ P π ) − 1 ∑ k = ⌊ t / 2 ⌋ + 1 t [ ∏ i = k + 1 t ( I − η i ( I − γ P π ) ) − ∏ i = k t ( I − η i ( I − γ P π ) ) ] = ( I − γ P π ) − 1 − ( I − γ P π ) − 1 ∏ i = ⌊ t / 2 ⌋ + 1 t ( I − η i ( I − γ P π ) ) ≤ ( I − γ P π ) − 1 , (197) \begin{aligned}
\sum_{k=\lfloor t/2\rfloor+1}^{t}\eta_k\prod_{i=k+1}^{t}\left(I-\eta_i(I-\gamma P_\pi)\right) \\
&= (I-\gamma P_\pi)^{-1}\sum_{k=\lfloor t/2\rfloor+1}^{t}\left[\prod_{i=k+1}^{t}\left(I-\eta_i(I-\gamma P_\pi)\right)-\prod_{i=k}^{t}\left(I-\eta_i(I-\gamma P_\pi)\right)\right] \\
&= (I-\gamma P_\pi)^{-1}-(I-\gamma P_\pi)^{-1}\prod_{i=\lfloor t/2\rfloor+1}^{t}\left(I-\eta_i(I-\gamma P_\pi)\right) \\
&\le(I-\gamma P_\pi)^{-1}, \tag*{(197)}
\end{aligned} k = ⌊ t /2 ⌋ + 1 ∑ t η k i = k + 1 ∏ t ( I − η i ( I − γ P π ) ) = ( I − γ P π ) − 1 k = ⌊ t /2 ⌋ + 1 ∑ t [ i = k + 1 ∏ t ( I − η i ( I − γ P π ) ) − i = k ∏ t ( I − η i ( I − γ P π ) ) ] = ( I − γ P π ) − 1 − ( I − γ P π ) − 1 i = ⌊ t /2 ⌋ + 1 ∏ t ( I − η i ( I − γ P π ) ) ≤ ( I − γ P π ) − 1 , (197)
where the inequality in the last line holds entrywise since ( I − γ P π ) − 1 (I-\gamma P_\pi)^{-1} ( I − γ P π ) − 1 and I − η i ( I − γ P π ) I-\eta_i(I-\gamma P_\pi) I − η i ( I − γ P π ) are both nonnegative matrices. Applying the preceding two bounds to (196) and using ( I − γ P π ) − 1 1 S = 1 1 − γ 1 S (I-\gamma P_\pi)^{-1}\mathbf{1}_S=\frac{1}{1-\gamma}\mathbf{1}_S ( I − γ P π ) − 1 1 S = 1 − γ 1 1 S , we reach
∑ k = ⌊ t / 2 ⌋ + 1 t Var k − 1 ( Z k ) ≤ 2 γ 2 log 3 T ( 1 − γ ) 3 T ( I − γ P π ) − 1 1 S = 2 γ 2 log 3 T ( 1 − γ ) 4 T 1 S . (198) \sum_{k=\lfloor t/2\rfloor+1}^{t}\operatorname{Var}_{k-1}(Z_k)
\le\frac{2\gamma^2\log^3 T}{(1-\gamma)^3T}(I-\gamma P_\pi)^{-1}\mathbf{1}_S
= \frac{2\gamma^2\log^3 T}{(1-\gamma)^4T}\mathbf{1}_S. \tag*{(198)} k = ⌊ t /2 ⌋ + 1 ∑ t Var k − 1 ( Z k ) ≤ ( 1 − γ ) 3 T 2 γ 2 log 3 T ( I − γ P π ) − 1 1 S = ( 1 − γ ) 4 T 2 γ 2 log 3 T 1 S . (198)
Combining (195) and (198) , and using γ 2 ≤ 1 \gamma^{2} \le1 γ 2 ≤ 1 , we obtain the deterministic bound
∥ W t ∥ ∞ ≤ 1 2 ( 1 − γ ) 2 T 3 + 2 log 3 T ( 1 − γ ) 4 T ≤ 3 log 3 T ( 1 − γ ) 4 T = : σ w 2 . (199) \lVert W_t\rVert_{\infty} \le\frac{1}{2(1-\gamma)^{2}T^{3}}+\frac{2\log^{3}T}{(1-\gamma)^{4}T} \le\frac{3\log^{3}T}{(1-\gamma)^{4}T} =: \sigma_{w}^{2}. \tag*{(199)} ∥ W t ∥ ∞ ≤ 2 ( 1 − γ ) 2 T 3 1 + ( 1 − γ ) 4 T 2 log 3 T ≤ ( 1 − γ ) 4 T 3 log 3 T =: σ w 2 . (199)
Equipped with the deterministic bounds (192) and (199) , we can first apply Freedman’s inequality while retaining the random cumulative variance W t W_t W t . To this end, take K = 2 ⌈ log 2 1 1 − γ ⌉ K = 2\left\lceil\log_{2}\frac{1}{1-\gamma}\right\rceil K = 2 ⌈ log 2 1 − γ 1 ⌉ . Since (177) and ϵ ≤ 1 / ( 1 − γ ) \epsilon\le1/(1-\gamma) ϵ ≤ 1/ ( 1 − γ ) ensure T ≥ max { 8 , 1 / ( 1 − γ ) } T \ge\max\{8,1/(1-\gamma)\} T ≥ max { 8 , 1/ ( 1 − γ )} for sufficiently large c 3 c_{3} c 3 , we have
σ w 2 2 K ≤ 3 log 3 T ( 1 − γ ) 2 T , log 4 S T K δ ≤ 3 log S T δ . (200) \frac{\sigma_{w}^{2}}{2^{K}} \le\frac{3\log^{3}T}{(1-\gamma)^{2}T}, \qquad\log\frac{4STK}{\delta} \le3\log\frac{ST}{\delta}. \tag*{(200)} 2 K σ w 2 ≤ ( 1 − γ ) 2 T 3 log 3 T , log δ 4 S T K ≤ 3 log δ S T . (200)
For a fixed t t t , applying Lemma 2.9 to each entry of ∑ k = 1 t Z k \sum_{k=1}^{t} Z_k ∑ k = 1 t Z k and taking a union bound over all S S S entries give, with probability at least 1 − δ / ( 2 T ) 1-\delta/(2T) 1 − δ / ( 2 T ) ,
∥ ∑ k = 1 t Z k ∥ ∞ ≤ 8 max { ∥ W t ∥ ∞ , σ w 2 2 K } log 4 S T K δ + 4 3 B log 4 S T K δ , (201) \left\lVert\sum_{k=1}^{t}Z_k\right\rVert_{\infty} \le\sqrt{8\max\left\{\lVert W_t\rVert_{\infty},\frac{\sigma_{w}^{2}}{2^{K}}\right\}\log\frac{4STK}{\delta}}+\frac{4}{3}B\log\frac{4STK}{\delta}, \tag*{(201)} k = 1 ∑ t Z k ∞ ≤ 8 max { ∥ W t ∥ ∞ , 2 K σ w 2 } log δ 4 S T K + 3 4 B log δ 4 S T K , (201)
with B B B and σ w 2 \sigma_{w}^{2} σ w 2 defined in (192) and (199) , respectively. It then boils down to deriving a sharper bound on ∥ W t ∥ ∞ \lVert W_t\rVert_{\infty} ∥ W t ∥ ∞ .
Step 3: exploiting the variance structure of W t W_t W t . To further sharpen the bound on W t W_t W t , the key idea is to relate the conditional variance of each TD update to the decrease in the squared iterates, which, as we shall see shortly, allows the weighted sum of conditional variances to be decomposed into a telescoping term, a martingale term, and a lower-order residual term. To begin, since k > t / 2 k>t/2 k > t /2 , Lemma 4.3 gives
η k ( t ) ≤ η k ≤ 2 log 3 T ( 1 − γ ) T = : η ˉ , t / 2 < k ≤ t . (202) \eta_{k}^{(t)} \le\eta_k \le\frac{2\log^{3}T}{(1-\gamma)T} =: \bar{\eta}, \qquad t/2<k\le t. \tag*{(202)} η k ( t ) ≤ η k ≤ ( 1 − γ ) T 2 log 3 T =: η ˉ , t /2 < k ≤ t . (202)
Combining this bound with (194) yields
∑ k = ⌊ t / 2 ⌋ + 1 t Var k − 1 ( Z k ) ≤ η ˉ γ 2 ∑ k = ⌊ t / 2 ⌋ + 1 t ∏ i = k + 1 t ( I − η i ( I − γ P π ) ) Var P π ( V k − 1 ) = : W ‾ t . (203) \sum_{k=\lfloor t/2\rfloor+1}^{t}\operatorname{Var}_{k-1}(Z_k) \le\bar{\eta}\gamma^{2}\sum_{k=\lfloor t/2\rfloor+1}^{t}\prod_{i=k+1}^{t}\left(I-\eta_i(I-\gamma P_{\pi})\right)\operatorname{Var}_{P_{\pi}}(V_{k-1}) =: \overline{W}_t. \tag*{(203)} k = ⌊ t /2 ⌋ + 1 ∑ t Var k − 1 ( Z k ) ≤ η ˉ γ 2 k = ⌊ t /2 ⌋ + 1 ∑ t i = k + 1 ∏ t ( I − η i ( I − γ P π ) ) Var P π ( V k − 1 ) =: W t . (203)
It then comes down to controlling W ‾ t \overline{W}_t W t . Let E k − 1 \mathbb{E}_{k-1} E k − 1 denote conditional expectation given the history through iteration k − 1 k-1 k − 1 . We first establish below a one-step relation between the conditional variance and the evolution of the squared TD iterates. This relation is a key ingredient in our refined analysis, which differs from the proof in Li et al. (2024a). Its proof is deferred to Section 4.6.3.
Lemma 4.4. For every k ≥ 1 k \ge1 k ≥ 1 ,
η k ( 1 − η k ) γ 2 Var P π ( V k − 1 ) ≤ ( I − η k ( I − γ P π ) ) ( V k − 1 ∘ V k − 1 ) − E k − 1 [ V k ∘ V k ] + 2 η k 1 − γ 1 S . (204) \eta_k(1-\eta_k)\gamma^{2}\operatorname{Var}_{P_{\pi}}(V_{k-1}) \le\left(I-\eta_k(I-\gamma P_{\pi})\right)(V_{k-1}\circ V_{k-1})-\mathbb{E}_{k-1}[V_k\circ V_k]+\frac{2\eta_k}{1-\gamma}\mathbf{1}_S. \tag*{(204)} η k ( 1 − η k ) γ 2 Var P π ( V k − 1 ) ≤ ( I − η k ( I − γ P π ) ) ( V k − 1 ∘ V k − 1 ) − E k − 1 [ V k ∘ V k ] + 1 − γ 2 η k 1 S . (204)
We now return to W ‾ t \overline{W}_t W t in (203) . The sample-size condition (177) and ϵ ≤ 1 / ( 1 − γ ) \epsilon\le1/(1-\gamma) ϵ ≤ 1/ ( 1 − γ ) ensure η ˉ ≤ 1 / 2 \bar{\eta}\le1/2 η ˉ ≤ 1/2 for sufficiently large c 3 c_3 c 3 . Applying Lemma 4.4 to the weighted sum in (203) , and adding and subtracting V k ∘ V k V_k\circ V_k V k ∘ V k inside the weighted sum, we obtain
W ‾ t ≤ η ˉ 1 − η ˉ { ∑ k = ⌊ t / 2 ⌋ + 1 t ∏ i = k + 1 t ( I − η i ( I − γ P π ) ) [ ( I − η k ( I − γ P π ) ) ( V k − 1 ∘ V k − 1 ) − V k ∘ V k ] + ∑ k = ⌊ t / 2 ⌋ + 1 t ∏ i = k + 1 t ( I − η i ( I − γ P π ) ) ( V k ∘ V k − E k − 1 [ V k ∘ V k ] ) + 2 1 − γ ∑ k = ⌊ t / 2 ⌋ + 1 t η k ∏ i = k + 1 t ( I − η i ( I − γ P π ) ) 1 S } . (205) \begin{aligned}
\overline{W}_t \le\frac{\bar{\eta}}{1-\bar{\eta}}\{\sum_{k=\lfloor t/2\rfloor+1}^{t}\prod_{i=k+1}^{t}(I-\eta_i(I-\gamma P_{\pi}))[(I-\eta_k(I-\gamma P_{\pi}))(V_{k-1}\circ V_{k-1})-V_k\circ V_k] \\
&+ \sum_{k=\lfloor t/2\rfloor+1}^{t}\prod_{i=k+1}^{t}(I-\eta_i(I-\gamma P_\pi))(V_k\circ V_k-\mathbb{E}_{k-1}[V_k\circ V_k])
+ \frac{2}{1-\gamma}\sum_{k=\lfloor t/2\rfloor+1}^{t}\eta_k\prod_{i=k+1}^{t}(I-\eta_i(I-\gamma P_\pi))\mathbf{1}_S
\}. \tag*{(205)}
\end{aligned} W t ≤ 1 − η ˉ η ˉ { k = ⌊ t /2 ⌋ + 1 ∑ t i = k + 1 ∏ t ( I − η i ( I − γ P π )) [( I − η k ( I − γ P π )) ( V k − 1 ∘ V k − 1 ) − V k ∘ V k ] + k = ⌊ t /2 ⌋ + 1 ∑ t i = k + 1 ∏ t ( I − η i ( I − γ P π )) ( V k ∘ V k − E k − 1 [ V k ∘ V k ]) + 1 − γ 2 k = ⌊ t /2 ⌋ + 1 ∑ t η k i = k + 1 ∏ t ( I − η i ( I − γ P π )) 1 S } . (205)
Note that the second sum in (205) consists of centered fluctuations. To express it as a martingale sum, define
Z ~ k : = ∏ i = k + 1 t ( I − η i ( I − γ P π ) ) ( V k ∘ V k − E k − 1 [ V k ∘ V k ] ) , k > t / 2. (206) \widetilde{Z}_k:=\prod_{i=k+1}^{t}\left(I-\eta_i(I-\gamma P_\pi)\right)\left(V_k\circ V_k-\mathbb{E}_{k-1}[V_k\circ V_k]\right),\qquad k>t/2. \tag*{(206)} Z k := i = k + 1 ∏ t ( I − η i ( I − γ P π ) ) ( V k ∘ V k − E k − 1 [ V k ∘ V k ] ) , k > t /2. (206)
The matrix weights are deterministic, so E k − 1 [ Z ~ k ] = 0 \mathbb{E}_{k-1}[\widetilde{Z}_k]=0 E k − 1 [ Z k ] = 0 and { Z ~ k } \{\widetilde{Z}_k\} { Z k } forms a martingale difference sequence. The first sum in (205) telescopes, resulting in
∑ k = ⌊ t / 2 ⌋ + 1 t ∏ i = k + 1 t ( I − η i ( I − γ P π ) ) { ( I − η k ( I − γ P π ) ) ( V k − 1 ∘ V k − 1 ) − V k ∘ V k } = ∏ i = ⌊ t / 2 ⌋ + 1 t ( I − η i ( I − γ P π ) ) ( V ⌊ t / 2 ⌋ ∘ V ⌊ t / 2 ⌋ ) − V t ∘ V t ≤ 1 ( 1 − γ ) 2 1 S , \begin{aligned}
\sum_{k=\lfloor t/2\rfloor+1}^{t}\prod_{i=k+1}^{t}\left(I-\eta_i(I-\gamma P_\pi)\right)\left\{\left(I-\eta_k(I-\gamma P_\pi)\right)(V_{k-1}\circ V_{k-1})-V_k\circ V_k\right\} \\
&=\prod_{i=\lfloor t/2\rfloor+1}^{t}\left(I-\eta_i(I-\gamma P_\pi)\right)(V_{\lfloor t/2\rfloor}\circ V_{\lfloor t/2\rfloor})-V_t\circ V_t \\
&\leq\frac{1}{(1-\gamma)^2}\mathbf{1}_S,
\end{aligned} k = ⌊ t /2 ⌋ + 1 ∑ t i = k + 1 ∏ t ( I − η i ( I − γ P π ) ) { ( I − η k ( I − γ P π ) ) ( V k − 1 ∘ V k − 1 ) − V k ∘ V k } = i = ⌊ t /2 ⌋ + 1 ∏ t ( I − η i ( I − γ P π ) ) ( V ⌊ t /2 ⌋ ∘ V ⌊ t /2 ⌋ ) − V t ∘ V t ≤ ( 1 − γ ) 2 1 1 S ,
where the last inequality uses Lemma 4.2 and the row-sum identity (187) . The third sum in (205) is at most 2 ( 1 − γ ) 2 1 S \frac{2}{(1-\gamma)^2}\mathbf{1}_S ( 1 − γ ) 2 2 1 S , due to the matrix telescoping calculation in (197) and ( I − γ P π ) − 1 1 S = 1 1 − γ 1 S (I-\gamma P_\pi)^{-1}\mathbf{1}_S=\frac{1}{1-\gamma}\mathbf{1}_S ( I − γ P π ) − 1 1 S = 1 − γ 1 1 S . Combining the above bounds with (206) , we obtain
W ‾ t ≤ η ‾ 1 − η ‾ ( 3 ( 1 − γ ) 2 1 S + ∑ k = ⌊ t / 2 ⌋ + 1 t Z ~ k ) . (207) \overline{W}_t\leq\frac{\overline{\eta}}{1-\overline{\eta}}\left(\frac{3}{(1-\gamma)^2}\mathbf{1}_S+\sum_{k=\lfloor t/2\rfloor+1}^{t}\widetilde{Z}_k\right). \tag*{(207)} W t ≤ 1 − η η ( 1 − γ ) 2 3 1 S + k = ⌊ t /2 ⌋ + 1 ∑ t Z k . (207)
Thus, controlling W ‾ t \overline{W}_t W t reduces to controlling the martingale sum ∑ k Z ~ k \sum_k\widetilde{Z}_k ∑ k Z k .
Step 4: applying martingale concentration to bound ∥ W t ∥ ∞ \|W_t\|_\infty ∥ W t ∥ ∞ . The centered term in (207) from the preceding step is itself a martingale sum. Applying Freedman’s inequality to this second martingale yields a self-bounding inequality for the variance proxy W t W_t W t , which ultimately provides the sharp bound on W t W_t W t needed in the original Freedman inequality. More precisely, applying Freedman’s inequality to the martingale sequence { Z ~ k } \{\widetilde{Z}_k\} { Z k } defined in (206) gives the following result.
Lemma 4.5. Let W ‾ t \overline{W}_t W t be defined in (203) , and η ‾ \overline{\eta} η be defined in (202) . Then, with probability at least 1 − δ / ( 2 T ) 1-\delta/(2T) 1 − δ / ( 2 T ) ,
∥ W ‾ t ∥ ∞ ≤ 16 η ‾ ( 1 − γ ) 2 , ∥ W t ∥ ∞ ≤ 33 log 3 T ( 1 − γ ) 3 T . (208) \|\overline{W}_t\|_\infty\leq\frac{16\overline{\eta}}{(1-\gamma)^2},\qquad\|W_t\|_\infty\leq\frac{33\log^3 T}{(1-\gamma)^3T}. \tag*{(208)} ∥ W t ∥ ∞ ≤ ( 1 − γ ) 2 16 η , ∥ W t ∥ ∞ ≤ ( 1 − γ ) 3 T 33 log 3 T . (208)
The proof of Lemma 4.5 is deferred to § 7.16.2 . It is noteworthy that the derivation of this lemma is a key point of departure from the analysis of Li et al. (2024a), enabling us to extend the guarantee to the full ε \varepsilon ε -range. Its main ingredient is a second martingale decomposition of the squared TD iterates, which exploits the variance structure of the MDP and yields the sharp dependence on the effective horizon.
Step 5: putting all pieces together. Thus far, we have shown that, for each fixed t ≥ T / ( c 2 log T ) t \ge T/(c_2 \log T) t ≥ T / ( c 2 log T ) , (201) and (208) hold simultaneously with probability at least 1 − δ / T 1-\delta/T 1 − δ / T . On this event, substituting the variance bound (208) into (201) , together with (192) and (200) , we see that
∥ ∑ k = 1 t Z k ∥ ∞ ≤ C 2 ( log 3 T ) ( log S T δ ) ( 1 − γ ) 3 T + C ( log 3 T ) ( log S T δ ) 2 ( 1 − γ ) 2 T ≤ C ( log 3 T ) ( log S T δ ) ( 1 − γ ) 3 T (209) \left\| \sum_{k=1}^{t} Z_k \right\|_{\infty} \le\frac{C}{2}\sqrt{\frac{(\log^{3} T)\left(\log\frac{ST}{\delta}\right)}{(1-\gamma)^{3}T}}+\frac{C(\log^{3} T)\left(\log\frac{ST}{\delta}\right)}{2(1-\gamma)^{2}T}\le C\sqrt{\frac{(\log^{3} T)\left(\log\frac{ST}{\delta}\right)}{(1-\gamma)^{3}T}} \tag*{(209)} k = 1 ∑ t Z k ∞ ≤ 2 C ( 1 − γ ) 3 T ( log 3 T ) ( log δ S T ) + 2 ( 1 − γ ) 2 T C ( log 3 T ) ( log δ S T ) ≤ C ( 1 − γ ) 3 T ( log 3 T ) ( log δ S T ) (209)
for some large enough universal constant C > 0 C>0 C > 0 . Plugging (209) into the error decomposition (190) and taking a union bound over all t t t obeying T / ( c 2 log T ) ≤ t ≤ T T/(c_2\log T)\le t\le T T / ( c 2 log T ) ≤ t ≤ T , we obtain
∥ Δ t ∥ ∞ ≤ 1 ( 1 − γ ) T 2 + C ( log 3 T ) ( log S T δ ) ( 1 − γ ) 3 T ≲ ( log 3 T ) ( log S T δ ) ( 1 − γ ) 3 T (210) \|\Delta_t\|_{\infty}\le\frac{1}{(1-\gamma)T^{2}}+C\sqrt{\frac{(\log^{3} T)\left(\log\frac{ST}{\delta}\right)}{(1-\gamma)^{3}T}}\lesssim\sqrt{\frac{(\log^{3} T)\left(\log\frac{ST}{\delta}\right)}{(1-\gamma)^{3}T}} \tag*{(210)} ∥ Δ t ∥ ∞ ≤ ( 1 − γ ) T 2 1 + C ( 1 − γ ) 3 T ( log 3 T ) ( log δ S T ) ≲ ( 1 − γ ) 3 T ( log 3 T ) ( log δ S T ) (210)
with probability at least 1 − δ 1-\delta 1 − δ . This concludes the proof.
Q-learning
Having established the appealing performance of synchronous TD learning for policy evaluation, we now turn our attention to the use of model-free algorithms for learning optimal Q-functions. In particular, we focus on the Q-learning algorithm introduced by Watkins (1989) and Watkins and Dayan (1992), one of the most celebrated model-free reinforcement learning methods.
Algorithm
Akin to TD learning, the Q-learning algorithm is designed as a stochastic approximation method for computing a fixed point of the Bellman operator. A key distinction is that the operator of interest is now the Bellman optimality operator (21) , rather than the Bellman consistency operator (17) . Unlike the latter, the Bellman optimality operator is nonlinear and nonsmooth (owing to the presence of the max operation), which introduces additional algorithmic and analytical challenges.
We now describe the algorithmic procedure of Q-learning. Following the stochastic approximation framework (171) , we maintain a running estimate of the optimal Q-function, Q t : S × A → R Q_t:\mathcal{S}\times\mathcal{A}\to\mathbb{R} Q t : S × A → R , for all t ≥ 0 t\ge0 t ≥ 0 , and at each iteration t t t , we update the Q-function estimate according to
Q t = ( 1 − η t ) Q t − 1 + η t T t ( Q t − 1 ) , (211) Q_t=(1-\eta_t)Q_{t-1}+\eta_t\mathcal{T}_t(Q_{t-1}), \tag*{(211)} Q t = ( 1 − η t ) Q t − 1 + η t T t ( Q t − 1 ) , (211)
where we initialize it such that 0 ≤ Q 0 ( s , a ) ≤ 1 1 − γ 0\le Q_0(s,a)\le\frac{1}{1-\gamma} 0 ≤ Q 0 ( s , a ) ≤ 1 − γ 1 for every state-action pair ( s , a ) (s,a) ( s , a ) . Here, η t ∈ ( 0 , 1 ] \eta_t\in(0,1] η t ∈ ( 0 , 1 ] denotes the learning rate at iteration t t t , and T t ( Q t − 1 ) \mathcal{T}_t(Q_{t-1}) T t ( Q t − 1 ) is constructed as an unbiased estimator of T ( Q t − 1 ) \mathcal{T}(Q_{t-1}) T ( Q t − 1 ) , where T ( ⋅ ) \mathcal{T}(\cdot) T ( ⋅ ) denotes the Bellman optimality operator (21) . With the generative model in place, T t ( Q t − 1 ) \mathcal{T}_t(Q_{t-1}) T t ( Q t − 1 ) can be constructed as follows:
for every pair ( s , a ) ∈ S × A (s,a)\in\mathcal{S}\times\mathcal{A} ( s , a ) ∈ S × A , draw an independent sample
s t ( s , a ) ∼ P ( ⋅ ∣ s , a ) s_t(s,a)\sim P(\cdot\mid s,a) s t ( s , a ) ∼ P ( ⋅ ∣ s , a )
by querying the generative model;
T t ( Q t − 1 ) ( s , a ) = r ( s , a ) + γ max a ′ ∈ A Q t − 1 ( s t ( s , a ) , a ′ ) . (212) \mathcal{T}_t(Q_{t-1})(s,a)=r(s,a)+\gamma\max_{a'\in\mathcal{A}}Q_{t-1}(s_t(s,a),a'). \tag*{(212)} T t ( Q t − 1 ) ( s , a ) = r ( s , a ) + γ a ′ ∈ A max Q t − 1 ( s t ( s , a ) , a ′ ) . (212)
Algorithm 4.2: Q-learning with a generative model
input: discount factor γ \gamma γ , reward function r r r , learning rates { η t } \{\eta_t\} { η t } , number of iterations T T T , initialization Q 0 Q_0 Q 0 .
for t = 1 , 2 , ⋯ , T t = 1,2,\cdots,T t = 1 , 2 , ⋯ , T do
draw a sample s t ( s , a ) ∼ i n d . P ( ⋅ ∣ s , a ) s_t(s,a) \mathrel{\overset{\mathrm{ind.}}{\sim}} P(\cdot\mid s,a) s t ( s , a ) ∼ ind. P ( ⋅ ∣ s , a ) for each ( s , a ) ∈ S × A (s,a) \in\mathcal{S} \times\mathcal{A} ( s , a ) ∈ S × A .
/* stochastic approximation. */
update Q t Q_t Q t according to
Q t = ( 1 − η t ) Q t − 1 + η t T t ( Q t − 1 ) , (213) Q_t = (1-\eta_t)Q_{t-1}+\eta_t\mathcal{T}_t(Q_{t-1}), \tag*{(213)} Q t = ( 1 − η t ) Q t − 1 + η t T t ( Q t − 1 ) , (213)
where for every ( s , a ) (s,a) ( s , a ) pair, the ( s , a ) (s,a) ( s , a ) -th entry of T t ( Q t − 1 ) \mathcal{T}_t(Q_{t-1}) T t ( Q t − 1 ) is
T t ( Q t − 1 ) ( s , a ) = r ( s , a ) + γ max a ′ ∈ A Q t − 1 ( s t ( s , a ) , a ′ ) . (214) \mathcal{T}_t(Q_{t-1})(s,a)=r(s,a)+\gamma\max_{a'\in\mathcal{A}}Q_{t-1}(s_t(s,a),a'). \tag*{(214)} T t ( Q t − 1 ) ( s , a ) = r ( s , a ) + γ a ′ ∈ A max Q t − 1 ( s t ( s , a ) , a ′ ) . (214)
This version of Q-learning is commonly referred to as synchronous Q-learning, since all entries of the Q-function are updated simultaneously at each iteration. In addition, the corresponding estimate for the optimal value function, V t : S → R V_t:\mathcal{S}\to\mathbb{R} V t : S → R , can be computed as
∀ s ∈ S : V t ( s ) : = max a ∈ A Q t ( s , a ) (215) \forall s\in\mathcal{S}: \quad V_t(s):=\max_{a\in\mathcal{A}}Q_t(s,a) \tag*{(215)} ∀ s ∈ S : V t ( s ) := a ∈ A max Q t ( s , a ) (215)
at iteration t t t . The complete description of the Q-learning procedure is summarized in Algorithm 4.2.
Theoretical guarantees
Having established the minimax optimality of TD learning over the accuracy range covered by the minimax lower bound, which can be viewed as a special case of Q-learning with number of actions A = 1 A=1 A = 1 , it is natural to conjecture that Q-learning should likewise attain near-optimal performance. However, this intuition breaks down when A ≥ 2 A\ge2 A ≥ 2 , as the multi-action setting fundamentally alters the situation. We make this precise via the following ℓ ∞ \ell_\infty ℓ ∞ -based sample complexity bound for Q-learning.
To state the result, we need to specify the learning rate schedule. In particular, we choose deterministic learning rates satisfying
1 1 + c 1 ( 1 − γ ) T log 3 T ≤ η t ≤ 1 1 + c 2 ( 1 − γ ) t log 3 T , 0 ≤ t ≤ T (216) \frac{1}{1+\frac{c_1(1-\gamma)T}{\log^3 T}}\le\eta_t\le\frac{1}{1+\frac{c_2(1-\gamma)t}{\log^3 T}},\qquad0\le t\le T \tag*{(216)} 1 + l o g 3 T c 1 ( 1 − γ ) T 1 ≤ η t ≤ 1 + l o g 3 T c 2 ( 1 − γ ) t 1 , 0 ≤ t ≤ T (216)
for some universal constants c 1 ≥ c 2 > 0 c_1\ge c_2>0 c 1 ≥ c 2 > 0 . Similar to the discussions in § 7.3 , such choices encompass both constant and rescaled linear learning rates (when appropriately chosen). The theorem below presents the state-of-the-art sample complexity analysis for Q-learning, originally established by Li et al. (2024a). In fact, it further extends the results of Li et al. (2024a) by broadening the admissible range of ε \varepsilon ε from ( 0 , 1 ] (0,1] ( 0 , 1 ] to the full range through a refined — and, indeed, simpler — analysis.
Theorem 4.6 (Extension of Li et al. , 2024a). Consider any δ ∈ ( 0 , 1 ) \delta\in(0,1) δ ∈ ( 0 , 1 ) , ε ∈ ( 0 , 1 1 − γ ] \varepsilon\in\left(0,\frac{1}{1-\gamma}\right] ε ∈ ( 0 , 1 − γ 1 ] , γ ∈ [ 1 / 2 , 1 ) \gamma\in[1/2,1) γ ∈ [ 1/2 , 1 ) , and A ≥ 2 A\ge2 A ≥ 2 . Suppose that the deterministic learning rates satisfy (216) for some small enough universal constants c 1 ≥ c 2 > 0 c_1\ge c_2>0 c 1 ≥ c 2 > 0 , and that the total number of iterations T ≥ 2 T\ge2 T ≥ 2 satisfies
T ≥ c 3 ( log 4 T ) ( log S A T δ ) ( 1 − γ ) 4 ε 2 (217) T\ge\frac{c_3(\log^4 T)\left(\log\frac{SAT}{\delta}\right)}{(1-\gamma)^4\varepsilon^2} \tag*{(217)} T ≥ ( 1 − γ ) 4 ε 2 c 3 ( log 4 T ) ( log δ S A T ) (217)
for some large enough universal constant c 3 > 0 c_3 > 0 c 3 > 0 . If the initialization obeys 0 ≤ Q 0 ( s , a ) ≤ 1 1 − γ 0 \le Q_0(s,a) \le\frac{1}{1-\gamma} 0 ≤ Q 0 ( s , a ) ≤ 1 − γ 1 for every ( s , a ) ∈ S × A (s,a) \in\mathcal{S} \times\mathcal{A} ( s , a ) ∈ S × A , then with probability at least 1 − δ 1-\delta 1 − δ , Q-learning (cf. Algorithm 4.2) achieves
∀ ( s , a ) ∈ S × A : ∣ Q T ( s , a ) − Q ∗ ( s , a ) ∣ ≤ ε . \forall(s,a) \in\mathcal{S} \times\mathcal{A}:\qquad\left|Q_T(s,a)-Q^*(s,a)\right| \le\varepsilon. ∀ ( s , a ) ∈ S × A : ∣ Q T ( s , a ) − Q ∗ ( s , a ) ∣ ≤ ε .
Remark 4.2 (Mean estimation guarantee). The high-probability guarantee in Theorem 4.6 can be readily converted into a bound on the mean estimation error. In light of the crude upper bound ∣ Q T ( s , a ) − Q ∗ ( s , a ) ∣ ≤ 1 1 − γ \left|Q_T(s,a)-Q^*(s,a)\right| \le\frac{1}{1-\gamma} ∣ Q T ( s , a ) − Q ∗ ( s , a ) ∣ ≤ 1 − γ 1 , one can take δ = ε ( 1 − γ ) / 2 \delta=\varepsilon(1-\gamma)/2 δ = ε ( 1 − γ ) /2 to obtain
E [ max ( s , a ) ∈ S × A ∣ Q T ( s , a ) − Q ∗ ( s , a ) ∣ ] ≤ ε ( 1 − δ ) + 1 1 − γ δ ≤ 2 ε , \mathbb{E}\left[\max_{(s,a)\in\mathcal{S}\times\mathcal{A}}\left|Q_T(s,a)-Q^*(s,a)\right|\right]\le\varepsilon(1-\delta)+\frac{1}{1-\gamma}\delta\le2\varepsilon, E [ ( s , a ) ∈ S × A max ∣ Q T ( s , a ) − Q ∗ ( s , a ) ∣ ] ≤ ε ( 1 − δ ) + 1 − γ 1 δ ≤ 2 ε ,
provided that the total number of iterations obeys
T ≥ c 3 ( log 4 T ) ( log 2 S A T ε ( 1 − γ ) ) ( 1 − γ ) 4 ε 2 . T\ge\frac{c_3(\log^4 T)\left(\log\frac{2SAT}{\varepsilon(1-\gamma)}\right)}{(1-\gamma)^4\varepsilon^2}. T ≥ ( 1 − γ ) 4 ε 2 c 3 ( log 4 T ) ( log ε ( 1 − γ ) 2 S A T ) .
In essence, Theorem 4.6 develops a non-asymptotic upper bound on the iteration complexity of Q-learning in the presence of the generative model. Recognizing that S A SA S A independent samples are drawn in each iteration — yielding a total of S A T SAT S A T samples overall — we can readily translate this result into the following sample complexity bound
O ~ ( S A ( 1 − γ ) 4 ε 2 ) , (218) \widetilde{O}\left(\frac{SA}{(1-\gamma)^4\varepsilon^2}\right), \tag*{(218)} O ( ( 1 − γ ) 4 ε 2 S A ) , (218)
which suffices for Q-learning to achieve ε \varepsilon ε -accuracy (for 0 < ε ≤ 1 / ( 1 − γ ) 0 < \varepsilon\le1/(1-\gamma) 0 < ε ≤ 1/ ( 1 − γ ) ) in the entrywise sense.
We provide further discussion about the learning rate choices. When T ≳ ( log 4 T ) ( log ( S A T / δ ) ) ( 1 − γ ) 4 ε 2 T \gtrsim\frac{(\log^4 T)(\log(SAT/\delta))}{(1-\gamma)^4\varepsilon^2} T ≳ ( 1 − γ ) 4 ε 2 ( l o g 4 T ) ( l o g ( S A T / δ )) , Condition (216) permits adopting the constant learning rates
η t ≡ 1 1 + c 1 ( 1 − γ ) T log 3 T = O ~ ( ( 1 − γ ) 3 ε 2 ) , 0 ≤ t ≤ T , \eta_t \equiv\frac{1}{1+\frac{c_1(1-\gamma)T}{\log^3 T}}=\widetilde{O}\left((1-\gamma)^3\varepsilon^2\right),\qquad0\le t\le T, η t ≡ 1 + l o g 3 T c 1 ( 1 − γ ) T 1 = O ( ( 1 − γ ) 3 ε 2 ) , 0 ≤ t ≤ T ,
depending almost solely on the discount factor γ \gamma γ and the target accuracy ε \varepsilon ε . Interestingly, both constant and rescaled linear learning rates satisfying (216) lead to the same sample complexity bound (in an order-wise sense), offering flexibility and robustness for practical use.
Nevertheless, the sample complexity bound in (218) , over the accuracy range covered by Theorem 3.7 , fails to match the minimax lower bound stated there. More specifically, the sample complexity bound of Q-learning incurs an additional O ~ ( 1 1 − γ ) \widetilde{O}\left(\frac{1}{1-\gamma}\right) O ( 1 − γ 1 ) multiplicative factor compared to what is achievable by its model-based counterpart (cf. Theorem 3.2 (Li et al., 2024c)). In sharp contrast to the A = 1 A=1 A = 1 regime — where TD learning attains near-optimal performance — the presence of multiple actions (A ≥ 2 A\ge2 A ≥ 2 ) appears to degrade the sample optimality of Q-learning.
Tightness and suboptimality
The gap between the upper bound in Theorem 4.6 and the minimax lower bound in Theorem 3.7 naturally raises the following questions: can the sample complexity upper bound for Q-learning be further improved, or is there an inherent bottleneck that prevents plain Q-learning from achieving optimal data efficiency? As it turns out, the horizon dependence in Theorem 4.6 is sharp in the regime of the lower bound below, which shows that Q-learning is provably suboptimal when there are at least two actions. This is confirmed by the lower bound below, originally established by Li et al. (2024a).
Theorem 4.7 (Li et al., 2024a). Assume that 3 / 4 ≤ γ < 1 3/4 \le\gamma< 1 3/4 ≤ γ < 1 and T ≥ c 3 ( 1 − γ ) 2 T \ge\frac{c_3}{(1-\gamma)^2} T ≥ ( 1 − γ ) 2 c 3 for some sufficiently large constant c 3 > 0 c_3 > 0 c 3 > 0 . Suppose that the initialization is Q 0 ≡ 0 Q_0 \equiv0 Q 0 ≡ 0 , and that the learning rates are taken to be either (i) η t = 1 1 + c η ( 1 − γ ) t \eta_t = \frac{1}{1+c_\eta(1-\gamma)t} η t = 1 + c η ( 1 − γ ) t 1 for any constant c η > 0 c_\eta> 0 c η > 0 for all t ≥ 0 t \ge0 t ≥ 0 , or (ii) η t ≡ η ∈ ( 0 , 1 ) \eta_t \equiv\eta\in(0,1) η t ≡ η ∈ ( 0 , 1 ) for all t ≥ 0 t \ge0 t ≥ 0 . There exists a γ \gamma γ -discounted MDP with S = 4 S = 4 S = 4 and A = 2 A = 2 A = 2 such that Q-learning (cf. Algorithm 4.2) obeys
max s ∈ S E [ ∣ V T ( s ) − V ⋆ ( s ) ∣ 2 ] ≥ c l b ( 1 − γ ) 4 T log 2 T (219) \max_{s \in\mathcal{S}} \mathbb{E}\left[|V_T(s)-V^\star(s)|^2\right] \ge\frac{c_{\mathrm{lb}}}{(1-\gamma)^4 T\log^2 T} \tag*{(219)} s ∈ S max E [ ∣ V T ( s ) − V ⋆ ( s ) ∣ 2 ] ≥ ( 1 − γ ) 4 T log 2 T c lb (219) for some universal constant c l b > 0 c_{\mathrm{lb}} > 0 c lb > 0 .
It is noteworthy that establishing an algorithm-specific lower bound requires techniques fundamentally different from those used in information-theoretic analyses (e.g., Fano’s lemma is no longer useful here). Although the hard MDP instance constructed in the proof of this theorem is partially inspired by the one in § 4.9 for deriving minimax lower bounds, the technical approach is drastically different. We defer the proof of this theorem to § 7.19 , which requires carefully tracking the dynamics of Q-learning throughout its execution.
We now discuss the implications of the algorithm-dependent lower bound established in Theorem 4.7 . This result shows that plain Q-learning cannot achieve ε \varepsilon ε -accuracy, in the sense that max s E [ ∣ V T ( s ) − V ⋆ ( s ) ∣ 2 ] ≤ ε 2 \max_{s}\mathbb{E}[|V_T(s)-V^\star(s)|^2] \le\varepsilon^2 max s E [ ∣ V T ( s ) − V ⋆ ( s ) ∣ 2 ] ≤ ε 2 , unless the number of iterations already exceeds
Ω ~ ( 1 ( 1 − γ ) 4 ε 2 ) , \widetilde{\Omega}\left(\frac{1}{(1-\gamma)^4\varepsilon^2}\right), Ω ( ( 1 − γ ) 4 ε 2 1 ) ,
within the iteration regime of Theorem 4.7 , when either constant or rescaled linear learning rates are used. This is a lower bound for the stated mean-square criterion; it does not assert a matching high-probability lower bound over the entire enlarged accuracy interval. As a consequence, the upper bound for Q-learning derived in Theorem 4.6 is tight in its dependence on the effective horizon 1 1 − γ \frac{1}{1-\gamma} 1 − γ 1 when A ≥ 2 A \ge2 A ≥ 2 . Crucially, this lower bound reveals an inherent suboptimality of plain Q-learning: its horizon dependence is provably worse than the minimax rate by a multiplicative factor of 1 1 − γ \frac{1}{1-\gamma} 1 − γ 1 (up to logarithmic factors) once there are at least two actions.
We note that the potential suboptimality of Q-learning has been recognized in the literature since its early development. A commonly cited explanation attributes such limitations to the overestimation of Q-values arising from the use of the max operator applied to empirical estimates (see, e.g., Thrun and Schwartz (1993), Hasselt (2010), and Van Hasselt et al. (2016)). Informally speaking, even in the idealistic scenario where Q t − 1 Q_{t-1} Q t − 1 forms a nearly unbiased estimate of Q ⋆ Q^\star Q ⋆ , taking the maximum over actions can introduce an inherent positive bias, that is,
E [ max a ′ ∈ A Q t − 1 ( s , a ′ ) ] ≥ max a ′ ∈ A E [ Q t − 1 ( s , a ′ ) ] ≈ max a ′ ∈ A E [ Q ⋆ ( s , a ′ ) ] , \mathbb{E}\left[\max_{a' \in\mathcal{A}} Q_{t-1}(s,a')\right] \ge\max_{a' \in\mathcal{A}} \mathbb{E}[Q_{t-1}(s,a')] \approx\max_{a' \in\mathcal{A}} \mathbb{E}[Q^\star(s,a')], E [ a ′ ∈ A max Q t − 1 ( s , a ′ ) ] ≥ a ′ ∈ A max E [ Q t − 1 ( s , a ′ )] ≈ a ′ ∈ A max E [ Q ⋆ ( s , a ′ )] ,
where max a ′ ∈ A Q t − 1 ( s , a ′ ) \max_{a' \in\mathcal{A}} Q_{t-1}(s,a') max a ′ ∈ A Q t − 1 ( s , a ′ ) is a key term in the empirical Bellman operator T t ( Q t − 1 ) \mathcal{T}_t(Q_{t-1}) T t ( Q t − 1 ) . In this sense, Theorems 4.6 and 4.7 taken together can be viewed as quantifying the impact of over-estimation on the sample complexity and convergence rate of plain Q-learning. A more quantitative comparison between over-estimation and under-estimation errors is provided towards the end of § 7.8 .
Analysis: a first attempt
Before presenting the proof of Theorem 4.6 , we first illustrate how to establish a looser upper bound O ~ ( 1 ( 1 − γ ) 5 ε 2 ) \widetilde{O}\left(\frac{1}{(1-\gamma)^5\varepsilon^2}\right) O ( ( 1 − γ ) 5 ε 2 1 ) on the iteration complexity, which exceeds the advertised bound in Theorem 4.6 by a multiplicative factor of 1 1 − γ \frac{1}{1-\gamma} 1 − γ 1 . Although suboptimal, this result features a simpler proof and introduces the error decomposition and comparison argument that we shall refine in the next subsection.
Akin to the analysis of TD learning, we adopt the vector and matrix notation introduced in § 2.5 , allowing us to express the Q-learning update rule (4.40) as
Q t = ( 1 − η t ) Q t − 1 + η t ( r + γ P t V t − 1 ) . (220) Q_t=(1-\eta_t)Q_{t-1}+\eta_t(r+\gamma P_tV_{t-1}). \tag*{(220)} Q t = ( 1 − η t ) Q t − 1 + η t ( r + γ P t V t − 1 ) . (220)
Here, the empirical transition matrix P t ∈ { 0 , 1 } S A × S P_t\in\{0,1\}^{SA\times S} P t ∈ { 0 , 1 } S A × S is defined as
∀ ( s , a , s ′ ) ∈ S × A × S : P t ( ( s , a ) , s ′ ) : = { 1 , if s ′ = s t ( s , a ) , 0 , otherwise , (221) \forall(s,a,s')\in\mathcal{S}\times\mathcal{A}\times\mathcal{S}:\quad P_t((s,a),s'):=\begin{cases}1,&\text{if }s'=s_t(s,a),\\0,&\text{otherwise},\end{cases} \tag*{(221)} ∀ ( s , a , s ′ ) ∈ S × A × S : P t (( s , a ) , s ′ ) := { 1 , 0 , if s ′ = s t ( s , a ) , otherwise , (221)
and the vector V t ∈ R S V_t\in\mathbb{R}^{S} V t ∈ R S is defined as V t ( s ) = max a Q t ( s , a ) V_t(s)=\max_a Q_t(s,a) V t ( s ) = max a Q t ( s , a ) for all s ∈ S s\in\mathcal{S} s ∈ S .
Preliminary facts. We begin with a lemma showing that, when initialized properly, each coordinate of both the Q Q Q -function and the value function estimates remains within [ 0 , 1 1 − γ ] \left[0,\frac{1}{1-\gamma}\right] [ 0 , 1 − γ 1 ] throughout the execution of the Q Q Q -learning algorithm. Its proof is deferred to § 7.17.1 .
Lemma 4.8. Assume that 0 ≤ Q 0 ( s , a ) ≤ 1 1 − γ 0\le Q_0(s,a)\le\frac{1}{1-\gamma} 0 ≤ Q 0 ( s , a ) ≤ 1 − γ 1 for all ( s , a ) ∈ S × A (s,a)\in\mathcal{S}\times\mathcal{A} ( s , a ) ∈ S × A . Then for every t ≥ 0 t\ge0 t ≥ 0 , we have
0 ≤ Q t ≤ 1 1 − γ 1 S A and 0 ≤ V t ≤ 1 1 − γ 1 S , (222) 0\le Q_t\le\frac{1}{1-\gamma}\mathbf{1}_{SA}\qquad\text{and}\qquad0\le V_t\le\frac{1}{1-\gamma}\mathbf{1}_{S}, \tag*{(222)} 0 ≤ Q t ≤ 1 − γ 1 1 S A and 0 ≤ V t ≤ 1 − γ 1 1 S , (222) provided that 0 ≤ η t ≤ 1 0\le\eta_t\le1 0 ≤ η t ≤ 1 for all t ≥ 0 t\ge0 t ≥ 0 . As an immediate consequence,
∥ Q t − Q ∗ ∥ ∞ ≤ 1 1 − γ for all t ≥ 0. (223) \lVert Q_t-Q^*\rVert_\infty\le\frac{1}{1-\gamma}\qquad\text{for all }t\ge0. \tag*{(223)} ∥ Q t − Q ∗ ∥ ∞ ≤ 1 − γ 1 for all t ≥ 0. (223)
Additionally, with a slight abuse of notation, we define
η i ( t ) : = { ∏ j = 1 t ( 1 − η j ) , if i = 0 , η i ∏ j = i + 1 t ( 1 − η j ) , if 0 < i < t , η t , if i = t , (224) \eta_i^{(t)}:=\begin{cases}
\prod_{j=1}^{t}(1-\eta_j),&\text{if }i=0,\\
\eta_i\prod_{j=i+1}^{t}(1-\eta_j),&\text{if }0<i<t,\\
\eta_t,&\text{if }i=t,
\end{cases} \tag*{(224)} η i ( t ) := ⎩ ⎨ ⎧ ∏ j = 1 t ( 1 − η j ) , η i ∏ j = i + 1 t ( 1 − η j ) , η t , if i = 0 , if 0 < i < t , if i = t , (224)
which captures the cumulative effect of the learning rates over time. Note that this definition differs from the TD counterpart in (180) in that it does not include the factor 1 − γ 1-\gamma 1 − γ . For the remainder of this subsection and the sharp analysis that follows, we set
η ˉ : = min { 1 , 2 log 3 T c 2 ( 1 − γ ) T } . (225) \bar{\eta}:=\min\left\{1,\frac{2\log^3 T}{c_2(1-\gamma)T}\right\}. \tag*{(225)} η ˉ := min { 1 , c 2 ( 1 − γ ) T 2 log 3 T } . (225)
The lemma below collects some useful properties of η i ( t ) \eta_i^{(t)} η i ( t ) , with its proof deferred to § 7.18 .
Lemma 4.9. Let η ˉ \bar{\eta} η ˉ be defined in (225) . For ⌊ T / 2 ⌋ < i ≤ t ≤ T \lfloor T/2\rfloor<i\le t\le T ⌊ T /2 ⌋ < i ≤ t ≤ T , the weights (224) satisfy
0 ≤ η i ( t ) ≤ η i ≤ η ˉ and ∑ i = ⌊ T / 2 ⌋ + 1 t ( η i ( t ) ) 2 ≤ η ˉ . (226) 0\le\eta_i^{(t)}\le\eta_i\le\bar{\eta}\qquad\text{and}\qquad\sum_{i=\lfloor T/2\rfloor+1}^{t}\left(\eta_i^{(t)}\right)^2\le\bar{\eta}. \tag*{(226)} 0 ≤ η i ( t ) ≤ η i ≤ η ˉ and i = ⌊ T /2 ⌋ + 1 ∑ t ( η i ( t ) ) 2 ≤ η ˉ . (226) Moreover, it holds that
∑ i = τ t η i ( t ) = { 1 − ∏ j = τ t ( 1 − η j ) , if 1 ≤ τ ≤ t , 1 , if τ = 0. (227) \sum_{i=\tau}^{t}\eta_i^{(t)}=\begin{cases}
1-\prod_{j=\tau}^{t}(1-\eta_j),&\text{if }1\le\tau\le t,\\
1,&\text{if }\tau=0.
\end{cases} \tag*{(227)} i = τ ∑ t η i ( t ) = { 1 − ∏ j = τ t ( 1 − η j ) , 1 , if 1 ≤ τ ≤ t , if τ = 0. (227)
Step 1: decomposing the error Q t − Q ⋆ Q_t - Q^\star Q t − Q ⋆ . Define the error sequence
Δ t : = Q t − Q ⋆ . (228) \Delta_t := Q_t - Q^\star. \tag*{(228)} Δ t := Q t − Q ⋆ . (228)
By the update rule (220) and the Bellman optimality equation Q ⋆ = r + γ P V ⋆ Q^\star= r + \gamma P V^\star Q ⋆ = r + γ P V ⋆ , we can decompose
Δ t = ( 1 − η t ) Q t − 1 + η t ( r + γ P t V t − 1 ) − Q ⋆ = ( 1 − η t ) ( Q t − 1 − Q ⋆ ) + η t ( r + γ P t V t − 1 − Q ⋆ ) = ( 1 − η t ) Δ t − 1 + γ η t ( P t V t − 1 − P V ⋆ ) = ( 1 − η t ) Δ t − 1 + γ η t P t ( V t − 1 − V ⋆ ) + γ η t ( P t − P ) V ⋆ . (229) \begin{aligned}
\Delta_t &= (1-\eta_t)Q_{t-1}+\eta_t(r+\gamma P_tV_{t-1})-Q^\star\\
&= (1-\eta_t)(Q_{t-1}-Q^\star)+\eta_t(r+\gamma P_tV_{t-1}-Q^\star) \\
&= (1-\eta_t)\Delta_{t-1}+\gamma\eta_t(P_tV_{t-1}-PV^\star) \\
&= (1-\eta_t)\Delta_{t-1}+\gamma\eta_tP_t(V_{t-1}-V^\star)+\gamma\eta_t(P_t-P)V^\star. \tag*{(229)}
\end{aligned} Δ t = ( 1 − η t ) Q t − 1 + η t ( r + γ P t V t − 1 ) − Q ⋆ = ( 1 − η t ) ( Q t − 1 − Q ⋆ ) + η t ( r + γ P t V t − 1 − Q ⋆ ) = ( 1 − η t ) Δ t − 1 + γ η t ( P t V t − 1 − P V ⋆ ) = ( 1 − η t ) Δ t − 1 + γ η t P t ( V t − 1 − V ⋆ ) + γ η t ( P t − P ) V ⋆ . (229)
Unrolling the recursion (229) from iteration ⌊ T / 2 ⌋ \lfloor T/2\rfloor ⌊ T /2 ⌋ gives
Δ t = ( ∏ j = ⌊ T / 2 ⌋ + 1 t ( 1 − η j ) ) Δ ⌊ T / 2 ⌋ + γ ∑ i = ⌊ T / 2 ⌋ + 1 t η i ( t ) P i ( V i − 1 − V ⋆ ) + γ ∑ i = ⌊ T / 2 ⌋ + 1 t η i ( t ) ( P i − P ) V ⋆ ⏟ = : ξ t , ⌊ T / 2 ⌋ < t ≤ T . (230) \begin{aligned}
\Delta_t={}&\left(\prod_{j=\lfloor T/2\rfloor+1}^{t}(1-\eta_j)\right)\Delta_{\lfloor T/2\rfloor}
+\gamma\sum_{i=\lfloor T/2\rfloor+1}^{t}\eta_i^{(t)}P_i(V_{i-1}-V^\star)\\
&+\gamma\underbrace{\sum_{i=\lfloor T/2\rfloor+1}^{t}\eta_i^{(t)}(P_i-P)V^\star}_{=:\xi_t},
\qquad\lfloor T/2\rfloor<t\le T. \tag*{(230)}
\end{aligned} Δ t = j = ⌊ T /2 ⌋ + 1 ∏ t ( 1 − η j ) Δ ⌊ T /2 ⌋ + γ i = ⌊ T /2 ⌋ + 1 ∑ t η i ( t ) P i ( V i − 1 − V ⋆ ) + γ =: ξ t i = ⌊ T /2 ⌋ + 1 ∑ t η i ( t ) ( P i − P ) V ⋆ , ⌊ T /2 ⌋ < t ≤ T . (230)
This identity decomposes the estimation error into three components: the error inherited from the previous ⌊ T / 2 ⌋ \lfloor T/2\rfloor ⌊ T /2 ⌋ iterations, the estimation error propagated thereafter, and the fluctuation term ξ t \xi_t ξ t , which we tackle separately below.
Step 2: controlling the estimation error without fluctuations. Given that
∥ V i − 1 − V ⋆ ∥ ∞ ≤ ∥ Q i − 1 − Q ⋆ ∥ ∞ = ∥ Δ i − 1 ∥ ∞ , \lVert V_{i-1}-V^\star\rVert_\infty\le\lVert Q_{i-1}-Q^\star\rVert_\infty=\lVert\Delta_{i-1}\rVert_\infty, ∥ V i − 1 − V ⋆ ∥ ∞ ≤ ∥ Q i − 1 − Q ⋆ ∥ ∞ = ∥ Δ i − 1 ∥ ∞ ,
combining (230) with the basic properties ∥ P i ∥ 1 = 1 \lVert P_i\rVert_1=1 ∥ P i ∥ 1 = 1 and ∥ Δ ⌊ T / 2 ⌋ ∥ ∞ ≤ 1 1 − γ \lVert\Delta_{\lfloor T/2\rfloor}\rVert_\infty\le\frac{1}{1-\gamma} ∥ Δ ⌊ T /2 ⌋ ∥ ∞ ≤ 1 − γ 1 implies that
∥ Δ t ∥ ∞ ≤ 1 1 − γ ∏ j = ⌊ T / 2 ⌋ + 1 t ( 1 − η j ) + γ ∑ i = ⌊ T / 2 ⌋ + 1 t η i ( t ) ∥ Δ i − 1 ∥ ∞ + ∥ ξ t ∥ ∞ . (231) \lVert\Delta_t\rVert_\infty\le\frac{1}{1-\gamma}\prod_{j=\lfloor T/2\rfloor+1}^{t}(1-\eta_j)+\gamma\sum_{i=\lfloor T/2\rfloor+1}^{t}\eta_i^{(t)}\lVert\Delta_{i-1}\rVert_\infty+\lVert\xi_t\rVert_\infty. \tag*{(231)} ∥ Δ t ∥ ∞ ≤ 1 − γ 1 j = ⌊ T /2 ⌋ + 1 ∏ t ( 1 − η j ) + γ i = ⌊ T /2 ⌋ + 1 ∑ t η i ( t ) ∥ Δ i − 1 ∥ ∞ + ∥ ξ t ∥ ∞ . (231)
To analyze this recursion, we introduce an auxiliary sequence that captures, in some sense, the deterministic part of the estimation error. Set c ⌊ T / 2 ⌋ : = 1 1 − γ c_{\lfloor T/2\rfloor}:=\frac{1}{1-\gamma} c ⌊ T /2 ⌋ := 1 − γ 1 , and for all t t t obeying ⌊ T / 2 ⌋ < t ≤ T \lfloor T/2\rfloor<t\le T ⌊ T /2 ⌋ < t ≤ T , define recursively that
c t : = 1 1 − γ ∏ j = ⌊ T / 2 ⌋ + 1 t ( 1 − η j ) + γ ∑ i = ⌊ T / 2 ⌋ + 1 t η i ( t ) c i − 1 . (232) c_t:=\frac{1}{1-\gamma}\prod_{j=\lfloor T/2\rfloor+1}^{t}(1-\eta_j)+\gamma\sum_{i=\lfloor T/2\rfloor+1}^{t}\eta_i^{(t)}c_{i-1}. \tag*{(232)} c t := 1 − γ 1 j = ⌊ T /2 ⌋ + 1 ∏ t ( 1 − η j ) + γ i = ⌊ T /2 ⌋ + 1 ∑ t η i ( t ) c i − 1 . (232)
Clearly, c t c_t c t satisfies similar recursion as the deterministic part of (231) , with the fluctuation term ∥ ξ t ∥ ∞ \lVert\xi_t\rVert_\infty ∥ ξ t ∥ ∞ omitted. Recalling from (224) that η i ( t ) = ( 1 − η t ) η i ( t − 1 ) \eta_i^{(t)}=(1-\eta_t)\eta_i^{(t-1)} η i ( t ) = ( 1 − η t ) η i ( t − 1 ) for i < t i<t i < t and η t ( t ) = η t \eta_t^{(t)}=\eta_t η t ( t ) = η t , one can rewrite the last summand in (232) to derive
c t = ( 1 − η t ) c t − 1 + γ η t c t − 1 = ( 1 − η t ( 1 − γ ) ) c t − 1 . c_t=(1-\eta_t)c_{t-1}+\gamma\eta_tc_{t-1}=(1-\eta_t(1-\gamma))c_{t-1}. c t = ( 1 − η t ) c t − 1 + γ η t c t − 1 = ( 1 − η t ( 1 − γ )) c t − 1 .
Iterating this scalar relation yields
c t = 1 1 − γ ∏ i = ⌊ T / 2 ⌋ + 1 t ( 1 − η i ( 1 − γ ) ) , ⌊ T / 2 ⌋ ≤ t ≤ T . (233) c_t=\frac{1}{1-\gamma}\prod_{i=\lfloor T/2\rfloor+1}^{t}(1-\eta_i(1-\gamma)),\qquad\lfloor T/2\rfloor\le t\le T. \tag*{(233)} c t = 1 − γ 1 i = ⌊ T /2 ⌋ + 1 ∏ t ( 1 − η i ( 1 − γ )) , ⌊ T /2 ⌋ ≤ t ≤ T . (233)
where an empty product equals one. Informally speaking, c t c_t c t reflects the error obtained by propagating the initial bound ∥ Δ ⌊ T / 2 ⌋ ∥ ∞ ≤ ( 1 − γ ) − 1 \lVert\Delta_{\lfloor T/2\rfloor}\rVert_\infty\le(1-\gamma)^{-1} ∥ Δ ⌊ T /2 ⌋ ∥ ∞ ≤ ( 1 − γ ) − 1 through the recursion (231) after setting ∥ ξ t ∥ ∞ = 0 \lVert\xi_t\rVert_\infty=0 ∥ ξ t ∥ ∞ = 0 . In particular, the factor 1 − η i ( 1 − γ ) 1-\eta_i(1-\gamma) 1 − η i ( 1 − γ ) captures the contraction factor governing the propagation of initialization error, analogous to the factor appearing in the TD bound (186) .
Step 3: accounting for the effect of fluctuations. We still need to take into account the fluctuation terms in (231) . Towards this end, we now show that these fluctuations affect the estimation error only through their largest magnitude over time. More precisely, we claim that
∥ Δ t ∥ ∞ ≤ c t + 1 1 − γ max ⌊ T / 2 ⌋ < j ≤ T ∥ ξ j ∥ ∞ for all ⌊ T / 2 ⌋ ≤ t ≤ T . (234) \lVert\Delta_t\rVert_\infty\le c_t+\frac{1}{1-\gamma}\max_{\lfloor T/2\rfloor<j\le T}\lVert\xi_j\rVert_\infty\qquad\text{for all }\lfloor T/2\rfloor\le t\le T. \tag*{(234)} ∥ Δ t ∥ ∞ ≤ c t + 1 − γ 1 ⌊ T /2 ⌋ < j ≤ T max ∥ ξ j ∥ ∞ for all ⌊ T /2 ⌋ ≤ t ≤ T . (234)
We prove this claim (234) by induction. The base case with t = ⌊ T / 2 ⌋ t=\lfloor T/2\rfloor t = ⌊ T /2 ⌋ follows from Lemma 4.8 and c ⌊ T / 2 ⌋ = ( 1 − γ ) − 1 c_{\lfloor T/2\rfloor}=(1-\gamma)^{-1} c ⌊ T /2 ⌋ = ( 1 − γ ) − 1 . Suppose the claim holds at all iterations prior to t t t . Substituting this bound into (231) , and using (232) and (227) , we arrive at
∥ Δ t ∥ ∞ ≤ c t + γ 1 − γ max ⌊ T / 2 ⌋ < j ≤ T ∥ ξ j ∥ ∞ ( 1 − ∏ j = ⌊ T / 2 ⌋ + 1 t ( 1 − η j ) ) + ∥ ξ t ∥ ∞ ≤ c t + 1 1 − γ max ⌊ T / 2 ⌋ < j ≤ T ∥ ξ j ∥ ∞ ( 1 − γ ∏ j = ⌊ T / 2 ⌋ + 1 t ( 1 − η j ) ) ≤ c t + 1 1 − γ max ⌊ T / 2 ⌋ < j ≤ T ∥ ξ j ∥ ∞ , \begin{aligned}
\lVert\Delta_t\rVert_\infty
&\le c_t+\frac{\gamma}{1-\gamma}\max_{\lfloor T/2\rfloor<j\le T}\lVert\xi_j\rVert_\infty\left(1-\prod_{j=\lfloor T/2\rfloor+1}^{t}(1-\eta_j)\right)+\lVert\xi_t\rVert_\infty\\
&\le c_t+\frac{1}{1-\gamma}\max_{\lfloor T/2\rfloor<j\le T}\lVert\xi_j\rVert_\infty\left(1-\gamma\prod_{j=\lfloor T/2\rfloor+1}^{t}(1-\eta_j)\right)\\
&\le c_t+\frac{1}{1-\gamma}\max_{\lfloor T/2\rfloor<j\le T}\lVert\xi_j\rVert_\infty,
\end{aligned} ∥ Δ t ∥ ∞ ≤ c t + 1 − γ γ ⌊ T /2 ⌋ < j ≤ T max ∥ ξ j ∥ ∞ 1 − j = ⌊ T /2 ⌋ + 1 ∏ t ( 1 − η j ) + ∥ ξ t ∥ ∞ ≤ c t + 1 − γ 1 ⌊ T /2 ⌋ < j ≤ T max ∥ ξ j ∥ ∞ 1 − γ j = ⌊ T /2 ⌋ + 1 ∏ t ( 1 − η j ) ≤ c t + 1 − γ 1 ⌊ T /2 ⌋ < j ≤ T max ∥ ξ j ∥ ∞ ,
where in the second inequality we use the fact γ 1 − γ ( 1 − C ) + 1 = 1 − γ C 1 − γ \frac{\gamma}{1-\gamma}(1-C)+1=\frac{1-\gamma C}{1-\gamma} 1 − γ γ ( 1 − C ) + 1 = 1 − γ 1 − γ C for any C > 0 C>0 C > 0 . This completes the induction, thus establishing (234) .
Step 4: bounding the size of fluctuations ξ t \xi_t ξ t . In order to make use of (234) , we need to control the sizes of the fluctuation terms { ξ t } \{\xi_t\} { ξ t } . For each fixed t t t , the summands in ξ t = γ ∑ i = ⌊ T / 2 ⌋ + 1 t η i ( t ) ( P i − P ) V ⋆ \xi_t=\gamma\sum_{i=\lfloor T/2\rfloor+1}^{t}\eta_i^{(t)}(P_i-P)V^\star ξ t = γ ∑ i = ⌊ T /2 ⌋ + 1 t η i ( t ) ( P i − P ) V ⋆ form a martingale difference sequence. Since 0 ≤ V ⋆ ≤ ( 1 − γ ) − 1 1 S 0\le V^\star\le(1-\gamma)^{-1}\mathbf{1}_S 0 ≤ V ⋆ ≤ ( 1 − γ ) − 1 1 S , it follows that
γ ∣ η i ( t ) ( P i − P ) V ⋆ ∣ ≤ γ η i ( t ) 1 − γ . \gamma\left|\eta_i^{(t)}(P_i-P)V^\star\right|\le\gamma\frac{\eta_i^{(t)}}{1-\gamma}. γ η i ( t ) ( P i − P ) V ⋆ ≤ γ 1 − γ η i ( t ) .
Applying the Azuma-Hoeffding inequality (see Lemma 2.8 ) to each coordinate of ξ t \xi_t ξ t , followed by a union bound over all S A SA S A coordinates and all ⌊ T / 2 ⌋ < t ≤ T \lfloor T/2\rfloor<t\le T ⌊ T /2 ⌋ < t ≤ T , we can demonstrate that with probability at least 1 − δ 1-\delta 1 − δ ,
∥ ξ t ∥ ∞ ≤ γ 1 − γ ∑ i = ⌊ T / 2 ⌋ + 1 t ( η i ( t ) ) 2 log 4 S A T δ ≤ γ 2 η log 4 S A T δ 1 − γ for all ⌊ T / 2 ⌋ ≤ t ≤ T . (235) \lVert\xi_t\rVert_\infty\le\frac{\gamma}{1-\gamma}\sqrt{\sum_{i=\lfloor T/2\rfloor+1}^{t}\left(\eta_i^{(t)}\right)^2\log\frac{4SAT}{\delta}}
\le\frac{\gamma\sqrt{2\eta\log\frac{4SAT}{\delta}}}{1-\gamma}\qquad\text{for all }\lfloor T/2\rfloor\le t\le T. \tag*{(235)} ∥ ξ t ∥ ∞ ≤ 1 − γ γ i = ⌊ T /2 ⌋ + 1 ∑ t ( η i ( t ) ) 2 log δ 4 S A T ≤ 1 − γ γ 2 η log δ 4 S A T for all ⌊ T /2 ⌋ ≤ t ≤ T . (235)
where the last inequality follows from Lemma 4.9 .
Step 5: bounding the term c T c_T c T . To finish the proof, it remains to bound c T c_T c T in (234) . By (233) and the elementary inequality 1 − x ≤ e − x 1-x \le e^{-x} 1 − x ≤ e − x , one has
c T ≤ 1 1 − γ exp { − ( 1 − γ ) ∑ i = ⌊ T / 2 ⌋ + 1 T η i } . (236) c_T \le\frac{1}{1-\gamma}\exp\left\{-(1-\gamma)\sum_{i=\lfloor T/2\rfloor+1}^{T}\eta_i\right\}. \tag*{(236)} c T ≤ 1 − γ 1 exp ⎩ ⎨ ⎧ − ( 1 − γ ) i = ⌊ T /2 ⌋ + 1 ∑ T η i ⎭ ⎬ ⎫ . (236)
From the sample-size condition (217) , it is easy to verify that log T ≥ 4 \log T \ge4 log T ≥ 4 and ( 1 − γ ) T ≥ log 3 T (1-\gamma)T \ge\log^3 T ( 1 − γ ) T ≥ log 3 T . Consequently, taking c 1 ≤ 1 c_1 \le1 c 1 ≤ 1 , the learning rate bound in (216) implies that
η i ≥ 1 1 + c 1 ( 1 − γ ) T / log 3 T ≥ log 3 T 2 ( 1 − γ ) T , ⌊ T / 2 ⌋ < i ≤ T , \eta_i \ge\frac{1}{1+c_1(1-\gamma)T/\log^3 T} \ge\frac{\log^3 T}{2(1-\gamma)T}, \qquad\lfloor T/2\rfloor< i \le T, η i ≥ 1 + c 1 ( 1 − γ ) T / log 3 T 1 ≥ 2 ( 1 − γ ) T log 3 T , ⌊ T /2 ⌋ < i ≤ T ,
which in turn yields
( 1 − γ ) ∑ i = ⌊ T / 2 ⌋ + 1 T η i ≥ ( 1 − γ ) T 2 ⋅ log 3 T 2 ( 1 − γ ) T = 1 4 log 3 T ≥ 4 log T . (1-\gamma)\sum_{i=\lfloor T/2\rfloor+1}^{T}\eta_i \ge(1-\gamma)\frac{T}{2}\cdot\frac{\log^3 T}{2(1-\gamma)T}=\frac{1}{4}\log^3 T\ge4\log T. ( 1 − γ ) i = ⌊ T /2 ⌋ + 1 ∑ T η i ≥ ( 1 − γ ) 2 T ⋅ 2 ( 1 − γ ) T log 3 T = 4 1 log 3 T ≥ 4 log T .
Substitution into (236) leads to
c T ≤ 1 ( 1 − γ ) T 4 . c_T\le\frac{1}{(1-\gamma)T^4}. c T ≤ ( 1 − γ ) T 4 1 .
Step 6: putting all this together. With the above bound on c T c_T c T in place, we can combine it with (235) and (234) to conclude that, with probability at least 1 − δ 1-\delta 1 − δ ,
∥ Q T − Q ∗ ∥ ∞ ≤ 1 ( 1 − γ ) T 4 + γ 2 η log 4 S A T δ ( 1 − γ ) 2 ≲ ( log 3 T ) ( log S A T δ ) ( 1 − γ ) 5 T . (237) \lVert Q_T-Q^*\rVert_\infty\le\frac{1}{(1-\gamma)T^4}+\frac{\gamma\sqrt{2\eta\log\frac{4SAT}{\delta}}}{(1-\gamma)^2}\lesssim\sqrt{\frac{(\log^3 T)\left(\log\frac{SAT}{\delta}\right)}{(1-\gamma)^5T}}. \tag*{(237)} ∥ Q T − Q ∗ ∥ ∞ ≤ ( 1 − γ ) T 4 1 + ( 1 − γ ) 2 γ 2 η log δ 4 S A T ≲ ( 1 − γ ) 5 T ( log 3 T ) ( log δ S A T ) . (237)
This in turn establishes the iteration complexity bound O ~ ( 1 ( 1 − γ ) 5 ε 2 ) \widetilde{O}\left(\frac{1}{(1-\gamma)^5\varepsilon^2}\right) O ( ( 1 − γ ) 5 ε 2 1 ) .
Discussion. As alluded to previously, the bound (237) , while admitting a concise proof, is loose by a factor of 1 1 − γ \frac{1}{1-\gamma} 1 − γ 1 . The gap stems from the use of the Azuma–Hoeffding inequality in controlling ξ t \xi_t ξ t , which relies solely on coarse magnitude bounds on the summands and fails to capture the variance structure inherent in the discounted infinite-horizon MDP. To obtain sharper guarantees, it is therefore essential to exploit this variance structure through more refined martingale concentration techniques, such as the Freedman inequality (see Lemma 2.9 ). However, even after incorporating this refinement, significant challenges remain: the resulting recursive relation becomes considerably more intricate and requires new analytical ideas to handle. These developments are the focus of the next subsection.
Sharp analysis: proof of Theorem 4.6
We now present the full proof of Theorem 4.6 , building in part on the ideas developed in the suboptimal analysis in § 7.7 . A key new idea is to introduce auxiliary comparison functions inspired by distributionally robust Bellman operators — a topic that we will revisit in detail in Section 7.1 — with uncertainty sets carefully calibrated to capture the variance of the sampling noise.
Preliminaries: distributional robustness. Before embarking on the analysis of Q-learning, we first single out some properties of distributionally robust RL, which may seem unrelated at first glance but will play a pivotal role in our sharp analysis. In contrast to the standard Bellman operator, which uses a single nominal transition kernel, a distributionally robust Bellman operator optimizes the expected value over a prescribed uncertainty set of plausible transition kernels. The following lemma characterizes how a χ 2 \chi^2 χ 2 -based uncertainty set induces a variance-dependent deviation from the nominal Bellman update. The proof is deferred to § 7.18.1 .
Lemma 4.10. For any p ∈ Δ ( S ) p \in\Delta(\mathcal{S}) p ∈ Δ ( S ) and ρ > 0 \rho> 0 ρ > 0 , define
U ρ ( p ) : = { q ∈ Δ ( S ) : χ 2 ( q ∥ p ) ≤ ρ , q ( s ) = 0 whenever p ( s ) = 0 } . (238) \mathcal{U}_{\rho}(p) := \left\{q \in\Delta(\mathcal{S}) : \chi^2(q \mathbin\Vert p) \le\rho,\ q(s) = 0\ \text{whenever}\ p(s) = 0\right\}. \tag*{(238)} U ρ ( p ) := { q ∈ Δ ( S ) : χ 2 ( q ∥ p ) ≤ ρ , q ( s ) = 0 whenever p ( s ) = 0 } . (238) where χ 2 ( ⋅ ∥ ⋅ ) \chi^2(\cdot\mathbin\Vert\cdot) χ 2 ( ⋅ ∥ ⋅ ) is the χ 2 \chi^2 χ 2 divergence defined in (4) . The following properties hold.
(i) For any V ∈ R S V \in\mathbb{R}^{S} V ∈ R S satisfying max s V ( s ) − min s V ( s ) ≤ 1 / ( 1 − γ ) \max_s V(s) - \min_s V(s) \le1/(1-\gamma) max s V ( s ) − min s V ( s ) ≤ 1/ ( 1 − γ ) , one has
ρ Var p ( V ) − ρ 1 − γ ≤ sup q ∈ U ρ ( p ) q T V − p T V ≤ ρ Var p ( V ) . (239) \sqrt{\rho\operatorname{Var}_p(V)}-\frac{\rho}{1-\gamma}
\le\sup_{q\in\mathcal{U}_{\rho}(p)}q^{\mathsf T}V-p^{\mathsf T}V
\le\sqrt{\rho\operatorname{Var}_p(V)}. \tag*{(239)} ρ Var p ( V ) − 1 − γ ρ ≤ q ∈ U ρ ( p ) sup q T V − p T V ≤ ρ Var p ( V ) . (239) The same bounds hold for p T V − inf q ∈ U ρ ( p ) q T V p^{\mathsf T}V-\inf_{q\in\mathcal{U}_{\rho}(p)}q^{\mathsf T}V p T V − inf q ∈ U ρ ( p ) q T V .
(ii) Consider an MDP with the same rewards and discount factor as the original MDP, but with transition kernel P ′ P' P ′ obeying P ′ ( ⋅ ∣ s , a ) ∈ U ρ ( P ( ⋅ ∣ s , a ) ) P'(\cdot\mid s,a)\in\mathcal{U}_{\rho}(P(\cdot\mid s,a)) P ′ ( ⋅ ∣ s , a ) ∈ U ρ ( P ( ⋅ ∣ s , a )) for every ( s , a ) ∈ S × A (s,a)\in\mathcal{S}\times\mathcal{A} ( s , a ) ∈ S × A . Its optimal Q-function, denoted by Q ⋆ , P ′ Q^{\star,P'} Q ⋆ , P ′ , satisfies
∥ Q ⋆ , P ′ − Q ⋆ ∥ ∞ ≤ ρ γ 2 ( 1 − γ ) 3 . (240) \left\|Q^{\star,P'}-Q^\star\right\|_{\infty}\le\sqrt{\frac{\rho\gamma}{2(1-\gamma)^3}}. \tag*{(240)} Q ⋆ , P ′ − Q ⋆ ∞ ≤ 2 ( 1 − γ ) 3 ρ γ . (240)
In words, part (i) of Lemma 4.10 shows that the largest change in the expected value over a χ 2 \chi^2 χ 2 -uncertainty set is controlled by the standard deviation, while part (ii) shows that the corresponding perturbation changes the optimal Q-function by only a controlled amount. As we will shortly see, these two properties allow us to construct auxiliary comparison functions that capture the variance-dependent fluctuations while remaining close to the optimal Q-function Q ⋆ Q^\star Q ⋆ .
Step 1: constructing comparison functions based on robust Bellman operators. We now leverage the above types of uncertainty sets to construct auxiliary comparison functions, whose roles will become clear momentarily. As detailed below, we employ χ 2 \chi^2 χ 2 -based perturbations of the transition kernels to capture the variance-dependent fluctuations, while using a constant reward shift to provide additional slack.
More precisely, set
ρ : = 8 η ˉ log 4 S A T δ and b : = 2 ρ 1 − γ , (241) \rho:= 8\bar{\eta}\log\frac{4SAT}{\delta}
\qquad\text{and}\qquad
b := \frac{2\rho}{1-\gamma}, \tag*{(241)} ρ := 8 η ˉ log δ 4 S A T and b := 1 − γ 2 ρ , (241)
where η ˉ \bar{\eta} η ˉ is defined in (225) . For any Q : S × A → R Q:\mathcal{S}\times\mathcal{A}\to\mathbb{R} Q : S × A → R , let V Q ( s ) : = max a ∈ A Q ( s , a ) V_Q(s):=\max_{a\in\mathcal{A}}Q(s,a) V Q ( s ) := max a ∈ A Q ( s , a ) for every s ∈ S s\in\mathcal{S} s ∈ S , and define the operators T + \mathcal{T}^{+} T + and T − \mathcal{T}^{-} T − such that, for every ( s , a ) ∈ S × A (s,a)\in\mathcal{S}\times\mathcal{A} ( s , a ) ∈ S × A ,
[ T + ( Q ) ] ( s , a ) : = r ( s , a ) + b + γ sup p ∈ U ρ ( P s , a ) p T V Q , (242) [\mathcal{T}^{+}(Q)](s,a):=r(s,a)+b+\gamma\sup_{p\in\mathcal{U}_{\rho}(P_{s,a})}p^{\mathsf T}V_Q, \tag*{(242)} [ T + ( Q )] ( s , a ) := r ( s , a ) + b + γ p ∈ U ρ ( P s , a ) sup p T V Q , (242)
[ T − ( Q ) ] ( s , a ) : = r ( s , a ) − b + γ inf p ∈ U ρ ( P s , a ) p T V Q . (243) [\mathcal{T}^{-}(Q)](s,a):=r(s,a)-b+\gamma\inf_{p\in\mathcal{U}_{\rho}(P_{s,a})}p^{\mathsf T}V_Q. \tag*{(243)} [ T − ( Q )] ( s , a ) := r ( s , a ) − b + γ p ∈ U ρ ( P s , a ) inf p T V Q . (243)
The lemma below shows that the robust Bellman-type operators T + \mathcal{T}^{+} T + and T − \mathcal{T}^{-} T − are both γ \gamma γ -contractions.
Lemma 4.11. The operators T + \mathcal{T}^{+} T + and T − \mathcal{T}^{-} T − defined in (4.69) satisfy, for any Q , Q ~ : S × A → R Q,\widetilde{Q}: \mathcal{S} \times\mathcal{A} \to\mathbb{R} Q , Q : S × A → R ,
∥ T + ( Q ) − T + ( Q ~ ) ∥ ∞ ≤ γ ∥ Q − Q ~ ∥ ∞ . (244) \left\|\mathcal{T}^{+}(Q)-\mathcal{T}^{+}(\widetilde{Q})\right\|_{\infty} \leq\gamma\left\|Q-\widetilde{Q}\right\|_{\infty}. \tag*{(244)} T + ( Q ) − T + ( Q ) ∞ ≤ γ Q − Q ∞ . (244) ∥ T − ( Q ) − T − ( Q ~ ) ∥ ∞ ≤ γ ∥ Q − Q ~ ∥ ∞ . (245) \left\|\mathcal{T}^{-}(Q)-\mathcal{T}^{-}(\widetilde{Q})\right\|_{\infty} \leq\gamma\left\|Q-\widetilde{Q}\right\|_{\infty}. \tag*{(245)} T − ( Q ) − T − ( Q ) ∞ ≤ γ Q − Q ∞ . (245)
Proof of Lemma 4.11. For every p ∈ Δ ( S ) p \in\Delta(\mathcal{S}) p ∈ Δ ( S ) , it is readily seen that
∣ p ⊤ V Q − p ⊤ V Q ~ ∣ ≤ ∥ V Q − V Q ~ ∥ ∞ ≤ ∥ Q − Q ~ ∥ ∞ . \left|p^{\top}V_Q-p^{\top}V_{\widetilde{Q}}\right| \leq\left\|V_Q-V_{\widetilde{Q}}\right\|_{\infty} \leq\left\|Q-\widetilde{Q}\right\|_{\infty}. p ⊤ V Q − p ⊤ V Q ≤ V Q − V Q ∞ ≤ Q − Q ∞ .
Using the elementary inequality
∣ sup p ∈ U f ( p ) − sup p ∈ U g ( p ) ∣ ≤ sup p ∈ U ∣ f ( p ) − g ( p ) ∣ \left|\sup_{p\in\mathcal{U}} f(p)-\sup_{p\in\mathcal{U}} g(p)\right| \leq\sup_{p\in\mathcal{U}}|f(p)-g(p)| p ∈ U sup f ( p ) − p ∈ U sup g ( p ) ≤ p ∈ U sup ∣ f ( p ) − g ( p ) ∣
we obtain, for every ( s , a ) ∈ S × A (s,a) \in\mathcal{S} \times\mathcal{A} ( s , a ) ∈ S × A ,
∣ [ T + ( Q ) ] ( s , a ) − [ T + ( Q ~ ) ] ( s , a ) ∣ ≤ γ ∥ Q − Q ~ ∥ ∞ . \left|\left[\mathcal{T}^{+}(Q)\right](s,a)-\left[\mathcal{T}^{+}(\widetilde{Q})\right](s,a)\right| \leq\gamma\left\|Q-\widetilde{Q}\right\|_{\infty}. [ T + ( Q ) ] ( s , a ) − [ T + ( Q ) ] ( s , a ) ≤ γ Q − Q ∞ .
Taking the maximum over ( s , a ) (s,a) ( s , a ) proves (244) . The proof of (245) is analogous and is therefore omitted for brevity.
Given the γ \gamma γ -contraction property of both T + \mathcal{T}^{+} T + and T − \mathcal{T}^{-} T − , we can define Q + Q^{+} Q + (resp. Q − Q^{-} Q − ) to be the unique fixed point of T + \mathcal{T}^{+} T + (resp. T − \mathcal{T}^{-} T − ). Writing V + ( s ) : = max a ∈ A Q + ( s , a ) V^{+}(s):=\max_{a\in\mathcal{A}}Q^{+}(s,a) V + ( s ) := max a ∈ A Q + ( s , a ) and V − ( s ) : = max a ∈ A Q − ( s , a ) V^{-}(s):=\max_{a\in\mathcal{A}}Q^{-}(s,a) V − ( s ) := max a ∈ A Q − ( s , a ) for every s ∈ S s\in\mathcal{S} s ∈ S , we arrive at
Q + ( s , a ) = r ( s , a ) + b + γ sup p ∈ U ρ ( P s , a ) p ⊤ V + , (246) Q^{+}(s,a)=r(s,a)+b+\gamma\sup_{p\in\mathcal{U}_{\rho}(P_{s,a})}p^{\top}V^{+}, \tag*{(246)} Q + ( s , a ) = r ( s , a ) + b + γ p ∈ U ρ ( P s , a ) sup p ⊤ V + , (246)
Q − ( s , a ) = r ( s , a ) − b + γ inf p ∈ U ρ ( P s , a ) p ⊤ V − . (247) Q^{-}(s,a)=r(s,a)-b+\gamma\inf_{p\in\mathcal{U}_{\rho}(P_{s,a})}p^{\top}V^{-}. \tag*{(247)} Q − ( s , a ) = r ( s , a ) − b + γ p ∈ U ρ ( P s , a ) inf p ⊤ V − . (247)
As we will shortly see, the fixed points Q + Q^{+} Q + and Q − Q^{-} Q − will serve as upper and lower comparison functions for the Q-learning iterates, respectively.
Before continuing, let us record a range bound that will be useful later. Subtracting the constant b / ( 1 − γ ) b/(1-\gamma) b / ( 1 − γ ) from (246) yields, for every ( s , a ) ∈ S × A (s,a) \in\mathcal{S} \times\mathcal{A} ( s , a ) ∈ S × A ,
Q + ( s , a ) − b 1 − γ = r ( s , a ) + γ sup p ∈ U ρ ( P s , a ) p ⊤ ( V + − b 1 − γ 1 S ) . (248) Q^{+}(s,a)-\frac{b}{1-\gamma}=r(s,a)+\gamma\sup_{p\in\mathcal{U}_{\rho}(P_{s,a})}p^{\top}\left(V^{+}-\frac{b}{1-\gamma}\mathbf{1}_{\mathcal{S}}\right). \tag*{(248)} Q + ( s , a ) − 1 − γ b = r ( s , a ) + γ p ∈ U ρ ( P s , a ) sup p ⊤ ( V + − 1 − γ b 1 S ) . (248)
Given that each uncertainty set is compact, we can choose a maximizer P s , a + ∈ U ρ ( P s , a ) P^{+}_{s,a}\in\mathcal{U}_{\rho}(P_{s,a}) P s , a + ∈ U ρ ( P s , a ) for each ( s , a ) (s,a) ( s , a ) . Let P + P^{+} P + denote the transition kernel whose ( s , a ) (s,a) ( s , a ) -th row is P s , a + P^{+}_{s,a} P s , a + . With this choice, (248) is the Bellman optimality equation for an MDP with reward function r r r and transition kernel P + P^{+} P + . By uniqueness of the solution to the Bellman equation, the corresponding optimal Q-function is
Q ⋆ , P + = Q + − b 1 − γ 1 S A . (249) Q^{\star,P^{+}}=Q^{+}-\frac{b}{1-\gamma}\mathbf{1}_{SA}. \tag*{(249)} Q ⋆ , P + = Q + − 1 − γ b 1 S A . (249)
Since the rewards lie in [ 0 , 1 ] [0,1] [ 0 , 1 ] , one immediately obtains
0 ≤ Q + − b 1 − γ 1 S A ≤ 1 1 − γ 1 S A . (250) 0\leq Q^{+}-\frac{b}{1-\gamma}\mathbf{1}_{SA}\leq\frac{1}{1-\gamma}\mathbf{1}_{SA}. \tag*{(250)} 0 ≤ Q + − 1 − γ b 1 S A ≤ 1 − γ 1 1 S A . (250)
Moreover, adding a constant does not change the range of a function. Therefore,
max s V + ( s ) − min s V + ( s ) ≤ 1 1 − γ . (251) \max_{s}V^{+}(s)-\min_{s}V^{+}(s)\leq\frac{1}{1-\gamma}. \tag*{(251)} s max V + ( s ) − s min V + ( s ) ≤ 1 − γ 1 . (251)
Applying the same argument to T − \mathcal{T}^{-} T − yields the corresponding range bounds for Q − Q^{-} Q − and V − V^{-} V − . Importantly, all four functions, Q + Q^{+} Q + , Q − Q^{-} Q − , V + V^{+} V + , and V − V^{-} V − , are deterministic and do not depend on the sampled transitions.
Step 2: decomposing the estimation error using the comparison functions. We now explain the role of the comparison function Q + Q^{+} Q + in (4.71). We focus on the upper bound; the lower bound follows by an analogous argument using Q − Q^{-} Q − . Decompose the estimation error as
Δ t = ( Q t − Q + ) + ( Q + − Q ⋆ ) . (252) \Delta_t = (Q_t - Q^{+}) + (Q^{+} - Q^{\star}). \tag*{(252)} Δ t = ( Q t − Q + ) + ( Q + − Q ⋆ ) . (252)
This decomposition separates the analysis into two parts: controlling the Q-learning iterates relative to Q + Q^{+} Q + , and controlling the distance between Q + Q^{+} Q + and the true optimal Q-function. To control the latter, combining Lemma 4.10 with property (250) gives
∥ Q + − Q ⋆ ∥ ∞ ≤ ∥ Q ⋆ , P + − Q ⋆ ∥ ∞ + ∥ Q ⋆ , P + − Q + ∥ ∞ ≤ γ ρ 2 ( 1 − γ ) 3 + 2 ρ ( 1 − γ ) 2 , (253) \lVert Q^{+} - Q^{\star}\rVert_{\infty} \le\lVert Q^{\star,P^{+}} - Q^{\star}\rVert_{\infty} + \lVert Q^{\star,P^{+}} - Q^{+}\rVert_{\infty}
\le\sqrt{\frac{\gamma\rho}{2(1-\gamma)^3}} + \frac{2\rho}{(1-\gamma)^2}, \tag*{(253)} ∥ Q + − Q ⋆ ∥ ∞ ≤ ∥ Q ⋆ , P + − Q ⋆ ∥ ∞ + ∥ Q ⋆ , P + − Q + ∥ ∞ ≤ 2 ( 1 − γ ) 3 γ ρ + ( 1 − γ ) 2 2 ρ , (253)
where we recall the definition of Q ⋆ , P + Q^{\star,P^{+}} Q ⋆ , P + in (249) and the choice of b b b in (241) .
Now we turn to the first term in (252) . Subtracting Q + Q^{+} Q + from the update (220) , and rearranging as in (229) , we can demonstrate that
Q t − Q + = ( 1 − η t ) ( Q t − 1 − Q + ) + γ η t P t ( V t − 1 − V + ) + γ η t ( P t − P ) V + − η t ( Q + − r − γ P V + ) . Q_t - Q^{+} = (1-\eta_t)(Q_{t-1}-Q^{+}) + \gamma\eta_t P_t(V_{t-1}-V^{+})
+ \gamma\eta_t(P_t-P)V^{+} - \eta_t(Q^{+}-r-\gamma PV^{+}). Q t − Q + = ( 1 − η t ) ( Q t − 1 − Q + ) + γ η t P t ( V t − 1 − V + ) + γ η t ( P t − P ) V + − η t ( Q + − r − γ P V + ) .
The final term can be understood as the Bellman residual of Q + Q^{+} Q + , which we denote as
Q r e s + : = Q + − r − γ P V + . (254) Q_{\mathrm{res}}^{+} := Q^{+} - r - \gamma PV^{+}. \tag*{(254)} Q res + := Q + − r − γ P V + . (254)
For t t t obeying ⌊ T / 2 ⌋ < t ≤ T \lfloor T/2\rfloor< t \le T ⌊ T /2 ⌋ < t ≤ T , unrolling the preceding recursion using (224) and (227) results in
Q t − Q + = ( ∏ j = ⌊ T / 2 ⌋ + 1 t ( 1 − η j ) ) ( Q ⌊ T / 2 ⌋ − Q + ) + γ ∑ i = ⌊ T / 2 ⌋ + 1 t η i ( t ) P i ( V i − 1 − V + ) + γ ∑ i = ⌊ T / 2 ⌋ + 1 t η i ( t ) ( P i − P ) V + − ( 1 − ∏ j = ⌊ T / 2 ⌋ + 1 t ( 1 − η j ) ) Q r e s + , (255) \begin{aligned}
Q_t-Q^{+}
={}&\left(\prod_{j=\lfloor T/2\rfloor+1}^{t}(1-\eta_j)\right)(Q_{\lfloor T/2\rfloor}-Q^{+})
+\gamma\sum_{i=\lfloor T/2\rfloor+1}^{t}\eta_i^{(t)}P_i(V_{i-1}-V^{+})\\
&+\gamma\sum_{i=\lfloor T/2\rfloor+1}^{t}\eta_i^{(t)}(P_i-P)V^{+}
-\left(1-\prod_{j=\lfloor T/2\rfloor+1}^{t}(1-\eta_j)\right)Q_{\mathrm{res}}^{+}, \tag*{(255)}
\end{aligned} Q t − Q + = j = ⌊ T /2 ⌋ + 1 ∏ t ( 1 − η j ) ( Q ⌊ T /2 ⌋ − Q + ) + γ i = ⌊ T /2 ⌋ + 1 ∑ t η i ( t ) P i ( V i − 1 − V + ) + γ i = ⌊ T /2 ⌋ + 1 ∑ t η i ( t ) ( P i − P ) V + − 1 − j = ⌊ T /2 ⌋ + 1 ∏ t ( 1 − η j ) Q res + , (255)
Here, ξ t + \xi_t^{+} ξ t + is the martingale fluctuation that reflects the randomness of the transition samples when the deterministic comparison value function V + V^{+} V + is used. Importantly, the comparison function Q + Q^{+} Q + is constructed so that its Bellman residual can absorb this fluctuation term ξ t + \xi_t^{+} ξ t + . In particular, if
∣ ξ t + ∣ ≤ Q r e s + (256) \lvert\xi_t^{+}\rvert\le Q_{\mathrm{res}}^{+} \tag*{(256)} ∣ ξ t + ∣ ≤ Q res + (256)
holds componentwise for all ⌊ T / 2 ⌋ < t ≤ T \lfloor T/2\rfloor< t \le T ⌊ T /2 ⌋ < t ≤ T , then the last two terms in (255) satisfy
ξ t + − ( 1 − ∏ j = ⌊ T / 2 ⌋ + 1 t ( 1 − η j ) ) Q r e s + ≤ ( ∏ j = ⌊ T / 2 ⌋ + 1 t ( 1 − η j ) ) Q r e s + . \xi_t^{+}-\left(1-\prod_{j=\lfloor T/2\rfloor+1}^{t}(1-\eta_j)\right)Q_{\mathrm{res}}^{+}
\le\left(\prod_{j=\lfloor T/2\rfloor+1}^{t}(1-\eta_j)\right)Q_{\mathrm{res}}^{+}. ξ t + − 1 − j = ⌊ T /2 ⌋ + 1 ∏ t ( 1 − η j ) Q res + ≤ j = ⌊ T /2 ⌋ + 1 ∏ t ( 1 − η j ) Q res + .
Thus, these two terms combine into a contribution weighted by the initial contraction factor. The remaining propagated error can then be controlled by an induction analogous to the one in § 7.7 . We verify next that Q r e s + Q_{\mathrm{res}}^{+} Q res + indeed dominates ξ t + \xi_t^{+} ξ t + .
Step 3: controlling the fluctuation term. We next verify the condition (256) . The key observation is that: since V + V^{+} V + is deterministic, the fluctuation term ξ t + \xi_t^{+} ξ t + is a martingale sum with a variance that can be expressed directly in terms of Var P ( V + ) \operatorname{Var}_{P}(V^{+}) Var P ( V + ) . We then show that the Bellman residual of Q + Q^{+} Q + is large enough to dominate this fluctuation.
More precisely, for each fixed t t t , the quantity ξ t + \xi_t^{+} ξ t + is a sum of martingale differences, which can be bounded via Freedman’s inequality (cf. Lemma 2.9 ). For a fixed coordinate ( s , a ) (s,a) ( s , a ) , write
ξ t + ( s , a ) = ∑ i = ⌊ T / 2 ⌋ + 1 t Z i + , with Z i + : = γ η i ( t ) [ ( P i − P ) V + ] ( s , a ) . \xi_t^{+}(s,a)=\sum_{i=\lfloor T/2\rfloor+1}^{t} Z_i^{+}, \qquad\text{with } Z_i^{+}:=\gamma\eta_i^{(t)}[(P_i-P)V^{+}](s,a). ξ t + ( s , a ) = i = ⌊ T /2 ⌋ + 1 ∑ t Z i + , with Z i + := γ η i ( t ) [( P i − P ) V + ] ( s , a ) .
As before, let E i \mathbb{E}_i E i denote conditional expectation given the history through iteration i i i . Since V + V^{+} V + is deterministic and E i − 1 [ P i ] = P \mathbb{E}_{i-1}[P_i]=P E i − 1 [ P i ] = P , it follows that
E i − 1 [ Z i + ] = 0 , ∣ Z i + ∣ ≤ η ˉ 1 − γ , \mathbb{E}_{i-1}[Z_i^{+}]=0,\qquad|Z_i^{+}|\leq\frac{\bar{\eta}}{1-\gamma}, E i − 1 [ Z i + ] = 0 , ∣ Z i + ∣ ≤ 1 − γ η ˉ ,
∑ i = ⌊ T / 2 ⌋ + 1 t Var i − 1 ( Z i + ) = γ 2 ∑ i = ⌊ T / 2 ⌋ + 1 t ( η i ( t ) ) 2 Var P s , a ( V + ) ≤ γ 2 η ˉ Var P s , a ( V + ) . \sum_{i=\lfloor T/2\rfloor+1}^{t}\operatorname{Var}_{i-1}(Z_i^{+})=\gamma^2\sum_{i=\lfloor T/2\rfloor+1}^{t}(\eta_i^{(t)})^2\operatorname{Var}_{P_{s,a}}(V^{+})\leq\gamma^2\bar{\eta}\operatorname{Var}_{P_{s,a}}(V^{+}). i = ⌊ T /2 ⌋ + 1 ∑ t Var i − 1 ( Z i + ) = γ 2 i = ⌊ T /2 ⌋ + 1 ∑ t ( η i ( t ) ) 2 Var P s , a ( V + ) ≤ γ 2 η ˉ Var P s , a ( V + ) .
Here, the bound on ∣ Z i + ∣ |Z_i^{+}| ∣ Z i + ∣ follows because each coordinate of ( P i − P ) V + (P_i-P)V^{+} ( P i − P ) V + is bounded by the range of V + V^{+} V + , which is at most 1 1 − γ \frac{1}{1-\gamma} 1 − γ 1 by (251) . The last inequality arises from (226) . Applying Freedman’s inequality (Lemma 2.9 ) with K 0 = 1 K_0=1 K 0 = 1 , followed by a union bound over all S A SA S A coordinates and all ⌊ T / 2 ⌋ < t ≤ T \lfloor T/2\rfloor<t\leq T ⌊ T /2 ⌋ < t ≤ T , we can show that, with probability at least 1 − δ / 2 1-\delta/2 1 − δ /2 ,
∣ ξ t + ∣ ≤ γ ρ Var P ( V + ) + γ ρ 6 ( 1 − γ ) 1 S A , for ⌊ T / 2 ⌋ < t ≤ T , (257) |\xi_t^{+}|\leq\gamma\sqrt{\rho\operatorname{Var}_{P}(V^{+})}+\frac{\gamma\rho}{6(1-\gamma)}\mathbf{1}_{SA},\qquad\text{for }\lfloor T/2\rfloor<t\leq T, \tag*{(257)} ∣ ξ t + ∣ ≤ γ ρ Var P ( V + ) + 6 ( 1 − γ ) γ ρ 1 S A , for ⌊ T /2 ⌋ < t ≤ T , (257)
where we recall the choice of ρ \rho ρ in (241) .
It remains to show that Q r e s + Q_{\mathrm{res}}^{+} Q res + dominates the right-hand side of (257) . Substituting (4.71) into the definition of Q r e s + Q_{\mathrm{res}}^{+} Q res + gives, for every ( s , a ) (s,a) ( s , a ) ,
Q r e s + ( s , a ) = b + γ ( sup p ∈ U ρ ( P s , a ) p T V + − P s , a V + ) . Q_{\mathrm{res}}^{+}(s,a)=b+\gamma\left(\sup_{p\in\mathcal{U}_{\rho}(P_{s,a})}p^{\mathsf{T}}V^{+}-P_{s,a}V^{+}\right). Q res + ( s , a ) = b + γ ( p ∈ U ρ ( P s , a ) sup p T V + − P s , a V + ) .
Applying part (i) of Lemma 4.10 , and using b = 2 ρ 1 − γ b=\frac{2\rho}{1-\gamma} b = 1 − γ 2 ρ together with the range bound (251) , we reach
γ ρ Var P ( V + ) + ( 2 − γ ) ρ 1 − γ 1 S A ≤ Q r e s + ≤ 2 ρ + ρ 1 − γ 1 S A , (258) \gamma\sqrt{\rho\operatorname{Var}_{P}(V^{+})}+\frac{(2-\gamma)\rho}{1-\gamma}\mathbf{1}_{SA}\leq Q_{\mathrm{res}}^{+}\leq\frac{2\rho+\sqrt{\rho}}{1-\gamma}\mathbf{1}_{SA}, \tag*{(258)} γ ρ Var P ( V + ) + 1 − γ ( 2 − γ ) ρ 1 S A ≤ Q res + ≤ 1 − γ 2 ρ + ρ 1 S A , (258)
where for the upper bound, we used Var P s , a ( V + ) ≤ ( 1 − γ ) − 2 \operatorname{Var}_{P_{s,a}}(V^{+})\leq(1-\gamma)^{-2} Var P s , a ( V + ) ≤ ( 1 − γ ) − 2 and γ ≤ 1 \gamma\leq1 γ ≤ 1 . Since 2 − γ ≥ γ / 6 2-\gamma\geq\gamma/6 2 − γ ≥ γ /6 , combining (257) and (258) , we conclude that
∣ ξ t + ∣ ≤ Q r e s + for all ⌊ T / 2 ⌋ < t ≤ T (259) |\xi_t^{+}|\leq Q_{\mathrm{res}}^{+}\qquad\text{for all }\lfloor T/2\rfloor<t\leq T \tag*{(259)} ∣ ξ t + ∣ ≤ Q res + for all ⌊ T /2 ⌋ < t ≤ T (259)
holds with probability at least 1 − δ / 2 1-\delta/2 1 − δ /2 . This establishes the absorption condition (256) discussed in Step 2.
Step 4: propagating the comparison bound through Q-learning iterates. We now use the residual bound from Step 3 to control the Q-learning iterates relative to the comparison function Q + Q^{+} Q + . Substituting (259) into (255) yields, componentwise,
Q t − Q + ≤ ( ∏ j = ⌊ T / 2 ⌋ + 1 t ( 1 − η j ) ) ( Q ⌊ T / 2 ⌋ − Q + + Q r e s + ) + γ ∑ i = ⌊ T / 2 ⌋ + 1 t η i ( t ) P i ( V i − 1 − V + ) . (260) Q_t-Q^{+}\leq\left(\prod_{j=\lfloor T/2\rfloor+1}^{t}(1-\eta_j)\right)(Q_{\lfloor T/2\rfloor}-Q^{+}+Q_{\mathrm{res}}^{+})+\gamma\sum_{i=\lfloor T/2\rfloor+1}^{t}\eta_i^{(t)}P_i(V_{i-1}-V^{+}). \tag*{(260)} Q t − Q + ≤ j = ⌊ T /2 ⌋ + 1 ∏ t ( 1 − η j ) ( Q ⌊ T /2 ⌋ − Q + + Q res + ) + γ i = ⌊ T /2 ⌋ + 1 ∑ t η i ( t ) P i ( V i − 1 − V + ) . (260)
The residual has therefore been absorbed into the initial term, leaving a recursion of the same form as the one analyzed in § 7.7 . Moreover, by Lemma 4.8 , Q ⌊ T / 2 ⌋ Q_{\lfloor T/2\rfloor} Q ⌊ T /2 ⌋ is bounded above by ( 1 − γ ) − 1 1 S A (1-\gamma)^{-1}\mathbf{1}_{SA} ( 1 − γ ) − 1 1 S A , while Q + ≥ 0 Q^{+}\ge0 Q + ≥ 0 . Combining these facts with the upper bound in (258) gives
Q ⌊ T / 2 ⌋ − Q + + Q r e s + ≤ 1 + 2 ρ + ρ 1 − γ 1 S A . Q_{\lfloor T/2\rfloor}-Q^{+}+Q_{\mathrm{res}}^{+}\leq\frac{1+2\rho+\sqrt{\rho}}{1-\gamma}\mathbf{1}_{SA}. Q ⌊ T /2 ⌋ − Q + + Q res + ≤ 1 − γ 1 + 2 ρ + ρ 1 S A .
To propagate this bound through the remaining recursion, we find it convenient to define
c ~ t : = 1 + 2 ρ + ρ 1 − γ ∏ i = ⌊ T / 2 ⌋ + 1 t ( 1 − η i ( 1 − γ ) ) , ⌊ T / 2 ⌋ ≤ t ≤ T , (261) \widetilde{c}_{t}:=\frac{1+2\rho+\sqrt{\rho}}{1-\gamma}\prod_{i=\lfloor T/2\rfloor+1}^{t}(1-\eta_i(1-\gamma)),\qquad\lfloor T/2\rfloor\leq t\leq T, \tag*{(261)} c t := 1 − γ 1 + 2 ρ + ρ i = ⌊ T /2 ⌋ + 1 ∏ t ( 1 − η i ( 1 − γ )) , ⌊ T /2 ⌋ ≤ t ≤ T , (261)
where an empty product equals one. Applying the same induction used to establish (234) in § 7.7 — now with (260) as the recursive inequality and c ~ t \widetilde{c}_t c t as the corresponding radius — and using (227) together with ∥ P i ∥ 1 = 1 \lVert P_i\rVert_1=1 ∥ P i ∥ 1 = 1 , we arrive at
Q t ≤ Q + + c ~ t 1 S A , ⌊ T / 2 ⌋ ≤ t ≤ T . (262) Q_t\leq Q^{+}+\widetilde{c}_t\mathbf{1}_{SA},\qquad\lfloor T/2\rfloor\leq t\leq T. \tag*{(262)} Q t ≤ Q + + c t 1 S A , ⌊ T /2 ⌋ ≤ t ≤ T . (262)
Thus, the Q-learning iterates are controlled by the comparison function Q + Q^{+} Q + up to a term that contracts at rate 1 − η i ( 1 − γ ) 1-\eta_i(1-\gamma) 1 − η i ( 1 − γ ) . Combining (253) and (262) with (252) gives
Δ T ≤ 1 + 2 ρ + ρ 1 − γ ∏ i = ⌊ T / 2 ⌋ + 1 T ( 1 − η i ( 1 − γ ) ) 1 S A + ( γ ρ 2 ( 1 − γ ) 3 + 2 ρ ( 1 − γ ) 2 ) 1 S A . (263) \Delta_T\leq\frac{1+2\rho+\sqrt{\rho}}{1-\gamma}\prod_{i=\lfloor T/2\rfloor+1}^{T}(1-\eta_i(1-\gamma))\mathbf{1}_{SA}
+\left(\sqrt{\frac{\gamma\rho}{2(1-\gamma)^3}}+\frac{2\rho}{(1-\gamma)^2}\right)\mathbf{1}_{SA}. \tag*{(263)} Δ T ≤ 1 − γ 1 + 2 ρ + ρ i = ⌊ T /2 ⌋ + 1 ∏ T ( 1 − η i ( 1 − γ )) 1 S A + ( 2 ( 1 − γ ) 3 γ ρ + ( 1 − γ ) 2 2 ρ ) 1 S A . (263)
Additionally, applying the same argument to the lower comparison function Q − Q^{-} Q − yields an upper bound on − Δ T -\Delta_T − Δ T of the same form. Combining the resulting upper and lower bounds, we conclude that, with probability at least 1 − δ 1-\delta 1 − δ ,
∥ Δ T ∥ ∞ ≤ 1 + 2 ρ + ρ 1 − γ ∏ i = ⌊ T / 2 ⌋ + 1 T ( 1 − η i ( 1 − γ ) ) + γ ρ 2 ( 1 − γ ) 3 + 2 ρ ( 1 − γ ) 2 . (264) \lVert\Delta_T\rVert_{\infty}\leq\frac{1+2\rho+\sqrt{\rho}}{1-\gamma}\prod_{i=\lfloor T/2\rfloor+1}^{T}(1-\eta_i(1-\gamma))
+\sqrt{\frac{\gamma\rho}{2(1-\gamma)^3}}+\frac{2\rho}{(1-\gamma)^2}. \tag*{(264)} ∥ Δ T ∥ ∞ ≤ 1 − γ 1 + 2 ρ + ρ i = ⌊ T /2 ⌋ + 1 ∏ T ( 1 − η i ( 1 − γ )) + 2 ( 1 − γ ) 3 γ ρ + ( 1 − γ ) 2 2 ρ . (264)
Step 5: completing the proof. We now verify that the right-hand side of (264) is at most ε \varepsilon ε under the sample-size condition (217) .
First, we bound the two terms involving ρ \rho ρ . Since S A T / δ ≥ 2 SAT/\delta\geq2 S A T / δ ≥ 2 ,
log 4 S A T δ ≤ 3 log S A T δ . \log\frac{4SAT}{\delta}\leq3\log\frac{SAT}{\delta}. log δ 4 S A T ≤ 3 log δ S A T .
Therefore, the choice (241) of ρ \rho ρ , together with (225) and the sample-size condition (217) , gives
ρ ≤ 48 ( log 3 T ) log S A T δ c 2 ( 1 − γ ) T ≤ 48 ε 2 ( 1 − γ ) 3 c 2 c 3 log T . (265) \rho\leq\frac{48(\log^3 T)\log\frac{SAT}{\delta}}{c_2(1-\gamma)T}\leq\frac{48\varepsilon^2(1-\gamma)^3}{c_2c_3\log T}. \tag*{(265)} ρ ≤ c 2 ( 1 − γ ) T 48 ( log 3 T ) log δ S A T ≤ c 2 c 3 log T 48 ε 2 ( 1 − γ ) 3 . (265)
Consequently,
γ ρ 2 ( 1 − γ ) 3 ≤ 2 ε 6 c 2 c 3 log T ≤ ε 4 , \sqrt{\frac{\gamma\rho}{2(1-\gamma)^3}}\leq2\varepsilon\sqrt{\frac{6}{c_2c_3\log T}}\leq\frac{\varepsilon}{4}, 2 ( 1 − γ ) 3 γ ρ ≤ 2 ε c 2 c 3 log T 6 ≤ 4 ε ,
2 ρ ( 1 − γ ) 2 ≤ 96 ε 2 ( 1 − γ ) c 2 c 3 log T ≤ ε 4 , \frac{2\rho}{(1-\gamma)^2} \le\frac{96\varepsilon^2(1-\gamma)}{c_2c_3\log T} \le\frac{\varepsilon}{4}, ( 1 − γ ) 2 2 ρ ≤ c 2 c 3 log T 96 ε 2 ( 1 − γ ) ≤ 4 ε ,
provided that the universal constants are chosen sufficiently large.
It remains to bound the first term on the right-hand side of (264) . Comparing (261) with (233) gives
c ~ T = ( 1 + 2 ρ + ρ ) c T . \widetilde{c}_T = (1+2\rho+\sqrt{\rho})c_T. c T = ( 1 + 2 ρ + ρ ) c T .
The preceding estimates imply ρ ≤ 1 / 16 \rho\le1/16 ρ ≤ 1/16 , and hence
1 + 2 ρ + ρ ≤ 2. 1+2\rho+\sqrt{\rho} \le2. 1 + 2 ρ + ρ ≤ 2.
Using the bound on c T c_T c T established at the end of § 7.7 , we obtain
c ~ T ≤ 2 c T ≤ 2 ( 1 − γ ) T 4 ≤ ε 2 . \widetilde{c}_T \le2c_T \le\frac{2}{(1-\gamma)T^4} \le\frac{\varepsilon}{2}. c T ≤ 2 c T ≤ ( 1 − γ ) T 4 2 ≤ 2 ε .
Here, the last inequality makes use of T ≥ 2 T \ge2 T ≥ 2 and
ε ≥ 1 ( 1 − γ ) T , \varepsilon\ge\frac{1}{(1-\gamma)\sqrt{T}}, ε ≥ ( 1 − γ ) T 1 ,
which follows from the sample-size condition (217) , provided that the constant c 3 c_3 c 3 is sufficiently large. Combining the three bounds with (264) yields
∥ Q T − Q ⋆ ∥ ∞ ≤ ε \lVert Q_T-Q^\star\rVert_\infty\le\varepsilon ∥ Q T − Q ⋆ ∥ ∞ ≤ ε
with probability at least 1 − δ 1-\delta 1 − δ , thereby completing the proof.
Discussion: over-estimation vs. under-estimation. Before concluding this subsection, we highlight an important difference between over-estimation and under-estimation in Q-learning. Consider an auxiliary TD-type iteration Q ~ t \widetilde{Q}_t Q t that employs the same initialization and transition samples as Q-learning, but follows a deterministic optimal policy π ⋆ \pi^\star π ⋆ instead of maximizing over actions. Since Q π ⋆ = Q ⋆ Q^{\pi^\star}=Q^\star Q π ⋆ = Q ⋆ , this auxiliary iteration evaluates the optimal policy. Moreover, because Q-learning maximizes over actions, one can anticipate that Q ~ t ≤ Q t \widetilde{Q}_t \le Q_t Q t ≤ Q t . Hence, the lower deviation of Q-learning can be bounded below by that of the auxiliary TD-type iteration. The TD-learning analysis gives
Q T − Q ⋆ ≥ Q ~ T − Q ⋆ ≥ − O ~ ( 1 ( 1 − γ ) 3 T ) 1 S A , Q_T-Q^\star\ge\widetilde{Q}_T-Q^\star\ge-\widetilde{O}\left(\sqrt{\frac{1}{(1-\gamma)^3T}}\right)\mathbf{1}_{SA}, Q T − Q ⋆ ≥ Q T − Q ⋆ ≥ − O ( ( 1 − γ ) 3 T 1 ) 1 S A ,
which is smaller in magnitude than the leading square-root term in the corresponding over-estimation bound by a factor of 1 / 1 − γ 1/\sqrt{1-\gamma} 1/ 1 − γ . In other words, over-estimation poses a more significant issue in Q-learning than under-estimation, an asymmetry reflected both in the proof and in practice.
Extension: asynchronous Q-learning
As noted previously, the Q-learning algorithm presented in Algorithm 4.2 is commonly referred to as synchronous Q-learning, reflecting the fact that the Q-values for all state–action pairs are updated simultaneously at each iteration. Another important practical variant is asynchronous Q-learning (Tsitsiklis, 1994; Jaakkola et al., 1993), in which only a single state–action pair is updated per iteration (typically following a Markovian trajectory, to be made precise shortly). Although the sampling mechanism in asynchronous Q-learning often deviates substantially from that of a generative model, its core algorithmic ideas and theoretical guarantees remain closely aligned with those of the synchronous counterpart. For this reason, we introduce this variant in this chapter as a natural extension of synchronous Q-learning.
Learning from Markovian samples
In contrast to the generative model setting, asynchronous Q Q Q -learning is most commonly applied in settings where samples are collected along a Markovian trajectory. To be precise, suppose that we have access to a sample trajectory { ( s t , a t , r t ) } t = 0 ∞ \{(s_t,a_t,r_t)\}_{t=0}^{\infty} {( s t , a t , r t ) } t = 0 ∞ , generated by the MDP of interest M = { S , A , P , r , γ } \mathcal{M}=\{\mathcal{S},\mathcal{A},P,r,\gamma\} M = { S , A , P , r , γ } under a stationary behavior policy π b \pi^{b} π b — so called because this policy dictates the actual “behavior” in generating this sample trajectory. Starting from an initial state s 0 s_0 s 0 , this trajectory evolves sequentially as
a t ∼ π b ( ⋅ ∣ s t ) , r t = r ( s t , a t ) , s t + 1 ∼ P ( ⋅ ∣ s t , a t ) , t ≥ 0 ; (266) a_t \sim\pi^{b}(\cdot\mid s_t),\quad r_t=r(s_t,a_t),\quad s_{t+1}\sim P(\cdot\mid s_t,a_t),\quad t\geq0; \tag*{(266)} a t ∼ π b ( ⋅ ∣ s t ) , r t = r ( s t , a t ) , s t + 1 ∼ P ( ⋅ ∣ s t , a t ) , t ≥ 0 ; (266)
see Figure 4.1 for an illustration. When π b \pi^{b} π b is stationary, the process { ( s t , a t ) } t = 0 ∞ \{(s_t,a_t)\}_{t=0}^{\infty} {( s t , a t ) } t = 0 ∞ can be viewed as a sample path of a time-homogeneous Markov chain. Notably, the behavior policy π b \pi^{b} π b may differ substantially from the target optimal policy π ⋆ \pi^{\star} π ⋆ . Throughout this section, we assume that asynchronous Q Q Q -learning operates on this single Markovian trajectory, with sample complexity naturally measured by the length of the observed trajectory required to achieve a prescribed level of accuracy.
Figure 4.1. In asynchronous Q-learning, one observes a single Markovian trajectory { ( s t , a t ) } t ≥ 0 \{(s_t,a_t)\}_{t\geq0} {( s t , a t ) } t ≥ 0 induced by a fixed behavior policy π b \pi^{b} π b , and updates the Q Q Q -estimate sequentially according to it.
Algorithm: asynchronous Q-learning
We now introduce the asynchronous Q-learning algorithm, in which only the state–action pair visited along the Markovian trajectory is updated at each iteration. Specifically, given the Markovian trajectory { s t , a t , r t } t = 0 ∞ \{s_t,a_t,r_t\}_{t=0}^{\infty} { s t , a t , r t } t = 0 ∞ generated by the behavior policy π b \pi^{b} π b , the asynchronous Q-learning algorithm maintains a Q Q Q -function estimate Q t : S × A → R Q_t:\mathcal{S}\times\mathcal{A}\to\mathbb{R} Q t : S × A → R at each time t ≥ 0 t\geq0 t ≥ 0 and updates only one entry at each iterate as follows:
Q t ( s t − 1 , a t − 1 ) = ( 1 − η t ) Q t − 1 ( s t − 1 , a t − 1 ) + η t T t ( Q t − 1 ) ( s t − 1 , a t − 1 ) , Q t ( s , a ) = Q t − 1 ( s , a ) , ∀ ( s , a ) ≠ ( s t − 1 , a t − 1 ) , (267) \begin{aligned}
Q_t(s_{t-1},a_{t-1})&=(1-\eta_t)Q_{t-1}(s_{t-1},a_{t-1})+\eta_t\mathcal{T}_t(Q_{t-1})(s_{t-1},a_{t-1}),\\
Q_t(s,a)&=Q_{t-1}(s,a),\qquad\forall(s,a)\ne(s_{t-1},a_{t-1}), \tag*{(267)}
\end{aligned} Q t ( s t − 1 , a t − 1 ) Q t ( s , a ) = ( 1 − η t ) Q t − 1 ( s t − 1 , a t − 1 ) + η t T t ( Q t − 1 ) ( s t − 1 , a t − 1 ) , = Q t − 1 ( s , a ) , ∀ ( s , a ) = ( s t − 1 , a t − 1 ) , (267)
Algorithm 4.3: Asynchronous Q-learning
input: discount factor γ \gamma γ , reward function r r r , learning rates { η t } \{\eta_t\} { η t } , number of iterations T T T , initialization Q 0 Q_0 Q 0 .
for t = 1 , 2 , ⋯ , T t = 1, 2, \cdots, T t = 1 , 2 , ⋯ , T do
draw action a t − 1 ∼ π b ( s t − 1 ) a_{t-1} \sim\pi_b(s_{t-1}) a t − 1 ∼ π b ( s t − 1 ) , observe reward r ( s t − 1 , a t − 1 ) r(s_{t-1}, a_{t-1}) r ( s t − 1 , a t − 1 ) , and draw next state s t ∼ P ( ⋅ ∣ s t − 1 , a t − 1 ) s_t \sim P(\cdot\mid s_{t-1}, a_{t-1}) s t ∼ P ( ⋅ ∣ s t − 1 , a t − 1 ) .
/* stochastic approximation. */
update Q t Q_t Q t according to
Q t ( s t − 1 , a t − 1 ) = ( 1 − η t ) Q t − 1 ( s t − 1 , a t − 1 ) + η t T t ( Q t − 1 ) ( s t − 1 , a t − 1 ) , Q t ( s , a ) = Q t − 1 ( s , a ) , ∀ ( s , a ) ≠ ( s t − 1 , a t − 1 ) , \begin{aligned}
Q_t(s_{t-1},a_{t-1}) &= (1-\eta_t)Q_{t-1}(s_{t-1},a_{t-1})+\eta_t\mathcal{T}_t(Q_{t-1})(s_{t-1},a_{t-1}),\\
Q_t(s,a) &= Q_{t-1}(s,a),\qquad\forall(s,a)\ne(s_{t-1},a_{t-1}),
\end{aligned} Q t ( s t − 1 , a t − 1 ) Q t ( s , a ) = ( 1 − η t ) Q t − 1 ( s t − 1 , a t − 1 ) + η t T t ( Q t − 1 ) ( s t − 1 , a t − 1 ) , = Q t − 1 ( s , a ) , ∀ ( s , a ) = ( s t − 1 , a t − 1 ) , where the ( s t − 1 , a t − 1 ) (s_{t-1},a_{t-1}) ( s t − 1 , a t − 1 ) -th entry of T t ( Q ) \mathcal{T}_t(Q) T t ( Q ) is given by
T t ( Q ) ( s t − 1 , a t − 1 ) = r ( s t − 1 , a t − 1 ) + γ max a ′ ∈ A Q ( s t , a ′ ) . \mathcal{T}_t(Q)(s_{t-1},a_{t-1})=r(s_{t-1},a_{t-1})+\gamma\max_{a'\in\mathcal{A}}Q(s_t,a'). T t ( Q ) ( s t − 1 , a t − 1 ) = r ( s t − 1 , a t − 1 ) + γ a ′ ∈ A max Q ( s t , a ′ ) .
whereas η t ∈ ( 0 , 1 ) \eta_t \in(0,1) η t ∈ ( 0 , 1 ) represents the learning rate, and T t \mathcal{T}_t T t denotes the empirical Bellman optimality operator w.r.t. the t t t -th sample transition ( s t − 1 , a t − 1 , s t ) (s_{t-1},a_{t-1},s_t) ( s t − 1 , a t − 1 , s t ) of the Markovian trajectory, defined as
T t ( Q ) ( s t − 1 , a t − 1 ) : = r ( s t − 1 , a t − 1 ) + γ max a ′ ∈ A Q ( s t , a ′ ) . (268) \mathcal{T}_t(Q)(s_{t-1},a_{t-1}) := r(s_{t-1},a_{t-1})+\gamma\max_{a'\in\mathcal{A}}Q(s_t,a'). \tag*{(268)} T t ( Q ) ( s t − 1 , a t − 1 ) := r ( s t − 1 , a t − 1 ) + γ a ′ ∈ A max Q ( s t , a ′ ) . (268)
It is self-evident that the term T t ( Q ) ( s t − 1 , a t − 1 ) \mathcal{T}_t(Q)(s_{t-1},a_{t-1}) T t ( Q ) ( s t − 1 , a t − 1 ) at iteration t t t can be viewed as a stochastic, unbiased estimate of the deterministic quantity T ( Q t − 1 ) ( s t − 1 , a t − 1 ) \mathcal{T}(Q_{t-1})(s_{t-1},a_{t-1}) T ( Q t − 1 ) ( s t − 1 , a t − 1 ) , where T ( ⋅ ) \mathcal{T}(\cdot) T ( ⋅ ) is the Bellman optimality operator defined in (21) . Consequently, the update of Q t ( s t − 1 , a t − 1 ) Q_t(s_{t-1},a_{t-1}) Q t ( s t − 1 , a t − 1 ) follows the paradigm of stochastic approximation, and asynchronous Q-learning can also be interpreted as a stochastic approximation scheme for computing the fixed point of the Bellman optimality operator. The corresponding value function estimate V t : S → R V_t:\mathcal{S}\to\mathbb{R} V t : S → R at time t t t is again given by
∀ s ∈ S : V t ( s ) : = max a ∈ A Q t ( s , a ) . \forall s\in\mathcal{S}: \qquad V_t(s):=\max_{a\in\mathcal{A}}Q_t(s,a). ∀ s ∈ S : V t ( s ) := a ∈ A max Q t ( s , a ) .
It is worth emphasizing that, at each time t t t , only a single entry — the one corresponding to the sampled state-action pair ( s t − 1 , a t − 1 ) (s_{t-1},a_{t-1}) ( s t − 1 , a t − 1 ) — is updated, while all other entries remain unchanged; see Figure 4.1 for an illustration. At a high level, this procedure resembles coordinate descent in optimization, with coordinates selected according to a Markov chain (Sun et al., 2020). In addition, although the initial estimate Q 0 Q_0 Q 0 can be chosen arbitrarily, we often set Q 0 ( s , a ) = 0 Q_0(s,a)=0 Q 0 ( s , a ) = 0 for all ( s , a ) (s,a) ( s , a ) unless otherwise noted. The complete asynchronous Q-learning algorithm is summarized in Algorithm 4.3.
Assumptions and key quantities
In contrast to the generative model setting — where one can obtain the same number of samples across all state–action pairs — it is generally unclear whether a Markovian sample trajectory will visit every state–action pair sufficiently often. Since insufficient visitation may result in a lack of critical information, we impose a standard assumption and introduce several key quantities, commonly used in the analysis of asynchronous Q-learning, to ensure adequate coverage of the state–action space by the sample trajectory.
Recall that when the behavior policy π b \pi^{\mathrm{b}} π b is stationary, the sequence { ( s t , a t ) } t = 0 ∞ \{(s_t,a_t)\}_{t=0}^{\infty} {( s t , a t ) } t = 0 ∞ forms a time-homogeneous Markov chain; in what follows, we denote by μ π b \mu_{\pi^{\mathrm{b}}} μ π b its stationary distribution. The following assumption concerns the uniform ergodicity of this chain.
Assumption 4.1. The Markov chain { ( s t , a t ) } t = 0 ∞ \{(s_t,a_t)\}_{t=0}^{\infty} {( s t , a t ) } t = 0 ∞ induced by the behavior policy π b \pi^{\mathrm{b}} π b (see (266) ) is assumed to be uniformly ergodic.
At a high level, uniform ergodicity ensures that, regardless of the initial state, the Markov chain eventually “forgets” its starting point and converges to the stationary distribution μ π b \mu_{\pi^{\mathrm{b}}} μ π b at a geometric rate. Consequently, aside from an initial burn-in period, the state–action visitation frequencies can be well approximated by the stationary distribution μ π b \mu_{\pi^{\mathrm{b}}} μ π b . See Paulin (2015, Section 1.2) for a formal definition of uniform ergodicity and its further implications.
With uniform ergodicity in place, two key quantities characterize, respectively, the information bottleneck in the stationary distribution and the rate of convergence toward it, both of which are critical in determining the performance of asynchronous Q Q Q -learning.
μ min : = min ( s , a ) ∈ S × A μ π b ( s , a ) . (269) \mu_{\min} := \min_{(s,a)\in\mathcal{S}\times\mathcal{A}} \mu_{\pi^{\mathrm{b}}}(s,a). \tag*{(269)} μ m i n := ( s , a ) ∈ S × A min μ π b ( s , a ) . (269)
This metric captures the information bottleneck induced by the least-visited state-action pair along the trajectory. In particular, a smaller μ min \mu_{\min} μ m i n indicates poorer coverage of the state-action space and, consequently, a higher chance that certain state-action pairs are insufficiently visited, which can adversely affect learning performance.
t m i x : = min { t ∣ max ( s , a ) ∈ S × A TV ( P t ( ⋅ ∣ s , a ) , μ π b ) ≤ 1 4 } . (270) t_{\mathrm{mix}} := \min\left\{t \mid\max_{(s,a)\in\mathcal{S}\times\mathcal{A}} \operatorname{TV}\left(\mathbb{P}^{t}(\cdot\mid s,a),\mu_{\pi^{\mathrm{b}}}\right) \le\frac{1}{4}\right\}. \tag*{(270)} t mix := min { t ∣ ( s , a ) ∈ S × A max TV ( P t ( ⋅ ∣ s , a ) , μ π b ) ≤ 4 1 } . (270)
where we recall that TV ( ⋅ , ⋅ ) \operatorname{TV}(\cdot,\cdot) TV ( ⋅ , ⋅ ) indicates the total-variation distance between two probability measures. Here, P t ( ⋅ ∣ s , a ) \mathbb{P}^{t}(\cdot\mid s,a) P t ( ⋅ ∣ s , a ) represents the distribution of ( s t , a t ) (s_t,a_t) ( s t , a t ) when the sample trajectory is initialized at ( s 0 , a 0 ) = ( s , a ) (s_0,a_0)=(s,a) ( s 0 , a 0 ) = ( s , a ) . In words, the mixing time characterizes how quickly the Markov chain can “forget” its initial state, quantifying (the order of) the time required for the distribution of the trajectory to approach its stationary distribution.
Theoretical guarantees
With the above assumption and key quantities in place, we are now ready to present the state-of-the-art theoretical guarantees for asynchronous Q Q Q -learning, originally established by Li et al., 2024a. For simplicity, the theorem below adopts a time-invariant learning rate schedule given by
η t ≡ η = c 1 log 3 T ( 1 − γ ) T μ min , 0 ≤ t ≤ T (271) \eta_t \equiv\eta= \frac{c_1\log^3 T}{(1-\gamma)T\mu_{\min}}, \qquad0 \le t \le T \tag*{(271)} η t ≡ η = ( 1 − γ ) T μ m i n c 1 log 3 T , 0 ≤ t ≤ T (271)
for some universal constants $0 < c_1 \le 1.
Theorem 4.12 (Li et al. (2024a)). Consider any δ ∈ ( 0 , 1 ) \delta\in(0,1) δ ∈ ( 0 , 1 ) , ϵ ∈ ( 0 , 1 ] \epsilon\in(0,1] ϵ ∈ ( 0 , 1 ] , and γ ∈ [ 1 / 2 , 1 ) \gamma\in[1/2,1) γ ∈ [ 1/2 , 1 ) . Suppose that Assumption 4.1 holds, and that the total number of iterations T T T exceeds
T ≥ c 2 log 2 ( S A T δ ) μ min max { log 3 T ( 1 − γ ) 4 ϵ 2 , t m i x 1 − γ } (272) T \ge\frac{c_2 \log^2\left(\frac{SAT}{\delta}\right)}{\mu_{\min}} \max\left\{\frac{\log^3 T}{(1-\gamma)^4\epsilon^2},\frac{t_{\mathrm{mix}}}{1-\gamma}\right\} \tag*{(272)} T ≥ μ m i n c 2 log 2 ( δ S A T ) max { ( 1 − γ ) 4 ϵ 2 log 3 T , 1 − γ t mix } (272) for some sufficiently large universal constant c 2 > 0 c_2>0 c 2 > 0 . If the initialization obeys 0 ≤ Q 0 ( s , a ) ≤ 1 1 − γ 0 \le Q_0(s,a) \le\frac{1}{1-\gamma} 0 ≤ Q 0 ( s , a ) ≤ 1 − γ 1 for all ( s , a ) ∈ S × A (s,a) \in\mathcal{S} \times\mathcal{A} ( s , a ) ∈ S × A , then asynchronous Q Q Q -learning (cf. Algorithm 4.3) with learning rates (271) achieves
max ( s , a ) ∈ S × A ∣ Q T ( s , a ) − Q ⋆ ( s , a ) ∣ ≤ ϵ \max_{(s,a)\in\mathcal{S}\times\mathcal{A}} \left|Q_T(s,a)-Q^\star(s,a)\right| \le\epsilon ( s , a ) ∈ S × A max ∣ Q T ( s , a ) − Q ⋆ ( s , a ) ∣ ≤ ϵ with probability at least 1 − δ 1-\delta 1 − δ .
Remark 4.3. As before, this high-probability result can be readily translated into a bound on the expected ℓ ∞ \ell_\infty ℓ ∞ -based estimation error with essentially the same sample complexity. We omit the details here and refer the reader to Li et al. (2024a) for details.
Given that each iteration of asynchronous Q Q Q -learning uses a single sample transition, Theorem 4.12 demonstrates that, with high probability, the total sample size needed for asynchronous Q Q Q -learning to yield entrywise ϵ \epsilon ϵ accuracy is
O ~ ( 1 μ min ( 1 − γ ) 4 ϵ 2 + t m i x μ min ( 1 − γ ) ) (273) \widetilde{O}\left(\frac{1}{\mu_{\min}(1-\gamma)^4\epsilon^2}+\frac{t_{\mathrm{mix}}}{\mu_{\min}(1-\gamma)}\right) \tag*{(273)} O ( μ m i n ( 1 − γ ) 4 ϵ 2 1 + μ m i n ( 1 − γ ) t mix ) (273)
provided that the learning rates are suitably chosen (see (271) ). We next provide several remarks on the two terms appearing in this bound.
Dependency on the minimum state-action occupancy. The first term in (273) mirrors the sample complexity of synchronous Q Q Q -learning. More specifically, recall that synchronous Q Q Q -learning achieves a sample complexity of O ~ ( S A ( 1 − γ ) 4 ϵ 2 ) \widetilde{O}\left(\frac{SA}{(1-\gamma)^4\epsilon^2}\right) O ( ( 1 − γ ) 4 ϵ 2 S A ) under a generative model (cf. (218) ). In comparison, the key distinction here is that the total number of state–action pairs (i.e., S A SA S A ) is replaced by 1 / μ min 1/\mu_{\min} 1/ μ m i n for asynchronous Q Q Q -learning, in order to account for the non-uniform visitation frequencies induced by the Markovian sampling process. Moreover, when the target accuracy ϵ \epsilon ϵ is sufficiently small, this term becomes the dominant contribution to the sample complexity of asynchronous Q Q Q -learning. The sharpness of this term has been further confirmed by a matching algorithm-dependent lower bound established in Li et al. (2024a, Theorem 5).
Dependency on the mixing time. The second term in (273) is nearly independent of the target accuracy (up to logarithmic factors) and can be interpreted as a burn-in time. Roughly speaking, it reflects the time required for asynchronous Q Q Q -learning to effectively emulate its synchronous counterpart despite the temporal dependence inherent in Markovian data. Notably, the mixing time of the Markov chain seems to only influence the length of this burn-in phase and becomes largely immaterial once the trajectory is sufficiently close to its stationary distribution.
For the sake of brevity, we omit the full proof of Theorem 4.12 , and refer to the synchronous and asynchronous analyses in Li et al. (2024a). Instead, we highlight a key observation that bridges the analysis of asynchronous and synchronous Q Q Q -learning. Specifically, over any time window of length T 0 = Θ ~ ( t m i x μ min ) T_0 = \widetilde{\Theta}\left(\frac{t_{\mathrm{mix}}}{\mu_{\min}}\right) T 0 = Θ ( μ m i n t mix ) , classical concentration inequalities for Markov chains (e.g., Paulin (2015, Theorem 3.14)) guarantee that each state-action pair ( s , a ) (s, a) ( s , a ) is visited at least on the order of
T 0 μ π b ( s , a ) ≥ T 0 μ min = Θ ~ ( t m i x ) T_0\mu_{\pi_b}(s, a) \ge T_0\mu_{\min} = \widetilde{\Theta}(t_{\mathrm{mix}}) T 0 μ π b ( s , a ) ≥ T 0 μ m i n = Θ ( t mix )
times with high probability. Consequently, it can be anticipated that within each such window of length Θ ~ ( t m i x μ min ) \widetilde{\Theta}\left(\frac{t_{\mathrm{mix}}}{\mu_{\min}}\right) Θ ( μ m i n t mix ) , the progress made by asynchronous Q-learning is at least comparable to — if not exceeding — that of Θ ~ ( t m i x ) \widetilde{\Theta}(t_{\mathrm{mix}}) Θ ( t mix ) iterations of synchronous Q-learning. This observation suggests a natural coupling between the two algorithms. When the total number of iterations T T T is large enough, the trajectory of asynchronous Q-learning can be partitioned into windows of length Θ ~ ( t m i x μ min ) \widetilde{\Theta}\left(\frac{t_{\mathrm{mix}}}{\mu_{\min}}\right) Θ ( μ m i n t mix ) , each of which can be compared to Θ ~ ( t m i x ) \widetilde{\Theta}(t_{\mathrm{mix}}) Θ ( t mix ) iterations of synchronous Q-learning. Once this correspondence is established, the performance guarantees for asynchronous Q-learning follow by leveraging the analysis developed for the synchronous setting. In particular, since synchronous Q-learning requires O ~ ( 1 ( 1 − γ ) 4 ε 2 ) \widetilde{O}\left(\frac{1}{(1-\gamma)^4\varepsilon^2}\right) O ( ( 1 − γ ) 4 ε 2 1 ) iterations to reach ε \varepsilon ε -accuracy, this argument suggests that asynchronous Q-learning requires O ~ ( 1 μ min ( 1 − γ ) 4 ε 2 ) \widetilde{O}\left(\frac{1}{\mu_{\min}(1-\gamma)^4\varepsilon^2}\right) O ( μ m i n ( 1 − γ ) 4 ε 2 1 ) iterations to attain the same level of accuracy, provided that T T T is large enough. We refer the interested reader to Li et al. (2024a) for the complete proof.
Before concluding this section, we briefly mention several minor extensions of the above results. First, careful readers may observe that the learning rate schedule in (271) requires prior knowledge of the stopping time T T T and, more importantly, the minimum state–action occupancy probability μ min \mu_{\min} μ m i n . Fortunately, this requirement can be eliminated by adopting a more data-driven, adaptive learning rate schedule without sacrificing performance; see Li et al. (2022b, Section III.D) for details. Moreover, it is worth emphasizing that “mixing” is not strictly necessary for the success of asynchronous Q-learning. The analysis can be readily extended to settings where the Markov chain underlying the sample trajectory does not mix but instead admits a finite cover time — namely, that within any time window of length exceeding t c o v e r t_{\mathrm{cover}} t cover , the event that all state–action pairs are visited at least once occurs with probability at least 1 / 2 1/2 1/2 . In this case, the analysis proceeds by partitioning the dynamics of asynchronous Q-learning into time windows of length Θ ~ ( t c o v e r ) \widetilde{\Theta}(t_{\mathrm{cover}}) Θ ( t cover ) (rather than those determined by the mixing time) and analyzing each window separately through comparisons with the synchronous counterpart. We refer the reader to Li et al. (2022b, Theorem 2) and the accompanying discussion therein for further details.
Extension: finite-horizon nonstationary MDPs
We now briefly discuss synchronous Q-learning for finite-horizon nonstationary MDPs. Both the algorithm and its analysis closely parallel those in the discounted infinite-horizon setting. We therefore omit the details here and refer interested readers to Li et al. (2021a) for a complete treatment.
In a nutshell, the algorithm maintains a sequence of Q-function estimates Q t , h : S × A → R Q_{t,h}:\mathcal{S}\times\mathcal{A}\to\mathbb{R} Q t , h : S × A → R for each iteration t t t and time step h h h . At each iteration t t t , the algorithm updates these estimates backward from h = H h=H h = H down to 1 via
Q t , h = ( 1 − η t ) Q t − 1 , h + η t T t , h ( Q t , h + 1 ) , (274) Q_{t,h} = (1-\eta_t)Q_{t-1,h}+\eta_t\mathcal{T}_{t,h}(Q_{t,h+1}), \tag*{(274)} Q t , h = ( 1 − η t ) Q t − 1 , h + η t T t , h ( Q t , h + 1 ) , (274)
where 0 < η t < 1 0<\eta_t<1 0 < η t < 1 denotes the learning rate for the t t t -th iteration, and T t , h \mathcal{T}_{t,h} T t , h is the empirical Bellman operator constructed from samples drawn from the generative model at step h h h of this iteration. More concretely, T t , h \mathcal{T}_{t,h} T t , h is defined as follows:
for each tuple ( s , a , h ) ∈ S × A × [ H ] (s,a,h) \in\mathcal{S} \times\mathcal{A} \times[H] ( s , a , h ) ∈ S × A × [ H ] , draw an independent next-state sample
s t , h ( s , a ) ∼ i n d . P h ( ⋅ ∣ s , a ) ; s_{t,h}(s,a) \mathrel{\mathop{\sim}^{\mathrm{ind.}}} P_h(\cdot\mid s,a); s t , h ( s , a ) ∼ ind. P h ( ⋅ ∣ s , a ) ;
T t , h ( Q t , h + 1 ) ( s , a ) : = r h ( s , a ) + max a ′ ∈ A Q t , h + 1 ( s t , h ( s , a ) , a ′ ) . \mathcal{T}_{t,h}(Q_{t,h+1})(s,a) := r_h(s,a) + \max_{a' \in\mathcal{A}} Q_{t,h+1}(s_{t,h}(s,a),a'). T t , h ( Q t , h + 1 ) ( s , a ) := r h ( s , a ) + a ′ ∈ A max Q t , h + 1 ( s t , h ( s , a ) , a ′ ) .
In addition, the value function estimate V t , h : S → R V_{t,h} : \mathcal{S} \to\mathbb{R} V t , h : S → R associated with the t t t -th iterate and step h h h is defined as
∀ s ∈ S : V t , h ( s ) : = max a ∈ A Q t , h ( s , a ) . (275) \forall s \in\mathcal{S} : \qquad V_{t,h}(s) := \max_{a \in\mathcal{A}} Q_{t,h}(s,a). \tag*{(275)} ∀ s ∈ S : V t , h ( s ) := a ∈ A max Q t , h ( s , a ) . (275)
To ensure the desired convergence guarantees, we also assume that the learning rates satisfy
1 1 + c 1 T H log 2 T ≤ η t ≤ 1 1 + c 2 t H log 2 T , 0 ≤ t ≤ T (276) \frac{1}{1 + \frac{c_1 T}{H \log^2 T}} \leq\eta_t \leq\frac{1}{1 + \frac{c_2 t}{H \log^2 T}}, \qquad0 \leq t \leq T \tag*{(276)} 1 + H l o g 2 T c 1 T 1 ≤ η t ≤ 1 + H l o g 2 T c 2 t 1 , 0 ≤ t ≤ T (276)
for sufficiently small universal constants c 1 ≥ c 2 > 0 c_1 \geq c_2 > 0 c 1 ≥ c 2 > 0 .
The next theorem establishes the ℓ ∞ \ell_\infty ℓ ∞ -based sample complexity of this synchronous Q-learning algorithm for finite-horizon MDPs.
Theorem 4.13 (Li et al. (2021a)). Consider any δ ∈ ( 0 , 1 ) \delta\in(0,1) δ ∈ ( 0 , 1 ) and ε ∈ ( 0 , 1 ] \varepsilon\in(0,1] ε ∈ ( 0 , 1 ] . Suppose that the learning rates obey (276) and the total number of iterations obeys
T ≥ c 3 H 4 ( log 3 T ) ( log S A T δ ) ε 2 (277) T \geq\frac{c_3 H^4 (\log^3 T) \left(\log\frac{SA T}{\delta}\right)}{\varepsilon^2} \tag*{(277)} T ≥ ε 2 c 3 H 4 ( log 3 T ) ( log δ S A T ) (277) for some sufficiently large universal constant c 3 > 0 c_3 > 0 c 3 > 0 . If the initialization obeys 0 ≤ Q 0 , h ( s , a ) ≤ H + 1 − h 0 \leq Q_{0,h}(s,a) \leq H + 1 - h 0 ≤ Q 0 , h ( s , a ) ≤ H + 1 − h for every ( s , a , h ) ∈ S × A × [ H ] (s,a,h) \in\mathcal{S} \times\mathcal{A} \times[H] ( s , a , h ) ∈ S × A × [ H ] , then the aforementioned synchronous Q-learning algorithm achieves
∀ ( s , a , h ) ∈ S × A × [ H ] : ∣ Q T , h ( s , a ) − Q h ∗ ( s , a ) ∣ ≤ ε \forall(s,a,h) \in\mathcal{S} \times\mathcal{A} \times[H] : \qquad\lvert Q_{T,h}(s,a) - Q_h^*(s,a) \rvert\leq\varepsilon ∀ ( s , a , h ) ∈ S × A × [ H ] : ∣ Q T , h ( s , a ) − Q h ∗ ( s , a )∣ ≤ ε with probability greater than $1-\delta.
Recalling that each iteration collects S A H SAH S A H independent samples, Theorem 4.13 indicates that synchronous Q-learning achieves a sample complexity of
O ~ ( S A H 5 ε 2 ) . \widetilde{O}\left(\frac{SAH^5}{\varepsilon^2}\right). O ( ε 2 S A H 5 ) .
for finite-horizon nonstationary MDPs. Notably, this sample complexity bound is a factor of H H H larger than the minimax-optimal complexity (167) attained by the model-based approach in § 5.1 , revealing a suboptimality gap for Q-learning analogous to that observed in the discounted infinite-horizon setting.
Appendix A: proof of auxiliary lemmas in Section 4.2
Proof of Lemma 4.2
We would like to apply induction to establish that, for any 0 ≤ t ≤ T 0 \leq t \leq T 0 ≤ t ≤ T ,
0 ≤ V t ≤ 1 1 − γ 1 S . (278) 0 \leq V_t \leq\frac{1}{1-\gamma}\mathbf{1}_{\mathcal{S}}. \tag*{(278)} 0 ≤ V t ≤ 1 − γ 1 1 S . (278)
Regarding the base case with t = 0 t = 0 t = 0 , this property is automatically guaranteed by our assumption. Next, suppose that (278) holds true for the ( t − 1 ) (t - 1) ( t − 1 ) -th iteration. Recalling that 0 ≤ η t ≤ 1 0 \leq\eta_t \leq1 0 ≤ η t ≤ 1 , 0 ≤ r π ≤ 1 S 0 \leq r_\pi\leq\mathbf{1}_S 0 ≤ r π ≤ 1 S , P t ≥ 0 P_t \geq0 P t ≥ 0 and ∥ P t ∥ 1 = 1 \lVert P_t\rVert_1 = 1 ∥ P t ∥ 1 = 1 , one can invoke the update rule (179) to show that
V t = ( 1 − η t ) V t − 1 + η t ( r π + γ P t V t − 1 ) ≥ 0 V_t = (1 - \eta_t)V_{t-1} + \eta_t(r_\pi+ \gamma P_tV_{t-1}) \geq0 V t = ( 1 − η t ) V t − 1 + η t ( r π + γ P t V t − 1 ) ≥ 0
and
V t = ( 1 − η t ) V t − 1 + η t ( r π + γ P t V t − 1 ) ≤ ( 1 − η t ) ∥ V t − 1 ∥ ∞ 1 S + η t ( ∥ r π ∥ ∞ + γ ∥ P t ∥ 1 ∥ V t − 1 ∥ ∞ ) 1 S ≤ ( 1 − η t ) 1 1 − γ 1 S + η t ( 1 + γ 1 − γ ) 1 S = 1 1 − γ 1 S . \begin{aligned}
V_t &= (1 - \eta_t)V_{t-1} + \eta_t(r_\pi+ \gamma P_tV_{t-1})\\
&\leq(1 - \eta_t)\lVert V_{t-1}\rVert_\infty\mathbf{1}_S + \eta_t(\lVert r_\pi\rVert_\infty+ \gamma\lVert P_t\rVert_1\lVert V_{t-1}\rVert_\infty)\mathbf{1}_S\\
&\leq(1 - \eta_t)\frac{1}{1-\gamma}\mathbf{1}_S + \eta_t\left(1 + \frac{\gamma}{1-\gamma}\right)\mathbf{1}_S = \frac{1}{1-\gamma}\mathbf{1}_S.
\end{aligned} V t = ( 1 − η t ) V t − 1 + η t ( r π + γ P t V t − 1 ) ≤ ( 1 − η t ) ∥ V t − 1 ∥ ∞ 1 S + η t (∥ r π ∥ ∞ + γ ∥ P t ∥ 1 ∥ V t − 1 ∥ ∞ ) 1 S ≤ ( 1 − η t ) 1 − γ 1 1 S + η t ( 1 + 1 − γ γ ) 1 S = 1 − γ 1 1 S .
This establishes (278) for the t t t -th iteration, which completes the induction argument for proving (278) . Moreover, taking together (278) and the fact that 0 ≤ V π ≤ 1 1 − γ 1 S 0 \leq V^\pi\leq\frac{1}{1-\gamma}\mathbf{1}_S 0 ≤ V π ≤ 1 − γ 1 1 S immediately reveals that
∥ V t − V π ∥ ∞ ≤ 1 1 − γ . \lVert V_t - V^\pi\rVert_\infty\leq\frac{1}{1-\gamma}. ∥ V t − V π ∥ ∞ ≤ 1 − γ 1 .
Proof of Lemma 4.3
Suppose that t ≥ T / ( c 2 log T ) t \geq T/(c_2\log T) t ≥ T / ( c 2 log T ) . We divide into two cases.
η k ( t ) ≤ ∏ i = t / 2 + 1 t ( 1 − η i ( 1 − γ ) ) ≤ (i) ( 1 − log 2 T 2 c 1 T ) t / 2 ≤ (ii) ( 1 − log 2 T 2 c 1 T ) T 2 c 2 log T = { ( 1 − log 2 T 2 c 1 T ) 2 c 1 T log 2 T } log T 4 c 1 c 2 ≤ (iii) 1 T 2 . (279) \begin{aligned}
\eta_k^{(t)} &\leq\prod_{i=t/2+1}^{t}(1-\eta_i(1-\gamma)) \mathrel{\overset{\text{(i)}}{\leq}} \left(1-\frac{\log^2 T}{2c_1T}\right)^{t/2} \mathrel{\overset{\text{(ii)}}{\leq}} \left(1-\frac{\log^2 T}{2c_1T}\right)^{\frac{T}{2c_2\log T}}\\
&= \left\{\left(1-\frac{\log^2 T}{2c_1T}\right)^{\frac{2c_1T}{\log^2 T}}\right\}^{\frac{\log T}{4c_1c_2}} \mathrel{\overset{\text{(iii)}}{\leq}} \frac{1}{T^2}. \tag*{(279)}
\end{aligned} η k ( t ) ≤ i = t /2 + 1 ∏ t ( 1 − η i ( 1 − γ )) ≤ (i) ( 1 − 2 c 1 T log 2 T ) t /2 ≤ (ii) ( 1 − 2 c 1 T log 2 T ) 2 c 2 l o g T T = ⎩ ⎨ ⎧ ( 1 − 2 c 1 T log 2 T ) l o g 2 T 2 c 1 T ⎭ ⎬ ⎫ 4 c 1 c 2 l o g T ≤ (iii) T 2 1 . (279)
Here, (i) follows from a direct consequence of (176) , i.e.,
( 1 − γ ) η i ≥ 1 − γ 1 + c 1 ( 1 − γ ) T log 2 T ≥ 1 − γ 2 c 1 ( 1 − γ ) T / log 2 T = log 2 T 2 c 1 T (1-\gamma)\eta_i \geq\frac{1-\gamma}{1+\frac{c_1(1-\gamma)T}{\log^2 T}} \geq\frac{1-\gamma}{2c_1(1-\gamma)T/\log^2 T} = \frac{\log^2 T}{2c_1T} ( 1 − γ ) η i ≥ 1 + l o g 2 T c 1 ( 1 − γ ) T 1 − γ ≥ 2 c 1 ( 1 − γ ) T / log 2 T 1 − γ = 2 c 1 T log 2 T
as long as T ≥ log 2 T c 1 ( 1 − γ ) T \geq\frac{\log^2 T}{c_1(1-\gamma)} T ≥ c 1 ( 1 − γ ) l o g 2 T (as ensured by (177) and ε ≤ 1 / ( 1 − γ ) \varepsilon\leq1/(1-\gamma) ε ≤ 1/ ( 1 − γ ) for sufficiently large c 3 c_3 c 3 ); (ii) is valid provided that t ≥ T / ( c 2 log T ) t \geq T/(c_2\log T) t ≥ T / ( c 2 log T ) ; and (iii) holds as long as c 1 c 2 ≤ 1 / 8 c_1c_2 \leq1/8 c 1 c 2 ≤ 1/8 and T ≥ c 1 − 1 log 2 T T \geq c_1^{-1}\log^2 T T ≥ c 1 − 1 log 2 T (again ensured by (177) ).
η k ( t ) ≤ η k ≤ 1 c 2 ( 1 − γ ) k / log 2 T < 2 log 3 T ( 1 − γ ) T . (280) \eta_k^{(t)} \leq\eta_k \leq\frac{1}{c_2(1-\gamma)k/\log^2 T} < \frac{2\log^3 T}{(1-\gamma)T}. \tag*{(280)} η k ( t ) ≤ η k ≤ c 2 ( 1 − γ ) k / log 2 T 1 < ( 1 − γ ) T 2 log 3 T . (280)
where the last inequality holds since k > t / 2 ≥ T / ( 2 c 2 log T ) k > t/2 \geq T/(2c_2\log T) k > t /2 ≥ T / ( 2 c 2 log T ) .
Proof of Lemma 4.4
For each state s s s , the TD update gives
V k ( s ) = ( 1 − η k ) V k − 1 ( s ) + η k ( r π ( s ) + γ ( P k V k − 1 ) ( s ) ) . V_k(s) = (1-\eta_k)V_{k-1}(s) + \eta_k\left(r_\pi(s) + \gamma(P_kV_{k-1})(s)\right). V k ( s ) = ( 1 − η k ) V k − 1 ( s ) + η k ( r π ( s ) + γ ( P k V k − 1 ) ( s ) ) .
Taking conditional expectations gives
E k − 1 [ V k ∘ V k ] = ( 1 − η k ) V k − 1 ∘ V k − 1 + η k E k − 1 [ ( r π + γ P k V k − 1 ) ∘ ( r π + γ P k V k − 1 ) ] \mathbb{E}_{k-1}[V_k \circ V_k] = (1-\eta_k)V_{k-1}\circ V_{k-1} + \eta_k\mathbb{E}_{k-1}\left[(r_\pi+\gamma P_kV_{k-1})\circ(r_\pi+\gamma P_kV_{k-1})\right] E k − 1 [ V k ∘ V k ] = ( 1 − η k ) V k − 1 ∘ V k − 1 + η k E k − 1 [ ( r π + γ P k V k − 1 ) ∘ ( r π + γ P k V k − 1 ) ]
− η k ( 1 − η k ) E k − 1 [ ( r π + γ P k V k − 1 − V k − 1 ) ∘ ( r π + γ P k V k − 1 − V k − 1 ) ] , (281) \quad-\eta_k(1-\eta_k)\mathbb{E}_{k-1}\left[(r_\pi+\gamma P_kV_{k-1}-V_{k-1})\circ(r_\pi+\gamma P_kV_{k-1}-V_{k-1})\right], \tag*{(281)} − η k ( 1 − η k ) E k − 1 [ ( r π + γ P k V k − 1 − V k − 1 ) ∘ ( r π + γ P k V k − 1 − V k − 1 ) ] , (281)
where we have used the elementary identity
( ( 1 − η ) x + η y ) 2 = ( 1 − η ) x 2 + η y 2 − η ( 1 − η ) ( y − x ) 2 . ((1-\eta)x+\eta y)^2=(1-\eta)x^2+\eta y^2-\eta(1-\eta)(y-x)^2. (( 1 − η ) x + η y ) 2 = ( 1 − η ) x 2 + η y 2 − η ( 1 − η ) ( y − x ) 2 .
Since r π ≤ 1 S r_\pi\leq1_{\mathcal{S}} r π ≤ 1 S and V k − 1 ≤ ( 1 − γ ) − 1 1 S V_{k-1}\leq(1-\gamma)^{-1}1_{\mathcal{S}} V k − 1 ≤ ( 1 − γ ) − 1 1 S , we have
E k − 1 [ ( r π + γ P k V k − 1 ) ∘ ( r π + γ P k V k − 1 ) ] = r π ∘ r π + 2 γ r π ∘ ( P π V k − 1 ) + γ 2 P π ( V k − 1 ∘ V k − 1 ) ≤ γ P π ( V k − 1 ∘ V k − 1 ) + 2 1 − γ 1 S , \begin{aligned}
\mathbb{E}_{k-1}\left[(r_\pi+\gamma P_kV_{k-1})\circ(r_\pi+\gamma P_kV_{k-1})\right]
=r_\pi\circ r_\pi+2\gamma r_\pi\circ(P_\pi V_{k-1})+\gamma^2P_\pi(V_{k-1}\circ V_{k-1}) \\
&\leq\gamma P_\pi(V_{k-1}\circ V_{k-1})+\frac{2}{1-\gamma}1_{\mathcal{S}},
\end{aligned} E k − 1 [ ( r π + γ P k V k − 1 ) ∘ ( r π + γ P k V k − 1 ) ] = r π ∘ r π + 2 γ r π ∘ ( P π V k − 1 ) + γ 2 P π ( V k − 1 ∘ V k − 1 ) ≤ γ P π ( V k − 1 ∘ V k − 1 ) + 1 − γ 2 1 S ,
and
E k − 1 [ ( r π + γ P k V k − 1 − V k − 1 ) ∘ ( r π + γ P k V k − 1 − V k − 1 ) ] ≥ γ 2 Var P π ( V k − 1 ) . \mathbb{E}_{k-1}\left[(r_\pi+\gamma P_kV_{k-1}-V_{k-1})\circ(r_\pi+\gamma P_kV_{k-1}-V_{k-1})\right]\geq\gamma^2\operatorname{Var}_{P_\pi}(V_{k-1}). E k − 1 [ ( r π + γ P k V k − 1 − V k − 1 ) ∘ ( r π + γ P k V k − 1 − V k − 1 ) ] ≥ γ 2 Var P π ( V k − 1 ) .
Substituting these bounds into (281) and rearranging terms, we conclude the proof.
Proof of Lemma 4.5
We intend to apply Freedman’s inequality to the martingale sequence { Z ~ k } \{\widetilde{Z}_k\} { Z k } defined in (206) . Towards this end, we first bound the magnitude and cumulative conditional variance of Z ~ k \widetilde{Z}_k Z k .
We start with bounding the magnitude max k : t / 2 < k ≤ t ∥ Z ~ k ∥ ∞ \max_{k:t/2<k\leq t}\|\widetilde{Z}_k\|_\infty max k : t /2 < k ≤ t ∥ Z k ∥ ∞ . Since conditionally on the past, each coordinate of V k V_k V k lies in an interval of length at most η k γ / ( 1 − γ ) \eta_k\gamma/(1-\gamma) η k γ / ( 1 − γ ) , and the squaring operator is 2 / ( 1 − γ ) 2/(1-\gamma) 2/ ( 1 − γ ) -Lipschitz on [ 0 , 1 / ( 1 − γ ) ] [0,1/(1-\gamma)] [ 0 , 1/ ( 1 − γ )] , it follows that
∥ V k ∘ V k − E k − 1 [ V k ∘ V k ] ∥ ∞ ≤ 2 η k ( 1 − γ ) 2 ≤ 2 η ˉ ( 1 − γ ) 2 , \|V_k\circ V_k-\mathbb{E}_{k-1}[V_k\circ V_k]\|_\infty\leq\frac{2\eta_k}{(1-\gamma)^2}\leq\frac{2\bar{\eta}}{(1-\gamma)^2}, ∥ V k ∘ V k − E k − 1 [ V k ∘ V k ] ∥ ∞ ≤ ( 1 − γ ) 2 2 η k ≤ ( 1 − γ ) 2 2 η ˉ ,
where we recall the definition of η ˉ \bar{\eta} η ˉ in (202) . This gives immediately
max k : t / 2 < k ≤ t ∥ Z ~ k ∥ ∞ ≤ 2 η ˉ ( 1 − γ ) 2 . \max_{k:t/2<k\leq t}\|\widetilde{Z}_k\|_\infty\leq\frac{2\bar{\eta}}{(1-\gamma)^2}. k : t /2 < k ≤ t max ∥ Z k ∥ ∞ ≤ ( 1 − γ ) 2 2 η ˉ .
We now move to the cumulative conditional variance ∑ k Var k − 1 ( Z ~ k ) \sum_k\operatorname{Var}_{k-1}(\widetilde{Z}_k) ∑ k Var k − 1 ( Z k ) . First, it is well known that Var ( X 2 ) ≤ 4 C max 2 Var ( X ) \operatorname{Var}(X^2)\leq4C_{\max}^2\operatorname{Var}(X) Var ( X 2 ) ≤ 4 C m a x 2 Var ( X ) for any random variable X X X obeying ∣ X ∣ ≤ C max |X|\leq C_{\max} ∣ X ∣ ≤ C m a x (see, e.g., Chen et al. (2021b, Lemma 30)), and hence
Var k − 1 ( V k ∘ V k ) ≤ 4 ( 1 − γ ) 2 Var k − 1 ( V k ) ≤ 4 η k 2 γ 2 ( 1 − γ ) 2 Var P π ( V k − 1 ) , \operatorname{Var}_{k-1}(V_k\circ V_k)\leq\frac{4}{(1-\gamma)^2}\operatorname{Var}_{k-1}(V_k)\leq\frac{4\eta_k^2\gamma^2}{(1-\gamma)^2}\operatorname{Var}_{P_\pi}(V_{k-1}), Var k − 1 ( V k ∘ V k ) ≤ ( 1 − γ ) 2 4 Var k − 1 ( V k ) ≤ ( 1 − γ ) 2 4 η k 2 γ 2 Var P π ( V k − 1 ) ,
where the last line is valid since Var k − 1 ( V k ) = η k 2 γ 2 Var P π ( V k − 1 ) \operatorname{Var}_{k-1}(V_k)=\eta_k^2\gamma^2\operatorname{Var}_{P_\pi}(V_{k-1}) Var k − 1 ( V k ) = η k 2 γ 2 Var P π ( V k − 1 ) . It then follows from (193) that
Var k − 1 ( Z ~ k ) ≤ 4 η k 2 γ 2 ( 1 − γ ) 2 ∏ i = k + 1 t ( I − η i ( I − γ P π ) ) Var P π ( V k − 1 ) . \operatorname{Var}_{k-1}(\widetilde{Z}_k)\leq\frac{4\eta_k^2\gamma^2}{(1-\gamma)^2}\prod_{i=k+1}^{t}\left(I-\eta_i(I-\gamma P_\pi)\right)\operatorname{Var}_{P_\pi}(V_{k-1}). Var k − 1 ( Z k ) ≤ ( 1 − γ ) 2 4 η k 2 γ 2 i = k + 1 ∏ t ( I − η i ( I − γ P π ) ) Var P π ( V k − 1 ) .
Using η k 2 ≤ η ‾ η k \eta_k^2\leq\overline{\eta}\eta_k η k 2 ≤ η η k and the definition (203) of W ‾ t \overline{W}_t W t , we conclude that
∑ k = ⌊ t / 2 ⌋ + 1 t Var k − 1 ( Z ~ k ) ≤ 4 ( 1 − γ ) 2 W ‾ t . \sum_{k=\lfloor t/2\rfloor+1}^{t}\operatorname{Var}_{k-1}(\widetilde{Z}_k)\leq\frac{4}{(1-\gamma)^2}\overline{W}_t. k = ⌊ t /2 ⌋ + 1 ∑ t Var k − 1 ( Z k ) ≤ ( 1 − γ ) 2 4 W t .
In addition, similar to the derivation in (199) , we record a crude bound of ∥ W ‾ t ∥ ∞ ≤ η ‾ ( 1 − γ ) 3 \|\overline{W}_t\|_\infty\leq\frac{\overline{\eta}}{(1-\gamma)^3} ∥ W t ∥ ∞ ≤ ( 1 − γ ) 3 η , and hence
∥ ∑ k = ⌊ t / 2 ⌋ + 1 t Var k − 1 ( Z ~ k ) ∥ ∞ ≤ 4 η ‾ ( 1 − γ ) 5 = : σ ~ w 2 . \left\|\sum_{k=\lfloor t/2\rfloor+1}^{t}\operatorname{Var}_{k-1}(\widetilde{Z}_k)\right\|_\infty\leq\frac{4\overline{\eta}}{(1-\gamma)^5}=:\widetilde{\sigma}_w^2. k = ⌊ t /2 ⌋ + 1 ∑ t Var k − 1 ( Z k ) ∞ ≤ ( 1 − γ ) 5 4 η =: σ w 2 .
Applying Freedman's inequality in Lemma 2.9 to each coordinate of ∑ k > t / 2 Z ~ k \sum_{k>t/2}\widetilde{Z}_k ∑ k > t /2 Z k with K = 2 ⌈ log 2 1 1 − γ ⌉ K=2\left\lceil\log_2\frac{1}{1-\gamma}\right\rceil K = 2 ⌈ log 2 1 − γ 1 ⌉ chosen as in Step 2, we have σ ~ w 2 2 K ≤ 4 η ‾ ( 1 − γ ) 4 \frac{\widetilde{\sigma}_w^2}{2^K}\leq\frac{4\overline{\eta}}{(1-\gamma)^4} 2 K σ w 2 ≤ ( 1 − γ ) 4 4 η . Taking a union bound over the S S S coordinates yields
∥ ∑ k = ⌊ t / 2 ⌋ + 1 t Z ~ k ∥ ∞ ≤ 32 ( 1 − γ ) 2 ( ∥ W ‾ t ∥ ∞ + η ‾ ( 1 − γ ) 2 ) log 4 S T K δ + 8 η ‾ 3 ( 1 − γ ) 2 log 4 S T K δ \left\|\sum_{k=\lfloor t/2\rfloor+1}^{t}\widetilde{Z}_k\right\|_\infty\leq\sqrt{\frac{32}{(1-\gamma)^2}\left(\|\overline{W}_t\|_\infty+\frac{\overline{\eta}}{(1-\gamma)^2}\right)\log\frac{4STK}{\delta}}+\frac{8\overline{\eta}}{3(1-\gamma)^2}\log\frac{4STK}{\delta} k = ⌊ t /2 ⌋ + 1 ∑ t Z k ∞ ≤ ( 1 − γ ) 2 32 ( ∥ W t ∥ ∞ + ( 1 − γ ) 2 η ) log δ 4 S T K + 3 ( 1 − γ ) 2 8 η log δ 4 S T K
with probability at least 1 − δ / ( 2 T ) 1-\delta/(2T) 1 − δ / ( 2 T ) . Further, the sample-size condition (200) and ϵ ≤ 1 1 − γ \epsilon\leq\frac{1}{1-\gamma} ϵ ≤ 1 − γ 1 give
η ‾ log 4 S T K δ ≤ 6 c 3 ( 1 − γ ) 2 ϵ 2 ≤ 1 64 . (282) \overline{\eta}\log\frac{4STK}{\delta}\leq\frac{6}{c_3}(1-\gamma)^2\epsilon^2\leq\frac{1}{64}. \tag*{(282)} η log δ 4 S T K ≤ c 3 6 ( 1 − γ ) 2 ϵ 2 ≤ 64 1 . (282)
Substituting the bound into (207) , and using ( 1 − η ‾ ) − 1 ≤ 2 (1-\overline{\eta})^{-1}\leq2 ( 1 − η ) − 1 ≤ 2 and (282) , we obtain
∥ W ‾ t ∥ ∞ ≤ 13 η ‾ 2 ( 1 − γ ) 2 + 2 η ‾ ( 1 − γ ) 2 ( ∥ W ‾ t ∥ ∞ + η ‾ ( 1 − γ ) 2 ) ≤ 1 2 ∥ W ‾ t ∥ ∞ + 8 η ‾ ( 1 − γ ) 2 , \begin{aligned}
\|\overline{W}_t\|_\infty\leq\frac{13\overline{\eta}}{2(1-\gamma)^2}+\sqrt{\frac{2\overline{\eta}}{(1-\gamma)^2}\left(\|\overline{W}_t\|_\infty+\frac{\overline{\eta}}{(1-\gamma)^2}\right)} \\
&\leq\frac{1}{2}\|\overline{W}_t\|_\infty+\frac{8\overline{\eta}}{(1-\gamma)^2},
\end{aligned} ∥ W t ∥ ∞ ≤ 2 ( 1 − γ ) 2 13 η + ( 1 − γ ) 2 2 η ( ∥ W t ∥ ∞ + ( 1 − γ ) 2 η ) ≤ 2 1 ∥ W t ∥ ∞ + ( 1 − γ ) 2 8 η ,
where the last step invokes the AM-GM inequality. Solving the above relation yields
∥ W ‾ t ∥ ∞ ≤ 16 η ‾ ( 1 − γ ) 2 . \|\overline{W}_t\|_\infty\leq\frac{16\overline{\eta}}{(1-\gamma)^2}. ∥ W t ∥ ∞ ≤ ( 1 − γ ) 2 16 η .
Combining this with the bound (195) and (203) , we arrive at
∥ W t ∥ ∞ ≤ γ 2 2 ( 1 − γ ) 2 T 3 + 16 η ‾ ( 1 − γ ) 2 ≤ 33 log 3 T ( 1 − γ ) 3 T . (283) \|W_t\|_\infty\leq\frac{\gamma^2}{2(1-\gamma)^2T^3}+\frac{16\overline{\eta}}{(1-\gamma)^2}\leq\frac{33\log^3 T}{(1-\gamma)^3T}. \tag*{(283)} ∥ W t ∥ ∞ ≤ 2 ( 1 − γ ) 2 T 3 γ 2 + ( 1 − γ ) 2 16 η ≤ ( 1 − γ ) 3 T 33 log 3 T . (283)
with probability at least 1 − δ / ( 2 T ) 1-\delta/(2T) 1 − δ / ( 2 T ) .
Appendix B: proof of auxiliary lemmas in Section 4.3
Proof of Lemma 4.8
The proof is similar to that of Lemma 4.2. Specifically, we would like to show by induction that, for every t ≥ 0 t\geq0 t ≥ 0 ,
0 ≤ Q t ≤ 1 1 − γ 1 S A and 0 ≤ V t ≤ 1 1 − γ 1 S . (284) 0\leq Q_t\leq\frac{1}{1-\gamma}\mathbf{1}_{SA}\quad\text{and}\quad0\leq V_t\leq\frac{1}{1-\gamma}\mathbf{1}_S. \tag*{(284)} 0 ≤ Q t ≤ 1 − γ 1 1 S A and 0 ≤ V t ≤ 1 − γ 1 1 S . (284)
We focus on proving the bounds for Q t Q_t Q t ; once these are established, the bounds for V t V_t V t follow immediately from the relation V t ( s ) = max a Q t ( s , a ) V_t(s)=\max_a Q_t(s,a) V t ( s ) = max a Q t ( s , a ) .
The base case t = 0 t=0 t = 0 follows directly from the assumption of the lemma. Now suppose that (284) holds for the ( t − 1 ) (t-1) ( t − 1 ) -th iterate. Since 0 ≤ η t ≤ 1 0\leq\eta_t\leq1 0 ≤ η t ≤ 1 , 0 ≤ r ≤ 1 S A 0\leq r\leq1_{SA} 0 ≤ r ≤ 1 S A , P t ≥ 0 P_t\geq0 P t ≥ 0 , and each row of P t P_t P t sums to one, the update rule (220) yields
Q t = ( 1 − η t ) Q t − 1 + η t ( r + γ P t V t − 1 ) ≥ 0 Q_t=(1-\eta_t)Q_{t-1}+\eta_t\left(r+\gamma P_tV_{t-1}\right)\geq0 Q t = ( 1 − η t ) Q t − 1 + η t ( r + γ P t V t − 1 ) ≥ 0
and
Q t = ( 1 − η t ) Q t − 1 + η t ( r + γ P t V t − 1 ) ≤ ( 1 − η t ) ∥ Q t − 1 ∥ ∞ 1 S A + η t ( ∥ r ∥ ∞ + γ ∥ P t ∥ 1 ∥ V t − 1 ∥ ∞ ) 1 S A ≤ ( 1 − η t ) 1 1 − γ 1 S A + η t ( 1 + γ 1 − γ ) 1 S A = 1 1 − γ 1 S A . \begin{aligned}
Q_t&=(1-\eta_t)Q_{t-1}+\eta_t\left(r+\gamma P_tV_{t-1}\right)\\
&\leq(1-\eta_t)\lVert Q_{t-1}\rVert_\infty1_{SA}+\eta_t\left(\lVert r\rVert_\infty+\gamma\lVert P_t\rVert_1\lVert V_{t-1}\rVert_\infty\right)1_{SA}\\
&\leq(1-\eta_t)\frac{1}{1-\gamma}1_{SA}+\eta_t\left(1+\frac{\gamma}{1-\gamma}\right)1_{SA}=\frac{1}{1-\gamma}1_{SA}.
\end{aligned} Q t = ( 1 − η t ) Q t − 1 + η t ( r + γ P t V t − 1 ) ≤ ( 1 − η t ) ∥ Q t − 1 ∥ ∞ 1 S A + η t ( ∥ r ∥ ∞ + γ ∥ P t ∥ 1 ∥ V t − 1 ∥ ∞ ) 1 S A ≤ ( 1 − η t ) 1 − γ 1 1 S A + η t ( 1 + 1 − γ γ ) 1 S A = 1 − γ 1 1 S A .
This verifies (284) for the t t t -th iterate and establishes (222) . The other claim (223) follows by combining (222) with the fact that 0 ≤ Q ⋆ ≤ 1 1 − γ 1 S A 0\leq Q^\star\leq\frac{1}{1-\gamma}1_{SA} 0 ≤ Q ⋆ ≤ 1 − γ 1 1 S A .
Proof of Lemma 4.9
From assumption (216) , we have 0 ≤ η j ≤ 1 0\leq\eta_j\leq1 0 ≤ η j ≤ 1 for all 1 ≤ j ≤ T 1\leq j\leq T 1 ≤ j ≤ T . For any ⌊ T / 2 ⌋ < i ≤ t ≤ T \lfloor T/2\rfloor<i\leq t\leq T ⌊ T /2 ⌋ < i ≤ t ≤ T , we invoke the upper bound in that assumption to obtain
0 ≤ η i ( t ) ≤ η i ≤ log 3 T c 2 ( 1 − γ ) i < 2 log 3 T c 2 ( 1 − γ ) T , 0\leq\eta_i^{(t)}\leq\eta_i\leq\frac{\log^3 T}{c_2(1-\gamma)i}<\frac{2\log^3 T}{c_2(1-\gamma)T}, 0 ≤ η i ( t ) ≤ η i ≤ c 2 ( 1 − γ ) i log 3 T < c 2 ( 1 − γ ) T 2 log 3 T ,
where the first two inequalities follow from (224) and the last inequality uses i > ⌊ T / 2 ⌋ i>\lfloor T/2\rfloor i > ⌊ T /2 ⌋ . Combining this with η i ≤ 1 \eta_i\leq1 η i ≤ 1 and the definition (225) of η ‾ \overline{\eta} η verifies that 0 ≤ η i ( t ) ≤ η i ≤ η ‾ 0\leq\eta_i^{(t)}\leq\eta_i\leq\overline{\eta} 0 ≤ η i ( t ) ≤ η i ≤ η .
Moreover, a telescoping argument reveals that
∑ i = 0 t η i ( t ) = ∏ j = 1 t ( 1 − η j ) + η 1 ∏ j = 2 t ( 1 − η j ) + η 2 ∏ j = 3 t ( 1 − η j ) + ⋯ + η t − 1 ( 1 − η t ) + η t = ∏ j = 2 t ( 1 − η j ) + η 2 ∏ j = 3 t ( 1 − η j ) + ⋯ + η t − 1 ( 1 − η t ) + η t = ⋯ = ( 1 − η t ) + η t = 1. \begin{aligned}
\sum_{i=0}^{t}\eta_i^{(t)}
&=\prod_{j=1}^{t}(1-\eta_j)+\eta_1\prod_{j=2}^{t}(1-\eta_j)+\eta_2\prod_{j=3}^{t}(1-\eta_j)+\cdots+\eta_{t-1}(1-\eta_t)+\eta_t\\
&=\prod_{j=2}^{t}(1-\eta_j)+\eta_2\prod_{j=3}^{t}(1-\eta_j)+\cdots+\eta_{t-1}(1-\eta_t)+\eta_t\\
&=\cdots=(1-\eta_t)+\eta_t=1.
\end{aligned} i = 0 ∑ t η i ( t ) = j = 1 ∏ t ( 1 − η j ) + η 1 j = 2 ∏ t ( 1 − η j ) + η 2 j = 3 ∏ t ( 1 − η j ) + ⋯ + η t − 1 ( 1 − η t ) + η t = j = 2 ∏ t ( 1 − η j ) + η 2 j = 3 ∏ t ( 1 − η j ) + ⋯ + η t − 1 ( 1 − η t ) + η t = ⋯ = ( 1 − η t ) + η t = 1.
In fact, the same argument starting at any 1 ≤ τ ≤ t 1\leq\tau\leq t 1 ≤ τ ≤ t yields
∏ j = τ t ( 1 − η j ) + ∑ i = τ t η i ( t ) = ∏ j = τ + 1 t ( 1 − η j ) + ∑ i = τ + 1 t η i ( t ) = ⋯ = 1 , \prod_{j=\tau}^{t}(1-\eta_j)+\sum_{i=\tau}^{t}\eta_i^{(t)}=\prod_{j=\tau+1}^{t}(1-\eta_j)+\sum_{i=\tau+1}^{t}\eta_i^{(t)}=\cdots=1, j = τ ∏ t ( 1 − η j ) + i = τ ∑ t η i ( t ) = j = τ + 1 ∏ t ( 1 − η j ) + i = τ + 1 ∑ t η i ( t ) = ⋯ = 1 ,
where an empty product equals one and an empty sum equals zero. Rearranging this identity, and taking it together with the preceding calculation for τ = 0 \tau=0 τ = 0 , we establish (227) .
Finally, it follows from η i ( t ) ≤ η ‾ \eta_i^{(t)}\leq\overline{\eta} η i ( t ) ≤ η and (227) that
∑ i = ⌊ T / 2 ⌋ + 1 t ( η i ( t ) ) 2 ≤ η ‾ ∑ i = ⌊ T / 2 ⌋ + 1 t η i ( t ) ≤ η ‾ . \sum_{i=\lfloor T/2\rfloor+1}^{t}\left(\eta_i^{(t)}\right)^2\leq\overline{\eta}\sum_{i=\lfloor T/2\rfloor+1}^{t}\eta_i^{(t)}\leq\overline{\eta}. i = ⌊ T /2 ⌋ + 1 ∑ t ( η i ( t ) ) 2 ≤ η i = ⌊ T /2 ⌋ + 1 ∑ t η i ( t ) ≤ η .
Proof of Lemma 4.10
Proof of part (i). For any given p p p and V V V , write σ 2 : = Var p ( V ) \sigma^2 := \operatorname{Var}_p(V) σ 2 := Var p ( V ) . For every q ∈ U ρ ( p ) q \in\mathcal{U}_{\rho}(p) q ∈ U ρ ( p ) , observe that
∣ q ⊤ V − p ⊤ V ∣ = ∣ ∑ s : p ( s ) > 0 ( q ( s ) − p ( s ) ) ( V ( s ) − p ⊤ V ) ∣ ≤ ( ∑ s : p ( s ) > 0 ( q ( s ) − p ( s ) ) 2 p ( s ) ) ( ∑ s : p ( s ) > 0 p ( s ) ( V ( s ) − p ⊤ V ) 2 ) = χ 2 ( q ∥ p ) σ ≤ ρ σ . \begin{aligned}
\left|q^{\top}V-p^{\top}V\right|=\left|\sum_{s:p(s)>0}(q(s)-p(s))(V(s)-p^{\top}V)\right| \\
&\leq\sqrt{\left(\sum_{s:p(s)>0}\frac{(q(s)-p(s))^2}{p(s)}\right)\left(\sum_{s:p(s)>0}p(s)(V(s)-p^{\top}V)^2\right)} \\
&=\sqrt{\chi^2(q\mathbin{\|}p)}\sigma\leq\sqrt{\rho}\sigma.
\end{aligned} q ⊤ V − p ⊤ V = s : p ( s ) > 0 ∑ ( q ( s ) − p ( s )) ( V ( s ) − p ⊤ V ) ≤ s : p ( s ) > 0 ∑ p ( s ) ( q ( s ) − p ( s ) ) 2 s : p ( s ) > 0 ∑ p ( s ) ( V ( s ) − p ⊤ V ) 2 = χ 2 ( q ∥ p ) σ ≤ ρ σ .
where the first equality uses ∑ s q ( s ) = ∑ s p ( s ) \sum_s q(s)=\sum_s p(s) ∑ s q ( s ) = ∑ s p ( s ) as well as the support restriction in (238) , and the second relation follows from the Cauchy-Schwarz inequality. This proves the upper bound in (239) .
To establish the lower bound in (239) , suppose first that σ ≥ ρ / ( 1 − γ ) \sigma\geq\sqrt{\rho}/(1-\gamma) σ ≥ ρ / ( 1 − γ ) , and consider a specific vector q 1 q_1 q 1 as follows
q 1 ( s ) = p ( s ) ( 1 + ρ σ ( V ( s ) − p ⊤ V ) ) , ∀ s ∈ S . q_1(s)=p(s)\left(1+\frac{\sqrt{\rho}}{\sigma}(V(s)-p^{\top}V)\right),\quad\forall s\in\mathcal{S}. q 1 ( s ) = p ( s ) ( 1 + σ ρ ( V ( s ) − p ⊤ V ) ) , ∀ s ∈ S .
Since ∣ V ( s ) − p ⊤ V ∣ ≤ 1 / ( 1 − γ ) \left|V(s)-p^{\top}V\right|\leq1/(1-\gamma) V ( s ) − p ⊤ V ≤ 1/ ( 1 − γ ) , every entry of q 1 q_1 q 1 is nonnegative. Moreover, direct calculation yields
∑ s q 1 ( s ) = 1 , χ 2 ( q 1 ∥ p ) = ρ σ 2 Var p ( V ) = ρ , q 1 ⊤ V − p ⊤ V = ρ σ . \sum_s q_1(s)=1,\qquad\chi^2(q_1\mathbin{\|}p)=\frac{\rho}{\sigma^2}\operatorname{Var}_p(V)=\rho,\qquad q_1^{\top}V-p^{\top}V=\sqrt{\rho}\sigma. s ∑ q 1 ( s ) = 1 , χ 2 ( q 1 ∥ p ) = σ 2 ρ Var p ( V ) = ρ , q 1 ⊤ V − p ⊤ V = ρ σ .
Consequently, q 1 q_1 q 1 lies within U ρ ( p ) \mathcal{U}_{\rho}(p) U ρ ( p ) and attains the upper bound. If instead 0 ≤ σ < ρ / ( 1 − γ ) 0\leq\sigma<\sqrt{\rho}/(1-\gamma) 0 ≤ σ < ρ / ( 1 − γ ) , then ρ σ − ρ / ( 1 − γ ) ≤ 0 \sqrt{\rho}\sigma-\rho/(1-\gamma)\leq0 ρ σ − ρ / ( 1 − γ ) ≤ 0 , whereas the choice q 2 = p q_2=p q 2 = p already gives q 2 ⊤ V − p ⊤ V = 0 q_2^{\top}V-p^{\top}V=0 q 2 ⊤ V − p ⊤ V = 0 . This proves (239) in all cases.
Finally, applying the above arguments to − V -V − V immediately establishes
ρ Var p ( V ) − ρ 1 − γ ≤ p ⊤ V − inf q ∈ U ρ ( p ) q ⊤ V ≤ ρ Var p ( V ) . \sqrt{\rho\operatorname{Var}_p(V)}-\frac{\rho}{1-\gamma}\leq p^{\top}V-\inf_{q\in\mathcal{U}_{\rho}(p)}q^{\top}V\leq\sqrt{\rho\operatorname{Var}_p(V)}. ρ Var p ( V ) − 1 − γ ρ ≤ p ⊤ V − q ∈ U ρ ( p ) inf q ⊤ V ≤ ρ Var p ( V ) .
Proof of part (ii). Let Q π , P ′ Q^{\pi,P'} Q π , P ′ (resp. Q π , P Q^{\pi,P} Q π , P ) denote the Q Q Q -function of policy π \pi π under the transition kernel P ′ P' P ′ (resp. P P P ). Let { ( s k , a k ) } k ≥ 0 \{(s_k,a_k)\}_{k\geq0} {( s k , a k ) } k ≥ 0 represent a sample trajectory generated under policy π \pi π , and let τ \tau τ be an independent random variable with
P ( τ = k ) = ( 1 − γ ) γ k , k = 0 , 1 , … . \mathbb{P}(\tau=k)=(1-\gamma)\gamma^k,\qquad k=0,1,\ldots. P ( τ = k ) = ( 1 − γ ) γ k , k = 0 , 1 , … .
The Q Q Q -function can then be expressed as the expected reward at the random stopping time τ \tau τ :
Q π , P ( s , a ) = ∑ k = 0 ∞ γ k E P [ r ( s k , a k ) ] = 1 1 − γ E P [ r ( s τ , a τ ) ] , (285) Q^{\pi,P}(s,a)=\sum_{k=0}^{\infty}\gamma^k\mathbb{E}_P[r(s_k,a_k)]=\frac{1}{1-\gamma}\mathbb{E}_P[r(s_\tau,a_\tau)], \tag*{(285)} Q π , P ( s , a ) = k = 0 ∑ ∞ γ k E P [ r ( s k , a k )] = 1 − γ 1 E P [ r ( s τ , a τ )] , (285)
with an analogous identity holding under P ′ P' P ′ . Here and below, E P [ ⋅ ] \mathbb{E}_P[\cdot] E P [ ⋅ ] denotes expectation w.r.t. the transition kernel P P P .
To proceed, we examine the distributions of randomly stopped trajectories under different transition kernels. Conditional on τ = k \tau=k τ = k , let P k ′ \mathbb{P}'_k P k ′ and P k \mathbb{P}_k P k denote the distributions of the trajectory ( s 0 , a 0 , … , s k , a k ) (s_0,a_0,\ldots,s_k,a_k) ( s 0 , a 0 , … , s k , a k ) under P ′ P' P ′ and P P P , respectively, assuming that the initial state–action pair and the policy are the same in both cases. The chain rule for KL divergence reveals that
KL ( P k ′ ∥ P k ) = ∑ j = 0 k − 1 E P ′ [ KL ( P ′ ( ⋅ ∣ s j , a j ) ∥ P ( ⋅ ∣ s j , a j ) ) ] ≤ k ρ . (286) \operatorname{KL}(\mathbb{P}'_k\mathbin{\|}\mathbb{P}_k)=\sum_{j=0}^{k-1}\mathbb{E}_{P'}\left[\operatorname{KL}\left(P'(\cdot\mid s_j,a_j)\mathbin{\|}P(\cdot\mid s_j,a_j)\right)\right]\leq k\rho. \tag*{(286)} KL ( P k ′ ∥ P k ) = j = 0 ∑ k − 1 E P ′ [ KL ( P ′ ( ⋅ ∣ s j , a j ) ∥ P ( ⋅ ∣ s j , a j ) ) ] ≤ k ρ . (286)
where we use the fact that for any q ∈ U ρ ( p ) q \in\mathcal{U}_{\rho}(p) q ∈ U ρ ( p ) , one has KL ( q ∥ p ) ≤ χ 2 ( q ∥ p ) ≤ ρ \operatorname{KL}(q \mathbin\Vert p) \le\chi^{2}(q \mathbin\Vert p) \le\rho KL ( q ∥ p ) ≤ χ 2 ( q ∥ p ) ≤ ρ (see (5) ).
Let P ′ \mathbb{P}^{\prime} P ′ and P \mathbb{P} P denote the joint distributions of the stopping time and the corresponding randomly stopped trajectory under the transition kernels P ′ P^{\prime} P ′ and P P P , respectively. Another application of the chain rule combined with (286) yields
KL ( P ′ ∥ P ) = ∑ k = 0 ∞ ( 1 − γ ) γ k KL ( P k ′ ∥ P k ) ≤ ρ ∑ k = 0 ∞ k ( 1 − γ ) γ k = ρ γ 1 − γ . \operatorname{KL}(\mathbb{P}^{\prime} \mathbin\Vert\mathbb{P}) = \sum_{k=0}^{\infty}(1-\gamma)\gamma^{k}\operatorname{KL}(P_{k}^{\prime} \mathbin\Vert P_{k}) \le\rho\sum_{k=0}^{\infty}k(1-\gamma)\gamma^{k} = \frac{\rho\gamma}{1-\gamma}. KL ( P ′ ∥ P ) = k = 0 ∑ ∞ ( 1 − γ ) γ k KL ( P k ′ ∥ P k ) ≤ ρ k = 0 ∑ ∞ k ( 1 − γ ) γ k = 1 − γ ρ γ .
Combining (285) with 0 ≤ r ≤ 1 0 \le r \le1 0 ≤ r ≤ 1 and Pinsker’s inequality (see (5) , or Tsybakov (2009, Lemma 2.5)), we can demonstrate that
∣ Q π , P ′ ( s , a ) − Q π , P ( s , a ) ∣ ≤ 1 1 − γ TV ( P ′ , P ) ≤ 1 1 − γ 1 2 KL ( P ′ ∥ P ) ≤ ρ γ 2 ( 1 − γ ) 3 . \left|Q^{\pi,P^{\prime}}(s,a)-Q^{\pi,P}(s,a)\right| \le\frac{1}{1-\gamma}\operatorname{TV}(\mathbb{P}^{\prime},\mathbb{P}) \le\frac{1}{1-\gamma}\sqrt{\frac{1}{2}\operatorname{KL}(\mathbb{P}^{\prime}\mathbin\Vert\mathbb{P})} \le\sqrt{\frac{\rho\gamma}{2(1-\gamma)^{3}}}. Q π , P ′ ( s , a ) − Q π , P ( s , a ) ≤ 1 − γ 1 TV ( P ′ , P ) ≤ 1 − γ 1 2 1 KL ( P ′ ∥ P ) ≤ 2 ( 1 − γ ) 3 ρ γ .
which holds uniformly over all ( s , a ) (s,a) ( s , a ) and all deterministic policy π \pi π . This in turn reveals that
∣ Q ⋆ , P ′ ( s , a ) − Q ⋆ , P ( s , a ) ∣ = ∣ max π Q π , P ′ ( s , a ) − max π Q π , P ( s , a ) ∣ ≤ max π ∣ Q π , P ′ ( s , a ) − Q π , P ( s , a ) ∣ , \left|Q^{\star,P^{\prime}}(s,a)-Q^{\star,P}(s,a)\right| = \left|\max_{\pi}Q^{\pi,P^{\prime}}(s,a)-\max_{\pi}Q^{\pi,P}(s,a)\right| \le\max_{\pi}\left|Q^{\pi,P^{\prime}}(s,a)-Q^{\pi,P}(s,a)\right|, Q ⋆ , P ′ ( s , a ) − Q ⋆ , P ( s , a ) = π max Q π , P ′ ( s , a ) − π max Q π , P ( s , a ) ≤ π max Q π , P ′ ( s , a ) − Q π , P ( s , a ) ,
thereby establishing (240) .
Appendix C: proof of the lower bound in Theorem 4.7
We now present the proof of the algorithm-dependent lower bound in Theorem 4.7 for Q-learning. For brevity, we focus on the case of time-invariant learning rates, i.e.,
η t ≡ η ∈ ( 0 , 1 ) , t ≥ 0 \eta_t \equiv\eta\in(0,1), \qquad t \ge0 η t ≡ η ∈ ( 0 , 1 ) , t ≥ 0
throughout the execution of the algorithm. The proof for the case of rescaled linear learning rates is similar and can be found in Li et al. (2024a). Without loss of generality, we assume
log T ≤ 1 1 − γ (287) \log T \le\frac{1}{1-\gamma} \tag*{(287)} log T ≤ 1 − γ 1 (287)
throughout the proof; otherwise the lower bound in Theorem 4.7 becomes looser than the minimax lower bound 1 ( 1 − γ ) 3 T \frac{1}{(1-\gamma)^{3}T} ( 1 − γ ) 3 T 1 established in Azar et al. (2013) (see Theorem 3.7).
Step 1: construction of a hard MDP
We begin by constructing a hard MDP instance M h a r d \mathcal{M}_{\mathrm{hard}} M hard with 4 states and 2 actions as follows; see Figure 4.2 for an illustration. The subsequent analysis will focus on tracking the Q-learning dynamics for this specific construction.
Figure 4.2. An illustration of the hard MDP in the proof of the lower bound for Q-learning.
S = { 0 , 1 , 2 , 3 } . \mathcal{S}=\{0,1,2,3\}. S = { 0 , 1 , 2 , 3 } .
Action space. To streamline the presentation, we allow the action space to vary across states. Let A s \mathcal{A}_{s} A s represent the action space associated with each state s ∈ S s \in\mathcal{S} s ∈ S , which we specify as follows:
A 0 = { 1 } , A 1 = { 1 , 2 } , A 2 = { 1 } , A 3 = { 1 } . \mathcal{A}_{0}=\{1\}, \qquad\mathcal{A}_{1}=\{1,2\}, \qquad\mathcal{A}_{2}=\{1\}, \qquad\mathcal{A}_{3}=\{1\}. A 0 = { 1 } , A 1 = { 1 , 2 } , A 2 = { 1 } , A 3 = { 1 } .
In other words, only state 1 admits more than one action, while all other states are associated with a single action.
from state 0 : P ( 0 ∣ 0 , 1 ) = 1 , r ( 0 , 1 ) = 0 , from state 1 : P ( 1 ∣ 1 , 1 ) = p , P ( 0 ∣ 1 , 1 ) = 1 − p , r ( 1 , 1 ) = 1 , P ( 1 ∣ 1 , 2 ) = p , P ( 0 ∣ 1 , 2 ) = 1 − p , r ( 1 , 2 ) = 1 , from state 2 : P ( 2 ∣ 2 , 1 ) = p , P ( 0 ∣ 2 , 1 ) = 1 − p , r ( 2 , 1 ) = 1 , from state 3 : P ( 3 ∣ 3 , 1 ) = 1 , r ( 3 , 1 ) = 1. \begin{aligned}
\text{from state 0}: \quad&P(0\mid0,1)=1, &&r(0,1)=0,\\
\text{from state 1}: \quad&P(1\mid1,1)=p, &&P(0\mid1,1)=1-p, &&r(1,1)=1,\\
& P(1\mid1,2)=p, &&P(0\mid1,2)=1-p, &&r(1,2)=1,\\
\text{from state 2}: \quad&P(2\mid2,1)=p, &&P(0\mid2,1)=1-p, &&r(2,1)=1,\\
\text{from state 3}: \quad&P(3\mid3,1)=1, &&r(3,1)=1.
\end{aligned} from state 0 : from state 1 : from state 2 : from state 3 : P ( 0 ∣ 0 , 1 ) = 1 , P ( 1 ∣ 1 , 1 ) = p , P ( 1 ∣ 1 , 2 ) = p , P ( 2 ∣ 2 , 1 ) = p , P ( 3 ∣ 3 , 1 ) = 1 , r ( 0 , 1 ) = 0 , P ( 0 ∣ 1 , 1 ) = 1 − p , P ( 0 ∣ 1 , 2 ) = 1 − p , P ( 0 ∣ 2 , 1 ) = 1 − p , r ( 3 , 1 ) = 1. r ( 1 , 1 ) = 1 , r ( 1 , 2 ) = 1 , r ( 2 , 1 ) = 1 ,
for some parameter p ∈ ( 0 , 1 ) p\in(0,1) p ∈ ( 0 , 1 ) . States 0 and 3 are absorbing, while from states 1 and 2 the process either remains in the same state or transitions to state 0. Moreover, state 0 yields zero immediate reward; once it is entered, no further rewards can be obtained.
In fact, the MDP construction bears some similarity with that in § 4.9.1 for the minimax lower bounds, with the key distinction being that the action space is now augmented to include at least two actions in order to capture the over-estimation issue of Q-learning. Motivated by the construction in § 4.9.1 (see (146) ), we choose the parameter p p p as
p = 4 γ − 1 3 γ . (288) p=\frac{4\gamma-1}{3\gamma}. \tag*{(288)} p = 3 γ 4 γ − 1 . (288)
A careful reader may also notice the inclusion of state 3, which is isolated and absorbing. As we shall see later, this state is introduced to simplify a small part of the analysis.
To facilitate the subsequent convergence analysis of Q-learning on this MDP, we first characterize its optimal value function and Q-function. The proof is deferred to the end of this section.
Lemma 4.14. The optimal value function and Q-function of the MDP M h a r d \mathcal{M}_{\mathrm{hard}} M hard constructed above satisfy
V ∗ ( 0 ) = Q ∗ ( 0 , 1 ) = 0 ; V^*(0)=Q^*(0,1)=0; V ∗ ( 0 ) = Q ∗ ( 0 , 1 ) = 0 ; V ∗ ( 1 ) = Q ∗ ( 1 , 1 ) = Q ∗ ( 1 , 2 ) = V ∗ ( 2 ) = Q ∗ ( 2 , 1 ) = 1 1 − γ p = 3 4 ( 1 − γ ) ; V^*(1)=Q^*(1,1)=Q^*(1,2)=V^*(2)=Q^*(2,1)=\frac{1}{1-\gamma p}=\frac{3}{4(1-\gamma)}; V ∗ ( 1 ) = Q ∗ ( 1 , 1 ) = Q ∗ ( 1 , 2 ) = V ∗ ( 2 ) = Q ∗ ( 2 , 1 ) = 1 − γ p 1 = 4 ( 1 − γ ) 3 ; V ∗ ( 3 ) = Q ∗ ( 3 , 1 ) = 1 1 − γ . V^*(3)=Q^*(3,1)=\frac{1}{1-\gamma}. V ∗ ( 3 ) = Q ∗ ( 3 , 1 ) = 1 − γ 1 .
Proof of Lemma 4.14. Since state 0 is absorbing with zero reward — so no further rewards are obtained once it is entered — we have
V ∗ ( 0 ) = Q ∗ ( 0 , 1 ) = 0. V^*(0)=Q^*(0,1)=0. V ∗ ( 0 ) = Q ∗ ( 0 , 1 ) = 0.
We now turn to states 1 and 2. Since actions 1 and 2 in state 1 are equivalent in both rewards and transition probabilities, it is clear that
Q ⋆ ( 1 , 1 ) = Q ⋆ ( 1 , 2 ) = V ⋆ ( 1 ) . (289) Q^\star(1,1) = Q^\star(1,2) = V^\star(1). \tag*{(289)} Q ⋆ ( 1 , 1 ) = Q ⋆ ( 1 , 2 ) = V ⋆ ( 1 ) . (289)
Also, it follows from the Bellman optimality equation (22) that
Q ⋆ ( 1 , 1 ) = r ( 1 , 1 ) + γ P ( 0 ∣ 1 , 1 ) V ⋆ ( 0 ) + γ P ( 1 ∣ 1 , 1 ) V ⋆ ( 1 ) . Q^\star(1,1) = r(1,1) + \gamma P(0 \mid1,1)V^\star(0) + \gamma P(1 \mid1,1)V^\star(1). Q ⋆ ( 1 , 1 ) = r ( 1 , 1 ) + γ P ( 0 ∣ 1 , 1 ) V ⋆ ( 0 ) + γ P ( 1 ∣ 1 , 1 ) V ⋆ ( 1 ) .
Combining this with (289) and rearranging terms, we reach
V ⋆ ( 1 ) = r ( 1 , 1 ) + γ P ( 0 ∣ 1 , 1 ) V ⋆ ( 0 ) 1 − γ P ( 1 ∣ 1 , 1 ) = 1 1 − γ p = 3 4 ( 1 − γ ) . V^\star(1) = \frac{r(1,1) + \gamma P(0 \mid1,1)V^\star(0)}{1 - \gamma P(1 \mid1,1)} = \frac{1}{1-\gamma p} = \frac{3}{4(1-\gamma)}. V ⋆ ( 1 ) = 1 − γ P ( 1 ∣ 1 , 1 ) r ( 1 , 1 ) + γ P ( 0 ∣ 1 , 1 ) V ⋆ ( 0 ) = 1 − γ p 1 = 4 ( 1 − γ ) 3 .
where the second identity follows since V ⋆ ( 0 ) = 0 V^\star(0)=0 V ⋆ ( 0 ) = 0 , and the third identity relies on the choice (288) of p p p . The derivations for V ⋆ ( 2 ) V^\star(2) V ⋆ ( 2 ) and Q ⋆ ( 2 , 1 ) Q^\star(2,1) Q ⋆ ( 2 , 1 ) are omitted as they follow from identical arguments.
When it comes to state 3, we invoke Bellman’s equation to obtain
V ⋆ ( 3 ) = Q ⋆ ( 3 , 1 ) = r ( 3 , 1 ) + γ P ( 3 ∣ 3 , 1 ) V ⋆ ( 3 ) = 1 + γ V ⋆ ( 3 ) . V^\star(3) = Q^\star(3,1) = r(3,1) + \gamma P(3 \mid3,1)V^\star(3) = 1 + \gamma V^\star(3). V ⋆ ( 3 ) = Q ⋆ ( 3 , 1 ) = r ( 3 , 1 ) + γ P ( 3 ∣ 3 , 1 ) V ⋆ ( 3 ) = 1 + γ V ⋆ ( 3 ) .
A little algebra then leads to V ⋆ ( 3 ) = 1 1 − γ V^\star(3)=\frac{1}{1-\gamma} V ⋆ ( 3 ) = 1 − γ 1 .
Step 2: preliminary calculation of Q-learning iterates
We now embark on the analysis of Q-learning for the MDP instance constructed in Step 1. As before, we let P t P_t P t encode the empirical transitions such that for any triple ( s , a , s ′ ) (s,a,s') ( s , a , s ′ ) ,
P t ( s ′ ∣ s , a ) : = { 1 , if s t ( s , a ) = s ′ , 0 , otherwise , (290) P_t(s' \mid s,a) := \begin{cases}
1, & \text{if }s_t(s,a)=s',\\
0, & \text{otherwise},
\end{cases} \tag*{(290)} P t ( s ′ ∣ s , a ) := { 1 , 0 , if s t ( s , a ) = s ′ , otherwise , (290)
where s t ( s , a ) s_t(s,a) s t ( s , a ) denotes the sample collected during the t t t -th iteration w.r.t. the ( s , a ) (s,a) ( s , a ) pair. Given that state 2 has a singleton action space, we simplify the notation by writing
P t ( s ′ ∣ 2 ) : = P t ( s ′ ∣ 2 , 1 ) . P_t(s' \mid2) := P_t(s' \mid2,1). P t ( s ′ ∣ 2 ) := P t ( s ′ ∣ 2 , 1 ) .
In what follows, we first introduce some notation and basic properties concerning the learning rates, and then slightly expand the Q-learning updates for each state.
Preliminary facts about learning rates. Let us introduce the following notation about learning rates which will be convenient for analyzing the Q-learning iterations:
η k ( t ) : = { η ( 1 − η ( 1 − γ p ) ) t − k , if 1 ≤ k ≤ t , ( 1 − η ( 1 − γ p ) ) t , if k = 0. (291) \eta_k^{(t)} := \begin{cases}
\eta(1-\eta(1-\gamma p))^{t-k}, & \text{if }1 \le k \le t,\\
(1-\eta(1-\gamma p))^t, & \text{if }k=0.
\end{cases} \tag*{(291)} η k ( t ) := { η ( 1 − η ( 1 − γ p ) ) t − k , ( 1 − η ( 1 − γ p ) ) t , if 1 ≤ k ≤ t , if k = 0. (291)
A direct geometric-sum calculation gives
η 0 ( t ) + ( 1 − γ p ) ∑ k = 1 t η k ( t ) = 1 (292) \eta_0^{(t)} + (1-\gamma p)\sum_{k=1}^{t}\eta_k^{(t)} = 1 \tag*{(292)} η 0 ( t ) + ( 1 − γ p ) k = 1 ∑ t η k ( t ) = 1 (292)
( 1 − η ( 1 − γ p ) ) t − τ + ( 1 − γ p ) ∑ k = τ + 1 t η k ( t ) = 1 (293) (1-\eta(1-\gamma p))^{t-\tau} + (1-\gamma p)\sum_{k=\tau+1}^{t}\eta_k^{(t)} = 1 \tag*{(293)} ( 1 − η ( 1 − γ p ) ) t − τ + ( 1 − γ p ) k = τ + 1 ∑ t η k ( t ) = 1 (293)
for any given integer 0 ≤ τ < t 0 \le\tau< t 0 ≤ τ < t .
State 0. To begin, since Q 0 = V 0 = 0 Q_0=V_0=0 Q 0 = V 0 = 0 and state 0 is absorbing, the Q-learning update rule implies that, for all t ≥ 1 t\ge1 t ≥ 1 ,
V t ( 0 ) = Q t ( 0 , 1 ) = ( 1 − η ) Q t − 1 ( 0 , 1 ) + η { r ( 0 , 1 ) + γ V t − 1 ( 0 ) } = ( 1 − η ( 1 − γ ) ) Q t − 1 ( 0 , 1 ) = ⋯ = ( 1 − η ( 1 − γ ) ) t Q 0 ( 0 , 1 ) = 0. (294) \begin{aligned}
V_t(0)=Q_t(0,1)&=(1-\eta)Q_{t-1}(0,1)+\eta\{r(0,1)+\gamma V_{t-1}(0)\}\\
&=(1-\eta(1-\gamma))Q_{t-1}(0,1)=\cdots\\
&=(1-\eta(1-\gamma))^tQ_0(0,1)=0. \tag*{(294)}
\end{aligned} V t ( 0 ) = Q t ( 0 , 1 ) = ( 1 − η ) Q t − 1 ( 0 , 1 ) + η { r ( 0 , 1 ) + γ V t − 1 ( 0 )} = ( 1 − η ( 1 − γ )) Q t − 1 ( 0 , 1 ) = ⋯ = ( 1 − η ( 1 − γ ) ) t Q 0 ( 0 , 1 ) = 0. (294)
State 1. For state 1, the Q-learning iterates satisfy
Q t ( 1 , 1 ) = ( 1 − η ) Q t − 1 ( 1 , 1 ) + η { 1 + γ P t ( 1 ∣ 1 , 1 ) V t − 1 ( 1 ) } , (295) Q_t(1,1)=(1-\eta)Q_{t-1}(1,1)+\eta\{1+\gamma P_t(1\mid1,1)V_{t-1}(1)\}, \tag*{(295)} Q t ( 1 , 1 ) = ( 1 − η ) Q t − 1 ( 1 , 1 ) + η { 1 + γ P t ( 1 ∣ 1 , 1 ) V t − 1 ( 1 )} , (295)
Q t ( 1 , 2 ) = ( 1 − η ) Q t − 1 ( 1 , 2 ) + η { 1 + γ P t ( 1 ∣ 1 , 2 ) V t − 1 ( 1 ) } . (296) Q_t(1,2)=(1-\eta)Q_{t-1}(1,2)+\eta\{1+\gamma P_t(1\mid1,2)V_{t-1}(1)\}. \tag*{(296)} Q t ( 1 , 2 ) = ( 1 − η ) Q t − 1 ( 1 , 2 ) + η { 1 + γ P t ( 1 ∣ 1 , 2 ) V t − 1 ( 1 )} . (296)
State 2. The Q-learning update combined with (294) gives
V t ( 2 ) = Q t ( 2 , 1 ) = ( 1 − η ) Q t − 1 ( 2 , 1 ) + η { r ( 2 , 1 ) + γ P t ( 2 ∣ 2 ) V t − 1 ( 2 ) + γ P t ( 0 ∣ 2 ) V t − 1 ( 0 ) } = ( 1 − η ) V t − 1 ( 2 ) + η { 1 + γ P t ( 2 ∣ 2 ) V t − 1 ( 2 ) } = ( 1 − η ( 1 − γ p ) ) V t − 1 ( 2 ) + η { 1 + γ ( P t ( 2 ∣ 2 ) − p ) V t − 1 ( 2 ) } . (297) \begin{aligned}
V_t(2)=Q_t(2,1)&=(1-\eta)Q_{t-1}(2,1)+\eta\{r(2,1)+\gamma P_t(2\mid2)V_{t-1}(2)+\gamma P_t(0\mid2)V_{t-1}(0)\}\\
&=(1-\eta)V_{t-1}(2)+\eta\{1+\gamma P_t(2\mid2)V_{t-1}(2)\}\\
&=(1-\eta(1-\gamma p))V_{t-1}(2)+\eta\{1+\gamma(P_t(2\mid2)-p)V_{t-1}(2)\}. \tag*{(297)}
\end{aligned} V t ( 2 ) = Q t ( 2 , 1 ) = ( 1 − η ) Q t − 1 ( 2 , 1 ) + η { r ( 2 , 1 ) + γ P t ( 2 ∣ 2 ) V t − 1 ( 2 ) + γ P t ( 0 ∣ 2 ) V t − 1 ( 0 )} = ( 1 − η ) V t − 1 ( 2 ) + η { 1 + γ P t ( 2 ∣ 2 ) V t − 1 ( 2 )} = ( 1 − η ( 1 − γ p )) V t − 1 ( 2 ) + η { 1 + γ ( P t ( 2 ∣ 2 ) − p ) V t − 1 ( 2 )} . (297)
Therefore, for any t t t and τ < t \tau<t τ < t , applying (297) recursively leads to
V t ( 2 ) = ( 1 − η ( 1 − γ p ) ) t − τ V τ ( 2 ) + ∑ k = τ + 1 t η k ( t ) + ∑ k = τ + 1 t η k ( t ) γ ( P k ( 2 ∣ 2 ) − p ) V k − 1 ( 2 ) , (298) V_t(2)=(1-\eta(1-\gamma p))^{t-\tau}V_\tau(2)+\sum_{k=\tau+1}^{t}\eta_k^{(t)}+\sum_{k=\tau+1}^{t}\eta_k^{(t)}\gamma(P_k(2\mid2)-p)V_{k-1}(2), \tag*{(298)} V t ( 2 ) = ( 1 − η ( 1 − γ p ) ) t − τ V τ ( 2 ) + k = τ + 1 ∑ t η k ( t ) + k = τ + 1 ∑ t η k ( t ) γ ( P k ( 2 ∣ 2 ) − p ) V k − 1 ( 2 ) , (298)
where η k ( t ) \eta_k^{(t)} η k ( t ) has been defined in (291) . Recalling from (293) that
∑ k = τ + 1 t η k ( t ) = 1 − ( 1 − η ( 1 − γ p ) ) t − τ 1 − γ p , \sum_{k=\tau+1}^{t}\eta_k^{(t)}=\frac{1-(1-\eta(1-\gamma p))^{t-\tau}}{1-\gamma p}, k = τ + 1 ∑ t η k ( t ) = 1 − γ p 1 − ( 1 − η ( 1 − γ p ) ) t − τ ,
we can rearrange terms in (298) to reach
V t ( 2 ) = 1 1 − γ p − ( 1 − η ( 1 − γ p ) ) t − τ [ 1 1 − γ p − V τ ( 2 ) ] + ∑ k = τ + 1 t η k ( t ) γ ( P k ( 2 ∣ 2 ) − p ) V k − 1 ( 2 ) . (299) V_t(2)=\frac{1}{1-\gamma p}-(1-\eta(1-\gamma p))^{t-\tau}\left[\frac{1}{1-\gamma p}-V_\tau(2)\right]+\sum_{k=\tau+1}^{t}\eta_k^{(t)}\gamma(P_k(2\mid2)-p)V_{k-1}(2). \tag*{(299)} V t ( 2 ) = 1 − γ p 1 − ( 1 − η ( 1 − γ p ) ) t − τ [ 1 − γ p 1 − V τ ( 2 ) ] + k = τ + 1 ∑ t η k ( t ) γ ( P k ( 2 ∣ 2 ) − p ) V k − 1 ( 2 ) . (299)
In particular, in the special case where τ = 0 \tau=0 τ = 0 (so that V τ ( 2 ) = V 0 ( 2 ) = 0 V_\tau(2)=V_0(2)=0 V τ ( 2 ) = V 0 ( 2 ) = 0 ), this identity simplifies to
V t ( 2 ) = 1 − η 0 ( t ) 1 − γ p + ∑ k = 1 t η k ( t ) γ ( P k ( 2 ∣ 2 ) − p ) V k − 1 ( 2 ) , (300) V_t(2)=\frac{1-\eta_0^{(t)}}{1-\gamma p}+\sum_{k=1}^{t}\eta_k^{(t)}\gamma(P_k(2\mid2)-p)V_{k-1}(2), \tag*{(300)} V t ( 2 ) = 1 − γ p 1 − η 0 ( t ) + k = 1 ∑ t η k ( t ) γ ( P k ( 2 ∣ 2 ) − p ) V k − 1 ( 2 ) , (300)
where we recall the definition of η 0 ( t ) \eta_0^{(t)} η 0 ( t ) in (291) . Moreover, we know that E [ P k ( 2 ∣ 2 ) ] = p \mathbb{E}[P_k(2\mid2)]=p E [ P k ( 2 ∣ 2 )] = p by construction, which together with the identity (300) and Lemma 4.14 leads to
E [ V t ( 2 ) ] = 1 − η 0 ( t ) 1 − γ p = 1 − ( 1 − 4 η 3 ( 1 − γ ) ) t 4 3 ( 1 − γ ) ; (301) \mathbb{E}[V_t(2)]=\frac{1-\eta_0^{(t)}}{1-\gamma p}=\frac{1-\left(1-\frac{4\eta}{3}(1-\gamma)\right)^t}{\frac{4}{3}(1-\gamma)}; \tag*{(301)} E [ V t ( 2 )] = 1 − γ p 1 − η 0 ( t ) = 3 4 ( 1 − γ ) 1 − ( 1 − 3 4 η ( 1 − γ ) ) t ; (301)
V ∗ ( 2 ) − E [ V t ( 2 ) ] = η 0 ( t ) 1 − γ p = ( 1 − 4 η 3 ( 1 − γ ) ) t 4 3 ( 1 − γ ) . (302) V^*(2)-\mathbb{E}[V_t(2)]=\frac{\eta_0^{(t)}}{1-\gamma p}=\frac{\left(1-\frac{4\eta}{3}(1-\gamma)\right)^t}{\frac{4}{3}(1-\gamma)}. \tag*{(302)} V ∗ ( 2 ) − E [ V t ( 2 )] = 1 − γ p η 0 ( t ) = 3 4 ( 1 − γ ) ( 1 − 3 4 η ( 1 − γ ) ) t . (302)
Similarly, applying the above argument to (299) and rearranging terms, we obtain, for any integer 0 ≤ τ < T 0 \le\tau< T 0 ≤ τ < T ,
V ∗ ( 2 ) − E [ V T ( 2 ) ] = ( 1 − η ( 1 − γ p ) ) T − τ ( 1 1 − γ p − E [ V τ ( 2 ) ] ) = ( 1 − η ( 1 − γ p ) ) T − τ ( V ∗ ( 2 ) − E [ V τ ( 2 ) ] ) . (303) \begin{aligned}
V^{*}(2)-\mathbb{E}[V_{T}(2)] &= (1-\eta(1-\gamma p))^{T-\tau}\left(\frac{1}{1-\gamma p}-\mathbb{E}[V_{\tau}(2)]\right) \\
&= (1-\eta(1-\gamma p))^{T-\tau}\left(V^{*}(2)-\mathbb{E}[V_{\tau}(2)]\right). \tag*{(303)}
\end{aligned} V ∗ ( 2 ) − E [ V T ( 2 )] = ( 1 − η ( 1 − γ p ) ) T − τ ( 1 − γ p 1 − E [ V τ ( 2 )] ) = ( 1 − η ( 1 − γ p ) ) T − τ ( V ∗ ( 2 ) − E [ V τ ( 2 )] ) . (303)
State 3. When it comes to state 3, it holds that
V t ( 3 ) = Q t ( 3 , 1 ) = ( 1 − η ) Q t − 1 ( 3 , 1 ) + η { r ( 3 , 1 ) + γ V t − 1 ( 3 ) } = ( 1 − η ( 1 − γ ) ) V t − 1 ( 3 ) + η . (304) \begin{aligned}
V_t(3) &= Q_t(3,1) = (1-\eta)Q_{t-1}(3,1)+\eta\{r(3,1)+\gamma V_{t-1}(3)\} \\
&= (1-\eta(1-\gamma))V_{t-1}(3)+\eta. \tag*{(304)}
\end{aligned} V t ( 3 ) = Q t ( 3 , 1 ) = ( 1 − η ) Q t − 1 ( 3 , 1 ) + η { r ( 3 , 1 ) + γ V t − 1 ( 3 )} = ( 1 − η ( 1 − γ )) V t − 1 ( 3 ) + η . (304)
Recognizing that V ∗ ( 3 ) = 1 1 − γ V^{*}(3)=\frac{1}{1-\gamma} V ∗ ( 3 ) = 1 − γ 1 (see Lemma 4.14) obeys
V ∗ ( 3 ) = ( 1 − η ( 1 − γ ) ) V ∗ ( 3 ) + η , V^{*}(3)=(1-\eta(1-\gamma))V^{*}(3)+\eta, V ∗ ( 3 ) = ( 1 − η ( 1 − γ )) V ∗ ( 3 ) + η ,
we can combine these two identities to arrive at
V t ( 3 ) − V ∗ ( 3 ) = ( 1 − η ( 1 − γ ) ) ( V t − 1 ( 3 ) − V ∗ ( 3 ) ) = ( 1 − η ( 1 − γ ) ) T ( V 0 ( 3 ) − V ∗ ( 3 ) ) = − 1 1 − γ ( 1 − η ( 1 − γ ) ) T . (305) \begin{aligned}
V_t(3)-V^{*}(3) &= (1-\eta(1-\gamma))(V_{t-1}(3)-V^{*}(3)) \\
&= (1-\eta(1-\gamma))^{T}(V_0(3)-V^{*}(3)) \\
&= -\frac{1}{1-\gamma}(1-\eta(1-\gamma))^{T}. \tag*{(305)}
\end{aligned} V t ( 3 ) − V ∗ ( 3 ) = ( 1 − η ( 1 − γ )) ( V t − 1 ( 3 ) − V ∗ ( 3 )) = ( 1 − η ( 1 − γ ) ) T ( V 0 ( 3 ) − V ∗ ( 3 )) = − 1 − γ 1 ( 1 − η ( 1 − γ ) ) T . (305)
Roadmap. In what follows, we divide the analysis into three parts based on the learning rate η \eta η : small (0 < η ≤ 4 ( 1 − γ ) T 0<\eta\le\frac{4}{(1-\gamma)T} 0 < η ≤ ( 1 − γ ) T 4 ), large (η ≥ 1 ( 1 − γ ) 2 T \eta\ge\frac{1}{(1-\gamma)^2T} η ≥ ( 1 − γ ) 2 T 1 ), and moderate (4 ( 1 − γ ) T < η ≤ 1 ( 1 − γ ) 2 T \frac{4}{(1-\gamma)T}<\eta\le\frac{1}{(1-\gamma)^2T} ( 1 − γ ) T 4 < η ≤ ( 1 − γ ) 2 T 1 ). Each regime is tackled in its own subsection below. In particular, the regimes with small and large learning rates will be analyzed by lower bounding the mean and variance of V ∗ ( 2 ) − V T ( 2 ) V^{*}(2)-V_T(2) V ∗ ( 2 ) − V T ( 2 ) , respectively, for which it suffices to analyze this state that has a single available action. The moderate learning rate regime presents the main technical challenge. For this regime, we analyze the dynamics of state 1, which, by admitting two actions, captures the over-estimation issue; the dynamics of state 3 will also be invoked later to facilitate the analysis.
Step 3: the regime with small learning rates
Suppose now the learning rate satisfies
0 < η ≤ 4 ( 1 − γ ) T , 0<\eta\le\frac{4}{(1-\gamma)T}, 0 < η ≤ ( 1 − γ ) T 4 ,
which, as we shall see, is fairly small. In this regime, it suffices to look at the dynamics associated with state 2 (which has a single action). Our strategy is to show that, under such a small learning rate, the bias V ∗ ( 2 ) − E [ V t ( 2 ) ] V^{*}(2)-\mathbb{E}[V_t(2)] V ∗ ( 2 ) − E [ V t ( 2 )] does not decay sufficiently quickly and therefore remains large after T T T iterations.
Towards this end, we first recall from (4.122) that
V ∗ ( 2 ) − E [ V T ( 2 ) ] = η 0 ( T ) 1 − γ p , V^{*}(2)-\mathbb{E}[V_T(2)]=\frac{\eta_0^{(T)}}{1-\gamma p}, V ∗ ( 2 ) − E [ V T ( 2 )] = 1 − γ p η 0 ( T ) ,
which reduces the problem to deriving a lower bound on η 0 ( T ) \eta_0^{(T)} η 0 ( T ) . Observe that
η ( 1 − γ p ) ≤ 1 − γ p = 4 ( 1 − γ ) 3 ≤ 1 3 . (306) \eta(1-\gamma p)\le1-\gamma p=\frac{4(1-\gamma)}{3}\le\frac{1}{3}. \tag*{(306)} η ( 1 − γ p ) ≤ 1 − γ p = 3 4 ( 1 − γ ) ≤ 3 1 . (306)
η ( 1 − γ p ) = 4 η ( 1 − γ ) 3 < 16 3 T , (307) \eta(1-\gamma p)=\frac{4\eta(1-\gamma)}{3}<\frac{16}{3T}, \tag*{(307)} η ( 1 − γ p ) = 3 4 η ( 1 − γ ) < 3 T 16 , (307)
provided that γ ≥ 3 / 4 \gamma\ge3/4 γ ≥ 3/4 and η ≤ 4 ( 1 − γ ) T \eta\le\frac{4}{(1-\gamma)T} η ≤ ( 1 − γ ) T 4 . As a consequence,
log η 0 ( T ) = T log ( 1 − η ( 1 − γ p ) ) ≥ − 1.5 T η ( 1 − γ p ) ≥ − 8 , \log\eta_0^{(T)}=T\log\left(1-\eta(1-\gamma p)\right)\ge-1.5T\eta(1-\gamma p)\ge-8, log η 0 ( T ) = T log ( 1 − η ( 1 − γ p ) ) ≥ − 1.5 T η ( 1 − γ p ) ≥ − 8 ,
where the first inequality holds due to the elementary fact log ( 1 − x ) ≥ − 1.5 x \log(1-x)\ge-1.5x log ( 1 − x ) ≥ − 1.5 x for all 0 ≤ x ≤ 0.5 0\le x\le0.5 0 ≤ x ≤ 0.5 . Taking the above results collectively yields
V ∗ ( 2 ) − E [ V T ( 2 ) ] = η 0 ( T ) 1 − γ p ≥ e − 8 1 − γ p = 3 4 e 8 ( 1 − γ ) , V^*(2)-\mathbb{E}[V_T(2)]=\frac{\eta_0^{(T)}}{1-\gamma p}\ge\frac{e^{-8}}{1-\gamma p}=\frac{3}{4e^8(1-\gamma)}, V ∗ ( 2 ) − E [ V T ( 2 )] = 1 − γ p η 0 ( T ) ≥ 1 − γ p e − 8 = 4 e 8 ( 1 − γ ) 3 ,
thus indicating that
E [ ( V ∗ ( 2 ) − V T ( 2 ) ) 2 ] ≥ ( V ∗ ( 2 ) − E [ V T ( 2 ) ] ) 2 ≥ 9 16 e 16 ( 1 − γ ) 2 ≥ 9 c 3 16 e 16 ( 1 − γ ) 4 T . (308) \mathbb{E}\left[(V^*(2)-V_T(2))^2\right]\ge\left(V^*(2)-\mathbb{E}[V_T(2)]\right)^2\ge\frac{9}{16e^{16}(1-\gamma)^2}
\ge\frac{9c_3}{16e^{16}(1-\gamma)^4T}. \tag*{(308)} E [ ( V ∗ ( 2 ) − V T ( 2 ) ) 2 ] ≥ ( V ∗ ( 2 ) − E [ V T ( 2 )] ) 2 ≥ 16 e 16 ( 1 − γ ) 2 9 ≥ 16 e 16 ( 1 − γ ) 4 T 9 c 3 . (308)
provided that T ≥ c 3 / ( 1 − γ ) 2 T\ge c_3/(1-\gamma)^2 T ≥ c 3 / ( 1 − γ ) 2 . This establishes the advertised lower bound (219) for this regime.
Step 4: the regime with large learning rates
Next, consider the regime with
η ≥ 1 ( 1 − γ ) 2 T , \eta\ge\frac{1}{(1-\gamma)^2T}, η ≥ ( 1 − γ ) 2 T 1 ,
corresponding to large learning rates. In this regime, it is again sufficient to look at state 2. In contrast to the small learning rate regime where we analyze the bias, we now aim to show that the variance of V T ( 2 ) V_T(2) V T ( 2 ) is large. This is motivated by the intuition that larger learning rates can lead to greater variability in estimating optimal values.
Before proceeding, we first note that if η > 4 ( 1 − γ ) T \eta>\frac{4}{(1-\gamma)T} η > ( 1 − γ ) T 4 (which covers the range η ≥ 1 ( 1 − γ ) 2 T \eta\ge\frac{1}{(1-\gamma)^2T} η ≥ ( 1 − γ ) 2 T 1 for γ ≥ 3 / 4 \gamma\ge3/4 γ ≥ 3/4 ), then one has
η 0 ( T ) = ( 1 − 4 η 3 ( 1 − γ ) ) T ≤ ( 1 − 4 η 3 ( 1 − γ ) ) 4 η 3 ( 1 − γ ) 4 ( 1 − γ ) η T 3 ≤ e − 4 ( 1 − γ ) η T 3 < e − 16 / 3 < 1 100 , (309) \eta_0^{(T)}=\left(1-\frac{4\eta}{3}(1-\gamma)\right)^T\le\left(1-\frac{4\eta}{3}(1-\gamma)\right)^{\frac{4\eta}{3(1-\gamma)}\frac{4(1-\gamma)\eta T}{3}}
\le e^{-\frac{4(1-\gamma)\eta T}{3}}<e^{-16/3}<\frac{1}{100}, \tag*{(309)} η 0 ( T ) = ( 1 − 3 4 η ( 1 − γ ) ) T ≤ ( 1 − 3 4 η ( 1 − γ ) ) 3 ( 1 − γ ) 4 η 3 4 ( 1 − γ ) η T ≤ e − 3 4 ( 1 − γ ) η T < e − 16/3 < 100 1 , (309)
where the last line makes use of the elementary inequality ( 1 − x ) 1 / x ≤ e − 1 (1-x)^{1/x}\le e^{-1} ( 1 − x ) 1/ x ≤ e − 1 for all 0 < x < 1 0<x<1 0 < x < 1 . This combined with (302) yields an upper bound on the bias:
∣ V ∗ ( 2 ) − E [ V T ( 2 ) ] ∣ = η 0 ( T ) 4 3 ( 1 − γ ) < 1 100 ( 1 − γ ) . (310) \left|V^*(2)-\mathbb{E}[V_T(2)]\right|=\frac{\eta_0^{(T)}}{\frac{4}{3}(1-\gamma)}<\frac{1}{100(1-\gamma)}. \tag*{(310)} ∣ V ∗ ( 2 ) − E [ V T ( 2 )] ∣ = 3 4 ( 1 − γ ) η 0 ( T ) < 100 ( 1 − γ ) 1 . (310)
We now turn to the variance of V T ( 2 ) V_T(2) V T ( 2 ) . Recalling the decomposition (299) of V T ( 2 ) V_T(2) V T ( 2 ) , we intend to exploit martingale properties to lower-bound its variance. Specifically, consider first a martingale sequence { Z k } 0 ≤ k ≤ T \{Z_k\}_{0\le k\le T} { Z k } 0 ≤ k ≤ T adapted to a filtration F 0 ⊆ F 1 ⊆ ⋯ ⊆ F T \mathcal{F}_0\subseteq\mathcal{F}_1\subseteq\cdots\subseteq\mathcal{F}_T F 0 ⊆ F 1 ⊆ ⋯ ⊆ F T , satisfying E [ Z k + 1 ∣ F k ] = 0 \mathbb{E}[Z_{k+1}\mid\mathcal{F}_k]=0 E [ Z k + 1 ∣ F k ] = 0 and E [ Z k ∣ F k ] = Z k \mathbb{E}[Z_k\mid\mathcal{F}_k]=Z_k E [ Z k ∣ F k ] = Z k for all 0 ≤ k ≤ T 0\le k\le T 0 ≤ k ≤ T . In addition, consider any 0 ≤ τ < T 0\le\tau<T 0 ≤ τ < T , and let W 0 W_0 W 0 be a random variable such that E [ W 0 ∣ F τ ] = W 0 \mathbb{E}[W_0 \mid\mathcal{F}_\tau] = W_0 E [ W 0 ∣ F τ ] = W 0 . Then, by the law of total variance and standard martingale properties, we have
Var ( W 0 + ∑ k = τ + 1 T Z k ) = E [ Var ( W 0 + ∑ k = τ + 1 T Z k | F T − 1 ) ] + Var ( E [ W 0 + ∑ k = τ + 1 T Z k | F T − 1 ] ) = ( i ) E [ Var ( Z T ∣ F T − 1 ) ] + Var ( W 0 + ∑ k = τ + 1 T − 1 Z k ) = ⋯ = ∑ k = τ + 1 T E [ Var ( Z k ∣ F k − 1 ) ] + Var ( W 0 ) ≥ ∑ k = τ + 1 T E [ Var ( Z k ∣ F k − 1 ) ] , (311) \begin{aligned}
\operatorname{Var}\left(W_0+\sum_{k=\tau+1}^{T} Z_k\right)
&=\mathbb{E}\left[\operatorname{Var}\left(W_0+\sum_{k=\tau+1}^{T} Z_k \,\middle|\, \mathcal{F}_{T-1}\right)\right]
+\operatorname{Var}\left(\mathbb{E}\left[W_0+\sum_{k=\tau+1}^{T} Z_k \,\middle|\, \mathcal{F}_{T-1}\right]\right)\\
&\overset{(i)}{=}\mathbb{E}\left[\operatorname{Var}\left(Z_T \mid\mathcal{F}_{T-1}\right)\right]
+\operatorname{Var}\left(W_0+\sum_{k=\tau+1}^{T-1} Z_k\right)=\cdots\\
&=\sum_{k=\tau+1}^{T}\mathbb{E}\left[\operatorname{Var}\left(Z_k \mid\mathcal{F}_{k-1}\right)\right]+\operatorname{Var}(W_0)\\
&\geq\sum_{k=\tau+1}^{T}\mathbb{E}\left[\operatorname{Var}\left(Z_k \mid\mathcal{F}_{k-1}\right)\right], \tag*{(311)}
\end{aligned} Var ( W 0 + k = τ + 1 ∑ T Z k ) = E [ Var ( W 0 + k = τ + 1 ∑ T Z k F T − 1 ) ] + Var ( E [ W 0 + k = τ + 1 ∑ T Z k F T − 1 ] ) = ( i ) E [ Var ( Z T ∣ F T − 1 ) ] + Var ( W 0 + k = τ + 1 ∑ T − 1 Z k ) = ⋯ = k = τ + 1 ∑ T E [ Var ( Z k ∣ F k − 1 ) ] + Var ( W 0 ) ≥ k = τ + 1 ∑ T E [ Var ( Z k ∣ F k − 1 ) ] , (311)
where (i) arises from the above assumptions on W 0 W_0 W 0 and { Z k } \{Z_k\} { Z k } . In words, this decomposes the variance of a martingale sum by summing the expected conditional variances over time. Consequently, for any 0 ≤ τ < T − 1 0 \leq\tau< T-1 0 ≤ τ < T − 1 , applying inequality (311) to the decomposition (299) (with τ \tau τ replaced by τ + 1 \tau+1 τ + 1 ) yields
Var ( V T ( 2 ) ) ≥ E [ ∑ k = τ + 2 T Var ( η k ( T ) γ ( P k ( 2 ∣ 2 ) − p ) V k − 1 ( 2 ) ∣ V k − 1 ( 2 ) ) ] = ∑ k = τ + 2 T ( η k ( T ) γ ) 2 p ( 1 − p ) E [ ( V k − 1 ( 2 ) ) 2 ] ≥ ( 1 − γ ) ( 4 γ − 1 ) 9 ∑ k = τ + 2 T ( η k ( T ) ) 2 ( E [ V k − 1 ( 2 ) ] ) 2 , (312) \begin{aligned}
\operatorname{Var}(V_T(2))
&\geq\mathbb{E}\left[\sum_{k=\tau+2}^{T}\operatorname{Var}\left(\eta_k^{(T)}\gamma(P_k(2\mid2)-p)V_{k-1}(2)\mid V_{k-1}(2)\right)\right]\\
&=\sum_{k=\tau+2}^{T}\left(\eta_k^{(T)}\gamma\right)^2p(1-p)\mathbb{E}\left[(V_{k-1}(2))^2\right]\\
&\geq\frac{(1-\gamma)(4\gamma-1)}{9}\sum_{k=\tau+2}^{T}\left(\eta_k^{(T)}\right)^2\left(\mathbb{E}[V_{k-1}(2)]\right)^2, \tag*{(312)}
\end{aligned} Var ( V T ( 2 )) ≥ E [ k = τ + 2 ∑ T Var ( η k ( T ) γ ( P k ( 2 ∣ 2 ) − p ) V k − 1 ( 2 ) ∣ V k − 1 ( 2 ) ) ] = k = τ + 2 ∑ T ( η k ( T ) γ ) 2 p ( 1 − p ) E [ ( V k − 1 ( 2 ) ) 2 ] ≥ 9 ( 1 − γ ) ( 4 γ − 1 ) k = τ + 2 ∑ T ( η k ( T ) ) 2 ( E [ V k − 1 ( 2 )] ) 2 , (312)
where the identity arises from the fact that P k ( 2 ∣ 2 ) P_k(2\mid2) P k ( 2 ∣ 2 ) is a Bernoulli random variable with mean p p p , and the last line uses the choice (288) of p p p and the elementary fact E [ X 2 ] ≥ ( E [ X ] ) 2 \mathbb{E}[X^2] \geq(\mathbb{E}[X])^2 E [ X 2 ] ≥ ( E [ X ] ) 2 . As we will see shortly, the rationale for replacing E [ ( V k − 1 ( 2 ) ) 2 ] \mathbb{E}[(V_{k-1}(2))^2] E [( V k − 1 ( 2 ) ) 2 ] with ( E [ V k − 1 ( 2 ) ] ) 2 (\mathbb{E}[V_{k-1}(2)])^2 ( E [ V k − 1 ( 2 )] ) 2 is that the mean E [ V k − 1 ( 2 ) ] \mathbb{E}[V_{k-1}(2)] E [ V k − 1 ( 2 )] becomes quite large beyond a certain time threshold and is easier to compute.
To further control (312) , we develop a lower bound on E [ V k − 1 ( 2 ) ] \mathbb{E}[V_{k-1}(2)] E [ V k − 1 ( 2 )] . Given (310) and the facts that V 0 ( 2 ) = 0 V_0(2)=0 V 0 ( 2 ) = 0 and V ∗ ( 2 ) = 3 4 ( 1 − γ ) V^*(2)=\frac{3}{4(1-\gamma)} V ∗ ( 2 ) = 4 ( 1 − γ ) 3 (see Lemma 4.14), one is allowed to choose 0 < τ ≤ T 0<\tau\leq T 0 < τ ≤ T such that
E [ V τ ( 2 ) ] ≥ 1 2 ( 1 − γ ) ≥ E [ V τ − 1 ( 2 ) ] . (313) \mathbb{E}[V_\tau(2)]\geq\frac{1}{2(1-\gamma)}\geq\mathbb{E}[V_{\tau-1}(2)]. \tag*{(313)} E [ V τ ( 2 )] ≥ 2 ( 1 − γ ) 1 ≥ E [ V τ − 1 ( 2 )] . (313)
It is also seen from (302) that E [ V k ( 2 ) ] \mathbb{E}[V_k(2)] E [ V k ( 2 )] is monotonically increasing in k k k , and hence
E [ V k ( 2 ) ] ≥ E [ V τ ( 2 ) ] ≥ 1 2 ( 1 − γ ) for all k ≥ τ ; \mathbb{E}[V_k(2)]\geq\mathbb{E}[V_\tau(2)]\geq\frac{1}{2(1-\gamma)}
\qquad\text{for all }k\geq\tau; E [ V k ( 2 )] ≥ E [ V τ ( 2 )] ≥ 2 ( 1 − γ ) 1 for all k ≥ τ ;
namely, E [ V k ( 2 ) ] \mathbb{E}[V_k(2)] E [ V k ( 2 )] becomes large once k k k exceeds the threshold τ \tau τ . Combine this inequality with (312) to yield
Var ( V T ( 2 ) ) ≥ ( 1 − γ ) ( 4 γ − 1 ) 9 ⋅ 1 4 ( 1 − γ ) 2 ∑ k = τ + 2 T ( η k ( T ) ) 2 = 4 γ − 1 36 ( 1 − γ ) η 2 ( 1 − ( 1 − η ( 1 − γ p ) ) 2 ( T − τ − 1 ) ) 1 − ( 1 − η ( 1 − γ p ) ) 2 , (314) \begin{aligned}
\operatorname{Var}(V_T(2))
&\geq\frac{(1-\gamma)(4\gamma-1)}{9}\cdot\frac{1}{4(1-\gamma)^2}\sum_{k=\tau+2}^{T}\left(\eta_k^{(T)}\right)^2\\
&=\frac{4\gamma-1}{36(1-\gamma)}
\frac{\eta^2\left(1-\left(1-\eta(1-\gamma p)\right)^{2(T-\tau-1)}\right)}
{1-\left(1-\eta(1-\gamma p)\right)^2}, \tag*{(314)}
\end{aligned} Var ( V T ( 2 )) ≥ 9 ( 1 − γ ) ( 4 γ − 1 ) ⋅ 4 ( 1 − γ ) 2 1 k = τ + 2 ∑ T ( η k ( T ) ) 2 = 36 ( 1 − γ ) 4 γ − 1 1 − ( 1 − η ( 1 − γ p ) ) 2 η 2 ( 1 − ( 1 − η ( 1 − γ p ) ) 2 ( T − τ − 1 ) ) , (314)
where the last line arises from (291) and calculates the sum of the geometric sequence. Recalling that η ( 1 − γ p ) ≤ 1 / 3 \eta(1-\gamma p)\le1/3 η ( 1 − γ p ) ≤ 1/3 (cf. (306) ), one can deduce that
( 1 − η ( 1 − γ p ) ) 2 ( T − τ − 1 ) ≤ 81 16 ( 1 − η ( 1 − γ p ) ) 2 ( T − τ + 1 ) = 81 16 ( V ⋆ ( 2 ) − E [ V T ( 2 ) ] ) 2 ( V ⋆ ( 2 ) − E [ V τ − 1 ( 2 ) ] ) 2 ≤ 81 ( 1 − γ ) 2 4 ( V ⋆ ( 2 ) − E [ V T ( 2 ) ] ) 2 ≤ 1 2 , \begin{aligned}
\left(1-\eta(1-\gamma p)\right)^{2(T-\tau-1)} \le\frac{81}{16}\left(1-\eta(1-\gamma p)\right)^{2(T-\tau+1)} \\
&= \frac{81}{16}\frac{\left(V^\star(2)-\mathbb{E}[V_T(2)]\right)^2}{\left(V^\star(2)-\mathbb{E}[V_{\tau-1}(2)]\right)^2} \\
&\le\frac{81(1-\gamma)^2}{4}\left(V^\star(2)-\mathbb{E}[V_T(2)]\right)^2 \le\frac{1}{2},
\end{aligned} ( 1 − η ( 1 − γ p ) ) 2 ( T − τ − 1 ) ≤ 16 81 ( 1 − η ( 1 − γ p ) ) 2 ( T − τ + 1 ) = 16 81 ( V ⋆ ( 2 ) − E [ V τ − 1 ( 2 )] ) 2 ( V ⋆ ( 2 ) − E [ V T ( 2 )] ) 2 ≤ 4 81 ( 1 − γ ) 2 ( V ⋆ ( 2 ) − E [ V T ( 2 )] ) 2 ≤ 2 1 ,
where the second line follows from (303) , and the two inequalities in the last line arise from (4.132) and (4.129)`, respectively. Moreover, it is also readily seen that
1 − ( 1 − η ( 1 − γ p ) ) 2 ≤ 2 η ( 1 − γ p ) = 8 η ( 1 − γ ) 3 . 1-\left(1-\eta(1-\gamma p)\right)^2\le2\eta(1-\gamma p)=\frac{8\eta(1-\gamma)}{3}. 1 − ( 1 − η ( 1 − γ p ) ) 2 ≤ 2 η ( 1 − γ p ) = 3 8 η ( 1 − γ ) .
Substitute the above two inequalities into (314) to yield
E [ ( V ⋆ ( 2 ) − V T ( 2 ) ) 2 ] ≥ Var ( V T ( 2 ) ) ≥ η ( 4 γ − 1 ) 192 ( 1 − γ ) 2 ≥ 4 γ − 1 192 ( 1 − γ ) 4 T ≥ 1 86 ( 1 − γ ) 4 T . (315) \begin{aligned}
\mathbb{E}\left[\left(V^\star(2)-V_T(2)\right)^2\right]\ge\operatorname{Var}(V_T(2))\ge\frac{\eta(4\gamma-1)}{192(1-\gamma)^2}\ge\frac{4\gamma-1}{192(1-\gamma)^4T} \\
&\ge\frac{1}{86(1-\gamma)^4T}. \tag*{(315)}
\end{aligned} E [ ( V ⋆ ( 2 ) − V T ( 2 ) ) 2 ] ≥ Var ( V T ( 2 )) ≥ 192 ( 1 − γ ) 2 η ( 4 γ − 1 ) ≥ 192 ( 1 − γ ) 4 T 4 γ − 1 ≥ 86 ( 1 − γ ) 4 T 1 . (315)
with the proviso that η ≥ 1 ( 1 − γ ) 2 T \eta\ge\frac{1}{(1-\gamma)^2T} η ≥ ( 1 − γ ) 2 T 1 and γ ≥ 3 / 4 \gamma\ge3/4 γ ≥ 3/4 . This proves the claimed lower bound (219) for this regime.
Step 5: the regime with moderate learning rates
We now turn to the regime of moderate learning rates, namely,
4 ( 1 − γ ) T < η < 1 ( 1 − γ ) 2 T . \frac{4}{(1-\gamma)T}<\eta<\frac{1}{(1-\gamma)^2T}. ( 1 − γ ) T 4 < η < ( 1 − γ ) 2 T 1 .
This is the regime in which Q-learning is most effective, and at the same time, the most challenging to analyze when developing lower bounds. Throughout this case, we make note of the property
η 0 ( T ) ≤ 1 100 , (316) \eta_0^{(T)}\le\frac{1}{100}, \tag*{(316)} η 0 ( T ) ≤ 100 1 , (316)
which has been justified previously in (309) . We shall focus attention on the dynamics w.r.t. state 1 and its associated value function V t ( 1 ) V_t(1) V t ( 1 ) . Note that the presence of 2 available actions at this state is deliberately designed to capture the over-estimation issue inherent in Q-learning.
Step 5.1: decoupling the impact of initialization. It is slightly more convenient to work with a Q-learning sequence initialized at the true optimal value. We therefore introduce an auxiliary sequence governed by the following update rule:
Q ^ t ( a ) = ( 1 − η ) Q ^ t − 1 ( a ) + η { 1 + γ P t ( 1 ∣ 1 , a ) V ^ t − 1 } . (317) \widehat{Q}_t(a)=(1-\eta)\widehat{Q}_{t-1}(a)+\eta\{1+\gamma P_t(1\mid1,a)\widehat{V}_{t-1}\}. \tag*{(317)} Q t ( a ) = ( 1 − η ) Q t − 1 ( a ) + η { 1 + γ P t ( 1 ∣ 1 , a ) V t − 1 } . (317)
with
V ^ t − 1 : = max a Q ^ t − 1 ( a ) and Q ^ 0 ( a ) : = Q ⋆ ( 1 , a ) = 1 1 − γ p . \widehat{V}_{t-1}:=\max_a\widehat{Q}_{t-1}(a)\qquad\text{and}\qquad\widehat{Q}_0(a):=Q^\star(1,a)=\frac{1}{1-\gamma p}. V t − 1 := a max Q t − 1 ( a ) and Q 0 ( a ) := Q ⋆ ( 1 , a ) = 1 − γ p 1 .
where Q ⋆ ( 1 , a ) Q^{\star}(1,a) Q ⋆ ( 1 , a ) has been computed in Lemma 4.14. Evidently, { Q ^ t ( a ) } \{\widehat{Q}_t(a)\} { Q t ( a )} corresponds to a Q-learning sequence initialized at the ground truth. To decouple the effect of initialization from the original dynamics of Q t ( 1 , a ) Q_t(1,a) Q t ( 1 , a ) , we show that the discrepancy resulting from the different initializations of Q ^ t ( a ) \widehat{Q}_t(a) Q t ( a ) and Q t ( 1 , a ) Q_t(1,a) Q t ( 1 , a ) vanishes geometrically, i.e.,
Q t ( 1 , a ) ≥ Q ^ t ( a ) − 1 1 − γ ( 1 − η ( 1 − γ ) ) t , a ∈ { 1 , 2 } . (318) Q_t(1,a) \ge\widehat{Q}_t(a)-\frac{1}{1-\gamma}(1-\eta(1-\gamma))^t,\qquad a\in\{1,2\}. \tag*{(318)} Q t ( 1 , a ) ≥ Q t ( a ) − 1 − γ 1 ( 1 − η ( 1 − γ ) ) t , a ∈ { 1 , 2 } . (318)
The proof is postponed to § 7.20.4 .
In light of this bound (318) , we focus on bounding the auxiliary sequence Q ^ t \widehat{Q}_t Q t in the sequel, and defer translating the result back to Q t ( 1 , a ) Q_t(1,a) Q t ( 1 , a ) until the final sub-step of the proof. Before proceeding, we record a crude lower bound on V ^ t \widehat{V}_t V t :
P { V ^ t ≥ 1 4 ( 1 − γ ) } ≥ 1 2 for all t ≤ T . (319) \mathbb{P}\left\{\widehat{V}_t\ge\frac{1}{4(1-\gamma)}\right\}\ge\frac{1}{2}\qquad\text{for all }t\le T. \tag*{(319)} P { V t ≥ 4 ( 1 − γ ) 1 } ≥ 2 1 for all t ≤ T . (319)
In other words, it shows that V ^ t \widehat{V}_t V t has the correct order with nontrivial probability. The proof is postponed to § 7.20.4 .
Step 5.2: deriving recursive relations about estimation errors. We now study the over-estimation error of the auxiliary sequence Q ^ t \widehat{Q}_t Q t (cf. (317) ). Define the error sequence
Δ t ( a ) : = Q ^ t ( a ) − Q ⋆ ( 1 , a ) , a = 1 , 2 ; (320) \Delta_t(a):=\widehat{Q}_t(a)-Q^{\star}(1,a),\qquad a=1,2; \tag*{(320)} Δ t ( a ) := Q t ( a ) − Q ⋆ ( 1 , a ) , a = 1 , 2 ; (320)
Δ t , max : = max a Δ t ( a ) . (321) \Delta_{t,\max}:=\max_a\Delta_t(a). \tag*{(321)} Δ t , m a x := a max Δ t ( a ) . (321)
Our objective is to show that Δ T , max \Delta_{T,\max} Δ T , m a x is large in expectation (so that Q ^ t ( a ) \widehat{Q}_t(a) Q t ( a ) overestimates Q ⋆ ( 1 , a ) Q^{\star}(1,a) Q ⋆ ( 1 , a ) ), thus putting the “over-estimation” issue that fundamentally slows convergence on a rigorous footing. To do so, a key step lies in establishing recursive relations about Δ t , max \Delta_{t,\max} Δ t , m a x . According to the update rule (317) and the Bellman optimality equation Q ⋆ ( 1 , a ) = 1 + γ p V ⋆ ( 1 ) Q^{\star}(1,a)=1+\gamma pV^{\star}(1) Q ⋆ ( 1 , a ) = 1 + γ p V ⋆ ( 1 ) , we have
Δ t ( a ) = ( 1 − η ) Q ^ t − 1 ( a ) + η ( 1 + γ P t ( 1 ∣ 1 , a ) V ^ t − 1 ) − Q ⋆ ( 1 , a ) = ( 1 − η ) Δ t − 1 ( a ) + η ( 1 + γ P t ( 1 ∣ 1 , a ) V ^ t − 1 ) − Q ⋆ ( 1 , a ) = ( 1 − η ) Δ t − 1 ( a ) + η γ ( P t ( 1 ∣ 1 , a ) V ^ t − 1 − p V ⋆ ( 1 ) ) = ( 1 − η ) Δ t − 1 ( a ) + η γ ( p ( V ^ t − 1 − V ⋆ ( 1 ) ) + ( P t ( 1 ∣ 1 , a ) − p ) V ^ t − 1 ) = ( 1 − η ) Δ t − 1 ( a ) + η γ ( p Δ t − 1 , max + ( P t ( 1 ∣ 1 , a ) − p ) V ^ t − 1 ) , \begin{aligned}
\Delta_t(a)
&=(1-\eta)\widehat{Q}_{t-1}(a)+\eta(1+\gamma P_t(1\mid1,a)\widehat{V}_{t-1})-Q^{\star}(1,a)\\
&=(1-\eta)\Delta_{t-1}(a)+\eta(1+\gamma P_t(1\mid1,a)\widehat{V}_{t-1})-Q^{\star}(1,a)\\
&=(1-\eta)\Delta_{t-1}(a)+\eta\gamma(P_t(1\mid1,a)\widehat{V}_{t-1}-pV^{\star}(1))\\
&=(1-\eta)\Delta_{t-1}(a)+\eta\gamma\bigl(p(\widehat{V}_{t-1}-V^{\star}(1))+(P_t(1\mid1,a)-p)\widehat{V}_{t-1}\bigr)\\
&=(1-\eta)\Delta_{t-1}(a)+\eta\gamma\bigl(p\Delta_{t-1,\max}+(P_t(1\mid1,a)-p)\widehat{V}_{t-1}\bigr),
\end{aligned} Δ t ( a ) = ( 1 − η ) Q t − 1 ( a ) + η ( 1 + γ P t ( 1 ∣ 1 , a ) V t − 1 ) − Q ⋆ ( 1 , a ) = ( 1 − η ) Δ t − 1 ( a ) + η ( 1 + γ P t ( 1 ∣ 1 , a ) V t − 1 ) − Q ⋆ ( 1 , a ) = ( 1 − η ) Δ t − 1 ( a ) + η γ ( P t ( 1 ∣ 1 , a ) V t − 1 − p V ⋆ ( 1 )) = ( 1 − η ) Δ t − 1 ( a ) + η γ ( p ( V t − 1 − V ⋆ ( 1 )) + ( P t ( 1 ∣ 1 , a ) − p ) V t − 1 ) = ( 1 − η ) Δ t − 1 ( a ) + η γ ( p Δ t − 1 , m a x + ( P t ( 1 ∣ 1 , a ) − p ) V t − 1 ) ,
where the last line holds since V ^ t − 1 − V ⋆ ( 1 ) = max a ( Q ^ t − 1 ( a ) − V ⋆ ( 1 ) ) = max a Δ t − 1 ( a ) \widehat{V}_{t-1}-V^{\star}(1)=\max_a(\widehat{Q}_{t-1}(a)-V^{\star}(1))=\max_a\Delta_{t-1}(a) V t − 1 − V ⋆ ( 1 ) = max a ( Q t − 1 ( a ) − V ⋆ ( 1 )) = max a Δ t − 1 ( a ) . Applying this relation recursively yields
Δ t ( a ) = ∑ k = 1 t η ( 1 − η ) t − k γ ( p Δ k − 1 , max + ( P k ( 1 ∣ 1 , a ) − p ) V ^ k − 1 ) , \Delta_t(a)=\sum_{k=1}^{t}\eta(1-\eta)^{t-k}\gamma\bigl(p\Delta_{k-1,\max}+(P_k(1\mid1,a)-p)\widehat{V}_{k-1}\bigr), Δ t ( a ) = k = 1 ∑ t η ( 1 − η ) t − k γ ( p Δ k − 1 , m a x + ( P k ( 1 ∣ 1 , a ) − p ) V k − 1 ) ,
where we use the initialization Δ 0 ( a ) = 0 \Delta_0(a)=0 Δ 0 ( a ) = 0 (by construction). Letting
ξ t ( a ) : = ∑ k = 1 t η ( 1 − η ) t − k γ ( P k ( 1 ∣ 1 , a ) − p ) V ^ k − 1 , (322) \xi_t(a):=\sum_{k=1}^{t}\eta(1-\eta)^{t-k}\gamma(P_k(1\mid1,a)-p)\widehat{V}_{k-1}, \tag*{(322)} ξ t ( a ) := k = 1 ∑ t η ( 1 − η ) t − k γ ( P k ( 1 ∣ 1 , a ) − p ) V k − 1 , (322)
ξ t , max : = max a ξ t ( a ) , (323) \xi_{t,\max}:=\max_a\xi_t(a), \tag*{(323)} ξ t , m a x := a max ξ t ( a ) , (323)
one can take the maximum over a a a in the above display about Δ t ( a ) \Delta_t(a) Δ t ( a ) to derive the following identity regarding the maximum error:
Δ t , max = ∑ k = 1 t η ( 1 − η ) t − k γ p Δ k − 1 , max + ξ t , max . (324) \Delta_{t,\max}=\sum_{k=1}^{t}\eta(1-\eta)^{t-k}\gamma p\Delta_{k-1,\max}+\xi_{t,\max}. \tag*{(324)} Δ t , m a x = k = 1 ∑ t η ( 1 − η ) t − k γ p Δ k − 1 , m a x + ξ t , m a x . (324)
Step 5.3: lower bounding E [ ξ t , max ] \mathbb{E}[\xi_{t,\max}] E [ ξ t , m a x ] . As can be seen from the recursion (324) , a crucial step is to control ξ t , max \xi_{t,\max} ξ t , m a x , which involves the maximum of multiple random quantities. Towards this end, we develop a lower bound on E [ ξ t , max ] \mathbb{E}[\xi_{t,\max}] E [ ξ t , m a x ] as follows; the proof is deferred to Section 4.8.7.
Lemma 4.15. Define a threshold
τ ^ : = max { τ | ( 1 − η ( 1 − γ p ) ) T − τ + 1 ≤ 6 7 } . (325) \widehat{\tau}:=\max\left\{\tau\ \middle|\ \left(1-\eta(1-\gamma p)\right)^{T-\tau+1}\leq\frac{6}{7}\right\}. \tag*{(325)} τ := max { τ ( 1 − η ( 1 − γ p ) ) T − τ + 1 ≤ 7 6 } . (325) Then there exists some universal constant c > 0 c>0 c > 0 such that
E [ ξ t , max ] ≥ c ( 1 − γ ) 2 T log T \mathbb{E}[\xi_{t,\max}]\geq\frac{c}{\sqrt{(1-\gamma)^2T\log T}} E [ ξ t , m a x ] ≥ ( 1 − γ ) 2 T log T c for all t ≥ τ ^ t\geq\widehat{\tau} t ≥ τ .
Remark 4.4. Property (316) guarantees that τ ^ \widehat{\tau} τ (cf. (325) ) is well-defined.
Remark 4.5. Note that E [ ξ t , max ] \mathbb{E}[\xi_{t,\max}] E [ ξ t , m a x ] is strictly positive due to the presence of the max operator together with the availability of more than one action. In contrast, when there is only a single action a a a , one readily has E [ ξ t , max ] = E [ ξ t ( a ) ] = 0 \mathbb{E}[\xi_{t,\max}]=\mathbb{E}[\xi_t(a)]=0 E [ ξ t , m a x ] = E [ ξ t ( a )] = 0 . This distinction gives rise to a fundamental difference in sample complexity between the cases A = 1 A=1 A = 1 and A ≥ 2 A\geq2 A ≥ 2 , leading respectively to the horizon dependency 1 ( 1 − γ ) 3 \frac{1}{(1-\gamma)^3} ( 1 − γ ) 3 1 and 1 ( 1 − γ ) 4 \frac{1}{(1-\gamma)^4} ( 1 − γ ) 4 1 .
Substituting (4.143) into (324) reveals that, for all t ≥ τ ^ t\geq\widehat{\tau} t ≥ τ ,
E [ Δ t , max ] ≥ ∑ k = 1 t η ( 1 − η ) t − k γ p E [ Δ k − 1 , max ] + c ( 1 − γ ) 2 T log T . (326) \mathbb{E}[\Delta_{t,\max}]\geq\sum_{k=1}^{t}\eta(1-\eta)^{t-k}\gamma p\mathbb{E}[\Delta_{k-1,\max}]+\frac{c}{\sqrt{(1-\gamma)^2T\log T}}. \tag*{(326)} E [ Δ t , m a x ] ≥ k = 1 ∑ t η ( 1 − η ) t − k γ p E [ Δ k − 1 , m a x ] + ( 1 − γ ) 2 T log T c . (326)
leaving us with a deterministic recursive relation to solve.
Step 5.4: solving the recursion (326) . To solve the above recursion, it is convenient to introduce the following auxiliary scalar sequence:
x t = ( 1 − η ) x t − 1 + η ( γ p x t − 1 + c ( 1 − γ ) 2 T log T ) . (327) x_t=(1-\eta)x_{t-1}+\eta\left(\gamma p x_{t-1}+\frac{c}{\sqrt{(1-\gamma)^2T\log T}}\right). \tag*{(327)} x t = ( 1 − η ) x t − 1 + η ( γ p x t − 1 + ( 1 − γ ) 2 T log T c ) . (327)
initialized at x τ ^ = 0 x_{\widehat{\tau}}=0 x τ = 0 , where τ ^ \widehat{\tau} τ has been defined in (325) . A direct comparison reveals that
E [ Δ T , max ] ≥ x T , (328) \mathbb{E}[\Delta_{T,\max}]\geq x_T, \tag*{(328)} E [ Δ T , m a x ] ≥ x T , (328)
which follows readily from the iterative expansion of (327) :
x t = ∑ k = τ ^ t η ( 1 − η ) t − k γ p x k − 1 + ∑ k = τ ^ T η ( 1 − η ) t − k c ( 1 − γ ) 2 T log T . x_t=\sum_{k=\widehat{\tau}}^{t}\eta(1-\eta)^{t-k}\gamma p x_{k-1}+\sum_{k=\widehat{\tau}}^{T}\eta(1-\eta)^{t-k}\frac{c}{\sqrt{(1-\gamma)^2T\log T}}. x t = k = τ ∑ t η ( 1 − η ) t − k γ p x k − 1 + k = τ ∑ T η ( 1 − η ) t − k ( 1 − γ ) 2 T log T c .
A straightforward calculation shows that the sequence (327) obeys
x T = c ( 1 − γ ) 2 T log T ∑ k = τ ^ T η ( 1 − η ( 1 − γ p ) ) T − k = c [ 1 − ( 1 − η ( 1 − γ p ) ) T − τ ^ + 1 ] ( 1 − γ p ) ( 1 − γ ) 2 T log T = 3 c [ 1 − ( 1 − η ( 1 − γ p ) ) T − τ ^ + 1 ] 4 ( 1 − γ ) 4 T log T ≥ 3 c 28 ( 1 − γ ) 4 T log T \begin{aligned}
x_T=\frac{c}{\sqrt{(1-\gamma)^2T\log T}}\sum_{k=\widehat{\tau}}^{T}\eta\left(1-\eta(1-\gamma p)\right)^{T-k} \\
&= \frac{c\left[1-\left(1-\eta(1-\gamma p)\right)^{T-\hat{\tau}+1}\right]}{(1-\gamma p)\sqrt{(1-\gamma)^{2T}}\log T} \\
&= \frac{3c\left[1-\left(1-\eta(1-\gamma p)\right)^{T-\hat{\tau}+1}\right]}{4\sqrt{(1-\gamma)^{4T}}\log T}
\ge\frac{3c}{28\sqrt{(1-\gamma)^{4T}}\log T}
\end{aligned} x T = ( 1 − γ ) 2 T log T c k = τ ∑ T η ( 1 − η ( 1 − γ p ) ) T − k = ( 1 − γ p ) ( 1 − γ ) 2 T log T c [ 1 − ( 1 − η ( 1 − γ p ) ) T − τ ^ + 1 ] = 4 ( 1 − γ ) 4 T log T 3 c [ 1 − ( 1 − η ( 1 − γ p ) ) T − τ ^ + 1 ] ≥ 28 ( 1 − γ ) 4 T log T 3 c
where the second equality follows by calculating the sum of the geometric sequence, and the last inequality holds as long as ( 1 − η ( 1 − γ p ) ) T − τ ^ + 1 ≤ 6 / 7 \left(1-\eta(1-\gamma p)\right)^{T-\hat{\tau}+1}\le6/7 ( 1 − η ( 1 − γ p ) ) T − τ ^ + 1 ≤ 6/7 (see (325) ). Combining this with (328) yields
E [ Δ T , max ] ≥ x T ≥ 3 c 28 ( 1 − γ ) 4 T log T . (329) \mathbb{E}\left[\Delta_{T,\max}\right]\ge x_T\ge\frac{3c}{28\sqrt{(1-\gamma)^{4T}}\log T}. \tag*{(329)} E [ Δ T , m a x ] ≥ x T ≥ 28 ( 1 − γ ) 4 T log T 3 c . (329)
Step 5.5: putting all this together. Thus far, we have established the claimed lower bound for the auxiliary sequence Q ^ t \widehat{Q}_t Q t . To complete the argument, it remains to connect it to the original sequence Q t Q_t Q t . To do so, combine the lower bound (329) with (318) to yield
E [ V T ( 1 ) − V ∗ ( 1 ) ] ≥ E [ Δ T , max − 1 1 − γ ( 1 − η ( 1 − γ ) ) T ] ≥ 3 c 28 ( 1 − γ ) 4 T log T − ( 1 − η ( 1 − γ ) ) T 1 − γ . \begin{aligned}
\mathbb{E}\left[V_T(1)-V^*(1)\right]\ge\mathbb{E}\left[\Delta_{T,\max}-\frac{1}{1-\gamma}\left(1-\eta(1-\gamma)\right)^T\right] \\
&\ge\frac{3c}{28\sqrt{(1-\gamma)^{4T}}\log T}-\frac{\left(1-\eta(1-\gamma)\right)^T}{1-\gamma}.
\end{aligned} E [ V T ( 1 ) − V ∗ ( 1 ) ] ≥ E [ Δ T , m a x − 1 − γ 1 ( 1 − η ( 1 − γ ) ) T ] ≥ 28 ( 1 − γ ) 4 T log T 3 c − 1 − γ ( 1 − η ( 1 − γ ) ) T .
As it turns out, the last term in the above display can be easily handled with the aid of V T ( 3 ) V_T(3) V T ( 3 ) . Specifically, taking this result and (305) collectively, we arrive at
E [ V T ( 1 ) − V ∗ ( 1 ) ] + E [ V ∗ ( 3 ) − V T ( 3 ) ] = E [ V T ( 1 ) − V ∗ ( 1 ) ] + ( 1 − η ( 1 − γ ) ) T 1 − γ ≥ 3 c 28 ( 1 − γ ) 4 T log T . \begin{aligned}
\mathbb{E}\left[V_T(1)-V^*(1)\right]+\mathbb{E}\left[V^*(3)-V_T(3)\right]
= \mathbb{E}\left[V_T(1)-V^*(1)\right]+\frac{\left(1-\eta(1-\gamma)\right)^T}{1-\gamma} \\
&\ge\frac{3c}{28\sqrt{(1-\gamma)^{4T}}\log T}.
\end{aligned} E [ V T ( 1 ) − V ∗ ( 1 ) ] + E [ V ∗ ( 3 ) − V T ( 3 ) ] = E [ V T ( 1 ) − V ∗ ( 1 ) ] + 1 − γ ( 1 − η ( 1 − γ ) ) T ≥ 28 ( 1 − γ ) 4 T log T 3 c .
Apply the elementary inequality max { a , b } ≥ 1 2 ( a + b ) \max\{a,b\}\ge\frac{1}{2}(a+b) max { a , b } ≥ 2 1 ( a + b ) to obtain
max { E [ ∣ V T ( 1 ) − V ∗ ( 1 ) ∣ ] , E [ ∣ V T ( 3 ) − V ∗ ( 3 ) ∣ ] } ≥ 1 2 { E [ V T ( 1 ) − V ∗ ( 1 ) ] + E [ V ∗ ( 3 ) − V T ( 3 ) ] } ≥ 3 c 56 ( 1 − γ ) 4 T log T , \begin{aligned}
\max\left\{\mathbb{E}\left[\left|V_T(1)-V^*(1)\right|\right],\mathbb{E}\left[\left|V_T(3)-V^*(3)\right|\right]\right\}
\ge\frac{1}{2}\left\{\mathbb{E}\left[V_T(1)-V^*(1)\right]+\mathbb{E}\left[V^*(3)-V_T(3)\right]\right\} \\
&\ge\frac{3c}{56\sqrt{(1-\gamma)^{4T}}\log T},
\end{aligned} max { E [ ∣ V T ( 1 ) − V ∗ ( 1 ) ∣ ] , E [ ∣ V T ( 3 ) − V ∗ ( 3 ) ∣ ] } ≥ 2 1 { E [ V T ( 1 ) − V ∗ ( 1 ) ] + E [ V ∗ ( 3 ) − V T ( 3 ) ] } ≥ 56 ( 1 − γ ) 4 T log T 3 c ,
yielding
max s E [ ∣ V T ( s ) − V ∗ ( s ) ∣ 2 ] ≥ max s ( E [ ∣ V T ( s ) − V ∗ ( s ) ∣ ] ) 2 ≥ ( 3 c 56 ( 1 − γ ) 4 T log T ) 2 = 9 c 2 56 2 ( 1 − γ ) 4 T log 2 T . \begin{aligned}
\max_s\mathbb{E}\left[\left|V_T(s)-V^*(s)\right|^2\right]\ge\max_s\left(\mathbb{E}\left[\left|V_T(s)-V^*(s)\right|\right]\right)^2 \\
&\ge\left(\frac{3c}{56\sqrt{(1-\gamma)^{4T}}\log T}\right)^2
= \frac{9c^2}{56^2(1-\gamma)^{4T}\log^2 T}.
\end{aligned} s max E [ ∣ V T ( s ) − V ∗ ( s ) ∣ 2 ] ≥ s max ( E [ ∣ V T ( s ) − V ∗ ( s ) ∣ ] ) 2 ≥ ( 56 ( 1 − γ ) 4 T log T 3 c ) 2 = 5 6 2 ( 1 − γ ) 4 T log 2 T 9 c 2 .
We have thus established the desired lower bound for the moderate learning-rate regime, which, together with the bounds in (308) and (315) for the other two regimes, completes the proof of Theorem 4.7.
Proof of auxiliary properties
Proof of property (318) . We prove this claim by induction. The base case t = 0 t=0 t = 0 follows trivially. Assume now the hypothesis holds for t − 1 t-1 t − 1 . By combining the inductive hypothesis (318) for the ( t − 1 ) (t-1) ( t − 1 ) -th iteration with the definitions of V t − 1 V_{t-1} V t − 1 and V ^ t − 1 \widehat{V}_{t-1} V t − 1 , we obtain
V t − 1 ( 1 ) = max a Q t − 1 ( 1 , a ) ≥ max a Q ^ t − 1 ( a ) − ( 1 − η ( 1 − γ ) ) t − 1 1 − γ = V ^ t − 1 − ( 1 − η ( 1 − γ ) ) t − 1 1 − γ . \begin{aligned}
V_{t-1}(1)=\max_a Q_{t-1}(1,a)\geq\max_a\widehat{Q}_{t-1}(a)-\frac{\left(1-\eta(1-\gamma)\right)^{t-1}}{1-\gamma} \\
&=\widehat{V}_{t-1}-\frac{\left(1-\eta(1-\gamma)\right)^{t-1}}{1-\gamma}.
\end{aligned} V t − 1 ( 1 ) = a max Q t − 1 ( 1 , a ) ≥ a max Q t − 1 ( a ) − 1 − γ ( 1 − η ( 1 − γ ) ) t − 1 = V t − 1 − 1 − γ ( 1 − η ( 1 − γ ) ) t − 1 .
Given the update rules of Q t ( 1 , a ) Q_t(1,a) Q t ( 1 , a ) and Q ^ t ( a ) \widehat{Q}_t(a) Q t ( a ) , their difference evolves as
Q t ( 1 , a ) − Q ^ t ( a ) = ( 1 − η ) ( Q t − 1 ( 1 , a ) − Q ^ t − 1 ( a ) ) + γ P t ( 1 ∣ 1 , a ) ( V t − 1 ( 1 ) − V ^ t − 1 ) ≥ − ( 1 − η ) ( 1 − η ( 1 − γ ) ) t − 1 1 − γ − η γ P t ( 1 ∣ 1 , a ) ( 1 − η ( 1 − γ ) ) t − 1 1 − γ ≥ − ( 1 − η ) ( 1 − η ( 1 − γ ) ) t − 1 1 − γ − η γ ( 1 − η ( 1 − γ ) ) t − 1 1 − γ = − ( 1 − η ( 1 − γ ) ) t 1 − γ . \begin{aligned}
Q_t(1,a)-\widehat{Q}_t(a)=(1-\eta)\left(Q_{t-1}(1,a)-\widehat{Q}_{t-1}(a)\right)+\gamma P_t(1\mid1,a)\left(V_{t-1}(1)-\widehat{V}_{t-1}\right) \\
&\geq-(1-\eta)\frac{\left(1-\eta(1-\gamma)\right)^{t-1}}{1-\gamma}-\eta\gamma P_t(1\mid1,a)\frac{\left(1-\eta(1-\gamma)\right)^{t-1}}{1-\gamma} \\
&\geq-(1-\eta)\frac{\left(1-\eta(1-\gamma)\right)^{t-1}}{1-\gamma}-\eta\gamma\frac{\left(1-\eta(1-\gamma)\right)^{t-1}}{1-\gamma} \\
&=-\frac{\left(1-\eta(1-\gamma)\right)^t}{1-\gamma}.
\end{aligned} Q t ( 1 , a ) − Q t ( a ) = ( 1 − η ) ( Q t − 1 ( 1 , a ) − Q t − 1 ( a ) ) + γ P t ( 1 ∣ 1 , a ) ( V t − 1 ( 1 ) − V t − 1 ) ≥ − ( 1 − η ) 1 − γ ( 1 − η ( 1 − γ ) ) t − 1 − η γ P t ( 1 ∣ 1 , a ) 1 − γ ( 1 − η ( 1 − γ ) ) t − 1 ≥ − ( 1 − η ) 1 − γ ( 1 − η ( 1 − γ ) ) t − 1 − η γ 1 − γ ( 1 − η ( 1 − γ ) ) t − 1 = − 1 − γ ( 1 − η ( 1 − γ ) ) t .
where the first inequality invokes the induction hypothesis for the ( t − 1 ) (t-1) ( t − 1 ) -th iteration, and the second follows since P t ( 1 ∣ 1 , a ) ≤ 1 P_t(1\mid1,a)\leq1 P t ( 1 ∣ 1 , a ) ≤ 1 . This completes the inductive step and, in turn, establishes the result for all t ≥ 0 t\geq0 t ≥ 0 .
Proof of property (319) . We find it convenient to introduce a second auxiliary sequence defined by the recursion:
Q ‾ t = ( 1 − η ) Q ‾ t − 1 + η { 1 + γ P t ( 1 ∣ 1 , 1 ) Q ‾ t − 1 } , (330) \overline{Q}_t=(1-\eta)\overline{Q}_{t-1}+\eta\{1+\gamma P_t(1\mid1,1)\overline{Q}_{t-1}\}, \tag*{(330)} Q t = ( 1 − η ) Q t − 1 + η { 1 + γ P t ( 1 ∣ 1 , 1 ) Q t − 1 } , (330)
Q ‾ 0 = V ∗ ( 1 ) = 1 1 − γ p . (331) \overline{Q}_0=V^*(1)=\frac{1}{1-\gamma p}. \tag*{(331)} Q 0 = V ∗ ( 1 ) = 1 − γ p 1 . (331)
In words, this sequence corresponds to the Q-learning iterates in a simplified setting with only a single available action, effectively removing the max operator. Given that V ^ t = max a Q ^ t ( a ) ≥ Q ^ t ( 1 ) \widehat{V}_t=\max_a\widehat{Q}_t(a)\geq\widehat{Q}_t(1) V t = max a Q t ( a ) ≥ Q t ( 1 ) , it is straightforward to verify that, for all t ≥ 0 t\geq0 t ≥ 0 ,
Q ^ t ( 1 ) ≥ ( 1 − η ) Q ^ t − 1 ( 1 ) + η { 1 + γ P t ( 1 ∣ 1 , 1 ) Q ^ t − 1 ( 1 ) } ≥ Q ‾ t . (332) \widehat{Q}_t(1)\geq(1-\eta)\widehat{Q}_{t-1}(1)+\eta\{1+\gamma P_t(1\mid1,1)\widehat{Q}_{t-1}(1)\}\geq\overline{Q}_t. \tag*{(332)} Q t ( 1 ) ≥ ( 1 − η ) Q t − 1 ( 1 ) + η { 1 + γ P t ( 1 ∣ 1 , 1 ) Q t − 1 ( 1 )} ≥ Q t . (332)
We use the sequence Q ‾ t \overline{Q}_t Q t to lower-bound V ^ t \widehat{V}_t V t , as its simpler structure allows us to more easily characterize the evolution of the lower envelope of the Q-iterates. First, observe that Q ‾ t \overline{Q}_t Q t admits the decomposition:
Q ‾ t = ( 1 − η ( 1 − γ p ) ) Q ‾ t − 1 + η { 1 + γ ( P t ( 1 ∣ 1 , 1 ) − p ) Q ‾ t − 1 } = ( 1 − η ( 1 − γ p ) ) t Q ‾ 0 + ∑ k = 1 t η ( 1 − η ( 1 − γ p ) ) t − k { 1 + γ ( P k ( 1 ∣ 1 , 1 ) − p ) Q ‾ k − 1 } = η 0 ( t ) 1 1 − γ p + ∑ k = 1 t η k ( t ) + ∑ k = 1 t η k ( t ) γ ( P k ( 1 ∣ 1 , 1 ) − p ) Q ‾ k − 1 = 1 1 − γ p + ∑ k = 1 t η k ( t ) γ ( P k ( 1 ∣ 1 , 1 ) − p ) Q ‾ k − 1 ⏟ = : z k . (333) \begin{aligned}
\overline{Q}_t&=(1-\eta(1-\gamma p))\overline{Q}_{t-1}+\eta\{1+\gamma(P_t(1\mid1,1)-p)\overline{Q}_{t-1}\}\\
&=(1-\eta(1-\gamma p))^t\overline{Q}_0+\sum_{k=1}^{t}\eta(1-\eta(1-\gamma p))^{t-k}\{1+\gamma(P_k(1\mid1,1)-p)\overline{Q}_{k-1}\}\\
&=\eta_0^{(t)}\frac{1}{1-\gamma p}+\sum_{k=1}^{t}\eta_k^{(t)}+\sum_{k=1}^{t}\eta_k^{(t)}\gamma(P_k(1\mid1,1)-p)\overline{Q}_{k-1}\\
&=\frac{1}{1-\gamma p}+\sum_{k=1}^{t}\underbrace{\eta_k^{(t)}\gamma(P_k(1\mid1,1)-p)\overline{Q}_{k-1}}_{=:z_k}. \tag*{(333)}
\end{aligned} Q t = ( 1 − η ( 1 − γ p )) Q t − 1 + η { 1 + γ ( P t ( 1 ∣ 1 , 1 ) − p ) Q t − 1 } = ( 1 − η ( 1 − γ p ) ) t Q 0 + k = 1 ∑ t η ( 1 − η ( 1 − γ p ) ) t − k { 1 + γ ( P k ( 1 ∣ 1 , 1 ) − p ) Q k − 1 } = η 0 ( t ) 1 − γ p 1 + k = 1 ∑ t η k ( t ) + k = 1 ∑ t η k ( t ) γ ( P k ( 1 ∣ 1 , 1 ) − p ) Q k − 1 = 1 − γ p 1 + k = 1 ∑ t =: z k η k ( t ) γ ( P k ( 1 ∣ 1 , 1 ) − p ) Q k − 1 . (333)
where the last line results from (292) . It then comes down to controlling ∑ k z k \sum_k z_k ∑ k z k .
By construction, the sequence { z k } \{z_k\} { z k } forms a martingale satisfying
E [ z k ∣ P k − 1 ( 1 ∣ 1 , 1 ) , … , P 1 ( 1 ∣ 1 , 1 ) ] = 0 and ∣ z k ∣ ≤ max 1 ≤ k ≤ t η k ( t ) ⋅ γ p 1 − γ . \mathbb{E}\left[z_k \mid P_{k-1}(1 \mid1,1),\ldots,P_1(1 \mid1,1)\right]=0
\quad\text{and}\quad
|z_k|\leq\max_{1\leq k\leq t}\eta_k^{(t)}\cdot\frac{\gamma p}{1-\gamma}. E [ z k ∣ P k − 1 ( 1 ∣ 1 , 1 ) , … , P 1 ( 1 ∣ 1 , 1 ) ] = 0 and ∣ z k ∣ ≤ 1 ≤ k ≤ t max η k ( t ) ⋅ 1 − γ γ p .
where the last inequality follows from the basic property 0 ≤ Q ‾ k − 1 ≤ 1 1 − γ 0\leq\overline{Q}_{k-1}\leq\frac{1}{1-\gamma} 0 ≤ Q k − 1 ≤ 1 − γ 1 (akin to Lemma 4.8) and the fact that ∣ P k ( 1 ∣ 1 , 1 ) − p ∣ ≤ max { p , 1 − p } = p |P_k(1\mid1,1)-p|\leq\max\{p,1-p\}=p ∣ P k ( 1 ∣ 1 , 1 ) − p ∣ ≤ max { p , 1 − p } = p since p = 4 γ − 1 3 γ p=\frac{4\gamma-1}{3\gamma} p = 3 γ 4 γ − 1 and γ ≥ 3 / 4 \gamma\geq3/4 γ ≥ 3/4 . In order to invoke Freedman’s inequality to control (333) , we need to bound the conditional variance. Given these properties and the fact that P k ( 1 ∣ 1 , 1 ) P_k(1\mid1,1) P k ( 1 ∣ 1 , 1 ) is a Bernoulli random variable with mean p p p , we obtain
∑ k = 1 t Var ( z k ∣ P k − 1 ( 1 ∣ 1 , 1 ) , … , P 1 ( 1 ∣ 1 , 1 ) ) = ∑ k = 1 t ( η k ( t ) ) 2 p ( 1 − p ) ( γ Q ‾ k − 1 ) 2 ≤ max 1 ≤ k ≤ t η k ( t ) ⋅ ∑ k = 1 t η k ( t ) ⋅ 1 3 ( 1 − γ ) ≤ max 1 ≤ k ≤ t η k ( t ) 4 ( 1 − γ ) 2 . \begin{aligned}
\sum_{k=1}^{t}\operatorname{Var}\left(z_k\mid P_{k-1}(1\mid1,1),\ldots,P_1(1\mid1,1)\right)
=\sum_{k=1}^{t}\left(\eta_k^{(t)}\right)^2p(1-p)\left(\gamma\overline{Q}_{k-1}\right)^2 \\
&\leq\max_{1\leq k\leq t}\eta_k^{(t)}\cdot\sum_{k=1}^{t}\eta_k^{(t)}\cdot\frac{1}{3(1-\gamma)}
\leq\frac{\max_{1\leq k\leq t}\eta_k^{(t)}}{4(1-\gamma)^2}.
\end{aligned} k = 1 ∑ t Var ( z k ∣ P k − 1 ( 1 ∣ 1 , 1 ) , … , P 1 ( 1 ∣ 1 , 1 ) ) = k = 1 ∑ t ( η k ( t ) ) 2 p ( 1 − p ) ( γ Q k − 1 ) 2 ≤ 1 ≤ k ≤ t max η k ( t ) ⋅ k = 1 ∑ t η k ( t ) ⋅ 3 ( 1 − γ ) 1 ≤ 4 ( 1 − γ ) 2 max 1 ≤ k ≤ t η k ( t ) .
Here, the penultimate inequality relies on the fact 0 ≤ Q ‾ k − 1 ≤ 1 1 − γ 0\leq\overline{Q}_{k-1}\leq\frac{1}{1-\gamma} 0 ≤ Q k − 1 ≤ 1 − γ 1 (akin to Lemma 4.8) and the choice of p p p , whereas the last inequality results from the following condition (derived through (292) )
∑ k = 1 t η k ( t ) = ( 1 − η 0 ( t ) ) 1 1 − γ p ≤ 1 1 − γ p = 3 4 ( 1 − γ ) . \sum_{k=1}^{t}\eta_k^{(t)}=(1-\eta_0^{(t)})\frac{1}{1-\gamma p}\leq\frac{1}{1-\gamma p}=\frac{3}{4(1-\gamma)}. k = 1 ∑ t η k ( t ) = ( 1 − η 0 ( t ) ) 1 − γ p 1 ≤ 1 − γ p 1 = 4 ( 1 − γ ) 3 .
Freedman’s inequality (see Lemma 2.9 with K 0 = 1 K_0=1 K 0 = 1 ) then implies that
∣ ∑ k = 1 t z k ∣ ≤ 4 max 1 ≤ k ≤ t η k ( t ) ( 1 − γ ) 2 log 2 δ + 4 max 1 ≤ k ≤ t η k ( t ) 3 ( 1 − γ ) log 2 δ (334) \left|\sum_{k=1}^{t}z_k\right|
\leq
\sqrt{\frac{4\max_{1\leq k\leq t}\eta_k^{(t)}}{(1-\gamma)^2}\log\frac{2}{\delta}
+\frac{4\max_{1\leq k\leq t}\eta_k^{(t)}}{3(1-\gamma)}\log\frac{2}{\delta}} \tag*{(334)} k = 1 ∑ t z k ≤ ( 1 − γ ) 2 4 max 1 ≤ k ≤ t η k ( t ) log δ 2 + 3 ( 1 − γ ) 4 max 1 ≤ k ≤ t η k ( t ) log δ 2 (334)
with probability at least 1 − δ 1-\delta 1 − δ .
To continue, observe that
max 1 ≤ k ≤ t η k ( t ) ≤ η ≤ 1 ( 1 − γ ) 2 T ≤ 1 50 , for all t ≥ 1 , (335) \max_{1\leq k\leq t}\eta_k^{(t)}\leq\eta\leq\frac{1}{(1-\gamma)^2T}\leq\frac{1}{50},
\qquad\text{for all }t\geq1, \tag*{(335)} 1 ≤ k ≤ t max η k ( t ) ≤ η ≤ ( 1 − γ ) 2 T 1 ≤ 50 1 , for all t ≥ 1 , (335)
provided that T ≥ 50 ( 1 − γ ) 2 T\geq\frac{50}{(1-\gamma)^2} T ≥ ( 1 − γ ) 2 50 . Setting δ = 1 / 2 \delta=1/2 δ = 1/2 in (334) then yields
∑ k = 1 t z k ≥ − 1 2 ( 1 − γ ) \sum_{k=1}^{t}z_k\geq-\frac{1}{2(1-\gamma)} k = 1 ∑ t z k ≥ − 2 ( 1 − γ ) 1
with probability at least 1 / 2 1/2 1/2 . Combine this with the decomposition (333) and the property (332) to conclude
V ^ t ≥ Q ^ t ( 1 ) ≥ Q ‾ t ≥ 1 1 − γ p − 1 2 ( 1 − γ ) = 1 4 ( 1 − γ ) \widehat{V}_t\geq\widehat{Q}_t(1)\geq\overline{Q}_t\geq\frac{1}{1-\gamma p}-\frac{1}{2(1-\gamma)}
=\frac{1}{4(1-\gamma)} V t ≥ Q t ( 1 ) ≥ Q t ≥ 1 − γ p 1 − 2 ( 1 − γ ) 1 = 4 ( 1 − γ ) 1
with probability at least 1 / 2 1/2 1/2 , where the last identity relies on the choice (288) of p p p . This completes the proof of (319) .
Proof of Lemma 4.15
In light of the definition (4.140), it is convenient to express, for every t t t ,
E [ ξ t , max ] = E [ ξ t ( 1 ) + ξ t ( 2 ) + ∣ ξ t ( 1 ) − ξ t ( 2 ) ∣ 2 ] = E [ ∣ ξ t ( 1 ) − ξ t ( 2 ) ∣ 2 ] . = 1 2 E [ ∣ ∑ k = 1 t η ( 1 − η ) t − k γ ( P k ( 1 ∣ 1 , 1 ) − P k ( 1 ∣ 1 , 2 ) ) V ^ k − 1 ∣ ] , \begin{aligned}
\mathbb{E}[\xi_{t,\max}]
=\mathbb{E}\left[\frac{\xi_t(1)+\xi_t(2)+|\xi_t(1)-\xi_t(2)|}{2}\right]
=\mathbb{E}\left[\frac{|\xi_t(1)-\xi_t(2)|}{2}\right]. \\
&= \frac{1}{2}\mathbb{E}\left[\left|\sum_{k=1}^{t}\eta(1-\eta)^{t-k}\gamma\left(P_k(1\mid1,1)-P_k(1\mid1,2)\right)\widehat{V}_{k-1}\right|\right],
\end{aligned} E [ ξ t , m a x ] = E [ 2 ξ t ( 1 ) + ξ t ( 2 ) + ∣ ξ t ( 1 ) − ξ t ( 2 ) ∣ ] = E [ 2 ∣ ξ t ( 1 ) − ξ t ( 2 ) ∣ ] . = 2 1 E [ k = 1 ∑ t η ( 1 − η ) t − k γ ( P k ( 1 ∣ 1 , 1 ) − P k ( 1 ∣ 1 , 2 ) ) V k − 1 ] ,
where we have used the fact that E [ ξ t ( a ) ] = 0 \mathbb{E}[\xi_t(a)]=0 E [ ξ t ( a )] = 0 . Importantly, the summation within the absolute value operator is a martingale sum. To be more precise, define
ζ t : = ∑ k = 1 t η ( 1 − η ) t − k γ ( P k ( 1 ∣ 1 , 1 ) − P k ( 1 ∣ 1 , 2 ) ) V ^ k − 1 , \zeta_t := \sum_{k=1}^{t}\eta(1-\eta)^{t-k}\gamma\left(P_k(1\mid1,1)-P_k(1\mid1,2)\right)\widehat{V}_{k-1}, ζ t := k = 1 ∑ t η ( 1 − η ) t − k γ ( P k ( 1 ∣ 1 , 1 ) − P k ( 1 ∣ 1 , 2 ) ) V k − 1 ,
where the underbraced summand is labeled = : z k =:z_k =: z k .
where { z k } \{z_k\} { z k } forms a martingale sequence since
E [ z k ∣ { P j ( 1 ∣ 1 , 1 ) , P j ( 1 ∣ 1 , 2 ) } 1 ≤ j < k ] = 0. \mathbb{E}\left[z_k\mid\left\{P_j(1\mid1,1),P_j(1\mid1,2)\right\}_{1\leq j<k}\right]=0. E [ z k ∣ { P j ( 1 ∣ 1 , 1 ) , P j ( 1 ∣ 1 , 2 ) } 1 ≤ j < k ] = 0.
In order to control E [ ∣ ζ t ∣ ] \mathbb{E}[|\zeta_t|] E [ ∣ ζ t ∣ ] , we first seek to analyze the tail of ∣ ζ t ∣ |\zeta_t| ∣ ζ t ∣ . To do so, we claim that, for any 0 < δ < 1 0<\delta<1 0 < δ < 1 ,
P { ∣ ζ t ∣ ≥ 8 log 2 δ 3 ( 1 − γ ) ∑ k = 1 t η 2 [ ( 1 − η ) t − k ] 2 + 4 η log 2 δ 3 ( 1 − γ ) } ≤ δ . (336) \mathbb{P}\left\{|\zeta_t|\geq\sqrt{\frac{8\log\frac{2}{\delta}}{3(1-\gamma)}\sum_{k=1}^{t}\eta^2\left[(1-\eta)^{t-k}\right]^2+\frac{4\eta\log\frac{2}{\delta}}{3(1-\gamma)}}\right\}\leq\delta. \tag*{(336)} P ⎩ ⎨ ⎧ ∣ ζ t ∣ ≥ 3 ( 1 − γ ) 8 log δ 2 k = 1 ∑ t η 2 [ ( 1 − η ) t − k ] 2 + 3 ( 1 − γ ) 4 η log δ 2 ⎭ ⎬ ⎫ ≤ δ . (336)
To validate this bound, we first notice that
∣ z k ∣ ≤ max k : 1 ≤ k ≤ t η ( 1 − η ) t − k ⋅ 1 1 − γ ≤ η 1 − γ , (337) |z_k|\leq\max_{k:1\leq k\leq t}\eta(1-\eta)^{t-k}\cdot\frac{1}{1-\gamma}\leq\frac{\eta}{1-\gamma}, \tag*{(337)} ∣ z k ∣ ≤ k : 1 ≤ k ≤ t max η ( 1 − η ) t − k ⋅ 1 − γ 1 ≤ 1 − γ η , (337)
where we have used ∣ V ^ k − 1 ∣ ≤ 1 1 − γ |\widehat{V}_{k-1}|\leq\frac{1}{1-\gamma} ∣ V k − 1 ∣ ≤ 1 − γ 1 according to Lemma Lemma 4.8 . In addition, we can further bound the aggregate condition variance as
∑ k = 1 t Var ( z k ∣ P k − 1 ( 1 ∣ 1 , 1 ) , P k − 1 ( 1 ∣ 1 , 2 ) , … , P 1 ( 1 ∣ 1 , 1 ) , P 1 ( 1 ∣ 1 , 2 ) ) = ∑ k = 1 t η 2 [ ( 1 − η ) t − k ] 2 ⋅ 2 p ( 1 − p ) ⋅ ( γ V ^ k − 1 ) 2 ≤ ∑ k = 1 t η 2 [ ( 1 − η ) t − k ] 2 ⋅ 2 3 ( 1 − γ ) , \begin{aligned}
\sum_{k=1}^{t}\operatorname{Var}\left(z_k\mid P_{k-1}(1\mid1,1),P_{k-1}(1\mid1,2),\ldots,P_1(1\mid1,1),P_1(1\mid1,2)\right) \\
&=\sum_{k=1}^{t}\eta^2\left[(1-\eta)^{t-k}\right]^2\cdot2p(1-p)\cdot\left(\gamma\widehat{V}_{k-1}\right)^2 \\
&\leq\sum_{k=1}^{t}\eta^2\left[(1-\eta)^{t-k}\right]^2\cdot\frac{2}{3(1-\gamma)},
\end{aligned} k = 1 ∑ t Var ( z k ∣ P k − 1 ( 1 ∣ 1 , 1 ) , P k − 1 ( 1 ∣ 1 , 2 ) , … , P 1 ( 1 ∣ 1 , 1 ) , P 1 ( 1 ∣ 1 , 2 ) ) = k = 1 ∑ t η 2 [ ( 1 − η ) t − k ] 2 ⋅ 2 p ( 1 − p ) ⋅ ( γ V k − 1 ) 2 ≤ k = 1 ∑ t η 2 [ ( 1 − η ) t − k ] 2 ⋅ 3 ( 1 − γ ) 2 ,
where the last inequality comes from the fact that ∣ V ^ k − 1 ∣ ≤ 1 1 − γ |\widehat{V}_{k-1}|\leq\frac{1}{1-\gamma} ∣ V k − 1 ∣ ≤ 1 − γ 1 , the assumption that γ ≥ 3 / 4 \gamma\geq3/4 γ ≥ 3/4 and the choice p = 4 γ − 1 3 γ p=\frac{4\gamma-1}{3\gamma} p = 3 γ 4 γ − 1 . These bounds, taken together with Freedman’s inequality (see Lemma 2.9 with K 0 = 1 K_0=1 K 0 = 1 ), justify the tail bound (336) .
Next, we make note of a crude deterministic bound:
∣ ζ t ∣ ≤ ∑ k = 1 t ∣ z k ∣ ≤ [ ∑ k = 1 t η ( 1 − η ) t − k ] ⋅ 1 1 − γ ≤ 1 1 − γ , (338) |\zeta_t|\leq\sum_{k=1}^{t}|z_k|\leq\left[\sum_{k=1}^{t}\eta(1-\eta)^{t-k}\right]\cdot\frac{1}{1-\gamma}\leq\frac{1}{1-\gamma}, \tag*{(338)} ∣ ζ t ∣ ≤ k = 1 ∑ t ∣ z k ∣ ≤ [ k = 1 ∑ t η ( 1 − η ) t − k ] ⋅ 1 − γ 1 ≤ 1 − γ 1 , (338)
which is a consequence of the facts that ∣ V ^ k − 1 ∣ ≤ 1 1 − γ |\widehat{V}_{k-1}|\leq\frac{1}{1-\gamma} ∣ V k − 1 ∣ ≤ 1 − γ 1 (see Lemma 4.8 ) and ∑ k = 1 t η ( 1 − η ) t − k ≤ η ⋅ 1 − ( 1 − η ) t 1 − ( 1 − η ) = 1 \sum_{k=1}^{t}\eta(1-\eta)^{t-k}\leq\eta\cdot\frac{1-(1-\eta)^t}{1-(1-\eta)}=1 ∑ k = 1 t η ( 1 − η ) t − k ≤ η ⋅ 1 − ( 1 − η ) 1 − ( 1 − η ) t = 1 . This immediately implies that
δ 0 : = ( 1 − γ ) 2 2 E [ ∣ ζ t ∣ 2 ] ≤ 1 2 . \delta_0:=\frac{(1-\gamma)^2}{2}\mathbb{E}[|\zeta_t|^2]\leq\frac{1}{2}. δ 0 := 2 ( 1 − γ ) 2 E [ ∣ ζ t ∣ 2 ] ≤ 2 1 .
Thus, setting δ = δ 0 ∈ ( 0 , 1 ) \delta= \delta_0 \in(0,1) δ = δ 0 ∈ ( 0 , 1 ) in (336) reveals that, with probability at least 1 − δ 0 1-\delta_0 1 − δ 0 ,
∣ ζ t ∣ ≤ B : = 8 log 2 δ 0 3 ( 1 − γ ) ∑ k = 1 t η 2 [ ( 1 − η ) t − k ] 2 + 4 η log 2 δ 0 3 ( 1 − γ ) . (339) |\zeta_t| \le B := \sqrt{\frac{8\log\frac{2}{\delta_0}}{3(1-\gamma)}\sum_{k=1}^{t}\eta^2\left[(1-\eta)^{t-k}\right]^2+\frac{4\eta\log\frac{2}{\delta_0}}{3(1-\gamma)}}. \tag*{(339)} ∣ ζ t ∣ ≤ B := 3 ( 1 − γ ) 8 log δ 0 2 k = 1 ∑ t η 2 [ ( 1 − η ) t − k ] 2 + 3 ( 1 − γ ) 4 η log δ 0 2 . (339)
When T ≥ 1 ( 1 − γ ) 2 T \ge\frac{1}{(1-\gamma)^2} T ≥ ( 1 − γ ) 2 1 , one can guarantee that
E [ ξ t , max ] = 1 2 E [ ∣ ζ t ∣ ] ≥ 1 2 E [ ∣ ζ t ∣ 1 ( ∣ ζ t ∣ ≤ B ) ] ≥ 1 2 B E [ ∣ ζ t ∣ 2 1 ( ∣ ζ t ∣ ≤ B ) ] = 1 2 B { E [ ∣ ζ t ∣ 2 ] − E [ ∣ ζ t ∣ 2 1 ( ∣ ζ t ∣ > B ) ] } ≥ (i) 1 2 B { E [ ∣ ζ t ∣ 2 ] − 1 ( 1 − γ ) 2 P ( ∣ ζ t ∣ > B ) } ≥ 1 2 B { E [ ∣ ζ t ∣ 2 ] − δ ( 1 − γ ) 2 } ≥ (ii) 1 4 B E [ ∣ ζ t ∣ 2 ] . (340) \begin{aligned}
\mathbb{E}[\xi_{t,\max}]
&=\frac{1}{2}\mathbb{E}[|\zeta_t|]
\ge\frac{1}{2}\mathbb{E}[|\zeta_t|\mathbf{1}(|\zeta_t|\le B)]
\ge\frac{1}{2B}\mathbb{E}[|\zeta_t|^2\mathbf{1}(|\zeta_t|\le B)]\\
&=\frac{1}{2B}\left\{\mathbb{E}[|\zeta_t|^2]-\mathbb{E}[|\zeta_t|^2\mathbf{1}(|\zeta_t|>B)]\right\}\\
&\mathrel{\overset{\text{(i)}}{\ge}}\frac{1}{2B}\left\{\mathbb{E}[|\zeta_t|^2]-\frac{1}{(1-\gamma)^2}\mathbb{P}(|\zeta_t|>B)\right\}\\
&\ge\frac{1}{2B}\left\{\mathbb{E}[|\zeta_t|^2]-\frac{\delta}{(1-\gamma)^2}\right\}
\mathrel{\overset{\text{(ii)}}{\ge}}\frac{1}{4B}\mathbb{E}[|\zeta_t|^2]. \tag*{(340)}
\end{aligned} E [ ξ t , m a x ] = 2 1 E [ ∣ ζ t ∣ ] ≥ 2 1 E [ ∣ ζ t ∣ 1 ( ∣ ζ t ∣ ≤ B )] ≥ 2 B 1 E [ ∣ ζ t ∣ 2 1 ( ∣ ζ t ∣ ≤ B )] = 2 B 1 { E [ ∣ ζ t ∣ 2 ] − E [ ∣ ζ t ∣ 2 1 ( ∣ ζ t ∣ > B )] } ≥ (i) 2 B 1 { E [ ∣ ζ t ∣ 2 ] − ( 1 − γ ) 2 1 P ( ∣ ζ t ∣ > B ) } ≥ 2 B 1 { E [ ∣ ζ t ∣ 2 ] − ( 1 − γ ) 2 δ } ≥ (ii) 4 B 1 E [ ∣ ζ t ∣ 2 ] . (340)
Here, (i) follows from (338) , while (ii) holds due to the choice of δ 0 \delta_0 δ 0 .
It thus suffices to lower bound E [ ∣ ζ t ∣ 2 ] \mathbb{E}[|\zeta_t|^2] E [ ∣ ζ t ∣ 2 ] for t ≥ τ ^ t\ge\hat{\tau} t ≥ τ ^ , where we recall the definition
τ ^ : = max { τ | ( 1 − η ( 1 − γ p ) ) T − τ + 1 ≤ 6 7 } . (341) \hat{\tau}:=\max\left\{\tau\ \middle|\ (1-\eta(1-\gamma^p))^{T-\tau+1}\le\frac{6}{7}\right\}. \tag*{(341)} τ ^ := max { τ ( 1 − η ( 1 − γ p ) ) T − τ + 1 ≤ 7 6 } . (341)
According to (316) , it satisfies 1 ≤ τ ^ ≤ T 1\le\hat{\tau}\le T 1 ≤ τ ^ ≤ T . First, observe that for any t ≥ τ ^ t\ge\hat{\tau} t ≥ τ ^ , the martingale properties about { z k } \{z_k\} { z k } give
E [ ∣ ζ t ∣ 2 ] ≥ ∑ k = 1 t E [ Var ( η ( 1 − η ) t − k γ ( P k ( 1 ∣ 1 , 1 ) − P k ( 1 ∣ 1 , 2 ) ) V ^ k − 1 ∣ V ^ k − 1 ) ] ≥ ∑ k = 1 t E [ Var ( η ( 1 − η ) t − k γ ( P k ( 1 ∣ 1 , 1 ) − P k ( 1 ∣ 1 , 2 ) ) V ^ k − 1 ∣ V ^ k − 1 ) ⋅ 1 { V ^ k − 1 ≥ 1 4 ( 1 − γ ) } ] ≥ ∑ k = 1 t η 2 [ ( 1 − η ) t − k ] 2 γ 2 ⋅ 2 p ( 1 − p ) 16 ( 1 − γ ) 2 P { V ^ k − 1 ≥ 1 4 ( 1 − γ ) } , \begin{aligned}
\mathbb{E}[|\zeta_t|^2]
&\ge\sum_{k=1}^{t}\mathbb{E}\left[\operatorname{Var}\left(\eta(1-\eta)^{t-k}\gamma(P_k(1\mid1,1)-P_k(1\mid1,2))\widehat{V}_{k-1}\mid\widehat{V}_{k-1}\right)\right]\\
&\ge\sum_{k=1}^{t}\mathbb{E}\left[\operatorname{Var}\left(\eta(1-\eta)^{t-k}\gamma(P_k(1\mid1,1)-P_k(1\mid1,2))\widehat{V}_{k-1}\mid\widehat{V}_{k-1}\right)\right.\\
&\qquad\left.\cdot\mathbf{1}\left\{\widehat{V}_{k-1}\ge\frac{1}{4(1-\gamma)}\right\}\right]\\
&\ge\sum_{k=1}^{t}\eta^2\left[(1-\eta)^{t-k}\right]^2\gamma^2\cdot\frac{2p(1-p)}{16(1-\gamma)^2}\mathbb{P}\left\{\widehat{V}_{k-1}\ge\frac{1}{4(1-\gamma)}\right\},
\end{aligned} E [ ∣ ζ t ∣ 2 ] ≥ k = 1 ∑ t E [ Var ( η ( 1 − η ) t − k γ ( P k ( 1 ∣ 1 , 1 ) − P k ( 1 ∣ 1 , 2 )) V k − 1 ∣ V k − 1 ) ] ≥ k = 1 ∑ t E [ Var ( η ( 1 − η ) t − k γ ( P k ( 1 ∣ 1 , 1 ) − P k ( 1 ∣ 1 , 2 )) V k − 1 ∣ V k − 1 ) ⋅ 1 { V k − 1 ≥ 4 ( 1 − γ ) 1 } ] ≥ k = 1 ∑ t η 2 [ ( 1 − η ) t − k ] 2 γ 2 ⋅ 16 ( 1 − γ ) 2 2 p ( 1 − p ) P { V k − 1 ≥ 4 ( 1 − γ ) 1 } ,
where the first line relies on (311) , and the last step makes use of the constraint V ^ k − 1 ≥ 1 4 ( 1 − γ ) \widehat{V}_{k-1}\ge\frac{1}{4(1-\gamma)} V k − 1 ≥ 4 ( 1 − γ ) 1 .
To further control the right-hand side, we take τ ′ : = max { t − 1 η , 1 } \tau':=\max\left\{t-\frac{1}{\eta},1\right\} τ ′ := max { t − η 1 , 1 } and show that
E [ ∣ ζ t ∣ 2 ] ≥ (i) 1 2 ∑ k = 1 t η 2 [ ( 1 − η ) t − k ] 2 γ 2 ⋅ 2 p ( 1 − p ) 16 ( 1 − γ ) 2 ≥ 1 48 ( 1 − γ ) ∑ k = 1 t η 2 [ ( 1 − η ) t − k ] 2 ≥ 1 48 ( 1 − γ ) ∑ k = τ ′ t η 2 [ ( 1 − η ) t − k ] 2 ≥ (ii) η 1200 ( 1 − γ ) . (342) \begin{aligned}
\mathbb{E}[|\zeta_t|^2]
&\mathrel{\overset{\text{(i)}}{\ge}}\frac{1}{2}\sum_{k=1}^{t}\eta^2\left[(1-\eta)^{t-k}\right]^2\gamma^2\cdot\frac{2p(1-p)}{16(1-\gamma)^2}\\
&\ge\frac{1}{48(1-\gamma)}\sum_{k=1}^{t}\eta^2\left[(1-\eta)^{t-k}\right]^2\\
&\ge\frac{1}{48(1-\gamma)}\sum_{k=\tau'}^{t}\eta^2\left[(1-\eta)^{t-k}\right]^2
\mathrel{\overset{\text{(ii)}}{\ge}}\frac{\eta}{1200(1-\gamma)}. \tag*{(342)}
\end{aligned} E [ ∣ ζ t ∣ 2 ] ≥ (i) 2 1 k = 1 ∑ t η 2 [ ( 1 − η ) t − k ] 2 γ 2 ⋅ 16 ( 1 − γ ) 2 2 p ( 1 − p ) ≥ 48 ( 1 − γ ) 1 k = 1 ∑ t η 2 [ ( 1 − η ) t − k ] 2 ≥ 48 ( 1 − γ ) 1 k = τ ′ ∑ t η 2 [ ( 1 − η ) t − k ] 2 ≥ (ii) 1200 ( 1 − γ ) η . (342)
where (i) arises from (319) , and (ii) is valid as long as the following property holds for t ≥ τ ^ t\ge\hat{\tau} t ≥ τ ^ (which will be proven towards the end of this subsection)
∑ k = τ ′ t η 2 [ ( 1 − η ) t − k ] 2 ≥ 1 24 η . (343) \sum_{k=\tau'}^{t}\eta^2\left[(1-\eta)^{t-k}\right]^2\ge\frac{1}{24}\eta. \tag*{(343)} k = τ ′ ∑ t η 2 [ ( 1 − η ) t − k ] 2 ≥ 24 1 η . (343)
We are now well equipped to control E [ ξ t , max ] \mathbb{E}[\xi_{t,\max}] E [ ξ t , m a x ] using the property (340) . Recalling the expression of B B B in (339) , we see that bounding E [ ∣ ζ t ∣ 2 ] / B \mathbb{E}[|\zeta_t|^2]/B E [ ∣ ζ t ∣ 2 ] / B reduces to controlling the following two quantities:
E [ ∣ ζ t ∣ 2 ] η 1 − γ log 2 δ 0 and E [ ∣ ζ t ∣ 2 ] log 2 δ 0 1 − γ ∑ k = 1 t η 2 [ ( 1 − η ) t − k ] 2 . (344) \frac{\mathbb{E}[|\zeta_t|^2]}{\frac{\eta}{1-\gamma}\log\frac{2}{\delta_0}}
\quad\text{and}\quad
\frac{\mathbb{E}[|\zeta_t|^2]}{\sqrt{\frac{\log\frac{2}{\delta_0}}{1-\gamma}\sum_{k=1}^{t}\eta^2[(1-\eta)^{t-k}]^2}}. \tag*{(344)} 1 − γ η log δ 0 2 E [ ∣ ζ t ∣ 2 ] and 1 − γ l o g δ 0 2 ∑ k = 1 t η 2 [( 1 − η ) t − k ] 2 E [ ∣ ζ t ∣ 2 ] . (344)
log 1 δ 0 = − log ( 1 − γ ) 2 2 E [ ∣ ζ t ∣ 2 ] ≤ − log ( 1 − γ ) η 2400 ≤ log ( 9600 T ) , (345) \log\frac{1}{\delta_0}
=-\log\frac{(1-\gamma)^2}{2}\mathbb{E}[|\zeta_t|^2]
\leq-\log\frac{(1-\gamma)\eta}{2400}
\leq\log(9600T), \tag*{(345)} log δ 0 1 = − log 2 ( 1 − γ ) 2 E [ ∣ ζ t ∣ 2 ] ≤ − log 2400 ( 1 − γ ) η ≤ log ( 9600 T ) , (345)
where the first inequality is due to (342) , and the second inequality arises since η ≥ 4 ( 1 − γ ) T \eta\geq\frac{4}{(1-\gamma)T} η ≥ ( 1 − γ ) T 4 (given the range of the learning rates in this case). Combine this with (342) to yield
E [ ∣ ζ t ∣ 2 ] η 1 − γ log 2 δ 0 ≳ 1 log T . \frac{\mathbb{E}[|\zeta_t|^2]}{\frac{\eta}{1-\gamma}\log\frac{2}{\delta_0}}\gtrsim\frac{1}{\log T}. 1 − γ η log δ 0 2 E [ ∣ ζ t ∣ 2 ] ≳ log T 1 .
E [ ∣ ζ t ∣ 2 ] log 2 δ 0 1 − γ ∑ k = 1 t η 2 [ ( 1 − η ) t − k ] 2 ≳ (i) 1 ( 1 − γ ) log T ∑ k = 1 t η 2 [ ( 1 − η ) t − k ] 2 \frac{\mathbb{E}[|\zeta_t|^2]}{\sqrt{\frac{\log\frac{2}{\delta_0}}{1-\gamma}\sum_{k=1}^{t}\eta^2[(1-\eta)^{t-k}]^2}}
\gtrsim^{\text{(i)}}
\sqrt{\frac{1}{(1-\gamma)\log T}\sum_{k=1}^{t}\eta^2[(1-\eta)^{t-k}]^2} 1 − γ l o g δ 0 2 ∑ k = 1 t η 2 [( 1 − η ) t − k ] 2 E [ ∣ ζ t ∣ 2 ] ≳ (i) ( 1 − γ ) log T 1 k = 1 ∑ t η 2 [( 1 − η ) t − k ] 2
≳ (ii) η ( 1 − γ ) log T ≳ (iii) 1 ( 1 − γ ) 2 T log T . \gtrsim^{\text{(ii)}}
\sqrt{\frac{\eta}{(1-\gamma)\log T}}
\gtrsim^{\text{(iii)}}
\frac{1}{\sqrt{(1-\gamma)^2T\log T}}. ≳ (ii) ( 1 − γ ) log T η ≳ (iii) ( 1 − γ ) 2 T log T 1 .
Here, (i) follows from (342) and (345) since
E [ ∣ ζ t ∣ 2 ] ≳ 1 1 − γ ∑ k = 1 t η 2 [ ( 1 − η ) t − k ] 2 and log 2 δ 0 ≲ log T ; \mathbb{E}[|\zeta_t|^2]\gtrsim\frac{1}{1-\gamma}\sum_{k=1}^{t}\eta^2[(1-\eta)^{t-k}]^2
\quad\text{and}\quad
\log\frac{2}{\delta_0}\lesssim\log T; E [ ∣ ζ t ∣ 2 ] ≳ 1 − γ 1 k = 1 ∑ t η 2 [( 1 − η ) t − k ] 2 and log δ 0 2 ≲ log T ;
(ii) arises from (343) ; and (iii) relies on the fact η ≳ 1 ( 1 − γ ) T \eta\gtrsim\frac{1}{(1-\gamma)T} η ≳ ( 1 − γ ) T 1 (given the range of the learning rates considered in this step).
Substituting the above relations into (340) and using the expression of B B B in (339) , we reach
E [ ξ t , max ] ≥ 1 4 B E [ ∣ ζ t ∣ 2 ] ≥ c ( 1 − γ ) 2 T log T \mathbb{E}[\xi_{t,\max}]
\geq\frac{1}{4B}\mathbb{E}[|\zeta_t|^2]
\geq\frac{c}{\sqrt{(1-\gamma)^2T\log T}} E [ ξ t , m a x ] ≥ 4 B 1 E [ ∣ ζ t ∣ 2 ] ≥ ( 1 − γ ) 2 T log T c
for some numerical constant c > 0 c>0 c > 0 . Thus, this proves property (4.143), as long as the claim (343) can be established.
Proof of the claim (343) . Recall that τ ′ : = max { t − 1 η , 1 } \tau' := \max\left\{t-\frac{1}{\eta},1\right\} τ ′ := max { t − η 1 , 1 } . For any k k k obeying τ ′ ≤ k ≤ t \tau'\leq k\leq t τ ′ ≤ k ≤ t , one has
η ( 1 − η ) t − k ≥ η ( 1 − η ) t − τ ′ ≥ η ( 1 − η ) 1 / η ≥ 1 3 η , \eta(1-\eta)^{t-k}\geq\eta(1-\eta)^{t-\tau'}\geq\eta(1-\eta)^{1/\eta}\geq\frac{1}{3}\eta, η ( 1 − η ) t − k ≥ η ( 1 − η ) t − τ ′ ≥ η ( 1 − η ) 1/ η ≥ 3 1 η ,
where the last inequality is valid as long as η ≤ 0.1 \eta\leq0.1 η ≤ 0.1 . Moreover, it is seen that for any t ≥ τ ˉ t\geq\bar{\tau} t ≥ τ ˉ ,
∑ k = τ ′ t η ( 1 − η ) t − k = 1 − ( 1 − η ) t − τ ′ + 1 ≥ 1 − max { ( 1 − η ) 1 / η + 1 , ( 1 − η ) t } \sum_{k=\tau'}^{t}\eta(1-\eta)^{t-k}
=1-(1-\eta)^{t-\tau'+1}
\geq1-\max\left\{(1-\eta)^{1/\eta+1},(1-\eta)^t\right\} k = τ ′ ∑ t η ( 1 − η ) t − k = 1 − ( 1 − η ) t − τ ′ + 1 ≥ 1 − max { ( 1 − η ) 1/ η + 1 , ( 1 − η ) t }
≥ 1 − max { e − 1 , ( 1 − η ) τ ^ } ≥ 1 8 . \ge1-\max\left\{e^{-1},(1-\eta)^{\widehat{\tau}}\right\}\ge\frac{1}{8}. ≥ 1 − max { e − 1 , ( 1 − η ) τ } ≥ 8 1 .
Here, the first identity can be derived via the sum of a geometric sequence; to see why the last inequality holds, note that according to the definition (341) and property (316) ,
( 1 − η ( 1 − γ ρ ) ) T − τ ^ > 6 7 \left(1-\eta(1-\gamma\rho)\right)^{T-\widehat{\tau}}>\frac{6}{7} ( 1 − η ( 1 − γ ρ ) ) T − τ > 7 6
⟹ ( 1 − η ( 1 − γ ρ ) ) τ ^ < ( 1 − η ( 1 − γ ρ ) ) T 6 / 7 = η 0 ( T ) 6 / 7 ≤ 1 / 100 6 / 7 < 7 8 , \Longrightarrow\quad\left(1-\eta(1-\gamma\rho)\right)^{\widehat{\tau}}<\frac{\left(1-\eta(1-\gamma\rho)\right)^T}{6/7}=\frac{\eta_0^{(T)}}{6/7}\le\frac{1/100}{6/7}<\frac{7}{8}, ⟹ ( 1 − η ( 1 − γ ρ ) ) τ < 6/7 ( 1 − η ( 1 − γ ρ ) ) T = 6/7 η 0 ( T ) ≤ 6/7 1/100 < 8 7 ,
which in turn implies that
( 1 − η ) τ ^ ≤ ( 1 − η ( 1 − γ ρ ) ) τ ^ ≤ 7 8 . (346) (1-\eta)^{\widehat{\tau}}\le\left(1-\eta(1-\gamma\rho)\right)^{\widehat{\tau}}\le\frac{7}{8}. \tag*{(346)} ( 1 − η ) τ ≤ ( 1 − η ( 1 − γ ρ ) ) τ ≤ 8 7 . (346)
Putting the preceding two bounds together yields, for any t ≥ τ ^ t\ge\widehat{\tau} t ≥ τ ,
∑ k = τ ′ t η 2 [ ( 1 − η ) t − k ] 2 ≥ min k : τ ′ ≤ k ≤ t { η ( 1 − η ) t − k } ∑ k = τ ′ t η ( 1 − η ) t − k ≥ 1 3 η ⋅ 1 8 ≥ 1 24 η \begin{aligned}
\sum_{k=\tau'}^{t}\eta^2\left[(1-\eta)^{t-k}\right]^2\ge\min_{k:\tau'\le k\le t}\left\{\eta(1-\eta)^{t-k}\right\}\sum_{k=\tau'}^{t}\eta(1-\eta)^{t-k} \\
&\ge\frac{1}{3}\eta\cdot\frac{1}{8}\ge\frac{1}{24}\eta
\end{aligned} k = τ ′ ∑ t η 2 [ ( 1 − η ) t − k ] 2 ≥ k : τ ′ ≤ k ≤ t min { η ( 1 − η ) t − k } k = τ ′ ∑ t η ( 1 − η ) t − k ≥ 3 1 η ⋅ 8 1 ≥ 24 1 η
as claimed.
Notes
The asymptotic convergence of TD learning and Q-learning — namely, their convergence behavior as the number of iterations tends to infinity, with all other parameters held fixed — has been extensively studied since its early development, e.g., Tsitsiklis and Van Roy (1997), Tsitsiklis (1994), Jaakkola et al. (1993), Szepesvári (1997), Borkar and Meyn (2000), Devraj and Meyn (2021), and Li et al. (2023e). Recently, substantial research efforts have shifted toward understanding the non-asymptotic performance of model-free methods, which elucidates how key problem parameters — in addition to the number of iterations — affect algorithmic performance. For TD learning, finite-time, finite-sample analyses have been developed in a growing body of recent work, including but not limited to Bhandari et al. (2018), Lakshminarayanan and Szepesvari (2018), Srikant and Ying (2019), Xu et al. (2019), Doan et al. (2019), Cai et al. (2019), Chen et al. (2020c), Mou et al. (2020), Kaledin et al. (2020), Chen et al. (2021d), Khamaru et al. (2021), Li et al. (2024a), Chen et al. (2022d), Patil et al. (2023), Li et al. (2024d), Mou et al. (2024), Samsonov et al. (2024b), Srikant (2025), Chen et al. (2025c), and Durmus et al. (2025). In parallel, finite-time, finite-sample performance guarantees for Q-learning and its variants have been established in, e.g., Kearns and Singh (1999), Even-Dar and Mansour (2003), Murphy (2005), Beck and Srikant (2012), Azar et al. (2011), Wainwright (2019b), Wainwright (2019c), Chen et al. (2020c), Qu and Wierman (2020), Xu and Gu (2020), Li et al. (2022b), Chen et al. (2024b), Chen et al. (2022d), Li et al. (2024a), Jin et al. (2024b), and Jiao et al. (2026).
Since TD learning and Q-learning can be viewed as instances of stochastic approximation, a notable portion of the existing theoretical analyses builds upon classical techniques from stochastic approximation theory, along with various modern extensions of these techniques. Prominent examples include the ODE method (e.g., Borkar and Meyn (2000), Borkar et al. (2025), and Meyn (2022)), Stein’s method (e.g., Srikant and Ying (2019) and Srikant (2025)), Lyapunov-drift approach (e.g., Chen et al. (2024b), Cayci et al. (2023), Chen et al. (2025d), and Chen and Maguluri (2026), and extensions to two-time-scale settings (e.g., Konda and Tsitsiklis (2004)), among others. See also the tutorial Meyn (2023) for the stability of Q-learning with linear function approximation. The continued development of these techniques has provided versatile tools for analyzing not only TD learning and Q-learning, but also a range of model-free variants, including TD learning with gradient correction (TDC), gradient TD learning, Zap Q-learning, SARSA, momentum Q-learning, double Q-learning, and so on (Borkar and Meyn, 2000; Sutton et al., 2009; Maei et al., 2009; Gupta et al., 2019; Liu et al., 2015; Devraj and Meyn, 2017; Xu et al., 2019; Xu and Liang, 2021; Zou et al., 2019; Weng et al., 2020b; Weng et al., 2020a; Xiong et al., 2020). It is worth noting, however, that these stochastic approximation techniques alone may be insufficient to establish the sharp performance guarantees for TD learning and Q-learning in Theorems 4.1 and 4.6 . Achieving these tight guarantees requires additional proof ideas that carefully exploit the intrinsic variance structure of the MDP, a key ingredient underlying the state-of-the-art analysis in Li et al. (2024a).
In addition, while the sample inefficiency and overestimation bias of Q-learning have long been observed in practice — motivating algorithmic variants such as double Q-learning (Hasselt, 2010) — formal lower bounds on the iteration complexity of Q-learning were not established until Li et al. (2024a). Notably, Wainwright (2019b) conjectured, based on numerical evidence, that the tight dependence of the sample complexity of Q-learning on the effective horizon scales as O ~ ( 1 ( 1 − γ ) 4 ) \widetilde{O}\left(\frac{1}{(1-\gamma)^4}\right) O ( ( 1 − γ ) 4 1 ) . However, prior to Li et al. (2024a), neither matching upper nor lower bounds confirming this scaling were available.
With the aim of improving the sample complexity of Q-learning while preserving its model-free nature, more sophisticated algorithmic techniques have been proposed and analyzed in the literature. For example, a variance-reduced variant of Q-learning — drawing inspiration from variance-reduction methods in stochastic optimization, particularly stochastic variance-reduced gradient (SVRG) (Johnson and Zhang, 2013) — has been shown to achieve the minimax-optimal sample complexity for all ε ∈ ( 0 , 1 ] \varepsilon\in(0,1] ε ∈ ( 0 , 1 ] (Wainwright, 2019c). At a high level, this approach partitions the iterative process into epochs. At the beginning of each epoch, a batch of samples is used to construct increasingly more accurate reference estimates of the value function. Subsequent stochastic approximation updates incorporate these references by combining them with fresh stochastic samples to reduce variability, thereby accelerating convergence. We refer interested readers to Wainwright (2019c), Li et al. (2022b), Mou et al. (2022), and Xia et al. (2024) for further details on both synchronous and asynchronous Q-learning with variance reduction. Notably, however, no existing model-free algorithm is known to attain minimax optimality when ε ≫ 1 \varepsilon\gg1 ε ≫ 1 . This suggests the inherent statistical suboptimality of model-free approaches in the most sample-hungry regime. Finally, we note that variance-reduction techniques have also been incorporated into TD learning for policy evaluation; see, for example, Khamaru et al. (2021), Xu et al. (2020b), and Li et al. (2023c).
Online RL
We now turn to online reinforcement learning, where an agent interacts with an unknown environment sequentially to collect data and learn a near-optimal policy. In contrast to the generative model setting — where arbitrary state–action pairs can be queried on demand — an agent in online RL must collect data by executing policies and observing trajectories in real time. This restriction couples data collection with decision-making, necessitating carefully designed exploration strategies that acquire informative samples without sacrificing long-term performance. Consequently, a central challenge of online RL lies in optimally balancing exploration and exploitation under a limited sampling budget.
Problem formulation
In this chapter, we focus primarily on online episodic RL formulated in finite-horizon nonstationary MDPs, a setting that has been studied most extensively and for which the statistical limits have been characterized even in the most sample-limited regime. Its counterpart in discounted infinite-horizon MDPs remains not fully solved to date (see, e.g., Lattimore and Hutter (2012), Strehl et al. (2006), Dong et al. (2019), Zhang et al. (2021f), Szita and Szepesvári (2010), Zhou et al. (2021b), and Ji and Li (2023)) and calls for further investigation.
Sampling protocol. To be precise, we model the unknown environment as a finite-horizon nonstationary MDP M = ( S , A , H , P , r ) \mathcal{M} = (\mathcal{S}, \mathcal{A}, H, P, r) M = ( S , A , H , P , r ) . For ease of exposition, we assume that the reward function r = { r h } h = 1 H r = \{r_h\}_{h=1}^{H} r = { r h } h = 1 H is fully known to the agent, an assumption that can be relaxed substantially without fundamentally altering the underlying statistical limits and incurs only minor additional technical complications. By contrast, the transition kernel P = { P h } h = 1 H P = \{P_h\}_{h=1}^{H} P = { P h } h = 1 H is assumed to be unknown a priori, constituting the primary source of uncertainty in the online RL setting studied in this chapter. The agent is permitted to interact with the MDP over K K K episodes, each of length H H H .
In each episode k = 1 , … , K k = 1, \ldots, K k = 1 , … , K , the sampling protocol of an online RL algorithm proceeds as follows:
(i) Initialization. The initial state s 1 k s_1^k s 1 k of each episode k k k is drawn independently according to
s 1 k ∼ i . i . d . ρ , k = 1 , ⋯ , K , (347) s_1^k \overset{\mathrm{i.i.d.}}{\sim} \rho,\qquad k=1,\cdots,K, \tag*{(347)} s 1 k ∼ i.i.d. ρ , k = 1 , ⋯ , K , (347)
where ρ ∈ Δ ( S ) \rho\in\Delta(\mathcal{S}) ρ ∈ Δ ( S ) denotes an (unknown) initial state distribution.
(ii) Policy selection. Based on the information collected in episodes 1 1 1 through k − 1 k-1 k − 1 , the agent selects a policy π k = { π h k } h = 1 H \pi^k=\{\pi_h^k\}_{h=1}^H π k = { π h k } h = 1 H .
(iii) Sampling. The agent executes policy π k \pi^k π k in the MDP M \mathcal{M} M , generating a sample trajectory { ( s h k , a h k , r h k ) } 1 ≤ h ≤ H \{(s_h^k,a_h^k,r_h^k)\}_{1\le h\le H} {( s h k , a h k , r h k ) } 1 ≤ h ≤ H , where s h k s_h^k s h k , a h k a_h^k a h k and r h k = r h ( s h k , a h k ) r_h^k=r_h(s_h^k,a_h^k) r h k = r h ( s h k , a h k ) denote the state, action and immediate reward at step h h h of this episode, respectively. Here, starting from s 1 k s_1^k s 1 k , the trajectory is rolled out sequentially for h = 1 , … , H h=1,\ldots,H h = 1 , … , H according to
a h k ∼ π h k ( ⋅ ∣ s h k ) and s h + 1 k ∼ P h ( ⋅ ∣ s h k , a h k ) . (348) a_h^k \sim\pi_h^k(\cdot\mid s_h^k)
\qquad\text{and}\qquad
s_{h+1}^k \sim P_h(\cdot\mid s_h^k,a_h^k). \tag*{(348)} a h k ∼ π h k ( ⋅ ∣ s h k ) and s h + 1 k ∼ P h ( ⋅ ∣ s h k , a h k ) . (348)
For notational convenience, we set r H + 1 = 0 r_{H+1}=0 r H + 1 = 0 throughout this chapter. Consequently, learning the terminal transition kernel P H P_H P H is unnecessary, as all actions yield zero rewards at step H + 1 H+1 H + 1 . Since each trajectory has length H H H , the total sample size can be understood as
Exploration-exploitation tradeoff. A fundamental challenge in online RL arises from the exploration–exploitation tradeoff: the agent must collect informative samples to learn the unknown transition kernel, while simultaneously seeking policies that maximize cumulative rewards. Compared to the generative model setting, at least two aspects significantly complicate matters.
Coverage of the state-action space. Effective learning typically requires adequate coverage of the state–action space, which is difficult to guarantee under online sampling protocols. In particular, the agent does not know a priori which regions of the state–action space are reachable through sample rollouts, nor how to reach them. This contrasts sharply to the generative model setting, where one can query arbitrary state–action pairs at any step of the MDP.
Complicated statistical dependency across time. Samples in online RL are collected sequentially and depend on the agent’s past decisions. This creates complex statistical dependencies across both samples and time. In contrast, the samples studied in Chapters 3 and 4 under the generative model are statistically independent, a feature that greatly simplifies both algorithm design and theoretical analysis.
Performance metric: regret. To evaluate the performance of an online RL algorithm, arguably one of the most widely used metrics is the (cumulative) regret over all K K K episodes:
Regret ( K ) : = ∑ k = 1 K ( V 1 ∗ ( s 1 k ) − V 1 π k ( s 1 k ) ) . (349) \operatorname{Regret}(K) := \sum_{k=1}^{K}\left(V_1^*(s_1^k)-V_1^{\pi^k}(s_1^k)\right). \tag*{(349)} Regret ( K ) := k = 1 ∑ K ( V 1 ∗ ( s 1 k ) − V 1 π k ( s 1 k ) ) . (349)
This metric quantifies the aggregate performance gap — expressed in terms of the resulting value functions — between the agent and an oracle that always acts optimally. Importantly, regret provides a natural way to formalize the exploration–exploitation tradeoff: it captures both the cost of exploration, since exploratory actions may yield lower rewards, and the cost of insufficient exploitation, as inaccurate model estimates arising from limited data can adversely affect decision-making. Achieving low regret implies that the algorithm can efficiently identify near-optimal policies while keeping the cumulative performance loss under control. The main focus of this chapter is the design of online RL algorithms with provably small cumulative regret Regret ( K ) \operatorname{Regret}(K) Regret ( K ) .
Connection between regret and PAC statistical guarantees. Careful readers may observe that the policies { π k } 1 ≤ k ≤ K \{\pi^k\}_{1\le k\le K} { π k } 1 ≤ k ≤ K deployed across episodes are generally different, which naturally raises the question of whether one can instead learn a single policy that attains near-optimal values. Indeed, this question was the primary focus of Chapters 3–4 and falls within the PAC (probably approximately correct) learning framework, which seeks a single policy estimate π ^ \hat{\pi} π ^ satisfying
V 1 ∗ ( ρ ) − V 1 π ^ ( ρ ) ≤ ε . V_1^*(\rho)-V_1^{\hat{\pi}}(\rho)\le\varepsilon. V 1 ∗ ( ρ ) − V 1 π ^ ( ρ ) ≤ ε .
Here, we recall that ρ \rho ρ denotes the state distribution generating the initial states { s 1 k } 1 ≤ k ≤ K \{s_1^k\}_{1\le k\le K} { s 1 k } 1 ≤ k ≤ K (see (347) ). Fortunately, regret guarantees can often be readily translated into corresponding PAC learning guarantees (see, e.g., Jin et al. (2018)).
To illustrate this connection, consider an online RL algorithm that, for any MDP with S S S states, A A A actions and horizon H H H , satisfies the expected regret bound
E [ Regret ( K ) ] ≤ C S , A , H K 1 − α (350) \mathbb{E}[\operatorname{Regret}(K)]\le C_{S,A,H}K^{1-\alpha} \tag*{(350)} E [ Regret ( K )] ≤ C S , A , H K 1 − α (350)
for some α ∈ [ 0 , 1 ] \alpha\in[0,1] α ∈ [ 0 , 1 ] and some quantity C S , A , H C_{S,A,H} C S , A , H that may depend on ( S , A , H ) (S,A,H) ( S , A , H ) . Here, the expectation is taken over all randomness arising from the initial states, sample transitions, and the learning algorithm’s internal randomization. Now construct the policy estimate π ^ \hat{\pi} π ^ by selecting one of the deployed policies uniformly at random:
π ^ ∼ Unif ( π 1 , π 2 , … , π K ) . (351) \hat{\pi}\sim\operatorname{Unif}(\pi^1,\pi^2,\ldots,\pi^K). \tag*{(351)} π ^ ∼ Unif ( π 1 , π 2 , … , π K ) . (351)
Then with probability at least 1 − δ 1-\delta 1 − δ ,
V 1 ∗ ( ρ ) − V 1 π ^ ( ρ ) ≤ C S , A , H + 1 K − α δ − 1 . (352) V_1^*(\rho)-V_1^{\hat{\pi}}(\rho)\le C_{S,A,H+1}K^{-\alpha}\delta^{-1}. \tag*{(352)} V 1 ∗ ( ρ ) − V 1 π ^ ( ρ ) ≤ C S , A , H + 1 K − α δ − 1 . (352)
Proof of property (352) . Let us first consider the special case where the initial state distribution ρ \rho ρ is supported on a single state s s s , namely, ρ ( s ) = 1 \rho(s)=1 ρ ( s ) = 1 and ρ ( s ′ ) = 0 \rho(s')=0 ρ ( s ′ ) = 0 for all s ′ ≠ s s'\ne s s ′ = s . Markov’s inequality asserts that, with probability at least 1 − δ 1-\delta 1 − δ ,
V 1 ∗ ( s ) − V 1 π ^ ( s ) ≤ δ − 1 E [ V 1 ∗ ( s ) − V 1 π ^ ( s ) ] = 1 K δ ∑ k = 1 K E [ V 1 ∗ ( s ) − V 1 π k ( s ) ] = 1 K δ E [ Regret ( K ) ] ≤ C S , A , H K − α δ − 1 , \begin{aligned}
V_1^*(s)-V_1^{\hat{\pi}}(s)
&\le\delta^{-1}\mathbb{E}[V_1^*(s)-V_1^{\hat{\pi}}(s)] \\
&= \frac{1}{K\delta}\sum_{k=1}^{K}\mathbb{E}[V_1^*(s)-V_1^{\pi^k}(s)] \\
&= \frac{1}{K\delta}\mathbb{E}[\operatorname{Regret}(K)]
\le C_{S,A,H}K^{-\alpha}\delta^{-1},
\end{aligned} V 1 ∗ ( s ) − V 1 π ^ ( s ) ≤ δ − 1 E [ V 1 ∗ ( s ) − V 1 π ^ ( s )] = K δ 1 k = 1 ∑ K E [ V 1 ∗ ( s ) − V 1 π k ( s )] = K δ 1 E [ Regret ( K )] ≤ C S , A , H K − α δ − 1 ,
where the first equality follows from the definition (351) of π ^ \hat{\pi} π ^ .
We now extend the argument to an arbitrary initial state distribution ρ \rho ρ . To this end, we augment the MDP by introducing an additional step 0, consisting of a single state s 0 s_0 s 0 , a single action a 0 a_0 a 0 , zero immediate reward (i.e., r 0 ( s 0 , a 0 ) = 0 r_0(s_0,a_0)=0 r 0 ( s 0 , a 0 ) = 0 ), and transition kernel P 0 ( ⋅ ∣ s 0 , a 0 ) = ρ P_0(\cdot\mid s_0,a_0)=\rho P 0 ( ⋅ ∣ s 0 , a 0 ) = ρ . Since there is only one available action, this additional step involves no decision-making; accordingly, with a slight abuse of notation, we continue to denote by π \pi π the obvious extension of a policy π \pi π for the original MDP and write V 0 π V_0^\pi V 0 π for its value function starting from step 0. The online RL algorithm can therefore be applied to the augmented MDP without modification, yielding
E [ Regret ( K ) ] = ∑ k = 1 K E [ V 0 ∗ ( s 0 ) − V 0 π k ( s 0 ) ] ≤ C S , A , H + 1 K 1 − α (353) \mathbb{E}[\operatorname{Regret}(K)] = \sum_{k=1}^{K} \mathbb{E}\left[V_0^*(s_0) - V_0^{\pi^k}(s_0)\right] \le C_{S,A,H+1}K^{1-\alpha} \tag*{(353)} E [ Regret ( K )] = k = 1 ∑ K E [ V 0 ∗ ( s 0 ) − V 0 π k ( s 0 ) ] ≤ C S , A , H + 1 K 1 − α (353)
Given that the augmented MDP has a singleton initial state, the preceding argument applies directly, implying that
V 0 ∗ ( s 0 ) − V 0 π ^ ( s 0 ) ≤ C S , A , H + 1 K − α δ − 1 (354) V_0^*(s_0) - V_0^{\widehat{\pi}}(s_0) \le C_{S,A,H+1}K^{-\alpha}\delta^{-1} \tag*{(354)} V 0 ∗ ( s 0 ) − V 0 π ( s 0 ) ≤ C S , A , H + 1 K − α δ − 1 (354)
with probability at least 1 − δ 1-\delta 1 − δ . To finish up, observe that for every policy π \pi π , one has V 0 π ( s 0 ) = V 1 π ( ρ ) V_0^\pi(s_0)=V_1^\pi(\rho) V 0 π ( s 0 ) = V 1 π ( ρ ) , as the additional initial step incurs zero reward and admits only a single action. This together with (354) completes the proof.
This simple argument demonstrates that sublinear regret naturally yields sample-efficient PAC statistical guarantees. For example, if an online RL algorithm enjoys the regret bound E [ Regret ( K ) ] ≤ O ~ ( S A H 3 K ) \mathbb{E}[\operatorname{Regret}(K)] \le\widetilde{O}(\sqrt{SAH^3K}) E [ Regret ( K )] ≤ O ( S A H 3 K ) for any MDP with S S S states, A A A actions and horizon length H H H , then the randomly selected policy π ^ \widehat{\pi} π achieves (by choosing δ = 1 log ( S A H ) \delta= \frac{1}{\log(SAH)} δ = l o g ( S A H ) 1 in (352) )
V 1 ∗ ( s ) − V 1 π ^ ( s ) ≤ O ~ ( S A H 3 K ) V_1^*(s) - V_1^{\widehat{\pi}}(s) \le\widetilde{O}\left(\sqrt{\frac{SAH^3}{K}}\right) V 1 ∗ ( s ) − V 1 π ( s ) ≤ O ( K S A H 3 )
with probability 1 − o ( 1 ) 1-o(1) 1 − o ( 1 ) ; consequently, to ensure V 1 ∗ ( s ) − V 1 π ^ ( s ) ≤ ε V_1^*(s)-V_1^{\widehat{\pi}}(s) \le\varepsilon V 1 ∗ ( s ) − V 1 π ( s ) ≤ ε with probability approaching 1, it suffices to run this online RL algorithm for
K = Θ ~ ( S A H 3 ε 2 ) K = \widetilde{\Theta}\left(\frac{SAH^3}{\varepsilon^2}\right) K = Θ ( ε 2 S A H 3 )
episodes.
A detour: the UCB algorithm for multi-armed bandits
Before embarking on the design and analysis of online RL algorithms, we take a brief detour to study multi-armed bandits, a special case of online RL where state transitions are absent or S = 1 S=1 S = 1 . In this setting, there is no transition kernel to estimate; instead, we consider the scenario where the observed rewards are noisy, so that model uncertainty arises solely from the stochasticity of the rewards. As it turns out, several key ideas developed in the multi-armed bandit literature pave the way for the design of efficient online RL algorithms.
Setting: stochastic bandits
Consider a standard stochastic bandit model, where an agent interacts sequentially with the unknown environment over T T T rounds. The key components of this bandit model are described as follows.
Actions. The agent has access to a set of A A A arms (i.e., actions), denoted by A = { 1 , … , A } \mathcal{A}=\{1,\ldots,A\} A = { 1 , … , A } . One can picture a gambler standing in front of a row of A A A slot machines, each with its own arm; at each round, the gambler needs to decide which machine to play.
Mean rewards. Each arm a ∈ A a \in\mathcal{A} a ∈ A is associated with a time-invariant mean reward r ( a ) ∈ [ 0 , 1 ] r(a) \in[0,1] r ( a ) ∈ [ 0 , 1 ] . In the gambling analogy, each slot machine has its own payout rate. The optimal mean reward is denoted by
r ⋆ : = max a ∈ A r ( a ) , (355) r^\star:= \max_{a \in\mathcal{A}} r(a), \tag*{(355)} r ⋆ := a ∈ A max r ( a ) , (355)
with the optimal arm defined correspondingly as
a ⋆ : = arg max a ∈ A r ( a ) . (356) a^\star:= \arg\max_{a \in\mathcal{A}} r(a). \tag*{(356)} a ⋆ := arg a ∈ A max r ( a ) . (356)
Crucially, these mean rewards are unknown to the agent and can only be learned through sequential interaction with the environment over time.
Realized rewards. At each round t t t , nature generates a noisy reward R a , t R_{a,t} R a , t independently for each arm a ∈ A a \in\mathcal{A} a ∈ A , given by
R a , t = r ( a ) + z a , t , with z a , t ∼ i . i . d . N ( 0 , 1 ) . (357) R_{a,t} = r(a) + z_{a,t}, \qquad\text{with } z_{a,t} \overset{\mathrm{i.i.d.}}{\sim} \mathcal{N}(0,1). \tag*{(357)} R a , t = r ( a ) + z a , t , with z a , t ∼ i.i.d. N ( 0 , 1 ) . (357)
While we assume i.i.d. standard Gaussian noise for simplicity of exposition, this assumption can be significantly relaxed without affecting the key algorithmic principles and insights (see, e.g., Lattimore and Szepesvári (2020)).
Observations (bandit feedback). The agent operates under bandit feedback, meaning that it does not observe the full collection { R a , t } a ∈ A \{R_{a,t}\}_{a \in\mathcal{A}} { R a , t } a ∈ A of noisy rewards at each round. Instead, at round t t t , the agent selects a single arm a t ∈ A a_t \in\mathcal{A} a t ∈ A — potentially based on previous actions and observations — and observes only the corresponding realized reward R a t , t R_{a_t,t} R a t , t .
Naturally, the agent would aim to maximize its cumulative payoff over T T T rounds. To evaluate the performance of a bandit algorithm (which determines how actions are selected over time), we introduce the (cumulative) regret as
Regret ( T ) : = T r ⋆ − ∑ t = 1 T r ( a t ) , (358) \operatorname{Regret}(T) := Tr^\star- \sum_{t=1}^{T} r(a_t), \tag*{(358)} Regret ( T ) := T r ⋆ − t = 1 ∑ T r ( a t ) , (358)
which measures the total expected loss in reward relative to always playing the optimal arm. Here, we employ the mean reward r ( a t ) r(a_t) r ( a t ) rather than the realized reward R a t , t R_{a_t,t} R a t , t to marginalize out observation noise, which ensures non-negativity of regret. Notably, Regret ( T ) \operatorname{Regret}(T) Regret ( T ) is itself a random variable, as the sequence of actions { a t } \{a_t\} { a t } is a function of the history of noisy rewards.
Moreover, the regret admits a simple yet useful decomposition. Let N T ( a ) N_T(a) N T ( a ) denote the number of times arm a a a is pulled over T T T rounds. Then the regret can be expressed in terms of { N T ( a ) } \{N_T(a)\} { N T ( a )} and the suboptimality gaps:
Regret ( T ) = ∑ t = 1 T Δ ( a t ) = ∑ a ∈ A Δ ( a ) N T ( a ) with Δ ( a ) : = r ⋆ − r ( a ) . (359) \operatorname{Regret}(T) = \sum_{t=1}^{T} \Delta(a_t) = \sum_{a \in\mathcal{A}} \Delta(a)N_T(a) \quad\text{with } \Delta(a) := r^\star- r(a). \tag*{(359)} Regret ( T ) = t = 1 ∑ T Δ ( a t ) = a ∈ A ∑ Δ ( a ) N T ( a ) with Δ ( a ) := r ⋆ − r ( a ) . (359)
Given that the mean rewards are unknown a priori , the agent must balance exploring undersampled arms with exploiting those currently deemed favorable. Despite its apparent simplicity, this classical bandit formulation already encapsulates a central challenge of sequential decision-making under uncertainty: the interplay between learning, decision-making, and data collection in real time.
Optimism in the face of uncertainty
We now introduce a core design principle underlying a broad family of bandit algorithms. On the one hand, if an arm a a a has been pulled only a small number of times, the corresponding estimate of its mean reward r a r_a r a can be highly volatile and therefore unreliable; this motivates playing the arm more often in order to improve the accuracy of the reward estimate. On the other hand, if the estimated reward of arm a a a is lower than that of other arms, this arm is likely suboptimal, suggesting that it should be pulled less frequently. In summary,
high uncertainty (i.e., low confidence) in an arm’s reward estimate can incentivize further exploration of that arm;
a large reward estimate encourages more frequent pulls of that arm.
One strategy to unify these two competing principles is to introduce an optimistic estimate of the form:
r ^ ( a ) = empirical reward estimate for arm a + its uncertainty , \widehat{r}(a) = \text{empirical reward estimate for arm } a + \text{ its uncertainty}, r ( a ) = empirical reward estimate for arm a + its uncertainty ,
and then select the arm with the highest value of r ^ ( a ) \widehat{r}(a) r ( a ) , provided that this quantity is constructed appropriately. Since r ^ ( a ) \widehat{r}(a) r ( a ) acts as an optimistic estimate of the reward r ( a ) r(a) r ( a ) (in the sense that it intentionally biases the estimate upward to account for uncertainty), this idea is commonly referred to as the optimism principle in the face of uncertainty, originally introduced by Lai and Robbins (1985).
While conceptually natural, this paradigm raises a fundamental question: how should the uncertainty bonus be calibrated relative to the empirical reward estimate? Evidently, the choice of this trade-off is crucial for achieving optimal regret and is, in general, far from obvious.
The upper confidence bound (UCB) algorithm
Somewhat surprisingly, a simple yet elegant approach — known as the upper confidence bound (UCB) algorithm, originally developed by Lai and Robbins (1985) and Lai (1987) — is sufficient to achieve the optimal exploration-exploitation trade-off.
More precisely, the UCB algorithm operates by selecting, at each round t t t , the arm with the largest upper confidence bound on its mean reward, where these confidence bounds are constructed using the samples observed up to and including time t t t . Specifically, for each round t t t , the agent pulls the arm
a t = arg max a ∈ A r t − 1 u c b ( a ) , (360) a_t = \underset{a \in\mathcal{A}}{\arg\max}\ r_{t-1}^{\mathrm{ucb}}(a), \tag*{(360)} a t = a ∈ A arg max r t − 1 ucb ( a ) , (360)
where r t u c b ( a ) r_t^{\mathrm{ucb}}(a) r t ucb ( a ) represents the UCB estimate for arm a a a , defined as
r t u c b ( a ) = r ‾ t ( a ) + 2 log T 2 δ N t ( a ) . (361) r_t^{\mathrm{ucb}}(a) = \overline{r}_t(a) + \sqrt{\frac{2 \log\frac{T^2}{\delta}}{N_t(a)}}. \tag*{(361)} r t ucb ( a ) = r t ( a ) + N t ( a ) 2 log δ T 2 . (361)
Here, 0 < δ < 1 0 < \delta< 1 0 < δ < 1 is some confidence parameter (whose role will become clear shortly), N t ( a ) N_t(a) N t ( a ) denotes the number of times arm a a a has been pulled up to round t t t , and r ‾ t ( a ) \overline{r}_t(a) r t ( a ) represents the empirical mean reward of arm a a a computed from all observations collected so far:
r ‾ t ( a ) = 1 N t ( a ) ∑ i = 1 t R a , i 1 ( arm a is pulled at round i ) . (362) \overline{r}_t(a) = \frac{1}{N_t(a)} \sum_{i=1}^{t} R_{a,i} \mathbf{1}(\text{arm }a\text{ is pulled at round }i). \tag*{(362)} r t ( a ) = N t ( a ) 1 i = 1 ∑ t R a , i 1 ( arm a is pulled at round i ) . (362)
Algorithm 5.1: The upper confidence bound (UCB) algorithm for online stochastic bandits
input: number of rounds T T T , confidence parameter δ \delta δ .
initialize N ( a ) ← 0 N(a) \leftarrow0 N ( a ) ← 0 , r s u m ( a ) ← 0 r^{\mathrm{sum}}(a) \leftarrow0 r sum ( a ) ← 0 , and r ˉ ( a ) ← 0 \bar{r}(a) \leftarrow0 r ˉ ( a ) ← 0 for all a ∈ A a \in\mathcal{A} a ∈ A . // pull arms based on UCBs.
for t = 1 , … , T t = 1,\ldots,T t = 1 , … , T do /* add bonus. */
compute a t ∈ arg max a ∈ A r u c b ( a ) a_t \in\arg\max_{a \in\mathcal{A}} r^{\mathrm{ucb}}(a) a t ∈ arg max a ∈ A r ucb ( a ) , where
r u c b ( a ) ← r ˉ ( a ) + 2 log T 2 δ N ( a ) . r^{\mathrm{ucb}}(a) \leftarrow\bar{r}(a) + \sqrt{\frac{2\log\frac{T^2}{\delta}}{N(a)}}. r ucb ( a ) ← r ˉ ( a ) + N ( a ) 2 log δ T 2 .
pull arm a t a_t a t at round t t t , and receive reward R a t , t R_{a_t,t} R a t , t .
set N ( a t ) ← N ( a t ) + 1 N(a_t) \leftarrow N(a_t) + 1 N ( a t ) ← N ( a t ) + 1 , r s u m ( a t ) ← r s u m ( a t ) + R a t , t r^{\mathrm{sum}}(a_t) \leftarrow r^{\mathrm{sum}}(a_t) + R_{a_t,t} r sum ( a t ) ← r sum ( a t ) + R a t , t , and r ˉ ( a t ) ← r s u m ( a t ) / N ( a t ) \bar{r}(a_t) \leftarrow r^{\mathrm{sum}}(a_t)/N(a_t) r ˉ ( a t ) ← r sum ( a t ) / N ( a t ) .
The complete procedure of this algorithm is summarized in Algorithm 5.1.
The specific form of the uncertainty term 2 log ( T 2 / δ ) N t ( a ) \sqrt{\frac{2\log(T^2/\delta)}{N_t(a)}} N t ( a ) 2 l o g ( T 2 / δ ) in (361) — often dubbed the exploration bonus — is motivated by the use of concentration inequalities to quantify statistical uncertainty. To make this precise, recall from standard Gaussian tail bounds that a Gaussian random variable Z ∼ N ( μ , σ 2 ) Z \sim\mathcal{N}(\mu,\sigma^2) Z ∼ N ( μ , σ 2 ) satisfies (see, e.g., Vershynin (2018, Chapter 2))
P ( Z ≤ μ − τ ) ≤ exp ( − τ 2 2 σ 2 ) \mathbb{P}(Z \le\mu-\tau) \le\exp\left(-\frac{\tau^2}{2\sigma^2}\right) P ( Z ≤ μ − τ ) ≤ exp ( − 2 σ 2 τ 2 )
for any τ ≥ 0 \tau\ge0 τ ≥ 0 , and as a result,
P ( Z + σ 2 log 1 δ ≥ μ ) > 1 − δ \mathbb{P}\left(Z+\sigma\sqrt{2\log\frac{1}{\delta}}\ge\mu\right)>1-\delta P ( Z + σ 2 log δ 1 ≥ μ ) > 1 − δ
for any 0 < δ < 1 0<\delta<1 0 < δ < 1 . Now observe that, under the Gaussian reward model (357) , one has r ˉ t ( a ) − r ( a ) = N ( 0 , 1 N t ( a ) ) \bar{r}_t(a)-r(a)=\mathcal{N}\left(0,\frac{1}{N_t(a)}\right) r ˉ t ( a ) − r ( a ) = N ( 0 , N t ( a ) 1 ) . Applying the above Gaussian tail bound thus yields
P ( r t u c b ( a ) ≥ r ( a ) ) = P ( r ˉ t ( a ) + 2 log T 2 δ N a ( t ) ≥ r ( a ) ) > 1 − δ T 2 , (363) \mathbb{P}\left(r_t^{\mathrm{ucb}}(a)\ge r(a)\right)=\mathbb{P}\left(\bar{r}_t(a)+\sqrt{\frac{2\log\frac{T^2}{\delta}}{N_a(t)}}\ge r(a)\right)>1-\frac{\delta}{T^2}, \tag*{(363)} P ( r t ucb ( a ) ≥ r ( a ) ) = P r ˉ t ( a ) + N a ( t ) 2 log δ T 2 ≥ r ( a ) > 1 − T 2 δ , (363)
and similarly,
P ( r ( a ) + 2 log T 2 δ N t ( a ) ≥ r ˉ t ( a ) ) > 1 − δ T 2 . (364) \mathbb{P}\left(r(a)+\sqrt{\frac{2\log\frac{T^2}{\delta}}{N_t(a)}}\ge\bar{r}_t(a)\right)>1-\frac{\delta}{T^2}. \tag*{(364)} P r ( a ) + N t ( a ) 2 log δ T 2 ≥ r ˉ t ( a ) > 1 − T 2 δ . (364)
Consequently, the exploration bonus serves as a high-probability upper bound on the reward estimation error, while r t u c b ( a ) r_t^{\mathrm{ucb}}(a) r t ucb ( a ) acts as a statistically valid upper confidence bound on the unknown mean reward r ( a ) r(a) r ( a ) — hence the name of the algorithm. This interpretation makes clear that the UCB algorithm makes decision according to an optimistic, yet statistically plausible, view of their mean rewards.
Theoretical guarantees
To rigorously demonstrate the efficacy of the UCB algorithm, we present its regret guarantees in the following theorem; the proof is provided in Section 5.2.5.
Theorem 5.1. Consider any 0 < δ < 1 0 < \delta< 1 0 < δ < 1 , and suppose that T ≥ A ≥ 2 T \ge A \ge2 T ≥ A ≥ 2 . Recall that Δ ( a ) = r ⋆ − r ( a ) ∈ [ 0 , 1 ] \Delta(a) = r^\star- r(a) \in[0, 1] Δ ( a ) = r ⋆ − r ( a ) ∈ [ 0 , 1 ] for all a ∈ A a \in\mathcal{A} a ∈ A . Then the expected regret of the UCB algorithm (see Algorithm 5.1) satisfies
E [ Regret ( T ) ] ≤ 8 δ A T log ( T 2 δ ) . (365) \mathbb{E}[\operatorname{Regret}(T)] \le8\delta\sqrt{AT\log\left(\frac{T^2}{\delta}\right)}. \tag*{(365)} E [ Regret ( T )] ≤ 8 δ A T log ( δ T 2 ) . (365) In addition, it also holds that
E [ Regret ( T ) ] ≤ 10 log ( T 2 δ ) ∑ a : Δ ( a ) > 0 1 Δ ( a ) . (366) \mathbb{E}[\operatorname{Regret}(T)] \le10\log\left(\frac{T^2}{\delta}\right)\sum_{a:\Delta(a)>0}\frac{1}{\Delta(a)}. \tag*{(366)} E [ Regret ( T )] ≤ 10 log ( δ T 2 ) a : Δ ( a ) > 0 ∑ Δ ( a ) 1 . (366)
Remark 5.1. These results can also be stated as high-probability regret bounds (Lattimore and Szepesvári, 2020). However, since bandits are not the primary focus of this monograph and are introduced mainly to motivate certain algorithmic ideas, we restrict our attention to the expected regret bounds for brevity.
This theorem provides two complementary results: a worst-case regret bound (365) that holds uniformly over all problem instances, and a more refined instance-dependent bound (366) that explicitly captures the role of the suboptimality gaps { Δ ( a ) } a ∈ A \{\Delta(a)\}_{a\in\mathcal{A}} { Δ ( a ) } a ∈ A . A few remarks about these regret guarantees are in order.
E [ Regret ( T ) ] ≤ O ~ ( T ) . \mathbb{E}[\operatorname{Regret}(T)] \le\widetilde{O}(\sqrt{T}). E [ Regret ( T )] ≤ O ( T ) .
In the language of online learning (Shalev-Shwartz, 2012), this implies that UCB is a no-regret learning algorithm, meaning that the average regret per round, 1 T E [ Regret ( T ) ] \frac{1}{T}\mathbb{E}[\operatorname{Regret}(T)] T 1 E [ Regret ( T )] , vanishes asymptotically as T → ∞ T \to\infty T → ∞ . In other words, although the agent may initially incur losses while exploring uncertain arms, its long-term performance eventually approaches that of the optimal arm. Encouragingly, the regret bound O ~ ( A T ) \widetilde{O}(\sqrt{AT}) O ( A T ) in (365) is minimax-optimal up to logarithmic factors, revealing that the optimism principle achieves essentially optimal worst-case performance. In fact, the logarithmic factor in (365) can be shaved through more refined algorithmic and analysis techniques (see Audibert and Bubeck (2009)). See, e.g., Lattimore and Szepesvári (2020, Section 15.2) for a matching lower bound.
The instance-dependent regret bound (366) , which is often tighter than the worst-case bound (365) , reveals the intrinsic difficulty of a bandit instance through the suboptimality gaps. In particular, this regret bound scales inversely with the harmonic mean of the gaps:
\mathbb{E}[\operatorname{Regret}(T)] \le\widetilde{O}\left(\frac{A}{\operatorname{harmonic\mathchar`-mean}(\{\Delta(a)\}_{a\ne a^\star})}\right).
where we define, for any sequence of positive numbers { x i } 1 ≤ i ≤ n \{x_i\}_{1\le i\le n} { x i } 1 ≤ i ≤ n ,
\operatorname{harmonic\mathchar`-mean}(\{x_i\}_{1\le i\le n}) := \frac{n}{x_1^{-1}+\cdots+x_n^{-1}}.
For instance, when Δ ( a ) ≳ 1 \Delta(a) \gtrsim1 Δ ( a ) ≳ 1 holds for all suboptimal arms and when A = O ( 1 ) A = O(1) A = O ( 1 ) , the expected regret bound reduces to
E [ Regret ( T ) ] ≤ O ~ ( 1 ) , \mathbb{E}[\operatorname{Regret}(T)] \le\widetilde{O}(1), E [ Regret ( T )] ≤ O ( 1 ) ,
illustrating that, for favorable problem instances, the regret can be considerably smaller than the worst-case O ~ ( T ) \widetilde{O}(\sqrt{T}) O ( T ) scaling.
Intuitive explanation. Before presenting the proof of Theorem 5.1, we take a moment to provide some intuition for the regret bounds. Recall the regret decomposition in (359) :
E [ Regret ( T ) ] = ∑ a ∈ A Δ ( a ) E [ N T ( a ) ] , (367) \mathbb{E}[\operatorname{Regret}(T)] = \sum_{a \in\mathcal{A}} \Delta(a)\mathbb{E}[N_T(a)], \tag*{(367)} E [ Regret ( T )] = a ∈ A ∑ Δ ( a ) E [ N T ( a )] , (367)
where, by a slight abuse of notation, N T ( a ) N_T(a) N T ( a ) denotes the total number of pulls of arm a a a over the T T T rounds. This suggests controlling E [ N T ( a ) ] \mathbb{E}[N_T(a)] E [ N T ( a )] for each arm a a a . As it turns out, a key step of the proof is to show that, for any suboptimal arm a a a ,
E [ N T ( a ) ] ≤ O ~ ( ( Δ ( a ) ) − 2 ) . \mathbb{E}[N_T(a)] \le\widetilde{O}\left((\Delta(a))^{-2}\right). E [ N T ( a )] ≤ O ( ( Δ ( a ) ) − 2 ) .
This scaling is intuitive. If N T ( a ) N_T(a) N T ( a ) becomes significantly larger than ( Δ ( a ) ) − 2 (\Delta(a))^{-2} ( Δ ( a ) ) − 2 , then the empirical estimate r ˉ ( a ) \bar{r}(a) r ˉ ( a ) concentrates sharply around the true mean r ( a ) r(a) r ( a ) , making it distinguishable from that of the optimal arm a ⋆ a^\star a ⋆ . More precisely, recall that the reward estimation error r ˉ a − r ( a ) ∼ N ( 0 , 1 / N T ( a ) ) \bar{r}_a - r(a) \sim\mathcal{N}(0,1/N_T(a)) r ˉ a − r ( a ) ∼ N ( 0 , 1/ N T ( a )) and is therefore on the order of 1 / N T ( a ) 1/\sqrt{N_T(a)} 1/ N T ( a ) ; once this uncertainty is much smaller than the suboptimality gap Δ ( a ) = r ⋆ − r ( a ) \Delta(a) = r^\star- r(a) Δ ( a ) = r ⋆ − r ( a ) , the learner can reliably identify arm a a a as suboptimal and will no longer select it. Consequently, the summand Δ ( a ) E [ N T ( a ) ] \Delta(a)\mathbb{E}[N_T(a)] Δ ( a ) E [ N T ( a )] in (367) obeys
Δ ( a ) E [ N T ( a ) ] ≤ Δ ( a ) O ~ ( ( Δ ( a ) ) − 2 ) = O ~ ( 1 / Δ ( a ) ) . \Delta(a)\mathbb{E}[N_T(a)] \le\Delta(a)\widetilde{O}\left((\Delta(a))^{-2}\right) = \widetilde{O}(1/\Delta(a)). Δ ( a ) E [ N T ( a )] ≤ Δ ( a ) O ( ( Δ ( a ) ) − 2 ) = O ( 1/Δ ( a )) .
Summing over all arms suggests the claimed instance-dependent regret bound (366) .
Analysis: proof of Theorem 5.1
To begin with, we observe that the first A A A rounds constitute an initialization phase, during which each arm is pulled once and the corresponding noisy reward is observed. Indeed, if an arm a a a has never been selected, then N ( a ) = 0 N(a) = 0 N ( a ) = 0 and hence r u c b ( a ) = ∞ r^{\mathrm{ucb}}(a) = \infty r ucb ( a ) = ∞ . Consequently, the algorithm always selects an unsampled arm whenever one exists, so that each arm must be pulled exactly once during the first A A A rounds, yielding an initial empirical mean estimate for every arm.
In what follows, we prove Theorem 5.1 with the aid of the regret decomposition (367) , focusing on t ≥ A t \ge A t ≥ A . For notational clarity, we use N t ( a ) N_t(a) N t ( a ) , r t u c b ( a ) r_t^{\mathrm{ucb}}(a) r t ucb ( a ) , and r ˉ t ( a ) \bar{r}_t(a) r ˉ t ( a ) to denote the values of the running variables N ( a ) N(a) N ( a ) , r u c b ( a ) r^{\mathrm{ucb}}(a) r ucb ( a ) , and r ˉ ( a ) \bar{r}(a) r ˉ ( a ) , respectively, at the end of round t t t .
Step 1: connecting N t ( a ) N_t(a) N t ( a ) with the arm-pulling probability. Consider any given a ∈ A a \in\mathcal{A} a ∈ A . We first identify a condition under which the UCB estimate r t u c b ( a ) r_t^{\mathrm{ucb}}(a) r t ucb ( a ) of a suboptimal arm a a a is smaller than that of the optimal arm a ⋆ a^\star a ⋆ (cf. (356) ), and hence arm a a a will not be pulled at the beginning of round t + 1 t+1 t + 1 . More specifically, if
N t ( a ) > 8 log T 2 δ ( Δ ( a ) ) 2 (368) N_t(a) > \frac{8\log\frac{T^2}{\delta}}{(\Delta(a))^2} \tag*{(368)} N t ( a ) > ( Δ ( a ) ) 2 8 log δ T 2 (368)
holds for t ≥ A t \ge A t ≥ A , then with probability at least 1 − 2 δ / T 2 1 - 2\delta/T^2 1 − 2 δ / T 2 ,
r t u c b ( a ) = r ˉ t ( a ) + 2 log T 2 / δ N t ( a ) ≤ ( i ) r ( a ) + 2 2 log T 2 / δ N t ( a ) < ( i i ) r ( a ∗ ) ≤ ( i i i ) r t u c b ( a ∗ ) . r_t^{\mathrm{ucb}}(a) = \bar{r}_t(a) + \sqrt{\frac{2\log T^2/\delta}{N_t(a)}} \mathrel{\mathop{\le}^{\mathrm{(i)}}} r(a) + 2\sqrt{\frac{2\log T^2/\delta}{N_t(a)}} \mathrel{\mathop{<}^{\mathrm{(ii)}}} r(a^*) \mathrel{\mathop{\le}^{\mathrm{(iii)}}} r_t^{\mathrm{ucb}}(a^*). r t ucb ( a ) = r ˉ t ( a ) + N t ( a ) 2 log T 2 / δ ≤ ( i ) r ( a ) + 2 N t ( a ) 2 log T 2 / δ < ( ii ) r ( a ∗ ) ≤ ( iii ) r t ucb ( a ∗ ) .
where (i) results from (364) , (ii) is a consequence of Condition (368) , and (iii) follows from (363) . As a result, for any t ≥ A t \ge A t ≥ A and any given a ∈ A a \in\mathcal{A} a ∈ A ,
P ( a t + 1 = a , N t ( a ) > 8 log T 2 / δ ( Δ ( a ) ) 2 ) = P ( a t + 1 = a ∣ N t ( a ) > 8 log T 2 / δ ( Δ ( a ) ) 2 ) P ( N t ( a ) > 8 log T 2 / δ ( Δ ( a ) ) 2 ) ≤ P ( a t + 1 = a ∣ N t ( a ) > 8 log T 2 / δ ( Δ ( a ) ) 2 ) ≤ 2 δ T 2 . (369) \mathbb{P}\left(a_{t+1}=a,N_t(a)>\frac{8\log T^2/\delta}{(\Delta(a))^2}\right)
= \mathbb{P}\left(a_{t+1}=a\mid N_t(a)>\frac{8\log T^2/\delta}{(\Delta(a))^2}\right)\mathbb{P}\left(N_t(a)>\frac{8\log T^2/\delta}{(\Delta(a))^2}\right)
\le\mathbb{P}\left(a_{t+1}=a\mid N_t(a)>\frac{8\log T^2/\delta}{(\Delta(a))^2}\right)\le\frac{2\delta}{T^2}. \tag*{(369)} P ( a t + 1 = a , N t ( a ) > ( Δ ( a ) ) 2 8 log T 2 / δ ) = P ( a t + 1 = a ∣ N t ( a ) > ( Δ ( a ) ) 2 8 log T 2 / δ ) P ( N t ( a ) > ( Δ ( a ) ) 2 8 log T 2 / δ ) ≤ P ( a t + 1 = a ∣ N t ( a ) > ( Δ ( a ) ) 2 8 log T 2 / δ ) ≤ T 2 2 δ . (369)
Step 2: deriving an instance-dependent bound on E [ N T ( a ) ] \mathbb{E}[N_T(a)] E [ N T ( a )] . Equipped with the properties derived in Step 1, we can demonstrate that, for any arm a a a with Δ ( a ) > 0 \Delta(a)>0 Δ ( a ) > 0 ,
E [ N T ( a ) ] = 1 + E [ ∑ t = A T − 1 1 ( a t + 1 = a ) ] = 1 + E [ ∑ t = A T − 1 1 ( a t + 1 = a , N t ( a ) ≤ 8 log T 2 / δ ( Δ ( a ) ) 2 ) ] + E [ ∑ t = A T − 1 1 ( a t + 1 = a , N t ( a ) > 8 log T 2 / δ ( Δ ( a ) ) 2 ) ] \begin{aligned}
\mathbb{E}[N_T(a)] = 1+\mathbb{E}\left[\sum_{t=A}^{T-1}\mathbb{1}(a_{t+1}=a)\right] \\
&=1+\mathbb{E}\left[\sum_{t=A}^{T-1}\mathbb{1}\left(a_{t+1}=a,N_t(a)\le\frac{8\log T^2/\delta}{(\Delta(a))^2}\right)\right]+\mathbb{E}\left[\sum_{t=A}^{T-1}\mathbb{1}\left(a_{t+1}=a,N_t(a)>\frac{8\log T^2/\delta}{(\Delta(a))^2}\right)\right]
\end{aligned} E [ N T ( a )] = 1 + E [ t = A ∑ T − 1 1 ( a t + 1 = a ) ] = 1 + E [ t = A ∑ T − 1 1 ( a t + 1 = a , N t ( a ) ≤ ( Δ ( a ) ) 2 8 log T 2 / δ ) ] + E [ t = A ∑ T − 1 1 ( a t + 1 = a , N t ( a ) > ( Δ ( a ) ) 2 8 log T 2 / δ ) ]
≤ 1 + 8 log T 2 / δ ( Δ ( a ) ) 2 + ∑ t = A T − 1 P ( a t + 1 = a , N t ( a ) > 8 log T 2 / δ ( Δ ( a ) ) 2 ) \le1+\frac{8\log T^2/\delta}{(\Delta(a))^2}+\sum_{t=A}^{T-1}\mathbb{P}\left(a_{t+1}=a,N_t(a)>\frac{8\log T^2/\delta}{(\Delta(a))^2}\right) ≤ 1 + ( Δ ( a ) ) 2 8 log T 2 / δ + t = A ∑ T − 1 P ( a t + 1 = a , N t ( a ) > ( Δ ( a ) ) 2 8 log T 2 / δ )
≤ 1 + 8 log T 2 / δ ( Δ ( a ) ) 2 + ∑ t = A T − 1 2 δ T 2 ≤ 8 log T 2 / δ ( Δ ( a ) ) 2 + 2. (370) \begin{aligned}
\le1+\frac{8\log T^2/\delta}{(\Delta(a))^2}+\sum_{t=A}^{T-1}\frac{2\delta}{T^2} \\
&\le\frac{8\log T^2/\delta}{(\Delta(a))^2}+2. \tag*{(370)}
\end{aligned} ≤ 1 + ( Δ ( a ) ) 2 8 log T 2 / δ + t = A ∑ T − 1 T 2 2 δ ≤ ( Δ ( a ) ) 2 8 log T 2 / δ + 2. (370)
Here, the first inequality follows because arm a a a is pulled exactly once during the first A A A rounds, and thereafter the event
{ a t + 1 = a , N t ( a ) ≤ 8 log ( T 2 / δ ) ( Δ ( a ) ) 2 } \left\{a_{t+1}=a,\;N_t(a)\le\frac{8\log(T^2/\delta)}{(\Delta(a))^2}\right\} { a t + 1 = a , N t ( a ) ≤ ( Δ ( a ) ) 2 8 log ( T 2 / δ ) }
can occur at most 8 log ( T 2 / δ ) ( Δ ( a ) ) 2 \frac{8\log(T^2/\delta)}{(\Delta(a))^2} ( Δ ( a ) ) 2 8 l o g ( T 2 / δ ) times, as each occurrence increments N t ( a ) N_t(a) N t ( a ) by one. The penultimate line in (370) results from (369) , whereas the last inequality in (370) holds since δ ≤ 1 \delta\le1 δ ≤ 1 and T ≥ A ≥ 2 T\ge A\ge2 T ≥ A ≥ 2 .
Step 3: establishing the instance-dependent regret bound. Substituting (370) into the regret decomposition (367) establishes the advertised instance-dependent regret bound (i.e., (366) ):
E [ Regret ( T ) ] ≤ ∑ a : Δ ( a ) > 0 Δ ( a ) ( 8 log T 2 / δ ( Δ ( a ) ) 2 + 2 ) ≤ ∑ a : Δ ( a ) > 0 10 log T 2 / δ Δ ( a ) , \mathbb{E}[\operatorname{Regret}(T)]\le\sum_{a:\Delta(a)>0}\Delta(a)\left(\frac{8\log T^2/\delta}{(\Delta(a))^2}+2\right)\le\sum_{a:\Delta(a)>0}\frac{10\log T^2/\delta}{\Delta(a)}, E [ Regret ( T )] ≤ a : Δ ( a ) > 0 ∑ Δ ( a ) ( ( Δ ( a ) ) 2 8 log T 2 / δ + 2 ) ≤ a : Δ ( a ) > 0 ∑ Δ ( a ) 10 log T 2 / δ ,
provided that Δ ( a ) ≤ 1 \Delta(a)\le1 Δ ( a ) ≤ 1 holds for all a ∈ A a\in\mathcal{A} a ∈ A .
Step 4: establishing the worst-case regret bound. With the instance-dependent bound (370) on E [ N T ( a ) ] \mathbb{E}[N_T(a)] E [ N T ( a )] in place, we can proceed to establish the claimed worst-case regret bound (365) . Considering any threshold Δ > 0 \Delta> 0 Δ > 0 (to be specified shortly), we make the following observations.
E [ N T ( a ) ] ≤ 8 log T 2 δ ( Δ ( a ) ) 2 + 2. \mathbb{E}[N_T(a)] \le\frac{8\log\frac{T^2}{\delta}}{(\Delta(a))^2} + 2. E [ N T ( a )] ≤ ( Δ ( a ) ) 2 8 log δ T 2 + 2.
∑ a E [ N T ( a ) ] ≤ T . \sum_a \mathbb{E}[N_T(a)] \le T. a ∑ E [ N T ( a )] ≤ T .
Combine these bounds with the regret decomposition (367) to reach
E [ Regret ( T ) ] ≤ ∑ a : Δ ( a ) ≥ Δ Δ ( a ) ( 8 log T 2 δ ( Δ ( a ) ) 2 + 2 ) + Δ ∑ a : Δ a < Δ E [ N T ( a ) ] ≤ 8 A log T 2 δ Δ + 2 A + T Δ , \begin{aligned}
\mathbb{E}[\operatorname{Regret}(T)] \le\sum_{a:\Delta(a)\ge\Delta} \Delta(a)\left(\frac{8\log\frac{T^2}{\delta}}{(\Delta(a))^2}+2\right)+\Delta\sum_{a:\Delta_a<\Delta}\mathbb{E}[N_T(a)] \\
&\le\frac{8A\log\frac{T^2}{\delta}}{\Delta}+2A+T\Delta,
\end{aligned} E [ Regret ( T )] ≤ a : Δ ( a ) ≥ Δ ∑ Δ ( a ) ( ( Δ ( a ) ) 2 8 log δ T 2 + 2 ) + Δ a : Δ a < Δ ∑ E [ N T ( a )] ≤ Δ 8 A log δ T 2 + 2 A + T Δ ,
where the last line also uses the assumption that Δ ( a ) ≤ 1 \Delta(a) \le1 Δ ( a ) ≤ 1 for each a ∈ A a \in\mathcal{A} a ∈ A . Choosing Δ = 4 A log ( T 2 / δ ) T \Delta= \sqrt{\frac{4A\log(T^2/\delta)}{T}} Δ = T 4 A l o g ( T 2 / δ ) , we arrive at
E [ Regret ( T ) ] ≤ 6 A T log T 2 δ + 2 A ≤ 8 A T log T 2 δ , \mathbb{E}[\operatorname{Regret}(T)] \le6\sqrt{AT\log\frac{T^2}{\delta}}+2A \le8\sqrt{AT\log\frac{T^2}{\delta}}, E [ Regret ( T )] ≤ 6 A T log δ T 2 + 2 A ≤ 8 A T log δ T 2 ,
provided that T ≥ A T \ge A T ≥ A . This completes the proof.
A model-based online RL algorithm: UCB-VI
Recall that in the generative model setting, model-based algorithms have been shown to achieve minimax-optimal sample complexity, even in the most data-hungry regime. Such results provide strong incentives for studying the model-based paradigm in the online RL setting as well. Given the proven efficiency of the optimism principle in multi-armed bandits (see § 8.2 ), it is natural to investigate how this principle can be effectively integrated into model-based online RL algorithms.
Algorithmic ideas
To begin with, the model-based policy learning algorithm introduced previously consists of a model estimation stage, where an empirical estimate of the probability transition kernel is constructed from collected data, followed by a planning stage that computes a policy by treating the empirical model as if it were the true environment. This naturally suggests a paradigm that integrates model-based policy learning with online decision-making and sampling, which operates as follows in each episode.
While conceptually intuitive, this approach lacks one crucial component: exploration. By focusing exclusively on exploitation — optimizing solely w.r.t. the current empirical model — the agent may fail to sufficiently explore regions of the state-action space that are poorly understood yet potentially highly rewarding.
To address this issue, a natural strategy is to incorporate the optimism principle in the face of uncertainty into the above framework, namely, to guide exploration by acting according to an optimistic view of the world. Compared with stochastic bandits (§ 8.2 ), where uncertainty arises primarily from noisy rewards, the online RL formulation considered herein faces uncertainty stemming from unknown transition kernels. Consequently, there are multiple objects around which one may construct confidence regions, including the transition kernel itself and the corresponding optimal value and Q Q Q -functions. The former (i.e., confidence regions for the transition kernel) led to some of the earliest UCB-style algorithms for online RL, most notably UCRL and UCRL2 (Auer and Ortner, 2006; Jaksch et al., 2010), whereas the latter (i.e., optimism over the optimal value and Q Q Q -function) led to the development of UCB-VI (Azar et al., 2017), the first online RL algorithm achieving asymptotically near-optimal regret. Since the primary focus of this work is sample efficiency, we concentrate on the UCB-VI algorithm and its variants.
In a nutshell, the UCB-VI algorithm — short for upper confidence bound value iteration and originally proposed by Azar et al. (2017) — implements the optimism principle by constructing upper confidence bounds on the optimal Q Q Q -function in each episode, and subsequently performing policy updates and data collection in accordance with these optimistic Q Q Q -value estimates rather than the unadjusted empirical estimates. To be more precise, in each episode k k k , the algorithm proceeds as follows, assuming the Q Q Q -value estimates are initialized optimistically as Q h u c b ( s , a ) = H Q_h^{\mathrm{ucb}}(s,a)=H Q h ucb ( s , a ) = H for all ( s , a , h ) ∈ S × A × [ H ] (s,a,h)\in\mathcal{S}\times\mathcal{A}\times[H] ( s , a , h ) ∈ S × A × [ H ] .
Sampling. Compute the current policy π u c b = { π h u c b } 1 ≤ h ≤ H \pi^{\mathrm{ucb}}=\{\pi_h^{\mathrm{ucb}}\}_{1\le h\le H} π ucb = { π h ucb } 1 ≤ h ≤ H w.r.t. the optimistic Q Q Q -estimates { Q h u c b } 1 ≤ h ≤ H \{Q_h^{\mathrm{ucb}}\}_{1\le h\le H} { Q h ucb } 1 ≤ h ≤ H via a greedy rule, i.e.,
π h u c b ( s ) ← arg max a Q h u c b ( s , a ) , ∀ ( s , h ) ∈ S × [ H ] . \pi_h^{\mathrm{ucb}}(s)\gets\arg\max_a Q_h^{\mathrm{ucb}}(s,a),\qquad\forall(s,h)\in\mathcal{S}\times[H]. π h ucb ( s ) ← arg a max Q h ucb ( s , a ) , ∀ ( s , h ) ∈ S × [ H ] .
Armed with the updated policy estimate π u c b \pi^{\mathrm{ucb}} π ucb , the agent executes π u c b \pi^{\mathrm{ucb}} π ucb in the MDP at the beginning of the episode, collecting a sample trajectory denoted by
( s 1 k , a 1 k , … , s H k , a H k , s H + 1 k ) . (s_1^k,a_1^k,\ldots,s_H^k,a_H^k,s_{H+1}^k). ( s 1 k , a 1 k , … , s H k , a H k , s H + 1 k ) .
Model estimation. Compute an empirical estimate P ^ = { P ^ h ∈ R S A × S } 1 ≤ h ≤ H \widehat{P}=\{\widehat{P}_h\in\mathbb{R}^{S A\times S}\}_{1\le h\le H} P = { P h ∈ R S A × S } 1 ≤ h ≤ H of the transition probability kernel using all previously collected samples. More specifically, the empirical model P ^ \widehat{P} P is constructed such that, for all ( s , a , h , s ′ ) ∈ S × A × [ H ] × S (s,a,h,s')\in\mathcal{S}\times\mathcal{A}\times[H]\times\mathcal{S} ( s , a , h , s ′ ) ∈ S × A × [ H ] × S ,
P ^ h ( s ′ ∣ s , a ) = N h ( s , a , s ′ ) N h ( s , a ) , (371) \widehat{P}_h(s'\mid s,a)=\frac{N_h(s,a,s')}{N_h(s,a)}, \tag*{(371)} P h ( s ′ ∣ s , a ) = N h ( s , a ) N h ( s , a , s ′ ) , (371)
where N h ( s , a ) N_h(s,a) N h ( s , a ) denotes the number of visits to the state-action-step tuple ( s , a , h ) (s,a,h) ( s , a , h ) up to the current episode, and N h ( s , a , s ′ ) N_h(s,a,s') N h ( s , a , s ′ ) denotes the number of such visits that result in a transition to state s ′ s' s ′ at step h + 1 h+1 h + 1 . Throughout this section, we often use the notation P ^ s , a , h : = [ P ^ h ( s ′ ∣ s , a ) ] s ′ ∈ S \widehat{P}_{s,a,h}:=[\widehat{P}_h(s'\mid s,a)]_{s'\in\mathcal{S}} P s , a , h := [ P h ( s ′ ∣ s , a ) ] s ′ ∈ S to denote the empirical transition vector associated with the state-action pair ( s , a ) (s,a) ( s , a ) at step h h h . If the tuple ( s , a , h ) (s,a,h) ( s , a , h ) has not been visited, one may simply set P ^ s , a , h ( s ′ ) = 1 / S \widehat{P}_{s,a,h}(s')=1/S P s , a , h ( s ′ ) = 1/ S for all s ′ ∈ S s'\in\mathcal{S} s ′ ∈ S , so that P ^ s , a , h \widehat{P}_{s,a,h} P s , a , h remains a valid probability distribution.
Algorithm 5.2: The upper confidence bound value iteration (UCB-VI) algorithm for episodic online RL
input: number of episodes K K K , reward function r r r , confidence parameter δ \delta δ , constants c 1 , c 2 > 0 c_1, c_2 > 0 c 1 , c 2 > 0 .
initialize N h ( s , a , s ′ ) ← 0 N_h(s,a,s') \leftarrow0 N h ( s , a , s ′ ) ← 0 , N h ( s , a ) ← 0 N_h(s,a) \leftarrow0 N h ( s , a ) ← 0 , Q h u c b ( s , a ) ← H Q_h^{\mathrm{ucb}}(s,a) \leftarrow H Q h ucb ( s , a ) ← H , V h u c b ( s ) ← H V_h^{\mathrm{ucb}}(s) \leftarrow H V h ucb ( s ) ← H , V H + 1 u c b ( s ) ← 0 V_{H+1}^{\mathrm{ucb}}(s) \leftarrow0 V H + 1 ucb ( s ) ← 0 , P ^ s , a , h ( s ′ ) ← 1 / S \widehat{P}_{s,a,h}(s') \leftarrow1/S P s , a , h ( s ′ ) ← 1/ S for all ( s , a , s ′ , h ) ∈ S × A × S × [ H ] (s,a,s',h) \in\mathcal{S} \times\mathcal{A} \times\mathcal{S} \times[H] ( s , a , s ′ , h ) ∈ S × A × S × [ H ] .
for k = 1 , ⋯ , K k = 1,\cdots,K k = 1 , ⋯ , K do /* policy update. */
compute policy π u c b \pi^{\mathrm{ucb}} π ucb such that π h u c b ( s ) ← arg max a Q h u c b ( s , a ) \pi_h^{\mathrm{ucb}}(s) \leftarrow\arg\max_a Q_h^{\mathrm{ucb}}(s,a) π h ucb ( s ) ← arg max a Q h ucb ( s , a ) for all ( s , h ) ∈ S × [ H ] (s,h) \in\mathcal{S} \times[H] ( s , h ) ∈ S × [ H ] .
observe the initial state s 1 ∼ i n d . ρ s_1 \mathrel{\mathop{\sim}^{\mathrm{ind.}}} \rho s 1 ∼ ind. ρ of a new episode.
for h = 1 , ⋯ , H h = 1,\cdots,H h = 1 , ⋯ , H do /* sampling. */
take a h = π h u c b ( s h ) a_h = \pi_h^{\mathrm{ucb}}(s_h) a h = π h ucb ( s h ) , observe s h + 1 s_{h+1} s h + 1 ; set ( s , a , s ′ ) ← ( s h , a h , s h + 1 ) (s,a,s') \leftarrow(s_h,a_h,s_{h+1}) ( s , a , s ′ ) ← ( s h , a h , s h + 1 ) . /* empirical model update. */
N h ( s , a ) ← N h ( s , a ) + 1 N_h(s,a) \leftarrow N_h(s,a) + 1 N h ( s , a ) ← N h ( s , a ) + 1 ; N h ( s , a , s ′ ) ← N h ( s , a , s ′ ) + 1 N_h(s,a,s') \leftarrow N_h(s,a,s') + 1 N h ( s , a , s ′ ) ← N h ( s , a , s ′ ) + 1 .
P ^ s , a , h ( s ~ ) ← N h ( s , a , s ~ ) N h ( s , a ) \widehat{P}_{s,a,h}(\widetilde{s}) \leftarrow\frac{N_h(s,a,\widetilde{s})}{N_h(s,a)} P s , a , h ( s ) ← N h ( s , a ) N h ( s , a , s ) for all s ~ ∈ S \widetilde{s} \in\mathcal{S} s ∈ S . /* optimistic updates of Q-functions & values. */
for h = H , ⋯ , 1 h = H,\cdots,1 h = H , ⋯ , 1 do
for ( s , a ) ∈ S × A (s,a) \in\mathcal{S} \times\mathcal{A} ( s , a ) ∈ S × A do
compute the bonus b h ( s , a ) b_h(s,a) b h ( s , a ) according to (5.20); update
Q h u c b ( s , a ) ← min { r h ( s , a ) + ⟨ P ^ s , a , h , V h + 1 u c b ⟩ + b h ( s , a ) , H } (372) Q_h^{\mathrm{ucb}}(s,a) \leftarrow\min\left\{r_h(s,a) + \left\langle\widehat{P}_{s,a,h}, V_{h+1}^{\mathrm{ucb}}\right\rangle+ b_h(s,a), H\right\} \tag*{(372)} Q h ucb ( s , a ) ← min { r h ( s , a ) + ⟨ P s , a , h , V h + 1 ucb ⟩ + b h ( s , a ) , H } (372)
V h u c b ( s ) ← max a Q h u c b ( s , a ) . (373) V_h^{\mathrm{ucb}}(s) \leftarrow\max_a Q_h^{\mathrm{ucb}}(s,a). \tag*{(373)} V h ucb ( s ) ← a max Q h ucb ( s , a ) . (373)
Optimistic Q-function estimation. Maintain upper confidence estimates of the optimal Q-function and value function. More concretely, working backward for h = H , … , 1 h = H,\ldots,1 h = H , … , 1 , one computes, for every state-action-step tuple,
Q h u c b ( s , a ) ← min { r h ( s , a ) + ⟨ P ^ h ( ⋅ ∣ s , a ) , V h + 1 u c b ⟩ + b h ( s , a ) , H } Q_h^{\mathrm{ucb}}(s,a) \leftarrow\min\left\{r_h(s,a) + \left\langle\widehat{P}_h(\cdot\mid s,a), V_{h+1}^{\mathrm{ucb}}\right\rangle+ b_h(s,a), H\right\} Q h ucb ( s , a ) ← min { r h ( s , a ) + ⟨ P h ( ⋅ ∣ s , a ) , V h + 1 ucb ⟩ + b h ( s , a ) , H }
V h u c b ( s ) ← max a Q h u c b ( s , a ) V_h^{\mathrm{ucb}}(s) \leftarrow\max_a Q_h^{\mathrm{ucb}}(s,a) V h ucb ( s ) ← a max Q h ucb ( s , a )
from the initial value V H + 1 u c b = 0 V_{H+1}^{\mathrm{ucb}} = 0 V H + 1 ucb = 0 , where Q u c b Q^{\mathrm{ucb}} Q ucb (resp. V u c b V^{\mathrm{ucb}} V ucb ) indicates the running estimate for the Q-function (resp. value function), and b h ( s , a ) ≥ 0 b_h(s,a) \ge0 b h ( s , a ) ≥ 0 is some suitably chosen exploration bonus that accounts for statistical uncertainty. Here, the iterate Q h u c b ( s , a ) Q_h^{\mathrm{ucb}}(s,a) Q h ucb ( s , a ) is explicitly capped at H H H to reflect the known feasible range [ 0 , H ] [0,H] [ 0 , H ] .
The full procedure is described in Algorithm 5.2.
At a high level, the UCB-VI algorithm makes decisions adaptively by favoring actions with either large estimated Q-values or high uncertainty in the Q-value estimates, thereby balancing exploitation with exploration in a principled manner. From a computational standpoint, each episode requires running a value-iteration-type procedure, whose complexity is roughly comparable to that of solving a fully specified finite-horizon MDP. Consequently, the overall computational complexity of UCB-VI is polynomial in the problem parameters.
Choices of exploration bonuses
A crucial design element of UCB-VI is the exploration bonus b h ( s , a ) b_h(s,a) b h ( s , a ) , which needs to be carefully constructed to ensure a valid, yet tight, upper confidence bound on the optimal Q Q Q -function (i.e., Q h u c b ( s , a ) ≥ Q h ⋆ ( s , a ) Q_h^{\mathrm{ucb}}(s,a) \ge Q_h^\star(s,a) Q h ucb ( s , a ) ≥ Q h ⋆ ( s , a ) ). As is evident from the algorithm description, the primary source of uncertainty in Q-function estimation in each iteration of dynamic programming arises from the term ⟨ P ^ h ( ⋅ ∣ s , a ) , V h + 1 u c b ⟩ \langle\widehat{P}_h(\cdot\mid s,a), V_{h+1}^{\mathrm{ucb}}\rangle ⟨ P h ( ⋅ ∣ s , a ) , V h + 1 ucb ⟩ appearing in (372) , whereas the ideal quantity would be the population-level counterpart ⟨ P h ( ⋅ ∣ s , a ) , V h + 1 u c b ⟩ \langle P_h(\cdot\mid s,a), V_{h+1}^{\mathrm{ucb}}\rangle ⟨ P h ( ⋅ ∣ s , a ) , V h + 1 ucb ⟩ . Two common choices for designing b h ( s , a ) b_h(s,a) b h ( s , a ) are obtained by applying the Hoeffding inequality or the Bernstein inequality to control the deviation
⟨ P ^ h ( ⋅ ∣ s , a ) − P h ( ⋅ ∣ s , a ) , V h + 1 u c b ⟩ , (374) \left\langle\widehat{P}_h(\cdot\mid s,a)-P_h(\cdot\mid s,a),V_{h+1}^{\mathrm{ucb}}\right\rangle, \tag*{(374)} ⟨ P h ( ⋅ ∣ s , a ) − P h ( ⋅ ∣ s , a ) , V h + 1 ucb ⟩ , (374)
which we introduce below. Throughout, we let δ > 0 \delta> 0 δ > 0 denote a confidence parameter (so that 1 − δ 1-\delta 1 − δ is the target success probability), and recall that N h ( s , a ) N_h(s,a) N h ( s , a ) is the visitation count of the state-action-step tuple ( s , a , h ) (s,a,h) ( s , a , h ) .
b h ( s , a ) = Θ ~ ( H 2 N h ( s , a ) ) . b_h(s,a)=\widetilde{\Theta}\left(\sqrt{\frac{H^2}{N_h(s,a)}}\right). b h ( s , a ) = Θ ( N h ( s , a ) H 2 ) .
This choice of upper confidence width is obtained by bounding the deviation term in (374) via the Hoeffding inequality (see Lemma 2.4 ) together with the fact that, by construction, the optimistic value estimate V h + 1 u c b ( s ) V_{h+1}^{\mathrm{ucb}}(s) V h + 1 ucb ( s ) falls within [ 0 , H ] [0,H] [ 0 , H ] .
Bernstein-style bonus. In contrast, the Bernstein-type bonus incorporates variance information to form tighter confidence bounds, often leading to improved statistical guarantees compared to their Hoeffding-based counterparts. By applying the Bernstein inequality (see Lemma 2.5 ) and using the fact that ∥ V h + 1 u c b ∥ ∞ ≤ H \|V_{h+1}^{\mathrm{ucb}}\|_\infty\le H ∥ V h + 1 ucb ∥ ∞ ≤ H , we see that the deviation term (374) can be bounded, with high probability, by
O ~ ( Var P s , a , h ( V h + 1 u c b ) N h ( s , a ) + H N h ( s , a ) ) . \widetilde{O}\left(\sqrt{\frac{\operatorname{Var}_{P_{s,a,h}}(V_{h+1}^{\mathrm{ucb}})}{N_h(s,a)}}+\frac{H}{N_h(s,a)}\right). O N h ( s , a ) Var P s , a , h ( V h + 1 ucb ) + N h ( s , a ) H .
where we recall the definition of the variance term in (61) . Nevertheless, since the true transition kernel P s , a , h P_{s,a,h} P s , a , h is unknown, this variance term Var P s , a , h ( ⋅ ) \operatorname{Var}_{P_{s,a,h}}(\cdot) Var P s , a , h ( ⋅ ) cannot be evaluated directly in practice. A natural remedy is to replace it with the empirical variance Var P ^ s , a , h ( ⋅ ) \operatorname{Var}_{\widehat{P}_{s,a,h}}(\cdot) Var P s , a , h ( ⋅ ) computed under the empirical model. This yields the following Bernstein-style exploration bonus:
b h ( s , a ) = Θ ~ ( Var P ^ s , a , h ( V h + 1 u c b ) N h ( s , a ) + H N h ( s , a ) ) . b_h(s,a)=\widetilde{\Theta}\left(\sqrt{\frac{\operatorname{Var}_{\widehat{P}_{s,a,h}}(V_{h+1}^{\mathrm{ucb}})}{N_h(s,a)}}+\frac{H}{N_h(s,a)}\right). b h ( s , a ) = Θ N h ( s , a ) Var P s , a , h ( V h + 1 ucb ) + N h ( s , a ) H .
arising from the empirical Bernstein inequality stated in Lemma 2.6 .
We shall shortly see — to be formalized in Lemma 5.5 — that the above exploration bonuses suffice to guarantee, with high probability, that Q h u c b ( s , a ) ≥ Q h ⋆ ( s , a ) Q_h^{\mathrm{ucb}}(s,a) \ge Q_h^\star(s,a) Q h ucb ( s , a ) ≥ Q h ⋆ ( s , a ) holds for every state-action-step tuple ( s , a , h ) (s,a,h) ( s , a , h ) , thereby offering valid upper confidence bounds.
Motivated by insights from the generative model setting, which highlight the importance of exploiting variance structure for data-efficient design, this monograph focuses on adopting and analyzing Bernstein-style bonuses within UCB-VI and its variants. To facilitate precise discussion and analysis, we shall work with the following concrete choice throughout this chapter:
b h ( s , a ) = c 1 Var P ^ s , a , h ( V h + 1 u c b ) log 1 δ ′ N h ( s , a ) + c 2 H log 1 δ ′ N h ( s , a ) , (375) b_h(s,a)=c_1\sqrt{\frac{\operatorname{Var}_{\widehat{P}_{s,a,h}}(V_{h+1}^{\mathrm{ucb}})\log\frac{1}{\delta'}}{N_h(s,a)}}+c_2\frac{H\log\frac{1}{\delta'}}{N_h(s,a)}, \tag*{(375)} b h ( s , a ) = c 1 N h ( s , a ) Var P s , a , h ( V h + 1 ucb ) log δ ′ 1 + c 2 N h ( s , a ) H log δ ′ 1 , (375)
where c 1 , c 2 > 0 c_1,c_2>0 c 1 , c 2 > 0 are some universal constants, and δ ′ \delta' δ ′ is a suitably adjusted confidence parameter. Throughout the analysis of UCB-VI, we take
δ ′ = δ 100 S 2 A 2 H 2 K 2 ; (376) \delta'=\frac{\delta}{100S^2A^2H^2K^2}; \tag*{(376)} δ ′ = 100 S 2 A 2 H 2 K 2 δ ; (376)
this particular choice of δ ′ \delta' δ ′ made solely for technical convenience and should not be interpreted as a practical recommendation.
Remark 5.2. We note that the form of (5.20) differs from the Bernstein-style bonus proposed in the original work of Azar et al. (2017); instead, we adopt a simplified choice suggested by Zhang et al. (2021e) and Zhang et al. (2025e) in their design of MVP, an algorithm we will shortly introduce in § 8.8 .
Theoretical guarantees
As alluded to previously, the development of the UCB-VI algorithm (Azar et al., 2017) marked a milestone in online RL theory, leading to the first algorithm provably achieving asymptotically minimax-optimal regret. Its statistical guarantees are formally stated in the following theorem.
Theorem 5.2 (Azar et al. (2017)). Consider any K ≥ 1 K \ge1 K ≥ 1 and any 0 < δ < 1 0<\delta<1 0 < δ < 1 . Then with probability at least 1 − δ 1-\delta 1 − δ , the UCB-VI algorithm in Algorithm 5.2 with Bernstein-style bonuses (5.20) achieves
Regret ( K ) ≤ O ~ ( S A H 3 K + S 2 A H 3 ) , (377) \operatorname{Regret}(K)\le\widetilde{O}\left(\sqrt{SAH^3K}+S^2AH^3\right), \tag*{(377)} Regret ( K ) ≤ O ( S A H 3 K + S 2 A H 3 ) , (377) provided that the constants c 1 , c 2 > 0 c_1,c_2>0 c 1 , c 2 > 0 in (5.20) are suitably chosen. Here, in the definition of Regret ( K ) \operatorname{Regret}(K) Regret ( K ) (cf. (349) ), π k \pi^k π k and s 1 k s_1^k s 1 k denote, respectively, the sampling policy deployed and the initial state observed in episode k k k of Algorithm 5.2.
Remark 5.3. It is noteworthy that the regret bound presented in Azar et al. (2017) differs from that in Theorem 5.2 in its dependence on the horizon H H H , since the former considers finite-horizon stationary MDPs, whereas Theorem 5.2 is stated for finite-horizon nonstationary MDPs. Intriguingly, this also unveils that the UCB-VI algorithm is asymptotically optimal for finite-horizon stationary MDPs.
Notably, once the number of episodes is sufficiently large, i.e.,
K ≳ S 3 A H 3 , (378) K\gtrsim S^3AH^3, \tag*{(378)} K ≳ S 3 A H 3 , (378)
the lower-order term S 2 A H 3 S^2AH^3 S 2 A H 3 becomes negligible, and the regret bound in Theorem 5.2 simplifies to
Regret ( K ) ≤ O ~ ( S A H 3 K ) . (379) \operatorname{Regret}(K)\le\widetilde{O}\left(\sqrt{SAH^3K}\right). \tag*{(379)} Regret ( K ) ≤ O ( S A H 3 K ) . (379)
As we will rigorize in § 8.12 , this matches the minimax lower bound up to logarithmic factors. Consequently, UCB-VI is asymptotically optimal in terms of regret.
Additionally, when K ≳ S 3 A H 3 K \gtrsim S^3AH^3 K ≳ S 3 A H 3 , the above asymptotic regret bound (379) can be readily converted into a PAC-style sample complexity guarantee via the regret-to-PAC conversion in (5.3) from § 8.1 . Specifically, if one outputs a policy π ^ \hat{\pi} π ^ drawn uniformly at random from those sampling policies deployed over the K K K episodes, then
V 1 ⋆ ( ρ ) − V 1 π ^ ( ρ ) ≤ ε V_1^{\star}(\rho)-V_1^{\hat{\pi}}(\rho)\leq\varepsilon V 1 ⋆ ( ρ ) − V 1 π ^ ( ρ ) ≤ ε
holds with probability approaching one, provided that the total number of collected samples satisfies
T = K H ≥ Ω ~ ( S A H 4 ε 2 ) T=KH\geq\widetilde{\Omega}\left(\frac{SAH^4}{\varepsilon^2}\right) T = K H ≥ Ω ( ε 2 S A H 4 )
and ε \varepsilon ε is sufficiently small. Somewhat surprisingly, this PAC-style sample complexity coincides, up to logarithmic factors, with the minimax-optimal sample complexity limit under the generative model setting (see § 5.1 ). In some sense, this result suggests that, at least in the asymptotic regime and from a minimax perspective, online exploration via optimism incurs little additional statistical cost compared to sampling from an idealized generative model.
Nevertheless, the statistical optimality of UCB-VI is guaranteed only after a substantial burn-in period of order K ≳ S 3 A H 3 K \gtrsim S^3AH^3 K ≳ S 3 A H 3 , which can be prohibitively large in practical scenarios where the available number of episodes is limited. This limitation of the existing theoretical guarantees for UCB-VI has motivated a line of subsequent work aimed at reducing or eliminating the burn-in cost while retaining optimal regret guarantees.
Notation and intuition
Before embarking on the proof, we provide some intuition for why the asymptotic regret scales as O ~ ( S A H 3 K ) \widetilde{O}(\sqrt{SAH^3K}) O ( S A H 3 K ) .
Notation. To facilitate the discussion and subsequent analysis, we introduce augmented notation to explicitly index the iterates of Algorithm 5.2 by the episode number k k k as follows.
( s 1 k , a 1 k , … , s H k , a H k , s H + 1 k ) (s_1^k,a_1^k,\ldots,s_H^k,a_H^k,s_{H+1}^k) ( s 1 k , a 1 k , … , s H k , a H k , s H + 1 k ) : the sample trajectory observed during episode k k k , with s h k s_h^k s h k (resp. a h k a_h^k a h k ) representing the state (resp. action) at step h h h .
P ^ s , a , h k ∈ R S \widehat{P}_{s,a,h}^k \in\mathbb{R}^S P s , a , h k ∈ R S : the latest empirical transition probability vector P ^ s , a , h \widehat{P}_{s,a,h} P s , a , h computed before episode k k k .
N h k ( s , a ) N_h^k(s,a) N h k ( s , a ) : the visitation count N h ( s , a ) N_h(s,a) N h ( s , a ) for the ( s , a , h ) (s,a,h) ( s , a , h ) -tuple before episode k k k .
b h k ( s , a ) ≥ 0 b_h^k(s,a) \geq0 b h k ( s , a ) ≥ 0 : the latest bonus term b h ( s , a ) b_h(s,a) b h ( s , a ) before episode k k k .
Heuristic arguments. Below, we provide a heuristic discussion of the regret analysis. To simplify the exposition, we consider the asymptotic regime with K → ∞ K \to\infty K → ∞ , where we may informally regard V h k ≈ V h ⋆ ≈ V h π k V_h^k \approx V_h^\star\approx V_h^{\pi^k} V h k ≈ V h ⋆ ≈ V h π k and P ^ k ≈ P \widehat{P}^k \approx P P k ≈ P . These approximations allow us to focus on the main bottlenecks while ignoring lower-order error terms.
As in the multi-armed bandit setting in § 8.2 , an appropriate choice of exploration bonuses guarantees that the value estimates remain optimistic, namely, V 1 k ≥ V 1 ⋆ V_1^k \ge V_1^\star V 1 k ≥ V 1 ⋆ (which will be rigorized shortly). This optimism property immediately yields
Regret ( K ) = ∑ k = 1 K ( V 1 ⋆ ( s 1 k ) − V 1 π k ( s 1 k ) ) ≤ ∑ k = 1 K ( V 1 k ( s 1 k ) − V 1 π k ( s 1 k ) ) , (380) \operatorname{Regret}(K) = \sum_{k=1}^{K} \left(V_1^\star(s_1^k) - V_1^{\pi^k}(s_1^k)\right) \le\sum_{k=1}^{K} \left(V_1^k(s_1^k) - V_1^{\pi^k}(s_1^k)\right), \tag*{(380)} Regret ( K ) = k = 1 ∑ K ( V 1 ⋆ ( s 1 k ) − V 1 π k ( s 1 k ) ) ≤ k = 1 ∑ K ( V 1 k ( s 1 k ) − V 1 π k ( s 1 k ) ) , (380)
which is oftentimes more convenient to work with because π k \pi^k π k is chosen greedily w.r.t. the optimistic Q Q Q -values; as a result, both V 1 k V_1^k V 1 k and V 1 π k V_1^{\pi^k} V 1 π k can be obtained through compatible Bellman recursions, allowing their difference to be propagated recursively and conveniently across the horizon.
To better understand this over-estimation on the right-hand side of (380) , recall that in each episode k k k , the value estimate V 1 k V_1^k V 1 k is computed via backward induction from step H H H back to step 1 (see (5.18)). At each step h h h , the Q Q Q -estimate is inflated by a Bernstein-style bonus term:
b h k ( s , a ) ≲ Var P ^ s , a , h k ( V h + 1 k ) N h k ( s , a ) ≈ Var P s , a , h ( V h + 1 π k ) N h k ( s , a ) (ignoring log factors) , b_h^k(s,a) \lesssim\sqrt{\frac{\operatorname{Var}_{\widehat{P}_{s,a,h}^k}(V_{h+1}^k)}{N_h^k(s,a)}} \approx\sqrt{\frac{\operatorname{Var}_{P_{s,a,h}}(V_{h+1}^{\pi^k})}{N_h^k(s,a)}} \quad\text{(ignoring log factors)}, b h k ( s , a ) ≲ N h k ( s , a ) Var P s , a , h k ( V h + 1 k ) ≈ N h k ( s , a ) Var P s , a , h ( V h + 1 π k ) (ignoring log factors) ,
where we omit the O ( 1 / N h k ( s , a ) ) O(1/N_h^k(s,a)) O ( 1/ N h k ( s , a )) lower-order term for simplicity, as the variance-dependent term typically dominates in the asymptotic regime with K → ∞ K \to\infty K → ∞ . While this bonus is designed to compensate for model uncertainty, it also systematically inflates the value estimates. Since the regret is bounded above by the optimistic over-estimation in (380) , it is natural to quantify the cumulative amount of optimism introduced by these bonus terms:
E o v e r : = ∑ k = 1 K ∑ h = 1 H b h k ( s h k , a h k ) ≲ ∑ k = 1 K ∑ h = 1 H Var P s h k , a h k , h ( V h + 1 π k ) N h k ( s h k , a h k ) . \mathcal{E}_{\mathrm{over}} := \sum_{k=1}^{K}\sum_{h=1}^{H} b_h^k(s_h^k,a_h^k) \lesssim\sum_{k=1}^{K}\sum_{h=1}^{H}\sqrt{\frac{\operatorname{Var}_{P_{s_h^k,a_h^k,h}}(V_{h+1}^{\pi^k})}{N_h^k(s_h^k,a_h^k)}}. E over := k = 1 ∑ K h = 1 ∑ H b h k ( s h k , a h k ) ≲ k = 1 ∑ K h = 1 ∑ H N h k ( s h k , a h k ) Var P s h k , a h k , h ( V h + 1 π k ) .
To bound this cumulative over-estimation error, a standard application of the Cauchy-Schwarz inequality separates this quantity into two components:
E o v e r ≲ ∑ k , h 1 N h k ( s h k , a h k ) ⋅ ∑ k , h Var P s h k , a h k , h ( V h + 1 π k ) . (381) \mathcal{E}_{\mathrm{over}} \lesssim\sqrt{\sum_{k,h}\frac{1}{N_h^k(s_h^k,a_h^k)}}\cdot\sqrt{\sum_{k,h}\operatorname{Var}_{P_{s_h^k,a_h^k,h}}(V_{h+1}^{\pi^k})}. \tag*{(381)} E over ≲ k , h ∑ N h k ( s h k , a h k ) 1 ⋅ k , h ∑ Var P s h k , a h k , h ( V h + 1 π k ) . (381)
As it turns out, these two sums can be readily controlled via the following basic results.
Lemma 5.3. For any K ≥ 2 K \ge2 K ≥ 2 , Algorithm 5.2 satisfies
∑ k = 1 K ∑ h = 1 H 1 max { N h k ( s h k , a h k ) , 1 } ≤ 2 S A H log 2 K . (382) \sum_{k=1}^{K}\sum_{h=1}^{H}\frac{1}{\max\{N_h^k(s_h^k,a_h^k),1\}} \le2SAH\log_2 K. \tag*{(382)} k = 1 ∑ K h = 1 ∑ H max { N h k ( s h k , a h k ) , 1 } 1 ≤ 2 S A H log 2 K . (382)
Lemma 5.4. Fix a deterministic policy π \pi π . Generate a sample trajectory ( s 1 , a 1 , … , s H , a H ) (s_1,a_1,\ldots,s_H,a_H) ( s 1 , a 1 , … , s H , a H ) by executing policy π \pi π in the MDP. Then one has
E [ ∑ h = 1 H Var P s h , a h , h ( V h + 1 π ) ] ≤ H 2 , (383) \mathbb{E}\left[\sum_{h=1}^{H}\operatorname{Var}_{P_{s_h,a_h,h}}(V_{h+1}^{\pi})\right] \le H^2, \tag*{(383)} E [ h = 1 ∑ H Var P s h , a h , h ( V h + 1 π ) ] ≤ H 2 , (383) where the expectation is over the randomness of the sample trajectory.
Remark 5.4. The variance bound in Lemma 5.4 parallels the variance-based upper bounds — such as (90) in § 4.5 — that were previously established for, and play a pivotal role in the analysis of, γ \gamma γ -discounted infinite-horizon MDPs.
Assuming that the cumulative variance term (i.e., the second sum appearing in the upper bound (381) ) concentrates around its expectation, the preceding two lemmas imply that, with high probability,
E o v e r ≲ O ~ ( S A H ⋅ K H 2 ) = O ~ ( S A H 3 K ) . (384) \mathcal{E}_{\mathrm{over}} \lesssim\widetilde{O}\left(\sqrt{SAH}\cdot\sqrt{KH^2}\right)=\widetilde{O}\left(\sqrt{SAH^3K}\right). \tag*{(384)} E over ≲ O ( S A H ⋅ K H 2 ) = O ( S A H 3 K ) . (384)
Furthermore, since the right-hand side of (380) is often orderwise dominated by this over-estimation error (as we shall shortly rigorize), the above heuristic discussion suggests that, asymptotically,
Regret ( K ) ≲ O ~ ( S A H 3 K ) . \operatorname{Regret}(K)\lesssim\widetilde{O}\left(\sqrt{SAH^3K}\right). Regret ( K ) ≲ O ( S A H 3 K ) .
Finally, we note that the cumulative variance bound in Lemma 5.4 — whose statement and proof are both remarkably simple — is the key ingredient behind the asymptotically optimal regret guarantee. Indeed, if one were to bound each summand in (383) naively by H 2 H^2 H 2 without exploiting the dependence structure across the horizon, the resulting cumulative variance could scale as H 3 H^3 H 3 , thereby precluding asymptotically optimal regret guarantees. This observation further highlights the importance of incorporating variance information into the design of exploration bonuses.
Proof of Lemma 5.3. For any given ( s , a , h ) (s,a,h) ( s , a , h ) , it is easily seen that
∑ k = 1 K 1 max { N h k ( s h k , a h k ) , 1 } 1 { ( s , a ) = ( s h k , a h k ) } ≤ 1 + ∑ k = 1 K − 1 1 k ≤ 2 log 2 K . \sum_{k=1}^{K}\frac{1}{\max\{N_h^k(s_h^k,a_h^k),1\}}\mathbb{1}\{(s,a)=(s_h^k,a_h^k)\}\leq1+\sum_{k=1}^{K-1}\frac{1}{k}\leq2\log_2 K. k = 1 ∑ K max { N h k ( s h k , a h k ) , 1 } 1 1 {( s , a ) = ( s h k , a h k )} ≤ 1 + k = 1 ∑ K − 1 k 1 ≤ 2 log 2 K .
This holds since: (i) when ( s , a ) (s,a) ( s , a ) is visited for the first time, we have N h k ( s , a ) = 0 N_h^k(s,a)=0 N h k ( s , a ) = 0 by definition, so the corresponding summand equals 1; (ii) thereafter, each subsequent visit to ( s , a ) (s,a) ( s , a ) increments its associated visitation count by 1. Summing over all ( s , a , h ) (s,a,h) ( s , a , h ) tuples completes the proof. □
Proof of Lemma 5.4. Consider first any fixed pair ( s , h ) ∈ S × [ H ] (s,h)\in\mathcal{S}\times[H] ( s , h ) ∈ S × [ H ] . It is easy to verify that
Var P s , π h ( s ) , h ( V h + 1 π ) = ∑ s ′ ∈ S P s , π h ( s ) , h ( s ′ ) [ V h + 1 π ( s ′ ) − ⟨ P s , π h ( s ) , h , V h + 1 π ⟩ ] 2 = ∑ s ′ ∈ S P s , π h ( s ) , h ( s ′ ) [ V h + 1 π ( s ′ ) − V h π ( s ) + r h ( s , π h ( s ) ) ] 2 = E s ′ ∼ P s , π h ( s ) , h [ ( V h + 1 π ( s ′ ) − V h π ( s ) + r h ( s , π h ( s ) ) ) 2 ] , (385) \begin{aligned}
\operatorname{Var}_{P_{s,\pi_h(s),h}}\left(V_{h+1}^{\pi}\right)
&=\sum_{s'\in\mathcal{S}}P_{s,\pi_h(s),h}(s')\left[V_{h+1}^{\pi}(s')-\left\langle P_{s,\pi_h(s),h},V_{h+1}^{\pi}\right\rangle\right]^2\\
&=\sum_{s'\in\mathcal{S}}P_{s,\pi_h(s),h}(s')\left[V_{h+1}^{\pi}(s')-V_h^{\pi}(s)+r_h(s,\pi_h(s))\right]^2\\
&=\mathbb{E}_{s'\sim P_{s,\pi_h(s),h}}\left[\left(V_{h+1}^{\pi}(s')-V_h^{\pi}(s)+r_h(s,\pi_h(s))\right)^2\right], \tag*{(385)}
\end{aligned} Var P s , π h ( s ) , h ( V h + 1 π ) = s ′ ∈ S ∑ P s , π h ( s ) , h ( s ′ ) [ V h + 1 π ( s ′ ) − ⟨ P s , π h ( s ) , h , V h + 1 π ⟩ ] 2 = s ′ ∈ S ∑ P s , π h ( s ) , h ( s ′ ) [ V h + 1 π ( s ′ ) − V h π ( s ) + r h ( s , π h ( s )) ] 2 = E s ′ ∼ P s , π h ( s ) , h [ ( V h + 1 π ( s ′ ) − V h π ( s ) + r h ( s , π h ( s )) ) 2 ] , (385)
where the second identity results from the Bellman equation. The cumulative variance over the horizon then satisfies
E [ ∑ h = 1 H Var P s h , a h , h ( V h + 1 π ) ] = ( i ) ∑ h = 1 H E [ ( V h + 1 π ( s h + 1 ) − V h π ( s h ) + r h ( s h , a h ) ) 2 ] \mathbb{E}\left[\sum_{h=1}^{H}\operatorname{Var}_{P_{s_h,a_h,h}}\left(V_{h+1}^{\pi}\right)\right]\mathrel{\overset{(i)}{=}}\sum_{h=1}^{H}\mathbb{E}\left[\left(V_{h+1}^{\pi}(s_{h+1})-V_h^{\pi}(s_h)+r_h(s_h,a_h)\right)^2\right] E [ h = 1 ∑ H Var P s h , a h , h ( V h + 1 π ) ] = ( i ) h = 1 ∑ H E [ ( V h + 1 π ( s h + 1 ) − V h π ( s h ) + r h ( s h , a h ) ) 2 ]
= ( i i ) E [ ( ∑ h = 1 H [ V h + 1 π ( s h + 1 ) − V h π ( s h ) + r h ( s h , a h ) ] ) 2 ] . = E [ ( ∑ h = 1 H r h ( s h , a h ) − V 1 π ( s 1 ) ) 2 ] ≤ H 2 . \begin{aligned}
\mathrel{\overset{(ii)}{=}}\mathbb{E}\left[\left(\sum_{h=1}^{H}\left[V_{h+1}^{\pi}(s_{h+1})-V_h^{\pi}(s_h)+r_h(s_h,a_h)\right]\right)^2\right]. \\
&= \mathbb{E}\left[\left(\sum_{h=1}^{H} r_h(s_h,a_h)-V_1^\pi(s_1)\right)^2\right]\le H^2.
\end{aligned} = ( ii ) E ( h = 1 ∑ H [ V h + 1 π ( s h + 1 ) − V h π ( s h ) + r h ( s h , a h ) ] ) 2 . = E ( h = 1 ∑ H r h ( s h , a h ) − V 1 π ( s 1 ) ) 2 ≤ H 2 .
Here, (i) follows by using (385) and recalling that a h = π h ( s h ) a_h=\pi_h(s_h) a h = π h ( s h ) in the sample trajectory; (ii) comes from the Markov property of the trajectory, which ensures that the cross terms vanish; (iii) arises from telescoping together with the condition V H + 1 π = 0 V_{H+1}^\pi=0 V H + 1 π = 0 ; and (iv) holds because both ∑ h = 1 H r h ( s h , a h ) \sum_{h=1}^{H}r_h(s_h,a_h) ∑ h = 1 H r h ( s h , a h ) and V 1 π ( s 1 ) V_1^\pi(s_1) V 1 π ( s 1 ) reside within the interval [ 0 , H ] [0,H] [ 0 , H ] .
Analysis: proof of Theorem 5.2
In this subsection, we provide the proof of Theorem 5.2. While the proof below differs somewhat from the original arguments of Azar et al. (2017) — though remaining similar in spirit — it provides a treatment that is more coherent with the proof of the MVP algorithm to be presented in the next section. For notational simplicity, we introduce a logarithmic term ι b \iota_b ι b as follows:
ι b : = 40000 ( log 2 K ) 3 log ( 3 S A H ) log 1 δ ′ , (386) \iota_b := 40000(\log_2 K)^3\log(3SAH)\log\frac{1}{\delta'}, \tag*{(386)} ι b := 40000 ( log 2 K ) 3 log ( 3 S A H ) log δ ′ 1 , (386)
where δ ′ \delta' δ ′ is defined in (376) . We will also adopt the notation introduced in § 8.6.2 .
Before embarking on the main proof, we first observe that when K ≤ ι b S A H 2 K\le\iota_bSAH^2 K ≤ ι b S A H 2 , the regret bound in Theorem 5.2 follows trivially. Indeed, since the value function always lies within [ 0 , H ] [0,H] [ 0 , H ] , one has
Regret ( K ) = ∑ k = 1 K ( V 1 ⋆ ( s 1 k ) − V 1 π k ( s 1 k ) ) ≤ H K ≤ S A H 3 ι b ≤ O ~ ( S A H 3 K + S 2 A H 3 ) , \begin{aligned}
\operatorname{Regret}(K)=\sum_{k=1}^{K}\left(V_1^\star(s_1^k)-V_1^{\pi^k}(s_1^k)\right)\le HK\le SAH^3\iota_b \\
&\le\widetilde{O}\left(\sqrt{SAH^3K}+S^2AH^3\right),
\end{aligned} Regret ( K ) = k = 1 ∑ K ( V 1 ⋆ ( s 1 k ) − V 1 π k ( s 1 k ) ) ≤ H K ≤ S A H 3 ι b ≤ O ( S A H 3 K + S 2 A H 3 ) ,
which already satisfies the claimed regret bound in this regime. Consequently, throughout the remainder of the proof, it suffices to assume that
K ≥ S A H 2 ι b . (387) K\ge SAH^2\iota_b. \tag*{(387)} K ≥ S A H 2 ι b . (387)
Step 1: justifying optimism. We first establish that the running Q-function and value function estimates in Algorithm 5.2 serve as valid upper bounds for the true optimal Q-function and value function, respectively. This property provides a formal justification for the optimism principle underlying UCB-VI. Note that the same optimism property continues to hold if the exploration bonuses are replaced by confidence widths derived from alternative concentration inequalities, such as Hoeffding’s inequality.
Lemma 5.5 (Optimism). Suppose that the constants c 1 , c 2 > 0 c_1,c_2>0 c 1 , c 2 > 0 in (5.20) obey c 1 ≥ 4 c_1\ge4 c 1 ≥ 4 and c 2 ≥ c 1 2 c_2\ge c_1^2 c 2 ≥ c 1 2 . With probability exceeding 1 − 2 S A H K δ ′ 1-2SAHK\delta' 1 − 2 S A H K δ ′ ,
Q h k ( s , a ) ≥ Q h ⋆ ( s , a ) and V h k ( s ) ≥ V h ⋆ ( s ) (388) Q_h^k(s,a)\ge Q_h^\star(s,a)\qquad\text{and}\qquad V_h^k(s)\ge V_h^\star(s) \tag*{(388)} Q h k ( s , a ) ≥ Q h ⋆ ( s , a ) and V h k ( s ) ≥ V h ⋆ ( s ) (388) hold simultaneously for all ( s , a , h , k ) ∈ S × A × [ H ] × [ K ] (s,a,h,k)\in\mathcal{S}\times\mathcal{A}\times[H]\times[K] ( s , a , h , k ) ∈ S × A × [ H ] × [ K ] .
Intuitively, this result holds because the exploration bonus added in each iteration is sufficiently large to dominate the uncertainty in the Bellman-type updates (5.18) arising from the use of empirical model estimates. The proof is deferred to § 8.15.1 . Unless stated otherwise, all subsequent analysis is carried out on the high-probability event under which the inequalities in (388) hold.
Step 2: decomposing regret. In view of the optimism property established in Lemma 5.5 , the regret admits the following upper bound
Regret ( K ) = ∑ k = 1 K ( V 1 ⋆ ( s 1 k ) − V 1 π k ( s 1 k ) ) ≤ ∑ k = 1 K ( V 1 k ( s 1 k ) − V 1 π k ( s 1 k ) ) (389) \operatorname{Regret}(K) = \sum_{k=1}^{K}\left(V_1^\star(s_1^k)-V_1^{\pi^k}(s_1^k)\right) \le\sum_{k=1}^{K}\left(V_1^k(s_1^k)-V_1^{\pi^k}(s_1^k)\right) \tag*{(389)} Regret ( K ) = k = 1 ∑ K ( V 1 ⋆ ( s 1 k ) − V 1 π k ( s 1 k ) ) ≤ k = 1 ∑ K ( V 1 k ( s 1 k ) − V 1 π k ( s 1 k ) ) (389)
with probability at least 1 − 2 S A H K δ ′ 1-2SAHK\delta^\prime 1 − 2 S A H K δ ′ . The remainder of the analysis therefore focuses on upper bounding the right-hand side of (389) .
Remark 5.5. To better understand the rationale behind replacing V 1 ⋆ V_1^\star V 1 ⋆ with V 1 k V_1^k V 1 k , note that V 1 k ( s 1 k ) = max a Q 1 k ( s 1 k , a ) = Q 1 k ( s 1 k , π 1 k ( s 1 k ) ) V_1^k(s_1^k)=\max_a Q_1^k(s_1^k,a)=Q_1^k(s_1^k,\pi_1^k(s_1^k)) V 1 k ( s 1 k ) = max a Q 1 k ( s 1 k , a ) = Q 1 k ( s 1 k , π 1 k ( s 1 k )) due to the greedy policy update (cf. line 4 4 4 of Algorithm 5.2). Consequently, both V 1 k ( s 1 k ) V_1^k(s_1^k) V 1 k ( s 1 k ) and V 1 π k ( s 1 k ) V_1^{\pi^k}(s_1^k) V 1 π k ( s 1 k ) are, in some sense, aligned in terms of the selected actions. This alignment proves convenient when recursively expanding the difference V 1 k ( s 1 k ) − V 1 π k ( s 1 k ) V_1^k(s_1^k)-V_1^{\pi^k}(s_1^k) V 1 k ( s 1 k ) − V 1 π k ( s 1 k ) using Bellman-type updates.
To control the right-hand side of (389) , we begin by deriving the following upper bound on the optimistic value function estimate.
Lemma 5.6. For every 1 ≤ k ≤ K 1\leq k\leq K 1 ≤ k ≤ K , one has
V 1 k ( s 1 k ) ≤ ∑ h = 1 H { ⟨ P ^ s h k , a h k , h − P s h k , a h k , h , V h + 1 k ⟩ + b h k ( s h k , a h k ) + r h ( s h k , a h k ) + ⟨ P s h k , a h k , h − e s h + 1 k , V h + 1 k ⟩ } . V_1^k(s_1^k)\leq\sum_{h=1}^{H}\left\{\left\langle\widehat{P}_{s_h^k,a_h^k,h}-P_{s_h^k,a_h^k,h},V_{h+1}^k\right\rangle+b_h^k(s_h^k,a_h^k)+r_h(s_h^k,a_h^k)+\left\langle P_{s_h^k,a_h^k,h}-e_{s_{h+1}^k},V_{h+1}^k\right\rangle\right\}. V 1 k ( s 1 k ) ≤ h = 1 ∑ H { ⟨ P s h k , a h k , h − P s h k , a h k , h , V h + 1 k ⟩ + b h k ( s h k , a h k ) + r h ( s h k , a h k ) + ⟨ P s h k , a h k , h − e s h + 1 k , V h + 1 k ⟩ } . Here and throughout, e s e_s e s denotes the s s s -th standard basis vector.
Intuitively, this recurrence relation is derived by recursively unrolling V 1 k ( s 1 k ) V_1^k(s_1^k) V 1 k ( s 1 k ) via its update rule through the entire horizon. Crucially, both the empirical model estimation error P ^ s h k , a h k , h − P s h k , a h k , h \widehat{P}_{s_h^k,a_h^k,h}-P_{s_h^k,a_h^k,h} P s h k , a h k , h − P s h k , a h k , h and the sampling noise P s h k , a h k , h − e s h + 1 k P_{s_h^k,a_h^k,h}-e_{s_{h+1}^k} P s h k , a h k , h − e s h + 1 k are zero-mean fluctuations, whose cumulative contributions may potentially be controlled via suitable concentration inequalities. We include the proof here given its central role in the analysis of UCB-type online RL algorithms.
Proof of Lemma 5.6. According to the update rule (5.18), the value function estimates for each 1 ≤ h ≤ H 1\leq h\leq H 1 ≤ h ≤ H and 1 ≤ k ≤ K 1\leq k\leq K 1 ≤ k ≤ K satisfy
V h k ( s h k ) = max a ∈ A Q h k ( s h k , a ) = ( i ) Q h k ( s h k , a h k ) V_h^k(s_h^k)=\max_{a\in\mathcal{A}}Q_h^k(s_h^k,a)\mathrel{\overset{(i)}{=} }Q_h^k(s_h^k,a_h^k) V h k ( s h k ) = a ∈ A max Q h k ( s h k , a ) = ( i ) Q h k ( s h k , a h k ) ≤ r h ( s h k , a h k ) + P ^ s h k , a h k , h V h + 1 k + b h k ( s h k , a h k ) \leq r_h(s_h^k,a_h^k)+\widehat{P}_{s_h^k,a_h^k,h}V_{h+1}^k+b_h^k(s_h^k,a_h^k) ≤ r h ( s h k , a h k ) + P s h k , a h k , h V h + 1 k + b h k ( s h k , a h k ) = ⟨ P ^ s h k , a h k , h − P s h k , a h k , h , V h + 1 k ⟩ + b h k ( s h k , a h k ) + r h ( s h k , a h k ) + ⟨ P s h k , a h k , h − e s h + 1 k , V h + 1 k ⟩ + V h + 1 k ( s h + 1 k ) , =\left\langle\widehat{P}_{s_h^k,a_h^k,h}-P_{s_h^k,a_h^k,h},V_{h+1}^k\right\rangle+b_h^k(s_h^k,a_h^k)+r_h(s_h^k,a_h^k)+\left\langle P_{s_h^k,a_h^k,h}-e_{s_{h+1}^k},V_{h+1}^k\right\rangle+V_{h+1}^k(s_{h+1}^k), = ⟨ P s h k , a h k , h − P s h k , a h k , h , V h + 1 k ⟩ + b h k ( s h k , a h k ) + r h ( s h k , a h k ) + ⟨ P s h k , a h k , h − e s h + 1 k , V h + 1 k ⟩ + V h + 1 k ( s h + 1 k ) , where (i) is valid due to the greedy rule for selecting a h k a_h^k a h k . Importantly, this recurrence relation links the value function estimate at step h h h (i.e., V h k ( s h k ) V_h^k(s_h^k) V h k ( s h k ) ) to its counterpart at step h + 1 h+1 h + 1 (i.e., V h + 1 k ( s h + 1 k ) V_{h+1}^k(s_{h+1}^k) V h + 1 k ( s h + 1 k ) ). By unrolling this relation recursively from h = 1 h=1 h = 1 to H H H , we arrive at
V 1 k ( s 1 k ) ≤ ∑ h = 1 H { ⟨ P ^ s h k , a h k , h − P s h k , a h k , h , V h + 1 k ⟩ + b h k ( s h k , a h k ) + r h ( s h k , a h k ) + ⟨ P s h k , a h k , h − e s h + 1 k , V h + 1 k ⟩ } , V_1^k(s_1^k)\leq\sum_{h=1}^{H}\left\{\left\langle\widehat{P}_{s_h^k,a_h^k,h}-P_{s_h^k,a_h^k,h},V_{h+1}^k\right\rangle+b_h^k(s_h^k,a_h^k)+r_h(s_h^k,a_h^k)+\left\langle P_{s_h^k,a_h^k,h}-e_{s_{h+1}^k},V_{h+1}^k\right\rangle\right\}, V 1 k ( s 1 k ) ≤ h = 1 ∑ H { ⟨ P s h k , a h k , h − P s h k , a h k , h , V h + 1 k ⟩ + b h k ( s h k , a h k ) + r h ( s h k , a h k ) + ⟨ P s h k , a h k , h − e s h + 1 k , V h + 1 k ⟩ } , where the recursive expansion terminates at step H H H since V H + 1 k = 0 V_{H+1}^k=0 V H + 1 k = 0 .
Substituting the regret bound from Lemma 5.6 into the decomposition (389) reveals that, with probability at least 1 − 2 S A H K δ ′ 1-2SAHK\delta' 1 − 2 S A H K δ ′ ,
(5.33) ≤ ∑ k = 1 K ∑ h = 1 H ⟨ P ^ s h k , a h k , h k − P s h k , a h k , h , V h + 1 k ⟩ + ∑ k = 1 K ∑ h = 1 H b h k ( s h k , a h k ) + ∑ k = 1 K ∑ h = 1 H ⟨ P s h k , a h k , h − e s h + 1 k , V h + 1 k ⟩ + ∑ k = 1 K ( ∑ h = 1 H r h ( s h k , a h k ) − V 1 π k ( s 1 k ) ) . (390) \begin{aligned}
\text{(5.33)}\le{}&\sum_{k=1}^{K}\sum_{h=1}^{H}\left\langle\widehat{P}^{k}_{s_h^k,a_h^k,h}-P_{s_h^k,a_h^k,h},V_{h+1}^k\right\rangle+\sum_{k=1}^{K}\sum_{h=1}^{H}b_h^k(s_h^k,a_h^k)\\
&+\sum_{k=1}^{K}\sum_{h=1}^{H}\left\langle P_{s_h^k,a_h^k,h}-e_{s_{h+1}^k},V_{h+1}^k\right\rangle+\sum_{k=1}^{K}\left(\sum_{h=1}^{H}r_h(s_h^k,a_h^k)-V_1^{\pi^k}(s_1^k)\right). \tag*{(390)}
\end{aligned} (5.33) ≤ k = 1 ∑ K h = 1 ∑ H ⟨ P s h k , a h k , h k − P s h k , a h k , h , V h + 1 k ⟩ + k = 1 ∑ K h = 1 ∑ H b h k ( s h k , a h k ) + k = 1 ∑ K h = 1 ∑ H ⟨ P s h k , a h k , h − e s h + 1 k , V h + 1 k ⟩ + k = 1 ∑ K ( h = 1 ∑ H r h ( s h k , a h k ) − V 1 π k ( s 1 k ) ) . (390)
Therefore, it remains to bound each of the four terms { T i } i = 1 4 \{T_i\}_{i=1}^{4} { T i } i = 1 4 . Before proceeding, we briefly comment on each term.
The term T 3 T_3 T 3 can be controlled relatively directly using martingale concentration inequalities, since s h + 1 k s_{h+1}^k s h + 1 k is a newly collected sample that is conditionally independent of V h + 1 k V_{h+1}^k V h + 1 k given what has happened prior to episode k k k . For similar reasons, the term T 4 T_4 T 4 can also be handled via martingale concentration arguments.
The most challenging term is T 1 T_1 T 1 , due to the intricate statistical dependence between P ^ s h k , a h k , h k \widehat{P}_{s_h^k,a_h^k,h}^k P s h k , a h k , h k and V h + 1 k V_{h+1}^k V h + 1 k . Addressing this dependence requires considerably more delicate analysis. In fact, as we shall see, the difficulty of handling this term in the moderate-to-small K K K regime also motivates the design of the MVP algorithm to be introduced in § 8.8 .
In what follows, we bound these terms separately. As will become clear shortly, the resulting bounds for these terms are inherently intertwined, ultimately leading to a system of interdependent inequalities that are solved jointly in the final step. To facilitate the analysis, we also introduce a few quantities concerning cumulative variance or cumulative second moments across time:
T 5 : = ∑ k = 1 K ∑ h = 1 H Var P ^ s h k , a h k , h k ( V h + 1 k ) , (391) T_5:=\sum_{k=1}^{K}\sum_{h=1}^{H}\operatorname{Var}_{\widehat{P}_{s_h^k,a_h^k,h}^k}\left(V_{h+1}^k\right), \tag*{(391)} T 5 := k = 1 ∑ K h = 1 ∑ H Var P s h k , a h k , h k ( V h + 1 k ) , (391)
T 6 u c b : = ∑ k = 1 K ∑ h = 1 H Var P s h k , a h k , h ( V h + 1 k ) , (392) T_6^{\mathrm{ucb}}:=\sum_{k=1}^{K}\sum_{h=1}^{H}\operatorname{Var}_{P_{s_h^k,a_h^k,h}}\left(V_{h+1}^k\right), \tag*{(392)} T 6 ucb := k = 1 ∑ K h = 1 ∑ H Var P s h k , a h k , h ( V h + 1 k ) , (392)
T 6 o p t : = ∑ k = 1 K ∑ h = 1 H Var P s h k , a h k , h ( V h + 1 ⋆ ) , (393) T_6^{\mathrm{opt}}:=\sum_{k=1}^{K}\sum_{h=1}^{H}\operatorname{Var}_{P_{s_h^k,a_h^k,h}}\left(V_{h+1}^{\star}\right), \tag*{(393)} T 6 opt := k = 1 ∑ K h = 1 ∑ H Var P s h k , a h k , h ( V h + 1 ⋆ ) , (393)
T 6 l o w : = ∑ k = 1 K ∑ h = 1 H Var P s h k , a h k , h ( V h + 1 π k ) , (394) T_6^{\mathrm{low}}:=\sum_{k=1}^{K}\sum_{h=1}^{H}\operatorname{Var}_{P_{s_h^k,a_h^k,h}}\left(V_{h+1}^{\pi^k}\right), \tag*{(394)} T 6 low := k = 1 ∑ K h = 1 ∑ H Var P s h k , a h k , h ( V h + 1 π k ) , (394)
T 6 d i f f : = ∑ k = 1 K ∑ h = 1 H ⟨ P s h k , a h k , h , ( V h + 1 k − V h + 1 π k ) 2 ⟩ . (395) T_6^{\mathrm{diff}}:=\sum_{k=1}^{K}\sum_{h=1}^{H}\left\langle P_{s_h^k,a_h^k,h},\left(V_{h+1}^k-V_{h+1}^{\pi^k}\right)^2\right\rangle. \tag*{(395)} T 6 diff := k = 1 ∑ K h = 1 ∑ H ⟨ P s h k , a h k , h , ( V h + 1 k − V h + 1 π k ) 2 ⟩ . (395)
Here, T 5 T_5 T 5 aggregates empirical variance terms, while the remaining quantities correspond to cumulative true variances or second moments associated with objects related to value functions.
Step 3: bounding the term T 1 T_1 T 1 in (390) . We begin with the term T 1 T_1 T 1 , which is particularly challenging to cope with due to the complicated statistical dependency between P ^ s h k , a h k , h k \widehat{P}^{k}_{s_h^k,a_h^k,h} P s h k , a h k , h k and V h + 1 k V^k_{h+1} V h + 1 k .
To see this, note that P ^ s , a , h k \widehat{P}^{k}_{s,a,h} P s , a , h k is constructed from all previous samples associated with ( s , a , h ) (s,a,h) ( s , a , h ) , while V h + 1 k V^k_{h+1} V h + 1 k is itself computed based on past samples. Consequently, these two objects exhibit complicated statistical dependence.
One common strategy for circumventing this technical difficulty is to replace V h + 1 k V^k_{h+1} V h + 1 k with the optimal value function V h + 1 ⋆ V^\star_{h+1} V h + 1 ⋆ , the latter of which is fixed and therefore statistically independent of the sampling process. This strategy has been adopted in a large body of existing analyses of model-based online RL algorithms (e.g., Azar et al. (2017), Dann et al. (2017), Zanette and Brunskill (2019), and Zhang et al. (2021e)). To be precise, let us decompose
T 1 = ∑ k , h ⟨ P ^ s h k , a h k , h k − P s h k , a h k , h k , V h + 1 ⋆ ⟩ ⏟ = : W 1 + ∑ k , h ⟨ P ^ s h k , a h k , h k − P s h k , a h k , h k , V h + 1 k − V h + 1 ⋆ ⟩ ⏟ = : W 2 . (396) T_1 = \underbrace{\sum_{k,h}\left\langle\widehat{P}^{k}_{s_h^k,a_h^k,h}-P^{k}_{s_h^k,a_h^k,h},V^\star_{h+1}\right\rangle}_{=:W_1}+\underbrace{\sum_{k,h}\left\langle\widehat{P}^{k}_{s_h^k,a_h^k,h}-P^{k}_{s_h^k,a_h^k,h},V^k_{h+1}-V^\star_{h+1}\right\rangle}_{=:W_2}. \tag*{(396)} T 1 = =: W 1 k , h ∑ ⟨ P s h k , a h k , h k − P s h k , a h k , h k , V h + 1 ⋆ ⟩ + =: W 2 k , h ∑ ⟨ P s h k , a h k , h k − P s h k , a h k , h k , V h + 1 k − V h + 1 ⋆ ⟩ . (396)
The rationale behind the decomposition (396) is two-fold: the first term W 1 W_1 W 1 can be conveniently controlled via concentration inequalities given that V h + 1 ⋆ V^\star_{h+1} V h + 1 ⋆ is fixed, whereas the second term W 2 W_2 W 2 acts as a higher-order residual that would vanish asymptotically as the value estimate V h + 1 k V^k_{h+1} V h + 1 k converges to V h + 1 ⋆ V^\star_{h+1} V h + 1 ⋆ for large enough k k k .
The lemma below develops upper bounds that connect W 1 W_1 W 1 and W 2 W_2 W 2 with cumulative variances. The proof is deferred to § 8.15.2 .
Lemma 5.7. Recall the definition of W 1 W_1 W 1 and W 2 W_2 W 2 in (396) , and set
W 3 : = ∑ k = 1 K ∑ h = 1 H ⟨ P ^ s h k , a h k , h k − P s h k , a h k , h k , ( V h + 1 k ) 2 ⟩ . (397) W_3 := \sum_{k=1}^{K}\sum_{h=1}^{H}\left\langle\widehat{P}^{k}_{s_h^k,a_h^k,h}-P^{k}_{s_h^k,a_h^k,h},(V^k_{h+1})^2\right\rangle. \tag*{(397)} W 3 := k = 1 ∑ K h = 1 ∑ H ⟨ P s h k , a h k , h k − P s h k , a h k , h k , ( V h + 1 k ) 2 ⟩ . (397) With probability greater than 1 − 12 S 2 A H K δ ′ 1-12S^2AHK\delta' 1 − 12 S 2 A H K δ ′ , one has
W 1 ≤ 1 8 S A H T 6 l o w ι b + 1 8 S A H T 6 d i f f ι b + 1 8 S A H 2 ι b , W_1 \le\frac{1}{8}\sqrt{SAHT_6^{\mathrm{low}}\iota_b}+\frac{1}{8}\sqrt{SAHT_6^{\mathrm{diff}}\iota_b}+\frac{1}{8}SAH^2\iota_b, W 1 ≤ 8 1 S A H T 6 low ι b + 8 1 S A H T 6 diff ι b + 8 1 S A H 2 ι b , W 2 ≤ 1 8 S 2 A H T 6 d i f f ι b + 1 8 S 2 A H 2 ι b , W_2 \le\frac{1}{8}\sqrt{S^2AHT_6^{\mathrm{diff}}\iota_b}+\frac{1}{8}S^2AH^2\iota_b, W 2 ≤ 8 1 S 2 A H T 6 diff ι b + 8 1 S 2 A H 2 ι b , W 3 ≤ 1 8 S A H 3 T 6 l o w ι b + 1 8 S 2 A H 3 T 6 d i f f ι b + 1 8 S 2 A H 3 ι b , W_3 \le\frac{1}{8}\sqrt{SAH^3T_6^{\mathrm{low}}\iota_b}+\frac{1}{8}\sqrt{S^2AH^3T_6^{\mathrm{diff}}\iota_b}+\frac{1}{8}S^2AH^3\iota_b, W 3 ≤ 8 1 S A H 3 T 6 low ι b + 8 1 S 2 A H 3 T 6 diff ι b + 8 1 S 2 A H 3 ι b , where T 6 l o w T_6^{\mathrm{low}} T 6 low and T 6 d i f f T_6^{\mathrm{diff}} T 6 diff are defined in (5.35).
Remark 5.6. While the primary goal of this step is to bound W 1 W_1 W 1 and W 2 W_2 W 2 , the same proof techniques can be readily adapted to upper bound W 3 W_3 W 3 as well, which will appear in the subsequent analysis.
Remark 5.7. A careful reader may notice that, in addition to the difference in the variance terms, the bound on W 2 W_2 W 2 contains an extra factor in S S S compared to the bound on W 1 W_1 W 1 . This discrepancy arises because, in the analysis of W 1 W_1 W 1 , the value function V h + 1 ⋆ V^\star_{h+1} V h + 1 ⋆ is fixed and statistically independent of the data, which substantially simplifies the argument. In contrast, the term W 2 W_2 W 2 involves the difference V h + 1 k − V h + 1 ⋆ V^k_{h+1}-V^\star_{h+1} V h + 1 k − V h + 1 ⋆ , which exhibits intricate statistical dependence on the data and consequently incurs an additional factor of S S S .
Combining Lemma 5.7 with (396) , we reach
T 1 = W 1 + W 2 ≤ 1 4 S A H T 6 l o w ι b + 1 4 S 2 A H T 6 d i f f ι b + 1 4 S 2 A H 2 ι b = : T 1 u b . (398) T_1=W_1+W_2\le\frac{1}{4}\sqrt{SAHT_6^{\mathrm{low}}\iota_b}+\frac{1}{4}\sqrt{S^2AHT_6^{\mathrm{diff}}\iota_b}+\frac{1}{4}S^2AH^2\iota_b=:T_1^{\mathrm{ub}}. \tag*{(398)} T 1 = W 1 + W 2 ≤ 4 1 S A H T 6 low ι b + 4 1 S 2 A H T 6 diff ι b + 4 1 S 2 A H 2 ι b =: T 1 ub . (398)
with probability greater than 1 − 12 S 2 A H K δ ′ 1-12S^2AHK\delta' 1 − 12 S 2 A H K δ ′ . It is also worth noting that the variance terms T 6 l o w T_6^{\mathrm{low}} T 6 low and T 6 d i f f T_6^{\mathrm{diff}} T 6 diff remain random quantities and, therefore, require additional arguments to control, which we shall accomplish in the ensuing steps.
Step 4: bounding the terms T 2 T_2 T 2 , T 3 T_3 T 3 and T 4 T_4 T 4 in (390) . This step is dedicated to bounding the terms T 2 T_2 T 2 , T 3 T_3 T 3 and T 4 T_4 T 4 arising in the regret decomposition (390) , as formalized in the lemma below.
Lemma 5.8. With probability exceeding 1 − 4 ( log 2 ( K H ) ) δ ′ 1-4(\log_2(KH))\delta' 1 − 4 ( log 2 ( K H )) δ ′ , one has
T 2 ≤ 1 2 S A H T 5 ι b + 1 2 S A H 2 ι b , (399) T_2 \leq\frac{1}{2}\sqrt{SAHT_5\iota_b}+\frac{1}{2}SAH^2\iota_b, \tag*{(399)} T 2 ≤ 2 1 S A H T 5 ι b + 2 1 S A H 2 ι b , (399) ∣ T 3 ∣ ≤ 8 T 6 u c b log 1 δ ′ + 5 H log 1 δ ′ ≤ 4 T 6 l o w log 1 δ ′ + 4 T 6 d i f f log 1 δ ′ + 5 H log 1 δ ′ , (400) \begin{aligned}
|T_3| &\leq\sqrt{8T_6^{\mathrm{ucb}}\log\frac{1}{\delta'}}+5H\log\frac{1}{\delta'}\\
&\leq4\sqrt{T_6^{\mathrm{low}}\log\frac{1}{\delta'}}+4\sqrt{T_6^{\mathrm{diff}}\log\frac{1}{\delta'}}+5H\log\frac{1}{\delta'}, \tag*{(400)}
\end{aligned} ∣ T 3 ∣ ≤ 8 T 6 ucb log δ ′ 1 + 5 H log δ ′ 1 ≤ 4 T 6 low log δ ′ 1 + 4 T 6 diff log δ ′ 1 + 5 H log δ ′ 1 , (400) ∣ T 4 ∣ ≤ 2 K H 2 log 1 δ ′ , (401) |T_4| \leq\sqrt{2KH^2\log\frac{1}{\delta'}}, \tag*{(401)} ∣ T 4 ∣ ≤ 2 K H 2 log δ ′ 1 , (401) provided that ι b ≥ 8 max { c 1 2 , c 2 } ( log 2 K ) log 1 δ ′ \iota_b \geq8\max\{c_1^2,c_2\}(\log_2 K)\log\frac{1}{\delta'} ι b ≥ 8 max { c 1 2 , c 2 } ( log 2 K ) log δ ′ 1 . Here, T 5 T_5 T 5 , T 6 u c b T_6^{\mathrm{ucb}} T 6 ucb , T 6 l o w T_6^{\mathrm{low}} T 6 low and T 6 d i f f T_6^{\mathrm{diff}} T 6 diff are defined in (5.35).
At a high level, T 3 T_3 T 3 and T 4 T_4 T 4 are sums of zero-mean fluctuations and are bounded via martingale concentration inequalities, whereas the bound on T 2 T_2 T 2 follows from the Cauchy-Schwarz inequality alongside basic properties of the visitation counts. The proof is deferred to Section 5.7.3.
Step 5: combining the bounds on { T i } i = 1 4 \{T_i\}_{i=1}^4 { T i } i = 1 4 . Taking the preceding bounds (398) and (5.39) together with (390) , we obtain
∑ k = 1 K ( V 1 k ( s 1 k ) − V 1 π k ( s 1 k ) ) ≤ T 1 + T 2 + T 3 + T 4 ≤ T 1 u b + T 2 + 4 T 6 l o w log 1 δ ′ + 4 T 6 d i f f log 1 δ ′ + 7 K H 2 log 1 δ ′ ≤ T 0 u b . (402) \begin{aligned}
\sum_{k=1}^{K}\left(V_1^k(s_1^k)-V_1^{\pi^k}(s_1^k)\right)
&\leq T_1+T_2+T_3+T_4\\
&\leq T_1^{\mathrm{ub}}+T_2+4\sqrt{T_6^{\mathrm{low}}\log\frac{1}{\delta'}}+4\sqrt{T_6^{\mathrm{diff}}\log\frac{1}{\delta'}}+7\sqrt{KH^2}\log\frac{1}{\delta'}\\
&\leq T_0^{\mathrm{ub}}. \tag*{(402)}
\end{aligned} k = 1 ∑ K ( V 1 k ( s 1 k ) − V 1 π k ( s 1 k ) ) ≤ T 1 + T 2 + T 3 + T 4 ≤ T 1 ub + T 2 + 4 T 6 low log δ ′ 1 + 4 T 6 diff log δ ′ 1 + 7 K H 2 log δ ′ 1 ≤ T 0 ub . (402)
with probability at least 1 − 16 S 2 A H K δ ′ 1-16S^2AHK\delta' 1 − 16 S 2 A H K δ ′ , where we define
T 0 u b : = 3 8 S A H T 6 l o w ι b + 3 8 S 2 A H T 6 d i f f ι b + 1 4 S 2 A H 2 ι b + 7 K H 2 log 1 δ ′ + T 2 . (403) T_0^{\mathrm{ub}}:=\frac{3}{8}\sqrt{SAHT_6^{\mathrm{low}}\iota_b}+\frac{3}{8}\sqrt{S^2AHT_6^{\mathrm{diff}}\iota_b}+\frac{1}{4}S^2AH^2\iota_b+7\sqrt{KH^2}\log\frac{1}{\delta'}+T_2. \tag*{(403)} T 0 ub := 8 3 S A H T 6 low ι b + 8 3 S 2 A H T 6 diff ι b + 4 1 S 2 A H 2 ι b + 7 K H 2 log δ ′ 1 + T 2 . (403)
Before proceeding, we note that the same arguments used to bound ∑ k = 1 K ( V h k ( s h k ) − V h π k ( s h k ) ) \sum_{k=1}^{K}(V_h^k(s_h^k)-V_h^{\pi^k}(s_h^k)) ∑ k = 1 K ( V h k ( s h k ) − V h π k ( s h k )) extend directly to ∑ k = 1 K ( V h k ( s h k ) − V h π k ( s h k ) ) \sum_{k=1}^{K}(V_h^k(s_h^k)-V_h^{\pi^k}(s_h^k)) ∑ k = 1 K ( V h k ( s h k ) − V h π k ( s h k )) for any 1 < h ≤ H 1<h\leq H 1 < h ≤ H . Combining this observation with a union bound yields that, with probability at least 1 − 16 S 2 A H 2 K δ ′ 1-16S^2AH^2K\delta' 1 − 16 S 2 A H 2 K δ ′ ,
∑ k = 1 K ( V h k ( s h k ) − V h π k ( s h k ) ) ≤ T 0 u b (404) \sum_{k=1}^{K}\left(V_h^k(s_h^k)-V_h^{\pi^k}(s_h^k)\right)\leq T_0^{\mathrm{ub}} \tag*{(404)} k = 1 ∑ K ( V h k ( s h k ) − V h π k ( s h k ) ) ≤ T 0 ub (404)
holds simultaneously for all 1 ≤ h ≤ H 1\leq h\leq H 1 ≤ h ≤ H , with T 0 u b T_0^{\mathrm{ub}} T 0 ub defined in (403) .
Step 6: bounding the second moments in (5.35). Thus far, the bounds on T 1 T_1 T 1 , T 2 T_2 T 2 and T 3 T_3 T 3 (see Lemmas 5.7 and 5.8 ) — and hence the bound on T 0 u b T_0^{\mathrm{ub}} T 0 ub — depend explicitly on the cumulative variances T 5 T_5 T 5 , T 6 l o w T_6^{\mathrm{low}} T 6 low and T 6 d i f f T_6^{\mathrm{diff}} T 6 diff (cf. (5.35)). To obtain the desirable regret bound, it remains to derive asymptotically tight upper bounds for these aggregate second moments. This is accomplished via the following lemma, whose proof is provided in § 8.15.4 .
Lemma 5.9. With probability at least 1 − 14 S 2 A H 2 K δ ′ 1-14S^2AH^2K\delta' 1 − 14 S 2 A H 2 K δ ′ ,
T 5 ≤ W 3 + 2 H T 2 + 2 K H 2 + 32 H 2 T 6 u b log 1 δ ′ + 5 H 2 log 1 δ ′ , (405) T_5 \le W_3 + 2HT_2 + 2KH^2 + \sqrt{32H^2T_6^{\mathrm{ub}}\log\frac{1}{\delta'}} + 5H^2\log\frac{1}{\delta'}, \tag*{(405)} T 5 ≤ W 3 + 2 H T 2 + 2 K H 2 + 32 H 2 T 6 ub log δ ′ 1 + 5 H 2 log δ ′ 1 , (405) T 6 l o w ≤ K H 2 + 2 H 6 K log 1 δ ′ , (406) T_6^{\mathrm{low}} \le KH^2 + \sqrt{2H^6K\log\frac{1}{\delta'}}, \tag*{(406)} T 6 low ≤ K H 2 + 2 H 6 K log δ ′ 1 , (406) T 6 d i f f ≤ 2 H 2 T 0 u b + 18 H 2 log 1 δ ′ , (407) T_6^{\mathrm{diff}} \le2H^2T_0^{\mathrm{ub}} + 18H^2\log\frac{1}{\delta'}, \tag*{(407)} T 6 diff ≤ 2 H 2 T 0 ub + 18 H 2 log δ ′ 1 , (407) where W 3 W_3 W 3 and T 0 u b T_0^{\mathrm{ub}} T 0 ub are defined in (397) and (5.40), respectively. In addition, under the condition (387) , one further has
T 6 l o w ≤ 2 K H 2 , (408) T_6^{\mathrm{low}} \le2KH^2, \tag*{(408)} T 6 low ≤ 2 K H 2 , (408) T 5 ≤ 6 K H 2 + 1 2 S 2 A H 3 T 6 d i f f ι b + 4 S 2 A H 3 ι b (409) T_5 \le6KH^2 + \frac{1}{2}\sqrt{S^2AH^3T_6^{\mathrm{diff}}\iota_b + 4S^2AH^3\iota_b} \tag*{(409)} T 5 ≤ 6 K H 2 + 2 1 S 2 A H 3 T 6 diff ι b + 4 S 2 A H 3 ι b (409) with probability at least 1 − 30 S 2 A H 2 K δ ′ 1-30S^2AH^2K\delta' 1 − 30 S 2 A H 2 K δ ′ .
Remark 5.8. Asymptotically (as K → ∞ K \to\infty K → ∞ ), it is natural to expect these variance quantities to satisfy T 5 ≈ T 6 l o w ≈ T 6 o p t ≈ T 6 d i f f ≲ K H 2 T_5 \approx T_6^{\mathrm{low}} \approx T_6^{\mathrm{opt}} \approx T_6^{\mathrm{diff}} \lesssim KH^2 T 5 ≈ T 6 low ≈ T 6 opt ≈ T 6 diff ≲ K H 2 , where the last inequality follows from (408) (thanks to the Markov structure of the cumulative variance). Importantly, both establishing and exploiting the bound T 6 l o w ≲ K H 2 T_6^{\mathrm{low}} \lesssim KH^2 T 6 low ≲ K H 2 are crucial for obtaining the asymptotically optimal regret guarantees.
Step 7: putting all pieces together. We now combine the preceding (interdependent) bounds to derive a clean upper bound on the regret.
First, substituting (409) into (399) in Lemma 5.8 yields
T 2 ≤ 1 2 S A H T 5 ι b + 1 2 S A H 2 ι b ≤ 1 2 6 S A H 3 K ι b + 1 2 S 2 A H 2 ( 1 2 A H T 6 d i f f ι b + 4 S A H 2 ι b ) ι b + 1 2 S A H 2 ι b ≤ 1 2 6 S A H 3 K ι b + 1 4 S 2 A H 2 ι b + 1 8 A H T 6 d i f f ι b + S A H 2 ι b + 1 2 S A H 2 ι b ≤ 1 2 6 S A H 3 K ι b + 1 8 A H T 6 d i f f ι b + 2 S 2 A H 2 ι b , \begin{aligned}
T_2 &\le\frac{1}{2}\sqrt{SAHT_5\iota_b}+\frac{1}{2}SAH^2\iota_b \\
&\le\frac{1}{2}\sqrt{6SAH^3K\iota_b}+\frac{1}{2}\sqrt{S^2AH^2\left(\frac{1}{2}\sqrt{AHT_6^{\mathrm{diff}}\iota_b+4SAH^2\iota_b}\right)\iota_b}+\frac{1}{2}SAH^2\iota_b \\
&\le\frac{1}{2}\sqrt{6SAH^3K\iota_b}+\frac{1}{4}S^2AH^2\iota_b+\frac{1}{8}\sqrt{AHT_6^{\mathrm{diff}}\iota_b}+SAH^2\iota_b+\frac{1}{2}SAH^2\iota_b \\
&\le\frac{1}{2}\sqrt{6SAH^3K\iota_b}+\frac{1}{8}\sqrt{AHT_6^{\mathrm{diff}}\iota_b}+2S^2AH^2\iota_b,
\end{aligned} T 2 ≤ 2 1 S A H T 5 ι b + 2 1 S A H 2 ι b ≤ 2 1 6 S A H 3 K ι b + 2 1 S 2 A H 2 ( 2 1 A H T 6 diff ι b + 4 S A H 2 ι b ) ι b + 2 1 S A H 2 ι b ≤ 2 1 6 S A H 3 K ι b + 4 1 S 2 A H 2 ι b + 8 1 A H T 6 diff ι b + S A H 2 ι b + 2 1 S A H 2 ι b ≤ 2 1 6 S A H 3 K ι b + 8 1 A H T 6 diff ι b + 2 S 2 A H 2 ι b ,
where the second inequality follows from the elementary fact a + b ≤ a + b \sqrt{a+b}\le\sqrt{a}+\sqrt{b} a + b ≤ a + b , and the penultimate inequality is due to the AM-GM inequality. Substituting this into the definition of T 0 u b T_0^{\mathrm{ub}} T 0 ub (cf. (5.40)) and combining terms, we reach
T 0 u b ≤ 3 8 S A H T 6 l o w ι b + S 2 A H T 6 d i f f ι b + 9 S A H 3 K ι b + 2 S 2 A H 2 ι b T_0^{\mathrm{ub}} \le\frac{3}{8}\sqrt{SAHT_6^{\mathrm{low}}\iota_b}+\sqrt{S^2AHT_6^{\mathrm{diff}}\iota_b}+9\sqrt{SAH^3K\iota_b}+2S^2AH^2\iota_b T 0 ub ≤ 8 3 S A H T 6 low ι b + S 2 A H T 6 diff ι b + 9 S A H 3 K ι b + 2 S 2 A H 2 ι b
≤ 10 S A H 3 K ι b + S 2 A H T 6 d i f f ι b + 2 S 2 A H 2 ι b . (410) \le10\sqrt{SAH^3K\iota_b}+\sqrt{S^2AHT_6^{\mathrm{diff}}\iota_b}+2S^2AH^2\iota_b. \tag*{(410)} ≤ 10 S A H 3 K ι b + S 2 A H T 6 diff ι b + 2 S 2 A H 2 ι b . (410)
where the last inequality results from (408) .
To conclude the proof, observe that the AM-GM inequality implies
S 2 A H T 6 d i f f b ≤ 1 4 H 2 T 6 d i f f b + S 2 A H 3 ℓ b ≤ 1 2 T 0 u b + 6 S 2 A H 3 ℓ b , \sqrt{S^2 A H T_6^{\mathrm{diff}}}_{\mathrm{b}} \le\frac{1}{4H^2}T_6^{\mathrm{diff}}{}_{\mathrm{b}} + S^2 A H^3 \ell_{\mathrm{b}} \le\frac{1}{2}T_0^{\mathrm{ub}} + 6S^2 A H^3 \ell_{\mathrm{b}}, S 2 A H T 6 diff b ≤ 4 H 2 1 T 6 diff b + S 2 A H 3 ℓ b ≤ 2 1 T 0 ub + 6 S 2 A H 3 ℓ b ,
where the last inequality follows from (407) in Lemma 5.9. Combining this bound with (410) and rearranging terms, we can readily conclude that
T 0 u b ≲ S A H 3 K ℓ b + S 2 A H 3 ℓ b T_0^{\mathrm{ub}} \lesssim\sqrt{S A H^3 K}\ell_{\mathrm{b}} + S^2 A H^3 \ell_{\mathrm{b}} T 0 ub ≲ S A H 3 K ℓ b + S 2 A H 3 ℓ b
with probability at least 1 − 50 S 2 A H 2 K δ ′ 1 - 50S^2AH^2K\delta' 1 − 50 S 2 A H 2 K δ ′ . Together with the choice of δ ′ \delta' δ ′ in (386) , this completes the proof of Theorem 5.2.
An improved model-based online RL algorithm: MVP
As noted in Section 5.3.3, the UCB-VI algorithm achieves asymptotically optimal regret only after an excessively large burn-in phase characterized by (378) . How to overcome this limitation and obtain regret guarantees that are optimal for all sample sizes has therefore become a fundamentally important research direction, drawing considerable attention in the years following the introduction of UCB-VI. This objective was recently accomplished by Zhang et al. (2024c) and Zhang et al. (2025e), which established that a variant of the optimistic model-based algorithms — called Monotonic Value Propagation (MVP), originally proposed by Zhang et al. (2021e) — attains minimax-optimal regret (up to logarithmic factors) for any number K K K of sample trajectories.
Revisiting UCB-VI in the sample-starved regime
At a high level, the MVP algorithm can be viewed as a refinement of UCB-VI in Algorithm 5.2, with only a handful of algorithmic modifications. As we shall see, these seemingly modest algorithmic changes lead to substantially stronger regret guarantees in the sample-starved regime. To motivate these algorithmic ideas, we first revisit the main bottleneck in the analysis of UCB-VI that limits its theoretical performance in the most sample-constrained settings.
As discussed in Step 2 of Section 5.3.5, the regret of UCB-VI admits a decomposition of the form (390) , consisting of four terms { T i } 1 ≤ i ≤ 4 \{T_i\}_{1\leq i\leq4} { T i } 1 ≤ i ≤ 4 . As already noted there, the terms T 3 T_3 T 3 and T 4 T_4 T 4 can be readily handled using standard martingale concentration inequalities, while T 2 T_2 T 2 can likewise be controlled through the variance-based inequalities developed there. The main obstacle is the term
T 1 = ∑ k = 1 K ∑ h = 1 H ⟨ P ^ s h k , a h k , h k − P s h k , a h k , h k , V h + 1 k ⟩ , (411) T_1 = \sum_{k=1}^{K}\sum_{h=1}^{H}\left\langle\widehat{P}^{k}_{s_h^k,a_h^k,h} - P^{k}_{s_h^k,a_h^k,h}, V^{k}_{h+1}\right\rangle, \tag*{(411)} T 1 = k = 1 ∑ K h = 1 ∑ H ⟨ P s h k , a h k , h k − P s h k , a h k , h k , V h + 1 k ⟩ , (411)
whose analysis is complicated due to the intricate statistical dependence between the empirical model P ^ k \widehat{P}^{k} P k and the value estimate V h + 1 k V^{k}_{h+1} V h + 1 k ; indeed, both P ^ k \widehat{P}^{k} P k and V k V^{k} V k depend on the entire collection of samples gathered prior to episode k k k , precluding a direct application of standard concentration arguments. Below, we discuss two fundamentally different strategies for overcoming this obstacle.
Strategy 1: replacing V k V^k V k with V ⋆ V^\star V ⋆ . Thus far, the analysis of UCB-VI copes with T 1 T_1 T 1 by first replacing V h + 1 k V^k_{h+1} V h + 1 k in (411) with the optimal value function V h + 1 ⋆ V^\star_{h+1} V h + 1 ⋆ — thereby enabling the use of standard concentration techniques given that V h + 1 ⋆ V^\star_{h+1} V h + 1 ⋆ is fixed and statistically independent from samples — and then controlling the resulting approximation error induced by the discrepancy between V h + 1 k V_{h+1}^k V h + 1 k and V h + 1 ∗ V_{h+1}^* V h + 1 ∗ . Although this approach is asymptotically well justified — indeed, one anticipates V h + 1 k V_{h+1}^k V h + 1 k to converge to V h + 1 ∗ V_{h+1}^* V h + 1 ∗ as the number of episodes K → ∞ K \to\infty K → ∞ — the gap between V h + 1 k V_{h+1}^k V h + 1 k and V h + 1 ∗ V_{h+1}^* V h + 1 ∗ can remain substantial and non-negligible in the regime where K K K is only moderate or small. As a result, the induced approximation error can be difficult to control sharply in this regime, ultimately giving rise to the large burn-in cost appearing in Theorem 5.2.
Strategy 2: a covering-type argument. We now describe an alternative approach for analyzing this term T 1 T_1 T 1 , based on a covering-type argument. A brief sketch of this argument is provided below, which serves to motivate both the algorithmic design and analysis of MVP.
To keep the exposition simple and highlight the main ideas, we keep the argument below informal. Recall that N h k ( s , a ) N_h^k(s,a) N h k ( s , a ) denotes the visitation count of ( s , a , h ) (s,a,h) ( s , a , h ) prior to episode k k k . Intuitively, once the collection of visitation counts at step h h h (i.e., { N h k ( s , a ) } ( s , a , k ) ∈ S × A × [ K ] \{N_h^k(s,a)\}_{(s,a,k)\in\mathcal{S}\times\mathcal{A}\times[K]} { N h k ( s , a ) } ( s , a , k ) ∈ S × A × [ K ] ) are given, the empirical transition kernel P ^ s , a , h k \widehat{P}_{s,a,h}^k P s , a , h k and the value estimate V h + 1 k V_{h+1}^k V h + 1 k become conditionally independent; to see this, it suffices to note that P ^ s , a , h k \widehat{P}_{s,a,h}^k P s , a , h k depends only on the samples collected at step h h h , whereas V h + 1 k V_{h+1}^k V h + 1 k is computed solely from samples collected at later steps via the backward Bellman recursion (5.18). Now suppose, for the sake of intuition, that these visitation counts were fixed in advance and statistically independent of the empirical transition kernels { P ^ s , a , h k } \{\widehat{P}_{s,a,h}^k\} { P s , a , h k } (a heuristic assumption that is not technically accurate, but useful for an informal discussion). Under this heuristic viewpoint, standard concentration techniques could be applied directly to control
∑ k , h ( P ^ s h k , a h k , h k − P s h k , a h k , h k ) V h + 1 k . \sum_{k,h}\left(\widehat{P}_{s_h^k,a_h^k,h}^k-P_{s_h^k,a_h^k,h}^k\right)V_{h+1}^k. k , h ∑ ( P s h k , a h k , h k − P s h k , a h k , h k ) V h + 1 k .
The remaining step would be to remove the artificial independence assumption by taking a union bound over all possible realizations of N h k ( s , a ) N_h^k(s,a) N h k ( s , a ) . The main obstacle, however, is combinatorial: the number of such visitation-count configurations grows exponentially with K K K , making a naive union bound prohibitively costly and hence significantly weakening the resulting regret guarantee.
Nevertheless, this heuristic argument points to a potentially useful idea: if the effective number of admissible visitation-count configurations could be reduced — say, to a quantity polynomial in K K K — then the covering-based argument outlined above could become viable. This important observation motivates the design of MVP, which we introduce next.
Algorithmic ideas: doubling-based updates
We now introduce the MVP algorithm by singling out its key difference from the UCB–VI algorithm presented in Algorithm 5.2. A key distinguishing feature of MVP is its epoch-based update scheme for the empirical transition kernels, inspired by the doubling-trick framework adopted in Jaksch et al. (2010). More specifically, the learning process is partitioned into consecutive epochs according to a simple doubling rule: whenever there exists a tuple ( s , a , h ) (s,a,h) ( s , a , h ) whose visitation count reaches a power of two, the current epoch terminates. At that point, MVP reconstructs the empirical model, recomputes the UCB-based optimistic Q-functions and value functions based on the updated transition kernels, and then initiates a new epoch with the corresponding updated policy. Within each epoch, the sampling policy is determined by the optimistic Q-function estimates computed prior to the beginning of that epoch; consequently, the same policy is used to collect data throughout all episodes within the epoch.
Algorithm 5.3: The monotonic value propagation (MVP) algorithm for episodic online RL
input: number of episodes K K K , reward function r r r , confidence parameter δ \delta δ , constants c 1 , c 2 > 0 c_1,c_2>0 c 1 , c 2 > 0 .
initialize N h a l l ( s , a ) ← 0 N_h^{\mathrm{all}}(s,a)\leftarrow0 N h all ( s , a ) ← 0 , N h ( s , a , s ′ ) ← 0 N_h(s,a,s')\leftarrow0 N h ( s , a , s ′ ) ← 0 , N h ( s , a ) ← 0 N_h(s,a)\leftarrow0 N h ( s , a ) ← 0 , Q h u c b ( s , a ) ← H Q_h^{\mathrm{ucb}}(s,a)\leftarrow H Q h ucb ( s , a ) ← H , V h u c b ( s ) ← H V_h^{\mathrm{ucb}}(s)\leftarrow H V h ucb ( s ) ← H , V H + 1 u c b ( s ) ← 0 V_{H+1}^{\mathrm{ucb}}(s)\leftarrow0 V H + 1 ucb ( s ) ← 0 , P ^ s , a , h ( s ′ ) ← 1 / S \widehat{P}_{s,a,h}(s')\leftarrow1/S P s , a , h ( s ′ ) ← 1/ S for all ( s , a , s ′ , h ) ∈ S × A × S × [ H ] (s,a,s',h)\in\mathcal{S}\times\mathcal{A}\times\mathcal{S}\times[H] ( s , a , s ′ , h ) ∈ S × A × S × [ H ] .
for k = 1 , ⋯ , K k=1,\cdots,K k = 1 , ⋯ , K do /* policy update. */
update policy π u c b \pi^{\mathrm{ucb}} π ucb such that π h u c b ( s ) ← arg max a Q h u c b ( s , a ) \pi_h^{\mathrm{ucb}}(s)\leftarrow\arg\max_a Q_h^{\mathrm{ucb}}(s,a) π h ucb ( s ) ← arg max a Q h ucb ( s , a ) for all ( s , h ) ∈ S × [ H ] (s,h)\in\mathcal{S}\times[H] ( s , h ) ∈ S × [ H ] .
observe the initial state s 1 ∼ i n d . ρ s_1\mathrel{\mathop{\sim}^{\mathrm{ind.}}}\rho s 1 ∼ ind. ρ of a new episode.
for h = 1 , ⋯ , H h=1,\cdots,H h = 1 , ⋯ , H do /* sampling. */
take a h = π h u c b ( s h ) a_h=\pi_h^{\mathrm{ucb}}(s_h) a h = π h ucb ( s h ) , observe s h + 1 s_{h+1} s h + 1 ; set ( s , a , s ′ ) ← ( s h , a h , s h + 1 ) (s,a,s')\leftarrow(s_h,a_h,s_{h+1}) ( s , a , s ′ ) ← ( s h , a h , s h + 1 ) .
N h a l l ( s , a ) ← N h a l l ( s , a ) + 1 N_h^{\mathrm{all}}(s,a)\leftarrow N_h^{\mathrm{all}}(s,a)+1 N h all ( s , a ) ← N h all ( s , a ) + 1 ; N h ( s , a , s ′ ) ← N h ( s , a , s ′ ) + 1 N_h(s,a,s')\leftarrow N_h(s,a,s')+1 N h ( s , a , s ′ ) ← N h ( s , a , s ′ ) + 1 . /* updates using data of this epoch. */
if N h a l l ( s , a ) ∈ { 1 , 2 , ⋯ , 2 log 2 K } N_h^{\mathrm{all}}(s,a)\in\{1,2,\cdots,2^{\log_2 K}\} N h all ( s , a ) ∈ { 1 , 2 , ⋯ , 2 l o g 2 K } then
N h ( s , a ) ← ∑ s ~ N h ( s , a , s ~ ) N_h(s,a)\leftarrow\sum_{\widetilde{s}}N_h(s,a,\widetilde{s}) N h ( s , a ) ← ∑ s N h ( s , a , s ) . // # visits in this epoch.
P ^ s , a , h ( s ~ ) ← N h ( s , a , s ~ ) N h ( s , a ) \widehat{P}_{s,a,h}(\widetilde{s})\leftarrow\frac{N_h(s,a,\widetilde{s})}{N_h(s,a)} P s , a , h ( s ) ← N h ( s , a ) N h ( s , a , s ) for all s ~ ∈ S \widetilde{s}\in\mathcal{S} s ∈ S . // empirical model.
set triggered = true; N h ( s , a , s ~ ) ← 0 N_h(s,a,\widetilde{s})\leftarrow0 N h ( s , a , s ) ← 0 for all s ~ ∈ S \widetilde{s}\in\mathcal{S} s ∈ S . /* optimistic Q-estimation for this epoch. */
if triggered = true then
set triggered = false.
for h = H , ⋯ , 1 h=H,\cdots,1 h = H , ⋯ , 1 do
for ( s , a ) ∈ S × A (s,a)\in\mathcal{S}\times\mathcal{A} ( s , a ) ∈ S × A do
set the bonus b h ( s , a ) b_h(s,a) b h ( s , a ) according to (5.20); compute
Q h u c b ( s , a ) ← min { r h ( s , a ) + ⟨ P ^ s , a , h , V h + 1 u c b ⟩ + b h ( s , a ) , H } (412) Q_h^{\mathrm{ucb}}(s,a)\leftarrow\min\left\{r_h(s,a)+\langle\widehat{P}_{s,a,h},V_{h+1}^{\mathrm{ucb}}\rangle+b_h(s,a),H\right\} \tag*{(412)} Q h ucb ( s , a ) ← min { r h ( s , a ) + ⟨ P s , a , h , V h + 1 ucb ⟩ + b h ( s , a ) , H } (412)
V h u c b ( s ) ← max a Q h u c b ( s , a ) . (413) V_h^{\mathrm{ucb}}(s)\leftarrow\max_a Q_h^{\mathrm{ucb}}(s,a). \tag*{(413)} V h ucb ( s ) ← a max Q h ucb ( s , a ) . (413)
Clearly, this epoch-based paradigm differs from the original UCB-VI algorithm in its update frequency: whereas UCB-VI updates the empirical transition model and recomputes the Q-estimates and value estimates after every episode, the epoch-based paradigm in MVP performs these updates only at the end of each epoch. An important property ensured by the careful design of the doubling update rule is that: for each each ( s , a , h ) (s,a,h) ( s , a , h ) , the associated empirical transition probability is updated only when its visitation count equals a power of two, and therefore no more than O ( log 2 K ) O(\log_2 K) O ( log 2 K ) times over the entire learning process. As we shall see, this structural property plays a central role in substantially reducing the effective covering number required in the covering-based argument mentioned in § 8.8.1 . More specifically, it induces a modified version of the visitation-count sequence { N h k ( s , a ) } ( s , a , k ) ∈ S × A × [ K ] \{N_h^k(s,a)\}_{(s,a,k)\in\mathcal{S}\times\mathcal{A}\times[K]} { N h k ( s , a ) } ( s , a , k ) ∈ S × A × [ K ] , which tracks the empirical model currently in use, rather than the most recently collected samples. Owing to the epoch-based update scheme for the empirical models, the number of admissible configurations of this modified visitation-count sequence is dramatically smaller than that arising in the original UCB-VI algorithm.
The complete procedure of MVP is presented in Algorithm 5.3, which adopts the same variance-aware Bernstein-style exploration bonuses as those recommended in Algorithm 5.2. In terms of notation, we now let N h a l l ( s , a ) N_h^{\mathrm{all}}(s,a) N h all ( s , a ) indicate the total visitation count of ( s , a , h ) (s,a,h) ( s , a , h ) accumulated over all episodes up to the current one, while N h ( s , a ) N_h(s,a) N h ( s , a ) (resp. N h ( s , a , s ~ ) N_h(s,a,\widetilde{s}) N h ( s , a , s ) ) tracks the visitation count of ( s , a , h ) (s,a,h) ( s , a , h ) (resp. ( s , a , h , s ~ ) (s,a,h,\widetilde{s}) ( s , a , h , s ) ) within the current epoch. In addition, the flag triggered is used to indicate whether we have reached the end of the current epoch.
Remark 5.9. The name Monotonic Value Propagation , coined by Zhang et al. (2021e), stems from the fact that the exploration bonuses (5.20) guarantee a certain monotonicity property in value propagation. For instance, the function f ( p , v , n ) f(p,v,n) f ( p , v , n ) defined in (499) in § 8.15.1 — which resembles a key operation arising in certain optimistic Bellman recursions — is non-decreasing with respect to v v v , where v v v typically represents a vector of value estimates. Notably, this monotonicity property has already been built into the version of UCB-VI presented in Algorithm 5.2; this version differs in this respect from the original UCB-VI algorithm of Azar et al. (2017) (see also Remark 5.2).
Theoretical guarantees
The epoch-based design of MVP, in conjunction with the optimistic model-based framework, turns out to be sufficient to overcome the suboptimal regret guarantees of UCB-VI in the sample-limited regime. As stated in the following theorem, MVP achieves minimax-optimal regret uniformly over the entire range K ≥ 1 K \ge1 K ≥ 1 .
Theorem 5.10 (Zhang et al., 2024c; Zhang et al., 2025e). Consider any K ≥ 1 K \ge1 K ≥ 1 and any 0 < δ < 1 0 < \delta< 1 0 < δ < 1 . With probability at least 1 − δ 1-\delta 1 − δ , the MVP algorithm (see Algorithm 5.3) with Bernstein-style bonuses (5.20) achieves
Regret ( K ) ≤ O ~ ( min { S A H 3 K , H K } ) , (414) \operatorname{Regret}(K) \le\widetilde{O}\left(\min\left\{\sqrt{SAH^{3}K},HK\right\}\right), \tag*{(414)} Regret ( K ) ≤ O ( min { S A H 3 K , H K } ) , (414) provided that the constants c 1 , c 2 > 0 c_1,c_2 > 0 c 1 , c 2 > 0 in (5.20) are suitably chosen. Here, in the definition of Regret ( K ) \operatorname{Regret}(K) Regret ( K ) (cf. (349) ), π k \pi^k π k and s 1 k s_1^k s 1 k denote, respectively, the policy deployed and the initial state observed in episode k k k of Algorithm 5.3.
As we shall confirm shortly in § 8.12 (see Theorem 5.19 ), the above regret guarantee for MVP matches the fundamental minimax lower bound up to logarithmic factors, regardless of how large K K K is. In contrast to UCB-VI, this means that MVP essentially eliminates the burn-in phase required to attain statistically optimal performance, with only logarithmic factors remaining. By converting this regret guarantee into a PAC-style sample complexity bound (using arguments in § 8.1 ), we immediately obtain the following implication: for an arbitrary accuracy level ε ∈ ( 0 , H ] \varepsilon\in(0,H] ε ∈ ( 0 , H ] , the MVP algorithm is guaranteed to yield an ε \varepsilon ε -optimal policy with high probability using at most
O ~ ( S A H 3 ε 2 ) episodes , (415) \widetilde{O}\left(\frac{SAH^{3}}{\varepsilon^{2}}\right)\ \text{episodes}, \tag*{(415)} O ( ε 2 S A H 3 ) episodes , (415)
where the output policy can be drawn, for instance, uniformly at random from the policy sequence { π k } k = 1 K \{\pi^k\}_{k=1}^{K} { π k } k = 1 K . Equivalently, this corresponds to O ~ ( S A H 4 ε 2 ) \widetilde{O}\left(\frac{SAH^{4}}{\varepsilon^{2}}\right) O ( ε 2 S A H 4 ) sample transitions, since each sample trajectory has length H H H . It is noteworthy that this sample complexity is information-theoretically unimprovable even when one has access to an idealized generative model (cf. § 5.1 ).
Taken together, the above results further underscore the statistical optimality of model-based methods in yet another fundamental RL setting: online RL. To date, MVP remains the only online RL algorithm known to provably attain minimax-optimal regret without incurring any burn-in cost.
A note of caution is warranted, however. At present, it remains unclear whether UCB-VI is fundamentally suboptimal or whether its large burn-in cost is merely an artifact of existing proof techniques, given the absence of algorithm-specific lower bounds for UCB-VI. Having said that, MVP enjoys an additional practical advantage over UCB-VI: a substantially lower switching cost. Owing to the doubling-based update rule, the sampling policy in MVP is no longer updated at every episode and may remain fixed for extended periods of time. This low-switching property can significantly reduce deployment and operational costs in applications where changing sampling policies is expensive.
Finally, it is noteworthy that, among the canonical MDP settings commonly studied in online RL, finite-horizon nonstationary MDPs remain the only setting for which minimax-optimal regret without a burn-in period is currently known to be achievable. How to establish similar results in other prominent settings, such as discounted infinite-horizon MDPs and finite-horizon stationary MDPs, remains open. Resolving these questions will likely require new ideas in both theory and algorithm design.
Following the analysis of UCB-VI, it is convenient to introduce the following logarithmic factor:
ι b : = 40000 ( log 2 K ) 3 log ( 3 S A H ) log 1 δ ′ with δ ′ = 100 δ S 2 A 2 H 2 K 2 . (416) \iota_b := 40000(\log_2 K)^3 \log(3SAH) \log\frac{1}{\delta'}
\qquad\text{with} \qquad
\delta' = \frac{100\delta}{S^2 A^2 H^2 K^2}. \tag*{(416)} ι b := 40000 ( log 2 K ) 3 log ( 3 S A H ) log δ ′ 1 with δ ′ = S 2 A 2 H 2 K 2 100 δ . (416)
As in § 8.7 , we first note that the desired regret bound in Theorem 5.10 holds trivially whenever K ≤ S A H ι b K \le SAH\iota_b K ≤ S A H ι b , since
Regret ( K ) = ∑ k = 1 K ( V 1 ⋆ ( s 1 k ) − V 1 π k ( s 1 k ) ) ≤ H K = min { S A H 3 K ι b , H K } . \operatorname{Regret}(K) = \sum_{k=1}^{K} \left(V_1^{\star}(s_1^k) - V_1^{\pi^k}(s_1^k)\right) \le HK = \min\left\{\sqrt{SAH^3K\iota_b},HK\right\}. Regret ( K ) = k = 1 ∑ K ( V 1 ⋆ ( s 1 k ) − V 1 π k ( s 1 k ) ) ≤ H K = min { S A H 3 K ι b , H K } .
It therefore suffices, for the remainder of this subsection, to focus on the regime where
K ≥ S A H ι b . (417) K \ge SAH\iota_b. \tag*{(417)} K ≥ S A H ι b . (417)
Preparation: an expanded set of samples from a generative model. We now introduce a seemingly different, yet closely related, sampling mechanism in which independent samples are generated from a generative model. Specifically, before online learning begins, we use a generative model to draw a collection of independent samples, as defined below.
Definition 5.1 (Expanded samples). Let D e x p a n d \mathcal{D}^{\mathrm{expand}} D expand be a set of S A H K SAHK S A H K independent samples generated as follows: for each ( s , a , h ) ∈ S × A × [ H ] (s,a,h) \in\mathcal{S} \times\mathcal{A} \times[H] ( s , a , h ) ∈ S × A × [ H ] , draw K K K independent samples ( s , a , h , s ′ ( i ) ) (s,a,h,s'^{(i)}) ( s , a , h , s ′ ( i ) ) obeying
s ′ ( i ) ∼ i n d . P s , a , h for all 1 ≤ i ≤ K . s'^{(i)} \mathrel{\overset{\mathrm{ind.}}{\sim}} P_{s,a,h}
\qquad\text{for all } 1 \le i \le K. s ′ ( i ) ∼ ind. P s , a , h for all 1 ≤ i ≤ K .
Importantly, D e x p a n d \mathcal{D}^{\mathrm{expand}} D expand can be coupled with the original set of samples, denoted by D o r i g \mathcal{D}^{\mathrm{orig}} D orig . Specifically, one first generates D e x p a n d \mathcal{D}^{\mathrm{expand}} D expand before online learning begins, and thereafter, whenever the online process queries ( s , a , h ) (s,a,h) ( s , a , h ) , one returns an unused sample from D e x p a n d \mathcal{D}^{\mathrm{expand}} D expand , without replacement. This coupling allows us to analyze the online process through the view of D e x p a n d \mathcal{D}^{\mathrm{expand}} D expand , which is sometimes more convenient. Unless stated otherwise, all subsequent proofs are carried out under this coupling between D o r i g \mathcal{D}^{\mathrm{orig}} D orig and D e x p a n d \mathcal{D}^{\mathrm{expand}} D expand .
Notation. As in the analysis of UCB-VI, it is convenient to introduce the following notation to index the iterates in Algorithm 5.3 explicitly by the episode number k k k . Some of the notation defined below is analogous to that previously used in the analysis of UCB-VI in § 8.7 .
P ^ s , a , h k ∈ R S \widehat{P}_{s,a,h}^{k} \in\mathbb{R}^{S} P s , a , h k ∈ R S : the latest update of the empirical transition probability vector P ^ s , a , h \widehat{P}_{s,a,h} P s , a , h before episode k k k .
b h k ( s , a ) ≥ 0 b_{h}^{k}(s,a) \ge0 b h k ( s , a ) ≥ 0 : the latest update of the exploration bonus term b h ( s , a ) b_h(s,a) b h ( s , a ) before episode k k k .
N h k , a l l ( s , a ) N_{h}^{k,\mathrm{all}}(s,a) N h k , all ( s , a ) : the total visitation count N h a l l ( s , a ) N_{h}^{\mathrm{all}}(s,a) N h all ( s , a ) of the ( s , a , h ) (s,a,h) ( s , a , h ) -tuple before episode k k k .
N h k ( s , a ) N_{h}^{k}(s,a) N h k ( s , a ) : the visitation count N h ( s , a ) N_h(s,a) N h ( s , a ) of the ( s , a , h ) (s,a,h) ( s , a , h ) -tuple of the latest doubling batch used to compute P ^ s , a , h \widehat{P}_{s,a,h} P s , a , h before episode k k k . When N h k , a l l ( s , a ) = 0 N_{h}^{k,\mathrm{all}}(s,a)=0 N h k , all ( s , a ) = 0 , set N h k ( s , a ) = 1 N_{h}^{k}(s,a)=1 N h k ( s , a ) = 1 for ease of presentation.
We shall also make use of the following notation for the empirical transition probability vector indexed by the batch number.
For any j ≥ 2 j \ge2 j ≥ 2 (resp. j = 1 j=1 j = 1 ), let P ^ s , a , h ( j ) \widehat{P}_{s,a,h}^{(j)} P s , a , h ( j ) denote the empirical transition probability vector for ( s , a , h ) (s,a,h) ( s , a , h ) computed from the j j j -th batch (resp. the first batch) of data, namely, the { 2 j − 2 + i } i = 1 2 j − 2 \{2^{j-2}+i\}_{i=1}^{2^{j-2}} { 2 j − 2 + i } i = 1 2 j − 2 -th samples (resp. the 1st sample) associated with ( s , a , h ) (s,a,h) ( s , a , h ) . For completeness, we set the empirical transition probability vector for the 0-th batch to be P ^ s , a , h ( 0 ) ( s ′ ) = 1 / S \widehat{P}_{s,a,h}^{(0)}(s')=1/S P s , a , h ( 0 ) ( s ′ ) = 1/ S for every s ′ ∈ S s' \in\mathcal{S} s ′ ∈ S .
With the coupling between D e x p a n d \mathcal{D}^{\mathrm{expand}} D expand and D o r i g \mathcal{D}^{\mathrm{orig}} D orig in mind, we will use P ^ s , a , h ( j ) \widehat{P}_{s,a,h}^{(j)} P s , a , h ( j ) interchangeably to denote the empirical probability vector constructed from the j j j -th batch of data from either D o r i g \mathcal{D}^{\mathrm{orig}} D orig or D e x p a n d \mathcal{D}^{\mathrm{expand}} D expand , as long as no confusion arises.
Recap: a basic regret decomposition. Repeating the arguments in Steps 1-2 of § 8.7 for UCB-VI yields the same optimism property (388) and regret decomposition (390) for MVP; specifically, with probability at least 1 − 2 S A H K δ ′ 1-2SAHK\delta' 1 − 2 S A H K δ ′ , we have
Q h k ( s , a ) ≥ Q h ⋆ ( s , a ) and V h k ( s ) ≥ V h π ⋆ ( s ) (418) Q_h^k(s,a) \ge Q_h^{\star}(s,a) \qquad\text{and} \qquad V_h^k(s) \ge V_h^{\pi^\star}(s) \tag*{(418)} Q h k ( s , a ) ≥ Q h ⋆ ( s , a ) and V h k ( s ) ≥ V h π ⋆ ( s ) (418)
for all ( s , a , h , k ) ∈ S × A × [ H ] × [ K ] (s,a,h,k) \in\mathcal{S} \times\mathcal{A} \times[H] \times[K] ( s , a , h , k ) ∈ S × A × [ H ] × [ K ] , and
Regret ( K ) ≤ ∑ k = 1 K ∑ h = 1 H ⟨ P ^ s h k , a h k , h − P s h k , a h k , h , V h + 1 k ⟩ ⏟ = : T 1 + ∑ k = 1 K ∑ h = 1 H b h k ( s h k , a h k ) ⏟ = : T 2 + ∑ k = 1 K ∑ h = 1 H ⟨ P s h k , a h k , h − e s h + 1 k , V h + 1 k ⟩ ⏟ = : T 3 + ∑ k = 1 K ( ∑ h = 1 H r h ( s h k , a h k ) − V 1 π k ( s 1 k ) ) . (419) \begin{aligned}
\operatorname{Regret}(K) \le{}&
\sum_{k=1}^{K}\sum_{h=1}^{H}
\underbrace{\left\langle\widehat{P}_{s_h^k,a_h^k,h}-P_{s_h^k,a_h^k,h},V_{h+1}^k\right\rangle}_{=:T_1}
+\sum_{k=1}^{K}\sum_{h=1}^{H}
\underbrace{b_h^k(s_h^k,a_h^k)}_{=:T_2}\\
&+\sum_{k=1}^{K}\sum_{h=1}^{H}
\underbrace{\left\langle P_{s_h^k,a_h^k,h}-e_{s_{h+1}^k},V_{h+1}^k\right\rangle}_{=:T_3}
+\sum_{k=1}^{K}\left(\sum_{h=1}^{H}r_h(s_h^k,a_h^k)-V_1^{\pi^k}(s_1^k)\right). \tag*{(419)}
\end{aligned} Regret ( K ) ≤ k = 1 ∑ K h = 1 ∑ H =: T 1 ⟨ P s h k , a h k , h − P s h k , a h k , h , V h + 1 k ⟩ + k = 1 ∑ K h = 1 ∑ H =: T 2 b h k ( s h k , a h k ) + k = 1 ∑ K h = 1 ∑ H =: T 3 ⟨ P s h k , a h k , h − e s h + 1 k , V h + 1 k ⟩ + k = 1 ∑ K ( h = 1 ∑ H r h ( s h k , a h k ) − V 1 π k ( s 1 k ) ) . (419)
The terms T 2 , … , T 4 T_2,\ldots,T_4 T 2 , … , T 4 can be handled in a similar manner as in the analysis of UCB-VI in § 8.7 . As alluded to previously, obtaining a sharp bound on T 1 T_1 T 1 constitutes one of the main objectives of the analysis that follows.
Step 1: a compressed representation of visitation counts { N h k ( s , a ) } \{N_h^k(s,a)\} { N h k ( s , a )} . As discussed in § 8.8.1 , a potential covering-based analysis strategy (see Strategy 2 therein) is to first condition on the visitation counts { N h k ( s , a ) } \{N_h^k(s,a)\} { N h k ( s , a )} when carrying out the concentration-of-measure analysis, and then take a union bound over all possible configurations of these visitation counts. A crucial step in implementing this strategy is therefore to control the total number of admissible configurations — namely, an appropriate covering number — for { N h k ( s , a ) } \{N_h^k(s,a)\} { N h k ( s , a )} . To this end, we introduce the notion of a profile , a term coined by Zhang et al. (2025e), which enables a compact representation of the visitation counts.
Definition 5.2 (Profile). Consider any collection { N h k , a l l ( s , a ) } \{N_h^{k,\mathrm{all}}(s,a)\} { N h k , all ( s , a )} . For any ( s , a , h , k ) ∈ S × A × [ H ] × [ K ] (s,a,h,k) \in\mathcal{S} \times\mathcal{A} \times[H] \times[K] ( s , a , h , k ) ∈ S × A × [ H ] × [ K ] , define
I s , a , h k : = { max { j ∈ Z : 2 j − 1 ≤ N h k , a l l ( s , a ) } , if N h k , a l l ( s , a ) > 0 , 0 , if N h k , a l l ( s , a ) = 0. I_{s,a,h}^k :=
\begin{cases}
\max\{j \in\mathbb{Z}: 2^{j-1} \leq N_h^{k,\mathrm{all}}(s,a)\}, & \text{if } N_h^{k,\mathrm{all}}(s,a)>0,\\
0, & \text{if } N_h^{k,\mathrm{all}}(s,a)=0.
\end{cases} I s , a , h k := { max { j ∈ Z : 2 j − 1 ≤ N h k , all ( s , a )} , 0 , if N h k , all ( s , a ) > 0 , if N h k , all ( s , a ) = 0. The profile of the k k k -th episode (1 ≤ k ≤ K 1 \leq k \leq K 1 ≤ k ≤ K ) and the corresponding total profile are then defined, respectively, as
I k : = { I s , a , h k } ( s , a , h ) ∈ S × A × [ H ] and I : = { I k } k = 1 K . \mathcal{I}^k := \{I_{s,a,h}^k\}_{(s,a,h)\in\mathcal{S}\times\mathcal{A}\times[H]}
\qquad\text{and}\qquad
\mathcal{I}:=\{\mathcal{I}^k\}_{k=1}^K. I k := { I s , a , h k } ( s , a , h ) ∈ S × A × [ H ] and I := { I k } k = 1 K .
By construction, given a total profile I \mathcal{I} I associated with { N h k , a l l ( s , a ) } \{N_h^{k,\mathrm{all}}(s,a)\} { N h k , all ( s , a )} , the empirical transition kernels satisfy
P ^ s , a , h k = P ^ s , a , h ( I s , a , h k ) , ∀ ( s , a , h , k ) ∈ S × A × [ H ] × [ K ] ; (420) \widehat{P}_{s,a,h}^k=\widehat{P}_{s,a,h}^{(I_{s,a,h}^k)},\qquad\forall(s,a,h,k)\in\mathcal{S}\times\mathcal{A}\times[H]\times[K]; \tag*{(420)} P s , a , h k = P s , a , h ( I s , a , h k ) , ∀ ( s , a , h , k ) ∈ S × A × [ H ] × [ K ] ; (420)
here, P ^ s , a , h ( j ) \widehat{P}_{s,a,h}^{(j)} P s , a , h ( j ) is as defined earlier in this subsection. Thus, a total profile fully determines when and where the empirical transition models are updated. Moreover, since each visitation count N h k ( s , a ) N_h^k(s,a) N h k ( s , a ) is recomputed only when the corresponding empirical model is updated (see line 10 of Algorithm 5.3), the total profile serves as a compact representation of the entire collection { N h k ( s , a ) } \{N_h^k(s,a)\} { N h k ( s , a )} .
To quantify the extent of the resulting compression (which stems from the doubling update mechanism), we next bound the total number of admissible profiles.
Lemma 5.11. Suppose that K ≥ S A H log 2 K K \geq SAH\log_2 K K ≥ S A H log 2 K . Then the number of all possible total profiles induced by Algorithm 5.3 is at most
( 4 S A H K ) S A H log 2 K + 1 . (4SAHK)^{SAH\log_2 K+1}. ( 4 S A H K ) S A H l o g 2 K + 1 .
Proof. The key idea is to consider the following set:
C : = { I = { I 1 , … , I K } | I 1 ≤ I 2 ≤ ⋯ ≤ I K , I k ∈ { 0 , 1 , … , log 2 K } S A H for all 1 ≤ k ≤ K } . (421) \mathcal{C}:=\left\{\mathcal{I}=\{\mathcal{I}^1,\ldots,\mathcal{I}^K\}\,\middle|\,\mathcal{I}^1\leq\mathcal{I}^2\leq\cdots\leq\mathcal{I}^K,\quad
\mathcal{I}^k\in\{0,1,\ldots,\log_2 K\}^{SAH}\ \text{for all }1\leq k\leq K\right\}. \tag*{(421)} C := { I = { I 1 , … , I K } I 1 ≤ I 2 ≤ ⋯ ≤ I K , I k ∈ { 0 , 1 , … , log 2 K } S A H for all 1 ≤ k ≤ K } . (421)
Since the visitation counts { N h k , a l l ( s , a ) } \{N_h^{k,\mathrm{all}}(s,a)\} { N h k , all ( s , a )} are non-decreasing in k k k , every total profile induced by Algorithm 5.3 satisfies the monotonicity constraint I 1 ≤ I 2 ≤ ⋯ ≤ I K \mathcal{I}^1\leq\mathcal{I}^2\leq\cdots\leq\mathcal{I}^K I 1 ≤ I 2 ≤ ⋯ ≤ I K , and hence belongs to C \mathcal{C} C . It therefore suffices to show that
∣ C ∣ ≤ ( 4 S A H K ) S A H log 2 K + 1 . (422) |\mathcal{C}|\leq(4SAHK)^{SAH\log_2 K+1}. \tag*{(422)} ∣ C ∣ ≤ ( 4 S A H K ) S A H l o g 2 K + 1 . (422)
The proof of this cardinality bound is deferred to § 8.16.1 . □
In contrast to the uncompressed setting (such as that arising in UCB–VI), where the visitation counts may admit exponentially many configurations in K K K , the doubling update mechanism adopted in Algorithm 5.3 leads to a substantial compression: by Lemma 5.11 , the exponent governing the number of admissible configurations grows only logarithmically with K K K . This compression is a key ingredient that makes the covering-based analysis strategy effective.
Before proceeding, we record another useful property of the visitation counts. This result is analogous to Lemma 5.3, but concerns the modified visitation counts associated with the epoch-based updates in MVP.
Lemma 5.12. For any K ≥ 2 K \ge2 K ≥ 2 , Algorithm 5.3 satisfies
∑ k = 1 K ∑ h = 1 H 1 max { N h k ( s h k , a h k ) , 1 } ≤ 4 S A H log 2 K . \sum_{k=1}^{K} \sum_{h=1}^{H} \frac{1}{\max\{N_{h}^{k}(s_{h}^{k},a_{h}^{k}),1\}} \le4SAH\log_{2}K. k = 1 ∑ K h = 1 ∑ H max { N h k ( s h k , a h k ) , 1 } 1 ≤ 4 S A H log 2 K . Proof. Fix any ( s , a , h ) ∈ S × A × [ H ] (s,a,h) \in\mathcal{S} \times\mathcal{A} \times[H] ( s , a , h ) ∈ S × A × [ H ] . By the doubling rule in Algorithm 5.3, the visitation count N h k ( s , a ) N_{h}^{k}(s,a) N h k ( s , a ) is within a factor of two of N h k , a l l ( s , a ) N_{h}^{k,\mathrm{all}}(s,a) N h k , all ( s , a ) . Consequently,
∑ k = 1 K 1 max { N h k ( s h k , a h k ) , 1 } 1 { ( s , a ) = ( s h k , a h k ) } ≤ ∑ k = 1 K 2 max { N h k , a l l ( s h k , a h k ) , 1 } 1 { ( s , a ) = ( s h k , a h k ) } \sum_{k=1}^{K} \frac{1}{\max\{N_{h}^{k}(s_{h}^{k},a_{h}^{k}),1\}}\mathbb{1}\{(s,a)=(s_{h}^{k},a_{h}^{k})\}
\le\sum_{k=1}^{K} \frac{2}{\max\{N_{h}^{k,\mathrm{all}}(s_{h}^{k},a_{h}^{k}),1\}}\mathbb{1}\{(s,a)=(s_{h}^{k},a_{h}^{k})\} k = 1 ∑ K max { N h k ( s h k , a h k ) , 1 } 1 1 {( s , a ) = ( s h k , a h k )} ≤ k = 1 ∑ K max { N h k , all ( s h k , a h k ) , 1 } 2 1 {( s , a ) = ( s h k , a h k )} ≤ 4 log 2 K , \le4\log_{2}K, ≤ 4 log 2 K , where the last inequality follows from the same argument as in the proof of Lemma 5.3 .
Step 2: decomposing T 1 T_{1} T 1 based on the doubling updates. With the notion of profiles in place, we now turn to the term T 1 T_{1} T 1 in (419) . To facilitate the analysis, we first introduce some additional notation.
I t r u e = { I 1 , t r u e , … , I K , t r u e } \mathcal{I}^{\mathrm{true}}=\{\mathcal{I}^{1,\mathrm{true}},\ldots,\mathcal{I}^{K,\mathrm{true}}\} I true = { I 1 , true , … , I K , true } , with I k , t r u e = { I s , a , h k , t r u e } \mathcal{I}^{k,\mathrm{true}}=\{\mathcal{I}_{s,a,h}^{k,\mathrm{true}}\} I k , true = { I s , a , h k , true } , denotes the total profile induced by the actual visitation counts { N h k , a l l ( s , a ) } \{N_{h}^{k,\mathrm{all}}(s,a)\} { N h k , all ( s , a )} generated during the online learning process;
k l , j , s , a , h k_{l,j,s,a,h} k l , j , s , a , h : the episode index corresponding to the ( 2 l − 1 + j ) (2^{l-1}+j) ( 2 l − 1 + j ) -th visit to ( s , a , h ) (s,a,h) ( s , a , h ) during the online learning process.
In view of (420) , we can reorganize T 1 T_{1} T 1 according to the doubling updates as follows
T 1 = ∑ k = 1 K ∑ h = 1 H ⟨ P ^ s h k , a h k , h k − P s h k , a h k , h , V h + 1 k ⟩ = ∑ k = 1 K ∑ h = 1 H ⟨ P ^ s h k , a h k , h ( I s h k , a h k , h k , t r u e ) − P s h k , a h k , h , V h + 1 k ⟩ = ∑ l = 0 log 2 K ∑ s , a , h ⟨ P ^ s , a , h ( l ) − P s , a , h , ∑ k = 1 K V h + 1 k 1 { ( s h k , a h k ) = ( s , a ) , I s , a , h k , t r u e = l } ⟩ ≤ ∑ l = 1 log 2 K ∑ s , a , h ⟨ P ^ s , a , h ( l ) − P s , a , h , ∑ k = 1 K V h + 1 k 1 { ( s h k , a h k ) = ( s , a ) , I s , a , h k , t r u e = l } ⟩ + S A H 2 = ∑ l = 1 log 2 K ∑ j = 1 2 l − 1 ∑ s , a , h ⟨ P ^ s , a , h ( l ) − P s , a , h , V h + 1 k l , j , s , a , h ⟩ + S A H 2 , (423) \begin{aligned}
T_{1}
&= \sum_{k=1}^{K}\sum_{h=1}^{H}\left\langle\widehat{P}_{s_{h}^{k},a_{h}^{k},h}^{k}-P_{s_{h}^{k},a_{h}^{k},h},V_{h+1}^{k}\right\rangle\\
&= \sum_{k=1}^{K}\sum_{h=1}^{H}\left\langle\widehat{P}_{s_{h}^{k},a_{h}^{k},h}^{(\mathcal{I}_{s_{h}^{k},a_{h}^{k},h}^{k,\mathrm{true}})}-P_{s_{h}^{k},a_{h}^{k},h},V_{h+1}^{k}\right\rangle\\
&= \sum_{l=0}^{\log_{2}K}\sum_{s,a,h}\left\langle\widehat{P}_{s,a,h}^{(l)}-P_{s,a,h},\sum_{k=1}^{K}V_{h+1}^{k}\mathbb{1}\{(s_{h}^{k},a_{h}^{k})=(s,a),\mathcal{I}_{s,a,h}^{k,\mathrm{true}}=l\}\right\rangle\\
&\le\sum_{l=1}^{\log_{2}K}\sum_{s,a,h}\left\langle\widehat{P}_{s,a,h}^{(l)}-P_{s,a,h},\sum_{k=1}^{K}V_{h+1}^{k}\mathbb{1}\{(s_{h}^{k},a_{h}^{k})=(s,a),\mathcal{I}_{s,a,h}^{k,\mathrm{true}}=l\}\right\rangle+SAH^{2} \\
&= \sum_{l=1}^{\log_{2}K}\sum_{j=1}^{2^{l-1}}\sum_{s,a,h}\left\langle\widehat{P}_{s,a,h}^{(l)}-P_{s,a,h},V_{h+1}^{k_{l,j,s,a,h}}\right\rangle+SAH^{2}, \tag*{(423)}
\end{aligned} T 1 = k = 1 ∑ K h = 1 ∑ H ⟨ P s h k , a h k , h k − P s h k , a h k , h , V h + 1 k ⟩ = k = 1 ∑ K h = 1 ∑ H ⟨ P s h k , a h k , h ( I s h k , a h k , h k , true ) − P s h k , a h k , h , V h + 1 k ⟩ = l = 0 ∑ l o g 2 K s , a , h ∑ ⟨ P s , a , h ( l ) − P s , a , h , k = 1 ∑ K V h + 1 k 1 {( s h k , a h k ) = ( s , a ) , I s , a , h k , true = l } ⟩ ≤ l = 1 ∑ l o g 2 K s , a , h ∑ ⟨ P s , a , h ( l ) − P s , a , h , k = 1 ∑ K V h + 1 k 1 {( s h k , a h k ) = ( s , a ) , I s , a , h k , true = l } ⟩ + S A H 2 = l = 1 ∑ l o g 2 K j = 1 ∑ 2 l − 1 s , a , h ∑ ⟨ P s , a , h ( l ) − P s , a , h , V h + 1 k l , j , s , a , h ⟩ + S A H 2 , (423)
where, for notational convenience, we set V h + 1 k = 0 V_{h+1}^{k}=0 V h + 1 k = 0 for all k > K k>K k > K . Here, the second line results from (420) , the penultimate line follows from the bound 0 ≤ V h + 1 k ( s ) ≤ H 0\le V_{h+1}^{k}(s)\le H 0 ≤ V h + 1 k ( s ) ≤ H for all s ∈ S s\in\mathcal{S} s ∈ S , while the last identity follows from the fact that, for each ( s , a , h ) (s,a,h) ( s , a , h ) and l ≥ 1 l\ge1 l ≥ 1 , the episodes with I s , a , h k , t r u e = l I_{s,a,h}^{k,\mathrm{true}}=l I s , a , h k , true = l correspond to the ( 2 l − 1 + j ) (2^{l-1}+j) ( 2 l − 1 + j ) -th visits to ( s , a , h ) (s,a,h) ( s , a , h ) , for 1 ≤ j ≤ 2 l − 1 1\le j\le2^{l-1} 1 ≤ j ≤ 2 l − 1 .
The key point of (423) is that it reorganizes the original sum by the doubling level l l l , grouping together all terms associated with the same empirical transition estimate P ^ s , a , h ( l ) \widehat{P}_{s,a,h}^{(l)} P s , a , h ( l ) . It therefore suggests first controlling the term
∑ s , a , h ⟨ P ^ s , a , h ( l ) − P s , a , h , V h + 1 k l , j , s , a , h ⟩ . (424) \sum_{s,a,h}\left\langle\widehat{P}_{s,a,h}^{(l)}-P_{s,a,h},V_{h+1}^{k_{l,j,s,a,h}}\right\rangle. \tag*{(424)} s , a , h ∑ ⟨ P s , a , h ( l ) − P s , a , h , V h + 1 k l , j , s , a , h ⟩ . (424)
As before, the main challenge is the intricate statistical dependence between P ^ s , a , h ( l ) \widehat{P}_{s,a,h}^{(l)} P s , a , h ( l ) and V h + 1 k l , j , s , a , h V_{h+1}^{k_{l,j,s,a,h}} V h + 1 k l , j , s , a , h . We address this dependence issue next.
Step 3: bounding the term (424) via coupling and covering. To obtain a sharp bound on (424) , we employ a covering argument based on the profiles introduced above. We first describe the main idea, before providing the details.
For any fixed total profile I \mathcal{I} I and fixed l ≤ log 2 K l\le\log_{2}K l ≤ log 2 K , derive a high-probability bound for weighted sums of the form
∑ s , a , h ( P ^ s , a , h ( l ) − P s , a , h ) X h + 1 , s , a , (425) \sum_{s,a,h}\left(\widehat{P}_{s,a,h}^{(l)}-P_{s,a,h}\right)X_{h+1,s,a}, \tag*{(425)} s , a , h ∑ ( P s , a , h ( l ) − P s , a , h ) X h + 1 , s , a , (425)
where each vector X h + 1 , s , a X_{h+1,s,a} X h + 1 , s , a is a deterministic function of the (fixed) profile I \mathcal{I} I and the samples collected at steps h ′ ≥ h + 1 h'\ge h+1 h ′ ≥ h + 1 . Here, under the view of the expanded sample set D e x p a n d \mathcal{D}^{\mathrm{expand}} D expand (cf. Definition 5.1), P ^ s , a , h ( l ) \widehat{P}_{s,a,h}^{(l)} P s , a , h ( l ) is constructed from the l l l -th batch of samples associated with ( s , a , h ) (s,a,h) ( s , a , h ) . These samples are independent of those determining X h + 1 , s , a X_{h+1,s,a} X h + 1 , s , a once I \mathcal{I} I is fixed, allowing us to invoke standard martingale concentration inequalities.
Take a union bound over all admissible total profiles I \mathcal{I} I . In view of the profile-counting bound in Lemma 5.11, this yields uniform control of (425) simultaneously over all profiles I \mathcal{I} I and all associated sequences { X h + 1 , s , a } \{X_{h+1,s,a}\} { X h + 1 , s , a } .
Show that the above uniform bound can be applied to the decomposition in (423) to obtain the desired bound.
Step 3.1: establishing uniform bounds on (425) . We first carry out steps i) and ii). To formalize the class of vectors { X h , s , a } \{X_{h,s,a}\} { X h , s , a } appearing in (425) , we introduce the following assumption.
Assumption 5.1. For each total profile I ∈ C \mathcal{I}\in\mathcal{C} I ∈ C (cf. (421) ), suppose the set { X h , I } 1 ≤ h ≤ H \{\mathcal{X}_{h,\mathcal{I}}\}_{1\le h\le H} { X h , I } 1 ≤ h ≤ H satisfies the following properties for every 1 ≤ h ≤ H 1\le h\le H 1 ≤ h ≤ H :
X h , I \mathcal{X}_{h,\mathcal{I}} X h , I is a set of no more than K + 1 K+1 K + 1 nonnegative vectors in R S \mathbb{R}^{\mathcal{S}} R S , including the all-zero vector 0 0 0 , and each vector X ∈ X h , I X\in\mathcal{X}_{h,\mathcal{I}} X ∈ X h , I obeys ∥ X ∥ ∞ ≤ H \lVert X\rVert_{\infty}\le H ∥ X ∥ ∞ ≤ H ;
X h + 1 , I \mathcal{X}_{h+1,\mathcal{I}} X h + 1 , I is a collection determined by a deterministic mapping of I \mathcal{I} I , the reward function r r r , and the empirical distribution vectors (from the view of the expanded sample set D e x p a n d \mathcal{D}^{\mathrm{expand}} D expand )
{ P ^ s , a , h ′ ( r s , a , h ′ ) } h ′ ∈ ( h , H ] , ( s , a , k ) ∈ S × A × [ K ] . \left\{\widehat{P}_{s,a,h'}^{(r_{s,a,h'})}\right\}_{h'\in(h,H],\,(s,a,k)\in\mathcal{S}\times\mathcal{A}\times[K]}. { P s , a , h ′ ( r s , a , h ′ ) } h ′ ∈ ( h , H ] , ( s , a , k ) ∈ S × A × [ K ] .
For each collection { X h , I } \{\mathcal{X}_{h,\mathcal{I}}\} { X h , I } satisfying the above assumptions, we can readily establish a uniform concentration bound by combining Freedman’s martingale concentration inequality with a union bound; the proof is postponed to § 8.17 .
Lemma 5.13. Suppose that K ≥ S A H log 2 K K \ge SAH\log_2 K K ≥ S A H log 2 K . For each I ∈ C \mathcal{I} \in\mathcal{C} I ∈ C (cf. (421) ), consider a set { X h , I } 1 ≤ h ≤ H \{\mathcal{X}_{h,\mathcal{I}}\}_{1\le h\le H} { X h , I } 1 ≤ h ≤ H satisfying Assumption 5.1. Then, with probability at least 1 − δ ′ 1-\delta' 1 − δ ′ ,
∑ ( s , a , h ) ∈ S × A × [ H ] ⟨ P ^ s , a , h ( l ) − P s , a , h , X h + 1 , s , a ⟩ ≤ ∑ ( s , a , h ) ∈ S × A × [ H ] max { ⟨ P ^ s , a , h ( l ) − P s , a , h , X h + 1 , s , a ⟩ , 0 } \sum_{(s,a,h)\in\mathcal{S}\times\mathcal{A}\times[H]}\left\langle\widehat{P}^{(l)}_{s,a,h}-P_{s,a,h},X_{h+1,s,a}\right\rangle
\le
\sum_{(s,a,h)\in\mathcal{S}\times\mathcal{A}\times[H]}\max\left\{\left\langle\widehat{P}^{(l)}_{s,a,h}-P_{s,a,h},X_{h+1,s,a}\right\rangle,0\right\} ( s , a , h ) ∈ S × A × [ H ] ∑ ⟨ P s , a , h ( l ) − P s , a , h , X h + 1 , s , a ⟩ ≤ ( s , a , h ) ∈ S × A × [ H ] ∑ max { ⟨ P s , a , h ( l ) − P s , a , h , X h + 1 , s , a ⟩ , 0 } ≤ S A H ι b 2 l log 2 K ∑ s , a , h Var P s , a , h ( X h + 1 , s , a ) + S A H 2 ι b 2 l log 2 K (426) \le\sqrt{\frac{SAH\iota_b}{2^l\log_2 K}\sum_{s,a,h}\operatorname{Var}_{P_{s,a,h}}(X_{h+1,s,a})+\frac{SAH^2\iota_b}{2^l\log_2 K}} \tag*{(426)} ≤ 2 l log 2 K S A H ι b s , a , h ∑ Var P s , a , h ( X h + 1 , s , a ) + 2 l log 2 K S A H 2 ι b (426) holds simultaneously for all I ∈ C \mathcal{I}\in\mathcal{C} I ∈ C , all 2 ≤ l ≤ log 2 K + 1 2\le l\le\log_2 K+1 2 ≤ l ≤ log 2 K + 1 , and all sequences { X h , s , a } ( s , a , h ) ∈ S × A × [ H ] \{X_{h,s,a}\}_{(s,a,h)\in\mathcal{S}\times\mathcal{A}\times[H]} { X h , s , a } ( s , a , h ) ∈ S × A × [ H ] obeying X h , s , a ∈ X h , I X_{h,s,a}\in\mathcal{X}_{h,\mathcal{I}} X h , s , a ∈ X h , I , ∀ ( s , a , h ) ∈ S × A × [ H ] \forall(s,a,h)\in\mathcal{S}\times\mathcal{A}\times[H] ∀ ( s , a , h ) ∈ S × A × [ H ] .
Step 3.2: coupling back to the online process. Next, we carry out iii) through the following lemma. This result is proven by combining the uniform bound in Lemma 5.13 with the decomposition (423) , which allows one to rigorously couple D e x p a n d \mathcal{D}^{\mathrm{expand}} D expand with the online learning process. The proof is deferred to § 8.17.1 .
Lemma 5.14. Suppose that K ≥ S A H log 2 K K \ge SAH\log_2 K K ≥ S A H log 2 K , and recall the definition of ι b \iota_b ι b in (416) . With probability exceeding 1 − δ ′ 1-\delta' 1 − δ ′ , we have
∑ k = 1 K ∑ h = 1 H ⟨ P ^ s h k , a h k , h k − P s h k , a h k , h , V h + 1 k ⟩ ≤ ∑ k = 1 K ∑ h = 1 H max { ⟨ P ^ s h k , a h k , h k − P s h k , a h k , h , V h + 1 k ⟩ , 0 } \sum_{k=1}^{K}\sum_{h=1}^{H}\left\langle\widehat{P}^{k}_{s_h^k,a_h^k,h}-P_{s_h^k,a_h^k,h},V_{h+1}^k\right\rangle
\le
\sum_{k=1}^{K}\sum_{h=1}^{H}\max\left\{\left\langle\widehat{P}^{k}_{s_h^k,a_h^k,h}-P_{s_h^k,a_h^k,h},V_{h+1}^k\right\rangle,0\right\} k = 1 ∑ K h = 1 ∑ H ⟨ P s h k , a h k , h k − P s h k , a h k , h , V h + 1 k ⟩ ≤ k = 1 ∑ K h = 1 ∑ H max { ⟨ P s h k , a h k , h k − P s h k , a h k , h , V h + 1 k ⟩ , 0 } ≤ S A H ι b ∑ k = 1 K ∑ h = 1 H Var P s h k , a h k , h ( V h + 1 k ) + S A H 2 ι b \le\sqrt{SAH\iota_b\sum_{k=1}^{K}\sum_{h=1}^{H}\operatorname{Var}_{P_{s_h^k,a_h^k,h}}(V_{h+1}^k)+SAH^2\iota_b} ≤ S A H ι b k = 1 ∑ K h = 1 ∑ H Var P s h k , a h k , h ( V h + 1 k ) + S A H 2 ι b and
∑ k = 1 K ∑ h = 1 H ⟨ P ^ s h k , a h k , h k − P s h k , a h k , h , ( V h + 1 k ) 2 ⟩ ≤ S A H 3 ι b ∑ k = 1 K ∑ h = 1 H Var P s h k , a h k , h ( ( V h + 1 k ) 2 ) + S A H 3 ι b . \sum_{k=1}^{K}\sum_{h=1}^{H}\left\langle\widehat{P}^{k}_{s_h^k,a_h^k,h}-P_{s_h^k,a_h^k,h},(V_{h+1}^k)^2\right\rangle
\le
\sqrt{SAH^3\iota_b\sum_{k=1}^{K}\sum_{h=1}^{H}\operatorname{Var}_{P_{s_h^k,a_h^k,h}}((V_{h+1}^k)^2)+SAH^3\iota_b}. k = 1 ∑ K h = 1 ∑ H ⟨ P s h k , a h k , h k − P s h k , a h k , h , ( V h + 1 k ) 2 ⟩ ≤ S A H 3 ι b k = 1 ∑ K h = 1 ∑ H Var P s h k , a h k , h (( V h + 1 k ) 2 ) + S A H 3 ι b .
Remark 5.10. In addition to controlling T 1 T_1 T 1 , Lemma 5.14 provides bounds on the two related quantities:
∑ k , h max { ⟨ P ^ s h k , a h k , h k − P s h k , a h k , h , V h + 1 k ⟩ , 0 } \sum_{k,h}\max\left\{\left\langle\widehat{P}^{k}_{s_h^k,a_h^k,h}-P_{s_h^k,a_h^k,h},V_{h+1}^k\right\rangle,0\right\} k , h ∑ max { ⟨ P s h k , a h k , h k − P s h k , a h k , h , V h + 1 k ⟩ , 0 }
and
∑ k , h ⟨ P ^ s h k , a h k , h k − P s h k , a h k , h , ( V h + 1 k ) 2 ⟩ . \sum_{k,h}\left\langle\widehat{P}^{k}_{s_h^k,a_h^k,h}-P_{s_h^k,a_h^k,h},(V_{h+1}^k)^2\right\rangle. k , h ∑ ⟨ P s h k , a h k , h k − P s h k , a h k , h , ( V h + 1 k ) 2 ⟩ .
These bounds follow from the same argument and will be useful in the subsequent analysis.
Step 4: combining and solving the coupled inequalities. We now assemble the estimates (e.g., bounds on the quantities T 1 , … , T 4 T_1,\ldots,T_4 T 1 , … , T 4 in (419) ) established in the previous steps and complete the regret analysis. As in the proof of UCB-VI (§ 8.7 ), it is convenient to introduce the following auxiliary quantities:
T 5 : = ∑ k = 1 K ∑ h = 1 H Var P ^ s h k , a h k , h k ( V h + 1 k ) , (427) T_5 := \sum_{k=1}^{K}\sum_{h=1}^{H}\operatorname{Var}_{\widehat{P}^{k}_{s_h^k,a_h^k,h}}(V_{h+1}^k), \tag*{(427)} T 5 := k = 1 ∑ K h = 1 ∑ H Var P s h k , a h k , h k ( V h + 1 k ) , (427)
T 6 u c b : = ∑ k = 1 K ∑ h = 1 H Var P s h k , a h k , h ( V h + 1 k ) , (428) T_6^{\mathrm{ucb}} := \sum_{k=1}^{K} \sum_{h=1}^{H} \operatorname{Var}_{P_{s_h^k,a_h^k,h}}\left(V_{h+1}^k\right), \tag*{(428)} T 6 ucb := k = 1 ∑ K h = 1 ∑ H Var P s h k , a h k , h ( V h + 1 k ) , (428)
W 3 : = ∑ k = 1 K ∑ h = 1 H ⟨ P ^ s h k , a h k , h − P s h k , a h k , h , ( V h + 1 k ) 2 ⟩ . (429) W_3 := \sum_{k=1}^{K} \sum_{h=1}^{H} \left\langle\widehat{P}_{s_h^k,a_h^k,h} - P_{s_h^k,a_h^k,h}, \left(V_{h+1}^k\right)^2 \right\rangle. \tag*{(429)} W 3 := k = 1 ∑ K h = 1 ∑ H ⟨ P s h k , a h k , h − P s h k , a h k , h , ( V h + 1 k ) 2 ⟩ . (429)
W 4 : = ∑ k = 1 K ∑ h = 1 H max { ⟨ P ^ s h k , a h k , h − P s h k , a h k , h , V h + 1 k ⟩ , 0 } , (430) W_4 := \sum_{k=1}^{K} \sum_{h=1}^{H} \max\left\{\left\langle\widehat{P}_{s_h^k,a_h^k,h} - P_{s_h^k,a_h^k,h}, V_{h+1}^k \right\rangle,0\right\}, \tag*{(430)} W 4 := k = 1 ∑ K h = 1 ∑ H max { ⟨ P s h k , a h k , h − P s h k , a h k , h , V h + 1 k ⟩ , 0 } , (430)
These quantities naturally arise in the variance decomposition and allow the different error terms to be coupled through a small collection of inequalities.
We begin by bounding T 1 T_1 T 1 (cf. (419) ), W 3 W_3 W 3 and W 4 W_4 W 4 (cf. (5.60)) in terms of T 6 u c b T_6^{\mathrm{ucb}} T 6 ucb (cf. (428) ). In view of Lemma 5.14, with probability at least 1 − δ ′ 1-\delta' 1 − δ ′ ,
T 1 ≤ W 4 ≤ T 6 u c b S A H ι b + S A H 2 ι b ; (431) T_1 \le W_4 \le\sqrt{T_6^{\mathrm{ucb}}SAH\iota_b}+SAH^2\iota_b; \tag*{(431)} T 1 ≤ W 4 ≤ T 6 ucb S A H ι b + S A H 2 ι b ; (431)
W 3 ≤ T 6 u c b S A H 3 ι b + S A H 3 ι b . (432) W_3 \le\sqrt{T_6^{\mathrm{ucb}}SAH^3\iota_b}+SAH^3\iota_b. \tag*{(432)} W 3 ≤ T 6 ucb S A H 3 ι b + S A H 3 ι b . (432)
The other terms T 2 T_2 T 2 , T 3 T_3 T 3 , and T 4 T_4 T 4 (cf. (419) ) are controlled exactly as in the analysis of UCB-VI (see Lemma 5.8), yielding
T 2 ≤ 1 2 S A H T 5 ι b + 1 2 S A H 2 ι b (433) T_2 \le\frac{1}{2}\sqrt{SAHT_5\iota_b}+\frac{1}{2}SAH^2\iota_b \tag*{(433)} T 2 ≤ 2 1 S A H T 5 ι b + 2 1 S A H 2 ι b (433)
∣ T 3 ∣ ≤ T 6 u c b ι b + H ι b (434) \lvert T_3\rvert\le\sqrt{T_6^{\mathrm{ucb}}\iota_b}+H\iota_b \tag*{(434)} ∣ T 3 ∣ ≤ T 6 ucb ι b + H ι b (434)
∣ T 4 ∣ ≤ K H 2 ι b (435) \lvert T_4\rvert\le\sqrt{KH^2\iota_b} \tag*{(435)} ∣ T 4 ∣ ≤ K H 2 ι b (435)
with probability at least 1 − ( 4 log 2 K ) δ ′ 1-(4\log_2 K)\delta' 1 − ( 4 log 2 K ) δ ′ . In addition, repeating the same arguments as in the proof of Lemma 5.9 reveals that, with probability at least 1 − 2 δ ′ log ( K H ) 1-2\delta'\log(KH) 1 − 2 δ ′ log ( K H ) ,
T 5 ≤ W 3 + 2 H T 2 + H 2 T 6 u c b ι b + K H 2 ι b , (436) T_5 \le W_3+2HT_2+\sqrt{H^2T_6^{\mathrm{ucb}}\iota_b}+KH^2\iota_b, \tag*{(436)} T 5 ≤ W 3 + 2 H T 2 + H 2 T 6 ucb ι b + K H 2 ι b , (436)
At this point, the only term that has not yet been controlled is T 6 u c b T_6^{\mathrm{ucb}} T 6 ucb , which is still intertwined with the other quantities. In the analysis of UCB-VI (§ 8.7 ), this quantity was bounded indirectly by first controlling T 6 l o w T_6^{\mathrm{low}} T 6 low and T 6 d i f f T_6^{\mathrm{diff}} T 6 diff separately. Here, we instead bound T 6 u c b T_6^{\mathrm{ucb}} T 6 ucb directly in terms of T 2 T_2 T 2 and W 4 W_4 W 4 , thereby closing the system of inequalities. This result is formalized in the following lemma; the proof is provided in § 8.18 .
Lemma 5.15. With probability at least 1 − 4 S A H K δ ′ 1-4SAHK\delta' 1 − 4 S A H K δ ′ ,
T 6 u c b ≤ 4 H T 2 + 4 H W 4 + K H 2 ι b . (437) T_6^{\mathrm{ucb}} \le4HT_2+4HW_4+KH^2\iota_b. \tag*{(437)} T 6 ucb ≤ 4 H T 2 + 4 H W 4 + K H 2 ι b . (437)
Thus, T 6 u c b T_6^{\mathrm{ucb}} T 6 ucb is now bounded entirely in terms of quantities that have already been bounded, completing the chain of recursive estimates.
We now have a closed system of inequalities. Solving this system of inequalities in (5.61) and combining the resulting estimates with the regret decomposition (419) , we can establish that
Regret ( K ) ≤ T 1 + T 2 + T 3 + T 4 ≲ S A H 3 K ι b (438) \operatorname{Regret}(K) \le T_1+T_2+T_3+T_4 \lesssim\sqrt{SAH^3K\iota_b} \tag*{(438)} Regret ( K ) ≤ T 1 + T 2 + T 3 + T 4 ≲ S A H 3 K ι b (438)
as claimed. Since solving the coupled inequalities involves only routine algebraic manipulations, we defer the details to § 8.18.1 .
Model-free online RL
Having established the statistical efficiency of model-based methods for online RL, we now shift our attention to model-free algorithms. In contrast to their model-based counterparts, the best currently known model-free approaches still fail to achieve minimax-optimal regret guarantees over the full range of sample sizes. For this reason, rather than providing a treatment as comprehensive as that for model-based methods, we restrict ourselves to a brief overview of the main ideas underlying optimistic Q-learning algorithms.
Broadly speaking, the key question is how to incorporate the principle of optimism into Q-learning in order to encourage efficient exploration, for instance by constructing suitable upper confidence estimates for the optimal Q-values. One of the most notable developments along this direction is the optimistic Q-learning algorithm — commonly referred to as Q-UCB— developed by Jin et al. (2018), whose complete procedure is summarized in Algorithm 5.4. Compared to the vanilla Q-learning algorithm introduced in Chapter 4, the update rule of Q-UCB incorporates two key modifications:
First, the Q-learning update now incorporates an exploration bonus b n b_n b n , analogous to the bonus used in the UCB-VI algorithm, in order to ensure that the learned Q-function remains an optimistic estimate (i.e., an upper confidence bound) of the optimal Q-function. More specifically, the optimistic Q-learning update takes the form
Q h u c b ( s h , a h ) ← ( 1 − η n ) Q h u c b ( s h , a h ) + η n { r h ( s h , a h ) + V h + 1 u c b ( s h + 1 ) + b n } , Q_h^{\mathrm{ucb}}(s_h,a_h) \leftarrow(1-\eta_n)Q_h^{\mathrm{ucb}}(s_h,a_h)+\eta_n\{r_h(s_h,a_h)+V_{h+1}^{\mathrm{ucb}}(s_{h+1})+b_n\}, Q h ucb ( s h , a h ) ← ( 1 − η n ) Q h ucb ( s h , a h ) + η n { r h ( s h , a h ) + V h + 1 ucb ( s h + 1 ) + b n } ,
V h u c b ( s h ) ← min { max a ′ ∈ A Q h u c b ( s h , a ′ ) , H } . V_h^{\mathrm{ucb}}(s_h) \leftarrow\min\left\{\max_{a'\in\mathcal{A}}Q_h^{\mathrm{ucb}}(s_h,a'),H\right\}. V h ucb ( s h ) ← min { a ′ ∈ A max Q h ucb ( s h , a ′ ) , H } .
where Q h u c b Q_h^{\mathrm{ucb}} Q h ucb (resp. V h u c b V_h^{\mathrm{ucb}} V h ucb ) denotes the running estimate of the optimal Q-function (resp. value function), and b n b_n b n denotes the exploration bonus depending on the visitation count n n n described above. A simple and natural choice of the bonus term can be derived from Hoeffding-type concentration inequalities:
b n = c b H 3 log ( S A T / δ ) n , (439) b_n=c_b\sqrt{\frac{H^3\log(SAT/\delta)}{n}}, \tag*{(439)} b n = c b n H 3 log ( S A T / δ ) , (439)
where c b > 0 c_b>0 c b > 0 is a suitable universal constant, and δ ∈ ( 0 , 1 ) \delta\in(0,1) δ ∈ ( 0 , 1 ) is the usual confidence parameter.
Second, the learning rate for each visited entry ( s h , a h ) (s_h,a_h) ( s h , a h ) is chosen adaptively according to the following linearly rescaled schedule:
η n = H + 1 H + n , (440) \eta_n=\frac{H+1}{H+n}, \tag*{(440)} η n = H + n H + 1 , (440)
where n n n denotes the number of visits accumulated thus far for ( s h , a h ) (s_h,a_h) ( s h , a h ) at step h h h . This carefully designed learning-rate schedule plays an important role in controlling error propagation across the horizon and, consequently, in achieving a regret bound with polynomial dependence on H H H .
The resulting regret guarantee for this model-free online RL algorithm, originally established by Jin et al. (2018), is summarized below.
Theorem 5.16 (Jin et al. (2018)). For any K ≥ 1 K\ge1 K ≥ 1 and any 0 < δ < 1 0<\delta<1 0 < δ < 1 , the Q-UCB algorithm in Algorithm 5.4 with Hoeffding-style bonuses (439) achieves
Regret ( K ) ≤ O ~ ( S A H 5 K ) (441) \operatorname{Regret}(K)\le\widetilde{O}\left(\sqrt{SAH^5K}\right) \tag*{(441)} Regret ( K ) ≤ O ( S A H 5 K ) (441) with probability at least 1 − δ 1-\delta 1 − δ , provided that the constant c b > 0 c_b>0 c b > 0 in (439) is suitably chosen.
Algorithm 5.4: Q-learning with upper confidence bounds (Q-UCB) for episodic online RL
input: number of episodes K K K , reward function r r r , confidence parameter δ \delta δ .
initialize N h ( s , a ) ← 0 N_h(s,a) \leftarrow0 N h ( s , a ) ← 0 , Q h u c b ( s , a ) ← H Q_h^{\mathrm{ucb}}(s,a) \leftarrow H Q h ucb ( s , a ) ← H , and V H + 1 ( s ) ← 0 V_{H+1}(s) \leftarrow0 V H + 1 ( s ) ← 0 for all ( s , a , h ) ∈ S × A × [ H ] (s,a,h) \in\mathcal{S} \times\mathcal{A} \times[H] ( s , a , h ) ∈ S × A × [ H ] .
for k = 1 , ⋯ , K k = 1,\cdots,K k = 1 , ⋯ , K do / policy update. /
update policy π \pi π such that π h ( s ) ← arg max a Q h u c b ( s , a ) \pi_h(s) \leftarrow\arg\max_a Q_h^{\mathrm{ucb}}(s,a) π h ( s ) ← arg max a Q h ucb ( s , a ) for all ( s , h ) ∈ S × [ H ] (s,h) \in\mathcal{S} \times[H] ( s , h ) ∈ S × [ H ] .
observe the initial state s 1 ∼ i n d . ρ s_1 \mathrel{\mathop{\sim}^{\mathrm{ind.}}} \rho s 1 ∼ ind. ρ from a new episode.
for h = 1 , ⋯ , H h = 1,\cdots,H h = 1 , ⋯ , H do / sampling. /
take a h = π h ( s h ) a_h = \pi_h(s_h) a h = π h ( s h ) , observe s h + 1 s_{h+1} s h + 1 ; set ( s , a , s ′ ) ← ( s h , a h , s h + 1 ) (s,a,s') \leftarrow(s_h,a_h,s_{h+1}) ( s , a , s ′ ) ← ( s h , a h , s h + 1 ) . / update of visitation counts. /
N h ( s , a ) ← N h ( s , a ) + 1 N_h(s,a) \leftarrow N_h(s,a)+1 N h ( s , a ) ← N h ( s , a ) + 1 ; n ← N h ( s , a ) n \leftarrow N_h(s,a) n ← N h ( s , a ) . / optimistic update of Q-functions & values. /
compute η n \eta_n η n and b n b_n b n according to (440) and (439) .
update
Q h u c b ( s , a ) ← ( 1 − η n ) Q h u c b ( s , a ) + η n { r h ( s , a ) + V h + 1 u c b ( s ′ ) + b n } , (442) Q_h^{\mathrm{ucb}}(s,a) \leftarrow(1-\eta_n)Q_h^{\mathrm{ucb}}(s,a)+\eta_n\{r_h(s,a)+V_{h+1}^{\mathrm{ucb}}(s')+b_n\}, \tag*{(442)} Q h ucb ( s , a ) ← ( 1 − η n ) Q h ucb ( s , a ) + η n { r h ( s , a ) + V h + 1 ucb ( s ′ ) + b n } , (442) V h u c b ( s ) ← min { max a ′ ∈ A Q h u c b ( s , a ′ ) , H } . (443) V_h^{\mathrm{ucb}}(s) \leftarrow\min\left\{\max_{a'\in\mathcal{A}}Q_h^{\mathrm{ucb}}(s,a'),H\right\}. \tag*{(443)} V h ucb ( s ) ← min { a ′ ∈ A max Q h ucb ( s , a ′ ) , H } . (443)
In comparison with the statistical guarantees achieved by model-based methods (see, e.g., Theorem 5.10 ), the regret guarantee in Theorem 5.16 remains statistically suboptimal in its dependence on the horizon length H H H . This dependence can be improved through more refined uncertainty quantification for the exploration bonus. For instance, incorporating Bernstein-style exploration bonuses reduces the regret bound to O ~ ( S A H 4 K ) \widetilde{O}(\sqrt{SAH^4K}) O ( S A H 4 K ) for sufficiently large K K K (Jin et al. (2018)).
Moreover, additional variance-reduction techniques — often referred to as reference-advantage decomposition in the online model-free RL literature (Zhang et al., 2020f; Li et al., 2021b) — can further improve the regret guarantee to become asymptotically minimax optimal. Nevertheless, these refined guarantees still rely on an extremely large burn-in phase before statistically optimal behavior is attained. We refer interested readers to the aforementioned work for further details.
Analysis: proof of Theorem 5.16
In this subsection, we provide the proof of Theorem 5.16 , following the proof strategy developed in Jin et al. (2018, Section 4).
Preliminary properties of the learning rates. We begin by recording several useful properties of the learning rates (440) . For notational convenience, we also introduce two related sequences, defined for integers N ≥ 0 N \ge0 N ≥ 0 and n ≥ 1 n \ge1 n ≥ 1 :
η n N : = { η n ∏ i = n + 1 N ( 1 − η i ) , if N > n , η n , if N = n , 0 , if N < n . (444) \eta_n^N :=
\begin{cases}
\eta_n\prod_{i=n+1}^{N}(1-\eta_i), & \text{if } N>n,\\
\eta_n, & \text{if } N=n,\\
0, & \text{if } N<n.
\end{cases} \tag*{(444)} η n N := ⎩ ⎨ ⎧ η n ∏ i = n + 1 N ( 1 − η i ) , η n , 0 , if N > n , if N = n , if N < n . (444)
and
η 0 N : = { ∏ i = 1 N ( 1 − η i ) = 0 , if N > 0 , 1 , if N = 0. (445) \eta_0^N :=
\begin{cases}
\prod_{i=1}^{N}(1-\eta_i)=0, & \text{if } N>0,\\
1, & \text{if } N=0.
\end{cases} \tag*{(445)} η 0 N := { ∏ i = 1 N ( 1 − η i ) = 0 , 1 , if N > 0 , if N = 0. (445)
As can be easily verified, it holds that
∑ n = 1 N η n N = { 1 , if N > 0 , 0 , if N = 0. (446) \sum_{n=1}^{N}\eta_n^N =
\begin{cases}
1, & \text{if } N>0,\\
0, & \text{if } N=0.
\end{cases} \tag*{(446)} n = 1 ∑ N η n N = { 1 , 0 , if N > 0 , if N = 0. (446)
The following properties play an important role in the analysis; the proof is deferred to § 8.19.1 .
Lemma 5.17. For the learning rates in (440) , we have, for every integer N ≥ 1 N \ge1 N ≥ 1 ,
1 N ≤ ∑ n = 1 N η n N n ≤ 2 N , (447) \frac{1}{\sqrt{N}} \le\sum_{n=1}^{N}\frac{\eta_n^N}{\sqrt{n}} \le\frac{2}{\sqrt{N}}, \tag*{(447)} N 1 ≤ n = 1 ∑ N n η n N ≤ N 2 , (447) ∑ n = 1 N ( η n N ) 2 ≤ 2 H N . (448) \sum_{n=1}^{N}(\eta_n^N)^2 \le\frac{2H}{N}. \tag*{(448)} n = 1 ∑ N ( η n N ) 2 ≤ N 2 H . (448) In addition, for every integer n ≥ 1 n \ge1 n ≥ 1 , one has
∑ N = n ∞ η n N = 1 + 1 H . (449) \sum_{N=n}^{\infty}\eta_n^N = 1+\frac{1}{H}. \tag*{(449)} N = n ∑ ∞ η n N = 1 + H 1 . (449)
Remark 5.11. The tail identity (449) will be used below to control how an estimation error generated at one visit propagates to subsequent visits.
Notation. In the sequel, we adopt the following convenient notation, indexed explicitly by the episode number k k k .
( s h k , a h k ) (s_h^k,a_h^k) ( s h k , a h k ) : the state-action pair encountered and chosen at step h h h in the k k k -th episode;
N h k ( s , a ) N_h^k(s,a) N h k ( s , a ) : the visitation count N h ( s , a ) N_h(s,a) N h ( s , a ) for the ( s , a , h ) (s,a,h) ( s , a , h ) -tuple before episode k k k ;
b h k ( s , a ) ≥ 0 b_h^k(s,a) \ge0 b h k ( s , a ) ≥ 0 : the latest bonus term b h ( s , a ) b_h(s,a) b h ( s , a ) before episode k k k ;
P h k ∈ { 0 , 1 } 1 × S P_h^k \in\{0,1\}^{1\times S} P h k ∈ { 0 , 1 } 1 × S : the empirical transition at time step h h h in the k k k -th episode, namely,
∀ s ∈ S : P h k ( s ) = 1 { s = s h + 1 k } . (450) \forall s \in\mathcal{S}: \qquad P_h^k(s)=\mathbb{1}\{s=s_{h+1}^k\}. \tag*{(450)} ∀ s ∈ S : P h k ( s ) = 1 { s = s h + 1 k } . (450)
Q h k ( s , a ) Q_h^k(s,a) Q h k ( s , a ) and V h k ( s ) V_h^k(s) V h k ( s ) : the latest Q h u c b ( s , a ) Q_h^{\mathrm{ucb}}(s,a) Q h ucb ( s , a ) and V h u c b ( s ) V_h^{\mathrm{ucb}}(s) V h ucb ( s ) before episode k k k .
Also, throughout this analysis, we denote
ι : = log ( S A K H / δ ) . (451) \iota:= \log(SAKH/\delta). \tag*{(451)} ι := log ( S A K H / δ ) . (451)
Step 1: a recursive expression for Q-estimation errors. We first unroll the Q-learning updates to derive a recursive expression for the Q iterates. For any ( s , a , h , k ) ∈ S × A × [ H ] × [ K ] (s,a,h,k) \in\mathcal{S} \times\mathcal{A} \times[H] \times[K] ( s , a , h , k ) ∈ S × A × [ H ] × [ K ] , suppose that ( s , a ) (s,a) ( s , a ) was previously visited t t t times at step h h h , and denote by k 1 < ⋯ < k t < k k_1 < \cdots< k_t < k k 1 < ⋯ < k t < k these previous episodes. Armed with the notation (5.67), unrolling the corresponding updates then yields
Q h k ( s , a ) = η 0 t H + ∑ i = 1 t η i t { r h ( s , a ) + V h + 1 k i ( s h + 1 k i ) + b h k i ( s , a ) } . (452) Q_h^k(s,a)=\eta_0^tH+\sum_{i=1}^{t}\eta_i^t\left\{r_h(s,a)+V_{h+1}^{k_i}(s_{h+1}^{k_i})+b_h^{k_i}(s,a)\right\}. \tag*{(452)} Q h k ( s , a ) = η 0 t H + i = 1 ∑ t η i t { r h ( s , a ) + V h + 1 k i ( s h + 1 k i ) + b h k i ( s , a ) } . (452)
where we use the initialization Q h 0 = H Q^0_h = H Q h 0 = H . Combining this with the Bellman equation Q h ⋆ ( s , a ) = r h ( s , a ) + P s , a , h V h + 1 ⋆ Q^\star_h(s,a) = r_h(s,a) + P_{s,a,h}V^\star_{h+1} Q h ⋆ ( s , a ) = r h ( s , a ) + P s , a , h V h + 1 ⋆ results in
Q h k ( s , a ) − Q h ⋆ ( s , a ) = η 0 t ( H − Q h ⋆ ( s , a ) ) + ∑ i = 1 t η i t { ( V h + 1 k i − V h + 1 ⋆ ) ( s h + 1 k i ) + ( P h k i − P s , a , h ) V h + 1 ⋆ + b h k i ( s , a ) } . (453) Q^k_h(s,a) - Q^\star_h(s,a) = \eta^t_0(H-Q^\star_h(s,a)) + \sum_{i=1}^{t}\eta^t_i\left\{(V^{k_i}_{h+1}-V^\star_{h+1})(s^{k_i}_{h+1}) + (P^{k_i}_h-P_{s,a,h})V^\star_{h+1} + b^{k_i}_h(s,a)\right\}. \tag*{(453)} Q h k ( s , a ) − Q h ⋆ ( s , a ) = η 0 t ( H − Q h ⋆ ( s , a )) + i = 1 ∑ t η i t { ( V h + 1 k i − V h + 1 ⋆ ) ( s h + 1 k i ) + ( P h k i − P s , a , h ) V h + 1 ⋆ + b h k i ( s , a ) } . (453)
where we have used the identity η 0 t + ∑ i = 1 t η i t = 1 \eta^t_0+\sum_{i=1}^{t}\eta^t_i=1 η 0 t + ∑ i = 1 t η i t = 1 (cf. (446) ). This expression decomposes the estimation error into a weighted combination of several components: the influence of the initial estimate, the propagated error from previously updated value functions, the statistical fluctuations of the observed transitions, and the exploration bonuses.
Step 2: optimism. We next establish that the Q-estimates are “optimistic,” namely, they can upper bound the optimal Q-function. This property, as stated in the lemma below, parallels Lemma 5.5 for the model-based approach. Here, t t t and the indices k 1 < ⋯ < k t k_1 < \cdots< k_t k 1 < ⋯ < k t are defined as in Step 1.
Lemma 5.18. Suppose that the constant c b c_b c b in the bonus choice (439) is large enough. Then with probability at least 1 − δ 1-\delta 1 − δ ,
0 ≤ ( Q h k − Q h ⋆ ) ( s , a ) ≤ η 0 t H + ∑ i = 1 t η i t ( V h + 1 k i − V h + 1 ⋆ ) ( s h + 1 k i ) + β t , (454) 0 \leq(Q^k_h-Q^\star_h)(s,a) \leq\eta^t_0H+\sum_{i=1}^{t}\eta^t_i(V^{k_i}_{h+1}-V^\star_{h+1})(s^{k_i}_{h+1})+\beta_t, \tag*{(454)} 0 ≤ ( Q h k − Q h ⋆ ) ( s , a ) ≤ η 0 t H + i = 1 ∑ t η i t ( V h + 1 k i − V h + 1 ⋆ ) ( s h + 1 k i ) + β t , (454) holds simultaneously for all ( s , a , h , k ) ∈ S × A × [ H ] × [ K ] (s,a,h,k)\in\mathcal{S}\times\mathcal{A}\times[H]\times[K] ( s , a , h , k ) ∈ S × A × [ H ] × [ K ] , where β t \beta_t β t is some residual term obeying
β 0 = 0 , β t ≤ C H 3 ι t for t ≥ 1 (455) \beta_0=0,\qquad\beta_t\leq C\sqrt{\frac{H^3\iota}{t}}\quad\text{for }t\geq1 \tag*{(455)} β 0 = 0 , β t ≤ C t H 3 ι for t ≥ 1 (455) for some absolute constant C > 0 C>0 C > 0 . Then with probability exceeding 1 − δ 1-\delta 1 − δ , for all ( s , a , h ) (s,a,h) ( s , a , h ) we have
Q h k ( s , a ) ≥ Q h ⋆ ( s , a ) and V h k ( s ) ≥ V h ⋆ ( s ) . (456) Q^k_h(s,a)\geq Q^\star_h(s,a)\qquad\text{and}\qquad V^k_h(s)\geq V^\star_h(s). \tag*{(456)} Q h k ( s , a ) ≥ Q h ⋆ ( s , a ) and V h k ( s ) ≥ V h ⋆ ( s ) . (456)
The proof of this lemma is deferred to § 8.20 . Intuitively, when the exploration bonus is chosen sufficiently large, it compensates for the statistical uncertainty in the Bellman updates and thereby ensures that the resulting Q-estimates remain optimistic. The bound on β t \beta_t β t in (455) is tailored to the Hoeffding-style exploration bonus in (439) and can be improved by adopting more refined bonuses, such as Bernstein-style bonuses.
Step 3: regret decomposition. We now proceed with the regret analysis. Similar to the analysis of UCB-VI, we first replace the unknown optimal value by the optimistic value estimate: in view of Lemma 5.18 (see (456) ), with probability at least 1 − δ 1-\delta 1 − δ ,
Regret ( K ) : = ∑ k = 1 K ( V 1 ⋆ ( s 1 k ) − V 1 π k ( s 1 k ) ) ≤ ∑ k = 1 K ( V 1 ⋆ ( s 1 k ) − V 1 π k ( s 1 k ) ) . (457) \operatorname{Regret}(K):=\sum_{k=1}^{K}\left(V^\star_1(s^k_1)-V^{\pi^k}_1(s^k_1)\right)\leq\sum_{k=1}^{K}\left(V^\star_1(s^k_1)-V^{\pi^k}_1(s^k_1)\right). \tag*{(457)} Regret ( K ) := k = 1 ∑ K ( V 1 ⋆ ( s 1 k ) − V 1 π k ( s 1 k ) ) ≤ k = 1 ∑ K ( V 1 ⋆ ( s 1 k ) − V 1 π k ( s 1 k ) ) . (457)
Now, with a slight abuse of notation, let t = N h k ( s h k , a h k ) t=N^k_h(s^k_h,a^k_h) t = N h k ( s h k , a h k ) represent the number of times ( s h k , a h k ) (s^k_h,a^k_h) ( s h k , a h k ) has previously been visited at step h h h , and let k 1 < ⋯ < k t < k k_1<\cdots<k_t<k k 1 < ⋯ < k t < k denote the indices of the corresponding episodes. Since policy π k \pi^k π k is chosen greedily w.r.t. Q h k Q^k_h Q h k , we can use (443) to show that
( V h k − V h π k ) ( s h k ) ≤ max a Q h k ( s h k , a ) − V h π k ( s h k ) = Q h k ( s h k , a h k ) − V h π k ( s h k ) ≤ ( Q h k − Q h π k ) ( s h k , a h k ) = ( Q h k − Q h ⋆ ) ( s h k , a h k ) + P s h k , a h k , h ( V h + 1 ⋆ − V h + 1 π k ) , (458) \begin{aligned}
(V^k_h-V^{\pi^k}_h)(s^k_h)\leq\max_a Q^k_h(s^k_h,a)-V^{\pi^k}_h(s^k_h)=Q^k_h(s^k_h,a^k_h)-V^{\pi^k}_h(s^k_h) \\
&\leq(Q_h^k-Q_h^{\pi^k})(s_h^k,a_h^k)=(Q_h^k-Q_h^\star)(s_h^k,a_h^k)+P_{s_h^k,a_h^k,h}(V_{h+1}^\star-V_{h+1}^{\pi^k}), \tag*{(458)}
\end{aligned} ( V h k − V h π k ) ( s h k ) ≤ a max Q h k ( s h k , a ) − V h π k ( s h k ) = Q h k ( s h k , a h k ) − V h π k ( s h k ) ≤ ( Q h k − Q h π k ) ( s h k , a h k ) = ( Q h k − Q h ⋆ ) ( s h k , a h k ) + P s h k , a h k , h ( V h + 1 ⋆ − V h + 1 π k ) , (458)
where the last relation is valid since, according to the Bellman equations,
Q h ⋆ ( s , a ) − Q h π ( s , a ) = P s , a , h V h + 1 ⋆ − P s , a , h V h + 1 π . Q_h^\star(s,a)-Q_h^{\pi}(s,a)=P_{s,a,h}V_{h+1}^\star-P_{s,a,h}V_{h+1}^{\pi}. Q h ⋆ ( s , a ) − Q h π ( s , a ) = P s , a , h V h + 1 ⋆ − P s , a , h V h + 1 π .
Combine this with Lemma 5.18 to show that
( V h k − V h π k ) ( s h k ) ≤ η 0 t H + ∑ i = 1 t η i t ( V h + 1 k i − V h + 1 ⋆ ) ( s h + 1 k i ) + β t + P s h k , a h k , h ( V h + 1 ⋆ − V h + 1 π k ) = η 0 t H + ∑ i = 1 t η i t ( V h + 1 k i − V h + 1 ⋆ ) ( s h + 1 k i ) + β t + ( V h + 1 ⋆ − V h + 1 π k ) ( s h + 1 k ) + ( P s h k , a h k , h − P h k ) ( V h + 1 ⋆ − V h + 1 π k ) , (459) \begin{aligned}
(V_h^k-V_h^{\pi^k})(s_h^k)
&\leq\eta_0^t H+\sum_{i=1}^{t}\eta_i^t(V_{h+1}^{k_i}-V_{h+1}^\star)(s_{h+1}^{k_i})+\beta_t+P_{s_h^k,a_h^k,h}(V_{h+1}^\star-V_{h+1}^{\pi^k})\\
&=\eta_0^t H+\sum_{i=1}^{t}\eta_i^t(V_{h+1}^{k_i}-V_{h+1}^\star)(s_{h+1}^{k_i})+\beta_t\\
&\quad+(V_{h+1}^\star-V_{h+1}^{\pi^k})(s_{h+1}^k)+(P_{s_h^k,a_h^k,h}-P_h^k)(V_{h+1}^\star-V_{h+1}^{\pi^k}), \tag*{(459)}
\end{aligned} ( V h k − V h π k ) ( s h k ) ≤ η 0 t H + i = 1 ∑ t η i t ( V h + 1 k i − V h + 1 ⋆ ) ( s h + 1 k i ) + β t + P s h k , a h k , h ( V h + 1 ⋆ − V h + 1 π k ) = η 0 t H + i = 1 ∑ t η i t ( V h + 1 k i − V h + 1 ⋆ ) ( s h + 1 k i ) + β t + ( V h + 1 ⋆ − V h + 1 π k ) ( s h + 1 k ) + ( P s h k , a h k , h − P h k ) ( V h + 1 ⋆ − V h + 1 π k ) , (459)
where the last term in (459) is a martingale difference term. Importantly, the upper bound in (459) converts the regret at step h h h into a one-step propagation of value estimation error from step h + 1 h+1 h + 1 , together with a martingale fluctuation caused by the sampled transition.
Step 4: summing one-step recursion. The next step is to aggregate the one-step recursion (459) over episodes. At a high level, we control the bonus terms by summing their learning-rate-weighted contributions and regroup the propagated value estimation errors according to when they were generated. As it turns out, the tail-sum property (449) of the learning rates is crucial, as it ensures that each past error has only a bounded total influence on future updates.
To be precise, we introduce more notation. For each ( k , h ) (k,h) ( k , h ) , denote by t ( k , h ) = N h k ( s h k , a h k ) t(k,h)=N_h^k(s_h^k,a_h^k) t ( k , h ) = N h k ( s h k , a h k ) the number of episodes prior to episode k k k in which ( s h k , a h k ) (s_h^k,a_h^k) ( s h k , a h k ) was visited at step h h h , and let k 1 ( k , h ) < ⋯ < k t ( k , h ) < k k_1(k,h)<\cdots<k_{t(k,h)}<k k 1 ( k , h ) < ⋯ < k t ( k , h ) < k denote the indices of these episodes. We now handle the first three terms in (459) separately.
∑ k = 1 K η 0 t ( k , h ) H = ∑ k = 1 K η 0 N h k ( s h k , a h k ) H = ∑ k = 1 K H ⋅ 1 { N h k ( s h k , a h k ) = 0 } ≤ S A H . (460) \sum_{k=1}^{K}\eta_0^{t(k,h)}H=\sum_{k=1}^{K}\eta_0^{N_h^k(s_h^k,a_h^k)}H=\sum_{k=1}^{K}H\cdot\mathbb{1}\{N_h^k(s_h^k,a_h^k)=0\}\leq SAH. \tag*{(460)} k = 1 ∑ K η 0 t ( k , h ) H = k = 1 ∑ K η 0 N h k ( s h k , a h k ) H = k = 1 ∑ K H ⋅ 1 { N h k ( s h k , a h k ) = 0 } ≤ S A H . (460)
Regarding the third term on the right-hand side of (459) , observe that for each ( s , a ) (s,a) ( s , a ) , the visitation count takes the values 0 , 1 , … , N h K + 1 ( s , a ) − 1 0,1,\ldots,N_h^{K+1}(s,a)-1 0 , 1 , … , N h K + 1 ( s , a ) − 1 over its successive visits. Thus, it follows from Lemma 5.18 (see (455) ) that
∑ k = 1 K β t ( k , h ) = ∑ k = 1 K β N h k ( s h k , a h k ) = ∑ s , a ∑ n = 1 N h K + 1 ( s , a ) β n ≤ C ∑ s , a ∑ n = 1 N h K + 1 ( s , a ) H 3 ι n ≤ 2 C H 3 ι ∑ s , a N h K + 1 ( s , a ) ≤ 2 C H 3 ι ∑ s , a 1 ∑ s , a N h K + 1 ( s , a ) = 2 C H 3 S A K ι , (461) \begin{aligned}
\sum_{k=1}^{K}\beta_{t(k,h)}
&=\sum_{k=1}^{K}\beta_{N_h^k(s_h^k,a_h^k)}
=\sum_{s,a}\sum_{n=1}^{N_h^{K+1}(s,a)}\beta_n
\leq C\sum_{s,a}\sum_{n=1}^{N_h^{K+1}(s,a)}\sqrt{\frac{H^3\iota}{n}}\\
&\leq2C\sqrt{H^3\iota}\sum_{s,a}\sqrt{N_h^{K+1}(s,a)}
\leq2C\sqrt{H^3\iota}\sqrt{\sum_{s,a}1}\sqrt{\sum_{s,a}N_h^{K+1}(s,a)}
=2C\sqrt{H^3SAK\iota}, \tag*{(461)}
\end{aligned} k = 1 ∑ K β t ( k , h ) = k = 1 ∑ K β N h k ( s h k , a h k ) = s , a ∑ n = 1 ∑ N h K + 1 ( s , a ) β n ≤ C s , a ∑ n = 1 ∑ N h K + 1 ( s , a ) n H 3 ι ≤ 2 C H 3 ι s , a ∑ N h K + 1 ( s , a ) ≤ 2 C H 3 ι s , a ∑ 1 s , a ∑ N h K + 1 ( s , a ) = 2 C H 3 S A K ι , (461)
where (i) uses the elementary inequality ∑ n = 1 N 1 n ≤ 2 N \sum_{n=1}^{N}\frac{1}{\sqrt{n}}\leq2\sqrt{N} ∑ n = 1 N n 1 ≤ 2 N , (ii) follows from the Cauchy–Schwarz inequality, and (iii) relies on the fact that ∑ s , a N h K + 1 ( s , a ) = K \sum_{s,a}N_h^{K+1}(s,a)=K ∑ s , a N h K + 1 ( s , a ) = K .
We next control the second term on the right-hand side of (459) . For each state-action pair ( s , a ) (s,a) ( s , a ) , let τ s , a , h ( i ) \tau_{s,a,h}(i) τ s , a , h ( i ) denote the episode index of the i i i -th visit to ( s , a ) (s,a) ( s , a ) at step h h h . Thus, whenever N h k ( s h k , a h k ) ≥ 1 N_h^{k}(s_h^k,a_h^k) \ge1 N h k ( s h k , a h k ) ≥ 1 , we have k i ( k , h ) = τ s h k , a h k , h ( i ) k_i(k,h)=\tau_{s_h^k,a_h^k,h}(i) k i ( k , h ) = τ s h k , a h k , h ( i ) . For brevity, define
Δ h + 1 ℓ : = ( V h + 1 ℓ − V h + 1 ⋆ ) ( s h + 1 ℓ ) ≥ 0 , \Delta_{h+1}^{\ell}:=(V_{h+1}^{\ell}-V_{h+1}^{\star})(s_{h+1}^{\ell})\ge0, Δ h + 1 ℓ := ( V h + 1 ℓ − V h + 1 ⋆ ) ( s h + 1 ℓ ) ≥ 0 ,
where the nonnegativity is due to optimism (see Lemma 5.18). For any given ( s , a , h ) (s,a,h) ( s , a , h ) , the contribution made by the i i i -th visit to ( s , a , h ) (s,a,h) ( s , a , h ) appears in every later episode k k k for which ( s h k , a h k ) = ( s , a ) (s_h^k,a_h^k)=(s,a) ( s h k , a h k ) = ( s , a ) ; in particular, if the i i i -th visit occurs in episode τ s , a , h ( i ) \tau_{s,a,h}(i) τ s , a , h ( i ) , then its coefficient in a later visit in episode k k k is η i N h k ( s , a ) \eta_i^{N_h^k(s,a)} η i N h k ( s , a ) . Therefore, regrouping the summation according to the episode in which each term is generated, we obtain
∑ k = 1 K ∑ i = 1 t ( k , h ) η i t ( k , h ) ( V h + 1 k i ( k , h ) − V h + 1 ⋆ ) ( s h + 1 k i ( k , h ) ) = ∑ s , a ∑ i = 1 N h K + 1 ( s , a ) Δ h + 1 τ s , a , h ( i ) ∑ k : ( s h k , a h k ) = ( s , a ) , N h k ( s , a ) ≥ i η i N h k ( s , a ) ≤ ( 1 + 1 H ) ∑ s , a ∑ i = 1 N h K + 1 ( s , a ) Δ h + 1 τ s , a , h ( i ) , (462) \begin{aligned}
\sum_{k=1}^{K}\sum_{i=1}^{t(k,h)}\eta_i^{t(k,h)}(V_{h+1}^{k_i(k,h)}-V_{h+1}^{\star})(s_{h+1}^{k_i(k,h)})
&=\sum_{s,a}\sum_{i=1}^{N_h^{K+1}(s,a)}\Delta_{h+1}^{\tau_{s,a,h}(i)}
\sum_{\substack{k:(s_h^k,a_h^k)=(s,a),\\N_h^k(s,a)\ge i}}\eta_i^{N_h^k(s,a)}\\
&\le\left(1+\frac{1}{H}\right)\sum_{s,a}\sum_{i=1}^{N_h^{K+1}(s,a)}\Delta_{h+1}^{\tau_{s,a,h}(i)}, \tag*{(462)}
\end{aligned} k = 1 ∑ K i = 1 ∑ t ( k , h ) η i t ( k , h ) ( V h + 1 k i ( k , h ) − V h + 1 ⋆ ) ( s h + 1 k i ( k , h ) ) = s , a ∑ i = 1 ∑ N h K + 1 ( s , a ) Δ h + 1 τ s , a , h ( i ) k : ( s h k , a h k ) = ( s , a ) , N h k ( s , a ) ≥ i ∑ η i N h k ( s , a ) ≤ ( 1 + H 1 ) s , a ∑ i = 1 ∑ N h K + 1 ( s , a ) Δ h + 1 τ s , a , h ( i ) , (462)
where the last inequality is valid since Δ h + 1 ℓ ≥ 0 \Delta_{h+1}^{\ell}\ge0 Δ h + 1 ℓ ≥ 0 , and according to (449) ,
∑ k : ( s h k , a h k ) = ( s , a ) , N h k ( s , a ) ≥ i η i N h k ( s , a ) ≤ ∑ N = i ∞ η i N = 1 + 1 H . \sum_{\substack{k:(s_h^k,a_h^k)=(s,a),\\N_h^k(s,a)\ge i}}\eta_i^{N_h^k(s,a)}
\le\sum_{N=i}^{\infty}\eta_i^N=1+\frac{1}{H}. k : ( s h k , a h k ) = ( s , a ) , N h k ( s , a ) ≥ i ∑ η i N h k ( s , a ) ≤ N = i ∑ ∞ η i N = 1 + H 1 .
Indeed, this is a key benefit of the learning rate schedule (440) : although an estimation error may propagate through many future updates, its total effective weight remains well controlled. Further, observe that { ( s , a , i ) : 1 ≤ i ≤ N h K + 1 ( s , a ) } \{(s,a,i):1\le i\le N_h^{K+1}(s,a)\} {( s , a , i ) : 1 ≤ i ≤ N h K + 1 ( s , a )} are in one-to-one correspondence with the K K K visits made at step h h h , and therefore,
∑ s , a ∑ i = 1 N h K + 1 ( s , a ) Δ h + 1 τ s , a , h ( i ) = ∑ k = 1 K ( V h + 1 k − V h + 1 ⋆ ) ( s h + 1 k ) . \sum_{s,a}\sum_{i=1}^{N_h^{K+1}(s,a)}\Delta_{h+1}^{\tau_{s,a,h}(i)}
=\sum_{k=1}^{K}(V_{h+1}^{k}-V_{h+1}^{\star})(s_{h+1}^{k}). s , a ∑ i = 1 ∑ N h K + 1 ( s , a ) Δ h + 1 τ s , a , h ( i ) = k = 1 ∑ K ( V h + 1 k − V h + 1 ⋆ ) ( s h + 1 k ) .
Combine it with (462) to reach
∑ k = 1 K ∑ i = 1 t ( k , h ) η i t ( k , h ) ( V h + 1 k i ( k , h ) − V h + 1 ⋆ ) ( s h + 1 k i ) ≤ ( 1 + 1 H ) ∑ k = 1 K ( V h + 1 k − V h + 1 ⋆ ) ( s h + 1 k ) . (463) \sum_{k=1}^{K}\sum_{i=1}^{t(k,h)}\eta_i^{t(k,h)}(V_{h+1}^{k_i(k,h)}-V_{h+1}^{\star})(s_{h+1}^{k_i})
\le\left(1+\frac{1}{H}\right)\sum_{k=1}^{K}(V_{h+1}^{k}-V_{h+1}^{\star})(s_{h+1}^{k}). \tag*{(463)} k = 1 ∑ K i = 1 ∑ t ( k , h ) η i t ( k , h ) ( V h + 1 k i ( k , h ) − V h + 1 ⋆ ) ( s h + 1 k i ) ≤ ( 1 + H 1 ) k = 1 ∑ K ( V h + 1 k − V h + 1 ⋆ ) ( s h + 1 k ) . (463)
Step 5: recursion across horizon. We now turn the bounds from Step 4 into a recursion over the horizon. Substituting (460) , (461) and (463) into (459) , we reach
∑ k = 1 K ( V h k − V h π k ) ( s h k ) ≤ S A H + ( 1 + 1 H ) ∑ k = 1 K ( V h + 1 k − V h + 1 ⋆ ) ( s h + 1 k ) + 2 C H 3 S A K + ∑ k = 1 K ( V h + 1 ⋆ − V h + 1 π k ) ( s h + 1 k ) + ∑ k = 1 K ( P s h k , a h k , h − P h k ) ( V h + 1 k − V h + 1 π k ) ≤ S A H + ( 1 + 1 H ) ∑ k = 1 K ( V h + 1 k − V h + 1 π k ) ( s h + 1 k ) + 2 C H 3 S A K ι + ∑ k = 1 K ( P s h k a h k , h − P h k ) ( V h + 1 ⋆ − V h + 1 π k ) , (464) \begin{aligned}
\sum_{k=1}^{K}(V_h^k-V_h^{\pi^k})(s_h^k)
&\le SAH+\left(1+\frac{1}{H}\right)\sum_{k=1}^{K}(V_{h+1}^k-V_{h+1}^{\star})(s_{h+1}^k)+2C\sqrt{H^3SAK}\\
&\quad+\sum_{k=1}^{K}(V_{h+1}^{\star}-V_{h+1}^{\pi^k})(s_{h+1}^k)+\sum_{k=1}^{K}(P_{s_h^k,a_h^k,h}-P_h^k)(V_{h+1}^{k}-V_{h+1}^{\pi^k}) \\
&\le SAH+\left(1+\frac{1}{H}\right)\sum_{k=1}^{K}\left(V_{h+1}^{k}-V_{h+1}^{\pi^{k}}\right)\left(s_{h+1}^{k}\right)+2C\sqrt{H^{3}SAK\iota}
+\sum_{k=1}^{K}\left(P_{s_{h}^{k}a_{h}^{k},h}-P_{h}^{k}\right)\left(V_{h+1}^{\star}-V_{h+1}^{\pi^{k}}\right), \tag*{(464)}
\end{aligned} k = 1 ∑ K ( V h k − V h π k ) ( s h k ) ≤ S A H + ( 1 + H 1 ) k = 1 ∑ K ( V h + 1 k − V h + 1 ⋆ ) ( s h + 1 k ) + 2 C H 3 S A K + k = 1 ∑ K ( V h + 1 ⋆ − V h + 1 π k ) ( s h + 1 k ) + k = 1 ∑ K ( P s h k , a h k , h − P h k ) ( V h + 1 k − V h + 1 π k ) ≤ S A H + ( 1 + H 1 ) k = 1 ∑ K ( V h + 1 k − V h + 1 π k ) ( s h + 1 k ) + 2 C H 3 S A K ι + k = 1 ∑ K ( P s h k a h k , h − P h k ) ( V h + 1 ⋆ − V h + 1 π k ) , (464)
where we have used the fact that V h + 1 ⋆ ≥ V h + 1 π k V_{h+1}^{\star}\ge V_{h+1}^{\pi^{k}} V h + 1 ⋆ ≥ V h + 1 π k . Iterating this inequality from h = 1 h=1 h = 1 to h = H h=H h = H propagates the next-step error all the way to the terminal step:
∑ k = 1 K ( V 1 k − V 1 π k ) ( s 1 k ) ≤ O ( H 2 S A + H 5 S A K ι + ∑ h = 1 H ∑ k = 1 K ( P s h k a h k , h − P h k ) ( V h + 1 ⋆ − V h + 1 π k ) ) , (465) \sum_{k=1}^{K}\left(V_{1}^{k}-V_{1}^{\pi^{k}}\right)\left(s_{1}^{k}\right)\le O\left(H^{2}SA+\sqrt{H^{5}SAK\iota}+\sum_{h=1}^{H}\sum_{k=1}^{K}\left(P_{s_{h}^{k}a_{h}^{k},h}-P_{h}^{k}\right)\left(V_{h+1}^{\star}-V_{h+1}^{\pi^{k}}\right)\right), \tag*{(465)} k = 1 ∑ K ( V 1 k − V 1 π k ) ( s 1 k ) ≤ O ( H 2 S A + H 5 S A K ι + h = 1 ∑ H k = 1 ∑ K ( P s h k a h k , h − P h k ) ( V h + 1 ⋆ − V h + 1 π k ) ) , (465)
where the recursion terminates at step H + 1 H+1 H + 1 since V H + 1 k = V H + 1 π k = 0 V_{H+1}^{k}=V_{H+1}^{\pi^{k}}=0 V H + 1 k = V H + 1 π k = 0 , and we have used the elementary inequality ( 1 + 1 / H ) H ≤ e (1+1/H)^{H}\le e ( 1 + 1/ H ) H ≤ e to bound the accumulated multiplicative factor.
It remains to bound the last term (i.e., the martingale term) in (465) . Given that ∥ V h + 1 ⋆ − V h + 1 π k ∥ ∞ ≤ H \lVert V_{h+1}^{\star}-V_{h+1}^{\pi^{k}}\rVert_{\infty}\le H ∥ V h + 1 ⋆ − V h + 1 π k ∥ ∞ ≤ H , the Azuma-Hoeffding inequality (Lemma 2.8 ) gives, with probability at least 1 − δ 1-\delta 1 − δ ,
∣ ∑ h = 1 H ∑ k = 1 K ( P s h k a h k , h − P h k ) ( V h + 1 ⋆ − V h + 1 π k ) ∣ ≤ O ( H H K ι ) . (466) \left|\sum_{h=1}^{H}\sum_{k=1}^{K}\left(P_{s_{h}^{k}a_{h}^{k},h}-P_{h}^{k}\right)\left(V_{h+1}^{\star}-V_{h+1}^{\pi^{k}}\right)\right|\le O(H\sqrt{HK\iota}). \tag*{(466)} h = 1 ∑ H k = 1 ∑ K ( P s h k a h k , h − P h k ) ( V h + 1 ⋆ − V h + 1 π k ) ≤ O ( H H K ι ) . (466)
Substitution into (465) gives
Regret ( K ) ≤ O ( H 5 S A K ι + H 2 S A ) . (467) \operatorname{Regret}(K)\le O\left(\sqrt{H^{5}SAK\iota}+H^{2}SA\right). \tag*{(467)} Regret ( K ) ≤ O ( H 5 S A K ι + H 2 S A ) . (467)
Step 6: removing the lower-order term. The preceding argument yields the intermediate bound (467) , which contains the additional term H 2 S A H^{2}SA H 2 S A . We now show that this term can be absorbed into the leading term. Since the regret per episode is at most H H H , we always have Regret ( K ) ≤ H K \operatorname{Regret}(K)\le HK Regret ( K ) ≤ H K . If K ≤ H 3 S A ι K\le H^{3}SA\iota K ≤ H 3 S A ι , then
Regret ( K ) ≤ H K ≤ H 5 S A K ι , \operatorname{Regret}(K)\le HK\le\sqrt{H^{5}SAK\iota}, Regret ( K ) ≤ H K ≤ H 5 S A K ι ,
so the desired bound follows directly from the trivial regret bound. If, on the other hand, K > H 3 S A ι K>H^{3}SA\iota K > H 3 S A ι , then it follows from (467) that
Regret ( K ) ≤ O ( H 5 S A K ι + H 2 S A ) = O ~ ( S A H 5 K ) . (468) \operatorname{Regret}(K)\le O\left(\sqrt{H^{5}SAK\iota}+H^{2}SA\right)=\widetilde{O}\left(\sqrt{SAH^{5}K}\right). \tag*{(468)} Regret ( K ) ≤ O ( H 5 S A K ι + H 2 S A ) = O ( S A H 5 K ) . (468)
Combining these two cases establishes Theorem 5.16 .
Minimax lower bounds
Having established intriguing regret upper bounds for online RL algorithms — particularly for the model-based approach — we now turn to the complementary question of whether these guarantees are statistically optimal. More precisely, we seek to understand the smallest regret that any online RL algorithm can achieve in the worst case over a suitable class of MDPs. Such minimax lower bounds provide fundamental performance benchmarks, allowing us to assess the statistical optimality of existing algorithms and determine whether the regret guarantees established in the previous sections can be further improved.
A line of research has been devoted to this question (e.g., Jaksch et al. (2010), Bartlett and Tewari (2009), Osband and Van Roy (2016), Jin et al. (2018), and Domingues et al. (2021)), building in part upon techniques originally developed for multi-armed bandits. In what follows, we present a minimax regret lower bound, originally established in Jin et al. (2018) and Domingues et al. (2021), which holds uniformly over all values of K K K .
Theorem 5.19. Consider any S ≥ 2 S \ge2 S ≥ 2 , A ≥ 3 A \ge3 A ≥ 3 , H ≥ 2 H \ge2 H ≥ 2 , and K ≥ 1 K \ge1 K ≥ 1 . There exists a collection C m d p \mathcal{C}_{\mathrm{mdp}} C mdp of finite-horizon nonstationary MDPs — each with S S S states, at most A A A actions per state, and horizon length H H H — together with an initial state distribution ρ ∈ Δ ( S ) \rho\in\Delta(S) ρ ∈ Δ ( S ) , such that
inf π ^ sup M ∈ C m d p E M [ ∑ k = 1 K ( V 1 ∗ ( ρ ) − V 1 π k ( ρ ) ) ] ≥ c 0 min { H 3 S A K , H K } \inf_{\hat{\pi}} \sup_{\mathcal{M} \in\mathcal{C}_{\mathrm{mdp}}} \mathbb{E}_{\mathcal{M}}\left[\sum_{k=1}^{K}\left(V_1^*(\rho)-V_1^{\pi^k}(\rho)\right)\right] \ge c_0 \min\left\{\sqrt{H^3SAK},HK\right\} π ^ inf M ∈ C mdp sup E M [ k = 1 ∑ K ( V 1 ∗ ( ρ ) − V 1 π k ( ρ ) ) ] ≥ c 0 min { H 3 S A K , H K } holds for some universal constant c 0 > 0 c_0 > 0 c 0 > 0 , where the expectation is taken w.r.t. the randomness under the MDP M \mathcal{M} M , while the infimum is over all online learning algorithms π ^ \hat{\pi} π ^ . Here, π ^ \hat{\pi} π ^ generates a sequence of policies π ^ = { π k } k = 1 K \hat{\pi} = \{\pi^k\}_{k=1}^K π ^ = { π k } k = 1 K , and each π k \pi^k π k may depend arbitrarily on the trajectories collected before episode k k k (but not on any observations from episode k k k or later episodes).
Since a regret of at most H K HK H K is trivially achievable — because the cumulative reward collected over K K K episodes cannot exceed H K HK H K — we shall often drop the linear term and simply write
Ω ( S A H 3 K ) if K ≥ S A H . (469) \Omega\left(\sqrt{SAH^3K}\right) \qquad\text{if } K \ge SAH. \tag*{(469)} Ω ( S A H 3 K ) if K ≥ S A H . (469)
Importantly, when combined with the regret upper bound for the MVP algorithm in Theorem Theorem 5.10 , this lower bound rigorously confirms the minimax optimality of model-based online RL (modulo some logarithmic factors).
Although the lower bound itself was originally established in Jin et al. (2018) and Domingues et al. (2021), the proof presented here follows a different route. Rather than adapting the existing arguments in Jin et al. (2018) and Domingues et al. (2021), we develop the proof from first principles using information-theoretic techniques. The argument proceeds in two stages. We first introduce a closely related bandit problem and use it to establish the minimax lower bound for a carefully constructed family of two-state MDPs (§ 8.13 ). We then lift this result to general S S S -state MDPs (§ 8.14 ). Besides providing an alternative proof, this derivation makes explicit the information-theoretic connection between multi-armed bandits and online RL.
Analysis: lower bound for a simpler case with S = 2 S = 2 S = 2
We begin by considering the simpler setting with S = 2 S = 2 S = 2 . Besides being considerably easier to analyze, this construction captures the essential statistical bottleneck of online RL. In § 8.14 , we will extend this construction to general state spaces by combining multiple independent copies of the two-state MDPs.
To be precise, our goal in this subsection is to construct a family C m d p s u b \mathcal{C}_{\mathrm{mdp}}^{\mathrm{sub}} C mdp sub of MDPs — each with S = 2 S = 2 S = 2 states, at most A A A actions per state (A ≥ 3 A \ge3 A ≥ 3 ), and horizon length H ≥ 2 H \ge2 H ≥ 2 — as well as an initial state distribution ρ \rho ρ , such that
inf π ^ max M ∈ C m d p s u b E M [ ∑ k = 1 K ( V 1 ∗ ( ρ ) − V 1 π k ( ρ ) ) ] ≥ c l b min { H 3 A K , H K } (470) \inf_{\hat{\pi}} \max_{\mathcal{M} \in\mathcal{C}_{\mathrm{mdp}}^{\mathrm{sub}}} \mathbb{E}_{\mathcal{M}}\left[\sum_{k=1}^{K}\left(V_1^*(\rho)-V_1^{\pi^k}(\rho)\right)\right] \ge c_{\mathrm{lb}} \min\left\{\sqrt{H^3AK},HK\right\} \tag*{(470)} π ^ inf M ∈ C mdp sub max E M [ k = 1 ∑ K ( V 1 ∗ ( ρ ) − V 1 π k ( ρ ) ) ] ≥ c lb min { H 3 A K , H K } (470)
holds for some universal constant c l b > 0 c_{\mathrm{lb}} > 0 c lb > 0 .
Construction of the MDPs. As before, we begin by constructing a collection C m d p s u b \mathcal{C}_{\mathrm{mdp}}^{\mathrm{sub}} C mdp sub of finite-horizon nonstationary MDPs. Each MDP M \mathcal{M} M in this collection is specified by the following components.
A 0 = { 0 } and A 1 = { 1 , … , A } . \mathcal{A}_0=\{0\}\qquad\text{and}\qquad\mathcal{A}_1=\{1,\ldots,A\}. A 0 = { 0 } and A 1 = { 1 , … , A } .
r h ( 0 , 0 ) = 0 and r h ( 1 , a ) = 1 for every a ∈ A 1 . (471) r_h(0,0)=0\qquad\text{and}\qquad r_h(1,a)=1\quad\text{for every }a\in\mathcal{A}_1. \tag*{(471)} r h ( 0 , 0 ) = 0 and r h ( 1 , a ) = 1 for every a ∈ A 1 . (471)
Optimal actions and probability transition kernel. Since the action space A 0 \mathcal{A}_0 A 0 associated with state 0 is a singleton, it suffices to specify the optimal action at state 1 for each step 1 ≤ h ≤ H 1\leq h\leq H 1 ≤ h ≤ H . For an MDP M ∈ C m d p s u b \mathcal{M}\in\mathcal{C}_{\mathrm{mdp}}^{\mathrm{sub}} M ∈ C mdp sub , we denote these optimal actions by
a 1 ⋆ , M , … , a H ⋆ , M ∈ A 1 (472) a_1^{\star,\mathcal{M}},\ldots,a_H^{\star,\mathcal{M}}\in\mathcal{A}_1 \tag*{(472)} a 1 ⋆ , M , … , a H ⋆ , M ∈ A 1 (472)
with a h ⋆ , M a_h^{\star,\mathcal{M}} a h ⋆ , M denoting the optimal action at state 1 and step h h h . The probability transition kernel of M \mathcal{M} M , denoted by P M P^{\mathcal{M}} P M , is then defined by
P h M ( 0 ∣ 0 , 0 ) = 1 , (473) P_h^{\mathcal{M}}(0\mid0,0)=1, \tag*{(473)} P h M ( 0 ∣ 0 , 0 ) = 1 , (473)
P h M ( 0 ∣ 1 , a ) = q a h ⋆ , M ( a ) , P h M ( 1 ∣ 1 , a ) = 1 − q a h ⋆ , M ( a ) (474) P_h^{\mathcal{M}}(0\mid1,a)=q_{a_h^{\star,\mathcal{M}}}(a),\qquad P_h^{\mathcal{M}}(1\mid1,a)=1-q_{a_h^{\star,\mathcal{M}}}(a) \tag*{(474)} P h M ( 0 ∣ 1 , a ) = q a h ⋆ , M ( a ) , P h M ( 1 ∣ 1 , a ) = 1 − q a h ⋆ , M ( a ) (474)
for every h ∈ [ H ] h\in[H] h ∈ [ H ] and a ∈ A 1 a\in\mathcal{A}_1 a ∈ A 1 . By construction, state 0 is absorbing and yields zero reward. Thus, the behavior of the MDP is largely determined by the dynamics at state 1. We will specify the function q a h ⋆ , M ( ⋅ ) q_{a_h^{\star,\mathcal{M}}}(\cdot) q a h ⋆ , M ( ⋅ ) next and verify that the resulting transition kernel is indeed consistent with the designated optimal actions (472) .
Initial state distribution. In what follows, we take the initial state distribution ρ \rho ρ to be concentrated on state 1, namely,
ρ ( s ) = { 1 , if s = 1 ; 0 , else . (475) \rho(s)=
\begin{cases}
1, & \text{if }s=1;\\
0, & \text{else}.
\end{cases} \tag*{(475)} ρ ( s ) = { 1 , 0 , if s = 1 ; else . (475)
It remains to specify the functions q a h ⋆ , M ( ⋅ ) q_{a_h^{\star,\mathcal{M}}}(\cdot) q a h ⋆ , M ( ⋅ ) appearing in (5.92). For notational simplicity, consider an arbitrary sequence of optimal actions a 1 ⋆ , … , a H ⋆ a_1^\star,\ldots,a_H^\star a 1 ⋆ , … , a H ⋆ . For each h ∈ [ H ] h\in[H] h ∈ [ H ] , we define
when a h ⋆ ≠ 1 : q a h ⋆ ( a ) = { 1 H − ε 0 H 2 , if a = a h ⋆ , 1 H − ε 0 2 H 2 , if a = 1 , 1 H , else , (476) \text{when }a_h^\star\ne1:\qquad
q_{a_h^\star}(a)=
\begin{cases}
\frac{1}{H}-\frac{\varepsilon_0}{H^2}, & \text{if }a=a_h^\star,\\
\frac{1}{H}-\frac{\varepsilon_0}{2H^2}, & \text{if }a=1,\\
\frac{1}{H}, & \text{else},
\end{cases} \tag*{(476)} when a h ⋆ = 1 : q a h ⋆ ( a ) = ⎩ ⎨ ⎧ H 1 − H 2 ε 0 , H 1 − 2 H 2 ε 0 , H 1 , if a = a h ⋆ , if a = 1 , else , (476)
when a h ⋆ = 1 : q a h ⋆ ( a ) = { 1 H − ε 0 2 H 2 , if a = a h ⋆ = 1 , 1 H , else . (477) \text{when }a_h^\star=1:\qquad
q_{a_h^\star}(a)=
\begin{cases}
\frac{1}{H}-\frac{\varepsilon_0}{2H^2}, & \text{if }a=a_h^\star=1,\\
\frac{1}{H}, & \text{else}.
\end{cases} \tag*{(477)} when a h ⋆ = 1 : q a h ⋆ ( a ) = { H 1 − 2 H 2 ε 0 , H 1 , if a = a h ⋆ = 1 , else . (477)
where the parameter 0 < ε 0 ≤ H / 2 0<\varepsilon_0\leq H/2 0 < ε 0 ≤ H /2 will be specified later (see (488) ). The consistency of the resulting transition kernel with the designated optimal actions is immediate. Indeed, since state 0 is absorbing and yields zero reward, maximizing the value function is equivalent to minimizing q h ⋆ ( a ) q_h^{\star}(a) q h ⋆ ( a ) over a ∈ A 1 a \in\mathcal{A}_1 a ∈ A 1 . By (5.94), this minimum is attained uniquely at a = a h ⋆ a = a_h^{\star} a = a h ⋆ for every step h h h .
We now specify the collection C m d p s u b \mathcal{C}_{\mathrm{mdp}}^{\mathrm{sub}} C mdp sub , where we allow the optimal actions to vary independently across steps. Specifically, for each
i = ( i 1 , … , i H ) ∈ [ A ] H , i = (i_1,\ldots,i_H) \in[A]^H, i = ( i 1 , … , i H ) ∈ [ A ] H ,
let M i \mathcal{M}_i M i denote the MDP constructed above with
a h ⋆ , M i = i h , 1 ≤ h ≤ H . (478) a_h^{\star,\mathcal{M}_i} = i_h, \qquad1 \le h \le H. \tag*{(478)} a h ⋆ , M i = i h , 1 ≤ h ≤ H . (478)
We then define
C m d p s u b : = { M i : i ∈ [ A ] H } ⟹ ∣ C m d p s u b ∣ = A H . (479) \mathcal{C}_{\mathrm{mdp}}^{\mathrm{sub}} := \{\mathcal{M}_i : i \in[A]^H\}
\qquad\Longrightarrow\qquad
\lvert\mathcal{C}_{\mathrm{mdp}}^{\mathrm{sub}}\rvert= A^H. \tag*{(479)} C mdp sub := { M i : i ∈ [ A ] H } ⟹ ∣ C mdp sub ∣ = A H . (479)
Proof strategy. The proof of the lower bound for this family proceeds in two main steps. First, we exploit a key feature of the above construction to relate the suboptimality gap of an arbitrary policy to its probability of selecting suboptimal actions, thereby reducing the regret analysis to controlling the probability of such action-selection errors. The following lemma formalizes this connection; its proof is deferred to § 8.21.1 .
Lemma 5.20. Suppose that H ≥ 2 H \ge2 H ≥ 2 . For every M ∈ C m d p s u b \mathcal{M} \in\mathcal{C}_{\mathrm{mdp}}^{\mathrm{sub}} M ∈ C mdp sub and every policy π \pi π , one has
V 1 ⋆ ( 1 ) − V 1 π ( 1 ) ≥ ∑ h = 1 H ε 0 ( H − h ) 16 H 2 ( 1 − π h ( a h ⋆ , M ∣ 1 ) ) . (480) V_1^{\star}(1) - V_1^{\pi}(1) \ge\sum_{h=1}^{H} \frac{\varepsilon_0(H-h)}{16H^2}\left(1-\pi_h(a_h^{\star,\mathcal{M}} \mid1)\right). \tag*{(480)} V 1 ⋆ ( 1 ) − V 1 π ( 1 ) ≥ h = 1 ∑ H 16 H 2 ε 0 ( H − h ) ( 1 − π h ( a h ⋆ , M ∣ 1 ) ) . (480)
Second, we develop a bandit statistical testing argument to show that the probability of selecting a suboptimal action remains bounded away from zero until sufficiently many observations are collected.
A detour: a lower bound on a bandit problem. To control the right-hand side of (480) , we take a brief detour to establish a lower bound in a closely related bandit setting. The formulation is the following.
Consider a collection C b a n d i t \mathcal{C}_{\mathrm{bandit}} C bandit consisting of A A A bandit instances, where each instance has A A A arms.
For i ∈ [ A ] i \in[A] i ∈ [ A ] , let B i \mathcal{B}_i B i denote the bandit instance whose unique optimal arm is a B i ⋆ = i a_{\mathcal{B}_i}^{\star}=i a B i ⋆ = i . The mean reward of arm a a a in B i \mathcal{B}_i B i is given by
r B i ( a ) = 1 − q i ( a ) , r_{\mathcal{B}_i}(a) = 1 - q_i(a), r B i ( a ) = 1 − q i ( a ) ,
where q i ( ⋅ ) q_i(\cdot) q i ( ⋅ ) is defined in (5.94). More concretely, pulling arm a a a in round t t t in the bandit instance B i \mathcal{B}_i B i yields an independent Bernoulli reward as
R a , t = { 1 , with probability 1 − q i ( a ) , 0 , else. R_{a,t} =
\begin{cases}
1, & \text{with probability } 1-q_i(a),\\
0, & \text{else.}
\end{cases} R a , t = { 1 , 0 , with probability 1 − q i ( a ) , else.
In what follows, let P B i \mathbb{P}_{\mathcal{B}_i} P B i denote the probability distribution under the bandit instance B i \mathcal{B}_i B i . As before, the learner interacts with the bandit instance in a sequential manner; at each round t t t , the learner selects an arm based only on the observations collected prior to round t t t .
We now apply the lower bound in Lemma 2.11 to develop a round-wise lower bound for this collection of bandit instances. The proof is deferred to § 8.22 .
Lemma 5.21. Consider the collection C b a n d i t \mathcal{C}_{\mathrm{bandit}} C bandit of bandit instances defined above. Suppose that A ≥ 3 A \ge3 A ≥ 3 , H ≥ 2 H \ge2 H ≥ 2 , and 0 < ε 0 ≤ H / 2 0 < \varepsilon_0 \le H/2 0 < ε 0 ≤ H /2 . Then for any round t ≤ H 3 A 18 ε 0 2 t \le\frac{H^3 A}{18\varepsilon_0^2} t ≤ 18 ε 0 2 H 3 A , one has
inf π ~ 1 A ∑ i = 1 A P B i ( a t + 1 ≠ i ) ≥ 1 3 . \inf_{\widetilde{\pi}} \frac{1}{A}\sum_{i=1}^{A}\mathbb{P}_{\mathcal{B}_i}(a_{t+1}\ne i) \ge\frac{1}{3}. π inf A 1 i = 1 ∑ A P B i ( a t + 1 = i ) ≥ 3 1 . where the infimum is taken over all online bandit algorithms π ~ \widetilde{\pi} π . Here, for a given algorithm π ~ \widetilde{\pi} π , we denote by { π t } t ≥ 1 \{\pi^t\}_{t\ge1} { π t } t ≥ 1 the resulting sequence of policies, where π t \pi^t π t depends only on observations collected prior to round t t t ; the arm a t a_t a t is then selected according to π t \pi^t π t .
In words, the lemma states that, at any round t + 1 t+1 t + 1 , the probability of selecting a suboptimal action — based on the observations collected through round t t t — is bounded away from zero. Importantly, this is a per-round guarantee that holds whenever t t t is not too large. As will become clear shortly, it is a little more convenient to develop this result in terms of the average probability of error rather than the minimax probability of error.
Regret analysis. We now return to the two-state MDP construction and leverage the preceding bandit lower bound in Lemma 5.21 . The key observation is that the information available for identifying the optimal action in the MDP has precisely the bandit structure studied above: under M \mathcal{M} M , whenever the learner takes action a a a at state 1 1 1 , the resulting transition corresponds to a Bernoulli observation with mean 1 − q a h ⋆ , M ( a ) 1-q_{a_h^{\star,\mathcal{M}}}(a) 1 − q a h ⋆ , M ( a ) , matching the reward distribution obtained by pulling arm a a a in B i \mathcal{B}_i B i if a h ⋆ , M = i a_h^{\star,\mathcal{M}}=i a h ⋆ , M = i . Hence, when choosing an action at any given step and episode, the learner faces the same statistical problem of identifying the unknown optimal action from sequential bandit feedback. Lemma 5.21 can therefore be used to lower bound the probability of selecting a suboptimal action in the MDP.
Before proceeding, we first introduce, for convenience of analysis, an auxiliary action variable in order to distinguish the action selected by the policy in state 1 from the action actually executed along the realized trajectory. For every episode k ∈ [ K ] k \in[K] k ∈ [ K ] and step h ∈ [ H ] h \in[H] h ∈ [ H ] , let
a ~ h k ∼ π h k ( ⋅ ∣ 1 ) (481) \widetilde{a}_h^k \sim\pi_h^k(\cdot\mid1) \tag*{(481)} a h k ∼ π h k ( ⋅ ∣ 1 ) (481)
conditional on what happens before episode k k k , regardless of whether the realized state s h k s_h^k s h k equals 1. Thus, a ~ h k \widetilde{a}_h^k a h k is an auxiliary action used only in the analysis. By construction,
E M [ 1 − π h k ( a h ⋆ , M ∣ 1 ) ] = P M ( a ~ h k ≠ a h ⋆ , M ) . (482) \mathbb{E}_{\mathcal{M}}\left[1-\pi_h^k(a_h^{\star,\mathcal{M}}\mid1)\right]=\mathbb{P}_{\mathcal{M}}\left(\widetilde{a}_h^k\ne a_h^{\star,\mathcal{M}}\right). \tag*{(482)} E M [ 1 − π h k ( a h ⋆ , M ∣ 1 ) ] = P M ( a h k = a h ⋆ , M ) . (482)
Conditional on the optimal actions at steps other than h h h , reaching state 1 at step h h h is independent of a h ⋆ , M a_h^{\star,\mathcal{M}} a h ⋆ , M , and the trajectory following the step-h h h transition provides no further information about it. Treating episodes that do not reach state 1 as dummy bandit rounds, the learner can therefore be viewed as a bandit algorithm run for k − 1 k-1 k − 1 rounds before episode k k k .
Now consider any online learning algorithm π ~ \widetilde{\pi} π . In view of (480) and (482) , one has
E M [ ∑ k = 1 K ( V 1 ⋆ ( 1 ) − V 1 π k ( 1 ) ) ] ≥ ∑ k = 1 K ∑ h = 1 H ε 0 ( H − h ) 16 H 2 P M ( a ~ h k ≠ a h ⋆ , M ) (483) \mathbb{E}_{\mathcal{M}}\left[\sum_{k=1}^{K}\left(V_1^{\star}(1)-V_1^{\pi^k}(1)\right)\right]
\ge\sum_{k=1}^{K}\sum_{h=1}^{H}\frac{\varepsilon_0(H-h)}{16H^2}\mathbb{P}_{\mathcal{M}}\left(\widetilde{a}_h^k\ne a_h^{\star,\mathcal{M}}\right) \tag*{(483)} E M [ k = 1 ∑ K ( V 1 ⋆ ( 1 ) − V 1 π k ( 1 ) ) ] ≥ k = 1 ∑ K h = 1 ∑ H 16 H 2 ε 0 ( H − h ) P M ( a h k = a h ⋆ , M ) (483)
for every M ∈ C m d p s u b \mathcal{M} \in\mathcal{C}_{\mathrm{mdp}}^{\mathrm{sub}} M ∈ C mdp sub . It follows that
inf π ^ max i ∈ [ A ] H E M i [ ∑ k = 1 K ( V 1 ∗ ( 1 ) − V 1 π k ( 1 ) ) ] ≥ inf π ^ 1 A H ∑ i ∈ [ A ] H E M i [ ∑ k = 1 K ( V 1 ∗ ( 1 ) − V 1 π k ( 1 ) ) ] ≥ ∑ k = 1 K ∑ h = 1 H ε 0 ( H − h ) 16 H 2 inf π ^ 1 A H ∑ i ∈ [ A ] H P M i ( a ~ h k ≠ i h ) . (484) \inf_{\widehat{\pi}} \max_{i \in[A]^H} \mathbb{E}_{\mathcal{M}_i}\left[\sum_{k=1}^{K}\left(V_1^*(1)-V_1^{\pi^k}(1)\right)\right] \ge\inf_{\widehat{\pi}} \frac{1}{A^H} \sum_{i \in[A]^H} \mathbb{E}_{\mathcal{M}_i}\left[\sum_{k=1}^{K}\left(V_1^*(1)-V_1^{\pi^k}(1)\right)\right]
\ge\sum_{k=1}^{K}\sum_{h=1}^{H} \frac{\varepsilon_0(H-h)}{16H^2}\inf_{\widehat{\pi}}\frac{1}{A^H}\sum_{i \in[A]^H}\mathbb{P}_{\mathcal{M}_i}\left(\widetilde{a}_h^k \ne i_h\right). \tag*{(484)} π inf i ∈ [ A ] H max E M i [ k = 1 ∑ K ( V 1 ∗ ( 1 ) − V 1 π k ( 1 ) ) ] ≥ π inf A H 1 i ∈ [ A ] H ∑ E M i [ k = 1 ∑ K ( V 1 ∗ ( 1 ) − V 1 π k ( 1 ) ) ] ≥ k = 1 ∑ K h = 1 ∑ H 16 H 2 ε 0 ( H − h ) π inf A H 1 i ∈ [ A ] H ∑ P M i ( a h k = i h ) . (484)
Next, we seek to lower bound (484) . For each h ∈ [ H ] h \in[H] h ∈ [ H ] , write
i − h : = ( i 1 , … , i h − 1 , i h + 1 , … , i H ) . i_{-h}:=(i_1,\ldots,i_{h-1},i_{h+1},\ldots,i_H). i − h := ( i 1 , … , i h − 1 , i h + 1 , … , i H ) .
Fix any i − h ∈ [ A ] H − 1 i_{-h} \in[A]^{H-1} i − h ∈ [ A ] H − 1 . Conditional on i − h i_{-h} i − h , the preceding construction induces a valid bandit algorithm for identifying i h i_h i h from at most k − 1 k-1 k − 1 informative observations. Therefore, applying Lemma 5.21 with t = k − 1 t=k-1 t = k − 1 gives
1 A ∑ i h = 1 A P M ( i h , i − h ) ( a ~ h k ≠ i h ) ≥ 1 3 , (485) \frac{1}{A}\sum_{i_h=1}^{A}\mathbb{P}_{\mathcal{M}_{(i_h,i_{-h})}}\left(\widetilde{a}_h^k \ne i_h\right)\ge\frac{1}{3}, \tag*{(485)} A 1 i h = 1 ∑ A P M ( i h , i − h ) ( a h k = i h ) ≥ 3 1 , (485)
provided that k − 1 ≤ A H 3 18 ε 0 2 k-1\le\frac{AH^3}{18\varepsilon_0^2} k − 1 ≤ 18 ε 0 2 A H 3 . Averaging (485) over i − h ∈ [ A ] H − 1 i_{-h}\in[A]^{H-1} i − h ∈ [ A ] H − 1 and then taking the infimum over π ^ \widehat{\pi} π yields
inf π ^ 1 A H ∑ i ∈ [ A ] H P M i ( a ~ h k ≠ i h ) ≥ 1 3 . (486) \inf_{\widehat{\pi}}\frac{1}{A^H}\sum_{i\in[A]^H}\mathbb{P}_{\mathcal{M}_i}\left(\widetilde{a}_h^k\ne i_h\right)\ge\frac{1}{3}. \tag*{(486)} π inf A H 1 i ∈ [ A ] H ∑ P M i ( a h k = i h ) ≥ 3 1 . (486)
Substituting (486) into (484) reveals that: if K − 1 ≤ A H 3 18 ε 0 2 K-1\le\frac{AH^3}{18\varepsilon_0^2} K − 1 ≤ 18 ε 0 2 A H 3 , then
inf π ^ max i ∈ [ A ] H E M i [ ∑ k = 1 K ( V 1 ∗ ( 1 ) − V 1 π k ( 1 ) ) ] ≥ 1 3 ∑ k = 1 K ∑ h = 1 H ε 0 ( H − h ) 16 H 2 = ε 0 K ( H − 1 ) 96 H ≥ ε 0 K 192 . (487) \inf_{\widehat{\pi}}\max_{i\in[A]^H}\mathbb{E}_{\mathcal{M}_i}\left[\sum_{k=1}^{K}\left(V_1^*(1)-V_1^{\pi^k}(1)\right)\right]\ge\frac{1}{3}\sum_{k=1}^{K}\sum_{h=1}^{H}\frac{\varepsilon_0(H-h)}{16H^2}=\frac{\varepsilon_0K(H-1)}{96H}\ge\frac{\varepsilon_0K}{192}. \tag*{(487)} π inf i ∈ [ A ] H max E M i [ k = 1 ∑ K ( V 1 ∗ ( 1 ) − V 1 π k ( 1 ) ) ] ≥ 3 1 k = 1 ∑ K h = 1 ∑ H 16 H 2 ε 0 ( H − h ) = 96 H ε 0 K ( H − 1 ) ≥ 192 ε 0 K . (487)
To finish up, choosing
ε 0 = min { H 3 A 18 K , H 2 } , (488) \varepsilon_0=\min\left\{\sqrt{\frac{H^3A}{18K}},\frac{H}{2}\right\}, \tag*{(488)} ε 0 = min { 18 K H 3 A , 2 H } , (488)
clearly satisfies K − 1 ≤ A H 3 18 ε 0 2 K-1\le\frac{AH^3}{18\varepsilon_0^2} K − 1 ≤ 18 ε 0 2 A H 3 and ε 0 ≤ H / 2 \varepsilon_0\le H/2 ε 0 ≤ H /2 . Substituting this choice into (487) , we arrive at
inf π ^ max M ∈ C m d p s u b E M [ ∑ k = 1 K ( V 1 ∗ ( 1 ) − V 1 π k ( 1 ) ) ] ≥ inf π ^ 1 ∣ C m d p s u b ∣ ∑ M ∈ C m d p s u b E M [ ∑ k = 1 K ( V 1 ∗ ( 1 ) − V 1 π k ( 1 ) ) ] ≥ ε 0 K 192 ≳ min { H 3 A K , H K } . (489) \inf_{\widehat{\pi}}\max_{\mathcal{M}\in\mathcal{C}_{\mathrm{mdp}}^{\mathrm{sub}}}\mathbb{E}_{\mathcal{M}}\left[\sum_{k=1}^{K}\left(V_1^*(1)-V_1^{\pi^k}(1)\right)\right]\ge\inf_{\widehat{\pi}}\frac{1}{\lvert\mathcal{C}_{\mathrm{mdp}}^{\mathrm{sub}}\rvert}\sum_{\mathcal{M}\in\mathcal{C}_{\mathrm{mdp}}^{\mathrm{sub}}}\mathbb{E}_{\mathcal{M}}\left[\sum_{k=1}^{K}\left(V_1^*(1)-V_1^{\pi^k}(1)\right)\right]
\ge\frac{\varepsilon_0K}{192}\gtrsim\min\left\{\sqrt{H^3AK},HK\right\}. \tag*{(489)} π inf M ∈ C mdp sub max E M [ k = 1 ∑ K ( V 1 ∗ ( 1 ) − V 1 π k ( 1 ) ) ] ≥ π inf ∣ C mdp sub ∣ 1 M ∈ C mdp sub ∑ E M [ k = 1 ∑ K ( V 1 ∗ ( 1 ) − V 1 π k ( 1 ) ) ] ≥ 192 ε 0 K ≳ min { H 3 A K , H K } . (489)
This taken together with the choice of ρ \rho ρ (cf. (475) ) establishes the advertised lower bound (470) for the case with S = 2 S=2 S = 2 .
Analysis: lower bound for general S S S (proof of Theorem 5.19)
We now demonstrate how the regret lower bound established in Section 5.6.1 for S = 2 S=2 S = 2 can be lifted to general S ≥ 2 S\ge2 S ≥ 2 . The key idea is to embed S / 2 S/2 S /2 independent copies of the two-state hard instances into a single MDP. Since each episode starts in exactly one of these copies, the K K K trajectories are effectively distributed across S / 2 S/2 S /2 independent learning problems, allowing us to apply the two-state lower bound separately to each sub-MDP. Throughout the proof, we assume, for simplicity, that S S S is even.
Construction of the MDPs and notation. We begin by constructing a family C m d p \mathcal{C}_{\mathrm{mdp}} C mdp of MDPs.
Each MDP M \mathcal{M} M in this family is composed of S / 2 S/2 S /2 disjoint sub-MDPs, each containing 2 states.
For each 1 ≤ j ≤ S / 2 1 \le j \le S/2 1 ≤ j ≤ S /2 , the j j j -th sub-MDP of M \mathcal{M} M , denoted by M s u b , j \mathcal{M}^{\mathrm{sub},j} M sub , j , has state space { j , j + S / 2 } \{j,j+S/2\} { j , j + S /2 } and is a relabeled copy of an MDP in C m d p s u b \mathcal{C}_{\mathrm{mdp}}^{\mathrm{sub}} C mdp sub introduced in § 8.13 , with states 1 and 0 therein corresponding to j j j and j + S / 2 j+S/2 j + S /2 , respectively.
More precisely, we define C m d p \mathcal{C}_{\mathrm{mdp}} C mdp to consist of all MDPs obtained by choosing each of the S / 2 S/2 S /2 sub-MDPs independently from C m d p s u b \mathcal{C}_{\mathrm{mdp}}^{\mathrm{sub}} C mdp sub . Equivalently, up to the state relabeling described above,
C m d p : = ( C m d p s u b ) S / 2 . (490) \mathcal{C}_{\mathrm{mdp}} := \left(\mathcal{C}_{\mathrm{mdp}}^{\mathrm{sub}}\right)^{S/2}. \tag*{(490)} C mdp := ( C mdp sub ) S /2 . (490)
Thus, each M ∈ C m d p \mathcal{M} \in\mathcal{C}_{\mathrm{mdp}} M ∈ C mdp can be identified with a tuple
M = ( M s u b , 1 , … , M s u b , S / 2 ) , M s u b , j ∈ C m d p s u b . (491) \mathcal{M} = \left(\mathcal{M}^{\mathrm{sub},1},\ldots,\mathcal{M}^{\mathrm{sub},S/2}\right), \qquad\mathcal{M}^{\mathrm{sub},j} \in\mathcal{C}_{\mathrm{mdp}}^{\mathrm{sub}}. \tag*{(491)} M = ( M sub , 1 , … , M sub , S /2 ) , M sub , j ∈ C mdp sub . (491)
ρ ( s ) = { 2 / S , if 1 ≤ s ≤ S / 2 , 0 , else . (492) \rho(s) =
\begin{cases}
2/S, & \text{if } 1 \le s \le S/2,\\
0, & \text{else}.
\end{cases} \tag*{(492)} ρ ( s ) = { 2/ S , 0 , if 1 ≤ s ≤ S /2 , else . (492)
Decomposition across the sub-MDPs. Let
J k : = s 1 k ∈ [ S / 2 ] , J : = ( J 1 , … , J K ) , K j : = ∑ k = 1 K 1 { J k = j } . J_k := s_1^k \in[S/2], \qquad\mathbf{J} := (J_1,\ldots,J_K), \qquad K_j := \sum_{k=1}^{K} \mathbf{1}\{J_k=j\}. J k := s 1 k ∈ [ S /2 ] , J := ( J 1 , … , J K ) , K j := k = 1 ∑ K 1 { J k = j } .
Thus, J k J_k J k identifies the sub-MDP visited in episode k k k , and K j K_j K j is the number of episodes assigned to the j j j -th sub-MDP. Since the initial states are drawn independently from ρ \rho ρ , the random vector J \mathbf{J} J is independent of the identities of all sub-MDPs.
We now place the uniform product prior on C m d p \mathcal{C}_{\mathrm{mdp}} C mdp ; namely, the sub-MDPs { M s u b , j } 1 ≤ j ≤ S / 2 \{\mathcal{M}^{\mathrm{sub},j}\}_{1\le j\le S/2} { M sub , j } 1 ≤ j ≤ S /2 are drawn independently and uniformly from C m d p s u b \mathcal{C}_{\mathrm{mdp}}^{\mathrm{sub}} C mdp sub . Fix an initial-state sequence J \mathbf{J} J and a component j j j . Under this product prior, the behavior of π ^ \widehat{\pi} π during the K j K_j K j episodes assigned to component j j j induces a valid randomized K j K_j K j -episode algorithm for the two-state problem. Indeed, the interactions with all other components can be incorporated into the internal randomization of this induced algorithm: their identities are drawn independently from the product prior, and their trajectories can be simulated between successive visits to component j j j . Although these simulated interactions may depend on the observations previously collected from component j j j , they contain no additional information about the identity of M s u b , j \mathcal{M}^{\mathrm{sub},j} M sub , j .
Consequently, the two-state lower bound (489) applies conditionally on J \mathbf{J} J . In particular, for every j ∈ [ S / 2 ] j \in[S/2] j ∈ [ S /2 ] ,
1 ∣ C m d p ∣ ∑ M ∈ C m d p E M [ ∑ k : J k = j ( V 1 ∗ ( j ) − V 1 π k ( j ) ) | J ] ≥ ε 0 K j 192 1 { K j ≤ A H 3 18 ε 0 2 } . (493) \frac{1}{|\mathcal{C}_{\mathrm{mdp}}|}\sum_{\mathcal{M}\in\mathcal{C}_{\mathrm{mdp}}}\mathbb{E}_{\mathcal{M}}\left[\sum_{k:J_k=j}\left(V_1^*(j)-V_1^{\pi^k}(j)\right)\,\middle|\,\mathbf{J}\right]\ge\frac{\varepsilon_0 K_j}{192}\mathbf{1}\left\{K_j\le\frac{AH^3}{18\varepsilon_0^2}\right\}. \tag*{(493)} ∣ C mdp ∣ 1 M ∈ C mdp ∑ E M k : J k = j ∑ ( V 1 ∗ ( j ) − V 1 π k ( j ) ) J ≥ 192 ε 0 K j 1 { K j ≤ 18 ε 0 2 A H 3 } . (493)
Moreover, since J k ∼ ρ J_k\sim\rho J k ∼ ρ independently of the policy π k \pi^k π k selected before episode k k k , for every M ∈ C m d p \mathcal{M}\in\mathcal{C}_{\mathrm{mdp}} M ∈ C mdp one has
E M [ V 1 ∗ ( J k ) − V 1 π k ( J k ) ] = E M [ V 1 ∗ ( ρ ) − V 1 π k ( ρ ) ] . (494) \mathbb{E}_{\mathcal{M}}\left[V_1^*(J_k)-V_1^{\pi^k}(J_k)\right]=\mathbb{E}_{\mathcal{M}}\left[V_1^*(\rho)-V_1^{\pi^k}(\rho)\right]. \tag*{(494)} E M [ V 1 ∗ ( J k ) − V 1 π k ( J k ) ] = E M [ V 1 ∗ ( ρ ) − V 1 π k ( ρ ) ] . (494)
Summing (493) over the components and averaging over J \mathbf{J} J yield
inf π ^ max M ∈ C m d p E M [ ∑ k = 1 K ( V 1 ⋆ ( ρ ) − V 1 π k ( ρ ) ) ] ≥ inf π ^ 1 ∣ C m d p ∣ ∑ M ∈ C m d p E M [ ∑ k = 1 K ( V 1 ⋆ ( J k ) − V 1 π k ( J k ) ) ] ≥ E J [ ∑ j = 1 S / 2 ε 0 K j 192 1 { K j ≤ A H 3 18 ε 0 2 } ] . (495) \begin{aligned}
\inf_{\hat{\pi}}\max_{\mathcal{M}\in\mathcal{C}_{\mathrm{mdp}}}\mathbb{E}_{\mathcal{M}}\left[\sum_{k=1}^{K}\left(V_1^\star(\rho)-V_1^{\pi^k}(\rho)\right)\right]\geq\inf_{\hat{\pi}}\frac{1}{|\mathcal{C}_{\mathrm{mdp}}|}\sum_{\mathcal{M}\in\mathcal{C}_{\mathrm{mdp}}}\mathbb{E}_{\mathcal{M}}\left[\sum_{k=1}^{K}\left(V_1^\star(J_k)-V_1^{\pi^k}(J_k)\right)\right] \\
&\geq\mathbb{E}_{\mathbf{J}}\left[\sum_{j=1}^{S/2}\frac{\varepsilon_0K_j}{192}\mathbf{1}\left\{K_j\leq\frac{AH^3}{18\varepsilon_0^2}\right\}\right]. \tag*{(495)}
\end{aligned} π ^ inf M ∈ C mdp max E M [ k = 1 ∑ K ( V 1 ⋆ ( ρ ) − V 1 π k ( ρ ) ) ] ≥ π ^ inf ∣ C mdp ∣ 1 M ∈ C mdp ∑ E M [ k = 1 ∑ K ( V 1 ⋆ ( J k ) − V 1 π k ( J k ) ) ] ≥ E J j = 1 ∑ S /2 192 ε 0 K j 1 { K j ≤ 18 ε 0 2 A H 3 } . (495)
In what follows, we look at two cases separately.
Case 1: K ≥ S / 2 K\geq S/2 K ≥ S /2 . We begin with the regime K ≥ S / 2 K\geq S/2 K ≥ S /2 . Continue the bound (495) to derive
inf π ^ max M ∈ C m d p E M [ ∑ k = 1 K ( V 1 ⋆ ( ρ ) − V 1 π k ( ρ ) ) ] ≥ E s 1 , … , s K ∼ i . i . d . ρ [ ∑ j = 1 S / 2 ε 0 K j 192 ] − ε 0 192 E s 1 , … , s K ∼ i . i . d . ρ [ ∑ j = 1 S / 2 K j 1 { K j > A H 3 18 ε 0 2 } ] = ε 0 K 192 − ε 0 192 ∑ j = 1 S / 2 E s 1 , … , s K ∼ i . i . d . ρ [ K j 1 { K j > A H 3 18 ε 0 2 } ] . (496) \begin{aligned}
\inf_{\hat{\pi}}\max_{\mathcal{M}\in\mathcal{C}_{\mathrm{mdp}}}\mathbb{E}_{\mathcal{M}}\left[\sum_{k=1}^{K}\left(V_1^\star(\rho)-V_1^{\pi^k}(\rho)\right)\right] \\
&\geq\mathbb{E}_{s_1,\ldots,s_K\stackrel{\mathrm{i.i.d.}}{\sim}\rho}\left[\sum_{j=1}^{S/2}\frac{\varepsilon_0K_j}{192}\right]-\frac{\varepsilon_0}{192}\mathbb{E}_{s_1,\ldots,s_K\stackrel{\mathrm{i.i.d.}}{\sim}\rho}\left[\sum_{j=1}^{S/2}K_j\mathbf{1}\left\{K_j>\frac{AH^3}{18\varepsilon_0^2}\right\}\right] \\
&=\frac{\varepsilon_0K}{192}-\frac{\varepsilon_0}{192}\sum_{j=1}^{S/2}\mathbb{E}_{s_1,\ldots,s_K\stackrel{\mathrm{i.i.d.}}{\sim}\rho}\left[K_j\mathbf{1}\left\{K_j>\frac{AH^3}{18\varepsilon_0^2}\right\}\right]. \tag*{(496)}
\end{aligned} π ^ inf M ∈ C mdp max E M [ k = 1 ∑ K ( V 1 ⋆ ( ρ ) − V 1 π k ( ρ ) ) ] ≥ E s 1 , … , s K ∼ i.i.d. ρ j = 1 ∑ S /2 192 ε 0 K j − 192 ε 0 E s 1 , … , s K ∼ i.i.d. ρ j = 1 ∑ S /2 K j 1 { K j > 18 ε 0 2 A H 3 } = 192 ε 0 K − 192 ε 0 j = 1 ∑ S /2 E s 1 , … , s K ∼ i.i.d. ρ [ K j 1 { K j > 18 ε 0 2 A H 3 } ] . (496)
Since K j ∼ Binomial ( K , 2 / S ) K_j\sim\operatorname{Binomial}(K,2/S) K j ∼ Binomial ( K , 2/ S ) , Lemma 2.7 tells us that: if ε 0 2 ≤ S A H 3 1800 K \varepsilon_0^2\leq\frac{SAH^3}{1800K} ε 0 2 ≤ 1800 K S A H 3 , then
E [ K j 1 { K j > A H 3 18 ε 0 2 } ] ≤ E [ K j 1 { K j > 100 K S } ] = 100 K S P { K j > 100 K S } + ∫ 100 K / S ∞ P { K j > τ } d τ ≤ 100 K S exp ( − 19 K S ) + 2 ∫ 100 K / S ∞ exp ( − 0.19 τ ) d τ ≤ 111 K S exp ( − 19 K S ) ≤ 0.01 K S , \begin{aligned}
\mathbb{E}\left[K_j\mathbf{1}\left\{K_j>\frac{AH^3}{18\varepsilon_0^2}\right\}\right]\leq\mathbb{E}\left[K_j\mathbf{1}\left\{K_j>\frac{100K}{S}\right\}\right] \\
&=\frac{100K}{S}\mathbb{P}\left\{K_j>\frac{100K}{S}\right\}+\int_{100K/S}^{\infty}\mathbb{P}\{K_j>\tau\}\,d\tau\\
&\leq\frac{100K}{S}\exp\left(-\frac{19K}{S}\right)+2\int_{100K/S}^{\infty}\exp(-0.19\tau)\,d\tau\\
&\leq\frac{111K}{S}\exp\left(-\frac{19K}{S}\right)\leq\frac{0.01K}{S},
\end{aligned} E [ K j 1 { K j > 18 ε 0 2 A H 3 } ] ≤ E [ K j 1 { K j > S 100 K } ] = S 100 K P { K j > S 100 K } + ∫ 100 K / S ∞ P { K j > τ } d τ ≤ S 100 K exp ( − S 19 K ) + 2 ∫ 100 K / S ∞ exp ( − 0.19 τ ) d τ ≤ S 111 K exp ( − S 19 K ) ≤ S 0.01 K ,
where the second line relies on the basic property
E [ X 1 { X > x } ] = x P ( X > x ) + ∫ x ∞ P ( X > τ ) d τ . \mathbb{E}\left[X\mathbf{1}\{X>x\}\right]=x\mathbb{P}(X>x)+\int_x^\infty\mathbb{P}(X>\tau)\,d\tau. E [ X 1 { X > x } ] = x P ( X > x ) + ∫ x ∞ P ( X > τ ) d τ .
Substitution into (496) shows that: if ε 0 2 ≤ S A H 3 1800 K \varepsilon_0^2\leq\frac{SAH^3}{1800K} ε 0 2 ≤ 1800 K S A H 3 and ε 0 ≤ H / 2 \varepsilon_0\leq H/2 ε 0 ≤ H /2 , then
inf π ^ max M ∈ C m d p E M [ ∑ k = 1 K ( V 1 ⋆ ( ρ ) − V 1 π k ( ρ ) ) ] ≥ ε 0 K 192 − ε 0 ( S / 2 ) 192 ⋅ 0.01 K S = 0.995 ε 0 K 192 . \inf_{\hat{\pi}}\max_{\mathcal{M}\in\mathcal{C}_{\mathrm{mdp}}}\mathbb{E}_{\mathcal{M}}\left[\sum_{k=1}^{K}\left(V_1^\star(\rho)-V_1^{\pi^k}(\rho)\right)\right]\geq\frac{\varepsilon_0K}{192}-\frac{\varepsilon_0(S/2)}{192}\cdot\frac{0.01K}{S}=\frac{0.995\varepsilon_0K}{192}. π ^ inf M ∈ C mdp max E M [ k = 1 ∑ K ( V 1 ⋆ ( ρ ) − V 1 π k ( ρ ) ) ] ≥ 192 ε 0 K − 192 ε 0 ( S /2 ) ⋅ S 0.01 K = 192 0.995 ε 0 K .
Case 2: K < S / 2 K<S/2 K < S /2 . We now turn to the regime K < S / 2 K<S/2 K < S /2 , where it suffices to focus on states that are visited exactly once.
To begin with, observe that if ε 0 ≤ H / 2 \varepsilon_0\leq H/2 ε 0 ≤ H /2 , then
A H 3 18 ε 0 2 ≥ 2 A H 9 ≥ 4 3 > 1 , \frac{AH^3}{18\varepsilon_0^2}\geq\frac{2AH}{9}\geq\frac{4}{3}>1, 18 ε 0 2 A H 3 ≥ 9 2 A H ≥ 3 4 > 1 ,
provided that A ≥ 3 A \ge3 A ≥ 3 and H ≥ 2 H \ge2 H ≥ 2 . Hence, every j j j obeying K j = 1 K_j = 1 K j = 1 satisfies A H 3 18 ε 0 2 > K j \frac{AH^3}{18\varepsilon_0^2} > K_j 18 ε 0 2 A H 3 > K j . Taking this collectively with (495) , we see that
inf π max M ∈ C m d p E M [ ∑ k = 1 K ( V 1 ∗ ( ρ ) − V 1 π k ( ρ ) ) ] ≥ E s 1 1 , … , s 1 K ∼ i . i . d . ρ [ ∑ j = 1 S / 2 ε 0 K j 192 1 { K j ≤ A H 3 18 ε 0 2 } ] ≥ E s 1 1 , … , s 1 K ∼ i . i . d . ρ [ ∑ j = 1 S / 2 ε 0 K j 192 1 { K j = 1 } ] = ε 0 192 ∑ j = 1 S / 2 P ( K j = 1 ) . (497) \begin{aligned}
\inf_{\pi} \max_{\mathcal{M} \in\mathcal{C}_{\mathrm{mdp}}} \mathbb{E}_{\mathcal{M}}\left[\sum_{k=1}^{K}\left(V_1^*(\rho)-V_1^{\pi^k}(\rho)\right)\right] \ge\mathbb{E}_{s_1^1,\ldots,s_1^K \overset{\mathrm{i.i.d.}}{\sim}\rho}\left[\sum_{j=1}^{S/2}\frac{\varepsilon_0 K_j}{192}\mathbf{1}\left\{K_j \le\frac{AH^3}{18\varepsilon_0^2}\right\}\right] \\
&\ge\mathbb{E}_{s_1^1,\ldots,s_1^K \overset{\mathrm{i.i.d.}}{\sim}\rho}\left[\sum_{j=1}^{S/2}\frac{\varepsilon_0 K_j}{192}\mathbf{1}\{K_j=1\}\right] \\
&= \frac{\varepsilon_0}{192}\sum_{j=1}^{S/2}\mathbb{P}(K_j=1). \tag*{(497)}
\end{aligned} π inf M ∈ C mdp max E M [ k = 1 ∑ K ( V 1 ∗ ( ρ ) − V 1 π k ( ρ ) ) ] ≥ E s 1 1 , … , s 1 K ∼ i.i.d. ρ j = 1 ∑ S /2 192 ε 0 K j 1 { K j ≤ 18 ε 0 2 A H 3 } ≥ E s 1 1 , … , s 1 K ∼ i.i.d. ρ j = 1 ∑ S /2 192 ε 0 K j 1 { K j = 1 } = 192 ε 0 j = 1 ∑ S /2 P ( K j = 1 ) . (497)
Recognizing that K j ∼ Binomial ( K , 2 / S ) K_j \sim\operatorname{Binomial}(K,2/S) K j ∼ Binomial ( K , 2/ S ) , one can deduce that
∑ j = 1 S / 2 P ( K j = 1 ) = S 2 ⋅ K 2 S ( 1 − 2 S ) K − 1 = K ( 1 − 2 S ) K − 1 ≥ K 4 , \sum_{j=1}^{S/2}\mathbb{P}(K_j=1)=\frac{S}{2}\cdot K\frac{2}{S}\left(1-\frac{2}{S}\right)^{K-1}=K\left(1-\frac{2}{S}\right)^{K-1}\ge\frac{K}{4}, j = 1 ∑ S /2 P ( K j = 1 ) = 2 S ⋅ K S 2 ( 1 − S 2 ) K − 1 = K ( 1 − S 2 ) K − 1 ≥ 4 K ,
where the last inequality follows from K < S / 2 K<S/2 K < S /2 . Consequently, we arrive at
inf π max M ∈ C m d p E M [ ∑ k = 1 K ( V 1 ∗ ( ρ ) − V 1 π k ( ρ ) ) ] ≥ ε 0 K 768 . (498) \inf_{\pi} \max_{\mathcal{M} \in\mathcal{C}_{\mathrm{mdp}}} \mathbb{E}_{\mathcal{M}}\left[\sum_{k=1}^{K}\left(V_1^*(\rho)-V_1^{\pi^k}(\rho)\right)\right]\ge\frac{\varepsilon_0K}{768}. \tag*{(498)} π inf M ∈ C mdp max E M [ k = 1 ∑ K ( V 1 ∗ ( ρ ) − V 1 π k ( ρ ) ) ] ≥ 768 ε 0 K . (498)
Putting all this together. To finish up, we take
ε 0 = min { S A H 3 1800 K , H 2 } \varepsilon_0=\min\left\{\sqrt{\frac{SAH^3}{1800K}},\frac{H}{2}\right\} ε 0 = min { 1800 K S A H 3 , 2 H }
in the above bounds (for both Case 1 and Case 2) to conclude that
inf π max M ∈ C m d p E M [ ∑ k = 1 K ( V 1 ∗ ( ρ ) − V 1 π k ( ρ ) ) ] ≳ min { S A H 3 K , H K } . \inf_{\pi} \max_{\mathcal{M} \in\mathcal{C}_{\mathrm{mdp}}} \mathbb{E}_{\mathcal{M}}\left[\sum_{k=1}^{K}\left(V_1^*(\rho)-V_1^{\pi^k}(\rho)\right)\right]\gtrsim\min\left\{\sqrt{SAH^3K},HK\right\}. π inf M ∈ C mdp max E M [ k = 1 ∑ K ( V 1 ∗ ( ρ ) − V 1 π k ( ρ ) ) ] ≳ min { S A H 3 K , H K } .
Appendix A: proof of auxiliary lemmas in Section 5.3.5
Proof of Lemma 5.5
We begin by introducing a scalar-valued function f f f , which facilitates the subsequent analysis:
f ( p , v , n ) : = ⟨ p , v ⟩ + max { c 1 Var p ( v ) log 1 δ n , c 1 2 H log 1 δ n } , (499) f(p,v,n):=\langle p,v\rangle+\max\left\{c_1\sqrt{\frac{\operatorname{Var}_p(v)\log\frac{1}{\delta}}{n}},\frac{c_1^2H\log\frac{1}{\delta}}{n}\right\}, \tag*{(499)} f ( p , v , n ) := ⟨ p , v ⟩ + max ⎩ ⎨ ⎧ c 1 n Var p ( v ) log δ 1 , n c 1 2 H log δ 1 ⎭ ⎬ ⎫ , (499)
defined for any probability vector p = [ p s ] 1 ≤ s ≤ S ∈ Δ S p=[p_s]_{1\le s\le S}\in\Delta^S p = [ p s ] 1 ≤ s ≤ S ∈ Δ S , any nonnegative vector v = [ v s ] 1 ≤ s ≤ S ∈ R S v=[v_s]_{1\le s\le S}\in\mathbb{R}^S v = [ v s ] 1 ≤ s ≤ S ∈ R S obeying ∥ v ∥ ∞ ≤ H \lVert v\rVert_\infty\le H ∥ v ∥ ∞ ≤ H , and any positive integer n n n . A key property of this function, which we rely on in this proof, is its monotonicity with respect to v v v , namely,
f ( p , v , n ) is non-decreasing in each entry of v . (500) f(p,v,n)\text{ is non-decreasing in each entry of }v. \tag*{(500)} f ( p , v , n ) is non-decreasing in each entry of v . (500)
The proof of this property is postponed to the end of this subsection. We now present the proof of Lemma 5.5 using the property (500) . Consider any tuple ( h , k , s , a ) (h,k,s,a) ( h , k , s , a ) , and we divide into two cases.
Case 1: N h k ( s , a ) ≤ 2 N_h^k(s,a) \le2 N h k ( s , a ) ≤ 2 . In this situation, the update rule (5.18) in conjunction with the bonus (5.20) yields the following trivial bounds:
Q h k ( s , a ) = H ≥ Q h ⋆ ( s , a ) and V h k ( s ) ≥ Q h k ( s , a ) = H ≥ V h ⋆ ( s ) , Q_h^k(s,a) = H \ge Q_h^\star(s,a) \qquad\text{and} \qquad V_h^k(s) \ge Q_h^k(s,a) = H \ge V_h^\star(s), Q h k ( s , a ) = H ≥ Q h ⋆ ( s , a ) and V h k ( s ) ≥ Q h k ( s , a ) = H ≥ V h ⋆ ( s ) ,
provided that the constant c 2 c_2 c 2 in (5.20) exceeds c 2 ≥ 2 c_2 \ge2 c 2 ≥ 2 .
Case 2: N h k ( s , a ) > 2 N_h^k(s,a) > 2 N h k ( s , a ) > 2 . Suppose now, by way of induction, that Q h + 1 k ≥ Q h + 1 ⋆ Q_{h+1}^k \ge Q_{h+1}^\star Q h + 1 k ≥ Q h + 1 ⋆ , which necessarily implies that V h + 1 k ≥ V h + 1 ⋆ V_{h+1}^k \ge V_{h+1}^\star V h + 1 k ≥ V h + 1 ⋆ . If Q h k ( s , a ) = H Q_h^k(s,a) = H Q h k ( s , a ) = H , then Q h k ( s , a ) ≥ Q h ⋆ ( s , a ) Q_h^k(s,a) \ge Q_h^\star(s,a) Q h k ( s , a ) ≥ Q h ⋆ ( s , a ) holds trivially. Therefore, it suffices to restrict attention to those state-action pairs obeying Q h k ( s , a ) < H Q_h^k(s,a) < H Q h k ( s , a ) < H in the sequel.
In view of the update rule (5.18) and the bonus (5.20), when Q h k ( s , a ) < H Q_h^k(s,a) < H Q h k ( s , a ) < H , we have
Q h k ( s , a ) = r h ( s , a ) + ⟨ P ^ s , a , h k , V h + 1 k ⟩ + c 1 Var P ^ s , a , h k ( V h + 1 k ) log 1 δ ′ N h k ( s , a ) + c 2 H log 1 δ ′ N h k ( s , a ) ≥ r h ( s , a ) + ( c 2 − c 1 2 ) H log 1 δ ′ N h k ( s , a ) + f ( P ^ s , a , h k , V h + 1 k , N h k ( s , a ) ) ≥ r h ( s , a ) + f ( P ^ s , a , h k , V h + 1 ⋆ , N h k ( s , a ) ) . (501) \begin{aligned}
Q_h^k(s,a)
&= r_h(s,a) + \langle\widehat{P}_{s,a,h}^k,V_{h+1}^k\rangle+ c_1\sqrt{\frac{\operatorname{Var}_{\widehat{P}_{s,a,h}^k}(V_{h+1}^k)\log\frac{1}{\delta'}}{N_h^k(s,a)}} + c_2\frac{H\log\frac{1}{\delta'}}{N_h^k(s,a)} \\
&\ge r_h(s,a) + \frac{(c_2-c_1^2)H\log\frac{1}{\delta'}}{N_h^k(s,a)} + f(\widehat{P}_{s,a,h}^k,V_{h+1}^k,N_h^k(s,a)) \\
&\ge r_h(s,a) + f(\widehat{P}_{s,a,h}^k,V_{h+1}^\star,N_h^k(s,a)). \tag*{(501)}
\end{aligned} Q h k ( s , a ) = r h ( s , a ) + ⟨ P s , a , h k , V h + 1 k ⟩ + c 1 N h k ( s , a ) Var P s , a , h k ( V h + 1 k ) log δ ′ 1 + c 2 N h k ( s , a ) H log δ ′ 1 ≥ r h ( s , a ) + N h k ( s , a ) ( c 2 − c 1 2 ) H log δ ′ 1 + f ( P s , a , h k , V h + 1 k , N h k ( s , a )) ≥ r h ( s , a ) + f ( P s , a , h k , V h + 1 ⋆ , N h k ( s , a )) . (501)
as long as c 2 ≥ c 1 2 c_2 \ge c_1^2 c 2 ≥ c 1 2 , where the last inequality results from the claim (500) and the property V h + 1 k ≥ V h + 1 ⋆ V_{h+1}^k \ge V_{h+1}^\star V h + 1 k ≥ V h + 1 ⋆ . Moreover, applying the empirical Bernstein inequality in Lemma 2.6 yields
P { ∣ ⟨ P ^ s , a , h k − P s , a , h , V h + 1 ⋆ ⟩ ∣ ≥ 2 Var P ^ s , a , h k ( V h + 1 ⋆ ) log 1 δ ′ N h k ( s , a ) + 14 H log 1 δ ′ 3 N h k ( s , a ) } ≤ P { ∣ ⟨ P ^ s , a , h k − P s , a , h , V h + 1 ⋆ ⟩ ∣ ≥ 2 Var P ^ s , a , h k ( V h + 1 ⋆ ) log 1 δ ′ N h k ( s , a ) − 1 + 7 H log 1 δ ′ 3 ( N h k ( s , a ) − 1 ) } ≤ 2 δ ′ . \begin{aligned}
\mathbb{P}\left\{\left|\langle\widehat{P}_{s,a,h}^k-P_{s,a,h},V_{h+1}^\star\rangle\right| \ge2\sqrt{\frac{\operatorname{Var}_{\widehat{P}_{s,a,h}^k}(V_{h+1}^\star)\log\frac{1}{\delta'}}{N_h^k(s,a)}}+\frac{14H\log\frac{1}{\delta'}}{3N_h^k(s,a)}\right\} \\
&\le\mathbb{P}\left\{\left|\langle\widehat{P}_{s,a,h}^k-P_{s,a,h},V_{h+1}^\star\rangle\right| \ge\sqrt{\frac{2\operatorname{Var}_{\widehat{P}_{s,a,h}^k}(V_{h+1}^\star)\log\frac{1}{\delta'}}{N_h^k(s,a)-1}+\frac{7H\log\frac{1}{\delta'}}{3\left(N_h^k(s,a)-1\right)}}\right\} \le2\delta'.
\end{aligned} P ⎩ ⎨ ⎧ ⟨ P s , a , h k − P s , a , h , V h + 1 ⋆ ⟩ ≥ 2 N h k ( s , a ) Var P s , a , h k ( V h + 1 ⋆ ) log δ ′ 1 + 3 N h k ( s , a ) 14 H log δ ′ 1 ⎭ ⎬ ⎫ ≤ P ⎩ ⎨ ⎧ ⟨ P s , a , h k − P s , a , h , V h + 1 ⋆ ⟩ ≥ N h k ( s , a ) − 1 2 Var P s , a , h k ( V h + 1 ⋆ ) log δ ′ 1 + 3 ( N h k ( s , a ) − 1 ) 7 H log δ ′ 1 ⎭ ⎬ ⎫ ≤ 2 δ ′ .
This in turn implies that, with probability exceeding 1 − 2 δ ′ 1-2\delta' 1 − 2 δ ′ ,
\begin{aligned}
f(\widehat{P}_{s,a,h}^k,V_{h+1}^\star,N_h^k(s,a))
&\ge\langle P_{s,a,h},V_{h+1}^\star\rangle+\langle\widehat{P}_{s,a,h}^k-P_{s,a,h},V_{h+1}^\star\rangle\\
&\quad+\frac{c_1}{2}\sqrt{\frac{\operatorname{Var}_{\widehat{P}_{s,a,h}^k}(V_{h+1}^\star)\log\frac{1}{\delta'}}{N_h^k(s,a)}+\frac{c_1^2H\log\frac{1}{\delta'}}{2N_h^k(s,a)} \\
&\ge\langle P_{s,a,h},V_{h+1}^\star\rangle+\left(\frac{c_1^2}{2}-\frac{14}{3}\right)\frac{H\log\frac{1}{\delta'}}{N_h^k(s,a)} \\
&\quad+\left(\frac{c_1}{2}-2\right)\sqrt{\frac{\operatorname{Var}_{\widehat{P}_{s,a,h}^k}(V_{h+1}^\star)\log\frac{1}{\delta'}}{N_h^k(s,a)}} \\
&\ge\langle P_{s,a,h},V_{h+1}^\star\rangle,
\end{aligned}}
where the last line holds as long as c 1 ≥ 4 c_1 \ge4 c 1 ≥ 4 . Substitution into (501) then yields that, with probability at least 1 − 2 δ ′ 1-2\delta' 1 − 2 δ ′ ,
Q h k ( s , a ) ≥ r h ( s , a ) + ⟨ P s , a , h , V h + 1 ⋆ ⟩ = Q h ⋆ ( s , a ) , Q_h^k(s,a) \ge r_h(s,a)+\langle P_{s,a,h},V_{h+1}^\star\rangle=Q_h^\star(s,a), Q h k ( s , a ) ≥ r h ( s , a ) + ⟨ P s , a , h , V h + 1 ⋆ ⟩ = Q h ⋆ ( s , a ) ,
where the identity is due to the Bellman equation (2.40).
To finish up, taking the union bound over all ( s , a , h , k ) ∈ S × A × [ H ] × [ K ] (s,a,h,k)\in\mathcal{S}\times\mathcal{A}\times[H]\times[K] ( s , a , h , k ) ∈ S × A × [ H ] × [ K ] concludes the proof.
Putting all this together. With the above two cases in place, one can invoke standard induction arguments to deduce that: with probability at least 1 − 4 S A H K δ ′ 1-4SAHK\delta' 1 − 4 S A H K δ ′ , one has Q h k ( s , a ) ≥ Q h ⋆ ( s , a ) Q_h^k(s,a) \ge Q_h^\star(s,a) Q h k ( s , a ) ≥ Q h ⋆ ( s , a ) and V h k = max a Q h k ( s , a ) ≥ max a Q h ⋆ ( s , a ) = V h ⋆ ( s ) V_h^k = \max_a Q_h^k(s,a) \ge\max_a Q_h^\star(s,a) = V_h^\star(s) V h k = max a Q h k ( s , a ) ≥ max a Q h ⋆ ( s , a ) = V h ⋆ ( s ) for every ( s , a , h , k ) (s,a,h,k) ( s , a , h , k ) , thus completing the proof.
Proof of property (500) . Fix any index 1 ≤ s ≤ S 1 \le s \le S 1 ≤ s ≤ S , and consider f f f as a function of the s s s -th entry v s v_s v s while freezing p , n p,n p , n and all other entries of v v v . We begin with the following observations:
f f f is a continuous function w.r.t. v v v ;
f f f is differentiable in v s v_s v s , except for at most two possible points that make c 1 Var p ( v ) log 1 δ ′ n = c 1 2 H log 1 δ ′ n c_1\sqrt{\frac{\operatorname{Var}_p(v)\log\frac{1}{\delta'}}{n}}=c_1^2\frac{H\log\frac{1}{\delta'}}{n} c 1 n Var p ( v ) l o g δ ′ 1 = c 1 2 n H l o g δ ′ 1 (due to the presence of non-differentiable points of the max operator and the quadratic nature of Var p ( v ) \operatorname{Var}_p(v) Var p ( v ) ).
By virtue of the continuity of f f f , establishing that f f f is non-decreasing in v s v_s v s reduces to verifying that ∂ f ( p , v , n ) ∂ v s ≥ 0 \frac{\partial f(p,v,n)}{\partial v_s} \ge0 ∂ v s ∂ f ( p , v , n ) ≥ 0 at all differentiable points.
To this end, we compute the partial derivative at any differentiable point v s v_s v s as follows:
∂ f ( p , v , n ) ∂ v s = p s + c 1 1 { c 1 Var p ( v ) log 1 δ ′ n ≥ c 1 2 H log 1 δ ′ n } p s ( v s − ⟨ p , v ⟩ ) log 1 δ ′ n Var p ( v ) = p s + 1 ( A v ) ⋅ c 1 H log 1 δ ′ n Var p ( v ) log 1 δ ′ ⋅ p s ( v s − ⟨ p , v ⟩ ) H , (502) \begin{aligned}
\frac{\partial f(p,v,n)}{\partial v_s}
= p_s+c_1\mathbf{1}\left\{c_1\sqrt{\frac{\operatorname{Var}_p(v)\log\frac{1}{\delta'}}{n}}\ge c_1^2\frac{H\log\frac{1}{\delta'}}{n}\right\}\frac{p_s(v_s-\langle p,v\rangle)\sqrt{\log\frac{1}{\delta'}}}{\sqrt{n\operatorname{Var}_p(v)}} \\
&= p_s+\mathbf{1}(A_v)\cdot\frac{c_1H\log\frac{1}{\delta'}}{\sqrt{n\operatorname{Var}_p(v)\log\frac{1}{\delta'}}}\cdot\frac{p_s(v_s-\langle p,v\rangle)}{H}, \tag*{(502)}
\end{aligned} ∂ v s ∂ f ( p , v , n ) = p s + c 1 1 ⎩ ⎨ ⎧ c 1 n Var p ( v ) log δ ′ 1 ≥ c 1 2 n H log δ ′ 1 ⎭ ⎬ ⎫ n Var p ( v ) p s ( v s − ⟨ p , v ⟩) log δ ′ 1 = p s + 1 ( A v ) ⋅ n Var p ( v ) log δ ′ 1 c 1 H log δ ′ 1 ⋅ H p s ( v s − ⟨ p , v ⟩) , (502)
where we introduce the event A v : = { n Var p ( v ) log 1 δ ′ ≥ c 1 H log 1 δ ′ } A_v:=\left\{\sqrt{n\operatorname{Var}_p(v)\log\frac{1}{\delta'}}\ge c_1H\log\frac{1}{\delta'}\right\} A v := { n Var p ( v ) log δ ′ 1 ≥ c 1 H log δ ′ 1 } . We now divide into two cases.
If v s − ⟨ p , v ⟩ ≥ 0 v_s-\langle p,v\rangle\ge0 v s − ⟨ p , v ⟩ ≥ 0 , then it is trivially seen that ∂ f ( p , v , n ) ∂ v s ≥ 0 \frac{\partial f(p,v,n)}{\partial v_s}\ge0 ∂ v s ∂ f ( p , v , n ) ≥ 0 .
If v s − ⟨ p , v ⟩ < 0 v_s-\langle p,v\rangle<0 v s − ⟨ p , v ⟩ < 0 , then one can continue the derivation in (502) and use the definition of A v A_v A v to reach
∂ f ( p , v , n ) ∂ v s ≥ p s + p s v s − ⟨ p , v ⟩ H = p s H + v s − ⟨ p , v ⟩ H ≥ 0 , \frac{\partial f(p,v,n)}{\partial v_s}\ge p_s+p_s\frac{v_s-\langle p,v\rangle}{H}=p_s\frac{H+v_s-\langle p,v\rangle}{H}\ge0, ∂ v s ∂ f ( p , v , n ) ≥ p s + p s H v s − ⟨ p , v ⟩ = p s H H + v s − ⟨ p , v ⟩ ≥ 0 ,
where the last inequality holds since v s ≥ 0 v_s\ge0 v s ≥ 0 and ⟨ p , v ⟩ ≤ H \langle p,v\rangle\le H ⟨ p , v ⟩ ≤ H .
Combining these two cases immediately establishes property (500) . □
Proof of Lemma 5.7
Bounding the quantity W 1 W_1 W 1 . We begin by applying Bernstein’s inequality (cf. Lemma 2.5) in conjunction with a union bound over all tuples ( s , a , h , k ) ∈ S × A × [ H ] × [ K ] (s,a,h,k)\in\mathcal{S}\times\mathcal{A}\times[H]\times[K] ( s , a , h , k ) ∈ S × A × [ H ] × [ K ] . More specifically, since ∥ V h + 1 ⋆ ∥ ∞ ≤ H \lVert V_{h+1}^\star\rVert_\infty\le H ∥ V h + 1 ⋆ ∥ ∞ ≤ H , it holds with probability at least 1 − 4 S A H K δ ′ 1-4SAHK\delta' 1 − 4 S A H K δ ′ that
∣ P ^ s , a , h k − P s , a , h , V h + 1 ⋆ ∣ ≤ 2 Var P s , a , h ( V h + 1 ⋆ ) log 1 δ ′ N h k ( s , a ) + H log 1 δ ′ N h k ( s , a ) (503) \left|\widehat{P}_{s,a,h}^k-P_{s,a,h},V_{h+1}^\star\right|
\le\sqrt{\frac{2\operatorname{Var}_{P_{s,a,h}}(V_{h+1}^\star)\log\frac{1}{\delta'}}{N_h^k(s,a)}}+\frac{H\log\frac{1}{\delta'}}{N_h^k(s,a)} \tag*{(503)} P s , a , h k − P s , a , h , V h + 1 ⋆ ≤ N h k ( s , a ) 2 Var P s , a , h ( V h + 1 ⋆ ) log δ ′ 1 + N h k ( s , a ) H log δ ′ 1 (503)
simultaneously for all ( s , a , h , k ) ∈ S × A × [ H ] × [ K ] (s,a,h,k)\in\mathcal{S}\times\mathcal{A}\times[H]\times[K] ( s , a , h , k ) ∈ S × A × [ H ] × [ K ] . Using this relation to bound each summand in W 1 W_1 W 1 , we obtain
W 1 ≤ ∑ k = 1 K ∑ h = 1 H 2 Var P s h k , a h k ( V h + 1 ⋆ ) log 1 δ ′ N h k ( s h k , a h k ) + ∑ k = 1 K ∑ h = 1 H H log 1 δ ′ N h k ( s h k , a h k ) ≤ ∑ k , h 2 log 1 δ ′ N h k ( s h k , a h k ) ∑ k , h Var P s h k , a h k , h ( V h + 1 k ⋆ ) + H log 1 δ ′ ∑ k , h 1 N h k ( s h k , a h k ) \begin{aligned}
W_1\le\sum_{k=1}^{K}\sum_{h=1}^{H}\sqrt{\frac{2\operatorname{Var}_{P_{s_h^k,a_h^k}}(V_{h+1}^\star)\log\frac{1}{\delta'}}{N_h^k(s_h^k,a_h^k)}}+\sum_{k=1}^{K}\sum_{h=1}^{H}\frac{H\log\frac{1}{\delta'}}{N_h^k(s_h^k,a_h^k)} \\
&\le\sqrt{\sum_{k,h} \frac{2\log\frac{1}{\delta'}}{N_h^k(s_h^k,a_h^k)}\sqrt{\sum_{k,h}\operatorname{Var}_{P_{s_h^k,a_h^k,h}}(V_{h+1}^{k\star})}+H\log\frac{1}{\delta'}\sum_{k,h}\frac{1}{N_h^k(s_h^k,a_h^k)}}
\end{aligned} W 1 ≤ k = 1 ∑ K h = 1 ∑ H N h k ( s h k , a h k ) 2 Var P s h k , a h k ( V h + 1 ⋆ ) log δ ′ 1 + k = 1 ∑ K h = 1 ∑ H N h k ( s h k , a h k ) H log δ ′ 1 ≤ k , h ∑ N h k ( s h k , a h k ) 2 log δ ′ 1 k , h ∑ Var P s h k , a h k , h ( V h + 1 k ⋆ ) + H log δ ′ 1 k , h ∑ N h k ( s h k , a h k ) 1
≤ 2 S A H T 6 o p t ( log 2 K ) log 1 δ ′ + 2 S A H 2 ( log 2 K ) log 1 δ ′ (504) \le2\sqrt{SAHT_6^{\mathrm{opt}}(\log_2 K)\log\frac{1}{\delta'}}+2SAH^2(\log_2 K)\log\frac{1}{\delta'} \tag*{(504)} ≤ 2 S A H T 6 opt ( log 2 K ) log δ ′ 1 + 2 S A H 2 ( log 2 K ) log δ ′ 1 (504)
with probability greater than 1 − 4 S A H K δ ′ 1-4SAHK\delta' 1 − 4 S A H K δ ′ , where the second line invokes the Cauchy-Schwarz inequality, and the last inequality makes use of Lemma Lemma 5.3 and the definition (393) of T 6 o p t T_6^{\mathrm{opt}} T 6 opt .
To finish up, note that for any two random variables X X X and Y Y Y ,
Var ( X + Y ) ≤ 2 Var ( X ) + 2 Var ( Y ) ≤ 2 Var ( X ) + 2 E [ Y 2 ] . \operatorname{Var}(X+Y)\le2\operatorname{Var}(X)+2\operatorname{Var}(Y)\le2\operatorname{Var}(X)+2\mathbb{E}[Y^2]. Var ( X + Y ) ≤ 2 Var ( X ) + 2 Var ( Y ) ≤ 2 Var ( X ) + 2 E [ Y 2 ] .
This combined with the property V h + 1 k ≥ V h + 1 ⋆ ≥ V h + 1 π k V_{h+1}^k\ge V_{h+1}^{\star}\ge V_{h+1}^{\pi^k} V h + 1 k ≥ V h + 1 ⋆ ≥ V h + 1 π k (see Lemma Lemma 5.5 ) gives rise to
Var P s h k , a h k , h ( V h + 1 ⋆ ) ≤ 2 Var P s h k , a h k , h ( V h + 1 π k ) + 2 ⟨ P s h k , a h k , h , ( V h + 1 ⋆ − V h + 1 π k ) 2 ⟩ \operatorname{Var}_{P_{s_h^k,a_h^k,h}}(V_{h+1}^{\star})\le2\operatorname{Var}_{P_{s_h^k,a_h^k,h}}(V_{h+1}^{\pi^k})+2\left\langle P_{s_h^k,a_h^k,h},(V_{h+1}^{\star}-V_{h+1}^{\pi^k})^2\right\rangle Var P s h k , a h k , h ( V h + 1 ⋆ ) ≤ 2 Var P s h k , a h k , h ( V h + 1 π k ) + 2 ⟨ P s h k , a h k , h , ( V h + 1 ⋆ − V h + 1 π k ) 2 ⟩
≤ 2 Var P s h k , a h k , h ( V h + 1 π k ) + 2 ⟨ P s h k , a h k , h , ( V h + 1 k − V h + 1 π k ) 2 ⟩ , \le2\operatorname{Var}_{P_{s_h^k,a_h^k,h}}(V_{h+1}^{\pi^k})+2\left\langle P_{s_h^k,a_h^k,h},(V_{h+1}^{k}-V_{h+1}^{\pi^k})^2\right\rangle, ≤ 2 Var P s h k , a h k , h ( V h + 1 π k ) + 2 ⟨ P s h k , a h k , h , ( V h + 1 k − V h + 1 π k ) 2 ⟩ ,
which taken collectively with the definition (5.35) results in
T 6 o p t ≤ 2 T 6 l o w + 2 T 6 d i f f (505) T_6^{\mathrm{opt}}\le2T_6^{\mathrm{low}}+2T_6^{\mathrm{diff}} \tag*{(505)} T 6 opt ≤ 2 T 6 low + 2 T 6 diff (505)
⟹ T 6 o p t ≤ 2 T 6 l o w + 2 T 6 d i f f . (506) \Longrightarrow\quad\sqrt{T_6^{\mathrm{opt}}}\le\sqrt{2T_6^{\mathrm{low}}}+\sqrt{2T_6^{\mathrm{diff}}}. \tag*{(506)} ⟹ T 6 opt ≤ 2 T 6 low + 2 T 6 diff . (506)
This together with (504) and the definition of ι b \iota_b ι b (cf. (386) ) establishes the claimed bound
W 1 ≤ 1 8 ι b S A H T 6 l o w + 1 8 ι b S A H T 6 d i f f + 1 8 ι b S A H 2 . W_1\le\frac{1}{8}\sqrt{\iota_bSAHT_6^{\mathrm{low}}}+\frac{1}{8}\sqrt{\iota_bSAHT_6^{\mathrm{diff}}}+\frac{1}{8}\iota_bSAH^2. W 1 ≤ 8 1 ι b S A H T 6 low + 8 1 ι b S A H T 6 diff + 8 1 ι b S A H 2 .
Remark 5.13. Repeating the above arguments for (5.122) also yields
T 6 u c b ≤ 2 T 6 l o w + 2 T 6 d i f f (507) T_6^{\mathrm{ucb}}\le2T_6^{\mathrm{low}}+2T_6^{\mathrm{diff}} \tag*{(507)} T 6 ucb ≤ 2 T 6 low + 2 T 6 diff (507)
⟹ T 6 u c b ≤ 2 T 6 l o w + 2 T 6 d i f f , (508) \Longrightarrow\quad\sqrt{T_6^{\mathrm{ucb}}}\le\sqrt{2T_6^{\mathrm{low}}}+\sqrt{2T_6^{\mathrm{diff}}}, \tag*{(508)} ⟹ T 6 ucb ≤ 2 T 6 low + 2 T 6 diff , (508)
which will prove useful in the subsequent analysis.
Bounding the quantity W 2 W_2 W 2 . We now turn our attention to the term W 2 W_2 W 2 in (396) . Setting
Δ h + 1 k : = V h + 1 k − V h + 1 ⋆ \Delta_{h+1}^k:=V_{h+1}^k-V_{h+1}^{\star} Δ h + 1 k := V h + 1 k − V h + 1 ⋆
for notational convenience and denoting by v ( s ) v(s) v ( s ) the s s s -th entry of a vector v v v , we derive
⟨ P ^ s , a , h k − P s , a , h , V h + 1 k − V h + 1 ⋆ ⟩ = ∑ s ′ ∈ S ( P ^ s , a , h k ( s ′ ) − P s , a , h ( s ′ ) ) Δ h + 1 k ( s ′ ) \left\langle\widehat{P}_{s,a,h}^k-P_{s,a,h},V_{h+1}^k-V_{h+1}^{\star}\right\rangle
=\sum_{s'\in\mathcal{S}}\left(\widehat{P}_{s,a,h}^k(s')-P_{s,a,h}(s')\right)\Delta_{h+1}^k(s') ⟨ P s , a , h k − P s , a , h , V h + 1 k − V h + 1 ⋆ ⟩ = s ′ ∈ S ∑ ( P s , a , h k ( s ′ ) − P s , a , h ( s ′ ) ) Δ h + 1 k ( s ′ )
= ( i ) ∑ s ′ ∈ S ( P ^ s , a , h k ( s ′ ) − P s , a , h ( s ′ ) ) Δ h + 1 k ( s ′ ) − { ∑ s ′ ∈ S ( P ^ s , a , h k ( s ′ ) − P s , a , h ( s ′ ) ) } ⟨ P s , a , h , Δ h + 1 k ⟩ = ∑ s ′ ∈ S ( P ^ s , a , h k ( s ′ ) − P s , a , h ( s ′ ) ) ( Δ h + 1 k ( s ′ ) − ⟨ P s , a , h , Δ h + 1 k ⟩ ) ≤ ∑ s ′ ∈ S ∣ P ^ s , a , h k ( s ′ ) − P s , a , h ( s ′ ) ∣ ⋅ ∣ Δ h + 1 k ( s ′ ) − ⟨ P s , a , h , Δ h + 1 k ⟩ ∣ . (509) \begin{aligned}
\overset{(i)}{=}\sum_{s'\in\mathcal{S}}\left(\widehat{P}_{s,a,h}^k(s')-P_{s,a,h}(s')\right)\Delta_{h+1}^k(s')-\left\{\sum_{s'\in\mathcal{S}}\left(\widehat{P}_{s,a,h}^k(s')-P_{s,a,h}(s')\right)\right\}\left\langle P_{s,a,h},\Delta_{h+1}^k\right\rangle\\
&=\sum_{s'\in\mathcal{S}}\left(\widehat{P}_{s,a,h}^k(s')-P_{s,a,h}(s')\right)\left(\Delta_{h+1}^k(s')-\left\langle P_{s,a,h},\Delta_{h+1}^k\right\rangle\right) \\
&\le\sum_{s'\in\mathcal{S}}\left|\widehat{P}_{s,a,h}^k(s')-P_{s,a,h}(s')\right|\cdot\left|\Delta_{h+1}^k(s')-\left\langle P_{s,a,h},\Delta_{h+1}^k\right\rangle\right|. \tag*{(509)}
\end{aligned} = ( i ) s ′ ∈ S ∑ ( P s , a , h k ( s ′ ) − P s , a , h ( s ′ ) ) Δ h + 1 k ( s ′ ) − { s ′ ∈ S ∑ ( P s , a , h k ( s ′ ) − P s , a , h ( s ′ ) ) } ⟨ P s , a , h , Δ h + 1 k ⟩ = s ′ ∈ S ∑ ( P s , a , h k ( s ′ ) − P s , a , h ( s ′ ) ) ( Δ h + 1 k ( s ′ ) − ⟨ P s , a , h , Δ h + 1 k ⟩ ) ≤ s ′ ∈ S ∑ P s , a , h k ( s ′ ) − P s , a , h ( s ′ ) ⋅ Δ h + 1 k ( s ′ ) − ⟨ P s , a , h , Δ h + 1 k ⟩ . (509)
where (i) follows since ∑ s ′ P ^ s , a , h k ( s ′ ) = ∑ s ′ P s , a , h ( s ′ ) = 1 \sum_{s'}\widehat{P}^{k}_{s,a,h}(s')=\sum_{s'}P_{s,a,h}(s')=1 ∑ s ′ P s , a , h k ( s ′ ) = ∑ s ′ P s , a , h ( s ′ ) = 1 . The Bernstein inequality (cf. Lemma 2.5 ) combined with the union bound tells us that, with probability greater than 1 − 4 S 2 A H K δ ′ 1-4S^{2}AHK\delta' 1 − 4 S 2 A H K δ ′ ,
∣ P ^ s , a , h k ( s ′ ) − P s , a , h ( s ′ ) ∣ = ∣ ⟨ P ^ s , a , h k − P s , a , h , e s ′ ⟩ ∣ ≤ 2 Var P s , a , h ( e s ′ ) log 1 δ ′ N h k ( s , a ) + 2 log 1 δ ′ 3 N h k ( s , a ) ≤ 2 P s , a , h ( s ′ ) log 1 δ ′ N h k ( s , a ) + log 1 δ ′ N h k ( s , a ) . (510) \begin{aligned}
\left|\widehat{P}^{k}_{s,a,h}(s')-P_{s,a,h}(s')\right|=\left|\left\langle\widehat{P}^{k}_{s,a,h}-P_{s,a,h},e_{s'}\right\rangle\right| \\
&\leq\sqrt{\frac{2\operatorname{Var}_{P_{s,a,h}}(e_{s'})\log\frac{1}{\delta'}}{N^{k}_{h}(s,a)}+\frac{2\log\frac{1}{\delta'}}{3N^{k}_{h}(s,a)}} \\
&\leq\sqrt{\frac{2P_{s,a,h}(s')\log\frac{1}{\delta'}}{N^{k}_{h}(s,a)}+\frac{\log\frac{1}{\delta'}}{N^{k}_{h}(s,a)}}. \tag*{(510)}
\end{aligned} P s , a , h k ( s ′ ) − P s , a , h ( s ′ ) = ⟨ P s , a , h k − P s , a , h , e s ′ ⟩ ≤ N h k ( s , a ) 2 Var P s , a , h ( e s ′ ) log δ ′ 1 + 3 N h k ( s , a ) 2 log δ ′ 1 ≤ N h k ( s , a ) 2 P s , a , h ( s ′ ) log δ ′ 1 + N h k ( s , a ) log δ ′ 1 . (510)
holds simultaneously for all ( s , a , s ′ , h , k ) ∈ S × A × S × [ H ] × [ K ] (s,a,s',h,k)\in\mathcal{S}\times\mathcal{A}\times\mathcal{S}\times[H]\times[K] ( s , a , s ′ , h , k ) ∈ S × A × S × [ H ] × [ K ] , where the last inequality uses the fact that Var P s , a , h ( e s ′ ) = P s , a , h ( s ′ ) ( 1 − P s , a , h ( s ′ ) ) ≤ P s , a , h ( s ′ ) \operatorname{Var}_{P_{s,a,h}}(e_{s'})=P_{s,a,h}(s')(1-P_{s,a,h}(s'))\leq P_{s,a,h}(s') Var P s , a , h ( e s ′ ) = P s , a , h ( s ′ ) ( 1 − P s , a , h ( s ′ )) ≤ P s , a , h ( s ′ ) . Substitution into (509) then leads to
⟨ P ^ s , a , h k − P s , a , h , V h + 1 k − V h + 1 ⋆ ⟩ ≤ ∑ s ′ ∈ S ( 2 P s , a , h ( s ′ ) log 1 δ ′ N h k ( s , a ) + log 1 δ ′ N h k ( s , a ) ) ∣ Δ h + 1 k ( s ′ ) − ⟨ P s , a , h , Δ h + 1 k ⟩ ∣ \begin{aligned}
\left\langle\widehat{P}^{k}_{s,a,h}-P_{s,a,h},V^{k}_{h+1}-V^{\star}_{h+1}\right\rangle\\
&\leq\sum_{s'\in\mathcal{S}}\left(\sqrt{\frac{2P_{s,a,h}(s')\log\frac{1}{\delta'}}{N^{k}_{h}(s,a)}+\frac{\log\frac{1}{\delta'}}{N^{k}_{h}(s,a)}}\right)\left|\Delta^{k}_{h+1}(s')-\left\langle P_{s,a,h},\Delta^{k}_{h+1}\right\rangle\right|
\end{aligned} ⟨ P s , a , h k − P s , a , h , V h + 1 k − V h + 1 ⋆ ⟩ ≤ s ′ ∈ S ∑ ( N h k ( s , a ) 2 P s , a , h ( s ′ ) log δ ′ 1 + N h k ( s , a ) log δ ′ 1 ) Δ h + 1 k ( s ′ ) − ⟨ P s , a , h , Δ h + 1 k ⟩
≤ (ii) 2 S log 1 δ ′ N h k ( s , a ) ∑ s ′ ∈ S P s , a , h ( s ′ ) ∣ Δ h + 1 k ( s ′ ) − ⟨ P s , a , h , Δ h + 1 k ⟩ ∣ 2 + ∑ s ′ ∈ S log 1 δ ′ N h k ( s , a ) ∣ Δ h + 1 k ( s ′ ) − ⟨ P s , a , h , Δ h + 1 k ⟩ ∣ \underset{\text{(ii)}}{\leq}\sqrt{\frac{2S\log\frac{1}{\delta'}}{N^{k}_{h}(s,a)}}\sqrt{\sum_{s'\in\mathcal{S}}P_{s,a,h}(s')\left|\Delta^{k}_{h+1}(s')-\left\langle P_{s,a,h},\Delta^{k}_{h+1}\right\rangle\right|^{2}}+\sum_{s'\in\mathcal{S}}\frac{\log\frac{1}{\delta'}}{N^{k}_{h}(s,a)}\left|\Delta^{k}_{h+1}(s')-\left\langle P_{s,a,h},\Delta^{k}_{h+1}\right\rangle\right| (ii) ≤ N h k ( s , a ) 2 S log δ ′ 1 s ′ ∈ S ∑ P s , a , h ( s ′ ) Δ h + 1 k ( s ′ ) − ⟨ P s , a , h , Δ h + 1 k ⟩ 2 + s ′ ∈ S ∑ N h k ( s , a ) log δ ′ 1 Δ h + 1 k ( s ′ ) − ⟨ P s , a , h , Δ h + 1 k ⟩
≤ (iii) 2 S log 1 δ ′ N h k ( s , a ) Var P s , a , h ( V h + 1 k − V h + 1 ⋆ ) + S H log 1 δ ′ N h k ( s , a ) . \underset{\text{(iii)}}{\leq}\sqrt{\frac{2S\log\frac{1}{\delta'}}{N^{k}_{h}(s,a)}}\sqrt{\operatorname{Var}_{P_{s,a,h}}\left(V^{k}_{h+1}-V^{\star}_{h+1}\right)}+\frac{SH\log\frac{1}{\delta'}}{N^{k}_{h}(s,a)}. (iii) ≤ N h k ( s , a ) 2 S log δ ′ 1 Var P s , a , h ( V h + 1 k − V h + 1 ⋆ ) + N h k ( s , a ) S H log δ ′ 1 .
where (ii) follows from the Cauchy-Schwarz inequality, and (iii) results from the definition of Var P s , a , h ( V ) \operatorname{Var}_{P_{s,a,h}}(V) Var P s , a , h ( V ) and the fact that Δ h + 1 k ( s ) ∈ [ 0 , H ] \Delta^{k}_{h+1}(s)\in[0,H] Δ h + 1 k ( s ) ∈ [ 0 , H ] for all s ∈ S s\in\mathcal{S} s ∈ S (since both V h + 1 k ( s ) V^{k}_{h+1}(s) V h + 1 k ( s ) and V h + 1 ⋆ ( s ) V^{\star}_{h+1}(s) V h + 1 ⋆ ( s ) lie in [ 0 , H ] [0,H] [ 0 , H ] ). Repeating similar arguments as in (504) , we arrive at
W 2 ≤ 2 S 2 A H ( log 2 K ) log 1 δ ′ ∑ k , h Var P s h k , a h k , h k ( V h + 1 k − V h + 1 ⋆ ) + 2 S 2 A H 2 ( log 2 K ) log 1 δ ′ ≤ 1 8 ν b S 2 A H T 6 d i f f + 1 8 ν b S 2 A H 2 . (511) \begin{aligned}
W_{2}\leq2\sqrt{S^{2}AH(\log_{2}K)\log\frac{1}{\delta'}}\sqrt{\sum_{k,h}\operatorname{Var}_{P^{k}_{s_{h}^{k},a_{h}^{k},h}}\left(V^{k}_{h+1}-V^{\star}_{h+1}\right)}+2S^{2}AH^{2}(\log_{2}K)\log\frac{1}{\delta'} \\
&\leq\frac{1}{8}\sqrt{\nu_{b}S^{2}AHT^{\mathrm{diff}}_{6}}+\frac{1}{8}\nu_{b}S^{2}AH^{2}. \tag*{(511)}
\end{aligned} W 2 ≤ 2 S 2 A H ( log 2 K ) log δ ′ 1 k , h ∑ Var P s h k , a h k , h k ( V h + 1 k − V h + 1 ⋆ ) + 2 S 2 A H 2 ( log 2 K ) log δ ′ 1 ≤ 8 1 ν b S 2 A H T 6 diff + 8 1 ν b S 2 A H 2 . (511)
with probability at least 1 − 4 S 2 A H K δ ′ 1-4S^{2}AHK\delta' 1 − 4 S 2 A H K δ ′ , where we recall the definition of T 6 d i f f T^{\mathrm{diff}}_{6} T 6 diff in (395) . Here, the last line follows since, according to the property V h + 1 k ≥ V h + 1 ⋆ ≥ V h + 1 π k V^{k}_{h+1}\geq V^{\star}_{h+1}\geq V^{\pi^{k}}_{h+1} V h + 1 k ≥ V h + 1 ⋆ ≥ V h + 1 π k (see Lemma 5.5 ),
Var P s , a , h ( V h + 1 k − V h + 1 ⋆ ) ≤ ⟨ P s , a , h , ( V h + 1 k − V h + 1 ⋆ ) 2 ⟩ ≤ ⟨ P s , a , h , ( V h + 1 k − V h + 1 π k ) 2 ⟩ , \operatorname{Var}_{P_{s,a,h}}\left(V^{k}_{h+1}-V^{\star}_{h+1}\right)\leq\left\langle P_{s,a,h},\left(V^{k}_{h+1}-V^{\star}_{h+1}\right)^{2}\right\rangle\leq\left\langle P_{s,a,h},\left(V^{k}_{h+1}-V^{\pi^{k}}_{h+1}\right)^{2}\right\rangle, Var P s , a , h ( V h + 1 k − V h + 1 ⋆ ) ≤ ⟨ P s , a , h , ( V h + 1 k − V h + 1 ⋆ ) 2 ⟩ ≤ ⟨ P s , a , h , ( V h + 1 k − V h + 1 π k ) 2 ⟩ ,
⟹ ∑ k , h Var P s h k , a h k , h k ( V h + 1 k − V h + 1 ⋆ ) ≤ T 6 d i f f . (512) \Longrightarrow\quad\sum_{k,h}\operatorname{Var}_{P^{k}_{s_{h}^{k},a_{h}^{k},h}}\left(V^{k}_{h+1}-V^{\star}_{h+1}\right)\leq T^{\mathrm{diff}}_{6}. \tag*{(512)} ⟹ k , h ∑ Var P s h k , a h k , h k ( V h + 1 k − V h + 1 ⋆ ) ≤ T 6 diff . (512)
Bounding the quantity W 3 W_{3} W 3 . Akin to the decomposition (396) , we split W 3 W_{3} W 3 into two components:
W 3 = ∑ k , h ⟨ P ^ s h k , a h k , h k − P s h k , a h k , h , ( V h + 1 ⋆ ) 2 ⟩ + ∑ k , h ⟨ P ^ s h k , a h k , h k − P s h k , a h k , h , ( V h + 1 k ) 2 − ( V h + 1 ⋆ ) 2 ⟩ . (513) W_{3}=\sum_{k,h}\left\langle\widehat{P}^{k}_{s_{h}^{k},a_{h}^{k},h}-P_{s_{h}^{k},a_{h}^{k},h},(V^{\star}_{h+1})^{2}\right\rangle+\sum_{k,h}\left\langle\widehat{P}^{k}_{s_{h}^{k},a_{h}^{k},h}-P_{s_{h}^{k},a_{h}^{k},h},(V^{k}_{h+1})^{2}-(V^{\star}_{h+1})^{2}\right\rangle. \tag*{(513)} W 3 = k , h ∑ ⟨ P s h k , a h k , h k − P s h k , a h k , h , ( V h + 1 ⋆ ) 2 ⟩ + k , h ∑ ⟨ P s h k , a h k , h k − P s h k , a h k , h , ( V h + 1 k ) 2 − ( V h + 1 ⋆ ) 2 ⟩ . (513)
The resulting quantities W 4 W_{4} W 4 and W 5 W_{5} W 5 can then be controlled using arguments analogous to those employed for W 1 W_{1} W 1 and W 2 W_{2} W 2 , respectively. Additionally, it is helpful to record the following basic variance bound, quoted from Chen et al. (2021b, Lemma 30).
Lemma 5.22. Let X X X be a random variable, and denote by C max C_{\max} C m a x the largest possible magnitude of X X X . Then we have Var ( X 2 ) ≤ 4 C max 2 Var ( X ) \operatorname{Var}(X^2) \le4C_{\max}^2\operatorname{Var}(X) Var ( X 2 ) ≤ 4 C m a x 2 Var ( X ) and Var ( ( X − E [ X ] ) 2 ) ≤ C max 2 Var ( X ) \operatorname{Var}((X-\mathbb{E}[X])^2) \le C_{\max}^2\operatorname{Var}(X) Var (( X − E [ X ] ) 2 ) ≤ C m a x 2 Var ( X ) .
Proof of Lemma 5.22 . From the elementary fact Var ( U + V ) ≤ 2 Var ( U ) + 2 Var ( V ) \operatorname{Var}(U+V) \le2\operatorname{Var}(U)+2\operatorname{Var}(V) Var ( U + V ) ≤ 2 Var ( U ) + 2 Var ( V ) , we can derive
Var ( X 2 ) ≤ 2 Var ( ( X − E [ X ] ) X ) + 2 Var ( E [ X ] X ) ≤ 2 E [ ( X − E [ X ] ) 2 X 2 ] + 2 ( E [ X ] ) 2 Var ( X ) ≤ 2 C max 2 E [ ( X − E [ X ] ) 2 ] + 2 C max 2 Var ( X ) = 4 C max 2 Var ( X ) . \begin{aligned}
\operatorname{Var}(X^2) &\le2\operatorname{Var}((X-\mathbb{E}[X])X)+2\operatorname{Var}(\mathbb{E}[X]X)\\
&\le2\mathbb{E}[(X-\mathbb{E}[X])^2X^2]+2(\mathbb{E}[X])^2\operatorname{Var}(X)\\
&\le2C_{\max}^2\mathbb{E}[(X-\mathbb{E}[X])^2]+2C_{\max}^2\operatorname{Var}(X)=4C_{\max}^2\operatorname{Var}(X).
\end{aligned} Var ( X 2 ) ≤ 2 Var (( X − E [ X ]) X ) + 2 Var ( E [ X ] X ) ≤ 2 E [( X − E [ X ] ) 2 X 2 ] + 2 ( E [ X ] ) 2 Var ( X ) ≤ 2 C m a x 2 E [( X − E [ X ] ) 2 ] + 2 C m a x 2 Var ( X ) = 4 C m a x 2 Var ( X ) .
Similarly, we have
Var ( ( X − E [ X ] ) 2 ) ≤ E [ ( X − E [ X ] ) 4 ] ≤ C max 2 E [ ( X − E [ X ] ) 2 ] = C max 2 Var ( X ) \operatorname{Var}((X-\mathbb{E}[X])^2)\le\mathbb{E}[(X-\mathbb{E}[X])^4]\le C_{\max}^2\mathbb{E}[(X-\mathbb{E}[X])^2]=C_{\max}^2\operatorname{Var}(X) Var (( X − E [ X ] ) 2 ) ≤ E [( X − E [ X ] ) 4 ] ≤ C m a x 2 E [( X − E [ X ] ) 2 ] = C m a x 2 Var ( X )
as claimed.
Regarding the term W 4 W_4 W 4 in (513) , repeating the arguments in (504) yields, with probability greater than 1 − 4 S A H K δ ′ 1-4SAHK\delta' 1 − 4 S A H K δ ′ , that
W 4 ≤ 2 S A H ( log 2 K ) log 1 δ ′ ∑ k , h Var P h s k , a k , h ( ( V h + 1 ⋆ ) 2 ) + 2 S A H 3 ( log 2 K ) log 1 δ ′ W_4\le2\sqrt{SAH(\log_2 K)\log\frac{1}{\delta'}\sum_{k,h}\operatorname{Var}_{P_h^{s_k,a_k,h}}\left((V_{h+1}^{\star})^2\right)}+2SAH^3(\log_2 K)\log\frac{1}{\delta'} W 4 ≤ 2 S A H ( log 2 K ) log δ ′ 1 k , h ∑ Var P h s k , a k , h ( ( V h + 1 ⋆ ) 2 ) + 2 S A H 3 ( log 2 K ) log δ ′ 1
≤ 4 S A H 3 T 6 o p t ( log 2 K ) log 1 δ ′ + 2 S A H 3 ( log 2 K ) log 1 δ ′ ≤ 1 16 ι b S A H 3 T 6 l o w + 1 16 ι b S A H 3 T 6 d i f f + 1 16 ι b S A H 3 . \begin{aligned}
\le4\sqrt{SAH^3\mathcal{T}_6^{\mathrm{opt}}(\log_2 K)\log\frac{1}{\delta'}}+2SAH^3(\log_2 K)\log\frac{1}{\delta'} \\
&\le\frac{1}{16}\sqrt{\iota_bSAH^3\mathcal{T}_6^{\mathrm{low}}}+\frac{1}{16}\sqrt{\iota_bSAH^3\mathcal{T}_6^{\mathrm{diff}}}+\frac{1}{16}\iota_bSAH^3.
\end{aligned} ≤ 4 S A H 3 T 6 opt ( log 2 K ) log δ ′ 1 + 2 S A H 3 ( log 2 K ) log δ ′ 1 ≤ 16 1 ι b S A H 3 T 6 low + 16 1 ι b S A H 3 T 6 diff + 16 1 ι b S A H 3 .
Here, the last inequality follows from (5.122), whereas the penultimate inequality applies Lemma 5.22 and the property ∥ V h + 1 ⋆ ∥ ∞ ≤ H \lVert V_{h+1}^{\star}\rVert_{\infty}\le H ∥ V h + 1 ⋆ ∥ ∞ ≤ H to obtain
Var P h s k , a k , h ( ( V h + 1 ⋆ ) 2 ) ≤ 4 H 2 Var P h s k , a k , h ( V h + 1 ⋆ ) . \operatorname{Var}_{P_h^{s_k,a_k,h}}\left((V_{h+1}^{\star})^2\right)\le4H^2\operatorname{Var}_{P_h^{s_k,a_k,h}}(V_{h+1}^{\star}). Var P h s k , a k , h ( ( V h + 1 ⋆ ) 2 ) ≤ 4 H 2 Var P h s k , a k , h ( V h + 1 ⋆ ) .
When it comes to the term W 5 W_5 W 5 in (513) , repeat the arguments for (511) to derive
W 5 ≤ 2 S 2 A H ( log 2 K ) log 1 δ ′ ∑ k , h Var P s , a , h ( ( V h + 1 k ) 2 − ( V h + 1 ⋆ ) 2 ) + 2 S 2 A H 3 ( log 2 K ) log 1 δ ′ . W_5\le2\sqrt{S^2AH(\log_2 K)\log\frac{1}{\delta'}\sum_{k,h}\operatorname{Var}_{P_{s,a,h}}\left((V_{h+1}^k)^2-(V_{h+1}^{\star})^2\right)}+2S^2AH^3(\log_2 K)\log\frac{1}{\delta'}. W 5 ≤ 2 S 2 A H ( log 2 K ) log δ ′ 1 k , h ∑ Var P s , a , h ( ( V h + 1 k ) 2 − ( V h + 1 ⋆ ) 2 ) + 2 S 2 A H 3 ( log 2 K ) log δ ′ 1 .
Furthermore, recalling the property H ≥ V h + 1 k ( s ) ≥ V h + 1 ⋆ ( s ) ≥ V h + 1 π k ( s ) ≥ 0 H\ge V_{h+1}^k(s)\ge V_{h+1}^{\star}(s)\ge V_{h+1}^{\pi_k}(s)\ge0 H ≥ V h + 1 k ( s ) ≥ V h + 1 ⋆ ( s ) ≥ V h + 1 π k ( s ) ≥ 0 that holds for all s ∈ S s\in\mathcal{S} s ∈ S (see Lemma 5.5 ), one can deduce that
Var P s , a , h ( ( V h + 1 k ) 2 − ( V h + 1 ⋆ ) 2 ) ≤ ⟨ P s , a , h , ( ( V h + 1 k ) 2 − ( V h + 1 ⋆ ) 2 ) 2 ⟩ ≤ ⟨ P s , a , h , ( ( V h + 1 k ) 2 − ( V h + 1 π k ) 2 ) 2 ⟩ = ⟨ P s , a , h , ( ( V h + 1 k − V h + 1 π k ) ∘ ( V h + 1 k + V h + 1 π k ) ) 2 ⟩ ≤ 4 H 2 ⟨ P s , a , h , ( V h + 1 k − V h + 1 π k ) 2 ⟩ . \begin{aligned}
\operatorname{Var}_{P_{s,a,h}}\left((V_{h+1}^k)^2-(V_{h+1}^{\star})^2\right)
&\le\left\langle P_{s,a,h},\left((V_{h+1}^k)^2-(V_{h+1}^{\star})^2\right)^2\right\rangle\\
&\le\left\langle P_{s,a,h},\left((V_{h+1}^k)^2-(V_{h+1}^{\pi_k})^2\right)^2\right\rangle\\
&=\left\langle P_{s,a,h},\left((V_{h+1}^k-V_{h+1}^{\pi_k})\circ(V_{h+1}^k+V_{h+1}^{\pi_k})\right)^2\right\rangle\\
&\le4H^2\left\langle P_{s,a,h},(V_{h+1}^k-V_{h+1}^{\pi_k})^2\right\rangle.
\end{aligned} Var P s , a , h ( ( V h + 1 k ) 2 − ( V h + 1 ⋆ ) 2 ) ≤ ⟨ P s , a , h , ( ( V h + 1 k ) 2 − ( V h + 1 ⋆ ) 2 ) 2 ⟩ ≤ ⟨ P s , a , h , ( ( V h + 1 k ) 2 − ( V h + 1 π k ) 2 ) 2 ⟩ = ⟨ P s , a , h , ( ( V h + 1 k − V h + 1 π k ) ∘ ( V h + 1 k + V h + 1 π k ) ) 2 ⟩ ≤ 4 H 2 ⟨ P s , a , h , ( V h + 1 k − V h + 1 π k ) 2 ⟩ .
Taking this inequality together with the definition (5.35), we arrive at
W 5 ≤ 4 S 2 A H 3 T 6 d i f f ( log 2 K ) log 1 δ ′ + 2 S 2 A H 3 ( log 2 K ) log 1 δ ′ ≤ 1 16 ι b S 2 A H 3 T 6 d i f f + 1 16 ι b S 2 A H 3 . \begin{aligned}
W_5\le4\sqrt{S^2AH^3\mathcal{T}_6^{\mathrm{diff}}(\log_2 K)\log\frac{1}{\delta'}+2S^2AH^3(\log_2 K)\log\frac{1}{\delta'}} \\
&\le\frac{1}{16}\sqrt{\iota_bS^2AH^3\mathcal{T}_6^{\mathrm{diff}}}+\frac{1}{16}\iota_bS^2AH^3.
\end{aligned} W 5 ≤ 4 S 2 A H 3 T 6 diff ( log 2 K ) log δ ′ 1 + 2 S 2 A H 3 ( log 2 K ) log δ ′ 1 ≤ 16 1 ι b S 2 A H 3 T 6 diff + 16 1 ι b S 2 A H 3 .
Substituting the preceding bounds on W 4 W_4 W 4 and W 5 W_5 W 5 into (513) , we conclude that
W 3 ≤ W 4 + W 5 ≤ 1 8 ι b S A H 3 T 6 l o w + 1 8 ι b S 2 A H 3 T 6 d i f f + 1 8 ι b S 2 A H 3 . W_3\le W_4+W_5\le\frac{1}{8}\sqrt{\iota_bSAH^3\mathcal{T}_6^{\mathrm{low}}}+\frac{1}{8}\sqrt{\iota_bS^2AH^3\mathcal{T}_6^{\mathrm{diff}}}+\frac{1}{8}\iota_bS^2AH^3. W 3 ≤ W 4 + W 5 ≤ 8 1 ι b S A H 3 T 6 low + 8 1 ι b S 2 A H 3 T 6 diff + 8 1 ι b S 2 A H 3 .
Proof of Lemma 5.8
Bounding the term T 2 T_2 T 2 . We begin by analyzing T 2 T_2 T 2 (cf. (390) ). Recalling the definition of the bonus term (5.20), we have
T 2 = ∑ k = 1 K ∑ h = 1 H b h k ( s h k , a h k ) = c 1 ∑ k , h Var P s h k , a h k ( V h + 1 k ) log 1 δ ′ N h k ( s h k , a h k ) + c 2 ∑ k , h H log 1 δ ′ N h k ( s h k , a h k ) . (514) T_2=\sum_{k=1}^{K}\sum_{h=1}^{H}b_h^k(s_h^k,a_h^k)=c_1\sum_{k,h}\sqrt{\frac{\operatorname{Var}_{P_{s_h^k,a_h^k}}(V_{h+1}^k)\log\frac{1}{\delta'}}{N_h^k(s_h^k,a_h^k)}}+c_2\sum_{k,h}\frac{H\log\frac{1}{\delta'}}{N_h^k(s_h^k,a_h^k)}. \tag*{(514)} T 2 = k = 1 ∑ K h = 1 ∑ H b h k ( s h k , a h k ) = c 1 k , h ∑ N h k ( s h k , a h k ) Var P s h k , a h k ( V h + 1 k ) log δ ′ 1 + c 2 k , h ∑ N h k ( s h k , a h k ) H log δ ′ 1 . (514)
Applying the Cauchy–Schwarz inequality to the first sum in (514) yields
T 2 ≤ c 1 ∑ k , h log 1 δ ′ N h k ( s h k , a h k ) ∑ k , h Var P s h k , a h k ( V h + 1 k ) + c 2 H ( log 1 δ ′ ) ∑ k , h 1 N h k ( s h k , a h k ) = c 1 ∑ k , h log 1 δ ′ N h k ( s h k , a h k ) T 5 + c 2 H ( log 1 δ ′ ) ∑ k , h 1 N h k ( s h k , a h k ) ≤ c 1 2 S A H ( log 2 K ) ( log 1 δ ′ ) T 5 + 2 c 2 S A H 2 ( log 2 K ) log 1 δ ′ ≤ 1 2 S A H T 5 ι b + 1 2 S A H 2 ι b , \begin{aligned}
T_2\leq c_1\sqrt{\sum_{k,h}\frac{\log\frac{1}{\delta'}}{N_h^k(s_h^k,a_h^k)}}\sqrt{\sum_{k,h}\operatorname{Var}_{P_{s_h^k,a_h^k}}(V_{h+1}^k)}+c_2H\left(\log\frac{1}{\delta'}\right)\sum_{k,h}\frac{1}{N_h^k(s_h^k,a_h^k)} \\
&=c_1\sqrt{\sum_{k,h}\frac{\log\frac{1}{\delta'}}{N_h^k(s_h^k,a_h^k)}}\sqrt{T_5}+c_2H\left(\log\frac{1}{\delta'}\right)\sum_{k,h}\frac{1}{N_h^k(s_h^k,a_h^k)} \\
&\leq c_1\sqrt{2SAH(\log_2 K)\left(\log\frac{1}{\delta'}\right)T_5}+2c_2SAH^2(\log_2 K)\log\frac{1}{\delta'} \\
&\leq\frac{1}{2}\sqrt{SAHT_5\iota_b}+\frac{1}{2}SAH^2\iota_b,
\end{aligned} T 2 ≤ c 1 k , h ∑ N h k ( s h k , a h k ) log δ ′ 1 k , h ∑ Var P s h k , a h k ( V h + 1 k ) + c 2 H ( log δ ′ 1 ) k , h ∑ N h k ( s h k , a h k ) 1 = c 1 k , h ∑ N h k ( s h k , a h k ) log δ ′ 1 T 5 + c 2 H ( log δ ′ 1 ) k , h ∑ N h k ( s h k , a h k ) 1 ≤ c 1 2 S A H ( log 2 K ) ( log δ ′ 1 ) T 5 + 2 c 2 S A H 2 ( log 2 K ) log δ ′ 1 ≤ 2 1 S A H T 5 ι b + 2 1 S A H 2 ι b ,
where the second relation uses the definition (391) of T 5 T_5 T 5 , the penultimate line invokes the visitation count property (382) , and the last line holds as long as ι b ≥ 8 max { c 1 2 , c 2 } ( log 2 K ) log 1 δ ′ \iota_b\geq8\max\{c_1^2,c_2\}(\log_2 K)\log\frac{1}{\delta'} ι b ≥ 8 max { c 1 2 , c 2 } ( log 2 K ) log δ ′ 1 .
Bounding the term T 3 T_3 T 3 . Next, we bound the term T 3 T_3 T 3 defined in (390) . Recall that, by definition, the value function estimate V h + 1 k V_{h+1}^k V h + 1 k is computed prior to the start of episode k k k . Consequently, conditional on the state-action pair ( s h k , a h k ) (s_h^k,a_h^k) ( s h k , a h k ) and the history up to the beginning of episode k k k , the next-state indicator vector e s h + 1 k e_{s_{h+1}^k} e s h + 1 k is statistically independent of V h + 1 k V_{h+1}^k V h + 1 k and has conditional mean P s h k , a h k , h P_{s_h^k,a_h^k,h} P s h k , a h k , h . Furthermore, since all entries of V h + 1 k V_{h+1}^k V h + 1 k lie within [ 0 , H ] [0,H] [ 0 , H ] , one has the deterministic bound:
∣ ⟨ P s h k , a h k , h − e s h + 1 k , V h + 1 k ⟩ ∣ ≤ H . \left|\left\langle P_{s_h^k,a_h^k,h}-e_{s_{h+1}^k},V_{h+1}^k\right\rangle\right|\leq H. ⟨ P s h k , a h k , h − e s h + 1 k , V h + 1 k ⟩ ≤ H .
Additionally, it is easily seen that T 6 u c b T_6^{\mathrm{ucb}} T 6 ucb (cf. (392) ) obeys the trivial upper bound
T 6 u c b ≤ ∑ k = 1 K ∑ h = 1 H ⟨ P s h k , a h k , h , ( V h + 1 k ) 2 ⟩ ≤ K H 3 . (515) T_6^{\mathrm{ucb}}\leq\sum_{k=1}^{K}\sum_{h=1}^{H}\left\langle P_{s_h^k,a_h^k,h},(V_{h+1}^k)^2\right\rangle\leq KH^3. \tag*{(515)} T 6 ucb ≤ k = 1 ∑ K h = 1 ∑ H ⟨ P s h k , a h k , h , ( V h + 1 k ) 2 ⟩ ≤ K H 3 . (515)
Therefore, using the definition of T 6 u c b T_6^{\mathrm{ucb}} T 6 ucb in (392) and its trivial upper bound (515) , one can invoke the Freedman inequality (see Lemma 2.9 with K 0 = log 2 ( K H ) K_0=\log_2(KH) K 0 = log 2 ( K H ) ) to show that, with probability at least 1 − 2 δ ′ log 2 ( K H ) 1-2\delta'\log_2(KH) 1 − 2 δ ′ log 2 ( K H ) ,
∣ T 3 ∣ ≤ 8 max { T 6 u c b , H 2 } log 1 δ ′ + 4 3 H log 1 δ ′ ≤ 8 T 6 u c b log 1 δ ′ + 8 H log 1 δ ′ + 4 3 H log 1 δ ′ ≤ 8 T 6 u c b log 1 δ ′ + 5 H log 1 δ ′ \begin{aligned}
|T_3|\leq\sqrt{8\max\{T_6^{\mathrm{ucb}},H^2\}\log\frac{1}{\delta'}}+\frac{4}{3}H\log\frac{1}{\delta'} \\
&\leq\sqrt{8T_6^{\mathrm{ucb}}\log\frac{1}{\delta'}}+\sqrt{8}H\log\frac{1}{\delta'}+\frac{4}{3}H\log\frac{1}{\delta'} \\
&\leq\sqrt{8T_6^{\mathrm{ucb}}\log\frac{1}{\delta'}}+5H\log\frac{1}{\delta'}
\end{aligned} ∣ T 3 ∣ ≤ 8 max { T 6 ucb , H 2 } log δ ′ 1 + 3 4 H log δ ′ 1 ≤ 8 T 6 ucb log δ ′ 1 + 8 H log δ ′ 1 + 3 4 H log δ ′ 1 ≤ 8 T 6 ucb log δ ′ 1 + 5 H log δ ′ 1
≤ 4 T 6 l o w log 1 δ ′ + 4 T 6 d i f f log 1 δ ′ + 5 H log 1 δ ′ , \leq4\sqrt{T_6^{\mathrm{low}}\log\frac{1}{\delta'}}+4\sqrt{T_6^{\mathrm{diff}}\log\frac{1}{\delta'}}+5H\log\frac{1}{\delta'}, ≤ 4 T 6 low log δ ′ 1 + 4 T 6 diff log δ ′ 1 + 5 H log δ ′ 1 ,
where the second line uses the elementary inequality max { a , b } ≤ a + b \sqrt{\max\{a,b\}}\leq\sqrt{a}+\sqrt{b} max { a , b } ≤ a + b , and the last inequality arises from (5.123).
Bounding the term T 4 T_4 T 4 . We now turn attention to the term T 4 T_4 T 4 , i.e.,
T 4 = ∑ k = 1 K ( ∑ h = 1 H r h ( s h k , a h k ) − V 1 π k ( s 1 k ) ) (516) T_4 = \sum_{k=1}^{K}\left(\sum_{h=1}^{H} r_h(s_h^k,a_h^k)-V_1^{\pi^k}(s_1^k)\right) \tag*{(516)} T 4 = k = 1 ∑ K ( h = 1 ∑ H r h ( s h k , a h k ) − V 1 π k ( s 1 k ) ) (516)
Note that conditional on π k \pi^k π k , E k E_k E k is a zero-mean random variable bounded in magnitude by H H H . An application of the Azuma-Hoeffding inequality (cf. Lemma 2.8 ) guarantees that, with probability exceeding 1 − 2 δ ′ 1-2\delta' 1 − 2 δ ′ ,
∣ T 4 ∣ = ∣ ∑ k = 1 K E k ∣ ≤ 2 K H 2 log 1 δ ′ . \lvert T_4\rvert=\left\lvert\sum_{k=1}^{K}E_k\right\rvert\leq\sqrt{2KH^2\log\frac{1}{\delta'}}. ∣ T 4 ∣ = k = 1 ∑ K E k ≤ 2 K H 2 log δ ′ 1 .
Proof of Lemma 5.9
To facilitate the analysis, we introduce the following auxiliary quantity:
T 8 : = ∑ k = 1 K ∑ h = 1 H ⟨ P s h k , a h k , h − e s h + 1 k , ( V h + 1 k ) 2 ⟩ (517) T_8:=\sum_{k=1}^{K}\sum_{h=1}^{H}\left\langle P_{s_h^k,a_h^k,h}-e_{s_{h+1}^k},(V_{h+1}^k)^2\right\rangle \tag*{(517)} T 8 := k = 1 ∑ K h = 1 ∑ H ⟨ P s h k , a h k , h − e s h + 1 k , ( V h + 1 k ) 2 ⟩ (517)
Since this quantity will appear in the upper bounds on several other terms, we begin by deriving an upper bound for T 8 T_8 T 8 .
Bounding the term T 8 T_8 T 8 . According to Lemma 5.22 , Var ( X 2 ) ≤ 4 H 2 Var ( X ) \operatorname{Var}(X^2)\leq4H^2\operatorname{Var}(X) Var ( X 2 ) ≤ 4 H 2 Var ( X ) holds for any random variable X X X with support on [ − H , H ] [-H,H] [ − H , H ] . This implies that
Var P s h k , a h k , h ( ( V h + 1 k ) 2 ) ≤ 4 H 2 Var P s h k , a h k , h ( V h + 1 k ) . (518) \operatorname{Var}_{P_{s_h^k,a_h^k,h}}\left((V_{h+1}^k)^2\right)\leq4H^2\operatorname{Var}_{P_{s_h^k,a_h^k,h}}\left(V_{h+1}^k\right). \tag*{(518)} Var P s h k , a h k , h ( ( V h + 1 k ) 2 ) ≤ 4 H 2 Var P s h k , a h k , h ( V h + 1 k ) . (518)
Therefore, invoking Freedman’s inequality (see Lemma 2.9 with K 0 = log 2 ( K H ) K_0=\log_2(KH) K 0 = log 2 ( K H ) ) together with the trivial bound Var P s h k , a h k , h ( ( V h + 1 k ) 2 ) ≤ H 4 \operatorname{Var}_{P_{s_h^k,a_h^k,h}}\left((V_{h+1}^k)^2\right)\leq H^4 Var P s h k , a h k , h ( ( V h + 1 k ) 2 ) ≤ H 4 reveals that, with probability at least 1 − 2 δ ′ log 2 ( K H ) 1-2\delta'\log_2(KH) 1 − 2 δ ′ log 2 ( K H ) ,
∣ T 8 ∣ ≤ 8 max { ∑ k , h Var P s h k , a h k , h ( ( V h + 1 k ) 2 ) , H 4 } log 1 δ ′ + 4 3 H 2 log 1 δ ′ ≤ 8 max { ∑ k , h Var P s h k , a h k , h ( ( V h + 1 k ) 2 ) , H 4 } log 1 δ ′ + 8 H 4 log 1 δ ′ + 4 3 H 2 log 1 δ ′ ≤ 8 ∑ k , h Var P s h k , a h k , h ( ( V h + 1 k ) 2 ) log 1 δ ′ + 5 H 2 log 1 δ ′ (519) \begin{aligned}
\lvert T_8\rvert
&\leq\sqrt{8\max\left\{\sum_{k,h}\operatorname{Var}_{P_{s_h^k,a_h^k,h}}\left((V_{h+1}^k)^2\right),H^4\right\}\log\frac{1}{\delta'}+\frac{4}{3}H^2\log\frac{1}{\delta'}}\\
&\leq\sqrt{8\max\left\{\sum_{k,h}\operatorname{Var}_{P_{s_h^k,a_h^k,h}}\left((V_{h+1}^k)^2\right),H^4\right\}\log\frac{1}{\delta'}+\sqrt{8H^4\log\frac{1}{\delta'}}+\frac{4}{3}H^2\log\frac{1}{\delta'}}\\
&\leq\sqrt{8\sum_{k,h}\operatorname{Var}_{P_{s_h^k,a_h^k,h}}\left((V_{h+1}^k)^2\right)\log\frac{1}{\delta'}+5H^2\log\frac{1}{\delta'}} \tag*{(519)}
\end{aligned} ∣ T 8 ∣ ≤ 8 max ⎩ ⎨ ⎧ k , h ∑ Var P s h k , a h k , h ( ( V h + 1 k ) 2 ) , H 4 ⎭ ⎬ ⎫ log δ ′ 1 + 3 4 H 2 log δ ′ 1 ≤ 8 max ⎩ ⎨ ⎧ k , h ∑ Var P s h k , a h k , h ( ( V h + 1 k ) 2 ) , H 4 ⎭ ⎬ ⎫ log δ ′ 1 + 8 H 4 log δ ′ 1 + 3 4 H 2 log δ ′ 1 ≤ 8 k , h ∑ Var P s h k , a h k , h ( ( V h + 1 k ) 2 ) log δ ′ 1 + 5 H 2 log δ ′ 1 (519)
≤ 32 H 2 ∑ k , h Var P s h k , a h k , h ( V h + 1 k ) log 1 δ ′ + 5 H 2 log 1 δ ′ = 32 H 2 T 6 u c b log 1 δ ′ + 5 H 2 log 1 δ ′ , (520) \begin{aligned}
&\leq\sqrt{32H^2\sum_{k,h}\operatorname{Var}_{P_{s_h^k,a_h^k,h}}\left(V_{h+1}^k\right)\log\frac{1}{\delta'}+5H^2\log\frac{1}{\delta'}}\\
&=\sqrt{32H^2T_6^{\mathrm{ucb}}\log\frac{1}{\delta'}+5H^2\log\frac{1}{\delta'}}, \tag*{(520)}
\end{aligned} ≤ 32 H 2 k , h ∑ Var P s h k , a h k , h ( V h + 1 k ) log δ ′ 1 + 5 H 2 log δ ′ 1 = 32 H 2 T 6 ucb log δ ′ 1 + 5 H 2 log δ ′ 1 , (520)
where the second line is valid since max { a , b } ≤ a + b \sqrt{\max\{a,b\}}\leq\sqrt{a}+\sqrt{b} max { a , b } ≤ a + b , the penultimate line follows from (518) , and the last identity uses the definition (392) of T 6 u c b T_6^{\mathrm{ucb}} T 6 ucb .
Bounding the term T 5 T_5 T 5 (cf. (391) ). Regarding the term T 5 T_5 T 5 , expanding the variance terms gives
T 5 = ∑ k = 1 K ∑ h = 1 H ( ⟨ P ^ s h k , a h k , h k , ( V h + 1 k ) 2 ⟩ − ⟨ P ^ s h k , a h k , h k , V h + 1 k ⟩ 2 ) = ∑ k = 1 K ∑ h = 1 H ⟨ P ^ s h k , a h k , h k − P s h k , a h k , h , ( V h + 1 k ) 2 ⟩ + ∑ k = 1 K ∑ h = 1 H ⟨ P s h k , a h k , h − e s h + 1 k , ( V h + 1 k ) 2 ⟩ + ∑ k = 1 K ∑ h = 1 H ( V h + 1 k ( s h + 1 k ) ) 2 − ∑ k = 1 K ∑ h = 1 H ( ⟨ P ^ s h k , a h k , h k , V h + 1 k ⟩ ) 2 = W 3 + T 8 + ∑ k = 1 K ∑ h = 1 H ( V h + 1 k ( s h + 1 k ) ) 2 − ∑ k = 1 K ∑ h = 1 H ( ⟨ P ^ s h k , a h k , h k , V h + 1 k ⟩ ) 2 ≤ W 3 + T 8 + ∑ k = 1 K ∑ h = 2 H ( V h k ( s h k ) ) 2 − ∑ k = 1 K ∑ h = 2 H ( ⟨ P ^ s h k , a h k , h k , V h + 1 k ⟩ ) 2 , (521) \begin{aligned}
T_5
&= \sum_{k=1}^{K} \sum_{h=1}^{H} \left(\left\langle\widehat{P}^{k}_{s_h^k,a_h^k,h},(V^k_{h+1})^2\right\rangle-\left\langle\widehat{P}^{k}_{s_h^k,a_h^k,h},V^k_{h+1}\right\rangle^2\right)\\
&= \sum_{k=1}^{K} \sum_{h=1}^{H}\left\langle\widehat{P}^{k}_{s_h^k,a_h^k,h}-P_{s_h^k,a_h^k,h},(V^k_{h+1})^2\right\rangle+\sum_{k=1}^{K} \sum_{h=1}^{H}\left\langle P_{s_h^k,a_h^k,h}-e^k_{s_{h+1}},(V^k_{h+1})^2\right\rangle\\
&\quad+\sum_{k=1}^{K} \sum_{h=1}^{H}(V^k_{h+1}(s^k_{h+1}))^2-\sum_{k=1}^{K} \sum_{h=1}^{H}\left(\left\langle\widehat{P}^{k}_{s_h^k,a_h^k,h},V^k_{h+1}\right\rangle\right)^2\\
&= W_3+T_8+\sum_{k=1}^{K} \sum_{h=1}^{H}(V^k_{h+1}(s^k_{h+1}))^2-\sum_{k=1}^{K} \sum_{h=1}^{H}\left(\left\langle\widehat{P}^{k}_{s_h^k,a_h^k,h},V^k_{h+1}\right\rangle\right)^2\\
&\leq W_3+T_8+\sum_{k=1}^{K} \sum_{h=2}^{H}(V^k_h(s^k_h))^2-\sum_{k=1}^{K} \sum_{h=2}^{H}\left(\left\langle\widehat{P}^{k}_{s_h^k,a_h^k,h},V^k_{h+1}\right\rangle\right)^2, \tag*{(521)}
\end{aligned} T 5 = k = 1 ∑ K h = 1 ∑ H ( ⟨ P s h k , a h k , h k , ( V h + 1 k ) 2 ⟩ − ⟨ P s h k , a h k , h k , V h + 1 k ⟩ 2 ) = k = 1 ∑ K h = 1 ∑ H ⟨ P s h k , a h k , h k − P s h k , a h k , h , ( V h + 1 k ) 2 ⟩ + k = 1 ∑ K h = 1 ∑ H ⟨ P s h k , a h k , h − e s h + 1 k , ( V h + 1 k ) 2 ⟩ + k = 1 ∑ K h = 1 ∑ H ( V h + 1 k ( s h + 1 k ) ) 2 − k = 1 ∑ K h = 1 ∑ H ( ⟨ P s h k , a h k , h k , V h + 1 k ⟩ ) 2 = W 3 + T 8 + k = 1 ∑ K h = 1 ∑ H ( V h + 1 k ( s h + 1 k ) ) 2 − k = 1 ∑ K h = 1 ∑ H ( ⟨ P s h k , a h k , h k , V h + 1 k ⟩ ) 2 ≤ W 3 + T 8 + k = 1 ∑ K h = 2 ∑ H ( V h k ( s h k ) ) 2 − k = 1 ∑ K h = 2 ∑ H ( ⟨ P s h k , a h k , h k , V h + 1 k ⟩ ) 2 , (521)
where W 3 W_3 W 3 and T 8 T_8 T 8 are defined in (397) and (517) , respectively. Here, in the last line we reindex the first sum (by replacing h + 1 h+1 h + 1 with h h h ) and make use of the fact that V H + 1 k = 0 V^k_{H+1}=0 V H + 1 k = 0 .
To continue, we make the observation that
( V h k ( s h k ) ) 2 − ( ⟨ P ^ s h k , a h k , h k , V h + 1 k ⟩ ) 2 = ( V h k ( s h k ) + ⟨ P ^ s h k , a h k , h k , V h + 1 k ⟩ ) ( V h k ( s h k ) − ⟨ P ^ s h k , a h k , h k , V h + 1 k ⟩ ) ≤ 2 H max { V h k ( s h k ) − ⟨ P ^ s h k , a h k , h k , V h + 1 k ⟩ , 0 } = 2 H max { Q h k ( s h k , a h k ) − ⟨ P ^ s h k , a h k , h k , V h + 1 k ⟩ , 0 } , \begin{aligned}
(V^k_h(s^k_h))^2-\left(\left\langle\widehat{P}^{k}_{s_h^k,a_h^k,h},V^k_{h+1}\right\rangle\right)^2
&=\left(V^k_h(s^k_h)+\left\langle\widehat{P}^{k}_{s_h^k,a_h^k,h},V^k_{h+1}\right\rangle\right)\left(V^k_h(s^k_h)-\left\langle\widehat{P}^{k}_{s_h^k,a_h^k,h},V^k_{h+1}\right\rangle\right)\\
&\leq2H\max\left\{V^k_h(s^k_h)-\left\langle\widehat{P}^{k}_{s_h^k,a_h^k,h},V^k_{h+1}\right\rangle,0\right\}\\
&=2H\max\left\{Q^k_h(s^k_h,a^k_h)-\left\langle\widehat{P}^{k}_{s_h^k,a_h^k,h},V^k_{h+1}\right\rangle,0\right\},
\end{aligned} ( V h k ( s h k ) ) 2 − ( ⟨ P s h k , a h k , h k , V h + 1 k ⟩ ) 2 = ( V h k ( s h k ) + ⟨ P s h k , a h k , h k , V h + 1 k ⟩ ) ( V h k ( s h k ) − ⟨ P s h k , a h k , h k , V h + 1 k ⟩ ) ≤ 2 H max { V h k ( s h k ) − ⟨ P s h k , a h k , h k , V h + 1 k ⟩ , 0 } = 2 H max { Q h k ( s h k , a h k ) − ⟨ P s h k , a h k , h k , V h + 1 k ⟩ , 0 } ,
where the inequality holds since 0 ≤ V h k ( s ) ≤ H 0\leq V^k_h(s)\leq H 0 ≤ V h k ( s ) ≤ H for all s ∈ S s\in\mathcal{S} s ∈ S , and the last identity holds since, due to the greedy policy choice, V h k ( s h k ) = max a Q h k ( s h k , a ) = Q h k ( s h k , a h k ) V^k_h(s^k_h)=\max_a Q^k_h(s^k_h,a)=Q^k_h(s^k_h,a^k_h) V h k ( s h k ) = max a Q h k ( s h k , a ) = Q h k ( s h k , a h k ) . Substitution back into (521) yields
T 5 ≤ W 3 + T 8 + 2 H ∑ k = 1 K ∑ h = 2 H max { Q h k ( s h k , a h k ) − ⟨ P ^ s h k , a h k , h k , V h + 1 k ⟩ , 0 } ≤ W 3 + T 8 + 2 H ∑ k = 1 K ∑ h = 2 H b h k ( s h k , a h k ) + 2 H ∑ k = 1 K ∑ h = 2 H r h ( s h k , a h k ) ≤ W 3 + T 8 + 2 H T 2 + 2 K H 2 , \begin{aligned}
T_5
&\leq W_3+T_8+2H\sum_{k=1}^{K}\sum_{h=2}^{H}\max\left\{Q^k_h(s^k_h,a^k_h)-\left\langle\widehat{P}^{k}_{s_h^k,a_h^k,h},V^k_{h+1}\right\rangle,0\right\}\\
&\leq W_3+T_8+2H\sum_{k=1}^{K}\sum_{h=2}^{H}b^k_h(s^k_h,a^k_h)+2H\sum_{k=1}^{K}\sum_{h=2}^{H}r_h(s^k_h,a^k_h)\\
&\leq W_3+T_8+2HT_2+2KH^2,
\end{aligned} T 5 ≤ W 3 + T 8 + 2 H k = 1 ∑ K h = 2 ∑ H max { Q h k ( s h k , a h k ) − ⟨ P s h k , a h k , h k , V h + 1 k ⟩ , 0 } ≤ W 3 + T 8 + 2 H k = 1 ∑ K h = 2 ∑ H b h k ( s h k , a h k ) + 2 H k = 1 ∑ K h = 2 ∑ H r h ( s h k , a h k ) ≤ W 3 + T 8 + 2 H T 2 + 2 K H 2 ,
where the second line is a direct consequence of the update rule (5.18), and the last line uses the definition (399) of T 2 T_2 T 2 combined with the assumption that r h ( s , a ) ≤ 1 r_h(s,a)\leq1 r h ( s , a ) ≤ 1 . Taking this collectively with (520) establishes the claimed bound (405) .
Bounding the term T 6 l o w T_6^{\mathrm{low}} T 6 low (cf. (394) ). Let F k \mathcal{F}_k F k denote the σ \sigma σ -field generated by all events prior to episode k k k , and define
U k : = ∑ h = 1 H Var P s h k , a h k , h ( V h + 1 k ) . U_k:=\sum_{h=1}^{H}\operatorname{Var}_{P_{s_h^k,a_h^k,h}}(V^k_{h+1}). U k := h = 1 ∑ H Var P s h k , a h k , h ( V h + 1 k ) .
It then follows directly from Lemma 5.4 that
E [ U k ∣ F k ] ≤ H 2 . \mathbb{E}[U_k\mid\mathcal{F}_k]\leq H^2. E [ U k ∣ F k ] ≤ H 2 .
Moreover, since ∥ V h + 1 π ∥ ∞ ≤ H \lVert V^{\pi}_{h+1}\rVert_{\infty}\le H ∥ V h + 1 π ∥ ∞ ≤ H for every policy π \pi π , we trivially have U k ≤ H 3 U_k\le H^3 U k ≤ H 3 . Applying the Azuma–Hoeffding inequality (Lemma 2.8 ) therefore yields that, with probability at least 1 − 2 δ ′ 1-2\delta' 1 − 2 δ ′ ,
T 6 l o w = ∑ k = 1 K U k ≤ E [ ∑ k = 1 K U k ] + 2 H 6 K log 1 δ ′ ≤ K H 2 + 2 H 6 K log 1 δ ′ . T_6^{\mathrm{low}}=\sum_{k=1}^{K}U_k\le\mathbb{E}\left[\sum_{k=1}^{K}U_k\right]+\sqrt{2H^6K\log\frac{1}{\delta'}}\le KH^2+\sqrt{2H^6K\log\frac{1}{\delta'}}. T 6 low = k = 1 ∑ K U k ≤ E [ k = 1 ∑ K U k ] + 2 H 6 K log δ ′ 1 ≤ K H 2 + 2 H 6 K log δ ′ 1 .
Bounding the term T 6 d i f f T_6^{\mathrm{diff}} T 6 diff (cf. (395) ). Given that H ≥ V h + 1 k ( s ) ≥ V h + 1 ∗ ( s ) ≥ V h + 1 π k ( s ) ≥ 0 H\ge V^k_{h+1}(s)\ge V^*_{h+1}(s)\ge V^{\pi^k}_{h+1}(s)\ge0 H ≥ V h + 1 k ( s ) ≥ V h + 1 ∗ ( s ) ≥ V h + 1 π k ( s ) ≥ 0 for all s ∈ S s\in\mathcal{S} s ∈ S (see Lemma 5.5 ), we can derive
T 6 d i f f = ∑ k = 1 K ∑ h = 1 H ⟨ P s h k , a h k , h k , ( V h + 1 k − V h + 1 π k ) 2 ⟩ ≤ H ∑ k = 1 K ∑ h = 1 H ⟨ P s h k , a h k , h k , V h + 1 k − V h + 1 π k ⟩ = H ∑ k = 1 K ∑ h = 1 H ⟨ P s h k , a h k , h k − e s h + 1 k , V h + 1 k − V h + 1 π k ⟩ + H ∑ k = 1 K ∑ h = 1 H ( V h + 1 k ( s h + 1 k ) − V h + 1 π k ( s h + 1 k ) ) . (522) \begin{aligned}
T_6^{\mathrm{diff}}=\sum_{k=1}^{K}\sum_{h=1}^{H}\left\langle P^{k}_{s_h^k,a_h^k,h},\left(V^k_{h+1}-V^{\pi^k}_{h+1}\right)^2\right\rangle\\
&\le H\sum_{k=1}^{K}\sum_{h=1}^{H}\left\langle P^{k}_{s_h^k,a_h^k,h},V^k_{h+1}-V^{\pi^k}_{h+1}\right\rangle\\
&=H\sum_{k=1}^{K}\sum_{h=1}^{H}\left\langle P^{k}_{s_h^k,a_h^k,h}-e_{s^k_{h+1}},V^k_{h+1}-V^{\pi^k}_{h+1}\right\rangle+H\sum_{k=1}^{K}\sum_{h=1}^{H}\left(V^k_{h+1}(s^k_{h+1})-V^{\pi^k}_{h+1}(s^k_{h+1})\right). \tag*{(522)}
\end{aligned} T 6 diff = k = 1 ∑ K h = 1 ∑ H ⟨ P s h k , a h k , h k , ( V h + 1 k − V h + 1 π k ) 2 ⟩ ≤ H k = 1 ∑ K h = 1 ∑ H ⟨ P s h k , a h k , h k , V h + 1 k − V h + 1 π k ⟩ = H k = 1 ∑ K h = 1 ∑ H ⟨ P s h k , a h k , h k − e s h + 1 k , V h + 1 k − V h + 1 π k ⟩ + H k = 1 ∑ K h = 1 ∑ H ( V h + 1 k ( s h + 1 k ) − V h + 1 π k ( s h + 1 k ) ) . (522)
leaving us with two terms to control.
The first term J 1 J_1 J 1 in (522) can be controlled using the same argument as for T 8 T_8 T 8 . Repeating the arguments in (519) tells us that, with probability at least 1 − 2 δ ′ log 2 ( K H ) 1-2\delta'\log_2(KH) 1 − 2 δ ′ log 2 ( K H ) ,
∣ J 1 ∣ ≤ 8 ∑ k , h Var s h k , a h k ( V h + 1 k − V h + 1 π k ) log 1 δ ′ + 5 H log 1 δ ′ ≤ 8 ∑ k , h ⟨ P s h k , a h k , h k , ( V h + 1 k − V h + 1 π k ) 2 ⟩ log 1 δ ′ + 5 H log 1 δ ′ = 8 T 6 d i f f log 1 δ ′ + 5 H log 1 δ ′ ≤ 1 2 H T 6 d i f f + 4 H log 1 δ ′ + 5 H log 1 δ ′ = 1 2 H T 6 d i f f + 9 H log 1 δ ′ . (523) \begin{aligned}
\lvert J_1\rvert\le\sqrt{8\sum_{k,h}\operatorname{Var}_{s_h^k,a_h^k}\left(V^k_{h+1}-V^{\pi^k}_{h+1}\right)\log\frac{1}{\delta'}+5H\log\frac{1}{\delta'}} \\
&\le\sqrt{8\sum_{k,h}\left\langle P^k_{s_h^k,a_h^k,h},\left(V^k_{h+1}-V^{\pi^k}_{h+1}\right)^2\right\rangle\log\frac{1}{\delta'}+5H\log\frac{1}{\delta'}} \\
&=\sqrt{8T_6^{\mathrm{diff}}\log\frac{1}{\delta'}+5H\log\frac{1}{\delta'}} \\
&\le\frac{1}{2H}T_6^{\mathrm{diff}}+4H\log\frac{1}{\delta'}+5H\log\frac{1}{\delta'} \\
&=\frac{1}{2H}T_6^{\mathrm{diff}}+9H\log\frac{1}{\delta'}. \tag*{(523)}
\end{aligned} ∣ J 1 ∣ ≤ 8 k , h ∑ Var s h k , a h k ( V h + 1 k − V h + 1 π k ) log δ ′ 1 + 5 H log δ ′ 1 ≤ 8 k , h ∑ ⟨ P s h k , a h k , h k , ( V h + 1 k − V h + 1 π k ) 2 ⟩ log δ ′ 1 + 5 H log δ ′ 1 = 8 T 6 diff log δ ′ 1 + 5 H log δ ′ 1 ≤ 2 H 1 T 6 diff + 4 H log δ ′ 1 + 5 H log δ ′ 1 = 2 H 1 T 6 diff + 9 H log δ ′ 1 . (523)
where the penultimate inequality applies the AM-GM inequality.
When it comes to the second term J 2 J_2 J 2 in (522) , it follows from (404) that, with probability at least 1 − 9 S 2 A H 2 K δ ′ 1-9S^2AH^2K\delta' 1 − 9 S 2 A H 2 K δ ′ ,
∣ J 2 ∣ ≤ H T 0 u b . (524) \lvert J_2\rvert\le HT_0^{\mathrm{ub}}. \tag*{(524)} ∣ J 2 ∣ ≤ H T 0 ub . (524)
The above two bounds taken collectively demonstrate that
T 6 d i f f ≤ H J 1 + H J 2 ≤ 1 2 T 6 d i f f + 9 H 2 log 1 δ ′ + H 2 T 0 u b , T_6^{\mathrm{diff}}\le HJ_1+HJ_2\le\frac{1}{2}T_6^{\mathrm{diff}}+9H^2\log\frac{1}{\delta'}+H^2T_0^{\mathrm{ub}}, T 6 diff ≤ H J 1 + H J 2 ≤ 2 1 T 6 diff + 9 H 2 log δ ′ 1 + H 2 T 0 ub ,
thus implying that, with probability at least 1 − 10 S 2 A H 2 K δ ′ 1-10S^2AH^2K\delta' 1 − 10 S 2 A H 2 K δ ′ ,
T 6 d i f f ≤ 2 H 2 T 0 u b + 18 H 2 log 1 δ ′ . T_6^{\mathrm{diff}}\le2H^2T_0^{\mathrm{ub}}+18H^2\log\frac{1}{\delta'}. T 6 diff ≤ 2 H 2 T 0 ub + 18 H 2 log δ ′ 1 .
Establishing the bounds (5.43). Under the condition (387) , it follows from (406) that
T 6 l o w ≤ K H 2 + 2 H 6 K log 1 δ ≤ 2 K H 2 . T_{6}^{\mathrm{low}} \le KH^{2}+\sqrt{2H^{6}K\log\frac{1}{\delta}} \le2KH^{2}. T 6 low ≤ K H 2 + 2 H 6 K log δ 1 ≤ 2 K H 2 .
Moreover, combining (405) with Lemma 5.7 and (5.123) (an inequality in the proof of Lemma 5.7 in Section 5.7.2) leads to
T 5 ≤ 2 H T 2 + 2 K H 2 + 1 4 S A H 3 T 6 l o w ι b + 1 4 S 2 A H 3 T 6 d i f f ι b + 1 4 S 2 A H 3 ι b ≤ (i) 2 H T 2 + 2 K H 2 + 1 4 2 S A H 5 K ι b + 1 4 S 2 A H 3 T 6 d i f f ι b + 1 4 S 2 A H 3 ι b ≤ (ii) S A H 3 T 5 ι b + 3 K H 2 + 5 4 S A H 3 ι b + 1 4 S 2 A H 3 T 6 d i f f ι b + 1 4 S 2 A H 3 ι b ≤ (iii) 1 2 T 5 + 3 K H 2 + 1 4 S 2 A H 3 T 6 d i f f ι b + 2 S 2 A H 3 ι b , \begin{aligned}
T_{5} &\le2HT_{2}+2KH^{2}+\frac{1}{4}\sqrt{SAH^{3}T_{6}^{\mathrm{low}}\iota_{b}}+\frac{1}{4}\sqrt{S^{2}AH^{3}T_{6}^{\mathrm{diff}}\iota_{b}}+\frac{1}{4}S^{2}AH^{3}\iota_{b}\\
&\underset{\text{(i)}}{\le}2HT_{2}+2KH^{2}+\frac{1}{4}\sqrt{2SAH^{5}K\iota_{b}}+\frac{1}{4}\sqrt{S^{2}AH^{3}T_{6}^{\mathrm{diff}}\iota_{b}}+\frac{1}{4}S^{2}AH^{3}\iota_{b}\\
&\underset{\text{(ii)}}{\le}\sqrt{SAH^{3}T_{5}\iota_{b}}+3KH^{2}+\frac{5}{4}SAH^{3}\iota_{b}+\frac{1}{4}\sqrt{S^{2}AH^{3}T_{6}^{\mathrm{diff}}\iota_{b}}+\frac{1}{4}S^{2}AH^{3}\iota_{b}\\
&\underset{\text{(iii)}}{\le}\frac{1}{2}T_{5}+3KH^{2}+\frac{1}{4}\sqrt{S^{2}AH^{3}T_{6}^{\mathrm{diff}}\iota_{b}}+2S^{2}AH^{3}\iota_{b},
\end{aligned} T 5 ≤ 2 H T 2 + 2 K H 2 + 4 1 S A H 3 T 6 low ι b + 4 1 S 2 A H 3 T 6 diff ι b + 4 1 S 2 A H 3 ι b (i) ≤ 2 H T 2 + 2 K H 2 + 4 1 2 S A H 5 K ι b + 4 1 S 2 A H 3 T 6 diff ι b + 4 1 S 2 A H 3 ι b (ii) ≤ S A H 3 T 5 ι b + 3 K H 2 + 4 5 S A H 3 ι b + 4 1 S 2 A H 3 T 6 diff ι b + 4 1 S 2 A H 3 ι b (iii) ≤ 2 1 T 5 + 3 K H 2 + 4 1 S 2 A H 3 T 6 diff ι b + 2 S 2 A H 3 ι b ,
and as a result,
⟹ T 5 ≤ 6 K H 2 + 1 2 S 2 A H 3 T 6 d i f f ι b + 4 S 2 A H 3 ι b . \Longrightarrow\quad T_{5}\le6KH^{2}+\frac{1}{2}\sqrt{S^{2}AH^{3}T_{6}^{\mathrm{diff}}\iota_{b}}+4S^{2}AH^{3}\iota_{b}. ⟹ T 5 ≤ 6 K H 2 + 2 1 S 2 A H 3 T 6 diff ι b + 4 S 2 A H 3 ι b .
Here, (i) follows from (408) , (ii) invokes Lemma 5.8 and the AM-GM inequality 1 4 2 S A H 5 K ι b ≤ K H 2 + 1 4 S A H 3 ι b \frac{1}{4}\sqrt{2SAH^{5}K\iota_{b}}\le KH^{2}+\frac{1}{4}SAH^{3}\iota_{b} 4 1 2 S A H 5 K ι b ≤ K H 2 + 4 1 S A H 3 ι b , while (iii) applies the AM-GM inequality S A H 3 T 5 ι b ≤ 1 2 T 5 + 1 2 S A H 3 ι b \sqrt{SAH^{3}T_{5}\iota_{b}}\le\frac{1}{2}T_{5}+\frac{1}{2}SAH^{3}\iota_{b} S A H 3 T 5 ι b ≤ 2 1 T 5 + 2 1 S A H 3 ι b .
Appendix B: proof of auxiliary lemmas in Section 5.4.4
Proof of Lemma 5.11
Recalling the set C \mathcal{C} C defined in (421) , it suffices to establish (422) . To this end, set
M = log 2 K and N = S A H . M=\log_{2}K\qquad\text{and}\qquad N=SAH. M = log 2 K and N = S A H .
We begin by introducing the following collections of “distinct” profiles:
C ( l ) d i s t i n c t : = { I = { I 1 , … , I l } ∣ I 1 ≤ ⋯ ≤ I l , I τ ∈ { 0 , 1 , … , M } N and I τ ≠ I τ + 1 ( ∀ τ ) } ; \mathcal{C}^{\mathrm{distinct}}_{(l)}:=\left\{\mathcal{I}=\{\mathcal{I}^{1},\ldots,\mathcal{I}^{l}\}\mid\mathcal{I}^{1}\le\cdots\le\mathcal{I}^{l},\ \mathcal{I}^{\tau}\in\{0,1,\ldots,M\}^{N}\ \text{and}\ \mathcal{I}^{\tau}\ne\mathcal{I}^{\tau+1}\ (\forall\tau)\right\}; C ( l ) distinct := { I = { I 1 , … , I l } ∣ I 1 ≤ ⋯ ≤ I l , I τ ∈ { 0 , 1 , … , M } N and I τ = I τ + 1 ( ∀ τ ) } ;
C d i s t i n c t : = ⋃ l ≥ 1 C ( l ) d i s t i n c t . \mathcal{C}^{\mathrm{distinct}}:=\bigcup_{l\ge1}\mathcal{C}^{\mathrm{distinct}}_{(l)}. C distinct := l ≥ 1 ⋃ C ( l ) distinct .
In words, C ( l ) d i s t i n c t \mathcal{C}^{\mathrm{distinct}}_{(l)} C ( l ) distinct consists of all non-decreasing paths of length l l l in { 0 , 1 , … , M } N \{0,1,\ldots,M\}^{N} { 0 , 1 , … , M } N whose points are all distinct, while C d i s t i n c t \mathcal{C}^{\mathrm{distinct}} C distinct collects all such paths of arbitrary length.
We first relate ∣ C ∣ |\mathcal{C}| ∣ C ∣ to ∣ C d i s t i n c t ∣ |\mathcal{C}^{\mathrm{distinct}}| ∣ C distinct ∣ . Define the projection operator Proj : C → C d i s t i n c t \operatorname{Proj}:\mathcal{C}\to\mathcal{C}^{\mathrm{distinct}} Proj : C → C distinct that maps each I ∈ C \mathcal{I}\in\mathcal{C} I ∈ C to the path I d i s t i n c t \mathcal{I}^{\mathrm{distinct}} I distinct obtained by removing all repeated elements from I \mathcal{I} I . For each I d i s t i n c t ∈ C d i s t i n c t \mathcal{I}^{\mathrm{distinct}}\in\mathcal{C}^{\mathrm{distinct}} I distinct ∈ C distinct , consider its inverse image
B ( I d i s t i n c t ) : = { I ∈ C ∣ Proj ( I ) = I d i s t i n c t } . \mathcal{B}(\mathcal{I}^{\mathrm{distinct}}):=\{\mathcal{I}\in\mathcal{C}\mid\operatorname{Proj}(\mathcal{I})=\mathcal{I}^{\mathrm{distinct}}\}. B ( I distinct ) := { I ∈ C ∣ Proj ( I ) = I distinct } .
Since I d i s t i n c t \mathcal{I}^{\mathrm{distinct}} I distinct is a non-decreasing path with distinct points in { 0 , 1 , … , M } N \{0,1,\ldots,M\}^{N} { 0 , 1 , … , M } N , its length is at most M N + 1 MN+1 M N + 1 ; indeed, along such a path, the sum of the N N N coordinates strictly increases at every step and ranges from 0 0 0 to at most M N MN M N . Consequently, the cardinality of B ( I d i s t i n c t ) \mathcal{B}(\mathcal{I}^{\mathrm{distinct}}) B ( I distinct ) is bounded by the number of nonnegative integer solutions to
∑ i = 1 M N + 1 x i = K and x i ∈ N for all 1 ≤ i ≤ M N + 1. \sum_{i=1}^{MN+1}x_{i}=K\qquad\text{and}\qquad x_{i}\in\mathbb{N}\ \text{for all }1\le i\le MN+1. i = 1 ∑ M N + 1 x i = K and x i ∈ N for all 1 ≤ i ≤ M N + 1.
Elementary combinatorial arguments then reveal that
∣ B ( I d i s t i n c t ) ∣ ≤ ( K + M N M N ) ≤ ( K + M N ) M N ≤ ( 2 K ) M N \left|\mathcal{B}(\mathcal{I}^{\mathrm{distinct}})\right| \le\binom{K+MN}{MN} \le(K+MN)^{MN} \le(2K)^{MN} B ( I distinct ) ≤ ( M N K + M N ) ≤ ( K + M N ) M N ≤ ( 2 K ) M N
for each I d i s t i n c t \mathcal{I}^{\mathrm{distinct}} I distinct , provided that K ≥ M N = S A H log 2 K K \ge MN = SAH\log_{2}K K ≥ M N = S A H log 2 K . Summing over all I d i s t i n c t ∈ C d i s t i n c t \mathcal{I}^{\mathrm{distinct}} \in\mathcal{C}^{\mathrm{distinct}} I distinct ∈ C distinct gives
∣ C ∣ ≤ ∑ I d i s t i n c t ∈ C d i s t i n c t ∣ B ( I d i s t i n c t ) ∣ ≤ ∣ C d i s t i n c t ∣ ⋅ ( 2 K ) M N . (525) |\mathcal{C}| \le\sum_{\mathcal{I}^{\mathrm{distinct}}\in\mathcal{C}^{\mathrm{distinct}}}\left|\mathcal{B}(\mathcal{I}^{\mathrm{distinct}})\right| \le\left|\mathcal{C}^{\mathrm{distinct}}\right|\cdot(2K)^{MN}. \tag*{(525)} ∣ C ∣ ≤ I distinct ∈ C distinct ∑ B ( I distinct ) ≤ C distinct ⋅ ( 2 K ) M N . (525)
It remains to bound ∣ C d i s t i n c t ∣ \left|\mathcal{C}^{\mathrm{distinct}}\right| C distinct . To do so, let us first look at the set C d i s t i n c t ( M N + 1 ) \mathcal{C}^{\mathrm{distinct}}(MN+1) C distinct ( M N + 1 ) , as each path in C d i s t i n c t \mathcal{C}^{\mathrm{distinct}} C distinct cannot have length more than M N + 1 MN+1 M N + 1 . For each I d i s t i n c t = { I ~ 1 , I ~ 2 , … , I ~ M N + 1 } ∈ C d i s t i n c t ( M N + 1 ) \mathcal{I}^{\mathrm{distinct}}=\{\widetilde{\mathcal{I}}^{1},\widetilde{\mathcal{I}}^{2},\ldots,\widetilde{\mathcal{I}}^{MN+1}\}\in\mathcal{C}^{\mathrm{distinct}}(MN+1) I distinct = { I 1 , I 2 , … , I M N + 1 } ∈ C distinct ( M N + 1 ) , we necessarily have
I ~ 1 = [ 0 , 0 , … , 0 ] T \widetilde{\mathcal{I}}^{1}=[0,0,\ldots,0]^{\mathsf{T}} I 1 = [ 0 , 0 , … , 0 ] T and I ~ M N + 1 = [ M , M , … , M ] T \widetilde{\mathcal{I}}^{MN+1}=[M,M,\ldots,M]^{\mathsf{T}} I M N + 1 = [ M , M , … , M ] T ;
In other words, each I d i s t i n c t ∈ C d i s t i n c t ( M N + 1 ) \mathcal{I}^{\mathrm{distinct}}\in\mathcal{C}^{\mathrm{distinct}}(MN+1) I distinct ∈ C distinct ( M N + 1 ) can be viewed as an M N MN M N -step path from [ 0 , 0 , … , 0 ] T [0,0,\ldots,0]^{\mathsf{T}} [ 0 , 0 , … , 0 ] T to [ M , M , … , M ] T [M,M,\ldots,M]^{\mathsf{T}} [ M , M , … , M ] T , with each step incrementing one of the N N N coordinates. Since there are at most N N N choices at each step,
∣ C d i s t i n c t ( M N + 1 ) ∣ ≤ N M N . \left|\mathcal{C}^{\mathrm{distinct}}(MN+1)\right| \le N^{MN}. C distinct ( M N + 1 ) ≤ N M N .
Finally, every path I d i s t i n c t ∈ C d i s t i n c t \mathcal{I}^{\mathrm{distinct}}\in\mathcal{C}^{\mathrm{distinct}} I distinct ∈ C distinct can be extended to some maximal path I ~ d i s t i n c t ∈ C d i s t i n c t ( M N + 1 ) \widetilde{\mathcal{I}}^{\mathrm{distinct}}\in\mathcal{C}^{\mathrm{distinct}}(MN+1) I distinct ∈ C distinct ( M N + 1 ) . For each such maximal path, there are at most 2 M N + 1 2^{MN+1} 2 M N + 1 ways to select a subsequence of its points. It thus follows that
∣ C d i s t i n c t ∣ ≤ 2 M N + 1 ∣ C d i s t i n c t ( M N + 1 ) ∣ ≤ ( 2 N ) M N + 1 , \left|\mathcal{C}^{\mathrm{distinct}}\right| \le2^{MN+1}\left|\mathcal{C}^{\mathrm{distinct}}(MN+1)\right| \le(2N)^{MN+1}, C distinct ≤ 2 M N + 1 C distinct ( M N + 1 ) ≤ ( 2 N ) M N + 1 ,
which taken collectively with (525) establishes the advertised result
∣ C ∣ ≤ ( 2 K ) M N ∣ C d i s t i n c t ∣ ≤ ( 4 K N ) M N + 1 ≤ ( 4 K N ) M N + 1 . |\mathcal{C}| \le(2K)^{MN}\left|\mathcal{C}^{\mathrm{distinct}}\right| \le(4KN)^{MN+1} \le(4KN)^{MN+1}. ∣ C ∣ ≤ ( 2 K ) M N C distinct ≤ ( 4 K N ) M N + 1 ≤ ( 4 K N ) M N + 1 .
Proof of Lemma 5.13
We begin by fixing a total profile I ∈ C \mathcal{I}\in\mathcal{C} I ∈ C , an integer l l l satisfying 2 ≤ l ≤ log 2 K + 1 2\le l\le\log_{2}K+1 2 ≤ l ≤ log 2 K + 1 , and a feasible sequence { X h , s , a } \{X_{h,s,a}\} { X h , s , a } . Recall that
P ^ s , a , h ( l ) \widehat{P}^{(l)}_{s,a,h} P s , a , h ( l ) is constructed from the l l l -th batch of data, which consists of 2 l − 2 2^{l-2} 2 l − 2 independent samples from D e x p a n d \mathcal{D}^{\mathrm{expand}} D expand (see Definition 5.1);
each X h + 1 , s , a X_{h+1,s,a} X h + 1 , s , a is a deterministic mapping of I \mathcal{I} I and the empirical transition models associated with steps h ′ ∈ [ h + 1 , H ] h'\in[h+1,H] h ′ ∈ [ h + 1 , H ] .
Consequently, for each ( s , a , h ) (s,a,h) ( s , a , h ) , the inner product ⟨ P ^ s , a , h ( l ) − P s , a , h , X h + 1 , s , a ⟩ \langle\widehat{P}^{(l)}_{s,a,h}-P_{s,a,h},X_{h+1,s,a}\rangle ⟨ P s , a , h ( l ) − P s , a , h , X h + 1 , s , a ⟩ can be expressed as the sum of 2 l − 2 2^{l-2} 2 l − 2 centered random variables, each bounded in magnitude by H / 2 l − 2 H/2^{l-2} H / 2 l − 2 . Thus, in the sum of interest ∑ s , a , h ⟨ P ^ s , a , h ( l ) − P s , a , h , X h + 1 , s , a ⟩ \sum_{s,a,h}\langle\widehat{P}^{(l)}_{s,a,h}-P_{s,a,h},X_{h+1,s,a}\rangle ∑ s , a , h ⟨ P s , a , h ( l ) − P s , a , h , X h + 1 , s , a ⟩ , by ordering these random variables backward in h h h (from H H H to 1), and arbitrarily over ( s , a ) (s,a) ( s , a ) and the samples within each batch, we obtain a martingale difference sequence with total conditional variance
W = 1 2 l − 2 ∑ s , a , h Var P s , a , h ( X h + 1 , s , a ) . W = \frac{1}{2^{l-2}}\sum_{s,a,h}\operatorname{Var}_{P_{s,a,h}}(X_{h+1,s,a}). W = 2 l − 2 1 s , a , h ∑ Var P s , a , h ( X h + 1 , s , a ) .
Additionally, since ∥ X h + 1 , s , a ∥ ∞ ≤ H \lVert X_{h+1,s,a}\rVert_{\infty} \le H ∥ X h + 1 , s , a ∥ ∞ ≤ H , we have the crude upper bound
W ≤ S A H 3 2 l − 2 = : σ 2 . W \le\frac{SAH^3}{2^{l-2}} =: \sigma^2. W ≤ 2 l − 2 S A H 3 =: σ 2 .
We can therefore apply Lemma 2.9 with K 0 = log 2 ( S A H K ) K_0 = \log_2(SAHK) K 0 = log 2 ( S A H K ) . Using 2 l − 2 ≤ K / 2 2^{l-2} \le K/2 2 l − 2 ≤ K /2 together with the elementary inequality max { a , b } ≤ a + b \sqrt{\max\{a,b\}} \le\sqrt{a}+\sqrt{b} max { a , b } ≤ a + b , we obtain, with probability at least 1 − δ ′ 1-\delta' 1 − δ ′ ,
∑ s , a , h ⟨ P ^ s , a , h ( l ) − P s , a , h , X h + 1 , s , a ⟩ ≤ 8 2 l − 2 ∑ s , a , h Var P s , a , h ( X h + 1 , s , a ) log 2 log 2 ( S A H K ) δ ′ + 4 H 2 l − 2 log 2 log 2 ( S A H K ) δ ′ . (526) \sum_{s,a,h}\left\langle\widehat{P}^{(l)}_{s,a,h}-P_{s,a,h},X_{h+1,s,a}\right\rangle
\le
\sqrt{\frac{8}{2^{l-2}}\sum_{s,a,h}\operatorname{Var}_{P_{s,a,h}}(X_{h+1,s,a})\log\frac{2\log_2(SAHK)}{\delta'}}
+\frac{4H}{2^{l-2}}\log\frac{2\log_2(SAHK)}{\delta'}. \tag*{(526)} s , a , h ∑ ⟨ P s , a , h ( l ) − P s , a , h , X h + 1 , s , a ⟩ ≤ 2 l − 2 8 s , a , h ∑ Var P s , a , h ( X h + 1 , s , a ) log δ ′ 2 log 2 ( S A H K ) + 2 l − 2 4 H log δ ′ 2 log 2 ( S A H K ) . (526)
We next make this bound uniform over all admissible choices. For any profile I \mathcal{I} I , Assumption 5.1 ensures that each X h , s , a X_{h,s,a} X h , s , a has at most K + 1 K+1 K + 1 possible choices. Hence, the number of feasible sequences { X h , s , a } ( s , a , h ) ∈ S × A × [ H ] \{X_{h,s,a}\}_{(s,a,h)\in\mathcal{S}\times\mathcal{A}\times[H]} { X h , s , a } ( s , a , h ) ∈ S × A × [ H ] is at most ( K + 1 ) S A H ≤ ( 2 K ) S A H (K+1)^{SAH}\le(2K)^{SAH} ( K + 1 ) S A H ≤ ( 2 K ) S A H . Moreover, Lemma 5.11 ensures that the number of admissible total profiles I \mathcal{I} I is at most ( 4 S A H K ) 2 S A H log 2 K (4SAHK)^{2SAH\log_2 K} ( 4 S A H K ) 2 S A H l o g 2 K . Taking a union bound over all profiles, all feasible sequences, and all 2 ≤ l ≤ log 2 K + 1 2\le l\le\log_2K+1 2 ≤ l ≤ log 2 K + 1 , and replacing δ ′ \delta' δ ′ in (526) by
δ ′ ( 4 S A H K ) 2 S A H log 2 K ( 2 K ) S A H log 2 K , \frac{\delta'}{(4SAHK)^{2SAH\log_2 K}(2K)^{SAH}\log_2K}, ( 4 S A H K ) 2 S A H l o g 2 K ( 2 K ) S A H log 2 K δ ′ ,
we conclude that, with probability at least 1 − δ ′ 1-\delta' 1 − δ ′ ,
∑ s , a , h ⟨ P ^ s , a , h ( l ) − P s , a , h , X h + 1 , s , a ⟩ ≤ 8 2 l − 2 ∑ s , a , h Var P s , a , h ( X h + 1 , s , a ) B 0 + 4 H 2 l − 2 B 0 ≤ 8 2 l − 2 ∑ s , a , h Var P s , a , h ( X h + 1 , s , a ) ( 6 S A H log 2 2 K + log 1 δ ′ ) + 4 H 2 l − 2 ( 6 S A H log 2 2 K + log 1 δ ′ ) ≤ S A H ι b 2 l log 2 K ∑ s , a , h Var P s , a , h ( X h + 1 , s , a ) + S A H 2 ι b 2 l log 2 K . (527) \begin{aligned}
\sum_{s,a,h}\left\langle\widehat{P}^{(l)}_{s,a,h}-P_{s,a,h},X_{h+1,s,a}\right\rangle
\le
\sqrt{\frac{8}{2^{l-2}}\sum_{s,a,h}\operatorname{Var}_{P_{s,a,h}}(X_{h+1,s,a})B_0}
+\frac{4H}{2^{l-2}}B_0 \\
&\le
\sqrt{\frac{8}{2^{l-2}}\sum_{s,a,h}\operatorname{Var}_{P_{s,a,h}}(X_{h+1,s,a})\left(6SAH\log_2^2K+\log\frac{1}{\delta'}\right)}
+\frac{4H}{2^{l-2}}\left(6SAH\log_2^2K+\log\frac{1}{\delta'}\right) \\
&\le
\sqrt{\frac{SAH\iota_b}{2^l\log_2K}\sum_{s,a,h}\operatorname{Var}_{P_{s,a,h}}(X_{h+1,s,a})+\frac{SAH^2\iota_b}{2^l\log_2K}}. \tag*{(527)}
\end{aligned} s , a , h ∑ ⟨ P s , a , h ( l ) − P s , a , h , X h + 1 , s , a ⟩ ≤ 2 l − 2 8 s , a , h ∑ Var P s , a , h ( X h + 1 , s , a ) B 0 + 2 l − 2 4 H B 0 ≤ 2 l − 2 8 s , a , h ∑ Var P s , a , h ( X h + 1 , s , a ) ( 6 S A H log 2 2 K + log δ ′ 1 ) + 2 l − 2 4 H ( 6 S A H log 2 2 K + log δ ′ 1 ) ≤ 2 l log 2 K S A H ι b s , a , h ∑ Var P s , a , h ( X h + 1 , s , a ) + 2 l log 2 K S A H 2 ι b . (527)
holds simultaneously for all I ∈ C \mathcal{I}\in\mathcal{C} I ∈ C , all 2 ≤ l ≤ log 2 K + 1 2\le l\le\log_2K+1 2 ≤ l ≤ log 2 K + 1 , and all feasible sequences { X h , s , a } ( s , a , h ) ∈ S × A × [ H ] \{X_{h,s,a}\}_{(s,a,h)\in\mathcal{S}\times\mathcal{A}\times[H]} { X h , s , a } ( s , a , h ) ∈ S × A × [ H ] . Here, we recall the definition of ι b \iota_b ι b in (416) , and in the second line above we set
B 0 = 2 S A H ( log 2 K ) log ( 4 S A H K ) + S A H log ( 2 K ) + log 2 log 2 2 ( S A H K ) δ ′ . B_0=2SAH(\log_2K)\log(4SAHK)+SAH\log(2K)+\log\frac{2\log_2^2(SAHK)}{\delta'}. B 0 = 2 S A H ( log 2 K ) log ( 4 S A H K ) + S A H log ( 2 K ) + log δ ′ 2 log 2 2 ( S A H K ) .
It remains to establish the corresponding bound for the positive parts. To this end, recall that 0 ∈ X h + 1 , I 0\in\mathcal{X}_{h+1,\mathcal{I}} 0 ∈ X h + 1 , I . Given any feasible sequence { X h , s , a } \{X_{h,s,a}\} { X h , s , a } , define another feasible sequence { X ~ h + 1 , s , a } \{\widetilde{X}_{h+1,s,a}\} { X h + 1 , s , a } by
X ~ h + 1 , s , a = { X h + 1 , s , a , if ⟨ P ^ s , a , h ( l ) − P s , a , h , X h + 1 , s , a ⟩ ≥ 0 , 0 , otherwise. \widetilde{X}_{h+1,s,a}=
\begin{cases}
X_{h+1,s,a}, & \text{if }\left\langle\widehat{P}^{(l)}_{s,a,h}-P_{s,a,h},X_{h+1,s,a}\right\rangle\ge0,\\
0, & \text{otherwise.}
\end{cases} X h + 1 , s , a = { X h + 1 , s , a , 0 , if ⟨ P s , a , h ( l ) − P s , a , h , X h + 1 , s , a ⟩ ≥ 0 , otherwise.
By construction,
∑ s , a , h max { ⟨ P ^ s , a , h ( l ) − P s , a , h , X h + 1 , s , a ⟩ , 0 } = ∑ s , a , h ⟨ P ^ s , a , h ( l ) − P s , a , h , X ~ h + 1 , s , a ⟩ . \sum_{s,a,h}\max\left\{\left\langle\widehat{P}^{(l)}_{s,a,h}-P_{s,a,h},X_{h+1,s,a}\right\rangle,0\right\}=\sum_{s,a,h}\left\langle\widehat{P}^{(l)}_{s,a,h}-P_{s,a,h},\widetilde{X}_{h+1,s,a}\right\rangle. s , a , h ∑ max { ⟨ P s , a , h ( l ) − P s , a , h , X h + 1 , s , a ⟩ , 0 } = s , a , h ∑ ⟨ P s , a , h ( l ) − P s , a , h , X h + 1 , s , a ⟩ .
Since (527) holds uniformly over all feasible sequences, it applies in particular to { X ~ h + 1 , s , a } \{\widetilde{X}_{h+1,s,a}\} { X h + 1 , s , a } . This yields the desired positive-part bound and completes the proof.
Proof of Lemma 5.14
We begin with the following claim, whose proof is deferred to the end of this subsection.
Claim 5.1. With probability exceeding 1 − δ ′ 1-\delta' 1 − δ ′ ,
∑ s , a , h ⟨ P ^ s , a , h ( l ) − P s , a , h , V h + 1 k l , j , s , a , h ⟩ ≤ S A H ι b 2 l log 2 K ∑ s , a , h Var P s , a , h ( V h + 1 k l , j , s , a , h ) + S A H 2 ι b 2 l log 2 K (528) \sum_{s,a,h}\left\langle\widehat{P}^{(l)}_{s,a,h}-P_{s,a,h},V_{h+1}^{k_{l,j,s,a,h}}\right\rangle\le\sqrt{\frac{SAH\iota_b}{2^l\log_2 K}\sum_{s,a,h}\operatorname{Var}_{P_{s,a,h}}\left(V_{h+1}^{k_{l,j,s,a,h}}\right)}+\frac{SAH^2\iota_b}{2^l\log_2 K} \tag*{(528)} s , a , h ∑ ⟨ P s , a , h ( l ) − P s , a , h , V h + 1 k l , j , s , a , h ⟩ ≤ 2 l log 2 K S A H ι b s , a , h ∑ Var P s , a , h ( V h + 1 k l , j , s , a , h ) + 2 l log 2 K S A H 2 ι b (528) holds simultaneously for all l = 1 , … , log 2 K l=1,\ldots,\log_2 K l = 1 , … , log 2 K and all j = 1 , … , 2 l − 1 j=1,\ldots,2^{l-1} j = 1 , … , 2 l − 1 . Here, we recall that k l , j , s , a , h k_{l,j,s,a,h} k l , j , s , a , h denotes the episode index corresponding to the ( 2 l − 1 + j ) (2^{l-1}+j) ( 2 l − 1 + j ) -th visit to ( s , a , h ) (s,a,h) ( s , a , h ) during the online learning process.
Assuming the validity of Claim 5.1 for the moment, we combine it with the decomposition (423) to obtain
∑ k = 1 K ∑ h = 1 H ⟨ P ^ s h k , a h k , h − P s h k , a h k , V h + 1 k ⟩ ≤ ∑ l = 1 log 2 K ∑ j = 1 2 l − 1 ∑ s , a , h ⟨ P ^ s , a , h ( l ) − P s , a , h , V h + 1 k l , j , s , a , h ⟩ + S A H 2 ≤ ∑ l = 1 log 2 K ∑ j = 1 2 l − 1 S A H ι b 2 l log 2 K ∑ s , a , h Var P s , a , h ( V h + 1 k l , j , s , a , h ) + ∑ l = 1 log 2 K ∑ j = 1 2 l − 1 S A H 2 ι b 2 l log 2 K + S A H 2 ≤ ( i ) S A H ι b ∑ l = 1 log 2 K ∑ j = 1 2 l − 1 ∑ s , a , h Var P s , a , h ( V h + 1 k l , j , s , a , h ) + 1 2 S A H 2 ι b + S A H 2 ≤ ( i i ) S A H ι b ∑ k = 1 K ∑ h = 1 H V s h k , a h k , h ( V h + 1 k ) + S A H 2 ι b . \begin{aligned}
\sum_{k=1}^{K}\sum_{h=1}^{H}\left\langle\widehat{P}_{s_h^k,a_h^k,h}-P_{s_h^k,a_h^k},V_{h+1}^k\right\rangle
&\le\sum_{l=1}^{\log_2 K}\sum_{j=1}^{2^{l-1}}\sum_{s,a,h}\left\langle\widehat{P}^{(l)}_{s,a,h}-P_{s,a,h},V_{h+1}^{k_{l,j,s,a,h}}\right\rangle+SAH^2\\
&\le\sum_{l=1}^{\log_2 K}\sum_{j=1}^{2^{l-1}}\sqrt{\frac{SAH\iota_b}{2^l\log_2 K}\sum_{s,a,h}\operatorname{Var}_{P_{s,a,h}}\left(V_{h+1}^{k_{l,j,s,a,h}}\right)}+\sum_{l=1}^{\log_2 K}\sum_{j=1}^{2^{l-1}}\frac{SAH^2\iota_b}{2^l\log_2 K}+SAH^2\\
&\mathrel{\overset{(i)}{\le}}\sqrt{SAH\iota_b\sum_{l=1}^{\log_2 K}\sum_{j=1}^{2^{l-1}}\sum_{s,a,h}\operatorname{Var}_{P_{s,a,h}}\left(V_{h+1}^{k_{l,j,s,a,h}}\right)}+\frac{1}{2}SAH^2\iota_b+SAH^2\\
&\mathrel{\overset{(ii)}{\le}}\sqrt{SAH\iota_b\sum_{k=1}^{K}\sum_{h=1}^{H}V_{s_h^k,a_h^k,h}\left(V_{h+1}^k\right)}+SAH^2\iota_b.
\end{aligned} k = 1 ∑ K h = 1 ∑ H ⟨ P s h k , a h k , h − P s h k , a h k , V h + 1 k ⟩ ≤ l = 1 ∑ l o g 2 K j = 1 ∑ 2 l − 1 s , a , h ∑ ⟨ P s , a , h ( l ) − P s , a , h , V h + 1 k l , j , s , a , h ⟩ + S A H 2 ≤ l = 1 ∑ l o g 2 K j = 1 ∑ 2 l − 1 2 l log 2 K S A H ι b s , a , h ∑ Var P s , a , h ( V h + 1 k l , j , s , a , h ) + l = 1 ∑ l o g 2 K j = 1 ∑ 2 l − 1 2 l log 2 K S A H 2 ι b + S A H 2 ≤ ( i ) S A H ι b l = 1 ∑ l o g 2 K j = 1 ∑ 2 l − 1 s , a , h ∑ Var P s , a , h ( V h + 1 k l , j , s , a , h ) + 2 1 S A H 2 ι b + S A H 2 ≤ ( ii ) S A H ι b k = 1 ∑ K h = 1 ∑ H V s h k , a h k , h ( V h + 1 k ) + S A H 2 ι b .
Here, (i) follows from the Cauchy-Schwarz inequality; and (ii) is valid due to our notational convention that V h + 1 k = 0 V_{h+1}^k=0 V h + 1 k = 0 ( ∀ k > K ) (\forall k>K) ( ∀ k > K ) and the identity
∑ k = 1 K ∑ h = 1 H Var P s h k , a h k , h ( V h + 1 k ) = ∑ l = 1 log 2 K ∑ s , a , h ∑ j = 1 2 l − 1 Var P s , a , h ( V h + 1 k l , j , s , a , h ) + ∑ k = 1 K ∑ h = 1 H 1 { N h k , a l l ( s h k , a h k ) = 0 } Var P s h k , a h k , h ( V h + 1 k ) . \begin{aligned}
\sum_{k=1}^{K}\sum_{h=1}^{H}\operatorname{Var}_{P_{s_h^k,a_h^k,h}}\left(V_{h+1}^k\right)
&=\sum_{l=1}^{\log_2 K}\sum_{s,a,h}\sum_{j=1}^{2^{l-1}}\operatorname{Var}_{P_{s,a,h}}\left(V_{h+1}^{k_{l,j,s,a,h}}\right)\\
&\quad+\sum_{k=1}^{K}\sum_{h=1}^{H}\mathbb{1}\left\{N_h^{k,\mathrm{all}}(s_h^k,a_h^k)=0\right\}\operatorname{Var}_{P_{s_h^k,a_h^k,h}}\left(V_{h+1}^k\right).
\end{aligned} k = 1 ∑ K h = 1 ∑ H Var P s h k , a h k , h ( V h + 1 k ) = l = 1 ∑ l o g 2 K s , a , h ∑ j = 1 ∑ 2 l − 1 Var P s , a , h ( V h + 1 k l , j , s , a , h ) + k = 1 ∑ K h = 1 ∑ H 1 { N h k , all ( s h k , a h k ) = 0 } Var P s h k , a h k , h ( V h + 1 k ) .
This proves the desired bound on ∑ k , h ⟨ P ^ s h k , a h k , h − P s h k , a h k , h , V h + 1 k ⟩ \sum_{k,h}\left\langle\widehat{P}_{s_h^k,a_h^k,h}-P_{s_h^k,a_h^k,h},V_{h+1}^k\right\rangle ∑ k , h ⟨ P s h k , a h k , h − P s h k , a h k , h , V h + 1 k ⟩ , provided that Claim 5.1 is valid. Before proceeding to the proof of Claim 5.1, we note that the same argument also controls the two related quantities ∑ k , h max { ⟨ P ^ s h k , a h k , h − P s h k , a h k , h , V h + 1 k ⟩ , 0 } \sum_{k,h}\max\{\left\langle\widehat{P}_{s_h^k,a_h^k,h}-P_{s_h^k,a_h^k,h},V_{h+1}^k\right\rangle,0\} ∑ k , h max { ⟨ P s h k , a h k , h − P s h k , a h k , h , V h + 1 k ⟩ , 0 } and ∑ k , h ⟨ P ^ s h k , a h k , h − P s h k , a h k , h , ( V h + 1 k ) 2 ⟩ \sum_{k,h}\left\langle\widehat{P}_{s_h^k,a_h^k,h}-P_{s_h^k,a_h^k,h},(V_{h+1}^k)^2\right\rangle ∑ k , h ⟨ P s h k , a h k , h − P s h k , a h k , h , ( V h + 1 k ) 2 ⟩ ; we omit the nearly identical derivations for brevity. In particular, the latter quantity can be shown to satisfy
∑ k = 1 K ∑ h = 1 H ⟨ P ^ s h k , a h k , h − P s h k , a h k , h , ( V h + 1 k ) 2 ⟩ ≤ S A H ι b 4 ∑ k = 1 K ∑ h = 1 H Var P s h k , a h k , h ( ( V h + 1 k ) 2 ) + S A H 3 ι b ≤ S A H 3 ι b ∑ k = 1 K ∑ h = 1 H Var P s h k , a h k , h ( V h + 1 k ) + S A H 3 ι b . \begin{aligned}
\sum_{k=1}^{K}\sum_{h=1}^{H}\left\langle\widehat{P}_{s_h^k,a_h^k,h}-P_{s_h^k,a_h^k,h},(V_{h+1}^k)^2\right\rangle\le\sqrt{\frac{SAH\iota_b}{4}\sum_{k=1}^{K}\sum_{h=1}^{H}\operatorname{Var}_{P_{s_h^k,a_h^k,h}}\left((V_{h+1}^k)^2\right)+SAH^3\iota_b} \\
&\le\sqrt{SAH^3\iota_b\sum_{k=1}^{K}\sum_{h=1}^{H}\operatorname{Var}_{P_{s_h^k,a_h^k,h}}\left(V_{h+1}^k\right)+SAH^3\iota_b}.
\end{aligned} k = 1 ∑ K h = 1 ∑ H ⟨ P s h k , a h k , h − P s h k , a h k , h , ( V h + 1 k ) 2 ⟩ ≤ 4 S A H ι b k = 1 ∑ K h = 1 ∑ H Var P s h k , a h k , h ( ( V h + 1 k ) 2 ) + S A H 3 ι b ≤ S A H 3 ι b k = 1 ∑ K h = 1 ∑ H Var P s h k , a h k , h ( V h + 1 k ) + S A H 3 ι b .
where the last inequality follows from Lemma 5.22 and the fact that 0 ≤ V h + 1 k ( s ) ≤ H 0 \le V_{h+1}^k(s) \le H 0 ≤ V h + 1 k ( s ) ≤ H for all s ∈ S s \in\mathcal{S} s ∈ S .
Proof of Claim 5.1. To apply Lemma 5.13 , we need to construct the collections { X h , I } \{\mathcal{X}_{h,\mathcal{I}}\} { X h , I } so that they contain the value estimates { V h k } \{V_h^k\} { V h k } generated by the actual online process. To this end, for each fixed total profile I ∈ C \mathcal{I} \in\mathcal{C} I ∈ C , we introduce an auxiliary version of MVP that operates on the expanded sample set D e x p a n d \mathcal{D}^{\mathrm{expand}} D expand while following the prescribed total profile I \mathcal{I} I . Specifically, for each I ∈ C \mathcal{I} \in\mathcal{C} I ∈ C (cf. (421) ), consider the updates in Algorithm 5.5 operating upon the expanded sample set D e x p a n d \mathcal{D}^{\mathrm{expand}} D expand .
Algorithm 5.5: Monotonic Value Propagation for a given total profile I ∈ C \mathcal{I} \in\mathcal{C} I ∈ C (MVP(I \mathcal{I} I ))
initialization: set V H + 1 k , I ( s ) ← H V_{H+1}^{k,\mathcal{I}}(s) \leftarrow H V H + 1 k , I ( s ) ← H for all s ∈ S s \in\mathcal{S} s ∈ S and 1 ≤ k ≤ K 1 \le k \le K 1 ≤ k ≤ K .
for k = 1 , 2 , … , K k = 1,2,\ldots,K k = 1 , 2 , … , K do
for h = H , H − 1 , … , 1 h = H,H-1,\ldots,1 h = H , H − 1 , … , 1 do
for ( s , a ) ∈ S × A (s,a) \in\mathcal{S} \times\mathcal{A} ( s , a ) ∈ S × A do
j ← I s , a , h k , n ← 2 j − 2 , j \leftarrow\mathcal{I}_{s,a,h}^k,\quad n \leftarrow2^{j-2}, j ← I s , a , h k , n ← 2 j − 2 ,
b h ( s , a ) ← c 1 Var P ^ s , a , h ( j ) ( V h + 1 k , I ) log 1 δ max { n , 1 } + c 2 H log 1 δ max { n , 1 } , b_h(s,a) \leftarrow c_1\sqrt{\frac{\operatorname{Var}_{\widehat{P}_{s,a,h}^{(j)}}(V_{h+1}^{k,\mathcal{I}})\log\frac{1}{\delta}}{\max\{n,1\}}}+c_2\frac{H\log\frac{1}{\delta}}{\max\{n,1\}}, b h ( s , a ) ← c 1 m a x { n , 1 } Var P s , a , h ( j ) ( V h + 1 k , I ) l o g δ 1 + c 2 m a x { n , 1 } H l o g δ 1 ,
Q h k , I ( s , a ) ← min { r h ( s , a ) + ⟨ P ^ s , a , h ( j ) , V h + 1 k , I ⟩ + b h ( s , a ) , H } , Q_h^{k,\mathcal{I}}(s,a) \leftarrow\min\left\{r_h(s,a)+\left\langle\widehat{P}_{s,a,h}^{(j)},V_{h+1}^{k,\mathcal{I}}\right\rangle+b_h(s,a),H\right\}, Q h k , I ( s , a ) ← min { r h ( s , a ) + ⟨ P s , a , h ( j ) , V h + 1 k , I ⟩ + b h ( s , a ) , H } ,
V h k , I ( s ) ← max a Q h k , I ( s , a ) V_h^{k,\mathcal{I}}(s) \leftarrow\max_a Q_h^{k,\mathcal{I}}(s,a) V h k , I ( s ) ← max a Q h k , I ( s , a ) .
Now for each h ∈ [ H ] h \in[H] h ∈ [ H ] and I ∈ C \mathcal{I} \in\mathcal{C} I ∈ C , define
X h , I : = { V h k , I ∣ 1 ≤ k ≤ K } ∪ { 0 } , ∀ h ∈ [ H ] and I ∈ C . (529) \mathcal{X}_{h,\mathcal{I}}:=\left\{V_h^{k,\mathcal{I}}\mid1\le k\le K\right\}\cup\{0\},\qquad\forall h\in[H]\text{ and }\mathcal{I}\in\mathcal{C}. \tag*{(529)} X h , I := { V h k , I ∣ 1 ≤ k ≤ K } ∪ { 0 } , ∀ h ∈ [ H ] and I ∈ C . (529)
By construction, the collections { X h , I } \{\mathcal{X}_{h,\mathcal{I}}\} { X h , I } satisfy Assumption 5.1. Hence, Lemma 5.13 implies that
∑ s , a , h ⟨ P ^ s , a , h ( l ) − P s , a , h , X h + 1 , s , a ⟩ ≤ S A H ι b 2 l log 2 K ∑ s , a , h Var P s , a , h ( X h + 1 , s , a ) + S A H 2 ι b 2 l log 2 K (530) \sum_{s,a,h}\left\langle\widehat{P}_{s,a,h}^{(l)}-P_{s,a,h},X_{h+1,s,a}\right\rangle\le\sqrt{\frac{SAH\iota_b}{2^l\log_2 K}\sum_{s,a,h}\operatorname{Var}_{P_{s,a,h}}(X_{h+1,s,a})+\frac{SAH^2\iota_b}{2^l\log_2 K}} \tag*{(530)} s , a , h ∑ ⟨ P s , a , h ( l ) − P s , a , h , X h + 1 , s , a ⟩ ≤ 2 l log 2 K S A H ι b s , a , h ∑ Var P s , a , h ( X h + 1 , s , a ) + 2 l log 2 K S A H 2 ι b (530)
simultaneously for all l = 1 , … , log 2 K l=1,\ldots,\log_2 K l = 1 , … , log 2 K , all I ∈ C \mathcal{I}\in\mathcal{C} I ∈ C , and all sequences { X h , s , a } \{X_{h,s,a}\} { X h , s , a } obeying X h , s , a ∈ X h , I X_{h,s,a}\in\mathcal{X}_{h,\mathcal{I}} X h , s , a ∈ X h , I , ∀ ( s , a , h ) \forall(s,a,h) ∀ ( s , a , h ) .
It remains to specialize this uniform bound to the profile I t r u e \mathcal{I}^{\mathrm{true}} I true generated by the actual online process. Under the coupling between D e x p a n d \mathcal{D}^{\mathrm{expand}} D expand and D o r i g \mathcal{D}^{\mathrm{orig}} D orig described after Definition 5.1, the value function estimates produced by the online process coincide with those generated by the auxiliary algorithm under the profile I t r u e I^{\mathrm{true}} I true , and hence
V h k = V h k , I t r u e ∈ X h , I t r u e , 1 ≤ k ≤ K . (531) V_h^k = V_h^{k,I^{\mathrm{true}}} \in\mathcal{X}_{h,I^{\mathrm{true}}}, \qquad1 \leq k \leq K. \tag*{(531)} V h k = V h k , I true ∈ X h , I true , 1 ≤ k ≤ K . (531)
The claim then follows directly from (531) and the uniform bound (530) .
Proof of Lemma 5.15
We begin by decomposing T 6 u c b T_6^{\mathrm{ucb}} T 6 ucb (cf. (428) ) as follows
T 6 u c b = ∑ k = 1 K ∑ h = 1 H ⟨ P s h k , a h k , h , ( V h + 1 k ) 2 ⟩ − ∑ k = 1 K ∑ h = 1 H ( ⟨ P s h k , a h k , h , V h + 1 k ⟩ ) 2 = ∑ k = 1 K ∑ h = 1 H ⟨ P s h k , a h k , h − e s h + 1 k , ( V h + 1 k ) 2 ⟩ + ∑ k = 1 K ∑ h = 1 H ( V h + 1 k ( s h + 1 k ) ) 2 − ∑ k = 1 K ∑ h = 1 H ( ⟨ P s h k , a h k , h , V h + 1 k ⟩ ) 2 = ( i ) ∑ k = 1 K ∑ h = 1 H ⟨ P s h k , a h k , h − e s h + 1 k , ( V h + 1 k ) 2 ⟩ + ∑ k = 1 K ∑ h = 2 H ( V h k ( s h k ) ) 2 − ∑ k = 1 K ∑ h = 1 H ( ⟨ P s h k , a h k , h , V h + 1 k ⟩ ) 2 ≤ ( i i ) T 8 + 2 H ∑ k = 1 K ∑ h = 1 H max { V h k ( s h k ) − ⟨ P s h k , a h k , h , V h + 1 k ⟩ , 0 } = ( i i i ) T 8 + 2 H ∑ k = 1 K ∑ h = 1 H max { Q h k ( s h k , a h k ) − ⟨ P s h k , a h k , h , V h + 1 k ⟩ , 0 } . \begin{aligned}
T_6^{\mathrm{ucb}}
&= \sum_{k=1}^{K}\sum_{h=1}^{H}\left\langle P_{s_h^k,a_h^k,h},(V_{h+1}^k)^2\right\rangle-\sum_{k=1}^{K}\sum_{h=1}^{H}\left(\left\langle P_{s_h^k,a_h^k,h},V_{h+1}^k\right\rangle\right)^2\\
&= \sum_{k=1}^{K}\sum_{h=1}^{H}\left\langle P_{s_h^k,a_h^k,h}-e_{s_{h+1}^k},(V_{h+1}^k)^2\right\rangle+\sum_{k=1}^{K}\sum_{h=1}^{H}(V_{h+1}^k(s_{h+1}^k))^2-\sum_{k=1}^{K}\sum_{h=1}^{H}\left(\left\langle P_{s_h^k,a_h^k,h},V_{h+1}^k\right\rangle\right)^2\\
&\mathrel{\overset{\mathrm{(i)}}{=}} \sum_{k=1}^{K}\sum_{h=1}^{H}\left\langle P_{s_h^k,a_h^k,h}-e_{s_{h+1}^k},(V_{h+1}^k)^2\right\rangle+\sum_{k=1}^{K}\sum_{h=2}^{H}(V_h^k(s_h^k))^2-\sum_{k=1}^{K}\sum_{h=1}^{H}\left(\left\langle P_{s_h^k,a_h^k,h},V_{h+1}^k\right\rangle\right)^2\\
&\mathrel{\overset{\mathrm{(ii)}}{\leq}} T_8+2H\sum_{k=1}^{K}\sum_{h=1}^{H}\max\left\{V_h^k(s_h^k)-\left\langle P_{s_h^k,a_h^k,h},V_{h+1}^k\right\rangle,0\right\}\\
&\mathrel{\overset{\mathrm{(iii)}}{=}} T_8+2H\sum_{k=1}^{K}\sum_{h=1}^{H}\max\left\{Q_h^k(s_h^k,a_h^k)-\left\langle P_{s_h^k,a_h^k,h},V_{h+1}^k\right\rangle,0\right\}.
\end{aligned} T 6 ucb = k = 1 ∑ K h = 1 ∑ H ⟨ P s h k , a h k , h , ( V h + 1 k ) 2 ⟩ − k = 1 ∑ K h = 1 ∑ H ( ⟨ P s h k , a h k , h , V h + 1 k ⟩ ) 2 = k = 1 ∑ K h = 1 ∑ H ⟨ P s h k , a h k , h − e s h + 1 k , ( V h + 1 k ) 2 ⟩ + k = 1 ∑ K h = 1 ∑ H ( V h + 1 k ( s h + 1 k ) ) 2 − k = 1 ∑ K h = 1 ∑ H ( ⟨ P s h k , a h k , h , V h + 1 k ⟩ ) 2 = ( i ) k = 1 ∑ K h = 1 ∑ H ⟨ P s h k , a h k , h − e s h + 1 k , ( V h + 1 k ) 2 ⟩ + k = 1 ∑ K h = 2 ∑ H ( V h k ( s h k ) ) 2 − k = 1 ∑ K h = 1 ∑ H ( ⟨ P s h k , a h k , h , V h + 1 k ⟩ ) 2 ≤ ( ii ) T 8 + 2 H k = 1 ∑ K h = 1 ∑ H max { V h k ( s h k ) − ⟨ P s h k , a h k , h , V h + 1 k ⟩ , 0 } = ( iii ) T 8 + 2 H k = 1 ∑ K h = 1 ∑ H max { Q h k ( s h k , a h k ) − ⟨ P s h k , a h k , h , V h + 1 k ⟩ , 0 } .
Here, (i) replaces h + 1 h+1 h + 1 with h h h in the second sum and uses V H + 1 k = 0 V_{H+1}^k=0 V H + 1 k = 0 ; in (ii), we invokes the definition of T 8 T_8 T 8 in (517) together with the elementary inequality
( V h k ( s h k ) ) 2 − ( ⟨ P s h k , a h k , h , V h + 1 k ⟩ ) 2 = ( V h k ( s h k ) − ⟨ P s h k , a h k , h , V h + 1 k ⟩ ) ( V h k ( s h k ) + ⟨ P s h k , a h k , h , V h + 1 k ⟩ ) ≤ 2 H max { V h k ( s h k ) − ⟨ P s h k , a h k , h , V h + 1 k ⟩ , 0 } ; \left(V_h^k(s_h^k)\right)^2-\left(\left\langle P_{s_h^k,a_h^k,h},V_{h+1}^k\right\rangle\right)^2
=\left(V_h^k(s_h^k)-\left\langle P_{s_h^k,a_h^k,h},V_{h+1}^k\right\rangle\right)\left(V_h^k(s_h^k)+\left\langle P_{s_h^k,a_h^k,h},V_{h+1}^k\right\rangle\right)
\leq2H\max\left\{V_h^k(s_h^k)-\left\langle P_{s_h^k,a_h^k,h},V_{h+1}^k\right\rangle,0\right\}; ( V h k ( s h k ) ) 2 − ( ⟨ P s h k , a h k , h , V h + 1 k ⟩ ) 2 = ( V h k ( s h k ) − ⟨ P s h k , a h k , h , V h + 1 k ⟩ ) ( V h k ( s h k ) + ⟨ P s h k , a h k , h , V h + 1 k ⟩ ) ≤ 2 H max { V h k ( s h k ) − ⟨ P s h k , a h k , h , V h + 1 k ⟩ , 0 } ;
and (iii) uses the greedy update V h k ( s h k ) = max a Q h k ( s h k , a ) = Q h k ( s h k , a h k ) V_h^k(s_h^k)=\max_a Q_h^k(s_h^k,a)=Q_h^k(s_h^k,a_h^k) V h k ( s h k ) = max a Q h k ( s h k , a ) = Q h k ( s h k , a h k ) .
The remaining issue is that the update rule (5.46) is expressed in terms of the empirical transition kernel P ^ k \widehat{P}^k P k , whereas the preceding display still involves the true transition kernel P P P . To address this, we incorporate P ^ s h k , a h k , h k \widehat{P}_{s_h^k,a_h^k,h}^k P s h k , a h k , h k into the max operator to yield
T 6 u c b ≤ T 8 + 2 H ∑ k = 1 K ∑ h = 1 H max { Q h k ( s h k , a h k ) − ⟨ P ^ s h k , a h k , h k , V h + 1 k ⟩ , 0 } + 2 H ∑ k = 1 K ∑ h = 1 H max { ⟨ P ^ s h k , a h k , h k − P s h k , a h k , h , V h + 1 k ⟩ , 0 } ≤ T 8 + 2 H ∑ k = 1 K ∑ h = 1 H b h k ( s h k , a h k ) + 2 H ∑ k = 1 K ∑ h = 1 H r h ( s h k , a h k ) + 2 H W 4 ≤ T 8 + 2 H T 2 + 2 K H 2 + 2 H W 4 . (532) \begin{aligned}
T_6^{\mathrm{ucb}}
&\leq T_8+2H\sum_{k=1}^{K}\sum_{h=1}^{H}\max\left\{Q_h^k(s_h^k,a_h^k)-\left\langle\widehat{P}_{s_h^k,a_h^k,h}^k,V_{h+1}^k\right\rangle,0\right\}\\
&\quad+2H\sum_{k=1}^{K}\sum_{h=1}^{H}\max\left\{\left\langle\widehat{P}_{s_h^k,a_h^k,h}^k-P_{s_h^k,a_h^k,h},V_{h+1}^k\right\rangle,0\right\}\\
&\leq T_8+2H\sum_{k=1}^{K}\sum_{h=1}^{H}b_h^k(s_h^k,a_h^k)+2H\sum_{k=1}^{K}\sum_{h=1}^{H}r_h(s_h^k,a_h^k)+2HW_4\\
&\leq T_8+2HT_2+2KH^2+2HW_4. \tag*{(532)}
\end{aligned} T 6 ucb ≤ T 8 + 2 H k = 1 ∑ K h = 1 ∑ H max { Q h k ( s h k , a h k ) − ⟨ P s h k , a h k , h k , V h + 1 k ⟩ , 0 } + 2 H k = 1 ∑ K h = 1 ∑ H max { ⟨ P s h k , a h k , h k − P s h k , a h k , h , V h + 1 k ⟩ , 0 } ≤ T 8 + 2 H k = 1 ∑ K h = 1 ∑ H b h k ( s h k , a h k ) + 2 H k = 1 ∑ K h = 1 ∑ H r h ( s h k , a h k ) + 2 H W 4 ≤ T 8 + 2 H T 2 + 2 K H 2 + 2 H W 4 . (532)
where the second inequality follows from the update rule (5.46) together with the definition of W 4 W_4 W 4 in (430) . Combining this inequality with the bound (520) on T 8 T_8 T 8 gives
T 6 u c b ≤ 2 H T 2 + 6 K H 2 + 32 H 2 T 6 u c b log 1 δ ′ + 3 H 2 log 1 δ ′ + 2 H W 4 T_6^{\mathrm{ucb}}\leq2HT_2+6KH^2+\sqrt{32H^2T_6^{\mathrm{ucb}}\log\frac{1}{\delta'}+3H^2\log\frac{1}{\delta'}}+2HW_4 T 6 ucb ≤ 2 H T 2 + 6 K H 2 + 32 H 2 T 6 ucb log δ ′ 1 + 3 H 2 log δ ′ 1 + 2 H W 4
≤ 2 H T 2 + 6 K H 2 + 1 2 T 6 u c b + 16 H 2 log 1 δ ′ + 3 H 2 log 1 δ ′ + 2 H W 4 , \le2HT_2 + 6KH^2 + \frac{1}{2}T_6^{\mathrm{ucb}} + 16H^2\log\frac{1}{\delta'} + 3H^2\log\frac{1}{\delta'} + 2HW_4, ≤ 2 H T 2 + 6 K H 2 + 2 1 T 6 ucb + 16 H 2 log δ ′ 1 + 3 H 2 log δ ′ 1 + 2 H W 4 ,
where the last line applied the AM-GM inequality. Rearranging terms and using the definition (416) of ι b \iota_b ι b yield, with probability exceeding 1 − 4 S A H K δ ′ 1-4SAHK\delta' 1 − 4 S A H K δ ′ ,
T 6 u c b ≤ 4 H T 2 + 4 H W 4 + K H 2 ι b . T_6^{\mathrm{ucb}} \le4HT_2 + 4HW_4 + KH^2\iota_b. T 6 ucb ≤ 4 H T 2 + 4 H W 4 + K H 2 ι b .
Solving the inequalities in (5.61)
We now solve the intertwined inequalities in (5.61). The strategy is to successively eliminate the auxiliary quantities T 6 u c b T_6^{\mathrm{ucb}} T 6 ucb and T 5 T_5 T 5 , after which the remaining bounds follow immediately.
We first tackle T 6 u c b T_6^{\mathrm{ucb}} T 6 ucb . Substituting (431) into (437) and invoking the AM-GM inequality give
T 6 u c b ≤ 4 H T 2 + 4 H T 6 u c b S A H ι b + 4 S A H 3 ι b + K H 2 ι b T_6^{\mathrm{ucb}} \le4HT_2 + 4H\sqrt{T_6^{\mathrm{ucb}}SAH\iota_b} + 4SAH^3\iota_b + KH^2\iota_b T 6 ucb ≤ 4 H T 2 + 4 H T 6 ucb S A H ι b + 4 S A H 3 ι b + K H 2 ι b
≤ 4 H T 2 + 1 2 T 6 u c b + 8 S A H 3 ι b + 4 S A H 3 ι b + K H 2 ι b , \le4HT_2 + \frac{1}{2}T_6^{\mathrm{ucb}} + 8SAH^3\iota_b + 4SAH^3\iota_b + KH^2\iota_b, ≤ 4 H T 2 + 2 1 T 6 ucb + 8 S A H 3 ι b + 4 S A H 3 ι b + K H 2 ι b ,
which together with a little algebra yields
T 6 u c b ≤ 8 H T 2 + 24 S A H 3 ι b + 2 K H 2 ι b . (533) T_6^{\mathrm{ucb}} \le8HT_2 + 24SAH^3\iota_b + 2KH^2\iota_b. \tag*{(533)} T 6 ucb ≤ 8 H T 2 + 24 S A H 3 ι b + 2 K H 2 ι b . (533)
Next, we turn to T 5 T_5 T 5 . It follows from (436) and (432) that
T 5 ≤ 2 H T 6 u c b S A H ι b + 2 H T 2 + S A H 3 ι b + K H 2 ι b ≤ 1 2 T 6 u c b + 2 S A H 3 ι b + 2 H T 2 + S A H 3 ι b + K H 2 ι b \begin{aligned}
T_5 \le2H\sqrt{T_6^{\mathrm{ucb}}SAH\iota_b} + 2HT_2 + SAH^3\iota_b + KH^2\iota_b \\
&\le\frac{1}{2}T_6^{\mathrm{ucb}} + 2SAH^3\iota_b + 2HT_2 + SAH^3\iota_b + KH^2\iota_b
\end{aligned} T 5 ≤ 2 H T 6 ucb S A H ι b + 2 H T 2 + S A H 3 ι b + K H 2 ι b ≤ 2 1 T 6 ucb + 2 S A H 3 ι b + 2 H T 2 + S A H 3 ι b + K H 2 ι b
≤ 4 H T 2 + 12 S A H 3 ι b + K H 2 ι b + 2 S A H 3 ι b + 2 H T 2 + S A H 3 ι b + K H 2 ι b , \le4HT_2 + 12SAH^3\iota_b + KH^2\iota_b + 2SAH^3\iota_b + 2HT_2 + SAH^3\iota_b + KH^2\iota_b, ≤ 4 H T 2 + 12 S A H 3 ι b + K H 2 ι b + 2 S A H 3 ι b + 2 H T 2 + S A H 3 ι b + K H 2 ι b ,
where the second inequality arises from the AM-GM inequality, and the third inequality results from (533) . Combine terms to arrive at
T 5 ≤ 6 H T 2 + 15 S A H 3 ι b + 2 K H 2 ι b . (534) T_5 \le6HT_2 + 15SAH^3\iota_b + 2KH^2\iota_b. \tag*{(534)} T 5 ≤ 6 H T 2 + 15 S A H 3 ι b + 2 K H 2 ι b . (534)
In addition, it is also seen from (433) and the AM-GM inequality that
T 2 ≤ 1 12 H T 5 + 3 S A H 2 ι b + 1 2 S A H 2 ι b , (535) T_2 \le\frac{1}{12H}T_5 + 3SAH^2\iota_b + \frac{1}{2}SAH^2\iota_b, \tag*{(535)} T 2 ≤ 12 H 1 T 5 + 3 S A H 2 ι b + 2 1 S A H 2 ι b , (535)
which combined with (534) reveals that
T 5 ≤ 1 2 T 5 + 36 S A H 3 ι b + 2 K H 2 ι b , ⟹ T 5 ≤ 72 S A H 3 ι b + 4 K H 2 ι b . (536) T_5 \le\frac{1}{2}T_5 + 36SAH^3\iota_b + 2KH^2\iota_b,
\qquad\Longrightarrow\qquad
T_5 \le72SAH^3\iota_b + 4KH^2\iota_b. \tag*{(536)} T 5 ≤ 2 1 T 5 + 36 S A H 3 ι b + 2 K H 2 ι b , ⟹ T 5 ≤ 72 S A H 3 ι b + 4 K H 2 ι b . (536)
Armed with inequality (536) , we can immediately combine it with (433) and (533) to yield
T 2 ≲ S A H 3 K ι b + S A H 2 ι b , (537) T_2 \lesssim\sqrt{SAH^3K\iota_b} + SAH^2\iota_b, \tag*{(537)} T 2 ≲ S A H 3 K ι b + S A H 2 ι b , (537)
T 6 u c b ≲ S A H 3 ι b + K H 2 ι b . (538) T_6^{\mathrm{ucb}} \lesssim SAH^3\iota_b + KH^2\iota_b. \tag*{(538)} T 6 ucb ≲ S A H 3 ι b + K H 2 ι b . (538)
Substitution into (431) and (434) then leads to
T 1 + ∣ T 3 ∣ ≲ S A H 3 K ι b + S A H 2 ι b . T_1 + |T_3| \lesssim\sqrt{SAH^3K\iota_b} + SAH^2\iota_b. T 1 + ∣ T 3 ∣ ≲ S A H 3 K ι b + S A H 2 ι b .
Putting these two bounds collectively with (537) and (435) , we reach
Regret ( K ) ≤ T 1 + T 2 + T 3 + T 4 ≲ S A H 3 K ι b + S A H 2 ι b ≍ S A H 3 K ι b , \operatorname{Regret}(K) \le T_1 + T_2 + T_3 + T_4 \lesssim\sqrt{SAH^3K\iota_b} + SAH^2\iota_b \asymp\sqrt{SAH^3K\iota_b}, Regret ( K ) ≤ T 1 + T 2 + T 3 + T 4 ≲ S A H 3 K ι b + S A H 2 ι b ≍ S A H 3 K ι b ,
provided that K ≥ S A H K \ge SAH K ≥ S A H .
Appendix C: proof of auxiliary lemmas in Section 5.5
Proof of Lemma 5.17
Let us first prove (448) . In view of the definitions (5.64) and (5.67a), for any 1 ≤ n ≤ N 1 \le n \le N 1 ≤ n ≤ N ,
η n N = H + 1 H + n ∏ i = n + 1 N i − 1 H + i = ( H + 1 ) ∏ i = n N − 1 i ∏ i = n N ( H + i ) (539) \eta_n^N = \frac{H+1}{H+n}\prod_{i=n+1}^{N}\frac{i-1}{H+i} = \frac{(H+1)\prod_{i=n}^{N-1}i}{\prod_{i=n}^{N}(H+i)} \tag*{(539)} η n N = H + n H + 1 i = n + 1 ∏ N H + i i − 1 = ∏ i = n N ( H + i ) ( H + 1 ) ∏ i = n N − 1 i (539)
= H + 1 H + N ∏ i = n N − 1 i H + i ≤ H + 1 H + N ≤ 2 H N . (540) = \frac{H+1}{H+N}\prod_{i=n}^{N-1}\frac{i}{H+i} \le\frac{H+1}{H+N} \le\frac{2H}{N}. \tag*{(540)} = H + N H + 1 i = n ∏ N − 1 H + i i ≤ H + N H + 1 ≤ N 2 H . (540)
This combined with the fact ∑ n = 1 N η n N = 1 \sum_{n=1}^{N}\eta_n^N=1 ∑ n = 1 N η n N = 1 (see (446) ) implies that
∑ n = 1 N ( η n N ) 2 ≤ ( max n : 1 ≤ n ≤ N η n N ) ∑ n = 1 N η n N ≤ 2 H N . (541) \sum_{n=1}^{N}(\eta_n^N)^2 \le\left(\max_{n:1\le n\le N}\eta_n^N\right)\sum_{n=1}^{N}\eta_n^N \le\frac{2H}{N}. \tag*{(541)} n = 1 ∑ N ( η n N ) 2 ≤ ( n : 1 ≤ n ≤ N max η n N ) n = 1 ∑ N η n N ≤ N 2 H . (541)
We next verify (449) . For any given N ≥ n ≥ 1 N \ge n \ge1 N ≥ n ≥ 1 , recall that
η n N = H + 1 H + n a N with a N : = { ∏ i = n + 1 N i − 1 H + i , if N > n , 1 , if N = n . (542) \eta_n^N=\frac{H+1}{H+n}a_N
\quad\text{with}\quad
a_N:=\begin{cases}
\prod_{i=n+1}^{N}\frac{i-1}{H+i}, & \text{if }N>n,\\
1, & \text{if }N=n.
\end{cases} \tag*{(542)} η n N = H + n H + 1 a N with a N := { ∏ i = n + 1 N H + i i − 1 , 1 , if N > n , if N = n . (542)
It is easily seen that
a N = H + N H a N − H + N + 1 H a N + 1 , (543) a_N=\frac{H+N}{H}a_N-\frac{H+N+1}{H}a_{N+1}, \tag*{(543)} a N = H H + N a N − H H + N + 1 a N + 1 , (543)
and hence the partial sum of a N a_N a N telescopes (for any L ≥ n L\ge n L ≥ n ):
∑ N = n L a N = H + n H − H + L + 1 H a L + 1 . (544) \sum_{N=n}^{L}a_N=\frac{H+n}{H}-\frac{H+L+1}{H}a_{L+1}. \tag*{(544)} N = n ∑ L a N = H H + n − H H + L + 1 a L + 1 . (544)
Recognizing that the last term H + L + 1 H a L + 1 \frac{H+L+1}{H}a_{L+1} H H + L + 1 a L + 1 vanishes as L → ∞ L\to\infty L → ∞ , we arrive at
∑ N = n ∞ η n N = ∑ N = n ∞ H + 1 H + n a N = H + 1 H + n ⋅ H + n H = 1 + 1 H . (545) \sum_{N=n}^{\infty}\eta_n^N=\sum_{N=n}^{\infty}\frac{H+1}{H+n}a_N=\frac{H+1}{H+n}\cdot\frac{H+n}{H}=1+\frac{1}{H}. \tag*{(545)} N = n ∑ ∞ η n N = N = n ∑ ∞ H + n H + 1 a N = H + n H + 1 ⋅ H H + n = 1 + H 1 . (545)
It remains to establish property (447) . When N = 1 N=1 N = 1 , this property holds trivially. Now suppose that N ≥ 2 N\ge2 N ≥ 2 . Making use of the basic relation η n N = ( 1 − η N ) η n N − 1 \eta_n^N=(1-\eta_N)\eta_n^{N-1} η n N = ( 1 − η N ) η n N − 1 for all 1 ≤ n ≤ N − 1 1\le n\le N-1 1 ≤ n ≤ N − 1 , we observe
∑ n = 1 N η n N n = η N N + ( 1 − η N ) ∑ n = 1 N − 1 η n N − 1 n . (546) \sum_{n=1}^{N}\frac{\eta_n^N}{\sqrt{n}}=\frac{\eta_N}{\sqrt{N}}+(1-\eta_N)\sum_{n=1}^{N-1}\frac{\eta_n^{N-1}}{\sqrt{n}}. \tag*{(546)} n = 1 ∑ N n η n N = N η N + ( 1 − η N ) n = 1 ∑ N − 1 n η n N − 1 . (546)
We prove the property by induction. Suppose for the moment that the property holds for N − 1 N-1 N − 1 , i.e.,
1 N − 1 ≤ ∑ n = 1 N − 1 η n N − 1 n ≤ 2 N − 1 . (547) \frac{1}{\sqrt{N-1}}\le\sum_{n=1}^{N-1}\frac{\eta_n^{N-1}}{\sqrt{n}}\le\frac{2}{\sqrt{N-1}}. \tag*{(547)} N − 1 1 ≤ n = 1 ∑ N − 1 n η n N − 1 ≤ N − 1 2 . (547)
Then it is readily seen from (546) that
∑ n = 1 N η n N n = η N N + ( 1 − η N ) ∑ n = 1 N − 1 η n N − 1 n ≥ η N N + 1 − η N N − 1 ≥ η N N + 1 − η N N = 1 N , (548) \begin{aligned}
\sum_{n=1}^{N}\frac{\eta_n^N}{\sqrt{n}}
&=\frac{\eta_N}{\sqrt{N}}+(1-\eta_N)\sum_{n=1}^{N-1}\frac{\eta_n^{N-1}}{\sqrt{n}}\\
&\ge\frac{\eta_N}{\sqrt{N}}+\frac{1-\eta_N}{\sqrt{N-1}}
\ge\frac{\eta_N}{\sqrt{N}}+\frac{1-\eta_N}{\sqrt{N}}
=\frac{1}{\sqrt{N}}, \tag*{(548)}
\end{aligned} n = 1 ∑ N n η n N = N η N + ( 1 − η N ) n = 1 ∑ N − 1 n η n N − 1 ≥ N η N + N − 1 1 − η N ≥ N η N + N 1 − η N = N 1 , (548)
where the first inequality comes from (547) . Similarly,
∑ n = 1 N η n N n = η N N + ( 1 − η N ) ∑ n = 1 N − 1 η n N − 1 n ≤ η N N + 2 ( 1 − η N ) N − 1 = H + 1 N ( H + N ) + 2 N − 1 H + N ≤ H + 1 N ( H + N ) + 2 N H + N = 1 N ( H + 1 H + N + 2 N H + N ) ≤ 2 N , \begin{aligned}
\sum_{n=1}^{N} \frac{\eta_n^N}{\sqrt{n}} &= \frac{\eta_N}{\sqrt{N}}+(1-\eta_N)\sum_{n=1}^{N-1}\frac{\eta_n^{N-1}}{\sqrt{n}} \\
&\le\frac{\eta_N}{\sqrt{N}}+\frac{2(1-\eta_N)}{\sqrt{N-1}}=\frac{H+1}{\sqrt{N}(H+N)}+\frac{2\sqrt{N-1}}{H+N} \\
&\le\frac{H+1}{\sqrt{N}(H+N)}+\frac{2\sqrt{N}}{H+N} \\
&=\frac{1}{\sqrt{N}}\left(\frac{H+1}{H+N}+\frac{2N}{H+N}\right)\le\frac{2}{\sqrt{N}},
\end{aligned} n = 1 ∑ N n η n N = N η N + ( 1 − η N ) n = 1 ∑ N − 1 n η n N − 1 ≤ N η N + N − 1 2 ( 1 − η N ) = N ( H + N ) H + 1 + H + N 2 N − 1 ≤ N ( H + N ) H + 1 + H + N 2 N = N 1 ( H + N H + 1 + H + N 2 N ) ≤ N 2 ,
where the last inequality follows since H ≥ 1 H \ge1 H ≥ 1 . This completes the induction and finishes the proof.
Proof of Lemma 5.18
Consider any given ( s , a , h ) ∈ S × A × [ H ] (s,a,h) \in\mathcal{S} \times\mathcal{A} \times[H] ( s , a , h ) ∈ S × A × [ H ] . When t = 0 t=0 t = 0 , the claim follows directly from Q 0 k ( s , a ) = H Q_0^k(s,a)=H Q 0 k ( s , a ) = H (as no update has yet occurred), 0 ≤ Q ⋆ ( s , a ) ≤ H 0 \le Q^\star(s,a) \le H 0 ≤ Q ⋆ ( s , a ) ≤ H and η 0 0 = 1 \eta_0^0=1 η 0 0 = 1 (cf. (445) ). Therefore, it suffices to focus on the case t ≥ 1 t \ge1 t ≥ 1 in the remainder of this proof.
As before, let k i k_i k i denote the index of the episode in which ( s , a ) (s,a) ( s , a ) is visited at step h h h for the i i i -th time. As can be readily seen, the sequence
Z i : = ( P h k i − P s , a , h ) V h + 1 ⋆ (549) Z_i := (P_h^{k_i}-P_{s,a,h})V_{h+1}^\star \tag*{(549)} Z i := ( P h k i − P s , a , h ) V h + 1 ⋆ (549)
forms a martingale difference sequence satisfying ∣ Z i ∣ ≤ H |Z_i| \le H ∣ Z i ∣ ≤ H . Therefore, by applying the Azuma-Hoeffding inequality (cf. Lemma 2.8) and a union bound over all ( s , a , h ) (s,a,h) ( s , a , h ) and all visit counts 1 ≤ t ≤ K 1 \le t \le K 1 ≤ t ≤ K , we can show that, with probability at least 1 − δ 1-\delta 1 − δ ,
∣ ∑ i = 1 t η i t Z i ∣ ≤ C 1 H ∑ i = 1 t ( η i t ) 2 ≤ C 2 H 3 ι t (550) \left|\sum_{i=1}^{t}\eta_i^t Z_i\right|\le C_1H\sqrt{\sum_{i=1}^{t}(\eta_i^t)^2}\le C_2\sqrt{\frac{H^3\iota}{t}} \tag*{(550)} i = 1 ∑ t η i t Z i ≤ C 1 H i = 1 ∑ t ( η i t ) 2 ≤ C 2 t H 3 ι (550)
holds simultaneously for all ( s , a , h , t ) (s,a,h,t) ( s , a , h , t ) , where C 1 , C 2 > 0 C_1,C_2>0 C 1 , C 2 > 0 are some universal constants, and the last inequality is due to (448) . On the same event, if the constant c b c_b c b in (439) is chosen sufficiently large, then it follows from (447) that
∑ i = 1 t η i t b h k i ( s , a ) = c b H 3 ι ∑ i = 1 t η i t i ≥ c b H 3 ι t ≥ ∣ ∑ i = 1 t η i t Z i ∣ . (551) \sum_{i=1}^{t}\eta_i^t b_h^{k_i}(s,a)=c_b\sqrt{H^3\iota}\sum_{i=1}^{t}\frac{\eta_i^t}{\sqrt{i}}\ge c_b\sqrt{\frac{H^3\iota}{t}}\ge\left|\sum_{i=1}^{t}\eta_i^t Z_i\right|. \tag*{(551)} i = 1 ∑ t η i t b h k i ( s , a ) = c b H 3 ι i = 1 ∑ t i η i t ≥ c b t H 3 ι ≥ i = 1 ∑ t η i t Z i . (551)
Similar arguments combined with (447) also give
2 ∑ i = 1 t η i t b h k i ( s , a ) ≤ 4 c b H 3 ι t . (552) 2\sum_{i=1}^{t}\eta_i^t b_h^{k_i}(s,a)\le4c_b\sqrt{\frac{H^3\iota}{t}}. \tag*{(552)} 2 i = 1 ∑ t η i t b h k i ( s , a ) ≤ 4 c b t H 3 ι . (552)
Therefore, we can readily establish (5.74) by combining (453) , (551) and (552) .
Next, we prove the optimism property (456) by induction. The property holds trivially for the base case with k = 0 k=0 k = 0 due to the choice of initialization. Now, assume that (456) holds for all episodes up to k k k . In view of the recursion (453) and (551) , one has
( Q h k − Q h ⋆ ) ( s , a ) ≥ η 0 t ( H − Q h ⋆ ( s , a ) ) + ∑ i = 1 t η i t ( V h + 1 k i − V h + 1 ⋆ ) ( s h + 1 k i ) ≥ 0. (Q_h^k-Q_h^\star)(s,a)\ge\eta_0^t(H-Q_h^\star(s,a))+\sum_{i=1}^{t}\eta_i^t(V_{h+1}^{k_i}-V_{h+1}^\star)(s_{h+1}^{k_i})\ge0. ( Q h k − Q h ⋆ ) ( s , a ) ≥ η 0 t ( H − Q h ⋆ ( s , a )) + i = 1 ∑ t η i t ( V h + 1 k i − V h + 1 ⋆ ) ( s h + 1 k i ) ≥ 0.
It then follows that
max a ∈ A Q h k ( s , a ) ≥ max a ∈ A Q h ⋆ ( s , a ) = V h ⋆ ( s ) , \max_{a\in\mathcal{A}} Q_h^k(s,a) \ge\max_{a\in\mathcal{A}} Q_h^\star(s,a) = V_h^\star(s), a ∈ A max Q h k ( s , a ) ≥ a ∈ A max Q h ⋆ ( s , a ) = V h ⋆ ( s ) ,
and the operation V h k ( s ) = min { H , max a Q h k ( s , a ) } V_h^k(s)=\min\{H,\max_{a}Q_h^k(s,a)\} V h k ( s ) = min { H , max a Q h k ( s , a )} preserves the inequality since V h ⋆ ( s ) ≤ H V_h^\star(s)\le H V h ⋆ ( s ) ≤ H . This completes the induction and hence the proof.
Appendix D: proof of auxiliary lemmas in Section 5.6
Proof of Lemma 5.20
We begin by establishing several basic properties of the constructed MDPs, which will then be used to prove the desired lower bound.
Basic properties of these MDPs. We now isolate several basic properties that hold for an arbitrary MDP M ∈ C m d p s u b \mathcal{M}\in\mathcal{C}_{\mathrm{mdp}}^{\mathrm{sub}} M ∈ C mdp sub . For notational convenience, we suppress the dependence of quantities (e.g., Q h π Q_h^\pi Q h π , V h π V_h^\pi V h π , d h π d_h^\pi d h π , a h ⋆ , M a_h^{\star,\mathcal{M}} a h ⋆ , M , P h M P_h^\mathcal{M} P h M ) on the underlying MDP M \mathcal{M} M , as long as it is clear from the context.
Q h π ( 0 , 0 ) = 0 for all 1 ≤ h ≤ H . (553) Q_h^\pi(0,0)=0 \qquad\text{for all }1\le h\le H. \tag*{(553)} Q h π ( 0 , 0 ) = 0 for all 1 ≤ h ≤ H . (553)
Also, since V H + 1 π ( 0 ) = V H + 1 π ( 1 ) = 0 V_{H+1}^\pi(0)=V_{H+1}^\pi(1)=0 V H + 1 π ( 0 ) = V H + 1 π ( 1 ) = 0 , we see from (471) that
V H π ( 1 ) = ∑ a ∈ A 1 π H ( a ∣ 1 ) r H ( 1 , a ) = 1 , (554) V_H^\pi(1)=\sum_{a\in\mathcal{A}_1}\pi_H(a\mid1)r_H(1,a)=1, \tag*{(554)} V H π ( 1 ) = a ∈ A 1 ∑ π H ( a ∣ 1 ) r H ( 1 , a ) = 1 , (554)
V h π ( 0 ) = r h ( 0 , 0 ) = 0 for all h . (555) V_h^\pi(0)=r_h(0,0)=0 \qquad\text{for all }h. \tag*{(555)} V h π ( 0 ) = r h ( 0 , 0 ) = 0 for all h . (555)
Moreover, for any 1 ≤ h ≤ H 1\le h\le H 1 ≤ h ≤ H and any a ∈ A 1 a\in\mathcal{A}_1 a ∈ A 1 , the Bellman equation (cf. (59) ) together with (471) and (553) gives
Q h π ( 1 , a ) = r h ( 1 , a ) + P h ( 1 ∣ 1 , a ) V h + 1 π ( 1 ) + P h ( 0 ∣ 1 , a ) V h + 1 π ( 0 ) = 1 + P h ( 1 ∣ 1 , a ) V h + 1 π ( 1 ) . (556) \begin{aligned}
Q_h^\pi(1,a)
&=r_h(1,a)+P_h(1\mid1,a)V_{h+1}^\pi(1)+P_h(0\mid1,a)V_{h+1}^\pi(0)\\
&=1+P_h(1\mid1,a)V_{h+1}^\pi(1). \tag*{(556)}
\end{aligned} Q h π ( 1 , a ) = r h ( 1 , a ) + P h ( 1 ∣ 1 , a ) V h + 1 π ( 1 ) + P h ( 0 ∣ 1 , a ) V h + 1 π ( 0 ) = 1 + P h ( 1 ∣ 1 , a ) V h + 1 π ( 1 ) . (556)
According to the construction (5.92) and (5.94), we have
P h ( 1 ∣ 1 , a ) ≥ 1 − 1 / H for all a ∈ A 1 , (557) P_h(1\mid1,a)\ge1-1/H \qquad\text{for all }a\in\mathcal{A}_1, \tag*{(557)} P h ( 1 ∣ 1 , a ) ≥ 1 − 1/ H for all a ∈ A 1 , (557)
which combined with (556) immediately leads to
Q h π ( 1 , a ) ≥ 1 + ( 1 − 1 / H ) V h + 1 π ( 1 ) for all a ∈ A 1 , ⟹ V h π ( 1 ) ≥ 1 + ( 1 − 1 / H ) V h + 1 π ( 1 ) . \begin{aligned}
Q_h^\pi(1,a)&\ge1+(1-1/H)V_{h+1}^\pi(1) \qquad\text{for all }a\in\mathcal{A}_1,\\
&\Longrightarrow V_h^\pi(1)\ge1+(1-1/H)V_{h+1}^\pi(1).
\end{aligned} Q h π ( 1 , a ) ≥ 1 + ( 1 − 1/ H ) V h + 1 π ( 1 ) for all a ∈ A 1 , ⟹ V h π ( 1 ) ≥ 1 + ( 1 − 1/ H ) V h + 1 π ( 1 ) .
Iterating this recursion and using V H π ( 1 ) = 1 V_H^\pi(1)=1 V H π ( 1 ) = 1 (cf. (554) ) yield
V h π ( 1 ) ≥ 1 + ( 1 − 1 / H ) { 1 + ( 1 − 1 / H ) V h + 2 π ( 1 ) } ≥ ⋯ ≥ ∑ i = 0 H − h ( 1 − 1 / H ) i = 1 − ( 1 − 1 / H ) H − h + 1 1 / H ≥ 1 − e − ( H − h + 1 ) / H 1 / H ≥ 1 2 H − h + 1 H 1 / H = H − h + 1 2 , (558) \begin{aligned}
V_h^\pi(1)
&\ge1+(1-1/H)\left\{1+(1-1/H)V_{h+2}^\pi(1)\right\}\ge\cdots\\
&\ge\sum_{i=0}^{H-h}(1-1/H)^i
=\frac{1-(1-1/H)^{H-h+1}}{1/H}
\ge\frac{1-e^{-(H-h+1)/H}}{1/H}\\
&\ge\frac{\frac{1}{2}\frac{H-h+1}{H}}{1/H}
=\frac{H-h+1}{2}, \tag*{(558)}
\end{aligned} V h π ( 1 ) ≥ 1 + ( 1 − 1/ H ) { 1 + ( 1 − 1/ H ) V h + 2 π ( 1 ) } ≥ ⋯ ≥ i = 0 ∑ H − h ( 1 − 1/ H ) i = 1/ H 1 − ( 1 − 1/ H ) H − h + 1 ≥ 1/ H 1 − e − ( H − h + 1 ) / H ≥ 1/ H 2 1 H H − h + 1 = 2 H − h + 1 , (558)
where we have used the elementary inequalities 1 − x ≤ exp ( − x ) 1-x\le\exp(-x) 1 − x ≤ exp ( − x ) and 1 − exp ( − x ) ≥ x / 2 1-\exp(-x)\ge x/2 1 − exp ( − x ) ≥ x /2 for 0 ≤ x ≤ 1 0\le x\le1 0 ≤ x ≤ 1 . Additionally, it also follows directly from (556) that
V h ⋆ ( 1 ) = Q h ⋆ ( 1 , a h ⋆ ) = 1 + P h ( 1 ∣ 1 , a h ⋆ ) V h + 1 ⋆ ( 1 ) . (559) V_h^\star(1)=Q_h^\star(1,a_h^\star)=1+P_h(1\mid1,a_h^\star)V_{h+1}^\star(1). \tag*{(559)} V h ⋆ ( 1 ) = Q h ⋆ ( 1 , a h ⋆ ) = 1 + P h ( 1 ∣ 1 , a h ⋆ ) V h + 1 ⋆ ( 1 ) . (559)
Lower bounding the suboptimality gap. Now, we analyze the suboptimality gap associated with policy π \pi π . Towards this, we first use (556) and (559) to demonstrate that
V h ∗ ( 1 ) − V h π ( 1 ) = 1 + P h ( 1 ∣ 1 , a h ∗ ) V h + 1 ∗ ( 1 ) − { 1 + E a h ∼ π h ( ⋅ ∣ 1 ) [ P h ( 1 ∣ 1 , a h ) V h + 1 π ( 1 ) ] } = E a h ∼ π h ( ⋅ ∣ 1 ) [ P h ( 1 ∣ 1 , a h ∗ ) V h + 1 ∗ ( 1 ) − P h ( 1 ∣ 1 , a h ) V h + 1 π ( 1 ) ] = E a h ∼ π h ( ⋅ ∣ 1 ) [ P h ( 1 ∣ 1 , a h ∗ ) ( V h + 1 ∗ ( 1 ) − V h + 1 π ( 1 ) ) ] + E a h ∼ π h ( ⋅ ∣ 1 ) [ ( P h ( 1 ∣ 1 , a h ∗ ) − P h ( 1 ∣ 1 , a h ) ) V h + 1 π ( 1 ) ] ≥ ( 1 − 1 H ) ( V h + 1 ∗ ( 1 ) − V h + 1 π ( 1 ) ) + ε 0 2 H 2 V h + 1 π ( 1 ) E a h ∼ π h ( ⋅ ∣ 1 ) [ 1 ( a h ≠ a h ∗ ) ] ≥ ( 1 − 1 H ) ( V h + 1 ∗ ( 1 ) − V h + 1 π ( 1 ) ) + ε 0 ( H − h ) 4 H 2 ( 1 − π h ( a h ∗ ∣ 1 ) ) , (560) \begin{aligned}
V_h^*(1)-V_h^\pi(1)
&=1+P_h(1\mid1,a_h^*)V_{h+1}^*(1)-\left\{1+\mathbb{E}_{a_h\sim\pi_h(\cdot\mid1)}\left[P_h(1\mid1,a_h)V_{h+1}^\pi(1)\right]\right\}\\
&=\mathbb{E}_{a_h\sim\pi_h(\cdot\mid1)}\left[P_h(1\mid1,a_h^*)V_{h+1}^*(1)-P_h(1\mid1,a_h)V_{h+1}^\pi(1)\right]\\
&=\mathbb{E}_{a_h\sim\pi_h(\cdot\mid1)}\left[P_h(1\mid1,a_h^*)\left(V_{h+1}^*(1)-V_{h+1}^\pi(1)\right)\right]\\
&\quad+\mathbb{E}_{a_h\sim\pi_h(\cdot\mid1)}\left[\left(P_h(1\mid1,a_h^*)-P_h(1\mid1,a_h)\right)V_{h+1}^\pi(1)\right]\\
&\geq\left(1-\frac{1}{H}\right)\left(V_{h+1}^*(1)-V_{h+1}^\pi(1)\right)+\frac{\varepsilon_0}{2H^2}V_{h+1}^\pi(1)\mathbb{E}_{a_h\sim\pi_h(\cdot\mid1)}\left[\mathbb{1}(a_h\ne a_h^*)\right]\\
&\geq\left(1-\frac{1}{H}\right)\left(V_{h+1}^*(1)-V_{h+1}^\pi(1)\right)+\frac{\varepsilon_0(H-h)}{4H^2}\left(1-\pi_h(a_h^*\mid1)\right), \tag*{(560)}
\end{aligned} V h ∗ ( 1 ) − V h π ( 1 ) = 1 + P h ( 1 ∣ 1 , a h ∗ ) V h + 1 ∗ ( 1 ) − { 1 + E a h ∼ π h ( ⋅ ∣ 1 ) [ P h ( 1 ∣ 1 , a h ) V h + 1 π ( 1 ) ] } = E a h ∼ π h ( ⋅ ∣ 1 ) [ P h ( 1 ∣ 1 , a h ∗ ) V h + 1 ∗ ( 1 ) − P h ( 1 ∣ 1 , a h ) V h + 1 π ( 1 ) ] = E a h ∼ π h ( ⋅ ∣ 1 ) [ P h ( 1 ∣ 1 , a h ∗ ) ( V h + 1 ∗ ( 1 ) − V h + 1 π ( 1 ) ) ] + E a h ∼ π h ( ⋅ ∣ 1 ) [ ( P h ( 1 ∣ 1 , a h ∗ ) − P h ( 1 ∣ 1 , a h ) ) V h + 1 π ( 1 ) ] ≥ ( 1 − H 1 ) ( V h + 1 ∗ ( 1 ) − V h + 1 π ( 1 ) ) + 2 H 2 ε 0 V h + 1 π ( 1 ) E a h ∼ π h ( ⋅ ∣ 1 ) [ 1 ( a h = a h ∗ ) ] ≥ ( 1 − H 1 ) ( V h + 1 ∗ ( 1 ) − V h + 1 π ( 1 ) ) + 4 H 2 ε 0 ( H − h ) ( 1 − π h ( a h ∗ ∣ 1 ) ) , (560)
where the last relation follows from (558) , and the penultimate inequality holds true since, according to (5.92) and (5.94),
P h ( 1 ∣ 1 , a h ∗ ) ≥ 1 − 1 / H , P_h(1\mid1,a_h^*)\geq1-1/H, P h ( 1 ∣ 1 , a h ∗ ) ≥ 1 − 1/ H ,
P h ( 1 ∣ 1 , a h ∗ ) − P h ( 1 ∣ 1 , a h ) ≥ ε 0 2 H 2 1 ( a h ≠ a h ∗ ) . P_h(1\mid1,a_h^*)-P_h(1\mid1,a_h)\geq\frac{\varepsilon_0}{2H^2}\mathbb{1}(a_h\ne a_h^*). P h ( 1 ∣ 1 , a h ∗ ) − P h ( 1 ∣ 1 , a h ) ≥ 2 H 2 ε 0 1 ( a h = a h ∗ ) .
Applying the above property (560) recursively, we arrive at
V 1 ∗ ( 1 ) − V 1 π ( 1 ) ≥ ( 1 − 1 H ) H ( V H + 1 ∗ ( 1 ) − V H + 1 π ( 1 ) ) + ∑ h = 1 H ( 1 − 1 H ) h ε 0 ( H − h ) 4 H 2 ( 1 − π h ( a h ∗ ∣ 1 ) ) = ∑ h = 1 H ( 1 − 1 H ) h ε 0 ( H − h ) 4 H 2 ( 1 − π h ( a h ∗ ∣ 1 ) ) ≥ ∑ h = 1 H ε 0 ( H − h ) 16 H 2 ( 1 − π h ( a h ∗ ∣ 1 ) ) . \begin{aligned}
V_1^*(1)-V_1^\pi(1)
&\geq\left(1-\frac{1}{H}\right)^H\left(V_{H+1}^*(1)-V_{H+1}^\pi(1)\right)+\sum_{h=1}^{H}\left(1-\frac{1}{H}\right)^h\frac{\varepsilon_0(H-h)}{4H^2}\left(1-\pi_h(a_h^*\mid1)\right)\\
&=\sum_{h=1}^{H}\left(1-\frac{1}{H}\right)^h\frac{\varepsilon_0(H-h)}{4H^2}\left(1-\pi_h(a_h^*\mid1)\right)\\
&\geq\sum_{h=1}^{H}\frac{\varepsilon_0(H-h)}{16H^2}\left(1-\pi_h(a_h^*\mid1)\right).
\end{aligned} V 1 ∗ ( 1 ) − V 1 π ( 1 ) ≥ ( 1 − H 1 ) H ( V H + 1 ∗ ( 1 ) − V H + 1 π ( 1 ) ) + h = 1 ∑ H ( 1 − H 1 ) h 4 H 2 ε 0 ( H − h ) ( 1 − π h ( a h ∗ ∣ 1 ) ) = h = 1 ∑ H ( 1 − H 1 ) h 4 H 2 ε 0 ( H − h ) ( 1 − π h ( a h ∗ ∣ 1 ) ) ≥ h = 1 ∑ H 16 H 2 ε 0 ( H − h ) ( 1 − π h ( a h ∗ ∣ 1 ) ) .
where the last inequality holds since ( 1 − 1 / H ) h ≥ ( 1 − 1 / H ) H ≥ 1 / 4 \left(1-1/H\right)^h\geq\left(1-1/H\right)^H\geq1/4 ( 1 − 1/ H ) h ≥ ( 1 − 1/ H ) H ≥ 1/4 if H ≥ 2 H\geq2 H ≥ 2 . Consequently, this lower bound hinges upon how often policy π \pi π selects suboptimal actions at each step.
Proof of Lemma 5.21
For notational convenience, let a t a_t a t denote the arm selected at round t t t and let R t = R a t , t R_t=R_{a_t,t} R t = R a t , t denote the corresponding observed reward. Define
x t : = { a 1 , R 1 , … , a t , R t } x_t:=\{a_1,R_1,\ldots,a_t,R_t\} x t := { a 1 , R 1 , … , a t , R t }
to be the trajectory observed up to the end of round t t t , and let P B i ( t ) \mathbb{P}_{\mathcal{B}_i}^{(t)} P B i ( t ) denote the distribution of x t x_t x t under the bandit instance B i \mathcal{B}_i B i . We also denote the conditional reward distribution under B i \mathcal{B}_i B i by
P i ( R ∣ a ) : = { 1 − q i ( a ) , if R = 1 , q i ( a ) , if R = 0. (561) P_i(R\mid a):=
\begin{cases}
1-q_i(a), & \text{if } R=1,\\
q_i(a), & \text{if } R=0.
\end{cases} \tag*{(561)} P i ( R ∣ a ) := { 1 − q i ( a ) , q i ( a ) , if R = 1 , if R = 0. (561)
Consider any online algorithm π ^ \hat{\pi} π ^ that generates a sequence of policies { π t } t ≥ 1 \{\pi^t\}_{t\ge1} { π t } t ≥ 1 , where the action at round t t t depends only on the past trajectory x t − 1 x_{t-1} x t − 1 . With a slight abuse of notation, we write
π t ( ⋅ ∣ x t − 1 ) \pi^t(\cdot\mid x_{t-1}) π t ( ⋅ ∣ x t − 1 )
for the action-selection distribution at round t t t . Here, π t ( ⋅ ∣ ⋅ ) \pi^t(\cdot\mid\cdot) π t ( ⋅ ∣ ⋅ ) denotes the underlying algorithmic rule, while the realized action-selection distribution at round t t t relies on the observed history x t − 1 x_{t-1} x t − 1 . Under the bandit instance B i B_i B i , the distribution of the trajectory x t x_t x t admits the following factorization:
P B i ( t ) ( x t ) = ∏ τ = 1 t π τ ( a τ ∣ x τ − 1 ) P i ( R τ ∣ a τ ) . \mathbb{P}_{B_i}^{(t)}(x_t)=\prod_{\tau=1}^{t}\pi^\tau(a_\tau\mid x_{\tau-1})P_i(R_\tau\mid a_\tau). P B i ( t ) ( x t ) = τ = 1 ∏ t π τ ( a τ ∣ x τ − 1 ) P i ( R τ ∣ a τ ) .
where P i ( ⋅ ∣ a t ) P_i(\cdot\mid a_t) P i ( ⋅ ∣ a t ) is defined in (561) . This implies that
P B 1 ( t ) ( x t ) P B i ( t ) ( x t ) = ∏ τ = 1 t P 1 ( R τ ∣ a τ ) P i ( R τ ∣ a τ ) , \frac{\mathbb{P}_{B_1}^{(t)}(x_t)}{\mathbb{P}_{B_i}^{(t)}(x_t)}
=\prod_{\tau=1}^{t}\frac{P_1(R_\tau\mid a_\tau)}{P_i(R_\tau\mid a_\tau)}, P B i ( t ) ( x t ) P B 1 ( t ) ( x t ) = τ = 1 ∏ t P i ( R τ ∣ a τ ) P 1 ( R τ ∣ a τ ) ,
meaning that the likelihood ratio decomposes into a product of per-round reward likelihood ratios. Consequently, the chain rule for KL divergence gives
KL ( P B 1 ( t ) ∥ P B i ( t ) ) = ∑ τ = 1 t E B 1 [ KL ( P 1 ( ⋅ ∣ a τ ) ∥ P i ( ⋅ ∣ a τ ) ) ] , (562) \operatorname{KL}\left(\mathbb{P}_{B_1}^{(t)} \,\middle\|\, \mathbb{P}_{B_i}^{(t)}\right)
=\sum_{\tau=1}^{t}\mathbb{E}_{B_1}\left[\operatorname{KL}\left(P_1(\cdot\mid a_\tau) \,\middle\|\, P_i(\cdot\mid a_\tau)\right)\right], \tag*{(562)} KL ( P B 1 ( t ) P B i ( t ) ) = τ = 1 ∑ t E B 1 [ KL ( P 1 ( ⋅ ∣ a τ ) ∥ P i ( ⋅ ∣ a τ ) ) ] , (562)
where the expectation is taken with respect to the randomness of a τ a_\tau a τ under B 1 B_1 B 1 .
Next, we bound each KL divergence term appearing in the summation on the right-hand side of (562) . Suppose that ε 0 ≤ H / 2 \varepsilon_0 \le H/2 ε 0 ≤ H /2 and H ≥ 2 H \ge2 H ≥ 2 . Recalling that the conditional reward distributions (561) are Bernoulli and letting KL ( p ∥ q ) \operatorname{KL}(p \parallel q) KL ( p ∥ q ) denote the KL divergence between Bernoulli distributions with means p p p and q q q , respectively, we can invoke Lemma 2.13 to show the following:
KL ( P 1 ( ⋅ ∣ a ) ∥ P i ( ⋅ ∣ a ) ) = KL ( q 1 ( a ) ∥ q i ( a ) ) ≤ ( q i ( a ) − q 1 ( a ) ) 2 q i ( a ) ( 1 − q i ( a ) ) ≤ ( ε 0 / H 2 ) 2 1 / ( 4 H ) = 4 ε 0 2 H 3 . \operatorname{KL}\left(P_1(\cdot\mid a) \,\middle\|\, P_i(\cdot\mid a)\right)
=\operatorname{KL}\left(q_1(a) \,\middle\|\, q_i(a)\right)
\le\frac{\left(q_i(a)-q_1(a)\right)^2}{q_i(a)\left(1-q_i(a)\right)}
\le\frac{(\varepsilon_0/H^2)^2}{1/(4H)}
=\frac{4\varepsilon_0^2}{H^3}. KL ( P 1 ( ⋅ ∣ a ) ∥ P i ( ⋅ ∣ a ) ) = KL ( q 1 ( a ) ∥ q i ( a ) ) ≤ q i ( a ) ( 1 − q i ( a ) ) ( q i ( a ) − q 1 ( a ) ) 2 ≤ 1/ ( 4 H ) ( ε 0 / H 2 ) 2 = H 3 4 ε 0 2 .
Here, the last line is a consequence of the properties (see (5.94))
∣ q i ( a ) − q 1 ( a ) ∣ ≤ ε 0 H 2 , q i ( a ) ≥ 1 H − ε 0 H 2 ≥ 1 2 H , 1 − q i ( a ) ≥ 1 − 1 H ≥ 1 2 . \left|q_i(a)-q_1(a)\right|\le\frac{\varepsilon_0}{H^2},\qquad q_i(a)\ge\frac{1}{H}-\frac{\varepsilon_0}{H^2}\ge\frac{1}{2H},\qquad1-q_i(a)\ge1-\frac{1}{H}\ge\frac{1}{2}. ∣ q i ( a ) − q 1 ( a ) ∣ ≤ H 2 ε 0 , q i ( a ) ≥ H 1 − H 2 ε 0 ≥ 2 H 1 , 1 − q i ( a ) ≥ 1 − H 1 ≥ 2 1 .
KL ( P 1 ( ⋅ ∣ a ) ∥ P i ( ⋅ ∣ a ) ) = 0. \operatorname{KL}\left(P_1(\cdot\mid a) \,\middle\|\, P_i(\cdot\mid a)\right)=0. KL ( P 1 ( ⋅ ∣ a ) ∥ P i ( ⋅ ∣ a ) ) = 0.
These two properties about the KL divergence taken together yield
∑ i = 1 A KL ( P 1 ( ⋅ ∣ a ) ∥ P i ( ⋅ ∣ a ) ) = KL ( P 1 ( ⋅ ∣ a ) ∥ P a ( ⋅ ∣ a ) ) ≤ 4 ε 0 2 H 3 , \sum_{i=1}^{A}\operatorname{KL}\left(P_1(\cdot\mid a) \,\middle\|\, P_i(\cdot\mid a)\right)
=\operatorname{KL}\left(P_1(\cdot\mid a) \,\middle\|\, P_a(\cdot\mid a)\right)
\le\frac{4\varepsilon_0^2}{H^3}, i = 1 ∑ A KL ( P 1 ( ⋅ ∣ a ) ∥ P i ( ⋅ ∣ a ) ) = KL ( P 1 ( ⋅ ∣ a ) ∥ P a ( ⋅ ∣ a ) ) ≤ H 3 4 ε 0 2 ,
which combined with (562) leads to
1 A ∑ i = 1 A KL ( P B 1 ( t ) ∥ P B i ( t ) ) = 1 A ∑ τ = 1 t E B 1 [ ∑ i = 1 A KL ( P 1 ( ⋅ ∣ a τ ) ∥ P i ( ⋅ ∣ a τ ) ) ] ≤ ∑ τ = 1 t 4 ε 0 2 A H 3 = 4 ε 0 2 t A H 3 . (563) \frac{1}{A}\sum_{i=1}^{A}\operatorname{KL}\left(\mathbb{P}_{B_1}^{(t)} \,\middle\|\, \mathbb{P}_{B_i}^{(t)}\right)
=\frac{1}{A}\sum_{\tau=1}^{t}\mathbb{E}_{B_1}\left[\sum_{i=1}^{A}\operatorname{KL}\left(P_1(\cdot\mid a_\tau) \,\middle\|\, P_i(\cdot\mid a_\tau)\right)\right]
\le\sum_{\tau=1}^{t}\frac{4\varepsilon_0^2}{AH^3}
=\frac{4\varepsilon_0^2t}{AH^3}. \tag*{(563)} A 1 i = 1 ∑ A KL ( P B 1 ( t ) P B i ( t ) ) = A 1 τ = 1 ∑ t E B 1 [ i = 1 ∑ A KL ( P 1 ( ⋅ ∣ a τ ) ∥ P i ( ⋅ ∣ a τ ) ) ] ≤ τ = 1 ∑ t A H 3 4 ε 0 2 = A H 3 4 ε 0 2 t . (563)
With (563) in place, applying Lemma 2.11 then reveals that
inf π ^ 1 A ∑ i = 1 A P B i ( a t + 1 ≠ i ) ≥ 1 − 1 A − A − 1 2 A 2 ∑ i = 2 A KL ( P B 1 ( t ) ∥ P B i ( t ) ) \inf_{\hat{\pi}}\frac{1}{A}\sum_{i=1}^{A}\mathbb{P}_{\mathcal{B}_i}(a_{t+1}\ne i) \ge1-\frac{1}{A}-\sqrt{\frac{A-1}{2A^2}\sum_{i=2}^{A}\operatorname{KL}\left(\mathbb{P}_{\mathcal{B}_1}^{(t)} \,\middle\|\, \mathbb{P}_{\mathcal{B}_i}^{(t)}\right)} π ^ inf A 1 i = 1 ∑ A P B i ( a t + 1 = i ) ≥ 1 − A 1 − 2 A 2 A − 1 i = 2 ∑ A KL ( P B 1 ( t ) P B i ( t ) )
≥ 1 − 1 A − 2 ( A − 1 ) ε 0 2 t A 2 H 3 \ge1-\frac{1}{A}-\sqrt{\frac{2(A-1)\varepsilon_0^2t}{A^2H^3}} ≥ 1 − A 1 − A 2 H 3 2 ( A − 1 ) ε 0 2 t
≥ 1 − 1 A − A − 1 9 A ≥ 1 3 , \ge1-\frac{1}{A}-\sqrt{\frac{A-1}{9A}}\ge\frac{1}{3}, ≥ 1 − A 1 − 9 A A − 1 ≥ 3 1 ,
provided that t ≤ A H 3 18 ε 0 2 t\le\frac{AH^3}{18\varepsilon_0^2} t ≤ 18 ε 0 2 A H 3 and A ≥ 3 A\ge3 A ≥ 3 . This concludes the proof.
Notes
Multi-armed bandit problems have deep roots in the statistics literature. First formulated by Thompson (1933), the bandit framework, along with the notion of regret, was later popularized by Herbert Robbins (e.g., the seminal work of Robbins (1952)). The use of upper confidence bounds in bandit problems was initially suggested by Lai and Robbins (1985) and Lai (1987). Subsequent research established asymptotic guarantees for UCB-style algorithms (e.g., Lai (1987), Katehakis and Robbins (1995), Agrawal (1995), and Burnetas and Katehakis (1996)), while non-asymptotic regret analyses were later developed by, among others, Auer et al. (2002), Bubeck and Cesa-Bianchi (2012), Russo and Van Roy (2013), and Cappé et al. (2013). By adaptively tuning the exploration bonuses, one can further obtain anytime guarantees for UCB-style algorithms (i.e., without requiring prior knowledge of the time horizon) and remove the extra logarithmic factor in the regret bound (Auer et al., 2002; Audibert and Bubeck, 2009; Degenne and Perchet, 2016). We refer the interested reader to Lattimore and Szepesvári (2020) for a comprehensive discussion of the historical development of bandit algorithms and related bibliographical remarks.
The optimism principle has subsequently become a central design paradigm in online RL. Building on ideas that originated in the bandit literature, a large body of work has developed optimistic algorithms for sequential learning and decision-making in unknown environments (e.g., Auer and Ortner (2006), Jaksch et al. (2010), Bartlett and Tewari (2009), Filippi et al. (2010), Dann and Brunskill (2015), Dann et al. (2017), Azar et al. (2017), Jin et al. (2018), Dann et al. (2019), Zanette and Brunskill (2019), and Bourel et al. (2020)). One of the earliest contributions in this direction is the work of Jaksch et al. (2010), which proposed the UCRL2 algorithm for weakly communicating MDPs. The key idea is to construct confidence regions for the transition kernel and use them to identify policies that are optimistic w.r.t. the set of plausible MDPs. This UCRL2 algorithm builds upon earlier optimistic methods, such as the UCRL algorithm proposed by Auer and Ortner (2006). When applied to episodic finite-horizon MDPs, however, the regret guarantees obtained via the UCRL2 analysis are not minimax optimal, exhibiting a gap of at least a factor of H 2 S \sqrt{H^2S} H 2 S relative to the minimax limit.
In comparison to UCRL2 and its variants, a more sample-efficient paradigm is to construct upper confidence bounds for the optimal Q Q Q -functions and value functions and use these confidence estimates to guide exploration. This paradigm has led to a sequence of sample-efficient algorithms for online RL, including both model-based methods (e.g., Azar et al. (2017), Zhang et al. (2021e), Ménard et al. (2021), and Zhang et al. (2025e)) and model-free methods (e.g., Jin et al. (2018), Bai et al. (2019), Yang et al. (2021), and Zhang et al. (2025b). When combined with confidence bounds derived from empirical Bernstein inequalities, which exploit variance information, model-based algorithms based on this paradigm can attain asymptotically optimal regret guarantees. For model-free RL, however, variance-aware bonuses alone are insufficient to achieve asymptotically optimal regret, which motivates the development of more sophisticated variance-reduction techniques (Zhang et al., 2020f; Li et al., 2021b; Li et al., 2023a; Zhang et al., 2026a) (originally referred to as reference-advantage decomposition by Zhang et al. (2020f)). Determining how to optimally leverage optimism to achieve minimax-optimal regret without incurring high burn-in cost remained an open problem until the work Zhang et al. (2024c) and Zhang et al. (2025e), which established the minimax optimality of model-based online RL for any sample size. In contrast, the most sample-efficient model-free algorithm currently known, due to Li et al. (2021b), still incurs a burn-in sample size on the order of S A poly ( H ) SA\operatorname{poly}(H) S A poly ( H ) .
Beyond the setting studied in this chapter, a variety of related problem formulations have been investigated, motivated by different practical considerations. One prominent example is reward-free (or reward-agnostic) exploration, where the goal is to collect data during an exploration phase without knowledge of the reward function, while still enabling near-optimal performance once the reward function is subsequently specified (e.g., Jin et al. (2020a), Zanette et al. (2020b), Zhang et al. (2020d), Zhang et al. (2020e), Wagenmaker et al. (2022), Li et al. (2024e), Wang et al. (2020b), and Zhang et al. (2021d)). Another direction is the study of “horizon-free RL,” which studies finite-horizon stationary MDPs under the assumption that the total reward accumulated within each episode is bounded by 1; the objective is to design algorithms whose regret bounds almost do not scale with the horizon length H H H , thereby mitigating the adverse effect of long horizon (Jiang and Agarwal, 2018; Zanette and Brunskill, 2019; Zhang et al., 2021e; Li et al., 2022c; Zhang et al., 2024d; Wang et al., 2025b)). There has also been a line of work that focuses on reducing the switching cost, namely the number of times the learner changes its sampling policy throughout the learning process, motivated by practical applications in which policy deployment is costly (Bai et al., 2019; Qiao and Wang, 2022; Qiao et al., 2022; Huang et al., 2022; Zhao et al., 2024; Zhang et al., 2026b)). Additionally, low-regret algorithms for constrained MDPs have been developed in, e.g., Efroni et al. (2020), Brantley et al. (2020), Ding et al. (2021), Liu et al. (2021), Bura et al. (2022), Wei et al. (2022), and Wei et al. (2023). Furthermore, the efficient use of the optimism principle under low-complexity function approximation — particularly linear function approximation — has itself become a major research direction; we refer the reader to, e.g., Yang and Wang (2020), Jin et al. (2020b), Zanette et al. (2020a), Du et al. (2020), Yang et al. (2020b), Modi et al. (2020), Ayoub et al. (2020), Wang et al. (2020a), Zanette 2021, Wang et al. (2021b), Weisz et al. (2021), Wang et al. (2021c), Li et al. (2022b), Zhou et al. (2021a), Jin et al. (2021a), Du et al. (2021), Yin et al. (2022a), and He et al. (2023) and the references therein for details.
Beyond confidence-interval-based methods, several alternative approaches to incentivizing exploration have also been extensively studied in both theory and practice. One notable approach is posterior sampling (Osband et al., 2013; Osband and Van Roy, 2017), with representative examples including Thompson sampling (Gopalan and Mannor, 2015; Agrawal and Jia, 2017; Ouyang et al., 2017; Agrawal et al., 2026), feel-Good Thompson sampling (Zhang, 2022), and learning rate randomization (Tiapkin et al., 2023; Wang et al., 2026). Another line of work promotes exploration by regularizing the maximum likelihood estimate of the model or value functions toward more optimistic versions, such as Hung et al. (2021), Mete et al. (2021), Yang et al. (2025a), and Liu et al. (2023b). Randomized exploration, when suitably designed, has also been shown to achieve efficient exploration (Xiong et al., 2022). Collectively, these developments showcase a diverse array of algorithmic ideas available for balancing exploration and exploitation.
Offline RL
Despite the promise of online RL, the limited capability of online data collection in various real-world applications presents a fundamental bottleneck to carrying the success of RL over to broader scenarios. For example, in applications such as clinical trials, autonomous systems, and online advertising, real-time data acquisition is expensive, high-stakes, and/or time-consuming. To circumvent this bottleneck, a plausible idea is to make more effective use of previously collected data, since historical datasets may contain information that transfers to new tasks. For instance, the state-transition dynamics encountered in past tasks may sometimes partially resemble those arising in new tasks. The potential of this data-driven paradigm has been explored and recognized in a diverse array of contexts, including but not limited to robotic manipulation (Ebert et al. (2018)), autonomous driving (Diehl et al. (2021)), and healthcare (Tang and Wiens (2021)); see Levine et al. (2020) and Prudencio et al. (2022) for recent surveys. The subfield of RL based solely on historical data, without further interaction with the environment, is now commonly referred to as offline RL or batch RL (Lange et al. (2012); Levine et al. (2020)).
Problem formulation and challenges
To facilitate a precise technical exposition, we begin by introducing the settings and objectives that will serve as the primary focus of this chapter. We then discuss the key challenges that arise in these settings, together with some important problem parameters introduced in response to these challenges.
Data generation mechanism. The effectiveness and potential of offline RL hinge critically upon the quality of the historical dataset, denoted throughout by D \mathcal{D} D . For concreteness, this chapter focuses primarily on an independent sampling model tailored to discounted infinite-horizon MDPs to describe the sample-generating process underlying D \mathcal{D} D .
To be precise, suppose the learner has access to a batch dataset D = { ( s i , a i , s i ′ ) } 1 ≤ i ≤ N \mathcal{D} = \{(s_i, a_i, s'_i)\}_{1\leq i\leq N} D = {( s i , a i , s i ′ ) } 1 ≤ i ≤ N comprising N N N sample transitions. These samples are generated according to an unknown distribution $d^b \in Δ ( S × A ) \Delta(\mathcal{S} \times\mathcal{A}) Δ ( S × A ) and the true transition kernel P P P of the underlying MDP, namely,
( s i , a i ) ∼ i n d . d b and s i ′ ∼ i n d . P ( ⋅ ∣ s i , a i ) , 1 ≤ i ≤ N . (564) (s_i, a_i) \mathrel{\mathop{\sim}^{\mathrm{ind.}}} d^{\mathrm{b}} \quad\text{and} \quad s_i' \mathrel{\mathop{\sim}^{\mathrm{ind.}}} P(\cdot\mid s_i, a_i), \qquad1 \le i \le N. \tag*{(564)} ( s i , a i ) ∼ ind. d b and s i ′ ∼ ind. P ( ⋅ ∣ s i , a i ) , 1 ≤ i ≤ N . (564)
This coincides with the sampling model proposed in the seminal work Rashidinejad et al. (2021). In practical applications, such samples may have been generated by executing one or more “behavior policies” in the MDP, which motivates the notation b in the distribution d b d^{\mathrm{b}} d b . As in previous chapters, we assume for simplicity that the exact reward functions are available to the learner; handling unknown rewards introduces only moderate additional technical challenges.
Remark 6.1. The independent sampling model can often be extended to more realistic data collection mechanisms, such as the setting in which the learner observes a single Markovian trajectory; see, e.g., Yan et al. (2023). We focus on independent sampling here to streamline the exposition.
Goal. With the batch dataset D \mathcal{D} D in hand, the goal of offline RL in this setting is to learn, in a data-efficient manner, a policy π ^ \widehat{\pi} π that attains near-optimal value with respect to a prescribed test state distribution ρ ∈ Δ ( S ) \rho\in\Delta(\mathcal{S}) ρ ∈ Δ ( S ) (which is not known to the learner a priori ). More precisely, for a prescribed precision level ε \varepsilon ε , we seek to identify an ε \varepsilon ε -optimal policy π ^ \widehat{\pi} π satisfying
V ∗ ( ρ ) − V π ^ ( ρ ) ≤ ε (565) V^*(\rho) - V^{\widehat{\pi}}(\rho) \le\varepsilon \tag*{(565)} V ∗ ( ρ ) − V π ( ρ ) ≤ ε (565)
with high probability. A desirable offline RL algorithm would achieve this target accuracy using as few samples from D \mathcal{D} D as possible. As in Chapters 3 and 5, particular emphasis is placed on attaining minimal sample complexity uniformly across the full range of accuracy levels, namely for all ε ∈ ( 0 , 1 1 − γ ] \varepsilon\in\left(0, \frac{1}{1-\gamma}\right] ε ∈ ( 0 , 1 − γ 1 ] .
Challenges. Unlike online RL, offline RL precludes active interaction with the environment to collect additional data. Consequently, the learner must rely entirely on the static dataset D \mathcal{D} D . The statistical efficiency of offline RL is therefore fundamentally dictated by how informative this historical dataset is for learning a near-optimal policy. Below, we highlight two fundamental challenges arising from this restricted data generation mechanism, both of which must be carefully addressed in order to achieve statistically efficient offline RL.
Distribution shift. For the most part, the historical dataset is generated by one or more behavior policies that differ from the target optimal policy. As a result, the state-action distribution induced by the target policy may deviate substantially from the sampling distribution d b d^{\mathrm{b}} d b underlying the dataset D \mathcal{D} D . A central challenge in offline RL is therefore how to effectively leverage past data despite this mismatch between the data distribution and the one induced by the optimal policy.
Limited data coverage. Ideally, one would like the dataset D \mathcal{D} D to contain sufficiently many samples for every state-action pair, so that the performance of all candidate policies can be accurately evaluated. In practice, however, such uniform coverage is often unrealistic, since the learner has no control over the past data collection process. Fortunately, uniform coverage may be unnecessary, given that the ultimate objective is only to identify a single near-optimal policy rather than accurately evaluate every policy.
Key parameters: single-policy concentrability. To quantitatively capture the distribution shift between the target occupancy measure and the underlying data distribution, we introduce a key quantity previously introduced in Rashidinejad et al. (2021).
Definition 6.1 (Single-policy concentrability for infinite-horizon MDPs). For any given initial distribution ρ ∈ Δ ( S ) \rho\in\Delta(\mathcal{S}) ρ ∈ Δ ( S ) , the single-policy concentrability coefficient underlying the batch dataset D \mathcal{D} D (see (564) ) is defined as
C ρ ⋆ : = max ( s , a ) ∈ S × A d ⋆ ( s , a ; ρ ) d b ( s , a ) , (566) C_{\rho}^{\star} := \max_{(s,a)\in\mathcal{S}\times\mathcal{A}} \frac{d^{\star}(s,a;\rho)}{d^{b}(s,a)}, \tag*{(566)} C ρ ⋆ := ( s , a ) ∈ S × A max d b ( s , a ) d ⋆ ( s , a ; ρ ) , (566) where d ⋆ d^{\star} d ⋆ represents the discounted occupancy distribution induced by the optimal deterministic policy π ⋆ \pi^{\star} π ⋆ (see (16) ), and the ratio 0 / 0 0/0 0/0 is interpreted as 0.
Remark 6.2. It is immediate to see that C ρ ⋆ ≥ 1 C_{\rho}^{\star} \ge1 C ρ ⋆ ≥ 1 , since
C ρ ⋆ = max ( s , a ) ∈ S × A d ⋆ ( s , a ; ρ ) d b ( s , a ) ≥ ∑ s , a d ⋆ ( s , a ; ρ ) ∑ s , a d b ( s , a ) = 1 1 = 1. C_{\rho}^{\star} = \max_{(s,a)\in\mathcal{S}\times\mathcal{A}} \frac{d^{\star}(s,a;\rho)}{d^{b}(s,a)} \ge\frac{\sum_{s,a}d^{\star}(s,a;\rho)}{\sum_{s,a}d^{b}(s,a)} = \frac{1}{1} = 1. C ρ ⋆ = ( s , a ) ∈ S × A max d b ( s , a ) d ⋆ ( s , a ; ρ ) ≥ ∑ s , a d b ( s , a ) ∑ s , a d ⋆ ( s , a ; ρ ) = 1 1 = 1.
In words, C ρ ⋆ C_{\rho}^{\star} C ρ ⋆ quantifies the distribution mismatch between the target occupancy distribution d ⋆ d^{\star} d ⋆ (induced by the optimal policy starting from the initial distribution ρ \rho ρ ) and the underlying data distribution d b d^{b} d b through the maximum density ratio. Since the value function in (8) is defined in a discounted manner, the occupancy distribution d ⋆ d^{\star} d ⋆ is likewise chosen to be discounted. When C ρ ⋆ C_{\rho}^{\star} C ρ ⋆ is close to 1, the batch dataset D \mathcal{D} D may be viewed as containing near-expert data, meaning that its induced distribution closely aligns with that induced by the optimal policy and therefore reflects, to some extent, how an expert makes decisions. In this favorable scenario, the offline RL problem becomes intimately connected to the problem of “imitation learning,” where it may suffice to simply imitate the expert’s behavior.
Moreover, C ρ ⋆ C_{\rho}^{\star} C ρ ⋆ is referred to as a “single-policy” concentrability coefficient because it concerns only the optimal policy π ⋆ \pi^{\star} π ⋆ . This assumption is substantially weaker than a uniform coverage requirement, or more precisely, the all-policy concentrability assumption adopted in earlier work, e.g., Munos (2007), Farahmand et al. (2010), Fan et al. (2019), and Chen and Jiang (2019). Note that the latter requires a uniform density-ratio bound over all policies and therefore demands the dataset to be highly exploratory, or equivalently, to provide adequate coverage of all state-action pairs.
Given that C ρ ⋆ C_{\rho}^{\star} C ρ ⋆ is introduced to capture the challenges arising from both distribution mismatch and limited data coverage, we will treat it as a key problem parameter throughout the remainder of this chapter, and develop both sample complexity upper bounds and minimax lower bounds primarily in terms of C ρ ⋆ C_{\rho}^{\star} C ρ ⋆ .
Remark 6.3. A refined version of C ρ ⋆ C_{\rho}^{\star} C ρ ⋆ , called the single-policy clipped concentrability coefficient (Li et al. (2024b)), can be obtained via clipping:
C ρ , c l i p p e d ⋆ : = max ( s , a ) ∈ S × A min { d ⋆ ( s , a ; ρ ) , 1 S } d b ( s , a ) . (567) C_{\rho,\mathrm{clipped}}^{\star} := \max_{(s,a)\in\mathcal{S}\times\mathcal{A}} \frac{\min\left\{d^{\star}(s,a;\rho),\frac{1}{S}\right\}}{d^{b}(s,a)}. \tag*{(567)} C ρ , clipped ⋆ := ( s , a ) ∈ S × A max d b ( s , a ) min { d ⋆ ( s , a ; ρ ) , S 1 } . (567)
Clearly, one has C ρ , c l i p p e d ⋆ ≤ C ρ ⋆ C_{\rho,\mathrm{clipped}}^{\star} \le C_{\rho}^{\star} C ρ , clipped ⋆ ≤ C ρ ⋆ . In addition, unlike C ρ ⋆ C_{\rho}^{\star} C ρ ⋆ , which always satisfies C ρ ⋆ ≥ 1 C_{\rho}^{\star} \ge1 C ρ ⋆ ≥ 1 , the clipped coefficient can be as low as 1 / S 1/S 1/ S , a property that can sometimes lead to improved statistical guarantees, especially when d b d^{b} d b is close to a uniform distribution, such as the generative model setting. As it turns out, many of the upper and lower bounds developed in this chapter continue to hold with C ρ ⋆ C_{\rho}^{\star} C ρ ⋆ replaced by C ρ , c l i p p e d ⋆ C_{\rho,\mathrm{clipped}}^{\star} C ρ , clipped ⋆ ; see Li et al. (2024b) for details.
A detour: the LCB algorithm for multi-armed bandits
Before turning to offline RL in full generality, it is instructive to first examine a much simpler setting that already captures two central challenges: distribution mismatch and limited data coverage. To build intuition, we begin with the classical multi-armed bandit problem to introduce the principle of pessimism in the face of uncertainty — a principle that will play a central role more broadly in offline RL.
Setting: offline stochastic bandits
The basic setup of the multi-armed bandit problem has been introduced in § 8.2 . We briefly recap the setting and describe the corresponding offline data-generation mechanism.
Bandits. Consider an A A A -armed stochastic bandit setting, where each arm a ∈ A = { 1 , … , A } a \in\mathcal{A} = \{1,\ldots,A\} a ∈ A = { 1 , … , A } is associated with a mean reward r a ∈ [ 0 , 1 ] r_a \in[0,1] r a ∈ [ 0 , 1 ] . The optimal mean reward and the corresponding optimal arm are denoted by r ⋆ r^\star r ⋆ and a ⋆ a^\star a ⋆ , respectively (see (5.6)).
Offline data. The offline dataset D \mathcal{D} D consists of N N N samples { ( a i , R i ) } i = 1 N \{(a_i,R_i)\}_{i=1}^{N} {( a i , R i ) } i = 1 N generated independently according to some joint distribution. More concretely, each action a i a_i a i is sampled as
a i ∼ i . i . d . d b , i = 1 , … , N (568) a_i \mathrel{\overset{\mathrm{i.i.d.}}{\sim}} d^b,\qquad i=1,\ldots,N \tag*{(568)} a i ∼ i.i.d. d b , i = 1 , … , N (568)
for some action distribution d b ∈ Δ ( A ) d^b \in\Delta(\mathcal{A}) d b ∈ Δ ( A ) , after which the corresponding (noisy) reward R i R_i R i is generated according to
R i = r a i + z i with z i ∼ i . i . d . N ( 0 , 1 ) . (569) R_i = r_{a_i} + z_i \qquad\text{with } z_i \mathrel{\overset{\mathrm{i.i.d.}}{\sim}} \mathcal{N}(0,1). \tag*{(569)} R i = r a i + z i with z i ∼ i.i.d. N ( 0 , 1 ) . (569)
Single-policy concentrability. Let d ⋆ d^\star d ⋆ denote the distribution induced by the optimal policy, i.e., always pulling the optimal arm a ⋆ a^\star a ⋆ ,
d ⋆ ( a ) : = { 1 , if a = a ⋆ , 0 , else. (570) d^\star(a) := \begin{cases} 1, & \text{if } a = a^\star, \\ 0, & \text{else.} \end{cases} \tag*{(570)} d ⋆ ( a ) := { 1 , 0 , if a = a ⋆ , else. (570)
Then the single-policy concentrability coefficient defined in (566) reduces to
C ⋆ : = max a ∈ A d ⋆ ( a ) d b ( a ) = 1 d b ( a ⋆ ) . (571) C^\star:= \max_{a \in\mathcal{A}} \frac{d^\star(a)}{d^b(a)} = \frac{1}{d^b(a^\star)}. \tag*{(571)} C ⋆ := a ∈ A max d b ( a ) d ⋆ ( a ) = d b ( a ⋆ ) 1 . (571)
Here, we drop the dependence on ρ \rho ρ in the notation C ρ ⋆ C^\star_\rho C ρ ⋆ , since the bandit setting contains only a single state.
Goal. Armed with the offline dataset D \mathcal{D} D , our goal is to learn an arm a ^ \widehat{a} a that achieves a suboptimality gap
r ( a ⋆ ) − r ( a ^ ) r(a^\star) - r(\widehat{a}) r ( a ⋆ ) − r ( a )
as small as possible. Note that this notion of suboptimality also plays a central role in online multi-armed bandits through the definition of regret; see (358) in § 8.2 .
Pessimism in the face of uncertainty
We now introduce some core ideas that enable statistically efficient offline learning in bandit problems. In contrast to online bandits, which rely heavily on the optimism principle (see § 8.2 ) to guide exploration, offline settings do not permit active exploration. As a result, optimism-based approaches are inherently inapplicable, necessitating a fundamentally different set of ideas.
A first attempt: the empirical best arm. A natural strategy for identifying the optimal arm is to select the one with the highest empirical mean reward. Denoting by N a N_a N a the number of times arm a a a is pulled in D \mathcal{D} D , the empirical reward estimate for each arm a ∈ A a \in\mathcal{A} a ∈ A is given by
r ˉ a : = { 1 N a ∑ t = 1 N R t 1 { a t = a } , if N a > 0 , 0 , otherwise . (572) \bar{r}_a :=
\begin{cases}
\frac{1}{N_a}\sum_{t=1}^{N} R_t \mathbb{1}\{a_t=a\}, & \text{if } N_a>0,\\
0, & \text{otherwise}.
\end{cases} \tag*{(572)} r ˉ a := { N a 1 ∑ t = 1 N R t 1 { a t = a } , 0 , if N a > 0 , otherwise . (572)
which coincides with the maximum likelihood estimator of the mean reward under the Gaussian noise assumption. The empirical best arm is then defined as
a e m p : = arg max a ∈ A r ˉ a . (573) a^{\mathrm{emp}} := \arg\max_{a\in\mathcal{A}} \bar{r}_a. \tag*{(573)} a emp := arg a ∈ A max r ˉ a . (573)
Although intuitive, this empirical best-arm estimator can perform poorly when certain arms are only infrequently sampled in the dataset, making their mean rewards difficult to estimate reliably. In fact, one can construct examples showing that even in the favorable regime where C ⋆ ≈ 1 C^\star\approx1 C ⋆ ≈ 1 , selecting the empirical best arm may still incur a non-vanishing suboptimality gap regardless of how large the sample size becomes. This issue arises from the high statistical uncertainty associated with poorly explored suboptimal arms, which can lead to over-estimation of their rewards with non-vanishing probability. One concrete example is given below.
Example 6.1. Suppose that A = { 1 , 2 } \mathcal{A}=\{1,2\} A = { 1 , 2 } , and that the corresponding mean rewards are given by r 1 = 1 r_1=1 r 1 = 1 and r 2 = 0 r_2=0 r 2 = 0 . Assume that a i = 1 a_i=1 a i = 1 for all i = 1 , … , N − 1 i=1,\ldots,N-1 i = 1 , … , N − 1 , and a N = 2 a_N=2 a N = 2 . Then for any N ≥ 2 N\ge2 N ≥ 2 , we have
E [ r ⋆ − r ˉ a e m p ] ≥ 0.075. \mathbb{E}[r^\star-\bar{r}_{a^{\mathrm{emp}}}] \ge0.075. E [ r ⋆ − r ˉ a emp ] ≥ 0.075.
Remark 6.4. In this example, the optimal arm is arm 1, which is sampled N − 1 N-1 N − 1 times out of a total of N N N observations. Consequently, the corresponding C ⋆ C^\star C ⋆ is exceedingly close to 1.
Proof of Example 6.1. Under the reward model, the empirical reward estimates satisfy r ˉ 1 ∼ N ( 1 , 1 N − 1 ) \bar{r}_1\sim\mathcal{N}\left(1,\frac{1}{N-1}\right) r ˉ 1 ∼ N ( 1 , N − 1 1 ) and r ˉ 2 ∼ N ( 0 , 1 ) \bar{r}_2\sim\mathcal{N}(0,1) r ˉ 2 ∼ N ( 0 , 1 ) , and hence
P ( r ˉ 1 < 1 ) = 0.5 and P ( r ˉ 2 ≥ 1 ) ≥ 0.15. \mathbb{P}(\bar{r}_1<1)=0.5 \qquad\text{and}\qquad\mathbb{P}(\bar{r}_2\ge1)\ge0.15. P ( r ˉ 1 < 1 ) = 0.5 and P ( r ˉ 2 ≥ 1 ) ≥ 0.15.
It then follows from the statistical independence between r ˉ 1 \bar{r}_1 r ˉ 1 and r ˉ 2 \bar{r}_2 r ˉ 2 that
E [ r 1 − r ˉ a e m p ] = P ( r ˉ 2 > r ˉ 1 ) ≥ P ( r ˉ 1 < 1 ) P ( r ˉ 2 ≥ 1 ) ≥ 0.075 \mathbb{E}[r_1-\bar{r}_{a^{\mathrm{emp}}}]
=\mathbb{P}(\bar{r}_2>\bar{r}_1)
\ge\mathbb{P}(\bar{r}_1<1)\mathbb{P}(\bar{r}_2\ge1)
\ge0.075 E [ r 1 − r ˉ a emp ] = P ( r ˉ 2 > r ˉ 1 ) ≥ P ( r ˉ 1 < 1 ) P ( r ˉ 2 ≥ 1 ) ≥ 0.075
as claimed.
The lower confidence bound (LCB) algorithm. As illustrated in Example 6.1, the main issue with the empirical best-arm estimator is the systematic over-estimation of the rewards of poorly sampled arms, caused by large variance. A natural strategy toward mitigating this issue is therefore to properly adjust the empirical reward estimates by accounting for variance information.
This idea leads to the principle of pessimism (or conservatism) in the face of uncertainty: arms with fewer samples should be treated more conservatively, since their empirical rewards are more volatile and less reliable. As a concrete method, one can penalize the empirical reward estimate by a penalty function b a b_a b a that decreases as the sample count N a N_a N a (i.e., the number of times arm a a a is sampled) increases. Rather than selecting the arm with the largest empirical reward, the pessimism principle instead recommends choosing the arm according to the lower confidence bound:
a l c b = arg max a ∈ A r a l c b with r a l c b = r ‾ a − b a . a^{\mathrm{lcb}} = \underset{a \in\mathcal{A}}{\arg\max} r_a^{\mathrm{lcb}} \qquad\text{with } r_a^{\mathrm{lcb}} = \overline{r}_a - b_a. a lcb = a ∈ A arg max r a lcb with r a lcb = r a − b a .
A concrete choice of the penalty function is
b a = 2 log ( 4 A δ ) N a (574) b_a = \sqrt{\frac{2\log\left(\frac{4A}{\delta}\right)}{N_a}} \tag*{(574)} b a = N a 2 log ( δ 4 A ) (574)
with a prescribed confidence parameter δ ∈ ( 0 , 1 ) \delta\in(0,1) δ ∈ ( 0 , 1 ) , which serves as a confidence width derived from standard Gaussian tail bounds and resembles the exploration bonus (cf. (361) ) adopted in the UCB algorithm. Intuitively, the quantity r a l c b r_a^{\mathrm{lcb}} r a lcb can be viewed as a lower confidence bound on the true reward r a r_a r a , motivating the name LCB algorithm. The full procedure is summarized in Algorithm 6.1.
Careful readers may notice that the LCB algorithm bears some resemblance to the UCB algorithm presented in § 8.2 . However, whereas UCB adds an exploration bonus to encourage exploration in online settings, the LCB approach subtracts a confidence penalty to guard against overestimation in the absence of exploration.
Remark 6.5. As a sanity check, when applying Algorithm 6.1 to Example 6.1, we can readily verify that r 1 l c b ≈ r 1 = 1 r_1^{\mathrm{lcb}} \approx r_1 = 1 r 1 lcb ≈ r 1 = 1 for large enough N N N , and r 2 l c b ≤ r 2 = 0 r_2^{\mathrm{lcb}} \leq r_2 = 0 r 2 lcb ≤ r 2 = 0 . Consequently, the LCB algorithm selects arm 1 (the optimal arm) and rules out arm 2 with high probability. Part of the reason this approach works well in the present example is that the algorithm effectively “trusts” the preference of the offline dataset: since the optimal arm is sampled frequently in the data, the estimate of r 1 r_1 r 1 becomes highly reliable, allowing the algorithm to confidently distinguish it from the suboptimal arm.
Theoretical guarantees
We now turn to the statistical guarantees of the LCB algorithm. As it turns out, the pessimism principle described above proves highly effective in coping with the challenges inherent to offline settings. This is formalized in the following theorem, originally established by Rashidinejad et al. (2021).
Theorem 6.1. Suppose that the sample size satisfies N ≥ 8 C ∗ log ( 2 / δ ) N \geq8C^* \log(2/\delta) N ≥ 8 C ∗ log ( 2/ δ ) . Then with probability at least 1 − δ 1-\delta 1 − δ , the action a l c b a^{\mathrm{lcb}} a lcb returned by the LCB algorithm (cf. Algorithm 6.1) with penalty function (574) achieves
r ∗ − r a l c b ≤ 4 C ∗ log ( 4 A / δ ) N . (575) r^* - r_{a^{\mathrm{lcb}}} \leq4\sqrt{\frac{C^*\log(4A/\delta)}{N}}. \tag*{(575)} r ∗ − r a lcb ≤ 4 N C ∗ log ( 4 A / δ ) . (575)
Algorithm 6.1: The lower confidence bound (LCB) algorithm for multi-armed bandits
input: batch dataset D = { ( a i , R i ) } i = 1 N \mathcal{D} = \{(a_i, R_i)\}_{i=1}^{N} D = {( a i , R i ) } i = 1 N , number of arms A \mathcal{A} A , confidence parameter δ ∈ ( 0 , 1 ) \delta\in(0,1) δ ∈ ( 0 , 1 ) .
for a ∈ A a \in\mathcal{A} a ∈ A do
set N a ← ∑ i = 1 N 1 { a i = a } N_a \gets\sum_{i=1}^{N} 1\{a_i = a\} N a ← ∑ i = 1 N 1 { a i = a } . /* visitation count. */
compute empirical reward
r ˉ a = { 1 N a ∑ i = 1 N R i 1 { a i = a } , if N a > 0 , 0 , otherwise. \bar{r}_a =
\begin{cases}
\frac{1}{N_a}\sum_{i=1}^{N} R_i 1\{a_i = a\}, & \text{if } N_a > 0,\\
0, & \text{otherwise.}
\end{cases} r ˉ a = { N a 1 ∑ i = 1 N R i 1 { a i = a } , 0 , if N a > 0 , otherwise.
set b a ← 2 log ( 4 A / δ ) N a b_a \gets\sqrt{\frac{2\log(4\mathcal{A}/\delta)}{N_a}} b a ← N a 2 l o g ( 4 A / δ ) . /* LCB penalty. */
output: a l c b = arg max a r a l c b a^{\mathrm{lcb}} = \arg\max_a r_a^{\mathrm{lcb}} a lcb = arg max a r a lcb , where r a l c b = r ˉ a − b a r_a^{\mathrm{lcb}} = \bar{r}_a - b_a r a lcb = r ˉ a − b a .
Before embarking on the proof of this result, we make several remarks. To achieve any target accuracy 0 < ε < 1 0 < \varepsilon< 1 0 < ε < 1 (in the sense that r ⋆ − r a l c b ≤ ε r^\star- r_{a^{\mathrm{lcb}}} \le\varepsilon r ⋆ − r a lcb ≤ ε ), it suffices for the LCB algorithm to use
O ~ ( C ⋆ ε 2 ) (576) \widetilde{O}\left(\frac{C^\star}{\varepsilon^2}\right) \tag*{(576)} O ( ε 2 C ⋆ ) (576)
samples from the offline dataset D \mathcal{D} D . This stands in sharp contrast to the empirical best-arm estimator, which in certain scenarios may be fundamentally incapable of achieving o ( 1 ) o(1) o ( 1 ) statistical error. This theorem therefore provides rigorous evidence that the pessimism principle can significantly outperform the vanilla empirical approach.
Notably, the sample complexity of the LCB algorithm scales linearly with the single-policy concentrability coefficient, without requiring this coefficient to be known a priori. This unveils that the statistical efficiency of the pessimistic approach automatically adapts to the intrinsic quality of the offline dataset, as characterized by C ⋆ C^\star C ⋆ , rather than the number of arms A \mathcal{A} A .
Intuitive explanation. Prior to delving into the proof of Theorem 6.1, it is instructive to first develop some intuition for the result. Suppose, for simplicity, that every suboptimal arm a a a satisfies r a < r ⋆ − ε r_a < r^\star- \varepsilon r a < r ⋆ − ε , so that achieving ε \varepsilon ε -accuracy amounts to ensuring that any suboptimal arm is unlikely to be selected.
Recall from the definition (571) that C ⋆ = 1 / d b ( a ⋆ ) C^\star= 1/d^b(a^\star) C ⋆ = 1/ d b ( a ⋆ ) . Consequently, the number of observations collected for the optimal arm a ⋆ a^\star a ⋆ scales as
N a ⋆ ≍ N d b ( a ⋆ ) ≍ N / C ⋆ . N_{a^\star} \asymp N d^b(a^\star) \asymp N/C^\star. N a ⋆ ≍ N d b ( a ⋆ ) ≍ N / C ⋆ .
As a result, when the sample size N N N is sufficiently large, both the empirical estimate r ˉ a ⋆ \bar{r}_{a^\star} r ˉ a ⋆ and the corresponding lower confidence bound r a ⋆ l c b r_{a^\star}^{\mathrm{lcb}} r a ⋆ lcb concentrate tightly around the true reward:
r ˉ a ⋆ ≈ r a ⋆ l c b = r a ⋆ − O ( 1 N a ⋆ ) = r ⋆ − O ( C ⋆ N ) , \bar{r}_{a^\star} \approx r_{a^\star}^{\mathrm{lcb}} = r_{a^\star} - O\left(\frac{1}{\sqrt{N_{a^\star}}}\right) = r^\star- O\left(\sqrt{\frac{C^\star}{N}}\right), r ˉ a ⋆ ≈ r a ⋆ lcb = r a ⋆ − O ( N a ⋆ 1 ) = r ⋆ − O ( N C ⋆ ) ,
where logarithmic factors are omitted for simplicity. Therefore, the lower confidence bound r a ⋆ l c b r_{a^\star}^{\mathrm{lcb}} r a ⋆ lcb can be statistically separated from the mean reward r a r_a r a of any suboptimal arm — and hence also from its lower confidence bound r a l c b r_a^{\mathrm{lcb}} r a lcb — provided that
C ⋆ N ≪ ε ⟺ N ≫ C ⋆ ε 2 . \sqrt{\frac{C^\star}{N}} \ll\varepsilon
\quad\Longleftrightarrow\quad
N \gg\frac{C^\star}{\varepsilon^2}. N C ⋆ ≪ ε ⟺ N ≫ ε 2 C ⋆ .
This heuristic captures the key intuition underlying the analysis.
Analysis: proof of Theorem 6.1
We begin by establishing two high-probability events. First, akin to the application of the Gaussian tail bounds (5.11) in the online setting, one can show that
r a l c b = r ˉ a − b a ≤ r a ≤ r ˉ a + b a for all 1 ≤ a ≤ A , (577) r_a^{\mathrm{lcb}} = \bar{r}_a - b_a \le r_a \le\bar{r}_a + b_a \qquad\text{for all } 1 \le a \le A, \tag*{(577)} r a lcb = r ˉ a − b a ≤ r a ≤ r ˉ a + b a for all 1 ≤ a ≤ A , (577)
with probability at least 1 − δ / 2 1-\delta/2 1 − δ /2 . This guarantees that r a l c b r_a^{\mathrm{lcb}} r a lcb is indeed a pessimistic estimate of the true mean reward r a r_a r a . Second, the concentration bound in Lemma 2.7 tells us that
N a ⋆ ≥ E [ N a ⋆ ] − 1 2 N d b ( a ⋆ ) = 1 2 N d b ( a ⋆ ) (578) N_{a^\star} \ge\mathbb{E}[N_{a^\star}] - \frac{1}{2}Nd^b(a^\star) = \frac{1}{2}Nd^b(a^\star) \tag*{(578)} N a ⋆ ≥ E [ N a ⋆ ] − 2 1 N d b ( a ⋆ ) = 2 1 N d b ( a ⋆ ) (578)
with probability at least 1 − exp ( − N d b ( a ⋆ ) / 8 ) 1-\exp(-Nd^b(a^\star)/8) 1 − exp ( − N d b ( a ⋆ ) /8 ) . Consequently, if N ≥ 8 C ⋆ log ( 2 / δ ) N \ge8C^\star\log(2/\delta) N ≥ 8 C ⋆ log ( 2/ δ ) , then (578) holds with probability exceeding 1 − δ / 2 1-\delta/2 1 − δ /2 .
Conditioning on the intersection of the two events above, we proceed to compare the optimal arm a ⋆ a^\star a ⋆ with the pessimistic choice a l c b a^{\mathrm{lcb}} a lcb . It is readily seen from (577) that
r a ⋆ ≤ r ˉ a ⋆ + b a ⋆ = r ˉ a ⋆ − b a ⋆ + 2 b a ⋆ . r_{a^\star} \le\bar{r}_{a^\star}+b_{a^\star}=\bar{r}_{a^\star}-b_{a^\star}+2b_{a^\star}. r a ⋆ ≤ r ˉ a ⋆ + b a ⋆ = r ˉ a ⋆ − b a ⋆ + 2 b a ⋆ .
In view of the definition of a l c b a^{\mathrm{lcb}} a lcb , we have r ˉ a ⋆ − b a ⋆ ≤ r ˉ a l c b − b a l c b \bar{r}_{a^\star}-b_{a^\star}\le\bar{r}_{a^{\mathrm{lcb}}}-b_{a^{\mathrm{lcb}}} r ˉ a ⋆ − b a ⋆ ≤ r ˉ a lcb − b a lcb , which combined with (577) again yields
r a ⋆ ≤ r ˉ a l c b − b a l c b + 2 b a ⋆ ≤ r a l c b + 2 b a ⋆ . (579) r_{a^\star}\le\bar{r}_{a^{\mathrm{lcb}}}-b_{a^{\mathrm{lcb}}}+2b_{a^\star}\le r_{a^{\mathrm{lcb}}}+2b_{a^\star}. \tag*{(579)} r a ⋆ ≤ r ˉ a lcb − b a lcb + 2 b a ⋆ ≤ r a lcb + 2 b a ⋆ . (579)
Therefore, to bound the suboptimality gap r a ⋆ − r a l c b r_{a^\star}-r_{a^{\mathrm{lcb}}} r a ⋆ − r a lcb , it suffices to control the penalty term b a ⋆ b_{a^\star} b a ⋆ , which reduces to lower bounding N a ⋆ N_{a^\star} N a ⋆ . By virtue of (578) and the definition C ⋆ = 1 / d b ( a ⋆ ) C^\star=1/d^b(a^\star) C ⋆ = 1/ d b ( a ⋆ ) (cf. (571) ), we have
N a ⋆ ≥ 1 2 N d b ( a ⋆ ) = N 2 C ⋆ > 0 , N_{a^\star}\ge\frac{1}{2}Nd^b(a^\star)=\frac{N}{2C^\star}>0, N a ⋆ ≥ 2 1 N d b ( a ⋆ ) = 2 C ⋆ N > 0 ,
which further implies that
b a ⋆ = 2 log ( 4 A / δ ) N a ⋆ ≤ 4 C ⋆ log ( 4 A / δ ) N . b_{a^\star}=\sqrt{\frac{2\log(4A/\delta)}{N_{a^\star}}}\le\sqrt{\frac{4C^\star\log(4A/\delta)}{N}}. b a ⋆ = N a ⋆ 2 log ( 4 A / δ ) ≤ N 4 C ⋆ log ( 4 A / δ ) .
Substituting this into (579) , we arrive at
r a ⋆ − r a l c b ≤ 2 b a ⋆ ≤ 4 C ⋆ log ( 4 A / δ ) N . r_{a^\star}-r_{a^{\mathrm{lcb}}}\le2b_{a^\star}\le4\sqrt{\frac{C^\star\log(4A/\delta)}{N}}. r a ⋆ − r a lcb ≤ 2 b a ⋆ ≤ 4 N C ⋆ log ( 4 A / δ ) .
with probability at least 1 − δ 1-\delta 1 − δ . This establishes Theorem 6.1.
Model-based offline RL
Thus far, we have demonstrated the effectiveness of the pessimism principle in the simpler setting of offline multi-armed bandits. This naturally motivates extending the pessimism principle to offline RL, with the goal of addressing the challenges of distribution mismatch and limited data coverage in much more general offline settings.
As it turns out, despite the different sources of uncertainty in offline bandits and offline RL — namely, noisy rewards in the former and unknown transition models in the latter — the pessimism