Introduction
Constrained Markov Decision Processes (CMDPs) provide a principled framework for constrained reinforcement learning with both reward objectives and safety requirements [4 , 22 ] . This framework is especially relevant in safety-critical domains, and a growing line of provably constrained RL algorithms has been developed within this framework [8 , 9 , 16 , 24 , 27 , 39 ] . These works primarily focus on the tabular setting, where guarantees scale with the cardinality of the state space and become unsuitable for continuous or high-dimensional environments. Linear CMDP methods [5 , 13 , 17 , 18 , 26 , 42 ] partially address this issue by assuming that transitions are linearly represented by known low-dimensional features, yielding state-dimension-independent bounds. However, existing tabular/linear CMDP methods all rely on a priori knowledge of the latent feature representations. In the absence of such prior feature knowledge, their guarantees no longer apply, leaving CMDPs with unknown features insufficiently addressed.
Representation learning has been effective for discovering low-dimensional transition features in unconstrained MDPs [2 , 29 , 31 , 36 , 37 , 41 , 44 ] . Extending this idea to CMDPs is more challenging because representation error now affects both reward optimization and constraint evaluation. The
Preprint. learner must identify latent dynamics from online data, use the learned representation to guide exploration, and still control cumulative constraint violation [2 , 16 ] . This creates a coupling between representation coverage and constraint-aware policy optimization that is absent from unconstrained low-rank RL and is not handled by existing CMDP results with known features.
In this paper, we develop primal-dual representation-learning algorithms for low-rank continuous-state CMDPs with unknown transition features. During feature learning, the algorithms interact with the environment via an ϵ \epsilon ϵ -greedy mixture execution policy and use the collected samples to learn transition features by maximum-likelihood estimation (MLE). For soft violation criterion, where only the total constraint shortfall across episodes is charged, our algorithm, REP-PD-TEG, uses the learned features to construct uncertainty bonuses, and plan with the bonus-augmented Lagrangian objective that couples reward maximization, utility optimization, and exploration. The algorithm outputs the return policies with a primal-dual update, and admits both return-policy and execution-policy guarantees. For the stricter per-episode hard-violation metric, where each episode’s shortfall is charged separately, we develop REP-PD-H-TEG, which keeps the representation-learning procedure but plans under a tight lower-confidence utility certificate and conducts a stabilized dual search to obtain the return policy. Specifically, we make the following contributions:
We propose the first theoretically guaranteed representation-learning method for low-rank continuous-state CMDPs with unknown features. Starting from the occupancy of the current return policy, it uses ϵ \epsilon ϵ -greedy auxiliary probes for action coverage, fits transition features by MLE, and constructs a merged-covariance uncertainty bonus. We quantify the effect of decaying-ϵ \epsilon ϵ exploration for both return and execution policies.
For the aggregate soft-violation criterion, we propose REP-PD-TEG, which updates the return policy through an optimistic Lagrangian whose reward and dual-weighted utility terms are both bonus-augmented based on the learned features. We show that REP-PD-TEG achieves return policy regret and soft violation of order O ~ ( K 1 / 2 + p ) \widetilde{O}(K^{1/2+p}) O ( K 1/2 + p ) sample complexity under the schedule ϵ k = k − p \epsilon_k = k^{-p} ϵ k = k − p , where ϵ k \epsilon_k ϵ k is the exploration level in episode k k k . In particular, this recovers the optimal O ~ ( K ) \widetilde{O}(\sqrt{K}) O ( K ) return policy rate when p = 0 p = 0 p = 0 , which corresponds to uniform auxiliary probing over actions during the whole procedure. We also prove an execution policy overall bound: executing the ϵ k \epsilon_k ϵ k -greedy exploration adds O ( K 1 − p ) O(K^{1-p}) O ( K 1 − p ) complexity, and the balanced choice p = 1 / 4 p = 1/4 p = 1/4 yields a O ~ ( K 3 / 4 ) \widetilde{O}(K^{3/4}) O ( K 3/4 ) execution policy regret and soft violation up to problem-dependent factors.
For the stricter per-episode hard-violation criterion, we propose REP-PD-H-TEG, which uses the learned features to conduct a stabilized dual search and certifies feasibility through a bonus-corrected lower-confidence utility certificate. For return policies, we obtain regret of order O ~ ( K 3 / 4 + p ) \widetilde{O}(K^{3/4+p}) O ( K 3/4 + p ) and hard violation of order O ~ ( K 1 / 2 + p ) \widetilde{O}(K^{1/2+p}) O ( K 1/2 + p ) ; taking p = 0 p = 0 p = 0 again gives uniform probing and the clean O ~ ( K ) \widetilde{O}(\sqrt{K}) O ( K ) hard-violation dependence. For execution policies, we show that the cost of ϵ k \epsilon_k ϵ k -greedy exploration is also O ( K 1 − p ) O(K^{1-p}) O ( K 1 − p ) and choosing p = 1 / 8 p = 1/8 p = 1/8 gives a O ~ ( K 7 / 8 ) \widetilde{O}(K^{7/8}) O ( K 7/8 ) regret and hard violation up to problem-dependent factors.
Appendix H provides the empirical study of REP-PD-TEG on a simulated job-scheduling task. Besides, we also implement an approximate neural network version of REP-PD-TEG on the Safety Gym task to validate the effectiveness of our representation learning procedure for a complex state-action space.
Remark on the probing schedule. In tabular and linear CMDPs, the state or feature representation is known, so the analyzed policy is the execution policy. In low-rank CMDPs, the transition features must be learned online, and auxiliary ϵ k \epsilon_k ϵ k -greedy probing is needed to obtain coverage over the state-action space for representation learning. This creates a distinction between the return-policy sequence and the execution policy. The latter is especially relevant in constrained RL because exploratory behavior can incur real training-time costs—for example, robot collisions or excessive contact forces—even when the final return policy is safe. The two REP-PD-TEG rows in Table 1 report, respectively, the sample complexity of the best return policy specialization (p = 0 p = 0 p = 0 ) and the best execution policy specialization (p = 1 / 4 p = 1/4 p = 1/4 ), where p p p denotes the probing strength with ϵ k = k − p \epsilon_k = k^{-p} ϵ k = k − p . Similarly, the two REP-PD-H-TEG rows in Table 2 show the best return policy complexity with p = 0 p = 0 p = 0 and the balanced execution policy complexity under the hard specialization p = 1 / 8 p = 1/8 p = 1/8 , respectively.
Table 2.
Table 1 : Comparison of soft-violation bounds. The two REP-PD-TEG rows report the complexity of the best return policy specialization and the best execution policy specialization, respectively.
Table 1.
Table 2 : Comparison of hard-violation bounds. The two REP-PD-H-TEG rows report the complexity of the best return policy specialization and the balanced execution policy specialization, respectively.
Related Work
Provably Constrained RL CMDPs provide a rigorous framework for safe RL, balancing reward maximization with cumulative cost constraints [4 , 19 ] . Several practical constrained RL methods have been developed on this foundation, including policy-gradient, Lagrangian, shielding, and model-based approaches [1 , 6 , 7 , 11 , 21 , 35 ] . On the theoretical side, there has been a growing body of work on provably safe reinforcement learning. Many works [8 , 9 , 14 , 16 , 24 , 25 , 27 , 39 ] consider the tabular setting. Among the best known regret and constraint-violation guarantees are of order O ~ ( K ) \widetilde{O}(\sqrt{K}) O ( K ) [16 , 27 ] . However, their polynomial dependence on the cardinality of state space ∣ S ∣ |\mathcal{S}| ∣ S ∣ makes them unsuitable for large or continuous state spaces.
Recent advances in linear CMDPs [5 , 13 , 17 , 18 , 26 , 30 , 42 , 43 ] achieve dimension-independent sample complexity by assuming known low-dimensional features ϕ ∗ ( s , a ) \phi^{*}(s,a) ϕ ∗ ( s , a ) . They enforce safety through confidence intervals, optimistic bonuses, primal-dual updates, pessimistic constraints, or safe baselines. However, their reliance on predefined latent features, assumed to be known a priori or handcrafted, limits applicability when transition structures are unknown and must be learned from data. Our setting instead has unknown latent features that must be learned online. To our knowledge, the only prior work on low-rank CMDPs is [20 ] , which studies the sample complexity of identifying a near-optimal feasible policy after training. In contrast, our work analyzes the entire learning trajectory and provides regret and constraint-violation guarantees.
Representation Learning for Unconstrained MDPs A growing body of research has focused on theoretically grounded representation learning in RL, aiming to derive low-dimensional features that efficiently capture environment dynamics. For example, works such as [15 , 28 , 46 ] investigated representation learning in block MDPs, leveraging hierarchical or modular state decompositions to improve sample efficiency. [38 ] studied representations under a Gaussian noise model, establishing guarantees for feature learning in noisy environments. Low-Rank MDPs have emerged as a pivotal framework for representation learning in structured environments. Within this regime, model-based approaches [2 , 36 , 41 , 44 , 45 ] learn latent transition features by leveraging model classes of transition probabilities, achieving sample-efficient exploration under realizability assumptions. In contrast, model-free methods like [29 ] eschew explicit transition models and show efficiency under minimal reachability conditions. The framework has recently expanded into reward-free exploration [10 , 20 ] and multi-agent Markov games [31 , 34 , 47 , 48 ] . However, these works do not address online safety constraints in CMDPs. Consequently, this work gives the first representation-learning framework for low-rank CMDPs that bridges feature discovery and provable online safety.
Background
We consider an episodic constrained MDP, denoted by ( S , A , P ∗ , r , g , γ , d 1 ) (\mathcal{S}, \mathcal{A}, P^{\ast}, r, g, \gamma, d_1) ( S , A , P ∗ , r , g , γ , d 1 ) where S \mathcal{S} S is the state space, A \mathcal{A} A is the action space, P ∗ P^{\ast} P ∗ is the transition function, r r r is the reward function, g g g is the utility function, γ \gamma γ is the discounted factor, and d 1 d_1 d 1 is the initial distribution. Following prior work [23 , 40 , 41 ] , we assume trajectory reward is normalized, i.e., for any trajectory { s h , a h } h = 1 ∞ \{s_h, a_h\}_{h=1}^{\infty} { s h , a h } h = 1 ∞ , we have
∑ h = 1 ∞ γ h r ( s h , a h ) ∈ [ 0 , 1 ] and ∑ h = 1 ∞ γ h g ( s h , a h ) ∈ [ 0 , 1 ] . \sum_{h=1}^{\infty} \gamma^h r(s_h, a_h) \in[0,1] \quad\text{and} \quad\sum_{h=1}^{\infty} \gamma^h g(s_h, a_h) \in[0,1]. h = 1 ∑ ∞ γ h r ( s h , a h ) ∈ [ 0 , 1 ] and h = 1 ∑ ∞ γ h g ( s h , a h ) ∈ [ 0 , 1 ] .
In an infinite-horizon discounted CMDP, each episode starts with state s 1 ∼ d 1 s_1 \sim d_1 s 1 ∼ d 1 . Then at each step h h h , the agent observes state s h ∈ S s_h \in\mathcal{S} s h ∈ S , picks an action a h ∈ A a_h \in\mathcal{A} a h ∈ A , and receives a reward r ( s h , a h ) r(s_h, a_h) r ( s h , a h ) and a utility g ( s h , a h ) g(s_h, a_h) g ( s h , a h ) . The MDP then transitions to s h + 1 s_{h+1} s h + 1 which is drawn from P ∗ ( ⋅ ∣ s h , a h ) P^{\ast}(\cdot\mid s_h, a_h) P ∗ ( ⋅ ∣ s h , a h ) . In this paper, we assume access to reward and utility function feedback, but does not require access to the transition model. For any s ∈ S s \in\mathcal{S} s ∈ S , π ( a ∣ s ) \pi(a \mid s) π ( a ∣ s ) denotes the probability of selecting action a ∈ A a \in\mathcal{A} a ∈ A when the state is s s s under policy π \pi π . Let d P ∗ π ( s ) d_{P^{\ast}}^{\pi}(s) d P ∗ π ( s ) be the state occupancy measure
d P ∗ π ( s ) = ( 1 − γ ) ∑ h = 1 ∞ γ h − 1 P π ∗ ( s h = s ∣ s 1 ∼ d 1 ) . d_{P^{\ast}}^{\pi}(s) = (1-\gamma)\sum_{h=1}^{\infty}\gamma^{h-1}P_{\pi}^{\ast}(s_h=s \mid s_1\sim d_1). d P ∗ π ( s ) = ( 1 − γ ) h = 1 ∑ ∞ γ h − 1 P π ∗ ( s h = s ∣ s 1 ∼ d 1 ) .
where P π ∗ ( s h = s ∣ s 1 ∼ d 1 ) P_{\pi}^{\ast}(s_h=s\mid s_1\sim d_1) P π ∗ ( s h = s ∣ s 1 ∼ d 1 ) is the probability that s h = s s_h=s s h = s , after starting at state s 1 ∼ d 1 s_1\sim d_1 s 1 ∼ d 1 and following π \pi π thereafter. Also, let d P ∗ π ( s , a ) = π ( a ∣ s ) d P ∗ π ( s ) d_{P^{\ast}}^{\pi}(s,a)=\pi(a|s)d_{P^{\ast}}^{\pi}(s) d P ∗ π ( s , a ) = π ( a ∣ s ) d P ∗ π ( s ) be the state-action occupancy measure. We define the expected value function for the discounted reward under policy π \pi π starting from state s s s as:
V P ∗ , r π ( s ) = E P ∗ , π [ ∑ h = 1 ∞ γ h r ( s h , a h ) | s 1 = s ] , (1) V_{P^{\ast},r}^{\pi}(s)=\mathbb{E}_{P^{\ast},\pi}\left[\sum_{h=1}^{\infty}\gamma^h r(s_h,a_h)\middle|s_1=s\right], \tag*{(1)} V P ∗ , r π ( s ) = E P ∗ , π [ h = 1 ∑ ∞ γ h r ( s h , a h ) s 1 = s ] , (1)
where E \mathbb{E} E is taken with respect to the policy π \pi π and the transition function P ∗ P^{\ast} P ∗ . Similarly, we define the action-value function for the reward as:
Q P ∗ , r π ( s , a ) = E P ∗ , π [ ∑ h = 1 ∞ γ h r ( s h , a h ) | s 1 = s , a 1 = a ] . (2) Q_{P^{\ast},r}^{\pi}(s,a)=\mathbb{E}_{P^{\ast},\pi}\left[\sum_{h=1}^{\infty}\gamma^h r(s_h,a_h)\middle|s_1=s,a_1=a\right]. \tag*{(2)} Q P ∗ , r π ( s , a ) = E P ∗ , π [ h = 1 ∑ ∞ γ h r ( s h , a h ) s 1 = s , a 1 = a ] . (2)
For simplicity, when the context is clear, we sometimes abbreviate V P ∗ , r π V_{P^{\ast},r}^{\pi} V P ∗ , r π and Q P ∗ , r π Q_{P^{\ast},r}^{\pi} Q P ∗ , r π as V r π V_r^{\pi} V r π and Q r π Q_r^{\pi} Q r π , respectively. We also define the value function for the utility V g π ( s ) V_g^{\pi}(s) V g π ( s ) and the action-value function for the utility Q g π ( s , a ) Q_g^{\pi}(s,a) Q g π ( s , a ) analogously by replacing r r r with g g g in (1) and (2), respectively. We denote V j π ( s ) V_j^{\pi}(s) V j π ( s ) , and Q j π ( s , a ) Q_j^{\pi}(s,a) Q j π ( s , a ) for j = r , g j=r,g j = r , g .
Definition 2.1. For brevity, we denote P ∗ V j π ( s , a ) = E s ′ ∼ P ∗ ( ⋅ ∣ s , a ) V j π ( s ′ ) P^{\ast}V_j^{\pi}(s,a)=\mathbb{E}_{s'\sim P^{\ast}(\cdot|s,a)}V_j^{\pi}(s') P ∗ V j π ( s , a ) = E s ′ ∼ P ∗ ( ⋅ ∣ s , a ) V j π ( s ′ ) for j = r , g j=r,g j = r , g .
Using this notation, the Bellman’s equation associated with the policy π \pi π becomes
Q j π ( s , a ) = ( j + γ P ∗ V j π ) ( s , a ) , j ∈ { r , g } . Q_j^{\pi}(s,a)=(j+\gamma P^{\ast}V_j^{\pi})(s,a), \qquad j\in\{r,g\}. Q j π ( s , a ) = ( j + γ P ∗ V j π ) ( s , a ) , j ∈ { r , g } .
Note that V j π ( s ) = ⟨ π ( ⋅ ∣ s ) , Q j π ( s , ⋅ ) ⟩ A V_j^{\pi}(s)=\langle\pi(\cdot|s),Q_j^{\pi}(s,\cdot)\rangle_{\mathcal{A}} V j π ( s ) = ⟨ π ( ⋅ ∣ s ) , Q j π ( s , ⋅ ) ⟩ A , where ⟨ π ( ⋅ ∣ s ) , Q j π ( s , ⋅ ) ⟩ A = ∑ a ∈ A π h ( a ∣ s ) Q j π ( s , a ) \langle\pi(\cdot|s),Q_j^{\pi}(s,\cdot)\rangle_{\mathcal{A}}=\sum_{a\in\mathcal{A}}\pi_h(a|s)Q_j^{\pi}(s,a) ⟨ π ( ⋅ ∣ s ) , Q j π ( s , ⋅ ) ⟩ A = ∑ a ∈ A π h ( a ∣ s ) Q j π ( s , a ) .
The objective of the learning agent is to find an optimal policy that maximizes the expected reward while satisfying a safety constraint on the expected utility:
max π V r π ( s 1 ) , subject to V g π ( s 1 ) ≥ b . (3) \max_{\pi} V_r^{\pi}(s_1), \qquad\text{subject to } V_g^{\pi}(s_1)\ge b. \tag*{(3)} π max V r π ( s 1 ) , subject to V g π ( s 1 ) ≥ b . (3)
where b ∈ ( 0 , 1 ] b \in(0,1] b ∈ ( 0 , 1 ] is a given threshold. While we focus on a single constraint for simplicity, our method naturally extends to handle multiple constraints. We denote the optimal policy as π ∗ \pi^* π ∗ which solves the above optimization problem.
In the absence of prior knowledge about the constraint, the agent must balance exploration and exploitation while ensuring that the policy satisfies the constraint. To this end, we allow the policy to occasionally violate the constraint and aim to minimize the regret and the total constraint violations over K K K episodes. This approach is consistent with existing literature on constrained RL [12 , 13 , 16 ] .
To formally measure the performance of the policy sequence { π k } 1 K \{\pi_k\}_{1}^{K} { π k } 1 K , we define the regret as
Regret ( K ) = ∑ k = 1 K V r π ∗ ( s 1 ) − V r π k ( s 1 ) , \operatorname{Regret}(K)=\sum_{k=1}^{K} V_r^{\pi^*}(s_1)-V_r^{\pi_k}(s_1), Regret ( K ) = k = 1 ∑ K V r π ∗ ( s 1 ) − V r π k ( s 1 ) ,
and the soft/hard constraint violations as
Violation s o f t ( K ) = [ ∑ k = 1 K ( b − V g π k ( s 1 ) ) ] + , Violation h a r d ( K ) = ∑ k = 1 K [ b − V g π k ( s 1 ) ] + , \operatorname{Violation}_{\mathrm{soft}}(K)=\left[\sum_{k=1}^{K}\left(b-V_g^{\pi_k}(s_1)\right)\right]_+,\quad\operatorname{Violation}_{\mathrm{hard}}(K)=\sum_{k=1}^{K}\left[b-V_g^{\pi_k}(s_1)\right]_+, Violation soft ( K ) = [ k = 1 ∑ K ( b − V g π k ( s 1 ) ) ] + , Violation hard ( K ) = k = 1 ∑ K [ b − V g π k ( s 1 ) ] + ,
where [ z ] + = max { z , 0 } [z]_+ = \max\{z,0\} [ z ] + = max { z , 0 } . The regret measures the gap between the total reward value by following the optimal policy π ∗ \pi^* π ∗ , and the total reward value obtained by following agent’s policy π k \pi_k π k at episode k k k over K K K episodes. The soft constraint violation quantifies the cumulative difference between the desired constraint threshold and the achieved utility, allowing for both positive and negative deviations over time. In contrast, the hard constraint violation is a stricter metric that measures only the cumulative positive deviations. As shown in [18 ] , even zero soft violation may lead to hard constraint violation that grows linearly with the number of episodes, while soft constraint violation can only grow sub-linearly with the number of episodes if the hard constraint violation grows sub-linearly.
These metrics capture cumulative notions of safety rather than strict zero-violation safe exploration. In particular, soft violation is the standard online-CMDP notion for budget-style or long-run average constraints, while hard violation is a stricter cumulative metric that counts only positive deviations. Throughout the paper, our guarantees concern cumulative violation control; they do not imply per-episode safety during learning.
Dual problem and Slater’s Condition. We then introduce some additional notation that will be used throughout this paper.
Definition 2.2. For any policy π \pi π and Lagrangian multiplier Y Y Y , we define the composite value function and Q-function as V π , Y ( s ) = V r π ( s ) + Y V g π ( s ) V^{\pi,Y}(s)=V_r^\pi(s)+YV_g^\pi(s) V π , Y ( s ) = V r π ( s ) + Y V g π ( s ) and Q π , Y ( s , a ) = Q r π ( s , a ) + Y Q g π ( s , a ) Q^{\pi,Y}(s,a)=Q_r^\pi(s,a)+YQ_g^\pi(s,a) Q π , Y ( s , a ) = Q r π ( s , a ) + Y Q g π ( s , a ) .
We can cast problem (3) as a saddle point problem max π min Y L ( π , Y ) \max_{\pi}\min_Y\mathcal{L}(\pi,Y) max π min Y L ( π , Y ) , with Lagrangian L ( π , Y ) = V r π ( s 1 ) + Y ( V g π ( s 1 ) − b ) = V r π , Y − Y b \mathcal{L}(\pi,Y)=V_r^\pi(s_1)+Y(V_g^\pi(s_1)-b)=V_r^{\pi,Y}-Yb L ( π , Y ) = V r π ( s 1 ) + Y ( V g π ( s 1 ) − b ) = V r π , Y − Y b . Although the Lagrangian is non-concave in π \pi π [3 ] , the strong duality holds [32 ] . Hence, there exists an optimal Lagrangian multiplier Y ∗ Y^* Y ∗ such that max π L ( π , Y ∗ ) \max_{\pi}\mathcal{L}(\pi,Y^*) max π L ( π , Y ∗ ) recovers the optimal reward value function.
We assume the following Slater’s condition in this paper.
Assumption 2.3 (Slater’s Condition). There exists θ > 0 \theta>0 θ > 0 , and a policy π ˉ \bar{\pi} π ˉ , such that V g π ˉ ( s 1 ) ≥ b + θ V_g^{\bar{\pi}}(s_1)\ge b+\theta V g π ˉ ( s 1 ) ≥ b + θ .
The Slater’s condition is mild in practice and widely used in prior work [13 , 16 , 33 ] ; it also implies that the optimal dual variable is bounded.
Lemma 2.4 (Boundedness of Y ∗ Y^* Y ∗ ). The optimal dual-variable Y ∗ Y^* Y ∗ satisfies V r π ∗ ( s 1 ) − V r π ˉ ( s 1 ) θ ≤ Y ∗ ≤ 1 θ \frac{V_r^{\pi^*}(s_1)-V_r^{\bar{\pi}}(s_1)}{\theta}\le Y^*\le\frac{1}{\theta} θ V r π ∗ ( s 1 ) − V r π ˉ ( s 1 ) ≤ Y ∗ ≤ θ 1 .
Low-rank CMDP. In this paper, we study low-rank CMDPs, which are characterized by a low-rank decomposition of the transition function. Specifically, we assume the following:
Assumption 2.5 (low-rank CMDP). The transition function of the CMDP P ∗ : S × A → Δ ( S ) \mathcal{P}^*: \mathcal{S}\times\mathcal{A}\to\Delta(\mathcal{S}) P ∗ : S × A → Δ ( S ) admits a low rank decomposition with rank d ∈ N d\in\mathbb{N} d ∈ N if there exists two embedding functions ϕ ∗ : S × A → R d , μ ∗ : S → R d \phi^*:\mathcal{S}\times\mathcal{A}\to\mathbb{R}^d,\mu^*:\mathcal{S}\to\mathbb{R}^d ϕ ∗ : S × A → R d , μ ∗ : S → R d such that
∀ s , s ′ ∈ S , a ∈ A : P ∗ ( s ′ ∣ s , a ) = ⟨ ϕ ∗ ( s , a ) , μ ∗ ( s ′ ) ⟩ . \forall s,s'\in\mathcal{S},\ a\in\mathcal{A}: P^*(s'|s,a)=\langle\phi^*(s,a),\mu^*(s')\rangle. ∀ s , s ′ ∈ S , a ∈ A : P ∗ ( s ′ ∣ s , a ) = ⟨ ϕ ∗ ( s , a ) , μ ∗ ( s ′ )⟩ . For normalization, we assume ∥ ϕ ∗ ( s , a ) ∥ ≤ 1 \|\phi^*(s,a)\|\le1 ∥ ϕ ∗ ( s , a ) ∥ ≤ 1 for all ( s , a ) (s,a) ( s , a ) and for any functions f : S → [ 0 , 1 ] f:\mathcal{S}\to[0,1] f : S → [ 0 , 1 ] ,
∥ ∫ μ ∗ ( s ) f ( s ) d s ∥ 2 ≤ d . \left\|\int\mu^*(s)f(s)\,ds\right\|_2\le\sqrt{d}. ∫ μ ∗ ( s ) f ( s ) d s 2 ≤ d .
Remark 2.6. The low-rank CMDP assumption, characterized by the existence of low-dimensional embedding functions ϕ ⋆ ( s , a ) \phi^{\star}(s,a) ϕ ⋆ ( s , a ) and μ ⋆ ( s ′ ) \mu^{\star}(s') μ ⋆ ( s ′ ) , is widely adopted structural assumption in RL [2 , 37 , 41 ] . This assumption encompasses several common settings as special cases, including tabular CMDPs, linear CMDPs, and block CMDPs.
Stochastic-oracle feedback model. Following prior low-rank representation-learning work, we study an occupancy-based stochastic-oracle setting. The learner may query states distributed as d π P ⋆ d_{\pi}^{P^{\star}} d π P ⋆ together with auxiliary transitions that are used only for representation learning and bonus construction. Practically, a discounted-occupancy state can be sampled by resetting to the initial distribution and rolling in under π \pi π for a geometrically distributed number of steps. Here K K K counts outer oracle rounds, not raw environment transitions; implementing K K K queries this way uses O ( K / ( 1 − γ ) + 3 K ) O(K/(1-\gamma)+3K) O ( K / ( 1 − γ ) + 3 K ) transitions in expectation. Our regret and violation bounds do not charge the complete roll-in trajectories. Likewise, the execution-policy corollaries transfer values to the stationary ϵ \epsilon ϵ -mixture policy rather than giving an end-to-end guarantee for the entire three-transition probing trajectory.
Function approximation setup and MLE oracle. Since the transition function P ⋆ P^{\star} P ⋆ is unknown, we employ a function class M = { ( μ , ϕ ) : μ ∈ Θ , ϕ ∈ Γ } \mathcal{M}=\{(\mu,\phi):\mu\in\Theta,\phi\in\Gamma\} M = {( μ , ϕ ) : μ ∈ Θ , ϕ ∈ Γ } to characterize μ ⋆ \mu^{\star} μ ⋆ and ϕ ⋆ \phi^{\star} ϕ ⋆ , and by extension, to approximate P ⋆ P^{\star} P ⋆ . Our formal guarantees use a finite realizable class, ∣ M ∣ < ∞ |\mathcal{M}|<\infty ∣ M ∣ < ∞ and P ⋆ ∈ M P^{\star}\in\mathcal{M} P ⋆ ∈ M ; this restricts the candidate representations rather than the cardinality of the state space. Finite realizability and logarithmic dependence on ∣ M ∣ |\mathcal{M}| ∣ M ∣ are standard in low-rank representation-learning analyses [2 , 41 ] . To select the appropriate transition P ^ \widehat{P} P from the function class M \mathcal{M} M , we assume access to a supervised learning-style maximum-likelihood estimation (MLE) oracle.
Definition 2.7 (MLE Oracle). Given a model class M \mathcal{M} M and a dataset D \mathcal{D} D in the form of ( s , a , s ′ ) (s,a,s') ( s , a , s ′ ) , the MLE oracle returns the maximum likelihood estimator P ^ = ( μ ^ , ϕ ^ ) = arg max ( μ , ϕ ) ∈ M E D ln ( μ ( s ′ ) ⊤ ϕ ( s , a ) ) \widehat{P}=(\widehat{\mu},\widehat{\phi})=\mathop{\arg\max}_{(\mu,\phi)\in\mathcal{M}}\mathbb{E}_{\mathcal{D}}\ln(\mu(s')^{\top}\phi(s,a)) P = ( μ , ϕ ) = arg max ( μ , ϕ ) ∈ M E D ln ( μ ( s ′ ) ⊤ ϕ ( s , a )) .
Soft Violation via Cumulative Primal-Dual Control
We first study low-rank CMDPs under the soft cumulative-violation metric, where constraint deficits and surpluses are aggregated before taking the positive part. The learner must identify an unknown low-rank transition representation while updating policies through a primal-dual procedure: return-policy data may lack action coverage, whereas always-uniform actions are less policy-adaptive. REP-PD-TEG (**Representation-learning Primal-Dual with Tertiary decaying-ϵ \epsilon ϵ -Greedy probing) resolves this tension by keeping the probing kernel close to the previous return policy while retaining a uniform action floor; the tertiary transition supplies the coverage needed for optimism. Unless stated otherwise, regret and soft violation are evaluated on the return policies { π k } k = 1 K \{\pi_k\}_{k=1}^{K} { π k } k = 1 K ; auxiliary tuples are used only for model estimation and bonus construction. We later quantify the cost of charging the execution policy.
REP-PD-TEG Algorithm
Algorithm 1 gives REP-PD-TEG. At episode k k k , the algorithm interacts with the environment using the execution policy q k ( ⋅ ∣ s ) = q k ′ ( ⋅ ∣ s ) = ( 1 − ϵ k ) π k − 1 ( ⋅ ∣ s ) + ϵ k U ( A ) q_k(\cdot\mid s)=q'_k(\cdot\mid s)=(1-\epsilon_k)\pi_{k-1}(\cdot\mid s)+\epsilon_k U(\mathcal{A}) q k ( ⋅ ∣ s ) = q k ′ ( ⋅ ∣ s ) = ( 1 − ϵ k ) π k − 1 ( ⋅ ∣ s ) + ϵ k U ( A ) , where U ( A ) U(\mathcal{A}) U ( A ) is the uniform distribution over the action space and ϵ 1 ≥ ϵ 2 ≥ ⋯ ≥ ϵ K > 0 \epsilon_1\geq\epsilon_2\geq\cdots\geq\epsilon_K>0 ϵ 1 ≥ ϵ 2 ≥ ⋯ ≥ ϵ K > 0 . The oracle returns one tertiary tuple ( s , a , s ′ , a ′ , s ′ ′ , a ′ ′ , s ˉ ) (s,a,s',a',s'',a'',\bar{s}) ( s , a , s ′ , a ′ , s ′′ , a ′′ , s ˉ ) : it first samples s ∼ d π k − 1 P ⋆ s\sim d_{\pi_{k-1}}^{P^{\star}} s ∼ d π k − 1 P ⋆ , then alternates actions from q k q_k q k , q k ′ q'_k q k ′ , q k ′ ′ q''_k q k ′′ with transitions from P ⋆ P^{\star} P ⋆ . The resulting three transitions are stored in separate replay buffers denoted as D k D_k D k , D k ′ D'_k D k ′ , and D k ′ ′ D''_k D k ′′ . After updating the replay buffers, REP-PD-TEG fits P ^ k = ( μ ^ k , ϕ ^ k ) \widehat{P}_k=(\widehat{\mu}_k,\widehat{\phi}_k) P k = ( μ k , ϕ k ) by MLE on D k + D k ′ + D k ′ ′ D_k+D'_k+D''_k D k + D k ′ + D k ′′ . The third stream is required by the present merged-covariance design, not by MLE alone. Since the covariance contains both D k D_k D k and D k ′ D'_k D k ′ , the one-step-back proof produces propagated quadratic terms under both primary and secondary designs: D k ′ D'_k D k ′ controls the former, while the one-step pushforward D k ′ ′ D''_k D k ′′ supplies an MLE-controlled denominator for the latter. We therefore do not place D k ′ ′ D''_k D k ′′ in the covariance, which would create another propagated level. The learned features are then utilized to calculate the empirical covariance matrix Σ ^ k m i x \widehat{\Sigma}^{\mathrm{mix}}_k Σ k mix , which quantifies the uncertainty associated with these features. Based on the empirical covariance matrix, an exploration bonus, b ^ k ( x , u ) \widehat{b}_k(x,u) b k ( x , u ) , is constructed as a scaled Mahalanobis norm of the learned feature ϕ ^ k \widehat{\phi}_k ϕ k with respect to Σ ^ k m i x \widehat{\Sigma}^{\mathrm{mix}}_k Σ k mix . This bonus induces uncertainty-aware planning under the learned model by augmenting both the reward and utility value estimates, after which the return policy and dual variable are updated according to the final two steps of Algorithm 1.
Algorithm 1 REP-PD-TEG
1: Input: Parameters $\{\lambda_k\}_{k=1}^{K}$, $\{\alpha_k\}_{k=1}^{K}$, $\eta$, $\xi$, $\{\epsilon_k\}_{k=1}^{K}$, model class $\mathcal{M}=\{(\mu,\phi):\mu\in\Theta,\phi\in\Gamma\}$, number of episodes $K$.
2: Initialize return policy $\pi_0(\cdot\mid s)=U(\mathcal{A})$, $Y_1=0$, and $D_0=D'_0=D''_0=\varnothing$.
3: for $k=1,\ldots,K$ do
4: Set execution policy $q_k=q'_k=q''_k=(1-\epsilon_k)\pi_{k-1}+\epsilon_kU(\mathcal{A})$.
5: Query one auxiliary tertiary tuple $(s,a,s',a',s'',a'',\bar{s})$, where $s\sim d^{\pi_{k-1}}_{P^\star}$, $a\sim q_k(\cdot\mid s)$, $s'\sim P^\star(\cdot\mid s,a)$, $a'\sim q'_k(\cdot\mid s')$, $s''\sim P^\star(\cdot\mid s',a')$, $a''\sim q''_k(\cdot\mid s'')$, $\bar{s}\sim P^\star(\cdot\mid s'',a'')$.
6: Update $D_k=D_{k-1}\cup\{(s,a,s')\}$, $D'_k=D'_{k-1}\cup\{(s',a',s'')\}$, $D''_k=D''_{k-1}\cup\{(s'',a'',\bar{s})\}$.
7: Fit the low-rank transition model by MLE:
$$
\widehat{P}_k=(\widehat{\mu}_k,\widehat{\phi}_k)\in\underset{(\mu,\phi)\in\mathcal{M}}{\arg\max}\ \mathbb{E}_{D_k+D'_k+D''_k}\left[\log\mu(x')^\top\phi(x,u)\right].
$$
8: Form the merged empirical covariance
$$
\widehat{\Sigma}^{\mathrm{mix}}_k=
\sum_{(x,u,x')\in D_k}\widehat{\phi}_k(x,u)\widehat{\phi}_k(x,u)^\top+
\sum_{(x,u,x')\in D'_k}\widehat{\phi}_k(x,u)\widehat{\phi}_k(x,u)^\top+
\lambda_k I.
$$
9: Set the covariance mixture bonus
$$
\widehat{b}_k(x,u)=\min\left\{\alpha_k\sqrt{\widehat{\phi}_k(x,u)^\top(\widehat{\Sigma}^{\mathrm{mix}}_k)^{-1}\widehat{\phi}_k(x,u)},\bar{b}\right\}.
$$
10: Update the return policy $\pi_k\in\arg\max_\pi\left(Q^{\pi}_{\widehat{P}_k,r+\widehat{b}_k}+Y_kQ^{\pi}_{\widehat{P}_k,g+\widehat{b}_k}\right)$.
11: Update the dual variable $Y_{k+1}=\Pi_{[0,\xi]}\left(Y_k+\eta\left(b-V^{\pi_k}_{\widehat{P}_k,g+\widehat{b}_k}(s_1)\right)\right)$.
12: end for
13: Return $\pi_1,\ldots,\pi_K$.
The dual variable is projected into [ 0 , ξ ] [0,\xi] [ 0 , ξ ] and we set ξ = 2 / θ \xi=2/\theta ξ = 2/ θ in the analysis. Reward optimism promotes exploration, while utility optimism is valid for soft violation because episode-level deficits and surpluses are aggregated. Thus one episode consists of auxiliary tertiary sampling, representation learning, bonus construction, optimistic primal update, and projected dual update.
Algorithm 1 uses exact MLE and exact optimistic-Lagrangian planning to isolate the statistical challenge of learning unknown representations. The proof exposes a conditional interface for other learners: a representation method may replace MLE if it establishes the same prediction event under all three probing designs, with any excess prediction error added to the representation-error term. Likewise, full-planning suboptimality enters through the cumulative planning term. A raw MLE objective gap need not imply the required prediction event, so the present theorems do not claim guarantees for arbitrary approximate oracles, misspecified models, or infinite classes. Spectral representation learning [37 ] is an example of a non-MLE alternative, but its three-design event would have to be verified separately here.
Polynomial-Schedule Guarantees
We state the guarantee for polynomial schedules ϵ k = k − p \epsilon_k=k^{-p} ϵ k = k − p , which make the exploration–deployment tradeoff explicit. The return-policy bound captures the statistical cost of learning the representation from auxiliary probing data, while the execution-policy statement adds the standard performance-difference cost of actually deploying the mixed probing policy.
Theorem 3.1 (Return policy and execution policy guarantees for polynomial probing). *Suppose Assumptions 2.3 and 2.5 hold and the finite model class is realizable, ∣ M ∣ < ∞ |\mathcal{M}|<\infty ∣ M ∣ < ∞ and P ⋆ ∈ M \mathcal{P}^\star\in\mathcal{M} P ⋆ ∈ M . Let ϵ k = k − p \epsilon_k=k^{-p} ϵ k = k − p for some p ∈ [ 0 , 1 / 2 ) p\in[0,1/2) p ∈ [ 0 , 1/2 ) . Run Algorithm 1 with α k = O ~ ( ( ∣ A ∣ k p + d 2 ) γ log ( ∣ M ∣ k / δ ) ) \alpha_k=\widetilde{O}\left(\sqrt{(|\mathcal{A}|k^p+d^2)\gamma\log(|\mathcal{M}|k/\delta)}\right) α k = O ( ( ∣ A ∣ k p + d 2 ) γ log ( ∣ M ∣ k / δ ) ) , λ k = Θ ~ ( ( ( ∣ A ∣ k p + d 2 ) / d ) log ( ∣ M ∣ k / δ ) ) \lambda_k=\widetilde{\Theta}\left(((|\mathcal{A}|k^p+d^2)/d)\log(|\mathcal{M}|k/\delta)\right) λ k = Θ ( (( ∣ A ∣ k p + d 2 ) / d ) log ( ∣ M ∣ k / δ ) ) , b ˉ ≥ max k ≤ K α k / λ k \bar{b}\geq\max_{k\leq K}\alpha_k/\sqrt{\lambda_k} b ˉ ≥ max k ≤ K α k / λ k , η = 2 ( 1 − γ ) / ( θ b ˉ K ) \eta=2(1-\gamma)/(\theta\bar{b}\sqrt{K}) η = 2 ( 1 − γ ) / ( θ b ˉ K ) , ξ = 2 / θ \xi=2/\theta ξ = 2/ θ . Then, with probability at least 1 − δ 1-\delta 1 − δ ,
Regret r e t u r n ( K ) ≤ O ~ ( ∣ A ∣ b ˉ d 2 K 1 / 2 + p θ ( 1 − γ ) 2 ) , Violation s o f t r e t u r n ( K ) ≤ O ~ ( ∣ A ∣ b ˉ d 2 K 1 / 2 + p ( 1 − γ ) 2 ) . \operatorname{Regret}^{\mathrm{return}}(K)\leq\widetilde{O}\left(\frac{|\mathcal{A}|\bar{b}d^2K^{1/2+p}}{\theta(1-\gamma)^2}\right),\qquad
\operatorname{Violation}^{\mathrm{return}}_{\mathrm{soft}}(K)\leq\widetilde{O}\left(\frac{|\mathcal{A}|\bar{b}d^2K^{1/2+p}}{(1-\gamma)^2}\right). Regret return ( K ) ≤ O ( θ ( 1 − γ ) 2 ∣ A ∣ b ˉ d 2 K 1/2 + p ) , Violation soft return ( K ) ≤ O ( ( 1 − γ ) 2 ∣ A ∣ b ˉ d 2 K 1/2 + p ) .
Moreover, for the execution policy π ~ k = ( 1 − ϵ k ) π k + ϵ k U ( A ) \widetilde{\pi}_k=(1-\epsilon_k)\pi_k+\epsilon_k U(\mathcal{A}) π k = ( 1 − ϵ k ) π k + ϵ k U ( A ) ,
Regret e x e c ( K ) ≤ Regret r e t u r n ( K ) + 2 1 − γ ∑ k = 1 K k − p , \operatorname{Regret}^{\mathrm{exec}}(K)\leq\operatorname{Regret}^{\mathrm{return}}(K)+\frac{2}{1-\gamma}\sum_{k=1}^{K}k^{-p}, Regret exec ( K ) ≤ Regret return ( K ) + 1 − γ 2 k = 1 ∑ K k − p ,
Violation s o f t e x e c ( K ) ≤ Violation s o f t r e t u r n ( K ) + 2 1 − γ ∑ k = 1 K k − p . \operatorname{Violation}^{\mathrm{exec}}_{\mathrm{soft}}(K)\leq\operatorname{Violation}^{\mathrm{return}}_{\mathrm{soft}}(K)+\frac{2}{1-\gamma}\sum_{k=1}^{K}k^{-p}. Violation soft exec ( K ) ≤ Violation soft return ( K ) + 1 − γ 2 k = 1 ∑ K k − p .
The proof is given in Appendix D. Generally, our analyses combine a tuple-level MLE event, a pathwise-to-averaged bridge, tertiary-aware optimism for the merged bonus, and a cumulative overestimation bound for the dual update. The schedule parameter p p p trades coverage for deployment cost: smaller p p p improves the return-policy rate, while larger p p p reduces the cost of executing the exploratory mixture. Since ∑ k = 1 K k − p ≤ K 1 − p / ( 1 − p ) \sum_{k=1}^{K}k^{-p}\leq K^{1-p}/(1-p) ∑ k = 1 K k − p ≤ K 1 − p / ( 1 − p ) , the return-policy term scales as O ~ ( K 1 / 2 + p ) \widetilde{O}(K^{1/2+p}) O ( K 1/2 + p ) , whereas execution adds O ( K 1 − p / ( 1 − γ ) ) O(K^{1-p}/(1-\gamma)) O ( K 1 − p / ( 1 − γ )) .
Corollary 3.2 (Best return policy and execution policy rates). Under the same setting as Theorem 3.1, the following two specializations hold.
First, if p = 0 p=0 p = 0 , equivalently ϵ k ≡ 1 \epsilon_k\equiv1 ϵ k ≡ 1 , then the probing kernel in Algorithm 1 becomes uniform:
q k = q k ′ = q k ′ ′ = U ( A ) . q_k=q'_k=q''_k=U(\mathcal{A}). q k = q k ′ = q k ′′ = U ( A ) . The return policies satisfy
Regret r e t u r n ( K ) ≤ O ~ ( ∣ A ∣ b ˉ d θ 2 K θ ( 1 − γ ) 2 ) , Violation s o f t r e t u r n ( K ) ≤ O ~ ( ∣ A ∣ b ˉ d θ 2 K ( 1 − γ ) 2 ) . \operatorname{Regret}^{\mathrm{return}}(K)\leq\widetilde{O}\left(\frac{|\mathcal{A}|\bar{b}d_\theta^2\sqrt{K}}{\theta(1-\gamma)^2}\right),\qquad
\operatorname{Violation}^{\mathrm{return}}_{\mathrm{soft}}(K)\leq\widetilde{O}\left(\frac{|\mathcal{A}|\bar{b}d_\theta^2\sqrt{K}}{(1-\gamma)^2}\right). Regret return ( K ) ≤ O ( θ ( 1 − γ ) 2 ∣ A ∣ b ˉ d θ 2 K ) , Violation soft return ( K ) ≤ O ( ( 1 − γ ) 2 ∣ A ∣ b ˉ d θ 2 K ) . This is the best return policy specialization, and reduces to uniform tertiary probing.
Second, if p = 1 / 4 p=1/4 p = 1/4 , then the return policies satisfy
Regret r e t u r n ( K ) ≤ O ~ ( ∣ A ∣ b ˉ d θ 2 K 3 / 4 θ ( 1 − γ ) 2 ) , Violation s o f t r e t u r n ( K ) ≤ O ~ ( ∣ A ∣ b ˉ d θ 2 K 3 / 4 ( 1 − γ ) 2 ) . \operatorname{Regret}^{\mathrm{return}}(K)\leq\widetilde{O}\left(\frac{|\mathcal{A}|\bar{b}d_\theta^2K^{3/4}}{\theta(1-\gamma)^2}\right),\qquad
\operatorname{Violation}^{\mathrm{return}}_{\mathrm{soft}}(K)\leq\widetilde{O}\left(\frac{|\mathcal{A}|\bar{b}d_\theta^2K^{3/4}}{(1-\gamma)^2}\right). Regret return ( K ) ≤ O ( θ ( 1 − γ ) 2 ∣ A ∣ b ˉ d θ 2 K 3/4 ) , Violation soft return ( K ) ≤ O ( ( 1 − γ ) 2 ∣ A ∣ b ˉ d θ 2 K 3/4 ) . The execution policy gap is also O ( K 3 / 4 / ( 1 − γ ) ) O(K^{3/4}/(1-\gamma)) O ( K 3/4 / ( 1 − γ )) , and the overall sample complexities of the execution policies become
Regret e x e c ( K ) ≤ O ~ ( K 3 / 4 ) , Violation s o f t e x e c ( K ) ≤ O ~ ( K 3 / 4 ) \operatorname{Regret}^{\mathrm{exec}}(K)\leq\widetilde{O}(K^{3/4}),\qquad
\operatorname{Violation}^{\mathrm{exec}}_{\mathrm{soft}}(K)\leq\widetilde{O}(K^{3/4}) Regret exec ( K ) ≤ O ( K 3/4 ) , Violation soft exec ( K ) ≤ O ( K 3/4 ) up to logarithmic and problem-dependent factors.
Corollary 3.2 highlights the schedule tradeoff: p = 0 p=0 p = 0 is statistically sharp for return policies, while p = 1 / 4 p=1/4 p = 1/4 balances representation learning with the cost of using the execution policy. Thus the return-policy theorem isolates representation-learning cost, and the execution policy corollary quantifies the extra price of deploying the probes as real behavior.
Hard Violation via Per-Episode Certification
We now consider the hard cumulative-violation metric. Unlike soft violation, hard violation does not allow constraint surplus in one episode to offset constraint deficit in another. Therefore the optimistic utility estimate used by the soft primal-dual update is insufficient for certifying feasibility at the episode level: an optimistic estimate may exceed the threshold b b b even when the true utility remains below it. REP-PD-H-TEG (the Hard-violation version of REP-PD-TEG) therefore keeps the same representation-learning procedure but changes the policy decision: policy search remains optimistic, while a tight feasibility test on the lower-confidence utility of the policy is explicitly conducted.
Lower-Certificate Planning in REP-PD-H-TEG
As shown in Algorithm 2 in Appendix F, REP-PD-H-TEG uses the same tertiary data collection, MLE representation update, merged covariance, and bonus construction as REP-PD-TEG. The hard-specific change lies in how the policy is selected after the bonus is built. In REP-PD-TEG, the optimistic utility value can be used in the dual update because violation is aggregated across episodes. In the hard violation setting, however, the policy executed in the current episode must be certified feasible on its own. Thus REP-PD-H-TEG introduces a lower-confidence utility certificate. Write Γ k : = ∣ A ∣ / ϵ k \Gamma_k := |\mathcal{A}|/\epsilon_k Γ k := ∣ A ∣/ ϵ k and ζ k : = ln ( ∣ M ∣ k / δ ) / k \zeta_k := \ln(|\mathcal{M}|k/\delta)/k ζ k := ln ( ∣ M ∣ k / δ ) / k , and choose a sufficiently large universal calibration constant C c e r t C_{\mathrm{cert}} C cert . The one-sided statistical correction and the certificate are
ℓ k : = C c e r t Γ k ζ k 1 − γ , U k ( π ) : = [ V P ^ k , g − b ^ k π ( s 1 ) − ℓ k ] + , \ell_k := C_{\mathrm{cert}}\sqrt{\frac{\Gamma_k\zeta_k}{1-\gamma}}, \qquad\mathcal{U}_k(\pi) := \left[V^{\pi}_{\widehat{P}_k,g-\widehat{b}_k}(s_1)-\ell_k\right]_+, ℓ k := C cert 1 − γ Γ k ζ k , U k ( π ) := [ V P k , g − b k π ( s 1 ) − ℓ k ] + ,
where [ x ] + = max { x , 0 } [x]_+ = \max\{x,0\} [ x ] + = max { x , 0 } . On the representation-learning good event, the one-step-back argument gives, simultaneously for every queried policy,
U k ( π ) ≤ V P ⋆ , g π ( s 1 ) . \mathcal{U}_k(\pi) \le V^{\pi}_{P^\star,g}(s_1). U k ( π ) ≤ V P ⋆ , g π ( s 1 ) .
Hence U k ( π ) ≥ b \mathcal{U}_k(\pi) \ge b U k ( π ) ≥ b implies true feasibility. The corresponding two-sided width is needed only in the analysis of the selected policy and is
w k ( π ) : = 2 ℓ k + 2 V P ^ k , b ^ k π ( s 1 ) , V P ⋆ , g π ( s 1 ) ≤ U k ( π ) + w k ( π ) . w_k(\pi) := 2\ell_k + 2V^{\pi}_{\widehat{P}_k,\widehat{b}_k}(s_1), \qquad V^{\pi}_{P^\star,g}(s_1) \le\mathcal{U}_k(\pi) + w_k(\pi). w k ( π ) := 2 ℓ k + 2 V P k , b k π ( s 1 ) , V P ⋆ , g π ( s 1 ) ≤ U k ( π ) + w k ( π ) .
Separating ℓ k \ell_k ℓ k from w k ( π ) w_k(\pi) w k ( π ) is essential: the former makes the stopping rule conservative, while the latter accounts for the pessimistic bonus and is summed only along the returned policy sequence.
Instead of choosing a single policy at the current dual variable, REP-PD-H-TEG performs a capped dual search within the episode in the planning step. For each queried dual level Y Y Y , it solves the optimistic Lagrangian planning problem
π ^ k , Y ∈ arg max π ( Q P ^ k , r + b ^ k π + Y Q P ^ k , g + b ^ k π ) , \widehat{\pi}_{k,Y} \in\mathop{\arg\max}_{\pi}\left(Q^{\pi}_{\widehat{P}_k,r+\widehat{b}_k}+YQ^{\pi}_{\widehat{P}_k,g+\widehat{b}_k}\right), π k , Y ∈ arg max π ( Q P k , r + b k π + Y Q P k , g + b k π ) ,
and then forms a statewise softmax candidate from the corresponding optimistic reward-plus-utility scores. The bonus makes planning optimistic and uncertainty-aware, exactly as in REP-PD-TEG; the stopping rule is different. REP-PD-H-TEG deploys the first candidate whose lower-confidence certificate satisfies U k ( π k , Y ) ≥ b \mathcal{U}_k(\pi_{k,Y}) \ge b U k ( π k , Y ) ≥ b . If no candidate is certified before the cap Y = K 1 / 4 Y=K^{1/4} Y = K 1/4 , the algorithm deploys the final capped-dual candidate. Thus optimistic bonus-augmented values are used to search for good policies, while the pessimistic bonus-based certificate decides whether a candidate is safe enough to execute.
Capped certificate search in one outer round.
For each grid level Y ≤ Y max Y \le Y_{\max} Y ≤ Y m a x , compute the optimistic candidate π k , Y \pi_{k,Y} π k , Y .
Evaluate U k ( π k , Y ) \mathcal{U}_k(\pi_{k,Y}) U k ( π k , Y ) ; if it is at least b b b , deploy π k , Y \pi_{k,Y} π k , Y and stop.
Otherwise increase Y Y Y and repeat the optimistic-plan/certificate test.
If no earlier candidate passes, deploy the candidate at Y max Y_{\max} Y m a x .
Hard-Violation Guarantees
We state the sample complexity result of REP-PD-H-TEG in this section. The proof decomposes into a front-end regret bound, a lower-certificate reduction, and a Slater-induced search-success argument; the full proof is given in Appendices F and G.
Table 1 (PDF p. 9).
The positive score gap is not used to learn the representation or to establish the lower-certificate inequality; it is used by the current hard proof to control the softmax approximation along the capped dual search.
Theorem 4.1 (Return policy and execution policy hard guarantees for REP-PD-H-TEG). Suppose Assumptions 2.3, 2.5, and C.1 hold, the finite model class is realizable (∣ M ∣ < ∞ |\mathcal{M}| < \infty ∣ M ∣ < ∞ , P ⋆ ∈ M P^\star\in\mathcal{M} P ⋆ ∈ M ), and fix δ ∈ ( 0 , 1 ) \delta\in(0,1) δ ∈ ( 0 , 1 ) . Run Algorithm 2 with ϵ k = k − p \epsilon_k = k^{-p} ϵ k = k − p for some p ∈ [ 0 , 1 / 2 ) p \in[0,1/2) p ∈ [ 0 , 1/2 ) , and set α k = Θ ~ ( ( ∣ A ∣ k p + d 2 ) γ log ( ∣ M ∣ k / δ ) ) \alpha_k = \widetilde{\Theta}\left(\sqrt{(|\mathcal{A}|k^p+d^2)\gamma\log(|\mathcal{M}|k/\delta)}\right) α k = Θ ( ( ∣ A ∣ k p + d 2 ) γ log ( ∣ M ∣ k / δ ) ) with a sufficiently large leading constant, $\lambda_k = \widetilde{\Theta}((|\mathcal{A}|k^p +
( d 2 ) / d ) log ( ∣ M ∣ k / δ ) ) (d^2)/d)\log(|\mathcal{M}|k/\delta)) ( d 2 ) / d ) log ( ∣ M ∣ k / δ )) , b ˉ ≥ max k ≤ K α k / λ k \bar{b} \ge\max_{k \le K} \alpha_k/\sqrt{\lambda_k} b ˉ ≥ max k ≤ K α k / λ k , ℓ k = C c e r t Γ k ζ k / ( 1 − γ ) \ell_k = C_{\mathrm{cert}}\sqrt{\Gamma_k}\zeta_k/(1-\gamma) ℓ k = C cert Γ k ζ k / ( 1 − γ ) for a sufficiently large universal C c e r t C_{\mathrm{cert}} C cert , Y max = K 1 / 4 Y_{\max}=K^{1/4} Y m a x = K 1/4 , τ h = Θ ( 1 / ( ( 1 − γ ) K log ∣ A ∣ ) ) \tau_h=\Theta(1/((1-\gamma)\sqrt{K}\log|\mathcal{A}|)) τ h = Θ ( 1/ (( 1 − γ ) K log ∣ A ∣ )) , η h = Θ ( τ h ( 1 − γ ) / K 3 / 4 ) \eta_h=\Theta(\tau_h(1-\gamma)/K^{3/4}) η h = Θ ( τ h ( 1 − γ ) / K 3/4 ) . Then, with probability at least 1 − δ 1-\delta 1 − δ , the return policies satisfy
Regret r e t u r n ( K ) ≤ O ~ ( [ ∣ A ∣ b ˉ d 2 ( 1 − γ ) 2 + ∣ A ∣ 1 − γ ] K 3 / 4 + p ) , \operatorname{Regret}^{\mathrm{return}}(K) \le\widetilde{O}\left(\left[\frac{|\mathcal{A}|\bar{b}d^2}{(1-\gamma)^2}+\sqrt{\frac{|\mathcal{A}|}{1-\gamma}}\right]K^{3/4+p}\right), Regret return ( K ) ≤ O ( [ ( 1 − γ ) 2 ∣ A ∣ b ˉ d 2 + 1 − γ ∣ A ∣ ] K 3/4 + p ) ,
Violation h a r d r e t u r n ( K ) ≤ O ~ ( 1 θ [ ∣ A ∣ b ˉ d 2 ( 1 − γ ) 2 + ∣ A ∣ 1 − γ ] K 1 / 2 + p ) + O ( θ − 4 ) . \operatorname{Violation}^{\mathrm{return}}_{\mathrm{hard}}(K) \le\widetilde{O}\left(\frac{1}{\theta}\left[\frac{|\mathcal{A}|\bar{b}d^2}{(1-\gamma)^2}+\sqrt{\frac{|\mathcal{A}|}{1-\gamma}}\right]K^{1/2+p}\right)+O(\theta^{-4}). Violation hard return ( K ) ≤ O ( θ 1 [ ( 1 − γ ) 2 ∣ A ∣ b ˉ d 2 + 1 − γ ∣ A ∣ ] K 1/2 + p ) + O ( θ − 4 ) .
The hard-violation bound is sublinear for p < 1 / 2 p<1/2 p < 1/2 , while the regret bound is sublinear only when p < 1 / 4 p<1/4 p < 1/4 . Moreover, for the execution policy π ~ k = ( 1 − ϵ k ) π k + ϵ k U ( A ) \tilde{\pi}_k=(1-\epsilon_k)\pi_k+\epsilon_k U(\mathcal{A}) π ~ k = ( 1 − ϵ k ) π k + ϵ k U ( A ) , we have Regret e x e c ( K ) ≤ Regret r e t u r n ( K ) + 2 1 − γ ∑ k = 1 K k − p \operatorname{Regret}^{\mathrm{exec}}(K)\le\operatorname{Regret}^{\mathrm{return}}(K)+\frac{2}{1-\gamma}\sum_{k=1}^{K}k^{-p} Regret exec ( K ) ≤ Regret return ( K ) + 1 − γ 2 ∑ k = 1 K k − p and Violation h a r d e x e c ( K ) ≤ Violation h a r d r e t u r n ( K ) + 2 1 − γ ∑ k = 1 K k − p \operatorname{Violation}^{\mathrm{exec}}_{\mathrm{hard}}(K)\le\operatorname{Violation}^{\mathrm{return}}_{\mathrm{hard}}(K)+\frac{2}{1-\gamma}\sum_{k=1}^{K}k^{-p} Violation hard exec ( K ) ≤ Violation hard return ( K ) + 1 − γ 2 ∑ k = 1 K k − p .
The K 1 / 2 + p K^{1/2+p} K 1/2 + p hard-violation term is the representation-learning price, and the extra K 1 / 4 K^{1/4} K 1/4 in regret comes from capped in-episode dual search; the execution policy bounds follow similar performance-difference argument from Section 3.
Corollary 4.2 (Best return policy and execution policy hard rates). Under the same setting as Theorem 4.1, the following two specializations hold.
First , if p = 0 p=0 p = 0 , equivalently ϵ k ≡ 1 \epsilon_k\equiv1 ϵ k ≡ 1 , then REP-PD-H-TEG uses q k = q k ′ = q k ′ ′ = U ( A ) q_k=q'_k=q''_k=U(\mathcal{A}) q k = q k ′ = q k ′′ = U ( A ) and the return policies satisfy
Regret r e t u r n ( K ) ≤ O ~ ( [ ∣ A ∣ b ˉ d 2 ( 1 − γ ) 2 + ∣ A ∣ 1 − γ ] K 3 / 4 ) , \operatorname{Regret}^{\mathrm{return}}(K) \le\widetilde{O}\left(\left[\frac{|\mathcal{A}|\bar{b}d^2}{(1-\gamma)^2}+\sqrt{\frac{|\mathcal{A}|}{1-\gamma}}\right]K^{3/4}\right), Regret return ( K ) ≤ O ( [ ( 1 − γ ) 2 ∣ A ∣ b ˉ d 2 + 1 − γ ∣ A ∣ ] K 3/4 ) , Violation h a r d r e t u r n ( K ) ≤ O ~ ( 1 θ [ ∣ A ∣ b ˉ d 2 ( 1 − γ ) 2 + ∣ A ∣ 1 − γ ] K ) + O ( θ − 4 ) . \operatorname{Violation}^{\mathrm{return}}_{\mathrm{hard}}(K) \le\widetilde{O}\left(\frac{1}{\theta}\left[\frac{|\mathcal{A}|\bar{b}d^2}{(1-\gamma)^2}+\sqrt{\frac{|\mathcal{A}|}{1-\gamma}}\right]\sqrt{K}\right)+O(\theta^{-4}). Violation hard return ( K ) ≤ O ( θ 1 [ ( 1 − γ ) 2 ∣ A ∣ b ˉ d 2 + 1 − γ ∣ A ∣ ] K ) + O ( θ − 4 ) . Regret r e t u r n ( K ) ≤ O ~ ( [ ∣ A ∣ b ˉ d 2 ( 1 − γ ) 2 + ∣ A ∣ 1 − γ ] K 7 / 8 ) , \operatorname{Regret}^{\mathrm{return}}(K) \le\widetilde{O}\left(\left[\frac{|\mathcal{A}|\bar{b}d^2}{(1-\gamma)^2}+\sqrt{\frac{|\mathcal{A}|}{1-\gamma}}\right]K^{7/8}\right), Regret return ( K ) ≤ O ( [ ( 1 − γ ) 2 ∣ A ∣ b ˉ d 2 + 1 − γ ∣ A ∣ ] K 7/8 ) , Violation h a r d r e t u r n ( K ) ≤ O ~ ( 1 θ [ ∣ A ∣ b ˉ d 2 ( 1 − γ ) 2 + ∣ A ∣ 1 − γ ] K 5 / 8 ) + O ( θ − 4 ) . \operatorname{Violation}^{\mathrm{return}}_{\mathrm{hard}}(K) \le\widetilde{O}\left(\frac{1}{\theta}\left[\frac{|\mathcal{A}|\bar{b}d^2}{(1-\gamma)^2}+\sqrt{\frac{|\mathcal{A}|}{1-\gamma}}\right]K^{5/8}\right)+O(\theta^{-4}). Violation hard return ( K ) ≤ O ( θ 1 [ ( 1 − γ ) 2 ∣ A ∣ b ˉ d 2 + 1 − γ ∣ A ∣ ] K 5/8 ) + O ( θ − 4 ) . For p = 1 / 8 p=1/8 p = 1/8 , the execution-policy gap is O ( K 7 / 8 / ( 1 − γ ) ) O(K^{7/8}/(1-\gamma)) O ( K 7/8 / ( 1 − γ )) ; hence both execution-policy regret and hard violation are O ~ ( K 7 / 8 ) \widetilde{O}(K^{7/8}) O ( K 7/8 ) up to problem-dependent factors, with exploratory actions determining the balanced rate.
Thus p = 0 p=0 p = 0 is sharp for return policies, whereas p = 1 / 8 p=1/8 p = 1/8 balances the hard-regret front end K 3 / 4 + p K^{3/4+p} K 3/4 + p with the execution gap K 1 − p K^{1-p} K 1 − p .
Conclusion
We develop representation-learning algorithms for low-rank continuous-state CMDPs with unknown transition features. REP-PD-TEG combines auxiliary probing, covariance bonuses, and optimistic primal-dual planning for soft-violation control; REP-PD-H-TEG adds a lower-confidence utility certificate and stabilized dual search for hard-violation control. Both provide return- and execution-policy guarantees.
Limitations. The theorems assume finite realizability and exact MLE/planning; alternative learners require the same three-design prediction guarantee. The hard result additionally uses a positive optimistic-score gap. The occupancy oracle is implementable with resets and geometric roll-ins, but our bounds do not charge full roll-in trajectories. Finally, the experiments illustrate the mechanism rather than exhaustively evaluating hard certification.
Acknowledgments and Disclosure of Funding
Funding. There was no funding in direct support of this work.
Competing interests. The authors declare no competing interests.
Appendix Roadmap
The appendix is organized as follows. Appendix A fixes notation and records the basic covariance concentration tool. Appendix B collects the MLE, elliptical-potential, simulation, and primal-dual lemmas used throughout the proofs. Appendix C states the technical ingredients specific to the hard-violation analysis. Appendix D proves the tertiary-aware decaying-ϵ \epsilon ϵ representation-learning result for REP-PD-TEG, while Appendix E gives the p = 0 p=0 p = 0 uniform specialization. Appendix F derives the one-sided hard certificate and proves the decaying-schedule result, while Appendix G records its p = 0 p=0 p = 0 specialization. Appendix H contains the experimental details.
Notation and Auxiliary Concentration
This appendix fixes the occupancy notation used throughout the proof and recalls the empirical-to-population covariance comparison that underlies the uncertainty bonus. Later appendices replace the uniform probing distribution below by the tertiary decaying-ϵ \epsilon ϵ design, but the same comparison principle is repeatedly used.
We use d π ∗ π d_{\pi_*}^{\pi} d π ∗ π for the normalized discounted state occupancy and d π ∗ π ( s , a ) = d π ∗ π ( s ) π ( a ∣ s ) d_{\pi_*}^{\pi}(s,a)=d_{\pi_*}^{\pi}(s)\pi(a\mid s) d π ∗ π ( s , a ) = d π ∗ π ( s ) π ( a ∣ s ) for the state-action occupancy. For the uniform probing baseline, let ρ k ( s ) = k − 1 ∑ i = 0 k − 1 d π ∗ π i ( s ) \rho_k(s)=k^{-1}\sum_{i=0}^{k-1}d_{\pi_*}^{\pi_i}(s) ρ k ( s ) = k − 1 ∑ i = 0 k − 1 d π ∗ π i ( s ) , and let ρ k ′ \rho'_k ρ k ′ be the marginal distribution of s ′ s' s ′ generated by first drawing s ∼ ρ k s\sim\rho_k s ∼ ρ k , then a ∼ U ( A ) a\sim U(\mathcal{A}) a ∼ U ( A ) , and finally s ′ ∼ P ∗ ( ⋅ ∣ s , a ) s'\sim P^*(\cdot\mid s,a) s ′ ∼ P ∗ ( ⋅ ∣ s , a ) . For a feature map ϕ \phi ϕ , define
Σ ρ k × U ( A ) , ϕ : = k E s ∼ ρ k , a ∼ U ( A ) [ ϕ ( s , a ) ϕ ( s , a ) ⊤ ] + λ k I . \Sigma_{\rho_k\times U(\mathcal{A}),\phi}:=k\mathbb{E}_{s\sim\rho_k,a\sim U(\mathcal{A})}[\phi(s,a)\phi(s,a)^\top]+\lambda_k I. Σ ρ k × U ( A ) , ϕ := k E s ∼ ρ k , a ∼ U ( A ) [ ϕ ( s , a ) ϕ ( s , a ) ⊤ ] + λ k I .
Lemma A.1 (Concentration of the bonus term). Let
Σ ^ k , ϕ : = ∑ i = 0 k − 1 ϕ ( s ( i ) , a ( i ) ) ϕ ( s ( i ) , a ( i ) ) ⊤ + λ k I , λ k = Θ ( d log ( k ∣ Φ ∣ / δ ) ) . \widehat{\Sigma}_{k,\phi}:=\sum_{i=0}^{k-1}\phi(s^{(i)},a^{(i)})\phi(s^{(i)},a^{(i)})^\top+\lambda_k I,\qquad\lambda_k=\Theta(d\log(k|\Phi|/\delta)). Σ k , ϕ := i = 0 ∑ k − 1 ϕ ( s ( i ) , a ( i ) ) ϕ ( s ( i ) , a ( i ) ) ⊤ + λ k I , λ k = Θ ( d log ( k ∣Φ∣/ δ )) . Then, with probability at least 1 − δ 1-\delta 1 − δ , simultaneously for all k ∈ K + k\in\mathbb{K}^+ k ∈ K + , all ϕ ∈ Φ \phi\in\Phi ϕ ∈ Φ , and all ( s , a ) (s,a) ( s , a ) ,
c 1 ∥ ϕ ( s , a ) ∥ Σ ρ k × U ( A ) , ϕ − 1 ≤ ∥ ϕ ( s , a ) ∥ Σ ^ k , ϕ − 1 ≤ c 2 ∥ ϕ ( s , a ) ∥ Σ ρ k × U ( A ) , ϕ − 1 , c_1\|\phi(s,a)\|_{\Sigma_{\rho_k\times U(\mathcal{A}),\phi}^{-1}}\leq\|\phi(s,a)\|_{\widehat{\Sigma}_{k,\phi}^{-1}}\leq c_2\|\phi(s,a)\|_{\Sigma_{\rho_k\times U(\mathcal{A}),\phi}^{-1}}, c 1 ∥ ϕ ( s , a ) ∥ Σ ρ k × U ( A ) , ϕ − 1 ≤ ∥ ϕ ( s , a ) ∥ Σ k , ϕ − 1 ≤ c 2 ∥ ϕ ( s , a ) ∥ Σ ρ k × U ( A ) , ϕ − 1 , where c 1 , c 2 > 0 c_1,c_2>0 c 1 , c 2 > 0 are universal constants.
Proof. This is the standard empirical-to-population covariance comparison for the uniform probing design; see Lemma 11 of Uehara et al. [41 ] . In the repaired analysis we apply the same concentration argument to block increments generated by the predictable auxiliary streams.
Auxiliary Lemmas
This appendix collects standard auxiliary tools used by both the soft and hard analyses. We first recall the MLE guarantee for low-rank transition models, then state the elliptical-potential and simulation lemmas used to aggregate model error and convert value differences into regret and violation bounds. The MLE statement is a direct consequence of Theorem 21 of Agarwal et al. [2 ] , specialized to the notation of our algorithms.
Lemma B.1 (MLE guarantee). For a fixed episode k k k , with probability 1 − δ 1-\delta 1 − δ ,
E s ∼ { 0.5 ρ k + 0.5 ρ k ′ } , a ∼ U ( A ) [ ∥ P ^ k ( ⋅ ∣ s , a ) − P ∗ ( ⋅ ∣ s , a ) ∥ 1 2 ] ≤ ζ , \mathbb{E}_{s\sim\{0.5\rho_k+0.5\rho'_k\},a\sim U(\mathcal{A})}\left[\|\widehat{P}_k(\cdot\mid s,a)-P^*(\cdot\mid s,a)\|_1^2\right]\leq\zeta, E s ∼ { 0.5 ρ k + 0.5 ρ k ′ } , a ∼ U ( A ) [ ∥ P k ( ⋅ ∣ s , a ) − P ∗ ( ⋅ ∣ s , a ) ∥ 1 2 ] ≤ ζ , where ζ : = ln ( ∣ M ∣ / δ ) k \zeta:=\frac{\ln(|\mathcal{M}|/\delta)}{k} ζ := k l n ( ∣ M ∣/ δ ) . As a straightforward corollary, with probability 1 − δ 1-\delta 1 − δ , ∀ k ∈ K + \forall k\in\mathbb{K}^+ ∀ k ∈ K + ,
E s ∼ { 0.5 ρ k + 0.5 ρ k ′ } , a ∼ U ( A ) [ ∥ P ^ k ( ⋅ ∣ s , a ) − P ∗ ( ⋅ ∣ s , a ) ∥ 1 2 ] ≤ 0.5 ζ k , \mathbb{E}_{s\sim\{0.5\rho_k+0.5\rho'_k\},a\sim U(\mathcal{A})}\left[\|\widehat{P}_k(\cdot\mid s,a)-P^*(\cdot\mid s,a)\|_1^2\right]\leq0.5\zeta_k, E s ∼ { 0.5 ρ k + 0.5 ρ k ′ } , a ∼ U ( A ) [ ∥ P k ( ⋅ ∣ s , a ) − P ∗ ( ⋅ ∣ s , a ) ∥ 1 2 ] ≤ 0.5 ζ k , where ζ k : = ln ( ∣ M ∣ k / δ ) k \zeta_k:=\frac{\ln(|\mathcal{M}|k/\delta)}{k} ζ k := k l n ( ∣ M ∣ k / δ ) .
The next two lemmas are the standard determinant-trace inequalities used in the elliptical-potential argument; see Lemma G.2 of Agarwal et al. [2 ] .
Lemma B.2. Consider the following process. For k = 1 , ⋯ , K k = 1, \cdots, K k = 1 , ⋯ , K , M k = M k − 1 + G k M_k = M_{k-1} + G_k M k = M k − 1 + G k with M 0 = λ 0 I M_0 = \lambda_0 I M 0 = λ 0 I and G k G_k G k being a positive semidefinite matrix with eigenvalues upper-bounded by 1. We have that:
2 ln det ( M K ) − 2 ln det ( λ 0 I ) ≥ ∑ k = 1 K Tr ( G k M k − 1 − 1 ) . 2 \ln\det(M_K) - 2 \ln\det(\lambda_0 I) \ge\sum_{k=1}^{K} \operatorname{Tr}(G_k M_{k-1}^{-1}). 2 ln det ( M K ) − 2 ln det ( λ 0 I ) ≥ k = 1 ∑ K Tr ( G k M k − 1 − 1 ) .
Lemma B.3 (Potential function lemma). Suppose Tr ( G k ) ≤ B 2 \operatorname{Tr}(G_k) \le B^2 Tr ( G k ) ≤ B 2 .
2 ln det ( M K ) − 2 ln det ( λ 0 I ) ≤ d ln ( 1 + K B 2 d λ 0 ) . 2 \ln\det(M_K) - 2 \ln\det(\lambda_0 I) \le d \ln\left(1 + \frac{K B^2}{d \lambda_0}\right). 2 ln det ( M K ) − 2 ln det ( λ 0 I ) ≤ d ln ( 1 + d λ 0 K B 2 ) .
Lemma B.4 (Simulation lemma). For any c ∈ { r , g } c \in\{r,g\} c ∈ { r , g } , two CMDPs ( P ′ , c + b ) (P', c+b) ( P ′ , c + b ) and ( P , c ) (P,c) ( P , c ) , and any policy π \pi π , we have
V P ′ , c + b π − V P , c π = 1 1 − γ E d π P ′ [ b ( s , a ) + γ E P ′ Q P , c π ( s ′ , π ) − γ E P Q P , c π ( s ′ , π ) ] . V_{P',c+b}^{\pi} - V_{P,c}^{\pi} = \frac{1}{1-\gamma}\mathbb{E}_{d_{\pi}^{P'}}\left[b(s,a) + \gamma\mathbb{E}_{P'}Q_{P,c}^{\pi}(s',\pi) - \gamma\mathbb{E}_{P}Q_{P,c}^{\pi}(s',\pi)\right]. V P ′ , c + b π − V P , c π = 1 − γ 1 E d π P ′ [ b ( s , a ) + γ E P ′ Q P , c π ( s ′ , π ) − γ E P Q P , c π ( s ′ , π ) ] . V P ′ , c + b π − V P , c π = 1 1 − γ E d π P ′ [ b ( s , a ) + γ E P ′ Q P ′ , c + b π ( s ′ , π ) − γ E P Q P ′ , c + b π ( s ′ , π ) ] . V_{P',c+b}^{\pi} - V_{P,c}^{\pi} = \frac{1}{1-\gamma}\mathbb{E}_{d_{\pi}^{P'}}\left[b(s,a) + \gamma\mathbb{E}_{P'}Q_{P',c+b}^{\pi}(s',\pi) - \gamma\mathbb{E}_{P}Q_{P',c+b}^{\pi}(s',\pi)\right]. V P ′ , c + b π − V P , c π = 1 − γ 1 E d π P ′ [ b ( s , a ) + γ E P ′ Q P ′ , c + b π ( s ′ , π ) − γ E P Q P ′ , c + b π ( s ′ , π ) ] .
Finally, we use the following primal-dual reduction, which is Lemma 9 of Ding et al. [13 ] in our notation.
Lemma B.5. Let Y ∗ Y^* Y ∗ be the optimal dual variable, and C ≥ 2 Y ∗ C \ge2Y^* C ≥ 2 Y ∗ , then, if
V r π ∗ ( s 1 ) − V r π ~ ( s 1 ) + C [ b − V g π ~ ( s 1 ) ] + ≤ δ V_{r}^{\pi^*}(s_1) - V_{r}^{\widetilde{\pi}}(s_1) + C[b - V_{g}^{\widetilde{\pi}}(s_1)]_+ \le\delta V r π ∗ ( s 1 ) − V r π ( s 1 ) + C [ b − V g π ( s 1 ) ] + ≤ δ then
[ b − V g π ~ ( s 1 ) ] + ≤ 2 δ C . [b - V_{g}^{\widetilde{\pi}}(s_1)]_+ \le\frac{2\delta}{C}. [ b − V g π ( s 1 ) ] + ≤ C 2 δ .
Hard-Constraint Technical Lemmas
This appendix collects the hard-constraint technical ingredients that are reused by the lower-certificate analyses in Appendices § 14 and § 13 . The hard-violation guarantees in this draft are stated through lower-confidence certificates and Slater-induced control of cap episodes; no optimistic feasibility check is used as a hard certificate.
Assumption C.1 (Positive action gap). For every episode k k k , queried dual level Y Y Y , and state s s s , the maximizer of the optimistic Lagrangian action score used by Algorithm 2 is separated from every suboptimal action by at least δ min > 0 \delta_{\min} > 0 δ m i n > 0 . The constant is uniform over k k k , Y Y Y , s s s .
Lemma C.2 (Softmax sensitivity). Let π k Y \pi_k^Y π k Y and π k Y + η \pi_k^{Y+\eta} π k Y + η be the softmax policies produced by two neighboring dual levels Y Y Y and Y + η Y+\eta Y + η with temperature τ \tau τ . Then
∣ V g π k Y ( s 1 ) − V g π k Y + η ( s 1 ) ∣ ≤ O ( η τ ( 1 − γ ) ) . \left|V_g^{\pi_k^Y}(s_1) - V_g^{\pi_k^{Y+\eta}}(s_1)\right| \le O\left(\frac{\eta}{\tau(1-\gamma)}\right). V g π k Y ( s 1 ) − V g π k Y + η ( s 1 ) ≤ O ( τ ( 1 − γ ) η ) .
Proof. Let Δ = sup s ∣ V g π k Y ( s ) − V g π k Y + η ( s ) ∣ \Delta= \sup_s \left|V_g^{\pi_k^Y}(s) - V_g^{\pi_k^{Y+\eta}}(s)\right| Δ = sup s V g π k Y ( s ) − V g π k Y + η ( s ) . Comparing the two Bellman equations gives
Δ ≤ γ Δ + sup s ∥ π k Y ( ⋅ ∣ s ) − π k Y + η ( ⋅ ∣ s ) ∥ 1 . \Delta\le\gamma\Delta+ \sup_s \left\|\pi_k^Y(\cdot\mid s) - \pi_k^{Y+\eta}(\cdot\mid s)\right\|_1. Δ ≤ γ Δ + s sup π k Y ( ⋅ ∣ s ) − π k Y + η ( ⋅ ∣ s ) 1 .
The softmax map is O ( 1 / τ ) O(1/\tau) O ( 1/ τ ) -Lipschitz in its input. Increasing the dual level by η \eta η changes the softmax score by O ( η ) O(\eta) O ( η ) under the normalized utility scale, and therefore
sup s ∥ π k Y ( ⋅ ∣ s ) − π k Y + η ( ⋅ ∣ s ) ∥ 1 ≤ O ( η / τ ) . \sup_s \left\|\pi_k^Y(\cdot\mid s) - \pi_k^{Y+\eta}(\cdot\mid s)\right\|_1 \le O(\eta/\tau). s sup π k Y ( ⋅ ∣ s ) − π k Y + η ( ⋅ ∣ s ) 1 ≤ O ( η / τ ) .
Rearranging the Bellman inequality proves the claim.
Decaying ϵ \epsilon ϵ -Greedy Probing: Tertiary-Aware Analysis
This appendix proves the decaying-ϵ \epsilon ϵ tertiary-aware result used in § 3 . The proof is organized around three steps:
the naive two-level decaying-ϵ \epsilon ϵ probing rule is not enough to close the optimism step;
a minimal tertiary-aware repair restores the missing denominator;
for this repaired algorithm, one can prove a regret guarantee for the return policy sequence { π k } k = 1 K \{\pi_k\}_{k=1}^{K} { π k } k = 1 K .
The guarantees in this appendix are for the return policies of Algorithm 1. The execution-policy transfer in § 3 is obtained afterward through a performance-difference argument. The same tertiary-aware front end also supports soft-violation control once the cumulative optimistic overestimation of the utility is bounded explicitly. The hard-violation analysis in later appendices uses the same representation-learning front end, but replaces optimistic feasibility checks with a lower-confidence certificate.
Why the Naive Two-Level Variant Is Not Enough
The natural first attempt is the two-level decaying-ϵ \epsilon ϵ rule
q i ( ⋅ ∣ s ) = q i ′ ( ⋅ ∣ s ) = ( 1 − ϵ i ) π i − 1 ( ⋅ ∣ s ) + ϵ i U ( A ) . q_i(\cdot\mid s)=q_i'(\cdot\mid s)=(1-\epsilon_i)\pi_{i-1}(\cdot\mid s)+\epsilon_i U(\mathcal{A}). q i ( ⋅ ∣ s ) = q i ′ ( ⋅ ∣ s ) = ( 1 − ϵ i ) π i − 1 ( ⋅ ∣ s ) + ϵ i U ( A ) .
This already improves the sampling rule relative to pure uniform probing, but it does not fully solve the coverage problem needed by the optimism proof.
Proposition D.1 (Two-level obstruction). There is no universal constant C < ∞ C<\infty C < ∞ such that the pointwise comparison
∥ ϕ ^ k ( s , a ) ∥ ( Σ k p r i ) − 1 ≤ C ∥ ϕ ^ k ( s , a ) ∥ ( Σ k m i x ) − 1 , ∀ ( s , a ) , \left\|\hat{\phi}_k(s,a)\right\|_{(\Sigma_k^{\mathrm{pri}})^{-1}}\leq C\left\|\hat{\phi}_k(s,a)\right\|_{(\Sigma_k^{\mathrm{mix}})^{-1}},\qquad\forall(s,a), ϕ ^ k ( s , a ) ( Σ k pri ) − 1 ≤ C ϕ ^ k ( s , a ) ( Σ k mix ) − 1 , ∀ ( s , a ) , holds for every episode k k k under the naive two-level decaying-ϵ \epsilon ϵ probing rule.
Proof. Consider a deterministic two-state MDP with states s 0 , s 1 s_0,s_1 s 0 , s 1 and two actions a s t a y , a j u m p a_{\mathrm{stay}},a_{\mathrm{jump}} a stay , a jump . Let the deployed policy always choose a s t a y a_{\mathrm{stay}} a stay at s 0 s_0 s 0 , let s 1 s_1 s 1 be absorbing, and let a j u m p a_{\mathrm{jump}} a jump be the only action that can move the process from s 0 s_0 s 0 to s 1 s_1 s 1 . Choose a bounded feature map such that
ϕ ^ k ( s 0 , a ) = e 1 , ϕ ^ k ( s 1 , a ) = e 2 , ∀ a ∈ A . \hat{\phi}_k(s_0,a)=e_1,\qquad\hat{\phi}_k(s_1,a)=e_2,\qquad\forall a\in\mathcal{A}. ϕ ^ k ( s 0 , a ) = e 1 , ϕ ^ k ( s 1 , a ) = e 2 , ∀ a ∈ A .
Then all primary samples stay at s 0 s_0 s 0 , so
Σ k p r i = k e 1 e 1 T + λ k I . \Sigma_k^{\mathrm{pri}}=k e_1e_1^{\mathsf{T}}+\lambda_k I. Σ k pri = k e 1 e 1 T + λ k I .
However, the secondary stream reaches s 1 s_1 s 1 with probability of order ϵ k / ∣ A ∣ \epsilon_k/|\mathcal{A}| ϵ k /∣ A ∣ , hence
Σ k m i x ⪰ k e 1 e 1 T + k ϵ k ∣ A ∣ e 2 e 2 T + λ k I . \Sigma_k^{\mathrm{mix}}\succeq k e_1e_1^{\mathsf{T}}+\frac{k\epsilon_k}{|\mathcal{A}|}e_2e_2^{\mathsf{T}}+\lambda_k I. Σ k mix ⪰ k e 1 e 1 T + ∣ A ∣ k ϵ k e 2 e 2 T + λ k I .
Evaluating the two norms at any point with feature vector e 2 e_2 e 2 gives
∥ e 2 ∥ ( Σ k p r i ) − 1 = 1 λ k , ∥ e 2 ∥ ( Σ k m i x ) − 1 ≤ 1 λ k + k ϵ k / ∣ A ∣ . \left\|e_2\right\|_{(\Sigma_k^{\mathrm{pri}})^{-1}}=\frac{1}{\sqrt{\lambda_k}},\qquad\left\|e_2\right\|_{(\Sigma_k^{\mathrm{mix}})^{-1}}\leq\frac{1}{\sqrt{\lambda_k+k\epsilon_k/|\mathcal{A}|}}. ∥ e 2 ∥ ( Σ k pri ) − 1 = λ k 1 , ∥ e 2 ∥ ( Σ k mix ) − 1 ≤ λ k + k ϵ k /∣ A ∣ 1 .
Therefore
∥ e 2 ∥ ( Σ k p r i ) − 1 ∥ e 2 ∥ ( Σ k m i x ) − 1 ≥ 1 + k ϵ k ∣ A ∣ λ k , \frac{\left\|e_2\right\|_{(\Sigma_k^{\mathrm{pri}})^{-1}}}{\left\|e_2\right\|_{(\Sigma_k^{\mathrm{mix}})^{-1}}}\geq\sqrt{1+\frac{k\epsilon_k}{|\mathcal{A}|\lambda_k}}, ∥ e 2 ∥ ( Σ k mix ) − 1 ∥ e 2 ∥ ( Σ k pri ) − 1 ≥ 1 + ∣ A ∣ λ k k ϵ k ,
which rules out a universal constant bridge. □ \square □
Proposition D.1 explains why the clean theorem is not for the naive two-level rule. The minimal repair for the present merged-covariance design is to add one more probe level. This obstruction does not rule out a primary-only alternative that fits on D + D ′ D+D' D + D ′ but constructs its covariance from D D D alone, as in the architecture of Uehara et al. [41 ] . Such a design removes the secondary covariance term but requires a different potential argument; the current regret aggregation cannot be transferred to it directly.
The Tertiary-Aware Repaired Algorithm
For each round i ∈ [ K ] i \in[K] i ∈ [ K ] , define the primary, secondary, and tertiary state marginals by
ρ i ( s ) : = d P ⋆ π i − 1 ( s ) , \rho_i(s) := d_{P^\star}^{\pi_{i-1}}(s), ρ i ( s ) := d P ⋆ π i − 1 ( s ) ,
ρ i ′ ( s ′ ) : = E ( s , a ) ∼ ρ i × q i [ P ⋆ ( s ′ ∣ s , a ) ] , \rho'_i(s') := \mathbb{E}_{(s,a)\sim\rho_i \times q_i}\left[P^\star(s' \mid s,a)\right], ρ i ′ ( s ′ ) := E ( s , a ) ∼ ρ i × q i [ P ⋆ ( s ′ ∣ s , a ) ] ,
ρ i ′ ′ ( s ′ ′ ) : = E ( s , a ) ∼ ρ i ′ × q i ′ [ P ⋆ ( s ′ ′ ∣ s , a ) ] . \rho''_i(s'') := \mathbb{E}_{(s,a)\sim\rho'_i \times q'_i}\left[P^\star(s'' \mid s,a)\right]. ρ i ′′ ( s ′′ ) := E ( s , a ) ∼ ρ i ′ × q i ′ [ P ⋆ ( s ′′ ∣ s , a ) ] .
The corresponding probing distributions are
ν i ( s , a ) : = ρ i ( s ) q i ( a ∣ s ) , ν i ′ ( s , a ) : = ρ i ′ ( s ) q i ′ ( a ∣ s ) , ν i ′ ′ ( s , a ) : = ρ i ′ ′ ( s ) q i ′ ′ ( a ∣ s ) . \nu_i(s,a) := \rho_i(s)q_i(a \mid s), \qquad\nu'_i(s,a) := \rho'_i(s)q'_i(a \mid s), \qquad\nu''_i(s,a) := \rho''_i(s)q''_i(a \mid s). ν i ( s , a ) := ρ i ( s ) q i ( a ∣ s ) , ν i ′ ( s , a ) := ρ i ′ ( s ) q i ′ ( a ∣ s ) , ν i ′′ ( s , a ) := ρ i ′′ ( s ) q i ′′ ( a ∣ s ) .
We write
ν ˉ k p r i : = 1 k ∑ i = 1 k ν i , ν ˉ k s e c : = 1 k ∑ i = 1 k ν i ′ , ν ˉ k t e r : = 1 k ∑ i = 1 k ν i ′ ′ , \bar{\nu}^{\mathrm{pri}}_k := \frac{1}{k}\sum_{i=1}^{k}\nu_i, \qquad\bar{\nu}^{\mathrm{sec}}_k := \frac{1}{k}\sum_{i=1}^{k}\nu'_i, \qquad\bar{\nu}^{\mathrm{ter}}_k := \frac{1}{k}\sum_{i=1}^{k}\nu''_i, ν ˉ k pri := k 1 i = 1 ∑ k ν i , ν ˉ k sec := k 1 i = 1 ∑ k ν i ′ , ν ˉ k ter := k 1 i = 1 ∑ k ν i ′′ ,
and
ν ˉ k a l l : = 1 3 ( ν ˉ k p r i + ν ˉ k s e c + ν ˉ k t e r ) . \bar{\nu}^{\mathrm{all}}_k := \frac{1}{3}\left(\bar{\nu}^{\mathrm{pri}}_k+\bar{\nu}^{\mathrm{sec}}_k+\bar{\nu}^{\mathrm{ter}}_k\right). ν ˉ k all := 3 1 ( ν ˉ k pri + ν ˉ k sec + ν ˉ k ter ) .
The repaired probing kernels are
q k ( ⋅ ∣ s ) = q k ′ ( ⋅ ∣ s ) = q k ′ ′ ( ⋅ ∣ s ) = ( 1 − ϵ k ) π k − 1 ( ⋅ ∣ s ) + ϵ k U ( A ) , q_k(\cdot\mid s) = q'_k(\cdot\mid s) = q''_k(\cdot\mid s) = (1-\epsilon_k)\pi_{k-1}(\cdot\mid s) + \epsilon_k U(\mathcal{A}), q k ( ⋅ ∣ s ) = q k ′ ( ⋅ ∣ s ) = q k ′′ ( ⋅ ∣ s ) = ( 1 − ϵ k ) π k − 1 ( ⋅ ∣ s ) + ϵ k U ( A ) ,
where
ϵ 1 ≥ ϵ 2 ≥ ⋯ ≥ ϵ K > 0. \epsilon_1 \ge\epsilon_2 \ge\cdots\ge\epsilon_K > 0. ϵ 1 ≥ ϵ 2 ≥ ⋯ ≥ ϵ K > 0.
We also write
Γ k : = ∣ A ∣ ϵ k , Λ k : = 1 1 − ϵ k + ϵ k / ∣ A ∣ . \Gamma_k := \frac{|\mathcal{A}|}{\epsilon_k}, \qquad\Lambda_k := \frac{1}{1-\epsilon_k+\epsilon_k/|\mathcal{A}|}. Γ k := ϵ k ∣ A ∣ , Λ k := 1 − ϵ k + ϵ k /∣ A ∣ 1 .
The coverage coefficients used below are domination constants between the target distributions appearing in the proof and the three empirical probing streams. For two state-action distributions η , ν \eta,\nu η , ν , define
D ( η ∥ ν ) : = inf { c ≥ 0 : E η [ g 2 ] ≤ c E ν [ g 2 ] for all g } . D(\eta\Vert\nu) := \inf\left\{c \ge0 : \mathbb{E}_{\eta}[g^2] \le c\mathbb{E}_{\nu}[g^2]\ \text{for all } g\right\}. D ( η ∥ ν ) := inf { c ≥ 0 : E η [ g 2 ] ≤ c E ν [ g 2 ] for all g } .
When densities exist, this is the essential supremum of d η / d ν d\eta/d\nu d η / d ν . For a state-action distribution ν \nu ν , let T π ν T_\pi\nu T π ν denote the one-step pushforward obtained by drawing ( s ~ , a ~ ) ∼ ν (\tilde{s},\tilde{a}) \sim\nu ( s ~ , a ~ ) ∼ ν , s ∼ P ⋆ ( ⋅ ∣ s ~ , a ~ ) s \sim P^\star(\cdot\mid\tilde{s},\tilde{a}) s ∼ P ⋆ ( ⋅ ∣ s ~ , a ~ ) , and a ∼ π ( ⋅ ∣ s ) a \sim\pi(\cdot\mid s) a ∼ π ( ⋅ ∣ s ) . With this notation,
C k i n i t ( π ) : = D ( ( 1 − γ ) d 1 ⊗ π ∥ ν ˉ k p r i ) , \mathcal{C}^{\mathrm{init}}_k(\pi) := D\left((1-\gamma)d_1 \otimes\pi\,\middle\Vert\, \bar{\nu}^{\mathrm{pri}}_k\right), C k init ( π ) := D ( ( 1 − γ ) d 1 ⊗ π ν ˉ k pri ) ,
C k n e x t ( π ) : = D ( T π ν ˉ k p r i ∥ ν ˉ k s e c ) , \mathcal{C}^{\mathrm{next}}_k(\pi) := D\left(T_\pi\bar{\nu}^{\mathrm{pri}}_k \,\middle\Vert\, \bar{\nu}^{\mathrm{sec}}_k\right), C k next ( π ) := D ( T π ν ˉ k pri ν ˉ k sec ) ,
C k , 1 t r i , n e x t ( π ) : = C k n e x t ( π ) , \mathcal{C}^{\mathrm{tri,next}}_{k,1}(\pi) := \mathcal{C}^{\mathrm{next}}_k(\pi), C k , 1 tri , next ( π ) := C k next ( π ) ,
C k , 2 t r i , n e x t ( π ) : = D ( T π ν ˉ k s e c ∥ ν ˉ k t e r ) , \mathcal{C}^{\mathrm{tri,next}}_{k,2}(\pi) := D\left(T_\pi\bar{\nu}^{\mathrm{sec}}_k \,\middle\Vert\, \bar{\nu}^{\mathrm{ter}}_k\right), C k , 2 tri , next ( π ) := D ( T π ν ˉ k sec ν ˉ k ter ) ,
C k t r i , n e x t ( π ) : = max { C k , 1 t r i , n e x t ( π ) , C k , 2 t r i , n e x t ( π ) } . \mathcal{C}^{\mathrm{tri,next}}_k(\pi) := \max\left\{\mathcal{C}^{\mathrm{tri,next}}_{k,1}(\pi), \mathcal{C}^{\mathrm{tri,next}}_{k,2}(\pi)\right\}. C k tri , next ( π ) := max { C k , 1 tri , next ( π ) , C k , 2 tri , next ( π ) } .
Thus C k i n i t \mathcal{C}^{\mathrm{init}}_k C k init compares the initial discounted mass to the primary stream, C k n e x t \mathcal{C}^{\mathrm{next}}_k C k next compares the primary-to-secondary one-step design, and C k t r i , n e x t \mathcal{C}^{\mathrm{tri,next}}_k C k tri , next is the maximum of the two one-step comparisons needed for the tertiary one-step-back argument.
The algorithm analyzed in this appendix is the main-text Algorithm 1. The repair relative to the earlier two-stream protocol is entirely in the auxiliary sampling design: it adds one more probe-action / transition pair, while keeping the bonus covariance merged over the primary and secondary tuples.
Supporting Lemmas
We now record the small collection of lemmas needed for the final theorem.
Lemma D.2 (Three-stream MLE event at tuple level). Under realizability P ⋆ ∈ M P^\star\in\mathcal{M} P ⋆ ∈ M , with probability at least 1 − δ 1-\delta 1 − δ , simultaneously for all k ∈ [ K ] k \in[K] k ∈ [ K ] ,
E ( s , a ) ∼ ν ~ k a l l [ ∥ P ^ k ( ⋅ ∣ s , a ) − P ⋆ ( ⋅ ∣ s , a ) ∥ 1 2 ] ≲ ln ( ∣ M ∣ k / δ ) k = : ζ k . \mathbb{E}_{(s,a)\sim\widetilde{\nu}^{\mathrm{all}}_k}\left[\left\|\widehat{P}_k(\cdot\mid s,a)-P^\star(\cdot\mid s,a)\right\|_1^2\right]\lesssim\frac{\ln(|\mathcal{M}|k/\delta)}{k}=:\zeta_k. E ( s , a ) ∼ ν k all [ P k ( ⋅ ∣ s , a ) − P ⋆ ( ⋅ ∣ s , a ) 1 2 ] ≲ k ln ( ∣ M ∣ k / δ ) =: ζ k . Here
ν ~ k s e c : = 1 k ∑ i = 1 k δ s i ′ q i ′ ( ⋅ ∣ s i ′ ) , ν ~ k t e r : = 1 k ∑ i = 1 k δ s i ′ ′ q i ′ ′ ( ⋅ ∣ s i ′ ′ ) , \widetilde{\nu}^{\mathrm{sec}}_k:=\frac{1}{k}\sum_{i=1}^{k}\delta_{s'_i}q'_i(\cdot\mid s'_i),\qquad
\widetilde{\nu}^{\mathrm{ter}}_k:=\frac{1}{k}\sum_{i=1}^{k}\delta_{s''_i}q''_i(\cdot\mid s''_i), ν k sec := k 1 i = 1 ∑ k δ s i ′ q i ′ ( ⋅ ∣ s i ′ ) , ν k ter := k 1 i = 1 ∑ k δ s i ′′ q i ′′ ( ⋅ ∣ s i ′′ ) , and
ν ~ k a l l : = 1 3 ( ν ~ k p r i + ν ~ k s e c + ν ~ k t e r ) . \widetilde{\nu}^{\mathrm{all}}_k:=\frac{1}{3}\left(\widetilde{\nu}^{\mathrm{pri}}_k+\widetilde{\nu}^{\mathrm{sec}}_k+\widetilde{\nu}^{\mathrm{ter}}_k\right). ν k all := 3 1 ( ν k pri + ν k sec + ν k ter ) .
Proof. For episode i i i , write the realized tertiary tuple as ( s i , a i , s i ′ , a i ′ , s i ′ ′ , a i ′ ′ , s ˉ i ) (s_i,a_i,s'_i,a'_i,s''_i,a''_i,\bar{s}_i) ( s i , a i , s i ′ , a i ′ , s i ′′ , a i ′′ , s ˉ i ) , where s ˉ i ∼ P ⋆ ( ⋅ ∣ s i ′ ′ , a i ′ ′ ) \bar{s}_i\sim P^\star(\cdot\mid s''_i,a''_i) s ˉ i ∼ P ⋆ ( ⋅ ∣ s i ′′ , a i ′′ ) is the next state of the third probe transition. Define
x 3 i − 2 : = ( s i , a i ) , y 3 i − 2 : = s i ′ , x_{3i-2}:=(s_i,a_i),\qquad y_{3i-2}:=s'_i, x 3 i − 2 := ( s i , a i ) , y 3 i − 2 := s i ′ ,
x 3 i − 1 : = ( s i ′ , a i ′ ) , y 3 i − 1 : = s i ′ ′ , x_{3i-1}:=(s'_i,a'_i),\qquad y_{3i-1}:=s''_i, x 3 i − 1 := ( s i ′ , a i ′ ) , y 3 i − 1 := s i ′′ ,
and
x 3 i : = ( s i ′ ′ , a i ′ ′ ) , y 3 i : = s ˉ i . x_{3i}:=(s''_i,a''_i),\qquad y_{3i}:=\bar{s}_i. x 3 i := ( s i ′′ , a i ′′ ) , y 3 i := s ˉ i .
Let F 3 i − 3 \mathcal{F}_{3i-3} F 3 i − 3 be the sigma-field generated by all data collected before episode i i i starts. Recursively define
F 3 i − 2 : = σ ( F 3 i − 3 , x 3 i − 2 , y 3 i − 2 ) , F 3 i − 1 : = σ ( F 3 i − 2 , x 3 i − 1 , y 3 i − 1 ) , \mathcal{F}_{3i-2}:=\sigma(\mathcal{F}_{3i-3},x_{3i-2},y_{3i-2}),\qquad
\mathcal{F}_{3i-1}:=\sigma(\mathcal{F}_{3i-2},x_{3i-1},y_{3i-1}), F 3 i − 2 := σ ( F 3 i − 3 , x 3 i − 2 , y 3 i − 2 ) , F 3 i − 1 := σ ( F 3 i − 2 , x 3 i − 1 , y 3 i − 1 ) ,
and
F 3 i : = σ ( F 3 i − 1 , x 3 i , y 3 i ) . \mathcal{F}_{3i}:=\sigma(\mathcal{F}_{3i-1},x_{3i},y_{3i}). F 3 i := σ ( F 3 i − 1 , x 3 i , y 3 i ) .
Then the predictable covariate designs are
D 3 i − 2 : = Law ( x 3 i − 2 ∣ F 3 i − 3 ) = ν i , D_{3i-2}:=\operatorname{Law}(x_{3i-2}\mid\mathcal{F}_{3i-3})=\nu_i, D 3 i − 2 := Law ( x 3 i − 2 ∣ F 3 i − 3 ) = ν i ,
D 3 i − 1 : = Law ( x 3 i − 1 ∣ F 3 i − 2 ) = δ s i ′ q i ′ ( ⋅ ∣ s i ′ ) , D_{3i-1}:=\operatorname{Law}(x_{3i-1}\mid\mathcal{F}_{3i-2})=\delta_{s'_i}q'_i(\cdot\mid s'_i), D 3 i − 1 := Law ( x 3 i − 1 ∣ F 3 i − 2 ) = δ s i ′ q i ′ ( ⋅ ∣ s i ′ ) ,
D 3 i : = Law ( x 3 i ∣ F 3 i − 1 ) = δ s i ′ ′ q i ′ ′ ( ⋅ ∣ s i ′ ′ ) . D_{3i}:=\operatorname{Law}(x_{3i}\mid\mathcal{F}_{3i-1})=\delta_{s''_i}q''_i(\cdot\mid s''_i). D 3 i := Law ( x 3 i ∣ F 3 i − 1 ) = δ s i ′′ q i ′′ ( ⋅ ∣ s i ′′ ) .
Hence the full covariate sequence
x 1 , x 2 , … , x 3 k x_1,x_2,\ldots,x_{3k} x 1 , x 2 , … , x 3 k
is a predictable design sequence with respect to the tuple-level filtration { F t } t = 0 3 k \{\mathcal{F}_t\}_{t=0}^{3k} { F t } t = 0 3 k in the sense of [2 Theorem 21] . Moreover, the responses satisfy
y t ∼ P ⋆ ( ⋅ ∣ x t ) for every t ≤ 3 k , y_t\sim P^\star(\cdot\mid x_t)\qquad\text{for every }t\le3k, y t ∼ P ⋆ ( ⋅ ∣ x t ) for every t ≤ 3 k ,
and realizability gives P ⋆ ∈ M P^\star\in\mathcal{M} P ⋆ ∈ M .
Apply Theorem 21 of [2 ] to the first 3 k 3k 3 k samples, with confidence level δ / k 2 \delta/k^2 δ / k 2 . This yields, for every fixed k k k ,
1 3 k ∑ t = 1 3 k E x ∼ D t [ ∥ P ^ k ( ⋅ ∣ x ) − P ⋆ ( ⋅ ∣ x ) ∥ 1 2 ] ≲ ln ( ∣ M ∣ k / δ ) k , \frac{1}{3k}\sum_{t=1}^{3k}\mathbb{E}_{x\sim D_t}\left[\left\|\widehat{P}_k(\cdot\mid x)-P^\star(\cdot\mid x)\right\|_1^2\right]\lesssim\frac{\ln(|\mathcal{M}|k/\delta)}{k}, 3 k 1 t = 1 ∑ 3 k E x ∼ D t [ P k ( ⋅ ∣ x ) − P ⋆ ( ⋅ ∣ x ) 1 2 ] ≲ k ln ( ∣ M ∣ k / δ ) ,
where D t D_t D t denotes the predictable design of the t t t -th tuple. By the definitions above,
1 3 k ∑ t = 1 3 k D t = 1 3 ( ν ~ k p r i + ν ~ k s e c + ν ~ k t e r ) = ν ~ k a l l , \frac{1}{3k}\sum_{t=1}^{3k}D_t=\frac{1}{3}\left(\widetilde{\nu}^{\mathrm{pri}}_k+\widetilde{\nu}^{\mathrm{sec}}_k+\widetilde{\nu}^{\mathrm{ter}}_k\right)=\widetilde{\nu}^{\mathrm{all}}_k, 3 k 1 t = 1 ∑ 3 k D t = 3 1 ( ν k pri + ν k sec + ν k ter ) = ν k all ,
as a random probability measure. Therefore the left-hand side is precisely
E ( s , a ) ∼ ν ~ k a l l [ ∥ P ^ k ( ⋅ ∣ s , a ) − P ⋆ ( ⋅ ∣ s , a ) ∥ 1 2 ] . \mathbb{E}_{(s,a)\sim\widetilde{\nu}^{\mathrm{all}}_k}\left[\left\|\widehat{P}_k(\cdot\mid s,a)-P^\star(\cdot\mid s,a)\right\|_1^2\right]. E ( s , a ) ∼ ν k all [ P k ( ⋅ ∣ s , a ) − P ⋆ ( ⋅ ∣ s , a ) 1 2 ] .
Finally, a union bound over all k ∈ [ K ] k\in[K] k ∈ [ K ] turns the fixed-k k k statement into the simultaneous event claimed in the lemma, with the extra ln k \ln k ln k absorbed into ζ k \zeta_k ζ k .
Lemma D.4 (Pathwise-to-averaged bridge for the MLE mismatch class). For each candidate model P ∈ M P\in\mathcal{M} P ∈ M , define
h P ( s , a ) : = ∥ P ( ⋅ ∣ s , a ) − P ⋆ ( ⋅ ∣ s , a ) ∥ 1 2 . h_P(s,a):=\lVert P(\cdot\mid s,a)-P^\star(\cdot\mid s,a)\rVert_1^2. h P ( s , a ) := ∥ P ( ⋅ ∣ s , a ) − P ⋆ ( ⋅ ∣ s , a ) ∥ 1 2 . Then there is a universal constant C < ∞ C<\infty C < ∞ such that, with probability at least 1 − δ 1-\delta 1 − δ , simultaneously for all k ∈ [ K ] k\in[K] k ∈ [ K ] and all P ∈ M P\in\mathcal{M} P ∈ M ,
E ν ~ k s e c [ h P ] ≤ 2 E ν ˉ k s e c [ h P ] + C ln ( ∣ M ∣ k / δ ) k , \mathbb{E}_{\tilde{\nu}^{\mathrm{sec}}_k}[h_P]\le2\mathbb{E}_{\bar{\nu}^{\mathrm{sec}}_k}[h_P]+C\frac{\ln(|\mathcal{M}|k/\delta)}{k}, E ν ~ k sec [ h P ] ≤ 2 E ν ˉ k sec [ h P ] + C k ln ( ∣ M ∣ k / δ ) , and
E ν ~ k t e r [ h P ] ≤ 2 E ν ˉ k t e r [ h P ] + C ln ( ∣ M ∣ k / δ ) k . \mathbb{E}_{\tilde{\nu}^{\mathrm{ter}}_k}[h_P]\le2\mathbb{E}_{\bar{\nu}^{\mathrm{ter}}_k}[h_P]+C\frac{\ln(|\mathcal{M}|k/\delta)}{k}. E ν ~ k ter [ h P ] ≤ 2 E ν ˉ k ter [ h P ] + C k ln ( ∣ M ∣ k / δ ) . Consequently, on the intersection of this event with Lemma D.2 , we have simultaneously for all k ∈ [ K ] k\in[K] k ∈ [ K ] ,
E ν ˉ k s e c [ f k 2 ] ≲ ζ k , E ν ˉ k t e r [ f k 2 ] ≲ ζ k . \mathbb{E}_{\bar{\nu}^{\mathrm{sec}}_k}[f_k^2]\lesssim\zeta_k,\qquad
\mathbb{E}_{\bar{\nu}^{\mathrm{ter}}_k}[f_k^2]\lesssim\zeta_k. E ν ˉ k sec [ f k 2 ] ≲ ζ k , E ν ˉ k ter [ f k 2 ] ≲ ζ k . In particular, this lemma is used later only to transfer the internal MLE-mismatch terms from the pathwise designs to the averaged designs. The generic test function g g g in the one-step-back and optimism lemmas is handled directly under the deterministic averaged measures ν ˉ k s e c , ν ˉ k t e r \bar{\nu}^{\mathrm{sec}}_k,\bar{\nu}^{\mathrm{ter}}_k ν ˉ k sec , ν ˉ k ter through the coverage coefficients, so no stronger uniform bridge over an adaptive class is invoked.
Proof. Fix a candidate model P ∈ M P\in\mathcal{M} P ∈ M and write h : = h P h:=h_P h := h P . Since ∥ P ( ⋅ ∣ s , a ) − P ⋆ ( ⋅ ∣ s , a ) ∥ 1 ≤ 2 \lVert P(\cdot\mid s,a)-P^\star(\cdot\mid s,a)\rVert_1\le2 ∥ P ( ⋅ ∣ s , a ) − P ⋆ ( ⋅ ∣ s , a ) ∥ 1 ≤ 2 , we have
0 ≤ h ( s , a ) ≤ 4 ∀ ( s , a ) . 0\le h(s,a)\le4\qquad\forall(s,a). 0 ≤ h ( s , a ) ≤ 4 ∀ ( s , a ) .
We first treat the secondary stream. For each episode i i i , define
X i s e c ( h ) : = h ( s i ′ , a i ′ ) − E ν ˉ i s e c [ h ] = h ( s i ′ , a i ′ ) − E ν i ′ [ h ] . X_i^{\mathrm{sec}}(h):=h(s_i',a_i')-\mathbb{E}_{\bar{\nu}^{\mathrm{sec}}_i}[h]=h(s_i',a_i')-\mathbb{E}_{\nu_i'}[h]. X i sec ( h ) := h ( s i ′ , a i ′ ) − E ν ˉ i sec [ h ] = h ( s i ′ , a i ′ ) − E ν i ′ [ h ] .
Let G i \mathcal{G}_i G i denote the sigma-field generated by all tuples up to the end of episode i i i . Because ( s i ′ , a i ′ ) (s_i',a_i') ( s i ′ , a i ′ ) has conditional mean design ν i ′ \nu_i' ν i ′ given G i − 1 \mathcal{G}_{i-1} G i − 1 , we have
E [ X i s e c ( h ) ∣ G i − 1 ] = 0. \mathbb{E}[X_i^{\mathrm{sec}}(h)\mid\mathcal{G}_{i-1}]=0. E [ X i sec ( h ) ∣ G i − 1 ] = 0.
Moreover,
∣ X i s e c ( h ) ∣ ≤ 4 , \lvert X_i^{\mathrm{sec}}(h)\rvert\le4, ∣ X i sec ( h )∣ ≤ 4 ,
and, using h 2 ≤ 4 h h^2\le4h h 2 ≤ 4 h on [ 0 , 4 ] [0,4] [ 0 , 4 ] ,
E [ ( X i s e c ( h ) ) 2 ∣ G i − 1 ] ≤ E [ h ( s i ′ , a i ′ ) 2 ∣ G i − 1 ] ≤ 4 E [ h ( s i ′ , a i ′ ) ∣ G i − 1 ] = 4 E ν i ′ [ h ] . \mathbb{E}\left[(X_i^{\mathrm{sec}}(h))^2\mid\mathcal{G}_{i-1}\right]
\le\mathbb{E}[h(s_i',a_i')^2\mid\mathcal{G}_{i-1}]
\le4\mathbb{E}[h(s_i',a_i')\mid\mathcal{G}_{i-1}]
=4\mathbb{E}_{\nu_i'}[h]. E [ ( X i sec ( h ) ) 2 ∣ G i − 1 ] ≤ E [ h ( s i ′ , a i ′ ) 2 ∣ G i − 1 ] ≤ 4 E [ h ( s i ′ , a i ′ ) ∣ G i − 1 ] = 4 E ν i ′ [ h ] .
Hence the martingale S k s e c ( h ) : = ∑ i = 1 k X i s e c ( h ) S_k^{\mathrm{sec}}(h):=\sum_{i=1}^kX_i^{\mathrm{sec}}(h) S k sec ( h ) := ∑ i = 1 k X i sec ( h ) has predictable quadratic variation bounded by
V k s e c ( h ) ≤ 4 ∑ i = 1 k E ν i ′ [ h ] = 4 k E ν ˉ k s e c [ h ] . V_k^{\mathrm{sec}}(h)\le4\sum_{i=1}^k\mathbb{E}_{\nu_i'}[h]=4k\mathbb{E}_{\bar{\nu}^{\mathrm{sec}}_k}[h]. V k sec ( h ) ≤ 4 i = 1 ∑ k E ν i ′ [ h ] = 4 k E ν ˉ k sec [ h ] .
Apply Freedman’s inequality with confidence level δ k , P s e c : = δ / ( 4 ∣ M ∣ k 2 ) \delta_{k,P}^{\mathrm{sec}}:=\delta/(4|\mathcal{M}|k^2) δ k , P sec := δ / ( 4∣ M ∣ k 2 ) , and write ℓ k s e c : = ln ( 1 / δ k , P s e c ) \ell_k^{\mathrm{sec}}:=\ln(1/\delta_{k,P}^{\mathrm{sec}}) ℓ k sec := ln ( 1/ δ k , P sec ) . Then, with probability at least 1 − δ k , P s e c 1-\delta_{k,P}^{\mathrm{sec}} 1 − δ k , P sec ,
− S k s e c ( h ) ≤ 2 V k s e c ( h ) ℓ k s e c + 4 3 ℓ k s e c . -S_k^{\mathrm{sec}}(h)\le\sqrt{2V_k^{\mathrm{sec}}(h)\ell_k^{\mathrm{sec}}}+\frac{4}{3}\ell_k^{\mathrm{sec}}. − S k sec ( h ) ≤ 2 V k sec ( h ) ℓ k sec + 3 4 ℓ k sec .
Dividing by k k k and recalling that
− S k s e c ( h ) k = E ν k s e c [ h ] − E ν ˉ k s e c [ h ] , -\frac{S_k^{\mathrm{sec}}(h)}{k}=\mathbb{E}_{\nu_k^{\mathrm{sec}}}[h]-\mathbb{E}_{\bar{\nu}_k^{\mathrm{sec}}}[h], − k S k sec ( h ) = E ν k sec [ h ] − E ν ˉ k sec [ h ] ,
we obtain
E ν ˉ k s e c [ h ] − E ν k s e c [ h ] ≤ 8 E ν ˉ k s e c [ h ] ℓ k s e c k + 4 3 ℓ k s e c k . \mathbb{E}_{\bar{\nu}_k^{\mathrm{sec}}}[h]-\mathbb{E}_{\nu_k^{\mathrm{sec}}}[h]\leq\sqrt{\frac{8\mathbb{E}_{\bar{\nu}_k^{\mathrm{sec}}}[h]\ell_k^{\mathrm{sec}}}{k}}+\frac{4}{3}\frac{\ell_k^{\mathrm{sec}}}{k}. E ν ˉ k sec [ h ] − E ν k sec [ h ] ≤ k 8 E ν ˉ k sec [ h ] ℓ k sec + 3 4 k ℓ k sec .
Using 2 a b ≤ a 2 + 2 b 2\sqrt{ab}\leq\frac{a}{2}+2b 2 ab ≤ 2 a + 2 b with a = E ν ˉ k s e c [ h ] a=\mathbb{E}_{\bar{\nu}_k^{\mathrm{sec}}}[h] a = E ν ˉ k sec [ h ] and b = 8 ℓ k s e c / k b=8\ell_k^{\mathrm{sec}}/k b = 8 ℓ k sec / k gives
8 E ν ˉ k s e c [ h ] ℓ k s e c k ≤ 1 2 E ν ˉ k s e c [ h ] + 16 ℓ k s e c k . \sqrt{\frac{8\mathbb{E}_{\bar{\nu}_k^{\mathrm{sec}}}[h]\ell_k^{\mathrm{sec}}}{k}}\leq\frac{1}{2}\mathbb{E}_{\bar{\nu}_k^{\mathrm{sec}}}[h]+16\frac{\ell_k^{\mathrm{sec}}}{k}. k 8 E ν ˉ k sec [ h ] ℓ k sec ≤ 2 1 E ν ˉ k sec [ h ] + 16 k ℓ k sec .
Therefore
E ν ˉ k s e c [ h ] ≤ 2 E ν k s e c [ h ] + 104 3 ℓ k s e c k . \mathbb{E}_{\bar{\nu}_k^{\mathrm{sec}}}[h]\leq2\mathbb{E}_{\nu_k^{\mathrm{sec}}}[h]+\frac{104}{3}\frac{\ell_k^{\mathrm{sec}}}{k}. E ν ˉ k sec [ h ] ≤ 2 E ν k sec [ h ] + 3 104 k ℓ k sec .
Since ℓ k s e c = ln ( 4 ∣ M ∣ k 2 / δ ) ≲ ln ( ∣ M ∣ k / δ ) \ell_k^{\mathrm{sec}}=\ln(4|\mathcal{M}|k^2/\delta)\lesssim\ln(|\mathcal{M}|k/\delta) ℓ k sec = ln ( 4∣ M ∣ k 2 / δ ) ≲ ln ( ∣ M ∣ k / δ ) , this yields the desired secondary bound after adjusting the universal constant C C C .
The tertiary bound is identical. Indeed, defining
X i t e r ( h ) : = h ( s i ′ ′ , a i ′ ′ ) − E ν i ′ ′ [ h ] , X_i^{\mathrm{ter}}(h):=h(s_i^{\prime\prime},a_i^{\prime\prime})-\mathbb{E}_{\nu_{i}^{\prime\prime}}[h], X i ter ( h ) := h ( s i ′′ , a i ′′ ) − E ν i ′′ [ h ] ,
we again have
E [ X i t e r ( h ) ∣ G i − 1 ] = 0 , ∣ X i t e r ( h ) ∣ ≤ 4 , \mathbb{E}[X_i^{\mathrm{ter}}(h)\mid\mathcal{G}_{i-1}]=0,\qquad\lvert X_i^{\mathrm{ter}}(h)\rvert\leq4, E [ X i ter ( h ) ∣ G i − 1 ] = 0 , ∣ X i ter ( h )∣ ≤ 4 ,
and
E [ ( X i t e r ( h ) ) 2 ∣ G i − 1 ] ≤ 4 E ν i ′ ′ [ h ] . \mathbb{E}\left[\left(X_i^{\mathrm{ter}}(h)\right)^2\mid\mathcal{G}_{i-1}\right]\leq4\mathbb{E}_{\nu_i^{\prime\prime}}[h]. E [ ( X i ter ( h ) ) 2 ∣ G i − 1 ] ≤ 4 E ν i ′′ [ h ] .
Running the same Freedman argument and taking a union bound over the secondary and tertiary streams, over all P ∈ M P\in\mathcal{M} P ∈ M , and over all k ∈ [ K ] k\in[K] k ∈ [ K ] , gives the first two displays in the lemma.
Finally, set P = P ^ k P=\widehat{P}_k P = P k . Since P ^ k ∈ M \widehat{P}_k\in\mathcal{M} P k ∈ M , we may plug h P ^ k = f k 2 h_{\widehat{P}_k}=f_k^2 h P k = f k 2 into the simultaneous event. By Lemma D.2,
E ν ˉ k a l l [ f k 2 ] ≲ ζ k . \mathbb{E}_{\bar{\nu}_k^{\mathrm{all}}}[f_k^2]\lesssim\zeta_k. E ν ˉ k all [ f k 2 ] ≲ ζ k .
Because
ν ~ k a l l = 1 3 ( ν k p r i + ν k s e c + ν k t e r ) , \widetilde{\nu}_k^{\mathrm{all}}=\frac{1}{3}\left(\nu_k^{\mathrm{pri}}+\nu_k^{\mathrm{sec}}+\nu_k^{\mathrm{ter}}\right), ν k all = 3 1 ( ν k pri + ν k sec + ν k ter ) ,
and all three component expectations are nonnegative,
E ν k s e c [ f k 2 ] ≤ 3 E ν ~ k a l l [ f k 2 ] ≲ ζ k , \mathbb{E}_{\nu_k^{\mathrm{sec}}}[f_k^2]\leq3\mathbb{E}_{\widetilde{\nu}_k^{\mathrm{all}}}[f_k^2]\lesssim\zeta_k, E ν k sec [ f k 2 ] ≤ 3 E ν k all [ f k 2 ] ≲ ζ k ,
and likewise
E ν k t e r [ f k 2 ] ≲ ζ k . \mathbb{E}_{\nu_k^{\mathrm{ter}}}[f_k^2]\lesssim\zeta_k. E ν k ter [ f k 2 ] ≲ ζ k .
Substituting these into the secondary and tertiary bridge inequalities and absorbing the extra ln ( ∣ M ∣ k / δ ) / k \ln(|\mathcal{M}|k/\delta)/k ln ( ∣ M ∣ k / δ ) / k term into ζ k \zeta_k ζ k proves the final display.
Lemma D.5 (Primary-covariance deployed-to-probe transfer). Fix an episode k k k and define
Σ k , ⋆ p r i : = k E ν k p r i [ ϕ ⋆ ( s , a ) ϕ ⋆ ( s , a ) ⊤ ] + λ k I . \Sigma_{k,\star}^{\mathrm{pri}}:=k\mathbb{E}_{\nu_k^{\mathrm{pri}}}\left[\phi^\star(s,a)\phi^\star(s,a)^\top\right]+\lambda_k I. Σ k , ⋆ pri := k E ν k pri [ ϕ ⋆ ( s , a ) ϕ ⋆ ( s , a ) ⊤ ] + λ k I . Then for every policy π \pi π and every measurable g : S × A → R g:\mathcal{S}\times\mathcal{A}\to\mathbb{R} g : S × A → R with ∥ g ∥ ∞ ≤ B , \lVert g\rVert_\infty\leq B, ∥ g ∥ ∞ ≤ B ,
∣ E d π i n i t [ g ] ∣ ≲ γ E d π i n i t [ ∥ ϕ ⋆ ( s ~ , a ~ ) ∥ ( Σ k , ⋆ p r i ) − 1 k ζ k n e x t ( π ) E ν k s e c [ g 2 ] + λ k d B 2 + ( 1 − γ ) ζ k i n i t ( π ) E ν k p r i [ g 2 ] ] . \left|\mathbb{E}_{d_{\pi}^{\mathrm{init}}}[g]\right|\lesssim\sqrt{\gamma}\mathbb{E}_{d_{\pi}^{\mathrm{init}}}\left[\lVert\phi^\star(\widetilde{s},\widetilde{a})\rVert_{(\Sigma_{k,\star}^{\mathrm{pri}})^{-1}}\sqrt{k\zeta_k^{\mathrm{next}}(\pi)\mathbb{E}_{\nu_k^{\mathrm{sec}}}[g^2]+\lambda_kdB^2}+\sqrt{(1-\gamma)\zeta_k^{\mathrm{init}}(\pi)\mathbb{E}_{\nu_k^{\mathrm{pri}}}[g^2]}\right]. E d π init [ g ] ≲ γ E d π init [ ∥ ϕ ⋆ ( s , a ) ∥ ( Σ k , ⋆ pri ) − 1 k ζ k next ( π ) E ν k sec [ g 2 ] + λ k d B 2 + ( 1 − γ ) ζ k init ( π ) E ν k pri [ g 2 ] ] .
Proof. Define
m g ( s ) : = E a ∼ π ( ⋅ ∣ s ) [ g ( s , a ) ] , T ⋆ π g ( s ~ , a ~ ) : = E s ∼ P ⋆ ( ⋅ ∣ s , a ~ ) [ m g ( s ) ] . m_g(s) := \mathbb{E}_{a\sim\pi(\cdot\mid s)}[g(s,a)], \qquad T_\star^\pi g(\tilde{s},\tilde{a}) := \mathbb{E}_{s\sim P^\star(\cdot\mid s,\tilde{a})}[m_g(s)]. m g ( s ) := E a ∼ π ( ⋅ ∣ s ) [ g ( s , a )] , T ⋆ π g ( s ~ , a ~ ) := E s ∼ P ⋆ ( ⋅ ∣ s , a ~ ) [ m g ( s )] .
Then ∣ m g ( s ) ∣ ≤ B |m_g(s)| \le B ∣ m g ( s ) ∣ ≤ B , and the discounted-occupancy recursion gives
E d π ⋆ [ g ] = γ E d π ⋆ [ T ⋆ π g ] + ( 1 − γ ) E s ∼ d 1 [ m g ( s ) ] . (4) \mathbb{E}_{d_{\pi^\star}}[g] = \gamma\mathbb{E}_{d_{\pi^\star}}[T_\star^\pi g] + (1-\gamma)\mathbb{E}_{s\sim d_1}[m_g(s)]. \tag*{(4)} E d π ⋆ [ g ] = γ E d π ⋆ [ T ⋆ π g ] + ( 1 − γ ) E s ∼ d 1 [ m g ( s )] . (4)
The initialization term is bounded by Cauchy–Schwarz and the definition of C k i n i t ( π ) \mathcal{C}^{\mathrm{init}}_k(\pi) C k init ( π ) :
( 1 − γ ) ∣ E s ∼ d 1 [ m g ( s ) ] ∣ ≤ ( 1 − γ ) C k i n i t ( π ) E ν k p r i [ g 2 ] . (1-\gamma)\left|\mathbb{E}_{s\sim d_1}[m_g(s)]\right| \le\sqrt{(1-\gamma)\mathcal{C}^{\mathrm{init}}_k(\pi)\mathbb{E}_{\nu_k^{\mathrm{pri}}}[g^2]}. ( 1 − γ ) ∣ E s ∼ d 1 [ m g ( s )] ∣ ≤ ( 1 − γ ) C k init ( π ) E ν k pri [ g 2 ] .
Next write v g ⋆ : = ∫ μ ⋆ ( s ) m g ( s ) d s v_g^\star:= \int\mu^\star(s)m_g(s)\,ds v g ⋆ := ∫ μ ⋆ ( s ) m g ( s ) d s . Since T ⋆ π g ( s ~ , a ~ ) = ϕ ⋆ ( s ~ , a ~ ) ⊤ v g ⋆ T_\star^\pi g(\tilde{s},\tilde{a}) = \phi^\star(\tilde{s},\tilde{a})^\top v_g^\star T ⋆ π g ( s ~ , a ~ ) = ϕ ⋆ ( s ~ , a ~ ) ⊤ v g ⋆ , Cauchy–Schwarz gives
E d π ⋆ [ T ⋆ π g ] ≤ E d π ⋆ [ ∥ ϕ ⋆ ( s ~ , a ~ ) ∥ ( Σ k , ⋆ p r i ) − 1 ] ∥ v g ⋆ ∥ Σ k , ⋆ p r i . \mathbb{E}_{d_{\pi^\star}}[T_\star^\pi g] \le\mathbb{E}_{d_{\pi^\star}}\left[\|\phi^\star(\tilde{s},\tilde{a})\|_{(\Sigma_{k,\star}^{\mathrm{pri}})^{-1}}\right]\|v_g^\star\|_{\Sigma_{k,\star}^{\mathrm{pri}}}. E d π ⋆ [ T ⋆ π g ] ≤ E d π ⋆ [ ∥ ϕ ⋆ ( s ~ , a ~ ) ∥ ( Σ k , ⋆ pri ) − 1 ] ∥ v g ⋆ ∥ Σ k , ⋆ pri .
Moreover, ∥ v g ⋆ ∥ 2 ≤ d B \|v_g^\star\|_2 \le\sqrt{d}B ∥ v g ⋆ ∥ 2 ≤ d B , and
∥ v g ⋆ ∥ Σ k , ⋆ p r i 2 = k E ν k p r i [ ( T ⋆ π g ) 2 ] + λ k ∥ v g ⋆ ∥ 2 2 . \|v_g^\star\|_{\Sigma_{k,\star}^{\mathrm{pri}}}^2 = k\mathbb{E}_{\nu_k^{\mathrm{pri}}}[(T_\star^\pi g)^2] + \lambda_k\|v_g^\star\|_2^2. ∥ v g ⋆ ∥ Σ k , ⋆ pri 2 = k E ν k pri [( T ⋆ π g ) 2 ] + λ k ∥ v g ⋆ ∥ 2 2 .
By Jensen and the definition of C k n e x t ( π ) \mathcal{C}^{\mathrm{next}}_k(\pi) C k next ( π ) ,
E ν k p r i [ ( T ⋆ π g ) 2 ] ≤ C k n e x t ( π ) E ν k s e c [ g 2 ] . \mathbb{E}_{\nu_k^{\mathrm{pri}}}[(T_\star^\pi g)^2] \le\mathcal{C}^{\mathrm{next}}_k(\pi)\mathbb{E}_{\nu_k^{\mathrm{sec}}}[g^2]. E ν k pri [( T ⋆ π g ) 2 ] ≤ C k next ( π ) E ν k sec [ g 2 ] .
Substituting these bounds into (4) gives the claim. □
Lemma D.6 (Merged covariance comparison). For each episode k k k , define
Σ k m i x : = k E ν ˉ k p r i [ ϕ ^ k ϕ ^ k ⊤ ] + k E ν ˉ k s e c [ ϕ ^ k ϕ ^ k ⊤ ] + λ k I . \Sigma_k^{\mathrm{mix}} := k\mathbb{E}_{\bar{\nu}_k^{\mathrm{pri}}}[\hat{\phi}_k\hat{\phi}_k^\top] + k\mathbb{E}_{\bar{\nu}_k^{\mathrm{sec}}}[\hat{\phi}_k\hat{\phi}_k^\top] + \lambda_k I. Σ k mix := k E ν ˉ k pri [ ϕ ^ k ϕ ^ k ⊤ ] + k E ν ˉ k sec [ ϕ ^ k ϕ ^ k ⊤ ] + λ k I . There exist universal constants 0 < c 1 ≤ c 2 < ∞ 0 < c_1 \le c_2 < \infty 0 < c 1 ≤ c 2 < ∞ such that, with probability at least 1 − δ 1-\delta 1 − δ , simultaneously for all k ∈ [ K ] k\in[K] k ∈ [ K ] and all ( s , a ) (s,a) ( s , a ) ,
c 1 ∥ ϕ ^ k ( s , a ) ∥ ( Σ k m i x ) − 1 ≤ ∥ ϕ ^ k ( s , a ) ∥ ( Σ ^ k m i x ) − 1 ≤ c 2 ∥ ϕ ^ k ( s , a ) ∥ ( Σ k m i x ) − 1 . c_1\|\hat{\phi}_k(s,a)\|_{(\Sigma_k^{\mathrm{mix}})^{-1}} \le\|\hat{\phi}_k(s,a)\|_{(\widehat{\Sigma}_k^{\mathrm{mix}})^{-1}} \le c_2\|\hat{\phi}_k(s,a)\|_{(\Sigma_k^{\mathrm{mix}})^{-1}}. c 1 ∥ ϕ ^ k ( s , a ) ∥ ( Σ k mix ) − 1 ≤ ∥ ϕ ^ k ( s , a ) ∥ ( Σ k mix ) − 1 ≤ c 2 ∥ ϕ ^ k ( s , a ) ∥ ( Σ k mix ) − 1 . Proof. Fix an episode k k k and first work with an arbitrary candidate feature map ϕ ∈ Γ \phi\in\Gamma ϕ ∈ Γ . Define the episode-block matrix
H i ( ϕ ) : = ϕ ( s i , a i ) ϕ ( s i , a i ) ⊤ + ϕ ( s i ′ , a i ′ ) ϕ ( s i ′ , a i ′ ) ⊤ , i ∈ [ k ] . H_i(\phi) := \phi(s_i,a_i)\phi(s_i,a_i)^\top+ \phi(s_i',a_i')\phi(s_i',a_i')^\top,\qquad i\in[k]. H i ( ϕ ) := ϕ ( s i , a i ) ϕ ( s i , a i ) ⊤ + ϕ ( s i ′ , a i ′ ) ϕ ( s i ′ , a i ′ ) ⊤ , i ∈ [ k ] . Let G i \mathcal{G}_i G i be the sigma-field generated by all tuples collected up to the end of episode i i i , with G 0 \mathcal{G}_0 G 0 the history before episode 1. Define the predictable episode-level conditional mean H ˉ i ( ϕ ) : = E [ H i ( ϕ ) ∣ G i − 1 ] \bar{H}_i(\phi) := \mathbb{E}[H_i(\phi)\mid\mathcal{G}_{i-1}] H ˉ i ( ϕ ) := E [ H i ( ϕ ) ∣ G i − 1 ] . By construction, conditioned on G i − 1 \mathcal{G}_{i-1} G i − 1 the primary tuple is sampled from ν i \nu_i ν i , while the secondary tuple has conditional mean design ν i ′ \nu_i' ν i ′ after marginalizing over the primary transition and the secondary probe action. Therefore
H ˉ i ( ϕ ) = E ν i [ ϕ ϕ ⊤ ] + E ν i ′ [ ϕ ϕ ⊤ ] , ∑ i = 1 k H ˉ i ( ϕ ) = k E ν ˉ k p r i [ ϕ ϕ ⊤ ] + k E ν ˉ k s e c [ ϕ ϕ ⊤ ] . \bar{H}_i(\phi) = \mathbb{E}_{\nu_i}[\phi\phi^\top] + \mathbb{E}_{\nu_i'}[\phi\phi^\top],
\qquad
\sum_{i=1}^{k}\bar{H}_i(\phi) = k\mathbb{E}_{\bar{\nu}_k^{\mathrm{pri}}}[\phi\phi^\top] + k\mathbb{E}_{\bar{\nu}_k^{\mathrm{sec}}}[\phi\phi^\top]. H ˉ i ( ϕ ) = E ν i [ ϕ ϕ ⊤ ] + E ν i ′ [ ϕ ϕ ⊤ ] , i = 1 ∑ k H ˉ i ( ϕ ) = k E ν ˉ k pri [ ϕ ϕ ⊤ ] + k E ν ˉ k sec [ ϕ ϕ ⊤ ] . Now define
Σ k m i x ( ϕ ) : = ∑ i = 1 k H ˉ i ( ϕ ) + λ k I , Σ ^ k m i x ( ϕ ) : = ∑ i = 1 k H i ( ϕ ) + λ k I . \Sigma_k^{\mathrm{mix}}(\phi) := \sum_{i=1}^{k}\bar{H}_i(\phi) + \lambda_k I,\qquad
\widehat{\Sigma}_k^{\mathrm{mix}}(\phi) := \sum_{i=1}^{k}H_i(\phi) + \lambda_k I. Σ k mix ( ϕ ) := i = 1 ∑ k H ˉ i ( ϕ ) + λ k I , Σ k mix ( ϕ ) := i = 1 ∑ k H i ( ϕ ) + λ k I . Under the bounded-feature assumption used throughout the paper, H i ( ϕ ) H_i(\phi) H i ( ϕ ) is positive semidefinite with operator norm at most 2. Hence the matrix martingale
M t ( ϕ ) : = ∑ u = 1 t ( H u ( ϕ ) − H ˉ u ( ϕ ) ) M_t(\phi) := \sum_{u=1}^{t}\left(H_u(\phi)-\bar{H}_u(\phi)\right) M t ( ϕ ) := u = 1 ∑ t ( H u ( ϕ ) − H ˉ u ( ϕ ) ) satisfies the standard bounded-increment conditions for episode-block matrix concentration with respect to { G i } i = 0 k \{\mathcal{G}_i\}_{i=0}^{k} { G i } i = 0 k . In particular, the same self-normalized matrix martingale argument used in the
proof of Lemma A.1 applies to these block increments. After choosing λ k ≳ d ln ( ∣ M ∣ k / δ ) \lambda_k \gtrsim d\ln(|\mathcal{M}|k/\delta) λ k ≳ d ln ( ∣ M ∣ k / δ ) , there is an event of probability at least 1 − δ 1-\delta 1 − δ on which
∥ ( Σ k m i x ( ϕ ) ) − 1 / 2 M k ( ϕ ) ( Σ k m i x ( ϕ ) ) − 1 / 2 ∥ o p ≤ 1 2 , \left\|(\Sigma_k^{\mathrm{mix}}(\phi))^{-1/2}M_k(\phi)(\Sigma_k^{\mathrm{mix}}(\phi))^{-1/2}\right\|_{\mathrm{op}} \le\frac{1}{2}, ( Σ k mix ( ϕ ) ) − 1/2 M k ( ϕ ) ( Σ k mix ( ϕ ) ) − 1/2 op ≤ 2 1 ,
simultaneously for all k ∈ [ K ] k \in[K] k ∈ [ K ] and all ϕ ∈ Γ \phi\in\Gamma ϕ ∈ Γ . Equivalently,
− 1 2 Σ k m i x ( ϕ ) ⪯ M k ( ϕ ) ⪯ 1 2 Σ k m i x ( ϕ ) . -\frac{1}{2}\Sigma_k^{\mathrm{mix}}(\phi) \preceq M_k(\phi) \preceq\frac{1}{2}\Sigma_k^{\mathrm{mix}}(\phi). − 2 1 Σ k mix ( ϕ ) ⪯ M k ( ϕ ) ⪯ 2 1 Σ k mix ( ϕ ) .
Since Σ ^ k m i x ( ϕ ) = Σ k m i x ( ϕ ) + M k ( ϕ ) \widehat{\Sigma}_k^{\mathrm{mix}}(\phi)=\Sigma_k^{\mathrm{mix}}(\phi)+M_k(\phi) Σ k mix ( ϕ ) = Σ k mix ( ϕ ) + M k ( ϕ ) , this immediately yields
1 2 Σ k m i x ( ϕ ) ⪯ Σ ^ k m i x ( ϕ ) ⪯ 3 2 Σ k m i x ( ϕ ) . \frac{1}{2}\Sigma_k^{\mathrm{mix}}(\phi) \preceq\widehat{\Sigma}_k^{\mathrm{mix}}(\phi) \preceq\frac{3}{2}\Sigma_k^{\mathrm{mix}}(\phi). 2 1 Σ k mix ( ϕ ) ⪯ Σ k mix ( ϕ ) ⪯ 2 3 Σ k mix ( ϕ ) .
Specializing to ϕ = ϕ ^ k \phi=\widehat{\phi}_k ϕ = ϕ k and inverting the PSD inequalities gives
1 2 Σ k m i x ⪯ Σ ^ k m i x ⪯ 3 2 Σ k m i x , \frac{1}{2}\Sigma_k^{\mathrm{mix}} \preceq\widehat{\Sigma}_k^{\mathrm{mix}} \preceq\frac{3}{2}\Sigma_k^{\mathrm{mix}}, 2 1 Σ k mix ⪯ Σ k mix ⪯ 2 3 Σ k mix ,
2 3 ( Σ k m i x ) − 1 ⪯ ( Σ ^ k m i x ) − 1 ⪯ 2 ( Σ k m i x ) − 1 . \frac{2}{3}(\Sigma_k^{\mathrm{mix}})^{-1} \preceq(\widehat{\Sigma}_k^{\mathrm{mix}})^{-1} \preceq2(\Sigma_k^{\mathrm{mix}})^{-1}. 3 2 ( Σ k mix ) − 1 ⪯ ( Σ k mix ) − 1 ⪯ 2 ( Σ k mix ) − 1 .
Applying the resulting quadratic-form comparison to ϕ ^ k ( s , a ) \widehat{\phi}_k(s,a) ϕ k ( s , a ) proves
2 3 ∥ ϕ ^ k ( s , a ) ∥ ( Σ k m i x ) − 1 ≤ ∥ ϕ ^ k ( s , a ) ∥ ( Σ ^ k m i x ) − 1 ≤ 2 ∥ ϕ ^ k ( s , a ) ∥ ( Σ k m i x ) − 1 . \sqrt{\frac{2}{3}}\|\widehat{\phi}_k(s,a)\|_{(\Sigma_k^{\mathrm{mix}})^{-1}} \le\|\widehat{\phi}_k(s,a)\|_{(\widehat{\Sigma}_k^{\mathrm{mix}})^{-1}} \le\sqrt{2}\|\widehat{\phi}_k(s,a)\|_{(\Sigma_k^{\mathrm{mix}})^{-1}}. 3 2 ∥ ϕ k ( s , a ) ∥ ( Σ k mix ) − 1 ≤ ∥ ϕ k ( s , a ) ∥ ( Σ k mix ) − 1 ≤ 2 ∥ ϕ k ( s , a ) ∥ ( Σ k mix ) − 1 .
Thus the claim holds with any universal constants c 1 ≤ 2 / 3 c_1 \le\sqrt{2/3} c 1 ≤ 2/3 and c 2 ≥ 2 c_2 \ge\sqrt{2} c 2 ≥ 2 .
Proposition D.7 (Concrete tertiary coverage bound). Under Algorithm 1, for every episode k k k and every policy π \pi π ,
G k , 1 t r i , n e x t ( π ) ≤ Γ k , G k , 2 t r i , n e x t ( π ) ≤ Γ k , G k t r i , n e x t ( π ) ≤ Γ k . \mathcal{G}_{k,1}^{\mathrm{tri,next}}(\pi) \le\Gamma_k,\qquad
\mathcal{G}_{k,2}^{\mathrm{tri,next}}(\pi) \le\Gamma_k,\qquad
\mathcal{G}_{k}^{\mathrm{tri,next}}(\pi) \le\Gamma_k. G k , 1 tri , next ( π ) ≤ Γ k , G k , 2 tri , next ( π ) ≤ Γ k , G k tri , next ( π ) ≤ Γ k . Proof. Because the schedule is nonincreasing,
ϵ i ≥ ϵ k ∀ i ≤ k . \epsilon_i \ge\epsilon_k \qquad\forall i \le k. ϵ i ≥ ϵ k ∀ i ≤ k . Hence every probing kernel among the first k k k episodes satisfies
q i ( a ∣ s ) ≥ ϵ k ∣ A ∣ , q i ′ ( a ∣ s ) ≥ ϵ k ∣ A ∣ , q i ′ ′ ( a ∣ s ) ≥ ϵ k ∣ A ∣ . q_i(a\mid s) \ge\frac{\epsilon_k}{|\mathcal{A}|},\qquad
q_i'(a\mid s) \ge\frac{\epsilon_k}{|\mathcal{A}|},\qquad
q_i''(a\mid s) \ge\frac{\epsilon_k}{|\mathcal{A}|}. q i ( a ∣ s ) ≥ ∣ A ∣ ϵ k , q i ′ ( a ∣ s ) ≥ ∣ A ∣ ϵ k , q i ′′ ( a ∣ s ) ≥ ∣ A ∣ ϵ k . For the first tertiary coefficient, let
μ k , 1 π ( s , a ) : = E ( s ~ , a ~ ) ∼ ν ‾ k p r i ( ⋅ ∣ s , a ~ ) [ π ( a ∣ s ) 1 { s ∈ d s } ] \mu_{k,1}^{\pi}(s,a) := \mathbb{E}_{( \widetilde{s},\widetilde{a})\sim\overline{\nu}_{k}^{\mathrm{pri}}(\cdot\mid s,\widetilde{a})}\left[\pi(a\mid s)1\{s\in ds\}\right] μ k , 1 π ( s , a ) := E ( s , a ) ∼ ν k pri ( ⋅ ∣ s , a ) [ π ( a ∣ s ) 1 { s ∈ d s } ] denote the pushforward state-action measure appearing in the definition of G k , 1 t r i , n e x t ( π ) \mathcal{G}_{k,1}^{\mathrm{tri,next}}(\pi) G k , 1 tri , next ( π ) . Since
ν ‾ k s e c ( s , a ) = 1 k ∑ i = 1 k ρ i ′ ( s ) q i ′ ( a ∣ s ) ≥ ϵ k ∣ A ∣ ⋅ 1 k ∑ i = 1 k ρ i ′ ( s ) , \overline{\nu}_k^{\mathrm{sec}}(s,a) = \frac{1}{k}\sum_{i=1}^{k}\rho_i'(s)q_i'(a\mid s) \ge\frac{\epsilon_k}{|\mathcal{A}|}\cdot\frac{1}{k}\sum_{i=1}^{k}\rho_i'(s), ν k sec ( s , a ) = k 1 i = 1 ∑ k ρ i ′ ( s ) q i ′ ( a ∣ s ) ≥ ∣ A ∣ ϵ k ⋅ k 1 i = 1 ∑ k ρ i ′ ( s ) , we have, pointwise,
μ k , 1 π ( s , a ) ≤ ∣ A ∣ ϵ k ν ‾ k s e c ( s , a ) = Γ k ν ‾ k s e c ( s , a ) . \mu_{k,1}^{\pi}(s,a) \le\frac{|\mathcal{A}|}{\epsilon_k}\overline{\nu}_k^{\mathrm{sec}}(s,a) = \Gamma_k\overline{\nu}_k^{\mathrm{sec}}(s,a). μ k , 1 π ( s , a ) ≤ ϵ k ∣ A ∣ ν k sec ( s , a ) = Γ k ν k sec ( s , a ) . Therefore for every nonnegative measurable g 2 g^2 g 2 ,
E μ k , 1 π [ g 2 ] ≤ Γ k E ν ‾ k s e c [ g 2 ] , \mathbb{E}_{\mu_{k,1}^{\pi}}[g^2] \le\Gamma_k\mathbb{E}_{\overline{\nu}_k^{\mathrm{sec}}}[g^2], E μ k , 1 π [ g 2 ] ≤ Γ k E ν k sec [ g 2 ] , which proves
G k , 1 t r i , n e x t ( π ) ≤ Γ k . \mathcal{G}_{k,1}^{\mathrm{tri,next}}(\pi) \le\Gamma_k. G k , 1 tri , next ( π ) ≤ Γ k .
The proof for E k , 2 t r i , n e x t ( π ) \mathcal{E}^{\mathrm{tri,next}}_{k,2}(\pi) E k , 2 tri , next ( π ) is identical. If
μ k , 2 π ( s , a ) : = E ( s ~ , a ~ ) ∼ ν ˉ k s e c s ∼ P ⋆ ( ⋅ ∣ s ~ , a ~ ) [ π ( a ∣ s ) 1 { s ∈ d s } ] , \mu^{\pi}_{k,2}(s,a) := \mathbb{E}_{\substack{(\tilde{s},\tilde{a})\sim\bar{\nu}^{\mathrm{sec}}_k\\s\sim P^{\star}(\cdot\mid\tilde{s},\tilde{a})}}\left[\pi(a\mid s)\mathbf{1}\{s\in ds\}\right], μ k , 2 π ( s , a ) := E ( s ~ , a ~ ) ∼ ν ˉ k sec s ∼ P ⋆ ( ⋅ ∣ s ~ , a ~ ) [ π ( a ∣ s ) 1 { s ∈ d s } ] ,
then
ν ˉ k t e r ( s , a ) ≥ ϵ k ∣ A ∣ ⋅ 1 k ∑ i = 1 k ρ i ′ ′ ( s ) , \bar{\nu}^{\mathrm{ter}}_k(s,a) \ge\frac{\epsilon_k}{|\mathcal{A}|}\cdot\frac{1}{k}\sum_{i=1}^{k}\rho_i''(s), ν ˉ k ter ( s , a ) ≥ ∣ A ∣ ϵ k ⋅ k 1 i = 1 ∑ k ρ i ′′ ( s ) ,
while by construction
μ k , 2 π ( s , a ) = 1 k ∑ i = 1 k ρ i ′ ′ ( s ) π ( a ∣ s ) . \mu^{\pi}_{k,2}(s,a)=\frac{1}{k}\sum_{i=1}^{k}\rho_i''(s)\pi(a\mid s). μ k , 2 π ( s , a ) = k 1 i = 1 ∑ k ρ i ′′ ( s ) π ( a ∣ s ) .
Using π ( a ∣ s ) ≤ 1 \pi(a\mid s)\le1 π ( a ∣ s ) ≤ 1 and the lower bound on ν ˉ k t e r \bar{\nu}^{\mathrm{ter}}_k ν ˉ k ter , we obtain the pointwise domination
μ k , 2 π ( s , a ) ≤ Γ k ν ˉ k t e r ( s , a ) . \mu^{\pi}_{k,2}(s,a)\le\Gamma_k\bar{\nu}^{\mathrm{ter}}_k(s,a). μ k , 2 π ( s , a ) ≤ Γ k ν ˉ k ter ( s , a ) .
Therefore, for every nonnegative measurable g 2 g^2 g 2 ,
E μ k , 2 π [ g 2 ] ≤ Γ k E ν ˉ k t e r [ g 2 ] , \mathbb{E}_{\mu^{\pi}_{k,2}}[g^2]\le\Gamma_k\mathbb{E}_{\bar{\nu}^{\mathrm{ter}}_k}[g^2], E μ k , 2 π [ g 2 ] ≤ Γ k E ν ˉ k ter [ g 2 ] ,
which proves
E k , 2 t r i , n e x t ( π ) ≤ Γ k . \mathcal{E}^{\mathrm{tri,next}}_{k,2}(\pi)\le\Gamma_k. E k , 2 tri , next ( π ) ≤ Γ k .
Taking the maximum of the two coefficients completes the proof. □ \square □
Lemma D.8 (Tertiary-aware mixed one-step-back inequality). Fix an episode k k k and define f k ( s , a ) : = ∥ P ^ k ( ⋅ ∣ s , a ) − P ⋆ ( ⋅ ∣ s , a ) ∥ 1 f_k(s,a):=\|\widehat{P}_k(\cdot\mid s,a)-P^{\star}(\cdot\mid s,a)\|_1 f k ( s , a ) := ∥ P k ( ⋅ ∣ s , a ) − P ⋆ ( ⋅ ∣ s , a ) ∥ 1 and
Σ k m i x : = k E ν ˉ k p r i [ ϕ ^ k ϕ ^ k ⊤ ] + k E ν ˉ k s e c [ ϕ ^ k ϕ ^ k ⊤ ] + λ k I . \Sigma_k^{\mathrm{mix}}:=k\mathbb{E}_{\bar{\nu}^{\mathrm{pri}}_k}[\hat{\phi}_k\hat{\phi}_k^{\top}]+k\mathbb{E}_{\bar{\nu}^{\mathrm{sec}}_k}[\hat{\phi}_k\hat{\phi}_k^{\top}]+\lambda_k I. Σ k mix := k E ν ˉ k pri [ ϕ ^ k ϕ ^ k ⊤ ] + k E ν ˉ k sec [ ϕ ^ k ϕ ^ k ⊤ ] + λ k I . Also define
M k m i x ( π ) : = E d k π [ ∥ ϕ k ( s ~ , a ~ ) ∥ ( Σ k m i x ) − 1 ] , G k , 23 ( g ) : = E ν ˉ k s e c [ g 2 ] + E ν ˉ k t e r [ g 2 ] . M_k^{\mathrm{mix}}(\pi):=\mathbb{E}_{d_k^{\pi}}\left[\|\phi_k(\tilde{s},\tilde{a})\|_{(\Sigma_k^{\mathrm{mix}})^{-1}}\right],\qquad
G_{k,23}(g):=\mathbb{E}_{\bar{\nu}^{\mathrm{sec}}_k}[g^2]+\mathbb{E}_{\bar{\nu}^{\mathrm{ter}}_k}[g^2]. M k mix ( π ) := E d k π [ ∥ ϕ k ( s ~ , a ~ ) ∥ ( Σ k mix ) − 1 ] , G k , 23 ( g ) := E ν ˉ k sec [ g 2 ] + E ν ˉ k ter [ g 2 ] . Then for every policy π \pi π and every measurable g : S × A → R g:\mathcal{S}\times\mathcal{A}\to\mathbb{R} g : S × A → R with ∥ g ∥ ∞ ≤ B \|g\|_{\infty}\le B ∥ g ∥ ∞ ≤ B ,
∣ E d k π [ g ] ∣ ≲ γ M k m i x ( π ) E k t r i , n e x t ( π ) G k , 23 ( g ) + λ k d B 2 + k B 2 ζ k + ( 1 − γ ) E k i n i t ( π ) E ν ˉ k p r i [ g 2 ] , \left|\mathbb{E}_{d_k^{\pi}}[g]\right|
\lesssim
\sqrt{\gamma M_k^{\mathrm{mix}}(\pi)}
\sqrt{\mathcal{E}^{\mathrm{tri,next}}_k(\pi)G_{k,23}(g)+\lambda_k d B^2+kB^2\zeta_k}
+\sqrt{(1-\gamma)\mathcal{E}^{\mathrm{init}}_k(\pi)\mathbb{E}_{\bar{\nu}^{\mathrm{pri}}_k}[g^2]}, E d k π [ g ] ≲ γ M k mix ( π ) E k tri , next ( π ) G k , 23 ( g ) + λ k d B 2 + k B 2 ζ k + ( 1 − γ ) E k init ( π ) E ν ˉ k pri [ g 2 ] , on the events of Lemmas D.2 and D.4.
Proof. Define
m g ( s ) : = E a ∼ π ( ⋅ ∣ s ) [ g ( s , a ) ] , m_g(s):=\mathbb{E}_{a\sim\pi(\cdot\mid s)}[g(s,a)], m g ( s ) := E a ∼ π ( ⋅ ∣ s ) [ g ( s , a )] , T k π g ( s ~ , a ~ ) : = E s ∼ P ^ k ( ⋅ ∣ s ~ , a ~ ) [ m g ( s ) ] , T_k^{\pi}\,g(\tilde{s},\tilde{a}):=\mathbb{E}_{s\sim\widehat{P}_k(\cdot\mid\tilde{s},\tilde{a})}[m_g(s)], T k π g ( s ~ , a ~ ) := E s ∼ P k ( ⋅ ∣ s ~ , a ~ ) [ m g ( s )] , T π , ⋆ g ( s ~ , a ~ ) : = E s ∼ P ⋆ ( ⋅ ∣ s ~ , a ~ ) [ m g ( s ) ] . T^{\pi,\star}\,g(\tilde{s},\tilde{a}):=\mathbb{E}_{s\sim P^{\star}(\cdot\mid\tilde{s},\tilde{a})}[m_g(s)]. T π , ⋆ g ( s ~ , a ~ ) := E s ∼ P ⋆ ( ⋅ ∣ s ~ , a ~ ) [ m g ( s )] . Since ∥ g ∥ ∞ ≤ B \|g\|_{\infty}\le B ∥ g ∥ ∞ ≤ B , we also have ∥ m g ∥ ∞ ≤ B \|m_g\|_{\infty}\le B ∥ m g ∥ ∞ ≤ B . The discounted occupancy recursion under P ^ k \widehat{P}_k P k gives
E d k π [ g ] = γ E d k π [ T k π g ] + ( 1 − γ ) E s ∼ d 1 [ m g ( s ) ] . \mathbb{E}_{d_k^{\pi}}[g]=\gamma\mathbb{E}_{d_k^{\pi}}[T_k^{\pi}g]+(1-\gamma)\mathbb{E}_{s\sim d_1}[m_g(s)]. E d k π [ g ] = γ E d k π [ T k π g ] + ( 1 − γ ) E s ∼ d 1 [ m g ( s )] . We treat the two terms on the right separately.
For the initialization term, write
E s ∼ d 1 [ m g ( s ) ] = ∑ s , a d 1 ( s ) π ( a ∣ s ) g ( s , a ) . \mathbb{E}_{s\sim d_1}[m_g(s)]=\sum_{s,a}d_1(s)\pi(a\mid s)g(s,a). E s ∼ d 1 [ m g ( s )] = s , a ∑ d 1 ( s ) π ( a ∣ s ) g ( s , a ) . By the definition of E k i n i t ( π ) \mathcal{E}^{\mathrm{init}}_k(\pi) E k init ( π ) ,
( 1 − γ ) d 1 ( s ) π ( a ∣ s ) ≤ E k i n i t ( π ) ν ˉ k p r i ( s , a ) ∀ ( s , a ) (1-\gamma)d_1(s)\pi(a\mid s)\le\mathcal{E}^{\mathrm{init}}_k(\pi)\bar{\nu}^{\mathrm{pri}}_k(s,a)\qquad\forall(s,a) ( 1 − γ ) d 1 ( s ) π ( a ∣ s ) ≤ E k init ( π ) ν ˉ k pri ( s , a ) ∀ ( s , a )
on the support of ν k p r i \nu_k^{\mathrm{pri}} ν k pri . Therefore Cauchy-Schwarz gives
( 1 − γ ) ∣ E s ∼ d 1 [ m g ( s ) ] ∣ ≤ ( 1 − γ ) C k i n i t ( π ) E ν k p r i [ g 2 ] . (1-\gamma)\left|\mathbb{E}_{s\sim d_1}[m_g(s)]\right| \le\sqrt{(1-\gamma)\mathcal{C}_k^{\mathrm{init}}(\pi)\mathbb{E}_{\nu_k^{\mathrm{pri}}}[g^2]}. ( 1 − γ ) ∣ E s ∼ d 1 [ m g ( s )] ∣ ≤ ( 1 − γ ) C k init ( π ) E ν k pri [ g 2 ] .
For the propagated term, define v g P ^ : = ∫ μ ^ k ( s ) m g ( s ) d s v_g^{\hat{P}} := \int\hat{\mu}_k(s)m_g(s)\,ds v g P ^ := ∫ μ ^ k ( s ) m g ( s ) d s . Because T P ^ k π g ( s ~ , a ~ ) = ϕ ^ k ( s ~ , a ~ ) ⊤ v g P ^ T_{\hat{P}_k}^{\pi}g(\tilde{s},\tilde{a})=\hat{\phi}_k(\tilde{s},\tilde{a})^\top v_g^{\hat{P}} T P ^ k π g ( s ~ , a ~ ) = ϕ ^ k ( s ~ , a ~ ) ⊤ v g P ^ , Cauchy-Schwarz under ( Σ k m i x ) − 1 (\Sigma_k^{\mathrm{mix}})^{-1} ( Σ k mix ) − 1 yields
E d P ^ k π [ T P ^ k π g ] ≤ M k m i x ( π ) ∥ v g P ^ ∥ Σ k m i x . \mathbb{E}_{d_{\hat{P}_k}^{\pi}}[T_{\hat{P}_k}^{\pi}g]\le M_k^{\mathrm{mix}}(\pi)\|v_g^{\hat{P}}\|_{\Sigma_k^{\mathrm{mix}}}. E d P ^ k π [ T P ^ k π g ] ≤ M k mix ( π ) ∥ v g P ^ ∥ Σ k mix .
Moreover, ∥ v g P ^ ∥ 2 ≤ d B \|v_g^{\hat{P}}\|_2\le\sqrt{d}B ∥ v g P ^ ∥ 2 ≤ d B , so
∥ v g P ^ ∥ Σ k m i x 2 = k E ν k p r i [ ( T P ^ k π g ) 2 ] + k E ν k s e c [ ( T P ^ k π g ) 2 ] + λ k ∥ v g P ^ ∥ 2 2 . \|v_g^{\hat{P}}\|_{\Sigma_k^{\mathrm{mix}}}^2
= k\mathbb{E}_{\nu_k^{\mathrm{pri}}}[(T_{\hat{P}_k}^{\pi}g)^2]
+k\mathbb{E}_{\nu_k^{\mathrm{sec}}}[(T_{\hat{P}_k}^{\pi}g)^2]
+\lambda_k\|v_g^{\hat{P}}\|_2^2. ∥ v g P ^ ∥ Σ k mix 2 = k E ν k pri [( T P ^ k π g ) 2 ] + k E ν k sec [( T P ^ k π g ) 2 ] + λ k ∥ v g P ^ ∥ 2 2 .
We now control the two quadratic terms. By definition of f k f_k f k ,
∣ T P ^ k π g ( s ~ , a ~ ) − T ⋆ π g ( s ~ , a ~ ) ∣ ≤ B f k ( s ~ , a ~ ) . \left|T_{\hat{P}_k}^{\pi}g(\tilde{s},\tilde{a})-T_{\star}^{\pi}g(\tilde{s},\tilde{a})\right|\le Bf_k(\tilde{s},\tilde{a}). T P ^ k π g ( s ~ , a ~ ) − T ⋆ π g ( s ~ , a ~ ) ≤ B f k ( s ~ , a ~ ) .
Hence, using ( a + b ) 2 ≤ 2 a 2 + 2 b 2 (a+b)^2\le2a^2+2b^2 ( a + b ) 2 ≤ 2 a 2 + 2 b 2 ,
E ν k p r i [ ( T P ^ k π g ) 2 ] ≲ E ν k p r i [ ( T ⋆ π g ) 2 ] + B 2 E ν k p r i [ f k 2 ] , \mathbb{E}_{\nu_k^{\mathrm{pri}}}[(T_{\hat{P}_k}^{\pi}g)^2]
\lesssim\mathbb{E}_{\nu_k^{\mathrm{pri}}}[(T_{\star}^{\pi}g)^2]+B^2\mathbb{E}_{\nu_k^{\mathrm{pri}}}[f_k^2], E ν k pri [( T P ^ k π g ) 2 ] ≲ E ν k pri [( T ⋆ π g ) 2 ] + B 2 E ν k pri [ f k 2 ] ,
E ν k s e c [ ( T P ^ k π g ) 2 ] ≲ E ν k s e c [ ( T ⋆ π g ) 2 ] + B 2 E ν k s e c [ f k 2 ] . \mathbb{E}_{\nu_k^{\mathrm{sec}}}[(T_{\hat{P}_k}^{\pi}g)^2]
\lesssim\mathbb{E}_{\nu_k^{\mathrm{sec}}}[(T_{\star}^{\pi}g)^2]+B^2\mathbb{E}_{\nu_k^{\mathrm{sec}}}[f_k^2]. E ν k sec [( T P ^ k π g ) 2 ] ≲ E ν k sec [( T ⋆ π g ) 2 ] + B 2 E ν k sec [ f k 2 ] .
On the events of Lemmas D.2 and D.4,
E ν k p r i [ f k 2 ] ≲ ζ k , E ν k s e c [ f k 2 ] ≲ ζ k , \mathbb{E}_{\nu_k^{\mathrm{pri}}}[f_k^2]\lesssim\zeta_k,\qquad
\mathbb{E}_{\nu_k^{\mathrm{sec}}}[f_k^2]\lesssim\zeta_k, E ν k pri [ f k 2 ] ≲ ζ k , E ν k sec [ f k 2 ] ≲ ζ k ,
because the primary term is one component of the pathwise average ν k a l l \nu_k^{\mathrm{all}} ν k all , while the secondary term is provided directly by Lemma D.4. This is the only place where a pathwise-to-averaged transfer is used: the generic g g g -dependence remains entirely inside G k , 23 ( g ) = E ν k s e c [ g 2 ] + E ν k t e r [ g 2 ] G_{k,23}(g)=\mathbb{E}_{\nu_k^{\mathrm{sec}}}[g^2]+\mathbb{E}_{\nu_k^{\mathrm{ter}}}[g^2] G k , 23 ( g ) = E ν k sec [ g 2 ] + E ν k ter [ g 2 ] and is handled through the deterministic coverage coefficients.
Next, by Jensen’s inequality,
E ν k p r i [ ( T ⋆ π g ) 2 ] ≤ E μ π k , 1 [ g 2 ] ≤ C k , 1 t r i , n e x t ( π ) E ν k s e c [ g 2 ] , \mathbb{E}_{\nu_k^{\mathrm{pri}}}[(T_{\star}^{\pi}g)^2]\le\mathbb{E}_{\mu_{\pi_{k,1}}}[g^2]\le\mathcal{C}_{k,1}^{\mathrm{tri,next}}(\pi)\mathbb{E}_{\nu_k^{\mathrm{sec}}}[g^2], E ν k pri [( T ⋆ π g ) 2 ] ≤ E μ π k , 1 [ g 2 ] ≤ C k , 1 tri , next ( π ) E ν k sec [ g 2 ] ,
E ν k s e c [ ( T ⋆ π g ) 2 ] ≤ C k , 2 t r i , n e x t ( π ) E ν k t e r [ g 2 ] . \mathbb{E}_{\nu_k^{\mathrm{sec}}}[(T_{\star}^{\pi}g)^2]\le\mathcal{C}_{k,2}^{\mathrm{tri,next}}(\pi)\mathbb{E}_{\nu_k^{\mathrm{ter}}}[g^2]. E ν k sec [( T ⋆ π g ) 2 ] ≤ C k , 2 tri , next ( π ) E ν k ter [ g 2 ] .
Combining the last four displays and using
C k t r i , n e x t ( π ) = max { C k , 1 t r i , n e x t ( π ) , C k , 2 t r i , n e x t ( π ) } , \mathcal{C}_k^{\mathrm{tri,next}}(\pi)=\max\left\{\mathcal{C}_{k,1}^{\mathrm{tri,next}}(\pi),\mathcal{C}_{k,2}^{\mathrm{tri,next}}(\pi)\right\}, C k tri , next ( π ) = max { C k , 1 tri , next ( π ) , C k , 2 tri , next ( π ) } ,
we obtain
∥ v g P ^ ∥ Σ k m i x 2 ≲ k C k t r i , n e x t ( π ) G k , 23 ( g ) + λ k d B 2 + k B 2 ζ k . \|v_g^{\hat{P}}\|_{\Sigma_k^{\mathrm{mix}}}^2
\lesssim k\mathcal{C}_k^{\mathrm{tri,next}}(\pi)G_{k,23}(g)+\lambda_kdB^2+kB^2\zeta_k. ∥ v g P ^ ∥ Σ k mix 2 ≲ k C k tri , next ( π ) G k , 23 ( g ) + λ k d B 2 + k B 2 ζ k .
Substituting this and the initialization bound into the discounted-occupancy recursion proves the claimed inequality. □
Lemma D.9 (Direct optimism for the repaired merged bonus). Fix an episode k k k and a policy π \pi π . Define
b k , m i x , p o p ( s , a ) : = α k m i x ∥ ϕ ^ k ( s , a ) ∥ ( Σ k m i x ) − 1 , b_{k,\mathrm{mix,pop}}(s,a):=\alpha_k^{\mathrm{mix}}\|\hat{\phi}_k(s,a)\|_{(\Sigma_k^{\mathrm{mix}})^{-1}}, b k , mix , pop ( s , a ) := α k mix ∥ ϕ ^ k ( s , a ) ∥ ( Σ k mix ) − 1 , b k , m i x , e m p ( s , a ) : = α k e m p ∥ ϕ ^ k ( s , a ) ∥ ( Σ ^ k m i x ) − 1 . b_{k,\mathrm{mix,emp}}(s,a):=\alpha_k^{\mathrm{emp}}\|\hat{\phi}_k(s,a)\|_{(\hat{\Sigma}_k^{\mathrm{mix}})^{-1}}. b k , mix , emp ( s , a ) := α k emp ∥ ϕ ^ k ( s , a ) ∥ ( Σ ^ k mix ) − 1 . Assume the events of Lemmas D.2, D.4, and the covariance comparison event of Lemma D.6 all hold. If
α k m i x ≳ γ ( k C k t r i , n e x t ( π ) ζ k + λ k d + k ζ k ) , \alpha_k^{\mathrm{mix}}\gtrsim\sqrt{\gamma\left(k\mathcal{C}_k^{\mathrm{tri,next}}(\pi)\zeta_k+\lambda_kd+k\zeta_k\right)}, α k mix ≳ γ ( k C k tri , next ( π ) ζ k + λ k d + k ζ k ) , and
α k e m p ≥ α k m i x / c 1 , \alpha_k^{\mathrm{emp}}\ge\alpha_k^{\mathrm{mix}}/c_1, α k emp ≥ α k mix / c 1 , then
V P ^ k , r + b k , m i x , e m p π − V P ⋆ , r π ≳ − C k i n i t ( π ) ζ k 1 − γ . V_{\hat{P}_k,r+b_{k,\mathrm{mix,emp}}}^{\pi}-V_{P^\star,r}^{\pi}\gtrsim-\sqrt{\frac{\mathcal{C}_k^{\mathrm{init}}(\pi)\zeta_k}{1-\gamma}}. V P ^ k , r + b k , mix , emp π − V P ⋆ , r π ≳ − 1 − γ C k init ( π ) ζ k . Under the nonbinding-cap condition b ^ k = b k , m i x , e m p \hat{b}_k=b_{k,\mathrm{mix,emp}} b ^ k = b k , mix , emp , the same bound holds with b k , m i x , e m p b_{k,\mathrm{mix,emp}} b k , mix , emp replaced by the actual algorithmic bonus b ^ k \hat{b}_k b ^ k .
Proof. Apply the first form of the simulation lemma (Lemma B.4) to the pair ( P ^ k , r + b k , m i x , p o p ) (\widehat{P}_k,r+b_{k,\mathrm{mix,pop}}) ( P k , r + b k , mix , pop ) and ( P ⋆ , r ) (P^\star,r) ( P ⋆ , r ) . This gives
( 1 − γ ) ( V P ^ k , r + b k , m i x , p o p π − V P ⋆ , r π ) = E d P ^ k π [ b k , m i x , p o p ( s , a ) γ E P ^ k [ V P ⋆ , r π ( s ′ ) ] − γ E P ⋆ [ V P ⋆ , r π ( s ′ ) ] ] . (1-\gamma)\left(V^{\pi}_{\widehat{P}_k,r+b_{k,\mathrm{mix,pop}}}-V^{\pi}_{P^\star,r}\right)=\mathbb{E}_{d^{\pi}_{\widehat{P}_k}}\left[b_{k,\mathrm{mix,pop}}(s,a)\gamma\mathbb{E}_{\widehat{P}_k}\left[V^{\pi}_{P^\star,r}(s')\right]-\gamma\mathbb{E}_{P^\star}\left[V^{\pi}_{P^\star,r}(s')\right]\right]. ( 1 − γ ) ( V P k , r + b k , mix , pop π − V P ⋆ , r π ) = E d P k π [ b k , mix , pop ( s , a ) γ E P k [ V P ⋆ , r π ( s ′ ) ] − γ E P ⋆ [ V P ⋆ , r π ( s ′ ) ] ] .
Because the total discounted reward is normalized to lie in [ 0 , 1 ] [0,1] [ 0 , 1 ] , we have
∥ V P ⋆ , r π ∥ ∞ ≤ 1 , ∣ E P ^ k [ V P ⋆ , r π ( s ′ ) ] − E P ⋆ [ V P ⋆ , r π ( s ′ ) ] ∣ ≤ f k ( s , a ) . \left\|V^{\pi}_{P^\star,r}\right\|_\infty\le1,\qquad\left|\mathbb{E}_{\widehat{P}_k}\left[V^{\pi}_{P^\star,r}(s')\right]-\mathbb{E}_{P^\star}\left[V^{\pi}_{P^\star,r}(s')\right]\right|\le f_k(s,a). V P ⋆ , r π ∞ ≤ 1 , E P k [ V P ⋆ , r π ( s ′ ) ] − E P ⋆ [ V P ⋆ , r π ( s ′ ) ] ≤ f k ( s , a ) .
Therefore
( 1 − γ ) ( V P ^ k , r + b k , m i x , p o p π − V P ⋆ , r π ) ≥ E d P ^ k π [ b k , m i x , p o p ( s , a ) ] − γ E d P ^ k π [ f k ] . (1-\gamma)\left(V^{\pi}_{\widehat{P}_k,r+b_{k,\mathrm{mix,pop}}}-V^{\pi}_{P^\star,r}\right)\ge\mathbb{E}_{d^{\pi}_{\widehat{P}_k}}\left[b_{k,\mathrm{mix,pop}}(s,a)\right]-\gamma\mathbb{E}_{d^{\pi}_{\widehat{P}_k}}\left[f_k\right]. ( 1 − γ ) ( V P k , r + b k , mix , pop π − V P ⋆ , r π ) ≥ E d P k π [ b k , mix , pop ( s , a ) ] − γ E d P k π [ f k ] .
For brevity, define
Ω k ( π ) : = k ζ k t r i , n e x t ( π ) ζ k + λ k d + k ζ k . \Omega_k(\pi):=k\zeta_k^{\mathrm{tri,next}}(\pi)\zeta_k+\lambda_kd+k\zeta_k. Ω k ( π ) := k ζ k tri , next ( π ) ζ k + λ k d + k ζ k .
Now apply Lemma D.8 with g = f k g=f_k g = f k and B = 2 B=2 B = 2 . On the events assumed in the lemma,
E d P ^ k π [ f k ] ≲ γ M k m i x ( π ) Ω k ( π ) + ( 1 − γ ) ζ k i n i t ( π ) ζ k . \mathbb{E}_{d^{\pi}_{\widehat{P}_k}}[f_k]\lesssim\sqrt{\gamma M_k^{\mathrm{mix}}(\pi)}\sqrt{\Omega_k(\pi)}+\sqrt{(1-\gamma)\zeta_k^{\mathrm{init}}(\pi)\zeta_k}. E d P k π [ f k ] ≲ γ M k mix ( π ) Ω k ( π ) + ( 1 − γ ) ζ k init ( π ) ζ k .
By the choice of α k m i x \alpha_k^{\mathrm{mix}} α k mix , α k m i x ≳ γ Ω k ( π ) \alpha_k^{\mathrm{mix}}\gtrsim\sqrt{\gamma\Omega_k(\pi)} α k mix ≳ γ Ω k ( π ) , so the first term on the right is absorbed by
E d P ^ k π [ b k , m i x , p o p ] = α k m i x M k m i x ( π ) . \mathbb{E}_{d^{\pi}_{\widehat{P}_k}}[b_{k,\mathrm{mix,pop}}]=\alpha_k^{\mathrm{mix}}M_k^{\mathrm{mix}}(\pi). E d P k π [ b k , mix , pop ] = α k mix M k mix ( π ) .
Consequently,
V P ^ k , r + b k , m i x , p o p π − V P ⋆ , r π ≳ − ζ k i n i t ( π ) ζ k 1 − γ . V^{\pi}_{\widehat{P}_k,r+b_{k,\mathrm{mix,pop}}}-V^{\pi}_{P^\star,r}\gtrsim-\sqrt{\frac{\zeta_k^{\mathrm{init}}(\pi)\zeta_k}{1-\gamma}}. V P k , r + b k , mix , pop π − V P ⋆ , r π ≳ − 1 − γ ζ k init ( π ) ζ k .
Next, Lemma D.6 gives
∥ ϕ ^ k ( s , a ) ∥ ( Σ ^ k m i x ) − 1 ≥ c 1 ∥ ϕ ^ k ( s , a ) ∥ ( Σ k m i x ) − 1 , \left\|\widehat{\phi}_k(s,a)\right\|_{(\widehat{\Sigma}^{\mathrm{mix}}_k)^{-1}}\ge c_1\left\|\widehat{\phi}_k(s,a)\right\|_{(\Sigma^{\mathrm{mix}}_k)^{-1}}, ϕ k ( s , a ) ( Σ k mix ) − 1 ≥ c 1 ϕ k ( s , a ) ( Σ k mix ) − 1 ,
and therefore α k e m p ≥ α k m i x / c 1 \alpha_k^{\mathrm{emp}}\ge\alpha_k^{\mathrm{mix}}/c_1 α k emp ≥ α k mix / c 1 implies the pointwise domination b k , m i x , e m p ( s , a ) ≥ b k , m i x , p o p ( s , a ) b_{k,\mathrm{mix,emp}}(s,a)\ge b_{k,\mathrm{mix,pop}}(s,a) b k , mix , emp ( s , a ) ≥ b k , mix , pop ( s , a ) . Monotonicity of the simulation lemma in the bonus term then yields
V P ^ k , r + b k , m i x , e m p π − V P ⋆ , r π ≳ − ζ k i n i t ( π ) ζ k 1 − γ . V^{\pi}_{\widehat{P}_k,r+b_{k,\mathrm{mix,emp}}}-V^{\pi}_{P^\star,r}\gtrsim-\sqrt{\frac{\zeta_k^{\mathrm{init}}(\pi)\zeta_k}{1-\gamma}}. V P k , r + b k , mix , emp π − V P ⋆ , r π ≳ − 1 − γ ζ k init ( π ) ζ k .
Finally, under the nonbinding-cap condition, b ^ k = b k , m i x , e m p \widehat{b}_k=b_{k,\mathrm{mix,emp}} b k = b k , mix , emp , so the same bound holds for the actual algorithmic bonus.
Final Aggregation
We now combine the optimism step with the deployed-to-probe transfer and the elliptical-potential argument.
Lemma D.10 (Tertiary-aware primal-dual aggregation). Fix an episode k k k and define
Δ k : = ( V P ⋆ , r π ⋆ ( s 1 ) − V P ⋆ , r π k ( s 1 ) ) + Y k ( V P ⋆ , g π ⋆ ( s 1 ) − V P ⋆ , g π k ( s 1 ) ) , \Delta_k:=\left(V^{\pi^\star}_{P^\star,r}(s_1)-V^{\pi_k}_{P^\star,r}(s_1)\right)+Y_k\left(V^{\pi^\star}_{P^\star,g}(s_1)-V^{\pi_k}_{P^\star,g}(s_1)\right), Δ k := ( V P ⋆ , r π ⋆ ( s 1 ) − V P ⋆ , r π k ( s 1 ) ) + Y k ( V P ⋆ , g π ⋆ ( s 1 ) − V P ⋆ , g π k ( s 1 ) ) , A k : = E d P ⋆ π k [ ∥ ϕ ⋆ ( s ~ , a ~ ) ∥ ( Σ k , ⋆ p r i ) − 1 ] , A_k:=\mathbb{E}_{d^{\pi_k}_{P^\star}}\left[\left\|\phi^\star(\widetilde{s},\widetilde{a})\right\|_{(\Sigma^{\mathrm{pri}}_{k,\star})^{-1}}\right], A k := E d P ⋆ π k [ ∥ ϕ ⋆ ( s , a ) ∥ ( Σ k , ⋆ pri ) − 1 ] , f k ( s , a ) : = ∥ P ^ k ( ⋅ ∣ s , a ) − P ⋆ ( ⋅ ∣ s , a ) ∥ 1 . f_k(s,a):=\left\|\widehat{P}_k(\cdot\mid s,a)-P^\star(\cdot\mid s,a)\right\|_1. f k ( s , a ) := P k ( ⋅ ∣ s , a ) − P ⋆ ( ⋅ ∣ s , a ) 1 . Assume, for every episode k k k ,
the cap is nonbinding:
b ‾ ≥ max 1 ≤ j ≤ K α j e m p λ j ; \overline{b}\ge\max_{1\le j\le K}\frac{\alpha_j^{\mathrm{emp}}}{\sqrt{\lambda_j}}; b ≥ 1 ≤ j ≤ K max λ j α j emp ; the events of Lemmas D.2 and D.4 hold;
Lemma D.9 holds with π = π ⋆ \pi= \pi^\star π = π ⋆ for both the reward objective r r r and the utility objective g g g .
Then, under Algorithm 1,
Δ k ≲ 1 + ξ 1 − γ A k γ d Γ k ( α k e m p ) 2 + γ λ k d ˉ 2 + ( 1 + ξ ) b ˉ ( 1 − γ ) 2 A k γ ( k Γ k ζ k + λ k d ) + 1 + ξ 1 − γ α k e m p Γ k d ( 1 − γ ) k + ( 1 + ξ ) b ˉ ( 1 − γ ) 2 Γ k ζ k ( 1 − γ ) + ( 1 + ξ ) Γ k ζ k 1 − γ . \Delta_k \lesssim\frac{1+\xi}{1-\gamma} A_k \sqrt{\gamma d \Gamma_k\left(\alpha_k^{\mathrm{emp}}\right)^2+\gamma\lambda_k\bar{d}^2}+\frac{(1+\xi)\bar{b}}{(1-\gamma)^2}A_k\sqrt{\gamma(k\Gamma_k\zeta_k+\lambda_k d)}
+\frac{1+\xi}{1-\gamma}\alpha_k^{\mathrm{emp}}\sqrt{\frac{\Gamma_k d(1-\gamma)}{k}}+\frac{(1+\xi)\bar{b}}{(1-\gamma)^2}\sqrt{\Gamma_k\zeta_k(1-\gamma)}+(1+\xi)\sqrt{\frac{\Gamma_k\zeta_k}{1-\gamma}}. Δ k ≲ 1 − γ 1 + ξ A k γ d Γ k ( α k emp ) 2 + γ λ k d ˉ 2 + ( 1 − γ ) 2 ( 1 + ξ ) b ˉ A k γ ( k Γ k ζ k + λ k d ) + 1 − γ 1 + ξ α k emp k Γ k d ( 1 − γ ) + ( 1 − γ ) 2 ( 1 + ξ ) b ˉ Γ k ζ k ( 1 − γ ) + ( 1 + ξ ) 1 − γ Γ k ζ k .
Consequently,
∑ k = 1 K Δ k ≲ 1 + ξ 1 − γ ∑ k = 1 K A k γ d Γ k ( α k e m p ) 2 + γ λ k d ˉ 2 + ( 1 + ξ ) b ˉ ( 1 − γ ) 2 ∑ k = 1 K A k γ ( k Γ k ζ k + λ k d ) + 1 + ξ 1 − γ ∑ k = 1 K α k e m p Γ k d ( 1 − γ ) k + ( 1 + ξ ) b ˉ ( 1 − γ ) 2 ∑ k = 1 K Γ k ζ k ( 1 − γ ) + ( 1 + ξ ) ∑ k = 1 K Γ k ζ k 1 − γ . \sum_{k=1}^{K}\Delta_k \lesssim\frac{1+\xi}{1-\gamma}\sum_{k=1}^{K}A_k\sqrt{\gamma d\Gamma_k\left(\alpha_k^{\mathrm{emp}}\right)^2+\gamma\lambda_k\bar{d}^2}+\frac{(1+\xi)\bar{b}}{(1-\gamma)^2}\sum_{k=1}^{K}A_k\sqrt{\gamma(k\Gamma_k\zeta_k+\lambda_kd)}
+\frac{1+\xi}{1-\gamma}\sum_{k=1}^{K}\alpha_k^{\mathrm{emp}}\sqrt{\frac{\Gamma_kd(1-\gamma)}{k}}
+\frac{(1+\xi)\bar{b}}{(1-\gamma)^2}\sum_{k=1}^{K}\sqrt{\Gamma_k\zeta_k(1-\gamma)}+(1+\xi)\sum_{k=1}^{K}\sqrt{\frac{\Gamma_k\zeta_k}{1-\gamma}}. k = 1 ∑ K Δ k ≲ 1 − γ 1 + ξ k = 1 ∑ K A k γ d Γ k ( α k emp ) 2 + γ λ k d ˉ 2 + ( 1 − γ ) 2 ( 1 + ξ ) b ˉ k = 1 ∑ K A k γ ( k Γ k ζ k + λ k d ) + 1 − γ 1 + ξ k = 1 ∑ K α k emp k Γ k d ( 1 − γ ) + ( 1 − γ ) 2 ( 1 + ξ ) b ˉ k = 1 ∑ K Γ k ζ k ( 1 − γ ) + ( 1 + ξ ) k = 1 ∑ K 1 − γ Γ k ζ k .
Proof. Fix an episode k k k . By the definition of Δ k \Delta_k Δ k ,
Δ k = ( V P ⋆ , r ⋆ − V P k π , r ⋆ ) + Y k ( V P ⋆ , g ⋆ − V P k π , g ⋆ ) . \Delta_k=\left(V_{P^\star,r}^\star-V_{P^\pi_k,r}^\star\right)+Y_k\left(V_{P^\star,g}^\star-V_{P^\pi_k,g}^\star\right). Δ k = ( V P ⋆ , r ⋆ − V P k π , r ⋆ ) + Y k ( V P ⋆ , g ⋆ − V P k π , g ⋆ ) .
Because 0 ≤ Y k ≤ ξ 0\leq Y_k\leq\xi 0 ≤ Y k ≤ ξ and Lemma D.9 holds at π ⋆ \pi^\star π ⋆ for both the reward and utility objectives,
V P ⋆ , r ⋆ ≤ V P k , r + b ^ k π ⋆ + ζ k i n i t ( π ⋆ ) ζ k 1 − γ , V P ⋆ , g ⋆ ≤ V P k , g + b ^ k π ⋆ + ζ k i n i t ( π ⋆ ) ζ k 1 − γ . V_{P^\star,r}^\star\leq V_{P_k,r+\hat b_k}^{\pi^\star}+\sqrt{\frac{\zeta_k^{\mathrm{init}}(\pi^\star)\zeta_k}{1-\gamma}},\qquad V_{P^\star,g}^\star\leq V_{P_k,g+\hat b_k}^{\pi^\star}+\sqrt{\frac{\zeta_k^{\mathrm{init}}(\pi^\star)\zeta_k}{1-\gamma}}. V P ⋆ , r ⋆ ≤ V P k , r + b ^ k π ⋆ + 1 − γ ζ k init ( π ⋆ ) ζ k , V P ⋆ , g ⋆ ≤ V P k , g + b ^ k π ⋆ + 1 − γ ζ k init ( π ⋆ ) ζ k .
Since every primary probing distribution has the uniform floor ϵ k / ∣ A ∣ \epsilon_k/|\mathcal{A}| ϵ k /∣ A ∣ and ρ i ( s ) = d P ⋆ π i − 1 ( s ) ≥ ( 1 − γ ) d 1 ( s ) \rho_i(s)=d_{P^\star}^{\pi_{i-1}}(s)\geq(1-\gamma)d_1(s) ρ i ( s ) = d P ⋆ π i − 1 ( s ) ≥ ( 1 − γ ) d 1 ( s ) , we have ν ˉ k p r i ( s , a ) ≥ ( 1 − γ ) d 1 ( s ) ϵ k / ∣ A ∣ \bar{\nu}_k^{\mathrm{pri}}(s,a)\geq(1-\gamma)d_1(s)\epsilon_k/|\mathcal{A}| ν ˉ k pri ( s , a ) ≥ ( 1 − γ ) d 1 ( s ) ϵ k /∣ A ∣ , hence ζ k i n i t ( π ⋆ ) ≤ Γ k \zeta_k^{\mathrm{init}}(\pi^\star)\leq\Gamma_k ζ k init ( π ⋆ ) ≤ Γ k . Therefore
Δ k ≤ Δ ~ k + ( 1 + ξ ) Γ k ζ k 1 − γ . \Delta_k\leq\widetilde{\Delta}_k+(1+\xi)\sqrt{\frac{\Gamma_k\zeta_k}{1-\gamma}}. Δ k ≤ Δ k + ( 1 + ξ ) 1 − γ Γ k ζ k .
where
Δ ~ k : = ( V P k , r + b ^ k π ⋆ − V P ⋆ , r π ⋆ ) + Y k ( V P k , g + b ^ k π ⋆ − V P ⋆ , g π ⋆ ) . \widetilde{\Delta}_k:=\left(V_{P_k,r+\hat b_k}^{\pi^\star}-V_{P^\star,r}^{\pi^\star}\right)+Y_k\left(V_{P_k,g+\hat b_k}^{\pi^\star}-V_{P^\star,g}^{\pi^\star}\right). Δ k := ( V P k , r + b ^ k π ⋆ − V P ⋆ , r π ⋆ ) + Y k ( V P k , g + b ^ k π ⋆ − V P ⋆ , g π ⋆ ) .
By the definition of π k \pi_k π k as the maximizer of the optimistic empirical Lagrangian,
V P k , r + b ^ k π ⋆ + Y k V P k , g + b ^ k π ⋆ ≤ V P k , r + b ^ k π k + Y k V P k , g + b ^ k π k . V_{P_k,r+\hat b_k}^{\pi^\star}+Y_kV_{P_k,g+\hat b_k}^{\pi^\star}\leq V_{P_k,r+\hat b_k}^{\pi_k}+Y_kV_{P_k,g+\hat b_k}^{\pi_k}. V P k , r + b ^ k π ⋆ + Y k V P k , g + b ^ k π ⋆ ≤ V P k , r + b ^ k π k + Y k V P k , g + b ^ k π k .
Hence
Δ ~ k ≤ ( V P k , r + b ^ k π k − V P ⋆ , r π ⋆ ) + Y k ( V P k , g + b ^ k π k − V P ⋆ , g π ⋆ ) . \widetilde{\Delta}_k\leq\left(V_{P_k,r+\hat b_k}^{\pi_k}-V_{P^\star,r}^{\pi^\star}\right)+Y_k\left(V_{P_k,g+\hat b_k}^{\pi_k}-V_{P^\star,g}^{\pi^\star}\right). Δ k ≤ ( V P k , r + b ^ k π k − V P ⋆ , r π ⋆ ) + Y k ( V P k , g + b ^ k π k − V P ⋆ , g π ⋆ ) .
Define
W k , r : = V P k , r + b ^ k π k , W k , g : = V P k , g + b ^ k π k , W_{k,r}:=V_{P_k,r+\hat b_k}^{\pi_k},\qquad W_{k,g}:=V_{P_k,g+\hat b_k}^{\pi_k}, W k , r := V P k , r + b ^ k π k , W k , g := V P k , g + b ^ k π k ,
Δ P k , P ⋆ W ( s , a ) : = E P k [ W ( s ′ ) ] − E P ⋆ [ W ( s ′ ) ] . \Delta_{P_k,P^\star}W(s,a):=\mathbb{E}_{P_k}[W(s')]-\mathbb{E}_{P^\star}[W(s')]. Δ P k , P ⋆ W ( s , a ) := E P k [ W ( s ′ )] − E P ⋆ [ W ( s ′ )] .
Apply the second form of the simulation lemma (Lemma B.4) to each bracket. For the reward term,
V P k , r + b ^ k π k − V P ⋆ , r π k = 1 1 − γ E d P ⋆ π k [ b ^ k ( s , a ) + γ Δ P k , P ⋆ W k , r ( s , a ) ] . V_{P_k,r+\hat b_k}^{\pi_k}-V_{P^\star,r}^{\pi_k}=\frac{1}{1-\gamma}\mathbb{E}_{d_{P^\star}^{\pi_k}}\left[\hat b_k(s,a)+\gamma\Delta_{P_k,P^\star}W_{k,r}(s,a)\right]. V P k , r + b ^ k π k − V P ⋆ , r π k = 1 − γ 1 E d P ⋆ π k [ b ^ k ( s , a ) + γ Δ P k , P ⋆ W k , r ( s , a ) ] .
The analogous identity holds for the utility term. Since ∥ b ^ k ∥ ∞ ≤ b ˉ \lVert\hat b_k\rVert_\infty\leq\bar b ∥ b ^ k ∥ ∞ ≤ b ˉ , the normalized reward and utility assumptions imply, up to universal constants,
∥ V P k , r + b ^ k π k ∥ ∞ ≲ b ˉ 1 − γ , ∥ V P k , g + b ^ k π k ∥ ∞ ≲ b ˉ 1 − γ . (5) \left\lVert V_{P_k,r+\hat b_k}^{\pi_k}\right\rVert_\infty\lesssim\frac{\bar b}{1-\gamma},\qquad\left\lVert V_{P_k,g+\hat b_k}^{\pi_k}\right\rVert_\infty\lesssim\frac{\bar b}{1-\gamma}. \tag*{(5)} V P k , r + b ^ k π k ∞ ≲ 1 − γ b ˉ , V P k , g + b ^ k π k ∞ ≲ 1 − γ b ˉ . (5)
Thus
Δ ~ k ≲ 1 + ξ 1 − γ E d p k π [ b ^ k ] + ( 1 + ξ ) b ‾ ( 1 − γ ) 2 E d p k π [ f k ] . \widetilde{\Delta}_k \lesssim\frac{1+\xi}{1-\gamma}\mathbb{E}_{d^\pi_{p_k}}[\widehat{b}_k] + \frac{(1+\xi)\overline{b}}{(1-\gamma)^2}\mathbb{E}_{d^\pi_{p_k}}[f_k]. Δ k ≲ 1 − γ 1 + ξ E d p k π [ b k ] + ( 1 − γ ) 2 ( 1 + ξ ) b E d p k π [ f k ] .
Combining this with the previous reduction gives
Δ k ≲ 1 + ξ 1 − γ E d p k π [ b ^ k ] + ( 1 + ξ ) b ‾ ( 1 − γ ) 2 E d p k π [ f k ] + ( 1 + ξ ) Γ k ζ k 1 − γ . \Delta_k \lesssim\frac{1+\xi}{1-\gamma}\mathbb{E}_{d^\pi_{p_k}}[\widehat{b}_k] + \frac{(1+\xi)\overline{b}}{(1-\gamma)^2}\mathbb{E}_{d^\pi_{p_k}}[f_k] + (1+\xi)\sqrt{\frac{\Gamma_k\zeta_k}{1-\gamma}}. Δ k ≲ 1 − γ 1 + ξ E d p k π [ b k ] + ( 1 − γ ) 2 ( 1 + ξ ) b E d p k π [ f k ] + ( 1 + ξ ) 1 − γ Γ k ζ k .
We now bound the two expectations separately.
For the bonus term, Lemma D.6 implies
b ^ k ( s , a ) ≲ α k e m p ∥ ϕ ^ k ( s , a ) ∥ ( Σ k m i x ) − 1 . \widehat{b}_k(s,a) \lesssim\alpha_k^{\mathrm{emp}}\lVert\widehat{\phi}_k(s,a)\rVert_{(\Sigma_k^{\mathrm{mix}})^{-1}}. b k ( s , a ) ≲ α k emp ∥ ϕ k ( s , a ) ∥ ( Σ k mix ) − 1 .
Moreover,
k E ν ˉ k p r i [ ∥ ϕ ^ k ∥ ( Σ k m i x ) − 1 2 ] ≤ d , k\mathbb{E}_{\bar{\nu}_k^{\mathrm{pri}}}\left[\lVert\widehat{\phi}_k\rVert^2_{(\Sigma_k^{\mathrm{mix}})^{-1}}\right]\le d, k E ν ˉ k pri [ ∥ ϕ k ∥ ( Σ k mix ) − 1 2 ] ≤ d ,
and the same bound holds with ν ˉ k p r i \bar{\nu}_k^{\mathrm{pri}} ν ˉ k pri replaced by ν ˉ k s e c \bar{\nu}_k^{\mathrm{sec}} ν ˉ k sec , because each term is upper-bounded by
Tr ( ( k E ν ˉ k p r i [ ϕ ^ k ϕ ^ k ⊤ ] + k E ν ˉ k s e c [ ϕ ^ k ϕ ^ k ⊤ ] ) ( Σ k m i x ) − 1 ) ≤ d . \operatorname{Tr}\left(\left(k\mathbb{E}_{\bar{\nu}_k^{\mathrm{pri}}}[\widehat{\phi}_k\widehat{\phi}_k^\top]+k\mathbb{E}_{\bar{\nu}_k^{\mathrm{sec}}}[\widehat{\phi}_k\widehat{\phi}_k^\top]\right)(\Sigma_k^{\mathrm{mix}})^{-1}\right)\le d. Tr ( ( k E ν ˉ k pri [ ϕ k ϕ k ⊤ ] + k E ν ˉ k sec [ ϕ k ϕ k ⊤ ] ) ( Σ k mix ) − 1 ) ≤ d .
Applying Lemma D.5 with g = b ^ k g=\widehat{b}_k g = b k and B = b ‾ B=\overline{b} B = b , and using C k n e x t ( π k ) ≤ Γ k C_k^{\mathrm{next}}(\pi_k)\le\Gamma_k C k next ( π k ) ≤ Γ k and C k i n i t ( π k ) ≤ Γ k C_k^{\mathrm{init}}(\pi_k)\le\Gamma_k C k init ( π k ) ≤ Γ k , gives
E d p k π [ b ^ k ] ≲ A k γ d Γ k ( α k e m p ) 2 + γ λ k b ‾ 2 + α k e m p Γ k d ( 1 − γ ) k . \mathbb{E}_{d^\pi_{p_k}}[\widehat{b}_k]\lesssim A_k\sqrt{\gamma d\Gamma_k(\alpha_k^{\mathrm{emp}})^2+\gamma\lambda_k\overline{b}^2+\alpha_k^{\mathrm{emp}}\sqrt{\frac{\Gamma_kd(1-\gamma)}{k}}}. E d p k π [ b k ] ≲ A k γ d Γ k ( α k emp ) 2 + γ λ k b 2 + α k emp k Γ k d ( 1 − γ ) .
For the model-error term, apply Lemma D.5 with g = f k g=f_k g = f k and B = 2 B=2 B = 2 . By Lemmas D.2 and D.4,
E ν ˉ k p r i [ f k 2 ] ≲ ζ k , E ν ˉ k s e c [ f k 2 ] ≲ ζ k , \mathbb{E}_{\bar{\nu}_k^{\mathrm{pri}}}[f_k^2]\lesssim\zeta_k,\qquad\mathbb{E}_{\bar{\nu}_k^{\mathrm{sec}}}[f_k^2]\lesssim\zeta_k, E ν ˉ k pri [ f k 2 ] ≲ ζ k , E ν ˉ k sec [ f k 2 ] ≲ ζ k ,
and again C k n e x t ( π k ) ≤ Γ k C_k^{\mathrm{next}}(\pi_k)\le\Gamma_k C k next ( π k ) ≤ Γ k and C k i n i t ( π k ) ≤ Γ k C_k^{\mathrm{init}}(\pi_k)\le\Gamma_k C k init ( π k ) ≤ Γ k . Hence
E d p k π [ f k ] ≲ A k γ ( k Γ k ζ k + λ k d ) + Γ k ζ k ( 1 − γ ) . \mathbb{E}_{d^\pi_{p_k}}[f_k]\lesssim A_k\sqrt{\gamma(k\Gamma_k\zeta_k+\lambda_kd)}+\sqrt{\Gamma_k\zeta_k(1-\gamma)}. E d p k π [ f k ] ≲ A k γ ( k Γ k ζ k + λ k d ) + Γ k ζ k ( 1 − γ ) .
Substituting the last two displays into the bound on Δ k \Delta_k Δ k yields the episode-wise inequality claimed in the lemma. Summing over k = 1 , … , K k=1,\ldots,K k = 1 , … , K proves the cumulative bound.
Lemma D.11 (Tertiary-aware deployed-to-probe potential reduction). Under Algorithm 1,
Ψ K : = ∑ k = 1 K A k ≲ 1 λ 1 + d ln ( 1 + K d λ 1 ) ∑ k = 2 K Λ k . \Psi_K:=\sum_{k=1}^{K}A_k\lesssim\frac{1}{\sqrt{\lambda_1}}+\sqrt{d\ln\left(1+\frac{K}{d\lambda_1}\right)\sum_{k=2}^{K}\Lambda_k}. Ψ K := k = 1 ∑ K A k ≲ λ 1 1 + d ln ( 1 + d λ 1 K ) k = 2 ∑ K Λ k . In particular, since Λ k ≤ Γ k \Lambda_k\le\Gamma_k Λ k ≤ Γ k ,
Ψ K ≲ 1 λ 1 + d ln ( 1 + K d λ 1 ) ∑ k = 2 K Γ k . \Psi_K\lesssim\frac{1}{\sqrt{\lambda_1}}+\sqrt{d\ln\left(1+\frac{K}{d\lambda_1}\right)\sum_{k=2}^{K}\Gamma_k}. Ψ K ≲ λ 1 1 + d ln ( 1 + d λ 1 K ) k = 2 ∑ K Γ k .
Proof. The first episode is trivial:
A 1 ≤ 1 λ 1 , A_1\le\frac{1}{\sqrt{\lambda_1}}, A 1 ≤ λ 1 1 ,
because ( Σ 1 , ⋆ p r i ) − 1 ⪯ λ 1 − 1 I (\Sigma_{1,\star}^{\mathrm{pri}})^{-1}\preceq\lambda_1^{-1}I ( Σ 1 , ⋆ pri ) − 1 ⪯ λ 1 − 1 I and ∥ ϕ ⋆ ( s , a ) ∥ 2 ≤ 1 \lVert\phi^\star(s,a)\rVert_2\le1 ∥ ϕ ⋆ ( s , a ) ∥ 2 ≤ 1 .
Now fix k ≤ K − 1 k\le K-1 k ≤ K − 1 . The relation
q k + 1 ( a ∣ s ) = ( 1 − ϵ k + 1 ) π k ( a ∣ s ) + ϵ k + 1 U ( a ) q_{k+1}(a\mid s)=(1-\epsilon_{k+1})\pi_k(a\mid s)+\epsilon_{k+1}U(a) q k + 1 ( a ∣ s ) = ( 1 − ϵ k + 1 ) π k ( a ∣ s ) + ϵ k + 1 U ( a )
implies the pointwise ratio bound
π k ( a ∣ s ) q k + 1 ( a ∣ s ) ≤ Λ k + 1 . \frac{\pi_k(a\mid s)}{q_{k+1}(a\mid s)}\le\Lambda_{k+1}. q k + 1 ( a ∣ s ) π k ( a ∣ s ) ≤ Λ k + 1 .
Therefore
A k ≤ Λ k + 1 E ( s , a ) ∼ ν k + 1 [ ∥ ϕ ⋆ ( s , a ) ∥ ( Σ k , ⋆ p r i ) − 1 2 ] . A_k \le\sqrt{\Lambda_{k+1}}\sqrt{\mathbb{E}_{(s,a)\sim\nu_{k+1}}\left[\left\|\phi^\star(s,a)\right\|^2_{(\Sigma^{\mathrm{pri}}_{k,\star})^{-1}}\right]}. A k ≤ Λ k + 1 E ( s , a ) ∼ ν k + 1 [ ∥ ϕ ⋆ ( s , a ) ∥ ( Σ k , ⋆ pri ) − 1 2 ] .
Indeed,
A k = E ( s , a ) ∼ d ϕ ⋆ π k ⋆ [ ∥ ϕ ⋆ ( s , a ) ∥ ( Σ k , ⋆ p r i ) − 1 ] ≤ E d ϕ ⋆ π k ⋆ [ d ϕ ⋆ π k ⋆ ( s , a ) ν k + 1 ( s , a ) ] E ( s , a ) ∼ ν k + 1 [ ∥ ϕ ⋆ ( s , a ) ∥ ( Σ k , ⋆ p r i ) − 1 2 ] ≤ Λ k + 1 E ( s , a ) ∼ ν k + 1 [ ∥ ϕ ⋆ ( s , a ) ∥ ( Σ k , ⋆ p r i ) − 1 2 ] . \begin{aligned}
A_k = \mathbb{E}_{(s,a)\sim d^{\pi_k^\star}_{\phi^\star}}\left[\left\|\phi^\star(s,a)\right\|_{(\Sigma^{\mathrm{pri}}_{k,\star})^{-1}}\right] \\
&\le\sqrt{\mathbb{E}_{d^{\pi_k^\star}_{\phi^\star}}\left[\frac{d^{\pi_k^\star}_{\phi^\star}(s,a)}{\nu_{k+1}(s,a)}\right]}\sqrt{\mathbb{E}_{(s,a)\sim\nu_{k+1}}\left[\left\|\phi^\star(s,a)\right\|^2_{(\Sigma^{\mathrm{pri}}_{k,\star})^{-1}}\right]} \\
&\le\sqrt{\Lambda_{k+1}}\sqrt{\mathbb{E}_{(s,a)\sim\nu_{k+1}}\left[\left\|\phi^\star(s,a)\right\|^2_{(\Sigma^{\mathrm{pri}}_{k,\star})^{-1}}\right]}.
\end{aligned} A k = E ( s , a ) ∼ d ϕ ⋆ π k ⋆ [ ∥ ϕ ⋆ ( s , a ) ∥ ( Σ k , ⋆ pri ) − 1 ] ≤ E d ϕ ⋆ π k ⋆ [ ν k + 1 ( s , a ) d ϕ ⋆ π k ⋆ ( s , a ) ] E ( s , a ) ∼ ν k + 1 [ ∥ ϕ ⋆ ( s , a ) ∥ ( Σ k , ⋆ pri ) − 1 2 ] ≤ Λ k + 1 E ( s , a ) ∼ ν k + 1 [ ∥ ϕ ⋆ ( s , a ) ∥ ( Σ k , ⋆ pri ) − 1 2 ] .
For each k ≤ K − 1 k \le K-1 k ≤ K − 1 , define
G k + 1 : = E ( s , a ) ∼ ν k + 1 [ ϕ ⋆ ( s , a ) ϕ ⋆ ( s , a ) ⊤ ] . G_{k+1}:=\mathbb{E}_{(s,a)\sim\nu_{k+1}}\left[\phi^\star(s,a)\phi^\star(s,a)^\top\right]. G k + 1 := E ( s , a ) ∼ ν k + 1 [ ϕ ⋆ ( s , a ) ϕ ⋆ ( s , a ) ⊤ ] .
Then
Σ k , ⋆ p r i = λ k I + ∑ i = 1 k G i . \Sigma^{\mathrm{pri}}_{k,\star}=\lambda_k I+\sum_{i=1}^{k}G_i. Σ k , ⋆ pri = λ k I + i = 1 ∑ k G i .
Since λ k + 1 ≥ λ k \lambda_{k+1}\ge\lambda_k λ k + 1 ≥ λ k , we may apply Lemma B.2 to the PSD increments G 2 , … , G K G_2,\ldots,G_K G 2 , … , G K and obtain
∑ k = 1 K − 1 Tr ( G k + 1 ( Σ k , ⋆ p r i ) − 1 ) ≲ ln det ( Σ K , ⋆ p r i ) − ln det ( λ 1 I ) . \sum_{k=1}^{K-1}\operatorname{Tr}\left(G_{k+1}(\Sigma^{\mathrm{pri}}_{k,\star})^{-1}\right)\lesssim\ln\det(\Sigma^{\mathrm{pri}}_{K,\star})-\ln\det(\lambda_1 I). k = 1 ∑ K − 1 Tr ( G k + 1 ( Σ k , ⋆ pri ) − 1 ) ≲ ln det ( Σ K , ⋆ pri ) − ln det ( λ 1 I ) .
Because
E ( s , a ) ∼ ν k + 1 [ ∥ ϕ ⋆ ( s , a ) ∥ ( Σ k , ⋆ p r i ) − 1 2 ] = Tr ( G k + 1 ( Σ k , ⋆ p r i ) − 1 ) , \mathbb{E}_{(s,a)\sim\nu_{k+1}}\left[\left\|\phi^\star(s,a)\right\|^2_{(\Sigma^{\mathrm{pri}}_{k,\star})^{-1}}\right]=\operatorname{Tr}\left(G_{k+1}(\Sigma^{\mathrm{pri}}_{k,\star})^{-1}\right), E ( s , a ) ∼ ν k + 1 [ ∥ ϕ ⋆ ( s , a ) ∥ ( Σ k , ⋆ pri ) − 1 2 ] = Tr ( G k + 1 ( Σ k , ⋆ pri ) − 1 ) ,
we deduce
∑ k = 1 K − 1 E ( s , a ) ∼ ν k + 1 [ ∥ ϕ ⋆ ( s , a ) ∥ ( Σ k , ⋆ p r i ) − 1 2 ] ≲ ln det ( Σ K , ⋆ p r i ) − ln det ( λ 1 I ) . \sum_{k=1}^{K-1}\mathbb{E}_{(s,a)\sim\nu_{k+1}}\left[\left\|\phi^\star(s,a)\right\|^2_{(\Sigma^{\mathrm{pri}}_{k,\star})^{-1}}\right]\lesssim\ln\det(\Sigma^{\mathrm{pri}}_{K,\star})-\ln\det(\lambda_1 I). k = 1 ∑ K − 1 E ( s , a ) ∼ ν k + 1 [ ∥ ϕ ⋆ ( s , a ) ∥ ( Σ k , ⋆ pri ) − 1 2 ] ≲ ln det ( Σ K , ⋆ pri ) − ln det ( λ 1 I ) .
Lemma B.3 and the bound ∥ ϕ ⋆ ( s , a ) ∥ 2 ≤ 1 \|\phi^\star(s,a)\|_2\le1 ∥ ϕ ⋆ ( s , a ) ∥ 2 ≤ 1 imply
ln det ( Σ K , ⋆ p r i ) − ln det ( λ 1 I ) ≲ d ln ( 1 + K d λ 1 ) . \ln\det(\Sigma^{\mathrm{pri}}_{K,\star})-\ln\det(\lambda_1 I)\lesssim d\ln\left(1+\frac{K}{d\lambda_1}\right). ln det ( Σ K , ⋆ pri ) − ln det ( λ 1 I ) ≲ d ln ( 1 + d λ 1 K ) .
Finally, Cauchy-Schwarz gives
∑ k = 1 K − 1 A k ≤ ∑ k = 2 K Λ k ∑ k = 1 K − 1 E ( s , a ) ∼ ν k + 1 [ ∥ ϕ ⋆ ( s , a ) ∥ ( Σ k , ⋆ p r i ) − 1 2 ] ≲ d ln ( 1 + K d λ 1 ) ∑ k = 2 K Λ k . \sum_{k=1}^{K-1}A_k\le\sqrt{\sum_{k=2}^{K}\Lambda_k}\sqrt{\sum_{k=1}^{K-1}\mathbb{E}_{(s,a)\sim\nu_{k+1}}\left[\left\|\phi^\star(s,a)\right\|^2_{(\Sigma^{\mathrm{pri}}_{k,\star})^{-1}}\right]}\lesssim\sqrt{d\ln\left(1+\frac{K}{d\lambda_1}\right)\sum_{k=2}^{K}\Lambda_k}. k = 1 ∑ K − 1 A k ≤ k = 2 ∑ K Λ k k = 1 ∑ K − 1 E ( s , a ) ∼ ν k + 1 [ ∥ ϕ ⋆ ( s , a ) ∥ ( Σ k , ⋆ pri ) − 1 2 ] ≲ d ln ( 1 + d λ 1 K ) k = 2 ∑ K Λ k .
Adding the bound on A 1 A_1 A 1 proves the result.
Lemma D.12 (Summable optimistic overestimation of the utility). Fix an episode k k k and define
U ^ k : = V P ^ k , g ^ + b ^ k π k ( s 1 ) , U k : = V P ⋆ , g π k ( s 1 ) . \widehat{U}_k:=V^{\pi_k}_{\widehat{P}_k,\widehat{g}+\widehat{b}_k}(s_1),\qquad U_k:=V^{\pi_k}_{P^\star,g}(s_1). U k := V P k , g + b k π k ( s 1 ) , U k := V P ⋆ , g π k ( s 1 ) . Assume Algorithm 1, the nonbinding-cap condition, and the events of Lemmas D.2, D.4, and D.6. Then, episode-wise,
[ U ^ k − U k ] + ≲ 1 1 − γ A k α k e m p γ d Γ k + b ˉ 1 − γ A k γ λ k d [\widehat{U}_k-U_k]_+\lesssim\frac{1}{1-\gamma}A_k\alpha_k^{\mathrm{emp}}\sqrt{\gamma d\Gamma_k}+\frac{\bar b}{1-\gamma}A_k\sqrt{\gamma\lambda_k d} [ U k − U k ] + ≲ 1 − γ 1 A k α k emp γ d Γ k + 1 − γ b ˉ A k γ λ k d + b ˉ ( 1 − γ ) 2 A k γ ( k Γ k ζ k + λ k d ) + 1 1 − γ α k e m p Γ k d ( 1 − γ ) k + b ˉ ( 1 − γ ) 2 Γ k ζ k ( 1 − γ ) . \qquad+\frac{\bar b}{(1-\gamma)^2}A_k\sqrt{\gamma(k\Gamma_k\zeta_k+\lambda_k d)}
+\frac{1}{1-\gamma}\alpha_k^{\mathrm{emp}}\sqrt{\frac{\Gamma_k d(1-\gamma)}{k}}+\frac{\bar b}{(1-\gamma)^2}\sqrt{\Gamma_k\zeta_k(1-\gamma)}. + ( 1 − γ ) 2 b ˉ A k γ ( k Γ k ζ k + λ k d ) + 1 − γ 1 α k emp k Γ k d ( 1 − γ ) + ( 1 − γ ) 2 b ˉ Γ k ζ k ( 1 − γ ) . Consequently, under the tuning of Proposition D.13 ,
∑ k = 1 K [ U ^ k − U k ] + ≤ O ~ ( b ˉ d 2 ( 1 − γ ) 2 [ Γ K ∑ k = 2 K Λ k + ∑ k = 1 K Γ k k ] ) . \sum_{k=1}^{K}[\widehat{U}_k-U_k]_+\le\widetilde{O}\left(\frac{\bar b d^2}{(1-\gamma)^2}\left[\sqrt{\Gamma_K\sum_{k=2}^{K}\Lambda_k}+\sum_{k=1}^{K}\frac{\Gamma_k}{\sqrt{k}}\right]\right). k = 1 ∑ K [ U k − U k ] + ≤ O ( 1 − γ ) 2 b ˉ d 2 Γ K k = 2 ∑ K Λ k + k = 1 ∑ K k Γ k .
Proof. Apply the second form of the simulation lemma (Lemma B.4 ) to the utility objective:
U ^ k − U k = 1 1 − γ E d P ⋆ π k [ b ^ k ( s , a ) + γ Δ P ^ k , P ⋆ W k , g ( s , a ) ] . \widehat{U}_k-U_k=\frac{1}{1-\gamma}\mathbb{E}_{d_{P^\star}^{\pi_k}}\left[\widehat{b}_k(s,a)+\gamma\Delta_{\widehat{P}_k,P^\star}W_{k,g}(s,a)\right]. U k − U k = 1 − γ 1 E d P ⋆ π k [ b k ( s , a ) + γ Δ P k , P ⋆ W k , g ( s , a ) ] .
where
W k , g : = V P ^ k , g + b ^ k π k , Δ P ^ k , P ⋆ W ( s , a ) : = E P ^ k [ W ( s ′ ) ] − E P ⋆ [ W ( s ′ ) ] . W_{k,g}:=V_{\widehat{P}_k,g+\widehat{b}_k}^{\pi_k},\qquad\Delta_{\widehat{P}_k,P^\star}W(s,a):=\mathbb{E}_{\widehat{P}_k}[W(s')]-\mathbb{E}_{P^\star}[W(s')]. W k , g := V P k , g + b k π k , Δ P k , P ⋆ W ( s , a ) := E P k [ W ( s ′ )] − E P ⋆ [ W ( s ′ )] .
Since ∥ b ^ k ∥ ∞ ≤ b ˉ \|\widehat{b}_k\|_\infty\leq\bar{b} ∥ b k ∥ ∞ ≤ b ˉ and the utility is normalized to lie in [ 0 , 1 ] [0,1] [ 0 , 1 ] , we have ∥ W k , g ∥ ∞ ≲ b ˉ / ( 1 − γ ) \|W_{k,g}\|_\infty\lesssim\bar{b}/(1-\gamma) ∥ W k , g ∥ ∞ ≲ b ˉ / ( 1 − γ ) . Therefore
[ U ^ k − U k ] + ≤ 1 1 − γ E d P ⋆ π k [ b ^ k ] + b ˉ ( 1 − γ ) 2 E d P ⋆ π k [ f k ] . [\widehat{U}_k-U_k]_+\leq\frac{1}{1-\gamma}\mathbb{E}_{d_{P^\star}^{\pi_k}}[\widehat{b}_k]+\frac{\bar{b}}{(1-\gamma)^2}\mathbb{E}_{d_{P^\star}^{\pi_k}}[f_k]. [ U k − U k ] + ≤ 1 − γ 1 E d P ⋆ π k [ b k ] + ( 1 − γ ) 2 b ˉ E d P ⋆ π k [ f k ] .
The bonus term is controlled exactly as in the proof of Lemma D.10:
E d P ⋆ π k [ b ^ k ] ≲ A k γ d Γ k ( α k e m p ) 2 + γ λ k b ˉ 2 + α k e m p Γ k d ( 1 − γ ) k , \mathbb{E}_{d_{P^\star}^{\pi_k}}[\widehat{b}_k]\lesssim A_k\sqrt{\gamma d\Gamma_k(\alpha_k^{\mathrm{emp}})^2}+\gamma\lambda_k\bar{b}^2+\alpha_k^{\mathrm{emp}}\sqrt{\frac{\Gamma_k d(1-\gamma)}{k}}, E d P ⋆ π k [ b k ] ≲ A k γ d Γ k ( α k emp ) 2 + γ λ k b ˉ 2 + α k emp k Γ k d ( 1 − γ ) ,
and the model-mismatch term obeys
E d P ⋆ π k [ f k ] ≲ A k γ ( k Γ k ζ k + λ k d ) + Γ k ζ k ( 1 − γ ) . \mathbb{E}_{d_{P^\star}^{\pi_k}}[f_k]\lesssim A_k\sqrt{\gamma(k\Gamma_k\zeta_k+\lambda_k d)}+\sqrt{\Gamma_k\zeta_k(1-\gamma)}. E d P ⋆ π k [ f k ] ≲ A k γ ( k Γ k ζ k + λ k d ) + Γ k ζ k ( 1 − γ ) .
Substituting the last two displays proves the episode-wise bound.
For the cumulative statement, one substitutes the same choices of α k e m p \alpha_k^{\mathrm{emp}} α k emp , λ k \lambda_k λ k , ζ k \zeta_k ζ k as in Proposition D.13 . The two A k A_k A k -weighted sums are then of order
O ~ ( b ˉ d 2 ( 1 − γ ) 2 Γ K Ψ K ) , Ψ K : = ∑ k = 1 K A k , \widetilde{O}\left(\frac{\bar{b}d^2}{(1-\gamma)^2}\sqrt{\Gamma_K}\Psi_K\right),\qquad\Psi_K:=\sum_{k=1}^{K}A_k, O ( ( 1 − γ ) 2 b ˉ d 2 Γ K Ψ K ) , Ψ K := k = 1 ∑ K A k ,
while the two remaining sums are of order
O ~ ( b ˉ d 2 ( 1 − γ ) 2 ∑ k = 1 K Γ k k ) . \widetilde{O}\left(\frac{\bar{b}d^2}{(1-\gamma)^2}\sum_{k=1}^{K}\frac{\Gamma_k}{\sqrt{k}}\right). O ( ( 1 − γ ) 2 b ˉ d 2 k = 1 ∑ K k Γ k ) .
Finally Lemma D.11 gives
Γ K Ψ K ≤ O ~ ( Γ K ∑ k = 2 K Λ k ) , \sqrt{\Gamma_K}\Psi_K\leq\widetilde{O}\left(\sqrt{\Gamma_K\sum_{k=2}^{K}\Lambda_k}\right), Γ K Ψ K ≤ O Γ K k = 2 ∑ K Λ k ,
which proves the cumulative bound.
Proposition D.13 (Theorem-level regret and soft-violation dependence). Assume Algorithm 1 and suppose the high-probability events required by Lemmas D.9 , D.10, and D.11 hold simultaneously. Choose
α k m i x = α k e m p = O ~ ( ( Γ k + d 2 ) γ ln ( ∣ M ∣ k / δ ) ) , \alpha_k^{\mathrm{mix}}=\alpha_k^{\mathrm{emp}}=\widetilde{O}\left(\sqrt{(\Gamma_k+d^2)\gamma\ln(|\mathcal{M}|k/\delta)}\right), α k mix = α k emp = O ( ( Γ k + d 2 ) γ ln ( ∣ M ∣ k / δ ) ) , λ k = Θ ~ ( Γ k + d 2 d ln ( ∣ M ∣ k / δ ) ) , \lambda_k=\widetilde{\Theta}\left(\frac{\Gamma_k+d^2}{d}\ln(|\mathcal{M}|k/\delta)\right), λ k = Θ ( d Γ k + d 2 ln ( ∣ M ∣ k / δ ) ) , and choose
η = 2 ( 1 − γ ) θ b ˉ K , ξ = Θ ~ ( 1 / θ ) , \eta=\frac{2(1-\gamma)}{\theta\bar{b}\sqrt{K}},\qquad\xi=\widetilde{\Theta}(1/\theta), η = θ b ˉ K 2 ( 1 − γ ) , ξ = Θ ( 1/ θ ) , and assume the cap is nonbinding:
b ˉ ≥ max 1 ≤ j ≤ K α j e m p λ j . \bar{b}\geq\max_{1\leq j\leq K}\frac{\alpha_j^{\mathrm{emp}}}{\sqrt{\lambda_j}}. b ˉ ≥ 1 ≤ j ≤ K max λ j α j emp . For a nonnegative sequence z 2 , … , z K z_2,\ldots,z_K z 2 , … , z K , define
Ξ K ( z ) : = Γ K ∑ k = 2 K z k + ∑ k = 1 K Γ k k . \Xi_K(z):=\sqrt{\Gamma_K\sum_{k=2}^{K}z_k+\sum_{k=1}^{K}\frac{\Gamma_k}{\sqrt{k}}}. Ξ K ( z ) := Γ K k = 2 ∑ K z k + k = 1 ∑ K k Γ k .
Then
∑ k = 1 K Δ k ≤ O ~ ( ( 1 + ξ ) b ˉ d 2 ( 1 − γ ) 2 Ξ K ( Λ ) ) . \sum_{k=1}^{K} \Delta_k \le\widetilde{O}\left(\frac{(1+\xi)\bar{b}d^2}{(1-\gamma)^2}\Xi_K(\Lambda)\right). k = 1 ∑ K Δ k ≤ O ( ( 1 − γ ) 2 ( 1 + ξ ) b ˉ d 2 Ξ K ( Λ ) ) .
Consequently, taking the comparator dual level to be Y = 0 Y=0 Y = 0 yields
Regret r e t u r n ( K ) ≤ O ~ ( b ˉ d 2 θ ( 1 − γ ) 2 Ξ K ( Λ ) ) . \operatorname{Regret}^{\mathrm{return}}(K) \le\widetilde{O}\left(\frac{\bar{b}d^2}{\theta(1-\gamma)^2}\Xi_K(\Lambda)\right). Regret return ( K ) ≤ O ( θ ( 1 − γ ) 2 b ˉ d 2 Ξ K ( Λ ) ) .
Moreover,
Violation s o f t r e t u r n ( K ) ≤ O ~ ( b ˉ d 2 ( 1 − γ ) 2 Ξ K ( Λ ) ) . \operatorname{Violation}^{\mathrm{return}}_{\mathrm{soft}}(K) \le\widetilde{O}\left(\frac{\bar{b}d^2}{(1-\gamma)^2}\Xi_K(\Lambda)\right). Violation soft return ( K ) ≤ O ( ( 1 − γ ) 2 b ˉ d 2 Ξ K ( Λ ) ) .
In particular, since Λ k ≤ Γ k = ∣ A ∣ / ϵ k \Lambda_k \le\Gamma_k = |\mathcal{A}|/\epsilon_k Λ k ≤ Γ k = ∣ A ∣/ ϵ k ,
Regret r e t u r n ( K ) ≤ O ~ ( b ˉ d 2 θ ( 1 − γ ) 2 Ξ K ( Γ ) ) . \operatorname{Regret}^{\mathrm{return}}(K) \le\widetilde{O}\left(\frac{\bar{b}d^2}{\theta(1-\gamma)^2}\Xi_K(\Gamma)\right). Regret return ( K ) ≤ O ( θ ( 1 − γ ) 2 b ˉ d 2 Ξ K ( Γ ) ) .
Likewise,
Violation s o f t r e t u r n ( K ) ≤ O ~ ( b ˉ d 2 ( 1 − γ ) 2 Ξ K ( Γ ) ) . \operatorname{Violation}^{\mathrm{return}}_{\mathrm{soft}}(K) \le\widetilde{O}\left(\frac{\bar{b}d^2}{(1-\gamma)^2}\Xi_K(\Gamma)\right). Violation soft return ( K ) ≤ O ( ( 1 − γ ) 2 b ˉ d 2 Ξ K ( Γ ) ) .
For ϵ k = k − p \epsilon_k = k^{-p} ϵ k = k − p with p ∈ ( 0 , 1 / 2 ) p \in(0,1/2) p ∈ ( 0 , 1/2 ) , this becomes
Regret r e t u r n ( K ) ≤ O ~ ( K 1 / 2 + p ) , Violation s o f t r e t u r n ( K ) ≤ O ~ ( K 1 / 2 + p ) . \operatorname{Regret}^{\mathrm{return}}(K) \le\widetilde{O}\left(K^{1/2+p}\right), \qquad\operatorname{Violation}^{\mathrm{return}}_{\mathrm{soft}}(K) \le\widetilde{O}\left(K^{1/2+p}\right). Regret return ( K ) ≤ O ( K 1/2 + p ) , Violation soft return ( K ) ≤ O ( K 1/2 + p ) .
up to problem-dependent prefactors, with the regret bound carrying the extra θ − 1 \theta^{-1} θ − 1 dependence. Moreover, under the displayed tuning of λ k \lambda_k λ k ,
α k e m p λ k = O ~ ( d ) , \frac{\alpha_k^{\mathrm{emp}}}{\sqrt{\lambda_k}} = \widetilde{O}(\sqrt{d}), λ k α k emp = O ( d ) ,
so the nonbinding-cap condition is compatible with any b ˉ = Ω ~ ( d ) \bar{b} = \widetilde{\Omega}(\sqrt{d}) b ˉ = Ω ( d ) and does not introduce any additional K K K -dependence.
Proof. Start from Lemma D.10 and substitute the tunings
α k e m p = O ~ ( ( Γ k + d 2 ) γ ln ( ∣ M ∣ k / δ ) ) , λ k = Θ ~ ( Γ k + d 2 d ln ( ∣ M ∣ k / δ ) ) , \alpha_k^{\mathrm{emp}} = \widetilde{O}\left(\sqrt{(\Gamma_k+d^2)\gamma\ln(|\mathcal{M}|k/\delta)}\right), \qquad\lambda_k = \widetilde{\Theta}\left(\frac{\Gamma_k+d^2}{d}\ln(|\mathcal{M}|k/\delta)\right), α k emp = O ( ( Γ k + d 2 ) γ ln ( ∣ M ∣ k / δ ) ) , λ k = Θ ( d Γ k + d 2 ln ( ∣ M ∣ k / δ ) ) ,
together with
ζ k = O ~ ( 1 / k ) . \zeta_k = \widetilde{O}(1/k). ζ k = O ( 1/ k ) .
Under these choices, λ k d = Θ ~ ( ( Γ k + d 2 ) ln ( ∣ M ∣ k / δ ) ) \lambda_k d = \widetilde{\Theta}((\Gamma_k+d^2)\ln(|\mathcal{M}|k/\delta)) λ k d = Θ (( Γ k + d 2 ) ln ( ∣ M ∣ k / δ )) , so every place in Lemma D.10 where λ k d \lambda_k d λ k d appears is of the same order as the dominant ( Γ k + d 2 ) (\Gamma_k+d^2) ( Γ k + d 2 ) term already present in α k e m p 2 \alpha_k^{\mathrm{emp}2} α k emp 2 . Therefore enlarging λ k \lambda_k λ k to make the cap nonbinding does not change the final K K K -dependence.
We also check the cap condition explicitly:
α k e m p λ k = O ~ ( ( Γ k + d 2 ) ln ( ∣ M ∣ k / δ ) ( ( Γ k + d 2 ) / d ) ln ( ∣ M ∣ k / δ ) ) = O ~ ( d ) . \frac{\alpha_k^{\mathrm{emp}}}{\sqrt{\lambda_k}} = \widetilde{O}\left(\sqrt{\frac{(\Gamma_k+d^2)\ln(|\mathcal{M}|k/\delta)}{((\Gamma_k+d^2)/d)\ln(|\mathcal{M}|k/\delta)}}\right) = \widetilde{O}(\sqrt{d}). λ k α k emp = O ( (( Γ k + d 2 ) / d ) ln ( ∣ M ∣ k / δ ) ( Γ k + d 2 ) ln ( ∣ M ∣ k / δ ) ) = O ( d ) .
Hence any choice b ˉ = Ω ~ ( d ) \bar{b} = \widetilde{\Omega}(\sqrt{d}) b ˉ = Ω ( d ) is sufficient to ensure the nonbinding-cap condition uniformly over k ≤ K k \le K k ≤ K .
Because Γ k \Gamma_k Γ k is nondecreasing, the dominant dependence in every episode-wise term may be upper bounded by Γ K \Gamma_K Γ K inside the coefficients that multiply A k A_k A k . Consequently the two A k A_k A k -weighted sums in Lemma D.10 are both of order
O ~ ( ( 1 + ξ ) b ˉ d 2 ( 1 − γ ) 2 Γ K Ψ K ) , Ψ K : = ∑ k = 1 K A k , \widetilde{O}\left(\frac{(1+\xi)\bar{b}d^2}{(1-\gamma)^2}\sqrt{\Gamma_K}\,\Psi_K\right), \qquad\Psi_K := \sum_{k=1}^{K} A_k, O ( ( 1 − γ ) 2 ( 1 + ξ ) b ˉ d 2 Γ K Ψ K ) , Ψ K := k = 1 ∑ K A k ,
The three remaining sums in Lemma D.10 contain no factor A k A_k A k ; after substituting the same choices of α k e m p \alpha_k^{\mathrm{emp}} α k emp , λ k \lambda_k λ k , ζ k \zeta_k ζ k , each of them is bounded by
O ~ ( ( 1 + ξ ) b ˉ d 2 ( 1 − γ ) 2 ∑ k = 1 K Γ k k ) . \widetilde{O}\left(\frac{(1+\xi)\bar{b}d^2}{(1-\gamma)^2}\sum_{k=1}^{K}\frac{\Gamma_k}{\sqrt{k}}\right). O ( ( 1 − γ ) 2 ( 1 + ξ ) b ˉ d 2 k = 1 ∑ K k Γ k ) .
Now apply Lemma D.11, which gives
Ψ K ≲ 1 λ 1 + d ln ( 1 + K d λ 1 ) ∑ k = 2 K Λ k . \Psi_K \lesssim\frac{1}{\sqrt{\lambda_1}}+\sqrt{d\ln\left(1+\frac{K}{d\lambda_1}\right)\sum_{k=2}^{K}\Lambda_k}. Ψ K ≲ λ 1 1 + d ln ( 1 + d λ 1 K ) k = 2 ∑ K Λ k .
After absorbing the harmless initial term 1 / λ 1 1/\sqrt{\lambda_1} 1/ λ 1 into the polylogarithmic factor, we obtain
Γ K Ψ K ≤ O ~ ( Γ K ∑ k = 2 K Λ k ) . \sqrt{\Gamma_K}\Psi_K\leq\widetilde{O}\left(\sqrt{\Gamma_K\sum_{k=2}^{K}\Lambda_k}\right). Γ K Ψ K ≤ O Γ K k = 2 ∑ K Λ k .
This is exactly the quantity Ξ K ( Λ ) \Xi_K(\Lambda) Ξ K ( Λ ) defined in the proposition, so
∑ k = 1 K Δ k ≤ O ~ ( ( 1 + ξ ) b ˉ d 2 ( 1 − γ ) 2 Ξ K ( Λ ) ) . \sum_{k=1}^{K}\Delta_k\leq\widetilde{O}\left(\frac{(1+\xi)\bar{b}d^2}{(1-\gamma)^2}\Xi_K(\Lambda)\right). k = 1 ∑ K Δ k ≤ O ( ( 1 − γ ) 2 ( 1 + ξ ) b ˉ d 2 Ξ K ( Λ ) ) .
The projection step in Algorithm 1 is identical to that of the main algorithm, so the proof of Lemma 2.4 applies verbatim and gives, for every Y ∈ [ 0 , ξ ] Y\in[0,\xi] Y ∈ [ 0 , ξ ] ,
∑ k = 1 K ( Y − Y k ) ( b − V P ^ k , g + b ^ k π k ( s 1 ) ) ≤ Y 2 2 η + η b ˉ 2 K 2 ( 1 − γ ) 2 . \sum_{k=1}^{K}(Y-Y_k)\left(b-V_{\widehat{P}_k,g+\widehat{b}_k}^{\pi_k}(s_1)\right)\leq\frac{Y^2}{2\eta}+\frac{\eta\bar{b}^2K}{2(1-\gamma)^2}. k = 1 ∑ K ( Y − Y k ) ( b − V P k , g + b k π k ( s 1 ) ) ≤ 2 η Y 2 + 2 ( 1 − γ ) 2 η b ˉ 2 K .
Setting Y = 0 Y=0 Y = 0 and combining this with Lemma D.10 yields
∑ k = 1 K ( V P ⋆ , r π ⋆ ( s 1 ) − V P ⋆ , r π k ( s 1 ) ) ≲ ∑ k = 1 K Δ k + η b ˉ 2 K 2 ( 1 − γ ) 2 . \sum_{k=1}^{K}\left(V_{P^\star,r}^{\pi^\star}(s_1)-V_{P^\star,r}^{\pi_k}(s_1)\right)\lesssim\sum_{k=1}^{K}\Delta_k+\frac{\eta\bar{b}^2K}{2(1-\gamma)^2}. k = 1 ∑ K ( V P ⋆ , r π ⋆ ( s 1 ) − V P ⋆ , r π k ( s 1 ) ) ≲ k = 1 ∑ K Δ k + 2 ( 1 − γ ) 2 η b ˉ 2 K .
Substituting the bound on ∑ k Δ k \sum_k\Delta_k ∑ k Δ k gives a right-hand side
O ~ ( b ˉ d 2 θ ( 1 − γ ) 2 Ξ K ( Λ ) ) + O ( b ˉ K θ ( 1 − γ ) ) , \widetilde{O}\left(\frac{\bar{b}d^2}{\theta(1-\gamma)^2}\Xi_K(\Lambda)\right)+O\left(\frac{\bar{b}\sqrt{K}}{\theta(1-\gamma)}\right), O ( θ ( 1 − γ ) 2 b ˉ d 2 Ξ K ( Λ ) ) + O ( θ ( 1 − γ ) b ˉ K ) ,
where η = 2 ( 1 − γ ) θ b ˉ K \eta=\frac{2(1-\gamma)}{\theta\bar{b}\sqrt{K}} η = θ b ˉ K 2 ( 1 − γ ) . The second term is lower order than the first whenever ϵ k = k − p \epsilon_k=k^{-p} ϵ k = k − p with p > 0 p>0 p > 0 , since then Ξ K ( Λ ) = Ω ~ ( K 1 / 2 + p ) \Xi_K(\Lambda)=\widetilde{\Omega}(K^{1/2+p}) Ξ K ( Λ ) = Ω ( K 1/2 + p ) . This proves the displayed regret bound.
We now turn to the soft-violation term. Define
U ^ k : = V P ^ k , g + b ^ k π k ( s 1 ) , U k : = V P ⋆ , g π ⋆ ( s 1 ) , S K : = ∑ k = 1 K ( b − U k ) . \widehat{U}_k:=V_{\widehat{P}_k,g+\widehat{b}_k}^{\pi_k}(s_1),\qquad U_k:=V_{P^\star,g}^{\pi^\star}(s_1),\qquad S_K:=\sum_{k=1}^{K}(b-U_k). U k := V P k , g + b k π k ( s 1 ) , U k := V P ⋆ , g π ⋆ ( s 1 ) , S K := k = 1 ∑ K ( b − U k ) .
Because V P ⋆ , g π ⋆ ( s 1 ) ≥ b V_{P^\star,g}^{\pi^\star}(s_1)\geq b V P ⋆ , g π ⋆ ( s 1 ) ≥ b , we have
V P ⋆ , r π ⋆ ( s 1 ) − V P ^ k , r π k ( s 1 ) + Y k ( b − U k ) ≤ Δ k ∀ k ∈ [ K ] . V_{P^\star,r}^{\pi^\star}(s_1)-V_{\widehat{P}_k,r}^{\pi_k}(s_1)+Y_k(b-U_k)\leq\Delta_k\qquad\forall k\in[K]. V P ⋆ , r π ⋆ ( s 1 ) − V P k , r π k ( s 1 ) + Y k ( b − U k ) ≤ Δ k ∀ k ∈ [ K ] .
Hence, for every Y ∈ [ 0 , ξ ] Y\in[0,\xi] Y ∈ [ 0 , ξ ] ,
Regret r e t u r n ( K ) + Y S K ≤ ∑ k = 1 K Δ k + ∑ k = 1 K ( Y − Y k ) ( b − U ^ k ) + ∑ k = 1 K ( Y − Y k ) ( U ^ k − U k ) . \operatorname{Regret}^{\mathrm{return}}(K)+YS_K\leq\sum_{k=1}^{K}\Delta_k+\sum_{k=1}^{K}(Y-Y_k)(b-\widehat{U}_k)+\sum_{k=1}^{K}(Y-Y_k)(\widehat{U}_k-U_k). Regret return ( K ) + Y S K ≤ k = 1 ∑ K Δ k + k = 1 ∑ K ( Y − Y k ) ( b − U k ) + k = 1 ∑ K ( Y − Y k ) ( U k − U k ) .
Choose Y = ξ Y=\xi Y = ξ when S K > 0 S_K>0 S K > 0 and Y = 0 Y=0 Y = 0 otherwise. Then
Regret r e t u r n ( K ) + ξ [ S K ] + ≤ ∑ k = 1 K Δ k + ξ 2 2 η + η b ˉ 2 K 2 ( 1 − γ ) 2 + ξ ∑ k = 1 K [ U ^ k − U k ] + . \operatorname{Regret}^{\mathrm{return}}(K)+\xi[S_K]_+\leq\sum_{k=1}^{K}\Delta_k+\frac{\xi^2}{2\eta}+\frac{\eta\bar{b}^2K}{2(1-\gamma)^2}+\xi\sum_{k=1}^{K}[\widehat{U}_k-U_k]_+. Regret return ( K ) + ξ [ S K ] + ≤ k = 1 ∑ K Δ k + 2 η ξ 2 + 2 ( 1 − γ ) 2 η b ˉ 2 K + ξ k = 1 ∑ K [ U k − U k ] + .
Now use Lemma D.12 together with the bound on ∑ k Δ k \sum_k \Delta_k ∑ k Δ k . This gives
Regret r e t u r n ( K ) + ξ [ S K ] + ≤ O ~ ( ξ b ˉ d 2 ( 1 − γ ) 2 Ξ K ( Λ ) ) + O ( ξ b ˉ K 1 − γ ) . \operatorname{Regret}^{\mathrm{return}}(K)+\xi[S_K]_+ \le\widetilde{O}\left(\frac{\xi\bar{b}d^2}{(1-\gamma)^2}\Xi_K(\Lambda)\right)+O\left(\frac{\xi\bar{b}\sqrt{K}}{1-\gamma}\right). Regret return ( K ) + ξ [ S K ] + ≤ O ( ( 1 − γ ) 2 ξ b ˉ d 2 Ξ K ( Λ ) ) + O ( 1 − γ ξ b ˉ K ) .
By linearity of discounted occupancy measures, there exists a policy π ′ \pi' π ′ whose reward and utility values equal the episode averages:
V P ⋆ , r π ′ ( s 1 ) = 1 K ∑ k = 1 K V P ⋆ , r π k ( s 1 ) , V P ⋆ , g π ′ ( s 1 ) = 1 K ∑ k = 1 K V P ⋆ , g π k ( s 1 ) . V_{P^\star,r}^{\pi'}(s_1)=\frac{1}{K}\sum_{k=1}^{K}V_{P^\star,r}^{\pi_k}(s_1), \qquad V_{P^\star,g}^{\pi'}(s_1)=\frac{1}{K}\sum_{k=1}^{K}V_{P^\star,g}^{\pi_k}(s_1). V P ⋆ , r π ′ ( s 1 ) = K 1 k = 1 ∑ K V P ⋆ , r π k ( s 1 ) , V P ⋆ , g π ′ ( s 1 ) = K 1 k = 1 ∑ K V P ⋆ , g π k ( s 1 ) .
Therefore
V P ⋆ , r π ⋆ ( s 1 ) − V P ⋆ , r π ′ ( s 1 ) + ξ [ b − V P ⋆ , g π ′ ( s 1 ) ] + ≤ 1 K ( Regret r e t u r n ( K ) + ξ [ S K ] + ) . V_{P^\star,r}^{\pi^\star}(s_1)-V_{P^\star,r}^{\pi'}(s_1)+\xi[b-V_{P^\star,g}^{\pi'}(s_1)]_+ \le\frac{1}{K}\left(\operatorname{Regret}^{\mathrm{return}}(K)+\xi[S_K]_+\right). V P ⋆ , r π ⋆ ( s 1 ) − V P ⋆ , r π ′ ( s 1 ) + ξ [ b − V P ⋆ , g π ′ ( s 1 ) ] + ≤ K 1 ( Regret return ( K ) + ξ [ S K ] + ) .
By Lemma 2.4, the optimal dual variable satisfies Y ⋆ ≤ 1 / θ Y^\star\le1/\theta Y ⋆ ≤ 1/ θ , so the constant hidden in ξ = Θ ~ ( 1 / θ ) \xi=\widetilde{\Theta}(1/\theta) ξ = Θ ( 1/ θ ) may be chosen so that ξ ≥ 2 Y ⋆ \xi\ge2Y^\star ξ ≥ 2 Y ⋆ . Applying Lemma B.5 yields
[ S K ] + ≤ 2 ξ ( Regret r e t u r n ( K ) + ξ [ S K ] + ) . [S_K]_+ \le\frac{2}{\xi}\left(\operatorname{Regret}^{\mathrm{return}}(K)+\xi[S_K]_+\right). [ S K ] + ≤ ξ 2 ( Regret return ( K ) + ξ [ S K ] + ) .
Since ξ − 1 = O ~ ( θ ) \xi^{-1}=\widetilde{O}(\theta) ξ − 1 = O ( θ ) and Ξ K ( Λ ) = Ω ~ ( K ) \Xi_K(\Lambda)=\widetilde{\Omega}(\sqrt{K}) Ξ K ( Λ ) = Ω ( K ) , the lower-order O ( ξ b ˉ K / ( 1 − γ ) ) O(\xi\bar{b}\sqrt{K}/(1-\gamma)) O ( ξ b ˉ K / ( 1 − γ )) term is absorbed, and we obtain
Violation s o f t r e t u r n ( K ) = [ S K ] + ≤ O ~ ( b ˉ d 2 ( 1 − γ ) 2 Ξ K ( Λ ) ) . \operatorname{Violation}^{\mathrm{return}}_{\mathrm{soft}}(K)=[S_K]_+\le\widetilde{O}\left(\frac{\bar{b}d^2}{(1-\gamma)^2}\Xi_K(\Lambda)\right). Violation soft return ( K ) = [ S K ] + ≤ O ( ( 1 − γ ) 2 b ˉ d 2 Ξ K ( Λ ) ) .
This proves the displayed soft-violation bound. □
Proposition D.14 (Execution policy corollary for a stationary-mixture variant). The guarantees in Proposition D.13 are stated for the return policy sequence { π k } k = 1 K \{\pi_k\}_{k=1}^{K} { π k } k = 1 K under the discounted-occupancy oracle protocol. The present proposition records a separate fully online interpretation in which episode k k k executes the stationary mixed policy
π ~ k ( ⋅ ∣ s ) : = ( 1 − ϵ k ) π k ( ⋅ ∣ s ) + ϵ k U ( A ) . \widetilde{\pi}_k(\cdot\mid s):=(1-\epsilon_k)\pi_k(\cdot\mid s)+\epsilon_k U(\mathcal{A}). π k ( ⋅ ∣ s ) := ( 1 − ϵ k ) π k ( ⋅ ∣ s ) + ϵ k U ( A ) . This online mixture is not the probing protocol analyzed above; it is included only to translate return policy guarantees into execution policy guarantees.
Define
Regret e x e c ( K ) : = ∑ k = 1 K ( V r π ⋆ ( s 1 ) − V r π ~ k ( s 1 ) ) , \operatorname{Regret}^{\mathrm{exec}}(K):=\sum_{k=1}^{K}\left(V_{r}^{\pi^\star}(s_1)-V_{r}^{\widetilde{\pi}_k}(s_1)\right), Regret exec ( K ) := k = 1 ∑ K ( V r π ⋆ ( s 1 ) − V r π k ( s 1 ) ) , Violation s o f t e x e c ( K ) : = [ ∑ k = 1 K ( b − V g π ~ k ( s 1 ) ) ] + . \operatorname{Violation}^{\mathrm{exec}}_{\mathrm{soft}}(K):=\left[\sum_{k=1}^{K}\left(b-V_{g}^{\widetilde{\pi}_k}(s_1)\right)\right]_+. Violation soft exec ( K ) := [ k = 1 ∑ K ( b − V g π k ( s 1 ) ) ] + . Then
Regret e x e c ( K ) ≤ Regret r e t u r n ( K ) + 2 1 − γ ∑ k = 1 K ϵ k , \operatorname{Regret}^{\mathrm{exec}}(K)\le\operatorname{Regret}^{\mathrm{return}}(K)+\frac{2}{1-\gamma}\sum_{k=1}^{K}\epsilon_k, Regret exec ( K ) ≤ Regret return ( K ) + 1 − γ 2 k = 1 ∑ K ϵ k , Violation s o f t e x e c ( K ) ≤ Violation s o f t r e t u r n ( K ) + 2 1 − γ ∑ k = 1 K ϵ k . \operatorname{Violation}^{\mathrm{exec}}_{\mathrm{soft}}(K)\le\operatorname{Violation}^{\mathrm{return}}_{\mathrm{soft}}(K)+\frac{2}{1-\gamma}\sum_{k=1}^{K}\epsilon_k. Violation soft exec ( K ) ≤ Violation soft return ( K ) + 1 − γ 2 k = 1 ∑ K ϵ k . In particular, if ϵ k = k − p \epsilon_k=k^{-p} ϵ k = k − p with p ∈ [ 1 / 4 , 1 / 2 ) p\in[1/4,1/2) p ∈ [ 1/4 , 1/2 ) , then Proposition D.13 implies
Regret e x e c ( K ) , Violation s o f t e x e c ( K ) ≤ O ~ ( K 1 / 2 + p ) , \operatorname{Regret}^{\mathrm{exec}}(K),\ \operatorname{Violation}^{\mathrm{exec}}_{\mathrm{soft}}(K)\le\widetilde{O}\left(K^{1/2+p}\right), Regret exec ( K ) , Violation soft exec ( K ) ≤ O ( K 1/2 + p ) , which matches the asymptotic orders of the return policy bounds.
Proof. Fix an episode k k k . For j ∈ { r , g } j\in\{r,g\} j ∈ { r , g } , define A j π k ( s , a ) : = Q j π k ( s , a ) − V j π k ( s ) A_j^{\pi_k}(s,a):=Q_j^{\pi_k}(s,a)-V_j^{\pi_k}(s) A j π k ( s , a ) := Q j π k ( s , a ) − V j π k ( s ) . By the trajectory-level normalization introduced in the background section, ∣ A j π k ( s , a ) ∣ ≤ 1 \lvert A_j^{\pi_k}(s,a)\rvert\le1 ∣ A j π k ( s , a )∣ ≤ 1 for both j = r j=r j = r and j = g j=g j = g . The performance-difference lemma with normalized discounted occupancy measure gives
( 1 − γ ) ( V j π ~ k ( s 1 ) − V j π k ( s 1 ) ) = E s ∼ d P ⋆ π ~ k [ ∑ a ∈ A ( π ~ k ( a ∣ s ) − π k ( a ∣ s ) ) A j π k ( s , a ) ] . (1-\gamma)\left(V_j^{\widetilde{\pi}_k}(s_1)-V_j^{\pi_k}(s_1)\right)=\mathbb{E}_{s\sim d_{P^\star}^{\widetilde{\pi}_k}}\left[\sum_{a\in\mathcal{A}}\left(\widetilde{\pi}_k(a\mid s)-\pi_k(a\mid s)\right)A_j^{\pi_k}(s,a)\right]. ( 1 − γ ) ( V j π k ( s 1 ) − V j π k ( s 1 ) ) = E s ∼ d P ⋆ π k [ a ∈ A ∑ ( π k ( a ∣ s ) − π k ( a ∣ s ) ) A j π k ( s , a ) ] .
Hence
D k , j ( s ) : = ∑ a ∈ A ∣ π ~ k ( a ∣ s ) − π k ( a ∣ s ) ∣ ⋅ ∣ A j π k ( s , a ) ∣ . D_{k,j}(s) := \sum_{a \in\mathcal{A}} \left|\tilde{\pi}_k(a \mid s) - \pi_k(a \mid s)\right| \cdot\left|A_j^{\pi_k}(s,a)\right|. D k , j ( s ) := a ∈ A ∑ ∣ π ~ k ( a ∣ s ) − π k ( a ∣ s ) ∣ ⋅ A j π k ( s , a ) .
Therefore
∣ V j π ~ k ( s 1 ) − V j π k ( s 1 ) ∣ ≤ 1 1 − γ E s ∼ d P ∗ π k [ D k , j ( s ) ] ≤ 1 1 − γ E s ∼ d P ∗ π k [ ∥ π ~ k ( ⋅ ∣ s ) − π k ( ⋅ ∣ s ) ∥ 1 ] . \left|V_j^{\tilde{\pi}_k}(s_1) - V_j^{\pi_k}(s_1)\right| \leq\frac{1}{1-\gamma}\mathbb{E}_{s \sim d_{P_*}^{\pi_k}}\left[D_{k,j}(s)\right] \leq\frac{1}{1-\gamma}\mathbb{E}_{s \sim d_{P_*}^{\pi_k}}\left[\left\|\tilde{\pi}_k(\cdot\mid s) - \pi_k(\cdot\mid s)\right\|_1\right]. V j π ~ k ( s 1 ) − V j π k ( s 1 ) ≤ 1 − γ 1 E s ∼ d P ∗ π k [ D k , j ( s ) ] ≤ 1 − γ 1 E s ∼ d P ∗ π k [ ∥ π ~ k ( ⋅ ∣ s ) − π k ( ⋅ ∣ s ) ∥ 1 ] .
Now
π ~ k ( ⋅ ∣ s ) − π k ( ⋅ ∣ s ) = ϵ k ( U ( A ) − π k ( ⋅ ∣ s ) ) , ∥ π ~ k ( ⋅ ∣ s ) − π k ( ⋅ ∣ s ) ∥ 1 ≤ ϵ k ∥ U ( A ) − π k ( ⋅ ∣ s ) ∥ 1 ≤ 2 ϵ k . \tilde{\pi}_k(\cdot\mid s)-\pi_k(\cdot\mid s)=\epsilon_k\left(U(\mathcal{A})-\pi_k(\cdot\mid s)\right), \qquad\left\|\tilde{\pi}_k(\cdot\mid s)-\pi_k(\cdot\mid s)\right\|_1 \leq\epsilon_k\left\|U(\mathcal{A})-\pi_k(\cdot\mid s)\right\|_1 \leq2\epsilon_k. π ~ k ( ⋅ ∣ s ) − π k ( ⋅ ∣ s ) = ϵ k ( U ( A ) − π k ( ⋅ ∣ s ) ) , ∥ π ~ k ( ⋅ ∣ s ) − π k ( ⋅ ∣ s ) ∥ 1 ≤ ϵ k ∥ U ( A ) − π k ( ⋅ ∣ s ) ∥ 1 ≤ 2 ϵ k .
Substituting yields, for both j = r j=r j = r and j = g j=g j = g ,
∣ V j π ~ k ( s 1 ) − V j π k ( s 1 ) ∣ ≤ 2 ϵ k 1 − γ . \left|V_j^{\tilde{\pi}_k}(s_1)-V_j^{\pi_k}(s_1)\right| \leq\frac{2\epsilon_k}{1-\gamma}. V j π ~ k ( s 1 ) − V j π k ( s 1 ) ≤ 1 − γ 2 ϵ k .
Applying this bound with j = r j=r j = r , we obtain
Regret exec ( K ) = ∑ k = 1 K ( V r π ∗ ( s 1 ) − V r π k ( s 1 ) ) + ∑ k = 1 K ( V r π k ( s 1 ) − V r π ~ k ( s 1 ) ) ≤ Regret return ( K ) + 2 1 − γ ∑ k = 1 K ϵ k . \begin{aligned}
\operatorname{Regret}^{\operatorname{exec}}(K) = \sum_{k=1}^{K}\left(V_r^{\pi^*}(s_1)-V_r^{\pi_k}(s_1)\right)+\sum_{k=1}^{K}\left(V_r^{\pi_k}(s_1)-V_r^{\tilde{\pi}_k}(s_1)\right) \\
&\leq\operatorname{Regret}^{\operatorname{return}}(K)+\frac{2}{1-\gamma}\sum_{k=1}^{K}\epsilon_k.
\end{aligned} Regret exec ( K ) = k = 1 ∑ K ( V r π ∗ ( s 1 ) − V r π k ( s 1 ) ) + k = 1 ∑ K ( V r π k ( s 1 ) − V r π ~ k ( s 1 ) ) ≤ Regret return ( K ) + 1 − γ 2 k = 1 ∑ K ϵ k .
Applying the same bound with j = g j=g j = g , together with [ x + y ] + ≤ [ x ] + + y [x+y]_+ \leq[x]_+ + y [ x + y ] + ≤ [ x ] + + y for y ≥ 0 y \geq0 y ≥ 0 , yields
Violation soft exec ( K ) = [ ∑ k = 1 K ( b − V g π ~ k ( s 1 ) ) ] + ≤ [ ∑ k = 1 K ( b − V g π k ( s 1 ) ) ] + + ∑ k = 1 K ∣ V g π ~ k ( s 1 ) − V g π k ( s 1 ) ∣ ≤ Violation soft return ( K ) + 2 1 − γ ∑ k = 1 K ϵ k . \begin{aligned}
\operatorname{Violation}^{\operatorname{exec}}_{\operatorname{soft}}(K)=\left[\sum_{k=1}^{K}\left(b-V_g^{\tilde{\pi}_k}(s_1)\right)\right]_+ \\
&\leq\left[\sum_{k=1}^{K}\left(b-V_g^{\pi_k}(s_1)\right)\right]_+ + \sum_{k=1}^{K}\left|V_g^{\tilde{\pi}_k}(s_1)-V_g^{\pi_k}(s_1)\right| \\
&\leq\operatorname{Violation}^{\operatorname{return}}_{\operatorname{soft}}(K)+\frac{2}{1-\gamma}\sum_{k=1}^{K}\epsilon_k.
\end{aligned} Violation soft exec ( K ) = [ k = 1 ∑ K ( b − V g π ~ k ( s 1 ) ) ] + ≤ [ k = 1 ∑ K ( b − V g π k ( s 1 ) ) ] + + k = 1 ∑ K V g π ~ k ( s 1 ) − V g π k ( s 1 ) ≤ Violation soft return ( K ) + 1 − γ 2 k = 1 ∑ K ϵ k .
Finally, if ϵ k = k − p \epsilon_k=k^{-p} ϵ k = k − p with p ∈ [ 1 / 4 , 1 / 2 ) p \in[1/4,1/2) p ∈ [ 1/4 , 1/2 ) , then Proposition D.13 gives
Regret return ( K ) ≤ O ~ ( K 1 / 2 + p ) , ∑ k = 1 K ϵ k = ∑ k = 1 K k − p = O ( K 1 − p ) . \operatorname{Regret}^{\operatorname{return}}(K) \leq\widetilde{O}(K^{1/2+p}), \qquad\sum_{k=1}^{K}\epsilon_k=\sum_{k=1}^{K}k^{-p}=O(K^{1-p}). Regret return ( K ) ≤ O ( K 1/2 + p ) , k = 1 ∑ K ϵ k = k = 1 ∑ K k − p = O ( K 1 − p ) .
Since p ≥ 1 / 4 p \geq1/4 p ≥ 1/4 is equivalent to 1 − p ≤ 1 / 2 + p 1-p \leq1/2+p 1 − p ≤ 1/2 + p , the additional execution term is of no larger order than the returned-policy terms. Hence Regret exec ( K ) ≤ O ~ ( K 1 / 2 + p ) \operatorname{Regret}^{\operatorname{exec}}(K) \leq\widetilde{O}(K^{1/2+p}) Regret exec ( K ) ≤ O ( K 1/2 + p ) . The same comparison applies to Violation soft exec ( K ) \operatorname{Violation}^{\operatorname{exec}}_{\operatorname{soft}}(K) Violation soft exec ( K ) , because the returned-policy soft-violation term is already of order O ~ ( K 1 / 2 + p ) \widetilde{O}(K^{1/2+p}) O ( K 1/2 + p ) . Therefore Violation soft exec ( K ) ≤ O ~ ( K 1 / 2 + p ) \operatorname{Violation}^{\operatorname{exec}}_{\operatorname{soft}}(K) \leq\widetilde{O}(K^{1/2+p}) Violation soft exec ( K ) ≤ O ( K 1/2 + p ) .
Uniform Exploration as a Tertiary-Aware Repaired Variant
This appendix specializes the decaying-ϵ \epsilon ϵ analysis in Appendix § 11 to the constant schedule ϵ k ≡ 1 \epsilon_k \equiv1 ϵ k ≡ 1 . This case is useful as a clean benchmark: all probe actions are drawn uniformly, while the tertiary probe level and merged bonus covariance are retained so that the repaired optimism proof still applies.
Throughout this appendix, we set
q k ( ⋅ ∣ s ) = q k ′ ( ⋅ ∣ s ) = q k ′ ′ ( ⋅ ∣ s ) = U ( A ) , ∀ k ∈ [ K ] . q_k(\cdot\mid s)=q'_k(\cdot\mid s)=q''_k(\cdot\mid s)=U(\mathcal{A}), \qquad\forall k \in[K]. q k ( ⋅ ∣ s ) = q k ′ ( ⋅ ∣ s ) = q k ′′ ( ⋅ ∣ s ) = U ( A ) , ∀ k ∈ [ K ] .
Equivalently, this is Algorithm 1 with ϵ k ≡ 1 \epsilon_k \equiv1 ϵ k ≡ 1 . Therefore
Γ k = Λ k = ∣ A ∣ , ∀ k ∈ [ K ] . \Gamma_k=\Lambda_k=|\mathcal{A}|, \qquad\forall k \in[K]. Γ k = Λ k = ∣ A ∣ , ∀ k ∈ [ K ] .
Uniform Specialization
For each round i ∈ [ K ] i \in[K] i ∈ [ K ] , define the primary, secondary, and tertiary state marginals by
ρ i ( s ) : = d P ⋆ π i − 1 ( s ) , ρ i ′ ( s ′ ) : = E ρ i × U ( A ) P ⋆ ( s ′ ∣ s , a ) , ρ i ′ ′ ( s ′ ′ ) : = E ρ i ′ × U ( A ) P ⋆ ( s ′ ′ ∣ s , a ) . \rho_i(s) := d_{P^\star}^{\pi_{i-1}}(s), \qquad\rho_i'(s') := \mathbb{E}_{\rho_i \times U(\mathcal{A})}P^\star(s' \mid s,a), \qquad\rho_i''(s'') := \mathbb{E}_{\rho_i' \times U(\mathcal{A})}P^\star(s'' \mid s,a). ρ i ( s ) := d P ⋆ π i − 1 ( s ) , ρ i ′ ( s ′ ) := E ρ i × U ( A ) P ⋆ ( s ′ ∣ s , a ) , ρ i ′′ ( s ′′ ) := E ρ i ′ × U ( A ) P ⋆ ( s ′′ ∣ s , a ) .
The corresponding probing distributions are
ν i ( s , a ) : = ρ i ( s ) U ( a ) , ν i ′ ( s , a ) : = ρ i ′ ( s ) U ( a ) , ν i ′ ′ ( s , a ) : = ρ i ′ ′ ( s ) U ( a ) . \nu_i(s,a) := \rho_i(s)U(a), \qquad\nu_i'(s,a) := \rho_i'(s)U(a), \qquad\nu_i''(s,a) := \rho_i''(s)U(a). ν i ( s , a ) := ρ i ( s ) U ( a ) , ν i ′ ( s , a ) := ρ i ′ ( s ) U ( a ) , ν i ′′ ( s , a ) := ρ i ′′ ( s ) U ( a ) .
We write
ν ˉ k p r i : = 1 k ∑ i = 1 k ν i , ν ˉ k s e c : = 1 k ∑ i = 1 k ν i ′ , ν ˉ k t e r : = 1 k ∑ i = 1 k ν i ′ ′ . \bar{\nu}_k^{\mathrm{pri}} := \frac{1}{k}\sum_{i=1}^{k}\nu_i, \qquad\bar{\nu}_k^{\mathrm{sec}} := \frac{1}{k}\sum_{i=1}^{k}\nu_i', \qquad\bar{\nu}_k^{\mathrm{ter}} := \frac{1}{k}\sum_{i=1}^{k}\nu_i''. ν ˉ k pri := k 1 i = 1 ∑ k ν i , ν ˉ k sec := k 1 i = 1 ∑ k ν i ′ , ν ˉ k ter := k 1 i = 1 ∑ k ν i ′′ .
No separate pseudocode is needed: this is exactly Algorithm 1 with the constant schedule ϵ k ≡ 1 \epsilon_k \equiv1 ϵ k ≡ 1 . Relative to the general algorithm, the only change is that all probing kernels are fixed to U ( A ) U(\mathcal{A}) U ( A ) ; the tertiary tuple, merged covariance, model fitting step, and primal-dual update remain the same.
Supporting Specializations
The tuple-level MLE event, the pathwise-to-averaged bridge for the squared MLE-mismatch class, and the merged covariance comparison from Appendix § 11 apply verbatim to Algorithm 1 with ϵ k ≡ 1 \epsilon_k \equiv1 ϵ k ≡ 1 . The only new point is that all coverage coefficients collapse to explicit constants.
Proposition E.1 (Uniform coverage specialization). Under Algorithm 1 with ϵ k ≡ 1 \epsilon_k \equiv1 ϵ k ≡ 1 , for every episode k k k and every policy π \pi π ,
C k i n i t ( π ) ≤ ∣ A ∣ , C k n e x t ( π ) ≤ ∣ A ∣ , \mathfrak{C}_k^{\mathrm{init}}(\pi) \le|\mathcal{A}|, \qquad\mathfrak{C}_k^{\mathrm{next}}(\pi) \le|\mathcal{A}|, C k init ( π ) ≤ ∣ A ∣ , C k next ( π ) ≤ ∣ A ∣ , and
C k , 1 t r i , n e x t ( π ) ≤ ∣ A ∣ , C k , 2 t r i , n e x t ( π ) ≤ ∣ A ∣ , C k t r i , n e x t ( π ) ≤ ∣ A ∣ . \mathfrak{C}_{k,1}^{\mathrm{tri,next}}(\pi) \le|\mathcal{A}|, \qquad\mathfrak{C}_{k,2}^{\mathrm{tri,next}}(\pi) \le|\mathcal{A}|, \qquad\mathfrak{C}_{k}^{\mathrm{tri,next}}(\pi) \le|\mathcal{A}|. C k , 1 tri , next ( π ) ≤ ∣ A ∣ , C k , 2 tri , next ( π ) ≤ ∣ A ∣ , C k tri , next ( π ) ≤ ∣ A ∣. Proof. Because every probe kernel is uniform,
ν ˉ k p r i ( s , a ) = 1 ∣ A ∣ ⋅ 1 k ∑ i = 1 k ρ i ( s ) . \bar{\nu}_k^{\mathrm{pri}}(s,a) = \frac{1}{|\mathcal{A}|} \cdot\frac{1}{k}\sum_{i=1}^{k}\rho_i(s). ν ˉ k pri ( s , a ) = ∣ A ∣ 1 ⋅ k 1 i = 1 ∑ k ρ i ( s ) . and every discounted occupancy satisfies
ρ i ( s ) = d P ⋆ π i − 1 ( s ) ≥ ( 1 − γ ) d 1 ( s ) . \rho_i(s) = d_{P^\star}^{\pi_{i-1}}(s) \ge(1-\gamma)d_1(s). ρ i ( s ) = d P ⋆ π i − 1 ( s ) ≥ ( 1 − γ ) d 1 ( s ) . Hence
ν ˉ k p r i ( s , a ) ≥ ( 1 − γ ) d 1 ( s ) ∣ A ∣ , \bar{\nu}_k^{\mathrm{pri}}(s,a) \ge\frac{(1-\gamma)d_1(s)}{|\mathcal{A}|}, ν ˉ k pri ( s , a ) ≥ ∣ A ∣ ( 1 − γ ) d 1 ( s ) , which implies
C k i n i t ( π ) ≤ ∣ A ∣ \mathfrak{C}_k^{\mathrm{init}}(\pi) \le|\mathcal{A}| C k init ( π ) ≤ ∣ A ∣ for every policy π \pi π . The bound on C k n e x t ( π ) \mathfrak{C}_k^{\mathrm{next}}(\pi) C k next ( π ) follows exactly as in the uniform version of the original proof: the next-step pushforward state-action measure has density at most ∣ A ∣ |\mathcal{A}| ∣ A ∣ times ν ˉ k s e c \bar{\nu}_k^{\mathrm{sec}} ν ˉ k sec , because the action component of ν ˉ k s e c \bar{\nu}_k^{\mathrm{sec}} ν ˉ k sec is uniform.
The tertiary coefficients are the specialization of Proposition D.7 to the constant schedule ϵ k ≡ 1 \epsilon_k \equiv1 ϵ k ≡ 1 , for which Γ k = ∣ A ∣ \Gamma_k = |\mathcal{A}| Γ k = ∣ A ∣ .
Proposition E.2 (Uniform potential reduction). Under Algorithm 1 with ϵ k ≡ 1 \epsilon_k \equiv1 ϵ k ≡ 1 ,
Ψ K : = ∑ k = 1 K A k ≲ 1 λ 1 + ∣ A ∣ d K ln ( 1 + K d λ 1 ) . \Psi_K := \sum_{k=1}^{K} A_k \lesssim\frac{1}{\sqrt{\lambda_1}} + \sqrt{|\mathcal{A}|dK\ln\left(1+\frac{K}{d\lambda_1}\right)}. Ψ K := k = 1 ∑ K A k ≲ λ 1 1 + ∣ A ∣ d K ln ( 1 + d λ 1 K ) . Proof. This is the specialization of Lemma D.11 to Λ k = ∣ A ∣ \Lambda_k = |\mathcal{A}| Λ k = ∣ A ∣ for all k k k :
Ψ K ≲ 1 λ 1 + d ln ( 1 + K d λ 1 ) ∑ k = 2 K Λ k . \Psi_K \lesssim\frac{1}{\sqrt{\lambda_1}} + \sqrt{d\ln\left(1+\frac{K}{d\lambda_1}\right)\sum_{k=2}^{K}\Lambda_k}. Ψ K ≲ λ 1 1 + d ln ( 1 + d λ 1 K ) k = 2 ∑ K Λ k . Since ∑ k = 2 K Λ k = ( K − 1 ) ∣ A ∣ \sum_{k=2}^{K}\Lambda_k = (K-1)|\mathcal{A}| ∑ k = 2 K Λ k = ( K − 1 ) ∣ A ∣ , the claim follows.
Theorem-Level Derivation
Theorem E.3 (Return policy regret and soft violation for p = 0 p = 0 p = 0 ). Assume Algorithm 1 is run with ϵ k ≡ 1 \epsilon_k \equiv1 ϵ k ≡ 1 , and suppose the high-probability events required by Lemmas D.9, D.10, and D.11 hold simultaneously. Choose
α k max = α k e m p = O ~ ( ( ∣ A ∣ + d 2 ) γ ln ( ∣ M ∣ k / δ ) ) , \alpha_k^{\max} = \alpha_k^{\mathrm{emp}} = \widetilde{O}\left(\sqrt{(\lvert\mathcal{A}\rvert+ d^2)\gamma\ln(\lvert\mathcal{M}\rvert k/\delta)}\right), α k m a x = α k emp = O ( (∣ A ∣ + d 2 ) γ ln (∣ M ∣ k / δ ) ) , λ k = Θ ~ ( ∣ A ∣ + d 2 d ln ( ∣ M ∣ k / δ ) ) , \lambda_k = \widetilde{\Theta}\left(\frac{\lvert\mathcal{A}\rvert+ d^2}{d}\ln(\lvert\mathcal{M}\rvert k/\delta)\right), λ k = Θ ( d ∣ A ∣ + d 2 ln (∣ M ∣ k / δ ) ) , and
η = 2 ( 1 − γ ) θ b ˉ K , ξ = Θ ~ ( 1 / θ ) , \eta= \frac{2(1-\gamma)}{\theta\bar{b}\sqrt{K}}, \qquad\xi= \widetilde{\Theta}(1/\theta), η = θ b ˉ K 2 ( 1 − γ ) , ξ = Θ ( 1/ θ ) , and assume the cap is nonbinding:
b ˉ ≥ max 1 ≤ j ≤ K α j e m p λ j . \bar{b} \ge\max_{1\le j\le K}\frac{\alpha_j^{\mathrm{emp}}}{\sqrt{\lambda_j}}. b ˉ ≥ 1 ≤ j ≤ K max λ j α j emp . Then, with probability at least 1 − δ 1-\delta 1 − δ ,
Regret r e t u r n ( K ) ≤ O ~ ( ∣ A ∣ b ˉ d 2 K θ ( 1 − γ ) 2 ) , Violation s o f t r e t u r n ( K ) ≤ O ~ ( ∣ A ∣ b ˉ d 2 K ( 1 − γ ) 2 ) . ∗ \operatorname{Regret}^{\mathrm{return}}(K) \le\widetilde{O}\left(\frac{\lvert\mathcal{A}\rvert\bar{b}d^2\sqrt{K}}{\theta(1-\gamma)^2}\right), \qquad\operatorname{Violation}_{\mathrm{soft}}^{\mathrm{return}}(K) \le\widetilde{O}\left(\frac{\lvert\mathcal{A}\rvert\bar{b}d^2\sqrt{K}}{(1-\gamma)^2}\right).* Regret return ( K ) ≤ O ( θ ( 1 − γ ) 2 ∣ A ∣ b ˉ d 2 K ) , Violation soft return ( K ) ≤ O ( ( 1 − γ ) 2 ∣ A ∣ b ˉ d 2 K ) . ∗ Proof. This is a direct specialization of Proposition D.13 to the constant schedule ϵ k ≡ 1 \epsilon_k \equiv1 ϵ k ≡ 1 . Under this choice,
Γ k = Λ k = ∣ A ∣ , ∀ k ∈ [ K ] . \Gamma_k = \Lambda_k = \lvert\mathcal{A}\rvert, \qquad\forall k \in[K]. Γ k = Λ k = ∣ A ∣ , ∀ k ∈ [ K ] . Therefore the quantity
Ξ K ( Λ ) : = Γ K ∑ k = 2 K Λ k + ∑ k = 1 K Γ k k \Xi_K(\Lambda) := \sqrt{\Gamma_K\sum_{k=2}^{K}\Lambda_k + \sum_{k=1}^{K}\frac{\Gamma_k}{\sqrt{k}}} Ξ K ( Λ ) := Γ K k = 2 ∑ K Λ k + k = 1 ∑ K k Γ k satisfies
Ξ K ( Λ ) = ∣ A ∣ ⋅ ( K − 1 ) ∣ A ∣ + ∣ A ∣ ∑ k = 1 K 1 k ≲ ∣ A ∣ K . \Xi_K(\Lambda) = \sqrt{\lvert\mathcal{A}\rvert\cdot(K-1)\lvert\mathcal{A}\rvert+ \lvert\mathcal{A}\rvert\sum_{k=1}^{K}\frac{1}{\sqrt{k}}} \lesssim\lvert\mathcal{A}\rvert\sqrt{K}. Ξ K ( Λ ) = ∣ A ∣ ⋅ ( K − 1 ) ∣ A ∣ + ∣ A ∣ k = 1 ∑ K k 1 ≲ ∣ A ∣ K . Substituting this into Proposition D.13 yields
Regret r e t u r n ( K ) ≤ O ~ ( b ˉ d 2 θ ( 1 − γ ) 2 Ξ K ( Λ ) ) ≤ O ~ ( ∣ A ∣ b ˉ d 2 K θ ( 1 − γ ) 2 ) , \operatorname{Regret}^{\mathrm{return}}(K) \le\widetilde{O}\left(\frac{\bar{b}d^2}{\theta(1-\gamma)^2}\Xi_K(\Lambda)\right) \le\widetilde{O}\left(\frac{\lvert\mathcal{A}\rvert\bar{b}d^2\sqrt{K}}{\theta(1-\gamma)^2}\right), Regret return ( K ) ≤ O ( θ ( 1 − γ ) 2 b ˉ d 2 Ξ K ( Λ ) ) ≤ O ( θ ( 1 − γ ) 2 ∣ A ∣ b ˉ d 2 K ) , and
Violation s o f t r e t u r n ( K ) ≤ O ~ ( b ˉ d 2 ( 1 − γ ) 2 Ξ K ( Λ ) ) ≤ O ~ ( ∣ A ∣ b ˉ d 2 K ( 1 − γ ) 2 ) . \operatorname{Violation}_{\mathrm{soft}}^{\mathrm{return}}(K) \le\widetilde{O}\left(\frac{\bar{b}d^2}{(1-\gamma)^2}\Xi_K(\Lambda)\right) \le\widetilde{O}\left(\frac{\lvert\mathcal{A}\rvert\bar{b}d^2\sqrt{K}}{(1-\gamma)^2}\right). Violation soft return ( K ) ≤ O ( ( 1 − γ ) 2 b ˉ d 2 Ξ K ( Λ ) ) ≤ O ( ( 1 − γ ) 2 ∣ A ∣ b ˉ d 2 K ) . It remains only to note that the displayed tuning of λ k \lambda_k λ k makes the cap condition compatible with a K K K -independent choice of b ˉ \bar{b} b ˉ . Indeed,
α k e m p λ k = O ~ ( ( ∣ A ∣ + d 2 ) ln ( ∣ M ∣ k / δ ) ( ( ∣ A ∣ + d 2 ) / d ) ln ( ∣ M ∣ k / δ ) ) = O ~ ( d ) , \frac{\alpha_k^{\mathrm{emp}}}{\sqrt{\lambda_k}} = \widetilde{O}\left(\sqrt{\frac{(\lvert\mathcal{A}\rvert+d^2)\ln(\lvert\mathcal{M}\rvert k/\delta)}{((\lvert\mathcal{A}\rvert+d^2)/d)\ln(\lvert\mathcal{M}\rvert k/\delta)}}\right) = \widetilde{O}(\sqrt{d}), λ k α k emp = O ( ((∣ A ∣ + d 2 ) / d ) ln (∣ M ∣ k / δ ) (∣ A ∣ + d 2 ) ln (∣ M ∣ k / δ ) ) = O ( d ) , so any b ˉ = Ω ~ ( d ) \bar{b} = \widetilde{\Omega}(\sqrt{d}) b ˉ = Ω ( d ) is sufficient, exactly as in Appendix D.
Constraint discussion. The regret proof uses the same representation-learning and optimism ingredients as Appendix D. For soft violation, one additional ingredient is needed: the cumulative overestimation lemma controls how much the optimistic utility used by the dual update can exceed the true utility. This term is summable under the repaired tertiary design, so the standard projected dual argument yields the stated soft-violation bound.
Remark E.4 (Relation to the p = 0 p = 0 p = 0 specialization). The uniform protocol is exactly the ϵ k ≡ 1 \epsilon_k \equiv1 ϵ k ≡ 1 specialization of Algorithm 1 in Section 3. Consequently, this appendix gives the fully explicit return-policy theorem for the p = 0 p = 0 p = 0 schedule, with the familiar O ~ ( ∣ A ∣ d 2 K ) \widetilde{O}(\lvert\mathcal{A}\rvert d^2\sqrt{K}) O (∣ A ∣ d 2 K ) dependence up to the same problem-dependent prefactors and logarithmic terms.
Hard Constraints Under Decaying Tertiary Probing
This appendix gives the repaired hard-violation analysis. The key distinction is between the one-sided correction used by the algorithm and the two-sided width used only in the proof. In particular, we do not assume an external certificate event and we do not require a summable radius uniformly over all policies queried by the inner loop.
Algorithm and Certificate
The representation-learning front end is the same as in Algorithm 1. At episode k k k , let
q k = q k ′ = q k ′ ′ : = ( 1 − ϵ k ) π k − 1 + ϵ k U ( A ) , Γ k : = ∣ A ∣ ϵ k . q_k=q'_k=q''_k:=(1-\epsilon_k)\pi_{k-1}+\epsilon_k U(\mathcal{A}), \qquad\Gamma_k:=\frac{|\mathcal{A}|}{\epsilon_k}. q k = q k ′ = q k ′′ := ( 1 − ϵ k ) π k − 1 + ϵ k U ( A ) , Γ k := ϵ k ∣ A ∣ .
and let
ζ k : = ln ( ∣ M ∣ k / δ ) k . \zeta_k:=\frac{\ln(|\mathcal{M}|k/\delta)}{k}. ζ k := k ln ( ∣ M ∣ k / δ ) .
After fitting P ^ k = ( μ ^ k , ϕ ^ k ) \widehat{P}_k=(\widehat{\mu}_k,\widehat{\phi}_k) P k = ( μ k , ϕ k ) on the three streams and forming Σ ^ k m i x \widehat{\Sigma}^{\mathrm{mix}}_k Σ k mix , define
b ^ k ( s , a ) : = min { α k ∥ ϕ ^ k ( s , a ) ∥ ( Σ ^ k m i x ) − 1 , b ˉ } . \widehat{b}_k(s,a):=\min\left\{\alpha_k\left\|\widehat{\phi}_k(s,a)\right\|_{(\widehat{\Sigma}^{\mathrm{mix}}_k)^{-1}},\bar{b}\right\}. b k ( s , a ) := min { α k ϕ k ( s , a ) ( Σ k mix ) − 1 , b ˉ } .
For a sufficiently large universal calibration constant C c e r t C_{\mathrm{cert}} C cert , set
ℓ k : = C c e r t Γ k ζ k 1 − γ , U k ( π ) : = [ V P ^ k , g − b ^ k π ( s 1 ) − ℓ k ] + . (6) \ell_k:=C_{\mathrm{cert}}\sqrt{\frac{\Gamma_k\zeta_k}{1-\gamma}},\qquad U_k(\pi):=\left[V^{\pi}_{\widehat{P}_k,g-\widehat{b}_k}(s_1)-\ell_k\right]_+. \tag*{(6)} ℓ k := C cert 1 − γ Γ k ζ k , U k ( π ) := [ V P k , g − b k π ( s 1 ) − ℓ k ] + . (6)
The quantity ℓ k \ell_k ℓ k is the one-sided statistical correction used by the algorithm. For analysis, define the policy-dependent selected width
w k ( π ) : = 2 ℓ k + 2 V P ^ k , b ^ k π ( s 1 ) . (7) w_k(\pi):=2\ell_k+2V^{\pi}_{\widehat{P}_k,\widehat{b}_k}(s_1). \tag*{(7)} w k ( π ) := 2 ℓ k + 2 V P k , b k π ( s 1 ) . (7)
Unlike the correction ℓ k \ell_k ℓ k , the width w k ( π ) w_k(\pi) w k ( π ) is not required as an algorithmic input.
Let Y max : = K 1 / 4 Y_{\max}:=K^{1/4} Y m a x := K 1/4 , and use the capped grid
Y K : = { 0 , η h , 2 η h , … } ∩ [ 0 , Y max ] ∪ { Y max } . \mathcal{Y}_K:=\{0,\eta_h,2\eta_h,\ldots\}\cap[0,Y_{\max}]\cup\{Y_{\max}\}. Y K := { 0 , η h , 2 η h , … } ∩ [ 0 , Y m a x ] ∪ { Y m a x } .
Thus the cap itself is always queried even when Y max / η h Y_{\max}/\eta_h Y m a x / η h is not an integer.
Derived Certificate Event
Let E r e p \mathcal{E}_{\mathrm{rep}} E rep be the intersection of the tuple-level MLE, pathwise bridge, and covariance-comparison events in Lemmas D.2 , D.4 , and D.6 . Replacing δ \delta δ by a constant fraction in the individual events makes P ( E r e p ) ≥ 1 − δ \mathbb{P}(\mathcal{E}_{\mathrm{rep}})\ge1-\delta P ( E rep ) ≥ 1 − δ .
Lemma F.1 (One-sided certificate and selected-policy interval). Assume E r e p \mathcal{E}_{\mathrm{rep}} E rep , the nonbinding-cap condition
b ˉ ≥ max j ≤ K α j λ j , \bar{b}\ge\max_{j\le K}\frac{\alpha_j}{\sqrt{\lambda_j}}, b ˉ ≥ j ≤ K max λ j α j , and the parameter lower bound on α k \alpha_k α k in Lemma D.9 with C k t r i , n e x t ( π ) ≤ Γ k C_k^{\mathrm{tri,next}}(\pi)\le\Gamma_k C k tri , next ( π ) ≤ Γ k . For a sufficiently large C c e r t C_{\mathrm{cert}} C cert , simultaneously for every episode k k k and every policy π \pi π ,
U k ( π ) ≤ V P ⋆ , g π ( s 1 ) ≤ U k ( π ) + w k ( π ) . (8) U_k(\pi)\le V^{\pi}_{P^\star,g}(s_1)\le U_k(\pi)+w_k(\pi). \tag*{(8)} U k ( π ) ≤ V P ⋆ , g π ( s 1 ) ≤ U k ( π ) + w k ( π ) . (8)
Proof. We first verify that the residual in the one-step-back argument is uniform in π \pi π . Since every primary occupancy contains the initial component ( 1 − γ ) d 1 (1-\gamma)d_1 ( 1 − γ ) d 1 , and every probing kernel up to episode k k k has action floor ϵ k / ∣ A ∣ \epsilon_k/|\mathcal{A}| ϵ k /∣ A ∣ ,
ν ˉ k p r i ( s , a ) ≥ ( 1 − γ ) d 1 ( s ) ϵ k ∣ A ∣ . \bar{\nu}^{\mathrm{pri}}_k(s,a)\ge(1-\gamma)d_1(s)\frac{\epsilon_k}{|\mathcal{A}|}. ν ˉ k pri ( s , a ) ≥ ( 1 − γ ) d 1 ( s ) ∣ A ∣ ϵ k .
Consequently
E k i n i t ( π ) = D ( ( 1 − γ ) d 1 ⊗ π ∥ ν ˉ k p r i ) ≤ Γ k \mathcal{E}^{\mathrm{init}}_k(\pi)=D\left((1-\gamma)d_1\otimes\pi\;\middle\|\;\bar{\nu}^{\mathrm{pri}}_k\right)\le\Gamma_k E k init ( π ) = D ( ( 1 − γ ) d 1 ⊗ π ν ˉ k pri ) ≤ Γ k
Algorithm 2 REP-PD-H-TEG with an explicit one-sided certificate
Input: Parameters { λ k , α k , ϵ k } k = 1 K \{\lambda_k,\alpha_k,\epsilon_k\}_{k=1}^{K} { λ k , α k , ϵ k } k = 1 K , confidence δ \delta δ , calibration C c e r t C_{\mathrm{cert}} C cert , temperature τ h \tau_h τ h , grid step η h \eta_h η h , model class M \mathcal{M} M , and episodes K K K .
Initialize π 0 ( ⋅ ∣ s ) = U ( A ) \pi_0(\cdot\mid s)=U(\mathcal{A}) π 0 ( ⋅ ∣ s ) = U ( A ) and D 0 = D 0 ′ = D 0 ′ ′ = ∅ D_0=D'_0=D''_0=\varnothing D 0 = D 0 ′ = D 0 ′′ = ∅ .
for k = 1 , … , K k=1,\ldots,K k = 1 , … , K do
Set q k = q k ′ = q k ′ ′ = ( 1 − ϵ k ) π k − 1 + ϵ k U ( A ) q_k=q'_k=q''_k=(1-\epsilon_k)\pi_{k-1}+\epsilon_kU(\mathcal{A}) q k = q k ′ = q k ′′ = ( 1 − ϵ k ) π k − 1 + ϵ k U ( A ) .
Query one tertiary tuple and update D k , D k ′ , D k ′ ′ D_k,D'_k,D''_k D k , D k ′ , D k ′′ exactly as in Algorithm 1.
Fit
P ^ k = ( μ ^ k , ϕ ^ k ) ∈ arg max ( μ , ϕ ) ∈ M E D k + D k ′ + D k ′ ′ [ log μ ( x ′ T ϕ ( x , u ) ) ] . \widehat{P}_k=(\widehat{\mu}_k,\widehat{\phi}_k)\in\underset{(\mu,\phi)\in\mathcal{M}}{\arg\max}\ \mathbb{E}_{D_k+D'_k+D''_k}\left[\log\mu(x'^{\mathsf T}\phi(x,u))\right]. P k = ( μ k , ϕ k ) ∈ ( μ , ϕ ) ∈ M arg max E D k + D k ′ + D k ′′ [ log μ ( x ′ T ϕ ( x , u )) ] .
Form Σ ^ k m i x \widehat{\Sigma}^{\mathrm{mix}}_k Σ k mix and b ^ k \widehat{b}_k b k as above, and set
ℓ k = C c e r t ∣ A ∣ ln ( ∣ M ∣ k / δ ) ϵ k k ( 1 − γ ) . \ell_k=C_{\mathrm{cert}}\sqrt{\frac{|\mathcal{A}|\ln(|\mathcal{M}|k/\delta)}{\epsilon_k k(1-\gamma)}}. ℓ k = C cert ϵ k k ( 1 − γ ) ∣ A ∣ ln ( ∣ M ∣ k / δ ) .
Set found ← \leftarrow ← false.
for Y Y Y in the increasing grid Y K \mathcal{Y}_K Y K do
Compute
π ^ k , Y ∈ arg max π ( Q P ^ k , r + b ^ k π + Y Q P ^ k , g + b ^ k π ) . \widehat{\pi}_{k,Y}\in\underset{\pi}{\arg\max}\left(Q^{\pi}_{\widehat{P}_k,r+\widehat{b}_k}+YQ^{\pi}_{\widehat{P}_k,g+\widehat{b}_k}\right). π k , Y ∈ π arg max ( Q P k , r + b k π + Y Q P k , g + b k π ) .
Form the statewise softmax candidate
π k , Y ( ⋅ ∣ x ) = softmax τ h ( Q P ^ k , r + b ^ k π ^ k , Y ( x , ⋅ ) + Y Q P ^ k , g + b ^ k π ^ k , Y ( x , ⋅ ) ) . \pi_{k,Y}(\cdot\mid x)=\operatorname{softmax}_{\tau_h}\left(Q^{\widehat{\pi}_{k,Y}}_{\widehat{P}_k,r+\widehat{b}_k}(x,\cdot)+YQ^{\widehat{\pi}_{k,Y}}_{\widehat{P}_k,g+\widehat{b}_k}(x,\cdot)\right). π k , Y ( ⋅ ∣ x ) = softmax τ h ( Q P k , r + b k π k , Y ( x , ⋅ ) + Y Q P k , g + b k π k , Y ( x , ⋅ ) ) .
Compute
U k ( π k , Y ) = [ V P ^ k , g − b ^ k π k , Y ( s 1 ) − ℓ k ] + . \mathcal{U}_k(\pi_{k,Y})=\left[V^{\pi_{k,Y}}_{\widehat{P}_k,g-\widehat{b}_k}(s_1)-\ell_k\right]_+. U k ( π k , Y ) = [ V P k , g − b k π k , Y ( s 1 ) − ℓ k ] + .
if U k ( π k , Y ) ≥ b \mathcal{U}_k(\pi_{k,Y})\ge b U k ( π k , Y ) ≥ b then
Set ( π k , Y k ) ← ( π k , Y , Y ) (\pi_k,Y_k)\leftarrow(\pi_{k,Y},Y) ( π k , Y k ) ← ( π k , Y , Y ) and found ← \leftarrow ← true.
break
end if
end for
if found = false then
Set ( π k , Y k ) ← ( π k , Y max , Y max ) (\pi_k,Y_k)\leftarrow(\pi_{k,Y_{\max}},Y_{\max}) ( π k , Y k ) ← ( π k , Y m a x , Y m a x ) .
end if
end for
Return π 1 , … , π K \pi_1,\ldots,\pi_K π 1 , … , π K .
for every policy π \pi π . Proposition D.7 likewise gives E k t r i , n e x t ( π ) ≤ Γ k \mathcal{E}^{\mathrm{tri,next}}_k(\pi)\le\Gamma_k E k tri , next ( π ) ≤ Γ k uniformly in π \pi π .
Write f k ( s , a ) = ∥ P ^ k ( ⋅ ∣ s , a ) − P ⋆ ( ⋅ ∣ s , a ) ∥ 1 f_k(s,a)=\|\widehat{P}_k(\cdot\mid s,a)-P^\star(\cdot\mid s,a)\|_1 f k ( s , a ) = ∥ P k ( ⋅ ∣ s , a ) − P ⋆ ( ⋅ ∣ s , a ) ∥ 1 . The first form of the simulation lemma gives
( 1 − γ ) ( V P ^ k , g − b ^ k π ( s 1 ) − V P ⋆ , g π ( s 1 ) ) = E P ^ k π [ − b ^ k ( s , a ) + γ ( E P ^ k − E P ⋆ ) V P ⋆ , g π ( s ′ ) ] . (9) (1-\gamma)\left(V^{\pi}_{\widehat{P}_k,g-\widehat{b}_k}(s_1)-V^{\pi}_{P^\star,g}(s_1)\right)
=\mathbb{E}_{\widehat{P}_k}^{\pi}\left[-\widehat{b}_k(s,a)+\gamma\left(\mathbb{E}_{\widehat{P}_k}-\mathbb{E}_{P^\star}\right)V^{\pi}_{P^\star,g}(s')\right]. \tag*{(9)} ( 1 − γ ) ( V P k , g − b k π ( s 1 ) − V P ⋆ , g π ( s 1 ) ) = E P k π [ − b k ( s , a ) + γ ( E P k − E P ⋆ ) V P ⋆ , g π ( s ′ ) ] . (9)
The normalized utility assumption implies ∥ V P ⋆ , g π ∥ ∞ ≤ 1 \|V^{\pi}_{P^\star,g}\|_\infty\le1 ∥ V P ⋆ , g π ∥ ∞ ≤ 1 . Applying Lemma D.8 with g = f k g=f_k g = f k , using E k i n i t ( π ) , E k t r i , n e x t ( π ) ≤ Γ k \mathcal{E}^{\mathrm{init}}_k(\pi),\mathcal{E}^{\mathrm{tri,next}}_k(\pi)\le\Gamma_k E k init ( π ) , E k tri , next ( π ) ≤ Γ k , and choosing α k \alpha_k α k to absorb the covariance-dependent term yields
V P ^ k , g − b ^ k π ( s 1 ) − V P ⋆ , g π ( s 1 ) ≤ C c e r t Γ k ζ k 1 − γ = ℓ k . (10) V^{\pi}_{\widehat{P}_k,g-\widehat{b}_k}(s_1)-V^{\pi}_{P^\star,g}(s_1)\le C_{\mathrm{cert}}\sqrt{\frac{\Gamma_k\zeta_k}{1-\gamma}}=\ell_k. \tag*{(10)} V P k , g − b k π ( s 1 ) − V P ⋆ , g π ( s 1 ) ≤ C cert 1 − γ Γ k ζ k = ℓ k . (10)
The proof is deterministic on E r e p \mathcal{E}_{\mathrm{rep}} E rep , so no additional union bound over the adaptively queried policies is required. Since the true utility is nonnegative, (G.5) proves the first inequality in (G.3) after positive-part clipping.
Repeating the direct-optimism argument of Lemma D.9 for the normalized utility signal g g g gives
V P ⋆ , g π ( s 1 ) ≤ V P ^ k , g + b ^ k π ( s 1 ) + ℓ k . (11) V^{\pi}_{P^\star,g}(s_1)\le V^{\pi}_{\widehat{P}_k,g+\widehat{b}_k}(s_1)+\ell_k. \tag*{(11)} V P ⋆ , g π ( s 1 ) ≤ V P k , g + b k π ( s 1 ) + ℓ k . (11)
For the same model and policy, discounted value is linear in the signal:
V P ^ k , g + b ^ k π ⋆ ( s 1 ) − V P ^ k , g − b ^ k π ⋆ ( s 1 ) = 2 V P ^ k , b ^ k π ⋆ ( s 1 ) . (12) V^{\pi^\star}_{\widehat{P}_k,g+\widehat{b}_k}(s_1)-V^{\pi^\star}_{\widehat{P}_k,g-\widehat{b}_k}(s_1)=2V^{\pi^\star}_{\widehat{P}_k,\widehat{b}_k}(s_1). \tag*{(12)} V P k , g + b k π ⋆ ( s 1 ) − V P k , g − b k π ⋆ ( s 1 ) = 2 V P k , b k π ⋆ ( s 1 ) . (12)
Combining (G.6), (G.7), and the raw lower endpoint V P ^ k , g − b ^ k π ⋆ − ℓ k V^{\pi^\star}_{\widehat{P}_k,g-\widehat{b}_k}-\ell_k V P k , g − b k π ⋆ − ℓ k proves the upper inequality with w k ( π ) = 2 ℓ k + 2 V P ^ k , b ^ k π ⋆ w_k(\pi)=2\ell_k+2V^{\pi^\star}_{\widehat{P}_k,\widehat{b}_k} w k ( π ) = 2 ℓ k + 2 V P k , b k π ⋆ . Replacing a negative raw endpoint by zero only tightens this upper bound because V P ⋆ , g π ⋆ ≥ 0 V^{\pi^\star}_{P^\star,g}\ge0 V P ⋆ , g π ⋆ ≥ 0 .
For the selected policy sequence, define
B K : = O ~ ( b ˉ d 2 ( 1 − γ ) 2 [ Γ K ∑ k = 2 K Λ k + ∑ k = 1 K Γ k k ] ) . (13) \mathcal{B}_K:=\widetilde{O}\left(\frac{\bar{b}d^2}{(1-\gamma)^2}\left[\sqrt{\Gamma_K\sum_{k=2}^{K}\Lambda_k+\sum_{k=1}^{K}\frac{\Gamma_k}{\sqrt{k}}}\right]\right). \tag*{(13)} B K := O ( 1 − γ ) 2 b ˉ d 2 Γ K k = 2 ∑ K Λ k + k = 1 ∑ K k Γ k . (13)
where
Λ k : = 1 1 − ϵ k + ϵ k / ∣ A ∣ . \Lambda_k:=\frac{1}{1-\epsilon_k+\epsilon_k/|\mathcal{A}|}. Λ k := 1 − ϵ k + ϵ k /∣ A ∣ 1 .
Lemma F.2 (Selected-policy width summability). On E r e p \mathcal{E}_{\mathrm{rep}} E rep , the policy sequence returned by Algorithm 2 satisfies
∑ k = 1 K w k ( π k ) ≤ B K + 2 ∑ k = 1 K ℓ k . (14) \sum_{k=1}^{K}w_k(\pi_k)\le\mathcal{B}_K+2\sum_{k=1}^{K}\ell_k. \tag*{(14)} k = 1 ∑ K w k ( π k ) ≤ B K + 2 k = 1 ∑ K ℓ k . (14)
Proof. Condition on the history before episode k k k . Apply the second form of the simulation lemma to the signal b ^ k \widehat{b}_k b k , with no additional bonus:
V P ^ k , b ^ k π k ( s 1 ) − V P ⋆ , b ^ k π k ( s 1 ) = γ 1 − γ E d P ⋆ π [ ( E P ^ k − E P ⋆ ) V P ^ k , b ^ k π k ( s ′ ) ] . V^{\pi_k}_{\widehat{P}_k,\widehat{b}_k}(s_1)-V^{\pi_k}_{P^\star,\widehat{b}_k}(s_1)=\frac{\gamma}{1-\gamma}\mathbb{E}_{d^\pi_{P^\star}}\left[\left(\mathbb{E}_{\widehat{P}_k}-\mathbb{E}_{P^\star}\right)V^{\pi_k}_{\widehat{P}_k,\widehat{b}_k}(s')\right]. V P k , b k π k ( s 1 ) − V P ⋆ , b k π k ( s 1 ) = 1 − γ γ E d P ⋆ π [ ( E P k − E P ⋆ ) V P k , b k π k ( s ′ ) ] .
Because 0 ≤ b ^ k ≤ b ˉ 0\le\widehat{b}_k\le\bar{b} 0 ≤ b k ≤ b ˉ , ∥ V P ^ k , b ^ k π k ∥ ∞ ≤ b ˉ / ( 1 − γ ) \lVert V^{\pi_k}_{\widehat{P}_k,\widehat{b}_k}\rVert_\infty\le\bar{b}/(1-\gamma) ∥ V P k , b k π k ∥ ∞ ≤ b ˉ / ( 1 − γ ) , and hence
V P ^ k , b ^ k π k ( s 1 ) ≤ 1 1 − γ E d P ⋆ π [ b ^ k ] + γ b ˉ ( 1 − γ ) 2 E d P ⋆ π [ f k ] . (15) V^{\pi_k}_{\widehat{P}_k,\widehat{b}_k}(s_1)\le\frac{1}{1-\gamma}\mathbb{E}_{d^\pi_{P^\star}}[\widehat{b}_k]+\frac{\gamma\bar{b}}{(1-\gamma)^2}\mathbb{E}_{d^\pi_{P^\star}}[f_k]. \tag*{(15)} V P k , b k π k ( s 1 ) ≤ 1 − γ 1 E d P ⋆ π [ b k ] + ( 1 − γ ) 2 γ b ˉ E d P ⋆ π [ f k ] . (15)
The two expectations in (G.10) are exactly the bonus and model-error terms bounded in the proof of Lemma D.10; their cumulative bound follows from Lemma D.11 and is B K \mathcal{B}_K B K up to universal constants. Summing (G.2) and using (G.10) proves (G.9). Importantly, this step is applied only to the returned sequence { π k } \{\pi_k\} { π k } , for which the deployed-to-probe coupling used by Lemma D.11 is valid.
For a polynomial schedule ϵ k = k − p \epsilon_k=k^{-p} ϵ k = k − p ,
∑ k = 1 K ℓ k ≤ O ~ ( ∣ A ∣ 1 − γ K ( 1 + p ) / 2 ) . (16) \sum_{k=1}^{K}\ell_k\le\widetilde{O}\left(\sqrt{\frac{|\mathcal{A}|}{1-\gamma}K^{(1+p)/2}}\right). \tag*{(16)} k = 1 ∑ K ℓ k ≤ O ( 1 − γ ∣ A ∣ K ( 1 + p ) /2 ) . (16)
while
B K ≤ O ~ ( ∣ A ∣ b ˉ d 2 ( 1 − γ ) 2 K 1 / 2 + p ) . (17) \mathcal{B}_K\le\widetilde{O}\left(\frac{|\mathcal{A}|\bar{b}d^2}{(1-\gamma)^2}K^{1/2+p}\right). \tag*{(17)} B K ≤ O ( ( 1 − γ ) 2 ∣ A ∣ b ˉ d 2 K 1/2 + p ) . (17)
Thus the correction term is no larger in K K K than the selected-width term.
Hard Front End
For the selected dual level Y k Y_k Y k , define the Lagrangian front end
σ ~ K : = ∑ k = 1 K [ V P ⋆ , r π ⋆ ( s 1 ) − V P ⋆ , r π k ( s 1 ) + Y k ( V P ⋆ , g π ⋆ ( s 1 ) − V P ⋆ , g π k ( s 1 ) ) ] . (18) \widetilde{\sigma}_K:=\sum_{k=1}^{K}\left[V^{\pi^\star}_{P^\star,r}(s_1)-V^{\pi_k}_{P^\star,r}(s_1)+Y_k\left(V^{\pi^\star}_{P^\star,g}(s_1)-V^{\pi_k}_{P^\star,g}(s_1)\right)\right]. \tag*{(18)} σ K := k = 1 ∑ K [ V P ⋆ , r π ⋆ ( s 1 ) − V P ⋆ , r π k ( s 1 ) + Y k ( V P ⋆ , g π ⋆ ( s 1 ) − V P ⋆ , g π k ( s 1 ) ) ] . (18)
Let σ k s m \sigma_k^{\mathrm{sm}} σ k sm be the value suboptimality introduced by replacing the exact optimistic planner at the selected dual level with its statewise softmax candidate. Under Assumption C.1,
∑ k = 1 K σ k s m ≤ O ~ ( K ∣ A ∣ ( 1 − γ ) 2 exp ( − δ min τ h ) ) . (19) \sum_{k=1}^{K}\sigma_k^{\mathrm{sm}}\le\widetilde{O}\left(\frac{K|\mathcal{A}|}{(1-\gamma)^2}\exp\left(-\frac{\delta_{\min}}{\tau_h}\right)\right). \tag*{(19)} k = 1 ∑ K σ k sm ≤ O ( ( 1 − γ ) 2 K ∣ A ∣ exp ( − τ h δ m i n ) ) . (19)
Theorem F.3 (Decaying hard front end). Assume E r e p \mathcal{E}_{\mathrm{rep}} E rep , the nonbinding bonus cap, and Assumption C.1. With the same α k \alpha_k α k , λ k \lambda_k λ k as in the soft analysis and 0 ≤ Y k ≤ Y max 0 \le Y_k \le Y_{\max} 0 ≤ Y k ≤ Y m a x ,
F ~ K ≤ O ~ ( b ˉ d 2 Y max ( 1 − γ ) 2 Γ K ∑ k = 2 K Λ k + ∑ k = 1 K Γ k k + ∑ k = 1 K σ k s m ) . (20) \widetilde{F}_K \le\widetilde{O}\left(\frac{\bar{b}d^2Y_{\max}}{(1-\gamma)^2}\sqrt{\Gamma_K\sum_{k=2}^{K}\Lambda_k+\sum_{k=1}^{K}\frac{\Gamma_k}{\sqrt{k}}}+\sum_{k=1}^{K}\sigma_k^{\mathrm{sm}}\right). \tag*{(20)} F K ≤ O ( 1 − γ ) 2 b ˉ d 2 Y m a x Γ K k = 2 ∑ K Λ k + k = 1 ∑ K k Γ k + k = 1 ∑ K σ k sm . (20)
Proof. At each selected dual level, the optimistic Lagrangian comparison is the same as in Lemma D.10, with the utility-side statistical terms multiplied by Y k ≤ Y max Y_k \le Y_{\max} Y k ≤ Y m a x . The reward-side terms are unchanged. The returned policy is a softmax approximation of the optimistic planner, so its value loss contributes σ k s m \sigma_k^{\mathrm{sm}} σ k sm . Applying Lemma D.11 to the returned sequence and summing gives (G.15). No certificate width is used in this front-end argument.
Selected-Certificate Reduction
Let K b r \mathcal{K}_{\mathrm{br}} K br contain episodes that stop at a certified grid point, and let K c a p \mathcal{K}_{\mathrm{cap}} K cap contain episodes that reach Y max Y_{\max} Y m a x without certification. Define the certificate grid sensitivity
s g r i d : = C g r i d η h τ h ( 1 − γ ) , (21) s_{\mathrm{grid}} := C_{\mathrm{grid}}\frac{\eta_h}{\tau_h(1-\gamma)}, \tag*{(21)} s grid := C grid τ h ( 1 − γ ) η h , (21)
where C g r i d C_{\mathrm{grid}} C grid is the universal constant from the softmax-sensitivity Lemma C.2. The positive-part clipping is 1-Lipschitz, so neighboring certificates differ by at most s g r i d s_{\mathrm{grid}} s grid .
Proposition F.4 (Selected-certificate regret reduction). On E r e p \mathcal{E}_{\mathrm{rep}} E rep ,
Regret r e t u r n ( K ) ≤ F ~ K + Y max ( ∑ k = 1 K w k ( π k ) + K s g r i d ) , (22) \operatorname{Regret}^{\mathrm{return}}(K) \le\widetilde{F}_K+Y_{\max}\left(\sum_{k=1}^{K}w_k(\pi_k)+Ks_{\mathrm{grid}}\right), \tag*{(22)} Regret return ( K ) ≤ F K + Y m a x ( k = 1 ∑ K w k ( π k ) + K s grid ) , (22) and
Violation h a r d r e t u r n ( K ) ≤ ∑ k ∈ K c a p [ b − V P ⋆ , g π k ( s 1 ) ] + . (23) \operatorname{Violation}_{\mathrm{hard}}^{\mathrm{return}}(K)\le\sum_{k\in\mathcal{K}_{\mathrm{cap}}}\left[b-V^{\pi_k}_{P^\star,g}(s_1)\right]_+. \tag*{(23)} Violation hard return ( K ) ≤ k ∈ K cap ∑ [ b − V P ⋆ , g π k ( s 1 ) ] + . (23)
Proof. If k ∈ K b r k\in\mathcal{K}_{\mathrm{br}} k ∈ K br , then U k ( π k ) ≥ b U_k(\pi_k)\ge b U k ( π k ) ≥ b , so Lemma F.1 gives V P ⋆ , g π k ( s 1 ) ≥ b V^{\pi_k}_{P^\star,g}(s_1)\ge b V P ⋆ , g π k ( s 1 ) ≥ b . Break episodes therefore contribute zero hard violation, proving (G.18).
To convert the Lagrangian front end into reward regret, use V P ⋆ , g π k ( s 1 ) ≥ b V^{\pi_k}_{P^\star,g}(s_1)\ge b V P ⋆ , g π k ( s 1 ) ≥ b :
Regret r e t u r n ( K ) ≤ F ~ K + ∑ k = 1 K Y k [ V P ⋆ , g π k ( s 1 ) − b ] + . (24) \operatorname{Regret}^{\mathrm{return}}(K)\le\widetilde{F}_K+\sum_{k=1}^{K}Y_k\left[V^{\pi_k}_{P^\star,g}(s_1)-b\right]_+. \tag*{(24)} Regret return ( K ) ≤ F K + k = 1 ∑ K Y k [ V P ⋆ , g π k ( s 1 ) − b ] + . (24)
If a break occurs at Y k > 0 Y_k>0 Y k > 0 , the preceding grid point failed the certificate. Grid sensitivity and Lemma F.1 therefore imply
[ V P ⋆ , g π k ( s 1 ) − b ] + ≤ s g r i d + w k ( π k ) . \left[V^{\pi_k}_{P^\star,g}(s_1)-b\right]_+\le s_{\mathrm{grid}}+w_k(\pi_k). [ V P ⋆ , g π k ( s 1 ) − b ] + ≤ s grid + w k ( π k ) .
When Y k = 0 Y_k=0 Y k = 0 , the corresponding term in (G.19) is zero. On a cap episode, the certificate at the explicitly queried cap failed, so
[ V P ⋆ , g π k ( s 1 ) − b ] + ≤ w k ( π k ) . \left[V^{\pi_k}_{P^\star,g}(s_1)-b\right]_+\le w_k(\pi_k). [ V P ⋆ , g π k ( s 1 ) − b ] + ≤ w k ( π k ) .
Using Y k ≤ Y max Y_k\le Y_{\max} Y k ≤ Y m a x proves (G.17).
Hard Violation at the Executed Cap Policy
The remaining argument is performed at the cap policy that is actually returned and subsequently enters the probing kernel; it does not use an unexecuted fixed-Y θ Y_\theta Y θ candidate.
For c ∈ { r , g } c\in\{r,g\} c ∈ { r , g } , define the selected-policy optimistic overestimation
o k , c : = [ V P ^ k , c ^ + b ^ k π k ( s 1 ) − V P ⋆ , c π k ( s 1 ) ] + . (25) o_{k,c}:=\left[V^{\pi_k}_{\widehat{P}_k,\widehat{c}+\widehat{b}_k}(s_1)-V^{\pi_k}_{P^\star,c}(s_1)\right]_+. \tag*{(25)} o k , c := [ V P k , c + b k π k ( s 1 ) − V P ⋆ , c π k ( s 1 ) ] + . (25)
The proof of Lemma D.12 applies to the returned sequence of the hard algorithm and to either normalized signal r r r or g g g ; hence
∑ k = 1 K o k , r ≤ B K , ∑ k = 1 K o k , g ≤ B K . (26) \sum_{k=1}^{K} o_{k,r} \le\mathcal{B}_K,\qquad\sum_{k=1}^{K} o_{k,g} \le\mathcal{B}_K. \tag*{(26)} k = 1 ∑ K o k , r ≤ B K , k = 1 ∑ K o k , g ≤ B K . (26)
Lemma F.5 (Cap-level Slater pressure). Let π ˉ \bar{\pi} π ˉ satisfy the Slater condition V P ⋆ , g π ˉ ( s 1 ) ≥ b + θ V^{\bar{\pi}}_{P^\star,g}(s_1) \ge b+\theta V P ⋆ , g π ˉ ( s 1 ) ≥ b + θ . For any k ∈ K c a p k \in\mathcal{K}_{\mathrm{cap}} k ∈ K cap , if Y max θ ≥ 1 Y_{\max}\theta\ge1 Y m a x θ ≥ 1 , then
[ b − V P ⋆ , g π k ( s 1 ) ] + ≤ o k , r Y max + o k , g + ( 1 + 1 Y max ) ℓ k + σ k s m Y max . (27) \left[b-V^{\pi_k}_{P^\star,g}(s_1)\right]_+ \le\frac{o_{k,r}}{Y_{\max}}+o_{k,g}+\left(1+\frac{1}{Y_{\max}}\right)\ell_k+\frac{\sigma_k^{\mathrm{sm}}}{Y_{\max}}. \tag*{(27)} [ b − V P ⋆ , g π k ( s 1 ) ] + ≤ Y m a x o k , r + o k , g + ( 1 + Y m a x 1 ) ℓ k + Y m a x σ k sm . (27) Proof. At the cap, softmax planning gives
V P ^ k , r + b ^ k π k ( s 1 ) + Y max V P ^ k , g + b ^ k π k ( s 1 ) ≥ V P ^ k , r + b ^ k π ˉ ( s 1 ) + Y max V P ^ k , g + b ^ k π ˉ ( s 1 ) − σ k s m . (28) V^{\pi_k}_{\widehat{P}_k,r+\widehat{b}_k}(s_1)+Y_{\max}V^{\pi_k}_{\widehat{P}_k,g+\widehat{b}_k}(s_1)
\ge V^{\bar{\pi}}_{\widehat{P}_k,r+\widehat{b}_k}(s_1)+Y_{\max}V^{\bar{\pi}}_{\widehat{P}_k,g+\widehat{b}_k}(s_1)-\sigma_k^{\mathrm{sm}}. \tag*{(28)} V P k , r + b k π k ( s 1 ) + Y m a x V P k , g + b k π k ( s 1 ) ≥ V P k , r + b k π ˉ ( s 1 ) + Y m a x V P k , g + b k π ˉ ( s 1 ) − σ k sm . (28) Direct optimism, with ϵ k i n i t ( π ˉ ) ≤ Γ k \epsilon_k^{\mathrm{init}}(\bar{\pi}) \le\Gamma_k ϵ k init ( π ˉ ) ≤ Γ k , yields
V P ^ k , c + b ^ k π ˉ ( s 1 ) ≥ V P ⋆ , c π ˉ ( s 1 ) − ℓ k ( c ∈ { r , g } ) . (29) V^{\bar{\pi}}_{\widehat{P}_k,c+\widehat{b}_k}(s_1)\ge V^{\bar{\pi}}_{P^\star,c}(s_1)-\ell_k\qquad(c\in\{r,g\}). \tag*{(29)} V P k , c + b k π ˉ ( s 1 ) ≥ V P ⋆ , c π ˉ ( s 1 ) − ℓ k ( c ∈ { r , g }) . (29) The definition of o k , c o_{k,c} o k , c upper-bounds the left-hand optimistic values in (G.23) by the corresponding true values plus o k , c o_{k,c} o k , c . Consequently,
V P ⋆ , r π k ( s 1 ) + Y max V P ⋆ , g π k ( s 1 ) + o k , r + Y max o k , g ≥ V P ⋆ , r π ˉ ( s 1 ) + Y max V P ⋆ , g π ˉ ( s 1 ) − ( 1 + Y max ) ℓ k − σ k s m . (30) V^{\pi_k}_{P^\star,r}(s_1)+Y_{\max}V^{\pi_k}_{P^\star,g}(s_1)+o_{k,r}+Y_{\max}o_{k,g}
\ge V^{\bar{\pi}}_{P^\star,r}(s_1)+Y_{\max}V^{\bar{\pi}}_{P^\star,g}(s_1)-(1+Y_{\max})\ell_k-\sigma_k^{\mathrm{sm}}. \tag*{(30)} V P ⋆ , r π k ( s 1 ) + Y m a x V P ⋆ , g π k ( s 1 ) + o k , r + Y m a x o k , g ≥ V P ⋆ , r π ˉ ( s 1 ) + Y m a x V P ⋆ , g π ˉ ( s 1 ) − ( 1 + Y m a x ) ℓ k − σ k sm . (30) If v k = [ b − V P ⋆ , g π k ( s 1 ) ] + > 0 v_k=[b-V^{\pi_k}_{P^\star,g}(s_1)]_+>0 v k = [ b − V P ⋆ , g π k ( s 1 ) ] + > 0 , the Slater condition and the normalization ∣ V P ⋆ , r π ˉ − V P ⋆ , r π ˉ ∣ ≤ 1 \lvert V^{\bar{\pi}}_{P^\star,r}-V^{\bar{\pi}}_{P^\star,r}\rvert\le1 ∣ V P ⋆ , r π ˉ − V P ⋆ , r π ˉ ∣ ≤ 1 imply
Y max ( θ + v k ) ≤ 1 + o k , r + Y max o k , g + ( 1 + Y max ) ℓ k + σ k s m . Y_{\max}(\theta+v_k)\le1+o_{k,r}+Y_{\max}o_{k,g}+(1+Y_{\max})\ell_k+\sigma_k^{\mathrm{sm}}. Y m a x ( θ + v k ) ≤ 1 + o k , r + Y m a x o k , g + ( 1 + Y m a x ) ℓ k + σ k sm . Since Y max θ ≥ 1 Y_{\max}\theta\ge1 Y m a x θ ≥ 1 , subtracting this term and dividing by Y max Y_{\max} Y m a x proves (G.22). The claim is trivial when v k = 0 v_k=0 v k = 0 .
Proposition F.6 (Decaying Slater-induced hard violation). Assume the hypotheses of Proposition F.4. If Y max θ ≥ 1 Y_{\max}\theta\ge1 Y m a x θ ≥ 1 , then
Violation h a r d r e t u r n ( K ) ≤ ( 1 + 1 Y max ) ( 2 B K + ∑ k = 1 K ℓ k + ∑ k = 1 K σ k s m ) (31) \operatorname{Violation}^{\mathrm{return}}_{\mathrm{hard}}(K)\le\left(1+\frac{1}{Y_{\max}}\right)\left(2\mathcal{B}_K+\sum_{k=1}^{K}\ell_k+\sum_{k=1}^{K}\sigma_k^{\mathrm{sm}}\right) \tag*{(31)} Violation hard return ( K ) ≤ ( 1 + Y m a x 1 ) ( 2 B K + k = 1 ∑ K ℓ k + k = 1 ∑ K σ k sm ) (31) up to universal constants. If Y max θ < 1 Y_{\max}\theta<1 Y m a x θ < 1 , the trivial bound Violation h a r d r e t u r n ( K ) ≤ K ≤ θ − 4 \operatorname{Violation}^{\mathrm{return}}_{\mathrm{hard}}(K)\le K\le\theta^{-4} Violation hard return ( K ) ≤ K ≤ θ − 4 applies.
Proof. Combine (G.18) with Lemma F.5, sum over cap episodes, and use (G.21). Extending each nonnegative sum from K c a p \mathcal{K}_{\mathrm{cap}} K cap to all episodes gives (G.26). The second statement follows from Y max = K 1 / 4 Y_{\max}=K^{1/4} Y m a x = K 1/4 .
Polynomial Schedules
Corollary F.7 (Polynomial hard rates). Let ϵ k = k − p \epsilon_k=k^{-p} ϵ k = k − p with p ∈ [ 0 , 1 / 2 ) p\in[0,1/2) p ∈ [ 0 , 1/2 ) , and choose
α k = Θ ~ ( ( Γ k + d 2 ) γ log ( ∣ M ∣ k / δ ) ) \alpha_k=\widetilde{\Theta}\left(\sqrt{(\Gamma_k+d^2)\gamma\log(|\mathcal{M}|k/\delta)}\right) α k = Θ ( ( Γ k + d 2 ) γ log ( ∣ M ∣ k / δ ) ) with a sufficiently large leading constant,
λ k = Θ ~ ( Γ k + d 2 d log ( ∣ M ∣ k / δ ) ) . \lambda_k=\widetilde{\Theta}\left(\frac{\Gamma_k+d^2}{d}\log(|\mathcal{M}|k/\delta)\right). λ k = Θ ( d Γ k + d 2 log ( ∣ M ∣ k / δ ) ) . τ h = Θ ( 1 ( 1 − γ ) K log ∣ A ∣ ) , η h = Θ ( τ h ( 1 − γ ) K 3 / 4 ) . \tau_h=\Theta\left(\frac{1}{(1-\gamma)\sqrt{K\log|\mathcal{A}|}}\right),\qquad
\eta_h=\Theta\left(\frac{\tau_h(1-\gamma)}{K^{3/4}}\right). τ h = Θ ( ( 1 − γ ) K log ∣ A ∣ 1 ) , η h = Θ ( K 3/4 τ h ( 1 − γ ) ) . Under the assumptions of Theorem F.3 and Proposition F.6, with probability at least 1 − δ 1-\delta 1 − δ ,
Regret r e t u r n ( K ) ≤ O ~ ( [ ∣ A ∣ b ˉ d 2 ( 1 − γ ) 2 + ∣ A ∣ 1 − γ ] K 3 / 4 + p ) . (32) \operatorname{Regret}^{\mathrm{return}}(K)\le\widetilde{O}\left(\left[\frac{|\mathcal{A}|\bar{b}d^2}{(1-\gamma)^2}+\sqrt{\frac{|\mathcal{A}|}{1-\gamma}}\right]K^{3/4+p}\right). \tag*{(32)} Regret return ( K ) ≤ O ( [ ( 1 − γ ) 2 ∣ A ∣ b ˉ d 2 + 1 − γ ∣ A ∣ ] K 3/4 + p ) . (32)
and
Violation h a r d r e t u r n ( K ) ≤ O ~ ( 1 θ [ ∣ A ∣ b ˉ d 2 ( 1 − γ ) 2 + ∣ A ∣ 1 − γ ] K 1 / 2 + p ) + O ( θ − 4 ) . (33) \operatorname{Violation}_{\mathrm{hard}}^{\mathrm{return}}(K) \le\widetilde{O}\left(\frac{1}{\theta}\left[\frac{|\mathcal{A}|\bar{b}d^{2}}{(1-\gamma)^{2}}+\sqrt{\frac{|\mathcal{A}|}{1-\gamma}}\right]K^{1/2+p}\right)+O(\theta^{-4}). \tag*{(33)} Violation hard return ( K ) ≤ O ( θ 1 [ ( 1 − γ ) 2 ∣ A ∣ b ˉ d 2 + 1 − γ ∣ A ∣ ] K 1/2 + p ) + O ( θ − 4 ) . (33)
Proof. For polynomial schedules,
Γ K ∑ k = 2 K Λ k + ∑ k = 1 K Γ k k ≤ O ~ ( ∣ A ∣ K 1 / 2 + p ) . \sqrt{\Gamma_K\sum_{k=2}^{K}\Lambda_k+\sum_{k=1}^{K}\frac{\Gamma_k}{\sqrt{k}}}\le\widetilde{O}(|\mathcal{A}|K^{1/2+p}). Γ K k = 2 ∑ K Λ k + k = 1 ∑ K k Γ k ≤ O ( ∣ A ∣ K 1/2 + p ) .
Equations (G.11)–(G.12), Theorem F.3 , Proposition F.4, and Y max = K 1 / 4 Y_{\max}=K^{1/4} Y m a x = K 1/4 give (G.27); the grid contribution is
Y max K s g r i d = O ~ ( K 1 / 2 ) . Y_{\max}K_{\mathrm{sgrid}}=\widetilde{O}(K^{1/2}). Y m a x K sgrid = O ( K 1/2 ) .
Proposition F.6 gives (G.28). We display the harmless factor θ − 1 \theta^{-1} θ − 1 to match the theorem’s problem-dependent Slater convention; for normalized utility θ ≤ 1 \theta\le1 θ ≤ 1 . The exponential softmax residual under the stated temperature is lower order.
The p = 0 p=0 p = 0 Hard Specialization
This appendix specializes the derived-certificate proof in Appendix F to uniform tertiary probing. It is included separately to make the clean return-policy rate transparent.
When p = 0 p=0 p = 0 , ϵ k ≡ 1 \epsilon_k\equiv1 ϵ k ≡ 1 , and hence
q k = q k ′ = q k ′ ′ = U ( A ) , Γ k = ∣ A ∣ , Λ k = ∣ A ∣ . q_k=q'_k=q''_k=U(\mathcal{A}),\qquad\Gamma_k=|\mathcal{A}|,\qquad\Lambda_k=|\mathcal{A}|. q k = q k ′ = q k ′′ = U ( A ) , Γ k = ∣ A ∣ , Λ k = ∣ A ∣.
The algorithmic lower correction becomes
ℓ k = C c e r t ∣ A ∣ ln ( ∣ M ∣ k / δ ) k ( 1 − γ ) . (34) \ell_k=C_{\mathrm{cert}}\sqrt{\frac{|\mathcal{A}|\ln(|\mathcal{M}|k/\delta)}{k(1-\gamma)}}. \tag*{(34)} ℓ k = C cert k ( 1 − γ ) ∣ A ∣ ln ( ∣ M ∣ k / δ ) . (34)
and Lemma F.1 gives
U ‾ k ( π ) ≤ V ⋆ , g π ( s 1 ) ≤ U ‾ k ( π ) + w k ( π ) \underline{U}_k(\pi)\le V^{\pi}_{\star,g}(s_1)\le\overline{U}_k(\pi)+w_k(\pi) U k ( π ) ≤ V ⋆ , g π ( s 1 ) ≤ U k ( π ) + w k ( π )
for every queried policy. The algorithm uses only the first inequality; the second is summed only for the returned sequence.
Lemma G.1 (p = 0 p=0 p = 0 selected correction and width). On the representation-learning good event,
∑ k = 1 K ℓ k ≤ O ~ ( ∣ A ∣ K 1 − γ ) , \sum_{k=1}^{K}\ell_k\le\widetilde{O}\left(\sqrt{\frac{|\mathcal{A}|K}{1-\gamma}}\right), k = 1 ∑ K ℓ k ≤ O ( 1 − γ ∣ A ∣ K ) , and
∑ k = 1 K w k ( π k ) ≤ O ~ ( [ ∣ A ∣ b ˉ d 2 ( 1 − γ ) 2 + ∣ A ∣ 1 − γ ] K ) . (35) \sum_{k=1}^{K}w_k(\pi_k)\le\widetilde{O}\left(\left[\frac{|\mathcal{A}|\bar{b}d^{2}}{(1-\gamma)^{2}}+\sqrt{\frac{|\mathcal{A}|}{1-\gamma}}\right]\sqrt{K}\right). \tag*{(35)} k = 1 ∑ K w k ( π k ) ≤ O ( [ ( 1 − γ ) 2 ∣ A ∣ b ˉ d 2 + 1 − γ ∣ A ∣ ] K ) . (35) Proof. The first display follows by summing ℓ k = O ~ ( k − 1 / 2 ) \ell_k=\widetilde{O}(k^{-1/2}) ℓ k = O ( k − 1/2 ) . The second follows from Lemma F.2 with Γ k = Λ k = ∣ A ∣ \Gamma_k=\Lambda_k=|\mathcal{A}| Γ k = Λ k = ∣ A ∣ .
Proposition G.2 (p = 0 p=0 p = 0 certificate reduction). Under the hypotheses of Proposition F.4,
Regret r e t u r n ( K ) ≤ O ~ ( [ ∣ A ∣ b ˉ d 2 ( 1 − γ ) 2 + ∣ A ∣ 1 − γ ] K 3 / 4 ) . (36) \operatorname{Regret}^{\mathrm{return}}(K)\le\widetilde{O}\left(\left[\frac{|\mathcal{A}|\bar{b}d^{2}}{(1-\gamma)^{2}}+\sqrt{\frac{|\mathcal{A}|}{1-\gamma}}\right]K^{3/4}\right). \tag*{(36)} Regret return ( K ) ≤ O ( [ ( 1 − γ ) 2 ∣ A ∣ b ˉ d 2 + 1 − γ ∣ A ∣ ] K 3/4 ) . (36) and hard violation is supported only on cap episodes.
Proof. For p = 0 p=0 p = 0 , the hard front end in Theorem F.3 is O ~ ( ∣ A ∣ b ˉ d 2 K 3 / 4 / ( 1 − γ ) 2 ) \widetilde{O}(|\mathcal{A}|\bar{b}d^{2}K^{3/4}/(1-\gamma)^{2}) O ( ∣ A ∣ b ˉ d 2 K 3/4 / ( 1 − γ ) 2 ) . Apply Proposition F.4 and Lemma G.1 . With η h = Θ ( τ h ( 1 − γ ) / K 3 / 4 ) \eta_h=\Theta(\tau_h(1-\gamma)/K^{3/4}) η h = Θ ( τ h ( 1 − γ ) / K 3/4 ) , the grid term is
Y max K s g r i d = O ~ ( K 1 / 2 ) , Y_{\max}K_{\mathrm{sgrid}}=\widetilde{O}(K^{1/2}), Y m a x K sgrid = O ( K 1/2 ) , and is lower order.
Theorem G.3 (Slater-induced hard violation for p = 0 p = 0 p = 0 ). Suppose Assumptions 2.3, 2.5, and C.1 hold, ∣ M ∣ < ∞ |\mathcal{M}| < \infty ∣ M ∣ < ∞ , P ⋆ ∈ M P^\star\in\mathcal{M} P ⋆ ∈ M , and the representation-learning good event holds. Then Algorithm 2 with ϵ k ≡ 1 \epsilon_k \equiv1 ϵ k ≡ 1 satisfies
Violation h a r d r e t u r n ( K ) ≤ O ~ ( 1 θ [ ∣ A ∣ d ˉ 2 ( 1 − γ ) 2 + ∣ A ∣ 1 − γ ] K ) + O ( θ − 4 ) . (37) \operatorname{Violation}_{\mathrm{hard}}^{\mathrm{return}}(K) \le\widetilde{O}\left(\frac{1}{\theta}\left[\frac{|\mathcal{A}|\bar{d}^2}{(1-\gamma)^2}+\sqrt{\frac{|\mathcal{A}|}{1-\gamma}}\right]\sqrt{K}\right)+O(\theta^{-4}). \tag*{(37)} Violation hard return ( K ) ≤ O ( θ 1 [ ( 1 − γ ) 2 ∣ A ∣ d ˉ 2 + 1 − γ ∣ A ∣ ] K ) + O ( θ − 4 ) . (37)
Proof. Break episodes are feasible by the one-sided inequality in Lemma F.1. At a cap episode, the returned policy is the actually queried Y max Y_{\max} Y m a x policy, so Lemma F.5 applies and its overestimation terms are summable along the returned sequence. Proposition F.6 and Lemma G.1 give (H.4). If K 1 / 4 θ < 1 K^{1/4}\theta< 1 K 1/4 θ < 1 , the trivial K ≤ θ − 4 K \le\theta^{-4} K ≤ θ − 4 bound accounts for the last term.
Experiments
Job scheduling.
To validate the convergence of REP-PD-TEG in CMDPs, we first evaluate it in a simulated job scheduling environment. This toy domain lets us visualize how the learned representation, optimistic bonus, and primal-dual update interact under unknown dynamics.
Following [17 , 18 ] , we design a constrained MDP with state space S = { 0 , 1 , … , 9 } \mathcal{S} = \{0, 1, \ldots, 9\} S = { 0 , 1 , … , 9 } , where s s s represents the number of pending jobs. The agent chooses either to withhold jobs (a = 0 a = 0 a = 0 ) or submit them (a = 1 a = 1 a = 1 ) over a horizon H = 10 H = 10 H = 10 . The transition dynamics follow:
s h + 1 = { max { s h − 2 a h , 0 } with probability 0.8 max { s h − a h , 0 } with probability 0.1 s h otherwise. s_{h+1} =
\begin{cases}
\max\{s_h - 2a_h, 0\} & \text{with probability } 0.8 \\
\max\{s_h - a_h, 0\} & \text{with probability } 0.1 \\
s_h & \text{otherwise.}
\end{cases} s h + 1 = ⎩ ⎨ ⎧ max { s h − 2 a h , 0 } max { s h − a h , 0 } s h with probability 0.8 with probability 0.1 otherwise.
We assume that at time steps from 3 to 6, the reward is 1 − 0.9 a 1 - 0.9a 1 − 0.9 a , while at all other time steps the reward is 1 − 0.2 a 1 - 0.2a 1 − 0.2 a . This mimics a system in which job processing becomes temporarily expensive. The utility function g ( s , a , s ′ ) = s − s ′ 2 g(s, a, s') = \frac{s-s'}{2} g ( s , a , s ′ ) = 2 s − s ′ quantifies job-processing efficiency, with per-episode constraint
∑ h = 0 H − 1 g ( s h , a h , s h + 1 ) ≥ b = 2. \sum_{h=0}^{H-1} g(s_h, a_h, s_{h+1}) \ge b = 2. h = 0 ∑ H − 1 g ( s h , a h , s h + 1 ) ≥ b = 2.
We do not hand-craft transition features in this domain. Instead, the appendix implementation learns latent transition factors ϕ ^ ( s , a ) \hat{\phi}(s, a) ϕ ^ ( s , a ) and μ ^ ( s ′ ) \hat{\mu}(s') μ ^ ( s ′ ) directly from the collected transitions using one-hot encodings of the discrete states and actions. After each episode, the newly collected transitions are appended to the dataset, the representation learner is updated on the accumulated data for multiple gradient steps, and the primal-dual policy update is then executed with the induced exploration bonus.
To demonstrate the effectiveness of our constraint-aware design, we compare against REP-UCB [41 ] , an unconstrained low-rank MDP approach using similar representation learning but without dual-variable updates, and a linear method [17 ] where the transition feature is known. As shown in Figure 1 , REP-PD-TEG has higher regret than REP-UCB on this toy instance but substantially lower cumulative constraint violation, illustrating the tradeoff induced by constraint-aware learning. Despite not knowing the transition feature, REP-PD-TEG is comparable in scale to the known-feature linear method of Ghosh et al. [17 ] on both metrics. Because this illustrative plot reports a single run without uncertainty bands, we do not infer statistical significance or an empirical asymptotic rate from it.
Figure 1. Illustrative single-run comparison of REP-PD-TEG, REP-UCB, and the known-feature linear method of Ghosh et al. [17 ] on the job-scheduling CMDP.
Practical Neural Implementation
Neural Safety Gym implementation. To probe scalability beyond the toy domain, we instantiate a practical neural variant, denoted REP-PD-NN for neural REP-PD-TEG. The theoretical algorithm uses a per-round full-batch MLE representation-learning step, which is difficult to scale directly to continuous-control domains. REP-PD-NN therefore replaces this step with a mini-batch SPEDER-style stochastic representation objective [37 ] , while preserving the primal-dual safe RL structure. The learned state-action representation is used as the input to neural reward and cost critics, and the policy is trained with a Lagrangian-style constrained policy update. We evaluate this variant on Safety Gym tasks [35 ] .
Algorithm 3 REP-PD-NN
Input: CMDP environment E \mathcal{E} E , cost budget d c o s t d_{\mathrm{cost}} d cost , replay buffer D \mathcal{D} D , warm-up steps W W W , gradient steps per update G G G , dual stepsize ρ \rho ρ .
Parameters: representation η \eta η , actor θ \theta θ , reward critics { ψ i } i = 1 2 \{\psi_i\}_{i=1}^{2} { ψ i } i = 1 2 , cost critics { ω i } i = 1 2 \{\omega_i\}_{i=1}^{2} { ω i } i = 1 2 , multiplier λ \lambda λ .
Initialize target networks η ˉ ← η \bar{\eta} \leftarrow\eta η ˉ ← η , ψ ˉ i ← ψ i \bar{\psi}_i \leftarrow\psi_i ψ ˉ i ← ψ i , and ω ˉ i ← ω i \bar{\omega}_i \leftarrow\omega_i ω ˉ i ← ω i .
Collect W W W transitions with a random or stochastic policy and store them in D \mathcal{D} D .
for t = 1 , … , T t = 1,\ldots,T t = 1 , … , T do
Sample a t ∼ π θ ( ⋅ ∣ s t ) a_t \sim\pi_\theta(\cdot|s_t) a t ∼ π θ ( ⋅ ∣ s t ) , execute a t a_t a t in E \mathcal{E} E , and store ( s t , a t , r t , c t , s t + 1 , m t ) (s_t,a_t,r_t,c_t,s_{t+1},m_t) ( s t , a t , r t , c t , s t + 1 , m t ) in D \mathcal{D} D .
if an episode terminates with cumulative cost J ^ c \hat{J}_c J ^ c then
Update λ ← [ λ + ρ ( J ^ c − d c o s t ) ] + \lambda\leftarrow[\lambda+ \rho(\hat{J}_c-d_{\mathrm{cost}})]_+ λ ← [ λ + ρ ( J ^ c − d cost ) ] + .
end if
for g = 1 , … , G g = 1,\ldots,G g = 1 , … , G do
Sample a mini-batch B \mathcal{B} B from D \mathcal{D} D .
Update the representation network by minimizing J r e p ( η ) J_{\mathrm{rep}}(\eta) J rep ( η ) on B \mathcal{B} B .
Encode current pairs with ϕ η ( s , a ) \phi_\eta(s,a) ϕ η ( s , a ) and target pairs with ϕ η ˉ ( s ′ , a ′ ) \phi_{\bar{\eta}}(s',a') ϕ η ˉ ( s ′ , a ′ ) .
Update reward critics by minimizing the SAC reward Bellman loss.
Update cost critics by minimizing the cost Bellman loss.
Update the actor by minimizing J π ( θ ) J_\pi(\theta) J π ( θ ) with the current multiplier λ \lambda λ .
Optionally update the entropy temperature α \alpha α .
Soft-update target networks η ˉ \bar{\eta} η ˉ , { ψ ˉ i } i = 1 2 \{\bar{\psi}_i\}_{i=1}^{2} { ψ ˉ i } i = 1 2 , and { ω ˉ i } i = 1 2 \{\bar{\omega}_i\}_{i=1}^{2} { ω ˉ i } i = 1 2 .
end for
end for
Return policy π θ \pi_\theta π θ and representation ϕ η \phi_\eta ϕ η .
Let ϕ η ( s , a ) \phi_\eta(s,a) ϕ η ( s , a ) denote the learned state-action representation, and J r e p ( η ) J_{\mathrm{rep}}(\eta) J rep ( η ) denote the objective of representation learning. The actor is trained with the Lagrangian objective
J π ( θ ) = E [ α log π θ ( a ∣ s ) − min i Q ψ i r ( ϕ η ( s , a ) ) + λ min i Q ω i c ( ϕ η ( s , a ) ) ] . J_\pi(\theta) = \mathbb{E}\left[\alpha\log\pi_\theta(a|s) - \min_i Q^{r}_{\psi_i}(\phi_\eta(s,a)) + \lambda\min_i Q^{c}_{\omega_i}(\phi_\eta(s,a))\right]. J π ( θ ) = E [ α log π θ ( a ∣ s ) − i min Q ψ i r ( ϕ η ( s , a )) + λ i min Q ω i c ( ϕ η ( s , a )) ] .
where λ \lambda λ is the Lagrange multiplier for the cost constraint.
Raw-state baseline. For the raw-state baseline, we use the same Lagrangian training skeleton but remove the representation update. The critics and actor receive raw state-action pairs ( s , a ) (s,a) ( s , a ) instead of ϕ η ( s , a ) \phi_\eta(s,a) ϕ η ( s , a ) . Thus, any performance difference is attributable to the learned representation rather than to a different safe RL backbone.
Table 3 reports the completed positive results after 1M interaction steps and 3 random seeds, using the final deterministic evaluation checkpoint (eval_return / eval_cost). Across the three reported tasks, REP-PD-NN substantially improves final evaluation return and also reduces final evaluation cost relative to the raw baseline. These results indicate that a learned representation can improve both task performance and safety on a meaningful subset of continuous-control safe RL tasks.
Table 3. Safety Gym results for the practical neural implementation REP-PD-NN . Metrics are the mean and standard deviation over 3 seeds at the final deterministic evaluation checkpoint after 1M environment steps.
Computational Resources
All experiments were run on a local institutional server equipped with eight NVIDIA RTX 4090 GPUs, each with 24GB of memory, and standard x86_64 CPUs. The tabular job-scheduling experiment is lightweight and can be reproduced on a CPU-only machine. Each Safety Gym training run uses one GPU; the eight-GPU server was used only to run independent seeds and methods in parallel, and is not required for reproducing any single run. The implementation uses Python with PyTorch, CUDA-enabled GPU execution, MuJoCo, and Safety Gym.
Table 4 summarizes the approximate compute required to reproduce the reported experiments. For Safety Gym, each environment is evaluated with two methods, raw and REP-PD-NN, and three random seeds, resulting in six runs per environment and eighteen final training runs in total.
Table 4. Approximate compute required to reproduce the reported experiments. Runtime denotes wall-clock time for one run. Total compute is over the reported methods and seeds.
Thus, the reported Safety Gym results require approximately 144–360 GPU-hours in total. When all eight GPUs are used in parallel, the final runs can be completed in roughly 18–45 wall-clock hours. During method development, we also ran preliminary hyperparameter sweeps and diagnostic experiments that are not required to reproduce the reported tables. These preliminary runs used additional GPU time of the same order as the final reported experiment set; failed or preliminary runs were excluded from the reported results.