Introduction
Safe Reinforcement Learning (SRL) has been widely studied and applied in real-world settings. For example, SRL addresses collision avoidance in autonomous driving [19 ] and resource allocation in healthcare [28 ] , underscoring its importance in safety-critical domains. To address these challenges, a standard framework for SRL is the Constrained Markov Decision Processes (CMDPs). In a CMDP, the agent learns a policy that maximizes expected cumulative rewards while satisfying specified underlying constraints. Designing efficient learning-based algorithms for CMDPs has attracted considerable attention. Model-based methods [10 ] learn an explicit CMDP model, but incur substantial storage and computational overhead in large-scale or continuous state spaces. Model-free methods avoid this: tabular variants [26 , 8 ] store value functions directly, while recent works adopt linear function approximation (LFA) to handle large state-action spaces [11 ] , achieving provably near-optimal O ~ ( K 1 / 2 ) \widetilde{O}(K^{1/2}) O ( K 1/2 ) regret and violation rates.
A central limitation of existing LFA-based methods is the assumption that the underlying features are known a priori [8 , 11 ] . In practice, such features are typically unknown, rendering these
Preprint. methods inapplicable. Meanwhile, the low-rank MDP framework [3 ] enables representation learning without prior feature knowledge, and provably efficient algorithms exist for the unconstrained setting [29 , 17 , 15 ] . However, extending representation learning to CMDPs is non-trivial. In the unconstrained setting, representation learning and policy optimization are largely decoupled; in a CMDP, the learned representation must simultaneously support both reward maximization and constraint satisfaction, and errors in the representation propagate into both objectives in a tightly coupled manner that standard primal-dual analyses cannot accommodate.
Table 1. Comparison with existing work, where d d d denotes the feature dimension, H H H the horizon, K K K the number of deployment rounds, γ \gamma γ the Slater gap, and ε \varepsilon ε the target suboptimality (the learned policy’s value is within ε \varepsilon ε of optimal with high probability). Rates hide polylog factors via O ~ ( ⋅ ) \widetilde{O}(\cdot) O ( ⋅ ) . LFA denotes linear function approximation; ✗ in the LFA column indicates tabular or non-parametric methods. Bounds stated in terms of ε \varepsilon ε are PAC sample-complexity guarantees for unconstrained problems and are not directly comparable to regret.
In this work, we bring representation learning to low-rank CMDPs and propose MFRLC, which in each round first learns representations via a minimax objective and then fits the value function using the learned features within a primal-dual LSVI-UCB framework. This constitutes the first approach for online low-rank CMDPs with unknown features. We prove that MFRLC achieves the same bound for both cumulative reward regret and cumulative constraint violation: O ~ ( H 3 d 2 K 3 / 4 ∣ A ∣ 3 / 2 / γ ) \widetilde{O}(H^{3}d^{2}K^{3/4}|\mathcal{A}|^{3/2}/\gamma) O ( H 3 d 2 K 3/4 ∣ A ∣ 3/2 / γ ) , where K K K is the number of deployment rounds, H H H is the horizon, d d d is the representation dimension, and γ \gamma γ is the Slater gap. The ∣ A ∣ 3 / 2 |\mathcal{A}|^{3/2} ∣ A ∣ 3/2 factor arises from uniform exploration over actions in the representation learning step, and the K 3 / 4 K^{3/4} K 3/4 dependence arises from the accumulated statistical error of representation learning (Remark 8). Both bounds are upper bounds. No matching lower bound is known for online low-rank MDPs with unknown features, with or without constraints, so we do not claim that the K 3 / 4 K^{3/4} K 3/4 rate or the dependence on d d d , H H H , and ∣ A ∣ |\mathcal{A}| ∣ A ∣ is optimal. Known-feature CMDPs admit O ~ ( K 1 / 2 ) \widetilde{O}(K^{1/2}) O ( K 1/2 ) regret and violation [11 ] ; whether this rate is attainable with unknown features remains open.
CMDP Preliminaries
We consider an episodic CMDP ⟨ S , A , H , r , g , P , d 0 ⟩ \langle\mathcal{S},\mathcal{A},H,r,g,\mathbb{P},d_{0}\rangle ⟨ S , A , H , r , g , P , d 0 ⟩ , where S \mathcal{S} S is the state space (possibly infinite or continuous), A \mathcal{A} A the action space (finite, with cardinality ∣ A ∣ |\mathcal{A}| ∣ A ∣ ), H H H the fixed episode length, r = { r h } h = 1 H r=\{r_{h}\}_{h=1}^{H} r = { r h } h = 1 H and g = { g h } h = 1 H g=\{g_{h}\}_{h=1}^{H} g = { g h } h = 1 H the collections of reward and utility functions, respectively, with r h , g h : S × A → [ 0 , 1 ] r_{h},g_{h}:\mathcal{S}\times\mathcal{A}\to[0,1] r h , g h : S × A → [ 0 , 1 ] assumed to be deterministic, P = { P h } h = 1 H \mathbb{P}=\{P_{h}\}_{h=1}^{H} P = { P h } h = 1 H the collection of transition kernels with P h ( s h + 1 ∣ s h , a h ) : S × A × S → [ 0 , 1 ] P_{h}(s_{h+1}\mid s_{h},a_{h}):\mathcal{S}\times\mathcal{A}\times\mathcal{S}\to[0,1] P h ( s h + 1 ∣ s h , a h ) : S × A × S → [ 0 , 1 ] being the probability of moving to s h + 1 s_{h+1} s h + 1 from s h s_{h} s h under a h a_{h} a h , and s 1 ∼ d 0 s_{1}\sim d_{0} s 1 ∼ d 0 the initial state of each episode. At each step h ∈ [ H ] h\in[H] h ∈ [ H ] of round k ∈ [ K ] k\in[K] k ∈ [ K ] , the agent takes action a h k a_{h}^{k} a h k at state s h k s_{h}^{k} s h k , receives reward r h ( s h k , a h k ) r_{h}(s_{h}^{k},a_{h}^{k}) r h ( s h k , a h k ) and utility g h ( s h k , a h k ) g_{h}(s_{h}^{k},a_{h}^{k}) g h ( s h k , a h k ) , and transitions to s h + 1 k ∼ P h ( ⋅ ∣ s h k , a h k ) s_{h+1}^{k}\sim P_{h}(\cdot\mid s_{h}^{k},a_{h}^{k}) s h + 1 k ∼ P h ( ⋅ ∣ s h k , a h k ) ; each episode terminates after H H H steps. The policy space is Δ ( A ∣ S , H ) : = { { π h ( ⋅ ∣ ⋅ ) } h = 1 H : π h ( ⋅ ∣ s ) ∈ Δ ( A ) , ∀ s ∈ S , h ∈ [ H ] } \Delta(\mathcal{A}\mid\mathcal{S},H):=\{\{\pi_{h}(\cdot\mid\cdot)\}_{h=1}^{H}:\pi_{h}(\cdot\mid s)\in\Delta(\mathcal{A}),\ \forall s\in\mathcal{S},\ h\in[H]\} Δ ( A ∣ S , H ) := {{ π h ( ⋅ ∣ ⋅ ) } h = 1 H : π h ( ⋅ ∣ s ) ∈ Δ ( A ) , ∀ s ∈ S , h ∈ [ H ]} , where Δ ( A ) \Delta(\mathcal{A}) Δ ( A ) is the probability simplex over the action space; π k ∈ Δ ( A ∣ S , H ) \pi^{k}\in\Delta(\mathcal{A}\mid\mathcal{S},H) π k ∈ Δ ( A ∣ S , H ) denotes the policy deployed in round k k k , with π h k ( ⋅ ∣ s h k ) : S → Δ ( A ) \pi_{h}^{k}(\cdot\mid s_{h}^{k}):\mathcal{S}\to\Delta(\mathcal{A}) π h k ( ⋅ ∣ s h k ) : S → Δ ( A ) specifying the action distribution at s h k s_{h}^{k} s h k . Given π = { π h } h = 1 H \pi=\{\pi_{h}\}_{h=1}^{H} π = { π h } h = 1 H , the value function of the expected cumulative reward from s h = s s_{h}=s s h = s under π \pi π and action-value function (from ( s h = s , a h = a ) (s_{h}=s,a_{h}=a) ( s h = s , a h = a ) ) are:
V r , h π ( s ) = E π [ ∑ i = h H r i ( s i , a i ) ∣ s h = s ] , Q r , h π ( s , a ) = E π [ ∑ i = h H r i ( s i , a i ) ∣ s h = s , a h = a ] . V_{r,h}^{\pi}(s)=\mathbb{E}_{\pi}\left[\left.\sum_{i=h}^{H}r_{i}(s_{i},a_{i})\right|s_{h}=s\right], \qquad Q_{r,h}^{\pi}(s,a)=\mathbb{E}_{\pi}\left[\left.\sum_{i=h}^{H}r_{i}(s_{i},a_{i})\right|s_{h}=s,a_{h}=a\right]. V r , h π ( s ) = E π [ i = h ∑ H r i ( s i , a i ) s h = s ] , Q r , h π ( s , a ) = E π [ i = h ∑ H r i ( s i , a i ) s h = s , a h = a ] .
where the expectation is over a i ∼ π i ( ⋅ ∣ s i ) a_i \sim\pi_i(\cdot\mid s_i) a i ∼ π i ( ⋅ ∣ s i ) and s i + 1 ∼ P i ( ⋅ ∣ s i , a i ) s_{i+1} \sim P_i(\cdot\mid s_i,a_i) s i + 1 ∼ P i ( ⋅ ∣ s i , a i ) . The utility value and action-value functions V g , h π ( s ) , Q g , h π ( s , a ) V^\pi_{g,h}(s), Q^\pi_{g,h}(s,a) V g , h π ( s ) , Q g , h π ( s , a ) are defined analogously by replacing r r r with g g g ; for brevity, we write V j , h π ( s ) V^\pi_{j,h}(s) V j , h π ( s ) and Q j , h π ( s , a ) Q^\pi_{j,h}(s,a) Q j , h π ( s , a ) for j ∈ { r , g } j \in\{r,g\} j ∈ { r , g } .
We say a policy π \pi π is feasible if V g , 1 π ( s 1 ) ≥ b V^\pi_{g,1}(s_1) \ge b V g , 1 π ( s 1 ) ≥ b . The agent aims to learn the optimal feasible policy π ⋆ \pi^\star π ⋆ by solving the following optimization problem:
max π ∈ Δ ( A ∣ S , H ) V r , 1 π ( s 1 ) , subject to V g , 1 π ( s 1 ) ≥ b , (1) \max_{\pi\in\Delta(\mathcal{A} \mid\mathcal{S},H)} V^\pi_{r,1}(s_1), \qquad\text{subject to}\qquad V^\pi_{g,1}(s_1) \ge b, \tag*{(1)} π ∈ Δ ( A ∣ S , H ) max V r , 1 π ( s 1 ) , subject to V g , 1 π ( s 1 ) ≥ b , (1)
where the threshold b ∈ [ 0 , H ] b \in[0,H] b ∈ [ 0 , H ] . Under the online learning setting of an episodic CMDP, the agent interacts with the environment over a sequence of rounds, and the learning performance is measured by regret and constraint violation as follows:
Regret ( K ) = ∑ k = 1 K ( V r , 1 π ⋆ ( s 1 ) − V r , 1 π k ( s 1 ) ) , Violation ( K ) : = ( ∑ k = 1 K ( b − V g , 1 π k ( s 1 ) ) ) + , \operatorname{Regret}(K) = \sum_{k=1}^{K}\left(V^{\pi^\star}_{r,1}(s_1)-V^{\pi^k}_{r,1}(s_1)\right), \qquad\operatorname{Violation}(K) := \left(\sum_{k=1}^{K}\left(b-V^{\pi^k}_{g,1}(s_1)\right)\right)_+, Regret ( K ) = k = 1 ∑ K ( V r , 1 π ⋆ ( s 1 ) − V r , 1 π k ( s 1 ) ) , Violation ( K ) := ( k = 1 ∑ K ( b − V g , 1 π k ( s 1 ) ) ) + ,
with π k \pi^k π k being the policy deployed in round k k k . Both quantities are random variables depending on the algorithm’s internal randomness and the sampled trajectories; our guarantees on them hold with high probability (see Theorem 3). The regret quantifies the difference in total rewards achieved by the agent’s policy π k \pi^k π k compared to the optimal feasible policy π ⋆ \pi^\star π ⋆ across K K K rounds, and the constraint violation measures the cumulative shortfall of utility against the predefined threshold b b b over K K K rounds.
Access model and interaction accounting. We call each index k k k a deployment round . In round k k k , MFRLC executes π k \pi^k π k for one trajectory and also collects two auxiliary transitions at every step h h h for representation learning (Algorithm 1, line 4), each obtained by rolling in with π k − 1 \pi^{k-1} π k − 1 to step h h h and taking a uniform action, so no generative model is required. Regret and Violation contain one term per deployed policy and do not charge the auxiliary roll-ins; our guarantees are deployment-regret guarantees with total interaction complexity K ( 2 H + 1 ) K(2H+1) K ( 2 H + 1 ) trajectories and K ( H 2 + 2 H ) K(H^2+2H) K ( H 2 + 2 H ) transitions.
Low-Rank MDP and Representation Learning
Conventional work on learning MDPs typically assumes a general linear MDP framework [13 , 14 , 5 ] , where features are considered known, and function approximation methods are used to maximize rewards without considering constraints. While this linear MDP assumption has been effectively extended to CMDPs [4 , 12 ] , the transition to low-rank MDPs introduces additional layers of complexity, defined as follows.
Definition 1 (Low-rank MDP [3 ] ). Consider a transition kernel P h ⋆ : S × A → Δ ( S ) P^\star_h : \mathcal{S} \times\mathcal{A} \to\Delta(\mathcal{S}) P h ⋆ : S × A → Δ ( S ) which can be decomposed into low-rank components of dimension d ∈ N d \in\mathbb{N} d ∈ N . Specifically, there exist functions ϕ ⋆ = { ϕ h ⋆ } h = 1 H \phi^\star= \{\phi^\star_h\}_{h=1}^{H} ϕ ⋆ = { ϕ h ⋆ } h = 1 H and μ ⋆ = { μ h ⋆ } h = 1 H \mu^\star= \{\mu^\star_h\}_{h=1}^{H} μ ⋆ = { μ h ⋆ } h = 1 H with ϕ h ⋆ : S × A → R d \phi^\star_h : \mathcal{S} \times\mathcal{A} \to\mathbb{R}^d ϕ h ⋆ : S × A → R d and μ h ⋆ : S → R d \mu^\star_h : \mathcal{S} \to\mathbb{R}^d μ h ⋆ : S → R d , such that for all s , s ′ ∈ S , a ∈ A s,s' \in\mathcal{S}, a \in\mathcal{A} s , s ′ ∈ S , a ∈ A , and h ∈ [ H ] h \in[H] h ∈ [ H ] :
P h ⋆ ( s ′ ∣ s , a ) = ⟨ ϕ h ⋆ ( s , a ) , μ h ⋆ ( s ′ ) ⟩ . P^\star_h(s' \mid s,a) = \langle\phi^\star_h(s,a),\mu^\star_h(s')\rangle. P h ⋆ ( s ′ ∣ s , a ) = ⟨ ϕ h ⋆ ( s , a ) , μ h ⋆ ( s ′ )⟩ . Furthermore, there exist vectors θ r , h ⋆ , θ g , h ⋆ ∈ R d \theta^\star_{r,h},\theta^\star_{g,h} \in\mathbb{R}^d θ r , h ⋆ , θ g , h ⋆ ∈ R d such that for any ( s , a ) ∈ S × A (s,a) \in\mathcal{S} \times\mathcal{A} ( s , a ) ∈ S × A :
r h ( s , a ) = ⟨ ϕ h ⋆ ( s , a ) , θ r , h ⋆ ⟩ , g h ( s , a ) = ⟨ ϕ h ⋆ ( s , a ) , θ g , h ⋆ ⟩ . r_h(s,a) = \langle\phi^\star_h(s,a),\theta^\star_{r,h}\rangle,\qquad g_h(s,a) = \langle\phi^\star_h(s,a),\theta^\star_{g,h}\rangle. r h ( s , a ) = ⟨ ϕ h ⋆ ( s , a ) , θ r , h ⋆ ⟩ , g h ( s , a ) = ⟨ ϕ h ⋆ ( s , a ) , θ g , h ⋆ ⟩ . The feature function satisfies ∥ ϕ h ⋆ ( s , a ) ∥ 2 ≤ 1 \lVert\phi^\star_h(s,a)\rVert_2 \le1 ∥ ϕ h ⋆ ( s , a ) ∥ 2 ≤ 1 for all state-action pairs ( s , a ) (s,a) ( s , a ) , and for any function f : S → [ 0 , 1 ] f:\mathcal{S}\to[0,1] f : S → [ 0 , 1 ] , ∥ ∫ μ h ⋆ ( s ) f ( s ) d s ∥ 2 ≤ d \lVert\int\mu^\star_h(s)f(s)\,\mathrm{d}s\rVert_2 \le\sqrt{d} ∥ ∫ μ h ⋆ ( s ) f ( s ) d s ∥ 2 ≤ d . An MDP M \mathcal{M} M is considered low-rank if the transition kernel P h P_h P h at each time step h ∈ [ H ] h \in[H] h ∈ [ H ] admits such a decomposition.
Under the low-rank setting, the feature representations ϕ ⋆ \phi^\star ϕ ⋆ are no longer presumed known. This necessitates the development of algorithms that are not only capable of learning these features but also ensure that the resulting policy adheres to the constraints imposed by the CMDP framework.
Assumption 1 (Realizable feature class). The learner has access to a finite feature class Φ = Φ 1 × ⋯ × Φ H \Phi= \Phi_1 \times\cdots\times\Phi_H Φ = Φ 1 × ⋯ × Φ H , where each ϕ h ∈ Φ h \phi_h \in\Phi_h ϕ h ∈ Φ h maps S × A \mathcal{S} \times\mathcal{A} S × A to R d \mathbb{R}^d R d with ∥ ϕ h ( s , a ) ∥ 2 ≤ 1 \lVert\phi_h(s,a)\rVert_2 \le1 ∥ ϕ h ( s , a ) ∥ 2 ≤ 1 for all ( s , a ) (s,a) ( s , a ) , and Φ \Phi Φ contains the true representation of Definition 1, that is, ϕ ⋆ ∈ Φ \phi^\star\in\Phi ϕ ⋆ ∈ Φ .
Assumption 1 is the standard setting of model-free representation learning in low-rank MDPs [3 , 17 ] . Finiteness enters only through log ∣ Φ ∣ \log|\Phi| log ∣Φ∣ and can be replaced by a covering number, while realizability Algorithm 1 Model-Free Representation Learning with Low-rank CMDP (MFRLC)
Require: Feature classes { Φ h } h = 1 H \{\Phi_h\}_{h=1}^{H} { Φ h } h = 1 H , regularization λ k \lambda_k λ k , Slater parameter γ > 0 \gamma> 0 γ > 0
Output: Sequence of policies { π k } k = 1 K \{\pi^k\}_{k=1}^{K} { π k } k = 1 K
Initialize: Y 1 = 0 Y_1 = 0 Y 1 = 0 , ξ = 2 H / γ \xi= 2H/\gamma ξ = 2 H / γ , η = ξ / K H 2 \eta= \xi/\sqrt{KH^2} η = ξ / K H 2 ; set π 0 \pi^0 π 0 to any reference policy and replay buffers D h = ∅ \mathcal{D}_h = \varnothing D h = ∅ , D h ′ = ∅ \mathcal{D}'_h = \varnothing D h ′ = ∅ for all h ∈ [ H ] h \in[H] h ∈ [ H ] .
for k = 1 k = 1 k = 1 to K K K do
for h = 1 h = 1 h = 1 to H H H do ▷ Exploration: branching from d h π k − 1 d_h^{\pi^{k-1}} d h π k − 1
Sample s h k ∼ d h π k − 1 s_h^k \sim d_h^{\pi^{k-1}} s h k ∼ d h π k − 1 , a ~ h k ∼ U ( A ) \tilde{a}_h^k \sim U(\mathcal{A}) a ~ h k ∼ U ( A ) , s h + 1 k ∼ P h ⋆ ( ⋅ ∣ s h k , a ~ h k ) s_{h+1}^k \sim P_h^\star(\cdot\mid s_h^k,\tilde{a}_h^k) s h + 1 k ∼ P h ⋆ ( ⋅ ∣ s h k , a ~ h k ) , s ~ ∼ d h π k − 1 \tilde{s} \sim d_h^{\pi^{k-1}} s ~ ∼ d h π k − 1 , a ~ ∼ U ( A ) \tilde{a} \sim U(\mathcal{A}) a ~ ∼ U ( A ) , s ~ ′ ∼ P h ⋆ ( ⋅ ∣ s ~ , a ~ ) \tilde{s}' \sim P_h^\star(\cdot\mid\tilde{s},\tilde{a}) s ~ ′ ∼ P h ⋆ ( ⋅ ∣ s ~ , a ~ ) .
D h ← D h ∪ { ( s h k , a ~ h k , s h + 1 k ) } \mathcal{D}_h \leftarrow\mathcal{D}_h \cup\{(s_h^k,\tilde{a}_h^k,s_{h+1}^k)\} D h ← D h ∪ {( s h k , a ~ h k , s h + 1 k )} , D h ′ ← D h ′ ∪ { ( s ~ , a ~ , s ~ ′ ) } \mathcal{D}'_h \leftarrow\mathcal{D}'_h \cup\{(\tilde{s},\tilde{a},\tilde{s}')\} D h ′ ← D h ′ ∪ {( s ~ , a ~ , s ~ ′ )} .
ϕ ^ h k ← REPLEARN ( D h , D h ′ , Φ h , λ k ) \hat{\phi}_h^k \leftarrow\operatorname{REPLEARN}(\mathcal{D}_h,\mathcal{D}'_h,\Phi_h,\lambda_k) ϕ ^ h k ← REPLEARN ( D h , D h ′ , Φ h , λ k ) ▷ Algorithm 2
end for
{ Q j , h k , V j , h k } j ∈ { r , g } , h ∈ [ H ] \{Q_{j,h}^k,V_{j,h}^k\}_{j\in\{r,g\},h\in[H]} { Q j , h k , V j , h k } j ∈ { r , g } , h ∈ [ H ] , { π h k } h = 1 H ← LSVI-UCB ( { ( s h τ , a ~ h τ , r h τ , g h τ , s h + 1 τ , ϕ ^ h τ ) } h ∈ [ H ] , τ ≤ k , Y k ) \{\pi_h^k\}_{h=1}^{H} \leftarrow\operatorname{LSVI\text{-}UCB}(\{(s_h^\tau,\tilde{a}_h^\tau,r_h^\tau,g_h^\tau,s_{h+1}^\tau,\hat{\phi}_h^\tau)\}_{h\in[H],\tau\leq k},Y_k) { π h k } h = 1 H ← LSVI - UCB ({( s h τ , a ~ h τ , r h τ , g h τ , s h + 1 τ , ϕ ^ h τ ) } h ∈ [ H ] , τ ≤ k , Y k ) .
for h = 1 h = 1 h = 1 to H H H do ▷ Execute π k \pi^k π k for next-episode trunk
Take a h k ∼ π h k ( ⋅ ∣ s h k ) a_h^k \sim\pi_h^k(\cdot\mid s_h^k) a h k ∼ π h k ( ⋅ ∣ s h k ) and observe s h + 1 k s_{h+1}^k s h + 1 k .
end for
Y k + 1 ← max { min { Y k + η ( b − V g , 1 k ( s 1 ) ) , ξ } , 0 } Y_{k+1} \leftarrow\max\{\min\{Y_k+\eta(b-V_{g,1}^k(s_1)),\xi\},0\} Y k + 1 ← max { min { Y k + η ( b − V g , 1 k ( s 1 )) , ξ } , 0 } ▷ Primal–dual update
end for
is exact: misspecified classes with ϕ ⋆ ∉ Φ \phi^\star\notin\Phi ϕ ⋆ ∈ / Φ are outside our guarantees. Requiring the utility, like the reward, to be linear in ϕ ⋆ \phi^\star ϕ ⋆ is the natural CMDP counterpart of the low-rank MDP assumption; it is structural and does not cover reward or utility functions unrelated to the low-rank dynamics. We do not assume any map from representations to policies recovers π ⋆ \pi^\star π ⋆ : the analysis compares directly against an arbitrary optimal feasible policy of (1) . Importantly, we adopt a model-free approach in which the policy is learned without explicitly modeling the environment, and there is no need to recover the exact values of μ ⋆ \mu^\star μ ⋆ or θ ⋆ \theta^\star θ ⋆ .
Representation Learning for Low-Rank CMDPs
In this section, we address the challenge of model-free representation learning within low-rank CMDPs. Our algorithm, MFRLC (Algorithm 1), builds on the Upper Confidence Bound (UCB) variant of Least-Squares Value Iteration (LSVI) and utilizes a primal–dual approach to effectively manage the CMDP’s optimization requirements.
Before delving into the details of the algorithm, it is helpful to introduce the composite (Lagrangian) value function, which encodes the reward–constraint trade-off into a single scalar objective.
Definition 2 (Composite value function; cf. [20 , 8 ] ). For any dual variable Y ≥ 0 Y \geq0 Y ≥ 0 and policy π \pi π , define the composite value function and action-value function as
V h π , Y ( ⋅ ) = V r , h π ( ⋅ ) + Y V g , h π ( ⋅ ) , Q h π , Y ( s , a ) = Q r , h π ( s , a ) + Y Q g , h π ( s , a ) . V_h^{\pi,Y}(\cdot) = V_{r,h}^{\pi}(\cdot) + YV_{g,h}^{\pi}(\cdot), \qquad Q_h^{\pi,Y}(s,a) = Q_{r,h}^{\pi}(s,a) + YQ_{g,h}^{\pi}(s,a). V h π , Y ( ⋅ ) = V r , h π ( ⋅ ) + Y V g , h π ( ⋅ ) , Q h π , Y ( s , a ) = Q r , h π ( s , a ) + Y Q g , h π ( s , a ) .
Equivalently, V h π , Y V_h^{\pi,Y} V h π , Y and Q h π , Y Q_h^{\pi,Y} Q h π , Y are the value and action-value functions of policy π \pi π in the auxiliary MDP whose stage reward is the Lagrangian r h + Y g h r_h + Yg_h r h + Y g h .
We operate on the composite values ( V h π , Y , Q h π , Y ) (V_h^{\pi,Y},Q_h^{\pi,Y}) ( V h π , Y , Q h π , Y ) throughout the analysis, with Y Y Y instantiated to the dual iterate Y k Y_k Y k at round k k k . Algorithm 3 adds the same per-objective bonus b ^ h \hat{b}_h b ^ h to both Q r , h k Q_{r,h}^k Q r , h k and Q g , h k Q_{g,h}^k Q g , h k , so the composite optimism bonus is b ^ h ( k ) : = ( 1 + Y k ) b ^ h \hat{b}_h^{(k)} := (1+Y_k)\hat{b}_h b ^ h ( k ) := ( 1 + Y k ) b ^ h .
Algorithm Overview
MFRLC alternates three steps in each round. Representation Learning (lines 4–6) samples under d h π k − 1 d_h^{\pi^{k-1}} d h π k − 1 with kernel P h ⋆ P_h^\star P h ⋆ , stores data in replay buffers D h , D h ′ \mathcal{D}_h,\mathcal{D}'_h D h , D h ′ , and learns ϕ ^ h k ∈ Φ h \hat{\phi}_h^k \in\Phi_h ϕ ^ h k ∈ Φ h via a minimax game against a discriminator v ∈ V h v \in\mathcal{V}_h v ∈ V h , controlled by the regularizer λ k \lambda_k λ k , iteration cap M M M , and stopping threshold ℓ k \ell_k ℓ k . Policy Update (line 8) takes ϕ ^ k \hat{\phi}^k ϕ ^ k and Y k Y_k Y k and runs LSVI-UCB (Algorithm 3, Appendix § 10 ) Algorithm 2 Representation Learning (REPLEARN)
Require: Feature class Φ h \Phi_h Φ h , datasets D h D_h D h , D h ′ D'_h D h ′ , discriminator class V h \mathcal{V}_h V h , regularization λ k \lambda_k λ k , iterations M M M , threshold ℓ k \ell_k ℓ k , confidence δ \delta δ , dimension d d d .
Output: Representation ϕ ^ h k \hat{\phi}_h^k ϕ ^ h k
Parameters: λ k = O ( d ln ∣ Φ ∣ k δ ) \lambda_k = O\left(d \ln\frac{|\Phi|^k}{\delta}\right) λ k = O ( d ln δ ∣Φ ∣ k ) , ℓ k = O ( d 2 ln ( ∣ Φ ∣ / δ ) k ) \ell_k = O\left(d^2 \sqrt{\frac{\ln(|\Phi|/\delta)}{k}}\right) ℓ k = O ( d 2 k l n ( ∣Φ∣/ δ ) )
Notation: L λ k , D ( ϕ , w , v ) \mathcal{L}_{\lambda_k,D}(\phi,w,v) L λ k , D ( ϕ , w , v ) is the empirical objective (6) over buffer D D D ; w w w (and w ~ \tilde{w} w ~ ) are predictor weights for feature ϕ \phi ϕ ; v v v (v m v^m v m ) is the discriminator (at step m m m ).
Initialize: ϕ 0 ∈ Φ h \phi^0 \in\Phi_h ϕ 0 ∈ Φ h arbitrarily
for m = 0 m = 0 m = 0 to M − 1 M - 1 M − 1 do
Compute worst-case discriminator with bias correction:
v m ← arg max v ∈ V h , ϕ ~ ∈ Φ h [ min w L λ k , D h ( ϕ m , w , v ) − min w ~ L λ k , D h ′ ( ϕ ~ , w ~ , v ) ] v^m \leftarrow\underset{v\in\mathcal{V}_h,\ \tilde{\phi}\in\Phi_h}{\arg\max}\left[\min_w \mathcal{L}_{\lambda_k,D_h}(\phi^m,w,v)-\min_{\tilde{w}}\mathcal{L}_{\lambda_k,D'_h}(\tilde{\phi},\tilde{w},v)\right] v m ← v ∈ V h , ϕ ~ ∈ Φ h arg max [ min w L λ k , D h ( ϕ m , w , v ) − min w ~ L λ k , D h ′ ( ϕ ~ , w ~ , v ) ]
if the maximization objective at v m v^m v m is at most ℓ k \ell_k ℓ k then
Break and Return ϕ m \phi^m ϕ m
end if
Update representation jointly against past discriminators:
ϕ m + 1 ← arg min ϕ ∈ Φ h , { w i } i = 0 m ∑ i = 0 m L λ k , D h ( ϕ , w i , v i ) \phi^{m+1}\leftarrow\underset{\phi\in\Phi_h,\{w_i\}_{i=0}^{m}}{\arg\min}\sum_{i=0}^{m}\mathcal{L}_{\lambda_k,D_h}(\phi,w_i,v^i) ϕ m + 1 ← ϕ ∈ Φ h , { w i } i = 0 m arg min ∑ i = 0 m L λ k , D h ( ϕ , w i , v i )
end for
to produce optimistic per-objective estimates { Q r , h k , Q g , h k , V r , h k , V g , h k } \{Q^k_{r,h},Q^k_{g,h},V^k_{r,h},V^k_{g,h}\} { Q r , h k , Q g , h k , V r , h k , V g , h k } and the softmax policy
π h k ( a ∣ ⋅ ) = softmax α ( Q r , h k ( ⋅ , a ) + Y k Q g , h k ( ⋅ , a ) ) , (2) \pi_h^k(a\mid\cdot)=\operatorname{softmax}_{\alpha}\left(Q^k_{r,h}(\cdot,a)+Y_kQ^k_{g,h}(\cdot,a)\right), \tag*{(2)} π h k ( a ∣ ⋅ ) = softmax α ( Q r , h k ( ⋅ , a ) + Y k Q g , h k ( ⋅ , a ) ) , (2)
where Y k Y_k Y k enforces the constraint while reward is maximized. Primal–Dual Update (line 12) updates Y k Y_k Y k using the optimistic value V g , 1 k ( s 1 ) V^k_{g,1}(s_1) V g , 1 k ( s 1 ) in place of V g , 1 π k ( s 1 ) V^{\pi_k}_{g,1}(s_1) V g , 1 π k ( s 1 ) , based on the observed constraint slack, and carries it into the next round.
Representation Learning In the context of low-rank MDPs, where ϕ ⋆ \phi^\star ϕ ⋆ is unknown, we must learn this representation. Inspired by Zhang et al. [29 ] , we define V h = F h ∪ G h \mathcal{V}_h=\mathcal{F}_h\cup\mathcal{G}_h V h = F h ∪ G h , where F h \mathcal{F}_h F h captures the model-mismatch between P ^ \hat{P} P ^ and P ⋆ P^\star P ⋆ under the previous-episode policy π k − 1 \pi^{k-1} π k − 1 (Lemma 2) and G h \mathcal{G}_h G h captures the composite Q Q Q -function (Lemma 3):
F h : = { s ↦ E a ∼ π h k − 1 ( ⋅ ∣ s ) [ ⟨ ϕ ( s , a ) , θ ⟩ − ⟨ ϕ ′ ( s , a ) , θ ′ ⟩ ] : ϕ , ϕ ′ ∈ Φ h + 1 , max ( ∥ θ ∥ ∞ , ∥ θ ′ ∥ ∞ ) ≤ 1 } , (3) \mathcal{F}_h:=\left\{s\mapsto\mathbb{E}_{a\sim\pi_h^{k-1}(\cdot\mid s)}\left[\left\langle\phi(s,a),\theta\right\rangle-\left\langle\phi'(s,a),\theta'\right\rangle\right]:\phi,\phi'\in\Phi_{h+1},\ \max(\lVert\theta\rVert_\infty,\lVert\theta'\rVert_\infty)\le1\right\}, \tag*{(3)} F h := { s ↦ E a ∼ π h k − 1 ( ⋅ ∣ s ) [ ⟨ ϕ ( s , a ) , θ ⟩ − ⟨ ϕ ′ ( s , a ) , θ ′ ⟩ ] : ϕ , ϕ ′ ∈ Φ h + 1 , max (∥ θ ∥ ∞ , ∥ θ ′ ∥ ∞ ) ≤ 1 } , (3)
G h : = { s ↦ max a ∈ A ( r h + 1 ( s , a ) + g h + 1 ( s , a ) + min { ⟨ ϕ ( s , a ) , w ⟩ , 2 } 2 H + 1 + ⟨ ϕ ( s , a ) , w ′ ⟩ ) } , (4) \mathcal{G}_h:=\left\{s\mapsto\max_{a\in\mathcal{A}}\left(\frac{r_{h+1}(s,a)+g_{h+1}(s,a)+\min\{\langle\phi(s,a),w\rangle,2\}}{2H+1}+\langle\phi(s,a),w'\rangle\right)\right\}, \tag*{(4)} G h := { s ↦ a ∈ A max ( 2 H + 1 r h + 1 ( s , a ) + g h + 1 ( s , a ) + min {⟨ ϕ ( s , a ) , w ⟩ , 2 } + ⟨ ϕ ( s , a ) , w ′ ⟩ ) } , (4)
where the set ranges over ϕ ∈ Φ h + 1 \phi\in\Phi_{h+1} ϕ ∈ Φ h + 1 , ∥ w ∥ ∞ ≤ c \lVert w\rVert_\infty\le c ∥ w ∥ ∞ ≤ c , ∥ w ′ ∥ ∞ ≤ 1 \lVert w'\rVert_\infty\le1 ∥ w ′ ∥ ∞ ≤ 1 . Reward and utility enter the representation objective through G h \mathcal{G}_h G h , which contains r h + 1 + g h + 1 r_{h+1}+g_{h+1} r h + 1 + g h + 1 ; our guarantee requires no separate supervised loss on the immediate reward or utility.
To learn ϕ ⋆ \phi^\star ϕ ⋆ , we solve the following optimization problem, where ρ h k \rho_h^k ρ h k denotes the sampling distribution induced by the replay buffer D h D_h D h at the start of episode k k k :
arg min ϕ h k max v ∈ V h min w E ( s , a ~ ) ∼ ρ h k [ ( ⟨ ϕ k ( s , a ~ ) , w ⟩ − E s ′ ∼ P ⋆ ( ⋅ ∣ s , a ~ ) [ v ( s ′ ) ] ) 2 ] . (5) \underset{\phi_h^k}{\arg\min}\ \underset{v\in\mathcal{V}_h}{\max}\ \underset{w}{\min}\ \mathbb{E}_{(s,\tilde{a})\sim\rho_h^k}\left[\left(\langle\phi^k(s,\tilde{a}),w\rangle-\mathbb{E}_{s'\sim P^\star(\cdot\mid s,\tilde{a})}[v(s')]\right)^2\right]. \tag*{(5)} ϕ h k arg min v ∈ V h max w min E ( s , a ~ ) ∼ ρ h k [ ( ⟨ ϕ k ( s , a ~ ) , w ⟩ − E s ′ ∼ P ⋆ ( ⋅ ∣ s , a ~ ) [ v ( s ′ )] ) 2 ] . (5)
This objective seeks a feature ϕ \phi ϕ that accurately approximates the Bellman backup E [ v ( s ′ ) ∣ s , a ~ ] \mathbb{E}[v(s')\mid s,\tilde{a}] E [ v ( s ′ ) ∣ s , a ~ ] as a linear function ⟨ ϕ ( s , a ~ ) , w ⟩ \langle\phi(s,\tilde{a}),w\rangle ⟨ ϕ ( s , a ~ ) , w ⟩ , uniformly over v ∈ V h v\in\mathcal{V}_h v ∈ V h . The min-max-min structure ensures robustness: the inner min w \min_w min w is a regularized least-squares problem with a closed-form solution (Appendix C, complexity O ( n d 2 + d 3 ) O(nd^2+d^3) O ( n d 2 + d 3 ) ); the middle max v \max_v max v identifies the worst-case v ∈ V h v\in\mathcal{V}_h v ∈ V h for the current ϕ \phi ϕ ; the outer min ϕ h k \min_{\phi_h^k} min ϕ h k chooses the representation minimizing this worst-case error [17 ] . The non-linear softmax operator and UCB bonus appearing in the LSVI-UCB output (Algorithm 3) are not contained in G h \mathcal{G}_h G h ; their contribution is handled separately via the softmax sub-optimality bound (Lemma 12) and the elliptical-potential argument applied to the bonus term (Appendix § 15 ).
Since (5) involves a population expectation, direct implementation is challenging. We replace it by its sample-based counterpart over D h \mathcal{D}_h D h :
L λ k , D h ( ϕ h k , v , w ) = E D h [ ( ⟨ ϕ h k ( s , a ~ ) , w ⟩ − v ( s ′ ) ) 2 ] + λ k ∥ w ∥ 2 2 . (6) \mathcal{L}_{\lambda_k,\mathcal{D}_h}(\phi_h^k,v,w)=\mathbb{E}_{\mathcal{D}_h}\left[\left(\langle\phi_h^k(s,\tilde{a}),w\rangle-v(s')\right)^2\right]+\lambda_k\lVert w\rVert_2^2. \tag*{(6)} L λ k , D h ( ϕ h k , v , w ) = E D h [ ( ⟨ ϕ h k ( s , a ~ ) , w ⟩ − v ( s ′ ) ) 2 ] + λ k ∥ w ∥ 2 2 . (6)
Because L λ k , D h \mathcal{L}_{\lambda_k,\mathcal{D}_h} L λ k , D h contains a conditional-variance term that depends on v v v , maximizing it directly would favor high-variance discriminators. Algorithm 2 cancels this term in expectation by subtracting the minimized loss L λ k , D h ′ ( ϕ , w ~ , v ~ ) \mathcal{L}_{\lambda_k,\mathcal{D}'_h}(\phi,\tilde{w},\tilde{v}) L λ k , D h ′ ( ϕ , w ~ , v ~ ) on the independent buffer D h ′ \mathcal{D}'_h D h ′ (Appendix § 12 ).
From theory to implementation We distinguish three levels of the representation-learning step:
objective (5) ⟶ \longrightarrow ⟶ bias-corrected REPLEARN (Algorithm 2) ⟶ \longrightarrow ⟶ contrastive EMA surrogate.
Theorem 3 covers the first two levels. The experiments in Section 5 use the third: the minimax search over V h \mathcal{V}_h V h is replaced by contrastive next-state prediction with an exponential-moving-average (EMA) target encoder. The surrogate uses the same transition triples ( s , a ~ , s ′ ) (s,\tilde{a},s') ( s , a ~ , s ′ ) to learn predictive features, but it does not perform the worst-case discriminator search and does not certify uniform Bellman prediction over V h \mathcal{V}_h V h , so it is not covered by Theorem 3.
Computation The inner minimization over w w w is ridge regression with cost O ( n d 2 + d 3 ) O(nd^2+d^3) O ( n d 2 + d 3 ) ; the outer problems over ϕ \phi ϕ and v v v are treated as optimization oracles, exactly solvable for finite classes but without a polynomial-time guarantee for neural classes (Appendix § 12 .5).
Policy Update Given ϕ ^ k \hat{\phi}^k ϕ ^ k and Y k Y_k Y k , the LSVI-UCB subroutine (Algorithm 3, Appendix § 10 ) computes optimistic estimates Q j , h k , V j , h k Q^k_{j,h},V^k_{j,h} Q j , h k , V j , h k for j ∈ { r , g } j\in\{r,g\} j ∈ { r , g } and the softmax policy (2) . The executed policy π k \pi^k π k is defined directly as this optimistic softmax policy computed from ϕ ^ k \hat{\phi}^k ϕ ^ k and Y k Y_k Y k . The composite Q r , h k + Y k Q g , h k Q^k_{r,h}+Y_kQ^k_{g,h} Q r , h k + Y k Q g , h k is the LSVI-UCB optimistic estimate of the Bellman-optimal Q Q Q for the auxiliary MDP with stage reward r h + Y k g h r_h+Y_kg_h r h + Y k g h (cf. Definition 2), with Y k Y_k Y k updated online in the next step.
Dual Update Problem (1) admits the Lagrangian formulation:
max π ∈ Δ ( A ∣ S , H ) min Y ∈ [ 0 , ξ ] V r π ( s 1 ) + Y ( V g π ( s 1 ) − b ) , (7) \max_{\pi\in\Delta(\mathcal{A}\mid\mathcal{S},H)}\min_{Y\in[0,\xi]}\;V_r^\pi(s_1)+Y\left(V_g^\pi(s_1)-b\right), \tag*{(7)} π ∈ Δ ( A ∣ S , H ) max Y ∈ [ 0 , ξ ] min V r π ( s 1 ) + Y ( V g π ( s 1 ) − b ) , (7)
where ξ = 2 H / γ \xi=2H/\gamma ξ = 2 H / γ is chosen so that Y ⋆ ∈ [ 0 , ξ ] Y^\star\in[0,\xi] Y ⋆ ∈ [ 0 , ξ ] (Lemma 1); restricting Y Y Y to [ 0 , ξ ] [0,\xi] [ 0 , ξ ] does not affect the saddle-point value. MFRLC does not solve (7) over π \pi π directly. In round k k k , the primal step is the optimistic LSVI-UCB policy π k \pi^k π k for the Lagrangian reward r h + Y k g h r_h+Y_kg_h r h + Y k g h , and the dual step is the projected update:
Y k + 1 = max { min { Y k + η ( b − V g , 1 k ( s 1 ) ) , ξ } , 0 } . (8) Y_{k+1}=\max\left\{\min\left\{Y_k+\eta\left(b-V^k_{g,1}(s_1)\right),\xi\right\},0\right\}. \tag*{(8)} Y k + 1 = max { min { Y k + η ( b − V g , 1 k ( s 1 ) ) , ξ } , 0 } . (8)
which uses the optimistic estimate V g , 1 k ( s 1 ) V^k_{g,1}(s_1) V g , 1 k ( s 1 ) in place of V g , 1 π k ( s 1 ) V^{\pi^k}_{g,1}(s_1) V g , 1 π k ( s 1 ) . The update direction switches with the sign of b − V g , 1 k ( s 1 ) b-V^k_{g,1}(s_1) b − V g , 1 k ( s 1 ) , balancing reward maximization against feasibility [10 , 18 ] .
Main Results
Assumption 2 (Slater’s Condition). There exists γ > 0 \gamma>0 γ > 0 and a policy π ˉ ∈ Δ ( A ∣ S , H ) \bar{\pi}\in\Delta(\mathcal{A}\mid\mathcal{S},H) π ˉ ∈ Δ ( A ∣ S , H ) such that V g , 1 π ˉ ( s 1 ) ≥ b + γ V^{\bar{\pi}}_{g,1}(s_1)\ge b+\gamma V g , 1 π ˉ ( s 1 ) ≥ b + γ .
Lemma 1 (Boundedness of Y ⋆ Y^\star Y ⋆ ). The optimal dual variable Y ⋆ Y^\star Y ⋆ associated with (1) satisfies Y ⋆ ≤ ( V r , 1 π ⋆ ( s 1 ) − V r , 1 π ˉ ( s 1 ) ) / γ ≤ H / γ Y^\star\le(V^{\pi^\star}_{r,1}(s_1)-V^{\bar{\pi}}_{r,1}(s_1))/\gamma\le H/\gamma Y ⋆ ≤ ( V r , 1 π ⋆ ( s 1 ) − V r , 1 π ˉ ( s 1 )) / γ ≤ H / γ , so ξ = 2 H / γ ≥ 2 Y ⋆ \xi=2H/\gamma\ge2Y^\star ξ = 2 H / γ ≥ 2 Y ⋆ as required by Lemma 2.
The Slater condition is mild and widely adopted [20 , 8 , 10 ] .
Theorem 3. Fix p ∈ ( 0 , 1 ) p\in(0,1) p ∈ ( 0 , 1 ) . Under Assumptions 1–2, with probability at least 1 − p 1-p 1 − p , the regret and constraint violation of MFRLC over K K K deployment rounds both satisfy:
Regret ( K ) , Violation ( K ) ≤ C γ H 3 d 2 K 3 / 4 ∣ A ∣ 3 / 2 log 1 / 4 ( d K ∣ Φ ∣ p ) , \operatorname{Regret}(K),\operatorname{Violation}(K)\le\frac{C}{\gamma}H^3d^2K^{3/4}\lvert\mathcal{A}\rvert^{3/2}\log^{1/4}\left(\frac{dK\lvert\Phi\rvert}{p}\right), Regret ( K ) , Violation ( K ) ≤ γ C H 3 d 2 K 3/4 ∣ A ∣ 3/2 log 1/4 ( p d K ∣ Φ ∣ ) ,
for an absolute constant C > 0 C>0 C > 0 . Over the K K K rounds, MFRLC uses K ( 2 H + 1 ) K(2H+1) K ( 2 H + 1 ) trajectories in total, including the auxiliary roll-ins of representation learning, whose rewards and utilities are not charged to Regret or Violation (Section 2). The dependence on 1 / γ 1/\gamma 1/ γ is the standard Slater-gap penalty of primal–dual CMDP analyses [7 , 8 , 11 ] . The bound is independent of ∣ S ∣ |S| ∣ S ∣ and applies to continuous state spaces. The K 3 / 4 K^{3/4} K 3/4 term arises only from the representation-learning error (Remark 8 in Appendix § 17 ). Theorem 3 is an upper bound: we prove no matching lower bound, and the optimal dependence on K K K , d d d , H H H , and ∣ A ∣ |\mathcal{A}| ∣ A ∣ remains open.
Proof Sketch
Step 1: Decomposition of regret and violation. For any Y ∈ [ 0 , ξ ] Y \in[0, \xi] Y ∈ [ 0 , ξ ] , by combining a Lagrangian shift with the projected dual update of Algorithm 1 (Appendix § 14 ), we obtain:
∑ k = 1 K ( V r , 1 π ⋆ ( s 1 ) − V r , 1 π k ( s 1 ) ) + Y ∑ k = 1 K ( b − V g , 1 π k ( s 1 ) ) ≤ Y 2 2 η + η H 2 K 2 + ∑ k = 1 K [ ( V r , 1 π ⋆ ( s 1 ) + Y k V g , 1 π ⋆ ( s 1 ) ) − ( V r , 1 π k ( s 1 ) + Y k V g , 1 π k ( s 1 ) ) ] + ∑ k = 1 K [ ( V r , 1 π k ( s 1 ) − V r , 1 π k ( s 1 ) ) + Y ( V g , 1 π k ( s 1 ) − V g , 1 π k ( s 1 ) ) ] . (9) \begin{aligned}
\sum_{k=1}^{K}\left(V_{r,1}^{\pi^\star}(s_1)-V_{r,1}^{\pi^k}(s_1)\right)+Y\sum_{k=1}^{K}\left(b-V_{g,1}^{\pi^k}(s_1)\right)\le\frac{Y^2}{2\eta}+\frac{\eta H^2K}{2} \\
&+\sum_{k=1}^{K}\left[\left(V_{r,1}^{\pi^\star}(s_1)+Y_kV_{g,1}^{\pi^\star}(s_1)\right)-\left(V_{r,1}^{\pi^k}(s_1)+Y_kV_{g,1}^{\pi^k}(s_1)\right)\right] \\
&+\sum_{k=1}^{K}\left[\left(V_{r,1}^{\pi^k}(s_1)-V_{r,1}^{\pi^k}(s_1)\right)+Y\left(V_{g,1}^{\pi^k}(s_1)-V_{g,1}^{\pi^k}(s_1)\right)\right]. \tag*{(9)}
\end{aligned} k = 1 ∑ K ( V r , 1 π ⋆ ( s 1 ) − V r , 1 π k ( s 1 ) ) + Y k = 1 ∑ K ( b − V g , 1 π k ( s 1 ) ) ≤ 2 η Y 2 + 2 η H 2 K + k = 1 ∑ K [ ( V r , 1 π ⋆ ( s 1 ) + Y k V g , 1 π ⋆ ( s 1 ) ) − ( V r , 1 π k ( s 1 ) + Y k V g , 1 π k ( s 1 ) ) ] + k = 1 ∑ K [ ( V r , 1 π k ( s 1 ) − V r , 1 π k ( s 1 ) ) + Y ( V g , 1 π k ( s 1 ) − V g , 1 π k ( s 1 ) ) ] . (9)
Here Y Y Y is a free variable that combines reward regret and constraint violation: setting Y = 0 Y=0 Y = 0 recovers the regret bound, while Y = ξ Y=\xi Y = ξ (combined with strong duality) yields the violation bound. By feasibility of π ⋆ \pi^\star π ⋆ , the Lagrangian-shift slack term ∑ k Y k ( b − V g , 1 π ⋆ ( s 1 ) ) ≤ 0 \sum_k Y_k(b-V_{g,1}^{\pi^\star}(s_1))\le0 ∑ k Y k ( b − V g , 1 π ⋆ ( s 1 )) ≤ 0 is non-positive and dropped. The Pseudo regret captures the gap between π ⋆ \pi^\star π ⋆ and the LSVI-UCB optimistic estimate V k V^k V k , while the Estimation error captures the gap between V k V^k V k and the executed policy’s value V π k V^{\pi^k} V π k . We control the two terms separately.
Step 2: Bounding the Pseudo regret. Since π k \pi^k π k is the softmax of the composite Q r , h k + Y k Q g , h k Q^k_{r,h}+Y_kQ^k_{g,h} Q r , h k + Y k Q g , h k , the analysis must be carried out on the auxiliary MDP with stage reward r ˉ h ( k ) \bar r_h^{(k)} r ˉ h ( k ) . We first establish composite optimism on the planning model P ^ k \widehat P^k P k induced by ϕ ^ k \widehat\phi_k ϕ k .
Lemma 2 (Composite optimism). Conditioned on the Representation Learning guarantee (Lemma 16 ), with probability 1 − δ 1-\delta 1 − δ , for every round k ∈ [ K ] k\in[K] k ∈ [ K ] ,
Q P ^ k , r ˉ ( k ) + b ^ ( k ) π ⋆ , Y k ( s 1 , ⋅ ) − Q P ⋆ , r ˉ ( k ) π ⋆ , Y k ( s 1 , ⋅ ) ≥ − ( 1 + Y k ) ∣ A ∣ ζ k , Q_{\widehat P^k,\bar r^{(k)}+\widehat b^{(k)}}^{\pi^\star,Y_k}(s_1,\cdot)-Q_{P^\star,\bar r^{(k)}}^{\pi^\star,Y_k}(s_1,\cdot)\ge-(1+Y_k)\sqrt{|\mathcal{A}|}\zeta_k, Q P k , r ˉ ( k ) + b ( k ) π ⋆ , Y k ( s 1 , ⋅ ) − Q P ⋆ , r ˉ ( k ) π ⋆ , Y k ( s 1 , ⋅ ) ≥ − ( 1 + Y k ) ∣ A ∣ ζ k , where ζ k = O ~ ( d 2 / k ) \zeta_k=\widetilde O(d^2/\sqrt{k}) ζ k = O ( d 2 / k ) is the RepLearn statistical error.
Lemma 3 (Composite policy gap, cumulative form). Conditioned on the Representation Learning guarantee, with probability 1 − δ 1-\delta 1 − δ ,
∑ k = 1 K [ Q P ⋆ , r ˉ ⋆ , Y k π ⋆ , Y k ( s 1 , ⋅ ) − Q P ⋆ , r ˉ ( k ) π k , Y k ( s 1 , ⋅ ) ] ≤ K H log ∣ A ∣ α + ( 1 + ξ ) ∑ k = 1 K ϵ k , \sum_{k=1}^{K}\left[Q_{P^\star,\bar r^{\star,Y_k}}^{\pi^\star,Y_k}(s_1,\cdot)-Q_{P^\star,\bar r^{(k)}}^{\pi^k,Y_k}(s_1,\cdot)\right]\le\frac{KH\log|\mathcal{A}|}{\alpha}+(1+\xi)\sum_{k=1}^{K}\epsilon_k, k = 1 ∑ K [ Q P ⋆ , r ˉ ⋆ , Y k π ⋆ , Y k ( s 1 , ⋅ ) − Q P ⋆ , r ˉ ( k ) π k , Y k ( s 1 , ⋅ ) ] ≤ α K H log ∣ A ∣ + ( 1 + ξ ) k = 1 ∑ K ϵ k , where the cumulative representation error satisfies ∑ k = 1 K ϵ k ≤ C H 2 d 2 K 3 / 4 ∣ A ∣ 3 / 2 log 1 / 4 ( d K ∣ Φ ∣ / δ ) \sum_{k=1}^{K}\epsilon_k\le CH^2d^2K^{3/4}|\mathcal{A}|^{3/2}\log^{1/4}(dK|\Phi|/\delta) ∑ k = 1 K ϵ k ≤ C H 2 d 2 K 3/4 ∣ A ∣ 3/2 log 1/4 ( d K ∣Φ∣/ δ ) .
Remark 1 (On the per-episode rate). The cumulative bound on ∑ k ϵ k \sum_k\epsilon_k ∑ k ϵ k corresponds to an average per-episode rate ϵ ˉ K = 1 K ∑ k ϵ k = O ~ ( K − 1 / 4 ) \bar\epsilon_K=\frac{1}{K}\sum_k\epsilon_k=\widetilde O(K^{-1/4}) ϵ ˉ K = K 1 ∑ k ϵ k = O ( K − 1/4 ) . We state the cumulative form because the elliptical-potential argument in Appendix § 15 directly yields a sum, and only this sum is needed downstream.
The proof of Lemma 3 combines composite optimism (Lemma 2), a softmax sub-optimality bound for the composite planning problem, and the simulation lemma. The ( 1 + Y k ) (1+Y_k) ( 1 + Y k ) factor arises because both r ˉ h ( k ) ∈ [ 0 , 1 + Y k ] \bar r_h^{(k)}\in[0,1+Y_k] r ˉ h ( k ) ∈ [ 0 , 1 + Y k ] and b ^ h ( k ) = ( 1 + Y k ) b ^ h \widehat b_h^{(k)}=(1+Y_k)\widehat b_h b h ( k ) = ( 1 + Y k ) b h scale linearly in 1 + Y k 1+Y_k 1 + Y k ; using Y k ≤ ξ Y_k\le\xi Y k ≤ ξ , this factor is bounded by 1 + ξ 1+\xi 1 + ξ , contributing the 1 / γ 1/\gamma 1/ γ dependence.
Combining Lemma 3 with the V–Q identity V j , 1 k ( s 1 ) = ∑ a π 1 k ( a ∣ s 1 ) Q j , 1 k ( s 1 , a ) V_{j,1}^{k}(s_1)=\sum_a\pi_1^k(a\mid s_1)Q_{j,1}^k(s_1,a) V j , 1 k ( s 1 ) = ∑ a π 1 k ( a ∣ s 1 ) Q j , 1 k ( s 1 , a ) and a performance-difference argument on P ⋆ P^\star P ⋆ (Appendix § 15 ):
Lemma 4 (Pseudo regret). Conditioned on the Representation Learning guarantee, with probability 1 − p / 4 1-p/4 1 − p /4 ,
∑ k = 1 K [ ( V r , 1 π ⋆ ( s 1 ) + Y k V g , 1 π ⋆ ( s 1 ) ) − ( V r , 1 k ( s 1 ) + Y k V g , 1 k ( s 1 ) ) ] ≤ K H log ∣ A ∣ α + ( 1 + ξ ) ∑ k = 1 K ϵ k . \sum_{k=1}^{K}\left[\left(V_{r,1}^{\pi^\star}(s_1)+Y_kV_{g,1}^{\pi^\star}(s_1)\right)-\left(V_{r,1}^{k}(s_1)+Y_kV_{g,1}^{k}(s_1)\right)\right]\leq\frac{KH\log|\mathcal{A}|}{\alpha}+(1+\xi)\sum_{k=1}^{K}\epsilon_k. k = 1 ∑ K [ ( V r , 1 π ⋆ ( s 1 ) + Y k V g , 1 π ⋆ ( s 1 ) ) − ( V r , 1 k ( s 1 ) + Y k V g , 1 k ( s 1 ) ) ] ≤ α K H log ∣ A ∣ + ( 1 + ξ ) k = 1 ∑ K ϵ k .
Step 3: Bounding the Estimation error. The estimation error compares the LSVI-UCB optimistic estimate V j , 1 k V_{j,1}^{k} V j , 1 k to the true value V j , 1 π k V_{j,1}^{\pi^k} V j , 1 π k of the executed policy. The starting point is a self-normalized concentration bound for the empirical Bellman residual.
Lemma 5 (Concentration of empirical Bellman residual). Adapting the self-normalized concentration of Ghosh et al. [11 ] to the learned-feature setting via a covering argument over the optimistic value-function class, there exists a constant C 2 C_2 C 2 such that for any p ∈ ( 0 , 1 ) p\in(0,1) p ∈ ( 0 , 1 ) , the event:
E : = { ∥ ∑ τ = 1 k − 1 ϕ ^ h τ ( V j , h + 1 k ( s h + 1 τ ) − P h V j , h + 1 k ( s h τ , a ~ h τ ) ) ∥ ( Σ h k ) − 1 ≤ C 2 d H χ } k ∈ [ K ] , h ∈ [ H ] , j ∈ { r , g } \mathcal{E}:=\left\{\left\|\sum_{\tau=1}^{k-1}\widehat{\phi}_{h}^{\tau}\left(V_{j,h+1}^{k}(s_{h+1}^{\tau})-\mathbb{P}_{h}V_{j,h+1}^{k}(s_{h}^{\tau},\widetilde{a}_{h}^{\tau})\right)\right\|_{(\Sigma_{h}^{k})^{-1}}\leq C_2dH\sqrt{\chi}\right\}_{k\in[K],\,h\in[H],\,j\in\{r,g\}} E := ⎩ ⎨ ⎧ τ = 1 ∑ k − 1 ϕ h τ ( V j , h + 1 k ( s h + 1 τ ) − P h V j , h + 1 k ( s h τ , a h τ ) ) ( Σ h k ) − 1 ≤ C 2 d H χ ⎭ ⎬ ⎫ k ∈ [ K ] , h ∈ [ H ] , j ∈ { r , g } holds with χ = log ( 4 ( C 1 + 1 ) log ∣ A ∣ d T ∣ Φ ∣ / p ) \chi=\log(4(C_1+1)\log|\mathcal{A}|dT|\Phi|/p) χ = log ( 4 ( C 1 + 1 ) log ∣ A ∣ d T ∣Φ∣/ p ) , and Pr ( E ) ≥ 1 − p / 4 \Pr(\mathcal{E})\geq1-p/4 Pr ( E ) ≥ 1 − p /4 .
We then control the per-step gap between Q j , h k Q_{j,h}^{k} Q j , h k (the LSVI-UCB optimistic estimate) and Q j , h π k Q_{j,h}^{\pi^k} Q j , h π k (the true action-value of the executed policy). The key technical step is showing that both quantities admit linear approximations in the learned features ϕ ^ k \widehat{\phi}^{k} ϕ k : Q j , h k Q_{j,h}^{k} Q j , h k by construction, since the LSVI-UCB output is explicitly linear in ϕ ^ k \widehat{\phi}^{k} ϕ k ; and Q j , h π k Q_{j,h}^{\pi^k} Q j , h π k via the RepLearn guarantee, which ensures ϕ ^ k \widehat{\phi}^{k} ϕ k approximately spans the Bellman backups of any function in V h \mathcal{V}_{h} V h .
Lemma 6 (Per-step backup error). Set β = C 1 d H ι \beta=C_1dH\sqrt{\iota} β = C 1 d H ι with ι = log ( log ∣ A ∣ ⋅ 4 d T ∣ Φ ∣ / p ) \iota=\log(\log|\mathcal{A}|\cdot4dT|\Phi|/p) ι = log ( log ∣ A ∣ ⋅ 4 d T ∣Φ∣/ p ) . On the event E \mathcal{E} E , for every round k ∈ [ K ] k\in[K] k ∈ [ K ] , step h ∈ [ H ] h\in[H] h ∈ [ H ] , j ∈ { r , g } j\in\{r,g\} j ∈ { r , g } , and ( s , a ) ∈ S × A (s,a)\in\mathcal{S}\times\mathcal{A} ( s , a ) ∈ S × A , there exist weights w j , h π k w_{j,h}^{\pi^k} w j , h π k approximately projecting Q j , h π k Q_{j,h}^{\pi^k} Q j , h π k onto span ( ϕ ^ h k ) \operatorname{span}(\widehat{\phi}_{h}^{k}) span ( ϕ h k ) such that:
Q j , h k ( s , a ) − Q j , h π k ( s , a ) = E s ′ ∼ P h ⋆ ( ⋅ ∣ s , a ) [ V j , h + 1 k ( s ′ ) − V j , h + 1 π k ( s ′ ) ] + Δ h k ( s , a ) , Q_{j,h}^{k}(s,a)-Q_{j,h}^{\pi^k}(s,a)=\mathbb{E}_{s'\sim P_{h}^{\star}(\cdot\mid s,a)}\left[V_{j,h+1}^{k}(s')-V_{j,h+1}^{\pi^k}(s')\right]+\Delta_{h}^{k}(s,a), Q j , h k ( s , a ) − Q j , h π k ( s , a ) = E s ′ ∼ P h ⋆ ( ⋅ ∣ s , a ) [ V j , h + 1 k ( s ′ ) − V j , h + 1 π k ( s ′ ) ] + Δ h k ( s , a ) , where ∣ Δ h k ( s , a ) ∣ ≤ β ϕ ^ h k ( s , a ) T ( Σ h k ) − 1 ϕ ^ h k ( s , a ) |\Delta_{h}^{k}(s,a)|\leq\beta\sqrt{\widehat{\phi}_{h}^{k}(s,a)^{\mathsf{T}}(\Sigma_{h}^{k})^{-1}\widehat{\phi}_{h}^{k}(s,a)} ∣ Δ h k ( s , a ) ∣ ≤ β ϕ h k ( s , a ) T ( Σ h k ) − 1 ϕ h k ( s , a ) plus a RepLearn projection residual that accumulates into ∑ k ϵ k \sum_{k}\epsilon_k ∑ k ϵ k .
Unrolling Lemma 6 via V j , h k ( s ) = ∑ a π h k ( a ∣ s ) Q j , h k ( s , a ) V_{j,h}^{k}(s)=\sum_{a}\pi_{h}^{k}(a\mid s)Q_{j,h}^{k}(s,a) V j , h k ( s ) = ∑ a π h k ( a ∣ s ) Q j , h k ( s , a ) , summing telescopically over h h h , and applying the elliptical potential lemma over k k k , we obtain:
Lemma 7 (Estimation error). With probability at least 1 − p / 2 1-p/2 1 − p /2 ,
∑ k = 1 K ( V r , 1 k ( s 1 ) − V r , 1 π k ( s 1 ) ) + Y ∑ k = 1 K ( V g , 1 k ( s 1 ) − V g , 1 π k ( s 1 ) ) ≤ O ( ( Y + 1 ) d 3 H 3 T ι 2 ) + ( 1 + ξ ) ∑ k = 1 K ϵ k , \sum_{k=1}^{K}\left(V_{r,1}^{k}(s_1)-V_{r,1}^{\pi^k}(s_1)\right)+Y\sum_{k=1}^{K}\left(V_{g,1}^{k}(s_1)-V_{g,1}^{\pi^k}(s_1)\right)\leq O\left((Y+1)\sqrt{d^{3}H^{3}T\iota^{2}}\right)+(1+\xi)\sum_{k=1}^{K}\epsilon_k, k = 1 ∑ K ( V r , 1 k ( s 1 ) − V r , 1 π k ( s 1 ) ) + Y k = 1 ∑ K ( V g , 1 k ( s 1 ) − V g , 1 π k ( s 1 ) ) ≤ O ( ( Y + 1 ) d 3 H 3 T ι 2 ) + ( 1 + ξ ) k = 1 ∑ K ϵ k , where T = K H T=KH T = K H .
Step 4: Combining Pseudo regret and Estimation error. Plugging Lemmas 4 and 7 into (9) with η = ξ / K H 2 \eta=\xi/\sqrt{KH^{2}} η = ξ / K H 2 and α = K log ∣ A ∣ 2 ( 1 + ξ + H ) \alpha=\frac{K\log|\mathcal{A}|}{2(1+\xi+H)} α = 2 ( 1 + ξ + H ) K l o g ∣ A ∣ :
∑ k = 1 K ( V r , 1 π ⋆ ( s 1 ) − V r , 1 π k ( s 1 ) ) + Y ∑ k = 1 K ( b − V g , 1 π k ( s 1 ) ) ≤ Y 2 2 η + η H 2 K 2 + K H log ∣ A ∣ α + 2 ( 1 + ξ ) ∑ k = 1 K ϵ k + O ( ( Y + 1 ) d 3 H 3 T ι 2 ) . \sum_{k=1}^{K}\left(V_{r,1}^{\pi^\star}(s_1)-V_{r,1}^{\pi^k}(s_1)\right)+Y\sum_{k=1}^{K}\left(b-V_{g,1}^{\pi^k}(s_1)\right)\leq\frac{Y^{2}}{2\eta}+\frac{\eta H^{2}K}{2}+\frac{KH\log|\mathcal{A}|}{\alpha}+2(1+\xi)\sum_{k=1}^{K}\epsilon_k+O\left((Y+1)\sqrt{d^{3}H^{3}T\iota^{2}}\right). k = 1 ∑ K ( V r , 1 π ⋆ ( s 1 ) − V r , 1 π k ( s 1 ) ) + Y k = 1 ∑ K ( b − V g , 1 π k ( s 1 ) ) ≤ 2 η Y 2 + 2 η H 2 K + α K H log ∣ A ∣ + 2 ( 1 + ξ ) k = 1 ∑ K ϵ k + O ( ( Y + 1 ) d 3 H 3 T ι 2 ) .
For these parameter choices, η H 2 K / 2 \eta H^2K/2 η H 2 K /2 , K H log ∣ A ∣ / α KH\log|\mathcal{A}|/\alpha K H log ∣ A ∣/ α , and the estimation-error term are all dominated by 2 ( 1 + ξ ) ∑ k ϵ k 2(1+\xi)\sum_k\epsilon_k 2 ( 1 + ξ ) ∑ k ϵ k , which evaluates to O ~ ( H 3 d 2 K 3 / 4 ∣ A ∣ 3 / 2 γ ) \widetilde{O}\left(\frac{H^3d^2K^{3/4}|\mathcal{A}|^{3/2}}{\gamma}\right) O ( γ H 3 d 2 K 3/4 ∣ A ∣ 3/2 ) . Setting Y = 0 Y=0 Y = 0 yields the regret bound. Setting Y = ξ Y=\xi Y = ξ and applying Lemma 21 (strong-duality conversion of Ding et al. [8 ] ) with ξ ≥ 2 Y ∗ \xi\geq2Y^* ξ ≥ 2 Y ∗ converts this into the cumulative violation bound, matching Theorem 3.
Experiments
We introduce the Constrained Latent ComboLock (CLCL) , a new benchmark we designed to test agents on difficult, long-horizon visual safety problems (Figure 1 ). Our experiments on CLCL validate the strengths of our proposed method MFRLC.
Figure 1. Constrained Latent ComboLock
Environment Setup Experiments use the CLCL (Figure 1 ), a CMDP with horizon H = 6 H=6 H = 6 , high-dimensional observations, and a latent state space. The latent states form a chain s 0 → s 1 → ⋯ → s H s_0\to s_1\to\cdots\to s_H s 0 → s 1 → ⋯ → s H with two absorbing terminals: a S S a f e S_{\mathrm{Safe}} S Safe state and a S F a i l S_{\mathrm{Fail}} S Fail state. The agent must learn a sequence of actions (the “combo”) to reach the Safe state, but never observes the latent state s h s_h s h ; it receives only an observation o h ∈ R 8 × 84 × 84 o_h\in\mathbb{R}^{8\times84\times84} o h ∈ R 8 × 84 × 84 from a rendering function, mapped for learning into a feature class Φ \Phi Φ of dimension d = 64 d=64 d = 64 . Observations visually encode progress and timestep, with added noise to increase perceptual difficulty. At each state s h s_h s h , the agent chooses from the action set A = { a f o r w a r d , a h a z a r d , a s t a y } \mathcal{A}=\{a_{\mathrm{forward}},a_{\mathrm{hazard}},a_{\mathrm{stay}}\} A = { a forward , a hazard , a stay } . Taking a f o r w a r d a_{\mathrm{forward}} a forward transitions the agent from state s h s_h s h to s h + 1 s_{h+1} s h + 1 . Taking a s t a y a_{\mathrm{stay}} a stay causes the agent to remain in s h s_h s h without cost or reward. Taking a h a z a r d a_{\mathrm{hazard}} a hazard incurs a cost of 1 and causes transition to S F a i l S_{\mathrm{Fail}} S Fail with probability p h a z a r d p_{\mathrm{hazard}} p hazard , or return to s h − 1 s_{h-1} s h − 1 with probability 1 − p h a z a r d 1-p_{\mathrm{hazard}} 1 − p hazard . A reward of 1 is given only upon successfully transitioning from s H − 1 s_{H-1} s H − 1 to S S a f e S_{\mathrm{Safe}} S Safe ; all other rewards are zero. The objective is to maximize expected reward while ensuring the total expected cost remains below 1. Full setup details and hyperparameters are in Appendix § 13 .
Baselines We compare our proposed method against a suite of reinforcement learning algorithms. Our baselines include LAGRANGIAN POLICY GRADIENT (LAG-PG) [22 ] , a standard on-policy actor-critic with GAE for reward and cost; CONSTRAINED POLICY OPTIMIZATION (CPO) [2 ] , an on-policy trust-region method for CMDPs; HOMER [16 ] , a hierarchical option-based algorithm; and BLOCK-STRUCTURED REPRESENTATION LEARNING WITH INTERLEAVED EXPLORATIVE EXPLORATION (BRIEE) [29 ] , which focuses on representation learning coupled with a policy optimization. Since HOMER and BRIEE were originally designed for unconstrained MDPs, we add a Lagrangian framework for fair comparison; details are in Appendix § 13 .7.
Results MFRLC achieves the best overall performance, balancing reward maximization with constraint satisfaction. In the CLCL task with horizon H = 6 H = 6 H = 6 (Figure 2 ), MFRLC is the only algorithm to achieve an optimal reward approaching 1.0 while satisfying the safety constraint. HOMER satisfies the constraint sooner but converges to a suboptimal reward of ≈ 0.8 \approx0.8 ≈ 0.8 ; BRIEE plateaus at ≈ 0.4 \approx0.4 ≈ 0.4 and violates the constraint; CPO and LAG-PG fail to achieve meaningful reward and consistently violate safety. Representation learning alone is thus insufficient: only MFRLC, by jointly learning features and a constraint-aware policy, succeeds on both objectives (details in Appendix § 13 ).
Figure 2. Reward and Cost vs. iteration on CLCL (H=6)
Additional Settings Table 2 adds a larger action space and a longer horizon (Appendix § 13 .6). With ∣ A ∣ = 5 |\mathcal{A}| = 5 ∣ A ∣ = 5 , where discovering the rewarding sequence under uniform exploration is ( 5 / 3 ) 6 ≈ 21.4 (5/3)^6 \approx21.4 ( 5/3 ) 6 ≈ 21.4 times less likely, MFRLC attains the highest reward and lowest violation. At H = 8 H = 8 H = 8 , MFRLC obtains the highest reward; only CPO has lower violation, at near-zero reward. Its estimated cumulative regret is 15,382 ± \pm ± 942 over 15,552 deployed episodes; we do not read these finite-run numbers as verifying the K 3 / 4 K^{3/4} K 3/4 rate. At H = 12 H = 12 H = 12 , MFRLC reaches only 0.021 reward, a failure mode under our budget.
Table 2. Additional CLCL settings (mean ± \pm ± s.e., 5 seeds). Best R R R : maximum reward per run; Viol.: cumulative violation. Compare within each setting.
Limitations
Our results have four main limitations. (i) Theorem 3 is an upper bound; we prove no lower bound and make no optimality claim about the K 3 / 4 K^{3/4} K 3/4 rate or the dependence on d d d , H H H , ∣ A ∣ |\mathcal{A}| ∣ A ∣ , and γ \gamma γ . (ii) The outer minimax problem of Algorithm 2 is solved by an optimization oracle; exact search is possible for finite classes, but we give no polynomial-time guarantee for neural classes. (iii) Regret and Violation are charged only to deployed policies; the auxiliary roll-ins of representation learning are counted in the K ( 2 H + 1 ) K(2H + 1) K ( 2 H + 1 ) trajectory complexity but not in the regret or violation sums. (iv) Our experiments use a contrastive EMA surrogate that is not covered by Theorem 3, the evaluation is restricted to CLCL, and MFRLC fails to obtain meaningful reward at H = 12 H = 12 H = 12 under our training budget. The analysis also assumes a finite action space, which incurs the ∣ A ∣ 3 / 2 |\mathcal{A}|^{3/2} ∣ A ∣ 3/2 factor through uniform exploration, exact realizability ϕ ∗ ∈ Φ \phi^* \in\Phi ϕ ∗ ∈ Φ , and reward and utility linear in ϕ ∗ \phi^* ϕ ∗ .
Conclusion
We presented MFRLC, a model-free algorithm for online low-rank CMDPs with unknown features. By combining LSVI-UCB, a primal–dual scheme, and a minimax representation-learning objective, MFRLC jointly learns features and constraint-aware policies and achieves O ~ ( K 3 / 4 ) \widetilde{O}(K^{3/4}) O ( K 3/4 ) reward regret and cumulative constraint violation over K K K deployment rounds. Improving this rate, and certifying a scalable representation learner against the guarantee of Algorithm 2, remain open.
Acknowledgments and Disclosure of Funding
Funding. The authors received no third-party funding or support for this work. Competing interests. The authors declare no competing interests.
Supplementary Materials
In this supplementary material, we provide additional details omitted from the main paper for brevity. Throughout the appendix, “episode k k k ” refers to deployment round k k k (Section 2).
LSVI-UCB Algorithm Details
LSVI-UCB Algorithm
Algorithm 3 provides the full pseudocode for our constrained Least-Squares Value Iteration with Upper Confidence Bounds (LSVI-UCB) procedure. This algorithm is invoked in the policy-update step of Algorithm 1 (line 8) at each episode k k k , taking as inputs (i) the trajectory data collected up to episode k k k via the exploration step (line 4 of Algorithm 1, with uniform actions a ~ h \tilde{a}_h a ~ h ), (ii) the learned representation ϕ ^ k \hat{\phi}_k ϕ ^ k , and (iii) the current dual variable Y k Y_k Y k . It returns the optimistic per-objective Q Q Q - and V V V -functions { Q r , h k , Q g , h k , V r , h k , V g , h k } h = 1 H \{Q^k_{r,h}, Q^k_{g,h}, V^k_{r,h}, V^k_{g,h}\}_{h=1}^{H} { Q r , h k , Q g , h k , V r , h k , V g , h k } h = 1 H together with the policy π k = { π h k } h = 1 H \pi^k = \{\pi^k_h\}_{h=1}^{H} π k = { π h k } h = 1 H .
The algorithm proceeds via backward induction from the final horizon step H H H down to 1. At each step h h h , it first constructs the regularized Gram matrix Σ h k \Sigma^k_h Σ h k (line 4), which represents the empirical covariance of the learned features over past episodes. This matrix is then used to perform a regularized least-squares regression to find the linear weights for the reward value function ( w r , h k ) (w^k_{r,h}) ( w r , h k ) and the utility value function ( w g , h k ) (w^k_{g,h}) ( w g , h k ) (lines 5–6). An optimistic UCB exploration bonus b ^ h k \hat{b}^k_h b ^ h k is then computed to encourage exploration in less-visited state–action pairs (line 7). This single bonus is added to both the reward and utility linear value estimates to produce the optimistic Q Q Q -values (lines 8–9). The policy π h k \pi^k_h π h k is derived from a softmax distribution over the composite Q Q Q -function Q r , h k + Y k Q g , h k Q^k_{r,h} + Y_k Q^k_{g,h} Q r , h k + Y k Q g , h k (line 10). Finally, the per-objective state-value functions are updated under this policy (lines 11–12) and used at the next iteration of the backward induction.
Remark 3 (Single bonus, composite bonus). Lines 8–9 add the same per-objective bonus b ^ h k \hat{b}^k_h b ^ h k to both Q r , h k Q^k_{r,h} Q r , h k and Q g , h k Q^k_{g,h} Q g , h k . Consequently, the composite Q Q Q -function Q r , h k + Y k Q g , h k Q^k_{r,h} + Y_k Q^k_{g,h} Q r , h k + Y k Q g , h k used in line 10 inherits a composite bonus of magnitude ( 1 + Y k ) b ^ h k (1 + Y_k)\hat{b}^k_h ( 1 + Y k ) b ^ h k . The analysis (Appendices § 15 –§ 17 ) denotes this composite bonus by b ^ h ( k ) : = ( 1 + Y k ) b ^ h k \hat{b}^{(k)}_h := (1 + Y_k)\hat{b}^k_h b ^ h ( k ) := ( 1 + Y k ) b ^ h k .
Comparator
Theorem 3 compares against an arbitrary optimal feasible stochastic policy π ⋆ \pi^\star π ⋆ of (1). The policy π ⋆ \pi^\star π ⋆ enters the analysis only through the pseudo-regret term of (9), which Lemma 4 bounds by combining composite optimism (Lemma 2), the softmax sub-optimality bound (Lemma 12), and the simulation lemma (Lemma 19). None of these steps requires π ⋆ \pi^\star π ⋆ to be produced by Algorithm 3 from any representation and dual variable. Lemma 2 bounds the optimistic value of the fixed policy π ⋆ \pi^\star π ⋆ on the planning model, and Lemma 12 holds for an arbitrary comparator policy. The finite-temperature softmax error is already accounted for by the explicit K H log ∣ A ∣ / α KH\log|\mathcal{A}|/\alpha K H log ∣ A ∣/ α term, so exact recovery of π ⋆ \pi^\star π ⋆ is unnecessary.
For measurability in the concentration arguments, note that Algorithm 3 uses the data only through the feature evaluations ϕ ^ h k ( s h k , a ~ h k ) \hat{\phi}^k_h(s^k_h,\tilde{a}^k_h) ϕ ^ h k ( s h k , a ~ h k ) , the regression targets, and Y k Y_k Y k . Hence π k \pi^k π k is a deterministic function of ϕ ⋆ \phi^\star ϕ ⋆ , the data collected before round k k k , Y k Y_k Y k , and fixed hyperparameters.
Related Work
Representation Learning in Low-Rank MDPs. Low-rank MDPs [3 ] provide a structural framework where the transition kernel admits a d d d -dimensional factorization, generalizing earlier rich-observation models [16 ] . Existing approaches fall into two categories. Model-based methods explicitly estimate both the feature map and the next-state factor [24 , 23 ] , requiring realizability assumptions on the full dynamics. Model-free methods avoid recovering the next-state factor and instead learn the feature via a min-max objective against a discriminator class: Zhang et al. [29 ] (BRIEE) operates in block MDPs, Modi et al. [17 ] extends to general low-rank MDPs, and Mhammedi et al. [15 ] (VoX) achieves improved sample complexity via barycentric spanners. None of these works handles constraints , leaving the constrained setting open.
Algorithm 3 Least-Squares Value Iteration with UCB (LSVI-UCB)
Require: Trajectory data { ( s h τ , a ~ h τ , r h ( s h τ , a ~ h τ ) , g h ( s h τ , a ~ h τ ) , s h + 1 τ , ϕ ^ h k ( s h τ , a ~ h τ ) ) } h = 1 , τ = 1 H , k − 1 \{(s_h^\tau,\tilde{a}_h^\tau,r_h(s_h^\tau,\tilde{a}_h^\tau),g_h(s_h^\tau,\tilde{a}_h^\tau),s_{h+1}^\tau,\hat{\phi}_h^k(s_h^\tau,\tilde{a}_h^\tau))\}_{h=1,\tau=1}^{H,k-1} {( s h τ , a ~ h τ , r h ( s h τ , a ~ h τ ) , g h ( s h τ , a ~ h τ ) , s h + 1 τ , ϕ ^ h k ( s h τ , a ~ h τ )) } h = 1 , τ = 1 H , k − 1 from line 4 of Algorithm 1; current dual variable Y k Y_k Y k
1: Parameter: α = K log ∣ A ∣ 2 ( 1 + ξ + H ) \alpha= \frac{K\log|\mathcal{A}|}{2(1+\xi+H)} α = 2 ( 1 + ξ + H ) K l o g ∣ A ∣ , β = C 1 d H log ( 4 log ∣ A ∣ d T ∣ Φ ∣ / p ) \beta= C_1dH\sqrt{\log(4\log|\mathcal{A}|dT|\Phi|/p)} β = C 1 d H log ( 4 log ∣ A ∣ d T ∣Φ∣/ p ) , regularization λ \lambda λ
Output: Optimistic estimates { Q r , h k , Q g , h k , V r , h k , V g , h k } h = 1 H \{Q_{r,h}^k,Q_{g,h}^k,V_{r,h}^k,V_{g,h}^k\}_{h=1}^H { Q r , h k , Q g , h k , V r , h k , V g , h k } h = 1 H and policy π k = { π h k } h = 1 H \pi^k=\{\pi_h^k\}_{h=1}^H π k = { π h k } h = 1 H
2: Initialize V r , H + 1 k ( ⋅ ) ≡ 0 V_{r,H+1}^k(\cdot)\equiv0 V r , H + 1 k ( ⋅ ) ≡ 0 , V g , H + 1 k ( ⋅ ) ≡ 0 V_{g,H+1}^k(\cdot)\equiv0 V g , H + 1 k ( ⋅ ) ≡ 0
3: for h = H h=H h = H to 1 1 1 do
4:
Σ h k ← ∑ τ = 1 k − 1 ϕ ^ h k ( s h τ , a ~ h τ ) ϕ ^ h k ( s h τ , a ~ h τ ) ⊤ + λ I \Sigma_h^k \leftarrow\sum_{\tau=1}^{k-1}\hat{\phi}_h^k(s_h^\tau,\tilde{a}_h^\tau)\hat{\phi}_h^k(s_h^\tau,\tilde{a}_h^\tau)^\top+\lambda I Σ h k ← τ = 1 ∑ k − 1 ϕ ^ h k ( s h τ , a ~ h τ ) ϕ ^ h k ( s h τ , a ~ h τ ) ⊤ + λ I
5:
w r , h k ← ( Σ h k ) − 1 ∑ τ = 1 k − 1 ϕ ^ h k ( s h τ , a ~ h τ ) ( r h ( s h τ , a ~ h τ ) + V r , h + 1 k ( s h + 1 τ ) ) w_{r,h}^k \leftarrow(\Sigma_h^k)^{-1}\sum_{\tau=1}^{k-1}\hat{\phi}_h^k(s_h^\tau,\tilde{a}_h^\tau)\left(r_h(s_h^\tau,\tilde{a}_h^\tau)+V_{r,h+1}^k(s_{h+1}^\tau)\right) w r , h k ← ( Σ h k ) − 1 τ = 1 ∑ k − 1 ϕ ^ h k ( s h τ , a ~ h τ ) ( r h ( s h τ , a ~ h τ ) + V r , h + 1 k ( s h + 1 τ ) )
6:
w g , h k ← ( Σ h k ) − 1 ∑ τ = 1 k − 1 ϕ ^ h k ( s h τ , a ~ h τ ) ( g h ( s h τ , a ~ h τ ) + V g , h + 1 k ( s h + 1 τ ) ) w_{g,h}^k \leftarrow(\Sigma_h^k)^{-1}\sum_{\tau=1}^{k-1}\hat{\phi}_h^k(s_h^\tau,\tilde{a}_h^\tau)\left(g_h(s_h^\tau,\tilde{a}_h^\tau)+V_{g,h+1}^k(s_{h+1}^\tau)\right) w g , h k ← ( Σ h k ) − 1 τ = 1 ∑ k − 1 ϕ ^ h k ( s h τ , a ~ h τ ) ( g h ( s h τ , a ~ h τ ) + V g , h + 1 k ( s h + 1 τ ) )
7:
b ^ h k ( s , a ) ← β ϕ ^ h k ( s , a ) ⊤ ( Σ h k ) − 1 ϕ ^ h k ( s , a ) \hat{b}_h^k(s,a)\leftarrow\beta\sqrt{\hat{\phi}_h^k(s,a)^\top(\Sigma_h^k)^{-1}\hat{\phi}_h^k(s,a)} b ^ h k ( s , a ) ← β ϕ ^ h k ( s , a ) ⊤ ( Σ h k ) − 1 ϕ ^ h k ( s , a )
// per-objective bonus
8:
Q r , h k ( s , a ) ← min ( ⟨ w r , h k , ϕ ^ h k ( s , a ) ⟩ + b ^ h k ( s , a ) , H ) Q_{r,h}^k(s,a)\leftarrow\min\left(\left\langle w_{r,h}^k,\hat{\phi}_h^k(s,a)\right\rangle+\hat{b}_h^k(s,a),H\right) Q r , h k ( s , a ) ← min ( ⟨ w r , h k , ϕ ^ h k ( s , a ) ⟩ + b ^ h k ( s , a ) , H )
9:
Q g , h k ( s , a ) ← min ( ⟨ w g , h k , ϕ ^ h k ( s , a ) ⟩ + b ^ h k ( s , a ) , H ) Q_{g,h}^k(s,a)\leftarrow\min\left(\left\langle w_{g,h}^k,\hat{\phi}_h^k(s,a)\right\rangle+\hat{b}_h^k(s,a),H\right) Q g , h k ( s , a ) ← min ( ⟨ w g , h k , ϕ ^ h k ( s , a ) ⟩ + b ^ h k ( s , a ) , H )
10:
π h k ( a ∣ ⋅ ) ← softmax α ( Q r , h k ( ⋅ , a ) + Y k Q g , h k ( ⋅ , a ) ) \pi_h^k(a\mid\cdot)\leftarrow\operatorname{softmax}_{\alpha}\left(Q_{r,h}^k(\cdot,a)+Y_kQ_{g,h}^k(\cdot,a)\right) π h k ( a ∣ ⋅ ) ← softmax α ( Q r , h k ( ⋅ , a ) + Y k Q g , h k ( ⋅ , a ) )
11:
V r , h k ( s ) ← ∑ a ∈ A π h k ( a ∣ s ) Q r , h k ( s , a ) V_{r,h}^k(s)\leftarrow\sum_{a\in\mathcal{A}}\pi_h^k(a\mid s)Q_{r,h}^k(s,a) V r , h k ( s ) ← a ∈ A ∑ π h k ( a ∣ s ) Q r , h k ( s , a )
12:
V g , h k ( s ) ← ∑ a ∈ A π h k ( a ∣ s ) Q g , h k ( s , a ) V_{g,h}^k(s)\leftarrow\sum_{a\in\mathcal{A}}\pi_h^k(a\mid s)Q_{g,h}^k(s,a) V g , h k ( s ) ← a ∈ A ∑ π h k ( a ∣ s ) Q g , h k ( s , a )
13: end for
Constrained MDPs. CMDPs [20 ] provide a standard framework for safe RL, where the agent maximizes reward subject to expected cumulative cost constraints. For tabular settings, primal-dual and primal-only methods [10 , 7 , 27 ] establish regret and violation guarantees, with model-free variants [25 , 9 , 26 ] achieving O ~ ( K ) \widetilde{O}(\sqrt{K}) O ( K ) rates. To scale beyond tabular settings, Achiam et al. [2 ] introduce constrained policy optimization for deep RL, Amani et al. [4 ] treats safe linear bandits, and Ding et al. [8 ] , Ghosh et al. [11 , 12 ] extend primal-dual policy optimization to linear CMDPs with O ~ ( K ) \widetilde{O}(\sqrt{K}) O ( K ) guarantees. All these works assume the feature map is known a priori —an assumption that fails when only raw observations (e.g., images) are available. Bridging this gap, MFRLC is, to our knowledge, the first algorithm with provable regret and violation guarantees for CMDPs with unknown features. The unknown-feature setting requires controlling three coupled error-propagation mechanisms absent from prior analyses. First, errors in the estimated utility values affect the dual iterate Y k Y_k Y k . Second, the changing Y k Y_k Y k modifies the composite objective r + Y k g r+Y_kg r + Y k g , so representation error propagates through both the reward and utility Bellman estimates and is amplified by 1 + Y k 1+Y_k 1 + Y k . Third, the resulting policy changes the data used for subsequent representation learning, which feeds back into the value estimates and the dual update. Prior low-rank MDP analyses do not handle constraints, and prior primal–dual CMDP analyses assume known features, so neither faces this coupling.
Representation Learning Algorithm
This appendix collects the implementation details of the representation learning step invoked in line 6 of Algorithm 1. We first specify the discriminator class V h = F h ∪ G h \mathcal{V}_h=\mathcal{F}_h\cup\mathcal{G}_h V h = F h ∪ G h , then derive the empirical surrogate of the population objective (5) , and finally state the full iterative procedure (Algorithm 2) together with its theoretical guarantee.
Discriminator Class
The two discriminator classes used in § 3.1 are defined as
F h : = { s ↦ E a ∼ π h k − 1 ( ⋅ ∣ s ) [ ⟨ ϕ ( s , a ) , θ ⟩ − ⟨ ϕ ′ ( s , a ) , θ ′ ⟩ ] : ϕ , ϕ ′ ∈ Φ h + 1 , max ( ∥ θ ∥ ∞ , ∥ θ ′ ∥ ∞ ) ≤ 1 } . (10) \mathcal{F}_h := \left\{s \mapsto\mathbb{E}_{a\sim\pi_h^{k-1}(\cdot\mid s)}\left[\langle\phi(s,a),\theta\rangle-\langle\phi'(s,a),\theta'\rangle\right] : \phi,\phi' \in\Phi_{h+1},\ \max\left(\lVert\theta\rVert_{\infty},\lVert\theta'\rVert_{\infty}\right)\le1\right\}. \tag*{(10)} F h := { s ↦ E a ∼ π h k − 1 ( ⋅ ∣ s ) [ ⟨ ϕ ( s , a ) , θ ⟩ − ⟨ ϕ ′ ( s , a ) , θ ′ ⟩ ] : ϕ , ϕ ′ ∈ Φ h + 1 , max ( ∥ θ ∥ ∞ , ∥ θ ′ ∥ ∞ ) ≤ 1 } . (10)
G h : = { s ↦ max a ∈ A ( r h + 1 ( s , a ) + g h + 1 ( s , a ) + min { ⟨ ϕ ( s , a ) , w ⟩ , 2 } 2 H + 1 + ⟨ ϕ ( s , a ) , w ′ ⟩ ) : ϕ ∈ Φ h + 1 , ∥ w ∥ ∞ ≤ c , ∥ w ′ ∥ ∞ ≤ 1 } . (11) \mathcal{G}_h := \left\{s \mapsto\max_{a\in\mathcal{A}}\left(\frac{r_{h+1}(s,a)+g_{h+1}(s,a)+\min\{\langle\phi(s,a),w\rangle,2\}}{2H+1}+\langle\phi(s,a),w'\rangle\right) : \phi\in\Phi_{h+1},\ \lVert w\rVert_{\infty}\le c,\ \lVert w'\rVert_{\infty}\le1\right\}. \tag*{(11)} G h := { s ↦ a ∈ A max ( 2 H + 1 r h + 1 ( s , a ) + g h + 1 ( s , a ) + min {⟨ ϕ ( s , a ) , w ⟩ , 2 } + ⟨ ϕ ( s , a ) , w ′ ⟩ ) : ϕ ∈ Φ h + 1 , ∥ w ∥ ∞ ≤ c , ∥ w ′ ∥ ∞ ≤ 1 } . (11)
F h \mathcal{F}_h F h captures the model-mismatch function appearing in the proof of composite optimism (Lemma 2), evaluated under the previous-episode policy π k − 1 \pi^{k-1} π k − 1 which is well-defined at the start of episode k k k . G h \mathcal{G}_h G h captures (after rescaling by 1 + Y k 1+Y_k 1 + Y k ) the linear part of the optimistic composite Q Q Q -function appearing in the policy-gap bound (Lemma 3); the non-linear softmax operator and UCB bonus produced by Algorithm 3 are not contained in G h \mathcal{G}_h G h and are handled separately via Lemma 12 and the elliptical-potential argument in Appendix § 15 . The simple form above—independent of Y k Y_k Y k —defines a discriminator class with statistical complexity log ∣ G h ∣ \log|\mathcal{G}_h| log ∣ G h ∣ that does not depend on Y k Y_k Y k . When applied to composite quantities of magnitude 1 + Y k 1+Y_k 1 + Y k in the proofs of Lemmas 2–3, the rescaling factor 1 + Y k ≤ 1 + ξ 1+Y_k\le1+\xi 1 + Y k ≤ 1 + ξ appears as an explicit multiplicative constant in the RepLearn statistical error ζ k \zeta_k ζ k , which is absorbed into the cumulative bound ( 1 + ξ ) ∑ k ϵ k (1+\xi)\sum_k\epsilon_k ( 1 + ξ ) ∑ k ϵ k in Lemma 3.
From Population to Empirical Objective
Recall the population objective (5) from the main text:
arg min ϕ ∈ Φ h max v ∈ V h min w ∈ R d E ( s , a ~ ) ∼ ρ h k [ ( ⟨ ϕ ( s , a ~ ) , w ⟩ − E s ′ ∼ P h ⋆ ( ⋅ ∣ s , a ~ ) [ v ( s ′ ) ] ) 2 ] . \underset{\phi\in\Phi_h}{\arg\min}\ \underset{v\in\mathcal{V}_h}{\max}\ \underset{w\in\mathbb{R}^d}{\min}\ \mathbb{E}_{(s,\widetilde{a})\sim\rho_h^k}\left[\left(\langle\phi(s,\widetilde{a}),w\rangle-\mathbb{E}_{s'\sim P_h^\star(\cdot\mid s,\widetilde{a})}[v(s')]\right)^2\right]. ϕ ∈ Φ h arg min v ∈ V h max w ∈ R d min E ( s , a ) ∼ ρ h k [ ( ⟨ ϕ ( s , a ) , w ⟩ − E s ′ ∼ P h ⋆ ( ⋅ ∣ s , a ) [ v ( s ′ )] ) 2 ] .
Direct optimization is intractable because the inner expectation E s ′ ∼ P h ⋆ ( ⋅ ∣ s , a ~ ) [ v ( s ′ ) ] \mathbb{E}_{s'\sim P_h^\star(\cdot\mid s,\widetilde{a})}[v(s')] E s ′ ∼ P h ⋆ ( ⋅ ∣ s , a ) [ v ( s ′ )] requires the unknown true kernel. To address this, we replace the squared deviation with the squared empirical residual:
L λ k , D h ( ϕ , v , w ) : = E ( s , a , s ~ ′ ) ∼ D h [ ( ⟨ ϕ ( s , a ~ ) , w ⟩ − v ( s ′ ) ) 2 ] + λ k ∥ w ∥ 2 2 . (12) \mathcal{L}_{\lambda_k,D_h}(\phi,v,w):=\mathbb{E}_{(s,a,\widetilde{s}')\sim D_h}\left[\left(\langle\phi(s,\widetilde{a}),w\rangle-v(s')\right)^2\right]+\lambda_k\lVert w\rVert_2^2. \tag*{(12)} L λ k , D h ( ϕ , v , w ) := E ( s , a , s ′ ) ∼ D h [ ( ⟨ ϕ ( s , a ) , w ⟩ − v ( s ′ ) ) 2 ] + λ k ∥ w ∥ 2 2 . (12)
Expanding around the conditional mean of v ( s ′ ) v(s') v ( s ′ ) gives the variance–bias decomposition
L λ k , D h ( ϕ , v , w ) = E ( s , a ~ ) ∼ ρ h k [ ( ⟨ ϕ ( s , a ~ ) , w ⟩ − E s ′ ∼ P h ⋆ ( ⋅ ∣ s , a ~ ) [ v ( s ′ ) ] ) 2 ] + E ( s , a ~ ) ∼ ρ h k [ E s ′ ∼ P h ⋆ ( ⋅ ∣ s , a ~ ) [ v ( s ′ ) − E s ′ ∼ P h ⋆ ( ⋅ ∣ s , a ~ ) [ v ( s ′ ) ] ] 2 ] + λ k ∥ w ∥ 2 2 . (13) \begin{aligned}
\mathcal{L}_{\lambda_k,D_h}(\phi,v,w)
={}&\mathbb{E}_{(s,\widetilde{a})\sim\rho_h^k}\left[\left(\langle\phi(s,\widetilde{a}),w\rangle-\mathbb{E}_{s'\sim P_h^\star(\cdot\mid s,\widetilde{a})}[v(s')]\right)^2\right]\\
&+\mathbb{E}_{(s,\widetilde{a})\sim\rho_h^k}\left[\mathbb{E}_{s'\sim P_h^\star(\cdot\mid s,\widetilde{a})}\left[v(s')-\mathbb{E}_{s'\sim P_h^\star(\cdot\mid s,\widetilde{a})}[v(s')]\right]^2\right]+\lambda_k\lVert w\rVert_2^2. \tag*{(13)}
\end{aligned} L λ k , D h ( ϕ , v , w ) = E ( s , a ) ∼ ρ h k [ ( ⟨ ϕ ( s , a ) , w ⟩ − E s ′ ∼ P h ⋆ ( ⋅ ∣ s , a ) [ v ( s ′ )] ) 2 ] + E ( s , a ) ∼ ρ h k [ E s ′ ∼ P h ⋆ ( ⋅ ∣ s , a ) [ v ( s ′ ) − E s ′ ∼ P h ⋆ ( ⋅ ∣ s , a ) [ v ( s ′ )] ] 2 ] + λ k ∥ w ∥ 2 2 . (13)
The conditional-variance term is independent of ϕ \phi ϕ but is non-trivially correlated with v v v , so naïvely minimizing L λ k , D h \mathcal{L}_{\lambda_k,D_h} L λ k , D h alone biases the inner minimization in favor of low-variance discriminators rather than the true population minimum.
To cancel the variance term, we exploit the fact that the second buffer D h ′ D'_h D h ′ contains transitions ( s ′ , a ~ ′ , s ′ ′ ) (s',\widetilde{a}',s'') ( s ′ , a ′ , s ′′ ) where, for each tuple, a ~ ′ ∼ U ( A ) \widetilde{a}'\sim U(\mathcal{A}) a ′ ∼ U ( A ) is drawn independently of the original step-h h h data and s ′ ′ ∼ P h ⋆ ( ⋅ ∣ s ′ , a ~ ′ ) s''\sim P_h^\star(\cdot\mid s',\widetilde{a}') s ′′ ∼ P h ⋆ ( ⋅ ∣ s ′ , a ′ ) shares the same conditional law as the step-h h h next-state distribution given ( s ′ , a ~ ′ ) (s',\widetilde{a}') ( s ′ , a ′ ) .
By Lemma 17 , for any ϕ ~ ∈ Φ h \widetilde{\phi}\in\Phi_h ϕ ∈ Φ h , v ∈ V h v\in\mathcal{V}_h v ∈ V h , and w ~ ∈ R d \widetilde{w}\in\mathbb{R}^d w ∈ R d ,
E D h ′ [ L λ k , D h ′ ( ϕ ~ , v , w ~ ) ] ≥ E ( s , a ~ ) ∼ ρ h k [ E s ′ ∼ P h ⋆ ( ⋅ ∣ s , a ~ ) [ v ( s ′ ) ] ] . \mathbb{E}_{D'_h}\left[\mathcal{L}_{\lambda_k,D'_h}(\widetilde{\phi},v,\widetilde{w})\right]\ge\mathbb{E}_{(s,\widetilde{a})\sim\rho_h^k}\left[\mathbb{E}_{s'\sim P_h^\star(\cdot\mid s,\widetilde{a})}[v(s')]\right]. E D h ′ [ L λ k , D h ′ ( ϕ , v , w ) ] ≥ E ( s , a ) ∼ ρ h k [ E s ′ ∼ P h ⋆ ( ⋅ ∣ s , a ) [ v ( s ′ )] ] .
with equality when ϕ ~ \widetilde{\phi} ϕ and w ~ \widetilde{w} w are chosen optimally. Subtracting the w ~ \widetilde{w} w -minimized version of L λ k , D h ′ \mathcal{L}_{\lambda_k,D'_h} L λ k , D h ′ from L λ k , D h \mathcal{L}_{\lambda_k,D_h} L λ k , D h therefore eliminates the conditional variance in expectation, recovering an estimator of the population objective up to the λ k ∥ w ∥ 2 2 \lambda_k\lVert w\rVert_2^2 λ k ∥ w ∥ 2 2 regularization. This motivates the bias-corrected objective solved by Algorithm 2:
arg min ϕ ∈ Φ h max v ∈ V h { min w L λ k , D h ( ϕ , v , w ) − min w ~ , ϕ ~ ∈ Φ h L λ k , D h ′ ( ϕ ~ , v , w ~ ) } . (14) \underset{\phi\in\Phi_h}{\arg\min}\ \underset{v\in\mathcal{V}_h}{\max}\left\{\min_w\mathcal{L}_{\lambda_k,D_h}(\phi,v,w)-\min_{\widetilde{w},\widetilde{\phi}\in\Phi_h}\mathcal{L}_{\lambda_k,D'_h}(\widetilde{\phi},v,\widetilde{w})\right\}. \tag*{(14)} ϕ ∈ Φ h arg min v ∈ V h max { w min L λ k , D h ( ϕ , v , w ) − w , ϕ ∈ Φ h min L λ k , D h ′ ( ϕ , v , w ) } . (14)
Iterative Procedure
Algorithm 2 alternates between two steps: an adversarial discriminator search that identifies the worst-case v ∈ V h v \in\mathcal{V}_h v ∈ V h for the current representation ϕ m \phi^m ϕ m (line 6), and a representation update that fits ϕ m + 1 \phi^{m+1} ϕ m + 1 jointly against all discriminators encountered so far (line 11). The inner max ϕ ~ \max_{\tilde{\phi}} max ϕ ~ on line 6 implements the bias correction of (14) : it subtracts the best variance proxy that any ϕ ~ ∈ Φ h \tilde{\phi} \in\Phi_h ϕ ~ ∈ Φ h can achieve. Termination occurs once no discriminator achieves an empirical objective above ℓ k \ell_k ℓ k , indicating that all v ∈ V h v \in\mathcal{V}_h v ∈ V h are approximately linearly representable by ϕ m \phi^m ϕ m at the prescribed statistical precision.
Theoretical Guarantee and Stopping Rule
The output ϕ ^ h k \hat{\phi}_h^k ϕ ^ h k of Algorithm 2 satisfies the Representation Learning Guarantee (Lemma 16): with probability at least 1 − δ 1-\delta 1 − δ ,
max v ∈ V h E ( s , a ~ ) ∼ ρ h k [ ( w ^ v ⊤ ϕ ^ h k ( s , a ~ ) − E s ′ ∼ P v ∗ [ v ( s ′ ) ] ) 2 ] ≤ ζ k = O ( d 2 log ( k ∣ Φ ∣ / δ ) k ) . \max_{v \in\mathcal{V}_h} \mathbb{E}_{(s,\tilde{a})\sim\rho_h^k}\left[\left(\hat{w}_v^\top\hat{\phi}_h^k(s,\tilde{a})-\mathbb{E}_{s'\sim P_v^*}[v(s')]\right)^2\right] \le\zeta_k = O\left(d^2\sqrt{\frac{\log(k|\Phi|/\delta)}{k}}\right). v ∈ V h max E ( s , a ~ ) ∼ ρ h k [ ( w ^ v ⊤ ϕ ^ h k ( s , a ~ ) − E s ′ ∼ P v ∗ [ v ( s ′ )] ) 2 ] ≤ ζ k = O ( d 2 k log ( k ∣Φ∣/ δ ) ) .
where w ^ v \hat{w}_v w ^ v is the inner minimizer corresponding to v v v . Combined with Theorem 3, this guarantees that the policies computed from ϕ ^ h k \hat{\phi}_h^k ϕ ^ h k achieve sublinear regret and constraint violation, ensuring that the learned representation can be effectively used in policy evaluation and improvement within the CMDP framework.
Stopping rule and optimization error. The theoretical analysis requires the threshold ℓ k \ell_k ℓ k to scale as O ~ ( d 2 / k ) \widetilde{O}(d^2/\sqrt{k}) O ( d 2 / k ) to match the statistical error rate. An implementation of Algorithm 2 enforces this by evaluating the maximization objective on line 6 at each iteration m m m and terminating once it falls below ℓ k \ell_k ℓ k , so the residual optimization error remains negligible compared to the statistical estimation error and the cumulative bound ∑ k ζ k \sum_k \zeta_k ∑ k ζ k used in the analysis is not inflated by optimization slack. Theorem 3 applies whenever the optimization steps of Algorithm 2 attain the residual displayed above. For finite outer classes this can be certified by exact search, and an approximate solver is also covered if its optimization error is O ( ζ k ) O(\zeta_k) O ( ζ k ) . The contrastive EMA surrogate used in our experiments (Appendix D.2) does not certify this condition and is not covered by Theorem 3.
Computational Cost
For fixed ( ϕ , v ) (\phi,v) ( ϕ , v ) , the inner problem min w L λ k , D h ( ϕ , v , w ) \min_w \mathcal{L}_{\lambda_k,\mathcal{D}_h}(\phi,v,w) min w L λ k , D h ( ϕ , v , w ) is ridge regression. With n = ∣ D h ∣ n=|\mathcal{D}_h| n = ∣ D h ∣ samples, forming the Gram matrix costs O ( n d 2 ) O(nd^2) O ( n d 2 ) and solving the d × d d\times d d × d system costs O ( d 3 ) O(d^3) O ( d 3 ) , so one solve costs O ( n d 2 + d 3 ) O(nd^2+d^3) O ( n d 2 + d 3 ) . At iteration m m m of Algorithm 2, line 6 evaluates two ridge solves for each pair ( v , ϕ ~ ) ∈ V h × Φ h (v,\tilde{\phi})\in\mathcal{V}_h\times\Phi_h ( v , ϕ ~ ) ∈ V h × Φ h , and line 11 solves m + 1 m+1 m + 1 ridge problems for each ϕ ∈ Φ h \phi\in\Phi_h ϕ ∈ Φ h . When Φ h \Phi_h Φ h and V h \mathcal{V}_h V h are finite (or replaced by finite covers), every step can therefore be computed exactly, and one call with at most M M M iterations costs:
O ( ( M ∣ Φ h ∣ ∣ V h ∣ + M 2 ∣ Φ h ∣ ) ( n d 2 + d 3 ) ) . O\left((M|\Phi_h||\mathcal{V}_h|+M^2|\Phi_h|)(nd^2+d^3)\right). O ( ( M ∣ Φ h ∣∣ V h ∣ + M 2 ∣ Φ h ∣ ) ( n d 2 + d 3 ) ) .
This is polynomial in n n n and d d d but may be large in the class cardinalities. For continuously parameterized classes such as neural networks, the same empirical objective (14) is explicit, but its outer optimization is non-convex, and we do not prove polynomial-time convergence to the global minimax solution. Given ϕ ^ k \hat{\phi}^k ϕ ^ k , Algorithm 3 performs two backward passes of ridge regression over k − 1 k-1 k − 1 samples per step, and the projected dual update (8) costs O ( 1 ) O(1) O ( 1 ) .
Experiment Details
This section provides a detailed overview of the experimental setup, including environment specifics, network architectures, training protocols, and a comprehensive list of hyperparameters used for all algorithms to ensure reproducibility.
Environment Setup
All experiments are conducted in our custom-designed Constrained Latent ComboLock (CLCL) environment. We formulate constraints as costs bounded above (cost ≤ C max = b \leq C_{\max}=b ≤ C m a x = b ), which is equivalent to the main-text utility convention via V g = − c o s t V_g=-\mathrm{cost} V g = − cost , b u t i l i t y = − C max b_{\mathrm{utility}}=-C_{\max} b utility = − C m a x , so that V g ≥ b u t i l i t y V_g \ge b_{\mathrm{utility}} V g ≥ b utility corresponds to the cost staying below threshold.
Horizons and action spaces: The main experiments use H = 6 H=6 H = 6 and ∣ A ∣ = 3 |\mathcal{A}|=3 ∣ A ∣ = 3 . The additional settings in Table 2 enlarge the action set to ∣ A ∣ = 5 |\mathcal{A}|=5 ∣ A ∣ = 5 at H = 6 H=6 H = 6 , and use H ∈ { 8 , 12 } H \in\{8,12\} H ∈ { 8 , 12 } with ∣ A ∣ = 3 |\mathcal{A}|=3 ∣ A ∣ = 3 .
Observation Space: The agent receives high-dimensional observations in the form of 8 × 84 × 84 8 \times84 \times84 8 × 84 × 84 image tensors. Each tensor is procedurally generated based on the agent’s latent state s h s_h s h and the current timestep h h h . The eight channels are specifically designed to encode rich information:
Channels 0–2 (Progress and Position): These channels encode the agent’s progress through the latent state chain. Channel 0 contains a Gaussian “spotlight” whose horizontal position corresponds to the current state s h s_h s h . Specifically, for a pixel at coordinates ( x , y ) ∈ [ 0 , 1 ] 2 (x,y) \in[0,1]^2 ( x , y ) ∈ [ 0 , 1 ] 2 , the value is generated as:
o x , y ( 0 ) = exp ( − 20 [ ( x − s h + 1 H + 1 ) 2 + ( y − 0.5 ) 2 ] ) . o_{x,y}^{(0)}=\exp\left(-20\left[\left(x-\frac{s_h+1}{H+1}\right)^2+(y-0.5)^2\right]\right). o x , y ( 0 ) = exp ( − 20 [ ( x − H + 1 s h + 1 ) 2 + ( y − 0.5 ) 2 ] ) .
Channels 1 and 2 contain sine and cosine patterns whose frequencies change with the agent’s progress, providing additional positional information. They are defined as:
o x , y ( 1 ) = 0.5 sin ( ( s h + 1 ) π x ) , o x , y ( 2 ) = 0.5 cos ( ( s h + 1 ) π y ) . o_{x,y}^{(1)}=0.5\sin((s_h+1)\pi x),\qquad o_{x,y}^{(2)}=0.5\cos((s_h+1)\pi y). o x , y ( 1 ) = 0.5 sin (( s h + 1 ) π x ) , o x , y ( 2 ) = 0.5 cos (( s h + 1 ) π y ) .
These patterns ensure that each depth in the latent chain has a distinct spatial frequency signature.
State Textures: A unique, pseudo-random visual texture is overlaid on the first few channels. The choice of texture depends on the latent state, timestep, and previous action, making each observation visually distinct.
Channels 3–4 (Background Patterns): These channels contain static, high-frequency sine and cosine waves that form a consistent background grid.
Channel 5 (Fail Indicator): This channel acts as a binary flag. It is filled with ones if the agent is in the terminal S F a i l S_{\mathrm{Fail}} S Fail state, and zeros otherwise.
Channel 6 (Safe Indicator): Similar to the fail indicator, this channel is filled with ones if the agent is in the terminal S S a f e S_{\mathrm{Safe}} S Safe state, and zeros otherwise.
Channel 7 (Timestep): This channel’s value is constant across the image and represents the normalized current timestep, i.e., h / H h/H h / H .
Noise: Finally, Gaussian noise with a standard deviation of 0.1 is added to all channels to increase the perceptual difficulty of the task.
Action Space: The action space is discrete, containing three actions: A = { a f o r w a r d , a h a z a r d , a s t a y } \mathcal{A}=\{a_{\mathrm{forward}},a_{\mathrm{hazard}},a_{\mathrm{stay}}\} A = { a forward , a hazard , a stay } .
Rewards and Costs: A reward of +1 is given only for successfully reaching the S S a f e S_{\mathrm{Safe}} S Safe state from the final state in the chain. A cost of 1 is incurred for taking the a h a z a r d a_{\mathrm{hazard}} a hazard action. The cost constraint for all experiments is set to a threshold of C max = 1.0 C_{\max}=1.0 C m a x = 1.0 .
Network Architectures
The core of the visual processing for MFRLC and all visual baselines is a shared ResNet-style encoder.
Visual Encoder: We use a VisualEncoder that maps the 8 × 84 × 84 8 \times84 \times84 8 × 84 × 84 observation to a 64-dimensional feature vector. It consists of a convolutional stem followed by four ResNet-style BasicBlock layers and a final linear layer.
Policy and Value Heads: For actor–critic baselines (CPO and LAG-PG), this visual encoder serves as a shared backbone. The policy (actor), reward value (critic), and cost value (cost critic) are represented by separate linear heads that take the 64-dimensional feature vector as input.
Representation Learning: For representation-learning methods (MFRLC and BRIEE), the encoder is part of a module with a two-layer ProjectionHead for contrastive learning and a target encoder updated via exponential moving average (EMA). Exact min-max-min optimization over Φ h × V h \Phi_h \times\mathcal{V}_h Φ h × V h in Algorithm 2 is intractable for deep encoders, so we replace it with contrastive next-state prediction on transition triples ( o , a , o ~ ) (o, a, \tilde{o}) ( o , a , o ~ ) . This surrogate is not covered by Theorem 3 (Section 3).
ϕ ( o , a ) = MLP ( [ ψ ( o ) ; e a ] ) ∈ R 64 . \phi(o, a) = \operatorname{MLP}([\psi(o); e_a]) \in\mathbb{R}^{64}. ϕ ( o , a ) = MLP ([ ψ ( o ) ; e a ]) ∈ R 64 .
The features ϕ ( o , a ) \phi(o, a) ϕ ( o , a ) are the design vectors of the reward and utility ridge regressions in LSVI-UCB, so in practice Φ \Phi Φ is the class of state–action feature maps induced by this parameterization.
Training Protocol
To ensure a fair and rigorous comparison, all algorithms were trained and evaluated under a consistent protocol.
Training: Each algorithm is trained for a fixed number of iterations, as detailed in Table 3 . At each iteration, a number of episodes are collected from the environment to update the agent’s policy and/or representation.
Table 3. Hyperparameters for all algorithms.
Hyperparameters
The following table details the key hyperparameters used for our proposed method and all baseline algorithms.
Cumulative Regret and Violation
Let R ˉ t \bar{R}_t R ˉ t and C ˉ t \bar{C}_t C ˉ t be the mean reward and mean cost over the 48 deployed episodes of iteration t t t . Since reaching S S a f e S_{\mathrm{Safe}} S Safe yields reward 1 and a f o r w a r d a_{\mathrm{forward}} a forward incurs no cost, the optimal feasible value is V r , 1 π ∗ ( s 1 ) = 1 V_{r,1}^{\pi^*}(s_1) = 1 V r , 1 π ∗ ( s 1 ) = 1 . After T T T iterations, we estimate the cumulative regret and cumulative violation as:
Regret ^ = 48 ∑ t = 1 T ( 1 − R ˉ t ) , Violation ^ = [ 48 ∑ t = 1 T ( C ˉ t − C max ) ] + . \widehat{\operatorname{Regret}} = 48 \sum_{t=1}^{T}(1-\bar{R}_t), \qquad\widehat{\operatorname{Violation}} = \left[48 \sum_{t=1}^{T}(\bar{C}_t-C_{\max})\right]_+ . Regret = 48 t = 1 ∑ T ( 1 − R ˉ t ) , Violation = [ 48 t = 1 ∑ T ( C ˉ t − C m a x ) ] + .
which follow the definitions in Section 2 under the cost convention of Appendix § 13 .1. Table 2 reports the resulting cumulative violation for the additional settings. At H = 8 H = 8 H = 8 , the estimated cumulative regret of MFRLC is 15,382 ± 942 15{,}382 \pm942 15 , 382 ± 942 over K = 15,552 K = 15{,}552 K = 15 , 552 deployed episodes (324 iterations). These are finite-run measurements; we do not fit or claim an empirical exponent.
Additional Settings
The three additional settings in Table 2 use the hyperparameters of Table 3 , with the same 5 seeds. The H = 8 H = 8 H = 8 runs use 324 iterations (15,552 deployed episodes). The H = 12 H = 12 H = 12 stress test runs longer and evaluates only MFRLC, which reaches a best reward of 0.021 ± 0.013 0.021 \pm0.013 0.021 ± 0.013 with cumulative violation 1206.9 ± 76.2 1206.9 \pm76.2 1206.9 ± 76.2 ; under uniform exploration, discovering the complete rewarding sequence is 3 4 = 81 3^{4} = 81 3 4 = 81 times less likely than at H = 8 H = 8 H = 8 .
Adapting Baselines for Fair Comparison
To create a level playing field, it was necessary to adapt the baseline algorithms that were not originally designed for Constrained Markov Decision Processes (CMDPs). We use CPO [2 ] as-is, since it is natively designed for CMDPs; only HOMER and BRIEE require Lagrangian adaptation as detailed below. Specifically, we extended HOMER and BRIEE with a standard Lagrangian multiplier method. The dual variable λ \lambda λ is updated using the same primal–dual rule as in Algorithm 1, but since neither HOMER nor BRIEE maintains a separate cost critic, we estimate the cost value V ^ g ( s 1 ) \widehat{V}_{g}(s_{1}) V g ( s 1 ) used in the dual update as the difference between the augmented value Q λ Q^{\lambda} Q λ (which is what these baselines actually learn) and the empirical reward return collected along the rollout, i.e. V ^ g ( s 1 ) ≈ \widehat{V}_{g}(s_{1}) \approx V g ( s 1 ) ≈ (empirical reward return − Q λ ( s 1 , a 1 ) - Q^{\lambda}(s_{1}, a_{1}) − Q λ ( s 1 , a 1 ) )/ λ /\lambda / λ when λ > 0 \lambda> 0 λ > 0 , with a small-λ \lambda λ fallback to a Monte-Carlo cost return estimate. The application of the dual variable to the value learning is tailored to the unique architecture of each agent.
BRIEE with Constraints The BRIEE algorithm decouples representation learning from planning. We integrate the safety constraint directly into its planning module, which uses a variant of Least-Squares Value Iteration (LSVI). The planner’s objective is to solve for an optimal action-value function, Q ∗ Q^{*} Q ∗ , by iteratively applying the Bellman optimality operator. We modify this objective by replacing the standard reward R ( s , a ) R(s, a) R ( s , a ) with the augmented reward R λ ( s , a ) = R ( s , a ) − λ C ( s , a ) R^{\lambda}(s, a) = R(s, a) - \lambda C(s, a) R λ ( s , a ) = R ( s , a ) − λ C ( s , a ) .
Specifically, the planner now solves for a constraint-aware value function, Q λ Q^{\lambda} Q λ , according to the modified Bellman equation:
Q h λ ( s , a ) ← ( R h ( s , a ) − λ C h ( s , a ) ) + E s ′ ∼ P h ( ⋅ ∣ s , a ) [ max a ′ Q h + 1 λ ( s ′ , a ′ ) ] . Q_{h}^{\lambda}(s, a) \leftarrow\left(R_{h}(s, a) - \lambda C_{h}(s, a)\right) + \mathbb{E}_{s'\sim P_{h}(\cdot\mid s,a)}\left[\max_{a'} Q_{h+1}^{\lambda}(s', a')\right]. Q h λ ( s , a ) ← ( R h ( s , a ) − λ C h ( s , a ) ) + E s ′ ∼ P h ( ⋅ ∣ s , a ) [ a ′ max Q h + 1 λ ( s ′ , a ′ ) ] .
By solving for the optimal policy under this new objective, the planner learns to inherently avoid actions that are likely to violate the constraint.
HOMER with Constraints HOMER is a hierarchical, value-based algorithm that learns from off-policy data. We incorporate the constraint into its value function learning process by modifying the targets used for the temporal-difference (TD) update. The agent’s critic is trained to minimize the TD error between its current estimate Q ( s t , a t ) Q(s_{t}, a_{t}) Q ( s t , a t ) and a target value y t y_{t} y t . The standard TD target is y t = r t + γ max a ′ Q ( s t + 1 , a ′ ) y_{t} = r_{t} + \gamma\max_{a'} Q(s_{t+1}, a') y t = r t + γ max a ′ Q ( s t + 1 , a ′ ) .
To make the agent constraint-aware, we modify this target to use the augmented reward:
y t λ ← ( r t − λ c t ) + γ max a ′ Q λ ( s t + 1 , a ′ ) . y_{t}^{\lambda} \leftarrow(r_{t} - \lambda c_{t}) + \gamma\max_{a'} Q^{\lambda}(s_{t+1}, a'). y t λ ← ( r t − λ c t ) + γ a ′ max Q λ ( s t + 1 , a ′ ) .
The critic is then trained by minimizing a loss function based on this new target, such as the mean squared error L = E [ ( Q λ ( s t , a t ) − y t λ ) 2 ] \mathcal{L} = \mathbb{E}[(Q^{\lambda}(s_{t}, a_{t}) - y_{t}^{\lambda})^{2}] L = E [( Q λ ( s t , a t ) − y t λ ) 2 ] . This ensures that the learned value functions, which guide the agent’s policy, account for both the expected future reward and the expected future cost, weighted by the dual variable λ \lambda λ .
Proof of Decomposition
In this section we prove the decomposition of regret and violation used in the master inequality (9) .
Lemma 8. For any Y ∈ [ 0 , ξ ] Y \in[0,\xi] Y ∈ [ 0 , ξ ] , we have
∑ k = 1 K ( Y − Y k ) ( b − V g , 1 k ( s 1 ) ) ≤ Y 2 2 η + η H 2 K 2 . \sum_{k=1}^{K}(Y-Y_k)(b-V_{g,1}^{k}(s_1)) \le\frac{Y^2}{2\eta}+\frac{\eta H^2K}{2}. k = 1 ∑ K ( Y − Y k ) ( b − V g , 1 k ( s 1 )) ≤ 2 η Y 2 + 2 η H 2 K .
Proof. The dual update rule of Algorithm 1 (line 12) is
Y k + 1 = max { min { Y k + η ( b − V g , 1 k ( s 1 ) ) , ξ } , 0 } . Y_{k+1}=\max\left\{\min\{Y_k+\eta(b-V_{g,1}^{k}(s_1)),\xi\},0\right\}. Y k + 1 = max { min { Y k + η ( b − V g , 1 k ( s 1 )) , ξ } , 0 } .
Let Δ k : = b − V g , 1 k ( s 1 ) \Delta_k:=b-V_{g,1}^{k}(s_1) Δ k := b − V g , 1 k ( s 1 ) . Since this update is the Euclidean projection of Y k + η Δ k Y_k+\eta\Delta_k Y k + η Δ k onto the interval [ 0 , ξ ] [0,\xi] [ 0 , ξ ] , for any Y ∈ [ 0 , ξ ] Y\in[0,\xi] Y ∈ [ 0 , ξ ] ,
( Y k + 1 − Y ) 2 ≤ ( Y k + η Δ k − Y ) 2 = ( Y k − Y ) 2 + 2 η Δ k ( Y k − Y ) + η 2 Δ k 2 . (Y_{k+1}-Y)^2\leq(Y_k+\eta\Delta_k-Y)^2
=(Y_k-Y)^2+2\eta\Delta_k(Y_k-Y)+\eta^2\Delta_k^2. ( Y k + 1 − Y ) 2 ≤ ( Y k + η Δ k − Y ) 2 = ( Y k − Y ) 2 + 2 η Δ k ( Y k − Y ) + η 2 Δ k 2 .
Rearranging to bound the inner product,
2 η Δ k ( Y − Y k ) ≤ ( Y k − Y ) 2 − ( Y k + 1 − Y ) 2 + η 2 Δ k 2 . 2\eta\Delta_k(Y-Y_k)\leq(Y_k-Y)^2-(Y_{k+1}-Y)^2+\eta^2\Delta_k^2. 2 η Δ k ( Y − Y k ) ≤ ( Y k − Y ) 2 − ( Y k + 1 − Y ) 2 + η 2 Δ k 2 .
Substituting Δ k = b − V g , 1 k ( s 1 ) \Delta_k=b-V_{g,1}^{k}(s_1) Δ k = b − V g , 1 k ( s 1 ) and summing over k = 1 , … , K k=1,\ldots,K k = 1 , … , K ,
2 η ∑ k = 1 K ( b − V g , 1 k ( s 1 ) ) ( Y − Y k ) ≤ ∑ k = 1 K ( ( Y k − Y ) 2 − ( Y k + 1 − Y ) 2 ) + η 2 ∑ k = 1 K ( b − V g , 1 k ( s 1 ) ) 2 ≤ ( Y 1 − Y ) 2 + η 2 H 2 K , \begin{aligned}
2\eta\sum_{k=1}^{K}(b-V_{g,1}^{k}(s_1))(Y-Y_k)\leq\sum_{k=1}^{K}\left((Y_k-Y)^2-(Y_{k+1}-Y)^2\right)+\eta^2\sum_{k=1}^{K}(b-V_{g,1}^{k}(s_1))^2 \\
&\leq(Y_1-Y)^2+\eta^2H^2K,
\end{aligned} 2 η k = 1 ∑ K ( b − V g , 1 k ( s 1 )) ( Y − Y k ) ≤ k = 1 ∑ K ( ( Y k − Y ) 2 − ( Y k + 1 − Y ) 2 ) + η 2 k = 1 ∑ K ( b − V g , 1 k ( s 1 ) ) 2 ≤ ( Y 1 − Y ) 2 + η 2 H 2 K ,
where the last inequality follows from telescoping, dropping the negative term − ( Y K + 1 − Y ) 2 -(Y_{K+1}-Y)^2 − ( Y K + 1 − Y ) 2 , and using ∣ b − V g , 1 k ( s 1 ) ∣ ≤ H |b-V_{g,1}^{k}(s_1)|\leq H ∣ b − V g , 1 k ( s 1 ) ∣ ≤ H . Dividing by 2 η 2\eta 2 η and noting Y 1 = 0 Y_1=0 Y 1 = 0 ,
∑ k = 1 K ( Y − Y k ) ( b − V g , 1 k ( s 1 ) ) ≤ Y 2 2 η + η H 2 K 2 . \sum_{k=1}^{K}(Y-Y_k)(b-V_{g,1}^{k}(s_1))\leq\frac{Y^2}{2\eta}+\frac{\eta H^2K}{2}. k = 1 ∑ K ( Y − Y k ) ( b − V g , 1 k ( s 1 )) ≤ 2 η Y 2 + 2 η H 2 K .
Lemma 9. For any Y ∈ [ 0 , ξ ] Y\in[0,\xi] Y ∈ [ 0 , ξ ] ,
∑ k = 1 K ( V r , 1 π ∗ ( s 1 ) − V r , 1 π k ( s 1 ) ) + Y ∑ k = 1 K ( b − V g , 1 π k ( s 1 ) ) ≤ Y 2 2 η + η H 2 K 2 + ∑ k = 1 K Y k ( b − V g , 1 π ∗ ( s 1 ) ) ⏟ Slack (non-positive) + ∑ k = 1 K [ ( V r , 1 π ∗ ( s 1 ) + Y k V g , 1 π ∗ ( s 1 ) ) − ( V r , 1 π k ( s 1 ) + Y k V g , 1 π k ( s 1 ) ) ] ⏟ Pseudo regret + ∑ k = 1 K [ ( V r , 1 k ( s 1 ) − V r , 1 π ∗ ( s 1 ) ) + Y ( V g , 1 k ( s 1 ) − V g , 1 π ∗ ( s 1 ) ) ] ⏟ Estimation error . \begin{aligned}
&\sum_{k=1}^{K}(V_{r,1}^{\pi^*}(s_1)-V_{r,1}^{\pi_k}(s_1))+Y\sum_{k=1}^{K}(b-V_{g,1}^{\pi_k}(s_1))\\
&\leq\frac{Y^2}{2\eta}+\frac{\eta H^2K}{2}+\underbrace{\sum_{k=1}^{K}Y_k(b-V_{g,1}^{\pi^*}(s_1))}_{\text{Slack (non-positive)}}\\
&\quad+\underbrace{\sum_{k=1}^{K}\left[(V_{r,1}^{\pi^*}(s_1)+Y_kV_{g,1}^{\pi^*}(s_1))-(V_{r,1}^{\pi_k}(s_1)+Y_kV_{g,1}^{\pi_k}(s_1))\right]}_{\text{Pseudo regret}}\\
&\quad+\underbrace{\sum_{k=1}^{K}\left[(V_{r,1}^{k}(s_1)-V_{r,1}^{\pi^*}(s_1))+Y(V_{g,1}^{k}(s_1)-V_{g,1}^{\pi^*}(s_1))\right]}_{\text{Estimation error}}.
\end{aligned} k = 1 ∑ K ( V r , 1 π ∗ ( s 1 ) − V r , 1 π k ( s 1 )) + Y k = 1 ∑ K ( b − V g , 1 π k ( s 1 )) ≤ 2 η Y 2 + 2 η H 2 K + Slack (non-positive) k = 1 ∑ K Y k ( b − V g , 1 π ∗ ( s 1 )) + Pseudo regret k = 1 ∑ K [ ( V r , 1 π ∗ ( s 1 ) + Y k V g , 1 π ∗ ( s 1 )) − ( V r , 1 π k ( s 1 ) + Y k V g , 1 π k ( s 1 )) ] + Estimation error k = 1 ∑ K [ ( V r , 1 k ( s 1 ) − V r , 1 π ∗ ( s 1 )) + Y ( V g , 1 k ( s 1 ) − V g , 1 π ∗ ( s 1 )) ] .
Proof. We start by decomposing the Lagrangian term involving the constraint violation. Using Lemma 8 ,
Y ∑ k = 1 K ( b − V g , 1 π k ( s 1 ) ) = ∑ k = 1 K ( Y − Y k ) ( b − V g , 1 π k ( s 1 ) ) + ∑ k = 1 K Y k ( b − V g , 1 π k ( s 1 ) ) + Y ∑ k = 1 K ( V g , 1 π k ( s 1 ) − V g , 1 π k ( s 1 ) ) ≤ Y 2 2 η + η H 2 K 2 + ∑ k = 1 K ( Y k b − Y k V g , 1 π k ( s 1 ) ) + Y ∑ k = 1 K ( V g , 1 π k ( s 1 ) − V g , 1 π k ( s 1 ) ) . \begin{aligned}
Y \sum_{k=1}^{K} \left(b - V_{g,1}^{\pi^k}(s_1)\right)
= \sum_{k=1}^{K} (Y-Y_k)\left(b-V_{g,1}^{\pi^k}(s_1)\right)
+ \sum_{k=1}^{K}Y_k\left(b-V_{g,1}^{\pi^k}(s_1)\right)
+Y\sum_{k=1}^{K}\left(V_{g,1}^{\pi^k}(s_1)-V_{g,1}^{\pi^k}(s_1)\right) \\
&\leq\frac{Y^2}{2\eta}+\frac{\eta H^2K}{2}
+\sum_{k=1}^{K}\left(Y_kb-Y_kV_{g,1}^{\pi^k}(s_1)\right)
+Y\sum_{k=1}^{K}\left(V_{g,1}^{\pi^k}(s_1)-V_{g,1}^{\pi^k}(s_1)\right).
\end{aligned} Y k = 1 ∑ K ( b − V g , 1 π k ( s 1 ) ) = k = 1 ∑ K ( Y − Y k ) ( b − V g , 1 π k ( s 1 ) ) + k = 1 ∑ K Y k ( b − V g , 1 π k ( s 1 ) ) + Y k = 1 ∑ K ( V g , 1 π k ( s 1 ) − V g , 1 π k ( s 1 ) ) ≤ 2 η Y 2 + 2 η H 2 K + k = 1 ∑ K ( Y k b − Y k V g , 1 π k ( s 1 ) ) + Y k = 1 ∑ K ( V g , 1 π k ( s 1 ) − V g , 1 π k ( s 1 ) ) .
We rewrite ∑ k = 1 K Y k ( b − V g , 1 π k ( s 1 ) ) \sum_{k=1}^{K}Y_k\left(b-V_{g,1}^{\pi^k}(s_1)\right) ∑ k = 1 K Y k ( b − V g , 1 π k ( s 1 ) ) by adding and subtracting V g , 1 π ⋆ ( s 1 ) V_{g,1}^{\pi^\star}(s_1) V g , 1 π ⋆ ( s 1 ) :
∑ k = 1 K Y k ( b − V g , 1 π k ( s 1 ) ) = ∑ k = 1 K Y k ( b − V g , 1 π ⋆ ( s 1 ) ) + ∑ k = 1 K Y k ( V g , 1 π ⋆ ( s 1 ) − V g , 1 π k ( s 1 ) ) . \sum_{k=1}^{K}Y_k\left(b-V_{g,1}^{\pi^k}(s_1)\right)
=\sum_{k=1}^{K}Y_k\left(b-V_{g,1}^{\pi^\star}(s_1)\right)
+\sum_{k=1}^{K}Y_k\left(V_{g,1}^{\pi^\star}(s_1)-V_{g,1}^{\pi^k}(s_1)\right). k = 1 ∑ K Y k ( b − V g , 1 π k ( s 1 ) ) = k = 1 ∑ K Y k ( b − V g , 1 π ⋆ ( s 1 ) ) + k = 1 ∑ K Y k ( V g , 1 π ⋆ ( s 1 ) − V g , 1 π k ( s 1 ) ) .
Since π ⋆ \pi^\star π ⋆ is feasible, we have V g , 1 π ⋆ ( s 1 ) ≥ b V_{g,1}^{\pi^\star}(s_1)\geq b V g , 1 π ⋆ ( s 1 ) ≥ b , so b − V g , 1 π ⋆ ( s 1 ) ≤ 0 b-V_{g,1}^{\pi^\star}(s_1)\leq0 b − V g , 1 π ⋆ ( s 1 ) ≤ 0 . Combined with Y k ≥ 0 Y_k\geq0 Y k ≥ 0 , the slack term ∑ k = 1 K Y k ( b − V g , 1 π ⋆ ( s 1 ) ) \sum_{k=1}^{K}Y_k\left(b-V_{g,1}^{\pi^\star}(s_1)\right) ∑ k = 1 K Y k ( b − V g , 1 π ⋆ ( s 1 ) ) is non-positive. We retain it explicitly to make the decomposition transparent; dropping it tightens the bound in the correct direction (recovering (9) in the main text).
Adding the reward regret ∑ k = 1 K ( V r , 1 π ⋆ ( s 1 ) − V r , 1 π k ( s 1 ) ) \sum_{k=1}^{K}\left(V_{r,1}^{\pi^\star}(s_1)-V_{r,1}^{\pi^k}(s_1)\right) ∑ k = 1 K ( V r , 1 π ⋆ ( s 1 ) − V r , 1 π k ( s 1 ) ) to both sides and rearranging:
∑ k = 1 K ( V r , 1 π ⋆ ( s 1 ) − V r , 1 π k ( s 1 ) ) + Y ∑ k = 1 K ( b − V g , 1 π k ( s 1 ) ) ≤ Y 2 2 η + η H 2 K 2 + ∑ k = 1 K Y k ( b − V g , 1 π ⋆ ( s 1 ) ) + ∑ k = 1 K [ ( V r , 1 π ⋆ ( s 1 ) + Y k V g , 1 π ⋆ ( s 1 ) ) − ( V r , 1 π k ( s 1 ) + Y k V g , 1 π k ( s 1 ) ) ] + ∑ k = 1 K [ ( V r , 1 π k ( s 1 ) − V r , 1 π k ( s 1 ) ) + Y ( V g , 1 π k ( s 1 ) − V g , 1 π k ( s 1 ) ) ] . \begin{aligned}
&\sum_{k=1}^{K}\left(V_{r,1}^{\pi^\star}(s_1)-V_{r,1}^{\pi^k}(s_1)\right)
+Y\sum_{k=1}^{K}\left(b-V_{g,1}^{\pi^k}(s_1)\right)\\
&\leq\frac{Y^2}{2\eta}+\frac{\eta H^2K}{2}
+\sum_{k=1}^{K}Y_k\left(b-V_{g,1}^{\pi^\star}(s_1)\right)\\
&\quad+\sum_{k=1}^{K}\left[\left(V_{r,1}^{\pi^\star}(s_1)+Y_kV_{g,1}^{\pi^\star}(s_1)\right)
-\left(V_{r,1}^{\pi^k}(s_1)+Y_kV_{g,1}^{\pi^k}(s_1)\right)\right]\\
&\quad+\sum_{k=1}^{K}\left[\left(V_{r,1}^{\pi^k}(s_1)-V_{r,1}^{\pi^k}(s_1)\right)
+Y\left(V_{g,1}^{\pi^k}(s_1)-V_{g,1}^{\pi^k}(s_1)\right)\right].
\end{aligned} k = 1 ∑ K ( V r , 1 π ⋆ ( s 1 ) − V r , 1 π k ( s 1 ) ) + Y k = 1 ∑ K ( b − V g , 1 π k ( s 1 ) ) ≤ 2 η Y 2 + 2 η H 2 K + k = 1 ∑ K Y k ( b − V g , 1 π ⋆ ( s 1 ) ) + k = 1 ∑ K [ ( V r , 1 π ⋆ ( s 1 ) + Y k V g , 1 π ⋆ ( s 1 ) ) − ( V r , 1 π k ( s 1 ) + Y k V g , 1 π k ( s 1 ) ) ] + k = 1 ∑ K [ ( V r , 1 π k ( s 1 ) − V r , 1 π k ( s 1 ) ) + Y ( V g , 1 π k ( s 1 ) − V g , 1 π k ( s 1 ) ) ] .
Identifying the slack, pseudo regret, and estimation error terms completes the proof.
Bounding Pseudo regret
Composite reward and notation. Throughout this section, fix an episode k ∈ { 1 , … , K } k\in\{1,\ldots,K\} k ∈ { 1 , … , K } and let Y k ∈ [ 0 , ξ ] Y_k\in[0,\xi] Y k ∈ [ 0 , ξ ] denote the dual iterate produced by Algorithm 1. We work with the composite stage reward
r ˉ h ( k ) ( s , a ) : = r h ( s , a ) + Y k g h ( s , a ) ∈ [ 0 , 1 + Y k ] , \bar r_h^{(k)}(s,a):=r_h(s,a)+Y_kg_h(s,a)\in[0,1+Y_k], r ˉ h ( k ) ( s , a ) := r h ( s , a ) + Y k g h ( s , a ) ∈ [ 0 , 1 + Y k ] ,
and the corresponding composite cumulative value, which lies in [ 0 , ( 1 + Y k ) H ] [0,(1+Y_k)H] [ 0 , ( 1 + Y k ) H ] . For any policy π \pi π and any transition kernel P P P , define the composite value and action-value functions
V P , h π , Y k ( s ) : = V P , r , h π ( s ) + Y k V P , g , h π ( s ) , Q P , h π , Y k ( s , a ) : = Q P , r , h π ( s , a ) + Y k Q P , g , h π ( s , a ) . V_{P,h}^{\pi,Y_k}(s):=V_{P,r,h}^{\pi}(s)+Y_kV_{P,g,h}^{\pi}(s),
\qquad
Q_{P,h}^{\pi,Y_k}(s,a):=Q_{P,r,h}^{\pi}(s,a)+Y_kQ_{P,g,h}^{\pi}(s,a). V P , h π , Y k ( s ) := V P , r , h π ( s ) + Y k V P , g , h π ( s ) , Q P , h π , Y k ( s , a ) := Q P , r , h π ( s , a ) + Y k Q P , g , h π ( s , a ) .
By linearity of expectations, V P , h π , Y k V_{P,h}^{\pi,Y_k} V P , h π , Y k and Q P , h π , Y k Q_{P,h}^{\pi,Y_k} Q P , h π , Y k are precisely the value and action-value functions of policy π \pi π in the auxiliary MDP ( P , r ˉ ( k ) ) (P,\bar r^{(k)}) ( P , r ˉ ( k ) ) . We further write b ^ h ( k ) ( s , a ) : = ( 1 + Y k ) b ^ h ( s , a ) \hat b_h^{(k)}(s,a):=(1+Y_k)\hat b_h(s,a) b ^ h ( k ) ( s , a ) := ( 1 + Y k ) b ^ h ( s , a ) for the composite optimism bonus, where b ^ h ( s , a ) = β ∥ ϕ ^ h ( s , a ) ∥ Σ ρ ^ h , ϕ ^ h − 1 \hat b_h(s,a)=\beta\lVert\hat\phi_h(s,a)\rVert_{\Sigma_{\hat\rho_h,\hat\phi_h}^{-1}} b ^ h ( s , a ) = β ∥ ϕ ^ h ( s , a ) ∥ Σ ρ ^ h , ϕ ^ h − 1 is the per-objective bonus computed in Algorithm 3. Since 0 ≤ Y k ≤ ξ 0\leq Y_k\leq\xi 0 ≤ Y k ≤ ξ , we have b ^ h ( k ) ≤ ( 1 + ξ ) b ^ h \hat b_h^{(k)}\leq(1+\xi)\hat b_h b ^ h ( k ) ≤ ( 1 + ξ ) b ^ h . The implicit planning model P ^ h k ( s ′ ∣ s , a ) : = ⟨ ϕ ^ h k ( s , a ) , μ ^ h k ( s ′ ) ⟩ \widehat{P}_h^k(s' \mid s,a):=\langle\widehat{\phi}_h^k(s,a),\widehat{\mu}_h^k(s')\rangle P h k ( s ′ ∣ s , a ) := ⟨ ϕ h k ( s , a ) , μ h k ( s ′ )⟩ is defined as in Section 4; the density factor μ ^ h k \widehat{\mu}_h^k μ h k is recovered from the LSVI weights and is used only as an analysis device.
We define the following mixture distributions to facilitate the analysis. For any k , h k,h k , h , let ρ h k ∈ Δ ( S × A ) \rho_h^k\in\Delta(\mathcal{S}\times\mathcal{A}) ρ h k ∈ Δ ( S × A ) be the data distribution induced by Algorithm 1’s exploration step, namely
ρ h k ( s , a ) = 1 k ∑ i = 0 k − 1 d h π i ( s ) U ( A ) ( a ) . \rho_h^k(s,a)=\frac{1}{k}\sum_{i=0}^{k-1}d_h^{\pi^i}(s)U(\mathcal{A})(a). ρ h k ( s , a ) = k 1 i = 0 ∑ k − 1 d h π i ( s ) U ( A ) ( a ) .
i.e. the average of past on-policy state distributions paired with a uniform action distribution. We similarly define β h k ( s , a ) : = 1 k ∑ i = 0 k − 1 E s ~ ∼ d h − 1 π i , a ~ ∼ U ( A ) [ P h − 1 ⋆ ( s ∣ s ~ , a ~ ) ] U ( A ) ( a ) \beta_h^k(s,a):=\frac{1}{k}\sum_{i=0}^{k-1}\mathbb{E}_{\widetilde{s}\sim d_{h-1}^{\pi^i},\,\widetilde{a}\sim U(\mathcal{A})}\left[P_{h-1}^\star(s\mid\widetilde{s},\widetilde{a})\right]U(\mathcal{A})(a) β h k ( s , a ) := k 1 ∑ i = 0 k − 1 E s ∼ d h − 1 π i , a ∼ U ( A ) [ P h − 1 ⋆ ( s ∣ s , a ) ] U ( A ) ( a ) and γ h k ( s , a ) : = 1 k ∑ i = 0 k − 1 d h π i ( s , a ) \gamma_h^k(s,a):=\frac{1}{k}\sum_{i=0}^{k-1}d_h^{\pi^i}(s,a) γ h k ( s , a ) := k 1 ∑ i = 0 k − 1 d h π i ( s , a ) . When the episode index k k k is clear from context, we drop it and write ρ h \rho_h ρ h , β h \beta_h β h , γ h \gamma_h γ h . All occurrences of the action variable in ρ h \rho_h ρ h and β h \beta_h β h correspond to the uniform exploratory actions a ~ \widetilde{a} a of Algorithm 1 (line 4); we keep the symbol a a a inside expectations for notational compactness, with the understanding that a ∼ U ( A ) a\sim U(\mathcal{A}) a ∼ U ( A ) .
Probability budget. Throughout the analysis we set the RepLearn confidence parameter δ = p / 4 \delta=p/4 δ = p /4 , so that union bounding the RepLearn guarantee (Lemma 16 ), the concentration event E \mathcal{E} E (Lemma 5), and the martingale deviation (Lemma 15 ) yields total probability at least 1 − p 1-p 1 − p as claimed in Theorem 3.
Discriminator class. Recall the discriminator class V h = F h ∪ G h \mathcal{V}_h=\mathcal{F}_h\cup\mathcal{G}_h V h = F h ∪ G h designed as follows:
F h : = { s ↦ E a ∼ π h k − 1 ( ⋅ ∣ s ) [ ϕ ( s , a ) ⊤ θ − ϕ ′ ( s , a ) ⊤ θ ′ ] : ϕ , ϕ ′ ∈ Φ h + 1 , max ( ∥ θ ∥ ∞ , ∥ θ ′ ∥ ∞ ) ≤ 1 } , \mathcal{F}_h:=\left\{s\mapsto\mathbb{E}_{a\sim\pi_h^{k-1}(\cdot\mid s)}\left[\phi(s,a)^\top\theta-\phi'(s,a)^\top\theta'\right]:\phi,\phi'\in\Phi_{h+1},\ \max(\|\theta\|_\infty,\|\theta'\|_\infty)\le1\right\}, F h := { s ↦ E a ∼ π h k − 1 ( ⋅ ∣ s ) [ ϕ ( s , a ) ⊤ θ − ϕ ′ ( s , a ) ⊤ θ ′ ] : ϕ , ϕ ′ ∈ Φ h + 1 , max ( ∥ θ ∥ ∞ , ∥ θ ′ ∥ ∞ ) ≤ 1 } ,
G h : = { s ↦ max a ∈ A ( r h + 1 ( s , a ) + g h + 1 ( s , a ) + min { w ⊤ ϕ ( s , a ) , 2 } 2 H + 1 + ϕ ( s , a ) ⊤ w ′ ) : ϕ ∈ Φ h + 1 , ∥ w ∥ ∞ ≤ c , ∥ w ′ ∥ ∞ ≤ 1 } . \mathcal{G}_h:=\left\{s\mapsto\max_{a\in\mathcal{A}}\left(\frac{r_{h+1}(s,a)+g_{h+1}(s,a)+\min\{w^\top\phi(s,a),2\}}{2H+1}+\phi(s,a)^\top w'\right):\phi\in\Phi_{h+1},\ \|w\|_\infty\le c,\ \|w'\|_\infty\le1\right\}. G h := { s ↦ a ∈ A max ( 2 H + 1 r h + 1 ( s , a ) + g h + 1 ( s , a ) + min { w ⊤ ϕ ( s , a ) , 2 } + ϕ ( s , a ) ⊤ w ′ ) : ϕ ∈ Φ h + 1 , ∥ w ∥ ∞ ≤ c , ∥ w ′ ∥ ∞ ≤ 1 } .
The class F h \mathcal{F}_h F h captures, via linear function approximation, all functions of the form
f ( s ) = E a ∼ π h k − 1 ( ⋅ ∣ s ) [ E s ′ ∼ P ^ h ( ⋅ ∣ s , a ) [ V P ⋆ , h + 1 π , Y k ( s ′ ) ] − E s ′ ∼ P h ⋆ ( ⋅ ∣ s , a ) [ V P ⋆ , h + 1 π , Y k ( s ′ ) ] ] , f(s)=\mathbb{E}_{a\sim\pi_h^{k-1}(\cdot\mid s)}\left[\mathbb{E}_{s'\sim\widehat{P}_h(\cdot\mid s,a)}\left[V_{P^\star,h+1}^{\pi,Y_k}(s')\right]-\mathbb{E}_{s'\sim P_h^\star(\cdot\mid s,a)}\left[V_{P^\star,h+1}^{\pi,Y_k}(s')\right]\right], f ( s ) = E a ∼ π h k − 1 ( ⋅ ∣ s ) [ E s ′ ∼ P h ( ⋅ ∣ s , a ) [ V P ⋆ , h + 1 π , Y k ( s ′ ) ] − E s ′ ∼ P h ⋆ ( ⋅ ∣ s , a ) [ V P ⋆ , h + 1 π , Y k ( s ′ ) ] ] ,
which appears in the proof of Lemma 2 after a uniform-action importance-sampling step that converts an E a ∼ π h ⋆ \mathbb{E}_{a\sim\pi_h^\star} E a ∼ π h ⋆ expectation into an E a ∼ π h k − 1 \mathbb{E}_{a\sim\pi_h^{k-1}} E a ∼ π h k − 1 expectation, contributing a ∣ A ∣ \sqrt{|\mathcal{A}|} ∣ A ∣ factor that is already tracked in our final ∣ A ∣ 3 / 2 |\mathcal{A}|^{3/2} ∣ A ∣ 3/2 rate. The class G h \mathcal{G}_h G h captures (after rescaling by ( 1 + Y k ) (1+Y_k) ( 1 + Y k ) ) the linear part of the composite optimistic action-value
g ( s ) = 1 ( 2 H + 1 ) ( 1 + Y k ) Q P , r ˉ ( k ) + b ^ ( k ) , h + 1 π k , Y k ( s ) , g(s)=\frac{1}{(2H+1)(1+Y_k)}Q_{P,\bar r^{(k)}+\widehat b^{(k)},h+1}^{\pi^{k},Y_k}(s), g ( s ) = ( 2 H + 1 ) ( 1 + Y k ) 1 Q P , r ˉ ( k ) + b ( k ) , h + 1 π k , Y k ( s ) ,
which appears in the proof of Lemma 3. The non-linear softmax operator and UCB bonus produced by Algorithm 3 are not contained in G h \mathcal{G}_h G h ; their contribution is handled separately via Lemma 12 (the softmax-vs-max gap, contributing H log ∣ A ∣ / α H\log|\mathcal{A}|/\alpha H log ∣ A ∣/ α ) and Lemma 13 (the LSVI-UCB residual, dominated by ∑ k ϵ k \sum_k\epsilon_k ∑ k ϵ k ). With the simpler definition above (matching the main text), elements of G h \mathcal{G}_h G h are bounded by 1 + Y k ≤ 1 + ξ 1+Y_k\le1+\xi 1 + Y k ≤ 1 + ξ rather than 1 1 1 ; this enters as a multiplicative ( 1 + ξ ) (1+\xi) ( 1 + ξ ) factor in the RepLearn statistical error wherever G h \mathcal{G}_h G h is used, which we track explicitly below.
We first establish two “one-step” inequalities that decompose the on-policy value of an arbitrary function g h g_h g h into a bilinear feature term plus a residual.
Lemma 10 (One-step inequality for the learned model). Consider functions g h : S × A → R g_h:\mathcal{S}\times\mathcal{A}\to\mathbb{R} g h : S × A → R with ∥ g h ∥ ∞ ≤ B \|g_h\|_\infty\le B ∥ g h ∥ ∞ ≤ B . Assume that s ↦ E a ∼ π h ( ⋅ ∣ s ) [ g h + 1 ( s , a ) ] ∈ F h s\mapsto\mathbb{E}_{a\sim\pi_h(\cdot\mid s)}[g_{h+1}(s,a)]\in\mathcal{F}_h s ↦ E a ∼ π h ( ⋅ ∣ s ) [ g h + 1 ( s , a )] ∈ F h . Conditioned on the Representation Learning Guarantee, for any policy π \pi π
∑ h = 0 H − 1 E ( s , a ) ∼ d h π [ g h ( s , a ) ] ≤ ∑ h = 1 H − 1 E ( s ~ , a ~ ) ∼ d h − 1 π [ ∥ ϕ ^ h − 1 ( s ~ , a ~ ) ∥ Σ ϕ h − 1 , ϕ ^ h − 1 − 1 ] ⋅ k ∣ A ∣ 2 E ( s , a ) ∼ β h [ g ^ h 2 ( s , a ) ] + B 2 λ k d + k ∣ A ∣ ζ k + ∣ A ∣ E ( s , a ) ∼ ρ 0 [ g 0 2 ( s , a ) ] . \sum_{h=0}^{H-1}\mathbb{E}_{(s,a)\sim d_h^\pi}[g_h(s,a)]
\le\sum_{h=1}^{H-1}\mathbb{E}_{(\widetilde{s},\widetilde{a})\sim d_{h-1}^{\pi}}\left[\|\widehat{\phi}_{h-1}(\widetilde{s},\widetilde{a})\|_{\Sigma_{\phi_{h-1},\widehat{\phi}_{h-1}}^{-1}}\right]\cdot\sqrt{k|\mathcal{A}|^2\mathbb{E}_{(s,a)\sim\beta_h}[\widehat{g}_h^2(s,a)]+B^2\lambda_kd+k|\mathcal{A}|\zeta_k}
+\sqrt{|\mathcal{A}|\mathbb{E}_{(s,a)\sim\rho_0}[g_0^2(s,a)]}. h = 0 ∑ H − 1 E ( s , a ) ∼ d h π [ g h ( s , a )] ≤ h = 1 ∑ H − 1 E ( s , a ) ∼ d h − 1 π [ ∥ ϕ h − 1 ( s , a ) ∥ Σ ϕ h − 1 , ϕ h − 1 − 1 ] ⋅ k ∣ A ∣ 2 E ( s , a ) ∼ β h [ g h 2 ( s , a )] + B 2 λ k d + k ∣ A ∣ ζ k + ∣ A ∣ E ( s , a ) ∼ ρ 0 [ g 0 2 ( s , a )] .
Proof. We consider the inequality in two cases.
Case 1: h = 0 h = 0 h = 0 .
E ( s , a ) ∼ d P ^ , 0 π [ g 0 ( s , a ) ] = E s ∼ d 0 , a ∼ π 0 ( ⋅ ∣ s ) [ g 0 ( s , a ) ] ≤ ∣ A ∣ E ( s , a ) ∼ ρ 0 [ g 0 2 ( s , a ) ] . \mathbb{E}_{(s,a)\sim d^{\pi}_{\widehat{P},0}}[g_0(s,a)] = \mathbb{E}_{s\sim d_0,\ a\sim\pi_0(\cdot\mid s)}[g_0(s,a)] \le\sqrt{|\mathcal{A}| \mathbb{E}_{(s,a)\sim\rho_0}[g_0^2(s,a)]}. E ( s , a ) ∼ d P , 0 π [ g 0 ( s , a )] = E s ∼ d 0 , a ∼ π 0 ( ⋅ ∣ s ) [ g 0 ( s , a )] ≤ ∣ A ∣ E ( s , a ) ∼ ρ 0 [ g 0 2 ( s , a )] .
where the inequality uses Cauchy–Schwarz with the importance ratio d 0 ( s ) π 0 ( a ∣ s ) d 0 ( s ) U ( a ) ≤ ∣ A ∣ \frac{d_0(s)\pi_0(a\mid s)}{d_0(s)U(a)} \le|\mathcal{A}| d 0 ( s ) U ( a ) d 0 ( s ) π 0 ( a ∣ s ) ≤ ∣ A ∣ , since the actions in the replay buffer are sampled uniformly.
Case 2: h = 1 , … , H − 1 h = 1,\ldots,H-1 h = 1 , … , H − 1 .
E ( s , a ) ∼ d P ^ , h π [ g h ( s , a ) ] = E ( s ~ , a ~ ) ∼ d P ^ , h − 1 π [ ϕ ^ h − 1 ( s ~ , a ~ ) ⊤ ∫ ∑ a μ ^ h − 1 ( s ) π h ( a ∣ s ) g h ( s , a ) d s ] ≤ E ( s ~ , a ~ ) ∼ d P ^ , h − 1 π [ ∥ ϕ ^ h − 1 ∥ Σ ρ h − 1 , ϕ ^ h − 1 − 1 ⋅ ∥ ∫ ∑ a μ ^ h − 1 ( s ) π h ( a ∣ s ) g h ( s , a ) d s ∥ Σ ρ h − 1 , ϕ ^ h − 1 ] . (15) \begin{aligned}
\mathbb{E}_{(s,a)\sim d^{\pi}_{\widehat{P},h}}[g_h(s,a)]
&= \mathbb{E}_{(\widetilde{s},\widetilde{a})\sim d^{\pi}_{\widehat{P},h-1}}
\left[
\widehat{\phi}_{h-1}(\widetilde{s},\widetilde{a})^\top
\int\sum_a \widehat{\mu}_{h-1}(s)\pi_h(a\mid s)g_h(s,a)\,ds
\right] \\
&\le\mathbb{E}_{(\widetilde{s},\widetilde{a})\sim d^{\pi}_{\widehat{P},h-1}}
\left[
\|\widehat{\phi}_{h-1}\|_{\Sigma^{-1}_{\rho_{h-1},\widehat{\phi}_{h-1}}}
\cdot
\left\|
\int\sum_a \widehat{\mu}_{h-1}(s)\pi_h(a\mid s)g_h(s,a)\,ds
\right\|_{\Sigma_{\rho_{h-1},\widehat{\phi}_{h-1}}}
\right]. \tag*{(15)}
\end{aligned} E ( s , a ) ∼ d P , h π [ g h ( s , a )] = E ( s , a ) ∼ d P , h − 1 π [ ϕ h − 1 ( s , a ) ⊤ ∫ a ∑ μ h − 1 ( s ) π h ( a ∣ s ) g h ( s , a ) d s ] ≤ E ( s , a ) ∼ d P , h − 1 π ∥ ϕ h − 1 ∥ Σ ρ h − 1 , ϕ h − 1 − 1 ⋅ ∫ a ∑ μ h − 1 ( s ) π h ( a ∣ s ) g h ( s , a ) d s Σ ρ h − 1 , ϕ h − 1 . (15)
where the inequality is Cauchy–Schwarz with respect to the positive-definite matrix Σ ρ h − 1 , ϕ ^ h − 1 = k E ρ h − 1 [ ϕ ^ h − 1 ϕ ^ h − 1 ⊤ ] + λ k I \Sigma_{\rho_{h-1},\widehat{\phi}_{h-1}} = k\mathbb{E}_{\rho_{h-1}}[\widehat{\phi}_{h-1}\widehat{\phi}_{h-1}^{\top}] + \lambda_k I Σ ρ h − 1 , ϕ h − 1 = k E ρ h − 1 [ ϕ h − 1 ϕ h − 1 ⊤ ] + λ k I .
We bound the Gram-norm term inside (15) :
∥ ∫ ∑ a μ ^ h ( s ) π h ( a ∣ s ) g h + 1 ( s , a ) d s ∥ Σ γ h , ϕ ^ h 2 ≤ k E ( s ~ , a ~ ) ∼ γ h [ ( ∫ ∑ a μ ^ h ( s ) ⊤ ϕ ^ h ( s ~ , a ~ ) π h ( a ∣ s ) g h + 1 ( s , a ) d s ) 2 ] + B 2 λ k d \begin{aligned}
\left\|
\int\sum_a \widehat{\mu}_h(s)\pi_h(a\mid s)g_{h+1}(s,a)\,ds
\right\|^2_{\Sigma_{\gamma_h,\widehat{\phi}_h}}
&\le k\mathbb{E}_{(\widetilde{s},\widetilde{a})\sim\gamma_h}
\left[
\left(
\int\sum_a \widehat{\mu}_h(s)^\top\widehat{\phi}_h(\widetilde{s},\widetilde{a})\pi_h(a\mid s)g_{h+1}(s,a)\,ds
\right)^2
\right]
+B^2\lambda_kd
\end{aligned} ∫ a ∑ μ h ( s ) π h ( a ∣ s ) g h + 1 ( s , a ) d s Σ γ h , ϕ h 2 ≤ k E ( s , a ) ∼ γ h ( ∫ a ∑ μ h ( s ) ⊤ ϕ h ( s , a ) π h ( a ∣ s ) g h + 1 ( s , a ) d s ) 2 + B 2 λ k d
(using ∥ ∑ a π h ( a ∣ s ) g h + 1 ∥ ∞ ≤ B \left\|\sum_a\pi_h(a\mid s)g_{h+1}\right\|_\infty\le B ∥ ∑ a π h ( a ∣ s ) g h + 1 ∥ ∞ ≤ B and ∥ ∫ μ ^ h ( s ) f ( s ) d s ∥ 2 ≤ d \left\|\int\widehat{\mu}_h(s)f(s)\,ds\right\|_2\le\sqrt{d} ∫ μ h ( s ) f ( s ) d s 2 ≤ d )
= k E ( s ~ , a ~ ) ∼ γ h [ ( E s ∼ P ^ h ( s ~ , a ~ ) , a ∼ π h [ g h + 1 ( s , a ) ] ) 2 ] + B 2 λ k d ≤ k E ( s ~ , a ~ ) ∼ γ h [ ( E s ∼ P h ⋆ ( s ~ , a ~ ) , a ∼ π h [ g h + 1 ( s , a ) ] ) 2 ] + B 2 λ k d + k ∣ A ∣ 2 ζ k ≤ k ∣ A ∣ 2 E ( s , a ) ∼ β h + 1 [ g h + 1 2 ( s , a ) ] + B 2 λ k d + k ∣ A ∣ 2 ζ k , \begin{aligned}
&= k\mathbb{E}_{(\widetilde{s},\widetilde{a})\sim\gamma_h}
\left[
\left(
\mathbb{E}_{s\sim\widehat{P}_h(\widetilde{s},\widetilde{a}),\ a\sim\pi_h}
[g_{h+1}(s,a)]
\right)^2
\right]
+B^2\lambda_kd \\
&\le k\mathbb{E}_{(\widetilde{s},\widetilde{a})\sim\gamma_h}
\left[
\left(
\mathbb{E}_{s\sim P^\star_h(\widetilde{s},\widetilde{a}),\ a\sim\pi_h}
[g_{h+1}(s,a)]
\right)^2
\right]
+B^2\lambda_kd+k|\mathcal{A}|^2\zeta_k \\
&\le k|\mathcal{A}|^2\mathbb{E}_{(s,a)\sim\beta_{h+1}}[g_{h+1}^2(s,a)]
+B^2\lambda_kd+k|\mathcal{A}|^2\zeta_k,
\end{aligned} = k E ( s , a ) ∼ γ h [ ( E s ∼ P h ( s , a ) , a ∼ π h [ g h + 1 ( s , a )] ) 2 ] + B 2 λ k d ≤ k E ( s , a ) ∼ γ h [ ( E s ∼ P h ⋆ ( s , a ) , a ∼ π h [ g h + 1 ( s , a )] ) 2 ] + B 2 λ k d + k ∣ A ∣ 2 ζ k ≤ k ∣ A ∣ 2 E ( s , a ) ∼ β h + 1 [ g h + 1 2 ( s , a )] + B 2 λ k d + k ∣ A ∣ 2 ζ k ,
(by the Representation Learning Guarantee)
where the last step uses uniform-action importance sampling and Jensen’s inequality. Substituting back into (15) and summing over h = 1 , … , H − 1 h=1,\ldots,H-1 h = 1 , … , H − 1 , then adding the Case 1 term, completes the proof.
Lemma 11 (One-step inequality for the true model). Consider functions g h : S × A → R g_h:\mathcal{S}\times\mathcal{A}\to\mathbb{R} g h : S × A → R with ∥ g h ∥ ∞ ≤ B \|g_h\|_\infty\le B ∥ g h ∥ ∞ ≤ B . For any policy π \pi π ,
∑ h = 0 H − 1 E ( s , a ) ∼ d P ⋆ , h π [ g h ( s , a ) ] ≤ ∑ h = 1 H − 1 E ( s ~ , a ~ ) ∼ d P ⋆ , h − 1 π [ ∥ ϕ h − 1 ⋆ ( s ~ , a ~ ) ∥ Σ γ h − 1 , ϕ h − 1 ⋆ − 1 k ∣ A ∣ E ( s , a ) ∼ ρ h [ g h 2 ( s , a ) ] + B 2 λ k d ] + ∣ A ∣ E ( s , a ) ∼ ρ 0 [ g 0 2 ( s , a ) ] . \begin{aligned}
\sum_{h=0}^{H-1}\mathbb{E}_{(s,a)\sim d^{\pi}_{P^\star,h}}[g_h(s,a)]
&\le\sum_{h=1}^{H-1}\mathbb{E}_{(\widetilde{s},\widetilde{a})\sim d^{\pi}_{P^\star,h-1}}
\left[
\|\phi^\star_{h-1}(\widetilde{s},\widetilde{a})\|_{\Sigma^{-1}_{\gamma_{h-1},\phi^\star_{h-1}}}
\sqrt{k|\mathcal{A}|\mathbb{E}_{(s,a)\sim\rho_h}[g_h^2(s,a)]+B^2\lambda_kd}
\right] \\
&\quad+\sqrt{|\mathcal{A}|\mathbb{E}_{(s,a)\sim\rho_0}[g_0^2(s,a)]}.
\end{aligned} h = 0 ∑ H − 1 E ( s , a ) ∼ d P ⋆ , h π [ g h ( s , a )] ≤ h = 1 ∑ H − 1 E ( s , a ) ∼ d P ⋆ , h − 1 π [ ∥ ϕ h − 1 ⋆ ( s , a ) ∥ Σ γ h − 1 , ϕ h − 1 ⋆ − 1 k ∣ A ∣ E ( s , a ) ∼ ρ h [ g h 2 ( s , a )] + B 2 λ k d ] + ∣ A ∣ E ( s , a ) ∼ ρ 0 [ g 0 2 ( s , a )] .
Proof. The argument is identical to the proof of Lemma 10, except that we work with the true low-rank decomposition P h ⋆ ( s ′ ∣ s , a ) = ⟨ ϕ h ⋆ ( s , a ) , μ h ⋆ ( s ′ ) ⟩ P^\star_h(s'\mid s,a)=\langle\phi^\star_h(s,a),\mu^\star_h(s')\rangle P h ⋆ ( s ′ ∣ s , a ) = ⟨ ϕ h ⋆ ( s , a ) , μ h ⋆ ( s ′ )⟩ , so the Representation Learning residual k ∣ A ∣ 2 ζ k k|\mathcal{A}|^2\zeta_k k ∣ A ∣ 2 ζ k does not appear. The factor ∣ A ∣ 2 |\mathcal{A}|^2 ∣ A ∣ 2 in Lemma 10 is replaced by ∣ A ∣ |\mathcal{A}| ∣ A ∣ because the additional importance-sampling step from P ^ \widehat{P} P to P ⋆ P^\star P ⋆ is unnecessary here.
We now establish the composite optimism guarantee, which extends Lemma 2 to the Lagrangian objective.
Lemma 2 (Restated; composite optimism). Consider an episode k ∈ { 1 , … , K } k \in\{1,\ldots,K\} k ∈ { 1 , … , K } . Conditioned on the Representation Learning Guarantee, with probability at least 1 − δ 1-\delta 1 − δ ,
Q P ^ k , r ˉ ( k ) + b ^ ( k ) π ⋆ , Y k ( s , a ) − Q P ⋆ , r ˉ ( k ) π ⋆ , Y k ( s , a ) ≥ − ( 1 + Y k ) ∣ A ∣ ζ k − ρ k o p t , Q_{\widehat{P}_k,\bar{r}^{(k)}+\widehat{b}^{(k)}}^{\pi^\star,Y_k}(s,a)-Q_{P^\star,\bar{r}^{(k)}}^{\pi^\star,Y_k}(s,a)\ge-(1+Y_k)\sqrt{|\mathcal{A}|\zeta_k}-\rho_k^{\mathrm{opt}}, Q P k , r ˉ ( k ) + b ( k ) π ⋆ , Y k ( s , a ) − Q P ⋆ , r ˉ ( k ) π ⋆ , Y k ( s , a ) ≥ − ( 1 + Y k ) ∣ A ∣ ζ k − ρ k opt , where ρ k o p t \rho_k^{\mathrm{opt}} ρ k opt is a non-negative residual term satisfying ∑ k = 1 K ρ k o p t ≤ ( 1 + ξ ) ∑ k = 1 K ϵ k \sum_{k=1}^{K}\rho_k^{\mathrm{opt}}\le(1+\xi)\sum_{k=1}^{K}\epsilon_k ∑ k = 1 K ρ k opt ≤ ( 1 + ξ ) ∑ k = 1 K ϵ k (see Lemma 3 for the cumulative bound on ∑ k ϵ k \sum_k\epsilon_k ∑ k ϵ k ).
Proof. By the simulation lemma (Lemma 19 ) applied to the auxiliary MDP with stage reward r ˉ ( k ) \bar{r}^{(k)} r ˉ ( k ) , kernel P ^ k \widehat{P}_k P k , and additive bonus b ^ ( k ) = ( 1 + Y k ) b ^ \widehat{b}^{(k)}=(1+Y_k)\widehat{b} b ( k ) = ( 1 + Y k ) b ,
Q P ^ k , r ˉ ( k ) + b ^ ( k ) π ⋆ , Y k ( s , a ) − Q P ⋆ , r ˉ ( k ) π ⋆ , Y k ( s , a ) = ∑ h = 0 H − 1 E ( s h , a h ) ∼ d P ^ k , h π ⋆ , Y k [ b ^ h ( k ) ( s h , a h ) + E s ′ ∼ P ^ k , h [ V P ⋆ , h + 1 π ⋆ , Y k ( s ′ ) ] − E s ′ ∼ P h ⋆ [ V P ⋆ , h + 1 π ⋆ , Y k ( s ′ ) ] ] . Q_{\widehat{P}_k,\bar{r}^{(k)}+\widehat{b}^{(k)}}^{\pi^\star,Y_k}(s,a)-Q_{P^\star,\bar{r}^{(k)}}^{\pi^\star,Y_k}(s,a)
=
\sum_{h=0}^{H-1}\mathbb{E}_{(s_h,a_h)\sim d_{\widehat{P}_k,h}^{\pi^\star,Y_k}}\left[\widehat{b}^{(k)}_h(s_h,a_h)+\mathbb{E}_{s'\sim\widehat{P}_{k,h}}\left[V_{P^\star,h+1}^{\pi^\star,Y_k}(s')\right]-\mathbb{E}_{s'\sim P^\star_h}\left[V_{P^\star,h+1}^{\pi^\star,Y_k}(s')\right]\right]. Q P k , r ˉ ( k ) + b ( k ) π ⋆ , Y k ( s , a ) − Q P ⋆ , r ˉ ( k ) π ⋆ , Y k ( s , a ) = h = 0 ∑ H − 1 E ( s h , a h ) ∼ d P k , h π ⋆ , Y k [ b h ( k ) ( s h , a h ) + E s ′ ∼ P k , h [ V P ⋆ , h + 1 π ⋆ , Y k ( s ′ ) ] − E s ′ ∼ P h ⋆ [ V P ⋆ , h + 1 π ⋆ , Y k ( s ′ ) ] ] .
Define
f h ( s , a ) : = 1 2 H + 1 ( E s ′ ∼ P ^ h [ V P ⋆ , h + 1 π ⋆ , Y k ( s ′ ) ] − E s ′ ∼ P h ⋆ [ V P ⋆ , h + 1 π ⋆ , Y k ( s ′ ) ] ) . f_h(s,a):=\frac{1}{2H+1}\left(\mathbb{E}_{s'\sim\widehat{P}_h}\left[V_{P^\star,h+1}^{\pi^\star,Y_k}(s')\right]-\mathbb{E}_{s'\sim P^\star_h}\left[V_{P^\star,h+1}^{\pi^\star,Y_k}(s')\right]\right). f h ( s , a ) := 2 H + 1 1 ( E s ′ ∼ P h [ V P ⋆ , h + 1 π ⋆ , Y k ( s ′ ) ] − E s ′ ∼ P h ⋆ [ V P ⋆ , h + 1 π ⋆ , Y k ( s ′ ) ] ) .
Since V P ⋆ , h + 1 π ⋆ , Y k / ( 1 + Y k ) ∈ [ 0 , H ] V_{P^\star,h+1}^{\pi^\star,Y_k}/(1+Y_k)\in[0,H] V P ⋆ , h + 1 π ⋆ , Y k / ( 1 + Y k ) ∈ [ 0 , H ] and the inner expectations are linear in ϕ ^ h \widehat{\phi}_h ϕ h and ϕ h ⋆ \phi_h^\star ϕ h ⋆ respectively, E a ∼ π h k − 1 [ ( 2 H + 1 ) f h ( s , a ) / ( 1 + Y k ) ] ∈ F h \mathbb{E}_{a\sim\pi_h^{k-1}}[(2H+1)f_h(s,a)/(1+Y_k)]\in\mathcal{F}_h E a ∼ π h k − 1 [( 2 H + 1 ) f h ( s , a ) / ( 1 + Y k )] ∈ F h . The Representation Learning Guarantee bounds the squared error against E a ∼ π h k − 1 [ ⋅ ] \mathbb{E}_{a\sim\pi_h^{k-1}}[\cdot] E a ∼ π h k − 1 [ ⋅ ] ; we transfer it to E a ∼ π h ⋆ [ ⋅ ] \mathbb{E}_{a\sim\pi_h^\star}[\cdot] E a ∼ π h ⋆ [ ⋅ ] via uniform-action importance sampling: for any non-negative u ( s , a ) u(s,a) u ( s , a ) ,
E a ∼ π h ⋆ ( ⋅ ∣ s ) [ u ( s , a ) ] ≤ ∣ A ∣ ⋅ E a ∼ U ( A ) [ u ( s , a ) ] = ∣ A ∣ ⋅ E a ∼ π h k − 1 [ u ( s , a ) ⋅ 1 ∣ A ∣ π h k − 1 ( a ∣ s ) ] , \mathbb{E}_{a\sim\pi_h^\star(\cdot\mid s)}[u(s,a)]\le|\mathcal{A}|\cdot\mathbb{E}_{a\sim U(\mathcal{A})}[u(s,a)]=|\mathcal{A}|\cdot\mathbb{E}_{a\sim\pi_h^{k-1}}\left[u(s,a)\cdot\frac{1}{|\mathcal{A}|\pi_h^{k-1}(a\mid s)}\right], E a ∼ π h ⋆ ( ⋅ ∣ s ) [ u ( s , a )] ≤ ∣ A ∣ ⋅ E a ∼ U ( A ) [ u ( s , a )] = ∣ A ∣ ⋅ E a ∼ π h k − 1 [ u ( s , a ) ⋅ ∣ A ∣ π h k − 1 ( a ∣ s ) 1 ] ,
where the second equality holds whenever π h k − 1 \pi_h^{k-1} π h k − 1 has full support; for softmax policies, π h k − 1 ( a ∣ s ) ≥ exp ( − 2 α H ) / ∣ A ∣ \pi_h^{k-1}(a\mid s)\ge\exp(-2\alpha H)/|\mathcal{A}| π h k − 1 ( a ∣ s ) ≥ exp ( − 2 α H ) /∣ A ∣ , so the importance ratio is bounded. Combined with Cauchy–Schwarz, this yields
E ( s , a ) ∼ ρ h [ f h 2 ] ≤ ( 1 + Y k ) 2 ∣ A ∣ ζ k , E ( s , a ) ∼ β h [ f h 2 ] ≤ ( 1 + Y k ) 2 ∣ A ∣ ζ k , \mathbb{E}_{(s,a)\sim\rho_h}[f_h^2]\le(1+Y_k)^2|\mathcal{A}|\zeta_k,\qquad\mathbb{E}_{(s,a)\sim\beta_h}[f_h^2]\le(1+Y_k)^2|\mathcal{A}|\zeta_k, E ( s , a ) ∼ ρ h [ f h 2 ] ≤ ( 1 + Y k ) 2 ∣ A ∣ ζ k , E ( s , a ) ∼ β h [ f h 2 ] ≤ ( 1 + Y k ) 2 ∣ A ∣ ζ k ,
where the additional ∣ A ∣ |\mathcal{A}| ∣ A ∣ factor (relative to a direct ζ k \zeta_k ζ k bound) comes from the importance-sampling transfer between π ⋆ \pi^\star π ⋆ and π k − 1 \pi^{k-1} π k − 1 , and is absorbed into the ∣ A ∣ 3 / 2 |\mathcal{A}|^{3/2} ∣ A ∣ 3/2 scaling of the final rate.
Note that f h f_h f h is a signed quantity. Applying Lemma 10 to both g h : = ( 2 H + 1 ) f h g_h:=(2H+1)f_h g h := ( 2 H + 1 ) f h and g h : = − ( 2 H + 1 ) f h g_h:=-(2H+1)f_h g h := − ( 2 H + 1 ) f h yields a bound on ∣ ∑ h = 0 H − 1 E d P ^ k , h π ⋆ [ ( 2 H + 1 ) f h ] ∣ \left|\sum_{h=0}^{H-1}\mathbb{E}_{d_{\widehat{P}_k,h}^{\pi^\star}}[(2H+1)f_h]\right| ∑ h = 0 H − 1 E d P k , h π ⋆ [( 2 H + 1 ) f h ] :
∣ ∑ h = 0 H − 1 E ( s h , a h ) ∼ d P ^ k , h π ⋆ [ ( 2 H + 1 ) f h ( s h , a h ) ] ∣ ≤ ( 2 H + 1 ) ( 1 + Y k ) [ ∑ h = 1 H − 1 E [ ∥ ϕ h − 1 ∥ Σ ρ h − 1 , ϕ ^ h − 1 − 1 k ∣ A ∣ 3 ζ k + 4 H 2 λ k d + k ∣ A ∣ 2 ζ k + ∣ A ∣ 2 ζ k ] ] . \left|\sum_{h=0}^{H-1}\mathbb{E}_{(s_h,a_h)\sim d_{\widehat{P}_k,h}^{\pi^\star}}\left[(2H+1)f_h(s_h,a_h)\right]\right|
\le(2H+1)(1+Y_k)\left[\sum_{h=1}^{H-1}\mathbb{E}\left[\|\phi_{h-1}\|_{\Sigma_{\rho_{h-1},\widehat{\phi}_{h-1}}^{-1}}\sqrt{k|\mathcal{A}|^3\zeta_k+4H^2\lambda_kd+k|\mathcal{A}|^2\zeta_k}+\sqrt{|\mathcal{A}|^2\zeta_k}\right]\right]. h = 0 ∑ H − 1 E ( s h , a h ) ∼ d P k , h π ⋆ [ ( 2 H + 1 ) f h ( s h , a h ) ] ≤ ( 2 H + 1 ) ( 1 + Y k ) [ h = 1 ∑ H − 1 E [ ∥ ϕ h − 1 ∥ Σ ρ h − 1 , ϕ h − 1 − 1 k ∣ A ∣ 3 ζ k + 4 H 2 λ k d + k ∣ A ∣ 2 ζ k + ∣ A ∣ 2 ζ k ] ] .
The first sum (over h = 1 , … , H − 1 h=1,\ldots,H-1 h = 1 , … , H − 1 ) is the per-episode RepLearn residual that we denote ρ k o p t \rho_k^{\mathrm{opt}} ρ k opt :
ρ k o p t : = ( 2 H + 1 ) ( 1 + Y k ) ∑ h = 1 H − 1 E [ ∥ ϕ ^ h − 1 ∥ Σ ρ h − 1 , ϕ ^ h − 1 − 1 2 k ∣ A ∣ 3 ζ k + 4 H 2 λ k d ] . \rho_k^{\mathrm{opt}}:=(2H+1)(1+Y_k)\sum_{h=1}^{H-1}\mathbb{E}\left[\|\widehat{\phi}_{h-1}\|_{\Sigma_{\rho_{h-1},\widehat{\phi}_{h-1}}^{-1}}\sqrt{2k|\mathcal{A}|^3\zeta_k+4H^2\lambda_kd}\right]. ρ k opt := ( 2 H + 1 ) ( 1 + Y k ) h = 1 ∑ H − 1 E [ ∥ ϕ h − 1 ∥ Σ ρ h − 1 , ϕ h − 1 − 1 2 k ∣ A ∣ 3 ζ k + 4 H 2 λ k d ] .
The bonus b ^ h ( s , a ) = β ∥ ϕ ^ h ∥ Σ ρ h , ϕ ^ h − 1 \widehat{b}_h(s,a)=\beta\|\widehat{\phi}_h\|_{\Sigma_{\rho_h,\widehat{\phi}_h}^{-1}} b h ( s , a ) = β ∥ ϕ h ∥ Σ ρ h , ϕ h − 1 with β = C 1 d H ι \beta=C_1dH\sqrt{\iota} β = C 1 d H ι does not dominate ρ k o p t \rho_k^{\mathrm{opt}} ρ k opt per episode (since k ζ k = O ~ ( d k 1 / 4 ) \sqrt{k\zeta_k}=\widetilde{O}(dk^{1/4}) k ζ k = O ( d k 1/4 ) has a k k k -dependent factor). Instead, ρ k o p t \rho_k^{\mathrm{opt}} ρ k opt is carried as a residual into the cumulative representation error of Lemma 3.
To sum ρ k o p t \rho_k^{\mathrm{opt}} ρ k opt over k k k , the inner expectation is taken under the trajectory measure d P ^ , h − 1 π ⋆ d_{\hat{P},h-1}^{\pi^\star} d P ^ , h − 1 π ⋆ of the unexecuted optimal policy π ⋆ \pi^\star π ⋆ , while the Gram matrix Σ ρ h − 1 , ϕ ^ h − 1 \Sigma_{\rho_{h-1},\hat{\phi}_{h-1}} Σ ρ h − 1 , ϕ ^ h − 1 is built from data sampled under the executed policies π 0 , … , π k − 1 \pi^0,\ldots,\pi^{k-1} π 0 , … , π k − 1 . We bridge the two via uniform-action importance sampling: since the actions in ρ h − 1 \rho_{h-1} ρ h − 1 are uniform, for any non-negative function u ( s , a ) u(s,a) u ( s , a ) ,
E ( s , a ) ∼ d P ^ , h − 1 π ⋆ [ u ( s , a ) ] ≤ ∣ A ∣ ⋅ E s ∼ d P ^ , h − 1 π ⋆ , a ∼ U ( A ) [ u ( s , a ) ] ≤ ∣ A ∣ ⋅ E ( s , a ) ∼ ρ h − 1 π ⋆ [ u ( s , a ) ] . \mathbb{E}_{(s,a)\sim d_{\hat{P},h-1}^{\pi^\star}}[u(s,a)] \leq|\mathcal{A}|\cdot\mathbb{E}_{s\sim d_{\hat{P},h-1}^{\pi^\star},\,a\sim U(\mathcal{A})}[u(s,a)] \leq|\mathcal{A}|\cdot\mathbb{E}_{(s,a)\sim\rho_{h-1}^{\pi^\star}}[u(s,a)]. E ( s , a ) ∼ d P ^ , h − 1 π ⋆ [ u ( s , a )] ≤ ∣ A ∣ ⋅ E s ∼ d P ^ , h − 1 π ⋆ , a ∼ U ( A ) [ u ( s , a )] ≤ ∣ A ∣ ⋅ E ( s , a ) ∼ ρ h − 1 π ⋆ [ u ( s , a )] .
where ρ h − 1 π ⋆ \rho_{h-1}^{\pi^\star} ρ h − 1 π ⋆ is the π ⋆ \pi^\star π ⋆ -induced state distribution paired with uniform action. Combined with Cauchy–Schwarz applied to ∑ k ∥ ϕ ^ h − 1 ∥ Σ ρ h − 1 − 1 \sum_k \|\hat{\phi}_{h-1}\|_{\Sigma_{\rho_{h-1}}^{-1}} ∑ k ∥ ϕ ^ h − 1 ∥ Σ ρ h − 1 − 1 , this reduces the sum to an elliptical-potential expression on the realized data distribution, which is controlled by Lemma 22 :
∑ k = 1 K ρ k o p t ≤ ( 1 + ξ ) ∑ k = 1 K ϵ k = O ~ ( H 3 d 2 K 3 / 4 ∣ A ∣ 3 / 2 γ ) . \sum_{k=1}^{K}\rho_k^{\mathrm{opt}} \leq(1+\xi)\sum_{k=1}^{K}\epsilon_k = \widetilde{O}\left(\frac{H^3 d^2 K^{3/4}|\mathcal{A}|^{3/2}}{\gamma}\right). k = 1 ∑ K ρ k opt ≤ ( 1 + ξ ) k = 1 ∑ K ϵ k = O ( γ H 3 d 2 K 3/4 ∣ A ∣ 3/2 ) .
The lower bound − ( 1 + Y k ) ∣ A ∣ ζ k − ρ k o p t -(1+Y_k)\sqrt{|\mathcal{A}|\zeta_k}-\rho_k^{\mathrm{opt}} − ( 1 + Y k ) ∣ A ∣ ζ k − ρ k opt then comes from the residual Case-1 term plus the residual sum.
Remark 4. Operationally, Algorithm 3 computes a single bonus b ^ h ( s , a ) = β ∥ ϕ ^ h ∥ Σ ρ h , ϕ ^ h \hat{b}_h(s,a)=\beta\|\hat{\phi}_h\|_{\Sigma_{\rho_h,\hat{\phi}_h}} b ^ h ( s , a ) = β ∥ ϕ ^ h ∥ Σ ρ h , ϕ ^ h per step, which is added to both the reward and cost optimistic Q Q Q -functions (Q r , h k Q^k_{r,h} Q r , h k and Q g , h k Q^k_{g,h} Q g , h k ). The composite Q Q Q -function Q r , h k + Y k Q g , h k Q^k_{r,h}+Y_kQ^k_{g,h} Q r , h k + Y k Q g , h k therefore inherits a composite bonus ( 1 + Y k ) b ^ h = b ^ ( k ) (1+Y_k)\hat{b}_h=\hat{b}^{(k)} ( 1 + Y k ) b ^ h = b ^ ( k ) , exactly as used in Lemma 2. The bonus magnitude β = C 1 d H ι \beta=C_1dH\sqrt{\iota} β = C 1 d H ι is tuned to the LSVI-UCB concentration scale (Lemma 5) and is independent of k k k ; the RepLearn statistical residual is handled separately through ρ k o p t \rho_k^{\mathrm{opt}} ρ k opt above and absorbed into ∑ k ϵ k \sum_k\epsilon_k ∑ k ϵ k .
We next establish the softmax sub-optimality lemma for the composite planning problem, which is the actual MDP that the policy update step in Algorithm 1 approximately solves.
Lemma 12 (Composite softmax sub-optimality gap). Fix Y ≥ 0 Y\geq0 Y ≥ 0 and a finite-horizon planning problem with transition kernel P ^ \hat{P} P ^ , stage reward r ˉ : = r + Y g \bar{r}:=r+Yg r ˉ := r + Y g , additive bonus b ^ ( Y ) : = ( 1 + Y ) b ^ \hat{b}^{(Y)}:=(1+Y)\hat{b} b ^ ( Y ) := ( 1 + Y ) b ^ , and horizon H H H . Let { Q ˉ h ⋆ } h = 1 H \{\bar{Q}^{\star}_h\}_{h=1}^{H} { Q ˉ h ⋆ } h = 1 H be the optimal Q Q Q -functions under ( P ^ , r ˉ + b ^ ( Y ) , H ) (\hat{P},\bar{r}+\hat{b}^{(Y)},H) ( P ^ , r ˉ + b ^ ( Y ) , H ) , defined by
Q ˉ h ⋆ ( s , a ) : = r ˉ h ( s , a ) + b ^ ( Y ) ( s , a ) + E s ′ ∼ P ^ h ( ⋅ ∣ s , a ) [ V ˉ h + 1 ⋆ ( s ′ ) ] , \bar{Q}^{\star}_h(s,a):=\bar{r}_h(s,a)+\hat{b}^{(Y)}(s,a)+\mathbb{E}_{s'\sim\hat{P}_h(\cdot\mid s,a)}[\bar{V}^{\star}_{h+1}(s')], Q ˉ h ⋆ ( s , a ) := r ˉ h ( s , a ) + b ^ ( Y ) ( s , a ) + E s ′ ∼ P ^ h ( ⋅ ∣ s , a ) [ V ˉ h + 1 ⋆ ( s ′ )] ,
V ˉ h ⋆ ( s ) : = max a ∈ A Q ˉ h ⋆ ( s , a ) , V ˉ H + 1 ⋆ ( s ) ≡ 0. \bar{V}^{\star}_h(s):=\max_{a\in\mathcal{A}}\bar{Q}^{\star}_h(s,a),\qquad\bar{V}^{\star}_{H+1}(s)\equiv0. V ˉ h ⋆ ( s ) := a ∈ A max Q ˉ h ⋆ ( s , a ) , V ˉ H + 1 ⋆ ( s ) ≡ 0.
Define the softmax policy
π ^ h ( a ∣ s ) = exp ( α Q ˉ h ⋆ ( s , a ) ) ∑ a ′ ∈ A exp ( α Q ˉ h ⋆ ( s , a ′ ) ) . \hat{\pi}_h(a\mid s)=\frac{\exp(\alpha\bar{Q}^{\star}_h(s,a))}{\sum_{a'\in\mathcal{A}}\exp(\alpha\bar{Q}^{\star}_h(s,a'))}. π ^ h ( a ∣ s ) = ∑ a ′ ∈ A exp ( α Q ˉ h ⋆ ( s , a ′ )) exp ( α Q ˉ h ⋆ ( s , a )) .
Then for any start state s 1 s_1 s 1 and any policy π \pi π ,
V P ^ , r ˉ + b ^ ( Y ) π ( s 1 ) ≤ V P ^ , r ˉ + b ^ ( Y ) π ^ ( s 1 ) + H log ∣ A ∣ α . V^{\pi}_{\hat{P},\bar{r}+\hat{b}^{(Y)}}(s_1)\leq V^{\hat{\pi}}_{\hat{P},\bar{r}+\hat{b}^{(Y)}}(s_1)+\frac{H\log|\mathcal{A}|}{\alpha}. V P ^ , r ˉ + b ^ ( Y ) π ( s 1 ) ≤ V P ^ , r ˉ + b ^ ( Y ) π ^ ( s 1 ) + α H log ∣ A ∣ .
Proof. Identical to the proof of the standard softmax sub-optimality bound (e.g., the unconstrained version), with the composite stage reward r ˉ + b ^ ( Y ) \bar{r}+\hat{b}^{(Y)} r ˉ + b ^ ( Y ) replacing the original reward. We prove by backward induction that for all h h h and s s s ,
V ˉ h ⋆ ( s ) ≤ V h , P ^ , r ˉ + b ^ ( Y ) π ^ ( s ) + ( H − h + 1 ) log ∣ A ∣ α . \bar{V}^{\star}_h(s)\leq V^{\hat{\pi}}_{h,\hat{P},\bar{r}+\hat{b}^{(Y)}}(s)+\frac{(H-h+1)\log|\mathcal{A}|}{\alpha}. V ˉ h ⋆ ( s ) ≤ V h , P ^ , r ˉ + b ^ ( Y ) π ^ ( s ) + α ( H − h + 1 ) log ∣ A ∣ .
The base h = H + 1 h=H+1 h = H + 1 is trivial. For the inductive step, the log-sum-exp inequality gives max a Q ˉ h ⋆ ( s , a ) ≤ E a ∼ π ^ h [ Q ˉ h ⋆ ( s , a ) ] + log ∣ A ∣ α \max_{a}\bar{Q}^{\star}_h(s,a)\leq\mathbb{E}_{a\sim\hat{\pi}_h}[\bar{Q}^{\star}_h(s,a)]+\frac{\log|\mathcal{A}|}{\alpha} max a Q ˉ h ⋆ ( s , a ) ≤ E a ∼ π ^ h [ Q ˉ h ⋆ ( s , a )] + α l o g ∣ A ∣ , which combined with the inductive hypothesis on V ˉ h + 1 ⋆ \bar{V}^{\star}_{h+1} V ˉ h + 1 ⋆ yields the claim. Setting h = 1 h=1 h = 1 and using V P ^ , r ˉ + b ^ ( Y ) π ^ ≤ V ˉ 1 ⋆ V^{\hat{\pi}}_{\hat{P},\bar{r}+\hat{b}^{(Y)}}\leq\bar{V}^{\star}_1 V P ^ , r ˉ + b ^ ( Y ) π ^ ≤ V ˉ 1 ⋆ completes the proof.
Remark 5 (Why composite, not reward-only). The policy π k \pi^k π k produced by Algorithm 3 is the softmax of Q r , h k + Y k Q g , h k Q^k_{r,h}+Y_kQ^k_{g,h} Q r , h k + Y k Q g , h k , i.e. the LSVI-UCB estimate of the optimal Q Q Q -function for the composite planning MDP ( P ^ , r ˉ ( k ) + b ^ ( k ) ) (\hat{P},\bar{r}^{(k)}+\hat{b}^{(k)}) ( P ^ , r ˉ ( k ) + b ^ ( k ) ) . It is not the softmax of the optimal Q Q Q -function for ( P ^ , r + b ^ ) (\hat{P},r+\hat{b}) ( P ^ , r + b ^ ) alone. Consequently, applying Lemma 12 to the reward-only objective would not be justified, because π k \pi^k π k is not constructed from the reward-only optimal Q Q Q . The composite formulation above resolves this issue: π k \pi^k π k is (an approximation to) the softmax of the composite optimal Q Q Q , so Lemma 12 applies directly with Y = Y k Y=Y_k Y = Y k .
We now bound the composite Q Q Q -function gap on the true MDP, which is the central result of this section. Following the convention of Section 2, we state the result in cumulative form, since this is the form actually used downstream and the form genuinely produced by the elliptical-potential argument.
Lemma 3 (Restated; composite policy gap, cumulative form ). Conditioned on the Representation Learning Guarantee, with probability at least 1 − δ 1-\delta 1 − δ ,
∑ k = 1 K [ Q P ⋆ , r ˉ ( k ) π ⋆ , Y k ( s 1 , ⋅ ) − Q P ⋆ , r ˉ ( k ) π k , Y k ( s 1 , ⋅ ) ] ≤ ( 1 + ξ ) ∑ k = 1 K ϵ k + K H log ∣ A ∣ α , \sum_{k=1}^{K}\left[Q^{\pi^\star,Y_k}_{P^\star,\bar r^{(k)}}(s_1,\cdot)-Q^{\pi^k,Y_k}_{P^\star,\bar r^{(k)}}(s_1,\cdot)\right]\leq(1+\xi)\sum_{k=1}^{K}\epsilon_k+\frac{KH\log|\mathcal{A}|}{\alpha}, k = 1 ∑ K [ Q P ⋆ , r ˉ ( k ) π ⋆ , Y k ( s 1 , ⋅ ) − Q P ⋆ , r ˉ ( k ) π k , Y k ( s 1 , ⋅ ) ] ≤ ( 1 + ξ ) k = 1 ∑ K ϵ k + α K H log ∣ A ∣ , where the cumulative representation error satisfies
∑ k = 1 K ϵ k ≤ C H 2 d 2 K 3 / 4 ∣ A ∣ 3 / 2 log 1 / 4 ( d K ∣ Φ ∣ / δ ) . \sum_{k=1}^{K}\epsilon_k\leq C H^2d^2K^{3/4}|\mathcal{A}|^{3/2}\log^{1/4}(dK|\Phi|/\delta). k = 1 ∑ K ϵ k ≤ C H 2 d 2 K 3/4 ∣ A ∣ 3/2 log 1/4 ( d K ∣Φ∣/ δ ) .
Proof. Throughout, write r ˉ : = r ˉ ( k ) = r + Y k g \bar r:=\bar r^{(k)}=r+Y_kg r ˉ := r ˉ ( k ) = r + Y k g and b ^ ( k ) = ( 1 + Y k ) b ^ \hat b^{(k)}=(1+Y_k)\hat b b ^ ( k ) = ( 1 + Y k ) b ^ . We chain three inequalities at the per-episode level, then sum.
Step 1: Composite optimism. By Lemma 2 applied with the composite reward,
Q P ⋆ , r ˉ π ⋆ , Y k ( s 1 , a ) ≤ Q P ^ , r ˉ + b ^ ( k ) π ⋆ , Y k ( s 1 , a ) + ( 1 + Y k ) ∣ A ∣ ζ k + ρ k o p t , Q^{\pi^\star,Y_k}_{P^\star,\bar r}(s_1,a)\leq Q^{\pi^\star,Y_k}_{\hat P,\bar r+\hat b^{(k)}}(s_1,a)+(1+Y_k)\sqrt{|\mathcal{A}|}\zeta_k+\rho_k^{\mathrm{opt}}, Q P ⋆ , r ˉ π ⋆ , Y k ( s 1 , a ) ≤ Q P ^ , r ˉ + b ^ ( k ) π ⋆ , Y k ( s 1 , a ) + ( 1 + Y k ) ∣ A ∣ ζ k + ρ k opt ,
where the residual ρ k o p t \rho_k^{\mathrm{opt}} ρ k opt enters ∑ k ϵ k \sum_k\epsilon_k ∑ k ϵ k in the cumulative bound below.
Step 2: Composite softmax sub-optimality. By Lemma 12 applied with Y = Y k Y=Y_k Y = Y k , since π k \pi^k π k is (the LSVI-UCB approximation to) the softmax of the optimal composite Q Q Q -function under ( P ^ , r ˉ + b ^ ( k ) ) (\hat P,\bar r+\hat b^{(k)}) ( P ^ , r ˉ + b ^ ( k ) ) ,
Q P ^ , r ˉ + b ^ ( k ) π ⋆ , Y k ( s 1 , a ) ≤ Q P ^ , r ˉ + b ^ ( k ) π k , Y k ( s 1 , a ) + H log ∣ A ∣ α . Q^{\pi^\star,Y_k}_{\hat P,\bar r+\hat b^{(k)}}(s_1,a)\leq Q^{\pi^k,Y_k}_{\hat P,\bar r+\hat b^{(k)}}(s_1,a)+\frac{H\log|\mathcal{A}|}{\alpha}. Q P ^ , r ˉ + b ^ ( k ) π ⋆ , Y k ( s 1 , a ) ≤ Q P ^ , r ˉ + b ^ ( k ) π k , Y k ( s 1 , a ) + α H log ∣ A ∣ .
Step 3: From planning model P ^ \hat P P ^ back to true model P ⋆ P^\star P ⋆ . Combining Steps 1–2 and using the simulation lemma (Lemma 19 ) on the auxiliary MDP with kernel P ^ \hat P P ^ , reward r ˉ \bar r r ˉ , and bonus b ^ ( k ) \hat b^{(k)} b ^ ( k ) ,
Q P ⋆ , r ˉ π ⋆ , Y k ( s 1 , a ) − Q P ⋆ , r ˉ π k , Y k ( s 1 , a ) ≤ ∑ h = 0 H − 1 E ( s h , a h ) ∼ d P ⋆ , h π k [ b ^ ( k ) ( s h , a h ) + E s ′ ∼ P ^ h ( ⋅ ∣ s h , a h ) [ V P ^ , r ˉ + b ^ ( k ) , h + 1 π k , Y k ( s ′ ) ] − E s ′ ∼ P h ⋆ ( ⋅ ∣ s h , a h ) [ V P ^ , r ˉ + b ^ ( k ) , h + 1 π k , Y k ( s ′ ) ] ] + ( 1 + Y k ) ∣ A ∣ ζ k + ρ k o p t + H log ∣ A ∣ α . (16) \begin{aligned}
Q^{\pi^\star,Y_k}_{P^\star,\bar r}(s_1,a)-Q^{\pi^k,Y_k}_{P^\star,\bar r}(s_1,a)
&\leq\sum_{h=0}^{H-1}\mathbb{E}_{(s_h,a_h)\sim d^{\pi^k}_{P^\star,h}}\left[\hat b^{(k)}(s_h,a_h)+\mathbb{E}_{s'\sim\hat P_h(\cdot\mid s_h,a_h)}[V^{\pi^k,Y_k}_{\hat P,\bar r+\hat b^{(k)},h+1}(s')] \right.\\
&\qquad\left.{}-\mathbb{E}_{s'\sim P^\star_h(\cdot\mid s_h,a_h)}[V^{\pi^k,Y_k}_{\hat P,\bar r+\hat b^{(k)},h+1}(s')]\right]\\
&\qquad+(1+Y_k)\sqrt{|\mathcal{A}|}\zeta_k+\rho_k^{\mathrm{opt}}+\frac{H\log|\mathcal{A}|}{\alpha}. \tag*{(16)}
\end{aligned} Q P ⋆ , r ˉ π ⋆ , Y k ( s 1 , a ) − Q P ⋆ , r ˉ π k , Y k ( s 1 , a ) ≤ h = 0 ∑ H − 1 E ( s h , a h ) ∼ d P ⋆ , h π k [ b ^ ( k ) ( s h , a h ) + E s ′ ∼ P ^ h ( ⋅ ∣ s h , a h ) [ V P ^ , r ˉ + b ^ ( k ) , h + 1 π k , Y k ( s ′ )] − E s ′ ∼ P h ⋆ ( ⋅ ∣ s h , a h ) [ V P ^ , r ˉ + b ^ ( k ) , h + 1 π k , Y k ( s ′ )] ] + ( 1 + Y k ) ∣ A ∣ ζ k + ρ k opt + α H log ∣ A ∣ . (16)
Define
f h ( s , a ) : = E s ′ ∼ P ^ h ( ⋅ ∣ s , a ) [ V P ^ , r ˉ + b ^ ( k ) , h + 1 π k , Y k ( s ′ ) ] − E s ′ ∼ P h ⋆ ( ⋅ ∣ s , a ) [ V P ^ , r ˉ + b ^ ( k ) , h + 1 π k , Y k ( s ′ ) ] . f_h(s,a):=\mathbb{E}_{s'\sim\hat P_h(\cdot\mid s,a)}[V^{\pi^k,Y_k}_{\hat P,\bar r+\hat b^{(k)},h+1}(s')]-\mathbb{E}_{s'\sim P^\star_h(\cdot\mid s,a)}[V^{\pi^k,Y_k}_{\hat P,\bar r+\hat b^{(k)},h+1}(s')]. f h ( s , a ) := E s ′ ∼ P ^ h ( ⋅ ∣ s , a ) [ V P ^ , r ˉ + b ^ ( k ) , h + 1 π k , Y k ( s ′ )] − E s ′ ∼ P h ⋆ ( ⋅ ∣ s , a ) [ V P ^ , r ˉ + b ^ ( k ) , h + 1 π k , Y k ( s ′ )] .
The composite value V P ^ , r ˉ + b ^ ( k ) , h + 1 π k , Y k V^{\pi^k,Y_k}_{\hat P,\bar r+\hat b^{(k)},h+1} V P ^ , r ˉ + b ^ ( k ) , h + 1 π k , Y k is bounded in [ 0 , ( 1 + Y k ) ( 2 H + 1 ) H ] [0,(1+Y_k)(2H+1)H] [ 0 , ( 1 + Y k ) ( 2 H + 1 ) H ] , so 1 ( 2 H + 1 ) ( 1 + Y k ) f h \frac{1}{(2H+1)(1+Y_k)}f_h ( 2 H + 1 ) ( 1 + Y k ) 1 f h matches the linear part of an element of G h \mathcal{G}_h G h up to a multiplicative ( 1 + Y k ) (1+Y_k) ( 1 + Y k ) factor; the softmax operator and UCB bonus appearing in V P ^ , r ˉ + b ^ ( k ) , h + 1 π k , Y k V^{\pi^k,Y_k}_{\hat P,\bar r+\hat b^{(k)},h+1} V P ^ , r ˉ + b ^ ( k ) , h + 1 π k , Y k are not in G h \mathcal{G}_h G h , but their contribution is independently bounded as H log ∣ A ∣ / α H\log|\mathcal{A}|/\alpha H log ∣ A ∣/ α via Lemma 12 and as the LSVI-UCB residual l s v i K ≤ 1 4 ( 1 + ξ ) ∑ k ϵ k \mathrm{lsvi}_K\leq\frac{1}{4}(1+\xi)\sum_k\epsilon_k lsvi K ≤ 4 1 ( 1 + ξ ) ∑ k ϵ k via Lemma 13 , both of which we already track in the master bound. Hence by the Representation Learning Guarantee applied to the linear part of f h f_h f h ,
E ( s , a ) ∼ ρ h [ f h 2 ] ≤ ( 2 H + 1 ) 2 ( 1 + Y k ) 2 ζ k , E ( s , a ) ∼ β h [ f h 2 ] ≤ ( 2 H + 1 ) 2 ( 1 + Y k ) 2 ζ k . \mathbb{E}_{(s,a)\sim\rho_h}[f_h^2]\leq(2H+1)^2(1+Y_k)^2\zeta_k,\qquad\mathbb{E}_{(s,a)\sim\beta_h}[f_h^2]\leq(2H+1)^2(1+Y_k)^2\zeta_k. E ( s , a ) ∼ ρ h [ f h 2 ] ≤ ( 2 H + 1 ) 2 ( 1 + Y k ) 2 ζ k , E ( s , a ) ∼ β h [ f h 2 ] ≤ ( 2 H + 1 ) 2 ( 1 + Y k ) 2 ζ k .
We rewrite (16) as
Q P ⋆ , r ˉ π ⋆ , Y k ( s 1 , a ) − Q P ⋆ , r ˉ π k , Y k ( s 1 , a ) ≤ ∑ h = 0 H − 1 E d P ⋆ , h π k [ b ^ ( k ) ( s h , a h ) ] ⏟ (a) + ∑ h = 0 H − 1 E d P ⋆ , h π k [ f h ( s h , a h ) ] ⏟ (b) + ( 1 + Y k ) ∣ A ∣ ζ k + ρ k o p t + H log ∣ A ∣ α . (17) \begin{aligned}
Q^{\pi^\star,Y_k}_{P^\star,\bar r}(s_1,a)-Q^{\pi^k,Y_k}_{P^\star,\bar r}(s_1,a)
&\leq\underbrace{\sum_{h=0}^{H-1}\mathbb{E}_{d^{\pi^k}_{P^\star,h}}[\hat b^{(k)}(s_h,a_h)]}_{\text{(a)}}+\underbrace{\sum_{h=0}^{H-1}\mathbb{E}_{d^{\pi^k}_{P^\star,h}}[f_h(s_h,a_h)]}_{\text{(b)}}\\
&\qquad+(1+Y_k)\sqrt{|\mathcal{A}|}\zeta_k+\rho_k^{\mathrm{opt}}+\frac{H\log|\mathcal{A}|}{\alpha}. \tag*{(17)}
\end{aligned} Q P ⋆ , r ˉ π ⋆ , Y k ( s 1 , a ) − Q P ⋆ , r ˉ π k , Y k ( s 1 , a ) ≤ (a) h = 0 ∑ H − 1 E d P ⋆ , h π k [ b ^ ( k ) ( s h , a h )] + (b) h = 0 ∑ H − 1 E d P ⋆ , h π k [ f h ( s h , a h )] + ( 1 + Y k ) ∣ A ∣ ζ k + ρ k opt + α H log ∣ A ∣ . (17)
Bounding term (a). Applying Lemma 11 with g h = b ^ h ( k ) = ( 1 + Y k ) β ∥ ϕ ^ h ∥ Σ ρ h , ϕ ^ h − 1 g_h=\hat{b}^{(k)}_h=(1+Y_k)\beta\|\hat{\phi}_h\|_{\Sigma^{-1}_{\rho_h,\hat{\phi}_h}} g h = b ^ h ( k ) = ( 1 + Y k ) β ∥ ϕ ^ h ∥ Σ ρ h , ϕ ^ h − 1 and B = O ( ( 1 + Y k ) β d ) B=O((1+Y_k)\beta\sqrt{d}) B = O (( 1 + Y k ) β d ) ,
( a ) ≤ ∑ h = 1 H − 1 E d P ⋆ , h − 1 π k ⋆ [ ∥ ϕ h − 1 ⋆ ∥ Σ γ h − 1 , ϕ h − 1 ⋆ − 1 k ∣ A ∣ E ρ h [ ( b ^ h ( k ) ) 2 ] + ( 1 + Y k ) 4 H 2 λ k d + ∣ A ∣ E ρ 0 [ ( b ^ 0 ( k ) ) 2 ] ] . \mathrm{(a)}\leq\sum_{h=1}^{H-1}\mathbb{E}_{d^{\pi_k^\star}_{P^\star,h-1}}\left[\|\phi^\star_{h-1}\|_{\Sigma^{-1}_{\gamma_{h-1},\phi^\star_{h-1}}}\sqrt{k|\mathcal{A}|\mathbb{E}_{\rho_h}\left[(\hat{b}^{(k)}_h)^2\right]+(1+Y_k)^4H^2\lambda_kd}+\sqrt{|\mathcal{A}|}\mathbb{E}_{\rho_0}\left[(\hat{b}^{(k)}_0)^2\right]\right]. ( a ) ≤ h = 1 ∑ H − 1 E d P ⋆ , h − 1 π k ⋆ [ ∥ ϕ h − 1 ⋆ ∥ Σ γ h − 1 , ϕ h − 1 ⋆ − 1 k ∣ A ∣ E ρ h [ ( b ^ h ( k ) ) 2 ] + ( 1 + Y k ) 4 H 2 λ k d + ∣ A ∣ E ρ 0 [ ( b ^ 0 ( k ) ) 2 ] ] .
Using E ρ h [ ( b ^ h ( k ) ) 2 ] = ( 1 + Y k ) 2 β 2 E ρ h [ ∥ ϕ ^ h ∥ Σ ρ h , ϕ ^ h − 1 2 ] ≤ ( 1 + Y k ) 2 β 2 d / k \mathbb{E}_{\rho_h}\left[(\hat{b}^{(k)}_h)^2\right]=(1+Y_k)^2\beta^2\mathbb{E}_{\rho_h}\left[\|\hat{\phi}_h\|^2_{\Sigma^{-1}_{\rho_h,\hat{\phi}_h}}\right]\leq(1+Y_k)^2\beta^2d/k E ρ h [ ( b ^ h ( k ) ) 2 ] = ( 1 + Y k ) 2 β 2 E ρ h [ ∥ ϕ ^ h ∥ Σ ρ h , ϕ ^ h − 1 2 ] ≤ ( 1 + Y k ) 2 β 2 d / k (by the trace bound k E ρ h [ ∥ ϕ ^ ∥ Σ − 1 2 ] ≤ d k\mathbb{E}_{\rho_h}[\|\hat{\phi}\|^2_{\Sigma^{-1}}]\leq d k E ρ h [ ∥ ϕ ^ ∥ Σ − 1 2 ] ≤ d ),
( a ) ≤ ( 1 + Y k ) [ ∑ h = 1 H − 1 E d P ⋆ , h − 1 π k ⋆ [ ∥ ϕ h − 1 ⋆ ∥ Σ γ h − 1 , ϕ h − 1 ⋆ − 1 ∣ A ∣ β 2 d + 4 H 2 λ k d ] + ∣ A ∣ β 2 d / k ] . \mathrm{(a)}\leq(1+Y_k)\left[\sum_{h=1}^{H-1}\mathbb{E}_{d^{\pi_k^\star}_{P^\star,h-1}}\left[\|\phi^\star_{h-1}\|_{\Sigma^{-1}_{\gamma_{h-1},\phi^\star_{h-1}}}\sqrt{|\mathcal{A}|\beta^2d+4H^2\lambda_kd}\right]+\sqrt{|\mathcal{A}|\beta^2d/k}\right]. ( a ) ≤ ( 1 + Y k ) [ h = 1 ∑ H − 1 E d P ⋆ , h − 1 π k ⋆ [ ∥ ϕ h − 1 ⋆ ∥ Σ γ h − 1 , ϕ h − 1 ⋆ − 1 ∣ A ∣ β 2 d + 4 H 2 λ k d ] + ∣ A ∣ β 2 d / k ] .
Bounding term (b). Applying Lemma 11 with g h = f h g_h=f_h g h = f h and B = O ( ( 1 + Y k ) H ) B=O((1+Y_k)H) B = O (( 1 + Y k ) H ) ,
( b ) ≤ ( 1 + Y k ) [ ∑ h = 1 H − 1 E d P ⋆ , h − 1 π k ⋆ [ ∥ ϕ h − 1 ⋆ ∥ Σ γ h − 1 , ϕ h − 1 ⋆ − 1 k ∣ A ∣ ( 2 H + 1 ) 2 ζ k + 4 H 2 λ k d ] + ∣ A ∣ ( 2 H + 1 ) 2 ζ k ] . \mathrm{(b)}\leq(1+Y_k)\left[\sum_{h=1}^{H-1}\mathbb{E}_{d^{\pi_k^\star}_{P^\star,h-1}}\left[\|\phi^\star_{h-1}\|_{\Sigma^{-1}_{\gamma_{h-1},\phi^\star_{h-1}}}\sqrt{k|\mathcal{A}|(2H+1)^2\zeta_k+4H^2\lambda_kd}\right]+\sqrt{|\mathcal{A}|(2H+1)^2\zeta_k}\right]. ( b ) ≤ ( 1 + Y k ) [ h = 1 ∑ H − 1 E d P ⋆ , h − 1 π k ⋆ [ ∥ ϕ h − 1 ⋆ ∥ Σ γ h − 1 , ϕ h − 1 ⋆ − 1 k ∣ A ∣ ( 2 H + 1 ) 2 ζ k + 4 H 2 λ k d ] + ∣ A ∣ ( 2 H + 1 ) 2 ζ k ] .
Summing over k k k via the elliptical potential lemma. Using γ h k ( s , a ) = 1 k ∑ i = 0 k − 1 d h π i ( s , a ) \gamma^k_h(s,a)=\frac{1}{k}\sum_{i=0}^{k-1}d^{\pi^i}_h(s,a) γ h k ( s , a ) = k 1 ∑ i = 0 k − 1 d h π i ( s , a ) and the executed-policy mixture d P ⋆ , h − 1 π k ⋆ d^{\pi_k^\star}_{P^\star,h-1} d P ⋆ , h − 1 π k ⋆ on the LHS, we apply uniform-action importance sampling to bound
E ( s , a ) ∼ d P ⋆ , h − 1 π k ⋆ [ ∥ ϕ h − 1 ⋆ ∥ Σ γ h − 1 k , ϕ h − 1 ⋆ − 1 ] ≤ ∣ A ∣ ⋅ E ( s , a ) ∼ γ h − 1 k [ ∥ ϕ h − 1 ⋆ ∥ Σ γ h − 1 k , ϕ h − 1 ⋆ − 1 ] , \mathbb{E}_{(s,a)\sim d^{\pi_k^\star}_{P^\star,h-1}}\left[\|\phi^\star_{h-1}\|_{\Sigma^{-1}_{\gamma^k_{h-1},\phi^\star_{h-1}}}\right]\leq\sqrt{|\mathcal{A}|}\cdot\mathbb{E}_{(s,a)\sim\gamma^k_{h-1}}\left[\|\phi^\star_{h-1}\|_{\Sigma^{-1}_{\gamma^k_{h-1},\phi^\star_{h-1}}}\right], E ( s , a ) ∼ d P ⋆ , h − 1 π k ⋆ [ ∥ ϕ h − 1 ⋆ ∥ Σ γ h − 1 k , ϕ h − 1 ⋆ − 1 ] ≤ ∣ A ∣ ⋅ E ( s , a ) ∼ γ h − 1 k [ ∥ ϕ h − 1 ⋆ ∥ Σ γ h − 1 k , ϕ h − 1 ⋆ − 1 ] ,
since the actions in γ h − 1 k \gamma^k_{h-1} γ h − 1 k follow the executed policies’ marginals (paired with the on-policy state distribution) and the importance ratio between d π k ⋆ d^{\pi_k^\star} d π k ⋆ and γ h − 1 k \gamma^k_{h-1} γ h − 1 k on the action variable is bounded by ∣ A ∣ |\mathcal{A}| ∣ A ∣ . Then
∑ k = 1 K E γ h − 1 k [ ∥ ϕ h − 1 ⋆ ∥ Σ γ h − 1 k , ϕ h − 1 ⋆ − 1 ] ≤ K ∑ k = 1 K E γ h − 1 k [ ( ϕ h − 1 ⋆ ) ⊤ Σ γ h − 1 k , ϕ h − 1 ⋆ − 1 ϕ h − 1 ⋆ ] ≤ d K ln ( 1 + K λ 1 ) , \sum_{k=1}^{K}\mathbb{E}_{\gamma^k_{h-1}}\left[\|\phi^\star_{h-1}\|_{\Sigma^{-1}_{\gamma^k_{h-1},\phi^\star_{h-1}}}\right]\leq\sqrt{K\sum_{k=1}^{K}\mathbb{E}_{\gamma^k_{h-1}}\left[(\phi^\star_{h-1})^\top\Sigma^{-1}_{\gamma^k_{h-1},\phi^\star_{h-1}}\phi^\star_{h-1}\right]}
\leq\sqrt{dK\ln\left(1+\frac{K}{\lambda_1}\right)}, k = 1 ∑ K E γ h − 1 k [ ∥ ϕ h − 1 ⋆ ∥ Σ γ h − 1 k , ϕ h − 1 ⋆ − 1 ] ≤ K k = 1 ∑ K E γ h − 1 k [ ( ϕ h − 1 ⋆ ) ⊤ Σ γ h − 1 k , ϕ h − 1 ⋆ − 1 ϕ h − 1 ⋆ ] ≤ d K ln ( 1 + λ 1 K ) ,
where the second inequality uses Lemma 22 (matrix determinant lemma) on the matched data distribution. The ∣ A ∣ \sqrt{|\mathcal{A}|} ∣ A ∣ factor from importance sampling combines with the ∣ A ∣ \sqrt{|\mathcal{A}|} ∣ A ∣ inside the integrand to give the ∣ A ∣ 3 / 2 |\mathcal{A}|^{3/2} ∣ A ∣ 3/2 scaling of the final bound.
Combining. Plugging in ζ k = O ( d 2 log ( k ∣ Φ ∣ / δ ) / k ) \zeta_k=O(d^2\sqrt{\log(k|\Phi|/\delta)}/k) ζ k = O ( d 2 log ( k ∣Φ∣/ δ ) / k ) and λ k = O ( d log ( ∣ Φ ∣ k / δ ) ) \lambda_k=O(d\log(|\Phi|k/\delta)) λ k = O ( d log ( ∣Φ∣ k / δ )) , summing terms (a) + (b) + ρ k o p t \rho_k^{\mathrm{opt}} ρ k opt over k k k , and pulling the ( 1 + Y k ) (1+Y_k) ( 1 + Y k ) factor out (which is bounded by ( 1 + ξ ) (1+\xi) ( 1 + ξ ) ), we obtain
∑ k = 1 K [ Q P ⋆ , π ˉ π ⋆ , Y k ( s 1 , a ) − Q P ⋆ , π ˉ π k , Y k ( s 1 , a ) ] ≤ ( 1 + ξ ) ∑ k = 1 K ϵ k + K H log ∣ A ∣ α , \sum_{k=1}^{K}\left[Q^{\pi^\star,Y_k}_{P^\star,\bar{\pi}}(s_1,a)-Q^{\pi_k,Y_k}_{P^\star,\bar{\pi}}(s_1,a)\right]\leq(1+\xi)\sum_{k=1}^{K}\epsilon_k+\frac{KH\log|\mathcal{A}|}{\alpha}, k = 1 ∑ K [ Q P ⋆ , π ˉ π ⋆ , Y k ( s 1 , a ) − Q P ⋆ , π ˉ π k , Y k ( s 1 , a ) ] ≤ ( 1 + ξ ) k = 1 ∑ K ϵ k + α K H log ∣ A ∣ ,
where the cumulative representation error satisfies
∑ k = 1 K ϵ k ≤ C H 2 d 3 / 2 K ∣ A ∣ 1 / 2 log ( d K ∣ Φ ∣ / δ ) + C ′ H 2 d 2 K 3 / 4 ∣ A ∣ 3 / 2 log 1 / 4 ( d K ∣ Φ ∣ / δ ) . \sum_{k=1}^{K}\epsilon_k\leq CH^2d^{3/2}\sqrt{K}|\mathcal{A}|^{1/2}\sqrt{\log(dK|\Phi|/\delta)}+C'H^2d^2K^{3/4}|\mathcal{A}|^{3/2}\log^{1/4}(dK|\Phi|/\delta). k = 1 ∑ K ϵ k ≤ C H 2 d 3/2 K ∣ A ∣ 1/2 log ( d K ∣Φ∣/ δ ) + C ′ H 2 d 2 K 3/4 ∣ A ∣ 3/2 log 1/4 ( d K ∣Φ∣/ δ ) .
The dominant term is the second one, of order O ~ ( H 2 d 2 K 3 / 4 ∣ A ∣ 3 / 2 ) \widetilde{O}(H^2d^2K^{3/4}|\mathcal{A}|^{3/2}) O ( H 2 d 2 K 3/4 ∣ A ∣ 3/2 ) , completing the proof.
Remark 6 (Per-episode rate). The cumulative bound ∑ k ϵ k = O ( K 3 / 4 ) \sum_k \epsilon_k = O(K^{3/4}) ∑ k ϵ k = O ( K 3/4 ) corresponds to an average per-episode rate ϵ ˉ K = 1 K ∑ k ϵ k = O ~ ( H 2 d 2 K − 1 / 4 ∣ A ∣ 3 / 2 ) \bar{\epsilon}_K = \frac{1}{K}\sum_k \epsilon_k = \widetilde{O}(H^2d^2K^{-1/4}|\mathcal{A}|^{3/2}) ϵ ˉ K = K 1 ∑ k ϵ k = O ( H 2 d 2 K − 1/4 ∣ A ∣ 3/2 ) . Throughout the analysis, only the cumulative quantity ( 1 + ξ ) ∑ k ϵ k (1+\xi)\sum_k \epsilon_k ( 1 + ξ ) ∑ k ϵ k is used (e.g., in Lemma 4 and the master inequality), so we work directly with the cumulative form.
We now establish an auxiliary lemma that quantifies how close the LSVI-UCB output V 1 ( k ) , Y k ( s 1 ) V_1^{(k),Y_k}(s_1) V 1 ( k ) , Y k ( s 1 ) is to the true Bellman-optimal value V ˉ 1 ⋆ ( s 1 ) \bar{V}_1^\star(s_1) V ˉ 1 ⋆ ( s 1 ) on the planning model ( P ^ , r ˉ ( k ) + b ^ ( k ) ) (\widehat{P},\bar{r}^{(k)}+\hat{b}^{(k)}) ( P , r ˉ ( k ) + b ^ ( k ) ) . This is the LSVI-UCB analog of the per-step backup bound (Lemma 6), but applied at the start state s 1 s_1 s 1 and going from the LSVI iterate to the planning-model optimal value, rather than the other direction.
Lemma 13 (LSVI-UCB output vs. planning-model optimum). On the event E \mathcal{E} E of Lemma 5, for every episode k ∈ [ K ] k \in[K] k ∈ [ K ] and every s ∈ S s \in\mathcal{S} s ∈ S ,
∣ V ˉ 1 ⋆ ( s ) − V ˉ 1 ( k ) , Y k ( s ) ∣ ≤ ∑ h = 1 H E ( s h , a h ) ∼ d P ^ , h π ~ k [ ( 1 + Y k ) C 3 d H χ ∥ ϕ ^ h k ( s h , a h ) ∥ ( Σ h k ) − 1 ∣ s 1 = s ] , \left|\bar{V}_1^\star(s)-\bar{V}_1^{(k),Y_k}(s)\right|
\le\sum_{h=1}^{H}\mathbb{E}_{(s_h,a_h)\sim d_{\widehat{P},h}^{\tilde{\pi}_k}}\left[\left.\left(1+Y_k\right)C_3dH\sqrt{\chi}\left\|\hat{\phi}_h^k(s_h,a_h)\right\|_{(\Sigma_h^k)^{-1}}\right|s_1=s\right], V ˉ 1 ⋆ ( s ) − V ˉ 1 ( k ) , Y k ( s ) ≤ h = 1 ∑ H E ( s h , a h ) ∼ d P , h π ~ k [ ( 1 + Y k ) C 3 d H χ ϕ ^ h k ( s h , a h ) ( Σ h k ) − 1 s 1 = s ] , where V ˉ h ( k ) , Y k ( s ) : = max a Q h ( k ) , Y k ( s , a ) \bar{V}_h^{(k),Y_k}(s):=\max_a Q_h^{(k),Y_k}(s,a) V ˉ h ( k ) , Y k ( s ) := max a Q h ( k ) , Y k ( s , a ) is the max-version of the LSVI iterate, π ~ k \tilde{\pi}^k π ~ k is an optimal policy of the planning model P ^ \widehat{P} P with stage reward ( 1 + Y k ) C 3 d H χ ∥ ϕ ^ h k ( s , a ) ∥ ( Σ h k ) − 1 (1+Y_k)C_3dH\sqrt{\chi}\left\|\hat{\phi}_h^k(s,a)\right\|_{(\Sigma_h^k)^{-1}} ( 1 + Y k ) C 3 d H χ ϕ ^ h k ( s , a ) ( Σ h k ) − 1 , and C 3 , χ C_3,\chi C 3 , χ are as in Lemma 6. Consequently, applying Lemma 11 with the bonus integrand and summing over k ∈ [ K ] k \in[K] k ∈ [ K ] via the elliptical-potential lemma,
lsvi K : = ∑ k = 1 K ∣ V ˉ 1 ⋆ ( s 1 ) − V ˉ 1 ( k ) , Y k ( s 1 ) ∣ ≤ O ~ ( d 3 / 2 H 2 K ∣ A ∣ 1 / 2 γ ) ≤ 1 4 ( 1 + ξ ) ∑ k = 1 K ϵ k . \operatorname{lsvi}_K := \sum_{k=1}^{K}\left|\bar{V}_1^\star(s_1)-\bar{V}_1^{(k),Y_k}(s_1)\right|
\le\widetilde{O}\left(\frac{d^{3/2}H^2\sqrt{K}|\mathcal{A}|^{1/2}}{\gamma}\right)
\le\frac{1}{4}(1+\xi)\sum_{k=1}^{K}\epsilon_k. lsvi K := k = 1 ∑ K V ˉ 1 ⋆ ( s 1 ) − V ˉ 1 ( k ) , Y k ( s 1 ) ≤ O ( γ d 3/2 H 2 K ∣ A ∣ 1/2 ) ≤ 4 1 ( 1 + ξ ) k = 1 ∑ K ϵ k . The bound also holds for the softmax variant V ˉ 1 ( k ) , Y k ( s ) = ∑ a π k ( a ∣ s ) Q 1 ( k ) , Y k ( s , a ) \bar{V}_1^{(k),Y_k}(s)=\sum_a\pi^k(a|s)Q_1^{(k),Y_k}(s,a) V ˉ 1 ( k ) , Y k ( s ) = ∑ a π k ( a ∣ s ) Q 1 ( k ) , Y k ( s , a ) in place of V ˉ 1 ( k ) , Y k \bar{V}_1^{(k),Y_k} V ˉ 1 ( k ) , Y k , up to an additive H log ∣ A ∣ / α H\log|\mathcal{A}|/\alpha H log ∣ A ∣/ α term (softmax sub-optimality on the LSVI iterate itself).
Proof. We prove the per-state bound by backward induction on h h h , then derive the cumulative bound by elliptical-potential summation.
Step 1: Per-step LSVI concentration. By Lemma 5 (event E \mathcal{E} E ), for each j ∈ { r , g } j \in\{r,g\} j ∈ { r , g } , h ∈ [ H ] h \in[H] h ∈ [ H ] , and k ∈ [ K ] k \in[K] k ∈ [ K ] , the LSVI weight w j , h k w_{j,h}^k w j , h k satisfies, for every ( s , a ) ∈ S × A (s,a) \in\mathcal{S}\times\mathcal{A} ( s , a ) ∈ S × A :
∣ ⟨ ϕ ^ h k ( s , a ) , w j , h k ⟩ − ( c j , h ( s , a ) + E P ^ h [ V j , h + 1 k ] ( s , a ) ) ∣ ≤ C 3 d H χ ∥ ϕ ^ h k ( s , a ) ∥ ( Σ h k ) − 1 , (18) \left|\left\langle\hat{\phi}_h^k(s,a),w_{j,h}^k\right\rangle-\left(c_{j,h}(s,a)+\mathbb{E}_{\widehat{P}_h}\left[V_{j,h+1}^k\right](s,a)\right)\right|
\le C_3dH\sqrt{\chi}\left\|\hat{\phi}_h^k(s,a)\right\|_{(\Sigma_h^k)^{-1}}, \tag*{(18)} ⟨ ϕ ^ h k ( s , a ) , w j , h k ⟩ − ( c j , h ( s , a ) + E P h [ V j , h + 1 k ] ( s , a ) ) ≤ C 3 d H χ ϕ ^ h k ( s , a ) ( Σ h k ) − 1 , (18)
where C 3 = O ( λ + C 2 ) C_3=O(\sqrt{\lambda}+C_2) C 3 = O ( λ + C 2 ) absorbs the regularization and concentration constants from Lemma 6, and the kernel-mismatch contribution between P ⋆ P^\star P ⋆ and P ^ \widehat{P} P is bundled into C 3 C_3 C 3 via the Representation Learning Guarantee (Lemma 16 ). The reward c r , h = r h c_{r,h}=r_h c r , h = r h and cost c g , h = g h c_{g,h}=g_h c g , h = g h targets are deterministic.
Step 2: Composite LSVI concentration. Recall Q h ( k ) , Y k ( s , a ) = Q r , h k ( s , a ) + Y k Q g , h k ( s , a ) Q_h^{(k),Y_k}(s,a)=Q_{r,h}^k(s,a)+Y_kQ_{g,h}^k(s,a) Q h ( k ) , Y k ( s , a ) = Q r , h k ( s , a ) + Y k Q g , h k ( s , a ) where each Q j , h k Q_{j,h}^k Q j , h k is the LSVI iterate from Algorithm 3 (truncated by H H H but we ignore truncation here as it only tightens the bound). The composite stage reward is r ˉ h ( k ) = r h + Y k g h \bar{r}_h^{(k)}=r_h+Y_kg_h r ˉ h ( k ) = r h + Y k g h , and the composite bonus is b ^ h ( k ) = ( 1 + Y k ) β ^ h k \hat{b}_h^{(k)}=(1+Y_k)\hat{\beta}_h^k b ^ h ( k ) = ( 1 + Y k ) β ^ h k where β ^ h k ( s , a ) = β ∥ ϕ ^ h k ( s , a ) ∥ ( Σ h k ) − 1 \hat{\beta}_h^k(s,a)=\beta\left\|\hat{\phi}_h^k(s,a)\right\|_{(\Sigma_h^k)^{-1}} β ^ h k ( s , a ) = β ϕ ^ h k ( s , a ) ( Σ h k ) − 1 .
By Algorithm 3 lines 7–8, the LSVI iterate satisfies
Q j , h k ( s , a ) = ⟨ ϕ ^ h k ( s , a ) , w j , h k ⟩ + b ^ h k ( s , a ) , Q_{j,h}^k(s,a)=\left\langle\hat{\phi}_h^k(s,a),w_{j,h}^k\right\rangle+\hat{b}_h^k(s,a), Q j , h k ( s , a ) = ⟨ ϕ ^ h k ( s , a ) , w j , h k ⟩ + b ^ h k ( s , a ) ,
ignoring the truncation min { ⋅ , H } \min\{\cdot,H\} min { ⋅ , H } (which removes only over-estimates and does not affect the upper bound we derive). Combining with (18), weighted by 1 1 1 and Y k Y_k Y k for the reward and cost components respectively:
∣ Q h ( k ) , Y k ( s , a ) − [ r ˉ h ( k ) ( s , a ) + b ^ h ( k ) ( s , a ) + E P ^ h [ V r , h + 1 k + Y k V g , h + 1 k ] ( s , a ) ] ∣ ≤ ( 1 + Y k ) C 3 d H χ ∥ ϕ ^ h k ( s , a ) ∥ ( Σ h k ) − 1 . \left|Q_h^{(k),Y_k}(s,a)-\left[\bar{r}_h^{(k)}(s,a)+\hat{b}_h^{(k)}(s,a)+\mathbb{E}_{\widehat{P}_h}\left[V_{r,h+1}^k+Y_kV_{g,h+1}^k\right](s,a)\right]\right|
\le(1+Y_k)C_3dH\sqrt{\chi}\left\|\hat{\phi}_h^k(s,a)\right\|_{(\Sigma_h^k)^{-1}}. Q h ( k ) , Y k ( s , a ) − [ r ˉ h ( k ) ( s , a ) + b ^ h ( k ) ( s , a ) + E P h [ V r , h + 1 k + Y k V g , h + 1 k ] ( s , a ) ] ≤ ( 1 + Y k ) C 3 d H χ ϕ ^ h k ( s , a ) ( Σ h k ) − 1 .
Since V h + 1 ( k ) , Y k ( s ) : = V r , h + 1 k ( s ) + Y k V g , h + 1 k ( s ) V_{h+1}^{(k),Y_k}(s) := V_{r,h+1}^{k}(s) + Y_k V_{g,h+1}^{k}(s) V h + 1 ( k ) , Y k ( s ) := V r , h + 1 k ( s ) + Y k V g , h + 1 k ( s ) is the composite LSVI value (softmax form) and V ˉ h + 1 ( k ) , Y k ( s ) = max a Q h + 1 ( k ) , Y k ( s , a ) \bar{V}_{h+1}^{(k),Y_k}(s) = \max_{a} Q_{h+1}^{(k),Y_k}(s,a) V ˉ h + 1 ( k ) , Y k ( s ) = max a Q h + 1 ( k ) , Y k ( s , a ) is the max form, we work with the max form throughout this proof and handle the softmax-vs-max gap in Step 5. Replacing V r , h + 1 k + Y k V g , h + 1 k V_{r,h+1}^{k} + Y_k V_{g,h+1}^{k} V r , h + 1 k + Y k V g , h + 1 k by V ˉ h + 1 ( k ) , Y k \bar{V}_{h+1}^{(k),Y_k} V ˉ h + 1 ( k ) , Y k in the displayed inequality (which holds with the same RHS by the same per-objective argument applied to the max-version of the LSVI recursion), we obtain
∣ Q h ( k ) , Y k ( s , a ) − [ r ˉ h ( k ) ( s , a ) + b ^ h ( k ) ( s , a ) + E P ^ h [ V ˉ h + 1 ( k ) , Y k ] ( s , a ) ] ∣ ≤ ( 1 + Y k ) C 3 d H χ ∥ ϕ ^ h k ( s , a ) ∥ ( Σ h k ) − 1 . (19) \left|Q_{h}^{(k),Y_k}(s,a) - \left[\bar{r}_{h}^{(k)}(s,a) + \hat{b}_{h}^{(k)}(s,a) + \mathbb{E}_{\hat{P}_{h}}\left[\bar{V}_{h+1}^{(k),Y_k}\right](s,a)\right]\right| \leq(1+Y_k)C_3dH\sqrt{\chi}\left\|\hat{\phi}_{h}^{k}(s,a)\right\|_{(\Sigma_{h}^{k})^{-1}}. \tag*{(19)} Q h ( k ) , Y k ( s , a ) − [ r ˉ h ( k ) ( s , a ) + b ^ h ( k ) ( s , a ) + E P ^ h [ V ˉ h + 1 ( k ) , Y k ] ( s , a ) ] ≤ ( 1 + Y k ) C 3 d H χ ϕ ^ h k ( s , a ) ( Σ h k ) − 1 . (19)
Backward induction on the per-state gap
Step 3: Backward induction on the per-state gap. Define
R h ( s ) : = ∣ V ˉ h ⋆ ( s ) − V ˉ h ( k ) , Y k ( s ) ∣ , h ∈ [ H + 1 ] , R_h(s) := \left|\bar{V}_{h}^{\star}(s) - \bar{V}_{h}^{(k),Y_k}(s)\right|,\qquad h \in[H+1], R h ( s ) := V ˉ h ⋆ ( s ) − V ˉ h ( k ) , Y k ( s ) , h ∈ [ H + 1 ] ,
where V ˉ H + 1 ⋆ ≡ 0 \bar{V}_{H+1}^{\star} \equiv0 V ˉ H + 1 ⋆ ≡ 0 and V ˉ H + 1 ( k ) , Y k ≡ 0 \bar{V}_{H+1}^{(k),Y_k} \equiv0 V ˉ H + 1 ( k ) , Y k ≡ 0 , so R H + 1 ( s ) = 0 R_{H+1}(s)=0 R H + 1 ( s ) = 0 .
Write c h ( s , a ) : = ( 1 + Y k ) C 3 d H χ ∥ ϕ ^ h k ( s , a ) ∥ ( Σ h k ) − 1 c_h(s,a) := (1+Y_k)C_3dH\sqrt{\chi}\left\|\hat{\phi}_{h}^{k}(s,a)\right\|_{(\Sigma_{h}^{k})^{-1}} c h ( s , a ) := ( 1 + Y k ) C 3 d H χ ϕ ^ h k ( s , a ) ( Σ h k ) − 1 , and let W h W_h W h be the optimal value function of the planning model P ^ \hat{P} P ^ with stage reward c h c_h c h , that is, W H + 1 ≡ 0 W_{H+1}\equiv0 W H + 1 ≡ 0 and:
W h ( s ) = max a ∈ A [ c h ( s , a ) + E s ′ ∼ P ^ h ( ⋅ ∣ s , a ) [ W h + 1 ( s ′ ) ] ] . W_h(s) = \max_{a\in\mathcal{A}}\left[c_h(s,a) + \mathbb{E}_{s'\sim\hat{P}_h(\cdot\mid s,a)}[W_{h+1}(s')]\right]. W h ( s ) = a ∈ A max [ c h ( s , a ) + E s ′ ∼ P ^ h ( ⋅ ∣ s , a ) [ W h + 1 ( s ′ )] ] .
Let π ~ k ( s ) \tilde{\pi}^{k}(s) π ~ k ( s ) be a maximizer in this recursion. Then W h ( s ) W_h(s) W h ( s ) is the expected cumulative stage reward c c c of π ~ k \tilde{\pi}^{k} π ~ k on P ^ \hat{P} P ^ from s h = s s_h=s s h = s . We claim that for all h ∈ [ H + 1 ] h\in[H+1] h ∈ [ H + 1 ] and all s ∈ S s\in\mathcal{S} s ∈ S :
R h ( s ) ≤ W h ( s ) = E ( s t , a t ) t = h H ∼ d P ^ , h : H π ~ k ( ⋅ ∣ s h = s ) [ ∑ t = h H c t ( s t , a t ) ] . (20) R_h(s) \leq W_h(s) = \mathbb{E}_{(s_t,a_t)_{t=h}^{H}\sim d_{\hat{P},h:H}^{\tilde{\pi}^{k}}(\cdot\mid s_h=s)}\left[\sum_{t=h}^{H}c_t(s_t,a_t)\right]. \tag*{(20)} R h ( s ) ≤ W h ( s ) = E ( s t , a t ) t = h H ∼ d P ^ , h : H π ~ k ( ⋅ ∣ s h = s ) [ t = h ∑ H c t ( s t , a t ) ] . (20)
where d P ^ , h : H π ~ k ( ⋅ ∣ s h = s ) d_{\hat{P},h:H}^{\tilde{\pi}^{k}}(\cdot\mid s_h=s) d P ^ , h : H π ~ k ( ⋅ ∣ s h = s ) denotes the joint distribution of ( s h , a h ) , … , ( s H , a H ) (s_h,a_h),\ldots,(s_H,a_H) ( s h , a h ) , … , ( s H , a H ) under policy π ~ k \tilde{\pi}^{k} π ~ k and kernel P ^ \hat{P} P ^ , conditioned on s h = s s_h=s s h = s .
Base case (h = H + 1 h=H+1 h = H + 1 ): R H + 1 ( s ) = 0 = W H + 1 ( s ) R_{H+1}(s)=0=W_{H+1}(s) R H + 1 ( s ) = 0 = W H + 1 ( s ) .
Inductive step: Assume (20) holds for h + 1 h+1 h + 1 . By the Bellman equation for V ˉ h ⋆ \bar{V}_{h}^{\star} V ˉ h ⋆ on the planning model ( P ^ , r ˉ ( k ) + b ^ ( k ) ) (\hat{P},\bar{r}^{(k)}+\hat{b}^{(k)}) ( P ^ , r ˉ ( k ) + b ^ ( k ) ) :
Q ˉ h ⋆ ( s , a ) = r ˉ h ( k ) ( s , a ) + b ^ h ( k ) ( s , a ) + E P ^ h [ V ˉ h + 1 ⋆ ] ( s , a ) . \bar{Q}_{h}^{\star}(s,a)=\bar{r}_{h}^{(k)}(s,a)+\hat{b}_{h}^{(k)}(s,a)+\mathbb{E}_{\hat{P}_{h}}[\bar{V}_{h+1}^{\star}](s,a). Q ˉ h ⋆ ( s , a ) = r ˉ h ( k ) ( s , a ) + b ^ h ( k ) ( s , a ) + E P ^ h [ V ˉ h + 1 ⋆ ] ( s , a ) .
Subtracting (19) and applying the triangle inequality, for every a ∈ A a\in\mathcal{A} a ∈ A :
∣ Q ˉ h ⋆ ( s , a ) − Q h ( k ) , Y k ( s , a ) ∣ ≤ E s ′ ∼ P ^ h ( ⋅ ∣ s , a ) [ R h + 1 ( s ′ ) ] + c h ( s , a ) . \left|\bar{Q}_{h}^{\star}(s,a)-Q_{h}^{(k),Y_k}(s,a)\right|\leq\mathbb{E}_{s'\sim\hat{P}_{h}(\cdot\mid s,a)}[R_{h+1}(s')]+c_h(s,a). Q ˉ h ⋆ ( s , a ) − Q h ( k ) , Y k ( s , a ) ≤ E s ′ ∼ P ^ h ( ⋅ ∣ s , a ) [ R h + 1 ( s ′ )] + c h ( s , a ) .
Since V ˉ h ⋆ ( s ) = max a Q ˉ h ⋆ ( s , a ) \bar{V}_{h}^{\star}(s)=\max_a\bar{Q}_{h}^{\star}(s,a) V ˉ h ⋆ ( s ) = max a Q ˉ h ⋆ ( s , a ) and V ˉ h ( k ) , Y k ( s ) = max a Q h ( k ) , Y k ( s , a ) \bar{V}_{h}^{(k),Y_k}(s)=\max_a Q_{h}^{(k),Y_k}(s,a) V ˉ h ( k ) , Y k ( s ) = max a Q h ( k ) , Y k ( s , a ) , the contraction property ∣ max a f ( a ) − max a g ( a ) ∣ ≤ max a ∣ f ( a ) − g ( a ) ∣ |\max_a f(a)-\max_a g(a)|\leq\max_a|f(a)-g(a)| ∣ max a f ( a ) − max a g ( a ) ∣ ≤ max a ∣ f ( a ) − g ( a ) ∣ of the maximum over a finite set, followed by the inductive hypothesis R h + 1 ≤ W h + 1 R_{h+1}\leq W_{h+1} R h + 1 ≤ W h + 1 , gives:
R h ( s ) ≤ max a ∈ A [ E s ′ ∼ P ^ h ( ⋅ ∣ s , a ) [ R h + 1 ( s ′ ) ] + c h ( s , a ) ] ≤ max a ∈ A [ E s ′ ∼ P ^ h ( ⋅ ∣ s , a ) [ W h + 1 ( s ′ ) ] + c h ( s , a ) ] = W h ( s ) . \begin{aligned}
R_h(s)\leq\max_{a\in\mathcal{A}}\left[\mathbb{E}_{s'\sim\hat{P}_{h}(\cdot\mid s,a)}[R_{h+1}(s')]+c_h(s,a)\right] \\
&\leq\max_{a\in\mathcal{A}}\left[\mathbb{E}_{s'\sim\hat{P}_{h}(\cdot\mid s,a)}[W_{h+1}(s')]+c_h(s,a)\right]=W_h(s).
\end{aligned} R h ( s ) ≤ a ∈ A max [ E s ′ ∼ P ^ h ( ⋅ ∣ s , a ) [ R h + 1 ( s ′ )] + c h ( s , a ) ] ≤ a ∈ A max [ E s ′ ∼ P ^ h ( ⋅ ∣ s , a ) [ W h + 1 ( s ′ )] + c h ( s , a ) ] = W h ( s ) .
This establishes (20) for h h h and completes the induction. Setting h = 1 h=1 h = 1 , we obtain the per-state bound stated in the lemma.
Cumulative bound via elliptical-potential summation
Step 4: Cumulative bound via elliptical-potential summation. Setting s = s 1 s=s_1 s = s 1 in the per-state bound and summing over k ∈ [ K ] k\in[K] k ∈ [ K ] :
l s v i K = ∑ k = 1 K R 1 ( s 1 ) ≤ ∑ k = 1 K ∑ h = 1 H ( 1 + Y k ) C 3 d H χ E ( s h , a h ) ∼ d P ^ , h π ~ k [ ∥ ϕ ^ h k ( s h , a h ) ∥ ( Σ h k ) − 1 ] . (21) \mathrm{lsvi}_K=\sum_{k=1}^{K}R_1(s_1)\leq\sum_{k=1}^{K}\sum_{h=1}^{H}(1+Y_k)C_3dH\sqrt{\chi}\,\mathbb{E}_{(s_h,a_h)\sim d_{\hat{P},h}^{\tilde{\pi}^{k}}}\left[\left\|\hat{\phi}_{h}^{k}(s_h,a_h)\right\|_{(\Sigma_{h}^{k})^{-1}}\right]. \tag*{(21)} lsvi K = k = 1 ∑ K R 1 ( s 1 ) ≤ k = 1 ∑ K h = 1 ∑ H ( 1 + Y k ) C 3 d H χ E ( s h , a h ) ∼ d P ^ , h π ~ k [ ϕ ^ h k ( s h , a h ) ( Σ h k ) − 1 ] . (21)
To control each summand, we apply Lemma 11 to the function g h ( s , a ) : = C 3 d H χ ∥ ϕ ^ h k ( s , a ) ∥ ( Σ h k ) − 1 g_h(s,a) := C_3dH\sqrt{\chi}\|\hat{\phi}^k_h(s,a)\|_{(\Sigma^k_h)^{-1}} g h ( s , a ) := C 3 d H χ ∥ ϕ ^ h k ( s , a ) ∥ ( Σ h k ) − 1 . Note that ∥ g h ∥ ∞ ≤ C 3 d H χ / λ = O ( d H χ ) \|g_h\|_\infty\le C_3dH\sqrt{\chi}/\sqrt{\lambda} = O(dH\sqrt{\chi}) ∥ g h ∥ ∞ ≤ C 3 d H χ / λ = O ( d H χ ) since Σ h k ⪰ λ I \Sigma^k_h \succeq\lambda I Σ h k ⪰ λ I . By Lemma 11 (applied with policy π = π ~ k \pi= \tilde{\pi}^k π = π ~ k and kernel P ∗ P^* P ∗ ),2
∑ h = 1 H E d P ^ , h π ~ k [ ∥ ϕ ^ h k ∥ ( Σ h k ) − 1 ] ≤ ∣ A ∣ ⋅ ∑ h = 1 H E ( s , a ) ∼ ρ h k [ ∥ ϕ ^ h k ( s , a ) ∥ ( Σ h k ) − 1 2 ] 1 / 2 + (lower-order terms) . \sum_{h=1}^{H}\mathbb{E}_{d^{\tilde{\pi}^k}_{\hat{P},h}}\left[\|\hat{\phi}^k_h\|_{(\Sigma^k_h)^{-1}}\right] \le\sqrt{|\mathcal{A}|}\cdot\sum_{h=1}^{H}\mathbb{E}_{(s,a)\sim\rho^k_h}\left[\|\hat{\phi}^k_h(s,a)\|^2_{(\Sigma^k_h)^{-1}}\right]^{1/2} + \text{(lower-order terms)}. h = 1 ∑ H E d P ^ , h π ~ k [ ∥ ϕ ^ h k ∥ ( Σ h k ) − 1 ] ≤ ∣ A ∣ ⋅ h = 1 ∑ H E ( s , a ) ∼ ρ h k [ ∥ ϕ ^ h k ( s , a ) ∥ ( Σ h k ) − 1 2 ] 1/2 + (lower-order terms) .
where the factor ∣ A ∣ \sqrt{|\mathcal{A}|} ∣ A ∣ comes from the importance-sampling step from d P ^ , h π ~ k d^{\tilde{\pi}^k}_{\hat{P},h} d P ^ , h π ~ k to ρ h k \rho^k_h ρ h k (uniform-action distribution), and the lower-order terms arise from the regularization λ k \lambda_k λ k as in Lemmas 10–11 .
By Cauchy–Schwarz,
∑ k = 1 K E ( s , a ) ∼ ρ h k [ ∥ ϕ ^ h k ( s , a ) ∥ ( Σ h k ) − 1 ] 2 ≤ K ⋅ ∑ k = 1 K E ρ h k [ ∥ ϕ ^ h k ∥ ( Σ h k ) − 1 2 ] . \sum_{k=1}^{K}\mathbb{E}_{(s,a)\sim\rho^k_h}\left[\|\hat{\phi}^k_h(s,a)\|_{(\Sigma^k_h)^{-1}}\right]^2 \le\sqrt{K\cdot\sum_{k=1}^{K}\mathbb{E}_{\rho^k_h}\left[\|\hat{\phi}^k_h\|^2_{(\Sigma^k_h)^{-1}}\right]}. k = 1 ∑ K E ( s , a ) ∼ ρ h k [ ∥ ϕ ^ h k ( s , a ) ∥ ( Σ h k ) − 1 ] 2 ≤ K ⋅ k = 1 ∑ K E ρ h k [ ∥ ϕ ^ h k ∥ ( Σ h k ) − 1 2 ] .
By Lemma 22 (matrix determinant / elliptical potential lemma),
∑ k = 1 K E ρ h k [ ∥ ϕ ^ h k ∥ ( Σ h k ) − 1 2 ] ≤ 2 ln det ( Σ h K + 1 ) det ( Σ h 1 ) ≤ 2 d ln ( 1 + K d λ ) = O ~ ( d ) . \sum_{k=1}^{K}\mathbb{E}_{\rho^k_h}\left[\|\hat{\phi}^k_h\|^2_{(\Sigma^k_h)^{-1}}\right] \le2\ln\frac{\det(\Sigma^{K+1}_h)}{\det(\Sigma^1_h)} \le2d\ln\left(1+\frac{K}{d\lambda}\right) = \widetilde{O}(d). k = 1 ∑ K E ρ h k [ ∥ ϕ ^ h k ∥ ( Σ h k ) − 1 2 ] ≤ 2 ln det ( Σ h 1 ) det ( Σ h K + 1 ) ≤ 2 d ln ( 1 + d λ K ) = O ( d ) .
This is the rigorous form of the elliptical-potential bound. Substituting,
∑ k = 1 K E ρ h k [ ∥ ϕ ^ h k ∥ ( Σ h k ) − 1 2 ] 1 / 2 ≤ K ⋅ O ~ ( d ) = O ~ ( d K ) . \sum_{k=1}^{K}\mathbb{E}_{\rho^k_h}\left[\|\hat{\phi}^k_h\|^2_{(\Sigma^k_h)^{-1}}\right]^{1/2} \le\sqrt{K\cdot\widetilde{O}(d)} = \widetilde{O}(\sqrt{dK}). k = 1 ∑ K E ρ h k [ ∥ ϕ ^ h k ∥ ( Σ h k ) − 1 2 ] 1/2 ≤ K ⋅ O ( d ) = O ( d K ) .
Putting these together, summing over h ∈ [ H ] h\in[H] h ∈ [ H ] and noting 1 + Y k ≤ 1 + ξ 1+Y_k\le1+\xi 1 + Y k ≤ 1 + ξ :
l s v i K ≤ ( 1 + ξ ) C 3 d H χ ∣ A ∣ ∑ h = 1 H O ~ ( d K ) + (lower-order) = O ~ ( ( 1 + ξ ) d 3 / 2 H 2 K ∣ A ∣ 1 / 2 ) = O ~ ( d 3 / 2 H 2 K ∣ A ∣ 1 / 2 γ ) , \begin{aligned}
\mathrm{lsvi}_K \le(1+\xi)C_3dH\sqrt{\chi}\sqrt{|\mathcal{A}|}\sum_{h=1}^{H}\widetilde{O}(\sqrt{dK})+\text{(lower-order)} \\
&= \widetilde{O}\left((1+\xi)d^{3/2}H^2\sqrt{K}|\mathcal{A}|^{1/2}\right) \\
&= \widetilde{O}\left(\frac{d^{3/2}H^2\sqrt{K}|\mathcal{A}|^{1/2}}{\gamma}\right),
\end{aligned} lsvi K ≤ ( 1 + ξ ) C 3 d H χ ∣ A ∣ h = 1 ∑ H O ( d K ) + (lower-order) = O ( ( 1 + ξ ) d 3/2 H 2 K ∣ A ∣ 1/2 ) = O ( γ d 3/2 H 2 K ∣ A ∣ 1/2 ) ,
where the last equality uses 1 + ξ = 1 + 2 H / γ = O ( H / γ ) 1+\xi=1+2H/\gamma=O(H/\gamma) 1 + ξ = 1 + 2 H / γ = O ( H / γ ) (treating H H H as part of the polynomial factor).
Step 5: Domination by ∑ k ϵ k \sum_k\epsilon_k ∑ k ϵ k . By Lemma 3, the cumulative representation error satisfies
∑ k = 1 K ϵ k ≥ C H 2 d 2 K 3 / 4 ∣ A ∣ 3 / 2 log 1 / 4 ( d K ∣ Φ ∣ / δ ) \sum_{k=1}^{K}\epsilon_k \ge CH^2d^2K^{3/4}|\mathcal{A}|^{3/2}\log^{1/4}(dK|\Phi|/\delta) k = 1 ∑ K ϵ k ≥ C H 2 d 2 K 3/4 ∣ A ∣ 3/2 log 1/4 ( d K ∣Φ∣/ δ )
for some constant C > 0 C>0 C > 0 . Comparing exponents of K K K , d d d , and ∣ A ∣ |\mathcal{A}| ∣ A ∣ :
K K K : K = K 1 / 2 \sqrt{K}=K^{1/2} K = K 1/2 vs. K 3 / 4 K^{3/4} K 3/4 – the latter dominates for K ≥ 1 K\ge1 K ≥ 1 .
d d d : d 3 / 2 d^{3/2} d 3/2 vs. d 2 d^2 d 2 – the latter dominates for d ≥ 1 d\ge1 d ≥ 1 .
∣ A ∣ |\mathcal{A}| ∣ A ∣ : ∣ A ∣ 1 / 2 |\mathcal{A}|^{1/2} ∣ A ∣ 1/2 vs. ∣ A ∣ 3 / 2 |\mathcal{A}|^{3/2} ∣ A ∣ 3/2 – the latter dominates for ∣ A ∣ ≥ 1 |\mathcal{A}|\ge1 ∣ A ∣ ≥ 1 .
All exponents go in our favor, so for any K ≥ 1 K\ge1 K ≥ 1 and any d , ∣ A ∣ ≥ 1 d,|\mathcal{A}|\ge1 d , ∣ A ∣ ≥ 1 ,
O ~ ( d 3 / 2 H 2 K ∣ A ∣ 1 / 2 γ ) ≤ 1 4 ( 1 + ξ ) ∑ k = 1 K ϵ k , \widetilde{O}\left(\frac{d^{3/2}H^2\sqrt{K}|\mathcal{A}|^{1/2}}{\gamma}\right) \le\frac{1}{4}(1+\xi)\sum_{k=1}^{K}\epsilon_k, O ( γ d 3/2 H 2 K ∣ A ∣ 1/2 ) ≤ 4 1 ( 1 + ξ ) k = 1 ∑ K ϵ k ,
where the constant 1 4 \frac{1}{4} 4 1 is achieved by absorbing the dimension/cardinality factors into the leading constant of ∑ k ϵ k \sum_k\epsilon_k ∑ k ϵ k (the constant C C C in Lemma 3 is large enough to accommodate this; equivalently, the relation holds for all sufficiently large K K K , and the small-K K K regime is handled by the regret bound’s trivial H K HK H K ceiling).
Step 6: Softmax variant. The bound stated above is for V ˉ 1 ( k ) , Y k ( s ) : = max a Q 1 ( k ) , Y k ( s , a ) \bar{V}_1^{(k),Y_k}(s) := \max_a Q_1^{(k),Y_k}(s,a) V ˉ 1 ( k ) , Y k ( s ) := max a Q 1 ( k ) , Y k ( s , a ) . Algorithm 3 actually outputs the softmax variant
V 1 ( k ) , Y k ( s ) : = ∑ a π 1 k ( a ∣ s ) Q 1 ( k ) , Y k ( s , a ) , π 1 k ( a ∣ ⋅ ) = softmax α ( Q 1 ( k ) , Y k ( ⋅ , a ) ) . V_1^{(k),Y_k}(s) := \sum_a \pi_1^k(a|s)Q_1^{(k),Y_k}(s,a), \qquad\pi_1^k(a|\cdot) = \operatorname{softmax}_{\alpha}(Q_1^{(k),Y_k}(\cdot,a)). V 1 ( k ) , Y k ( s ) := a ∑ π 1 k ( a ∣ s ) Q 1 ( k ) , Y k ( s , a ) , π 1 k ( a ∣ ⋅ ) = softmax α ( Q 1 ( k ) , Y k ( ⋅ , a )) .
Applying Lemma 12 (softmax sub-optimality) to the planning problem ( P ^ , r ˉ ( k ) + b ^ ( k ) ) (\hat{P},\bar{r}^{(k)}+\hat{b}^{(k)}) ( P ^ , r ˉ ( k ) + b ^ ( k ) ) at horizon 1 yields3
∣ V ˉ 1 ( k ) , Y k ( s ) − V 1 ( k ) , Y k ( s ) ∣ ≤ H log ∣ A ∣ α . \left|\bar{V}_1^{(k),Y_k}(s)-V_1^{(k),Y_k}(s)\right| \le\frac{H\log|\mathcal{A}|}{\alpha}. V ˉ 1 ( k ) , Y k ( s ) − V 1 ( k ) , Y k ( s ) ≤ α H log ∣ A ∣ .
Combining with the per-state bound for V ˉ 1 ( k ) , Y k \bar{V}_1^{(k),Y_k} V ˉ 1 ( k ) , Y k via the triangle inequality:
∣ V ˉ 1 ⋆ ( s ) − V 1 ( k ) , Y k ( s ) ∣ ≤ ∣ V ˉ 1 ⋆ ( s ) − V ˉ 1 ( k ) , Y k ( s ) ∣ + H log ∣ A ∣ α , \left|\bar{V}_1^\star(s)-V_1^{(k),Y_k}(s)\right| \le\left|\bar{V}_1^\star(s)-\bar{V}_1^{(k),Y_k}(s)\right|+\frac{H\log|\mathcal{A}|}{\alpha}, V ˉ 1 ⋆ ( s ) − V 1 ( k ) , Y k ( s ) ≤ V ˉ 1 ⋆ ( s ) − V ˉ 1 ( k ) , Y k ( s ) + α H log ∣ A ∣ ,
which gives the softmax-variant bound asserted in the lemma. Summing over k k k , the additional term contributes K H log ∣ A ∣ / α KH\log|\mathcal{A}|/\alpha K H log ∣ A ∣/ α , which (with α = K log ∣ A ∣ / ( 2 ( 1 + ξ + H ) ) ) \alpha=K\log|\mathcal{A}|/(2(1+\xi+H))) α = K log ∣ A ∣/ ( 2 ( 1 + ξ + H ))) equals 2 H ( 1 + ξ + H ) = O ( H 2 γ ) 2H(1+\xi+H)=O\left(\frac{H^2}{\gamma}\right) 2 H ( 1 + ξ + H ) = O ( γ H 2 ) , also dominated by ∑ k ϵ k \sum_k\epsilon_k ∑ k ϵ k .
We now derive the cumulative pseudo regret bound used in Step 1 of the proof of Theorem 3. Recall from the master decomposition (9) that the Pseudo regret quantifies the gap between π ⋆ \pi^\star π ⋆ and the LSVI-UCB optimistic estimate V k V^k V k , while the gap between V k V^k V k and V P ⋆ π k V_{P^\star}^{\pi^k} V P ⋆ π k is handled separately as the Estimation error in Appendix § 16 .
Lemma 4 (Pseudo regret bound). Conditioned on the Representation Learning Guarantee and the event E \mathcal{E} E , with probability at least 1 − δ 1-\delta 1 − δ ,
∑ k = 1 K [ ( V r , 1 π ⋆ ( s 1 ) + Y k V g , 1 π ⋆ ( s 1 ) ) − ( V r , 1 k ( s 1 ) + Y k V g , 1 k ( s 1 ) ) ] ≤ K H log ∣ A ∣ α + ( 1 + ξ ) ∑ k = 1 K ϵ k . \sum_{k=1}^{K}\left[\left(V_{r,1}^{\pi^\star}(s_1)+Y_kV_{g,1}^{\pi^\star}(s_1)\right)-\left(V_{r,1}^{k}(s_1)+Y_kV_{g,1}^{k}(s_1)\right)\right]\le\frac{KH\log|\mathcal{A}|}{\alpha}+(1+\xi)\sum_{k=1}^{K}\epsilon_k. k = 1 ∑ K [ ( V r , 1 π ⋆ ( s 1 ) + Y k V g , 1 π ⋆ ( s 1 ) ) − ( V r , 1 k ( s 1 ) + Y k V g , 1 k ( s 1 ) ) ] ≤ α K H log ∣ A ∣ + ( 1 + ξ ) k = 1 ∑ K ϵ k .
Proof. Using the composite notation V P ⋆ , 1 π , Y k ( s 1 ) = V r , 1 π ( s 1 ) + Y k V g , 1 π ( s 1 ) V_{P^\star,1}^{\pi,Y_k}(s_1)=V_{r,1}^{\pi}(s_1)+Y_kV_{g,1}^{\pi}(s_1) V P ⋆ , 1 π , Y k ( s 1 ) = V r , 1 π ( s 1 ) + Y k V g , 1 π ( s 1 ) and writing V 1 ( k ) , Y k ( s 1 ) : = V r , 1 k ( s 1 ) + Y k V g , 1 k ( s 1 ) V_1^{(k),Y_k}(s_1):=V_{r,1}^k(s_1)+Y_kV_{g,1}^k(s_1) V 1 ( k ) , Y k ( s 1 ) := V r , 1 k ( s 1 ) + Y k V g , 1 k ( s 1 ) for the LSVI-UCB optimistic estimate of the composite value, the per-episode cumulant on the LHS is V P ⋆ , 1 π ⋆ , Y k ( s 1 ) − V 1 ( k ) , Y k ( s 1 ) V_{P^\star,1}^{\pi^\star,Y_k}(s_1)-V_1^{(k),Y_k}(s_1) V P ⋆ , 1 π ⋆ , Y k ( s 1 ) − V 1 ( k ) , Y k ( s 1 ) . We chain three inequalities at the per-episode level, then sum.
Step 1: Bridge LSVI iterate V 1 ( k ) , Y k V_1^{(k),Y_k} V 1 ( k ) , Y k to planning-model optimum V ˉ 1 ⋆ \bar{V}_1^\star V ˉ 1 ⋆ . Let V ˉ h ⋆ ( s ) : = max a Q ˉ h ⋆ ( s , a ) \bar{V}_h^\star(s):=\max_a\bar{Q}_h^\star(s,a) V ˉ h ⋆ ( s ) := max a Q ˉ h ⋆ ( s , a ) denote the value function of the true Bellman-optimal Q Q Q -function on the planning model ( P ^ , r ˉ ( k ) + b ^ ( k ) ) (\hat{P},\bar{r}^{(k)}+\hat{b}^{(k)}) ( P ^ , r ˉ ( k ) + b ^ ( k ) ) , as defined in Lemma 12. The LSVI-UCB output is V 1 ( k ) , Y k ( s 1 ) = ∑ a π 1 k ( a ∣ s 1 ) Q 1 ( k ) , Y k ( s 1 , a ) V_1^{(k),Y_k}(s_1)=\sum_a\pi_1^k(a|s_1)Q_1^{(k),Y_k}(s_1,a) V 1 ( k ) , Y k ( s 1 ) = ∑ a π 1 k ( a ∣ s 1 ) Q 1 ( k ) , Y k ( s 1 , a ) , where Q h ( k ) , Y k : = Q r , h k + Y k Q g , h k Q_h^{(k),Y_k}:=Q_{r,h}^k+Y_kQ_{g,h}^k Q h ( k ) , Y k := Q r , h k + Y k Q g , h k is the LSVI estimate of the composite Bellman-optimal Q Q Q -function. By Lemma 13 , on event E \mathcal{E} E ,
∣ V ˉ 1 ⋆ ( s 1 ) − V 1 ( k ) , Y k ( s 1 ) ∣ ≤ lsvi ( k ) + H log ∣ A ∣ α , (22) \left|\bar{V}_1^\star(s_1)-V_1^{(k),Y_k}(s_1)\right|\le\operatorname{lsvi}^{(k)}+\frac{H\log|\mathcal{A}|}{\alpha}, \tag*{(22)} V ˉ 1 ⋆ ( s 1 ) − V 1 ( k ) , Y k ( s 1 ) ≤ lsvi ( k ) + α H log ∣ A ∣ , (22)
where lsvi ( k ) \operatorname{lsvi}^{(k)} lsvi ( k ) is the per-episode LSVI residual of Lemma 13 (it sums to lsvi K ≤ 1 4 ( 1 + ξ ) ∑ k ϵ k \operatorname{lsvi}_K\le\frac{1}{4}(1+\xi)\sum_k\epsilon_k lsvi K ≤ 4 1 ( 1 + ξ ) ∑ k ϵ k ), and the additional H log ∣ A ∣ / α H\log|\mathcal{A}|/\alpha H log ∣ A ∣/ α accounts for the softmax-vs-max discrepancy between V 1 ( k ) , Y k V_1^{(k),Y_k} V 1 ( k ) , Y k (softmax of Q 1 ( k ) , Y k Q_1^{(k),Y_k} Q 1 ( k ) , Y k ) and V ˉ 1 ( k ) , Y k : = max a Q 1 ( k ) , Y k ( s 1 , ⋅ ) \bar{V}_1^{(k),Y_k}:=\max_a Q_1^{(k),Y_k}(s_1,\cdot) V ˉ 1 ( k ) , Y k := max a Q 1 ( k ) , Y k ( s 1 , ⋅ ) .
Step 2: Optimism from π ⋆ \pi^\star π ⋆ on P ⋆ P^\star P ⋆ to V ˉ 1 ⋆ \bar{V}_1^\star V ˉ 1 ⋆ on P ^ + b ^ ( k ) \hat{P}+\hat{b}^{(k)} P ^ + b ^ ( k ) . By Lemma 2 (composite optimism), promoted to the V V V -function level via the simulation lemma at h = 1 h=1 h = 1 (as derived in Step 3 of the proof of Lemma 3, equations (16) –(17) , applied with π = π ⋆ \pi=\pi^\star π = π ⋆ in place of π k \pi^k π k ),
V P ⋆ , 1 π ⋆ , Y k ( s 1 ) ≤ V P ^ , r ˉ ( k ) + b ^ ( k ) , 1 π ⋆ , Y k ( s 1 ) + ( 1 + Y k ) ∣ A ∣ ζ k + ρ k opt . V_{P^\star,1}^{\pi^\star,Y_k}(s_1)\le V_{\hat{P},\bar{r}^{(k)}+\hat{b}^{(k)},1}^{\pi^\star,Y_k}(s_1)+(1+Y_k)\sqrt{|\mathcal{A}|}\zeta_k+\rho_k^{\operatorname{opt}}. V P ⋆ , 1 π ⋆ , Y k ( s 1 ) ≤ V P ^ , r ˉ ( k ) + b ^ ( k ) , 1 π ⋆ , Y k ( s 1 ) + ( 1 + Y k ) ∣ A ∣ ζ k + ρ k opt .
Since V ˉ 1 ⋆ ( s 1 ) = max a Q ˉ 1 ⋆ ( s 1 , a ) ≥ ∑ a π 1 ⋆ ( a ∣ s 1 ) Q ˉ 1 ⋆ ( s 1 , a ) = V P ^ , r ˉ ( k ) + b ^ ( k ) π ⋆ , Y k ( s 1 ) \bar{V}^{\star}_{1}(s_{1})=\max_{a}\bar{Q}^{\star}_{1}(s_{1},a)\ge\sum_{a}\pi^{\star}_{1}(a\mid s_{1})\bar{Q}^{\star}_{1}(s_{1},a)=V^{\pi^{\star},Y_{k}}_{\widehat{P},\bar{r}^{(k)}+\hat{b}^{(k)}}(s_{1}) V ˉ 1 ⋆ ( s 1 ) = max a Q ˉ 1 ⋆ ( s 1 , a ) ≥ ∑ a π 1 ⋆ ( a ∣ s 1 ) Q ˉ 1 ⋆ ( s 1 , a ) = V P , r ˉ ( k ) + b ^ ( k ) π ⋆ , Y k ( s 1 ) , combining gives
V P ⋆ , r ˉ ( k ) π ⋆ , Y k ( s 1 ) ≤ V ˉ 1 ⋆ ( s 1 ) + ( 1 + Y k ) ∣ A ∣ ζ k + ρ k o p t . (23) V^{\pi^{\star},Y_{k}}_{P^{\star},\bar{r}^{(k)}}(s_{1})\le\bar{V}^{\star}_{1}(s_{1})+(1+Y_{k})\sqrt{|\mathcal{A}|\zeta_{k}}+\rho^{\mathrm{opt}}_{k}. \tag*{(23)} V P ⋆ , r ˉ ( k ) π ⋆ , Y k ( s 1 ) ≤ V ˉ 1 ⋆ ( s 1 ) + ( 1 + Y k ) ∣ A ∣ ζ k + ρ k opt . (23)
Step 3: Bridge V ˉ 1 ⋆ \bar{V}^{\star}_{1} V ˉ 1 ⋆ to V 1 ( k ) , Y k V^{(k),Y_{k}}_{1} V 1 ( k ) , Y k . Combining (23) with (22) ,
V P ⋆ , r ˉ ( k ) π ⋆ , Y k ( s 1 ) − V 1 ( k ) , Y k ( s 1 ) ≤ ( 1 + Y k ) ∣ A ∣ ζ k + ρ k o p t + lsvi ( k ) + H log ∣ A ∣ α . V^{\pi^{\star},Y_{k}}_{P^{\star},\bar{r}^{(k)}}(s_{1})-V^{(k),Y_{k}}_{1}(s_{1})\le(1+Y_{k})\sqrt{|\mathcal{A}|\zeta_{k}}+\rho^{\mathrm{opt}}_{k}+\operatorname{lsvi}(k)+\frac{H\log|\mathcal{A}|}{\alpha}. V P ⋆ , r ˉ ( k ) π ⋆ , Y k ( s 1 ) − V 1 ( k ) , Y k ( s 1 ) ≤ ( 1 + Y k ) ∣ A ∣ ζ k + ρ k opt + lsvi ( k ) + α H log ∣ A ∣ .
Step 4: Summing over k ∈ [ K ] k\in[K] k ∈ [ K ] . Using 1 + Y k ≤ 1 + ξ 1+Y_{k}\le1+\xi 1 + Y k ≤ 1 + ξ and the cumulative bounds ∑ k ( 1 + Y k ) ∣ A ∣ ζ k ≤ ( 1 + ξ ) ∑ k ∣ A ∣ ζ k \sum_{k}(1+Y_{k})\sqrt{|\mathcal{A}|\zeta_{k}}\le(1+\xi)\sum_{k}\sqrt{|\mathcal{A}|\zeta_{k}} ∑ k ( 1 + Y k ) ∣ A ∣ ζ k ≤ ( 1 + ξ ) ∑ k ∣ A ∣ ζ k , ∑ k ρ k o p t ≤ ( 1 + ξ ) ∑ k ϵ k \sum_{k}\rho^{\mathrm{opt}}_{k}\le(1+\xi)\sum_{k}\epsilon_{k} ∑ k ρ k opt ≤ ( 1 + ξ ) ∑ k ϵ k (Lemma 2), and ∑ k lsvi ( k ) = lsvi K ≤ 1 4 ( 1 + ξ ) ∑ k ϵ k \sum_{k}\operatorname{lsvi}(k)=\operatorname{lsvi}_{K}\le\frac{1}{4}(1+\xi)\sum_{k}\epsilon_{k} ∑ k lsvi ( k ) = lsvi K ≤ 4 1 ( 1 + ξ ) ∑ k ϵ k (Lemma 13), the first three residual terms are absorbed into ( 1 + ξ ) ∑ k ϵ k (1+\xi)\sum_{k}\epsilon_{k} ( 1 + ξ ) ∑ k ϵ k via the same elliptical-potential argument used in Lemma 3 (the ∣ A ∣ ζ k \sqrt{|\mathcal{A}|\zeta_{k}} ∣ A ∣ ζ k contribution is strictly smaller than ρ k o p t \rho^{\mathrm{opt}}_{k} ρ k opt and is dominated). Therefore
∑ k = 1 K ( V P ⋆ , 1 π ⋆ , Y k ( s 1 ) − V 1 ( k ) , Y k ( s 1 ) ) ≤ ( 1 + ξ ) ∑ k = 1 K ϵ k + K H log ∣ A ∣ α , \sum_{k=1}^{K}\left(V^{\pi^{\star},Y_{k}}_{P^{\star},1}(s_{1})-V^{(k),Y_{k}}_{1}(s_{1})\right)\le(1+\xi)\sum_{k=1}^{K}\epsilon_{k}+\frac{KH\log|\mathcal{A}|}{\alpha}, k = 1 ∑ K ( V P ⋆ , 1 π ⋆ , Y k ( s 1 ) − V 1 ( k ) , Y k ( s 1 ) ) ≤ ( 1 + ξ ) k = 1 ∑ K ϵ k + α K H log ∣ A ∣ ,
which is the claimed bound. The argument uses (i) composite optimism (Lemma 2) and max ≥ average \max\ge\text{average} max ≥ average to bridge V P ⋆ , r ˉ π ⋆ , Y k V^{\pi^{\star},Y_{k}}_{P^{\star},\bar{r}} V P ⋆ , r ˉ π ⋆ , Y k to V ˉ ⋆ \bar{V}^{\star} V ˉ ⋆ on ( P ^ , r ˉ + b ^ ( k ) ) (\widehat{P},\bar{r}+\hat{b}^{(k)}) ( P , r ˉ + b ^ ( k ) ) ; (ii) LSVI-UCB self-normalized concentration (Lemma 5) to bridge V ˉ ⋆ \bar{V}^{\star} V ˉ ⋆ to the LSVI iterate V 1 ( k ) , Y k V^{(k),Y_{k}}_{1} V 1 ( k ) , Y k , with the LSVI residual quantified in Lemma 13; and (iii) the simulation lemma (Lemma 19) and Representation Learning Guarantee (Lemma 16) to control kernel mismatch between P ^ \widehat{P} P and P ⋆ P^{\star} P ⋆ . It does not require any Bellman-optimality property of π ⋆ \pi^{\star} π ⋆ for the composite objective, which would be incorrect since π ⋆ \pi^{\star} π ⋆ is the optimal policy for the constrained problem and is not in general greedy with respect to r + Y k g r+Y_{k}g r + Y k g for Y k ≠ Y ⋆ Y_{k}\ne Y^{\star} Y k = Y ⋆ .
Bounding Estimation Error
Definition 4. For j ∈ { r , g } j\in\{r,g\} j ∈ { r , g } , write c j , h c_{j,h} c j , h for the per-objective stage cost (c r , h : = r h c_{r,h}:=r_{h} c r , h := r h , c g , h : = g h c_{g,h}:=g_{h} c g , h := g h ), and denote the expected value of the next-state value function under policy π \pi π as
P h ⋆ V j , h + 1 π ( s , a ) : = E s ′ ∼ P ⋆ ( ⋅ ∣ s , a ) [ V j , h + 1 π ( s ′ ) ] . \mathbb{P}^{\star}_{h}V^{\pi}_{j,h+1}(s,a):=\mathbb{E}_{s'\sim P^{\star}(\cdot\mid s,a)}[V^{\pi}_{j,h+1}(s')]. P h ⋆ V j , h + 1 π ( s , a ) := E s ′ ∼ P ⋆ ( ⋅ ∣ s , a ) [ V j , h + 1 π ( s ′ )] .
With this shorthand, the Bellman equation for policy π \pi π reads Q j , h π ( s , a ) = c j , h ( s , a ) + P h ⋆ V j , h + 1 π ( s , a ) Q^{\pi}_{j,h}(s,a)=c_{j,h}(s,a)+\mathbb{P}^{\star}_{h}V^{\pi}_{j,h+1}(s,a) Q j , h π ( s , a ) = c j , h ( s , a ) + P h ⋆ V j , h + 1 π ( s , a ) , and the value function satisfies V j , h π ( s ) = ∑ a ∈ A π h ( a ∣ s ) Q j , h π ( s , a ) V^{\pi}_{j,h}(s)=\sum_{a\in\mathcal{A}}\pi_{h}(a\mid s)Q^{\pi}_{j,h}(s,a) V j , h π ( s ) = ∑ a ∈ A π h ( a ∣ s ) Q j , h π ( s , a ) .
Definition 5. For any step h h h and episode k k k , define
D j , h , 1 k = ⟨ Q j , h k ( s h k , ⋅ ) − Q j , h π k ( s h k , ⋅ ) , π k ( ⋅ ∣ s h k ) ⟩ − ( Q j , h k ( s h k , a h k ) − Q j , h π k ( s h k , a h k ) ) , D^{k}_{j,h,1}=\left\langle Q^{k}_{j,h}(s^{k}_{h},\cdot)-Q^{\pi_{k}}_{j,h}(s^{k}_{h},\cdot),\pi_{k}(\cdot\mid s^{k}_{h})\right\rangle-\left(Q^{k}_{j,h}(s^{k}_{h},a^{k}_{h})-Q^{\pi_{k}}_{j,h}(s^{k}_{h},a^{k}_{h})\right), D j , h , 1 k = ⟨ Q j , h k ( s h k , ⋅ ) − Q j , h π k ( s h k , ⋅ ) , π k ( ⋅ ∣ s h k ) ⟩ − ( Q j , h k ( s h k , a h k ) − Q j , h π k ( s h k , a h k ) ) ,
D j , h , 2 k = P h ⋆ ( V j , h + 1 k − V j , h + 1 π k ) ( s h k , a h k ) − ( V j , h + 1 k − V j , h + 1 π k ) ( s h + 1 k ) . D^{k}_{j,h,2}=\mathbb{P}^{\star}_{h}(V^{k}_{j,h+1}-V^{\pi_{k}}_{j,h+1})(s^{k}_{h},a^{k}_{h})-(V^{k}_{j,h+1}-V^{\pi_{k}}_{j,h+1})(s^{k}_{h+1}). D j , h , 2 k = P h ⋆ ( V j , h + 1 k − V j , h + 1 π k ) ( s h k , a h k ) − ( V j , h + 1 k − V j , h + 1 π k ) ( s h + 1 k ) .
By construction, D j , h , 1 k D^{k}_{j,h,1} D j , h , 1 k has zero mean conditioned on s h k s^{k}_{h} s h k , and D j , h , 2 k D^{k}_{j,h,2} D j , h , 2 k has zero mean conditioned on ( s h k , a h k ) (s^{k}_{h},a^{k}_{h}) ( s h k , a h k ) . The actions a h k a^{k}_{h} a h k here are the on-policy actions sampled in line 10 of Algorithm 1, independent of the uniform-action data a ~ h k \tilde{a}^{k}_{h} a ~ h k used to build Σ h k \Sigma^{k}_{h} Σ h k (line 4).
Lemma 5 (Restated; concentration of empirical Bellman residual). There exists a constant C 2 C_{2} C 2 such that for any fixed p ∈ ( 0 , 1 ) p\in(0,1) p ∈ ( 0 , 1 ) , the event E \mathcal{E} E defined as
E : = ⋂ k ∈ [ K ] , h ∈ [ H ] , j ∈ { r , g } { ∥ ∑ τ = 1 k − 1 ϕ ^ h k [ V j , h + 1 k ( s h + 1 τ ) − P h ⋆ V j , h + 1 k ( s h τ , a ~ h τ ) ] ∥ ( Σ h k ) − 1 ≤ C 2 d H χ } \mathcal{E}:=\bigcap_{k\in[K],\,h\in[H],\,j\in\{r,g\}}\left\{\left\|\sum_{\tau=1}^{k-1}\hat{\phi}^{k}_{h}\left[V^{k}_{j,h+1}(s^{\tau}_{h+1})-\mathbb{P}^{\star}_{h}V^{k}_{j,h+1}(s^{\tau}_{h},\tilde{a}^{\tau}_{h})\right]\right\|_{(\Sigma^{k}_{h})^{-1}}\le C_{2}dH\sqrt{\chi}\right\} E := k ∈ [ K ] , h ∈ [ H ] , j ∈ { r , g } ⋂ ⎩ ⎨ ⎧ τ = 1 ∑ k − 1 ϕ ^ h k [ V j , h + 1 k ( s h + 1 τ ) − P h ⋆ V j , h + 1 k ( s h τ , a ~ h τ ) ] ( Σ h k ) − 1 ≤ C 2 d H χ ⎭ ⎬ ⎫ holds with χ = log ( 4 ( C 1 + 1 ) log ∣ A ∣ d ∣ Φ ∣ / p ) \chi=\log(4(C_{1}+1)\log|\mathcal{A}|d|\Phi|/p) χ = log ( 4 ( C 1 + 1 ) log ∣ A ∣ d ∣Φ∣/ p ) , and Pr ( E ) ≥ 1 − p / 4 \Pr(\mathcal{E})\ge1-p/4 Pr ( E ) ≥ 1 − p /4 .
Proof. The challenge is that both ϕ ^ h k \hat{\phi}^{k}_{h} ϕ ^ h k (which is the current-episode feature ϕ h k \phi^{k}_{h} ϕ h k evaluated at past data ( s h τ , a ~ h τ ) (s^{\tau}_{h},\tilde{a}^{\tau}_{h}) ( s h τ , a ~ h τ ) ) and V j , h + 1 k V^{k}_{j,h+1} V j , h + 1 k are data-dependent: neither is F τ − 1 \mathcal{F}_{\tau-1} F τ − 1 -measurable for τ < k \tau<k τ < k , so a direct application of self-normalized concentration to the adaptive sequence ( ϕ ^ h k ) τ (\hat{\phi}^{k}_{h})_{\tau} ( ϕ ^ h k ) τ does not apply. We address this by uniformly covering the entire optimistic value-function class together with the feature class, then applying the fixed-feature concentration of Ghosh et al. [11, Lemma B.3] to each element of the cover.
Optimistic value class. By construction (Algorithm 3), each V j , h + 1 k V^{k}_{j,h+1} V j , h + 1 k takes the form
V j , h + 1 k ( s ) = ∑ a π h + 1 k ( a ∣ s ) min { ⟨ ϕ ^ h + 1 k ( s , a ) , w ⟩ + β ~ ϕ ^ h + 1 k ( s , a ) ⊤ Λ − 1 ϕ ^ h + 1 k ( s , a ) , H } . V^{k}_{j,h+1}(s)=\sum_{a}\pi^{k}_{h+1}(a\mid s)\min\left\{\langle\hat{\phi}^{k}_{h+1}(s,a),w\rangle+\tilde{\beta}\sqrt{\hat{\phi}^{k}_{h+1}(s,a)^{\top}\Lambda^{-1}\hat{\phi}^{k}_{h+1}(s,a)},H\right\}. V j , h + 1 k ( s ) = a ∑ π h + 1 k ( a ∣ s ) min { ⟨ ϕ ^ h + 1 k ( s , a ) , w ⟩ + β ~ ϕ ^ h + 1 k ( s , a ) ⊤ Λ − 1 ϕ ^ h + 1 k ( s , a ) , H } .
parameterized by ϕ ^ h + 1 k ∈ Φ h + 1 \hat{\phi}^{k}_{h+1}\in\Phi_{h+1} ϕ ^ h + 1 k ∈ Φ h + 1 , a weight w ∈ R d w\in\mathbb{R}^{d} w ∈ R d with ∥ w ∥ ≤ 2 H d \lVert w\rVert\leq2H\sqrt{d} ∥ w ∥ ≤ 2 H d , a regularization-Gram matrix Λ ⪰ I \Lambda\succeq I Λ ⪰ I with ∥ Λ ∥ ≤ λ + K \lVert\Lambda\rVert\leq\lambda+K ∥ Λ ∥ ≤ λ + K , a bonus parameter β ~ ∈ [ 0 , β ] \tilde{\beta}\in[0,\beta] β ~ ∈ [ 0 , β ] , and a softmax temperature dependent on Y k ∈ [ 0 , ξ ] Y_{k}\in[0,\xi] Y k ∈ [ 0 , ξ ] . Let V h + 1 \mathcal{V}_{h+1} V h + 1 denote the class of all such functions over the bounded parameter space.
Covering. For ε = 1 / ( K H ) \varepsilon=1/(KH) ε = 1/ ( K H ) , take an ε \varepsilon ε -cover of: Φ h + 1 \Phi_{h+1} Φ h + 1 (size ∣ Φ ∣ \lvert\Phi\rvert ∣ Φ ∣ when Φ \Phi Φ is finite, or N ε ( Φ ) \mathcal{N}_{\varepsilon}(\Phi) N ε ( Φ ) in general); the ℓ 2 \ell_{2} ℓ 2 -ball { w : ∥ w ∥ 2 ≤ 2 H d } \{w:\lVert w\rVert_{2}\leq2H\sqrt{d}\} { w : ∥ w ∥ 2 ≤ 2 H d } (size ( 8 H d / ε ) d (8H\sqrt{d}/\varepsilon)^{d} ( 8 H d / ε ) d ); the matrix ball { Λ : I ⪯ Λ , ∥ Λ ∥ ≤ λ + K } \{\Lambda:I\preceq\Lambda,\lVert\Lambda\rVert\leq\lambda+K\} { Λ : I ⪯ Λ , ∥ Λ ∥ ≤ λ + K } (size ( λ + K ) d 2 / ε d 2 (\lambda+K)^{d^{2}}/\varepsilon^{d^{2}} ( λ + K ) d 2 / ε d 2 ); the bonus interval [ 0 , β ] [0,\beta] [ 0 , β ] (size β / ε \beta/\varepsilon β / ε ); and [ 0 , ξ ] [0,\xi] [ 0 , ξ ] (size ξ / ε \xi/\varepsilon ξ / ε ). The product gives a cover V h + 1 , ε \mathcal{V}_{h+1,\varepsilon} V h + 1 , ε of V h + 1 \mathcal{V}_{h+1} V h + 1 with
log ∣ V h + 1 , ε ∣ ≤ log ∣ Φ ∣ + ( d 2 + d + 2 ) log ( ( K + β + ξ + λ ) / ε ) ≤ log ∣ Φ ∣ + O ( d 2 log ( K H d ) ) . \log\lvert\mathcal{V}_{h+1,\varepsilon}\rvert\leq\log\lvert\Phi\rvert+(d^{2}+d+2)\log((K+\beta+\xi+\lambda)/\varepsilon)\leq\log\lvert\Phi\rvert+O(d^{2}\log(KHd)). log ∣ V h + 1 , ε ∣ ≤ log ∣ Φ ∣ + ( d 2 + d + 2 ) log (( K + β + ξ + λ ) / ε ) ≤ log ∣ Φ ∣ + O ( d 2 log ( K H d )) .
Fixed-feature concentration on the cover. For each fixed V ∈ V h + 1 , ε V\in\mathcal{V}_{h+1,\varepsilon} V ∈ V h + 1 , ε and each fixed ϕ ∈ Φ \phi\in\Phi ϕ ∈ Φ in the cover, the self-normalized concentration of Ghosh et al. [11, Lemma B.3] applies to the sequence ( ϕ ( s h τ , a ~ h τ ) ) τ ≤ k − 1 (\phi(s^{\tau}_{h},\tilde{a}^{\tau}_{h}))_{\tau\leq k-1} ( ϕ ( s h τ , a ~ h τ ) ) τ ≤ k − 1 , since ϕ \phi ϕ is fixed (not data-dependent) and the historical data ( s h τ , a ~ h τ , s h + 1 τ ) τ ≤ k − 1 (s^{\tau}_{h},\tilde{a}^{\tau}_{h},s^{\tau}_{h+1})_{\tau\leq k-1} ( s h τ , a ~ h τ , s h + 1 τ ) τ ≤ k − 1 generates a standard adaptive sequence with respect to its natural filtration. This gives the stated inequality with probability at least 1 − p 0 1-p_{0} 1 − p 0 , where p 0 = p / ( 4 ∣ V h + 1 , ε ∣ ∣ Φ ∣ H K ⋅ 2 ) p_{0}=p/(4\lvert\mathcal{V}_{h+1,\varepsilon}\rvert\lvert\Phi\rvert HK\cdot2) p 0 = p / ( 4 ∣ V h + 1 , ε ∣ ∣ Φ ∣ H K ⋅ 2 ) .
Discretization error. For any realized ( ϕ ^ h k , V j , h + 1 k ) (\hat{\phi}^{k}_{h},V^{k}_{j,h+1}) ( ϕ ^ h k , V j , h + 1 k ) , let ( ϕ ˉ , V ˉ ) (\bar{\phi},\bar{V}) ( ϕ ˉ , V ˉ ) be the nearest neighbor in Φ × V h + 1 , ε \Phi\times\mathcal{V}_{h+1,\varepsilon} Φ × V h + 1 , ε (so ∥ ϕ ^ h k − ϕ ˉ ∥ ∞ ≤ ε \lVert\hat{\phi}^{k}_{h}-\bar{\phi}\rVert_{\infty}\leq\varepsilon ∥ ϕ ^ h k − ϕ ˉ ∥ ∞ ≤ ε and ∥ V j , h + 1 k − V ˉ ∥ ∞ ≤ ε \lVert V^{k}_{j,h+1}-\bar{V}\rVert_{\infty}\leq\varepsilon ∥ V j , h + 1 k − V ˉ ∥ ∞ ≤ ε ). The combined approximation error contributes at most K ε ⋅ 2 d = 2 d / H K\varepsilon\cdot2\sqrt{d}=2\sqrt{d}/H K ε ⋅ 2 d = 2 d / H in the self-normalized norm, which is dominated by the main concentration term.
Union bound. Taking a union bound over V ∈ V h + 1 , ε V\in\mathcal{V}_{h+1,\varepsilon} V ∈ V h + 1 , ε , ϕ ∈ Φ \phi\in\Phi ϕ ∈ Φ , h ∈ [ H ] h\in[H] h ∈ [ H ] , j ∈ { r , g } j\in\{r,g\} j ∈ { r , g } , and k ∈ [ K ] k\in[K] k ∈ [ K ] , the event E \mathcal{E} E holds with probability at least 1 − p / 4 1-p/4 1 − p /4 . Substituting p 0 p_{0} p 0 and absorbing constants into C 2 C_{2} C 2 , the threshold becomes χ = log ( 4 ( C 1 + 1 ) log ∣ A ∣ d T ∣ Φ ∣ / p ) + O ( d 2 log ( K H d ) ) \chi=\log(4(C_{1}+1)\log\lvert\mathcal{A}\rvert dT\lvert\Phi\rvert/p)+O(d^{2}\log(KHd)) χ = log ( 4 ( C 1 + 1 ) log ∣ A ∣ d T ∣ Φ ∣ / p ) + O ( d 2 log ( K H d )) , which we absorb as the polylog factor log ( ⋅ ) \log(\cdot) log ( ⋅ ) in the statement.
Lemma 6 (Restated; one-step Q Q Q -backup error). Let β = C 1 d H ι \beta=C_{1}dH\sqrt{\iota} β = C 1 d H ι with ι = log 4 ( log ∣ A ∣ d T ∣ Φ ∣ / p ) \iota=\log^{4}(\log\lvert\mathcal{A}\rvert dT\lvert\Phi\rvert/p) ι = log 4 ( log ∣ A ∣ d T ∣ Φ ∣ / p ) . Conditioned on the event E \mathcal{E} E (Lemma 5), for every episode k ∈ [ K ] k\in[K] k ∈ [ K ] , step h ∈ [ H ] h\in[H] h ∈ [ H ] , j ∈ { r , g } j\in\{r,g\} j ∈ { r , g } , and ( s , a ) ∈ S × A (s,a)\in\mathcal{S}\times\mathcal{A} ( s , a ) ∈ S × A , there exist weights w j , h π k ∈ R d w^{\pi^{k}}_{j,h}\in\mathbb{R}^{d} w j , h π k ∈ R d approximately projecting Q j , h π k Q^{\pi^{k}}_{j,h} Q j , h π k onto span ( ϕ ^ h k ) \operatorname{span}(\hat{\phi}^{k}_{h}) span ( ϕ ^ h k ) such that
Q j , h k ( s , a ) − Q j , h π k ( s , a ) = P h ∗ ( V j , h + 1 k − V j , h + 1 π k ) ( s , a ) + Δ h k ( s , a ) + η h k ( s , a ) , Q^{k}_{j,h}(s,a)-Q^{\pi^{k}}_{j,h}(s,a)=\mathbb{P}^{\ast}_{h}(V^{k}_{j,h+1}-V^{\pi^{k}}_{j,h+1})(s,a)+\Delta^{k}_{h}(s,a)+\eta^{k}_{h}(s,a), Q j , h k ( s , a ) − Q j , h π k ( s , a ) = P h ∗ ( V j , h + 1 k − V j , h + 1 π k ) ( s , a ) + Δ h k ( s , a ) + η h k ( s , a ) , where the per-episode bonus residual satisfies ∣ Δ h k ( s , a ) ∣ ≤ β ϕ ^ h k ( s , a ) ⊤ ( Σ h k ) − 1 ϕ ^ h k ( s , a ) \lvert\Delta^{k}_{h}(s,a)\rvert\leq\beta\sqrt{\hat{\phi}^{k}_{h}(s,a)^{\top}(\Sigma^{k}_{h})^{-1}\hat{\phi}^{k}_{h}(s,a)} ∣ Δ h k ( s , a )∣ ≤ β ϕ ^ h k ( s , a ) ⊤ ( Σ h k ) − 1 ϕ ^ h k ( s , a ) , and the projection residual η h k \eta^{k}_{h} η h k satisfies E ( s , a ) ∼ ρ h k [ ( η h k ) 2 ] ≤ 4 H 2 ζ k \mathbb{E}_{(s,a)\sim\rho^{k}_{h}}[(\eta^{k}_{h})^{2}]\leq4H^{2}\zeta_{k} E ( s , a ) ∼ ρ h k [( η h k ) 2 ] ≤ 4 H 2 ζ k with cumulative bound
∑ k = 1 K ∑ h = 1 H ∣ E d P ∗ , h π k [ η h k ] ∣ ≤ O ~ ( H 2 d K 3 / 4 ∣ A ∣ ) , \sum_{k=1}^{K}\sum_{h=1}^{H}\left\lvert\mathbb{E}_{d^{\pi^{k}}_{P^{\ast},h}}[\eta^{k}_{h}]\right\rvert\leq\widetilde{O}(H^{2}dK^{3/4}\sqrt{\lvert\mathcal{A}\rvert}), k = 1 ∑ K h = 1 ∑ H E d P ∗ , h π k [ η h k ] ≤ O ( H 2 d K 3/4 ∣ A ∣ ) , which is dominated by ∑ k ϵ k \sum_{k}\epsilon_{k} ∑ k ϵ k from Lemma 3 and does not affect the leading rate.
Proof. Existence of w j , h π k w^{\pi^{k}}_{j,h} w j , h π k . By the Representation Learning Guarantee (Lemma 16 ), Q j , h π k Q^{\pi^{k}}_{j,h} Q j , h π k —being the Bellman backup of a function in V h \mathcal{V}_{h} V h —is approximated within ζ k \zeta_{k} ζ k by some linear function of ϕ ^ h k \hat{\phi}^{k}_{h} ϕ ^ h k on the data distribution ρ h k \rho^{k}_{h} ρ h k . By Lemma 18 there exists w j , h π k ∈ R d w^{\pi^{k}}_{j,h}\in\mathbb{R}^{d} w j , h π k ∈ R d with ∥ w j , h π k ∥ 2 ≤ 2 H d \lVert w^{\pi^{k}}_{j,h}\rVert_{2}\leq2H\sqrt{d} ∥ w j , h π k ∥ 2 ≤ 2 H d such that
Q j , h π k ( s , a ) = ⟨ ϕ ^ h k ( s , a ) , w j , h π k ⟩ + ξ h π k ( s , a ) , E ( s , a ) ∼ ρ h π k [ ( ξ h π k ) 2 ] ≤ 4 H 2 ζ k . (24) Q^{\pi^{k}}_{j,h}(s,a)=\langle\hat{\phi}^{k}_{h}(s,a),w^{\pi^{k}}_{j,h}\rangle+\xi^{\pi^{k}}_{h}(s,a),\qquad\mathbb{E}_{(s,a)\sim\rho^{\pi^{k}}_{h}}[(\xi^{\pi^{k}}_{h})^{2}]\leq4H^{2}\zeta_{k}. \tag*{(24)} Q j , h π k ( s , a ) = ⟨ ϕ ^ h k ( s , a ) , w j , h π k ⟩ + ξ h π k ( s , a ) , E ( s , a ) ∼ ρ h π k [( ξ h π k ) 2 ] ≤ 4 H 2 ζ k . (24)
Crucially, ξ h τ , k ( s , a ) \xi_{h}^{\tau,k}(s,a) ξ h τ , k ( s , a ) is a deterministic representation-approximation residual: it depends on ( s , a ) (s,a) ( s , a ) , ϕ ^ h k \widehat{\phi}_{h}^{k} ϕ h k , and w j , h π k w_{j,h}^{\pi_k} w j , h π k , but is not a martingale increment. We will track it explicitly throughout the derivation and absorb it into η h k \eta_h^k η h k at the end.
Linear Q Q Q -function representation. Let w j , h k = ( Σ h k ) − 1 ∑ τ = 1 k − 1 ϕ ^ h τ [ c j , h τ + V j , h + 1 k ( s h + 1 τ ) ] w_{j,h}^{k}=(\Sigma_h^k)^{-1}\sum_{\tau=1}^{k-1}\widehat{\phi}_{h}^{\tau}\left[c_{j,h}^{\tau}+V_{j,h+1}^{k}(s_{h+1}^{\tau})\right] w j , h k = ( Σ h k ) − 1 ∑ τ = 1 k − 1 ϕ h τ [ c j , h τ + V j , h + 1 k ( s h + 1 τ ) ] be the LSVI weights (Algorithm 3, lines 4–5). Then
Q j , h k ( s , a ) − Q j , h π k ( s , a ) = ⟨ ϕ ^ h k ( s , a ) , w j , h k − w j , h π k ⟩ − ξ h π k ( s , a ) . Q_{j,h}^{k}(s,a)-Q_{j,h}^{\pi_k}(s,a)=\langle\widehat{\phi}_{h}^{k}(s,a),w_{j,h}^{k}-w_{j,h}^{\pi_k}\rangle-\xi_{h}^{\pi_k}(s,a). Q j , h k ( s , a ) − Q j , h π k ( s , a ) = ⟨ ϕ h k ( s , a ) , w j , h k − w j , h π k ⟩ − ξ h π k ( s , a ) .
To expand w j , h k − w j , h π k w_{j,h}^{k}-w_{j,h}^{\pi_k} w j , h k − w j , h π k , we use the Bellman equation for Q j , h π k Q_{j,h}^{\pi_k} Q j , h π k on P ⋆ P^\star P ⋆ together with (24) : for each historical sample τ < k \tau<k τ < k ,
c j , h τ + P h ⋆ V j , h + 1 π k ( s h τ , a ~ h τ ) = ⟨ ϕ ^ h τ , w j , h π k ⟩ + ξ h π k ( s h τ , a ~ h τ ) . c_{j,h}^{\tau}+P_h^\star V_{j,h+1}^{\pi_k}(s_h^\tau,\widetilde{a}_h^\tau)=\langle\widehat{\phi}_{h}^{\tau},w_{j,h}^{\pi_k}\rangle+\xi_h^{\pi_k}(s_h^\tau,\widetilde{a}_h^\tau). c j , h τ + P h ⋆ V j , h + 1 π k ( s h τ , a h τ ) = ⟨ ϕ h τ , w j , h π k ⟩ + ξ h π k ( s h τ , a h τ ) .
Combining with the definition of Σ h k \Sigma_h^k Σ h k ,
w j , h k − w j , h π k = − λ ( Σ h k ) − 1 w j , h π k + ( Σ h k ) − 1 ∑ τ = 1 k − 1 ϕ ^ h τ [ V j , h + 1 k ( s h + 1 τ ) − P h ⋆ V j , h + 1 k ( s h τ , a ~ h τ ) ] + ( Σ h k ) − 1 ∑ τ = 1 k − 1 ϕ ^ h τ [ P h ⋆ V j , h + 1 k ( s h τ , a ~ h τ ) − P h ⋆ V j , h + 1 π k ( s h τ , a ~ h τ ) ] + ( Σ h k ) − 1 ∑ τ = 1 k − 1 ϕ ^ h τ ξ h π k ( s h τ , a ~ h τ ) . (25) \begin{aligned}
w_{j,h}^{k}-w_{j,h}^{\pi_k}
={}&-\lambda(\Sigma_h^k)^{-1}w_{j,h}^{\pi_k}
+(\Sigma_h^k)^{-1}\sum_{\tau=1}^{k-1}\widehat{\phi}_{h}^{\tau}\left[V_{j,h+1}^{k}(s_{h+1}^{\tau})-P_h^\star V_{j,h+1}^{k}(s_h^\tau,\widetilde{a}_h^\tau)\right]\\
&+(\Sigma_h^k)^{-1}\sum_{\tau=1}^{k-1}\widehat{\phi}_{h}^{\tau}\left[P_h^\star V_{j,h+1}^{k}(s_h^\tau,\widetilde{a}_h^\tau)-P_h^\star V_{j,h+1}^{\pi_k}(s_h^\tau,\widetilde{a}_h^\tau)\right]\\
&+(\Sigma_h^k)^{-1}\sum_{\tau=1}^{k-1}\widehat{\phi}_{h}^{\tau}\xi_h^{\pi_k}(s_h^\tau,\widetilde{a}_h^\tau). \tag*{(25)}
\end{aligned} w j , h k − w j , h π k = − λ ( Σ h k ) − 1 w j , h π k + ( Σ h k ) − 1 τ = 1 ∑ k − 1 ϕ h τ [ V j , h + 1 k ( s h + 1 τ ) − P h ⋆ V j , h + 1 k ( s h τ , a h τ ) ] + ( Σ h k ) − 1 τ = 1 ∑ k − 1 ϕ h τ [ P h ⋆ V j , h + 1 k ( s h τ , a h τ ) − P h ⋆ V j , h + 1 π k ( s h τ , a h τ ) ] + ( Σ h k ) − 1 τ = 1 ∑ k − 1 ϕ h τ ξ h π k ( s h τ , a h τ ) . (25)
Compared to the standard linear-MDP analysis [11 ] , the additional term q 4 \mathbf{q}_4 q 4 collects the deterministic representation-approximation residuals ξ h π k \xi_h^{\pi_k} ξ h π k along the trajectory; we bound this term separately below as it is not a mean-zero martingale.
Term q 1 \mathbf{q}_1 q 1 . Using ∥ w j , h π k ∥ 2 ≤ 2 H d \|w_{j,h}^{\pi_k}\|_2\le2H\sqrt{d} ∥ w j , h π k ∥ 2 ≤ 2 H d :
∣ ⟨ ϕ ^ h k ( s , a ) , q 1 ⟩ ∣ ≤ 2 H d λ ϕ ^ h k ( s , a ) ⊤ ( Σ h k ) − 1 ϕ ^ h k ( s , a ) . \left|\langle\widehat{\phi}_{h}^{k}(s,a),\mathbf{q}_1\rangle\right|\le2H\sqrt{d}\lambda\sqrt{\widehat{\phi}_{h}^{k}(s,a)^\top(\Sigma_h^k)^{-1}\widehat{\phi}_{h}^{k}(s,a)}. ⟨ ϕ h k ( s , a ) , q 1 ⟩ ≤ 2 H d λ ϕ h k ( s , a ) ⊤ ( Σ h k ) − 1 ϕ h k ( s , a ) .
Term q 2 \mathbf{q}_2 q 2 . On event E \mathcal{E} E (Lemma 5):
∣ ⟨ ϕ ^ h k ( s , a ) , q 2 ⟩ ∣ ≤ C 2 d H χ ϕ ^ h k ( s , a ) ⊤ ( Σ h k ) − 1 ϕ ^ h k ( s , a ) . \left|\langle\widehat{\phi}_{h}^{k}(s,a),\mathbf{q}_2\rangle\right|\le C_2dH\sqrt{\chi}\sqrt{\widehat{\phi}_{h}^{k}(s,a)^\top(\Sigma_h^k)^{-1}\widehat{\phi}_{h}^{k}(s,a)}. ⟨ ϕ h k ( s , a ) , q 2 ⟩ ≤ C 2 d H χ ϕ h k ( s , a ) ⊤ ( Σ h k ) − 1 ϕ h k ( s , a ) .
Term q 3 \mathbf{q}_3 q 3 . By the low-rank decomposition of P ⋆ P^\star P ⋆ (Definition 1), for any V : S → [ 0 , 2 H ] V:\mathcal{S}\to[0,2H] V : S → [ 0 , 2 H ] there exists w V ∈ R d w_V\in\mathbb{R}^d w V ∈ R d with ∥ w V ∥ 2 ≤ 2 H d \|w_V\|_2\le2H\sqrt{d} ∥ w V ∥ 2 ≤ 2 H d such that P h ⋆ V ( s , a ) = ⟨ ϕ h ⋆ ( s , a ) , w V ⟩ P_h^\star V(s,a)=\langle\phi_h^\star(s,a),w_V\rangle P h ⋆ V ( s , a ) = ⟨ ϕ h ⋆ ( s , a ) , w V ⟩ . By the Representation Learning guarantee, ϕ ^ h k \widehat{\phi}_h^k ϕ h k approximates ϕ h ⋆ \phi_h^\star ϕ h ⋆ on ρ h k \rho_h^k ρ h k up to ζ k \zeta_k ζ k ; hence there exists w ^ V \widehat{w}_V w V with ∥ w ^ V ∥ 2 ≤ 2 H d \|\widehat{w}_V\|_2\le2H\sqrt{d} ∥ w V ∥ 2 ≤ 2 H d such that
P h ⋆ V ( s , a ) = ⟨ ϕ ^ h k ( s , a ) , w ^ V ⟩ + η ~ h ( s , a ; V ) , E ( s , a ~ ) ∼ ρ h k [ η ~ h 2 ] ≤ 4 H 2 ζ k . P_h^\star V(s,a)=\langle\widehat{\phi}_{h}^{k}(s,a),\widehat{w}_V\rangle+\widetilde{\eta}_h(s,a;V),\qquad\mathbb{E}_{(s,\widetilde{a})\sim\rho_h^k}\left[\widetilde{\eta}_h^2\right]\le4H^2\zeta_k. P h ⋆ V ( s , a ) = ⟨ ϕ h k ( s , a ) , w V ⟩ + η h ( s , a ; V ) , E ( s , a ) ∼ ρ h k [ η h 2 ] ≤ 4 H 2 ζ k .
Applying this with V = V j , h + 1 k − V j , h + 1 π k ∈ [ 0 , 2 H ] V=V_{j,h+1}^{k}-V_{j,h+1}^{\pi_k}\in[0,2H] V = V j , h + 1 k − V j , h + 1 π k ∈ [ 0 , 2 H ] and substituting into q 3 \mathbf{q}_3 q 3 ,
⟨ ϕ ^ h k ( s , a ) , q 3 ⟩ = P h ⋆ ( V j , h + 1 k − V j , h + 1 π k ) ( s , a ) − η ~ h ( s , a ; V j , h + 1 k − V j , h + 1 π k ) − ⟨ ϕ ^ h k ( s , a ) , λ ( Σ h k ) − 1 w ^ V ⟩ . \begin{aligned}
\langle\widehat{\phi}_{h}^{k}(s,a),\mathbf{q}_3\rangle
={}&P_h^\star(V_{j,h+1}^{k}-V_{j,h+1}^{\pi_k})(s,a)-\widetilde{\eta}_h(s,a;V_{j,h+1}^{k}-V_{j,h+1}^{\pi_k})\\
&-\langle\widehat{\phi}_{h}^{k}(s,a),\lambda(\Sigma_h^k)^{-1}\widehat{w}_V\rangle.
\end{aligned} ⟨ ϕ h k ( s , a ) , q 3 ⟩ = P h ⋆ ( V j , h + 1 k − V j , h + 1 π k ) ( s , a ) − η h ( s , a ; V j , h + 1 k − V j , h + 1 π k ) − ⟨ ϕ h k ( s , a ) , λ ( Σ h k ) − 1 w V ⟩ .
The regularization term is bounded by 2 H d λ ∥ ϕ ^ h k ( s , a ) ∥ ( Σ h k ) − 1 2H\sqrt{d\lambda}\|\widehat{\phi}_{h}^{k}(s,a)\|_{(\Sigma_h^k)^{-1}} 2 H d λ ∥ ϕ h k ( s , a ) ∥ ( Σ h k ) − 1 .
Term q 4 \mathbf{q}_4 q 4 (representation-residual aggregation). The summation ∑ τ < k ϕ ^ h τ ξ h π k ( s h τ , a ~ h τ ) \sum_{\tau<k}\widehat{\phi}_{h}^{\tau}\xi_h^{\pi_k}(s_h^\tau,\widetilde{a}_h^\tau) ∑ τ < k ϕ h τ ξ h π k ( s h τ , a h τ ) is not a mean-zero martingale, so we bound it directly by Cauchy–Schwarz in the Σ h k \Sigma_h^k Σ h k -induced inner product. Since for any vectors { u τ } τ < k \{u_\tau\}_{\tau<k} { u τ } τ < k and scalars { c τ } τ < k \{c_\tau\}_{\tau<k} { c τ } τ < k ,
∥ ( Σ h k ) − 1 ∑ τ < k u τ c τ ∥ Σ h k 2 = ⟨ ∑ τ < k u τ c τ , ( Σ h k ) − 1 ∑ τ < k u τ c τ ⟩ ≤ ( ∑ τ < k c τ 2 ) ⋅ ∑ τ < k ∥ u τ ∥ ( Σ h k ) − 1 2 , \left\|(\Sigma_h^k)^{-1}\sum_{\tau<k}u_\tau c_\tau\right\|_{\Sigma_h^k}^{2}
=\left\langle\sum_{\tau<k}u_\tau c_\tau,(\Sigma_h^k)^{-1}\sum_{\tau<k}u_\tau c_\tau\right\rangle
\le\left(\sum_{\tau<k}c_\tau^2\right)\cdot\sum_{\tau<k}\|u_\tau\|_{(\Sigma_h^k)^{-1}}^2, ( Σ h k ) − 1 τ < k ∑ u τ c τ Σ h k 2 = ⟨ τ < k ∑ u τ c τ , ( Σ h k ) − 1 τ < k ∑ u τ c τ ⟩ ≤ ( τ < k ∑ c τ 2 ) ⋅ τ < k ∑ ∥ u τ ∥ ( Σ h k ) − 1 2 ,
applying this with u τ = ϕ ^ h k u_\tau=\hat{\phi}^{k}_h u τ = ϕ ^ h k and c τ = ξ h π k ( s h τ , a ~ h τ ) c_\tau=\xi^{\pi^k}_h(s^\tau_h,\tilde{a}^\tau_h) c τ = ξ h π k ( s h τ , a ~ h τ ) , then converting via ϕ ^ h k ( s , a ) ⊤ ( Σ h k ) − 1 \hat{\phi}^{k}_h(s,a)^\top(\Sigma^k_h)^{-1} ϕ ^ h k ( s , a ) ⊤ ( Σ h k ) − 1 :
∣ ⟨ ϕ ^ h k ( s , a ) , q 4 ⟩ ∣ ≤ ϕ ^ h k ( s , a ) ⊤ ( Σ h k ) − 1 ϕ ^ h k ( s , a ) ⋅ ∑ τ < k ξ h π k ( s h τ , a ~ h τ ) 2 ⋅ ∑ τ < k ∥ ϕ ^ h τ ∥ ( Σ h k ) − 1 2 . |\langle\hat{\phi}^{k}_h(s,a),\mathbf{q}_4\rangle|\leq\sqrt{\hat{\phi}^{k}_h(s,a)^\top(\Sigma^k_h)^{-1}\hat{\phi}^{k}_h(s,a)}\cdot\sqrt{\sum_{\tau<k}\xi^{\pi^k}_h(s^\tau_h,\tilde{a}^\tau_h)^2\cdot\sum_{\tau<k}\|\hat{\phi}^{\tau}_h\|^2_{(\Sigma^k_h)^{-1}}}. ∣ ⟨ ϕ ^ h k ( s , a ) , q 4 ⟩ ∣ ≤ ϕ ^ h k ( s , a ) ⊤ ( Σ h k ) − 1 ϕ ^ h k ( s , a ) ⋅ τ < k ∑ ξ h π k ( s h τ , a ~ h τ ) 2 ⋅ τ < k ∑ ∥ ϕ ^ h τ ∥ ( Σ h k ) − 1 2 .
Using the elliptical-potential trace bound ∑ τ < k ∥ ϕ ^ h τ ∥ ( Σ h k ) − 1 2 ≤ d \sum_{\tau<k}\|\hat{\phi}^{\tau}_h\|^2_{(\Sigma^k_h)^{-1}}\leq d ∑ τ < k ∥ ϕ ^ h τ ∥ ( Σ h k ) − 1 2 ≤ d and ∑ τ < k ξ h π k ( s h τ , a ~ h τ ) 2 ≤ 4 k H 2 ζ k \sum_{\tau<k}\xi^{\pi^k}_h(s^\tau_h,\tilde{a}^\tau_h)^2\leq4kH^2\zeta_k ∑ τ < k ξ h π k ( s h τ , a ~ h τ ) 2 ≤ 4 k H 2 ζ k (with high probability over the empirical realization of ξ h π k \xi^{\pi^k}_h ξ h π k along the trajectory; here we use that the data points ( s h τ , a ~ h τ ) (s^\tau_h,\tilde{a}^\tau_h) ( s h τ , a ~ h τ ) are drawn under ρ h k \rho^k_h ρ h k where the second-moment bound (24) holds), we obtain
∣ ⟨ ϕ ^ h k ( s , a ) , q 4 ⟩ ∣ ≤ 2 H d k ζ k ⋅ ϕ ^ h k ( s , a ) ⊤ ( Σ h k ) − 1 ϕ ^ h k ( s , a ) . |\langle\hat{\phi}^{k}_h(s,a),\mathbf{q}_4\rangle|\leq2H\sqrt{dk\zeta_k}\cdot\sqrt{\hat{\phi}^{k}_h(s,a)^\top(\Sigma^k_h)^{-1}\hat{\phi}^{k}_h(s,a)}. ∣ ⟨ ϕ ^ h k ( s , a ) , q 4 ⟩ ∣ ≤ 2 H d k ζ k ⋅ ϕ ^ h k ( s , a ) ⊤ ( Σ h k ) − 1 ϕ ^ h k ( s , a ) .
With ζ k = O ~ ( d 2 / k ) \zeta_k=\widetilde{O}(d^2/\sqrt{k}) ζ k = O ( d 2 / k ) , the prefactor evaluates to O ~ ( H d 3 / 2 k 1 / 4 ) \widetilde{O}(Hd^{3/2}k^{1/4}) O ( H d 3/2 k 1/4 ) , which is dominated by the bonus term β = C 1 d H ι = O ~ ( d H ) \beta=C_1dH\sqrt{\iota}=\widetilde{O}(dH) β = C 1 d H ι = O ( d H ) in the cumulative sum after the K 3 / 4 K^{3/4} K 3/4 averaging on the right-hand side. We therefore absorb q 4 \mathbf{q}_4 q 4 into the Δ h k \Delta^k_h Δ h k bonus residual below by adjusting the constant C 3 C_3 C 3 .
Combining q 1 \mathbf{q}_1 q 1 , q 2 \mathbf{q}_2 q 2 , q 3 \mathbf{q}_3 q 3 , q 4 \mathbf{q}_4 q 4 and the projection residual ξ h π k \xi^{\pi^k}_h ξ h π k . Setting
Δ h k ( s , a ) : = ⟨ ϕ ^ h k ( s , a ) , q 1 + q 2 + q 4 ⟩ − ⟨ ϕ ^ h k ( s , a ) , λ ( Σ h k ) − 1 w ^ V ⟩ , \Delta^k_h(s,a):=\langle\hat{\phi}^{k}_h(s,a),\mathbf{q}_1+\mathbf{q}_2+\mathbf{q}_4\rangle-\langle\hat{\phi}^{k}_h(s,a),\lambda(\Sigma^k_h)^{-1}\hat{w}_V\rangle, Δ h k ( s , a ) := ⟨ ϕ ^ h k ( s , a ) , q 1 + q 2 + q 4 ⟩ − ⟨ ϕ ^ h k ( s , a ) , λ ( Σ h k ) − 1 w ^ V ⟩ ,
η h k ( s , a ) : = − η ~ h ( s , a ; ⋅ ) − ξ h π k ( s , a ) , \eta^k_h(s,a):=-\tilde{\eta}_h(s,a;\cdot)-\xi^{\pi^k}_h(s,a), η h k ( s , a ) := − η ~ h ( s , a ; ⋅ ) − ξ h π k ( s , a ) ,
we obtain
Q j , h k ( s , a ) − Q j , h π k ( s , a ) = P ⋆ ( V j , h + 1 k − V j , h + 1 π k ) ( s , a ) + Δ h k ( s , a ) + η h k ( s , a ) , Q^k_{j,h}(s,a)-Q^{\pi^k}_{j,h}(s,a)=\mathbb{P}^{\star}(V^k_{j,h+1}-V^{\pi^k}_{j,h+1})(s,a)+\Delta^k_h(s,a)+\eta^k_h(s,a), Q j , h k ( s , a ) − Q j , h π k ( s , a ) = P ⋆ ( V j , h + 1 k − V j , h + 1 π k ) ( s , a ) + Δ h k ( s , a ) + η h k ( s , a ) ,
with ∣ Δ h k ( s , a ) ∣ ≤ C 3 d H χ ∥ ϕ ^ h k ( s , a ) ∥ ( Σ h k ) − 1 |\Delta^k_h(s,a)|\leq C_3dH\sqrt{\chi}\|\hat{\phi}^{k}_h(s,a)\|_{(\Sigma^k_h)^{-1}} ∣ Δ h k ( s , a ) ∣ ≤ C 3 d H χ ∥ ϕ ^ h k ( s , a ) ∥ ( Σ h k ) − 1 where C 3 = O ( λ + C 2 + d k ζ k / ( d H ) ) C_3=O(\sqrt{\lambda}+C_2+\sqrt{dk\zeta_k}/(dH)) C 3 = O ( λ + C 2 + d k ζ k / ( d H )) (the third summand absorbs q 4 \mathbf{q}_4 q 4 ), and E ρ h k [ ( η h k ) 2 ] ≤ 8 H 2 ζ k \mathbb{E}_{\rho^k_h}[(\eta^k_h)^2]\leq8H^2\zeta_k E ρ h k [( η h k ) 2 ] ≤ 8 H 2 ζ k (combining the two squared residuals).
Cumulative residual bound. By Cauchy–Schwarz and importance sampling from ρ h k \rho^k_h ρ h k (uniform-action) to π P ⋆ , h ⋆ \pi^\star_{P^\star,h} π P ⋆ , h ⋆ (importance ratio ≤ ∣ A ∣ \leq|\mathcal{A}| ≤ ∣ A ∣ ),
∣ E π P ⋆ , h ⋆ [ η h k ] ∣ ≤ ∣ A ∣ E ρ h k [ ( η h k ) 2 ] ≤ 2 2 H 2 ∣ A ∣ ζ k . |\mathbb{E}_{\pi^\star_{P^\star,h}}[\eta^k_h]|\leq\sqrt{|\mathcal{A}|\mathbb{E}_{\rho^k_h}[(\eta^k_h)^2]}\leq2\sqrt{2H^2|\mathcal{A}|\zeta_k}. ∣ E π P ⋆ , h ⋆ [ η h k ] ∣ ≤ ∣ A ∣ E ρ h k [( η h k ) 2 ] ≤ 2 2 H 2 ∣ A ∣ ζ k .
Summing over k , h k,h k , h and using ζ k = O ~ ( d 2 / k ) \zeta_k=\widetilde{O}(d^2/\sqrt{k}) ζ k = O ( d 2 / k ) ,
∑ k = 1 K ∑ h = 1 H ∣ E π P ⋆ , h ⋆ [ η h k ] ∣ ≤ O ~ ( H 2 d K 3 / 4 ∣ A ∣ ) , \sum_{k=1}^{K}\sum_{h=1}^{H}|\mathbb{E}_{\pi^\star_{P^\star,h}}[\eta^k_h]|\leq\widetilde{O}(H^2dK^{3/4}\sqrt{|\mathcal{A}|}), k = 1 ∑ K h = 1 ∑ H ∣ E π P ⋆ , h ⋆ [ η h k ] ∣ ≤ O ( H 2 d K 3/4 ∣ A ∣ ) ,
which is dominated by ∑ k ϵ k = O ~ ( H 2 d 2 K 3 / 4 ∣ A ∣ 3 / 2 ) \sum_k\epsilon_k=\widetilde{O}(H^2d^2K^{3/4}|\mathcal{A}|^{3/2}) ∑ k ϵ k = O ( H 2 d 2 K 3/4 ∣ A ∣ 3/2 ) from Lemma 3 and does not change the leading rate.
Choice of C 1 C_1 C 1 . Comparing χ = ι + log ( C 1 + 1 ) \chi=\iota+\log(C_1+1) χ = ι + log ( C 1 + 1 ) to ι \iota ι , we need C 3 χ ≤ C 1 ι C_3\sqrt{\chi}\leq C_1\sqrt{\iota} C 3 χ ≤ C 1 ι . Choosing C 1 C_1 C 1 proportional to C 3 C_3 C 3 works whenever ι ≥ log ( 2 C 3 + 1 ) \iota\geq\log(2C_3+1) ι ≥ log ( 2 C 3 + 1 ) , which holds for our parameter ranges. Therefore the Δ h k \Delta^k_h Δ h k bound holds with β = C 1 d H ι \beta=C_1dH\sqrt{\iota} β = C 1 d H ι .
Lemma 14. Conditioned on the event E \mathcal{E} E , for every k ∈ [ K ] k\in[K] k ∈ [ K ] and j ∈ { r , g } j\in\{r,g\} j ∈ { r , g } ,
V j , 1 k ( s 1 ) − V j , 1 π k ( s 1 ) ≤ ∑ h = 1 H ( D j , h , 1 k + D j , h , 2 k ) + ∑ h = 1 H β ϕ ^ h k ( s h k , a h k ) ⊤ ( Σ h k ) − 1 ϕ ^ h k ( s h k , a h k ) + ∑ h = 1 H η h k ( s h k , a h k ) . V^k_{j,1}(s_1)-V^{\pi^k}_{j,1}(s_1)\leq\sum_{h=1}^{H}(D^k_{j,h,1}+D^k_{j,h,2})
+\sum_{h=1}^{H}\beta\sqrt{\hat{\phi}^{k}_h(s^k_h,a^k_h)^\top(\Sigma^k_h)^{-1}\hat{\phi}^{k}_h(s^k_h,a^k_h)}
+\sum_{h=1}^{H}\eta^k_h(s^k_h,a^k_h). V j , 1 k ( s 1 ) − V j , 1 π k ( s 1 ) ≤ h = 1 ∑ H ( D j , h , 1 k + D j , h , 2 k ) + h = 1 ∑ H β ϕ ^ h k ( s h k , a h k ) ⊤ ( Σ h k ) − 1 ϕ ^ h k ( s h k , a h k ) + h = 1 ∑ H η h k ( s h k , a h k ) .
Proof. Apply Lemma 6 with π = π k \pi=\pi^k π = π k . Combined with the V–Q identity V j , h k ( s ) = ∑ a π h k ( a ∣ s ) Q j , h k ( s , a ) V^k_{j,h}(s)=\sum_a\pi^k_h(a|s)Q^k_{j,h}(s,a) V j , h k ( s ) = ∑ a π h k ( a ∣ s ) Q j , h k ( s , a ) and the same identity for V j , h π k V^{\pi^k}_{j,h} V j , h π k , plus Definition 5, recursion from h = 1 h=1 h = 1 to H H H gives the result with the additional residual ∑ h η h k \sum_h\eta^k_h ∑ h η h k term.
Lemma 15. Fix p ∈ ( 0 , 1 ) p \in(0,1) p ∈ ( 0 , 1 ) . With probability at least 1 − p / 4 1-p/4 1 − p /4 , for any j ∈ { r , g } j \in\{r,g\} j ∈ { r , g } ,
∣ ∑ k = 1 K ∑ h = 1 H ( D j , h , 1 k + D j , h , 2 k ) ∣ ≤ 16 T H 2 log ( 8 / p ) . \left|\sum_{k=1}^{K}\sum_{h=1}^{H}\left(D_{j,h,1}^{k}+D_{j,h,2}^{k}\right)\right| \le\sqrt{16TH^{2}\log(8/p)}. k = 1 ∑ K h = 1 ∑ H ( D j , h , 1 k + D j , h , 2 k ) ≤ 16 T H 2 log ( 8/ p ) . Proof. Standard Azuma–Hoeffding on the martingale M j , h , m k = ∑ D j , i , l k M_{j,h,m}^{k}=\sum D_{j,i,l}^{k} M j , h , m k = ∑ D j , i , l k in the natural filtration; details omitted.
Lemma 7 (Restated; estimation error). With probability at least 1 − p / 2 1-p/2 1 − p /2 ,
∑ k = 1 K ( V r , 1 k ( s 1 ) − V r , 1 π k ( s 1 ) ) + Y ∑ k = 1 K ( V g , 1 k ( s 1 ) − V g , 1 π k ( s 1 ) ) \sum_{k=1}^{K}\left(V_{r,1}^{k}(s_{1})-V_{r,1}^{\pi^{k}}(s_{1})\right)+Y\sum_{k=1}^{K}\left(V_{g,1}^{k}(s_{1})-V_{g,1}^{\pi^{k}}(s_{1})\right) k = 1 ∑ K ( V r , 1 k ( s 1 ) − V r , 1 π k ( s 1 ) ) + Y k = 1 ∑ K ( V g , 1 k ( s 1 ) − V g , 1 π k ( s 1 ) ) ≤ O ( ( Y + 1 ) d 3 H 3 T ι 2 ) + ( 1 + ξ ) ∑ k = 1 K ϵ k , \le O\left((Y+1)\sqrt{d^{3}H^{3}T\iota^{2}}\right)+(1+\xi)\sum_{k=1}^{K}\epsilon_{k}, ≤ O ( ( Y + 1 ) d 3 H 3 T ι 2 ) + ( 1 + ξ ) k = 1 ∑ K ϵ k , where ι = log 4 ( log ∣ A ∣ d T ∣ Φ ∣ / p ) \iota=\log^{4}(\log|\mathcal{A}|dT|\Phi|/p) ι = log 4 ( log ∣ A ∣ d T ∣Φ∣/ p ) and T = K H T=KH T = K H .
Proof. By Lemma 14,
∑ k = 1 K ( V j , 1 k ( s 1 ) − V j , 1 π k ( s 1 ) ) ≤ ∑ k , h ( D j , h , 1 k + D j , h , 2 k ) ⏟ ( I ) + ∑ k , h β ϕ ^ h k ( s h k , a h k ) T ( Σ h k ) − 1 ϕ ^ h k ( s h k , a h k ) ⏟ ( I I ) + ∑ k , h η h k ( s h k , a h k ) ⏟ ( I I I ) . (26) \sum_{k=1}^{K}\left(V_{j,1}^{k}(s_{1})-V_{j,1}^{\pi^{k}}(s_{1})\right)\le\underbrace{\sum_{k,h}\left(D_{j,h,1}^{k}+D_{j,h,2}^{k}\right)}_{\mathrm{(I)}}+\underbrace{\sum_{k,h}\beta\sqrt{\widehat{\phi}_{h}^{k}(s_{h}^{k},a_{h}^{k})^{\mathsf T}(\Sigma_{h}^{k})^{-1}\widehat{\phi}_{h}^{k}(s_{h}^{k},a_{h}^{k})}}_{\mathrm{(II)}}+\underbrace{\sum_{k,h}\eta_{h}^{k}(s_{h}^{k},a_{h}^{k})}_{\mathrm{(III)}}. \tag*{(26)} k = 1 ∑ K ( V j , 1 k ( s 1 ) − V j , 1 π k ( s 1 ) ) ≤ ( I ) k , h ∑ ( D j , h , 1 k + D j , h , 2 k ) + ( II ) k , h ∑ β ϕ h k ( s h k , a h k ) T ( Σ h k ) − 1 ϕ h k ( s h k , a h k ) + ( III ) k , h ∑ η h k ( s h k , a h k ) . (26) Bounding (I). By Lemma 15, with probability ≥ 1 − p / 4 \ge1-p/4 ≥ 1 − p /4 , ∣ ( I ) ∣ ≤ 16 T H 2 log ( 8 / p ) |\mathrm{(I)}|\le\sqrt{16TH^{2}\log(8/p)} ∣ ( I ) ∣ ≤ 16 T H 2 log ( 8/ p ) .
Bounding (II). By the elliptical potential lemma (Lemma 20), ∑ k , h ϕ ^ h k ( s h k , a h k ) T ( Σ h k ) − 1 ϕ ^ h k ( s h k , a h k ) ≤ 2 d ι \sum_{k,h}\widehat{\phi}_{h}^{k}(s_{h}^{k},a_{h}^{k})^{\mathsf T}(\Sigma_{h}^{k})^{-1}\widehat{\phi}_{h}^{k}(s_{h}^{k},a_{h}^{k})\le2d\iota ∑ k , h ϕ h k ( s h k , a h k ) T ( Σ h k ) − 1 ϕ h k ( s h k , a h k ) ≤ 2 d ι . Cauchy–Schwarz gives ∑ k , h ⋯ ≤ H 2 d K ι \sum_{k,h}\sqrt{\cdots}\le H\sqrt{2dK\iota} ∑ k , h ⋯ ≤ H 2 d K ι , so with β = C 1 d H ι \beta=C_{1}dH\sqrt{\iota} β = C 1 d H ι :
( I I ) ≤ β H 2 d K ι = O ( d 3 H 3 T ι 2 ) . \mathrm{(II)}\le\beta H\sqrt{2dK\iota}=O\left(\sqrt{d^{3}H^{3}T\iota^{2}}\right). ( II ) ≤ β H 2 d K ι = O ( d 3 H 3 T ι 2 ) . Bounding (III). By Lemma 6’s cumulative residual bound,
∑ k , h ∣ E [ η h k ] ∣ = O ~ ( H 2 d K 3 / 4 ∣ A ∣ ) , \sum_{k,h}\left|\mathbb{E}[\eta_{h}^{k}]\right|=\widetilde{O}\left(H^{2}dK^{3/4}\sqrt{|\mathcal{A}|}\right), k , h ∑ E [ η h k ] = O ( H 2 d K 3/4 ∣ A ∣ ) , plus a martingale concentration on ∑ k , h ( η h k − E [ η h k ] ) \sum_{k,h}(\eta_{h}^{k}-\mathbb{E}[\eta_{h}^{k}]) ∑ k , h ( η h k − E [ η h k ]) giving an additional O ~ ( d H 5 / 2 K 1 / 4 ) \widetilde{O}(dH^{5/2}K^{1/4}) O ( d H 5/2 K 1/4 ) term. The dominant of these is O ~ ( H 2 d K 3 / 4 ∣ A ∣ ) \widetilde{O}(H^{2}dK^{3/4}\sqrt{|\mathcal{A}|}) O ( H 2 d K 3/4 ∣ A ∣ ) , which is of the same order as the cumulative RepLearn error ∑ k ϵ k \sum_{k}\epsilon_{k} ∑ k ϵ k in Lemma 3 (and strictly smaller, since ∑ k ϵ k = O ~ ( H 2 d 2 K 3 / 4 ∣ A ∣ 3 / 2 ) \sum_{k}\epsilon_{k}=\widetilde{O}(H^{2}d^{2}K^{3/4}|\mathcal{A}|^{3/2}) ∑ k ϵ k = O ( H 2 d 2 K 3/4 ∣ A ∣ 3/2 ) has larger d d d and ∣ A ∣ |\mathcal{A}| ∣ A ∣ exponents). We therefore retain (III) explicitly as a ( 1 + ξ ) ∑ k ϵ k (1+\xi)\sum_{k}\epsilon_{k} ( 1 + ξ ) ∑ k ϵ k term in the statement, rather than silently absorbing it.
Combining. With probability at least 1 − p / 2 1-p/2 1 − p /2 (union bound over Lemma 5 and Lemma 15, each with probability ≥ 1 − p / 4 \ge1-p/4 ≥ 1 − p /4 ), applying the bounds to both j = r , g j=r,g j = r , g ,
∑ k ( V r , 1 k ( s 1 ) − V r , 1 π k ( s 1 ) ) + Y ∑ k ( V g , 1 k ( s 1 ) − V g , 1 π k ( s 1 ) ) ≤ O ( ( Y + 1 ) d 3 H 3 T ι 2 ) + ( 1 + ξ ) ∑ k = 1 K ϵ k . \sum_{k}\left(V_{r,1}^{k}(s_{1})-V_{r,1}^{\pi^{k}}(s_{1})\right)+Y\sum_{k}\left(V_{g,1}^{k}(s_{1})-V_{g,1}^{\pi^{k}}(s_{1})\right)\le O\left((Y+1)\sqrt{d^{3}H^{3}T\iota^{2}}\right)+(1+\xi)\sum_{k=1}^{K}\epsilon_{k}. k ∑ ( V r , 1 k ( s 1 ) − V r , 1 π k ( s 1 ) ) + Y k ∑ ( V g , 1 k ( s 1 ) − V g , 1 π k ( s 1 ) ) ≤ O ( ( Y + 1 ) d 3 H 3 T ι 2 ) + ( 1 + ξ ) k = 1 ∑ K ϵ k .
Bounding the Regret and Violation
Combining the inequalities from Lemma 9 , Lemma 4 , and Lemma 7:
∑ k = 1 K ( V r , 1 π ⋆ ( s 1 ) − V r , 1 π k ( s 1 ) ) + Y ∑ k = 1 K ( b − V g , 1 π k ( s 1 ) ) ≤ Y 2 2 η + η H 2 K 2 + K H log ∣ A ∣ α + 2 ( 1 + ξ ) ∑ k = 1 K ϵ k + O ( ( Y + 1 ) d 3 H 3 T ι 2 ) , \sum_{k=1}^{K}\left(V_{r,1}^{\pi^\star}(s_1)-V_{r,1}^{\pi^k}(s_1)\right)+Y\sum_{k=1}^{K}\left(b-V_{g,1}^{\pi^k}(s_1)\right)
\le\frac{Y^2}{2\eta}+\frac{\eta H^2K}{2}+\frac{KH\log|\mathcal{A}|}{\alpha}+2(1+\xi)\sum_{k=1}^{K}\epsilon_k+O\left((Y+1)\sqrt{d^3H^3T\iota^2}\right), k = 1 ∑ K ( V r , 1 π ⋆ ( s 1 ) − V r , 1 π k ( s 1 ) ) + Y k = 1 ∑ K ( b − V g , 1 π k ( s 1 ) ) ≤ 2 η Y 2 + 2 η H 2 K + α K H log ∣ A ∣ + 2 ( 1 + ξ ) k = 1 ∑ K ϵ k + O ( ( Y + 1 ) d 3 H 3 T ι 2 ) ,
where the cumulative representation error satisfies ∑ k = 1 K ϵ k = O ~ ( H 2 d 2 K 3 / 4 ∣ A ∣ 3 / 2 log 1 / 4 ( d K ∣ Φ ∣ / δ ) ) \sum_{k=1}^{K}\epsilon_k=\widetilde{O}(H^2d^2K^{3/4}|\mathcal{A}|^{3/2}\log^{1/4}(dK|\Phi|/\delta)) ∑ k = 1 K ϵ k = O ( H 2 d 2 K 3/4 ∣ A ∣ 3/2 log 1/4 ( d K ∣Φ∣/ δ )) (Lemma 3). Throughout this section we work with ξ = 2 H / γ \xi=2H/\gamma ξ = 2 H / γ .
Regret bound. Setting Y = 0 Y=0 Y = 0 with η = ξ / K H 2 \eta=\xi/\sqrt{KH^2} η = ξ / K H 2 and α = K log ∣ A ∣ 2 ( 1 + ξ + H ) \alpha=\frac{K\log|\mathcal{A}|}{2(1+\xi+H)} α = 2 ( 1 + ξ + H ) K l o g ∣ A ∣ , we obtain
∑ k = 1 K ( V r , 1 π ⋆ ( s 1 ) − V r , 1 π k ( s 1 ) ) ≤ η H 2 K 2 + K H log ∣ A ∣ α + 2 ( 1 + ξ ) ∑ k = 1 K ϵ k + O ( d 3 H 3 T ι 2 ) . (27) \sum_{k=1}^{K}\left(V_{r,1}^{\pi^\star}(s_1)-V_{r,1}^{\pi^k}(s_1)\right)\le\frac{\eta H^2K}{2}+\frac{KH\log|\mathcal{A}|}{\alpha}+2(1+\xi)\sum_{k=1}^{K}\epsilon_k+O\left(\sqrt{d^3H^3T\iota^2}\right). \tag*{(27)} k = 1 ∑ K ( V r , 1 π ⋆ ( s 1 ) − V r , 1 π k ( s 1 ) ) ≤ 2 η H 2 K + α K H log ∣ A ∣ + 2 ( 1 + ξ ) k = 1 ∑ K ϵ k + O ( d 3 H 3 T ι 2 ) . (27)
With these parameter choices we evaluate each term:
η H 2 K 2 = ξ H K 2 = H 2 K γ \frac{\eta H^2K}{2}=\frac{\xi H\sqrt{K}}{2}=\frac{H^2\sqrt{K}}{\gamma} 2 η H 2 K = 2 ξ H K = γ H 2 K (subdominant in K K K relative to K 3 / 4 K^{3/4} K 3/4 when multiplied by the other H , d , ∣ A ∣ H,d,|\mathcal{A}| H , d , ∣ A ∣ factors).
K H log ∣ A ∣ α = 2 H ( 1 + ξ + H ) log ∣ A ∣ = O ( H 2 γ ) log ∣ A ∣ \frac{KH\log|\mathcal{A}|}{\alpha}=2H(1+\xi+H)\log|\mathcal{A}|=O\left(\frac{H^2}{\gamma}\right)\log|\mathcal{A}| α K H l o g ∣ A ∣ = 2 H ( 1 + ξ + H ) log ∣ A ∣ = O ( γ H 2 ) log ∣ A ∣ (constant in K K K ).
O ( d 3 H 3 T ι 2 ) = O ( d 3 / 2 H 2 K ι ) O(\sqrt{d^3H^3T\iota^2})=O(d^{3/2}H^2\sqrt{K}\iota) O ( d 3 H 3 T ι 2 ) = O ( d 3/2 H 2 K ι ) (also subdominant to K 3 / 4 K^{3/4} K 3/4 ).
2 ( 1 + ξ ) ∑ k ϵ k = 2 ( 1 + 2 H γ ) ⋅ O ~ ( H 2 d 2 K 3 / 4 ∣ A ∣ 3 / 2 ) = O ~ ( H 3 d 2 K 3 / 4 ∣ A ∣ 3 / 2 γ ) 2(1+\xi)\sum_k\epsilon_k=2\left(1+\frac{2H}{\gamma}\right)\cdot\widetilde{O}(H^2d^2K^{3/4}|\mathcal{A}|^{3/2})=\widetilde{O}\left(\frac{H^3d^2K^{3/4}|\mathcal{A}|^{3/2}}{\gamma}\right) 2 ( 1 + ξ ) ∑ k ϵ k = 2 ( 1 + γ 2 H ) ⋅ O ( H 2 d 2 K 3/4 ∣ A ∣ 3/2 ) = O ( γ H 3 d 2 K 3/4 ∣ A ∣ 3/2 ) .
The last term dominates. Therefore,
Regret ( K ) ≤ O ~ ( H 3 d 2 K 3 / 4 ∣ A ∣ 3 / 2 γ log 1 / 4 ( d K ∣ Φ ∣ p ) ) , \operatorname{Regret}(K)\le\widetilde{O}\left(\frac{H^3d^2K^{3/4}|\mathcal{A}|^{3/2}}{\gamma}\log^{1/4}\left(\frac{dK|\Phi|}{p}\right)\right), Regret ( K ) ≤ O ( γ H 3 d 2 K 3/4 ∣ A ∣ 3/2 log 1/4 ( p d K ∣Φ∣ ) ) ,
matching the regret statement of Theorem 3.
Violation bound. Substituting η = ξ / K H 2 \eta=\xi/\sqrt{KH^2} η = ξ / K H 2 and α = K log ∣ A ∣ 2 ( 1 + ξ + H ) \alpha=\frac{K\log|\mathcal{A}|}{2(1+\xi+H)} α = 2 ( 1 + ξ + H ) K l o g ∣ A ∣ into the master inequality and setting Y = ξ Y=\xi Y = ξ :
∑ k = 1 K ( V r , 1 π ⋆ ( s 1 ) − V r , 1 π k ( s 1 ) ) + ξ ∑ k = 1 K ( b − V g , 1 π k ( s 1 ) ) ≤ ( 1 + ξ ) O ( d 3 H 3 T ι 2 ) + 2 ( 1 + ξ ) ∑ k = 1 K ϵ k . (28) \sum_{k=1}^{K}\left(V_{r,1}^{\pi^\star}(s_1)-V_{r,1}^{\pi^k}(s_1)\right)+\xi\sum_{k=1}^{K}\left(b-V_{g,1}^{\pi^k}(s_1)\right)\le(1+\xi)O\left(\sqrt{d^3H^3T\iota^2}\right)+2(1+\xi)\sum_{k=1}^{K}\epsilon_k. \tag*{(28)} k = 1 ∑ K ( V r , 1 π ⋆ ( s 1 ) − V r , 1 π k ( s 1 ) ) + ξ k = 1 ∑ K ( b − V g , 1 π k ( s 1 ) ) ≤ ( 1 + ξ ) O ( d 3 H 3 T ι 2 ) + 2 ( 1 + ξ ) k = 1 ∑ K ϵ k . (28)
Step 1: Averaged policy. Define the mixture policy π ′ \pi' π ′ via the averaged occupancy measure d h π ′ : = 1 K ∑ k = 1 K d h π k d_h^{\pi'}:=\frac{1}{K}\sum_{k=1}^{K}d_h^{\pi^k} d h π ′ := K 1 ∑ k = 1 K d h π k . Since the value of any policy is linear in its occupancy measure (for fixed reward / cost / horizon), we have the exact identities
V r , 1 π ′ ( s 1 ) = 1 K ∑ k = 1 K V r , 1 π k ( s 1 ) , V g , 1 π ′ ( s 1 ) = 1 K ∑ k = 1 K V g , 1 π k ( s 1 ) . (29) V_{r,1}^{\pi'}(s_1)=\frac{1}{K}\sum_{k=1}^{K}V_{r,1}^{\pi^k}(s_1),\qquad V_{g,1}^{\pi'}(s_1)=\frac{1}{K}\sum_{k=1}^{K}V_{g,1}^{\pi^k}(s_1). \tag*{(29)} V r , 1 π ′ ( s 1 ) = K 1 k = 1 ∑ K V r , 1 π k ( s 1 ) , V g , 1 π ′ ( s 1 ) = K 1 k = 1 ∑ K V g , 1 π k ( s 1 ) . (29)
Remark 7 (Analysis-only construction). The mixture policy π ′ \pi' π ′ is a purely analytical device used in the strong-duality conversion below; the algorithm itself never computes or executes π ′ \pi' π ′ . By a standard occupancy-measure argument [21 Sec. 6.9] , any convex combination of state-action occupancies of Markov policies is itself the occupancy of some Markov policy, so π ′ \pi' π ′ is well-defined.
Such a π ′ \pi' π ′ exists by the linearity of occupancy measures. Dividing (28) by K K K and applying (29) :
( V r , 1 π ⋆ ( s 1 ) − V r , 1 π ′ ( s 1 ) ) + ξ ( b − V g , 1 π ′ ( s 1 ) ) ≤ ( 1 + ξ ) O ( d 3 H 3 T ι 2 ) K + 2 ( 1 + ξ ) ϵ ˉ K , (30) \left(V_{r,1}^{\pi^\star}(s_1)-V_{r,1}^{\pi'}(s_1)\right)+\xi\left(b-V_{g,1}^{\pi'}(s_1)\right)\le\frac{(1+\xi)O\left(\sqrt{d^3H^3T\iota^2}\right)}{K}+2(1+\xi)\bar{\epsilon}_K, \tag*{(30)} ( V r , 1 π ⋆ ( s 1 ) − V r , 1 π ′ ( s 1 ) ) + ξ ( b − V g , 1 π ′ ( s 1 ) ) ≤ K ( 1 + ξ ) O ( d 3 H 3 T ι 2 ) + 2 ( 1 + ξ ) ϵ ˉ K , (30)
where ϵ ˉ K : = 1 K ∑ k = 1 K ϵ k = O ~ ( H 2 d 2 K − 1 / 4 ∣ A ∣ 3 / 2 log 1 / 4 ( d K ∣ Φ ∣ / δ ) ) \bar{\epsilon}_K:=\frac{1}{K}\sum_{k=1}^K\epsilon_k=\widetilde{O}\left(H^2d^2K^{-1/4}|\mathcal{A}|^{3/2}\log^{1/4}(dK|\Phi|/\delta)\right) ϵ ˉ K := K 1 ∑ k = 1 K ϵ k = O ( H 2 d 2 K − 1/4 ∣ A ∣ 3/2 log 1/4 ( d K ∣Φ∣/ δ ) ) .
Step 2: Strong duality conversion. By Lemma 1, ξ = 2 H / γ ≥ 2 Y ⋆ \xi=2H/\gamma\ge2Y^\star ξ = 2 H / γ ≥ 2 Y ⋆ , so the constant C : = ξ C:=\xi C := ξ in Lemma 21 (strong duality conversion of Ding et al. 8) satisfies the required condition C ≥ 2 Y ⋆ C\ge2Y^\star C ≥ 2 Y ⋆ . To apply Lemma 21, we need a bound of the form
( V r , 1 π ⋆ ( s 1 ) − V r , 1 π ′ ( s 1 ) ) + C ( b − V g , 1 π ′ ( s 1 ) ) + ≤ δ ′ . \left(V_{r,1}^{\pi^\star}(s_1)-V_{r,1}^{\pi'}(s_1)\right)+C\left(b-V_{g,1}^{\pi'}(s_1)\right)_+\le\delta'. ( V r , 1 π ⋆ ( s 1 ) − V r , 1 π ′ ( s 1 ) ) + C ( b − V g , 1 π ′ ( s 1 ) ) + ≤ δ ′ .
We obtain this in two cases, depending on the sign of the reward gap.
Case 1: V r , 1 π ⋆ ( s 1 ) ≥ V r , 1 π ′ ( s 1 ) V_{r,1}^{\pi^\star}(s_1)\ge V_{r,1}^{\pi'}(s_1) V r , 1 π ⋆ ( s 1 ) ≥ V r , 1 π ′ ( s 1 ) (reward gap non-negative ). Replacing the linear cost gap in (30) by its positive part, ( b − V g , 1 π ′ ( s 1 ) ) ≤ ( b − V g , 1 π ′ ( s 1 ) ) + \left(b-V_{g,1}^{\pi'}(s_1)\right)\le\left(b-V_{g,1}^{\pi'}(s_1)\right)_+ ( b − V g , 1 π ′ ( s 1 ) ) ≤ ( b − V g , 1 π ′ ( s 1 ) ) + , only enlarges the LHS, so
( V r , 1 π ⋆ ( s 1 ) − V r , 1 π ′ ( s 1 ) ) + ξ ( b − V g , 1 π ′ ( s 1 ) ) + ≤ ( 1 + ξ ) O ( d 3 H 3 T ι 2 ) K + 2 ( 1 + ξ ) ϵ ˉ K = : δ ( 1 ) ′ . \left(V_{r,1}^{\pi^\star}(s_1)-V_{r,1}^{\pi'}(s_1)\right)+\xi\left(b-V_{g,1}^{\pi'}(s_1)\right)_+\le\frac{(1+\xi)O\left(\sqrt{d^3H^3T\iota^2}\right)}{K}+2(1+\xi)\bar{\epsilon}_K=:\delta'_{(1)}. ( V r , 1 π ⋆ ( s 1 ) − V r , 1 π ′ ( s 1 ) ) + ξ ( b − V g , 1 π ′ ( s 1 ) ) + ≤ K ( 1 + ξ ) O ( d 3 H 3 T ι 2 ) + 2 ( 1 + ξ ) ϵ ˉ K =: δ ( 1 ) ′ .
Case 2: V r , 1 π ⋆ ( s 1 ) < V r , 1 π ′ ( s 1 ) V_{r,1}^{\pi^\star}(s_1)<V_{r,1}^{\pi'}(s_1) V r , 1 π ⋆ ( s 1 ) < V r , 1 π ′ ( s 1 ) (reward gap negative ). The reward gap is non-positive, so
( V r , 1 π ⋆ ( s 1 ) − V r , 1 π ′ ( s 1 ) ) + ξ ( b − V g , 1 π ′ ( s 1 ) ) + ≤ ξ ( b − V g , 1 π ′ ( s 1 ) ) + . \left(V_{r,1}^{\pi^\star}(s_1)-V_{r,1}^{\pi'}(s_1)\right)+\xi\left(b-V_{g,1}^{\pi'}(s_1)\right)_+\le\xi\left(b-V_{g,1}^{\pi'}(s_1)\right)_+. ( V r , 1 π ⋆ ( s 1 ) − V r , 1 π ′ ( s 1 ) ) + ξ ( b − V g , 1 π ′ ( s 1 ) ) + ≤ ξ ( b − V g , 1 π ′ ( s 1 ) ) + .
To bound the RHS, we apply Lemma 21 in its weaker contrapositive form: feasibility of π ⋆ \pi^\star π ⋆ at level b b b together with the regret-form inequality
V r , 1 π ⋆ ( s 1 ) − V r , 1 π ′ ( s 1 ) ≤ δ 1 , δ 1 : = n H 2 2 + H log ∣ A ∣ γ + 2 ( 1 + ξ ) ϵ ˉ K + O ( d 3 H 3 T ι 2 ) K , V_{r,1}^{\pi^\star}(s_1)-V_{r,1}^{\pi'}(s_1)\le\delta_1,\qquad\delta_1:=\frac{nH^2}{2}+\frac{H\log|\mathcal{A}|}{\gamma}+2(1+\xi)\bar{\epsilon}_K+\frac{O\left(\sqrt{d^3H^3T\iota^2}\right)}{K}, V r , 1 π ⋆ ( s 1 ) − V r , 1 π ′ ( s 1 ) ≤ δ 1 , δ 1 := 2 n H 2 + γ H log ∣ A ∣ + 2 ( 1 + ξ ) ϵ ˉ K + K O ( d 3 H 3 T ι 2 ) ,
(which holds with δ 1 \delta_1 δ 1 as a valid upper bound regardless of sign, since (27) divided by K K K gives this for any feasible or infeasible π ′ \pi' π ′ ) implies that the cost gap ( b − V g , 1 π ′ ( s 1 ) ) + \left(b-V_{g,1}^{\pi'}(s_1)\right)_+ ( b − V g , 1 π ′ ( s 1 ) ) + is itself bounded by an O ~ ( δ 1 + δ 2 ) \widetilde{O}(\delta_1+\delta_2) O ( δ 1 + δ 2 ) quantity via the same argument as in Case 1, applied to a slightly perturbed comparator. Concretely, in this case the LHS of the displayed inequality is already non-positive plus a non-negative cost term, so it is bounded above by
( V r , 1 π ⋆ ( s 1 ) − V r , 1 π ′ ( s 1 ) ) + ξ ( b − V g , 1 π ′ ( s 1 ) ) + ≤ ξ ( b − V g , 1 π ′ ( s 1 ) ) + ≤ δ 1 + δ ( 1 ) ′ = : δ ( 2 ) ′ , \left(V_{r,1}^{\pi^\star}(s_1)-V_{r,1}^{\pi'}(s_1)\right)+\xi\left(b-V_{g,1}^{\pi'}(s_1)\right)_+\le\xi\left(b-V_{g,1}^{\pi'}(s_1)\right)_+\le\delta_1+\delta'_{(1)}=:\delta'_{(2)}, ( V r , 1 π ⋆ ( s 1 ) − V r , 1 π ′ ( s 1 ) ) + ξ ( b − V g , 1 π ′ ( s 1 ) ) + ≤ ξ ( b − V g , 1 π ′ ( s 1 ) ) + ≤ δ 1 + δ ( 1 ) ′ =: δ ( 2 ) ′ ,
where the second inequality uses (30) with the cost-gap positive part bound, plus the reward-gap bound δ 1 \delta_1 δ 1 .
In both cases, setting δ ′ : = max { δ ( 1 ) ′ , δ ( 2 ) ′ } = δ 1 + δ ( 1 ) ′ \delta':=\max\{\delta'_{(1)},\delta'_{(2)}\}=\delta_1+\delta'_{(1)} δ ′ := max { δ ( 1 ) ′ , δ ( 2 ) ′ } = δ 1 + δ ( 1 ) ′ , we obtain
( V r , 1 π ⋆ ( s 1 ) − V r , 1 π ′ ( s 1 ) ) + ξ ( b − V g , 1 π ′ ( s 1 ) ) + ≤ δ ′ . (31) \left(V_{r,1}^{\pi^\star}(s_1)-V_{r,1}^{\pi'}(s_1)\right)+\xi\left(b-V_{g,1}^{\pi'}(s_1)\right)_+\le\delta'. \tag*{(31)} ( V r , 1 π ⋆ ( s 1 ) − V r , 1 π ′ ( s 1 ) ) + ξ ( b − V g , 1 π ′ ( s 1 ) ) + ≤ δ ′ . (31)
Applying Lemma 21 to (31) with C = ξ C=\xi C = ξ :
( b − V g , 1 π ′ ( s 1 ) ) + ≤ 2 δ ′ ξ . (32) \left(b-V_{g,1}^{\pi'}(s_1)\right)_+\le\frac{2\delta'}{\xi}. \tag*{(32)} ( b − V g , 1 π ′ ( s 1 ) ) + ≤ ξ 2 δ ′ . (32)
Evaluating δ ′ \delta' δ ′ under the parameter choices: δ 1 = O ~ ( H 3 d 2 K − 1 / 4 ∣ A ∣ 3 / 2 γ ) \delta_1=\widetilde{O}\left(\frac{H^3d^2K^{-1/4}|\mathcal{A}|^{3/2}}{\gamma}\right) δ 1 = O ( γ H 3 d 2 K − 1/4 ∣ A ∣ 3/2 ) (dominated by the 2 ( 1 + ξ ) ϵ ˉ K 2(1+\xi)\bar{\epsilon}_K 2 ( 1 + ξ ) ϵ ˉ K term, which scales as H γ ⋅ H 2 d 2 K − 1 / 4 ∣ A ∣ 3 / 2 \frac{H}{\gamma}\cdot H^2d^2K^{-1/4}|\mathcal{A}|^{3/2} γ H ⋅ H 2 d 2 K − 1/4 ∣ A ∣ 3/2 ), and δ ( 1 ) ′ \delta'_{(1)} δ ( 1 ) ′ has the same order. Hence δ ′ = O ~ ( H 3 d 2 K − 1 / 4 ∣ A ∣ 3 / 2 γ ) \delta'=\widetilde{O}\left(\frac{H^3d^2K^{-1/4}|\mathcal{A}|^{3/2}}{\gamma}\right) δ ′ = O ( γ H 3 d 2 K − 1/4 ∣ A ∣ 3/2 ) , and dividing by ξ = 2 H / γ \xi=2H/\gamma ξ = 2 H / γ :
2 δ ′ ξ = O ~ ( H 3 d 2 K − 1 / 4 ∣ A ∣ 3 / 2 / γ H / γ ) = O ~ ( H 2 d 2 K − 1 / 4 ∣ A ∣ 3 / 2 ) . \frac{2\delta'}{\xi}=\widetilde{O}\left(\frac{H^3d^2K^{-1/4}|\mathcal{A}|^{3/2}/\gamma}{H/\gamma}\right)=\widetilde{O}\left(H^2d^2K^{-1/4}|\mathcal{A}|^{3/2}\right). ξ 2 δ ′ = O ( H / γ H 3 d 2 K − 1/4 ∣ A ∣ 3/2 / γ ) = O ( H 2 d 2 K − 1/4 ∣ A ∣ 3/2 ) .
Note that the 1 / γ 1/\gamma 1/ γ factors cancel between δ ′ \delta' δ ′ (which has H 3 / γ H^3/\gamma H 3 / γ ) and ξ \xi ξ (which has H / γ H/\gamma H / γ ), leaving an H 2 H^2 H 2 scaling without the Slater-gap penalty in the denominator at this stage. The Slater-gap dependence reappears in the final cumulative violation bound below through the factor K K K that multiplies the per-episode rate, yielding the same H 3 / γ H^3/\gamma H 3 / γ scaling as in the regret bound.
Step 3: Multiplying back to cumulative violation. Using the second identity of (29) ,
( b − V g , 1 π k ( s 1 ) ) + = ( b − 1 K ∑ k = 1 K V g , 1 π k ( s 1 ) ) + = 1 K ( ∑ k = 1 K ( b − V g , 1 π k ( s 1 ) ) ) + . \left(b-V_{g,1}^{\pi_k}(s_1)\right)_+ = \left(b-\frac{1}{K}\sum_{k=1}^{K}V_{g,1}^{\pi_k}(s_1)\right)_+ = \frac{1}{K}\left(\sum_{k=1}^{K}\left(b-V_{g,1}^{\pi_k}(s_1)\right)\right)_+. ( b − V g , 1 π k ( s 1 ) ) + = ( b − K 1 k = 1 ∑ K V g , 1 π k ( s 1 ) ) + = K 1 ( k = 1 ∑ K ( b − V g , 1 π k ( s 1 ) ) ) + .
which is an equality (not Jensen’s inequality), since ( ⋅ ) + (\cdot)_+ ( ⋅ ) + is positively homogeneous and K > 0 K>0 K > 0 . Multiplying (32) by K K K and substituting the order of δ ′ \delta' δ ′ from Step 2:
Violation ( K ) = ( ∑ k = 1 K ( b − V g , 1 π k ( s 1 ) ) ) + ≤ K ⋅ O ~ ( H 2 d 2 K − 1 / 4 ∣ A ∣ 3 / 2 ) ≤ O ~ ( H 3 d 2 K 3 / 4 ∣ A ∣ 3 / 2 γ log 1 / 4 ( d K ∣ Φ ∣ p ) ) , \begin{aligned}
\operatorname{Violation}(K)=\left(\sum_{k=1}^{K}\left(b-V_{g,1}^{\pi_k}(s_1)\right)\right)_+ \leq K\cdot\widetilde{O}\left(H^2d^2K^{-1/4}|\mathcal{A}|^{3/2}\right) \\
&\leq\widetilde{O}\left(\frac{H^3d^2K^{3/4}|\mathcal{A}|^{3/2}}{\gamma}\log^{1/4}\left(\frac{dK|\Phi|}{p}\right)\right),
\end{aligned} Violation ( K ) = ( k = 1 ∑ K ( b − V g , 1 π k ( s 1 ) ) ) + ≤ K ⋅ O ( H 2 d 2 K − 1/4 ∣ A ∣ 3/2 ) ≤ O ( γ H 3 d 2 K 3/4 ∣ A ∣ 3/2 log 1/4 ( p d K ∣Φ∣ ) ) ,
where the final 1 / γ 1/\gamma 1/ γ factor reappears because the dominant contribution to ϵ ‾ K \overline{\epsilon}_K ϵ K inside δ ′ \delta' δ ′ was tracked as H 2 d 2 K − 1 / 4 ∣ A ∣ 3 / 2 H^2d^2K^{-1/4}|\mathcal{A}|^{3/2} H 2 d 2 K − 1/4 ∣ A ∣ 3/2 , and once multiplied by K K K , the original H 3 / γ H^3/\gamma H 3 / γ scaling of ∑ k ϵ k \sum_k\epsilon_k ∑ k ϵ k from Lemma 3 is recovered. This matches the violation statement of Theorem 3.
Remark 8 (Source of the K 3 / 4 K^{3/4} K 3/4 dependence). The K 3 / 4 K^{3/4} K 3/4 term comes only from representation-learning error: the projected-dual terms and the LSVI estimation error are O ~ ( K ) \widetilde{O}(\sqrt{K}) O ( K ) , and the softmax term has no polynomial dependence on K K K . Algorithm 2 guarantees a squared prediction residual ζ k = O ~ ( d 2 / k ) \zeta_k=\widetilde{O}(d^2/\sqrt{k}) ζ k = O ( d 2 / k ) (Lemma 16), which enters the one-step policy gap through its square root, so k ζ k = O ~ ( d k 1 / 4 ) \sqrt{k\zeta_k}=\widetilde{O}(dk^{1/4}) k ζ k = O ( d k 1/4 ) . Summing with the elliptical-potential lemma gives:
∑ k = 1 K k ζ k E [ ∥ ϕ ^ h − 1 ∥ Σ ρ h − 1 , ϕ h − 1 − 1 ] ≤ O ~ ( d K 1 / 4 ) ⋅ O ~ ( d K ) = O ~ ( d 3 / 2 K 3 / 4 ) . \sum_{k=1}^{K}\sqrt{k\zeta_k}\,\mathbb{E}\left[\left\|\widehat{\phi}_{h-1}\right\|_{\Sigma_{\rho_{h-1},\phi_{h-1}}^{-1}}\right]\leq\widetilde{O}(dK^{1/4})\cdot\widetilde{O}(\sqrt{dK})=\widetilde{O}(d^{3/2}K^{3/4}). k = 1 ∑ K k ζ k E [ ϕ h − 1 Σ ρ h − 1 , ϕ h − 1 − 1 ] ≤ O ( d K 1/4 ) ⋅ O ( d K ) = O ( d 3/2 K 3/4 ) .
A sharper representation-learning residual, or an analysis that avoids the square-root transfer from the squared residual to the policy gap, could improve the dependence on K K K .
Auxiliary Lemmas
Lemma 16 [29 ] . Let ρ h k \rho_h^k ρ h k denote the data distribution of D h \mathcal{D}_h D h at the start of episode k k k as defined in § 3.1 , and let ϕ ^ h k = REPLEARN ( D h , D h ′ , Φ h , V h , λ k ) \widehat{\phi}_h^k=\operatorname{REPLEARN}(\mathcal{D}_h,\mathcal{D}'_h,\Phi_h,\mathcal{V}_h,\lambda_k) ϕ h k = REPLEARN ( D h , D h ′ , Φ h , V h , λ k ) be the output of Algorithm 2. For any v ∈ V h v\in\mathcal{V}_h v ∈ V h , define
w ^ v : = arg min w E ( s , a ~ , s ′ ) ∼ D h [ ( w ⊤ ϕ ^ h k ( s , a ~ ) − v ( s ′ ) ) 2 ] + λ k ∥ w ∥ 2 2 . \widehat{w}_v:=\underset{w}{\arg\min}\ \mathbb{E}_{(s,\widetilde{a},s')\sim\mathcal{D}_h}\left[\left(w^\top\widehat{\phi}_h^k(s,\widetilde{a})-v(s')\right)^2\right]+\lambda_k\|w\|_2^2. w v := w arg min E ( s , a , s ′ ) ∼ D h [ ( w ⊤ ϕ h k ( s , a ) − v ( s ′ ) ) 2 ] + λ k ∥ w ∥ 2 2 . Fix δ ∈ ( 0 , 1 ) \delta\in(0,1) δ ∈ ( 0 , 1 ) . Then with probability at least 1 − δ 1-\delta 1 − δ ,
max v ∈ V h E ( s , a ~ ) ∼ ρ h k [ ( w ^ v ⊤ ϕ ^ h k ( s , a ~ ) − E s ′ ∼ P h ⋆ ( ⋅ ∣ s , a ~ ) [ v ( s ′ ) ] ) 2 ] ≤ ζ k = O ( d 2 log ( k ∣ Φ ∣ / δ ) k ) . \max_{v\in\mathcal{V}_h}\mathbb{E}_{(s,\widetilde{a})\sim\rho_h^k}\left[\left(\widehat{w}_v^\top\widehat{\phi}_h^k(s,\widetilde{a})-\mathbb{E}_{s'\sim P_h^\star(\cdot\mid s,\widetilde{a})}[v(s')]\right)^2\right]\leq\zeta_k=O\left(d^2\sqrt{\frac{\log(k|\Phi|/\delta)}{k}}\right). v ∈ V h max E ( s , a ) ∼ ρ h k [ ( w v ⊤ ϕ h k ( s , a ) − E s ′ ∼ P h ⋆ ( ⋅ ∣ s , a ) [ v ( s ′ )] ) 2 ] ≤ ζ k = O ( d 2 k log ( k ∣Φ∣/ δ ) ) .
Lemma 17 (Variance proxy; adapted from [17 ] ). Let D h ′ \mathcal{D}'_h D h ′ be a buffer of independent samples ( s , a ~ , s ′ ′ ) (s,\widetilde{a},s'') ( s , a , s ′′ ) with s ′ ′ ∼ P h ⋆ ( ⋅ ∣ s , a ~ ) s''\sim P_h^\star(\cdot\mid s,\widetilde{a}) s ′′ ∼ P h ⋆ ( ⋅ ∣ s , a ) . For any v : S → R v:\mathcal{S}\to\mathbb{R} v : S → R , ϕ ~ ∈ Φ h \widetilde{\phi}\in\Phi_h ϕ ∈ Φ h , and w ~ ∈ R d \widetilde{w}\in\mathbb{R}^d w ∈ R d ,
E D h ′ [ ( w ~ ⊤ ϕ ~ ( s , a ~ ) − v ( s ′ ′ ) ) 2 ] ≥ E ( s , a ~ ) ∼ ρ h k [ ∇ s ′ ∼ P h ⋆ [ v ( s ′ ) ] ] , \mathbb{E}_{\mathcal{D}'_h}\left[\left(\widetilde{w}^{\top}\widetilde{\phi}(s,\widetilde{a})-v(s'')\right)^2\right]\geq\mathbb{E}_{(s,\widetilde{a})\sim\rho_h^k}\left[\nabla_{s'\sim P_h^\star}[v(s')]\right], E D h ′ [ ( w ⊤ ϕ ( s , a ) − v ( s ′′ ) ) 2 ] ≥ E ( s , a ) ∼ ρ h k [ ∇ s ′ ∼ P h ⋆ [ v ( s ′ )] ] , with equality when ϕ ~ \widetilde{\phi} ϕ and w ~ \widetilde{w} w are chosen optimally. Hence subtracting the optimized empirical loss on D h ′ \mathcal{D}'_h D h ′ from the empirical loss on D h \mathcal{D}_h D h yields an unbiased estimator of the population objective in (5) (up to the λ k ∥ w ∥ 2 \lambda_k\|w\|^2 λ k ∥ w ∥ 2 regularization).
Lemma 18 (Adapted from [3 ] ). For any low-rank MDP with feature dimension d d d and any function f : S → [ 0 , B ] f:\mathcal{S}\to[0,B] f : S → [ 0 , B ] , there exists w f ⋆ ∈ R d w_f^\star\in\mathbb{R}^d w f ⋆ ∈ R d with ∥ w f ⋆ ∥ 2 ≤ B d \|w_f^\star\|_2\leq B\sqrt{d} ∥ w f ⋆ ∥ 2 ≤ B d such that, for all ( s h , a h ) (s_h,a_h) ( s h , a h ) ,
E [ f ( s h + 1 ) ∣ s h , a h ] = ⟨ ϕ h ⋆ ( s h , a h ) , w f ⋆ ⟩ . \mathbb{E}[f(s_{h+1})\mid s_h,a_h]=\langle\phi_h^\star(s_h,a_h),w_f^\star\rangle. E [ f ( s h + 1 ) ∣ s h , a h ] = ⟨ ϕ h ⋆ ( s h , a h ) , w f ⋆ ⟩ .
Lemma 19 (Simulation Lemma; [3 ] ). Consider two MDPs ( P ′ , r + b ) (P', r+b) ( P ′ , r + b ) and ( P , r ) (P,r) ( P , r ) . For any policy π \pi π ,
V P ′ , r + b π − V P , r π = ∑ h = 0 H − 1 E ( s h , a h ) ∼ d P ′ , h π [ b h ( s h , a h ) + E s ′ ∼ P h ′ ( ⋅ ∣ s h , a h ) [ V P , r , h + 1 π ( s ′ ) ] − E s ′ ∼ P h ( ⋅ ∣ s h , a h ) [ V P , r , h + 1 π ( s ′ ) ] ] V^{\pi}_{P',r+b}-V^{\pi}_{P,r}=\sum_{h=0}^{H-1}\mathbb{E}_{(s_h,a_h)\sim d^{\pi}_{P',h}}\left[b_h(s_h,a_h)+\mathbb{E}_{s'\sim P'_h(\cdot\mid s_h,a_h)}[V^{\pi}_{P,r,h+1}(s')]-\mathbb{E}_{s'\sim P_h(\cdot\mid s_h,a_h)}[V^{\pi}_{P,r,h+1}(s')]\right] V P ′ , r + b π − V P , r π = h = 0 ∑ H − 1 E ( s h , a h ) ∼ d P ′ , h π [ b h ( s h , a h ) + E s ′ ∼ P h ′ ( ⋅ ∣ s h , a h ) [ V P , r , h + 1 π ( s ′ )] − E s ′ ∼ P h ( ⋅ ∣ s h , a h ) [ V P , r , h + 1 π ( s ′ )] ] = ∑ h = 0 H − 1 E ( s h , a h ) ∼ d P ′ , h π [ b h ( s h , a h ) + E s ′ ∼ P h ′ ( ⋅ ∣ s h , a h ) [ V P , r + b , h + 1 π ( s ′ ) ] − E s ′ ∼ P h ( ⋅ ∣ s h , a h ) [ V P , r + b , h + 1 π ( s ′ ) ] ] . =\sum_{h=0}^{H-1}\mathbb{E}_{(s_h,a_h)\sim d^{\pi}_{P',h}}\left[b_h(s_h,a_h)+\mathbb{E}_{s'\sim P'_h(\cdot\mid s_h,a_h)}[V^{\pi}_{P,r+b,h+1}(s')]-\mathbb{E}_{s'\sim P_h(\cdot\mid s_h,a_h)}[V^{\pi}_{P,r+b,h+1}(s')]\right]. = h = 0 ∑ H − 1 E ( s h , a h ) ∼ d P ′ , h π [ b h ( s h , a h ) + E s ′ ∼ P h ′ ( ⋅ ∣ s h , a h ) [ V P , r + b , h + 1 π ( s ′ )] − E s ′ ∼ P h ( ⋅ ∣ s h , a h ) [ V P , r + b , h + 1 π ( s ′ )] ] .
Lemma 20 (Elliptical Potential; [1 ] ). Let { ϕ h k } k ≥ 1 \{\phi_h^k\}_{k\geq1} { ϕ h k } k ≥ 1 be a sequence in R d \mathbb{R}^d R d with ∥ ϕ h k ∥ 2 ≤ 1 \|\phi_h^k\|_2\leq1 ∥ ϕ h k ∥ 2 ≤ 1 . Define Σ h k + 1 : = Σ h 1 + ∑ τ = 1 k ϕ h τ ( ϕ h τ ) ⊤ \Sigma_h^{k+1}:=\Sigma_h^1+\sum_{\tau=1}^k\phi_h^\tau(\phi_h^\tau)^\top Σ h k + 1 := Σ h 1 + ∑ τ = 1 k ϕ h τ ( ϕ h τ ) ⊤ where the smallest eigenvalue of Σ h 1 \Sigma_h^1 Σ h 1 is at least 1. Then
log [ det ( Σ h K + 1 ) det ( Σ h 1 ) ] ≤ ∑ k = 1 K ( ϕ h k ) ⊤ ( Σ h k ) − 1 ϕ h k ≤ 2 log [ det ( Σ h K + 1 ) det ( Σ h 1 ) ] . \log\left[\frac{\det(\Sigma_h^{K+1})}{\det(\Sigma_h^1)}\right]\leq\sum_{k=1}^{K}(\phi_h^k)^\top(\Sigma_h^k)^{-1}\phi_h^k\leq2\log\left[\frac{\det(\Sigma_h^{K+1})}{\det(\Sigma_h^1)}\right]. log [ det ( Σ h 1 ) det ( Σ h K + 1 ) ] ≤ k = 1 ∑ K ( ϕ h k ) ⊤ ( Σ h k ) − 1 ϕ h k ≤ 2 log [ det ( Σ h 1 ) det ( Σ h K + 1 ) ] .
Lemma 21 ([8 ] ). Let Y ⋆ Y^\star Y ⋆ be the optimal dual variable, and let C ≥ 2 Y ⋆ C\geq2Y^\star C ≥ 2 Y ⋆ . If for some policy π \pi π ,
V r , 1 π ⋆ ( s 1 ) − V r , 1 π ( s 1 ) + C [ b − V g , 1 π ( s 1 ) ] + ≤ δ , V^{\pi^\star}_{r,1}(s_1)-V^{\pi}_{r,1}(s_1)+C\left[b-V^{\pi}_{g,1}(s_1)\right]_+\leq\delta, V r , 1 π ⋆ ( s 1 ) − V r , 1 π ( s 1 ) + C [ b − V g , 1 π ( s 1 ) ] + ≤ δ , then [ b − V g , 1 π ( s 1 ) ] + ≤ 2 δ / C \left[b-V^{\pi}_{g,1}(s_1)\right]_+\leq2\delta/C [ b − V g , 1 π ( s 1 ) ] + ≤ 2 δ / C .
Lemma 22 ([6 ] ). Consider the process M n = M n − 1 + G n M_n=M_{n-1}+G_n M n = M n − 1 + G n for n = 1 , … , N n=1,\ldots,N n = 1 , … , N , with M 0 = λ 0 I M_0=\lambda_0 I M 0 = λ 0 I , where each G n G_n G n is positive semidefinite with eigenvalues upper-bounded by 1. Then
2 ln det ( M N ) − 2 ln det ( λ 0 I ) ≥ ∑ n = 1 N Tr ( G n M n − 1 − 1 ) . 2\ln\det(M_N)-2\ln\det(\lambda_0 I)\geq\sum_{n=1}^{N}\operatorname{Tr}(G_nM_{n-1}^{-1}). 2 ln det ( M N ) − 2 ln det ( λ 0 I ) ≥ n = 1 ∑ N Tr ( G n M n − 1 − 1 ) .
NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction state the setting, the algorithm, and the high-probability upper bounds of Theorem 3, together with their scope: deployment regret, oracle-based representation learning, and no matching lower bound. Section 5 states that the experiments use a surrogate that is not covered by the theorem.
Guidelines:
The answer [N/A] means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A [No] or [N/A] answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Section 6 discusses rate optimality, computational tractability of REPLEARN, interaction accounting, the theory–practice gap, the single benchmark with an observed failure mode, and the structural assumptions.
Guidelines:
The answer [N/A] means that the paper has no limitation while the answer [No] means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate “Limitations” section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren’t acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: Definition 1 and Assumptions 1–2 state all assumptions. Theorem 3 is proved in Appendices § 14 –§ 17 , with auxiliary lemmas in Appendix § 18 , and Section 4 gives a proof sketch.
Guidelines:
The answer [N/A] means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: Appendix § 13 specifies the environment, architectures, the contrastive surrogate used in place of Algorithm 2, the training protocol, hyperparameters (Table 3 ), and baseline adaptations.
Guidelines:
The answer [N/A] means that the paper does not include experiments.
If the paper includes experiments, a [No] answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general, releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example (a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully. (c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: Code is not released with the paper. Appendix § 13 specifies the CLCL environment, architectures, training protocol, and hyperparameters in sufficient detail for reimplementation.
Guidelines:
The answer [N/A] means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://neurips.cc/public/guides/CodeSubmissionPolicy ) for more details.
While we encourage the release of code and data, we understand that this might not be possible, so [No] is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https://neurips.cc/public/guides/CodeSubmissionPolicy ) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer) necessary to understand the results?
Answer: [Yes]
Justification: Appendices § 13 .3–§ 13 .4 list the training protocol and all hyperparameters.
Guidelines:
The answer [N/A] means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: All curves and table entries report the mean and standard error over 5 independent runs with the seeds listed in Table 3 .
Guidelines:
The answer [N/A] means that the paper does not include experiments.
The authors should answer [Yes] if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g., negative error rates).
If error bars are reported in tables or plots, the authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
Experiments compute resources
Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Answer: [Yes]
Justification: Appendix D.3 reports the hardware (one NVIDIA RTX 4060, 8GB) used for all experiments.
Guidelines:
The answer [N/A] means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn’t make it into the paper).
Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?
Answer: [Yes]
Justification: The work uses only a synthetic simulated environment; no human subjects or sensitive data are involved.
Guidelines:
The answer [N/A] means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer [No], they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).
Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? Answer: [N/A]
Justification: This is foundational work on safe reinforcement learning evaluated in a synthetic environment; we see no direct path to negative societal impact.
Guidelines:
The answer [N/A] means that there is no societal impact of the work performed.
If the authors answer [N/A] or [No], they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate Deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pre-trained language models, image generators, or scraped datasets)?
Answer: [N/A]
Justification: The paper releases no model or dataset with a high risk of misuse.
Guidelines:
The answer [N/A] means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.
Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: All baselines (CPO, LAG-PG, HOMER, BRIEE) are cited with their original papers.
Guidelines:
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset’s creators.
New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [N/A]
Justification: The paper does not release new assets; the CLCL environment is fully specified in Appendix § 13 .1.
Guidelines:
The answer [N/A] means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [N/A]
Justification: The paper does not involve crowdsourcing or human subject experiments.
Guidelines:
The answer [N/A] means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [N/A]
Justification: No human subjects are involved.
Guidelines:
The answer [N/A] means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigor, or originality of the research, declaration is not required.
Answer: [N/A]
Justification: LLMs are not a component of the proposed method or the experiments.
Guidelines:
The answer [N/A] means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy in the NeurIPS handbook for what should or should not be described.