Introduction
Multi-step temporal-difference (TD) learning propagates rewards over longer horizons, making it a central tool for policy evaluation and credit assignment in reinforcement learning [28 , 4 , 18 , 41 , 34 ] . In off-policy learning, however, longer backups also accumulate the effects of mismatch between the behavior and target policies. Many methods handle this mismatch through products of per-step trace coefficients. Unclipped importance-sampling ratios can amplify variance, while clipping controls the weights but can weaken long-horizon learning signals [31 , 35 ] . Methods such as Retrace and Peng’s Q ( λ ) Q(\lambda) Q ( λ ) make different compromises between signal retention, fixed-point bias, and convergence conditions [31 , 24 , 15 ] . These trade-offs make the maximum trace length, or cap length , a consequential choice and motivate adapting the backup horizon to the encountered policy mismatch. Prior on-policy analysis offers a complementary perspective by relating sufficient backup length to temporal dependence [10 ] . Estimating the relevant mixing properties can be difficult, and the corresponding guarantees do not directly extend to off-policy sampling. This motivates a sample-based stopping rule that adapts the backup horizon without estimating mixing time:
Can idempotent coupling indicators enable adaptive backups that better balance long-horizon learning signals and target variability?
We introduce MINT (Meeting-time INdicators for Truncation). In its theoretical construction, behavior and target state processes are coupled, and a binary mask retains the trace until their first agreement. The cumulative mask remains zero thereafter, turning the backup into a randomly truncated sum of TD errors. For model-free implementation, an action-level surrogate uses the policy ratio at a single sampled action to decide whether to continue. This surrogate defines its own stopping process and requires neither transition-kernel access nor a mixing-time estimate. Figure 1 illustrates the empirical motivation on MuJoCo Hopper. Across the tested caps, MINT exhibits a favorable empirical bias–variance balance and maintains high returns. This motivates examining how stochastic truncation changes both learning performance and the resulting target estimates.
Figure 1. Cap-length sensitivity on MuJoCo Hopper (SAC). (a) Bias and variance at 100K and 1M steps for caps { 3 , 5 , 10 , 20 } \{3, 5, 10, 20\} { 3 , 5 , 10 , 20 } (darker shades indicate shorter caps). MINT exhibits a favorable empirical bias–variance balance across the tested caps; several baseline configurations have higher variance at longer caps. (b) Learning curves for the same caps. MINT attains high returns across the tested values, whereas baseline performance varies more with the cap.
Contributions. We provide (i) a coupling-based formulation of stochastic truncation and a model-free action-level implementation (§ 3 ); (ii) a survival-based variance bound at Q π Q^{\pi} Q π for the state-coupled estimator and separate sufficient contraction conditions for the state-coupled and practical surrogate operators under fixed-policy evaluation (Lemma 3.3 and Theorems B.9 and 4.1 ); and (iii) comparative evaluations demonstrating gains over multi-step baselines on MuJoCo and DMC, using a common cap without per-task cap tuning in the main MuJoCo comparison, alongside task-dependent benefits on Atari 100k, RWRL, and OGBench (§ 5 ).
Preliminaries
We consider a Markov decision process (MDP) ( S , A , P , P 0 , r , γ ) (\mathcal{S}, \mathcal{A}, \mathcal{P}, \mathcal{P}_0, r, \gamma) ( S , A , P , P 0 , r , γ ) with finite state and action spaces, transition kernel P : S × A → Δ ( S ) \mathcal{P} : \mathcal{S} \times\mathcal{A} \to\Delta(\mathcal{S}) P : S × A → Δ ( S ) , initial distribution P 0 \mathcal{P}_0 P 0 , bounded reward r : S × A → [ − r max , r max ] r : \mathcal{S} \times\mathcal{A} \to[-r_{\max}, r_{\max}] r : S × A → [ − r m a x , r m a x ] , and discount γ ∈ [ 0 , 1 ) \gamma\in[0,1) γ ∈ [ 0 , 1 ) [40 ] . For a policy π \pi π , the Bellman operator is T π Q : = r + γ P π Q T^{\pi}Q := r + \gamma\mathcal{P}^{\pi}Q T π Q := r + γ P π Q , where ( P π Q ) ( s , a ) : = E s ′ ∼ P ( ⋅ ∣ s , a ) , a ′ ∼ π ( ⋅ ∣ s ′ ) [ Q ( s ′ , a ′ ) ] (\mathcal{P}^{\pi}Q)(s,a) := \mathbb{E}_{s' \sim\mathcal{P}(\cdot\mid s,a),\,a' \sim\pi(\cdot\mid s')}[Q(s',a')] ( P π Q ) ( s , a ) := E s ′ ∼ P ( ⋅ ∣ s , a ) , a ′ ∼ π ( ⋅ ∣ s ′ ) [ Q ( s ′ , a ′ )] . Trajectories ( s t , a t , r t ) t ≥ 0 (s_t,a_t,r_t)_{t \ge0} ( s t , a t , r t ) t ≥ 0 are generated by a behavior policy β \beta β , and our goal is to estimate the action-value function Q π Q^{\pi} Q π of a target policy π ≠ β \pi\ne\beta π = β from these off-policy samples [34 , 41 ] .
The Retrace Operator and the Variance Explosion Problem
A central challenge in multi-step off-policy evaluation is correcting the distributional mismatch between β \beta β and π \pi π without inflating the variance of the multi-step return. The general Retrace operator [31 ] formalizes this correction through a sequence of per-step trace coefficients c i : = c ( s i , a i ) c_i := c(s_i,a_i) c i := c ( s i , a i ) :
R Q = Q + E β [ ∑ t = 0 ∞ ( γ λ ) t ( ∏ i = 1 t c i ) δ t Q ] , δ t Q : = r t + γ E π [ Q ( s t + 1 , ⋅ ) ] − Q ( s t , a t ) , (1) \mathcal{R}Q = Q + \mathbb{E}_{\beta}\left[\sum_{t=0}^{\infty}(\gamma\lambda)^t\left(\prod_{i=1}^{t}c_i\right)\delta_t^Q\right], \qquad\delta_t^Q := r_t + \gamma\mathbb{E}_{\pi}[Q(s_{t+1},\cdot)] - Q(s_t,a_t), \tag*{(1)} R Q = Q + E β [ t = 0 ∑ ∞ ( γ λ ) t ( i = 1 ∏ t c i ) δ t Q ] , δ t Q := r t + γ E π [ Q ( s t + 1 , ⋅ )] − Q ( s t , a t ) , (1)
with the convention ∏ i = 1 0 c i = 1 \prod_{i=1}^{0} c_i = 1 ∏ i = 1 0 c i = 1 and λ ∈ [ 0 , 1 ] \lambda\in[0,1] λ ∈ [ 0 , 1 ] controlling the geometric decay of the trace [7 ] . Existing multi-step off-policy methods instantiate Equation (1) by a choice of c i c_i c i : Retrace [31 ] uses c i = min ( 1 , π ( a i ∣ s i ) / β ( a i ∣ s i ) ) c_i = \min(1,\pi(a_i \mid s_i)/\beta(a_i \mid s_i)) c i = min ( 1 , π ( a i ∣ s i ) / β ( a i ∣ s i )) ; Harutyunyan’s Q π ( λ ) Q^{\pi}(\lambda) Q π ( λ ) uses c i = 1 c_i = 1 c i = 1 [19 ] . A sufficient conservative constraint is 0 ≤ c i ≤ π ( a i ∣ s i ) / β ( a i ∣ s i ) 0 \le c_i \le\pi(a_i \mid s_i)/\beta(a_i \mid s_i) 0 ≤ c i ≤ π ( a i ∣ s i ) / β ( a i ∣ s i ) , which guarantees contraction of the expected evaluation operator toward Q π Q^{\pi} Q π for fixed policies [31 ] .
The variance explosion problem. For a finite cap L L L , write Z t : = ( γ λ ) t ( ∏ i = 1 t c i ) δ t Q Z_t := (\gamma\lambda)^t \left(\prod_{i=1}^{t} c_i\right)\delta_t^Q Z t := ( γ λ ) t ( ∏ i = 1 t c i ) δ t Q . The estimator’s second moment has the exact expansion
E [ ( ∑ t = 0 L − 1 Z t ) 2 ] = ∑ t = 0 L − 1 ( γ λ ) 2 t E [ ( ∏ i = 1 t c i 2 ) ( δ t Q ) 2 ] ⏟ diagonal terms E [ Z t 2 ] + 2 ∑ 0 ≤ u < t < L E [ Z u Z t ] . (2) \mathbb{E}\left[\left(\sum_{t=0}^{L-1} Z_t\right)^2\right] = \underbrace{\sum_{t=0}^{L-1}(\gamma\lambda)^{2t}\mathbb{E}\left[\left(\prod_{i=1}^{t}c_i^2\right)(\delta_t^Q)^2\right]}_{\text{diagonal terms }\mathbb{E}[Z_t^2]} + 2\sum_{0\le u<t<L}\mathbb{E}[Z_uZ_t]. \tag*{(2)} E ( t = 0 ∑ L − 1 Z t ) 2 = diagonal terms E [ Z t 2 ] t = 0 ∑ L − 1 ( γ λ ) 2 t E [ ( i = 1 ∏ t c i 2 ) ( δ t Q ) 2 ] + 2 0 ≤ u < t < L ∑ E [ Z u Z t ] . (2)
Trace weights and TD errors can be dependent, and cross terms need not vanish at arbitrary Q Q Q . Unclipped importance-sampling ratios can amplify second moments over long traces; clipped coefficients need not exhibit this growth. Table 1 summarizes representative operators using the classification of Kozuno et al. [24 ] . Conservative methods such as Retrace bound coefficient products but may weaken long-horizon signals. Non-conservative methods can retain more signal but may introduce off-policy bias or require additional stability conditions [24 , 35 ] . These trade-offs can make performance sensitive to cap length, motivating task-specific tuning [20 ] .
Table 1. Multi-step off-policy operators for behavior and target policies. W t W_t W t excludes ( γ λ ) t (\gamma\lambda)^t ( γ λ ) t ; E [ W t 2 ] \mathbb{E}[W_t^2] E [ W t 2 ] describes squared weights, not the full estimator variance.
Mixing Time and Effective Horizon
Mixing time describes how quickly a Markov chain forgets its initial state. For an ergodic behavior chain with transition kernel P β \mathcal{P}^{\beta} P β and stationary distribution μ β \mu_{\beta} μ β , the total-variation (TV) mixing time is
τ m i x β : = min { t ≥ 0 : sup s ∥ ( P β ) t ( s , ⋅ ) − μ β ∥ T V ≤ 1 4 } . \tau_{\mathrm{mix}}^{\beta} := \min\left\{t\ge0:\sup_s\left\lVert(\mathcal{P}^{\beta})^t(s,\cdot)-\mu_{\beta}\right\rVert_{\mathrm{TV}}\le\frac{1}{4}\right\}. τ mix β := min { t ≥ 0 : s sup ( P β ) t ( s , ⋅ ) − μ β TV ≤ 4 1 } .
It measures convergence to stationarity uniformly over initial states [26 ] , but can be difficult to estimate directly from trajectories [46 ] . For on-policy evaluation (β = π \beta=\pi β = π ) with bounded rewards, Duan and Wainwright [10, Section 3.3.2, Eq. (29b)] obtain finite-sample bounds for multi-step kernel least-squares TD under their assumptions and the look-ahead condition
K ≳ min { H , τ ⋆ β } , H : = ( 1 − γ ) − 1 . (3) K \gtrsim\min\{H,\tau_{\star}^{\beta}\}, \qquad H := (1-\gamma)^{-1}. \tag*{(3)} K ≳ min { H , τ ⋆ β } , H := ( 1 − γ ) − 1 . (3)
Here τ ⋆ β \tau_{\star}^{\beta} τ ⋆ β denotes the mixing parameter used in their analysis, whose definition differs from the TV mixing time τ m i x β \tau_{\mathrm{mix}}^{\beta} τ mix β above; see Appendix § 8.1 .§ 7 for its definition and the detailed assumptions. This condition links the backup horizon to chain-level temporal dependence, but applying it requires knowledge of mixing properties. Moreover, the guarantees of Duan and Wainwright [10 ] rely on on-policy sampling and do not directly extend to β ≠ π \beta\ne\pi β = π . These limitations motivate our sample-based stopping rule for off-policy multi-step evaluation (§ 3 ).
The MINT Operator
We introduce MINT, which replaces products of real-valued trace coefficients with a binary stopping rule. We first define the state-coupled operator, then establish a variance bound at Q π Q^\pi Q π , and finally describe its model-free action-level surrogate. Let S t β , S t π S_t^\beta,S_t^\pi S t β , S t π start from a shared state. At policy-generated steps, maximally couple their policy-induced next-state kernels. Write H t \mathcal{H}_t H t for the joint pre-action history and G t = H t ∨ σ ( a t ) \mathcal{G}_t=\mathcal{H}_t\vee\sigma(a_t) G t = H t ∨ σ ( a t ) . At such steps, the behavior action has conditional law β ( ⋅ ∣ S t β ) \beta(\cdot\mid S_t^\beta) β ( ⋅ ∣ S t β ) given H t \mathcal{H}_t H t . The conditional environment law of ( r t , S t + 1 β ) (r_t,S_{t+1}^\beta) ( r t , S t + 1 β ) given G t \mathcal{G}_t G t must equal its law given ( S t β , a t ) (S_t^\beta,a_t) ( S t β , a t ) . The filtration and initial-action conditioning are specified in Appendix § 8.2 .§ 1 .
Definition 3.1 (State-coupled stopping rule). For i ≥ 1 i\ge1 i ≥ 1 and t ≥ 0 t\ge0 t ≥ 0 , we define
A i : = 1 { S i β ≠ S i π } , T β , π : = inf { i ≥ 1 : A i = 0 } , W t : = ∏ i = 1 t A i = 1 { T β , π > t } , A_i:=\mathbf{1}\{S_i^\beta\ne S_i^\pi\},\qquad T_{\beta,\pi}:=\inf\{i\ge1:A_i=0\},\qquad W_t:=\prod_{i=1}^{t}A_i=\mathbf{1}\{T_{\beta,\pi}>t\}, A i := 1 { S i β = S i π } , T β , π := inf { i ≥ 1 : A i = 0 } , W t := i = 1 ∏ t A i = 1 { T β , π > t } , with W 0 = 1 W_0=1 W 0 = 1 and inf ∅ = ∞ \inf\varnothing=\infty inf ∅ = ∞ .
The cumulative mask remains zero after first agreement, even if the states later separate. It is H t \mathcal{H}_t H t -measurable, hence G t \mathcal{G}_t G t -measurable, and cannot depend on the next reward or transition. We write s t = S t β s_t=S_t^\beta s t = S t β for brevity, and let E x , P x \mathbb{E}_x,\mathbb{P}_x E x , P x refer to the joint trajectory law conditioned on the starting pair x = ( s 0 , a 0 ) x=(s_0,a_0) x = ( s 0 , a 0 ) . Replacing ∏ i = 1 t c i \prod_{i=1}^{t}c_i ∏ i = 1 t c i in Equation (1) by W t W_t W t yields
( R β , λ Q ) ( x ) = Q ( x ) + E x [ ∑ t ≥ 0 ( γ λ ) t W t δ t Q ] = Q ( x ) + E x [ ∑ t = 0 T β , π − 1 ( γ λ ) t δ t Q ] . (4) (\mathcal{R}_{\beta,\lambda}Q)(x)=Q(x)+\mathbb{E}_x\left[\sum_{t\ge0}(\gamma\lambda)^tW_t\delta_t^Q\right]=Q(x)+\mathbb{E}_x\left[\sum_{t=0}^{T_{\beta,\pi}-1}(\gamma\lambda)^t\delta_t^Q\right]. \tag*{(4)} ( R β , λ Q ) ( x ) = Q ( x ) + E x [ t ≥ 0 ∑ ( γ λ ) t W t δ t Q ] = Q ( x ) + E x t = 0 ∑ T β , π − 1 ( γ λ ) t δ t Q . (4)
† ^{\dagger} † Peng’s return mixture has this TD-error representation with target π λ : = λ β + ( 1 − λ ) π \pi_\lambda:=\lambda\beta+(1-\lambda)\pi π λ := λ β + ( 1 − λ ) π , rather than π \pi π . ‡ ^{\ddagger} ‡ N n : = ( T β ) n T π \mathcal{N}_n:=(\mathcal{T}^\beta)^n\mathcal{T}^\pi N n := ( T β ) n T π is the ordinary fixed-horizon return (λ = 1 \lambda=1 λ = 1 ); its fixed point is generally behavior-dependent for n > 1 n>1 n > 1 .
The expectation averages environment and coupling randomness. Subscript p p p identifies the disagreement process; indicators can be dependent. The operator fixes Q π Q^\pi Q π ; uniqueness and convergence require the separate contraction condition in § 4 .
Variance at the Target Value Function
Two properties enter the variance analysis: conditional mean-zero TD errors eliminate cross terms, while idempotence turns squared binary masks into survival indicators.
Definition 3.2 (Discounted survival and policy mismatch). For starting pair x = ( s 0 , a 0 ) x=(s_0,a_0) x = ( s 0 , a 0 ) , write T = T β , π T=T_{\beta,\pi} T = T β , π and D s : = TV ( β ( ⋅ ∣ s ) , π ( ⋅ ∣ s ) ) D_s:=\operatorname{TV}(\beta(\cdot\mid s),\pi(\cdot\mid s)) D s := TV ( β ( ⋅ ∣ s ) , π ( ⋅ ∣ s )) . For γ ∈ ( 0 , 1 ) \gamma\in(0,1) γ ∈ ( 0 , 1 ) and λ ∈ [ 0 , 1 ] \lambda\in[0,1] λ ∈ [ 0 , 1 ] , we define
M x ( γ , λ ) : = ∑ t ≥ 0 ( γ λ ) t Pr x ( T > t ) , N x ( γ , λ ) : = ∑ t ≥ 1 ( γ λ ) t E x [ 1 { T > t } D S t ] . (5) M_x(\gamma,\lambda):=\sum_{t\ge0}(\gamma\lambda)^t\Pr_x(T>t),\qquad N_x(\gamma,\lambda):=\sum_{t\ge1}(\gamma\lambda)^t\mathbb{E}_x[\mathbf{1}\{T>t\}D_{S_t}]. \tag*{(5)} M x ( γ , λ ) := t ≥ 0 ∑ ( γ λ ) t x Pr ( T > t ) , N x ( γ , λ ) := t ≥ 1 ∑ ( γ λ ) t E x [ 1 { T > t } D S t ] . (5) M x M_x M x measures discounted trace survival; N x N_x N x weights policy mismatch along the surviving trace.
Lemma 3.3 (Martingale property, fixed point, and variance). Under the joint sampling conditions above , let δ t Q π \delta_t^{Q^\pi} δ t Q π be the TD error of Equation (1) at Q π Q^\pi Q π . Let C : = ∥ r ∥ ∞ + ( 1 + γ ) ∥ Q π ∥ ∞ < ∞ C:=\lVert r\rVert_\infty+(1+\gamma)\lVert Q^\pi\rVert_\infty<\infty C := ∥ r ∥ ∞ + ( 1 + γ ) ∥ Q π ∥ ∞ < ∞ . Then, for every starting pair x x x :
(i) Martingale property. E x [ δ t Q π ∣ G t ] = 0 \mathbb{E}_x[\delta_t^{Q^\pi}\mid\mathcal{G}_t]=0 E x [ δ t Q π ∣ G t ] = 0 and W t W_t W t is G t \mathcal{G}_t G t -measurable.
(ii) Fixed point. R p , λ Q π = Q π R_{p,\lambda}Q^\pi=Q^\pi R p , λ Q π = Q π .
(iii) Variance. For Δ ^ : = ∑ t ≥ 0 ( γ λ ) t W t δ t Q π \hat{\Delta}:=\sum_{t\ge0}(\gamma\lambda)^tW_t\delta_t^{Q^\pi} Δ ^ := ∑ t ≥ 0 ( γ λ ) t W t δ t Q π ,
Var x ( Δ ^ ) = E x [ Δ ^ 2 ] = ∑ t ≥ 0 ( γ λ ) 2 t E x [ W t ( δ t Q π ) 2 ] ≤ C 2 M x ( γ 2 , λ 2 ) . (6) \operatorname{Var}_x(\hat{\Delta})=\mathbb{E}_x[\hat{\Delta}^2]=\sum_{t\ge0}(\gamma\lambda)^{2t}\mathbb{E}_x[W_t(\delta_t^{Q^\pi})^2]\le C^2M_x(\gamma^2,\lambda^2). \tag*{(6)} Var x ( Δ ^ ) = E x [ Δ ^ 2 ] = t ≥ 0 ∑ ( γ λ ) 2 t E x [ W t ( δ t Q π ) 2 ] ≤ C 2 M x ( γ 2 , λ 2 ) . (6)
For u < t u<t u < t , the earlier TD error and both masks are measurable at G t \mathcal{G}_t G t , so the tower property removes the cross term. Idempotence then gives W t 2 = W t W_t^2=W_t W t 2 = W t , yielding the diagonal expression in Equation (6) . The full proof is in Appendix § 8.2 .§ 1 . The cross-term argument also holds for other suitably adapted weights; binary masks additionally express diagonal weights through survival probabilities, giving
Var x ( Δ ^ ) ≤ C 2 min { E x [ T β , π ] , 1 1 − γ 2 λ 2 } . \operatorname{Var}_x(\hat{\Delta})\le C^2\min\left\{\mathbb{E}_x[T_{\beta,\pi}],\frac{1}{1-\gamma^2\lambda^2}\right\}. Var x ( Δ ^ ) ≤ C 2 min { E x [ T β , π ] , 1 − γ 2 λ 2 1 } .
This bound concerns evaluation at Q π Q^\pi Q π ; it does not establish variance dominance over Retrace or describe the full error of a learned critic. It offers a survival-based perspective alongside the mixing-based analysis in § 2.2 . Contraction requires the additional bounds developed in § 4 .
Practical Implementation
Constructing the state coupling requires transition-kernel information. The implementation instead uses the action-level keep probability
p ^ i : = 1 − min ( 1 , π ( a i ∣ s i ) β ( a i ∣ s i ) ) . (7) \hat{p}_i:=1-\min\left(1,\frac{\pi(a_i\mid s_i)}{\beta(a_i\mid s_i)}\right). \tag*{(7)} p ^ i := 1 − min ( 1 , β ( a i ∣ s i ) π ( a i ∣ s i ) ) . (7)
Here β \beta β is the policy that generated the stored transition: its log-probability is recorded at collection time and compared with the current π \pi π at replay. Using randomness independent of subsequent rewards and transitions, draw A ^ i ∼ Bernoulli ( p ^ i ) \hat{A}_i \sim\operatorname{Bernoulli}(\hat{p}_i) A ^ i ∼ Bernoulli ( p ^ i ) and define
T ^ β , π : = inf { i ≥ 1 : A ^ i = 0 } , K ^ : = min ( K max , T ^ β , π ) , (8) \hat{T}_{\beta,\pi} := \inf\{i \ge1 : \hat{A}_i = 0\}, \qquad\hat{K} := \min(K_{\max}, \hat{T}_{\beta,\pi}), \tag*{(8)} T ^ β , π := inf { i ≥ 1 : A ^ i = 0 } , K ^ := min ( K m a x , T ^ β , π ) , (8)
with inf ∅ = ∞ \inf\varnothing= \infty inf ∅ = ∞ . The implementation fixes a hard cap K max K_{\max} K m a x ; the sampled stopping time selects the realized prefix within it. The state-coupled operator above is uncapped, whereas the surrogate evaluation theorem uses an explicit finite cap. The rule retains the trace on disagreement: when π = β \pi= \beta π = β , p ^ i = 0 \hat{p}_i = 0 p ^ i = 0 and the backup is one-step. A larger p ^ i \hat{p}_i p ^ i makes continuation more likely at that step. Unlike Retrace, the keep rule uses the complement of the clipped policy ratio. The expected effective horizon K e f f : = E [ K ^ ] K_{\mathrm{eff}} := \mathbb{E}[\hat{K}] K eff := E [ K ^ ] depends on the full sequence of keep probabilities.
Relation to the state coupling. The surrogate’s conditional mean is the action-level policy TV,
E a ∼ β ( ⋅ ∣ s ) [ p ^ ( s , a ) ] = D s ≥ d ( s ) : = TV ( P β ( s , ⋅ ) , P π ( s , ⋅ ) ) , \mathbb{E}_{a\sim\beta(\cdot\mid s)}[\hat{p}(s,a)] = D_s \ge d(s) := \operatorname{TV}(\mathcal{P}^{\beta}(s,\cdot),\mathcal{P}^{\pi}(s,\cdot)), E a ∼ β ( ⋅ ∣ s ) [ p ^ ( s , a )] = D s ≥ d ( s ) := TV ( P β ( s , ⋅ ) , P π ( s , ⋅ )) ,
by the data-processing inequality (Proposition A.4 ). This compares one-step kernels from a shared state; it does not identify its multi-step survival law with that of the state coupling. The expected surrogate evaluation has its own finite-cap contraction condition (Theorem B.9 and Appendix § 8.2 .8).
Theoretical Analysis
The variance lemma (Lemma 3.3 ) controls the estimator at Q π Q^\pi Q π . We next give a sufficient contraction condition for the state-coupled operator under an arbitrary behavior policy. Use M x = M x ( γ , λ ) M_x = M_x(\gamma,\lambda) M x = M x ( γ , λ ) and N x = N x ( γ , λ ) N_x = N_x(\gamma,\lambda) N x = N x ( γ , λ ) from Definition 3.2, retaining their starting-pair dependence.
Theorem 4.1 (Contraction of R β , λ \mathcal{R}_{\beta,\lambda} R β , λ ). Under the joint sampling conditions of § 3 , fix policies β \beta β , π \pi π , γ ∈ ( 0 , 1 ) \gamma\in(0,1) γ ∈ ( 0 , 1 ) , and λ ∈ [ 0 , 1 ] \lambda\in[0,1] λ ∈ [ 0 , 1 ] . We define
κ : = sup x { 1 − ( 1 − γ ) M x + 2 N x } . (9) \kappa:= \sup_x \left\{1 - (1-\gamma)M_x + 2N_x\right\}. \tag*{(9)} κ := x sup { 1 − ( 1 − γ ) M x + 2 N x } . (9) Then, for all bounded Q Q Q , Q ′ Q' Q ′ ,
∥ R β , λ Q − R β , λ Q ′ ∥ ∞ ≤ κ ∥ Q − Q ′ ∥ ∞ . (10) \lVert\mathcal{R}_{\beta,\lambda}Q - \mathcal{R}_{\beta,\lambda}Q' \rVert_\infty\le\kappa\lVert Q-Q' \rVert_\infty. \tag*{(10)} ∥ R β , λ Q − R β , λ Q ′ ∥ ∞ ≤ κ ∥ Q − Q ′ ∥ ∞ . (10) If κ < 1 \kappa< 1 κ < 1 , R β , λ \mathcal{R}_{\beta,\lambda} R β , λ is a κ \kappa κ -contraction with unique fixed point Q π Q^\pi Q π .
For each starting pair, 1 − ( 1 − γ ) M x 1-(1-\gamma)M_x 1 − ( 1 − γ ) M x is the total update-weight contribution and 2 N x 2N_x 2 N x bounds the negative-weight correction. In the on-policy case M x = 1 M_x = 1 M x = 1 and N x = 0 N_x = 0 N x = 0 for all x x x , recovering κ = γ \kappa= \gamma κ = γ ; the full proof is in Appendix § 8.2 .§ 2 . The first agreement time here differs from the behavior chain’s mixing time (Appendix § 8.2 .§ 6 ). The action-dependent surrogate has its own finite-cap contraction Algorithm 1 MINT target (summary). Full procedure: Algorithm 2.
Require: Trajectory with stored behavior log-probabilities; π \pi π , Q ‾ θ ˉ \overline{Q}_{\bar{\theta}} Q θ ˉ , γ \gamma γ , λ \lambda λ ; K max ≥ 1 K_{\max} \ge1 K m a x ≥ 1
For 1 ≤ i < K max 1 \le i < K_{\max} 1 ≤ i < K m a x , compute p ^ i \hat{p}_i p ^ i by Equation (7) and independently draw A ^ i ∼ Bernoulli ( p ^ i ) \hat{A}_i \sim\operatorname{Bernoulli}(\hat{p}_i) A ^ i ∼ Bernoulli ( p ^ i )
K ^ ← min ( K max , T ^ β , π ) \hat{K} \leftarrow\min(K_{\max}, \hat{T}_{\beta,\pi}) K ^ ← min ( K m a x , T ^ β , π )
return Q ‾ θ ˉ ( s 0 , a 0 ) + ∑ t = 0 K ^ − 1 ( γ λ ) t δ t Q ‾ θ ˉ \overline{Q}_{\bar{\theta}}(s_0, a_0) + \sum_{t=0}^{\hat{K}-1}(\gamma\lambda)^t\delta_t^{\overline{Q}_{\bar{\theta}}} Q θ ˉ ( s 0 , a 0 ) + ∑ t = 0 K ^ − 1 ( γ λ ) t δ t Q θ ˉ
theorem (Theorem B.9 and Table 4 ). A separate exact-control result for Markov trace operators, under additional overlap and policy-improvement hypotheses, is given in Theorem B.8 . We can also normalize the surviving mismatch by setting D w g t x : = N x / ( M x − 1 ) D_{\mathrm{wgt}}^x := N_x/(M_x - 1) D wgt x := N x / ( M x − 1 ) for M x > 1 M_x > 1 M x > 1 and D w g t x : = 0 D_{\mathrm{wgt}}^x := 0 D wgt x := 0 otherwise; let D ‾ w g t : = sup x D w g t x \overline{D}_{\mathrm{wgt}} := \sup_x D_{\mathrm{wgt}}^x D wgt := sup x D wgt x .
Table 4. Assumption usage by result. No listed result requires deterministic dynamics. State-coupled and action-level surrogate results have separate sufficient conditions.
Theorem 4.1 relates contraction to policy mismatch along surviving traces. To express this relationship as a horizon budget, let K λ : = sup x min { E x [ T ] , ( 1 − γ λ ) − 1 } K_\lambda:= \sup_x \min\{\mathbb{E}_x[T], (1-\gamma\lambda)^{-1}\} K λ := sup x min { E x [ T ] , ( 1 − γ λ ) − 1 } , so M x ≤ K λ M_x \le K_\lambda M x ≤ K λ . This survival bound is distinct from the hard cap K max K_{\max} K m a x ; Corollary C.1 gives two sufficient regimes. When D ‾ w g t ≤ ( 1 − γ ) / 2 \overline{D}_{\mathrm{wgt}} \le(1-\gamma)/2 D wgt ≤ ( 1 − γ ) /2 , no additional meeting-time restriction is needed. Otherwise, contraction is ensured by K λ < 2 D ‾ w g t / [ 2 D ‾ w g t − ( 1 − γ ) ] K_\lambda< 2\overline{D}_{\mathrm{wgt}}/[2\overline{D}_{\mathrm{wgt}}-(1-\gamma)] K λ < 2 D wgt / [ 2 D wgt − ( 1 − γ )] . Thus, the admissible horizon depends jointly on survival and the policy mismatch encountered before stopping. At γ = 0.9 \gamma= 0.9 γ = 0.9 , mismatch up to 0.05 0.05 0.05 requires no additional horizon restriction. At mismatch 0.1 0.1 0.1 , the requirement K λ < 2 K_\lambda< 2 K λ < 2 is met by λ = 0.5 \lambda= 0.5 λ = 0.5 , since K λ ≤ 1.818 K_\lambda\le1.818 K λ ≤ 1.818 . Figure 3 and Table 5 illustrate these budgets. Under additional overlap assumptions, a geometric specialization is given in Corollary B.1 and Figure 9 . The action-level surrogate instead has its finite-cap contraction conditions (Theorem B.9 and Corollary B.10).
Figure 3. State-coupled horizon budget. At γ = 0.9 \gamma= 0.9 γ = 0.9 , green satisfies the sufficient condition in Corollary C.1; pink is inconclusive. Guides show discount-only bounds for λ ∈ { 0.7 , 0.5 } \lambda\in\{0.7, 0.5\} λ ∈ { 0.7 , 0.5 } .
Figure 9. Geometric-overlap specialization at γ = 0.99 \gamma=0.99 γ = 0.99 . Under the cross-Doeblin and uniform initial-step overlap hypotheses of Corollary B.1, γ [ 1 + 2 D ˉ w g t λ ( 1 − ρ ) ] < 1 \gamma[1+2\bar{D}_{\mathrm{wgt}}\lambda(1-\rho)]<1 γ [ 1 + 2 D ˉ wgt λ ( 1 − ρ )] < 1 ensures contraction of the fixed-policy state-coupled operator. Dark-to-light green bands identify the satisfying subsets of λ ∈ { 1 , 0.7 , 0.5 , 0.3 } \lambda\in\{1,0.7,0.5,0.3\} λ ∈ { 1 , 0.7 , 0.5 , 0.3 } specified in the legend. Pink is inconclusive for all four displayed values; it does not indicate divergence. For each λ \lambda λ , its curve marks equality and the condition holds strictly above it. The inset enlarges the high-overlap region; open markers are hypothetical boundary points at D ˉ w g t = 0.2 \bar{D}_{\mathrm{wgt}}=0.2 D ˉ wgt = 0.2 . The comparison holds the summary coordinate D ˉ w g t \bar{D}_{\mathrm{wgt}} D ˉ wgt fixed. These state-coupled conditions do not certify action-level surrogate contraction or deep-RL convergence.
Table 5. State-coupled contraction budget at γ = 0.9 \gamma= 0.9 γ = 0.9 (Corollary C.1 ). λ ∗ = ( 1 − γ ) / ( 2 γ D ˉ w g t ) \lambda^{*} = (1-\gamma)/(2\gamma\bar{D}_{\mathrm{wgt}}) λ ∗ = ( 1 − γ ) / ( 2 γ D ˉ wgt ) is the critical λ \lambda λ below which the sufficient condition holds for any meeting-time distribution. Finite entries require K λ < K λ max K_{\lambda} < K_{\lambda}^{\max} K λ < K λ m a x ; ∞ \infty ∞ denotes no additional horizon restriction beyond the discount-only bound, not an infinite realized horizon.
Experiments
We evaluate data-efficient learning, adaptive trace lengths, task-dependent backup trade-offs, and behavior under stochastic dynamics:
RQ1 How does MINT compare with multi-step baselines?
RQ2 Does stochastic truncation produce a per-task adaptive trace length without estimating τ m i x \tau_{\mathrm{mix}} τ mix ?
RQ3 Under which task conditions does MINT improve the backup trade-off, and how do policy mismatch and target spread characterize these gains?
RQ4 How do target estimation and learned-critic behavior change under stochastic dynamics?
Experimental Setup
We evaluate MuJoCo [43 ] with SAC [16 ] and TD3 [14 ] , eleven tasks from the DeepMind Control Suite (DMC) [42 ] with SAC (task selection: Section F.2.1), and Atari 100k [23 ] with a DrQ(ε \varepsilon ε )-based variant [47 ] . Additional RWRL [11 ] and OGBench [33 ] experiments diagnose cap sensitivity and fixed-policy estimation. The five-task MuJoCo comparison uses a common cap of 5, λ = 0.7 \lambda= 0.7 λ = 0.7 for MINT, Peng’s Q ( λ ) Q(\lambda) Q ( λ ) , and Retrace, fixed SAC α = 0.2 \alpha= 0.2 α = 0.2 , and five seeds per method for 1M steps. The 11-task DMC comparison uses five seeds per method/task, 500K steps, γ = 0.99 \gamma= 0.99 γ = 0.99 , automatic entropy tuning, and cap length 10; MINT uses λ = 0.9 \lambda= 0.9 λ = 0.9 , while its n n n -step SAC baseline uses a pure uncorrected return with n = 10 n = 10 n = 10 . IQM uses 95% stratified-bootstrap intervals [3 ] . Settings are matched within each comparison; protocols and normalization are not pooled across suites. Atari settings are in Appendix F.3. The supplementary five-task DMC evaluation uses a separate 1M-step protocol; Table 13 distinguishes its seeds, targets, and score windows from the primary comparison.
Table 13. DMC evaluation protocols. Supplementary evaluation scores are not pooled into the primary eleven-task aggregates. The supplementary evaluation’s five-seed curves and CartPole re-run retain their separately reported configurations.
Benchmark Performance (RQ1)
MuJoCo. Figure 5 (a–Figure 5 b) compares aggregate IQM over five tasks for SAC (fixed-α \alpha α ) and TD3 based methods. SAC-MINT has the best aggregate IQM among all variants. SAC-Retrace matches MINT on humanoid-v2 and achieves lower returns on the remaining four tasks; Peng’s Q ( λ ) Q(\lambda) Q ( λ ) and uncorrected n n n -step often underperform one-step SAC. Per-task SAC and TD3 curves are in Figure 15 and Figure 16 .
Figure 5. Performance and adaptive truncation behavior on MuJoCo. (a–b) Aggregate normalized scores (IQM) of SAC (fixed-α \alpha α ) and TD3 variants over training steps; shaded regions show 95% stratified bootstrap confidence intervals across 5 seeds per task. (c–d) Mean truncation lengths by MINT across five MuJoCo environments, with ± 1 \pm1 ± 1 standard deviation across seeds. SAC horizons shorten over training, while TD3 horizons remain closer to the cap.
Figure 15. Performance with stochastic truncation. Across five MuJoCo benchmarks, MINT attains higher returns than the multi-step baselines on most tasks and at earlier training checkpoints. For baseline comparisons, we follow the convention of [24 ] and fix the cap length to n = 5 n=5 n = 5 for all multi-step methods.
Figure 16. Evaluation of TD3 variants over OpenAI MuJoCo environments.
11-task DMC Motivated by MINT’s fixed-policy contraction conditions and survival-based variance analysis, we examine whether adaptive truncation provides a useful balance between value-error propagation and target-sampling variability in actor–critic learning. We evaluate sample efficiency on eleven tasks selected from the DMC Easy & Medium set used by SimBa [25 ] (Section F.2.1), using a finite budget of 500K steps. MINT attains IQM 0.457 versus 0.366 for one-step SAC and 0.284 for n n n -step SAC (Figure 6 ): relative point-estimate gains of 24.7% and 60.8%. Mean returns improve on 10/11 and 9/11 tasks, respectively (Table 14 and Figure 20 ). Learning curves and profiles support the aggregate comparison (Figure 18 and Figure 19 ); task-group differences are examined in § 5.4 .
Figure 6. 11-task DMC aggregate performance at 500K steps (5 seeds). Scores are normalized by 1000. Rectangles show 95% bootstrap intervals; black ticks mark estimates.
Table 14. 11-task DMC final returns by reward accessibility. Last-five-evaluation average at 500K steps; mean ± standard deviation across five seeds. Groups use the fraction of uniform-random episodes containing positive reward. Gray literature stars in the task-curve figure use a different protocol and are not matched baselines.
Figure 18. 11-task DMC evaluation-time IQM. All eleven tasks and the original six-/five-task A/B split. Bands are pointwise 95% stratified-bootstrap intervals; final-window aggregate scores average five evaluations per run.
Figure 19. 11-task DMC final-score profiles. Fraction of pooled task/seed runs above return/1000 threshold τ \tau τ , with pointwise 95% bootstrap bands. Each task contributes five seeds.
Figure 20. 11-task DMC task learning curves. Mean over five seeds; bands are one standard error, not 95% intervals. Each evaluation averages ten episodes. Gray stars are published SAC reference values with a different action-repeat protocol.
Atari 100k. The DrQ-based MINT variant achieves higher mean HNS (0.512 vs. 0.465) and a slightly lower optimality gap (0.627 vs. 0.631) than published DrQ(ϵ \epsilon ϵ ) (Figure 8 and Table 25 ). The clearest task-group gains occur in the easy/human-optimal group (§ 5.4 ), while full-suite median (0.152/0.313) and IQM (0.251/0.280) remain lower. A direct extension of DrQ with the TD-error target learned poorly; we therefore modify the implementation as detailed in Appendix § 8.6 .§ 3 .
Figure 8. Atari 100k aggregate performance. Rectangles: 95% bootstrap intervals; black ticks: estimates. Diamonds: DrQ(ϵ \epsilon ϵ ) 100-run estimates [37 Table A.1] , without baseline intervals.
Table 25: Atari 26-game aggregate point estimates. MINT: primary, five seeds/game. Literature rows use published point estimates [3 , 37 ] ; the BBF row aggregates 50 runs/game from the authors’ released scores. Bottom rows are approximate estimates recovered from the rliable paper’s 10-run figure. The 10-run and 100-run DrQ references are reported separately. Higher is better except optimality gap.
Table 25.
Adaptive Horizons and Cap Sensitivity (RQ2)
MINT selects K ^ = min ( K max , T ^ β , π ) \hat{K} = \min(K_{\max}, \hat{T}_{\beta,\pi}) K ^ = min ( K m a x , T ^ β , π ) without estimating τ min \tau_{\min} τ m i n . MuJoCo lengths vary across tasks and training, with SAC generally shorter than TD3 (Figure 5 (c–Figure 5 d)). On Hopper, returns vary little across caps {3, 5, 10, 20}, unlike Peng’s Q ( λ ) Q(\lambda) Q ( λ ) and uncorrected n n n -step (Figure 1 b and Appendix § 8.12 .§ 1 ). Atari mean K K K falls from 6.20 to 5.05 as the fraction of matching behavior and current greedy action rises from 14.1% to 28.1% (Figure 37 ). DMC temperature effects are task-dependent (Figure 25 and Figure 32 ).
Figure 25. DMC temperature associations. Each of 55 runs uses 50 training records. Gray points are seeds; pink diamonds are task medians and segments are seed min–max ranges, not confidence intervals. Weak overall median correlations do not imply independence.
Figure 32. Fixed/auto-temperature comparison on CartPole-Swingup-Sparse. Cap L = 10 L = 10 L = 10 ; entropy coefficient (left) and realized length (right). The original plotted ranges are preserved. This protocol comparison is distinct from the within-run correlations in Section F.2.6.
Figure 37. Primary Atari agreement and horizon diagnostics. Group curves are equal-game averages of five-seed iteration means. Final scatter points are five-seed game means; correlations are descriptive across 26 games. Steps are nominal iteration counts times 10K; no smoothing is applied. Published DrQ game means are point references.
RWRL illustrates the value of leaving room for stopping. At fixed α = 0.05 \alpha= 0.05 α = 0.05 and 200K steps, increasing the cap tenfold changes mean length only from 3.9 to 4.8, with 99.5% stopping before cap 50 (Table 2 ). A separate auto-α \alpha α diagnostic fixes π \pi π at 1M and varies β \beta β over 900K, 500K, and 100K.
Table 2. RWRL cap ablation. Mean ± \pm ± std. across three seeds; actuator-noise levels pooled.
At cap 50, mean lengths are 7.06 → \to → 14.21 → \to → 45.96 for σ = 0.1 \sigma= 0.1 σ = 0.1 and 10.52 → \to → 18.10 → \to → 44.61 for σ = 0.3 \sigma= 0.3 σ = 0.3 . For recent β \beta β , cap 50 cuts only 0.007%/0.583% of traces; for the oldest β \beta β , these fractions rise to 79.73%/75.70% (Figure 40 ). Thus the cap margin needed for stopping to determine most lengths depends on the behavior distribution (Section G.1.1).
Figure 40. Surrogate stopping survival across behavior ages. Fixed π \pi π at 1M; solid/dashed/dotted curves use β \beta β at 900K/500K/100K (ages 100K/500K/900K). Each noise–age condition pools three seeds and 30,720 indicator draws. Dashed vertical lines mark caps 5/10/50; the horizontal axis is logarithmic. Recent and intermediate conditions have no censoring. At age 900K, 33.5%/3.0% of draws remain censored at the observation horizon 1000 for σ = 0.1 / 0.3 \sigma= 0.1/0.3 σ = 0.1/0.3 ; their unobserved tails are not extrapolated.
Task-Dependent Benefits (RQ3)
We group DMC tasks by the fraction of uniform-random episodes containing positive reward, as measured in Section F.2.2: rare (< 5%), occasional (10–50%), and frequent (100%). MINT improves mean returns over one-step SAC on all six DMC tasks with rare or occasional positive rewards, and over ten-step SAC on five of the six (Table 14 ). The largest aggregate gains occur on the occasional-reward Finger/Hopper tasks: IQM 0.453 versus 0.322/0.157 for one-step/ten-step SAC (Figure 21 ). Both strictly rare tasks improve over one-step, while Pendulum favors ten-step. Frequent-reward tasks retain near-one-step IQM while improving over ten-step. Task-level exceptions and supplementary task and λ \lambda λ sweeps are detailed in Appendix § 8.6 . Under Bellemare’s Atari exploration taxonomy [6 ] , MINT’s strongest task-group gains occur in the easy/human-optimal group: mean HNS is 0.689 versus 0.463 for published DrQ, the median of game means is 0.669 versus 0.429, and game means improve on 7/11 games (Figure 35 and Table 28 ). The largest gains occur on DemonAttack, Boxing, Assault, and CrazyClimber; their positive contributions to the suite mean are partly offset by lower hard/dense scores and mixed score-exploit results. More details are reported in Appendix § 8.6 .§ 3 .
Figure 21. DMC reward-accessibility groups. Endpoint IQM with 95% confidence-interval rectangles and black estimate ticks. Group sizes are 2/4/5; grouping is descriptive rather than a universal exploration taxonomy.
Table 28. Atari group performance. Mean HNS and median across game means for both methods; IQM pools MINT’s game/seed scores only. DrQ group IQM is unavailable. Wins count MINT game means above the published DrQ reference, not a probability of improvement. DrQ values use rounded published returns.
Figure 35. Group mean, median, and IQM with seed uncertainty. E/H: easy/human-optimal; E/X: easy/score-exploit; H/D: hard/dense; H/S: hard/sparse. Pink rectangles are MINT’s 95% percentile seed-bootstrap intervals (50,000 resamples, seeds independently resampled within each fixed game; no task resampling). Black vertical ticks are estimates; open diamonds are DrQ point references from published game means. Median is over game means, while IQM pools game/run scores. DrQ group IQM, baseline intervals, and difference intervals are unavailable. The sparse group’s tight seed interval does not establish robustness over task selection.
DMC diagnostics fix the policy and target critic and compare operators on the same behavior-trajectory segments (Section F.2.4). Pooling current and older behavior checkpoints within each stage and applying stage-specific seed exclusions, MINT has lower target spread than cap-ten λ \lambda λ -TD in 26/33 task–checkpoint comparisons across eleven tasks at 200K, 300K, and 500K, and pure ten-step in 27/33, with median ratios 0.916 and 0.759 (Figure 24 ). Figure 7 illustrates the 200K diagnostics: spread is lower in all four panels, while absolute aggregated bias is higher in three. A separate five-seed diagnostic reports lower target-variance point estimates than a shuffled-length control in 11/12 off-policy conditions, preserving the full stopping-length distribution (Section F.2.5). This supports a descriptive benefit from trajectory-dependent stopping, without establishing uniform variance dominance over fixed backups or a causal explanation for control gains. Operator details and counterexamples are provided in Sections F.2.4 and F.2.5.
Figure 7. DMC target diagnostics at 200K (RQ3). Green: MINT; red: fixed λ \lambda λ -TD traces (square: K = 10 K = 10 K = 10 ); blue: n n n -step. Dark-to-light: length 1–10. Five training seeds per task. Axes: absolute aggregated signed bias and target std., relative to 100; scales differ. Full results: Section F.2.4.
Figure 24. DMC target-spread diagnostics with the weighted TD-error reference. 33 task–stage cases at 200K/300K/500K, conditional on the seed filter in Table 15 . (a) Fixed-K K K λ \lambda λ -weighted TD trace standard deviation divided by its length-one value; gray lines are cases and the violet line is their pointwise median. (b) MINT/reference standard-deviation ratios; black diamonds are medians. The cap-ten reference is the fixed TD trace, not the Peng mixture. The mean-K K K comparison interpolates the fixed-length standard-deviation curve; it is not a randomized matched-length estimator.
Interpreting adaptive horizons. MINT adjusts reward propagation against exposure to off-policy segments and bootstrap estimates. Figure 4 shows task-dependent lengths across measurements up to and after 140K steps; its conditional references describe this trade-off without certifying an optimal horizon. The shorter early-stage traces in Figure 4 are consistent with greater overlap between the current and collection-time behavior policies, leading to more frequent stopping. As training progresses, policy evolution and replay of transitions collected under earlier policies can increase target–behavior mismatch and produce longer traces. Within the finite 500K-step budget, this evolving policy mixture may help explain the observed temporal differences in trace length.
Figure 4. Adaptive trace lengths and mismatch visualization on 11-task DMC. Points show mean trace lengths vs the action-based mismatch proxy χ λ p r o x y \chi_{\lambda}^{\mathrm{proxy}} χ λ proxy . Hollow/filled markers denote measurements at ≤ 140 \le140 ≤ 140 K/> 140 >140 > 140 K steps. Green contours show the fraction of discount-grid values satisfying an approximate contraction inequality; Blue lines mark calibrated transition-based horizon references (group-mean K r e f K_{\mathrm{ref}} K ref ); Gray curves show empirical 90th-quantile references. Trace lengths vary across tasks and steps. More details are in Section F.2.3.
Target and Learned-Critic Evaluation under Stochastic Dynamics (RQ4)
OGBench AntMaze crosses deterministic/stochastic dynamics with navigate/stitch datasets. A fitted Gaussian behavior model defines policy displacements h h h in standard-deviation units. All operators share a frozen approximate Monte-Carlo critic and finite-rollout reference. At h = 0.5 h = 0.5 h = 0.5 , MINT has lower RMSE than Retrace on all four datasets (Table 3 ). Its intrinsic variance is lower than a shuffled-K ^ \hat{K} K ^ control in seven of eight reported h ∈ { 0.5 , 1 } h \in\{0.5, 1\} h ∈ { 0.5 , 1 } cells (Table 3 and Table 39 ). This control preserves the length distribution but removes trajectory dependence. One-step has the lowest RMSE in most cells; at h = 2 h = 2 h = 2 , Retrace also overtakes MINT in the frozen-critic comparison (Appendix § 8.13 .§ 1 ).
Table 3. OGBench frozen-critic evaluation (h = 0.5 h = 0.5 h = 0.5 ). Approximate Monte-Carlo critic; 1 paired run per cell over 100 stratified evaluation states. Bold marks the lower value within each metric pair.
Table 39: OGBench frozen-critic evaluation at h = 1 h = 1 h = 1 (one paired run per cell, 100 stratified evaluation states). h h h : target-policy displacement in behavior standard deviations; t t t : paired t t t -statistic of the per-state absolute-error difference, MINT vs. Retrace. M = MINT, R = Retrace, shuf = shuffled-K ^ \hat{K} K ^ control. Results at h = 0.5 h = 0.5 h = 0.5 are split between Table 3 and Table 38 .
Table 39.
Learned-critic results reveal a propagation trade-off. Retrace has lower RMSE on all evaluated datasets at 100K updates. At 300K, MINT improves on Retrace on the deterministic maze at h = 1 h = 1 h = 1 , after slower propagation with K ^ ≈ 1.6 \hat{K} \approx1.6 K ^ ≈ 1.6 ; Retrace retains its advantage on the stochastic maze at h ∈ { 1 , 2 } h \in\{1,2\} h ∈ { 1 , 2 } . The shuffled-K ^ \hat{K} K ^ control is at least as accurate and remains stable at h = 2 h = 2 h = 2 on the stochastic maze (Appendix § 8.13 .§ 2 and Table 40 ).
Table 40. Learned-critic (fitted-Q evaluation) RMSE against the Monte-Carlo Q π Q^\pi Q π at 300k gradient steps, 5 seeds (one Retrace cell has 4). Bold marks the best median in each cell.
Discussion. The experiments support adaptive backup length as a practical estimator-design choice, with control-performance gains and lower target spread relative to long fixed backups. In RWRL, increasing the cap from 5 to 50 under fixed α \alpha α changes mean trace length only from 3.9 to 4.8, with 99.5% of traces stopping before cap 50 (Table 2 ). In OGBench frozen-critic evaluation, MINT achieves lower RMSE than Retrace on all four datasets at h = 0.5 h = 0.5 h = 0.5 and lower intrinsic variance than a shuffled-length control in seven of eight reported h ∈ { 0.5 , 1 } h \in\{0.5, 1\} h ∈ { 0.5 , 1 } conditions (Table 3 and Table 39 ). These findings highlight the practical value of adaptive stopping for horizon selection and target estimation.
Related Work
Multi-step off-policy methods differ in how they handle policy mismatch and preserve the target-policy value function. Retrace constrains trace coefficients to guarantee fixed-policy evaluation under arbitrary behavior policies [31 ] . Harutyunyan et al. [19 ] accumulates geometrically weighted TD errors without per-decision corrections, but requires additional policy-mismatch conditions for contraction; our expected operator recovers it when all traces are retained. Peng’s method generally converges to a behavior-dependent value function, although control convergence is possible when behavior suitably tracks a greedy policy [24 ] . MINT preserves the target-policy value function, with separate sufficient contraction conditions for its state-coupled and surrogate operators (Theorems B.9 and 4.1 ). Rowland et al. [35 ] analyze trade-offs among update variance, fixed-point bias, and contraction rate; Daley et al. [9 ] unify per-decision and trajectory-aware traces under a multistep operator. MINT continues a trace with the complement of the clipped policy ratio used by Retrace, making continuation depend on disagreement and reducing to one-step backups. Its binary masks express the state-coupled estimator’s variance bound at the target-policy value function through trace-survival probabilities (Lemma 3.3 ), without establishing variance dominance over Retrace. Importance Resampling resamples replay transitions [36 ] , whereas MINT truncates the given trajectory.
Mixing-time analyses offer a complementary perspective on temporal dependence in policy learning [17 , 1 , 32 , 45 ] , including average-reward methods without oracle mixing-time knowledge [39 ] . For on-policy multi-step evaluation, Duan and Wainwright [10 ] relate sufficient look-ahead to temporal dependence and model misspecification (§ 2.2 ); this does not directly give off-policy guarantees. MINT instead uses policy-mismatch-dependent stopping for discounted TD evaluation, motivated by coupling [29 , 27 , 21 ] . Behavior-chain mixing uses a same-kernel coalescent coupling; behavior–target first agreement controls state-coupled trace survival (Appendix § 8.2 .§ 6 ). The model-free surrogate defines a separate stopping process without transition-kernel access or mixing-time estimation.
Conclusion
MINT uses policy mismatch to adapt the backup horizon through binary stopping. Our analysis connects trace survival to variance and gives sufficient conditions for convergence in fixed-policy evaluation, while a model-free stopping rule makes the approach practical. Experiments show that adaptive truncation can improve learning performance and reduce target variability, although the benefits vary across tasks and benchmarks. These results motivate treating backup length as part of estimator design and adapting it to balance long-horizon learning signals with target variability.
Acknowledgments and Disclosure of Funding
This work is in part supported by the National Research Foundation of Korea (NRF, RS-2024-00451435(20%), RS-2024-00413957(10%), RS-2026-25547694(20%)), Institute of Information & communications Technology Planning & Evaluation (IITP, RS-2025-02305453(10%), RS-2025-02273157(10%), RS-2025-25442149(15%), RS-2021-II211343(15%)) grant funded by the Ministry of Science and ICT (MSIT), Institute of New Media and Communications (INMAC), and the BK21 FOUR program of the Education, Artificial Intelligence Graduate School Program (Seoul National University), and Research Program for Future ICT Pioneers, Seoul National University in 2026.
Notation and Background
This appendix collects the standard definitions used throughout the paper: total variation (TV) distance, the coupling lemma, maximal coupling, and uniform ergodicity. These are the analytic primitives behind the meeting-time analysis of Lemma B.2 and the contraction analysis of Theorem 4.1 .
Total Variation Distance
For any two probability distributions ν 1 \nu_1 ν 1 and ν 2 \nu_2 ν 2 over S \mathcal{S} S , the total variation (TV) distance ∥ ⋅ ∥ T V \lVert\cdot\rVert_{\mathrm{TV}} ∥ ⋅ ∥ TV is
∥ ν 1 − ν 2 ∥ T V ≔ 1 2 ∑ s ∈ S ∣ ν 1 ( s ) − ν 2 ( s ) ∣ = max A ⊆ S ∣ ν 1 ( A ) − ν 2 ( A ) ∣ \lVert\nu_1-\nu_2\rVert_{\mathrm{TV}} \coloneqq\frac{1}{2}\sum_{s\in\mathcal{S}} \lvert\nu_1(s)-\nu_2(s)\rvert= \max_{A\subseteq\mathcal{S}} \lvert\nu_1(A)-\nu_2(A)\rvert ∥ ν 1 − ν 2 ∥ TV : = 2 1 s ∈ S ∑ ∣ ν 1 ( s ) − ν 2 ( s )∣ = A ⊆ S max ∣ ν 1 ( A ) − ν 2 ( A )∣
where the norm ∥ ⋅ ∥ \lVert\cdot\rVert ∥ ⋅ ∥ corresponds to the L 1 L_1 L 1 metric. By definition, the TV distance takes values in [ 0 , 1 ] [0,1] [ 0 , 1 ] .
Coupling Lemma and Maximal Coupling
Lemma A.1 (Coupling Lemma). Let ν 1 \nu_1 ν 1 and ν 2 \nu_2 ν 2 be two probability distributions over a finite space S \mathcal{S} S . For every coupling ( X , Y ) (X,Y) ( X , Y ) of ν 1 \nu_1 ν 1 and ν 2 \nu_2 ν 2 ,
P ( X ≠ Y ) ≥ ∥ ν 1 − ν 2 ∥ T V . \mathbb{P}(X\ne Y)\geq\lVert\nu_1-\nu_2\rVert_{\mathrm{TV}}. P ( X = Y ) ≥ ∥ ν 1 − ν 2 ∥ TV . There exists a coupling attaining equality, called a maximal coupling:
P ( X ≠ Y ) = ∥ ν 1 − ν 2 ∥ T V = 1 − ∑ s ∈ S min ( ν 1 ( s ) , ν 2 ( s ) ) . (11) \mathbb{P}(X\ne Y)=\lVert\nu_1-\nu_2\rVert_{\mathrm{TV}}=1-\sum_{s\in\mathcal{S}}\min(\nu_1(s),\nu_2(s)). \tag*{(11)} P ( X = Y ) = ∥ ν 1 − ν 2 ∥ TV = 1 − s ∈ S ∑ min ( ν 1 ( s ) , ν 2 ( s )) . (11)
Maximal coupling minimizes the probability of disagreement P ( X ≠ Y ) \mathbb{P}(X\ne Y) P ( X = Y ) among all possible couplings of ν 1 \nu_1 ν 1 and ν 2 \nu_2 ν 2 . Throughout the paper we use the construction Equation (11) to relate the meeting time of two policy-induced chains to TV distances between their transition kernels.
We say the MDP is uniformly ergodic with parameter α e r g \alpha_{\mathrm{erg}} α erg if there exist a constant α e r g ∈ ( 0 , 1 ) \alpha_{\mathrm{erg}}\in(0,1) α erg ∈ ( 0 , 1 ) and C e r g > 0 C_{\mathrm{erg}}>0 C erg > 0 such that
max s ∈ S ∥ P β ( t ) ( s , ⋅ ) − μ β ∥ T V ≤ C e r g ( 1 − α e r g ) t \max_{s\in\mathcal{S}}\lVert\mathcal{P}_{\beta}^{(t)}(s,\cdot)-\mu_{\beta}\rVert_{\mathrm{TV}}\leq C_{\mathrm{erg}}(1-\alpha_{\mathrm{erg}})^t s ∈ S max ∥ P β ( t ) ( s , ⋅ ) − μ β ∥ TV ≤ C erg ( 1 − α erg ) t
for all t ∈ N t\in\mathbb{N} t ∈ N . Uniform ergodicity is a standard mild assumption in the analysis of perturbed Markov chains [30 , 38 , 22 ] .
Data-Processing Inequality for Total Variation
Lemma (TV contraction under Markov kernel). Let P P P and Q Q Q be two probability measures over a measurable space ( X , F ) (\mathcal{X},\mathcal{F}) ( X , F ) , and let f : X → Y f:\mathcal{X}\to\mathcal{Y} f : X → Y be a stochastic map, i.e., a Markov kernel such that f ( ⋅ ∣ x ) f(\cdot\mid x) f ( ⋅ ∣ x ) is a probability distribution over Y \mathcal{Y} Y for each x ∈ X x\in\mathcal{X} x ∈ X . Then the push-forward measures P Y , Q Y P_Y,Q_Y P Y , Q Y on Y \mathcal{Y} Y defined by
P Y ( B ) = ∫ X f ( B ∣ x ) d P ( x ) , Q Y ( B ) = ∫ X f ( B ∣ x ) d Q ( x ) P_Y(B)=\int_{\mathcal{X}} f(B\mid x)\,\mathrm{d}P(x),\qquad Q_Y(B)=\int_{\mathcal{X}} f(B\mid x)\,\mathrm{d}Q(x) P Y ( B ) = ∫ X f ( B ∣ x ) d P ( x ) , Q Y ( B ) = ∫ X f ( B ∣ x ) d Q ( x ) satisfy
T V ( P Y , Q Y ) ≤ T V ( P , Q ) . \mathrm{TV}(P_Y,Q_Y)\leq\mathrm{TV}(P,Q). TV ( P Y , Q Y ) ≤ TV ( P , Q ) .
Proof. Let μ = P − Q \mu=P-Q μ = P − Q , so μ ( X ) = 0 \mu(\mathcal{X})=0 μ ( X ) = 0 . For any measurable B ⊆ Y B\subseteq\mathcal{Y} B ⊆ Y , the function g B ( x ) = f ( B ∣ x ) g_B(x)=f(B\mid x) g B ( x ) = f ( B ∣ x ) lies in [ 0 , 1 ] [0,1] [ 0 , 1 ] . The variational form sup 0 ≤ g ≤ 1 ∣ ∫ g d μ ∣ = T V ( P , Q ) \sup_{0\leq g\leq1}\lvert\int g\,\mathrm{d}\mu\rvert=\mathrm{TV}(P,Q) sup 0 ≤ g ≤ 1 ∣ ∫ g d μ ∣ = TV ( P , Q ) therefore gives
T V ( P Y , Q Y ) = sup B ∣ ∫ g B d ( P − Q ) ∣ ≤ T V ( P , Q ) . \mathrm{TV}(P_Y,Q_Y)=\sup_B\left\lvert\int g_B\,\mathrm{d}(P-Q)\right\rvert\leq\mathrm{TV}(P,Q). TV ( P Y , Q Y ) = B sup ∫ g B d ( P − Q ) ≤ TV ( P , Q ) .
Interpretation. Applying a stochastic transformation can only reduce, not increase, the distinguishability of two distributions. In RL, this explains why the discrepancy between next-state distributions under two policies is bounded by their action-level difference:
TV ( ∑ a π ( a ∣ s ) P ( ⋅ ∣ s , a ) , ∑ a β ( a ∣ s ) P ( ⋅ ∣ s , a ) ) ≤ TV ( π ( ⋅ ∣ s ) , β ( ⋅ ∣ s ) ) . \operatorname{TV}\left(\sum_a \pi(a \mid s)P(\cdot\mid s,a), \sum_a \beta(a \mid s)P(\cdot\mid s,a)\right) \le\operatorname{TV}\left(\pi(\cdot\mid s),\beta(\cdot\mid s)\right). TV ( a ∑ π ( a ∣ s ) P ( ⋅ ∣ s , a ) , a ∑ β ( a ∣ s ) P ( ⋅ ∣ s , a ) ) ≤ TV ( π ( ⋅ ∣ s ) , β ( ⋅ ∣ s ) ) .
This principle underlies the use of action-level disagreement probabilities p ^ i \hat{p}_i p ^ i in MINT as a proxy for policy divergence propagated through the dynamics.
Standing Assumptions
This subsection states the kernel-level TV bound and cross-Doeblin condition used by the supporting results. The assumption usage table (Table 4 ) distinguishes these hypotheses from those of the main contraction theorem and the action-level surrogate theorem.
Assumption A.2 (Kernel-level TV bound). For the fixed policy pair, define the kernel discrepancy by
d : = sup s ∈ S d ( s ) , d ( s ) : = ∥ P β ( s , ⋅ ) − P π ( s , ⋅ ) ∥ TV . d := \sup_{s \in\mathcal{S}} d(s), \qquad d(s) := \left\|P^\beta(s,\cdot)-P^\pi(s,\cdot)\right\|_{\operatorname{TV}}. d := s ∈ S sup d ( s ) , d ( s ) := P β ( s , ⋅ ) − P π ( s , ⋅ ) TV . Thus d ∈ [ 0 , 1 ] d \in[0,1] d ∈ [ 0 , 1 ] and data processing gives d ≤ D : = sup s TV ( β ( ⋅ ∣ s ) , π ( ⋅ ∣ s ) ) d \le D := \sup_s \operatorname{TV}(\beta(\cdot\mid s),\pi(\cdot\mid s)) d ≤ D := sup s TV ( β ( ⋅ ∣ s ) , π ( ⋅ ∣ s )) . This comparison concerns the actual supremum defined here; an arbitrary looser numerical upper bound on d d d can exceed D D D .
Assumption A.3 (Cross-Doeblin). There exists ρ ∈ ( 0 , 1 − d ) \rho\in(0,1-d) ρ ∈ ( 0 , 1 − d ) such that for every pair of states ( s , s ′ ) ∈ S × S (s,s') \in\mathcal{S} \times\mathcal{S} ( s , s ′ ) ∈ S × S and every policy pair ( β , π ) (\beta,\pi) ( β , π ) ,
∥ P β ( s , ⋅ ) − P π ( s ′ , ⋅ ) ∥ TV ≤ 1 − ρ . \left\|P^\beta(s,\cdot)-P^\pi(s',\cdot)\right\|_{\operatorname{TV}} \le1-\rho. P β ( s , ⋅ ) − P π ( s ′ , ⋅ ) TV ≤ 1 − ρ . The cross-Doeblin condition requires the existence of a uniform overlap ρ ∈ ( 0 , 1 − d ) \rho\in(0,1-d) ρ ∈ ( 0 , 1 − d ) such that ∥ P β ( s , ⋅ ) − P π ( s ′ , ⋅ ) ∥ TV ≤ 1 − ρ \left\|P^\beta(s,\cdot)-P^\pi(s',\cdot)\right\|_{\operatorname{TV}} \le1-\rho P β ( s , ⋅ ) − P π ( s ′ , ⋅ ) TV ≤ 1 − ρ for all state pairs and all policies. Combined with the maximal coupling Equation (11) , it ensures that conditional on disagreement at step t t t , the coupled processes meet at step t + 1 t+1 t + 1 with probability at least ρ \rho ρ , which is what produces the geometric meeting-time tail used in Corollary B.1.
Where each assumption is used. Table 4 distinguishes the state-coupled analysis from the finite-state, fixed-policy surrogate theorem. The latter uses no deterministic-dynamics or cross-Doeblin hypothesis, but requires its own χ λ \chi_\lambda χ λ sufficient condition. The one-step DPI proposition is not a transfer theorem for contraction.
Cross-Doeblin refers to Assumption A.3; kernel TV d d d is defined in Assumption A.2. A dash means no additional condition is listed beyond the stated setup.
Cross-Doeblin and Two TV Quantities
Two TV quantities, two roles. By the data-processing inequality (Appendix A.4), the kernel-level TV bound d d d from Assumption A.2 satisfies d ≤ D : = sup s D s d \le D := \sup_s D_s d ≤ D := sup s D s . Both quantities measure off-policy discrepancy but at different “levels”:
d d d enters the cross-Doeblin assumption ρ ∈ ( 0 , 1 − d ) \rho\in(0,1-d) ρ ∈ ( 0 , 1 − d ) , controlling the kernel-level meeting rate.
D ˉ w g t \bar{D}_{\mathrm{wgt}} D ˉ wgt enters the contraction condition Equation (20) , controlling the per-step weight deviation in the operator analysis.
D s D_s D s is action-level policy mismatch, while D ˉ w g t \bar{D}_{\mathrm{wgt}} D ˉ wgt weights it by the state-coupled survival process. The implemented surrogate has its own survival process; its contraction correction weights χ λ ( s ) \chi_{\lambda}(s) χ λ ( s ) instead (Theorem B.9). These weighted quantities can differ.
Justification of the action-level approximation. For transitions from a shared state s s s , define d ( s ) = TV ( P β ( ⋅ ∣ s ) , P π ( ⋅ ∣ s ) ) d(s) = \operatorname{TV}(P^{\beta}(\cdot\mid s), P^{\pi}(\cdot\mid s)) d ( s ) = TV ( P β ( ⋅ ∣ s ) , P π ( ⋅ ∣ s )) . Data processing bounds d ( s ) d(s) d ( s ) by D s D_s D s . The implementation samples an action-dependent keep probability whose mean is D s D_s D s , as follows; this does not identify it with the disagreement probability of a coupled pair at different states.
(i) Always valid. Since min { 1 , π / β } ∈ [ 0 , 1 ] \min\{1,\pi/\beta\} \in[0,1] min { 1 , π / β } ∈ [ 0 , 1 ] , we have p ^ i ∈ [ 0 , 1 ] \hat{p}_i \in[0,1] p ^ i ∈ [ 0 , 1 ] regardless of the relative magnitudes of π ( a i ∣ s i ) \pi(a_i \mid s_i) π ( a i ∣ s i ) and β ( a i ∣ s i ) \beta(a_i \mid s_i) β ( a i ∣ s i ) , including the unbounded-ratio regime where π / β > 1 \pi/\beta> 1 π / β > 1 .
(ii) Unbiased estimator of the action-level policy TV. Taking the expectation over a i ∼ β ( ⋅ ∣ s i ) a_i \sim\beta(\cdot\mid s_i) a i ∼ β ( ⋅ ∣ s i ) and using the variational identity for total variation,
E a ∼ β [ min { 1 , π ( a ∣ s ) β ( a ∣ s ) } ] = ∑ a min { β ( a ∣ s ) , π ( a ∣ s ) } = 1 − TV ( β ( ⋅ ∣ s ) , π ( ⋅ ∣ s ) ) , (12) \mathbb{E}_{a\sim\beta}\left[\min\left\{1,\frac{\pi(a\mid s)}{\beta(a\mid s)}\right\}\right] = \sum_a \min\{\beta(a\mid s),\pi(a\mid s)\} = 1-\operatorname{TV}(\beta(\cdot\mid s),\pi(\cdot\mid s)), \tag*{(12)} E a ∼ β [ min { 1 , β ( a ∣ s ) π ( a ∣ s ) } ] = a ∑ min { β ( a ∣ s ) , π ( a ∣ s )} = 1 − TV ( β ( ⋅ ∣ s ) , π ( ⋅ ∣ s )) , (12)
E a ∼ β [ p ^ i ∣ s i ] = TV ( β ( ⋅ ∣ s i ) , π ( ⋅ ∣ s i ) ) = D s i . (13) \mathbb{E}_{a\sim\beta}[\hat{p}_i\mid s_i] = \operatorname{TV}(\beta(\cdot\mid s_i),\pi(\cdot\mid s_i)) = D_{s_i}. \tag*{(13)} E a ∼ β [ p ^ i ∣ s i ] = TV ( β ( ⋅ ∣ s i ) , π ( ⋅ ∣ s i )) = D s i . (13)
Thus, p ^ i \hat{p}_i p ^ i is an unbiased sample-level estimate of D s i D_{s_i} D s i . Its conditional mean, rather than each realized sample, upper-bounds d ( s i ) d(s_i) d ( s i ) , the one-step kernel TV from a shared state.
(iii) Fully model-free. p ^ i \hat{p}_i p ^ i depends only on the policy ratio at the sampled action and never queries the environment transition kernel, so it is computable in any actor-critic implementation (cf. Algorithm 1).
(iv) One-step data processing under arbitrary dynamics.
Proposition A.4 (One-step surrogate and kernel TV). For an arbitrary transition kernel, let d ( s ) : = TV ( P β ( ⋅ ∣ s ) , P π ( ⋅ ∣ s ) ) d(s) := \operatorname{TV}(\mathcal{P}^{\beta}(\cdot\mid s),\mathcal{P}^{\pi}(\cdot\mid s)) d ( s ) := TV ( P β ( ⋅ ∣ s ) , P π ( ⋅ ∣ s )) . Then
E a ∼ β ( ⋅ ∣ s ) [ p ^ ( s , a ) ] = D s ≥ d ( s ) . \mathbb{E}_{a\sim\beta(\cdot\mid s)}[\hat{p}(s,a)] = D_s \ge d(s). E a ∼ β ( ⋅ ∣ s ) [ p ^ ( s , a )] = D s ≥ d ( s ) . If the next states are maximally coupled from a shared state s s s , their one-step disagreement probability is d ( s ) d(s) d ( s ) . Equality D s = d ( s ) D_s=d(s) D s = d ( s ) holds, in particular, for deterministic dynamics with an injective action-to-next-state map.
Proof. The expectation identity is proved in (ii). Both next-state laws are the push-forwards of the two action laws through the common kernel a ↦ P ( ⋅ ∣ s , a ) a\mapsto\mathcal{P}(\cdot\mid s,a) a ↦ P ( ⋅ ∣ s , a ) , so data processing gives d ( s ) ≤ D s d(s)\le D_s d ( s ) ≤ D s . Maximal coupling identifies d ( s ) d(s) d ( s ) with one-step disagreement from that shared state. An injective deterministic map preserves TV, giving the stated sufficient equality case.
This proposition is an expectation-level, one-step statement. It neither gives a samplewise upper bound on kernel disagreement nor orders the multi-step stopping times once the two state processes differ. The action-dependent surrogate contraction theorem is Theorem B.9 ; it uses the surrogate’s own survival quantities and the negative-mass fraction χ λ \chi_\lambda χ λ .
Mixing-Time Parameters and On-Policy Look-Ahead
Minorization-based mixing parameter. The parameter τ ∗ β ≥ 1 \tau_*^\beta\ge1 τ ∗ β ≥ 1 in Equation (3) belongs to the common data-generating and target chain in the on-policy analysis of Duan and Wainwright [10 ] . Their mixing condition requires a probability measure ν β \nu_\beta ν β and a finite constant C ν ≥ 0 C_\nu\ge0 C ν ≥ 0 such that
P β ( s , B ) ≥ 1 τ ∗ β ν β ( B ) for all s and measurable B , \mathcal{P}^\beta(s,B) \ge\frac{1}{\tau_*^\beta}\nu_\beta(B) \quad\text{for all }s\text{ and measurable }B, P β ( s , B ) ≥ τ ∗ β 1 ν β ( B ) for all s and measurable B ,
∥ d ν β d μ β ∥ ∞ ≤ 1 + C ν . \left\|\frac{d\nu_\beta}{d\mu_\beta}\right\|_\infty\le1+C_\nu. d μ β d ν β ∞ ≤ 1 + C ν .
Thus all one-step transition distributions contain a common probability component of weight 1 / τ ∗ β 1/\tau_*^\beta 1/ τ ∗ β . This is a minorization condition, rather than the TV-threshold definition of τ m i x β \tau_{\mathrm{mix}}^\beta τ mix β in § 2.2 ; the two parameters are not identical by definition [10 Section 3.1.1, Eq. (11)] .
Scope of the look-ahead condition. Duan and Wainwright [10 ] analyze multi-step kernel least-squares TD estimates. Their error bounds distinguish martingale fluctuations from a Bellman-residual contribution affected by temporal dependence. In the uniformly bounded-reward setting, consistent with our assumption ∣ r ( s , a ) ∣ ≤ r max |r(s,a)|\le r_{\max} ∣ r ( s , a ) ∣ ≤ r m a x , Equation (3) is sufficient for their stated finite-sample bounds, subject to their mixing condition, boundedness and regularity assumptions on the kernel and its eigenfunctions, and sample-size requirements [10 Sections 3.1.1 and 3.3.2, Eq. (29b)] . The look-ahead condition alone is not an off-policy guarantee.
Behavior-chain mixing versus off-policy stopping. A coalescent coupling of two behavior chains from different initial states bounds τ m i x β \tau_{\mathrm{mix}}^\beta τ mix β . The behavior–target first agreement instead controls MINT trace survival. These two couplings and their stopping times are distinguished in Appendix § 8.2 .§ 6 .
Proofs
Proof of the Variance Lemma (Lemma 3.3)
Joint sampling and filtration. Let H t \mathcal{H}_t H t contain both state histories, all past actions and rewards, and coupling randomness revealed through time t t t , before sampling the current behavior action. At policy-generated steps, draw a t ∼ β ( ⋅ ∣ s t ) a_t \sim\beta(\cdot\mid s_t) a t ∼ β ( ⋅ ∣ s t ) conditionally on H t \mathcal{H}_t H t , and set G t = H t ∨ σ ( a t ) \mathcal{G}_t = \mathcal{H}_t \vee\sigma(a_t) G t = H t ∨ σ ( a t ) . For t ≥ 1 t \ge1 t ≥ 1 , the arrival-state mask A t A_t A t and cumulative mask W t W_t W t are H t \mathcal{H}_t H t -measurable, hence G t \mathcal{G}_t G t -measurable; W 0 = 1 W_0 = 1 W 0 = 1 is deterministic. The enlarged history contains no information from the next reward or transition beyond that allowed by the environment law. We require
L ( r t , s t + 1 ∣ G t ) = L ( r t , s t + 1 ∣ s t , a t ) . \mathcal{L}(r_t,s_{t+1}\mid\mathcal{G}_t)=\mathcal{L}(r_t,s_{t+1}\mid s_t,a_t). L ( r t , s t + 1 ∣ G t ) = L ( r t , s t + 1 ∣ s t , a t ) .
This compatibility is part of the construction, not a consequence of marginal coupling alone. For example, let Γ t ( u , v ) \Gamma_t(u,v) Γ t ( u , v ) be a maximal coupling of P β ( s t , ⋅ ) \mathcal{P}^{\beta}(s_t,\cdot) P β ( s t , ⋅ ) and P π ( S t π , ⋅ ) \mathcal{P}^{\pi}(S_t^\pi,\cdot) P π ( S t π , ⋅ ) , given H t \mathcal{H}_t H t . First draw a t ∼ β ( ⋅ ∣ s t ) a_t \sim\beta(\cdot\mid s_t) a t ∼ β ( ⋅ ∣ s t ) and the behavior reward and next state u u u from the environment. Then draw the auxiliary next state v v v from Γ t ( u , v ) / P β ( s t , u ) \Gamma_t(u,v)/\mathcal{P}^{\beta}(s_t,u) Γ t ( u , v ) / P β ( s t , u ) on positive-denominator rows. Averaging over a t a_t a t recovers Γ t \Gamma_t Γ t and both policy-induced state marginals; the behavior transition law given G t \mathcal{G}_t G t is unchanged. Zero-denominator rows can be assigned any conditional law, since they have zero probability under a policy-generated step. If the initial pair x x x is prescribed, use the induced conditional initial coupling, with a valid behavior transition on any null-support extension. All Pr x ( T > t ) \Pr_x(T>t) Pr x ( T > t ) refer to that specified initialization.
Conditional and marginal coupling. For i ≥ 1 i \ge1 i ≥ 1 , let p i : = Pr ( A i = 1 ∣ H i − 1 ) p_i := \Pr(A_i=1\mid\mathcal{H}_{i-1}) p i := Pr ( A i = 1 ∣ H i − 1 ) . At a policy-generated step, conditional maximal coupling gives
p i = TV ( P β ( S i − 1 β , ⋅ ) , P π ( S i − 1 π , ⋅ ) ) . p_i=\operatorname{TV}\left(\mathcal{P}^{\beta}(S^\beta_{i-1},\cdot),\mathcal{P}^{\pi}(S^\pi_{i-1},\cdot)\right). p i = TV ( P β ( S i − 1 β , ⋅ ) , P π ( S i − 1 π , ⋅ ) ) .
When the initial action is prescribed, the first-step law is instead the conditional initial coupling specified above. Conditional maximal coupling at each step does not imply maximal coupling of the unconditional time-i i i marginals. In general,
TV ( L ( S i β ) , L ( S i π ) ) ≤ Pr ( A i = 1 ) . \operatorname{TV}\left(\mathcal{L}(S^\beta_i),\mathcal{L}(S^\pi_i)\right)\leq\Pr(A_i=1). TV ( L ( S i β ) , L ( S i π ) ) ≤ Pr ( A i = 1 ) .
Survival functional and its bounds. For T = T β , π T=T_{\beta,\pi} T = T β , π , summing the geometric series along each stopped trajectory gives
M x ( γ , λ ) = E x [ 1 − ( γ λ ) T 1 − γ λ ] ≤ min { E x [ T ] , 1 1 − γ λ } , M_x(\gamma,\lambda)=\mathbb{E}_x\left[\frac{1-(\gamma\lambda)^T}{1-\gamma\lambda}\right]\leq\min\left\{\mathbb{E}_x[T],\frac{1}{1-\gamma\lambda}\right\}, M x ( γ , λ ) = E x [ 1 − γ λ 1 − ( γ λ ) T ] ≤ min { E x [ T ] , 1 − γ λ 1 } ,
with ( γ λ ) ∞ = 0 (\gamma\lambda)^\infty=0 ( γ λ ) ∞ = 0 . For a distribution of starting pairs, M M M without subscript denotes the corresponding average of M x M_x M x . The contraction bound in Theorem 4.1 instead retains each starting pair and takes a supremum; this average cannot replace that supremum.
Martingale and fixed point. Let δ t = δ t Q π \delta_t=\delta_t^{Q^\pi} δ t = δ t Q π , W t = ∏ i = 1 t A i = 1 { T > t } W_t=\prod_{i=1}^{t}A_i=\mathbf{1}\{T>t\} W t = ∏ i = 1 t A i = 1 { T > t } , and α = γ λ \alpha=\gamma\lambda α = γ λ . The Bellman identity and the conditional environment law give
E x [ δ t ∣ G t ] = ( T π Q π ) ( s t , a t ) − Q π ( s t , a t ) = 0. \mathbb{E}_x[\delta_t\mid\mathcal{G}_t]=(\mathcal{T}^\pi Q^\pi)(s_t,a_t)-Q^\pi(s_t,a_t)=0. E x [ δ t ∣ G t ] = ( T π Q π ) ( s t , a t ) − Q π ( s t , a t ) = 0.
The mask W t W_t W t is G t \mathcal{G}_t G t -measurable, hence E x [ W t δ t ] = 0 \mathbb{E}_x[W_t\delta_t]=0 E x [ W t δ t ] = 0 . The deterministic bound ∣ δ t ∣ ≤ C : = ∥ r ∥ ∞ + ( 1 + γ ) ∥ Q π ∥ ∞ |\delta_t|\leq C:=\|r\|_\infty+(1+\gamma)\|Q^\pi\|_\infty ∣ δ t ∣ ≤ C := ∥ r ∥ ∞ + ( 1 + γ ) ∥ Q π ∥ ∞ and α < 1 \alpha<1 α < 1 justify exchanging the sum and expectation. Thus E x [ Δ ^ ] = 0 \mathbb{E}_x[\widehat{\Delta}]=0 E x [ Δ ] = 0 and R ρ , λ Q π = Q π \mathcal{R}_{\rho,\lambda}Q^\pi=Q^\pi R ρ , λ Q π = Q π .
Cross terms and idempotence. For u < t u<t u < t , W u , W t , δ u W_u,W_t,\delta_u W u , W t , δ u are G t \mathcal{G}_t G t -measurable. Therefore
E x [ W u W t δ u δ t ] = E x [ W u W t δ u E x [ δ t ∣ G t ] ] = 0. \mathbb{E}_x[W_uW_t\delta_u\delta_t]=\mathbb{E}_x[W_uW_t\delta_u\mathbb{E}_x[\delta_t\mid\mathcal{G}_t]]=0. E x [ W u W t δ u δ t ] = E x [ W u W t δ u E x [ δ t ∣ G t ]] = 0.
This orthogonality applies to other adapted weights as well. For the binary masks, W t 2 = W t W_t^2=W_t W t 2 = W t additionally yields
Var x ( Δ ^ ) = E x [ Δ ^ 2 ] = ∑ t ≥ 0 α 2 t E x [ W t δ t 2 ] ≤ C 2 ∑ t ≥ 0 ( γ 2 λ 2 ) t Pr x ( T > t ) = C 2 M x ( γ 2 , λ 2 ) . (14) \begin{aligned}
\operatorname{Var}_x(\widehat{\Delta})=\mathbb{E}_x[\widehat{\Delta}^2]=\sum_{t\geq0}\alpha^{2t}\mathbb{E}_x[W_t\delta_t^2] \\
&\leq C^2\sum_{t\geq0}(\gamma^2\lambda^2)^t\Pr_x(T>t)=C^2M_x(\gamma^2,\lambda^2). \tag*{(14)}
\end{aligned} Var x ( Δ ) = E x [ Δ 2 ] = t ≥ 0 ∑ α 2 t E x [ W t δ t 2 ] ≤ C 2 t ≥ 0 ∑ ( γ 2 λ 2 ) t x Pr ( T > t ) = C 2 M x ( γ 2 , λ 2 ) . (14)
One can first expand finite sums and then pass to the limit: ∣ Δ ^ ∣ ≤ C / ( 1 − α ) |\widehat{\Delta}| \le C/(1-\alpha) ∣ Δ ∣ ≤ C / ( 1 − α ) gives dominated convergence. Averaging over a specified starting distribution gives the corresponding C 2 M ( γ 2 , λ 2 ) C^2M(\gamma^2,\lambda^2) C 2 M ( γ 2 , λ 2 ) bound. No claim of universal variance dominance over clipped trace methods follows from this calculation.
Proof of Main contraction
The normalized policy-TV average in § 4 is discounted and survival-weighted through M x M_x M x , N x N_x N x of Definition 3.2; D ˉ w g t \bar{D}_{\mathrm{wgt}} D ˉ wgt takes a supremum over starting pairs, not individual times. By definition,
N x = D w g t x ( M x − 1 ) ≤ D ˉ w g t ( M x − 1 ) , D ˉ w g t ≤ sup s D s = : D ≤ 1. N_x = D^{x}_{\mathrm{wgt}}(M_x-1) \le\bar{D}_{\mathrm{wgt}}(M_x-1), \qquad\bar{D}_{\mathrm{wgt}} \le\sup_s D_s =: D \le1. N x = D wgt x ( M x − 1 ) ≤ D ˉ wgt ( M x − 1 ) , D ˉ wgt ≤ s sup D s =: D ≤ 1.
Fix a starting pair x x x and use the joint sampling conditions of § 3 . Set W t = ∏ i = 1 t A i W_t = \prod_{i=1}^{t} A_i W t = ∏ i = 1 t A i , W 0 = 1 W_0=1 W 0 = 1 . The arrival-state mask A t A_t A t , W t − 1 W_{t-1} W t − 1 , and s t s_t s t are known before a t a_t a t is sampled; thus they are H t \mathcal{H}_t H t -measurable. The conditional action law is β ( ⋅ ∣ s t ) \beta(\cdot\mid s_t) β ( ⋅ ∣ s t ) . We retain A t A_t A t in this calculation instead of replacing it by a conditional mean taken with respect to a different filtration.
Signed coefficients. For q = Q − Q ′ q=Q-Q' q = Q − Q ′ , expanding the TD-error sum and telescoping gives
( R p , λ Q − R p , λ Q ′ ) ( x ) = ∑ t ≥ 1 γ t λ t − 1 E x [ W t − 1 ∑ b ( π ( b ∣ s t ) − λ A t β ( b ∣ s t ) ) q ( s t , b ) ] . (\mathcal{R}_{p,\lambda}Q-\mathcal{R}_{p,\lambda}Q')(x)=\sum_{t\ge1}\gamma^t\lambda^{t-1}\mathbb{E}_x\left[W_{t-1}\sum_b\left(\pi(b\mid s_t)-\lambda A_t\beta(b\mid s_t)\right)q(s_t,b)\right]. ( R p , λ Q − R p , λ Q ′ ) ( x ) = t ≥ 1 ∑ γ t λ t − 1 E x [ W t − 1 b ∑ ( π ( b ∣ s t ) − λ A t β ( b ∣ s t ) ) q ( s t , b ) ] .
The conditional environment law justifies the shift of the bootstrap term to the next visited state; the pre-action conditioning justifies averaging q ( s t , a t ) q(s_t,a_t) q ( s t , a t ) using β \beta β . Let w x ; y , b w_{x;y,b} w x ; y , b be the resulting signed coefficients and define
b x : = ∑ y , b ∣ w x ; y , b ∣ . (15) b_x:=\sum_{y,b}|w_{x;y,b}|. \tag*{(15)} b x := y , b ∑ ∣ w x ; y , b ∣. (15)
For any a ∈ { 0 , 1 } a\in\{0,1\} a ∈ { 0 , 1 } ,
∑ b ∣ π ( b ∣ s ) − λ a β ( b ∣ s ) ∣ = 1 − λ a + 2 ∑ b [ λ a β ( b ∣ s ) − π ( b ∣ s ) ] + \sum_b|\pi(b\mid s)-\lambda a\beta(b\mid s)|=1-\lambda a+2\sum_b[\lambda a\beta(b\mid s)-\pi(b\mid s)]_+ b ∑ ∣ π ( b ∣ s ) − λa β ( b ∣ s ) ∣ = 1 − λa + 2 b ∑ [ λa β ( b ∣ s ) − π ( b ∣ s ) ] +
≤ 1 − λ a + 2 λ a D s . (16) \le1-\lambda a+2\lambda aD_s. \tag*{(16)} ≤ 1 − λa + 2 λa D s . (16)
Indeed, λ a ≤ 1 \lambda a\le1 λa ≤ 1 and [ λ a β − π ] + ≤ λ a [ β − π ] + [\lambda a\beta-\pi]_+\le\lambda a[\beta-\pi]_+ [ λa β − π ] + ≤ λa [ β − π ] + , whose action sum is λ a D s \lambda aD_s λa D s . Consequently,
b x ≤ M 1 , x + 2 Δ λ , x , (17) b_x\le M_{1,x}+2\Delta_{\lambda,x}, \tag*{(17)} b x ≤ M 1 , x + 2 Δ λ , x , (17)
where
M 1 , x : = ∑ t ≥ 1 γ t λ t − 1 E x [ W t − 1 ( 1 − λ A t ) ] , Δ λ , x : = ∑ t ≥ 1 ( γ λ ) t E x [ W t D s t ] = N x . M_{1,x}:=\sum_{t\ge1}\gamma^t\lambda^{t-1}\mathbb{E}_x[W_{t-1}(1-\lambda A_t)],\qquad\Delta_{\lambda,x}:=\sum_{t\ge1}(\gamma\lambda)^t\mathbb{E}_x[W_tD_{s_t}]=N_x. M 1 , x := t ≥ 1 ∑ γ t λ t − 1 E x [ W t − 1 ( 1 − λ A t )] , Δ λ , x := t ≥ 1 ∑ ( γ λ ) t E x [ W t D s t ] = N x .
The survival identity W t = W t − 1 A t W_t=W_{t-1}A_t W t = W t − 1 A t gives
M 1 , x = γ M x − ( M x − 1 ) = 1 − ( 1 − γ ) M x . M_{1,x}=\gamma M_x-(M_x-1)=1-(1-\gamma)M_x. M 1 , x = γ M x − ( M x − 1 ) = 1 − ( 1 − γ ) M x .
Combining the two terms proves
b x ≤ κ x : = 1 − ( 1 − γ ) M x + 2 N x ≤ 1 − ( 1 − γ ) M x + 2 D ˉ w g t ( M x − 1 ) . (18) b_x\le\kappa_x:=1-(1-\gamma)M_x+2N_x\le1-(1-\gamma)M_x+2\bar{D}_{\mathrm{wgt}}(M_x-1). \tag*{(18)} b x ≤ κ x := 1 − ( 1 − γ ) M x + 2 N x ≤ 1 − ( 1 − γ ) M x + 2 D ˉ wgt ( M x − 1 ) . (18)
At λ = 0 \lambda=0 λ = 0 the same formula follows directly from R p , 0 = T \mathcal{R}_{p,0}=T R p , 0 = T ; equivalently, use the t = 1 t=1 t = 1 term in the telescoping expression. Consequently, for all bounded Q , Q ′ Q,Q' Q , Q ′ and every starting pair x x x ,
∣ R p , λ Q ( x ) − R p , λ Q ′ ( x ) ∣ ≤ κ x ∥ Q − Q ′ ∥ ∞ . (19) |\mathcal{R}_{p,\lambda}Q(x)-\mathcal{R}_{p,\lambda}Q'(x)|\le\kappa_x\|Q-Q'\|_\infty. \tag*{(19)} ∣ R p , λ Q ( x ) − R p , λ Q ′ ( x ) ∣ ≤ κ x ∥ Q − Q ′ ∥ ∞ . (19)
Taking the supremum over x x x proves the claimed Lipschitz bound. The fixed-point identity of Lemma 3.3 gives uniqueness and convergence when this supremum is below one.
Since 1 ≤ M x ≤ ( 1 − γ λ ) − 1 1\le M_x\le(1-\gamma\lambda)^{-1} 1 ≤ M x ≤ ( 1 − γ λ ) − 1 uniformly,
sup x { 1 − ( 1 − γ ) M x + 2 N x } < 1 ⟺ sup x 2 N x M x < 1 − γ . (20) \sup_x\{1-(1-\gamma)M_x+2N_x\}<1\quad\Longleftrightarrow\quad\sup_x\frac{2N_x}{M_x}<1-\gamma. \tag*{(20)} x sup { 1 − ( 1 − γ ) M x + 2 N x } < 1 ⟺ x sup M x 2 N x < 1 − γ . (20)
The uniform bounds on M x M_x M x preserve a strictly positive margin in both directions. Substituting N x ≤ D ˉ w g t ( M x − 1 ) N_x\le\bar{D}_{\mathrm{wgt}}(M_x-1) N x ≤ D ˉ wgt ( M x − 1 ) gives the sufficient condition
sup x { 2 D ˉ w g t M x − 1 M x } < 1 − γ . (21) \sup_x\left\{2\bar{D}_{\mathrm{wgt}}\frac{M_x-1}{M_x}\right\}<1-\gamma. \tag*{(21)} x sup { 2 D ˉ wgt M x M x − 1 } < 1 − γ . (21)
This uses the survival-weighted identity above without requiring a bound on the TV average at each individual time. A value of M M M averaged over an initial distribution cannot replace the supremum in Equation (9) . This proof concerns the state-coupled operator; replacing p i p_i p i by p ^ i \hat{p}_i p ^ i is not used. The action-dependent surrogate requires its own negative-weight bound, developed in Appendix § 8.2 .8.
Connection to the λ \lambda λ -dependent effective horizon. For every starting pair,
M x ≤ K λ x : = min ( E x [ T ] , 1 1 − γ λ ) . (22) M_x \le K_\lambda^x := \min\left(\mathbb{E}_x[T],\frac{1}{1-\gamma\lambda}\right). \tag*{(22)} M x ≤ K λ x := min ( E x [ T ] , 1 − γ λ 1 ) . (22)
K λ : = sup x K λ x ≤ 1 1 − γ λ . K_\lambda:= \sup_x K_\lambda^x \le\frac{1}{1-\gamma\lambda}. K λ := x sup K λ x ≤ 1 − γ λ 1 .
This follows from the pointwise inequality ∑ t = 0 T − 1 ( γ λ ) t ≤ min ( T , ( 1 − γ λ ) − 1 ) \sum_{t=0}^{T-1}(\gamma\lambda)^t \le\min(T,(1-\gamma\lambda)^{-1}) ∑ t = 0 T − 1 ( γ λ ) t ≤ min ( T , ( 1 − γ λ ) − 1 ) . Since ( u − 1 ) / u (u-1)/u ( u − 1 ) / u is increasing for u ≥ 1 u \ge1 u ≥ 1 , a uniform sufficient condition is
2 D ˉ w g t K λ − 1 K λ < 1 − γ . (23) 2\bar{D}_{\mathrm{wgt}}\frac{K_\lambda-1}{K_\lambda}<1-\gamma. \tag*{(23)} 2 D ˉ wgt K λ K λ − 1 < 1 − γ . (23)
The scalar K λ K_\lambda K λ here is a uniform bound over starting pairs, not an initial-distribution average. When the survival law is the same for all starting pairs, the formulas reduce to the corresponding scalar expressions.
Closed-Form Corollary and Its Proof
Corollary B.1 (Geometric meeting time — closed-form contraction). Suppose Assumption A.3 holds with parameter ρ ∈ ( 0 , 1 − d ) \rho\in(0,1-d) ρ ∈ ( 0 , 1 − d ) . Also require Pr x ( A 1 = 0 ) ≥ ρ \Pr_x(A_1=0)\ge\rho Pr x ( A 1 = 0 ) ≥ ρ uniformly over the prescribed starting pairs. This initial-step condition is needed when conditioning on the behavior’s initial action; it is not implied by overlap of policy-averaged kernels alone. Then, for every starting pair x x x , Pr x ( T β , π > t ) ≤ ( 1 − ρ ) t \Pr_x(T_{\beta,\pi}>t)\le(1-\rho)^t Pr x ( T β , π > t ) ≤ ( 1 − ρ ) t , so M x ≤ [ 1 − γ λ ( 1 − ρ ) ] − 1 M_x\le[1-\gamma\lambda(1-\rho)]^{-1} M x ≤ [ 1 − γ λ ( 1 − ρ ) ] − 1 uniformly. The uniform TV-based sufficient condition of Theorem 4.1 is ensured by the single-product condition
γ ( 1 + 2 D ˉ w g t λ ( 1 − ρ ) ) < 1. \gamma\left(1+2\bar{D}_{\mathrm{wgt}}\lambda(1-\rho)\right)<1. γ ( 1 + 2 D ˉ wgt λ ( 1 − ρ ) ) < 1. None of the factors D ˉ w g t \bar{D}_{\mathrm{wgt}} D ˉ wgt , λ \lambda λ , ( 1 − ρ ) (1-\rho) ( 1 − ρ ) needs to be small individually; only their product must be controlled.
Assumption A.3 asserts ∥ P β ( s , ⋅ ) − P π ( s ′ , ⋅ ) ∥ T V ≤ 1 − ρ \lVert P^\beta(s,\cdot)-P^\pi(s',\cdot)\rVert_{\mathrm{TV}}\leq1-\rho ∥ P β ( s , ⋅ ) − P π ( s ′ , ⋅ ) ∥ TV ≤ 1 − ρ uniformly over state pairs. Conditional maximal coupling gives a meeting probability of at least ρ \rho ρ at subsequent policy-generated transitions. Combining this with the explicit initial-step condition and iterating gives Pr x ( T β , π > t ) ≤ ( 1 − ρ ) t \Pr_x(T_{\beta,\pi}>t)\leq(1-\rho)^t Pr x ( T β , π > t ) ≤ ( 1 − ρ ) t for every x x x . Geometric summation gives
M x ≤ ∑ t ≥ 0 ( γ λ ( 1 − ρ ) ) t = 1 1 − γ λ ( 1 − ρ ) . M_x \leq\sum_{t\geq0}(\gamma\lambda(1-\rho))^t = \frac{1}{1-\gamma\lambda(1-\rho)}. M x ≤ t ≥ 0 ∑ ( γ λ ( 1 − ρ ) ) t = 1 − γ λ ( 1 − ρ ) 1 .
Writing U = [ 1 − γ λ ( 1 − ρ ) ] − 1 U=[1-\gamma\lambda(1-\rho)]^{-1} U = [ 1 − γ λ ( 1 − ρ ) ] − 1 , we have ( M x − 1 ) / M x ≤ ( U − 1 ) / U = γ λ ( 1 − ρ ) (M_x-1)/M_x\leq(U-1)/U=\gamma\lambda(1-\rho) ( M x − 1 ) / M x ≤ ( U − 1 ) / U = γ λ ( 1 − ρ ) uniformly in x x x . Substitution into Equation (21) gives the sufficient condition
γ ( 1 + 2 D ˉ w g t λ ( 1 − ρ ) ) < 1. \gamma(1+2\bar{D}_{\mathrm{wgt}}\lambda(1-\rho))<1. γ ( 1 + 2 D ˉ wgt λ ( 1 − ρ )) < 1.
Time-averaged disagreement bound
Lemma B.2 (Time-averaged disagreement; nearby chains). Consider a co-adapted coupling of the β \beta β and π \pi π state processes, with conditional maximal coupling at each policy-generated transition. Write F k \mathcal{F}_k F k for the joint history through time k k k before the next transition, A k = 1 { S k β ≠ S k π } A_k=1\{S_k^\beta\neq S_k^\pi\} A k = 1 { S k β = S k π } , and u k = E [ A k ] u_k=\mathbb{E}[A_k] u k = E [ A k ] . Under Assumptions A.2 and A.3 , set η = ρ + d ∈ ( 0 , 1 ) \eta=\rho+d\in(0,1) η = ρ + d ∈ ( 0 , 1 ) . For any initial joint law at time 1 1 1 and every integer t ≥ 1 t\geq1 t ≥ 1 ,
1 t ∑ k = 1 t u k ≤ d η + 1 − ( 1 − η ) t η t ( u 1 − d η ) . (24) \frac{1}{t}\sum_{k=1}^{t}u_k\leq\frac{d}{\eta}+\frac{1-(1-\eta)^t}{\eta t}\left(u_1-\frac{d}{\eta}\right). \tag*{(24)} t 1 k = 1 ∑ t u k ≤ η d + η t 1 − ( 1 − η ) t ( u 1 − η d ) . (24)
Proof. Conditional maximal coupling and the same-state kernel bound give
Pr ( A k + 1 = 1 ∣ F k ) ≤ d ( 1 − A k ) + ( 1 − ρ ) A k . \Pr(A_{k+1}=1\mid\mathcal{F}_k)\leq d(1-A_k)+(1-\rho)A_k. Pr ( A k + 1 = 1 ∣ F k ) ≤ d ( 1 − A k ) + ( 1 − ρ ) A k .
Indeed, on A k = 0 A_k=0 A k = 0 both kernels start at the same state, whereas on A k = 1 A_k=1 A k = 1 the cross-state overlap is at least ρ \rho ρ . Taking expectations yields the scalar recursion
u k + 1 ≤ d + ( 1 − ρ − d ) u k . (25) u_{k+1}\leq d+(1-\rho-d)u_k. \tag*{(25)} u k + 1 ≤ d + ( 1 − ρ − d ) u k . (25)
Let r = 1 − η ∈ ( 0 , 1 ) r=1-\eta\in(0,1) r = 1 − η ∈ ( 0 , 1 ) and q ∗ = d / η q_*=d/\eta q ∗ = d / η , so d + r q ∗ = q ∗ d+rq_*=q_* d + r q ∗ = q ∗ . Subtracting q ∗ q_* q ∗ and iterating the inequality, using r ≥ 0 r\geq0 r ≥ 0 , gives
u k − q ∗ ≤ r k − 1 ( u 1 − q ∗ ) . u_k-q_*\leq r^{k-1}(u_1-q_*). u k − q ∗ ≤ r k − 1 ( u 1 − q ∗ ) .
This remains valid when u 1 < q ∗ u_1<q_* u 1 < q ∗ . Summing for k = 1 , … , t k=1,\ldots,t k = 1 , … , t and using ∑ j = 0 t − 1 r j = ( 1 − r t ) / η \sum_{j=0}^{t-1}r^j=(1-r^t)/\eta ∑ j = 0 t − 1 r j = ( 1 − r t ) / η proves Equation (24) .
This recursion gives the expectation bound associated with the nearby-chain analysis of Johndrow and Mattingly [21 Theorem 9] , without a Poisson-equation construction. Its implication is
lim sup t → ∞ 1 t ∑ k = 1 t E [ A k ] ≤ d ρ + d . \limsup_{t\to\infty}\frac{1}{t}\sum_{k=1}^{t}\mathbb{E}[A_k]\leq\frac{d}{\rho+d}. t → ∞ lim sup t 1 k = 1 ∑ t E [ A k ] ≤ ρ + d d .
The correction in the displayed upper bound has magnitude at most ∣ u 1 − q ∗ ∣ / ( η t ) |u_1-q_*|/(\eta t) ∣ u 1 − q ∗ ∣/ ( η t ) , hence is O ( t − 1 ) O(t^{-1}) O ( t − 1 ) for fixed ( d , ρ ) (d,\rho) ( d , ρ ) . This does not assert convergence of the actual average to q ∗ q_* q ∗ , or monotone decay: the upper bound increases toward q ∗ q_* q ∗ if u 1 < q ∗ u_1<q_* u 1 < q ∗ . The first-agreement survival bound is proved separately in Theorem B.7 ; it is not a consequence of the Cesaro bound.
Perturbed Markov chain bound
Assumption B.3 (One-step Doeblin condition for the behavior kernel). For the fixed behavior kernel, there exists a β ∈ ( 0 , 1 ) a_\beta\in(0,1) a β ∈ ( 0 , 1 ) such that
δ ( P β ) : = sup s , s ′ ∥ P β ( s , ⋅ ) − P β ( s ′ , ⋅ ) ∥ T V ≤ 1 − a β . (26) \delta(P^\beta):=\sup_{s,s'}\lVert P^\beta(s,\cdot)-P^\beta(s',\cdot)\rVert_{\mathrm{TV}}\leq1-a_\beta. \tag*{(26)} δ ( P β ) := s , s ′ sup ∥ P β ( s , ⋅ ) − P β ( s ′ , ⋅ ) ∥ TV ≤ 1 − a β . (26)
This is the one-step Doeblin condition in Johndrow and Mattingly [21 Assumption 1] . It implies
∥ ν P β − ν ′ P β ∥ T V ≤ ( 1 − a β ) ∥ ν − ν ′ ∥ T V \lVert\nu P^\beta-\nu'P^\beta\rVert_{\mathrm{TV}} \le(1-a_\beta)\lVert\nu-\nu'\rVert_{\mathrm{TV}} ∥ ν P β − ν ′ P β ∥ TV ≤ ( 1 − a β ) ∥ ν − ν ′ ∥ TV
for probability laws ν , ν ′ \nu,\nu' ν , ν ′ . It is stronger than merely specifying an exponential envelope with arbitrary C e r g C_{\mathrm{erg}} C erg as in Appendix § 8.1 ; a β a_\beta a β is not identified with that envelope’s α e r g \alpha_{\mathrm{erg}} α erg . The assumption is local to the perturbation results below and is not an additional hypothesis of Theorem 4.1 .
Lemma B.4 (Stationary perturbation bound). Under Assumptions A.2 and B.3, let μ β \mu_\beta μ β and μ π \mu_\pi μ π be stationary laws of P β P^\beta P β and P π P^\pi P π , respectively. Then
∥ μ β − μ π ∥ T V ≤ min { 1 , d / a β } . \lVert\mu_\beta-\mu_\pi\rVert_{\mathrm{TV}} \le\min\{1,d/a_\beta\}. ∥ μ β − μ π ∥ TV ≤ min { 1 , d / a β } .
Proof. Stationarity, the triangle inequality, and the one-step contraction give
∥ μ β − μ π ∥ T V ≤ ∥ ( μ β − μ π ) P β ∥ T V + ∥ μ π ( P β − P π ) ∥ T V ≤ ( 1 − a β ) ∥ μ β − μ π ∥ T V + d . \lVert\mu_\beta-\mu_\pi\rVert_{\mathrm{TV}} \le\lVert(\mu_\beta-\mu_\pi)P^\beta\rVert_{\mathrm{TV}}+\lVert\mu_\pi(P^\beta-P^\pi)\rVert_{\mathrm{TV}}
\le(1-a_\beta)\lVert\mu_\beta-\mu_\pi\rVert_{\mathrm{TV}}+d. ∥ μ β − μ π ∥ TV ≤ ∥( μ β − μ π ) P β ∥ TV + ∥ μ π ( P β − P π ) ∥ TV ≤ ( 1 − a β ) ∥ μ β − μ π ∥ TV + d .
Rearrange and use that TV is at most one. No mixing rate for π \pi π is needed for this stationary comparison, provided its stationary law exists.
Lemma B.5 (Perturbed Markov chain TV bound). Under the hypotheses of Lemma B.4 , define b π ( t ) = sup s ∥ ( P π ) t ( s , ⋅ ) − μ π ∥ T V b_\pi(t)=\sup_s\lVert(P^\pi)^t(s,\cdot)-\mu_\pi\rVert_{\mathrm{TV}} b π ( t ) = sup s ∥( P π ) t ( s , ⋅ ) − μ π ∥ TV . For every initial state s s s and integer t ≥ 0 t\ge0 t ≥ 0 ,
∥ ( P β ) t ( s , ⋅ ) − μ β ∥ T V ≤ ∥ ( P β ) t ( s , ⋅ ) − ( P π ) t ( s , ⋅ ) ∥ T V + b π ( t ) + min { 1 , d / a β } . (27) \lVert(P^\beta)^t(s,\cdot)-\mu_\beta\rVert_{\mathrm{TV}}\le\lVert(P^\beta)^t(s,\cdot)-(P^\pi)^t(s,\cdot)\rVert_{\mathrm{TV}}+b_\pi(t)+\min\{1,d/a_\beta\}. \tag*{(27)} ∥( P β ) t ( s , ⋅ ) − μ β ∥ TV ≤ ∥( P β ) t ( s , ⋅ ) − ( P π ) t ( s , ⋅ ) ∥ TV + b π ( t ) + min { 1 , d / a β } . (27) If, additionally, π \pi π satisfies its own uniform-ergodicity bound b π ( t ) ≤ C π ( 1 − a π ) t b_\pi(t)\le C_\pi(1-a_\pi)^t b π ( t ) ≤ C π ( 1 − a π ) t , this envelope may replace b π ( t ) b_\pi(t) b π ( t ) on the right.
Proof. Insert ( P π ) t ( s , ⋅ ) (P^\pi)^t(s,\cdot) ( P π ) t ( s , ⋅ ) and μ π \mu_\pi μ π between ( P β ) t ( s , ⋅ ) (P^\beta)^t(s,\cdot) ( P β ) t ( s , ⋅ ) and μ β \mu_\beta μ β , apply the triangle inequality, and use Lemma B.4 .
The target-chain mixing term is explicit: behavior-chain ergodicity alone does not supply the same rate for π \pi π . For example, if d < a β / 2 d<a_\beta/2 d < a β /2 , the triangle inequality implies
δ ( P π ) ≤ δ ( P β ) + 2 d ≤ 1 − ( a β − 2 d ) , \delta(P^\pi)\le\delta(P^\beta)+2d\le1-(a_\beta-2d), δ ( P π ) ≤ δ ( P β ) + 2 d ≤ 1 − ( a β − 2 d ) ,
so one may take C π = 1 C_\pi=1 C π = 1 and a π = a β − 2 d a_\pi=a_\beta-2d a π = a β − 2 d . Alternatively, the all-policy cross-Doeblin condition of Assumption A.3 , applied separately to ( β , β ) (\beta,\beta) ( β , β ) and ( π , π ) (\pi,\pi) ( π , π ) , supplies one-step constants a β = a π = ρ a_\beta=a_\pi=\rho a β = a π = ρ when that stronger condition is imposed.
For a coupling of the two different kernels, the first term in Equation (27) is at most Pr ( S t β ≠ S t π ) \Pr(S^\beta_t\ne S^\pi_t) Pr ( S t β = S t π ) . This is a time-t t t disagreement probability, not generally bounded above by Pr ( T β , π > t ) \Pr(T_{\beta,\pi}>t) Pr ( T β , π > t ) : the chains can separate after their first agreement. Thus this perturbation decomposition does not by itself turn the MINT stopping time into a behavior-chain mixing-time bound.
Behavior-chain mixing and off-policy truncation: two couplings
Behavior–behavior coupling for estimating τ m i x β \tau^\beta_{\mathrm{mix}} τ mix β . Let ( X t , Y t ) (X_t,Y_t) ( X t , Y t ) be a co-adapted coupling of two copies of P β P^\beta P β , started at arbitrary states x , y x,y x , y . Choose the coupling to be coalescent: once the states agree, use the same transition for both copies thereafter. Its coalescence time is
τ c β : = inf { t ≥ 0 : X t = Y t } . \tau^\beta_c:=\inf\{t\ge0:X_t=Y_t\}. τ c β := inf { t ≥ 0 : X t = Y t } .
Unlike an off-policy coupling, both marginals here have the same kernel. Fix such a coupling for each initial pair and set m β = sup x , y E x , y [ τ c β ] m_\beta=\sup_{x,y}\mathbb{E}_{x,y}[\tau^\beta_c] m β = sup x , y E x , y [ τ c β ] .
Proposition B.6 (Behavior-chain mixing bound). Suppose P β P^\beta P β has stationary law μ β \mu_\beta μ β . For the coalescent couplings above and every integer t ≥ 0 t\ge0 t ≥ 0 ,
sup x ∥ ( P β ) t ( x , ⋅ ) − μ β ∥ T V ≤ sup x , y Pr ( τ c β > t ) . (28) \sup_x\lVert(P^\beta)^t(x,\cdot)-\mu_\beta\rVert_{\mathrm{TV}}\le\sup_{x,y}\Pr(\tau^\beta_c>t). \tag*{(28)} x sup ∥( P β ) t ( x , ⋅ ) − μ β ∥ TV ≤ x , y sup Pr ( τ c β > t ) . (28) If m β < ∞ m_\beta<\infty m β < ∞ , then
τ m i x β ≤ ⌈ 4 m β ⌉ . (29) \tau^\beta_{\mathrm{mix}}\le\lceil4m_\beta\rceil. \tag*{(29)} τ mix β ≤ ⌈ 4 m β ⌉ . (29)
Proof. Coalescence implies { X t ≠ Y t } = { τ c β > t } \{X_t \ne Y_t\} = \{\tau^\beta_c > t\} { X t = Y t } = { τ c β > t } . The coupling inequality bounds ∥ ( P β ) t ( x , ⋅ ) − ( P β ) t ( y , ⋅ ) ∥ T V \lVert(\mathcal{P}^\beta)^t(x,\cdot)-(\mathcal{P}^\beta)^t(y,\cdot)\rVert_{\mathrm{TV}} ∥( P β ) t ( x , ⋅ ) − ( P β ) t ( y , ⋅ ) ∥ TV by this tail. Integrate the second initial state against μ β \mu_\beta μ β and use stationarity to obtain Equation (28) . For t ≥ 1 t \ge1 t ≥ 1 , Markov’s inequality bounds its right-hand side by m β / t m_\beta/t m β / t . Taking t = ⌈ 4 m β ⌉ t = \lceil4m_\beta\rceil t = ⌈ 4 m β ⌉ proves Equation (29) ; if m β = 0 m_\beta= 0 m β = 0 , all initial states already agree and the claim is immediate.
This is the same-kernel coupling route to mixing-time bounds [26 Section 5.2] . It preserves the purpose of using two chains to bound the behavior policy’s mixing time. It is an analytic upper bound; estimating the supremum or its tail from trajectories requires additional statistical control, and is not supplied by an observed MINT stopping time.
Behavior–target coupling for the MINT survival horizon. Return to the distinct-kernel processes of Definition 3.1, with T β , π = inf { t ≥ 1 : S t β = S t π } T_{\beta,\pi} = \inf\{t \ge1 : S^\beta_t = S^\pi_t\} T β , π = inf { t ≥ 1 : S t β = S t π } and starting pair x = ( s 0 , a 0 ) x = (s_0,a_0) x = ( s 0 , a 0 ) . Here the cumulative mask is killed at first agreement, while the state processes may later separate. This stopping time controls the state-coupled trace and is distinct from τ c β \tau^\beta_c τ c β .
Theorem B.7 (Expected state-coupled truncation horizon). Suppose the conditional meeting probability at each transition after time 1, on survival, is at least ρ > 0 \rho> 0 ρ > 0 ; conditional maximal coupling under Assumption A.3 ensures this. Write u 1 , x = Pr x ( T β , π > 1 ) u_{1,x} = \Pr_x(T_{\beta,\pi} > 1) u 1 , x = Pr x ( T β , π > 1 ) . Then, for every integer t ≥ 1 t \ge1 t ≥ 1 ,
Pr x ( T β , π > t ) ≤ u 1 , x ( 1 − ρ ) t − 1 , E x [ T β , π ] ≤ 1 + u 1 , x ρ . (30) \Pr_x(T_{\beta,\pi} > t) \le u_{1,x}(1-\rho)^{t-1}, \qquad\mathbb{E}_x[T_{\beta,\pi}] \le1+\frac{u_{1,x}}{\rho}. \tag*{(30)} x Pr ( T β , π > t ) ≤ u 1 , x ( 1 − ρ ) t − 1 , E x [ T β , π ] ≤ 1 + ρ u 1 , x . (30) If the initial conditional coupling also satisfies u 1 , x ≤ 1 − ρ u_{1,x} \le1-\rho u 1 , x ≤ 1 − ρ , these simplify to
Pr x ( T β , π > t ) ≤ ( 1 − ρ ) t ( t ≥ 0 ) , E x [ T β , π ] ≤ ρ − 1 . \Pr_x(T_{\beta,\pi} > t) \le(1-\rho)^t \quad(t \ge0), \qquad\mathbb{E}_x[T_{\beta,\pi}] \le\rho^{-1}. x Pr ( T β , π > t ) ≤ ( 1 − ρ ) t ( t ≥ 0 ) , E x [ T β , π ] ≤ ρ − 1 . For a deterministic integer cap L ≥ 1 L \ge1 L ≥ 1 , define K s t a t e = min { L , T β , π } K_{\mathrm{state}} = \min\{L,T_{\beta,\pi}\} K state = min { L , T β , π } . Under this additional initial-step condition,
E x [ K s t a t e ] ≤ min { L , ρ − 1 } , M x ( γ , λ ) ≤ 1 1 − γ λ ( 1 − ρ ) . \mathbb{E}_x[K_{\mathrm{state}}] \le\min\{L,\rho^{-1}\}, \qquad M_x(\gamma,\lambda) \le\frac{1}{1-\gamma\lambda(1-\rho)}. E x [ K state ] ≤ min { L , ρ − 1 } , M x ( γ , λ ) ≤ 1 − γ λ ( 1 − ρ ) 1 .
Proof. For t ≥ 1 t \ge1 t ≥ 1 , { T β , π > t } \{T_{\beta,\pi} > t\} { T β , π > t } is measurable with respect to the joint history through time t t t . The conditional meeting bound gives
Pr x ( T β , π > t + 1 ) = E x [ 1 { T β , π > t } Pr ( A t + 1 = 1 ∣ F t ) ] ≤ ( 1 − ρ ) Pr x ( T β , π > t ) . \Pr_x(T_{\beta,\pi} > t+1) = \mathbb{E}_x\left[1\{T_{\beta,\pi} > t\}\Pr(A_{t+1}=1 \mid\mathcal{F}_t)\right] \le(1-\rho)\Pr_x(T_{\beta,\pi} > t). x Pr ( T β , π > t + 1 ) = E x [ 1 { T β , π > t } Pr ( A t + 1 = 1 ∣ F t ) ] ≤ ( 1 − ρ ) x Pr ( T β , π > t ) .
Iterate from time 1. The tail-sum identity gives E x [ T β , π ] = ∑ t ≥ 0 Pr x ( T β , π > t ) ≤ 1 + u 1 , x / ρ \mathbb{E}_x[T_{\beta,\pi}] = \sum_{t\ge0}\Pr_x(T_{\beta,\pi} > t) \le1+u_{1,x}/\rho E x [ T β , π ] = ∑ t ≥ 0 Pr x ( T β , π > t ) ≤ 1 + u 1 , x / ρ . If the initial-step condition holds, the geometric tail starts at time 0. Since K s t a t e ≤ L K_{\mathrm{state}} \le L K state ≤ L and K s t a t e ≤ T β , π K_{\mathrm{state}} \le T_{\beta,\pi} K state ≤ T β , π , its expectation satisfies the stated cap bound. Finally, sum the discounted tail to bound M x M_x M x .
The initial-step condition matters when conditioning on a prescribed action a 0 a_0 a 0 : overlap of policy-averaged kernels alone does not by itself imply overlap of that conditional first transition. Without it, the same argument still gives
E x [ K s t a t e ] ≤ min { L , 1 + u 1 , x / ρ } , M x ( γ , λ ) ≤ 1 + γ λ u 1 , x 1 − γ λ ( 1 − ρ ) . \mathbb{E}_x[K_{\mathrm{state}}] \le\min\{L,1+u_{1,x}/\rho\}, \qquad M_x(\gamma,\lambda) \le1+\frac{\gamma\lambda u_{1,x}}{1-\gamma\lambda(1-\rho)}. E x [ K state ] ≤ min { L , 1 + u 1 , x / ρ } , M x ( γ , λ ) ≤ 1 + 1 − γ λ ( 1 − ρ ) γ λ u 1 , x .
These bounds concern the state coupling. They do not bound the implemented action-level surrogate’s stopping time, whose survival law and contraction are analyzed separately in Appendix § 8.2 .8.
Why the two times cannot be substituted. Even when β = π \beta=\pi β = π , a common-state synchronous coupling has T β , π = 1 T_{\beta,\pi}=1 T β , π = 1 while the behavior chain can mix slowly. For example, with
P β = P π = ( 1 − ε ε ε 1 − ε ) , 0 < ε < 1 2 , \mathcal{P}^\beta=\mathcal{P}^\pi=
\begin{pmatrix}
1-\varepsilon& \varepsilon\\
\varepsilon& 1-\varepsilon
\end{pmatrix},
\qquad0<\varepsilon<\frac{1}{2}, P β = P π = ( 1 − ε ε ε 1 − ε ) , 0 < ε < 2 1 ,
the stationary law is uniform and ∥ ( P β ) t ( 0 , ⋅ ) − μ β ∥ T V = 1 2 ( 1 − 2 ε ) t \lVert(\mathcal{P}^\beta)^t(0,\cdot)-\mu_\beta\rVert_{\mathrm{TV}}=\frac{1}{2}(1-2\varepsilon)^t ∥( P β ) t ( 0 , ⋅ ) − μ β ∥ TV = 2 1 ( 1 − 2 ε ) t . Thus τ m i x β \tau^\beta_{\mathrm{mix}} τ mix β grows without bound as ε ↓ 0 \varepsilon\downarrow0 ε ↓ 0 although this same-start first agreement remains 1. The valid mixing bound uses the worst-initial-pair β \beta β –β \beta β coalescence time in Proposition B.6 ; the MINT horizon uses the β \beta β –π \pi π survival time in Theorem B.7 .
Online Control with Markov Traces
This subsection distinguishes Markovianity of a coupled pair from a Markov trace on the original state–action space. The online result below concerns the latter class of expected operators. It is a conditional control result, separate from the state-coupled evaluation theorem in Theorem 4.1 .
State coupling on an augmented space. For fixed β \beta β , π \pi π , let Z t = ( S t β , S t π ) Z_t = (S_t^\beta, S_t^\pi) Z t = ( S t β , S t π ) be a Markov coupling of the policy-induced state kernels, with joint transition kernel K β , π K_{\beta,\pi} K β , π on S × S \mathcal{S} \times\mathcal{S} S × S . Its marginals are P β ( ⋅ ∣ S t β ) \mathcal{P}^\beta(\cdot\mid S_t^\beta) P β ( ⋅ ∣ S t β ) and P π ( ⋅ ∣ S t π ) \mathcal{P}^\pi(\cdot\mid S_t^\pi) P π ( ⋅ ∣ S t π ) ; maximal coupling is one such construction. For z ′ = ( s β ′ , s π ′ ) z' = (s_\beta',s_\pi') z ′ = ( s β ′ , s π ′ ) , define the killed kernel
K ~ β , π ( z , z ′ ) : = K β , π ( z , z ′ ) 1 { s β ′ ≠ s π ′ } . (31) \widetilde{K}_{\beta,\pi}(z,z') := K_{\beta,\pi}(z,z')\mathbb{1}\{s_\beta' \ne s_\pi'\}. \tag*{(31)} K β , π ( z , z ′ ) := K β , π ( z , z ′ ) 1 { s β ′ = s π ′ } . (31)
If T = inf { t ≥ 1 : S t β = S t π } T = \inf\{t \ge1 : S_t^\beta= S_t^\pi\} T = inf { t ≥ 1 : S t β = S t π } , then
Pr z ( T > t ) = ( K ~ β , π t 1 ) ( z ) , \Pr_z(T > t) = (\widetilde{K}_{\beta,\pi}^{t}\mathbf{1})(z), z Pr ( T > t ) = ( K β , π t 1 ) ( z ) ,
∑ t ≥ 0 ( γ λ ) t Pr z ( T > t ) = [ ( I − γ λ K ~ β , π ) − 1 1 ] ( z ) . (32) \sum_{t \ge0}(\gamma\lambda)^t\Pr_z(T > t) = [(I-\gamma\lambda\widetilde{K}_{\beta,\pi})^{-1}\mathbf{1}](z). \tag*{(32)} t ≥ 0 ∑ ( γ λ ) t z Pr ( T > t ) = [( I − γ λ K β , π ) − 1 1 ] ( z ) . (32)
Killing here terminates the trace at the first meeting; it does not require the two original chains to remain together after meeting. Thus the coupled construction can be Markov on the augmented space even when its survival cannot be described by a single keep function on the behavior state–action space. In particular, under maximal coupling the next disagreement probability depends on both coordinates through TV ( P β ( ⋅ ∣ s β ) , P π ( ⋅ ∣ s π ) ) \operatorname{TV}(\mathcal{P}^\beta(\cdot\mid s_\beta),\mathcal{P}^\pi(\cdot\mid s_\pi)) TV ( P β ( ⋅ ∣ s β ) , P π ( ⋅ ∣ s π )) . Removing the target-policy state coordinate does not necessarily preserve a Markov trace representation. An online theorem for the augmented construction and a projected update of Q ( s , a ) Q(s,a) Q ( s , a ) would require an additional operator argument; Equation (32) alone does not establish this extension.
Markov-trace setup for the control result. Let S , A \mathcal{S},\mathcal{A} S , A be finite, rewards be bounded, γ ∈ ( 0 , 1 ) \gamma\in(0,1) γ ∈ ( 0 , 1 ) , and λ ∈ [ 0 , 1 ] \lambda\in[0,1] λ ∈ [ 0 , 1 ] . At control iteration n n n , fix β n \beta_n β n , π n \pi_n π n and a keep function g n : S × A → [ 0 , 1 ] g_n:\mathcal{S}\times\mathcal{A}\to[0,1] g n : S × A → [ 0 , 1 ] . These functions may be selected using Q n Q_n Q n , but are held fixed when applying that iteration’s operator to any input Q Q Q . Given a behavior trajectory including its rewards, draw independent masks B i ( n ) ∼ Bernoulli ( g n ( s i , a i ) ) B_i^{(n)} \sim\operatorname{Bernoulli}(g_n(s_i,a_i)) B i ( n ) ∼ Bernoulli ( g n ( s i , a i )) with independent randomness. Write W t ( n ) = ∏ i = 1 t B i ( n ) W_t^{(n)}=\prod_{i=1}^{t}B_i^{(n)} W t ( n ) = ∏ i = 1 t B i ( n ) and W 0 ( n ) = 1 W_0^{(n)}=1 W 0 ( n ) = 1 . The masks do not inspect the following transition or reward. Define
R n Q ( x ) : = Q ( x ) + E β n , x [ ∑ t ≥ 0 ( γ λ ) t W t ( n ) δ t Q , π n ] , (33) \mathcal{R}_nQ(x):=Q(x)+\mathbb{E}_{\beta_n,x}\left[\sum_{t\ge0}(\gamma\lambda)^tW_t^{(n)}\delta_t^{Q,\pi_n}\right], \tag*{(33)} R n Q ( x ) := Q ( x ) + E β n , x [ t ≥ 0 ∑ ( γ λ ) t W t ( n ) δ t Q , π n ] , (33)
where δ t Q , π n = r t + γ E b ∼ π n ( ⋅ ∣ s t + 1 ) Q ( s t + 1 , b ) − Q ( s t , a t ) \delta_t^{Q,\pi_n}=r_t+\gamma\mathbb{E}_{b\sim\pi_n(\cdot\mid s_{t+1})}Q(s_{t+1},b)-Q(s_t,a_t) δ t Q , π n = r t + γ E b ∼ π n ( ⋅ ∣ s t + 1 ) Q ( s t + 1 , b ) − Q ( s t , a t ) . The index n n n denotes a control iteration, whereas t t t denotes time within its behavior trajectory. This is an uncapped population operator. The implemented action-level rule is the case g n = h n g_n=h_n g n = h n from Equation (45) ; the coupled-state indicators of Theorem 4.1 are not identified with g n g_n g n by notation alone.
Theorem B.8 (Exact control convergence with Markov traces). Under the Markov-trace setup above, consider Q n + 1 = R n Q n Q_{n+1}=\mathcal{R}_nQ_n Q n + 1 = R n Q n . Assume:
(A1) Evaluation fixed point. R n Q π n = Q π n \mathcal{R}_nQ^{\pi_n}=Q^{\pi_n} R n Q π n = Q π n for every n n n . This identity follows from the specified operator, as shown below.
(A2) Monotone policy improvement. T π n + 1 Q n + 1 ≥ T π n Q n + 1 \mathcal{T}^{\pi_{n+1}}Q_{n+1}\ge\mathcal{T}^{\pi_n}Q_{n+1} T π n + 1 Q n + 1 ≥ T π n Q n + 1 componentwise.
(A3) Initialization. T π 0 Q 0 ≥ Q 0 \mathcal{T}^{\pi_0}Q_0\ge Q_0 T π 0 Q 0 ≥ Q 0 and Q 0 ≤ Q ∗ Q_0\le Q^* Q 0 ≤ Q ∗ componentwise.
(A4) Coefficient overlap. λ β n ( a ∣ s ) g n ( s , a ) ≤ π n ( a ∣ s ) \lambda\beta_n(a\mid s)g_n(s,a)\le\pi_n(a\mid s) λ β n ( a ∣ s ) g n ( s , a ) ≤ π n ( a ∣ s ) for every ( s , a ) (s,a) ( s , a ) and every n n n .
(A5) Asymptotic greediness. For the Bellman optimality operator T \mathcal{T} T ,
ε n : = ∥ T Q n − T π n Q n ∥ ∞ ⟶ 0. \varepsilon_n:=\|\mathcal{T}Q_n-\mathcal{T}^{\pi_n}Q_n\|_\infty\longrightarrow0. ε n := ∥ T Q n − T π n Q n ∥ ∞ ⟶ 0. This requirement is separate from (A2).
Then Q n ↑ Q ∗ Q_n \uparrow Q^* Q n ↑ Q ∗ componentwise and ∥ Q n − Q ∗ ∥ ∞ → 0 \|Q_n-Q^*\|_\infty\to0 ∥ Q n − Q ∗ ∥ ∞ → 0 .
Proof. Markov factorization and the evaluation fixed point. For a keep function g g g and behavior policy β \beta β , define
C β , g ( ( s , a ) , ( s ′ , b ) ) : = P ( s ′ ∣ s , a ) β ( b ∣ s ′ ) g ( s ′ , b ) . (34) C_{\beta,g}((s,a),(s',b)) := P(s' \mid s,a)\beta(b \mid s')g(s',b). \tag*{(34)} C β , g (( s , a ) , ( s ′ , b )) := P ( s ′ ∣ s , a ) β ( b ∣ s ′ ) g ( s ′ , b ) . (34)
Write C n = C β n , g n C_n=C_{\beta_n,g_n} C n = C β n , g n . Since C n ≥ 0 C_n \ge0 C n ≥ 0 and C n 1 ≤ 1 C_n\mathbf{1} \le1 C n 1 ≤ 1 , the resolvent
N n : = ( I − γ λ C n ) − 1 = ∑ j ≥ 0 ( γ λ C n ) j \mathcal{N}_n := (I-\gamma\lambda C_n)^{-1} = \sum_{j\ge0}(\gamma\lambda C_n)^j N n := ( I − γ λ C n ) − 1 = j ≥ 0 ∑ ( γ λ C n ) j
exists and satisfies N n ≥ I \mathcal{N}_n \ge I N n ≥ I entrywise and 1 ≤ N n 1 ≤ ( 1 − γ λ ) − 1 1 1 \le\mathcal{N}_n\mathbf{1} \le(1-\gamma\lambda)^{-1}\mathbf{1} 1 ≤ N n 1 ≤ ( 1 − γ λ ) − 1 1 . Conditional mask independence and the Markov property give
E β n , x [ W t ( n ) δ t Q , π n ] = [ C n t ( T π n Q − Q ) ] ( x ) . (35) \mathbb{E}_{\beta_n,x}[W_t^{(n)}\delta_t^{Q,\pi_n}] = [C_n^t(\mathcal{T}^{\pi_n}Q-Q)](x). \tag*{(35)} E β n , x [ W t ( n ) δ t Q , π n ] = [ C n t ( T π n Q − Q )] ( x ) . (35)
The bounded geometric sum permits exchanging expectation and summation, hence
R n Q = Q + N n ( T π n Q − Q ) . (36) \mathcal{R}_nQ=Q+\mathcal{N}_n(\mathcal{T}^{\pi_n}Q-Q). \tag*{(36)} R n Q = Q + N n ( T π n Q − Q ) . (36)
This proves (A1) directly. In fact, invertibility of N n \mathcal{N}_n N n makes Q π n Q^{\pi_n} Q π n the unique fixed point even before (A4) is used.
Matrix order and non-negativity. We use only
N n ( I − γ λ C n ) = ( I − γ λ C n ) N n = I . \mathcal{N}_n(I-\gamma\lambda C_n)=(I-\gamma\lambda C_n)\mathcal{N}_n=I. N n ( I − γ λ C n ) = ( I − γ λ C n ) N n = I .
Thus N n \mathcal{N}_n N n commutes with C n C_n C n . No commutation with P π n \mathcal{P}^{\pi_n} P π n , or between operators from different control iterations, is assumed. The linear part of Equation (36) is
R n Q − R n Q ′ = H n ( Q − Q ′ ) , \mathcal{R}_nQ-\mathcal{R}_nQ'=H_n(Q-Q'), R n Q − R n Q ′ = H n ( Q − Q ′ ) ,
H n : = I + N n ( γ P π n − I ) = γ N n ( P π n − λ C n ) . (37) H_n:=I+\mathcal{N}_n(\gamma\mathcal{P}^{\pi_n}-I)
=\gamma\mathcal{N}_n(\mathcal{P}^{\pi_n}-\lambda C_n). \tag*{(37)} H n := I + N n ( γ P π n − I ) = γ N n ( P π n − λ C n ) . (37)
Indeed, I = N n ( I − γ λ C n ) I=\mathcal{N}_n(I-\gamma\lambda C_n) I = N n ( I − γ λ C n ) gives this identity without moving P π n \mathcal{P}^{\pi_n} P π n across N n \mathcal{N}_n N n . Under (A4),
( P π n − λ C n ) ( ( s , a ) , ( s ′ , b ) ) = P ( s ′ ∣ s , a ) [ π n ( b ∣ s ′ ) − λ β n ( b ∣ s ′ ) g n ( s ′ , b ) ] ≥ 0. (\mathcal{P}^{\pi_n}-\lambda C_n)((s,a),(s',b))=P(s'\mid s,a)[\pi_n(b\mid s')-\lambda\beta_n(b\mid s')g_n(s',b)]\ge0. ( P π n − λ C n ) (( s , a ) , ( s ′ , b )) = P ( s ′ ∣ s , a ) [ π n ( b ∣ s ′ ) − λ β n ( b ∣ s ′ ) g n ( s ′ , b )] ≥ 0.
Consequently H n ≥ 0 H_n\ge0 H n ≥ 0 , so R n \mathcal{R}_n R n preserves componentwise order. Also,
H n 1 = γ N n 1 − γ λ N n C n 1 = 1 − ( 1 − γ ) N n 1 ≤ γ 1. (38) H_n\mathbf{1}=\gamma\mathcal{N}_n\mathbf{1}-\gamma\lambda\mathcal{N}_nC_n\mathbf{1}
= \mathbf{1}-(1-\gamma)\mathcal{N}_n\mathbf{1}\le\gamma\mathbf{1}. \tag*{(38)} H n 1 = γ N n 1 − γ λ N n C n 1 = 1 − ( 1 − γ ) N n 1 ≤ γ 1 . (38)
Therefore
∥ H n ∥ ∞ ≤ γ , (39) \lVert H_n\rVert_\infty\le\gamma, \tag*{(39)} ∥ H n ∥ ∞ ≤ γ , (39)
and the fixed-point identity yields
∥ Q n + 1 − Q π n ∥ ∞ ≤ γ ∥ Q n − Q π n ∥ ∞ . (40) \lVert Q_{n+1}-Q^{\pi_n}\rVert_\infty\le\gamma\lVert Q_n-Q^{\pi_n}\rVert_\infty. \tag*{(40)} ∥ Q n + 1 − Q π n ∥ ∞ ≤ γ ∥ Q n − Q π n ∥ ∞ . (40)
Monotonicity of the iterates. Let b n : = T π n Q n − Q n b_n:=\mathcal{T}^{\pi_n}Q_n-Q_n b n := T π n Q n − Q n . The residual after the update with the same target policy is
T π n Q n + 1 − Q n + 1 = b n + ( γ P π n − I ) N n b n = γ ( P π n − λ C n ) N n b n . \mathcal{T}^{\pi_n}Q_{n+1}-Q_{n+1}=b_n+(\gamma\mathcal{P}^{\pi_n}-I)\mathcal{N}_nb_n
=\gamma(\mathcal{P}^{\pi_n}-\lambda C_n)\mathcal{N}_nb_n. T π n Q n + 1 − Q n + 1 = b n + ( γ P π n − I ) N n b n = γ ( P π n − λ C n ) N n b n .
Here N n − I = γ λ C n N n \mathcal{N}_n-I=\gamma\lambda C_n\mathcal{N}_n N n − I = γ λ C n N n is used with N n \mathcal{N}_n N n remaining on the right. In particular, this expression is not obtained by commuting P π n \mathcal{P}^{\pi_n} P π n and N n \mathcal{N}_n N n . Conditions (A2) and (A4) give
b n + 1 ≥ γ ( P π n − λ C n ) N n b n ≥ 0 (41) b_{n+1}\ge\gamma(\mathcal{P}^{\pi_n}-\lambda C_n)\mathcal{N}_nb_n\ge0 \tag*{(41)} b n + 1 ≥ γ ( P π n − λ C n ) N n b n ≥ 0 (41)
whenever b n ≥ 0 b_n\ge0 b n ≥ 0 . Since b 0 ≥ 0 b_0\ge0 b 0 ≥ 0 by (A3), induction proves b n ≥ 0 b_n\ge0 b n ≥ 0 for every n n n , and
Q n + 1 − Q n = N n b n ≥ 0. (42) Q_{n+1}-Q_n=\mathcal{N}_nb_n\ge0. \tag*{(42)} Q n + 1 − Q n = N n b n ≥ 0. (42)
Upper bound and optimality. Because T π n Q ∗ ≤ T Q ∗ = Q ∗ \mathcal{T}^{\pi_n}Q^*\le\mathcal{T}Q^*=Q^* T π n Q ∗ ≤ T Q ∗ = Q ∗ and N n ≥ 0 \mathcal{N}_n\ge0 N n ≥ 0 , Equation (36) implies R n Q ∗ ≤ Q ∗ \mathcal{R}_nQ^*\le Q^* R n Q ∗ ≤ Q ∗ . Order preservation then gives
Q n ≤ Q ∗ ⟹ Q n + 1 = R n Q n ≤ R n Q ∗ ≤ Q ∗ . Q_n\le Q^* \quad\Longrightarrow\quad Q_{n+1}=\mathcal{R}_nQ_n\le\mathcal{R}_nQ^*\le Q^*. Q n ≤ Q ∗ ⟹ Q n + 1 = R n Q n ≤ R n Q ∗ ≤ Q ∗ .
Together with (A3), this bounds the monotone sequence above by Q ∗ Q^* Q ∗ . Finiteness of S × A \mathcal{S} \times\mathcal{A} S × A gives convergence to some Q ∞ ≤ Q ∗ Q_\infty\le Q^* Q ∞ ≤ Q ∗ in supremum norm, and hence ∥ Q n + 1 − Q n ∥ ∞ → 0 \lVert Q_{n+1} - Q_n \rVert_\infty\to0 ∥ Q n + 1 − Q n ∥ ∞ → 0 . Since N n ≥ I \mathcal{N}_n \ge I N n ≥ I entrywise and b n ≥ 0 b_n \ge0 b n ≥ 0 ,
0 ≤ b n ≤ N n b n = Q n + 1 − Q n , ∥ b n ∥ ∞ → 0. 0 \le b_n \le\mathcal{N}_n b_n = Q_{n+1} - Q_n, \qquad\lVert b_n \rVert_\infty\to0. 0 ≤ b n ≤ N n b n = Q n + 1 − Q n , ∥ b n ∥ ∞ → 0.
Now apply (A5):
∥ T Q n − Q n ∥ ∞ ≤ ε n + ∥ b n ∥ ∞ → 0. (43) \lVert\mathcal{T} Q_n - Q_n \rVert_\infty\le\varepsilon_n + \lVert b_n \rVert_\infty\to0. \tag*{(43)} ∥ T Q n − Q n ∥ ∞ ≤ ε n + ∥ b n ∥ ∞ → 0. (43)
The γ \gamma γ -contraction of T \mathcal{T} T yields
∥ Q n − Q ∗ ∥ ∞ ≤ ∥ T Q n − Q n ∥ ∞ 1 − γ → 0 , \lVert Q_n - Q^* \rVert_\infty\le\frac{\lVert\mathcal{T} Q_n - Q_n \rVert_\infty}{1-\gamma} \to0, ∥ Q n − Q ∗ ∥ ∞ ≤ 1 − γ ∥ T Q n − Q n ∥ ∞ → 0 ,
so Q ∞ = Q ∗ Q_\infty= Q^* Q ∞ = Q ∗ .
Roles of the assumptions and the surrogate specialization. As in Munos et al. [31, Theorem 2], (A2) and (A5) perform distinct tasks: monotone policy improvement preserves the residual sign, whereas asymptotic greediness rules out convergence to a nonoptimal policy value. Choosing every π n \pi_n π n greedy with respect to Q n Q_n Q n satisfies (A5) with ε n = 0 \varepsilon_n = 0 ε n = 0 and also supplies (A2) at the next update. The Markov-trace setup supplies (A1), while (A3) initializes monotonicity. Condition (A4) is a quantitative coefficient-overlap condition, not merely a common-support assumption: it makes both H n H_n H n and the residual propagation matrix in Equation (41) nonnegative.
For g n = h n = 1 − min ( 1 , π n / β n ) g_n = h_n = 1 - \min(1, \pi_n/\beta_n) g n = h n = 1 − min ( 1 , π n / β n ) on the support of β n \beta_n β n , β n h n = [ β n − π n ] + \beta_n h_n = [\beta_n - \pi_n]_+ β n h n = [ β n − π n ] + . Thus (A4) is equivalent to
π n ( a ∣ s ) ≥ λ 1 + λ β n ( a ∣ s ) for all ( s , a ) with β n ( a ∣ s ) > 0. (44) \pi_n(a \mid s) \ge\frac{\lambda}{1+\lambda}\beta_n(a \mid s) \quad\text{for all } (s,a) \text{ with } \beta_n(a \mid s) > 0. \tag*{(44)} π n ( a ∣ s ) ≥ 1 + λ λ β n ( a ∣ s ) for all ( s , a ) with β n ( a ∣ s ) > 0. (44)
Indeed, where β n ≤ π n \beta_n \le\pi_n β n ≤ π n the coefficient constraint is automatic; otherwise it is λ ( β n − π n ) ≤ π n \lambda(\beta_n - \pi_n) \le\pi_n λ ( β n − π n ) ≤ π n , which rearranges to Equation (44) . At λ = 0 \lambda= 0 λ = 0 the condition is automatic. This equivalence concerns the action-level surrogate; it is not a re-expression of a general paired-state survival indicator. Unlike the fixed-policy χ λ \chi_\lambda χ λ condition of Theorem B.9 , (A4) is used here to obtain the order preservation needed for control.
Deep-RL extensions: entropy, KL constraints, and smoothing. Entropy-regularized actors, including SAC [16 ] , and KL-based policy constraints motivate considering policy pairs with substantial overlap. To invoke this theorem for the surrogate, however, a quantitative bound such as inf s , a : β n ( a ∣ s ) > 0 π n ( a ∣ s ) / β n ( a ∣ s ) ≥ λ / ( 1 + λ ) \inf_{s,a:\beta_n(a\mid s)>0} \pi_n(a\mid s)/\beta_n(a\mid s) \ge\lambda/(1+\lambda) inf s , a : β n ( a ∣ s ) > 0 π n ( a ∣ s ) / β n ( a ∣ s ) ≥ λ / ( 1 + λ ) must hold; full support, entropy regularization, or a finite KL bound alone is not used as a proof of (A4). This discussion concerns possible extensions of the operator analysis rather than an automatic guarantee for actor–critic training.
For deterministic-policy methods such as TD3, introducing explicitly specified smoothed policies can make density ratios available. An application to those smoothed policies would still require the coefficient-overlap and policy-improvement hypotheses above. Extending the result to the deterministic limit requires a separate continuity argument and is not established by the present theorem. Likewise, entropy-regularized optimal control requires the corresponding soft Bellman operator and policy-improvement conditions if its optimum is the object of interest.
Scope of the control result. The theorem concerns exact, uncapped, finite-dimensional updates of Equation (33) . It does not assert that arbitrary state-coupled traces admit this factorization on ( s , a ) (s,a) ( s , a ) , or that augmenting the pair state alone establishes the same control result after projection. Finite-cap evaluation retains terminal bootstrap terms as in Theorem B.9 ; a finite-cap control extension requires its own corresponding argument. Sample-based updates, nonlinear critics, and simultaneously learned actors are further extensions. These distinctions preserve the state-coupled evaluation theorem as a separate main result.
Contraction of the Action-Level Surrogate
This subsection analyzes the implemented action-level stopping rule directly, separately from the state-coupled operator of Theorem 4.1 . The behavior policy may be arbitrary; contraction requires the sufficient condition below. No transition-kernel coupling or cross-Doeblin hypothesis is used.
Setup and survival quantities. Let S , A \mathcal{S}, \mathcal{A} S , A be finite, rewards be bounded, and β , π \beta, \pi β , π be fixed policies. Fix γ ∈ ( 0 , 1 ) \gamma\in(0,1) γ ∈ ( 0 , 1 ) , λ ∈ [ 0 , 1 ] \lambda\in[0,1] λ ∈ [ 0 , 1 ] , and an integer hard cap L = K max ≥ 1 L = K_{\max} \ge1 L = K m a x ≥ 1 . Write α = γ λ \alpha= \gamma\lambda α = γ λ and
h ( s , a ) : = 1 − min ( 1 , π ( a ∣ s ) β ( a ∣ s ) ) . (45) h(s,a) := 1-\min\left(1,\frac{\pi(a\mid s)}{\beta(a\mid s)}\right). \tag*{(45)} h ( s , a ) := 1 − min ( 1 , β ( a ∣ s ) π ( a ∣ s ) ) . (45)
The ratio is needed only where β ( a ∣ s ) > 0 \beta(a\mid s)>0 β ( a ∣ s ) > 0 ; set h = 0 h=0 h = 0 otherwise. Thus β ( a ∣ s ) h ( s , a ) = [ β ( a ∣ s ) − π ( a ∣ s ) ] + \beta(a\mid s)h(s,a)=[\beta(a\mid s)-\pi(a\mid s)]_{+} β ( a ∣ s ) h ( s , a ) = [ β ( a ∣ s ) − π ( a ∣ s ) ] + everywhere. Given a behavior trajectory, including its rewards, draw independent masks A ^ i ∼ Bernoulli ( h ( s i , a i ) ) \widehat{A}_i\sim\operatorname{Bernoulli}(h(s_i,a_i)) A i ∼ Bernoulli ( h ( s i , a i )) using independent randomness. In particular, the mask at i i i does not inspect the next transition or reward. Define W t = ∏ i = 1 t A ^ i W_t=\prod_{i=1}^{t}\widehat{A}_i W t = ∏ i = 1 t A i , W 0 = 1 W_0=1 W 0 = 1 , and T ^ = inf { i ≥ 1 : A ^ i = 0 } \widehat{T}=\inf\{i\ge1:\widehat{A}_i=0\} T = inf { i ≥ 1 : A i = 0 } , with inf ∅ = ∞ \inf\varnothing=\infty inf ∅ = ∞ . The expected surrogate operator is
R λ , L s u r Q ( x ) : = Q ( x ) + E x [ ∑ t = 0 L − 1 α t W t δ t Q ] , x = ( s 0 , a 0 ) . (46) \mathcal{R}^{\mathrm{sur}}_{\lambda,L}Q(x):=Q(x)+\mathbb{E}_x\left[\sum_{t=0}^{L-1}\alpha^tW_t\delta_t^Q\right],\qquad x=(s_0,a_0). \tag*{(46)} R λ , L sur Q ( x ) := Q ( x ) + E x [ t = 0 ∑ L − 1 α t W t δ t Q ] , x = ( s 0 , a 0 ) . (46)
The cap in this definition is exactly the implementation’s min ( L , T ^ ) \min(L,\widehat{T}) min ( L , T ) ; it is not an uncapped approximation.
Let D s = TV ( β ( ⋅ ∣ s ) , π ( ⋅ ∣ s ) ) D_s=\operatorname{TV}(\beta(\cdot\mid s),\pi(\cdot\mid s)) D s = TV ( β ( ⋅ ∣ s ) , π ( ⋅ ∣ s )) . Define the negative-coefficient mass and its normalized fraction by
e λ ( s ) : = ∑ a [ λ β ( a ∣ s ) h ( s , a ) − π ( a ∣ s ) ] + = ∑ a [ λ β ( a ∣ s ) − ( 1 + λ ) π ( a ∣ s ) ] + . (47) e_\lambda(s):=\sum_a[\lambda\beta(a\mid s)h(s,a)-\pi(a\mid s)]_+
=\sum_a[\lambda\beta(a\mid s)-(1+\lambda)\pi(a\mid s)]_+. \tag*{(47)} e λ ( s ) := a ∑ [ λ β ( a ∣ s ) h ( s , a ) − π ( a ∣ s ) ] + = a ∑ [ λ β ( a ∣ s ) − ( 1 + λ ) π ( a ∣ s ) ] + . (47)
χ λ ( s ) : = { e λ ( s ) / ( λ D s ) , λ D s > 0 , 0 , λ D s = 0. \chi_\lambda(s):=
\begin{cases}
e_\lambda(s)/(\lambda D_s), & \lambda D_s>0,\\
0, & \lambda D_s=0.
\end{cases} χ λ ( s ) := { e λ ( s ) / ( λ D s ) , 0 , λ D s > 0 , λ D s = 0.
Since 0 ≤ e λ ( s ) ≤ λ D s 0\le e_\lambda(s)\le\lambda D_s 0 ≤ e λ ( s ) ≤ λ D s , we have χ λ ( s ) ∈ [ 0 , 1 ] \chi_\lambda(s)\in[0,1] χ λ ( s ) ∈ [ 0 , 1 ] . This fraction measures the negative-weight correction left by the action-dependent keep rule; it is generally different from the policy TV D s D_s D s .
For every starting pair x x x , define
M ^ L x : = ∑ t = 0 L − 1 α t Pr x ( T ^ > t ) , (48) \widehat{M}_L^x:=\sum_{t=0}^{L-1}\alpha^t\Pr_x(\widehat{T}>t), \tag*{(48)} M L x := t = 0 ∑ L − 1 α t x Pr ( T > t ) , (48)
χ ‾ λ , L x : = ∑ t = 1 L − 1 α t E x [ W t χ λ ( s t ) ] M ^ L x − 1 . \overline{\chi}_{\lambda,L}^x:=
\frac{\sum_{t=1}^{L-1}\alpha^t\mathbb{E}_x[W_t\chi_\lambda(s_t)]}{\widehat{M}_L^x-1}. χ λ , L x := M L x − 1 ∑ t = 1 L − 1 α t E x [ W t χ λ ( s t )] .
where the second quantity is 0 if its denominator vanishes. All survival probabilities here belong to the surrogate’s own stopping process. In particular, neither M ^ L x \widehat{M}_L^x M L x nor χ ‾ λ , L x \overline{\chi}_{\lambda,L}^x χ λ , L x is inferred by substituting p ^ i \widehat{p}_i p i into the state-level coupling analysis.
Theorem B.9 (Finite-cap surrogate contraction). Under the setup above, R λ , L s u r \mathcal{R}^{\mathrm{sur}}_{\lambda,L} R λ , L sur has the unique fixed point Q π Q^\pi Q π . Moreover, for all Q , Q ′ Q,Q' Q , Q ′ , its supremum-norm Lipschitz modulus is bounded by
κ χ , L : = sup x { 1 − ( 1 − γ ) M ^ L x + 2 χ ‾ λ , L x ( M ^ L x − 1 ) } . (49) \kappa_{\chi,L}:=\sup_x\left\{1-(1-\gamma)\widehat{M}_L^x+2\overline{\chi}_{\lambda,L}^x(\widehat{M}_L^x-1)\right\}. \tag*{(49)} κ χ , L := x sup { 1 − ( 1 − γ ) M L x + 2 χ λ , L x ( M L x − 1 ) } . (49) Consequently, if κ χ , L < 1 \kappa_{\chi,L}<1 κ χ , L < 1 , the exact iteration Q n + 1 = R λ , L s u r Q n Q_{n+1}=\mathcal{R}^{\mathrm{sur}}_{\lambda,L}Q_n Q n + 1 = R λ , L sur Q n satisfies
∥ Q n − Q π ∥ ∞ ≤ κ χ , L n ∥ Q 0 − Q π ∥ ∞ . \lVert Q_n-Q^\pi\rVert_\infty\le\kappa_{\chi,L}^n\lVert Q_0-Q^\pi\rVert_\infty. ∥ Q n − Q π ∥ ∞ ≤ κ χ , L n ∥ Q 0 − Q π ∥ ∞ . The condition κ χ , L < 1 \kappa_{\chi,L}<1 κ χ , L < 1 is sufficient, not necessary.
Proof. Expected operator and fixed point. On state–action pairs define the nonnegative matrix
C ( ( s , a ) , ( s ′ , b ) ) : = P ( s ′ ∣ s , a ) β ( b ∣ s ′ ) h ( s ′ , b ) , S L : = ∑ j = 0 L − 1 ( α C ) j , S 0 : = 0. (50) C((s,a),(s',b)):=\mathcal{P}(s'\mid s,a)\beta(b\mid s')h(s',b),\qquad S_L:=\sum_{j=0}^{L-1}(\alpha C)^j,\qquad S_0:=0. \tag*{(50)} C (( s , a ) , ( s ′ , b )) := P ( s ′ ∣ s , a ) β ( b ∣ s ′ ) h ( s ′ , b ) , S L := j = 0 ∑ L − 1 ( α C ) j , S 0 := 0. (50)
Because C 1 ≤ 1 C\mathbf{1}\le1 C 1 ≤ 1 and α < 1 \alpha<1 α < 1 , ∥ α C ∥ ∞ < 1 \lVert\alpha C\rVert_\infty<1 ∥ α C ∥ ∞ < 1 . Conditional mask independence gives E [ W t ∣ trajectory ] = ∏ i = 1 t h ( s i , a i ) \mathbb{E}[W_t\mid\text{trajectory}]=\prod_{i=1}^{t}h(s_i,a_i) E [ W t ∣ trajectory ] = ∏ i = 1 t h ( s i , a i ) . The Markov property and the conditional mean of the TD residual therefore yield
E x [ W t δ t Q ] = ( C t ( T π Q − Q ) ) ( x ) , R λ , L s u r Q = Q + S L ( T π Q − Q ) . \mathbb{E}_x[W_t\delta_t^Q]=(C^t(\mathcal{T}^\pi Q-Q))(x),\qquad
\mathcal{R}^{\mathrm{sur}}_{\lambda,L}Q=Q+S_L(\mathcal{T}^\pi Q-Q). E x [ W t δ t Q ] = ( C t ( T π Q − Q )) ( x ) , R λ , L sur Q = Q + S L ( T π Q − Q ) .
The factorization S L = ( I − ( α C ) L ) ( I − α C ) − 1 S_L = (I - (\alpha C)^L)(I - \alpha C)^{-1} S L = ( I − ( α C ) L ) ( I − α C ) − 1 shows that S L S_L S L is invertible. Hence R λ , L s u r Q = Q \mathcal{R}^{\mathrm{sur}}_{\lambda,L}Q = Q R λ , L sur Q = Q if and only if T π Q = Q \mathcal{T}^{\pi}Q = Q T π Q = Q , proving uniqueness of the fixed point Q π Q^\pi Q π .
Finite-cap boundary. The linear part of the affine operator is
H L : = I + S L ( γ P π − I ) = γ S L − 1 ( P π − λ C ) + γ ( α C ) L − 1 P π . (51) H_L := I + S_L(\gamma P^\pi- I)
= \gamma S_{L-1}(P^\pi- \lambda C) + \gamma(\alpha C)^{L-1}P^\pi. \tag*{(51)} H L := I + S L ( γ P π − I ) = γ S L − 1 ( P π − λ C ) + γ ( α C ) L − 1 P π . (51)
This follows by expanding the finite sums; no commutativity between C C C and P π P^\pi P π is used. The last term is the terminal bootstrap contribution, and S L − 1 S_{L-1} S L − 1 , rather than S L S_L S L , multiplies the signed interior contribution. Also,
S L 1 = ( M ^ L x ) x , H L 1 = 1 − ( 1 − γ ) S L 1. (52) S_L\mathbf{1} = (\widehat{M}^{x}_L)_x,\qquad H_L\mathbf{1} = 1 - (1-\gamma)S_L\mathbf{1}. \tag*{(52)} S L 1 = ( M L x ) x , H L 1 = 1 − ( 1 − γ ) S L 1 . (52)
Negative mass. Let E λ : = ( λ C − P π ) + E_\lambda:= (\lambda C - P^\pi)_+ E λ := ( λ C − P π ) + , with positive part and absolute values taken entrywise. Then
∣ P π − λ C ∣ = P π − λ C + 2 E λ . \lvert P^\pi- \lambda C\rvert= P^\pi- \lambda C + 2E_\lambda. ∣ P π − λ C ∣ = P π − λ C + 2 E λ .
All other factors in Equation (51) are nonnegative, so
∣ H L ∣ 1 ≤ 1 − ( 1 − γ ) S L 1 + 2 γ S L − 1 E λ 1. (53) \lvert H_L\rvert\mathbf{1} \le1 - (1-\gamma)S_L\mathbf{1} + 2\gamma S_{L-1}E_\lambda\mathbf{1}. \tag*{(53)} ∣ H L ∣ 1 ≤ 1 − ( 1 − γ ) S L 1 + 2 γ S L − 1 E λ 1 . (53)
The state-level mass of E λ E_\lambda E λ is exactly e λ e_\lambda e λ from Equation (47) . If χ ~ ( s , a ) : = χ λ ( s ) \widetilde{\chi}(s,a) := \chi_\lambda(s) χ ( s , a ) := χ λ ( s ) , then
( E λ 1 ) ( s , a ) = ∑ s ′ P ( s ′ ∣ s , a ) e λ ( s ′ ) = λ ( C χ ~ ) ( s , a ) , (E_\lambda\mathbf{1})(s,a) = \sum_{s'}P(s'\mid s,a)e_\lambda(s') = \lambda(C\widetilde{\chi})(s,a), ( E λ 1 ) ( s , a ) = s ′ ∑ P ( s ′ ∣ s , a ) e λ ( s ′ ) = λ ( C χ ) ( s , a ) ,
where ∑ b β ( b ∣ s ′ ) h ( s ′ , b ) = D s ′ \sum_b\beta(b\mid s')h(s',b)=D_{s'} ∑ b β ( b ∣ s ′ ) h ( s ′ , b ) = D s ′ was used. Therefore
γ ( S L − 1 E λ 1 ) ( x ) = ∑ t = 1 L − 1 α t ( C t χ ~ ) ( x ) = ∑ t = 1 L − 1 α t E x [ W t χ λ ( s t ) ] = χ ‾ λ , L ( M ^ L x − 1 ) . \begin{aligned}
\gamma(S_{L-1}E_\lambda\mathbf{1})(x) = \sum_{t=1}^{L-1}\alpha^t(C^t\widetilde{\chi})(x) \\
&= \sum_{t=1}^{L-1}\alpha^t\mathbb{E}_x[W_t\chi_\lambda(s_t)] \\
&= \overline{\chi}_{\lambda,L}(\widehat{M}^{x}_L-1).
\end{aligned} γ ( S L − 1 E λ 1 ) ( x ) = t = 1 ∑ L − 1 α t ( C t χ ) ( x ) = t = 1 ∑ L − 1 α t E x [ W t χ λ ( s t )] = χ λ , L ( M L x − 1 ) .
Substituting into Equation (53) and taking the maximum row absolute sum proves ∥ H L ∥ ∞ ≤ κ χ , L \lVert H_L\rVert_\infty\le\kappa_{\chi,L} ∥ H L ∥ ∞ ≤ κ χ , L . Since R λ , L s u r Q − R λ , L s u r Q ′ = H L ( Q − Q ′ ) \mathcal{R}^{\mathrm{sur}}_{\lambda,L}Q-\mathcal{R}^{\mathrm{sur}}_{\lambda,L}Q'=H_L(Q-Q') R λ , L sur Q − R λ , L sur Q ′ = H L ( Q − Q ′ ) , contraction and the stated geometric convergence follow when κ χ , L < 1 \kappa_{\chi,L}<1 κ χ , L < 1 . □
Corollary B.10 (A sufficient overlap condition). If π ( a ∣ s ) ≥ λ β ( a ∣ s ) / ( 1 + λ ) \pi(a\mid s)\ge\lambda\beta(a\mid s)/(1+\lambda) π ( a ∣ s ) ≥ λ β ( a ∣ s ) / ( 1 + λ ) for all ( s , a ) (s,a) ( s , a ) , then R λ , L s u r \mathcal{R}^{\mathrm{sur}}_{\lambda,L} R λ , L sur is a γ \gamma γ -contraction for every L ≥ 1 L\ge1 L ≥ 1 .
Proof. The condition is equivalent to λ β h ≤ π \lambda\beta h\le\pi λ β h ≤ π , so e λ = 0 e_\lambda=0 e λ = 0 and E λ = 0 E_\lambda=0 E λ = 0 . Equation (51) then gives H L ≥ 0 H_L\ge0 H L ≥ 0 . Since S L 1 ≥ 1 S_L\mathbf{1}\ge1 S L 1 ≥ 1 , Equation (52) gives ∥ H L ∥ ∞ ≤ γ \lVert H_L\rVert_\infty\le\gamma ∥ H L ∥ ∞ ≤ γ . This is the pointwise overlap condition appearing in Equation (44) , used here solely for fixed-policy evaluation. □
Boundary cases and scope. At L = 1 L=1 L = 1 or λ = 0 \lambda=0 λ = 0 the surrogate operator reduces to T π \mathcal{T}^{\pi} T π . For an infinite cap, the same proof uses S = ( I − α C ) − 1 S=(I-\alpha C)^{-1} S = ( I − α C ) − 1 in place of both S L S_L S L and S L − 1 S_{L-1} S L − 1 , and the survival sums extend to infinity. A bound at or above 1 is inconclusive: exact finite-dimensional iteration converges from every initial value if and only if ρ ( H L ) < 1 \rho(H_L)<1 ρ ( H L ) < 1 , so the sufficient norm condition ∥ H L ∥ ∞ < 1 \lVert H_L\rVert_\infty<1 ∥ H L ∥ ∞ < 1 is not necessary. These are population results for fixed policies. Finite-sample concentration, sample-based stochastic approximation, changing policies, and nonlinear critic projection require additional analysis. In particular, averaging the complete action-dependent backup in a minibatch does not replace h ( s , a ) h(s,a) h ( s , a ) by its state mean D s D_s D s inside that backup.
Why a contraction condition is needed. Consider two states 0, 1 and two deterministic actions: stay preserves the state and flip switches it. Let β \beta β choose each action with probability 1 / 2 1/2 1/2 , let π \pi π always flip, and set rewards to zero. Then h ( s , s t a y ) = 1 h(s,\mathrm{stay})=1 h ( s , stay ) = 1 and h ( s , f l i p ) = 0 h(s,\mathrm{flip})=0 h ( s , flip ) = 0 . For v ( 0 ) = 1 , v ( 1 ) = − 1 v(0)=1,v(1)=-1 v ( 0 ) = 1 , v ( 1 ) = − 1 , define Q 0 ( s , a ) = v ( f ( s , a ) ) Q_0(s,a)=v(f(s,a)) Q 0 ( s , a ) = v ( f ( s , a )) , where f f f is the deterministic next-state map. We have C Q 0 = Q 0 / 2 CQ_0=Q_0/2 C Q 0 = Q 0 /2 and P π Q 0 = − Q 0 P^\pi Q_0=-Q_0 P π Q 0 = − Q 0 , hence
H L Q 0 = μ L Q 0 , μ L = 1 − ( 1 + γ ) ∑ t = 0 L − 1 ( γ λ / 2 ) t . H_LQ_0=\mu_LQ_0,\qquad\mu_L=1-(1+\gamma)\sum_{t=0}^{L-1}(\gamma\lambda/2)^t. H L Q 0 = μ L Q 0 , μ L = 1 − ( 1 + γ ) t = 0 ∑ L − 1 ( γ λ /2 ) t .
At γ = 0.99 \gamma= 0.99 γ = 0.99 , λ = 0.7 \lambda= 0.7 λ = 0.7 , L = 5 L = 5 L = 5 , this eigenvalue is μ L ≈ − 2.02993 \mu_L \approx-2.02993 μ L ≈ − 2.02993 , so exact expected iteration can diverge even with a finite cap. This example does not satisfy the stated sufficient contraction condition; it explains why that condition cannot be omitted. It makes no claim about small-step stochastic updates or the empirical SAC/TD3 training runs.
A Common Conservative TV Condition
Corollary B.11 (A common sufficient condition). Let D = sup s TV ( β ( ⋅ ∣ s ) , π ( ⋅ ∣ s ) ) D = \sup_s \operatorname{TV}(\beta(\cdot\mid s), \pi(\cdot\mid s)) D = sup s TV ( β ( ⋅ ∣ s ) , π ( ⋅ ∣ s )) . Under the respective fixed-policy hypotheses of Theorems B.9 and 4.1 , the condition
γ ( 1 + 2 λ D ) < 1 (54) \gamma(1 + 2\lambda D) < 1 \tag*{(54)} γ ( 1 + 2 λ D ) < 1 (54) ensures contraction of each expected evaluation operator toward Q π Q^\pi Q π ; for the surrogate this holds for every finite cap L ≥ 1 L \ge1 L ≥ 1 .
Proof. Set ε 0 = 1 − γ − 2 γ λ D > 0 \varepsilon_0 = 1 - \gamma- 2\gamma\lambda D > 0 ε 0 = 1 − γ − 2 γ λ D > 0 . For the state-coupled operator, N x ≤ D ( M x − 1 ) N_x \le D(M_x - 1) N x ≤ D ( M x − 1 ) and M x − 1 ≤ γ λ M x M_x - 1 \le\gamma\lambda M_x M x − 1 ≤ γ λ M x give
1 − ( 1 − γ ) M x + 2 N x ≤ 1 − ε 0 M x ≤ 1 − ε 0 < 1 1 - (1 - \gamma)M_x + 2N_x \le1 - \varepsilon_0 M_x \le1 - \varepsilon_0 < 1 1 − ( 1 − γ ) M x + 2 N x ≤ 1 − ε 0 M x ≤ 1 − ε 0 < 1 uniformly over x x x . For the surrogate, e λ ( s ) ≤ λ D s ≤ λ D e_\lambda(s) \le\lambda D_s \le\lambda D e λ ( s ) ≤ λ D s ≤ λ D , so E λ 1 ≤ λ D 1 E_\lambda\mathbf{1} \le\lambda D\mathbf{1} E λ 1 ≤ λ D 1 . Since S L − 1 1 ≤ S L 1 S_{L-1}\mathbf{1} \le S_L\mathbf{1} S L − 1 1 ≤ S L 1 , Equation (53) gives
∣ H L ∣ 1 ≤ 1 − ε 0 S L 1 ≤ ( 1 − ε 0 ) 1. \lvert H_L\rvert\mathbf{1} \le1 - \varepsilon_0 S_L\mathbf{1} \le(1 - \varepsilon_0)\mathbf{1}. ∣ H L ∣ 1 ≤ 1 − ε 0 S L 1 ≤ ( 1 − ε 0 ) 1 . The separate fixed-point results complete both claims.
For λ > 0 \lambda> 0 λ > 0 , Equation (54) requires D < ( 1 − γ ) / ( 2 γ λ ) D < (1 - \gamma)/(2\gamma\lambda) D < ( 1 − γ ) / ( 2 γ λ ) . At the main MuJoCo setting ( γ , λ ) = ( 0.99 , 0.7 ) (\gamma, \lambda) = (0.99, 0.7) ( γ , λ ) = ( 0.99 , 0.7 ) , this is D < 0.00722 D < 0.00722 D < 0.00722 uniformly over states. It is a restrictive shared baseline, not a certificate for the experimental regime. The survival-dependent state condition and the surrogate’s negative-mass condition can be less restrictive; their relative restrictiveness depends on their stopping distributions and actionwise policy ratios. The common condition neither identifies the two stopping processes nor transfers one theorem to the other operator.
Extended Theoretical Discussion
Reading the survival and overlap diagrams. Figure 3 shows the general mismatch–horizon budget at γ = 0.9 \gamma= 0.9 γ = 0.9 , without a cross-Doeblin assumption. Figure 9 , placed with the geometric corollary in Appendix § 8.2 .§ 3 , instead shows the overlap specialization at γ = 0.99 \gamma= 0.99 γ = 0.99 . The two diagrams describe sufficient conditions for the state-coupled operator, not measured deep-RL operating points.
Four Regimes of Theorem 1
For the scalar examples below, suppose the survival law is the same for all starting pairs and write M = M x M = M_x M = M x . The uniform modulus reduces to κ = 1 − ( 1 − γ ) M + 2 D ‾ w g t ( M − 1 ) \kappa= 1 - (1 - \gamma)M + 2\overline{D}_{\mathrm{wgt}}(M - 1) κ = 1 − ( 1 − γ ) M + 2 D wgt ( M − 1 ) , and the uniform TV-based sufficient condition Equation (21) reduces to 2 D ‾ w g t ( M − 1 ) / M < 1 − γ 2\overline{D}_{\mathrm{wgt}}(M - 1)/M < 1 - \gamma 2 D wgt ( M − 1 ) / M < 1 − γ . For nonuniform survival laws, the supremum over starting pairs must be retained. The main text discusses Regime 3 (geometric meeting); the other three regimes are recorded here for completeness.
Regime 1 — On-policy limit (D ‾ w g t → 0 \overline{D}_{\mathrm{wgt}} \to0 D wgt → 0 ). For every starting pair, N x ≤ D ‾ w g t ( M x − 1 ) ≤ D ‾ w g t γ / ( 1 − γ ) N_x \le\overline{D}_{\mathrm{wgt}}(M_x - 1) \le\overline{D}_{\mathrm{wgt}}\gamma/(1 - \gamma) N x ≤ D wgt ( M x − 1 ) ≤ D wgt γ / ( 1 − γ ) , so the contraction value is uniformly D ‾ w g t = 0 \overline{D}_{\mathrm{wgt}} = 0 D wgt = 0 for fixed γ \gamma γ , λ \lambda λ . At D ‾ w g t = 0 \overline{D}_{\mathrm{wgt}} = 0 D wgt = 0 , κ x = 1 − ( 1 − γ ) M x ≤ γ \kappa_x = 1 - (1 - \gamma)M_x \le\gamma κ x = 1 − ( 1 − γ ) M x ≤ γ for every x x x . In the on-policy case, M x = 1 M_x = 1 M x = 1 and N x = 0 N_x = 0 N x = 0 for all x x x , recovering κ = γ \kappa= \gamma κ = γ .
Regime 2 — Deterministic meeting at T = 1 T = 1 T = 1 . Pr ( T > 0 ) = 1 \Pr(T > 0) = 1 Pr ( T > 0 ) = 1 , Pr ( T > t ) = 0 \Pr(T > t) = 0 Pr ( T > t ) = 0 for t ≥ 1 t \ge1 t ≥ 1 , so M = 1 M = 1 M = 1 and κ ≤ γ \kappa\le\gamma κ ≤ γ . One-step Bellman is recovered for any D ‾ w g t \overline{D}_{\mathrm{wgt}} D wgt , reflecting that with immediate meeting there is no off-policy correction to apply.
Regime 3 — Geometric meeting time, T ∼ Geom ( q ) T \sim\operatorname{Geom}(q) T ∼ Geom ( q ) . Pr ( T > t ) = ( 1 − q ) t \Pr(T > t) = (1-q)^t Pr ( T > t ) = ( 1 − q ) t , so M = 1 / ( 1 − γ λ ( 1 − q ) ) M = 1/(1-\gamma\lambda(1-q)) M = 1/ ( 1 − γ λ ( 1 − q )) and ( M − 1 ) / M = γ λ ( 1 − q ) (M-1)/M = \gamma\lambda(1-q) ( M − 1 ) / M = γ λ ( 1 − q ) . Substituting into Equation (21) gives the closed-form sufficient condition
γ ( 1 + 2 D ˉ w g t λ ( 1 − q ) ) < 1. \gamma(1+2\bar{D}_{\mathrm{wgt}}\lambda(1-q))<1. γ ( 1 + 2 D ˉ wgt λ ( 1 − q )) < 1.
The sufficient-condition region widens as the per-step meeting probability q q q grows: at q = 1 q=1 q = 1 we recover Regime 2; at q = 0 q=0 q = 0 we recover Regime 4 below. Here the survival law is exactly geometric. In Corollary B.1, the cross-Doeblin parameter ρ \rho ρ instead supplies a geometric upper bound on the survival tail; equality is not required.
Regime 4 — Worst case (T = ∞ T=\infty T = ∞ ). The two coupled trajectories never meet. Pr ( T > t ) = 1 \Pr(T>t)=1 Pr ( T > t ) = 1 for all t t t , so M = 1 / ( 1 − γ λ ) M=1/(1-\gamma\lambda) M = 1/ ( 1 − γ λ ) and ( M − 1 ) / M = γ λ (M-1)/M=\gamma\lambda ( M − 1 ) / M = γ λ . The condition becomes
γ ( 1 + 2 D ˉ w g t λ ) < 1. \gamma(1+2\bar{D}_{\mathrm{wgt}}\lambda)<1. γ ( 1 + 2 D ˉ wgt λ ) < 1.
At D ˉ w g t = 1 , λ = 1 \bar{D}_{\mathrm{wgt}}=1,\lambda=1 D ˉ wgt = 1 , λ = 1 this gives γ < 1 / 3 \gamma<1/3 γ < 1/3 , the most pessimistic case. But at D ˉ w g t = 0.1 , λ = 1 \bar{D}_{\mathrm{wgt}}=0.1,\lambda=1 D ˉ wgt = 0.1 , λ = 1 it admits γ < 1 / 1.2 ≈ 0.833 \gamma<1/1.2\approx0.833 γ < 1/1.2 ≈ 0.833 ; at D ˉ w g t = 0.01 \bar{D}_{\mathrm{wgt}}=0.01 D ˉ wgt = 0.01 it admits γ < 0.98 \gamma<0.98 γ < 0.98 . Even when the trajectories never meet, reducing D ˉ w g t \bar{D}_{\mathrm{wgt}} D ˉ wgt allows the sufficient condition to hold at larger discount factors.
Practical Regime Corollaries
Theorem 4.1 holds for any γ ∈ ( 0 , 1 ) \gamma\in(0,1) γ ∈ ( 0 , 1 ) , but the worst-case quantitative bound becomes vacuous when γ \gamma γ is close to 1 unless D ˉ w g t \bar{D}_{\mathrm{wgt}} D ˉ wgt is correspondingly small. In this subsection we specialize to the moderate-discount regime γ ≤ 0.9 \gamma\leq0.9 γ ≤ 0.9 and give two explicit sufficient conditions: a worst-case (ℓ ∞ \ell_\infty ℓ ∞ ) form and an averaged (L 1 L^1 L 1 ) form. Both follow from the pointwise bound of Theorem 4.1 . The averaged result is an L ∞ → L 1 L^\infty\to L^1 L ∞ → L 1 one-step error bound; the averaging step requires neither cross-Doeblin nor ergodicity and is not an iteration-convergence argument.
Untruncated TD-error reference. Setting A i ≡ 1 A_i\equiv1 A i ≡ 1 in Equation (4) gives Q + ( I − γ λ P β ) − 1 ( T π Q − Q ) Q+(I-\gamma\lambda P^\beta)^{-1}(\mathcal{T}^\pi Q-Q) Q + ( I − γ λ P β ) − 1 ( T π Q − Q ) , the Q π ( λ ) Q^\pi(\lambda) Q π ( λ ) evaluation operator of Harutyunyan et al. [19 ] . It retains Q π Q^\pi Q π as its fixed point. This is distinct from Peng’s mixture-of-returns operator in Table 1 . The no-survival reference M x = ( 1 − γ λ ) − 1 M_x=(1-\gamma\lambda)^{-1} M x = ( 1 − γ λ ) − 1 gives the sufficient bound γ ( 1 + 2 D ˉ w g t λ ) < 1 \gamma(1+2\bar{D}_{\mathrm{wgt}}\lambda)<1 γ ( 1 + 2 D ˉ wgt λ ) < 1 from the current analysis. Comparisons with this expression concern sufficient bounds; they do not identify a convergence region for Peng’s algorithm.
Corollary C.1 (Practical worst-case contraction at moderate discount). Adopt the operator setup of Theorem 4.1 . Let K λ = sup x min ( E x [ T β , π ] , ( 1 − γ λ ) − 1 ) K_\lambda=\sup_x\min(\mathbb{E}_x[T_{\beta,\pi}],(1-\gamma\lambda)^{-1}) K λ = sup x min ( E x [ T β , π ] , ( 1 − γ λ ) − 1 ) be the uniform effective-horizon bound in Equation (22) . Fix γ ∈ ( 0 , 1 ) \gamma\in(0,1) γ ∈ ( 0 , 1 ) .
(A) Free-trace regime. If D ˉ w g t ≤ 1 − γ 2 \bar{D}_{\mathrm{wgt}}\leq\frac{1-\gamma}{2} D ˉ wgt ≤ 2 1 − γ , then κ < 1 \kappa<1 κ < 1 for every K λ ≥ 1 K_\lambda\geq1 K λ ≥ 1 (equivalently, every meeting-time distribution and every λ ∈ [ 0 , 1 ] \lambda\in[0,1] λ ∈ [ 0 , 1 ] ).
(B) Bounded-trace regime. If D ˉ w g t > 1 − γ 2 \bar{D}_{\mathrm{wgt}}>\frac{1-\gamma}{2} D ˉ wgt > 2 1 − γ , then κ < 1 \kappa<1 κ < 1 whenever
K λ < 2 D ˉ w g t 2 D ˉ w g t − ( 1 − γ ) = : K λ max ( γ , D ˉ w g t ) . (55) K_\lambda<\frac{2\bar{D}_{\mathrm{wgt}}}{2\bar{D}_{\mathrm{wgt}}-(1-\gamma)}=:K_\lambda^{\max}(\gamma,\bar{D}_{\mathrm{wgt}}). \tag*{(55)} K λ < 2 D ˉ wgt − ( 1 − γ ) 2 D ˉ wgt =: K λ m a x ( γ , D ˉ wgt ) . (55) In particular, choosing λ < λ ∗ ( γ , D ˉ w g t ) : = 1 − γ 2 γ D ˉ w g t \lambda<\lambda^*(\gamma,\bar{D}_{\mathrm{wgt}}):=\frac{1-\gamma}{2\gamma\bar{D}_{\mathrm{wgt}}} λ < λ ∗ ( γ , D ˉ wgt ) := 2 γ D ˉ wgt 1 − γ forces 1 1 − γ λ < K λ max \frac{1}{1-\gamma\lambda}<K_\lambda^{\max} 1 − γ λ 1 < K λ m a x and therefore guarantees contraction for every meeting-time distribution.
Proof. For every x x x , Equation (18) gives κ x ≤ 1 − ( 1 − γ ) M x + 2 D ˉ w g t ( M x − 1 ) \kappa_x\leq1-(1-\gamma)M_x+2\bar{D}_{\mathrm{wgt}}(M_x-1) κ x ≤ 1 − ( 1 − γ ) M x + 2 D ˉ wgt ( M x − 1 ) . In Regime (A), the coefficient 2 D ˉ w g t − ( 1 − γ ) 2\bar{D}_{\mathrm{wgt}}-(1-\gamma) 2 D ˉ wgt − ( 1 − γ ) of M x M_x M x is non-positive, so M x ≥ 1 M_x\geq1 M x ≥ 1 yields κ x ≤ γ \kappa_x\leq\gamma κ x ≤ γ uniformly. In Regime (B), this coefficient is positive, so M x ≤ K λ M_x\leq K_\lambda M x ≤ K λ yields a uniform upper bound strictly below 1 when K λ < K λ max K_\lambda<K_\lambda^{\max} K λ < K λ m a x . The λ ∗ \lambda^* λ ∗ statement follows from ( 1 − γ λ ∗ ) − 1 = K λ max (1-\gamma\lambda^*)^{-1}=K_\lambda^{\max} ( 1 − γ λ ∗ ) − 1 = K λ m a x . □
Remark C.2 (Specialization to γ = 0.9 \gamma=0.9 γ = 0.9 ). At γ = 0.9 \gamma=0.9 γ = 0.9 the free-trace threshold is D ˉ w g t ≤ 0.05 \bar{D}_{\mathrm{wgt}}\leq0.05 D ˉ wgt ≤ 0.05 . In the bounded-trace regime, D ˉ w g t = 0.10 \bar{D}_{\mathrm{wgt}}=0.10 D ˉ wgt = 0.10 requires K λ < 2 K_\lambda<2 K λ < 2 , while D ˉ w g t = 0.50 \bar{D}_{\mathrm{wgt}}=0.50 D ˉ wgt = 0.50 requires K λ < 1.111 … K_\lambda<1.111\ldots K λ < 1.111 … . These are fixed-policy state-coupled guarantees. In the free-trace regime, no additional meeting-time restriction is needed; this does not certify the action-level surrogate or deep-RL training.
Corollary C.3 (Averaged one-step error bound). Adopt the state-coupled operator setup and the pointwise Lipschitz bound of Theorem 4.1 , without requiring its uniform contraction condition. Fix a probability distribution ν \nu ν over starting pairs x = ( s , a ) x=(s,a) x = ( s , a ) , and write
M ˉ : = E x ∼ ν [ M x ] , ∥ f ∥ L 1 ( ν ) : = E x ∼ ν [ ∣ f ( x ) ∣ ] . \bar{M} := \mathbb{E}_{x\sim\nu}[M_x], \qquad\|f\|_{L^1(\nu)} := \mathbb{E}_{x\sim\nu}[|f(x)|]. M ˉ := E x ∼ ν [ M x ] , ∥ f ∥ L 1 ( ν ) := E x ∼ ν [ ∣ f ( x ) ∣ ] . Then for every bounded Q Q Q ,
∥ R p , λ Q − Q π ∥ L 1 ( ν ) ≤ κ ⋆ ∥ Q − Q π ∥ ∞ , (56) \|R_{p,\lambda}Q-Q^{\pi}\|_{L^1(\nu)} \le\kappa^{\star}\|Q-Q^{\pi}\|_{\infty}, \tag*{(56)} ∥ R p , λ Q − Q π ∥ L 1 ( ν ) ≤ κ ⋆ ∥ Q − Q π ∥ ∞ , (56) where
κ ⋆ : = 1 − ( 1 − γ ) M ˉ + 2 D ˉ w g t ( M ˉ − 1 ) . (57) \kappa^{\star} := 1-(1-\gamma)\bar{M}+2\bar{D}_{\mathrm{wgt}}(\bar{M}-1). \tag*{(57)} κ ⋆ := 1 − ( 1 − γ ) M ˉ + 2 D ˉ wgt ( M ˉ − 1 ) . (57) The coefficient is below one in either of the following regimes:
If D ˉ w g t ≤ ( 1 − γ ) / 2 \bar{D}_{\mathrm{wgt}} \le(1-\gamma)/2 D ˉ wgt ≤ ( 1 − γ ) /2 , then κ ⋆ ≤ γ < 1 \kappa^{\star}\le\gamma<1 κ ⋆ ≤ γ < 1 for every admissible M ˉ ≥ 1 \bar{M}\ge1 M ˉ ≥ 1 .
If D ˉ w g t > ( 1 − γ ) / 2 \bar{D}_{\mathrm{wgt}} > (1-\gamma)/2 D ˉ wgt > ( 1 − γ ) /2 , then κ ⋆ < 1 \kappa^{\star}<1 κ ⋆ < 1 whenever M ˉ < 2 D ˉ w g t / [ 2 D ˉ w g t − ( 1 − γ ) ] \bar{M}<2\bar{D}_{\mathrm{wgt}}/[2\bar{D}_{\mathrm{wgt}}-(1-\gamma)] M ˉ < 2 D ˉ wgt / [ 2 D ˉ wgt − ( 1 − γ )] .
No cross-Doeblin or mixing assumption is needed for this averaging step. If the behavior chain has a stationary law μ β \mu_{\beta} μ β , one may choose ν = μ β ⊗ β \nu=\mu_{\beta}\otimes\beta ν = μ β ⊗ β as a special case.
Proof. The fixed-point identity and the pointwise bound of Theorem 4.1 give
∣ R p , λ Q ( x ) − Q π ( x ) ∣ ≤ [ 1 − ( 1 − γ ) M x + 2 D ˉ w g t ( M x − 1 ) ] ∥ Q − Q π ∥ ∞ . |R_{p,\lambda}Q(x)-Q^{\pi}(x)|\le[1-(1-\gamma)M_x+2\bar{D}_{\mathrm{wgt}}(M_x-1)]\|Q-Q^{\pi}\|_{\infty}. ∣ R p , λ Q ( x ) − Q π ( x ) ∣ ≤ [ 1 − ( 1 − γ ) M x + 2 D ˉ wgt ( M x − 1 )] ∥ Q − Q π ∥ ∞ .
Integrating with respect to ν \nu ν gives Equation (56) . The two regimes follow by considering the sign of 2 D ˉ w g t − ( 1 − γ ) 2\bar{D}_{\mathrm{wgt}}-(1-\gamma) 2 D ˉ wgt − ( 1 − γ ) and using M ˉ ≥ 1 \bar{M}\ge1 M ˉ ≥ 1 .
This is an L ∞ → L 1 ( ν ) L^{\infty}\to L^1(\nu) L ∞ → L 1 ( ν ) one-step error bound. Even if κ ⋆ < 1 \kappa^{\star}<1 κ ⋆ < 1 , it is not an L 1 ( ν ) L^1(\nu) L 1 ( ν ) contraction, since the right-hand side uses the supremum norm. Consequently it does not by itself prove convergence of repeated updates; the uniform sufficient condition of Theorem 4.1 supplies that separate guarantee.
Relation to the empirical deep-RL settings. The free-trace threshold ( 1 − γ ) / 2 (1-\gamma)/2 ( 1 − γ ) /2 is 0.005 at γ = 0.99 \gamma=0.99 γ = 0.99 . The RWRL runs report mean action-level keep probabilities between 0.70 and 0.99 depending on the arm, but these statistics do not identify the state-coupled survival quantities or certify the χ λ \chi_{\lambda} χ λ condition of Theorem B.9 . The measured κ ^ = 0.84 ± 0.13 \hat{\kappa}=0.84\pm0.13 κ ^ = 0.84 ± 0.13 at K max = 5 K_{\max}=5 K m a x = 5 , below 1 on 90% of probes (Appendix § 8.12 .§ 3 ), is empirical evidence about visited-distribution backup gains, not an estimate of κ ⋆ \kappa^{\star} κ ⋆ or a certificate of κ χ , L < 1 \kappa_{\chi,L}<1 κ χ , L < 1 .
Extended Toy-Chain Validation
Diagnostic study on tabular CliffWalking
Assumptions A.2 and A.3 specify a kernel-level TV bound d d d and uniform cross-policy overlap ρ \rho ρ for the coupling analysis. Our diagnostic study on tabular CliffWalking (Figure 10 (a)) illustrates their relation to transition structure: policy deviations induce near-deterministic branching and absorbing transitions, with state-wise TV distances close to 1 and cross-state overlap near zero. As shown in Figure 10 (b), both the state-wise discrepancies d ( s ) d(s) d ( s ) and the overlap matrix ρ ( s , s ′ ) \rho(s,s') ρ ( s , s ′ ) between an optimal policy and a uniformly random policy exhibit extreme mismatch across most states—reflecting structural properties of the MDP rather than policy proximity. Figure 10 (c) evaluates the RHS of Lemma B.2 at several illustrative ( d , ρ ) (d, \rho) ( d , ρ ) pairs. These curves bound time-averaged expected disagreement under their stated hypotheses; they do not verify a positive uniform ρ \rho ρ for CliffWalking or imply a meeting-time tail. The latter requires the separate conditional overlap assumptions in Theorem B.7 .
Figure 10. Diagnostic study on tabular CliffWalking. (a-top) Environment visualization for CliffWalking. (a-bottom) State-wise total variation distance d ( s ) d(s) d ( s ) between an optimal policy and a uniformly random policy, illustrating pronounced off-policy discrepancies near the cliff region. (b) Pairwise overlap matrix ρ ( s , s ′ ) \rho(s, s') ρ ( s , s ′ ) between transition kernels induced by the two policies; near-zero overlap across most state pairs highlights a strong structural mismatch. (c) Time-averaged expected-disagreement bounds from Lemma B.2 for illustrative ( d , ρ ) (d, \rho) ( d , ρ ) pairs; these curves do not certify a meeting-time bound for the displayed environment.
Single-Trap chain refinements
Main-text second-moment experiment. For Figure 2 , the chain uses N = 20 N = 20 N = 20 , states { 0 , … , N } \{0, \ldots, N\} { 0 , … , N } , reward r = + 1 r = +1 r = + 1 at the trap state N N N , slippery transitions with ξ = 0.1 \xi= 0.1 ξ = 0.1 , and reflecting boundaries. The behavior policy is uniform over { − 1 , + 1 } \{-1, +1\} { − 1 , + 1 } ; the target is identical except π ( + 1 ∣ N ) = 1 \pi(+1 \mid N) = 1 π ( + 1 ∣ N ) = 1 . We evaluate the multi-step Bellman-residual estimator at trace lengths { 1 , 5 , 10 , 20 , 50 } \{1, 5, 10, 20, 50\} { 1 , 5 , 10 , 20 , 50 } with γ = 0.9 \gamma= 0.9 γ = 0.9 and λ = 1 \lambda= 1 λ = 1 . Both axes are logarithmic. Error bars are ± 1 \pm1 ± 1 standard deviation over 30 replicates and are smaller than the plot markers. At t = 50 t = 50 t = 50 , the second moments of Retrace and Peng’s Q ( λ ) Q(\lambda) Q ( λ ) are 1.43 × 1.43\times 1.43 × and 2.28 × 2.28\times 2.28 × that of MINT; these comparisons concern only the tested trace lengths and this localized-mismatch setup.
Figure 2. Single-Trap chain: empirical second moment. (a) Chain MDP. (b) Second moment of the multi-step Bellman residual Δ ^ \hat{\Delta} Δ ^ over trace lengths at γ = 0.9 \gamma= 0.9 γ = 0.9 , λ = 1 \lambda= 1 λ = 1 (log–log axes). Error bars (±1 standard deviation, 30 replicates) are smaller than the markers. At t = 50 t = 50 t = 50 , Retrace and Peng’s Q ( λ ) Q(\lambda) Q ( λ ) are 1.43× and 2.28× MINT, whose curve levels off over the tested lengths. Setup: Appendix § 8.4 .§ 2 .
We extend the Single-Trap chain validation of the main text with two additional refinements: data-processing gap (refinement (ii)) and cross-Doeblin restart (refinement (iii)). Setup follows the main text: Single-Trap chain with states { 0 , … , N } \{0,\ldots,N\} { 0 , … , N } , reflective boundaries, two actions { − 1 , + 1 } \{-1,+1\} { − 1 , + 1 } , β \beta β uniform, π \pi π identical to β \beta β except π ( + 1 ∣ N ) = 1 \pi(+1\mid N)=1 π ( + 1 ∣ N ) = 1 at the trap state. The per-state action TV is D s = 0 D_s=0 D s = 0 everywhere except D N = 0.5 D_N=0.5 D N = 0.5 , so the worst-case bound is D ˉ = 0.5 \bar{D}=0.5 D ˉ = 0.5 regardless of dynamics.
Refinement (ii): Data-processing gap on slippery transitions
With slippery transitions (intended action with probability 1 − ξ 1-\xi 1 − ξ , a uniform random action otherwise), the action-level TV D ˉ = 0.5 \bar{D}=0.5 D ˉ = 0.5 is unchanged, while the one-step kernel TV at the trap shrinks to 0.25 0.25 0.25 at ξ = 0.5 \xi=0.5 ξ = 0.5 . The plotted kernel-weighted diagnostic drops from 0.11 0.11 0.11 to 0.06 0.06 0.06 , while the policy-TV diagnostic remains around 0.12 0.12 0.12 (Figure 11 ). This illustrates a one-step data-processing gap. Kernel TV cannot replace policy TV in the correction term of Theorem 4.1 : that proof bounds signed coefficients on state–action values before any push-forward through the dynamics. The weighted kernel quantity is a diagnostic, not a contraction certificate. The implemented surrogate has a separate χ λ \chi\lambda χ λ correction (Theorem B.9 ).
Figure 11. Data-processing gap on the slippery Single-Trap chain (N = 50 N=50 N = 50 , γ = 0.9 \gamma=0.9 γ = 0.9 , λ = 1 \lambda=1 λ = 1 , slippery ξ \xi ξ varied). (a) Action-level D s D_s D s (red) is unchanged by slippery dynamics, while kernel-level p s p_s p s (green) shrinks at the trap state. (b) The plotted policy-TV and kernel-TV weighted diagnostics separate with ξ \xi ξ ; the smaller kernel diagnostic is not substituted into the contraction condition. (c) Cross-Doeblin ρ = 0 \rho=0 ρ = 0 for the chain MDP without restart. (d) The left-hand side of the sufficient condition in Corollary B.1 exceeds 1 throughout; the restart modification introduces ρ > 0 \rho>0 ρ > 0 .
Refinement (iii): Cross-Doeblin restart and the no-survival reference
We add a uniform-restart component (with probability q r s t q_{\mathrm{rst}} q rst , transition is uniform over all states; with probability 1 − q r s t 1-q_{\mathrm{rst}} 1 − q rst , slippery dynamics) to obtain ρ ≈ q r s t \rho\approx q_{\mathrm{rst}} ρ ≈ q rst (Figure 12 (a)). We compare MINT’s contraction LHS γ ( 1 + 2 D ˉ w g t λ ( 1 − ρ ) ) \gamma(1+2\bar{D}_{\mathrm{wgt}}\lambda(1-\rho)) γ ( 1 + 2 D ˉ wgt λ ( 1 − ρ )) against the no-survival reference LHS γ ( 1 + 2 D ˉ w g t λ ) \gamma(1+2\bar{D}_{\mathrm{wgt}}\lambda) γ ( 1 + 2 D ˉ wgt λ ) (the ρ = 0 \rho=0 ρ = 0 case, which is the sufficient bound). Table 6 reports the measured ratio 1 / ( 1 − ρ ) 1/(1-\rho) 1/ ( 1 − ρ ) alongside both LHS values. An illustrative parameter setting is q r s t = 0.20 q_{\mathrm{rst}}=0.20 q rst = 0.20 : MINT’s LHS drops to 0.985 < 1 0.985<1 0.985 < 1 (contraction), while the reference LHS is 1.006 > 1 1.006>1 1.006 > 1 (no contraction guarantee). The empirical region ratio matches the theoretical 1 / ( 1 − ρ ) 1/(1-\rho) 1/ ( 1 − ρ ) to three decimal places across ρ ∈ { 0.1 , 0.3 , 0.5 } \rho\in\{0.1,0.3,0.5\} ρ ∈ { 0.1 , 0.3 , 0.5 } , providing a quantitative comparison of these sufficient bounds.
Table 6. Uniform-restart diagnostic: MINT and the no-survival reference bound on the restart chain (N = 30 N=30 N = 30 , γ = 0.9 \gamma=0.9 γ = 0.9 , λ = 1 \lambda=1 λ = 1 , ξ = 0.1 \xi=0.1 ξ = 0.1 ). The contraction condition is γ ( 1 + 2 D ˉ w g t λ ( 1 − ρ ) ) < 1 \gamma(1+2\bar{D}_{\mathrm{wgt}}\lambda(1-\rho))<1 γ ( 1 + 2 D ˉ wgt λ ( 1 − ρ )) < 1 for MINT and the ρ = 0 \rho=0 ρ = 0 specialization for the reference. At q r s t = 0.20 q_{\mathrm{rst}}=0.20 q rst = 0.20 : MINT contracts (✓) while the reference bound is inconclusive.
Figure 12. Uniform-restart diagnostic: cross-Doeblin overlap and the region satisfying MINT’s sufficient contraction condition on the restart chain (N = 30 N=30 N = 30 , γ = 0.9 \gamma=0.9 γ = 0.9 , ξ = 0.1 \xi=0.1 ξ = 0.1 ). (a) Measured ρ \rho ρ matches restart probability q r s t q_{\mathrm{rst}} q rst exactly. (b) The plotted policy-TV and kernel-TV diagnostics decrease with ρ \rho ρ . Only the policy-TV quantity enters the stated contraction bound. (c) MINT’s contraction LHS (green) drops below 1 at q r s t = 0.20 q_{\mathrm{rst}}=0.20 q rst = 0.20 , while the no-survival reference LHS (red) does not until q r s t = 0.30 q_{\mathrm{rst}}=0.30 q rst = 0.30 . (d) Empirical region ratio matches the theoretical 1 / ( 1 − ρ ) 1/(1-\rho) 1/ ( 1 − ρ ) to three decimal places.
Summary of refinements. The diagnostics have separate roles. Survival weighting gives the plotted policy-TV value D ˉ w g t ≈ 0.1 \bar{D}_{\mathrm{wgt}}\approx0.1 D ˉ wgt ≈ 0.1 , compared with the worst-case action TV D ˉ = 0.5 \bar{D}=0.5 D ˉ = 0.5 . Slippery dynamics give a smaller kernel-TV diagnostic, p ˉ w g t ≈ 0.06 \bar{p}_{\mathrm{wgt}}\approx0.06 p ˉ wgt ≈ 0.06 . For any common nonnegative normalized weights ω s \omega_s ω s , DPI implies ∑ s ω s d ( s ) ≤ ∑ s ω s D s \sum_s\omega_s d(s)\leq\sum_s\omega_sD_s ∑ s ω s d ( s ) ≤ ∑ s ω s D s , but this does not justify replacing D ˉ w g t \bar{D}_{\mathrm{wgt}} D ˉ wgt by p ˉ w g t \bar{p}_{\mathrm{wgt}} p ˉ wgt in Theorem 4.1 . Cross-Doeblin overlap separately changes the geometric-tail sufficient bound by the factor 1 − ρ 1-\rho 1 − ρ ; the displayed threshold ratio 1 / ( 1 − ρ ) 1/(1-\rho) 1/ ( 1 − ρ ) compares those two state-coupled sufficient bounds with the same policy-TV parameter. These observations do not compose into a multiplicative contraction improvement and do not transfer a state-coupled guarantee to the implemented surrogate.
High-Mismatch Ablations on the Single-Trap Chain
We compare stopping rules in a controlled high-mismatch stress test, using the single-trap dynamics but different policy pairs from the localized-mismatch refinements in Figure 11 and Figure 12 . The policies below differ at every state. Results from these settings are not pooled with, or compared in absolute scale to, the localized-mismatch experiments.
Setup and measured quantities. We use N = 20 N = 20 N = 20 , γ = 0.9 \gamma= 0.9 γ = 0.9 , λ = 1 \lambda= 1 λ = 1 , slippery transitions with probability 0.1, and a 50-step evaluation window. The representative setting is
β ( + 1 ∣ s ) = 0.05 , π ( + 1 ∣ s ) = 0.95 , D s = 0.90 for every s . \beta(+1 \mid s) = 0.05,\qquad\pi(+1 \mid s) = 0.95,\qquad D_s = 0.90 \quad\text{for every } s. β ( + 1 ∣ s ) = 0.05 , π ( + 1 ∣ s ) = 0.95 , D s = 0.90 for every s .
An additional comparison uses β ( + 1 ∣ s ) = 0.005 \beta(+1 \mid s) = 0.005 β ( + 1 ∣ s ) = 0.005 and π ( + 1 ∣ s ) = 0.995 \pi(+1 \mid s) = 0.995 π ( + 1 ∣ s ) = 0.995 , giving D s = 0.99 D_s = 0.99 D s = 0.99 at every state. Within each setting, all variants use the same pool of behavior trajectories: 20 replicates of 20,000 trajectories. The anchor is always retained, W 0 = 1 W_0 = 1 W 0 = 1 , and W t = ∏ i = 1 t B i W_t = \prod_{i=1}^{t} B_i W t = ∏ i = 1 t B i for continuation indicators B i B_i B i . We report the mean retained horizon and the sampling variance of
Δ ^ t = ∑ u = 0 t − 1 ( γ λ ) u W u δ u Q π \widehat{\Delta}_t = \sum_{u=0}^{t-1}(\gamma\lambda)^u W_u \delta_u^{Q^\pi} Δ t = u = 0 ∑ t − 1 ( γ λ ) u W u δ u Q π
at t ∈ { 1 , 5 , 10 , 20 , 50 } t \in\{1, 5, 10, 20, 50\} t ∈ { 1 , 5 , 10 , 20 , 50 } . The variance plots concern evaluation at Q π Q^\pi Q π ; the separate perturbed-critic experiment below measures sampled backup MSE.
Stopping rules and controls.
Action-level surrogate: draw B i ∼ Bernoulli ( h ( s i , a i ) ) B_i \sim\operatorname{Bernoulli}(h(s_i,a_i)) B i ∼ Bernoulli ( h ( s i , a i )) , where h = 1 − min ( 1 , π / β ) h = 1 - \min(1,\pi/\beta) h = 1 − min ( 1 , π / β ) is the keep probability in Equation (7) . Set T ^ = inf { i ≥ 1 : B i = 0 } \widehat{T} = \inf\{i \ge1 : B_i = 0\} T = inf { i ≥ 1 : B i = 0 } . With no additional cap we retain min ( 50 , T ^ ) \min(50,\widehat{T}) min ( 50 , T ) terms; the capped variant retains min ( 10 , T ^ ) \min(10,\widehat{T}) min ( 10 , T ) terms.
Constant-p p p controls: draw independent B i ∼ Bernoulli ( p ) B_i \sim\operatorname{Bernoulli}(p) B i ∼ Bernoulli ( p ) , with p p p denoting keep probability. The sweep uses p ∈ { 0.1 , 0.5 , 0.9 } p \in\{0.1, 0.5, 0.9\} p ∈ { 0.1 , 0.5 , 0.9 } . Separate rate-matched controls choose p p p so that ∑ u = 0 49 p u \sum_{u=0}^{49}p^u ∑ u = 0 49 p u matches either the surrogate or the state-coupled reference horizon. The two matching targets are distinguished in the legends.
State-coupled reference (“oracle” in the legends): use the first-agreement survival mask from the coupled state processes. This is a reference requiring state-coupling information, not a minimum-MSE estimator or an upper bound on practical performance.
Symmetric log-ratio heuristic (“symmetric TV” in the legends): use
h s y m ( s , a ) = clip ( 1 2 ∣ log π ( a ∣ s ) − log β ( a ∣ s ) ∣ , 0 , 1 ) (58) h_{\mathrm{sym}}(s,a) = \operatorname{clip}\left(\frac{1}{2}\left|\log\pi(a \mid s) - \log\beta(a \mid s)\right|,0,1\right) \tag*{(58)} h sym ( s , a ) = clip ( 2 1 ∣ log π ( a ∣ s ) − log β ( a ∣ s ) ∣ , 0 , 1 ) (58)
as the keep probability. This heuristic is not an exact TV estimator, and the surrogate contraction theorem is not asserted for this alternative rule.
A: Comparison with horizon-matched random truncation. Figure 13 (a,Figure 13 d) includes controls matched separately to the surrogate and to the state-coupled reference. In these state-independent policy-mismatch settings, the surrogate does not show a consistent advantage over its horizon-matched constant-p p p control across variance and backup MSE. Thus these tests do not support a general claim that the surrogate improves on random truncation at a matched horizon. They also do not establish equivalence of the estimators: the surrogate mask depends on the sampled action, whereas the constant-p p p mask is independent of that action. A setting with constant D s D_s D s provides limited evidence about benefits from adapting to statewise variation in policy mismatch.
Figure 13. Representative high-mismatch ablation. D s = 0.90 D_s = 0.90 D s = 0.90 . Policies are β ( + 1 ∣ s ) = 0.05 \beta(+1 \mid s) = 0.05 β ( + 1 ∣ s ) = 0.05 and π ( + 1 ∣ s ) = 0.95 \pi(+1 \mid s) = 0.95 π ( + 1 ∣ s ) = 0.95 at every state. (a) Variance versus mean retained horizon, including random controls matched separately to the surrogate and state-coupled reference. (b) The symmetric log-ratio heuristic saturates and retains all 50 steps; the surrogate has lower variance here. (c) Cap 10 reduces the measured horizon from 10.0 to 6.52 with little change in variance. The inset displays the right-censored stopping time min ( T ^ , 50 ) \min(\hat{T}, 50) min ( T ^ , 50 ) ; the final bin includes T ^ ≥ 50 \hat{T} \ge50 T ^ ≥ 50 , and about 39% of traces reach cap 10. (d) Sampled backup MSE at Q 0 = Q π + ξ Q_0 = Q^\pi+ \xi Q 0 = Q π + ξ , ξ ∼ N ( 0 , I ) \xi\sim N(0, I) ξ ∼ N ( 0 , I ) ; the cap lowers the mean from 13.8 to 9.5. Error bars: (a)–(c), ± 1 \pm1 ± 1 SD across 20 trajectory replicates; (d), bootstrap percentile 95% CI for the mean across 20 perturbation seeds. These MSEs are not expected-operator contraction measurements.
B: Saturation of the symmetric heuristic. For the opposing policies here, 1 2 ∣ log ( π / β ) ∣ = 1 2 log 19 ≈ 1.472 \frac{1}{2}|\log(\pi/\beta)| = \frac{1}{2}\log19 \approx1.472 2 1 ∣ log ( π / β ) ∣ = 2 1 log 19 ≈ 1.472 at D s = 0.90 D_s = 0.90 D s = 0.90 , and 1 2 log 199 ≈ 2.647 \frac{1}{2}\log199 \approx2.647 2 1 log 199 ≈ 2.647 at D s = 0.99 D_s = 0.99 D s = 0.99 , for either action. Hence Equation (58) is identically one: the symmetric heuristic retains the full 50-step window. In the representative setting its variance is approximately 0.160, versus 0.129 for the surrogate, and its backup MSE is higher (Figure 13 (b,Figure 13 d)). The variance curves are closer at D s = 0.99 D_s = 0.99 D s = 0.99 (Figure 14 ), indicating that their separation depends on the policy pair. This comparison diagnoses saturation of this particular heuristic, rather than proving that one clipping form is uniformly superior.
Figure 14. Additional high-mismatch comparison: D s = 0.99 D_s = 0.99 D s = 0.99 . Policies are β ( + 1 ∣ s ) = 0.005 \beta(+1 \mid s) = 0.005 β ( + 1 ∣ s ) = 0.005 and π ( + 1 ∣ s ) = 0.995 \pi(+1 \mid s) = 0.995 π ( + 1 ∣ s ) = 0.995 at every state; panel definitions and error-bar conventions match Figure 13 . Cap 10 reduces the measured surrogate horizon from 39.5 to 9.57 and mean backup MSE from 40.8 to 16.1, while variance changes only from 0.143 to 0.141. About 91% of traces reach cap 10. The inset is right-censored at 50, so the final bin includes T ^ ≥ 50 \hat{T} \ge50 T ^ ≥ 50 . The state-coupled reference has horizon 44.5 and is not a minimum-MSE benchmark. The matched-random controls and variance comparisons do not establish a general surrogate advantage across policy pairs.
C: The cap reduces horizon and sampled backup MSE. For a state-independent pair β ( + 1 ) = b \beta(+1) = b β ( + 1 ) = b , π ( + 1 ) = q > b \pi(+1) = q > b π ( + 1 ) = q > b , the surrogate has
h ( + 1 ) = 0 , h ( − 1 ) = q − b 1 − b , E β [ h ∣ s ] = q − b = D . h(+1) = 0,\qquad h(-1) = \frac{q-b}{1-b},\qquad\mathbb{E}_{\beta}[h \mid s] = q-b = D. h ( + 1 ) = 0 , h ( − 1 ) = 1 − b q − b , E β [ h ∣ s ] = q − b = D .
Conditional on the pre-action history, the mean keep probability is therefore always D D D . Iterating the survival recursion gives
Pr ( T ^ > u ) = D u , E [ min ( L , T ^ ) ] = ∑ u = 0 L − 1 D u = 1 − D L 1 − D . (59) \Pr(\hat{T} > u) = D^{u}, \qquad\mathbb{E}[\min(L,\hat{T})] = \sum_{u=0}^{L-1} D^{u} = \frac{1-D^{L}}{1-D}. \tag*{(59)} Pr ( T ^ > u ) = D u , E [ min ( L , T ^ )] = u = 0 ∑ L − 1 D u = 1 − D 1 − D L . (59)
At D = 0.90 D = 0.90 D = 0.90 , the analytic horizons are 9.948 for L = 50 L = 50 L = 50 and 6.513 for L = 10 L = 10 L = 10 , consistent with the measured 10.0 and 6.52. At D = 0.99 D = 0.99 D = 0.99 , they are 39.499 and 9.562, consistent with 39.5 and 9.57. The probabilities of reaching the cap are Pr ( T ^ ≥ 10 ) = D 9 \Pr(\hat{T} \ge10) = D^{9} Pr ( T ^ ≥ 10 ) = D 9 , namely 38.7% and 91.4%, consistent with the insets. These marginal survival identities do not make the surrogate independent of the trajectory.
For backup MSE, set Q 0 = Q π + ξ Q_{0} = Q^{\pi} + \xi Q 0 = Q π + ξ with ξ ∼ N ( 0 , I ) \xi\sim\mathcal{N}(0,I) ξ ∼ N ( 0 , I ) over state–action pairs. Each of 20 perturbation seeds fixes one ξ \xi ξ shared across the trajectory replicates and variants. We measure
M S E b a c k u p = E [ ( R ^ Q 0 ( s 0 , a 0 ) − Q π ( s 0 , a 0 ) ) 2 ] , (60) \mathrm{MSE}_{\mathrm{backup}} = \mathbb{E}\left[\left(\widehat{R}Q_{0}(s_{0},a_{0}) - Q^{\pi}(s_{0},a_{0})\right)^{2}\right], \tag*{(60)} MSE backup = E [ ( R Q 0 ( s 0 , a 0 ) − Q π ( s 0 , a 0 ) ) 2 ] , (60)
where R ^ Q 0 \widehat{R}Q_{0} R Q 0 is the sampled finite-window TD-error backup. The table reports mean ± \pm ± standard deviation across perturbation seeds. Figure panels (a)–(c) show ± 1 \pm1 ± 1 standard deviation of the variance estimates across 20 trajectory replicates; panel (d) shows bootstrap percentile 95% confidence intervals for the mean across the 20 perturbation seeds. The perturbation scale in panel (d) is σ = 1 \sigma= 1 σ = 1 , distinct from the slippery-transition probability.
Table 7. High-mismatch surrogate and cap comparison at a 50-step evaluation window. Backup MSE is mean ± \pm ± SD over 20 perturbation seeds; variance is measured at Q π Q^{\pi} Q π . No additional cap means that the trace is still limited by the 50-step window.
In the representative D s = 0.90 D_{s} = 0.90 D s = 0.90 case, cap 10 lowers mean backup MSE from 13.8 to 9.5 (31.2%), while variance changes only from 0.129 to 0.128 (about 0.8%). In the D s = 0.99 D_{s} = 0.99 D s = 0.99 comparison, it lowers mean backup MSE from 40.8 to 16.1 (60.5%), while variance changes from 0.143 to 0.141 (about 1.4%). The mean MSE decrease is accompanied by lower MSE in 16 of the 20 reported perturbation seeds in the latter setting, with a median paired capped/uncapped ratio of 0.49. These observations support the cap’s role in controlling the sampled backup under the tested mismatch and perturbation distribution, not a claim of large variance reduction or an optimal choice of cap.
The state-coupled reference remains long: its measured horizons are 40.4 and 44.5, with backup MSE 29.2 ± \pm ± 32.4 and 41.7 ± \pm ± 48.3 at D s = 0.90 D_{s} = 0.90 D s = 0.90 and 0.99, respectively. Thus the cap does not match the surrogate horizon to the reference horizon. The smaller MSE of a much shorter capped backup does not establish an ordering between the state-coupled and surrogate contraction conditions. More aggressive constant-p p p truncation can yield still smaller MSE in this perturbation experiment, as shown in panel (d); the experiment does not measure the credit-assignment benefit of long backups in learned-critic training.
Scope of the MSE comparison. For a fixed Q 0 Q_{0} Q 0 and starting pair, the sampled MSE decomposes as
E [ ( R ^ Q 0 − Q π ) 2 ] = ( R Q 0 − Q π ) 2 + Var ( R ^ Q 0 ) , R Q 0 = E [ R ^ Q 0 ] . \mathbb{E}\left[(\widehat{R}Q_{0} - Q^{\pi})^{2}\right] = (RQ_{0} - Q^{\pi})^{2} + \operatorname{Var}(\widehat{R}Q_{0}), \qquad RQ_{0} = \mathbb{E}[\widehat{R}Q_{0}]. E [ ( R Q 0 − Q π ) 2 ] = ( R Q 0 − Q π ) 2 + Var ( R Q 0 ) , R Q 0 = E [ R Q 0 ] .
It therefore combines expected-backup error with sampling variance. A reduction in this quantity does not prove sup-norm contraction of the expected operator or convergence of repeated updates. These are stress tests of fixed-policy estimators; they do not supply a contraction certificate for the deep-RL runs. These perturbation experiments use σ = 1 \sigma= 1 σ = 1 ; no claim of robustness across perturbation scales is made from these figures.
Additional comparison at D s = 0.99 D_s = 0.99 D s = 0.99 . Figure 14 uses the more extreme opposing policy pair. It produces a much longer surrogate horizon and a larger gap between capped and uncapped backup MSE, while retaining the same qualitative conclusions about the matched-random comparison and cap-induced variance change. The comparisons are within this policy setting; they do not identify the absolute variance or MSE of different policy settings with one another.
Reproducibility. The experiment and plotting entry points are scripts/single_trap_ablation.py and paper_figures/ablation_single_trap.py. The D s = 0.99 D_s = 0.99 D s = 0.99 run is identified by outputs/single_trap_ablation/mismatch_b0.005_q0.995.json; the policy pair stated above identifies the representative D s = 0.90 D_s = 0.90 D s = 0.90 setting. Trajectory-length curves and stopping-time histograms use the same high-mismatch policy pairs and protocols as the corresponding backup comparisons.
Implementation Details
Algorithm 2 MINT multi-step target construction — ordinary-residual form matching Algorithm 1; SAC/TD3 substitutions are specified below.
Require: target policy π \pi π , current critic Q θ Q_\theta Q θ , target critic Q θ ˉ Q_{\bar{\theta}} Q θ ˉ , sampled trajectory ( s 0 , a 0 , r 0 , s 1 , a 1 , r 1 , … , s H ) (s_0, a_0, r_0, s_1, a_1, r_1, \ldots, s_H) ( s 0 , a 0 , r 0 , s 1 , a 1 , r 1 , … , s H ) from replay B \mathcal{B} B together with the stored behavior log-probabilities log β ( a i ∣ s i ) \log\beta(a_i \mid s_i) log β ( a i ∣ s i ) recorded at collection time (so β \beta β is the data-generating policy of that trajectory, not the current actor), discount γ \gamma γ , decay λ ∈ [ 0 , 1 ] \lambda\in[0,1] λ ∈ [ 0 , 1 ] , hard cap K max K_{\max} K m a x (a protocol constant, fixed to 5 in the main continuous-control comparisons).
Ensure: multi-step target y ^ \hat{y} y ^ for ( s 0 , a 0 ) (s_0, a_0) ( s 0 , a 0 ) .
101: [Disagreement estimate] For i = 1 , … , K max − 1 i = 1, \ldots, K_{\max} - 1 i = 1 , … , K m a x − 1 , compute
p ^ i ← 1 − min ( 1 , π ( a i ∣ s i ) β ( a i ∣ s i ) ) \hat{p}_i \leftarrow1 - \min\left(1, \frac{\pi(a_i \mid s_i)}{\beta(a_i \mid s_i)}\right) p ^ i ← 1 − min ( 1 , β ( a i ∣ s i ) π ( a i ∣ s i ) )
(action-level surrogate, Equation (7) ).
102: [Sample stochastic mask] Draw A ^ i ∼ Bernoulli ( p ^ i ) \hat{A}_i \sim\operatorname{Bernoulli}(\hat{p}_i) A ^ i ∼ Bernoulli ( p ^ i ) independently for i = 1 , … , K max − 1 i = 1, \ldots, K_{\max} - 1 i = 1 , … , K m a x − 1 using independent randomness.
103: [Effective horizon] Let T ^ ← inf { 1 ≤ i < K max : A ^ i = 0 } \hat{T} \leftarrow\inf\{1 \leq i < K_{\max} : \hat{A}_i = 0\} T ^ ← inf { 1 ≤ i < K m a x : A ^ i = 0 } , with inf ∅ = ∞ \inf\varnothing= \infty inf ∅ = ∞ ; set K ^ ← min ( K max , T ^ ) \hat{K} \leftarrow\min(K_{\max}, \hat{T}) K ^ ← min ( K m a x , T ^ ) .
104: [Trace-mask product] For t = 0 , 1 , … , K ^ − 1 t = 0, 1, \ldots, \hat{K} - 1 t = 0 , 1 , … , K ^ − 1 , define w t ← ( γ λ ) t ∏ i = 1 t A ^ i w_t \leftarrow(\gamma\lambda)^t \prod_{i=1}^{t} \hat{A}_i w t ← ( γ λ ) t ∏ i = 1 t A ^ i , so w t = ( γ λ ) t w_t = (\gamma\lambda)^t w t = ( γ λ ) t until step T ^ − 1 \hat{T} - 1 T ^ − 1 and w t = 0 w_t = 0 w t = 0 afterwards.
105: [Bellman residuals] For t = 0 , … , K ^ − 1 t = 0, \ldots, \hat{K} - 1 t = 0 , … , K ^ − 1 , compute the Bellman residual δ t Q \delta_t^Q δ t Q (cf. Equation (1) )
δ t ← r t + γ E a ′ ∼ π ( ⋅ ∣ s t + 1 ) Q θ ˉ ( s t + 1 , a ′ ) − Q θ ˉ ( s t , a t ) . \delta_t \leftarrow r_t + \gamma\mathbb{E}_{a' \sim\pi(\cdot\mid s_{t+1})} Q_{\bar{\theta}}(s_{t+1}, a') - Q_{\bar{\theta}}(s_t, a_t). δ t ← r t + γ E a ′ ∼ π ( ⋅ ∣ s t + 1 ) Q θ ˉ ( s t + 1 , a ′ ) − Q θ ˉ ( s t , a t ) .
106: [Multi-step target]
y ^ ← Q θ ˉ ( s 0 , a 0 ) + ∑ t = 0 K ^ − 1 w t δ t . \hat{y} \leftarrow Q_{\bar{\theta}}(s_0, a_0) + \sum_{t=0}^{\hat{K}-1} w_t \delta_t. y ^ ← Q θ ˉ ( s 0 , a 0 ) + t = 0 ∑ K ^ − 1 w t δ t .
107: [Critic update] Take a gradient step on ( Q θ ( s 0 , a 0 ) − y ^ ) 2 (Q_\theta(s_0, a_0) - \hat{y})^2 ( Q θ ( s 0 , a 0 ) − y ^ ) 2 ; refresh Q θ ˉ Q_{\bar{\theta}} Q θ ˉ and update π \pi π as in the base actor–critic algorithm.
In this section, we describe the full implementation details of MINT. Following the standard practice in off-policy RL, we use the PyTorch version of the implementations in OpenAI SpinningUp [2 ] .
Experimental Setup We compare MINT with four baseline methods, a conventional one-step method, uncorrected multi-step method, Retrace [31 ] and Peng’s Q ( λ ) Q(\lambda) Q ( λ ) [24 ] . Given a randomly sampled trajectory ( s 0 , a 0 , r 0 , s 1 , a 1 , r 1 , s 2 , ⋯ ) (s_0, a_0, r_0, s_1, a_1, r_1, s_2, \cdots) ( s 0 , a 0 , r 0 , s 1 , a 1 , r 1 , s 2 , ⋯ ) , where Q θ ˉ Q_{\bar{\theta}} Q θ ˉ denotes the target Q Q Q -function, and a ~ ϕ ( s i ) \tilde{a}_\phi(s_i) a ~ ϕ ( s i ) is a sample from π ϕ ( ⋅ ∣ s i ) \pi_\phi(\cdot\mid s_i) π ϕ ( ⋅ ∣ s i ) . The detailed targets for the Q Q Q -function of all algorithms are described in Table 8 . Multi-step targets train the critic at the sampled anchor ( s 0 , a 0 ) (s_0, a_0) ( s 0 , a 0 ) . The actor retains its backbone objective at replay states using fresh actor actions; for SAC this maximizes the critic value minus the log-policy term, as specified in Equation (66) .
Table 8. Consistent finite-cap targets for the ordinary Bellman residual used in the main text. Here L L L is deterministic; MINT’s realized prefix length is K ^ ≤ L \hat{K}\le L K ^ ≤ L . The maximum-entropy substitutions are specified below.
Training and evaluation. For all algorithms, we use [ 256 , 256 ] [256, 256] [ 256 , 256 ] -sized multi-layer perceptrons (MLPs) for all neural networks. We train with 1M environment steps for OpenAI MuJoCo and evaluate the agent every 1000 steps using a deterministic policy in 10 episodes.
Finite-cap target definitions and backward recursion. All formulas below use a fixed target critic Q θ ˉ Q_{\bar{\theta}} Q θ ˉ during target construction. Write q i = Q θ ˉ ( s i , a i ) q_i = Q_{\bar{\theta}}(s_i, a_i) q i = Q θ ˉ ( s i , a i ) and v i = E a ∼ π ( ⋅ ∣ s i ) Q θ ˉ ( s i , a ) v_i = \mathbb{E}_{a \sim\pi(\cdot\mid s_i)} Q_{\bar{\theta}}(s_i, a) v i = E a ∼ π ( ⋅ ∣ s i ) Q θ ˉ ( s i , a ) . With twin target critics, the fixed function used here can be Q θ ˉ = min j = 1 , 2 Q θ ˉ j Q_{\bar{\theta}} = \min_{j=1,2} Q_{\bar{\theta}_j} Q θ ˉ = min j = 1 , 2 Q θ ˉ j . The stored actions a i a_i a i and fresh target-policy actions in v i v_i v i have different roles. In particular, v i v_i v i is an expectation at s i s_i s i , not a maximization over the current state.
Let L ≥ 1 L \geq1 L ≥ 1 be the available deterministic cap and let c i c_i c i exclude λ \lambda λ , as in Equation (1) . Set
b 0 = 1 , b t = ∏ j = 1 t c j , δ i = r i + γ v i + 1 − q i . b_0 = 1, \qquad b_t = \prod_{j=1}^{t} c_j, \qquad\delta_i = r_i + \gamma v_{i+1} - q_i. b 0 = 1 , b t = j = 1 ∏ t c j , δ i = r i + γ v i + 1 − q i .
The finite-cap TD-error target is
Y L ( c ) = q 0 + ∑ i = 0 L − 1 ( γ λ ) i b i δ i . (61) Y_L(c)=q_0+\sum_{i=0}^{L-1}(\gamma\lambda)^i b_i\delta_i. \tag*{(61)} Y L ( c ) = q 0 + i = 0 ∑ L − 1 ( γ λ ) i b i δ i . (61)
For MINT, c i = A ^ i c_i=\hat{A}_i c i = A ^ i , so b i = 1 { T ^ > i } b_i=1\{\hat{T}>i\} b i = 1 { T ^ > i } and terms after K ^ − 1 \hat{K}-1 K ^ − 1 , with K ^ = min ( L , T ^ ) \hat{K}=\min(L,\hat{T}) K ^ = min ( L , T ^ ) , vanish. Thus Equation (61) is exactly the target in the main-text Algorithm 1 and the full Algorithm 2; it includes the anchor q 0 q_0 q 0 and uses λ \lambda λ exactly once per continuation step.
An equivalent backward computation is
G L − 1 = r L − 1 + γ v L , G i = r i + γ v i + 1 + γ λ c i + 1 ( G i + 1 − q i + 1 ) , i = L − 2 , … , 0 , Y L ( c ) = G 0 . (62) \begin{aligned}
G_{L-1}&=r_{L-1}+\gamma v_L,\\
G_i&=r_i+\gamma v_{i+1}+\gamma\lambda c_{i+1}(G_{i+1}-q_{i+1}),\qquad i=L-2,\ldots,0,\\
Y_L(c)&=G_0. \tag*{(62)}
\end{aligned} G L − 1 G i Y L ( c ) = r L − 1 + γ v L , = r i + γ v i + 1 + γ λ c i + 1 ( G i + 1 − q i + 1 ) , i = L − 2 , … , 0 , = G 0 . (62)
To verify the identity, subtract q i q_i q i from the recursion and substitute backward: each continuation contributes a factor γ λ c i + 1 \gamma\lambda c_{i+1} γ λ c i + 1 , giving Equation (61) . For MINT one may start instead at G ^ K ^ − 1 = r K ^ − 1 + γ v K ^ \hat{G}_{\hat{K}-1}=r_{\hat{K}-1}+\gamma v_{\hat{K}} G ^ K ^ − 1 = r K ^ − 1 + γ v K ^ and recurse only over the retained prefix. The boundary uses a bootstrap at s K ^ s_{\hat{K}} s K ^ even when the trace ends because of the cap. There is no subtraction of an unsampled q L q_L q L at the boundary.
For comparison with reward-return formulas, define c ~ i = λ c i \tilde{c}_i=\lambda c_i c ~ i = λ c i for 1 ≤ i < L 1\le i<L 1 ≤ i < L and impose c ~ L = 0 \tilde{c}_L=0 c ~ L = 0 . Expanding Equation (61) gives
Y L ( c ) = ∑ i = 0 L − 1 γ i ( ∏ j = 1 i c ~ j ) [ r i + γ ( v i + 1 − c ~ i + 1 q i + 1 ) ] . (63) Y_L(c)=\sum_{i=0}^{L-1}\gamma^i\left(\prod_{j=1}^{i}\tilde{c}_j\right)\left[r_i+\gamma(v_{i+1}-\tilde{c}_{i+1}q_{i+1})\right]. \tag*{(63)} Y L ( c ) = i = 0 ∑ L − 1 γ i ( j = 1 ∏ i c ~ j ) [ r i + γ ( v i + 1 − c ~ i + 1 q i + 1 ) ] . (63)
The term c ~ L q L \tilde{c}_Lq_L c ~ L q L is defined to be zero and can be omitted from evaluation. This terminal convention prevents an extra subtraction at a deterministic cap; it does not remove the final value bootstrap. At L = 1 L=1 L = 1 , every row below reduces to r 0 + γ v 1 r_0+\gamma v_1 r 0 + γ v 1 .
Peng’s row is a finite mixture of ordinary multi-step returns [24 ] , not the c i ≡ 1 c_i\equiv1 c i ≡ 1 specialization of Equation (61) with target π \pi π . Its expanded form is
P 0 = ∑ i = 0 L − 1 ( γ λ ) i [ r i + γ ( 1 − λ ) v i + 1 ] + ( γ λ ) L v L . (64) P_0=\sum_{i=0}^{L-1}(\gamma\lambda)^i[r_i+\gamma(1-\lambda)v_{i+1}]+(\gamma\lambda)^Lv_L. \tag*{(64)} P 0 = i = 0 ∑ L − 1 ( γ λ ) i [ r i + γ ( 1 − λ ) v i + 1 ] + ( γ λ ) L v L . (64)
The last term is essential: at λ = 1 \lambda=1 λ = 1 this is the uncorrected L L L -step return including γ L v L \gamma^Lv_L γ L v L ; at λ = 0 \lambda=0 λ = 0 it is one-step. By contrast, c i ≡ 1 c_i\equiv1 c i ≡ 1 in Equation (61) is the finite-cap TD-error trace with unit coefficients c i = 1 c_i=1 c i = 1 , retaining ( γ λ ) i (\gamma\lambda)^i ( γ λ ) i weights and target π \pi π .
SAC soft residual and actor update. For SAC [16 ] , replace v i v_i v i in the TD-error constructions by
v i s o f t = E a ∼ π ϕ ( ⋅ ∣ s i ) [ min j = 1 , 2 Q θ ˉ j ( s i , a ) − α e n t log π ϕ ( a ∣ s i ) ] . (65) v_i^{\mathrm{soft}}=\mathbb{E}_{a\sim\pi_\phi(\cdot\mid s_i)}\left[\min_{j=1,2}Q_{\bar{\theta}_j}(s_i,a)-\alpha_{\mathrm{ent}}\log\pi_\phi(a\mid s_i)\right]. \tag*{(65)} v i soft = E a ∼ π ϕ ( ⋅ ∣ s i ) [ j = 1 , 2 min Q θ ˉ j ( s i , a ) − α ent log π ϕ ( a ∣ s i ) ] . (65)
and set δ i s o f t = r i + γ v i + 1 s o f t − q i \delta_i^{\mathrm{soft}}=r_i+\gamma v_{i+1}^{\mathrm{soft}}-q_i δ i soft = r i + γ v i + 1 soft − q i , with q i = min j Q θ ˉ j ( s i , a i ) q_i=\min_jQ_{\bar{\theta}_j}(s_i,a_i) q i = min j Q θ ˉ j ( s i , a i ) . An action drawn freshly from π ϕ \pi_\phi π ϕ gives the usual Monte Carlo estimate of this bootstrap. MINT and Retrace then use the same sum and backward recursion with this soft residual. The stopping rule and cap boundary are unchanged. This is an explicitly entropy-regularized extension; the main-text evaluation results concern the ordinary Bellman residual and Q π Q^\pi Q π stated there.
For a full maximum-entropy version of the reward-return baselines, intermediate entropy terms must also be included. With e i = − α e n t log π ϕ ( a i ∣ s i ) e_i=-\alpha_{\mathrm{ent}}\log\pi_\phi(a_i\mid s_i) e i = − α ent log π ϕ ( a i ∣ s i ) for stored actions 1 ≤ i < L 1\le i<L 1 ≤ i < L , the corresponding uncorrected soft return is
∑ i = 0 L − 1 γ i r i + ∑ i = 1 L − 1 γ i e i + γ L v L s o f t . \sum_{i=0}^{L-1}\gamma^i r_i+\sum_{i=1}^{L-1}\gamma^i e_i+\gamma^{L}v_L^{\mathrm{soft}}. i = 0 ∑ L − 1 γ i r i + i = 1 ∑ L − 1 γ i e i + γ L v L soft .
The finite mixture of these soft returns has boundary P L − 1 s o f t = r L − 1 + γ v L s o f t P_{L-1}^{\mathrm{soft}}=r_{L-1}+\gamma v_L^{\mathrm{soft}} P L − 1 soft = r L − 1 + γ v L soft and recursion
P i s o f t = r i + γ [ ( 1 − λ ) v i + 1 s o f t + λ ( P i + 1 s o f t + e i + 1 ) ] ( i < L − 1 ) . P_i^{\mathrm{soft}}=r_i+\gamma\left[(1-\lambda)v_{i+1}^{\mathrm{soft}}+\lambda\left(P_{i+1}^{\mathrm{soft}}+e_{i+1}\right)\right]\qquad(i<L-1). P i soft = r i + γ [ ( 1 − λ ) v i + 1 soft + λ ( P i + 1 soft + e i + 1 ) ] ( i < L − 1 ) .
These formulas specify full soft-return versions; inserting a soft value only at the end of an ordinary reward sum defines a different target. No entropy of the fixed starting action is added to a Q Q Q -target.
The SAC actor is optimized by gradient ascent on
E s ∼ B , a ∼ π ϕ ( ⋅ ∣ s ) [ min j Q θ j ( s , a ) − α e n t log π ϕ ( a ∣ s ) ] , (66) \mathbb{E}_{s\sim\mathcal{B},\,a\sim\pi_\phi(\cdot\mid s)}
\left[\min_j Q_{\theta_j}(s,a)-\alpha_{\mathrm{ent}}\log\pi_\phi(a\mid s)\right], \tag*{(66)} E s ∼ B , a ∼ π ϕ ( ⋅ ∣ s ) [ j min Q θ j ( s , a ) − α ent log π ϕ ( a ∣ s ) ] , (66)
using online critics and reparameterized actor actions. The multi-step critic target is detached during critic regression; it is not substituted for the actor’s differentiable critic objective. Here α e n t \alpha_{\mathrm{ent}} α ent is the SAC entropy coefficient, distinct from the discount–trace product γ λ \gamma\lambda γ λ .
TD3 bootstrap and the policy used in the ratio. For TD3 [14 ] , the bootstrap has no entropy term and uses the clipped double target critic at a smoothed target action:
a ~ i = clip A ( μ ˉ ϕ ( s i ) + clip ( ϵ i , − c , c ) ) , ϵ i ∼ N ( 0 , σ t a r g 2 I ) , \tilde{a}_i=\operatorname{clip}_{A}\left(\bar{\mu}_\phi(s_i)+\operatorname{clip}(\epsilon_i,-c,c)\right),\qquad\epsilon_i\sim\mathcal{N}(0,\sigma_{\mathrm{targ}}^2 I), a ~ i = clip A ( μ ˉ ϕ ( s i ) + clip ( ϵ i , − c , c ) ) , ϵ i ∼ N ( 0 , σ targ 2 I ) ,
v i T D 3 = E [ min j Q ˉ θ ˉ j ( s i , a ~ i ) ] . v_i^{\mathrm{TD3}}=\mathbb{E}\left[\min_j\bar{Q}_{\bar{\theta}_j}(s_i,\tilde{a}_i)\right]. v i TD3 = E [ j min Q ˉ θ ˉ j ( s i , a ~ i ) ] .
During TD3 data collection, the behavior policy adds Gaussian exploration noise with standard deviation 0.1 to the actor output and clips the action to the valid bounds. Its log-probability log β ( a i ∣ s i ) \log\beta(a_i\mid s_i) log β ( a i ∣ s i ) is computed at collection time and stored with the transition. At replay, the same stored action is evaluated under the smoothed target-actor law above, with σ t a r g = 0.2 \sigma_{\mathrm{targ}}=0.2 σ targ = 0.2 and noise clip c = 0.5 c=0.5 c = 0.5 . Both probability evaluations account for clipping, including the induced probability masses at clipping boundaries. The stopping rule uses the stored behavior value and the replay-time target value through min ( 1 , π / β ) = exp ( min ( 0 , log π − log β ) ) \min(1,\pi/\beta)=\exp(\min(0,\log\pi-\log\beta)) min ( 1 , π / β ) = exp ( min ( 0 , log π − log β )) . During uniform-random warm-up, the stored behavior value is that of the uniform action law. Thus the ratio compares the stochastic collection and target-action laws, rather than assigning a Gaussian density to the deterministic actor itself.
For true terminal transitions set the subsequent bootstrap and continuation to zero. A deterministic trace cap retains the bootstrap at its last observed non-state; a time-limit truncation should not be treated as a true terminal transition solely because the recording window ended.
Methods and Hyperparameters. We use two one-step RL algorithms, SAC and TD3, for the multi-step extension.
Twin-Delayed Deep Deterministic Policy Gradient (TD3). TD3 [14 ] adopts the same training pipeline and neural network architecture as DDPG, but introduces several improvements to address overestimation bias in Q-learning. TD3 uses two critic networks, Q θ 1 ( s , a ) Q_{\theta_1}(s,a) Q θ 1 ( s , a ) and Q θ 2 ( s , a ) Q_{\theta_2}(s,a) Q θ 2 ( s , a ) , with independent parameter sets θ 1 \theta_1 θ 1 and θ 2 \theta_2 θ 2 . This twin-critic design follows double Q-learning [44 ] .
Soft Actor-Critic (SAC). SAC [16 ] adopts the same training pipeline and architecture as DDPG/TD3 but introduces a maximum-entropy objective. SAC also maintains two critic networks to reduce overestimation bias.
We adopt the default hyperparameters from the OpenAI SpinningUp codebase. The cap length denotes the upper limit of the sub-trajectory length for the baseline algorithms (uncorrected n n n -step, Retrace, PQL). Detailed values are reported below.
Algorithm. The detailed multi-step target construction is given as Algorithm 1 in the main text (§ 3.2 ). It replaces the multi-step critic target in the SAC and TD3 implementations evaluated here; the trace construction computes p ^ i \hat{p}_i p ^ i , draws the keep masks, and selects the retained prefix. Its critic target is the finite-cap residual sum in Equation (61) , with the explicit SAC or TD3 bootstrap substitution above.
Experimental Details. We implement MINT and the baselines in PyTorch and evaluate them using the standard off-policy RL protocol in Google Dopamine [8 ] . Code is available at https://github.com/seungyubhan/mint .
Numerical treatment of the policy ratio. Here, log-probability denotes log-mass for a discrete policy and log-density for a continuous policy. For SAC’s tanh-squashed Gaussian actors, evaluating π ( a i ∣ s i ) / β ( a i ∣ s i ) \pi(a_i \mid s_i)/\beta(a_i \mid s_i) π ( a i ∣ s i ) / β ( a i ∣ s i ) at a stored action requires three precautions. (i) Stored behavior log-probability. log β ( a i ∣ s i ) \log\beta(a_i \mid s_i) log β ( a i ∣ s i ) is computed and stored at collection time (during the uniform warm-up it is the log-probability under the uniform box distribution), so β \beta β is always the data-generating policy of that transition and no density model of the drifting replay distribution is needed (Algorithm 2). (ii) Squash correction. The log-probability of a stored action under π \pi π is evaluated by inverting the squash, u = atanh ( a / a max ) u = \operatorname{atanh}(a/a_{\max}) u = atanh ( a / a m a x ) with the argument clipped to [ − 1 + 10 − 6 , 1 − 10 − 6 ] [-1+10^{-6}, 1-10^{-6}] [ − 1 + 1 0 − 6 , 1 − 1 0 − 6 ] to guard the saturated boundary, and adding the tanh log-Jacobian; the policy log-standard-deviation is clamped to [ − 20 , 2 ] [-20, 2] [ − 20 , 2 ] . (iii) Log-space ratio. The clipped ratio is formed as min ( 1 , π / β ) = exp ( min ( 0 , log π − log β ) ) \min(1,\pi/\beta)=\exp(\min(0,\log\pi-\log\beta)) min ( 1 , π / β ) = exp ( min ( 0 , log π − log β )) : clamping the log-ratio at 0 is exactly the minimum against 1 and removes overflow, while underflow of the exponential for very negative log-ratios yields p ^ i → 1 \hat{p}_i \to1 p ^ i → 1 , the correct limit rather than a NaN. p ^ i \hat{p}_i p ^ i is finally clamped to [ 0 , 1 ] [0,1] [ 0 , 1 ] and the anchor step t = 0 t=0 t = 0 is always kept. These steps keep the ratio stable when π \pi π becomes sharply peaked relative to β \beta β (the saturation regime of Appendix § 8.12 .§ 1 ).
Additional Experimental Results
MuJoCo: Evaluation Protocol and Additional Results
Evaluation Protocol and Score Normalization
We conducted experiments following the evaluation protocol proposed in [3 ] . This protocol emphasizes robust evaluation through interquartile mean (IQM) scores and normalized performance.
The main MuJoCo comparison uses the five-seed-per-method-and-task protocol stated in Appendix § 8.14 , with scores aggregated after min–max normalization. Additional experiments report their own seed counts. The main learning curves cover 1,000K environment interactions; Table 11 lists the reference scores used for normalization. Metric computation was conducted using the official codebase of [3 ] . For min–max normalization, expert and random scores were taken from the Minari extension [12 ] of D4RL [13 ] ; the Humanoid benchmark follows Minari scores, while the remaining tasks use D4RL values.
Table 11. Reference scores for min–max normalization. Random and expert performance values are taken from D4RL [13 ] and Minari [12 ] .
Across both SAC- and TD3-based experiments, MINT attains higher IQM at earlier training checkpoints than the compared multi-step methods and baselines. Only MINT surpasses the expert score within the reported 1,000K-step budget.
Sensitivity to λ \lambda λ
λ \lambda λ is shared by MINT, Peng’s Q ( λ ) Q(\lambda) Q ( λ ) , and Retrace: the main table runs MINT, Peng’s Q ( λ ) Q(\lambda) Q ( λ ) , and Retrace at the same λ = 0.7 \lambda=0.7 λ = 0.7 , chosen to match the Peng’s Q ( λ ) Q(\lambda) Q ( λ ) protocol rather than to optimize our operator.
Figure 17. Comparison of SAC, SAC-MINT, and MBPO across four MuJoCo benchmarks. We use different multi-step configurations per environment: halfcheetah and walker use a standard multi-step setting (cap length = 5 = 5 = 5 , λ = 0.7 \lambda= 0.7 λ = 0.7 ), whereas ant and hopper follow the longer-horizon model-based configuration inspired by MBPO (cap length = 25 = 25 = 25 , λ = 0.1 \lambda= 0.1 λ = 0.1 ). The curves compare the sample efficiency of SAC, MINT, and MBPO under these task-specific settings; MINT often attains returns comparable to MBPO while using a model-free algorithm.
It is nevertheless a hyperparameter, setting the ( γ λ ) t (\gamma\lambda)^t ( γ λ ) t weight on every term of the backup. Table 12 reports the full sweep on MuJoCo Ant-v2 and Hopper-v2 (fixed α = 0.2 \alpha= 0.2 α = 0.2 , 1M steps, final return averaged over the last five evaluations, mean ± \pm ± s.d. across seeds) without selecting favorable settings. Return decreases with λ \lambda λ on Ant-v2 and is comparatively stable on Hopper-v2 (about 9% variation over the range); from λ = 0.7 \lambda= 0.7 λ = 0.7 to 0.9 the change is − 12.9 % -12.9\% − 12.9% on Ant-v2 and + 3.5 % +3.5\% + 3.5% on Hopper-v2, with overlapping mean ± \pm ± s.d. ranges in both cases. Two qualifications apply: λ = 0.7 \lambda= 0.7 λ = 0.7 is not the best setting in this sweep (the two highest mean returns on Ant-v2 occur at 0.1 and 0.3), consistent with its having been fixed for protocol matching; and because the baselines were not swept, we make no claim about how the gap between methods changes at other λ \lambda λ . Kozuno et al. [24 ] likewise find Peng’s Q ( λ ) Q(\lambda) Q ( λ ) sensitive to λ \lambda λ on DMControl (their Appendix J.8), with the best overall performance near λ = 0.7 \lambda= 0.7 λ = 0.7 , so λ \lambda λ sensitivity is not unique to MINT. In the reported sweep the cap is fixed at 5 independently of λ \lambda λ (Equation (8) ). For fixed policies and a fixed trajectory, λ \lambda λ changes the ( γ λ ) t (\gamma\lambda)^t ( γ λ ) t weights without changing the stopping rule; across training runs, it can also affect the learned critic and policy.
Table 12. λ \lambda λ sweep for SAC-MINT (fixed α = 0.2 \alpha= 0.2 α = 0.2 , K max = 5 K_{\max} = 5 K m a x = 5 , 1M steps; mean ± \pm ± s.d. across 2–3 seeds per cell). The main-table value is λ = 0.7 \lambda= 0.7 λ = 0.7 .
DMC: Main Results and Additional Diagnostics
11-Task Evaluation: Setup and Task Selection
Task selection and scope. Our eleven-task evaluation is a study-specific subset of the DeepMind Control Suite [42 ] , rather than a separate official benchmark. We selected tasks from the 20-task DMC Easy & Medium set used by SimBa [25 ] : six tasks with indicator or height-gated reward (Cartpole swingup sparse, Pendulum swingup, Finger turn easy/hard, and Hopper stand/hop), and five dense-reward tasks with action dimension at most six (Walker walk/run, Cheetah run, Acrobot swingup, and Fish swim). The dense-task choice considered published SAC performance indicating potential for improvement and the availability of existing task implementations; this is a purposive subset, not a random or exhaustive sample of DMC. The comparison applies a common protocol to all eleven selected tasks. The evaluated subset excludes Ball-in-Cup catch and Acrobot swingup sparse. Table 14 identifies the exact evaluation set. The reward-frequency groups reported later are a separate descriptive measurement under a uniform-random policy; they are not official DMC task classes. Conclusions are restricted to this set and protocol.
Common protocol. We train eleven DMC tasks with five seeds per method/task (165 runs) for 500K environment steps. All methods use γ = 0.99 \gamma= 0.99 γ = 0.99 , automatic entropy tuning initialized at α = 0.2 \alpha= 0.2 α = 0.2 , batch size 256, two 256-unit hidden layers, learning rates 3 × 10 − 4 3 \times10^{-4} 3 × 1 0 − 4 , and one update per step after 5K steps. We evaluate ten episodes every 10K steps. Performance averages the last five evaluations within each run; task tables report the mean and standard deviation over five seeds. Aggregate scores divide returns by 1000. No action repeat is applied. MINT uses λ = 0.9 \lambda= 0.9 λ = 0.9 , K max = 10 K_{\max} = 10 K m a x = 10 , and stored collection-time behavior log-probabilities. SAC-multistep is a pure uncorrected ten-step return with γ 10 \gamma^{10} γ 10 bootstrapping; it does not use λ \lambda λ weighting. Consequently, this comparison does not isolate stochastic stopping as the only changed target ingredient. Only runs using this common protocol contribute to the reported aggregates.
11-Task Performance and Task Groups
Aggregate learning and distribution. The aggregate in Figure 6 uses final-window scores. The IQM curve instead aggregates scores at each evaluation checkpoint; its final point can differ from the final-window IQM. The performance profile pools the 55 task/seed final scores per method. At normalized score 0.5, MINT exceeds the threshold in 28/55 runs, versus 21/55 and 15/55 for one-step and ten-step. These are descriptive threshold comparisons, not proof of dominance at every threshold.
Task-specific gains and exceptions. Finger easy/hard and Walker walk/run favor MINT over both baselines at the reported endpoint. Hopper recovers relative to pure ten-step, but Hopper-Stand has large seed variability. One-step is stronger on Cheetah, and ten-step is stronger on Pendulum and Fish. Acrobot-Swingup has low absolute return despite improvement over one-step; this evaluation uses the dense-reward task, distinct from Acrobot-Swingup-Sparse.
Descriptive task groups. Reward accessibility is measured as the fraction of uniform-random episodes containing positive reward. Rare (< 5%) comprises Cartpole sparse and Pendulum; occasional (10–50%) comprises Finger easy/hard and Hopper stand/hop; frequent (100%) comprises Walker walk/run, Cheetah, Acrobot, and Fish. The corresponding MINT IQMs are 0.392, 0.453, and 0.491, versus one-step 0.320/0.322/0.487 and ten-step 0.440/0.157/0.327. The occasional group has probabilities of improvement 0.78 [0.63, 0.91] over one-step and 0.88 [0.74, 1.00] over ten-step. Action groups contain 3/2/6 tasks; the two-dimensional group is entirely Finger. The learned-K grouping is post hoc. Across the six rare/occasional tasks, MINT improves mean return over one-step in all six and over ten-step in five; Pendulum is the exception to the latter comparison. The strictly rare group contains only two tasks and is reported separately. These partitions describe reward accessibility under a random policy, not the exploration taxonomy used for Atari, and do not establish causal effects of reward frequency, action dimension, or length.
Figure 22. DMC action-dimensionality groups. Endpoint IQM, bootstrap intervals and estimates. The two-dimensional group contains only Finger tasks, confounding action dimension with task identity.
Figure 23. DMC learned-horizon groups. Endpoint IQM grouped by MINT mean length after training. Grouping is post hoc and does not certify an optimal horizon.
Mismatch–Horizon Diagnostics and Reference Construction
Figure 4 relates measured trace lengths to conditional mismatch references. For logged mean keep p p p and λ = 0.9 \lambda= 0.9 λ = 0.9 , a state-homogeneous symmetric-binary closure gives
χ λ p r o x y = [ ( 1 + 2 λ ) p − 1 ] + 2 λ p . \chi_{\lambda}^{\mathrm{proxy}} = \frac{\left[(1 + 2\lambda)p - 1\right]_{+}}{2\lambda p}. χ λ proxy = 2 λ p [ ( 1 + 2 λ ) p − 1 ] + .
The vertical coordinate J J J is the logged mean length. Let S L ( u ) = ∑ t = 0 L − 1 u t S_L(u) = \sum_{t=0}^{L-1} u^t S L ( u ) = ∑ t = 0 L − 1 u t , solve S 10 ( q ) = J S_{10}(q) = J S 10 ( q ) = J , and set M γ ( J ) = S 10 ( γ λ q ) M_{\gamma}(J) = S_{10}(\gamma\lambda q) M γ ( J ) = S 10 ( γ λ q ) . Green contours count the fraction of γ ∈ { 0.01 , 0.02 , … , 0.99 } \gamma\in\{0.01, 0.02, \ldots, 0.99\} γ ∈ { 0.01 , 0.02 , … , 0.99 } satisfying
2 χ λ p r o x y ( 1 − M γ ( J ) − 1 ) < 1 − γ . 2\chi_{\lambda}^{\mathrm{proxy}}\left(1 - M_{\gamma}(J)^{-1}\right) < 1 - \gamma. 2 χ λ proxy ( 1 − M γ ( J ) − 1 ) < 1 − γ .
Logged p p p and reconstructed q q q can differ. Thus these contours combine two conditional approximations and are neither sample-wise contraction certificates, convergence probabilities, nor measured operator gains. Gray curves fit monotone cubic-spline 90th-quantile regression to all positive-proxy records within each panel (2,696 records in total), then add 0.6 steps for display. They are not confidence bounds; solid/dashed portions distinguish supported and sparse ranges. Of these records, 457 are displayed; hollow/filled marks distinguish steps at or below/above 140K.
At γ 0 = 0.99 \gamma_0 = 0.99 γ 0 = 0.99 , blue guides combine a checkpoint block-transition proxy ρ ^ e f f \hat{\rho}_{\mathrm{eff}} ρ ^ eff on visited support with trace-mass bounds. This is not uniform minorization. Task-specific C e C_e C e is calibrated to at least 50% coverage of broadly visited checkpoint cells; uncapped K r e f K_{\mathrm{ref}} K ref values are summarized by task medians and equal-task group means (3.82, 6.08, 7.77). The two Finger tasks have only three and two calibration cells, respectively. Same-data calibration provides conditional references, not independent evidence of noise reduction. The on-policy motivation from Duan and Wainwright [10 ] does not transfer its guarantee to off-policy learning or these empirical proxies.
Target Operators and Bias–Variance Diagnostics
Weighted operator definitions. For a shared behavior segment, let q t = Q θ ˉ ( s t , a t ) q_t = Q_{\bar{\theta}}(s_t, a_t) q t = Q θ ˉ ( s t , a t ) and let v t + 1 v_{t+1} v t + 1 be the sampled soft target-policy value used by the diagnostic code. Define δ t = r t + γ v t + 1 − q t \delta_t = r_t + \gamma v_{t+1} - q_t δ t = r t + γ v t + 1 − q t . The fixed-length λ \lambda λ -weighted TD-error trace (lam in the diagnostic code) is
G K λ = q 0 + ∑ t = 0 K − 1 ( γ λ ) t δ t = r 0 + γ v 1 + ∑ t = 1 K − 1 ( γ λ ) t ( r t + γ v t + 1 − q t ) . (67) G_K^{\lambda} = q_0 + \sum_{t=0}^{K-1}(\gamma\lambda)^t\delta_t = r_0 + \gamma v_1 + \sum_{t=1}^{K-1}(\gamma\lambda)^t(r_t + \gamma v_{t+1} - q_t). \tag*{(67)} G K λ = q 0 + t = 0 ∑ K − 1 ( γ λ ) t δ t = r 0 + γ v 1 + t = 1 ∑ K − 1 ( γ λ ) t ( r t + γ v t + 1 − q t ) . (67)
This is a finite-cap Q π ( λ ) Q^\pi(\lambda) Q π ( λ ) -type TD-error trace [19 ] , retaining the λ \lambda λ weights but using unit importance-trace coefficients. The code evaluates the same sum as ∑ t < K ( γ λ ) t [ r t + γ v t + 1 − γ λ 1 { t < K − 1 } q t + 1 ] \sum_{t<K}(\gamma\lambda)^t[r_t + \gamma v_{t+1} - \gamma\lambda1\{t < K - 1\}q_{t+1}] ∑ t < K ( γ λ ) t [ r t + γ v t + 1 − γ λ 1 { t < K − 1 } q t + 1 ] . MINT selects G K ^ λ G_{\hat{K}}^\lambda G K ^ λ , where K ^ \hat{K} K ^ is the retained prefix from Bernoulli draws with keep probability 1 − min ( 1 , π ( a t ∣ s t ) / β ( a t ∣ s t ) ) 1 - \min(1, \pi(a_t \mid s_t)/\beta(a_t \mid s_t)) 1 − min ( 1 , π ( a t ∣ s t ) / β ( a t ∣ s t )) . Its random length depends on the segment. Pure n n n -step instead uses N n = ∑ t < n γ t r t + γ n v n N_n = \sum_{t<n}\gamma^t r_t + \gamma^n v_n N n = ∑ t < n γ t r t + γ n v n .
Peng’s finite-cap mixture is P K λ = ( 1 − λ ) ∑ n = 1 K − 1 λ n − 1 N n + λ K − 1 N K P_K^\lambda= (1 - \lambda)\sum_{n=1}^{K-1}\lambda^{n-1}N_n + \lambda^{K-1}N_K P K λ = ( 1 − λ ) ∑ n = 1 K − 1 λ n − 1 N n + λ K − 1 N K [24 ] , as in Equation (64) . With shared v t v_t v t , the pathwise difference is
G K λ − P K λ = ∑ t = 1 K − 1 ( γ λ ) t ( v t − q t ) . (68) G_K^\lambda- P_K^\lambda= \sum_{t=1}^{K-1}(\gamma\lambda)^t(v_t - q_t). \tag*{(68)} G K λ − P K λ = t = 1 ∑ K − 1 ( γ λ ) t ( v t − q t ) . (68)
Thus the diagnostic’s 1am curve, including its K = 10 K = 10 K = 10 square, is not a measurement of Peng’s return mixture off policy. The planes use “fixed λ \lambda λ -TD trace” and “fixed trace (K = 10 K = 10 K = 10 )” accordingly. No Peng-mixture target comparison is inferred from those points. This distinction does not change the independently trained Peng baselines in the MuJoCo experiments.
Shared evaluation and aggregation. Policy and learned target critics are frozen at 100K, 200K, 300K, or 500K checkpoints of the SAC-MINT run; these are not oracle-critic measurements. Behavior segments are generated with that checkpoint and its available predecessors, rather than drawn from the replay buffer. Operators share segments within each evaluated cell. Bias is mean target minus a finite-rollout soft-Q π Q^\pi Q π Monte-Carlo reference; finite-rollout and sampling error remain in that reference. The plane plots the absolute value of the seed-aggregated signed bias, rather than mean absolute error. Target spread is the within-anchor standard deviation aggregated over anchors and behavior checkpoints by a root-mean-variance calculation within each seed, then averaged over seeds. MINT spread also includes randomness in the stopping draws. Neither coordinate has a confidence interval in these planes. The plotted bias is a finite-backup diagnostic with a learned critic, not the fixed-point bias of the population operator. Its change therefore need not contradict preservation of Q π i n Q^{\pi_{\mathrm{in}}} Q π in in fixed-policy evaluation. These target coordinates complement the control returns rather than establish their cause. Values are expressed as percentages relative to 100 = 1 / ( 1 − γ ) 1/(1-\gamma) 1/ ( 1 − γ ) ; this is not an upper bound on entropy-regularized value. The same seed set is used for all operators in each task–stage panel.
Stage-specific exclusions. The aggregation excludes a seed at a stage if any included behavior checkpoint has absolute one-step target bias divided by 100 above 0.5. This is a diagnostic filter, not an independent certification that the remaining critic is accurate. It applies to target diagnostics only; the control-performance aggregates keep all five seeds. Table 15 gives retained counts and excluded seed identities. In particular, Pendulum at 500K retains only seed 0; Finger–Hard retains two seeds. These late-stage panels are descriptive conditional estimates.
Table 15. Seeds retained in target diagnostics. Each cell gives the retained count, followed by excluded zero-based seed IDs in parentheses when nonempty. Training and control-score aggregates use all five seeds.
Aggregate spread and the bias trade-off. At 200K/300K/500K there are 33 task–stage cases. Fixed-K K K λ \lambda λ -weighted TD trace spread increases monotonically with length in 32; the median length-ten/length-one standard-deviation ratio is 4.09. MINT is below the cap-ten TD trace in 26/33 (median ratio 0.916), and below pure ten-step in 27/33 (0.759). For multidimensional-action tasks, both counts are 23/24, with median ratios 0.911 and 0.733, respectively. In 20/33 cases, a lower spread than pure ten-step coexists with no smaller absolute bias. Against an interpolated fixed-K K K λ \lambda λ -weighted TD trace at MINT’s measured mean length, the median ratio is 1.077 (below one in 10/33). That interpolation is not a randomized matched-length control. The aggregate statistics therefore support reductions relative to long fixed backups without establishing uniform bias reduction or an additional equal-length benefit from adaptive stopping.
Selected lower-spread examples and counterexamples. Figure 7 shows Finger–Turn–Hard, Hopper–Hop, Cartpole–Swingup–Sparse, and Walker–Run at 200K. The one-step-bias filter retains 3/4/5/5 seeds: Finger–Turn–Hard excludes seeds 0 and 4; Hopper–Hop excludes seed 1. No additional filter is introduced. MINT/fixed-K K K = 10 standard-deviation ratios are 0.699, 0.806, 0.704, and 0.760. Within-anchor variance is lower in 3/3, 4/4, 3/5, and 5/5 retained seeds, respectively. In particular, the aggregated reduction on Cartpole–Swingup–Sparse does not imply a decrease in every seed. Absolute aggregated bias is higher on Finger–Turn–Hard (11.051 versus 10.689), Hopper–Hop (1.364 versus 1.317), and Walker–Run (0.714 versus 0.700), but lower on Cartpole–Swingup–Sparse (0.211 versus 0.218). These descriptive comparisons do not establish statistical significance or represent all tasks.
All-task planes are collected in Section F.2.7. At 300K, Finger–Easy reduces standard deviation from 1.095 to 0.650 relative to the cap-ten TD trace while bias rises from 11.647 to 11.947. Walker–Walk reduces both spread (0.418 to 0.381) and bias (0.253 to 0.224); Fish has a modest spread reduction (0.638 to 0.593) with higher bias (16.485 to 16.754). Acrobot is a counterexample: spread rises from 0.068 to 0.122 and bias from 2.484 to 2.595; pure ten-step is also lower in both coordinates (Figure 28 ). The all-task results and 26/33 lower-spread count provide the broader context for the selected main-text examples.
Figure 28. All-task DMC bias–variance plane at 300K. Eleven task panels; 50/55 task–seed cells retained (Table 15 ). Red: fixed-K K K λ \lambda λ -weighted TD trace, square: K = 10 K = 10 K = 10 ; blue: pure n n n -step; green diamond: MINT. Dark-to-light points mark length 1–10. Horizontal: absolute aggregated signed bias; vertical: target standard deviation. Both are percentages relative to the normalization value 100, not a soft-value upper bound. Panel scales differ and no confidence intervals are shown. This figure includes all eleven tasks at 300K, including the Acrobot counterexample. Finger-Easy, Walker-Walk, Fish, and Acrobot show spread savings, modest changes, and a negative case under the same stage-specific aggregation.
Reward Retention and Target Variance Diagnostics
Protocol and scope. We evaluate five training seeds (0–4), fixing the policy and learned target critic from each MINT checkpoint. Table 16 summarizes the measurement. Each behavior distribution is a single checkpoint rollout, rather than training replay. Policy pairs are reported separately, without pooling the π = β \pi= \beta π = β boundary with off-policy pairs or applying the stage-specific seed filter of Section F.2.4. The reward-frequency table covers five tasks; the subsequent retention and target tables report four of these tasks (FH, CS, WR, and AS), excluding HH. Task codes are FH: Finger-Turn-Hard; CS: Cartpole-Swingup-Sparse; WR: Walker-Run; AS: Acrobot-Swingup; and HH: Hopper-Hop.
Table 16. Main reward-retention and target-diagnostic protocol.
Reward frequency and conditional retention. For a reward-bearing window, define
R r e t ( K ) = ∑ t < K ( γ λ ) t ∣ r t ∣ ∑ t < L ( γ λ ) t ∣ r t ∣ , ∑ t < L ( γ λ ) t ∣ r t ∣ > 0. R_{\mathrm{ret}}(K) = \frac{\sum_{t<K}(\gamma\lambda)^t\lvert r_t\rvert}{\sum_{t<L}(\gamma\lambda)^t\lvert r_t\rvert}, \qquad\sum_{t<L}(\gamma\lambda)^t\lvert r_t\rvert> 0. R ret ( K ) = ∑ t < L ( γ λ ) t ∣ r t ∣ ∑ t < K ( γ λ ) t ∣ r t ∣ , t < L ∑ ( γ λ ) t ∣ r t ∣ > 0.
The shuffled-K K K control preserves MINT’s stopping-length distribution while removing its alignment with the behavior window. Retention measures use of already encountered rewards, separately from reward discovery. Table 17 reports reward availability before conditioning on reward-bearing windows.
Table 17. Reward availability in the sampled behavior windows. Policy-pair entries are in thousands of environment steps.
Target moments. For each anchor x = ( s 0 , a 0 ) x = (s_0, a_0) x = ( s 0 , a 0 ) , let μ m ( x ) \mu_m(x) μ m ( x ) and v m ( x ) v_m(x) v m ( x ) be the conditional mean and variance of method m m m ’s target over behavior windows and stopping draws. The reported squared bias averages [ μ m ( x ) − Q M C π ( x ) ] 2 [\mu_m(x)-Q^\pi_{\mathrm{MC}}(x)]^2 [ μ m ( x ) − Q MC π ( x ) ] 2 over anchors, rather than squaring an aggregated signed error. Variance averages v m ( x ) v_m(x) v m ( x ) , and target MSE combines these two quantities; finite-rollout reference error remains. These are learned-critic finite-backup diagnostics, not operator fixed-point bias. This distinction also separates the present squared-bias statistic from the horizontal coordinate of Figure 7 .
For stochastic targets, with D = 32 D=32 D = 32 draws per window, the variance estimator accounting for within-window sampling is
v ^ = Var ^ w ( Y ‾ w ) + ( 1 − D − 1 ) Avg ^ w Var ^ d ( Y w d ) , \widehat{v}=\widehat{\operatorname{Var}}_w(\overline{Y}_w)+(1-D^{-1})\widehat{\operatorname{Avg}}_w\widehat{\operatorname{Var}}_d(Y_{wd}), v = Var w ( Y w ) + ( 1 − D − 1 ) Avg w Var d ( Y w d ) ,
where both sample variances use their corresponding independent sample counts. Fixed-length targets use the sample variance across windows. Table 18 gives MINT’s absolute values in unnormalized target units; values are seed means.
Table 18: MINT target statistics in the main measurement. Squared bias is computed per anchor before averaging.
Table 18.
Table 19: Reward retention: shuffled-K K K minus MINT. Negative differences favor MINT; intervals are for paired seed differences.
Table 19. Retention is defined only for reward-bearing windows. Dashes indicate fewer than five valid training seeds.
Variance at a matched length distribution. The variance point estimates favor MINT over shuffled-K K K in 11/12 off-policy task-pair conditions: all three pairs on FH, CS, and AS, and two on WR. On FH, the reductions relative to shuffled-K K K are approximately 6%, 45%, and 23% for (200, 100), (500, 400), and (500, 100) K, respectively. The corresponding reductions are approximately 64–68% on CS and 65–86% on AS; WR differences are small and mixed. These percentages are reconstructed from rounded summary values. Unlike the interpolated mean-length comparison in Section F.2.4, this control preserves the full marginal stopping-length distribution. The results therefore support a descriptive variance benefit from the alignment between stopping length and trajectory in this measurement. The reported intervals concern MSE and retention, not variance differences, so the 11/12 count is not a significance count.
FH also has higher retention than shuffled-K K K in the two β = 100 K \beta= 100\mathrm{K} β = 100 K conditions, with differences 0.092 and 0.106 and intervals excluding zero. Its (500, 400)K retention interval includes zero. CS retention differences favor MINT, but are defined in only three or four seeds. AS shows substantial variance reductions with little retention difference, so the variance pattern is not confined to reward-retention gains.
Fixed-length references and error decomposition. For completeness, Tables 20 to 22 reproduce the target-MSE comparisons and their squared-bias and variance differences. Every difference is reference method minus MINT. The interval and direction columns apply only to Δ M S E \Delta\mathrm{MSE} Δ MSE . The variance point estimates do not establish dominance over fixed traces: MINT has lower variance than fixed-10 in only 1/12 off-policy conditions, and higher variance than fixed-1 in all 12. At the π = β \pi= \beta π = β boundary it is approximately one-step and has lower variance than fixed-10 in all four tasks. Thus the matched-distribution benefit is distinct from a general variance advantage over long fixed backups. Comparisons with Section F.2.4 must also account for that diagnostic’s averaging across current and older behavior checkpoints, seed exclusions, and standard-deviation aggregation.
Table 20. Target comparison: shuffled-K K K minus MINT. All four tasks use five training seeds per policy pair. Lower/Higher means that the Δ \Delta Δ MSE interval lies wholly below/above zero. Variance differences are point estimates without separate intervals.
Table 21. Target comparison: fixed TD trace, K = 10 K = 10 K = 10 minus MINT. All four tasks use five training seeds per policy pair. Lower/Higher means that the Δ \Delta Δ MSE interval lies wholly below/above zero. Variance differences are point estimates without separate intervals.
Table 22. Target comparison: fixed TD trace, K = 1 K = 1 K = 1 minus MINT. All four tasks use five training seeds per policy pair. Lower/Higher means that the Δ \Delta Δ MSE interval lies wholly below/above zero. Variance differences are point estimates without separate intervals.
Table 23. MSE interval directions over the 16 task–pair conditions. Entries count lower / includes zero / higher for reference minus MINT, using the within-window variance estimator defined above.
Interpretation and limits. The main result is higher reward retention on selected sparse-task pairs together with lower variance than a shuffled control at the same length distribution. This is a finite-checkpoint target result and does not identify the cause of the separately reported control-performance gains. Absolute target MSE is dominated by squared bias in these cells; neither it nor target variance alone measures the usefulness of an update for policy improvement. The measurements do not separate action-independent value offsets from action-dependent errors. No online shuffled-K K K learning comparison is provided. At π = β \pi= \beta π = β , MINT and fixed-1 targets agree to within 1.7 × 10 − 5 1.7 \times10^{-5} 1.7 × 1 0 − 5 in the reported check; tiny stored-versus-recomputed log-probability differences can produce occasional non-one-step draws.
SAC Temperature and Retained Length
All three DMC training methods use automatic entropy tuning. The median within-run Spearman correlations over 55 MINT runs are − 0.098 -0.098 − 0.098 for α \alpha α versus K K K and − 0.045 -0.045 − 0.045 for α \alpha α versus logged keep probability; first-difference medians are − 0.028 -0.028 − 0.028 and − 0.031 -0.031 − 0.031 . Opposite task-specific signs do not support a universal temperature-to-horizon map. These autocorrelated series are descriptive rather than independent-observation tests. Temperature participates in learning, but continuation depends on current and stored behavior policies; the association does not identify a causal mechanism.
All-Task and All-Stage Bias–Variance Results
Reading the training stages. At 100K, the only behavior checkpoint equals the target policy, so the stopping rule is one-step; this is a reference case rather than evidence that adaptation improves an off-policy target. Later stages include older behavior checkpoints. Comparing stages changes policies, critics, and behavior mixtures together, so movement in these planes is not a controlled time trend of one fixed operator. At 500K, Finger-Hard, Cheetah, and Acrobot retain visible cases where MINT does not improve both coordinates relative to the cap-ten TD trace. The planes complement the control returns rather than proving that a target-level coordinate change causes the training-score difference.
Figure 26. All-task DMC bias–variance plane at 100K. Eleven task panels; 51/55 task–seed cells retained (Table 15 ). Red: fixed-K K K λ \lambda λ -weighted TD trace, square: K = 10 K = 10 K = 10 ; blue: pure n n n -step; green diamond: MINT. Dark-to-light points mark length 1–10. Horizontal: absolute aggregated signed bias; vertical: target standard deviation. Both are percentages relative to the normalization value 100, not a soft-value upper bound. Panel scales differ and no confidence intervals are shown. The available behavior and target checkpoints coincide. MINT is one-step by construction; overlapping marks can hide its diamond. The other fixed-length targets remain useful reference curves with learned critics.
Figure 27. All-task DMC bias–variance plane at 200K. Eleven task panels; 50/55 task–seed cells retained (Table 15 ). Red: fixed-K K K λ \lambda λ -weighted TD trace, square: K = 10 K = 10 K = 10 ; blue: pure n n n -step; green diamond: MINT. Dark-to-light points mark length 1–10. Horizontal: absolute aggregated signed bias; vertical: target standard deviation. Both are percentages relative to the normalization value 100, not a soft-value upper bound. Panel scales differ and no confidence intervals are shown. The first off-policy stage includes current and older behavior checkpoints. Finger/Hopper and Walker panels show different trade-offs; compare both coordinates within a panel rather than reading a single universal frontier.
Figure 29. All-task DMC bias–variance plane at 500K. Eleven task panels; 45/55 task–seed cells retained (Table 15 ). Red: fixed-K K K λ \lambda λ -weighted TD trace, square: K = 10 K = 10 K = 10 ; blue: pure n n n -step; green diamond: MINT. Dark-to-light points mark length 1–10. Horizontal: absolute aggregated signed bias; vertical: target standard deviation. Both are percentages relative to the normalization value 100, not a soft-value upper bound. Panel scales differ and no confidence intervals are shown. These panels use the 500K checkpoints. Pendulum retains one seed and Finger-Hard two; these panels do not characterize all five trained runs. Negative cases remain visible, so lower spread or bias is not asserted uniformly.
Additional CartPole Balance and Hopper Transition Diagnostics
The CartPole-Balance-Sparse λ \lambda λ sweep and Hopper transition-distance diagnostic use separate protocols from Table 13 . Figure 30 documents parameter sensitivity and normalized next-state distances; the latter is not a policy-TV estimate. One-step matching and empirical backup gains are examined separately in Appendices § 8.12 .§ 2 and G.3.
Figure 30. (a) CartPole-Balance-Sparse: the original λ ∈ { 0.3 , 0.7 , 0.9 , 1 } \lambda\in\{0.3, 0.7, 0.9, 1\} λ ∈ { 0.3 , 0.7 , 0.9 , 1 } comparison. MINT succeeds at λ = 0.3 \lambda= 0.3 λ = 0.3 and 0.7 0.7 0.7 ; the poor-performing 0.9 0.9 0.9 and 1 1 1 settings are also retained. (b) Normalized next-state distances on Hopper (median ≈ 0.36 \approx0.36 ≈ 0.36 , 90th percentile ≈ 0.64 \approx0.64 ≈ 0.64 ), used as descriptive dynamics diagnostics; they do not certify the surrogate contraction condition of Theorem B.9 .
Supplementary Five-Task DMC Evaluation
This supplementary evaluation uses the distinct conditions in Table 13 . Its original SAC-multistep label does not by itself identify the current pure ten-step baseline.
Setup. The tasks are CartPole-Swingup-Sparse, Hopper-Stand, Finger-Turn-Easy, Acrobot-Swingup-Sparse, and Ball-in-Cup-Catch. Table 24 reports three-seed, 1M-step conditions, averaging the final 100K steps. A separate five-seed CartPole-Swingup-Sparse re-run at L = 10 L = 10 L = 10 gives 617 ± 113 617 \pm113 617 ± 113 ; both its cap and seed set differ from the table’s L = 5 L = 5 L = 5 condition. The five-seed curves in Figure 31 retain their original curve protocol.
Table 24. DMControl ablation. Mean evaluation return at 1M environment steps (mean₁₀ ± std, 3 seeds). Bold marks the best per row. SAC-MINT improves over uncorrected multi-step on CartPole and Hopper; both learn on Finger, saturate on Ball-in-Cup, and fail on the tested Acrobot setting.
Figure 31. Supplementary DMC learning curves. Five seeds, 95% bootstrap intervals, 1M steps. The Hopper-Stand one-step reference is its final return. These curves retain their original settings, separately from the three-seed table and the CartPole re-run (Table 13 and Table 24 ).
Task-specific outcomes. MINT improves on the original multistep baseline on CartPole and Hopper-Stand, while Finger-Turn-Easy favors that baseline. Acrobot-Swingup-Sparse remains unsolved and Ball-in-Cup-Catch saturates for both reported methods. These comparisons do not isolate a bias–variance mechanism or imply that backup adaptation solves reward discovery.
Fixed and Automatic Temperature on CartPole-Swingup-Sparse
The CartPole-Swingup-Sparse comparison uses cap L = 10 L = 10 L = 10 (Figure 32 ). Fixed α = 0.2 \alpha= 0.2 α = 0.2 is reported with a diffuse policy before reward, K ^ ≈ 2 \hat{K} \approx2 K ^ ≈ 2 , and failed learning. Under automatic tuning, reward is observed, α \alpha α later settles at 0.06–0.10, and K ^ \hat{K} K ^ approaches 5.7 with successful learning. Entropy alone does not determine the target–behavior discrepancy or stopping law. The supplementary evaluation also reports this regime on Hopper-Stand and early α \alpha α collapse before reward on sparse Acrobot; these observations do not isolate the cause of that failure.
Atari 100k: Primary Results and Distributional Limits
Scope and protocol. The primary result uses the DrQ(ε \varepsilon ε )-based MINT configuration with λ = 0.9 \lambda= 0.9 λ = 0.9 , cap length 10, and five seeds on each of 26 games (130 runs). Only λ = 0.7 \lambda= 0.7 λ = 0.7 is retained as an additional ablation (seeds 5–9, versus primary seeds 0–4). Published DrQ(ε \varepsilon ε ) 100-run references are used. The full-suite aggregate is shown in the main text (Figure 8 ); the numerical table below supports that comparison. Primary MINT uses hard target updates every 100 gradient steps and target clipping to [ − 100 , 100 ] [-100, 100] [ − 100 , 100 ] . These changes accompany truncation and λ \lambda λ weighting, so their effects are not isolated. The Atari target removes intermediate V − Q V-Q V − Q corrections and changes target updates and clipping; it is a modified truncated λ \lambda λ -return, so the original TD-error operator’s fixed-point guarantee is not asserted for it.
TD-error targets and target-update sensitivity. Direct application of the correction-preserving TD-error target learned poorly on Breakout, Qbert, and Seaquest. Frequent target copying was accompanied by large critic values; increasing the copy period to 100 gradient steps did not resolve every tested setting. This motivated the stabilized variant specified above. We suspect that frequent target updates in this DrQ backbone amplify feedback through the TD(λ \lambda λ ) target, with residual sensitivity even at period 100. This is a hypothesis about the backbone–target interaction: the diagnostics do not distinguish overfitting from other bootstrapping instability, and changing the copy period does not uniformly improve the unmodified DrQ backbone. An isolated test would vary the period while holding the return, clipping, and update budget fixed, and compare training-replay errors with held-out episode and Monte-Carlo errors.
Full-suite aggregate and distributional performance
Tables 25 and 26 provide broader benchmark context [3 , 37 ] . MINT exceeds DER’s IQM point reference, but is below the SPR, EfficientZero, and BBF point references in full-suite IQM. Its DrQ-relative strengths must therefore be read as task-dependent benefits rather than a claim of leading Atari 100k performance. Runs, algorithms, and evaluation details differ; these references are not matched target ablations. The primary DrQ comparison elsewhere uses the 100-run table. The 10-run figure-derived references below are kept separate and approximate; their interval endpoints are not recomputed.
Run-level coverage and comparison scope. Figure 33 shows the pooled game/run distribution. At HNS 0.1, the reported fractions are MINT 0.585, approximate 10-run DrQ 0.692, and BBF 0.900; at HNS 1.0 they are 0.192, 0.135, and 0.445. These are run fractions, whereas Figure 36 counts game means. The lower-threshold deficit and stronger upper tail against DrQ are consistent across these two views, without distributional dominance. The reported probability of improvement against
Figure 33. Pooled-run Atari performance profile. Fraction of game/run scores above HNS threshold, with pointwise 95% intervals. MINT: primary, five runs/game; BBF: 50 author runs/game. DrQ is the approximate 10-run profile recovered from the rliable paper, not the 100-run game-mean reference used in the group analysis. Lines connect the 201 evaluated thresholds without smoothing or additional interpolation; bands report the archived pointwise intervals. This differs from the game-mean profile in Figure 36 .
Figure 36. Atari game-mean profile and IQM progression. Left: fraction of 26 game means above HNS threshold τ \tau τ ; MINT averages five seeds/game and baselines use published game means. This is not a pooled-run profile. Right: MINT IQM by nominal training step (five evaluation episodes at 10K–90K, 100 at the endpoint); DrQ/DER horizontal lines are final-IQM references, not baseline learning curves. Bands show pointwise bootstrap intervals for MINT; final-reference comparisons do not establish a learning-speed advantage.
Table 26: Full-suite estimates and uncertainty. Point estimate [95% stratified-bootstrap interval] from the archived aggregate results. Entries use numerical interval endpoints, which may differ slightly from those in the figures. BBF is published context, not a matched comparison.
Table 26.
BBF is 0.0544 [0.0395, 0.0692] (26 games); no DrQ probability is inferred without baseline run data. Neither the low BBF probability nor a positive DrQ group mean is a causal test of truncation.
Evaluation-time performance. Table 27 supplements the IQM curve with all four aggregate metrics. The nominal 10K–90K checkpoints use five evaluation episodes, whereas the final checkpoint uses 100. These checkpoints are evaluations of MINT, not a matched DrQ learning curve or a wall-clock comparison. They should not be read as a test of uniform improvement between adjacent checkpoints.
Table 27. Primary MINT evaluation checkpoints. Point estimate [95% interval]; 26 games and five training seeds. The larger final evaluation budget differs from the intermediate budget.
Task-group performance
Group benefits and concentration. We apply the rough exploration taxonomy of Bellemare et al. [6 Appendix C.2] to the 26 games: easy/human-optimal (11), easy/score-exploit (6), hard/dense (7), and hard/sparse (2). Group mean HNS is MINT/DrQ 0.689/0.463, 0.741/0.875, 0.077/0.151, and 0.372/0.343, respectively. These are comparisons against published game means, without baseline-run uncertainty. In that taxonomy, easy/hard concerns whether local exploration can find a high-scoring policy; the hard class is then divided by reward density. Reward density and exploration difficulty are different axes, and these historical labels do not certify coverage in a 100K run. Human-optimal is a category name, not a statement that this agent reaches human score; score-exploit does not mean that MINT learned an exploit. The sparse group contains only two games and PrivateEye remains near random.
Figure 34. Atari task groups under Bellemare et al. [6 ] . E/H: easy/human-optimal; E/X: easy/score-exploit; H/D: hard/dense; H/S: hard/sparse (11/6/7/2 games). (a) Equal-game mean HNS. (b) Game-mean differences and group means against published DrQ(ε \varepsilon ε ). No baseline-run confidence intervals or subgroup pooled-run IQMs are inferred.
Group performance beyond the mean. Table 28 and Figure 35 add median and pooled-run IQM to the mean-based view. Easy/human-optimal has MINT/DrQ mean 0.689/0.463 and median of game means 0.669/0.429, with 7/11 game means higher. Thus the positive point-reference comparison is visible in the group’s median as well as its mean. The MINT pooled-run IQM is 0.521, but DrQ’s group IQM cannot be recovered from its game means. The remaining seven games after removing the four largest gains still have a negative average difference: group central improvement and concentrated contributions are distinct observations.
Easy/score-exploit is weaker in both mean (0.741/0.875) and median (0.398/0.423), with 2/6 higher game means. Hard/dense is lower in both (0.077/0.151 mean; 0.075/0.124 median), with only 1/7 higher. Hard/sparse has mean and median 0.372/0.343, but consists of just Freeway and PrivateEye; seed uncertainty does not measure sensitivity to choosing these two tasks. These are descriptive group effects against published references, not evidence of significance or a learned exploration mechanism.
Table 29. Where the Atari mean difference comes from. Equal-game arithmetic decomposition against rounded published game means, without a confidence interval for the difference.
The four largest-gain games are DemonAttack, Boxing, Assault, and CrazyClimber. Together they contribute +0.118419 to the full-suite mean difference; the other 22 contribute −0.071307. The remaining seven easy/human games have mean difference −0.085. Eighteen of the twenty MINT runs from those four games lie in the upper tail excluded by the full-suite IQM (32 lowest and 32 highest scores removed, 66 retained). Thus large game-specific gains raise mean HNS while leaving lower median and IQM point estimates. IQM describes central performance rather than failing to reflect an improvement. This does not establish a uniform distributional advantage or a general exploration improvement.
At HNS 0.1, MINT exceeds the threshold in 16/26 games versus 20/26 for DrQ; at HNS 1.0, the ordering reverses to 5/26 versus 3/26. This crossing profile is consistent with stronger upper-tail outcomes and weaker low-threshold coverage.
Interpreting group effects and realized horizons
Backup design under shared exploration. The variant retains the DrQ backbone’s ε \varepsilon ε -greedy schedule and adds no directed-exploration objective. This makes backup design under standard exploration the relevant empirical question. Identical exploration schedules do not ensure identical state-visitation or replay distributions: changes in value estimates alter greedy actions, state visitation, and replay contents. Consequently, the present online comparison does not isolate learning from identical experience. Nor can a common exploration limitation explain the relative losses by itself, since the published DrQ reference scores higher on many of the same games.
Task structure, hypotheses, and counterexamples. Seven of eleven easy/human-optimal game means improve, but four games account for the large positive contribution and the other seven have a negative average difference. Six of seven hard/dense games have lower means; MsPacman’s small positive difference is the exception. Thus neither easy exploration nor dense reward alone determines benefit. We hypothesize that agreement-driven truncation can help when useful reward experiences are repeatedly available, by changing how observed rewards and bootstrapped values enter the target. Scores alone do not measure reward frequency, the fraction of rewards retained before stopping, or bootstrap accuracy, so this remains a mechanism hypothesis.
Pong is an easy/human-optimal counterexample: its mean HNS is lower by 0.169 against the published game reference. Qbert is lower by 0.067. The sparse group’s small positive mean difference combines Freeway’s +0.057 with PrivateEye’s near-zero score; it does not show that exploration bottlenecks were solved. Game-level differences use published DrQ means, not baseline-run uncertainty. As emphasized by Badia et al. [5 ] , finding rewards and assigning credit over long delays are separate challenges. Their Skiing/Solaris examples are absent from this 26-game suite; our group results do not establish an advantage on those long-delay tasks.
Reward retention versus bootstrapping. For a realized length K K K , shared state-value evaluations, and before clipping, the correction-free target is a truncated mixture of n n n -step returns. Changing K K K and λ \lambda λ changes both which observed rewards contribute and where value estimates are used. Shorter traces can reduce reliance on distant replay segments while increasing reliance on nearby bootstrap estimates; neither effect is guaranteed to help. Agreement with a greedy action does not certify that reward has been reached or that the bootstrap value is accurate. Target-update delay and clipping also change, preventing attribution of score differences to stopping alone. Direct tests would hold replay, target updates, clipping, and value evaluations fixed while comparing fixed-cap λ \lambda λ mixtures and trajectory-independent matched-length or shuffled-length controls, together with reward-inclusion and bootstrap-error measurements. These measurements have not been made here.
Agreement and realized length. Mean K K K falls from 6.20 to 5.05 while GreedyMatch rises from 14.1% to 28.1%; final five-seed game means have rank correlation − 0.932 -0.932 − 0.932 . Both short and long lengths include unsuccessful games. GreedyMatch is the fraction of matching behavior and current greedy (argmax) actions, rather than the survival-conditioned agreement probability of the theory. These are descriptive relationships, not an optimal-horizon certificate.
The first-to-final nominal checkpoints are 10K and 100K interactions. Final means range from Freeway’s 1.83 to PrivateEye’s 7.33. Assault (3.74) and DemonAttack (3.92) improve with short traces, while Pong (4.32) and Qbert (4.28) lose; BankHeist (6.33) and PrivateEye retain long traces without a corresponding gain. Thus trace length alone does not predict benefit. The logged mean also does not determine the discounted trace mass E [ ∑ t = 0 K − 1 ( γ λ ) t ] \mathbb{E}[\sum_{t=0}^{K-1}(\gamma\lambda)^t] E [ ∑ t = 0 K − 1 ( γ λ ) t ] , which depends on the length distribution. It is neither a mixing-time estimate nor evidence of optimal credit assignment.
Additional λ = 0.7 \lambda= 0.7 λ = 0.7 Ablation
The additional variant has mean HNS 0.5098 and IQM 0.2115, versus primary 0.5118 and 0.2509. Its final mean K K K is longer (5.287 versus 5.052), while GreedyMatch is lower (24.37% versus 28.08%). Eleven game means rise and fifteen fall; probability of improvement is 0.4508 [0.3846, 0.5162]. For fixed policies and a fixed trajectory, the fifth-power TD-weight ratio is ( 0.7 / 0.9 ) 5 ≈ 0.285 (0.7/0.9)^5 \approx0.285 ( 0.7/0.9 ) 5 ≈ 0.285 ; this identity is not an explanation of the online score difference. The TD(λ \lambda λ ) target structure, target-update feedback, and learned policies interact, and the seed sets also differ. This comparison therefore does not isolate either the effect of realized horizon or the proposed target-update mechanism (Appendix § 8.6 .§ 3 ).
Figure 38. Additional Atari λ \lambda λ ablation. λ = 0.9 \lambda= 0.9 λ = 0.9 (seeds 0–4) versus 0.7 (seeds 5–9). IQM uses filled 95% bootstrap confidence-interval bars and black estimate ticks. Other panels retain horizon and game-group HNS differences. This is supplementary to the primary 0.9 result.
Table 30. Table 30: Primary Atari performance by game and task group. MINT raw return: mean ± \pm ± standard deviation over five seeds; DrQ return: published point reference. The final columns are human-normalized game means and their difference. Standard deviations are not confidence intervals; no uncertainty for the difference is inferred. Baseline protocols are not matched ablations. Values are rounded, including PrivateEye’s near-zero HNS.
Cap, Signal, and Contraction in the Deep-RL Regime
This appendix examines how the cap and disagreement signal affect the effective horizon (Appendix § 8.12 .§ 1 ), compares the action-level surrogate with a one-step ϵ \epsilon ϵ -tolerant matching diagnostic (Appendix § 8.12 .§ 2 ), and probes empirical backup gains under stochastic dynamics (Appendix § 8.12 .§ 3 ). The separate MuJoCo λ \lambda λ sweep is reported in Section F.1.2.
Common setup. We use RWRL walker_walk [11 ] , the Real-World RL Challenge variant of the DeepMind Control task, with a SAC backbone. RWRL provides a benchmark-defined actuator-noise channel: Gaussian noise of scale σ ∈ { 0.0 , 0.1 , 0.3 } \sigma\in\{0.0, 0.1, 0.3\} σ ∈ { 0.0 , 0.1 , 0.3 } is added to the executed action inside the environment, so that P ( s ′ ∣ s , a ) \mathcal{P}(s' \mid s,a) P ( s ′ ∣ s , a ) is non-degenerate at a specified level; we use this published mechanism rather than a perturbation of our own. For the fixed-α \alpha α 200K and auto-α \alpha α 780K training arms in Appendix § 8.12 .§ 1 , the backbone follows the main table (λ = 0.7 \lambda= 0.7 λ = 0.7 , γ = 0.99 \gamma= 0.99 γ = 0.99 , Table 10 ), with caps K max ∈ { 5 , 10 , 50 } K_{\max} \in\{5, 10, 50\} K m a x ∈ { 5 , 10 , 50 } . The fixed-checkpoint stopping diagnostic uses a separate auto-α \alpha α 1M protocol with λ = 0.9 \lambda= 0.9 λ = 0.9 (Section G.1.1). The backup-gain probes use the separate protocol stated in Appendix § 8.12 .§ 3 . The retained-horizon studies have two arms. The fixed-α \alpha α arm uses α = 0.05 \alpha= 0.05 α = 0.05 , 3 seeds, and 200k environment steps. With α = 0.2 \alpha= 0.2 α = 0.2 , the main MuJoCo setting, the observed return on this DMControl-derived task is approximately 27 after 80k steps. Its per-step reward is roughly 3.5 × 3.5\times 3.5 × smaller than Hopper’s, so α \alpha α is rescaled to the reward scale. The eleven-task DMC comparison instead uses automatic entropy tuning (Appendix § 8.6 .§ 2 ). The auto-α \alpha α arm learns α \alpha α and runs 5 seeds per ( σ , K max ) (\sigma, K_{\max}) ( σ , K m a x ) cell (15 trials per cap) for 780k steps. These runs are a K e f f K_{\mathrm{eff}} K eff diagnostic on MINT itself—the only baseline included is uncorrected n n n -step at the same cap—not a re-run of the main comparison. All statistics are read at a common checkpoint for every trial.
Table 10. SAC Hyperparameters (MuJoCo main experiments). The entropy coefficient is fixed at α = 0.2 \alpha=0.2 α = 0.2 (the SpinningUp default) for MINT and all multi-step baselines; no automatic entropy tuning is used in these runs. The DMC ablation and RWRL experiments use their separately stated entropy settings; the fixed-α \alpha α statement here is restricted to the main MuJoCo SAC comparison.
What is and is not measured. The diagnostics measure retained horizons and one-step matching, with the following scope. (i) We do not measure the multi-step state-coupling meeting time T β , π T_{\beta,\pi} T β , π of Definition 3.1 on continuous control, nor a correlation between it and K ^ \hat{K} K ^ . The measured state-level object is a calibrated one-step ϵ \epsilon ϵ -tolerant matching diagnostic, denoted m ^ ϵ ( s ) \hat{m}_{\epsilon}(s) m ^ ϵ ( s ) and compared with action-level TV in Appendix § 8.12 .§ 2 ; it is not identified with exact kernel TV d ( s ) d(s) d ( s ) ; the retained-horizon and early-stopping evidence is summarized in the main text (Table 2 ). (ii) We do not train an uncapped (K max = ∞ K_{\max} = \infty K m a x = ∞ ) arm; the largest training cap is 50. We separately measure fixed-checkpoint surrogate stopping without this cap, with right censoring at a verified rollout horizon of 1000 (Section G.1.1). Neither substitute establishes the multi-step coupling guarantee for the deployed estimator; they provide evidence about the selected horizon and an empirical association between action-level TV and the matching diagnostic.
Cap-Length Sensitivity and Horizon Diagnostics
The MuJoCo Hopper sweep in Figure 1 b tests K max ∈ { 3 , 5 , 10 , 20 } K_{\max} \in\{3, 5, 10, 20\} K m a x ∈ { 3 , 5 , 10 , 20 } over 1M environment steps. MINT has limited return sensitivity across these caps, while Peng’s Q ( λ ) Q(\lambda) Q ( λ ) and uncorrected n n n -step vary more. The RWRL studies below examine both early stopping and return sensitivity under distinct entropy settings; the controlled toy-chain experiment separately compares capped and uncapped sampled backups (Appendix § 8.4 .§ 5 ).
Table 2 in the main text reports the fixed-α \alpha α retained horizons and early-stopping fractions. Here we give the corresponding normalized horizons and mean keep probabilities (Table 31 ), followed by the separate auto-α \alpha α study.
Table 31. Additional diagnostics for the fixed-α \alpha α RWRL comparison in Table 2 : α = 0.05 \alpha= 0.05 α = 0.05 , 200k steps, three seeds, noise levels pooled.
Auto-α \alpha α diagnostic: the stopping distribution needs room. The auto-α \alpha α arm exhibits cap tracking (Table 32 ). For K max = 50 K_{\max} = 50 K m a x = 50 , K e f f K_{\mathrm{eff}} K eff rises during warm-up, peaks near 100k steps, and declines to 25% below its peak by 780k steps, with three quarters of traces stopping early. For K max ∈ { 5 , 10 } K_{\max} \in\{5, 10\} K m a x ∈ { 5 , 10 } it settles at ≈ \approx ≈ 97% of the cap. The learned α \alpha α converges to 0.002–0.014 in every arm (last column), accompanied by a concentrated target policy, reported log-ratios near − 38 -38 − 38 , and mean keep probabilities near 0.98 0.98 0.98 . Such small ratios make the clipped agreement weight negligible and continuation likely. At the same cap of 50, the separate fixed-α \alpha α arm has a lower mean keep probability (0.70 versus 0.98) and a shorter effective horizon (approximately 5 versus 21.9). These observations associate entropy settings with surrogate survival under the tested protocols; they do not isolate temperature as the cause or measure a state-coupling meeting time. We pool over σ \sigma σ because its effect on K e f f K_{\mathrm{eff}} K eff was smaller than the run-to-run spread, so we do not claim a trend with the noise level. A repeat of the 45-trial grid on different hardware with the same seeds reproduced every cell to within 0.1 in K e f f K_{\mathrm{eff}} K eff and 0.6 percentage points in the early-stopping fraction.
Table 32. Auto-α \alpha α arm: K e f f K_{\mathrm{eff}} K eff and Pr [ K ^ < K max ] \Pr[\hat{K} < K_{\max}] Pr [ K ^ < K m a x ] at 20k → 100k → 780k steps (15 trials per cap, σ \sigma σ pooled), and the learned α \alpha α at the end of training (5-seed mean).
Return sensitivity to cap length. Sweeping K max K_{\max} K m a x quantifies return reductions at smaller caps (Table 34 ). Reducing K max K_{\max} K m a x from 50 to 10 lowers K e f f K_{\mathrm{eff}} K eff by 56% but final return by only 7%; reducing it from 10 to 5 halves K e f f K_{\mathrm{eff}} K eff again and lowers return by 60%. These results show that the cap can remain consequential for the implemented surrogate. The fixed-α \alpha α arm shows the same direction at its two caps (603 ± \pm ± 110 at K max = 50 K_{\max}=50 K m a x = 50 versus 25 ± \pm ± 3 at K max = 5 K_{\max}=5 K m a x = 5 ; 200k steps, 3 seeds), so this sensitivity also occurs without automatic entropy tuning. These training comparisons change the learned policy and its exploration/replay distribution together with the cap; they do not isolate a target-only effect of cap length. Fixed-temperature learning is particularly limited at cap 5, while cap 50 attains a substantial intermediate return at the shorter 200K budget. The fixed and automatic-temperature arms also differ in duration and seed count. Together with the Hopper sweep, they support using a cap large enough to accommodate the stopping regime, rather than selecting one exact horizon for every task. The common cap of 5 in the main MuJoCo comparison remains a matched budget.
Table 34. Return as a function of the cap (auto-α \alpha α arm, 780k steps, 15 trials per cell; the only experiment with a K max = 10 K_{\max}=10 K m a x = 10 condition).
Table 33 breaks the K max ∈ { 10 , 50 } K_{\max} \in\{10,50\} K m a x ∈ { 10 , 50 } returns down by noise level: the agent learns a strong policy at every σ \sigma σ and performance is stable up to σ = 0.3 \sigma=0.3 σ = 0.3 , with the gain from K max = 10 K_{\max}=10 K m a x = 10 to 50 marginal, so the cap effect is not driven by any single noise level.
Table 33. Eval return on RWRL walker_walk by transition-noise level (auto-α \alpha α arm, 780k steps, mean ± \pm ± s.d. over 5 seeds; max return 1000).
Stopping Distributions across Behavior-Policy Ages
This fixed-checkpoint diagnostic uses RWRL walker_walk, Gaussian actuator noise σ ∈ { 0.1 , 0.3 } \sigma\in\{0.1,0.3\} σ ∈ { 0.1 , 0.3 } , and no observation noise. Policies were trained with automatic entropy tuning, automatic target entropy, λ = 0.9 \lambda=0.9 λ = 0.9 , and cap/window length 50. We fix τ \tau τ at 1M and separately evaluate β \beta β at 900K, 500K, and 100K: behavior ages 100K, 500K, and 900K. The target checkpoint’s τ \tau τ ranges from 0.004. to 0.009 across seeds. Each condition uses seeds 0–2, 40 states/seed sampled every 25 steps along a behavior rollout, and 256 indicator draws/state (30,720 draws). These draws share realized behavior trajectories; the independent training repeats are the three seeds. Different β \beta β conditions also change visited states, so this is not an intervention on age alone at identical states.
At each realized action we sample the indicator in Equation (7) , retain the anchor, and record T ^ = inf { i ≥ 1 : A ^ i = 0 } \hat{T} = \inf\{i \ge1 : \hat{A}_i = 0\} T ^ = inf { i ≥ 1 : A ^ i = 0 } without imposing the training cap. The plotted T T T is this action-level stopping time, not a state-coupling meeting time. Simulator state and episode clocks are restored for each measurement. Every age-sweep rollout records length 1000, time-limit truncation, and no true terminal; draws still surviving at this horizon are right-censored. Only measurements with complete episode-clock restoration are included.
Figure 39. Recent-behavior stopping distribution. Auto-α \alpha α , λ = 0.9 \lambda= 0.9 λ = 0.9 ; π \pi π at 1M, β \beta β at 900K, training cap 50. Panels pool three seeds, 40 states/seed, and 256 draws/state. Dashed caps 5/10/50 are applied to the same stopping draws. Means are 7.06/10.56, medians 5.5/6, and maxima 56/75 at σ = 0.1 / 0.3 \sigma= 0.1/0.3 σ = 0.1/0.3 ; neither condition has censored draws. Horizontal “meeting time” T T T denotes the surrogate stopping time T ^ \hat{T} T ^ .
For recent β \beta β , exactly 2/30,720 and 179/30,720 draws exceed 50 (0.007%/0.583%); another 0/55 stop exactly at 50. Thus Pr ( T ^ ≥ 50 ) \Pr(\hat{T} \ge50) Pr ( T ^ ≥ 50 ) is 0.007%/0.762%, distinct from strict cap cutting, Pr ( T ^ > 50 ) \Pr(\hat{T} > 50) Pr ( T ^ > 50 ) . Cap 50 therefore cuts fewer than 0.6% of draws in both recent-policy conditions.
Older behavior policies and cap intervention. Figure 40 and Table 35 show longer tails at older β \beta β . All caps below are applied after measurement to fixed policies and shared draws; these are not separately trained cap arms. Writing K L = min ( L , T ^ ) K_L = \min(L,\hat{T}) K L = min ( L , T ^ ) , the integer-time identity
Table 35. Behavior age and cap-50 intervention. Pooled draw summaries, fixed π \pi π at 1M. Values marked ≥ \ge ≥ are lower bounds obtained by recording censored draws at 1000. For L ≤ 100 L \le100 L ≤ 100 , E [ K L ] \mathbb{E}[K_L] E [ K L ] and Pr ( T ^ > L ) \Pr(\hat{T} > L) Pr ( T ^ > L ) are fully observed empirical summaries despite tail censoring. Percentages are descriptive, without an independent-draw confidence claim.
E [ K L + 1 ] − E [ K L ] = Pr ( T ^ > L ) \mathbb{E}[K_{L+1}] - \mathbb{E}[K_L] = \Pr(\hat{T} > L) E [ K L + 1 ] − E [ K L ] = Pr ( T ^ > L )
connects the survival curve directly to the marginal effect of increasing the cap.
Increasing the cap from 50 to 100 changes mean length by approximately 0/0.04 at the reported precision in the recent conditions, 0.71/2.66 at age 500K, and 37.10/34.42 at age 900K. The oldest conditions’ mean T ^ \hat{T} T ^ remains only lower-bounded, while their cap-50/100 summaries are observed. Seed variation is substantial: at σ = 0.1 \sigma= 0.1 σ = 0.1 , age 900K, all seed-0 draws are censored, and the per-seed horizon-truncated means are 1000.0/167.1/174.6. At age 500K, per-seed means are 20.9/8.9/15.0 and 12.3/42.7/12.2 for σ = 0.1 / 0.3 \sigma= 0.1/0.3 σ = 0.1/0.3 .
The same trials’ 1M training logs report mean K ^ = 19.6 \hat{K} = 19.6 K ^ = 19.6 –23.4 23.4 23.4 and cap-reach rates 19–27%. Replay mixes collection-time behavior ages of approximately 0–990K; none of the single-checkpoint conditions reconstructs that mixture, and the replay was not saved to verify mixture weights. The age sweep supports behavior-dependent cap margin, without proving that age alone explains the training gap or that a stopping distribution improves return or critic accuracy. It is separate from the fixed-α \alpha α 200K comparison in Table 2 and the λ = 0.7 \lambda= 0.7 λ = 0.7 , 780K training sweep above.
The Action-Level Surrogate and a One-Step Matching Diagnostic
Proposition A.4 gives the exact one-step relation E β [ p ^ i ∣ s ] = D s ≥ d ( s ) \mathbb{E}_{\beta}[\hat{p}_i \mid s] = D_s \ge d(s) E β [ p ^ i ∣ s ] = D s ≥ d ( s ) , where d ( s ) = TV ( P β ( ⋅ ∣ s ) , P π ( ⋅ ∣ s ) ) d(s) = \operatorname{TV}(P^\beta(\cdot\mid s), P^\pi(\cdot\mid s)) d ( s ) = TV ( P β ( ⋅ ∣ s ) , P π ( ⋅ ∣ s )) . RRWRL’s action-noise channel factorizes as a ↦ a ~ = clip ( a + ϵ ) ↦ s ′ = F ( s , a ~ ) a \mapsto\tilde{a} = \operatorname{clip}(a+\epsilon) \mapsto s' = F(s,\tilde{a}) a ↦ a ~ = clip ( a + ϵ ) ↦ s ′ = F ( s , a ~ ) , with the same Markov maps for both policies, so the relation applies to the exact resulting distributions. Closeness of next-state coordinates alone does not establish TV overlap.
On the K max = 50 K_{\max} = 50 K m a x = 50 auto-α \alpha α arm (seed 0), action-level D s D_s D s is evaluated by numerical quadrature of the two Gaussian policies. The simulator-based comparison uses a calibrated ϵ \epsilon ϵ -tolerant matching statistic m ^ ϵ ( s ) \hat{m}_{\epsilon}(s) m ^ ϵ ( s ) , with a β = π \beta= \pi β = π calibration control, at 250 replay states across four checkpoints. We report an ϵ \epsilon ϵ -tolerant matching diagnostic, rather than an estimate of exact kernel TV. Its observed ordering relative to action-level TV is an empirical diagnostic comparison, not a verification of DPI.
The measured ordering E β [ p ^ ] ≥ m ^ ϵ ( s ) \mathbb{E}_{\beta}[\hat{p}] \ge\hat{m}_{\epsilon}(s) E β [ p ^ ] ≥ m ^ ϵ ( s ) holds at all 250 retained states. The reported Spearman correlations are 0.95 across the arm/checkpoint cells and 0.86 across states (Table 36 ), with level gaps 0.23–0.40. The diagnostic excludes states whose calibration floor exceeds 2%, so this selection may affect the observed correlations and gaps. These are directional measurements on one task and one seed; they neither identify d ( s ) d(s) d ( s ) nor order multi-step stopping times or certify the χ λ \chi_{\lambda} χ λ contraction condition.
Table 36. Action-level TV versus the ϵ \epsilon ϵ -tolerant matching diagnostic on RWRL walker_walk (K max = 50 K_{\max}=50 K m a x = 50 arm, seed 0) at four checkpoints. E β [ p ^ ] ≥ m ^ ϵ ( s ) \mathbb{E}_{\beta}[\hat{p}] \ge\hat{m}_{\epsilon}(s) E β [ p ^ ] ≥ m ^ ϵ ( s ) at all 250 retained states; reported Spearman correlations are 0.95 across cells and 0.86 across states. The matching statistic is not an estimate of exact kernel TV.
Empirical Gain of the Deployed Operator
The ϵ \epsilon ϵ -tolerant matching diagnostic does not certify the sufficient condition of Theorem B.9 . We therefore probe the implemented backup empirically. Subtracting backups applied to two value functions cancels the reward and entropy terms and leaves a linear map on Δ = Q − Q ′ \Delta=Q-Q' Δ = Q − Q ′ ; its measured gain over the visited distribution is
κ ^ = ∥ T ^ Q − T ^ Q ′ ∥ ∥ Q − Q ′ ∥ \hat{\kappa}=\frac{\lVert\hat{T}Q-\hat{T}Q'\rVert}{\lVert Q-Q'\rVert} κ ^ = ∥ Q − Q ′ ∥ ∥ T ^ Q − T ^ Q ′ ∥
which we estimate by fitted power iteration over the visited distribution. The probes use the action-level surrogate on windows of length L = 10 L=10 L = 10 , with retained length K ^ p r o b e : = min ( T ^ β , π , K max , L ) \hat{K}_{\mathrm{probe}}:=\min(\hat{T}_{\beta,\pi},K_{\max},L) K ^ probe := min ( T ^ β , π , K m a x , L ) . Thus the nominal caps 5 and 50 give effective probe caps of 5 and 10, respectively. It is a fitted, visited-distribution diagnostic composed with critic projection. It is neither the mixed-norm state-level bound κ ⋆ \kappa^{\star} κ ⋆ of Corollary C.3 nor the global surrogate upper bound κ χ , L \kappa_{\chi,L} κ χ , L of Theorem B.9 .
Setup. The backup-gain probes use a separate auto-α \alpha α protocol on RWRL walker_walk with injected transition noise (σ ∈ { 0.1 , 0.3 } \sigma\in\{0.1,0.3\} σ ∈ { 0.1 , 0.3 } ), 100k training steps, 3 seeds, and 60 probes across training checkpoints. We use this arm because p ^ i \hat{p}_i p ^ i saturates there and many traces reach the effective probe cap. The policy pairs were trained at ( λ , γ ) = ( 0.9 , 0.999 ) (\lambda,\gamma)=(0.9,0.999) ( λ , γ ) = ( 0.9 , 0.999 ) . During the coefficient comparisons, these policy pairs and the sampled trajectories are held fixed while the backup coefficients are changed to the main-experiment values or the training values. The transition operator is evaluated on shared samples, so no density model of the kernel is needed.
Results. At the deployed coefficients (λ = 0.7 \lambda= 0.7 λ = 0.7 , γ = 0.99 \gamma= 0.99 γ = 0.99 , K max = 5 K_{\max} = 5 K m a x = 5 ), the mean empirical gain is κ ^ = 0.84 ± 0.13 \hat{\kappa} = 0.84 \pm0.13 κ ^ = 0.84 ± 0.13 , below 1 on 90% of probes (Table 37 , middle block), at checkpoints where the trace reaches the cap 88% of the time. The effective probe cap of 5 has a smaller measured gain than the effective cap of 10. The same policies at ( λ , γ ) = ( 0.9 , 0.999 ) (\lambda, \gamma) = (0.9, 0.999) ( λ , γ ) = ( 0.9 , 0.999 ) give κ ^ = 1.23 \hat{\kappa} = 1.23 κ ^ = 1.23 , below 1 on 27% of probes. These comparisons distinguish observed probe behavior without certifying a global operator norm. The backup-difference implementation is checked by a unit test; at λ = 0 \lambda= 0 λ = 0 , the measured gain is within about 2% of the one-step Bellman reference γ \gamma γ (bottom block).
Table 37. Empirical backup gain κ ^ \hat{\kappa} κ ^ on RWRL walker_walk with transition noise. Top: operating point (pooled over cap, noise, seed). Middle: deployed coefficients by nominal cap. Bottom: comparison at λ = 0 \lambda=0 λ = 0 with the one-step Bellman reference γ \gamma γ . Probes use windows of length L = 10 L=10 L = 10 , so the K max = 50 K_{\max}=50 K m a x = 50 row has an effective cap of 10. These are fitted probe measurements, not certificates of the global surrogate sufficient condition.
Scope. κ ^ \hat{\kappa} κ ^ uses a finite anchor sample, includes critic projection, and comes from one task. The dominant reported error is the power-iteration fit (median relative error ≈ 14 % \approx14\% ≈ 14% ), not captured by the seed s.d. These probes are empirical stability diagnostics rather than estimates of κ χ , L \kappa_{\chi,L} κ χ , L or proofs of convergence for the deep-RL runs. We do not claim a trend with training progress or noise level.
Stochastic Dynamics and Stitching: Off-Policy Evaluation on OGBench
MINT’s central claim concerns the multi-step target, and its fixed point is Q π Q^\pi Q π (Lemma 3.3 ). We therefore also evaluate it as a pure off-policy evaluation operator . Fixing the target policy removes the control-loop confounds (entropy tuning, exploration, replay non-stationarity) and provides a finite-rollout Monte Carlo reference for Q π Q^\pi Q π , so target errors are measured against that reference rather than inferred from returns. The experiment examines stochastic dynamics and datasets composed of shorter trajectories.
Setup. We use OGBench AntMaze [33 ] , whose observation is the full simulator state, so any dataset state can be restored and the target policy rolled out from it. The grid crosses stochasticity against stitching: antmaze-large (deterministic) and antmaze-teleport (stochastic), each with the navigate (1000-step trajectories) and stitch (200-step trajectories) datasets. The stochasticity is the benchmark’s own: entering a teleporter moves the agent to one of three exits uniformly at random, a discrete state jump that makes the kernel multimodal, unlike the unimodal action noise of Appendix § 8.12 . We use γ = 0.99 \gamma= 0.99 γ = 0.99 , window length L = K max = 50 L = K_{\max} = 50 L = K m a x = 50 , λ = 0.9 \lambda= 0.9 λ = 0.9 , and OGBench’s single-task reward relabeling. The behavior policy β \beta β is a state-conditioned Gaussian fit to each dataset. The target policy displaces its mean by h h h behavior standard deviations along a fixed unit direction d d d in action space, μ π ( s ) = μ β ( s ) + h σ β ( s ) d \mu_\pi(s) = \mu_\beta(s) + h\,\sigma_\beta(s)\,d μ π ( s ) = μ β ( s ) + h σ β ( s ) d with σ π = σ β \sigma_\pi= \sigma_\beta σ π = σ β , so h = 0 h = 0 h = 0 recovers π = β \pi= \beta π = β exactly and h h h measures policy divergence in units of behavior standard deviations. Operators: MINT, Retrace(λ \lambda λ ), and a shuffled-K ^ \hat{K} K ^ control that permutes MINT’s truncation patterns across the batch, preserving the marginal distribution of K ^ \hat{K} K ^ exactly while removing its association with each sampled state–action sequence. All operators are evaluated using the same states, windows, and critic values, so all comparisons are paired at the level of individual evaluation states.
The ratios use a fitted behavior model, and reported errors are relative to finite-rollout Monte Carlo references. Behavior-model misspecification and Monte Carlo uncertainty contribute additional errors beyond the ideal fixed-policy operator analysis. In particular, h = 0 h = 0 h = 0 matches the target to the fitted behavior model; it does not establish equality with the true data-generating policy.
Frozen Critic: Properties of the Operator Alone
With the critic fixed to the finite-rollout Monte Carlo estimate, the comparison holds that approximate critic constant across target operators. Table 3 in the main text reports RMSE and intrinsic sampling variance at h = 0.5 h = 0.5 h = 0.5 ; Table 38 adds bias (mean signed error) and the paired t t t -statistic of the per-state absolute-error difference between MINT and Retrace. Table 39 reports all metrics at h = 1 h = 1 h = 1 . Intrinsic variance is measured with the approximate frozen critic; the exact-Q π Q^\pi Q π bound of Lemma 3.3 is not directly a guarantee for this measurement.
Table 38: Additional frozen-critic statistics at h = 0.5 h = 0.5 h = 0.5 for the comparison in Table 3 . t t t : paired t t t -statistic of the per-state absolute-error difference, MINT vs. Retrace.
Table 38.
Paired comparisons and error metrics. Section 5.5 summarizes the target-level comparison; Table 3 , Table 38 and Table 39 give the per-cell measurements. Bias is a mean signed error and permits cancellation across states, so we also report RMSE: the MINT–Retrace ratios correspond to 2.1×–13.8× reductions in bias magnitude and the more modest 1.04×–2.96× reductions in RMSE. At h = 0.5 h = 0.5 h = 0.5 the paired per-state absolute-error comparison has t = 2.3 t = 2.3 t = 2.3 –4.0; at h = 1 h = 1 h = 1 it is within sampling noise in three of the four cells (t = 0.6 t = 0.6 t = 0.6 –2.2). We do not claim that the advantage grows with divergence: at h = 2 h = 2 h = 2 the ordering reverses, because Retrace’s coefficient λ min ( 1 , π / β ) \lambda\min(1, \pi/\beta) λ min ( 1 , π / β ) itself truncates more aggressively as π \pi π moves away from β \beta β . Across the full 16-cell comparison, MINT has lower Var i n t \operatorname{Var}_{\mathrm{int}} Var int than the shuffled-K ^ \hat{K} K ^ control in 15 cells; large-navigate at h = 0.5 h = 0.5 h = 0.5 is the exception. The control complements the rate-matched comparison of Appendix § 8.4 .§ 5 .
Stitching. The stitch datasets contain 200-step rather than 1000-step trajectories. At h = 0.5 h = 0.5 h = 0.5 , paired per-state absolute-error comparisons favor MINT over Retrace on both stitch datasets (t = 4.0 t = 4.0 t = 4.0 and 3.2), as on the corresponding navigate datasets (t = 2.5 t = 2.5 t = 2.5 and 2.3). These results show that the observed target-level advantage also occurs on the shorter-trajectory datasets.
Protocol and caveat. We use 100 evaluation states per cell, stratified by distance to a goal-reaching event, because uniformly sampled states have nearly constant Q π Q^\pi Q π on these mazes and would make the comparison uninformative. The reference Q π ( s 0 , a 0 ) Q^\pi(s_0, a_0) Q π ( s 0 , a 0 ) uses 100 Monte-Carlo rollouts per state; frozen-critic values along each window use 30 rollouts per state and are shared across all operators; stochastic operators are averaged over 32 draws per state. Each cell is a single paired run; the seed-replicated version is the learned-critic study below. A one-step baseline attains the lowest measured RMSE in most cells of this approximate-frozen-critic comparison; this is an empirical result rather than an optimality claim based on an exact critic. This panel is therefore a comparison among off-policy corrections , not evidence that MINT dominates every target estimator.
Learned Critic: Fitted-Q Evaluation and a Limitation
As a complementary experiment we run the same operators inside fitted-Q evaluation, training a newly initialized critic (5 seeds) and scoring it against the same Monte-Carlo Q π Q^\pi Q π . The ranking depends on the optimization horizon: at 100k gradient steps Retrace achieves lower RMSE on every evaluated dataset, and by 300k steps the ordering on the deterministic maze reverses—at h = 1 h = 1 h = 1 MINT truncates to K ^ ≈ 1.6 \hat{K} \approx1.6 K ^ ≈ 1.6 , propagates value more slowly, and needs roughly 200k updates to stabilize, but then reaches lower error on the deterministic maze. Evaluations before convergence therefore conflate propagation speed with asymptotic estimation quality. Table 40 reports RMSE at 300k steps; we report both median and mean because the mean is sensitive to late-training instability in two cells. A seed is classified as unstable when its final error is at least twice its best error during training.
Bias and value propagation. The stochastic-maze mean RMSE at h = 1 h = 1 h = 1 is 12.04 for Retrace and 17.20 for MINT. The ranking across optimization horizons described above is consistent with a bias–propagation trade-off: a shorter effective horizon can reduce bias per backup but slow value propagation, and which effect dominates depends on critic convergence. The main-text overview is in § 5.5 .
Instability at large divergence under stochastic dynamics. At h = 2 h = 2 h = 2 both corrected operators show late-training instability. On the stochastic maze all MINT seeds reach RMSE 15–20 mid-training before degrading, two of five severely (final RMSE 85 and 157); one Retrace seed on the deterministic maze ends at RMSE 59 against a median of 17.66. The instability is therefore not unique to MINT, but it is more frequent and more severe for MINT under stochastic dynamics. The shuffled-K ^ \hat{K} K ^ control preserves the marginal truncation-length distribution while removing its match to each sampled trajectory, and it is stable in both h = 2 h = 2 h = 2 cells (0/5 unstable seeds, s.d. 1.57 and 2.43). This contrast suggests that dependence between the truncation pattern and the sampled trajectory may contribute to instability under large mismatch and multimodal dynamics; it does not isolate state dependence from action- or trajectory-dependent effects. These results document a learned-critic limitation of MINT at large policy divergence under stochastic dynamics; whether learning-rate or target-network schedules mitigate it is left open. Finally, shuffled-K ^ \hat{K} K ^ performs at least as well as MINT in every learned-critic cell. The frozen-critic comparison reports lower intrinsic variance in 15 of 16 cells, but this advantage does not consistently translate into lower learned-critic error.
Compute and Reproducibility
Hardware. All experiments were run on a single workstation with the following configuration: Ubuntu 24.04.1 LTS (kernel 6.8.0); 2× NVIDIA RTX 5000 Ada Generation GPUs (32 GB GDDR6 each, driver 560.35.03, CUDA 12.6); an AMD Ryzen Threadripper PRO 7985WX CPU (64 cores / 128 threads); and 502 GB of system RAM. Each training run is pinned to a single GPU, and we run two parallel seeds at a time.
Compute breakdown. Estimated aggregate GPU time for the listed experimental components is reported below. GPU-hours sum device time across runs; simultaneous execution on two GPUs reduces elapsed wall-clock time, not this aggregate.
MuJoCo main (5 tasks × 2 backbones × 5 methods × 5 seeds × 1M steps): 250 single-GPU runs at approximately 6 hours per run give 1,500 GPU-hours (about 750 elapsed hours if both GPUs are continuously occupied).
Cap-length sweep (≥ 3 \ge3 ≥ 3 tasks × 4 caps × 4 baselines × 5 seeds × 1M steps): ≈ 360 GPU-hours .
DMControl sparse-reward (1–2 tasks × 5 baselines × 5 seeds × 1M steps): ≈ 120 GPU-hours .
DMControl truncation-kernel ablation (Section F.2.9; 5 tasks × 3 methods × 3 seeds × 1M steps, plus a 5-seed re-run of CartPole–Swingup–Sparse): ≈ 100 GPU-hours .
Ablations (clipped surrogate p ^ i \hat{p}_i p ^ i , clipping form, hard cap K max K_{\max} K m a x ): ≈ 80 GPU-hours .
Toy-chain variance experiments: < 1 hour on CPU only.
The subtotal of the listed GPU components is approximately 2,160 GPU-hours (1,500 + 360 + 120 + 100 + 80), excluding the separately reported CPU-only toy experiments. This is a subtotal for the listed components, rather than an accounting of every additional benchmark or exploratory run.
Reproducibility. All evaluation protocols follow OpenAI SpinningUp [2 ] unless noted otherwise; full hyperparameters are listed in Table 9 and Table 10 and the Q-target update rules in Table 8 .
Table 9. TD3 Hyperparameters