Introduction
Bilevel reinforcement learning (BRL) couples reward learning with policy adaptation: changing a reward changes both the responding policy and the trajectories used to evaluate the upper objective. This dependence underlies reward design and reinforcement learning from human feedback. Establishing convergence requires justifying the policy-optimization geometry and controlling value estimation while rewards, policies, and sampling distributions evolve together.
Tabular analyses exploit finite policy and value representations [24 , 30 ] . In particular, Yang et al. [30 ] use soft Bellman structure to obtain first-order upper-gradient calculations for a global regularized response. Zeng et al. [31 ] establish a single-loop first-order tabular actor–critic guarantee under regularization-dependent PL and additional Hessian conditions. Extending these guarantees beyond finite state–action representations requires controlling actor and critic approximation and learning errors.
Beyond tabular formulations, PARL [5 ] studies policy alignment with parameterized policies and accounts for the dependence of the upper sampling distribution on the lower response. Its formulation accommodates continuous states, but its A-PARL algorithm uses implicit hypergradient calculations involving mixed derivatives and an inverse lower-level Hessian. It is therefore not fully first order, and its convergence analysis assumes access to the required derivative oracles. Gaur et al. [10 ] establish a first-order O ~ ( ε − 3 ) \widetilde{O}(\varepsilon^{-3}) O ( ε − 3 ) sample-complexity bound for continuous state–action spaces under a uniform lower-level Polyak–Łojasiewicz (PL) condition.
Uniform PL does not follow from smooth policy parameterization alone. Our counterexample in Proposition C.52 has smooth positive action probabilities and exactly representable values, yet admits a suboptimal stationary point, violating PL. This motivates specifying an actor construction, deriving its required local geometry, and proving that the learning dynamics remain in the region where that geometry holds. Additionally, PL conditions in general do not hold for arbitrary RL setups. Thus works which rely on PL assumptions are not applicable for arbitrary actor and critic classes.
Additionally, critic learning introduces a distinct difficulty. The value targets change with both the reward and the policy, and the response correction uses differences between nearby policies divided by their separation. Separate bounds on the two critic errors do not directly control their difference after this normalization as the separation shrinks. A coupled guarantee must therefore establish persistent critic stability and track both the individual value errors and their normalized endpoint differences. Thus we have the following question.
Can we obtain improved sample-complexity guarantees for single-loop, first-order bilevel reinforcement learning beyond tabular settings, using explicit neural actors and nonlinear critics, while deriving the required local geometry and controlling their coupled learning errors?
We answer this question in the affirmative and address these challenges with explicit tied multilayer actors and a nonlinear critic built from fixed features. An initialization covariance condition, architectural derivative bounds, and fixed reference-policy KL regularization yield local curvature; a joint containment proof makes this geometry valid throughout training with high probability. Shared trajectories control the noise in the response correction. Under feature excitation and realizability of the regularized value family, our paired critic analysis establishes stability and tracking of the required normalized errors. Residual-controlled actor solves complete the finite, first-order implementation. Together, these ingredients yield O ~ ( ε − 2 ) \widetilde{O}(\varepsilon^{-2}) O ( ε − 2 ) sample complexity for O ( ε ) O(\varepsilon) O ( ε ) expected squared upper-gradient norm, including persistent critic training, preference sampling, and metric initialization. The guarantee covers continuous states and finite actions. under the stated architecture, initialization, regularity, and value-representation conditions.
Variance control for coupled response estimation. We estimate the policy-response correction by evaluating nearby policies on shared, appropriately weighted trajectories. This coupling keeps sampling variance controlled as the policies being compared become closer, allowing the response approximation to become more accurate without amplifying sampling noise. Together with paired critic-error control and persistent actor–critic tracking, this mechanism supports our improved sample complexity relative to Gaur et al. [10 ] .
Improved interaction complexity with single-loop first-order learning. We establish the O ~ ( ε − 2 ) \widetilde{O}(\varepsilon^{-2}) O ( ε − 2 ) interaction bound while reward, actor, and critic updates proceed together. Shared trajectory sampling controls response-estimation noise as the perturbation shrinks, and a combined descent and tracking potential accounts for all three learning processes and their sampling costs.
Explicit actor–critic architectures with coupled learning guarantees. We specify multilayer neural actors and a nonlinear critic built from fixed features. We show that smooth policies and exact value realizability do not imply uniform PL, and derive local actor curvature from initialization identifiability and fixed reference-KL regularization. We prove containment of the coupled iterates and stability of persistent critic learning, controlling critic errors as rewards, policies change. A continuous-state construction verifies the actor and critic conditions (Appendix § 10 .11).
Problem Formulation
MDP and sampling access. For a reward parameter x ∈ O x = R d x x \in\mathcal{O}_x = \mathbb{R}^{d_x} x ∈ O x = R d x , let M x = ( S , A , P , r x , ρ , γ ) \mathcal{M}_x = (\mathcal{S}, \mathcal{A}, P, r_x, \rho, \gamma) M x = ( S , A , P , r x , ρ , γ ) be a discounted Markov decision process: S \mathcal{S} S is compact and possibly continuous, A \mathcal{A} A is finite, P ( d s ′ ∣ s , a ) P(ds' \mid s,a) P ( d s ′ ∣ s , a ) is the transition law, r x ( s , a ) r_x(s,a) r x ( s , a ) is the reward, ρ \rho ρ is the reset distribution, and γ ∈ ( 0 , 1 ) \gamma\in(0,1) γ ∈ ( 0 , 1 ) is the discount. Only the reward and upper sample loss depend explicitly on x x x . We can independently reset from ρ \rho ρ , generate trajectories under the current policies, and query feedback on trajectory tuples. The reward and its first derivatives are evaluable; the transition kernel need not be known.
Multilayer neural actor. Let L L L be the hidden depth, m = ( m 1 , … , m L ) \mathbf{m} = (m_1,\ldots,m_L) m = ( m 1 , … , m L ) the hidden widths, and α ∈ R r \alpha\in\mathbb{R}^r α ∈ R r the trainable coordinates. A scalar score F m , α F_{\mathbf{m},\alpha} F m , α tilts a fixed strictly positive reference policy:
π m , α ( a ∣ s ) = π r e f ( a ∣ s ) e F m , α ( s , a ) ∑ b ∈ A π r e f ( b ∣ s ) e F m , α ( s , b ) . (1) \pi_{\mathbf{m},\alpha}(a \mid s) = \frac{\pi_{\mathrm{ref}}(a \mid s)e^{F_{\mathbf{m},\alpha}(s,a)}}{\sum_{b\in\mathcal{A}}\pi_{\mathrm{ref}}(b \mid s)e^{F_{\mathbf{m},\alpha}(s,b)}}. \tag*{(1)} π m , α ( a ∣ s ) = ∑ b ∈ A π ref ( b ∣ s ) e F m , α ( s , b ) π ref ( a ∣ s ) e F m , α ( s , a ) . (1)
Two coupled smooth multilayer networks define the score. Their weights vary through a fixed map of the same r r r coordinates around initialization; we call this map the retained actor chart. Depth L L L and dimension r r r remain fixed as widths vary, and F m , 0 = 0 F_{\mathbf{m},0}=0 F m , 0 = 0 gives π m , 0 = π r e f \pi_{\mathbf{m},0}=\pi_{\mathrm{ref}} π m , 0 = π ref . Appendix § 10 .§ 1 specifies the chart, normalization, inputs, and activations.
Lower and upper objectives. Let P m , α \mathbb{P}_{\mathbf{m},\alpha} P m , α be the trajectory law under reset s 0 ∼ ρ s_0 \sim\rho s 0 ∼ ρ , actions a t ∼ π m , α ( ⋅ ∣ s t ) a_t \sim\pi_{\mathbf{m},\alpha}(\cdot\mid s_t) a t ∼ π m , α ( ⋅ ∣ s t ) , and transitions P P P ; write E m , α \mathbb{E}_{\mathbf{m},\alpha} E m , α for its expectation. Fix tuple size I I I and trajectory length K K K independently of accuracy. The tuple ξ = ( ξ 1 , … , ξ I ) \xi= (\xi^1,\ldots,\xi^I) ξ = ( ξ 1 , … , ξ I ) contains independent length-K K K state–action trajectories, with product law ( P m , α K ) ⊗ I (\mathbb{P}_{\mathbf{m},\alpha}^{K})^{\otimes I} ( P m , α K ) ⊗ I . Feedback follows a fixed conditional law D ( ⋅ ∣ ξ ) \mathcal{D}(\cdot\mid\xi) D ( ⋅ ∣ ξ ) , independent of ( x , α ) (x,\alpha) ( x , α ) given the tuple. For sample loss L ( x ; ξ , Y ) L(x;\xi,Y) L ( x ; ξ , Y ) and fixed Kullback–Leibler (KL) regularization coefficient τ > 0 \tau> 0 τ > 0 , define
g τ , m ( x , α ) = ( 1 − γ ) E m , α ∑ t ≥ 0 γ t [ − r x ( s t , a t ) + τ log π m , α ( a t ∣ s t ) π r e f ( a t ∣ s t ) ] , (2) g_{\tau,\mathbf{m}}(x,\alpha) = (1-\gamma)\mathbb{E}_{\mathbf{m},\alpha}\sum_{t\geq0}\gamma^t\left[-r_x(s_t,a_t)+\tau\log\frac{\pi_{\mathbf{m},\alpha}(a_t \mid s_t)}{\pi_{\mathrm{ref}}(a_t \mid s_t)}\right], \tag*{(2)} g τ , m ( x , α ) = ( 1 − γ ) E m , α t ≥ 0 ∑ γ t [ − r x ( s t , a t ) + τ log π ref ( a t ∣ s t ) π m , α ( a t ∣ s t ) ] , (2)
f m ( x , α ) = E ξ ∼ ( P m , α K ) ⊗ I , Y ∼ D ( ⋅ ∣ ξ ) L ( x ; ξ , Y ) . (3) f_{\mathbf{m}}(x,\alpha) = \mathbb{E}_{\xi\sim(\mathbb{P}_{\mathbf{m},\alpha}^{K})^{\otimes I},\,Y\sim\mathcal{D}(\cdot\mid\xi)}L(x;\xi,Y). \tag*{(3)} f m ( x , α ) = E ξ ∼ ( P m , α K ) ⊗ I , Y ∼ D ( ⋅ ∣ ξ ) L ( x ; ξ , Y ) . (3)
Thus lower minimization maximizes regularized return, while the upper loss depends on the policy through its trajectory distribution. For Bradley–Terry comparisons, I = 2 I=2 I = 2 and Y ∈ { 0 , 1 } Y\in\{0,1\} Y ∈ { 0 , 1 } , with Y = 1 Y=1 Y = 1 favoring ξ 1 \xi^1 ξ 1 .
Writing ( s t i , a t i ) (s_t^i, a_t^i) ( s t i , a t i ) for trajectory i i i , the loss is
Δ x = ∑ t = 0 K − 1 [ r x ( s t 1 , a t 1 ) − r x ( s t 2 , a t 2 ) ] , L ( x ; ξ , Y ) = log ( 1 + e Δ x ) − Y Δ x . (4) \Delta_x = \sum_{t=0}^{K-1} [r_x(s_t^1, a_t^1) - r_x(s_t^2, a_t^2)], \qquad L(x; \xi, Y) = \log(1 + e^{\Delta_x}) - Y\Delta_x. \tag*{(4)} Δ x = t = 0 ∑ K − 1 [ r x ( s t 1 , a t 1 ) − r x ( s t 2 , a t 2 )] , L ( x ; ξ , Y ) = log ( 1 + e Δ x ) − Y Δ x . (4)
Correct specification of the preference model is unnecessary (Appendix § 10 .§ 2 ).
Bilevel problem and stationarity target. Let B g e o m = { α : ∥ α ∥ 2 < R g e o m } \mathcal{B}_{\mathrm{geom}} = \{\alpha: \|\alpha\|_2 < R_{\mathrm{geom}}\} B geom = { α : ∥ α ∥ 2 < R geom } . For the fixed τ \tau τ and radius supplied by § 4 , we study
min x ∈ R d x Φ τ , m ( x ) : = f m ( x , α 0 , m ∗ ( x ) ) , where α 0 , m ∗ ( x ) ∈ arg min α ∈ B g e o m g τ , m ( x , α ) . (5) \min_{x\in\mathbb{R}^{d_x}} \Phi_{\tau,m}(x) := f_m(x, \alpha_{0,m}^*(x)),
\qquad
\text{where }\alpha_{0,m}^*(x) \in\arg\min_{\alpha\in\mathcal{B}_{\mathrm{geom}}} g_{\tau,m}(x,\alpha). \tag*{(5)} x ∈ R d x min Φ τ , m ( x ) := f m ( x , α 0 , m ∗ ( x )) , where α 0 , m ∗ ( x ) ∈ arg α ∈ B geom min g τ , m ( x , α ) . (5)
The later conditions make this minimizer unique, interior, and smooth in x x x , with ∇ α g τ , m ( x , α 0 , m ∗ ) = 0 \nabla_\alpha g_{\tau,m}(x,\alpha_{0,m}^*) = 0 ∇ α g τ , m ( x , α 0 , m ∗ ) = 0 . The ball selects a local response in the full actor space R r \mathbb{R}^r R r ; subscript 0 denotes the unperturbed response. For accuracy ε > 0 \varepsilon> 0 ε > 0 , our goal is an output satisfying E ∥ ∇ Φ τ , m ( x o u t ) ∥ 2 = O ( ε ) \mathbb{E}\|\nabla\Phi_{\tau,m}(x_{\mathrm{out}})\|^2 = O(\varepsilon) E ∥∇ Φ τ , m ( x out ) ∥ 2 = O ( ε ) .
Value function and nonlinear critic. Hereafter suppress actor widths, write g = g τ g = g_\tau g = g τ , and use h x h_x h x , h α h_\alpha h α for partial gradients of a function h h h . Define q x , α = − r x + τ log ( π α / π r e f ) q_{x,\alpha} = -r_x + \tau\log(\pi_\alpha/\pi_{\mathrm{ref}}) q x , α = − r x + τ log ( π α / π ref ) and the unnormalized discounted action value
Q x , α t r u e ( s , a ) = E α s , a [ ∑ t ≥ 0 γ t q x , α ( s t , a t ) ] , Q_{x,\alpha}^{\mathrm{true}}(s,a) = \mathbb{E}_{\alpha}^{s,a}\left[\sum_{t\geq0}\gamma^t q_{x,\alpha}(s_t,a_t)\right], Q x , α true ( s , a ) = E α s , a [ t ≥ 0 ∑ γ t q x , α ( s t , a t ) ] ,
where E α s , a \mathbb{E}_{\alpha}^{s,a} E α s , a starts the process at ( s 0 , a 0 ) = ( s , a ) (s_0,a_0) = (s,a) ( s 0 , a 0 ) = ( s , a ) . For a fixed bounded feature map ϕ Q : S × A → R d Q \phi_Q : \mathcal{S}\times\mathcal{A}\to\mathbb{R}^{d_Q} ϕ Q : S × A → R d Q , coordinates θ ∈ R d Q \theta\in\mathbb{R}^{d_Q} θ ∈ R d Q , and scale m Q ≥ 1 m_Q\geq1 m Q ≥ 1 , use
Q m Q , θ ( s , a ) = m Q tanh ( ϕ Q ( s , a ) ⊤ θ / m Q ) . (6) Q_{m_Q,\theta}(s,a) = \sqrt{m_Q}\tanh\left(\phi_Q(s,a)^\top\theta/\sqrt{m_Q}\right). \tag*{(6)} Q m Q , θ ( s , a ) = m Q tanh ( ϕ Q ( s , a ) ⊤ θ / m Q ) . (6)
The scale controls nonlinearity at fixed dimension. § 3 trains this critic persistently; Appendix § 10 .11 jointly verifies the actor and critic assumptions in a continuous-state instance.
Response-chain correction. With response Jacobian D x α 0 ∗ ( x ) ∈ R r × d x D_x\alpha_0^*(x)\in\mathbb{R}^{r\times d_x} D x α 0 ∗ ( x ) ∈ R r × d x ,
∇ Φ τ ( x ) = f x ( x , α 0 ∗ ( x ) ) + D x α 0 ∗ ( x ) ⊤ f α ( x , α 0 ∗ ( x ) ) . (7) \nabla\Phi_\tau(x) = f_x(x,\alpha_0^*(x)) + D_x\alpha_0^*(x)^\top f_\alpha(x,\alpha_0^*(x)). \tag*{(7)} ∇ Φ τ ( x ) = f x ( x , α 0 ∗ ( x )) + D x α 0 ∗ ( x ) ⊤ f α ( x , α 0 ∗ ( x )) . (7)
The second term accounts for policy adaptation. The next section estimates this total gradient.
Proposed Approach
Response correction through paired actors. Following first-order penalty constructions [18 ] , let α c ∗ ( x ) \alpha_c^*(x) α c ∗ ( x ) solve g α + c f α = 0 g_\alpha+ cf_\alpha= 0 g α + c f α = 0 locally in B g e o m \mathcal{B}_{\mathrm{geom}} B geom for perturbation 0 < c ≤ min { c 0 , 1 } 0 < c \leq\min\{c_0,1\} 0 < c ≤ min { c 0 , 1 } and sufficiently small fixed c 0 c_0 c 0 (§ 4 ). Define
S c ∗ ( x ) : = α c ∗ ( x ) − α 0 ∗ ( x ) c . (8) S_c^*(x) := \frac{\alpha_c^*(x)-\alpha_0^*(x)}{c}. \tag*{(8)} S c ∗ ( x ) := c α c ∗ ( x ) − α 0 ∗ ( x ) . (8)
Persistent coordinates U , S ∈ R r U, S \in\mathbb{R}^{r} U , S ∈ R r track α 0 ∗ ( x ) , S c ∗ ( x ) \alpha_{0}^{*}(x), S_{c}^{*}(x) α 0 ∗ ( x ) , S c ∗ ( x ) ; the actors are U , U + c S U, U+cS U , U + c S . For p = ( x , U , S ) p=(x,U,S) p = ( x , U , S ) , set a ( p ) = g α ( x , U ) a(p)=g_{\alpha}(x,U) a ( p ) = g α ( x , U ) and
R c ( p ) = g α ( x , U + c S ) − g α ( x , U ) c + f α ( x , U + c S ) . (9) R_{c}(p)=\frac{g_{\alpha}(x,U+cS)-g_{\alpha}(x,U)}{c}+f_{\alpha}(x,U+cS). \tag*{(9)} R c ( p ) = c g α ( x , U + c S ) − g α ( x , U ) + f α ( x , U + c S ) . (9)
V c ( p ) = g x ( x , U + c S ) − g x ( x , U ) c + f x ( x , U + c S ) . (10) V_{c}(p)=\frac{g_{x}(x,U+cS)-g_{x}(x,U)}{c}+f_{x}(x,U+cS). \tag*{(10)} V c ( p ) = c g x ( x , U + c S ) − g x ( x , U ) + f x ( x , U + c S ) . (10)
At these response targets, R c = 0 R_{c}=0 R c = 0 and V c = ∇ Φ τ ( x ) + O ( c ) V_{c}=\nabla\Phi_{\tau}(x)+O(c) V c = ∇ Φ τ ( x ) + O ( c ) under the conditions of § 4 (Appendix § 9 .§ 1 ). Thus gradient differences recover policy adaptation using first-order evaluations.
Paired critic and certified neighborhoods. We learn two critics together through coordinates z = ( u , v ) z=(u,v) z = ( u , v ) : the base critic has parameter u u u , and the perturbed critic has parameter u + c v u+cv u + c v . Thus v v v tracks their separation relative to the perturbation scale c c c . To keep critic updates bounded, we project onto K Q K_{Q} K Q , the set of pairs satisfying ∥ u ∥ ≤ R u \lVert u\rVert\leq R_{u} ∥ u ∥ ≤ R u and ∥ v ∥ ≤ R v \lVert v\rVert\leq R_{v} ∥ v ∥ ≤ R v . The projection Π K Q W \Pi_{K_{Q}}^{W} Π K Q W uses the weighted distance induced by ∥ z ∥ W 2 = M ∥ u ∥ 2 + ∥ v ∥ 2 \lVert z\rVert_{W}^{2}=M\lVert u\rVert^{2}+\lVert v\rVert^{2} ∥ z ∥ W 2 = M ∥ u ∥ 2 + ∥ v ∥ 2 . Appendix § 9 .§ 2 constructs the positive radii and weight.
For the actors, let s > 0 s>0 s > 0 bound the normalized separation and r p r o x > 0 r_{\mathrm{prox}}>0 r prox > 0 be the radius of each local actor solve. The analysis uses a prospective neighborhood, defined by ∥ S ∥ < s \lVert S\rVert<s ∥ S ∥ < s and ∥ U ∥ < R g e o m − c 0 s − r p r o x / 2 \lVert U\rVert<R_{\mathrm{geom}}-c_{0}s-r_{\mathrm{prox}}/2 ∥ U ∥ < R geom − c 0 s − r prox /2 , within which proposed updates can be analyzed. The algorithm checks the stricter condition
Safe ( U , S ) : ∥ S ∥ < s / 2 , ∥ U ∥ < R g e o m − c 0 s − r p r o x . \operatorname{Safe}(U,S):\qquad\lVert S\rVert<s/2,\qquad\lVert U\rVert<R_{\mathrm{geom}}-c_{0}s-r_{\mathrm{prox}}. Safe ( U , S ) : ∥ S ∥ < s /2 , ∥ U ∥ < R geom − c 0 s − r prox .
The gap between these neighborhoods provides room for local actor updates before the next safety check. Appendix § 9 .§ 5 specifies the radii and proves that the iterates satisfy this test throughout training with high probability.
Finite actor updates. Each actor update balances following an estimated descent direction with keeping policy changes small. Set the actor step η y = r Q β \eta_{y}=r_{Q}\beta η y = r Q β and upper step η x = q η y \eta_{x}=q\eta_{y} η x = q η y , where β > 0 \beta>0 β > 0 is the critic step and r Q , q > 0 r_{Q},q>0 r Q , q > 0 are fixed ratios. Draw n n n independent reset states from ρ \rho ρ , separately from training data, and denote their empirical distribution by ρ ^ n \widehat{\rho}_{n} ρ n . Choose n n n for rejection budget δ m e t \delta_{\mathrm{met}} δ met using (188) , and check the initialization covariance using (191) . This empirical certificate supplies the local KL geometry needed for the updates; rejection returns x 0 x_{0} x 0 . Store the accepted states and reuse them throughout training for both actors.
For a frozen actor parameter b b b and estimated gradient h h h , define
ℓ b , h ( α ) = 1 n ∑ i = 1 n D K L ( π b ( ⋅ ∣ s i ) ∥ π α ( ⋅ ∣ s i ) ) + η y h T ( α − b ) . (11) \ell_{b,h}(\alpha)=\frac{1}{n}\sum_{i=1}^{n}D_{\mathrm{KL}}\left(\pi_{b}(\cdot\mid s_{i})\middle\|\pi_{\alpha}(\cdot\mid s_{i})\right)+\eta_{y}h^{\mathsf{T}}(\alpha-b). \tag*{(11)} ℓ b , h ( α ) = n 1 i = 1 ∑ n D KL ( π b ( ⋅ ∣ s i ) ∥ π α ( ⋅ ∣ s i ) ) + η y h T ( α − b ) . (11)
The KL term discourages large policy changes on the stored states, while the linear term encourages descent along − h -h − h . The map M ~ ( b , h ) \widetilde{M}(b,h) M ( b , h ) minimizes this objective approximately by projected gradient descent, starting at b b b and remaining within distance r p r o x r_{\mathrm{prox}} r prox . Under the certified conditions, the stopping rule in Proposition C.33 guarantees distance at most c η y 2 c\eta_{y}^{2} c η y 2 from the exact minimizer within J max J_{\max} J m a x updates. These numerical steps use stored states and finite action sums, requiring no fresh environment interactions. Appendix § 10 .§ 4 gives the implementation details and constants.
Sampling routines and timing. At each iteration, all sampling routines use frozen pre-update parameters; the rollout cap H H H is distinct from the preference length K K K . LowerPair uses the paired critics and preference tuples to estimate ( a , R c ) (a,R_{c}) ( a , R c ) , with C ^ = a ^ + c ^ R ^ c \widehat{C}=\widehat{a}+\widehat{c}\widehat{R}_{c} C = a + c R c exactly. OuterPair estimates V c V_{c} V c using reward derivatives and separate preference tuples. TDPair estimates the paired temporal-difference (TD) field in (222) , reusing each base-policy transition for both endpoint residuals. The sampling blocks are conditionally independent given the pre-draw history, which includes the metric sample. Preference tuples and labels come from the old perturbed actor, and the updated critic enters the next iteration. Appendix § 10 .§ 6 specifies the estimators and their interaction costs. Algorithm 2 gives a detailed expansion of one iteration of Algorithm 1, spelling out the sampling batches, estimator construction, and parameter updates.
Algorithm 1 Single-loop bilevel RL with a persistent paired critic
Require: Initial x 0 ∈ R d x x_0 \in\mathbb{R}^{d_x} x 0 ∈ R d x ; iteration count T T T ; c , τ , m Q , H , δ m e t c,\tau,m_Q,H,\delta_{\mathrm{met}} c , τ , m Q , H , δ met ; batch sizes; step sizes β , η y = r Q β , η x = q η y \beta,\eta_y=r_Q\beta,\eta_x=q\eta_y β , η y = r Q β , η x = q η y ; the specified geometry and radii.
Fix the chart; draw n n n independent reset states to form ρ ^ n \widehat{\rho}_n ρ n and certify it using (188) and (191) .
Set U 0 = S 0 = 0 U_0=S_0=0 U 0 = S 0 = 0 and deterministic z 0 ∈ K Q z_0\in\mathcal{K}_Q z 0 ∈ K Q ; return x 0 x_0 x 0 if the metric certificate or Safe ( U 0 , S 0 ) \operatorname{Safe}(U_0,S_0) Safe ( U 0 , S 0 ) fails.
for t = 0 , … , T − 1 t=0,\ldots,T-1 t = 0 , … , T − 1 do
Freeze p t = ( x t , U t , S t ) p_t=(x_t,U_t,S_t) p t = ( x t , U t , S t ) and z t z_t z t .
( a ^ t , R ^ t ) ← LowerPair ( p t , z t ; H ) (\widehat{a}_t,\widehat{R}_t)\leftarrow\operatorname{LowerPair}(p_t,z_t;H) ( a t , R t ) ← LowerPair ( p t , z t ; H ) ; C ^ t ← a ^ t + c R ^ t \widehat{C}_t\leftarrow\widehat{a}_t+c\widehat{R}_t C t ← a t + c R t exactly.
U t + 1 ← M ^ ( U t , a ^ t ) U_{t+1}\leftarrow\widehat{M}(U_t,\widehat{a}_t) U t + 1 ← M ( U t , a t ) ; α ^ t + 1 ← M ^ ( U t + c S t , C ^ t ) \widehat{\alpha}_{t+1}\leftarrow\widehat{M}(U_t+cS_t,\widehat{C}_t) α t + 1 ← M ( U t + c S t , C t ) .
S t + 1 ← ( α ^ t + 1 − U t + 1 ) / c S_{t+1}\leftarrow(\widehat{\alpha}_{t+1}-U_{t+1})/c S t + 1 ← ( α t + 1 − U t + 1 ) / c .
V ^ t ← OuterPair ( p t ; H ) \widehat{V}_t\leftarrow\operatorname{OuterPair}(p_t;H) V t ← OuterPair ( p t ; H ) ; x t + 1 ← x t − η x V ^ t x_{t+1}\leftarrow x_t-\eta_x\widehat{V}_t x t + 1 ← x t − η x V t .
F ^ t ← TDPair ( p t , z t ; H ) \widehat{F}_t\leftarrow\operatorname{TDPair}(p_t,z_t;H) F t ← TDPair ( p t , z t ; H ) ; z t + 1 ← Π K Q W ( z t − β F ^ t ) z_{t+1}\leftarrow\Pi_{\mathcal{K}_Q}^{W}(z_t-\beta\widehat{F}_t) z t + 1 ← Π K Q W ( z t − β F t ) .
return x 0 x_0 x 0 if Safe ( U t + 1 , S t + 1 ) \operatorname{Safe}(U_{t+1},S_{t+1}) Safe ( U t + 1 , S t + 1 ) fails.
end for
Draw an independent uniform index J ∈ { 0 , … , T − 1 } J\in\{0,\ldots,T-1\} J ∈ { 0 , … , T − 1 } ; return x J x_J x J .
Why shared samples matter. The response correction divides differences between nearby policy evaluations by c c c , which can amplify noise from independent endpoint estimates. Shared, appropriately weighted trajectories make the centered endpoint difference shrink with c c c , keeping the normalized sampling variance controlled as the deterministic approximation bias decreases. Steps 3 and 6 of the proof outline in § 4 establish this variance control and explain its role in the sample complexity.
Theoretical Analysis
Fix A > 0 A>0 A > 0 , B ‾ A = { α ∈ R r : ∥ α ∥ 2 ≤ A } \overline{B}_A=\{\alpha\in\mathbb{R}^r:\|\alpha\|_2\le A\} B A = { α ∈ R r : ∥ α ∥ 2 ≤ A } , and 0 < R g e o m ≤ A / 2 0<R_{\mathrm{geom}}\le A/2 0 < R geom ≤ A /2 . Derivatives use multilinear operator norms; order zero denotes the function. Bounds are uniform in admissible actor widths. After fixing the construction, global constants are independent of x x x , c c c , m Q m_Q m Q and accuracy unless stated otherwise. First we give the needed assumption as follows.
Assumption 4.1 (A1: primitive local regularity). For every compact Q ⋐ O x Q\Subset\mathcal{O}_x Q ⋐ O x and α ∈ B ‾ A \alpha\in\overline{B}_A α ∈ B A , require:
(i) Reward: r x ( s , a ) r_x(s,a) r x ( s , a ) is C 4 C^4 C 4 in x x x , with sup x ∈ Q , s , a ∥ D x k r x ( s , a ) ∥ ≤ C r , k ( Q ) \sup_{x\in Q,s,a}\|D_x^k r_x(s,a)\|\le C_{r,k}(Q) sup x ∈ Q , s , a ∥ D x k r x ( s , a ) ∥ ≤ C r , k ( Q ) for 0 ≤ k ≤ 4 0\le k\le4 0 ≤ k ≤ 4 .
(ii) Upper sample loss: L ( x ; ξ , Y ) L(x;\xi,Y) L ( x ; ξ , Y ) is C 4 C^4 C 4 in x x x , with sup x ∈ Q , ξ , Y ∥ D x k L ∥ ≤ C L , k ( Q ) \sup_{x\in Q,\xi,Y}\|D_x^kL\|\le C_{L,k}(Q) sup x ∈ Q , ξ , Y ∥ D x k L ∥ ≤ C L , k ( Q ) for 0 ≤ k ≤ 4 0\le k\le4 0 ≤ k ≤ 4 .
(iii) Actor score: the normalized tied construction satisfies sup m , α ∈ B ‾ A , s , a ∥ D α k F m , α ∥ ≤ C F , k ( A ) \sup_{m,\alpha\in\overline{B}_A,s,a}\|D_\alpha^kF_{m,\alpha}\|\le C_{F,k}(A) sup m , α ∈ B A , s , a ∥ D α k F m , α ∥ ≤ C F , k ( A ) for 1 ≤ k ≤ 4 1\le k\le4 1 ≤ k ≤ 4 .
All envelopes are finite and uniform in sample arguments and admissible widths; state or action differentiability is unnecessary.
Context. Related regularity appears in Marbach and Tsitsiklis [22 ] and Kwon et al. [18 ] . Fourth-order bounds control paired KL-update remainders; the specified architecture verifies item (iii).
Assumption 4.2 (A2: initialization identifiability). In the retained r r r -dimensional chart, let Z m , 0 = D α F m , 0 Z_{m,0}=D_\alpha F_{m,0} Z m , 0 = D α F m , 0 . For a width-independent λ e f f , L > 0 \lambda_{\mathrm{eff},L}>0 λ eff , L > 0 , require
G m , 0 : = E s ∼ ρ Cov a ∼ π r e f ( ⋅ ∣ s ) ( Z m , 0 ( s , a ) ) ⪰ λ e f f , L I r . (12) \mathcal{G}_{m,0}:=\mathbb{E}_{s\sim\rho}\operatorname{Cov}_{a\sim\pi_{\mathrm{ref}}(\cdot\mid s)}\left(Z_{m,0}(s,a)\right)\succeq\lambda_{\mathrm{eff},L}I_r. \tag*{(12)} G m , 0 := E s ∼ ρ Cov a ∼ π ref ( ⋅ ∣ s ) ( Z m , 0 ( s , a ) ) ⪰ λ eff , L I r . (12)
Context, verification, and derived geometry. Unlike the domain-wide Fisher conditions of Liu et al. [21 ] and Fatkhullin et al. [6 ] , A2 concerns only initialization. It admits a finite-sample certificate for an independently fixed chart (Proposition C.13 ) and analytical verification in Appendix § 10 .11. Architectural derivative bounds and global regularity in G below propagate this gap locally. Sufficiently strong fixed reference-KL regularization then gives
∇ α 2 [ g τ , m ( x , α ) + c f m ( x , α ) ] ⪰ μ τ I r , (13) \nabla_{\alpha}^{2}[g_{\tau,m}(x,\alpha)+c f_m(x,\alpha)]\succeq\mu_{\tau}I_r, \tag*{(13)} ∇ α 2 [ g τ , m ( x , α ) + c f m ( x , α )] ⪰ μ τ I r , (13)
uniformly over x x x , α ∈ B g e o m \alpha\in\mathcal{B}_{\mathrm{geom}} α ∈ B geom , and 0 ≤ c ≤ c 0 0\leq c\leq c_0 0 ≤ c ≤ c 0 , for μ τ > 0 \mu_{\tau}>0 μ τ > 0 (Proposition B.1 ). This yields unique smooth interior responses; under C1–C2 and the stated schedule, containment of the coupled iterates is proved (Theorem B.13 ).
Initialization identifiability and derived geometry. Gaur et al. [10 ] imposes uniform lower-level PL, which need not hold even for smooth positive policies (Proposition C.52). Our analysis instead starts from the initialization covariance condition in Assumption 4.2 , verified analytically for our explicit construction in Appendix § 10 .11. Together with regularity and fixed reference-KL regularization, this condition yields local response curvature (Proposition B.1 ). The coupled-update analysis then establishes containment within this region (Theorem B.13 ).
Assumption 4.3 (G: global upper regularity for preference losses). Use § 2 ’s fixed-dynamics, finite-action reset setting and objective (3) on O x = R d x O_x=\mathbb{R}^{d_x} O x = R d x , with fixed tuple size I I I and trajectory length K K K . Tuples and labels are freshly sampleable. The conditional feedback law is independent of ( x , α ) (x,\alpha) ( x , α ) given the trajectories, and L L L has no additional explicit actor dependence. For finite constants, uniformly in x x x and sample arguments,
∣ r x ∣ ≤ B r , ∥ ∇ x r x ∥ ≤ G r , ∥ ∇ x 2 r x ∥ o p ≤ L r , \lvert r_x\rvert\leq B_r,\qquad\lVert\nabla_xr_x\rVert\leq G_r,\qquad\lVert\nabla_x^2r_x\rVert_{\mathrm{op}}\leq L_r, ∣ r x ∣ ≤ B r , ∥ ∇ x r x ∥ ≤ G r , ∥ ∇ x 2 r x ∥ op ≤ L r ,
∣ L ∣ ≤ B L , ∥ ∇ x L ∥ ≤ G L , ∥ ∇ x 2 L ∥ o p ≤ L L . (14) \lvert L\rvert\leq B_L,\qquad\lVert\nabla_xL\rVert\leq G_L,\qquad\lVert\nabla_x^2L\rVert_{\mathrm{op}}\leq L_L. \tag*{(14)} ∣ L ∣ ≤ B L , ∥ ∇ x L ∥ ≤ G L , ∥ ∇ x 2 L ∥ op ≤ L L . (14)
These bounds hold for each realized label; A1’s local C 4 C^4 C 4 regularity is retained.
Context and role. Related global reward and loss envelopes appear in Yang et al. [30 ] . For Bradley–Terry, the reward bounds imply the loss envelopes (Proposition C.14 ). These bounds remain uniform as x x x changes. Together with A1–A2, they allow sufficiently strong fixed reference-KL regularization to yield actor curvature uniformly in x x x , without restricting the upper iterates to a prescribed compact set.
Critic assumptions. The next two assumptions concern feature excitation and value representation. Let ν U \nu_U ν U denote the normalized discounted state–action occupancy under π U \pi_U π U with reset distribution ρ \rho ρ . For a state–action function h h h , define K α h ( s , a ) = E [ h ( s ′ , a ′ ) ∣ s , a ] K_\alpha h(s,a)=\mathbb{E}[h(s',a')\mid s,a] K α h ( s , a ) = E [ h ( s ′ , a ′ ) ∣ s , a ] , where s ′ ∼ P ( ⋅ ∣ s , a ) s'\sim P(\cdot\mid s,a) s ′ ∼ P ( ⋅ ∣ s , a ) and a ′ ∼ π α ( ⋅ ∣ s ′ ) a'\sim\pi_\alpha(\cdot\mid s') a ′ ∼ π α ( ⋅ ∣ s ′ ) . In C1–C2, ( U , S ) (U,S) ( U , S ) ranges over the prospective neighborhood in § 3 ; C2 covers every interpolation α = U + t S \alpha=U+tS α = U + tS , 0 ≤ t ≤ c 0 0\leq t\leq c_0 0 ≤ t ≤ c 0 , including the prospective update segments of Appendix § 9 .§ 5 .
Assumption 4.4 (C1: fixed bounded critic features and excitation). The map φ Q : S × A → R d Q \varphi_Q:\mathcal{S}\times\mathcal{A}\to\mathbb{R}^{d_Q} φ Q : S × A → R d Q and dimension d Q d_Q d Q are fixed independently of x x x , actors, m Q m_Q m Q , and accuracy, with sup s , a ∥ φ Q ( s , a ) ∥ ≤ F Q < ∞ \sup_{s,a}\lVert\varphi_Q(s,a)\rVert\leq F_Q<\infty sup s , a ∥ φ Q ( s , a )∥ ≤ F Q < ∞ . Every base coordinate U U U in the stated neighborhood satisfies E ν U [ φ Q φ Q ⊤ ] ⪰ λ Q I d Q \mathbb{E}_{\nu_U}[\varphi_Q\varphi_Q^\top]\succeq\lambda_QI_{d_Q} E ν U [ φ Q φ Q ⊤ ] ⪰ λ Q I d Q for a uniform λ Q > 0 \lambda_Q>0 λ Q > 0 .
Context and role. C1 rules out unidentifiable critic coefficient directions under discounted reset occupancy, supporting stable TD learning. Related feature conditions appear in linear TD (Bhandari et al., [3 ] , § 2 ) and neural TD (Ke et al., [16 ] ).
Assumption 4.5 (C2: realizability of the regularized lower value). Choose sufficiently large finite τ \tau τ for actor geometry and initialization as in Proposition B.1 and Theorem B.13 , then keep it fixed before selecting critic constants. For every x ∈ R d x x\in\mathbb{R}^{d_x} x ∈ R d x and actor α \alpha α specified above, require
Q x , α t r u e = ( Id − γ K α ) − 1 q x , α ∈ span φ Q , (15) Q_{x,\alpha}^{\mathrm{true}}=(\operatorname{Id}-\gamma K_\alpha)^{-1}q_{x,\alpha}\in\operatorname{span}\varphi_Q, \tag*{(15)} Q x , α true = ( Id − γ K α ) − 1 q x , α ∈ span φ Q , (15)
where q x , α q_{x,\alpha} q x , α , Q x , α t r u e Q_{x,\alpha}^{\mathrm{true}} Q x , α true are in § 2 , Id \operatorname{Id} Id is the identity operator, and span φ Q = { φ Q ⊤ b : b ∈ R d Q } \operatorname{span}\varphi_Q=\{\varphi_Q^{\top}b:b\in\mathbb{R}^{d_Q}\} span φ Q = { φ Q ⊤ b : b ∈ R d Q } .
Context and role. C2 requires the fixed feature span to represent the regularized value family, including its actor-dependent KL cost. Related representation conditions appear in Jin et al. [14 ] ; neural TD retains projection error Ke et al. [16 ] .
Paired accuracy of the nonlinear critic. At the paired population TD target ( u ∗ , v ∗ ) = Z ( p ) (u^*,v^*) = Z(p) ( u ∗ , v ∗ ) = Z ( p ) from (50) , with p = ( x , U , S ) p=(x,U,S) p = ( x , U , S ) , define the value errors e 0 ∗ ( p ) = Q m Q , u ∗ − Q x , U t r u e e_0^*(p)=Q_{m_Q,u^*}-Q_{x,U}^{\mathrm{true}} e 0 ∗ ( p ) = Q m Q , u ∗ − Q x , U true and e c ∗ ( p ) = Q m Q , u ∗ + c v ∗ − Q x , U + c S t r u e e_c^*(p)=Q_{m_Q,u^*+cv^*}-Q_{x,U+cS}^{\mathrm{true}} e c ∗ ( p ) = Q m Q , u ∗ + c v ∗ − Q x , U + c S true . Appendix § 9 .§ 2 proves stability and
A Q : = sup p { ∥ e 0 ∗ ( p ) ∥ ∞ 2 + ∥ e c ∗ ( p ) − e 0 ∗ ( p ) c ∥ ∞ 2 } ≤ C m Q − 2 , (16) A_Q := \sup_p \left\{ \lVert e_0^*(p) \rVert_\infty^2 + \left\lVert\frac{e_c^*(p)-e_0^*(p)}{c} \right\rVert_\infty^2 \right\} \le C m_Q^{-2}, \tag*{(16)} A Q := p sup { ∥ e 0 ∗ ( p ) ∥ ∞ 2 + c e c ∗ ( p ) − e 0 ∗ ( p ) ∞ 2 } ≤ C m Q − 2 , (16)
uniformly over the prescribed domain, with C C C independent of admissible c , m Q c,m_Q c , m Q . The key point is that the critic-error difference is O ( c / m Q ) O(c/m_Q) O ( c / m Q ) : division by c c c preserves the approximation scale. With c 2 ≳ ε c^2 \gtrsim\varepsilon c 2 ≳ ε and m Q ≳ ε − 1 / 2 m_Q \gtrsim\varepsilon^{-1/2} m Q ≳ ε − 1/2 , the paired floor is O ( ε ) O(\varepsilon) O ( ε ) , whereas separate endpoint bounds yield only O ( 1 ) O(1) O ( 1 ) . This is a population approximation bound; sampling and moving-target errors from persistent training are controlled jointly with actor tracking, starting from any deterministic z 0 ∈ K Q z_0 \in\mathcal{K}_Q z 0 ∈ K Q (Appendix § 9 .§ 3 ).(Appendix § 9 .§ 3 ).
Main Result
Now with the assumptions in place the main theorem is stated as follows,
Theorem 4.6 (Single-loop stationarity with a trained paired critic). Fix γ \gamma γ , I I I , K K K , actor depth and retained dimension, and critic feature dimension. Assume A1, A2, G, and C1–C2 for the specified actor and critic (6), at the selected fixed τ \tau τ and neighborhoods. Initialize U 0 = S 0 = 0 U_0=S_0=0 U 0 = S 0 = 0 and any deterministic z 0 ∈ K Q z_0 \in\mathcal{K}_Q z 0 ∈ K Q . There exist fixed projection radii, actor buffers, positive step ratios r Q r_Q r Q , q q q , and positive constants independent of ε \varepsilon ε such that, for sufficiently small ε > 0 \varepsilon>0 ε > 0 , use
β = k β ε , η y = r Q β , η x = q η y , c 2 = k c ε , T = ⌈ k T ε − 2 ⌉ , \beta= k_\beta\varepsilon,\qquad\eta_y = r_Q\beta,\qquad\eta_x = q\eta_y,\qquad c^2 = k_c\varepsilon,\qquad T = \lceil k_T\varepsilon^{-2}\rceil, β = k β ε , η y = r Q β , η x = q η y , c 2 = k c ε , T = ⌈ k T ε − 2 ⌉ , H = Θ ( ( 1 − γ ) − 1 log ( 1 / ε ) ) , m Q ≥ max { m Q , 0 , C m ε − 1 / 2 } . (17) H = \Theta\left((1-\gamma)^{-1}\log(1/\varepsilon)\right),\qquad m_Q \ge\max\{m_{Q,0},C_m\varepsilon^{-1/2}\}. \tag*{(17)} H = Θ ( ( 1 − γ ) − 1 log ( 1/ ε ) ) , m Q ≥ max { m Q , 0 , C m ε − 1/2 } . (17) Choose constants as in Appendix § 9 .§ 5 , with squared truncation biases summing to o ( ε ) o(\varepsilon) o ( ε ) , positive TD/evaluation batches at most polynomial in H H H , and fixed preference batches as in Appendix § 10 .§ 6 . Run Algorithm 1 with δ m e t = ε / 2 \delta_{\mathrm{met}}=\varepsilon/2 δ met = ε /2 , its certified metric, and finite KL solves. Total certificate-failure probability is at most ε \varepsilon ε . The output x o u t x_{\mathrm{out}} x out , uniform among training iterates on success and equal to x 0 x_0 x 0 otherwise, satisfies
E ∥ ∇ Φ τ , m ( x o u t ) ∥ 2 = O ( ε ) , N s e g , N t r a n s = O ~ ( ε − 2 ) , (18) \mathbb{E}\lVert\nabla\Phi_{\tau,m}(x_{\mathrm{out}})\rVert^2 = O(\varepsilon),\qquad N_{\mathrm{seg}},N_{\mathrm{trans}}=\widetilde{O}(\varepsilon^{-2}), \tag*{(18)} E ∥ ∇ Φ τ , m ( x out ) ∥ 2 = O ( ε ) , N seg , N trans = O ( ε − 2 ) , (18) where N s e g , N t r a n s N_{\mathrm{seg}},N_{\mathrm{trans}} N seg , N trans count all sampled trajectory segments and environment transitions, including critic, evaluation, preference, and initialization data. The tilde suppresses logarithmic factors in 1 / ε 1/\varepsilon 1/ ε .
Interpretation of the guarantee. The theorem establishes stationarity of the upper objective induced by the selected policy response at fixed reference-KL regularization. The sample complexity includes all training and initialization data, including persistent critic learning from potentially inaccurate initial value estimates. Finite KL solves reuse stored metric states, with their numerical work accounted for separately (Appendices § 10 .§ 4 and § 9 .§ 6 ). Appendix § 7 .§ 1 compares our optimization target and sample complexity with prior guarantees. Under an additional domain PL condition, the selected response is a global lower minimizer as shown in (Corollary A.2 ).
Proof outline
Write Φ = Φ τ , m \Phi= \Phi_{\tau,m} Φ = Φ τ , m . The derived bounds ∥ ∇ Φ ∥ ≤ G Φ \lVert\nabla\Phi\rVert\le G_{\Phi} ∥ ∇Φ ∥ ≤ G Φ and Φ ≥ Φ ‾ \Phi\ge\underline{\Phi} Φ ≥ Φ hold globally in x x x , with Φ ‾ = 0 \underline{\Phi}=0 Φ = 0 for Bradley–Terry and − B L -B_L − B L otherwise (Equation (43) ). Conditional expectations E t E_t E t use the pre-draw history, including the accepted metric sample. Appendix § 9 follows the six steps below.
1. Derived actor geometry and first-order response correction. (Appendix § 9 .§ 1 .) A 2 A_2 A 2 , the reset component of occupancy, and the architectural derivative bounds yield local reference-KL curvature. Sufficiently strong fixed τ \tau τ dominates the reward and perturbation curvature, giving unique smooth interior base and perturbed responses uniformly in x x x (Proposition B.1 ). Their normalized difference yields V c ( x , α 0 ∗ ( x ) , S 0 ∗ ( x ) ) = ∇ Φ ( x ) + O ( c ) V_c(x,\alpha_0^*(x),S_0^*(x)) = \nabla\Phi(x) + O(c) V c ( x , α 0 ∗ ( x ) , S 0 ∗ ( x )) = ∇Φ ( x ) + O ( c ) (Lemma B.3 ). This identifies the first-order population direction to be tracked, with squared response-approximation bias O ( c 2 ) O(c^2) O ( c 2 ) and without assuming uniform lower-level PL.
2. Stable paired targets and normalized approximation errors. (Appendix § 9 .§ 2 .) The reset identity and Jensen’s inequality give, for square-integrable h h h ,
E ν U [ h ( h − γ K U h ) ] ≥ ( 1 − γ ) ∥ h ∥ L 2 ( ν U ) 2 . (19) \mathbb{E}_{\nu_U}[h(h-\gamma K_Uh)] \ge(1-\sqrt{\gamma})\lVert h\rVert_{L^2(\nu_U)}^2. \tag*{(19)} E ν U [ h ( h − γ K U h )] ≥ ( 1 − γ ) ∥ h ∥ L 2 ( ν U ) 2 . (19)
C1 transfers this coercivity to linear TD; the O ( c ) O(c) O ( c ) actor and O ( m Q − 1 ) O(m_Q^{-1}) O ( m Q − 1 ) nonlinear field and derivative perturbations preserve stability. This gives an interior paired target Z ( p ) Z(p) Z ( p ) with two bounded derivatives (Propositions B.5 and B.6 ). Under C2, comparing the TD roots and their actor derivatives with the realizable value family gives O ( m Q − 1 ) O(m_Q^{-1}) O ( m Q − 1 ) control of both the value error and its derivative along the actor path. Integrating this derivative between the endpoints yields ∥ e c ∗ ( p ) − e 0 ∗ ( p ) ∥ ∞ = O ( c / m Q ) \lVert e_c^*(p)-e_0^*(p)\rVert_\infty= O(c/m_Q) ∥ e c ∗ ( p ) − e 0 ∗ ( p ) ∥ ∞ = O ( c / m Q ) . Together with ∥ e 0 ∗ ( p ) ∥ ∞ = O ( m Q − 1 ) \lVert e_0^*(p)\rVert_\infty= O(m_Q^{-1}) ∥ e 0 ∗ ( p ) ∥ ∞ = O ( m Q − 1 ) , this establishes A Q = O ( m Q − 2 ) A_Q = O(m_Q^{-2}) A Q = O ( m Q − 2 ) in (16), uniformly in the response perturbation (Proposition B.7). Thus shrinking c c c does not amplify the normalized approximation error of the population critic targets. The next step accounts for stochastic tracking of these targets.
3. Shared samples, finite solves, and moving-target tracking. (Appendix § 9 .§ 3 .) Shared, appropriately weighted trajectories control endpoint fluctuations before division by c c c . On the certified neighborhoods, the centered endpoint difference has per-sample second moment O ( c 2 ) O(c^2) O ( c 2 ) , so the normalized correction has variance O ( B − 1 ) O(B^{-1}) O ( B − 1 ) for a batch of B B B shared trajectories, uniformly in c c c . Uniform geometric-length moments permit fixed positive batches (Lemma B.8 ). Evaluation uses the old critic and C ^ = a ^ + c R ^ \widehat{C}=\widehat{a}+c\widehat{R} C = a + c R , while a conditionally independent TD block tracks Z ( p ) Z(p) Z ( p ) (Lemma B.10 ). For moving targets, conditional independence and centering cancel linear noise cross terms, leaving the 1 / β 1/\beta 1/ β penalty only on the mean target drift (Lemma B.9 ). The empirical metric and finite KL solves give endpoint error at most c η γ 2 c\eta_\gamma^2 c η γ 2 , hence normalized error O ( η γ 2 ) O(\eta_\gamma^2) O ( η γ 2 ) . These solves reuse stored metric states and require no additional environment samples (Propositions C.33 and C.37).
4. Joint tracking contraction. (Appendix § 9 .§ 4 .) Define the critic tracking error C t = ∥ z t − Z ( p t ) ∥ W 2 C_t=\lVert z_t-Z(p_t)\rVert_W^2 C t = ∥ z t − Z ( p t ) ∥ W 2 . Let T t \mathcal{T}_t T t be the nonnegative actor tracking potential in (41), controlling errors in U t , S t U_t,S_t U t , S t . For a fixed positive weight λ \lambda λ , set
D t = T t + λ r Q C t . (20) D_t=\mathcal{T}_t+\lambda r_Q C_t. \tag*{(20)} D t = T t + λ r Q C t . (20)
Sufficiently small fixed step ratios absorb the mutual actor–critic error terms and give, for fixed ρ D , C D > 0 \rho_D,C_D>0 ρ D , C D > 0 (Theorem B.11 ),
E t D t + 1 ≤ ( 1 − ρ D η γ ) D t + C D η γ ( q G Φ 2 + ω + β ) , (21) \mathbb{E}_tD_{t+1}\le(1-\rho_D\eta_\gamma)D_t+C_D\eta_\gamma(qG_\Phi^2+\omega+\beta), \tag*{(21)} E t D t + 1 ≤ ( 1 − ρ D η γ ) D t + C D η γ ( q G Φ 2 + ω + β ) , (21)
ω = c 2 + A Q + b e v 2 + b T D 2 , \omega=c^2+A_Q+b_{\mathrm{ev}}^2+b_{\mathrm{TD}}^2, ω = c 2 + A Q + b ev 2 + b TD 2 ,
where b e v , b T D b_{\mathrm{ev}}, b_{\mathrm{TD}} b ev , b TD bound truncation biases. This inequality holds before exit and through the prospective next step in the larger analytic neighborhood. The term q G Φ 2 qG_{\Phi}^{2} q G Φ 2 bounds target movement for containment; Step 6 retains the gradient term explicitly to avoid introducing this constant into the stationarity floor.
5. Proving containment of the coupled iterates. (Appendix B.5) Choose actor geometry and fixed τ \tau τ , then critic radii and potential weights, followed by r Q r_Q r Q to accommodate the permitted initial critic error and q q q to control target drift. Buffered prospective-step estimates justify the local recursions before containment is known. A stopped martingale bound then controls D t D_t D t : under (17) , fluctuations of order η y log ( T / ε ) poly ( H ) \sqrt{\eta_y}\log(T/\varepsilon)\operatorname{poly}(H) η y log ( T / ε ) poly ( H ) fit inside the actor buffers. This excludes a first actor exit with conditional probability at least 1 − ε / 2 1-\varepsilon/2 1 − ε /2 for every accepted metric realization (Theorems B.12 and B.13 ). Projection contains the critic, and G G G supplies global bounds in the upper variable.
6. Upper stationarity and sample complexity. (Appendix B.6) With fixed positive weights and constants, and Δ Ψ t = Ψ t + 1 − Ψ t \Delta\Psi_t=\Psi_{t+1}-\Psi_t Δ Ψ t = Ψ t + 1 − Ψ t , upper descent and tracking give (Proposition B.14 )
Ψ t = Φ ( x t ) − Φ + a Ψ q T t + b Ψ q r Q C t , \Psi_t=\Phi(x_t)-\Phi+a_{\Psi}qT_t+b_{\Psi}qr_QC_t, Ψ t = Φ ( x t ) − Φ + a Ψ q T t + b Ψ q r Q C t ,
E t Δ Ψ t ≤ − a ∗ η x ( ∥ ∇ Φ ( x t ) ∥ 2 + T t + C t ) + C ∗ η x ( ω + β + η y + η x ) . (22) \mathbb{E}_t\Delta\Psi_t\leq-a_*\eta_x\left(\lVert\nabla\Phi(x_t)\rVert^2+T_t+C_t\right)+C_*\eta_x(\omega+\beta+\eta_y+\eta_x). \tag*{(22)} E t Δ Ψ t ≤ − a ∗ η x ( ∥ ∇Φ ( x t ) ∥ 2 + T t + C t ) + C ∗ η x ( ω + β + η y + η x ) . (22)
Telescoping with predictable pre-exit indicators and terminal nonnegativity avoids conditioning on future containment. Uniform output on success and fallback on failure yield
E ∥ ∇ Φ ( x o u t ) ∥ 2 ≤ C [ Ψ 0 η x T + ω + β + η y + η x ] + ε G Φ 2 . (23) \mathbb{E}\lVert\nabla\Phi(x_{\mathrm{out}})\rVert^2\leq C\left[\frac{\Psi_0}{\eta_xT}+\omega+\beta+\eta_y+\eta_x\right]+\varepsilon G_{\Phi}^{2}. \tag*{(23)} E ∥ ∇Φ ( x out ) ∥ 2 ≤ C [ η x T Ψ 0 + ω + β + η y + η x ] + ε G Φ 2 . (23)
The initial potential includes critic error. Under (17) , c 2 = Θ ( ε ) c^2=\Theta(\varepsilon) c 2 = Θ ( ε ) makes the squared response-secant bias O ( ε ) O(\varepsilon) O ( ε ) , while m Q − 2 = O ( ε ) m_Q^{-2}=O(\varepsilon) m Q − 2 = O ( ε ) gives the same order for the paired critic approximation floor. Truncation ensures b e v 2 + b T D 2 = o ( ε ) b_{\mathrm{ev}}^2+b_{\mathrm{TD}}^2=o(\varepsilon) b ev 2 + b TD 2 = o ( ε ) , hence ω = O ( ε ) \omega=O(\varepsilon) ω = O ( ε ) . The choices η x , η y , β = Θ ( ε ) \eta_x,\eta_y,\beta=\Theta(\varepsilon) η x , η y , β = Θ ( ε ) and T = Θ ( ε − 2 ) T=\Theta(\varepsilon^{-2}) T = Θ ( ε − 2 ) also give Ψ 0 / ( η x T ) = O ( ε ) \Psi_0/(\eta_xT)=O(\varepsilon) Ψ 0 / ( η x T ) = O ( ε ) , so every term in the bound is O ( ε ) O(\varepsilon) O ( ε ) . The uniform sampling bounds from Step 3 allow these choices without a polynomial increase in batch size as c c c shrinks. Counting TD, response-evaluation, preference, and initialization samples therefore gives O ~ ( ε − 2 ) \widetilde{O}(\varepsilon^{-2}) O ( ε − 2 ) sample complexity. Appendix A.1.1 explains the O ~ ( ε − 3 ) \widetilde{O}(\varepsilon^{-3}) O ( ε − 3 ) sample complexity established by Gaur et al. [10 ] and the source of our improved bound.
Empirical Results
We focus on interaction scaling in the exact continuous-state construction and end-to-end performance on Walker Walk.
We use the exact continuous-state, two-action construction in Appendix C.11, with a tied multilayer actor and a fixed-feature nonlinear critic. Training uses sampled trajectories; exact reference calculations provide the upper-gradient evaluation metric. Figure 1 (a) shows similar decay of the average squared upper-gradient norm for an initially exact zero critic and a displaced critic with inaccurate value estimates, consistent with the predicted N − 1 / 2 N^{-1/2} N − 1/2 trend and O ~ ( ε − 2 ) \widetilde{O}(\varepsilon^{-2}) O ( ε − 2 ) interaction complexity. The prior works in Table 1 do not report this direct check of the predicted stationarity scaling against total sampled interactions. On Walker Walk, we learn rewards from trajectory preferences and incorporate the response correction into reward updates while training a continuous-control policy. Figure 1 (b) shows higher mean returns than the adapted method of Gaur et al. [10 ] . Appendices D.1.1 and D.1.2 explain the scaling and initializations; Appendices D and E give protocols and implementation qualifications for the exact and end-to-end cases respectively.
Table 1. Analyzed BRL guarantees. SL: single-loop stochastic tracking; FO: first-order implementation. Nonlinear A–C analysis requires explicit nonlinear actors and critics with coupled training and critic-error control. Dashes mark unestablished guarantees or unreported comparable sample complexitys. Rates use O ( ε ) O(\varepsilon) O ( ε ) squared stationarity under distinct targets and assumptions. {sup}†Stoc-SoBiRL; its deterministic model-based variant uses $O(1)$ inner updates. {sup}‡Fixed regularization, Theorem 2 of version 6; best-iterate output. Our uniform-output bound concerns a selected local regularized response, globally optimal on D \mathcal{D} D under additional PL (Corollary A.2 ).
Figure 1. Interaction scaling and continuous control. (a) Average squared upper-gradient norm, G ‾ T = T − 1 ∑ t < T ∣ Φ ′ ( x t ) ∣ 2 \overline{G}_T = T^{-1} \sum_{t<T} |\Phi'(x_t)|^2 G T = T − 1 ∑ t < T ∣ Φ ′ ( x t ) ∣ 2 , versus total training cost N N N , for zero and displaced critic initialization. Error bars show 95% t t t intervals over ten seeds; the dashed line is an N − 1 / 2 N^{-1/2} N − 1/2 reference. (b) Walker Walk evaluation returns: means and one-standard-deviation bands across five matched training seeds.
Conclusion
We developed a single-loop, fully first-order bilevel RL method with explicit neural actors and a persistently trained nonlinear critic. We proved O ~ ( ε − 2 ) \widetilde{O}(\varepsilon^{-2}) O ( ε − 2 ) sample complexity for O ( ε ) O(\varepsilon) O ( ε ) expected squared upper-gradient norm under the stated assumptions.
AI use statement
Generative AI tools were used for writing assistance. The authors verified all mathematical statements, proofs, citations, and experimental results, and take full responsibility for the content of the submission.
Reproducibility Statement
The supplementary material includes the experiment code, dependencies, configurations, and seeds, with setup instructions in the README in order to rerun the experiments from scratch, generate all given figures.
Related Work
Standard reinforcement learning and actor–critic methods. Classical reinforcement learning (RL) methods optimize a single policy objective, with policy-gradient methods providing a framework for direct policy optimization under function approximation [28 , 25 ] . Actor–critic algorithms combine policy improvement with value-function estimation, with early convergence analyses establishing their properties under two-timescale stochastic approximation [17 , 4 , 8 ] . Subsequent work has developed finite-sample guarantees for temporal difference learning and actor–critic methods under function approximation and Markovian sampling [3 , 29 , 1 , 9 ] . Neural policy-gradient analyses have further established convergence and optimality guarantees under suitable overparameterization and representation assumptions [27 ] . These developments motivate our use of first-order policy updates and persistent critic learning. However, standard RL analyses typically optimize a fixed policy objective rather than an upper objective induced by an implicitly defined policy response. In bilevel RL, the lower-level policy changes with the upper-level parameter, introducing additional response-sensitivity and estimation errors. In particular, ordinary critic accuracy guarantees do not directly control the normalized differences of critic errors required by our response correction. Our analysis addresses this additional coupling through local response geometry, shared sampling, and paired critic tracking for the specified actor–critic architecture.
Bilevel optimization and RL. Implicit bilevel methods control Hessian inverses or iterative hypergradients ([7 , 11 , 13 ] ); penalty methods permit first-order stochastic updates ([18 ] ). In BRL, PARL uses implicit response calculations ([5 ] ), penalty approaches prescribe lower-solution accuracy ([24 ] ), and soft Bellman structure yields tabular hypergradients ([30 ] ). Gaur et al. [10 ] analyze continuous state–action spaces under uniform lower PL, with value-estimation error controlled by an approximation assumption and an imported actor–critic bound. Zeng et al. [31 ] give a single-loop tabular actor–critic guarantee under regularization-dependent PL and Hessian conditions. Table 1 compares assumptions; Section A.1 distinguishes targets.
Critic construction and coupled learning. Prior BRL implementations already use neural critics ([24 , 30 ] ). We analyze an explicit nonlinear critic whose persistent paired updates track moving values and control the normalized difference of endpoint errors as the response perturbation shrinks. Separate endpoint accuracy need not provide this control after division by the perturbation. The proof requires fixed bounded features, excitation, and realizability of the regularized value family (Appendices § 9 .§ 2 and § 10 .11); it does not establish unrestricted feature learning or nonlinear expressive superiority.
Policy geometry and TD. Natural policy gradients and trust regions motivate Fisher geometry ([15 , 23 , 1 ] ). Our initialization certificate yields local response curvature, in contrast to domain-wide Fisher nondegeneracy ([21 , 6 ] ). Feature-conditioned TD stability ([3 , 16 ] ) supports our nonlinear paired tracking proof. The context after A1–C2 details the restrictions.
Bilevel optimization. Bilevel optimization couples an upper objective to the solution of a parameterized lower problem. With smooth, strongly convex lower objectives, the unique lower response supports convergence analyses based on implicit differentiation or differentiation through lower updates ([11 , 13 ] ). For convex lower objectives with multiple minimizers, bilevel descent aggregation incorporates the upper objective when selecting among lower solutions ([19 ] ). Nonconvex lower problems introduce further difficulties: stationary points can be suboptimal, and the response reached by optimization can depend on initialization. Approaches to this setting include value-function reformulations ([20 ] ) and selection maps that explicitly specify which lower critical point defines the upper objective ([2 ] ). Our work develops this local-response perspective for BRL, deriving response curvature for explicit neural actors and controlling the coupled learning of policies and persistent nonlinear critics.
Comparison with lower bounds. Ji [12 ] establish an Ω ( δ − 4 ) \Omega(\delta^{-4}) Ω ( δ − 4 ) stochastic oracle lower bound for zero-respecting first-order methods on nonconvex-strongly-convex bilevel problems, where δ \delta δ denotes expected gradient-mapping norm accuracy. With fixed problem constants and δ = ϵ \delta= \sqrt{\epsilon} δ = ϵ , this yields an Ω ( ϵ − 2 ) \Omega(\epsilon^{-2}) Ω ( ϵ − 2 ) benchmark for squared stationarity accuracy, matching the accuracy exponent of our O ~ ( ϵ − 2 ) \widetilde{O}(\epsilon^{-2}) O ( ϵ − 2 ) interaction bound. This comparison does not establish minimax optimality for our structured RL setting, since the problem classes and oracle models differ; matching condition-number dependence also remains open.
Optimization target and comparison with prior guarantees
The target Φ τ , m \Phi_{\tau,\mathrm{m}} Φ τ , m uses fixed reference-KL regularization: c c c is an estimation perturbation with vanishing bias, whereas τ \tau τ remains part of the lower problem. Yang et al. [30 ] use a global entropy-regularized tabular response; Gaur et al. [10 ] global lower responses in their parameter class under uniform PL; Shen et al. [24 ] a penalty reformulation linked to lower optimality; and Zeng et al. [31 ] fixed and vanishing regularization under their geometric conditions. Our explicit structure yields local curvature and joint containment. We can show that under similar assumptions to Gaur et al. [10 ] we can attain the same stationarity guarantee for a selected globally optimal lower response.
Assumption A.1 (Optional PL on a prescribed actor domain). Fix the finite actor architecture m \mathrm{m} m and τ \tau τ of Theorem 4.6 , and a compact convex D ⊂ R r \mathcal{D} \subset\mathbb{R}^{r} D ⊂ R r with B ‾ g e o m ⊂ int D \overline{B}_{\mathrm{geom}} \subset\operatorname{int}\mathcal{D} B geom ⊂ int D , independent of x , c , m Q , ε x,c,m_Q,\varepsilon x , c , m Q , ε . With g τ , m , D ∗ ( x ) = min α ∈ D g τ , m ( x , α ) g_{\tau,\mathrm{m},\mathcal{D}}^*(x)=\min_{\alpha\in\mathcal{D}}g_{\tau,\mathrm{m}}(x,\alpha) g τ , m , D ∗ ( x ) = min α ∈ D g τ , m ( x , α ) , assume a constant μ P L > 0 \mu_{\mathrm{PL}}>0 μ PL > 0 independent of x , ε x,\varepsilon x , ε such that, for all x ∈ R d x x\in\mathbb{R}^{d_x} x ∈ R d x and α ∈ D \alpha\in\mathcal{D} α ∈ D ,
1 2 ∥ ∇ α g τ , m ( x , α ) ∥ 2 ≥ μ P L [ g τ , m ( x , α ) − g τ , m , D ∗ ( x ) ] . (24) \frac{1}{2}\lVert\nabla_{\alpha}g_{\tau,\mathrm{m}}(x,\alpha)\rVert^2 \ge\mu_{\mathrm{PL}}\left[g_{\tau,\mathrm{m}}(x,\alpha)-g_{\tau,\mathrm{m},\mathcal{D}}^*(x)\right]. \tag*{(24)} 2 1 ∥ ∇ α g τ , m ( x , α ) ∥ 2 ≥ μ PL [ g τ , m ( x , α ) − g τ , m , D ∗ ( x ) ] . (24) The gradient is the ambient Euclidean gradient, including at the boundary.
Corollary A.2 (Selected global regularized response). Under Theorem 4.6 ’s conditions and Assumption A.1 , α 0 , m ∗ ( x ) ∈ arg min α ∈ D g τ , m ( x , α ) \alpha_{0,\mathrm{m}}^*(x)\in\arg\min_{\alpha\in\mathcal{D}}g_{\tau,\mathrm{m}}(x,\alpha) α 0 , m ∗ ( x ) ∈ arg min α ∈ D g τ , m ( x , α ) for every x x x . Thus Φ τ , m , D s e l ( x ) : = f m ( x , α 0 , m ∗ ( x ) ) ≡ Φ τ , m ( x ) \Phi_{\tau,\mathrm{m},\mathcal{D}}^{\mathrm{sel}}(x):=f_{\mathrm{m}}(x,\alpha_{0,\mathrm{m}}^*(x))\equiv\Phi_{\tau,\mathrm{m}}(x) Φ τ , m , D sel ( x ) := f m ( x , α 0 , m ∗ ( x )) ≡ Φ τ , m ( x ) is a selected global-response objective, and the same output satisfies (18) with unchanged label and numerical costs.
Appendix § 10 .12 proves the corollary, gives selection-independence conditions, and rules out full-space PL for this bounded architecture. Initialization, fixed τ \tau τ , and upper stationarity are unchanged.
Source of the sample-complexity improvement
We compare the sample-complexity exponents for squared-gradient stationarity, retaining the distinctions between optimization targets and assumptions discussed above.
The published analysis. In the notation of Gaur et al. [10 ] , let σ \sigma σ denote the perturbation, T G , K G T_{\mathrm{G}},K_{\mathrm{G}} T G , K G the outer and inner iteration counts, n G , B G n_{\mathrm{G}},B_{\mathrm{G}} n G , B G the sampling budgets, and H G H_{\mathrm{G}} H G the rollout horizon. Writing S G S_{\mathrm{G}} S G for their averaged squared-stationarity criterion and ε a p p r o x , G \varepsilon_{\mathrm{approx},\mathrm{G}} ε approx , G for their approximation term, their bound has the form
S G ≤ O ~ ( 1 T G + e − K G σ 2 + 1 σ 2 n G + γ 2 H G σ 2 B G + σ 2 + ε a p p r o x , G ) . (25) S_{\mathrm{G}}\le\widetilde{O}\left(\frac{1}{T_{\mathrm{G}}}+\frac{e^{-K_{\mathrm{G}}}}{\sigma^2}+\frac{1}{\sigma^2 n_{\mathrm{G}}}+\frac{\gamma^{2H_{\mathrm{G}}}}{\sigma^2 B_{\mathrm{G}}}+\sigma^2+\varepsilon_{\mathrm{approx},\mathrm{G}}\right). \tag*{(25)} S G ≤ O ( T G 1 + σ 2 e − K G + σ 2 n G 1 + σ 2 B G γ 2 H G + σ 2 + ε approx , G ) . (25)
Their stated schedule uses, up to logarithmic factors,
σ 2 ≍ ε , T G ≍ ε − 1 , n G , B G ≍ ε − 2 , \sigma^2\asymp\varepsilon,\qquad T_{\mathrm{G}}\asymp\varepsilon^{-1},\qquad n_{\mathrm{G}},B_{\mathrm{G}}\asymp\varepsilon^{-2}, σ 2 ≍ ε , T G ≍ ε − 1 , n G , B G ≍ ε − 2 ,
K G , H G = O ( log ( 1 / ε ) ) . (26) K_{\mathrm{G}},H_{\mathrm{G}}=O(\log(1/\varepsilon)). \tag*{(26)} K G , H G = O ( log ( 1/ ε )) . (26)
This gives S G = O ~ ( ε + ε a p p r o x , G ) S_{\mathrm{G}}=\widetilde{O}(\varepsilon+\varepsilon_{\mathrm{approx},\mathrm{G}}) S G = O ( ε + ε approx , G ) and the sample count
N G = n G K G T G + B G K G H G T G + B G H G T G = O ~ ( ε − 3 ) . (27) N_{\mathrm{G}}=n_{\mathrm{G}}K_{\mathrm{G}}T_{\mathrm{G}}+B_{\mathrm{G}}K_{\mathrm{G}}H_{\mathrm{G}}T_{\mathrm{G}}+B_{\mathrm{G}}H_{\mathrm{G}}T_{\mathrm{G}}=\widetilde{O}(\varepsilon^{-3}). \tag*{(27)} N G = n G K G T G + B G K G H G T G + B G H G T G = O ( ε − 3 ) . (27)
The term 1 / ( σ 2 n G ) 1/(\sigma^2n_{\mathrm{G}}) 1/ ( σ 2 n G ) exposes the estimation cost: with σ 2 ≍ ε \sigma^2\asymp\varepsilon σ 2 ≍ ε , taking n G ≍ ε − 2 n_{\mathrm{G}}\asymp\varepsilon^{-2} n G ≍ ε − 2 makes this contribution O ( ε ) O(\varepsilon) O ( ε ) .
Joint control of normalized response errors. Our analysis controls endpoint differences jointly before dividing by the response perturbation. Let ζ ˉ 0 , ζ ˉ c \bar{\zeta}_0,\bar{\zeta}_c ζ ˉ 0 , ζ ˉ c denote the centered endpoint errors averaged over B B B shared trajectories. On the prescribed actor and critic neighborhoods,
E t ∥ ζ ˉ c − ζ ˉ 0 ∥ 2 ≤ C c 2 B , E t ∥ ζ ˉ c − ζ ˉ 0 c ∥ 2 ≤ C B , (28) \mathbb{E}_t\left\|\bar{\zeta}_c-\bar{\zeta}_0\right\|^2 \le\frac{Cc^2}{B}, \qquad\mathbb{E}_t\left\|\frac{\bar{\zeta}_c-\bar{\zeta}_0}{c}\right\|^2 \le\frac{C}{B}, \tag*{(28)} E t ζ ˉ c − ζ ˉ 0 2 ≤ B C c 2 , E t c ζ ˉ c − ζ ˉ 0 2 ≤ B C , (28)
with C C C independent of c c c and B B B (Lemma B.8 ). The shared-trajectory construction and its weighting therefore control the sampling variance uniformly as the perturbation shrinks.
The critic approximation requires a corresponding joint estimate. The paired population TD targets satisfy
∥ e 0 ∗ ( p ) ∥ ∞ 2 + ∥ e c ∗ ( p ) − e 0 ∗ ( p ) c ∥ ∞ 2 ≤ C m Q − 2 , (29) \left\|e_0^*(p)\right\|_\infty^2+\left\|\frac{e_c^*(p)-e_0^*(p)}{c}\right\|_\infty^2 \le Cm_Q^{-2}, \tag*{(29)} ∥ e 0 ∗ ( p ) ∥ ∞ 2 + c e c ∗ ( p ) − e 0 ∗ ( p ) ∞ 2 ≤ C m Q − 2 , (29)
uniformly over the prescribed neighborhood (Proposition B.7). Thus neither centered sampling noise nor the normalized critic approximation floor introduces an inverse power of c c c . The moving-target analysis additionally accounts for the error incurred while learning these critics persistently (Appendix § 9 .§ 3 ).
Propagation to the final sample complexity. The joint descent and tracking bound ((23) ) combines these estimates with the initial-error contribution Ψ 0 / ( η x T ) \Psi_0/(\eta_xT) Ψ 0 / ( η x T ) . Under ((17) ,
c 2 = Θ ( ε ) , m Q − 2 = O ( ε ) , c^2=\Theta(\varepsilon), \qquad m_Q^{-2}=O(\varepsilon), c 2 = Θ ( ε ) , m Q − 2 = O ( ε ) ,
η x , η y , β = Θ ( ε ) , T = Θ ( ε − 2 ) . (30) \eta_x,\eta_y,\beta=\Theta(\varepsilon), \qquad T=\Theta(\varepsilon^{-2}). \tag*{(30)} η x , η y , β = Θ ( ε ) , T = Θ ( ε − 2 ) . (30)
The squared secant bias, paired critic approximation floor, step-size-weighted stochastic terms, and initial-error contribution are then all O ( ε ) O(\varepsilon) O ( ε ) . The prescribed truncation and metric-certificate accuracies control the remaining terms.
Uniform sampling moments allow fixed or logarithmically growing batches. Consequently, counting all TD, response-evaluation, preference, and initialization samples gives
N o u r s = O ~ ( T ) = O ~ ( ε − 2 ) , (31) N_{\mathrm{ours}}=\widetilde{O}(T)=\widetilde{O}(\varepsilon^{-2}), \tag*{(31)} N ours = O ( T ) = O ( ε − 2 ) , (31)
including critic training. The improved exponent follows from joint control of errors in the divided response correction together with persistent actor–critic tracking. This comparison concerns the guarantees established by the two analyses under their respective assumptions.
Organization of the Appendix
The appendix separates the proof of the main theorem, its supporting constructions and estimates, and the experimental evidence. Appendix § 9 gives the main proof sequence; Appendix § 10 supplies its technical ingredients. Appendices D–E describe the experiments.
Appendix B: Main results. This section proves Theorem 4.6 in the order of the six-step proof outline in the main text. Section B.1 establishes the actor geometry and response correction. Section B.2 derives stable paired-TD targets and their approximation accuracy, while Section B.3 controls shared-sample estimation and persistent tracking. Sections B.4 and B.5 construct a contracting potential and exclude a first exit from the admissible region. Section B.6 then proves upper-objective stationarity and accounts for the data used in training and metric estimation.
Appendix C: Supporting results. Sections C.1–C.3 develop the multilayer actor construction, initialization certificates, preference-loss derivatives, and response calculus. Section C.4 establishes the common-KL geometry and implements the actor updates through an empirical metric and a finite local solver, including the effect of numerical errors. Sections C.5–C.8 provide the critic, sampling, conditional-tracking, and concentration estimates used in the main proof. Sections C.9 and C.10 discuss assumption verification and counterexamples to uniform lower-level PL guarantees. Section C.11 constructs a continuous-state instance with an explicit response, verified critic-feature conditions, and a nonzero chain correction; it also clarifies the interpretation of the on-policy preference objective. Section C.12 proves the optional global-response corollary, states selection-independence conditions, and explains the obstruction to uniform PL on the full actor space.
Appendix D: Controlled continuous-state experiments. This section specifies the actor and nonlinear critic, sampled training protocol, initialization comparisons, and reference calculations. It reports response-secant bias, shared versus independent sampling variance, critic approximation and tracking, and stationarity versus charged interactions. It also records the controls, uncertainty estimates, numerical safeguards, pilot history, and qualifications of the finite-budget scaling results.
Appendix E: End-to-end benchmarks. This section documents Walker Walk and Highway-fast, including the SAC implementation, preference collection, baselines, architectures, hyperparameters, learning curves, and checkpoint and per-seed returns. It also gives the uncertainty estimates and accounting of main and auxiliary training interactions.
Main Results
This section proves Theorem 4.6 under A1, A2, G and C1–C2, following the six steps of the main-text proof outline. The target is the selected fixed-reference-KL response objective; the dynamics are fixed, actions are finite, and preferences use fixed finite tuples. The proofs of the actor construction and auxiliary calculations are in Appendix § 10 .
Notation and sampling history. Suppress actor-width subscripts and write g = g τ g = g_{\tau} g = g τ , Φ = Φ τ , m \Phi= \Phi_{\tau,m} Φ = Φ τ , m , p = ( x , U , S ) p = (x, U, S) p = ( x , U , S ) , α 0 = U \alpha_0 = U α 0 = U , and α c = U + c S \alpha_c = U + cS α c = U + c S . All norms are Euclidean or induced operator norms unless a subscript is given. Let F t \mathcal{F}_t F t be the history before the current data blocks, including the old actors, upper parameter, critic, and independent initial metric sample. On acceptance, the common-KL law is fixed at p ^ n \hat{p}_n p ^ n throughout training, with the certificate of Proposition C.30 . Write E t = E [ ⋅ ∣ F t ] \mathbb{E}_t = \mathbb{E}[\cdot\mid\mathcal{F}_t] E t = E [ ⋅ ∣ F t ] . The lower, outer and TD blocks are independent conditional on this history; the two preference blocks are separate and use the old perturbed actor. If needed, G t \mathcal{G}_t G t additionally reveals the lower block and actor updates; outer samples remain fresh under that conditioning. The new critic is first used at the next iteration.
Constants denoted C C C may change between inequalities. They may depend on the fixed discount, tuple sizes, actor depth and retained dimensions, feature conditioning, and the sufficiently large but fixed τ \tau τ . They are independent of x x x , accuracy, 0 ≤ c ≤ c 0 0 \le c \le c_0 0 ≤ c ≤ c 0 , m Q ≥ m Q , 0 m_Q \ge m_{Q,0} m Q ≥ m Q , 0 , and admissible actor widths. Uniformity under later reductions of step ratios is stated where needed. L L L denotes actor depth, K K K preference trajectory length, H H H the truncation cap and N N N a geometric prefix length.
Policy transition and reset occupancy. For bounded measurable h h h , the policy transition operator and normalized discounted state–action occupancy used in C1–C2 are
( K α h ) ( s , a ) = E s ′ ∼ P ( ⋅ ∣ s , a ) ∑ a ′ ∈ A π α ( a ′ ∣ s ′ ) h ( s ′ , a ′ ) , (K_{\alpha}h)(s,a)=\mathbb{E}_{s'\sim P(\cdot\mid s,a)}\sum_{a'\in\mathcal{A}}\pi_{\alpha}(a'\mid s')h(s',a'), ( K α h ) ( s , a ) = E s ′ ∼ P ( ⋅ ∣ s , a ) a ′ ∈ A ∑ π α ( a ′ ∣ s ′ ) h ( s ′ , a ′ ) ,
ρ U ( d s , d a ) = ρ ( d s ) π U ( d a ∣ s ) , ν U = ( 1 − γ ) ∑ k ≥ 0 γ k ρ U K U k . (32) \rho^{U}(ds,da)=\rho(ds)\pi_{U}(da\mid s),\qquad\nu_{U}=(1-\gamma)\sum_{k\geq0}\gamma^{k}\rho_{U}K_{U}^{k}. \tag*{(32)} ρ U ( d s , d a ) = ρ ( d s ) π U ( d a ∣ s ) , ν U = ( 1 − γ ) k ≥ 0 ∑ γ k ρ U K U k . (32)
Thus ν U = ( 1 − γ ) ρ U + γ ν U K U \nu_{U}=(1-\gamma)\rho_{U}+\gamma\nu_{U}K_{U} ν U = ( 1 − γ ) ρ U + γ ν U K U ; its state marginal is d U γ d_{U}^{\gamma} d U γ .
Actor geometry and response correction
Write j ( x , α ) = ( 1 − γ ) E π α ∑ t ≥ 0 γ t r x ( s t , a t ) j(x,\alpha)=(1-\gamma)\mathbb{E}_{\pi_{\alpha}}\sum_{t\geq0}\gamma^{t}r_{x}(s_{t},a_{t}) j ( x , α ) = ( 1 − γ ) E π α ∑ t ≥ 0 γ t r x ( s t , a t ) and K ( α ) = ( 1 − γ ) E π α ∑ t ≥ 0 γ t log ( π α / π r e f ) ( a t ∣ s t ) \mathcal{K}(\alpha)=(1-\gamma)\mathbb{E}_{\pi_{\alpha}}\sum_{t\geq0}\gamma^{t}\log(\pi_{\alpha}/\pi_{\mathrm{ref}})(a_{t}\mid s_{t}) K ( α ) = ( 1 − γ ) E π α ∑ t ≥ 0 γ t log ( π α / π ref ) ( a t ∣ s t ) , so g = − j + τ K g=-j+\tau\mathcal{K} g = − j + τ K . Put h c = g + c f h_{c}=g+cf h c = g + c f . The construction in Appendix § 10 .§ 1 supplies actor derivative bounds C k = C k , L C_{k}=C_{k,L} C k = C k , L and the KL Hessian Lipschitz constant L K L_{K} L K , independent of x x x , width, and τ \tau τ . Let a 0 = ( 1 − γ ) λ e f f , L a_{0}=(1-\gamma)\lambda_{\mathrm{eff},L} a 0 = ( 1 − γ ) λ eff , L and κ = a 0 / 4 \kappa=a_{0}/4 κ = a 0 /4 , and choose
0 < R = R A = R g e o m ≤ min { A / 2 , a 0 / ( 2 L K ) , λ e f f , L / ( 2 C 2 ) } . (33) 0<R=R_{A}=R_{\mathrm{geom}}\leq\min\{A/2,a_{0}/(2L_{K}),\sqrt{\lambda_{\mathrm{eff},L}/(2C_{2})}\}. \tag*{(33)} 0 < R = R A = R geom ≤ min { A /2 , a 0 / ( 2 L K ) , λ eff , L / ( 2 C 2 ) } . (33)
Omit a restriction with zero denominator. Write B R B_{R} B R for the open ball and Θ R = B ‾ R \Theta_{R}=\overline{B}_{R} Θ R = B R for its closure. Define J q = B r A q ( γ ) J_{q}=B_{r}A_{q}(\gamma) J q = B r A q ( γ ) , G f = B L b 1 ( I K ) G_{f}=B_{L}b_{1}(I_{K}) G f = B L b 1 ( I K ) and L f = B L b 2 ( I K ) L_{f}=B_{L}b_{2}(I_{K}) L f = B L b 2 ( I K ) using the likelihood polynomials in Appendix § 10 .§ 2 . Fix a separation radius s = S t u b e > 0 s=S_{\mathrm{tube}}>0 s = S tube > 0 independently of τ \tau τ ; reduce the preliminary cap c 0 c_{0} c 0 so c 0 s ≤ R / 8 c_{0}s\leq R/8 c 0 s ≤ R /8 . The common-KL radius r p r o x ≤ R / 8 r_{\mathrm{prox}}\leq R/8 r prox ≤ R /8 is chosen in Appendix § 10 .§ 4 . All actor bounds below are used where ∥ S ∥ ≤ s \lVert S\rVert\leq s ∥ S ∥ ≤ s and U + c B s ⊂ B R U+cB_{s}\subset B_{R} U + c B s ⊂ B R , with the additional local-update buffer specified in Step B.5.
Proposition B.1 (One certified actor neighborhood for every x x x ). For a fixed perturbation cap c 0 c_{0} c 0 , choose a finite τ ≥ 1 \tau\geq1 τ ≥ 1 satisfying
τ ≥ 4 ( J 2 + c 0 L f ) a 0 , J 1 + c 0 G f κ τ < R 4 . (34) \tau\geq\frac{4(J_{2}+c_{0}L_{f})}{a_{0}},\qquad\frac{J_{1}+c_{0}G_{f}}{\kappa\tau}<\frac{R}{4}. \tag*{(34)} τ ≥ a 0 4 ( J 2 + c 0 L f ) , κ τ J 1 + c 0 G f < 4 R . (34) Then h c = g τ + c f h_{c}=g_{\tau}+cf h c = g τ + c f has actor Hessian at least μ τ I \mu_{\tau}I μ τ I , μ τ = κ τ \mu_{\tau}=\kappa\tau μ τ = κ τ , on B R B_{R} B R for every x x x and 0 ≤ c ≤ c 0 0\leq c\leq c_{0} 0 ≤ c ≤ c 0 . Its unique root on this ball obeys
∥ α c ⋆ ( x ) ∥ ≤ J 1 + c G f μ τ < R / 4 , ∥ α c ⋆ ( x ) − α 0 ⋆ ( x ) ∥ ≤ c G f μ τ . (35) \lVert\alpha_{c}^{\star}(x)\rVert\leq\frac{J_{1}+cG_{f}}{\mu_{\tau}}<R/4,\qquad\lVert\alpha_{c}^{\star}(x)-\alpha_{0}^{\star}(x)\rVert\leq\frac{cG_{f}}{\mu_{\tau}}. \tag*{(35)} ∥ α c ⋆ ( x )∥ ≤ μ τ J 1 + c G f < R /4 , ∥ α c ⋆ ( x ) − α 0 ⋆ ( x )∥ ≤ μ τ c G f . (35) The locally defined roots agree on overlaps and give a globally defined selected response on R d x \mathbb{R}^{d_{x}} R d x , smooth to the orders used below.
Proof. Lemma C.10 and the KL Hessian Lipschitz bound give K α α ⪰ ( a 0 / 2 ) I \mathcal{K}_{\alpha\alpha}\succeq(a_{0}/2)I K α α ⪰ ( a 0 /2 ) I on B R B_{R} B R . Lemma C.15 yields h c , α α ⪰ ( τ a 0 / 2 − J 2 − c 0 L f ) I ⪰ μ τ I h_{c,\alpha\alpha}\succeq(\tau a_{0}/2-J_{2}-c_{0}L_{f})I\succeq\mu_{\tau}I h c , α α ⪰ ( τ a 0 /2 − J 2 − c 0 L f ) I ⪰ μ τ I . Since the KL gradient vanishes at initialization, ∥ h c , α ( x , 0 ) ∥ ≤ J 1 + c G f \lVert h_{c,\alpha}(x,0)\rVert\leq J_{1}+cG_{f} ∥ h c , α ( x , 0 )∥ ≤ J 1 + c G f . On any sphere of radius strictly between ( J 1 + c G f ) / μ τ (J_{1}+cG_{f})/\mu_{\tau} ( J 1 + c G f ) / μ τ and R R R , strong monotonicity makes the gradient point outward. A minimizer on the closed smaller ball is therefore interior. Strong convexity gives uniqueness and the first distance bound. Subtract the root equations for c c c and 0 0 0 and use strong monotonicity to obtain the second bound. The implicit-function theorem applies because the actor Hessian is invertible; uniqueness identifies the local branches wherever they overlap. This asserts a selected local neural response, not an unrestricted global optimum.
Let G α x G_{\alpha x} G α x , G α x x G_{\alpha xx} G α xx , G α α x G_{\alpha\alpha x} G α α x , and G α α α G_{\alpha\alpha\alpha} G α α α denote the corresponding uniform derivative bounds for g τ g_{\tau} g τ , and write F x , F α , F x x , F α x , F α α F_{x},F_{\alpha},F_{xx},F_{\alpha x},F_{\alpha\alpha} F x , F α , F xx , F α x , F α α for those of f f f . Define
a 1 = μ τ − 1 G α x , a_{1}=\mu_{\tau}^{-1}G_{\alpha x}, a 1 = μ τ − 1 G α x ,
a 2 = μ τ − 1 ( G α x x + 2 G α α x a 1 + G α α α a 1 2 ) . (36) a_{2}=\mu_{\tau}^{-1}(G_{\alpha xx}+2G_{\alpha\alpha x}a_{1}+G_{\alpha\alpha\alpha}a_{1}^{2}). \tag*{(36)} a 2 = μ τ − 1 ( G α xx + 2 G α α x a 1 + G α α α a 1 2 ) . (36)
Corollary B.2 (Global gradient, Hessian, and value bounds). The selected response satisfies ∥ D x α 0 ⋆ ∥ ≤ a 1 \lVert D_x\alpha_0^\star\rVert\le a_1 ∥ D x α 0 ⋆ ∥ ≤ a 1 and ∥ D x 2 α 0 ⋆ ∥ ≤ a 2 \lVert D_x^2\alpha_0^\star\rVert\le a_2 ∥ D x 2 α 0 ⋆ ∥ ≤ a 2 . Consequently,
∥ ∇ Φ τ ( x ) ∥ ≤ F x + F α a 1 = : G Φ , (37) \lVert\nabla\Phi_\tau(x)\rVert\le F_x+F_{\alpha}a_1=:G_\Phi, \tag*{(37)} ∥ ∇ Φ τ ( x )∥ ≤ F x + F α a 1 =: G Φ , (37) ∥ ∇ 2 Φ τ ( x ) ∥ o p ≤ F x x + 2 F x α a 1 + F α α a 1 2 + F α a 2 = : L Φ . (38) \lVert\nabla^2\Phi_\tau(x)\rVert_{\mathrm{op}}\le F_{xx}+2F_{x\alpha}a_1+F_{\alpha\alpha}a_1^2+F_\alpha a_2=:L_\Phi. \tag*{(38)} ∥ ∇ 2 Φ τ ( x ) ∥ op ≤ F xx + 2 F xα a 1 + F α α a 1 2 + F α a 2 =: L Φ . (38) Also − B L ≤ Φ τ ( x ) ≤ B L -B_L\le\Phi_\tau(x)\le B_L − B L ≤ Φ τ ( x ) ≤ B L , with lower bound 0 for (138) . The gradient is globally Lipschitz, and the usual quadratic upper Taylor bound holds along every upper-update segment.
Proof. Set a ( x ) = α 0 ⋆ ( x ) a(x)=\alpha_0^\star(x) a ( x ) = α 0 ⋆ ( x ) , G 0 = g α α ( x , a ( x ) ) G_0=g_{\alpha\alpha}(x,a(x)) G 0 = g α α ( x , a ( x )) , and B = g α x ( x , a ( x ) ) B=g_{\alpha x}(x,a(x)) B = g α x ( x , a ( x )) . Differentiating g α ( x , a ( x ) ) = 0 g_\alpha(x,a(x))=0 g α ( x , a ( x )) = 0 gives D x a = − G 0 − 1 B D_xa=-G_0^{-1}B D x a = − G 0 − 1 B . In unit directions u , v u,v u , v , differentiating once more gives
G 0 D x 2 a [ u , v ] = − ( g α x x [ u , v ] + g α α x [ D x a [ u ] , v ] + g α α x [ D x a [ v ] , u ] + g α α α [ D x a [ u ] , D x a [ v ] ] ) . G_0D_x^2a[u,v]=-\left(g_{\alpha xx}[u,v]+g_{\alpha\alpha x}[D_xa[u],v]+g_{\alpha\alpha x}[D_xa[v],u]+g_{\alpha\alpha\alpha}[D_xa[u],D_xa[v]]\right). G 0 D x 2 a [ u , v ] = − ( g α xx [ u , v ] + g α α x [ D x a [ u ] , v ] + g α α x [ D x a [ v ] , u ] + g α α α [ D x a [ u ] , D x a [ v ]] ) . The inverse bound ∥ G 0 − 1 ∥ ≤ μ τ − 1 \lVert G_0^{-1}\rVert\le\mu_\tau^{-1} ∥ G 0 − 1 ∥ ≤ μ τ − 1 proves (36) . Applying the chain rule twice to f ( x , a ( x ) ) f(x,a(x)) f ( x , a ( x )) proves the two bounds. Integrating the bounded Hessian along a segment in R d x \mathbb{R}^{d_x} R d x proves the final statement.
Lemma B.3 (Uniform consistency of the response correction). With the population field V c V_c V c in (10) ,
∥ V c ( x , α 0 ⋆ ( x ) , S c ⋆ ( x ) ) − ∇ Φ τ ( x ) ∥ ≤ C s e c c , S c ⋆ = ( α c ⋆ − α 0 ⋆ ) / c , (39) \lVert V_c(x,\alpha_0^\star(x),S_c^\star(x))-\nabla\Phi_\tau(x)\rVert\le C_{\mathrm{sec}}c,\qquad S_c^\star=(\alpha_c^\star-\alpha_0^\star)/c, \tag*{(39)} ∥ V c ( x , α 0 ⋆ ( x ) , S c ⋆ ( x )) − ∇ Φ τ ( x )∥ ≤ C sec c , S c ⋆ = ( α c ⋆ − α 0 ⋆ ) / c , (39) uniformly over x x x , width, and 0 < c ≤ c 0 0<c\le c_0 0 < c ≤ c 0 . On the actor tube, the field is uniformly Lipschitz in ( U , S ) (U,S) ( U , S ) , with no inverse power of c c c .
Proof. For fixed x x x , implicit differentiation in c c c gives ∂ c α c ⋆ = − h c , α α − 1 f α ( x , α c ⋆ ) \partial_c\alpha_c^\star=-h_{c,\alpha\alpha}^{-1}f_\alpha(x,\alpha_c^\star) ∂ c α c ⋆ = − h c , α α − 1 f α ( x , α c ⋆ ) , hence a uniform bound G f / μ τ G_f/\mu_\tau G f / μ τ . A second derivative satisfies
∂ c 2 α c ⋆ = − h c , α α − 1 [ h c , α α α [ ∂ c α c ⋆ , ∂ c α c ⋆ ] + 2 f α α ∂ c α c ⋆ ] , \partial_c^2\alpha_c^\star=-h_{c,\alpha\alpha}^{-1}\left[h_{c,\alpha\alpha\alpha}[\partial_c\alpha_c^\star,\partial_c\alpha_c^\star]+2f_{\alpha\alpha}\partial_c\alpha_c^\star\right], ∂ c 2 α c ⋆ = − h c , α α − 1 [ h c , α α α [ ∂ c α c ⋆ , ∂ c α c ⋆ ] + 2 f α α ∂ c α c ⋆ ] , which is uniformly bounded by Lemma C.15. Taylor expansion in c c c of g x ( x , α c ⋆ ) g_x(x,\alpha_c^\star) g x ( x , α c ⋆ ) and the Lipschitz bound for f x ( x , ⋅ ) f_x(x,\cdot) f x ( x , ⋅ ) show that V c V_c V c differs by O ( c ) O(c) O ( c ) from f x − g x α g α α − 1 f α f_x-g_{x\alpha}g_{\alpha\alpha}^{-1}f_\alpha f x − g xα g α α − 1 f α , evaluated at the base response. This is the exact response-chain gradient. Finally,
V c ( x , U , S ) = ∫ 0 1 g x α ( x , U + θ c S ) S d θ + f x ( x , U + c S ) . V_c(x,U,S)=\int_0^1g_{x\alpha}(x,U+\theta cS)S\,d\theta+f_x(x,U+cS). V c ( x , U , S ) = ∫ 0 1 g xα ( x , U + θ c S ) S d θ + f x ( x , U + c S ) . Differentiating this integral in U , S U,S U , S uses only the bounded blocks g x α , g x α α , f x α g_{x\alpha},g_{x\alpha\alpha},f_{x\alpha} g xα , g xα α , f xα and bounded S S S .
Actor tracking potential. For c > 0 c>0 c > 0 , define the scaled potential
H c ( x , U , S ) = g ( x , U + c S ) − g ( x , U ) − c ⟨ g α ( x , U ) , S ⟩ c 2 + f ( x , U + c S ) − f ( x , U ) c . (40) \mathcal{H}_c(x,U,S)=\frac{g(x,U+cS)-g(x,U)-c\langle g_\alpha(x,U),S\rangle}{c^2}+\frac{f(x,U+cS)-f(x,U)}{c}. \tag*{(40)} H c ( x , U , S ) = c 2 g ( x , U + c S ) − g ( x , U ) − c ⟨ g α ( x , U ) , S ⟩ + c f ( x , U + c S ) − f ( x , U ) . (40)
Its S S S gradient is R c R_c R c from (9) . Let S ˉ c ( x , U ) \bar{S}_c(x,U) S ˉ c ( x , U ) be its unique interior minimizer over B ‾ s \overline{B}_s B s . Appendix C.3 proves existence for sufficiently large fixed τ \tau τ and the integral extension at c = 0 c=0 c = 0 . Set Δ 0 ( x , U ) = g ( x , U ) − g ( x , α 0 ⋆ ( x ) ) \Delta_0(x,U)=g(x,U)-g(x,\alpha_0^\star(x)) Δ 0 ( x , U ) = g ( x , U ) − g ( x , α 0 ⋆ ( x )) and Γ c ( x , U , S ) = H c ( x , U , S ) − H c ( x , U , S ˉ c ( x , U ) ) \Gamma_c(x,U,S)=\mathcal{H}_c(x,U,S)-\mathcal{H}_c(x,U,\bar{S}_c(x,U)) Γ c ( x , U , S ) = H c ( x , U , S ) − H c ( x , U , S ˉ c ( x , U )) . For the fixed positive weight λ S \lambda_S λ S selected in Lemma C.20, the nonnegative actor potential is
T ( x , U , S ) = Δ 0 ( x , U ) + λ S Γ c ( x , U , S ) . (41) \mathcal{T}(x,U,S)=\Delta_0(x,U)+\lambda_S\Gamma_c(x,U,S). \tag*{(41)} T ( x , U , S ) = Δ 0 ( x , U ) + λ S Γ c ( x , U , S ) . (41)
Lemma B.4. (Certified actor geometry and local update bounds). For the radii just selected, there are constants κ , E ∗ , J 1 , G f > 0 \kappa, E_*, J_1, G_f > 0 κ , E ∗ , J 1 , G f > 0 . A sufficient finite fixed τ \tau τ yields μ τ = κ τ \mu_\tau= \kappa\tau μ τ = κ τ and, uniformly in x x x and sufficiently small c c c ,
∥ α 0 ∗ ( x ) ∥ ≤ J 1 / μ τ , ∥ S c ∗ ( x ) ∥ ≤ G f / μ τ , \|\alpha_0^*(x)\| \le J_1/\mu_\tau,\qquad\|S_c^*(x)\| \le G_f/\mu_\tau, ∥ α 0 ∗ ( x ) ∥ ≤ J 1 / μ τ , ∥ S c ∗ ( x ) ∥ ≤ G f / μ τ , ∥ U − α 0 ∗ ( x ) ∥ 2 + ∥ S − S c ∗ ( x ) ∥ 2 ≤ E ∗ T ( x , U , S ) / μ τ , \|U-\alpha_0^*(x)\|^2+\|S-S_c^*(x)\|^2 \le E_* \mathcal{T}(x,U,S)/\mu_\tau, ∥ U − α 0 ∗ ( x ) ∥ 2 + ∥ S − S c ∗ ( x ) ∥ 2 ≤ E ∗ T ( x , U , S ) / μ τ , T ( x , 0 , 0 ) ≤ J 1 2 + λ S G f 2 2 μ τ . (42) \mathcal{T}(x,0,0)\le\frac{J_1^2+\lambda_S G_f^2}{2\mu_\tau}. \tag*{(42)} T ( x , 0 , 0 ) ≤ 2 μ τ J 1 2 + λ S G f 2 . (42) Here E ∗ , λ S E_*,\lambda_S E ∗ , λ S can be fixed independently of sufficiently large τ \tau τ . The needed gradients and full ( x , U , S ) (x,U,S) ( x , U , S ) Hessian of T \mathcal{T} T are uniformly bounded on the prospective actor tube. Moreover,
∥ ∇ Φ ∥ ≤ G Φ , ∥ ∇ 2 Φ ∥ ≤ L Φ , Φ ≥ Φ ‾ , \|\nabla\Phi\|\le G_\Phi,\qquad\|\nabla^2\Phi\|\le L_\Phi,\qquad\Phi\ge\overline{\Phi}, ∥∇Φ∥ ≤ G Φ , ∥ ∇ 2 Φ∥ ≤ L Φ , Φ ≥ Φ , ∥ a ∥ 2 + ∥ R c ∥ 2 ≤ C T , ∥ V c ∥ 2 ≤ C ( ∥ ∇ Φ ∥ 2 + T + c 2 ) . (43) \|\mathbf{a}\|^2+\|R_c\|^2\le C\mathcal{T},\qquad\|V_c\|^2\le C(\|\nabla\Phi\|^2+\mathcal{T}+c^2). \tag*{(43)} ∥ a ∥ 2 + ∥ R c ∥ 2 ≤ C T , ∥ V c ∥ 2 ≤ C ( ∥∇Φ ∥ 2 + T + c 2 ) . (43) For preference cross-entropy take Φ = 0 \Phi=0 Φ = 0 ; for a bounded general loss take − B L -B_L − B L .
Write P 0 = F m , ρ ^ n ( U ) − 1 P^0=F_{m,\widehat{\rho}_n}(U)^{-1} P 0 = F m , ρ n ( U ) − 1 , P c = F m , ρ ^ n ( U + c S ) − 1 P^c=F_{m,\widehat{\rho}_n}(U+cS)^{-1} P c = F m , ρ n ( U + c S ) − 1 and K c = ( P 0 − P c ) / c K_c=(P^0-P^c)/c K c = ( P 0 − P c ) / c . On metric acceptance these operators obey the same fixed bounds by Proposition C.30 . For compatible samples satisfying C ^ = a ^ + c R ^ \widehat{C}=\widehat{a}+c\widehat{R} C = a + c R , the finite actor updates admit
Δ U = − η y P 0 a ^ + r U , \Delta U=-\eta_yP^0\widehat{a}+r_U, Δ U = − η y P 0 a + r U , Δ S = − η y P c R ^ + η y K c a ^ + r S , Δ x = − η x V ^ . (44) \Delta S=-\eta_yP^c\widehat{R}+\eta_yK_c\widehat{a}+r_S,\qquad\Delta x=-\eta_x\widehat{V}. \tag*{(44)} Δ S = − η y P c R + η y K c a + r S , Δ x = − η x V . (44) Here r U = ρ U + e U r_U=\rho^U+e_U r U = ρ U + e U and r S = ρ S + e S r_S=\rho^S+e_S r S = ρ S + e S , where ρ U , ρ S \rho^U,\rho^S ρ U , ρ S are the exact reference-map remainders and the finite-solver errors satisfy ∥ e U ∥ ≤ c η y 2 \|e_U\|\le c\eta_y^2 ∥ e U ∥ ≤ c η y 2 , ∥ e S ∥ ≤ 2 η y 2 \|e_S\|\le2\eta_y^2 ∥ e S ∥ ≤ 2 η y 2 samplewise. The predictable operators are uniformly bounded and the noiseless ( U , S ) (U,S) ( U , S ) direction decreases T \mathcal{T} T by at least a fixed multiple of T \mathcal{T} T . The combined remainders have conditional first, second and fourth moments O ( η y 2 ) O(\eta_y^2) O ( η y 2 ) , O ( η y 4 ) O(\eta_y^4) O ( η y 4 ) and O ( η y 8 ) O(\eta_y^8) O ( η y 8 ) , respectively. Deterministic actor increments are η y poly ( H ) \eta_y\operatorname{poly}(H) η y poly ( H ) on sufficiently small prospective steps.
Proof. Proposition B.1 gives curvature and the two response displacements. Lemmas C.19 and C.20 prove the tracking self-bounds, full Hessian bound, distance control and initialization estimate without any compact upper domain. Corollary B.2 gives the objective bounds, and Lemma C.21 gives the corrected-field estimate. For the local updates, Lemmas C.24 and C.25 prove the exact reference expansion, including its inverse-metric correction. Proposition C.33 and Lemma C.35 give the stated implementation errors; no derivative of the numerical stopping rule is required. Lemma C.27 proves descent of its population direction. The moment assertions follow from Lemma C.45 and the sampled-field bounds in Lemma B.8 . These are moment conditions derived from the sampling protocol, and do not assume accurate critic values.
Stable paired TD targets and derived accuracy
For p = ( x , U , S ) p=(x,U,S) p = ( x , U , S ) , write α 0 = U \alpha_0=U α 0 = U , α c = U + c S \alpha_c=U+cS α c = U + c S and z = ( u , v ) z=(u,v) z = ( u , v ) , so the critics have coordinates u u u and u + c v u+cv u + c v . Let q x , α q_{x,\alpha} q x , α and Q x , α t r u e Q^{\mathrm{true}}_{x,\alpha} Q x , α true have the convention in (15) : the true value is an unnormalized discounted cost. The normalized state–action occupancy obeys
ν U = ( 1 − γ ) ρ U + γ ν U K U . (45) \nu_U=(1-\gamma)\rho_U+\gamma\nu_UK_U. \tag*{(45)} ν U = ( 1 − γ ) ρ U + γ ν U K U . (45)
Define the untruncated TD mean field (distinct from the cap H H H ) by
H ( x , U , α , θ ) = E ν U [ ( Q m Q , θ − q x , α − γ K α Q m Q , θ ) ∇ θ Q m Q , θ ] , H(x,U,\alpha,\theta)=\mathbb{E}_{\nu_U}\left[(Q_{m_Q,\theta}-q_{x,\alpha}-\gamma K_\alpha Q_{m_Q,\theta})\nabla_\theta Q_{m_Q,\theta}\right], H ( x , U , α , θ ) = E ν U [ ( Q m Q , θ − q x , α − γ K α Q m Q , θ ) ∇ θ Q m Q , θ ] ,
F ( p , z ) = ( H ( x , U , U , u ) [ H ( x , U , U + c S , u + c v ) − H ( x , U , U , u ) ] / c ) . (46) F(p,z)=
\begin{pmatrix}
H(x,U,U,u)\\
[H(x,U,U+cS,u+cv)-H(x,U,U,u)]/c
\end{pmatrix}. \tag*{(46)} F ( p , z ) = ( H ( x , U , U , u ) [ H ( x , U , U + c S , u + c v ) − H ( x , U , U , u )] / c ) . (46)
The operator K α K_{\alpha} K α includes the next-state and next-action expectations. This is a TD semigradient field. The sample field H ˉ \bar{H} H ˉ , with exact finite next-action sums, is defined in (222) ; U U U denotes its base sampling law and has no additional pathwise role in that integrand.
The projection set and weighted norm are K Q = B ‾ R u × B ‾ R v K_Q = \overline{B}_{R_u} \times\overline{B}_{R_v} K Q = B R u × B R v and ∥ z ∥ W 2 = M ∥ u ∥ 2 + ∥ v ∥ 2 \lVert z\rVert_W^2 = M\lVert u\rVert^2 + \lVert v\rVert^2 ∥ z ∥ W 2 = M ∥ u ∥ 2 + ∥ v ∥ 2 , with W = diag ( M I d Q , I d Q ) W = \operatorname{diag}(MI_{d_Q}, I_{d_Q}) W = diag ( M I d Q , I d Q ) . The radii and M > 0 M > 0 M > 0 are constructed below; their definition precedes the projected update in (223) . Let ℓ max \ell_{\max} ℓ m a x bound ∣ log ( π α / π r e f ) ∣ \lvert\log(\pi_{\alpha}/\pi_{\mathrm{ref}})\rvert ∣ log ( π α / π ref )∣ on the prospective actor neighborhood and put q max = B r + τ ℓ max q_{\max} = B_r + \tau\ell_{\max} q m a x = B r + τ ℓ m a x .
Proposition B.5 (Global TD coercivity and interior roots). Set μ 0 = ( 1 − γ ) λ Q \mu_0 = (1-\sqrt{\gamma})\lambda_Q μ 0 = ( 1 − γ ) λ Q . Choose
R Q > 4 q max F Q μ 0 , γ F Q 2 L π c 0 s ≤ μ 0 4 , (47) R_Q > \frac{4q_{\max}F_Q}{\mu_0}, \qquad\gamma F_Q^2L_{\pi}c_0s \le\frac{\mu_0}{4}, \tag*{(47)} R Q > μ 0 4 q m a x F Q , γ F Q 2 L π c 0 s ≤ 4 μ 0 , (47) where L π L_{\pi} L π is a uniform policy ℓ 1 \ell_1 ℓ 1 -Lipschitz constant. There is a finite m Q , 0 m_{Q,0} m Q , 0 such that, for every m Q ≥ m Q , 0 m_Q \ge m_{Q,0} m Q ≥ m Q , 0 , H ( x , U , α , ⋅ ) H(x,U,\alpha,\cdot) H ( x , U , α , ⋅ ) is strongly monotone on B R Q B_{R_Q} B R Q with modulus μ Q = μ 0 / 2 \mu_Q = \mu_0/2 μ Q = μ 0 /2 , and has a unique root θ ∗ ( x , U , α ) \theta^*(x,U,\alpha) θ ∗ ( x , U , α ) with norm strictly less than $R_Q/2. The bounds are global in x x x and uniform in m Q , c m_Q,c m Q , c .
Proof. Lemma C.39 gives the nonlinear critic derivative and linearization bounds in (220) . For every scalar v v v , (45) and Jensen imply ∥ K U v ∥ L 2 ( ν U ) ≤ γ − 1 / 2 ∥ v ∥ L 2 ( ν U ) \lVert K_Uv\rVert_{L^2(\nu_U)} \le\gamma^{-1/2}\lVert v\rVert_{L^2(\nu_U)} ∥ K U v ∥ L 2 ( ν U ) ≤ γ − 1/2 ∥ v ∥ L 2 ( ν U ) . Consequently
E ν U v ( v − γ K U v ) ≥ ( 1 − γ ) ∥ v ∥ L 2 ( ν U ) 2 . (48) \mathbb{E}_{\nu_U}v(v-\gamma K_Uv) \ge(1-\sqrt{\gamma})\lVert v\rVert_{L^2(\nu_U)}^2. \tag*{(48)} E ν U v ( v − γ K U v ) ≥ ( 1 − γ ) ∥ v ∥ L 2 ( ν U ) 2 . (48)
For a unit b ∈ R d Q b \in\mathbb{R}^{d_Q} b ∈ R d Q set v = b ⊤ ∇ Q m Q , θ v = b^{\top}\nabla Q_{m_Q,\theta} v = b ⊤ ∇ Q m Q , θ . Differentiating the TD field shows that its quadratic Jacobian form equals E v ( v − γ K α v ) \mathbb{E}v(v-\gamma K_{\alpha}v) E v ( v − γ K α v ) plus the TD residual times b ⊤ ∇ 2 Q b b^{\top}\nabla^2Qb b ⊤ ∇ 2 Q b . The residual has absolute value at most D Q = ( 1 + γ ) F Q R Q + q max D_Q = (1+\gamma)F_QR_Q + q_{\max} D Q = ( 1 + γ ) F Q R Q + q m a x . Replacing K α K_{\alpha} K α by K U K_U K U costs at most γ F Q 2 L π c 0 s \gamma F_Q^2L_{\pi}c_0s γ F Q 2 L π c 0 s . Also
E ν U ∇ Q ∇ Q ⊤ ⪰ sech 4 ( F Q R Q / m Q ) λ Q I . \mathbb{E}_{\nu_U}\nabla Q\nabla Q^{\top} \succeq\operatorname{sech}^4(F_QR_Q/\sqrt{m_Q})\lambda_QI. E ν U ∇ Q ∇ Q ⊤ ⪰ sech 4 ( F Q R Q / m Q ) λ Q I .
It follows that
sym D θ H ⪰ [ μ 0 sech 4 ( F Q R Q / m Q ) − 2 D Q F Q 3 R Q m Q − γ F Q 2 L π c 0 s ] I . (49) \operatorname{sym}D_{\theta}H \succeq\left[\mu_0\operatorname{sech}^4(F_QR_Q/\sqrt{m_Q})-\frac{2D_QF_Q^3R_Q}{m_Q}-\gamma F_Q^2L_{\pi}c_0s\right]I. \tag*{(49)} sym D θ H ⪰ [ μ 0 sech 4 ( F Q R Q / m Q ) − m Q 2 D Q F Q 3 R Q − γ F Q 2 L π c 0 s ] I . (49)
Choose m Q , 0 m_{Q,0} m Q , 0 so the first two deviations from μ 0 \mu_0 μ 0 total at most μ 0 / 4 \mu_0/4 μ 0 /4 . Integration along line segments proves strong monotonicity with modulus μ Q \mu_Q μ Q .
At zero, ∥ H ( x , U , α , 0 ) ∥ ≤ q max F Q \lVert H(x,U,\alpha,0)\rVert\le q_{\max}F_Q ∥ H ( x , U , α , 0 )∥ ≤ q m a x F Q . A variational-inequality solution exists on the closed ball by applying Brouwer to a projected field step. At the boundary its outward radial field satisfies ⟨ θ , H ( θ ) ⟩ ≥ μ Q R Q 2 − R Q q max F Q > 0 \langle\theta,H(\theta)\rangle\ge\mu_QR_Q^2-R_Qq_{\max}F_Q > 0 ⟨ θ , H ( θ )⟩ ≥ μ Q R Q 2 − R Q q m a x F Q > 0 , which excludes a boundary variational-inequality solution by testing against zero. Thus the solution is an interior root, unique by strong monotonicity. Its norm is at most q max F Q / μ Q < R Q / 2 q_{\max}F_Q/\mu_Q < R_Q/2 q m a x F Q / μ Q < R Q /2 .
With θ ∗ \theta^* θ ∗ the unique TD root just obtained, define its paired target
Z ( p ) = ( θ ∗ ( x , U , U ) , θ ∗ ( x , U , U + c S ) − θ ∗ ( x , U , U ) c ) (50) Z(p)=\left(\theta^*(x,U,U),\frac{\theta^*(x,U,U+cS)-\theta^*(x,U,U)}{c}\right) \tag*{(50)} Z ( p ) = ( θ ∗ ( x , U , U ) , c θ ∗ ( x , U , U + c S ) − θ ∗ ( x , U , U ) ) (50)
Proposition B.6 (Uniform paired roots, projection set, and target derivatives). There are fixed radii R u R_u R u , R v R_v R v and a weight M M M for which R u + c 0 R v < R Q R_u+c_0R_v<R_Q R u + c 0 R v < R Q , both endpoint critic parameters lie in the critic ball for every z ∈ K Q z\in K_Q z ∈ K Q , and the target Z ( p ) Z(p) Z ( p ) lies strictly inside K Q K_Q K Q with uniform slack. After reducing c 0 c_0 c 0 if needed, F ( p , ⋅ ) F(p,\cdot) F ( p , ⋅ ) is strongly monotone and Lipschitz in the W W W norm, with constants uniform in x , m Q , c x,m_Q,c x , m Q , c . Its sample field is bounded uniformly in c c c . The target is twice continuously differentiable in p p p , with
∥ D Z ∥ W ≤ L Z , ∥ D 2 Z ∥ W ≤ H Z (51) \lVert DZ\rVert_W \le L_Z,\qquad\lVert D^2Z\rVert_W \le H_Z \tag*{(51)} ∥ D Z ∥ W ≤ L Z , ∥ D 2 Z ∥ W ≤ H Z (51) uniformly over x x x and the prospective actor tube.
Proof. The inverse of D θ H D_\theta H D θ H has norm at most μ Q − 1 \mu_Q^{-1} μ Q − 1 . Implicit differentiation gives
D α θ ∗ = − ( D θ H ) − 1 D α H , D_\alpha\theta^* = -(D_\theta H)^{-1}D_\alpha H, D α θ ∗ = − ( D θ H ) − 1 D α H ,
uniformly bounded by Lemma C.40 . Integrating this derivative along U + t S U+tS U + tS bounds the second component of (50) by a constant times s s s . Choose R u R_u R u strictly between the uniform base-root bound and R Q R_Q R Q , and R v R_v R v strictly above that response bound. Reduce c 0 c_0 c 0 to get R u + c 0 R v < R Q R_u+c_0R_v<R_Q R u + c 0 R v < R Q . The root and derivative constants can first be bounded for the preliminary cap; reduction does not invalidate them.
Keeping the sample and base law fixed gives the integral identity
H ^ ( x , U , U + c S , u + c v ) − H ^ ( x , U , U , u ) c = ∫ 0 1 { D α H ^ ( x , U , U + t c S , u + t c v ) [ S ] + D θ H ^ ( x , U , U + t c S , u + t c v ) [ v ] } d t . (52) \frac{\widehat{H}(x,U,U+cS,u+cv)-\widehat{H}(x,U,U,u)}{c}
=
\int_0^1\left\{D_\alpha\widehat{H}(x,U,U+tcS,u+tcv)[S]
+D_\theta\widehat{H}(x,U,U+tcS,u+tcv)[v]\right\}\,\mathrm{d}t. \tag*{(52)} c H ( x , U , U + c S , u + c v ) − H ( x , U , U , u ) = ∫ 0 1 { D α H ( x , U , U + t c S , u + t c v ) [ S ] + D θ H ( x , U , U + t c S , u + t c v ) [ v ] } d t . (52)
It bounds the sample field without dividing two unrelated endpoint estimates by c c c . Its mean has block Jacobian
F z = ( D θ H 0 0 B c D θ H c ) , B c = [ D θ H c − D θ H 0 ] / c . F_z=
\begin{pmatrix}
D_\theta H_0 & 0\\
B_c & D_\theta H_c
\end{pmatrix},
\qquad
B_c=[D_\theta H_c-D_\theta H_0]/c. F z = ( D θ H 0 B c 0 D θ H c ) , B c = [ D θ H c − D θ H 0 ] / c .
The same integral formula bounds ∥ B c ∥ ≤ C B \lVert B_c\rVert\le C_B ∥ B c ∥ ≤ C B . For increments ( a , b ) (a,b) ( a , b ) its weighted quadratic form is at least M μ Q ∥ a ∥ 2 + μ Q ∥ b ∥ 2 − C B ∥ a ∥ ∥ b ∥ M\mu_Q\lVert a\rVert^2+\mu_Q\lVert b\rVert^2-C_B\lVert a\rVert\lVert b\rVert M μ Q ∥ a ∥ 2 + μ Q ∥ b ∥ 2 − C B ∥ a ∥ ∥ b ∥ . Choosing M ≥ 1 + C B 2 / μ Q 2 M\ge1+C_B^2/\mu_Q^2 M ≥ 1 + C B 2 / μ Q 2 gives strong monotonicity with modulus μ F = μ Q / 2 \mu_F=\mu_Q/2 μ F = μ Q /2 in the W W W norm. Bounded blocks give Lipschitz continuity, and the equations F = 0 F=0 F = 0 identify (50) .
Twice differentiating the mean integral (52) uses derivatives of H H H of total order at most three, with at most two x x x derivatives and at least one actor or critic derivative in each third-order block. Lemma C.40 is exactly sufficient, including the derivatives of the base law. Thus F p F_p F p , F z F_z F z , F p p F_{pp} F pp , F p z F_{pz} F p z , F z z F_{zz} F z z have uniform bounds. Applying the implicit-function theorem in the open neighborhood of each interior target gives
D Z = − F z − 1 F p , DZ=-F_z^{-1}F_p, D Z = − F z − 1 F p ,
D 2 Z [ a , b ] = − F z − 1 { F p p [ a , b ] + F p z [ a , D Z b ] + F z p [ D Z a , b ] + F z z [ D Z a , D Z b ] } . (53) D^2Z[a,b]=-F_z^{-1}\{F_{pp}[a,b]+F_{pz}[a,DZb]+F_{zp}[DZa,b]+F_{zz}[DZa,DZb]\}. \tag*{(53)} D 2 Z [ a , b ] = − F z − 1 { F pp [ a , b ] + F p z [ a , D Z b ] + F z p [ D Z a , b ] + F z z [ D Z a , D Z b ]} . (53)
Strong monotonicity bounds the inverse by μ F − 1 \mu_F^{-1} μ F − 1 in the weighted norm. For bounds L 1 L_1 L 1 on F p F_p F p and L 2 L_2 L 2 on the second blocks, one may take L Z = L 1 / μ F L_Z=L_1/\mu_F L Z = L 1 / μ F and H Z = L 2 ( 1 + L Z ) 2 / μ F H_Z=L_2(1+L_Z)^2/\mu_F H Z = L 2 ( 1 + L Z ) 2 / μ F . Uniqueness identifies the local branches on overlaps for every x ∈ R d x x\in\mathbb{R}^{d_x} x ∈ R d x . □
Proposition B.7 (The global paired approximation floor is derived). Under C2, write ( u ∗ , v ∗ ) = Z ( p ) (u^*,v^*)=Z(p) ( u ∗ , v ∗ ) = Z ( p ) and define the endpoint errors
e 0 ∗ ( p ) = Q m Q , u ∗ − Q x , U t r u e , e c ∗ ( p ) = Q m Q , u ∗ + c v ∗ − Q x , U + c S t r u e . e_0^*(p)=Q_{mQ,u^*}-Q_{x,U}^{\mathrm{true}},
\qquad
e_c^*(p)=Q_{mQ,u^*+cv^*}-Q_{x,U+cS}^{\mathrm{true}}. e 0 ∗ ( p ) = Q m Q , u ∗ − Q x , U true , e c ∗ ( p ) = Q m Q , u ∗ + c v ∗ − Q x , U + c S true . Then the squared paired approximation floor satisfies
A Q : = sup p { ∥ e 0 ∗ ( p ) ∥ ∞ 2 + ∥ e c ∗ ( p ) − e 0 ∗ ( p ) c ∥ ∞ 2 } ≤ C m Q − 2 . (54) A_Q:=\sup_p\left\{\lVert e_0^*(p)\rVert_\infty^2+
\left\lVert\frac{e_c^*(p)-e_0^*(p)}{c}\right\rVert_\infty^2\right\}
\le Cm_Q^{-2}. \tag*{(54)} A Q := p sup { ∥ e 0 ∗ ( p ) ∥ ∞ 2 + c e c ∗ ( p ) − e 0 ∗ ( p ) ∞ 2 } ≤ C m Q − 2 . (54) This bound and the required projected-root coefficient derivative bounds are uniform in x x x .
Proof. Fix x x x , U U U , S S S and vary the endpoint α t = U + t S \alpha_t = U + tS α t = U + tS , 0 ≤ t ≤ c 0 0 \le t \le c_0 0 ≤ t ≤ c 0 , keeping ν U \nu_U ν U fixed. The linear-feature TD field is A t θ − b t A_t\theta- b_t A t θ − b t , with
A t = E ν U [ φ Q ( φ Q − γ K α t φ Q ) T ] , b t = E ν U [ φ Q q x , α t ] . A_t = \mathbb{E}_{\nu_U}\left[\varphi_Q(\varphi_Q - \gamma K_{\alpha_t}\varphi_Q)^{\mathsf{T}}\right], \qquad b_t = \mathbb{E}_{\nu_U}\left[\varphi_Q q_{x,\alpha_t}\right]. A t = E ν U [ φ Q ( φ Q − γ K α t φ Q ) T ] , b t = E ν U [ φ Q q x , α t ] .
Equation (48) and the perturbation bound give sym A t ⪰ ( μ 0 / 2 ) I \operatorname{sym} A_t \succeq(\mu_0/2)I sym A t ⪰ ( μ 0 /2 ) I . Consequently its root w t = A t − 1 b t w_t = A_t^{-1}b_t w t = A t − 1 b t is uniformly bounded and lies inside B R Q / 2 B_{R_Q/2} B R Q /2 . All required coefficient derivatives follow by differentiating A t w t = b t A_tw_t = b_t A t w t = b t : A t A_t A t has only actor dependence, and b t b_t b t uses only the available reward derivatives and actor likelihood derivatives. In particular no smooth-coefficient hypothesis is needed in addition to C 2 C^2 C 2 . Exact realizability makes w t w_t w t the true-value coefficient, since the Bellman equation implies the TD equation and the latter has a unique root.
Let H t m Q H_t^{m_Q} H t m Q be the nonlinear field, with root θ t m Q \theta_t^{m_Q} θ t m Q . From (220) , uniformly over the critic ball,
∥ H t m Q − ( A t θ − b t ) ∥ + ∥ D θ H t m Q − A t ∥ + ∥ ∂ t H t m Q − ( A ˙ t θ − b ˙ t ) ∥ ≤ C / m Q . \left\|H_t^{m_Q} - (A_t\theta- b_t)\right\| + \left\|D_\theta H_t^{m_Q} - A_t\right\|
+ \left\|\partial_t H_t^{m_Q} - (\dot{A}_t\theta- \dot{b}_t)\right\| \le C/m_Q. H t m Q − ( A t θ − b t ) + D θ H t m Q − A t + ∂ t H t m Q − ( A ˙ t θ − b ˙ t ) ≤ C / m Q .
Strong monotonicity implies ∥ θ t m Q − w t ∥ ≤ C / m Q \left\|\theta_t^{m_Q} - w_t\right\| \le C/m_Q θ t m Q − w t ≤ C / m Q . Differentiating both root equations, with B t = D θ H t m Q ( θ t m Q ) B_t = D_\theta H_t^{m_Q}(\theta_t^{m_Q}) B t = D θ H t m Q ( θ t m Q ) , gives
B t ( θ ˙ t m Q − w ˙ t ) = − [ ∂ t H t m Q ( θ t m Q ) − ( A ˙ t w t − b ˙ t ) ] − ( B t − A t ) w ˙ t . B_t(\dot{\theta}_t^{m_Q} - \dot{w}_t) = -\left[\partial_t H_t^{m_Q}(\theta_t^{m_Q}) - (\dot{A}_t w_t - \dot{b}_t)\right] - (B_t - A_t)\dot{w}_t. B t ( θ ˙ t m Q − w ˙ t ) = − [ ∂ t H t m Q ( θ t m Q ) − ( A ˙ t w t − b ˙ t ) ] − ( B t − A t ) w ˙ t .
The inverse, coefficient, and derivative bounds imply ∥ θ ˙ t m Q − w ˙ t ∥ ≤ C / m Q \left\|\dot{\theta}_t^{m_Q} - \dot{w}_t\right\| \le C/m_Q θ ˙ t m Q − w ˙ t ≤ C / m Q . Therefore e m Q ( t ) = Q m Q , θ t m Q − φ Q T w t e_{m_Q}(t) = Q_{m_Q,\theta_t^{m_Q}} - \varphi_Q^{\mathsf{T}}w_t e m Q ( t ) = Q m Q , θ t m Q − φ Q T w t has ∥ e m Q ( t ) ∥ ∞ + ∥ e ˙ m Q ( t ) ∥ ∞ ≤ C / m Q \|e_{m_Q}(t)\|_\infty+ \|\dot{e}_{m_Q}(t)\|_\infty\le C/m_Q ∥ e m Q ( t ) ∥ ∞ + ∥ e ˙ m Q ( t ) ∥ ∞ ≤ C / m Q . Integrating e ˙ m Q \dot{e}_{m_Q} e ˙ m Q from 0 0 0 to c c c bounds the normalized error difference by C / m Q C/m_Q C / m Q . Squaring proves (54) . All constants were global in x x x before this calculation. □
Shared samples and persistent tracking
Condition on a realized accepted metric sample, included in F t \mathcal{F}_t F t . The three sampling routines in Appendix § 10 .§ 6 use the old p t p_t p t , z t z_t z t and conditionally independent data blocks. The numerical actor solvers use only the stored states and frozen lower block; Lemma C.36 preserves the conditional independence from the current TD block. All constants below are uniform over accepted metric realizations. Write b T D = C T D γ H ℓ b_{\mathrm{TD}} = C_{\mathrm{TD}}\gamma_H^\ell b TD = C TD γ H ℓ and b e v = C e v ( H + 1 ) ℓ γ H b_{\mathrm{ev}} = C_{\mathrm{ev}}(H+1)^\ell\gamma_H b ev = C ev ( H + 1 ) ℓ γ H for uniform truncation-bias bounds, with fixed constants and integer ℓ \ell ℓ supplied there. Define
T t = T ( p t ) , C t = ∥ z t − Z ( p t ) ∥ W 2 , X t = ∥ ∇ Φ ( x t ) ∥ 2 , ω = c 2 + A Q + b e v 2 + b T D 2 . (55) \mathcal{T}_t = \mathcal{T}(p_t), \qquad C_t = \|z_t - Z(p_t)\|_W^2, \qquad X_t = \|\nabla\Phi(x_t)\|^2, \qquad\omega= c^2 + A_Q + b_{\mathrm{ev}}^2 + b_{\mathrm{TD}}^2. \tag*{(55)} T t = T ( p t ) , C t = ∥ z t − Z ( p t ) ∥ W 2 , X t = ∥∇Φ ( x t ) ∥ 2 , ω = c 2 + A Q + b ev 2 + b TD 2 . (55)
All conditional inequalities below hold at a pre-exit point and through its prospective next step in the larger analytic actor neighborhood.
Lemma B.8 (Uniform oracle biases and moments). The squared conditional biases of the base, normalized-response, and corrected outer fields are at most
C ( C t + A Q + b e v 2 ) , b e v ≤ C ( H + 1 ) ℓ γ H (56) C(C_t + A_Q + b_{\mathrm{ev}}^2), \qquad b_{\mathrm{ev}} \le C(H+1)^\ell\gamma_H \tag*{(56)} C ( C t + A Q + b ev 2 ) , b ev ≤ C ( H + 1 ) ℓ γ H (56) for a fixed integer ℓ \ell ℓ . For the fixed-dynamics outer estimator its critic-dependent term can be omitted. Every fixed-order conditional sample moment is uniformly bounded in H H H ; deterministic envelopes are polynomial in H H H . All statements are uniform in x x x , m Q m_Q m Q , c c c . Fixed positive batch sizes already satisfy the normalized moment requirements; TD/evaluation batches may grow at most polynomially in H H H , with fixed preference batch sizes.
Proof. Lemma C.43 controls the base and normalized critic errors before likelihood weighting, then transfers them through the shared-prefix estimators. Lemmas C.41 and C.42 control those weights; Proposition C.44 supplies unbiased bounded preference terms. Their geometric tail and moment estimates prove the displayed bounds. Every data block uses the old critic, so centering is conditional on the stated pre-draw history. For fixed dynamics the outer field contains only reward derivatives and direct preference terms, and hence no critic error.
Lemma B.9 (Moving-target critic recursion with centered target movement). For sufficiently small steps, η x ≤ η y \eta_x \le\eta_y η x ≤ η y , and η y / β \eta_y/\beta η y / β sufficiently small, there are fixed positive k k k , C C C such that
E t Δ C t ≤ − k β C t + C η y 2 β T t + C η x 2 β X t + C β ω + C ( β 2 + η y 2 + η x 2 ) . (57) \mathbb{E}_t\Delta C_t \le-k\beta C_t + C\frac{\eta_y^2}{\beta}T_t + C\frac{\eta_x^2}{\beta}X_t + C\beta\omega+ C(\beta^2+\eta_y^2+\eta_x^2). \tag*{(57)} E t Δ C t ≤ − k β C t + C β η y 2 T t + C β η x 2 X t + C β ω + C ( β 2 + η y 2 + η x 2 ) . (57)
The constants can be fixed uniformly over all sufficiently small ratios η y / β \eta_y/\beta η y / β and η x / η y \eta_x/\eta_y η x / η y .
Proof. First hold the target at the old p t p_t p t . Projection is nonexpansive and Z ( p t ) ∈ K Q Z(p_t) \in\mathcal{K}_Q Z ( p t ) ∈ K Q . Strong monotonicity, Lipschitz continuity, bounded noise second moment, and (228) give
E t ∥ z t + 1 − Z ( p t ) ∥ W 2 ≤ ( 1 − a β ) C t + C β b T D 2 + C β 2 (58) \mathbb{E}_t\left\|z_{t+1}-Z(p_t)\right\|_W^2 \le(1-a\beta)C_t+C\beta b_{\mathrm{TD}}^2+C\beta^2 \tag*{(58)} E t ∥ z t + 1 − Z ( p t ) ∥ W 2 ≤ ( 1 − a β ) C t + C β b TD 2 + C β 2 (58)
for a fixed a > 0 a>0 a > 0 . Indeed expansion of the projected squared distance gives the cross term − 2 β ⟨ z t − Z , F ⟩ -2\beta\langle z_t-Z,F\rangle − 2 β ⟨ z t − Z , F ⟩ ; absorb the bias cross term by Young’s inequality, and the β 2 \beta^2 β 2 terms by a sufficiently small absolute step.
Write Δ p t = d t + w t + 1 \Delta p_t=d_t+w_{t+1} Δ p t = d t + w t + 1 , where d t = E t Δ p t d_t=\mathbb{E}_t\Delta p_t d t = E t Δ p t and E t w t + 1 = 0 \mathbb{E}_t w_{t+1}=0 E t w t + 1 = 0 . The current TD block is conditionally independent of this entire increment. Reveal that block first, and set e ˉ = z t + 1 − Z ( p t ) \bar e=z_{t+1}-Z(p_t) e ˉ = z t + 1 − Z ( p t ) , J = D Z ( p t ) J=DZ(p_t) J = D Z ( p t ) . Then e ˉ \bar e e ˉ , J J J , d t d_t d t are fixed under the enlarged conditioning and w t + 1 w_{t+1} w t + 1 stays centered. Taylor expansion gives
Z ( p t + Δ p t ) − Z ( p t ) = J ( d t + w t + 1 ) + R t , ∥ R t ∥ W ≤ 1 2 H Z ∥ Δ p t ∥ 2 . Z(p_t+\Delta p_t)-Z(p_t)=J(d_t+w_{t+1})+R_t,\qquad\|R_t\|_W\le\frac{1}{2}H_Z\|\Delta p_t\|^2. Z ( p t + Δ p t ) − Z ( p t ) = J ( d t + w t + 1 ) + R t , ∥ R t ∥ W ≤ 2 1 H Z ∥Δ p t ∥ 2 .
Both critic points defining e ˉ \bar e e ˉ lie in K Q \mathcal{K}_Q K Q , so ∥ e ˉ ∥ W \|\bar e\|_W ∥ e ˉ ∥ W is bounded by its diameter. Prospective increments have a fixed deterministic bound. The remainder cross terms and its square are consequently bounded by C ∥ d t + w t + 1 ∥ 2 C\|d_t+w_{t+1}\|^2 C ∥ d t + w t + 1 ∥ 2 . Centering cancels the linear cross term involving w t + 1 w_{t+1} w t + 1 . Use Young’s inequality only for the deterministic displacement J d t Jd_t J d t , and combine with (58) , to obtain
E t C t + 1 ≤ ( 1 − a ′ β ) C t + C β b T D 2 + C β 2 + C β ∥ d t ∥ 2 + C E t ∥ w t + 1 ∥ 2 . (59) \mathbb{E}_t C_{t+1}\le(1-a'\beta)C_t+C\beta b_{\mathrm{TD}}^2+C\beta^2+\frac{C}{\beta}\|d_t\|^2+C\mathbb{E}_t\|w_{t+1}\|^2. \tag*{(59)} E t C t + 1 ≤ ( 1 − a ′ β ) C t + C β b TD 2 + C β 2 + β C ∥ d t ∥ 2 + C E t ∥ w t + 1 ∥ 2 . (59)
In particular the centered target movement has no 1 / β 1/\beta 1/ β penalty.
Lemma C.45 gives the conditional mean, second moment and fourth moment estimates in (235) . It uses the paired reference-map remainder, the samplewise finite-solver error from Lemma C.35 , and the trained-critic oracle bias, including the fact that only the deterministic target drift is penalized by 1 / β 1/\beta 1/ β in (59) .
Substitute (235) into (59) . Choose a fixed upper bound on ( η y / β ) 2 (\eta_y/\beta)^2 ( η y / β ) 2 small enough to absorb the self-coupling C ( η y 2 + η x 2 ) C t / β C(\eta_y^2+\eta_x^2)C_t/\beta C ( η y 2 + η x 2 ) C t / β into the negative drift. Since η x ≤ η y / β ≤ 1 \eta_x\le\eta_y/\beta\le1 η x ≤ η y / β ≤ 1 , the remaining terms have the sizes in (57) . The constants before absorption are uniform over ratios at most one; fixing a sufficiently small upper bound and then reducing the ratios preserves them. □ \square □
Lemma B.10 (Actor tracking and outer descent with the trained critic). For sufficiently small steps and η x / η y \eta_x/\eta_y η x / η y below a fixed threshold, there are fixed positive a a a , b b b , d d d , u u u , v v v , w w w , C C C with
E t Δ Φ t ≤ − a η x X t + b η x T t + d η x C t + C η x ω + C η x 2 , (60) \mathbb{E}_t\Delta\Phi_t\le-a\eta_xX_t+b\eta_xT_t+d\eta_xC_t+C\eta_x\omega+C\eta_x^2, \tag*{(60)} E t Δ Φ t ≤ − a η x X t + b η x T t + d η x C t + C η x ω + C η x 2 , (60) E t Δ T t ≤ − u η y T t + v η x X t + w η y C t + C η y ω + C ( η y 2 + η x 2 ) . (61) \mathbb{E}_t\Delta T_t\le-u\eta_yT_t+v\eta_xX_t+w\eta_yC_t+C\eta_y\omega+C(\eta_y^2+\eta_x^2). \tag*{(61)} E t Δ T t ≤ − u η y T t + v η x X t + w η y C t + C η y ω + C ( η y 2 + η x 2 ) . (61) The constants can be fixed uniformly over subsequently smaller step-size ratios.
Proof. Use the simultaneous Taylor estimate of Lemma C.46 . The population actor direction decreases T \mathcal{T} T by ρ 0 T \rho_0\mathcal{T} ρ 0 T , where ρ 0 > 0 \rho_0 > 0 ρ 0 > 0 is fixed, and ∥ ∇ T ∥ ≤ C T \lVert\nabla\mathcal{T}\rVert\le C\sqrt{\mathcal{T}} ∥ ∇ T ∥ ≤ C T by Lemma C.19 . Thus lower conditional bias adds at most C η y T t C t + ω C\eta_y\sqrt{\mathcal{T}_t}\sqrt{\mathcal{C}_t}+\omega C η y T t C t + ω . Young’s inequality leaves ρ 0 η y T t / 4 \rho_0\eta_y\mathcal{T}_t/4 ρ 0 η y T t /4 and C η y ( C t + ω ) C\eta_y(\mathcal{C}_t+\omega) C η y ( C t + ω ) . The mean local-map remainders and all quadratic Taylor terms cost C ( η y 2 + η x 2 ) C(\eta_y^2+\eta_x^2) C ( η y 2 + η x 2 ) . The finite numerical error contributes at most C η y 2 C\eta_y^2 C η y 2 because the potential gradient is uniformly bounded on the prospective neighborhood. Equivalently, Proposition C.37 transfers each one-step inequality from the virtual exact step at the same old iterate.
The upper contribution is bounded by C η x ( T t + X t + C t + ω ) C\eta_x(\mathcal{T}_t+\mathcal{X}_t+\mathcal{C}_t+\omega) C η x ( T t + X t + C t + ω ) because ∥ V c ∥ ≤ C ( X t + T t + c ) \lVert V_c\rVert\le C(\sqrt{\mathcal{X}_t}+\sqrt{\mathcal{T}_t}+c) ∥ V c ∥ ≤ C ( X t + T t + c ) . Fix an upper bound on q = η x / η y q=\eta_x/\eta_y q = η x / η y small enough to absorb its T t \mathcal{T}_t T t coefficient. Since η x ≤ η y \eta_x\le\eta_y η x ≤ η y , the remaining terms give (61) , with constants unchanged when q q q is subsequently reduced. This calculation uses the full Hessian of the tracking potential and requires no smoothness of the chosen metric as a function of the iterates.
For the outer step, global L Φ L_\Phi L Φ -smoothness gives
E t Δ Φ t ≤ − η x ⟨ ∇ Φ ( x t ) , V c ( p t ) + e V , t ⟩ + 1 2 L Φ η x 2 E t ∥ V ^ t ∥ 2 . (62) \mathbb{E}_t\Delta\Phi_t\le-\eta_x\langle\nabla\Phi(x_t),V_c(p_t)+e_{V,t}\rangle+\frac{1}{2}L_\Phi\eta_x^2\mathbb{E}_t\lVert\widehat{V}_t\rVert^2. \tag*{(62)} E t Δ Φ t ≤ − η x ⟨ ∇Φ ( x t ) , V c ( p t ) + e V , t ⟩ + 2 1 L Φ η x 2 E t ∥ V t ∥ 2 . (62)
Here e V , t = E t V ^ t − V c ( p t ) e_{V,t}=\mathbb{E}_t\widehat{V}_t-V_c(p_t) e V , t = E t V t − V c ( p t ) . Lemma C.21 bounds the squared secant error by C ( T t + c 2 ) C(\mathcal{T}_t+c^2) C ( T t + c 2 ) , and Lemma B.8 bounds the additional squared bias by C ( C t + A Q + b e v 2 ) C(\mathcal{C}_t+A_Q+b_{\mathrm{ev}}^2) C ( C t + A Q + b ev 2 ) . Young’s inequality and bounded second moments yield (60). The actual fixed-dynamics outer estimator has no critic dependence; the displayed d η x C t d\eta_x\mathcal{C}_t d η x C t is a harmless conservative bound.
A contracting potential for containment
The step ratios r Q = η y / β r_Q=\eta_y/\beta r Q = η y / β and q = η x / η y q=\eta_x/\eta_y q = η x / η y are positive constants at most one. We first construct a contracting actor–critic potential; the objective-containing potential used for stationarity is defined separately in Step B.6.
Theorem B.11 (Combined tracking contraction). Choose λ > 0 \lambda>0 λ > 0 with λ k > w \lambda k>w λk > w , and reduce the admissible upper bound on r Q r_Q r Q so that λ C r Q 2 ≤ u / 2 \lambda C r_Q^2\le u/2 λ C r Q 2 ≤ u /2 in (57) . Define
D t = T t + λ r Q C t . (63) D_t=\mathcal{T}_t+\lambda r_Q\mathcal{C}_t. \tag*{(63)} D t = T t + λ r Q C t . (63) There are ρ D , C D > 0 \rho_D,C_D>0 ρ D , C D > 0 , uniform over subsequently smaller r Q , q r_Q,q r Q , q , such that
E t D t + 1 ≤ ( 1 − ρ D η y ) D t + η y D ε , D ε = C D ( q G Φ 2 + ω + β ) . (64) \mathbb{E}_tD_{t+1}\le(1-\rho_D\eta_y)D_t+\eta_yD_\varepsilon,\qquad D_\varepsilon=C_D(qG_\Phi^2+\omega+\beta). \tag*{(64)} E t D t + 1 ≤ ( 1 − ρ D η y ) D t + η y D ε , D ε = C D ( q G Φ 2 + ω + β ) . (64) Proof. Add (61) to λ r Q \lambda r_Q λ r Q times (57) . After dividing by η y \eta_y η y , the coefficients of T t , C t , X t \mathcal{T}_t,\mathcal{C}_t,\mathcal{X}_t T t , C t , X t are
− u + λ C r Q 2 , w − λ k , v q + λ C q 2 r Q 2 . -u+\lambda C r_Q^2,\qquad w-\lambda k,\qquad vq+\lambda Cq^2r_Q^2. − u + λ C r Q 2 , w − λk , v q + λ C q 2 r Q 2 . Let δ C = λ k − w > 0 \delta_C=\lambda k-w>0 δ C = λk − w > 0 and choose ρ D = min { u / 2 , δ C / λ } \rho_D=\min\{u/2,\delta_C/\lambda\} ρ D = min { u /2 , δ C / λ } . Since r Q ≤ 1 r_Q\le1 r Q ≤ 1 , the negative terms dominate − ρ D ( T t + λ r Q C t ) -\rho_D(\mathcal{T}_t+\lambda r_Q\mathcal{C}_t) − ρ D ( T t + λ r Q C t ) . The X t \mathcal{X}_t X t coefficient is at most ( v + λ C ) q (v+\lambda C)q ( v + λ C ) q , and X t ≤ G Φ 2 \mathcal{X}_t\le G_\Phi^2 X t ≤ G Φ 2 . The bias terms are at most C ( 1 + λ ) ω C(1+\lambda)\omega C ( 1 + λ ) ω . The divided step remainders equal, up to a fixed factor,
η y + η x 2 / η y + λ ( β + η y 2 / β + η x 2 / β ) ≤ C ( 1 + λ ) β . \eta_y+\eta_x^2/\eta_y+\lambda(\beta+\eta_y^2/\beta+\eta_x^2/\beta)\le C(1+\lambda)\beta. η y + η x 2 / η y + λ ( β + η y 2 / β + η x 2 / β ) ≤ C ( 1 + λ ) β . Absorb these constants into C D C_D C D . No inverse power of a subsequently reduced ratio enters either ρ D \rho_D ρ D or C D C_D C D . The finite solver adds at most C ( 1 + λ r Q ) η y 2 C(1+\lambda r_Q)\eta_y^2 C ( 1 + λ r Q ) η y 2 to the unweighted one-step comparison in Proposition C.37. After division by η y \eta_y η y , this is O ( η y ) ≤ O ( β ) O(\eta_y)\le O(\beta) O ( η y ) ≤ O ( β ) , already included in D ε D_\varepsilon D ε . Thus the recurrence concerns the implemented process, with enlarged fixed remainder constants.
Constructively excluding a first exit
Use the actor radius R A = R R_A = R R A = R and separation radius s s s from Step B.1. The local-solve radius r p r o x r_{\mathrm{prox}} r prox obeys (157) . Put
S max = s / 2 , c 0 s ≤ R A / 8 , r p r o x ≤ R A / 8 , b c a p = min { R A / 4 , s / 8 } , T c a p = b c a p 2 μ τ / E ∗ . (65) \begin{aligned}
S_{\max} &= s/2, & c_0s &\le R_A/8, & r_{\mathrm{prox}} &\le R_A/8,\\
b_{\mathrm{cap}} &= \min\{R_A/4,s/8\}, & T_{\mathrm{cap}} &= b_{\mathrm{cap}}^2\mu_\tau/E_*. \tag*{(65)}
\end{aligned} S m a x b cap = s /2 , = min { R A /4 , s /8 } , c 0 s T cap ≤ R A /8 , = b cap 2 μ τ / E ∗ . r prox ≤ R A /8 , (65)
The canonical observable certificate is
Safe ( U , S ) : ∥ S ∥ < s / 2 and ∥ U ∥ < R A − c 0 s − r p r o x . (66) \operatorname{Safe}(U,S):\quad\lVert S\rVert< s/2 \quad\text{and}\quad\lVert U\rVert< R_A-c_0s-r_{\mathrm{prox}}. \tag*{(66)} Safe ( U , S ) : ∥ S ∥ < s /2 and ∥ U ∥ < R A − c 0 s − r prox . (66)
Let τ a = inf { k ≥ 0 : Safe ( U k , S k ) fails } \tau_a=\inf\{k\ge0:\operatorname{Safe}(U_k,S_k)\text{ fails}\} τ a = inf { k ≥ 0 : Safe ( U k , S k ) fails } , with infimum + ∞ +\infty + ∞ for the empty set. The check includes k = T k=T k = T . The larger analytic neighborhood is ∥ S ∥ < s \lVert S\rVert< s ∥ S ∥ < s , ∥ U ∥ < R A − c 0 s − r p r o x / 2 \lVert U\rVert< R_A-c_0s-r_{\mathrm{prox}}/2 ∥ U ∥ < R A − c 0 s − r prox /2 ; it contains the prospective step from a safe point when the increments are sufficiently small. There is no upper-domain exit test. Lemma C.51 controls innovations through that prospective step, including projected critic updates.
Theorem B.12 (Maximum bound through a prospective first exit). Let τ a \tau_a τ a be the first completed iteration at which an actor certificate fails. Suppose (63)–(240) hold before and through that prospective step, and 0 < ρ D η y ≤ 1 0 < \rho_D\eta_y \le1 0 < ρ D η y ≤ 1 . With probability at least 1 − δ 1-\delta 1 − δ ,
max 0 ≤ k ≤ T ∧ τ a D k ≤ max { D 0 , D ε ρ D } + B D ( H ) 2 η y ρ D log 2 T δ . (67) \max_{0\le k\le T\wedge\tau_a}D_k\le\max\left\{D_0,\frac{D_\varepsilon}{\rho_D}\right\}+B_D(H)\sqrt{\frac{2\eta_y}{\rho_D}\log\frac{2T}{\delta}}. \tag*{(67)} 0 ≤ k ≤ T ∧ τ a max D k ≤ max { D 0 , ρ D D ε } + B D ( H ) ρ D 2 η y log δ 2 T . (67) Proof. Put a = 1 − ρ D η y a=1-\rho_D\eta_y a = 1 − ρ D η y and multiply each pre-exit innovation by the predictable indicator I j = 1 { j < τ a } I_j=\mathbf{1}_{\{j<\tau_a\}} I j = 1 { j < τ a } , setting it to zero afterward. For a deterministic k k k , on the event k ≤ τ a k\le\tau_a k ≤ τ a , iteration of the conditional inequality gives
D k ≤ a k D 0 + D ε ρ D ( 1 − a k ) + ∑ j = 0 k − 1 a k − 1 − j I j N j + 1 . D_k\le a^kD_0+\frac{D_\varepsilon}{\rho_D}(1-a^k)+\sum_{j=0}^{k-1}a^{k-1-j}I_jN_{j+1}. D k ≤ a k D 0 + ρ D D ε ( 1 − a k ) + j = 0 ∑ k − 1 a k − 1 − j I j N j + 1 . The last sum is a weighted martingale sum for this fixed k k k . Its squared increment bounds sum to at most η y 2 B D ( H ) 2 ∑ i ≥ 0 a 2 i ≤ η y B D ( H ) 2 / ρ D \eta_y^2B_D(H)^2\sum_{i\ge0}a^{2i}\le\eta_yB_D(H)^2/\rho_D η y 2 B D ( H ) 2 ∑ i ≥ 0 a 2 i ≤ η y B D ( H ) 2 / ρ D . The bounded-increment exponential martingale inequality bounds its upper tail by δ / ( 2 T ) \delta/(2T) δ / ( 2 T ) at the deviation in (67) . A union bound over k ≤ T k\le T k ≤ T gives the result (with slack in the failure probability). The first two deterministic terms are a convex combination. The comparison is asserted only through the exit; contraction of a frozen post-exit value is neither used nor needed.
Theorem B.13 (Constructive containment for the trained critic). Under Assumptions 4.1, 4.2, 4.3, 4.4, and 4.5, condition on any realized accepted metric sample from Definition C.28 . Use the finite actor solvers and specified critic and sampling algorithm, initialize U 0 = S 0 = 0 U_0=S_0=0 U 0 = S 0 = 0 and any deterministic z 0 ∈ K Q z_0\in\mathcal{K}_Q z 0 ∈ K Q . There are finite fixed choices of regularization, radii, weights, positive step ratios, and positive sampling/tuning constants, uniform over accepted metric realizations, such that, for all sufficiently small ε \varepsilon ε ,
β = k β ε , η y = r Q β , η x = q η y , c 2 = k c ε , T = ⌈ k T ε − 2 ⌉ , (68) \beta=k_\beta\varepsilon,\qquad\eta_y=r_Q\beta,\qquad\eta_x=q\eta_y,\qquad c^2=k_c\varepsilon,\qquad T=\lceil k_T\varepsilon^{-2}\rceil, \tag*{(68)} β = k β ε , η y = r Q β , η x = q η y , c 2 = k c ε , T = ⌈ k T ε − 2 ⌉ , (68) with m Q ≥ max { m Q , 0 , C m ε − 1 / 2 } m_Q\ge\max\{m_{Q,0},C_m\varepsilon^{-1/2}\} m Q ≥ max { m Q , 0 , C m ε − 1/2 } and logarithmic H H H , satisfies
P ( τ a ≤ T ∣ s 1 , … , s n ) ≤ δ a , δ a = ε / 2. (69) \mathbb{P}(\tau_a\le T\mid s_1,\ldots,s_n)\le\delta_a,\qquad\delta_a=\varepsilon/2. \tag*{(69)} P ( τ a ≤ T ∣ s 1 , … , s n ) ≤ δ a , δ a = ε /2. (69) The critic remains in K Q \mathcal{K}_Q K Q by construction. No upper compactness, accurate critic initialization, or explicit augmented containment margin is assumed.
Proof. 1. Fix actor geometry before critic constants. The actor radius and tube are selected from the actor certificate. Increase the finite regularization τ \tau τ until the actor curvature conditions hold and
J 1 / μ τ < R A / 4 , G f / μ τ < s / 4 , T 0 < T c a p / 4. (70) J_1/\mu_\tau< R_A/4,\qquad G_f/\mu_\tau< s/4,\qquad T_0 < T_{\mathrm{cap}}/4. \tag*{(70)} J 1 / μ τ < R A /4 , G f / μ τ < s /4 , T 0 < T cap /4. (70)
This is possible by (42) , since T 0 = O ( τ − 1 ) T_0 = O(\tau^{-1}) T 0 = O ( τ − 1 ) and T c a p = Θ ( τ ) T_{\mathrm{cap}} = \Theta(\tau) T cap = Θ ( τ ) . Now keep τ \tau τ fixed. C2 concerns the lower criterion at this selected value. Construct R Q , m Q , 0 , R u , R v , M R_Q,m_{Q,0},R_u,R_v,M R Q , m Q , 0 , R u , R v , M as in Step B.2, reducing c 0 c_0 c 0 within the preliminary cap as needed. The resulting constants may depend on this τ \tau τ ; the proof never increases τ \tau τ afterward.
2. Fix weights and then the ratios. Let C max = diam W ( K Q ) 2 C_{\max}=\operatorname{diam}_{\mathcal{W}}(K_Q)^2 C m a x = diam W ( K Q ) 2 , which bounds C 0 C_0 C 0 uniformly in x , m Q , c x,m_Q,c x , m Q , c . Fix the weight λ \lambda λ from Theorem B.11 . Also fix the stationarity weights a Ψ , b Ψ a_\Psi,b_\Psi a Ψ , b Ψ from Proposition B.14 in Step B.6 before choosing ratios. Those weights depend only on the already fixed recursion constants, so this forward reference introduces no circular choice. Choose r Q > 0 r_Q>0 r Q > 0 below all critic and combined-drift thresholds, with
λ r Q C max < T c a p / 4. (71) \lambda r_Q C_{\max}<T_{\mathrm{cap}}/4. \tag*{(71)} λ r Q C m a x < T cap /4. (71)
It follows that D 0 < T c a p / 2 D_0<T_{\mathrm{cap}}/2 D 0 < T cap /2 . This controls any bounded initial critic error without a preliminary training phase. Choose q > 0 q>0 q > 0 below both tracking and stationarity thresholds and so small that
C D q G Φ 2 / ρ D < T c a p / 4. (72) C_DqG_\Phi^2/\rho_D<T_{\mathrm{cap}}/4. \tag*{(72)} C D q G Φ 2 / ρ D < T cap /4. (72)
If G Φ = 0 G_\Phi=0 G Φ = 0 , this last restriction is vacuous. The constants C D , ρ D C_D,\rho_D C D , ρ D were fixed uniformly over smaller ratios, so these choices are noncircular.
3. Make the remaining floor and fluctuations small. After the deterministic prospective-step justification in Step 4, the maximum theorem applies with the following choices. Choose any fixed positive k β , k c , k T k_\beta,k_c,k_T k β , k c , k T , and take ε \varepsilon ε sufficiently small that all absolute step restrictions and c ≤ c 0 c\le c_0 c ≤ c 0 hold. Proposition B.7 gives A Q = O ( ε ) A_Q=O(\varepsilon) A Q = O ( ε ) with the stated m Q m_Q m Q . Take H = O ( ( 1 − γ ) − 1 log ( 1 / ε ) ) H=O((1-\gamma)^{-1}\log(1/\varepsilon)) H = O (( 1 − γ ) − 1 log ( 1/ ε )) large enough that b e v 2 + b T D 2 = o ( ε ) b_{\mathrm{ev}}^2+b_{\mathrm{TD}}^2=o(\varepsilon) b ev 2 + b TD 2 = o ( ε ) , and use the normalized batches from Lemma B.8 . Hence ω + β = O ( ε ) \omega+\beta=O(\varepsilon) ω + β = O ( ε ) and D ε / ρ D < T c a p / 2 D_\varepsilon/\rho_D<T_{\mathrm{cap}}/2 D ε / ρ D < T cap /2 for sufficiently small ε \varepsilon ε . With δ = δ a = ε / 2 \delta=\delta_a=\varepsilon/2 δ = δ a = ε /2 and the schedule (68) , the fluctuation term in (67) tends to zero: it is ε \sqrt{\varepsilon} ε times a polynomial in log ( 1 / ε ) \log(1/\varepsilon) log ( 1/ ε ) . It is eventually less than T c a p / 4 T_{\mathrm{cap}}/4 T cap /4 .
4. Justify the prospective step before using its inequality. At a pre-exit point, (66) leaves room for U + c B ‾ s U+c\overline{B}s U + c B s and the local solves. The samplewise normalized-pair bounds imply ∥ Δ U ∥ < r p r o x / 2 \lVert\Delta U\rVert<r_{\mathrm{prox}}/2 ∥ Δ U ∥ < r prox /2 and ∥ Δ S ∥ < s / 4 \lVert\Delta S\rVert<s/4 ∥ Δ S ∥ < s /4 when ε \varepsilon ε is sufficiently small, because each is at most ε poly ( H ) \varepsilon\operatorname{poly}(H) ε poly ( H ) . Thus the prospective endpoint and every segment to it lie in the larger actor tube with ∥ S ∥ < s \lVert S\rVert<s ∥ S ∥ < s and with the endpoint policies in B R A B_{R_A} B R A . Choose the preliminary neighborhoods with strict extra slack if a closed ball is used in a derivative bound. All local-solver and target derivative estimates therefore apply to this step before it is known to remain in the smaller certified set. The critic update stays in K Q K_Q K Q by projection, and its target at the prospective endpoint lies in that same set by Proposition B.6 . Upper segments require no buffer restriction because the estimates are global in x x x . In detail, each numerical iterate remains in its buffered local ball by projection. The virtual reference update has the stated deterministic increment bound; the implemented update differs by at most 5 η y 2 \sqrt{5}\eta_y^2 5 η y 2 in ( U , S ) (U,S) ( U , S ) by Lemma C.35 . This proves the prospective claim for both updates, and the segment between them, without first assuming containment (Lemma C.36 ).
5. Contradict the first exit. On the event from Theorem B.12 , the previous choices give D k < T c a p D_k<T_{\mathrm{cap}} D k < T cap for every k ≤ T / τ a k\le T/\tau_a k ≤ T / τ a . Since T k ≤ D k T_k\le D_k T k ≤ D k , (42) makes each actor error strictly smaller than b c a p b_{\mathrm{cap}} b cap . At a candidate first exit this implies
∥ U k ∥ < J 1 / μ τ + b c a p < R A / 2 , ∥ S k ∥ < G f / μ τ + b c a p < 3 s / 8 < s / 2. \lVert U_k\rVert<J_1/\mu_\tau+b_{\mathrm{cap}}<R_A/2,\qquad
\lVert S_k\rVert<G_f/\mu_\tau+b_{\mathrm{cap}}<3s/8<s/2. ∥ U k ∥ < J 1 / μ τ + b cap < R A /2 , ∥ S k ∥ < G f / μ τ + b cap < 3 s /8 < s /2.
Also c 0 s + r p r o x ≤ R A / 4 c_0s+r_{\mathrm{prox}}\le R_A/4 c 0 s + r prox ≤ R A /4 , so the distance of U k U_k U k from ∂ B R A \partial B_{R_A} ∂ B R A is strictly greater than the required buffer. Both certificates hold strictly, a contradiction. Therefore this event excludes τ a ≤ T \tau_a\le T τ a ≤ T and has conditional probability at least 1 − δ a 1-\delta_a 1 − δ a . Averaging this uniform conditional bound and adding the independent metric rejection budget δ m e t = ε / 2 \delta_{\mathrm{met}}=\varepsilon/2 δ met = ε /2 gives total failure probability at most ε \varepsilon ε . □ \square □
Stationarity and complete data accounting
The contracting potential D \mathcal{D} D proves containment. For stationarity we use positive weights on the actor and critic errors and include the lower-bounded selected objective.
Proposition B.14 (Augmented objective descent). Choose a Ψ , b Ψ > 0 a_{\Psi}, b_{\Psi} > 0 a Ψ , b Ψ > 0 with a Ψ u > b + 1 a_{\Psi}u > b + 1 a Ψ u > b + 1 and b Ψ k > d + a Ψ w + 1 b_{\Psi}k > d + a_{\Psi}w + 1 b Ψ k > d + a Ψ w + 1 , using the constants in (60)–(61). For sufficiently small fixed r Q , q r_Q, q r Q , q ,
Ψ t = Φ ( x t ) − Φ + a Ψ q T t + b Ψ q r Q C t (73) \Psi_t = \Phi(x_t) - \Phi+ a_{\Psi}qT_t + b_{\Psi}qr_QC_t \tag*{(73)} Ψ t = Φ ( x t ) − Φ + a Ψ q T t + b Ψ q r Q C t (73) satisfies
E t Δ Ψ t ≤ − a ∗ η x ( X t + T t + C t ) + C ∗ η x ( ω + β + η y + η x ) (74) \mathbb{E}_t\Delta\Psi_t \le-a_*\eta_x(X_t + T_t + C_t) + C_*\eta_x(\omega+ \beta+ \eta_y + \eta_x) \tag*{(74)} E t Δ Ψ t ≤ − a ∗ η x ( X t + T t + C t ) + C ∗ η x ( ω + β + η y + η x ) (74) through the prospective actor exit, with fixed positive constants.
Proof. Add (60), a Ψ η x / η y a_{\Psi}\eta_x/\eta_y a Ψ η x / η y times (61) , and b Ψ η x / β b_{\Psi}\eta_x/\beta b Ψ η x / β times (57) . After division by η x \eta_x η x , the three coefficients are bounded above by
− a + a Ψ v q + b Ψ C q 2 r Q 2 , b − a Ψ u + b Ψ C r Q 2 , d + a Ψ w − b Ψ k . -a + a_{\Psi}vq + b_{\Psi}Cq^2r_Q^2,\qquad b - a_{\Psi}u + b_{\Psi}Cr_Q^2,\qquad d + a_{\Psi}w - b_{\Psi}k. − a + a Ψ v q + b Ψ C q 2 r Q 2 , b − a Ψ u + b Ψ C r Q 2 , d + a Ψ w − b Ψ k .
Choose the weights first, then r Q r_Q r Q to make b Ψ C r Q 2 < 1 / 2 b_{\Psi}Cr_Q^2 < 1/2 b Ψ C r Q 2 < 1/2 , and q q q to make a Ψ v q + b Ψ C q 2 r Q 2 < a / 2 a_{\Psi}vq + b_{\Psi}Cq^2r_Q^2 < a/2 a Ψ v q + b Ψ C q 2 r Q 2 < a /2 . These can be imposed simultaneously with the containment choices. Weighted remainder terms have the bound in (74) .
Proof of Theorem 4.6 . Fix the actor initialization and retained chart, and take δ m e t = δ a = ε / 2 \delta_{\mathrm{met}} = \delta_a = \varepsilon/2 δ met = δ a = ε /2 . Proposition C.29 bounds metric rejection by δ m e t \delta_{\mathrm{met}} δ met . Conditional on any accepted metric sample, Proposition C.30 supplies the same Fisher constants uniformly on the actor ball. All following constants are independent of the accepted realization. The finite-solver errors are already included in the tracking recursions by Proposition C.37. In particular their weighted contribution to Ψ \Psi Ψ is at most C q η y 2 = C η x η y Cq\eta_y^2 = C\eta_x\eta_y C q η y 2 = C η x η y , which is absorbed by (74) .
For each such metric realization, stop at τ a \tau_a τ a and freeze the process afterward. Multiply (74) by the predictable pre-exit indicator and telescope, including the prospective exit step:
a ∗ η x ∑ t < T E [ 1 { t < τ a } X t ∣ s 1 , … , s n ] ≤ Ψ 0 + C ∗ η x T ( ω + β + η y + η x ) . a_*\eta_x\sum_{t<T}\mathbb{E}[\mathbf{1}_{\{t<\tau_a\}}X_t\mid s_1,\ldots,s_n]\le\Psi_0 + C_*\eta_xT(\omega+ \beta+ \eta_y + \eta_x). a ∗ η x t < T ∑ E [ 1 { t < τ a } X t ∣ s 1 , … , s n ] ≤ Ψ 0 + C ∗ η x T ( ω + β + η y + η x ) .
The terminal potential is nonnegative. For t < T t<T t < T , 1 { τ a > T } ≤ 1 { t < τ a } \mathbf{1}_{\{\tau_a>T\}}\le\mathbf{1}_{\{t<\tau_a\}} 1 { τ a > T } ≤ 1 { t < τ a } ; Theorem B.13 gives the uniform conditional bound P ( τ a ≤ T ∣ s 1 , … , s n ) ≤ δ a \mathbb{P}(\tau_a\le T\mid s_1,\ldots,s_n)\le\delta_a P ( τ a ≤ T ∣ s 1 , … , s n ) ≤ δ a . Draw the independent uniform output index on success and return x 0 x_0 x 0 on either failure. Averaging over the metric data and using ∥ ∇ Φ ( x 0 ) ∥ ≤ G Φ \lVert\nabla\Phi(x_0)\rVert\le G_{\Phi} ∥ ∇Φ ( x 0 )∥ ≤ G Φ gives
E ∥ ∇ Φ ( x o u t ) ∥ 2 ≤ C [ Ψ 0 η x T + c 2 + A Q + b e v 2 + b T D 2 + β + η y + η x ] + ( δ m e t + δ a ) G Φ 2 . (75) \mathbb{E}\left\lVert\nabla\Phi(x_{\mathrm{out}})\right\rVert^2
\le C\left[\frac{\Psi_0}{\eta_xT}+c^2+A_Q+b_{\mathrm{ev}}^2+b_{\mathrm{TD}}^2+\beta+\eta_y+\eta_x\right]+(\delta_{\mathrm{met}}+\delta_a)G_{\Phi}^2. \tag*{(75)} E ∥ ∇Φ ( x out ) ∥ 2 ≤ C [ η x T Ψ 0 + c 2 + A Q + b ev 2 + b TD 2 + β + η y + η x ] + ( δ met + δ a ) G Φ 2 . (75)
Here the initial potential is bounded uniformly over metric realizations: Φ ( x 0 ) − Φ ≤ 2 B L \Phi(x_0)-\Phi\le2B_L Φ ( x 0 ) − Φ ≤ 2 B L (or B L B_L B L for Bradley–Terry), and T 0 , C 0 T_0,C_0 T 0 , C 0 have the fixed bounds proved above. This argument conditions only on initial metric data and uses predictable stopping; it never conditions a descent inequality on future successful training. Metric acceptance reproduces the existing geometric constants, so no enlargement of the C2 neighborhood or increase in fixed τ \tau τ is needed. The union of metric rejection and actor exit has probability at most δ m e t + δ a = ε \delta_{\mathrm{met}}+\delta_a=\varepsilon δ met + δ a = ε .
Under schedule (17) , m Q ≥ max { m Q , 0 , C m ε − 1 / 2 } m_Q\ge\max\{m_{Q,0},C_m\varepsilon^{-1/2}\} m Q ≥ max { m Q , 0 , C m ε − 1/2 } gives A Q = O ( ε ) A_Q=O(\varepsilon) A Q = O ( ε ) , and logarithmic H H H makes b e v 2 + b T D 2 = o ( ε ) b_{\mathrm{ev}}^2+b_{\mathrm{TD}}^2=o(\varepsilon) b ev 2 + b TD 2 = o ( ε ) . Also ( η x T ) − 1 = O ( ε ) (\eta_xT)^{-1}=O(\varepsilon) ( η x T ) − 1 = O ( ε ) and every remaining term has order O ( ε ) O(\varepsilon) O ( ε ) . This proves the claimed stationarity bound.
Let B C B_C B C , B H B_H B H , B V B_V B V count TD, lower evaluation and outer discounted sample draws per iteration, and B f , a B_{f,a} B f , a , B f , x B_{f,x} B f , x the tuple-batch sizes. A TD or discounted evaluation draw uses at most H + 1 H+1 H + 1 transitions; a tuple uses at most I K IK I K transitions. The complete conservative ledger is
N s e g ≤ T [ B C + B H + B V + I ( B f , a + B f , x ) ] + n + N p i l o t , s e g , N t r a n s ≤ T [ ( H + 1 ) ( B C + B H + B V ) + I K ( B f , a + B f , x ) ] + n + N p i l o t , t r a n s . (76) \begin{aligned}
N_{\mathrm{seg}} &\le T[B_C+B_H+B_V+I(B_{f,a}+B_{f,x})]+n+N_{\mathrm{pilot,seg}},\\
N_{\mathrm{trans}} &\le T[(H+1)(B_C+B_H+B_V)+IK(B_{f,a}+B_{f,x})]+n+N_{\mathrm{pilot,trans}}. \tag*{(76)}
\end{aligned} N seg N trans ≤ T [ B C + B H + B V + I ( B f , a + B f , x )] + n + N pilot , seg , ≤ T [( H + 1 ) ( B C + B H + B V ) + I K ( B f , a + B f , x )] + n + N pilot , trans . (76)
The n n n charges the one-time metric reset sample from (188) . Each reset is conservatively charged as one segment and at most one transition; when no transition follows the reset its actual transition contribution is zero. Optional, separately collected initialization-pilot data are added once; they are unnecessary when A2 is assumed. The fixed-gap pilot in Proposition C.13 , if used, needs O ( log ( 1 / ε ) ) O(\log(1/\varepsilon)) O ( log ( 1/ ε )) draws. For fixed γ \gamma γ , r r r , G Z G_Z G Z , λ 0 \lambda_0 λ 0 , I I I , K K K , n = O ( log ( 1 / ε ) ) n=O(\log(1/\varepsilon)) n = O ( log ( 1/ ε )) . Discounted training batches are fixed or at most polynomial in H H H , preference batches are fixed, and labels are charged for both preference blocks. Thus N s e g , N t r a n s = O ~ ( ε − 2 ) N_{\mathrm{seg}},N_{\mathrm{trans}}=\widetilde{O}(\varepsilon^{-2}) N seg , N trans = O ( ε − 2 ) and N l a b e l s = O ( ε − 2 ) N_{\mathrm{labels}}=O(\varepsilon^{-2}) N labels = O ( ε − 2 ) ; every persistent TD update is counted. Reusing metric states during numerical optimization incurs no additional environment interactions or preference labels.
There are at most two finite local solves per completed training iteration. Proposition C.33 therefore gives
N K L − g r a d ≤ 2 T ( J max + 1 ) = O ( κ K ε − 2 log ( 1 / ε ) ) , κ K = M K , L m K , L . (77) N_{\mathrm{KL-grad}}\le2T(J_{\max}+1)=O\left(\kappa_K\varepsilon^{-2}\log(1/\varepsilon)\right),\qquad\kappa_K=\frac{M_{K,L}}{m_{K,L}}. \tag*{(77)} N KL − grad ≤ 2 T ( J m a x + 1 ) = O ( κ K ε − 2 log ( 1/ ε ) ) , κ K = m K , L M K , L . (77)
Indeed G p r o x ( H ) = poly ( H ) G_{\mathrm{prox}}(H)=\operatorname{poly}(H) G prox ( H ) = poly ( H ) , H = O ( log ( 1 / ε ) ) H=O(\log(1/\varepsilon)) H = O ( log ( 1/ ε )) and c η y = Θ ( ε 3 / 2 ) c\eta_y=\Theta(\varepsilon^{3/2}) c η y = Θ ( ε 3/2 ) in (205) , while κ K \kappa_K κ K is independent of n n n and accuracy. A finite-sum gradient has at most n ∣ A ∣ n|\mathcal{A}| n ∣ A ∣ state–action terms. If C a c t ( m , r ) C_{\mathrm{act}}(m,r) C act ( m , r ) bounds the arithmetic for one actor term, including its first derivative, the numerical work is
O ( T ( J max + 1 ) [ n ∣ A ∣ C a c t ( m , r ) + r ] ) . O\left(T(J_{\max}+1)[n|\mathcal{A}|C_{\mathrm{act}}(m,r)+r]\right). O ( T ( J m a x + 1 ) [ n ∣ A ∣ C act ( m , r ) + r ] ) .
Constructing the metric additionally needs n ∣ A ∣ n|\mathcal{A}| n ∣ A ∣ actor-feature evaluations, covariance accumulation and one r × r r\times r r × r spectral certificate. These finite computations are separate from environment interactions and from the other network arithmetic. Although endpoint parameter accuracy is c η y 2 = Θ ( ε 5 / 2 ) c\eta_y^2=\Theta(\varepsilon^{5/2}) c η y 2 = Θ ( ε 5/2 ) , geometric convergence of the local solver makes its additional iteration cost logarithmic; it does not introduce an ε − 5 / 2 \varepsilon^{-5/2} ε − 5/2 sample exponent.
Supporting Results
These construction details and auxiliary results support the six-step proof in Appendix § 9 . Assumptions and the selected target are those in the main text; no additional upper localization or oracle-accuracy assumption is introduced for the main theorem. The optional global-response results in Section C.12 explicitly state their additional conditions.
Actor construction and initialization certificates
The following normalized architecture supplies A1(iii). A2 is the separate initialization covariance condition; hidden width alone does not imply it. All construction constants are fixed before choosing τ \tau τ . Fix the hidden depth L L L , input width m 0 = d ψ m_0=d_\psi m 0 = d ψ , and hidden widths
m = ( m 1 , … , m L ) . \mathbf{m}=(m_1,\ldots,m_L). m = ( m 1 , … , m L ) .
For a branch parameter
ϑ = ( W 1 , … , W L ) , W ℓ ∈ R m ℓ × m ℓ − 1 , \vartheta=(W^1,\ldots,W^L),\qquad W^\ell\in\mathbb{R}^{m_\ell\times m_{\ell-1}}, ϑ = ( W 1 , … , W L ) , W ℓ ∈ R m ℓ × m ℓ − 1 ,
define
h ϑ 0 ( s , a ) = ψ ( s , a ) , (78) h_{\vartheta}^{0}(s,a)=\psi(s,a), \tag*{(78)} h ϑ 0 ( s , a ) = ψ ( s , a ) , (78)
z ϑ ℓ ( s , a ) = 1 m ℓ − 1 W ℓ h ϑ ℓ − 1 ( s , a ) , (79) z_{\vartheta}^{\ell}(s,a)=\frac{1}{\sqrt{m_{\ell-1}}}W^{\ell}h_{\vartheta}^{\ell-1}(s,a), \tag*{(79)} z ϑ ℓ ( s , a ) = m ℓ − 1 1 W ℓ h ϑ ℓ − 1 ( s , a ) , (79)
h ϑ ℓ ( s , a ) = σ ( z ϑ ℓ ( s , a ) ) , ℓ = 1 , … , L , (80) h_{\vartheta}^{\ell}(s,a)=\sigma\left(z_{\vartheta}^{\ell}(s,a)\right),\qquad\ell=1,\ldots,L, \tag*{(80)} h ϑ ℓ ( s , a ) = σ ( z ϑ ℓ ( s , a ) ) , ℓ = 1 , … , L , (80)
G m , ϑ ( s , a ) = 1 m L b T h ϑ L ( s , a ) , b j ∈ { − 1 , + 1 } . (81) G_{m,\vartheta}(s,a)=\frac{1}{\sqrt{m_L}}b^{\mathsf{T}}h_{\vartheta}^{L}(s,a),\qquad b_j\in\{-1,+1\}. \tag*{(81)} G m , ϑ ( s , a ) = m L 1 b T h ϑ L ( s , a ) , b j ∈ { − 1 , + 1 } . (81)
The activations are applied coordinatewise and the readout b b b is fixed. Equip the branch parameter space with the product Frobenius norm
∥ Δ ϑ ∥ p a r 2 : = ∑ ℓ = 1 L ∥ Δ W ℓ ∥ F 2 . (82) \lVert\Delta\vartheta\rVert_{\mathrm{par}}^2:=\sum_{\ell=1}^{L}\lVert\Delta W^{\ell}\rVert_{F}^{2}. \tag*{(82)} ∥ Δ ϑ ∥ par 2 := ℓ = 1 ∑ L ∥ Δ W ℓ ∥ F 2 . (82)
Let B m : R r → ∏ ℓ = 1 L R m ℓ × m ℓ − 1 B_m:\mathbb{R}^{r}\to\prod_{\ell=1}^{L}\mathbb{R}^{m_{\ell}\times m_{\ell-1}} B m : R r → ∏ ℓ = 1 L R m ℓ × m ℓ − 1 be an isometry:
B m T B m = I r . (83) B_m^{\mathsf{T}}B_m=I_r. \tag*{(83)} B m T B m = I r . (83)
For an initialization ϑ m , 0 \vartheta_{m,0} ϑ m , 0 , define the two antithetic branches for every α ∈ R r \alpha\in\mathbb{R}^{r} α ∈ R r by
ϑ m ± ( α ) = ϑ m , 0 ± 1 2 B m α , α ∈ R r . (84) \vartheta_m^{\pm}(\alpha)=\vartheta_{m,0}\pm\frac{1}{\sqrt{2}}B_m\alpha,\qquad\alpha\in\mathbb{R}^{r}. \tag*{(84)} ϑ m ± ( α ) = ϑ m , 0 ± 2 1 B m α , α ∈ R r . (84)
Fix an ambient radius A > 0 A>0 A > 0 and let B ‾ A = B ‾ ( 0 , A ) \overline{B}_A=\overline{B}(0,A) B A = B ( 0 , A ) . The geometry construction later selects 0 < R g e o m ≤ A / 2 0<R_{\mathrm{geom}}\le A/2 0 < R geom ≤ A /2 ; write
Θ R : = B ‾ ( 0 , R g e o m ) . (85) \Theta_R:=\overline{B}(0,R_{\mathrm{geom}}). \tag*{(85)} Θ R := B ( 0 , R geom ) . (85)
This closed ball is used only as a local geometry-certification tube. It is not the actor parameter space. The exact energy is
F m , α ( s , a ) = 1 2 [ G m , ϑ m + ( α ) ( s , a ) − G m , ϑ m − ( α ) ( s , a ) ] . (86) F_{m,\alpha}(s,a)=\frac{1}{\sqrt{2}}\left[G_{m,\vartheta_m^{+}(\alpha)}(s,a)-G_{m,\vartheta_m^{-}(\alpha)}(s,a)\right]. \tag*{(86)} F m , α ( s , a ) = 2 1 [ G m , ϑ m + ( α ) ( s , a ) − G m , ϑ m − ( α ) ( s , a ) ] . (86)
This is an exact nonlinear network. It can be implemented as one wider block-diagonal network with a fixed antisymmetric readout. At initialization,
F m , 0 ( s , a ) = 0 , D α 2 F m , 0 ( s , a ) = 0. (87) F_{m,0}(s,a)=0,\qquad D_{\alpha}^{2}F_{m,0}(s,a)=0. \tag*{(87)} F m , 0 ( s , a ) = 0 , D α 2 F m , 0 ( s , a ) = 0. (87)
The reference π r e f ( a ∣ s ) > 0 \pi_{\mathrm{ref}}(a\mid s)>0 π ref ( a ∣ s ) > 0 is fixed and independent of x x x . The policy is its finite exponential tilt:
d π m , α ( ⋅ ∣ s ) d π r e f ( ⋅ ∣ s ) ( a ) = exp { F m , α ( s , a ) − A m , s ( α ) } , (88) \frac{d\pi_{m,\alpha}(\cdot\mid s)}{d\pi_{\mathrm{ref}}(\cdot\mid s)}(a)=\exp\{F_{m,\alpha}(s,a)-A_{m,s}(\alpha)\}, \tag*{(88)} d π ref ( ⋅ ∣ s ) d π m , α ( ⋅ ∣ s ) ( a ) = exp { F m , α ( s , a ) − A m , s ( α )} , (88)
where
A m , s ( α ) = log ∑ a ′ ∈ A e F m , α ( s , a ′ ) π r e f ( a ′ ∣ s ) . (89) A_{m,s}(\alpha)=\log\sum_{a'\in\mathcal{A}}e^{F_{m,\alpha}(s,a')}\pi_{\mathrm{ref}}(a'\mid s). \tag*{(89)} A m , s ( α ) = log a ′ ∈ A ∑ e F m , α ( s , a ′ ) π ref ( a ′ ∣ s ) . (89)
Define the effective energy feature and policy score
Z m , α ( s , a ) : = ∇ α F m , α ( s , a ) , (90) Z_{m,\alpha}(s,a):=\nabla_{\alpha}F_{m,\alpha}(s,a), \tag*{(90)} Z m , α ( s , a ) := ∇ α F m , α ( s , a ) , (90)
Z ‾ m , α ( s , a ) : = Z m , α ( s , a ) − E a ′ ∼ π m , α ( ⋅ ∣ s ) [ Z m , α ( s , a ′ ) ] = ∇ α log π m , α ( a ∣ s ) . (91) \overline{Z}_{m,\alpha}(s,a):=Z_{m,\alpha}(s,a)-\mathbb{E}_{a'\sim\pi_{m,\alpha}(\cdot\mid s)}[Z_{m,\alpha}(s,a')]=\nabla_{\alpha}\log\pi_{m,\alpha}(a\mid s). \tag*{(91)} Z m , α ( s , a ) := Z m , α ( s , a ) − E a ′ ∼ π m , α ( ⋅ ∣ s ) [ Z m , α ( s , a ′ )] = ∇ α log π m , α ( a ∣ s ) . (91)
For a state law ν \nu ν , define
F m , ν ( α ) : = E s ∼ ν , a ∼ π m , α ( ⋅ ∣ s ) [ Z ‾ m , α ( s , a ) Z ‾ m , α ( s , a ) T ] . (92) \mathcal{F}_{m,\nu}(\alpha):=\mathbb{E}_{s\sim\nu,\,a\sim\pi_{m,\alpha}(\cdot\mid s)}\left[\overline{Z}_{m,\alpha}(s,a)\overline{Z}_{m,\alpha}(s,a)^{\mathsf{T}}\right]. \tag*{(92)} F m , ν ( α ) := E s ∼ ν , a ∼ π m , α ( ⋅ ∣ s ) [ Z m , α ( s , a ) Z m , α ( s , a ) T ] . (92)
Write F x , m ( α ) \mathcal{F}_{x,m}(\alpha) F x , m ( α ) when ν = d α π \nu=d_{\alpha}^{\pi} ν = d α π .
Construction requirements. The depth L L L , effective dimension r r r , and ambient radius A A A are independent of the hidden widths. The actor domain itself is R r \mathbb{R}^r R r . The input satisfies ∥ ψ ( s , a ) ∥ ≤ B ψ \|\psi(s,a)\| \le B_\psi ∥ ψ ( s , a ) ∥ ≤ B ψ . Each activation is bounded and belongs to C 4 ( R ) C^4(\mathbb{R}) C 4 ( R ) , with
∥ σ ℓ ( k ) ∥ ∞ ≤ M ℓ , k , k = 0 , 1 , 2 , 3 , 4. (93) \|\sigma_\ell^{(k)}\|_\infty\le M_{\ell,k}, \qquad k=0,1,2,3,4. \tag*{(93)} ∥ σ ℓ ( k ) ∥ ∞ ≤ M ℓ , k , k = 0 , 1 , 2 , 3 , 4. (93)
The initialized normalized layer operators satisfy
∥ W m , 0 ℓ ∥ o p m ℓ − 1 ≤ κ ℓ , ℓ = 1 , … , L , (94) \frac{\|W_{m,0}^{\ell}\|_{\mathrm{op}}}{\sqrt{m_{\ell-1}}} \le\kappa_\ell, \qquad\ell=1,\ldots,L, \tag*{(94)} m ℓ − 1 ∥ W m , 0 ℓ ∥ op ≤ κ ℓ , ℓ = 1 , … , L , (94)
with width-independent constants. The chart map is an isometry as in (83) . If it is selected from data, use data independent of the sample used to certify its covariance gap. Conditional analysis for a fixed accepted chart is also possible.
Chart spectral control and antithetic identities For a branch point ϑ m ± ( α ) \vartheta_m^{\pm}(\alpha) ϑ m ± ( α ) , set
A ± , α ℓ : = 1 m ℓ − 1 W ± , α ℓ . (95) A_{\pm,\alpha}^{\ell} := \frac{1}{\sqrt{m_{\ell-1}}}W_{\pm,\alpha}^{\ell}. \tag*{(95)} A ± , α ℓ := m ℓ − 1 1 W ± , α ℓ . (95)
Lemma C.1 (Uniform normalized spectral bound on the chart). Under Section C.1, on the ambient ball B ‾ A \overline{B}_A B A ,
∥ A ± , α ℓ ∥ o p ≤ κ ℓ + A 2 m ℓ − 1 ≤ κ ‾ ℓ : = κ ℓ + A 2 (96) \|A_{\pm,\alpha}^{\ell}\|_{\mathrm{op}} \le\kappa_\ell+\frac{A}{\sqrt{2m_{\ell-1}}}\le\overline{\kappa}_\ell:=\kappa_\ell+\frac{A}{\sqrt{2}} \tag*{(96)} ∥ A ± , α ℓ ∥ op ≤ κ ℓ + 2 m ℓ − 1 A ≤ κ ℓ := κ ℓ + 2 A (96) for every branch, chart point, and admissible width.
Proof. The layer-ℓ \ell ℓ component of B m α B_m\alpha B m α has Frobenius norm at most ∥ B m α ∥ p a r = ∥ α ∥ ≤ A \|B_m\alpha\|_{\mathrm{par}}=\|\alpha\|\le A ∥ B m α ∥ par = ∥ α ∥ ≤ A . Therefore its operator norm is at most A A A . Divide the triangle inequality for W m , 0 ℓ ± 2 − 1 / 2 ( B m α ) ℓ W_{m,0}^{\ell}\pm2^{-1/2}(B_m\alpha)^\ell W m , 0 ℓ ± 2 − 1/2 ( B m α ) ℓ by m ℓ − 1 \sqrt{m_{\ell-1}} m ℓ − 1 and use (94) .
Lemma C.2 (Antithetic initialization identities). The full two-branch chart is an isometry. Moreover,
F m , 0 ( s , a ) = 0 , (97) F_{m,0}(s,a)=0, \tag*{(97)} F m , 0 ( s , a ) = 0 , (97) Z m , 0 ( s , a ) = B m T ∇ ϑ G m , ϑ m , 0 ( s , a ) . (98) Z_{m,0}(s,a)=B_m^{\mathsf T}\nabla_{\vartheta}G_{m,\vartheta_{m,0}}(s,a). \tag*{(98)} Z m , 0 ( s , a ) = B m T ∇ ϑ G m , ϑ m , 0 ( s , a ) . (98) More generally, for directions u 1 , … , u k ∈ R r u_1,\ldots,u_k\in\mathbb{R}^r u 1 , … , u k ∈ R r ,
D α k F m , α [ u 1 , … , u k ] = 2 − ( k + 1 ) / 2 ( D ϑ k G ϑ + ( α ) [ B m u 1 , … , B m u k ] − ( − 1 ) k D ϑ k G ϑ − ( α ) [ B m u 1 , … , B m u k ] ) . (99) D_{\alpha}^{k}F_{m,\alpha}[u_1,\ldots,u_k]
=2^{-(k+1)/2}\left(D_{\vartheta}^{k}G_{\vartheta_+(\alpha)}[B_mu_1,\ldots,B_mu_k]
-(-1)^kD_{\vartheta}^{k}G_{\vartheta_-(\alpha)}[B_mu_1,\ldots,B_mu_k]\right). \tag*{(99)} D α k F m , α [ u 1 , … , u k ] = 2 − ( k + 1 ) /2 ( D ϑ k G ϑ + ( α ) [ B m u 1 , … , B m u k ] − ( − 1 ) k D ϑ k G ϑ − ( α ) [ B m u 1 , … , B m u k ] ) . (99) Proof. The map
u ⟼ ( 2 − 1 / 2 B m u , − 2 − 1 / 2 B m u ) u\longmapsto\left(2^{-1/2}B_mu,-2^{-1/2}B_mu\right) u ⟼ ( 2 − 1/2 B m u , − 2 − 1/2 B m u ) has squared product norm
1 2 ∥ B m u ∥ p a r 2 + 1 2 ∥ B m u ∥ p a r 2 = ∥ u ∥ 2 . \frac{1}{2}\|B_mu\|_{\mathrm{par}}^2+\frac{1}{2}\|B_mu\|_{\mathrm{par}}^2=\|u\|^2. 2 1 ∥ B m u ∥ par 2 + 2 1 ∥ B m u ∥ par 2 = ∥ u ∥ 2 . The two branch values agree at α = 0 \alpha=0 α = 0 , proving (97) . Differentiating once gives (98) ; repeated differentiation and the chain rule give (99) .
A width-independent mixed-derivative recursion For ℓ = 0 , … , L \ell=0,\ldots,L ℓ = 0 , … , L , define
q 0 : = B ψ m 0 , q ℓ : = M ℓ , 0 ( ℓ ≥ 1 ) . (100) q_0 := \frac{B_\psi}{\sqrt{m_0}}, \qquad q_\ell:= M_{\ell,0}\quad(\ell\ge1). \tag*{(100)} q 0 := m 0 B ψ , q ℓ := M ℓ , 0 ( ℓ ≥ 1 ) . (100)
Set H 0 , k = 0 H_{0,k}=0 H 0 , k = 0 for k = 1 , 2 , 3 , 4 k=1,2,3,4 k = 1 , 2 , 3 , 4 . Recursively, for ℓ = 1 , … , L \ell=1,\ldots,L ℓ = 1 , … , L , let
H ^ ℓ − 1 , 0 : = q ℓ − 1 , H ^ ℓ − 1 , j : = H ℓ − 1 , j ( j = 1 , 2 , 3 ) , (101) \widehat{H}_{\ell-1,0}:=q_{\ell-1},\qquad\widehat{H}_{\ell-1,j}:=H_{\ell-1,j}\quad(j=1,2,3), \tag*{(101)} H ℓ − 1 , 0 := q ℓ − 1 , H ℓ − 1 , j := H ℓ − 1 , j ( j = 1 , 2 , 3 ) , (101)
and
Z ℓ , k : = κ ˉ ℓ H ℓ − 1 , k + k H ^ ℓ − 1 , k − 1 , k = 1 , 2 , 3 , 4. (102) Z_{\ell,k}:=\bar{\kappa}_{\ell}H_{\ell-1,k}+k\widehat{H}_{\ell-1,k-1},\qquad k=1,2,3,4. \tag*{(102)} Z ℓ , k := κ ˉ ℓ H ℓ − 1 , k + k H ℓ − 1 , k − 1 , k = 1 , 2 , 3 , 4. (102)
Define
H ℓ , 1 : = M ℓ , 1 Z ℓ , 1 , (103) H_{\ell,1}:=M_{\ell,1}Z_{\ell,1}, \tag*{(103)} H ℓ , 1 := M ℓ , 1 Z ℓ , 1 , (103)
H ℓ , 2 : = M ℓ , 1 Z ℓ , 2 + M ℓ , 2 Z ℓ , 1 2 , (104) H_{\ell,2}:=M_{\ell,1}Z_{\ell,2}+M_{\ell,2}Z_{\ell,1}^{2}, \tag*{(104)} H ℓ , 2 := M ℓ , 1 Z ℓ , 2 + M ℓ , 2 Z ℓ , 1 2 , (104)
H ℓ , 3 : = M ℓ , 1 Z ℓ , 3 + 3 M ℓ , 2 Z ℓ , 1 Z ℓ , 2 + M ℓ , 3 Z ℓ , 1 3 , (105) H_{\ell,3}:=M_{\ell,1}Z_{\ell,3}+3M_{\ell,2}Z_{\ell,1}Z_{\ell,2}+M_{\ell,3}Z_{\ell,1}^{3}, \tag*{(105)} H ℓ , 3 := M ℓ , 1 Z ℓ , 3 + 3 M ℓ , 2 Z ℓ , 1 Z ℓ , 2 + M ℓ , 3 Z ℓ , 1 3 , (105)
H ℓ , 4 : = M ℓ , 1 Z ℓ , 4 + 4 M ℓ , 2 Z ℓ , 1 Z ℓ , 3 + 3 M ℓ , 2 Z ℓ , 2 2 + 6 M ℓ , 3 Z ℓ , 1 2 Z ℓ , 2 + M ℓ , 4 Z ℓ , 1 4 . (106) \begin{aligned}
H_{\ell,4}:={}&M_{\ell,1}Z_{\ell,4}+4M_{\ell,2}Z_{\ell,1}Z_{\ell,3}+3M_{\ell,2}Z_{\ell,2}^{2}\\
&+6M_{\ell,3}Z_{\ell,1}^{2}Z_{\ell,2}+M_{\ell,4}Z_{\ell,1}^{4}. \tag*{(106)}
\end{aligned} H ℓ , 4 := M ℓ , 1 Z ℓ , 4 + 4 M ℓ , 2 Z ℓ , 1 Z ℓ , 3 + 3 M ℓ , 2 Z ℓ , 2 2 + 6 M ℓ , 3 Z ℓ , 1 2 Z ℓ , 2 + M ℓ , 4 Z ℓ , 1 4 . (106)
All these numbers depend on fixed depth, the activation envelopes, the fixed ambient radius A A A , the input bound, and the normalized spectral envelopes, but not on the hidden widths.
Lemma C.3 (Exact mixed layer-derivative recursion). Fix a branch point and parameter directions Δ 1 ϑ , … , Δ k ϑ \Delta_1\vartheta,\ldots,\Delta_k\vartheta Δ 1 ϑ , … , Δ k ϑ with ∥ Δ i ϑ ∥ p a r ≤ 1 \lVert\Delta_i\vartheta\rVert_{\mathrm{par}}\le1 ∥ Δ i ϑ ∥ par ≤ 1 . For a nonempty subset I ⊆ { 1 , … , k } I\subseteq\{1,\ldots,k\} I ⊆ { 1 , … , k } , write
h I ℓ = D ϑ ∣ I ∣ h ℓ [ Δ i ϑ : i ∈ I ] , h_I^\ell=D_\vartheta^{|I|}h^\ell[\Delta_i\vartheta:i\in I], h I ℓ = D ϑ ∣ I ∣ h ℓ [ Δ i ϑ : i ∈ I ] , z I ℓ = D ϑ ∣ I ∣ z ℓ [ Δ i ϑ : i ∈ I ] . z_I^\ell=D_\vartheta^{|I|}z^\ell[\Delta_i\vartheta:i\in I]. z I ℓ = D ϑ ∣ I ∣ z ℓ [ Δ i ϑ : i ∈ I ] . Then
z I ℓ = A ℓ h I ℓ − 1 + 1 m ℓ − 1 ∑ i ∈ I Δ i W ℓ h I ∖ { i } ℓ − 1 , (107) z_I^\ell=A^\ell h_I^{\ell-1}+\frac{1}{\sqrt{m_{\ell-1}}}\sum_{i\in I}\Delta_iW^\ell h_{I\setminus\{i\}}^{\ell-1}, \tag*{(107)} z I ℓ = A ℓ h I ℓ − 1 + m ℓ − 1 1 i ∈ I ∑ Δ i W ℓ h I ∖ { i } ℓ − 1 , (107) where h ∅ ℓ = h ℓ h_\varnothing^\ell=h^\ell h ∅ ℓ = h ℓ . For ∣ I ∣ = k ≤ 4 |I|=k\le4 ∣ I ∣ = k ≤ 4 ,
∥ z I ℓ ∥ ≤ Z ℓ , k , ∥ h I ℓ ∥ ≤ H ℓ , k . (108) \lVert z_I^\ell\rVert\le Z_{\ell,k},\qquad\lVert h_I^\ell\rVert\le H_{\ell,k}. \tag*{(108)} ∥ z I ℓ ∥ ≤ Z ℓ , k , ∥ h I ℓ ∥ ≤ H ℓ , k . (108)
Proof. Because W ℓ W^\ell W ℓ is affine in the parameter, at most one derivative can fall on it. Applying the product rule to m ℓ − 1 − 1 / 2 W ℓ h ℓ − 1 m_{\ell-1}^{-1/2}W^\ell h^{\ell-1} m ℓ − 1 − 1/2 W ℓ h ℓ − 1 gives (107) exactly.
For the norm bound, first note that ∥ Δ i W ℓ ∥ o p ≤ ∥ Δ i W ℓ ∥ F ≤ 1 \lVert\Delta_iW^\ell\rVert_{\mathrm{op}}\le\lVert\Delta_iW^\ell\rVert_{\mathrm{F}}\le1 ∥ Δ i W ℓ ∥ op ≤ ∥ Δ i W ℓ ∥ F ≤ 1 . If I ∖ { i } = ∅ I\setminus\{i\}=\varnothing I ∖ { i } = ∅ , then
∥ h ℓ − 1 ∥ m ℓ − 1 ≤ q ℓ − 1 ; \frac{\lVert h^{\ell-1}\rVert}{\sqrt{m_{\ell-1}}}\le q_{\ell-1}; m ℓ − 1 ∥ h ℓ − 1 ∥ ≤ q ℓ − 1 ;
this follows from the input bound when ℓ = 1 \ell=1 ℓ = 1 and from bounded activation coordinates when ℓ ≥ 2 \ell\ge2 ℓ ≥ 2 . If I ∖ { i } I\setminus\{i\} I ∖ { i } is nonempty, induction gives ∥ h I ∖ { i } ℓ − 1 ∥ ≤ H ℓ − 1 , k − 1 \lVert h_{I\setminus\{i\}}^{\ell-1}\rVert\le H_{\ell-1,k-1} ∥ h I ∖ { i } ℓ − 1 ∥ ≤ H ℓ − 1 , k − 1 , and division by m ℓ − 1 ≥ 1 \sqrt{m_{\ell-1}}\ge1 m ℓ − 1 ≥ 1 can only reduce the bound. Together with Lemma C.1 , this proves ∥ z I ℓ ∥ ≤ Z ℓ , k \lVert z_I^\ell\rVert\le Z_{\ell,k} ∥ z I ℓ ∥ ≤ Z ℓ , k .
The coordinatewise Faà di Bruno formula is
h I ℓ = ∑ P ∈ P ( I ) σ ℓ ( ∣ P ∣ ) ( z ℓ ) ∘ ∏ B ∈ P z B ℓ . (109) h_I^\ell=\sum_{P\in\mathcal{P}(I)}\sigma_\ell^{(|P|)}(z^\ell)\circ\prod_{B\in P}z_B^\ell. \tag*{(109)} h I ℓ = P ∈ P ( I ) ∑ σ ℓ ( ∣ P ∣ ) ( z ℓ ) ∘ B ∈ P ∏ z B ℓ . (109)
where P ( I ) \mathcal{P}(I) P ( I ) is the set of partitions of I I I and products are coordinatewise. For vectors v 1 , … , v j v_1,\ldots,v_j v 1 , … , v j ,
∥ v 1 ⊙ ⋯ ⊙ v j ∥ 2 ≤ ∏ i = 1 j ∥ v i ∥ 2 . (110) \lVert v_1\odot\cdots\odot v_j\rVert_2\le\prod_{i=1}^{j}\lVert v_i\rVert_2. \tag*{(110)} ∥ v 1 ⊙ ⋯ ⊙ v j ∥ 2 ≤ i = 1 ∏ j ∥ v i ∥ 2 . (110)
The partitions of sets of sizes one through four have block-size multiplicities 1; 1, 1; 1, 3, 1; and 1, 4, 3, 6, 1, respectively. Substituting the Z Z Z bounds into (109) gives exactly (103) –(106) . Induction over ℓ \ell ℓ proves (108) .
Theorem C.4 (Exact deep derivative bounds). Under Section C.1, define
C k , L : = 2 ( 1 − k ) / 2 H L , k , k = 1 , 2 , 3 , 4. (111) C_{k,L}:=2^{(1-k)/2}H_{L,k},\qquad k=1,2,3,4. \tag*{(111)} C k , L := 2 ( 1 − k ) /2 H L , k , k = 1 , 2 , 3 , 4. (111) Then, uniformly over all admissible hidden widths, points of B ‾ A \overline{B}_A B A , and state-action pairs,
∥ D α k F m , α ( s , a ) ∥ o p ≤ C k , L , k = 1 , 2 , 3 , 4. (112) \left\lVert D_\alpha^kF_{m,\alpha}(s,a)\right\rVert_{\mathrm{op}}\le C_{k,L},\qquad k=1,2,3,4. \tag*{(112)} D α k F m , α ( s , a ) op ≤ C k , L , k = 1 , 2 , 3 , 4. (112) Consequently,
∥ Z m , α ( s , a ) ∥ ≤ C 1 , L , (113) \lVert Z_{m,\alpha}(s,a)\rVert\le C_{1,L}, \tag*{(113)} ∥ Z m , α ( s , a )∥ ≤ C 1 , L , (113) ∥ Z m , α ( s , a ) − Z m , β ( s , a ) ∥ ≤ C 2 , L ∥ α − β ∥ , (114) \lVert Z_{m,\alpha}(s,a)-Z_{m,\beta}(s,a)\rVert\le C_{2,L}\lVert\alpha-\beta\rVert, \tag*{(114)} ∥ Z m , α ( s , a ) − Z m , β ( s , a )∥ ≤ C 2 , L ∥ α − β ∥ , (114) ∣ F m , α ( s , a ) − F m , β ( s , a ) ∣ ≤ C 1 , L ∥ α − β ∥ . (115) \lvert F_{m,\alpha}(s,a)-F_{m,\beta}(s,a)\rvert\le C_{1,L}\lVert\alpha-\beta\rVert. \tag*{(115)} ∣ F m , α ( s , a ) − F m , β ( s , a )∣ ≤ C 1 , L ∥ α − β ∥ . (115) Proof. Apply Lemma C.3 at either branch to unit product-norm directions. Since ∥ b ∥ / m L = 1 \lVert b\rVert/\sqrt{m_L}=1 ∥ b ∥ / m L = 1 ,
∣ D ϑ k G [ Δ 1 ϑ , … , Δ k ϑ ] ∣ ≤ H L , k . \left\lvert D_\vartheta^kG[\Delta_1\vartheta,\ldots,\Delta_k\vartheta]\right\rvert\le H_{L,k}. D ϑ k G [ Δ 1 ϑ , … , Δ k ϑ ] ≤ H L , k . For unit u i ∈ R r u_i\in\mathbb{R}^r u i ∈ R r , the isometry gives ∥ B m u i ∥ p a r = 1 \lVert B_mu_i\rVert_{\mathrm{par}}=1 ∥ B m u i ∥ par = 1 . Now use (99) ; the sum of the two branch bounds is 2 ⋅ 2 − ( k + 1 ) / 2 H L , k = C k , L 2\cdot2^{-(k+1)/2}H_{L,k}=C_{k,L} 2 ⋅ 2 − ( k + 1 ) /2 H L , k = C k , L . The first consequence is the case k = 1 k=1 k = 1 . The remaining two follow by integrating the second and first derivatives, respectively, along the segment from β \beta β to α \alpha α .
Corollary C.5 (Uniform energy and density envelope). For every point of the smaller geometry chart Θ R = B ‾ R \Theta_R=\overline{B}_R Θ R = B R ,
∣ F m , α ( s , a ) ∣ ≤ C 1 , L R , (116) \lvert F_{m,\alpha}(s,a)\rvert\le C_{1,L}R, \tag*{(116)} ∣ F m , α ( s , a )∣ ≤ C 1 , L R , (116) e − 2 C 1 , L R ≤ d π m , α ( ⋅ ∣ s ) d π r e f ( ⋅ ∣ s ) ( a ) ≤ e 2 C 1 , L R , (117) e^{-2C_{1,L}R}\le\frac{d\pi_{m,\alpha}(\cdot\mid s)}{d\pi_{\mathrm{ref}}(\cdot\mid s)}(a)\le e^{2C_{1,L}R}, \tag*{(117)} e − 2 C 1 , L R ≤ d π ref ( ⋅ ∣ s ) d π m , α ( ⋅ ∣ s ) ( a ) ≤ e 2 C 1 , L R , (117) ∥ log d π m , α ( ⋅ ∣ s ) d π r e f ( ⋅ ∣ s ) ∥ ∞ , a ≤ 2 C 1 , L R . (118) \left\lVert\log\frac{d\pi_{m,\alpha}(\cdot\mid s)}{d\pi_{\mathrm{ref}}(\cdot\mid s)}\right\rVert_{\infty,a}\le2C_{1,L}R. \tag*{(118)} log d π ref ( ⋅ ∣ s ) d π m , α ( ⋅ ∣ s ) ∞ , a ≤ 2 C 1 , L R . (118) Proof. Use F m , 0 = 0 F_{m,0}=0 F m , 0 = 0 and (115) . The log normalizer lies between the minimum and maximum of the energy, so both the numerator and normalizer comparisons contribute at most C 1 , L R C_{1,L}R C 1 , L R .
Log-partition and log-policy derivatives For nonnegative numbers c 1 , … , c k c_1,\ldots,c_k c 1 , … , c k , define the complete exponential Bell polynomial
B k ( c 1 , … , c k ) : = ∑ n 1 , … , n k ≥ 0 ∑ j = 1 k j n j = k k ! ∏ j = 1 k n j ! ( j ! ) n j ∏ j = 1 k c j n j . (119) B_k(c_1,\ldots,c_k):=\sum_{\substack{n_1,\ldots,n_k\ge0\\ \sum_{j=1}^{k}jn_j=k}}\frac{k!}{\prod_{j=1}^{k}n_j!(j!)^{n_j}}\prod_{j=1}^{k}c_j^{n_j}. \tag*{(119)} B k ( c 1 , … , c k ) := n 1 , … , n k ≥ 0 ∑ j = 1 k j n j = k ∑ ∏ j = 1 k n j ! ( j ! ) n j k ! j = 1 ∏ k c j n j . (119)
Set $B_0:=1.
Lemma C.6 (Log-partition derivatives with explicit envelopes). Fix a state and a C 4 C^{4} C 4 scalar path t ↦ F t ( a ) t \mapsto F_t(a) t ↦ F t ( a ) . Suppose ∥ ∂ t j F t ∥ ∞ ≤ c j \lVert\partial_t^j F_t\rVert_{\infty} \le c_j ∥ ∂ t j F t ∥ ∞ ≤ c j for j = 1 , 2 , 3 , 4 j=1,2,3,4 j = 1 , 2 , 3 , 4 on an interval. Let
Z ( t ) = ∫ e F t ( a ) π r e f ( d a ∣ s ) , A ( t ) = log Z ( t ) . Z(t)=\int e^{F_t(a)}\pi_{\mathrm{ref}}(da\mid s), \qquad A(t)=\log Z(t). Z ( t ) = ∫ e F t ( a ) π ref ( d a ∣ s ) , A ( t ) = log Z ( t ) . Define b j = B j ( c 1 , … , c j ) b_j=B_j(c_1,\ldots,c_j) b j = B j ( c 1 , … , c j ) and recursively
a 1 : = b 1 , a k : = b k + ∑ j = 1 k − 1 ( k − 1 j − 1 ) a j b k − j , k = 2 , 3 , 4. (120) a_1:=b_1,\qquad a_k:=b_k+\sum_{j=1}^{k-1}\binom{k-1}{j-1}a_jb_{k-j},\qquad k=2,3,4. \tag*{(120)} a 1 := b 1 , a k := b k + j = 1 ∑ k − 1 ( j − 1 k − 1 ) a j b k − j , k = 2 , 3 , 4. (120) Then ∣ A ( k ) ( t ) ∣ ≤ a k \lvert A^{(k)}(t)\rvert\le a_k ∣ A ( k ) ( t )∣ ≤ a k for k = 1 , 2 , 3 , 4 k=1,2,3,4 k = 1 , 2 , 3 , 4 .
Proof. Faà di Bruno’s formula for e F t ( a ) e^{F_t(a)} e F t ( a ) gives
∂ t k e F t ( a ) = e F t ( a ) B k ( ∂ t F t ( a ) , … , ∂ t k F t ( a ) ) . \partial_t^k e^{F_t(a)}=e^{F_t(a)}B_k(\partial_tF_t(a),\ldots,\partial_t^kF_t(a)). ∂ t k e F t ( a ) = e F t ( a ) B k ( ∂ t F t ( a ) , … , ∂ t k F t ( a )) . After integration and division by Z ( t ) Z(t) Z ( t ) , ∣ Z ( k ) ( t ) / Z ( t ) ∣ ≤ b k \lvert Z^{(k)}(t)/Z(t)\rvert\le b_k ∣ Z ( k ) ( t ) / Z ( t )∣ ≤ b k . Differentiating the identity Z ′ = A ′ Z Z'=A'Z Z ′ = A ′ Z k − 1 k-1 k − 1 additional times gives
Z ( k ) Z = ∑ j = 1 k ( k − 1 j − 1 ) A ( j ) Z ( k − j ) Z . (121) \frac{Z^{(k)}}{Z}=\sum_{j=1}^{k}\binom{k-1}{j-1}A^{(j)}\frac{Z^{(k-j)}}{Z}. \tag*{(121)} Z Z ( k ) = j = 1 ∑ k ( j − 1 k − 1 ) A ( j ) Z Z ( k − j ) . (121) Isolate the j = k j=k j = k term and apply induction, the triangle inequality, and the bounds on Z ( q ) / Z Z^{(q)}/Z Z ( q ) / Z . This yields exactly (120) . □
Lemma C.7 (Polarization bound). Let T T T be a continuous symmetric k k k -linear form on a real Hilbert space and let p T ( h ) = T [ h , … , h ] p_T(h)=T[h,\ldots,h] p T ( h ) = T [ h , … , h ] . Then
∥ T ∥ o p ≤ k k k ! sup ∥ h ∥ ≤ 1 ∣ p T ( h ) ∣ . (122) \lVert T\rVert_{\mathrm{op}}\le\frac{k^k}{k!}\sup_{\lVert h\rVert\le1}\lvert p_T(h)\rvert. \tag*{(122)} ∥ T ∥ op ≤ k ! k k ∥ h ∥ ≤ 1 sup ∣ p T ( h )∣ . (122) Proof. The real polarization identity is
T [ h 1 , … , h k ] = 1 2 k k ! ∑ ϵ ∈ { − 1 , 1 } k ( ∏ i = 1 k ϵ i ) p T ( ∑ i = 1 k ϵ i h i ) . T[h_1,\ldots,h_k]=\frac{1}{2^k k!}\sum_{\epsilon\in\{-1,1\}^k}\left(\prod_{i=1}^{k}\epsilon_i\right)p_T\left(\sum_{i=1}^{k}\epsilon_i h_i\right). T [ h 1 , … , h k ] = 2 k k ! 1 ϵ ∈ { − 1 , 1 } k ∑ ( i = 1 ∏ k ϵ i ) p T ( i = 1 ∑ k ϵ i h i ) . For ∥ h i ∥ ≤ 1 \lVert h_i\rVert\le1 ∥ h i ∥ ≤ 1 , the argument of p T p_T p T has norm at most k k k . Homogeneity gives ∣ p T ( ∑ i ϵ i h i ) ∣ ≤ k k sup ∥ h ∥ ≤ 1 ∣ p T ( h ) ∣ \lvert p_T(\sum_i\epsilon_i h_i)\rvert\le k^k\sup_{\lVert h\rVert\le1}\lvert p_T(h)\rvert ∣ p T ( ∑ i ϵ i h i )∣ ≤ k k sup ∥ h ∥ ≤ 1 ∣ p T ( h )∣ . Summing 2 k 2^k 2 k terms proves the claim. □
Lemma C.8 (Width-uniform deep log-policy derivatives). For k = 1 , 2 , 3 , 4 k=1,2,3,4 k = 1 , 2 , 3 , 4 , let b k , L b_{k,L} b k , L and a k , L a_{k,L} a k , L be obtained from (119) –(120) with c j = C j , L c_j=C_{j,L} c j = C j , L , and define
M l o g , k , L : = k k k ! ( C k , L + a k , L ) . (123) M_{\mathrm{log},k,L}:=\frac{k^k}{k!}(C_{k,L}+a_{k,L}). \tag*{(123)} M log , k , L := k ! k k ( C k , L + a k , L ) . (123) Then, uniformly over widths, states, actions, and chart points,
∥ D α k log π m , α ( a ∣ s ) ∥ o p ≤ M l o g , k , L , k = 1 , 2 , 3 , 4. (124) \lVert D_\alpha^k\log\pi_{m,\alpha}(a\mid s)\rVert_{\mathrm{op}}\le M_{\mathrm{log},k,L},\qquad k=1,2,3,4. \tag*{(124)} ∥ D α k log π m , α ( a ∣ s ) ∥ op ≤ M log , k , L , k = 1 , 2 , 3 , 4. (124)
For every fixed state law ν \nu ν , the Fisher matrix is Lipschitz:
∥ F m , ν ( α ) − F m , ν ( β ) ∥ o p ≤ L F , L ∥ α − β ∥ , (125) \left\|\mathcal{F}_{m,\nu}(\alpha)-\mathcal{F}_{m,\nu}(\beta)\right\|_{\mathrm{op}} \le L_{F,L}\left\|\alpha-\beta\right\|, \tag*{(125)} ∥ F m , ν ( α ) − F m , ν ( β ) ∥ op ≤ L F , L ∥ α − β ∥ , (125)
where
L F , L : = 2 M log , 1 , L M log , 2 , L + M log , 1 , L 3 . (126) L_{F,L} := 2M_{\log,1,L}M_{\log,2,L}+M_{\log,1,L}^{3}. \tag*{(126)} L F , L := 2 M l o g , 1 , L M l o g , 2 , L + M l o g , 1 , L 3 . (126)
Consequently, on the Fisher-conditioned region,
∥ F m , ν ( α ) − 1 − F m , ν ( β ) − 1 ∥ o p ≤ L P , L ∥ α − β ∥ , L P , L : = L F , L λ F , L 2 . (127) \left\|\mathcal{F}_{m,\nu}(\alpha)^{-1}-\mathcal{F}_{m,\nu}(\beta)^{-1}\right\|_{\mathrm{op}} \le L_{P,L}\left\|\alpha-\beta\right\|,\qquad L_{P,L}:=\frac{L_{F,L}}{\lambda_{F,L}^{2}}. \tag*{(127)} F m , ν ( α ) − 1 − F m , ν ( β ) − 1 op ≤ L P , L ∥ α − β ∥ , L P , L := λ F , L 2 L F , L . (127)
Proof. Fix a unit direction h h h and apply Lemma C.6 to F t ( a ) = F m , α + t h ( s , a ) F_t(a)=\mathcal{F}_{m,\alpha+th}(s,a) F t ( a ) = F m , α + t h ( s , a ) . The deep derivative theorem gives ∥ ∂ t j F t ∥ ∞ ≤ C j , L \left\|\partial_t^jF_t\right\|_\infty\le C_{j,L} ∂ t j F t ∞ ≤ C j , L . Since log π α = F α − A s ( α ) \log\pi_\alpha=F_\alpha-A_s(\alpha) log π α = F α − A s ( α ) , the diagonal k k k th directional derivative has magnitude at most C k , L + a k , L C_{k,L}+a_{k,L} C k , L + a k , L . The derivative is a symmetric k k k -linear form, so Lemma C.7 gives (124) .
Let S α = ∇ α log π m , α S_\alpha=\nabla_\alpha\log\pi_{m,\alpha} S α = ∇ α log π m , α . For a unit direction h h h , differentiation of a policy expectation gives
D α E ν , π α [ S α S α T ] [ h ] = E ν , π α [ ( D S α [ h ] ) S α T + S α ( D S α [ h ] ) T + S α S α T ⟨ S α , h ⟩ ] . D_\alpha\mathbb{E}_{\nu,\pi_\alpha}[S_\alpha S_\alpha^{\mathsf{T}}][h]
=
\mathbb{E}_{\nu,\pi_\alpha}\left[(DS_\alpha[h])S_\alpha^{\mathsf{T}}+S_\alpha(DS_\alpha[h])^{\mathsf{T}}+S_\alpha S_\alpha^{\mathsf{T}}\langle S_\alpha,h\rangle\right]. D α E ν , π α [ S α S α T ] [ h ] = E ν , π α [ ( D S α [ h ]) S α T + S α ( D S α [ h ] ) T + S α S α T ⟨ S α , h ⟩ ] .
The operator norm is at most L F , L L_{F,L} L F , L . Integrating this derivative along the segment from β \beta β to $\alpha proves (125) . Finally use Lemma C.49 and the Fisher lower bound.
Reference-KL envelopes. Use the fixed ambient actor ball. Put χ γ = γ / ( 1 − γ ) \chi_\gamma=\gamma/(1-\gamma) χ γ = γ / ( 1 − γ ) and v 0 = 2 C 1 A v_0=2C_1A v 0 = 2 C 1 A . Since F 0 = 0 F_0=0 F 0 = 0 , ∣ F α ∣ ≤ C 1 A \lvert F_\alpha\rvert\le C_1A ∣ F α ∣ ≤ C 1 A and ∣ log ( π α / π r e f ) ∣ ≤ v 0 \left\lvert\log(\pi_\alpha/\pi_{\mathrm{ref}})\right\rvert\le v_0 ∣ log ( π α / π ref ) ∣ ≤ v 0 .
Let S 1 , S 2 , S 3 S_1,S_2,S_3 S 1 , S 2 , S 3 be uniform bounds on the first three derivatives of log π α \log\pi_\alpha log π α . They follow from differentiating the finite softmax and the energy bounds. One conservative choice is
S 1 = 2 C 1 , S 2 = 2 C 2 + C 1 2 , S 3 = 2 C 3 + 6 C 1 C 2 + 8 C 1 3 . S_1=2C_1,\qquad S_2=2C_2+C_1^2,\qquad S_3=2C_3+6C_1C_2+8C_1^3. S 1 = 2 C 1 , S 2 = 2 C 2 + C 1 2 , S 3 = 2 C 3 + 6 C 1 C 2 + 8 C 1 3 .
Define
P 1 = S 1 , P 2 = S 2 + S 1 2 , P 3 = S 3 + 3 S 1 S 2 + S 1 3 , D 1 = χ γ P 1 , D 2 = χ γ P 2 + 2 χ γ 2 P 1 2 , D 3 = χ γ P 3 + 6 χ γ 2 P 1 P 2 + 6 χ γ 3 P 1 3 . (128) \begin{aligned}
P_1&=S_1, & P_2&=S_2+S_1^2, & P_3&=S_3+3S_1S_2+S_1^3,\\
D_1&=\chi_\gamma P_1, & D_2&=\chi_\gamma P_2+2\chi_\gamma^2P_1^2, & D_3&=\chi_\gamma P_3+6\chi_\gamma^2P_1P_2+6\chi_\gamma^3P_1^3. \tag*{(128)}
\end{aligned} P 1 D 1 = S 1 , = χ γ P 1 , P 2 D 2 = S 2 + S 1 2 , = χ γ P 2 + 2 χ γ 2 P 1 2 , P 3 D 3 = S 3 + 3 S 1 S 2 + S 1 3 , = χ γ P 3 + 6 χ γ 2 P 1 P 2 + 6 χ γ 3 P 1 3 . (128)
Here the norm for signed measures is the dual of the supremum norm, without a factor 1 / 2 1/2 1/2 .
Lemma C.9 (Occupancy and KL envelopes). The total-variation norms of the first three policy derivatives are at most P i P_i P i , and those of the normalized state occupancy derivatives are at most D i D_i D i . Let
q 0 = v 0 , q 1 = S 1 + P 1 v 0 , q_0=v_0,\qquad q_1=S_1+P_1v_0, q 0 = v 0 , q 1 = S 1 + P 1 v 0 , q 2 = S 2 + 2 P 1 S 1 + P 2 v 0 , q 3 = S 3 + 3 P 1 S 2 + 3 P 2 S 1 + P 3 v 0 . q_2=S_2+2P_1S_1+P_2v_0,\qquad q_3=S_3+3P_1S_2+3P_2S_1+P_3v_0. q 2 = S 2 + 2 P 1 S 1 + P 2 v 0 , q 3 = S 3 + 3 P 1 S 2 + 3 P 2 S 1 + P 3 v 0 . Then
∥ D α 3 K ∥ ≤ L K : = q 3 + 3 D 1 q 2 + 3 D 2 q 1 + D 3 q 0 . (129) \left\|D_\alpha^3\mathcal{K}\right\|\le L_K:=q_3+3D_1q_2+3D_2q_1+D_3q_0. \tag*{(129)} D α 3 K ≤ L K := q 3 + 3 D 1 q 2 + 3 D 2 q 1 + D 3 q 0 . (129) In particular the KL Hessian is L K L_K L K -Lipschitz on actor segments in the ambient ball. All these constants are independent of τ \tau τ , width, and target accuracy.
Proof. Differentiate π = e log π \pi=e^{\log\pi} π = e l o g π and sum absolute values over actions to obtain P i P_i P i . The actor-induced Markov kernel T α T_\alpha T α is a contraction in the supremum norm, and ∥ D i T α ∥ ≤ P i \lVert D^iT_\alpha\rVert\le P_i ∥ D i T α ∥ ≤ P i . The normalized occupancy is
d α = ( 1 − γ ) ρ ( I − γ T α ) − 1 . d_\alpha=(1-\gamma)\rho(I-\gamma T_\alpha)^{-1}. d α = ( 1 − γ ) ρ ( I − γ T α ) − 1 .
Differentiate its resolvent identity. The resolvent norm is at most ( 1 − γ ) − 1 (1-\gamma)^{-1} ( 1 − γ ) − 1 , while ∥ d α ∥ = 1 \lVert d_\alpha\rVert=1 ∥ d α ∥ = 1 . Product differentiation gives exactly the bounds D i D_i D i in (128); the coefficients count the ordered product-rule terms. The statewise KL is k α ( s ) = ∑ a π α ( a ∣ s ) log ( π α ( a ∣ s ) π r e f ( a ∣ s ) ) k_\alpha(s)=\sum_a\pi_\alpha(a\mid s)\log\left(\frac{\pi_\alpha(a\mid s)}{\pi_{\mathrm{ref}}(a\mid s)}\right) k α ( s ) = ∑ a π α ( a ∣ s ) log ( π ref ( a ∣ s ) π α ( a ∣ s ) ) . Product differentiation bounds its derivatives by q i q_i q i . Finally differentiate K = d α k α \mathcal{K}=d_\alpha k_\alpha K = d α k α three times. The four groups of terms have bounds q 3 , 3 D 1 q 2 , 3 D 2 q 1 , D 3 q 0 q_3,3D_1q_2,3D_2q_1,D_3q_0 q 3 , 3 D 1 q 2 , 3 D 2 q 1 , D 3 q 0 . All differentiations are justified by the finite action sums, derivative envelopes, and discounted resolvent. The argument does not require transition likelihood ratios for derivatives in α \alpha α .
Lemma C.10 (Exact KL Hessian at initialization). With a 0 = ( 1 − γ ) λ e f f , L a_0=(1-\gamma)\lambda_{\mathrm{eff},L} a 0 = ( 1 − γ ) λ eff , L , on the fixed actor chart,
K ( 0 ) = 0 , ∇ α K ( 0 ) = 0 , ∇ α α 2 K ( 0 ) = F x , m ( 0 ) ⪰ a 0 I , (130) \mathcal{K}(0)=0,\quad\nabla_\alpha\mathcal{K}(0)=0,\quad\nabla^2_{\alpha\alpha}\mathcal{K}(0)=\mathcal{F}_{x,m}(0)\succeq a_0I, \tag*{(130)} K ( 0 ) = 0 , ∇ α K ( 0 ) = 0 , ∇ α α 2 K ( 0 ) = F x , m ( 0 ) ⪰ a 0 I , (130) where F x , m ( 0 ) \mathcal{F}_{x,m}(0) F x , m ( 0 ) is the occupancy-weighted policy Fisher matrix.
Proof. For every state s s s , k 0 ( s ) = 0 k_0(s)=0 k 0 ( s ) = 0 and D k 0 ( s ) = 0 Dk_0(s)=0 D k 0 ( s ) = 0 . At zero the score is Z 0 ( s , a ) − E π r e f Z 0 ( s , ⋅ ) Z_0(s,a)-\mathbb{E}_{\pi_{\mathrm{ref}}}Z_0(s,\cdot) Z 0 ( s , a ) − E π ref Z 0 ( s , ⋅ ) , so the information identity yields D 2 k 0 ( s ) = Cov π r e f ( Z 0 ( s , ⋅ ) ) D^2k_0(s)=\operatorname{Cov}_{\pi_{\mathrm{ref}}}(Z_0(s,\cdot)) D 2 k 0 ( s ) = Cov π ref ( Z 0 ( s , ⋅ )) . Differentiating K = d α k α \mathcal{K}=d_\alpha k_\alpha K = d α k α twice leaves only d 0 D 2 k 0 d_0D^2k_0 d 0 D 2 k 0 because the terms multiplying k 0 k_0 k 0 and D k 0 Dk_0 D k 0 vanish statewise. Finally d 0 ⪰ ( 1 − γ ) ρ d_0\succeq(1-\gamma)\rho d 0 ⪰ ( 1 − γ ) ρ and Assumption 4.2 give the lower bound. Normalization is essential: the bound contains ( 1 − γ ) (1-\gamma) ( 1 − γ ) , not its reciprocal.
Lemma C.11 (Persistence of the restricted initialization covariance). Under Assumption 4.2 and Theorem C.4 , for every α ∈ Θ R \alpha\in\Theta_R α ∈ Θ R ,
E s ∼ ρ Cov a ∼ π r e f ( ⋅ ∣ s ) ( Z m , α ( s , a ) ) ⪰ ( λ e f f , L − C 2 , L ∥ α ∥ ) + 2 I r . (131) \mathbb{E}_{s\sim\rho}\operatorname{Cov}_{a\sim\pi_{\mathrm{ref}}(\cdot\mid s)}\left(Z_{m,\alpha}(s,a)\right)\succeq\left(\sqrt{\lambda_{\mathrm{eff},L}}-C_{2,L}\lVert\alpha\rVert\right)_+^2I_r. \tag*{(131)} E s ∼ ρ Cov a ∼ π ref ( ⋅ ∣ s ) ( Z m , α ( s , a ) ) ⪰ ( λ eff , L − C 2 , L ∥ α ∥ ) + 2 I r . (131)
Proof. Fix a unit vector v ∈ R r v\in\mathbb{R}^r v ∈ R r and let the joint reference law be s ∼ ρ s\sim\rho s ∼ ρ , a ∼ π r e f ( ⋅ ∣ s ) a\sim\pi_{\mathrm{ref}}(\cdot\mid s) a ∼ π ref ( ⋅ ∣ s ) . Set X = v ⊤ Z m , 0 X=v^\top Z_{m,0} X = v ⊤ Z m , 0 and Y = v ⊤ Z m , α Y=v^\top Z_{m,\alpha} Y = v ⊤ Z m , α . By (114) ,
∥ X − Y ∥ L 2 ≤ C 2 , L ∥ α ∥ . \lVert X-Y\rVert_{L_2}\le C_{2,L}\lVert\alpha\rVert. ∥ X − Y ∥ L 2 ≤ C 2 , L ∥ α ∥ .
Conditional centering is a contraction, so
∥ Y − E [ Y ∣ s ] ∥ L 2 ≥ ∥ X − E [ X ∣ s ] ∥ L 2 − ∥ X − Y ∥ L 2 ≥ λ e f f , L − C 2 , L ∥ α ∥ , \begin{aligned}
\lVert Y-\mathbb{E}[Y\mid s]\rVert_{L_2}\ge\lVert X-\mathbb{E}[X\mid s]\rVert_{L_2}-\lVert X-Y\rVert_{L_2} \\
&\ge\sqrt{\lambda_{\mathrm{eff},L}}-C_{2,L}\lVert\alpha\rVert,
\end{aligned} ∥ Y − E [ Y ∣ s ] ∥ L 2 ≥ ∥ X − E [ X ∣ s ] ∥ L 2 − ∥ X − Y ∥ L 2 ≥ λ eff , L − C 2 , L ∥ α ∥ ,
where the final inequality is (12) . Since the left-hand side is nonnegative, take the positive part of the right-hand side before squaring, and then take the infimum over unit v v v . This proves the claim.
Lemma C.12 (Persistent discounted and common-state Fisher excitation). For the radius in (33) , for every α ∈ Θ R \alpha\in\Theta_R α ∈ Θ R and every state law ν ⪰ ( 1 − γ ) ρ \nu\succeq(1-\gamma)\rho ν ⪰ ( 1 − γ ) ρ ,
λ ‾ F , L I r ⪯ F m , ν ( α ) ⪯ λ ‾ F , L I r , (132) \underline{\lambda}_{F,L}I_r\preceq\mathcal{F}_{m,\nu}(\alpha)\preceq\overline{\lambda}_{F,L}I_r, \tag*{(132)} λ F , L I r ⪯ F m , ν ( α ) ⪯ λ F , L I r , (132) where
λ ‾ F , L = ( 1 − γ ) e − 2 C 1 , L R λ e f f , L / 4 , λ ‾ F , L = 4 C 1 , L 2 . (133) \underline{\lambda}_{F,L}=(1-\gamma)e^{-2C_{1,L}R\lambda_{\mathrm{eff},L}/4},\qquad\overline{\lambda}_{F,L}=4C_{1,L}^2. \tag*{(133)} λ F , L = ( 1 − γ ) e − 2 C 1 , L R λ eff , L /4 , λ F , L = 4 C 1 , L 2 . (133) In particular, this holds for each discounted occupancy and for every common mixture law (154) .
Proof. The upper bound follows from ∥ Z m , α ∥ ≤ 2 C 1 , L \lVert Z_{m,\alpha}\rVert\le2C_{1,L} ∥ Z m , α ∥ ≤ 2 C 1 , L . For the lower bound, Corollary C.5 gives, conditionally on each state,
π m , α ( ⋅ ∣ s ) ⪰ e − 2 C 1 , L R π r e f ( ⋅ ∣ s ) . \pi_{m,\alpha}(\cdot\mid s) \succeq e^{-2C_{1,L}R}\pi_{\mathrm{ref}}(\cdot\mid s). π m , α ( ⋅ ∣ s ) ⪰ e − 2 C 1 , L R π ref ( ⋅ ∣ s ) .
Apply Lemma C.47 conditionally in the action variable to Z m , α Z_{m,\alpha} Z m , α , integrate with respect to ν ⪰ ( 1 − γ ) ρ \nu\succeq(1-\gamma)\rho ν ⪰ ( 1 − γ ) ρ , and then use Lemma C.11 . For every unit v v v ,
v T F m , ν ( α ) v ≥ ( 1 − γ ) e − 2 C 1 , L R E s ∼ ρ Var a ∼ π r e f ( v T Z m , α ( s , a ) ) ≥ λ F , L . v^{\mathsf{T}}\mathcal{F}_{m,\nu}(\alpha)v \ge(1-\gamma)e^{-2C_{1,L}R}\mathbb{E}_{s\sim\rho}\operatorname{Var}_{a\sim\pi_{\mathrm{ref}}}\left(v^{\mathsf{T}}Z_{m,\alpha}(s,a)\right) \ge\lambda_{F,L}. v T F m , ν ( α ) v ≥ ( 1 − γ ) e − 2 C 1 , L R E s ∼ ρ Var a ∼ π ref ( v T Z m , α ( s , a ) ) ≥ λ F , L .
The common mixture dominates ( 1 − γ ) ρ (1-\gamma)\rho ( 1 − γ ) ρ because each of its two components does.
Optional one-time initialization certificate.
Proposition C.13 (Independent pilot certification). Fix a realized network and chart with ∥ Z m , 0 ∥ ≤ G Z : = C 1 , L \lVert Z_{m,0}\rVert\le G_Z := C_{1,L} ∥ Z m , 0 ∥ ≤ G Z := C 1 , L . Draw independent pilot triples
s i ∼ ρ , a i , a i ′ ∼ i . i . d . π r e f ( ⋅ ∣ s i ) , i = 1 , … , N , s_i \sim\rho,\qquad a_i,a_i' \mathrel{\overset{\mathrm{i.i.d.}}{\sim}} \pi_{\mathrm{ref}}(\cdot\mid s_i),\qquad i=1,\ldots,N, s i ∼ ρ , a i , a i ′ ∼ i.i.d. π ref ( ⋅ ∣ s i ) , i = 1 , … , N , and define
G ^ N = 1 2 N ∑ i = 1 N ( Z i − Z i ′ ) ( Z i − Z i ′ ) T , (134) \widehat{\mathcal{G}}_N=\frac{1}{2N}\sum_{i=1}^{N}(Z_i-Z_i')(Z_i-Z_i')^{\mathsf{T}}, \tag*{(134)} G N = 2 N 1 i = 1 ∑ N ( Z i − Z i ′ ) ( Z i − Z i ′ ) T , (134) where Z i = Z m , 0 ( s i , a i ) Z_i=Z_{m,0}(s_i,a_i) Z i = Z m , 0 ( s i , a i ) and Z i ′ = Z m , 0 ( s i , a i ′ ) Z_i'=Z_{m,0}(s_i,a_i') Z i ′ = Z m , 0 ( s i , a i ′ ) . Then
E [ G ^ N ∣ m , i n i t i a l i z a t i o n ] = G m , 0 . (135) \mathbb{E}[\widehat{\mathcal{G}}_N\mid m,\mathrm{initialization}]=\mathcal{G}_{m,0}. \tag*{(135)} E [ G N ∣ m , initialization ] = G m , 0 . (135) Moreover, there is a universal constant C B C_{\mathrm{B}} C B such that, with probability at least 1 − δ 1-\delta 1 − δ ,
∥ G ^ N − G m , 0 ∥ o p ≤ C B G Z 2 [ log ( 2 r / δ ) N + log ( 2 r / δ ) N ] = : ϵ N . (136) \left\lVert\widehat{\mathcal{G}}_N-\mathcal{G}_{m,0}\right\rVert_{\mathrm{op}}
\le C_{\mathrm{B}}G_Z^2\left[\sqrt{\frac{\log(2r/\delta)}{N}}+\frac{\log(2r/\delta)}{N}\right]=:\epsilon_N. \tag*{(136)} G N − G m , 0 op ≤ C B G Z 2 [ N log ( 2 r / δ ) + N log ( 2 r / δ ) ] =: ϵ N . (136) Consequently,
λ min ( G ^ N ) ≥ λ c e r t + ϵ N (137) \lambda_{\min}(\widehat{\mathcal{G}}_N)\ge\lambda_{\mathrm{cert}}+\epsilon_N \tag*{(137)} λ m i n ( G N ) ≥ λ cert + ϵ N (137) *certifies G m , 0 ⪰ λ c e r t I r \mathcal{G}_{m,0}\succeq\lambda_{\mathrm{cert}}I_r G m , 0 ⪰ λ cert I r with probability at least 1 − δ 1-\delta 1 − δ .
Proof. Conditionally on s i s_i s i , the two actions are independent and identically distributed, so
1 2 E [ ( Z i − Z i ′ ) ( Z i − Z i ′ ) T ∣ s i ] = Cov a ∼ π r e f ( ⋅ ∣ s i ) ( Z m , 0 ( s i , a ) ) . \frac{1}{2}\mathbb{E}\left[(Z_i-Z_i')(Z_i-Z_i')^{\mathsf{T}}\mid s_i\right]=\operatorname{Cov}_{a\sim\pi_{\mathrm{ref}}(\cdot\mid s_i)}(Z_{m,0}(s_i,a)). 2 1 E [ ( Z i − Z i ′ ) ( Z i − Z i ′ ) T ∣ s i ] = Cov a ∼ π ref ( ⋅ ∣ s i ) ( Z m , 0 ( s i , a )) .
Averaging over s i s_i s i proves (135) . Each summand in (134) is positive semidefinite and has operator norm at most 2 G Z 2 2G_Z^2 2 G Z 2 . Self-adjoint matrix Bernstein gives (136) after changing a universal numerical constant. Weyl’s inequality then proves the certification statement.
The chart must be fixed independently of the certification sample. Chart selection from data therefore requires a separate certificate split. The proposition bounds false acceptance by δ \delta δ ; it does not assert that every chart is accepted. For a fixed positive gap and slack, the sample size grows only logarithmically in 1 / δ 1/\delta 1/ δ . Charge any collected reset/pilot data in (76) . A2 can instead be assumed directly. The actor Fisher certificate is distinct from C1’s critic feature Gram gap.
Preference losses and global mixed-derivative bounds
Bradley–Terry losses satisfy explicit global envelopes For two trajectories and a realized binary label Y Y Y , put
Δ x = ∑ t = 0 K − 1 r x ( s t 1 , a t 1 ) − ∑ t = 0 K − 1 r x ( s t 2 , a t 2 ) , L = log ( 1 + e Δ x ) − Y Δ x . (138) \Delta_x = \sum_{t=0}^{K-1} r_x(s_t^1,a_t^1) - \sum_{t=0}^{K-1} r_x(s_t^2,a_t^2), \qquad L = \log(1+e^{\Delta_x}) - Y\Delta_x. \tag*{(138)} Δ x = t = 0 ∑ K − 1 r x ( s t 1 , a t 1 ) − t = 0 ∑ K − 1 r x ( s t 2 , a t 2 ) , L = log ( 1 + e Δ x ) − Y Δ x . (138)
The conditional law of Y Y Y given the trajectories is fixed when differentiating. In particular, it is not a reward-model label distribution whose parameter dependence has been omitted.
Proposition C.14 (Samplewise preference-loss envelopes). If ∣ r x ∣ ≤ B r |r_x| \le B_r ∣ r x ∣ ≤ B r , ∥ ∇ x r x ∥ ≤ G r \|\nabla_x r_x\| \le G_r ∥ ∇ x r x ∥ ≤ G r , and ∥ ∇ x 2 r x ∥ o p ≤ L r \|\nabla_x^2r_x\|_{\mathrm{op}} \le L_r ∥ ∇ x 2 r x ∥ op ≤ L r uniformly, then (138) obeys
0 ≤ L ≤ B L : = log 2 + 2 K B r , ∥ ∇ x L ∥ ≤ G L : = 2 K G r , 0 \le L \le B_L := \log2 + 2KB_r,\qquad\|\nabla_xL\| \le G_L := 2KG_r, 0 ≤ L ≤ B L := log 2 + 2 K B r , ∥ ∇ x L ∥ ≤ G L := 2 K G r , ∥ ∇ x 2 L ∥ o p ≤ L L : = K 2 G r 2 + 2 K L r . (139) \|\nabla_x^2L\|_{\mathrm{op}} \le L_L := K^2G_r^2 + 2KL_r. \tag*{(139)} ∥ ∇ x 2 L ∥ op ≤ L L := K 2 G r 2 + 2 K L r . (139)
These bounds hold for each realized label, and Φ τ ≥ 0 \Phi_\tau\ge0 Φ τ ≥ 0 .
Proof. The corresponding bounds for Δ x \Delta_x Δ x , its gradient, and its Hessian are 2 K B r 2KB_r 2 K B r , 2 K G r 2KG_r 2 K G r , and 2 K L r 2KL_r 2 K L r . For σ ( z ) = ( 1 + e − z ) − 1 \sigma(z)=(1+e^{-z})^{-1} σ ( z ) = ( 1 + e − z ) − 1 ,
∇ x L = ( σ ( Δ x ) − Y ) ∇ x Δ x , \nabla_xL=(\sigma(\Delta_x)-Y)\nabla_x\Delta_x, ∇ x L = ( σ ( Δ x ) − Y ) ∇ x Δ x ,
∇ x 2 L = σ ′ ( Δ x ) ∇ x Δ x ∇ x Δ x ⊤ + ( σ ( Δ x ) − Y ) ∇ x 2 Δ x . \nabla_x^2L=\sigma'(\Delta_x)\nabla_x\Delta_x\nabla_x\Delta_x^\top+(\sigma(\Delta_x)-Y)\nabla_x^2\Delta_x. ∇ x 2 L = σ ′ ( Δ x ) ∇ x Δ x ∇ x Δ x ⊤ + ( σ ( Δ x ) − Y ) ∇ x 2 Δ x .
Use ∣ σ − Y ∣ ≤ 1 |\sigma-Y|\le1 ∣ σ − Y ∣ ≤ 1 , 0 ≤ σ ′ ≤ 1 / 4 0\le\sigma'\le1/4 0 ≤ σ ′ ≤ 1/4 , and 0 ≤ L ≤ log 2 + ∣ Δ x ∣ 0\le L\le\log2+|\Delta_x| 0 ≤ L ≤ log 2 + ∣ Δ x ∣ . Taking the policy expectation preserves nonnegativity.
For example, r x ( s , a ) = B tanh ( x ⊤ φ ( s , a ) ) r_x(s,a)=B\tanh(x^\top\varphi(s,a)) r x ( s , a ) = B tanh ( x ⊤ φ ( s , a )) with sup s , a ∥ φ ( s , a ) ∥ ≤ C φ \sup_{s,a}\|\varphi(s,a)\|\le C_\varphi sup s , a ∥ φ ( s , a ) ∥ ≤ C φ satisfies the assumptions on unrestricted x x x , with B r = B B_r=B B r = B , G r = B C φ G_r=BC_\varphi G r = B C φ , and L r = 2 B C φ 2 L_r=2BC_\varphi^2 L r = 2 B C φ 2 . Its higher derivatives also exist and are locally uniformly bounded. Bounded output alone does not give the derivative envelopes for a generic reward network with every weight trainable. For a fixed offline comparison dataset independent of α \alpha α , this preference term instead has f α = 0 f_\alpha=0 f α = 0 ; the policy-generated objective in the theorem retains the response-chain contribution.
The mixed derivatives used by the proof Let M q M_q M q be the width-independent actor log-policy derivative bounds from Lemma C.8 , for q = 1 , … , 4 q=1,\ldots,4 q = 1 , … , 4 , on the ambient actor neighborhood. Define the likelihood-derivative polynomials
b 0 ( n ) = 1 , b 1 ( n ) = n M 1 , b_0(n)=1,\qquad b_1(n)=nM_1, b 0 ( n ) = 1 , b 1 ( n ) = n M 1 ,
b 2 ( n ) = n M 2 + n 2 M 1 2 , b_2(n)=nM_2+n^2M_1^2, b 2 ( n ) = n M 2 + n 2 M 1 2 ,
b 3 ( n ) = n M 3 + 3 n 2 M 1 M 2 + n 3 M 1 3 , b_3(n)=nM_3+3n^2M_1M_2+n^3M_1^3, b 3 ( n ) = n M 3 + 3 n 2 M 1 M 2 + n 3 M 1 3 ,
b 4 ( n ) = n M 4 + 4 n 2 M 1 M 3 + 3 n 2 M 2 2 + 6 n 3 M 1 2 M 2 + n 4 M 1 4 . b_4(n)=nM_4+4n^2M_1M_3+3n^2M_2^2+6n^3M_1^2M_2+n^4M_1^4. b 4 ( n ) = n M 4 + 4 n 2 M 1 M 3 + 3 n 2 M 2 2 + 6 n 3 M 1 2 M 2 + n 4 M 1 4 .
Set A q ( γ ) = ( 1 − γ ) ∑ t ≥ 0 γ t b q ( t + 1 ) A_q(\gamma)=(1-\gamma)\sum_{t\ge0}\gamma^t b_q(t+1) A q ( γ ) = ( 1 − γ ) ∑ t ≥ 0 γ t b q ( t + 1 ) , ( R 0 , R 1 , R 2 ) = ( B r , G r , L r ) (R_0,R_1,R_2)=(B_r,G_r,L_r) ( R 0 , R 1 , R 2 ) = ( B r , G r , L r ) , and ( F 0 , F 1 , F 2 ) = ( B L , G L , L L ) (F_0,F_1,F_2)=(B_L,G_L,L_L) ( F 0 , F 1 , F 2 ) = ( B L , G L , L L ) .
Lemma C.15 (Uniform mixed-derivative bounds on unrestricted x x x ). For 0 ≤ p ≤ 2 0\le p\le2 0 ≤ p ≤ 2 and 0 ≤ q ≤ 4 0\le q\le4 0 ≤ q ≤ 4 , the separated mixed derivatives exist and satisfy
∥ D x p D α q j ( x , α ) ∥ o p ≤ R p A q ( γ ) , (140) \|D_x^pD_\alpha^qj(x,\alpha)\|_{\mathrm{op}}\le R_pA_q(\gamma), \tag*{(140)} ∥ D x p D α q j ( x , α ) ∥ op ≤ R p A q ( γ ) , (140) ∥ D x p D α q f ( x , α ) ∥ o p ≤ F p b q ( I K ) . (141) \|D_x^pD_\alpha^qf(x,\alpha)\|_{\mathrm{op}}\le F_pb_q(IK). \tag*{(141)} ∥ D x p D α q f ( x , α ) ∥ op ≤ F p b q ( I K ) . (141)
uniformly in x x x and width. For p ≥ 1 p \ge1 p ≥ 1 , the corresponding derivatives of g τ = − j + τ K g_\tau=-j+\tau\mathcal{K} g τ = − j + τ K equal those of − j -j − j ; for p = 0 p=0 p = 0 , add τ \tau τ times the pure actor KL envelope. In particular, with M 0 = 2 C 1 , L A M_0=2C_{1,L}A M 0 = 2 C 1 , L A and M i = M log , i , L M_i=M_{\log,i,L} M i = M l o g , i , L for i ≥ 1 i\ge1 i ≥ 1 ,
∥ D α q K ( α ) ∥ o p ≤ ( 1 − γ ) ∑ t ≥ 0 γ t ∑ j = 0 q ( q j ) b j ( t + 1 ) M q − j < ∞ , 0 ≤ q ≤ 4. (142) \left\|D_\alpha^q\mathcal{K}(\alpha)\right\|_{\mathrm{op}} \le(1-\gamma)\sum_{t\ge0}\gamma^t\sum_{j=0}^{q}\binom{q}{j}b_j(t+1)M_{q-j}<\infty,\qquad0\le q\le4. \tag*{(142)} ∥ D α q K ( α ) ∥ op ≤ ( 1 − γ ) t ≥ 0 ∑ γ t j = 0 ∑ q ( j q ) b j ( t + 1 ) M q − j < ∞ , 0 ≤ q ≤ 4. (142)
No global bounds on pure third or fourth x x x derivatives are claimed.
Proof. The trajectory likelihood has log likelihood equal to a sum of n n n log-policy terms plus a parameter-independent term. Differentiating its exponential q q q times gives sums over set partitions; taking operator norms yields exactly b q ( n ) b_q(n) b q ( n ) as a bound on the likelihood derivative divided by the likelihood. Fixed dynamics imply that x x x derivatives act only on r x r_x r x or L L L , while actor derivatives act only on likelihoods. The feedback kernel has no derivative. For the reward at time t t t there are t + 1 t+1 t + 1 action factors, and the upper tuple has I K IK I K action factors. The global bounds give domination, and each A q ( γ ) A_q(\gamma) A q ( γ ) is finite because a geometric sequence times a fixed-degree polynomial is summable. This justifies differentiation under the expectations and the discounted sum. The same separated calculation gives mixed derivatives with p + q > 4 p+q>4 p + q > 4 when needed, using only p p p primitive x x x derivatives and q q q actor derivatives; it does not assume joint global C 6 C^6 C 6 bounds. Finally, K \mathcal{K} K is independent of x x x . For its time-t t t contribution, apply the product rule to the prefix likelihood and the terminal log-policy-ratio criterion. Assigning j j j derivatives to the likelihood gives b j ( t + 1 ) b_j(t+1) b j ( t + 1 ) ; the remaining derivatives of that criterion are bounded by M q − j M_{q-j} M q − j . Summing the ( q j ) \binom{q}{j} ( j q ) assignments and the discounted series proves (142) , including the fourth actor derivative used by the tracking Hessian.
For clarity, the largest derivative blocks used in the argument are listed below; their lower-order counterparts are also used.
Table 7.
A1’s compact-set C 4 C^4 C 4 statement is not invoked as a global bound. The needed discounted derivative tails are bounded by ( 1 − γ ) ∑ t ≥ H γ t R p b q ( t + 1 ) (1-\gamma)\sum_{t\ge H}\gamma^tR_pb_q(t+1) ( 1 − γ ) ∑ t ≥ H γ t R p b q ( t + 1 ) . They therefore admit the logarithmic truncation schedule. The length K K K of the preference objective is fixed by its definition and introduces no discounted truncation bias.
Response and tracking-potential calculus
This subsection supplies the local-root and gap calculations used in Step B.1. Its upper-domain bounds use only the separated derivative blocks in Lemma C.15 .
Lemma C.16 (Integral secants are uniform in the perturbation). On any segment in the analytic actor neighborhood,
G ( x , U + c S ) − G ( x , U ) c = ∫ 0 1 D α G ( x , U + t c S ) [ S ] d t . (143) \frac{G(x,U+cS)-G(x,U)}{c}=\int_0^1D_\alpha G(x,U+tcS)[S]\,dt. \tag*{(143)} c G ( x , U + c S ) − G ( x , U ) = ∫ 0 1 D α G ( x , U + t c S ) [ S ] d t . (143) If D α G D_\alpha G D α G is bounded and Lipschitz in the variables being compared, then the secant is Lipschitz in those variables and in bounded S S S , uniformly for 0 < c ≤ c 0 0<c\le c_0 0 < c ≤ c 0 . Its continuous value at zero is D α G ( x , U ) [ S ] D_\alpha G(x,U)[S] D α G ( x , U ) [ S ] . A Lipschitz constant L G L_G L G for D α G D_\alpha G D α G also gives error at most L G c ∥ S ∥ 2 / 2 L_Gc\lVert S\rVert^2/2 L G c ∥ S ∥ 2 /2 from this zero-perturbation value.
Proof. Apply the fundamental theorem of calculus to t ↦ G ( x , U + t c S ) t \mapsto G(x,U+tcS) t ↦ G ( x , U + t c S ) . For two triples, subtract the integral expressions and add and subtract the derivative at one common point. Its bounded norm controls S − S ′ S-S' S − S ′ ; its Lipschitz bound controls the remaining difference, using ∥ U + t c S − U ′ − t c S ′ ∥ ≤ ∥ U − U ′ ∥ + c 0 ∥ S − S ′ ∥ \lVert U+tcS-U'-tcS'\rVert\le\lVert U-U'\rVert+c_0\lVert S-S'\rVert ∥ U + t c S − U ′ − t c S ′ ∥ ≤ ∥ U − U ′ ∥ + c 0 ∥ S − S ′ ∥ . Subtracting D α G ( x , U ) [ S ] D_\alpha G(x,U)[S] D α G ( x , U ) [ S ] and integrating t t t proves the last bound.
Lemma C.17 (Uniformly Lipschitz population residual). On the prospective actor neighborhood, R c R_c R c is globally Lipschitz in x x x and locally uniformly Lipschitz in ( U , S ) (U,S) ( U , S ) , with constants independent of c c c and width. At the selected response pair,
R c ( x , α 0 ∗ ( x ) , S c ∗ ( x ) ) = 0. (144) R_c(x,\alpha_0^*(x),S_c^*(x))=0. \tag*{(144)} R c ( x , α 0 ∗ ( x ) , S c ∗ ( x )) = 0. (144)
Proof. Use Lemma C.16 for G = g α G=g_\alpha G = g α and add f α ( x , U + c S ) f_\alpha(x,U+cS) f α ( x , U + c S ) . The derivative blocks g α α x g_{\alpha\alpha x} g α α x , g α α α g_{\alpha\alpha\alpha} g α α α , f α x f_{\alpha x} f α x and f α α f_{\alpha\alpha} f α α are uniformly bounded by Lemma C.15 . Subtracting the base and perturbed response equations gives (144) . The segment between any two admissible U U U values preserves the proof-ball inclusion by convexity.
The scaled potential from (40) has the exact integral form
H c ( x , U , S ) = ∫ 0 1 ( 1 − t ) S T g α α ( x , U + t c S ) S d t + ∫ 0 1 ⟨ f α ( x , U + t c S ) , S ⟩ d t . (145) \mathcal{H}_c(x,U,S)=\int_0^1(1-t)S^{\mathsf T}g_{\alpha\alpha}(x,U+tcS)S\,dt
+\int_0^1\langle f_\alpha(x,U+tcS),S\rangle\,dt. \tag*{(145)} H c ( x , U , S ) = ∫ 0 1 ( 1 − t ) S T g α α ( x , U + t c S ) S d t + ∫ 0 1 ⟨ f α ( x , U + t c S ) , S ⟩ d t . (145)
This formula defines its continuous extension at c = 0 c=0 c = 0 without a divergent additive constant.
Proposition C.18 (Interior root of the scaled residual). If U + c B ‾ s ⊂ B R U+c\overline{B}_s\subset B_R U + c B s ⊂ B R and G f / μ τ < s G_f/\mu_\tau<s G f / μ τ < s , then H c ( x , U , ⋅ ) \mathcal{H}_c(x,U,\cdot) H c ( x , U , ⋅ ) has a unique interior minimizer S ‾ c ( x , U ) ∈ B s \overline{S}_c(x,U)\in B_s S c ( x , U ) ∈ B s , satisfying
R c ( x , U , S ‾ c ) = 0 , ∥ S ‾ c ( x , U ) ∥ ≤ G f / μ τ , S ‾ c ( x , α 0 ∗ ( x ) ) = S c ∗ ( x ) . (146) R_c(x,U,\overline{S}_c)=0,\qquad\lVert\overline{S}_c(x,U)\rVert\le G_f/\mu_\tau,\qquad\overline{S}_c(x,\alpha_0^*(x))=S_c^*(x). \tag*{(146)} R c ( x , U , S c ) = 0 , ∥ S c ( x , U )∥ ≤ G f / μ τ , S c ( x , α 0 ∗ ( x )) = S c ∗ ( x ) . (146) These statements hold uniformly in x x x and extend to c = 0 c=0 c = 0 .
Proof. Direct differentiation gives
∇ S H c = R c , D S R c = h c , α α ( x , U + c S ) ⪰ μ τ I . (147) \nabla_S\mathcal{H}_c=R_c,\qquad D_SR_c=h_{c,\alpha\alpha}(x,U+cS)\succeq\mu_\tau I. \tag*{(147)} ∇ S H c = R c , D S R c = h c , α α ( x , U + c S ) ⪰ μ τ I . (147)
At S = 0 S=0 S = 0 , R c = f α ( x , U ) R_c=f_\alpha(x,U) R c = f α ( x , U ) . On the boundary ∥ S ∥ = s \lVert S\rVert=s ∥ S ∥ = s , ⟨ R c , S ⟩ ≥ μ τ s 2 − G f s > 0 \langle R_c,S\rangle\ge\mu_\tau s^2-G_fs>0 ⟨ R c , S ⟩ ≥ μ τ s 2 − G f s > 0 . Thus a minimizer on the compact ball cannot lie on its boundary. Strong convexity gives uniqueness; comparing its zero gradient with R c ( x , U , 0 ) R_c(x,U,0) R c ( x , U , 0 ) gives the norm bound. Equation (144) and uniqueness identify the response-pair value. All arguments follow from the continuous integral formula at c = 0 c=0 c = 0 as well.
Lemma C.19 (Global gap regularity and self-bounds). Fix sufficiently large finite τ \tau τ and any fixed λ S > 0 \lambda_S>0 λ S > 0 . On the prospective actor neighborhood, S ‾ c \overline{S}_c S c is continuously differentiable, and T \mathcal{T} T is twice continuously differentiable. The following bounds are uniform in x x x , c c c and width:
1 2 μ τ ∥ U − α 0 ∗ ∥ 2 ≤ Δ 0 , 1 2 μ τ ∥ S − S ‾ c ∥ 2 ≤ Γ c , (148) \frac{1}{2}\mu_\tau\lVert U-\alpha_0^*\rVert^2\le\Delta_0,\qquad
\frac{1}{2}\mu_\tau\lVert S-\overline{S}_c\rVert^2\le\Gamma_c, \tag*{(148)} 2 1 μ τ ∥ U − α 0 ∗ ∥ 2 ≤ Δ 0 , 2 1 μ τ ∥ S − S c ∥ 2 ≤ Γ c , (148) ∥ a ∥ 2 ≥ 2 μ τ Δ 0 , ∥ R c ∥ 2 ≥ 2 μ τ Γ c , ∥ a ∥ 2 + ∥ R c ∥ 2 ≤ C T , (149) \lVert a\rVert^2\ge2\mu_\tau\Delta_0,\qquad
\lVert R_c\rVert^2\ge2\mu_\tau\Gamma_c,\qquad
\lVert a\rVert^2+\lVert R_c\rVert^2\le C\mathcal{T}, \tag*{(149)} ∥ a ∥ 2 ≥ 2 μ τ Δ 0 , ∥ R c ∥ 2 ≥ 2 μ τ Γ c , ∥ a ∥ 2 + ∥ R c ∥ 2 ≤ C T , (149) ∥ ∇ ( x , U , S ) T ∥ 2 ≤ C T , ∥ D ( x , U , S ) 2 T ∥ o p ≤ C . (150) \lVert\nabla_{(x,U,S)}\mathcal{T}\rVert^2\le C\mathcal{T},\qquad
\lVert D^2_{(x,U,S)}\mathcal{T}\rVert_{\mathrm{op}}\le C. \tag*{(150)} ∥ ∇ ( x , U , S ) T ∥ 2 ≤ C T , ∥ D ( x , U , S ) 2 T ∥ op ≤ C . (150) The gap values and their first derivatives are bounded on this neighborhood. These constants may depend on the fixed τ \tau τ .
Proof. Actor Hessian upper bounds and Lemma C.50 give the distance and gradient inequalities for both gaps. The implicit-function theorem applied to R c ( x , U , S ˉ c ) = 0 R_c(x,U,\bar{S}_c)=0 R c ( x , U , S ˉ c ) = 0 gives
D y S ˉ c = − ( D S R c ) − 1 D y R c , y = ( x , U ) . D_y\bar{S}_c=-(D_SR_c)^{-1}D_yR_c,\qquad y=(x,U). D y S ˉ c = − ( D S R c ) − 1 D y R c , y = ( x , U ) .
The inverse norm is at most μ τ − 1 \mu_\tau^{-1} μ τ − 1 . Two derivatives of (145) use only g α α x x g_{\alpha\alpha xx} g α α xx , g α α α x g_{\alpha\alpha\alpha x} g α α α x , g α α α α g_{\alpha\alpha\alpha\alpha} g α α α α and f α x x f_{\alpha xx} f α xx , f α α x f_{\alpha\alpha x} f α α x , f α α α f_{\alpha\alpha\alpha} f α α α , together with lower-order blocks. Lemma C.15 supplies uniform bounds for exactly these derivatives. Hence D y S ˉ c D_y\bar{S}_c D y S ˉ c and the full Hessian of H c \mathcal{H}_c H c are bounded.
For y = x , U y=x,U y = x , U , the envelope identities are
D y Γ c = D y H c ( x , U , S ) − D y H c ( x , U , S ˉ c ) , D x Δ 0 = g x ( x , U ) − g x ( x , α 0 ∗ ) . D_y\Gamma_c=D_y\mathcal{H}_c(x,U,S)-D_y\mathcal{H}_c(x,U,\bar{S}_c),\qquad D_x\Delta_0=g_x(x,U)-g_x(x,\alpha_0^*). D y Γ c = D y H c ( x , U , S ) − D y H c ( x , U , S ˉ c ) , D x Δ 0 = g x ( x , U ) − g x ( x , α 0 ∗ ) .
The first difference has norm at most C ∥ S − S ˉ c ∥ C\lVert S-\bar{S}_c\rVert C ∥ S − S ˉ c ∥ ; the second at most C ∥ U − α 0 ∗ ∥ C\lVert U-\alpha_0^*\rVert C ∥ U − α 0 ∗ ∥ . Together with D U Δ 0 = a D_U\Delta_0=a D U Δ 0 = a , D S Γ c = R c D_S\Gamma_c=R_c D S Γ c = R c and the gap inequalities, these prove the gradient self-bound. Differentiating the envelope identities once uses only D 2 H c D^2\mathcal{H}_c D 2 H c , D S S ˉ c D_S\bar{S}_c D S S ˉ c and D x α 0 ∗ D_x\alpha_0^* D x α 0 ∗ , whose bound is in Corollary B.2 . This proves the full Hessian bound without differentiating the minimizing separation twice. Upper Hessian bounds times squared actor distances bound the gap values. No pure third or fourth global x x x derivative is used.
Lemma C.20 (Regularization-uniform weight, distance control and initialization). Fix the actor and separation neighborhoods before τ \tau τ , and let p − = λ ˉ F , L − 1 p_-=\bar{\lambda}_{F,L}^{-1} p − = λ ˉ F , L − 1 , p + = λ ‾ F , L − 1 p_+=\underline{\lambda}_{F,L}^{-1} p + = λ F , L − 1 and K max = L P , L S K_{\max}=L_{P,L_S} K m a x = L P , L S . There are d ∗ , E ∗ > 0 d_*,E_*>0 d ∗ , E ∗ > 0 and a choice λ S > 0 \lambda_S>0 λ S > 0 , independent of sufficiently large τ \tau τ , such that
∥ U − α 0 ∗ ( x ) ∥ 2 + ∥ S − S c ∗ ( x ) ∥ 2 ≤ E ∗ μ τ T ( x , U , S ) , T ( x , 0 , 0 ) ≤ J 1 2 + λ S G f 2 2 μ τ . (151) \lVert U-\alpha_0^*(x)\rVert^2+\lVert S-S_c^*(x)\rVert^2\leq\frac{E_*}{\mu_\tau}\mathcal{T}(x,U,S),\qquad\mathcal{T}(x,0,0)\leq\frac{J_1^2+\lambda_SG_f^2}{2\mu_\tau}. \tag*{(151)} ∥ U − α 0 ∗ ( x ) ∥ 2 + ∥ S − S c ∗ ( x ) ∥ 2 ≤ μ τ E ∗ T ( x , U , S ) , T ( x , 0 , 0 ) ≤ 2 μ τ J 1 2 + λ S G f 2 . (151) One can also require ∥ D U S ˉ c ∥ ≤ d ∗ \lVert D_U\bar{S}_c\rVert\leq d_* ∥ D U S ˉ c ∥ ≤ d ∗ and ∥ ∇ U Γ c ∥ ≤ d ∗ ∥ R c ∥ \lVert\nabla_U\Gamma_c\rVert\leq d_*\lVert R_c\rVert ∥ ∇ U Γ c ∥ ≤ d ∗ ∥ R c ∥ on this neighborhood.
Proof. Choose a fixed τ 0 ≥ 1 \tau_0\geq1 τ 0 ≥ 1 above the curvature threshold in (34) . Since D α 3 K D_\alpha^3\mathcal{K} D α 3 K is bounded by L K L_K L K , differentiating the residual integral gives
∥ D U R c ∥ ≤ ( J 3 + τ L K ) s + L f ≤ D ∗ τ , D ∗ = ( J 3 / τ 0 + L K ) s + L f / τ 0 . \lVert D_UR_c\rVert\leq(J_3+\tau L_K)s+L_f\leq D_*\tau,\qquad D_*=(J_3/\tau_0+L_K)s+L_f/\tau_0. ∥ D U R c ∥ ≤ ( J 3 + τ L K ) s + L f ≤ D ∗ τ , D ∗ = ( J 3 / τ 0 + L K ) s + L f / τ 0 .
Set d ∗ = D ∗ / κ d_*=D_*/\kappa d ∗ = D ∗ / κ . Implicit differentiation bounds D U S ˉ c D_U\bar{S}_c D U S ˉ c by d ∗ d_* d ∗ . The envelope identity and strong monotonicity imply ∥ ∇ U Γ c ∥ ≤ D ∗ τ ∥ S − S ˉ c ∥ ≤ d ∗ ∥ R c ∥ \lVert\nabla_U\Gamma_c\rVert\leq D_*\tau\lVert S-\bar{S}_c\rVert\leq d_*\lVert R_c\rVert ∥ ∇ U Γ c ∥ ≤ D ∗ τ ∥ S − S ˉ c ∥ ≤ d ∗ ∥ R c ∥ . Choose
0 < λ S ≤ min { 1 , p − 2 4 ( p + + d ∗ + K max ) 2 } . (152) 0<\lambda_S\leq\min\left\{1,\frac{p_-^2}{4(p_++d_*+K_{\max})^2}\right\}. \tag*{(152)} 0 < λ S ≤ min { 1 , 4 ( p + + d ∗ + K m a x ) 2 p − 2 } . (152)
Omit the second restriction if its denominator is zero. This choice is independent of τ \tau τ and makes the population direction descend as proved in Lemma C.27 .
Integrating D U S ˉ c D_U\bar{S}_c D U S ˉ c on the segment from U U U to α 0 ∗ \alpha_0^* α 0 ∗ gives ∥ S ˉ c ( x , U ) − S c ∗ ( x ) ∥ ≤ d ∗ ∥ U − α 0 ∗ ∥ \lVert\bar{S}_c(x,U)-S_c^*(x)\rVert\leq d_*\lVert U-\alpha_0^*\rVert ∥ S ˉ c ( x , U ) − S c ∗ ( x )∥ ≤ d ∗ ∥ U − α 0 ∗ ∥ . The proof-ball inclusion holds along that segment by convexity. The triangle inequality and (148) yield the first bound with E ∗ = max { 2 ( 1 + 2 d ∗ 2 ) , 4 / λ S } E_*=\max\{2(1+2d_*^2),4/\lambda_S\} E ∗ = max { 2 ( 1 + 2 d ∗ 2 ) , 4/ λ S } . At U = S = 0 U=S=0 U = S = 0 , the KL gradient vanishes, so ∥ a ∥ ≤ J 1 \lVert a\rVert\leq J_1 ∥ a ∥ ≤ J 1 and ∥ R c ∥ = ∥ f α ( x , 0 ) ∥ ≤ G f \lVert R_c\rVert=\lVert f_\alpha(x,0)\rVert\leq G_f ∥ R c ∥ = ∥ f α ( x , 0 )∥ ≤ G f . Apply the residual-to-gap inequalities in (149) to prove the initialization bound. This argument holds for every x x x and requires no warm start.
Lemma C.21 (Tracked corrected-field error). Let V c ∗ ( x ) = V c ( x , α 0 ∗ ( x ) , S c ∗ ( x ) ) V_c^*(x)=V_c(x,\alpha_0^*(x),S_c^*(x)) V c ∗ ( x ) = V c ( x , α 0 ∗ ( x ) , S c ∗ ( x )) . Then
∥ V c ( p ) − ∇ Φ ( x ) ∥ 2 ≤ C ( T ( p ) + c 2 ) , ∥ V c ( p ) ∥ 2 ≤ C ( ∥ ∇ Φ ( x ) ∥ 2 + T ( p ) + c 2 ) . (153) \lVert V_c(p)-\nabla\Phi(x)\rVert^2\leq C(\mathcal{T}(p)+c^2),\qquad\lVert V_c(p)\rVert^2\leq C(\lVert\nabla\Phi(x)\rVert^2+\mathcal{T}(p)+c^2). \tag*{(153)} ∥ V c ( p ) − ∇Φ ( x ) ∥ 2 ≤ C ( T ( p ) + c 2 ) , ∥ V c ( p ) ∥ 2 ≤ C (∥ ∇Φ ( x ) ∥ 2 + T ( p ) + c 2 ) . (153)
Proof. Lemma B.3 gives the Lipschitz bound for V c V_c V c in the actor pair and ∥ V c ∗ − ∇ Φ ∥ ≤ C s e c c \lVert V_c^*-\nabla\Phi\rVert\leq C_{\mathrm{sec}}c ∥ V c ∗ − ∇Φ ∥ ≤ C sec c . Combining it with (151) gives the first inequality. Adding ∇ Φ \nabla\Phi ∇Φ and using the squared triangle inequality gives the second.
Common-KL geometry and finite local updates
Both actors use the same stored empirical state measure d ˉ t = ρ ^ n \bar{d}_t = \widehat{\rho}_n d ˉ t = ρ n from Definition C.28 . Its observable covariance certificate provides the same Fisher constants as the population law
ν t p o p = 1 2 d U t γ + 1 2 d U t + c S t γ (154) \nu_t^{\mathrm{pop}} = \frac{1}{2}dU_t^\gamma+ \frac{1}{2}dU_{t+c_S t}^\gamma \tag*{(154)} ν t pop = 2 1 d U t γ + 2 1 d U t + c S t γ (154)
The latter explains the original constants but is not evaluated by the algorithm. The metric data are drawn once, independently of the actor initialization and all subsequent training blocks, and their cost is included in (76) . A rejected covariance certificate returns x 0 x_0 x 0 . Conditional on acceptance, the following analytic reference maps apply to the actual empirical metric. Their finite implementation and its error propagation are proved below.
For a probability state law ν \nu ν and actor points α \alpha α , b b b , define
K m , ν ( α ∣ b ) = E s ∼ ν D K L ( π m , b ( ⋅ ∣ s ) ∥ π m , α ( ⋅ ∣ s ) ) (155) K_{m,\nu}(\alpha\mid b) = \mathbb{E}_{s \sim\nu} D_{\mathrm{KL}}\left(\pi_{m,b}(\cdot\mid s) \mathbin{\|} \pi_{m,\alpha}(\cdot\mid s)\right) \tag*{(155)} K m , ν ( α ∣ b ) = E s ∼ ν D KL ( π m , b ( ⋅ ∣ s ) ∥ π m , α ( ⋅ ∣ s ) ) (155)
and the exact local map
M m , ν , η ( b , v ) = arg min α ∈ B ( b , r p r o x ) { ⟨ v , α − b ⟩ + η − 1 K m , ν ( α ∣ b ) } . (156) M_{m,\nu,\eta}(b,v) = \underset{\alpha\in B(b,r_{\mathrm{prox}})}{\arg\min}\left\{\langle v,\alpha-b\rangle+ \eta^{-1}K_{m,\nu}(\alpha\mid b)\right\}. \tag*{(156)} M m , ν , η ( b , v ) = α ∈ B ( b , r prox ) arg min { ⟨ v , α − b ⟩ + η − 1 K m , ν ( α ∣ b ) } . (156)
Choose
0 < r p r o x ≤ min { R / 8 , λ F , L / ( 2 M log , 3 , L ) } , (157) 0 < r_{\mathrm{prox}} \le\min\left\{R/8,\lambda_{F,L}/(2M_{\log,3,L})\right\}, \tag*{(157)} 0 < r prox ≤ min { R /8 , λ F , L / ( 2 M l o g , 3 , L ) } , (157)
omitting a zero-denominator restriction. If G p r o x ( H ) ≥ 1 G_{\mathrm{prox}}(H) \ge1 G prox ( H ) ≥ 1 bounds both sampled actor directions, impose the deterministic local-step restriction
η y G p r o x ( H ) ≤ m K , L r p r o x / 2 , m K , L = λ F , L / 2. (158) \eta_y G_{\mathrm{prox}}(H) \le m_{K,L}r_{\mathrm{prox}}/2,\qquad m_{K,L} = \lambda_{F,L}/2. \tag*{(158)} η y G prox ( H ) ≤ m K , L r prox /2 , m K , L = λ F , L /2. (158)
The sampled bounds in Appendix § 10 .§ 6 make G p r o x ( H ) G_{\mathrm{prox}}(H) G prox ( H ) polynomial in H H H . This condition is satisfied by the headline schedule for sufficiently small accuracy. The exact map is used only as a mathematical reference. Algorithm 1 returns the residual-controlled finite approximation in Definition C.32 ; its error is incorporated below. For brevity, write
k m , ν ( α , b ) : = ∇ α K m , ν ( α ∣ b ) . (159) k_{m,\nu}(\alpha,b) := \nabla_\alpha K_{m,\nu}(\alpha\mid b). \tag*{(159)} k m , ν ( α , b ) := ∇ α K m , ν ( α ∣ b ) . (159)
Lemma C.22 (Derivatives of the oriented common KL). For every state law ν \nu ν ,
k m , ν ( α , b ) = − E ν , π m , b [ ∇ α log π m , α ( a ∣ s ) ] , (160) k_{m,\nu}(\alpha,b) = -\mathbb{E}_{\nu,\pi_{m,b}}\left[\nabla_\alpha\log\pi_{m,\alpha}(a \mid s)\right], \tag*{(160)} k m , ν ( α , b ) = − E ν , π m , b [ ∇ α log π m , α ( a ∣ s ) ] , (160) k m , ν ( b , b ) = 0 , (161) k_{m,\nu}(b,b) = 0, \tag*{(161)} k m , ν ( b , b ) = 0 , (161) D 1 k m , ν ( b , b ) = F m , ν ( b ) , (162) D_1 k_{m,\nu}(b,b) = F_{m,\nu}(b), \tag*{(162)} D 1 k m , ν ( b , b ) = F m , ν ( b ) , (162) D 2 k m , ν ( b , b ) = − F m , ν ( b ) . (163) D_2 k_{m,\nu}(b,b) = -F_{m,\nu}(b). \tag*{(163)} D 2 k m , ν ( b , b ) = − F m , ν ( b ) . (163) On the fixed ambient actor ball, all mixed derivatives of k m , ν k_{m,\nu} k m , ν of total order at most three are bounded uniformly in the widths and in the probability law ν \nu ν .
Proof. Differentiate the first argument of the oriented KL under the integral to obtain (160) . The conditional score identity E π b ∇ b log π b = 0 \mathbb{E}_{\pi_b}\nabla_b \log\pi_b = 0 E π b ∇ b log π b = 0 proves (161) . Differentiating this zero-score identity in a direction h h h gives
E π b [ D 2 log π b [ h ] + ∇ log π b ⟨ ∇ log π b , h ⟩ ] = 0. \mathbb{E}_{\pi_b}\left[D^2 \log\pi_b[h] + \nabla\log\pi_b \langle\nabla\log\pi_b,h\rangle\right] = 0. E π b [ D 2 log π b [ h ] + ∇ log π b ⟨ ∇ log π b , h ⟩ ] = 0.
Hence − E D 2 log π b -\mathbb{E}D^2 \log\pi_b − E D 2 log π b equals the Fisher matrix, proving (162) . Differentiating the expectation in (160) with respect to the sampling parameter b b b gives the negative Fisher matrix at the diagonal, proving (163) . Each further derivative is a finite sum of products of log-policy derivatives and score factors. Lemma C.8 bounds all terms through the required order.
Lemma C.23 (Uniform local KL conditioning). Let ν \nu ν be a probability law such that λ ‾ F , L I r ⪯ F m , ν ( b ) ⪯ λ ‾ F , L I r \underline{\lambda}_{F,L}I_r \preceq F_{m,\nu}(b) \preceq\overline{\lambda}_{F,L}I_r λ F , L I r ⪯ F m , ν ( b ) ⪯ λ F , L I r , let b ∈ Θ R b \in\Theta_R b ∈ Θ R , and suppose the segment from b b b to α \alpha α lies in the local geometry tube with ∥ α − b ∥ ≤ r p r o x \|\alpha-b\| \le r_{\mathrm{prox}} ∥ α − b ∥ ≤ r prox . Then
m K , L I r ⪯ ∇ α α 2 K m , ν ( α ∣ b ) ⪯ M K , L I r , (164) m_{K,L}I_r \preceq\nabla_{\alpha\alpha}^{2}\mathcal{K}_{m,\nu}(\alpha\mid b) \preceq M_{K,L}I_r, \tag*{(164)} m K , L I r ⪯ ∇ α α 2 K m , ν ( α ∣ b ) ⪯ M K , L I r , (164) where
m K , L : = 1 2 λ ‾ F , L , M K , L : = λ ‾ F , L + M log , 3 , L r p r o x . (165) m_{K,L} := \frac{1}{2}\underline{\lambda}_{F,L}, \qquad M_{K,L} := \overline{\lambda}_{F,L} + M_{\log,3,L}r_{\mathrm{prox}}. \tag*{(165)} m K , L := 2 1 λ F , L , M K , L := λ F , L + M l o g , 3 , L r prox . (165) Consequently,
m K , L 2 ∥ α − b ∥ 2 ≤ K m , ν ( α ∣ b ) ≤ M K , L 2 ∥ α − b ∥ 2 . (166) \frac{m_{K,L}}{2}\|\alpha-b\|^{2} \le\mathcal{K}_{m,\nu}(\alpha\mid b) \le\frac{M_{K,L}}{2}\|\alpha-b\|^{2}. \tag*{(166)} 2 m K , L ∥ α − b ∥ 2 ≤ K m , ν ( α ∣ b ) ≤ 2 M K , L ∥ α − b ∥ 2 . (166) Moreover,
∣ K m , ν ( b + Δ ∣ b ) − 1 2 ⟨ Δ , F m , ν ( b ) Δ ⟩ ∣ ≤ M log , 3 , L 6 ∥ Δ ∥ 3 . (167) \left|\mathcal{K}_{m,\nu}(b+\Delta\mid b)-\frac{1}{2}\langle\Delta,F_{m,\nu}(b)\Delta\rangle\right| \le\frac{M_{\log,3,L}}{6}\|\Delta\|^{3}. \tag*{(167)} K m , ν ( b + Δ ∣ b ) − 2 1 ⟨ Δ , F m , ν ( b ) Δ ⟩ ≤ 6 M l o g , 3 , L ∥Δ ∥ 3 . (167) Proof. By Lemma C.22 , the first-argument Hessian at the diagonal is the Fisher matrix. Since
∇ α α 2 K ( α ∣ b ) = − E ν , π b D α 2 log π α , \nabla_{\alpha\alpha}^{2}\mathcal{K}(\alpha\mid b)=-\mathbb{E}_{\nu,\pi_b}D_{\alpha}^{2}\log\pi_{\alpha}, ∇ α α 2 K ( α ∣ b ) = − E ν , π b D α 2 log π α , its derivative with respect to α \alpha α has norm at most M log , 3 , L M_{\log,3,L} M l o g , 3 , L . Therefore
∥ ∇ α α 2 K ( α ∣ b ) − F m , ν ( b ) ∥ o p ≤ M log , 3 , L ∥ α − b ∥ . \left\|\nabla_{\alpha\alpha}^{2}\mathcal{K}(\alpha\mid b)-F_{m,\nu}(b)\right\|_{\mathrm{op}} \le M_{\log,3,L}\|\alpha-b\|. ∇ α α 2 K ( α ∣ b ) − F m , ν ( b ) op ≤ M l o g , 3 , L ∥ α − b ∥. Use the stated Fisher bounds and (157) to obtain (164) . The implemented empirical metric satisfies these bounds by Proposition C.30 ; its proof does not assume domination by a population state law. Since the KL and its first derivative vanish at the diagonal, integrating the Hessian twice along the segment gives (166) . The integral third-order Taylor remainder gives (167) .
Lemma C.24 (Exact local proximal map and quantitative remainder). Let ν \nu ν satisfy the Fisher premise of Lemma C.23 , let b b b lie at distance at least r p r o x r_{\mathrm{prox}} r prox from ∂ Θ R \partial\Theta_R ∂ Θ R , and let ∥ v ∥ ≤ G p r o x ( H ) \|v\| \le G_{\mathrm{prox}}(H) ∥ v ∥ ≤ G prox ( H ) . Under (158) , the problem (156) has a unique interior minimizer M = M m , ν , η ( b , v ) M=M_{m,\nu,\eta}(b,v) M = M m , ν , η ( b , v ) satisfying
k m , ν ( M , b ) + η v = 0 , ∥ M − b ∥ ≤ η m K , L ∥ v ∥ ≤ 1 2 r p r o x . (168) k_{m,\nu}(M,b)+\eta v=0,\qquad\|M-b\|\le\frac{\eta}{m_{K,L}}\|v\|\le\frac{1}{2}r_{\mathrm{prox}}. \tag*{(168)} k m , ν ( M , b ) + η v = 0 , ∥ M − b ∥ ≤ m K , L η ∥ v ∥ ≤ 2 1 r prox . (168) Let
P m , ν ( b ) : = F m , ν ( b ) − 1 . (169) P_{m,\nu}(b):=\mathcal{F}_{m,\nu}(b)^{-1}. \tag*{(169)} P m , ν ( b ) := F m , ν ( b ) − 1 . (169) There is a C 1 C^{1} C 1 remainder Q m , ν Q_{m,\nu} Q m , ν such that
M m , ν , η ( b , v ) = b − η P m , ν ( b ) v + η 2 Q m , ν ( b , v , η ) , (170) M_{m,\nu,\eta}(b,v)=b-\eta P_{m,\nu}(b)v+\eta^{2}Q_{m,\nu}(b,v,\eta), \tag*{(170)} M m , ν , η ( b , v ) = b − η P m , ν ( b ) v + η 2 Q m , ν ( b , v , η ) , (170) and
∥ Q m , ν ( b , v , η ) ∥ ≤ C Q , L ∥ v ∥ 2 , (171) \|Q_{m,\nu}(b,v,\eta)\|\le C_{Q,L}\|v\|^{2}, \tag*{(171)} ∥ Q m , ν ( b , v , η ) ∥ ≤ C Q , L ∥ v ∥ 2 , (171) ∥ D b Q m , ν ( b , v , η ) ∥ o p ≤ C Q , L ∥ v ∥ 2 , (172) \|D_bQ_{m,\nu}(b,v,\eta)\|_{\mathrm{op}}\le C_{Q,L}\|v\|^{2}, \tag*{(172)} ∥ D b Q m , ν ( b , v , η ) ∥ op ≤ C Q , L ∥ v ∥ 2 , (172) ∥ D v Q m , ν ( b , v , η ) ∥ o p ≤ C Q , L ∥ v ∥ . (173) \|D_vQ_{m,\nu}(b,v,\eta)\|_{\mathrm{op}}\le C_{Q,L}\|v\|. \tag*{(173)} ∥ D v Q m , ν ( b , v , η ) ∥ op ≤ C Q , L ∥ v ∥. (173) The constant is uniform in the widths, ν \nu ν , and sufficiently small η \eta η ; it is also independent of H H H under (158) .
Proof. Strong convexity of the local objective gives uniqueness. For a constrained minimizer M M M , the variational inequality with comparison point b b b gives
⟨ k ( M , b ) + η v , b − M ⟩ ≥ 0. \langle k(M,b)+\eta v,b-M\rangle\ge0. ⟨ k ( M , b ) + η v , b − M ⟩ ≥ 0.
Since k ( b , b ) = 0 k(b,b)=0 k ( b , b ) = 0 and k ( ⋅ , b ) k(\cdot,b) k ( ⋅ , b ) is m K , L m_{K,L} m K , L -strongly monotone,
m K , L ∥ M − b ∥ 2 ≤ ⟨ k ( M , b ) , M − b ⟩ ≤ − η ⟨ v , M − b ⟩ ≤ η ∥ v ∥ ∥ M − b ∥ . m_{K,L}\lVert M-b\rVert^2 \le\langle k(M,b),M-b\rangle\le-\eta\langle v,M-b\rangle\le\eta\lVert v\rVert\lVert M-b\rVert. m K , L ∥ M − b ∥ 2 ≤ ⟨ k ( M , b ) , M − b ⟩ ≤ − η ⟨ v , M − b ⟩ ≤ η ∥ v ∥ ∥ M − b ∥ .
This proves the displacement bound. The tube and stepsize conditions make the solution strictly interior, so the first-order equation holds. Write M = b + η d M=b+\eta d M = b + η d . For η ≥ 0 \eta\ge0 η ≥ 0 , define the continuously extended map
Ψ ( d , b , v , η ) : = ∫ 0 1 D 1 k ( b + t η d , b ) [ d ] d t + v . (174) \Psi(d,b,v,\eta):=\int_0^1 D_1k(b+t\eta d,b)[d]\,dt+v. \tag*{(174)} Ψ ( d , b , v , η ) := ∫ 0 1 D 1 k ( b + t η d , b ) [ d ] d t + v . (174)
For η > 0 \eta>0 η > 0 , the fundamental theorem of calculus gives the exact identity
Ψ ( d , b , v , η ) = η − 1 k ( b + η d , b ) + v , (175) \Psi(d,b,v,\eta)=\eta^{-1}k(b+\eta d,b)+v, \tag*{(175)} Ψ ( d , b , v , η ) = η − 1 k ( b + η d , b ) + v , (175)
while at η = 0 \eta=0 η = 0 it becomes F m , ν ( b ) d + v = 0 \mathcal{F}_{m,\nu}(b)d+v=0 F m , ν ( b ) d + v = 0 . Differentiating (175) with respect to d d d gives, for η > 0 \eta>0 η > 0 ,
D d Ψ ( d , b , v , η ) = D 1 k ( b + η d , b ) , (176) D_d\Psi(d,b,v,\eta)=D_1k(b+\eta d,b), \tag*{(176)} D d Ψ ( d , b , v , η ) = D 1 k ( b + η d , b ) , (176)
and the same formula holds at η = 0 \eta=0 η = 0 by continuity with value F m , ν ( b ) \mathcal{F}_{m,\nu}(b) F m , ν ( b ) . Hence Lemma C.23 gives D d Ψ ⪰ m K , L I r D_d\Psi\succeq m_{K,L}I_r D d Ψ ⪰ m K , L I r throughout the local tube. Since the derivatives of k k k through order three are uniformly bounded, the quantitative implicit-function theorem gives a uniformly C 2 C^2 C 2 solution d = d ( b , v , η ) d=d(b,v,\eta) d = d ( b , v , η ) with d ( b , v , 0 ) = − P m , ν ( b ) v d(b,v,0)=-P_{m,\nu}(b)v d ( b , v , 0 ) = − P m , ν ( b ) v and ∥ d ∥ ≤ m K , L − 1 ∥ v ∥ \lVert d\rVert\le m_{K,L}^{-1}\lVert v\rVert ∥ d ∥ ≤ m K , L − 1 ∥ v ∥ .
Differentiate (174) with respect to η \eta η . Every term in ∂ η Ψ \partial_\eta\Psi ∂ η Ψ contains two factors of d d d , so ∥ ∂ η Ψ ∥ ≤ C L ∥ v ∥ 2 \lVert\partial_\eta\Psi\rVert\le C_L\lVert v\rVert^2 ∥ ∂ η Ψ ∥ ≤ C L ∥ v ∥ 2 . Uniform invertibility of D d Ψ D_d\Psi D d Ψ gives ∥ ∂ η d ∥ ≤ C L ∥ v ∥ 2 \lVert\partial_\eta d\rVert\le C_L\lVert v\rVert^2 ∥ ∂ η d ∥ ≤ C L ∥ v ∥ 2 . Differentiating this identity in b b b and v v v and using the bounded third derivatives of k k k gives
∥ D b ∂ η d ∥ ≤ C L ∥ v ∥ 2 , ∥ D v ∂ η d ∥ ≤ C L ∥ v ∥ . \lVert D_b\partial_\eta d\rVert\le C_L\lVert v\rVert^2,\qquad\lVert D_v\partial_\eta d\rVert\le C_L\lVert v\rVert. ∥ D b ∂ η d ∥ ≤ C L ∥ v ∥ 2 , ∥ D v ∂ η d ∥ ≤ C L ∥ v ∥ .
Finally,
d ( b , v , η ) + P ( b ) v = η ∫ 0 1 ∂ η d ( b , v , t η ) d t . d(b,v,\eta)+P(b)v=\eta\int_0^1\partial_\eta d(b,v,t\eta)\,dt. d ( b , v , η ) + P ( b ) v = η ∫ 0 1 ∂ η d ( b , v , t η ) d t .
Define Q Q Q by the integral on the right. Multiplying by η \eta η in M = b + η d M=b+\eta d M = b + η d gives (170) , and the three displayed derivative bounds follow from the preceding estimates. Only the displacement η ∥ v ∥ \eta\lVert v\rVert η ∥ v ∥ is restricted to the fixed local ball; the displayed powers of ∥ v ∥ \lVert v\rVert ∥ v ∥ account explicitly for the magnitude of the direction. Thus increasing the polynomial sample envelope with H H H does not enter the remainder constant. □
Lemma C.25 (c c c -uniform coupled expansion for the exact deep KL updates). Fix a common state law ν \nu ν and let
U + = M m , ν , η ( U , A ^ ) , U^+=M_{m,\nu,\eta}(U,\widehat A), U + = M m , ν , η ( U , A ) , U + + c S + = M m , ν , η ( U + c S , A ^ + c R ^ ) . U^++cS^+=M_{m,\nu,\eta}(U+cS,\widehat A+c\widehat R). U + + c S + = M m , ν , η ( U + c S , A + c R ) . Set
P 0 = P m , ν ( U ) , P c = P m , ν ( U + c S ) , K c ( U , S ) = P 0 − P c c . (177) P^0=P_{m,\nu}(U),\qquad P^c=P_{m,\nu}(U+cS),\qquad K_c(U,S)=\frac{P^0-P^c}{c}. \tag*{(177)} P 0 = P m , ν ( U ) , P c = P m , ν ( U + c S ) , K c ( U , S ) = c P 0 − P c . (177)
Then
U + − U = − η P 0 A ^ + ρ U , (178) U^{+}-U=-\eta P^{0}\widehat{A}+\rho^{U}, \tag*{(178)} U + − U = − η P 0 A + ρ U , (178)
S + − S = − η P c R ^ + η K c ( U , S ) A ^ + ρ S , (179) S^{+}-S=-\eta P^{c}\widehat{R}+\eta K_{c}(U,S)\widehat{A}+\rho^{S}, \tag*{(179)} S + − S = − η P c R + η K c ( U , S ) A + ρ S , (179)
∥ K c ( U , S ) ∥ o p ≤ L P , L ∥ S ∥ , (180) \lVert K_{c}(U,S)\rVert_{\mathrm{op}}\le L_{P,L}\lVert S\rVert, \tag*{(180)} ∥ K c ( U , S ) ∥ op ≤ L P , L ∥ S ∥ , (180)
where
∥ ρ U ∥ ≤ C Q , L η 2 ∥ A ^ ∥ 2 , (181) \lVert\rho^{U}\rVert\le C_{Q,L}\eta^{2}\lVert\widehat{A}\rVert^{2}, \tag*{(181)} ∥ ρ U ∥ ≤ C Q , L η 2 ∥ A ∥ 2 , (181)
∥ ρ S ∥ ≤ C Q , L η 2 [ ∥ S ∥ ( ∥ A ^ ∥ + c ∥ R ^ ∥ ) 2 + ( ∥ A ^ ∥ + c ∥ R ^ ∥ ) ∥ R ^ ∥ ] . (182) \lVert\rho^{S}\rVert\le C_{Q,L}\eta^{2}\left[\lVert S\rVert\left(\lVert\widehat{A}\rVert+c\lVert\widehat{R}\rVert\right)^{2}+\left(\lVert\widehat{A}\rVert+c\lVert\widehat{R}\rVert\right)\lVert\widehat{R}\rVert\right]. \tag*{(182)} ∥ ρ S ∥ ≤ C Q , L η 2 [ ∥ S ∥ ( ∥ A ∥ + c ∥ R ∥ ) 2 + ( ∥ A ∥ + c ∥ R ∥ ) ∥ R ∥ ] . (182)
Every constant is uniform as c ↓ 0 c\downarrow0 c ↓ 0 and over the hidden widths.
Proof. Apply (170) to both updates and subtract before dividing by c c c :
S + − S = − η P c ( A ^ + c R ^ ) − P 0 A ^ c + η 2 Q ( U + c S , A ^ + c R ^ , η ) − Q ( U , A ^ , η ) c . S^{+}-S=-\eta\frac{P^{c}\left(\widehat{A}+c\widehat{R}\right)-P^{0}\widehat{A}}{c}
+\eta^{2}\frac{Q\left(U+cS,\widehat{A}+c\widehat{R},\eta\right)-Q\left(U,\widehat{A},\eta\right)}{c}. S + − S = − η c P c ( A + c R ) − P 0 A + η 2 c Q ( U + c S , A + c R , η ) − Q ( U , A , η ) .
The inverse-Fisher Lipschitz estimate gives (180) . For the nonlinear remainder, apply the fundamental theorem of calculus along θ ↦ ( U + θ c S , A ^ + θ c R ^ ) \theta\mapsto\left(U+\theta cS,\widehat{A}+\theta c\widehat{R}\right) θ ↦ ( U + θ c S , A + θ c R ) and use (172) –(173) . This proves (182) ; the base estimate is immediate from (171) .
Lemma C.26 (Deterministic prospective actor increments). On the safe actor set, for sufficiently small absolute steps,
∥ U + − U ∥ + ∥ S + − S ∥ ≤ C η y poly ( H ) . \lVert U^{+}-U\rVert+\lVert S^{+}-S\rVert\le C\eta_{y}\operatorname{poly}(H). ∥ U + − U ∥ + ∥ S + − S ∥ ≤ C η y poly ( H ) . The constants are uniform in c c c , width and critic scale.
Proof. Lemma C.25 bounds the virtual exact increments by linear sampled directions plus quadratic local-map remainders. The directions are polynomial in H H H by Lemma C.43 , and ∥ S ∥ ≤ s \lVert S\rVert\le s ∥ S ∥ ≤ s , c ≤ c 0 c\le c_{0} c ≤ c 0 , with uniformly bounded inverse metrics. Lemma C.35 adds at most c η y 2 c\eta_{y}^{2} c η y 2 to the base increment and 2 η y 2 2\eta_{y}^{2} 2 η y 2 to the normalized increment. The displayed bound therefore holds for the finite implementation as well when c , η y ≤ 1 c,\eta_{y}\le1 c , η y ≤ 1 . Selecting accuracy small enough gives ∥ Δ U ∥ < r p r o x / 2 \lVert\Delta U\rVert<r_{\mathrm{prox}}/2 ∥ Δ U ∥ < r prox /2 and ∥ Δ S ∥ < s / 4 \lVert\Delta S\rVert<s/4 ∥ Δ S ∥ < s /4 . For all interpolation points used in the paired remainder proof, the base point is U + t c S U+tcS U + t c S and its direction is a ^ + t c R ^ \widehat{a}+tc\widehat{R} a + t c R , 0 ≤ t ≤ 1 0\le t\le1 0 ≤ t ≤ 1 . They obey the same buffered actor inclusion and direction bound, so the local map exists along the entire interpolation before containment is invoked.
The population direction. Let p − = λ ‾ F , L − 1 p_{-}=\overline{\lambda}_{F,L}^{-1} p − = λ F , L − 1 , p + = λ ‾ F , L − 1 p_{+}=\underline{\lambda}_{F,L}^{-1} p + = λ F , L − 1 and K max = L P , L s K_{\max}=L_{P,L}s K m a x = L P , L s . The constant d ∗ d_{*} d ∗ and the weight λ S \lambda_{S} λ S are fixed in Lemma C.20 .
Lemma C.27 (Population coupled direction is descending). Let A = a ( p ) = g α ( x , U ) A=a(p)=g_{\alpha}(x,U) A = a ( p ) = g α ( x , U ) and R = R c ( p ) R=R_{c}(p) R = R c ( p ) . Let
d U = − P 0 A d_{U}=-P^{0}A d U = − P 0 A d S = − P c R + K c ( U , S ) A , d_{S}=-P^{c}R+K_{c}(U,S)A, d S = − P c R + K c ( U , S ) A , where P 0 , P c P^{0},P^{c} P 0 , P c are inverse common-state Fisher matrices. Then
⟨ ∇ U T , d U ⟩ + ⟨ ∇ S T , d S ⟩ ≤ − ρ 0 , L T , ρ 0 , L : = p − min { μ τ , μ τ } . (183) \langle\nabla_{U}\mathcal{T},d_{U}\rangle+\langle\nabla_{S}\mathcal{T},d_{S}\rangle\le-\rho_{0,L}\mathcal{T},\qquad\rho_{0,L}:=p_{-}\min\{\mu_{\tau},\mu_{\tau}\}. \tag*{(183)} ⟨ ∇ U T , d U ⟩ + ⟨ ∇ S T , d S ⟩ ≤ − ρ 0 , L T , ρ 0 , L := p − min { μ τ , μ τ } . (183)
Proof. Since p − I ⪯ P 0 p_-I \preceq P^0 p − I ⪯ P 0 , P c ⪯ p + I P^c \preceq p_+I P c ⪯ p + I and ∥ K c ∥ ≤ K max \lVert K_c\rVert\le K_{\max} ∥ K c ∥ ≤ K m a x , the left-hand side is at most
− p − ∥ A ∥ 2 − λ S p − ∥ R ∥ 2 + λ S ( p + + d ∗ + K max ) ∥ A ∥ ∥ R ∥ . -p_-\lVert A\rVert^2-\lambda_Sp_-\lVert R\rVert^2+\lambda_S(p_++d_*+K_{\max})\lVert A\rVert\lVert R\rVert. − p − ∥ A ∥ 2 − λ S p − ∥ R ∥ 2 + λ S ( p + + d ∗ + K m a x ) ∥ A ∥ ∥ R ∥ .
Let D = p + + d ∗ + K max D=p_++d_*+K_{\max} D = p + + d ∗ + K m a x . Young’s inequality gives
λ S D ∥ A ∥ ∥ R ∥ ≤ p − 4 ∥ A ∥ 2 + λ S 2 D 2 p − ∥ R ∥ 2 . \lambda_SD\lVert A\rVert\lVert R\rVert\le\frac{p_-}{4}\lVert A\rVert^2+\frac{\lambda_S^2D^2}{p_-}\lVert R\rVert^2 . λ S D ∥ A ∥ ∥ R ∥ ≤ 4 p − ∥ A ∥ 2 + p − λ S 2 D 2 ∥ R ∥ 2 .
The choice (152) makes the last coefficient at most λ S p − / 4 \lambda_Sp_-/4 λ S p − /4 . Hence the directional derivative is at most − ( p − / 2 ) ∥ A ∥ 2 − ( λ S p − / 2 ) ∥ R ∥ 2 -(p_-/2)\lVert A\rVert^2-(\lambda_Sp_-/2)\lVert R\rVert^2 − ( p − /2 ) ∥ A ∥ 2 − ( λ S p − /2 ) ∥ R ∥ 2 . The PL inequalities for the base gap and Γ \Gamma Γ give ∥ A ∥ 2 ≥ 2 μ τ Δ 0 \lVert A\rVert^2\ge2\mu_\tau\Delta_0 ∥ A ∥ 2 ≥ 2 μ τ Δ 0 and ∥ R ∥ 2 ≥ 2 μ τ Γ \lVert R\rVert^2\ge2\mu_\tau\Gamma ∥ R ∥ 2 ≥ 2 μ τ Γ , proving (183) .
Throughout this subsection, suppress width subscripts and write Z α = ∇ α F m , α Z_\alpha=\nabla_\alpha F_{m,\alpha} Z α = ∇ α F m , α , G Z = C 1 , L G_Z=C_{1,L} G Z = C 1 , L , λ 0 = λ e f f , L \lambda_0=\lambda_{\mathrm{eff},L} λ 0 = λ eff , L and L 3 = M log , 3 , L L_3=M_{\log,3,L} L 3 = M l o g , 3 , L . The actor bounds in Appendix § 10 .§ 1 and the radius (33) give
∥ Z α ( s , a ) ∥ ≤ G Z , ∥ Z α ( s , a ) − Z b ( s , a ) ∥ ≤ C 2 , L ∥ α − b ∥ , (184) \lVert Z_\alpha(s,a)\rVert\le G_Z,\qquad\lVert Z_\alpha(s,a)-Z_b(s,a)\rVert\le C_{2,L}\lVert\alpha-b\rVert, \tag*{(184)} ∥ Z α ( s , a )∥ ≤ G Z , ∥ Z α ( s , a ) − Z b ( s , a )∥ ≤ C 2 , L ∥ α − b ∥ , (184)
G 0 : = E ρ Cov π r e f ( Z 0 ) ⪰ λ 0 I r , C 2 , L R ≤ λ 0 / 2 , (185) \mathcal{G}_0:=\mathbb{E}_{\rho}\operatorname{Cov}_{\pi_{\mathrm{ref}}}(Z_0)\succeq\lambda_0I_r,\qquad C_{2,L}R\le\sqrt{\lambda_0}/2, \tag*{(185)} G 0 := E ρ Cov π ref ( Z 0 ) ⪰ λ 0 I r , C 2 , L R ≤ λ 0 /2 , (185)
π α ( a ∣ s ) ≥ e − 2 G Z R π r e f ( a ∣ s ) , ∥ D α 3 log π α ( a ∣ s ) ∥ o p ≤ L 3 . (186) \pi_\alpha(a\mid s)\ge e^{-2G_ZR}\pi_{\mathrm{ref}}(a\mid s),\qquad\lVert D_\alpha^3\log\pi_\alpha(a\mid s)\rVert_{\mathrm{op}}\le L_3. \tag*{(186)} π α ( a ∣ s ) ≥ e − 2 G Z R π ref ( a ∣ s ) , ∥ D α 3 log π α ( a ∣ s ) ∥ op ≤ L 3 . (186)
All these inequalities hold on B ‾ R \overline{B}_R B R . In particular, the population constants already used in the exact reference-map analysis are
λ ‾ F , L = ( 1 − γ ) e − 2 G Z R λ 0 / 4 , λ ‾ F , L = 4 G Z 2 . (187) \underline{\lambda}_{F,L}=(1-\gamma)e^{-2G_ZR}\lambda_0/4,\qquad\overline{\lambda}_{F,L}=4G_Z^2. \tag*{(187)} λ F , L = ( 1 − γ ) e − 2 G Z R λ 0 /4 , λ F , L = 4 G Z 2 . (187)
The finite metric below preserves these constants, so it does not change the fixed regularization, the actor buffers or the domain required by C2. It is used only for update geometry: the population objectives, selected response and all trajectory laws for evaluation and TD retain their original definitions. Actor probabilities and derivatives are computed on finitely many states with full finite-action sums, in the same real-arithmetic model as the other neural calculations.
An empirical metric with the existing Fisher constants
Definition C.28 (One initial metric sample). Fix the realized actor initialization and retained chart before drawing metric data. For a chosen δ m e t ∈ ( 0 , 1 ) \delta_{\mathrm{met}}\in(0,1) δ met ∈ ( 0 , 1 ) , independently draw s 1 , … , s n s_1,\ldots,s_n s 1 , … , s n from ρ \rho ρ , with
n ≥ ⌈ 8 G Z 2 γ 2 λ 0 log r δ m e t ⌉ . (188) n\ge\left\lceil\frac{8G_Z^2}{\gamma^2\lambda_0}\log\frac{r}{\delta_{\mathrm{met}}}\right\rceil. \tag*{(188)} n ≥ ⌈ γ 2 λ 0 8 G Z 2 log δ met r ⌉ . (188) Define
ρ ^ n = 1 n ∑ i = 1 n δ s i , (189) \widehat{\rho}_n=\frac{1}{n}\sum_{i=1}^{n}\delta_{s_i}, \tag*{(189)} ρ n = n 1 i = 1 ∑ n δ s i , (189) B ( s ) = Cov a ∼ π r e f ( ⋅ ∣ s ) ( Z 0 ( s , a ) ) , G ^ n = 1 n ∑ i = 1 n B ( s i ) . (190) B(s)=\operatorname{Cov}_{a\sim\pi_{\mathrm{ref}}(\cdot\mid s)}(Z_0(s,a)),\qquad\widehat{\mathcal{G}}_n=\frac{1}{n}\sum_{i=1}^{n}B(s_i). \tag*{(190)} B ( s ) = Cov a ∼ π ref ( ⋅ ∣ s ) ( Z 0 ( s , a )) , G n = n 1 i = 1 ∑ n B ( s i ) . (190) Compute each covariance by summing over all actions. Accept the metric if
G ^ n ⪰ ( 1 − γ / 2 ) λ 0 I r . (191) \widehat{\mathcal{G}}_n\succeq(1-\gamma/2)\lambda_0I_r. \tag*{(191)} G n ⪰ ( 1 − γ /2 ) λ 0 I r . (191) If the certificate fails, return the prescribed fallback x 0 x_0 x 0 . Upon acceptance, use d ‾ t = ρ ^ n \overline{d}_t=\widehat{\rho}_n d t = ρ n for both endpoint updates at every iteration. All subsequent training blocks are generated using the specified sampling protocol, independently of the metric data conditional on the old parameters and history.
This is the actual initialization covariance under ρ ^ n \widehat{\rho}_n ρ n . The specified optional pilot based on two sampled actions estimates a population covariance and is a different matrix. The full action sums in (190) directly certify the empirical KL measure used here.
Proposition C.29 (Certificate acceptance probability). Conditional on any fixed initialization and chart satisfying A1–A2, Definition C.28 rejects with probability at most δ m e t \delta_{\mathrm{met}} δ met .
Proof. For every unit vector v v v ,
0 ≤ v ⊤ B ( s ) v = Var π r e f ( v ⊤ Z 0 ( s , ⋅ ) ) ≤ E π r e f ( v ⊤ Z 0 ( s , ⋅ ) ) 2 ≤ G Z 2 . 0 \le v^{\top}B(s)v = \operatorname{Var}_{\pi_{\mathrm{ref}}}(v^{\top}Z_0(s,\cdot)) \le\mathbb{E}_{\pi_{\mathrm{ref}}}(v^{\top}Z_0(s,\cdot))^2 \le G_Z^2. 0 ≤ v ⊤ B ( s ) v = Var π ref ( v ⊤ Z 0 ( s , ⋅ )) ≤ E π ref ( v ⊤ Z 0 ( s , ⋅ ) ) 2 ≤ G Z 2 .
Thus 0 ⪯ B ( s i ) ⪯ G Z 2 I r 0 \preceq B(s_i) \preceq G_Z^2 I_r 0 ⪯ B ( s i ) ⪯ G Z 2 I r , and E B ( s i ) = C 0 ⪰ λ 0 I r \mathbb{E}B(s_i)=\mathcal{C}_0 \succeq\lambda_0 I_r E B ( s i ) = C 0 ⪰ λ 0 I r . Let μ min = λ min ( ∑ i E B ( s i ) ) ≥ n λ 0 \mu_{\min}=\lambda_{\min}\left(\sum_i\mathbb{E}B(s_i)\right)\ge n\lambda_0 μ m i n = λ m i n ( ∑ i E B ( s i ) ) ≥ n λ 0 . The matrix Chernoff lower-tail inequality (Tropp, [26 ] , Corollary 5.2 and Remark 5.3) gives
P ( λ min ( ∑ i B ( s i ) ) ≤ ( 1 − t ) μ min ) ≤ r exp ( − t 2 μ min 2 G Z 2 ) , 0 < t < 1. \mathbb{P}\left(\lambda_{\min}\left(\sum_i B(s_i)\right)\le(1-t)\mu_{\min}\right)\le r\exp\left(-\frac{t^2\mu_{\min}}{2G_Z^2}\right), \qquad0<t<1. P ( λ m i n ( i ∑ B ( s i ) ) ≤ ( 1 − t ) μ m i n ) ≤ r exp ( − 2 G Z 2 t 2 μ m i n ) , 0 < t < 1.
The rejection event in (191) is contained in this event with t = γ / 2 t=\gamma/2 t = γ /2 . Hence
P ( metric rejection ) ≤ r exp ( − n γ 2 λ 0 8 G Z 2 ) ≤ δ m e t . (192) \mathbb{P}(\text{metric rejection})\le r\exp\left(-\frac{n\gamma^2\lambda_0}{8G_Z^2}\right)\le\delta_{\mathrm{met}}. \tag*{(192)} P ( metric rejection ) ≤ r exp ( − 8 G Z 2 n γ 2 λ 0 ) ≤ δ met . (192)
Proposition C.30 (Uniform empirical Fisher certificate). On the observable acceptance event, simultaneously for every α ∈ B ‾ R \alpha\in\overline{B}_R α ∈ B R
λ ‾ F , L I r ⪯ F ^ n ( α ) : = F m , ρ ^ n ( α ) ⪯ λ ‾ F , L I r . (193) \underline{\lambda}_{F,L}I_r\preceq\widehat{\mathcal{F}}_n(\alpha):=\mathcal{F}_{m,\widehat{\rho}_n}(\alpha)\preceq\overline{\lambda}_{F,L}I_r. \tag*{(193)} λ F , L I r ⪯ F n ( α ) := F m , ρ n ( α ) ⪯ λ F , L I r . (193) These are the constants in (187) . The inverse P n ( α ) = F ^ n ( α ) − 1 P_n(\alpha)=\widehat{\mathcal{F}}_n(\alpha)^{-1} P n ( α ) = F n ( α ) − 1 is uniformly bounded and Lipschitz, with bounds independent of n n n .
Proof. Fix a unit v v v , and use the joint law s ∼ ρ ^ n s\sim\widehat{\rho}_n s ∼ ρ n , a ∼ π r e f ( ⋅ ∣ s ) a\sim\pi_{\mathrm{ref}}(\cdot\mid s) a ∼ π ref ( ⋅ ∣ s ) . Conditional centering is an orthogonal projection in L 2 L^2 L 2 . Applying the triangle inequality to centered v ⊤ Z α v^{\top}Z_\alpha v ⊤ Z α and v ⊤ Z 0 v^{\top}Z_0 v ⊤ Z 0 , followed by (184) , gives
( E ρ ^ n Var π r e f ( v ⊤ Z α ) ) 1 / 2 ≥ ( E ρ ^ n Var π r e f ( v ⊤ Z 0 ) ) 1 / 2 − C 2 , L ∥ α ∥ . \left(\mathbb{E}_{\widehat{\rho}_n}\operatorname{Var}_{\pi_{\mathrm{ref}}}(v^{\top}Z_\alpha)\right)^{1/2}\ge\left(\mathbb{E}_{\widehat{\rho}_n}\operatorname{Var}_{\pi_{\mathrm{ref}}}(v^{\top}Z_0)\right)^{1/2}-C_{2,L}\|\alpha\|. ( E ρ n Var π ref ( v ⊤ Z α ) ) 1/2 ≥ ( E ρ n Var π ref ( v ⊤ Z 0 ) ) 1/2 − C 2 , L ∥ α ∥.
On acceptance, the first term on the right is at least ( 1 − γ / 2 ) λ 0 \sqrt{(1-\gamma/2)\lambda_0} ( 1 − γ /2 ) λ 0 . The right-hand side is positive on B ‾ R \overline{B}_R B R by (185) .
For any real-valued action function Y Y Y , the density lower bound in (186) implies
Var π α ( Y ) = inf a 0 ∈ R E π α ( Y − a 0 ) 2 ≥ e − 2 G Z R inf a 0 ∈ R E π r e f ( Y − a 0 ) 2 . \operatorname{Var}_{\pi_\alpha}(Y)=\inf_{a_0\in\mathbb{R}}\mathbb{E}_{\pi_\alpha}(Y-a_0)^2\ge e^{-2G_ZR}\inf_{a_0\in\mathbb{R}}\mathbb{E}_{\pi_{\mathrm{ref}}}(Y-a_0)^2. Var π α ( Y ) = a 0 ∈ R inf E π α ( Y − a 0 ) 2 ≥ e − 2 G Z R a 0 ∈ R inf E π ref ( Y − a 0 ) 2 .
Therefore
v ⊤ F ^ n ( α ) v ≥ e − 2 G Z R ( ( 1 − γ / 2 ) λ 0 − C 2 , L R ) 2 . (194) v^{\top}\widehat{\mathcal{F}}_n(\alpha)v\ge e^{-2G_ZR}\left(\sqrt{(1-\gamma/2)\lambda_0}-C_{2,L}R\right)^2. \tag*{(194)} v ⊤ F n ( α ) v ≥ e − 2 G Z R ( ( 1 − γ /2 ) λ 0 − C 2 , L R ) 2 . (194)
Set a = 1 − γ a=\sqrt{1-\gamma} a = 1 − γ . The root-mean-square inequality gives
1 − γ / 2 = 1 + a 2 2 ≥ 1 + a 2 . \sqrt{1-\gamma/2}=\sqrt{\frac{1+a^2}{2}}\ge\frac{1+a}{2}. 1 − γ /2 = 2 1 + a 2 ≥ 2 1 + a .
Combining this with (185) proves the lower bound in (193) . The centered feature has norm at most 2 G Z 2G_Z 2 G Z , proving the upper bound.
The specified log-policy derivative bounds give, for every fixed probability law ν \nu ν ,
∥ F m , ν ( α ) − F m , ν ( β ) ∥ o p ≤ L F ∥ α − β ∥ , \lVert\mathcal{F}_{m,\nu}(\alpha)-\mathcal{F}_{m,\nu}(\beta)\rVert_{\mathrm{op}} \le L_F\lVert\alpha-\beta\rVert, ∥ F m , ν ( α ) − F m , ν ( β ) ∥ op ≤ L F ∥ α − β ∥ ,
where L F L_F L F is independent of ν \nu ν . Apply this to ρ ^ n \widehat{\rho}_n ρ n , and use the identity A − 1 − B − 1 = A − 1 ( B − A ) B − 1 A^{-1}-B^{-1}=A^{-1}(B-A)B^{-1} A − 1 − B − 1 = A − 1 ( B − A ) B − 1 , to obtain
∥ P n ( α ) ∥ o p ≤ λ F , L − 1 , ∥ P n ( α ) − P n ( β ) ∥ o p ≤ L F λ F , L 2 ∥ α − β ∥ . (195) \lVert P_n(\alpha)\rVert_{\mathrm{op}} \le\lambda_{F,L}^{-1},\qquad\lVert P_n(\alpha)-P_n(\beta)\rVert_{\mathrm{op}} \le\frac{L_F}{\lambda_{F,L}^{2}}\lVert\alpha-\beta\rVert. \tag*{(195)} ∥ P n ( α ) ∥ op ≤ λ F , L − 1 , ∥ P n ( α ) − P n ( β ) ∥ op ≤ λ F , L 2 L F ∥ α − β ∥ . (195)
Remark C.31 (Why one sample works throughout training). The accepted covariance certificate implies (193) for all points of the actor ball, including later data-dependent iterates. No union bound over iterations or covering argument over actor parameters is needed. An atomic empirical measure need not dominate ( 1 − γ ) ρ (1-\gamma)\rho ( 1 − γ ) ρ ; the proof above establishes its Fisher bound directly. Nor is an approximation of the current population occupancy required: the existing update analysis uses the measure as a predictable, well-conditioned common metric.
The threshold ( 1 − γ / 2 ) λ 0 (1-\gamma/2)\lambda_0 ( 1 − γ /2 ) λ 0 is chosen to reproduce the existing constants. A half-gap threshold also gives positive curvature with suitable constants, but is not used for that purpose here. The γ − 2 \gamma^{-2} γ − 2 factor in (188) is an explicit sufficient cost of this choice; no optimal dependence on γ \gamma γ is claimed.
An explicit finite local solver
Fix accepted metric data, a buffered old actor b b b , and a frozen sampled direction h h h , with ∥ h ∥ ≤ G p r o x ( H ) \lVert h\rVert\le G_{\mathrm{prox}}(H) ∥ h ∥ ≤ G prox ( H ) . Set
K ^ n ( α ∣ b ) = 1 n ∑ i = 1 n D K L ( π b ( ⋅ ∣ s i ) ∥ π α ( ⋅ ∣ s i ) ) , (196) \widehat{K}_n(\alpha\mid b)=\frac{1}{n}\sum_{i=1}^{n}D_{\mathrm{KL}}\bigl(\pi_b(\cdot\mid s_i)\,\|\,\pi_\alpha(\cdot\mid s_i)\bigr), \tag*{(196)} K n ( α ∣ b ) = n 1 i = 1 ∑ n D KL ( π b ( ⋅ ∣ s i ) ∥ π α ( ⋅ ∣ s i ) ) , (196)
ℓ b , h ( α ) = K ^ n ( α ∣ b ) + η y − 1 ⟨ h , α − b ⟩ . (197) \ell_{b,h}(\alpha)=\widehat{K}_n(\alpha\mid b)+\eta_y^{-1}\langle h,\alpha-b\rangle. \tag*{(197)} ℓ b , h ( α ) = K n ( α ∣ b ) + η y − 1 ⟨ h , α − b ⟩ . (197)
This scaled objective has the same minimizer as the specified linear cost plus η y − 1 \eta_y^{-1} η y − 1 times KL. Every derivative in the numerical solver is with respect to α \alpha α , holding b b b , h h h , and the metric sample fixed.
The gradient is the finite sum
∇ ℓ b , h ( α ) = 1 n ∑ i = 1 n ∑ a ∈ A ( π α ( a ∣ s i ) − π b ( a ∣ s i ) ) Z α ( s i , a ) + η y − 1 h . (198) \nabla\ell_{b,h}(\alpha)=\frac{1}{n}\sum_{i=1}^{n}\sum_{a\in\mathcal{A}}\bigl(\pi_\alpha(a\mid s_i)-\pi_b(a\mid s_i)\bigr)Z_\alpha(s_i,a)+\eta_y^{-1}h. \tag*{(198)} ∇ ℓ b , h ( α ) = n 1 i = 1 ∑ n a ∈ A ∑ ( π α ( a ∣ s i ) − π b ( a ∣ s i ) ) Z α ( s i , a ) + η y − 1 h . (198)
Indeed, the derivative of KL is − ∑ a π b ( a ∣ s ) ∇ α log π α ( a ∣ s ) -\sum_a\pi_b(a\mid s)\nabla_\alpha\log\pi_\alpha(a\mid s) − ∑ a π b ( a ∣ s ) ∇ α log π α ( a ∣ s ) ; substitute the centered-feature identity from the softmax actor definition. The calculation uses actor first derivatives and finite action sums. It requires neither transition probabilities nor a population-state expectation.
Retain the local radius and step restrictions
0 < r p r o x ≤ min { R 8 , λ F , L 2 L 3 } , η y G p r o x ( H ) ≤ m K , L r p r o x 2 , (199) 0<r_{\mathrm{prox}}\le\min\left\{\frac{R}{8},\frac{\lambda_{F,L}}{2L_3}\right\},\qquad\eta_yG_{\mathrm{prox}}(H)\le\frac{m_{K,L}r_{\mathrm{prox}}}{2}, \tag*{(199)} 0 < r prox ≤ min { 8 R , 2 L 3 λ F , L } , η y G prox ( H ) ≤ 2 m K , L r prox , (199)
omitting the second radius restriction if L 3 = 0 L_3=0 L 3 = 0 , where
m K , L = λ F , L 2 , M K , L = λ ‾ F , L + L 3 r p r o x , κ K = M K , L m K , L . (200) m_{K,L}=\frac{\lambda_{F,L}}{2},\qquad M_{K,L}=\overline{\lambda}_{F,L}+L_3r_{\mathrm{prox}},\qquad\kappa_K=\frac{M_{K,L}}{m_{K,L}}. \tag*{(200)} m K , L = 2 λ F , L , M K , L = λ F , L + L 3 r prox , κ K = m K , L M K , L . (200)
Assume the local ball B b = B ‾ ( b , r p r o x ) \mathcal{B}_b=\overline{\mathcal{B}}(b,r_{\mathrm{prox}}) B b = B ( b , r prox ) lies in B ‾ R \overline{\mathcal{B}}_R B R . The specified observable safe-set test provides this buffer for both endpoint bases and their interpolation.
By Lemmas C.23 and C.24, ℓ b , h \ell_{b,h} ℓ b , h is m K , L m_{K,L} m K , L -strongly convex and M K , L M_{K,L} M K , L -smooth on B b \mathcal{B}_b B b , and its reference minimizer M ( b , h ) = M m , ρ ^ n , η y ( b , h ) M(b,h)=M_{m,\widehat{\rho}_n,\eta_y}(b,h) M ( b , h ) = M m , ρ n , η y ( b , h ) is interior, with ∇ ℓ b , h ( M ) = 0 \nabla\ell_{b,h}(M)=0 ∇ ℓ b , h ( M ) = 0 and ∥ M − b ∥ ≤ η y ∥ h ∥ / m K , L ≤ r p r o x / 2 \lVert M-b\rVert\le\eta_y\lVert h\rVert/m_{K,L}\le r_{\mathrm{prox}}/2 ∥ M − b ∥ ≤ η y ∥ h ∥ / m K , L ≤ r prox /2 . These are analysis points; the algorithm computes the finite routine below.
Definition C.32 (Residual-controlled projected gradient descent). Take 0 < c ≤ min { c 0 , 1 } 0 < c \le\min\{c_0,1\} 0 < c ≤ min { c 0 , 1 } and set
d s o l = c η y 2 , t r e s = m K , L c η y 2 . (201) d_{\mathrm{sol}}=c\eta_y^2,\qquad t_{\mathrm{res}}=m_{K,L}c\eta_y^2. \tag*{(201)} d sol = c η y 2 , t res = m K , L c η y 2 . (201) Initialize α ( 0 ) = b \alpha^{(0)}=b α ( 0 ) = b . At numerical iteration j j j , compute g ( j ) = ∇ ℓ b , h ( α ( j ) ) g^{(j)}=\nabla\ell_{b,h}(\alpha^{(j)}) g ( j ) = ∇ ℓ b , h ( α ( j ) ) from (198) . If
∥ g ( j ) ∥ ≤ t r e s , (202) \left\|g^{(j)}\right\|\le t_{\mathrm{res}}, \tag*{(202)} g ( j ) ≤ t res , (202) return M ~ ( b , h ) = α ( j ) \widetilde{M}(b,h)=\alpha^{(j)} M ( b , h ) = α ( j ) . Otherwise set
α ( j + 1 ) = Π B b ( α ( j ) − g ( j ) 2 M K , L ) (203) \alpha^{(j+1)}=\Pi_{\mathcal{B}_b}\left(\alpha^{(j)}-\frac{g^{(j)}}{2M_{K,L}}\right) \tag*{(203)} α ( j + 1 ) = Π B b ( α ( j ) − 2 M K , L g ( j ) ) (203) and continue. The Euclidean projection is explicit:
Π B b ( y ) = b + ( y − b ) min { 1 , r p r o x / ∥ y − b ∥ } , \Pi_{\mathcal{B}_b}(y)=b+(y-b)\min\{1,r_{\mathrm{prox}}/\|y-b\|\}, Π B b ( y ) = b + ( y − b ) min { 1 , r prox /∥ y − b ∥ } , with value b b b when y = b y=b y = b . The same frozen metric sample is reused at every numerical iteration.
Proposition C.33 (Finite termination and certified endpoint accuracy). Definition C.32 returns an endpoint satisfying
∥ M ~ ( b , h ) − M ( b , h ) ∥ ≤ c η y 2 . (204) \left\|\widetilde{M}(b,h)-M(b,h)\right\|\le c\eta_y^2. \tag*{(204)} M ( b , h ) − M ( b , h ) ≤ c η y 2 . (204) Define log + ( u ) = max { 0 , log u } \log_+(u)=\max\{0,\log u\} log + ( u ) = max { 0 , log u } for u > 0 u>0 u > 0 . The number of projected updates is at most
J max = ⌈ 2 κ K log + ( M K , L G p r o x ( H ) m K , L 2 c η y ) ⌉ . (205) J_{\max}=\left\lceil2\kappa_K\log_+\left(\frac{M_{K,L}G_{\mathrm{prox}}(H)}{m_{K,L}^2c\eta_y}\right)\right\rceil. \tag*{(205)} J m a x = ⌈ 2 κ K log + ( m K , L 2 c η y M K , L G prox ( H ) ) ⌉ . (205) At most J max + 1 J_{\max}+1 J m a x + 1 finite-sum gradients are evaluated.
Proof. Let M = M ( b , h ) M=M(b,h) M = M ( b , h ) . For α ∈ B b \alpha\in\mathcal{B}_b α ∈ B b , the integral Hessian along the segment from M M M to α \alpha α has eigenvalues in [ m K , L , M K , L ] [m_{K,L},M_{K,L}] [ m K , L , M K , L ] . Since ∇ ℓ b , h ( M ) = 0 \nabla\ell_{b,h}(M)=0 ∇ ℓ b , h ( M ) = 0 , the unprojected gradient map with step 1 / ( 2 M K , L ) 1/(2M_{K,L}) 1/ ( 2 M K , L ) contracts distance to M M M by at most
q K = 1 − m K , L 2 M K , L . q_K=1-\frac{m_{K,L}}{2M_{K,L}}. q K = 1 − 2 M K , L m K , L .
Projection onto the convex ball is nonexpansive and fixes M M M . Consequently,
∥ α ( j ) − M ∥ ≤ q K j ∥ b − M ∥ ≤ e − j / ( 2 κ K ) η y G p r o x ( H ) m K , L . (206) \left\|\alpha^{(j)}-M\right\|\le q_K^j\|b-M\|\le e^{-j/(2\kappa_K)}\frac{\eta_yG_{\mathrm{prox}}(H)}{m_{K,L}}. \tag*{(206)} α ( j ) − M ≤ q K j ∥ b − M ∥ ≤ e − j / ( 2 κ K ) m K , L η y G prox ( H ) . (206)
Local smoothness gives
∥ ∇ ℓ b , h ( α ( j ) ) ∥ ≤ M K , L ∥ α ( j ) − M ∥ ≤ M K , L η y G p r o x ( H ) m K , L e − j / ( 2 κ K ) . \left\|\nabla\ell_{b,h}(\alpha^{(j)})\right\|\le M_{K,L}\left\|\alpha^{(j)}-M\right\|\le\frac{M_{K,L}\eta_yG_{\mathrm{prox}}(H)}{m_{K,L}}e^{-j/(2\kappa_K)}. ∇ ℓ b , h ( α ( j ) ) ≤ M K , L α ( j ) − M ≤ m K , L M K , L η y G prox ( H ) e − j / ( 2 κ K ) .
This is at most m K , L c η y 2 m_{K,L}c\eta_y^2 m K , L c η y 2 by j = J max j=J_{\max} j = J m a x . Finally, strong monotonicity and the zero gradient at M M M imply
m K , L ∥ α − M ∥ 2 ≤ ⟨ ∇ ℓ b , h ( α ) , α − M ⟩ ≤ ∥ ∇ ℓ b , h ( α ) ∥ ∥ α − M ∥ . m_{K,L}\|\alpha-M\|^2\le\langle\nabla\ell_{b,h}(\alpha),\alpha-M\rangle\le\|\nabla\ell_{b,h}(\alpha)\|\|\alpha-M\|. m K , L ∥ α − M ∥ 2 ≤ ⟨ ∇ ℓ b , h ( α ) , α − M ⟩ ≤ ∥∇ ℓ b , h ( α ) ∥∥ α − M ∥.
At the stopping point, this yields (204) . All iterates stay in the local ball by projection, so the proof does not presuppose actor containment beyond the current buffered step.
Remark C.34 (Normalization and arithmetic). The stopping test is for the scaled objective (197) . If implemented using ⟨ h , α − b ⟩ + η y − 1 K ^ n n ( α ∣ b ) \langle h,\alpha-b\rangle+\eta_y^{-1}\widehat{K}_{\mathrm{nn}}(\alpha\mid b) ⟨ h , α − b ⟩ + η y − 1 K nn ( α ∣ b ) , the equivalent gradient-residual threshold is m K , L c η y m_{K,L}c\eta_y m K , L c η y . The parameter tolerance c η y 2 c\eta_y^2 c η y 2 is not an objective-value tolerance. With c = Θ ( ϵ ) c=\Theta(\sqrt{\epsilon}) c = Θ ( ϵ ) and η y = Θ ( ϵ ) \eta_y=\Theta(\epsilon) η y = Θ ( ϵ ) , it is O ( ϵ 5 / 2 ) O(\epsilon^{5/2}) O ( ϵ 5/2 ) ; geometric convergence gives a logarithmic iteration dependence on this accuracy. Neither the stopping rule nor the count above assumes an accurate optimization oracle. Gradient evaluations are the explicit finite neural computations in (198) .
Paired numerical errors and the sampling history
At an old iterate p = ( x , U , S ) p=(x,U,S) p = ( x , U , S ) , use the specified frozen lower estimates ( a ^ , R ^ ) (\hat{a},\hat{R}) ( a ^ , R ^ ) , and form C ^ = a ^ + c R ^ \hat{C}=\hat{a}+c\hat{R} C ^ = a ^ + c R ^ . The two bases and directions are
b 0 = U , h 0 = a ^ , b c = U + c S , h c = a ^ + c R ^ . b_0=U,\quad h_0=\hat{a},\qquad b_c=U+cS,\quad h_c=\hat{a}+c\hat{R}. b 0 = U , h 0 = a ^ , b c = U + c S , h c = a ^ + c R ^ .
Let M 0 = M ( b 0 , h 0 ) M_0=M(b_0,h_0) M 0 = M ( b 0 , h 0 ) , M c = M ( b c , h c ) M_c=M(b_c,h_c) M c = M ( b c , h c ) . These are mathematical reference points used only in the proof. The implemented updates are
U + = M ~ ( b 0 , h 0 ) , α c + = M ~ ( b c , h c ) , S + = α c + − U + c . (207) U^+=\widetilde{M}(b_0,h_0),\qquad\alpha_c^+=\widetilde{M}(b_c,h_c),\qquad S^+=\frac{\alpha_c^+-U^+}{c}. \tag*{(207)} U + = M ( b 0 , h 0 ) , α c + = M ( b c , h c ) , S + = c α c + − U + . (207)
Lemma C.35 (Samplewise paired error). At the same old iterate and with the same sampled directions, define the virtual exact update U ˉ + = M 0 \bar{U}^+=M_0 U ˉ + = M 0 , S ˉ + = ( M c − M 0 ) / c \bar{S}^+=(M_c-M_0)/c S ˉ + = ( M c − M 0 ) / c . Then, samplewise,
e U : = U + − U ˉ + , ∥ e U ∥ ≤ c η y 2 , (208) e_U:=U^+-\bar{U}^+,\qquad\lVert e_U\rVert\le c\eta_y^2, \tag*{(208)} e U := U + − U ˉ + , ∥ e U ∥ ≤ c η y 2 , (208) e S : = S + − S ˉ + , ∥ e S ∥ ≤ 2 η y 2 . (209) e_S:=S^+-\bar{S}^+,\qquad\lVert e_S\rVert\le2\eta_y^2. \tag*{(209)} e S := S + − S ˉ + , ∥ e S ∥ ≤ 2 η y 2 . (209) In particular, for c ≤ 1 c\le1 c ≤ 1 ,
∥ ( e U , e S ) ∥ ≤ 5 η y 2 . (210) \lVert(e_U,e_S)\rVert\le\sqrt{5}\,\eta_y^2. \tag*{(210)} ∥( e U , e S )∥ ≤ 5 η y 2 . (210) No differentiability of the stopping rule or coupling of numerical stopping times is required.
Proof. Set e 0 = M ~ ( b 0 , h 0 ) − M 0 e_0=\widetilde{M}(b_0,h_0)-M_0 e 0 = M ( b 0 , h 0 ) − M 0 and e c = M ~ ( b c , h c ) − M c e_c=\widetilde{M}(b_c,h_c)-M_c e c = M ( b c , h c ) − M c . Proposition C.33 gives ∥ e 0 ∥ , ∥ e c ∥ ≤ c η y 2 \lVert e_0\rVert,\lVert e_c\rVert\le c\eta_y^2 ∥ e 0 ∥ , ∥ e c ∥ ≤ c η y 2 . The identities e U = e 0 e_U=e_0 e U = e 0 and e S = ( e c − e 0 ) / c e_S=(e_c-e_0)/c e S = ( e c − e 0 ) / c prove the first two bounds. Their squared sum is at most ( c 2 + 4 ) η y 4 (c^2+4)\eta_y^4 ( c 2 + 4 ) η y 4 . □
For completeness, the exact reference-map expansion from Lemma C.25 therefore yields the implemented expansion
U + − U = − η y P 0 a ^ + ρ U + e U , (211) U^+-U=-\eta_yP^0\hat{a}+\rho_U+e_U, \tag*{(211)} U + − U = − η y P 0 a ^ + ρ U + e U , (211)
S + − S = − η y P c R ^ + η y K c a ^ + ρ S + e S , (212) S^+-S=-\eta_yP^c\hat{R}+\eta_yK_c\hat{a}+\rho_S+e_S, \tag*{(212)} S + − S = − η y P c R ^ + η y K c a ^ + ρ S + e S , (212)
P 0 = P n ( U ) , P c = P n ( U + c S ) , K c = P 0 − P c c . P^0=P_n(U),\qquad P^c=P_n(U+cS),\qquad K_c=\frac{P^0-P^c}{c}. P 0 = P n ( U ) , P c = P n ( U + c S ) , K c = c P 0 − P c .
The original remainders ρ U , ρ S \rho_U,\rho_S ρ U , ρ S have conditional first moments O ( η y 2 ) O(\eta_y^2) O ( η y 2 ) , second moments O ( η y 4 ) O(\eta_y^4) O ( η y 4 ) , and fourth moments O ( η y 8 ) O(\eta_y^8) O ( η y 8 ) . Adding ( e U , e S ) (e_U,e_S) ( e U , e S ) preserves these orders: its corresponding bounds follow immediately from (210) . The normalized inverse-metric difference is still bounded by ∥ K c ∥ o p ≤ L F λ F , L 2 ∥ S ∥ \lVert K_c\rVert_{\mathrm{op}}\le L_F\lambda_{F,L}^2\lVert S\rVert ∥ K c ∥ op ≤ L F λ F , L 2 ∥ S ∥ .
Lemma C.36 (Filtration and prospective-step validity). Include the entire initial metric sample in the pre-draw history F t \mathcal{F}_t F t . Use old parameters for all three training blocks, and run both local solvers solely with the frozen lower block and metric data. Then the simultaneous ( x , U , S ) (x,U,S) ( x , U , S ) increment remains conditionally independent of the current TD block. The conditional movement estimates of Appendix § 10 .7 retain their orders. For sufficiently small accuracy in the headline schedule, the implemented prospective step lies in the existing larger analytic actor neighborhood, before containment has been established.
Proof. Each numerical iterate and its stopping index are measurable functions of the old history and lower block. The outer update uses its independent old-parameter block. The TD block is independent of these two blocks conditional on F t \mathcal{F}_t F t , as stipulated in the existing protocol. Hence the conditional centering argument for moving-target TD applies. The mean and centered numerical perturbations have orders O ( η y 2 ) O(\eta_y^2) O ( η y 2 ) , with second and fourth moment bounds supplied by (210) ; these preserve the movement estimates.
At a safe old point, both local balls and the corresponding balls along the endpoint interpolation lie in B ‾ R \overline{B}_R B R . The projected inner iterations stay in their respective balls. The virtual exact coupled increments have deterministic bound C η y poly ( H ) C\eta_y\operatorname{poly}(H) C η y poly ( H ) by the specified paired expansion. The implemented increments add at most 5 η y 2 \sqrt{5}\eta_y^2 5 η y 2 . With logarithmic H H H and η y = Θ ( ϵ ) \eta_y=\Theta(\epsilon) η y = Θ ( ϵ ) , both tend to zero. In particular, the existing prospective restrictions ∥ Δ U ∥ < r p r o x / 2 \|\Delta U\|<r_{\mathrm{prox}}/2 ∥Δ U ∥ < r prox /2 and ∥ Δ S ∥ < s / 4 \|\Delta S\|<s/4 ∥Δ S ∥ < s /4 hold for sufficiently small ϵ \epsilon ϵ . The virtual and implemented points, and the segment between them, lie in the same convex buffered analytic neighborhood. This argument uses only the current safe point and therefore introduces no circular appeal to future containment.
Transfer to tracking, containment, and stationarity
Use the specified actor potential T ( p ) = Δ 0 ( x , U ) + λ S Γ c ( x , U , S ) \mathcal{T}(p)=\Delta_0(x,U)+\lambda_S\Gamma_c(x,U,S) T ( p ) = Δ 0 ( x , U ) + λ S Γ c ( x , U , S ) . To distinguish the moving critic target from the actor feature Z α Z_\alpha Z α , write that target as Z ( p ) \mathcal{Z}(p) Z ( p ) , and set
C ( p , z ) = ∥ z − Z ( p ) ∥ W 2 , X ( p ) = ∥ ∇ Φ ( x ) ∥ 2 , ω = c 2 + A Q + b e v 2 + b T D 2 . \mathcal{C}(p,z)=\|z-\mathcal{Z}(p)\|_W^2,\qquad X(p)=\|\nabla\Phi(x)\|^2,\qquad\omega=c^2+A_Q+b_{\mathrm{ev}}^2+b_{\mathrm{TD}}^2. C ( p , z ) = ∥ z − Z ( p ) ∥ W 2 , X ( p ) = ∥∇Φ ( x ) ∥ 2 , ω = c 2 + A Q + b ev 2 + b TD 2 .
The established bounds on the prospective neighborhood include
∥ ∇ p T ∥ ≤ L T , ∥ Z ( p ) − Z ( p ˉ ) ∥ W ≤ L Z ∥ p − p ˉ ∥ , Z ( p ) , z ∈ K Q . (213) \|\nabla_p\mathcal{T}\|\le L_T,\qquad\|\mathcal{Z}(p)-\mathcal{Z}(\bar p)\|_W\le L_Z\|p-\bar p\|,\qquad\mathcal{Z}(p),z\in\mathcal{K}_Q. \tag*{(213)} ∥ ∇ p T ∥ ≤ L T , ∥ Z ( p ) − Z ( p ˉ ) ∥ W ≤ L Z ∥ p − p ˉ ∥ , Z ( p ) , z ∈ K Q . (213)
These constants are global in the upper parameter and uniform in the critic scale and c c c . Put D Q = diam W ( K Q ) < ∞ D_Q=\operatorname{diam}_W(\mathcal{K}_Q)<\infty D Q = diam W ( K Q ) < ∞ .
Proposition C.37 (One-step transfer without new accuracy assumptions). At a safe old iterate, compare the implemented step with the virtual exact-proximal step at the same old iterate, using identical lower, outer, and TD blocks. The outer and critic updates themselves agree, and the next actor/upper tuples p + , p ˉ + p^+,\bar p^+ p + , p ˉ + satisfy
x + = x ˉ + , z + = z ˉ + , ∥ p + − p ˉ + ∥ ≤ 5 η y 2 . x^+=\bar{x}^+,\qquad z^+=\bar{z}^+,\qquad\|p^+-\bar{p}^+\|\le\sqrt{5}\eta_y^2. x + = x ˉ + , z + = z ˉ + , ∥ p + − p ˉ + ∥ ≤ 5 η y 2 . Consequently,
∣ T ( p + ) − T ( p ˉ + ) ∣ ≤ 5 L T η y 2 , (214) |\mathcal{T}(p^+)-\mathcal{T}(\bar p^+)|\le\sqrt{5}L_T\eta_y^2, \tag*{(214)} ∣ T ( p + ) − T ( p ˉ + ) ∣ ≤ 5 L T η y 2 , (214) ∣ C ( p + , z + ) − C ( p ˉ + , z + ) ∣ ≤ 2 5 D Q L Z η y 2 , (215) |\mathcal{C}(p^+,z^+)-\mathcal{C}(\bar p^+,z^+)|\le2\sqrt{5}D_QL_Z\eta_y^2, \tag*{(215)} ∣ C ( p + , z + ) − C ( p ˉ + , z + ) ∣ ≤ 2 5 D Q L Z η y 2 , (215) Φ ( x + ) − Φ ( x ˉ + ) = 0. (216) \Phi(x^+)-\Phi(\bar{x}^+)=0. \tag*{(216)} Φ ( x + ) − Φ ( x ˉ + ) = 0. (216) The existing conditional actor/critic tracking recursions and outer descent inequality thus retain their form, with enlarged second-order remainder constants.
Proof. Both the outer and TD routines use the old parameters, so using the same sampled blocks gives identical x + x^+ x + and z + z^+ z + . Lemma C.35 bounds the remaining difference. Integrating the actor-potential gradient along the segment supplied by Lemma C.36 gives (214) . For the critic error, both z + − Z ( p + ) z^+-\mathcal{Z}(p^+) z + − Z ( p + ) and z + − Z ( p ˉ + ) z^+-\mathcal{Z}(\bar p^+) z + − Z ( p ˉ + ) have W W W -norm at most D Q D_Q D Q . The difference of their squared norms is at most
2 D Q ∥ Z ( p + ) − Z ( p ˉ + ) ∥ W , 2D_Q\|\mathcal{Z}(p^+)-\mathcal{Z}(\bar p^+)\|_W, 2 D Q ∥ Z ( p + ) − Z ( p ˉ + ) ∥ W ,
which proves (215) . The last identity is immediate.
The specified exact-map inequalities are one-step inequalities uniform over safe old iterates. They can therefore be applied to the virtual step at each implemented iterate; a separate exact-solver trajectory is unnecessary. Take conditional expectations in the displayed samplewise bounds. Each additional actor or critic term is O ( η y 2 ) O(\eta_y^2) O ( η y 2 ) , already present in the corresponding recurrence. The outer descent inequality is unchanged.
In particular, the transferred recursions have the existing orders
E t Δ C t ≤ − k β C t + C η y 2 β T t + C η x 2 β X t + C β ω + C ( β 2 + η y 2 + η x 2 ) , (217) \mathbb{E}_t\Delta C_t \le-k\beta C_t + C\frac{\eta_y^2}{\beta}T_t + C\frac{\eta_x^2}{\beta}X_t + C\beta\omega+ C(\beta^2+\eta_y^2+\eta_x^2), \tag*{(217)} E t Δ C t ≤ − k β C t + C β η y 2 T t + C β η x 2 X t + C β ω + C ( β 2 + η y 2 + η x 2 ) , (217)
E t Δ T t ≤ − u η y T t + v η x X t + w η y C t + C η y ω + C ( η y 2 + η x 2 ) , (218) \mathbb{E}_t\Delta T_t \le-u\eta_yT_t + v\eta_xX_t + w\eta_yC_t + C\eta_y\omega+ C(\eta_y^2+\eta_x^2), \tag*{(218)} E t Δ T t ≤ − u η y T t + v η x X t + w η y C t + C η y ω + C ( η y 2 + η x 2 ) , (218)
E t Δ Φ t ≤ − a η x X t + b η x T t + d η x C t + C η x ω + C η x 2 . (219) \mathbb{E}_t\Delta\Phi_t \le-a\eta_xX_t + b\eta_xT_t + d\eta_xC_t + C\eta_x\omega+ C\eta_x^2. \tag*{(219)} E t Δ Φ t ≤ − a η x X t + b η x T t + d η x C t + C η x ω + C η x 2 . (219)
Here the positive drift constants and sufficiently small fixed step-ratio choices are those furnished by the specified proof; the numerical perturbation enters the indicated remainder terms. In particular, it does not create a new c − 1 c^{-1} c − 1 error floor.
Corollary C.38 (Containment and augmented descent). Let η y = r Q β \eta_y=r_Q\beta η y = r Q β , η x = q η y \eta_x=q\eta_y η x = q η y , with the sufficiently small fixed positive ratios selected in the analysis. The contracting potential D = T + λ D r Q C D=T+\lambda_Dr_QC D = T + λ D r Q C and stationarity potential
Ψ = Φ − Φ ‾ + a Ψ q T + b Ψ q r Q C \Psi=\Phi-\overline{\Phi}+a_\Psi qT+b_\Psi qr_QC Ψ = Φ − Φ + a Ψ q T + b Ψ q r Q C retain their required drift and innovation bounds. In particular, the new contribution to the one-step drift of Ψ \Psi Ψ is O ( η x η y ) O(\eta_x\eta_y) O ( η x η y ) , and the added contribution to the drift of D D D is O ( η y 2 ) O(\eta_y^2) O ( η y 2 ) .
Proof. For Ψ \Psi Ψ , multiply (214) by a Ψ q a_\Psi q a Ψ q and (215) by b Ψ q r Q b_\Psi qr_Q b Ψ q r Q . Since r Q ≤ 1 r_Q\le1 r Q ≤ 1 and q η y = η x q\eta_y=\eta_x q η y = η x , the extra term is bounded by C q η y 2 = C η x η y Cq\eta_y^2=C\eta_x\eta_y C q η y 2 = C η x η y . It is absorbed in the existing C η x ( ω + β + η y + η x ) C\eta_x(\omega+\beta+\eta_y+\eta_x) C η x ( ω + β + η y + η x ) remainder. For D D D , the additional term is C η y 2 C\eta_y^2 C η y 2 . After division by η y \eta_y η y , it is C η y ≤ C β C\eta_y\le C\beta C η y ≤ C β , again within the existing contracting floor. The difference between the implemented next D D D and the virtual exact next D D D is samplewise O ( η y 2 ) O(\eta_y^2) O ( η y 2 ) . Centering at most doubles this bound. The original innovation bound ∣ N t + 1 ∣ ≤ η y B D ( H ) |N_{t+1}|\le\eta_yB_D(H) ∣ N t + 1 ∣ ≤ η y B D ( H ) therefore remains valid after enlarging its polynomial envelope. Together with Lemma C.36 , the same stopped martingale maximum argument and first-exit contradiction apply.
Critic calculus and auxiliary TD estimates
The critic in (6) trains a fixed-dimensional chart θ \theta θ . Increasing m Q m_Q m Q changes its nonlinear scale, not its number of independent trainable coordinates. For integer m Q m_Q m Q , tied tanh units give this realization exactly.
Lemma C.39 (Nonlinear critic derivatives and linearization). On ∥ θ ∥ ≤ R Q \|\theta\|\le R_Q ∥ θ ∥ ≤ R Q , the following bounds hold uniformly over inputs:
∣ Q m Q , θ ∣ ≤ F Q R Q , ∥ ∇ Q m Q , θ ∥ ≤ F Q , |Q_{m_Q,\theta}|\le F_QR_Q,\qquad\|\nabla Q_{m_Q,\theta}\|\le F_Q, ∣ Q m Q , θ ∣ ≤ F Q R Q , ∥∇ Q m Q , θ ∥ ≤ F Q , ∥ Q m Q , θ − φ Q T θ ∥ ∞ ≤ F Q 3 R Q 3 3 m Q , ∥ ∇ Q m Q , θ − φ Q ∥ ∞ ≤ F Q 3 R Q 2 m Q , \|Q_{m_Q,\theta}-\varphi_Q^\mathsf{T}\theta\|_\infty\le\frac{F_Q^3R_Q^3}{3m_Q},\qquad\|\nabla Q_{m_Q,\theta}-\varphi_Q\|_\infty\le\frac{F_Q^3R_Q^2}{m_Q}, ∥ Q m Q , θ − φ Q T θ ∥ ∞ ≤ 3 m Q F Q 3 R Q 3 , ∥∇ Q m Q , θ − φ Q ∥ ∞ ≤ m Q F Q 3 R Q 2 , ∥ ∇ 2 Q m Q , θ ∥ ∞ ≤ 2 F Q 2 R Q m Q , ∥ ∇ 3 Q m Q , θ ∥ ∞ ≤ 2 F Q 2 m Q . (220) \|\nabla^2Q_{m_Q,\theta}\|_\infty\le\frac{2F_Q^2R_Q}{m_Q},\qquad\|\nabla^3Q_{m_Q,\theta}\|_\infty\le\frac{2F_Q^2}{m_Q}. \tag*{(220)} ∥ ∇ 2 Q m Q , θ ∥ ∞ ≤ m Q 2 F Q 2 R Q , ∥ ∇ 3 Q m Q , θ ∥ ∞ ≤ m Q 2 F Q 2 . (220) Fourth derivatives are also uniformly bounded for m Q ≥ 1 m_Q\ge1 m Q ≥ 1 .
Proof. Put t = φ Q T θ / m Q t=\varphi_Q^\mathsf{T}\theta/\sqrt{m_Q} t = φ Q T θ / m Q . Direct differentiation gives ∇ Q = sech 2 ( t ) φ Q \nabla Q=\operatorname{sech}^2(t)\varphi_Q ∇ Q = sech 2 ( t ) φ Q and ∇ 2 Q = − 2 sech 2 ( t ) tanh ( t ) φ Q ⊗ 2 / m Q \nabla^2Q=-2\operatorname{sech}^2(t)\tanh(t)\varphi_Q^{\otimes2}/\sqrt{m_Q} ∇ 2 Q = − 2 sech 2 ( t ) tanh ( t ) φ Q ⊗ 2 / m Q . Use ∣ tanh t ∣ ≤ ∣ t ∣ |\tanh t|\le|t| ∣ tanh t ∣ ≤ ∣ t ∣ , ∣ tanh t − t ∣ ≤ ∣ t ∣ 3 / 3 |\tanh t-t|\le|t|^3/3 ∣ tanh t − t ∣ ≤ ∣ t ∣ 3 /3 , and 1 − sech 2 t ≤ t 2 1-\operatorname{sech}^2t\le t^2 1 − sech 2 t ≤ t 2 . The third derivative is ( 4 sech 2 t − 6 sech 4 t ) φ Q ⊗ 3 / m Q (4\operatorname{sech}^2t-6\operatorname{sech}^4t)\varphi_Q^{\otimes3}/m_Q ( 4 sech 2 t − 6 sech 4 t ) φ Q ⊗ 3 / m Q , whose scalar coefficient has magnitude at most two. The fourth derivative is a bounded polynomial in tanh t \tanh t tanh t and sech t \operatorname{sech}t sech t , multiplied by φ Q ⊗ 4 / m Q 3 / 2 \varphi_Q^{\otimes4}/m_Q^{3/2} φ Q ⊗ 4 / m Q 3/2 . These formulas prove all claims.
Lemma C.40 (Anisotropic derivative bounds on unrestricted x x x ). Regard ( U , α , θ ) (U,\alpha,\theta) ( U , α , θ ) as independent variables and denote this group by y y y . On the indicated neighborhoods, every derivative block
D x a D y b H ( x , U , α , θ ) , 0 ≤ a ≤ 2 , a + b ≤ 3 , (221) D_x^aD_y^b\mathsf{H}(x,U,\alpha,\theta),\qquad0\le a\le2,\quad a+b\le3, \tag*{(221)} D x a D y b H ( x , U , α , θ ) , 0 ≤ a ≤ 2 , a + b ≤ 3 , (221)
is uniformly bounded. Derivatives of H ^ \hat{H} H ^ at a fixed transition, holding its sampling law fixed, have uniform envelopes of the corresponding orders. Derivatives of ν U \nu_U ν U are treated separately by likelihood differentiation and geometric moments; their prefix likelihood factors have polynomial length envelopes. Pure D x 3 H D_x^3H D x 3 H is not needed or claimed uniformly bounded.
Proof. The base occupancy is independent of x x x and α \alpha α . Its U U U derivatives through order three are controlled by differentiating the prefix likelihood under π U \pi_U π U . Bounded derivatives of the log-policy give degree-at-most-three polynomials in prefix length. The sums ∑ k ≥ 0 ( 1 − γ ) γ k ( k + 1 ) j \sum_{k\geq0}(1-\gamma)\gamma^k(k+1)^j ∑ k ≥ 0 ( 1 − γ ) γ k ( k + 1 ) j are finite. This proves differentiation under the discounted sum in total variation and supplies uniform bounds for those derivatives.
For the TD integrand, an x x x derivative acts only on q = − r x + τ log ( π α / π r e f ) q=-r_x+\tau\log(\pi_\alpha/\pi_{\mathrm{ref}}) q = − r x + τ log ( π α / π ref ) . Thus D x a q = − D x a r x D_x^a q=-D_x^a r_x D x a q = − D x a r x for a = 1 , 2 a=1,2 a = 1 , 2 , and D x a D α b q = 0 D_x^aD_\alpha^bq=0 D x a D α b q = 0 when a , b ≥ 1 a,b\geq1 a , b ≥ 1 . Actor derivatives act on K α K_\alpha K α , the entropy criterion, or the base likelihood. Three derivatives in θ \theta θ of the TD integrand use derivatives of Q Q Q through order four, uniformly bounded by (220) and its fourth-order analogue. Product differentiation and (14) give every block in (221) .
For example, at independent ( U , α , θ ) (U,\alpha,\theta) ( U , α , θ ) , H x x = E ν U [ r x x ∇ Q ] H_{xx}=\mathbb{E}_{\nu_U}[r_{xx}\nabla Q] H xx = E ν U [ r xx ∇ Q ] , H x x α = 0 H_{xx\alpha}=0 H xxα = 0 , and H x x θ = E ν U [ r x x ∇ 2 Q ] H_{xx\theta}=\mathbb{E}_{\nu_U}[r_{xx}\nabla^2Q] H xx θ = E ν U [ r xx ∇ 2 Q ] ; H x x U H_{xxU} H xxU differentiates only the base law in the first expression. These are the potentially delicate blocks needed after taking the normalized endpoint difference. No pure third x x x derivative enters. The same domination proves continuity of these derivatives, not merely a norm bound. □ \square □
Sampling routines and paired oracle estimates
All blocks use the pre-draw history and old parameters specified at the start of Appendix § 9 , including the stored accepted metric sample. The reset states used for that metric are independent of these training blocks and are reused only for the finite KL computations. Write M 1 = M l o g , 1 , L M_1=M_{\mathrm{log},1,L} M 1 = M log , 1 , L . Define the sample TD residual at a common transition by
H ^ ( x , U , α , θ ) = [ Q m Q , θ ( s , a ) − q x , α ( s , a ) − γ ∑ a ′ π α ( a ′ ∣ s ′ ) Q m Q , θ ( s ′ , a ′ ) ] ∇ θ Q m Q , θ ( s , a ) , \hat{H}(x,U,\alpha,\theta)=\left[Q_{m_Q,\theta}(s,a)-q_{x,\alpha}(s,a)-\gamma\sum_{a'}\pi_\alpha(a'\mid s')Q_{m_Q,\theta}(s',a')\right]\nabla_\theta Q_{m_Q,\theta}(s,a), H ^ ( x , U , α , θ ) = [ Q m Q , θ ( s , a ) − q x , α ( s , a ) − γ a ′ ∑ π α ( a ′ ∣ s ′ ) Q m Q , θ ( s ′ , a ′ ) ] ∇ θ Q m Q , θ ( s , a ) ,
F s a m p ( p , z ; s , a , s ′ ) = ( H ^ ( x , U , U , u ) , H ^ ( x , U , U + c S , u + c v ) − H ^ ( x , U , U , u ) c ) . (222) F^{\mathrm{samp}}(p,z;s,a,s')=\left(\hat{H}(x,U,U,u),\frac{\hat{H}(x,U,U+cS,u+cv)-\hat{H}(x,U,U,u)}{c}\right). \tag*{(222)} F samp ( p , z ; s , a , s ′ ) = ( H ^ ( x , U , U , u ) , c H ^ ( x , U , U + c S , u + c v ) − H ^ ( x , U , U , u ) ) . (222)
Here q x , α q_{x,\alpha} q x , α is the regularized lower criterion in (15) . Both endpoints use the same base-law transition; the next-action expectations are exact finite sums. With F ^ t , H \hat{F}_{t,H} F ^ t , H the truncated sample average, update
z t + 1 = Π K Q W ( z t − β F ^ t , H ) , K Q = B ‾ R u × B ‾ R v , ∥ ( u , v ) ∥ W 2 = M ∥ u ∥ 2 + ∥ v ∥ 2 . (223) z_{t+1}=\Pi^W_{K_Q}(z_t-\beta\hat{F}_{t,H}),\qquad K_Q=\overline{B}_{R_u}\times\overline{B}_{R_v},\qquad\|(u,v)\|_W^2=M\|u\|^2+\|v\|^2. \tag*{(223)} z t + 1 = Π K Q W ( z t − β F ^ t , H ) , K Q = B R u × B R v , ∥ ( u , v ) ∥ W 2 = M ∥ u ∥ 2 + ∥ v ∥ 2 . (223)
The critic construction in Step B.2 supplies fixed R u , R v , M R_u,R_v,M R u , R v , M and an admissible critic scale. Projection is a specified critic update; critic accuracy is proved. Initialize any deterministic z 0 ∈ K Q z_0\in K_Q z 0 ∈ K Q , without an accuracy or burn-in requirement. Both algorithms use the single actor test (66) , initially and after every completed iteration.
Explicit evaluation samplers. Write α 0 = U \alpha_0=U α 0 = U , α c = U + c S \alpha_c=U+cS α c = U + c S , θ 0 = u \theta_0=u θ 0 = u , θ c = u + c v \theta_c=u+cv θ c = u + c v , and Q j = Q m Q , θ j Q_j=Q_{m_Q,\theta_j} Q j = Q m Q , θ j . The critic represents the unnormalized discounted action value, whereas g τ g_\tau g τ is normalized by 1 − γ 1-\gamma 1 − γ . Consequently the following score-times-value actor estimator has no additional ( 1 − γ ) − 1 (1-\gamma)^{-1} ( 1 − γ ) − 1 factor. For a common-start mixture prefix ξ = ( s 0 , a 0 , … , s n , a n ) \xi=(s_0,a_0,\ldots,s_n,a_n) ξ = ( s 0 , a 0 , … , s n , a n ) with s 0 ∼ ρ s_0\sim\rho s 0 ∼ ρ , set
ℓ j ( ξ ) = ∏ k = 0 n π α j ( a k ∣ s k ) , w j ( ξ ) = 2 ℓ j ( ξ ) ℓ 0 ( ξ ) + ℓ c ( ξ ) , j ∈ { 0 , c } . (224) \ell_j(\xi)=\prod_{k=0}^{n}\pi_{\alpha_j}(a_k\mid s_k),\qquad
w_j(\xi)=\frac{2\ell_j(\xi)}{\ell_0(\xi)+\ell_c(\xi)},\qquad
j\in\{0,c\}. \tag*{(224)} ℓ j ( ξ ) = k = 0 ∏ n π α j ( a k ∣ s k ) , w j ( ξ ) = ℓ 0 ( ξ ) + ℓ c ( ξ ) 2 ℓ j ( ξ ) , j ∈ { 0 , c } . (224)
The terminal action is included. Shared reset and transition factors cancel in this ratio. At the same terminal state-action pair define
G j ( ξ ) = ∇ α log π α j ( a n ∣ s n ) Q j ( s n , a n ) + ∂ α q x , α j ( s n , a n ) , a ( ξ ) = w 0 G 0 , d ( ξ ) = w c G c − w 0 G 0 c , e ( ξ ) = − w c − w 0 c ∇ x r x ( s n , a n ) . (225) \begin{aligned}
G_j(\xi) &= \nabla_{\alpha} \log\pi_{\alpha_j}(a_n \mid s_n)Q_j(s_n,a_n) + \partial_{\alpha}q_{x,\alpha_j}(s_n,a_n),\\
a(\xi) &= w_0G_0,\qquad d(\xi) = \frac{w_cG_c-w_0G_0}{c},\qquad e(\xi) = -\frac{w_c-w_0}{c}\nabla_xr_x(s_n,a_n). \tag*{(225)}
\end{aligned} G j ( ξ ) a ( ξ ) = ∇ α log π α j ( a n ∣ s n ) Q j ( s n , a n ) + ∂ α q x , α j ( s n , a n ) , = w 0 G 0 , d ( ξ ) = c w c G c − w 0 G 0 , e ( ξ ) = − c w c − w 0 ∇ x r x ( s n , a n ) . (225)
The direct criterion derivative ∂ α q \partial_{\alpha}q ∂ α q differentiates the policy-ratio criterion, not the trained critic coordinates. For a fresh preference tuple and label at the old perturbed actor,
P α ( ξ , Y ) = L ( x ; ξ , Y ) ∑ i = 1 I ∑ k = 0 K − 1 ∇ α log π α c ( a k i ∣ s k i ) , P x ( ξ , Y ) = ∇ x L ( x ; ξ , Y ) . (226) P_{\alpha}(\xi,Y) = L(x;\xi,Y)\sum_{i=1}^{I}\sum_{k=0}^{K-1}\nabla_{\alpha}\log\pi_{\alpha_c}(a_k^i\mid s_k^i),\qquad P_x(\xi,Y)=\nabla_xL(x;\xi,Y). \tag*{(226)} P α ( ξ , Y ) = L ( x ; ξ , Y ) i = 1 ∑ I k = 0 ∑ K − 1 ∇ α log π α c ( a k i ∣ s k i ) , P x ( ξ , Y ) = ∇ x L ( x ; ξ , Y ) . (226)
These are unbiased for f α f_{\alpha} f α , f x f_x f x and satisfy ∥ P α ∥ ≤ B L I K M 1 \lVert P_{\alpha}\rVert\le B_LIKM_1 ∥ P α ∥ ≤ B L I K M 1 , ∥ P x ∥ ≤ G L \lVert P_x\rVert\le G_L ∥ P x ∥ ≤ G L . Neither is divided by c c c .
In Algorithm 2, a capped geometric draw means drawing N N N independently with Pr ( N = n ) = ( 1 − γ ) γ n \Pr(N=n)=(1-\gamma)\gamma^n Pr ( N = n ) = ( 1 − γ ) γ n and using a zero sample if N ≥ H N\ge H N ≥ H . For N < H N<H N < H , an equal-mixture prefix chooses J ∼ Unif { 0 , c } J\sim\operatorname{Unif}\{0,c\} J ∼ Unif { 0 , c } , resets from ρ \rho ρ , and follows π α J \pi_{\alpha_J} π α J for N N N transitions and a terminal action. This convention does not condition on N < H N<H N < H ; such conditioning would change the field normalization. The three blocks below define LowerPair, OuterPair, and TDPair in Algorithm 1.
Algorithm 2 One iteration: explicit sampling and persistent paired TD
Require: Frozen p = ( x , U , S ) p=(x,U,S) p = ( x , U , S ) , z = ( u , v ) z=(u,v) z = ( u , v ) , stored certified d ˉ = ρ ^ n \bar d=\hat{\rho}_n d ˉ = ρ ^ n ; H H H , B H B_H B H , B V B_V B V , B C B_C B C , B f , a B_{f,a} B f , a , B f , x B_{f,x} B f , x and the main algorithm’s steps and radii.
Ensure: New ( x + , U + , S + , z + ) (x^+,U^+,S^+,z^+) ( x + , U + , S + , z + ) or failure; all oracle arguments are old.
Set α 0 = U \alpha_0=U α 0 = U , α c = U + c S \alpha_c=U+cS α c = U + c S , θ 0 = u \theta_0=u θ 0 = u , θ c = u + c v \theta_c=u+cv θ c = u + c v .
for b = 1 , … , B H b=1,\ldots,B_H b = 1 , … , B H do
Independently draw geometric N b N_b N b , with Pr ( N b = n ) = ( 1 − γ ) γ n \Pr(N_b=n)=(1-\gamma)\gamma^n Pr ( N b = n ) = ( 1 − γ ) γ n .
if N b ≥ H N_b\ge H N b ≥ H then
Set ( a b , d b ) = ( 0 , 0 ) (a_b,d_b)=(0,0) ( a b , d b ) = ( 0 , 0 ) .
else
Draw an equal-mixture reset prefix ξ b \xi_b ξ b with N b N_b N b transitions and terminal action.
Compute ( a b , d b ) = ( a ( ξ b ) , d ( ξ b ) ) (a_b,d_b)=(a(\xi_b),d(\xi_b)) ( a b , d b ) = ( a ( ξ b ) , d ( ξ b )) using both old critics.
end if
end for
Independently draw B f , a B_{f,a} B f , a preference tuples and labels at old α c \alpha_c α c ; let P ^ α \hat P_\alpha P ^ α average P α P_\alpha P α .
a ^ = B H − 1 ∑ b a b \hat a=B_H^{-1}\sum_b a_b a ^ = B H − 1 ∑ b a b , R ^ = B H − 1 ∑ b d b + P ^ α \hat R=B_H^{-1}\sum_b d_b+\hat P_\alpha R ^ = B H − 1 ∑ b d b + P ^ α ; C ^ = a ^ + c R ^ \hat C=\hat a+c\hat R C ^ = a ^ + c R ^ .
Use Definition C.32 to compute U + = M ~ ( U , a ^ ) U^+=\widetilde M(U,\hat a) U + = M ( U , a ^ ) ; α c + = M ~ ( α c , C ^ ) \alpha_c^+=\widetilde M(\alpha_c,\hat C) α c + = M ( α c , C ^ ) ; S + = ( α c + − U + ) / c S^+=(\alpha_c^+-U^+)/c S + = ( α c + − U + ) / c .
for b = 1 , … , B V b=1,\ldots,B_V b = 1 , … , B V do
Independently draw a fresh capped geometric equal-mixture prefix at the old actor pair.
Set e b = 0 e_b=0 e b = 0 for a capped draw and e b = e ( ξ b ) e_b=e(\xi_b) e b = e ( ξ b ) otherwise.
end for
Independently draw B f , x B_{f,x} B f , x preference tuples and labels at old α c \alpha_c α c ; let P ^ x \hat P_x P ^ x average P x P_x P x .
V ^ = B V − 1 ∑ b e b + P ^ x \hat V=B_V^{-1}\sum_b e_b+\hat P_x V ^ = B V − 1 ∑ b e b + P ^ x ; x + = x − η x V ^ x^+=x-\eta_x\hat V x + = x − η x V ^ .
for b = 1 , … , B C b=1,\ldots,B_C b = 1 , … , B C do
Independently of both evaluation blocks, draw geometric N b N_b N b .
if N b ≥ H N_b\ge H N b ≥ H then
Set F b = 0 F_b=0 F b = 0 .
else
Reset from ρ \rho ρ and follow π U \pi_U π U for N b N_b N b transitions to s s s ; draw a ∼ π U ( ⋅ ∣ s ) a\sim\pi_U(\cdot\mid s) a ∼ π U ( ⋅ ∣ s ) , s ′ ∼ P ( ⋅ ∣ s , a ) s'\sim P(\cdot\mid s,a) s ′ ∼ P ( ⋅ ∣ s , a ) .
Evaluate F b = F s a m p ( p , z ; s , a , s ′ ) F_b=F^{\mathrm{samp}}(p,z;s,a,s') F b = F samp ( p , z ; s , a , s ′ ) from (222) using old p , z p,z p , z .
end if
end for
F ^ = B C − 1 ∑ b F b \hat F=B_C^{-1}\sum_b F_b F ^ = B C − 1 ∑ b F b ; z + = Π K Q W ( z − β F ^ ) z^+=\Pi^W_{K_Q}(z-\beta\hat F) z + = Π K Q W ( z − β F ^ ) .
if Safe ( U + , S + ) \operatorname{Safe}(U^+,S^+) Safe ( U + , S + ) fails then
return failure; Algorithm 1 returns the fixed x 0 x_0 x 0 .
end if
return ( x + , U + , S + , z + ) (x^+,U^+,S^+,z^+) ( x + , U + , S + , z + ) .
Mean identities and truncation. If a scalar test function v v v is integrable, geometric prefix sampling gives E [ v ( s N , a N ) ] = E ν α v \mathbb{E}[v(s_N,a_N)] = \mathbb{E}_{\nu_\alpha}v E [ v ( s N , a N )] = E ν α v . Differentiating the discounted Bellman equation for Q t r u e Q^{\mathrm{true}} Q true and multiplying by the normalized reset resolvent yields
g α ( x , α ) = E ν α [ ∇ α log π α Q x , α t r u e + ∂ α q x , α ] , g x ( x , α ) = E ν α [ − ∇ x r x ] . (227) \begin{aligned}
g_\alpha(x,\alpha) &= \mathbb{E}_{\nu_\alpha}[\nabla_\alpha\log\pi_\alpha Q^{\mathrm{true}}_{x,\alpha}+\partial_\alpha q_{x,\alpha}],\\
g_x(x,\alpha) &= \mathbb{E}_{\nu_\alpha}[-\nabla_x r_x]. \tag*{(227)}
\end{aligned} g α ( x , α ) g x ( x , α ) = E ν α [ ∇ α log π α Q x , α true + ∂ α q x , α ] , = E ν α [ − ∇ x r x ] . (227)
Indeed, differentiating Q = q + γ K α Q Q=q+\gamma K_\alpha Q Q = q + γ K α Q first accounts for the change in the current action law, the criterion and all future action laws. The resolvent collects these score terms over discounted occupancy; the lower normalization cancels the reset-resolvent factor. Fixed dynamics make the second identity immediate. No derivative is taken through learned critic coordinates. With zero output for N ≥ H N\ge H N ≥ H , the TD tail is
∥ E t F ^ t , H − F ( p t , z t ) ∥ W ≤ C T D γ H = b T D . (228) \left\|\mathbb{E}_t\widehat{F}_{t,H}-F(p_t,z_t)\right\|_W \le C_{\mathrm{TD}}\gamma^H=b_{\mathrm{TD}}. \tag*{(228)} E t F t , H − F ( p t , z t ) W ≤ C TD γ H = b TD . (228)
This is truncation, not conditioning on N < H N<H N < H .
Common-prefix balance weights. For fixed prefix length n n n (including n + 1 n+1 n + 1 action factors), let p 0 , p c p^0,p^c p 0 , p c have reset law ρ \rho ρ , common dynamics and endpoint policies; let q = ( p 0 + p c ) / 2 q=(p^0+p^c)/2 q = ( p 0 + p c ) /2 and w j = d p j / d q w_j=dp^j/dq w j = d p j / d q . Selecting the endpoint once per trajectory implements precisely this mixture.
Lemma C.41 (Balance-weight identities).
0 ≤ w 0 , w c ≤ 2 , w 0 + w c = 2 q -a.s. , E q [ w j Z ] = E p j [ Z ] , j ∈ { 0 , c } . (229) 0\le w_0,w_c\le2,\qquad w_0+w_c=2\quad q\text{-a.s.},\qquad\mathbb{E}_q[w_jZ]=\mathbb{E}_{p^j}[Z],\quad j\in\{0,c\}. \tag*{(229)} 0 ≤ w 0 , w c ≤ 2 , w 0 + w c = 2 q -a.s. , E q [ w j Z ] = E p j [ Z ] , j ∈ { 0 , c } . (229)
Proof. Since q ≥ p j / 2 q\ge p^j/2 q ≥ p j /2 , w j ≤ 2 w_j\le2 w j ≤ 2 . The identity w 0 + w c = 2 w_0+w_c=2 w 0 + w c = 2 follows from p 0 + p c = 2 q p^0+p^c=2q p 0 + p c = 2 q , and the expectation identity is the Radon–Nikodym change-of-measure formula.
Lemma C.42 (Divided balance-weight difference is uniform in c c c ). Let r ( ξ ) = log ( d p c / d p 0 ) ( ξ ) r(\xi)=\log(dp^c/dp^0)(\xi) r ( ξ ) = log ( d p c / d p 0 ) ( ξ ) . Then
w c − w 0 = 2 tanh ( r / 2 ) (230) w_c-w_0=2\tanh(r/2) \tag*{(230)} w c − w 0 = 2 tanh ( r /2 ) (230) and
∣ w c ( ξ ) − w 0 ( ξ ) c ∣ ≤ ( n + 1 ) M log , 1 , L ∥ S ∥ . (231) \left|\frac{w_c(\xi)-w_0(\xi)}{c}\right|\le(n+1)M_{\log,1,L}\lVert S\rVert. \tag*{(231)} c w c ( ξ ) − w 0 ( ξ ) ≤ ( n + 1 ) M l o g , 1 , L ∥ S ∥ . (231) Proof. Writing z = d p c / d p 0 = e r z=dp^c/dp^0=e^r z = d p c / d p 0 = e r gives w c = 2 z / ( 1 + z ) w_c=2z/(1+z) w c = 2 z / ( 1 + z ) and w 0 = 2 / ( 1 + z ) w_0=2/(1+z) w 0 = 2/ ( 1 + z ) , hence (230) . The common initial law and common environment dynamics cancel from the path likelihood ratio, leaving
r ( ξ ) = ∑ h = 0 n [ log π U + c S ( a h ∣ s h ) − log π U ( a h ∣ s h ) ] . r(\xi)=\sum_{h=0}^{n}[\log\pi_{U+cS}(a_h\mid s_h)-\log\pi_U(a_h\mid s_h)]. r ( ξ ) = h = 0 ∑ n [ log π U + c S ( a h ∣ s h ) − log π U ( a h ∣ s h )] . The integral mean-value formula and Lemma C.8 give ∣ r ∣ ≤ c ( n + 1 ) M log , 1 , L ∥ S ∥ |r|\le c(n+1)M_{\log,1,L}\lVert S\rVert ∣ r ∣ ≤ c ( n + 1 ) M l o g , 1 , L ∥ S ∥ . Since ∣ tanh z ∣ ≤ ∣ z ∣ |\tanh z|\le|z| ∣ tanh z ∣ ≤ ∣ z ∣ , (231) follows.
Lemma C.43 (Paired evaluation errors, tails and moments). For z ∗ = Z ( p ) z^*=Z(p) z ∗ = Z ( p ) and e j = Q m Q , θ j − Q x , α j t r u e e_j=Q_{m_Q,\theta_j}-Q^{\mathrm{true}}_{x,\alpha_j} e j = Q m Q , θ j − Q x , α j true ,
∥ e 0 ∥ ∞ 2 + ∥ e c − e 0 c ∥ ∞ 2 ≤ C ( ∥ z − z ∗ ∥ W 2 + A Q ) . (232) \lVert e_0\rVert_\infty^2+\left\lVert\frac{e_c-e_0}{c}\right\rVert_\infty^2\le C\left(\lVert z-z^*\rVert_W^2+A_Q\right). \tag*{(232)} ∥ e 0 ∥ ∞ 2 + c e c − e 0 ∞ 2 ≤ C ( ∥ z − z ∗ ∥ W 2 + A Q ) . (232) The base and normalized lower evaluation biases are therefore bounded in squared norm by C ( ∥ z − z ∗ ∥ W 2 + A Q + b e v 2 ) C(\lVert z-z^*\rVert_W^2+A_Q+b_{\mathrm{ev}}^2) C (∥ z − z ∗ ∥ W 2 + A Q + b ev 2 ) . The outer bias is bounded by b e v b_{\mathrm{ev}} b ev without critic error. For some fixed integer ℓ \ell ℓ , b e v ≤ C ( H + 1 ) ℓ γ H b_{\mathrm{ev}}\le C(H+1)^\ell\gamma^H b ev ≤ C ( H + 1 ) ℓ γ H . Every fixed-order conditional moment of the sampled directions is bounded independently of H H H , c c c , m Q m_Q m Q , x x x ; their samplewise envelopes are polynomial in H H H .
Proof. Use the integral
Q m Q , u + c v − Q m Q , u c = ∫ 0 1 ∇ Q m Q , u + t c v T v d t . \frac{Q_{mQ,u+cv}-Q_{mQ,u}}{c}=\int_0^1\nabla Q_{mQ,u+tcv}^{\mathsf T}v\,dt. c Q m Q , u + c v − Q m Q , u = ∫ 0 1 ∇ Q m Q , u + t c v T v d t .
The first two critic derivative bounds make this map Lipschitz in ( u , v ) (u,v) ( u , v ) , uniformly in c c c on K Q K_Q K Q . The base value is also Lipschitz. Subtract their values at z ∗ z^* z ∗ and use (54) to obtain (232) .
For lower score-times-value errors, set T j = ∇ log π α j e j T_j=\nabla\log\pi_{\alpha_j}e_j T j = ∇ log π α j e j . Both the scores and their normalized endpoint differences are bounded. Decompose before dividing:
w c T c − w 0 T 0 c = w c T c − T 0 c + w c − w 0 c T 0 , \frac{w_cT_c-w_0T_0}{c}=w_c\frac{T_c-T_0}{c}+\frac{w_c-w_0}{c}T_0, c w c T c − w 0 T 0 = w c c T c − T 0 + c w c − w 0 T 0 ,
T c − T 0 c = ∇ log π α c e c − e 0 c + ∇ log π α c − ∇ log π α 0 c e 0 . \frac{T_c-T_0}{c}=\nabla\log\pi_{\alpha_c}\frac{e_c-e_0}{c}+\frac{\nabla\log\pi_{\alpha_c}-\nabla\log\pi_{\alpha_0}}{c}e_0. c T c − T 0 = ∇ log π α c c e c − e 0 + c ∇ log π α c − ∇ log π α 0 e 0 .
Lemmas C.41 –C.42 bound this by C ( n + 1 ) ( ∥ z − z ∗ ∥ W + A Q ) C(n+1)(\lVert z-z^*\rVert_W+\sqrt{A_Q}) C ( n + 1 ) (∥ z − z ∗ ∥ W + A Q ) . The direct criterion derivatives introduce no critic error. Taking geometric expectations proves the untruncated bias bound. The outer sample in (225) uses only r x r_x r x derivatives.
On the projection set, base learned values and normalized differences are bounded even when inaccurate. Thus each untruncated direction is bounded by C ( n + 1 ) ℓ C(n+1)^\ell C ( n + 1 ) ℓ for a fixed degree. Every moment ∑ n ≥ 0 ( 1 − γ ) γ n ( n + 1 ) j \sum_{n\ge0}(1-\gamma)\gamma^n(n+1)^j ∑ n ≥ 0 ( 1 − γ ) γ n ( n + 1 ) j is finite. The removed tail is at most C ( H + 1 ) ℓ γ H C(H+1)^\ell\gamma^H C ( H + 1 ) ℓ γ H , by writing n = H + k n=H+k n = H + k and summing the geometric polynomial. Capping supplies the samplewise polynomial envelope; it does not worsen any fixed-order moment. Preference terms are bounded by Proposition C.44 . For TD, the integral identity (52) bounds the sample uniformly, and its omitted tail proves (228) .
Direct preference samples.
Proposition C.44 (Unbiased bounded tuple gradients). For a tuple and label sampled at any frozen actor α \alpha α , use P x P_x P x , P α P_\alpha P α from (226) , with α c \alpha_c α c replaced by α \alpha α . Conditionally on every pre-sampling history that fixes ( x , α ) (x,\alpha) ( x , α ) ,
E [ P x ∣ x , α ] = f x ( x , α ) , E [ P α ∣ x , α ] = f α ( x , α ) . (233) \mathbb{E}[P_x\mid x,\alpha]=f_x(x,\alpha),\qquad\mathbb{E}[P_\alpha\mid x,\alpha]=f_\alpha(x,\alpha). \tag*{(233)} E [ P x ∣ x , α ] = f x ( x , α ) , E [ P α ∣ x , α ] = f α ( x , α ) . (233) For every realized sample,
∥ P x ∥ ≤ G L , ∥ P α ∥ ≤ B L I K M 1 . (234) \lVert P_x\rVert\le G_L,\qquad\lVert P_\alpha\rVert\le B_LIKM_1. \tag*{(234)} ∥ P x ∥ ≤ G L , ∥ P α ∥ ≤ B L I K M 1 . (234) Averages F ^ x , F ^ α \widehat F_x,\widehat F_\alpha F x , F α of respectively B f , x B_{f,x} B f , x , B f , α B_{f,\alpha} B f , α independent tuple–label samples have conditional variance at most G L 2 / B f , x G_L^2/B_{f,x} G L 2 / B f , x and ( B L I K M 1 ) 2 / B f , α (B_LIKM_1)^2/B_{f,\alpha} ( B L I K M 1 ) 2 / B f , α .
Proof. The joint tuple–label law factors into the I I I trajectory laws and D \mathcal{D} D . Fixed initial law and dynamics and the parameter-independent label kernel imply that its actor log derivative is the score sum ∑ i , k ∇ α log π α ( a k i ∣ s k i ) \sum_{i,k}\nabla_\alpha\log\pi_\alpha(a_k^i\mid s_k^i) ∑ i , k ∇ α log π α ( a k i ∣ s k i ) ; its x x x derivative is zero. Differentiating under the finite expectation is justified by the displayed uniform envelopes. Since the sample loss has no explicit actor dependence, this gives (233) . The triangle inequality gives a score-sum bound of I K M 1 IKM_1 I K M 1 , proving (234) . Conditional independence and centering divide the second-moment bounds by the batch size. These statements concern each sampled label, rather than only a label-averaged population loss.
Batch sizes and timing. Fix positive preference batch sizes B f , α , B f , x B_{f,\alpha},B_{f,x} B f , α , B f , x independently of accuracy. One may take any positive TD/evaluation batch sizes B C B_C B C , B H B_H B H , B V B_V B V bounded by a fixed polynomial in H H H , including fixed sizes: the geometric moment bounds already give the uniform moments through order eight used in the proof. Averaging independent samples preserves those bounds and reduces centered second moments. Conditional on F t \mathcal{F}_t F t , each evaluation stage has its own independent tuple/label block, and TD is independent of both evaluation stages. The exact identity C ^ = a ^ + c ^ R ^ \widehat C=\widehat a+\widehat c\widehat R C = a + c R is samplewise. After conditioning further on G t \mathcal{G}_t G t , the outer block still has its old-parameter conditional mean. No lower labels are reused for the outer estimator, and the critic update is first used at iteration t + 1 t+1 t + 1 . The full interaction ledger is (76) .
Conditional tracking estimates
The results here use the derived trained-critic bias from Lemma B.8 , the residual-controlled finite solves, and the pre-draw history F t \mathcal{F}_t F t defined in Appendix § 9 . They do not introduce an independent oracle-accuracy assumption.
Lemma C.45 (Conditional means and moments of the simultaneous increment). Write Δ p t = p t + 1 − p t = d t + w t + 1 \Delta p_t=p_{t+1}-p_t=d_t+w_{t+1} Δ p t = p t + 1 − p t = d t + w t + 1 , where d t = E t Δ p t d_t=\mathbb{E}_t\Delta p_t d t = E t Δ p t and E t w t + 1 = 0 \mathbb{E}_t w_{t+1}=0 E t w t + 1 = 0 . For the prospective step,
∥ d t ∥ 2 ≤ C { ( η y 2 + η x 2 ) ( T t + C t + ω ) + η x 2 X t + ( η y 2 + η x 2 ) 2 } , \lVert d_t\rVert^2 \le C\{(\eta_y^2+\eta_x^2)(\mathcal{T}_t+\mathcal{C}_t+\omega)+\eta_x^2X_t+(\eta_y^2+\eta_x^2)^2\}, ∥ d t ∥ 2 ≤ C {( η y 2 + η x 2 ) ( T t + C t + ω ) + η x 2 X t + ( η y 2 + η x 2 ) 2 } , E t ∥ w t + 1 ∥ 2 ≤ C ( η y 2 + η x 2 ) , E t ∥ Δ p t ∥ 4 ≤ C ( η y + η x ) 4 . (235) \mathbb{E}_t\lVert w_{t+1}\rVert^2\le C(\eta_y^2+\eta_x^2),\qquad\mathbb{E}_t\lVert\Delta p_t\rVert^4\le C(\eta_y+\eta_x)^4. \tag*{(235)} E t ∥ w t + 1 ∥ 2 ≤ C ( η y 2 + η x 2 ) , E t ∥ Δ p t ∥ 4 ≤ C ( η y + η x ) 4 . (235) The combined reference-map and finite-solver remainders have conditional first, second and fourth moments O ( η y ) O(\eta_y) O ( η y ) , O ( η y 2 ) O(\eta_y^2) O ( η y 2 ) and O ( η y 8 ) O(\eta_y^8) O ( η y 8 ) , respectively. The entire Δ p t \Delta p_t Δ p t is conditionally independent of the current TD block. All constants are uniform over subsequently smaller step ratios.
Proof. For each sampled direction decompose, for example, a ^ = a + e a + ζ a \hat{a}=a+e_a+\zeta_a a ^ = a + e a + ζ a , where e a = E t a ^ − a e_a=\mathbb{E}_t\hat{a}-a e a = E t a ^ − a and E t ζ a = 0 \mathbb{E}_t\zeta_a=0 E t ζ a = 0 ; use the same decomposition for R ^ , V ^ \hat{R},\hat{V} R ^ , V ^ . The inverse metrics P 0 P^0 P 0 , P c P^c P c and correction K c K_c K c and the full metric sample are fixed under F t \mathcal{F}_t F t . Lemma C.25 and the samplewise bounds in Lemma C.35 give
Δ U = − η y P 0 a ^ + ρ U + e U , \Delta U=-\eta_yP^0\hat{a}+\rho^U+e_U, Δ U = − η y P 0 a ^ + ρ U + e U ,
Δ S = − η y P c R ^ + η y K c a ^ + ρ S + e S , \Delta S=-\eta_yP^c\hat{R}+\eta_yK_c\hat{a}+\rho^S+e_S, Δ S = − η y P c R ^ + η y K c a ^ + ρ S + e S ,
Δ x = − η x V ^ . \Delta x=-\eta_x\hat{V}. Δ x = − η x V ^ .
The reference-map remainders are η y 2 \eta_y^2 η y 2 times polynomials of degree at most two in the directions, with bounded S S S , c c c . Direction moments through order eight from Lemma B.8 bound their means, second moments and fourth moments. Moreover, ∥ ( e U , e S ) ∥ ≤ 5 η y 2 \lVert(e_U,e_S)\rVert\le\sqrt{5}\eta_y^2 ∥( e U , e S )∥ ≤ 5 η y 2 samplewise, so their means and centered parts preserve these moment orders. Centered first-order terms have zero conditional mean. Thus the population field bounds (43) and the squared bias bound (56) give the first line of (235) . The bounded second moments of all directions and the remainder estimates give its second line; the same argument using fourth moments of the increment gives the last bound. No independence between the base and scaled lower estimates is needed.
The two finite actor updates, including their stopping indices, are measurable functions of the lower block and stored metric sample, while the upper update is a function of its separate outer block. Both use old parameters. The TD block is independent of this pair of blocks under F t \mathcal{F}_t F t . Hence revealing TD first preserves the centering of w t + 1 w_{t+1} w t + 1 , exactly as used in the moving-target critic proof. Conditional fourth moments are finite without any bound on upper iterates, because the reward and loss envelopes are global. □
Lemma C.46 (Conditional Taylor transfer for actor tracking). Let d U p o p = − P 0 a d_U^{\mathrm{pop}}=-P^0a d U pop = − P 0 a and d S p o p = − P c R c + K c a d_S^{\mathrm{pop}}=-P^cR_c+K_ca d S pop = − P c R c + K c a . On a prospective step,
E t Δ T t ≤ η y ⟨ ∇ U , S T t , ( d U p o p , d S p o p ) ⟩ − η x ⟨ ∇ x T t , V c ( p t ) ⟩ + C ( η y + η x ) T t C t + ω + C ( η y 2 + η x 2 ) . (236) \mathbb{E}_t\Delta\mathcal{T}_t\le\eta_y\langle\nabla_{U,S}\mathcal{T}_t,(d_U^{\mathrm{pop}},d_S^{\mathrm{pop}})\rangle
-\eta_x\langle\nabla_x\mathcal{T}_t,V_c(p_t)\rangle+C(\eta_y+\eta_x)\sqrt{\mathcal{T}_t}\sqrt{\mathcal{C}_t+\omega}+C(\eta_y^2+\eta_x^2). \tag*{(236)} E t Δ T t ≤ η y ⟨ ∇ U , S T t , ( d U pop , d S pop )⟩ − η x ⟨ ∇ x T t , V c ( p t )⟩ + C ( η y + η x ) T t C t + ω + C ( η y 2 + η x 2 ) . (236)
Proof. Apply the quadratic Taylor formula in all variables ( x , U , S ) (x,U,S) ( x , U , S ) along the segment from p t p_t p t to p t + Δ p t p_t+\Delta p_t p t + Δ p t . Lemma C.19 bounds the full Hessian there, including mixed actor–upper blocks. Lemma C.45 therefore bounds the expected quadratic remainder by C ( η y 2 + η x 2 ) C(\eta_y^2+\eta_x^2) C ( η y 2 + η x 2 ) . Substitute the conditional mean of each update from its paired expansion. Centered first-order noise vanishes because the old gradient and metric are F t \mathcal{F}_t F t -measurable. The conditional means of the combined reference-map and numerical remainders cost C η y 2 C\eta_y^2 C η y 2 , since the old potential gradient is bounded. The remaining bias products are bounded by C ( η y + C(\eta_y+ C ( η y + η X ∥ ∇ T t ∥ C t + A Q + b e v 2 \eta_X\|\nabla T_t\|\sqrt{C_t+A_Q+b_{\mathrm{ev}}^2} η X ∥∇ T t ∥ C t + A Q + b ev 2 . The gradient self-bound gives the displayed inequality. This calculation also covers the lower-first implementation order: all fields remain functions of the old parameters, and fresh outer data retain their conditional mean after revealing G t \mathcal{G}_t G t . No conditional expectation is taken with respect to a future non-exit event.
All segmentwise statements below are used on the larger prospective actor neighborhood. Short Young and quadratic Taylor inequalities are applied directly in the main proofs.
Lemma C.47 (Covariance transfer under domination). If P ⪰ q Q P \succeq qQ P ⪰ q Q for probability measures P , Q P,Q P , Q and q ∈ ( 0 , 1 ] q \in(0,1] q ∈ ( 0 , 1 ] , then for every square-integrable vector X X X ,
Cov P ( X ) ⪰ q Cov Q ( X ) . \operatorname{Cov}_P(X) \succeq q\operatorname{Cov}_Q(X). Cov P ( X ) ⪰ q Cov Q ( X ) . Proof. For a vector v v v , write Y = v ⊤ X Y=v^\top X Y = v ⊤ X . The variational identity Var P ( Y ) = inf a ∈ R E P [ ( Y − a ) 2 ] \operatorname{Var}_P(Y)=\inf_{a\in\mathbb{R}}\mathbb{E}_P[(Y-a)^2] Var P ( Y ) = inf a ∈ R E P [( Y − a ) 2 ] gives
Var P ( Y ) ≥ q inf a ∈ R E Q [ ( Y − a ) 2 ] = q Var Q ( Y ) . \operatorname{Var}_P(Y) \ge q\inf_{a\in\mathbb{R}}\mathbb{E}_Q[(Y-a)^2] = q\operatorname{Var}_Q(Y). Var P ( Y ) ≥ q a ∈ R inf E Q [( Y − a ) 2 ] = q Var Q ( Y ) . Apply the scalar conclusion to every v v v .
Lemma C.48 (Conditional centering is an L 2 L_2 L 2 contraction). On a joint state-action probability space, let Π X = X − E [ X ∣ s ] \Pi X=X-\mathbb{E}[X\mid s] Π X = X − E [ X ∣ s ] . Then
∥ Π X ∥ L 2 ≤ ∥ X ∥ L 2 , ∥ Π X − Π Y ∥ L 2 ≤ ∥ X − Y ∥ L 2 . \|\Pi X\|_{L_2}\le\|X\|_{L_2},\qquad\|\Pi X-\Pi Y\|_{L_2}\le\|X-Y\|_{L_2}. ∥Π X ∥ L 2 ≤ ∥ X ∥ L 2 , ∥Π X − Π Y ∥ L 2 ≤ ∥ X − Y ∥ L 2 . Proof. Conditional expectation is the orthogonal projection onto the closed subspace of s s s -measurable random variables. Hence Π \Pi Π is the complementary orthogonal projection and has operator norm one.
Lemma C.49 (Inverse perturbation). If A A A and B B B are invertible matrices with ∥ A − 1 ∥ o p , ∥ B − 1 ∥ o p ≤ μ − 1 \|A^{-1}\|_{\mathrm{op}},\|B^{-1}\|_{\mathrm{op}}\le\mu^{-1} ∥ A − 1 ∥ op , ∥ B − 1 ∥ op ≤ μ − 1 , then
∥ A − 1 − B − 1 ∥ o p ≤ μ − 2 ∥ A − B ∥ o p . \|A^{-1}-B^{-1}\|_{\mathrm{op}}\le\mu^{-2}\|A-B\|_{\mathrm{op}}. ∥ A − 1 − B − 1 ∥ op ≤ μ − 2 ∥ A − B ∥ op . Proof. Use A − 1 − B − 1 = A − 1 ( B − A ) B − 1 A^{-1}-B^{-1}=A^{-1}(B-A)B^{-1} A − 1 − B − 1 = A − 1 ( B − A ) B − 1 .
Lemma C.50 (Gradient bounds from strong convexity and smoothness). Let H H H be differentiable, μ \mu μ -strongly convex, and have L H L_H L H -Lipschitz gradient on a convex set. Suppose its minimizer z ⋆ z^\star z ⋆ is interior, so ∇ H ( z ⋆ ) = 0 \nabla H(z^\star)=0 ∇ H ( z ⋆ ) = 0 . Then
μ 2 ∥ z − z ⋆ ∥ 2 ≤ H ( z ) − H ( z ⋆ ) , (237) \frac{\mu}{2}\|z-z^\star\|^2\le H(z)-H(z^\star), \tag*{(237)} 2 μ ∥ z − z ⋆ ∥ 2 ≤ H ( z ) − H ( z ⋆ ) , (237) 2 μ ( H ( z ) − H ( z ⋆ ) ) ≤ ∥ ∇ H ( z ) ∥ 2 , (238) 2\mu(H(z)-H(z^\star))\le\|\nabla H(z)\|^2, \tag*{(238)} 2 μ ( H ( z ) − H ( z ⋆ )) ≤ ∥∇ H ( z ) ∥ 2 , (238) ∥ ∇ H ( z ) ∥ 2 ≤ 2 L H 2 μ ( H ( z ) − H ( z ⋆ ) ) . (239) \|\nabla H(z)\|^2\le\frac{2L_H^2}{\mu}(H(z)-H(z^\star)). \tag*{(239)} ∥∇ H ( z ) ∥ 2 ≤ μ 2 L H 2 ( H ( z ) − H ( z ⋆ )) . (239) Proof. Strong convexity at z ⋆ z^\star z ⋆ gives (237) . Strong convexity at z z z gives
H ( z ⋆ ) ≥ H ( z ) + ⟨ ∇ H ( z ) , z ⋆ − z ⟩ + μ 2 ∥ z − z ⋆ ∥ 2 . H(z^\star)\ge H(z)+\langle\nabla H(z),z^\star-z\rangle+\frac{\mu}{2}\|z-z^\star\|^2. H ( z ⋆ ) ≥ H ( z ) + ⟨ ∇ H ( z ) , z ⋆ − z ⟩ + 2 μ ∥ z − z ⋆ ∥ 2 . Maximizing the right-hand upper bound on H ( z ) − H ( z ⋆ ) H(z)-H(z^\star) H ( z ) − H ( z ⋆ ) over the displacement yields H ( z ) − H ( z ⋆ ) ≤ ∥ ∇ H ( z ) ∥ 2 / ( 2 μ ) H(z)-H(z^\star)\le\|\nabla H(z)\|^2/(2\mu) H ( z ) − H ( z ⋆ ) ≤ ∥∇ H ( z ) ∥ 2 / ( 2 μ ) , proving (238) . Finally, ∥ ∇ H ( z ) ∥ ≤ L H ∥ z − z ⋆ ∥ \|\nabla H(z)\|\le L_H\|z-z^\star\| ∥∇ H ( z ) ∥ ≤ L H ∥ z − z ⋆ ∥ and (237) give (239) .
Lemma C.51 (Bounded innovations despite critic projection). There is a polynomial envelope B D ( H ) B_D(H) B D ( H ) such that, at pre-exit steps including the prospective first exit,
∣ N t + 1 ∣ ≤ η y B D ( H ) , N t + 1 = D t + 1 − E t D t + 1 . (240) \lvert N_{t+1}\rvert\le\eta_y B_D(H), \qquad N_{t+1} = D_{t+1} - \mathbb{E}_t D_{t+1}. \tag*{(240)} ∣ N t + 1 ∣ ≤ η y B D ( H ) , N t + 1 = D t + 1 − E t D t + 1 . (240) No differentiability or inactivity of the critic projection is needed.
Proof. Since z t ∈ K Q z_t \in\mathcal{K}_Q z t ∈ K Q , nonexpansiveness gives ∥ z t + 1 − z t ∥ W ≤ β ∥ F ^ t , H ∥ W ≤ C β \lVert z_{t+1}-z_t\rVert_W \le\beta\lVert\widehat{F}_{t,H}\rVert_W \le C\beta ∥ z t + 1 − z t ∥ W ≤ β ∥ F t , H ∥ W ≤ C β . Uniform target derivatives and (44) give ∥ Z ( p t + 1 ) − Z ( p t ) ∥ W ≤ C η y poly ( H ) \lVert Z(p_{t+1})-Z(p_t)\rVert_W \le C\eta_y\operatorname{poly}(H) ∥ Z ( p t + 1 ) − Z ( p t ) ∥ W ≤ C η y poly ( H ) . Both old and new error vectors have bounded norm because the critic iterate and prospective target lie in K Q \mathcal{K}_Q K Q . Hence
∣ C t + 1 − C t ∣ ≤ C [ β + η y poly ( H ) ] . \lvert C_{t+1}-C_t\rvert\le C[\beta+\eta_y\operatorname{poly}(H)]. ∣ C t + 1 − C t ∣ ≤ C [ β + η y poly ( H )] .
Multiplication by λ r Q \lambda r_Q λ r Q gives C η y poly ( H ) C\eta_y\operatorname{poly}(H) C η y poly ( H ) , using r Q β = η y r_Q\beta=\eta_y r Q β = η y and r Q ≤ 1 r_Q\le1 r Q ≤ 1 . Uniform first derivatives of the actor potential similarly give ∣ T t + 1 − T t ∣ ≤ C η y poly ( H ) \lvert T_{t+1}-T_t\rvert\le C\eta_y\operatorname{poly}(H) ∣ T t + 1 − T t ∣ ≤ C η y poly ( H ) . Centering doubles the resulting deterministic bound. Relative to the virtual exact step, Proposition C.37 changes the next D D D by at most C η y 2 C\eta_y^2 C η y 2 samplewise, and centering changes its innovation by at most 2 C η y 2 2C\eta_y^2 2 C η y 2 . This is absorbed by enlarging B D ( H ) B_D(H) B D ( H ) , without a c − 1 c^{-1} c − 1 factor. The argument is valid for all sufficiently small absolute steps on the reserved prospective buffers. □
Assumption verification and the PL counterexample
A1’s actor derivative requirement is verified by Theorem C.4 ; the fixed bounded input, normalized initialization and fixed chart are part of that construction. A2 is an identifiability requirement with the optional independent pilot in Proposition C.13 . Its local persistence does not imply global nondegeneracy for unrestricted neural parameters. For G, Proposition C.14 and the bounded tanh reward example in Appendix § 10 .§ 2 give explicit global envelopes. Neither requires a correctly specified fitted preference model.
For C1, a feature Gram gap under the base reset law ρ U \rho_U ρ U is sufficient by ν U ⪰ ( 1 − γ ) ρ U \nu_U \succeq(1-\gamma)\rho_U ν U ⪰ ( 1 − γ ) ρ U . This concerns fixed critic features, not the actor tangent features. A sufficient structural certificate for C2 is that the feature span contains r x r_x r x and the log-policy-ratio criterion separately and is invariant under every admitted K α K_\alpha K α . The Neumann series for ( I − γ K α ) − 1 (I-\gamma K_\alpha)^{-1} ( I − γ K α ) − 1 then stays in this finite-dimensional closed span. Finite state–action indicator features provide one such realization when their occupancy Gram matrix has a uniform gap. For continuous states the same span invariance and Gram gap are substantive restrictions. The theorem requires C2 at the selected fixed τ \tau τ ; increasing τ \tau τ or m Q m_Q m Q alone does not establish it. Even under exact C2, the finite nonlinear critic retains the derived O ( m Q − 2 ) O(m_Q^{-2}) O ( m Q − 2 ) squared paired approximation floor in (54) .
What generic policy smoothness does not imply.
Proposition C.52 (Smooth actors and exact critics do not imply uniform PL). Consider a one-state self-loop MDP with actions 0 0 0 , 1 1 1 , rewards r ( 1 ) = 1 r(1)=1 r ( 1 ) = 1 and r ( 0 ) = 0 r(0)=0 r ( 0 ) = 0 , and the actor
π θ ( 1 ) = 1 2 + 1 4 sin θ , π θ ( 0 ) = 1 − π θ ( 1 ) , θ ∈ [ − π , π ] . (241) \pi_\theta(1)=\frac{1}{2}+\frac{1}{4}\sin\theta,\qquad
\pi_\theta(0)=1-\pi_\theta(1),\qquad
\theta\in[-\pi,\pi]. \tag*{(241)} π θ ( 1 ) = 2 1 + 4 1 sin θ , π θ ( 0 ) = 1 − π θ ( 1 ) , θ ∈ [ − π , π ] . (241) The actor probabilities are uniformly positive and all required score derivatives are bounded. A tabular critic can represent the action-value function exactly. Nevertheless the normalized discounted return fails every uniform maximization PL inequality with a positive constant.
Proof. The one state is visited at every step, so the normalized return is
J ( θ ) = ( 1 − γ ) ∑ t ≥ 0 γ t π θ ( 1 ) = 1 2 + 1 4 sin θ . J(\theta)=(1-\gamma)\sum_{t\ge0}\gamma^t\pi_\theta(1)=\frac{1}{2}+\frac{1}{4}\sin\theta. J ( θ ) = ( 1 − γ ) t ≥ 0 ∑ γ t π θ ( 1 ) = 2 1 + 4 1 sin θ .
Its maximum is J ∗ = 3 / 4 J^* = 3/4 J ∗ = 3/4 at θ = π / 2 \theta= \pi/2 θ = π /2 , whereas
J ′ ( − π / 2 ) = 1 4 cos ( − π / 2 ) = 0 , J ∗ − J ( − π / 2 ) = 1 2 . J'(-\pi/2) = \frac{1}{4}\cos(-\pi/2) = 0,\qquad J^* - J(-\pi/2) = \frac{1}{2}. J ′ ( − π /2 ) = 4 1 cos ( − π /2 ) = 0 , J ∗ − J ( − π /2 ) = 2 1 .
Thus 1 2 ∥ ∇ J ( θ ) ∥ 2 ≥ μ ( J ∗ − J ( θ ) ) \frac{1}{2}\lVert\nabla J(\theta)\rVert^2 \ge\mu(J^* - J(\theta)) 2 1 ∥ ∇ J ( θ ) ∥ 2 ≥ μ ( J ∗ − J ( θ )) fails for every μ > 0 \mu> 0 μ > 0 . Equivalently, the minimization PL inequality fails for g = − J g = -J g = − J . Since both action probabilities lie in [ 1 / 4 , 3 / 4 ] [1/4, 3/4] [ 1/4 , 3/4 ] , their log derivatives are smooth and bounded on the compact domain. The ordinary discounted action values are Q θ ( a ) = r ( a ) + γ π θ ( 1 ) / ( 1 − γ ) Q_\theta(a) = r(a) + \gamma\pi_\theta(1)/(1-\gamma) Q θ ( a ) = r ( a ) + γ π θ ( 1 ) / ( 1 − γ ) , which an exact two-entry tabular critic represents.
The example establishes that generic actor smoothness, positive probabilities, and critic realizability do not imply uniform PL. It does not contradict a theorem that imposes PL separately. It is not a witness that our entire actor/assumption class strictly contains Gaur et al.’s class.
KL-regularized counterexamples and degenerate neural policies
The example in Proposition C.52 concerns the unregularized return. We now give an additional smooth policy with a spurious strict local minimum for every positive KL-regularization level, and explain a related obstruction in multilayer neural parameterizations.
Setting and the PL condition
Consider a discounted MDP with finite action set A A A , a strictly positive reference policy π r e f \pi_{\mathrm{ref}} π ref , and a parametric policy family { π θ : θ ∈ Θ } \{\pi_\theta: \theta\in\Theta\} { π θ : θ ∈ Θ } . For an upper parameter x x x and τ ≥ 0 \tau\ge0 τ ≥ 0 , the normalized lower cost is
g τ ( x , θ ) = ( 1 − γ ) E π θ ∑ t ≥ 0 γ t [ − r x ( s t , a t ) + τ log π θ ( a t ∣ s t ) π r e f ( a t ∣ s t ) ] . (242) g_\tau(x,\theta) = (1-\gamma)\mathbb{E}_{\pi_\theta}\sum_{t\ge0}\gamma^t\left[-r_x(s_t,a_t) + \tau\log\frac{\pi_\theta(a_t\mid s_t)}{\pi_{\mathrm{ref}}(a_t\mid s_t)}\right]. \tag*{(242)} g τ ( x , θ ) = ( 1 − γ ) E π θ t ≥ 0 ∑ γ t [ − r x ( s t , a t ) + τ log π ref ( a t ∣ s t ) π θ ( a t ∣ s t ) ] . (242)
as in (2) , so the regularizer is D K L ( π θ ∥ π r e f ) D_{\mathrm{KL}}(\pi_\theta\Vert\pi_{\mathrm{ref}}) D KL ( π θ ∥ π ref ) averaged over the discounted occupancy. Write g τ ∗ ( x ) = inf θ ∈ Θ g τ ( x , θ ) g_\tau^*(x) = \inf_{\theta\in\Theta}g_\tau(x,\theta) g τ ∗ ( x ) = inf θ ∈ Θ g τ ( x , θ ) .
A uniform lower PL condition with constant μ > 0 \mu> 0 μ > 0 on Θ \Theta Θ requires, for every x x x in the upper domain and every θ ∈ Θ \theta\in\Theta θ ∈ Θ ,
1 2 ∥ ∇ θ g τ ( x , θ ) ∥ 2 ≥ μ ( g τ ( x , θ ) − g τ ∗ ( x ) ) . (243) \frac{1}{2}\lVert\nabla_\theta g_\tau(x,\theta)\rVert^2 \ge\mu\left(g_\tau(x,\theta)-g_\tau^*(x)\right). \tag*{(243)} 2 1 ∥ ∇ θ g τ ( x , θ ) ∥ 2 ≥ μ ( g τ ( x , θ ) − g τ ∗ ( x ) ) . (243)
Consequently, a single pair ( x , θ ˉ ) (x,\bar{\theta}) ( x , θ ˉ ) with ∇ θ g τ ( x , θ ˉ ) = 0 \nabla_\theta g_\tau(x,\bar{\theta}) = 0 ∇ θ g τ ( x , θ ˉ ) = 0 and g τ ( x , θ ˉ ) > g τ ∗ ( x ) g_\tau(x,\bar{\theta}) > g_\tau^*(x) g τ ( x , θ ˉ ) > g τ ∗ ( x ) rules out (243) for every μ > 0 \mu> 0 μ > 0 .
A spurious local minimum for every regularization level
One-state reduction. Take one state with a self-loop, A = { 0 , 1 } A = \{0,1\} A = { 0 , 1 } , π r e f = ( 1 / 2 , 1 / 2 ) \pi_{\mathrm{ref}} = (1/2,1/2) π ref = ( 1/2 , 1/2 ) , and reward r x ( 1 ) = u ( x ) r_x(1) = u(x) r x ( 1 ) = u ( x ) , r x ( 0 ) = 0 r_x(0) = 0 r x ( 0 ) = 0 . Write p = π θ ( 1 ) p = \pi_\theta(1) p = π θ ( 1 ) and
k ( p ) = D K L ( Bern ( p ) ∥ Bern ( 1 2 ) ) = p log ( 2 p ) + ( 1 − p ) log ( 2 ( 1 − p ) ) . k(p) = D_{\mathrm{KL}}\left(\operatorname{Bern}(p)\,\middle\Vert\,\operatorname{Bern}\left(\frac{1}{2}\right)\right) = p\log(2p) + (1-p)\log(2(1-p)). k ( p ) = D KL ( Bern ( p ) Bern ( 2 1 ) ) = p log ( 2 p ) + ( 1 − p ) log ( 2 ( 1 − p )) .
The state is visited at every step, so (242) reduces to
g τ ( x , θ ) = h u , τ ( p ( θ ) ) , h u , τ ( p ) = − u p + τ k ( p ) . (244) g_\tau(x,\theta) = h_{u,\tau}(p(\theta)),\qquad h_{u,\tau}(p) = -up + \tau k(p). \tag*{(244)} g τ ( x , θ ) = h u , τ ( p ( θ )) , h u , τ ( p ) = − u p + τ k ( p ) . (244)
For τ > 0 \tau> 0 τ > 0 , h u , τ h_{u,\tau} h u , τ is strictly convex on ( 0 , 1 ) (0,1) ( 0 , 1 ) , with h u , τ ′ ( p ) = − u + τ log p 1 − p h'_{u,\tau}(p) = -u + \tau\log\frac{p}{1-p} h u , τ ′ ( p ) = − u + τ log 1 − p p .
The policy family. For θ ∈ R \theta\in\mathbb{R} θ ∈ R (equivalently, on the circle), let
p ( θ ) = 1 2 + cos θ + 1 2 cos 2 θ 6 . (245) p(\theta)=\frac{1}{2}+\frac{\cos\theta+\frac{1}{2}\cos2\theta}{6}. \tag*{(245)} p ( θ ) = 2 1 + 6 cos θ + 2 1 cos 2 θ . (245)
Lemma C.53 (Properties of the family). The map p p p is C ∞ C^\infty C ∞ and 2 π 2\pi 2 π -periodic, and its range is exactly [ 3 8 , 3 4 ] [\frac{3}{8},\frac{3}{4}] [ 8 3 , 4 3 ] . Within each period it has a strict global maximum p ( 0 ) = 3 4 p(0)=\frac{3}{4} p ( 0 ) = 4 3 , strict global minima p ( ± 2 π 3 ) = 3 8 p(\pm\frac{2\pi}{3})=\frac{3}{8} p ( ± 3 2 π ) = 8 3 , and a strict local maximum p ( π ) = 5 12 p(\pi)=\frac{5}{12} p ( π ) = 12 5 . All log-policy derivatives of every order are bounded.
Proof. Put φ ( θ ) = cos θ + 1 2 cos 2 θ \varphi(\theta)=\cos\theta+\frac{1}{2}\cos2\theta φ ( θ ) = cos θ + 2 1 cos 2 θ . Then φ ′ ( θ ) = − sin θ ( 1 + 2 cos θ ) \varphi'(\theta)=-\sin\theta(1+2\cos\theta) φ ′ ( θ ) = − sin θ ( 1 + 2 cos θ ) , so the critical points in ( − π , π ] (-\pi,\pi] ( − π , π ] are θ ∈ { 0 , π , ± 2 π 3 } \theta\in\{0,\pi,\pm\frac{2\pi}{3}\} θ ∈ { 0 , π , ± 3 2 π } , with values φ ( 0 ) = 3 2 \varphi(0)=\frac{3}{2} φ ( 0 ) = 2 3 , φ ( π ) = − 1 2 \varphi(\pi)=-\frac{1}{2} φ ( π ) = − 2 1 and φ ( ± 2 π 3 ) = − 3 4 \varphi(\pm\frac{2\pi}{3})=-\frac{3}{4} φ ( ± 3 2 π ) = − 4 3 . Since φ ′ ′ ( θ ) = − cos θ − 2 cos 2 θ \varphi''(\theta)=-\cos\theta-2\cos2\theta φ ′′ ( θ ) = − cos θ − 2 cos 2 θ , we get φ ′ ′ ( 0 ) = − 3 \varphi''(0)=-3 φ ′′ ( 0 ) = − 3 , φ ′ ′ ( π ) = − 1 \varphi''(\pi)=-1 φ ′′ ( π ) = − 1 and φ ′ ′ ( ± 2 π 3 ) = 3 2 \varphi''(\pm\frac{2\pi}{3})=\frac{3}{2} φ ′′ ( ± 3 2 π ) = 2 3 . This gives the classification and the range φ ∈ [ − 3 4 , 3 2 ] \varphi\in[-\frac{3}{4},\frac{3}{2}] φ ∈ [ − 4 3 , 2 3 ] , hence p ∈ [ 3 8 , 3 4 ] p\in[\frac{3}{8},\frac{3}{4}] p ∈ [ 8 3 , 4 3 ] . Both action probabilities lie in [ 1 4 , 5 8 ] ∪ [ 3 8 , 3 4 ] ⊂ [ 1 4 , 3 4 ] [\frac{1}{4},\frac{5}{8}]\cup[\frac{3}{8},\frac{3}{4}]\subset[\frac{1}{4},\frac{3}{4}] [ 4 1 , 8 5 ] ∪ [ 8 3 , 4 3 ] ⊂ [ 4 1 , 4 3 ] , so log π θ \log\pi_\theta log π θ is a smooth function of a trigonometric polynomial bounded away from the singularities of log \log log , and all of its derivatives are bounded.
Proposition C.54 (Spurious local minimum for every τ \tau τ ). Let u ≥ 0 u\ge0 u ≥ 0 and τ > 0 \tau>0 τ > 0 . In the setting (244) –(245) : (i) θ ˉ = π \bar{\theta}=\pi θ ˉ = π is a strict local minimizer of g τ ( x , ⋅ ) g_\tau(x,\cdot) g τ ( x , ⋅ ) with ∇ θ g τ ( x , π ) = 0 \nabla_\theta g_\tau(x,\pi)=0 ∇ θ g τ ( x , π ) = 0 ;
(ii) its suboptimality gap satisfies
g τ ( x , π ) − g τ ∗ ( x ) ≥ u 12 + τ k ( 5 12 ) > 0 ; (246) g_\tau(x,\pi)-g_\tau^*(x)\ge\frac{u}{12}+\tau k\left(\frac{5}{12}\right)>0; \tag*{(246)} g τ ( x , π ) − g τ ∗ ( x ) ≥ 12 u + τ k ( 12 5 ) > 0 ; (246) (iii) consequently, (243) fails for every μ > 0 \mu>0 μ > 0 on any parameter set containing π \pi π and a point θ \theta θ with p ( θ ) = 1 2 p(\theta)=\frac{1}{2} p ( θ ) = 2 1 .
Here k ( 5 12 ) = 5 12 log 5 6 + 7 12 log 7 6 ≈ 0.01395 k(\frac{5}{12})=\frac{5}{12}\log\frac{5}{6}+\frac{7}{12}\log\frac{7}{6}\approx0.01395 k ( 12 5 ) = 12 5 log 6 5 + 12 7 log 6 7 ≈ 0.01395 .
Proof. (i) By the chain rule, ∇ θ g τ ( x , θ ) = h u , τ ′ ( p ( θ ) ) p ′ ( θ ) \nabla_\theta g_\tau(x,\theta)=h'_{u,\tau}(p(\theta))p'(\theta) ∇ θ g τ ( x , θ ) = h u , τ ′ ( p ( θ )) p ′ ( θ ) , and p ′ ( π ) = 0 p'(\pi)=0 p ′ ( π ) = 0 by Lemma C.53. At p = 5 12 p=\frac{5}{12} p = 12 5 , h u , τ ′ ( 5 12 ) = − u + τ log 5 7 < 0 h'_{u,\tau}(\frac{5}{12})=-u+\tau\log\frac{5}{7}<0 h u , τ ′ ( 12 5 ) = − u + τ log 7 5 < 0 , because u ≥ 0 u\ge0 u ≥ 0 , τ > 0 \tau>0 τ > 0 and log 5 7 < 0 \log\frac{5}{7}<0 log 7 5 < 0 . Differentiating once more at a critical point of p p p gives
∂ θ 2 g τ ( x , π ) = h u , τ ′ ′ ( p ( π ) ) p ′ ( π ) 2 + h u , τ ′ ( p ( π ) ) p ′ ′ ( π ) = h u , τ ′ ( 5 12 ) ⋅ ( − 1 6 ) > 0 , \partial_\theta^2 g_\tau(x,\pi)=h''_{u,\tau}(p(\pi))p'(\pi)^2+h'_{u,\tau}(p(\pi))p''(\pi)=h'_{u,\tau}\left(\frac{5}{12}\right)\cdot\left(-\frac{1}{6}\right)>0, ∂ θ 2 g τ ( x , π ) = h u , τ ′′ ( p ( π )) p ′ ( π ) 2 + h u , τ ′ ( p ( π )) p ′′ ( π ) = h u , τ ′ ( 12 5 ) ⋅ ( − 6 1 ) > 0 ,
so π \pi π is a strict local minimizer.
(ii) The range of p p p contains 1 2 \frac{1}{2} 2 1 , so g τ ∗ ( x ) ≤ h u , τ ( 1 2 ) g_\tau^*(x)\le h_{u,\tau}(\frac{1}{2}) g τ ∗ ( x ) ≤ h u , τ ( 2 1 ) . Since k ( 1 2 ) = 0 k(\frac{1}{2})=0 k ( 2 1 ) = 0 ,
h u , τ ( 5 12 ) − h u , τ ( 1 2 ) = u ( 1 2 − 5 12 ) + τ ( k ( 5 12 ) − 0 ) = u 12 + τ k ( 5 12 ) . h_{u,\tau}\left(\frac{5}{12}\right)-h_{u,\tau}\left(\frac{1}{2}\right)=u\left(\frac{1}{2}-\frac{5}{12}\right)+\tau\left(k\left(\frac{5}{12}\right)-0\right)=\frac{u}{12}+\tau k\left(\frac{5}{12}\right). h u , τ ( 12 5 ) − h u , τ ( 2 1 ) = u ( 2 1 − 12 5 ) + τ ( k ( 12 5 ) − 0 ) = 12 u + τ k ( 12 5 ) .
This quantity is positive because k ( 5 12 ) > 0 k(\frac{5}{12})>0 k ( 12 5 ) > 0 and $\tau>0.
(iii) Combine (i) and (ii) with the observation following (243) .
Degenerate policy Jacobians and multilayer networks
Lemma C.56 (Vanishing policy Jacobian forces a critical point). Let A A A be finite, the rewards bounded, and π r e f \pi_{\mathrm{ref}} π ref strictly positive. Suppose R θ = π θ / π r e f R_\theta=\pi_\theta/\pi_{\mathrm{ref}} R θ = π θ / π ref is differentiable at θ ˉ \bar{\theta} θ ˉ as a map into bounded functions with the supremum norm, and that 0 < c ≤ R θ ( s , a ) ≤ C < ∞ 0<c\le R_\theta(s,a)\le C<\infty 0 < c ≤ R θ ( s , a ) ≤ C < ∞ on a neighborhood of θ ˉ \bar{\theta} θ ˉ . If D θ π θ ˉ ( a ∣ s ) = 0 D_\theta\pi_{\bar{\theta}}(a\mid s)=0 D θ π θ ˉ ( a ∣ s ) = 0 for every ( s , a ) (s,a) ( s , a ) , then ∇ θ g τ ( x , θ ˉ ) = 0 \nabla_\theta g_\tau(x,\bar{\theta})=0 ∇ θ g τ ( x , θ ˉ ) = 0 for every x x x and finite τ ≥ 0 \tau\ge0 τ ≥ 0 . If this point is suboptimal for some x x x , uniform lower PL fails on any parameter set containing it and a strictly better point. The differentiability and ratio bounds hold for finite state spaces with smooth positive policies and for smooth bounded-input neural exponential tilts on bounded parameter neighborhoods.
Proof. The pointwise derivative condition implies that the derivative of R θ R_\theta R θ at θ ˉ \bar{\theta} θ ˉ is the zero bounded function. As a function of R R R , the state transition operator is affine and bounded in the supremum norm. Its discounted resolvent ( I − γ T π ) − 1 (I-\gamma T_\pi)^{-1} ( I − γ T π ) − 1 is Fréchet differentiable, by the resolvent identity and ∥ ( I − γ T π ) − 1 ∥ ≤ ( 1 − γ ) − 1 \lVert(I-\gamma T_\pi)^{-1}\rVert\le(1-\gamma)^{-1} ∥( I − γ T π ) − 1 ∥ ≤ ( 1 − γ ) − 1 . The expected one-step cost is
∑ a π r e f ( a ∣ s ) R ( s , a ) [ − r x ( s , a ) + τ log R ( s , a ) ] , \sum_a \pi_{\mathrm{ref}}(a\mid s)R(s,a)[-r_x(s,a)+\tau\log R(s,a)], a ∑ π ref ( a ∣ s ) R ( s , a ) [ − r x ( s , a ) + τ log R ( s , a )] ,
which is differentiable in the same norm because R R R stays in [ c , C ] [c,C] [ c , C ] . Thus the normalized discounted objective is a differentiable function of R R R , and the chain rule gives a zero parameter gradient. A strictly positive objective gap then contradicts (243) .
Proposition C.57 (Zero hidden weights are degenerate for depth at least two). Let σ \sigma σ be continuously differentiable with σ ( 0 ) = 0 \sigma(0)=0 σ ( 0 ) = 0 (e.g. tanh). For depth L ≥ 2 L\ge2 L ≥ 2 , arbitrary widths, bounded input ψ ( s , a ) \psi(s,a) ψ ( s , a ) and fixed readout b b b , consider
h 0 = ψ , h ℓ = σ ( W ℓ h ℓ − 1 ) , ℓ = 1 , … , L , h^0=\psi,\qquad h^\ell=\sigma(W_\ell h^{\ell-1}),\qquad\ell=1,\ldots,L, h 0 = ψ , h ℓ = σ ( W ℓ h ℓ − 1 ) , ℓ = 1 , … , L , F θ ( s , a ) = b T h L , π θ ( a ∣ s ) ∝ π r e f ( a ∣ s ) e F θ ( s , a ) . F_\theta(s,a)=b^{\mathsf T}h^L,\qquad\pi_\theta(a\mid s)\propto\pi_{\mathrm{ref}}(a\mid s)e^{F_\theta(s,a)}. F θ ( s , a ) = b T h L , π θ ( a ∣ s ) ∝ π ref ( a ∣ s ) e F θ ( s , a ) . with θ = ( W 1 , … , W L ) \theta=(W_1,\ldots,W_L) θ = ( W 1 , … , W L ) , and let θ ˉ \bar{\theta} θ ˉ have W ℓ = 0 W_\ell=0 W ℓ = 0 for all ℓ \ell ℓ . Then D θ F θ ˉ ( s , a ) = 0 D_\theta F_{\bar{\theta}}(s,a)=0 D θ F θ ˉ ( s , a ) = 0 and D θ π θ ˉ ( a ∣ s ) = 0 D_\theta\pi_{\bar{\theta}}(a\mid s)=0 D θ π θ ˉ ( a ∣ s ) = 0 for all ( s , a ) (s,a) ( s , a ) , and π θ ˉ = π r e f \pi_{\bar{\theta}}=\pi_{\mathrm{ref}} π θ ˉ = π ref . The same holds for L = 1 L=1 L = 1 if the readout b b b is also trainable and set to zero. Any normalization constants such as m ℓ − 1 − 1 / 2 m_{\ell-1}^{-1/2} m ℓ − 1 − 1/2 do not change the conclusion.
Proof. At θ ˉ \bar{\theta} θ ˉ , h 1 = σ ( 0 ) = 0 h^1=\sigma(0)=0 h 1 = σ ( 0 ) = 0 , and inductively h ℓ = 0 h^\ell=0 h ℓ = 0 for all ℓ ≥ 1 \ell\ge1 ℓ ≥ 1 . Backpropagation gives ∂ F / ∂ W ℓ = δ ℓ ( h ℓ − 1 ) T \partial F/\partial W_\ell=\delta_\ell(h^{\ell-1})^{\mathsf T} ∂ F / ∂ W ℓ = δ ℓ ( h ℓ − 1 ) T , where δ L = b ⊙ σ ′ ( 0 ) \delta_L=b\odot\sigma'(0) δ L = b ⊙ σ ′ ( 0 ) and δ ℓ = ( W ℓ + 1 T δ ℓ + 1 ) ⊙ σ ′ ( 0 ) \delta_\ell=(W_{\ell+1}^{\mathsf T}\delta_{\ell+1})\odot\sigma'(0) δ ℓ = ( W ℓ + 1 T δ ℓ + 1 ) ⊙ σ ′ ( 0 ) for ℓ < L \ell<L ℓ < L . For ℓ ≥ 2 \ell\ge2 ℓ ≥ 2 , h ℓ − 1 = 0 h^{\ell-1}=0 h ℓ − 1 = 0 . For ℓ = 1 \ell=1 ℓ = 1 , δ 1 \delta_1 δ 1 contains the factor W 2 T = 0 W_2^{\mathsf T}=0 W 2 T = 0 , which uses L ≥ 2 L\ge2 L ≥ 2 . Hence D θ F θ ˉ = 0 D_\theta F_{\bar{\theta}}=0 D θ F θ ˉ = 0 . Since F θ ˉ = 0 F_{\bar{\theta}}=0 F θ ˉ = 0 , π θ ˉ = π r e f \pi_{\bar{\theta}}=\pi_{\mathrm{ref}} π θ ˉ = π ref , and D θ π θ ˉ = D F π ∘ D θ F θ ˉ = 0 D_\theta\pi_{\bar{\theta}}=D_F\pi\circ D_\theta F_{\bar{\theta}}=0 D θ π θ ˉ = D F π ∘ D θ F θ ˉ = 0 . For L = 1 L=1 L = 1 with trainable b = 0 b=0 b = 0 , ∂ F / ∂ b = h 1 = 0 \partial F/\partial b=h^1=0 ∂ F / ∂ b = h 1 = 0 and ∂ F / ∂ W 1 = ( b ⊙ σ ′ ( 0 ) ) ψ T = 0 \partial F/\partial W_1=(b\odot\sigma'(0))\psi^{\mathsf T}=0 ∂ F / ∂ W 1 = ( b ⊙ σ ′ ( 0 )) ψ T = 0 .
Corollary C.58 (Uniform PL fails on domains containing zero hidden weights). In the setting of Proposition C.57, suppose that for some upper parameter x x x and some θ ∈ Θ \theta\in\Theta θ ∈ Θ , g τ ( x , θ ) < g τ ( x , θ ˉ ) g_\tau(x,\theta)<g_\tau(x,\bar{\theta}) g τ ( x , θ ) < g τ ( x , θ ˉ ) . Then uniform lower PL (243) fails on Θ \Theta Θ for every μ > 0 \mu>0 μ > 0 and every τ ≥ 0 \tau\ge0 τ ≥ 0 for which this strict inequality holds.
Proof. Combine Proposition C.57 with Lemma C.56.
A concrete instance. Take a one-state bandit with A = { − 1 , + 1 } A=\{-1,+1\} A = { − 1 , + 1 } , π r e f = 1 2 \pi_{\mathrm{ref}}=\frac{1}{2} π ref = 2 1 , r x ( a ) = u ( x ) a r_x(a)=u(x)a r x ( a ) = u ( x ) a with u ( x ) > 0 u(x)>0 u ( x ) > 0 , input ψ ( a ) = a \psi(a)=a ψ ( a ) = a , depth L = 2 L=2 L = 2 , width one, tanh activations and readout b = 1 b=1 b = 1 . Then F θ ( a ) = tanh ( w 2 tanh ( w 1 a ) ) F_\theta(a)=\tanh(w_2\tanh(w_1a)) F θ ( a ) = tanh ( w 2 tanh ( w 1 a )) , which is odd in a a a , so
π θ ( + 1 ) = σ log ( 2 tanh ( w 2 tanh w 1 ) ) , σ log ( v ) = ( 1 + e − v ) − 1 . \pi_\theta(+1)=\sigma_{\log}(2\tanh(w_2\tanh w_1)),\qquad\sigma_{\log}(v)=(1+e^{-v})^{-1}. π θ ( + 1 ) = σ l o g ( 2 tanh ( w 2 tanh w 1 )) , σ l o g ( v ) = ( 1 + e − v ) − 1 .
Writing q = π θ ( + 1 ) q=\pi_\theta(+1) q = π θ ( + 1 ) , the cost is h ~ ( q ) = − u ( 2 q − 1 ) + τ k ( q ) \tilde{h}(q)=-u(2q-1)+\tau k(q) h ~ ( q ) = − u ( 2 q − 1 ) + τ k ( q ) , with h ~ ′ ( 1 2 ) = − 2 u < 0 \tilde{h}'(\frac{1}{2})=-2u<0 h ~ ′ ( 2 1 ) = − 2 u < 0 . Taking w 1 = w 2 = w > 0 w_1=w_2=w>0 w 1 = w 2 = w > 0 small gives q ∈ ( 1 2 , 1 2 + δ ) q\in(\frac{1}{2},\frac{1}{2}+\delta) q ∈ ( 2 1 , 2 1 + δ ) for small δ \delta δ , hence g τ ( x , θ ) < h ~ ( 1 2 ) = g τ ( x , θ ˉ ) g_\tau(x,\theta)<\tilde{h}(\frac{1}{2})=g_\tau(x,\bar{\theta}) g τ ( x , θ ) < h ~ ( 2 1 ) = g τ ( x , θ ˉ ) . So θ ˉ = ( 0 , 0 ) \bar{\theta}=(0,0) θ ˉ = ( 0 , 0 ) is a critical point that is suboptimal for every x x x with u ( x ) > 0 u(x)>0 u ( x ) > 0 and every τ ≥ 0 \tau\ge0 τ ≥ 0 . When u < τ u<\tau u < τ , the regularized optimum q ⋆ = σ log ( 2 u / τ ) q^{\star}=\sigma_{\log(2u/\tau)} q ⋆ = σ l o g ( 2 u / τ ) is representable, and the gap equals τ log cosh ( u / τ ) \tau\log\cosh(u/\tau) τ log cosh ( u / τ ) .
Remark C.59 (Continuous actions). Lemma C.56 carries over to Gaussian policies N ( μ θ ( s ) , Σ ) \mathcal{N}(\mu_\theta(s),\Sigma) N ( μ θ ( s ) , Σ ) with fixed covariance, provided the occupancy and cost are differentiable in the mean map. If the mean network has all hidden weights zero and depth at least two, then D θ μ θ = 0 D_\theta\mu_\theta=0 D θ μ θ = 0 by the argument of Proposition C.57, and the same conclusion follows. This gives the analogous obstruction for fixed-covariance continuous-action policies under the stated differentiability condition.
Relation to the local assumptions
A concrete continuous-state instance with a nontrivial response correction
We give a joint satisfiability example for A1, A2, G, and C1–C2 in a class with finitely many policy contexts. It has continuous state variation in rewards and transition probabilities, a nonlinear tied actor, and five fixed critic features. An attainable preference teacher produces a finite interior stationary point of the response-induced upper objective, at which the response-chain contribution is nonzero. The population illustration below evaluates this objective independently. Appendix § 11 additionally implements sampled single-loop training on the same instance, using its two-hidden-layer actor specialization and the persistent nonlinear paired critic.
A class with finite policy contexts and Bellman closure
Proposition C.62 (A sufficient structural certificate for C1–C2). Let S \mathcal{S} S be compact, A \mathcal{A} A finite, and { S j } j = 1 M \{\mathcal{S}_j\}_{j=1}^{M} { S j } j = 1 M a measurable partition. Write j ( s ) j(s) j ( s ) for the cell index. Suppose the following conditions hold.
(i) The fixed transition kernel has a finite-rank mixture P ( d s ′ ∣ s , a ) = ∑ ℓ = 1 M ′ w ℓ ( s , a ) μ ℓ ( d s ′ ) P(ds' \mid s,a)=\sum_{\ell=1}^{M'}w_\ell(s,a)\mu_\ell(ds') P ( d s ′ ∣ s , a ) = ∑ ℓ = 1 M ′ w ℓ ( s , a ) μ ℓ ( d s ′ ) , where the μ ℓ \mu_\ell μ ℓ are probability measures and the measurable weights are nonnegative and sum to one.
(ii) The tied actor uses the bounded, fixed input ψ ( s , a ) = ψ 0 ( j ( s ) , a ) \psi(s,a)=\psi_0(j(s),a) ψ ( s , a ) = ψ 0 ( j ( s ) , a ) , with fixed hidden depth and the normalized initialization, activation, and chart conditions of Appendix § 10 .§ 1 . The reference policy depends only on ( j ( s ) , a ) (j(s),a) ( j ( s ) , a ) and is strictly positive on every such pair.
(iii) The fixed feature map φ Q : S × A → R d Q \varphi_Q : S \times A \to\mathbb{R}^{d_Q} φ Q : S × A → R d Q is bounded. Its span contains all cell–action indicators, all w ℓ w_\ell w ℓ , and every bounded reward r x r_x r x .
(iv) For some λ 0 > 0 \lambda_0 > 0 λ 0 > 0 , E ρ ( d s ) π r e f ( d a ∣ s ) [ φ Q φ Q T ] ⪰ λ 0 I d Q \mathbb{E}_{\rho(ds)\pi_{\mathrm{ref}}(da|s)}[\varphi_Q\varphi_Q^{\mathsf T}] \succeq\lambda_0 I_{d_Q} E ρ ( d s ) π ref ( d a ∣ s ) [ φ Q φ Q T ] ⪰ λ 0 I d Q .
Then C2 holds for every finite τ > 0 \tau> 0 τ > 0 , every x x x , and every finite actor parameter. On an ambient ball ∥ α ∥ ≤ A \lVert\alpha\rVert\le A ∥ α ∥ ≤ A , C1 holds with
λ Q = ( 1 − γ ) c A λ 0 , c A = exp ( − 2 C F , 1 A ) , \lambda_Q = (1-\gamma)c_A\lambda_0,\qquad c_A = \exp(-2C_{F,1}A), λ Q = ( 1 − γ ) c A λ 0 , c A = exp ( − 2 C F , 1 A ) ,
where C F , 1 C_{F,1} C F , 1 is a width-uniform bound on ∥ D α F ∥ \lVert D_\alpha F\rVert ∥ D α F ∥ there and the tied construction has F 0 = 0 F_0 = 0 F 0 = 0 .
Proof. At fixed parameters the neural score is constant on each cell–action pair. The finite-action log normalizer is therefore constant within each state cell. Both terms of log ( π α / π r e f ) \log(\pi_\alpha/\pi_{\mathrm{ref}}) log ( π α / π ref ) belong to the cell-indicator span. The regularized cost q x , α = − r x + τ log ( π α / π r e f ) q_{x,\alpha}=-r_x+\tau\log(\pi_\alpha/\pi_{\mathrm{ref}}) q x , α = − r x + τ log ( π α / π ref ) thus belongs to V = span φ Q V=\operatorname{span}\varphi_Q V = span φ Q .
For every bounded v v v ,
K α v ( s , a ) = ∑ ℓ = 1 M ′ w ℓ ( s , a ) ∫ ∑ a ′ π α ( a ′ ∣ s ′ ) v ( s ′ , a ′ ) μ ℓ ( d s ′ ) ∈ V . K_\alpha v(s,a)=\sum_{\ell=1}^{M'}w_\ell(s,a)\int\sum_{a'}\pi_\alpha(a'|s')v(s',a')\,\mu_\ell(ds')\in V. K α v ( s , a ) = ℓ = 1 ∑ M ′ w ℓ ( s , a ) ∫ a ′ ∑ π α ( a ′ ∣ s ′ ) v ( s ′ , a ′ ) μ ℓ ( d s ′ ) ∈ V .
The Neumann series ∑ k ≥ 0 γ k K α k q x , α \sum_{k\ge0}\gamma^kK_\alpha^kq_{x,\alpha} ∑ k ≥ 0 γ k K α k q x , α converges uniformly and stays in the closed finite-dimensional space V V V . This proves C2. On the ambient ball, ∣ F α ∣ ≤ C F , 1 A |F_\alpha|\le C_{F,1}A ∣ F α ∣ ≤ C F , 1 A and π α / π r e f ≥ c A \pi_\alpha/\pi_{\mathrm{ref}}\ge c_A π α / π ref ≥ c A . Therefore ρ U ≥ c A ρ π r e f \rho_U\ge c_A\rho_{\pi_{\mathrm{ref}}} ρ U ≥ c A ρ π ref , and ν U ≥ ( 1 − γ ) ρ U \nu_U\ge(1-\gamma)\rho_U ν U ≥ ( 1 − γ ) ρ U transfers the reference Gram bound to C1. □
The bounded-feature condition is needed for C1; compactness alone does not bound an arbitrary measurable feature map. The normalized network conditions in the proposition supply A1(iii). Reward and preference regularity and A2 remain separate requirements, verified for the instance below. This proposition is a sufficient certificate within our broader assumption-based theorem.
Proposition C.63 (A log-normalizer obstruction under finite-rank dynamics). Let S = [ − 1 , 1 ] S=[-1,1] S = [ − 1 , 1 ] , A = { − 1 , + 1 } A=\{-1,+1\} A = { − 1 , + 1 } , π r e f = 1 / 2 \pi_{\mathrm{ref}}=1/2 π ref = 1/2 , and F α ( s , a ) = a α s F_\alpha(s,a)=a\alpha s F α ( s , a ) = a α s , for α \alpha α in an open interval containing zero. No fixed finite-dimensional subspace of bounded functions contains log ( π α / π r e f ) \log(\pi_\alpha/\pi_{\mathrm{ref}}) log ( π α / π ref ) for every such α \alpha α . If, in addition, the transition kernel has a fixed finite-rank mixture as in Proposition C.62, then, for every fixed τ > 0 \tau>0 τ > 0 and fixed bounded reward, no fixed finite feature space can represent all corresponding regularized values.
Proof. The log-policy ratio equals a α s − log cosh ( α s ) a\alpha s-\log\cosh(\alpha s) a α s − log cosh ( α s ) . If these functions belonged to a finite-dimensional space V V V , uniform finite differences and closedness would put all their α \alpha α -derivatives at zero in V V V . Uniform convergence of these differences follows from smoothness on a compact neighborhood in ( α , s ) (\alpha,s) ( α , s ) .
Every even Taylor coefficient of log cosh y \log\cosh y log cosh y is nonzero. For completeness, write tanh y = ∑ k ≥ 1 ( − 1 ) k − 1 a k y 2 k − 1 \tanh y=\sum_{k\ge1}(-1)^{k-1}a_k y^{2k-1} tanh y = ∑ k ≥ 1 ( − 1 ) k − 1 a k y 2 k − 1 near zero. The identity ( tanh y ) ′ = 1 − tanh 2 y (\tanh y)'=1-\tanh^2y ( tanh y ) ′ = 1 − tanh 2 y gives a 1 = 1 a_1=1 a 1 = 1 and ( 2 k − 1 ) a k = ∑ j = 1 k − 1 a j a k − j > 0 (2k-1)a_k=\sum_{j=1}^{k-1}a_ja_{k-j}>0 ( 2 k − 1 ) a k = ∑ j = 1 k − 1 a j a k − j > 0 for k ≥ 2 k\ge2 k ≥ 2 . Integration gives nonzero coefficients ( − 1 ) k − 1 a k / ( 2 k ) (-1)^{k-1}a_k/(2k) ( − 1 ) k − 1 a k / ( 2 k ) in log cosh y \log\cosh y log cosh y . Thus V V V would contain every s 2 k s^{2k} s 2 k , k ≥ 1 k\ge1 k ≥ 1 , contradicting their linear independence.
For the value claim let W = span { w ℓ } ℓ = 1 M ′ W=\operatorname{span}\{w_\ell\}_{\ell=1}^{M'} W = span { w ℓ } ℓ = 1 M ′ . The finite-rank kernel implies K α v ∈ W K_\alpha v\in W K α v ∈ W for every bounded v v v . If all Q α Q_\alpha Q α belonged to a fixed finite-dimensional V V V , the Bellman equation would imply
τ log ( π α / π r e f ) = Q α − γ K α Q α + r ∈ V + W + span { r } , \tau\log(\pi_\alpha/\pi_{\mathrm{ref}})=Q_\alpha-\gamma K_\alpha Q_\alpha+r\in V+W+\operatorname{span}\{r\}, τ log ( π α / π ref ) = Q α − γ K α Q α + r ∈ V + W + span { r } , contradicting the first part. □
Remark C.64 (The obstruction is not a necessity theorem for finite contexts.) The value conclusion above uses the finite-rank transition condition; failure of log-ratio membership alone does not imply failure of C2 for an arbitrary policy-dependent Bellman operator. Nor does this binary-logit example characterize every finite-action policy family. For instance, with three actions, let q ( s ) = 1 / 2 + s / 4 q(s) = 1/2+s/4 q ( s ) = 1/2 + s /4 and π r e f ( s ) = ( q ( s ) / 2 , ( 1 − q ( s ) ) / 2 , 1 / 2 ) \pi_{\mathrm{ref}}(s) = (q(s)/2, (1-q(s))/2, 1/2) π ref ( s ) = ( q ( s ) /2 , ( 1 − q ( s )) /2 , 1/2 ) . Take v = ( 1 , 1 , − 1 ) \mathbf{v} = (1,1,-1) v = ( 1 , 1 , − 1 ) and F α ( s , a ) = h ( α ) v a F_{\alpha}(s,a) = h(\alpha)v_a F α ( s , a ) = h ( α ) v a . The log normalizer is log cosh h ( α ) \log\cosh h(\alpha) log cosh h ( α ) , independent of s s s , so the log-policy ratio lies in span { 1 , v } \operatorname{span}\{1,\mathbf{v}\} span { 1 , v } although the policy varies continuously with s s s . With reset dynamics P ( ⋅ ∣ s , a ) = ρ P(\cdot\mid s,a) = \rho P ( ⋅ ∣ s , a ) = ρ , reward tanh ( x ) s v a \tanh(x)s v_a tanh ( x ) s v a , and features ( 1 , v a , s v a ) (1,v_a,sv_a) ( 1 , v a , s v a ) , C2 follows immediately. Under uniform ρ \rho ρ the reference Gram matrix is diag ( 1 , 1 , 1 / 3 ) \operatorname{diag}(1,1,1/3) diag ( 1 , 1 , 1/3 ) , so C1 also holds on bounded actor neighborhoods. Finite policy contexts are consequently a useful sufficient construction, not an additional necessary restriction on the general theorem.
The controlled process and preference model
Fix any discount factor γ ∈ ( 0 , 1 ) \gamma\in(0,1) γ ∈ ( 0 , 1 ) . Set
S = [ − 1 , 1 ] , A = { − 1 , + 1 } , ρ = Unif [ − 1 , 1 ] . \mathcal{S} = [-1,1], \qquad\mathcal{A} = \{-1,+1\}, \qquad\rho= \operatorname{Unif}[-1,1]. S = [ − 1 , 1 ] , A = { − 1 , + 1 } , ρ = Unif [ − 1 , 1 ] .
Use the Borel sigma algebra on S \mathcal{S} S . Define
I − = [ − 1 , 0 ) , I + = [ 0 , 1 ] , i ( s ) = { − 1 , s < 0 , + 1 , s ≥ 0. I_- = [-1,0), \qquad I_+ = [0,1], \qquad i(s) =
\begin{cases}
-1, & s < 0,\\
+1, & s \ge0.
\end{cases} I − = [ − 1 , 0 ) , I + = [ 0 , 1 ] , i ( s ) = { − 1 , + 1 , s < 0 , s ≥ 0.
and let μ − \mu_- μ − and μ + \mu_+ μ + be the uniform probability measures on I − I_- I − and I + I_+ I + , respectively. The endpoint convention is immaterial for these measures and fixes all functions pointwise. Define the transition kernel
P ( d s ′ ∣ s , a ) = ( 1 2 + a s 4 ) μ + ( d s ′ ) + ( 1 2 − a s 4 ) μ − ( d s ′ ) . (247) P(ds' \mid s,a) = \left(\frac{1}{2}+\frac{as}{4}\right)\mu_+(ds') + \left(\frac{1}{2}-\frac{as}{4}\right)\mu_-(ds'). \tag*{(247)} P ( d s ′ ∣ s , a ) = ( 2 1 + 4 a s ) μ + ( d s ′ ) + ( 2 1 − 4 a s ) μ − ( d s ′ ) . (247)
Both mixture weights lie in [ 1 / 4 , 3 / 4 ] [1/4,3/4] [ 1/4 , 3/4 ] . Thus this is a measurable probability kernel with continuous state distributions. Its mixture weights depend continuously on s s s and depend on the action. The reset law is ρ = ( μ − + μ + ) / 2 \rho= (\mu_-+\mu_+)/2 ρ = ( μ − + μ + ) /2 .
The upper parameter is x ∈ R x \in\mathbb{R} x ∈ R . Write u ( x ) = tanh x u(x) = \tanh x u ( x ) = tanh x and set
r x ( s , a ) = u ( x ) a s . (248) r_x(s,a) = u(x)as. \tag*{(248)} r x ( s , a ) = u ( x ) a s . (248)
Only the reward model and the fitted preference loss depend on x x x ; the transition kernel, reset law, and reference policy remain fixed.
For the upper objective, independently sample two trajectories ξ 1 , ξ 2 \xi^{1}, \xi^{2} ξ 1 , ξ 2 , each with K = 2 K = 2 K = 2 state–action pairs, using the current actor, the reset law, and (247) . Put
D ( ξ ) = ∑ t = 0 1 ( a t 1 s t 1 − a t 2 s t 2 ) , Δ x = u ( x ) D ( ξ ) , P ( Y = 1 ∣ ξ ) = σ ( κ D ( ξ ) ) , σ ( v ) = ( 1 + e − v ) − 1 , (249) \begin{aligned}
D(\xi) &= \sum_{t=0}^{1}(a_t^{1}s_t^{1}-a_t^{2}s_t^{2}), \qquad&\Delta_x &= u(x)D(\xi),\\
\mathbb{P}(Y=1\mid\xi) &= \sigma(\kappa D(\xi)), \qquad&\sigma(v) &= (1+e^{-v})^{-1}, \tag*{(249)}
\end{aligned} D ( ξ ) P ( Y = 1 ∣ ξ ) = t = 0 ∑ 1 ( a t 1 s t 1 − a t 2 s t 2 ) , = σ ( κ D ( ξ )) , Δ x σ ( v ) = u ( x ) D ( ξ ) , = ( 1 + e − v ) − 1 , (249)
L ( x ; ξ , Y ) = log ( 1 + e Δ x ) − Y Δ x . (250) L(x;\xi,Y)=\log(1+e^{\Delta_x})-Y\Delta_x. \tag*{(250)} L ( x ; ξ , Y ) = log ( 1 + e Δ x ) − Y Δ x . (250)
Fix κ ∈ ( 0 , 1 ) \kappa\in(0,1) κ ∈ ( 0 , 1 ) , with κ = 1 / 2 \kappa= 1/2 κ = 1/2 in the numerical illustration. Here the teacher reward is the fixed function r † ( s , a ) = κ a s r^{\dagger}(s,a)=\kappa as r † ( s , a ) = κa s . Conditional on the trajectories, the label law has no dependence on x x x or on the actor parameter. The loss has no additional explicit actor dependence. Fresh tuples and labels can be drawn independently.
A tied neural actor with an explicit initialization gap
Use π r e f ( a ∣ s ) = 1 / 2 \pi_{\mathrm{ref}}(a\mid s)=1/2 π ref ( a ∣ s ) = 1/2 . For every integer n ≥ 1 n\geq1 n ≥ 1 and retained parameter α = ( α − , α + ) ∈ R 2 \alpha=(\alpha_{-},\alpha_{+})\in\mathbb{R}^{2} α = ( α − , α + ) ∈ R 2 , define
h n ( t ) = n tanh ( t 2 n ) , F n , α ( s , a ) = a h n ( α i ( s ) ) . (251) h_n(t)=\sqrt{n}\tanh\left(\frac{t}{2\sqrt{n}}\right), \qquad F_{n,\alpha}(s,a)=a h_n(\alpha_{i(s)}). \tag*{(251)} h n ( t ) = n tanh ( 2 n t ) , F n , α ( s , a ) = a h n ( α i ( s ) ) . (251)
π n , α ( a ∣ s ) = exp { a h n ( α i ( s ) ) } 2 cosh ( h n ( α i ( s ) ) ) . (252) \pi_{n,\alpha}(a\mid s)=\frac{\exp\{a h_n(\alpha_{i(s)})\}}{2\cosh(h_n(\alpha_{i(s)}))}. \tag*{(252)} π n , α ( a ∣ s ) = 2 cosh ( h n ( α i ( s ) )) exp { a h n ( α i ( s ) )} . (252)
Write π n , α ( a ∣ i ) \pi_{n,\alpha}(a\mid i) π n , α ( a ∣ i ) for this region-constant probability. The two regions have independently trainable action preferences. In particular, the policy is state dependent whenever h n ( α − ) ≠ h n ( α + ) h_n(\alpha_{-})\ne h_n(\alpha_{+}) h n ( α − ) = h n ( α + ) .
Lemma C.66 (Realization by our tied architecture). The score in (251) is an exact one-hidden-layer specialization of the normalized antithetic actor, with input dimension 2, hidden width 2 n 2n 2 n , and retained dimension 2. It satisfies the actor part of A1 uniformly in n n n and satisfies A2 with λ e f f , L = 1 / 8 \lambda_{\mathrm{eff},L}=1/8 λ eff , L = 1/8 .
Proof. Let e − , e + \mathbf{e}_{-},\mathbf{e}_{+} e − , e + be the coordinate vectors in R 2 \mathbb{R}^{2} R 2 and use the bounded input ψ ( s , a ) = a e i ( s ) \psi(s,a)=a\mathbf{e}_{i(s)} ψ ( s , a ) = a e i ( s ) . Index the 2 n 2n 2 n hidden units by ( i , k ) (i,k) ( i , k ) , where i ∈ { − 1 , + 1 } i\in\{-1,+1\} i ∈ { − 1 , + 1 } and k = 1 , … , n k=1,\ldots,n k = 1 , … , n . For a 2 n 2n 2 n -by-2 weight matrix W W W , use the normalized branch
G n , W ( s , a ) = 1 2 n 1 2 n T tanh ( W ψ ( s , a ) 2 ) . G_{n,W}(s,a)=\frac{1}{\sqrt{2n}}\mathbf{1}_{2n}^{\mathsf{T}}\tanh\left(\frac{W\psi(s,a)}{\sqrt{2}}\right). G n , W ( s , a ) = 2 n 1 1 2 n T tanh ( 2 W ψ ( s , a ) ) .
The activation acts coordinatewise and the readout is fixed. Initialize W 0 = 0 W_{0}=0 W 0 = 0 and define the fixed chart B n B_n B n by
( B n α ) ( i , k ) , j = α i n 1 { j = i } . (B_n\alpha)_{(i,k),j}=\frac{\alpha_i}{\sqrt{n}}\mathbf{1}_{\{j=i\}}. ( B n α ) ( i , k ) , j = n α i 1 { j = i } .
Then ∥ B n α ∥ F 2 = α − 2 + α + 2 \lVert B_n\alpha\rVert_{F}^{2}=\alpha_{-}^{2}+\alpha_{+}^{2} ∥ B n α ∥ F 2 = α − 2 + α + 2 , so B n B_n B n is an isometry. Take the two branches W ± ( α ) = ± B n α / 2 W^{\pm}(\alpha)=\pm B_n\alpha/\sqrt{2} W ± ( α ) = ± B n α / 2 and their antithetic score F = ( G n , W + − G n , W − ) / 2 F=(G_{n,W^{+}}-G_{n,W^{-}})/\sqrt{2} F = ( G n , W + − G n , W − ) / 2 . At an input in region i i i , exactly n n n units have nonzero preactivation, equal to ± a α i / ( 2 n ) \pm a\alpha_i/(2\sqrt{n}) ± a α i / ( 2 n ) . Oddness of tanh gives
G n , W ± ( s , a ) = ± a n 2 tanh ( α i 2 n ) , G_{n,W^{\pm}}(s,a)=\pm a\sqrt{\frac{n}{2}}\tanh\left(\frac{\alpha_i}{2\sqrt{n}}\right), G n , W ± ( s , a ) = ± a 2 n tanh ( 2 n α i ) ,
and therefore gives exactly (251) . The initialization satisfies the normalized spectral requirement, and all hidden weights vary through the specified fixed chart.
For k = 1 , … , 4 k=1,\ldots,4 k = 1 , … , 4 ,
h n ( k ) ( t ) = 2 − k n ( 1 − k ) / 2 tanh ( k ) ( t 2 n ) . (253) h_n^{(k)}(t)=2^{-k}n^{(1-k)/2}\tanh^{(k)}\left(\frac{t}{2\sqrt{n}}\right). \tag*{(253)} h n ( k ) ( t ) = 2 − k n ( 1 − k ) /2 tanh ( k ) ( 2 n t ) . (253)
All derivatives of tanh \tanh tanh in this display are bounded on R \mathbb{R} R . Since n ( 1 − k ) / 2 ≤ 1 n^{(1-k)/2} \le1 n ( 1 − k ) /2 ≤ 1 , the required operator norms of D α k F n , α D_{\alpha}^{k}F_{n,\alpha} D α k F n , α are bounded independently of n n n , states, actions, and points in any fixed ambient ball. Also F n , 0 = 0 F_{n,0}=0 F n , 0 = 0 , D α 2 F n , 0 = 0 D_{\alpha}^{2}F_{n,0}=0 D α 2 F n , 0 = 0 , and h n ′ ( 0 ) = 1 / 2 h_n^{\prime}(0)=1/2 h n ′ ( 0 ) = 1/2 . Thus the initialization feature is
Z n , 0 ( s , a ) = D α F n , 0 ( s , a ) = a 2 e i ( s ) . Z_{n,0}(s,a)=D_{\alpha}F_{n,0}(s,a)=\frac{a}{2}e_i(s). Z n , 0 ( s , a ) = D α F n , 0 ( s , a ) = 2 a e i ( s ) .
Its conditional mean under the uniform reference policy is zero, and its conditional covariance is e i e i T / 4 e_i e_i^{T}/4 e i e i T /4 in region i i i . Both regions have reset probability 1 / 2 1/2 1/2 , whence
G n , 0 : = E s ∼ ρ Cov a ∼ π r e f ( ⋅ ∣ s ) ( Z n , 0 ( s , a ) ) = 1 8 I 2 . (254) \mathcal{G}_{n,0}:=\mathbb{E}_{s\sim\rho}\operatorname{Cov}_{a\sim\pi_{\mathrm{ref}}(\cdot\mid s)}(Z_{n,0}(s,a))=\frac{1}{8}I_2. \tag*{(254)} G n , 0 := E s ∼ ρ Cov a ∼ π ref ( ⋅ ∣ s ) ( Z n , 0 ( s , a )) = 8 1 I 2 . (254)
Lemma C.67 (Reward and upper-loss regularity). The reward and sample loss satisfy the remaining parts of A1 and all of G on the unrestricted upper domain R \mathbb{R} R , with
B r = 1 , G r = 1 , L r = 2 , B L = log 2 + 4 , G L = 4 , L L = 12. (255) B_r=1,\qquad G_r=1,\qquad L_r=2,\qquad B_L=\log2+4,\qquad G_L=4,\qquad L_L=12. \tag*{(255)} B r = 1 , G r = 1 , L r = 2 , B L = log 2 + 4 , G L = 4 , L L = 12. (255)
Proof. The bounds ∣ u ∣ ≤ 1 |u|\le1 ∣ u ∣ ≤ 1 , ∣ u ′ ∣ ≤ 1 |u'|\le1 ∣ u ′ ∣ ≤ 1 , and ∣ u ′ ′ ∣ ≤ 2 |u''|\le2 ∣ u ′′ ∣ ≤ 2 , together with ∣ a s ∣ ≤ 1 |a s|\le1 ∣ a s ∣ ≤ 1 , give the reward constants. The reward is C ∞ C^\infty C ∞ in x x x , with bounded derivatives through order four uniformly in ( s , a ) (s,a) ( s , a ) .
Since ∣ D ∣ ≤ 4 |D|\le4 ∣ D ∣ ≤ 4 , we have ∣ Δ x ∣ ≤ 4 |\Delta_x|\le4 ∣ Δ x ∣ ≤ 4 , ∣ ∂ x Δ x ∣ ≤ 4 |\partial_x\Delta_x|\le4 ∣ ∂ x Δ x ∣ ≤ 4 , and ∣ ∂ x 2 Δ x ∣ ≤ 8 |\partial_x^2\Delta_x|\le8 ∣ ∂ x 2 Δ x ∣ ≤ 8 . For each realized Y ∈ { 0 , 1 } Y\in\{0,1\} Y ∈ { 0 , 1 } ,
0 ≤ L ≤ log 2 + ∣ Δ x ∣ , 0\le L\le\log2+|\Delta_x|, 0 ≤ L ≤ log 2 + ∣ Δ x ∣ ,
∂ x L = ( σ ( Δ x ) − Y ) ∂ x Δ x , \partial_x L=(\sigma(\Delta_x)-Y)\partial_x\Delta_x, ∂ x L = ( σ ( Δ x ) − Y ) ∂ x Δ x ,
∂ x 2 L = σ ′ ( Δ x ) ( ∂ x Δ x ) 2 + ( σ ( Δ x ) − Y ) ∂ x 2 Δ x . \partial_x^2 L=\sigma'(\Delta_x)(\partial_x\Delta_x)^2+(\sigma(\Delta_x)-Y)\partial_x^2\Delta_x. ∂ x 2 L = σ ′ ( Δ x ) ( ∂ x Δ x ) 2 + ( σ ( Δ x ) − Y ) ∂ x 2 Δ x .
Use ∣ σ − Y ∣ ≤ 1 |\sigma-Y|\le1 ∣ σ − Y ∣ ≤ 1 and 0 ≤ σ ′ ≤ 1 / 4 0\le\sigma'\le1/4 0 ≤ σ ′ ≤ 1/4 to obtain (255) . Composition with the smooth softplus function, with ∣ D ∣ ≤ 4 |D|\le4 ∣ D ∣ ≤ 4 , also supplies the samplewise C 4 C^4 C 4 bounds required by A1. Equation (249) provides G’s parameter-independent conditional feedback law; the sampling construction provides its fixed tuple size, finite trajectory length, and fresh sample access.
Remark C.68 (A concrete two-hidden-layer variant). Cell constancy is preserved at every fixed hidden depth, but A2 must still be checked. In particular, initializing all hidden weights at zero with tanh \tanh tanh gives zero parameter gradients at depth at least two, and therefore fails A2.
Here is an explicit alternative that trains both hidden layers and has a gap independent of width. Give each hidden layer width 2 n 2n 2 n , with n n n units for each region. Use the same input and fixed all-ones readout as above. Set W 0 1 = 0 W_0^1=0 W 0 1 = 0 and let W 0 2 W_0^2 W 0 2 be block diagonal, each n n n -by-n n n block having every entry 2 / n \sqrt{2/n} 2/ n . Then ∥ W 0 2 / 2 n ∥ o p = 1 \lVert W_0^2/\sqrt{2n}\rVert_{\mathrm{op}}=1 ∥ W 0 2 / 2 n ∥ op = 1 . Define a product-space chart by
( B n 1 α ) ( i , k ) , j = α i 2 n 1 { j = i } , ( B n 2 α ) ( i , k ) , ( j , ℓ ) = α i 2 n 1 { j = i } . (B_n^1\alpha)_{(i,k),j}=\frac{\alpha_i}{\sqrt{2n}}\mathbf{1}\{j=i\},\qquad
(B_n^2\alpha)_{(i,k),(j,\ell)}=\frac{\alpha_i}{\sqrt{2n}}\mathbf{1}\{j=i\}. ( B n 1 α ) ( i , k ) , j = 2 n α i 1 { j = i } , ( B n 2 α ) ( i , k ) , ( j , ℓ ) = 2 n α i 1 { j = i } .
Each chart component has squared Frobenius norm ∥ α ∥ 2 / 2 \lVert\alpha\rVert^2/2 ∥ α ∥ 2 /2 , so the combined chart is an isometry. Use W ℓ ± = W ℓ 0 ± B n ℓ α / 2 W_\ell^\pm=W_\ell^0\pm B_n^\ell\alpha/\sqrt{2} W ℓ ± = W ℓ 0 ± B n ℓ α / 2 , our layer normalizations, and its antithetic readout. Writing t i = α i / ( 2 2 n ) t_i=\alpha_i/(2\sqrt{2n}) t i = α i / ( 2 2 n ) , direct substitution gives
F n , α ( 2 ) ( s , a ) = a H n ( α i ( s ) ) , H n ( α i ) = n 2 [ tanh ( ( 1 + t i ) tanh t i ) + tanh ( ( 1 − t i ) tanh t i ) ] . (256) F_{n,\alpha}^{(2)}(s,a)=aH_n(\alpha_i(s)),\qquad
H_n(\alpha_i)=\frac{\sqrt{n}}{2}\left[\tanh((1+t_i)\tanh t_i)+\tanh((1-t_i)\tanh t_i)\right]. \tag*{(256)} F n , α ( 2 ) ( s , a ) = a H n ( α i ( s )) , H n ( α i ) = 2 n [ tanh (( 1 + t i ) tanh t i ) + tanh (( 1 − t i ) tanh t i ) ] . (256)
Thus H n ′ ( 0 ) = 1 / ( 2 2 ) H_n'(0)=1/(2\sqrt{2}) H n ′ ( 0 ) = 1/ ( 2 2 ) and G n , 0 ( 2 ) = I 2 / 16 \mathcal{G}_{n,0}^{(2)}=I_2/16 G n , 0 ( 2 ) = I 2 /16 , for every n n n . The normalized initialization and the isometric chart supply the width-uniform derivative bounds. By uniform continuity of the first derivative, H n H_n H n is strictly increasing on a common neighborhood of zero. Choosing τ \tau τ sufficiently large puts the target logits i tanh ( x ) / ( 2 τ ) i\tanh(x)/(2\tau) i tanh ( x ) / ( 2 τ ) in its local image. The local response is then α i ∗ ( x ) = H n − 1 ( i tanh ( x ) / ( 2 τ ) ) \alpha_i^*(x)=H_n^{-1}(i\tanh(x)/(2\tau)) α i ∗ ( x ) = H n − 1 ( i tanh ( x ) / ( 2 τ )) . The same five critic features and all Bellman-closure arguments remain valid. The induced response policy, and hence the upper objective and the numerical computations below, are identical to those for the one-hidden-layer parameterization whenever this local response is used.
Five fixed critic features
Define
z ( s ) = 12 ( s − i ( s ) 2 ) , ϕ Q ( s , a ) = ( e i ( s ) , a a z ( s ) ) ∈ R 5 , (257) z(s)=\sqrt{12}\left(s-\frac{i(s)}{2}\right),\qquad\phi_Q(s,a)=\begin{pmatrix}e^{i(s),a}\\ az(s)\end{pmatrix}\in\mathbb{R}^5, \tag*{(257)} z ( s ) = 12 ( s − 2 i ( s ) ) , ϕ Q ( s , a ) = ( e i ( s ) , a a z ( s ) ) ∈ R 5 , (257)
where the first four coordinates are ordered as ( − 1 , − 1 ) (-1,-1) ( − 1 , − 1 ) , ( − 1 , + 1 ) (-1,+1) ( − 1 , + 1 ) , ( + 1 , − 1 ) (+1,-1) ( + 1 , − 1 ) , ( + 1 , + 1 ) (+1,+1) ( + 1 , + 1 ) . These features are fixed independently of x x x , α \alpha α , n n n , the critic scale m Q m_Q m Q , and target accuracy. On either region,
∣ z ( s ) ∣ ≤ 3 , E μ i z = 0 , E μ i z 2 = 1. (258) \lvert z(s)\rvert\le\sqrt{3},\qquad\mathbb{E}_{\mu_i}z=0,\qquad\mathbb{E}_{\mu_i}z^2=1. \tag*{(258)} ∣ z ( s )∣ ≤ 3 , E μ i z = 0 , E μ i z 2 = 1. (258)
Consequently ∥ ϕ Q ( s , a ) ∥ 2 ≤ 2 \lVert\phi_Q(s,a)\rVert_2\le2 ∥ ϕ Q ( s , a ) ∥ 2 ≤ 2 .
For a fixed actor width, suppress n n n in the transition-operator notation and set
( K α v ) ( s , a ) = E s ′ ∼ P ( ⋅ ∣ s , a ) ∑ a ′ ∈ A π n , α ( a ′ ∣ s ′ ) v ( s ′ , a ′ ) , (K_\alpha v)(s,a)=\mathbb{E}_{s'\sim P(\cdot\mid s,a)}\sum_{a'\in\mathcal{A}}\pi_{n,\alpha}(a'\mid s')v(s',a'), ( K α v ) ( s , a ) = E s ′ ∼ P ( ⋅ ∣ s , a ) a ′ ∈ A ∑ π n , α ( a ′ ∣ s ′ ) v ( s ′ , a ′ ) ,
ρ U ( d s , d a ) = ρ ( d s ) π n , U ( d a ∣ s ) , ν U = ( 1 − γ ) ∑ k = 0 ∞ γ k ρ U K U k . (259) \rho_U(ds,da)=\rho(ds)\pi_{n,U}(da\mid s),\qquad\nu_U=(1-\gamma)\sum_{k=0}^{\infty}\gamma^k\rho_UK_U^k. \tag*{(259)} ρ U ( d s , d a ) = ρ ( d s ) π n , U ( d a ∣ s ) , ν U = ( 1 − γ ) k = 0 ∑ ∞ γ k ρ U K U k . (259)
Lemma C.69 (C1: uniform feature excitation). For any fixed ambient actor radius A > 0 A>0 A > 0 , C1 holds throughout ∥ U ∥ 2 ≤ A \lVert U\rVert_2\le A ∥ U ∥ 2 ≤ A , uniformly in n ≥ 1 n\ge1 n ≥ 1 , with
d Q = 5 , F Q = 2 , λ Q = 1 − γ 2 ( 1 + e A ) . (260) d_Q=5,\qquad F_Q=2,\qquad\lambda_Q=\frac{1-\gamma}{2(1+e^A)}. \tag*{(260)} d Q = 5 , F Q = 2 , λ Q = 2 ( 1 + e A ) 1 − γ . (260)
Proof. The inequality ∣ tanh t ∣ ≤ ∣ t ∣ \lvert\tanh t\rvert\le\lvert t\rvert ∣ tanh t ∣ ≤ ∣ t ∣ gives ∣ h n ( U i ) ∣ ≤ ∣ U i ∣ / 2 ≤ A / 2 \lvert h_n(U_i)\rvert\le\lvert U_i\rvert/2\le A/2 ∣ h n ( U i )∣ ≤ ∣ U i ∣ /2 ≤ A /2 , so
π n , U ( a ∣ i ) ≥ p A : = 1 1 + e A . \pi_{n,U}(a\mid i)\ge p_A:=\frac{1}{1+e^A}. π n , U ( a ∣ i ) ≥ p A := 1 + e A 1 .
Within each region the policy is constant in s s s . Therefore the zero mean in (258) makes the cross moments between the first four features and the fifth feature vanish. The reset Gram matrix is exactly
E ρ U [ ϕ Q ϕ Q T ] = diag ( { 1 2 π n , U ( a ∣ i ) } i , a , 1 ) ≥ p A 2 I 5 . \mathbb{E}_{\rho_U}[\phi_Q\phi_Q^{\mathsf{T}}]=\operatorname{diag}\left(\left\{\frac{1}{2}\pi_{n,U}(a\mid i)\right\}_{i,a},1\right)\ge\frac{p_A}{2}I_5. E ρ U [ ϕ Q ϕ Q T ] = diag ( { 2 1 π n , U ( a ∣ i ) } i , a , 1 ) ≥ 2 p A I 5 .
Since ν U ≥ ( 1 − γ ) ρ U \nu_U\ge(1-\gamma)\rho_U ν U ≥ ( 1 − γ ) ρ U as measures, integration of the positive semidefinite matrix ϕ Q ϕ Q T \phi_Q\phi_Q^{\mathsf{T}} ϕ Q ϕ Q T gives (260) . This covers every smaller base-actor neighborhood used in Theorem 4.6 . □ \square □
Lemma C.70 (C2: exact realizability of the regularized value). For every finite τ > 0 \tau>0 τ > 0 , x ∈ R x\in\mathbb{R} x ∈ R , n ≥ 1 n\ge1 n ≥ 1 , and α ∈ R 2 \alpha\in\mathbb{R}^2 α ∈ R 2 , the regularized lower action value
q x , α ( s , a ) = − r x ( s , a ) + τ log π n , α ( a ∣ s ) π r e f ( a ∣ s ) , q_{x,\alpha}(s,a)=-r_x(s,a)+\tau\log\frac{\pi_{n,\alpha}(a\mid s)}{\pi_{\mathrm{ref}}(a\mid s)}, q x , α ( s , a ) = − r x ( s , a ) + τ log π ref ( a ∣ s ) π n , α ( a ∣ s ) ,
Q x , α t r u e = ( Id − γ K α ) − 1 q x , α (261) Q_{x,\alpha}^{\mathrm{true}}=(\operatorname{Id}-\gamma K_\alpha)^{-1}q_{x,\alpha} \tag*{(261)} Q x , α true = ( Id − γ K α ) − 1 q x , α (261)
belongs to span φ Q \varphi_Q φ Q . On ∥ α ∥ 2 ≤ A \lVert\alpha\rVert_2 \le A ∥ α ∥ 2 ≤ A ,
∥ q x , α ∥ ∞ ≤ 1 + τ A , ∥ Q x , α t r u e ∥ ∞ ≤ 1 + τ A 1 − γ , (262) \lVert q_{x,\alpha}\rVert_\infty\le1+\tau A,\qquad\lVert Q^{\mathrm{true}}_{x,\alpha}\rVert_\infty\le\frac{1+\tau A}{1-\gamma}, \tag*{(262)} ∥ q x , α ∥ ∞ ≤ 1 + τ A , ∥ Q x , α true ∥ ∞ ≤ 1 − γ 1 + τ A , (262)
uniformly in x x x and n n n .
Proof. First,
a s = a i ( s ) 2 + a z ( s ) 12 . (263) a_s=\frac{a\,i(s)}{2}+\frac{a z(s)}{\sqrt{12}}. \tag*{(263)} a s = 2 a i ( s ) + 12 a z ( s ) . (263)
The first term is constant on each region–action pair and the second is a multiple of the fifth feature. Thus r x ∈ span φ Q r_x\in\operatorname{span}\varphi_Q r x ∈ span φ Q . Also
log π n , α ( a ∣ s ) π r e f ( a ∣ s ) = a h n ( α i ( s ) ) − log cosh ( h n ( α i ( s ) ) ) , (264) \log\frac{\pi_{n,\alpha}(a\mid s)}{\pi_{\mathrm{ref}}(a\mid s)}
=a h_n(\alpha i_{(s)})-\log\cosh(h_n(\alpha i_{(s)})), \tag*{(264)} log π ref ( a ∣ s ) π n , α ( a ∣ s ) = a h n ( α i ( s ) ) − log cosh ( h n ( α i ( s ) )) , (264)
which is constant on each region–action pair. In particular q x , α = φ Q T c q_{x,\alpha}=\varphi_Q^{\mathsf T}c q x , α = φ Q T c with coefficients
c ( i , a ) = − u ( x ) a i 2 + τ [ a h n ( α i ) − log cosh ( h n ( α i ) ) ] , c 5 = − u ( x ) 12 . c_{(i,a)}=-\frac{u(x)ai}{2}+\tau\left[a h_n(\alpha_i)-\log\cosh(h_n(\alpha_i))\right],\qquad c_5=-\frac{u(x)}{\sqrt{12}}. c ( i , a ) = − 2 u ( x ) ai + τ [ a h n ( α i ) − log cosh ( h n ( α i )) ] , c 5 = − 12 u ( x ) .
For any bounded measurable v v v , define
m i ( v , α ) = ∫ I i ∑ a ′ ∈ A π n , α ( a ′ ∣ s ′ ) v ( s ′ , a ′ ) μ i ( d s ′ ) . m_i(v,\alpha)=\int_{I_i}\sum_{a'\in\mathcal{A}}\pi_{n,\alpha}(a'\mid s')v(s',a')\,\mu_i(ds'). m i ( v , α ) = ∫ I i a ′ ∈ A ∑ π n , α ( a ′ ∣ s ′ ) v ( s ′ , a ′ ) μ i ( d s ′ ) .
The transition kernel gives the exact identity
( K α v ) ( s , a ) = m + ( v , α ) + m − ( v , α ) 2 + a s 4 [ m + ( v , α ) − m − ( v , α ) ] . (265) (K_\alpha v)(s,a)=\frac{m_+(v,\alpha)+m_-(v,\alpha)}{2}+\frac{a s}{4}\left[m_+(v,\alpha)-m_-(v,\alpha)\right]. \tag*{(265)} ( K α v ) ( s , a ) = 2 m + ( v , α ) + m − ( v , α ) + 4 a s [ m + ( v , α ) − m − ( v , α ) ] . (265)
Both 1 1 1 and a s a s a s belong to span φ Q \operatorname{span}\varphi_Q span φ Q , so this span is invariant under K α K_\alpha K α . The Markov operator is a contraction in the supremum norm. Hence ∑ k ≥ 0 γ k K α k q x , α \sum_{k\ge0}\gamma^kK_\alpha^kq_{x,\alpha} ∑ k ≥ 0 γ k K α k q x , α converges uniformly. Every partial sum lies in the finite-dimensional, and therefore closed, span of φ Q \varphi_Q φ Q . Its limit is the value in (261) , which proves C2.
Finally, on the ambient ball, ∣ h n ( α i ) ∣ ≤ A / 2 \lvert h_n(\alpha_i)\rvert\le A/2 ∣ h n ( α i )∣ ≤ A /2 and 0 ≤ log cosh ( h n ( α i ) ) ≤ ∣ h n ( α i ) ∣ 0\le\log\cosh(h_n(\alpha_i))\le\lvert h_n(\alpha_i)\rvert 0 ≤ log cosh ( h n ( α i )) ≤ ∣ h n ( α i )∣ . Thus the absolute log-policy ratio is at most A A A . The reward bound and the discounted resolvent bound give (262) .
The Bellman-closure step is the same structural principle used for value representation in linear MDPs Jin et al. [14 ] . Here the additional membership of the log-policy ratio in the fixed feature span is explicitly verified in (264) . The unregularized transition has rank-two features ( w + , w − ) = ( 1 / 2 + a s / 4 , 1 / 2 − a s / 4 ) (w_+,w_-)=(1/2+a s/4,1/2-a s/4) ( w + , w − ) = ( 1/2 + a s /4 , 1/2 − a s /4 ) and measure vector ( μ + , μ − ) (\mu_+,\mu_-) ( μ + , μ − ) . The five-feature augmentation also represents the actor’s KL cost. The hypotheses of Proposition C.62 specialize to
M = M ′ = 2 , P = w + μ + + w − μ − , M=M'=2,\qquad P=w_+\mu_++w_-\mu_-, M = M ′ = 2 , P = w + μ + + w − μ − ,
ψ ( s , a ) = a e i ( s ) , π r e f ( a ∣ i ) = 1 / 2 , \psi(s,a)=a e_{i(s)},\qquad\pi_{\mathrm{ref}}(a\mid i)=1/2, ψ ( s , a ) = a e i ( s ) , π ref ( a ∣ i ) = 1/2 ,
∥ φ Q ∥ ≤ 2 , 1 , a s , { 1 ( i ( s ) , a ) = ( j , b ) } j , b ∈ span φ Q , \lVert\varphi_Q\rVert\le2,\quad1,\quad a s,\quad\left\{\mathbf{1}_{(i(s),a)=(j,b)}\right\}_{j,b}\in\operatorname{span}\varphi_Q, ∥ φ Q ∥ ≤ 2 , 1 , a s , { 1 ( i ( s ) , a ) = ( j , b ) } j , b ∈ span φ Q ,
w ± , r x ∈ span φ Q , E ρ π r e f φ Q φ Q T = diag ( 1 / 4 , 1 / 4 , 1 / 4 , 1 / 4 , 1 ) . w_\pm,r_x\in\operatorname{span}\varphi_Q,\qquad\mathbb{E}_{\rho\pi_{\mathrm{ref}}}\varphi_Q\varphi_Q^{\mathsf T}=\operatorname{diag}(1/4,1/4,1/4,1/4,1). w ± , r x ∈ span φ Q , E ρ π ref φ Q φ Q T = diag ( 1/4 , 1/4 , 1/4 , 1/4 , 1 ) .
Corollary C.71 (A fixed coefficient envelope). The coefficient vector w x , α w_{x,\alpha} w x , α in Q x , α t r u e = φ Q T w x , α Q^{\mathrm{true}}_{x,\alpha}=\varphi_Q^{\mathsf T}w_{x,\alpha} Q x , α true = φ Q T w x , α is unique and, on the ambient ball, satisfies
∥ w x , α ∥ 2 ≤ 2 ( 1 + τ A ) 1 − γ . \lVert w_{x,\alpha}\rVert_2\le\frac{2(1+\tau A)}{1-\gamma}. ∥ w x , α ∥ 2 ≤ 1 − γ 2 ( 1 + τ A ) . Proof. At the reference policy, the reset Gram matrix is diag ( 1 / 4 , 1 / 4 , 1 / 4 , 1 / 4 , 1 ) ⪰ I 5 / 4 \operatorname{diag}(1/4,1/4,1/4,1/4,1)\succeq I_5/4 diag ( 1/4 , 1/4 , 1/4 , 1/4 , 1 ) ⪰ I 5 /4 . This proves linear independence of the features. Applying the same Gram inequality to Q t r u e = φ Q T w Q^{\mathrm{true}}=\varphi_Q^{\mathsf T}w Q true = φ Q T w yields ∥ w ∥ 2 2 / 4 ≤ E ρ 0 ( Q t r u e ) 2 ≤ ∥ Q t r u e ∥ ∞ 2 \lVert w\rVert_2^2/4\le\mathbb{E}_{\rho_0}(Q^{\mathrm{true}})^2\le\lVert Q^{\mathrm{true}}\rVert_\infty^2 ∥ w ∥ 2 2 /4 ≤ E ρ 0 ( Q true ) 2 ≤ ∥ Q true ∥ ∞ 2 .
An explicit nonconstant lower response
Use the normalized lower objective (2) ,
g τ , n ( x , α ) = ( 1 − γ ) E n , α ∑ t = 0 ∞ γ t q x , α ( s t , a t ) . (266) g_{\tau,n}(x,\alpha)=(1-\gamma)\mathbb{E}_{n,\alpha}\sum_{t=0}^{\infty}\gamma^t q_{x,\alpha}(s_t,a_t). \tag*{(266)} g τ , n ( x , α ) = ( 1 − γ ) E n , α t = 0 ∑ ∞ γ t q x , α ( s t , a t ) . (266)
Proposition C.72 (Selected response and true critic in closed form). If τ > 1 / 2 \tau>1/2 τ > 1/2 , then, for every x ∈ R x\in\mathbb{R} x ∈ R and n ≥ 1 n\ge1 n ≥ 1 , (266) has the unique global minimizer
α i ∗ ( x ) = 2 n arctanh ( i u ( x ) 2 τ n ) , i ∈ { − 1 , + 1 } . (267) \alpha_i^*(x)=2\sqrt{n}\operatorname{arctanh}\left(\frac{i\,u(x)}{2\tau\sqrt{n}}\right),\qquad i\in\{-1,+1\}. \tag*{(267)} α i ∗ ( x ) = 2 n arctanh ( 2 τ n i u ( x ) ) , i ∈ { − 1 , + 1 } . (267) Its value and true regularized action value are
g τ , n ( x , α ∗ ( x ) ) = − τ log cosh ( u ( x ) 2 τ ) , (268) g_{\tau,n}(x,\alpha^*(x))=-\tau\log\cosh\left(\frac{u(x)}{2\tau}\right), \tag*{(268)} g τ , n ( x , α ∗ ( x )) = − τ log cosh ( 2 τ u ( x ) ) , (268) Q x , α ∗ ( x ) t r u e ( s , a ) = − u ( x ) 12 a z ( s ) − τ 1 − γ log cosh ( u ( x ) 2 τ ) . (269) Q^{\mathrm{true}}_{x,\alpha^*(x)}(s,a)=-\frac{u(x)}{\sqrt{12}}az(s)-\frac{\tau}{1-\gamma}\log\cosh\left(\frac{u(x)}{2\tau}\right). \tag*{(269)} Q x , α ∗ ( x ) true ( s , a ) = − 12 u ( x ) a z ( s ) − 1 − γ τ log cosh ( 2 τ u ( x ) ) . (269) Moreover,
sup n ≥ 1 , x ∈ R ∥ α ∗ ( x ) ∥ 2 ≤ 2 2 arctanh ( 1 2 τ ) , ∂ x α i ∗ ( x ) ∣ x = 0 = i τ . (270) \sup_{n\ge1,x\in\mathbb{R}}\|\alpha^*(x)\|_2\le2\sqrt{2}\operatorname{arctanh}\left(\frac{1}{2\tau}\right),\qquad\left.\partial_x\alpha_i^*(x)\right|_{x=0}=\frac{i}{\tau}. \tag*{(270)} n ≥ 1 , x ∈ R sup ∥ α ∗ ( x ) ∥ 2 ≤ 2 2 arctanh ( 2 τ 1 ) , ∂ x α i ∗ ( x ) ∣ x = 0 = τ i . (270)
Proof. Every state law generated from ρ \rho ρ is a mixture of μ − \mu_- μ − and μ + \mu_+ μ + , because this is true initially and every transition in (247) has that form. The normalized discounted state occupancy therefore has the form d α = d − ( α ) μ − + d + ( α ) μ + d_\alpha=d_-(\alpha)\mu_-+d_+(\alpha)\mu_+ d α = d − ( α ) μ − + d + ( α ) μ + , where d − + d + = 1 d_-+d_+=1 d − + d + = 1 and each d i ≥ ( 1 − γ ) / 2 > 0 d_i\ge(1-\gamma)/2>0 d i ≥ ( 1 − γ ) /2 > 0 .
Write h i = h n ( α i ) h_i=h_n(\alpha_i) h i = h n ( α i ) and p b ( a ) = e a b / ( 2 cosh b ) p_b(a)=e^{ab}/(2\cosh b) p b ( a ) = e ab / ( 2 cosh b ) . Under π h i \pi_{h_i} π h i , E a = tanh h i \mathbb{E}a=\tanh h_i E a = tanh h i . Since E μ i s = i / 2 \mathbb{E}_{\mu_i}s=i/2 E μ i s = i /2 , the average one-step cost in region i i i is
ℓ i ( x , h i ) = − i u ( x ) 2 tanh h i + τ [ h i tanh h i − log cosh h i ] . \ell_i(x,h_i)=-\frac{i\,u(x)}{2}\tanh h_i+\tau\left[h_i\tanh h_i-\log\cosh h_i\right]. ℓ i ( x , h i ) = − 2 i u ( x ) tanh h i + τ [ h i tanh h i − log cosh h i ] .
Let b i = i u ( x ) / ( 2 τ ) b_i=i\,u(x)/(2\tau) b i = i u ( x ) / ( 2 τ ) . Direct calculation of binary relative entropy gives
ℓ i ( x , h i ) = τ KL ( p h i ∥ p b i ) − τ log cosh ( u ( x ) 2 τ ) . \ell_i(x,h_i)=\tau\operatorname{KL}(p_{h_i}\|p_{b_i})-\tau\log\cosh\left(\frac{u(x)}{2\tau}\right). ℓ i ( x , h i ) = τ KL ( p h i ∥ p b i ) − τ log cosh ( 2 τ u ( x ) ) .
Consequently
g τ , n ( x , α ) = τ ∑ i ∈ { − 1 , + 1 } d i ( α ) KL ( p h i ∥ p b i ) − τ log cosh ( u ( x ) 2 τ ) . (271) g_{\tau,n}(x,\alpha)=\tau\sum_{i\in\{-1,+1\}}d_i(\alpha)\operatorname{KL}(p_{h_i}\|p_{b_i})-\tau\log\cosh\left(\frac{u(x)}{2\tau}\right). \tag*{(271)} g τ , n ( x , α ) = τ i ∈ { − 1 , + 1 } ∑ d i ( α ) KL ( p h i ∥ p b i ) − τ log cosh ( 2 τ u ( x ) ) . (271)
The dependence of d i d_i d i on the actor is retained in this identity. Both weights are strictly positive. Thus equality with the lower bound occurs exactly when h i = b i h_i=b_i h i = b i for both regions. Since h n h_n h n is strictly increasing with range ( − n , n ) (-\sqrt{n},\sqrt{n}) ( − n , n ) and ∣ b i ∣ < 1 ≤ n |b_i|<1\le\sqrt{n} ∣ b i ∣ < 1 ≤ n when τ > 1 / 2 \tau>1/2 τ > 1/2 , these equalities have the unique solution (267) . This also proves (268) .
At this response, (263) and (264) give
q x , α ∗ ( s , a ) = − u ( x ) 12 a z ( s ) − τ log cosh ( u ( x ) 2 τ ) . q_{x,\alpha^*}(s,a)=-\frac{u(x)}{\sqrt{12}}az(s)-\tau\log\cosh\left(\frac{u(x)}{2\tau}\right). q x , α ∗ ( s , a ) = − 12 u ( x ) a z ( s ) − τ log cosh ( 2 τ u ( x ) ) .
The function a z ( s ) az(s) a z ( s ) has zero mean under each μ i ( d s ) π n , α ∗ ( d a ∣ i ) \mu_i(ds)\pi_{n,\alpha^*}(da\mid i) μ i ( d s ) π n , α ∗ ( d a ∣ i ) . Hence K α ∗ ( a z ) = 0 K_{\alpha^*}(az)=0 K α ∗ ( a z ) = 0 and K α ∗ 1 = 1 K_{\alpha^*}1=1 K α ∗ 1 = 1 . The discounted resolvent therefore gives (269) directly.
For 0 ≤ t < 1 0 \le t < 1 0 ≤ t < 1 , the power series for inverse hyperbolic tangent gives
n arctanh ( t / n ) = ∑ k = 0 ∞ t 2 k + 1 ( 2 k + 1 ) n k ≤ arctanh t . \sqrt{n}\,\operatorname{arctanh}(t/\sqrt{n})=\sum_{k=0}^{\infty}\frac{t^{2k+1}}{(2k+1)n^k}\le\operatorname{arctanh}t. n arctanh ( t / n ) = k = 0 ∑ ∞ ( 2 k + 1 ) n k t 2 k + 1 ≤ arctanh t .
Apply this with t = ∣ u ( x ) ∣ / ( 2 τ ) t=|u(x)|/(2\tau) t = ∣ u ( x ) ∣/ ( 2 τ ) to obtain the norm bound. Differentiating (267) gives
∂ α i α i ∗ ( x ) = i sech 2 x τ ( 1 − u ( x ) 2 / ( 4 τ 2 n ) ) . \partial_{\alpha_i}\alpha_i^*(x)=\frac{i\,\operatorname{sech}^2x}{\tau\left(1-u(x)^2/(4\tau^2n)\right)}. ∂ α i α i ∗ ( x ) = τ ( 1 − u ( x ) 2 / ( 4 τ 2 n ) ) i sech 2 x .
which proves the final claim.
For any fixed target radius R 0 > 0 R_0>0 R 0 > 0 , the bound (270) is smaller than R 0 R_0 R 0 whenever
τ > 1 2 tanh ( R 0 / ( 2 2 ) ) . \tau>\frac{1}{2\tanh\left(R_0/(2\sqrt{2})\right)}. τ > 2 tanh ( R 0 / ( 2 2 ) ) 1 .
Thus τ \tau τ can be chosen large enough to meet our geometry and initialization requirements while placing the entire response family inside any prescribed smaller actor buffer. This same τ \tau τ is then fixed before selecting critic constants. For x ≠ 0 x\ne0 x = 0 the response gives different policies in the two regions, and (269) varies with the continuous state within each region.
An interior upper stationary point and a nonzero chain term
Let f n ( x , α ) ≡ E L ( x ; ξ , Y ) f_n(x,\alpha)\equiv\mathbb{E}L(x;\xi,Y) f n ( x , α ) ≡ E L ( x ; ξ , Y ) and Φ τ ( x ) = f n ( x , α ∗ ( x ) ) \Phi_\tau(x)=f_n(x,\alpha^*(x)) Φ τ ( x ) = f n ( x , α ∗ ( x )) . The conditional mean sample loss is
ℓ κ ( u , d ) = log ( 1 + e u d ) − u d σ ( κ d ) = log ( 2 cosh ( u d / 2 ) ) − u d 2 tanh ( κ d / 2 ) . (272) \ell_\kappa(u,d)=\log(1+e^{ud})-ud\,\sigma(\kappa d)=\log\left(2\cosh(ud/2)\right)-\frac{ud}{2}\tanh(\kappa d/2). \tag*{(272)} ℓ κ ( u , d ) = log ( 1 + e u d ) − u d σ ( κ d ) = log ( 2 cosh ( u d /2 ) ) − 2 u d tanh ( κ d /2 ) . (272)
It is even in d d d . For arbitrary actor coordinates, the exact score is
∂ α j log π n , α ( a ∣ s ) = 1 { i ( s ) = j } h n ′ ( α j ) [ a − tanh ( h n ( α j ) ) ] . \partial_{\alpha_j}\log\pi_{n,\alpha}(a\mid s)=\mathbf{1}_{\{i(s)=j\}}h_n'(\alpha_j)\left[a-\tanh\left(h_n(\alpha_j)\right)\right]. ∂ α j log π n , α ( a ∣ s ) = 1 { i ( s ) = j } h n ′ ( α j ) [ a − tanh ( h n ( α j ) ) ] .
Since the dynamics and conditional feedback law are fixed,
f α = E [ L ∑ ℓ = 1 2 ∑ t = 0 1 ∇ α log π n , α ( a t ℓ ∣ s t ℓ ) ] . (273) f_\alpha=\mathbb{E}\left[L\sum_{\ell=1}^{2}\sum_{t=0}^{1}\nabla_\alpha\log\pi_{n,\alpha}\left(a_t^\ell\mid s_t^\ell\right)\right]. \tag*{(273)} f α = E [ L ℓ = 1 ∑ 2 t = 0 ∑ 1 ∇ α log π n , α ( a t ℓ ∣ s t ℓ ) ] . (273)
Lemma C.73 (An exact one-dimensional quadrature formula). At the lower response put u = tanh x u=\tanh x u = tanh x and p = σ ( u / τ ) p=\sigma(u/\tau) p = σ ( u / τ ) . Define
B 4 ( y ) = 1 6 ∑ j = 0 4 ( − 1 ) j ( 4 j ) ( y − j ) + 3 , 0 ≤ y ≤ 4 , B_4(y)=\frac{1}{6}\sum_{j=0}^{4}(-1)^j\binom{4}{j}(y-j)_+^3,\qquad0\le y\le4, B 4 ( y ) = 6 1 j = 0 ∑ 4 ( − 1 ) j ( j 4 ) ( y − j ) + 3 , 0 ≤ y ≤ 4 , J k ( u ) = ∫ 0 4 B 4 ( y ) ℓ κ ( u , y − 4 + k ) d y , k = 0 , … , 4. (274) J_k(u)=\int_0^4 B_4(y)\ell_\kappa(u,y-4+k)\,dy,\qquad k=0,\ldots,4. \tag*{(274)} J k ( u ) = ∫ 0 4 B 4 ( y ) ℓ κ ( u , y − 4 + k ) d y , k = 0 , … , 4. (274) Here B 4 B_4 B 4 is the density of the sum of four independent uniform variables on [ 0 , 1 ] [0,1] [ 0 , 1 ] . Write v = p ( 1 − p ) v=p(1-p) v = p ( 1 − p ) and
( ω 0 , ω 1 , ω 2 , ω 3 , ω 4 ) = ( v 2 , 2 v − 4 v 2 , 1 − 4 v + 6 v 2 , 2 v − 4 v 2 , v 2 ) . (275) (\omega_0,\omega_1,\omega_2,\omega_3,\omega_4)=\left(v^2,\,2v-4v^2,\,1-4v+6v^2,\,2v-4v^2,\,v^2\right). \tag*{(275)} ( ω 0 , ω 1 , ω 2 , ω 3 , ω 4 ) = ( v 2 , 2 v − 4 v 2 , 1 − 4 v + 6 v 2 , 2 v − 4 v 2 , v 2 ) . (275)
Then the exact population objective and gradient decomposition are
Φ τ ( x ) = ∑ k = 0 4 ω k ( p ) J k ( u ) , \Phi_{\tau}(x)=\sum_{k=0}^{4}\omega_k(p)J_k(u), Φ τ ( x ) = k = 0 ∑ 4 ω k ( p ) J k ( u ) ,
A τ ( x ) : = f x ( x , α ∗ ( x ) ) = ( 1 − u 2 ) ∑ k = 0 4 ω k ( p ) J k ′ ( u ) , A_{\tau}(x):=f_x(x,\alpha^*(x))=(1-u^2)\sum_{k=0}^{4}\omega_k(p)J_k'(u), A τ ( x ) := f x ( x , α ∗ ( x )) = ( 1 − u 2 ) k = 0 ∑ 4 ω k ( p ) J k ′ ( u ) ,
C τ ( x ) : = D x α ∗ ( x ) T f α ( x , α ∗ ( x ) ) = ( 1 − u 2 ) p ( 1 − p ) τ ∑ k = 0 4 ω k ′ ( p ) J k ( u ) , C_{\tau}(x):=D_x\alpha^*(x)^{\mathsf T}f_{\alpha}(x,\alpha^*(x))=\frac{(1-u^2)p(1-p)}{\tau}\sum_{k=0}^{4}\omega_k'(p)J_k(u), C τ ( x ) := D x α ∗ ( x ) T f α ( x , α ∗ ( x )) = τ ( 1 − u 2 ) p ( 1 − p ) k = 0 ∑ 4 ω k ′ ( p ) J k ( u ) ,
Φ τ ′ ( x ) = A τ ( x ) + C τ ( x ) . (276) \Phi_{\tau}'(x)=A_{\tau}(x)+C_{\tau}(x). \tag*{(276)} Φ τ ′ ( x ) = A τ ( x ) + C τ ( x ) . (276)
In the derivative of J k J_k J k , the integrand is [ σ ( u d ) − σ ( κ d ) ] d [\sigma(ud)-\sigma(\kappa d)]d [ σ ( u d ) − σ ( κ d )] d , with d = y − 4 + k d=y-4+k d = y − 4 + k .
Proof. At the response the policy logits are h n ( α i ∗ ) = i u / ( 2 τ ) h_n(\alpha_i^*)=iu/(2\tau) h n ( α i ∗ ) = i u / ( 2 τ ) . Write each state as s t = i t R t s_t=i_tR_t s t = i t R t , where R t = ∣ s t ∣ R_t=|s_t| R t = ∣ s t ∣ , and define ϵ t = a t i t \epsilon_t=a_ti_t ϵ t = a t i t . At reset, R 0 R_0 R 0 is uniform on [ 0 , 1 ] [0,1] [ 0 , 1 ] and independent of i 0 i_0 i 0 . Every subsequent transition first selects a region and then draws a fresh uniform magnitude independent of the past. Conditional on every region and magnitude, the response policy has P ( ϵ t = + 1 ) = p \mathbb{P}(\epsilon_t=+1)=p P ( ϵ t = + 1 ) = p , independent of that region, magnitude, and the past. Thus the products a t s t = ϵ t R t a_ts_t=\epsilon_tR_t a t s t = ϵ t R t are independent across time, with independent uniform R t R_t R t and independent signs of probability p p p .
This independence concerns the products a t s t a_ts_t a t s t , not the raw states conditioned on an entire future region path. It accounts for the dependence of the transition weights on the preceding state and action. For two independent length-two trajectories,
D = ϵ 1 R 1 + ϵ 2 R 2 − ϵ 3 R 3 − ϵ 4 R 4 . D=\epsilon_1R_1+\epsilon_2R_2-\epsilon_3R_3-\epsilon_4R_4. D = ϵ 1 R 1 + ϵ 2 R 2 − ϵ 3 R 3 − ϵ 4 R 4 .
Conditional on the four signs in this sum, if k k k signs are positive then D D D has the law of a sum of four independent [ 0 , 1 ] [0,1] [ 0 , 1 ] uniforms minus 4 − k 4-k 4 − k . Replacing a negatively signed uniform − R -R − R by ( 1 − R ) − 1 (1-R)-1 ( 1 − R ) − 1 proves this claim. The probability generating polynomial of k k k is [ ( 1 − p ) + p z ] 2 [ p + ( 1 − p ) z ] 2 [(1-p)+pz]^2[p+(1-p)z]^2 [( 1 − p ) + p z ] 2 [ p + ( 1 − p ) z ] 2 ; its coefficients are exactly (275) . This proves the objective formula. Differentiating u ( x ) u(x) u ( x ) and p ( x ) p(x) p ( x ) gives u ′ = 1 − u 2 u'=1-u^2 u ′ = 1 − u 2 and p ′ = ( 1 − u 2 ) p ( 1 − p ) / τ p'=(1-u^2)p(1-p)/\tau p ′ = ( 1 − u 2 ) p ( 1 − p ) / τ . Differentiation of the integrals is justified on their compact domains. The derivative through u u u is the direct partial derivative, and the derivative through p p p is exactly the trajectory-law contribution in (273) . □
Proposition C.74 (A finite interior minimizer with a nonzero correction). For every κ ∈ ( 0 , 1 ) \kappa\in(0,1) κ ∈ ( 0 , 1 ) and τ > 1 / 2 \tau>1/2 τ > 1/2 , let x † = arctanh κ x^\dagger=\operatorname{arctanh}\kappa x † = arctanh κ . Then
Φ τ ′ ( 0 ) < 0 , Φ τ ′ ( x † ) > 0. \Phi_{\tau}'(0)<0,\qquad\Phi_{\tau}'(x^\dagger)>0. Φ τ ′ ( 0 ) < 0 , Φ τ ′ ( x † ) > 0. Consequently Φ τ \Phi_{\tau} Φ τ has an interior minimizer on [ 0 , x † ] [0,x^\dagger] [ 0 , x † ] . At every stationary point x ˉ ∈ ( 0 , x † ) \bar{x}\in(0,x^\dagger) x ˉ ∈ ( 0 , x † ) ,
A τ ( x ˉ ) < 0 , C τ ( x ˉ ) = − A τ ( x ˉ ) > 0 , ∣ C τ ( x ˉ ) ∣ ∣ A τ ( x ˉ ) ∣ = 1. (277) A_{\tau}(\bar{x})<0,\qquad C_{\tau}(\bar{x})=-A_{\tau}(\bar{x})>0,\qquad\frac{|C_{\tau}(\bar{x})|}{|A_{\tau}(\bar{x})|}=1. \tag*{(277)} A τ ( x ˉ ) < 0 , C τ ( x ˉ ) = − A τ ( x ˉ ) > 0 , ∣ A τ ( x ˉ ) ∣ ∣ C τ ( x ˉ ) ∣ = 1. (277) The chain term vanishes at x = 0 x=0 x = 0 .
Proof. At x = 0 x=0 x = 0 , the loss equals log 2 \log2 log 2 for every tuple and every label, so f α ( 0 , α ) = 0 f_{\alpha}(0,\alpha)=0 f α ( 0 , α ) = 0 and C τ ( 0 ) = 0 C_{\tau}(0)=0 C τ ( 0 ) = 0 . The direct derivative equals E [ ( 1 / 2 − σ ( κ D ) ) D ] \mathbb{E}[(1/2-\sigma(\kappa D))D] E [( 1/2 − σ ( κ D )) D ] , which is strictly negative: the integrand is negative whenever D ≠ 0 D\ne0 D = 0 , an event of probability one here.
At x = x † x=x^\dagger x = x † , u = κ u=\kappa u = κ and the direct derivative is zero. The conditional mean loss is the binary entropy H ( σ ( κ d ) ) H(\sigma(\kappa d)) H ( σ ( κ d )) , an even function strictly decreasing in ∣ d ∣ > 0 |d|>0 ∣ d ∣ > 0 . To compare the integrals in (274) , let X X X be the sum of four independent uniforms minus two. Its density b b b is symmetric and decreasing in ∣ X ∣ |X| ∣ X ∣ . For example, B 4 ( y ) = y 3 / 6 B_4(y)=y^3/6 B 4 ( y ) = y 3 /6 on [ 0 , 1 ] [0,1] [ 0 , 1 ] and B 4 ( y ) = [ y 3 − 4 ( y − 1 ) 3 ] / 6 B_4(y)=[y^3-4(y-1)^3]/6 B 4 ( y ) = [ y 3 − 4 ( y − 1 ) 3 ] /6 on [ 1 , 2 ] [1,2] [ 1 , 2 ] is increasing there, and symmetry supplies the other half of the density. For any c > 0 c>0 c > 0 , differentiating the convolution with the entropy function H κ H_\kappa H κ gives
d d c E H κ ( X + c ) = ∫ 0 ∞ H κ ′ ( z ) [ b ( z − c ) − b ( z + c ) ] d z < 0. \frac{d}{dc}\mathbb{E}H_\kappa(X+c)=\int_0^\infty H_\kappa'(z)[b(z-c)-b(z+c)]\,dz<0. d c d E H κ ( X + c ) = ∫ 0 ∞ H κ ′ ( z ) [ b ( z − c ) − b ( z + c )] d z < 0.
Indeed H κ ′ ( z ) < 0 H_\kappa'(z)<0 H κ ′ ( z ) < 0 for z > 0 z>0 z > 0 and the bracket is nonnegative, and strictly positive on a set of positive measure. It follows that J 2 ( κ ) > J 1 ( κ ) > J 0 ( κ ) J_2(\kappa)>J_1(\kappa)>J_0(\kappa) J 2 ( κ ) > J 1 ( κ ) > J 0 ( κ ) and J 4 − k ( κ ) = J k ( κ ) J_{4-k}(\kappa)=J_k(\kappa) J 4 − k ( κ ) = J k ( κ ) .
Holding u = κ u=\kappa u = κ fixed, write the objective as a polynomial in v = p ( 1 − p ) v=p(1-p) v = p ( 1 − p ) . Its derivative with respect to v v v is
4 v [ J 0 − J 2 ] + ( 4 − 16 v ) [ J 1 − J 2 ] < 0. 4v[J_0-J_2]+(4-16v)[J_1-J_2]<0. 4 v [ J 0 − J 2 ] + ( 4 − 16 v ) [ J 1 − J 2 ] < 0.
Here 0 < v < 1 / 4 0<v<1/4 0 < v < 1/4 , p > 1 / 2 p>1/2 p > 1/2 , and d v / d p = 1 − 2 p < 0 dv/dp=1-2p<0 d v / d p = 1 − 2 p < 0 . Since p ′ ( x † ) > 0 p'(x^\dagger)>0 p ′ ( x † ) > 0 , the response contribution is strictly positive. This proves the second endpoint sign.
A minimum on the compact interval exists, and the strict endpoint derivative signs exclude either endpoint. At every interior stationary point, 0 < u < κ 0<u<\kappa 0 < u < κ , so [ σ ( u D ) − σ ( κ D ) ] D < 0 [\sigma(uD)-\sigma(\kappa D)]D<0 [ σ ( u D ) − σ ( κ D )] D < 0 almost surely. Thus the direct term is strictly negative and stationarity forces the positive, nonzero cancellation in (277) .
The existence proof holds for every τ > 1 / 2 \tau>1/2 τ > 1/2 , including any larger fixed τ \tau τ selected by our geometry and containment constants. The threshold 1 / 2 1/2 1/2 is sufficient for the explicit response formula; it does not replace that operative algorithmic choice of τ \tau τ .
Remark C.75 (Preference-risk stationarity and the teacher objective). Let p † = σ ( κ D ) p^\dagger=\sigma(\kappa D) p † = σ ( κ D ) , p x = σ ( tanh ( x ) D ) p_x=\sigma(\tanh(x)D) p x = σ ( tanh ( x ) D ) , and let H B H_{\mathrm{B}} H B denote binary entropy. Conditional cross-entropy gives
Φ τ ( x ) = E ξ ∼ ( P α ∗ ( x ) ) ⊗ 2 [ H B ( p † ) + KL ( Ber ( p † ) ∥ Ber ( p x ) ) ] . (278) \Phi_\tau(x)=\mathbb{E}_{\xi\sim(\mathbb{P}_{\alpha^*(x)})^{\otimes2}}\left[H_{\mathrm{B}}(p^\dagger)+\operatorname{KL}(\operatorname{Ber}(p^\dagger)\mathbin\Vert\operatorname{Ber}(p_x))\right]. \tag*{(278)} Φ τ ( x ) = E ξ ∼ ( P α ∗ ( x ) ) ⊗ 2 [ H B ( p † ) + KL ( Ber ( p † ) ∥ Ber ( p x )) ] . (278)
At x † = arctanh κ x^\dagger=\operatorname{arctanh}\kappa x † = arctanh κ , the divergence and its derivative vanish, but the expected entropy can vary through the response-dependent trajectory law. Proposition C.74 gives Φ τ ′ ( x † ) > 0 \Phi_\tau'(x^\dagger)>0 Φ τ ′ ( x † ) > 0 and an interior minimizer x ˉ ∈ ( 0 , x † ) \bar{x}\in(0,x^\dagger) x ˉ ∈ ( 0 , x † ) .
The normalized return under the teacher reward r † ( s , a ) = κ a s r^\dagger(s,a)=\kappa as r † ( s , a ) = κa s is, at the selected response,
J † ( x ) : = ( 1 − γ ) E α ∗ ( x ) ∑ t ≥ 0 γ t r † ( s t , a t ) = κ 2 tanh ( tanh x 2 τ ) . (279) J^\dagger(x):=(1-\gamma)\mathbb{E}_{\alpha^*(x)}\sum_{t\geq0}\gamma^t r^\dagger(s_t,a_t)=\frac{\kappa}{2}\tanh\left(\frac{\tanh x}{2\tau}\right). \tag*{(279)} J † ( x ) := ( 1 − γ ) E α ∗ ( x ) t ≥ 0 ∑ γ t r † ( s t , a t ) = 2 κ tanh ( 2 τ tanh x ) . (279)
Indeed, conditional on either region, the state is uniform and E [ a ∣ i ] = i tanh ( tanh ( x ) / ( 2 τ ) ) \mathbb{E}[a\mid i]=i\tanh(\tanh(x)/(2\tau)) E [ a ∣ i ] = i tanh ( tanh ( x ) / ( 2 τ )) , so every time step has the displayed mean reward. This return is strictly increasing in x x x ; thus J † ( x ˉ ) < J † ( x † ) J^\dagger(\bar{x})<J^\dagger(x^\dagger) J † ( x ˉ ) < J † ( x † ) . The example establishes a nonzero response correction and also separates optimization of the stated on-policy risk from reward identification or teacher-return maximization.
A fully specified population illustration. Take γ = 1 / 2 \gamma=1/2 γ = 1/2 , κ = 1 / 2 \kappa=1/2 κ = 1/2 , n = 1 n=1 n = 1 , and τ = 2 \tau=2 τ = 2 . Deterministic quadrature of (274) gives an interior minimizer
x ˉ ≈ 0.5414613800 , x † ≈ 0.5493061443 , Φ 2 ( x ˉ ) ≈ 0.6559732584. (280) \bar{x}\approx0.5414613800,\qquad x^\dagger\approx0.5493061443,\qquad\Phi_2(\bar{x})\approx0.6559732584. \tag*{(280)} x ˉ ≈ 0.5414613800 , x † ≈ 0.5493061443 , Φ 2 ( x ˉ ) ≈ 0.6559732584. (280)
At x ˉ \bar{x} x ˉ , the direct and response terms are approximately − 0.001204848558 -0.001204848558 − 0.001204848558 and + 0.001204848558 +0.001204848558 + 0.001204848558 , respectively. Table 2 reports both absolute magnitudes and their ratio. The ratio equals one at a stationary point by cancellation; this identity alone should not be interpreted as a large correction throughout the domain.
Table 2. Population gradient decomposition for κ = 1 / 2 \kappa= 1/2 κ = 1/2 , τ = 2 \tau= 2 τ = 2 . The ratio at x † x^\dagger x † is undefined because its denominator is zero.
The density B 4 B_4 B 4 is polynomial between consecutive integers. Gauss–Legendre quadrature with 12, 24, and 48 nodes on each of its four pieces agreed at the checked points to less than 10 − 13 10^{-13} 1 0 − 13 in the objective and both gradient terms. An independent four-dimensional tensor quadrature with explicit enumeration of the 16 sign patterns agreed at x ˉ \bar{x} x ˉ to the same tolerance. These are numerical consistency checks, rather than formal interval-error certificates; the existence and nonzero-chain claims are established analytically above.
Figure 2. Population quantities evaluated by deterministic quadrature. Left: upper objective. Middle: full, direct, and response derivatives. Right: the same derivatives near the minimum, with the vertical scale multiplied by 10 3 10^3 1 0 3 . The dashed line is x ˉ \bar{x} x ˉ and the dotted line is x † x^\dagger x † . These curves illustrate the response mechanism; they are not stochastic convergence results.
For larger fixed regularization the correction remains nonzero, but becomes smaller, as Table 3 illustrates. In this symmetric example C τ = O ( τ − 2 ) C_\tau= O(\tau^{-2}) C τ = O ( τ − 2 ) on compact x x x intervals: p − 1 / 2 = O ( τ − 1 ) p^{-1/2} = O(\tau^{-1}) p − 1/2 = O ( τ − 1 ) and ω k ′ ( p ) = O ( p − 1 / 2 ) \omega_k'(p) = O(p^{-1/2}) ω k ′ ( p ) = O ( p − 1/2 ) in (276) . The τ = 2 \tau= 2 τ = 2 plot is a population illustration, not a certification of every conservative constant in the stochastic containment theorem.
Table 3. The finite stationary point and its nonzero correction persist as τ \tau τ increases, with κ = 1 / 2 \kappa= 1/2 κ = 1/2 .
Joint conclusion and exact critic targets
Proposition C.76 (Joint continuous-state instance). Fix an ambient actor radius A > 0 A > 0 A > 0 and γ ∈ ( 0 , 1 ) \gamma\in(0,1) γ ∈ ( 0 , 1 ) . Use (247)–(250), the actor (251)–(252), and the feature map (257) . Choose sufficiently large finite τ \tau τ as required by our actor geometry and containment construction, and then keep it fixed. The resulting instance satisfies A1, A2, G, C1, and C2 simultaneously, uniformly over actor widths 2 n 2n 2 n and over the prescribed actor neighborhoods contained in ∥ α ∥ 2 ≤ A \lVert\alpha\rVert_2 \le A ∥ α ∥ 2 ≤ A . The reward and upper domains are unrestricted in x ∈ R x \in\mathbb{R} x ∈ R . The actor has retained dimension 2, the critic has fixed feature dimension 5, and the constants in (254) , (255) , and (260) do not depend on critic scale or target accuracy.
Proof. Combine Lemmas C.66 , C.67, C.69, and C.70. The critic verifications hold on the full ambient ball, so they also cover base and perturbed actors and the local update segments used by the theorem. Proposition C.72 shows that the selected response can be placed strictly inside the chosen geometry neighborhood. No assumption about an independently accurate critic initialization is used in these verifications.
Remark C.77 (Scope and the nonlinear trained critic). The example has a finite-rank transition kernel and a policy that uses a two-region state encoding. State differentiability is not required by A1; the measurable encoding i ( s ) i(s) i ( s ) is therefore allowed. The transition probabilities and rewards retain within-region continuous state dependence. This establishes joint satisfiability within the stated model class.
C2 is a statement about the fixed linear span span φ Q \operatorname{span}\varphi_Q span φ Q . We train
Q m Q , θ ( s , a ) = m Q tanh ( φ Q ( s , a ) T θ m Q ) , Q_{m_Q,\theta}(s,a) = \sqrt{m_Q}\tanh\left(\frac{\varphi_Q(s,a)^\mathsf{T}\theta}{\sqrt{m_Q}}\right), Q m Q , θ ( s , a ) = m Q tanh ( m Q φ Q ( s , a ) T θ ) ,
whose finite nonlinear scale need not yield exact representation of the true value. For example, on ∥ θ ∥ 2 ≤ R Q \lVert\theta\rVert_2 \le R_Q ∥ θ ∥ 2 ≤ R Q the feature bound above gives the elementary estimate
∥ Q m Q , θ − φ Q T θ ∥ ∞ ≤ 8 R Q 3 3 m Q . \lVert Q_{m_Q,\theta}-\varphi_Q^\mathsf{T}\theta\rVert_\infty\le\frac{8R_Q^3}{3m_Q}. ∥ Q m Q , θ − φ Q T θ ∥ ∞ ≤ 3 m Q 8 R Q 3 .
Our paired-TD stability and tracking arguments are still used to obtain its derived squared paired approximation floor and interaction bound. Exact feature-span membership is the primitive realizability condition verified by this example.
Exact value coefficients for arbitrary actors. The one-dimensional response formula is not needed to compute the true value away from the response. Let the cell order be that of (257) . For j ∈ { − 1 , + 1 } j \in\{-1,+1\} j ∈ { − 1 , + 1 } define coefficient vectors v j , m j ( α ) ∈ R 5 v_j,m_j(\alpha) \in\mathbb{R}^5 v j , m j ( α ) ∈ R 5 by
( v j ) ( i , a ) = 1 2 + j a i 8 , ( v j ) 5 = j 4 12 , ( m j ) ( i , a ) = 1 { i = j } π n , α ( a ∣ i ) , ( m j ) 5 = 0. (v_j)_{(i,a)}=\frac{1}{2}+\frac{jai}{8},\qquad(v_j)_5=\frac{j}{4\sqrt{12}},\qquad(m_j)_{(i,a)}=\mathbf{1}_{\{i=j\}}\pi_{n,\alpha}(a\mid i),\qquad(m_j)_5=0. ( v j ) ( i , a ) = 2 1 + 8 j ai , ( v j ) 5 = 4 12 j , ( m j ) ( i , a ) = 1 { i = j } π n , α ( a ∣ i ) , ( m j ) 5 = 0.
Then w j ( s , a ) = φ Q ( s , a ) T v j w_j(s,a)=\varphi_Q(s,a)^\mathsf{T}v_j w j ( s , a ) = φ Q ( s , a ) T v j and E μ j π α φ Q = m j ( α ) \mathbb{E}_{\mu_j\pi_\alpha}\varphi_Q=m_j(\alpha) E μ j π α φ Q = m j ( α ) . Define
T α = ∑ j ∈ { − 1 , + 1 } v j m j ( α ) T , K α ( φ Q T b ) = φ Q T T α b . (281) T_\alpha=\sum_{j\in\{-1,+1\}}v_jm_j(\alpha)^\mathsf{T},\qquad K_\alpha(\varphi_Q^\mathsf{T}b)=\varphi_Q^\mathsf{T}T_\alpha b. \tag*{(281)} T α = j ∈ { − 1 , + 1 } ∑ v j m j ( α ) T , K α ( φ Q T b ) = φ Q T T α b . (281)
If c x , α c_{x,\alpha} c x , α is the coefficient vector for q q q in the proof of Lemma C.70, the true-value coefficients are exactly
w x , α = ( I 5 − γ T α ) − 1 c x , α . (282) w_{x,\alpha}=(I_5-\gamma T_\alpha)^{-1}c_{x,\alpha}. \tag*{(282)} w x , α = ( I 5 − γ T α ) − 1 c x , α . (282)
The inverse exists because the discounted Bellman operator is invertible on the invariant feature span and the features are linearly independent.
The finite nonlinear TD target is a different root. For a base actor U U U , the state occupancy weights can themselves be computed from a two-by-two matrix. Its rows are
P ‾ U ( i , + ) = 1 2 + i tanh ( h n ( U i ) ) 8 , P ‾ U ( i , − ) = 1 − P ‾ U ( i , + ) . \overline{P}_U(i,+)=\frac{1}{2}+\frac{i\tanh(h_n(U_i))}{8},\qquad\overline{P}_U(i,-)=1-\overline{P}_U(i,+). P U ( i , + ) = 2 1 + 8 i tanh ( h n ( U i )) , P U ( i , − ) = 1 − P U ( i , + ) .
With row-vector convention,
d U = ( 1 − γ ) ( 1 / 2 , 1 / 2 ) ( I 2 − γ P ‾ U ) − 1 , d_U=(1-\gamma)(1/2,1/2)(I_2-\gamma\overline{P}_U)^{-1}, d U = ( 1 − γ ) ( 1/2 , 1/2 ) ( I 2 − γ P U ) − 1 ,
ν U ( d s , d a ) = ∑ i d U ( i ) μ i ( d s ) π n , U ( a ∣ i ) . \nu_U(ds,da)=\sum_i d_U(i)\mu_i(ds)\pi_{n,U}(a\mid i). ν U ( d s , d a ) = i ∑ d U ( i ) μ i ( d s ) π n , U ( a ∣ i ) .
For finite critic scale, the relevant mean field remains
H m Q ( x , U , α , θ ) = E ν U [ ( Q m Q , θ − q x , α − γ K α Q m Q , θ ) ∇ θ Q m Q , θ ] . (283) H_{m_Q}(x,U,\alpha,\theta)=\mathbb{E}_{\nu_U}\left[(Q_{m_Q,\theta}-q_{x,\alpha}-\gamma K_\alpha Q_{m_Q,\theta})\nabla_\theta Q_{m_Q,\theta}\right]. \tag*{(283)} H m Q ( x , U , α , θ ) = E ν U [ ( Q m Q , θ − q x , α − γ K α Q m Q , θ ) ∇ θ Q m Q , θ ] . (283)
All integrals in this field reduce to finite action sums and one-dimensional within-region integrals. Our interior root θ m Q ∗ ( x , U , α ) \theta^*_{m_Q}(x,U,\alpha) θ m Q ∗ ( x , U , α ) can therefore be computed by a five-dimensional nonlinear root solve with controlled quadrature. For p a l g = ( x , U , S ) p_{\mathrm{alg}}=(x,U,S) p alg = ( x , U , S ) the paired target is
Z m Q ( p a l g ) = ( θ m Q ∗ ( x , U , U ) , θ m Q ∗ ( x , U , U + c S ) − θ m Q ∗ ( x , U , U ) c ) . (284) Z_{m_Q}(p_{\mathrm{alg}})=\left(\theta^*_{m_Q}(x,U,U),\frac{\theta^*_{m_Q}(x,U,U+cS)-\theta^*_{m_Q}(x,U,U)}{c}\right). \tag*{(284)} Z m Q ( p alg ) = ( θ m Q ∗ ( x , U , U ) , c θ m Q ∗ ( x , U , U + c S ) − θ m Q ∗ ( x , U , U ) ) . (284)
The linear solve (282) computes the exact value coefficients, not this nonlinear TD root. Monitoring the theorem’s ∥ z t − Z m Q ( p t ) ∥ W \lVert z_t-Z_{m_Q}(p_t)\rVert_W ∥ z t − Z m Q ( p t ) ∥ W therefore requires the nonlinear solves, with tolerances small enough after division by c c c .
Population evaluation away from the lower response. The local perturbed response solves g α ( x , α c ∗ ) + c f α ( x , α c ∗ ) = 0 g_\alpha(x,\alpha_c^*)+c f_\alpha(x,\alpha_c^*)=0 g α ( x , α c ∗ ) + c f α ( x , α c ∗ ) = 0 , with S c ∗ = ( α c ∗ − α 0 ∗ ) / c S_c^*=(\alpha_c^*-\alpha_0^*)/c S c ∗ = ( α c ∗ − α 0 ∗ ) / c . Away from the unperturbed response, preference expectations must retain the transition likelihoods; the independent-product reduction in Lemma C.73 is specific to the response policy. The exact population calculations provide evaluation quantities and do not replace the sampling estimators in Algorithm 1.
Conditional global lower responses on a prescribed domain
This subsection proves Corollary A.2 and specifies the scope of its additional assumption. Fix the finite actor architecture m \mathbf{m} m and the same fixed regularization parameter τ \tau τ as in Theorem 4.6 . Write a ( x ) = α 0 , m ∗ ( x ) a(x)=\alpha_{0,m}^*(x) a ( x ) = α 0 , m ∗ ( x ) and
g τ , m , D ∗ ( x ) : = min β ∈ D g τ , m ( x , β ) , A τ , m , D ∗ ( x ) : = arg min β ∈ D g τ , m ( x , β ) . g_{\tau,m,\mathcal{D}}^*(x):=\min_{\beta\in\mathcal{D}}g_{\tau,m}(x,\beta),\qquad\mathcal{A}_{\tau,m,\mathcal{D}}^*(x):=\arg\min_{\beta\in\mathcal{D}}g_{\tau,m}(x,\beta). g τ , m , D ∗ ( x ) := β ∈ D min g τ , m ( x , β ) , A τ , m , D ∗ ( x ) := arg β ∈ D min g τ , m ( x , β ) .
The minimum exists by continuity and compactness. The actor map remains defined on R r \mathbb{R}^r R r ; D \mathcal{D} D specifies the comparison domain for global lower optimality.
Proof of Corollary A.2. Proposition B.1, with c = 0 c = 0 c = 0 , gives a ( x ) ∈ B R g e o m / 4 ⊂ D a(x) \in B_{R_{\mathrm{geom}}/4} \subset D a ( x ) ∈ B R geom /4 ⊂ D and ∇ α g τ , m ( x , a ( x ) ) = 0 \nabla_{\alpha}g_{\tau,m}(x, a(x)) = 0 ∇ α g τ , m ( x , a ( x )) = 0 for every x x x . Assumption A.1 therefore yields
0 ≤ g τ , m ( x , a ( x ) ) − g τ , m , D ∗ ( x ) ≤ ∥ ∇ α g τ , m ( x , a ( x ) ) ∥ 2 2 μ P L = 0. (285) 0 \le g_{\tau,m}(x, a(x)) - g_{\tau,m,D}^{*}(x) \le\frac{\lVert\nabla_{\alpha}g_{\tau,m}(x, a(x))\rVert^{2}}{2\mu_{\mathrm{PL}}} = 0. \tag*{(285)} 0 ≤ g τ , m ( x , a ( x )) − g τ , m , D ∗ ( x ) ≤ 2 μ PL ∥ ∇ α g τ , m ( x , a ( x )) ∥ 2 = 0. (285)
Thus a ( x ) ∈ A τ , m , D ∗ ( x ) a(x) \in A_{\tau,m,D}^{*}(x) a ( x ) ∈ A τ , m , D ∗ ( x ) . The selected global-response objective Φ τ , m , D s e l ( x ) = f m ( x , a ( x ) ) \Phi_{\tau,m,D}^{\mathrm{sel}}(x) = f_m(x, a(x)) Φ τ , m , D sel ( x ) = f m ( x , a ( x )) equals Φ τ , m ( x ) \Phi_{\tau,m}(x) Φ τ , m ( x ) everywhere. Smoothness follows from the existing response construction, and their gradients agree. Theorem 4.6 , applied to the same algorithm and output, therefore gives expected squared gradient norm O ( ϵ ) O(\epsilon) O ( ϵ ) with O ~ ( ϵ − 2 ) \widetilde{O}(\epsilon^{-2}) O ( ϵ − 2 ) segments and transitions, and its unchanged label and numerical KL-solve costs.
Domain, constants, and algorithmic scope. The prescribed D D D is independent of x x x , perturbation c c c , critic scale m Q m_Q m Q , and accuracy ϵ \epsilon ϵ . A meaningful extension compares against actors beyond the neighborhood already covered by derived strong convexity. The additional PL condition uses the true fixed-τ \tau τ lower objective and its ambient Euclidean gradient, including at boundary points of D D D . It is not a projected-gradient condition and need not certify arbitrary constrained stationary points. The proof evaluates it at the interior, zero-gradient response. Compactness and inclusion of that response suffice for the identification argument; convexity is a convenient domain convention.
No projection onto D D D or new global lower solver is introduced. All initializations, local containment and failure-output rules, sampling blocks, parameter choices, and finite KL solves remain those of Theorem 4.6 . In particular, the argument imposes no PL condition on the perturbed objective h c = g τ , m + c f m h_c = g_{\tau,m} + c f_m h c = g τ , m + c f m and requires no critic realizability or training beyond the neighborhoods in C1–C2. Those conditions continue to justify the estimator locally. The extra assumption changes the interpretation of the selected response, not the algorithm or its distribution of outputs.
For a fixed architecture, μ P L \mu_{\mathrm{PL}} μ PL may depend on m m m , D D D , and τ \tau τ . It enters no step-size or sample-size choice in this corollary, so no new inverse-PL factor is introduced into the inherited interaction bound. Positivity separately at each x x x would already suffice in (285) ; uniformity in x x x is the stated stronger condition. If widths are instead selected as m = m ( ϵ ) m = m(\epsilon) m = m ( ϵ ) , impose the condition for every architecture used on an accuracy-independent domain and retain the target notation Φ τ , m ( ϵ ) , D s e l \Phi_{\tau,m(\epsilon),D}^{\mathrm{sel}} Φ τ , m ( ϵ ) , D sel . The argument does not identify targets at different widths, and does not need a width-uniform PL constant. The pointwise certificate
g τ , m ( x , α ) − g τ , m , D ∗ ( x ) ≤ ( 2 μ P L ) − 1 ∥ ∇ α g τ , m ( x , α ) ∥ 2 , α ∈ D , g_{\tau,m}(x,\alpha) - g_{\tau,m,D}^{*}(x) \le(2\mu_{\mathrm{PL}})^{-1}\lVert\nabla_{\alpha}g_{\tau,m}(x,\alpha)\rVert^{2}, \qquad\alpha\in D, g τ , m ( x , α ) − g τ , m , D ∗ ( x ) ≤ ( 2 μ PL ) − 1 ∥ ∇ α g τ , m ( x , α ) ∥ 2 , α ∈ D ,
would require a separate residual estimate to give a finite-sample lower-suboptimality bound for an approximate actor output.
Selection independence and optimization target. PL alone does not generally guarantee uniqueness of a lower minimizer. The corollary retains the specified smooth selection a ( x ) a(x) a ( x ) . If, in addition, A τ , m , D ∗ ( x ) A_{\tau,m,D}^{*}(x) A τ , m , D ∗ ( x ) is a singleton for every x x x , that selection is the unique global lower response. A weaker sufficient condition for a selection-independent upper objective is upper-loss invariance:
f m ( x , α ) = f m ( x , β ) for all α , β ∈ A τ , m , D ∗ ( x ) , for every x . (286) f_m(x,\alpha) = f_m(x,\beta) \quad\text{for all } \alpha,\beta\in A_{\tau,m,D}^{*}(x), \quad\text{for every } x. \tag*{(286)} f m ( x , α ) = f m ( x , β ) for all α , β ∈ A τ , m , D ∗ ( x ) , for every x . (286)
Then every choice of global lower solution induces the same scalar objective, which is smooth because it equals Φ τ , m \Phi_{\tau,m} Φ τ , m , even if some selections themselves are nonsmooth. Without either condition or another selection argument, the corollary does not identify its objective with the optimistic or pessimistic upper objective obtained by minimizing or maximizing f m f_m f m over the lower solution set.
The conclusion is upper stationarity for a selected global regularized lower response within the declared actor class and domain. Matching another bilevel target additionally requires matching the policy class, domain, regularization, and selection convention. It gives no unregularized-response comparison or guarantee from arbitrary actor initialization. In particular, τ \tau τ stays at the fixed value required for the original geometry; the analysis does not permit τ ↓ 0 \tau\downarrow0 τ ↓ 0 . PL only for the unregularized lower objective would not justify (285) .
Why a full-space PL assumption is unavailable here. The bounded activations and fixed readout in (81) and (93) imply, at every branch parameter, ∣ G m , ϑ ( s , a ) ∣ ≤ m L M , L , 0 \lvert G_{m,\vartheta}(s,a)\rvert\le\sqrt{m}L_{M,L,0} ∣ G m , ϑ ( s , a )∣ ≤ m L M , L , 0 . Consequently the antithetic energy (86) obeys
∣ F m , α ( s , a ) ∣ ≤ B F , m : = 2 m L M , L , 0 < ∞ ( α ∈ R r ) . (287) \lvert F_{m,\alpha}(s,a)\rvert\le B_{F,m} := \sqrt{2m}L_{M,L,0} < \infty\qquad(\alpha\in\mathbb{R}^{r}). \tag*{(287)} ∣ F m , α ( s , a )∣ ≤ B F , m := 2 m L M , L , 0 < ∞ ( α ∈ R r ) . (287)
For the exponential tilt (88) , (89) gives ∣ A m , s ( α ) ∣ ≤ B F , m \lvert A_{m,s}(\alpha)\rvert\le B_{F,m} ∣ A m , s ( α )∣ ≤ B F , m , and hence ∣ log ( π m , α / π r e f ) ∣ ≤ 2 B F , m \lvert\log(\pi_{m,\alpha}/\pi_{\mathrm{ref}})\rvert\le2B_{F,m} ∣ log ( π m , α / π ref )∣ ≤ 2 B F , m . The expected log ratio at each state is a nonnegative KL divergence. The reward bound and normalized discounting in (2) therefore yield
− B r ≤ g τ , m ( x , α ) ≤ B r + 2 τ B F , m for all x , α . (288) -B_r \le g_{\tau,m}(x,\alpha) \le B_r + 2\tau B_{F,m} \qquad\text{for all } x,\alpha. \tag*{(288)} − B r ≤ g τ , m ( x , α ) ≤ B r + 2 τ B F , m for all x , α . (288)
This bound needs fixed finite widths, but no uniform positive lower bound on reference action probabilities across states. For fixed x x x , the objective is C 2 C^2 C 2 on R r \mathbb{R}^{r} R r : bounded inputs, bounded activation derivatives, and finite layer weights give uniform actor and score derivative bounds on each compact parameter set. Differentiated length-t t t trajectory likelihoods have polynomial-in-t t t bounds, which are summable with γ t \gamma^t γ t , justifying twice differentiating the discounted expectation locally.
Lemma C.78 (Bounded objectives obstruct full-space uniform PL). Let v : R r → R v:\mathbb{R}^{r}\to\mathbb{R} v : R r → R be C 2 C^2 C 2 and bounded above, with v ∗ : = inf z ∈ R r v ( z ) > − ∞ v_* := \inf_{z\in\mathbb{R}^{r}} v(z) > -\infty v ∗ := inf z ∈ R r v ( z ) > − ∞ . If there is μ > 0 \mu>0 μ > 0 such that 1 2 ∥ ∇ v ( z ) ∥ 2 ≥ μ [ v ( z ) − v ∗ ] \frac{1}{2}\lVert\nabla v(z)\rVert^2 \ge\mu[v(z)-v_*] 2 1 ∥ ∇ v ( z ) ∥ 2 ≥ μ [ v ( z ) − v ∗ ] for every z ∈ R r z\in\mathbb{R}^{r} z ∈ R r , then v v v is constant.
Proof. If v v v is nonconstant, choose z 0 z_0 z 0 with v ( z 0 ) > v ∗ v(z_0)>v_* v ( z 0 ) > v ∗ and consider the maximal solution of z ˙ = ∇ v ( z ) \dot{z}=\nabla v(z) z ˙ = ∇ v ( z ) from z 0 z_0 z 0 . Local existence and uniqueness follow from C 2 C^2 C 2 regularity. With M = sup v < ∞ M=\sup v<\infty M = sup v < ∞ ,
∫ 0 t ∥ ∇ v ( z ( u ) ) ∥ 2 d u = v ( z ( t ) ) − v ( z 0 ) ≤ M − v ( z 0 ) , ∥ z ( t ) − z ( s ) ∥ ≤ t − s M − v ( z 0 ) \int_0^t \lVert\nabla v(z(u))\rVert^2\,du = v(z(t))-v(z_0)\le M-v(z_0), \qquad\lVert z(t)-z(s)\rVert\le\sqrt{t-s}\sqrt{M-v(z_0)} ∫ 0 t ∥ ∇ v ( z ( u )) ∥ 2 d u = v ( z ( t )) − v ( z 0 ) ≤ M − v ( z 0 ) , ∥ z ( t ) − z ( s )∥ ≤ t − s M − v ( z 0 )
for 0 ≤ s < t 0\le s<t 0 ≤ s < t in the solution interval. If the maximal forward time were finite, the second estimate would give a finite limit for z ( t ) z(t) z ( t ) there; local existence at that limit would extend the solution. Thus it exists for all t ≥ 0 t\ge0 t ≥ 0 . But e ( t ) = v ( z ( t ) ) − v ∗ > 0 e(t)=v(z(t))-v_*>0 e ( t ) = v ( z ( t )) − v ∗ > 0 satisfies e ˙ ( t ) = ∥ ∇ v ( z ( t ) ) ∥ 2 ≥ 2 μ e ( t ) \dot{e}(t)=\lVert\nabla v(z(t))\rVert^2\ge2\mu e(t) e ˙ ( t ) = ∥ ∇ v ( z ( t )) ∥ 2 ≥ 2 μ e ( t ) , so e ( t ) ≥ e ( 0 ) e 2 μ t e(t)\ge e(0)e^{2\mu t} e ( t ) ≥ e ( 0 ) e 2 μ t . This contradicts e ( t ) ≤ M − v ∗ e(t)\le M-v_* e ( t ) ≤ M − v ∗ . ▫
Apply Lemma C.78 to v ( α ) = g τ , m ( x , α ) v(\alpha)=g_{\tau,m}(x,\alpha) v ( α ) = g τ , m ( x , α ) at any fixed x x x . Equation (288) supplies both bounds, while Proposition B.1’s positive local Hessian makes v v v nonconstant for r ≥ 1 r\ge1 r ≥ 1 . A positive full-space uniform Euclidean PL constant is therefore impossible for this construction. Replacing D \mathcal{D} D by R r \mathbb{R}^{r} R r in Assumption A.1 would make the conditional extension vacuous in these nontrivial instances.
Compatibility on compact domains. The obstruction does not apply when ascent can leave the comparison domain. For illustration, v ( a ) = tanh 2 ( a ) v(a)=\tanh^2(a) v ( a ) = tanh 2 ( a ) satisfies on any [ − A , A ] [-A,A] [ − A , A ] , 0 < A < ∞ 0<A<\infty 0 < A < ∞ ,
1 2 ∣ v ′ ( a ) ∣ 2 = 2 sech 4 ( a ) v ( a ) ≥ 2 sech 4 ( A ) [ v ( a ) − min b ∈ [ − A , A ] v ( b ) ] . (289) \frac{1}{2}\lvert v'(a)\rvert^2 = 2\operatorname{sech}^4(a)v(a) \ge2\operatorname{sech}^4(A)\left[v(a)-\min_{b\in[-A,A]}v(b)\right]. \tag*{(289)} 2 1 ∣ v ′ ( a ) ∣ 2 = 2 sech 4 ( a ) v ( a ) ≥ 2 sech 4 ( A ) [ v ( a ) − b ∈ [ − A , A ] min v ( b ) ] . (289)
It is smooth, globally bounded, and nonconstant, with positive curvature near its minimum and a positive PL constant on each fixed compact interval. This illustrates compatibility with bounded nonlinearities; it does not verify Assumption A.1 for the RL examples.
A separate condition for upper optimality. Set V = Φ τ , m , D s e l V = \Phi^{\mathrm{sel}}_{\tau,m,\mathcal{D}} V = Φ τ , m , D sel . If a separately specified upper domain X \mathcal{X} X contains x o u t x_{\mathrm{out}} x out almost surely and, for V ∗ : = inf x ∈ X V ( x ) > − ∞ V_* := \inf_{x\in\mathcal{X}} V(x) > -\infty V ∗ := inf x ∈ X V ( x ) > − ∞ , one additionally has
1 2 ∥ ∇ V ( x ) ∥ 2 ≥ μ u p [ V ( x ) − V ∗ ] ( x ∈ X ) , μ u p > 0 , \frac{1}{2}\lVert\nabla V(x)\rVert^2 \ge\mu_{\mathrm{up}}[V(x)-V_*]\quad(x\in\mathcal{X}),\qquad\mu_{\mathrm{up}}>0, 2 1 ∥ ∇ V ( x ) ∥ 2 ≥ μ up [ V ( x ) − V ∗ ] ( x ∈ X ) , μ up > 0 ,
then taking expectations gives E [ V ( x o u t ) − V ∗ ] = O ( ϵ / μ u p ) \mathbb{E}[V(x_{\mathrm{out}})-V_*] = O(\epsilon/\mu_{\mathrm{up}}) E [ V ( x out ) − V ∗ ] = O ( ϵ / μ up ) . Neither the upper PL condition nor output membership in such a domain is asserted here. They do not follow from lower PL. In fact, Corollary B.2 gives a bounded C 2 C^2 C 2 upper objective, so Lemma C.78 also precludes positive full-space upper PL when that objective is nonconstant.
Continuous-state experimental details
This section specifies the sampled continuous-state experiment and its population diagnostics. The structural construction and its C 1 C^1 C 1 –C 2 C^2 C 2 verification appear in Appendix § 10 .11, especially Lemmas C.69 and C.70. We use the two-hidden-layer actor in Remark C.68. The code was ran on a 12700k processor with an RTX 5090 and 32GB RAM.
Instance, parameterization, and reference calculations
Let I − = [ − 1 , 0 ) I_-=[-1,0) I − = [ − 1 , 0 ) , I + = [ 0 , 1 ] I_+=[0,1] I + = [ 0 , 1 ] , and μ i = Unif ( I i ) \mu_i=\operatorname{Unif}(I_i) μ i = Unif ( I i ) . The reset distribution is uniform on [ − 1 , 1 ] [-1,1] [ − 1 , 1 ] , and the transition law is
P ( d s ′ ∣ s , a ) = ( 1 2 + a s 4 ) μ + ( d s ′ ) + ( 1 2 − a s 4 ) μ − ( d s ′ ) , a ∈ { − 1 , + 1 } . P(\mathrm{d}s'\mid s,a)=\left(\frac{1}{2}+\frac{as}{4}\right)\mu_+(\mathrm{d}s')+\left(\frac{1}{2}-\frac{as}{4}\right)\mu_-(\mathrm{d}s'),\qquad a\in\{-1,+1\}. P ( d s ′ ∣ s , a ) = ( 2 1 + 4 a s ) μ + ( d s ′ ) + ( 2 1 − 4 a s ) μ − ( d s ′ ) , a ∈ { − 1 , + 1 } .
We use γ = 1 / 2 \gamma=1/2 γ = 1/2 , reward r x ( s , a ) = tanh ( x ) a s r_x(s,a)=\tanh(x)as r x ( s , a ) = tanh ( x ) a s , and reference-KL coefficient τ = 2 \tau=2 τ = 2 with a uniform reference policy. The lower objective is the normalized discounted cost with q x , α = − r x + τ log ( π α / π r e f ) q_{x,\alpha}=-r_x+\tau\log(\pi_\alpha/\pi_{\mathrm{ref}}) q x , α = − r x + τ log ( π α / π ref ) . Two independent length-two state–action trajectories define D = ∑ t = 0 1 ( a t 1 s t 1 − a t 2 s t 2 ) D=\sum_{t=0}^{1}(a_t^1s_t^1-a_t^2s_t^2) D = ∑ t = 0 1 ( a t 1 s t 1 − a t 2 s t 2 ) ; the fixed feedback kernel is Y ∣ D ∼ Bernoulli ( σ ( D / 2 ) ) Y\mid D\sim\operatorname{Bernoulli}(\sigma(D/2)) Y ∣ D ∼ Bernoulli ( σ ( D /2 )) . The preference loss is
L ( x ; D , Y ) = log ( 1 + exp ( tanh ( x ) D ) ) − Y tanh ( x ) D . L(x;D,Y)=\log(1+\exp(\tanh(x)D))-Y\tanh(x)D. L ( x ; D , Y ) = log ( 1 + exp ( tanh ( x ) D )) − Y tanh ( x ) D .
The upper objective Φ ( x ) = f ( x , α 0 ∗ ( x ) ) \Phi(x)=f(x,\alpha_0^*(x)) Φ ( x ) = f ( x , α 0 ∗ ( x )) averages this loss over trajectories generated by the lower response. There is no upper projection or added quadratic upper regularizer.
The tied actor has two hidden layers of width eight and two trainable chart coordinates. Its logits are a H 4 ( α i ( s ) ) aH_4(\alpha_i(s)) a H 4 ( α i ( s )) , where
H 4 ( v ) = tanh ( ( 1 + t ) tanh t ) + tanh ( ( 1 − t ) tanh t ) , t = v / ( 2 8 ) . H_4(v)=\tanh((1+t)\tanh t)+\tanh((1-t)\tanh t),\qquad t=v/(2\sqrt{8}). H 4 ( v ) = tanh (( 1 + t ) tanh t ) + tanh (( 1 − t ) tanh t ) , t = v / ( 2 8 ) .
Both hidden layers vary with these coordinates. The critic is Q θ ( s , a ) = m Q tanh ( φ ( s , a ) ⊤ θ / m Q ) Q_\theta(s,a)=\sqrt{m_Q}\tanh(\varphi(s,a)^\top\theta/\sqrt{m_Q}) Q θ ( s , a ) = m Q tanh ( φ ( s , a ) ⊤ θ / m Q ) . The five fixed features comprise four region–action indicators and a 12 ( s − i ( s ) / 2 ) \sqrt{12}(s-i(s)/2) 12 ( s − i ( s ) /2 ) , with region sign i ( s ) ∈ { − 1 , + 1 } i(s)\in\{-1,+1\} i ( s ) ∈ { − 1 , + 1 } . Thus m Q m_Q m Q varies the critic’s nonlinearity while its parameter dimension remains five. Training samples continuous states without a state grid; the two-region structure permits separate deterministic reference calculations.
The learned variants use old-coordinate updates, independent random streams for the update blocks, no critic warmup, and no population targets. Exact responses, gradients, and TD roots are evaluated after training or in explicitly identified population studies.
Connection to the theoretical sample complexity
The ordinate in Figure 1 (a) matches the stationarity criterion of Theorem 4.6 . For a successful run and an independent index J J J uniform on { 0 , … , T − 1 } \{0,\ldots,T-1\} { 0 , … , T − 1 } ,
E J [ ∣ Φ ′ ( x J ) ∣ 2 ∣ x 0 , … , x T − 1 ] = 1 T ∑ t = 0 T − 1 ∣ Φ ′ ( x t ) ∣ 2 = G ‾ T . \mathbb{E}_{J}\left[\left\lvert\Phi'(x_J)\right\rvert^2 \mid x_0,\ldots,x_{T-1}\right] = \frac{1}{T}\sum_{t=0}^{T-1}\left\lvert\Phi'(x_t)\right\rvert^2 = \overline{G}_T. E J [ ∣ Φ ′ ( x J ) ∣ 2 ∣ x 0 , … , x T − 1 ] = T 1 t = 0 ∑ T − 1 ∣ Φ ′ ( x t ) ∣ 2 = G T .
Thus averaging the displayed run-level statistic across training seeds estimates the theorem’s uniform-output squared-gradient criterion; the prescribed fallback accounts for failed runs as described above. The theoretical relation
E ∥ ∇ Φ ( x o u t ) ∥ 2 = O ( ε ) , N = O ~ ( ε − 2 ) \mathbb{E}\left\lVert\nabla\Phi(x_{\mathrm{out}})\right\rVert^2 = O(\varepsilon), \qquad N = \widetilde{O}(\varepsilon^{-2}) E ∥ ∇Φ ( x out ) ∥ 2 = O ( ε ) , N = O ( ε − 2 )
has leading polynomial dependence N − 1 / 2 N^{-1/2} N − 1/2 when expressed as squared stationarity versus interaction budget, up to logarithmic factors. This is the reference slope shown in the main-text plot.
The experimental schedule also follows the theorem’s powers of the iteration budget. Writing ε T ≍ T − 1 / 2 \varepsilon_T \asymp T^{-1/2} ε T ≍ T − 1/2 gives
η x , η y , β , c 2 , m Q − 2 ≍ T − 1 / 2 , Ψ 0 η x T = O ( T − 1 / 2 ) , \eta_x,\eta_y,\beta,c^2,m_Q^{-2} \asymp T^{-1/2}, \qquad\frac{\Psi_0}{\eta_x T}=O(T^{-1/2}), η x , η y , β , c 2 , m Q − 2 ≍ T − 1/2 , η x T Ψ 0 = O ( T − 1/2 ) ,
for a uniformly bounded initial potential Ψ 0 \Psi_0 Ψ 0 . These are the terms in the stationarity bound (23) , with the remaining truncation and certificate terms controlled at the corresponding accuracy in the theorem. With fixed batches and geometric rollout lengths of bounded mean, charged interactions grow proportionally to T T T in expectation, apart from the initial metric cost. Accordingly, the fitted slopes near − 0.505 -0.505 − 0.505 and nearly constant N G ‾ T \sqrt{N}\,\overline{G}_T N G T provide finite-budget support for the predicted interaction scaling. An eightfold increase in budget predicts a reduction by approximately 8 = 2.828 \sqrt{8}=2.828 8 = 2.828 in this metric; the observed reductions are 2.856 and 2.859.
Role of the two critic initializations
The two curves use the same actor and reward initialization, matched training seeds, interaction budgets, and update schedules. They differ in the initial critic coordinates. The zero-initialization curve starts from u 0 = v 0 = 0 u_0=v_0=0 u 0 = v 0 = 0 , which represents the exact values at x 0 = U 0 = S 0 = 0 x_0=U_0=S_0=0 x 0 = U 0 = S 0 = 0 . The displaced-initialization curve starts from ∥ u 0 ∥ = 2 \lVert u_0\rVert=2 ∥ u 0 ∥ = 2 and ∥ v 0 ∥ = 4 \lVert v_0\rVert=4 ∥ v 0 ∥ = 4 , so its initial critics are inaccurate. Both settings subsequently train the critics through sampled TD updates while the reward and actor parameters evolve.
Theorem 4.6 permits any deterministic critic initialization in K Q \mathcal{K}_Q K Q . Initial critic error contributes to Ψ 0 \Psi_0 Ψ 0 in (23) ; with a displacement fixed across budgets, its contribution Ψ 0 / ( η x T ) \Psi_0/(\eta_x T) Ψ 0 / ( η x T ) retains the T − 1 / 2 T^{-1/2} T − 1/2 dependence. The theorem therefore allows initialization-dependent constants with the same interaction exponent. Comparing the two curves checks whether the predicted decay is also observed when training begins with inaccurate value estimates. Their similar slopes support this aspect of the guarantee, while the critic-error measurements in Section D.2 directly document recovery from the displaced initialization.
Critic initialization and recovery
We evaluate persistent critic learning from zero and displaced initializations using the full update in both cases. The fixed-budget study uses T = 2000 T=2000 T = 2000 , batch size 256 in each sampled block, geometric cap H = 32 H=32 H = 32 , c = 0.05 c=0.05 c = 0.05 , m Q = 256 m_Q=256 m Q = 256 , TD step β = 0.2 \beta=0.2 β = 0.2 , and actor and upper steps η y = η x = 0.02 \eta_y=\eta_x=0.02 η y = η x = 0.02 . The initial metric uses 4722 reset states; critic radii are R u = 4 R_u=4 R u = 4 and R v = 8 R_v=8 R v = 8 . The two initializations use matched seeds 100–109, with ten runs per initialization. All 20 runs complete.
At x 0 = U 0 = S 0 = 0 x_0 = U_0 = S_0 = 0 x 0 = U 0 = S 0 = 0 , zero critics represent the exact initial values. The displaced initialization has ∥ u 0 ∥ = 2 \lVert u_0\rVert= 2 ∥ u 0 ∥ = 2 and ∥ v 0 ∥ = 4 \lVert v_0\rVert= 4 ∥ v 0 ∥ = 4 , in fixed directions shared across seeds. Both settings train the critics from the first iteration, without critic warmup or population targets in the updates.
For evaluation, population TD roots at the current reward and actor parameters supply ( u ∗ , v ∗ ) (u^*, v^*) ( u ∗ , v ∗ ) . The weighted parameter tracking error is 100 ∥ u − u ∗ ∥ 2 + ∥ v − v ∗ ∥ 2 100\lVert u-u^*\rVert^2 + \lVert v-v^*\rVert^2 100 ∥ u − u ∗ ∥ 2 + ∥ v − v ∗ ∥ 2 , evaluated at 26 saved reference checkpoints per run. Table 4 reports initial and final seed-averaged errors for the displaced initialization. Value errors are measured against the true regularized values. The base metric is the squared sup-norm error; the paired metric is the squared sup-norm of the endpoint-error difference after division by c c c .
Table 4. Critic recovery from displaced initialization. Entries are means over ten training seeds; population reference calculations are used only for evaluation.
The reductions in both base and normalized paired value errors show recovery from initially inaccurate critics on this continuous-state instance while the reward and actor parameters evolve.
End-to-end continuous-state benchmark details
This section describes Walker Walk and Highway-fast. They both have continuous observations; Highway-fast has a finite action set. The separate continuous-state theory-aligned study has its own protocol. These practical SAC experiments evaluate end-to-end learning performance.
Cohort, uncertainty, and interaction accounting
Both figures use 5 training seeds. Each seed contributes the mean of ten held-out evaluation episodes at each checkpoint. Lines average these five means; bands show their sample standard deviation with denominator 5 − 1 5-1 5 − 1 , not a confidence interval or a pooled episode SD. Each preference run uses 100,000 main interactions and 500 labels. Walker Walk additionally uses 9,216 auxiliary response interactions per run. Highway-fast auxiliary interactions are listed below. Known-reward SAC uses 100,000 main interactions and no preference labels or auxiliary responses. The horizontal axis counts total training interactions: main plus cumulative auxiliary response transitions. For each common evaluation checkpoint, the plotted cost is averaged across the five seeds.
Table 5. Highway-fast auxiliary interactions, in addition to 100,000 main interactions per run.
Figure 3. End-to-end deterministic evaluation returns: mean and one sample SD across five matched training seeds, each contributing a ten-episode mean. The horizontal axis includes cumulative auxiliary response interactions; checkpoint costs are averaged across seeds. Walker Walk starts at its first measured checkpoint, and Highway-fast is shown after exploration warm-up.
End-to-end implementation
Environments. Walker Walk uses DeepMind Control, a 24-dimensional observation, a bounded six-dimensional continuous action, and 1000-step episodes with per-step task reward in [ 0 , 1 ] [0, 1] [ 0 , 1 ] . Highway-fast uses highway-fast-v0, with 25 continuous observation features: presence, position ( x , y ) (x, y) ( x , y ) , and velocity ( v x , v y ) (v_x, v_y) ( v x , v y ) for five vehicles. Its five actions are lane left, idle, lane right, faster, and slower. The configuration uses three lanes, 20 traffic vehicles, vehicle density 1, policy frequency 1 Hz, simulation frequency 5 Hz, and at most 50 policy steps per episode. Collisions terminate episodes; offroad_terminal is false. Task rewards are normalized. Returns are undiscounted sums of true environment rewards.
Actor and critics. Walker Walk uses a tanh-squashed diagonal-Gaussian actor with two 1024-unit ReLU layers, outputting action means and log standard deviations; the latter are clamped to [ − 5 , 2 ] [-5, 2] [ − 5 , 2 ] . Its two Q networks each map concatenated state/action through two 1024-unit ReLU layers to a scalar. Highway-fast uses a categorical actor with two 1024-unit ReLU layers and five logits. Each of its two Q networks maps the state through two 1024-unit ReLU layers to five action values. Discrete SAC computes the action expectation exactly. Highway actions are represented by five-dimensional one-hot vectors in the reward model.
Reward model and labels. Both preference methods use an ensemble of three reward networks, each with three 256-unit LeakyReLU layers and a scalar tanh output. Inputs concatenate state and action. Reward Adam learning rate is 3 × 10 − 4 3 \times10^{-4} 3 × 1 0 − 4 ; preference batch size and reward updates per query block are both 50. Segment length is 50 for Walker Walk and 10 for Highway-fast. Synthetic labels use segment task rewards, β = − 1 \beta= -1 β = − 1 , γ teacher = 1 \gamma_{\text{teacher}} = 1 γ teacher = 1 , zero mistake and skip probabilities, and equality-threshold coefficient 0.1. This coefficient scales the teacher’s equality margin; it is not an independent probability of assigning an equal label. Acquisition uses disagreement sampling from a ten-times-larger candidate batch.
Response and upper-update settings. Secant uses perturbation c = 0.1 c = 0.1 c = 0.1 and chain coefficient 1. Each response branch receives 100 shadow-SAC updates and supplies two auxiliary rollout episodes, capped at 256 steps on Walker Walk and 50 on Highway-fast. The 10k feedback block performs preference bootstrapping; nine subsequent blocks apply the upper correction, beginning at 15k. Across those blocks there are 1800 shadow-SAC updates (two branches per block).
Table 6. Common SAC and collection settings. Feedback settings apply to the two preference methods. Evaluation schedules are specified below.
The Gaur et al. [10 ] comparator uses the upper objective G + V ^ b a s e / σ G + \hat{V}_{\mathrm{base}}/\sigma G + V ^ base / σ , with σ = 0.05 \sigma= 0.05 σ = 0.05 on Walker Walk and σ = 0.08 \sigma= 0.08 σ = 0.08 on Highway-fast. This is the upper level loss used for the empirical results in Section F of the work. The rest of the pipeline, including both response branches, the perturbation c = 0.1 c = 0.1 c = 0.1 , their update counts, and auxiliary rollout collection, is retained. Thus c c c configures the retained response construction; it is not substituted for σ \sigma σ in the Gaur upper loss. The value contribution uses the base response’s normalized, truncated discounted learned-reward return and is averaged across reward-ensemble members. The positive sign is retained, including for negative learned returns. σ \sigma σ was chosen via grid search over [ 0.01 , 0.2 ] [0.01, 0.2] [ 0.01 , 0.2 ] .
Evaluation and display. Evaluation uses 10 held-out environment seeds: the Gaussian mean with squashing on Walker Walk and the categorical argmax on Highway-fast. Walker Walk uses saved actors at 25k, 50k, 75k, and 100k. The Walker curve begins at its first measured checkpoint, 25k main interactions; no artificial origin is included. Highway-fast evaluations are recorded at the first episode boundary after each 5k milestone (less than 50 steps after the nominal milestone); the exact step is retained in the data. The figure includes 10k–100k main-interaction checkpoints, displayed at their total training costs and omitting the earlier exploration period. The preference methods’ shared warm-up produces identical early policies within each seed; their upper updates first differ at 15k. Lines join available evaluations without smoothing, jitter, or synthetic intermediate measurements.
Reproduction material. The accompanying code regenerates figures and tables from the archived episode returns and verifies the recorded source hashes. Exact source versions and the support snapshot are supplied for reconstructing each historical training run. The code README distinguishes rebuilding reported figures from rerunning GPU training with the recorded environment.